Skip to content

High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.

feat: add browserbench task runner via PinchTab

Bosch9696ddf43192
diff --git a/dev b/dev
index c9d3ab13..e116a47a 100755
--- a/dev
+++ b/dev
@@ -44,6 +44,7 @@ COMMANDS=(
"bench:πŸ“Š:Benchmark lanes"
" pinchtab:πŸ“Š:Run PinchTab benchmark via Anthropic Messages API"
" agent-browser:πŸ“Š:Run agent-browser benchmark via Anthropic Messages API"
+ " browserbench:πŸ“Š:Run BrowserBench CSV tasks through PinchTab"
"doctor:🩺:Setup dev environment"
"check go:πŸ§ͺ:Go only"
"check dashboard:πŸ§ͺ:Dashboard only"
diff --git a/scripts/dev-bench.sh b/scripts/dev-bench.sh
index 8302f25b..096ee511 100755
--- a/scripts/dev-bench.sh
+++ b/scripts/dev-bench.sh
@@ -6,12 +6,14 @@ usage() {
Usage:
./dev bench pinchtab [options...]
./dev bench agent-browser [options...]
+ ./dev bench browserbench [options...]
Examples:
./dev bench pinchtab --dry-run
ANTHROPIC_API_KEY=... ./dev bench pinchtab --model claude-haiku-4-5-20251001
ANTHROPIC_API_KEY=... ./dev bench pinchtab --groups 0,1,2,3
ANTHROPIC_API_KEY=... ./dev bench agent-browser --max-turns 150
+ ANTHROPIC_API_KEY=... ./dev bench browserbench --tasks 5 --verbose
For optimization baseline (no API keys required):
./dev opt baseline
@@ -30,6 +32,9 @@ case "${mode}" in
pinchtab|agent-browser)
exec go run ./tests/tools/runner --lane "${mode}" --finalize --terse-summary "$@"
;;
+ browserbench)
+ exec go run ./tests/tools/runner browserbench "$@"
+ ;;
-h|--help|help)
usage
;;
diff --git a/skills/pinchtab/SKILL.md b/skills/pinchtab/SKILL.md
index 747a67c4..ed11ead3 100644
--- a/skills/pinchtab/SKILL.md
+++ b/skills/pinchtab/SKILL.md
@@ -120,8 +120,8 @@ pinchtab --server http://localhost:9868 snap -i -c # target a specific instance
### Navigation and tabs
```bash
-pinchtab nav <url> # auto-starts default local server; flags: --snap, --new-tab, --tab <id>, --block-images, --block-ads, --print-tab-id
-pinchtab back | forward | reload # all support --snap, --snap-diff, --text
+pinchtab nav <url> # auto-starts default local server; flags: --snap, --new-tab, --tab <id>, --block-images, --block-ads, --dismiss-banners, --print-tab-id
+pinchtab back | forward | reload # all support --snap, --snap-diff, --text, --dismiss-banners
pinchtab tab # list tabs
pinchtab tab <tab-id> # focus tab
pinchtab nav <url> --new-tab # force another tab
@@ -164,7 +164,7 @@ Guidance:
All interaction commands accept unified selectors (see Selectors above).
```bash
-pinchtab click <selector> # flags: --snap, --snap-diff, --text, --wait-nav, --x/--y (coords), --dialog-action accept|dismiss [--dialog-text "..."]
+pinchtab click <selector> # flags: --snap, --snap-diff, --text, --wait-nav, --dismiss-banners (with --wait-nav), --x/--y (coords), --dialog-action accept|dismiss [--dialog-text "..."]
pinchtab dblclick <selector>
pinchtab mouse move|down|up <selector|x y> # --button left|middle|right
pinchtab mouse wheel <ms> --dx <n> --dy <n>
@@ -183,6 +183,7 @@ Rules:
- **Prefer `--snap-diff`** with `click`, `fill`, `select`, `back`, `forward`, `reload` β€” returns `OK` + only changed elements. Use `--snap` when you need the full snapshot (first nav, major page change).
- Prefer `fill` for form entry; `type` only when the site depends on keystroke events.
- `click --wait-nav` when a click navigates. May return `{"success":true}` or `Error 409: unexpected page navigation` β€” treat 409 as success and verify with fresh `snap`/`text`.
+- `--dismiss-banners` on `nav`/`back`/`forward`/`reload` (and on `click --wait-nav`) runs a best-effort pass that clicks a visible Accept all / Got it / OK / Close / Dismiss button, or removes obvious cookie/consent/dialog/overlay containers. Use when a fresh page-load shows a modal that blocks interaction (typical symptom: `Error 500: action click: element is occluded`). Heuristic β€” can misfire on pages that label legitimate UI as `overlay` or `modal`; not a substitute for an explicit selector when one is known.
- Use low-level `mouse` only for drag handles, canvas widgets, or exact pointer sequences.
- JS dialogs: `--dialog-action accept|dismiss`, `--dialog-text` for `prompt()` responses.
- HTTP scroll action: `"scrollX"`/`"scrollY"` for pixel deltas, `"selector"` to scroll into view β€” `x`/`y` are viewport coords, not deltas.
diff --git a/tests/e2e/scenarios/cli/actions-extended.sh b/tests/e2e/scenarios/cli/actions-extended.sh
index 2a114d4d..f375e2a3 100644
--- a/tests/e2e/scenarios/cli/actions-extended.sh
+++ b/tests/e2e/scenarios/cli/actions-extended.sh
@@ -408,3 +408,138 @@ assert_output_contains "OK" "fill succeeded"
assert_output_contains "~" "snap-diff shows changes"
end_test
+
+# ─────────────────────────────────────────────────────────────────
+start_test "pinchtab press --snap"
+
+pt_ok nav "${FIXTURES_URL}/form.html"
+pt_ok focus --css "#username"
+pt_ok press Tab --snap
+
+# Output should be OK plus an interactive snapshot. The snapshot must mention
+# a textbox (form contains username/email/password inputs); confirms --snap
+# triggered the post-action snapshot fetch on a key press.
+assert_output_contains "OK" "press succeeded"
+assert_output_contains "textbox" "snap output contains form input"
+
+end_test
+
+# ─────────────────────────────────────────────────────────────────
+start_test "pinchtab press --snap-diff"
+
+pt_ok nav "${FIXTURES_URL}/form.html"
+pt_ok focus --css "#username"
+pt_ok snap # establish baseline
+pt_ok press Tab --snap-diff
+
+# Tab key shifts focus from #username to the next input (#email). Snap-diff
+# must show the diff header (+N ~N -N) β€” focus changes mark prior/next
+# elements as modified, so we expect a "~" change marker. Falls back to "+0"
+# header if the platform's accessibility tree doesn't surface focus shifts.
+assert_output_contains "OK" "press succeeded"
+assert_output_contains "+" "snap-diff shows compact diff header"
+
+end_test
+
+# ─────────────────────────────────────────────────────────────────
+# Cookie-banner dismissal β€” exercises the --dismiss-banners flag on the
+# nav/back/forward/reload/click code paths against a fixture page that
+# renders a fixed-position cookie banner with an "Accept all" button.
+# Phase 1 of the dismissal heuristic clicks the labelled button, which
+# triggers the fixture's onclick to set body[data-cookie-dismissed=true].
+# We use that attribute as the post-condition for a successful dismissal.
+# ─────────────────────────────────────────────────────────────────
+
+start_test "pinchtab nav --dismiss-banners clears cookie banner"
+
+pt_ok nav "${FIXTURES_URL}/cookie-banner.html" --dismiss-banners
+
+# The fixture's Accept-all handler sets body[data-cookie-dismissed=true] and
+# adds class=dismissed to #cookie-banner.
+pt_ok eval "document.body.dataset.cookieDismissed"
+assert_output_contains "true" "fixture's Accept-all handler ran"
+
+pt_ok eval "document.getElementById('cookie-banner').classList.contains('dismissed')"
+assert_output_contains "true" "cookie banner is hidden"
+
+end_test
+
+# ─────────────────────────────────────────────────────────────────
+start_test "pinchtab nav (no flag) leaves cookie banner up"
+
+pt_ok nav "${FIXTURES_URL}/cookie-banner.html"
+
+pt_ok eval "document.body.dataset.cookieDismissed || ''"
+# Should be empty β€” the helper did not run, and the user did not click.
+assert_output_not_contains "true" "Accept-all was NOT auto-clicked"
+
+pt_ok eval "document.getElementById('cookie-banner').classList.contains('dismissed')"
+assert_output_contains "false" "cookie banner is still visible"
+
+end_test
+
+# ─────────────────────────────────────────────────────────────────
+start_test "pinchtab reload --dismiss-banners clears re-shown banner"
+
+# Land on the page with the flag β€” the banner is dismissed.
+pt_ok nav "${FIXTURES_URL}/cookie-banner.html" --dismiss-banners
+pt_ok eval "document.body.dataset.cookieDismissed"
+assert_output_contains "true" "initial dismissal landed"
+
+# Reload WITHOUT the flag β€” the banner re-renders fresh, dataset wiped.
+pt_ok reload
+pt_ok eval "document.body.dataset.cookieDismissed || ''"
+assert_output_not_contains "true" "reload re-rendered the banner"
+
+# Reload WITH --dismiss-banners β€” should re-dismiss.
+pt_ok reload --dismiss-banners
+pt_ok eval "document.body.dataset.cookieDismissed"
+assert_output_contains "true" "reload --dismiss-banners cleared it again"
+
+end_test
+
+# ─────────────────────────────────────────────────────────────────
+start_test "pinchtab back --dismiss-banners clears banner on prior page"
+
+pt_ok nav "${FIXTURES_URL}/index.html"
+pt_ok nav "${FIXTURES_URL}/cookie-banner.html"
+pt_ok back --dismiss-banners
+
+# back should land us on index.html β€” banner-less. The dismissal helper
+# is allowed to run (it's a no-op when there are no matching elements);
+# we just verify back navigated and didn't error.
+assert_output_contains "index.html" "back navigated to prior URL"
+
+end_test
+
+# ─────────────────────────────────────────────────────────────────
+start_test "pinchtab forward --dismiss-banners clears banner on next page"
+
+pt_ok nav "${FIXTURES_URL}/index.html"
+pt_ok nav "${FIXTURES_URL}/cookie-banner.html"
+pt_ok back
+pt_ok forward --dismiss-banners
+
+# forward returns to cookie-banner.html β€” the helper should fire the
+# Accept-all click on the freshly-rendered banner.
+pt_ok eval "document.body.dataset.cookieDismissed"
+assert_output_contains "true" "forward --dismiss-banners cleared the banner"
+
+end_test
+
+# ─────────────────────────────────────────────────────────────────
+start_test "pinchtab click --wait-nav --dismiss-banners on landing page"
+
+# Start on a banner-less page that links to the cookie-banner page. Click
+# the link with --wait-nav so the click triggers a navigation; pair with
+# --dismiss-banners so the helper fires after the nav settles.
+pt_ok nav "${FIXTURES_URL}/index.html"
+
+# Inject a link to the cookie-banner fixture so the click triggers a nav.
+pt_ok eval "(() => { const a = document.createElement('a'); a.id = 'goto-cookie'; a.href = '${FIXTURES_URL}/cookie-banner.html'; a.textContent = 'go'; document.body.appendChild(a); return 'ok'; })()"
+pt_ok click --css "#goto-cookie" --wait-nav --dismiss-banners
+
+pt_ok eval "document.body.dataset.cookieDismissed"
+assert_output_contains "true" "click --wait-nav --dismiss-banners cleared the banner"
+
+end_test
diff --git a/tests/tools/browserbench/README.md b/tests/tools/browserbench/README.md
new file mode 100644
index 00000000..d369c224
--- /dev/null
+++ b/tests/tools/browserbench/README.md
@@ -0,0 +1,117 @@
+# BrowserBench Runbook
+
+Hand this file to a sub-agent that needs to run the BrowserBench harness end-to-end without prior context.
+
+## What you're doing
+
+Running info-extraction tasks from the [Halluminate BrowserBench](https://github.com/Halluminate/browserbench) dataset against PinchTab. Each task pre-navigates a tab to a starting URL, then drives an LLM agent through `./scripts/pt` browser primitives until it emits `FINAL_ANSWER:` or hits a turn cap. Output is a CSV with `success`/`agent_result`/`ground_truth` per task plus per-task artifacts (HAR, screenshot, console log, full command transcript).
+
+## Before you run
+
+Verify these in order β€” abort if any fails and report which:
+
+1. `docker info >/dev/null 2>&1` β€” Docker daemon is up.
+2. `[ -n "$ANTHROPIC_API_KEY" ]` (or `OPENAI_API_KEY`) β€” model API key is in env.
+3. `cd /Users/bosh/dev/pinchtab` β€” repo root is the working directory.
+
+## Run the bench
+
+Start with a small slice to confirm the harness works, then scale up:
+
+```bash
+# Smoke (5 tasks, fresh container, live progress)
+./dev bench browserbench --tasks 5 --verbose
+
+# Real run (30 tasks, reuse container so it's faster)
+./dev bench browserbench --tasks 30 --skip-init
+
+# One specific task (debugging)
+./dev bench browserbench --task-id 4 --verbose
+
+# Full dataset (β‰ˆ25–30 min wall time, expensive in tokens)
+./dev bench browserbench --skip-init
+```
+
+The wrapper calls `go run ./tests/tools/runner browserbench`. Both forms accept the same flags (`./dev bench browserbench --help`).
+
+Most-used flags:
+
+| Flag | Purpose |
+|---|---|
+| `--tasks N` | Cap how many tasks run (default: all 295) |
+| `--task-id ID` | Run exactly one task |
+| `--skip-init` | Reuse the existing container instead of `--force-recreate`-ing |
+| `--csv-file PATH` | Override the dataset (point at a local snapshot for reproducibility) |
+| `--max-turns N` | Per-task agent loop budget (default 80) |
+| `--verbose` / `-v` | Stream per-turn tool calls live |
+| `--model NAME` | Override the model (default `claude-haiku-4-5-20251001`) |
+
+## Where the results go
+
+```
+tests/tools/browserbench/results/
+└── pinchtab_browserbench_<TS>.csv # one row per task
+ pinchtab_browserbench_<TS>/ # same name minus extension
+ └── task-<id>/
+ β”œβ”€β”€ commands.ndjson # agent's full transcript
+ └── artifacts/
+ β”œβ”€β”€ final.png network.har console.log
+ β”œβ”€β”€ stealth-status.json
+ └── autosolver.json
+```
+
+CSV cols of interest: `task_id`, `success`, `agent_result`, `ground_truth`, `execution_time`, `error_message`, plus token counters. `success` = case-insensitive substring match between `agent_result` and `ground_truth`.
+
+The whole `results/` tree is gitignored.
+
+## Constraints β€” DO NOT do these
+
+These are the recurring failure modes from prior runs. Build them into any agent prompt you write or sub-agent you spawn:
+
+- **`./scripts/pt solve` does not exist.** The autosolver runs server-side only; there is no CLI subcommand. Don't attempt it.
+- **`pt instances`, `pt instance ...`, `pt server`, `pt config show`, reading `~/.pinchtab/config.json`** all return `403 session_scope_forbidden` inside the bench session. Don't probe them.
+- **Don't retry a click that failed with `element is occluded: top=...`.** Either dismiss the overlay (`pt nav --dismiss-banners`, see SKILL.md) or take a different path.
+- **Don't navigate off the starting domain** unless the task naturally leads there.
+
+## Reading the result
+
+After the run finishes:
+
+```bash
+# pass rate + which tasks failed
+python3 - tests/tools/browserbench/results/pinchtab_browserbench_*.csv <<'PY'
+import csv, sys, glob
+path = sorted(glob.glob(sys.argv[1]))[-1]
+rows = list(csv.DictReader(open(path)))
+ok = sum(1 for r in rows if r["success"] == "true")
+print(f"{ok}/{len(rows)} passed ({100*ok/len(rows):.1f}%) -- {path}")
+for r in rows:
+ if r["success"] != "true":
+ print(f" fail task={r['task_id']:>3} err={r['error_message'][:60]!r} url={r['starting_url']}")
+PY
+```
+
+For per-task diagnosis, the most informative artifact is `task-<id>/commands.ndjson` β€” every shell call the agent made, with stdout/stderr and exit codes. Look for repeated `unknown command "solve"` (prompt regression), `element is occluded` (banner problem), or `Just a moment...` (Cloudflare wall).
+
+## Report back
+
+When you finish, surface to the caller:
+
+1. The aggregate pass rate (`X/N passed`).
+2. The failing task IDs and the dominant failure mode for each (CAPTCHA, IP block, occlusion, timeout, etc.).
+3. The output CSV path so they can diff it against earlier runs.
+
+## Known un-winnable tasks
+
+These will fail under any client-side config β€” flag them rather than burning turns:
+
+- **IP / geo blocks** (need residential proxy): tasks 2 (dickssportinggoods), 98 (apartments), 112 (homes).
+- **Cloudflare Turnstile captchas** (need LLM-fallback or a paid solver API key): tasks 86 (studocu), 99 (bbb), 104 (doordash), 248 (hinative), 274 (cambridge), 276 (cars).
+
+Other failures are agent/prompt issues and worth investigating.
+
+## Reference
+
+- Container config: `tests/tools/config/pinchtab-benchmark.json` β€” `instanceDefaults.stealthLevel`, `autoSolver.*`. Edits applied on next bench run (or `docker compose ... up -d --force-recreate --no-deps pinchtab` from `tests/tools/`).
+- Skill the bench prompt loads: `skills/pinchtab/SKILL.md` (read this if you're spawning your own LLM agents β€” describes the full `./scripts/pt` surface).
+- Runner code: `tests/tools/runner/internal/bench/browserbench.go`.
diff --git a/tests/tools/config/pinchtab-benchmark.json b/tests/tools/config/pinchtab-benchmark.json
index 8dd76bed..bd7df4d0 100644
--- a/tests/tools/config/pinchtab-benchmark.json
+++ b/tests/tools/config/pinchtab-benchmark.json
@@ -6,9 +6,18 @@
},
"instanceDefaults": {
"maxTabs": 20,
- "stealthLevel": "light",
+ "stealthLevel": "full",
"headless": true
},
+ "autoSolver": {
+ "enabled": true,
+ "autoTrigger": true,
+ "triggerOnNavigate": false,
+ "triggerOnAction": true,
+ "maxAttempts": 3,
+ "solverTimeoutSec": 30,
+ "solvers": ["cloudflare", "semantic", "jschallenge"]
+ },
"multiInstance": {
"strategy": "always-on",
"allocationPolicy": "fcfs",
diff --git a/tests/tools/runner/internal/bench/browserbench.go b/tests/tools/runner/internal/bench/browserbench.go
new file mode 100644
index 00000000..1f7317b5
--- /dev/null
+++ b/tests/tools/runner/internal/bench/browserbench.go
@@ -0,0 +1,724 @@
+package bench
+
+import (
+ "encoding/csv"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "net/http"
+ "os"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "time"
+)
+
+const defaultBrowserBenchCSV = "https://raw.githubusercontent.com/Halluminate/browserbench/main/browserbench.csv"
+
+var errBrowserBenchHelp = errors.New("browserbench help")
+
+type BrowserBenchArgs struct {
+ Provider Provider
+ Model string
+ CSVFile string
+ Output string
+ Tasks int
+ TaskID string
+ SkipInit bool
+ Verbose bool
+ MaxTokens int
+ Temperature float64
+ MaxTurns int
+ TimeoutSeconds int
+ TurnDelayMs int
+ MaxInputTokens int
+ MaxOutputTokens int
+}
+
+type BrowserBenchTask struct {
+ TaskID string
+ StartingURL string
+ TaskDescription string
+ GroundTruthURL string
+ GroundTruth string
+}
+
+type BrowserBenchRow struct {
+ TaskID string
+ StartingURL string
+ TaskDescription string
+ GroundTruthURL string
+ GroundTruth string
+ Provider string
+ Timestamp string
+ Success bool
+ ErrorMessage string
+ AgentResult string
+ SessionURL string
+ ExecutionTimeSeconds string
+ PinchTabVersion string
+ StealthLevel string
+ SolverUsed string
+ SolveAttempts string
+ HARPath string
+ ScreenshotPath string
+ ConsoleLogPath string
+ CommandLogPath string
+ RequestCount string
+ InputTokens string
+ OutputTokens string
+ CacheCreationInputTokens string
+ CacheReadInputTokens string
+ TotalInputTokens string
+ TotalTokens string
+}
+
+func defaultBrowserBenchArgs() BrowserBenchArgs {
+ base := defaultArgs()
+ return BrowserBenchArgs{
+ CSVFile: defaultBrowserBenchCSV,
+ MaxTokens: base.MaxTokens,
+ Temperature: base.Temperature,
+ MaxTurns: 80,
+ TimeoutSeconds: base.TimeoutSeconds,
+ TurnDelayMs: 750,
+ MaxInputTokens: 0,
+ MaxOutputTokens: 0,
+ }
+}
+
+func parseBrowserBenchArgs(argv []string) (BrowserBenchArgs, error) {
+ a := defaultBrowserBenchArgs()
+ next := func(i *int, name string) (string, error) {
+ *i++
+ if *i >= len(argv) {
+ return "", fmt.Errorf("%s requires a value", name)
+ }
+ return argv[*i], nil
+ }
+
+ for i := 0; i < len(argv); i++ {
+ switch argv[i] {
+ case "--help", "-h":
+ return a, errBrowserBenchHelp
+ case "--provider":
+ v, err := next(&i, argv[i])
+ if err != nil {
+ return a, err
+ }
+ a.Provider = Provider(v)
+ case "--model":
+ v, err := next(&i, argv[i])
+ if err != nil {
+ return a, err
+ }
+ a.Model = v
+ case "--csv-file":
+ v, err := next(&i, argv[i])
+ if err != nil {
+ return a, err
+ }
+ a.CSVFile = v
+ case "--output":
+ v, err := next(&i, argv[i])
+ if err != nil {
+ return a, err
+ }
+ a.Output = v
+ case "--tasks":
+ v, err := next(&i, argv[i])
+ if err != nil {
+ return a, err
+ }
+ n, err := strconv.Atoi(v)
+ if err != nil {
+ return a, fmt.Errorf("--tasks: %w", err)
+ }
+ a.Tasks = n
+ case "--task-id":
+ v, err := next(&i, argv[i])
+ if err != nil {
+ return a, err
+ }
+ a.TaskID = v
+ case "--skip-init":
+ a.SkipInit = true
+ case "--verbose", "-v":
+ a.Verbose = true
+ case "--max-tokens":
+ v, err := next(&i, argv[i])
+ if err != nil {
+ return a, err
+ }
+ n, err := strconv.Atoi(v)
+ if err != nil {
+ return a, fmt.Errorf("--max-tokens: %w", err)
+ }
+ a.MaxTokens = n
+ case "--temperature":
+ v, err := next(&i, argv[i])
+ if err != nil {
+ return a, err
+ }
+ f, err := strconv.ParseFloat(v, 64)
+ if err != nil {
+ return a, fmt.Errorf("--temperature: %w", err)
+ }
+ a.Temperature = f
+ case "--max-turns":
+ v, err := next(&i, argv[i])
+ if err != nil {
+ return a, err
+ }
+ n, err := strconv.Atoi(v)
+ if err != nil {
+ return a, fmt.Errorf("--max-turns: %w", err)
+ }
+ a.MaxTurns = n
+ case "--timeout-seconds":
+ v, err := next(&i, argv[i])
+ if err != nil {
+ return a, err
+ }
+ n, err := strconv.Atoi(v)
+ if err != nil {
+ return a, fmt.Errorf("--timeout-seconds: %w", err)
+ }
+ a.TimeoutSeconds = n
+ case "--turn-delay-ms":
+ v, err := next(&i, argv[i])
+ if err != nil {
+ return a, err
+ }
+ n, err := strconv.Atoi(v)
+ if err != nil {
+ return a, fmt.Errorf("--turn-delay-ms: %w", err)
+ }
+ a.TurnDelayMs = n
+ case "--max-input-tokens":
+ v, err := next(&i, argv[i])
+ if err != nil {
+ return a, err
+ }
+ n, err := strconv.Atoi(v)
+ if err != nil {
+ return a, fmt.Errorf("--max-input-tokens: %w", err)
+ }
+ a.MaxInputTokens = n
+ case "--max-output-tokens":
+ v, err := next(&i, argv[i])
+ if err != nil {
+ return a, err
+ }
+ n, err := strconv.Atoi(v)
+ if err != nil {
+ return a, fmt.Errorf("--max-output-tokens: %w", err)
+ }
+ a.MaxOutputTokens = n
+ default:
+ return a, fmt.Errorf("unknown flag %q", argv[i])
+ }
+ }
+
+ if a.Tasks < 0 {
+ return a, fmt.Errorf("--tasks must be >= 0")
+ }
+ return a, nil
+}
+
+func writeBrowserBenchUsage(w io.Writer) {
+ _, _ = fmt.Fprint(w, `Usage:
+ runner browserbench [options]
+
+Options:
+ --provider anthropic|openai|fake
+ --model MODEL
+ --csv-file PATH_OR_URL BrowserBench CSV (default: Halluminate raw CSV)
+ --output PATH Output CSV path (default: timestamped in tests/tools/browserbench/results)
+ --tasks N Limit number of tasks
+ --task-id ID Run only one task_id
+ --skip-init Reuse existing benchmark container
+ --max-tokens N
+ --temperature N
+ --max-turns N
+ --timeout-seconds N
+ --turn-delay-ms N
+ --max-input-tokens N
+ --max-output-tokens N
+ --verbose, -v
+
+Example:
+ go run ./tests/tools/runner browserbench --provider anthropic --tasks 5 --verbose
+`)
+}
+
+func RunBrowserBench(argv []string, stdout, stderr io.Writer) int {
+ args, err := parseBrowserBenchArgs(argv)
+ if err != nil {
+ if errors.Is(err, errBrowserBenchHelp) {
+ writeBrowserBenchUsage(stdout)
+ return 0
+ }
+ _, _ = fmt.Fprintf(stderr, "runner browserbench: %v\n\n", err)
+ writeBrowserBenchUsage(stderr)
+ return 1
+ }
+
+ provider := resolveProvider(Args{Provider: args.Provider})
+ if err := validateAPIKeys(provider); err != nil {
+ _, _ = fmt.Fprintf(stderr, "runner browserbench: %v\n", err)
+ return 1
+ }
+ model := resolveModel(provider, args.Model)
+ toolsDir := resolveToolsDir()
+
+ if !args.SkipInit {
+ if err := setupPinchtabContainer(toolsDir, stdout, stderr); err != nil {
+ _, _ = fmt.Fprintf(stderr, "runner browserbench: %v\n", err)
+ return 1
+ }
+ }
+
+ tasks, err := loadBrowserBenchTasks(args.CSVFile)
+ if err != nil {
+ _, _ = fmt.Fprintf(stderr, "runner browserbench: load CSV: %v\n", err)
+ return 1
+ }
+ tasks = selectBrowserBenchTasks(tasks, args.TaskID, args.Tasks)
+ if len(tasks) == 0 {
+ _, _ = fmt.Fprintln(stderr, "runner browserbench: no tasks selected")
+ return 1
+ }
+
+ resultsDir := filepath.Join(toolsDir, "browserbench", "results")
+ if err := os.MkdirAll(resultsDir, 0o755); err != nil {
+ _, _ = fmt.Fprintf(stderr, "runner browserbench: create results dir: %v\n", err)
+ return 1
+ }
+
+ outputPath := args.Output
+ if outputPath == "" {
+ outputPath = filepath.Join(resultsDir, fmt.Sprintf("pinchtab_browserbench_%s.csv", time.Now().Format("20060102_150405")))
+ }
+ runDir := strings.TrimSuffix(outputPath, filepath.Ext(outputPath))
+ if err := os.MkdirAll(runDir, 0o755); err != nil {
+ _, _ = fmt.Fprintf(stderr, "runner browserbench: create run dir: %v\n", err)
+ return 1
+ }
+
+ shell, err := NewPersistentShell(toolsDir)
+ if err != nil {
+ _, _ = fmt.Fprintf(stderr, "runner browserbench: shell init failed: %v\n", err)
+ return 1
+ }
+ defer shell.Close(false)
+
+ pinchtabVersion := detectPinchTabVersion(shell)
+ rows := make([]BrowserBenchRow, 0, len(tasks))
+
+ for i, task := range tasks {
+ _, _ = fmt.Fprintf(stdout, "[browserbench] task %d/%d id=%s url=%s\n", i+1, len(tasks), task.TaskID, task.StartingURL)
+ row, err := runBrowserBenchTask(args, provider, model, task, runDir, shell, pinchtabVersion, stdout, stderr)
+ if err != nil {
+ row.ErrorMessage = err.Error()
+ }
+ rows = append(rows, row)
+ if err := writeBrowserBenchCSV(outputPath, rows); err != nil {
+ _, _ = fmt.Fprintf(stderr, "runner browserbench: write CSV: %v\n", err)
+ return 1
+ }
+ }
+
+ passed := 0
+ for _, row := range rows {
+ if row.Success {
+ passed++
+ }
+ }
+ _, _ = fmt.Fprintf(stdout, "[browserbench] wrote %s (%d/%d passed)\n", outputPath, passed, len(rows))
+ return 0
+}
+
+func loadBrowserBenchTasks(path string) ([]BrowserBenchTask, error) {
+ data, err := readFileOrURL(path)
+ if err != nil {
+ return nil, err
+ }
+ r := csv.NewReader(strings.NewReader(string(data)))
+ records, err := r.ReadAll()
+ if err != nil {
+ return nil, err
+ }
+ if len(records) < 2 {
+ return nil, fmt.Errorf("CSV has no data rows")
+ }
+ headers := make(map[string]int)
+ for i, h := range records[0] {
+ headers[strings.TrimSpace(strings.ToLower(h))] = i
+ }
+ required := []string{"task_id", "starting_url", "task_description", "ground_truth_url", "ground_truth"}
+ for _, key := range required {
+ if _, ok := headers[key]; !ok {
+ return nil, fmt.Errorf("missing required column %q", key)
+ }
+ }
+ var tasks []BrowserBenchTask
+ for _, rec := range records[1:] {
+ if len(rec) == 0 {
+ continue
+ }
+ tasks = append(tasks, BrowserBenchTask{
+ TaskID: rec[headers["task_id"]],
+ StartingURL: rec[headers["starting_url"]],
+ TaskDescription: rec[headers["task_description"]],
+ GroundTruthURL: rec[headers["ground_truth_url"]],
+ GroundTruth: rec[headers["ground_truth"]],
+ })
+ }
+ return tasks, nil
+}
+
+func selectBrowserBenchTasks(tasks []BrowserBenchTask, taskID string, limit int) []BrowserBenchTask {
+ selected := make([]BrowserBenchTask, 0, len(tasks))
+ for _, task := range tasks {
+ if taskID != "" && task.TaskID != taskID {
+ continue
+ }
+ selected = append(selected, task)
+ if limit > 0 && len(selected) >= limit {
+ break
+ }
+ }
+ return selected
+}
+
+func runBrowserBenchTask(args BrowserBenchArgs, provider, model string, task BrowserBenchTask, runDir string, shell *PersistentShell, pinchtabVersion string, stdout, stderr io.Writer) (BrowserBenchRow, error) {
+ row := BrowserBenchRow{
+ TaskID: task.TaskID,
+ StartingURL: task.StartingURL,
+ TaskDescription: task.TaskDescription,
+ GroundTruthURL: task.GroundTruthURL,
+ GroundTruth: task.GroundTruth,
+ Provider: provider,
+ Timestamp: time.Now().UTC().Format(time.RFC3339),
+ PinchTabVersion: pinchtabVersion,
+ }
+
+ taskDir := filepath.Join(runDir, fmt.Sprintf("task-%s", sanitizeTaskID(task.TaskID)))
+ artifactsDir := filepath.Join(taskDir, "artifacts")
+ _ = os.MkdirAll(artifactsDir, 0o755)
+ commandLogPath := filepath.Join(taskDir, "commands.ndjson")
+ row.CommandLogPath = commandLogPath
+
+ if _, _, err := shell.Run(fmt.Sprintf("mkdir -p %q && ./scripts/pt console --clear >/dev/null 2>&1 || true", artifactsDir), 30*time.Second); err != nil {
+ return row, err
+ }
+ _, _, _ = shell.Run(`curl -sS -X POST -H 'Authorization: Bearer benchmark-token' http://localhost:9867/network/clear >/dev/null 2>&1 || true`, 15*time.Second)
+ if _, _, err := shell.Run(fmt.Sprintf("./scripts/pt nav %q >/dev/null", task.StartingURL), 45*time.Second); err != nil {
+ return row, fmt.Errorf("pre-nav failed: %w", err)
+ }
+
+ runner := createRunner(provider, model, Args{
+ Provider: Provider(provider),
+ Model: model,
+ MaxTokens: args.MaxTokens,
+ Temperature: args.Temperature,
+ MaxInputTokens: args.MaxInputTokens,
+ MaxOutputTokens: args.MaxOutputTokens,
+ })
+
+ before := runner.Usage()
+ start := time.Now()
+ finalText, loopErr := runBrowserBenchAgentLoop(task, provider, shell, runner, args, commandLogPath, stdout)
+ elapsed := time.Since(start)
+ after := runner.Usage()
+ usage := diffUsage(after, before)
+
+ row.AgentResult = extractFinalAnswer(finalText)
+ row.ExecutionTimeSeconds = fmt.Sprintf("%.3f", elapsed.Seconds())
+ row.RequestCount = strconv.Itoa(usage.RequestCount)
+ row.InputTokens = strconv.Itoa(usage.InputTokens)
+ row.OutputTokens = strconv.Itoa(usage.OutputTokens)
+ row.CacheCreationInputTokens = strconv.Itoa(usage.CacheCreationInputTokens)
+ row.CacheReadInputTokens = strconv.Itoa(usage.CacheReadInputTokens)
+ row.TotalInputTokens = strconv.Itoa(usage.InputTokens + usage.CacheCreationInputTokens + usage.CacheReadInputTokens)
+ row.TotalTokens = strconv.Itoa(usage.InputTokens + usage.CacheCreationInputTokens + usage.CacheReadInputTokens + usage.OutputTokens)
+ if loopErr != nil {
+ row.ErrorMessage = loopErr.Error()
+ }
+
+ stealthPath := filepath.Join(artifactsDir, "stealth-status.json")
+ autosolverPath := filepath.Join(artifactsDir, "autosolver.json")
+ screenshotPath := filepath.Join(artifactsDir, "final.png")
+ harPath := filepath.Join(artifactsDir, "network.har")
+ consolePath := filepath.Join(artifactsDir, "console.log")
+ row.ScreenshotPath = screenshotPath
+ row.HARPath = harPath
+ row.ConsoleLogPath = consolePath
+
+ _, _, _ = shell.Run(fmt.Sprintf("curl -sS -H 'Authorization: Bearer benchmark-token' http://localhost:9867/stealth/status > %q || true", stealthPath), 15*time.Second)
+ _, _, _ = shell.Run(fmt.Sprintf("curl -sS -H 'Authorization: Bearer benchmark-token' http://localhost:9867/config/autosolver > %q || true", autosolverPath), 15*time.Second)
+ _, _, _ = shell.Run(fmt.Sprintf("curl -sS -H 'Authorization: Bearer benchmark-token' 'http://localhost:9867/screenshot?format=png&raw=true' > %q || true", screenshotPath), 30*time.Second)
+ _, _, _ = shell.Run(fmt.Sprintf("curl -sS -H 'Authorization: Bearer benchmark-token' 'http://localhost:9867/network/export?format=har' > %q || true", harPath), 30*time.Second)
+ _, _, _ = shell.Run(fmt.Sprintf("./scripts/pt console > %q 2>/dev/null || true", consolePath), 15*time.Second)
+
+ row.StealthLevel = readJSONField(stealthPath, "level")
+ row.SolverUsed, row.SolveAttempts = deriveSolveMetadata(commandLogPath)
+ row.Success = evaluateBrowserBenchAnswer(row.AgentResult, task.GroundTruth) && row.ErrorMessage == ""
+
+ return row, nil
+}
+
+func runBrowserBenchAgentLoop(task BrowserBenchTask, provider string, shell *PersistentShell, runner Runner, args BrowserBenchArgs, commandLogPath string, stdout io.Writer) (string, error) {
+ conversation := runner.InitialConversation(browserBenchUserPrompt(task))
+ systemPrompt := browserBenchSystemPrompt(filepath.Dir(filepath.Dir(resolveToolsDir())))
+ out := NewOutputWriter(stdout, args.Verbose)
+
+ for turn := 1; turn <= args.MaxTurns; turn++ {
+ if turn > 1 && args.TurnDelayMs > 0 {
+ time.Sleep(time.Duration(args.TurnDelayMs) * time.Millisecond)
+ }
+ usage := runner.Usage()
+ if args.MaxInputTokens > 0 && usage.InputTokens+usage.CacheCreationInputTokens+usage.CacheReadInputTokens >= args.MaxInputTokens {
+ return "", fmt.Errorf("input token budget exceeded")
+ }
+ if args.MaxOutputTokens > 0 && usage.OutputTokens >= args.MaxOutputTokens {
+ return "", fmt.Errorf("output token budget exceeded")
+ }
+
+ out.Turn(turn, args.MaxTurns)
+ spinner := out.StartSpinner("Solving BrowserBench task...")
+ response, err := runner.Send(systemPrompt, conversation)
+ spinner.Stop()
+ if err != nil {
+ return "", err
+ }
+
+ newUsage := runner.Usage()
+ out.APICall(runner.Provider(), runner.Model(),
+ newUsage.InputTokens-usage.InputTokens,
+ newUsage.OutputTokens-usage.OutputTokens,
+ newUsage.CacheCreationInputTokens-usage.CacheCreationInputTokens,
+ newUsage.CacheReadInputTokens-usage.CacheReadInputTokens)
+
+ toolCalls := runner.ExtractToolCalls(response, time.Duration(args.TimeoutSeconds)*time.Second)
+ if len(toolCalls) == 0 {
+ return runner.ExtractFinalText(response), nil
+ }
+ results := executeToolCallsVerbose(shell, toolCalls, commandLogPath, out)
+ conversation = runner.AppendToolResults(conversation, response, results)
+ conversation = CompactConversation(provider, conversation, "BrowserBench task in progress. Continue from the current browser state until you can answer with FINAL_ANSWER.")
+ }
+
+ return "", fmt.Errorf("max turns reached (%d)", args.MaxTurns)
+}
+
+func browserBenchSystemPrompt(repoRoot string) string {
+ base := `You are a precise web-task execution agent running BrowserBench-style tasks through PinchTab.
+
+Rules:
+- Use shell tool calls only.
+- Use only ./scripts/pt for browser actions.
+- The browser is already pointed at the task's starting URL when the task begins.
+- Do NOT create or manage PinchTab sessions yourself. Do not call ./scripts/pt session create, session login, or other session bootstrap commands.
+- Prefer PinchTab primitives such as snap, text, click, fill, press, wait, back, eval, console, and solve.
+- Do NOT invent convenience flags. "--snap" and "--snap-diff" are valid on some actions (for example nav, click, fill, select, scroll, back, forward, reload) but not on every command. In particular, do not use them with press. When unsure, run a separate "./scripts/pt snap -i -c" after the action.
+- If a site presents a challenge, you may use ./scripts/pt solve.
+- Stay on the starting website domain unless the task explicitly requires a page on the same site reached by navigation.
+- Do not fabricate answers.
+- When you are confident, end with a single line in this exact format: FINAL_ANSWER: <answer>
+- Keep the final answer short and factual.`
+
+ skill := LoadSkillContent(filepath.Join(repoRoot, "skills", "pinchtab", "SKILL.md"))
+ if skill == "" {
+ return base
+ }
+ return base + "\n\n# PinchTab Skill\n\n" + skill
+}
+
+func browserBenchUserPrompt(task BrowserBenchTask) string {
+ return fmt.Sprintf(`Complete this BrowserBench task.
+
+Starting URL: %s
+Task ID: %s
+Instruction: %s
+
+Important:
+- the browser is already opened on the starting URL
+- do not use the ground truth in your reasoning
+- return exactly one final line: FINAL_ANSWER: <answer>`, task.StartingURL, task.TaskID, task.TaskDescription)
+}
+
+func readFileOrURL(path string) ([]byte, error) {
+ if strings.HasPrefix(path, "http://") || strings.HasPrefix(path, "https://") {
+ client := &http.Client{Timeout: 30 * time.Second}
+ resp, err := client.Get(path)
+ if err != nil {
+ return nil, err
+ }
+ defer func() { _ = resp.Body.Close() }()
+ if resp.StatusCode < 200 || resp.StatusCode > 299 {
+ return nil, fmt.Errorf("GET %s: status %d", path, resp.StatusCode)
+ }
+ return io.ReadAll(resp.Body)
+ }
+ return os.ReadFile(path)
+}
+
+func writeBrowserBenchCSV(path string, rows []BrowserBenchRow) error {
+ if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
+ return err
+ }
+ f, err := os.Create(path)
+ if err != nil {
+ return err
+ }
+ defer func() { _ = f.Close() }()
+ w := csv.NewWriter(f)
+ headers := []string{
+ "task_id", "starting_url", "task_description", "ground_truth_url", "ground_truth",
+ "provider", "timestamp", "success", "error_message", "agent_result", "session_url", "execution_time",
+ "pinchtab_version", "stealth_level", "solver_used", "attempts",
+ "har_path", "screenshot_path", "console_log_path", "command_log_path",
+ "request_count", "input_tokens", "output_tokens", "cache_creation_input_tokens", "cache_read_input_tokens", "total_input_tokens", "total_tokens",
+ }
+ if err := w.Write(headers); err != nil {
+ return err
+ }
+ for _, row := range rows {
+ rec := []string{
+ row.TaskID, row.StartingURL, row.TaskDescription, row.GroundTruthURL, row.GroundTruth,
+ row.Provider, row.Timestamp, strconv.FormatBool(row.Success), row.ErrorMessage, row.AgentResult, row.SessionURL, row.ExecutionTimeSeconds,
+ row.PinchTabVersion, row.StealthLevel, row.SolverUsed, row.SolveAttempts,
+ row.HARPath, row.ScreenshotPath, row.ConsoleLogPath, row.CommandLogPath,
+ row.RequestCount, row.InputTokens, row.OutputTokens, row.CacheCreationInputTokens, row.CacheReadInputTokens, row.TotalInputTokens, row.TotalTokens,
+ }
+ if err := w.Write(rec); err != nil {
+ return err
+ }
+ }
+ w.Flush()
+ return w.Error()
+}
+
+func detectPinchTabVersion(shell *PersistentShell) string {
+ out, _, err := shell.Run("docker exec tools-pinchtab-1 sh -lc 'pinchtab --version 2>/dev/null || true'", 15*time.Second)
+ if err != nil {
+ return ""
+ }
+ lines := strings.Split(strings.TrimSpace(out), "\n")
+ if len(lines) == 0 {
+ return strings.TrimSpace(out)
+ }
+ return strings.TrimSpace(lines[len(lines)-1])
+}
+
+func sanitizeTaskID(v string) string {
+ v = strings.TrimSpace(v)
+ if v == "" {
+ return "unknown"
+ }
+ re := regexp.MustCompile(`[^a-zA-Z0-9._-]+`)
+ return re.ReplaceAllString(v, "-")
+}
+
+func extractFinalAnswer(text string) string {
+ for _, line := range strings.Split(text, "\n") {
+ line = strings.TrimSpace(line)
+ if strings.HasPrefix(strings.ToUpper(line), "FINAL_ANSWER:") {
+ return strings.TrimSpace(line[len("FINAL_ANSWER:"):])
+ }
+ }
+ return strings.TrimSpace(text)
+}
+
+func evaluateBrowserBenchAnswer(answer, truth string) bool {
+ a := normalizeAnswer(answer)
+ g := normalizeAnswer(truth)
+ if a == "" || g == "" {
+ return false
+ }
+ if g == "yes" || g == "no" {
+ return a == g || strings.HasPrefix(a, g+" ") || strings.Contains(a, " "+g+" ")
+ }
+ if a == g || strings.Contains(a, g) || strings.Contains(g, a) {
+ return true
+ }
+ return false
+}
+
+func normalizeAnswer(s string) string {
+ s = strings.ToLower(strings.TrimSpace(s))
+ s = strings.ReplaceAll(s, "’", "'")
+ s = strings.ReplaceAll(s, "β€œ", `"`)
+ s = strings.ReplaceAll(s, "”", `"`)
+ s = regexp.MustCompile(`\s+`).ReplaceAllString(s, " ")
+ s = strings.Trim(s, " .,!?:;\"'")
+ return s
+}
+
+func diffUsage(after, before UsageCounters) UsageCounters {
+ return UsageCounters{
+ RequestCount: after.RequestCount - before.RequestCount,
+ InputTokens: after.InputTokens - before.InputTokens,
+ OutputTokens: after.OutputTokens - before.OutputTokens,
+ CacheCreationInputTokens: after.CacheCreationInputTokens - before.CacheCreationInputTokens,
+ CacheReadInputTokens: after.CacheReadInputTokens - before.CacheReadInputTokens,
+ }
+}
+
+func readJSONField(path, key string) string {
+ data, err := os.ReadFile(path)
+ if err != nil {
+ return ""
+ }
+ var m map[string]any
+ if err := json.Unmarshal(data, &m); err != nil {
+ return ""
+ }
+ if v, ok := m[key]; ok {
+ return strings.TrimSpace(fmt.Sprint(v))
+ }
+ return ""
+}
+
+func deriveSolveMetadata(commandLogPath string) (string, string) {
+ data, err := os.ReadFile(commandLogPath)
+ if err != nil {
+ return "", ""
+ }
+ lines := strings.Split(string(data), "\n")
+ solver := ""
+ maxAttempt := 0
+ for _, line := range lines {
+ line = strings.TrimSpace(line)
+ if line == "" {
+ continue
+ }
+ var entry struct {
+ Output string `json:"output"`
+ }
+ if err := json.Unmarshal([]byte(line), &entry); err != nil {
+ continue
+ }
+ var payload struct {
+ Solver string `json:"solver"`
+ Attempts int `json:"attempts"`
+ }
+ if err := json.Unmarshal([]byte(entry.Output), &payload); err != nil {
+ continue
+ }
+ if payload.Solver != "" {
+ solver = payload.Solver
+ }
+ if payload.Attempts > maxAttempt {
+ maxAttempt = payload.Attempts
+ }
+ }
+ if maxAttempt == 0 {
+ return solver, ""
+ }
+ return solver, strconv.Itoa(maxAttempt)
+}
diff --git a/tests/tools/runner/internal/bench/browserbench_test.go b/tests/tools/runner/internal/bench/browserbench_test.go
new file mode 100644
index 00000000..1a2449ef
--- /dev/null
+++ b/tests/tools/runner/internal/bench/browserbench_test.go
@@ -0,0 +1,75 @@
+package bench
+
+import (
+ "os"
+ "path/filepath"
+ "testing"
+)
+
+func TestParseBrowserBenchArgs(t *testing.T) {
+ args, err := parseBrowserBenchArgs([]string{"--provider", "openai", "--tasks", "5", "--task-id", "12", "--verbose"})
+ if err != nil {
+ t.Fatalf("parseBrowserBenchArgs returned error: %v", err)
+ }
+ if args.Provider != ProviderOpenAI {
+ t.Fatalf("Provider = %q, want %q", args.Provider, ProviderOpenAI)
+ }
+ if args.Tasks != 5 {
+ t.Fatalf("Tasks = %d, want 5", args.Tasks)
+ }
+ if args.TaskID != "12" {
+ t.Fatalf("TaskID = %q, want 12", args.TaskID)
+ }
+ if !args.Verbose {
+ t.Fatal("Verbose = false, want true")
+ }
+}
+
+func TestSelectBrowserBenchTasks(t *testing.T) {
+ tasks := []BrowserBenchTask{{TaskID: "1"}, {TaskID: "2"}, {TaskID: "3"}}
+ selected := selectBrowserBenchTasks(tasks, "2", 0)
+ if len(selected) != 1 || selected[0].TaskID != "2" {
+ t.Fatalf("selected = %#v, want only task 2", selected)
+ }
+
+ selected = selectBrowserBenchTasks(tasks, "", 2)
+ if len(selected) != 2 || selected[1].TaskID != "2" {
+ t.Fatalf("limit selection = %#v, want first two tasks", selected)
+ }
+}
+
+func TestEvaluateBrowserBenchAnswer(t *testing.T) {
+ cases := []struct {
+ answer string
+ truth string
+ want bool
+ }{
+ {"FINAL_ANSWER: 3.4", "3.4", true},
+ {"yes", "Yes", true},
+ {"The answer is Without Me", "Without Me", true},
+ {"No", "Yes", false},
+ }
+ for _, tc := range cases {
+ if got := evaluateBrowserBenchAnswer(extractFinalAnswer(tc.answer), tc.truth); got != tc.want {
+ t.Fatalf("evaluateBrowserBenchAnswer(%q, %q) = %v, want %v", tc.answer, tc.truth, got, tc.want)
+ }
+ }
+}
+
+func TestDeriveSolveMetadata(t *testing.T) {
+ dir := t.TempDir()
+ path := filepath.Join(dir, "commands.ndjson")
+ content := `{"output":"{\"solver\":\"cloudflare\",\"attempts\":2}"}
+{"output":"{\"solver\":\"semantic\",\"attempts\":1}"}
+`
+ if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ solver, attempts := deriveSolveMetadata(path)
+ if solver != "semantic" {
+ t.Fatalf("solver = %q, want semantic", solver)
+ }
+ if attempts != "2" {
+ t.Fatalf("attempts = %q, want 2", attempts)
+ }
+}
diff --git a/tests/tools/runner/main.go b/tests/tools/runner/main.go
index 61704016..78145c0a 100644
--- a/tests/tools/runner/main.go
+++ b/tests/tools/runner/main.go
@@ -24,6 +24,8 @@ func run(argv []string, stdout, stderr io.Writer) int {
return e2e.Run(argv[1:], stdout, stderr)
case "bench":
return bench.Run(argv[1:], stdout, stderr)
+ case "browserbench":
+ return bench.RunBrowserBench(argv[1:], stdout, stderr)
case "verify-step", "record-step", "step-end":
return bench.Run(argv, stdout, stderr)
case "opt":
@@ -41,6 +43,7 @@ func run(argv []string, stdout, stderr io.Writer) int {
_, _ = fmt.Fprintln(stderr, "Use one of:")
_, _ = fmt.Fprintln(stderr, " runner e2e --suite basic")
_, _ = fmt.Fprintln(stderr, " runner bench --lane pinchtab")
+ _, _ = fmt.Fprintln(stderr, " runner browserbench --tasks 5")
_, _ = fmt.Fprintln(stderr, " runner --lane pinchtab")
_, _ = fmt.Fprintln(stderr, " runner step-end ...")
_, _ = fmt.Fprintln(stderr, " runner opt merge-reports ...")
diff --git a/tests/tools/scripts/probe-all.sh b/tests/tools/scripts/probe-all.sh
new file mode 100755
index 00000000..23ee9480
--- /dev/null
+++ b/tests/tools/scripts/probe-all.sh
@@ -0,0 +1,86 @@
+#!/usr/bin/env bash
+# Probe every starting_url in the BrowserBench dataset, recording the final URL/title/snap
+# and any captcha pauses. Designed to be run from tests/tools/.
+#
+# Output CSV: /tmp/bb_probe_full.csv
+# Live log: /tmp/bb_probe_full.log
+
+set -u
+
+SRC=/tmp/bb.csv
+OUT=/tmp/bb_probe_full.csv
+LOG=/tmp/bb_probe_full.log
+SERVER=http://localhost:9867
+TOKEN=benchmark-token
+PT="$(dirname "$0")/pt"
+
+# fresh CSV with header
+echo "task_id,starting_url,final_url,title,status,note" > "$OUT"
+: > "$LOG"
+
+# materialize task list to a temp file so child processes can't consume the loop's stdin
+TASKS=$(mktemp)
+trap 'rm -f "$TASKS"' EXIT
+python3 - "$SRC" "$TASKS" <<'PY'
+import csv, sys
+with open(sys.argv[1], newline="") as fh, open(sys.argv[2], "w") as out:
+ r = csv.DictReader(fh)
+ for row in r:
+ out.write(f"{row['task_id']}\t{row['starting_url']}\n")
+PY
+
+while IFS=$'\t' read -r task_id url <&3; do
+ ts=$(date +%H:%M:%S)
+ printf '[%s] %s -> %s\n' "$ts" "$task_id" "$url" | tee -a "$LOG"
+
+ # navigate (5s timeout via curl-style; rely on pt's own timeout)
+ "$PT" nav "$url" >/dev/null 2>&1 || true
+ sleep 4
+
+ # check if tab paused for human handoff
+ state=$(curl -sS -H "Authorization: Bearer $TOKEN" "$SERVER/tabs" \
+ | python3 -c 'import json,sys; d=json.load(sys.stdin); print((d.get("tabs") or [{}])[0].get("status",""))')
+
+ note=""
+ if [ "$state" = "paused_handoff" ]; then
+ reason=$(curl -sS -H "Authorization: Bearer $TOKEN" "$SERVER/tabs" \
+ | python3 -c 'import json,sys; d=json.load(sys.stdin); print((d.get("tabs") or [{}])[0].get("handoffReason",""))')
+ printf ' >> PAUSED reason=%s β€” solve in dashboard, then click resume\n' "$reason" | tee -a "$LOG"
+ note="paused:$reason"
+ # poll up to 5 minutes for human resume
+ for i in $(seq 1 60); do
+ sleep 5
+ state=$(curl -sS -H "Authorization: Bearer $TOKEN" "$SERVER/tabs" \
+ | python3 -c 'import json,sys; d=json.load(sys.stdin); print((d.get("tabs") or [{}])[0].get("status",""))')
+ [ "$state" != "paused_handoff" ] && break
+ done
+ if [ "$state" = "paused_handoff" ]; then
+ printf ' >> still paused after 5 min, force-resume + skip\n' | tee -a "$LOG"
+ curl -sS -X POST -H "Authorization: Bearer $TOKEN" "$SERVER/resume" >/dev/null 2>&1 || true
+ note="$note;timeout"
+ fi
+ fi
+
+ final_url=$("$PT" url 2>/dev/null | tr -d '\r' | head -1)
+ title=$("$PT" title 2>/dev/null | tr -d '\r' | head -1)
+
+ # classify the page
+ status="ok"
+ lc_title=$(printf '%s' "$title" | tr '[:upper:]' '[:lower:]')
+ case "$lc_title" in
+ *"just a moment"*|*"one moment"*) status="cloudflare_interstitial" ;;
+ *"access forbidden"*|*"access denied"*|*"blocked"*|*"403"*) status="blocked_403" ;;
+ *"page not found"*|*"404"*|*"not found"*) status="not_found" ;;
+ *"are you a robot"*|*"verify you are human"*|*"please verify"*|*"checking your browser"*) status="bot_check" ;;
+ *) status="ok" ;;
+ esac
+
+ # CSV-escape via python to be safe with commas in titles
+ python3 -c '
+import csv, sys
+w = csv.writer(sys.stdout)
+w.writerow(sys.argv[1:])
+' "$task_id" "$url" "$final_url" "$title" "$status" "$note" >> "$OUT"
+done 3< "$TASKS"
+
+echo "DONE" | tee -a "$LOG"