Skip to content

High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.

refactor: replace api-runner with go implementation

Luigi Agostid7216a104ddf
diff --git a/docs/deep-dive/agent-browser-benchmark.md b/docs/deep-dive/agent-browser-benchmark.md
index 9103f9f2..f4df250a 100644
--- a/docs/deep-dive/agent-browser-benchmark.md
+++ b/docs/deep-dive/agent-browser-benchmark.md
@@ -617,16 +617,16 @@ docker compose up -d --build
./scripts/run-optimization.sh
# 3. Run baseline (deterministic, ~30 seconds)
-./scripts/baseline_all.sh
+./scripts/baseline.sh
# 4. Run PinchTab agent lane (requires LLM, ~15 minutes)
-# Execute tasks from AGENT_TASKS.md using ./scripts/pt
+# Execute tasks from benchmark-run/index.md + benchmark-run/group-XX.md using setup-pinchtab.md and ./scripts/pt
# 5. Initialize agent-browser lane
./scripts/run-agent-browser-benchmark.sh
# 6. Run agent-browser lane (requires LLM, ~20 minutes)
-# Execute tasks from AGENT_BROWSER_TASKS.md using ./scripts/ab
+# Execute tasks from benchmark-run/index.md + benchmark-run/group-XX.md using setup-agent-browser.md and ./scripts/ab
# 7. Generate reports
./scripts/finalize-report.sh
diff --git a/scripts/dev-benchmark.sh b/scripts/dev-benchmark.sh
index 7c58fd04..117e3ad4 100755
--- a/scripts/dev-benchmark.sh
+++ b/scripts/dev-benchmark.sh
@@ -18,20 +18,19 @@ usage() {
cat <<'EOF'
Usage:
./dev benchmark baseline
- OPENAI_API_KEY=... ./dev benchmark agent [extra runner args...]
- OPENAI_API_KEY=... ./dev benchmark agent-browser [extra runner args...]
- ANTHROPIC_API_KEY=... ./dev benchmark agent [extra runner args...]
- ANTHROPIC_API_KEY=... ./dev benchmark agent-browser [extra runner args...]
+ OPENAI_API_KEY=... ./dev benchmark pinchtab [options...]
+ OPENAI_API_KEY=... ./dev benchmark agent-browser [options...]
+ ANTHROPIC_API_KEY=... ./dev benchmark pinchtab [options...]
+ ANTHROPIC_API_KEY=... ./dev benchmark agent-browser [options...]
Examples:
./dev benchmark baseline
- OPENAI_API_KEY=... ./dev benchmark agent --model gpt-5
- OPENAI_API_KEY=... ./dev benchmark agent-browser --model gpt-5
- ANTHROPIC_API_KEY=... ./dev benchmark agent --model claude-haiku-4-5-20251001
- ANTHROPIC_API_KEY=... ./dev benchmark agent-browser --model claude-haiku-4-5-20251001
- ANTHROPIC_API_KEY=... ./dev benchmark agent --provider anthropic --turn-delay-ms 3000
- ANTHROPIC_API_KEY=... ./dev benchmark agent --provider anthropic --profile common10
- ANTHROPIC_API_KEY=... ./dev benchmark agent-browser --provider anthropic --groups 0,1,2,3
+ ./dev benchmark pinchtab --dry-run
+ OPENAI_API_KEY=... ./dev benchmark pinchtab --model gpt-5
+ ANTHROPIC_API_KEY=... ./dev benchmark pinchtab --model claude-haiku-4-5-20251001
+ ANTHROPIC_API_KEY=... ./dev benchmark pinchtab --profile common10
+ ANTHROPIC_API_KEY=... ./dev benchmark pinchtab --max-input-tokens 50000
+ ANTHROPIC_API_KEY=... ./dev benchmark agent-browser --groups 0,1,2,3
EOF
}
@@ -52,12 +51,13 @@ case "${mode}" in
BASELINE_REPORT="$(resolve_current_report "${RESULTS_DIR}/current_baseline_report.txt")"
./scripts/finalize-report.sh "${BASELINE_REPORT}"
;;
- agent|agent-browser)
+ pinchtab|agent-browser)
if [[ -z "${OPENAI_API_KEY:-}" && -z "${ANTHROPIC_API_KEY:-}" ]]; then
echo "ERROR: OPENAI_API_KEY or ANTHROPIC_API_KEY is required for benchmark ${mode}" >&2
exit 1
fi
- exec node --experimental-strip-types ./scripts/run-api-benchmark.ts --lane "${mode}" --finalize "$@"
+ cd "${ROOT_DIR}"
+ exec go run ./tests/benchmark/cmd/api-runner --lane "${mode}" --finalize "$@"
;;
-h|--help|help)
usage
diff --git a/tests/benchmark/AGENT_BROWSER_TASKS.md b/tests/benchmark/AGENT_BROWSER_TASKS.md
deleted file mode 100644
index cdd891c3..00000000
--- a/tests/benchmark/AGENT_BROWSER_TASKS.md
+++ /dev/null
@@ -1,76 +0,0 @@
-# Agent Browser Benchmark
-
-Natural-language benchmark lane for running the same fixture tasks with
-[`agent-browser`](https://github.com/vercel-labs/agent-browser) instead of the
-PinchTab CLI.
-
-## Setup
-
-1. Read `./AGENT_BROWSER_INSTRUCTIONS.md` first. This is your benchmark guide to
- the wrapper and command workflow.
-
-2. Load the live CLI skill content before using browser commands:
-
-```bash
-./scripts/ab skills get agent-browser --full
-```
-
-3. Start the Docker benchmark environment and initialize a report:
-
-```bash
-cd tests/benchmark
-./scripts/run-agent-browser-benchmark.sh
-```
-
-4. Use the Docker-backed wrapper for every browser action:
-
-```bash
-./scripts/ab open http://fixtures/
-./scripts/ab snapshot -i -c
-./scripts/ab click @e2
-./scripts/ab fill @e3 "[email protected]"
-```
-
-5. Record each completed step as an answer/result first:
-
-```bash
-./scripts/record-step.sh --type agent-browser 1 1 answer \
- "opened fixtures home and got refs e1-e13" "completed"
-```
-
-`record-step.sh` will automatically calculate the number of `agent-browser`
-tool calls used since the previous recorded step by reading
-`results/agent_browser_commands.ndjson`.
-
-6. After the execution lane is done, run a separate verification pass with:
-
-```bash
-./scripts/verify-step.sh --type agent-browser 1 1 pass "matched expected homepage state"
-```
-
-## Environment
-
-- Fixtures: `http://fixtures/`
-- Session name: `benchmark` by default (`AGENT_BROWSER_SESSION` overrides)
-- Browser driver: Docker service `agent-browser`
-
-## Tooling Guidance
-
-Use `./AGENT_BROWSER_INSTRUCTIONS.md` as the primary benchmark operating guide.
-Reach for `./scripts/ab --help` only when the skill does not already answer the
-question.
-
-## Task Set
-
-Reuse the same benchmark task groups from [AGENT_TASKS.md](./AGENT_TASKS.md)
-for content extraction, search, forms, SPA state, login, e-commerce, exports,
-dialogs, async flows, drag/drop, keyboard, scrolling, and iframe interaction.
-
-The only setup difference is that Group 0 should validate the `agent-browser`
-lane instead of the PinchTab server:
-
-- 0.1 `./scripts/ab open http://fixtures/` succeeds
-- 0.2 `./scripts/ab snapshot -i -c` returns interactive refs
-- 0.3 session state persists across multiple `./scripts/ab ...` commands
-
-After that, continue with the same user-facing tasks from Group 1 onward.
diff --git a/tests/benchmark/AGENT_TASKS.md b/tests/benchmark/AGENT_TASKS.md
deleted file mode 100644
index 89ae8c70..00000000
--- a/tests/benchmark/AGENT_TASKS.md
+++ /dev/null
@@ -1,718 +0,0 @@
-# PinchTab Agent Benchmark
-
-Natural language tasks to test how well an agent uses PinchTab from skill docs alone.
-
-## Instructions
-
-1. Read `../../skills/pinchtab/SKILL.md` — this is your only guide
-2. For each task, figure out which commands to use
-3. **Log every command executed**
-4. Do not score your own work as pass/fail unless a task is trivially objective.
-5. Prefer recording each step as `answer` with the actual response/result:
- `./scripts/record-step.sh --type agent <group> <step> answer "<what you saw>" "notes"`
-6. A separate verification pass should later stamp the step with:
- `./scripts/verify-step.sh --type agent <group> <step> <pass|fail|skip> "verification notes"`
-
-### Recommended setup: env vars + `./scripts/pt` wrapper
-
-In normal PinchTab deployments the binary is installed on the host and
-invoked directly:
-
-```bash
-export PINCHTAB_TOKEN=<token>
-export PINCHTAB_TAB=$(pinchtab nav http://example.com)
-pinchtab snap -i -c
-pinchtab click '#submit'
-```
-
-The benchmark harness runs PinchTab **inside a Docker container**, so every
-invocation normally has to be prefixed with
-`docker exec -e PINCHTAB_TOKEN=... -e PINCHTAB_SERVER=... -e PINCHTAB_TAB=... benchmark-pinchtab-1 pinchtab ...`
-— ~140 characters of boilerplate per command. The repo ships a tiny wrapper
-at `tests/benchmark/scripts/pt` that handles the Docker preamble so you can
-use the same terse CLI as a host install:
-
-```bash
-# Capture the shared tab ID. pinchtab prints only the tab ID when stdout
-# is a pipe, so $(...) capture works directly.
-export PINCHTAB_TAB=$(./scripts/pt nav http://fixtures/)
-
-# Every subsequent tab-scoped command auto-targets $PINCHTAB_TAB:
-./scripts/pt snap -i -c
-./scripts/pt eval "document.title"
-./scripts/pt eval --await-promise "window.fetchPayload()"
-./scripts/pt click '#submit'
-./scripts/pt drag '#piece' --drag-x 12 --drag-y -158
-
-# Record answer/result for later verification:
-./scripts/record-step.sh --type agent 1 2 answer "clicked submit and saw success banner" "worked"
-```
-
-The wrapper is a ~60-line shell script; read `scripts/pt` for the exact
-forwarding rules. Flags after `pt` are forwarded verbatim to `pinchtab`, so
-anything that works with `pinchtab ...` works with `./scripts/pt ...`.
-
-Explicit `--tab <id>` on any command still wins over `PINCHTAB_TAB`.
-
-For deferred verification, keep the answer payload factual. Record the
-actual page text, count, answer, or state transition you observed, not your own
-judgment about whether that matches the benchmark oracle.
-
-## Environment
-
-- PinchTab: `http://localhost:9867`, token: `benchmark-token`
-- Fixtures: `http://fixtures/` (running in Docker as `fixtures` hostname)
-- Pages: `/`, `/wiki.html`, `/wiki-go.html`, `/articles.html`, `/search.html`,
- `/form.html`, `/dashboard.html`, `/ecommerce.html`, `/spa.html`, `/login.html`
-
----
-
-## Group 0: Setup & Diagnosis
-
-### 0.1 Server reachable
-Check that the PinchTab server is healthy.
-
-**Verify**: Server responds with `status: ok`.
-
-### 0.2 Auth is required
-Make a request to the server with a **wrong** token (`PINCHTAB_TOKEN=wrong-token ./scripts/pt health`) and confirm it is rejected. The `pt` wrapper always injects the benchmark token by default, so you must explicitly override it to test auth rejection.
-
-**Verify**: Response is HTTP 401 or contains `unauthorized`.
-
-### 0.3 Auth works with token
-Repeat the same request WITH the bearer token and confirm it succeeds.
-
-**Verify**: Response is HTTP 200.
-
-### 0.4 Instance available
-Confirm at least one Chrome instance is running. If none exist, start one.
-
-**Verify**: Health response shows `defaultInstance.status == "running"` (or after starting one, the new instance is running).
-
-### 0.5 List existing tabs
-Get the current list of open tabs.
-
-**Verify**: A list (possibly empty) is returned without error.
-
-### 0.6 Clean stale tabs
-If any tabs from previous runs are open, close them so the benchmark starts from a known state.
-
-**Verify**: After cleanup, the tab list is empty (or only contains a single about:blank tab).
-
-### 0.7 Network reach to target
-Navigate to `http://fixtures/` and confirm the fixtures server is reachable from PinchTab.
-
-**Verify**: Navigate returns successfully and the page contains benchmark content.
-
-### 0.8 Capture initial tab ID
-Save the tab ID returned by the navigate in 0.7. Use this tab ID for all subsequent tasks to avoid creating new tabs.
-
-**Verify**: A tab ID was captured and matches what `GET /tabs` reports.
-
----
-
-## Group 1: Reading & Extracting Real Content
-
-### 1.1 Get the full list of categories from the wiki index
-Navigate to `http://fixtures/wiki.html` and extract all category names with their article counts.
-
-**Verify**: Can name at least 2 categories and their article counts (e.g. "Programming Languages: 12 articles").
-
-### 1.2 Navigate by clicking a link
-From the wiki index, click the "Go (programming language)" link to navigate to the Go article.
-
-**Verify**: You are now on the Go article page (not the wiki index).
-
-### 1.3 Extract structured data from a table
-On the Go article, read the infobox table and answer: Who designed Go, and what year did it first appear?
-
-**Verify**: Answer contains "Robert Griesemer" (or "Rob Pike" or "Ken Thompson") and "2009".
-
-### 1.4 Count list items
-On the Go article, count how many key features are listed.
-
-**Verify**: Answer is 6 (verify against `FEATURE_COUNT_6`).
-
-### 1.5 Read all article headlines from articles page
-Navigate to `http://fixtures/articles.html` and list all article titles.
-
-**Verify**: Found at least 3 articles including "The Future of Artificial Intelligence".
-
-### 1.6 Read dashboard metrics
-Navigate to `http://fixtures/dashboard.html` and extract: Total Users, Revenue, and Conversion Rate.
-
-**Verify**: Found `24,582` users AND `$1,284,930` revenue.
-
----
-
-## Group 2: Search & Dynamic Interaction
-
-### 2.1 Use wiki search to find a page
-On `http://fixtures/wiki.html`, search for "golang" using the search form. Do not navigate directly — use the search input.
-
-**Verify**: Ended up on the Go article page after search.
-
-### 2.2 Search with no results
-On `http://fixtures/search.html`, search for something with no results (use "xyznonexistent").
-
-**Verify**: Page handled it gracefully (no crash, some response rendered).
-
-### 2.3 Search for AI content
-On `http://fixtures/search.html`, search for "artificial intelligence" and verify a result appeared.
-
-**Verify**: Result contains "The Future of Artificial Intelligence".
-
----
-
-## Group 3: Complex Form
-
-### 3.1 Fill and submit a complete form
-Navigate to `http://fixtures/form.html`. Complete the entire form:
-- Full name: "Agent Test User"
-- Email: "[email protected]"
-- Phone: "+44 20 9999 0000"
-- Country: United Kingdom
-- Subject: Technical Support
-- Message: "Testing PinchTab form automation"
-- Check newsletter
-- Set priority to High
-- Submit
-
-**Verify**: Form submitted successfully. Confirmation shows name "AGENT_TEST_USER".
-
-### 3.2 Reset and refill
-After submitting, if the form is still accessible (or navigate back), verify you can identify the reset button.
-
-**Verify**: Reset/back button or form element found in snapshot.
-
----
-
-## Group 4: SPA State Management
-
-### 4.1 Read initial app state
-Navigate to `http://fixtures/spa.html?reset=1` (the `?reset=1` query param clears any previous localStorage state so the SPA starts with its default 3 tasks). Read the current task list — how many tasks exist, how many are active vs done?
-
-**Verify**: Found 3 total, 2 active, 1 done (verify `TASK_STATS_TOTAL_3_ACTIVE_2_DONE_1`).
-
-### 4.2 Add a new high-priority task
-Add a task called "Automate deployment" with high priority.
-
-**Verify**: Task appeared in the list (`TASK_ADDED_AUTOMATE_DEPLOYMENT_PRIORITY_HIGH`).
-
-### 4.3 Delete a task
-Delete the task titled "Write benchmark tests".
-
-**Verify**: Task count changed (went from 4 to 3).
-
----
-
-## Group 5: Login Flow
-
-### 5.1 Attempt login with wrong credentials
-Navigate to `http://fixtures/login.html`. Try to log in with username "admin" and password "wrong".
-
-**Verify**: Error message appeared (`INVALID_CREDENTIALS_ERROR`).
-
-### 5.2 Login successfully
-Clear the form and log in with username "benchmark" / password "test456".
-
-**Verify**: Dashboard appeared after login (`VERIFY_LOGIN_SUCCESS_DASHBOARD`).
-
----
-
-## Group 6: Multi-Step E-commerce
-
-### 6.1 Research products before buying
-Navigate to `http://fixtures/ecommerce.html`. List all available products with their prices. Which product is out of stock?
-
-**Verify**: Found Wireless Headphones ($149.99), Smart Watch Pro ($299.99), Portable Charger ($49.99). Mechanical Keyboard is out of stock.
-
-### 6.2 Add two items and verify total
-Add Wireless Headphones and Smart Watch Pro to cart. What is the total?
-
-**Verify**: Cart total is $449.98 (149.99 + 299.99).
-
-### 6.3 Complete checkout
-Click checkout to complete the order.
-
-**Verify**: Order confirmation shows (`VERIFY_CHECKOUT_SUCCESS_ORDER`).
-
----
-
-## Group 7: Content + Interaction Combined
-
-### 7.1 Read and comment on Go article
-Navigate to `http://fixtures/wiki-go.html`. Read the article, then post a comment with rating 5 stars and text "Excellent reference".
-
-**Verify**: Comment posted (`COMMENT_POSTED_RATING_5_TEXT_RECEIVED`).
-
-### 7.2 Cross-page research task
-Navigate to wiki index, find which category has the most articles, then navigate to one of its listed items.
-
-**Verify**: Successfully navigated to at least one article page after reading the category counts.
-
----
-
-## Group 8: Error Handling
-
-### 8.1 Handle 404 gracefully
-Try to navigate to a page that doesn't exist: `http://fixtures/missing-page-abc.html`.
-
-**Verify**: Got a response (404 or error), no crash, server still responsive after.
-
-### 8.2 Handle missing element gracefully
-On any page, try to click an element with ID `#fake-button-that-does-not-exist`.
-
-**Verify**: Got a clear error message, not a crash or hang.
-
----
-
-## Group 9: Export
-
-### 9.1 Screenshot a complex page
-Navigate to `http://fixtures/dashboard.html` and take a screenshot.
-
-**Verify**: Screenshot generated (file saved or base64 returned).
-
-### 9.2 Export a page as PDF
-Export the dashboard as a PDF.
-
-**Verify**: PDF generated.
-
----
-
-## Group 10: Nested Interactions & Modal Dialogs
-
-### 10.1 Open and interact with modal on dashboard
-Navigate to `http://fixtures/dashboard.html`. Find and click the Settings button (selector: `#settings-btn`) to open the modal dialog.
-
-**Verify**: Modal appeared — snapshot contains "Dashboard Settings".
-
-### 10.2 Modify settings and close modal
-In the modal, select "Dark" from the theme dropdown (`#theme-select`), then click the Save button (`#modal-save`). After the modal closes, check the page content.
-
-**Verify**: Page contains `THEME_DARK_APPLIED`.
-
----
-
-## Group 11: State Persistence & Page Reload
-
-### 11.1 Add an item and verify after page reload
-Navigate to `http://fixtures/spa.html?reset=1` (starts with clean state). Add a task titled exactly "Persistent Task Test". Then navigate away to `http://fixtures/` and back to `http://fixtures/spa.html` (WITHOUT the reset param) to verify localStorage persistence.
-
-**Verify**: After reload, the task still appears in the list (`TASK_PERSISTENT_TEST_FOUND_AFTER_RELOAD`).
-
-### 11.2 Logout and log back in
-From the logged-in dashboard, click Sign Out to log out. Then log in again with username "benchmark" / password "test456".
-
-**Verify**: Successfully logged back in and dashboard shows `SESSION_RENEWED`.
-
----
-
-## Group 12: Multi-Page Navigation & Back Button
-
-### 12.1 Navigate through multiple pages and return
-Starting from `http://fixtures/`, navigate to wiki → Go article → back to wiki → back to home.
-
-**Verify**: Successfully returned to home page (title contains "Benchmark" or "Home").
-
-### 12.2 Compare data across pages
-Navigate to wiki.html, note the total article count from categories. Navigate to articles.html, count articles there. Compare totals.
-
-**Verify**: Can report totals from both pages and explain difference (`COMPARISON_DATA_FOUND`).
-
----
-
-## Group 13: Form State & Multi-Step Submission
-
-### 13.1 Submit form without email
-Navigate to `http://fixtures/form.html`. Fill only the name field ("Validator Test"), leave email blank, click Submit. The browser's native required-field validation will prevent submission.
-
-**Verify**: Submission blocked (form stays open, no success message shown).
-
-### 13.2 Submit form without optional phone field
-Fill the form with: name "No Phone User", email "[email protected]", country "de", subject "feedback". Leave the phone field empty. Submit.
-
-**Verify**: Submission succeeded and page shows `OPTIONAL_FIELD_SKIPPED_SUCCESS`.
-
----
-
-## Group 14: Dynamic Content Loading
-
-### 14.1 Load more products
-Navigate to `http://fixtures/ecommerce.html`. Find and click the "Load More Products" button to reveal additional products.
-
-**Verify**: Additional products appeared (`ADDITIONAL_PRODUCTS_LOADED`).
-
-### 14.2 Add a lazy-loaded product to cart
-After loading more products, add product #5 (USB-C Cable) to the cart.
-
-**Verify**: Cart shows the lazy-loaded item (`CART_UPDATED_WITH_LAZY_PRODUCT`).
-
----
-
-## Group 15: Complex Data Extraction & Aggregation
-
-### 15.1 Extract and sum financial data
-Navigate to `http://fixtures/dashboard.html`. Extract revenue and profit values, calculate profit margin.
-
-**Verify**: Correctly calculated: profit_margin = (profit / revenue) * 100 (`PROFIT_MARGIN_CALCULATED`).
-
-### 15.2 Build comparison table from multiple sources
-Visit these 3 pages and compare their feature counts and key features:
-- `http://fixtures/wiki-go.html` (Go: 6 features)
-- `http://fixtures/wiki-python.html` (Python: 7 features)
-- `http://fixtures/wiki-rust.html` (Rust: 5 features)
-
-Report which language has the most features and name 1 feature unique to each.
-
-**Verify**: Response is factually correct AND wiki-python.html contains `COMPARISON_TABLE_BUILT`.
-
----
-
-## Group 16: Hover & Tooltips
-
-### 16.1 Hover reveals hidden content
-Navigate to `http://fixtures/hovers.html`. Hover over the first avatar to reveal its hidden info.
-
-**Verify**: After hovering, snapshot contains `HOVER_REVEALED_USER_1`.
-
-### 16.2 Hover swap
-Hover over the second avatar.
-
-**Verify**: Snapshot contains `HOVER_REVEALED_USER_2`.
-
----
-
-## Group 17: Scrolling
-
-### 17.1 Scroll by pixels
-Navigate to `http://fixtures/scroll.html`. Scroll down approximately 1500 pixels.
-
-**Verify**: Snapshot or text contains `SCROLL_MIDDLE_MARKER` (the mid-page marker is now in view).
-
-### 17.2 Scroll to footer
-Continue scrolling until the footer is visible (or scroll directly to the footer element).
-
-**Verify**: Snapshot contains `SCROLL_REACHED_FOOTER`.
-
----
-
-## Group 18: File Download
-
-### 18.1 Download a file
-Download `http://fixtures/download-sample.txt` to a local file.
-
-**Verify**: The downloaded file exists and its content includes `DOWNLOAD_FILE_CONTENT_VERIFIED`.
-
----
-
-## Group 19: iFrame
-
-### 19.1 Read iframe content
-Navigate to `http://fixtures/iframe.html` and extract content from inside the embedded same-origin iframe.
-
-**Verify**: The iframe's inner content includes `IFRAME_INNER_CONTENT_LOADED`.
-
-### 19.2 Type into iframe input (native frame scope)
-Use `pinchtab frame '#content-frame'` to scope into the iframe, then `fill` the input with "Hello World" and `click` the Save button. Verify via a scoped `snap` — `text --full` doesn't pierce iframes. Reset with `pinchtab frame main` afterwards.
-
-**Verify**: Scoped snapshot contains `IFRAME_INPUT_RECEIVED_HELLO_WORLD`.
-
----
-
-## Group 20: Dialogs
-
-### 20.1 Accept alert
-Navigate to `http://fixtures/alerts.html`. Click the "Click for Alert" button and dismiss the alert.
-
-**Verify**: The page result contains `DIALOG_ALERT_DISMISSED`.
-
-### 20.2 Cancel confirm
-Click the "Click for Confirm" button and cancel the confirm dialog.
-
-**Verify**: The page result contains `DIALOG_CONFIRM_CANCELLED`.
-
----
-
-## Group 21: Async / awaitPromise
-
-### 21.1 Await a promise-returning function
-Navigate to `http://fixtures/async.html`. The page exposes `window.fetchPayload()`, which returns a `Promise` that resolves after a short delay. Use `eval` to call it and retrieve the **resolved** value, not a Promise wrapper.
-
-**Verify**: The resolved value contains `ASYNC_PAYLOAD_READY_42`.
-
-### 21.2 Await a promise resolving to an object
-On the same page, call `window.fetchUser()` and retrieve the resolved object so you can read a field from it.
-
-**Verify**: The resolved object's `name` field equals `ASYNC_USER_NAME_ADA`.
-
----
-
-## Group 22: Mouse Drag & Drop
-
-### 22.1 Drag a piece into Zone A
-Navigate to `http://fixtures/drag.html`. The page contains a draggable square (`#piece`) and three target zones (`#zone-a`, `#zone-b`, `#zone-c`). Drag the piece so its center ends up over **Zone A**.
-
-**Verify**: The page shows `LAST_DROP=DROP_ZONE_A_OK`.
-
-### 22.2 Drag to Zone B, then Zone C
-Without reloading the page, drag the piece next into **Zone B**, and then into **Zone C**. The page records an ordered drop sequence.
-
-**Verify**: The page shows `DROP_SEQUENCE=DROP_ZONE_A_OK,DROP_ZONE_B_OK,DROP_ZONE_C_OK` (all three drops in order).
-
----
-
-## Group 23: Async / Loading state
-
-### 23.1 Wait for async content
-Navigate to `http://fixtures/loading.html`. The page shows a spinner and the text "Loading..." for ~1.5 seconds, then replaces it with the final payload. Wait for the final content to appear before reporting completion.
-
-**Verify**: The final snapshot contains `VERIFY_LOADING_COMPLETE_88888`.
-
----
-
-## Group 24: Keyboard events
-
-### 24.1 Press Escape
-Navigate to `http://fixtures/keyboard.html` (it auto-focuses an input so keyboard events land). Press the Escape key.
-
-**Verify**: Page text contains `KEYBOARD_ESCAPE_PRESSED`.
-
-### 24.2 Press 'a' then Enter
-Without reloading, press the `a` key, then the `Enter` key.
-
-**Verify**: Page text now contains all three markers in order: `KEYBOARD_ESCAPE_PRESSED`, `KEYBOARD_KEY_A_PRESSED`, `KEYBOARD_ENTER_PRESSED`.
-
----
-
-## Group 25: Tab panels
-
-### 25.1 Switch to the Settings tab
-Navigate to `http://fixtures/tabs.html`. The Profile tab is shown by default. Click the Settings tab.
-
-**Verify**: After the click, the page content includes `TAB_SETTINGS_CONTENT` (and the Profile content is no longer visible).
-
-### 25.2 Switch to the Billing tab
-Click the Billing tab.
-
-**Verify**: Page content now includes `TAB_BILLING_CONTENT`.
-
----
-
-## Group 26: Accordion
-
-### 26.1 Open section A
-Navigate to `http://fixtures/accordion.html`. Click the header for Section A to expand it.
-
-**Verify**: Page text includes `ACCORDION_SECTION_A_OPEN`.
-
-### 26.2 Open section B
-Click the header for Section B. Because the accordion is exclusive-expand, Section A should close.
-
-**Verify**: Page text includes `ACCORDION_SECTION_B_OPEN`, and Section A's `aria-expanded` attribute is now `"false"` (use `eval` to check the attribute if needed).
-
----
-
-## Group 27: Contenteditable editor
-
-### 27.1 Type into the rich-text editor
-Navigate to `http://fixtures/editor.html`. Type the text `Hello rich text` into the `#editor` div. `contenteditable` elements don't have `.value`, so use `type` (keyboard events), not `fill`.
-
-**Verify**: Page text contains `EDITOR_CHARS=15` and the mirror shows `Hello rich text`.
-
-### 27.2 Commit by pressing Enter
-Press Enter (the editor intercepts Enter to commit the current buffer to a separate marker).
-
-**Verify**: Page text contains `EDITOR_COMMITTED=Hello rich text`.
-
----
-
-## Group 28: Range slider
-
-### 28.1 Move slider into HIGH bucket
-Navigate to `http://fixtures/range.html`. Set the `#volume` range slider to 90.
-
-**Verify**: Page text contains both `RANGE_VALUE_90` and `BUCKET_HIGH`.
-
-### 28.2 Move slider into LOW bucket
-Without reloading, change the slider to 10.
-
-**Verify**: Page text now contains `RANGE_VALUE_10` and `BUCKET_LOW`.
-
----
-
-## Group 29: Pagination
-
-### 29.1 Advance to page 2
-Navigate to `http://fixtures/pagination.html`. Click the Next button.
-
-**Verify**: Page text contains `PAGE_2_FIRST_ITEM` and `PAGE_2_OF_3`.
-
-### 29.2 Reach the last page; Next disabled
-Click Next once more.
-
-**Verify**: Page text contains `PAGE_3_FIRST_ITEM` and `PAGE_3_OF_3`, and the Next button's `disabled` property is `true` (use `eval` to check).
-
----
-
-## Group 30: Custom dropdown menu
-
-### 30.1 Pick the "Beta" item
-Navigate to `http://fixtures/dropdown.html`. The page has a custom dropdown — click the toggle button (`#dropdown-toggle`) to open the menu, then click the "Beta" item.
-
-**Verify**: Page text contains `DROPDOWN_SELECTED=BETA`.
-
-### 30.2 Reopen and pick "Gamma"
-Reopen the dropdown and select "Gamma". (Each selection auto-closes the menu, so you need to click the toggle again before selecting.)
-
-**Verify**: Page text contains `DROPDOWN_SELECTED=GAMMA`.
-
----
-
-## Group 31: Nested iframes (3 levels deep)
-
-### 31.1 Click a button in the deepest frame
-Navigate to `http://fixtures/iframe-nested.html`. The outer page embeds `#level-2`, which in turn embeds `#level-3`, which contains `#deep-button`. Drill through the two frame hops and click the button. Verify via a scoped `snap` before resetting to `main`.
-
-**Verify**: Scoped snapshot contains `DEEP_CLICKED=YES_LEVEL_3`.
-
----
-
-## Group 32: Dynamic iframe (inserted after load)
-
-### 32.1 Wait for a late iframe, then interact with it
-Navigate to `http://fixtures/iframe-dynamic.html`. The iframe is inserted ~1.2 s after load (use `wait --text "IFRAME_DYNAMIC_ATTACHED"`). Then scope into `#late-frame`, fill `#iframe-input` with "Late World", click `#iframe-submit`, and verify the inner result marker.
-
-**Verify**: Scoped snapshot contains `IFRAME_INPUT_RECEIVED_LATE_WORLD`.
-
----
-
-## Group 33: srcdoc iframe
-
-### 33.1 Interact with an inline-content iframe
-Navigate to `http://fixtures/iframe-srcdoc.html`. The iframe's content is inlined via `srcdoc`; it has no `src` URL. Scope into `#srcdoc-frame`, fill `#inline-input` with `srcdoc`, click `#inline-submit`, and verify.
-
-**Verify**: Scoped snapshot contains `INLINE_RECEIVED_SRCDOC`.
-
----
-
-## Group 34: Sandboxed iframe
-
-### 34.1 Click inside a sandboxed iframe
-Navigate to `http://fixtures/iframe-sandbox.html`. The iframe has `sandbox="allow-scripts allow-same-origin"`. Scope into `#sandboxed`, click `#sandbox-button`, and verify.
-
-**Verify**: Scoped snapshot contains `SANDBOX_CLICKED=YES`.
-
----
-
-## Group 35: Long-form article (Medium/Substack style)
-
-### 35.1 Read the article with default `text`
-Navigate to `http://fixtures/article.html`. Extract the page's main content using the default `text` (Readability) mode. Default mode is the right choice for article-style pages.
-
-**Verify**: The extracted text contains both `ARTICLE_PUBLISHED_2026_04_15` and `ARTICLE_WORD_COUNT_MARKER_323` (both inside the article body — Readability keeps them).
-
-### 35.2 See the chrome that Readability drops
-Re-extract with `text --full` and confirm the footer is included this time.
-
-**Verify**: `--full` output contains `FOOTER_COPYRIGHT_MARKER` (which the default mode trims).
-
----
-
-## Group 36: Search results page (SERP)
-
-### 36.1 Find a specific result by id
-Navigate to `http://fixtures/serp.html`. The page has 6 result cards (`#r-1`..`#r-6`). Extract just the third card's content. A scoped snapshot (`snap --selector "#r-3"`) is the direct path.
-
-**Verify**: The scoped output contains `RESULT_3_TITLE` and `RESULT_3_SNIPPET_MARKER`.
-
-### 36.2 Count all result cards
-Use a full text extraction to verify all six results are present in one pass.
-
-**Verify**: Output contains all of `RESULT_1_TITLE` through `RESULT_6_TITLE` and the summary `SERP_RESULT_COUNT_6`. Default Readability trims SERPs — use `text --full`.
-
----
-
-## Group 37: Q&A thread (Stack-Overflow style)
-
-### 37.1 Find the accepted answer id
-Navigate to `http://fixtures/qa.html`. The accepted answer carries `data-accepted="true"` on its `<div>` wrapper. Use `eval` to return that element's `id`.
-
-**Verify**: The `eval` result equals `"a-2"`.
-
-### 37.2 Extract the accepted answer's body
-Scope a snapshot to `#a-2` and verify the body content.
-
-**Verify**: Scoped snapshot contains `ANSWER_2_BODY_MARKER` and `ACCEPTED_ANSWER_ID_A2`.
-
----
-
-## Group 38: Pricing table
-
-### 38.1 Read just the Pro plan
-Navigate to `http://fixtures/pricing.html`. The page has three plan cards (`#plan-free`, `#plan-pro`, `#plan-enterprise`). Read only the Pro plan using a scoped snapshot.
-
-**Verify**: Scoped snapshot contains `PLAN_PRO_PRICE_29` and `PLAN_PRO_LIMIT_5000_requests per day`.
-
-### 38.2 Compare all three plans
-Extract the full page (pricing grids are a Readability anti-pattern — reach for `--full`).
-
-**Verify**: Output contains `PLAN_FREE_PRICE_0`, `PLAN_PRO_PRICE_29`, and `PLAN_ENTERPRISE_PRICE_CUSTOM`.
-
----
-
-## Summary
-
-| Group | Tasks | Description |
-|-------|-------|-------------|
-| 0 | 8 | Setup & Diagnosis |
-| 1 | 6 | Reading & Extracting Content |
-| 2 | 3 | Search & Dynamic Interaction |
-| 3 | 2 | Complex Form |
-| 4 | 3 | SPA State Management |
-| 5 | 2 | Login Flow |
-| 6 | 3 | Multi-Step E-commerce |
-| 7 | 2 | Content + Interaction Combined |
-| 8 | 2 | Error Handling |
-| 9 | 2 | Export |
-| 10 | 2 | Nested Interactions & Modal Dialogs |
-| 11 | 2 | State Persistence & Page Reload |
-| 12 | 2 | Multi-Page Navigation & Back Button |
-| 13 | 2 | Form State & Multi-Step Submission |
-| 14 | 2 | Dynamic Content Loading |
-| 15 | 2 | Complex Data Extraction & Aggregation |
-| 16 | 2 | Hover & Tooltips |
-| 17 | 2 | Scrolling |
-| 18 | 1 | File Download |
-| 19 | 2 | iFrame |
-| 20 | 2 | Dialogs |
-| 21 | 2 | Async / awaitPromise |
-| 22 | 2 | Mouse Drag & Drop |
-| 23 | 1 | Async / Loading state |
-| 24 | 2 | Keyboard events |
-| 25 | 2 | Tab panels |
-| 26 | 2 | Accordion |
-| 27 | 2 | Contenteditable editor |
-| 28 | 2 | Range slider |
-| 29 | 2 | Pagination |
-| 30 | 2 | Custom dropdown menu |
-| 31 | 1 | Nested iframes (3 levels) |
-| 32 | 1 | Dynamic iframe |
-| 33 | 1 | srcdoc iframe |
-| 34 | 1 | Sandboxed iframe |
-| 35 | 2 | Long-form article |
-| 36 | 2 | Search results page |
-| 37 | 2 | Q&A thread |
-| 38 | 2 | Pricing table |
-
-**Total: 85 tasks**
-
-## Key Differences from Baseline
-
-The agent must:
-- Choose between `/text`, `/snapshot`, `/action`, `/navigate` appropriately
-- Decide when to use `filter=interactive` vs full snapshot
-- Handle multi-step flows without step-by-step curl guidance
-- Extract and interpret structured data (tables, lists, counts)
-- Detect state changes after interactions
diff --git a/tests/benchmark/BASELINE_TASKS.md b/tests/benchmark/BASELINE_TASKS.md
deleted file mode 100644
index 16efe850..00000000
--- a/tests/benchmark/BASELINE_TASKS.md
+++ /dev/null
@@ -1,1830 +0,0 @@
-# PinchTab Benchmark Tasks (Baseline)
-
-Reproducible benchmark using explicit curl commands against controlled fixture pages.
-Each task maps 1:1 to an agent task in AGENT_TASKS.md for direct comparison.
-
-## Environment
-
-- **PinchTab Server**: `http://localhost:9867`
-- **Fixtures Server**: `http://fixtures/` (inside Docker network)
-- **Token**: `benchmark-token`
-- **Auth Header**: `Authorization: Bearer benchmark-token`
-
-## MANDATORY: Docker
-
-```bash
-cd ~/dev/pinchtab/tests/benchmark
-docker compose up -d --build
-# Wait for healthy: curl -sf -H "Authorization: Bearer benchmark-token" http://localhost:9867/health
-```
-
-## Recording
-
-```bash
-# Baseline execution answer:
-./scripts/record-step.sh --type baseline <group> <step> answer "<observed result>" "notes"
-
-# Immediate verification:
-./scripts/verify-step.sh --type baseline <group> <step> <pass|fail|skip> "verification notes"
-```
-
-**On failure, include in notes:**
-- What was expected
-- What was actually returned
-- HTTP status code / error message
-
-## Tab Reuse
-
-`POST /navigate` creates a new tab by default. To avoid multi-tab issues, the
-runner must:
-
-1. Capture `tabId` from the first navigate response (step 0.2).
-2. Pass `"tabId":"TAB_ID"` in every subsequent navigate request body.
-3. Use tab-scoped endpoints for actions and snapshots:
- - `POST /tabs/TAB_ID/action` instead of `POST /action`
- - `GET /tabs/TAB_ID/snapshot?...` instead of `GET /snapshot?...`
- - `GET /tabs/TAB_ID/text` instead of `GET /text`
- - `POST /tabs/TAB_ID/back` instead of `POST /back`
- - `GET /tabs/TAB_ID/screenshot` instead of `GET /screenshot`
- - `GET /tabs/TAB_ID/pdf` instead of `GET /pdf`
-
-All curl examples below use `TAB_ID` as a placeholder. Replace with the actual
-tab ID captured in step 0.2.
-
----
-
-## Group 0: Setup & Diagnosis
-
-### 0.1 Server reachable
-```bash
-curl -sf http://localhost:9867/health \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: `status == "ok"` in response body.
-
-### 0.2 Auth required
-```bash
-curl -s -o /dev/null -w "%{http_code}" http://localhost:9867/health
-```
-**Pass if**: HTTP status is `401` (auth rejected without token).
-
-### 0.3 Auth works with token
-```bash
-curl -s -o /dev/null -w "%{http_code}" http://localhost:9867/health \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: HTTP status is `200`.
-
-### 0.4 Instance available
-```bash
-curl -sf http://localhost:9867/health \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Response contains `defaultInstance.status == "running"`. If not, run:
-```bash
-curl -X POST http://localhost:9867/instances/start \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{}'
-```
-And re-verify the new instance is running.
-
-### 0.5 List existing tabs
-```bash
-curl -sf http://localhost:9867/tabs \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Returns a JSON array (possibly empty) without error.
-
-### 0.6 Clean stale tabs
-For each tab returned by step 0.5, close it:
-```bash
-curl -X POST http://localhost:9867/tabs/TAB_ID/close \
- -H "Authorization: Bearer benchmark-token"
-```
-Then verify cleanup:
-```bash
-curl -sf http://localhost:9867/tabs \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: After cleanup, the tab list is empty or contains only an about:blank tab.
-
-### 0.7 Network reach to target
-```bash
-curl -sf -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"url":"http://fixtures/"}'
-```
-**Capture**: Save the `tabId` from the JSON response. All subsequent commands use this tab ID.
-```bash
-curl -sf "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Navigate returns HTTP 200 AND snapshot contains `VERIFY_HOME_LOADED_12345`.
-
-### 0.8 Capture initial tab ID
-```bash
-curl -sf http://localhost:9867/tabs \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: The captured `TAB_ID` from step 0.7 appears in the tabs list.
-
----
-
-## Group 1: Reading & Extracting
-
-### 1.1 Wiki categories
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki.html"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains `VERIFY_WIKI_INDEX_55555` AND `COUNT_LANGUAGES_12` AND `COUNT_TOOLS_15`.
-
-### 1.2 Click a link
-```bash
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#link-go","waitNav":true}'
-```
-**Pass if**: HTTP 200 with `{"success":true}`.
-
-### 1.3 Table extraction
-```bash
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains `VERIFY_WIKI_GO_LANG_88888` AND `Robert Griesemer` AND `2009`.
-
-### 1.4 Count list items
-```bash
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains `FEATURE_COUNT_6`.
-
-### 1.5 Article headlines
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/articles.html"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains `The Future of Artificial Intelligence` AND `Climate Action in 2026` AND `Mars Colony`.
-
-### 1.6 Dashboard metrics
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/dashboard.html"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains `24,582` AND `$1,284,930` AND `4.28%`.
-
----
-
-## Group 2: Search & Dynamic
-
-### 2.1 Wiki search
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki.html"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#wiki-search-input","text":"golang"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#wiki-search-btn","waitNav":true}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Snapshot contains `VERIFY_WIKI_GO_LANG_88888` (search redirected to Go page).
-
-### 2.2 No results search
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/search.html"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#search-input","text":"xyznonexistent"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#search-btn"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: All curls return HTTP 200 AND snapshot shows no-results message.
-
-### 2.3 AI content search
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/search.html"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#search-input","text":"artificial intelligence"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#search-btn"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Snapshot contains `The Future of Artificial Intelligence`.
-
----
-
-## Group 3: Form
-
-### 3.1 Complete form
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/form.html"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#fullname","text":"John Benchmark"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#email","text":"[email protected]"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#phone","text":"+44 20 1234 5678"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"select","selector":"#country","value":"uk"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"select","selector":"#subject","value":"support"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#message","text":"This is a benchmark test message for PinchTab automation."}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#newsletter"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"input[name=priority][value=high]"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#submit-btn"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains `VERIFY_FORM_SUBMITTED_SUCCESS` AND `SUBMISSION_DATA_NAME_JOHN_BENCHMARK`.
-
-### 3.2 Reset/refill
-```bash
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?filter=interactive&format=compact" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains reset button or form element (after submission, page still has interactive elements).
-
----
-
-## Group 4: SPA
-
-### 4.1 Read app state
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/spa.html?reset=1"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains `VERIFY_SPA_PAGE_99999` AND `TASK_STATS_TOTAL_3_ACTIVE_2_DONE_1`.
-
-### 4.2 Add task
-```bash
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#new-task-input","text":"Deploy to production"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"select","selector":"#priority-select","value":"high"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#add-task-btn"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains `TASK_ADDED_DEPLOY_TO_PRODUCTION_PRIORITY_HIGH`.
-
-### 4.3 Delete task
-```bash
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#task-1 .delete-task"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Total count is `3` (started with 3, added 1, deleted 1 = 3).
-
----
-
-## Group 5: Login
-
-### 5.1 Invalid login
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/login.html"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#username","text":"baduser"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#password","text":"wrongpassword"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#login-btn"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains `INVALID_CREDENTIALS_ERROR`.
-
-### 5.2 Valid login
-```bash
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#username","text":"benchmark"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#password","text":"test456"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#login-btn"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains `VERIFY_LOGIN_SUCCESS_DASHBOARD` AND `SESSION_TOKEN_ACTIVE_TRUE`.
-
----
-
-## Group 6: E-commerce
-
-### 6.1 Research products
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/ecommerce.html"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains `VERIFY_SHOP_PAGE_44444` AND `$149.99` AND `$299.99` AND `$49.99` AND `Out of Stock`.
-
-### 6.2 Add to cart
-```bash
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#product-1 .add-to-cart"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#product-2 .add-to-cart"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains `CART_ITEM_WIRELESS_HEADPHONES` AND `CART_ITEM_SMART_WATCH_PRO` AND `449.98`.
-
-### 6.3 Checkout
-```bash
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#checkout-btn"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains `VERIFY_CHECKOUT_SUCCESS_ORDER` AND `ORDER_TOTAL_449_98`.
-
----
-
-## Group 7: Content + Interaction
-
-### 7.1 Read & comment
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki-go.html"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
- -H "Authorization: Bearer benchmark-token"
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#comment-text","text":"Great article on Go! Very comprehensive."}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"select","selector":"#comment-rating","value":"5"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#submit-comment"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: First snapshot contains `VERIFY_WIKI_GO_LANG_88888` AND final snapshot contains `COMMENT_POSTED_RATING_5_TEXT_RECEIVED`.
-
-### 7.2 Cross-page research
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki.html"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \
- -H "Authorization: Bearer benchmark-token"
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#link-go","waitNav":true}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: First snapshot contains `COUNT_LANGUAGES_12` AND second snapshot contains `VERIFY_WIKI_GO_LANG_88888`.
-
----
-
-## Group 8: Error Handling
-
-### 8.1 404 handling
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/nonexistent-page-xyz.html"}'
-```
-**Pass if**: Returns response without crash (HTTP 200 with error page, or structured error).
-
-### 8.2 Missing element
-```bash
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#element-that-does-not-exist"}'
-```
-**Pass if**: Error response with clear message (not crash).
-
----
-
-## Group 9: Export
-
-### 9.1 Screenshot
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/dashboard.html"}'
-
-curl http://localhost:9867/tabs/TAB_ID/screenshot \
- -H "Authorization: Bearer benchmark-token" \
- --output /tmp/benchmark-screenshot.png
-
-ls -la /tmp/benchmark-screenshot.png
-```
-**Pass if**: File exists and size > 10240 bytes.
-
-### 9.2 PDF export
-```bash
-curl http://localhost:9867/tabs/TAB_ID/pdf \
- -H "Authorization: Bearer benchmark-token" \
- --output /tmp/benchmark-dashboard.pdf
-
-ls -la /tmp/benchmark-dashboard.pdf
-```
-**Pass if**: File exists and size > 10240 bytes.
-
----
-
-## Group 10: Modals
-
-### 10.1 Open modal
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/dashboard.html"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#settings-btn"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Snapshot contains `Dashboard Settings`.
-
-### 10.2 Modal interaction
-```bash
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"select","selector":"#theme-select","value":"dark"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#modal-save"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains `THEME_DARK_APPLIED`.
-
----
-
-## Group 11: Persistence
-
-### 11.1 State after reload
-```bash
-# Start fresh (reset param clears localStorage on this load only)
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/spa.html?reset=1"}'
-
-# Add the persistent task
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#new-task-input","text":"Persistent Task Test"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#add-task-btn"}'
-
-# Navigate away then back WITHOUT reset param — state should persist in localStorage
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/"}'
-
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/spa.html"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains `TASK_PERSISTENT_TEST_FOUND_AFTER_RELOAD`.
-
-### 11.2 Logout/re-login
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/login.html"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#username","text":"benchmark"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#password","text":"test456"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#login-btn"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#logout-btn"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#username","text":"benchmark"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#password","text":"test456"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#login-btn"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains `VERIFY_LOGIN_SUCCESS_DASHBOARD` AND `SESSION_RENEWED`.
-
----
-
-## Group 12: Multi-page Nav
-
-### 12.1 Navigate & return
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/"}'
-
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki.html"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#link-go","waitNav":true}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/back \
- -H "Authorization: Bearer benchmark-token"
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/back \
- -H "Authorization: Bearer benchmark-token"
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Final snapshot contains `VERIFY_HOME_LOADED_12345` (returned to home).
-
-### 12.2 Cross-page compare
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki.html"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \
- -H "Authorization: Bearer benchmark-token"
-
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/articles.html"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Wiki snapshot contains `COUNT_LANGUAGES_12` AND articles snapshot contains article titles.
-
----
-
-## Group 13: Form Validation
-
-### 13.1 Required field
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/form.html"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#fullname","text":"Validator Test"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#submit-btn"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Snapshot does NOT contain `VERIFY_FORM_SUBMITTED_SUCCESS` (submission blocked by validation).
-
-### 13.2 Optional field
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/form.html"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#fullname","text":"No Phone User"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#email","text":"[email protected]"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"select","selector":"#country","value":"de"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"select","selector":"#subject","value":"feedback"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#submit-btn"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains `VERIFY_FORM_SUBMITTED_SUCCESS` AND `OPTIONAL_FIELD_SKIPPED_SUCCESS`.
-
----
-
-## Group 14: Dynamic Content
-
-### 14.1 Load more
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/ecommerce.html"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#load-more-btn"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains `ADDITIONAL_PRODUCTS_LOADED`.
-
-### 14.2 Lazy-loaded item
-```bash
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#product-5 .add-to-cart"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains `CART_UPDATED_WITH_LAZY_PRODUCT`.
-
----
-
-## Group 15: Data Aggregation
-
-### 15.1 Financial calc
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/dashboard.html"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains `$1,284,930` (revenue) AND `$384,930` (profit) AND `PROFIT_MARGIN_CALCULATED`.
-
-### 15.2 Multi-page comparison
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki-go.html"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
- -H "Authorization: Bearer benchmark-token"
-
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki-python.html"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
- -H "Authorization: Bearer benchmark-token"
-
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki-rust.html"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Go snapshot contains `FEATURE_COUNT_6` AND Python snapshot contains `FEATURE_COUNT_7` AND `COMPARISON_TABLE_BUILT` AND Rust snapshot contains `FEATURE_COUNT_5`.
-
----
-
-## Group 16: Hover & Tooltips
-
-### 16.1 Hover reveals info
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/hovers.html"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"hover","selector":"#avatar-1"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Snapshot contains `HOVER_REVEALED_USER_1`.
-
-### 16.2 Hover swap
-```bash
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"hover","selector":"#avatar-2"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Snapshot contains `HOVER_REVEALED_USER_2`.
-
----
-
-## Group 17: Scrolling
-
-### 17.1 Scroll by pixels
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/scroll.html"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"scroll","scrollY":1500}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Snapshot contains `SCROLL_MIDDLE_MARKER`.
-
-### 17.2 Scroll to footer
-```bash
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"scroll","selector":"#footer"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Snapshot contains `SCROLL_REACHED_FOOTER`.
-
----
-
-## Group 18: File Download
-
-### 18.1 Download a file
-```bash
-curl "http://localhost:9867/tabs/TAB_ID/download?url=http://fixtures/download-sample.txt" \
- -H "Authorization: Bearer benchmark-token" | \
- jq -r .data | base64 -d > /tmp/benchmark-download.txt
-
-grep "DOWNLOAD_FILE_CONTENT_VERIFIED" /tmp/benchmark-download.txt
-```
-**Pass if**: File exists and contains `DOWNLOAD_FILE_CONTENT_VERIFIED`.
-
-**Note**: The download endpoint returns JSON with `{contentType, data (base64), size, url}`, not the raw file. Decode `.data` to get the file contents. For PinchTab to download from internal hosts (like `fixtures`), the domain must be in `security.downloadAllowedDomains` config.
-
----
-
-## Group 19: iFrame
-
-### 19.1 Read iframe content
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/iframe.html"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Snapshot contains `IFRAME_INNER_CONTENT_LOADED`.
-
-### 19.2 Type into iframe input (native frame scope)
-```bash
-# Scope into the same-origin iframe
-curl -X POST http://localhost:9867/frame?tabId=TAB_ID \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"target":"#content-frame"}'
-
-# Selectors now resolve inside the iframe
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#iframe-input","text":"Hello World"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#iframe-submit"}'
-
-# Verify via a scoped snap (text --full doesn't pierce frames)
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
- -H "Authorization: Bearer benchmark-token"
-
-# Reset scope
-curl -X POST http://localhost:9867/frame?tabId=TAB_ID \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"target":"main"}'
-```
-**Pass if**: The scoped snapshot contains `IFRAME_INPUT_RECEIVED_HELLO_WORLD`.
-
-**Note**: `POST /frame` sets a stateful scope on the tab. Subsequent selector-based `/action` and `/snapshot` calls resolve inside the scoped iframe. Reset with `{"target":"main"}` when done. Cross-origin iframes are not currently exposed this way — fall back to `/evaluate` + `contentDocument` for those (when same-origin policy permits).
-
----
-
-## Group 20: Dialogs
-
-### 20.1 Accept alert
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/alerts.html"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#alert-btn","dialogAction":"accept"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Snapshot contains `DIALOG_ALERT_DISMISSED`.
-
-### 20.2 Cancel confirm
-```bash
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#confirm-btn","dialogAction":"dismiss"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Snapshot contains `DIALOG_CONFIRM_CANCELLED`.
-
-**Note**: Use `dialogAction` on the click action to auto-accept or auto-dismiss a JS dialog that the click opens. Without it, the click would hang until `/dialog` is called from a separate request.
-
----
-
-## Group 21: Async / awaitPromise
-
-### 21.1 Await a promise-returning function
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/async.html"}'
-
-# Without awaitPromise: returns {} (unresolved Promise representation)
-curl -X POST http://localhost:9867/tabs/TAB_ID/evaluate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"expression":"window.fetchPayload()","awaitPromise":true}'
-```
-**Pass if**: Response `.result` equals `"ASYNC_PAYLOAD_READY_42"`. Also verify that repeating the same call with `awaitPromise` omitted returns `{}` (proves `awaitPromise` is actually doing work).
-
-### 21.2 Await a promise resolving to an object
-```bash
-curl -X POST http://localhost:9867/tabs/TAB_ID/evaluate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"expression":"window.fetchUser()","awaitPromise":true}'
-```
-**Pass if**: Response `.result.name` equals `"ASYNC_USER_NAME_ADA"` (structured value survives the round-trip).
-
-**Note**: See SKILL.md and `references/api.md` — `awaitPromise:true` on `/evaluate` makes the server wait for the returned Promise to settle before responding. Omit the flag when you want the raw Promise reference.
-
----
-
-## Group 22: Mouse Drag & Drop
-
-### 22.1 Drag piece into Zone A (high-level `drag` action)
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/drag.html"}'
-
-# Piece center starts at (92, 358); Zone A center is at (104, 200).
-# Drag offset: (+12, -158).
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"drag","selector":"#piece","dragX":12,"dragY":-158}'
-
-curl "http://localhost:9867/tabs/TAB_ID/text" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Page text contains `LAST_DROP=DROP_ZONE_A_OK`.
-
-### 22.2 Low-level sequence: drag into Zone B, then Zone C
-```bash
-# Zone centers in viewport coords (for the default window size):
-# piece starts: (92, 358)
-# Zone A: (104, 200)
-# Zone B: (344, 200)
-# Zone C: (584, 400)
-# If the viewport differs, query these with /evaluate +
-# Array.from(['piece','zone-a','zone-b','zone-c']).map(id => ...getBoundingClientRect()).
-
-# Drag piece (now at Zone A) to Zone B via explicit mouse-down → mouse-move → mouse-up.
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"mouse-move","x":104,"y":200}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"mouse-down","x":104,"y":200,"button":"left"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"mouse-move","x":344,"y":200}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"mouse-up","x":344,"y":200,"button":"left"}'
-
-# Now drag Zone B → Zone C
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"mouse-down","x":344,"y":200,"button":"left"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"mouse-move","x":584,"y":400}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"mouse-up","x":584,"y":400,"button":"left"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/text" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Page text contains `DROP_SEQUENCE=DROP_ZONE_A_OK,DROP_ZONE_B_OK,DROP_ZONE_C_OK`.
-
-**Note**: Either path (the high-level `drag` action with `dragX`/`dragY` offsets, or the explicit `mouse-down → mouse-move → mouse-up` sequence at absolute viewport coordinates) works. Use `drag` for simple point-to-point drags, and the low-level trio when the target site depends on intermediate `mousemove` events or when you need precise pacing.
-
----
-
-## Group 23: Async / Loading state
-
-### 23.1 Wait for async content to replace a loading spinner
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/loading.html"}'
-
-# Poll the snapshot until the final marker appears, or call /wait with a text
-# predicate. Using /evaluate with awaitPromise is another valid path.
-for i in 1 2 3 4 5 6 7 8 9 10; do
- SNAP=$(curl -sf "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
- -H "Authorization: Bearer benchmark-token")
- echo "$SNAP" | grep -q "VERIFY_LOADING_COMPLETE_88888" && break
- sleep 0.3
-done
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Final snapshot contains `VERIFY_LOADING_COMPLETE_88888`.
-
-**Note**: The fixture delays replacing the spinner by ~1.5 s. Agents using
-`pinchtab wait --text "VERIFY_LOADING_COMPLETE_88888" --timeout 5000` hit
-this in a single call. Baseline callers can poll or `sleep` between
-snapshots.
-
----
-
-## Group 24: Keyboard events
-
-### 24.1 Press Escape to emit a marker
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/keyboard.html"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"press","key":"Escape"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Page text contains `KEYBOARD_ESCAPE_PRESSED`. Use `mode=raw` because the log div is short enough for Readability to drop it.
-
-### 24.2 Sequential keys (a then Enter)
-```bash
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"press","key":"a"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"press","key":"Enter"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/text" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Page text contains both `KEYBOARD_KEY_A_PRESSED` and `KEYBOARD_ENTER_PRESSED` (and the Escape marker from 24.1 is still there, proving the log accumulates).
-
-**Note**: `press` issues a single keypress and requires the page to already have focus on an element that listens for `keydown`. The fixture auto-focuses `#focus-input` on load.
-
----
-
-## Group 25: Tab panels
-
-### 25.1 Switch to Settings tab
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/tabs.html"}'
-
-# Verify initial panel
-curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \
- -H "Authorization: Bearer benchmark-token"
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#tab-settings"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Initial response contains `TAB_PROFILE_CONTENT` AND final response contains `TAB_SETTINGS_CONTENT` (and the profile content is now hidden).
-
-### 25.2 Switch to Billing tab
-```bash
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#tab-billing"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Response contains `TAB_BILLING_CONTENT`.
-
-**Note**: Tab panels toggle via the `hidden` attribute; the non-selected panels are absent from the visible text. Use `text?mode=raw` so the extraction reflects what the user actually sees.
-
----
-
-## Group 26: Accordion
-
-### 26.1 Open section A
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/accordion.html"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#section-a .section-header"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Response contains `ACCORDION_SECTION_A_OPEN`.
-
-### 26.2 Open section B — A auto-closes
-```bash
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#section-b .section-header"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \
- -H "Authorization: Bearer benchmark-token"
-
-# Also verify the A body text is no longer visible by inspecting aria-expanded.
-curl -X POST http://localhost:9867/tabs/TAB_ID/evaluate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"expression":"document.getElementById(\"section-a\").getAttribute(\"aria-expanded\")"}'
-```
-**Pass if**: Text response contains `ACCORDION_SECTION_B_OPEN`, AND the `evaluate` response for the section-a state returns `"false"`.
-
-**Note**: Accordion uses an exclusive-expand pattern (only one section open). CSS `max-height:0` collapses the closed bodies, but the text content is still in the DOM — confirming aria state is the most reliable check.
-
----
-
-## Group 27: Contenteditable editor
-
-### 27.1 Type into a `contenteditable` div
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/editor.html"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"type","selector":"#editor","text":"Hello rich text"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Response contains `EDITOR_CHARS=15` AND the mirror echoes `Hello rich text`.
-
-### 27.2 Commit by pressing Enter
-```bash
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"press","key":"Enter"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Response contains `EDITOR_COMMITTED=Hello rich text`.
-
-**Note**: `contenteditable` elements don't have a `.value` property, so `fill` won't work. Use `type` so each character is dispatched via keyboard events the browser routes into the editor. `press Enter` is intercepted (preventDefault) to commit the buffer instead of inserting a newline.
-
----
-
-## Group 28: Range slider
-
-### 28.1 Set range slider to HIGH bucket
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/range.html"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#volume","text":"90"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Response contains `RANGE_VALUE_90` AND `BUCKET_HIGH`.
-
-### 28.2 Move to LOW bucket
-```bash
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#volume","text":"10"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Response contains `RANGE_VALUE_10` AND `BUCKET_LOW`.
-
-**Note**: `fill` calls the native value setter, which fires both `input` and `change` events. That's enough to drive a page that listens via `addEventListener('input', ...)`. If the target only listens to pointer drag events, `drag --drag-x N` on the thumb would be needed — but most sites use `input`/`change`.
-
----
-
-## Group 29: Pagination
-
-### 29.1 Advance to page 2
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/pagination.html"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#next-btn"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Response contains `PAGE_2_FIRST_ITEM` AND `PAGE_2_OF_3` (and the page-1 marker is gone).
-
-### 29.2 Advance to last page; Next is disabled
-```bash
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#next-btn"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \
- -H "Authorization: Bearer benchmark-token"
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/evaluate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"expression":"document.getElementById(\"next-btn\").disabled"}'
-```
-**Pass if**: Text contains `PAGE_3_FIRST_ITEM` AND `PAGE_3_OF_3` AND the evaluate response is `true` (Next button is disabled on the last page).
-
-**Note**: Pagination buttons re-render the list in place. If you re-snap after the click, refs `e0..eN` for the list items will be different than before.
-
----
-
-## Group 30: Custom dropdown menu
-
-### 30.1 Open the dropdown and pick "Beta"
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/dropdown.html"}'
-
-# Open the menu
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#dropdown-toggle"}'
-
-# Pick the "Beta" item by its data-value attribute
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#dropdown-menu li[data-value=\"beta\"]"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Response contains `DROPDOWN_SELECTED=BETA`.
-
-### 30.2 Reopen and pick "Gamma"
-```bash
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#dropdown-toggle"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#dropdown-menu li[data-value=\"gamma\"]"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Response contains `DROPDOWN_SELECTED=GAMMA`.
-
-**Note**: This is a non-native dropdown — it's a `<button>` that toggles a `<ul>`. The menu items are not `<option>` elements, so `pinchtab select` won't work. Use two clicks (toggle, then item). Click-outside also dismisses the menu, so don't do anything between the two clicks that would synthesize a body click.
-
----
-
-## Group 31: Nested iframes (3 levels deep)
-
-### 31.1 Drill into the deepest frame and click its button
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/iframe-nested.html"}'
-
-# Hop into level 2
-curl -X POST http://localhost:9867/frame?tabId=TAB_ID \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"target":"#level-2"}'
-
-# Hop further into level 3 (selector is resolved relative to current scope)
-curl -X POST http://localhost:9867/frame?tabId=TAB_ID \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"target":"#level-3"}'
-
-# Click the level-3 button
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#deep-button"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
- -H "Authorization: Bearer benchmark-token"
-
-curl -X POST http://localhost:9867/frame?tabId=TAB_ID \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"target":"main"}'
-```
-**Pass if**: Scoped snapshot contains `DEEP_CLICKED=YES_LEVEL_3`.
-
-**Note**: Each `/frame` hop is relative to the current scope, so drilling
-into a nested frame requires one hop per level. Use `{"target":"main"}`
-to unwind to the top document in a single call.
-
----
-
-## Group 32: Dynamic iframe (inserted after load)
-
-### 32.1 Wait for a late-inserted iframe then interact with it
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/iframe-dynamic.html"}'
-
-# Wait for the marker that indicates the iframe was attached
-curl -X POST http://localhost:9867/tabs/TAB_ID/wait \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"text":"IFRAME_DYNAMIC_ATTACHED","timeout":5000}'
-
-# Scope into it and interact
-curl -X POST http://localhost:9867/frame?tabId=TAB_ID \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"target":"#late-frame"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#iframe-input","text":"Late World"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#iframe-submit"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
- -H "Authorization: Bearer benchmark-token"
-
-curl -X POST http://localhost:9867/frame?tabId=TAB_ID \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"target":"main"}'
-```
-**Pass if**: Scoped snapshot contains `IFRAME_INPUT_RECEIVED_LATE_WORLD`.
-
-**Note**: The iframe is added to the DOM ~1.2 s after load. Snapshotting
-immediately won't see it; `pinchtab wait --text` is the right primitive.
-
----
-
-## Group 33: srcdoc iframe (inline HTML, no src URL)
-
-### 33.1 Interact with an inline-content iframe
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/iframe-srcdoc.html"}'
-
-curl -X POST http://localhost:9867/frame?tabId=TAB_ID \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"target":"#srcdoc-frame"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#inline-input","text":"srcdoc"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#inline-submit"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
- -H "Authorization: Bearer benchmark-token"
-
-curl -X POST http://localhost:9867/frame?tabId=TAB_ID \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"target":"main"}'
-```
-**Pass if**: Scoped snapshot contains `INLINE_RECEIVED_SRCDOC`.
-
-**Note**: `srcdoc` iframes inherit the parent's origin (same-origin by
-default), so frame scope works the same way as `src`-based iframes. The
-returned `frameUrl` is `about:srcdoc`.
-
----
-
-## Group 34: Sandboxed iframe (allow-scripts allow-same-origin)
-
-### 34.1 Click a button inside a sandbox="allow-scripts allow-same-origin" iframe
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/iframe-sandbox.html"}'
-
-curl -X POST http://localhost:9867/frame?tabId=TAB_ID \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"target":"#sandboxed"}'
-
-curl -X POST http://localhost:9867/tabs/TAB_ID/action \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#sandbox-button"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
- -H "Authorization: Bearer benchmark-token"
-
-curl -X POST http://localhost:9867/frame?tabId=TAB_ID \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"target":"main"}'
-```
-**Pass if**: Scoped snapshot contains `SANDBOX_CLICKED=YES`.
-
-**Note**: `sandbox="allow-scripts allow-same-origin"` permits scripts and
-inherits the parent origin, so frame scope works. Sandboxes without
-`allow-same-origin` force a unique opaque origin and would behave like
-cross-origin — out of reach for the current `/frame` scope API.
-
----
-
-## Group 35: Long-form article (Medium/Substack style)
-
-### 35.1 Readability keeps the article body
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/article.html"}'
-
-# Default mode uses Readability; this is its sweet spot.
-curl "http://localhost:9867/tabs/TAB_ID/text" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Response contains `ARTICLE_PUBLISHED_2026_04_15` AND `ARTICLE_WORD_COUNT_MARKER_323` (both inside the main article body — Readability should retain them).
-
-### 35.2 `--full` picks up the surrounding chrome Readability drops
-```bash
-curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Response contains `ARTICLE_PUBLISHED_2026_04_15` AND `FOOTER_COPYRIGHT_MARKER`. Default Readability trims the footer; `--full` keeps it. Proves the two modes behave as documented.
-
----
-
-## Group 36: Search results page (SERP)
-
-### 36.1 Extract a specific result title by id
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/serp.html"}'
-
-# Use a scoped snapshot to extract just one card — faster than full text.
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?selector=%23r-3&format=compact&maxTokens=500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Snapshot contains `RESULT_3_TITLE` AND `RESULT_3_SNIPPET_MARKER`.
-
-### 36.2 Count all six result cards
-```bash
-curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Response contains `RESULT_1_TITLE`, `RESULT_2_TITLE`, `RESULT_3_TITLE`, `RESULT_4_TITLE`, `RESULT_5_TITLE`, `RESULT_6_TITLE` (all 6), AND `SERP_RESULT_COUNT_6`. A SERP is the canonical case where Readability's single-article assumption fails — `--full` is required to see the list.
-
----
-
-## Group 37: Q&A thread (Stack-Overflow style)
-
-### 37.1 Find the accepted answer
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/qa.html"}'
-
-# Use eval to find the accepted answer id — structured page data beats
-# text-parsing when the marker lives in an attribute.
-curl -X POST http://localhost:9867/tabs/TAB_ID/evaluate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"expression":"document.querySelector(\"[data-accepted=\\\"true\\\"]\").id"}'
-```
-**Pass if**: Response `.result` equals `"a-2"`. The accepted-answer marker is in a data attribute; this exercises `eval` as a structured alternative to text scraping.
-
-### 37.2 Extract the accepted answer body text
-```bash
-# Snapshot scoped to the accepted answer, with max-token budget
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?selector=%23a-2&format=compact&maxTokens=1500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Scoped snapshot contains `ANSWER_2_BODY_MARKER` AND `ACCEPTED_ANSWER_ID_A2`. Tests `snapshot?selector=` for scoped reads — a common pattern when a page has many independent sections.
-
----
-
-## Group 38: Pricing table
-
-### 38.1 Extract the Pro plan price
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/pricing.html"}'
-
-curl "http://localhost:9867/tabs/TAB_ID/snapshot?selector=%23plan-pro&format=compact&maxTokens=500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Snapshot contains `PLAN_PRO_PRICE_29` AND `PLAN_PRO_LIMIT_5000_requests per day`.
-
-### 38.2 Extract all three plan prices in one pass
-```bash
-curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Response contains `PLAN_FREE_PRICE_0`, `PLAN_PRO_PRICE_29`, AND `PLAN_ENTERPRISE_PRICE_CUSTOM`. Pricing tables are grid-heavy; Readability would pick one column. `--full` returns the whole surface.
-
----
-
-## Summary
-
-| Group | Tasks | Description |
-|-------|-------|-------------|
-| 0 | 8 | Setup & Diagnosis |
-| 1 | 6 | Reading & Extracting |
-| 2 | 3 | Search & Dynamic |
-| 3 | 2 | Form |
-| 4 | 3 | SPA |
-| 5 | 2 | Login |
-| 6 | 3 | E-commerce |
-| 7 | 2 | Content + Interaction |
-| 8 | 2 | Error Handling |
-| 9 | 2 | Export |
-| 10 | 2 | Modals |
-| 11 | 2 | Persistence |
-| 12 | 2 | Multi-page Nav |
-| 13 | 2 | Form Validation |
-| 14 | 2 | Dynamic Content |
-| 15 | 2 | Data Aggregation |
-| 16 | 2 | Hover & Tooltips |
-| 17 | 2 | Scrolling |
-| 18 | 1 | File Download |
-| 19 | 2 | iFrame |
-| 20 | 2 | Dialogs |
-| 21 | 2 | Async / awaitPromise |
-| 22 | 2 | Mouse Drag & Drop |
-| 23 | 1 | Async / Loading state |
-| 24 | 2 | Keyboard events |
-| 25 | 2 | Tab panels |
-| 26 | 2 | Accordion |
-| 27 | 2 | Contenteditable editor |
-| 28 | 2 | Range slider |
-| 29 | 2 | Pagination |
-| 30 | 2 | Custom dropdown menu |
-| 31 | 1 | Nested iframes (3 levels) |
-| 32 | 1 | Dynamic iframe |
-| 33 | 1 | srcdoc iframe |
-| 34 | 1 | Sandboxed iframe |
-| 35 | 2 | Long-form article (text extraction) |
-| 36 | 2 | Search results page (SERP) |
-| 37 | 2 | Q&A thread (Stack-Overflow style) |
-| 38 | 2 | Pricing table |
-
-**Total: 85 tasks**
-
-## Verification Strings
-
-| Page | String |
-|------|--------|
-| Home | `VERIFY_HOME_LOADED_12345` |
-| Articles | `VERIFY_ARTICLES_PAGE_67890` |
-| Search | `VERIFY_SEARCH_PAGE_11111` |
-| Form | `VERIFY_FORM_PAGE_22222` |
-| Dashboard | `VERIFY_DASHBOARD_PAGE_33333` |
-| Shop | `VERIFY_SHOP_PAGE_44444` |
-| Wiki Index | `VERIFY_WIKI_INDEX_55555` |
-| Login | `VERIFY_LOGIN_PAGE_77777` |
-| Go Article | `VERIFY_WIKI_GO_LANG_88888` |
-| SPA | `VERIFY_SPA_PAGE_99999` |
-| Python Article | `VERIFY_WIKI_PYTHON_LANG` |
-| Rust Article | `VERIFY_WIKI_RUST_LANG` |
-| Async | `VERIFY_ASYNC_PAGE_77777` |
-| Drag | `VERIFY_DRAG_PAGE_33333` |
-| Loading | `VERIFY_LOADING_PAGE_11111` |
-| Keyboard | `VERIFY_KEYBOARD_PAGE_22222` |
-| Tabs | `VERIFY_TABS_PAGE_55555` |
-| Accordion | `VERIFY_ACCORDION_PAGE_66666` |
-| Editor | `VERIFY_EDITOR_PAGE_44444` |
-| Range | `VERIFY_RANGE_PAGE_99999` |
-| Pagination | `VERIFY_PAGINATION_PAGE_88888` |
-| Dropdown | `VERIFY_DROPDOWN_PAGE_77777` |
-| Iframe Nested (outer) | `VERIFY_IFRAME_NESTED_OUTER` |
-| Iframe Nested (L2) | `VERIFY_IFRAME_NESTED_L2` |
-| Iframe Nested (L3) | `VERIFY_IFRAME_NESTED_L3` |
-| Iframe Dynamic | `VERIFY_IFRAME_DYNAMIC_OUTER` |
-| Iframe Srcdoc | `VERIFY_IFRAME_SRCDOC_OUTER` |
-| Iframe Sandbox | `VERIFY_IFRAME_SANDBOX_OUTER` |
-| Article | `VERIFY_ARTICLE_PAGE_41414` |
-| SERP | `SERP_PAGE_MARKER_88044` |
-| Q&A | `VERIFY_QA_PAGE_72000` |
-| Pricing | `VERIFY_PRICING_PAGE_30303` |
diff --git a/tests/benchmark/CRON_TASK.md b/tests/benchmark/CRON_TASK.md
deleted file mode 100644
index 75e31857..00000000
--- a/tests/benchmark/CRON_TASK.md
+++ /dev/null
@@ -1,142 +0,0 @@
-# PinchTab Optimization Cron Task
-
-**Goal: improve the agent lane against a known-good benchmark environment.**
-
-Baseline is not part of the recurring agent task. It is a direct shell
-verification that should be run after infrastructure, fixture, or benchmark
-changes to confirm the environment still behaves as expected.
-
----
-
-## Baseline Gate
-
-Run baseline manually when any of these change:
-
-- PinchTab server or CLI behavior
-- benchmark fixtures
-- `tests/benchmark/scripts/baseline.sh`
-- benchmark Docker setup
-
-Command:
-
-```bash
-cd ~/dev/pinchtab/tests/benchmark
-./scripts/run-optimization.sh
-./scripts/baseline.sh
-./scripts/finalize-report.sh "$(cat ./results/current_baseline_report.txt)"
-```
-
-Only continue with optimization work if baseline is green.
-
----
-
-## Recurring Optimization Task
-
-### Step 1 — Setup
-```bash
-cd ~/dev/pinchtab/tests/benchmark
-git checkout feat/benchmark && git pull --rebase origin feat/benchmark
-./scripts/run-optimization.sh
-# Note the TIMESTAMP from output
-```
-
-### Step 2 — Run Agent Benchmark
-Execute `AGENT_TASKS.md` using `skills/pinchtab/SKILL.md` as the guide.
-
-- Use the benchmark wrapper `./scripts/pt`
-- Record results with `record-step.sh --type agent`
-- Verify answers with `verify-step.sh`
-- Log every command executed
-- Do not use baseline as an execution guide
-
-### Step 3 — Gap Analysis
-Compare agent results against the latest green baseline report and classify failures:
-
-| Failure Type | Cause | Fix |
-|---|---|---|
-| **Wrong endpoint** | Agent used `/text` when should use `/snapshot` | Improve `SKILL.md` |
-| **Wrong selector** | Agent guessed selector incorrectly | Improve `SKILL.md` or fixture |
-| **Missing step** | Agent skipped a required action | Clarify `AGENT_TASKS.md` |
-| **Wrong URL** | Agent used wrong fixture path | Clarify `AGENT_TASKS.md` |
-| **API bug** | Endpoint behaves unexpectedly | Fix PinchTab code |
-| **Test ambiguity** | Verification string hard to find | Fix fixture or `scripts/baseline.sh` |
-
-### Step 4 — Make Exactly 1 Change
-
-Priority order:
-1. **API Bug** → Fix PinchTab Go code, commit as `fix: <description>`
-2. **Skill Gap** → Improve `SKILL.md`, commit as `docs(skill): <description>`
-3. **Test Ambiguity** → Fix fixture HTML or `scripts/baseline.sh`, commit as `test: <description>`
-4. **Agent Task Clarity** → Improve `AGENT_TASKS.md`, commit as `test(agent): <description>`
-5. **No Gaps Found** → Add new benchmark coverage, with matching baseline + agent cases
-
-**If agent is already at 100%**: add harder cases.
-
-### Step 5 — Verify the Fix Makes Sense
-Before committing, ask:
-
-- will this change help the agent pick the right tool next time?
-- if this is a skill/docs fix, does it directly address the observed failure?
-- if this is a benchmark fix, is the expected signal reachable via the real tool flow?
-
-### Step 6 — Commit & Push
-```bash
-cd ~/dev/pinchtab
-git add -A
-git commit -m "<type>: <clear description of what changed and why>"
-git push origin feat/benchmark
-```
-
-### Step 7 — Log the Run
-Append to `results/optimization_log.md`:
-
-```markdown
-## Run #N — YYYY-MM-DD HH:MM
-
-**Results:**
-- Baseline reference: X/Y (Z%)
-- Agent: X/Y (Z%)
-- Gap: N steps
-
-**Agent Failures:**
-- Step X.Y: [failure type] — [what went wrong]
-- Step X.Y: [failure type] — [what went wrong]
-
-**Root Cause:**
-[One clear sentence explaining the pattern]
-
-**Change Made:**
-- Type: api|skill|test|agent-task
-- Description: [What changed]
-- Expected impact: [Why this should close the gap]
-- Commit: [hash]
-
-**Next Focus:**
-[What to watch in the next run]
-```
-
-### Step 8 — Report to User
-Send a concise update:
-
-```text
-Run #N complete
-Baseline reference: X% | Agent: Y% | Gap: N steps
-Failure: [brief description]
-Fix: [what changed] ([commit])
-```
-
----
-
-## Success Criteria
-
-- **Short term**: Agent pass rate ≥ 95%
-- **Long term**: Agent matches the latest green baseline on all existing tests
-- **Ongoing**: When the gap closes, increase test complexity
-
-## What NOT to do
-
-- Do not run baseline as an agent task
-- Do not make multiple changes per run
-- Do not skip root-cause analysis
-- Do not weaken tests just to improve the score
-- Do not commit a change that does not directly address an observed failure
diff --git a/tests/benchmark/OPTIMIZATION_LOOP.md b/tests/benchmark/OPTIMIZATION_LOOP.md
deleted file mode 100644
index d8a1175e..00000000
--- a/tests/benchmark/OPTIMIZATION_LOOP.md
+++ /dev/null
@@ -1,166 +0,0 @@
-# PinchTab Auto-Optimization Loop
-
-Automated improvement cycle for PinchTab benchmarks.
-
-## Pre-requisite (Human, once)
-
-```bash
-cd ~/dev/pinchtab/tests/benchmark
-docker compose up -d --build
-# Wait for healthy:
-curl -sf -H "Authorization: Bearer benchmark-token" http://localhost:9867/health
-```
-
-## Loop Steps
-
-```
-┌─────────────────────────────────────────────────────────────┐
-│ 1. RUN AGENT BENCHMARK [Sub-agent] │
-│ - Runs every iteration │
-│ - Load AGENT_TASKS.md + skills/pinchtab/SKILL.md │
-│ - Execute tasks using only skill docs │
-│ - Record via: record-step.sh --type agent │
-│ - Output: results/agent_benchmark_<timestamp>.json │
-└─────────────────────────────────────────────────────────────┘
- ↓
-┌─────────────────────────────────────────────────────────────┐
-│ 2. CHECK FOR REGRESSION [Main agent] │
-│ - Read both JSON reports for pass/fail counts │
-│ - Compare against results/best_score.txt │
-│ - If worse: flag regression, prioritize fix/revert │
-│ - If equal or better: update best_score.txt │
-└─────────────────────────────────────────────────────────────┘
- ↓
-┌─────────────────────────────────────────────────────────────┐
-│ 3. ANALYZE DIFFERENCES [Main agent] │
-│ - Diff baseline vs agent results side by side │
-│ - Identify which steps agent failed but baseline passed │
-│ - Find patterns (wrong endpoint, selectors, etc.) │
-└─────────────────────────────────────────────────────────────┘
- ↓
-┌─────────────────────────────────────────────────────────────┐
-│ 4. PROPOSE 1 IMPROVEMENT [Main agent → Human]│
-│ Priority order: │
-│ a) Fix PinchTab CLI/REST bug if found │
-│ b) Improve skill documentation if agent confused │
-│ c) Add verification to existing test case │
-│ d) Add new test case for uncovered scenario │
-│ Present proposal to human for approval. │
-└─────────────────────────────────────────────────────────────┘
- ↓
-┌─────────────────────────────────────────────────────────────┐
-│ 5. IMPLEMENT (no commit) [Main agent] │
-│ - Make the single change │
-│ - Leave changes uncommitted for manual review │
-└─────────────────────────────────────────────────────────────┘
- ↓
-┌─────────────────────────────────────────────────────────────┐
-│ 6. LOG RUN [Main agent] │
-│ - Append to results/optimization_log.md │
-│ - Record: timestamp, pass rates, regression check, │
-│ change made, next focus │
-└─────────────────────────────────────────────────────────────┘
-```
-
-## Roles
-
-| Step | Who | What |
-|------|-----|------|
-| Pre-req | **Human** | Start Docker, verify health |
-| 1 Agent | **Sub-agent** | Execute agent tasks from skill docs, record results |
-| 2 Regression | **Main agent** | Compare against best_score.txt |
-| 3 Analyze | **Main agent** | Diff baseline vs agent, find patterns |
-| 4 Propose | **Main agent → Human** | Suggest 1 fix, get approval |
-| 5 Implement | **Main agent** | Make the change, leave uncommitted |
-| 6 Log | **Main agent** | Append to optimization_log.md |
-
-## Baseline is a Shell Verification, Not an Agent Run
-
-The baseline suite (`scripts/baseline.sh`) validates that the benchmark infrastructure itself works — fixtures serve correctly, PinchTab APIs behave as expected, and the test conditions are reachable. It's deterministic (no LLM involved), so once it's at 100% it stays at 100% unless something changes underneath it.
-
-Run baseline directly as a shell command:
-
-```bash
-./scripts/run-optimization.sh
-./scripts/baseline.sh
-```
-
-**The optimization loop does NOT treat baseline as an agent task and does NOT re-run it every iteration.** Instead, run the baseline manually when any of these change:
-
-- PinchTab server code (anything in `internal/` that affects the HTTP API)
-- Fixtures (anything in `tests/benchmark/fixtures/`)
-- `scripts/baseline.sh` itself (adding/modifying tests)
-- `docker-compose.yml` or the Docker image
-
-**Workflow for adding a new test case:**
-
-1. Write the executable baseline case in `scripts/baseline.sh`.
- If you keep `BASELINE_TASKS.md` around as reference material, update it second.
-2. Write the matching task in `AGENT_TASKS.md` (natural language)
-3. Write the summary row in `TEST_CASES.md`
-4. Add any new fixture files needed
-5. **Run the baseline shell verification** — it must reach 100% before the case is considered valid
-6. If baseline fails: fix the test/fixture/PinchTab until it passes
-7. Commit the new case — now the agent loop can use it
-
-Once baseline is green, the optimization loop only runs the agent suite.
-
-## Regression Detection
-
-Each run compares against `results/best_score.txt`, which tracks the highest pass rates achieved:
-
-```
-baseline=62/68
-agent=33/39
-```
-
-- **If current run is worse**: Regression detected. The improvement priority shifts to fixing or reverting the last change before proposing anything new.
-- **If equal or better**: Update `best_score.txt` with the new high-water mark.
-
-## Improvement Priority
-
-1. **API/CLI Bug** — If a curl command returns unexpected error, investigate PinchTab code
-2. **Skill Gap** — If agent uses wrong endpoint/approach, improve SKILL.md documentation
-3. **Benchmark Gap** — If tests don't verify important behavior, add verification
-4. **Coverage Gap** — If common scenarios aren't tested, add test cases
-
-## Log Format
-
-Each run appends to `results/optimization_log.md`:
-
-```markdown
-## Run #N — YYYY-MM-DD HH:MM
-
-**Results:**
-- Baseline: X/Y (Z%)
-- Agent: X/Y (Z%)
-
-**Regression Check:**
-- Best: baseline X/Y, agent X/Y
-- This run: [better | same | REGRESSION]
-
-**Analysis:**
-- [What differed between runs]
-- [Root cause identified]
-
-**Change Made:**
-- [Type: api|skill|benchmark]
-- [Description]
-- [Status: uncommitted — pending manual review]
-
-**Next Focus:**
-- [What to look at next run]
-```
-
-## Files
-
-- `scripts/baseline.sh` — Executable baseline lane source of truth
-- `scripts/run-optimization.sh` — Main loop script (initializes reports)
-- `scripts/record-step.sh` — Appends a step result to the current report
-- `results/optimization_log.md` — Run history (includes per-run metrics table)
-- `results/best_score.txt` — High-water mark for regression detection
-- Changes left uncommitted for manual review and commit
-
-Agent runs use the benchmark wrapper `./scripts/pt`, which forwards to the
-PinchTab CLI inside Docker while preserving `PINCHTAB_TAB` targeting. See
-`AGENT_TASKS.md` "Recommended setup" for the exact pattern.
diff --git a/tests/benchmark/README.md b/tests/benchmark/README.md
index b54fadf8..862dd2d1 100644
--- a/tests/benchmark/README.md
+++ b/tests/benchmark/README.md
@@ -17,24 +17,20 @@ cd tests/benchmark
# agent-browser lane
./scripts/run-agent-browser-benchmark.sh
-# Then read AGENT_BROWSER_INSTRUCTIONS.md and run tasks from AGENT_BROWSER_TASKS.md with ./scripts/ab
+# Then read setup-agent-browser.md and benchmark-run/index.md and run tasks with ./scripts/ab
./scripts/finalize-report.sh
-# optional provider API runner
-OPENAI_API_KEY=... node --experimental-strip-types ./scripts/run-api-benchmark.ts --lane agent --finalize
-OPENAI_API_KEY=... node --experimental-strip-types ./scripts/run-api-benchmark.ts --lane agent-browser --finalize
-ANTHROPIC_API_KEY=... node --experimental-strip-types ./scripts/run-api-benchmark.ts --lane agent --model claude-haiku-4-5-20251001 --finalize
-ANTHROPIC_API_KEY=... node --experimental-strip-types ./scripts/run-api-benchmark.ts --lane agent-browser --model claude-haiku-4-5-20251001 --finalize
-
-# repo entrypoints
+# repo entrypoints (recommended)
./dev benchmark baseline
-OPENAI_API_KEY=... ./dev benchmark agent
-OPENAI_API_KEY=... ./dev benchmark agent-browser
-ANTHROPIC_API_KEY=... ./dev benchmark agent
-ANTHROPIC_API_KEY=... ./dev benchmark agent-browser
-ANTHROPIC_API_KEY=... ./dev benchmark agent --provider anthropic --turn-delay-ms 3000
-ANTHROPIC_API_KEY=... ./dev benchmark agent --provider anthropic --profile common10
-ANTHROPIC_API_KEY=... ./dev benchmark agent-browser --provider anthropic --groups 0,1,2,3
+./dev benchmark pinchtab --dry-run
+OPENAI_API_KEY=... ./dev benchmark pinchtab
+ANTHROPIC_API_KEY=... ./dev benchmark pinchtab --profile common10
+ANTHROPIC_API_KEY=... ./dev benchmark pinchtab --max-input-tokens 50000
+ANTHROPIC_API_KEY=... ./dev benchmark agent-browser --groups 0,1,2,3
+
+# direct Go runner
+ANTHROPIC_API_KEY=... go run ./cmd/api-runner --lane pinchtab --finalize
+ANTHROPIC_API_KEY=... go run ./cmd/api-runner --lane agent-browser --finalize
```
## MANDATORY: Docker Environment
@@ -54,9 +50,10 @@ If Docker build fails or is skipped, the benchmark is **INVALID**.
| File | Purpose |
|------|---------|
| `../../skills/pinchtab/SKILL.md` | PinchTab skill (same as shipped product) |
-| `AGENT_BROWSER_INSTRUCTIONS.md` | Benchmark-local instructions for the `agent-browser` lane |
-| `BASELINE_TASKS.md` | Legacy descriptive baseline spec; not executed by the harness |
-| `AGENT_BROWSER_TASKS.md` | Equivalent task lane for `agent-browser` |
+| `setup-pinchtab.md` | PinchTab lane setup and wrapper guidance |
+| `setup-agent-browser.md` | agent-browser lane setup and wrapper guidance |
+| `benchmark-run/index.md` | Shared benchmark task index for the agent-driven lanes |
+| `benchmark-run/group-XX.md` | One markdown file per benchmark group |
| `scripts/baseline.sh` | Executable baseline lane source of truth |
| `scripts/run-optimization.sh` | Initialize PinchTab benchmark reports |
| `scripts/run-agent-browser-benchmark.sh` | Start fixtures + `agent-browser` and initialize a fresh report |
@@ -139,22 +136,22 @@ The benchmark runs PinchTab in Docker with:
Every step should record the observed answer/result:
```bash
-./scripts/record-step.sh --type agent <group> <step> <answer|fail|skip> "answer" "notes"
+./scripts/record-step.sh --type pinchtab <group> <step> <answer|fail|skip> "answer" "notes"
```
Example:
```bash
-./scripts/record-step.sh --type agent 1 1 answer "Navigation completed in 1.2s" "observed output"
-./scripts/record-step.sh --type agent 2 3 fail "Element not found"
+./scripts/record-step.sh --type pinchtab 1 1 answer "Navigation completed in 1.2s" "observed output"
+./scripts/record-step.sh --type pinchtab 2 3 fail "Element not found"
```
Deferred-verification example:
```bash
-./scripts/record-step.sh --type agent 1 1 answer \
+./scripts/record-step.sh --type pinchtab 1 1 answer \
"Found categories Programming Languages: 12, Databases: 8" \
"raw answer"
-./scripts/verify-step.sh --type agent 1 1 pass \
+./scripts/verify-step.sh --type pinchtab 1 1 pass \
"Answer satisfies the benchmark oracle"
```
diff --git a/tests/benchmark/benchmark-run/group-00.md b/tests/benchmark/benchmark-run/group-00.md
new file mode 100644
index 00000000..5e5fd34a
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-00.md
@@ -0,0 +1,44 @@
+# Group 0: Setup & Diagnosis
+
+### 0.1 Server reachable
+Check that the PinchTab server is healthy.
+
+**Verify**: Server responds with `status: ok`.
+
+### 0.2 Auth is required
+Make a request to the server with a **wrong** token (`PINCHTAB_TOKEN=wrong-token ./scripts/pt health`) and confirm it is rejected. The `pt` wrapper always injects the benchmark token by default, so you must explicitly override it to test auth rejection.
+
+**Verify**: Response is HTTP 401 or contains `unauthorized`.
+
+### 0.3 Auth works with token
+Repeat the same request WITH the bearer token and confirm it succeeds.
+
+**Verify**: Response is HTTP 200.
+
+### 0.4 Instance available
+Confirm at least one Chrome instance is running. If none exist, start one.
+
+**Verify**: Health response shows `defaultInstance.status == "running"` (or after starting one, the new instance is running).
+
+### 0.5 List existing tabs
+Get the current list of open tabs.
+
+**Verify**: A list (possibly empty) is returned without error.
+
+### 0.6 Clean stale tabs
+If any tabs from previous runs are open, close them so the benchmark starts from a known state.
+
+**Verify**: After cleanup, the tab list is empty (or only contains a single about:blank tab).
+
+### 0.7 Network reach to target
+Navigate to `http://fixtures/` and confirm the fixtures server is reachable from PinchTab.
+
+**Verify**: Navigate returns successfully and the page contains benchmark content.
+
+### 0.8 Capture initial tab ID
+Save the tab ID returned by the navigate in 0.7. Use this tab ID for all subsequent tasks to avoid creating new tabs.
+
+**Verify**: A tab ID was captured and matches what `GET /tabs` reports.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-01.md b/tests/benchmark/benchmark-run/group-01.md
new file mode 100644
index 00000000..9ae2ee54
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-01.md
@@ -0,0 +1,34 @@
+# Group 1: Reading & Extracting Real Content
+
+### 1.1 Get the full list of categories from the wiki index
+Navigate to `http://fixtures/wiki.html` and extract all category names with their article counts.
+
+**Verify**: Can name at least 2 categories and their article counts (e.g. "Programming Languages: 12 articles").
+
+### 1.2 Navigate by clicking a link
+From the wiki index, click the "Go (programming language)" link to navigate to the Go article.
+
+**Verify**: You are now on the Go article page (not the wiki index).
+
+### 1.3 Extract structured data from a table
+On the Go article, read the infobox table and answer: Who designed Go, and what year did it first appear?
+
+**Verify**: Answer contains "Robert Griesemer" (or "Rob Pike" or "Ken Thompson") and "2009".
+
+### 1.4 Count list items
+On the Go article, count how many key features are listed.
+
+**Verify**: Answer is 6 (verify against `FEATURE_COUNT_6`).
+
+### 1.5 Read all article headlines from articles page
+Navigate to `http://fixtures/articles.html` and list all article titles.
+
+**Verify**: Found at least 3 articles including "The Future of Artificial Intelligence".
+
+### 1.6 Read dashboard metrics
+Navigate to `http://fixtures/dashboard.html` and extract: Total Users, Revenue, and Conversion Rate.
+
+**Verify**: Found `24,582` users AND `$1,284,930` revenue.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-02.md b/tests/benchmark/benchmark-run/group-02.md
new file mode 100644
index 00000000..78be52a2
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-02.md
@@ -0,0 +1,19 @@
+# Group 2: Search & Dynamic Interaction
+
+### 2.1 Use wiki search to find a page
+On `http://fixtures/wiki.html`, search for "golang" using the search form. Do not navigate directly — use the search input.
+
+**Verify**: Ended up on the Go article page after search.
+
+### 2.2 Search with no results
+On `http://fixtures/search.html`, search for something with no results (use "xyznonexistent").
+
+**Verify**: Page handled it gracefully (no crash, some response rendered).
+
+### 2.3 Search for AI content
+On `http://fixtures/search.html`, search for "artificial intelligence" and verify a result appeared.
+
+**Verify**: Result contains "The Future of Artificial Intelligence".
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-03.md b/tests/benchmark/benchmark-run/group-03.md
new file mode 100644
index 00000000..34de80e3
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-03.md
@@ -0,0 +1,23 @@
+# Group 3: Complex Form
+
+### 3.1 Fill and submit a complete form
+Navigate to `http://fixtures/form.html`. Complete the entire form:
+- Full name: "Agent Test User"
+- Email: "[email protected]"
+- Phone: "+44 20 9999 0000"
+- Country: United Kingdom
+- Subject: Technical Support
+- Message: "Testing PinchTab form automation"
+- Check newsletter
+- Set priority to High
+- Submit
+
+**Verify**: Form submitted successfully. Confirmation shows name "AGENT_TEST_USER".
+
+### 3.2 Reset and refill
+After submitting, if the form is still accessible (or navigate back), verify you can identify the reset button.
+
+**Verify**: Reset/back button or form element found in snapshot.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-04.md b/tests/benchmark/benchmark-run/group-04.md
new file mode 100644
index 00000000..bf5978c4
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-04.md
@@ -0,0 +1,19 @@
+# Group 4: SPA State Management
+
+### 4.1 Read initial app state
+Navigate to `http://fixtures/spa.html?reset=1` (the `?reset=1` query param clears any previous localStorage state so the SPA starts with its default 3 tasks). Read the current task list — how many tasks exist, how many are active vs done?
+
+**Verify**: Found 3 total, 2 active, 1 done (verify `TASK_STATS_TOTAL_3_ACTIVE_2_DONE_1`).
+
+### 4.2 Add a new high-priority task
+Add a task called "Automate deployment" with high priority.
+
+**Verify**: Task appeared in the list (`TASK_ADDED_AUTOMATE_DEPLOYMENT_PRIORITY_HIGH`).
+
+### 4.3 Delete a task
+Delete the task titled "Write benchmark tests".
+
+**Verify**: Task count changed (went from 4 to 3).
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-05.md b/tests/benchmark/benchmark-run/group-05.md
new file mode 100644
index 00000000..ba3d630a
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-05.md
@@ -0,0 +1,14 @@
+# Group 5: Login Flow
+
+### 5.1 Attempt login with wrong credentials
+Navigate to `http://fixtures/login.html`. Try to log in with username "admin" and password "wrong".
+
+**Verify**: Error message appeared (`INVALID_CREDENTIALS_ERROR`).
+
+### 5.2 Login successfully
+Clear the form and log in with username "benchmark" / password "test456".
+
+**Verify**: Dashboard appeared after login (`VERIFY_LOGIN_SUCCESS_DASHBOARD`).
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-06.md b/tests/benchmark/benchmark-run/group-06.md
new file mode 100644
index 00000000..37c049f7
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-06.md
@@ -0,0 +1,19 @@
+# Group 6: Multi-Step E-commerce
+
+### 6.1 Research products before buying
+Navigate to `http://fixtures/ecommerce.html`. List all available products with their prices. Which product is out of stock?
+
+**Verify**: Found Wireless Headphones ($149.99), Smart Watch Pro ($299.99), Portable Charger ($49.99). Mechanical Keyboard is out of stock.
+
+### 6.2 Add two items and verify total
+Add Wireless Headphones and Smart Watch Pro to cart. What is the total?
+
+**Verify**: Cart total is $449.98 (149.99 + 299.99).
+
+### 6.3 Complete checkout
+Click checkout to complete the order.
+
+**Verify**: Order confirmation shows (`VERIFY_CHECKOUT_SUCCESS_ORDER`).
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-07.md b/tests/benchmark/benchmark-run/group-07.md
new file mode 100644
index 00000000..41c1838f
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-07.md
@@ -0,0 +1,14 @@
+# Group 7: Content + Interaction Combined
+
+### 7.1 Read and comment on Go article
+Navigate to `http://fixtures/wiki-go.html`. Read the article, then post a comment with rating 5 stars and text "Excellent reference".
+
+**Verify**: Comment posted (`COMMENT_POSTED_RATING_5_TEXT_RECEIVED`).
+
+### 7.2 Cross-page research task
+Navigate to wiki index, find which category has the most articles, then navigate to one of its listed items.
+
+**Verify**: Successfully navigated to at least one article page after reading the category counts.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-08.md b/tests/benchmark/benchmark-run/group-08.md
new file mode 100644
index 00000000..ddfc6817
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-08.md
@@ -0,0 +1,14 @@
+# Group 8: Error Handling
+
+### 8.1 Handle 404 gracefully
+Try to navigate to a page that doesn't exist: `http://fixtures/missing-page-abc.html`.
+
+**Verify**: Got a response (404 or error), no crash, server still responsive after.
+
+### 8.2 Handle missing element gracefully
+On any page, try to click an element with ID `#fake-button-that-does-not-exist`.
+
+**Verify**: Got a clear error message, not a crash or hang.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-09.md b/tests/benchmark/benchmark-run/group-09.md
new file mode 100644
index 00000000..5f2ccedf
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-09.md
@@ -0,0 +1,14 @@
+# Group 9: Export
+
+### 9.1 Screenshot a complex page
+Navigate to `http://fixtures/dashboard.html` and take a screenshot.
+
+**Verify**: Screenshot generated (file saved or base64 returned).
+
+### 9.2 Export a page as PDF
+Export the dashboard as a PDF.
+
+**Verify**: PDF generated.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-10.md b/tests/benchmark/benchmark-run/group-10.md
new file mode 100644
index 00000000..a948108e
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-10.md
@@ -0,0 +1,14 @@
+# Group 10: Nested Interactions & Modal Dialogs
+
+### 10.1 Open and interact with modal on dashboard
+Navigate to `http://fixtures/dashboard.html`. Find and click the Settings button (selector: `#settings-btn`) to open the modal dialog.
+
+**Verify**: Modal appeared — snapshot contains "Dashboard Settings".
+
+### 10.2 Modify settings and close modal
+In the modal, select "Dark" from the theme dropdown (`#theme-select`), then click the Save button (`#modal-save`). After the modal closes, check the page content.
+
+**Verify**: Page contains `THEME_DARK_APPLIED`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-11.md b/tests/benchmark/benchmark-run/group-11.md
new file mode 100644
index 00000000..e4b85aff
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-11.md
@@ -0,0 +1,14 @@
+# Group 11: State Persistence & Page Reload
+
+### 11.1 Add an item and verify after page reload
+Navigate to `http://fixtures/spa.html?reset=1` (starts with clean state). Add a task titled exactly "Persistent Task Test". Then navigate away to `http://fixtures/` and back to `http://fixtures/spa.html` (WITHOUT the reset param) to verify localStorage persistence.
+
+**Verify**: After reload, the task still appears in the list (`TASK_PERSISTENT_TEST_FOUND_AFTER_RELOAD`).
+
+### 11.2 Logout and log back in
+From the logged-in dashboard, click Sign Out to log out. Then log in again with username "benchmark" / password "test456".
+
+**Verify**: Successfully logged back in and dashboard shows `SESSION_RENEWED`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-12.md b/tests/benchmark/benchmark-run/group-12.md
new file mode 100644
index 00000000..c17fb7bf
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-12.md
@@ -0,0 +1,14 @@
+# Group 12: Multi-Page Navigation & Back Button
+
+### 12.1 Navigate through multiple pages and return
+Starting from `http://fixtures/`, navigate to wiki → Go article → back to wiki → back to home.
+
+**Verify**: Successfully returned to home page (title contains "Benchmark" or "Home").
+
+### 12.2 Compare data across pages
+Navigate to wiki.html, note the total article count from categories. Navigate to articles.html, count articles there. Compare totals.
+
+**Verify**: Can report totals from both pages and explain difference (`COMPARISON_DATA_FOUND`).
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-13.md b/tests/benchmark/benchmark-run/group-13.md
new file mode 100644
index 00000000..310a7fe6
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-13.md
@@ -0,0 +1,14 @@
+# Group 13: Form State & Multi-Step Submission
+
+### 13.1 Submit form without email
+Navigate to `http://fixtures/form.html`. Fill only the name field ("Validator Test"), leave email blank, click Submit. The browser's native required-field validation will prevent submission.
+
+**Verify**: Submission blocked (form stays open, no success message shown).
+
+### 13.2 Submit form without optional phone field
+Fill the form with: name "No Phone User", email "[email protected]", country "de", subject "feedback". Leave the phone field empty. Submit.
+
+**Verify**: Submission succeeded and page shows `OPTIONAL_FIELD_SKIPPED_SUCCESS`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-14.md b/tests/benchmark/benchmark-run/group-14.md
new file mode 100644
index 00000000..865d6b3f
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-14.md
@@ -0,0 +1,14 @@
+# Group 14: Dynamic Content Loading
+
+### 14.1 Load more products
+Navigate to `http://fixtures/ecommerce.html`. Find and click the "Load More Products" button to reveal additional products.
+
+**Verify**: Additional products appeared (`ADDITIONAL_PRODUCTS_LOADED`).
+
+### 14.2 Add a lazy-loaded product to cart
+After loading more products, add product #5 (USB-C Cable) to the cart.
+
+**Verify**: Cart shows the lazy-loaded item (`CART_UPDATED_WITH_LAZY_PRODUCT`).
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-15.md b/tests/benchmark/benchmark-run/group-15.md
new file mode 100644
index 00000000..adcc745a
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-15.md
@@ -0,0 +1,19 @@
+# Group 15: Complex Data Extraction & Aggregation
+
+### 15.1 Extract and sum financial data
+Navigate to `http://fixtures/dashboard.html`. Extract revenue and profit values, calculate profit margin.
+
+**Verify**: Correctly calculated: profit_margin = (profit / revenue) * 100 (`PROFIT_MARGIN_CALCULATED`).
+
+### 15.2 Build comparison table from multiple sources
+Visit these 3 pages and compare their feature counts and key features:
+- `http://fixtures/wiki-go.html` (Go: 6 features)
+- `http://fixtures/wiki-python.html` (Python: 7 features)
+- `http://fixtures/wiki-rust.html` (Rust: 5 features)
+
+Report which language has the most features and name 1 feature unique to each.
+
+**Verify**: Response is factually correct AND wiki-python.html contains `COMPARISON_TABLE_BUILT`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-16.md b/tests/benchmark/benchmark-run/group-16.md
new file mode 100644
index 00000000..73e24f19
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-16.md
@@ -0,0 +1,14 @@
+# Group 16: Hover & Tooltips
+
+### 16.1 Hover reveals hidden content
+Navigate to `http://fixtures/hovers.html`. Hover over the first avatar to reveal its hidden info.
+
+**Verify**: After hovering, snapshot contains `HOVER_REVEALED_USER_1`.
+
+### 16.2 Hover swap
+Hover over the second avatar.
+
+**Verify**: Snapshot contains `HOVER_REVEALED_USER_2`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-17.md b/tests/benchmark/benchmark-run/group-17.md
new file mode 100644
index 00000000..7ca19c86
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-17.md
@@ -0,0 +1,14 @@
+# Group 17: Scrolling
+
+### 17.1 Scroll by pixels
+Navigate to `http://fixtures/scroll.html`. Scroll down approximately 1500 pixels.
+
+**Verify**: Snapshot or text contains `SCROLL_MIDDLE_MARKER` (the mid-page marker is now in view).
+
+### 17.2 Scroll to footer
+Continue scrolling until the footer is visible (or scroll directly to the footer element).
+
+**Verify**: Snapshot contains `SCROLL_REACHED_FOOTER`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-18.md b/tests/benchmark/benchmark-run/group-18.md
new file mode 100644
index 00000000..0744fd1c
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-18.md
@@ -0,0 +1,9 @@
+# Group 18: File Download
+
+### 18.1 Download a file
+Download `http://fixtures/download-sample.txt` to a local file.
+
+**Verify**: The downloaded file exists and its content includes `DOWNLOAD_FILE_CONTENT_VERIFIED`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-19.md b/tests/benchmark/benchmark-run/group-19.md
new file mode 100644
index 00000000..bf8517a0
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-19.md
@@ -0,0 +1,14 @@
+# Group 19: iFrame
+
+### 19.1 Read iframe content
+Navigate to `http://fixtures/iframe.html` and extract content from inside the embedded same-origin iframe.
+
+**Verify**: The iframe's inner content includes `IFRAME_INNER_CONTENT_LOADED`.
+
+### 19.2 Type into iframe input (native frame scope)
+Use `pinchtab frame '#content-frame'` to scope into the iframe, then `fill` the input with "Hello World" and `click` the Save button. Verify via a scoped `snap` — `text --full` doesn't pierce iframes. Reset with `pinchtab frame main` afterwards.
+
+**Verify**: Scoped snapshot contains `IFRAME_INPUT_RECEIVED_HELLO_WORLD`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-20.md b/tests/benchmark/benchmark-run/group-20.md
new file mode 100644
index 00000000..1a3ba53d
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-20.md
@@ -0,0 +1,14 @@
+# Group 20: Dialogs
+
+### 20.1 Accept alert
+Navigate to `http://fixtures/alerts.html`. Click the "Click for Alert" button and dismiss the alert.
+
+**Verify**: The page result contains `DIALOG_ALERT_DISMISSED`.
+
+### 20.2 Cancel confirm
+Click the "Click for Confirm" button and cancel the confirm dialog.
+
+**Verify**: The page result contains `DIALOG_CONFIRM_CANCELLED`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-21.md b/tests/benchmark/benchmark-run/group-21.md
new file mode 100644
index 00000000..2ed0f123
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-21.md
@@ -0,0 +1,14 @@
+# Group 21: Async / awaitPromise
+
+### 21.1 Await a promise-returning function
+Navigate to `http://fixtures/async.html`. The page exposes `window.fetchPayload()`, which returns a `Promise` that resolves after a short delay. Use `eval` to call it and retrieve the **resolved** value, not a Promise wrapper.
+
+**Verify**: The resolved value contains `ASYNC_PAYLOAD_READY_42`.
+
+### 21.2 Await a promise resolving to an object
+On the same page, call `window.fetchUser()` and retrieve the resolved object so you can read a field from it.
+
+**Verify**: The resolved object's `name` field equals `ASYNC_USER_NAME_ADA`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-22.md b/tests/benchmark/benchmark-run/group-22.md
new file mode 100644
index 00000000..d069d637
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-22.md
@@ -0,0 +1,14 @@
+# Group 22: Mouse Drag & Drop
+
+### 22.1 Drag a piece into Zone A
+Navigate to `http://fixtures/drag.html`. The page contains a draggable square (`#piece`) and three target zones (`#zone-a`, `#zone-b`, `#zone-c`). Drag the piece so its center ends up over **Zone A**.
+
+**Verify**: The page shows `LAST_DROP=DROP_ZONE_A_OK`.
+
+### 22.2 Drag to Zone B, then Zone C
+Without reloading the page, drag the piece next into **Zone B**, and then into **Zone C**. The page records an ordered drop sequence.
+
+**Verify**: The page shows `DROP_SEQUENCE=DROP_ZONE_A_OK,DROP_ZONE_B_OK,DROP_ZONE_C_OK` (all three drops in order).
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-23.md b/tests/benchmark/benchmark-run/group-23.md
new file mode 100644
index 00000000..ce4cfeeb
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-23.md
@@ -0,0 +1,9 @@
+# Group 23: Async / Loading state
+
+### 23.1 Wait for async content
+Navigate to `http://fixtures/loading.html`. The page shows a spinner and the text "Loading..." for ~1.5 seconds, then replaces it with the final payload. Wait for the final content to appear before reporting completion.
+
+**Verify**: The final snapshot contains `VERIFY_LOADING_COMPLETE_88888`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-24.md b/tests/benchmark/benchmark-run/group-24.md
new file mode 100644
index 00000000..20192423
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-24.md
@@ -0,0 +1,14 @@
+# Group 24: Keyboard events
+
+### 24.1 Press Escape
+Navigate to `http://fixtures/keyboard.html` (it auto-focuses an input so keyboard events land). Press the Escape key.
+
+**Verify**: Page text contains `KEYBOARD_ESCAPE_PRESSED`.
+
+### 24.2 Press 'a' then Enter
+Without reloading, press the `a` key, then the `Enter` key.
+
+**Verify**: Page text now contains all three markers in order: `KEYBOARD_ESCAPE_PRESSED`, `KEYBOARD_KEY_A_PRESSED`, `KEYBOARD_ENTER_PRESSED`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-25.md b/tests/benchmark/benchmark-run/group-25.md
new file mode 100644
index 00000000..370fafd0
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-25.md
@@ -0,0 +1,14 @@
+# Group 25: Tab panels
+
+### 25.1 Switch to the Settings tab
+Navigate to `http://fixtures/tabs.html`. The Profile tab is shown by default. Click the Settings tab.
+
+**Verify**: After the click, the page content includes `TAB_SETTINGS_CONTENT` (and the Profile content is no longer visible).
+
+### 25.2 Switch to the Billing tab
+Click the Billing tab.
+
+**Verify**: Page content now includes `TAB_BILLING_CONTENT`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-26.md b/tests/benchmark/benchmark-run/group-26.md
new file mode 100644
index 00000000..df0f37d3
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-26.md
@@ -0,0 +1,14 @@
+# Group 26: Accordion
+
+### 26.1 Open section A
+Navigate to `http://fixtures/accordion.html`. Click the header for Section A to expand it.
+
+**Verify**: Page text includes `ACCORDION_SECTION_A_OPEN`.
+
+### 26.2 Open section B
+Click the header for Section B. Because the accordion is exclusive-expand, Section A should close.
+
+**Verify**: Page text includes `ACCORDION_SECTION_B_OPEN`, and Section A's `aria-expanded` attribute is now `"false"` (use `eval` to check the attribute if needed).
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-27.md b/tests/benchmark/benchmark-run/group-27.md
new file mode 100644
index 00000000..5b974508
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-27.md
@@ -0,0 +1,14 @@
+# Group 27: Contenteditable editor
+
+### 27.1 Type into the rich-text editor
+Navigate to `http://fixtures/editor.html`. Type the text `Hello rich text` into the `#editor` div. `contenteditable` elements don't have `.value`, so use `type` (keyboard events), not `fill`.
+
+**Verify**: Page text contains `EDITOR_CHARS=15` and the mirror shows `Hello rich text`.
+
+### 27.2 Commit by pressing Enter
+Press Enter (the editor intercepts Enter to commit the current buffer to a separate marker).
+
+**Verify**: Page text contains `EDITOR_COMMITTED=Hello rich text`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-28.md b/tests/benchmark/benchmark-run/group-28.md
new file mode 100644
index 00000000..5f352fae
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-28.md
@@ -0,0 +1,14 @@
+# Group 28: Range slider
+
+### 28.1 Move slider into HIGH bucket
+Navigate to `http://fixtures/range.html`. Set the `#volume` range slider to 90.
+
+**Verify**: Page text contains both `RANGE_VALUE_90` and `BUCKET_HIGH`.
+
+### 28.2 Move slider into LOW bucket
+Without reloading, change the slider to 10.
+
+**Verify**: Page text now contains `RANGE_VALUE_10` and `BUCKET_LOW`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-29.md b/tests/benchmark/benchmark-run/group-29.md
new file mode 100644
index 00000000..ed935bca
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-29.md
@@ -0,0 +1,14 @@
+# Group 29: Pagination
+
+### 29.1 Advance to page 2
+Navigate to `http://fixtures/pagination.html`. Click the Next button.
+
+**Verify**: Page text contains `PAGE_2_FIRST_ITEM` and `PAGE_2_OF_3`.
+
+### 29.2 Reach the last page; Next disabled
+Click Next once more.
+
+**Verify**: Page text contains `PAGE_3_FIRST_ITEM` and `PAGE_3_OF_3`, and the Next button's `disabled` property is `true` (use `eval` to check).
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-30.md b/tests/benchmark/benchmark-run/group-30.md
new file mode 100644
index 00000000..69c78d4a
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-30.md
@@ -0,0 +1,14 @@
+# Group 30: Custom dropdown menu
+
+### 30.1 Pick the "Beta" item
+Navigate to `http://fixtures/dropdown.html`. The page has a custom dropdown — click the toggle button (`#dropdown-toggle`) to open the menu, then click the "Beta" item.
+
+**Verify**: Page text contains `DROPDOWN_SELECTED=BETA`.
+
+### 30.2 Reopen and pick "Gamma"
+Reopen the dropdown and select "Gamma". (Each selection auto-closes the menu, so you need to click the toggle again before selecting.)
+
+**Verify**: Page text contains `DROPDOWN_SELECTED=GAMMA`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-31.md b/tests/benchmark/benchmark-run/group-31.md
new file mode 100644
index 00000000..030c91cb
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-31.md
@@ -0,0 +1,9 @@
+# Group 31: Nested iframes (3 levels deep)
+
+### 31.1 Click a button in the deepest frame
+Navigate to `http://fixtures/iframe-nested.html`. The outer page embeds `#level-2`, which in turn embeds `#level-3`, which contains `#deep-button`. Drill through the two frame hops and click the button. Verify via a scoped `snap` before resetting to `main`.
+
+**Verify**: Scoped snapshot contains `DEEP_CLICKED=YES_LEVEL_3`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-32.md b/tests/benchmark/benchmark-run/group-32.md
new file mode 100644
index 00000000..d46bbb7e
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-32.md
@@ -0,0 +1,9 @@
+# Group 32: Dynamic iframe (inserted after load)
+
+### 32.1 Wait for a late iframe, then interact with it
+Navigate to `http://fixtures/iframe-dynamic.html`. The iframe is inserted ~1.2 s after load (use `wait --text "IFRAME_DYNAMIC_ATTACHED"`). Then scope into `#late-frame`, fill `#iframe-input` with "Late World", click `#iframe-submit`, and verify the inner result marker.
+
+**Verify**: Scoped snapshot contains `IFRAME_INPUT_RECEIVED_LATE_WORLD`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-33.md b/tests/benchmark/benchmark-run/group-33.md
new file mode 100644
index 00000000..abe79613
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-33.md
@@ -0,0 +1,9 @@
+# Group 33: srcdoc iframe
+
+### 33.1 Interact with an inline-content iframe
+Navigate to `http://fixtures/iframe-srcdoc.html`. The iframe's content is inlined via `srcdoc`; it has no `src` URL. Scope into `#srcdoc-frame`, fill `#inline-input` with `srcdoc`, click `#inline-submit`, and verify.
+
+**Verify**: Scoped snapshot contains `INLINE_RECEIVED_SRCDOC`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-34.md b/tests/benchmark/benchmark-run/group-34.md
new file mode 100644
index 00000000..27adab4f
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-34.md
@@ -0,0 +1,9 @@
+# Group 34: Sandboxed iframe
+
+### 34.1 Click inside a sandboxed iframe
+Navigate to `http://fixtures/iframe-sandbox.html`. The iframe has `sandbox="allow-scripts allow-same-origin"`. Scope into `#sandboxed`, click `#sandbox-button`, and verify.
+
+**Verify**: Scoped snapshot contains `SANDBOX_CLICKED=YES`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-35.md b/tests/benchmark/benchmark-run/group-35.md
new file mode 100644
index 00000000..eb313fec
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-35.md
@@ -0,0 +1,14 @@
+# Group 35: Long-form article (Medium/Substack style)
+
+### 35.1 Read the article with default `text`
+Navigate to `http://fixtures/article.html`. Extract the page's main content using the default `text` (Readability) mode. Default mode is the right choice for article-style pages.
+
+**Verify**: The extracted text contains both `ARTICLE_PUBLISHED_2026_04_15` and `ARTICLE_WORD_COUNT_MARKER_323` (both inside the article body — Readability keeps them).
+
+### 35.2 See the chrome that Readability drops
+Re-extract with `text --full` and confirm the footer is included this time.
+
+**Verify**: `--full` output contains `FOOTER_COPYRIGHT_MARKER` (which the default mode trims).
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-36.md b/tests/benchmark/benchmark-run/group-36.md
new file mode 100644
index 00000000..8cd88cb7
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-36.md
@@ -0,0 +1,14 @@
+# Group 36: Search results page (SERP)
+
+### 36.1 Find a specific result by id
+Navigate to `http://fixtures/serp.html`. The page has 6 result cards (`#r-1`..`#r-6`). Extract just the third card's content. A scoped snapshot (`snap --selector "#r-3"`) is the direct path.
+
+**Verify**: The scoped output contains `RESULT_3_TITLE` and `RESULT_3_SNIPPET_MARKER`.
+
+### 36.2 Count all result cards
+Use a full text extraction to verify all six results are present in one pass.
+
+**Verify**: Output contains all of `RESULT_1_TITLE` through `RESULT_6_TITLE` and the summary `SERP_RESULT_COUNT_6`. Default Readability trims SERPs — use `text --full`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-37.md b/tests/benchmark/benchmark-run/group-37.md
new file mode 100644
index 00000000..67d2b2d3
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-37.md
@@ -0,0 +1,14 @@
+# Group 37: Q&A thread (Stack-Overflow style)
+
+### 37.1 Find the accepted answer id
+Navigate to `http://fixtures/qa.html`. The accepted answer carries `data-accepted="true"` on its `<div>` wrapper. Use `eval` to return that element's `id`.
+
+**Verify**: The `eval` result equals `"a-2"`.
+
+### 37.2 Extract the accepted answer's body
+Scope a snapshot to `#a-2` and verify the body content.
+
+**Verify**: Scoped snapshot contains `ANSWER_2_BODY_MARKER` and `ACCEPTED_ANSWER_ID_A2`.
+
+---
+
diff --git a/tests/benchmark/benchmark-run/group-38.md b/tests/benchmark/benchmark-run/group-38.md
new file mode 100644
index 00000000..81917733
--- /dev/null
+++ b/tests/benchmark/benchmark-run/group-38.md
@@ -0,0 +1,11 @@
+# Group 38: Pricing table
+
+### 38.1 Read just the Pro plan
+Navigate to `http://fixtures/pricing.html`. The page has three plan cards (`#plan-free`, `#plan-pro`, `#plan-enterprise`). Read only the Pro plan using a scoped snapshot.
+
+**Verify**: Scoped snapshot contains `PLAN_PRO_PRICE_29` and `PLAN_PRO_LIMIT_5000_requests per day`.
+
+### 38.2 Compare all three plans
+Extract the full page (pricing grids are a Readability anti-pattern — reach for `--full`).
+
+**Verify**: Output contains `PLAN_FREE_PRICE_0`, `PLAN_PRO_PRICE_29`, and `PLAN_ENTERPRISE_PRICE_CUSTOM`.
diff --git a/tests/benchmark/benchmark-run/index.md b/tests/benchmark/benchmark-run/index.md
new file mode 100644
index 00000000..27f90db6
--- /dev/null
+++ b/tests/benchmark/benchmark-run/index.md
@@ -0,0 +1,55 @@
+# Benchmark Run
+
+Natural-language benchmark task set for the agent-driven lanes.
+
+Read this index first, then read the relevant group files.
+
+## How to use this directory
+
+- `setup-pinchtab.md` and `setup-agent-browser.md` define lane-specific setup, wrapper usage, recording, and Group 0 behavior.
+- This directory defines the shared task groups.
+- Each group lives in its own file: `group-00.md` through `group-38.md`.
+- When running a subset, read only the selected group files.
+- When running the full suite, proceed in numeric order.
+
+## Group Files
+
+- `group-00.md` — Setup & Diagnosis
+- `group-01.md` — Reading & Extracting Real Content
+- `group-02.md` — Search & Dynamic Interaction
+- `group-03.md` — Complex Form
+- `group-04.md` — SPA State Management
+- `group-05.md` — Login Flow
+- `group-06.md` — Multi-Step E-commerce
+- `group-07.md` — Content + Interaction Combined
+- `group-08.md` — Error Handling
+- `group-09.md` — Export
+- `group-10.md` — Nested Interactions & Modal Dialogs
+- `group-11.md` — State Persistence & Page Reload
+- `group-12.md` — Multi-Page Navigation & Back Button
+- `group-13.md` — Form State & Multi-Step Submission
+- `group-14.md` — Dynamic Content Loading
+- `group-15.md` — Complex Data Extraction & Aggregation
+- `group-16.md` — Hover & Tooltips
+- `group-17.md` — Scrolling
+- `group-18.md` — File Download
+- `group-19.md` — iFrame
+- `group-20.md` — Dialogs
+- `group-21.md` — Async / awaitPromise
+- `group-22.md` — Mouse Drag & Drop
+- `group-23.md` — Async / Loading state
+- `group-24.md` — Keyboard events
+- `group-25.md` — Tab panels
+- `group-26.md` — Accordion
+- `group-27.md` — Contenteditable editor
+- `group-28.md` — Range slider
+- `group-29.md` — Pagination
+- `group-30.md` — Custom dropdown menu
+- `group-31.md` — Nested iframes (3 levels deep)
+- `group-32.md` — Dynamic iframe (inserted after load)
+- `group-33.md` — srcdoc iframe
+- `group-34.md` — Sandboxed iframe
+- `group-35.md` — Long-form article (Medium/Substack style)
+- `group-36.md` — Search results page (SERP)
+- `group-37.md` — Q&A thread (Stack-Overflow style)
+- `group-38.md` — Pricing table
diff --git a/tests/benchmark/cmd/api-runner/anthropic.go b/tests/benchmark/cmd/api-runner/anthropic.go
new file mode 100644
index 00000000..952c8219
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/anthropic.go
@@ -0,0 +1,219 @@
+package main
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "fmt"
+ "io"
+ "net/http"
+ "time"
+)
+
+const AnthropicAPIURL = "https://api.anthropic.com/v1/messages"
+
+type AnthropicRunner struct {
+ apiKey string
+ model string
+ maxTokens int
+ temperature float64
+ promptCaching bool
+ usage UsageCounters
+ client *http.Client
+ retryCfg RetryConfig
+}
+
+func NewAnthropicRunner(apiKey, model string, maxTokens int, temperature float64, promptCaching bool) *AnthropicRunner {
+ return &AnthropicRunner{
+ apiKey: apiKey,
+ model: model,
+ maxTokens: maxTokens,
+ temperature: temperature,
+ promptCaching: promptCaching,
+ client: &http.Client{Timeout: 5 * time.Minute},
+ retryCfg: DefaultRetryConfig(),
+ }
+}
+
+func (r *AnthropicRunner) Provider() string { return "anthropic" }
+func (r *AnthropicRunner) Source() string { return "anthropic-messages" }
+func (r *AnthropicRunner) Model() string { return r.model }
+func (r *AnthropicRunner) Usage() UsageCounters { return r.usage }
+
+func (r *AnthropicRunner) ToolDefinitions() interface{} {
+ return []map[string]interface{}{
+ {
+ "name": "run_command",
+ "description": "Run a shell command in a persistent bash session rooted at tests/benchmark. Environment changes such as cd/export persist across calls.",
+ "input_schema": map[string]interface{}{
+ "type": "object",
+ "properties": map[string]interface{}{
+ "command": map[string]interface{}{"type": "string"},
+ "timeout_seconds": map[string]interface{}{"type": "integer", "minimum": 1, "maximum": 600},
+ },
+ "required": []string{"command"},
+ },
+ },
+ }
+}
+
+func (r *AnthropicRunner) InitialConversation(userPrompt string) []interface{} {
+ return []interface{}{
+ map[string]interface{}{
+ "role": "user",
+ "content": userPrompt,
+ },
+ }
+}
+
+func (r *AnthropicRunner) Send(systemPrompt string, conversation []interface{}) (interface{}, error) {
+ body := map[string]interface{}{
+ "model": r.model,
+ "max_tokens": r.maxTokens,
+ "temperature": r.temperature,
+ "system": systemPrompt,
+ "tools": r.ToolDefinitions(),
+ "messages": conversation,
+ }
+ if r.promptCaching {
+ body["cache_control"] = map[string]string{"type": "ephemeral"}
+ }
+
+ payload, err := json.Marshal(body)
+ if err != nil {
+ return nil, fmt.Errorf("marshal request: %w", err)
+ }
+
+ var result map[string]interface{}
+ resp, err := DoWithRetry(context.Background(), r.retryCfg, func() (*http.Response, error) {
+ req, err := http.NewRequest("POST", AnthropicAPIURL, bytes.NewReader(payload))
+ if err != nil {
+ return nil, err
+ }
+ req.Header.Set("Content-Type", "application/json")
+ req.Header.Set("X-Api-Key", r.apiKey)
+ req.Header.Set("Anthropic-Version", "2023-06-01")
+ return r.client.Do(req)
+ })
+ if err != nil {
+ return nil, err
+ }
+ defer resp.Body.Close()
+
+ respBody, err := io.ReadAll(resp.Body)
+ if err != nil {
+ return nil, fmt.Errorf("read response: %w", err)
+ }
+
+ if err := json.Unmarshal(respBody, &result); err != nil {
+ return nil, fmt.Errorf("unmarshal response: %w", err)
+ }
+
+ if resp.StatusCode >= 400 {
+ return nil, fmt.Errorf("Anthropic API error %d: %s", resp.StatusCode, string(respBody))
+ }
+
+ r.updateUsage(result)
+ return result, nil
+}
+
+func (r *AnthropicRunner) updateUsage(result map[string]interface{}) {
+ usage, _ := result["usage"].(map[string]interface{})
+ if usage == nil {
+ return
+ }
+ r.usage.RequestCount++
+ r.usage.InputTokens += toInt(usage["input_tokens"])
+ r.usage.OutputTokens += toInt(usage["output_tokens"])
+ r.usage.CacheCreationInputTokens += toInt(usage["cache_creation_input_tokens"])
+ r.usage.CacheReadInputTokens += toInt(usage["cache_read_input_tokens"])
+}
+
+func toInt(v interface{}) int {
+ switch n := v.(type) {
+ case float64:
+ return int(n)
+ case int:
+ return n
+ default:
+ return 0
+ }
+}
+
+func (r *AnthropicRunner) ExtractToolCalls(response interface{}, defaultTimeout time.Duration) []ToolCall {
+ resp, _ := response.(map[string]interface{})
+ content, _ := resp["content"].([]interface{})
+ var calls []ToolCall
+ for _, item := range content {
+ m, _ := item.(map[string]interface{})
+ if m["type"] != "tool_use" {
+ continue
+ }
+ input, _ := m["input"].(map[string]interface{})
+ cmd, _ := input["command"].(string)
+ if cmd == "" {
+ continue
+ }
+ timeout := int(defaultTimeout.Seconds())
+ if t := toInt(input["timeout_seconds"]); t > 0 {
+ timeout = t
+ }
+ calls = append(calls, ToolCall{
+ ID: fmt.Sprintf("%v", m["id"]),
+ Command: cmd,
+ TimeoutSeconds: timeout,
+ })
+ }
+ return calls
+}
+
+func (r *AnthropicRunner) AppendToolResults(conversation []interface{}, response interface{}, results []ToolExecutionResult) []interface{} {
+ resp, _ := response.(map[string]interface{})
+ conversation = append(conversation, map[string]interface{}{
+ "role": "assistant",
+ "content": resp["content"],
+ })
+ if len(results) > 0 {
+ toolResults := make([]interface{}, len(results))
+ for i, res := range results {
+ toolResults[i] = map[string]interface{}{
+ "type": "tool_result",
+ "tool_use_id": res.ID,
+ "is_error": res.IsError,
+ "content": res.Content,
+ }
+ }
+ conversation = append(conversation, map[string]interface{}{
+ "role": "user",
+ "content": toolResults,
+ })
+ }
+ return conversation
+}
+
+func (r *AnthropicRunner) ExtractFinalText(response interface{}) string {
+ resp, _ := response.(map[string]interface{})
+ content, _ := resp["content"].([]interface{})
+ var texts []string
+ for _, item := range content {
+ m, _ := item.(map[string]interface{})
+ if m["type"] != "text" {
+ continue
+ }
+ if text, ok := m["text"].(string); ok && text != "" {
+ texts = append(texts, text)
+ }
+ }
+ return joinStrings(texts, "\n")
+}
+
+func joinStrings(strs []string, sep string) string {
+ if len(strs) == 0 {
+ return ""
+ }
+ result := strs[0]
+ for _, s := range strs[1:] {
+ result += sep + s
+ }
+ return result
+}
diff --git a/tests/benchmark/cmd/api-runner/anthropic_test.go b/tests/benchmark/cmd/api-runner/anthropic_test.go
new file mode 100644
index 00000000..219185ba
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/anthropic_test.go
@@ -0,0 +1,185 @@
+package main
+
+import (
+ "encoding/json"
+ "io"
+ "net/http"
+ "net/http/httptest"
+ "testing"
+ "time"
+)
+
+func TestAnthropicToolDefinitions(t *testing.T) {
+ r := NewAnthropicRunner("key", "claude-haiku-4-5-20251001", 4096, 0, true)
+ tools := r.ToolDefinitions().([]map[string]interface{})
+ if len(tools) != 1 {
+ t.Fatalf("got %d tools; want 1", len(tools))
+ }
+ if tools[0]["name"] != "run_command" {
+ t.Errorf("tool name = %v; want 'run_command'", tools[0]["name"])
+ }
+}
+
+func TestAnthropicInitialConversation(t *testing.T) {
+ r := NewAnthropicRunner("key", "claude-haiku-4-5-20251001", 4096, 0, true)
+ conv := r.InitialConversation("test prompt")
+ if len(conv) != 1 {
+ t.Fatalf("got %d messages; want 1", len(conv))
+ }
+ msg := conv[0].(map[string]interface{})
+ if msg["role"] != "user" {
+ t.Errorf("role = %v; want 'user'", msg["role"])
+ }
+ if msg["content"] != "test prompt" {
+ t.Errorf("content = %v; want 'test prompt'", msg["content"])
+ }
+}
+
+func TestAnthropicSend(t *testing.T) {
+ var receivedBody map[string]interface{}
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ if r.Method != "POST" {
+ t.Errorf("method = %s; want POST", r.Method)
+ }
+ if r.Header.Get("X-Api-Key") != "test-key" {
+ t.Errorf("X-Api-Key = %s; want test-key", r.Header.Get("X-Api-Key"))
+ }
+ if r.Header.Get("Anthropic-Version") != "2023-06-01" {
+ t.Errorf("Anthropic-Version = %s; want 2023-06-01", r.Header.Get("Anthropic-Version"))
+ }
+
+ body, _ := io.ReadAll(r.Body)
+ json.Unmarshal(body, &receivedBody)
+
+ w.Header().Set("Content-Type", "application/json")
+ json.NewEncoder(w).Encode(map[string]interface{}{
+ "content": []interface{}{
+ map[string]interface{}{"type": "text", "text": "Hello"},
+ },
+ "usage": map[string]interface{}{
+ "input_tokens": 100,
+ "output_tokens": 50,
+ },
+ })
+ }))
+ defer server.Close()
+
+ r := NewAnthropicRunner("test-key", "claude-haiku-4-5-20251001", 4096, 0.5, true)
+ r.retryCfg.Sleep = func(d time.Duration) {}
+
+ originalURL := AnthropicAPIURL
+ defer func() { _ = originalURL }()
+
+ conv := r.InitialConversation("hello")
+ r.client = &http.Client{Transport: &testTransport{server.URL}}
+
+ _, err := r.Send("system", conv)
+ if err != nil {
+ t.Fatal(err)
+ }
+
+ if receivedBody["model"] != "claude-haiku-4-5-20251001" {
+ t.Errorf("model = %v; want claude-haiku-4-5-20251001", receivedBody["model"])
+ }
+ if receivedBody["max_tokens"] != float64(4096) {
+ t.Errorf("max_tokens = %v; want 4096", receivedBody["max_tokens"])
+ }
+ if _, ok := receivedBody["cache_control"]; !ok {
+ t.Error("missing cache_control")
+ }
+
+ usage := r.Usage()
+ if usage.RequestCount != 1 {
+ t.Errorf("RequestCount = %d; want 1", usage.RequestCount)
+ }
+ if usage.InputTokens != 100 {
+ t.Errorf("InputTokens = %d; want 100", usage.InputTokens)
+ }
+}
+
+type testTransport struct {
+ targetURL string
+}
+
+func (t *testTransport) RoundTrip(req *http.Request) (*http.Response, error) {
+ newReq, _ := http.NewRequest(req.Method, t.targetURL, req.Body)
+ for k, v := range req.Header {
+ newReq.Header[k] = v
+ }
+ return http.DefaultTransport.RoundTrip(newReq)
+}
+
+func TestAnthropicExtractToolCalls(t *testing.T) {
+ r := NewAnthropicRunner("key", "model", 4096, 0, true)
+ response := map[string]interface{}{
+ "content": []interface{}{
+ map[string]interface{}{
+ "type": "tool_use",
+ "id": "toolu_123",
+ "input": map[string]interface{}{"command": "ls -la", "timeout_seconds": 30.0},
+ },
+ map[string]interface{}{
+ "type": "text",
+ "text": "Let me list files",
+ },
+ },
+ }
+
+ calls := r.ExtractToolCalls(response, 120*time.Second)
+ if len(calls) != 1 {
+ t.Fatalf("got %d calls; want 1", len(calls))
+ }
+ if calls[0].ID != "toolu_123" {
+ t.Errorf("ID = %s; want toolu_123", calls[0].ID)
+ }
+ if calls[0].Command != "ls -la" {
+ t.Errorf("Command = %s; want 'ls -la'", calls[0].Command)
+ }
+ if calls[0].TimeoutSeconds != 30 {
+ t.Errorf("TimeoutSeconds = %d; want 30", calls[0].TimeoutSeconds)
+ }
+}
+
+func TestAnthropicExtractFinalText(t *testing.T) {
+ r := NewAnthropicRunner("key", "model", 4096, 0, true)
+ response := map[string]interface{}{
+ "content": []interface{}{
+ map[string]interface{}{"type": "text", "text": "Hello"},
+ map[string]interface{}{"type": "tool_use", "id": "123"},
+ map[string]interface{}{"type": "text", "text": "World"},
+ },
+ }
+
+ text := r.ExtractFinalText(response)
+ if text != "Hello\nWorld" {
+ t.Errorf("got %q; want 'Hello\\nWorld'", text)
+ }
+}
+
+func TestAnthropicAppendToolResults(t *testing.T) {
+ r := NewAnthropicRunner("key", "model", 4096, 0, true)
+ conv := []interface{}{}
+ response := map[string]interface{}{
+ "content": []interface{}{
+ map[string]interface{}{"type": "tool_use", "id": "123"},
+ },
+ }
+ results := []ToolExecutionResult{
+ {ID: "123", IsError: false, Content: "output"},
+ }
+
+ conv = r.AppendToolResults(conv, response, results)
+ if len(conv) != 2 {
+ t.Fatalf("got %d messages; want 2", len(conv))
+ }
+
+ assistant := conv[0].(map[string]interface{})
+ if assistant["role"] != "assistant" {
+ t.Errorf("first message role = %v; want 'assistant'", assistant["role"])
+ }
+
+ user := conv[1].(map[string]interface{})
+ if user["role"] != "user" {
+ t.Errorf("second message role = %v; want 'user'", user["role"])
+ }
+}
diff --git a/tests/benchmark/cmd/api-runner/args.go b/tests/benchmark/cmd/api-runner/args.go
new file mode 100644
index 00000000..fee83b64
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/args.go
@@ -0,0 +1,298 @@
+// Package main implements the Go port of tests/benchmark/scripts/run-api-benchmark.ts.
+//
+// This file defines the command-line interface. The flag surface mirrors the
+// TypeScript runner so `./dev benchmark --runner go ...` can be swapped in
+// without callers changing their invocation. Two new flags, --dry-run and
+// --index-file, are added here so later steps can exercise the plan/prompt
+// assembly without needing network access or the default index.md location.
+package main
+
+import (
+ "errors"
+ "fmt"
+ "io"
+ "sort"
+ "strconv"
+ "strings"
+)
+
+// Lane is the benchmark lane (pinchtab or agent-browser).
+type Lane string
+
+const (
+ LanePinchtab Lane = "pinchtab"
+ LaneAgentBrowser Lane = "agent-browser"
+)
+
+// Provider identifies which API the runner talks to.
+type Provider string
+
+const (
+ ProviderAnthropic Provider = "anthropic"
+ ProviderOpenAI Provider = "openai"
+ ProviderFake Provider = "fake"
+ ProviderUnset Provider = ""
+)
+
+// Args is the resolved plan for a single invocation. It is intentionally a
+// plain-data struct so tests can construct one without touching os.Args.
+type Args struct {
+ Lane Lane
+ Provider Provider
+ Model string
+ Groups []int
+ Profile string
+ MaxTokens int
+ Temperature float64
+ MaxTurns int
+ MaxIdleTurns int
+ TimeoutSeconds int
+ TurnDelayMs int
+ ReportFile string
+ SkipInit bool
+ NoPromptCaching bool
+ Finalize bool
+ DryRun bool
+ IndexFile string
+ MaxInputTokens int
+ MaxOutputTokens int
+}
+
+// Defaults matches the literal defaults in the TypeScript runner. Keep this in
+// sync with parseArgs() in run-api-benchmark.ts.
+func defaultArgs() Args {
+ return Args{
+ MaxTokens: 4096,
+ Temperature: 0,
+ MaxTurns: 300,
+ MaxIdleTurns: 25,
+ TimeoutSeconds: 120,
+ TurnDelayMs: 1500,
+ }
+}
+
+const usageText = `Usage:
+ go run ./tests/benchmark/cmd/api-runner --lane pinchtab [options]
+ go run ./tests/benchmark/cmd/api-runner --lane agent-browser [options]
+
+Options:
+ --provider anthropic|openai|fake
+ --model MODEL
+ --groups 0,1,2,3
+ --profile common10
+ --max-tokens N
+ --temperature N
+ --max-turns N
+ --max-idle-turns N
+ --timeout-seconds N
+ --turn-delay-ms N
+ --report-file PATH
+ --skip-init
+ --no-prompt-caching
+ --finalize
+ --dry-run Print the resolved plan and exit 0 without network access
+ --index-file PATH Override path to tests/benchmark/benchmark-run/index.md
+ --max-input-tokens N Stop when cumulative input tokens exceed N (exit code 4)
+ --max-output-tokens N Stop when cumulative output tokens exceed N (exit code 4)
+`
+
+// ParseArgs walks argv manually (like the TS runner) so the flag surface and
+// error messages stay byte-identical. The stdlib `flag` package would reorder
+// output and reject `--groups 0,1,2` style values in some edge cases.
+func ParseArgs(argv []string) (Args, error) {
+ a := defaultArgs()
+
+ next := func(i *int, name string) (string, error) {
+ *i++
+ if *i >= len(argv) {
+ return "", fmt.Errorf("%s requires a value", name)
+ }
+ return argv[*i], nil
+ }
+
+ for i := 0; i < len(argv); i++ {
+ arg := argv[i]
+ switch arg {
+ case "--lane":
+ v, err := next(&i, arg)
+ if err != nil {
+ return a, err
+ }
+ a.Lane = Lane(v)
+ case "--provider":
+ v, err := next(&i, arg)
+ if err != nil {
+ return a, err
+ }
+ a.Provider = Provider(v)
+ case "--model":
+ v, err := next(&i, arg)
+ if err != nil {
+ return a, err
+ }
+ a.Model = v
+ case "--groups":
+ v, err := next(&i, arg)
+ if err != nil {
+ return a, err
+ }
+ groups, perr := parseGroups(v)
+ if perr != nil {
+ return a, perr
+ }
+ a.Groups = groups
+ case "--profile":
+ v, err := next(&i, arg)
+ if err != nil {
+ return a, err
+ }
+ a.Profile = v
+ case "--max-tokens":
+ v, err := next(&i, arg)
+ if err != nil {
+ return a, err
+ }
+ n, perr := strconv.Atoi(v)
+ if perr != nil {
+ return a, fmt.Errorf("--max-tokens: %w", perr)
+ }
+ a.MaxTokens = n
+ case "--temperature":
+ v, err := next(&i, arg)
+ if err != nil {
+ return a, err
+ }
+ f, perr := strconv.ParseFloat(v, 64)
+ if perr != nil {
+ return a, fmt.Errorf("--temperature: %w", perr)
+ }
+ a.Temperature = f
+ case "--max-turns":
+ v, err := next(&i, arg)
+ if err != nil {
+ return a, err
+ }
+ n, perr := strconv.Atoi(v)
+ if perr != nil {
+ return a, fmt.Errorf("--max-turns: %w", perr)
+ }
+ a.MaxTurns = n
+ case "--max-idle-turns":
+ v, err := next(&i, arg)
+ if err != nil {
+ return a, err
+ }
+ n, perr := strconv.Atoi(v)
+ if perr != nil {
+ return a, fmt.Errorf("--max-idle-turns: %w", perr)
+ }
+ a.MaxIdleTurns = n
+ case "--timeout-seconds":
+ v, err := next(&i, arg)
+ if err != nil {
+ return a, err
+ }
+ n, perr := strconv.Atoi(v)
+ if perr != nil {
+ return a, fmt.Errorf("--timeout-seconds: %w", perr)
+ }
+ a.TimeoutSeconds = n
+ case "--turn-delay-ms":
+ v, err := next(&i, arg)
+ if err != nil {
+ return a, err
+ }
+ n, perr := strconv.Atoi(v)
+ if perr != nil {
+ return a, fmt.Errorf("--turn-delay-ms: %w", perr)
+ }
+ a.TurnDelayMs = n
+ case "--report-file":
+ v, err := next(&i, arg)
+ if err != nil {
+ return a, err
+ }
+ a.ReportFile = v
+ case "--skip-init":
+ a.SkipInit = true
+ case "--no-prompt-caching":
+ a.NoPromptCaching = true
+ case "--finalize":
+ a.Finalize = true
+ case "--dry-run":
+ a.DryRun = true
+ case "--index-file":
+ v, err := next(&i, arg)
+ if err != nil {
+ return a, err
+ }
+ a.IndexFile = v
+ case "--max-input-tokens":
+ v, err := next(&i, arg)
+ if err != nil {
+ return a, err
+ }
+ n, perr := strconv.Atoi(v)
+ if perr != nil {
+ return a, fmt.Errorf("--max-input-tokens: %w", perr)
+ }
+ a.MaxInputTokens = n
+ case "--max-output-tokens":
+ v, err := next(&i, arg)
+ if err != nil {
+ return a, err
+ }
+ n, perr := strconv.Atoi(v)
+ if perr != nil {
+ return a, fmt.Errorf("--max-output-tokens: %w", perr)
+ }
+ a.MaxOutputTokens = n
+ case "-h", "--help", "help":
+ return a, errHelp
+ default:
+ return a, fmt.Errorf("unknown argument: %s", arg)
+ }
+ }
+
+ if a.Lane != LanePinchtab && a.Lane != LaneAgentBrowser {
+ return a, errors.New("--lane must be 'pinchtab' or 'agent-browser'")
+ }
+
+ return a, nil
+}
+
+// errHelp is a sentinel the caller uses to distinguish "user asked for help"
+// (exit 0) from "bad flags" (exit 1). It matches the TS runner's usage(0) vs
+// usage(1) behaviour.
+var errHelp = errors.New("help requested")
+
+// parseGroups splits "0,1,2,3" into sorted unique ints, matching the TS
+// implementation's `.map(Number).filter(Number.isInteger)` semantics.
+func parseGroups(raw string) ([]int, error) {
+ seen := make(map[int]struct{})
+ var out []int
+ for _, piece := range strings.Split(raw, ",") {
+ piece = strings.TrimSpace(piece)
+ if piece == "" {
+ continue
+ }
+ n, err := strconv.Atoi(piece)
+ if err != nil {
+ // TS filters silently; we do too, to preserve parity.
+ continue
+ }
+ if _, dup := seen[n]; dup {
+ continue
+ }
+ seen[n] = struct{}{}
+ out = append(out, n)
+ }
+ sort.Ints(out)
+ return out, nil
+}
+
+// WriteUsage prints the help block to the given writer. Kept as a helper so
+// tests can capture output without redirecting os.Stderr.
+func WriteUsage(w io.Writer) {
+ fmt.Fprint(w, usageText)
+}
diff --git a/tests/benchmark/cmd/api-runner/compact.go b/tests/benchmark/cmd/api-runner/compact.go
new file mode 100644
index 00000000..e787cd6b
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/compact.go
@@ -0,0 +1,59 @@
+package main
+
+const (
+ CompactAfterAnthropicMessages = 10
+ KeepRecentAnthropicMessages = 6
+ CompactAfterOpenAIItems = 16
+ KeepRecentOpenAIItems = 10
+)
+
+func CompactAnthropicConversation(conversation []interface{}, summary string) []interface{} {
+ if len(conversation) <= CompactAfterAnthropicMessages {
+ return conversation
+ }
+ head := conversation[0]
+ recentStart := len(conversation) - KeepRecentAnthropicMessages
+ if recentStart < 1 {
+ recentStart = 1
+ }
+ recent := conversation[recentStart:]
+
+ result := make([]interface{}, 0, 2+len(recent))
+ result = append(result, head)
+ result = append(result, map[string]interface{}{
+ "role": "user",
+ "content": summary,
+ })
+ result = append(result, recent...)
+ return result
+}
+
+func CompactOpenAIConversation(conversation []interface{}, summary string) []interface{} {
+ if len(conversation) <= CompactAfterOpenAIItems {
+ return conversation
+ }
+ head := conversation[0]
+ recentStart := len(conversation) - KeepRecentOpenAIItems
+ if recentStart < 1 {
+ recentStart = 1
+ }
+ recent := conversation[recentStart:]
+
+ result := make([]interface{}, 0, 2+len(recent))
+ result = append(result, head)
+ result = append(result, map[string]interface{}{
+ "role": "user",
+ "content": []interface{}{
+ map[string]interface{}{"type": "input_text", "text": summary},
+ },
+ })
+ result = append(result, recent...)
+ return result
+}
+
+func CompactConversation(provider string, conversation []interface{}, summary string) []interface{} {
+ if provider == "anthropic" {
+ return CompactAnthropicConversation(conversation, summary)
+ }
+ return CompactOpenAIConversation(conversation, summary)
+}
diff --git a/tests/benchmark/cmd/api-runner/compact_test.go b/tests/benchmark/cmd/api-runner/compact_test.go
new file mode 100644
index 00000000..cd2c56f9
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/compact_test.go
@@ -0,0 +1,88 @@
+package main
+
+import (
+ "testing"
+)
+
+func TestCompactAnthropicNoOp(t *testing.T) {
+ conv := make([]interface{}, 8)
+ for i := range conv {
+ conv[i] = map[string]interface{}{"role": "user", "content": "msg"}
+ }
+ result := CompactAnthropicConversation(conv, "summary")
+ if len(result) != 8 {
+ t.Errorf("got %d; want 8 (no compaction)", len(result))
+ }
+}
+
+func TestCompactAnthropicTriggered(t *testing.T) {
+ conv := make([]interface{}, 12)
+ conv[0] = map[string]interface{}{"role": "user", "content": "initial"}
+ for i := 1; i < 12; i++ {
+ conv[i] = map[string]interface{}{"role": "assistant", "content": i}
+ }
+
+ result := CompactAnthropicConversation(conv, "summary text")
+
+ if len(result) != 8 {
+ t.Errorf("got %d; want 8 (head + summary + 6 recent)", len(result))
+ }
+
+ summary := result[1].(map[string]interface{})
+ if summary["content"] != "summary text" {
+ t.Errorf("summary content = %v", summary["content"])
+ }
+}
+
+func TestCompactOpenAINoOp(t *testing.T) {
+ conv := make([]interface{}, 14)
+ for i := range conv {
+ conv[i] = map[string]interface{}{"type": "message"}
+ }
+ result := CompactOpenAIConversation(conv, "summary")
+ if len(result) != 14 {
+ t.Errorf("got %d; want 14 (no compaction)", len(result))
+ }
+}
+
+func TestCompactOpenAITriggered(t *testing.T) {
+ conv := make([]interface{}, 20)
+ conv[0] = map[string]interface{}{"role": "user", "content": "initial"}
+ for i := 1; i < 20; i++ {
+ conv[i] = map[string]interface{}{"type": "function_call", "id": i}
+ }
+
+ result := CompactOpenAIConversation(conv, "summary text")
+
+ if len(result) != 12 {
+ t.Errorf("got %d; want 12 (head + summary + 10 recent)", len(result))
+ }
+
+ summary := result[1].(map[string]interface{})
+ content := summary["content"].([]interface{})
+ if len(content) != 1 {
+ t.Fatalf("summary content length = %d; want 1", len(content))
+ }
+ part := content[0].(map[string]interface{})
+ if part["text"] != "summary text" {
+ t.Errorf("summary text = %v", part["text"])
+ }
+}
+
+func TestCompactConversationRouting(t *testing.T) {
+ conv := make([]interface{}, 20)
+ conv[0] = map[string]interface{}{"role": "user", "content": "initial"}
+ for i := 1; i < 20; i++ {
+ conv[i] = map[string]interface{}{"role": "assistant"}
+ }
+
+ anthropic := CompactConversation("anthropic", conv, "summary")
+ if len(anthropic) != 8 {
+ t.Errorf("anthropic compaction = %d; want 8", len(anthropic))
+ }
+
+ openai := CompactConversation("openai", conv, "summary")
+ if len(openai) != 12 {
+ t.Errorf("openai compaction = %d; want 12", len(openai))
+ }
+}
diff --git a/tests/benchmark/cmd/api-runner/fake.go b/tests/benchmark/cmd/api-runner/fake.go
new file mode 100644
index 00000000..b4ae564d
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/fake.go
@@ -0,0 +1,165 @@
+package main
+
+import (
+ "fmt"
+ "time"
+)
+
+type FakeResponse struct {
+ ToolCalls []ToolCall
+ FinalText string
+}
+
+type FakeRunner struct {
+ model string
+ responses []FakeResponse
+ turnIndex int
+ usage UsageCounters
+}
+
+func NewFakeRunner(model string, responses []FakeResponse) *FakeRunner {
+ return &FakeRunner{
+ model: model,
+ responses: responses,
+ }
+}
+
+func (r *FakeRunner) Provider() string { return "fake" }
+func (r *FakeRunner) Source() string { return "fake-runner" }
+func (r *FakeRunner) Model() string { return r.model }
+func (r *FakeRunner) Usage() UsageCounters { return r.usage }
+
+func (r *FakeRunner) ToolDefinitions() interface{} {
+ return []map[string]interface{}{
+ {
+ "name": "run_command",
+ "description": "Run a shell command",
+ "input_schema": map[string]interface{}{
+ "type": "object",
+ "properties": map[string]interface{}{
+ "command": map[string]interface{}{"type": "string"},
+ },
+ },
+ },
+ }
+}
+
+func (r *FakeRunner) InitialConversation(userPrompt string) []interface{} {
+ return []interface{}{
+ map[string]interface{}{
+ "role": "user",
+ "content": userPrompt,
+ },
+ }
+}
+
+func (r *FakeRunner) Send(systemPrompt string, conversation []interface{}) (interface{}, error) {
+ r.usage.RequestCount++
+ r.usage.InputTokens += 100
+ r.usage.OutputTokens += 50
+
+ if r.turnIndex >= len(r.responses) {
+ return map[string]interface{}{
+ "final_text": "No more scripted responses",
+ }, nil
+ }
+
+ resp := r.responses[r.turnIndex]
+ r.turnIndex++
+
+ if len(resp.ToolCalls) > 0 {
+ content := make([]interface{}, len(resp.ToolCalls))
+ for i, tc := range resp.ToolCalls {
+ content[i] = map[string]interface{}{
+ "type": "tool_use",
+ "id": tc.ID,
+ "input": map[string]interface{}{
+ "command": tc.Command,
+ "timeout_seconds": tc.TimeoutSeconds,
+ },
+ }
+ }
+ return map[string]interface{}{"content": content}, nil
+ }
+
+ return map[string]interface{}{
+ "content": []interface{}{
+ map[string]interface{}{
+ "type": "text",
+ "text": resp.FinalText,
+ },
+ },
+ }, nil
+}
+
+func (r *FakeRunner) ExtractToolCalls(response interface{}, defaultTimeout time.Duration) []ToolCall {
+ resp, _ := response.(map[string]interface{})
+ content, _ := resp["content"].([]interface{})
+ var calls []ToolCall
+ for _, item := range content {
+ m, _ := item.(map[string]interface{})
+ if m["type"] != "tool_use" {
+ continue
+ }
+ input, _ := m["input"].(map[string]interface{})
+ cmd, _ := input["command"].(string)
+ if cmd == "" {
+ continue
+ }
+ timeout := int(defaultTimeout.Seconds())
+ if t := toInt(input["timeout_seconds"]); t > 0 {
+ timeout = t
+ }
+ calls = append(calls, ToolCall{
+ ID: fmt.Sprintf("%v", m["id"]),
+ Command: cmd,
+ TimeoutSeconds: timeout,
+ })
+ }
+ return calls
+}
+
+func (r *FakeRunner) AppendToolResults(conversation []interface{}, response interface{}, results []ToolExecutionResult) []interface{} {
+ resp, _ := response.(map[string]interface{})
+ conversation = append(conversation, map[string]interface{}{
+ "role": "assistant",
+ "content": resp["content"],
+ })
+ if len(results) > 0 {
+ toolResults := make([]interface{}, len(results))
+ for i, res := range results {
+ toolResults[i] = map[string]interface{}{
+ "type": "tool_result",
+ "tool_use_id": res.ID,
+ "is_error": res.IsError,
+ "content": res.Content,
+ }
+ }
+ conversation = append(conversation, map[string]interface{}{
+ "role": "user",
+ "content": toolResults,
+ })
+ }
+ return conversation
+}
+
+func (r *FakeRunner) ExtractFinalText(response interface{}) string {
+ resp, _ := response.(map[string]interface{})
+
+ if text, ok := resp["final_text"].(string); ok {
+ return text
+ }
+
+ content, _ := resp["content"].([]interface{})
+ var texts []string
+ for _, item := range content {
+ m, _ := item.(map[string]interface{})
+ if m["type"] != "text" {
+ continue
+ }
+ if text, ok := m["text"].(string); ok && text != "" {
+ texts = append(texts, text)
+ }
+ }
+ return joinStrings(texts, "\n")
+}
diff --git a/tests/benchmark/cmd/api-runner/index.go b/tests/benchmark/cmd/api-runner/index.go
new file mode 100644
index 00000000..3749f4ee
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/index.go
@@ -0,0 +1,140 @@
+package main
+
+import (
+ "bufio"
+ "os"
+ "path/filepath"
+ "regexp"
+ "sort"
+ "strconv"
+ "strings"
+)
+
+var (
+ groupFileRegex = regexp.MustCompile(`^group-(\d+)\.md$`)
+ listItemRegex = regexp.MustCompile(`^-\s+` + "`?" + `group-(\d+)\.md` + "`?")
+ htmlCommentStart = regexp.MustCompile(`<!--`)
+ htmlCommentEnd = regexp.MustCompile(`-->`)
+ commentedListItem = regexp.MustCompile(`<!--\s*-\s+` + "`?" + `group-(\d+)\.md` + "`?")
+)
+
+type IndexFile struct {
+ Dir string
+ Active []int
+ Disabled []int
+}
+
+func ParseIndexFile(path string) (IndexFile, error) {
+ dir := filepath.Dir(path)
+ content, err := os.ReadFile(path)
+ if err != nil {
+ return IndexFile{Dir: dir}, err
+ }
+ return parseIndexContent(dir, string(content))
+}
+
+func parseIndexContent(dir, content string) (IndexFile, error) {
+ idx := IndexFile{Dir: dir}
+ disabledSet := make(map[int]struct{})
+ activeSet := make(map[int]struct{})
+
+ scanner := bufio.NewScanner(strings.NewReader(content))
+ inComment := false
+ inGroupSection := false
+
+ for scanner.Scan() {
+ line := scanner.Text()
+
+ if strings.HasPrefix(strings.TrimSpace(line), "## Group Files") {
+ inGroupSection = true
+ continue
+ }
+
+ if inGroupSection && strings.HasPrefix(strings.TrimSpace(line), "## ") {
+ break
+ }
+
+ if !inGroupSection {
+ continue
+ }
+
+ if htmlCommentStart.MatchString(line) && !htmlCommentEnd.MatchString(line) {
+ inComment = true
+ if m := listItemRegex.FindStringSubmatch(stripCommentStart(line)); m != nil {
+ n, _ := strconv.Atoi(m[1])
+ disabledSet[n] = struct{}{}
+ }
+ continue
+ }
+
+ if inComment {
+ if m := listItemRegex.FindStringSubmatch(line); m != nil {
+ n, _ := strconv.Atoi(m[1])
+ disabledSet[n] = struct{}{}
+ }
+ if htmlCommentEnd.MatchString(line) {
+ inComment = false
+ }
+ continue
+ }
+
+ if m := commentedListItem.FindStringSubmatch(line); m != nil {
+ n, _ := strconv.Atoi(m[1])
+ disabledSet[n] = struct{}{}
+ continue
+ }
+
+ if m := listItemRegex.FindStringSubmatch(line); m != nil {
+ n, _ := strconv.Atoi(m[1])
+ activeSet[n] = struct{}{}
+ }
+ }
+
+ for n := range activeSet {
+ idx.Active = append(idx.Active, n)
+ }
+ sort.Ints(idx.Active)
+
+ for n := range disabledSet {
+ idx.Disabled = append(idx.Disabled, n)
+ }
+ sort.Ints(idx.Disabled)
+
+ return idx, nil
+}
+
+func stripCommentStart(line string) string {
+ idx := strings.Index(line, "<!--")
+ if idx >= 0 {
+ return line[idx+4:]
+ }
+ return line
+}
+
+func ScanGroupFiles(dir string) ([]int, error) {
+ entries, err := os.ReadDir(dir)
+ if err != nil {
+ return nil, err
+ }
+ var groups []int
+ for _, e := range entries {
+ if e.IsDir() {
+ continue
+ }
+ if m := groupFileRegex.FindStringSubmatch(e.Name()); m != nil {
+ n, _ := strconv.Atoi(m[1])
+ groups = append(groups, n)
+ }
+ }
+ sort.Ints(groups)
+ return groups, nil
+}
+
+func (idx IndexFile) IsDisabled(group int) bool {
+ for _, d := range idx.Disabled {
+ if d == group {
+ return true
+ }
+ }
+ return false
+}
diff --git a/tests/benchmark/cmd/api-runner/index_test.go b/tests/benchmark/cmd/api-runner/index_test.go
new file mode 100644
index 00000000..a932090e
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/index_test.go
@@ -0,0 +1,169 @@
+package main
+
+import (
+ "bytes"
+ "os"
+ "path/filepath"
+ "reflect"
+ "testing"
+)
+
+func TestParseIndexAllActive(t *testing.T) {
+ content := `# Benchmark Run
+
+## Group Files
+
+- group-00.md — Setup
+- group-01.md — Reading
+- group-02.md — Search
+`
+ idx, err := parseIndexContent("/fake/dir", content)
+ if err != nil {
+ t.Fatal(err)
+ }
+ want := []int{0, 1, 2}
+ if !reflect.DeepEqual(idx.Active, want) {
+ t.Errorf("Active = %v; want %v", idx.Active, want)
+ }
+ if len(idx.Disabled) != 0 {
+ t.Errorf("Disabled = %v; want empty", idx.Disabled)
+ }
+}
+
+func TestParseIndexSingleLineComment(t *testing.T) {
+ content := `## Group Files
+
+- group-00.md — Setup
+<!-- - group-01.md — Reading (disabled) -->
+- group-02.md — Search
+`
+ idx, err := parseIndexContent("/fake/dir", content)
+ if err != nil {
+ t.Fatal(err)
+ }
+ wantActive := []int{0, 2}
+ wantDisabled := []int{1}
+ if !reflect.DeepEqual(idx.Active, wantActive) {
+ t.Errorf("Active = %v; want %v", idx.Active, wantActive)
+ }
+ if !reflect.DeepEqual(idx.Disabled, wantDisabled) {
+ t.Errorf("Disabled = %v; want %v", idx.Disabled, wantDisabled)
+ }
+}
+
+func TestParseIndexMultiLineComment(t *testing.T) {
+ content := `## Group Files
+
+- group-00.md — Setup
+<!--
+- group-01.md — Reading (disabled)
+- group-02.md — Search (disabled)
+-->
+- group-03.md — Form
+`
+ idx, err := parseIndexContent("/fake/dir", content)
+ if err != nil {
+ t.Fatal(err)
+ }
+ wantActive := []int{0, 3}
+ wantDisabled := []int{1, 2}
+ if !reflect.DeepEqual(idx.Active, wantActive) {
+ t.Errorf("Active = %v; want %v", idx.Active, wantActive)
+ }
+ if !reflect.DeepEqual(idx.Disabled, wantDisabled) {
+ t.Errorf("Disabled = %v; want %v", idx.Disabled, wantDisabled)
+ }
+}
+
+func TestParseIndexEmptyIndex(t *testing.T) {
+ content := `# Empty Index
+
+No group section here.
+`
+ idx, err := parseIndexContent("/fake/dir", content)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(idx.Active) != 0 || len(idx.Disabled) != 0 {
+ t.Errorf("Expected empty sets; got Active=%v Disabled=%v", idx.Active, idx.Disabled)
+ }
+}
+
+func TestParseIndexBacktickFilenames(t *testing.T) {
+ content := `## Group Files
+
+- ` + "`group-00.md`" + ` — Setup
+<!-- - ` + "`group-01.md`" + ` — disabled -->
+`
+ idx, err := parseIndexContent("/fake/dir", content)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !reflect.DeepEqual(idx.Active, []int{0}) {
+ t.Errorf("Active = %v; want [0]", idx.Active)
+ }
+ if !reflect.DeepEqual(idx.Disabled, []int{1}) {
+ t.Errorf("Disabled = %v; want [1]", idx.Disabled)
+ }
+}
+
+func TestScanGroupFiles(t *testing.T) {
+ dir := t.TempDir()
+ for _, name := range []string{"group-00.md", "group-05.md", "group-10.md", "setup.md"} {
+ if err := os.WriteFile(filepath.Join(dir, name), []byte("test"), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ }
+ groups, err := ScanGroupFiles(dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ want := []int{0, 5, 10}
+ if !reflect.DeepEqual(groups, want) {
+ t.Errorf("ScanGroupFiles = %v; want %v", groups, want)
+ }
+}
+
+func TestSelectGroupsExplicit(t *testing.T) {
+ idx := IndexFile{Dir: "/d", Active: []int{0, 1, 2}, Disabled: []int{3}}
+ args := Args{Groups: []int{1, 3}}
+ var stderr bytes.Buffer
+ sel, err := SelectGroups(args, idx, &stderr)
+ if err != nil {
+ t.Fatal(err)
+ }
+ want := []int{1, 3}
+ if !reflect.DeepEqual(sel.Selected, want) {
+ t.Errorf("Selected = %v; want %v", sel.Selected, want)
+ }
+ if len(sel.Warnings) != 1 {
+ t.Errorf("Expected 1 warning; got %d", len(sel.Warnings))
+ }
+}
+
+func TestSelectGroupsProfile(t *testing.T) {
+ idx := IndexFile{Dir: "/d", Active: []int{0, 1, 2, 3, 4, 5}}
+ args := Args{Profile: "common10"}
+ var stderr bytes.Buffer
+ sel, err := SelectGroups(args, idx, &stderr)
+ if err != nil {
+ t.Fatal(err)
+ }
+ want := []int{0, 1, 2, 3}
+ if !reflect.DeepEqual(sel.Selected, want) {
+ t.Errorf("Selected = %v; want %v", sel.Selected, want)
+ }
+}
+
+func TestSelectGroupsActiveOnly(t *testing.T) {
+ idx := IndexFile{Dir: "/d", Active: []int{0, 2, 4}}
+ args := Args{}
+ var stderr bytes.Buffer
+ sel, err := SelectGroups(args, idx, &stderr)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !reflect.DeepEqual(sel.Selected, idx.Active) {
+ t.Errorf("Selected = %v; want %v", sel.Selected, idx.Active)
+ }
+}
diff --git a/tests/benchmark/cmd/api-runner/loop.go b/tests/benchmark/cmd/api-runner/loop.go
new file mode 100644
index 00000000..e72d7c69
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/loop.go
@@ -0,0 +1,220 @@
+package main
+
+import (
+ "encoding/json"
+ "fmt"
+ "io"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "time"
+)
+
+type LoopConfig struct {
+ Lane Lane
+ Provider string
+ Model string
+ Groups []int
+ ReportFile string
+ MaxTurns int
+ MaxIdleTurns int
+ TimeoutSeconds int
+ TurnDelayMs int
+ Finalize bool
+ SkipInit bool
+ BenchDir string
+ CommandLogFile string
+ MaxInputTokens int
+ MaxOutputTokens int
+ Stdout io.Writer
+ Stderr io.Writer
+}
+
+type LoopResult struct {
+ ExitCode int
+ FinalText string
+ Usage UsageCounters
+}
+
+func RunLoop(cfg LoopConfig, runner Runner, shell *PersistentShell) LoopResult {
+ promptCfg := PromptConfig{
+ RepoRoot: filepath.Dir(filepath.Dir(cfg.BenchDir)),
+ BenchDir: cfg.BenchDir,
+ BenchRunDir: filepath.Join(cfg.BenchDir, "benchmark-run"),
+ SetupPTFile: filepath.Join(cfg.BenchDir, "setup-pinchtab.md"),
+ SetupABFile: filepath.Join(cfg.BenchDir, "setup-agent-browser.md"),
+ }
+
+ userPrompt := LaneUserPrompt(cfg.Lane, promptCfg, cfg.ReportFile, cfg.Groups)
+ conversation := runner.InitialConversation(userPrompt)
+ systemPromptText := SystemPrompt()
+
+ var finalText string
+ exitCode := 0
+ idleTurns := 0
+ lastAnswered := ReadProgress(cfg.ReportFile).Answered
+
+ clearCommandLog(cfg.CommandLogFile)
+
+ for turn := 1; turn <= cfg.MaxTurns; turn++ {
+ if turn > 1 && cfg.TurnDelayMs > 0 {
+ time.Sleep(time.Duration(cfg.TurnDelayMs) * time.Millisecond)
+ }
+
+ usage := runner.Usage()
+ if cfg.MaxInputTokens > 0 && usage.InputTokens+usage.CacheCreationInputTokens+usage.CacheReadInputTokens >= cfg.MaxInputTokens {
+ finalText = fmt.Sprintf("Budget exceeded: input tokens (%d) >= limit (%d)", usage.InputTokens+usage.CacheCreationInputTokens+usage.CacheReadInputTokens, cfg.MaxInputTokens)
+ exitCode = 4
+ break
+ }
+ if cfg.MaxOutputTokens > 0 && usage.OutputTokens >= cfg.MaxOutputTokens {
+ finalText = fmt.Sprintf("Budget exceeded: output tokens (%d) >= limit (%d)", usage.OutputTokens, cfg.MaxOutputTokens)
+ exitCode = 4
+ break
+ }
+
+ response, err := runner.Send(systemPromptText, conversation)
+ if err != nil {
+ finalText = fmt.Sprintf("API error: %v", err)
+ exitCode = 1
+ break
+ }
+
+ toolCalls := runner.ExtractToolCalls(response, time.Duration(cfg.TimeoutSeconds)*time.Second)
+ if len(toolCalls) > 0 {
+ results := executeToolCalls(shell, toolCalls, cfg.CommandLogFile)
+ conversation = runner.AppendToolResults(conversation, response, results)
+
+ summary := BuildProgressSummary(cfg.ReportFile)
+ conversation = CompactConversation(cfg.Provider, conversation, summary)
+
+ progress := ReadProgress(cfg.ReportFile)
+ if progress.Answered > lastAnswered {
+ idleTurns = 0
+ lastAnswered = progress.Answered
+ } else {
+ idleTurns++
+ }
+
+ if idleTurns >= cfg.MaxIdleTurns {
+ finalText = fmt.Sprintf("Stopped after %d consecutive turns without recording a benchmark step. Check %s for the command trace.",
+ idleTurns, cfg.CommandLogFile)
+ exitCode = 3
+ break
+ }
+ continue
+ }
+
+ finalText = runner.ExtractFinalText(response)
+ break
+ }
+
+ if finalText == "" {
+ finalText = fmt.Sprintf("Stopped after reaching max turns (%d).", cfg.MaxTurns)
+ exitCode = 2
+ }
+
+ if _, err := os.Stat(cfg.ReportFile); err == nil {
+ recordUsage(cfg.BenchDir, cfg.ReportFile, runner)
+ if cfg.Finalize {
+ finalizeReport(cfg.BenchDir, cfg.ReportFile)
+ }
+ }
+
+ if finalText != "" {
+ fmt.Fprintln(cfg.Stdout, finalText)
+ }
+
+ usage := runner.Usage()
+ fmt.Fprintf(cfg.Stdout,
+ "\n[run-usage] provider=%s requests=%d input=%d cache_create=%d cache_read=%d output=%d total=%d\n",
+ runner.Provider(),
+ usage.RequestCount,
+ usage.InputTokens,
+ usage.CacheCreationInputTokens,
+ usage.CacheReadInputTokens,
+ usage.OutputTokens,
+ usage.InputTokens+usage.CacheCreationInputTokens+usage.CacheReadInputTokens+usage.OutputTokens,
+ )
+
+ return LoopResult{
+ ExitCode: exitCode,
+ FinalText: finalText,
+ Usage: usage,
+ }
+}
+
+func clearCommandLog(path string) {
+ if path != "" {
+ os.WriteFile(path, []byte{}, 0o644)
+ }
+}
+
+func executeToolCalls(shell *PersistentShell, calls []ToolCall, logFile string) []ToolExecutionResult {
+ var results []ToolExecutionResult
+ for _, call := range calls {
+ output, exitCode, err := shell.Run(call.Command, time.Duration(call.TimeoutSeconds)*time.Second)
+ var result ToolExecutionResult
+ if err != nil {
+ result = ToolExecutionResult{
+ ID: call.ID,
+ IsError: true,
+ Content: fmt.Sprintf("$ %s\n[runner_error]\n%s", call.Command, err.Error()),
+ }
+ appendCommandLog(logFile, call.Command, -1, err.Error())
+ } else {
+ trimmed := TrimToolOutput(output)
+ result = ToolExecutionResult{
+ ID: call.ID,
+ IsError: exitCode != 0,
+ Content: FormatToolResult(call.Command, exitCode, output),
+ }
+ appendCommandLog(logFile, call.Command, exitCode, trimmed)
+ }
+ results = append(results, result)
+ }
+ return results
+}
+
+func appendCommandLog(path, command string, exitCode int, output string) {
+ if path == "" {
+ return
+ }
+ entry := map[string]interface{}{
+ "timestamp": time.Now().UTC().Format(time.RFC3339),
+ "command": command,
+ "exit_code": exitCode,
+ "output": output,
+ }
+ data, _ := json.Marshal(entry)
+ f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
+ if err != nil {
+ return
+ }
+ defer f.Close()
+ f.Write(append(data, '\n'))
+}
+
+func recordUsage(benchDir, reportFile string, runner Runner) {
+ usage := runner.Usage()
+ script := filepath.Join(benchDir, "scripts", "record-run-usage.sh")
+ cmd := exec.Command(script,
+ "--report-file", reportFile,
+ "--provider", runner.Provider(),
+ "--source", runner.Source(),
+ "--request-count", fmt.Sprintf("%d", usage.RequestCount),
+ "--input-tokens", fmt.Sprintf("%d", usage.InputTokens),
+ "--output-tokens", fmt.Sprintf("%d", usage.OutputTokens),
+ "--cache-creation-input-tokens", fmt.Sprintf("%d", usage.CacheCreationInputTokens),
+ "--cache-read-input-tokens", fmt.Sprintf("%d", usage.CacheReadInputTokens),
+ )
+ cmd.Dir = benchDir
+ cmd.Run()
+}
+
+func finalizeReport(benchDir, reportFile string) {
+ script := filepath.Join(benchDir, "scripts", "finalize-report.sh")
+ cmd := exec.Command(script, reportFile)
+ cmd.Dir = benchDir
+ cmd.Run()
+}
diff --git a/tests/benchmark/cmd/api-runner/loop_test.go b/tests/benchmark/cmd/api-runner/loop_test.go
new file mode 100644
index 00000000..67860331
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/loop_test.go
@@ -0,0 +1,279 @@
+package main
+
+import (
+ "bytes"
+ "encoding/json"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+)
+
+func TestLoopWithFakeRunner(t *testing.T) {
+ dir := t.TempDir()
+ reportFile := filepath.Join(dir, "report.json")
+ commandLog := filepath.Join(dir, "commands.ndjson")
+
+ os.WriteFile(reportFile, []byte(`{"totals":{"steps_answered":0},"steps":[]}`), 0o644)
+
+ responses := []FakeResponse{
+ {
+ ToolCalls: []ToolCall{
+ {ID: "call_1", Command: "echo hello", TimeoutSeconds: 10},
+ },
+ },
+ {
+ FinalText: "Benchmark complete",
+ },
+ }
+
+ runner := NewFakeRunner("fake-model", responses)
+ shell, err := NewPersistentShell(dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer shell.Close(false)
+
+ var stdout, stderr bytes.Buffer
+ cfg := LoopConfig{
+ Lane: LanePinchtab,
+ Provider: "fake",
+ Model: "fake-model",
+ Groups: []int{0},
+ ReportFile: reportFile,
+ MaxTurns: 10,
+ MaxIdleTurns: 5,
+ TimeoutSeconds: 30,
+ TurnDelayMs: 0,
+ BenchDir: dir,
+ CommandLogFile: commandLog,
+ Stdout: &stdout,
+ Stderr: &stderr,
+ }
+
+ result := RunLoop(cfg, runner, shell)
+
+ if result.ExitCode != 0 {
+ t.Errorf("ExitCode = %d; want 0", result.ExitCode)
+ }
+ if result.FinalText != "Benchmark complete" {
+ t.Errorf("FinalText = %q; want 'Benchmark complete'", result.FinalText)
+ }
+ if result.Usage.RequestCount != 2 {
+ t.Errorf("RequestCount = %d; want 2", result.Usage.RequestCount)
+ }
+
+ logData, err := os.ReadFile(commandLog)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !strings.Contains(string(logData), "echo hello") {
+ t.Error("command log missing 'echo hello'")
+ }
+
+ output := stdout.String()
+ if !strings.Contains(output, "Benchmark complete") {
+ t.Errorf("stdout missing final text: %s", output)
+ }
+ if !strings.Contains(output, "[run-usage]") {
+ t.Errorf("stdout missing usage line: %s", output)
+ }
+}
+
+func TestLoopIdleTurnLimit(t *testing.T) {
+ dir := t.TempDir()
+ reportFile := filepath.Join(dir, "report.json")
+ commandLog := filepath.Join(dir, "commands.ndjson")
+
+ os.WriteFile(reportFile, []byte(`{"totals":{"steps_answered":0},"steps":[]}`), 0o644)
+
+ responses := make([]FakeResponse, 10)
+ for i := range responses {
+ responses[i] = FakeResponse{
+ ToolCalls: []ToolCall{
+ {ID: "call", Command: "echo noop", TimeoutSeconds: 10},
+ },
+ }
+ }
+
+ runner := NewFakeRunner("fake-model", responses)
+ shell, err := NewPersistentShell(dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer shell.Close(false)
+
+ var stdout bytes.Buffer
+ cfg := LoopConfig{
+ Lane: LanePinchtab,
+ Provider: "fake",
+ Model: "fake-model",
+ ReportFile: reportFile,
+ MaxTurns: 20,
+ MaxIdleTurns: 3,
+ TimeoutSeconds: 30,
+ BenchDir: dir,
+ CommandLogFile: commandLog,
+ Stdout: &stdout,
+ Stderr: &bytes.Buffer{},
+ }
+
+ result := RunLoop(cfg, runner, shell)
+
+ if result.ExitCode != 3 {
+ t.Errorf("ExitCode = %d; want 3 (idle limit)", result.ExitCode)
+ }
+ if !strings.Contains(result.FinalText, "consecutive turns") {
+ t.Errorf("FinalText should mention consecutive turns: %s", result.FinalText)
+ }
+}
+
+func TestLoopMaxTurns(t *testing.T) {
+ dir := t.TempDir()
+ reportFile := filepath.Join(dir, "report.json")
+
+ os.WriteFile(reportFile, []byte(`{"totals":{"steps_answered":0},"steps":[]}`), 0o644)
+
+ responses := make([]FakeResponse, 100)
+ for i := range responses {
+ responses[i] = FakeResponse{
+ ToolCalls: []ToolCall{
+ {ID: "call", Command: "echo turn", TimeoutSeconds: 10},
+ },
+ }
+ }
+
+ runner := NewFakeRunner("fake-model", responses)
+ shell, err := NewPersistentShell(dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer shell.Close(false)
+
+ var stdout bytes.Buffer
+ cfg := LoopConfig{
+ Lane: LanePinchtab,
+ Provider: "fake",
+ Model: "fake-model",
+ ReportFile: reportFile,
+ MaxTurns: 3,
+ MaxIdleTurns: 100,
+ TimeoutSeconds: 30,
+ BenchDir: dir,
+ Stdout: &stdout,
+ Stderr: &bytes.Buffer{},
+ }
+
+ result := RunLoop(cfg, runner, shell)
+
+ if result.ExitCode != 2 {
+ t.Errorf("ExitCode = %d; want 2 (max turns)", result.ExitCode)
+ }
+ if !strings.Contains(result.FinalText, "max turns") {
+ t.Errorf("FinalText should mention max turns: %s", result.FinalText)
+ }
+}
+
+func TestAppendCommandLog(t *testing.T) {
+ dir := t.TempDir()
+ logFile := filepath.Join(dir, "log.ndjson")
+
+ appendCommandLog(logFile, "ls -la", 0, "output here")
+ appendCommandLog(logFile, "cat file", 1, "error")
+
+ data, err := os.ReadFile(logFile)
+ if err != nil {
+ t.Fatal(err)
+ }
+
+ lines := strings.Split(strings.TrimSpace(string(data)), "\n")
+ if len(lines) != 2 {
+ t.Fatalf("got %d lines; want 2", len(lines))
+ }
+
+ var entry map[string]interface{}
+ json.Unmarshal([]byte(lines[0]), &entry)
+ if entry["command"] != "ls -la" {
+ t.Errorf("first command = %v; want 'ls -la'", entry["command"])
+ }
+}
+
+func TestLoopBudgetLimit(t *testing.T) {
+ dir := t.TempDir()
+ reportFile := filepath.Join(dir, "report.json")
+
+ os.WriteFile(reportFile, []byte(`{"totals":{"steps_answered":0},"steps":[]}`), 0o644)
+
+ responses := make([]FakeResponse, 10)
+ for i := range responses {
+ responses[i] = FakeResponse{
+ ToolCalls: []ToolCall{
+ {ID: "call", Command: "echo test", TimeoutSeconds: 10},
+ },
+ }
+ }
+
+ runner := NewFakeRunner("fake-model", responses)
+ shell, err := NewPersistentShell(dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer shell.Close(false)
+
+ var stdout bytes.Buffer
+ cfg := LoopConfig{
+ Lane: LanePinchtab,
+ Provider: "fake",
+ Model: "fake-model",
+ ReportFile: reportFile,
+ MaxTurns: 20,
+ MaxIdleTurns: 100,
+ TimeoutSeconds: 30,
+ MaxInputTokens: 150,
+ BenchDir: dir,
+ Stdout: &stdout,
+ Stderr: &bytes.Buffer{},
+ }
+
+ result := RunLoop(cfg, runner, shell)
+
+ if result.ExitCode != 4 {
+ t.Errorf("ExitCode = %d; want 4 (budget exceeded)", result.ExitCode)
+ }
+ if !strings.Contains(result.FinalText, "Budget exceeded") {
+ t.Errorf("FinalText should mention budget: %s", result.FinalText)
+ }
+}
+
+func TestFakeRunner(t *testing.T) {
+ responses := []FakeResponse{
+ {ToolCalls: []ToolCall{{ID: "1", Command: "echo test"}}},
+ {FinalText: "done"},
+ }
+ runner := NewFakeRunner("test-model", responses)
+
+ if runner.Provider() != "fake" {
+ t.Errorf("Provider = %s; want 'fake'", runner.Provider())
+ }
+
+ conv := runner.InitialConversation("hello")
+ if len(conv) != 1 {
+ t.Fatalf("initial conv length = %d; want 1", len(conv))
+ }
+
+ resp1, _ := runner.Send("system", conv)
+ calls := runner.ExtractToolCalls(resp1, 60)
+ if len(calls) != 1 {
+ t.Fatalf("turn 1 calls = %d; want 1", len(calls))
+ }
+
+ resp2, _ := runner.Send("system", conv)
+ text := runner.ExtractFinalText(resp2)
+ if text != "done" {
+ t.Errorf("final text = %q; want 'done'", text)
+ }
+
+ if runner.Usage().RequestCount != 2 {
+ t.Errorf("RequestCount = %d; want 2", runner.Usage().RequestCount)
+ }
+}
diff --git a/tests/benchmark/cmd/api-runner/main.go b/tests/benchmark/cmd/api-runner/main.go
new file mode 100644
index 00000000..c283ae54
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/main.go
@@ -0,0 +1,244 @@
+// Command api-runner is the Go port of
+// tests/benchmark/scripts/run-api-benchmark.ts.
+//
+// Exit codes:
+//
+// 0 success (or --help)
+// 1 argument/setup error
+// 2 max turns reached
+// 3 idle turn limit reached
+// 4 budget exceeded
+package main
+
+import (
+ "errors"
+ "fmt"
+ "io"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "strings"
+)
+
+func main() {
+ os.Exit(run(os.Args[1:], os.Stdout, os.Stderr))
+}
+
+func run(argv []string, stdout, stderr io.Writer) int {
+ args, err := ParseArgs(argv)
+ if err != nil {
+ if errors.Is(err, errHelp) {
+ WriteUsage(stdout)
+ return 0
+ }
+ fmt.Fprintf(stderr, "api-runner: %v\n\n", err)
+ WriteUsage(stderr)
+ return 1
+ }
+
+ if args.DryRun {
+ fmt.Fprint(stdout, formatPlan(args))
+ return 0
+ }
+
+ benchDir := resolveBenchDir()
+ provider := resolveProvider(args)
+ model := resolveModel(provider, args.Model)
+ groups := resolveGroupsFromArgs(args)
+
+ reportFile := args.ReportFile
+ if reportFile == "" {
+ if args.SkipInit {
+ reportFile = resolveReportPath(benchDir, args.Lane)
+ } else {
+ reportFile = initializeLane(benchDir, args, provider, model)
+ }
+ }
+
+ fmt.Fprintf(stdout, "[benchmark-runner] provider=%s model=%s lane=%s groups=%s report=%s\n",
+ provider, model, args.Lane, formatGroups(groups), reportFile)
+
+ runner := createRunner(provider, model, args)
+ shell, err := NewPersistentShell(benchDir)
+ if err != nil {
+ fmt.Fprintf(stderr, "api-runner: shell init failed: %v\n", err)
+ return 1
+ }
+ defer shell.Close(false)
+
+ cfg := LoopConfig{
+ Lane: args.Lane,
+ Provider: provider,
+ Model: model,
+ Groups: groups,
+ ReportFile: reportFile,
+ MaxTurns: args.MaxTurns,
+ MaxIdleTurns: args.MaxIdleTurns,
+ TimeoutSeconds: args.TimeoutSeconds,
+ TurnDelayMs: args.TurnDelayMs,
+ Finalize: args.Finalize,
+ BenchDir: benchDir,
+ CommandLogFile: filepath.Join(benchDir, "results", "agent_commands.ndjson"),
+ MaxInputTokens: args.MaxInputTokens,
+ MaxOutputTokens: args.MaxOutputTokens,
+ Stdout: stdout,
+ Stderr: stderr,
+ }
+
+ result := RunLoop(cfg, runner, shell)
+ return result.ExitCode
+}
+
+func resolveBenchDir() string {
+ cwd, _ := os.Getwd()
+ dir := cwd
+ for {
+ if _, err := os.Stat(filepath.Join(dir, "go.mod")); err == nil {
+ return filepath.Join(dir, "tests", "benchmark")
+ }
+ parent := filepath.Dir(dir)
+ if parent == dir {
+ return filepath.Join(cwd, "tests", "benchmark")
+ }
+ dir = parent
+ }
+}
+
+func resolveProvider(args Args) string {
+ if args.Provider != ProviderUnset {
+ return string(args.Provider)
+ }
+ hasOpenAI := strings.TrimSpace(os.Getenv("OPENAI_API_KEY")) != ""
+ hasAnthropic := strings.TrimSpace(os.Getenv("ANTHROPIC_API_KEY")) != ""
+ if hasOpenAI && hasAnthropic {
+ fmt.Fprintln(os.Stderr, "warning: multiple providers configured, defaulting to anthropic")
+ return "anthropic"
+ }
+ if hasOpenAI {
+ return "openai"
+ }
+ if hasAnthropic {
+ return "anthropic"
+ }
+ return "anthropic"
+}
+
+func resolveModel(provider, explicit string) string {
+ if explicit != "" {
+ return explicit
+ }
+ if provider == "openai" {
+ if m := os.Getenv("OPENAI_MODEL"); m != "" {
+ return m
+ }
+ return "gpt-5"
+ }
+ if m := os.Getenv("ANTHROPIC_MODEL"); m != "" {
+ return m
+ }
+ return "claude-haiku-4-5-20251001"
+}
+
+func resolveGroupsFromArgs(args Args) []int {
+ if len(args.Groups) > 0 {
+ return args.Groups
+ }
+ switch args.Profile {
+ case "common10":
+ return []int{0, 1, 2, 3}
+ default:
+ return nil
+ }
+}
+
+func resolveReportPath(benchDir string, lane Lane) string {
+ ptrFile := filepath.Join(benchDir, "results", "current_agent_report.txt")
+ if lane == LaneAgentBrowser {
+ ptrFile = filepath.Join(benchDir, "results", "current_agent_browser_report.txt")
+ }
+ data, err := os.ReadFile(ptrFile)
+ if err != nil {
+ return ""
+ }
+ return strings.TrimSpace(string(data))
+}
+
+func initializeLane(benchDir string, args Args, provider, model string) string {
+ var script string
+ if args.Lane == LanePinchtab {
+ script = filepath.Join(benchDir, "scripts", "run-optimization.sh")
+ } else {
+ script = filepath.Join(benchDir, "scripts", "run-agent-browser-benchmark.sh")
+ }
+ cmd := exec.Command(script)
+ cmd.Dir = benchDir
+ cmd.Env = append(os.Environ(),
+ "BENCHMARK_MODEL="+model,
+ "BENCHMARK_RUNNER="+runnerSource(provider),
+ )
+ cmd.Stdout = os.Stdout
+ cmd.Stderr = os.Stderr
+ cmd.Run()
+ return resolveReportPath(benchDir, args.Lane)
+}
+
+func runnerSource(provider string) string {
+ if provider == "openai" {
+ return "openai-responses"
+ }
+ return "anthropic-messages"
+}
+
+func createRunner(provider, model string, args Args) Runner {
+ promptCaching := !args.NoPromptCaching
+ if provider == "openai" {
+ apiKey := os.Getenv("OPENAI_API_KEY")
+ return NewOpenAIRunner(apiKey, model, args.MaxTokens, args.Temperature, promptCaching)
+ }
+ if provider == "fake" {
+ return NewFakeRunner(model, nil)
+ }
+ apiKey := os.Getenv("ANTHROPIC_API_KEY")
+ return NewAnthropicRunner(apiKey, model, args.MaxTokens, args.Temperature, promptCaching)
+}
+
+func formatPlan(a Args) string {
+ var b strings.Builder
+ b.WriteString("api-runner (Go) — resolved plan\n")
+ fmt.Fprintf(&b, " lane: %s\n", a.Lane)
+ fmt.Fprintf(&b, " provider: %s\n", stringOr(string(a.Provider), "(auto-detect from env)"))
+ fmt.Fprintf(&b, " model: %s\n", stringOr(a.Model, "(provider default)"))
+ fmt.Fprintf(&b, " groups: %s\n", formatGroups(a.Groups))
+ fmt.Fprintf(&b, " profile: %s\n", stringOr(a.Profile, "(none)"))
+ fmt.Fprintf(&b, " max-tokens: %d\n", a.MaxTokens)
+ fmt.Fprintf(&b, " temperature: %g\n", a.Temperature)
+ fmt.Fprintf(&b, " max-turns: %d\n", a.MaxTurns)
+ fmt.Fprintf(&b, " max-idle-turns: %d\n", a.MaxIdleTurns)
+ fmt.Fprintf(&b, " timeout-seconds: %d\n", a.TimeoutSeconds)
+ fmt.Fprintf(&b, " turn-delay-ms: %d\n", a.TurnDelayMs)
+ fmt.Fprintf(&b, " report-file: %s\n", stringOr(a.ReportFile, "(auto-generated)"))
+ fmt.Fprintf(&b, " skip-init: %t\n", a.SkipInit)
+ fmt.Fprintf(&b, " no-prompt-caching: %t\n", a.NoPromptCaching)
+ fmt.Fprintf(&b, " finalize: %t\n", a.Finalize)
+ fmt.Fprintf(&b, " dry-run: %t\n", a.DryRun)
+ fmt.Fprintf(&b, " index-file: %s\n", stringOr(a.IndexFile, "(default)"))
+ return b.String()
+}
+
+func stringOr(v, fallback string) string {
+ if v == "" {
+ return fallback
+ }
+ return v
+}
+
+func formatGroups(groups []int) string {
+ if len(groups) == 0 {
+ return "all"
+ }
+ parts := make([]string, len(groups))
+ for i, g := range groups {
+ parts[i] = fmt.Sprintf("%d", g)
+ }
+ return strings.Join(parts, ",")
+}
diff --git a/tests/benchmark/cmd/api-runner/openai.go b/tests/benchmark/cmd/api-runner/openai.go
new file mode 100644
index 00000000..08d07114
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/openai.go
@@ -0,0 +1,215 @@
+package main
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "fmt"
+ "io"
+ "net/http"
+ "time"
+)
+
+const OpenAIResponsesURL = "https://api.openai.com/v1/responses"
+
+type OpenAIRunner struct {
+ baseURL string
+ apiKey string
+ model string
+ maxTokens int
+ temperature float64
+ promptCaching bool
+ usage UsageCounters
+ client *http.Client
+ retryCfg RetryConfig
+}
+
+func NewOpenAIRunner(apiKey, model string, maxTokens int, temperature float64, promptCaching bool) *OpenAIRunner {
+ return &OpenAIRunner{
+ baseURL: OpenAIResponsesURL,
+ apiKey: apiKey,
+ model: model,
+ maxTokens: maxTokens,
+ temperature: temperature,
+ promptCaching: promptCaching,
+ client: &http.Client{Timeout: 5 * time.Minute},
+ retryCfg: DefaultRetryConfig(),
+ }
+}
+
+func (r *OpenAIRunner) Provider() string { return "openai" }
+func (r *OpenAIRunner) Source() string { return "openai-responses" }
+func (r *OpenAIRunner) Model() string { return r.model }
+func (r *OpenAIRunner) Usage() UsageCounters { return r.usage }
+
+func (r *OpenAIRunner) ToolDefinitions() interface{} {
+ return []map[string]interface{}{
+ {
+ "type": "function",
+ "name": "run_command",
+ "description": "Run a shell command in a persistent bash session rooted at tests/benchmark. Environment changes such as cd/export persist across calls.",
+ "parameters": map[string]interface{}{
+ "type": "object",
+ "properties": map[string]interface{}{
+ "command": map[string]interface{}{"type": "string"},
+ "timeout_seconds": map[string]interface{}{"type": "integer", "minimum": 1, "maximum": 600},
+ },
+ "required": []string{"command"},
+ },
+ },
+ }
+}
+
+func (r *OpenAIRunner) InitialConversation(userPrompt string) []interface{} {
+ return []interface{}{
+ map[string]interface{}{
+ "role": "user",
+ "content": []interface{}{
+ map[string]interface{}{"type": "input_text", "text": userPrompt},
+ },
+ },
+ }
+}
+
+func (r *OpenAIRunner) Send(systemPrompt string, conversation []interface{}) (interface{}, error) {
+ body := map[string]interface{}{
+ "model": r.model,
+ "instructions": systemPrompt,
+ "tools": r.ToolDefinitions(),
+ "input": conversation,
+ "max_output_tokens": r.maxTokens,
+ }
+ if r.promptCaching {
+ body["prompt_cache_retention"] = "24h"
+ }
+
+ payload, err := json.Marshal(body)
+ if err != nil {
+ return nil, fmt.Errorf("marshal request: %w", err)
+ }
+
+ var result map[string]interface{}
+ resp, err := DoWithRetry(context.Background(), r.retryCfg, func() (*http.Response, error) {
+ req, err := http.NewRequest("POST", r.baseURL, bytes.NewReader(payload))
+ if err != nil {
+ return nil, err
+ }
+ req.Header.Set("Content-Type", "application/json")
+ req.Header.Set("Authorization", "Bearer "+r.apiKey)
+ return r.client.Do(req)
+ })
+ if err != nil {
+ return nil, err
+ }
+ defer resp.Body.Close()
+
+ respBody, err := io.ReadAll(resp.Body)
+ if err != nil {
+ return nil, fmt.Errorf("read response: %w", err)
+ }
+
+ if err := json.Unmarshal(respBody, &result); err != nil {
+ return nil, fmt.Errorf("unmarshal response: %w", err)
+ }
+
+ if resp.StatusCode >= 400 {
+ return nil, fmt.Errorf("OpenAI API error %d: %s", resp.StatusCode, string(respBody))
+ }
+
+ r.updateUsage(result)
+ return result, nil
+}
+
+func (r *OpenAIRunner) updateUsage(result map[string]interface{}) {
+ usage, _ := result["usage"].(map[string]interface{})
+ if usage == nil {
+ return
+ }
+ r.usage.RequestCount++
+
+ totalInput := toInt(usage["input_tokens"])
+ cached := 0
+ if details, ok := usage["input_tokens_details"].(map[string]interface{}); ok {
+ cached = toInt(details["cached_tokens"])
+ }
+ r.usage.InputTokens += max(0, totalInput-cached)
+ r.usage.OutputTokens += toInt(usage["output_tokens"])
+ r.usage.CacheReadInputTokens += cached
+}
+
+func (r *OpenAIRunner) ExtractToolCalls(response interface{}, defaultTimeout time.Duration) []ToolCall {
+ resp, _ := response.(map[string]interface{})
+ output, _ := resp["output"].([]interface{})
+ var calls []ToolCall
+ for _, item := range output {
+ m, _ := item.(map[string]interface{})
+ if m["type"] != "function_call" {
+ continue
+ }
+ argsStr, _ := m["arguments"].(string)
+ var args map[string]interface{}
+ if argsStr != "" {
+ json.Unmarshal([]byte(argsStr), &args)
+ }
+ cmd, _ := args["command"].(string)
+ if cmd == "" {
+ continue
+ }
+ timeout := int(defaultTimeout.Seconds())
+ if t := toInt(args["timeout_seconds"]); t > 0 {
+ timeout = t
+ }
+ callID := fmt.Sprintf("%v", m["call_id"])
+ if callID == "" || callID == "<nil>" {
+ callID = fmt.Sprintf("%v", m["id"])
+ }
+ calls = append(calls, ToolCall{
+ ID: callID,
+ Command: cmd,
+ TimeoutSeconds: timeout,
+ })
+ }
+ return calls
+}
+
+func (r *OpenAIRunner) AppendToolResults(conversation []interface{}, response interface{}, results []ToolExecutionResult) []interface{} {
+ resp, _ := response.(map[string]interface{})
+ output, _ := resp["output"].([]interface{})
+ conversation = append(conversation, output...)
+ for _, res := range results {
+ conversation = append(conversation, map[string]interface{}{
+ "type": "function_call_output",
+ "call_id": res.ID,
+ "output": res.Content,
+ })
+ }
+ return conversation
+}
+
+func (r *OpenAIRunner) ExtractFinalText(response interface{}) string {
+ resp, _ := response.(map[string]interface{})
+
+ if text, ok := resp["output_text"].(string); ok && text != "" {
+ return text
+ }
+
+ output, _ := resp["output"].([]interface{})
+ var texts []string
+ for _, item := range output {
+ m, _ := item.(map[string]interface{})
+ if m["type"] != "message" {
+ continue
+ }
+ content, _ := m["content"].([]interface{})
+ for _, part := range content {
+ p, _ := part.(map[string]interface{})
+ ptype, _ := p["type"].(string)
+ if ptype == "output_text" || ptype == "text" {
+ if text, ok := p["text"].(string); ok && text != "" {
+ texts = append(texts, text)
+ }
+ }
+ }
+ }
+ return joinStrings(texts, "\n")
+}
diff --git a/tests/benchmark/cmd/api-runner/openai_test.go b/tests/benchmark/cmd/api-runner/openai_test.go
new file mode 100644
index 00000000..99fb9de2
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/openai_test.go
@@ -0,0 +1,196 @@
+package main
+
+import (
+ "encoding/json"
+ "io"
+ "net/http"
+ "net/http/httptest"
+ "testing"
+ "time"
+)
+
+func TestOpenAIToolDefinitions(t *testing.T) {
+ r := NewOpenAIRunner("key", "gpt-5-mini", 4096, 0, true)
+ tools := r.ToolDefinitions().([]map[string]interface{})
+ if len(tools) != 1 {
+ t.Fatalf("got %d tools; want 1", len(tools))
+ }
+ if tools[0]["name"] != "run_command" {
+ t.Errorf("tool name = %v; want 'run_command'", tools[0]["name"])
+ }
+ if tools[0]["type"] != "function" {
+ t.Errorf("tool type = %v; want 'function'", tools[0]["type"])
+ }
+}
+
+func TestOpenAIInitialConversation(t *testing.T) {
+ r := NewOpenAIRunner("key", "gpt-5-mini", 4096, 0, true)
+ conv := r.InitialConversation("test prompt")
+ if len(conv) != 1 {
+ t.Fatalf("got %d messages; want 1", len(conv))
+ }
+ msg := conv[0].(map[string]interface{})
+ if msg["role"] != "user" {
+ t.Errorf("role = %v; want 'user'", msg["role"])
+ }
+ content := msg["content"].([]interface{})
+ if len(content) != 1 {
+ t.Fatalf("got %d content parts; want 1", len(content))
+ }
+ part := content[0].(map[string]interface{})
+ if part["type"] != "input_text" {
+ t.Errorf("content type = %v; want 'input_text'", part["type"])
+ }
+}
+
+func TestOpenAISend(t *testing.T) {
+ var receivedBody map[string]interface{}
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ if r.Method != "POST" {
+ t.Errorf("method = %s; want POST", r.Method)
+ }
+ auth := r.Header.Get("Authorization")
+ if auth != "Bearer test-key" {
+ t.Errorf("Authorization = %s; want 'Bearer test-key'", auth)
+ }
+
+ body, _ := io.ReadAll(r.Body)
+ json.Unmarshal(body, &receivedBody)
+
+ w.Header().Set("Content-Type", "application/json")
+ json.NewEncoder(w).Encode(map[string]interface{}{
+ "output": []interface{}{
+ map[string]interface{}{"type": "message", "content": []interface{}{
+ map[string]interface{}{"type": "output_text", "text": "Hello"},
+ }},
+ },
+ "usage": map[string]interface{}{
+ "input_tokens": 100,
+ "output_tokens": 50,
+ "input_tokens_details": map[string]interface{}{
+ "cached_tokens": 30,
+ },
+ },
+ })
+ }))
+ defer server.Close()
+
+ r := NewOpenAIRunner("test-key", "gpt-5-mini", 4096, 0.5, true)
+ r.baseURL = server.URL
+ r.retryCfg.Sleep = func(d time.Duration) {}
+
+ conv := r.InitialConversation("hello")
+ _, err := r.Send("system", conv)
+ if err != nil {
+ t.Fatal(err)
+ }
+
+ if receivedBody["model"] != "gpt-5-mini" {
+ t.Errorf("model = %v; want gpt-5-mini", receivedBody["model"])
+ }
+ if receivedBody["max_output_tokens"] != float64(4096) {
+ t.Errorf("max_output_tokens = %v; want 4096", receivedBody["max_output_tokens"])
+ }
+ if receivedBody["prompt_cache_retention"] != "24h" {
+ t.Errorf("prompt_cache_retention = %v; want '24h'", receivedBody["prompt_cache_retention"])
+ }
+
+ usage := r.Usage()
+ if usage.RequestCount != 1 {
+ t.Errorf("RequestCount = %d; want 1", usage.RequestCount)
+ }
+ if usage.InputTokens != 70 {
+ t.Errorf("InputTokens = %d; want 70 (100 - 30 cached)", usage.InputTokens)
+ }
+ if usage.CacheReadInputTokens != 30 {
+ t.Errorf("CacheReadInputTokens = %d; want 30", usage.CacheReadInputTokens)
+ }
+}
+
+func TestOpenAIExtractToolCalls(t *testing.T) {
+ r := NewOpenAIRunner("key", "model", 4096, 0, true)
+ response := map[string]interface{}{
+ "output": []interface{}{
+ map[string]interface{}{
+ "type": "function_call",
+ "call_id": "call_123",
+ "arguments": `{"command": "ls -la", "timeout_seconds": 30}`,
+ },
+ map[string]interface{}{
+ "type": "message",
+ },
+ },
+ }
+
+ calls := r.ExtractToolCalls(response, 120*time.Second)
+ if len(calls) != 1 {
+ t.Fatalf("got %d calls; want 1", len(calls))
+ }
+ if calls[0].ID != "call_123" {
+ t.Errorf("ID = %s; want call_123", calls[0].ID)
+ }
+ if calls[0].Command != "ls -la" {
+ t.Errorf("Command = %s; want 'ls -la'", calls[0].Command)
+ }
+ if calls[0].TimeoutSeconds != 30 {
+ t.Errorf("TimeoutSeconds = %d; want 30", calls[0].TimeoutSeconds)
+ }
+}
+
+func TestOpenAIExtractFinalText(t *testing.T) {
+ r := NewOpenAIRunner("key", "model", 4096, 0, true)
+
+ t.Run("output_text field", func(t *testing.T) {
+ response := map[string]interface{}{
+ "output_text": "Direct output",
+ }
+ text := r.ExtractFinalText(response)
+ if text != "Direct output" {
+ t.Errorf("got %q; want 'Direct output'", text)
+ }
+ })
+
+ t.Run("from output array", func(t *testing.T) {
+ response := map[string]interface{}{
+ "output": []interface{}{
+ map[string]interface{}{
+ "type": "message",
+ "content": []interface{}{
+ map[string]interface{}{"type": "output_text", "text": "Hello"},
+ map[string]interface{}{"type": "text", "text": "World"},
+ },
+ },
+ },
+ }
+ text := r.ExtractFinalText(response)
+ if text != "Hello\nWorld" {
+ t.Errorf("got %q; want 'Hello\\nWorld'", text)
+ }
+ })
+}
+
+func TestOpenAIAppendToolResults(t *testing.T) {
+ r := NewOpenAIRunner("key", "model", 4096, 0, true)
+ conv := []interface{}{}
+ response := map[string]interface{}{
+ "output": []interface{}{
+ map[string]interface{}{"type": "function_call", "call_id": "123"},
+ },
+ }
+ results := []ToolExecutionResult{
+ {ID: "123", IsError: false, Content: "output"},
+ }
+
+ conv = r.AppendToolResults(conv, response, results)
+ if len(conv) != 2 {
+ t.Fatalf("got %d items; want 2", len(conv))
+ }
+
+ funcOutput := conv[1].(map[string]interface{})
+ if funcOutput["type"] != "function_call_output" {
+ t.Errorf("type = %v; want 'function_call_output'", funcOutput["type"])
+ }
+ if funcOutput["call_id"] != "123" {
+ t.Errorf("call_id = %v; want '123'", funcOutput["call_id"])
+ }
+}
diff --git a/tests/benchmark/cmd/api-runner/prompt.go b/tests/benchmark/cmd/api-runner/prompt.go
new file mode 100644
index 00000000..b1f6b20c
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/prompt.go
@@ -0,0 +1,293 @@
+package main
+
+import (
+ "fmt"
+ "os"
+ "path/filepath"
+ "strings"
+)
+
+type PromptConfig struct {
+ RepoRoot string
+ BenchDir string
+ BenchRunDir string
+ SetupPTFile string
+ SetupABFile string
+}
+
+func DefaultPromptConfig() PromptConfig {
+ cwd, _ := os.Getwd()
+ repoRoot := findRepoRoot(cwd)
+ benchDir := filepath.Join(repoRoot, "tests", "benchmark")
+ return PromptConfig{
+ RepoRoot: repoRoot,
+ BenchDir: benchDir,
+ BenchRunDir: filepath.Join(benchDir, "benchmark-run"),
+ SetupPTFile: filepath.Join(benchDir, "setup-pinchtab.md"),
+ SetupABFile: filepath.Join(benchDir, "setup-agent-browser.md"),
+ }
+}
+
+func findRepoRoot(start string) string {
+ dir := start
+ for {
+ if _, err := os.Stat(filepath.Join(dir, "go.mod")); err == nil {
+ return dir
+ }
+ parent := filepath.Dir(dir)
+ if parent == dir {
+ return start
+ }
+ dir = parent
+ }
+}
+
+type LaneConfig struct {
+ Label string
+ SkillInstruction string
+ Wrapper string
+ RecordType string
+ WorkflowSummary []string
+ AdapterNotes []string
+ BootstrapCommands []string
+}
+
+func LanePromptConfig(lane Lane, cfg PromptConfig) LaneConfig {
+ if lane == LanePinchtab {
+ return LaneConfig{
+ Label: "PinchTab agent",
+ SkillInstruction: fmt.Sprintf("Read %s exactly once before acting.", cfg.SetupPTFile),
+ Wrapper: "./scripts/pt",
+ RecordType: "pinchtab",
+ WorkflowSummary: []string{
+ "use only ./scripts/pt for browser actions",
+ "keep one shared tab via PINCHTAB_TAB",
+ "use snap -i -c for actionable refs",
+ "use text or text --full for reading content",
+ "refresh refs after any navigation or DOM change",
+ },
+ AdapterNotes: []string{
+ `a navigation-expected click can return "Error 409: unexpected page navigation ..."; treat that as likely success and verify with a fresh snapshot/text read`,
+ "do not assume every command returns JSON; only parse JSON when the command actually returned JSON",
+ `the download endpoint returns JSON with base64 content in "data", not a local file path`,
+ `this environment uses BSD/macOS userland tools; avoid GNU-only flags such as "head -n -1"`,
+ },
+ BootstrapCommands: []string{
+ "./scripts/pt health",
+ "./scripts/pt tab",
+ "export PINCHTAB_TAB=$(./scripts/pt nav http://fixtures/)",
+ "./scripts/pt snap -i -c",
+ },
+ }
+ }
+ return LaneConfig{
+ Label: "agent-browser",
+ SkillInstruction: fmt.Sprintf("Read %s exactly once before acting, then load the official agent-browser skill exactly once with `./scripts/ab skills get agent-browser --full`.", cfg.SetupABFile),
+ Wrapper: "./scripts/ab",
+ RecordType: "agent-browser",
+ WorkflowSummary: []string{
+ "use only ./scripts/ab for browser actions",
+ "keep the shared benchmark session across commands",
+ "use snapshot -i -c for actionable refs",
+ "use fresh refs after any navigation or DOM change",
+ "reuse the current browser state instead of reopening pages unless needed",
+ },
+ AdapterNotes: []string{
+ "after navigation-triggering clicks, verify the resulting page with a fresh snapshot/text read instead of trusting the click response alone",
+ "do not assume every command returns JSON; parse only when the command actually returned JSON",
+ "for downloads, inspect returned content instead of assuming a file already exists locally",
+ `this environment uses BSD/macOS userland tools; avoid GNU-only flags such as "head -n -1"`,
+ },
+ BootstrapCommands: []string{
+ "./scripts/ab skills get agent-browser --full",
+ "./scripts/ab open http://fixtures/",
+ "./scripts/ab snapshot -i -c",
+ },
+ }
+}
+
+func SystemPrompt() string {
+ return `You are a precise benchmark execution agent. Use tools to inspect the repo and run the benchmark lane exactly as instructed.
+
+Rules:
+- Never fabricate command output or task results.
+- Use the shell tool for all file reads and command execution.
+- Do not use destructive commands such as rm -rf, git reset, or checkout.
+- After recording an answer, verify it immediately against the task oracle.
+- Prefer factual command output over long reasoning.`
+}
+
+func LaneSubsetInstructions(groups []int) string {
+ if len(groups) == 0 {
+ return "Execute the full benchmark task set."
+ }
+ parts := make([]string, len(groups))
+ for i, g := range groups {
+ parts[i] = fmt.Sprintf("%d", g)
+ }
+ return fmt.Sprintf(`Execute only these benchmark groups: %s.
+Do not attempt groups outside this subset.
+Treat all other groups as out of scope for this run rather than as failures.
+For the selected groups, execute every step in the group unless blocked.`, strings.Join(parts, ", "))
+}
+
+func readText(path string) string {
+ data, err := os.ReadFile(path)
+ if err != nil {
+ return ""
+ }
+ return strings.TrimSpace(string(data))
+}
+
+func BenchmarkRunGroupFile(dir string, group int) string {
+ return filepath.Join(dir, fmt.Sprintf("group-%02d.md", group))
+}
+
+func BenchmarkRunAllGroups(dir string) []int {
+ entries, err := os.ReadDir(dir)
+ if err != nil {
+ return nil
+ }
+ var groups []int
+ for _, e := range entries {
+ if e.IsDir() {
+ continue
+ }
+ if m := groupFileRegex.FindStringSubmatch(e.Name()); m != nil {
+ var n int
+ fmt.Sscanf(m[1], "%d", &n)
+ groups = append(groups, n)
+ }
+ }
+ return groups
+}
+
+func BenchmarkRunSelectedText(cfg PromptConfig, groups []int) string {
+ indexPath := filepath.Join(cfg.BenchRunDir, "index.md")
+ selected := groups
+ if len(selected) == 0 {
+ selected = BenchmarkRunAllGroups(cfg.BenchRunDir)
+ }
+ chunks := []string{readText(indexPath)}
+ for _, g := range selected {
+ path := BenchmarkRunGroupFile(cfg.BenchRunDir, g)
+ if content := readText(path); content != "" {
+ chunks = append(chunks, content)
+ }
+ }
+ return strings.Join(chunks, "\n\n")
+}
+
+func LaneTaskSourceInstructions(lane Lane, cfg PromptConfig, groups []int) string {
+ indexPath := filepath.Join(cfg.BenchRunDir, "index.md")
+
+ if len(groups) == 0 {
+ return fmt.Sprintf("Use the benchmark run index at %s plus all group files in %s.\n\n%s",
+ indexPath, cfg.BenchRunDir, BenchmarkRunSelectedText(cfg, nil))
+ }
+
+ if lane == LanePinchtab {
+ subset := BenchmarkRunSelectedText(cfg, groups)
+ return fmt.Sprintf("Use only this selected task subset from %s:\n\n%s", cfg.BenchRunDir, subset)
+ }
+
+ var sections []string
+ for _, g := range groups {
+ if g == 0 {
+ sections = append(sections, agentBrowserGroup0())
+ continue
+ }
+ path := BenchmarkRunGroupFile(cfg.BenchRunDir, g)
+ if content := readText(path); content != "" {
+ sections = append(sections, content)
+ }
+ }
+
+ return fmt.Sprintf("Use only this selected task subset for agent-browser: Group 0 from %s, Groups 1+ from %s.\n\n%s\n\n%s",
+ cfg.SetupABFile, cfg.BenchRunDir, readText(indexPath), strings.Join(sections, "\n\n"))
+}
+
+func agentBrowserGroup0() string {
+ return `## Group 0: Setup & Diagnosis (agent-browser lane)
+
+### 0.1 Open fixtures home
+Run ` + "`./scripts/ab open http://fixtures/`" + `.
+
+**Verify**: Open succeeds and the home page loads.
+
+### 0.2 Snapshot interactive refs
+Run ` + "`./scripts/ab snapshot -i -c`" + ` on the home page.
+
+**Verify**: Interactive refs are returned without error.
+
+### 0.3 Session persists across commands
+Use the same wrapper session across multiple commands and confirm browser state persists.
+
+**Verify**: A follow-up action can use the existing page/session state.`
+}
+
+func LaneUserPrompt(lane Lane, cfg PromptConfig, reportFile string, groups []int) string {
+ lc := LanePromptConfig(lane, cfg)
+ firstGroup := 0
+ if len(groups) > 0 {
+ firstGroup = groups[0]
+ }
+
+ var bootstrapLines []string
+ for i, cmd := range lc.BootstrapCommands {
+ bootstrapLines = append(bootstrapLines, fmt.Sprintf("%d. %s", i+1, cmd))
+ }
+ bootstrap := strings.Join(bootstrapLines, "\n")
+
+ taskInstr := LaneTaskSourceInstructions(lane, cfg, groups)
+ taskInstr = strings.ReplaceAll(taskInstr, "\n", "\n ")
+
+ subsetInstr := LaneSubsetInstructions(groups)
+ subsetInstr = strings.ReplaceAll(subsetInstr, "\n", "\n - ")
+
+ return fmt.Sprintf(`Work in this repo: %s
+
+Benchmark lane: %s execution.
+
+Requirements:
+- Follow a linear execution flow: skill once, selected groups, execute, record, verify, continue.
+- Do not read README, browse directories, or inspect unrelated files unless a command path is missing.
+- %s
+- Tool wrapper:
+ - %s
+- Workflow summary:
+ - %s
+- Adapter notes:
+ - %s
+- Task scope:
+ %s
+- For each completed step:
+ 1. record the observed answer with:
+ - ./scripts/record-step.sh --report-file %s --type %s <group> <step> answer "<what you saw>" "notes"
+ 2. immediately verify it with:
+ - ./scripts/verify-step.sh --report-file %s --type %s <group> <step> <pass|fail|skip> "verification notes"
+- If a step cannot be completed, record fail or skip in the same report.
+- Do not leave answered steps pending verification.
+- Keep commands concise. Prefer rg/sed/cat only when you must inspect a specific file.
+- After the skill step, begin actual benchmark execution immediately.
+- Start with this bootstrap command sequence before attempting the selected steps:
+%s
+- After the bootstrap, immediately execute Group %d step 1.
+- Subset selection:
+ - %s
+- Finish when all selected steps are executed or when you are blocked.
+
+Your final answer should briefly summarize completion status and the main blockers, if any.`,
+ cfg.RepoRoot,
+ lc.Label,
+ lc.SkillInstruction,
+ lc.Wrapper,
+ strings.Join(lc.WorkflowSummary, "\n - "),
+ strings.Join(lc.AdapterNotes, "\n - "),
+ taskInstr,
+ reportFile, lc.RecordType,
+ reportFile, lc.RecordType,
+ bootstrap,
+ firstGroup,
+ subsetInstr)
+}
diff --git a/tests/benchmark/cmd/api-runner/prompt_test.go b/tests/benchmark/cmd/api-runner/prompt_test.go
new file mode 100644
index 00000000..2fd4d060
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/prompt_test.go
@@ -0,0 +1,122 @@
+package main
+
+import (
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+)
+
+func TestSystemPromptGolden(t *testing.T) {
+ golden, err := os.ReadFile("testdata/golden/system-prompt.txt")
+ if err != nil {
+ t.Fatalf("read golden: %v", err)
+ }
+ got := SystemPrompt()
+ want := strings.TrimSpace(string(golden))
+ if got != want {
+ t.Errorf("SystemPrompt mismatch:\n--- want ---\n%s\n--- got ---\n%s", want, got)
+ }
+}
+
+func TestLaneSubsetInstructionsFull(t *testing.T) {
+ got := LaneSubsetInstructions(nil)
+ want := "Execute the full benchmark task set."
+ if got != want {
+ t.Errorf("got %q; want %q", got, want)
+ }
+}
+
+func TestLaneSubsetInstructionsSubset(t *testing.T) {
+ got := LaneSubsetInstructions([]int{0, 1, 2})
+ if !strings.Contains(got, "Execute only these benchmark groups: 0, 1, 2.") {
+ t.Errorf("missing group list in %q", got)
+ }
+ if !strings.Contains(got, "Do not attempt groups outside this subset.") {
+ t.Errorf("missing subset constraint in %q", got)
+ }
+}
+
+func TestLanePromptConfigAgent(t *testing.T) {
+ cfg := PromptConfig{
+ RepoRoot: "/repo",
+ BenchDir: "/repo/tests/benchmark",
+ BenchRunDir: "/repo/tests/benchmark/benchmark-run",
+ SetupPTFile: "/repo/tests/benchmark/setup-pinchtab.md",
+ SetupABFile: "/repo/tests/benchmark/setup-agent-browser.md",
+ }
+ lc := LanePromptConfig(LanePinchtab, cfg)
+ if lc.Label != "PinchTab agent" {
+ t.Errorf("Label = %q; want 'PinchTab agent'", lc.Label)
+ }
+ if lc.Wrapper != "./scripts/pt" {
+ t.Errorf("Wrapper = %q; want './scripts/pt'", lc.Wrapper)
+ }
+}
+
+func TestLanePromptConfigAgentBrowser(t *testing.T) {
+ cfg := PromptConfig{
+ RepoRoot: "/repo",
+ BenchDir: "/repo/tests/benchmark",
+ BenchRunDir: "/repo/tests/benchmark/benchmark-run",
+ SetupPTFile: "/repo/tests/benchmark/setup-pinchtab.md",
+ SetupABFile: "/repo/tests/benchmark/setup-agent-browser.md",
+ }
+ lc := LanePromptConfig(LaneAgentBrowser, cfg)
+ if lc.Label != "agent-browser" {
+ t.Errorf("Label = %q; want 'agent-browser'", lc.Label)
+ }
+ if lc.Wrapper != "./scripts/ab" {
+ t.Errorf("Wrapper = %q; want './scripts/ab'", lc.Wrapper)
+ }
+}
+
+func TestLaneUserPromptStructure(t *testing.T) {
+ cfg := DefaultPromptConfig()
+ reportFile := cfg.BenchDir + "/results/test_report.json"
+
+ prompt := LaneUserPrompt(LanePinchtab, cfg, reportFile, nil)
+
+ checks := []string{
+ "Work in this repo:",
+ "Benchmark lane: PinchTab agent execution.",
+ "./scripts/pt",
+ "record-step.sh",
+ "verify-step.sh",
+ "bootstrap command sequence",
+ }
+ for _, check := range checks {
+ if !strings.Contains(prompt, check) {
+ t.Errorf("prompt missing %q", check)
+ }
+ }
+}
+
+func TestBenchmarkRunGroupFile(t *testing.T) {
+ got := BenchmarkRunGroupFile("/dir", 5)
+ want := "/dir/group-05.md"
+ if got != want {
+ t.Errorf("got %q; want %q", got, want)
+ }
+}
+
+func TestBenchmarkRunAllGroups(t *testing.T) {
+ dir := t.TempDir()
+ for _, name := range []string{"group-00.md", "group-01.md", "group-10.md", "index.md"} {
+ os.WriteFile(filepath.Join(dir, name), []byte("test"), 0o644)
+ }
+ groups := BenchmarkRunAllGroups(dir)
+ if len(groups) != 3 {
+ t.Errorf("got %d groups; want 3", len(groups))
+ }
+}
+
+func TestAgentBrowserGroup0Content(t *testing.T) {
+ content := agentBrowserGroup0()
+ if !strings.Contains(content, "Group 0: Setup & Diagnosis (agent-browser lane)") {
+ t.Error("missing group 0 header")
+ }
+ if !strings.Contains(content, "./scripts/ab open http://fixtures/") {
+ t.Error("missing open command")
+ }
+}
diff --git a/tests/benchmark/cmd/api-runner/report.go b/tests/benchmark/cmd/api-runner/report.go
new file mode 100644
index 00000000..40cae2de
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/report.go
@@ -0,0 +1,129 @@
+package main
+
+import (
+ "encoding/json"
+ "fmt"
+ "os"
+ "regexp"
+ "strings"
+)
+
+const (
+ MaxSummarySteps = 12
+ MaxSummaryTextChars = 220
+)
+
+type ReportTotals struct {
+ StepsAnswered int `json:"steps_answered"`
+ StepsFailed int `json:"steps_failed"`
+ StepsSkipped int `json:"steps_skipped"`
+ StepsVerifiedPassed int `json:"steps_verified_passed"`
+ StepsVerifiedFailed int `json:"steps_verified_failed"`
+ StepsVerifiedSkipped int `json:"steps_verified_skipped"`
+ StepsPendingVerification int `json:"steps_pending_verification"`
+}
+
+type ReportStep struct {
+ ID string `json:"id"`
+ Status string `json:"status"`
+ Answer string `json:"answer"`
+ Notes string `json:"notes"`
+ Verification *struct {
+ Status string `json:"status"`
+ } `json:"verification,omitempty"`
+}
+
+type Report struct {
+ Totals ReportTotals `json:"totals"`
+ Steps []ReportStep `json:"steps"`
+}
+
+func ReadReport(path string) (*Report, error) {
+ data, err := os.ReadFile(path)
+ if err != nil {
+ return nil, err
+ }
+ var report Report
+ if err := json.Unmarshal(data, &report); err != nil {
+ return nil, err
+ }
+ return &report, nil
+}
+
+func shorten(text string, maxChars int) string {
+ ws := regexp.MustCompile(`\s+`)
+ normalized := strings.TrimSpace(ws.ReplaceAllString(text, " "))
+ if len(normalized) <= maxChars {
+ return normalized
+ }
+ return normalized[:maxChars-3] + "..."
+}
+
+func BuildProgressSummary(reportFile string) string {
+ report, err := ReadReport(reportFile)
+ if err != nil {
+ return "No benchmark report file exists yet."
+ }
+
+ totals := report.Totals
+ steps := report.Steps
+
+ startIdx := len(steps) - MaxSummarySteps
+ if startIdx < 0 {
+ startIdx = 0
+ }
+ recentSteps := steps[startIdx:]
+
+ var lines []string
+ lines = append(lines, "Benchmark progress summary from the external report.")
+ lines = append(lines, fmt.Sprintf("- answered=%d", totals.StepsAnswered))
+ lines = append(lines, fmt.Sprintf("- execution_failed=%d", totals.StepsFailed))
+ lines = append(lines, fmt.Sprintf("- execution_skipped=%d", totals.StepsSkipped))
+ lines = append(lines, fmt.Sprintf("- verified_passed=%d", totals.StepsVerifiedPassed))
+ lines = append(lines, fmt.Sprintf("- verified_failed=%d", totals.StepsVerifiedFailed))
+ lines = append(lines, fmt.Sprintf("- verified_skipped=%d", totals.StepsVerifiedSkipped))
+ lines = append(lines, fmt.Sprintf("- pending_verification=%d", totals.StepsPendingVerification))
+
+ if len(recentSteps) > 0 {
+ lines = append(lines, "Recent recorded steps:")
+ for _, step := range recentSteps {
+ verification := ""
+ if step.Verification != nil && step.Verification.Status != "" {
+ verification = fmt.Sprintf(" / verify=%s", step.Verification.Status)
+ }
+ answer := step.Answer
+ if answer == "" {
+ answer = step.Notes
+ }
+ lines = append(lines, fmt.Sprintf("- %s: status=%s%s; %s",
+ step.ID, step.Status, verification, shorten(answer, MaxSummaryTextChars)))
+ }
+ } else {
+ lines = append(lines, "No steps recorded yet.")
+ }
+
+ lines = append(lines, "Use the report file and benchmark wrappers as the source of truth.")
+ lines = append(lines, "Do not re-read large documentation files unless you are blocked on a specific syntax detail.")
+
+ return strings.Join(lines, "\n")
+}
+
+type Progress struct {
+ Answered int
+ Failed int
+ Skipped int
+ VerifiedPassed int
+}
+
+func ReadProgress(reportFile string) Progress {
+ report, err := ReadReport(reportFile)
+ if err != nil {
+ return Progress{}
+ }
+ return Progress{
+ Answered: report.Totals.StepsAnswered,
+ Failed: report.Totals.StepsFailed,
+ Skipped: report.Totals.StepsSkipped,
+ VerifiedPassed: report.Totals.StepsVerifiedPassed,
+ }
+}
diff --git a/tests/benchmark/cmd/api-runner/report_test.go b/tests/benchmark/cmd/api-runner/report_test.go
new file mode 100644
index 00000000..fbee7092
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/report_test.go
@@ -0,0 +1,108 @@
+package main
+
+import (
+ "encoding/json"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+)
+
+func TestShorten(t *testing.T) {
+ tests := []struct {
+ input string
+ max int
+ expected string
+ }{
+ {"hello world", 20, "hello world"},
+ {"hello world\n\tfoo", 20, "hello world foo"},
+ {"this is a very long string", 10, "this is..."},
+ }
+
+ for _, tc := range tests {
+ got := shorten(tc.input, tc.max)
+ if got != tc.expected {
+ t.Errorf("shorten(%q, %d) = %q; want %q", tc.input, tc.max, got, tc.expected)
+ }
+ }
+}
+
+func TestBuildProgressSummaryNoFile(t *testing.T) {
+ summary := BuildProgressSummary("/nonexistent/file.json")
+ if !strings.Contains(summary, "No benchmark report file") {
+ t.Errorf("unexpected summary for missing file: %q", summary)
+ }
+}
+
+func TestBuildProgressSummaryWithData(t *testing.T) {
+ dir := t.TempDir()
+ path := filepath.Join(dir, "report.json")
+
+ report := map[string]interface{}{
+ "totals": map[string]interface{}{
+ "steps_answered": 5,
+ "steps_failed": 1,
+ "steps_skipped": 0,
+ "steps_verified_passed": 4,
+ "steps_verified_failed": 1,
+ "steps_verified_skipped": 0,
+ "steps_pending_verification": 0,
+ },
+ "steps": []map[string]interface{}{
+ {"id": "0.1", "status": "answered", "answer": "ok", "verification": map[string]string{"status": "pass"}},
+ {"id": "0.2", "status": "answered", "answer": "done"},
+ {"id": "1.1", "status": "failed", "notes": "error occurred"},
+ },
+ }
+ data, _ := json.Marshal(report)
+ os.WriteFile(path, data, 0o644)
+
+ summary := BuildProgressSummary(path)
+
+ checks := []string{
+ "answered=5",
+ "execution_failed=1",
+ "verified_passed=4",
+ "0.1: status=answered / verify=pass",
+ "1.1: status=failed",
+ }
+ for _, check := range checks {
+ if !strings.Contains(summary, check) {
+ t.Errorf("summary missing %q:\n%s", check, summary)
+ }
+ }
+}
+
+func TestReadProgress(t *testing.T) {
+ dir := t.TempDir()
+ path := filepath.Join(dir, "report.json")
+
+ report := map[string]interface{}{
+ "totals": map[string]interface{}{
+ "steps_answered": 10,
+ "steps_failed": 2,
+ "steps_skipped": 1,
+ "steps_verified_passed": 8,
+ },
+ }
+ data, _ := json.Marshal(report)
+ os.WriteFile(path, data, 0o644)
+
+ p := ReadProgress(path)
+ if p.Answered != 10 {
+ t.Errorf("Answered = %d; want 10", p.Answered)
+ }
+ if p.Failed != 2 {
+ t.Errorf("Failed = %d; want 2", p.Failed)
+ }
+ if p.VerifiedPassed != 8 {
+ t.Errorf("VerifiedPassed = %d; want 8", p.VerifiedPassed)
+ }
+}
+
+func TestReadProgressMissingFile(t *testing.T) {
+ p := ReadProgress("/nonexistent.json")
+ if p.Answered != 0 || p.Failed != 0 {
+ t.Errorf("expected zero progress for missing file; got %+v", p)
+ }
+}
diff --git a/tests/benchmark/cmd/api-runner/retry.go b/tests/benchmark/cmd/api-runner/retry.go
new file mode 100644
index 00000000..76c927b5
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/retry.go
@@ -0,0 +1,105 @@
+package main
+
+import (
+ "context"
+ "fmt"
+ "io"
+ "net/http"
+ "strconv"
+ "time"
+)
+
+const (
+ MaxRetryAttempts = 3
+ MaxRetryBackoff = 15 * time.Second
+ InitialRetryDelay = 2 * time.Second
+)
+
+type RetryConfig struct {
+ MaxAttempts int
+ InitialDelay time.Duration
+ MaxDelay time.Duration
+ Clock func() time.Time
+ Sleep func(time.Duration)
+}
+
+func DefaultRetryConfig() RetryConfig {
+ return RetryConfig{
+ MaxAttempts: MaxRetryAttempts,
+ InitialDelay: InitialRetryDelay,
+ MaxDelay: MaxRetryBackoff,
+ Clock: time.Now,
+ Sleep: time.Sleep,
+ }
+}
+
+type DoFunc func() (*http.Response, error)
+
+func DoWithRetry(ctx context.Context, cfg RetryConfig, do DoFunc) (*http.Response, error) {
+ var lastErr error
+ delay := cfg.InitialDelay
+
+ for attempt := 0; attempt < cfg.MaxAttempts; attempt++ {
+ if ctx.Err() != nil {
+ return nil, ctx.Err()
+ }
+
+ resp, err := do()
+ if err != nil {
+ lastErr = err
+ if attempt < cfg.MaxAttempts-1 {
+ cfg.Sleep(delay)
+ delay = min(delay*2, cfg.MaxDelay)
+ }
+ continue
+ }
+
+ if resp.StatusCode < 400 {
+ return resp, nil
+ }
+
+ if !isRetryable(resp.StatusCode) {
+ return resp, nil
+ }
+
+ lastErr = fmt.Errorf("HTTP %d", resp.StatusCode)
+ io.Copy(io.Discard, resp.Body)
+ resp.Body.Close()
+
+ if attempt < cfg.MaxAttempts-1 {
+ retryDelay := parseRetryAfter(resp.Header.Get("Retry-After"), cfg.Clock())
+ if retryDelay > 0 {
+ delay = min(retryDelay, cfg.MaxDelay)
+ }
+ cfg.Sleep(delay)
+ delay = min(delay*2, cfg.MaxDelay)
+ }
+ }
+
+ return nil, fmt.Errorf("max retries exceeded: %w", lastErr)
+}
+
+func isRetryable(statusCode int) bool {
+ switch statusCode {
+ case 429, 500, 502, 503, 504:
+ return true
+ default:
+ return false
+ }
+}
+
+func parseRetryAfter(value string, now time.Time) time.Duration {
+ if value == "" {
+ return 0
+ }
+
+ if seconds, err := strconv.Atoi(value); err == nil {
+ return time.Duration(seconds) * time.Second
+ }
+
+ if t, err := http.ParseTime(value); err == nil {
+ return t.Sub(now)
+ }
+
+ return 0
+}
diff --git a/tests/benchmark/cmd/api-runner/retry_test.go b/tests/benchmark/cmd/api-runner/retry_test.go
new file mode 100644
index 00000000..f807d529
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/retry_test.go
@@ -0,0 +1,153 @@
+package main
+
+import (
+ "context"
+ "net/http"
+ "net/http/httptest"
+ "sync/atomic"
+ "testing"
+ "time"
+)
+
+func TestRetrySuccessOnFirstAttempt(t *testing.T) {
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.WriteHeader(200)
+ }))
+ defer server.Close()
+
+ cfg := DefaultRetryConfig()
+ cfg.Sleep = func(d time.Duration) {}
+
+ var callCount int32
+ resp, err := DoWithRetry(context.Background(), cfg, func() (*http.Response, error) {
+ atomic.AddInt32(&callCount, 1)
+ return http.Get(server.URL)
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ resp.Body.Close()
+
+ if callCount != 1 {
+ t.Errorf("callCount = %d; want 1", callCount)
+ }
+}
+
+func TestRetryOn429(t *testing.T) {
+ var callCount int32
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ count := atomic.AddInt32(&callCount, 1)
+ if count < 3 {
+ w.Header().Set("Retry-After", "1")
+ w.WriteHeader(429)
+ return
+ }
+ w.WriteHeader(200)
+ }))
+ defer server.Close()
+
+ cfg := DefaultRetryConfig()
+ var totalSleep time.Duration
+ cfg.Sleep = func(d time.Duration) { totalSleep += d }
+
+ resp, err := DoWithRetry(context.Background(), cfg, func() (*http.Response, error) {
+ return http.Get(server.URL)
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ resp.Body.Close()
+
+ if callCount != 3 {
+ t.Errorf("callCount = %d; want 3", callCount)
+ }
+ if totalSleep < 2*time.Second {
+ t.Errorf("totalSleep = %v; want >= 2s", totalSleep)
+ }
+}
+
+func TestRetryExhausted(t *testing.T) {
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.WriteHeader(429)
+ }))
+ defer server.Close()
+
+ cfg := DefaultRetryConfig()
+ cfg.Sleep = func(d time.Duration) {}
+
+ var callCount int32
+ _, err := DoWithRetry(context.Background(), cfg, func() (*http.Response, error) {
+ atomic.AddInt32(&callCount, 1)
+ return http.Get(server.URL)
+ })
+ if err == nil {
+ t.Fatal("expected error")
+ }
+ if callCount != 3 {
+ t.Errorf("callCount = %d; want 3", callCount)
+ }
+}
+
+func TestNoRetryOn400(t *testing.T) {
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.WriteHeader(400)
+ }))
+ defer server.Close()
+
+ cfg := DefaultRetryConfig()
+ cfg.Sleep = func(d time.Duration) {}
+
+ var callCount int32
+ resp, err := DoWithRetry(context.Background(), cfg, func() (*http.Response, error) {
+ atomic.AddInt32(&callCount, 1)
+ return http.Get(server.URL)
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ resp.Body.Close()
+
+ if callCount != 1 {
+ t.Errorf("callCount = %d; want 1 (no retry on 400)", callCount)
+ }
+ if resp.StatusCode != 400 {
+ t.Errorf("status = %d; want 400", resp.StatusCode)
+ }
+}
+
+func TestParseRetryAfterSeconds(t *testing.T) {
+ now := time.Now()
+ d := parseRetryAfter("5", now)
+ if d != 5*time.Second {
+ t.Errorf("got %v; want 5s", d)
+ }
+}
+
+func TestParseRetryAfterHTTPDate(t *testing.T) {
+ now := time.Date(2026, 4, 17, 12, 0, 0, 0, time.UTC)
+ future := now.Add(30 * time.Second)
+ d := parseRetryAfter(future.Format(http.TimeFormat), now)
+ if d < 29*time.Second || d > 31*time.Second {
+ t.Errorf("got %v; want ~30s", d)
+ }
+}
+
+func TestRetryContextCancellation(t *testing.T) {
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.WriteHeader(429)
+ }))
+ defer server.Close()
+
+ ctx, cancel := context.WithCancel(context.Background())
+ cancel()
+
+ cfg := DefaultRetryConfig()
+ cfg.Sleep = func(d time.Duration) {}
+
+ _, err := DoWithRetry(ctx, cfg, func() (*http.Response, error) {
+ return http.Get(server.URL)
+ })
+ if err != context.Canceled {
+ t.Errorf("got %v; want context.Canceled", err)
+ }
+}
diff --git a/tests/benchmark/cmd/api-runner/runner.go b/tests/benchmark/cmd/api-runner/runner.go
new file mode 100644
index 00000000..22696be9
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/runner.go
@@ -0,0 +1,36 @@
+package main
+
+import "time"
+
+type ToolCall struct {
+ ID string
+ Command string
+ TimeoutSeconds int
+}
+
+type ToolExecutionResult struct {
+ ID string
+ IsError bool
+ Content string
+}
+
+type UsageCounters struct {
+ RequestCount int
+ InputTokens int
+ OutputTokens int
+ CacheCreationInputTokens int
+ CacheReadInputTokens int
+}
+
+type Runner interface {
+ Provider() string
+ Source() string
+ Model() string
+ ToolDefinitions() interface{}
+ InitialConversation(userPrompt string) []interface{}
+ Send(systemPrompt string, conversation []interface{}) (interface{}, error)
+ ExtractToolCalls(response interface{}, defaultTimeout time.Duration) []ToolCall
+ AppendToolResults(conversation []interface{}, response interface{}, results []ToolExecutionResult) []interface{}
+ ExtractFinalText(response interface{}) string
+ Usage() UsageCounters
+}
diff --git a/tests/benchmark/cmd/api-runner/shell.go b/tests/benchmark/cmd/api-runner/shell.go
new file mode 100644
index 00000000..5c5f5d69
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/shell.go
@@ -0,0 +1,174 @@
+package main
+
+import (
+ "bufio"
+ "crypto/rand"
+ "encoding/hex"
+ "errors"
+ "fmt"
+ "io"
+ "os/exec"
+ "regexp"
+ "strings"
+ "sync"
+ "time"
+)
+
+const MaxToolOutputChars = 2400
+
+type PersistentShell struct {
+ cmd *exec.Cmd
+ stdin io.WriteCloser
+ stdout io.ReadCloser
+ stderr io.ReadCloser
+
+ mu sync.Mutex
+ buffer string
+ closed bool
+}
+
+func NewPersistentShell(workDir string) (*PersistentShell, error) {
+ cmd := exec.Command("/bin/bash")
+ cmd.Dir = workDir
+
+ stdin, err := cmd.StdinPipe()
+ if err != nil {
+ return nil, fmt.Errorf("stdin pipe: %w", err)
+ }
+
+ stdout, err := cmd.StdoutPipe()
+ if err != nil {
+ stdin.Close()
+ return nil, fmt.Errorf("stdout pipe: %w", err)
+ }
+
+ stderr, err := cmd.StderrPipe()
+ if err != nil {
+ stdin.Close()
+ stdout.Close()
+ return nil, fmt.Errorf("stderr pipe: %w", err)
+ }
+
+ if err := cmd.Start(); err != nil {
+ stdin.Close()
+ stdout.Close()
+ stderr.Close()
+ return nil, fmt.Errorf("start shell: %w", err)
+ }
+
+ ps := &PersistentShell{
+ cmd: cmd,
+ stdin: stdin,
+ stdout: stdout,
+ stderr: stderr,
+ }
+
+ go ps.drainStderr()
+
+ return ps, nil
+}
+
+func (ps *PersistentShell) drainStderr() {
+ io.Copy(io.Discard, ps.stderr)
+}
+
+func (ps *PersistentShell) Run(command string, timeout time.Duration) (output string, exitCode int, err error) {
+ ps.mu.Lock()
+ defer ps.mu.Unlock()
+
+ if ps.closed {
+ return "", -1, errors.New("shell is closed")
+ }
+
+ nonce := randomHex(8)
+ marker := fmt.Sprintf("__BENCH_DONE__%s", nonce)
+ markerRegex := regexp.MustCompile(fmt.Sprintf(`\n%s:(\d+)\r?\n`, regexp.QuoteMeta(marker)))
+
+ wrapped := fmt.Sprintf("%s\nprintf '\\n%s:%%s\\n' \"$?\"\n", command, marker)
+ if _, err := io.WriteString(ps.stdin, wrapped); err != nil {
+ return "", -1, fmt.Errorf("write command: %w", err)
+ }
+
+ deadline := time.Now().Add(timeout)
+ reader := bufio.NewReader(ps.stdout)
+
+ for time.Now().Before(deadline) {
+ remaining := time.Until(deadline)
+ if remaining < 10*time.Millisecond {
+ remaining = 10 * time.Millisecond
+ }
+
+ done := make(chan struct{})
+ var chunk []byte
+ var readErr error
+
+ go func() {
+ buf := make([]byte, 4096)
+ n, e := reader.Read(buf)
+ chunk = buf[:n]
+ readErr = e
+ close(done)
+ }()
+
+ select {
+ case <-done:
+ if readErr != nil && readErr != io.EOF {
+ return "", -1, fmt.Errorf("read stdout: %w", readErr)
+ }
+ ps.buffer += string(chunk)
+
+ if m := markerRegex.FindStringSubmatchIndex(ps.buffer); m != nil {
+ output = ps.buffer[:m[0]]
+ output = strings.TrimPrefix(output, "\n")
+ var code int
+ fmt.Sscanf(ps.buffer[m[2]:m[3]], "%d", &code)
+ ps.buffer = ps.buffer[m[1]:]
+ return output, code, nil
+ }
+ case <-time.After(remaining):
+ ps.closeLocked(true)
+ return "", -1, fmt.Errorf("command timed out after %v: %s", timeout, command)
+ }
+ }
+
+ ps.closeLocked(true)
+ return "", -1, fmt.Errorf("command timed out after %v: %s", timeout, command)
+}
+
+func (ps *PersistentShell) closeLocked(force bool) {
+ if ps.closed {
+ return
+ }
+ ps.closed = true
+
+ if force {
+ ps.cmd.Process.Kill()
+ } else {
+ io.WriteString(ps.stdin, "exit\n")
+ }
+ ps.stdin.Close()
+ ps.cmd.Wait()
+}
+
+func (ps *PersistentShell) Close(force bool) {
+ ps.mu.Lock()
+ defer ps.mu.Unlock()
+ ps.closeLocked(force)
+}
+
+func randomHex(n int) string {
+ b := make([]byte, n)
+ rand.Read(b)
+ return hex.EncodeToString(b)
+}
+
+func TrimToolOutput(text string) string {
+ if len(text) <= MaxToolOutputChars {
+ return text
+ }
+ return fmt.Sprintf("%s\n\n[output truncated: %d more chars]", text[:MaxToolOutputChars], len(text)-MaxToolOutputChars)
+}
+
+func FormatToolResult(command string, exitCode int, output string) string {
+ return fmt.Sprintf("$ %s\n[exit_code=%d]\n%s", command, exitCode, TrimToolOutput(output))
+}
diff --git a/tests/benchmark/cmd/api-runner/shell_test.go b/tests/benchmark/cmd/api-runner/shell_test.go
new file mode 100644
index 00000000..ac4f8df6
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/shell_test.go
@@ -0,0 +1,137 @@
+package main
+
+import (
+ "strings"
+ "testing"
+ "time"
+)
+
+func TestShellEcho(t *testing.T) {
+ ps, err := NewPersistentShell(t.TempDir())
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer ps.Close(false)
+
+ out, code, err := ps.Run("echo hello", 5*time.Second)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if code != 0 {
+ t.Errorf("exit code = %d; want 0", code)
+ }
+ if strings.TrimSpace(out) != "hello" {
+ t.Errorf("output = %q; want 'hello'", out)
+ }
+}
+
+func TestShellNonZeroExit(t *testing.T) {
+ ps, err := NewPersistentShell(t.TempDir())
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer ps.Close(false)
+
+ out, code, err := ps.Run("(exit 42)", 5*time.Second)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if code != 42 {
+ t.Errorf("exit code = %d; want 42", code)
+ }
+ if out != "" {
+ t.Errorf("output = %q; want empty", out)
+ }
+}
+
+func TestShellTimeout(t *testing.T) {
+ ps, err := NewPersistentShell(t.TempDir())
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer ps.Close(true)
+
+ _, _, err = ps.Run("sleep 60", 100*time.Millisecond)
+ if err == nil {
+ t.Fatal("expected timeout error")
+ }
+ if !strings.Contains(err.Error(), "timed out") {
+ t.Errorf("error = %v; want timeout", err)
+ }
+}
+
+func TestShellEnvPersistence(t *testing.T) {
+ ps, err := NewPersistentShell(t.TempDir())
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer ps.Close(false)
+
+ _, code1, err := ps.Run("export TESTVAR=foobar123", 5*time.Second)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if code1 != 0 {
+ t.Fatalf("export exit code = %d", code1)
+ }
+
+ out, code2, err := ps.Run("echo $TESTVAR", 5*time.Second)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if code2 != 0 {
+ t.Errorf("echo exit code = %d", code2)
+ }
+ if strings.TrimSpace(out) != "foobar123" {
+ t.Errorf("output = %q; want 'foobar123'", out)
+ }
+}
+
+func TestShellLargeOutput(t *testing.T) {
+ ps, err := NewPersistentShell(t.TempDir())
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer ps.Close(false)
+
+ out, code, err := ps.Run("seq 1 50000", 30*time.Second)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if code != 0 {
+ t.Errorf("exit code = %d; want 0", code)
+ }
+ lines := strings.Split(strings.TrimSpace(out), "\n")
+ if len(lines) != 50000 {
+ t.Errorf("got %d lines; want 50000", len(lines))
+ }
+ if lines[0] != "1" || lines[49999] != "50000" {
+ t.Errorf("first/last lines incorrect: %q / %q", lines[0], lines[49999])
+ }
+}
+
+func TestTrimToolOutput(t *testing.T) {
+ short := "hello world"
+ if got := TrimToolOutput(short); got != short {
+ t.Errorf("short text modified: %q", got)
+ }
+
+ long := strings.Repeat("x", 3000)
+ trimmed := TrimToolOutput(long)
+ if len(trimmed) > MaxToolOutputChars+100 {
+ t.Errorf("trimmed too long: %d chars", len(trimmed))
+ }
+ if !strings.Contains(trimmed, "[output truncated:") {
+ t.Error("missing truncation marker")
+ }
+}
+
+func TestFormatToolResult(t *testing.T) {
+ result := FormatToolResult("ls -la", 0, "file.txt")
+ if !strings.HasPrefix(result, "$ ls -la") {
+ t.Errorf("missing command prefix: %q", result)
+ }
+ if !strings.Contains(result, "[exit_code=0]") {
+ t.Errorf("missing exit code: %q", result)
+ }
+}
diff --git a/tests/benchmark/cmd/api-runner/tasks.go b/tests/benchmark/cmd/api-runner/tasks.go
new file mode 100644
index 00000000..3ce76b95
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/tasks.go
@@ -0,0 +1,70 @@
+package main
+
+import (
+ "fmt"
+ "io"
+ "sort"
+)
+
+type GroupSelection struct {
+ Selected []int
+ Warnings []string
+}
+
+func SelectGroups(args Args, idx IndexFile, stderr io.Writer) (GroupSelection, error) {
+ var sel GroupSelection
+
+ if len(args.Groups) > 0 {
+ for _, g := range args.Groups {
+ sel.Selected = append(sel.Selected, g)
+ if idx.IsDisabled(g) {
+ w := fmt.Sprintf("warning: group %d is disabled in index but explicitly requested", g)
+ sel.Warnings = append(sel.Warnings, w)
+ fmt.Fprintln(stderr, w)
+ }
+ }
+ sort.Ints(sel.Selected)
+ return sel, nil
+ }
+
+ profileGroups := resolveProfile(args.Profile)
+ activeSet := make(map[int]struct{})
+ for _, g := range idx.Active {
+ activeSet[g] = struct{}{}
+ }
+
+ if len(profileGroups) > 0 {
+ for _, g := range profileGroups {
+ if _, ok := activeSet[g]; ok {
+ sel.Selected = append(sel.Selected, g)
+ }
+ }
+ sort.Ints(sel.Selected)
+ return sel, nil
+ }
+
+ if len(idx.Active) > 0 {
+ sel.Selected = append(sel.Selected, idx.Active...)
+ return sel, nil
+ }
+
+ fallback, err := ScanGroupFiles(idx.Dir)
+ if err != nil {
+ return sel, fmt.Errorf("fallback scan: %w", err)
+ }
+ sel.Selected = fallback
+ return sel, nil
+}
+
+func resolveProfile(profile string) []int {
+ switch profile {
+ case "common10":
+ return []int{0, 1, 2, 3}
+ default:
+ return nil
+ }
+}
+
+func GroupFilePath(dir string, group int) string {
+ return fmt.Sprintf("%s/group-%02d.md", dir, group)
+}
diff --git a/tests/benchmark/cmd/api-runner/testdata/golden/progress-summary-sample.txt b/tests/benchmark/cmd/api-runner/testdata/golden/progress-summary-sample.txt
new file mode 100644
index 00000000..df1b2717
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/testdata/golden/progress-summary-sample.txt
@@ -0,0 +1,23 @@
+Benchmark progress summary from the external report.
+- answered=35
+- execution_failed=0
+- execution_skipped=0
+- verified_passed=35
+- verified_failed=0
+- verified_skipped=0
+- pending_verification=0
+Recent recorded steps:
+- 5.2: status=answer / verify=pass; VERIFY_LOGIN_SUCCESS_DASHBOARD
+- 6.1: status=answer / verify=pass; Headphones .99, Watch .99, Charger .99. Mechanical Keyboard out of stock.
+- 6.2: status=answer / verify=pass; Total: .98
+- 6.3: status=answer / verify=pass; VERIFY_CHECKOUT_SUCCESS_ORDER
+- 7.1: status=answer / verify=pass; COMMENT_POSTED_RATING_5_TEXT_RECEIVED
+- 7.2: status=answer / verify=pass; Navigated wiki -> Python article after reading categories (DevTools 15, Languages 12)
+- 8.1: status=answer / verify=pass; 404 Not Found, server responsive
+- 8.2: status=answer / verify=pass; Error 400: no element found
+- 9.1: status=answer / verify=pass; screenshot-20260416-233758.jpg (61566 bytes)
+- 9.2: status=answer / verify=pass; page-20260416-233759.pdf (46078 bytes)
+- 10.1: status=answer / verify=pass; Settings modal visible with theme dropdown
+- 10.2: status=answer / verify=pass; THEME_DARK_APPLIED
+Use the report file and benchmark wrappers as the source of truth.
+Do not re-read large documentation files unless you are blocked on a specific syntax detail.
\ No newline at end of file
diff --git a/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-all.txt b/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-all.txt
new file mode 100644
index 00000000..82541b23
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-all.txt
@@ -0,0 +1,692 @@
+Work in this repo: /sessions/eager-determined-cannon/mnt/pinchtab
+
+Benchmark lane: PinchTab agent execution.
+
+Requirements:
+- Follow a linear execution flow: skill once, selected groups, execute, record, verify, continue.
+- Do not read README, browse directories, or inspect unrelated files unless a command path is missing.
+- Read /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/setup-pinchtab.md exactly once before acting.
+- Tool wrapper:
+ - ./scripts/pt
+- Workflow summary:
+ - use only ./scripts/pt for browser actions
+ - keep one shared tab via PINCHTAB_TAB
+ - use snap -i -c for actionable refs
+ - use text or text --full for reading content
+ - refresh refs after any navigation or DOM change
+- Adapter notes:
+ - a navigation-expected click can return "Error 409: unexpected page navigation ..."; treat that as likely success and verify with a fresh snapshot/text read
+ - do not assume every command returns JSON; only parse JSON when the command actually returned JSON
+ - the download endpoint returns JSON with base64 content in "data", not a local file path
+ - this environment uses BSD/macOS userland tools; avoid GNU-only flags such as "head -n -1"
+- Task scope:
+ Use the benchmark run index at /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/benchmark-run/index.md plus all group files in /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/benchmark-run.
+
+ # Benchmark Run
+
+ Natural-language benchmark task set for the agent-driven lanes.
+
+ Read this index first, then read the relevant group files.
+
+ ## How to use this directory
+
+ - `setup-pinchtab.md` and `setup-agent-browser.md` define lane-specific setup, wrapper usage, recording, and Group 0 behavior.
+ - This directory defines the shared task groups.
+ - Each group lives in its own file: `group-00.md` through `group-38.md`.
+ - When running a subset, read only the selected group files.
+ - When running the full suite, proceed in numeric order.
+
+ ## Group Files
+
+ - `group-00.md` — Setup & Diagnosis
+ - `group-01.md` — Reading & Extracting Real Content
+ - `group-02.md` — Search & Dynamic Interaction
+ - `group-03.md` — Complex Form
+ - `group-04.md` — SPA State Management
+ - `group-05.md` — Login Flow
+ - `group-06.md` — Multi-Step E-commerce
+ - `group-07.md` — Content + Interaction Combined
+ - `group-08.md` — Error Handling
+ - `group-09.md` — Export
+ - `group-10.md` — Nested Interactions & Modal Dialogs
+ - `group-11.md` — State Persistence & Page Reload
+ - `group-12.md` — Multi-Page Navigation & Back Button
+ - `group-13.md` — Form State & Multi-Step Submission
+ - `group-14.md` — Dynamic Content Loading
+ - `group-15.md` — Complex Data Extraction & Aggregation
+ - `group-16.md` — Hover & Tooltips
+ - `group-17.md` — Scrolling
+ - `group-18.md` — File Download
+ - `group-19.md` — iFrame
+ - `group-20.md` — Dialogs
+ - `group-21.md` — Async / awaitPromise
+ - `group-22.md` — Mouse Drag & Drop
+ - `group-23.md` — Async / Loading state
+ - `group-24.md` — Keyboard events
+ - `group-25.md` — Tab panels
+ - `group-26.md` — Accordion
+ - `group-27.md` — Contenteditable editor
+ - `group-28.md` — Range slider
+ - `group-29.md` — Pagination
+ - `group-30.md` — Custom dropdown menu
+ - `group-31.md` — Nested iframes (3 levels deep)
+ - `group-32.md` — Dynamic iframe (inserted after load)
+ - `group-33.md` — srcdoc iframe
+ - `group-34.md` — Sandboxed iframe
+ - `group-35.md` — Long-form article (Medium/Substack style)
+ - `group-36.md` — Search results page (SERP)
+ - `group-37.md` — Q&A thread (Stack-Overflow style)
+ - `group-38.md` — Pricing table
+
+ # Group 0: Setup & Diagnosis
+
+ ### 0.1 Server reachable
+ Check that the PinchTab server is healthy.
+
+ **Verify**: Server responds with `status: ok`.
+
+ ### 0.2 Auth is required
+ Make a request to the server with a **wrong** token (`PINCHTAB_TOKEN=wrong-token ./scripts/pt health`) and confirm it is rejected. The `pt` wrapper always injects the benchmark token by default, so you must explicitly override it to test auth rejection.
+
+ **Verify**: Response is HTTP 401 or contains `unauthorized`.
+
+ ### 0.3 Auth works with token
+ Repeat the same request WITH the bearer token and confirm it succeeds.
+
+ **Verify**: Response is HTTP 200.
+
+ ### 0.4 Instance available
+ Confirm at least one Chrome instance is running. If none exist, start one.
+
+ **Verify**: Health response shows `defaultInstance.status == "running"` (or after starting one, the new instance is running).
+
+ ### 0.5 List existing tabs
+ Get the current list of open tabs.
+
+ **Verify**: A list (possibly empty) is returned without error.
+
+ ### 0.6 Clean stale tabs
+ If any tabs from previous runs are open, close them so the benchmark starts from a known state.
+
+ **Verify**: After cleanup, the tab list is empty (or only contains a single about:blank tab).
+
+ ### 0.7 Network reach to target
+ Navigate to `http://fixtures/` and confirm the fixtures server is reachable from PinchTab.
+
+ **Verify**: Navigate returns successfully and the page contains benchmark content.
+
+ ### 0.8 Capture initial tab ID
+ Save the tab ID returned by the navigate in 0.7. Use this tab ID for all subsequent tasks to avoid creating new tabs.
+
+ **Verify**: A tab ID was captured and matches what `GET /tabs` reports.
+
+ ---
+
+ # Group 1: Reading & Extracting Real Content
+
+ ### 1.1 Get the full list of categories from the wiki index
+ Navigate to `http://fixtures/wiki.html` and extract all category names with their article counts.
+
+ **Verify**: Can name at least 2 categories and their article counts (e.g. "Programming Languages: 12 articles").
+
+ ### 1.2 Navigate by clicking a link
+ From the wiki index, click the "Go (programming language)" link to navigate to the Go article.
+
+ **Verify**: You are now on the Go article page (not the wiki index).
+
+ ### 1.3 Extract structured data from a table
+ On the Go article, read the infobox table and answer: Who designed Go, and what year did it first appear?
+
+ **Verify**: Answer contains "Robert Griesemer" (or "Rob Pike" or "Ken Thompson") and "2009".
+
+ ### 1.4 Count list items
+ On the Go article, count how many key features are listed.
+
+ **Verify**: Answer is 6 (verify against `FEATURE_COUNT_6`).
+
+ ### 1.5 Read all article headlines from articles page
+ Navigate to `http://fixtures/articles.html` and list all article titles.
+
+ **Verify**: Found at least 3 articles including "The Future of Artificial Intelligence".
+
+ ### 1.6 Read dashboard metrics
+ Navigate to `http://fixtures/dashboard.html` and extract: Total Users, Revenue, and Conversion Rate.
+
+ **Verify**: Found `24,582` users AND `$1,284,930` revenue.
+
+ ---
+
+ # Group 2: Search & Dynamic Interaction
+
+ ### 2.1 Use wiki search to find a page
+ On `http://fixtures/wiki.html`, search for "golang" using the search form. Do not navigate directly — use the search input.
+
+ **Verify**: Ended up on the Go article page after search.
+
+ ### 2.2 Search with no results
+ On `http://fixtures/search.html`, search for something with no results (use "xyznonexistent").
+
+ **Verify**: Page handled it gracefully (no crash, some response rendered).
+
+ ### 2.3 Search for AI content
+ On `http://fixtures/search.html`, search for "artificial intelligence" and verify a result appeared.
+
+ **Verify**: Result contains "The Future of Artificial Intelligence".
+
+ ---
+
+ # Group 3: Complex Form
+
+ ### 3.1 Fill and submit a complete form
+ Navigate to `http://fixtures/form.html`. Complete the entire form:
+ - Full name: "Agent Test User"
+ - Email: "[email protected]"
+ - Phone: "+44 20 9999 0000"
+ - Country: United Kingdom
+ - Subject: Technical Support
+ - Message: "Testing PinchTab form automation"
+ - Check newsletter
+ - Set priority to High
+ - Submit
+
+ **Verify**: Form submitted successfully. Confirmation shows name "AGENT_TEST_USER".
+
+ ### 3.2 Reset and refill
+ After submitting, if the form is still accessible (or navigate back), verify you can identify the reset button.
+
+ **Verify**: Reset/back button or form element found in snapshot.
+
+ ---
+
+ # Group 4: SPA State Management
+
+ ### 4.1 Read initial app state
+ Navigate to `http://fixtures/spa.html?reset=1` (the `?reset=1` query param clears any previous localStorage state so the SPA starts with its default 3 tasks). Read the current task list — how many tasks exist, how many are active vs done?
+
+ **Verify**: Found 3 total, 2 active, 1 done (verify `TASK_STATS_TOTAL_3_ACTIVE_2_DONE_1`).
+
+ ### 4.2 Add a new high-priority task
+ Add a task called "Automate deployment" with high priority.
+
+ **Verify**: Task appeared in the list (`TASK_ADDED_AUTOMATE_DEPLOYMENT_PRIORITY_HIGH`).
+
+ ### 4.3 Delete a task
+ Delete the task titled "Write benchmark tests".
+
+ **Verify**: Task count changed (went from 4 to 3).
+
+ ---
+
+ # Group 5: Login Flow
+
+ ### 5.1 Attempt login with wrong credentials
+ Navigate to `http://fixtures/login.html`. Try to log in with username "admin" and password "wrong".
+
+ **Verify**: Error message appeared (`INVALID_CREDENTIALS_ERROR`).
+
+ ### 5.2 Login successfully
+ Clear the form and log in with username "benchmark" / password "test456".
+
+ **Verify**: Dashboard appeared after login (`VERIFY_LOGIN_SUCCESS_DASHBOARD`).
+
+ ---
+
+ # Group 6: Multi-Step E-commerce
+
+ ### 6.1 Research products before buying
+ Navigate to `http://fixtures/ecommerce.html`. List all available products with their prices. Which product is out of stock?
+
+ **Verify**: Found Wireless Headphones ($149.99), Smart Watch Pro ($299.99), Portable Charger ($49.99). Mechanical Keyboard is out of stock.
+
+ ### 6.2 Add two items and verify total
+ Add Wireless Headphones and Smart Watch Pro to cart. What is the total?
+
+ **Verify**: Cart total is $449.98 (149.99 + 299.99).
+
+ ### 6.3 Complete checkout
+ Click checkout to complete the order.
+
+ **Verify**: Order confirmation shows (`VERIFY_CHECKOUT_SUCCESS_ORDER`).
+
+ ---
+
+ # Group 7: Content + Interaction Combined
+
+ ### 7.1 Read and comment on Go article
+ Navigate to `http://fixtures/wiki-go.html`. Read the article, then post a comment with rating 5 stars and text "Excellent reference".
+
+ **Verify**: Comment posted (`COMMENT_POSTED_RATING_5_TEXT_RECEIVED`).
+
+ ### 7.2 Cross-page research task
+ Navigate to wiki index, find which category has the most articles, then navigate to one of its listed items.
+
+ **Verify**: Successfully navigated to at least one article page after reading the category counts.
+
+ ---
+
+ # Group 8: Error Handling
+
+ ### 8.1 Handle 404 gracefully
+ Try to navigate to a page that doesn't exist: `http://fixtures/missing-page-abc.html`.
+
+ **Verify**: Got a response (404 or error), no crash, server still responsive after.
+
+ ### 8.2 Handle missing element gracefully
+ On any page, try to click an element with ID `#fake-button-that-does-not-exist`.
+
+ **Verify**: Got a clear error message, not a crash or hang.
+
+ ---
+
+ # Group 9: Export
+
+ ### 9.1 Screenshot a complex page
+ Navigate to `http://fixtures/dashboard.html` and take a screenshot.
+
+ **Verify**: Screenshot generated (file saved or base64 returned).
+
+ ### 9.2 Export a page as PDF
+ Export the dashboard as a PDF.
+
+ **Verify**: PDF generated.
+
+ ---
+
+ # Group 10: Nested Interactions & Modal Dialogs
+
+ ### 10.1 Open and interact with modal on dashboard
+ Navigate to `http://fixtures/dashboard.html`. Find and click the Settings button (selector: `#settings-btn`) to open the modal dialog.
+
+ **Verify**: Modal appeared — snapshot contains "Dashboard Settings".
+
+ ### 10.2 Modify settings and close modal
+ In the modal, select "Dark" from the theme dropdown (`#theme-select`), then click the Save button (`#modal-save`). After the modal closes, check the page content.
+
+ **Verify**: Page contains `THEME_DARK_APPLIED`.
+
+ ---
+
+ # Group 11: State Persistence & Page Reload
+
+ ### 11.1 Add an item and verify after page reload
+ Navigate to `http://fixtures/spa.html?reset=1` (starts with clean state). Add a task titled exactly "Persistent Task Test". Then navigate away to `http://fixtures/` and back to `http://fixtures/spa.html` (WITHOUT the reset param) to verify localStorage persistence.
+
+ **Verify**: After reload, the task still appears in the list (`TASK_PERSISTENT_TEST_FOUND_AFTER_RELOAD`).
+
+ ### 11.2 Logout and log back in
+ From the logged-in dashboard, click Sign Out to log out. Then log in again with username "benchmark" / password "test456".
+
+ **Verify**: Successfully logged back in and dashboard shows `SESSION_RENEWED`.
+
+ ---
+
+ # Group 12: Multi-Page Navigation & Back Button
+
+ ### 12.1 Navigate through multiple pages and return
+ Starting from `http://fixtures/`, navigate to wiki → Go article → back to wiki → back to home.
+
+ **Verify**: Successfully returned to home page (title contains "Benchmark" or "Home").
+
+ ### 12.2 Compare data across pages
+ Navigate to wiki.html, note the total article count from categories. Navigate to articles.html, count articles there. Compare totals.
+
+ **Verify**: Can report totals from both pages and explain difference (`COMPARISON_DATA_FOUND`).
+
+ ---
+
+ # Group 13: Form State & Multi-Step Submission
+
+ ### 13.1 Submit form without email
+ Navigate to `http://fixtures/form.html`. Fill only the name field ("Validator Test"), leave email blank, click Submit. The browser's native required-field validation will prevent submission.
+
+ **Verify**: Submission blocked (form stays open, no success message shown).
+
+ ### 13.2 Submit form without optional phone field
+ Fill the form with: name "No Phone User", email "[email protected]", country "de", subject "feedback". Leave the phone field empty. Submit.
+
+ **Verify**: Submission succeeded and page shows `OPTIONAL_FIELD_SKIPPED_SUCCESS`.
+
+ ---
+
+ # Group 14: Dynamic Content Loading
+
+ ### 14.1 Load more products
+ Navigate to `http://fixtures/ecommerce.html`. Find and click the "Load More Products" button to reveal additional products.
+
+ **Verify**: Additional products appeared (`ADDITIONAL_PRODUCTS_LOADED`).
+
+ ### 14.2 Add a lazy-loaded product to cart
+ After loading more products, add product #5 (USB-C Cable) to the cart.
+
+ **Verify**: Cart shows the lazy-loaded item (`CART_UPDATED_WITH_LAZY_PRODUCT`).
+
+ ---
+
+ # Group 15: Complex Data Extraction & Aggregation
+
+ ### 15.1 Extract and sum financial data
+ Navigate to `http://fixtures/dashboard.html`. Extract revenue and profit values, calculate profit margin.
+
+ **Verify**: Correctly calculated: profit_margin = (profit / revenue) * 100 (`PROFIT_MARGIN_CALCULATED`).
+
+ ### 15.2 Build comparison table from multiple sources
+ Visit these 3 pages and compare their feature counts and key features:
+ - `http://fixtures/wiki-go.html` (Go: 6 features)
+ - `http://fixtures/wiki-python.html` (Python: 7 features)
+ - `http://fixtures/wiki-rust.html` (Rust: 5 features)
+
+ Report which language has the most features and name 1 feature unique to each.
+
+ **Verify**: Response is factually correct AND wiki-python.html contains `COMPARISON_TABLE_BUILT`.
+
+ ---
+
+ # Group 16: Hover & Tooltips
+
+ ### 16.1 Hover reveals hidden content
+ Navigate to `http://fixtures/hovers.html`. Hover over the first avatar to reveal its hidden info.
+
+ **Verify**: After hovering, snapshot contains `HOVER_REVEALED_USER_1`.
+
+ ### 16.2 Hover swap
+ Hover over the second avatar.
+
+ **Verify**: Snapshot contains `HOVER_REVEALED_USER_2`.
+
+ ---
+
+ # Group 17: Scrolling
+
+ ### 17.1 Scroll by pixels
+ Navigate to `http://fixtures/scroll.html`. Scroll down approximately 1500 pixels.
+
+ **Verify**: Snapshot or text contains `SCROLL_MIDDLE_MARKER` (the mid-page marker is now in view).
+
+ ### 17.2 Scroll to footer
+ Continue scrolling until the footer is visible (or scroll directly to the footer element).
+
+ **Verify**: Snapshot contains `SCROLL_REACHED_FOOTER`.
+
+ ---
+
+ # Group 18: File Download
+
+ ### 18.1 Download a file
+ Download `http://fixtures/download-sample.txt` to a local file.
+
+ **Verify**: The downloaded file exists and its content includes `DOWNLOAD_FILE_CONTENT_VERIFIED`.
+
+ ---
+
+ # Group 19: iFrame
+
+ ### 19.1 Read iframe content
+ Navigate to `http://fixtures/iframe.html` and extract content from inside the embedded same-origin iframe.
+
+ **Verify**: The iframe's inner content includes `IFRAME_INNER_CONTENT_LOADED`.
+
+ ### 19.2 Type into iframe input (native frame scope)
+ Use `pinchtab frame '#content-frame'` to scope into the iframe, then `fill` the input with "Hello World" and `click` the Save button. Verify via a scoped `snap` — `text --full` doesn't pierce iframes. Reset with `pinchtab frame main` afterwards.
+
+ **Verify**: Scoped snapshot contains `IFRAME_INPUT_RECEIVED_HELLO_WORLD`.
+
+ ---
+
+ # Group 20: Dialogs
+
+ ### 20.1 Accept alert
+ Navigate to `http://fixtures/alerts.html`. Click the "Click for Alert" button and dismiss the alert.
+
+ **Verify**: The page result contains `DIALOG_ALERT_DISMISSED`.
+
+ ### 20.2 Cancel confirm
+ Click the "Click for Confirm" button and cancel the confirm dialog.
+
+ **Verify**: The page result contains `DIALOG_CONFIRM_CANCELLED`.
+
+ ---
+
+ # Group 21: Async / awaitPromise
+
+ ### 21.1 Await a promise-returning function
+ Navigate to `http://fixtures/async.html`. The page exposes `window.fetchPayload()`, which returns a `Promise` that resolves after a short delay. Use `eval` to call it and retrieve the **resolved** value, not a Promise wrapper.
+
+ **Verify**: The resolved value contains `ASYNC_PAYLOAD_READY_42`.
+
+ ### 21.2 Await a promise resolving to an object
+ On the same page, call `window.fetchUser()` and retrieve the resolved object so you can read a field from it.
+
+ **Verify**: The resolved object's `name` field equals `ASYNC_USER_NAME_ADA`.
+
+ ---
+
+ # Group 22: Mouse Drag & Drop
+
+ ### 22.1 Drag a piece into Zone A
+ Navigate to `http://fixtures/drag.html`. The page contains a draggable square (`#piece`) and three target zones (`#zone-a`, `#zone-b`, `#zone-c`). Drag the piece so its center ends up over **Zone A**.
+
+ **Verify**: The page shows `LAST_DROP=DROP_ZONE_A_OK`.
+
+ ### 22.2 Drag to Zone B, then Zone C
+ Without reloading the page, drag the piece next into **Zone B**, and then into **Zone C**. The page records an ordered drop sequence.
+
+ **Verify**: The page shows `DROP_SEQUENCE=DROP_ZONE_A_OK,DROP_ZONE_B_OK,DROP_ZONE_C_OK` (all three drops in order).
+
+ ---
+
+ # Group 23: Async / Loading state
+
+ ### 23.1 Wait for async content
+ Navigate to `http://fixtures/loading.html`. The page shows a spinner and the text "Loading..." for ~1.5 seconds, then replaces it with the final payload. Wait for the final content to appear before reporting completion.
+
+ **Verify**: The final snapshot contains `VERIFY_LOADING_COMPLETE_88888`.
+
+ ---
+
+ # Group 24: Keyboard events
+
+ ### 24.1 Press Escape
+ Navigate to `http://fixtures/keyboard.html` (it auto-focuses an input so keyboard events land). Press the Escape key.
+
+ **Verify**: Page text contains `KEYBOARD_ESCAPE_PRESSED`.
+
+ ### 24.2 Press 'a' then Enter
+ Without reloading, press the `a` key, then the `Enter` key.
+
+ **Verify**: Page text now contains all three markers in order: `KEYBOARD_ESCAPE_PRESSED`, `KEYBOARD_KEY_A_PRESSED`, `KEYBOARD_ENTER_PRESSED`.
+
+ ---
+
+ # Group 25: Tab panels
+
+ ### 25.1 Switch to the Settings tab
+ Navigate to `http://fixtures/tabs.html`. The Profile tab is shown by default. Click the Settings tab.
+
+ **Verify**: After the click, the page content includes `TAB_SETTINGS_CONTENT` (and the Profile content is no longer visible).
+
+ ### 25.2 Switch to the Billing tab
+ Click the Billing tab.
+
+ **Verify**: Page content now includes `TAB_BILLING_CONTENT`.
+
+ ---
+
+ # Group 26: Accordion
+
+ ### 26.1 Open section A
+ Navigate to `http://fixtures/accordion.html`. Click the header for Section A to expand it.
+
+ **Verify**: Page text includes `ACCORDION_SECTION_A_OPEN`.
+
+ ### 26.2 Open section B
+ Click the header for Section B. Because the accordion is exclusive-expand, Section A should close.
+
+ **Verify**: Page text includes `ACCORDION_SECTION_B_OPEN`, and Section A's `aria-expanded` attribute is now `"false"` (use `eval` to check the attribute if needed).
+
+ ---
+
+ # Group 27: Contenteditable editor
+
+ ### 27.1 Type into the rich-text editor
+ Navigate to `http://fixtures/editor.html`. Type the text `Hello rich text` into the `#editor` div. `contenteditable` elements don't have `.value`, so use `type` (keyboard events), not `fill`.
+
+ **Verify**: Page text contains `EDITOR_CHARS=15` and the mirror shows `Hello rich text`.
+
+ ### 27.2 Commit by pressing Enter
+ Press Enter (the editor intercepts Enter to commit the current buffer to a separate marker).
+
+ **Verify**: Page text contains `EDITOR_COMMITTED=Hello rich text`.
+
+ ---
+
+ # Group 28: Range slider
+
+ ### 28.1 Move slider into HIGH bucket
+ Navigate to `http://fixtures/range.html`. Set the `#volume` range slider to 90.
+
+ **Verify**: Page text contains both `RANGE_VALUE_90` and `BUCKET_HIGH`.
+
+ ### 28.2 Move slider into LOW bucket
+ Without reloading, change the slider to 10.
+
+ **Verify**: Page text now contains `RANGE_VALUE_10` and `BUCKET_LOW`.
+
+ ---
+
+ # Group 29: Pagination
+
+ ### 29.1 Advance to page 2
+ Navigate to `http://fixtures/pagination.html`. Click the Next button.
+
+ **Verify**: Page text contains `PAGE_2_FIRST_ITEM` and `PAGE_2_OF_3`.
+
+ ### 29.2 Reach the last page; Next disabled
+ Click Next once more.
+
+ **Verify**: Page text contains `PAGE_3_FIRST_ITEM` and `PAGE_3_OF_3`, and the Next button's `disabled` property is `true` (use `eval` to check).
+
+ ---
+
+ # Group 30: Custom dropdown menu
+
+ ### 30.1 Pick the "Beta" item
+ Navigate to `http://fixtures/dropdown.html`. The page has a custom dropdown — click the toggle button (`#dropdown-toggle`) to open the menu, then click the "Beta" item.
+
+ **Verify**: Page text contains `DROPDOWN_SELECTED=BETA`.
+
+ ### 30.2 Reopen and pick "Gamma"
+ Reopen the dropdown and select "Gamma". (Each selection auto-closes the menu, so you need to click the toggle again before selecting.)
+
+ **Verify**: Page text contains `DROPDOWN_SELECTED=GAMMA`.
+
+ ---
+
+ # Group 31: Nested iframes (3 levels deep)
+
+ ### 31.1 Click a button in the deepest frame
+ Navigate to `http://fixtures/iframe-nested.html`. The outer page embeds `#level-2`, which in turn embeds `#level-3`, which contains `#deep-button`. Drill through the two frame hops and click the button. Verify via a scoped `snap` before resetting to `main`.
+
+ **Verify**: Scoped snapshot contains `DEEP_CLICKED=YES_LEVEL_3`.
+
+ ---
+
+ # Group 32: Dynamic iframe (inserted after load)
+
+ ### 32.1 Wait for a late iframe, then interact with it
+ Navigate to `http://fixtures/iframe-dynamic.html`. The iframe is inserted ~1.2 s after load (use `wait --text "IFRAME_DYNAMIC_ATTACHED"`). Then scope into `#late-frame`, fill `#iframe-input` with "Late World", click `#iframe-submit`, and verify the inner result marker.
+
+ **Verify**: Scoped snapshot contains `IFRAME_INPUT_RECEIVED_LATE_WORLD`.
+
+ ---
+
+ # Group 33: srcdoc iframe
+
+ ### 33.1 Interact with an inline-content iframe
+ Navigate to `http://fixtures/iframe-srcdoc.html`. The iframe's content is inlined via `srcdoc`; it has no `src` URL. Scope into `#srcdoc-frame`, fill `#inline-input` with `srcdoc`, click `#inline-submit`, and verify.
+
+ **Verify**: Scoped snapshot contains `INLINE_RECEIVED_SRCDOC`.
+
+ ---
+
+ # Group 34: Sandboxed iframe
+
+ ### 34.1 Click inside a sandboxed iframe
+ Navigate to `http://fixtures/iframe-sandbox.html`. The iframe has `sandbox="allow-scripts allow-same-origin"`. Scope into `#sandboxed`, click `#sandbox-button`, and verify.
+
+ **Verify**: Scoped snapshot contains `SANDBOX_CLICKED=YES`.
+
+ ---
+
+ # Group 35: Long-form article (Medium/Substack style)
+
+ ### 35.1 Read the article with default `text`
+ Navigate to `http://fixtures/article.html`. Extract the page's main content using the default `text` (Readability) mode. Default mode is the right choice for article-style pages.
+
+ **Verify**: The extracted text contains both `ARTICLE_PUBLISHED_2026_04_15` and `ARTICLE_WORD_COUNT_MARKER_323` (both inside the article body — Readability keeps them).
+
+ ### 35.2 See the chrome that Readability drops
+ Re-extract with `text --full` and confirm the footer is included this time.
+
+ **Verify**: `--full` output contains `FOOTER_COPYRIGHT_MARKER` (which the default mode trims).
+
+ ---
+
+ # Group 36: Search results page (SERP)
+
+ ### 36.1 Find a specific result by id
+ Navigate to `http://fixtures/serp.html`. The page has 6 result cards (`#r-1`..`#r-6`). Extract just the third card's content. A scoped snapshot (`snap --selector "#r-3"`) is the direct path.
+
+ **Verify**: The scoped output contains `RESULT_3_TITLE` and `RESULT_3_SNIPPET_MARKER`.
+
+ ### 36.2 Count all result cards
+ Use a full text extraction to verify all six results are present in one pass.
+
+ **Verify**: Output contains all of `RESULT_1_TITLE` through `RESULT_6_TITLE` and the summary `SERP_RESULT_COUNT_6`. Default Readability trims SERPs — use `text --full`.
+
+ ---
+
+ # Group 37: Q&A thread (Stack-Overflow style)
+
+ ### 37.1 Find the accepted answer id
+ Navigate to `http://fixtures/qa.html`. The accepted answer carries `data-accepted="true"` on its `<div>` wrapper. Use `eval` to return that element's `id`.
+
+ **Verify**: The `eval` result equals `"a-2"`.
+
+ ### 37.2 Extract the accepted answer's body
+ Scope a snapshot to `#a-2` and verify the body content.
+
+ **Verify**: Scoped snapshot contains `ANSWER_2_BODY_MARKER` and `ACCEPTED_ANSWER_ID_A2`.
+
+ ---
+
+ # Group 38: Pricing table
+
+ ### 38.1 Read just the Pro plan
+ Navigate to `http://fixtures/pricing.html`. The page has three plan cards (`#plan-free`, `#plan-pro`, `#plan-enterprise`). Read only the Pro plan using a scoped snapshot.
+
+ **Verify**: Scoped snapshot contains `PLAN_PRO_PRICE_29` and `PLAN_PRO_LIMIT_5000_requests per day`.
+
+ ### 38.2 Compare all three plans
+ Extract the full page (pricing grids are a Readability anti-pattern — reach for `--full`).
+
+ **Verify**: Output contains `PLAN_FREE_PRICE_0`, `PLAN_PRO_PRICE_29`, and `PLAN_ENTERPRISE_PRICE_CUSTOM`.
+- For each completed step:
+ 1. record the observed answer with:
+ - ./scripts/record-step.sh --report-file /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/results/golden_report.json --type pinchtab <group> <step> answer "<what you saw>" "notes"
+ 2. immediately verify it with:
+ - ./scripts/verify-step.sh --report-file /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/results/golden_report.json --type pinchtab <group> <step> <pass|fail|skip> "verification notes"
+- If a step cannot be completed, record fail or skip in the same report.
+- Do not leave answered steps pending verification.
+- Keep commands concise. Prefer rg/sed/cat only when you must inspect a specific file.
+- After the skill step, begin actual benchmark execution immediately.
+- Start with this bootstrap command sequence before attempting the selected steps:
+1. ./scripts/pt health
+2. ./scripts/pt tab
+3. export PINCHTAB_TAB=$(./scripts/pt nav http://fixtures/)
+4. ./scripts/pt snap -i -c
+- After the bootstrap, immediately execute Group 0 step 1.
+- Subset selection:
+ - Execute the full benchmark task set.
+- Finish when all selected steps are executed or when you are blocked.
+
+Your final answer should briefly summarize completion status and the main blockers, if any.
\ No newline at end of file
diff --git a/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-browser-all.txt b/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-browser-all.txt
new file mode 100644
index 00000000..c5815635
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-browser-all.txt
@@ -0,0 +1,691 @@
+Work in this repo: /sessions/eager-determined-cannon/mnt/pinchtab
+
+Benchmark lane: agent-browser execution.
+
+Requirements:
+- Follow a linear execution flow: skill once, selected groups, execute, record, verify, continue.
+- Do not read README, browse directories, or inspect unrelated files unless a command path is missing.
+- Read /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/setup-agent-browser.md exactly once before acting, then load the official agent-browser skill exactly once with `./scripts/ab skills get agent-browser --full`.
+- Tool wrapper:
+ - ./scripts/ab
+- Workflow summary:
+ - use only ./scripts/ab for browser actions
+ - keep the shared benchmark session across commands
+ - use snapshot -i -c for actionable refs
+ - use fresh refs after any navigation or DOM change
+ - reuse the current browser state instead of reopening pages unless needed
+- Adapter notes:
+ - after navigation-triggering clicks, verify the resulting page with a fresh snapshot/text read instead of trusting the click response alone
+ - do not assume every command returns JSON; parse only when the command actually returned JSON
+ - for downloads, inspect returned content instead of assuming a file already exists locally
+ - this environment uses BSD/macOS userland tools; avoid GNU-only flags such as "head -n -1"
+- Task scope:
+ Use the benchmark run index at /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/benchmark-run/index.md plus all group files in /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/benchmark-run.
+
+ # Benchmark Run
+
+ Natural-language benchmark task set for the agent-driven lanes.
+
+ Read this index first, then read the relevant group files.
+
+ ## How to use this directory
+
+ - `setup-pinchtab.md` and `setup-agent-browser.md` define lane-specific setup, wrapper usage, recording, and Group 0 behavior.
+ - This directory defines the shared task groups.
+ - Each group lives in its own file: `group-00.md` through `group-38.md`.
+ - When running a subset, read only the selected group files.
+ - When running the full suite, proceed in numeric order.
+
+ ## Group Files
+
+ - `group-00.md` — Setup & Diagnosis
+ - `group-01.md` — Reading & Extracting Real Content
+ - `group-02.md` — Search & Dynamic Interaction
+ - `group-03.md` — Complex Form
+ - `group-04.md` — SPA State Management
+ - `group-05.md` — Login Flow
+ - `group-06.md` — Multi-Step E-commerce
+ - `group-07.md` — Content + Interaction Combined
+ - `group-08.md` — Error Handling
+ - `group-09.md` — Export
+ - `group-10.md` — Nested Interactions & Modal Dialogs
+ - `group-11.md` — State Persistence & Page Reload
+ - `group-12.md` — Multi-Page Navigation & Back Button
+ - `group-13.md` — Form State & Multi-Step Submission
+ - `group-14.md` — Dynamic Content Loading
+ - `group-15.md` — Complex Data Extraction & Aggregation
+ - `group-16.md` — Hover & Tooltips
+ - `group-17.md` — Scrolling
+ - `group-18.md` — File Download
+ - `group-19.md` — iFrame
+ - `group-20.md` — Dialogs
+ - `group-21.md` — Async / awaitPromise
+ - `group-22.md` — Mouse Drag & Drop
+ - `group-23.md` — Async / Loading state
+ - `group-24.md` — Keyboard events
+ - `group-25.md` — Tab panels
+ - `group-26.md` — Accordion
+ - `group-27.md` — Contenteditable editor
+ - `group-28.md` — Range slider
+ - `group-29.md` — Pagination
+ - `group-30.md` — Custom dropdown menu
+ - `group-31.md` — Nested iframes (3 levels deep)
+ - `group-32.md` — Dynamic iframe (inserted after load)
+ - `group-33.md` — srcdoc iframe
+ - `group-34.md` — Sandboxed iframe
+ - `group-35.md` — Long-form article (Medium/Substack style)
+ - `group-36.md` — Search results page (SERP)
+ - `group-37.md` — Q&A thread (Stack-Overflow style)
+ - `group-38.md` — Pricing table
+
+ # Group 0: Setup & Diagnosis
+
+ ### 0.1 Server reachable
+ Check that the PinchTab server is healthy.
+
+ **Verify**: Server responds with `status: ok`.
+
+ ### 0.2 Auth is required
+ Make a request to the server with a **wrong** token (`PINCHTAB_TOKEN=wrong-token ./scripts/pt health`) and confirm it is rejected. The `pt` wrapper always injects the benchmark token by default, so you must explicitly override it to test auth rejection.
+
+ **Verify**: Response is HTTP 401 or contains `unauthorized`.
+
+ ### 0.3 Auth works with token
+ Repeat the same request WITH the bearer token and confirm it succeeds.
+
+ **Verify**: Response is HTTP 200.
+
+ ### 0.4 Instance available
+ Confirm at least one Chrome instance is running. If none exist, start one.
+
+ **Verify**: Health response shows `defaultInstance.status == "running"` (or after starting one, the new instance is running).
+
+ ### 0.5 List existing tabs
+ Get the current list of open tabs.
+
+ **Verify**: A list (possibly empty) is returned without error.
+
+ ### 0.6 Clean stale tabs
+ If any tabs from previous runs are open, close them so the benchmark starts from a known state.
+
+ **Verify**: After cleanup, the tab list is empty (or only contains a single about:blank tab).
+
+ ### 0.7 Network reach to target
+ Navigate to `http://fixtures/` and confirm the fixtures server is reachable from PinchTab.
+
+ **Verify**: Navigate returns successfully and the page contains benchmark content.
+
+ ### 0.8 Capture initial tab ID
+ Save the tab ID returned by the navigate in 0.7. Use this tab ID for all subsequent tasks to avoid creating new tabs.
+
+ **Verify**: A tab ID was captured and matches what `GET /tabs` reports.
+
+ ---
+
+ # Group 1: Reading & Extracting Real Content
+
+ ### 1.1 Get the full list of categories from the wiki index
+ Navigate to `http://fixtures/wiki.html` and extract all category names with their article counts.
+
+ **Verify**: Can name at least 2 categories and their article counts (e.g. "Programming Languages: 12 articles").
+
+ ### 1.2 Navigate by clicking a link
+ From the wiki index, click the "Go (programming language)" link to navigate to the Go article.
+
+ **Verify**: You are now on the Go article page (not the wiki index).
+
+ ### 1.3 Extract structured data from a table
+ On the Go article, read the infobox table and answer: Who designed Go, and what year did it first appear?
+
+ **Verify**: Answer contains "Robert Griesemer" (or "Rob Pike" or "Ken Thompson") and "2009".
+
+ ### 1.4 Count list items
+ On the Go article, count how many key features are listed.
+
+ **Verify**: Answer is 6 (verify against `FEATURE_COUNT_6`).
+
+ ### 1.5 Read all article headlines from articles page
+ Navigate to `http://fixtures/articles.html` and list all article titles.
+
+ **Verify**: Found at least 3 articles including "The Future of Artificial Intelligence".
+
+ ### 1.6 Read dashboard metrics
+ Navigate to `http://fixtures/dashboard.html` and extract: Total Users, Revenue, and Conversion Rate.
+
+ **Verify**: Found `24,582` users AND `$1,284,930` revenue.
+
+ ---
+
+ # Group 2: Search & Dynamic Interaction
+
+ ### 2.1 Use wiki search to find a page
+ On `http://fixtures/wiki.html`, search for "golang" using the search form. Do not navigate directly — use the search input.
+
+ **Verify**: Ended up on the Go article page after search.
+
+ ### 2.2 Search with no results
+ On `http://fixtures/search.html`, search for something with no results (use "xyznonexistent").
+
+ **Verify**: Page handled it gracefully (no crash, some response rendered).
+
+ ### 2.3 Search for AI content
+ On `http://fixtures/search.html`, search for "artificial intelligence" and verify a result appeared.
+
+ **Verify**: Result contains "The Future of Artificial Intelligence".
+
+ ---
+
+ # Group 3: Complex Form
+
+ ### 3.1 Fill and submit a complete form
+ Navigate to `http://fixtures/form.html`. Complete the entire form:
+ - Full name: "Agent Test User"
+ - Email: "[email protected]"
+ - Phone: "+44 20 9999 0000"
+ - Country: United Kingdom
+ - Subject: Technical Support
+ - Message: "Testing PinchTab form automation"
+ - Check newsletter
+ - Set priority to High
+ - Submit
+
+ **Verify**: Form submitted successfully. Confirmation shows name "AGENT_TEST_USER".
+
+ ### 3.2 Reset and refill
+ After submitting, if the form is still accessible (or navigate back), verify you can identify the reset button.
+
+ **Verify**: Reset/back button or form element found in snapshot.
+
+ ---
+
+ # Group 4: SPA State Management
+
+ ### 4.1 Read initial app state
+ Navigate to `http://fixtures/spa.html?reset=1` (the `?reset=1` query param clears any previous localStorage state so the SPA starts with its default 3 tasks). Read the current task list — how many tasks exist, how many are active vs done?
+
+ **Verify**: Found 3 total, 2 active, 1 done (verify `TASK_STATS_TOTAL_3_ACTIVE_2_DONE_1`).
+
+ ### 4.2 Add a new high-priority task
+ Add a task called "Automate deployment" with high priority.
+
+ **Verify**: Task appeared in the list (`TASK_ADDED_AUTOMATE_DEPLOYMENT_PRIORITY_HIGH`).
+
+ ### 4.3 Delete a task
+ Delete the task titled "Write benchmark tests".
+
+ **Verify**: Task count changed (went from 4 to 3).
+
+ ---
+
+ # Group 5: Login Flow
+
+ ### 5.1 Attempt login with wrong credentials
+ Navigate to `http://fixtures/login.html`. Try to log in with username "admin" and password "wrong".
+
+ **Verify**: Error message appeared (`INVALID_CREDENTIALS_ERROR`).
+
+ ### 5.2 Login successfully
+ Clear the form and log in with username "benchmark" / password "test456".
+
+ **Verify**: Dashboard appeared after login (`VERIFY_LOGIN_SUCCESS_DASHBOARD`).
+
+ ---
+
+ # Group 6: Multi-Step E-commerce
+
+ ### 6.1 Research products before buying
+ Navigate to `http://fixtures/ecommerce.html`. List all available products with their prices. Which product is out of stock?
+
+ **Verify**: Found Wireless Headphones ($149.99), Smart Watch Pro ($299.99), Portable Charger ($49.99). Mechanical Keyboard is out of stock.
+
+ ### 6.2 Add two items and verify total
+ Add Wireless Headphones and Smart Watch Pro to cart. What is the total?
+
+ **Verify**: Cart total is $449.98 (149.99 + 299.99).
+
+ ### 6.3 Complete checkout
+ Click checkout to complete the order.
+
+ **Verify**: Order confirmation shows (`VERIFY_CHECKOUT_SUCCESS_ORDER`).
+
+ ---
+
+ # Group 7: Content + Interaction Combined
+
+ ### 7.1 Read and comment on Go article
+ Navigate to `http://fixtures/wiki-go.html`. Read the article, then post a comment with rating 5 stars and text "Excellent reference".
+
+ **Verify**: Comment posted (`COMMENT_POSTED_RATING_5_TEXT_RECEIVED`).
+
+ ### 7.2 Cross-page research task
+ Navigate to wiki index, find which category has the most articles, then navigate to one of its listed items.
+
+ **Verify**: Successfully navigated to at least one article page after reading the category counts.
+
+ ---
+
+ # Group 8: Error Handling
+
+ ### 8.1 Handle 404 gracefully
+ Try to navigate to a page that doesn't exist: `http://fixtures/missing-page-abc.html`.
+
+ **Verify**: Got a response (404 or error), no crash, server still responsive after.
+
+ ### 8.2 Handle missing element gracefully
+ On any page, try to click an element with ID `#fake-button-that-does-not-exist`.
+
+ **Verify**: Got a clear error message, not a crash or hang.
+
+ ---
+
+ # Group 9: Export
+
+ ### 9.1 Screenshot a complex page
+ Navigate to `http://fixtures/dashboard.html` and take a screenshot.
+
+ **Verify**: Screenshot generated (file saved or base64 returned).
+
+ ### 9.2 Export a page as PDF
+ Export the dashboard as a PDF.
+
+ **Verify**: PDF generated.
+
+ ---
+
+ # Group 10: Nested Interactions & Modal Dialogs
+
+ ### 10.1 Open and interact with modal on dashboard
+ Navigate to `http://fixtures/dashboard.html`. Find and click the Settings button (selector: `#settings-btn`) to open the modal dialog.
+
+ **Verify**: Modal appeared — snapshot contains "Dashboard Settings".
+
+ ### 10.2 Modify settings and close modal
+ In the modal, select "Dark" from the theme dropdown (`#theme-select`), then click the Save button (`#modal-save`). After the modal closes, check the page content.
+
+ **Verify**: Page contains `THEME_DARK_APPLIED`.
+
+ ---
+
+ # Group 11: State Persistence & Page Reload
+
+ ### 11.1 Add an item and verify after page reload
+ Navigate to `http://fixtures/spa.html?reset=1` (starts with clean state). Add a task titled exactly "Persistent Task Test". Then navigate away to `http://fixtures/` and back to `http://fixtures/spa.html` (WITHOUT the reset param) to verify localStorage persistence.
+
+ **Verify**: After reload, the task still appears in the list (`TASK_PERSISTENT_TEST_FOUND_AFTER_RELOAD`).
+
+ ### 11.2 Logout and log back in
+ From the logged-in dashboard, click Sign Out to log out. Then log in again with username "benchmark" / password "test456".
+
+ **Verify**: Successfully logged back in and dashboard shows `SESSION_RENEWED`.
+
+ ---
+
+ # Group 12: Multi-Page Navigation & Back Button
+
+ ### 12.1 Navigate through multiple pages and return
+ Starting from `http://fixtures/`, navigate to wiki → Go article → back to wiki → back to home.
+
+ **Verify**: Successfully returned to home page (title contains "Benchmark" or "Home").
+
+ ### 12.2 Compare data across pages
+ Navigate to wiki.html, note the total article count from categories. Navigate to articles.html, count articles there. Compare totals.
+
+ **Verify**: Can report totals from both pages and explain difference (`COMPARISON_DATA_FOUND`).
+
+ ---
+
+ # Group 13: Form State & Multi-Step Submission
+
+ ### 13.1 Submit form without email
+ Navigate to `http://fixtures/form.html`. Fill only the name field ("Validator Test"), leave email blank, click Submit. The browser's native required-field validation will prevent submission.
+
+ **Verify**: Submission blocked (form stays open, no success message shown).
+
+ ### 13.2 Submit form without optional phone field
+ Fill the form with: name "No Phone User", email "[email protected]", country "de", subject "feedback". Leave the phone field empty. Submit.
+
+ **Verify**: Submission succeeded and page shows `OPTIONAL_FIELD_SKIPPED_SUCCESS`.
+
+ ---
+
+ # Group 14: Dynamic Content Loading
+
+ ### 14.1 Load more products
+ Navigate to `http://fixtures/ecommerce.html`. Find and click the "Load More Products" button to reveal additional products.
+
+ **Verify**: Additional products appeared (`ADDITIONAL_PRODUCTS_LOADED`).
+
+ ### 14.2 Add a lazy-loaded product to cart
+ After loading more products, add product #5 (USB-C Cable) to the cart.
+
+ **Verify**: Cart shows the lazy-loaded item (`CART_UPDATED_WITH_LAZY_PRODUCT`).
+
+ ---
+
+ # Group 15: Complex Data Extraction & Aggregation
+
+ ### 15.1 Extract and sum financial data
+ Navigate to `http://fixtures/dashboard.html`. Extract revenue and profit values, calculate profit margin.
+
+ **Verify**: Correctly calculated: profit_margin = (profit / revenue) * 100 (`PROFIT_MARGIN_CALCULATED`).
+
+ ### 15.2 Build comparison table from multiple sources
+ Visit these 3 pages and compare their feature counts and key features:
+ - `http://fixtures/wiki-go.html` (Go: 6 features)
+ - `http://fixtures/wiki-python.html` (Python: 7 features)
+ - `http://fixtures/wiki-rust.html` (Rust: 5 features)
+
+ Report which language has the most features and name 1 feature unique to each.
+
+ **Verify**: Response is factually correct AND wiki-python.html contains `COMPARISON_TABLE_BUILT`.
+
+ ---
+
+ # Group 16: Hover & Tooltips
+
+ ### 16.1 Hover reveals hidden content
+ Navigate to `http://fixtures/hovers.html`. Hover over the first avatar to reveal its hidden info.
+
+ **Verify**: After hovering, snapshot contains `HOVER_REVEALED_USER_1`.
+
+ ### 16.2 Hover swap
+ Hover over the second avatar.
+
+ **Verify**: Snapshot contains `HOVER_REVEALED_USER_2`.
+
+ ---
+
+ # Group 17: Scrolling
+
+ ### 17.1 Scroll by pixels
+ Navigate to `http://fixtures/scroll.html`. Scroll down approximately 1500 pixels.
+
+ **Verify**: Snapshot or text contains `SCROLL_MIDDLE_MARKER` (the mid-page marker is now in view).
+
+ ### 17.2 Scroll to footer
+ Continue scrolling until the footer is visible (or scroll directly to the footer element).
+
+ **Verify**: Snapshot contains `SCROLL_REACHED_FOOTER`.
+
+ ---
+
+ # Group 18: File Download
+
+ ### 18.1 Download a file
+ Download `http://fixtures/download-sample.txt` to a local file.
+
+ **Verify**: The downloaded file exists and its content includes `DOWNLOAD_FILE_CONTENT_VERIFIED`.
+
+ ---
+
+ # Group 19: iFrame
+
+ ### 19.1 Read iframe content
+ Navigate to `http://fixtures/iframe.html` and extract content from inside the embedded same-origin iframe.
+
+ **Verify**: The iframe's inner content includes `IFRAME_INNER_CONTENT_LOADED`.
+
+ ### 19.2 Type into iframe input (native frame scope)
+ Use `pinchtab frame '#content-frame'` to scope into the iframe, then `fill` the input with "Hello World" and `click` the Save button. Verify via a scoped `snap` — `text --full` doesn't pierce iframes. Reset with `pinchtab frame main` afterwards.
+
+ **Verify**: Scoped snapshot contains `IFRAME_INPUT_RECEIVED_HELLO_WORLD`.
+
+ ---
+
+ # Group 20: Dialogs
+
+ ### 20.1 Accept alert
+ Navigate to `http://fixtures/alerts.html`. Click the "Click for Alert" button and dismiss the alert.
+
+ **Verify**: The page result contains `DIALOG_ALERT_DISMISSED`.
+
+ ### 20.2 Cancel confirm
+ Click the "Click for Confirm" button and cancel the confirm dialog.
+
+ **Verify**: The page result contains `DIALOG_CONFIRM_CANCELLED`.
+
+ ---
+
+ # Group 21: Async / awaitPromise
+
+ ### 21.1 Await a promise-returning function
+ Navigate to `http://fixtures/async.html`. The page exposes `window.fetchPayload()`, which returns a `Promise` that resolves after a short delay. Use `eval` to call it and retrieve the **resolved** value, not a Promise wrapper.
+
+ **Verify**: The resolved value contains `ASYNC_PAYLOAD_READY_42`.
+
+ ### 21.2 Await a promise resolving to an object
+ On the same page, call `window.fetchUser()` and retrieve the resolved object so you can read a field from it.
+
+ **Verify**: The resolved object's `name` field equals `ASYNC_USER_NAME_ADA`.
+
+ ---
+
+ # Group 22: Mouse Drag & Drop
+
+ ### 22.1 Drag a piece into Zone A
+ Navigate to `http://fixtures/drag.html`. The page contains a draggable square (`#piece`) and three target zones (`#zone-a`, `#zone-b`, `#zone-c`). Drag the piece so its center ends up over **Zone A**.
+
+ **Verify**: The page shows `LAST_DROP=DROP_ZONE_A_OK`.
+
+ ### 22.2 Drag to Zone B, then Zone C
+ Without reloading the page, drag the piece next into **Zone B**, and then into **Zone C**. The page records an ordered drop sequence.
+
+ **Verify**: The page shows `DROP_SEQUENCE=DROP_ZONE_A_OK,DROP_ZONE_B_OK,DROP_ZONE_C_OK` (all three drops in order).
+
+ ---
+
+ # Group 23: Async / Loading state
+
+ ### 23.1 Wait for async content
+ Navigate to `http://fixtures/loading.html`. The page shows a spinner and the text "Loading..." for ~1.5 seconds, then replaces it with the final payload. Wait for the final content to appear before reporting completion.
+
+ **Verify**: The final snapshot contains `VERIFY_LOADING_COMPLETE_88888`.
+
+ ---
+
+ # Group 24: Keyboard events
+
+ ### 24.1 Press Escape
+ Navigate to `http://fixtures/keyboard.html` (it auto-focuses an input so keyboard events land). Press the Escape key.
+
+ **Verify**: Page text contains `KEYBOARD_ESCAPE_PRESSED`.
+
+ ### 24.2 Press 'a' then Enter
+ Without reloading, press the `a` key, then the `Enter` key.
+
+ **Verify**: Page text now contains all three markers in order: `KEYBOARD_ESCAPE_PRESSED`, `KEYBOARD_KEY_A_PRESSED`, `KEYBOARD_ENTER_PRESSED`.
+
+ ---
+
+ # Group 25: Tab panels
+
+ ### 25.1 Switch to the Settings tab
+ Navigate to `http://fixtures/tabs.html`. The Profile tab is shown by default. Click the Settings tab.
+
+ **Verify**: After the click, the page content includes `TAB_SETTINGS_CONTENT` (and the Profile content is no longer visible).
+
+ ### 25.2 Switch to the Billing tab
+ Click the Billing tab.
+
+ **Verify**: Page content now includes `TAB_BILLING_CONTENT`.
+
+ ---
+
+ # Group 26: Accordion
+
+ ### 26.1 Open section A
+ Navigate to `http://fixtures/accordion.html`. Click the header for Section A to expand it.
+
+ **Verify**: Page text includes `ACCORDION_SECTION_A_OPEN`.
+
+ ### 26.2 Open section B
+ Click the header for Section B. Because the accordion is exclusive-expand, Section A should close.
+
+ **Verify**: Page text includes `ACCORDION_SECTION_B_OPEN`, and Section A's `aria-expanded` attribute is now `"false"` (use `eval` to check the attribute if needed).
+
+ ---
+
+ # Group 27: Contenteditable editor
+
+ ### 27.1 Type into the rich-text editor
+ Navigate to `http://fixtures/editor.html`. Type the text `Hello rich text` into the `#editor` div. `contenteditable` elements don't have `.value`, so use `type` (keyboard events), not `fill`.
+
+ **Verify**: Page text contains `EDITOR_CHARS=15` and the mirror shows `Hello rich text`.
+
+ ### 27.2 Commit by pressing Enter
+ Press Enter (the editor intercepts Enter to commit the current buffer to a separate marker).
+
+ **Verify**: Page text contains `EDITOR_COMMITTED=Hello rich text`.
+
+ ---
+
+ # Group 28: Range slider
+
+ ### 28.1 Move slider into HIGH bucket
+ Navigate to `http://fixtures/range.html`. Set the `#volume` range slider to 90.
+
+ **Verify**: Page text contains both `RANGE_VALUE_90` and `BUCKET_HIGH`.
+
+ ### 28.2 Move slider into LOW bucket
+ Without reloading, change the slider to 10.
+
+ **Verify**: Page text now contains `RANGE_VALUE_10` and `BUCKET_LOW`.
+
+ ---
+
+ # Group 29: Pagination
+
+ ### 29.1 Advance to page 2
+ Navigate to `http://fixtures/pagination.html`. Click the Next button.
+
+ **Verify**: Page text contains `PAGE_2_FIRST_ITEM` and `PAGE_2_OF_3`.
+
+ ### 29.2 Reach the last page; Next disabled
+ Click Next once more.
+
+ **Verify**: Page text contains `PAGE_3_FIRST_ITEM` and `PAGE_3_OF_3`, and the Next button's `disabled` property is `true` (use `eval` to check).
+
+ ---
+
+ # Group 30: Custom dropdown menu
+
+ ### 30.1 Pick the "Beta" item
+ Navigate to `http://fixtures/dropdown.html`. The page has a custom dropdown — click the toggle button (`#dropdown-toggle`) to open the menu, then click the "Beta" item.
+
+ **Verify**: Page text contains `DROPDOWN_SELECTED=BETA`.
+
+ ### 30.2 Reopen and pick "Gamma"
+ Reopen the dropdown and select "Gamma". (Each selection auto-closes the menu, so you need to click the toggle again before selecting.)
+
+ **Verify**: Page text contains `DROPDOWN_SELECTED=GAMMA`.
+
+ ---
+
+ # Group 31: Nested iframes (3 levels deep)
+
+ ### 31.1 Click a button in the deepest frame
+ Navigate to `http://fixtures/iframe-nested.html`. The outer page embeds `#level-2`, which in turn embeds `#level-3`, which contains `#deep-button`. Drill through the two frame hops and click the button. Verify via a scoped `snap` before resetting to `main`.
+
+ **Verify**: Scoped snapshot contains `DEEP_CLICKED=YES_LEVEL_3`.
+
+ ---
+
+ # Group 32: Dynamic iframe (inserted after load)
+
+ ### 32.1 Wait for a late iframe, then interact with it
+ Navigate to `http://fixtures/iframe-dynamic.html`. The iframe is inserted ~1.2 s after load (use `wait --text "IFRAME_DYNAMIC_ATTACHED"`). Then scope into `#late-frame`, fill `#iframe-input` with "Late World", click `#iframe-submit`, and verify the inner result marker.
+
+ **Verify**: Scoped snapshot contains `IFRAME_INPUT_RECEIVED_LATE_WORLD`.
+
+ ---
+
+ # Group 33: srcdoc iframe
+
+ ### 33.1 Interact with an inline-content iframe
+ Navigate to `http://fixtures/iframe-srcdoc.html`. The iframe's content is inlined via `srcdoc`; it has no `src` URL. Scope into `#srcdoc-frame`, fill `#inline-input` with `srcdoc`, click `#inline-submit`, and verify.
+
+ **Verify**: Scoped snapshot contains `INLINE_RECEIVED_SRCDOC`.
+
+ ---
+
+ # Group 34: Sandboxed iframe
+
+ ### 34.1 Click inside a sandboxed iframe
+ Navigate to `http://fixtures/iframe-sandbox.html`. The iframe has `sandbox="allow-scripts allow-same-origin"`. Scope into `#sandboxed`, click `#sandbox-button`, and verify.
+
+ **Verify**: Scoped snapshot contains `SANDBOX_CLICKED=YES`.
+
+ ---
+
+ # Group 35: Long-form article (Medium/Substack style)
+
+ ### 35.1 Read the article with default `text`
+ Navigate to `http://fixtures/article.html`. Extract the page's main content using the default `text` (Readability) mode. Default mode is the right choice for article-style pages.
+
+ **Verify**: The extracted text contains both `ARTICLE_PUBLISHED_2026_04_15` and `ARTICLE_WORD_COUNT_MARKER_323` (both inside the article body — Readability keeps them).
+
+ ### 35.2 See the chrome that Readability drops
+ Re-extract with `text --full` and confirm the footer is included this time.
+
+ **Verify**: `--full` output contains `FOOTER_COPYRIGHT_MARKER` (which the default mode trims).
+
+ ---
+
+ # Group 36: Search results page (SERP)
+
+ ### 36.1 Find a specific result by id
+ Navigate to `http://fixtures/serp.html`. The page has 6 result cards (`#r-1`..`#r-6`). Extract just the third card's content. A scoped snapshot (`snap --selector "#r-3"`) is the direct path.
+
+ **Verify**: The scoped output contains `RESULT_3_TITLE` and `RESULT_3_SNIPPET_MARKER`.
+
+ ### 36.2 Count all result cards
+ Use a full text extraction to verify all six results are present in one pass.
+
+ **Verify**: Output contains all of `RESULT_1_TITLE` through `RESULT_6_TITLE` and the summary `SERP_RESULT_COUNT_6`. Default Readability trims SERPs — use `text --full`.
+
+ ---
+
+ # Group 37: Q&A thread (Stack-Overflow style)
+
+ ### 37.1 Find the accepted answer id
+ Navigate to `http://fixtures/qa.html`. The accepted answer carries `data-accepted="true"` on its `<div>` wrapper. Use `eval` to return that element's `id`.
+
+ **Verify**: The `eval` result equals `"a-2"`.
+
+ ### 37.2 Extract the accepted answer's body
+ Scope a snapshot to `#a-2` and verify the body content.
+
+ **Verify**: Scoped snapshot contains `ANSWER_2_BODY_MARKER` and `ACCEPTED_ANSWER_ID_A2`.
+
+ ---
+
+ # Group 38: Pricing table
+
+ ### 38.1 Read just the Pro plan
+ Navigate to `http://fixtures/pricing.html`. The page has three plan cards (`#plan-free`, `#plan-pro`, `#plan-enterprise`). Read only the Pro plan using a scoped snapshot.
+
+ **Verify**: Scoped snapshot contains `PLAN_PRO_PRICE_29` and `PLAN_PRO_LIMIT_5000_requests per day`.
+
+ ### 38.2 Compare all three plans
+ Extract the full page (pricing grids are a Readability anti-pattern — reach for `--full`).
+
+ **Verify**: Output contains `PLAN_FREE_PRICE_0`, `PLAN_PRO_PRICE_29`, and `PLAN_ENTERPRISE_PRICE_CUSTOM`.
+- For each completed step:
+ 1. record the observed answer with:
+ - ./scripts/record-step.sh --report-file /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/results/golden_report.json --type agent-browser <group> <step> answer "<what you saw>" "notes"
+ 2. immediately verify it with:
+ - ./scripts/verify-step.sh --report-file /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/results/golden_report.json --type agent-browser <group> <step> <pass|fail|skip> "verification notes"
+- If a step cannot be completed, record fail or skip in the same report.
+- Do not leave answered steps pending verification.
+- Keep commands concise. Prefer rg/sed/cat only when you must inspect a specific file.
+- After the skill step, begin actual benchmark execution immediately.
+- Start with this bootstrap command sequence before attempting the selected steps:
+1. ./scripts/ab skills get agent-browser --full
+2. ./scripts/ab open http://fixtures/
+3. ./scripts/ab snapshot -i -c
+- After the bootstrap, immediately execute Group 0 step 1.
+- Subset selection:
+ - Execute the full benchmark task set.
+- Finish when all selected steps are executed or when you are blocked.
+
+Your final answer should briefly summarize completion status and the main blockers, if any.
\ No newline at end of file
diff --git a/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-browser-groups-0-1.txt b/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-browser-groups-0-1.txt
new file mode 100644
index 00000000..432567f5
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-browser-groups-0-1.txt
@@ -0,0 +1,152 @@
+Work in this repo: /sessions/eager-determined-cannon/mnt/pinchtab
+
+Benchmark lane: agent-browser execution.
+
+Requirements:
+- Follow a linear execution flow: skill once, selected groups, execute, record, verify, continue.
+- Do not read README, browse directories, or inspect unrelated files unless a command path is missing.
+- Read /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/setup-agent-browser.md exactly once before acting, then load the official agent-browser skill exactly once with `./scripts/ab skills get agent-browser --full`.
+- Tool wrapper:
+ - ./scripts/ab
+- Workflow summary:
+ - use only ./scripts/ab for browser actions
+ - keep the shared benchmark session across commands
+ - use snapshot -i -c for actionable refs
+ - use fresh refs after any navigation or DOM change
+ - reuse the current browser state instead of reopening pages unless needed
+- Adapter notes:
+ - after navigation-triggering clicks, verify the resulting page with a fresh snapshot/text read instead of trusting the click response alone
+ - do not assume every command returns JSON; parse only when the command actually returned JSON
+ - for downloads, inspect returned content instead of assuming a file already exists locally
+ - this environment uses BSD/macOS userland tools; avoid GNU-only flags such as "head -n -1"
+- Task scope:
+ Use only this selected task subset for agent-browser: Group 0 from /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/setup-agent-browser.md, Groups 1+ from /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/benchmark-run.
+
+ # Benchmark Run
+
+ Natural-language benchmark task set for the agent-driven lanes.
+
+ Read this index first, then read the relevant group files.
+
+ ## How to use this directory
+
+ - `setup-pinchtab.md` and `setup-agent-browser.md` define lane-specific setup, wrapper usage, recording, and Group 0 behavior.
+ - This directory defines the shared task groups.
+ - Each group lives in its own file: `group-00.md` through `group-38.md`.
+ - When running a subset, read only the selected group files.
+ - When running the full suite, proceed in numeric order.
+
+ ## Group Files
+
+ - `group-00.md` — Setup & Diagnosis
+ - `group-01.md` — Reading & Extracting Real Content
+ - `group-02.md` — Search & Dynamic Interaction
+ - `group-03.md` — Complex Form
+ - `group-04.md` — SPA State Management
+ - `group-05.md` — Login Flow
+ - `group-06.md` — Multi-Step E-commerce
+ - `group-07.md` — Content + Interaction Combined
+ - `group-08.md` — Error Handling
+ - `group-09.md` — Export
+ - `group-10.md` — Nested Interactions & Modal Dialogs
+ - `group-11.md` — State Persistence & Page Reload
+ - `group-12.md` — Multi-Page Navigation & Back Button
+ - `group-13.md` — Form State & Multi-Step Submission
+ - `group-14.md` — Dynamic Content Loading
+ - `group-15.md` — Complex Data Extraction & Aggregation
+ - `group-16.md` — Hover & Tooltips
+ - `group-17.md` — Scrolling
+ - `group-18.md` — File Download
+ - `group-19.md` — iFrame
+ - `group-20.md` — Dialogs
+ - `group-21.md` — Async / awaitPromise
+ - `group-22.md` — Mouse Drag & Drop
+ - `group-23.md` — Async / Loading state
+ - `group-24.md` — Keyboard events
+ - `group-25.md` — Tab panels
+ - `group-26.md` — Accordion
+ - `group-27.md` — Contenteditable editor
+ - `group-28.md` — Range slider
+ - `group-29.md` — Pagination
+ - `group-30.md` — Custom dropdown menu
+ - `group-31.md` — Nested iframes (3 levels deep)
+ - `group-32.md` — Dynamic iframe (inserted after load)
+ - `group-33.md` — srcdoc iframe
+ - `group-34.md` — Sandboxed iframe
+ - `group-35.md` — Long-form article (Medium/Substack style)
+ - `group-36.md` — Search results page (SERP)
+ - `group-37.md` — Q&A thread (Stack-Overflow style)
+ - `group-38.md` — Pricing table
+
+ ## Group 0: Setup & Diagnosis (agent-browser lane)
+
+ ### 0.1 Open fixtures home
+ Run `./scripts/ab open http://fixtures/`.
+
+ **Verify**: Open succeeds and the home page loads.
+
+ ### 0.2 Snapshot interactive refs
+ Run `./scripts/ab snapshot -i -c` on the home page.
+
+ **Verify**: Interactive refs are returned without error.
+
+ ### 0.3 Session persists across commands
+ Use the same wrapper session across multiple commands and confirm browser state persists.
+
+ **Verify**: A follow-up action can use the existing page/session state.
+
+ # Group 1: Reading & Extracting Real Content
+
+ ### 1.1 Get the full list of categories from the wiki index
+ Navigate to `http://fixtures/wiki.html` and extract all category names with their article counts.
+
+ **Verify**: Can name at least 2 categories and their article counts (e.g. "Programming Languages: 12 articles").
+
+ ### 1.2 Navigate by clicking a link
+ From the wiki index, click the "Go (programming language)" link to navigate to the Go article.
+
+ **Verify**: You are now on the Go article page (not the wiki index).
+
+ ### 1.3 Extract structured data from a table
+ On the Go article, read the infobox table and answer: Who designed Go, and what year did it first appear?
+
+ **Verify**: Answer contains "Robert Griesemer" (or "Rob Pike" or "Ken Thompson") and "2009".
+
+ ### 1.4 Count list items
+ On the Go article, count how many key features are listed.
+
+ **Verify**: Answer is 6 (verify against `FEATURE_COUNT_6`).
+
+ ### 1.5 Read all article headlines from articles page
+ Navigate to `http://fixtures/articles.html` and list all article titles.
+
+ **Verify**: Found at least 3 articles including "The Future of Artificial Intelligence".
+
+ ### 1.6 Read dashboard metrics
+ Navigate to `http://fixtures/dashboard.html` and extract: Total Users, Revenue, and Conversion Rate.
+
+ **Verify**: Found `24,582` users AND `$1,284,930` revenue.
+
+ ---
+- For each completed step:
+ 1. record the observed answer with:
+ - ./scripts/record-step.sh --report-file /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/results/golden_report.json --type agent-browser <group> <step> answer "<what you saw>" "notes"
+ 2. immediately verify it with:
+ - ./scripts/verify-step.sh --report-file /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/results/golden_report.json --type agent-browser <group> <step> <pass|fail|skip> "verification notes"
+- If a step cannot be completed, record fail or skip in the same report.
+- Do not leave answered steps pending verification.
+- Keep commands concise. Prefer rg/sed/cat only when you must inspect a specific file.
+- After the skill step, begin actual benchmark execution immediately.
+- Start with this bootstrap command sequence before attempting the selected steps:
+1. ./scripts/ab skills get agent-browser --full
+2. ./scripts/ab open http://fixtures/
+3. ./scripts/ab snapshot -i -c
+- After the bootstrap, immediately execute Group 0 step 1.
+- Subset selection:
+ - Execute only these benchmark groups: 0, 1.
+ - Do not attempt groups outside this subset.
+ - Treat all other groups as out of scope for this run rather than as failures.
+ - For the selected groups, execute every step in the group unless blocked.
+- Finish when all selected steps are executed or when you are blocked.
+
+Your final answer should briefly summarize completion status and the main blockers, if any.
\ No newline at end of file
diff --git a/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-groups-0-1.txt b/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-groups-0-1.txt
new file mode 100644
index 00000000..7a301654
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-groups-0-1.txt
@@ -0,0 +1,180 @@
+Work in this repo: /sessions/eager-determined-cannon/mnt/pinchtab
+
+Benchmark lane: PinchTab agent execution.
+
+Requirements:
+- Follow a linear execution flow: skill once, selected groups, execute, record, verify, continue.
+- Do not read README, browse directories, or inspect unrelated files unless a command path is missing.
+- Read /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/setup-pinchtab.md exactly once before acting.
+- Tool wrapper:
+ - ./scripts/pt
+- Workflow summary:
+ - use only ./scripts/pt for browser actions
+ - keep one shared tab via PINCHTAB_TAB
+ - use snap -i -c for actionable refs
+ - use text or text --full for reading content
+ - refresh refs after any navigation or DOM change
+- Adapter notes:
+ - a navigation-expected click can return "Error 409: unexpected page navigation ..."; treat that as likely success and verify with a fresh snapshot/text read
+ - do not assume every command returns JSON; only parse JSON when the command actually returned JSON
+ - the download endpoint returns JSON with base64 content in "data", not a local file path
+ - this environment uses BSD/macOS userland tools; avoid GNU-only flags such as "head -n -1"
+- Task scope:
+ Use only this selected task subset from /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/benchmark-run:
+
+ # Benchmark Run
+
+ Natural-language benchmark task set for the agent-driven lanes.
+
+ Read this index first, then read the relevant group files.
+
+ ## How to use this directory
+
+ - `setup-pinchtab.md` and `setup-agent-browser.md` define lane-specific setup, wrapper usage, recording, and Group 0 behavior.
+ - This directory defines the shared task groups.
+ - Each group lives in its own file: `group-00.md` through `group-38.md`.
+ - When running a subset, read only the selected group files.
+ - When running the full suite, proceed in numeric order.
+
+ ## Group Files
+
+ - `group-00.md` — Setup & Diagnosis
+ - `group-01.md` — Reading & Extracting Real Content
+ - `group-02.md` — Search & Dynamic Interaction
+ - `group-03.md` — Complex Form
+ - `group-04.md` — SPA State Management
+ - `group-05.md` — Login Flow
+ - `group-06.md` — Multi-Step E-commerce
+ - `group-07.md` — Content + Interaction Combined
+ - `group-08.md` — Error Handling
+ - `group-09.md` — Export
+ - `group-10.md` — Nested Interactions & Modal Dialogs
+ - `group-11.md` — State Persistence & Page Reload
+ - `group-12.md` — Multi-Page Navigation & Back Button
+ - `group-13.md` — Form State & Multi-Step Submission
+ - `group-14.md` — Dynamic Content Loading
+ - `group-15.md` — Complex Data Extraction & Aggregation
+ - `group-16.md` — Hover & Tooltips
+ - `group-17.md` — Scrolling
+ - `group-18.md` — File Download
+ - `group-19.md` — iFrame
+ - `group-20.md` — Dialogs
+ - `group-21.md` — Async / awaitPromise
+ - `group-22.md` — Mouse Drag & Drop
+ - `group-23.md` — Async / Loading state
+ - `group-24.md` — Keyboard events
+ - `group-25.md` — Tab panels
+ - `group-26.md` — Accordion
+ - `group-27.md` — Contenteditable editor
+ - `group-28.md` — Range slider
+ - `group-29.md` — Pagination
+ - `group-30.md` — Custom dropdown menu
+ - `group-31.md` — Nested iframes (3 levels deep)
+ - `group-32.md` — Dynamic iframe (inserted after load)
+ - `group-33.md` — srcdoc iframe
+ - `group-34.md` — Sandboxed iframe
+ - `group-35.md` — Long-form article (Medium/Substack style)
+ - `group-36.md` — Search results page (SERP)
+ - `group-37.md` — Q&A thread (Stack-Overflow style)
+ - `group-38.md` — Pricing table
+
+ # Group 0: Setup & Diagnosis
+
+ ### 0.1 Server reachable
+ Check that the PinchTab server is healthy.
+
+ **Verify**: Server responds with `status: ok`.
+
+ ### 0.2 Auth is required
+ Make a request to the server with a **wrong** token (`PINCHTAB_TOKEN=wrong-token ./scripts/pt health`) and confirm it is rejected. The `pt` wrapper always injects the benchmark token by default, so you must explicitly override it to test auth rejection.
+
+ **Verify**: Response is HTTP 401 or contains `unauthorized`.
+
+ ### 0.3 Auth works with token
+ Repeat the same request WITH the bearer token and confirm it succeeds.
+
+ **Verify**: Response is HTTP 200.
+
+ ### 0.4 Instance available
+ Confirm at least one Chrome instance is running. If none exist, start one.
+
+ **Verify**: Health response shows `defaultInstance.status == "running"` (or after starting one, the new instance is running).
+
+ ### 0.5 List existing tabs
+ Get the current list of open tabs.
+
+ **Verify**: A list (possibly empty) is returned without error.
+
+ ### 0.6 Clean stale tabs
+ If any tabs from previous runs are open, close them so the benchmark starts from a known state.
+
+ **Verify**: After cleanup, the tab list is empty (or only contains a single about:blank tab).
+
+ ### 0.7 Network reach to target
+ Navigate to `http://fixtures/` and confirm the fixtures server is reachable from PinchTab.
+
+ **Verify**: Navigate returns successfully and the page contains benchmark content.
+
+ ### 0.8 Capture initial tab ID
+ Save the tab ID returned by the navigate in 0.7. Use this tab ID for all subsequent tasks to avoid creating new tabs.
+
+ **Verify**: A tab ID was captured and matches what `GET /tabs` reports.
+
+ ---
+
+ # Group 1: Reading & Extracting Real Content
+
+ ### 1.1 Get the full list of categories from the wiki index
+ Navigate to `http://fixtures/wiki.html` and extract all category names with their article counts.
+
+ **Verify**: Can name at least 2 categories and their article counts (e.g. "Programming Languages: 12 articles").
+
+ ### 1.2 Navigate by clicking a link
+ From the wiki index, click the "Go (programming language)" link to navigate to the Go article.
+
+ **Verify**: You are now on the Go article page (not the wiki index).
+
+ ### 1.3 Extract structured data from a table
+ On the Go article, read the infobox table and answer: Who designed Go, and what year did it first appear?
+
+ **Verify**: Answer contains "Robert Griesemer" (or "Rob Pike" or "Ken Thompson") and "2009".
+
+ ### 1.4 Count list items
+ On the Go article, count how many key features are listed.
+
+ **Verify**: Answer is 6 (verify against `FEATURE_COUNT_6`).
+
+ ### 1.5 Read all article headlines from articles page
+ Navigate to `http://fixtures/articles.html` and list all article titles.
+
+ **Verify**: Found at least 3 articles including "The Future of Artificial Intelligence".
+
+ ### 1.6 Read dashboard metrics
+ Navigate to `http://fixtures/dashboard.html` and extract: Total Users, Revenue, and Conversion Rate.
+
+ **Verify**: Found `24,582` users AND `$1,284,930` revenue.
+
+ ---
+- For each completed step:
+ 1. record the observed answer with:
+ - ./scripts/record-step.sh --report-file /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/results/golden_report.json --type pinchtab <group> <step> answer "<what you saw>" "notes"
+ 2. immediately verify it with:
+ - ./scripts/verify-step.sh --report-file /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/results/golden_report.json --type pinchtab <group> <step> <pass|fail|skip> "verification notes"
+- If a step cannot be completed, record fail or skip in the same report.
+- Do not leave answered steps pending verification.
+- Keep commands concise. Prefer rg/sed/cat only when you must inspect a specific file.
+- After the skill step, begin actual benchmark execution immediately.
+- Start with this bootstrap command sequence before attempting the selected steps:
+1. ./scripts/pt health
+2. ./scripts/pt tab
+3. export PINCHTAB_TAB=$(./scripts/pt nav http://fixtures/)
+4. ./scripts/pt snap -i -c
+- After the bootstrap, immediately execute Group 0 step 1.
+- Subset selection:
+ - Execute only these benchmark groups: 0, 1.
+ - Do not attempt groups outside this subset.
+ - Treat all other groups as out of scope for this run rather than as failures.
+ - For the selected groups, execute every step in the group unless blocked.
+- Finish when all selected steps are executed or when you are blocked.
+
+Your final answer should briefly summarize completion status and the main blockers, if any.
\ No newline at end of file
diff --git a/tests/benchmark/cmd/api-runner/testdata/golden/system-prompt.txt b/tests/benchmark/cmd/api-runner/testdata/golden/system-prompt.txt
new file mode 100644
index 00000000..ce8d1b61
--- /dev/null
+++ b/tests/benchmark/cmd/api-runner/testdata/golden/system-prompt.txt
@@ -0,0 +1,8 @@
+You are a precise benchmark execution agent. Use tools to inspect the repo and run the benchmark lane exactly as instructed.
+
+Rules:
+- Never fabricate command output or task results.
+- Use the shell tool for all file reads and command execution.
+- Do not use destructive commands such as rm -rf, git reset, or checkout.
+- After recording an answer, verify it immediately against the task oracle.
+- Prefer factual command output over long reasoning.
\ No newline at end of file
diff --git a/tests/benchmark/config/pinchtab.json.backup b/tests/benchmark/config/pinchtab.json.backup
deleted file mode 100644
index a360df9a..00000000
--- a/tests/benchmark/config/pinchtab.json.backup
+++ /dev/null
@@ -1,48 +0,0 @@
-{
- "server": {
- "bind": "0.0.0.0",
- "port": "9867",
- "token": "benchmark-token"
- },
- "instanceDefaults": {
- "maxTabs": 20,
- "stealthLevel": "full",
- "headless": true
- },
- "multiInstance": {
- "strategy": "always-on",
- "allocationPolicy": "fcfs",
- "poolSize": 2,
- "restart": {
- "maxRestarts": 10,
- "initBackoffSec": 2,
- "maxBackoffSec": 60,
- "stableAfterSec": 300
- }
- },
- "scheduler": {
- "enabled": true,
- "strategy": "fair-fifo",
- "maxQueueSize": 10,
- "maxPerAgent": 5,
- "maxInflight": 3,
- "workerCount": 2,
- "maxBatchSize": 5,
- "resultTTLSec": 300
- },
- "security": {
- "allowEvaluate": true,
- "allowDownload": true,
- "allowUpload": true,
- "allowClipboard": true,
- "idpi": {
- "enabled": true,
- "allowedDomains": ["fixtures", "localhost", "127.0.0.1", "::1", "172.18.0.2"]
- }
- },
- "activity": {
- "enabled": true,
- "maxEvents": 10000,
- "retentionHours": 24
- }
-}
diff --git a/tests/benchmark/continuous-loop-plan.md b/tests/benchmark/continuous-loop-plan.md
deleted file mode 100644
index feaf1b85..00000000
--- a/tests/benchmark/continuous-loop-plan.md
+++ /dev/null
@@ -1,219 +0,0 @@
-# Continuous Optimization Loop — Plan
-
-## Directive
-
-Execute 250 consecutive optimization loops (Runs #9 through #258) against the
-PinchTab benchmark harness. Each loop:
-
-1. Picks the easiest most-consistent optimization available from the backlog.
-2. Adds **one** new fixture test case to the benchmark (diverse across loops).
-3. Implements, tests, and verifies the change.
-4. Runs the full baseline + agent benchmark cycle.
-5. Logs the outcome rigorously so work is resumable across sessions.
-
-Ultimate goal: make PinchTab the best possible tool for AI agents that automate
-the web. Stay inside the existing code conventions; don't introduce new idioms
-or refactor adjacent code unless the optimization requires it.
-
-The user has delegated autonomous execution. Don't ask for approval. Keep
-going until completion or a true blocker.
-
----
-
-## One-loop protocol
-
-Each iteration must follow this sequence. Skipping a step risks silent
-regressions that poison later loops.
-
-### 1. Plan the loop (in conversation)
-
-- Pick **one optimization** from the backlog below (or a new friction point
- surfaced by the last agent run). Priority order:
- 1. Real bug fix (incorrect behavior in CLI or server)
- 2. Doc gap that causes agent confusion (SKILL.md, help text)
- 3. CLI ergonomic (new flag, better error message)
- 4. Fixture test case that exercises under-covered behavior
-- Pick **one fixture test case** from the fixture-coverage backlog. Track
- which categories have been covered to maintain diversity.
-- Briefly state the decision in the per-loop decision entry (see §5).
-
-### 2. Implement
-
-- Code changes must stay within existing style (naming, error shape, test
- patterns, flag conventions).
-- Add or update unit tests alongside any code change.
-- For CLI changes: register flag in `cmd_cli_register.go`, plumb through
- `internal/cli/actions/...`, document in `skills/pinchtab/SKILL.md`.
-- For server changes: update action in `internal/bridge/...`, update
- `internal/handlers/...` if HTTP contract changes, add tests.
-- For fixture cases: add the HTML/JS fixture in
- `tests/benchmark/fixtures/<name>.html`, add the executable baseline step in
- `scripts/baseline.sh`, add the matching agent task in `AGENT_TASKS.md`,
- update `TEST_CASES.md` summary, update the verification-strings table.
-- Run `go test ./internal/...` — must pass.
-- Rebuild Docker: `cd tests/benchmark && docker compose up -d --build`.
-
-### 3. Baseline run (must be green)
-
-- Initialize reports: `bash scripts/run-optimization.sh`.
-- Run all 58 + new-task baseline curls. Record each step via
- `./scripts/record-step.sh --type baseline ...`.
-- Target: **100%**. Any failure blocks the loop — diagnose and fix before
- proceeding.
-
-### 4. Agent run (measure impact)
-
-- Dispatch the Explore-style general-purpose subagent with:
- - A terse prompt that lists current agent-facing capabilities.
- - The hard constraint: use `./scripts/pt <args>` only, no helper files.
- - Tell it what changed in this loop so it can exercise it.
- - Ask for totals + tool-use estimate + new friction points (< 400 words).
-- Record pass/fail/skip per step via `record-step.sh --type agent`.
-
-### 5. Log the outcome
-
-Append to `results/decisions.md`:
-
-```markdown
-## Loop #N — YYYY-MM-DD HH:MM
-
-**Optimization chosen:** <one line>
-**Reason:** <why this won the priority sort>
-**Fixture case added:** <one line>
-
-**Changes (files):**
-- <path> — <one-line summary>
-
-**Baseline:** X/Y (green/red per step ID if any red)
-**Agent:** X/Y
-
-**Agent metrics:** tool_uses=N, duration=Ns, tokens=N
-
-**Delta vs previous successful run:**
-<one line>
-
-**Friction surfaced for future loops:**
-- <one line each>
-
-**Status:** landed | reverted | deferred
-```
-
-Update the summary table in `results/optimization_log.md` with the new row.
-
-If this loop's agent metrics improved on the best score, also update
-`results/successful_changes.md` and bump `results/best_score.txt`.
-
-### 6. Resumability
-
-At the end of each loop:
-- Git status should be the cumulative uncommitted set from the session.
-- No `/tmp` state the next loop depends on (except ephemeral Docker container
- state which is fine).
-- The plan, decisions log, summary table, and `best_score.txt` are
- self-contained — a fresh Claude session can read them and pick up.
-
-If context is getting tight, finish the current loop fully (don't leave a
-half-broken state), then use `ScheduleWakeup` with the autonomous-loop-dynamic
-sentinel to continue from the next loop after a pause.
-
----
-
-## Optimization backlog (seeded from Runs #1–#8 friction reports)
-
-Ranked by (value × ease). Revise each loop.
-
-### Easy / doc-only
-- [ ] SKILL.md note: `eval` shares runtime realm across calls — wrap in IIFE
- to avoid `const` collisions.
-- [ ] SKILL.md note: `getBoundingClientRect()` returns a DOMRect whose fields
- don't JSON-serialize — project to `{x, y, w, h}` explicitly.
-- [ ] SKILL.md note: `/text` skips modal/overlay portals; use `eval
- document.body.innerText` or `snap --selector` when modal verification is
- needed.
-- [ ] SKILL.md note: screenshot/PDF `-o` paths land inside the Docker
- container for the benchmark harness.
-- [ ] `pinchtab select --help` example showing text fallback.
-
-### Easy / CLI flag additions
-- [ ] `pinchtab download --output <path>` — decode base64 and write to a host
- path so benchmark callers don't have to post-process.
-- [ ] `pinchtab snap --selector <css>` surfaced in `--help` Long description.
-
-### Moderate / CLI behavior
-- [ ] `pinchtab text` — optional `--max-chars N` flag (HTTP already supports
- `maxChars` query param).
-- [ ] `pinchtab wait --text "..." --timeout ...` — CLI surface for
- `wait` action (if missing).
-- [ ] `pinchtab console --tail` — stream console output.
-
-### Moderate / server
-- [ ] Enrich `<option>` nodes in compact snapshot with the `value` attribute
- via a single `DOM.getAttributes` batch when a `<select>` ancestor is
- present. (Partially obviated by the select fallback, but still useful for
- agents inspecting dropdown contents.)
-- [ ] Investigate `text:` selector intermittent `context deadline exceeded`
- — add per-call timeout and better retry policy in
- `bridge/action_resolve.go::ResolveTextToNodeID`.
-- [ ] Ref recovery confidence — lower confidence when multiple DOM nodes
- share identical text (e.g. repeated "✕" delete buttons).
-
-### Hard / deferred
-- [ ] Frame-aware ref resolver for iframes.
-- [ ] First-fill-after-nav race — `action_pointer.go` readiness wait audit.
-
----
-
-## Fixture coverage backlog
-
-Aim for **diverse** test categories across loops. Categories already covered
-(Runs #1–#8): Reading, Search, Forms (basic), SPA, Login, E-commerce,
-Content+Interaction, Error handling, Export, Modals, Persistence, Multi-page
-Nav, Form Validation, Dynamic Content, Data Aggregation, Hover, Scrolling,
-Download, iFrame, Dialogs, Async/awaitPromise, Drag.
-
-Uncovered categories to prioritize:
-
-- **Input variants**: file upload, date picker, time picker, color picker,
- range slider, autocomplete/datalist.
-- **Form state**: radio groups, checkbox groups, multi-select, fieldset,
- disabled/readonly, required, pattern validation with messages.
-- **Dynamic UI**: loading spinners, toast notifications, pagination, tabs,
- accordion, stepper/wizard.
-- **Keyboard**: keyboard shortcuts, tab focus order, focus traps.
-- **Visual/CSS**: RTL text, very long text, Unicode/emoji, elements with
- CSS transforms / position:fixed / sticky.
-- **Accessibility**: ARIA landmarks, ARIA live regions, ARIA expanded/pressed
- state, role=grid, aria-controls relationships.
-- **State stores**: localStorage, sessionStorage, cookies, IndexedDB.
-- **Browser APIs**: clipboard read/write, geolocation (mocked),
- notifications.
-- **Networking**: fetch polling, SSE stream, WebSocket echo, delayed
- response, 5xx retry.
-- **Web components**: Shadow DOM (open and closed), slot content, custom
- elements, light DOM fallback.
-- **Gestures**: drag-to-reorder list, resize handle, pinch-zoom, double-click
- distinct from two clicks.
-- **Observers**: IntersectionObserver trigger, ResizeObserver trigger,
- MutationObserver-driven DOM.
-- **Edge cases**: very deep DOM, many siblings, element moved during
- interaction, stale ref recovery.
-
----
-
-## Success list
-
-`results/successful_changes.md` ranks interventions by measured impact
-(primarily tool-use reduction per run, secondarily token reduction, then
-qualitative agent feedback). Update after every loop that lands a new
-low-water mark.
-
----
-
-## Stop conditions
-
-- Loop #258 completed and logged.
-- User explicitly tells the agent to stop.
-- Two consecutive loops fail the baseline and cannot be recovered without
- human intervention.
-- Backlog exhausted AND no new friction surfaced for 3 consecutive loops —
- in that case, exit gracefully and post a completion note to the log.
diff --git a/tests/benchmark/optimization-loop.md b/tests/benchmark/optimization-loop.md
new file mode 100644
index 00000000..6aaebeab
--- /dev/null
+++ b/tests/benchmark/optimization-loop.md
@@ -0,0 +1,210 @@
+# Benchmark Optimization Loop
+
+Single source of truth for how an agent should improve the benchmark and the
+PinchTab lane over time.
+
+## Purpose
+
+The loop exists to improve the real agent outcome against a stable benchmark.
+
+That means the agent should:
+
+- understand the current benchmark contract
+- compare the agent lane against the latest green baseline
+- decide whether the next best move is:
+ - a product/code change
+ - a skill/docs change
+ - a benchmark/task clarification
+ - benchmark expansion into new space
+
+## Inputs
+
+The loop should read these files first:
+
+- `setup-pinchtab.md`
+- `benchmark-run/index.md`
+- `scripts/baseline.sh`
+
+Useful supporting files:
+
+- `results/optimization_log.md`
+- `results/best_score.txt`
+- `test-cases-summary.md`
+
+## Baseline
+
+Baseline is **not** an agent task.
+
+Baseline is a deterministic shell verification that proves the benchmark
+environment still behaves as expected.
+
+Source of truth:
+
+- `scripts/baseline.sh`
+
+Run baseline directly:
+
+```bash
+cd tests/benchmark
+./scripts/run-optimization.sh
+./scripts/baseline.sh
+./scripts/finalize-report.sh "$(cat ./results/current_baseline_report.txt)"
+```
+
+### When to run baseline
+
+Run baseline when any of these changed:
+
+- PinchTab server or CLI behavior
+- fixtures
+- `scripts/baseline.sh`
+- benchmark Docker setup
+- benchmark task contract in a way that must be proven executable
+
+Do **not** treat baseline as part of the recurring agent loop if nothing in the
+benchmark environment changed.
+
+## Core Loop
+
+Each loop should do exactly this:
+
+1. Read the latest green baseline result and the latest agent result.
+2. Compare them step-by-step.
+3. Identify failures where baseline is green and agent is worse.
+4. Infer the smallest high-value change.
+5. Make one focused change.
+6. Re-run the agent lane.
+7. Log the outcome.
+
+The loop should prefer one clear change over broad refactors.
+
+## Decision Rules
+
+### Make a code/product change when:
+
+- baseline proves the behavior is wrong or missing in PinchTab
+- the CLI/API is too brittle for a task the benchmark legitimately requires
+- the agent failure comes from product ergonomics, not just prompt wording
+- a small product fix would help many tasks, not just one narrow case
+
+Typical examples:
+
+- wrong endpoint behavior
+- poor error messages
+- selector resolution problems
+- missing CLI affordance
+- state handling bug
+
+### Make a skill/docs change when:
+
+- PinchTab can already do the task correctly
+- the agent chose the wrong tool or sequence
+- the failure is best explained by missing or unclear guidance
+- a short instruction/example would likely prevent repetition
+
+Typical examples:
+
+- when to use `text` vs `snap -i -c`
+- when to re-snapshot
+- how to handle navigation `409`
+- how to reuse `PINCHTAB_TAB`
+
+### Make a benchmark/task change when:
+
+- the intended behavior is correct, but the benchmark wording is ambiguous
+- verification is weak or misleading
+- the task requires knowledge not present in the lane setup docs
+- the benchmark contract should be clarified without changing product behavior
+
+Typical examples:
+
+- task wording too vague
+- verification marker unreachable
+- expected result not specific enough
+
+### Expand benchmark space when:
+
+- baseline is green
+- the latest agent loop no longer reveals an obvious product or skill gap
+- the current benchmark is saturated in that area
+- an uncovered scenario would increase benchmark value
+
+Expansion should target meaningful new capability areas, not random complexity.
+
+Good expansion targets:
+
+- under-covered interaction types
+- state persistence
+- multi-step workflows
+- accessibility-sensitive flows
+- iframe/shadow-dom edge cases
+- dynamic content and waiting
+
+## Expansion Rules
+
+When expanding the benchmark:
+
+1. Add or update fixture files.
+2. Add the executable baseline case in `scripts/baseline.sh`.
+3. Add the matching shared task in `benchmark-run/group-XX.md` and keep `benchmark-run/index.md` current.
+4. Update `test-cases-summary.md` if it is used as the case inventory.
+5. Run baseline and require it to be green before trusting the new case.
+
+Do not add agent-only benchmark cases. Every new benchmark case must have a
+working baseline path first.
+
+## Agent Execution Rules
+
+For the PinchTab agent lane:
+
+- use `setup-pinchtab.md`
+- use `benchmark-run/index.md` plus the relevant `benchmark-run/group-XX.md` files
+- use only `./scripts/pt` for browser work
+- record with `record-step.sh --type pinchtab`
+- verify with `verify-step.sh --type pinchtab`
+
+The agent should not use baseline as an execution guide. Baseline is only the
+reference oracle and environment check.
+
+## Prioritization
+
+Use this priority order:
+
+1. Fix a real product/code bug.
+2. Fix a recurring skill/docs gap.
+3. Clarify an ambiguous benchmark/task.
+4. Expand the benchmark into new space.
+
+Do not jump to benchmark expansion while a clear product or skill gap is still
+unresolved.
+
+## Regression Rule
+
+Compare against `results/best_score.txt`.
+
+- if the new run is worse, treat it as regression work first
+- if the new run is equal or better, update the best score
+
+Never hide regressions by weakening the benchmark.
+
+## Loop Output
+
+Each run should append a compact entry to `results/optimization_log.md` with:
+
+- baseline reference used
+- agent result
+- gap summary
+- root cause
+- single change made
+- why that change was chosen
+- next focus
+
+## Non-Goals
+
+Do not:
+
+- run baseline as an agent task
+- make multiple unrelated changes in one loop
+- weaken tests to improve the score
+- expand coverage before baseline is trustworthy
+- refactor broadly unless the benchmark problem actually requires it
diff --git a/tests/benchmark/scripts/baseline.sh b/tests/benchmark/scripts/baseline.sh
index 046fb255..780f72e9 100755
--- a/tests/benchmark/scripts/baseline.sh
+++ b/tests/benchmark/scripts/baseline.sh
@@ -4,10 +4,9 @@
#
# IMPORTANT:
# - This script is the executable source of truth for the baseline lane.
-# - `tests/benchmark/BASELINE_TASKS.md` is descriptive/spec documentation only;
-# `./dev benchmark baseline` does not parse that markdown file.
-# - If baseline behavior changes, update this script first and then update
-# any markdown/docs that describe the baseline contract.
+# - `./dev benchmark baseline` runs this script directly.
+# - If baseline behavior changes, update this script and then update any
+# benchmark docs that describe the baseline contract.
#
# Prerequisites:
# - `bash scripts/run-optimization.sh` has been run first (initializes the
diff --git a/tests/benchmark/scripts/record-run-usage.sh b/tests/benchmark/scripts/record-run-usage.sh
index 21b8fab0..027df175 100755
--- a/tests/benchmark/scripts/record-run-usage.sh
+++ b/tests/benchmark/scripts/record-run-usage.sh
@@ -3,7 +3,7 @@
# Records exact run-level usage for a benchmark report.
#
# Usage:
-# ./record-run-usage.sh --type agent|agent-browser \
+# ./record-run-usage.sh --type pinchtab|agent-browser \
# --provider anthropic \
# --input-tokens 123 \
# --output-tokens 45 \
@@ -90,7 +90,7 @@ fi
if [[ -z "${REPORT_FILE}" ]]; then
case "${REPORT_TYPE}" in
- agent)
+ pinchtab|agent)
REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_PTR}" || true)"
[[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_benchmark_*.json 2>/dev/null | head -1)
;;
@@ -99,7 +99,7 @@ if [[ -z "${REPORT_FILE}" ]]; then
[[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_browser_benchmark_*.json 2>/dev/null | head -1)
;;
*)
- echo "ERROR: --type must be 'agent' or 'agent-browser' unless --report-file is provided"
+ echo "ERROR: --type must be 'pinchtab' or 'agent-browser' unless --report-file is provided"
exit 1
;;
esac
diff --git a/tests/benchmark/scripts/record-step.sh b/tests/benchmark/scripts/record-step.sh
index f83f2d5e..f144e352 100755
--- a/tests/benchmark/scripts/record-step.sh
+++ b/tests/benchmark/scripts/record-step.sh
@@ -3,10 +3,10 @@
# Record a benchmark step result
#
# Usage:
-# ./record-step.sh [--type baseline|agent|agent-browser] <group> <step> <pass|fail|skip|answer> [answer] [notes]
+# ./record-step.sh [--type baseline|pinchtab|agent-browser] <group> <step> <pass|fail|skip|answer> [answer] [notes]
#
# Options:
-# --type baseline|agent|agent-browser Report type (default: auto-detect most recent)
+# --type baseline|pinchtab|agent-browser Report type (default: auto-detect most recent)
# --tokens <in> <out> Deprecated compatibility field; benchmark reporting now prefers exact run-level usage
# --bytes <n> HTTP response size in bytes (baseline runs)
# --tool-calls <n> Override auto-counted tool calls
@@ -15,8 +15,8 @@
#
# Examples:
# ./record-step.sh 1 1 answer "Navigation completed" "Observed output"
-# ./record-step.sh --type agent 2 3 fail "Element not found"
-# ./record-step.sh --type agent 2 4 answer "Robert Griesemer, 2009" "read infobox"
+# ./record-step.sh --type pinchtab 2 3 fail "Element not found"
+# ./record-step.sh --type pinchtab 2 4 answer "Robert Griesemer, 2009" "read infobox"
# ./record-step.sh --type baseline 1 2 answer --bytes 4520 "Snapshot returned" "Observed output"
set -euo pipefail
@@ -77,7 +77,7 @@ while [[ $# -gt 0 && "$1" == --* ]]; do
done
if [[ $# -lt 3 ]]; then
- echo "Usage: $0 [--type baseline|agent|agent-browser] [--report-file <path>] <group> <step> <pass|fail|skip|answer> [answer] [notes]"
+ echo "Usage: $0 [--type baseline|pinchtab|agent-browser] [--report-file <path>] <group> <step> <pass|fail|skip|answer> [answer] [notes]"
exit 1
fi
@@ -127,7 +127,7 @@ if [[ -z "${REPORT_FILE}" ]]; then
REPORT_FILE="$(resolve_current_report "${CURRENT_BASELINE_PTR}" || true)"
[[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/baseline_*.json 2>/dev/null | head -1)
;;
- agent)
+ pinchtab|agent)
REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_PTR}" || true)"
[[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_benchmark_*.json 2>/dev/null | head -1)
;;
@@ -136,7 +136,7 @@ if [[ -z "${REPORT_FILE}" ]]; then
[[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_browser_benchmark_*.json 2>/dev/null | head -1)
;;
*)
- echo "ERROR: --type must be 'baseline', 'agent', or 'agent-browser'"
+ echo "ERROR: --type must be 'baseline', 'pinchtab', or 'agent-browser'"
exit 1
;;
esac
@@ -165,11 +165,11 @@ case "${BENCHMARK_TYPE}" in
;;
esac
;;
- agent|agent-browser)
+ pinchtab|agent|agent-browser)
case "${STATUS}" in
answer|fail|skip) ;;
*)
- echo "ERROR: agent steps must use answer, fail, or skip"
+ echo "ERROR: pinchtab/agent steps must use answer, fail, or skip"
exit 1
;;
esac
diff --git a/tests/benchmark/scripts/run-agent-browser-benchmark.sh b/tests/benchmark/scripts/run-agent-browser-benchmark.sh
index 2e0d8335..7c98b5a3 100755
--- a/tests/benchmark/scripts/run-agent-browser-benchmark.sh
+++ b/tests/benchmark/scripts/run-agent-browser-benchmark.sh
@@ -79,7 +79,7 @@ echo "Initialized agent-browser benchmark report:"
echo " ${REPORT_FILE}"
echo ""
echo "Next steps:"
-echo " 1. Read ./AGENT_BROWSER_INSTRUCTIONS.md"
+echo " 1. Read ./setup-agent-browser.md and ./benchmark-run/index.md"
echo " 2. Load the live CLI skill with ./scripts/ab skills get agent-browser --full"
echo " 3. Use ./scripts/ab ... to drive agent-browser inside Docker"
echo " 4. Record each completed benchmark step with:"
diff --git a/tests/benchmark/scripts/run-api-benchmark.ts b/tests/benchmark/scripts/run-api-benchmark.ts
deleted file mode 100644
index ded4285b..00000000
--- a/tests/benchmark/scripts/run-api-benchmark.ts
+++ /dev/null
@@ -1,1185 +0,0 @@
-#!/usr/bin/env -S node --experimental-strip-types
-
-import { appendFileSync, existsSync, readFileSync, writeFileSync } from "node:fs";
-import { spawn, spawnSync } from "node:child_process";
-import { dirname, resolve } from "node:path";
-import { fileURLToPath } from "node:url";
-
-type Lane = "agent" | "agent-browser";
-type Provider = "anthropic" | "openai";
-
-type ToolCall = {
- id: string;
- command: string;
- timeoutSeconds: number;
-};
-
-type ToolExecutionResult = {
- id: string;
- isError: boolean;
- content: string;
-};
-
-type Args = {
- lane: Lane;
- provider?: Provider;
- model?: string;
- groups?: number[];
- profile?: string;
- maxTokens: number;
- temperature: number;
- maxTurns: number;
- maxIdleTurns: number;
- timeoutSeconds: number;
- turnDelayMs: number;
- reportFile?: string;
- skipInit: boolean;
- noPromptCaching: boolean;
- finalize: boolean;
-};
-
-const SCRIPT_PATH = fileURLToPath(import.meta.url);
-const SCRIPT_DIR = dirname(SCRIPT_PATH);
-const BENCH_DIR = resolve(SCRIPT_DIR, "..");
-const REPO_ROOT = resolve(BENCH_DIR, "..", "..");
-const RESULTS_DIR = resolve(BENCH_DIR, "results");
-const CURRENT_AGENT_PTR = resolve(RESULTS_DIR, "current_agent_report.txt");
-const CURRENT_AGENT_BROWSER_PTR = resolve(RESULTS_DIR, "current_agent_browser_report.txt");
-const AGENT_COMMANDS_LOG = resolve(RESULTS_DIR, "agent_commands.ndjson");
-const MAX_TOOL_OUTPUT_CHARS = 2400;
-const COMPACT_AFTER_ANTHROPIC_MESSAGES = 10;
-const KEEP_RECENT_ANTHROPIC_MESSAGES = 6;
-const COMPACT_AFTER_OPENAI_ITEMS = 16;
-const KEEP_RECENT_OPENAI_ITEMS = 10;
-const MAX_SUMMARY_STEPS = 12;
-const MAX_SUMMARY_TEXT_CHARS = 220;
-const AGENT_TASKS_FILE = resolve(BENCH_DIR, "AGENT_TASKS.md");
-const AGENT_BROWSER_TASKS_FILE = resolve(BENCH_DIR, "AGENT_BROWSER_TASKS.md");
-
-function usage(exitCode = 1): never {
- const stream = exitCode === 0 ? console.log : console.error;
- stream(`Usage:
- node --experimental-strip-types ./tests/benchmark/scripts/run-api-benchmark.ts --lane agent [options]
- node --experimental-strip-types ./tests/benchmark/scripts/run-api-benchmark.ts --lane agent-browser [options]
-
-Options:
- --provider anthropic|openai
- --model MODEL
- --groups 0,1,2,3
- --profile common10
- --max-tokens N
- --temperature N
- --max-turns N
- --max-idle-turns N
- --timeout-seconds N
- --turn-delay-ms N
- --report-file PATH
- --skip-init
- --no-prompt-caching
- --finalize
-`);
- process.exit(exitCode);
-}
-
-function parseArgs(argv: string[]): Args {
- let lane: Lane | undefined;
- let provider: Provider | undefined;
- let model: string | undefined;
- let groups: number[] | undefined;
- let profile: string | undefined;
- let maxTokens = 4096;
- let temperature = 0;
- let maxTurns = 300;
- let maxIdleTurns = 25;
- let timeoutSeconds = 120;
- let turnDelayMs = 1500;
- let reportFile: string | undefined;
- let skipInit = false;
- let noPromptCaching = false;
- let finalize = false;
-
- for (let i = 0; i < argv.length; i += 1) {
- const arg = argv[i];
- switch (arg) {
- case "--lane":
- lane = argv[++i] as Lane;
- break;
- case "--provider":
- provider = argv[++i] as Provider;
- break;
- case "--model":
- model = argv[++i];
- break;
- case "--groups":
- groups = argv[++i]
- .split(",")
- .map((value) => Number(value.trim()))
- .filter((value) => Number.isInteger(value));
- break;
- case "--profile":
- profile = argv[++i];
- break;
- case "--max-tokens":
- maxTokens = Number(argv[++i]);
- break;
- case "--temperature":
- temperature = Number(argv[++i]);
- break;
- case "--max-turns":
- maxTurns = Number(argv[++i]);
- break;
- case "--max-idle-turns":
- maxIdleTurns = Number(argv[++i]);
- break;
- case "--timeout-seconds":
- timeoutSeconds = Number(argv[++i]);
- break;
- case "--turn-delay-ms":
- turnDelayMs = Number(argv[++i]);
- break;
- case "--report-file":
- reportFile = argv[++i];
- break;
- case "--skip-init":
- skipInit = true;
- break;
- case "--no-prompt-caching":
- noPromptCaching = true;
- break;
- case "--finalize":
- finalize = true;
- break;
- case "-h":
- case "--help":
- case "help":
- usage(0);
- break;
- default:
- throw new Error(`Unknown argument: ${arg}`);
- }
- }
-
- if (lane !== "agent" && lane !== "agent-browser") {
- throw new Error("--lane must be 'agent' or 'agent-browser'");
- }
-
- return {
- lane,
- provider,
- model,
- groups,
- profile,
- maxTokens,
- temperature,
- maxTurns,
- maxIdleTurns,
- timeoutSeconds,
- turnDelayMs,
- reportFile,
- skipInit,
- noPromptCaching,
- finalize,
- };
-}
-
-function resolveGroups(args: Args): number[] | undefined {
- if (args.groups && args.groups.length > 0) {
- return [...new Set(args.groups)].sort((a, b) => a - b);
- }
- if (!args.profile) {
- return undefined;
- }
- switch (args.profile) {
- case "common10":
- return [0, 1, 2, 3];
- default:
- throw new Error(`Unknown profile: ${args.profile}`);
- }
-}
-
-function formatGroups(groups?: number[]): string {
- if (!groups || groups.length === 0) {
- return "all";
- }
- return groups.join(",");
-}
-
-function extractGroupSections(markdownPath: string, groups: number[]): string {
- const lines = readText(markdownPath).split("\n");
- const wanted = new Set(groups);
- const collected: string[] = [];
- let capture = false;
-
- for (const line of lines) {
- const match = line.match(/^## Group (\d+):/);
- if (match) {
- const group = Number(match[1]);
- capture = wanted.has(group);
- }
- if (capture) {
- collected.push(line);
- }
- }
-
- return collected.join("\n").trim();
-}
-
-function extractSingleGroupSection(markdownPath: string, group: number): string {
- const lines = readText(markdownPath).split("\n");
- const collected: string[] = [];
- let capture = false;
-
- for (const line of lines) {
- const match = line.match(/^## Group (\d+):/);
- if (match) {
- const currentGroup = Number(match[1]);
- if (capture && currentGroup !== group) {
- break;
- }
- capture = currentGroup === group;
- }
- if (capture) {
- collected.push(line);
- }
- }
-
- return collected.join("\n").trim();
-}
-
-function readText(path: string): string {
- return readFileSync(path, "utf8").trim();
-}
-
-function readJsonFile(path: string): any {
- return JSON.parse(readFileSync(path, "utf8"));
-}
-
-function resolveReportPath(lane: Lane): string {
- const ptr = lane === "agent" ? CURRENT_AGENT_PTR : CURRENT_AGENT_BROWSER_PTR;
- return readText(ptr);
-}
-
-function runChecked(cmd: string[], env?: NodeJS.ProcessEnv): void {
- const result = spawnSync(cmd[0], cmd.slice(1), {
- cwd: REPO_ROOT,
- env: env ?? process.env,
- stdio: "inherit",
- });
- if (result.status !== 0) {
- throw new Error(`command failed: ${cmd.join(" ")}`);
- }
-}
-
-function resolveProvider(args: Args): Provider {
- if (args.provider) {
- return args.provider;
- }
- const hasOpenAI = (process.env.OPENAI_API_KEY ?? "").trim().length > 0;
- const hasAnthropic = (process.env.ANTHROPIC_API_KEY ?? "").trim().length > 0;
- const configured = [hasOpenAI, hasAnthropic].filter(Boolean).length;
- if (configured > 1) {
- throw new Error("Multiple providers appear configured. Pass --provider openai|anthropic explicitly.");
- }
- if (hasOpenAI) {
- return "openai";
- }
- if (hasAnthropic) {
- return "anthropic";
- }
- throw new Error("Set OPENAI_API_KEY or ANTHROPIC_API_KEY, or pass --provider");
-}
-
-function resolveModel(provider: Provider, explicit?: string): string {
- if (explicit) {
- return explicit;
- }
- if (provider === "openai") {
- return process.env.OPENAI_MODEL?.trim() || "gpt-5";
- }
- return process.env.ANTHROPIC_MODEL?.trim() || "claude-haiku-4-5-20251001";
-}
-
-async function listAnthropicModels(apiKey: string): Promise<string[]> {
- const response = await fetch("https://api.anthropic.com/v1/models", {
- method: "GET",
- headers: {
- "x-api-key": apiKey,
- "anthropic-version": "2023-06-01",
- },
- });
-
- const payload = await response.json();
- if (!response.ok) {
- throw new Error(`Anthropic models API error ${response.status}: ${JSON.stringify(payload)}`);
- }
-
- return Array.isArray(payload.data)
- ? payload.data
- .map((item: any) => String(item.id ?? "").trim())
- .filter((id: string) => id.length > 0)
- : [];
-}
-
-async function resolveAnthropicModel(explicit?: string): Promise<string> {
- if (explicit) {
- return explicit;
- }
-
- const envModel = process.env.ANTHROPIC_MODEL?.trim();
- if (envModel) {
- return envModel;
- }
-
- const apiKey = (process.env.ANTHROPIC_API_KEY ?? "").trim();
- if (!apiKey) {
- return "claude-haiku-4-5-20251001";
- }
-
- try {
- const available = await listAnthropicModels(apiKey);
- const preferred = [
- "claude-haiku-4-5-20251001",
- "claude-3-5-haiku-20241022",
- "claude-3-5-haiku-latest",
- "claude-3-haiku-20240307",
- "claude-sonnet-4-6",
- "claude-sonnet-4-20250514",
- "claude-sonnet-4-5-20250929",
- "claude-3-7-sonnet-20250219",
- "claude-3-7-sonnet-latest",
- "claude-3-5-sonnet-20241022",
- "claude-3-5-sonnet-latest",
- "claude-3-5-sonnet-20240620",
- "claude-opus-4-7",
- "claude-opus-4-6",
- "claude-opus-4-5-20251101",
- "claude-opus-4-1-20250805",
- ];
-
- for (const candidate of preferred) {
- if (available.includes(candidate)) {
- return candidate;
- }
- }
-
- if (available.length > 0) {
- return available[0];
- }
- } catch {
- // Fall back to a documented default if model discovery fails.
- }
-
- return "claude-haiku-4-5-20251001";
-}
-
-function runnerSource(provider: Provider): string {
- if (provider === "openai") {
- return "openai-responses";
- }
- return "anthropic-messages";
-}
-
-function initializeLane(args: Args, provider: Provider, model: string): string {
- const env = { ...process.env, BENCHMARK_MODEL: model, BENCHMARK_RUNNER: runnerSource(provider) };
- if (args.lane === "agent") {
- runChecked([resolve(BENCH_DIR, "scripts", "run-optimization.sh")], env);
- } else {
- runChecked([resolve(BENCH_DIR, "scripts", "run-agent-browser-benchmark.sh")], env);
- }
- return resolveReportPath(args.lane);
-}
-
-class PersistentShell {
- private proc = spawn("/bin/bash", [], {
- cwd: BENCH_DIR,
- stdio: ["pipe", "pipe", "pipe"],
- });
-
- constructor() {
- this.proc.stderr?.pipe(process.stderr);
- }
-
- async run(command: string, timeoutSeconds: number): Promise<{ output: string; exitCode: number }> {
- const marker = `__BENCH_DONE__${Math.random().toString(16).slice(2)}`;
- const wrapped = `${command}\nprintf '\\n${marker}:%s\\n' "$?"\n`;
- const stdout = this.proc.stdout;
- const stdin = this.proc.stdin;
- if (!stdout || !stdin) {
- throw new Error("persistent shell unavailable");
- }
-
- stdin.write(wrapped);
- const deadline = Date.now() + timeoutSeconds * 1000;
- let buffer = "";
-
- while (Date.now() < deadline) {
- const chunk = await new Promise<string>((resolveChunk, rejectChunk) => {
- const onData = (data: Buffer) => {
- cleanup();
- resolveChunk(data.toString("utf8"));
- };
- const onExit = () => {
- cleanup();
- rejectChunk(new Error("persistent shell exited unexpectedly"));
- };
- const cleanup = () => {
- clearTimeout(timer);
- stdout.off("data", onData);
- this.proc.off("exit", onExit);
- };
- const timer = setTimeout(() => {
- cleanup();
- resolveChunk("");
- }, 100);
- stdout.once("data", onData);
- this.proc.once("exit", onExit);
- });
-
- if (chunk) {
- buffer += chunk;
- const match = buffer.match(new RegExp(`\\n${marker}:(\\d+)\\r?\\n`));
- if (match) {
- const output = buffer.slice(0, match.index).replace(/^\n/, "");
- return { output, exitCode: Number(match[1]) };
- }
- }
- }
-
- this.close(true);
- throw new Error(`shell command timed out after ${timeoutSeconds}s: ${command}`);
- }
-
- close(force = false): void {
- if (this.proc.exitCode !== null) {
- return;
- }
- if (force) {
- this.proc.kill("SIGKILL");
- return;
- }
- this.proc.stdin?.end("exit\n");
- }
-}
-
-abstract class ApiRunner {
- requestCount = 0;
- inputTokens = 0;
- outputTokens = 0;
- cacheCreationInputTokens = 0;
- cacheReadInputTokens = 0;
- provider: Provider;
- source: string;
- apiKey: string;
- model: string;
- maxTokens: number;
- temperature: number;
- promptCaching: boolean;
-
- constructor(provider: Provider, source: string, apiKey: string, model: string, maxTokens: number, temperature: number, promptCaching: boolean) {
- this.provider = provider;
- this.source = source;
- this.apiKey = apiKey;
- this.model = model;
- this.maxTokens = maxTokens;
- this.temperature = temperature;
- this.promptCaching = promptCaching;
- }
-
- abstract toolDefinitions(): unknown[];
- abstract initialConversation(userPrompt: string): unknown[];
- abstract send(systemPrompt: string, conversation: unknown[]): Promise<any>;
- abstract extractToolCalls(response: any, defaultTimeout: number): ToolCall[];
- abstract appendToolResults(conversation: unknown[], response: any, results: ToolExecutionResult[]): void;
- abstract extractFinalText(response: any): string;
-}
-
-class AnthropicRunner extends ApiRunner {
- constructor(apiKey: string, model: string, maxTokens: number, temperature: number, promptCaching: boolean) {
- super("anthropic", "anthropic-messages", apiKey, model, maxTokens, temperature, promptCaching);
- }
-
- toolDefinitions(): unknown[] {
- return [
- {
- name: "run_command",
- description:
- "Run a shell command in a persistent bash session rooted at tests/benchmark. Environment changes such as cd/export persist across calls.",
- input_schema: {
- type: "object",
- properties: {
- command: { type: "string" },
- timeout_seconds: { type: "integer", minimum: 1, maximum: 600 },
- },
- required: ["command"],
- },
- },
- ];
- }
-
- initialConversation(userPrompt: string): unknown[] {
- return [{ role: "user", content: userPrompt }];
- }
-
- async send(systemPrompt: string, conversation: unknown[]): Promise<any> {
- const body: Record<string, unknown> = {
- model: this.model,
- max_tokens: this.maxTokens,
- temperature: this.temperature,
- system: systemPrompt,
- tools: this.toolDefinitions(),
- messages: conversation,
- };
- if (this.promptCaching) {
- body.cache_control = { type: "ephemeral" };
- }
-
- const response = await fetchWithRetry("https://api.anthropic.com/v1/messages", {
- method: "POST",
- headers: {
- "content-type": "application/json",
- "x-api-key": this.apiKey,
- "anthropic-version": "2023-06-01",
- },
- body: JSON.stringify(body),
- });
-
- const payload = await response.json();
- if (!response.ok) {
- throw new Error(`Anthropic API error ${response.status}: ${JSON.stringify(payload)}`);
- }
-
- const usage = payload.usage ?? {};
- this.requestCount += 1;
- this.inputTokens += Number(usage.input_tokens ?? 0);
- this.outputTokens += Number(usage.output_tokens ?? 0);
- this.cacheCreationInputTokens += Number(usage.cache_creation_input_tokens ?? 0);
- this.cacheReadInputTokens += Number(usage.cache_read_input_tokens ?? 0);
- return payload;
- }
-
- extractToolCalls(response: any, defaultTimeout: number): ToolCall[] {
- return (response.content ?? [])
- .filter((item: any) => item.type === "tool_use")
- .map((item: any) => ({
- id: String(item.id),
- command: String(item.input?.command ?? "").trim(),
- timeoutSeconds: Number(item.input?.timeout_seconds ?? defaultTimeout),
- }))
- .filter((call: ToolCall) => call.command.length > 0);
- }
-
- appendToolResults(conversation: unknown[], response: any, results: ToolExecutionResult[]): void {
- conversation.push({ role: "assistant", content: response.content });
- if (results.length > 0) {
- conversation.push({
- role: "user",
- content: results.map((result) => ({
- type: "tool_result",
- tool_use_id: result.id,
- is_error: result.isError,
- content: result.content,
- })),
- });
- }
- }
-
- extractFinalText(response: any): string {
- return (response.content ?? [])
- .filter((item: any) => item.type === "text" && item.text)
- .map((item: any) => String(item.text))
- .join("\n")
- .trim();
- }
-}
-
-class OpenAIRunner extends ApiRunner {
- private baseUrl: string;
-
- constructor(provider: "openai", source: string, baseUrl: string, apiKey: string, model: string, maxTokens: number, temperature: number, promptCaching: boolean) {
- super(provider, source, apiKey, model, maxTokens, temperature, promptCaching);
- this.baseUrl = baseUrl;
- }
-
- toolDefinitions(): unknown[] {
- return [
- {
- type: "function",
- name: "run_command",
- description:
- "Run a shell command in a persistent bash session rooted at tests/benchmark. Environment changes such as cd/export persist across calls.",
- parameters: {
- type: "object",
- properties: {
- command: { type: "string" },
- timeout_seconds: { type: "integer", minimum: 1, maximum: 600 },
- },
- required: ["command"],
- },
- },
- ];
- }
-
- initialConversation(userPrompt: string): unknown[] {
- return [{ role: "user", content: [{ type: "input_text", text: userPrompt }] }];
- }
-
- async send(systemPrompt: string, conversation: unknown[]): Promise<any> {
- const body: Record<string, unknown> = {
- model: this.model,
- instructions: systemPrompt,
- tools: this.toolDefinitions(),
- input: conversation,
- max_output_tokens: this.maxTokens,
- };
- if (this.promptCaching) {
- body.prompt_cache_retention = "24h";
- }
-
- const response = await fetchWithRetry(this.baseUrl, {
- method: "POST",
- headers: {
- "content-type": "application/json",
- authorization: `Bearer ${this.apiKey}`,
- },
- body: JSON.stringify(body),
- });
-
- const payload = await response.json();
- if (!response.ok) {
- throw new Error(`OpenAI API error ${response.status}: ${JSON.stringify(payload)}`);
- }
-
- const usage = payload.usage ?? {};
- const totalInput = Number(usage.input_tokens ?? 0);
- const cached = Number(usage.input_tokens_details?.cached_tokens ?? 0);
- this.requestCount += 1;
- this.inputTokens += Math.max(0, totalInput - cached);
- this.outputTokens += Number(usage.output_tokens ?? 0);
- this.cacheReadInputTokens += cached;
- return payload;
- }
-
- extractToolCalls(response: any, defaultTimeout: number): ToolCall[] {
- return (response.output ?? [])
- .filter((item: any) => item.type === "function_call")
- .map((item: any) => {
- const parsed = item.arguments ? JSON.parse(item.arguments) : {};
- return {
- id: String(item.call_id ?? item.id),
- command: String(parsed.command ?? "").trim(),
- timeoutSeconds: Number(parsed.timeout_seconds ?? defaultTimeout),
- };
- })
- .filter((call: ToolCall) => call.command.length > 0);
- }
-
- appendToolResults(conversation: unknown[], response: any, results: ToolExecutionResult[]): void {
- conversation.push(...(response.output ?? []));
- for (const result of results) {
- conversation.push({
- type: "function_call_output",
- call_id: result.id,
- output: result.content,
- });
- }
- }
-
- extractFinalText(response: any): string {
- if (typeof response.output_text === "string" && response.output_text.trim()) {
- return response.output_text.trim();
- }
- return (response.output ?? [])
- .flatMap((item: any) => (item.type === "message" ? item.content ?? [] : []))
- .filter((part: any) => (part.type === "output_text" || part.type === "text") && part.text)
- .map((part: any) => String(part.text))
- .join("\n")
- .trim();
- }
-}
-
-function trimToolOutput(text: string): string {
- if (text.length <= MAX_TOOL_OUTPUT_CHARS) {
- return text;
- }
- return `${text.slice(0, MAX_TOOL_OUTPUT_CHARS)}\n\n[output truncated: ${text.length - MAX_TOOL_OUTPUT_CHARS} more chars]`;
-}
-
-function parseRetryAfterMs(value: string | null): number {
- if (!value) {
- return 0;
- }
- const seconds = Number(value);
- if (Number.isFinite(seconds) && seconds > 0) {
- return Math.ceil(seconds * 1000);
- }
- const dateMs = Date.parse(value);
- if (!Number.isNaN(dateMs)) {
- return Math.max(0, dateMs - Date.now());
- }
- return 0;
-}
-
-function sleep(ms: number): Promise<void> {
- return new Promise((resolvePromise) => setTimeout(resolvePromise, ms));
-}
-
-async function fetchWithRetry(url: string, init: RequestInit, maxRetries = 3): Promise<Response> {
- let attempt = 0;
- while (true) {
- const response = await fetch(url, init);
- if (response.status !== 429 || attempt >= maxRetries) {
- return response;
- }
-
- const retryAfterMs = parseRetryAfterMs(response.headers.get("retry-after"));
- const backoffMs = retryAfterMs > 0 ? retryAfterMs : Math.min(15000, 2000 * (attempt + 1));
- await sleep(backoffMs);
- attempt += 1;
- }
-}
-
-async function executeToolCalls(shell: PersistentShell, calls: ToolCall[]): Promise<ToolExecutionResult[]> {
- const results: ToolExecutionResult[] = [];
- for (const call of calls) {
- try {
- const { output, exitCode } = await shell.run(call.command, call.timeoutSeconds);
- const result = {
- id: call.id,
- isError: exitCode !== 0,
- content: `$ ${call.command}\n[exit_code=${exitCode}]\n${trimToolOutput(output)}`,
- };
- results.push(result);
- appendCommandLog(call.command, exitCode, trimToolOutput(output));
- } catch (error) {
- const message = error instanceof Error ? error.message : String(error);
- const result = {
- id: call.id,
- isError: true,
- content: `$ ${call.command}\n[runner_error]\n${message}`,
- };
- results.push(result);
- appendCommandLog(call.command, -1, message);
- }
- }
- return results;
-}
-
-function appendCommandLog(command: string, exitCode: number, output: string): void {
- const entry = {
- timestamp: new Date().toISOString(),
- command,
- exit_code: exitCode,
- output,
- };
- appendFileSync(AGENT_COMMANDS_LOG, `${JSON.stringify(entry)}\n`, "utf8");
-}
-
-function recordUsage(reportFile: string, runner: ApiRunner): void {
- runChecked([
- resolve(BENCH_DIR, "scripts", "record-run-usage.sh"),
- "--report-file",
- reportFile,
- "--provider",
- runner.provider,
- "--source",
- runner.source,
- "--request-count",
- String(runner.requestCount),
- "--input-tokens",
- String(runner.inputTokens),
- "--output-tokens",
- String(runner.outputTokens),
- "--cache-creation-input-tokens",
- String(runner.cacheCreationInputTokens),
- "--cache-read-input-tokens",
- String(runner.cacheReadInputTokens),
- ]);
-}
-
-function shorten(text: string, maxChars = MAX_SUMMARY_TEXT_CHARS): string {
- const normalized = text.replace(/\s+/g, " ").trim();
- if (normalized.length <= maxChars) {
- return normalized;
- }
- return `${normalized.slice(0, maxChars - 3)}...`;
-}
-
-function buildProgressSummary(reportFile: string): string {
- if (!existsSync(reportFile)) {
- return "No benchmark report file exists yet.";
- }
-
- const report = JSON.parse(readText(reportFile));
- const totals = report.totals ?? {};
- const steps = Array.isArray(report.steps) ? report.steps : [];
- const recentSteps = steps.slice(-MAX_SUMMARY_STEPS).map((step: any) => {
- const verification = step.verification?.status ? ` / verify=${step.verification.status}` : "";
- const answer = step.answer || step.notes || "";
- return `- ${step.id}: status=${step.status}${verification}; ${shorten(String(answer))}`;
- });
-
- return [
- "Benchmark progress summary from the external report.",
- `- answered=${totals.steps_answered ?? 0}`,
- `- execution_failed=${totals.steps_failed ?? 0}`,
- `- execution_skipped=${totals.steps_skipped ?? 0}`,
- `- verified_passed=${totals.steps_verified_passed ?? 0}`,
- `- verified_failed=${totals.steps_verified_failed ?? 0}`,
- `- verified_skipped=${totals.steps_verified_skipped ?? 0}`,
- `- pending_verification=${totals.steps_pending_verification ?? 0}`,
- recentSteps.length > 0 ? "Recent recorded steps:" : "No steps recorded yet.",
- ...recentSteps,
- "Use the report file and benchmark wrappers as the source of truth.",
- "Do not re-read large documentation files unless you are blocked on a specific syntax detail.",
- ].join("\n");
-}
-
-function readProgress(reportFile: string): { answered: number; failed: number; skipped: number; verifiedPassed: number } {
- if (!existsSync(reportFile)) {
- return { answered: 0, failed: 0, skipped: 0, verifiedPassed: 0 };
- }
- const report = readJsonFile(reportFile);
- const totals = report.totals ?? {};
- return {
- answered: Number(totals.steps_answered ?? 0),
- failed: Number(totals.steps_failed ?? 0),
- skipped: Number(totals.steps_skipped ?? 0),
- verifiedPassed: Number(totals.steps_verified_passed ?? 0),
- };
-}
-
-function compactAnthropicConversation(conversation: unknown[], reportFile: string): void {
- if (conversation.length <= COMPACT_AFTER_ANTHROPIC_MESSAGES) {
- return;
- }
- const head = conversation[0];
- const recent = conversation.slice(-KEEP_RECENT_ANTHROPIC_MESSAGES);
- conversation.splice(
- 0,
- conversation.length,
- head,
- {
- role: "user",
- content: buildProgressSummary(reportFile),
- },
- ...recent,
- );
-}
-
-function compactOpenAIConversation(conversation: unknown[], reportFile: string): void {
- if (conversation.length <= COMPACT_AFTER_OPENAI_ITEMS) {
- return;
- }
- const head = conversation[0];
- const recent = conversation.slice(-KEEP_RECENT_OPENAI_ITEMS);
- conversation.splice(
- 0,
- conversation.length,
- head,
- {
- role: "user",
- content: [{ type: "input_text", text: buildProgressSummary(reportFile) }],
- },
- ...recent,
- );
-}
-
-function compactConversation(provider: Provider, conversation: unknown[], reportFile: string): void {
- if (provider === "anthropic") {
- compactAnthropicConversation(conversation, reportFile);
- return;
- }
- compactOpenAIConversation(conversation, reportFile);
-}
-
-function laneSubsetInstructions(groups?: number[]): string {
- if (!groups || groups.length === 0) {
- return "Execute the full benchmark task set.";
- }
-
- return [
- `Execute only these benchmark groups: ${groups.join(", ")}.`,
- "Do not attempt groups outside this subset.",
- "Treat all other groups as out of scope for this run rather than as failures.",
- "For the selected groups, execute every step in the group unless blocked.",
- ].join("\n");
-}
-
-function laneTaskSourceInstructions(lane: Lane, groups?: number[]): string {
- const fullTaskFile = lane === "agent" ? AGENT_TASKS_FILE : AGENT_BROWSER_TASKS_FILE;
- if (!groups || groups.length === 0) {
- return [
- `Read the full task file at ${fullTaskFile}.`,
- "Execute the full benchmark task set.",
- ].join("\n");
- }
-
- if (lane === "agent") {
- const subset = extractGroupSections(AGENT_TASKS_FILE, groups);
- return [
- `Use only this selected task subset from ${AGENT_TASKS_FILE}:`,
- "",
- subset,
- ].join("\n");
- }
-
- const sections: string[] = [];
- for (const group of groups) {
- if (group === 0) {
- sections.push([
- "## Group 0: Setup & Diagnosis (agent-browser lane)",
- "",
- "### 0.1 Open fixtures home",
- "Run `./scripts/ab open http://fixtures/`.",
- "",
- "**Verify**: Open succeeds and the home page loads.",
- "",
- "### 0.2 Snapshot interactive refs",
- "Run `./scripts/ab snapshot -i -c` on the home page.",
- "",
- "**Verify**: Interactive refs are returned without error.",
- "",
- "### 0.3 Session persists across commands",
- "Use the same wrapper session across multiple commands and confirm browser state persists.",
- "",
- "**Verify**: A follow-up action can use the existing page/session state.",
- ].join("\n"));
- continue;
- }
-
- const section = extractSingleGroupSection(AGENT_TASKS_FILE, group);
- if (section) {
- sections.push(section);
- }
- }
-
- return [
- `Use only this selected task subset for agent-browser: Group 0 from ${AGENT_BROWSER_TASKS_FILE}, Groups 1+ from ${AGENT_TASKS_FILE}.`,
- "",
- sections.join("\n\n"),
- ].join("\n");
-}
-
-type LanePromptConfig = {
- label: string;
- skillInstruction: string;
- wrapper: string;
- recordType: string;
- workflowSummary: string[];
- adapterNotes: string[];
- bootstrapCommands: string[];
-};
-
-function lanePromptConfig(lane: Lane): LanePromptConfig {
- if (lane === "agent") {
- return {
- label: "PinchTab agent",
- skillInstruction: `Read ${resolve(REPO_ROOT, "skills", "pinchtab", "SKILL.md")} exactly once before acting.`,
- wrapper: "./scripts/pt",
- recordType: "agent",
- workflowSummary: [
- "use only ./scripts/pt for browser actions",
- "keep one shared tab via PINCHTAB_TAB",
- "use snap -i -c for actionable refs",
- "use text or text --full for reading content",
- "refresh refs after any navigation or DOM change",
- ],
- adapterNotes: [
- 'a navigation-expected click can return "Error 409: unexpected page navigation ..."; treat that as likely success and verify with a fresh snapshot/text read',
- "do not assume every command returns JSON; only parse JSON when the command actually returned JSON",
- 'the download endpoint returns JSON with base64 content in "data", not a local file path',
- 'this environment uses BSD/macOS userland tools; avoid GNU-only flags such as "head -n -1"',
- ],
- bootstrapCommands: [
- "./scripts/pt health",
- "./scripts/pt tab",
- "export PINCHTAB_TAB=$(./scripts/pt nav http://fixtures/)",
- "./scripts/pt snap -i -c",
- ],
- };
- }
-
- return {
- label: "agent-browser",
- skillInstruction: "Load the official agent-browser skill exactly once with `./scripts/ab skills get agent-browser --full` before acting.",
- wrapper: "./scripts/ab",
- recordType: "agent-browser",
- workflowSummary: [
- "use only ./scripts/ab for browser actions",
- "keep the shared benchmark session across commands",
- "use snapshot -i -c for actionable refs",
- "use fresh refs after any navigation or DOM change",
- "reuse the current browser state instead of reopening pages unless needed",
- ],
- adapterNotes: [
- "after navigation-triggering clicks, verify the resulting page with a fresh snapshot/text read instead of trusting the click response alone",
- "do not assume every command returns JSON; parse only when the command actually returned JSON",
- "for downloads, inspect returned content instead of assuming a file already exists locally",
- 'this environment uses BSD/macOS userland tools; avoid GNU-only flags such as "head -n -1"',
- ],
- bootstrapCommands: [
- "./scripts/ab skills get agent-browser --full",
- "./scripts/ab open http://fixtures/",
- "./scripts/ab snapshot -i -c",
- ],
- };
-}
-
-function laneUserPrompt(lane: Lane, reportFile: string, groups?: number[]): string {
- const config = lanePromptConfig(lane);
- const firstGroup = groups && groups.length > 0 ? groups[0] : 0;
- const bootstrap = config.bootstrapCommands.map((command, index) => `${index + 1}. ${command}`).join("\n");
-
- return `
-Work in this repo: ${REPO_ROOT}
-
-Benchmark lane: ${config.label} execution.
-
-Requirements:
-- Follow a linear execution flow: skill once, selected groups, execute, record, verify, continue.
-- Do not read README, browse directories, or inspect unrelated files unless a command path is missing.
-- ${config.skillInstruction}
-- Tool wrapper:
- - ${config.wrapper}
-- Workflow summary:
- - ${config.workflowSummary.join("\n - ")}
-- Adapter notes:
- - ${config.adapterNotes.join("\n - ")}
-- Task scope:
- ${laneTaskSourceInstructions(lane, groups).replace(/\n/g, "\n ")}
-- For each completed step:
- 1. record the observed answer with:
- - ./scripts/record-step.sh --report-file ${reportFile} --type ${config.recordType} <group> <step> answer "<what you saw>" "notes"
- 2. immediately verify it with:
- - ./scripts/verify-step.sh --report-file ${reportFile} --type ${config.recordType} <group> <step> <pass|fail|skip> "verification notes"
-- If a step cannot be completed, record fail or skip in the same report.
-- Do not leave answered steps pending verification.
-- Keep commands concise. Prefer rg/sed/cat only when you must inspect a specific file.
-- After the skill step, begin actual benchmark execution immediately.
-- Start with this bootstrap command sequence before attempting the selected steps:
-${bootstrap}
-- After the bootstrap, immediately execute Group ${firstGroup} step 1.
-- Subset selection:
- - ${laneSubsetInstructions(groups).replace(/\n/g, "\n - ")}
-- Finish when all selected steps are executed or when you are blocked.
-
-Your final answer should briefly summarize completion status and the main blockers, if any.
-`.trim();
-}
-
-function systemPrompt(): string {
- return `
-You are a precise benchmark execution agent. Use tools to inspect the repo and run the benchmark lane exactly as instructed.
-
-Rules:
-- Never fabricate command output or task results.
-- Use the shell tool for all file reads and command execution.
-- Do not use destructive commands such as rm -rf, git reset, or checkout.
-- After recording an answer, verify it immediately against the task oracle.
-- Prefer factual command output over long reasoning.
-`.trim();
-}
-
-function createRunner(provider: Provider, model: string, args: Args): ApiRunner {
- if (provider === "openai") {
- const apiKey = (process.env.OPENAI_API_KEY ?? "").trim();
- if (!apiKey) {
- throw new Error("OPENAI_API_KEY is required for provider=openai");
- }
- return new OpenAIRunner(
- "openai",
- "openai-responses",
- "https://api.openai.com/v1/responses",
- apiKey,
- model,
- args.maxTokens,
- args.temperature,
- !args.noPromptCaching,
- );
- }
-
- const apiKey = (process.env.ANTHROPIC_API_KEY ?? "").trim();
- if (!apiKey) {
- throw new Error("ANTHROPIC_API_KEY is required for provider=anthropic");
- }
- return new AnthropicRunner(apiKey, model, args.maxTokens, args.temperature, !args.noPromptCaching);
-}
-
-async function main(): Promise<number> {
- try {
- const args = parseArgs(process.argv.slice(2));
- const provider = resolveProvider(args);
- const model = provider === "anthropic" ? await resolveAnthropicModel(args.model) : resolveModel(provider, args.model);
- const groups = resolveGroups(args);
- const reportFile = args.reportFile || (args.skipInit ? resolveReportPath(args.lane) : initializeLane(args, provider, model));
- writeFileSync(AGENT_COMMANDS_LOG, "", "utf8");
-
- console.log(`[benchmark-runner] provider=${provider} model=${model} lane=${args.lane} groups=${formatGroups(groups)} report=${reportFile}`);
-
- const runner = createRunner(provider, model, args);
- const shell = new PersistentShell();
- const conversation = runner.initialConversation(laneUserPrompt(args.lane, reportFile, groups));
-
- let finalText = "";
- let exitCode = 0;
- let idleTurns = 0;
- let lastAnswered = readProgress(reportFile).answered;
-
- try {
- for (let turn = 1; turn <= args.maxTurns; turn += 1) {
- if (turn > 1 && args.turnDelayMs > 0) {
- await sleep(args.turnDelayMs);
- }
- const response = await runner.send(systemPrompt(), conversation);
- const toolCalls = runner.extractToolCalls(response, args.timeoutSeconds);
- if (toolCalls.length > 0) {
- const results = await executeToolCalls(shell, toolCalls);
- runner.appendToolResults(conversation, response, results);
- compactConversation(runner.provider, conversation, reportFile);
- const progress = readProgress(reportFile);
- if (progress.answered > lastAnswered) {
- idleTurns = 0;
- lastAnswered = progress.answered;
- } else {
- idleTurns += 1;
- }
- if (idleTurns >= args.maxIdleTurns) {
- finalText = `Stopped after ${idleTurns} consecutive turns without recording a benchmark step. Check ${AGENT_COMMANDS_LOG} for the command trace.`;
- exitCode = 3;
- break;
- }
- continue;
- }
-
- finalText = runner.extractFinalText(response);
- break;
- }
-
- if (!finalText) {
- finalText = `Stopped after reaching max turns (${args.maxTurns}).`;
- exitCode = 2;
- }
- } finally {
- shell.close();
- if (existsSync(reportFile)) {
- recordUsage(reportFile, runner);
- }
- if (args.finalize && existsSync(reportFile)) {
- runChecked([resolve(BENCH_DIR, "scripts", "finalize-report.sh"), reportFile]);
- }
- }
-
- if (finalText) {
- console.log(finalText);
- }
- console.log(
- `\n[run-usage] provider=${runner.provider} requests=${runner.requestCount} input=${runner.inputTokens} cache_create=${runner.cacheCreationInputTokens} cache_read=${runner.cacheReadInputTokens} output=${runner.outputTokens} total=${runner.inputTokens + runner.cacheCreationInputTokens + runner.cacheReadInputTokens + runner.outputTokens}`,
- );
- return exitCode;
- } catch (error) {
- console.error(error instanceof Error ? error.message : String(error));
- return 1;
- }
-}
-
-process.exitCode = await main();
diff --git a/tests/benchmark/scripts/run-optimization.sh b/tests/benchmark/scripts/run-optimization.sh
index 43016e2e..8beeade3 100755
--- a/tests/benchmark/scripts/run-optimization.sh
+++ b/tests/benchmark/scripts/run-optimization.sh
@@ -82,7 +82,7 @@ EOF
cat > "${AGENT_REPORT}" << EOF
{
"benchmark": {
- "type": "agent",
+ "type": "pinchtab",
"run_number": ${RUN_NUMBER},
"timestamp": "${TIMESTAMP}",
"model": "${BENCHMARK_MODEL:-unknown}",
diff --git a/tests/benchmark/scripts/verify-answers.sh b/tests/benchmark/scripts/verify-answers.sh
index 11ced7d1..67470774 100755
--- a/tests/benchmark/scripts/verify-answers.sh
+++ b/tests/benchmark/scripts/verify-answers.sh
@@ -3,7 +3,7 @@
# One-off verifier for Loop #31 answers. Reads the agent report, matches each
# step's `.answer` against an expected marker/substring set, and invokes
# verify-step.sh with pass/fail. The marker map below is hand-derived from
-# BASELINE_TASKS.md "Pass if" criteria.
+# the baseline benchmark oracle implemented in scripts/baseline.sh.
#
# Usage:
# ./verify_answers_for_loop31.sh results/agent_benchmark_20260415_233421.json
diff --git a/tests/benchmark/scripts/verify-step.sh b/tests/benchmark/scripts/verify-step.sh
index d6439d40..86668394 100755
--- a/tests/benchmark/scripts/verify-step.sh
+++ b/tests/benchmark/scripts/verify-step.sh
@@ -4,7 +4,7 @@
# step.
#
# Usage:
-# ./verify-step.sh [--type baseline|agent|agent-browser] [--report-file <path>] <group> <step> <pass|fail|skip> "notes"
+# ./verify-step.sh [--type baseline|pinchtab|agent-browser] [--report-file <path>] <group> <step> <pass|fail|skip> "notes"
set -euo pipefail
@@ -35,7 +35,7 @@ while [[ $# -gt 0 && "$1" == --* ]]; do
done
if [[ $# -lt 3 ]]; then
- echo "Usage: $0 [--type baseline|agent|agent-browser] [--report-file <path>] <group> <step> <pass|fail|skip> [notes]"
+ echo "Usage: $0 [--type baseline|pinchtab|agent-browser] [--report-file <path>] <group> <step> <pass|fail|skip> [notes]"
exit 1
fi
@@ -70,7 +70,7 @@ if [[ -z "${REPORT_FILE}" ]]; then
REPORT_FILE="$(resolve_current_report "${CURRENT_BASELINE_PTR}" || true)"
[[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/baseline_*.json 2>/dev/null | head -1)
;;
- agent)
+ pinchtab|agent)
REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_PTR}" || true)"
[[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_benchmark_*.json 2>/dev/null | head -1)
;;
@@ -79,7 +79,7 @@ if [[ -z "${REPORT_FILE}" ]]; then
[[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_browser_benchmark_*.json 2>/dev/null | head -1)
;;
*)
- echo "ERROR: --type must be 'baseline', 'agent', or 'agent-browser'"
+ echo "ERROR: --type must be 'baseline', 'pinchtab', or 'agent-browser'"
exit 1
;;
esac
diff --git a/tests/benchmark/AGENT_BROWSER_INSTRUCTIONS.md b/tests/benchmark/setup-agent-browser.md
similarity index 56%
rename from tests/benchmark/AGENT_BROWSER_INSTRUCTIONS.md
rename to tests/benchmark/setup-agent-browser.md
index d35e46af..ebf279df 100644
--- a/tests/benchmark/AGENT_BROWSER_INSTRUCTIONS.md
+++ b/tests/benchmark/setup-agent-browser.md
@@ -1,13 +1,31 @@
-# Agent Browser Instructions
+# Agent Browser Setup
-These are benchmark-specific instructions for running the `agent-browser` lane.
-They are not the official `agent-browser` skill. For the live CLI skill, load it
-through the benchmark wrapper:
+Lane-specific setup and operating guidance for running the benchmark with
+`agent-browser`.
+
+## Read Order
+
+1. Read `./benchmark-run/index.md`
+2. Load the live skill through the wrapper:
```bash
./scripts/ab skills get agent-browser --full
```
+This file covers setup, wrapper usage, and Group 0 differences. Shared task
+groups live in `benchmark-run/`.
+
+## Recording
+
+Record each completed step as factual `answer`, `fail`, or `skip`, then verify:
+
+```bash
+./scripts/record-step.sh --type agent-browser <group> <step> answer "<what you saw>" "notes"
+./scripts/verify-step.sh --type agent-browser <group> <step> <pass|fail|skip> "verification notes"
+```
+
+Do not self-grade inside the answer payload. Keep the answer factual.
+
## Wrapper
For benchmark runs in this repo, do not call `agent-browser` directly. Use:
@@ -65,6 +83,8 @@ cd tests/benchmark
- Fixtures: `http://fixtures/`
- Session name: `benchmark` by default (`AGENT_BROWSER_SESSION` overrides)
- Browser driver: Docker service `agent-browser`
+- Pages: `/`, `/wiki.html`, `/wiki-go.html`, `/articles.html`, `/search.html`,
+ `/form.html`, `/dashboard.html`, `/ecommerce.html`, `/spa.html`, `/login.html`
## Operating Guidance
@@ -73,3 +93,13 @@ cd tests/benchmark
- Prefer refs from `snapshot -i -c` over brittle selectors
- Re-snapshot after navigation or any DOM-changing action
- Keep one session for the whole benchmark lane unless a task explicitly needs a reset
+
+## Group 0 Override
+
+For the `agent-browser` lane, Group 0 differs from `benchmark-run/group-00.md`:
+
+- 0.1 `./scripts/ab open http://fixtures/` succeeds
+- 0.2 `./scripts/ab snapshot -i -c` returns interactive refs
+- 0.3 session state persists across multiple `./scripts/ab ...` commands
+
+From Group 1 onward, use `benchmark-run/group-01.md` through `benchmark-run/group-38.md`.
diff --git a/tests/benchmark/setup-pinchtab.md b/tests/benchmark/setup-pinchtab.md
new file mode 100644
index 00000000..0acc8cdd
--- /dev/null
+++ b/tests/benchmark/setup-pinchtab.md
@@ -0,0 +1,73 @@
+# PinchTab Setup
+
+Lane-specific setup and operating guidance for running the benchmark with
+PinchTab.
+
+## Read Order
+
+1. Read `../../skills/pinchtab/SKILL.md`
+2. Read `./benchmark-run/index.md`
+
+`benchmark-run/index.md` contains the shared task map. This file only covers
+setup, wrapper usage, and PinchTab-specific expectations.
+
+## Recording
+
+Record each completed step as factual `answer`, `fail`, or `skip`, then verify:
+
+```bash
+./scripts/record-step.sh --type pinchtab <group> <step> answer "<what you saw>" "notes"
+./scripts/verify-step.sh --type pinchtab <group> <step> <pass|fail|skip> "verification notes"
+```
+
+Do not self-grade inside the answer payload. Keep the answer factual.
+
+## Environment
+
+- PinchTab: `http://localhost:9867`, token: `benchmark-token`
+- Fixtures: `http://fixtures/` (running in Docker as `fixtures` hostname)
+- Pages: `/`, `/wiki.html`, `/wiki-go.html`, `/articles.html`, `/search.html`,
+ `/form.html`, `/dashboard.html`, `/ecommerce.html`, `/spa.html`, `/login.html`
+
+## Wrapper
+
+Use only:
+
+```bash
+./scripts/pt ...
+```
+
+Do not call `pinchtab` directly for benchmark runs.
+
+The wrapper:
+
+- executes the CLI inside the benchmark Docker container
+- forwards `PINCHTAB_TOKEN`, `PINCHTAB_SERVER`, and `PINCHTAB_TAB`
+- preserves one shared tab across commands when `PINCHTAB_TAB` is set
+
+## Recommended Flow
+
+```bash
+./scripts/pt health
+./scripts/pt tab
+export PINCHTAB_TAB=$(./scripts/pt nav http://fixtures/)
+./scripts/pt snap -i -c
+```
+
+After that:
+
+- use `./scripts/pt snap -i -c` for actionable refs
+- use `./scripts/pt text` or `./scripts/pt text --full` for reading
+- re-snapshot after navigation or DOM changes
+- record and verify each step as you go
+
+## PinchTab-Specific Notes
+
+- A navigation-triggering click can return `Error 409: unexpected page navigation ...` and still have succeeded. Verify with a fresh snapshot or text read.
+- Do not assume every command returns JSON. Parse only when the output is actually JSON.
+- The download endpoint returns JSON with base64 content in `data`, not a local file path.
+- `PINCHTAB_TAB` should be reused for the whole lane unless a task explicitly needs a new tab.
+
+## Group 0
+
+For the PinchTab lane, run Group 0 from `benchmark-run/group-00.md`.
diff --git a/tests/benchmark/TEST_CASES.md b/tests/benchmark/test-cases-summary.md
similarity index 100%
rename from tests/benchmark/TEST_CASES.md
rename to tests/benchmark/test-cases-summary.md