High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.
refactor: replace api-runner with go implementation
Luigi Agostid7216a104ddf
diff --git a/docs/deep-dive/agent-browser-benchmark.md b/docs/deep-dive/agent-browser-benchmark.mdindex 9103f9f2..f4df250a 100644--- a/docs/deep-dive/agent-browser-benchmark.md+++ b/docs/deep-dive/agent-browser-benchmark.md@@ -617,16 +617,16 @@ docker compose up -d --build./scripts/run-optimization.sh# 3. Run baseline (deterministic, ~30 seconds)-./scripts/baseline_all.sh+./scripts/baseline.sh# 4. Run PinchTab agent lane (requires LLM, ~15 minutes)-# Execute tasks from AGENT_TASKS.md using ./scripts/pt+# Execute tasks from benchmark-run/index.md + benchmark-run/group-XX.md using setup-pinchtab.md and ./scripts/pt# 5. Initialize agent-browser lane./scripts/run-agent-browser-benchmark.sh# 6. Run agent-browser lane (requires LLM, ~20 minutes)-# Execute tasks from AGENT_BROWSER_TASKS.md using ./scripts/ab+# Execute tasks from benchmark-run/index.md + benchmark-run/group-XX.md using setup-agent-browser.md and ./scripts/ab# 7. Generate reports./scripts/finalize-report.shdiff --git a/scripts/dev-benchmark.sh b/scripts/dev-benchmark.shindex 7c58fd04..117e3ad4 100755--- a/scripts/dev-benchmark.sh+++ b/scripts/dev-benchmark.sh@@ -18,20 +18,19 @@ usage() {cat <<'EOF'Usage:./dev benchmark baseline- OPENAI_API_KEY=... ./dev benchmark agent [extra runner args...]- OPENAI_API_KEY=... ./dev benchmark agent-browser [extra runner args...]- ANTHROPIC_API_KEY=... ./dev benchmark agent [extra runner args...]- ANTHROPIC_API_KEY=... ./dev benchmark agent-browser [extra runner args...]+ OPENAI_API_KEY=... ./dev benchmark pinchtab [options...]+ OPENAI_API_KEY=... ./dev benchmark agent-browser [options...]+ ANTHROPIC_API_KEY=... ./dev benchmark pinchtab [options...]+ ANTHROPIC_API_KEY=... ./dev benchmark agent-browser [options...]Examples:./dev benchmark baseline- OPENAI_API_KEY=... ./dev benchmark agent --model gpt-5- OPENAI_API_KEY=... ./dev benchmark agent-browser --model gpt-5- ANTHROPIC_API_KEY=... ./dev benchmark agent --model claude-haiku-4-5-20251001- ANTHROPIC_API_KEY=... ./dev benchmark agent-browser --model claude-haiku-4-5-20251001- ANTHROPIC_API_KEY=... ./dev benchmark agent --provider anthropic --turn-delay-ms 3000- ANTHROPIC_API_KEY=... ./dev benchmark agent --provider anthropic --profile common10- ANTHROPIC_API_KEY=... ./dev benchmark agent-browser --provider anthropic --groups 0,1,2,3+ ./dev benchmark pinchtab --dry-run+ OPENAI_API_KEY=... ./dev benchmark pinchtab --model gpt-5+ ANTHROPIC_API_KEY=... ./dev benchmark pinchtab --model claude-haiku-4-5-20251001+ ANTHROPIC_API_KEY=... ./dev benchmark pinchtab --profile common10+ ANTHROPIC_API_KEY=... ./dev benchmark pinchtab --max-input-tokens 50000+ ANTHROPIC_API_KEY=... ./dev benchmark agent-browser --groups 0,1,2,3EOF}@@ -52,12 +51,13 @@ case "${mode}" inBASELINE_REPORT="$(resolve_current_report "${RESULTS_DIR}/current_baseline_report.txt")"./scripts/finalize-report.sh "${BASELINE_REPORT}";;- agent|agent-browser)+ pinchtab|agent-browser)if [[ -z "${OPENAI_API_KEY:-}" && -z "${ANTHROPIC_API_KEY:-}" ]]; thenecho "ERROR: OPENAI_API_KEY or ANTHROPIC_API_KEY is required for benchmark ${mode}" >&2exit 1fi- exec node --experimental-strip-types ./scripts/run-api-benchmark.ts --lane "${mode}" --finalize "$@"+ cd "${ROOT_DIR}"+ exec go run ./tests/benchmark/cmd/api-runner --lane "${mode}" --finalize "$@";;-h|--help|help)usagediff --git a/tests/benchmark/AGENT_BROWSER_TASKS.md b/tests/benchmark/AGENT_BROWSER_TASKS.mddeleted file mode 100644index cdd891c3..00000000--- a/tests/benchmark/AGENT_BROWSER_TASKS.md+++ /dev/null@@ -1,76 +0,0 @@-# Agent Browser Benchmark--Natural-language benchmark lane for running the same fixture tasks with-[`agent-browser`](https://github.com/vercel-labs/agent-browser) instead of the-PinchTab CLI.--## Setup--1. Read `./AGENT_BROWSER_INSTRUCTIONS.md` first. This is your benchmark guide to- the wrapper and command workflow.--2. Load the live CLI skill content before using browser commands:--```bash-./scripts/ab skills get agent-browser --full-```--3. Start the Docker benchmark environment and initialize a report:--```bash-cd tests/benchmark-./scripts/run-agent-browser-benchmark.sh-```--4. Use the Docker-backed wrapper for every browser action:--```bash-./scripts/ab open http://fixtures/-./scripts/ab snapshot -i -c-./scripts/ab click @e2-./scripts/ab fill @e3 "[email protected]"-```--5. Record each completed step as an answer/result first:--```bash-./scripts/record-step.sh --type agent-browser 1 1 answer \- "opened fixtures home and got refs e1-e13" "completed"-```--`record-step.sh` will automatically calculate the number of `agent-browser`-tool calls used since the previous recorded step by reading-`results/agent_browser_commands.ndjson`.--6. After the execution lane is done, run a separate verification pass with:--```bash-./scripts/verify-step.sh --type agent-browser 1 1 pass "matched expected homepage state"-```--## Environment--- Fixtures: `http://fixtures/`-- Session name: `benchmark` by default (`AGENT_BROWSER_SESSION` overrides)-- Browser driver: Docker service `agent-browser`--## Tooling Guidance--Use `./AGENT_BROWSER_INSTRUCTIONS.md` as the primary benchmark operating guide.-Reach for `./scripts/ab --help` only when the skill does not already answer the-question.--## Task Set--Reuse the same benchmark task groups from [AGENT_TASKS.md](./AGENT_TASKS.md)-for content extraction, search, forms, SPA state, login, e-commerce, exports,-dialogs, async flows, drag/drop, keyboard, scrolling, and iframe interaction.--The only setup difference is that Group 0 should validate the `agent-browser`-lane instead of the PinchTab server:--- 0.1 `./scripts/ab open http://fixtures/` succeeds-- 0.2 `./scripts/ab snapshot -i -c` returns interactive refs-- 0.3 session state persists across multiple `./scripts/ab ...` commands--After that, continue with the same user-facing tasks from Group 1 onward.diff --git a/tests/benchmark/AGENT_TASKS.md b/tests/benchmark/AGENT_TASKS.mddeleted file mode 100644index 89ae8c70..00000000--- a/tests/benchmark/AGENT_TASKS.md+++ /dev/null@@ -1,718 +0,0 @@-# PinchTab Agent Benchmark--Natural language tasks to test how well an agent uses PinchTab from skill docs alone.--## Instructions--1. Read `../../skills/pinchtab/SKILL.md` — this is your only guide-2. For each task, figure out which commands to use-3. **Log every command executed**-4. Do not score your own work as pass/fail unless a task is trivially objective.-5. Prefer recording each step as `answer` with the actual response/result:- `./scripts/record-step.sh --type agent <group> <step> answer "<what you saw>" "notes"`-6. A separate verification pass should later stamp the step with:- `./scripts/verify-step.sh --type agent <group> <step> <pass|fail|skip> "verification notes"`--### Recommended setup: env vars + `./scripts/pt` wrapper--In normal PinchTab deployments the binary is installed on the host and-invoked directly:--```bash-export PINCHTAB_TOKEN=<token>-export PINCHTAB_TAB=$(pinchtab nav http://example.com)-pinchtab snap -i -c-pinchtab click '#submit'-```--The benchmark harness runs PinchTab **inside a Docker container**, so every-invocation normally has to be prefixed with-`docker exec -e PINCHTAB_TOKEN=... -e PINCHTAB_SERVER=... -e PINCHTAB_TAB=... benchmark-pinchtab-1 pinchtab ...`-— ~140 characters of boilerplate per command. The repo ships a tiny wrapper-at `tests/benchmark/scripts/pt` that handles the Docker preamble so you can-use the same terse CLI as a host install:--```bash-# Capture the shared tab ID. pinchtab prints only the tab ID when stdout-# is a pipe, so $(...) capture works directly.-export PINCHTAB_TAB=$(./scripts/pt nav http://fixtures/)--# Every subsequent tab-scoped command auto-targets $PINCHTAB_TAB:-./scripts/pt snap -i -c-./scripts/pt eval "document.title"-./scripts/pt eval --await-promise "window.fetchPayload()"-./scripts/pt click '#submit'-./scripts/pt drag '#piece' --drag-x 12 --drag-y -158--# Record answer/result for later verification:-./scripts/record-step.sh --type agent 1 2 answer "clicked submit and saw success banner" "worked"-```--The wrapper is a ~60-line shell script; read `scripts/pt` for the exact-forwarding rules. Flags after `pt` are forwarded verbatim to `pinchtab`, so-anything that works with `pinchtab ...` works with `./scripts/pt ...`.--Explicit `--tab <id>` on any command still wins over `PINCHTAB_TAB`.--For deferred verification, keep the answer payload factual. Record the-actual page text, count, answer, or state transition you observed, not your own-judgment about whether that matches the benchmark oracle.--## Environment--- PinchTab: `http://localhost:9867`, token: `benchmark-token`-- Fixtures: `http://fixtures/` (running in Docker as `fixtures` hostname)-- Pages: `/`, `/wiki.html`, `/wiki-go.html`, `/articles.html`, `/search.html`,- `/form.html`, `/dashboard.html`, `/ecommerce.html`, `/spa.html`, `/login.html`-------## Group 0: Setup & Diagnosis--### 0.1 Server reachable-Check that the PinchTab server is healthy.--**Verify**: Server responds with `status: ok`.--### 0.2 Auth is required-Make a request to the server with a **wrong** token (`PINCHTAB_TOKEN=wrong-token ./scripts/pt health`) and confirm it is rejected. The `pt` wrapper always injects the benchmark token by default, so you must explicitly override it to test auth rejection.--**Verify**: Response is HTTP 401 or contains `unauthorized`.--### 0.3 Auth works with token-Repeat the same request WITH the bearer token and confirm it succeeds.--**Verify**: Response is HTTP 200.--### 0.4 Instance available-Confirm at least one Chrome instance is running. If none exist, start one.--**Verify**: Health response shows `defaultInstance.status == "running"` (or after starting one, the new instance is running).--### 0.5 List existing tabs-Get the current list of open tabs.--**Verify**: A list (possibly empty) is returned without error.--### 0.6 Clean stale tabs-If any tabs from previous runs are open, close them so the benchmark starts from a known state.--**Verify**: After cleanup, the tab list is empty (or only contains a single about:blank tab).--### 0.7 Network reach to target-Navigate to `http://fixtures/` and confirm the fixtures server is reachable from PinchTab.--**Verify**: Navigate returns successfully and the page contains benchmark content.--### 0.8 Capture initial tab ID-Save the tab ID returned by the navigate in 0.7. Use this tab ID for all subsequent tasks to avoid creating new tabs.--**Verify**: A tab ID was captured and matches what `GET /tabs` reports.-------## Group 1: Reading & Extracting Real Content--### 1.1 Get the full list of categories from the wiki index-Navigate to `http://fixtures/wiki.html` and extract all category names with their article counts.--**Verify**: Can name at least 2 categories and their article counts (e.g. "Programming Languages: 12 articles").--### 1.2 Navigate by clicking a link-From the wiki index, click the "Go (programming language)" link to navigate to the Go article.--**Verify**: You are now on the Go article page (not the wiki index).--### 1.3 Extract structured data from a table-On the Go article, read the infobox table and answer: Who designed Go, and what year did it first appear?--**Verify**: Answer contains "Robert Griesemer" (or "Rob Pike" or "Ken Thompson") and "2009".--### 1.4 Count list items-On the Go article, count how many key features are listed.--**Verify**: Answer is 6 (verify against `FEATURE_COUNT_6`).--### 1.5 Read all article headlines from articles page-Navigate to `http://fixtures/articles.html` and list all article titles.--**Verify**: Found at least 3 articles including "The Future of Artificial Intelligence".--### 1.6 Read dashboard metrics-Navigate to `http://fixtures/dashboard.html` and extract: Total Users, Revenue, and Conversion Rate.--**Verify**: Found `24,582` users AND `$1,284,930` revenue.-------## Group 2: Search & Dynamic Interaction--### 2.1 Use wiki search to find a page-On `http://fixtures/wiki.html`, search for "golang" using the search form. Do not navigate directly — use the search input.--**Verify**: Ended up on the Go article page after search.--### 2.2 Search with no results-On `http://fixtures/search.html`, search for something with no results (use "xyznonexistent").--**Verify**: Page handled it gracefully (no crash, some response rendered).--### 2.3 Search for AI content-On `http://fixtures/search.html`, search for "artificial intelligence" and verify a result appeared.--**Verify**: Result contains "The Future of Artificial Intelligence".-------## Group 3: Complex Form--### 3.1 Fill and submit a complete form-Navigate to `http://fixtures/form.html`. Complete the entire form:-- Full name: "Agent Test User"-- Email: "[email protected]"-- Phone: "+44 20 9999 0000"-- Country: United Kingdom-- Subject: Technical Support-- Message: "Testing PinchTab form automation"-- Check newsletter-- Set priority to High-- Submit--**Verify**: Form submitted successfully. Confirmation shows name "AGENT_TEST_USER".--### 3.2 Reset and refill-After submitting, if the form is still accessible (or navigate back), verify you can identify the reset button.--**Verify**: Reset/back button or form element found in snapshot.-------## Group 4: SPA State Management--### 4.1 Read initial app state-Navigate to `http://fixtures/spa.html?reset=1` (the `?reset=1` query param clears any previous localStorage state so the SPA starts with its default 3 tasks). Read the current task list — how many tasks exist, how many are active vs done?--**Verify**: Found 3 total, 2 active, 1 done (verify `TASK_STATS_TOTAL_3_ACTIVE_2_DONE_1`).--### 4.2 Add a new high-priority task-Add a task called "Automate deployment" with high priority.--**Verify**: Task appeared in the list (`TASK_ADDED_AUTOMATE_DEPLOYMENT_PRIORITY_HIGH`).--### 4.3 Delete a task-Delete the task titled "Write benchmark tests".--**Verify**: Task count changed (went from 4 to 3).-------## Group 5: Login Flow--### 5.1 Attempt login with wrong credentials-Navigate to `http://fixtures/login.html`. Try to log in with username "admin" and password "wrong".--**Verify**: Error message appeared (`INVALID_CREDENTIALS_ERROR`).--### 5.2 Login successfully-Clear the form and log in with username "benchmark" / password "test456".--**Verify**: Dashboard appeared after login (`VERIFY_LOGIN_SUCCESS_DASHBOARD`).-------## Group 6: Multi-Step E-commerce--### 6.1 Research products before buying-Navigate to `http://fixtures/ecommerce.html`. List all available products with their prices. Which product is out of stock?--**Verify**: Found Wireless Headphones ($149.99), Smart Watch Pro ($299.99), Portable Charger ($49.99). Mechanical Keyboard is out of stock.--### 6.2 Add two items and verify total-Add Wireless Headphones and Smart Watch Pro to cart. What is the total?--**Verify**: Cart total is $449.98 (149.99 + 299.99).--### 6.3 Complete checkout-Click checkout to complete the order.--**Verify**: Order confirmation shows (`VERIFY_CHECKOUT_SUCCESS_ORDER`).-------## Group 7: Content + Interaction Combined--### 7.1 Read and comment on Go article-Navigate to `http://fixtures/wiki-go.html`. Read the article, then post a comment with rating 5 stars and text "Excellent reference".--**Verify**: Comment posted (`COMMENT_POSTED_RATING_5_TEXT_RECEIVED`).--### 7.2 Cross-page research task-Navigate to wiki index, find which category has the most articles, then navigate to one of its listed items.--**Verify**: Successfully navigated to at least one article page after reading the category counts.-------## Group 8: Error Handling--### 8.1 Handle 404 gracefully-Try to navigate to a page that doesn't exist: `http://fixtures/missing-page-abc.html`.--**Verify**: Got a response (404 or error), no crash, server still responsive after.--### 8.2 Handle missing element gracefully-On any page, try to click an element with ID `#fake-button-that-does-not-exist`.--**Verify**: Got a clear error message, not a crash or hang.-------## Group 9: Export--### 9.1 Screenshot a complex page-Navigate to `http://fixtures/dashboard.html` and take a screenshot.--**Verify**: Screenshot generated (file saved or base64 returned).--### 9.2 Export a page as PDF-Export the dashboard as a PDF.--**Verify**: PDF generated.-------## Group 10: Nested Interactions & Modal Dialogs--### 10.1 Open and interact with modal on dashboard-Navigate to `http://fixtures/dashboard.html`. Find and click the Settings button (selector: `#settings-btn`) to open the modal dialog.--**Verify**: Modal appeared — snapshot contains "Dashboard Settings".--### 10.2 Modify settings and close modal-In the modal, select "Dark" from the theme dropdown (`#theme-select`), then click the Save button (`#modal-save`). After the modal closes, check the page content.--**Verify**: Page contains `THEME_DARK_APPLIED`.-------## Group 11: State Persistence & Page Reload--### 11.1 Add an item and verify after page reload-Navigate to `http://fixtures/spa.html?reset=1` (starts with clean state). Add a task titled exactly "Persistent Task Test". Then navigate away to `http://fixtures/` and back to `http://fixtures/spa.html` (WITHOUT the reset param) to verify localStorage persistence.--**Verify**: After reload, the task still appears in the list (`TASK_PERSISTENT_TEST_FOUND_AFTER_RELOAD`).--### 11.2 Logout and log back in-From the logged-in dashboard, click Sign Out to log out. Then log in again with username "benchmark" / password "test456".--**Verify**: Successfully logged back in and dashboard shows `SESSION_RENEWED`.-------## Group 12: Multi-Page Navigation & Back Button--### 12.1 Navigate through multiple pages and return-Starting from `http://fixtures/`, navigate to wiki → Go article → back to wiki → back to home.--**Verify**: Successfully returned to home page (title contains "Benchmark" or "Home").--### 12.2 Compare data across pages-Navigate to wiki.html, note the total article count from categories. Navigate to articles.html, count articles there. Compare totals.--**Verify**: Can report totals from both pages and explain difference (`COMPARISON_DATA_FOUND`).-------## Group 13: Form State & Multi-Step Submission--### 13.1 Submit form without email-Navigate to `http://fixtures/form.html`. Fill only the name field ("Validator Test"), leave email blank, click Submit. The browser's native required-field validation will prevent submission.--**Verify**: Submission blocked (form stays open, no success message shown).--### 13.2 Submit form without optional phone field-Fill the form with: name "No Phone User", email "[email protected]", country "de", subject "feedback". Leave the phone field empty. Submit.--**Verify**: Submission succeeded and page shows `OPTIONAL_FIELD_SKIPPED_SUCCESS`.-------## Group 14: Dynamic Content Loading--### 14.1 Load more products-Navigate to `http://fixtures/ecommerce.html`. Find and click the "Load More Products" button to reveal additional products.--**Verify**: Additional products appeared (`ADDITIONAL_PRODUCTS_LOADED`).--### 14.2 Add a lazy-loaded product to cart-After loading more products, add product #5 (USB-C Cable) to the cart.--**Verify**: Cart shows the lazy-loaded item (`CART_UPDATED_WITH_LAZY_PRODUCT`).-------## Group 15: Complex Data Extraction & Aggregation--### 15.1 Extract and sum financial data-Navigate to `http://fixtures/dashboard.html`. Extract revenue and profit values, calculate profit margin.--**Verify**: Correctly calculated: profit_margin = (profit / revenue) * 100 (`PROFIT_MARGIN_CALCULATED`).--### 15.2 Build comparison table from multiple sources-Visit these 3 pages and compare their feature counts and key features:-- `http://fixtures/wiki-go.html` (Go: 6 features)-- `http://fixtures/wiki-python.html` (Python: 7 features)-- `http://fixtures/wiki-rust.html` (Rust: 5 features)--Report which language has the most features and name 1 feature unique to each.--**Verify**: Response is factually correct AND wiki-python.html contains `COMPARISON_TABLE_BUILT`.-------## Group 16: Hover & Tooltips--### 16.1 Hover reveals hidden content-Navigate to `http://fixtures/hovers.html`. Hover over the first avatar to reveal its hidden info.--**Verify**: After hovering, snapshot contains `HOVER_REVEALED_USER_1`.--### 16.2 Hover swap-Hover over the second avatar.--**Verify**: Snapshot contains `HOVER_REVEALED_USER_2`.-------## Group 17: Scrolling--### 17.1 Scroll by pixels-Navigate to `http://fixtures/scroll.html`. Scroll down approximately 1500 pixels.--**Verify**: Snapshot or text contains `SCROLL_MIDDLE_MARKER` (the mid-page marker is now in view).--### 17.2 Scroll to footer-Continue scrolling until the footer is visible (or scroll directly to the footer element).--**Verify**: Snapshot contains `SCROLL_REACHED_FOOTER`.-------## Group 18: File Download--### 18.1 Download a file-Download `http://fixtures/download-sample.txt` to a local file.--**Verify**: The downloaded file exists and its content includes `DOWNLOAD_FILE_CONTENT_VERIFIED`.-------## Group 19: iFrame--### 19.1 Read iframe content-Navigate to `http://fixtures/iframe.html` and extract content from inside the embedded same-origin iframe.--**Verify**: The iframe's inner content includes `IFRAME_INNER_CONTENT_LOADED`.--### 19.2 Type into iframe input (native frame scope)-Use `pinchtab frame '#content-frame'` to scope into the iframe, then `fill` the input with "Hello World" and `click` the Save button. Verify via a scoped `snap` — `text --full` doesn't pierce iframes. Reset with `pinchtab frame main` afterwards.--**Verify**: Scoped snapshot contains `IFRAME_INPUT_RECEIVED_HELLO_WORLD`.-------## Group 20: Dialogs--### 20.1 Accept alert-Navigate to `http://fixtures/alerts.html`. Click the "Click for Alert" button and dismiss the alert.--**Verify**: The page result contains `DIALOG_ALERT_DISMISSED`.--### 20.2 Cancel confirm-Click the "Click for Confirm" button and cancel the confirm dialog.--**Verify**: The page result contains `DIALOG_CONFIRM_CANCELLED`.-------## Group 21: Async / awaitPromise--### 21.1 Await a promise-returning function-Navigate to `http://fixtures/async.html`. The page exposes `window.fetchPayload()`, which returns a `Promise` that resolves after a short delay. Use `eval` to call it and retrieve the **resolved** value, not a Promise wrapper.--**Verify**: The resolved value contains `ASYNC_PAYLOAD_READY_42`.--### 21.2 Await a promise resolving to an object-On the same page, call `window.fetchUser()` and retrieve the resolved object so you can read a field from it.--**Verify**: The resolved object's `name` field equals `ASYNC_USER_NAME_ADA`.-------## Group 22: Mouse Drag & Drop--### 22.1 Drag a piece into Zone A-Navigate to `http://fixtures/drag.html`. The page contains a draggable square (`#piece`) and three target zones (`#zone-a`, `#zone-b`, `#zone-c`). Drag the piece so its center ends up over **Zone A**.--**Verify**: The page shows `LAST_DROP=DROP_ZONE_A_OK`.--### 22.2 Drag to Zone B, then Zone C-Without reloading the page, drag the piece next into **Zone B**, and then into **Zone C**. The page records an ordered drop sequence.--**Verify**: The page shows `DROP_SEQUENCE=DROP_ZONE_A_OK,DROP_ZONE_B_OK,DROP_ZONE_C_OK` (all three drops in order).-------## Group 23: Async / Loading state--### 23.1 Wait for async content-Navigate to `http://fixtures/loading.html`. The page shows a spinner and the text "Loading..." for ~1.5 seconds, then replaces it with the final payload. Wait for the final content to appear before reporting completion.--**Verify**: The final snapshot contains `VERIFY_LOADING_COMPLETE_88888`.-------## Group 24: Keyboard events--### 24.1 Press Escape-Navigate to `http://fixtures/keyboard.html` (it auto-focuses an input so keyboard events land). Press the Escape key.--**Verify**: Page text contains `KEYBOARD_ESCAPE_PRESSED`.--### 24.2 Press 'a' then Enter-Without reloading, press the `a` key, then the `Enter` key.--**Verify**: Page text now contains all three markers in order: `KEYBOARD_ESCAPE_PRESSED`, `KEYBOARD_KEY_A_PRESSED`, `KEYBOARD_ENTER_PRESSED`.-------## Group 25: Tab panels--### 25.1 Switch to the Settings tab-Navigate to `http://fixtures/tabs.html`. The Profile tab is shown by default. Click the Settings tab.--**Verify**: After the click, the page content includes `TAB_SETTINGS_CONTENT` (and the Profile content is no longer visible).--### 25.2 Switch to the Billing tab-Click the Billing tab.--**Verify**: Page content now includes `TAB_BILLING_CONTENT`.-------## Group 26: Accordion--### 26.1 Open section A-Navigate to `http://fixtures/accordion.html`. Click the header for Section A to expand it.--**Verify**: Page text includes `ACCORDION_SECTION_A_OPEN`.--### 26.2 Open section B-Click the header for Section B. Because the accordion is exclusive-expand, Section A should close.--**Verify**: Page text includes `ACCORDION_SECTION_B_OPEN`, and Section A's `aria-expanded` attribute is now `"false"` (use `eval` to check the attribute if needed).-------## Group 27: Contenteditable editor--### 27.1 Type into the rich-text editor-Navigate to `http://fixtures/editor.html`. Type the text `Hello rich text` into the `#editor` div. `contenteditable` elements don't have `.value`, so use `type` (keyboard events), not `fill`.--**Verify**: Page text contains `EDITOR_CHARS=15` and the mirror shows `Hello rich text`.--### 27.2 Commit by pressing Enter-Press Enter (the editor intercepts Enter to commit the current buffer to a separate marker).--**Verify**: Page text contains `EDITOR_COMMITTED=Hello rich text`.-------## Group 28: Range slider--### 28.1 Move slider into HIGH bucket-Navigate to `http://fixtures/range.html`. Set the `#volume` range slider to 90.--**Verify**: Page text contains both `RANGE_VALUE_90` and `BUCKET_HIGH`.--### 28.2 Move slider into LOW bucket-Without reloading, change the slider to 10.--**Verify**: Page text now contains `RANGE_VALUE_10` and `BUCKET_LOW`.-------## Group 29: Pagination--### 29.1 Advance to page 2-Navigate to `http://fixtures/pagination.html`. Click the Next button.--**Verify**: Page text contains `PAGE_2_FIRST_ITEM` and `PAGE_2_OF_3`.--### 29.2 Reach the last page; Next disabled-Click Next once more.--**Verify**: Page text contains `PAGE_3_FIRST_ITEM` and `PAGE_3_OF_3`, and the Next button's `disabled` property is `true` (use `eval` to check).-------## Group 30: Custom dropdown menu--### 30.1 Pick the "Beta" item-Navigate to `http://fixtures/dropdown.html`. The page has a custom dropdown — click the toggle button (`#dropdown-toggle`) to open the menu, then click the "Beta" item.--**Verify**: Page text contains `DROPDOWN_SELECTED=BETA`.--### 30.2 Reopen and pick "Gamma"-Reopen the dropdown and select "Gamma". (Each selection auto-closes the menu, so you need to click the toggle again before selecting.)--**Verify**: Page text contains `DROPDOWN_SELECTED=GAMMA`.-------## Group 31: Nested iframes (3 levels deep)--### 31.1 Click a button in the deepest frame-Navigate to `http://fixtures/iframe-nested.html`. The outer page embeds `#level-2`, which in turn embeds `#level-3`, which contains `#deep-button`. Drill through the two frame hops and click the button. Verify via a scoped `snap` before resetting to `main`.--**Verify**: Scoped snapshot contains `DEEP_CLICKED=YES_LEVEL_3`.-------## Group 32: Dynamic iframe (inserted after load)--### 32.1 Wait for a late iframe, then interact with it-Navigate to `http://fixtures/iframe-dynamic.html`. The iframe is inserted ~1.2 s after load (use `wait --text "IFRAME_DYNAMIC_ATTACHED"`). Then scope into `#late-frame`, fill `#iframe-input` with "Late World", click `#iframe-submit`, and verify the inner result marker.--**Verify**: Scoped snapshot contains `IFRAME_INPUT_RECEIVED_LATE_WORLD`.-------## Group 33: srcdoc iframe--### 33.1 Interact with an inline-content iframe-Navigate to `http://fixtures/iframe-srcdoc.html`. The iframe's content is inlined via `srcdoc`; it has no `src` URL. Scope into `#srcdoc-frame`, fill `#inline-input` with `srcdoc`, click `#inline-submit`, and verify.--**Verify**: Scoped snapshot contains `INLINE_RECEIVED_SRCDOC`.-------## Group 34: Sandboxed iframe--### 34.1 Click inside a sandboxed iframe-Navigate to `http://fixtures/iframe-sandbox.html`. The iframe has `sandbox="allow-scripts allow-same-origin"`. Scope into `#sandboxed`, click `#sandbox-button`, and verify.--**Verify**: Scoped snapshot contains `SANDBOX_CLICKED=YES`.-------## Group 35: Long-form article (Medium/Substack style)--### 35.1 Read the article with default `text`-Navigate to `http://fixtures/article.html`. Extract the page's main content using the default `text` (Readability) mode. Default mode is the right choice for article-style pages.--**Verify**: The extracted text contains both `ARTICLE_PUBLISHED_2026_04_15` and `ARTICLE_WORD_COUNT_MARKER_323` (both inside the article body — Readability keeps them).--### 35.2 See the chrome that Readability drops-Re-extract with `text --full` and confirm the footer is included this time.--**Verify**: `--full` output contains `FOOTER_COPYRIGHT_MARKER` (which the default mode trims).-------## Group 36: Search results page (SERP)--### 36.1 Find a specific result by id-Navigate to `http://fixtures/serp.html`. The page has 6 result cards (`#r-1`..`#r-6`). Extract just the third card's content. A scoped snapshot (`snap --selector "#r-3"`) is the direct path.--**Verify**: The scoped output contains `RESULT_3_TITLE` and `RESULT_3_SNIPPET_MARKER`.--### 36.2 Count all result cards-Use a full text extraction to verify all six results are present in one pass.--**Verify**: Output contains all of `RESULT_1_TITLE` through `RESULT_6_TITLE` and the summary `SERP_RESULT_COUNT_6`. Default Readability trims SERPs — use `text --full`.-------## Group 37: Q&A thread (Stack-Overflow style)--### 37.1 Find the accepted answer id-Navigate to `http://fixtures/qa.html`. The accepted answer carries `data-accepted="true"` on its `<div>` wrapper. Use `eval` to return that element's `id`.--**Verify**: The `eval` result equals `"a-2"`.--### 37.2 Extract the accepted answer's body-Scope a snapshot to `#a-2` and verify the body content.--**Verify**: Scoped snapshot contains `ANSWER_2_BODY_MARKER` and `ACCEPTED_ANSWER_ID_A2`.-------## Group 38: Pricing table--### 38.1 Read just the Pro plan-Navigate to `http://fixtures/pricing.html`. The page has three plan cards (`#plan-free`, `#plan-pro`, `#plan-enterprise`). Read only the Pro plan using a scoped snapshot.--**Verify**: Scoped snapshot contains `PLAN_PRO_PRICE_29` and `PLAN_PRO_LIMIT_5000_requests per day`.--### 38.2 Compare all three plans-Extract the full page (pricing grids are a Readability anti-pattern — reach for `--full`).--**Verify**: Output contains `PLAN_FREE_PRICE_0`, `PLAN_PRO_PRICE_29`, and `PLAN_ENTERPRISE_PRICE_CUSTOM`.-------## Summary--| Group | Tasks | Description |-|-------|-------|-------------|-| 0 | 8 | Setup & Diagnosis |-| 1 | 6 | Reading & Extracting Content |-| 2 | 3 | Search & Dynamic Interaction |-| 3 | 2 | Complex Form |-| 4 | 3 | SPA State Management |-| 5 | 2 | Login Flow |-| 6 | 3 | Multi-Step E-commerce |-| 7 | 2 | Content + Interaction Combined |-| 8 | 2 | Error Handling |-| 9 | 2 | Export |-| 10 | 2 | Nested Interactions & Modal Dialogs |-| 11 | 2 | State Persistence & Page Reload |-| 12 | 2 | Multi-Page Navigation & Back Button |-| 13 | 2 | Form State & Multi-Step Submission |-| 14 | 2 | Dynamic Content Loading |-| 15 | 2 | Complex Data Extraction & Aggregation |-| 16 | 2 | Hover & Tooltips |-| 17 | 2 | Scrolling |-| 18 | 1 | File Download |-| 19 | 2 | iFrame |-| 20 | 2 | Dialogs |-| 21 | 2 | Async / awaitPromise |-| 22 | 2 | Mouse Drag & Drop |-| 23 | 1 | Async / Loading state |-| 24 | 2 | Keyboard events |-| 25 | 2 | Tab panels |-| 26 | 2 | Accordion |-| 27 | 2 | Contenteditable editor |-| 28 | 2 | Range slider |-| 29 | 2 | Pagination |-| 30 | 2 | Custom dropdown menu |-| 31 | 1 | Nested iframes (3 levels) |-| 32 | 1 | Dynamic iframe |-| 33 | 1 | srcdoc iframe |-| 34 | 1 | Sandboxed iframe |-| 35 | 2 | Long-form article |-| 36 | 2 | Search results page |-| 37 | 2 | Q&A thread |-| 38 | 2 | Pricing table |--**Total: 85 tasks**--## Key Differences from Baseline--The agent must:-- Choose between `/text`, `/snapshot`, `/action`, `/navigate` appropriately-- Decide when to use `filter=interactive` vs full snapshot-- Handle multi-step flows without step-by-step curl guidance-- Extract and interpret structured data (tables, lists, counts)-- Detect state changes after interactionsdiff --git a/tests/benchmark/BASELINE_TASKS.md b/tests/benchmark/BASELINE_TASKS.mddeleted file mode 100644index 16efe850..00000000--- a/tests/benchmark/BASELINE_TASKS.md+++ /dev/null@@ -1,1830 +0,0 @@-# PinchTab Benchmark Tasks (Baseline)--Reproducible benchmark using explicit curl commands against controlled fixture pages.-Each task maps 1:1 to an agent task in AGENT_TASKS.md for direct comparison.--## Environment--- **PinchTab Server**: `http://localhost:9867`-- **Fixtures Server**: `http://fixtures/` (inside Docker network)-- **Token**: `benchmark-token`-- **Auth Header**: `Authorization: Bearer benchmark-token`--## MANDATORY: Docker--```bash-cd ~/dev/pinchtab/tests/benchmark-docker compose up -d --build-# Wait for healthy: curl -sf -H "Authorization: Bearer benchmark-token" http://localhost:9867/health-```--## Recording--```bash-# Baseline execution answer:-./scripts/record-step.sh --type baseline <group> <step> answer "<observed result>" "notes"--# Immediate verification:-./scripts/verify-step.sh --type baseline <group> <step> <pass|fail|skip> "verification notes"-```--**On failure, include in notes:**-- What was expected-- What was actually returned-- HTTP status code / error message--## Tab Reuse--`POST /navigate` creates a new tab by default. To avoid multi-tab issues, the-runner must:--1. Capture `tabId` from the first navigate response (step 0.2).-2. Pass `"tabId":"TAB_ID"` in every subsequent navigate request body.-3. Use tab-scoped endpoints for actions and snapshots:- - `POST /tabs/TAB_ID/action` instead of `POST /action`- - `GET /tabs/TAB_ID/snapshot?...` instead of `GET /snapshot?...`- - `GET /tabs/TAB_ID/text` instead of `GET /text`- - `POST /tabs/TAB_ID/back` instead of `POST /back`- - `GET /tabs/TAB_ID/screenshot` instead of `GET /screenshot`- - `GET /tabs/TAB_ID/pdf` instead of `GET /pdf`--All curl examples below use `TAB_ID` as a placeholder. Replace with the actual-tab ID captured in step 0.2.-------## Group 0: Setup & Diagnosis--### 0.1 Server reachable-```bash-curl -sf http://localhost:9867/health \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: `status == "ok"` in response body.--### 0.2 Auth required-```bash-curl -s -o /dev/null -w "%{http_code}" http://localhost:9867/health-```-**Pass if**: HTTP status is `401` (auth rejected without token).--### 0.3 Auth works with token-```bash-curl -s -o /dev/null -w "%{http_code}" http://localhost:9867/health \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: HTTP status is `200`.--### 0.4 Instance available-```bash-curl -sf http://localhost:9867/health \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Response contains `defaultInstance.status == "running"`. If not, run:-```bash-curl -X POST http://localhost:9867/instances/start \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{}'-```-And re-verify the new instance is running.--### 0.5 List existing tabs-```bash-curl -sf http://localhost:9867/tabs \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Returns a JSON array (possibly empty) without error.--### 0.6 Clean stale tabs-For each tab returned by step 0.5, close it:-```bash-curl -X POST http://localhost:9867/tabs/TAB_ID/close \- -H "Authorization: Bearer benchmark-token"-```-Then verify cleanup:-```bash-curl -sf http://localhost:9867/tabs \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: After cleanup, the tab list is empty or contains only an about:blank tab.--### 0.7 Network reach to target-```bash-curl -sf -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"url":"http://fixtures/"}'-```-**Capture**: Save the `tabId` from the JSON response. All subsequent commands use this tab ID.-```bash-curl -sf "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Navigate returns HTTP 200 AND snapshot contains `VERIFY_HOME_LOADED_12345`.--### 0.8 Capture initial tab ID-```bash-curl -sf http://localhost:9867/tabs \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: The captured `TAB_ID` from step 0.7 appears in the tabs list.-------## Group 1: Reading & Extracting--### 1.1 Wiki categories-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki.html"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains `VERIFY_WIKI_INDEX_55555` AND `COUNT_LANGUAGES_12` AND `COUNT_TOOLS_15`.--### 1.2 Click a link-```bash-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#link-go","waitNav":true}'-```-**Pass if**: HTTP 200 with `{"success":true}`.--### 1.3 Table extraction-```bash-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains `VERIFY_WIKI_GO_LANG_88888` AND `Robert Griesemer` AND `2009`.--### 1.4 Count list items-```bash-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains `FEATURE_COUNT_6`.--### 1.5 Article headlines-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/articles.html"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains `The Future of Artificial Intelligence` AND `Climate Action in 2026` AND `Mars Colony`.--### 1.6 Dashboard metrics-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/dashboard.html"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains `24,582` AND `$1,284,930` AND `4.28%`.-------## Group 2: Search & Dynamic--### 2.1 Wiki search-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki.html"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#wiki-search-input","text":"golang"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#wiki-search-btn","waitNav":true}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Snapshot contains `VERIFY_WIKI_GO_LANG_88888` (search redirected to Go page).--### 2.2 No results search-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/search.html"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#search-input","text":"xyznonexistent"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#search-btn"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: All curls return HTTP 200 AND snapshot shows no-results message.--### 2.3 AI content search-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/search.html"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#search-input","text":"artificial intelligence"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#search-btn"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Snapshot contains `The Future of Artificial Intelligence`.-------## Group 3: Form--### 3.1 Complete form-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/form.html"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#fullname","text":"John Benchmark"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#email","text":"[email protected]"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#phone","text":"+44 20 1234 5678"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"select","selector":"#country","value":"uk"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"select","selector":"#subject","value":"support"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#message","text":"This is a benchmark test message for PinchTab automation."}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#newsletter"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"input[name=priority][value=high]"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#submit-btn"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains `VERIFY_FORM_SUBMITTED_SUCCESS` AND `SUBMISSION_DATA_NAME_JOHN_BENCHMARK`.--### 3.2 Reset/refill-```bash-curl "http://localhost:9867/tabs/TAB_ID/snapshot?filter=interactive&format=compact" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains reset button or form element (after submission, page still has interactive elements).-------## Group 4: SPA--### 4.1 Read app state-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/spa.html?reset=1"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains `VERIFY_SPA_PAGE_99999` AND `TASK_STATS_TOTAL_3_ACTIVE_2_DONE_1`.--### 4.2 Add task-```bash-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#new-task-input","text":"Deploy to production"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"select","selector":"#priority-select","value":"high"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#add-task-btn"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains `TASK_ADDED_DEPLOY_TO_PRODUCTION_PRIORITY_HIGH`.--### 4.3 Delete task-```bash-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#task-1 .delete-task"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Total count is `3` (started with 3, added 1, deleted 1 = 3).-------## Group 5: Login--### 5.1 Invalid login-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/login.html"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#username","text":"baduser"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#password","text":"wrongpassword"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#login-btn"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains `INVALID_CREDENTIALS_ERROR`.--### 5.2 Valid login-```bash-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#username","text":"benchmark"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#password","text":"test456"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#login-btn"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains `VERIFY_LOGIN_SUCCESS_DASHBOARD` AND `SESSION_TOKEN_ACTIVE_TRUE`.-------## Group 6: E-commerce--### 6.1 Research products-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/ecommerce.html"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains `VERIFY_SHOP_PAGE_44444` AND `$149.99` AND `$299.99` AND `$49.99` AND `Out of Stock`.--### 6.2 Add to cart-```bash-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#product-1 .add-to-cart"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#product-2 .add-to-cart"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains `CART_ITEM_WIRELESS_HEADPHONES` AND `CART_ITEM_SMART_WATCH_PRO` AND `449.98`.--### 6.3 Checkout-```bash-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#checkout-btn"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains `VERIFY_CHECKOUT_SUCCESS_ORDER` AND `ORDER_TOTAL_449_98`.-------## Group 7: Content + Interaction--### 7.1 Read & comment-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki-go.html"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \- -H "Authorization: Bearer benchmark-token"--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#comment-text","text":"Great article on Go! Very comprehensive."}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"select","selector":"#comment-rating","value":"5"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#submit-comment"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: First snapshot contains `VERIFY_WIKI_GO_LANG_88888` AND final snapshot contains `COMMENT_POSTED_RATING_5_TEXT_RECEIVED`.--### 7.2 Cross-page research-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki.html"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \- -H "Authorization: Bearer benchmark-token"--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#link-go","waitNav":true}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: First snapshot contains `COUNT_LANGUAGES_12` AND second snapshot contains `VERIFY_WIKI_GO_LANG_88888`.-------## Group 8: Error Handling--### 8.1 404 handling-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/nonexistent-page-xyz.html"}'-```-**Pass if**: Returns response without crash (HTTP 200 with error page, or structured error).--### 8.2 Missing element-```bash-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#element-that-does-not-exist"}'-```-**Pass if**: Error response with clear message (not crash).-------## Group 9: Export--### 9.1 Screenshot-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/dashboard.html"}'--curl http://localhost:9867/tabs/TAB_ID/screenshot \- -H "Authorization: Bearer benchmark-token" \- --output /tmp/benchmark-screenshot.png--ls -la /tmp/benchmark-screenshot.png-```-**Pass if**: File exists and size > 10240 bytes.--### 9.2 PDF export-```bash-curl http://localhost:9867/tabs/TAB_ID/pdf \- -H "Authorization: Bearer benchmark-token" \- --output /tmp/benchmark-dashboard.pdf--ls -la /tmp/benchmark-dashboard.pdf-```-**Pass if**: File exists and size > 10240 bytes.-------## Group 10: Modals--### 10.1 Open modal-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/dashboard.html"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#settings-btn"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Snapshot contains `Dashboard Settings`.--### 10.2 Modal interaction-```bash-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"select","selector":"#theme-select","value":"dark"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#modal-save"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains `THEME_DARK_APPLIED`.-------## Group 11: Persistence--### 11.1 State after reload-```bash-# Start fresh (reset param clears localStorage on this load only)-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/spa.html?reset=1"}'--# Add the persistent task-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#new-task-input","text":"Persistent Task Test"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#add-task-btn"}'--# Navigate away then back WITHOUT reset param — state should persist in localStorage-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/"}'--curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/spa.html"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains `TASK_PERSISTENT_TEST_FOUND_AFTER_RELOAD`.--### 11.2 Logout/re-login-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/login.html"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#username","text":"benchmark"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#password","text":"test456"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#login-btn"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#logout-btn"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#username","text":"benchmark"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#password","text":"test456"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#login-btn"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains `VERIFY_LOGIN_SUCCESS_DASHBOARD` AND `SESSION_RENEWED`.-------## Group 12: Multi-page Nav--### 12.1 Navigate & return-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/"}'--curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki.html"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#link-go","waitNav":true}'--curl -X POST http://localhost:9867/tabs/TAB_ID/back \- -H "Authorization: Bearer benchmark-token"--curl -X POST http://localhost:9867/tabs/TAB_ID/back \- -H "Authorization: Bearer benchmark-token"--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Final snapshot contains `VERIFY_HOME_LOADED_12345` (returned to home).--### 12.2 Cross-page compare-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki.html"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \- -H "Authorization: Bearer benchmark-token"--curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/articles.html"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Wiki snapshot contains `COUNT_LANGUAGES_12` AND articles snapshot contains article titles.-------## Group 13: Form Validation--### 13.1 Required field-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/form.html"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#fullname","text":"Validator Test"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#submit-btn"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Snapshot does NOT contain `VERIFY_FORM_SUBMITTED_SUCCESS` (submission blocked by validation).--### 13.2 Optional field-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/form.html"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#fullname","text":"No Phone User"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#email","text":"[email protected]"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"select","selector":"#country","value":"de"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"select","selector":"#subject","value":"feedback"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#submit-btn"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains `VERIFY_FORM_SUBMITTED_SUCCESS` AND `OPTIONAL_FIELD_SKIPPED_SUCCESS`.-------## Group 14: Dynamic Content--### 14.1 Load more-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/ecommerce.html"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#load-more-btn"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains `ADDITIONAL_PRODUCTS_LOADED`.--### 14.2 Lazy-loaded item-```bash-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#product-5 .add-to-cart"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains `CART_UPDATED_WITH_LAZY_PRODUCT`.-------## Group 15: Data Aggregation--### 15.1 Financial calc-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/dashboard.html"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains `$1,284,930` (revenue) AND `$384,930` (profit) AND `PROFIT_MARGIN_CALCULATED`.--### 15.2 Multi-page comparison-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki-go.html"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \- -H "Authorization: Bearer benchmark-token"--curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki-python.html"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \- -H "Authorization: Bearer benchmark-token"--curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki-rust.html"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Go snapshot contains `FEATURE_COUNT_6` AND Python snapshot contains `FEATURE_COUNT_7` AND `COMPARISON_TABLE_BUILT` AND Rust snapshot contains `FEATURE_COUNT_5`.-------## Group 16: Hover & Tooltips--### 16.1 Hover reveals info-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/hovers.html"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"hover","selector":"#avatar-1"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Snapshot contains `HOVER_REVEALED_USER_1`.--### 16.2 Hover swap-```bash-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"hover","selector":"#avatar-2"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Snapshot contains `HOVER_REVEALED_USER_2`.-------## Group 17: Scrolling--### 17.1 Scroll by pixels-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/scroll.html"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"scroll","scrollY":1500}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Snapshot contains `SCROLL_MIDDLE_MARKER`.--### 17.2 Scroll to footer-```bash-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"scroll","selector":"#footer"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Snapshot contains `SCROLL_REACHED_FOOTER`.-------## Group 18: File Download--### 18.1 Download a file-```bash-curl "http://localhost:9867/tabs/TAB_ID/download?url=http://fixtures/download-sample.txt" \- -H "Authorization: Bearer benchmark-token" | \- jq -r .data | base64 -d > /tmp/benchmark-download.txt--grep "DOWNLOAD_FILE_CONTENT_VERIFIED" /tmp/benchmark-download.txt-```-**Pass if**: File exists and contains `DOWNLOAD_FILE_CONTENT_VERIFIED`.--**Note**: The download endpoint returns JSON with `{contentType, data (base64), size, url}`, not the raw file. Decode `.data` to get the file contents. For PinchTab to download from internal hosts (like `fixtures`), the domain must be in `security.downloadAllowedDomains` config.-------## Group 19: iFrame--### 19.1 Read iframe content-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/iframe.html"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Snapshot contains `IFRAME_INNER_CONTENT_LOADED`.--### 19.2 Type into iframe input (native frame scope)-```bash-# Scope into the same-origin iframe-curl -X POST http://localhost:9867/frame?tabId=TAB_ID \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"target":"#content-frame"}'--# Selectors now resolve inside the iframe-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#iframe-input","text":"Hello World"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#iframe-submit"}'--# Verify via a scoped snap (text --full doesn't pierce frames)-curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \- -H "Authorization: Bearer benchmark-token"--# Reset scope-curl -X POST http://localhost:9867/frame?tabId=TAB_ID \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"target":"main"}'-```-**Pass if**: The scoped snapshot contains `IFRAME_INPUT_RECEIVED_HELLO_WORLD`.--**Note**: `POST /frame` sets a stateful scope on the tab. Subsequent selector-based `/action` and `/snapshot` calls resolve inside the scoped iframe. Reset with `{"target":"main"}` when done. Cross-origin iframes are not currently exposed this way — fall back to `/evaluate` + `contentDocument` for those (when same-origin policy permits).-------## Group 20: Dialogs--### 20.1 Accept alert-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/alerts.html"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#alert-btn","dialogAction":"accept"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Snapshot contains `DIALOG_ALERT_DISMISSED`.--### 20.2 Cancel confirm-```bash-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#confirm-btn","dialogAction":"dismiss"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Snapshot contains `DIALOG_CONFIRM_CANCELLED`.--**Note**: Use `dialogAction` on the click action to auto-accept or auto-dismiss a JS dialog that the click opens. Without it, the click would hang until `/dialog` is called from a separate request.-------## Group 21: Async / awaitPromise--### 21.1 Await a promise-returning function-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/async.html"}'--# Without awaitPromise: returns {} (unresolved Promise representation)-curl -X POST http://localhost:9867/tabs/TAB_ID/evaluate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"expression":"window.fetchPayload()","awaitPromise":true}'-```-**Pass if**: Response `.result` equals `"ASYNC_PAYLOAD_READY_42"`. Also verify that repeating the same call with `awaitPromise` omitted returns `{}` (proves `awaitPromise` is actually doing work).--### 21.2 Await a promise resolving to an object-```bash-curl -X POST http://localhost:9867/tabs/TAB_ID/evaluate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"expression":"window.fetchUser()","awaitPromise":true}'-```-**Pass if**: Response `.result.name` equals `"ASYNC_USER_NAME_ADA"` (structured value survives the round-trip).--**Note**: See SKILL.md and `references/api.md` — `awaitPromise:true` on `/evaluate` makes the server wait for the returned Promise to settle before responding. Omit the flag when you want the raw Promise reference.-------## Group 22: Mouse Drag & Drop--### 22.1 Drag piece into Zone A (high-level `drag` action)-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/drag.html"}'--# Piece center starts at (92, 358); Zone A center is at (104, 200).-# Drag offset: (+12, -158).-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"drag","selector":"#piece","dragX":12,"dragY":-158}'--curl "http://localhost:9867/tabs/TAB_ID/text" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Page text contains `LAST_DROP=DROP_ZONE_A_OK`.--### 22.2 Low-level sequence: drag into Zone B, then Zone C-```bash-# Zone centers in viewport coords (for the default window size):-# piece starts: (92, 358)-# Zone A: (104, 200)-# Zone B: (344, 200)-# Zone C: (584, 400)-# If the viewport differs, query these with /evaluate +-# Array.from(['piece','zone-a','zone-b','zone-c']).map(id => ...getBoundingClientRect()).--# Drag piece (now at Zone A) to Zone B via explicit mouse-down → mouse-move → mouse-up.-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"mouse-move","x":104,"y":200}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"mouse-down","x":104,"y":200,"button":"left"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"mouse-move","x":344,"y":200}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"mouse-up","x":344,"y":200,"button":"left"}'--# Now drag Zone B → Zone C-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"mouse-down","x":344,"y":200,"button":"left"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"mouse-move","x":584,"y":400}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"mouse-up","x":584,"y":400,"button":"left"}'--curl "http://localhost:9867/tabs/TAB_ID/text" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Page text contains `DROP_SEQUENCE=DROP_ZONE_A_OK,DROP_ZONE_B_OK,DROP_ZONE_C_OK`.--**Note**: Either path (the high-level `drag` action with `dragX`/`dragY` offsets, or the explicit `mouse-down → mouse-move → mouse-up` sequence at absolute viewport coordinates) works. Use `drag` for simple point-to-point drags, and the low-level trio when the target site depends on intermediate `mousemove` events or when you need precise pacing.-------## Group 23: Async / Loading state--### 23.1 Wait for async content to replace a loading spinner-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/loading.html"}'--# Poll the snapshot until the final marker appears, or call /wait with a text-# predicate. Using /evaluate with awaitPromise is another valid path.-for i in 1 2 3 4 5 6 7 8 9 10; do- SNAP=$(curl -sf "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \- -H "Authorization: Bearer benchmark-token")- echo "$SNAP" | grep -q "VERIFY_LOADING_COMPLETE_88888" && break- sleep 0.3-done--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Final snapshot contains `VERIFY_LOADING_COMPLETE_88888`.--**Note**: The fixture delays replacing the spinner by ~1.5 s. Agents using-`pinchtab wait --text "VERIFY_LOADING_COMPLETE_88888" --timeout 5000` hit-this in a single call. Baseline callers can poll or `sleep` between-snapshots.-------## Group 24: Keyboard events--### 24.1 Press Escape to emit a marker-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/keyboard.html"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"press","key":"Escape"}'--curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Page text contains `KEYBOARD_ESCAPE_PRESSED`. Use `mode=raw` because the log div is short enough for Readability to drop it.--### 24.2 Sequential keys (a then Enter)-```bash-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"press","key":"a"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"press","key":"Enter"}'--curl "http://localhost:9867/tabs/TAB_ID/text" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Page text contains both `KEYBOARD_KEY_A_PRESSED` and `KEYBOARD_ENTER_PRESSED` (and the Escape marker from 24.1 is still there, proving the log accumulates).--**Note**: `press` issues a single keypress and requires the page to already have focus on an element that listens for `keydown`. The fixture auto-focuses `#focus-input` on load.-------## Group 25: Tab panels--### 25.1 Switch to Settings tab-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/tabs.html"}'--# Verify initial panel-curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \- -H "Authorization: Bearer benchmark-token"--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#tab-settings"}'--curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Initial response contains `TAB_PROFILE_CONTENT` AND final response contains `TAB_SETTINGS_CONTENT` (and the profile content is now hidden).--### 25.2 Switch to Billing tab-```bash-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#tab-billing"}'--curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Response contains `TAB_BILLING_CONTENT`.--**Note**: Tab panels toggle via the `hidden` attribute; the non-selected panels are absent from the visible text. Use `text?mode=raw` so the extraction reflects what the user actually sees.-------## Group 26: Accordion--### 26.1 Open section A-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/accordion.html"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#section-a .section-header"}'--curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Response contains `ACCORDION_SECTION_A_OPEN`.--### 26.2 Open section B — A auto-closes-```bash-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#section-b .section-header"}'--curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \- -H "Authorization: Bearer benchmark-token"--# Also verify the A body text is no longer visible by inspecting aria-expanded.-curl -X POST http://localhost:9867/tabs/TAB_ID/evaluate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"expression":"document.getElementById(\"section-a\").getAttribute(\"aria-expanded\")"}'-```-**Pass if**: Text response contains `ACCORDION_SECTION_B_OPEN`, AND the `evaluate` response for the section-a state returns `"false"`.--**Note**: Accordion uses an exclusive-expand pattern (only one section open). CSS `max-height:0` collapses the closed bodies, but the text content is still in the DOM — confirming aria state is the most reliable check.-------## Group 27: Contenteditable editor--### 27.1 Type into a `contenteditable` div-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/editor.html"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"type","selector":"#editor","text":"Hello rich text"}'--curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Response contains `EDITOR_CHARS=15` AND the mirror echoes `Hello rich text`.--### 27.2 Commit by pressing Enter-```bash-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"press","key":"Enter"}'--curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Response contains `EDITOR_COMMITTED=Hello rich text`.--**Note**: `contenteditable` elements don't have a `.value` property, so `fill` won't work. Use `type` so each character is dispatched via keyboard events the browser routes into the editor. `press Enter` is intercepted (preventDefault) to commit the buffer instead of inserting a newline.-------## Group 28: Range slider--### 28.1 Set range slider to HIGH bucket-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/range.html"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#volume","text":"90"}'--curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Response contains `RANGE_VALUE_90` AND `BUCKET_HIGH`.--### 28.2 Move to LOW bucket-```bash-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#volume","text":"10"}'--curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Response contains `RANGE_VALUE_10` AND `BUCKET_LOW`.--**Note**: `fill` calls the native value setter, which fires both `input` and `change` events. That's enough to drive a page that listens via `addEventListener('input', ...)`. If the target only listens to pointer drag events, `drag --drag-x N` on the thumb would be needed — but most sites use `input`/`change`.-------## Group 29: Pagination--### 29.1 Advance to page 2-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/pagination.html"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#next-btn"}'--curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Response contains `PAGE_2_FIRST_ITEM` AND `PAGE_2_OF_3` (and the page-1 marker is gone).--### 29.2 Advance to last page; Next is disabled-```bash-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#next-btn"}'--curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \- -H "Authorization: Bearer benchmark-token"--curl -X POST http://localhost:9867/tabs/TAB_ID/evaluate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"expression":"document.getElementById(\"next-btn\").disabled"}'-```-**Pass if**: Text contains `PAGE_3_FIRST_ITEM` AND `PAGE_3_OF_3` AND the evaluate response is `true` (Next button is disabled on the last page).--**Note**: Pagination buttons re-render the list in place. If you re-snap after the click, refs `e0..eN` for the list items will be different than before.-------## Group 30: Custom dropdown menu--### 30.1 Open the dropdown and pick "Beta"-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/dropdown.html"}'--# Open the menu-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#dropdown-toggle"}'--# Pick the "Beta" item by its data-value attribute-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#dropdown-menu li[data-value=\"beta\"]"}'--curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Response contains `DROPDOWN_SELECTED=BETA`.--### 30.2 Reopen and pick "Gamma"-```bash-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#dropdown-toggle"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#dropdown-menu li[data-value=\"gamma\"]"}'--curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Response contains `DROPDOWN_SELECTED=GAMMA`.--**Note**: This is a non-native dropdown — it's a `<button>` that toggles a `<ul>`. The menu items are not `<option>` elements, so `pinchtab select` won't work. Use two clicks (toggle, then item). Click-outside also dismisses the menu, so don't do anything between the two clicks that would synthesize a body click.-------## Group 31: Nested iframes (3 levels deep)--### 31.1 Drill into the deepest frame and click its button-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/iframe-nested.html"}'--# Hop into level 2-curl -X POST http://localhost:9867/frame?tabId=TAB_ID \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"target":"#level-2"}'--# Hop further into level 3 (selector is resolved relative to current scope)-curl -X POST http://localhost:9867/frame?tabId=TAB_ID \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"target":"#level-3"}'--# Click the level-3 button-curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#deep-button"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \- -H "Authorization: Bearer benchmark-token"--curl -X POST http://localhost:9867/frame?tabId=TAB_ID \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"target":"main"}'-```-**Pass if**: Scoped snapshot contains `DEEP_CLICKED=YES_LEVEL_3`.--**Note**: Each `/frame` hop is relative to the current scope, so drilling-into a nested frame requires one hop per level. Use `{"target":"main"}`-to unwind to the top document in a single call.-------## Group 32: Dynamic iframe (inserted after load)--### 32.1 Wait for a late-inserted iframe then interact with it-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/iframe-dynamic.html"}'--# Wait for the marker that indicates the iframe was attached-curl -X POST http://localhost:9867/tabs/TAB_ID/wait \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"text":"IFRAME_DYNAMIC_ATTACHED","timeout":5000}'--# Scope into it and interact-curl -X POST http://localhost:9867/frame?tabId=TAB_ID \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"target":"#late-frame"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#iframe-input","text":"Late World"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#iframe-submit"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \- -H "Authorization: Bearer benchmark-token"--curl -X POST http://localhost:9867/frame?tabId=TAB_ID \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"target":"main"}'-```-**Pass if**: Scoped snapshot contains `IFRAME_INPUT_RECEIVED_LATE_WORLD`.--**Note**: The iframe is added to the DOM ~1.2 s after load. Snapshotting-immediately won't see it; `pinchtab wait --text` is the right primitive.-------## Group 33: srcdoc iframe (inline HTML, no src URL)--### 33.1 Interact with an inline-content iframe-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/iframe-srcdoc.html"}'--curl -X POST http://localhost:9867/frame?tabId=TAB_ID \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"target":"#srcdoc-frame"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#inline-input","text":"srcdoc"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#inline-submit"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \- -H "Authorization: Bearer benchmark-token"--curl -X POST http://localhost:9867/frame?tabId=TAB_ID \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"target":"main"}'-```-**Pass if**: Scoped snapshot contains `INLINE_RECEIVED_SRCDOC`.--**Note**: `srcdoc` iframes inherit the parent's origin (same-origin by-default), so frame scope works the same way as `src`-based iframes. The-returned `frameUrl` is `about:srcdoc`.-------## Group 34: Sandboxed iframe (allow-scripts allow-same-origin)--### 34.1 Click a button inside a sandbox="allow-scripts allow-same-origin" iframe-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/iframe-sandbox.html"}'--curl -X POST http://localhost:9867/frame?tabId=TAB_ID \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"target":"#sandboxed"}'--curl -X POST http://localhost:9867/tabs/TAB_ID/action \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#sandbox-button"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \- -H "Authorization: Bearer benchmark-token"--curl -X POST http://localhost:9867/frame?tabId=TAB_ID \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"target":"main"}'-```-**Pass if**: Scoped snapshot contains `SANDBOX_CLICKED=YES`.--**Note**: `sandbox="allow-scripts allow-same-origin"` permits scripts and-inherits the parent origin, so frame scope works. Sandboxes without-`allow-same-origin` force a unique opaque origin and would behave like-cross-origin — out of reach for the current `/frame` scope API.-------## Group 35: Long-form article (Medium/Substack style)--### 35.1 Readability keeps the article body-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/article.html"}'--# Default mode uses Readability; this is its sweet spot.-curl "http://localhost:9867/tabs/TAB_ID/text" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Response contains `ARTICLE_PUBLISHED_2026_04_15` AND `ARTICLE_WORD_COUNT_MARKER_323` (both inside the main article body — Readability should retain them).--### 35.2 `--full` picks up the surrounding chrome Readability drops-```bash-curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Response contains `ARTICLE_PUBLISHED_2026_04_15` AND `FOOTER_COPYRIGHT_MARKER`. Default Readability trims the footer; `--full` keeps it. Proves the two modes behave as documented.-------## Group 36: Search results page (SERP)--### 36.1 Extract a specific result title by id-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/serp.html"}'--# Use a scoped snapshot to extract just one card — faster than full text.-curl "http://localhost:9867/tabs/TAB_ID/snapshot?selector=%23r-3&format=compact&maxTokens=500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Snapshot contains `RESULT_3_TITLE` AND `RESULT_3_SNIPPET_MARKER`.--### 36.2 Count all six result cards-```bash-curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Response contains `RESULT_1_TITLE`, `RESULT_2_TITLE`, `RESULT_3_TITLE`, `RESULT_4_TITLE`, `RESULT_5_TITLE`, `RESULT_6_TITLE` (all 6), AND `SERP_RESULT_COUNT_6`. A SERP is the canonical case where Readability's single-article assumption fails — `--full` is required to see the list.-------## Group 37: Q&A thread (Stack-Overflow style)--### 37.1 Find the accepted answer-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/qa.html"}'--# Use eval to find the accepted answer id — structured page data beats-# text-parsing when the marker lives in an attribute.-curl -X POST http://localhost:9867/tabs/TAB_ID/evaluate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"expression":"document.querySelector(\"[data-accepted=\\\"true\\\"]\").id"}'-```-**Pass if**: Response `.result` equals `"a-2"`. The accepted-answer marker is in a data attribute; this exercises `eval` as a structured alternative to text scraping.--### 37.2 Extract the accepted answer body text-```bash-# Snapshot scoped to the accepted answer, with max-token budget-curl "http://localhost:9867/tabs/TAB_ID/snapshot?selector=%23a-2&format=compact&maxTokens=1500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Scoped snapshot contains `ANSWER_2_BODY_MARKER` AND `ACCEPTED_ANSWER_ID_A2`. Tests `snapshot?selector=` for scoped reads — a common pattern when a page has many independent sections.-------## Group 38: Pricing table--### 38.1 Extract the Pro plan price-```bash-curl -X POST http://localhost:9867/navigate \- -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/pricing.html"}'--curl "http://localhost:9867/tabs/TAB_ID/snapshot?selector=%23plan-pro&format=compact&maxTokens=500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Snapshot contains `PLAN_PRO_PRICE_29` AND `PLAN_PRO_LIMIT_5000_requests per day`.--### 38.2 Extract all three plan prices in one pass-```bash-curl "http://localhost:9867/tabs/TAB_ID/text?mode=raw&format=text" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Response contains `PLAN_FREE_PRICE_0`, `PLAN_PRO_PRICE_29`, AND `PLAN_ENTERPRISE_PRICE_CUSTOM`. Pricing tables are grid-heavy; Readability would pick one column. `--full` returns the whole surface.-------## Summary--| Group | Tasks | Description |-|-------|-------|-------------|-| 0 | 8 | Setup & Diagnosis |-| 1 | 6 | Reading & Extracting |-| 2 | 3 | Search & Dynamic |-| 3 | 2 | Form |-| 4 | 3 | SPA |-| 5 | 2 | Login |-| 6 | 3 | E-commerce |-| 7 | 2 | Content + Interaction |-| 8 | 2 | Error Handling |-| 9 | 2 | Export |-| 10 | 2 | Modals |-| 11 | 2 | Persistence |-| 12 | 2 | Multi-page Nav |-| 13 | 2 | Form Validation |-| 14 | 2 | Dynamic Content |-| 15 | 2 | Data Aggregation |-| 16 | 2 | Hover & Tooltips |-| 17 | 2 | Scrolling |-| 18 | 1 | File Download |-| 19 | 2 | iFrame |-| 20 | 2 | Dialogs |-| 21 | 2 | Async / awaitPromise |-| 22 | 2 | Mouse Drag & Drop |-| 23 | 1 | Async / Loading state |-| 24 | 2 | Keyboard events |-| 25 | 2 | Tab panels |-| 26 | 2 | Accordion |-| 27 | 2 | Contenteditable editor |-| 28 | 2 | Range slider |-| 29 | 2 | Pagination |-| 30 | 2 | Custom dropdown menu |-| 31 | 1 | Nested iframes (3 levels) |-| 32 | 1 | Dynamic iframe |-| 33 | 1 | srcdoc iframe |-| 34 | 1 | Sandboxed iframe |-| 35 | 2 | Long-form article (text extraction) |-| 36 | 2 | Search results page (SERP) |-| 37 | 2 | Q&A thread (Stack-Overflow style) |-| 38 | 2 | Pricing table |--**Total: 85 tasks**--## Verification Strings--| Page | String |-|------|--------|-| Home | `VERIFY_HOME_LOADED_12345` |-| Articles | `VERIFY_ARTICLES_PAGE_67890` |-| Search | `VERIFY_SEARCH_PAGE_11111` |-| Form | `VERIFY_FORM_PAGE_22222` |-| Dashboard | `VERIFY_DASHBOARD_PAGE_33333` |-| Shop | `VERIFY_SHOP_PAGE_44444` |-| Wiki Index | `VERIFY_WIKI_INDEX_55555` |-| Login | `VERIFY_LOGIN_PAGE_77777` |-| Go Article | `VERIFY_WIKI_GO_LANG_88888` |-| SPA | `VERIFY_SPA_PAGE_99999` |-| Python Article | `VERIFY_WIKI_PYTHON_LANG` |-| Rust Article | `VERIFY_WIKI_RUST_LANG` |-| Async | `VERIFY_ASYNC_PAGE_77777` |-| Drag | `VERIFY_DRAG_PAGE_33333` |-| Loading | `VERIFY_LOADING_PAGE_11111` |-| Keyboard | `VERIFY_KEYBOARD_PAGE_22222` |-| Tabs | `VERIFY_TABS_PAGE_55555` |-| Accordion | `VERIFY_ACCORDION_PAGE_66666` |-| Editor | `VERIFY_EDITOR_PAGE_44444` |-| Range | `VERIFY_RANGE_PAGE_99999` |-| Pagination | `VERIFY_PAGINATION_PAGE_88888` |-| Dropdown | `VERIFY_DROPDOWN_PAGE_77777` |-| Iframe Nested (outer) | `VERIFY_IFRAME_NESTED_OUTER` |-| Iframe Nested (L2) | `VERIFY_IFRAME_NESTED_L2` |-| Iframe Nested (L3) | `VERIFY_IFRAME_NESTED_L3` |-| Iframe Dynamic | `VERIFY_IFRAME_DYNAMIC_OUTER` |-| Iframe Srcdoc | `VERIFY_IFRAME_SRCDOC_OUTER` |-| Iframe Sandbox | `VERIFY_IFRAME_SANDBOX_OUTER` |-| Article | `VERIFY_ARTICLE_PAGE_41414` |-| SERP | `SERP_PAGE_MARKER_88044` |-| Q&A | `VERIFY_QA_PAGE_72000` |-| Pricing | `VERIFY_PRICING_PAGE_30303` |diff --git a/tests/benchmark/CRON_TASK.md b/tests/benchmark/CRON_TASK.mddeleted file mode 100644index 75e31857..00000000--- a/tests/benchmark/CRON_TASK.md+++ /dev/null@@ -1,142 +0,0 @@-# PinchTab Optimization Cron Task--**Goal: improve the agent lane against a known-good benchmark environment.**--Baseline is not part of the recurring agent task. It is a direct shell-verification that should be run after infrastructure, fixture, or benchmark-changes to confirm the environment still behaves as expected.-------## Baseline Gate--Run baseline manually when any of these change:--- PinchTab server or CLI behavior-- benchmark fixtures-- `tests/benchmark/scripts/baseline.sh`-- benchmark Docker setup--Command:--```bash-cd ~/dev/pinchtab/tests/benchmark-./scripts/run-optimization.sh-./scripts/baseline.sh-./scripts/finalize-report.sh "$(cat ./results/current_baseline_report.txt)"-```--Only continue with optimization work if baseline is green.-------## Recurring Optimization Task--### Step 1 — Setup-```bash-cd ~/dev/pinchtab/tests/benchmark-git checkout feat/benchmark && git pull --rebase origin feat/benchmark-./scripts/run-optimization.sh-# Note the TIMESTAMP from output-```--### Step 2 — Run Agent Benchmark-Execute `AGENT_TASKS.md` using `skills/pinchtab/SKILL.md` as the guide.--- Use the benchmark wrapper `./scripts/pt`-- Record results with `record-step.sh --type agent`-- Verify answers with `verify-step.sh`-- Log every command executed-- Do not use baseline as an execution guide--### Step 3 — Gap Analysis-Compare agent results against the latest green baseline report and classify failures:--| Failure Type | Cause | Fix |-|---|---|---|-| **Wrong endpoint** | Agent used `/text` when should use `/snapshot` | Improve `SKILL.md` |-| **Wrong selector** | Agent guessed selector incorrectly | Improve `SKILL.md` or fixture |-| **Missing step** | Agent skipped a required action | Clarify `AGENT_TASKS.md` |-| **Wrong URL** | Agent used wrong fixture path | Clarify `AGENT_TASKS.md` |-| **API bug** | Endpoint behaves unexpectedly | Fix PinchTab code |-| **Test ambiguity** | Verification string hard to find | Fix fixture or `scripts/baseline.sh` |--### Step 4 — Make Exactly 1 Change--Priority order:-1. **API Bug** → Fix PinchTab Go code, commit as `fix: <description>`-2. **Skill Gap** → Improve `SKILL.md`, commit as `docs(skill): <description>`-3. **Test Ambiguity** → Fix fixture HTML or `scripts/baseline.sh`, commit as `test: <description>`-4. **Agent Task Clarity** → Improve `AGENT_TASKS.md`, commit as `test(agent): <description>`-5. **No Gaps Found** → Add new benchmark coverage, with matching baseline + agent cases--**If agent is already at 100%**: add harder cases.--### Step 5 — Verify the Fix Makes Sense-Before committing, ask:--- will this change help the agent pick the right tool next time?-- if this is a skill/docs fix, does it directly address the observed failure?-- if this is a benchmark fix, is the expected signal reachable via the real tool flow?--### Step 6 — Commit & Push-```bash-cd ~/dev/pinchtab-git add -A-git commit -m "<type>: <clear description of what changed and why>"-git push origin feat/benchmark-```--### Step 7 — Log the Run-Append to `results/optimization_log.md`:--```markdown-## Run #N — YYYY-MM-DD HH:MM--**Results:**-- Baseline reference: X/Y (Z%)-- Agent: X/Y (Z%)-- Gap: N steps--**Agent Failures:**-- Step X.Y: [failure type] — [what went wrong]-- Step X.Y: [failure type] — [what went wrong]--**Root Cause:**-[One clear sentence explaining the pattern]--**Change Made:**-- Type: api|skill|test|agent-task-- Description: [What changed]-- Expected impact: [Why this should close the gap]-- Commit: [hash]--**Next Focus:**-[What to watch in the next run]-```--### Step 8 — Report to User-Send a concise update:--```text-Run #N complete-Baseline reference: X% | Agent: Y% | Gap: N steps-Failure: [brief description]-Fix: [what changed] ([commit])-```-------## Success Criteria--- **Short term**: Agent pass rate ≥ 95%-- **Long term**: Agent matches the latest green baseline on all existing tests-- **Ongoing**: When the gap closes, increase test complexity--## What NOT to do--- Do not run baseline as an agent task-- Do not make multiple changes per run-- Do not skip root-cause analysis-- Do not weaken tests just to improve the score-- Do not commit a change that does not directly address an observed failurediff --git a/tests/benchmark/OPTIMIZATION_LOOP.md b/tests/benchmark/OPTIMIZATION_LOOP.mddeleted file mode 100644index d8a1175e..00000000--- a/tests/benchmark/OPTIMIZATION_LOOP.md+++ /dev/null@@ -1,166 +0,0 @@-# PinchTab Auto-Optimization Loop--Automated improvement cycle for PinchTab benchmarks.--## Pre-requisite (Human, once)--```bash-cd ~/dev/pinchtab/tests/benchmark-docker compose up -d --build-# Wait for healthy:-curl -sf -H "Authorization: Bearer benchmark-token" http://localhost:9867/health-```--## Loop Steps--```-┌─────────────────────────────────────────────────────────────┐-│ 1. RUN AGENT BENCHMARK [Sub-agent] │-│ - Runs every iteration │-│ - Load AGENT_TASKS.md + skills/pinchtab/SKILL.md │-│ - Execute tasks using only skill docs │-│ - Record via: record-step.sh --type agent │-│ - Output: results/agent_benchmark_<timestamp>.json │-└─────────────────────────────────────────────────────────────┘- ↓-┌─────────────────────────────────────────────────────────────┐-│ 2. CHECK FOR REGRESSION [Main agent] │-│ - Read both JSON reports for pass/fail counts │-│ - Compare against results/best_score.txt │-│ - If worse: flag regression, prioritize fix/revert │-│ - If equal or better: update best_score.txt │-└─────────────────────────────────────────────────────────────┘- ↓-┌─────────────────────────────────────────────────────────────┐-│ 3. ANALYZE DIFFERENCES [Main agent] │-│ - Diff baseline vs agent results side by side │-│ - Identify which steps agent failed but baseline passed │-│ - Find patterns (wrong endpoint, selectors, etc.) │-└─────────────────────────────────────────────────────────────┘- ↓-┌─────────────────────────────────────────────────────────────┐-│ 4. PROPOSE 1 IMPROVEMENT [Main agent → Human]│-│ Priority order: │-│ a) Fix PinchTab CLI/REST bug if found │-│ b) Improve skill documentation if agent confused │-│ c) Add verification to existing test case │-│ d) Add new test case for uncovered scenario │-│ Present proposal to human for approval. │-└─────────────────────────────────────────────────────────────┘- ↓-┌─────────────────────────────────────────────────────────────┐-│ 5. IMPLEMENT (no commit) [Main agent] │-│ - Make the single change │-│ - Leave changes uncommitted for manual review │-└─────────────────────────────────────────────────────────────┘- ↓-┌─────────────────────────────────────────────────────────────┐-│ 6. LOG RUN [Main agent] │-│ - Append to results/optimization_log.md │-│ - Record: timestamp, pass rates, regression check, │-│ change made, next focus │-└─────────────────────────────────────────────────────────────┘-```--## Roles--| Step | Who | What |-|------|-----|------|-| Pre-req | **Human** | Start Docker, verify health |-| 1 Agent | **Sub-agent** | Execute agent tasks from skill docs, record results |-| 2 Regression | **Main agent** | Compare against best_score.txt |-| 3 Analyze | **Main agent** | Diff baseline vs agent, find patterns |-| 4 Propose | **Main agent → Human** | Suggest 1 fix, get approval |-| 5 Implement | **Main agent** | Make the change, leave uncommitted |-| 6 Log | **Main agent** | Append to optimization_log.md |--## Baseline is a Shell Verification, Not an Agent Run--The baseline suite (`scripts/baseline.sh`) validates that the benchmark infrastructure itself works — fixtures serve correctly, PinchTab APIs behave as expected, and the test conditions are reachable. It's deterministic (no LLM involved), so once it's at 100% it stays at 100% unless something changes underneath it.--Run baseline directly as a shell command:--```bash-./scripts/run-optimization.sh-./scripts/baseline.sh-```--**The optimization loop does NOT treat baseline as an agent task and does NOT re-run it every iteration.** Instead, run the baseline manually when any of these change:--- PinchTab server code (anything in `internal/` that affects the HTTP API)-- Fixtures (anything in `tests/benchmark/fixtures/`)-- `scripts/baseline.sh` itself (adding/modifying tests)-- `docker-compose.yml` or the Docker image--**Workflow for adding a new test case:**--1. Write the executable baseline case in `scripts/baseline.sh`.- If you keep `BASELINE_TASKS.md` around as reference material, update it second.-2. Write the matching task in `AGENT_TASKS.md` (natural language)-3. Write the summary row in `TEST_CASES.md`-4. Add any new fixture files needed-5. **Run the baseline shell verification** — it must reach 100% before the case is considered valid-6. If baseline fails: fix the test/fixture/PinchTab until it passes-7. Commit the new case — now the agent loop can use it--Once baseline is green, the optimization loop only runs the agent suite.--## Regression Detection--Each run compares against `results/best_score.txt`, which tracks the highest pass rates achieved:--```-baseline=62/68-agent=33/39-```--- **If current run is worse**: Regression detected. The improvement priority shifts to fixing or reverting the last change before proposing anything new.-- **If equal or better**: Update `best_score.txt` with the new high-water mark.--## Improvement Priority--1. **API/CLI Bug** — If a curl command returns unexpected error, investigate PinchTab code-2. **Skill Gap** — If agent uses wrong endpoint/approach, improve SKILL.md documentation-3. **Benchmark Gap** — If tests don't verify important behavior, add verification-4. **Coverage Gap** — If common scenarios aren't tested, add test cases--## Log Format--Each run appends to `results/optimization_log.md`:--```markdown-## Run #N — YYYY-MM-DD HH:MM--**Results:**-- Baseline: X/Y (Z%)-- Agent: X/Y (Z%)--**Regression Check:**-- Best: baseline X/Y, agent X/Y-- This run: [better | same | REGRESSION]--**Analysis:**-- [What differed between runs]-- [Root cause identified]--**Change Made:**-- [Type: api|skill|benchmark]-- [Description]-- [Status: uncommitted — pending manual review]--**Next Focus:**-- [What to look at next run]-```--## Files--- `scripts/baseline.sh` — Executable baseline lane source of truth-- `scripts/run-optimization.sh` — Main loop script (initializes reports)-- `scripts/record-step.sh` — Appends a step result to the current report-- `results/optimization_log.md` — Run history (includes per-run metrics table)-- `results/best_score.txt` — High-water mark for regression detection-- Changes left uncommitted for manual review and commit--Agent runs use the benchmark wrapper `./scripts/pt`, which forwards to the-PinchTab CLI inside Docker while preserving `PINCHTAB_TAB` targeting. See-`AGENT_TASKS.md` "Recommended setup" for the exact pattern.diff --git a/tests/benchmark/README.md b/tests/benchmark/README.mdindex b54fadf8..862dd2d1 100644--- a/tests/benchmark/README.md+++ b/tests/benchmark/README.md@@ -17,24 +17,20 @@ cd tests/benchmark# agent-browser lane./scripts/run-agent-browser-benchmark.sh-# Then read AGENT_BROWSER_INSTRUCTIONS.md and run tasks from AGENT_BROWSER_TASKS.md with ./scripts/ab+# Then read setup-agent-browser.md and benchmark-run/index.md and run tasks with ./scripts/ab./scripts/finalize-report.sh-# optional provider API runner-OPENAI_API_KEY=... node --experimental-strip-types ./scripts/run-api-benchmark.ts --lane agent --finalize-OPENAI_API_KEY=... node --experimental-strip-types ./scripts/run-api-benchmark.ts --lane agent-browser --finalize-ANTHROPIC_API_KEY=... node --experimental-strip-types ./scripts/run-api-benchmark.ts --lane agent --model claude-haiku-4-5-20251001 --finalize-ANTHROPIC_API_KEY=... node --experimental-strip-types ./scripts/run-api-benchmark.ts --lane agent-browser --model claude-haiku-4-5-20251001 --finalize--# repo entrypoints+# repo entrypoints (recommended)./dev benchmark baseline-OPENAI_API_KEY=... ./dev benchmark agent-OPENAI_API_KEY=... ./dev benchmark agent-browser-ANTHROPIC_API_KEY=... ./dev benchmark agent-ANTHROPIC_API_KEY=... ./dev benchmark agent-browser-ANTHROPIC_API_KEY=... ./dev benchmark agent --provider anthropic --turn-delay-ms 3000-ANTHROPIC_API_KEY=... ./dev benchmark agent --provider anthropic --profile common10-ANTHROPIC_API_KEY=... ./dev benchmark agent-browser --provider anthropic --groups 0,1,2,3+./dev benchmark pinchtab --dry-run+OPENAI_API_KEY=... ./dev benchmark pinchtab+ANTHROPIC_API_KEY=... ./dev benchmark pinchtab --profile common10+ANTHROPIC_API_KEY=... ./dev benchmark pinchtab --max-input-tokens 50000+ANTHROPIC_API_KEY=... ./dev benchmark agent-browser --groups 0,1,2,3++# direct Go runner+ANTHROPIC_API_KEY=... go run ./cmd/api-runner --lane pinchtab --finalize+ANTHROPIC_API_KEY=... go run ./cmd/api-runner --lane agent-browser --finalize```## MANDATORY: Docker Environment@@ -54,9 +50,10 @@ If Docker build fails or is skipped, the benchmark is **INVALID**.| File | Purpose ||------|---------|| `../../skills/pinchtab/SKILL.md` | PinchTab skill (same as shipped product) |-| `AGENT_BROWSER_INSTRUCTIONS.md` | Benchmark-local instructions for the `agent-browser` lane |-| `BASELINE_TASKS.md` | Legacy descriptive baseline spec; not executed by the harness |-| `AGENT_BROWSER_TASKS.md` | Equivalent task lane for `agent-browser` |+| `setup-pinchtab.md` | PinchTab lane setup and wrapper guidance |+| `setup-agent-browser.md` | agent-browser lane setup and wrapper guidance |+| `benchmark-run/index.md` | Shared benchmark task index for the agent-driven lanes |+| `benchmark-run/group-XX.md` | One markdown file per benchmark group || `scripts/baseline.sh` | Executable baseline lane source of truth || `scripts/run-optimization.sh` | Initialize PinchTab benchmark reports || `scripts/run-agent-browser-benchmark.sh` | Start fixtures + `agent-browser` and initialize a fresh report |@@ -139,22 +136,22 @@ The benchmark runs PinchTab in Docker with:Every step should record the observed answer/result:```bash-./scripts/record-step.sh --type agent <group> <step> <answer|fail|skip> "answer" "notes"+./scripts/record-step.sh --type pinchtab <group> <step> <answer|fail|skip> "answer" "notes"```Example:```bash-./scripts/record-step.sh --type agent 1 1 answer "Navigation completed in 1.2s" "observed output"-./scripts/record-step.sh --type agent 2 3 fail "Element not found"+./scripts/record-step.sh --type pinchtab 1 1 answer "Navigation completed in 1.2s" "observed output"+./scripts/record-step.sh --type pinchtab 2 3 fail "Element not found"```Deferred-verification example:```bash-./scripts/record-step.sh --type agent 1 1 answer \+./scripts/record-step.sh --type pinchtab 1 1 answer \"Found categories Programming Languages: 12, Databases: 8" \"raw answer"-./scripts/verify-step.sh --type agent 1 1 pass \+./scripts/verify-step.sh --type pinchtab 1 1 pass \"Answer satisfies the benchmark oracle"```diff --git a/tests/benchmark/benchmark-run/group-00.md b/tests/benchmark/benchmark-run/group-00.mdnew file mode 100644index 00000000..5e5fd34a--- /dev/null+++ b/tests/benchmark/benchmark-run/group-00.md@@ -0,0 +1,44 @@+# Group 0: Setup & Diagnosis++### 0.1 Server reachable+Check that the PinchTab server is healthy.++**Verify**: Server responds with `status: ok`.++### 0.2 Auth is required+Make a request to the server with a **wrong** token (`PINCHTAB_TOKEN=wrong-token ./scripts/pt health`) and confirm it is rejected. The `pt` wrapper always injects the benchmark token by default, so you must explicitly override it to test auth rejection.++**Verify**: Response is HTTP 401 or contains `unauthorized`.++### 0.3 Auth works with token+Repeat the same request WITH the bearer token and confirm it succeeds.++**Verify**: Response is HTTP 200.++### 0.4 Instance available+Confirm at least one Chrome instance is running. If none exist, start one.++**Verify**: Health response shows `defaultInstance.status == "running"` (or after starting one, the new instance is running).++### 0.5 List existing tabs+Get the current list of open tabs.++**Verify**: A list (possibly empty) is returned without error.++### 0.6 Clean stale tabs+If any tabs from previous runs are open, close them so the benchmark starts from a known state.++**Verify**: After cleanup, the tab list is empty (or only contains a single about:blank tab).++### 0.7 Network reach to target+Navigate to `http://fixtures/` and confirm the fixtures server is reachable from PinchTab.++**Verify**: Navigate returns successfully and the page contains benchmark content.++### 0.8 Capture initial tab ID+Save the tab ID returned by the navigate in 0.7. Use this tab ID for all subsequent tasks to avoid creating new tabs.++**Verify**: A tab ID was captured and matches what `GET /tabs` reports.++---+diff --git a/tests/benchmark/benchmark-run/group-01.md b/tests/benchmark/benchmark-run/group-01.mdnew file mode 100644index 00000000..9ae2ee54--- /dev/null+++ b/tests/benchmark/benchmark-run/group-01.md@@ -0,0 +1,34 @@+# Group 1: Reading & Extracting Real Content++### 1.1 Get the full list of categories from the wiki index+Navigate to `http://fixtures/wiki.html` and extract all category names with their article counts.++**Verify**: Can name at least 2 categories and their article counts (e.g. "Programming Languages: 12 articles").++### 1.2 Navigate by clicking a link+From the wiki index, click the "Go (programming language)" link to navigate to the Go article.++**Verify**: You are now on the Go article page (not the wiki index).++### 1.3 Extract structured data from a table+On the Go article, read the infobox table and answer: Who designed Go, and what year did it first appear?++**Verify**: Answer contains "Robert Griesemer" (or "Rob Pike" or "Ken Thompson") and "2009".++### 1.4 Count list items+On the Go article, count how many key features are listed.++**Verify**: Answer is 6 (verify against `FEATURE_COUNT_6`).++### 1.5 Read all article headlines from articles page+Navigate to `http://fixtures/articles.html` and list all article titles.++**Verify**: Found at least 3 articles including "The Future of Artificial Intelligence".++### 1.6 Read dashboard metrics+Navigate to `http://fixtures/dashboard.html` and extract: Total Users, Revenue, and Conversion Rate.++**Verify**: Found `24,582` users AND `$1,284,930` revenue.++---+diff --git a/tests/benchmark/benchmark-run/group-02.md b/tests/benchmark/benchmark-run/group-02.mdnew file mode 100644index 00000000..78be52a2--- /dev/null+++ b/tests/benchmark/benchmark-run/group-02.md@@ -0,0 +1,19 @@+# Group 2: Search & Dynamic Interaction++### 2.1 Use wiki search to find a page+On `http://fixtures/wiki.html`, search for "golang" using the search form. Do not navigate directly — use the search input.++**Verify**: Ended up on the Go article page after search.++### 2.2 Search with no results+On `http://fixtures/search.html`, search for something with no results (use "xyznonexistent").++**Verify**: Page handled it gracefully (no crash, some response rendered).++### 2.3 Search for AI content+On `http://fixtures/search.html`, search for "artificial intelligence" and verify a result appeared.++**Verify**: Result contains "The Future of Artificial Intelligence".++---+diff --git a/tests/benchmark/benchmark-run/group-03.md b/tests/benchmark/benchmark-run/group-03.mdnew file mode 100644index 00000000..34de80e3--- /dev/null+++ b/tests/benchmark/benchmark-run/group-03.md@@ -0,0 +1,23 @@+# Group 3: Complex Form++### 3.1 Fill and submit a complete form+Navigate to `http://fixtures/form.html`. Complete the entire form:+- Full name: "Agent Test User"+- Email: "[email protected]"+- Phone: "+44 20 9999 0000"+- Country: United Kingdom+- Subject: Technical Support+- Message: "Testing PinchTab form automation"+- Check newsletter+- Set priority to High+- Submit++**Verify**: Form submitted successfully. Confirmation shows name "AGENT_TEST_USER".++### 3.2 Reset and refill+After submitting, if the form is still accessible (or navigate back), verify you can identify the reset button.++**Verify**: Reset/back button or form element found in snapshot.++---+diff --git a/tests/benchmark/benchmark-run/group-04.md b/tests/benchmark/benchmark-run/group-04.mdnew file mode 100644index 00000000..bf5978c4--- /dev/null+++ b/tests/benchmark/benchmark-run/group-04.md@@ -0,0 +1,19 @@+# Group 4: SPA State Management++### 4.1 Read initial app state+Navigate to `http://fixtures/spa.html?reset=1` (the `?reset=1` query param clears any previous localStorage state so the SPA starts with its default 3 tasks). Read the current task list — how many tasks exist, how many are active vs done?++**Verify**: Found 3 total, 2 active, 1 done (verify `TASK_STATS_TOTAL_3_ACTIVE_2_DONE_1`).++### 4.2 Add a new high-priority task+Add a task called "Automate deployment" with high priority.++**Verify**: Task appeared in the list (`TASK_ADDED_AUTOMATE_DEPLOYMENT_PRIORITY_HIGH`).++### 4.3 Delete a task+Delete the task titled "Write benchmark tests".++**Verify**: Task count changed (went from 4 to 3).++---+diff --git a/tests/benchmark/benchmark-run/group-05.md b/tests/benchmark/benchmark-run/group-05.mdnew file mode 100644index 00000000..ba3d630a--- /dev/null+++ b/tests/benchmark/benchmark-run/group-05.md@@ -0,0 +1,14 @@+# Group 5: Login Flow++### 5.1 Attempt login with wrong credentials+Navigate to `http://fixtures/login.html`. Try to log in with username "admin" and password "wrong".++**Verify**: Error message appeared (`INVALID_CREDENTIALS_ERROR`).++### 5.2 Login successfully+Clear the form and log in with username "benchmark" / password "test456".++**Verify**: Dashboard appeared after login (`VERIFY_LOGIN_SUCCESS_DASHBOARD`).++---+diff --git a/tests/benchmark/benchmark-run/group-06.md b/tests/benchmark/benchmark-run/group-06.mdnew file mode 100644index 00000000..37c049f7--- /dev/null+++ b/tests/benchmark/benchmark-run/group-06.md@@ -0,0 +1,19 @@+# Group 6: Multi-Step E-commerce++### 6.1 Research products before buying+Navigate to `http://fixtures/ecommerce.html`. List all available products with their prices. Which product is out of stock?++**Verify**: Found Wireless Headphones ($149.99), Smart Watch Pro ($299.99), Portable Charger ($49.99). Mechanical Keyboard is out of stock.++### 6.2 Add two items and verify total+Add Wireless Headphones and Smart Watch Pro to cart. What is the total?++**Verify**: Cart total is $449.98 (149.99 + 299.99).++### 6.3 Complete checkout+Click checkout to complete the order.++**Verify**: Order confirmation shows (`VERIFY_CHECKOUT_SUCCESS_ORDER`).++---+diff --git a/tests/benchmark/benchmark-run/group-07.md b/tests/benchmark/benchmark-run/group-07.mdnew file mode 100644index 00000000..41c1838f--- /dev/null+++ b/tests/benchmark/benchmark-run/group-07.md@@ -0,0 +1,14 @@+# Group 7: Content + Interaction Combined++### 7.1 Read and comment on Go article+Navigate to `http://fixtures/wiki-go.html`. Read the article, then post a comment with rating 5 stars and text "Excellent reference".++**Verify**: Comment posted (`COMMENT_POSTED_RATING_5_TEXT_RECEIVED`).++### 7.2 Cross-page research task+Navigate to wiki index, find which category has the most articles, then navigate to one of its listed items.++**Verify**: Successfully navigated to at least one article page after reading the category counts.++---+diff --git a/tests/benchmark/benchmark-run/group-08.md b/tests/benchmark/benchmark-run/group-08.mdnew file mode 100644index 00000000..ddfc6817--- /dev/null+++ b/tests/benchmark/benchmark-run/group-08.md@@ -0,0 +1,14 @@+# Group 8: Error Handling++### 8.1 Handle 404 gracefully+Try to navigate to a page that doesn't exist: `http://fixtures/missing-page-abc.html`.++**Verify**: Got a response (404 or error), no crash, server still responsive after.++### 8.2 Handle missing element gracefully+On any page, try to click an element with ID `#fake-button-that-does-not-exist`.++**Verify**: Got a clear error message, not a crash or hang.++---+diff --git a/tests/benchmark/benchmark-run/group-09.md b/tests/benchmark/benchmark-run/group-09.mdnew file mode 100644index 00000000..5f2ccedf--- /dev/null+++ b/tests/benchmark/benchmark-run/group-09.md@@ -0,0 +1,14 @@+# Group 9: Export++### 9.1 Screenshot a complex page+Navigate to `http://fixtures/dashboard.html` and take a screenshot.++**Verify**: Screenshot generated (file saved or base64 returned).++### 9.2 Export a page as PDF+Export the dashboard as a PDF.++**Verify**: PDF generated.++---+diff --git a/tests/benchmark/benchmark-run/group-10.md b/tests/benchmark/benchmark-run/group-10.mdnew file mode 100644index 00000000..a948108e--- /dev/null+++ b/tests/benchmark/benchmark-run/group-10.md@@ -0,0 +1,14 @@+# Group 10: Nested Interactions & Modal Dialogs++### 10.1 Open and interact with modal on dashboard+Navigate to `http://fixtures/dashboard.html`. Find and click the Settings button (selector: `#settings-btn`) to open the modal dialog.++**Verify**: Modal appeared — snapshot contains "Dashboard Settings".++### 10.2 Modify settings and close modal+In the modal, select "Dark" from the theme dropdown (`#theme-select`), then click the Save button (`#modal-save`). After the modal closes, check the page content.++**Verify**: Page contains `THEME_DARK_APPLIED`.++---+diff --git a/tests/benchmark/benchmark-run/group-11.md b/tests/benchmark/benchmark-run/group-11.mdnew file mode 100644index 00000000..e4b85aff--- /dev/null+++ b/tests/benchmark/benchmark-run/group-11.md@@ -0,0 +1,14 @@+# Group 11: State Persistence & Page Reload++### 11.1 Add an item and verify after page reload+Navigate to `http://fixtures/spa.html?reset=1` (starts with clean state). Add a task titled exactly "Persistent Task Test". Then navigate away to `http://fixtures/` and back to `http://fixtures/spa.html` (WITHOUT the reset param) to verify localStorage persistence.++**Verify**: After reload, the task still appears in the list (`TASK_PERSISTENT_TEST_FOUND_AFTER_RELOAD`).++### 11.2 Logout and log back in+From the logged-in dashboard, click Sign Out to log out. Then log in again with username "benchmark" / password "test456".++**Verify**: Successfully logged back in and dashboard shows `SESSION_RENEWED`.++---+diff --git a/tests/benchmark/benchmark-run/group-12.md b/tests/benchmark/benchmark-run/group-12.mdnew file mode 100644index 00000000..c17fb7bf--- /dev/null+++ b/tests/benchmark/benchmark-run/group-12.md@@ -0,0 +1,14 @@+# Group 12: Multi-Page Navigation & Back Button++### 12.1 Navigate through multiple pages and return+Starting from `http://fixtures/`, navigate to wiki → Go article → back to wiki → back to home.++**Verify**: Successfully returned to home page (title contains "Benchmark" or "Home").++### 12.2 Compare data across pages+Navigate to wiki.html, note the total article count from categories. Navigate to articles.html, count articles there. Compare totals.++**Verify**: Can report totals from both pages and explain difference (`COMPARISON_DATA_FOUND`).++---+diff --git a/tests/benchmark/benchmark-run/group-13.md b/tests/benchmark/benchmark-run/group-13.mdnew file mode 100644index 00000000..310a7fe6--- /dev/null+++ b/tests/benchmark/benchmark-run/group-13.md@@ -0,0 +1,14 @@+# Group 13: Form State & Multi-Step Submission++### 13.1 Submit form without email+Navigate to `http://fixtures/form.html`. Fill only the name field ("Validator Test"), leave email blank, click Submit. The browser's native required-field validation will prevent submission.++**Verify**: Submission blocked (form stays open, no success message shown).++### 13.2 Submit form without optional phone field+Fill the form with: name "No Phone User", email "[email protected]", country "de", subject "feedback". Leave the phone field empty. Submit.++**Verify**: Submission succeeded and page shows `OPTIONAL_FIELD_SKIPPED_SUCCESS`.++---+diff --git a/tests/benchmark/benchmark-run/group-14.md b/tests/benchmark/benchmark-run/group-14.mdnew file mode 100644index 00000000..865d6b3f--- /dev/null+++ b/tests/benchmark/benchmark-run/group-14.md@@ -0,0 +1,14 @@+# Group 14: Dynamic Content Loading++### 14.1 Load more products+Navigate to `http://fixtures/ecommerce.html`. Find and click the "Load More Products" button to reveal additional products.++**Verify**: Additional products appeared (`ADDITIONAL_PRODUCTS_LOADED`).++### 14.2 Add a lazy-loaded product to cart+After loading more products, add product #5 (USB-C Cable) to the cart.++**Verify**: Cart shows the lazy-loaded item (`CART_UPDATED_WITH_LAZY_PRODUCT`).++---+diff --git a/tests/benchmark/benchmark-run/group-15.md b/tests/benchmark/benchmark-run/group-15.mdnew file mode 100644index 00000000..adcc745a--- /dev/null+++ b/tests/benchmark/benchmark-run/group-15.md@@ -0,0 +1,19 @@+# Group 15: Complex Data Extraction & Aggregation++### 15.1 Extract and sum financial data+Navigate to `http://fixtures/dashboard.html`. Extract revenue and profit values, calculate profit margin.++**Verify**: Correctly calculated: profit_margin = (profit / revenue) * 100 (`PROFIT_MARGIN_CALCULATED`).++### 15.2 Build comparison table from multiple sources+Visit these 3 pages and compare their feature counts and key features:+- `http://fixtures/wiki-go.html` (Go: 6 features)+- `http://fixtures/wiki-python.html` (Python: 7 features)+- `http://fixtures/wiki-rust.html` (Rust: 5 features)++Report which language has the most features and name 1 feature unique to each.++**Verify**: Response is factually correct AND wiki-python.html contains `COMPARISON_TABLE_BUILT`.++---+diff --git a/tests/benchmark/benchmark-run/group-16.md b/tests/benchmark/benchmark-run/group-16.mdnew file mode 100644index 00000000..73e24f19--- /dev/null+++ b/tests/benchmark/benchmark-run/group-16.md@@ -0,0 +1,14 @@+# Group 16: Hover & Tooltips++### 16.1 Hover reveals hidden content+Navigate to `http://fixtures/hovers.html`. Hover over the first avatar to reveal its hidden info.++**Verify**: After hovering, snapshot contains `HOVER_REVEALED_USER_1`.++### 16.2 Hover swap+Hover over the second avatar.++**Verify**: Snapshot contains `HOVER_REVEALED_USER_2`.++---+diff --git a/tests/benchmark/benchmark-run/group-17.md b/tests/benchmark/benchmark-run/group-17.mdnew file mode 100644index 00000000..7ca19c86--- /dev/null+++ b/tests/benchmark/benchmark-run/group-17.md@@ -0,0 +1,14 @@+# Group 17: Scrolling++### 17.1 Scroll by pixels+Navigate to `http://fixtures/scroll.html`. Scroll down approximately 1500 pixels.++**Verify**: Snapshot or text contains `SCROLL_MIDDLE_MARKER` (the mid-page marker is now in view).++### 17.2 Scroll to footer+Continue scrolling until the footer is visible (or scroll directly to the footer element).++**Verify**: Snapshot contains `SCROLL_REACHED_FOOTER`.++---+diff --git a/tests/benchmark/benchmark-run/group-18.md b/tests/benchmark/benchmark-run/group-18.mdnew file mode 100644index 00000000..0744fd1c--- /dev/null+++ b/tests/benchmark/benchmark-run/group-18.md@@ -0,0 +1,9 @@+# Group 18: File Download++### 18.1 Download a file+Download `http://fixtures/download-sample.txt` to a local file.++**Verify**: The downloaded file exists and its content includes `DOWNLOAD_FILE_CONTENT_VERIFIED`.++---+diff --git a/tests/benchmark/benchmark-run/group-19.md b/tests/benchmark/benchmark-run/group-19.mdnew file mode 100644index 00000000..bf8517a0--- /dev/null+++ b/tests/benchmark/benchmark-run/group-19.md@@ -0,0 +1,14 @@+# Group 19: iFrame++### 19.1 Read iframe content+Navigate to `http://fixtures/iframe.html` and extract content from inside the embedded same-origin iframe.++**Verify**: The iframe's inner content includes `IFRAME_INNER_CONTENT_LOADED`.++### 19.2 Type into iframe input (native frame scope)+Use `pinchtab frame '#content-frame'` to scope into the iframe, then `fill` the input with "Hello World" and `click` the Save button. Verify via a scoped `snap` — `text --full` doesn't pierce iframes. Reset with `pinchtab frame main` afterwards.++**Verify**: Scoped snapshot contains `IFRAME_INPUT_RECEIVED_HELLO_WORLD`.++---+diff --git a/tests/benchmark/benchmark-run/group-20.md b/tests/benchmark/benchmark-run/group-20.mdnew file mode 100644index 00000000..1a3ba53d--- /dev/null+++ b/tests/benchmark/benchmark-run/group-20.md@@ -0,0 +1,14 @@+# Group 20: Dialogs++### 20.1 Accept alert+Navigate to `http://fixtures/alerts.html`. Click the "Click for Alert" button and dismiss the alert.++**Verify**: The page result contains `DIALOG_ALERT_DISMISSED`.++### 20.2 Cancel confirm+Click the "Click for Confirm" button and cancel the confirm dialog.++**Verify**: The page result contains `DIALOG_CONFIRM_CANCELLED`.++---+diff --git a/tests/benchmark/benchmark-run/group-21.md b/tests/benchmark/benchmark-run/group-21.mdnew file mode 100644index 00000000..2ed0f123--- /dev/null+++ b/tests/benchmark/benchmark-run/group-21.md@@ -0,0 +1,14 @@+# Group 21: Async / awaitPromise++### 21.1 Await a promise-returning function+Navigate to `http://fixtures/async.html`. The page exposes `window.fetchPayload()`, which returns a `Promise` that resolves after a short delay. Use `eval` to call it and retrieve the **resolved** value, not a Promise wrapper.++**Verify**: The resolved value contains `ASYNC_PAYLOAD_READY_42`.++### 21.2 Await a promise resolving to an object+On the same page, call `window.fetchUser()` and retrieve the resolved object so you can read a field from it.++**Verify**: The resolved object's `name` field equals `ASYNC_USER_NAME_ADA`.++---+diff --git a/tests/benchmark/benchmark-run/group-22.md b/tests/benchmark/benchmark-run/group-22.mdnew file mode 100644index 00000000..d069d637--- /dev/null+++ b/tests/benchmark/benchmark-run/group-22.md@@ -0,0 +1,14 @@+# Group 22: Mouse Drag & Drop++### 22.1 Drag a piece into Zone A+Navigate to `http://fixtures/drag.html`. The page contains a draggable square (`#piece`) and three target zones (`#zone-a`, `#zone-b`, `#zone-c`). Drag the piece so its center ends up over **Zone A**.++**Verify**: The page shows `LAST_DROP=DROP_ZONE_A_OK`.++### 22.2 Drag to Zone B, then Zone C+Without reloading the page, drag the piece next into **Zone B**, and then into **Zone C**. The page records an ordered drop sequence.++**Verify**: The page shows `DROP_SEQUENCE=DROP_ZONE_A_OK,DROP_ZONE_B_OK,DROP_ZONE_C_OK` (all three drops in order).++---+diff --git a/tests/benchmark/benchmark-run/group-23.md b/tests/benchmark/benchmark-run/group-23.mdnew file mode 100644index 00000000..ce4cfeeb--- /dev/null+++ b/tests/benchmark/benchmark-run/group-23.md@@ -0,0 +1,9 @@+# Group 23: Async / Loading state++### 23.1 Wait for async content+Navigate to `http://fixtures/loading.html`. The page shows a spinner and the text "Loading..." for ~1.5 seconds, then replaces it with the final payload. Wait for the final content to appear before reporting completion.++**Verify**: The final snapshot contains `VERIFY_LOADING_COMPLETE_88888`.++---+diff --git a/tests/benchmark/benchmark-run/group-24.md b/tests/benchmark/benchmark-run/group-24.mdnew file mode 100644index 00000000..20192423--- /dev/null+++ b/tests/benchmark/benchmark-run/group-24.md@@ -0,0 +1,14 @@+# Group 24: Keyboard events++### 24.1 Press Escape+Navigate to `http://fixtures/keyboard.html` (it auto-focuses an input so keyboard events land). Press the Escape key.++**Verify**: Page text contains `KEYBOARD_ESCAPE_PRESSED`.++### 24.2 Press 'a' then Enter+Without reloading, press the `a` key, then the `Enter` key.++**Verify**: Page text now contains all three markers in order: `KEYBOARD_ESCAPE_PRESSED`, `KEYBOARD_KEY_A_PRESSED`, `KEYBOARD_ENTER_PRESSED`.++---+diff --git a/tests/benchmark/benchmark-run/group-25.md b/tests/benchmark/benchmark-run/group-25.mdnew file mode 100644index 00000000..370fafd0--- /dev/null+++ b/tests/benchmark/benchmark-run/group-25.md@@ -0,0 +1,14 @@+# Group 25: Tab panels++### 25.1 Switch to the Settings tab+Navigate to `http://fixtures/tabs.html`. The Profile tab is shown by default. Click the Settings tab.++**Verify**: After the click, the page content includes `TAB_SETTINGS_CONTENT` (and the Profile content is no longer visible).++### 25.2 Switch to the Billing tab+Click the Billing tab.++**Verify**: Page content now includes `TAB_BILLING_CONTENT`.++---+diff --git a/tests/benchmark/benchmark-run/group-26.md b/tests/benchmark/benchmark-run/group-26.mdnew file mode 100644index 00000000..df0f37d3--- /dev/null+++ b/tests/benchmark/benchmark-run/group-26.md@@ -0,0 +1,14 @@+# Group 26: Accordion++### 26.1 Open section A+Navigate to `http://fixtures/accordion.html`. Click the header for Section A to expand it.++**Verify**: Page text includes `ACCORDION_SECTION_A_OPEN`.++### 26.2 Open section B+Click the header for Section B. Because the accordion is exclusive-expand, Section A should close.++**Verify**: Page text includes `ACCORDION_SECTION_B_OPEN`, and Section A's `aria-expanded` attribute is now `"false"` (use `eval` to check the attribute if needed).++---+diff --git a/tests/benchmark/benchmark-run/group-27.md b/tests/benchmark/benchmark-run/group-27.mdnew file mode 100644index 00000000..5b974508--- /dev/null+++ b/tests/benchmark/benchmark-run/group-27.md@@ -0,0 +1,14 @@+# Group 27: Contenteditable editor++### 27.1 Type into the rich-text editor+Navigate to `http://fixtures/editor.html`. Type the text `Hello rich text` into the `#editor` div. `contenteditable` elements don't have `.value`, so use `type` (keyboard events), not `fill`.++**Verify**: Page text contains `EDITOR_CHARS=15` and the mirror shows `Hello rich text`.++### 27.2 Commit by pressing Enter+Press Enter (the editor intercepts Enter to commit the current buffer to a separate marker).++**Verify**: Page text contains `EDITOR_COMMITTED=Hello rich text`.++---+diff --git a/tests/benchmark/benchmark-run/group-28.md b/tests/benchmark/benchmark-run/group-28.mdnew file mode 100644index 00000000..5f352fae--- /dev/null+++ b/tests/benchmark/benchmark-run/group-28.md@@ -0,0 +1,14 @@+# Group 28: Range slider++### 28.1 Move slider into HIGH bucket+Navigate to `http://fixtures/range.html`. Set the `#volume` range slider to 90.++**Verify**: Page text contains both `RANGE_VALUE_90` and `BUCKET_HIGH`.++### 28.2 Move slider into LOW bucket+Without reloading, change the slider to 10.++**Verify**: Page text now contains `RANGE_VALUE_10` and `BUCKET_LOW`.++---+diff --git a/tests/benchmark/benchmark-run/group-29.md b/tests/benchmark/benchmark-run/group-29.mdnew file mode 100644index 00000000..ed935bca--- /dev/null+++ b/tests/benchmark/benchmark-run/group-29.md@@ -0,0 +1,14 @@+# Group 29: Pagination++### 29.1 Advance to page 2+Navigate to `http://fixtures/pagination.html`. Click the Next button.++**Verify**: Page text contains `PAGE_2_FIRST_ITEM` and `PAGE_2_OF_3`.++### 29.2 Reach the last page; Next disabled+Click Next once more.++**Verify**: Page text contains `PAGE_3_FIRST_ITEM` and `PAGE_3_OF_3`, and the Next button's `disabled` property is `true` (use `eval` to check).++---+diff --git a/tests/benchmark/benchmark-run/group-30.md b/tests/benchmark/benchmark-run/group-30.mdnew file mode 100644index 00000000..69c78d4a--- /dev/null+++ b/tests/benchmark/benchmark-run/group-30.md@@ -0,0 +1,14 @@+# Group 30: Custom dropdown menu++### 30.1 Pick the "Beta" item+Navigate to `http://fixtures/dropdown.html`. The page has a custom dropdown — click the toggle button (`#dropdown-toggle`) to open the menu, then click the "Beta" item.++**Verify**: Page text contains `DROPDOWN_SELECTED=BETA`.++### 30.2 Reopen and pick "Gamma"+Reopen the dropdown and select "Gamma". (Each selection auto-closes the menu, so you need to click the toggle again before selecting.)++**Verify**: Page text contains `DROPDOWN_SELECTED=GAMMA`.++---+diff --git a/tests/benchmark/benchmark-run/group-31.md b/tests/benchmark/benchmark-run/group-31.mdnew file mode 100644index 00000000..030c91cb--- /dev/null+++ b/tests/benchmark/benchmark-run/group-31.md@@ -0,0 +1,9 @@+# Group 31: Nested iframes (3 levels deep)++### 31.1 Click a button in the deepest frame+Navigate to `http://fixtures/iframe-nested.html`. The outer page embeds `#level-2`, which in turn embeds `#level-3`, which contains `#deep-button`. Drill through the two frame hops and click the button. Verify via a scoped `snap` before resetting to `main`.++**Verify**: Scoped snapshot contains `DEEP_CLICKED=YES_LEVEL_3`.++---+diff --git a/tests/benchmark/benchmark-run/group-32.md b/tests/benchmark/benchmark-run/group-32.mdnew file mode 100644index 00000000..d46bbb7e--- /dev/null+++ b/tests/benchmark/benchmark-run/group-32.md@@ -0,0 +1,9 @@+# Group 32: Dynamic iframe (inserted after load)++### 32.1 Wait for a late iframe, then interact with it+Navigate to `http://fixtures/iframe-dynamic.html`. The iframe is inserted ~1.2 s after load (use `wait --text "IFRAME_DYNAMIC_ATTACHED"`). Then scope into `#late-frame`, fill `#iframe-input` with "Late World", click `#iframe-submit`, and verify the inner result marker.++**Verify**: Scoped snapshot contains `IFRAME_INPUT_RECEIVED_LATE_WORLD`.++---+diff --git a/tests/benchmark/benchmark-run/group-33.md b/tests/benchmark/benchmark-run/group-33.mdnew file mode 100644index 00000000..abe79613--- /dev/null+++ b/tests/benchmark/benchmark-run/group-33.md@@ -0,0 +1,9 @@+# Group 33: srcdoc iframe++### 33.1 Interact with an inline-content iframe+Navigate to `http://fixtures/iframe-srcdoc.html`. The iframe's content is inlined via `srcdoc`; it has no `src` URL. Scope into `#srcdoc-frame`, fill `#inline-input` with `srcdoc`, click `#inline-submit`, and verify.++**Verify**: Scoped snapshot contains `INLINE_RECEIVED_SRCDOC`.++---+diff --git a/tests/benchmark/benchmark-run/group-34.md b/tests/benchmark/benchmark-run/group-34.mdnew file mode 100644index 00000000..27adab4f--- /dev/null+++ b/tests/benchmark/benchmark-run/group-34.md@@ -0,0 +1,9 @@+# Group 34: Sandboxed iframe++### 34.1 Click inside a sandboxed iframe+Navigate to `http://fixtures/iframe-sandbox.html`. The iframe has `sandbox="allow-scripts allow-same-origin"`. Scope into `#sandboxed`, click `#sandbox-button`, and verify.++**Verify**: Scoped snapshot contains `SANDBOX_CLICKED=YES`.++---+diff --git a/tests/benchmark/benchmark-run/group-35.md b/tests/benchmark/benchmark-run/group-35.mdnew file mode 100644index 00000000..eb313fec--- /dev/null+++ b/tests/benchmark/benchmark-run/group-35.md@@ -0,0 +1,14 @@+# Group 35: Long-form article (Medium/Substack style)++### 35.1 Read the article with default `text`+Navigate to `http://fixtures/article.html`. Extract the page's main content using the default `text` (Readability) mode. Default mode is the right choice for article-style pages.++**Verify**: The extracted text contains both `ARTICLE_PUBLISHED_2026_04_15` and `ARTICLE_WORD_COUNT_MARKER_323` (both inside the article body — Readability keeps them).++### 35.2 See the chrome that Readability drops+Re-extract with `text --full` and confirm the footer is included this time.++**Verify**: `--full` output contains `FOOTER_COPYRIGHT_MARKER` (which the default mode trims).++---+diff --git a/tests/benchmark/benchmark-run/group-36.md b/tests/benchmark/benchmark-run/group-36.mdnew file mode 100644index 00000000..8cd88cb7--- /dev/null+++ b/tests/benchmark/benchmark-run/group-36.md@@ -0,0 +1,14 @@+# Group 36: Search results page (SERP)++### 36.1 Find a specific result by id+Navigate to `http://fixtures/serp.html`. The page has 6 result cards (`#r-1`..`#r-6`). Extract just the third card's content. A scoped snapshot (`snap --selector "#r-3"`) is the direct path.++**Verify**: The scoped output contains `RESULT_3_TITLE` and `RESULT_3_SNIPPET_MARKER`.++### 36.2 Count all result cards+Use a full text extraction to verify all six results are present in one pass.++**Verify**: Output contains all of `RESULT_1_TITLE` through `RESULT_6_TITLE` and the summary `SERP_RESULT_COUNT_6`. Default Readability trims SERPs — use `text --full`.++---+diff --git a/tests/benchmark/benchmark-run/group-37.md b/tests/benchmark/benchmark-run/group-37.mdnew file mode 100644index 00000000..67d2b2d3--- /dev/null+++ b/tests/benchmark/benchmark-run/group-37.md@@ -0,0 +1,14 @@+# Group 37: Q&A thread (Stack-Overflow style)++### 37.1 Find the accepted answer id+Navigate to `http://fixtures/qa.html`. The accepted answer carries `data-accepted="true"` on its `<div>` wrapper. Use `eval` to return that element's `id`.++**Verify**: The `eval` result equals `"a-2"`.++### 37.2 Extract the accepted answer's body+Scope a snapshot to `#a-2` and verify the body content.++**Verify**: Scoped snapshot contains `ANSWER_2_BODY_MARKER` and `ACCEPTED_ANSWER_ID_A2`.++---+diff --git a/tests/benchmark/benchmark-run/group-38.md b/tests/benchmark/benchmark-run/group-38.mdnew file mode 100644index 00000000..81917733--- /dev/null+++ b/tests/benchmark/benchmark-run/group-38.md@@ -0,0 +1,11 @@+# Group 38: Pricing table++### 38.1 Read just the Pro plan+Navigate to `http://fixtures/pricing.html`. The page has three plan cards (`#plan-free`, `#plan-pro`, `#plan-enterprise`). Read only the Pro plan using a scoped snapshot.++**Verify**: Scoped snapshot contains `PLAN_PRO_PRICE_29` and `PLAN_PRO_LIMIT_5000_requests per day`.++### 38.2 Compare all three plans+Extract the full page (pricing grids are a Readability anti-pattern — reach for `--full`).++**Verify**: Output contains `PLAN_FREE_PRICE_0`, `PLAN_PRO_PRICE_29`, and `PLAN_ENTERPRISE_PRICE_CUSTOM`.diff --git a/tests/benchmark/benchmark-run/index.md b/tests/benchmark/benchmark-run/index.mdnew file mode 100644index 00000000..27f90db6--- /dev/null+++ b/tests/benchmark/benchmark-run/index.md@@ -0,0 +1,55 @@+# Benchmark Run++Natural-language benchmark task set for the agent-driven lanes.++Read this index first, then read the relevant group files.++## How to use this directory++- `setup-pinchtab.md` and `setup-agent-browser.md` define lane-specific setup, wrapper usage, recording, and Group 0 behavior.+- This directory defines the shared task groups.+- Each group lives in its own file: `group-00.md` through `group-38.md`.+- When running a subset, read only the selected group files.+- When running the full suite, proceed in numeric order.++## Group Files++- `group-00.md` — Setup & Diagnosis+- `group-01.md` — Reading & Extracting Real Content+- `group-02.md` — Search & Dynamic Interaction+- `group-03.md` — Complex Form+- `group-04.md` — SPA State Management+- `group-05.md` — Login Flow+- `group-06.md` — Multi-Step E-commerce+- `group-07.md` — Content + Interaction Combined+- `group-08.md` — Error Handling+- `group-09.md` — Export+- `group-10.md` — Nested Interactions & Modal Dialogs+- `group-11.md` — State Persistence & Page Reload+- `group-12.md` — Multi-Page Navigation & Back Button+- `group-13.md` — Form State & Multi-Step Submission+- `group-14.md` — Dynamic Content Loading+- `group-15.md` — Complex Data Extraction & Aggregation+- `group-16.md` — Hover & Tooltips+- `group-17.md` — Scrolling+- `group-18.md` — File Download+- `group-19.md` — iFrame+- `group-20.md` — Dialogs+- `group-21.md` — Async / awaitPromise+- `group-22.md` — Mouse Drag & Drop+- `group-23.md` — Async / Loading state+- `group-24.md` — Keyboard events+- `group-25.md` — Tab panels+- `group-26.md` — Accordion+- `group-27.md` — Contenteditable editor+- `group-28.md` — Range slider+- `group-29.md` — Pagination+- `group-30.md` — Custom dropdown menu+- `group-31.md` — Nested iframes (3 levels deep)+- `group-32.md` — Dynamic iframe (inserted after load)+- `group-33.md` — srcdoc iframe+- `group-34.md` — Sandboxed iframe+- `group-35.md` — Long-form article (Medium/Substack style)+- `group-36.md` — Search results page (SERP)+- `group-37.md` — Q&A thread (Stack-Overflow style)+- `group-38.md` — Pricing tablediff --git a/tests/benchmark/cmd/api-runner/anthropic.go b/tests/benchmark/cmd/api-runner/anthropic.gonew file mode 100644index 00000000..952c8219--- /dev/null+++ b/tests/benchmark/cmd/api-runner/anthropic.go@@ -0,0 +1,219 @@+package main++import (+ "bytes"+ "context"+ "encoding/json"+ "fmt"+ "io"+ "net/http"+ "time"+)++const AnthropicAPIURL = "https://api.anthropic.com/v1/messages"++type AnthropicRunner struct {+ apiKey string+ model string+ maxTokens int+ temperature float64+ promptCaching bool+ usage UsageCounters+ client *http.Client+ retryCfg RetryConfig+}++func NewAnthropicRunner(apiKey, model string, maxTokens int, temperature float64, promptCaching bool) *AnthropicRunner {+ return &AnthropicRunner{+ apiKey: apiKey,+ model: model,+ maxTokens: maxTokens,+ temperature: temperature,+ promptCaching: promptCaching,+ client: &http.Client{Timeout: 5 * time.Minute},+ retryCfg: DefaultRetryConfig(),+ }+}++func (r *AnthropicRunner) Provider() string { return "anthropic" }+func (r *AnthropicRunner) Source() string { return "anthropic-messages" }+func (r *AnthropicRunner) Model() string { return r.model }+func (r *AnthropicRunner) Usage() UsageCounters { return r.usage }++func (r *AnthropicRunner) ToolDefinitions() interface{} {+ return []map[string]interface{}{+ {+ "name": "run_command",+ "description": "Run a shell command in a persistent bash session rooted at tests/benchmark. Environment changes such as cd/export persist across calls.",+ "input_schema": map[string]interface{}{+ "type": "object",+ "properties": map[string]interface{}{+ "command": map[string]interface{}{"type": "string"},+ "timeout_seconds": map[string]interface{}{"type": "integer", "minimum": 1, "maximum": 600},+ },+ "required": []string{"command"},+ },+ },+ }+}++func (r *AnthropicRunner) InitialConversation(userPrompt string) []interface{} {+ return []interface{}{+ map[string]interface{}{+ "role": "user",+ "content": userPrompt,+ },+ }+}++func (r *AnthropicRunner) Send(systemPrompt string, conversation []interface{}) (interface{}, error) {+ body := map[string]interface{}{+ "model": r.model,+ "max_tokens": r.maxTokens,+ "temperature": r.temperature,+ "system": systemPrompt,+ "tools": r.ToolDefinitions(),+ "messages": conversation,+ }+ if r.promptCaching {+ body["cache_control"] = map[string]string{"type": "ephemeral"}+ }++ payload, err := json.Marshal(body)+ if err != nil {+ return nil, fmt.Errorf("marshal request: %w", err)+ }++ var result map[string]interface{}+ resp, err := DoWithRetry(context.Background(), r.retryCfg, func() (*http.Response, error) {+ req, err := http.NewRequest("POST", AnthropicAPIURL, bytes.NewReader(payload))+ if err != nil {+ return nil, err+ }+ req.Header.Set("Content-Type", "application/json")+ req.Header.Set("X-Api-Key", r.apiKey)+ req.Header.Set("Anthropic-Version", "2023-06-01")+ return r.client.Do(req)+ })+ if err != nil {+ return nil, err+ }+ defer resp.Body.Close()++ respBody, err := io.ReadAll(resp.Body)+ if err != nil {+ return nil, fmt.Errorf("read response: %w", err)+ }++ if err := json.Unmarshal(respBody, &result); err != nil {+ return nil, fmt.Errorf("unmarshal response: %w", err)+ }++ if resp.StatusCode >= 400 {+ return nil, fmt.Errorf("Anthropic API error %d: %s", resp.StatusCode, string(respBody))+ }++ r.updateUsage(result)+ return result, nil+}++func (r *AnthropicRunner) updateUsage(result map[string]interface{}) {+ usage, _ := result["usage"].(map[string]interface{})+ if usage == nil {+ return+ }+ r.usage.RequestCount+++ r.usage.InputTokens += toInt(usage["input_tokens"])+ r.usage.OutputTokens += toInt(usage["output_tokens"])+ r.usage.CacheCreationInputTokens += toInt(usage["cache_creation_input_tokens"])+ r.usage.CacheReadInputTokens += toInt(usage["cache_read_input_tokens"])+}++func toInt(v interface{}) int {+ switch n := v.(type) {+ case float64:+ return int(n)+ case int:+ return n+ default:+ return 0+ }+}++func (r *AnthropicRunner) ExtractToolCalls(response interface{}, defaultTimeout time.Duration) []ToolCall {+ resp, _ := response.(map[string]interface{})+ content, _ := resp["content"].([]interface{})+ var calls []ToolCall+ for _, item := range content {+ m, _ := item.(map[string]interface{})+ if m["type"] != "tool_use" {+ continue+ }+ input, _ := m["input"].(map[string]interface{})+ cmd, _ := input["command"].(string)+ if cmd == "" {+ continue+ }+ timeout := int(defaultTimeout.Seconds())+ if t := toInt(input["timeout_seconds"]); t > 0 {+ timeout = t+ }+ calls = append(calls, ToolCall{+ ID: fmt.Sprintf("%v", m["id"]),+ Command: cmd,+ TimeoutSeconds: timeout,+ })+ }+ return calls+}++func (r *AnthropicRunner) AppendToolResults(conversation []interface{}, response interface{}, results []ToolExecutionResult) []interface{} {+ resp, _ := response.(map[string]interface{})+ conversation = append(conversation, map[string]interface{}{+ "role": "assistant",+ "content": resp["content"],+ })+ if len(results) > 0 {+ toolResults := make([]interface{}, len(results))+ for i, res := range results {+ toolResults[i] = map[string]interface{}{+ "type": "tool_result",+ "tool_use_id": res.ID,+ "is_error": res.IsError,+ "content": res.Content,+ }+ }+ conversation = append(conversation, map[string]interface{}{+ "role": "user",+ "content": toolResults,+ })+ }+ return conversation+}++func (r *AnthropicRunner) ExtractFinalText(response interface{}) string {+ resp, _ := response.(map[string]interface{})+ content, _ := resp["content"].([]interface{})+ var texts []string+ for _, item := range content {+ m, _ := item.(map[string]interface{})+ if m["type"] != "text" {+ continue+ }+ if text, ok := m["text"].(string); ok && text != "" {+ texts = append(texts, text)+ }+ }+ return joinStrings(texts, "\n")+}++func joinStrings(strs []string, sep string) string {+ if len(strs) == 0 {+ return ""+ }+ result := strs[0]+ for _, s := range strs[1:] {+ result += sep + s+ }+ return result+}diff --git a/tests/benchmark/cmd/api-runner/anthropic_test.go b/tests/benchmark/cmd/api-runner/anthropic_test.gonew file mode 100644index 00000000..219185ba--- /dev/null+++ b/tests/benchmark/cmd/api-runner/anthropic_test.go@@ -0,0 +1,185 @@+package main++import (+ "encoding/json"+ "io"+ "net/http"+ "net/http/httptest"+ "testing"+ "time"+)++func TestAnthropicToolDefinitions(t *testing.T) {+ r := NewAnthropicRunner("key", "claude-haiku-4-5-20251001", 4096, 0, true)+ tools := r.ToolDefinitions().([]map[string]interface{})+ if len(tools) != 1 {+ t.Fatalf("got %d tools; want 1", len(tools))+ }+ if tools[0]["name"] != "run_command" {+ t.Errorf("tool name = %v; want 'run_command'", tools[0]["name"])+ }+}++func TestAnthropicInitialConversation(t *testing.T) {+ r := NewAnthropicRunner("key", "claude-haiku-4-5-20251001", 4096, 0, true)+ conv := r.InitialConversation("test prompt")+ if len(conv) != 1 {+ t.Fatalf("got %d messages; want 1", len(conv))+ }+ msg := conv[0].(map[string]interface{})+ if msg["role"] != "user" {+ t.Errorf("role = %v; want 'user'", msg["role"])+ }+ if msg["content"] != "test prompt" {+ t.Errorf("content = %v; want 'test prompt'", msg["content"])+ }+}++func TestAnthropicSend(t *testing.T) {+ var receivedBody map[string]interface{}+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ if r.Method != "POST" {+ t.Errorf("method = %s; want POST", r.Method)+ }+ if r.Header.Get("X-Api-Key") != "test-key" {+ t.Errorf("X-Api-Key = %s; want test-key", r.Header.Get("X-Api-Key"))+ }+ if r.Header.Get("Anthropic-Version") != "2023-06-01" {+ t.Errorf("Anthropic-Version = %s; want 2023-06-01", r.Header.Get("Anthropic-Version"))+ }++ body, _ := io.ReadAll(r.Body)+ json.Unmarshal(body, &receivedBody)++ w.Header().Set("Content-Type", "application/json")+ json.NewEncoder(w).Encode(map[string]interface{}{+ "content": []interface{}{+ map[string]interface{}{"type": "text", "text": "Hello"},+ },+ "usage": map[string]interface{}{+ "input_tokens": 100,+ "output_tokens": 50,+ },+ })+ }))+ defer server.Close()++ r := NewAnthropicRunner("test-key", "claude-haiku-4-5-20251001", 4096, 0.5, true)+ r.retryCfg.Sleep = func(d time.Duration) {}++ originalURL := AnthropicAPIURL+ defer func() { _ = originalURL }()++ conv := r.InitialConversation("hello")+ r.client = &http.Client{Transport: &testTransport{server.URL}}++ _, err := r.Send("system", conv)+ if err != nil {+ t.Fatal(err)+ }++ if receivedBody["model"] != "claude-haiku-4-5-20251001" {+ t.Errorf("model = %v; want claude-haiku-4-5-20251001", receivedBody["model"])+ }+ if receivedBody["max_tokens"] != float64(4096) {+ t.Errorf("max_tokens = %v; want 4096", receivedBody["max_tokens"])+ }+ if _, ok := receivedBody["cache_control"]; !ok {+ t.Error("missing cache_control")+ }++ usage := r.Usage()+ if usage.RequestCount != 1 {+ t.Errorf("RequestCount = %d; want 1", usage.RequestCount)+ }+ if usage.InputTokens != 100 {+ t.Errorf("InputTokens = %d; want 100", usage.InputTokens)+ }+}++type testTransport struct {+ targetURL string+}++func (t *testTransport) RoundTrip(req *http.Request) (*http.Response, error) {+ newReq, _ := http.NewRequest(req.Method, t.targetURL, req.Body)+ for k, v := range req.Header {+ newReq.Header[k] = v+ }+ return http.DefaultTransport.RoundTrip(newReq)+}++func TestAnthropicExtractToolCalls(t *testing.T) {+ r := NewAnthropicRunner("key", "model", 4096, 0, true)+ response := map[string]interface{}{+ "content": []interface{}{+ map[string]interface{}{+ "type": "tool_use",+ "id": "toolu_123",+ "input": map[string]interface{}{"command": "ls -la", "timeout_seconds": 30.0},+ },+ map[string]interface{}{+ "type": "text",+ "text": "Let me list files",+ },+ },+ }++ calls := r.ExtractToolCalls(response, 120*time.Second)+ if len(calls) != 1 {+ t.Fatalf("got %d calls; want 1", len(calls))+ }+ if calls[0].ID != "toolu_123" {+ t.Errorf("ID = %s; want toolu_123", calls[0].ID)+ }+ if calls[0].Command != "ls -la" {+ t.Errorf("Command = %s; want 'ls -la'", calls[0].Command)+ }+ if calls[0].TimeoutSeconds != 30 {+ t.Errorf("TimeoutSeconds = %d; want 30", calls[0].TimeoutSeconds)+ }+}++func TestAnthropicExtractFinalText(t *testing.T) {+ r := NewAnthropicRunner("key", "model", 4096, 0, true)+ response := map[string]interface{}{+ "content": []interface{}{+ map[string]interface{}{"type": "text", "text": "Hello"},+ map[string]interface{}{"type": "tool_use", "id": "123"},+ map[string]interface{}{"type": "text", "text": "World"},+ },+ }++ text := r.ExtractFinalText(response)+ if text != "Hello\nWorld" {+ t.Errorf("got %q; want 'Hello\\nWorld'", text)+ }+}++func TestAnthropicAppendToolResults(t *testing.T) {+ r := NewAnthropicRunner("key", "model", 4096, 0, true)+ conv := []interface{}{}+ response := map[string]interface{}{+ "content": []interface{}{+ map[string]interface{}{"type": "tool_use", "id": "123"},+ },+ }+ results := []ToolExecutionResult{+ {ID: "123", IsError: false, Content: "output"},+ }++ conv = r.AppendToolResults(conv, response, results)+ if len(conv) != 2 {+ t.Fatalf("got %d messages; want 2", len(conv))+ }++ assistant := conv[0].(map[string]interface{})+ if assistant["role"] != "assistant" {+ t.Errorf("first message role = %v; want 'assistant'", assistant["role"])+ }++ user := conv[1].(map[string]interface{})+ if user["role"] != "user" {+ t.Errorf("second message role = %v; want 'user'", user["role"])+ }+}diff --git a/tests/benchmark/cmd/api-runner/args.go b/tests/benchmark/cmd/api-runner/args.gonew file mode 100644index 00000000..fee83b64--- /dev/null+++ b/tests/benchmark/cmd/api-runner/args.go@@ -0,0 +1,298 @@+// Package main implements the Go port of tests/benchmark/scripts/run-api-benchmark.ts.+//+// This file defines the command-line interface. The flag surface mirrors the+// TypeScript runner so `./dev benchmark --runner go ...` can be swapped in+// without callers changing their invocation. Two new flags, --dry-run and+// --index-file, are added here so later steps can exercise the plan/prompt+// assembly without needing network access or the default index.md location.+package main++import (+ "errors"+ "fmt"+ "io"+ "sort"+ "strconv"+ "strings"+)++// Lane is the benchmark lane (pinchtab or agent-browser).+type Lane string++const (+ LanePinchtab Lane = "pinchtab"+ LaneAgentBrowser Lane = "agent-browser"+)++// Provider identifies which API the runner talks to.+type Provider string++const (+ ProviderAnthropic Provider = "anthropic"+ ProviderOpenAI Provider = "openai"+ ProviderFake Provider = "fake"+ ProviderUnset Provider = ""+)++// Args is the resolved plan for a single invocation. It is intentionally a+// plain-data struct so tests can construct one without touching os.Args.+type Args struct {+ Lane Lane+ Provider Provider+ Model string+ Groups []int+ Profile string+ MaxTokens int+ Temperature float64+ MaxTurns int+ MaxIdleTurns int+ TimeoutSeconds int+ TurnDelayMs int+ ReportFile string+ SkipInit bool+ NoPromptCaching bool+ Finalize bool+ DryRun bool+ IndexFile string+ MaxInputTokens int+ MaxOutputTokens int+}++// Defaults matches the literal defaults in the TypeScript runner. Keep this in+// sync with parseArgs() in run-api-benchmark.ts.+func defaultArgs() Args {+ return Args{+ MaxTokens: 4096,+ Temperature: 0,+ MaxTurns: 300,+ MaxIdleTurns: 25,+ TimeoutSeconds: 120,+ TurnDelayMs: 1500,+ }+}++const usageText = `Usage:+ go run ./tests/benchmark/cmd/api-runner --lane pinchtab [options]+ go run ./tests/benchmark/cmd/api-runner --lane agent-browser [options]++Options:+ --provider anthropic|openai|fake+ --model MODEL+ --groups 0,1,2,3+ --profile common10+ --max-tokens N+ --temperature N+ --max-turns N+ --max-idle-turns N+ --timeout-seconds N+ --turn-delay-ms N+ --report-file PATH+ --skip-init+ --no-prompt-caching+ --finalize+ --dry-run Print the resolved plan and exit 0 without network access+ --index-file PATH Override path to tests/benchmark/benchmark-run/index.md+ --max-input-tokens N Stop when cumulative input tokens exceed N (exit code 4)+ --max-output-tokens N Stop when cumulative output tokens exceed N (exit code 4)+`++// ParseArgs walks argv manually (like the TS runner) so the flag surface and+// error messages stay byte-identical. The stdlib `flag` package would reorder+// output and reject `--groups 0,1,2` style values in some edge cases.+func ParseArgs(argv []string) (Args, error) {+ a := defaultArgs()++ next := func(i *int, name string) (string, error) {+ *i+++ if *i >= len(argv) {+ return "", fmt.Errorf("%s requires a value", name)+ }+ return argv[*i], nil+ }++ for i := 0; i < len(argv); i++ {+ arg := argv[i]+ switch arg {+ case "--lane":+ v, err := next(&i, arg)+ if err != nil {+ return a, err+ }+ a.Lane = Lane(v)+ case "--provider":+ v, err := next(&i, arg)+ if err != nil {+ return a, err+ }+ a.Provider = Provider(v)+ case "--model":+ v, err := next(&i, arg)+ if err != nil {+ return a, err+ }+ a.Model = v+ case "--groups":+ v, err := next(&i, arg)+ if err != nil {+ return a, err+ }+ groups, perr := parseGroups(v)+ if perr != nil {+ return a, perr+ }+ a.Groups = groups+ case "--profile":+ v, err := next(&i, arg)+ if err != nil {+ return a, err+ }+ a.Profile = v+ case "--max-tokens":+ v, err := next(&i, arg)+ if err != nil {+ return a, err+ }+ n, perr := strconv.Atoi(v)+ if perr != nil {+ return a, fmt.Errorf("--max-tokens: %w", perr)+ }+ a.MaxTokens = n+ case "--temperature":+ v, err := next(&i, arg)+ if err != nil {+ return a, err+ }+ f, perr := strconv.ParseFloat(v, 64)+ if perr != nil {+ return a, fmt.Errorf("--temperature: %w", perr)+ }+ a.Temperature = f+ case "--max-turns":+ v, err := next(&i, arg)+ if err != nil {+ return a, err+ }+ n, perr := strconv.Atoi(v)+ if perr != nil {+ return a, fmt.Errorf("--max-turns: %w", perr)+ }+ a.MaxTurns = n+ case "--max-idle-turns":+ v, err := next(&i, arg)+ if err != nil {+ return a, err+ }+ n, perr := strconv.Atoi(v)+ if perr != nil {+ return a, fmt.Errorf("--max-idle-turns: %w", perr)+ }+ a.MaxIdleTurns = n+ case "--timeout-seconds":+ v, err := next(&i, arg)+ if err != nil {+ return a, err+ }+ n, perr := strconv.Atoi(v)+ if perr != nil {+ return a, fmt.Errorf("--timeout-seconds: %w", perr)+ }+ a.TimeoutSeconds = n+ case "--turn-delay-ms":+ v, err := next(&i, arg)+ if err != nil {+ return a, err+ }+ n, perr := strconv.Atoi(v)+ if perr != nil {+ return a, fmt.Errorf("--turn-delay-ms: %w", perr)+ }+ a.TurnDelayMs = n+ case "--report-file":+ v, err := next(&i, arg)+ if err != nil {+ return a, err+ }+ a.ReportFile = v+ case "--skip-init":+ a.SkipInit = true+ case "--no-prompt-caching":+ a.NoPromptCaching = true+ case "--finalize":+ a.Finalize = true+ case "--dry-run":+ a.DryRun = true+ case "--index-file":+ v, err := next(&i, arg)+ if err != nil {+ return a, err+ }+ a.IndexFile = v+ case "--max-input-tokens":+ v, err := next(&i, arg)+ if err != nil {+ return a, err+ }+ n, perr := strconv.Atoi(v)+ if perr != nil {+ return a, fmt.Errorf("--max-input-tokens: %w", perr)+ }+ a.MaxInputTokens = n+ case "--max-output-tokens":+ v, err := next(&i, arg)+ if err != nil {+ return a, err+ }+ n, perr := strconv.Atoi(v)+ if perr != nil {+ return a, fmt.Errorf("--max-output-tokens: %w", perr)+ }+ a.MaxOutputTokens = n+ case "-h", "--help", "help":+ return a, errHelp+ default:+ return a, fmt.Errorf("unknown argument: %s", arg)+ }+ }++ if a.Lane != LanePinchtab && a.Lane != LaneAgentBrowser {+ return a, errors.New("--lane must be 'pinchtab' or 'agent-browser'")+ }++ return a, nil+}++// errHelp is a sentinel the caller uses to distinguish "user asked for help"+// (exit 0) from "bad flags" (exit 1). It matches the TS runner's usage(0) vs+// usage(1) behaviour.+var errHelp = errors.New("help requested")++// parseGroups splits "0,1,2,3" into sorted unique ints, matching the TS+// implementation's `.map(Number).filter(Number.isInteger)` semantics.+func parseGroups(raw string) ([]int, error) {+ seen := make(map[int]struct{})+ var out []int+ for _, piece := range strings.Split(raw, ",") {+ piece = strings.TrimSpace(piece)+ if piece == "" {+ continue+ }+ n, err := strconv.Atoi(piece)+ if err != nil {+ // TS filters silently; we do too, to preserve parity.+ continue+ }+ if _, dup := seen[n]; dup {+ continue+ }+ seen[n] = struct{}{}+ out = append(out, n)+ }+ sort.Ints(out)+ return out, nil+}++// WriteUsage prints the help block to the given writer. Kept as a helper so+// tests can capture output without redirecting os.Stderr.+func WriteUsage(w io.Writer) {+ fmt.Fprint(w, usageText)+}diff --git a/tests/benchmark/cmd/api-runner/compact.go b/tests/benchmark/cmd/api-runner/compact.gonew file mode 100644index 00000000..e787cd6b--- /dev/null+++ b/tests/benchmark/cmd/api-runner/compact.go@@ -0,0 +1,59 @@+package main++const (+ CompactAfterAnthropicMessages = 10+ KeepRecentAnthropicMessages = 6+ CompactAfterOpenAIItems = 16+ KeepRecentOpenAIItems = 10+)++func CompactAnthropicConversation(conversation []interface{}, summary string) []interface{} {+ if len(conversation) <= CompactAfterAnthropicMessages {+ return conversation+ }+ head := conversation[0]+ recentStart := len(conversation) - KeepRecentAnthropicMessages+ if recentStart < 1 {+ recentStart = 1+ }+ recent := conversation[recentStart:]++ result := make([]interface{}, 0, 2+len(recent))+ result = append(result, head)+ result = append(result, map[string]interface{}{+ "role": "user",+ "content": summary,+ })+ result = append(result, recent...)+ return result+}++func CompactOpenAIConversation(conversation []interface{}, summary string) []interface{} {+ if len(conversation) <= CompactAfterOpenAIItems {+ return conversation+ }+ head := conversation[0]+ recentStart := len(conversation) - KeepRecentOpenAIItems+ if recentStart < 1 {+ recentStart = 1+ }+ recent := conversation[recentStart:]++ result := make([]interface{}, 0, 2+len(recent))+ result = append(result, head)+ result = append(result, map[string]interface{}{+ "role": "user",+ "content": []interface{}{+ map[string]interface{}{"type": "input_text", "text": summary},+ },+ })+ result = append(result, recent...)+ return result+}++func CompactConversation(provider string, conversation []interface{}, summary string) []interface{} {+ if provider == "anthropic" {+ return CompactAnthropicConversation(conversation, summary)+ }+ return CompactOpenAIConversation(conversation, summary)+}diff --git a/tests/benchmark/cmd/api-runner/compact_test.go b/tests/benchmark/cmd/api-runner/compact_test.gonew file mode 100644index 00000000..cd2c56f9--- /dev/null+++ b/tests/benchmark/cmd/api-runner/compact_test.go@@ -0,0 +1,88 @@+package main++import (+ "testing"+)++func TestCompactAnthropicNoOp(t *testing.T) {+ conv := make([]interface{}, 8)+ for i := range conv {+ conv[i] = map[string]interface{}{"role": "user", "content": "msg"}+ }+ result := CompactAnthropicConversation(conv, "summary")+ if len(result) != 8 {+ t.Errorf("got %d; want 8 (no compaction)", len(result))+ }+}++func TestCompactAnthropicTriggered(t *testing.T) {+ conv := make([]interface{}, 12)+ conv[0] = map[string]interface{}{"role": "user", "content": "initial"}+ for i := 1; i < 12; i++ {+ conv[i] = map[string]interface{}{"role": "assistant", "content": i}+ }++ result := CompactAnthropicConversation(conv, "summary text")++ if len(result) != 8 {+ t.Errorf("got %d; want 8 (head + summary + 6 recent)", len(result))+ }++ summary := result[1].(map[string]interface{})+ if summary["content"] != "summary text" {+ t.Errorf("summary content = %v", summary["content"])+ }+}++func TestCompactOpenAINoOp(t *testing.T) {+ conv := make([]interface{}, 14)+ for i := range conv {+ conv[i] = map[string]interface{}{"type": "message"}+ }+ result := CompactOpenAIConversation(conv, "summary")+ if len(result) != 14 {+ t.Errorf("got %d; want 14 (no compaction)", len(result))+ }+}++func TestCompactOpenAITriggered(t *testing.T) {+ conv := make([]interface{}, 20)+ conv[0] = map[string]interface{}{"role": "user", "content": "initial"}+ for i := 1; i < 20; i++ {+ conv[i] = map[string]interface{}{"type": "function_call", "id": i}+ }++ result := CompactOpenAIConversation(conv, "summary text")++ if len(result) != 12 {+ t.Errorf("got %d; want 12 (head + summary + 10 recent)", len(result))+ }++ summary := result[1].(map[string]interface{})+ content := summary["content"].([]interface{})+ if len(content) != 1 {+ t.Fatalf("summary content length = %d; want 1", len(content))+ }+ part := content[0].(map[string]interface{})+ if part["text"] != "summary text" {+ t.Errorf("summary text = %v", part["text"])+ }+}++func TestCompactConversationRouting(t *testing.T) {+ conv := make([]interface{}, 20)+ conv[0] = map[string]interface{}{"role": "user", "content": "initial"}+ for i := 1; i < 20; i++ {+ conv[i] = map[string]interface{}{"role": "assistant"}+ }++ anthropic := CompactConversation("anthropic", conv, "summary")+ if len(anthropic) != 8 {+ t.Errorf("anthropic compaction = %d; want 8", len(anthropic))+ }++ openai := CompactConversation("openai", conv, "summary")+ if len(openai) != 12 {+ t.Errorf("openai compaction = %d; want 12", len(openai))+ }+}diff --git a/tests/benchmark/cmd/api-runner/fake.go b/tests/benchmark/cmd/api-runner/fake.gonew file mode 100644index 00000000..b4ae564d--- /dev/null+++ b/tests/benchmark/cmd/api-runner/fake.go@@ -0,0 +1,165 @@+package main++import (+ "fmt"+ "time"+)++type FakeResponse struct {+ ToolCalls []ToolCall+ FinalText string+}++type FakeRunner struct {+ model string+ responses []FakeResponse+ turnIndex int+ usage UsageCounters+}++func NewFakeRunner(model string, responses []FakeResponse) *FakeRunner {+ return &FakeRunner{+ model: model,+ responses: responses,+ }+}++func (r *FakeRunner) Provider() string { return "fake" }+func (r *FakeRunner) Source() string { return "fake-runner" }+func (r *FakeRunner) Model() string { return r.model }+func (r *FakeRunner) Usage() UsageCounters { return r.usage }++func (r *FakeRunner) ToolDefinitions() interface{} {+ return []map[string]interface{}{+ {+ "name": "run_command",+ "description": "Run a shell command",+ "input_schema": map[string]interface{}{+ "type": "object",+ "properties": map[string]interface{}{+ "command": map[string]interface{}{"type": "string"},+ },+ },+ },+ }+}++func (r *FakeRunner) InitialConversation(userPrompt string) []interface{} {+ return []interface{}{+ map[string]interface{}{+ "role": "user",+ "content": userPrompt,+ },+ }+}++func (r *FakeRunner) Send(systemPrompt string, conversation []interface{}) (interface{}, error) {+ r.usage.RequestCount+++ r.usage.InputTokens += 100+ r.usage.OutputTokens += 50++ if r.turnIndex >= len(r.responses) {+ return map[string]interface{}{+ "final_text": "No more scripted responses",+ }, nil+ }++ resp := r.responses[r.turnIndex]+ r.turnIndex++++ if len(resp.ToolCalls) > 0 {+ content := make([]interface{}, len(resp.ToolCalls))+ for i, tc := range resp.ToolCalls {+ content[i] = map[string]interface{}{+ "type": "tool_use",+ "id": tc.ID,+ "input": map[string]interface{}{+ "command": tc.Command,+ "timeout_seconds": tc.TimeoutSeconds,+ },+ }+ }+ return map[string]interface{}{"content": content}, nil+ }++ return map[string]interface{}{+ "content": []interface{}{+ map[string]interface{}{+ "type": "text",+ "text": resp.FinalText,+ },+ },+ }, nil+}++func (r *FakeRunner) ExtractToolCalls(response interface{}, defaultTimeout time.Duration) []ToolCall {+ resp, _ := response.(map[string]interface{})+ content, _ := resp["content"].([]interface{})+ var calls []ToolCall+ for _, item := range content {+ m, _ := item.(map[string]interface{})+ if m["type"] != "tool_use" {+ continue+ }+ input, _ := m["input"].(map[string]interface{})+ cmd, _ := input["command"].(string)+ if cmd == "" {+ continue+ }+ timeout := int(defaultTimeout.Seconds())+ if t := toInt(input["timeout_seconds"]); t > 0 {+ timeout = t+ }+ calls = append(calls, ToolCall{+ ID: fmt.Sprintf("%v", m["id"]),+ Command: cmd,+ TimeoutSeconds: timeout,+ })+ }+ return calls+}++func (r *FakeRunner) AppendToolResults(conversation []interface{}, response interface{}, results []ToolExecutionResult) []interface{} {+ resp, _ := response.(map[string]interface{})+ conversation = append(conversation, map[string]interface{}{+ "role": "assistant",+ "content": resp["content"],+ })+ if len(results) > 0 {+ toolResults := make([]interface{}, len(results))+ for i, res := range results {+ toolResults[i] = map[string]interface{}{+ "type": "tool_result",+ "tool_use_id": res.ID,+ "is_error": res.IsError,+ "content": res.Content,+ }+ }+ conversation = append(conversation, map[string]interface{}{+ "role": "user",+ "content": toolResults,+ })+ }+ return conversation+}++func (r *FakeRunner) ExtractFinalText(response interface{}) string {+ resp, _ := response.(map[string]interface{})++ if text, ok := resp["final_text"].(string); ok {+ return text+ }++ content, _ := resp["content"].([]interface{})+ var texts []string+ for _, item := range content {+ m, _ := item.(map[string]interface{})+ if m["type"] != "text" {+ continue+ }+ if text, ok := m["text"].(string); ok && text != "" {+ texts = append(texts, text)+ }+ }+ return joinStrings(texts, "\n")+}diff --git a/tests/benchmark/cmd/api-runner/index.go b/tests/benchmark/cmd/api-runner/index.gonew file mode 100644index 00000000..3749f4ee--- /dev/null+++ b/tests/benchmark/cmd/api-runner/index.go@@ -0,0 +1,140 @@+package main++import (+ "bufio"+ "os"+ "path/filepath"+ "regexp"+ "sort"+ "strconv"+ "strings"+)++var (+ groupFileRegex = regexp.MustCompile(`^group-(\d+)\.md$`)+ listItemRegex = regexp.MustCompile(`^-\s+` + "`?" + `group-(\d+)\.md` + "`?")+ htmlCommentStart = regexp.MustCompile(`<!--`)+ htmlCommentEnd = regexp.MustCompile(`-->`)+ commentedListItem = regexp.MustCompile(`<!--\s*-\s+` + "`?" + `group-(\d+)\.md` + "`?")+)++type IndexFile struct {+ Dir string+ Active []int+ Disabled []int+}++func ParseIndexFile(path string) (IndexFile, error) {+ dir := filepath.Dir(path)+ content, err := os.ReadFile(path)+ if err != nil {+ return IndexFile{Dir: dir}, err+ }+ return parseIndexContent(dir, string(content))+}++func parseIndexContent(dir, content string) (IndexFile, error) {+ idx := IndexFile{Dir: dir}+ disabledSet := make(map[int]struct{})+ activeSet := make(map[int]struct{})++ scanner := bufio.NewScanner(strings.NewReader(content))+ inComment := false+ inGroupSection := false++ for scanner.Scan() {+ line := scanner.Text()++ if strings.HasPrefix(strings.TrimSpace(line), "## Group Files") {+ inGroupSection = true+ continue+ }++ if inGroupSection && strings.HasPrefix(strings.TrimSpace(line), "## ") {+ break+ }++ if !inGroupSection {+ continue+ }++ if htmlCommentStart.MatchString(line) && !htmlCommentEnd.MatchString(line) {+ inComment = true+ if m := listItemRegex.FindStringSubmatch(stripCommentStart(line)); m != nil {+ n, _ := strconv.Atoi(m[1])+ disabledSet[n] = struct{}{}+ }+ continue+ }++ if inComment {+ if m := listItemRegex.FindStringSubmatch(line); m != nil {+ n, _ := strconv.Atoi(m[1])+ disabledSet[n] = struct{}{}+ }+ if htmlCommentEnd.MatchString(line) {+ inComment = false+ }+ continue+ }++ if m := commentedListItem.FindStringSubmatch(line); m != nil {+ n, _ := strconv.Atoi(m[1])+ disabledSet[n] = struct{}{}+ continue+ }++ if m := listItemRegex.FindStringSubmatch(line); m != nil {+ n, _ := strconv.Atoi(m[1])+ activeSet[n] = struct{}{}+ }+ }++ for n := range activeSet {+ idx.Active = append(idx.Active, n)+ }+ sort.Ints(idx.Active)++ for n := range disabledSet {+ idx.Disabled = append(idx.Disabled, n)+ }+ sort.Ints(idx.Disabled)++ return idx, nil+}++func stripCommentStart(line string) string {+ idx := strings.Index(line, "<!--")+ if idx >= 0 {+ return line[idx+4:]+ }+ return line+}++func ScanGroupFiles(dir string) ([]int, error) {+ entries, err := os.ReadDir(dir)+ if err != nil {+ return nil, err+ }+ var groups []int+ for _, e := range entries {+ if e.IsDir() {+ continue+ }+ if m := groupFileRegex.FindStringSubmatch(e.Name()); m != nil {+ n, _ := strconv.Atoi(m[1])+ groups = append(groups, n)+ }+ }+ sort.Ints(groups)+ return groups, nil+}++func (idx IndexFile) IsDisabled(group int) bool {+ for _, d := range idx.Disabled {+ if d == group {+ return true+ }+ }+ return false+}diff --git a/tests/benchmark/cmd/api-runner/index_test.go b/tests/benchmark/cmd/api-runner/index_test.gonew file mode 100644index 00000000..a932090e--- /dev/null+++ b/tests/benchmark/cmd/api-runner/index_test.go@@ -0,0 +1,169 @@+package main++import (+ "bytes"+ "os"+ "path/filepath"+ "reflect"+ "testing"+)++func TestParseIndexAllActive(t *testing.T) {+ content := `# Benchmark Run++## Group Files++- group-00.md — Setup+- group-01.md — Reading+- group-02.md — Search+`+ idx, err := parseIndexContent("/fake/dir", content)+ if err != nil {+ t.Fatal(err)+ }+ want := []int{0, 1, 2}+ if !reflect.DeepEqual(idx.Active, want) {+ t.Errorf("Active = %v; want %v", idx.Active, want)+ }+ if len(idx.Disabled) != 0 {+ t.Errorf("Disabled = %v; want empty", idx.Disabled)+ }+}++func TestParseIndexSingleLineComment(t *testing.T) {+ content := `## Group Files++- group-00.md — Setup+<!-- - group-01.md — Reading (disabled) -->+- group-02.md — Search+`+ idx, err := parseIndexContent("/fake/dir", content)+ if err != nil {+ t.Fatal(err)+ }+ wantActive := []int{0, 2}+ wantDisabled := []int{1}+ if !reflect.DeepEqual(idx.Active, wantActive) {+ t.Errorf("Active = %v; want %v", idx.Active, wantActive)+ }+ if !reflect.DeepEqual(idx.Disabled, wantDisabled) {+ t.Errorf("Disabled = %v; want %v", idx.Disabled, wantDisabled)+ }+}++func TestParseIndexMultiLineComment(t *testing.T) {+ content := `## Group Files++- group-00.md — Setup+<!--+- group-01.md — Reading (disabled)+- group-02.md — Search (disabled)+-->+- group-03.md — Form+`+ idx, err := parseIndexContent("/fake/dir", content)+ if err != nil {+ t.Fatal(err)+ }+ wantActive := []int{0, 3}+ wantDisabled := []int{1, 2}+ if !reflect.DeepEqual(idx.Active, wantActive) {+ t.Errorf("Active = %v; want %v", idx.Active, wantActive)+ }+ if !reflect.DeepEqual(idx.Disabled, wantDisabled) {+ t.Errorf("Disabled = %v; want %v", idx.Disabled, wantDisabled)+ }+}++func TestParseIndexEmptyIndex(t *testing.T) {+ content := `# Empty Index++No group section here.+`+ idx, err := parseIndexContent("/fake/dir", content)+ if err != nil {+ t.Fatal(err)+ }+ if len(idx.Active) != 0 || len(idx.Disabled) != 0 {+ t.Errorf("Expected empty sets; got Active=%v Disabled=%v", idx.Active, idx.Disabled)+ }+}++func TestParseIndexBacktickFilenames(t *testing.T) {+ content := `## Group Files++- ` + "`group-00.md`" + ` — Setup+<!-- - ` + "`group-01.md`" + ` — disabled -->+`+ idx, err := parseIndexContent("/fake/dir", content)+ if err != nil {+ t.Fatal(err)+ }+ if !reflect.DeepEqual(idx.Active, []int{0}) {+ t.Errorf("Active = %v; want [0]", idx.Active)+ }+ if !reflect.DeepEqual(idx.Disabled, []int{1}) {+ t.Errorf("Disabled = %v; want [1]", idx.Disabled)+ }+}++func TestScanGroupFiles(t *testing.T) {+ dir := t.TempDir()+ for _, name := range []string{"group-00.md", "group-05.md", "group-10.md", "setup.md"} {+ if err := os.WriteFile(filepath.Join(dir, name), []byte("test"), 0o644); err != nil {+ t.Fatal(err)+ }+ }+ groups, err := ScanGroupFiles(dir)+ if err != nil {+ t.Fatal(err)+ }+ want := []int{0, 5, 10}+ if !reflect.DeepEqual(groups, want) {+ t.Errorf("ScanGroupFiles = %v; want %v", groups, want)+ }+}++func TestSelectGroupsExplicit(t *testing.T) {+ idx := IndexFile{Dir: "/d", Active: []int{0, 1, 2}, Disabled: []int{3}}+ args := Args{Groups: []int{1, 3}}+ var stderr bytes.Buffer+ sel, err := SelectGroups(args, idx, &stderr)+ if err != nil {+ t.Fatal(err)+ }+ want := []int{1, 3}+ if !reflect.DeepEqual(sel.Selected, want) {+ t.Errorf("Selected = %v; want %v", sel.Selected, want)+ }+ if len(sel.Warnings) != 1 {+ t.Errorf("Expected 1 warning; got %d", len(sel.Warnings))+ }+}++func TestSelectGroupsProfile(t *testing.T) {+ idx := IndexFile{Dir: "/d", Active: []int{0, 1, 2, 3, 4, 5}}+ args := Args{Profile: "common10"}+ var stderr bytes.Buffer+ sel, err := SelectGroups(args, idx, &stderr)+ if err != nil {+ t.Fatal(err)+ }+ want := []int{0, 1, 2, 3}+ if !reflect.DeepEqual(sel.Selected, want) {+ t.Errorf("Selected = %v; want %v", sel.Selected, want)+ }+}++func TestSelectGroupsActiveOnly(t *testing.T) {+ idx := IndexFile{Dir: "/d", Active: []int{0, 2, 4}}+ args := Args{}+ var stderr bytes.Buffer+ sel, err := SelectGroups(args, idx, &stderr)+ if err != nil {+ t.Fatal(err)+ }+ if !reflect.DeepEqual(sel.Selected, idx.Active) {+ t.Errorf("Selected = %v; want %v", sel.Selected, idx.Active)+ }+}diff --git a/tests/benchmark/cmd/api-runner/loop.go b/tests/benchmark/cmd/api-runner/loop.gonew file mode 100644index 00000000..e72d7c69--- /dev/null+++ b/tests/benchmark/cmd/api-runner/loop.go@@ -0,0 +1,220 @@+package main++import (+ "encoding/json"+ "fmt"+ "io"+ "os"+ "os/exec"+ "path/filepath"+ "time"+)++type LoopConfig struct {+ Lane Lane+ Provider string+ Model string+ Groups []int+ ReportFile string+ MaxTurns int+ MaxIdleTurns int+ TimeoutSeconds int+ TurnDelayMs int+ Finalize bool+ SkipInit bool+ BenchDir string+ CommandLogFile string+ MaxInputTokens int+ MaxOutputTokens int+ Stdout io.Writer+ Stderr io.Writer+}++type LoopResult struct {+ ExitCode int+ FinalText string+ Usage UsageCounters+}++func RunLoop(cfg LoopConfig, runner Runner, shell *PersistentShell) LoopResult {+ promptCfg := PromptConfig{+ RepoRoot: filepath.Dir(filepath.Dir(cfg.BenchDir)),+ BenchDir: cfg.BenchDir,+ BenchRunDir: filepath.Join(cfg.BenchDir, "benchmark-run"),+ SetupPTFile: filepath.Join(cfg.BenchDir, "setup-pinchtab.md"),+ SetupABFile: filepath.Join(cfg.BenchDir, "setup-agent-browser.md"),+ }++ userPrompt := LaneUserPrompt(cfg.Lane, promptCfg, cfg.ReportFile, cfg.Groups)+ conversation := runner.InitialConversation(userPrompt)+ systemPromptText := SystemPrompt()++ var finalText string+ exitCode := 0+ idleTurns := 0+ lastAnswered := ReadProgress(cfg.ReportFile).Answered++ clearCommandLog(cfg.CommandLogFile)++ for turn := 1; turn <= cfg.MaxTurns; turn++ {+ if turn > 1 && cfg.TurnDelayMs > 0 {+ time.Sleep(time.Duration(cfg.TurnDelayMs) * time.Millisecond)+ }++ usage := runner.Usage()+ if cfg.MaxInputTokens > 0 && usage.InputTokens+usage.CacheCreationInputTokens+usage.CacheReadInputTokens >= cfg.MaxInputTokens {+ finalText = fmt.Sprintf("Budget exceeded: input tokens (%d) >= limit (%d)", usage.InputTokens+usage.CacheCreationInputTokens+usage.CacheReadInputTokens, cfg.MaxInputTokens)+ exitCode = 4+ break+ }+ if cfg.MaxOutputTokens > 0 && usage.OutputTokens >= cfg.MaxOutputTokens {+ finalText = fmt.Sprintf("Budget exceeded: output tokens (%d) >= limit (%d)", usage.OutputTokens, cfg.MaxOutputTokens)+ exitCode = 4+ break+ }++ response, err := runner.Send(systemPromptText, conversation)+ if err != nil {+ finalText = fmt.Sprintf("API error: %v", err)+ exitCode = 1+ break+ }++ toolCalls := runner.ExtractToolCalls(response, time.Duration(cfg.TimeoutSeconds)*time.Second)+ if len(toolCalls) > 0 {+ results := executeToolCalls(shell, toolCalls, cfg.CommandLogFile)+ conversation = runner.AppendToolResults(conversation, response, results)++ summary := BuildProgressSummary(cfg.ReportFile)+ conversation = CompactConversation(cfg.Provider, conversation, summary)++ progress := ReadProgress(cfg.ReportFile)+ if progress.Answered > lastAnswered {+ idleTurns = 0+ lastAnswered = progress.Answered+ } else {+ idleTurns+++ }++ if idleTurns >= cfg.MaxIdleTurns {+ finalText = fmt.Sprintf("Stopped after %d consecutive turns without recording a benchmark step. Check %s for the command trace.",+ idleTurns, cfg.CommandLogFile)+ exitCode = 3+ break+ }+ continue+ }++ finalText = runner.ExtractFinalText(response)+ break+ }++ if finalText == "" {+ finalText = fmt.Sprintf("Stopped after reaching max turns (%d).", cfg.MaxTurns)+ exitCode = 2+ }++ if _, err := os.Stat(cfg.ReportFile); err == nil {+ recordUsage(cfg.BenchDir, cfg.ReportFile, runner)+ if cfg.Finalize {+ finalizeReport(cfg.BenchDir, cfg.ReportFile)+ }+ }++ if finalText != "" {+ fmt.Fprintln(cfg.Stdout, finalText)+ }++ usage := runner.Usage()+ fmt.Fprintf(cfg.Stdout,+ "\n[run-usage] provider=%s requests=%d input=%d cache_create=%d cache_read=%d output=%d total=%d\n",+ runner.Provider(),+ usage.RequestCount,+ usage.InputTokens,+ usage.CacheCreationInputTokens,+ usage.CacheReadInputTokens,+ usage.OutputTokens,+ usage.InputTokens+usage.CacheCreationInputTokens+usage.CacheReadInputTokens+usage.OutputTokens,+ )++ return LoopResult{+ ExitCode: exitCode,+ FinalText: finalText,+ Usage: usage,+ }+}++func clearCommandLog(path string) {+ if path != "" {+ os.WriteFile(path, []byte{}, 0o644)+ }+}++func executeToolCalls(shell *PersistentShell, calls []ToolCall, logFile string) []ToolExecutionResult {+ var results []ToolExecutionResult+ for _, call := range calls {+ output, exitCode, err := shell.Run(call.Command, time.Duration(call.TimeoutSeconds)*time.Second)+ var result ToolExecutionResult+ if err != nil {+ result = ToolExecutionResult{+ ID: call.ID,+ IsError: true,+ Content: fmt.Sprintf("$ %s\n[runner_error]\n%s", call.Command, err.Error()),+ }+ appendCommandLog(logFile, call.Command, -1, err.Error())+ } else {+ trimmed := TrimToolOutput(output)+ result = ToolExecutionResult{+ ID: call.ID,+ IsError: exitCode != 0,+ Content: FormatToolResult(call.Command, exitCode, output),+ }+ appendCommandLog(logFile, call.Command, exitCode, trimmed)+ }+ results = append(results, result)+ }+ return results+}++func appendCommandLog(path, command string, exitCode int, output string) {+ if path == "" {+ return+ }+ entry := map[string]interface{}{+ "timestamp": time.Now().UTC().Format(time.RFC3339),+ "command": command,+ "exit_code": exitCode,+ "output": output,+ }+ data, _ := json.Marshal(entry)+ f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)+ if err != nil {+ return+ }+ defer f.Close()+ f.Write(append(data, '\n'))+}++func recordUsage(benchDir, reportFile string, runner Runner) {+ usage := runner.Usage()+ script := filepath.Join(benchDir, "scripts", "record-run-usage.sh")+ cmd := exec.Command(script,+ "--report-file", reportFile,+ "--provider", runner.Provider(),+ "--source", runner.Source(),+ "--request-count", fmt.Sprintf("%d", usage.RequestCount),+ "--input-tokens", fmt.Sprintf("%d", usage.InputTokens),+ "--output-tokens", fmt.Sprintf("%d", usage.OutputTokens),+ "--cache-creation-input-tokens", fmt.Sprintf("%d", usage.CacheCreationInputTokens),+ "--cache-read-input-tokens", fmt.Sprintf("%d", usage.CacheReadInputTokens),+ )+ cmd.Dir = benchDir+ cmd.Run()+}++func finalizeReport(benchDir, reportFile string) {+ script := filepath.Join(benchDir, "scripts", "finalize-report.sh")+ cmd := exec.Command(script, reportFile)+ cmd.Dir = benchDir+ cmd.Run()+}diff --git a/tests/benchmark/cmd/api-runner/loop_test.go b/tests/benchmark/cmd/api-runner/loop_test.gonew file mode 100644index 00000000..67860331--- /dev/null+++ b/tests/benchmark/cmd/api-runner/loop_test.go@@ -0,0 +1,279 @@+package main++import (+ "bytes"+ "encoding/json"+ "os"+ "path/filepath"+ "strings"+ "testing"+)++func TestLoopWithFakeRunner(t *testing.T) {+ dir := t.TempDir()+ reportFile := filepath.Join(dir, "report.json")+ commandLog := filepath.Join(dir, "commands.ndjson")++ os.WriteFile(reportFile, []byte(`{"totals":{"steps_answered":0},"steps":[]}`), 0o644)++ responses := []FakeResponse{+ {+ ToolCalls: []ToolCall{+ {ID: "call_1", Command: "echo hello", TimeoutSeconds: 10},+ },+ },+ {+ FinalText: "Benchmark complete",+ },+ }++ runner := NewFakeRunner("fake-model", responses)+ shell, err := NewPersistentShell(dir)+ if err != nil {+ t.Fatal(err)+ }+ defer shell.Close(false)++ var stdout, stderr bytes.Buffer+ cfg := LoopConfig{+ Lane: LanePinchtab,+ Provider: "fake",+ Model: "fake-model",+ Groups: []int{0},+ ReportFile: reportFile,+ MaxTurns: 10,+ MaxIdleTurns: 5,+ TimeoutSeconds: 30,+ TurnDelayMs: 0,+ BenchDir: dir,+ CommandLogFile: commandLog,+ Stdout: &stdout,+ Stderr: &stderr,+ }++ result := RunLoop(cfg, runner, shell)++ if result.ExitCode != 0 {+ t.Errorf("ExitCode = %d; want 0", result.ExitCode)+ }+ if result.FinalText != "Benchmark complete" {+ t.Errorf("FinalText = %q; want 'Benchmark complete'", result.FinalText)+ }+ if result.Usage.RequestCount != 2 {+ t.Errorf("RequestCount = %d; want 2", result.Usage.RequestCount)+ }++ logData, err := os.ReadFile(commandLog)+ if err != nil {+ t.Fatal(err)+ }+ if !strings.Contains(string(logData), "echo hello") {+ t.Error("command log missing 'echo hello'")+ }++ output := stdout.String()+ if !strings.Contains(output, "Benchmark complete") {+ t.Errorf("stdout missing final text: %s", output)+ }+ if !strings.Contains(output, "[run-usage]") {+ t.Errorf("stdout missing usage line: %s", output)+ }+}++func TestLoopIdleTurnLimit(t *testing.T) {+ dir := t.TempDir()+ reportFile := filepath.Join(dir, "report.json")+ commandLog := filepath.Join(dir, "commands.ndjson")++ os.WriteFile(reportFile, []byte(`{"totals":{"steps_answered":0},"steps":[]}`), 0o644)++ responses := make([]FakeResponse, 10)+ for i := range responses {+ responses[i] = FakeResponse{+ ToolCalls: []ToolCall{+ {ID: "call", Command: "echo noop", TimeoutSeconds: 10},+ },+ }+ }++ runner := NewFakeRunner("fake-model", responses)+ shell, err := NewPersistentShell(dir)+ if err != nil {+ t.Fatal(err)+ }+ defer shell.Close(false)++ var stdout bytes.Buffer+ cfg := LoopConfig{+ Lane: LanePinchtab,+ Provider: "fake",+ Model: "fake-model",+ ReportFile: reportFile,+ MaxTurns: 20,+ MaxIdleTurns: 3,+ TimeoutSeconds: 30,+ BenchDir: dir,+ CommandLogFile: commandLog,+ Stdout: &stdout,+ Stderr: &bytes.Buffer{},+ }++ result := RunLoop(cfg, runner, shell)++ if result.ExitCode != 3 {+ t.Errorf("ExitCode = %d; want 3 (idle limit)", result.ExitCode)+ }+ if !strings.Contains(result.FinalText, "consecutive turns") {+ t.Errorf("FinalText should mention consecutive turns: %s", result.FinalText)+ }+}++func TestLoopMaxTurns(t *testing.T) {+ dir := t.TempDir()+ reportFile := filepath.Join(dir, "report.json")++ os.WriteFile(reportFile, []byte(`{"totals":{"steps_answered":0},"steps":[]}`), 0o644)++ responses := make([]FakeResponse, 100)+ for i := range responses {+ responses[i] = FakeResponse{+ ToolCalls: []ToolCall{+ {ID: "call", Command: "echo turn", TimeoutSeconds: 10},+ },+ }+ }++ runner := NewFakeRunner("fake-model", responses)+ shell, err := NewPersistentShell(dir)+ if err != nil {+ t.Fatal(err)+ }+ defer shell.Close(false)++ var stdout bytes.Buffer+ cfg := LoopConfig{+ Lane: LanePinchtab,+ Provider: "fake",+ Model: "fake-model",+ ReportFile: reportFile,+ MaxTurns: 3,+ MaxIdleTurns: 100,+ TimeoutSeconds: 30,+ BenchDir: dir,+ Stdout: &stdout,+ Stderr: &bytes.Buffer{},+ }++ result := RunLoop(cfg, runner, shell)++ if result.ExitCode != 2 {+ t.Errorf("ExitCode = %d; want 2 (max turns)", result.ExitCode)+ }+ if !strings.Contains(result.FinalText, "max turns") {+ t.Errorf("FinalText should mention max turns: %s", result.FinalText)+ }+}++func TestAppendCommandLog(t *testing.T) {+ dir := t.TempDir()+ logFile := filepath.Join(dir, "log.ndjson")++ appendCommandLog(logFile, "ls -la", 0, "output here")+ appendCommandLog(logFile, "cat file", 1, "error")++ data, err := os.ReadFile(logFile)+ if err != nil {+ t.Fatal(err)+ }++ lines := strings.Split(strings.TrimSpace(string(data)), "\n")+ if len(lines) != 2 {+ t.Fatalf("got %d lines; want 2", len(lines))+ }++ var entry map[string]interface{}+ json.Unmarshal([]byte(lines[0]), &entry)+ if entry["command"] != "ls -la" {+ t.Errorf("first command = %v; want 'ls -la'", entry["command"])+ }+}++func TestLoopBudgetLimit(t *testing.T) {+ dir := t.TempDir()+ reportFile := filepath.Join(dir, "report.json")++ os.WriteFile(reportFile, []byte(`{"totals":{"steps_answered":0},"steps":[]}`), 0o644)++ responses := make([]FakeResponse, 10)+ for i := range responses {+ responses[i] = FakeResponse{+ ToolCalls: []ToolCall{+ {ID: "call", Command: "echo test", TimeoutSeconds: 10},+ },+ }+ }++ runner := NewFakeRunner("fake-model", responses)+ shell, err := NewPersistentShell(dir)+ if err != nil {+ t.Fatal(err)+ }+ defer shell.Close(false)++ var stdout bytes.Buffer+ cfg := LoopConfig{+ Lane: LanePinchtab,+ Provider: "fake",+ Model: "fake-model",+ ReportFile: reportFile,+ MaxTurns: 20,+ MaxIdleTurns: 100,+ TimeoutSeconds: 30,+ MaxInputTokens: 150,+ BenchDir: dir,+ Stdout: &stdout,+ Stderr: &bytes.Buffer{},+ }++ result := RunLoop(cfg, runner, shell)++ if result.ExitCode != 4 {+ t.Errorf("ExitCode = %d; want 4 (budget exceeded)", result.ExitCode)+ }+ if !strings.Contains(result.FinalText, "Budget exceeded") {+ t.Errorf("FinalText should mention budget: %s", result.FinalText)+ }+}++func TestFakeRunner(t *testing.T) {+ responses := []FakeResponse{+ {ToolCalls: []ToolCall{{ID: "1", Command: "echo test"}}},+ {FinalText: "done"},+ }+ runner := NewFakeRunner("test-model", responses)++ if runner.Provider() != "fake" {+ t.Errorf("Provider = %s; want 'fake'", runner.Provider())+ }++ conv := runner.InitialConversation("hello")+ if len(conv) != 1 {+ t.Fatalf("initial conv length = %d; want 1", len(conv))+ }++ resp1, _ := runner.Send("system", conv)+ calls := runner.ExtractToolCalls(resp1, 60)+ if len(calls) != 1 {+ t.Fatalf("turn 1 calls = %d; want 1", len(calls))+ }++ resp2, _ := runner.Send("system", conv)+ text := runner.ExtractFinalText(resp2)+ if text != "done" {+ t.Errorf("final text = %q; want 'done'", text)+ }++ if runner.Usage().RequestCount != 2 {+ t.Errorf("RequestCount = %d; want 2", runner.Usage().RequestCount)+ }+}diff --git a/tests/benchmark/cmd/api-runner/main.go b/tests/benchmark/cmd/api-runner/main.gonew file mode 100644index 00000000..c283ae54--- /dev/null+++ b/tests/benchmark/cmd/api-runner/main.go@@ -0,0 +1,244 @@+// Command api-runner is the Go port of+// tests/benchmark/scripts/run-api-benchmark.ts.+//+// Exit codes:+//+// 0 success (or --help)+// 1 argument/setup error+// 2 max turns reached+// 3 idle turn limit reached+// 4 budget exceeded+package main++import (+ "errors"+ "fmt"+ "io"+ "os"+ "os/exec"+ "path/filepath"+ "strings"+)++func main() {+ os.Exit(run(os.Args[1:], os.Stdout, os.Stderr))+}++func run(argv []string, stdout, stderr io.Writer) int {+ args, err := ParseArgs(argv)+ if err != nil {+ if errors.Is(err, errHelp) {+ WriteUsage(stdout)+ return 0+ }+ fmt.Fprintf(stderr, "api-runner: %v\n\n", err)+ WriteUsage(stderr)+ return 1+ }++ if args.DryRun {+ fmt.Fprint(stdout, formatPlan(args))+ return 0+ }++ benchDir := resolveBenchDir()+ provider := resolveProvider(args)+ model := resolveModel(provider, args.Model)+ groups := resolveGroupsFromArgs(args)++ reportFile := args.ReportFile+ if reportFile == "" {+ if args.SkipInit {+ reportFile = resolveReportPath(benchDir, args.Lane)+ } else {+ reportFile = initializeLane(benchDir, args, provider, model)+ }+ }++ fmt.Fprintf(stdout, "[benchmark-runner] provider=%s model=%s lane=%s groups=%s report=%s\n",+ provider, model, args.Lane, formatGroups(groups), reportFile)++ runner := createRunner(provider, model, args)+ shell, err := NewPersistentShell(benchDir)+ if err != nil {+ fmt.Fprintf(stderr, "api-runner: shell init failed: %v\n", err)+ return 1+ }+ defer shell.Close(false)++ cfg := LoopConfig{+ Lane: args.Lane,+ Provider: provider,+ Model: model,+ Groups: groups,+ ReportFile: reportFile,+ MaxTurns: args.MaxTurns,+ MaxIdleTurns: args.MaxIdleTurns,+ TimeoutSeconds: args.TimeoutSeconds,+ TurnDelayMs: args.TurnDelayMs,+ Finalize: args.Finalize,+ BenchDir: benchDir,+ CommandLogFile: filepath.Join(benchDir, "results", "agent_commands.ndjson"),+ MaxInputTokens: args.MaxInputTokens,+ MaxOutputTokens: args.MaxOutputTokens,+ Stdout: stdout,+ Stderr: stderr,+ }++ result := RunLoop(cfg, runner, shell)+ return result.ExitCode+}++func resolveBenchDir() string {+ cwd, _ := os.Getwd()+ dir := cwd+ for {+ if _, err := os.Stat(filepath.Join(dir, "go.mod")); err == nil {+ return filepath.Join(dir, "tests", "benchmark")+ }+ parent := filepath.Dir(dir)+ if parent == dir {+ return filepath.Join(cwd, "tests", "benchmark")+ }+ dir = parent+ }+}++func resolveProvider(args Args) string {+ if args.Provider != ProviderUnset {+ return string(args.Provider)+ }+ hasOpenAI := strings.TrimSpace(os.Getenv("OPENAI_API_KEY")) != ""+ hasAnthropic := strings.TrimSpace(os.Getenv("ANTHROPIC_API_KEY")) != ""+ if hasOpenAI && hasAnthropic {+ fmt.Fprintln(os.Stderr, "warning: multiple providers configured, defaulting to anthropic")+ return "anthropic"+ }+ if hasOpenAI {+ return "openai"+ }+ if hasAnthropic {+ return "anthropic"+ }+ return "anthropic"+}++func resolveModel(provider, explicit string) string {+ if explicit != "" {+ return explicit+ }+ if provider == "openai" {+ if m := os.Getenv("OPENAI_MODEL"); m != "" {+ return m+ }+ return "gpt-5"+ }+ if m := os.Getenv("ANTHROPIC_MODEL"); m != "" {+ return m+ }+ return "claude-haiku-4-5-20251001"+}++func resolveGroupsFromArgs(args Args) []int {+ if len(args.Groups) > 0 {+ return args.Groups+ }+ switch args.Profile {+ case "common10":+ return []int{0, 1, 2, 3}+ default:+ return nil+ }+}++func resolveReportPath(benchDir string, lane Lane) string {+ ptrFile := filepath.Join(benchDir, "results", "current_agent_report.txt")+ if lane == LaneAgentBrowser {+ ptrFile = filepath.Join(benchDir, "results", "current_agent_browser_report.txt")+ }+ data, err := os.ReadFile(ptrFile)+ if err != nil {+ return ""+ }+ return strings.TrimSpace(string(data))+}++func initializeLane(benchDir string, args Args, provider, model string) string {+ var script string+ if args.Lane == LanePinchtab {+ script = filepath.Join(benchDir, "scripts", "run-optimization.sh")+ } else {+ script = filepath.Join(benchDir, "scripts", "run-agent-browser-benchmark.sh")+ }+ cmd := exec.Command(script)+ cmd.Dir = benchDir+ cmd.Env = append(os.Environ(),+ "BENCHMARK_MODEL="+model,+ "BENCHMARK_RUNNER="+runnerSource(provider),+ )+ cmd.Stdout = os.Stdout+ cmd.Stderr = os.Stderr+ cmd.Run()+ return resolveReportPath(benchDir, args.Lane)+}++func runnerSource(provider string) string {+ if provider == "openai" {+ return "openai-responses"+ }+ return "anthropic-messages"+}++func createRunner(provider, model string, args Args) Runner {+ promptCaching := !args.NoPromptCaching+ if provider == "openai" {+ apiKey := os.Getenv("OPENAI_API_KEY")+ return NewOpenAIRunner(apiKey, model, args.MaxTokens, args.Temperature, promptCaching)+ }+ if provider == "fake" {+ return NewFakeRunner(model, nil)+ }+ apiKey := os.Getenv("ANTHROPIC_API_KEY")+ return NewAnthropicRunner(apiKey, model, args.MaxTokens, args.Temperature, promptCaching)+}++func formatPlan(a Args) string {+ var b strings.Builder+ b.WriteString("api-runner (Go) — resolved plan\n")+ fmt.Fprintf(&b, " lane: %s\n", a.Lane)+ fmt.Fprintf(&b, " provider: %s\n", stringOr(string(a.Provider), "(auto-detect from env)"))+ fmt.Fprintf(&b, " model: %s\n", stringOr(a.Model, "(provider default)"))+ fmt.Fprintf(&b, " groups: %s\n", formatGroups(a.Groups))+ fmt.Fprintf(&b, " profile: %s\n", stringOr(a.Profile, "(none)"))+ fmt.Fprintf(&b, " max-tokens: %d\n", a.MaxTokens)+ fmt.Fprintf(&b, " temperature: %g\n", a.Temperature)+ fmt.Fprintf(&b, " max-turns: %d\n", a.MaxTurns)+ fmt.Fprintf(&b, " max-idle-turns: %d\n", a.MaxIdleTurns)+ fmt.Fprintf(&b, " timeout-seconds: %d\n", a.TimeoutSeconds)+ fmt.Fprintf(&b, " turn-delay-ms: %d\n", a.TurnDelayMs)+ fmt.Fprintf(&b, " report-file: %s\n", stringOr(a.ReportFile, "(auto-generated)"))+ fmt.Fprintf(&b, " skip-init: %t\n", a.SkipInit)+ fmt.Fprintf(&b, " no-prompt-caching: %t\n", a.NoPromptCaching)+ fmt.Fprintf(&b, " finalize: %t\n", a.Finalize)+ fmt.Fprintf(&b, " dry-run: %t\n", a.DryRun)+ fmt.Fprintf(&b, " index-file: %s\n", stringOr(a.IndexFile, "(default)"))+ return b.String()+}++func stringOr(v, fallback string) string {+ if v == "" {+ return fallback+ }+ return v+}++func formatGroups(groups []int) string {+ if len(groups) == 0 {+ return "all"+ }+ parts := make([]string, len(groups))+ for i, g := range groups {+ parts[i] = fmt.Sprintf("%d", g)+ }+ return strings.Join(parts, ",")+}diff --git a/tests/benchmark/cmd/api-runner/openai.go b/tests/benchmark/cmd/api-runner/openai.gonew file mode 100644index 00000000..08d07114--- /dev/null+++ b/tests/benchmark/cmd/api-runner/openai.go@@ -0,0 +1,215 @@+package main++import (+ "bytes"+ "context"+ "encoding/json"+ "fmt"+ "io"+ "net/http"+ "time"+)++const OpenAIResponsesURL = "https://api.openai.com/v1/responses"++type OpenAIRunner struct {+ baseURL string+ apiKey string+ model string+ maxTokens int+ temperature float64+ promptCaching bool+ usage UsageCounters+ client *http.Client+ retryCfg RetryConfig+}++func NewOpenAIRunner(apiKey, model string, maxTokens int, temperature float64, promptCaching bool) *OpenAIRunner {+ return &OpenAIRunner{+ baseURL: OpenAIResponsesURL,+ apiKey: apiKey,+ model: model,+ maxTokens: maxTokens,+ temperature: temperature,+ promptCaching: promptCaching,+ client: &http.Client{Timeout: 5 * time.Minute},+ retryCfg: DefaultRetryConfig(),+ }+}++func (r *OpenAIRunner) Provider() string { return "openai" }+func (r *OpenAIRunner) Source() string { return "openai-responses" }+func (r *OpenAIRunner) Model() string { return r.model }+func (r *OpenAIRunner) Usage() UsageCounters { return r.usage }++func (r *OpenAIRunner) ToolDefinitions() interface{} {+ return []map[string]interface{}{+ {+ "type": "function",+ "name": "run_command",+ "description": "Run a shell command in a persistent bash session rooted at tests/benchmark. Environment changes such as cd/export persist across calls.",+ "parameters": map[string]interface{}{+ "type": "object",+ "properties": map[string]interface{}{+ "command": map[string]interface{}{"type": "string"},+ "timeout_seconds": map[string]interface{}{"type": "integer", "minimum": 1, "maximum": 600},+ },+ "required": []string{"command"},+ },+ },+ }+}++func (r *OpenAIRunner) InitialConversation(userPrompt string) []interface{} {+ return []interface{}{+ map[string]interface{}{+ "role": "user",+ "content": []interface{}{+ map[string]interface{}{"type": "input_text", "text": userPrompt},+ },+ },+ }+}++func (r *OpenAIRunner) Send(systemPrompt string, conversation []interface{}) (interface{}, error) {+ body := map[string]interface{}{+ "model": r.model,+ "instructions": systemPrompt,+ "tools": r.ToolDefinitions(),+ "input": conversation,+ "max_output_tokens": r.maxTokens,+ }+ if r.promptCaching {+ body["prompt_cache_retention"] = "24h"+ }++ payload, err := json.Marshal(body)+ if err != nil {+ return nil, fmt.Errorf("marshal request: %w", err)+ }++ var result map[string]interface{}+ resp, err := DoWithRetry(context.Background(), r.retryCfg, func() (*http.Response, error) {+ req, err := http.NewRequest("POST", r.baseURL, bytes.NewReader(payload))+ if err != nil {+ return nil, err+ }+ req.Header.Set("Content-Type", "application/json")+ req.Header.Set("Authorization", "Bearer "+r.apiKey)+ return r.client.Do(req)+ })+ if err != nil {+ return nil, err+ }+ defer resp.Body.Close()++ respBody, err := io.ReadAll(resp.Body)+ if err != nil {+ return nil, fmt.Errorf("read response: %w", err)+ }++ if err := json.Unmarshal(respBody, &result); err != nil {+ return nil, fmt.Errorf("unmarshal response: %w", err)+ }++ if resp.StatusCode >= 400 {+ return nil, fmt.Errorf("OpenAI API error %d: %s", resp.StatusCode, string(respBody))+ }++ r.updateUsage(result)+ return result, nil+}++func (r *OpenAIRunner) updateUsage(result map[string]interface{}) {+ usage, _ := result["usage"].(map[string]interface{})+ if usage == nil {+ return+ }+ r.usage.RequestCount++++ totalInput := toInt(usage["input_tokens"])+ cached := 0+ if details, ok := usage["input_tokens_details"].(map[string]interface{}); ok {+ cached = toInt(details["cached_tokens"])+ }+ r.usage.InputTokens += max(0, totalInput-cached)+ r.usage.OutputTokens += toInt(usage["output_tokens"])+ r.usage.CacheReadInputTokens += cached+}++func (r *OpenAIRunner) ExtractToolCalls(response interface{}, defaultTimeout time.Duration) []ToolCall {+ resp, _ := response.(map[string]interface{})+ output, _ := resp["output"].([]interface{})+ var calls []ToolCall+ for _, item := range output {+ m, _ := item.(map[string]interface{})+ if m["type"] != "function_call" {+ continue+ }+ argsStr, _ := m["arguments"].(string)+ var args map[string]interface{}+ if argsStr != "" {+ json.Unmarshal([]byte(argsStr), &args)+ }+ cmd, _ := args["command"].(string)+ if cmd == "" {+ continue+ }+ timeout := int(defaultTimeout.Seconds())+ if t := toInt(args["timeout_seconds"]); t > 0 {+ timeout = t+ }+ callID := fmt.Sprintf("%v", m["call_id"])+ if callID == "" || callID == "<nil>" {+ callID = fmt.Sprintf("%v", m["id"])+ }+ calls = append(calls, ToolCall{+ ID: callID,+ Command: cmd,+ TimeoutSeconds: timeout,+ })+ }+ return calls+}++func (r *OpenAIRunner) AppendToolResults(conversation []interface{}, response interface{}, results []ToolExecutionResult) []interface{} {+ resp, _ := response.(map[string]interface{})+ output, _ := resp["output"].([]interface{})+ conversation = append(conversation, output...)+ for _, res := range results {+ conversation = append(conversation, map[string]interface{}{+ "type": "function_call_output",+ "call_id": res.ID,+ "output": res.Content,+ })+ }+ return conversation+}++func (r *OpenAIRunner) ExtractFinalText(response interface{}) string {+ resp, _ := response.(map[string]interface{})++ if text, ok := resp["output_text"].(string); ok && text != "" {+ return text+ }++ output, _ := resp["output"].([]interface{})+ var texts []string+ for _, item := range output {+ m, _ := item.(map[string]interface{})+ if m["type"] != "message" {+ continue+ }+ content, _ := m["content"].([]interface{})+ for _, part := range content {+ p, _ := part.(map[string]interface{})+ ptype, _ := p["type"].(string)+ if ptype == "output_text" || ptype == "text" {+ if text, ok := p["text"].(string); ok && text != "" {+ texts = append(texts, text)+ }+ }+ }+ }+ return joinStrings(texts, "\n")+}diff --git a/tests/benchmark/cmd/api-runner/openai_test.go b/tests/benchmark/cmd/api-runner/openai_test.gonew file mode 100644index 00000000..99fb9de2--- /dev/null+++ b/tests/benchmark/cmd/api-runner/openai_test.go@@ -0,0 +1,196 @@+package main++import (+ "encoding/json"+ "io"+ "net/http"+ "net/http/httptest"+ "testing"+ "time"+)++func TestOpenAIToolDefinitions(t *testing.T) {+ r := NewOpenAIRunner("key", "gpt-5-mini", 4096, 0, true)+ tools := r.ToolDefinitions().([]map[string]interface{})+ if len(tools) != 1 {+ t.Fatalf("got %d tools; want 1", len(tools))+ }+ if tools[0]["name"] != "run_command" {+ t.Errorf("tool name = %v; want 'run_command'", tools[0]["name"])+ }+ if tools[0]["type"] != "function" {+ t.Errorf("tool type = %v; want 'function'", tools[0]["type"])+ }+}++func TestOpenAIInitialConversation(t *testing.T) {+ r := NewOpenAIRunner("key", "gpt-5-mini", 4096, 0, true)+ conv := r.InitialConversation("test prompt")+ if len(conv) != 1 {+ t.Fatalf("got %d messages; want 1", len(conv))+ }+ msg := conv[0].(map[string]interface{})+ if msg["role"] != "user" {+ t.Errorf("role = %v; want 'user'", msg["role"])+ }+ content := msg["content"].([]interface{})+ if len(content) != 1 {+ t.Fatalf("got %d content parts; want 1", len(content))+ }+ part := content[0].(map[string]interface{})+ if part["type"] != "input_text" {+ t.Errorf("content type = %v; want 'input_text'", part["type"])+ }+}++func TestOpenAISend(t *testing.T) {+ var receivedBody map[string]interface{}+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ if r.Method != "POST" {+ t.Errorf("method = %s; want POST", r.Method)+ }+ auth := r.Header.Get("Authorization")+ if auth != "Bearer test-key" {+ t.Errorf("Authorization = %s; want 'Bearer test-key'", auth)+ }++ body, _ := io.ReadAll(r.Body)+ json.Unmarshal(body, &receivedBody)++ w.Header().Set("Content-Type", "application/json")+ json.NewEncoder(w).Encode(map[string]interface{}{+ "output": []interface{}{+ map[string]interface{}{"type": "message", "content": []interface{}{+ map[string]interface{}{"type": "output_text", "text": "Hello"},+ }},+ },+ "usage": map[string]interface{}{+ "input_tokens": 100,+ "output_tokens": 50,+ "input_tokens_details": map[string]interface{}{+ "cached_tokens": 30,+ },+ },+ })+ }))+ defer server.Close()++ r := NewOpenAIRunner("test-key", "gpt-5-mini", 4096, 0.5, true)+ r.baseURL = server.URL+ r.retryCfg.Sleep = func(d time.Duration) {}++ conv := r.InitialConversation("hello")+ _, err := r.Send("system", conv)+ if err != nil {+ t.Fatal(err)+ }++ if receivedBody["model"] != "gpt-5-mini" {+ t.Errorf("model = %v; want gpt-5-mini", receivedBody["model"])+ }+ if receivedBody["max_output_tokens"] != float64(4096) {+ t.Errorf("max_output_tokens = %v; want 4096", receivedBody["max_output_tokens"])+ }+ if receivedBody["prompt_cache_retention"] != "24h" {+ t.Errorf("prompt_cache_retention = %v; want '24h'", receivedBody["prompt_cache_retention"])+ }++ usage := r.Usage()+ if usage.RequestCount != 1 {+ t.Errorf("RequestCount = %d; want 1", usage.RequestCount)+ }+ if usage.InputTokens != 70 {+ t.Errorf("InputTokens = %d; want 70 (100 - 30 cached)", usage.InputTokens)+ }+ if usage.CacheReadInputTokens != 30 {+ t.Errorf("CacheReadInputTokens = %d; want 30", usage.CacheReadInputTokens)+ }+}++func TestOpenAIExtractToolCalls(t *testing.T) {+ r := NewOpenAIRunner("key", "model", 4096, 0, true)+ response := map[string]interface{}{+ "output": []interface{}{+ map[string]interface{}{+ "type": "function_call",+ "call_id": "call_123",+ "arguments": `{"command": "ls -la", "timeout_seconds": 30}`,+ },+ map[string]interface{}{+ "type": "message",+ },+ },+ }++ calls := r.ExtractToolCalls(response, 120*time.Second)+ if len(calls) != 1 {+ t.Fatalf("got %d calls; want 1", len(calls))+ }+ if calls[0].ID != "call_123" {+ t.Errorf("ID = %s; want call_123", calls[0].ID)+ }+ if calls[0].Command != "ls -la" {+ t.Errorf("Command = %s; want 'ls -la'", calls[0].Command)+ }+ if calls[0].TimeoutSeconds != 30 {+ t.Errorf("TimeoutSeconds = %d; want 30", calls[0].TimeoutSeconds)+ }+}++func TestOpenAIExtractFinalText(t *testing.T) {+ r := NewOpenAIRunner("key", "model", 4096, 0, true)++ t.Run("output_text field", func(t *testing.T) {+ response := map[string]interface{}{+ "output_text": "Direct output",+ }+ text := r.ExtractFinalText(response)+ if text != "Direct output" {+ t.Errorf("got %q; want 'Direct output'", text)+ }+ })++ t.Run("from output array", func(t *testing.T) {+ response := map[string]interface{}{+ "output": []interface{}{+ map[string]interface{}{+ "type": "message",+ "content": []interface{}{+ map[string]interface{}{"type": "output_text", "text": "Hello"},+ map[string]interface{}{"type": "text", "text": "World"},+ },+ },+ },+ }+ text := r.ExtractFinalText(response)+ if text != "Hello\nWorld" {+ t.Errorf("got %q; want 'Hello\\nWorld'", text)+ }+ })+}++func TestOpenAIAppendToolResults(t *testing.T) {+ r := NewOpenAIRunner("key", "model", 4096, 0, true)+ conv := []interface{}{}+ response := map[string]interface{}{+ "output": []interface{}{+ map[string]interface{}{"type": "function_call", "call_id": "123"},+ },+ }+ results := []ToolExecutionResult{+ {ID: "123", IsError: false, Content: "output"},+ }++ conv = r.AppendToolResults(conv, response, results)+ if len(conv) != 2 {+ t.Fatalf("got %d items; want 2", len(conv))+ }++ funcOutput := conv[1].(map[string]interface{})+ if funcOutput["type"] != "function_call_output" {+ t.Errorf("type = %v; want 'function_call_output'", funcOutput["type"])+ }+ if funcOutput["call_id"] != "123" {+ t.Errorf("call_id = %v; want '123'", funcOutput["call_id"])+ }+}diff --git a/tests/benchmark/cmd/api-runner/prompt.go b/tests/benchmark/cmd/api-runner/prompt.gonew file mode 100644index 00000000..b1f6b20c--- /dev/null+++ b/tests/benchmark/cmd/api-runner/prompt.go@@ -0,0 +1,293 @@+package main++import (+ "fmt"+ "os"+ "path/filepath"+ "strings"+)++type PromptConfig struct {+ RepoRoot string+ BenchDir string+ BenchRunDir string+ SetupPTFile string+ SetupABFile string+}++func DefaultPromptConfig() PromptConfig {+ cwd, _ := os.Getwd()+ repoRoot := findRepoRoot(cwd)+ benchDir := filepath.Join(repoRoot, "tests", "benchmark")+ return PromptConfig{+ RepoRoot: repoRoot,+ BenchDir: benchDir,+ BenchRunDir: filepath.Join(benchDir, "benchmark-run"),+ SetupPTFile: filepath.Join(benchDir, "setup-pinchtab.md"),+ SetupABFile: filepath.Join(benchDir, "setup-agent-browser.md"),+ }+}++func findRepoRoot(start string) string {+ dir := start+ for {+ if _, err := os.Stat(filepath.Join(dir, "go.mod")); err == nil {+ return dir+ }+ parent := filepath.Dir(dir)+ if parent == dir {+ return start+ }+ dir = parent+ }+}++type LaneConfig struct {+ Label string+ SkillInstruction string+ Wrapper string+ RecordType string+ WorkflowSummary []string+ AdapterNotes []string+ BootstrapCommands []string+}++func LanePromptConfig(lane Lane, cfg PromptConfig) LaneConfig {+ if lane == LanePinchtab {+ return LaneConfig{+ Label: "PinchTab agent",+ SkillInstruction: fmt.Sprintf("Read %s exactly once before acting.", cfg.SetupPTFile),+ Wrapper: "./scripts/pt",+ RecordType: "pinchtab",+ WorkflowSummary: []string{+ "use only ./scripts/pt for browser actions",+ "keep one shared tab via PINCHTAB_TAB",+ "use snap -i -c for actionable refs",+ "use text or text --full for reading content",+ "refresh refs after any navigation or DOM change",+ },+ AdapterNotes: []string{+ `a navigation-expected click can return "Error 409: unexpected page navigation ..."; treat that as likely success and verify with a fresh snapshot/text read`,+ "do not assume every command returns JSON; only parse JSON when the command actually returned JSON",+ `the download endpoint returns JSON with base64 content in "data", not a local file path`,+ `this environment uses BSD/macOS userland tools; avoid GNU-only flags such as "head -n -1"`,+ },+ BootstrapCommands: []string{+ "./scripts/pt health",+ "./scripts/pt tab",+ "export PINCHTAB_TAB=$(./scripts/pt nav http://fixtures/)",+ "./scripts/pt snap -i -c",+ },+ }+ }+ return LaneConfig{+ Label: "agent-browser",+ SkillInstruction: fmt.Sprintf("Read %s exactly once before acting, then load the official agent-browser skill exactly once with `./scripts/ab skills get agent-browser --full`.", cfg.SetupABFile),+ Wrapper: "./scripts/ab",+ RecordType: "agent-browser",+ WorkflowSummary: []string{+ "use only ./scripts/ab for browser actions",+ "keep the shared benchmark session across commands",+ "use snapshot -i -c for actionable refs",+ "use fresh refs after any navigation or DOM change",+ "reuse the current browser state instead of reopening pages unless needed",+ },+ AdapterNotes: []string{+ "after navigation-triggering clicks, verify the resulting page with a fresh snapshot/text read instead of trusting the click response alone",+ "do not assume every command returns JSON; parse only when the command actually returned JSON",+ "for downloads, inspect returned content instead of assuming a file already exists locally",+ `this environment uses BSD/macOS userland tools; avoid GNU-only flags such as "head -n -1"`,+ },+ BootstrapCommands: []string{+ "./scripts/ab skills get agent-browser --full",+ "./scripts/ab open http://fixtures/",+ "./scripts/ab snapshot -i -c",+ },+ }+}++func SystemPrompt() string {+ return `You are a precise benchmark execution agent. Use tools to inspect the repo and run the benchmark lane exactly as instructed.++Rules:+- Never fabricate command output or task results.+- Use the shell tool for all file reads and command execution.+- Do not use destructive commands such as rm -rf, git reset, or checkout.+- After recording an answer, verify it immediately against the task oracle.+- Prefer factual command output over long reasoning.`+}++func LaneSubsetInstructions(groups []int) string {+ if len(groups) == 0 {+ return "Execute the full benchmark task set."+ }+ parts := make([]string, len(groups))+ for i, g := range groups {+ parts[i] = fmt.Sprintf("%d", g)+ }+ return fmt.Sprintf(`Execute only these benchmark groups: %s.+Do not attempt groups outside this subset.+Treat all other groups as out of scope for this run rather than as failures.+For the selected groups, execute every step in the group unless blocked.`, strings.Join(parts, ", "))+}++func readText(path string) string {+ data, err := os.ReadFile(path)+ if err != nil {+ return ""+ }+ return strings.TrimSpace(string(data))+}++func BenchmarkRunGroupFile(dir string, group int) string {+ return filepath.Join(dir, fmt.Sprintf("group-%02d.md", group))+}++func BenchmarkRunAllGroups(dir string) []int {+ entries, err := os.ReadDir(dir)+ if err != nil {+ return nil+ }+ var groups []int+ for _, e := range entries {+ if e.IsDir() {+ continue+ }+ if m := groupFileRegex.FindStringSubmatch(e.Name()); m != nil {+ var n int+ fmt.Sscanf(m[1], "%d", &n)+ groups = append(groups, n)+ }+ }+ return groups+}++func BenchmarkRunSelectedText(cfg PromptConfig, groups []int) string {+ indexPath := filepath.Join(cfg.BenchRunDir, "index.md")+ selected := groups+ if len(selected) == 0 {+ selected = BenchmarkRunAllGroups(cfg.BenchRunDir)+ }+ chunks := []string{readText(indexPath)}+ for _, g := range selected {+ path := BenchmarkRunGroupFile(cfg.BenchRunDir, g)+ if content := readText(path); content != "" {+ chunks = append(chunks, content)+ }+ }+ return strings.Join(chunks, "\n\n")+}++func LaneTaskSourceInstructions(lane Lane, cfg PromptConfig, groups []int) string {+ indexPath := filepath.Join(cfg.BenchRunDir, "index.md")++ if len(groups) == 0 {+ return fmt.Sprintf("Use the benchmark run index at %s plus all group files in %s.\n\n%s",+ indexPath, cfg.BenchRunDir, BenchmarkRunSelectedText(cfg, nil))+ }++ if lane == LanePinchtab {+ subset := BenchmarkRunSelectedText(cfg, groups)+ return fmt.Sprintf("Use only this selected task subset from %s:\n\n%s", cfg.BenchRunDir, subset)+ }++ var sections []string+ for _, g := range groups {+ if g == 0 {+ sections = append(sections, agentBrowserGroup0())+ continue+ }+ path := BenchmarkRunGroupFile(cfg.BenchRunDir, g)+ if content := readText(path); content != "" {+ sections = append(sections, content)+ }+ }++ return fmt.Sprintf("Use only this selected task subset for agent-browser: Group 0 from %s, Groups 1+ from %s.\n\n%s\n\n%s",+ cfg.SetupABFile, cfg.BenchRunDir, readText(indexPath), strings.Join(sections, "\n\n"))+}++func agentBrowserGroup0() string {+ return `## Group 0: Setup & Diagnosis (agent-browser lane)++### 0.1 Open fixtures home+Run ` + "`./scripts/ab open http://fixtures/`" + `.++**Verify**: Open succeeds and the home page loads.++### 0.2 Snapshot interactive refs+Run ` + "`./scripts/ab snapshot -i -c`" + ` on the home page.++**Verify**: Interactive refs are returned without error.++### 0.3 Session persists across commands+Use the same wrapper session across multiple commands and confirm browser state persists.++**Verify**: A follow-up action can use the existing page/session state.`+}++func LaneUserPrompt(lane Lane, cfg PromptConfig, reportFile string, groups []int) string {+ lc := LanePromptConfig(lane, cfg)+ firstGroup := 0+ if len(groups) > 0 {+ firstGroup = groups[0]+ }++ var bootstrapLines []string+ for i, cmd := range lc.BootstrapCommands {+ bootstrapLines = append(bootstrapLines, fmt.Sprintf("%d. %s", i+1, cmd))+ }+ bootstrap := strings.Join(bootstrapLines, "\n")++ taskInstr := LaneTaskSourceInstructions(lane, cfg, groups)+ taskInstr = strings.ReplaceAll(taskInstr, "\n", "\n ")++ subsetInstr := LaneSubsetInstructions(groups)+ subsetInstr = strings.ReplaceAll(subsetInstr, "\n", "\n - ")++ return fmt.Sprintf(`Work in this repo: %s++Benchmark lane: %s execution.++Requirements:+- Follow a linear execution flow: skill once, selected groups, execute, record, verify, continue.+- Do not read README, browse directories, or inspect unrelated files unless a command path is missing.+- %s+- Tool wrapper:+ - %s+- Workflow summary:+ - %s+- Adapter notes:+ - %s+- Task scope:+ %s+- For each completed step:+ 1. record the observed answer with:+ - ./scripts/record-step.sh --report-file %s --type %s <group> <step> answer "<what you saw>" "notes"+ 2. immediately verify it with:+ - ./scripts/verify-step.sh --report-file %s --type %s <group> <step> <pass|fail|skip> "verification notes"+- If a step cannot be completed, record fail or skip in the same report.+- Do not leave answered steps pending verification.+- Keep commands concise. Prefer rg/sed/cat only when you must inspect a specific file.+- After the skill step, begin actual benchmark execution immediately.+- Start with this bootstrap command sequence before attempting the selected steps:+%s+- After the bootstrap, immediately execute Group %d step 1.+- Subset selection:+ - %s+- Finish when all selected steps are executed or when you are blocked.++Your final answer should briefly summarize completion status and the main blockers, if any.`,+ cfg.RepoRoot,+ lc.Label,+ lc.SkillInstruction,+ lc.Wrapper,+ strings.Join(lc.WorkflowSummary, "\n - "),+ strings.Join(lc.AdapterNotes, "\n - "),+ taskInstr,+ reportFile, lc.RecordType,+ reportFile, lc.RecordType,+ bootstrap,+ firstGroup,+ subsetInstr)+}diff --git a/tests/benchmark/cmd/api-runner/prompt_test.go b/tests/benchmark/cmd/api-runner/prompt_test.gonew file mode 100644index 00000000..2fd4d060--- /dev/null+++ b/tests/benchmark/cmd/api-runner/prompt_test.go@@ -0,0 +1,122 @@+package main++import (+ "os"+ "path/filepath"+ "strings"+ "testing"+)++func TestSystemPromptGolden(t *testing.T) {+ golden, err := os.ReadFile("testdata/golden/system-prompt.txt")+ if err != nil {+ t.Fatalf("read golden: %v", err)+ }+ got := SystemPrompt()+ want := strings.TrimSpace(string(golden))+ if got != want {+ t.Errorf("SystemPrompt mismatch:\n--- want ---\n%s\n--- got ---\n%s", want, got)+ }+}++func TestLaneSubsetInstructionsFull(t *testing.T) {+ got := LaneSubsetInstructions(nil)+ want := "Execute the full benchmark task set."+ if got != want {+ t.Errorf("got %q; want %q", got, want)+ }+}++func TestLaneSubsetInstructionsSubset(t *testing.T) {+ got := LaneSubsetInstructions([]int{0, 1, 2})+ if !strings.Contains(got, "Execute only these benchmark groups: 0, 1, 2.") {+ t.Errorf("missing group list in %q", got)+ }+ if !strings.Contains(got, "Do not attempt groups outside this subset.") {+ t.Errorf("missing subset constraint in %q", got)+ }+}++func TestLanePromptConfigAgent(t *testing.T) {+ cfg := PromptConfig{+ RepoRoot: "/repo",+ BenchDir: "/repo/tests/benchmark",+ BenchRunDir: "/repo/tests/benchmark/benchmark-run",+ SetupPTFile: "/repo/tests/benchmark/setup-pinchtab.md",+ SetupABFile: "/repo/tests/benchmark/setup-agent-browser.md",+ }+ lc := LanePromptConfig(LanePinchtab, cfg)+ if lc.Label != "PinchTab agent" {+ t.Errorf("Label = %q; want 'PinchTab agent'", lc.Label)+ }+ if lc.Wrapper != "./scripts/pt" {+ t.Errorf("Wrapper = %q; want './scripts/pt'", lc.Wrapper)+ }+}++func TestLanePromptConfigAgentBrowser(t *testing.T) {+ cfg := PromptConfig{+ RepoRoot: "/repo",+ BenchDir: "/repo/tests/benchmark",+ BenchRunDir: "/repo/tests/benchmark/benchmark-run",+ SetupPTFile: "/repo/tests/benchmark/setup-pinchtab.md",+ SetupABFile: "/repo/tests/benchmark/setup-agent-browser.md",+ }+ lc := LanePromptConfig(LaneAgentBrowser, cfg)+ if lc.Label != "agent-browser" {+ t.Errorf("Label = %q; want 'agent-browser'", lc.Label)+ }+ if lc.Wrapper != "./scripts/ab" {+ t.Errorf("Wrapper = %q; want './scripts/ab'", lc.Wrapper)+ }+}++func TestLaneUserPromptStructure(t *testing.T) {+ cfg := DefaultPromptConfig()+ reportFile := cfg.BenchDir + "/results/test_report.json"++ prompt := LaneUserPrompt(LanePinchtab, cfg, reportFile, nil)++ checks := []string{+ "Work in this repo:",+ "Benchmark lane: PinchTab agent execution.",+ "./scripts/pt",+ "record-step.sh",+ "verify-step.sh",+ "bootstrap command sequence",+ }+ for _, check := range checks {+ if !strings.Contains(prompt, check) {+ t.Errorf("prompt missing %q", check)+ }+ }+}++func TestBenchmarkRunGroupFile(t *testing.T) {+ got := BenchmarkRunGroupFile("/dir", 5)+ want := "/dir/group-05.md"+ if got != want {+ t.Errorf("got %q; want %q", got, want)+ }+}++func TestBenchmarkRunAllGroups(t *testing.T) {+ dir := t.TempDir()+ for _, name := range []string{"group-00.md", "group-01.md", "group-10.md", "index.md"} {+ os.WriteFile(filepath.Join(dir, name), []byte("test"), 0o644)+ }+ groups := BenchmarkRunAllGroups(dir)+ if len(groups) != 3 {+ t.Errorf("got %d groups; want 3", len(groups))+ }+}++func TestAgentBrowserGroup0Content(t *testing.T) {+ content := agentBrowserGroup0()+ if !strings.Contains(content, "Group 0: Setup & Diagnosis (agent-browser lane)") {+ t.Error("missing group 0 header")+ }+ if !strings.Contains(content, "./scripts/ab open http://fixtures/") {+ t.Error("missing open command")+ }+}diff --git a/tests/benchmark/cmd/api-runner/report.go b/tests/benchmark/cmd/api-runner/report.gonew file mode 100644index 00000000..40cae2de--- /dev/null+++ b/tests/benchmark/cmd/api-runner/report.go@@ -0,0 +1,129 @@+package main++import (+ "encoding/json"+ "fmt"+ "os"+ "regexp"+ "strings"+)++const (+ MaxSummarySteps = 12+ MaxSummaryTextChars = 220+)++type ReportTotals struct {+ StepsAnswered int `json:"steps_answered"`+ StepsFailed int `json:"steps_failed"`+ StepsSkipped int `json:"steps_skipped"`+ StepsVerifiedPassed int `json:"steps_verified_passed"`+ StepsVerifiedFailed int `json:"steps_verified_failed"`+ StepsVerifiedSkipped int `json:"steps_verified_skipped"`+ StepsPendingVerification int `json:"steps_pending_verification"`+}++type ReportStep struct {+ ID string `json:"id"`+ Status string `json:"status"`+ Answer string `json:"answer"`+ Notes string `json:"notes"`+ Verification *struct {+ Status string `json:"status"`+ } `json:"verification,omitempty"`+}++type Report struct {+ Totals ReportTotals `json:"totals"`+ Steps []ReportStep `json:"steps"`+}++func ReadReport(path string) (*Report, error) {+ data, err := os.ReadFile(path)+ if err != nil {+ return nil, err+ }+ var report Report+ if err := json.Unmarshal(data, &report); err != nil {+ return nil, err+ }+ return &report, nil+}++func shorten(text string, maxChars int) string {+ ws := regexp.MustCompile(`\s+`)+ normalized := strings.TrimSpace(ws.ReplaceAllString(text, " "))+ if len(normalized) <= maxChars {+ return normalized+ }+ return normalized[:maxChars-3] + "..."+}++func BuildProgressSummary(reportFile string) string {+ report, err := ReadReport(reportFile)+ if err != nil {+ return "No benchmark report file exists yet."+ }++ totals := report.Totals+ steps := report.Steps++ startIdx := len(steps) - MaxSummarySteps+ if startIdx < 0 {+ startIdx = 0+ }+ recentSteps := steps[startIdx:]++ var lines []string+ lines = append(lines, "Benchmark progress summary from the external report.")+ lines = append(lines, fmt.Sprintf("- answered=%d", totals.StepsAnswered))+ lines = append(lines, fmt.Sprintf("- execution_failed=%d", totals.StepsFailed))+ lines = append(lines, fmt.Sprintf("- execution_skipped=%d", totals.StepsSkipped))+ lines = append(lines, fmt.Sprintf("- verified_passed=%d", totals.StepsVerifiedPassed))+ lines = append(lines, fmt.Sprintf("- verified_failed=%d", totals.StepsVerifiedFailed))+ lines = append(lines, fmt.Sprintf("- verified_skipped=%d", totals.StepsVerifiedSkipped))+ lines = append(lines, fmt.Sprintf("- pending_verification=%d", totals.StepsPendingVerification))++ if len(recentSteps) > 0 {+ lines = append(lines, "Recent recorded steps:")+ for _, step := range recentSteps {+ verification := ""+ if step.Verification != nil && step.Verification.Status != "" {+ verification = fmt.Sprintf(" / verify=%s", step.Verification.Status)+ }+ answer := step.Answer+ if answer == "" {+ answer = step.Notes+ }+ lines = append(lines, fmt.Sprintf("- %s: status=%s%s; %s",+ step.ID, step.Status, verification, shorten(answer, MaxSummaryTextChars)))+ }+ } else {+ lines = append(lines, "No steps recorded yet.")+ }++ lines = append(lines, "Use the report file and benchmark wrappers as the source of truth.")+ lines = append(lines, "Do not re-read large documentation files unless you are blocked on a specific syntax detail.")++ return strings.Join(lines, "\n")+}++type Progress struct {+ Answered int+ Failed int+ Skipped int+ VerifiedPassed int+}++func ReadProgress(reportFile string) Progress {+ report, err := ReadReport(reportFile)+ if err != nil {+ return Progress{}+ }+ return Progress{+ Answered: report.Totals.StepsAnswered,+ Failed: report.Totals.StepsFailed,+ Skipped: report.Totals.StepsSkipped,+ VerifiedPassed: report.Totals.StepsVerifiedPassed,+ }+}diff --git a/tests/benchmark/cmd/api-runner/report_test.go b/tests/benchmark/cmd/api-runner/report_test.gonew file mode 100644index 00000000..fbee7092--- /dev/null+++ b/tests/benchmark/cmd/api-runner/report_test.go@@ -0,0 +1,108 @@+package main++import (+ "encoding/json"+ "os"+ "path/filepath"+ "strings"+ "testing"+)++func TestShorten(t *testing.T) {+ tests := []struct {+ input string+ max int+ expected string+ }{+ {"hello world", 20, "hello world"},+ {"hello world\n\tfoo", 20, "hello world foo"},+ {"this is a very long string", 10, "this is..."},+ }++ for _, tc := range tests {+ got := shorten(tc.input, tc.max)+ if got != tc.expected {+ t.Errorf("shorten(%q, %d) = %q; want %q", tc.input, tc.max, got, tc.expected)+ }+ }+}++func TestBuildProgressSummaryNoFile(t *testing.T) {+ summary := BuildProgressSummary("/nonexistent/file.json")+ if !strings.Contains(summary, "No benchmark report file") {+ t.Errorf("unexpected summary for missing file: %q", summary)+ }+}++func TestBuildProgressSummaryWithData(t *testing.T) {+ dir := t.TempDir()+ path := filepath.Join(dir, "report.json")++ report := map[string]interface{}{+ "totals": map[string]interface{}{+ "steps_answered": 5,+ "steps_failed": 1,+ "steps_skipped": 0,+ "steps_verified_passed": 4,+ "steps_verified_failed": 1,+ "steps_verified_skipped": 0,+ "steps_pending_verification": 0,+ },+ "steps": []map[string]interface{}{+ {"id": "0.1", "status": "answered", "answer": "ok", "verification": map[string]string{"status": "pass"}},+ {"id": "0.2", "status": "answered", "answer": "done"},+ {"id": "1.1", "status": "failed", "notes": "error occurred"},+ },+ }+ data, _ := json.Marshal(report)+ os.WriteFile(path, data, 0o644)++ summary := BuildProgressSummary(path)++ checks := []string{+ "answered=5",+ "execution_failed=1",+ "verified_passed=4",+ "0.1: status=answered / verify=pass",+ "1.1: status=failed",+ }+ for _, check := range checks {+ if !strings.Contains(summary, check) {+ t.Errorf("summary missing %q:\n%s", check, summary)+ }+ }+}++func TestReadProgress(t *testing.T) {+ dir := t.TempDir()+ path := filepath.Join(dir, "report.json")++ report := map[string]interface{}{+ "totals": map[string]interface{}{+ "steps_answered": 10,+ "steps_failed": 2,+ "steps_skipped": 1,+ "steps_verified_passed": 8,+ },+ }+ data, _ := json.Marshal(report)+ os.WriteFile(path, data, 0o644)++ p := ReadProgress(path)+ if p.Answered != 10 {+ t.Errorf("Answered = %d; want 10", p.Answered)+ }+ if p.Failed != 2 {+ t.Errorf("Failed = %d; want 2", p.Failed)+ }+ if p.VerifiedPassed != 8 {+ t.Errorf("VerifiedPassed = %d; want 8", p.VerifiedPassed)+ }+}++func TestReadProgressMissingFile(t *testing.T) {+ p := ReadProgress("/nonexistent.json")+ if p.Answered != 0 || p.Failed != 0 {+ t.Errorf("expected zero progress for missing file; got %+v", p)+ }+}diff --git a/tests/benchmark/cmd/api-runner/retry.go b/tests/benchmark/cmd/api-runner/retry.gonew file mode 100644index 00000000..76c927b5--- /dev/null+++ b/tests/benchmark/cmd/api-runner/retry.go@@ -0,0 +1,105 @@+package main++import (+ "context"+ "fmt"+ "io"+ "net/http"+ "strconv"+ "time"+)++const (+ MaxRetryAttempts = 3+ MaxRetryBackoff = 15 * time.Second+ InitialRetryDelay = 2 * time.Second+)++type RetryConfig struct {+ MaxAttempts int+ InitialDelay time.Duration+ MaxDelay time.Duration+ Clock func() time.Time+ Sleep func(time.Duration)+}++func DefaultRetryConfig() RetryConfig {+ return RetryConfig{+ MaxAttempts: MaxRetryAttempts,+ InitialDelay: InitialRetryDelay,+ MaxDelay: MaxRetryBackoff,+ Clock: time.Now,+ Sleep: time.Sleep,+ }+}++type DoFunc func() (*http.Response, error)++func DoWithRetry(ctx context.Context, cfg RetryConfig, do DoFunc) (*http.Response, error) {+ var lastErr error+ delay := cfg.InitialDelay++ for attempt := 0; attempt < cfg.MaxAttempts; attempt++ {+ if ctx.Err() != nil {+ return nil, ctx.Err()+ }++ resp, err := do()+ if err != nil {+ lastErr = err+ if attempt < cfg.MaxAttempts-1 {+ cfg.Sleep(delay)+ delay = min(delay*2, cfg.MaxDelay)+ }+ continue+ }++ if resp.StatusCode < 400 {+ return resp, nil+ }++ if !isRetryable(resp.StatusCode) {+ return resp, nil+ }++ lastErr = fmt.Errorf("HTTP %d", resp.StatusCode)+ io.Copy(io.Discard, resp.Body)+ resp.Body.Close()++ if attempt < cfg.MaxAttempts-1 {+ retryDelay := parseRetryAfter(resp.Header.Get("Retry-After"), cfg.Clock())+ if retryDelay > 0 {+ delay = min(retryDelay, cfg.MaxDelay)+ }+ cfg.Sleep(delay)+ delay = min(delay*2, cfg.MaxDelay)+ }+ }++ return nil, fmt.Errorf("max retries exceeded: %w", lastErr)+}++func isRetryable(statusCode int) bool {+ switch statusCode {+ case 429, 500, 502, 503, 504:+ return true+ default:+ return false+ }+}++func parseRetryAfter(value string, now time.Time) time.Duration {+ if value == "" {+ return 0+ }++ if seconds, err := strconv.Atoi(value); err == nil {+ return time.Duration(seconds) * time.Second+ }++ if t, err := http.ParseTime(value); err == nil {+ return t.Sub(now)+ }++ return 0+}diff --git a/tests/benchmark/cmd/api-runner/retry_test.go b/tests/benchmark/cmd/api-runner/retry_test.gonew file mode 100644index 00000000..f807d529--- /dev/null+++ b/tests/benchmark/cmd/api-runner/retry_test.go@@ -0,0 +1,153 @@+package main++import (+ "context"+ "net/http"+ "net/http/httptest"+ "sync/atomic"+ "testing"+ "time"+)++func TestRetrySuccessOnFirstAttempt(t *testing.T) {+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ w.WriteHeader(200)+ }))+ defer server.Close()++ cfg := DefaultRetryConfig()+ cfg.Sleep = func(d time.Duration) {}++ var callCount int32+ resp, err := DoWithRetry(context.Background(), cfg, func() (*http.Response, error) {+ atomic.AddInt32(&callCount, 1)+ return http.Get(server.URL)+ })+ if err != nil {+ t.Fatal(err)+ }+ resp.Body.Close()++ if callCount != 1 {+ t.Errorf("callCount = %d; want 1", callCount)+ }+}++func TestRetryOn429(t *testing.T) {+ var callCount int32+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ count := atomic.AddInt32(&callCount, 1)+ if count < 3 {+ w.Header().Set("Retry-After", "1")+ w.WriteHeader(429)+ return+ }+ w.WriteHeader(200)+ }))+ defer server.Close()++ cfg := DefaultRetryConfig()+ var totalSleep time.Duration+ cfg.Sleep = func(d time.Duration) { totalSleep += d }++ resp, err := DoWithRetry(context.Background(), cfg, func() (*http.Response, error) {+ return http.Get(server.URL)+ })+ if err != nil {+ t.Fatal(err)+ }+ resp.Body.Close()++ if callCount != 3 {+ t.Errorf("callCount = %d; want 3", callCount)+ }+ if totalSleep < 2*time.Second {+ t.Errorf("totalSleep = %v; want >= 2s", totalSleep)+ }+}++func TestRetryExhausted(t *testing.T) {+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ w.WriteHeader(429)+ }))+ defer server.Close()++ cfg := DefaultRetryConfig()+ cfg.Sleep = func(d time.Duration) {}++ var callCount int32+ _, err := DoWithRetry(context.Background(), cfg, func() (*http.Response, error) {+ atomic.AddInt32(&callCount, 1)+ return http.Get(server.URL)+ })+ if err == nil {+ t.Fatal("expected error")+ }+ if callCount != 3 {+ t.Errorf("callCount = %d; want 3", callCount)+ }+}++func TestNoRetryOn400(t *testing.T) {+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ w.WriteHeader(400)+ }))+ defer server.Close()++ cfg := DefaultRetryConfig()+ cfg.Sleep = func(d time.Duration) {}++ var callCount int32+ resp, err := DoWithRetry(context.Background(), cfg, func() (*http.Response, error) {+ atomic.AddInt32(&callCount, 1)+ return http.Get(server.URL)+ })+ if err != nil {+ t.Fatal(err)+ }+ resp.Body.Close()++ if callCount != 1 {+ t.Errorf("callCount = %d; want 1 (no retry on 400)", callCount)+ }+ if resp.StatusCode != 400 {+ t.Errorf("status = %d; want 400", resp.StatusCode)+ }+}++func TestParseRetryAfterSeconds(t *testing.T) {+ now := time.Now()+ d := parseRetryAfter("5", now)+ if d != 5*time.Second {+ t.Errorf("got %v; want 5s", d)+ }+}++func TestParseRetryAfterHTTPDate(t *testing.T) {+ now := time.Date(2026, 4, 17, 12, 0, 0, 0, time.UTC)+ future := now.Add(30 * time.Second)+ d := parseRetryAfter(future.Format(http.TimeFormat), now)+ if d < 29*time.Second || d > 31*time.Second {+ t.Errorf("got %v; want ~30s", d)+ }+}++func TestRetryContextCancellation(t *testing.T) {+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ w.WriteHeader(429)+ }))+ defer server.Close()++ ctx, cancel := context.WithCancel(context.Background())+ cancel()++ cfg := DefaultRetryConfig()+ cfg.Sleep = func(d time.Duration) {}++ _, err := DoWithRetry(ctx, cfg, func() (*http.Response, error) {+ return http.Get(server.URL)+ })+ if err != context.Canceled {+ t.Errorf("got %v; want context.Canceled", err)+ }+}diff --git a/tests/benchmark/cmd/api-runner/runner.go b/tests/benchmark/cmd/api-runner/runner.gonew file mode 100644index 00000000..22696be9--- /dev/null+++ b/tests/benchmark/cmd/api-runner/runner.go@@ -0,0 +1,36 @@+package main++import "time"++type ToolCall struct {+ ID string+ Command string+ TimeoutSeconds int+}++type ToolExecutionResult struct {+ ID string+ IsError bool+ Content string+}++type UsageCounters struct {+ RequestCount int+ InputTokens int+ OutputTokens int+ CacheCreationInputTokens int+ CacheReadInputTokens int+}++type Runner interface {+ Provider() string+ Source() string+ Model() string+ ToolDefinitions() interface{}+ InitialConversation(userPrompt string) []interface{}+ Send(systemPrompt string, conversation []interface{}) (interface{}, error)+ ExtractToolCalls(response interface{}, defaultTimeout time.Duration) []ToolCall+ AppendToolResults(conversation []interface{}, response interface{}, results []ToolExecutionResult) []interface{}+ ExtractFinalText(response interface{}) string+ Usage() UsageCounters+}diff --git a/tests/benchmark/cmd/api-runner/shell.go b/tests/benchmark/cmd/api-runner/shell.gonew file mode 100644index 00000000..5c5f5d69--- /dev/null+++ b/tests/benchmark/cmd/api-runner/shell.go@@ -0,0 +1,174 @@+package main++import (+ "bufio"+ "crypto/rand"+ "encoding/hex"+ "errors"+ "fmt"+ "io"+ "os/exec"+ "regexp"+ "strings"+ "sync"+ "time"+)++const MaxToolOutputChars = 2400++type PersistentShell struct {+ cmd *exec.Cmd+ stdin io.WriteCloser+ stdout io.ReadCloser+ stderr io.ReadCloser++ mu sync.Mutex+ buffer string+ closed bool+}++func NewPersistentShell(workDir string) (*PersistentShell, error) {+ cmd := exec.Command("/bin/bash")+ cmd.Dir = workDir++ stdin, err := cmd.StdinPipe()+ if err != nil {+ return nil, fmt.Errorf("stdin pipe: %w", err)+ }++ stdout, err := cmd.StdoutPipe()+ if err != nil {+ stdin.Close()+ return nil, fmt.Errorf("stdout pipe: %w", err)+ }++ stderr, err := cmd.StderrPipe()+ if err != nil {+ stdin.Close()+ stdout.Close()+ return nil, fmt.Errorf("stderr pipe: %w", err)+ }++ if err := cmd.Start(); err != nil {+ stdin.Close()+ stdout.Close()+ stderr.Close()+ return nil, fmt.Errorf("start shell: %w", err)+ }++ ps := &PersistentShell{+ cmd: cmd,+ stdin: stdin,+ stdout: stdout,+ stderr: stderr,+ }++ go ps.drainStderr()++ return ps, nil+}++func (ps *PersistentShell) drainStderr() {+ io.Copy(io.Discard, ps.stderr)+}++func (ps *PersistentShell) Run(command string, timeout time.Duration) (output string, exitCode int, err error) {+ ps.mu.Lock()+ defer ps.mu.Unlock()++ if ps.closed {+ return "", -1, errors.New("shell is closed")+ }++ nonce := randomHex(8)+ marker := fmt.Sprintf("__BENCH_DONE__%s", nonce)+ markerRegex := regexp.MustCompile(fmt.Sprintf(`\n%s:(\d+)\r?\n`, regexp.QuoteMeta(marker)))++ wrapped := fmt.Sprintf("%s\nprintf '\\n%s:%%s\\n' \"$?\"\n", command, marker)+ if _, err := io.WriteString(ps.stdin, wrapped); err != nil {+ return "", -1, fmt.Errorf("write command: %w", err)+ }++ deadline := time.Now().Add(timeout)+ reader := bufio.NewReader(ps.stdout)++ for time.Now().Before(deadline) {+ remaining := time.Until(deadline)+ if remaining < 10*time.Millisecond {+ remaining = 10 * time.Millisecond+ }++ done := make(chan struct{})+ var chunk []byte+ var readErr error++ go func() {+ buf := make([]byte, 4096)+ n, e := reader.Read(buf)+ chunk = buf[:n]+ readErr = e+ close(done)+ }()++ select {+ case <-done:+ if readErr != nil && readErr != io.EOF {+ return "", -1, fmt.Errorf("read stdout: %w", readErr)+ }+ ps.buffer += string(chunk)++ if m := markerRegex.FindStringSubmatchIndex(ps.buffer); m != nil {+ output = ps.buffer[:m[0]]+ output = strings.TrimPrefix(output, "\n")+ var code int+ fmt.Sscanf(ps.buffer[m[2]:m[3]], "%d", &code)+ ps.buffer = ps.buffer[m[1]:]+ return output, code, nil+ }+ case <-time.After(remaining):+ ps.closeLocked(true)+ return "", -1, fmt.Errorf("command timed out after %v: %s", timeout, command)+ }+ }++ ps.closeLocked(true)+ return "", -1, fmt.Errorf("command timed out after %v: %s", timeout, command)+}++func (ps *PersistentShell) closeLocked(force bool) {+ if ps.closed {+ return+ }+ ps.closed = true++ if force {+ ps.cmd.Process.Kill()+ } else {+ io.WriteString(ps.stdin, "exit\n")+ }+ ps.stdin.Close()+ ps.cmd.Wait()+}++func (ps *PersistentShell) Close(force bool) {+ ps.mu.Lock()+ defer ps.mu.Unlock()+ ps.closeLocked(force)+}++func randomHex(n int) string {+ b := make([]byte, n)+ rand.Read(b)+ return hex.EncodeToString(b)+}++func TrimToolOutput(text string) string {+ if len(text) <= MaxToolOutputChars {+ return text+ }+ return fmt.Sprintf("%s\n\n[output truncated: %d more chars]", text[:MaxToolOutputChars], len(text)-MaxToolOutputChars)+}++func FormatToolResult(command string, exitCode int, output string) string {+ return fmt.Sprintf("$ %s\n[exit_code=%d]\n%s", command, exitCode, TrimToolOutput(output))+}diff --git a/tests/benchmark/cmd/api-runner/shell_test.go b/tests/benchmark/cmd/api-runner/shell_test.gonew file mode 100644index 00000000..ac4f8df6--- /dev/null+++ b/tests/benchmark/cmd/api-runner/shell_test.go@@ -0,0 +1,137 @@+package main++import (+ "strings"+ "testing"+ "time"+)++func TestShellEcho(t *testing.T) {+ ps, err := NewPersistentShell(t.TempDir())+ if err != nil {+ t.Fatal(err)+ }+ defer ps.Close(false)++ out, code, err := ps.Run("echo hello", 5*time.Second)+ if err != nil {+ t.Fatal(err)+ }+ if code != 0 {+ t.Errorf("exit code = %d; want 0", code)+ }+ if strings.TrimSpace(out) != "hello" {+ t.Errorf("output = %q; want 'hello'", out)+ }+}++func TestShellNonZeroExit(t *testing.T) {+ ps, err := NewPersistentShell(t.TempDir())+ if err != nil {+ t.Fatal(err)+ }+ defer ps.Close(false)++ out, code, err := ps.Run("(exit 42)", 5*time.Second)+ if err != nil {+ t.Fatal(err)+ }+ if code != 42 {+ t.Errorf("exit code = %d; want 42", code)+ }+ if out != "" {+ t.Errorf("output = %q; want empty", out)+ }+}++func TestShellTimeout(t *testing.T) {+ ps, err := NewPersistentShell(t.TempDir())+ if err != nil {+ t.Fatal(err)+ }+ defer ps.Close(true)++ _, _, err = ps.Run("sleep 60", 100*time.Millisecond)+ if err == nil {+ t.Fatal("expected timeout error")+ }+ if !strings.Contains(err.Error(), "timed out") {+ t.Errorf("error = %v; want timeout", err)+ }+}++func TestShellEnvPersistence(t *testing.T) {+ ps, err := NewPersistentShell(t.TempDir())+ if err != nil {+ t.Fatal(err)+ }+ defer ps.Close(false)++ _, code1, err := ps.Run("export TESTVAR=foobar123", 5*time.Second)+ if err != nil {+ t.Fatal(err)+ }+ if code1 != 0 {+ t.Fatalf("export exit code = %d", code1)+ }++ out, code2, err := ps.Run("echo $TESTVAR", 5*time.Second)+ if err != nil {+ t.Fatal(err)+ }+ if code2 != 0 {+ t.Errorf("echo exit code = %d", code2)+ }+ if strings.TrimSpace(out) != "foobar123" {+ t.Errorf("output = %q; want 'foobar123'", out)+ }+}++func TestShellLargeOutput(t *testing.T) {+ ps, err := NewPersistentShell(t.TempDir())+ if err != nil {+ t.Fatal(err)+ }+ defer ps.Close(false)++ out, code, err := ps.Run("seq 1 50000", 30*time.Second)+ if err != nil {+ t.Fatal(err)+ }+ if code != 0 {+ t.Errorf("exit code = %d; want 0", code)+ }+ lines := strings.Split(strings.TrimSpace(out), "\n")+ if len(lines) != 50000 {+ t.Errorf("got %d lines; want 50000", len(lines))+ }+ if lines[0] != "1" || lines[49999] != "50000" {+ t.Errorf("first/last lines incorrect: %q / %q", lines[0], lines[49999])+ }+}++func TestTrimToolOutput(t *testing.T) {+ short := "hello world"+ if got := TrimToolOutput(short); got != short {+ t.Errorf("short text modified: %q", got)+ }++ long := strings.Repeat("x", 3000)+ trimmed := TrimToolOutput(long)+ if len(trimmed) > MaxToolOutputChars+100 {+ t.Errorf("trimmed too long: %d chars", len(trimmed))+ }+ if !strings.Contains(trimmed, "[output truncated:") {+ t.Error("missing truncation marker")+ }+}++func TestFormatToolResult(t *testing.T) {+ result := FormatToolResult("ls -la", 0, "file.txt")+ if !strings.HasPrefix(result, "$ ls -la") {+ t.Errorf("missing command prefix: %q", result)+ }+ if !strings.Contains(result, "[exit_code=0]") {+ t.Errorf("missing exit code: %q", result)+ }+}diff --git a/tests/benchmark/cmd/api-runner/tasks.go b/tests/benchmark/cmd/api-runner/tasks.gonew file mode 100644index 00000000..3ce76b95--- /dev/null+++ b/tests/benchmark/cmd/api-runner/tasks.go@@ -0,0 +1,70 @@+package main++import (+ "fmt"+ "io"+ "sort"+)++type GroupSelection struct {+ Selected []int+ Warnings []string+}++func SelectGroups(args Args, idx IndexFile, stderr io.Writer) (GroupSelection, error) {+ var sel GroupSelection++ if len(args.Groups) > 0 {+ for _, g := range args.Groups {+ sel.Selected = append(sel.Selected, g)+ if idx.IsDisabled(g) {+ w := fmt.Sprintf("warning: group %d is disabled in index but explicitly requested", g)+ sel.Warnings = append(sel.Warnings, w)+ fmt.Fprintln(stderr, w)+ }+ }+ sort.Ints(sel.Selected)+ return sel, nil+ }++ profileGroups := resolveProfile(args.Profile)+ activeSet := make(map[int]struct{})+ for _, g := range idx.Active {+ activeSet[g] = struct{}{}+ }++ if len(profileGroups) > 0 {+ for _, g := range profileGroups {+ if _, ok := activeSet[g]; ok {+ sel.Selected = append(sel.Selected, g)+ }+ }+ sort.Ints(sel.Selected)+ return sel, nil+ }++ if len(idx.Active) > 0 {+ sel.Selected = append(sel.Selected, idx.Active...)+ return sel, nil+ }++ fallback, err := ScanGroupFiles(idx.Dir)+ if err != nil {+ return sel, fmt.Errorf("fallback scan: %w", err)+ }+ sel.Selected = fallback+ return sel, nil+}++func resolveProfile(profile string) []int {+ switch profile {+ case "common10":+ return []int{0, 1, 2, 3}+ default:+ return nil+ }+}++func GroupFilePath(dir string, group int) string {+ return fmt.Sprintf("%s/group-%02d.md", dir, group)+}diff --git a/tests/benchmark/cmd/api-runner/testdata/golden/progress-summary-sample.txt b/tests/benchmark/cmd/api-runner/testdata/golden/progress-summary-sample.txtnew file mode 100644index 00000000..df1b2717--- /dev/null+++ b/tests/benchmark/cmd/api-runner/testdata/golden/progress-summary-sample.txt@@ -0,0 +1,23 @@+Benchmark progress summary from the external report.+- answered=35+- execution_failed=0+- execution_skipped=0+- verified_passed=35+- verified_failed=0+- verified_skipped=0+- pending_verification=0+Recent recorded steps:+- 5.2: status=answer / verify=pass; VERIFY_LOGIN_SUCCESS_DASHBOARD+- 6.1: status=answer / verify=pass; Headphones .99, Watch .99, Charger .99. Mechanical Keyboard out of stock.+- 6.2: status=answer / verify=pass; Total: .98+- 6.3: status=answer / verify=pass; VERIFY_CHECKOUT_SUCCESS_ORDER+- 7.1: status=answer / verify=pass; COMMENT_POSTED_RATING_5_TEXT_RECEIVED+- 7.2: status=answer / verify=pass; Navigated wiki -> Python article after reading categories (DevTools 15, Languages 12)+- 8.1: status=answer / verify=pass; 404 Not Found, server responsive+- 8.2: status=answer / verify=pass; Error 400: no element found+- 9.1: status=answer / verify=pass; screenshot-20260416-233758.jpg (61566 bytes)+- 9.2: status=answer / verify=pass; page-20260416-233759.pdf (46078 bytes)+- 10.1: status=answer / verify=pass; Settings modal visible with theme dropdown+- 10.2: status=answer / verify=pass; THEME_DARK_APPLIED+Use the report file and benchmark wrappers as the source of truth.+Do not re-read large documentation files unless you are blocked on a specific syntax detail.\ No newline at end of filediff --git a/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-all.txt b/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-all.txtnew file mode 100644index 00000000..82541b23--- /dev/null+++ b/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-all.txt@@ -0,0 +1,692 @@+Work in this repo: /sessions/eager-determined-cannon/mnt/pinchtab++Benchmark lane: PinchTab agent execution.++Requirements:+- Follow a linear execution flow: skill once, selected groups, execute, record, verify, continue.+- Do not read README, browse directories, or inspect unrelated files unless a command path is missing.+- Read /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/setup-pinchtab.md exactly once before acting.+- Tool wrapper:+ - ./scripts/pt+- Workflow summary:+ - use only ./scripts/pt for browser actions+ - keep one shared tab via PINCHTAB_TAB+ - use snap -i -c for actionable refs+ - use text or text --full for reading content+ - refresh refs after any navigation or DOM change+- Adapter notes:+ - a navigation-expected click can return "Error 409: unexpected page navigation ..."; treat that as likely success and verify with a fresh snapshot/text read+ - do not assume every command returns JSON; only parse JSON when the command actually returned JSON+ - the download endpoint returns JSON with base64 content in "data", not a local file path+ - this environment uses BSD/macOS userland tools; avoid GNU-only flags such as "head -n -1"+- Task scope:+ Use the benchmark run index at /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/benchmark-run/index.md plus all group files in /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/benchmark-run.++ # Benchmark Run++ Natural-language benchmark task set for the agent-driven lanes.++ Read this index first, then read the relevant group files.++ ## How to use this directory++ - `setup-pinchtab.md` and `setup-agent-browser.md` define lane-specific setup, wrapper usage, recording, and Group 0 behavior.+ - This directory defines the shared task groups.+ - Each group lives in its own file: `group-00.md` through `group-38.md`.+ - When running a subset, read only the selected group files.+ - When running the full suite, proceed in numeric order.++ ## Group Files++ - `group-00.md` — Setup & Diagnosis+ - `group-01.md` — Reading & Extracting Real Content+ - `group-02.md` — Search & Dynamic Interaction+ - `group-03.md` — Complex Form+ - `group-04.md` — SPA State Management+ - `group-05.md` — Login Flow+ - `group-06.md` — Multi-Step E-commerce+ - `group-07.md` — Content + Interaction Combined+ - `group-08.md` — Error Handling+ - `group-09.md` — Export+ - `group-10.md` — Nested Interactions & Modal Dialogs+ - `group-11.md` — State Persistence & Page Reload+ - `group-12.md` — Multi-Page Navigation & Back Button+ - `group-13.md` — Form State & Multi-Step Submission+ - `group-14.md` — Dynamic Content Loading+ - `group-15.md` — Complex Data Extraction & Aggregation+ - `group-16.md` — Hover & Tooltips+ - `group-17.md` — Scrolling+ - `group-18.md` — File Download+ - `group-19.md` — iFrame+ - `group-20.md` — Dialogs+ - `group-21.md` — Async / awaitPromise+ - `group-22.md` — Mouse Drag & Drop+ - `group-23.md` — Async / Loading state+ - `group-24.md` — Keyboard events+ - `group-25.md` — Tab panels+ - `group-26.md` — Accordion+ - `group-27.md` — Contenteditable editor+ - `group-28.md` — Range slider+ - `group-29.md` — Pagination+ - `group-30.md` — Custom dropdown menu+ - `group-31.md` — Nested iframes (3 levels deep)+ - `group-32.md` — Dynamic iframe (inserted after load)+ - `group-33.md` — srcdoc iframe+ - `group-34.md` — Sandboxed iframe+ - `group-35.md` — Long-form article (Medium/Substack style)+ - `group-36.md` — Search results page (SERP)+ - `group-37.md` — Q&A thread (Stack-Overflow style)+ - `group-38.md` — Pricing table++ # Group 0: Setup & Diagnosis++ ### 0.1 Server reachable+ Check that the PinchTab server is healthy.++ **Verify**: Server responds with `status: ok`.++ ### 0.2 Auth is required+ Make a request to the server with a **wrong** token (`PINCHTAB_TOKEN=wrong-token ./scripts/pt health`) and confirm it is rejected. The `pt` wrapper always injects the benchmark token by default, so you must explicitly override it to test auth rejection.++ **Verify**: Response is HTTP 401 or contains `unauthorized`.++ ### 0.3 Auth works with token+ Repeat the same request WITH the bearer token and confirm it succeeds.++ **Verify**: Response is HTTP 200.++ ### 0.4 Instance available+ Confirm at least one Chrome instance is running. If none exist, start one.++ **Verify**: Health response shows `defaultInstance.status == "running"` (or after starting one, the new instance is running).++ ### 0.5 List existing tabs+ Get the current list of open tabs.++ **Verify**: A list (possibly empty) is returned without error.++ ### 0.6 Clean stale tabs+ If any tabs from previous runs are open, close them so the benchmark starts from a known state.++ **Verify**: After cleanup, the tab list is empty (or only contains a single about:blank tab).++ ### 0.7 Network reach to target+ Navigate to `http://fixtures/` and confirm the fixtures server is reachable from PinchTab.++ **Verify**: Navigate returns successfully and the page contains benchmark content.++ ### 0.8 Capture initial tab ID+ Save the tab ID returned by the navigate in 0.7. Use this tab ID for all subsequent tasks to avoid creating new tabs.++ **Verify**: A tab ID was captured and matches what `GET /tabs` reports.++ ---++ # Group 1: Reading & Extracting Real Content++ ### 1.1 Get the full list of categories from the wiki index+ Navigate to `http://fixtures/wiki.html` and extract all category names with their article counts.++ **Verify**: Can name at least 2 categories and their article counts (e.g. "Programming Languages: 12 articles").++ ### 1.2 Navigate by clicking a link+ From the wiki index, click the "Go (programming language)" link to navigate to the Go article.++ **Verify**: You are now on the Go article page (not the wiki index).++ ### 1.3 Extract structured data from a table+ On the Go article, read the infobox table and answer: Who designed Go, and what year did it first appear?++ **Verify**: Answer contains "Robert Griesemer" (or "Rob Pike" or "Ken Thompson") and "2009".++ ### 1.4 Count list items+ On the Go article, count how many key features are listed.++ **Verify**: Answer is 6 (verify against `FEATURE_COUNT_6`).++ ### 1.5 Read all article headlines from articles page+ Navigate to `http://fixtures/articles.html` and list all article titles.++ **Verify**: Found at least 3 articles including "The Future of Artificial Intelligence".++ ### 1.6 Read dashboard metrics+ Navigate to `http://fixtures/dashboard.html` and extract: Total Users, Revenue, and Conversion Rate.++ **Verify**: Found `24,582` users AND `$1,284,930` revenue.++ ---++ # Group 2: Search & Dynamic Interaction++ ### 2.1 Use wiki search to find a page+ On `http://fixtures/wiki.html`, search for "golang" using the search form. Do not navigate directly — use the search input.++ **Verify**: Ended up on the Go article page after search.++ ### 2.2 Search with no results+ On `http://fixtures/search.html`, search for something with no results (use "xyznonexistent").++ **Verify**: Page handled it gracefully (no crash, some response rendered).++ ### 2.3 Search for AI content+ On `http://fixtures/search.html`, search for "artificial intelligence" and verify a result appeared.++ **Verify**: Result contains "The Future of Artificial Intelligence".++ ---++ # Group 3: Complex Form++ ### 3.1 Fill and submit a complete form+ Navigate to `http://fixtures/form.html`. Complete the entire form:+ - Full name: "Agent Test User"+ - Email: "[email protected]"+ - Phone: "+44 20 9999 0000"+ - Country: United Kingdom+ - Subject: Technical Support+ - Message: "Testing PinchTab form automation"+ - Check newsletter+ - Set priority to High+ - Submit++ **Verify**: Form submitted successfully. Confirmation shows name "AGENT_TEST_USER".++ ### 3.2 Reset and refill+ After submitting, if the form is still accessible (or navigate back), verify you can identify the reset button.++ **Verify**: Reset/back button or form element found in snapshot.++ ---++ # Group 4: SPA State Management++ ### 4.1 Read initial app state+ Navigate to `http://fixtures/spa.html?reset=1` (the `?reset=1` query param clears any previous localStorage state so the SPA starts with its default 3 tasks). Read the current task list — how many tasks exist, how many are active vs done?++ **Verify**: Found 3 total, 2 active, 1 done (verify `TASK_STATS_TOTAL_3_ACTIVE_2_DONE_1`).++ ### 4.2 Add a new high-priority task+ Add a task called "Automate deployment" with high priority.++ **Verify**: Task appeared in the list (`TASK_ADDED_AUTOMATE_DEPLOYMENT_PRIORITY_HIGH`).++ ### 4.3 Delete a task+ Delete the task titled "Write benchmark tests".++ **Verify**: Task count changed (went from 4 to 3).++ ---++ # Group 5: Login Flow++ ### 5.1 Attempt login with wrong credentials+ Navigate to `http://fixtures/login.html`. Try to log in with username "admin" and password "wrong".++ **Verify**: Error message appeared (`INVALID_CREDENTIALS_ERROR`).++ ### 5.2 Login successfully+ Clear the form and log in with username "benchmark" / password "test456".++ **Verify**: Dashboard appeared after login (`VERIFY_LOGIN_SUCCESS_DASHBOARD`).++ ---++ # Group 6: Multi-Step E-commerce++ ### 6.1 Research products before buying+ Navigate to `http://fixtures/ecommerce.html`. List all available products with their prices. Which product is out of stock?++ **Verify**: Found Wireless Headphones ($149.99), Smart Watch Pro ($299.99), Portable Charger ($49.99). Mechanical Keyboard is out of stock.++ ### 6.2 Add two items and verify total+ Add Wireless Headphones and Smart Watch Pro to cart. What is the total?++ **Verify**: Cart total is $449.98 (149.99 + 299.99).++ ### 6.3 Complete checkout+ Click checkout to complete the order.++ **Verify**: Order confirmation shows (`VERIFY_CHECKOUT_SUCCESS_ORDER`).++ ---++ # Group 7: Content + Interaction Combined++ ### 7.1 Read and comment on Go article+ Navigate to `http://fixtures/wiki-go.html`. Read the article, then post a comment with rating 5 stars and text "Excellent reference".++ **Verify**: Comment posted (`COMMENT_POSTED_RATING_5_TEXT_RECEIVED`).++ ### 7.2 Cross-page research task+ Navigate to wiki index, find which category has the most articles, then navigate to one of its listed items.++ **Verify**: Successfully navigated to at least one article page after reading the category counts.++ ---++ # Group 8: Error Handling++ ### 8.1 Handle 404 gracefully+ Try to navigate to a page that doesn't exist: `http://fixtures/missing-page-abc.html`.++ **Verify**: Got a response (404 or error), no crash, server still responsive after.++ ### 8.2 Handle missing element gracefully+ On any page, try to click an element with ID `#fake-button-that-does-not-exist`.++ **Verify**: Got a clear error message, not a crash or hang.++ ---++ # Group 9: Export++ ### 9.1 Screenshot a complex page+ Navigate to `http://fixtures/dashboard.html` and take a screenshot.++ **Verify**: Screenshot generated (file saved or base64 returned).++ ### 9.2 Export a page as PDF+ Export the dashboard as a PDF.++ **Verify**: PDF generated.++ ---++ # Group 10: Nested Interactions & Modal Dialogs++ ### 10.1 Open and interact with modal on dashboard+ Navigate to `http://fixtures/dashboard.html`. Find and click the Settings button (selector: `#settings-btn`) to open the modal dialog.++ **Verify**: Modal appeared — snapshot contains "Dashboard Settings".++ ### 10.2 Modify settings and close modal+ In the modal, select "Dark" from the theme dropdown (`#theme-select`), then click the Save button (`#modal-save`). After the modal closes, check the page content.++ **Verify**: Page contains `THEME_DARK_APPLIED`.++ ---++ # Group 11: State Persistence & Page Reload++ ### 11.1 Add an item and verify after page reload+ Navigate to `http://fixtures/spa.html?reset=1` (starts with clean state). Add a task titled exactly "Persistent Task Test". Then navigate away to `http://fixtures/` and back to `http://fixtures/spa.html` (WITHOUT the reset param) to verify localStorage persistence.++ **Verify**: After reload, the task still appears in the list (`TASK_PERSISTENT_TEST_FOUND_AFTER_RELOAD`).++ ### 11.2 Logout and log back in+ From the logged-in dashboard, click Sign Out to log out. Then log in again with username "benchmark" / password "test456".++ **Verify**: Successfully logged back in and dashboard shows `SESSION_RENEWED`.++ ---++ # Group 12: Multi-Page Navigation & Back Button++ ### 12.1 Navigate through multiple pages and return+ Starting from `http://fixtures/`, navigate to wiki → Go article → back to wiki → back to home.++ **Verify**: Successfully returned to home page (title contains "Benchmark" or "Home").++ ### 12.2 Compare data across pages+ Navigate to wiki.html, note the total article count from categories. Navigate to articles.html, count articles there. Compare totals.++ **Verify**: Can report totals from both pages and explain difference (`COMPARISON_DATA_FOUND`).++ ---++ # Group 13: Form State & Multi-Step Submission++ ### 13.1 Submit form without email+ Navigate to `http://fixtures/form.html`. Fill only the name field ("Validator Test"), leave email blank, click Submit. The browser's native required-field validation will prevent submission.++ **Verify**: Submission blocked (form stays open, no success message shown).++ ### 13.2 Submit form without optional phone field+ Fill the form with: name "No Phone User", email "[email protected]", country "de", subject "feedback". Leave the phone field empty. Submit.++ **Verify**: Submission succeeded and page shows `OPTIONAL_FIELD_SKIPPED_SUCCESS`.++ ---++ # Group 14: Dynamic Content Loading++ ### 14.1 Load more products+ Navigate to `http://fixtures/ecommerce.html`. Find and click the "Load More Products" button to reveal additional products.++ **Verify**: Additional products appeared (`ADDITIONAL_PRODUCTS_LOADED`).++ ### 14.2 Add a lazy-loaded product to cart+ After loading more products, add product #5 (USB-C Cable) to the cart.++ **Verify**: Cart shows the lazy-loaded item (`CART_UPDATED_WITH_LAZY_PRODUCT`).++ ---++ # Group 15: Complex Data Extraction & Aggregation++ ### 15.1 Extract and sum financial data+ Navigate to `http://fixtures/dashboard.html`. Extract revenue and profit values, calculate profit margin.++ **Verify**: Correctly calculated: profit_margin = (profit / revenue) * 100 (`PROFIT_MARGIN_CALCULATED`).++ ### 15.2 Build comparison table from multiple sources+ Visit these 3 pages and compare their feature counts and key features:+ - `http://fixtures/wiki-go.html` (Go: 6 features)+ - `http://fixtures/wiki-python.html` (Python: 7 features)+ - `http://fixtures/wiki-rust.html` (Rust: 5 features)++ Report which language has the most features and name 1 feature unique to each.++ **Verify**: Response is factually correct AND wiki-python.html contains `COMPARISON_TABLE_BUILT`.++ ---++ # Group 16: Hover & Tooltips++ ### 16.1 Hover reveals hidden content+ Navigate to `http://fixtures/hovers.html`. Hover over the first avatar to reveal its hidden info.++ **Verify**: After hovering, snapshot contains `HOVER_REVEALED_USER_1`.++ ### 16.2 Hover swap+ Hover over the second avatar.++ **Verify**: Snapshot contains `HOVER_REVEALED_USER_2`.++ ---++ # Group 17: Scrolling++ ### 17.1 Scroll by pixels+ Navigate to `http://fixtures/scroll.html`. Scroll down approximately 1500 pixels.++ **Verify**: Snapshot or text contains `SCROLL_MIDDLE_MARKER` (the mid-page marker is now in view).++ ### 17.2 Scroll to footer+ Continue scrolling until the footer is visible (or scroll directly to the footer element).++ **Verify**: Snapshot contains `SCROLL_REACHED_FOOTER`.++ ---++ # Group 18: File Download++ ### 18.1 Download a file+ Download `http://fixtures/download-sample.txt` to a local file.++ **Verify**: The downloaded file exists and its content includes `DOWNLOAD_FILE_CONTENT_VERIFIED`.++ ---++ # Group 19: iFrame++ ### 19.1 Read iframe content+ Navigate to `http://fixtures/iframe.html` and extract content from inside the embedded same-origin iframe.++ **Verify**: The iframe's inner content includes `IFRAME_INNER_CONTENT_LOADED`.++ ### 19.2 Type into iframe input (native frame scope)+ Use `pinchtab frame '#content-frame'` to scope into the iframe, then `fill` the input with "Hello World" and `click` the Save button. Verify via a scoped `snap` — `text --full` doesn't pierce iframes. Reset with `pinchtab frame main` afterwards.++ **Verify**: Scoped snapshot contains `IFRAME_INPUT_RECEIVED_HELLO_WORLD`.++ ---++ # Group 20: Dialogs++ ### 20.1 Accept alert+ Navigate to `http://fixtures/alerts.html`. Click the "Click for Alert" button and dismiss the alert.++ **Verify**: The page result contains `DIALOG_ALERT_DISMISSED`.++ ### 20.2 Cancel confirm+ Click the "Click for Confirm" button and cancel the confirm dialog.++ **Verify**: The page result contains `DIALOG_CONFIRM_CANCELLED`.++ ---++ # Group 21: Async / awaitPromise++ ### 21.1 Await a promise-returning function+ Navigate to `http://fixtures/async.html`. The page exposes `window.fetchPayload()`, which returns a `Promise` that resolves after a short delay. Use `eval` to call it and retrieve the **resolved** value, not a Promise wrapper.++ **Verify**: The resolved value contains `ASYNC_PAYLOAD_READY_42`.++ ### 21.2 Await a promise resolving to an object+ On the same page, call `window.fetchUser()` and retrieve the resolved object so you can read a field from it.++ **Verify**: The resolved object's `name` field equals `ASYNC_USER_NAME_ADA`.++ ---++ # Group 22: Mouse Drag & Drop++ ### 22.1 Drag a piece into Zone A+ Navigate to `http://fixtures/drag.html`. The page contains a draggable square (`#piece`) and three target zones (`#zone-a`, `#zone-b`, `#zone-c`). Drag the piece so its center ends up over **Zone A**.++ **Verify**: The page shows `LAST_DROP=DROP_ZONE_A_OK`.++ ### 22.2 Drag to Zone B, then Zone C+ Without reloading the page, drag the piece next into **Zone B**, and then into **Zone C**. The page records an ordered drop sequence.++ **Verify**: The page shows `DROP_SEQUENCE=DROP_ZONE_A_OK,DROP_ZONE_B_OK,DROP_ZONE_C_OK` (all three drops in order).++ ---++ # Group 23: Async / Loading state++ ### 23.1 Wait for async content+ Navigate to `http://fixtures/loading.html`. The page shows a spinner and the text "Loading..." for ~1.5 seconds, then replaces it with the final payload. Wait for the final content to appear before reporting completion.++ **Verify**: The final snapshot contains `VERIFY_LOADING_COMPLETE_88888`.++ ---++ # Group 24: Keyboard events++ ### 24.1 Press Escape+ Navigate to `http://fixtures/keyboard.html` (it auto-focuses an input so keyboard events land). Press the Escape key.++ **Verify**: Page text contains `KEYBOARD_ESCAPE_PRESSED`.++ ### 24.2 Press 'a' then Enter+ Without reloading, press the `a` key, then the `Enter` key.++ **Verify**: Page text now contains all three markers in order: `KEYBOARD_ESCAPE_PRESSED`, `KEYBOARD_KEY_A_PRESSED`, `KEYBOARD_ENTER_PRESSED`.++ ---++ # Group 25: Tab panels++ ### 25.1 Switch to the Settings tab+ Navigate to `http://fixtures/tabs.html`. The Profile tab is shown by default. Click the Settings tab.++ **Verify**: After the click, the page content includes `TAB_SETTINGS_CONTENT` (and the Profile content is no longer visible).++ ### 25.2 Switch to the Billing tab+ Click the Billing tab.++ **Verify**: Page content now includes `TAB_BILLING_CONTENT`.++ ---++ # Group 26: Accordion++ ### 26.1 Open section A+ Navigate to `http://fixtures/accordion.html`. Click the header for Section A to expand it.++ **Verify**: Page text includes `ACCORDION_SECTION_A_OPEN`.++ ### 26.2 Open section B+ Click the header for Section B. Because the accordion is exclusive-expand, Section A should close.++ **Verify**: Page text includes `ACCORDION_SECTION_B_OPEN`, and Section A's `aria-expanded` attribute is now `"false"` (use `eval` to check the attribute if needed).++ ---++ # Group 27: Contenteditable editor++ ### 27.1 Type into the rich-text editor+ Navigate to `http://fixtures/editor.html`. Type the text `Hello rich text` into the `#editor` div. `contenteditable` elements don't have `.value`, so use `type` (keyboard events), not `fill`.++ **Verify**: Page text contains `EDITOR_CHARS=15` and the mirror shows `Hello rich text`.++ ### 27.2 Commit by pressing Enter+ Press Enter (the editor intercepts Enter to commit the current buffer to a separate marker).++ **Verify**: Page text contains `EDITOR_COMMITTED=Hello rich text`.++ ---++ # Group 28: Range slider++ ### 28.1 Move slider into HIGH bucket+ Navigate to `http://fixtures/range.html`. Set the `#volume` range slider to 90.++ **Verify**: Page text contains both `RANGE_VALUE_90` and `BUCKET_HIGH`.++ ### 28.2 Move slider into LOW bucket+ Without reloading, change the slider to 10.++ **Verify**: Page text now contains `RANGE_VALUE_10` and `BUCKET_LOW`.++ ---++ # Group 29: Pagination++ ### 29.1 Advance to page 2+ Navigate to `http://fixtures/pagination.html`. Click the Next button.++ **Verify**: Page text contains `PAGE_2_FIRST_ITEM` and `PAGE_2_OF_3`.++ ### 29.2 Reach the last page; Next disabled+ Click Next once more.++ **Verify**: Page text contains `PAGE_3_FIRST_ITEM` and `PAGE_3_OF_3`, and the Next button's `disabled` property is `true` (use `eval` to check).++ ---++ # Group 30: Custom dropdown menu++ ### 30.1 Pick the "Beta" item+ Navigate to `http://fixtures/dropdown.html`. The page has a custom dropdown — click the toggle button (`#dropdown-toggle`) to open the menu, then click the "Beta" item.++ **Verify**: Page text contains `DROPDOWN_SELECTED=BETA`.++ ### 30.2 Reopen and pick "Gamma"+ Reopen the dropdown and select "Gamma". (Each selection auto-closes the menu, so you need to click the toggle again before selecting.)++ **Verify**: Page text contains `DROPDOWN_SELECTED=GAMMA`.++ ---++ # Group 31: Nested iframes (3 levels deep)++ ### 31.1 Click a button in the deepest frame+ Navigate to `http://fixtures/iframe-nested.html`. The outer page embeds `#level-2`, which in turn embeds `#level-3`, which contains `#deep-button`. Drill through the two frame hops and click the button. Verify via a scoped `snap` before resetting to `main`.++ **Verify**: Scoped snapshot contains `DEEP_CLICKED=YES_LEVEL_3`.++ ---++ # Group 32: Dynamic iframe (inserted after load)++ ### 32.1 Wait for a late iframe, then interact with it+ Navigate to `http://fixtures/iframe-dynamic.html`. The iframe is inserted ~1.2 s after load (use `wait --text "IFRAME_DYNAMIC_ATTACHED"`). Then scope into `#late-frame`, fill `#iframe-input` with "Late World", click `#iframe-submit`, and verify the inner result marker.++ **Verify**: Scoped snapshot contains `IFRAME_INPUT_RECEIVED_LATE_WORLD`.++ ---++ # Group 33: srcdoc iframe++ ### 33.1 Interact with an inline-content iframe+ Navigate to `http://fixtures/iframe-srcdoc.html`. The iframe's content is inlined via `srcdoc`; it has no `src` URL. Scope into `#srcdoc-frame`, fill `#inline-input` with `srcdoc`, click `#inline-submit`, and verify.++ **Verify**: Scoped snapshot contains `INLINE_RECEIVED_SRCDOC`.++ ---++ # Group 34: Sandboxed iframe++ ### 34.1 Click inside a sandboxed iframe+ Navigate to `http://fixtures/iframe-sandbox.html`. The iframe has `sandbox="allow-scripts allow-same-origin"`. Scope into `#sandboxed`, click `#sandbox-button`, and verify.++ **Verify**: Scoped snapshot contains `SANDBOX_CLICKED=YES`.++ ---++ # Group 35: Long-form article (Medium/Substack style)++ ### 35.1 Read the article with default `text`+ Navigate to `http://fixtures/article.html`. Extract the page's main content using the default `text` (Readability) mode. Default mode is the right choice for article-style pages.++ **Verify**: The extracted text contains both `ARTICLE_PUBLISHED_2026_04_15` and `ARTICLE_WORD_COUNT_MARKER_323` (both inside the article body — Readability keeps them).++ ### 35.2 See the chrome that Readability drops+ Re-extract with `text --full` and confirm the footer is included this time.++ **Verify**: `--full` output contains `FOOTER_COPYRIGHT_MARKER` (which the default mode trims).++ ---++ # Group 36: Search results page (SERP)++ ### 36.1 Find a specific result by id+ Navigate to `http://fixtures/serp.html`. The page has 6 result cards (`#r-1`..`#r-6`). Extract just the third card's content. A scoped snapshot (`snap --selector "#r-3"`) is the direct path.++ **Verify**: The scoped output contains `RESULT_3_TITLE` and `RESULT_3_SNIPPET_MARKER`.++ ### 36.2 Count all result cards+ Use a full text extraction to verify all six results are present in one pass.++ **Verify**: Output contains all of `RESULT_1_TITLE` through `RESULT_6_TITLE` and the summary `SERP_RESULT_COUNT_6`. Default Readability trims SERPs — use `text --full`.++ ---++ # Group 37: Q&A thread (Stack-Overflow style)++ ### 37.1 Find the accepted answer id+ Navigate to `http://fixtures/qa.html`. The accepted answer carries `data-accepted="true"` on its `<div>` wrapper. Use `eval` to return that element's `id`.++ **Verify**: The `eval` result equals `"a-2"`.++ ### 37.2 Extract the accepted answer's body+ Scope a snapshot to `#a-2` and verify the body content.++ **Verify**: Scoped snapshot contains `ANSWER_2_BODY_MARKER` and `ACCEPTED_ANSWER_ID_A2`.++ ---++ # Group 38: Pricing table++ ### 38.1 Read just the Pro plan+ Navigate to `http://fixtures/pricing.html`. The page has three plan cards (`#plan-free`, `#plan-pro`, `#plan-enterprise`). Read only the Pro plan using a scoped snapshot.++ **Verify**: Scoped snapshot contains `PLAN_PRO_PRICE_29` and `PLAN_PRO_LIMIT_5000_requests per day`.++ ### 38.2 Compare all three plans+ Extract the full page (pricing grids are a Readability anti-pattern — reach for `--full`).++ **Verify**: Output contains `PLAN_FREE_PRICE_0`, `PLAN_PRO_PRICE_29`, and `PLAN_ENTERPRISE_PRICE_CUSTOM`.+- For each completed step:+ 1. record the observed answer with:+ - ./scripts/record-step.sh --report-file /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/results/golden_report.json --type pinchtab <group> <step> answer "<what you saw>" "notes"+ 2. immediately verify it with:+ - ./scripts/verify-step.sh --report-file /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/results/golden_report.json --type pinchtab <group> <step> <pass|fail|skip> "verification notes"+- If a step cannot be completed, record fail or skip in the same report.+- Do not leave answered steps pending verification.+- Keep commands concise. Prefer rg/sed/cat only when you must inspect a specific file.+- After the skill step, begin actual benchmark execution immediately.+- Start with this bootstrap command sequence before attempting the selected steps:+1. ./scripts/pt health+2. ./scripts/pt tab+3. export PINCHTAB_TAB=$(./scripts/pt nav http://fixtures/)+4. ./scripts/pt snap -i -c+- After the bootstrap, immediately execute Group 0 step 1.+- Subset selection:+ - Execute the full benchmark task set.+- Finish when all selected steps are executed or when you are blocked.++Your final answer should briefly summarize completion status and the main blockers, if any.\ No newline at end of filediff --git a/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-browser-all.txt b/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-browser-all.txtnew file mode 100644index 00000000..c5815635--- /dev/null+++ b/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-browser-all.txt@@ -0,0 +1,691 @@+Work in this repo: /sessions/eager-determined-cannon/mnt/pinchtab++Benchmark lane: agent-browser execution.++Requirements:+- Follow a linear execution flow: skill once, selected groups, execute, record, verify, continue.+- Do not read README, browse directories, or inspect unrelated files unless a command path is missing.+- Read /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/setup-agent-browser.md exactly once before acting, then load the official agent-browser skill exactly once with `./scripts/ab skills get agent-browser --full`.+- Tool wrapper:+ - ./scripts/ab+- Workflow summary:+ - use only ./scripts/ab for browser actions+ - keep the shared benchmark session across commands+ - use snapshot -i -c for actionable refs+ - use fresh refs after any navigation or DOM change+ - reuse the current browser state instead of reopening pages unless needed+- Adapter notes:+ - after navigation-triggering clicks, verify the resulting page with a fresh snapshot/text read instead of trusting the click response alone+ - do not assume every command returns JSON; parse only when the command actually returned JSON+ - for downloads, inspect returned content instead of assuming a file already exists locally+ - this environment uses BSD/macOS userland tools; avoid GNU-only flags such as "head -n -1"+- Task scope:+ Use the benchmark run index at /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/benchmark-run/index.md plus all group files in /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/benchmark-run.++ # Benchmark Run++ Natural-language benchmark task set for the agent-driven lanes.++ Read this index first, then read the relevant group files.++ ## How to use this directory++ - `setup-pinchtab.md` and `setup-agent-browser.md` define lane-specific setup, wrapper usage, recording, and Group 0 behavior.+ - This directory defines the shared task groups.+ - Each group lives in its own file: `group-00.md` through `group-38.md`.+ - When running a subset, read only the selected group files.+ - When running the full suite, proceed in numeric order.++ ## Group Files++ - `group-00.md` — Setup & Diagnosis+ - `group-01.md` — Reading & Extracting Real Content+ - `group-02.md` — Search & Dynamic Interaction+ - `group-03.md` — Complex Form+ - `group-04.md` — SPA State Management+ - `group-05.md` — Login Flow+ - `group-06.md` — Multi-Step E-commerce+ - `group-07.md` — Content + Interaction Combined+ - `group-08.md` — Error Handling+ - `group-09.md` — Export+ - `group-10.md` — Nested Interactions & Modal Dialogs+ - `group-11.md` — State Persistence & Page Reload+ - `group-12.md` — Multi-Page Navigation & Back Button+ - `group-13.md` — Form State & Multi-Step Submission+ - `group-14.md` — Dynamic Content Loading+ - `group-15.md` — Complex Data Extraction & Aggregation+ - `group-16.md` — Hover & Tooltips+ - `group-17.md` — Scrolling+ - `group-18.md` — File Download+ - `group-19.md` — iFrame+ - `group-20.md` — Dialogs+ - `group-21.md` — Async / awaitPromise+ - `group-22.md` — Mouse Drag & Drop+ - `group-23.md` — Async / Loading state+ - `group-24.md` — Keyboard events+ - `group-25.md` — Tab panels+ - `group-26.md` — Accordion+ - `group-27.md` — Contenteditable editor+ - `group-28.md` — Range slider+ - `group-29.md` — Pagination+ - `group-30.md` — Custom dropdown menu+ - `group-31.md` — Nested iframes (3 levels deep)+ - `group-32.md` — Dynamic iframe (inserted after load)+ - `group-33.md` — srcdoc iframe+ - `group-34.md` — Sandboxed iframe+ - `group-35.md` — Long-form article (Medium/Substack style)+ - `group-36.md` — Search results page (SERP)+ - `group-37.md` — Q&A thread (Stack-Overflow style)+ - `group-38.md` — Pricing table++ # Group 0: Setup & Diagnosis++ ### 0.1 Server reachable+ Check that the PinchTab server is healthy.++ **Verify**: Server responds with `status: ok`.++ ### 0.2 Auth is required+ Make a request to the server with a **wrong** token (`PINCHTAB_TOKEN=wrong-token ./scripts/pt health`) and confirm it is rejected. The `pt` wrapper always injects the benchmark token by default, so you must explicitly override it to test auth rejection.++ **Verify**: Response is HTTP 401 or contains `unauthorized`.++ ### 0.3 Auth works with token+ Repeat the same request WITH the bearer token and confirm it succeeds.++ **Verify**: Response is HTTP 200.++ ### 0.4 Instance available+ Confirm at least one Chrome instance is running. If none exist, start one.++ **Verify**: Health response shows `defaultInstance.status == "running"` (or after starting one, the new instance is running).++ ### 0.5 List existing tabs+ Get the current list of open tabs.++ **Verify**: A list (possibly empty) is returned without error.++ ### 0.6 Clean stale tabs+ If any tabs from previous runs are open, close them so the benchmark starts from a known state.++ **Verify**: After cleanup, the tab list is empty (or only contains a single about:blank tab).++ ### 0.7 Network reach to target+ Navigate to `http://fixtures/` and confirm the fixtures server is reachable from PinchTab.++ **Verify**: Navigate returns successfully and the page contains benchmark content.++ ### 0.8 Capture initial tab ID+ Save the tab ID returned by the navigate in 0.7. Use this tab ID for all subsequent tasks to avoid creating new tabs.++ **Verify**: A tab ID was captured and matches what `GET /tabs` reports.++ ---++ # Group 1: Reading & Extracting Real Content++ ### 1.1 Get the full list of categories from the wiki index+ Navigate to `http://fixtures/wiki.html` and extract all category names with their article counts.++ **Verify**: Can name at least 2 categories and their article counts (e.g. "Programming Languages: 12 articles").++ ### 1.2 Navigate by clicking a link+ From the wiki index, click the "Go (programming language)" link to navigate to the Go article.++ **Verify**: You are now on the Go article page (not the wiki index).++ ### 1.3 Extract structured data from a table+ On the Go article, read the infobox table and answer: Who designed Go, and what year did it first appear?++ **Verify**: Answer contains "Robert Griesemer" (or "Rob Pike" or "Ken Thompson") and "2009".++ ### 1.4 Count list items+ On the Go article, count how many key features are listed.++ **Verify**: Answer is 6 (verify against `FEATURE_COUNT_6`).++ ### 1.5 Read all article headlines from articles page+ Navigate to `http://fixtures/articles.html` and list all article titles.++ **Verify**: Found at least 3 articles including "The Future of Artificial Intelligence".++ ### 1.6 Read dashboard metrics+ Navigate to `http://fixtures/dashboard.html` and extract: Total Users, Revenue, and Conversion Rate.++ **Verify**: Found `24,582` users AND `$1,284,930` revenue.++ ---++ # Group 2: Search & Dynamic Interaction++ ### 2.1 Use wiki search to find a page+ On `http://fixtures/wiki.html`, search for "golang" using the search form. Do not navigate directly — use the search input.++ **Verify**: Ended up on the Go article page after search.++ ### 2.2 Search with no results+ On `http://fixtures/search.html`, search for something with no results (use "xyznonexistent").++ **Verify**: Page handled it gracefully (no crash, some response rendered).++ ### 2.3 Search for AI content+ On `http://fixtures/search.html`, search for "artificial intelligence" and verify a result appeared.++ **Verify**: Result contains "The Future of Artificial Intelligence".++ ---++ # Group 3: Complex Form++ ### 3.1 Fill and submit a complete form+ Navigate to `http://fixtures/form.html`. Complete the entire form:+ - Full name: "Agent Test User"+ - Email: "[email protected]"+ - Phone: "+44 20 9999 0000"+ - Country: United Kingdom+ - Subject: Technical Support+ - Message: "Testing PinchTab form automation"+ - Check newsletter+ - Set priority to High+ - Submit++ **Verify**: Form submitted successfully. Confirmation shows name "AGENT_TEST_USER".++ ### 3.2 Reset and refill+ After submitting, if the form is still accessible (or navigate back), verify you can identify the reset button.++ **Verify**: Reset/back button or form element found in snapshot.++ ---++ # Group 4: SPA State Management++ ### 4.1 Read initial app state+ Navigate to `http://fixtures/spa.html?reset=1` (the `?reset=1` query param clears any previous localStorage state so the SPA starts with its default 3 tasks). Read the current task list — how many tasks exist, how many are active vs done?++ **Verify**: Found 3 total, 2 active, 1 done (verify `TASK_STATS_TOTAL_3_ACTIVE_2_DONE_1`).++ ### 4.2 Add a new high-priority task+ Add a task called "Automate deployment" with high priority.++ **Verify**: Task appeared in the list (`TASK_ADDED_AUTOMATE_DEPLOYMENT_PRIORITY_HIGH`).++ ### 4.3 Delete a task+ Delete the task titled "Write benchmark tests".++ **Verify**: Task count changed (went from 4 to 3).++ ---++ # Group 5: Login Flow++ ### 5.1 Attempt login with wrong credentials+ Navigate to `http://fixtures/login.html`. Try to log in with username "admin" and password "wrong".++ **Verify**: Error message appeared (`INVALID_CREDENTIALS_ERROR`).++ ### 5.2 Login successfully+ Clear the form and log in with username "benchmark" / password "test456".++ **Verify**: Dashboard appeared after login (`VERIFY_LOGIN_SUCCESS_DASHBOARD`).++ ---++ # Group 6: Multi-Step E-commerce++ ### 6.1 Research products before buying+ Navigate to `http://fixtures/ecommerce.html`. List all available products with their prices. Which product is out of stock?++ **Verify**: Found Wireless Headphones ($149.99), Smart Watch Pro ($299.99), Portable Charger ($49.99). Mechanical Keyboard is out of stock.++ ### 6.2 Add two items and verify total+ Add Wireless Headphones and Smart Watch Pro to cart. What is the total?++ **Verify**: Cart total is $449.98 (149.99 + 299.99).++ ### 6.3 Complete checkout+ Click checkout to complete the order.++ **Verify**: Order confirmation shows (`VERIFY_CHECKOUT_SUCCESS_ORDER`).++ ---++ # Group 7: Content + Interaction Combined++ ### 7.1 Read and comment on Go article+ Navigate to `http://fixtures/wiki-go.html`. Read the article, then post a comment with rating 5 stars and text "Excellent reference".++ **Verify**: Comment posted (`COMMENT_POSTED_RATING_5_TEXT_RECEIVED`).++ ### 7.2 Cross-page research task+ Navigate to wiki index, find which category has the most articles, then navigate to one of its listed items.++ **Verify**: Successfully navigated to at least one article page after reading the category counts.++ ---++ # Group 8: Error Handling++ ### 8.1 Handle 404 gracefully+ Try to navigate to a page that doesn't exist: `http://fixtures/missing-page-abc.html`.++ **Verify**: Got a response (404 or error), no crash, server still responsive after.++ ### 8.2 Handle missing element gracefully+ On any page, try to click an element with ID `#fake-button-that-does-not-exist`.++ **Verify**: Got a clear error message, not a crash or hang.++ ---++ # Group 9: Export++ ### 9.1 Screenshot a complex page+ Navigate to `http://fixtures/dashboard.html` and take a screenshot.++ **Verify**: Screenshot generated (file saved or base64 returned).++ ### 9.2 Export a page as PDF+ Export the dashboard as a PDF.++ **Verify**: PDF generated.++ ---++ # Group 10: Nested Interactions & Modal Dialogs++ ### 10.1 Open and interact with modal on dashboard+ Navigate to `http://fixtures/dashboard.html`. Find and click the Settings button (selector: `#settings-btn`) to open the modal dialog.++ **Verify**: Modal appeared — snapshot contains "Dashboard Settings".++ ### 10.2 Modify settings and close modal+ In the modal, select "Dark" from the theme dropdown (`#theme-select`), then click the Save button (`#modal-save`). After the modal closes, check the page content.++ **Verify**: Page contains `THEME_DARK_APPLIED`.++ ---++ # Group 11: State Persistence & Page Reload++ ### 11.1 Add an item and verify after page reload+ Navigate to `http://fixtures/spa.html?reset=1` (starts with clean state). Add a task titled exactly "Persistent Task Test". Then navigate away to `http://fixtures/` and back to `http://fixtures/spa.html` (WITHOUT the reset param) to verify localStorage persistence.++ **Verify**: After reload, the task still appears in the list (`TASK_PERSISTENT_TEST_FOUND_AFTER_RELOAD`).++ ### 11.2 Logout and log back in+ From the logged-in dashboard, click Sign Out to log out. Then log in again with username "benchmark" / password "test456".++ **Verify**: Successfully logged back in and dashboard shows `SESSION_RENEWED`.++ ---++ # Group 12: Multi-Page Navigation & Back Button++ ### 12.1 Navigate through multiple pages and return+ Starting from `http://fixtures/`, navigate to wiki → Go article → back to wiki → back to home.++ **Verify**: Successfully returned to home page (title contains "Benchmark" or "Home").++ ### 12.2 Compare data across pages+ Navigate to wiki.html, note the total article count from categories. Navigate to articles.html, count articles there. Compare totals.++ **Verify**: Can report totals from both pages and explain difference (`COMPARISON_DATA_FOUND`).++ ---++ # Group 13: Form State & Multi-Step Submission++ ### 13.1 Submit form without email+ Navigate to `http://fixtures/form.html`. Fill only the name field ("Validator Test"), leave email blank, click Submit. The browser's native required-field validation will prevent submission.++ **Verify**: Submission blocked (form stays open, no success message shown).++ ### 13.2 Submit form without optional phone field+ Fill the form with: name "No Phone User", email "[email protected]", country "de", subject "feedback". Leave the phone field empty. Submit.++ **Verify**: Submission succeeded and page shows `OPTIONAL_FIELD_SKIPPED_SUCCESS`.++ ---++ # Group 14: Dynamic Content Loading++ ### 14.1 Load more products+ Navigate to `http://fixtures/ecommerce.html`. Find and click the "Load More Products" button to reveal additional products.++ **Verify**: Additional products appeared (`ADDITIONAL_PRODUCTS_LOADED`).++ ### 14.2 Add a lazy-loaded product to cart+ After loading more products, add product #5 (USB-C Cable) to the cart.++ **Verify**: Cart shows the lazy-loaded item (`CART_UPDATED_WITH_LAZY_PRODUCT`).++ ---++ # Group 15: Complex Data Extraction & Aggregation++ ### 15.1 Extract and sum financial data+ Navigate to `http://fixtures/dashboard.html`. Extract revenue and profit values, calculate profit margin.++ **Verify**: Correctly calculated: profit_margin = (profit / revenue) * 100 (`PROFIT_MARGIN_CALCULATED`).++ ### 15.2 Build comparison table from multiple sources+ Visit these 3 pages and compare their feature counts and key features:+ - `http://fixtures/wiki-go.html` (Go: 6 features)+ - `http://fixtures/wiki-python.html` (Python: 7 features)+ - `http://fixtures/wiki-rust.html` (Rust: 5 features)++ Report which language has the most features and name 1 feature unique to each.++ **Verify**: Response is factually correct AND wiki-python.html contains `COMPARISON_TABLE_BUILT`.++ ---++ # Group 16: Hover & Tooltips++ ### 16.1 Hover reveals hidden content+ Navigate to `http://fixtures/hovers.html`. Hover over the first avatar to reveal its hidden info.++ **Verify**: After hovering, snapshot contains `HOVER_REVEALED_USER_1`.++ ### 16.2 Hover swap+ Hover over the second avatar.++ **Verify**: Snapshot contains `HOVER_REVEALED_USER_2`.++ ---++ # Group 17: Scrolling++ ### 17.1 Scroll by pixels+ Navigate to `http://fixtures/scroll.html`. Scroll down approximately 1500 pixels.++ **Verify**: Snapshot or text contains `SCROLL_MIDDLE_MARKER` (the mid-page marker is now in view).++ ### 17.2 Scroll to footer+ Continue scrolling until the footer is visible (or scroll directly to the footer element).++ **Verify**: Snapshot contains `SCROLL_REACHED_FOOTER`.++ ---++ # Group 18: File Download++ ### 18.1 Download a file+ Download `http://fixtures/download-sample.txt` to a local file.++ **Verify**: The downloaded file exists and its content includes `DOWNLOAD_FILE_CONTENT_VERIFIED`.++ ---++ # Group 19: iFrame++ ### 19.1 Read iframe content+ Navigate to `http://fixtures/iframe.html` and extract content from inside the embedded same-origin iframe.++ **Verify**: The iframe's inner content includes `IFRAME_INNER_CONTENT_LOADED`.++ ### 19.2 Type into iframe input (native frame scope)+ Use `pinchtab frame '#content-frame'` to scope into the iframe, then `fill` the input with "Hello World" and `click` the Save button. Verify via a scoped `snap` — `text --full` doesn't pierce iframes. Reset with `pinchtab frame main` afterwards.++ **Verify**: Scoped snapshot contains `IFRAME_INPUT_RECEIVED_HELLO_WORLD`.++ ---++ # Group 20: Dialogs++ ### 20.1 Accept alert+ Navigate to `http://fixtures/alerts.html`. Click the "Click for Alert" button and dismiss the alert.++ **Verify**: The page result contains `DIALOG_ALERT_DISMISSED`.++ ### 20.2 Cancel confirm+ Click the "Click for Confirm" button and cancel the confirm dialog.++ **Verify**: The page result contains `DIALOG_CONFIRM_CANCELLED`.++ ---++ # Group 21: Async / awaitPromise++ ### 21.1 Await a promise-returning function+ Navigate to `http://fixtures/async.html`. The page exposes `window.fetchPayload()`, which returns a `Promise` that resolves after a short delay. Use `eval` to call it and retrieve the **resolved** value, not a Promise wrapper.++ **Verify**: The resolved value contains `ASYNC_PAYLOAD_READY_42`.++ ### 21.2 Await a promise resolving to an object+ On the same page, call `window.fetchUser()` and retrieve the resolved object so you can read a field from it.++ **Verify**: The resolved object's `name` field equals `ASYNC_USER_NAME_ADA`.++ ---++ # Group 22: Mouse Drag & Drop++ ### 22.1 Drag a piece into Zone A+ Navigate to `http://fixtures/drag.html`. The page contains a draggable square (`#piece`) and three target zones (`#zone-a`, `#zone-b`, `#zone-c`). Drag the piece so its center ends up over **Zone A**.++ **Verify**: The page shows `LAST_DROP=DROP_ZONE_A_OK`.++ ### 22.2 Drag to Zone B, then Zone C+ Without reloading the page, drag the piece next into **Zone B**, and then into **Zone C**. The page records an ordered drop sequence.++ **Verify**: The page shows `DROP_SEQUENCE=DROP_ZONE_A_OK,DROP_ZONE_B_OK,DROP_ZONE_C_OK` (all three drops in order).++ ---++ # Group 23: Async / Loading state++ ### 23.1 Wait for async content+ Navigate to `http://fixtures/loading.html`. The page shows a spinner and the text "Loading..." for ~1.5 seconds, then replaces it with the final payload. Wait for the final content to appear before reporting completion.++ **Verify**: The final snapshot contains `VERIFY_LOADING_COMPLETE_88888`.++ ---++ # Group 24: Keyboard events++ ### 24.1 Press Escape+ Navigate to `http://fixtures/keyboard.html` (it auto-focuses an input so keyboard events land). Press the Escape key.++ **Verify**: Page text contains `KEYBOARD_ESCAPE_PRESSED`.++ ### 24.2 Press 'a' then Enter+ Without reloading, press the `a` key, then the `Enter` key.++ **Verify**: Page text now contains all three markers in order: `KEYBOARD_ESCAPE_PRESSED`, `KEYBOARD_KEY_A_PRESSED`, `KEYBOARD_ENTER_PRESSED`.++ ---++ # Group 25: Tab panels++ ### 25.1 Switch to the Settings tab+ Navigate to `http://fixtures/tabs.html`. The Profile tab is shown by default. Click the Settings tab.++ **Verify**: After the click, the page content includes `TAB_SETTINGS_CONTENT` (and the Profile content is no longer visible).++ ### 25.2 Switch to the Billing tab+ Click the Billing tab.++ **Verify**: Page content now includes `TAB_BILLING_CONTENT`.++ ---++ # Group 26: Accordion++ ### 26.1 Open section A+ Navigate to `http://fixtures/accordion.html`. Click the header for Section A to expand it.++ **Verify**: Page text includes `ACCORDION_SECTION_A_OPEN`.++ ### 26.2 Open section B+ Click the header for Section B. Because the accordion is exclusive-expand, Section A should close.++ **Verify**: Page text includes `ACCORDION_SECTION_B_OPEN`, and Section A's `aria-expanded` attribute is now `"false"` (use `eval` to check the attribute if needed).++ ---++ # Group 27: Contenteditable editor++ ### 27.1 Type into the rich-text editor+ Navigate to `http://fixtures/editor.html`. Type the text `Hello rich text` into the `#editor` div. `contenteditable` elements don't have `.value`, so use `type` (keyboard events), not `fill`.++ **Verify**: Page text contains `EDITOR_CHARS=15` and the mirror shows `Hello rich text`.++ ### 27.2 Commit by pressing Enter+ Press Enter (the editor intercepts Enter to commit the current buffer to a separate marker).++ **Verify**: Page text contains `EDITOR_COMMITTED=Hello rich text`.++ ---++ # Group 28: Range slider++ ### 28.1 Move slider into HIGH bucket+ Navigate to `http://fixtures/range.html`. Set the `#volume` range slider to 90.++ **Verify**: Page text contains both `RANGE_VALUE_90` and `BUCKET_HIGH`.++ ### 28.2 Move slider into LOW bucket+ Without reloading, change the slider to 10.++ **Verify**: Page text now contains `RANGE_VALUE_10` and `BUCKET_LOW`.++ ---++ # Group 29: Pagination++ ### 29.1 Advance to page 2+ Navigate to `http://fixtures/pagination.html`. Click the Next button.++ **Verify**: Page text contains `PAGE_2_FIRST_ITEM` and `PAGE_2_OF_3`.++ ### 29.2 Reach the last page; Next disabled+ Click Next once more.++ **Verify**: Page text contains `PAGE_3_FIRST_ITEM` and `PAGE_3_OF_3`, and the Next button's `disabled` property is `true` (use `eval` to check).++ ---++ # Group 30: Custom dropdown menu++ ### 30.1 Pick the "Beta" item+ Navigate to `http://fixtures/dropdown.html`. The page has a custom dropdown — click the toggle button (`#dropdown-toggle`) to open the menu, then click the "Beta" item.++ **Verify**: Page text contains `DROPDOWN_SELECTED=BETA`.++ ### 30.2 Reopen and pick "Gamma"+ Reopen the dropdown and select "Gamma". (Each selection auto-closes the menu, so you need to click the toggle again before selecting.)++ **Verify**: Page text contains `DROPDOWN_SELECTED=GAMMA`.++ ---++ # Group 31: Nested iframes (3 levels deep)++ ### 31.1 Click a button in the deepest frame+ Navigate to `http://fixtures/iframe-nested.html`. The outer page embeds `#level-2`, which in turn embeds `#level-3`, which contains `#deep-button`. Drill through the two frame hops and click the button. Verify via a scoped `snap` before resetting to `main`.++ **Verify**: Scoped snapshot contains `DEEP_CLICKED=YES_LEVEL_3`.++ ---++ # Group 32: Dynamic iframe (inserted after load)++ ### 32.1 Wait for a late iframe, then interact with it+ Navigate to `http://fixtures/iframe-dynamic.html`. The iframe is inserted ~1.2 s after load (use `wait --text "IFRAME_DYNAMIC_ATTACHED"`). Then scope into `#late-frame`, fill `#iframe-input` with "Late World", click `#iframe-submit`, and verify the inner result marker.++ **Verify**: Scoped snapshot contains `IFRAME_INPUT_RECEIVED_LATE_WORLD`.++ ---++ # Group 33: srcdoc iframe++ ### 33.1 Interact with an inline-content iframe+ Navigate to `http://fixtures/iframe-srcdoc.html`. The iframe's content is inlined via `srcdoc`; it has no `src` URL. Scope into `#srcdoc-frame`, fill `#inline-input` with `srcdoc`, click `#inline-submit`, and verify.++ **Verify**: Scoped snapshot contains `INLINE_RECEIVED_SRCDOC`.++ ---++ # Group 34: Sandboxed iframe++ ### 34.1 Click inside a sandboxed iframe+ Navigate to `http://fixtures/iframe-sandbox.html`. The iframe has `sandbox="allow-scripts allow-same-origin"`. Scope into `#sandboxed`, click `#sandbox-button`, and verify.++ **Verify**: Scoped snapshot contains `SANDBOX_CLICKED=YES`.++ ---++ # Group 35: Long-form article (Medium/Substack style)++ ### 35.1 Read the article with default `text`+ Navigate to `http://fixtures/article.html`. Extract the page's main content using the default `text` (Readability) mode. Default mode is the right choice for article-style pages.++ **Verify**: The extracted text contains both `ARTICLE_PUBLISHED_2026_04_15` and `ARTICLE_WORD_COUNT_MARKER_323` (both inside the article body — Readability keeps them).++ ### 35.2 See the chrome that Readability drops+ Re-extract with `text --full` and confirm the footer is included this time.++ **Verify**: `--full` output contains `FOOTER_COPYRIGHT_MARKER` (which the default mode trims).++ ---++ # Group 36: Search results page (SERP)++ ### 36.1 Find a specific result by id+ Navigate to `http://fixtures/serp.html`. The page has 6 result cards (`#r-1`..`#r-6`). Extract just the third card's content. A scoped snapshot (`snap --selector "#r-3"`) is the direct path.++ **Verify**: The scoped output contains `RESULT_3_TITLE` and `RESULT_3_SNIPPET_MARKER`.++ ### 36.2 Count all result cards+ Use a full text extraction to verify all six results are present in one pass.++ **Verify**: Output contains all of `RESULT_1_TITLE` through `RESULT_6_TITLE` and the summary `SERP_RESULT_COUNT_6`. Default Readability trims SERPs — use `text --full`.++ ---++ # Group 37: Q&A thread (Stack-Overflow style)++ ### 37.1 Find the accepted answer id+ Navigate to `http://fixtures/qa.html`. The accepted answer carries `data-accepted="true"` on its `<div>` wrapper. Use `eval` to return that element's `id`.++ **Verify**: The `eval` result equals `"a-2"`.++ ### 37.2 Extract the accepted answer's body+ Scope a snapshot to `#a-2` and verify the body content.++ **Verify**: Scoped snapshot contains `ANSWER_2_BODY_MARKER` and `ACCEPTED_ANSWER_ID_A2`.++ ---++ # Group 38: Pricing table++ ### 38.1 Read just the Pro plan+ Navigate to `http://fixtures/pricing.html`. The page has three plan cards (`#plan-free`, `#plan-pro`, `#plan-enterprise`). Read only the Pro plan using a scoped snapshot.++ **Verify**: Scoped snapshot contains `PLAN_PRO_PRICE_29` and `PLAN_PRO_LIMIT_5000_requests per day`.++ ### 38.2 Compare all three plans+ Extract the full page (pricing grids are a Readability anti-pattern — reach for `--full`).++ **Verify**: Output contains `PLAN_FREE_PRICE_0`, `PLAN_PRO_PRICE_29`, and `PLAN_ENTERPRISE_PRICE_CUSTOM`.+- For each completed step:+ 1. record the observed answer with:+ - ./scripts/record-step.sh --report-file /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/results/golden_report.json --type agent-browser <group> <step> answer "<what you saw>" "notes"+ 2. immediately verify it with:+ - ./scripts/verify-step.sh --report-file /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/results/golden_report.json --type agent-browser <group> <step> <pass|fail|skip> "verification notes"+- If a step cannot be completed, record fail or skip in the same report.+- Do not leave answered steps pending verification.+- Keep commands concise. Prefer rg/sed/cat only when you must inspect a specific file.+- After the skill step, begin actual benchmark execution immediately.+- Start with this bootstrap command sequence before attempting the selected steps:+1. ./scripts/ab skills get agent-browser --full+2. ./scripts/ab open http://fixtures/+3. ./scripts/ab snapshot -i -c+- After the bootstrap, immediately execute Group 0 step 1.+- Subset selection:+ - Execute the full benchmark task set.+- Finish when all selected steps are executed or when you are blocked.++Your final answer should briefly summarize completion status and the main blockers, if any.\ No newline at end of filediff --git a/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-browser-groups-0-1.txt b/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-browser-groups-0-1.txtnew file mode 100644index 00000000..432567f5--- /dev/null+++ b/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-browser-groups-0-1.txt@@ -0,0 +1,152 @@+Work in this repo: /sessions/eager-determined-cannon/mnt/pinchtab++Benchmark lane: agent-browser execution.++Requirements:+- Follow a linear execution flow: skill once, selected groups, execute, record, verify, continue.+- Do not read README, browse directories, or inspect unrelated files unless a command path is missing.+- Read /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/setup-agent-browser.md exactly once before acting, then load the official agent-browser skill exactly once with `./scripts/ab skills get agent-browser --full`.+- Tool wrapper:+ - ./scripts/ab+- Workflow summary:+ - use only ./scripts/ab for browser actions+ - keep the shared benchmark session across commands+ - use snapshot -i -c for actionable refs+ - use fresh refs after any navigation or DOM change+ - reuse the current browser state instead of reopening pages unless needed+- Adapter notes:+ - after navigation-triggering clicks, verify the resulting page with a fresh snapshot/text read instead of trusting the click response alone+ - do not assume every command returns JSON; parse only when the command actually returned JSON+ - for downloads, inspect returned content instead of assuming a file already exists locally+ - this environment uses BSD/macOS userland tools; avoid GNU-only flags such as "head -n -1"+- Task scope:+ Use only this selected task subset for agent-browser: Group 0 from /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/setup-agent-browser.md, Groups 1+ from /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/benchmark-run.++ # Benchmark Run++ Natural-language benchmark task set for the agent-driven lanes.++ Read this index first, then read the relevant group files.++ ## How to use this directory++ - `setup-pinchtab.md` and `setup-agent-browser.md` define lane-specific setup, wrapper usage, recording, and Group 0 behavior.+ - This directory defines the shared task groups.+ - Each group lives in its own file: `group-00.md` through `group-38.md`.+ - When running a subset, read only the selected group files.+ - When running the full suite, proceed in numeric order.++ ## Group Files++ - `group-00.md` — Setup & Diagnosis+ - `group-01.md` — Reading & Extracting Real Content+ - `group-02.md` — Search & Dynamic Interaction+ - `group-03.md` — Complex Form+ - `group-04.md` — SPA State Management+ - `group-05.md` — Login Flow+ - `group-06.md` — Multi-Step E-commerce+ - `group-07.md` — Content + Interaction Combined+ - `group-08.md` — Error Handling+ - `group-09.md` — Export+ - `group-10.md` — Nested Interactions & Modal Dialogs+ - `group-11.md` — State Persistence & Page Reload+ - `group-12.md` — Multi-Page Navigation & Back Button+ - `group-13.md` — Form State & Multi-Step Submission+ - `group-14.md` — Dynamic Content Loading+ - `group-15.md` — Complex Data Extraction & Aggregation+ - `group-16.md` — Hover & Tooltips+ - `group-17.md` — Scrolling+ - `group-18.md` — File Download+ - `group-19.md` — iFrame+ - `group-20.md` — Dialogs+ - `group-21.md` — Async / awaitPromise+ - `group-22.md` — Mouse Drag & Drop+ - `group-23.md` — Async / Loading state+ - `group-24.md` — Keyboard events+ - `group-25.md` — Tab panels+ - `group-26.md` — Accordion+ - `group-27.md` — Contenteditable editor+ - `group-28.md` — Range slider+ - `group-29.md` — Pagination+ - `group-30.md` — Custom dropdown menu+ - `group-31.md` — Nested iframes (3 levels deep)+ - `group-32.md` — Dynamic iframe (inserted after load)+ - `group-33.md` — srcdoc iframe+ - `group-34.md` — Sandboxed iframe+ - `group-35.md` — Long-form article (Medium/Substack style)+ - `group-36.md` — Search results page (SERP)+ - `group-37.md` — Q&A thread (Stack-Overflow style)+ - `group-38.md` — Pricing table++ ## Group 0: Setup & Diagnosis (agent-browser lane)++ ### 0.1 Open fixtures home+ Run `./scripts/ab open http://fixtures/`.++ **Verify**: Open succeeds and the home page loads.++ ### 0.2 Snapshot interactive refs+ Run `./scripts/ab snapshot -i -c` on the home page.++ **Verify**: Interactive refs are returned without error.++ ### 0.3 Session persists across commands+ Use the same wrapper session across multiple commands and confirm browser state persists.++ **Verify**: A follow-up action can use the existing page/session state.++ # Group 1: Reading & Extracting Real Content++ ### 1.1 Get the full list of categories from the wiki index+ Navigate to `http://fixtures/wiki.html` and extract all category names with their article counts.++ **Verify**: Can name at least 2 categories and their article counts (e.g. "Programming Languages: 12 articles").++ ### 1.2 Navigate by clicking a link+ From the wiki index, click the "Go (programming language)" link to navigate to the Go article.++ **Verify**: You are now on the Go article page (not the wiki index).++ ### 1.3 Extract structured data from a table+ On the Go article, read the infobox table and answer: Who designed Go, and what year did it first appear?++ **Verify**: Answer contains "Robert Griesemer" (or "Rob Pike" or "Ken Thompson") and "2009".++ ### 1.4 Count list items+ On the Go article, count how many key features are listed.++ **Verify**: Answer is 6 (verify against `FEATURE_COUNT_6`).++ ### 1.5 Read all article headlines from articles page+ Navigate to `http://fixtures/articles.html` and list all article titles.++ **Verify**: Found at least 3 articles including "The Future of Artificial Intelligence".++ ### 1.6 Read dashboard metrics+ Navigate to `http://fixtures/dashboard.html` and extract: Total Users, Revenue, and Conversion Rate.++ **Verify**: Found `24,582` users AND `$1,284,930` revenue.++ ---+- For each completed step:+ 1. record the observed answer with:+ - ./scripts/record-step.sh --report-file /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/results/golden_report.json --type agent-browser <group> <step> answer "<what you saw>" "notes"+ 2. immediately verify it with:+ - ./scripts/verify-step.sh --report-file /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/results/golden_report.json --type agent-browser <group> <step> <pass|fail|skip> "verification notes"+- If a step cannot be completed, record fail or skip in the same report.+- Do not leave answered steps pending verification.+- Keep commands concise. Prefer rg/sed/cat only when you must inspect a specific file.+- After the skill step, begin actual benchmark execution immediately.+- Start with this bootstrap command sequence before attempting the selected steps:+1. ./scripts/ab skills get agent-browser --full+2. ./scripts/ab open http://fixtures/+3. ./scripts/ab snapshot -i -c+- After the bootstrap, immediately execute Group 0 step 1.+- Subset selection:+ - Execute only these benchmark groups: 0, 1.+ - Do not attempt groups outside this subset.+ - Treat all other groups as out of scope for this run rather than as failures.+ - For the selected groups, execute every step in the group unless blocked.+- Finish when all selected steps are executed or when you are blocked.++Your final answer should briefly summarize completion status and the main blockers, if any.\ No newline at end of filediff --git a/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-groups-0-1.txt b/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-groups-0-1.txtnew file mode 100644index 00000000..7a301654--- /dev/null+++ b/tests/benchmark/cmd/api-runner/testdata/golden/prompt-agent-groups-0-1.txt@@ -0,0 +1,180 @@+Work in this repo: /sessions/eager-determined-cannon/mnt/pinchtab++Benchmark lane: PinchTab agent execution.++Requirements:+- Follow a linear execution flow: skill once, selected groups, execute, record, verify, continue.+- Do not read README, browse directories, or inspect unrelated files unless a command path is missing.+- Read /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/setup-pinchtab.md exactly once before acting.+- Tool wrapper:+ - ./scripts/pt+- Workflow summary:+ - use only ./scripts/pt for browser actions+ - keep one shared tab via PINCHTAB_TAB+ - use snap -i -c for actionable refs+ - use text or text --full for reading content+ - refresh refs after any navigation or DOM change+- Adapter notes:+ - a navigation-expected click can return "Error 409: unexpected page navigation ..."; treat that as likely success and verify with a fresh snapshot/text read+ - do not assume every command returns JSON; only parse JSON when the command actually returned JSON+ - the download endpoint returns JSON with base64 content in "data", not a local file path+ - this environment uses BSD/macOS userland tools; avoid GNU-only flags such as "head -n -1"+- Task scope:+ Use only this selected task subset from /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/benchmark-run:++ # Benchmark Run++ Natural-language benchmark task set for the agent-driven lanes.++ Read this index first, then read the relevant group files.++ ## How to use this directory++ - `setup-pinchtab.md` and `setup-agent-browser.md` define lane-specific setup, wrapper usage, recording, and Group 0 behavior.+ - This directory defines the shared task groups.+ - Each group lives in its own file: `group-00.md` through `group-38.md`.+ - When running a subset, read only the selected group files.+ - When running the full suite, proceed in numeric order.++ ## Group Files++ - `group-00.md` — Setup & Diagnosis+ - `group-01.md` — Reading & Extracting Real Content+ - `group-02.md` — Search & Dynamic Interaction+ - `group-03.md` — Complex Form+ - `group-04.md` — SPA State Management+ - `group-05.md` — Login Flow+ - `group-06.md` — Multi-Step E-commerce+ - `group-07.md` — Content + Interaction Combined+ - `group-08.md` — Error Handling+ - `group-09.md` — Export+ - `group-10.md` — Nested Interactions & Modal Dialogs+ - `group-11.md` — State Persistence & Page Reload+ - `group-12.md` — Multi-Page Navigation & Back Button+ - `group-13.md` — Form State & Multi-Step Submission+ - `group-14.md` — Dynamic Content Loading+ - `group-15.md` — Complex Data Extraction & Aggregation+ - `group-16.md` — Hover & Tooltips+ - `group-17.md` — Scrolling+ - `group-18.md` — File Download+ - `group-19.md` — iFrame+ - `group-20.md` — Dialogs+ - `group-21.md` — Async / awaitPromise+ - `group-22.md` — Mouse Drag & Drop+ - `group-23.md` — Async / Loading state+ - `group-24.md` — Keyboard events+ - `group-25.md` — Tab panels+ - `group-26.md` — Accordion+ - `group-27.md` — Contenteditable editor+ - `group-28.md` — Range slider+ - `group-29.md` — Pagination+ - `group-30.md` — Custom dropdown menu+ - `group-31.md` — Nested iframes (3 levels deep)+ - `group-32.md` — Dynamic iframe (inserted after load)+ - `group-33.md` — srcdoc iframe+ - `group-34.md` — Sandboxed iframe+ - `group-35.md` — Long-form article (Medium/Substack style)+ - `group-36.md` — Search results page (SERP)+ - `group-37.md` — Q&A thread (Stack-Overflow style)+ - `group-38.md` — Pricing table++ # Group 0: Setup & Diagnosis++ ### 0.1 Server reachable+ Check that the PinchTab server is healthy.++ **Verify**: Server responds with `status: ok`.++ ### 0.2 Auth is required+ Make a request to the server with a **wrong** token (`PINCHTAB_TOKEN=wrong-token ./scripts/pt health`) and confirm it is rejected. The `pt` wrapper always injects the benchmark token by default, so you must explicitly override it to test auth rejection.++ **Verify**: Response is HTTP 401 or contains `unauthorized`.++ ### 0.3 Auth works with token+ Repeat the same request WITH the bearer token and confirm it succeeds.++ **Verify**: Response is HTTP 200.++ ### 0.4 Instance available+ Confirm at least one Chrome instance is running. If none exist, start one.++ **Verify**: Health response shows `defaultInstance.status == "running"` (or after starting one, the new instance is running).++ ### 0.5 List existing tabs+ Get the current list of open tabs.++ **Verify**: A list (possibly empty) is returned without error.++ ### 0.6 Clean stale tabs+ If any tabs from previous runs are open, close them so the benchmark starts from a known state.++ **Verify**: After cleanup, the tab list is empty (or only contains a single about:blank tab).++ ### 0.7 Network reach to target+ Navigate to `http://fixtures/` and confirm the fixtures server is reachable from PinchTab.++ **Verify**: Navigate returns successfully and the page contains benchmark content.++ ### 0.8 Capture initial tab ID+ Save the tab ID returned by the navigate in 0.7. Use this tab ID for all subsequent tasks to avoid creating new tabs.++ **Verify**: A tab ID was captured and matches what `GET /tabs` reports.++ ---++ # Group 1: Reading & Extracting Real Content++ ### 1.1 Get the full list of categories from the wiki index+ Navigate to `http://fixtures/wiki.html` and extract all category names with their article counts.++ **Verify**: Can name at least 2 categories and their article counts (e.g. "Programming Languages: 12 articles").++ ### 1.2 Navigate by clicking a link+ From the wiki index, click the "Go (programming language)" link to navigate to the Go article.++ **Verify**: You are now on the Go article page (not the wiki index).++ ### 1.3 Extract structured data from a table+ On the Go article, read the infobox table and answer: Who designed Go, and what year did it first appear?++ **Verify**: Answer contains "Robert Griesemer" (or "Rob Pike" or "Ken Thompson") and "2009".++ ### 1.4 Count list items+ On the Go article, count how many key features are listed.++ **Verify**: Answer is 6 (verify against `FEATURE_COUNT_6`).++ ### 1.5 Read all article headlines from articles page+ Navigate to `http://fixtures/articles.html` and list all article titles.++ **Verify**: Found at least 3 articles including "The Future of Artificial Intelligence".++ ### 1.6 Read dashboard metrics+ Navigate to `http://fixtures/dashboard.html` and extract: Total Users, Revenue, and Conversion Rate.++ **Verify**: Found `24,582` users AND `$1,284,930` revenue.++ ---+- For each completed step:+ 1. record the observed answer with:+ - ./scripts/record-step.sh --report-file /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/results/golden_report.json --type pinchtab <group> <step> answer "<what you saw>" "notes"+ 2. immediately verify it with:+ - ./scripts/verify-step.sh --report-file /sessions/eager-determined-cannon/mnt/pinchtab/tests/benchmark/results/golden_report.json --type pinchtab <group> <step> <pass|fail|skip> "verification notes"+- If a step cannot be completed, record fail or skip in the same report.+- Do not leave answered steps pending verification.+- Keep commands concise. Prefer rg/sed/cat only when you must inspect a specific file.+- After the skill step, begin actual benchmark execution immediately.+- Start with this bootstrap command sequence before attempting the selected steps:+1. ./scripts/pt health+2. ./scripts/pt tab+3. export PINCHTAB_TAB=$(./scripts/pt nav http://fixtures/)+4. ./scripts/pt snap -i -c+- After the bootstrap, immediately execute Group 0 step 1.+- Subset selection:+ - Execute only these benchmark groups: 0, 1.+ - Do not attempt groups outside this subset.+ - Treat all other groups as out of scope for this run rather than as failures.+ - For the selected groups, execute every step in the group unless blocked.+- Finish when all selected steps are executed or when you are blocked.++Your final answer should briefly summarize completion status and the main blockers, if any.\ No newline at end of filediff --git a/tests/benchmark/cmd/api-runner/testdata/golden/system-prompt.txt b/tests/benchmark/cmd/api-runner/testdata/golden/system-prompt.txtnew file mode 100644index 00000000..ce8d1b61--- /dev/null+++ b/tests/benchmark/cmd/api-runner/testdata/golden/system-prompt.txt@@ -0,0 +1,8 @@+You are a precise benchmark execution agent. Use tools to inspect the repo and run the benchmark lane exactly as instructed.++Rules:+- Never fabricate command output or task results.+- Use the shell tool for all file reads and command execution.+- Do not use destructive commands such as rm -rf, git reset, or checkout.+- After recording an answer, verify it immediately against the task oracle.+- Prefer factual command output over long reasoning.\ No newline at end of filediff --git a/tests/benchmark/config/pinchtab.json.backup b/tests/benchmark/config/pinchtab.json.backupdeleted file mode 100644index a360df9a..00000000--- a/tests/benchmark/config/pinchtab.json.backup+++ /dev/null@@ -1,48 +0,0 @@-{- "server": {- "bind": "0.0.0.0",- "port": "9867",- "token": "benchmark-token"- },- "instanceDefaults": {- "maxTabs": 20,- "stealthLevel": "full",- "headless": true- },- "multiInstance": {- "strategy": "always-on",- "allocationPolicy": "fcfs",- "poolSize": 2,- "restart": {- "maxRestarts": 10,- "initBackoffSec": 2,- "maxBackoffSec": 60,- "stableAfterSec": 300- }- },- "scheduler": {- "enabled": true,- "strategy": "fair-fifo",- "maxQueueSize": 10,- "maxPerAgent": 5,- "maxInflight": 3,- "workerCount": 2,- "maxBatchSize": 5,- "resultTTLSec": 300- },- "security": {- "allowEvaluate": true,- "allowDownload": true,- "allowUpload": true,- "allowClipboard": true,- "idpi": {- "enabled": true,- "allowedDomains": ["fixtures", "localhost", "127.0.0.1", "::1", "172.18.0.2"]- }- },- "activity": {- "enabled": true,- "maxEvents": 10000,- "retentionHours": 24- }-}diff --git a/tests/benchmark/continuous-loop-plan.md b/tests/benchmark/continuous-loop-plan.mddeleted file mode 100644index feaf1b85..00000000--- a/tests/benchmark/continuous-loop-plan.md+++ /dev/null@@ -1,219 +0,0 @@-# Continuous Optimization Loop — Plan--## Directive--Execute 250 consecutive optimization loops (Runs #9 through #258) against the-PinchTab benchmark harness. Each loop:--1. Picks the easiest most-consistent optimization available from the backlog.-2. Adds **one** new fixture test case to the benchmark (diverse across loops).-3. Implements, tests, and verifies the change.-4. Runs the full baseline + agent benchmark cycle.-5. Logs the outcome rigorously so work is resumable across sessions.--Ultimate goal: make PinchTab the best possible tool for AI agents that automate-the web. Stay inside the existing code conventions; don't introduce new idioms-or refactor adjacent code unless the optimization requires it.--The user has delegated autonomous execution. Don't ask for approval. Keep-going until completion or a true blocker.-------## One-loop protocol--Each iteration must follow this sequence. Skipping a step risks silent-regressions that poison later loops.--### 1. Plan the loop (in conversation)--- Pick **one optimization** from the backlog below (or a new friction point- surfaced by the last agent run). Priority order:- 1. Real bug fix (incorrect behavior in CLI or server)- 2. Doc gap that causes agent confusion (SKILL.md, help text)- 3. CLI ergonomic (new flag, better error message)- 4. Fixture test case that exercises under-covered behavior-- Pick **one fixture test case** from the fixture-coverage backlog. Track- which categories have been covered to maintain diversity.-- Briefly state the decision in the per-loop decision entry (see §5).--### 2. Implement--- Code changes must stay within existing style (naming, error shape, test- patterns, flag conventions).-- Add or update unit tests alongside any code change.-- For CLI changes: register flag in `cmd_cli_register.go`, plumb through- `internal/cli/actions/...`, document in `skills/pinchtab/SKILL.md`.-- For server changes: update action in `internal/bridge/...`, update- `internal/handlers/...` if HTTP contract changes, add tests.-- For fixture cases: add the HTML/JS fixture in- `tests/benchmark/fixtures/<name>.html`, add the executable baseline step in- `scripts/baseline.sh`, add the matching agent task in `AGENT_TASKS.md`,- update `TEST_CASES.md` summary, update the verification-strings table.-- Run `go test ./internal/...` — must pass.-- Rebuild Docker: `cd tests/benchmark && docker compose up -d --build`.--### 3. Baseline run (must be green)--- Initialize reports: `bash scripts/run-optimization.sh`.-- Run all 58 + new-task baseline curls. Record each step via- `./scripts/record-step.sh --type baseline ...`.-- Target: **100%**. Any failure blocks the loop — diagnose and fix before- proceeding.--### 4. Agent run (measure impact)--- Dispatch the Explore-style general-purpose subagent with:- - A terse prompt that lists current agent-facing capabilities.- - The hard constraint: use `./scripts/pt <args>` only, no helper files.- - Tell it what changed in this loop so it can exercise it.- - Ask for totals + tool-use estimate + new friction points (< 400 words).-- Record pass/fail/skip per step via `record-step.sh --type agent`.--### 5. Log the outcome--Append to `results/decisions.md`:--```markdown-## Loop #N — YYYY-MM-DD HH:MM--**Optimization chosen:** <one line>-**Reason:** <why this won the priority sort>-**Fixture case added:** <one line>--**Changes (files):**-- <path> — <one-line summary>--**Baseline:** X/Y (green/red per step ID if any red)-**Agent:** X/Y--**Agent metrics:** tool_uses=N, duration=Ns, tokens=N--**Delta vs previous successful run:**-<one line>--**Friction surfaced for future loops:**-- <one line each>--**Status:** landed | reverted | deferred-```--Update the summary table in `results/optimization_log.md` with the new row.--If this loop's agent metrics improved on the best score, also update-`results/successful_changes.md` and bump `results/best_score.txt`.--### 6. Resumability--At the end of each loop:-- Git status should be the cumulative uncommitted set from the session.-- No `/tmp` state the next loop depends on (except ephemeral Docker container- state which is fine).-- The plan, decisions log, summary table, and `best_score.txt` are- self-contained — a fresh Claude session can read them and pick up.--If context is getting tight, finish the current loop fully (don't leave a-half-broken state), then use `ScheduleWakeup` with the autonomous-loop-dynamic-sentinel to continue from the next loop after a pause.-------## Optimization backlog (seeded from Runs #1–#8 friction reports)--Ranked by (value × ease). Revise each loop.--### Easy / doc-only-- [ ] SKILL.md note: `eval` shares runtime realm across calls — wrap in IIFE- to avoid `const` collisions.-- [ ] SKILL.md note: `getBoundingClientRect()` returns a DOMRect whose fields- don't JSON-serialize — project to `{x, y, w, h}` explicitly.-- [ ] SKILL.md note: `/text` skips modal/overlay portals; use `eval- document.body.innerText` or `snap --selector` when modal verification is- needed.-- [ ] SKILL.md note: screenshot/PDF `-o` paths land inside the Docker- container for the benchmark harness.-- [ ] `pinchtab select --help` example showing text fallback.--### Easy / CLI flag additions-- [ ] `pinchtab download --output <path>` — decode base64 and write to a host- path so benchmark callers don't have to post-process.-- [ ] `pinchtab snap --selector <css>` surfaced in `--help` Long description.--### Moderate / CLI behavior-- [ ] `pinchtab text` — optional `--max-chars N` flag (HTTP already supports- `maxChars` query param).-- [ ] `pinchtab wait --text "..." --timeout ...` — CLI surface for- `wait` action (if missing).-- [ ] `pinchtab console --tail` — stream console output.--### Moderate / server-- [ ] Enrich `<option>` nodes in compact snapshot with the `value` attribute- via a single `DOM.getAttributes` batch when a `<select>` ancestor is- present. (Partially obviated by the select fallback, but still useful for- agents inspecting dropdown contents.)-- [ ] Investigate `text:` selector intermittent `context deadline exceeded`- — add per-call timeout and better retry policy in- `bridge/action_resolve.go::ResolveTextToNodeID`.-- [ ] Ref recovery confidence — lower confidence when multiple DOM nodes- share identical text (e.g. repeated "✕" delete buttons).--### Hard / deferred-- [ ] Frame-aware ref resolver for iframes.-- [ ] First-fill-after-nav race — `action_pointer.go` readiness wait audit.-------## Fixture coverage backlog--Aim for **diverse** test categories across loops. Categories already covered-(Runs #1–#8): Reading, Search, Forms (basic), SPA, Login, E-commerce,-Content+Interaction, Error handling, Export, Modals, Persistence, Multi-page-Nav, Form Validation, Dynamic Content, Data Aggregation, Hover, Scrolling,-Download, iFrame, Dialogs, Async/awaitPromise, Drag.--Uncovered categories to prioritize:--- **Input variants**: file upload, date picker, time picker, color picker,- range slider, autocomplete/datalist.-- **Form state**: radio groups, checkbox groups, multi-select, fieldset,- disabled/readonly, required, pattern validation with messages.-- **Dynamic UI**: loading spinners, toast notifications, pagination, tabs,- accordion, stepper/wizard.-- **Keyboard**: keyboard shortcuts, tab focus order, focus traps.-- **Visual/CSS**: RTL text, very long text, Unicode/emoji, elements with- CSS transforms / position:fixed / sticky.-- **Accessibility**: ARIA landmarks, ARIA live regions, ARIA expanded/pressed- state, role=grid, aria-controls relationships.-- **State stores**: localStorage, sessionStorage, cookies, IndexedDB.-- **Browser APIs**: clipboard read/write, geolocation (mocked),- notifications.-- **Networking**: fetch polling, SSE stream, WebSocket echo, delayed- response, 5xx retry.-- **Web components**: Shadow DOM (open and closed), slot content, custom- elements, light DOM fallback.-- **Gestures**: drag-to-reorder list, resize handle, pinch-zoom, double-click- distinct from two clicks.-- **Observers**: IntersectionObserver trigger, ResizeObserver trigger,- MutationObserver-driven DOM.-- **Edge cases**: very deep DOM, many siblings, element moved during- interaction, stale ref recovery.-------## Success list--`results/successful_changes.md` ranks interventions by measured impact-(primarily tool-use reduction per run, secondarily token reduction, then-qualitative agent feedback). Update after every loop that lands a new-low-water mark.-------## Stop conditions--- Loop #258 completed and logged.-- User explicitly tells the agent to stop.-- Two consecutive loops fail the baseline and cannot be recovered without- human intervention.-- Backlog exhausted AND no new friction surfaced for 3 consecutive loops —- in that case, exit gracefully and post a completion note to the log.diff --git a/tests/benchmark/optimization-loop.md b/tests/benchmark/optimization-loop.mdnew file mode 100644index 00000000..6aaebeab--- /dev/null+++ b/tests/benchmark/optimization-loop.md@@ -0,0 +1,210 @@+# Benchmark Optimization Loop++Single source of truth for how an agent should improve the benchmark and the+PinchTab lane over time.++## Purpose++The loop exists to improve the real agent outcome against a stable benchmark.++That means the agent should:++- understand the current benchmark contract+- compare the agent lane against the latest green baseline+- decide whether the next best move is:+ - a product/code change+ - a skill/docs change+ - a benchmark/task clarification+ - benchmark expansion into new space++## Inputs++The loop should read these files first:++- `setup-pinchtab.md`+- `benchmark-run/index.md`+- `scripts/baseline.sh`++Useful supporting files:++- `results/optimization_log.md`+- `results/best_score.txt`+- `test-cases-summary.md`++## Baseline++Baseline is **not** an agent task.++Baseline is a deterministic shell verification that proves the benchmark+environment still behaves as expected.++Source of truth:++- `scripts/baseline.sh`++Run baseline directly:++```bash+cd tests/benchmark+./scripts/run-optimization.sh+./scripts/baseline.sh+./scripts/finalize-report.sh "$(cat ./results/current_baseline_report.txt)"+```++### When to run baseline++Run baseline when any of these changed:++- PinchTab server or CLI behavior+- fixtures+- `scripts/baseline.sh`+- benchmark Docker setup+- benchmark task contract in a way that must be proven executable++Do **not** treat baseline as part of the recurring agent loop if nothing in the+benchmark environment changed.++## Core Loop++Each loop should do exactly this:++1. Read the latest green baseline result and the latest agent result.+2. Compare them step-by-step.+3. Identify failures where baseline is green and agent is worse.+4. Infer the smallest high-value change.+5. Make one focused change.+6. Re-run the agent lane.+7. Log the outcome.++The loop should prefer one clear change over broad refactors.++## Decision Rules++### Make a code/product change when:++- baseline proves the behavior is wrong or missing in PinchTab+- the CLI/API is too brittle for a task the benchmark legitimately requires+- the agent failure comes from product ergonomics, not just prompt wording+- a small product fix would help many tasks, not just one narrow case++Typical examples:++- wrong endpoint behavior+- poor error messages+- selector resolution problems+- missing CLI affordance+- state handling bug++### Make a skill/docs change when:++- PinchTab can already do the task correctly+- the agent chose the wrong tool or sequence+- the failure is best explained by missing or unclear guidance+- a short instruction/example would likely prevent repetition++Typical examples:++- when to use `text` vs `snap -i -c`+- when to re-snapshot+- how to handle navigation `409`+- how to reuse `PINCHTAB_TAB`++### Make a benchmark/task change when:++- the intended behavior is correct, but the benchmark wording is ambiguous+- verification is weak or misleading+- the task requires knowledge not present in the lane setup docs+- the benchmark contract should be clarified without changing product behavior++Typical examples:++- task wording too vague+- verification marker unreachable+- expected result not specific enough++### Expand benchmark space when:++- baseline is green+- the latest agent loop no longer reveals an obvious product or skill gap+- the current benchmark is saturated in that area+- an uncovered scenario would increase benchmark value++Expansion should target meaningful new capability areas, not random complexity.++Good expansion targets:++- under-covered interaction types+- state persistence+- multi-step workflows+- accessibility-sensitive flows+- iframe/shadow-dom edge cases+- dynamic content and waiting++## Expansion Rules++When expanding the benchmark:++1. Add or update fixture files.+2. Add the executable baseline case in `scripts/baseline.sh`.+3. Add the matching shared task in `benchmark-run/group-XX.md` and keep `benchmark-run/index.md` current.+4. Update `test-cases-summary.md` if it is used as the case inventory.+5. Run baseline and require it to be green before trusting the new case.++Do not add agent-only benchmark cases. Every new benchmark case must have a+working baseline path first.++## Agent Execution Rules++For the PinchTab agent lane:++- use `setup-pinchtab.md`+- use `benchmark-run/index.md` plus the relevant `benchmark-run/group-XX.md` files+- use only `./scripts/pt` for browser work+- record with `record-step.sh --type pinchtab`+- verify with `verify-step.sh --type pinchtab`++The agent should not use baseline as an execution guide. Baseline is only the+reference oracle and environment check.++## Prioritization++Use this priority order:++1. Fix a real product/code bug.+2. Fix a recurring skill/docs gap.+3. Clarify an ambiguous benchmark/task.+4. Expand the benchmark into new space.++Do not jump to benchmark expansion while a clear product or skill gap is still+unresolved.++## Regression Rule++Compare against `results/best_score.txt`.++- if the new run is worse, treat it as regression work first+- if the new run is equal or better, update the best score++Never hide regressions by weakening the benchmark.++## Loop Output++Each run should append a compact entry to `results/optimization_log.md` with:++- baseline reference used+- agent result+- gap summary+- root cause+- single change made+- why that change was chosen+- next focus++## Non-Goals++Do not:++- run baseline as an agent task+- make multiple unrelated changes in one loop+- weaken tests to improve the score+- expand coverage before baseline is trustworthy+- refactor broadly unless the benchmark problem actually requires itdiff --git a/tests/benchmark/scripts/baseline.sh b/tests/benchmark/scripts/baseline.shindex 046fb255..780f72e9 100755--- a/tests/benchmark/scripts/baseline.sh+++ b/tests/benchmark/scripts/baseline.sh@@ -4,10 +4,9 @@## IMPORTANT:# - This script is the executable source of truth for the baseline lane.-# - `tests/benchmark/BASELINE_TASKS.md` is descriptive/spec documentation only;-# `./dev benchmark baseline` does not parse that markdown file.-# - If baseline behavior changes, update this script first and then update-# any markdown/docs that describe the baseline contract.+# - `./dev benchmark baseline` runs this script directly.+# - If baseline behavior changes, update this script and then update any+# benchmark docs that describe the baseline contract.## Prerequisites:# - `bash scripts/run-optimization.sh` has been run first (initializes thediff --git a/tests/benchmark/scripts/record-run-usage.sh b/tests/benchmark/scripts/record-run-usage.shindex 21b8fab0..027df175 100755--- a/tests/benchmark/scripts/record-run-usage.sh+++ b/tests/benchmark/scripts/record-run-usage.sh@@ -3,7 +3,7 @@# Records exact run-level usage for a benchmark report.## Usage:-# ./record-run-usage.sh --type agent|agent-browser \+# ./record-run-usage.sh --type pinchtab|agent-browser \# --provider anthropic \# --input-tokens 123 \# --output-tokens 45 \@@ -90,7 +90,7 @@ fiif [[ -z "${REPORT_FILE}" ]]; thencase "${REPORT_TYPE}" in- agent)+ pinchtab|agent)REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_PTR}" || true)"[[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_benchmark_*.json 2>/dev/null | head -1);;@@ -99,7 +99,7 @@ if [[ -z "${REPORT_FILE}" ]]; then[[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_browser_benchmark_*.json 2>/dev/null | head -1);;*)- echo "ERROR: --type must be 'agent' or 'agent-browser' unless --report-file is provided"+ echo "ERROR: --type must be 'pinchtab' or 'agent-browser' unless --report-file is provided"exit 1;;esacdiff --git a/tests/benchmark/scripts/record-step.sh b/tests/benchmark/scripts/record-step.shindex f83f2d5e..f144e352 100755--- a/tests/benchmark/scripts/record-step.sh+++ b/tests/benchmark/scripts/record-step.sh@@ -3,10 +3,10 @@# Record a benchmark step result## Usage:-# ./record-step.sh [--type baseline|agent|agent-browser] <group> <step> <pass|fail|skip|answer> [answer] [notes]+# ./record-step.sh [--type baseline|pinchtab|agent-browser] <group> <step> <pass|fail|skip|answer> [answer] [notes]## Options:-# --type baseline|agent|agent-browser Report type (default: auto-detect most recent)+# --type baseline|pinchtab|agent-browser Report type (default: auto-detect most recent)# --tokens <in> <out> Deprecated compatibility field; benchmark reporting now prefers exact run-level usage# --bytes <n> HTTP response size in bytes (baseline runs)# --tool-calls <n> Override auto-counted tool calls@@ -15,8 +15,8 @@## Examples:# ./record-step.sh 1 1 answer "Navigation completed" "Observed output"-# ./record-step.sh --type agent 2 3 fail "Element not found"-# ./record-step.sh --type agent 2 4 answer "Robert Griesemer, 2009" "read infobox"+# ./record-step.sh --type pinchtab 2 3 fail "Element not found"+# ./record-step.sh --type pinchtab 2 4 answer "Robert Griesemer, 2009" "read infobox"# ./record-step.sh --type baseline 1 2 answer --bytes 4520 "Snapshot returned" "Observed output"set -euo pipefail@@ -77,7 +77,7 @@ while [[ $# -gt 0 && "$1" == --* ]]; dodoneif [[ $# -lt 3 ]]; then- echo "Usage: $0 [--type baseline|agent|agent-browser] [--report-file <path>] <group> <step> <pass|fail|skip|answer> [answer] [notes]"+ echo "Usage: $0 [--type baseline|pinchtab|agent-browser] [--report-file <path>] <group> <step> <pass|fail|skip|answer> [answer] [notes]"exit 1fi@@ -127,7 +127,7 @@ if [[ -z "${REPORT_FILE}" ]]; thenREPORT_FILE="$(resolve_current_report "${CURRENT_BASELINE_PTR}" || true)"[[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/baseline_*.json 2>/dev/null | head -1);;- agent)+ pinchtab|agent)REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_PTR}" || true)"[[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_benchmark_*.json 2>/dev/null | head -1);;@@ -136,7 +136,7 @@ if [[ -z "${REPORT_FILE}" ]]; then[[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_browser_benchmark_*.json 2>/dev/null | head -1);;*)- echo "ERROR: --type must be 'baseline', 'agent', or 'agent-browser'"+ echo "ERROR: --type must be 'baseline', 'pinchtab', or 'agent-browser'"exit 1;;esac@@ -165,11 +165,11 @@ case "${BENCHMARK_TYPE}" in;;esac;;- agent|agent-browser)+ pinchtab|agent|agent-browser)case "${STATUS}" inanswer|fail|skip) ;;*)- echo "ERROR: agent steps must use answer, fail, or skip"+ echo "ERROR: pinchtab/agent steps must use answer, fail, or skip"exit 1;;esacdiff --git a/tests/benchmark/scripts/run-agent-browser-benchmark.sh b/tests/benchmark/scripts/run-agent-browser-benchmark.shindex 2e0d8335..7c98b5a3 100755--- a/tests/benchmark/scripts/run-agent-browser-benchmark.sh+++ b/tests/benchmark/scripts/run-agent-browser-benchmark.sh@@ -79,7 +79,7 @@ echo "Initialized agent-browser benchmark report:"echo " ${REPORT_FILE}"echo ""echo "Next steps:"-echo " 1. Read ./AGENT_BROWSER_INSTRUCTIONS.md"+echo " 1. Read ./setup-agent-browser.md and ./benchmark-run/index.md"echo " 2. Load the live CLI skill with ./scripts/ab skills get agent-browser --full"echo " 3. Use ./scripts/ab ... to drive agent-browser inside Docker"echo " 4. Record each completed benchmark step with:"diff --git a/tests/benchmark/scripts/run-api-benchmark.ts b/tests/benchmark/scripts/run-api-benchmark.tsdeleted file mode 100644index ded4285b..00000000--- a/tests/benchmark/scripts/run-api-benchmark.ts+++ /dev/null@@ -1,1185 +0,0 @@-#!/usr/bin/env -S node --experimental-strip-types--import { appendFileSync, existsSync, readFileSync, writeFileSync } from "node:fs";-import { spawn, spawnSync } from "node:child_process";-import { dirname, resolve } from "node:path";-import { fileURLToPath } from "node:url";--type Lane = "agent" | "agent-browser";-type Provider = "anthropic" | "openai";--type ToolCall = {- id: string;- command: string;- timeoutSeconds: number;-};--type ToolExecutionResult = {- id: string;- isError: boolean;- content: string;-};--type Args = {- lane: Lane;- provider?: Provider;- model?: string;- groups?: number[];- profile?: string;- maxTokens: number;- temperature: number;- maxTurns: number;- maxIdleTurns: number;- timeoutSeconds: number;- turnDelayMs: number;- reportFile?: string;- skipInit: boolean;- noPromptCaching: boolean;- finalize: boolean;-};--const SCRIPT_PATH = fileURLToPath(import.meta.url);-const SCRIPT_DIR = dirname(SCRIPT_PATH);-const BENCH_DIR = resolve(SCRIPT_DIR, "..");-const REPO_ROOT = resolve(BENCH_DIR, "..", "..");-const RESULTS_DIR = resolve(BENCH_DIR, "results");-const CURRENT_AGENT_PTR = resolve(RESULTS_DIR, "current_agent_report.txt");-const CURRENT_AGENT_BROWSER_PTR = resolve(RESULTS_DIR, "current_agent_browser_report.txt");-const AGENT_COMMANDS_LOG = resolve(RESULTS_DIR, "agent_commands.ndjson");-const MAX_TOOL_OUTPUT_CHARS = 2400;-const COMPACT_AFTER_ANTHROPIC_MESSAGES = 10;-const KEEP_RECENT_ANTHROPIC_MESSAGES = 6;-const COMPACT_AFTER_OPENAI_ITEMS = 16;-const KEEP_RECENT_OPENAI_ITEMS = 10;-const MAX_SUMMARY_STEPS = 12;-const MAX_SUMMARY_TEXT_CHARS = 220;-const AGENT_TASKS_FILE = resolve(BENCH_DIR, "AGENT_TASKS.md");-const AGENT_BROWSER_TASKS_FILE = resolve(BENCH_DIR, "AGENT_BROWSER_TASKS.md");--function usage(exitCode = 1): never {- const stream = exitCode === 0 ? console.log : console.error;- stream(`Usage:- node --experimental-strip-types ./tests/benchmark/scripts/run-api-benchmark.ts --lane agent [options]- node --experimental-strip-types ./tests/benchmark/scripts/run-api-benchmark.ts --lane agent-browser [options]--Options:- --provider anthropic|openai- --model MODEL- --groups 0,1,2,3- --profile common10- --max-tokens N- --temperature N- --max-turns N- --max-idle-turns N- --timeout-seconds N- --turn-delay-ms N- --report-file PATH- --skip-init- --no-prompt-caching- --finalize-`);- process.exit(exitCode);-}--function parseArgs(argv: string[]): Args {- let lane: Lane | undefined;- let provider: Provider | undefined;- let model: string | undefined;- let groups: number[] | undefined;- let profile: string | undefined;- let maxTokens = 4096;- let temperature = 0;- let maxTurns = 300;- let maxIdleTurns = 25;- let timeoutSeconds = 120;- let turnDelayMs = 1500;- let reportFile: string | undefined;- let skipInit = false;- let noPromptCaching = false;- let finalize = false;-- for (let i = 0; i < argv.length; i += 1) {- const arg = argv[i];- switch (arg) {- case "--lane":- lane = argv[++i] as Lane;- break;- case "--provider":- provider = argv[++i] as Provider;- break;- case "--model":- model = argv[++i];- break;- case "--groups":- groups = argv[++i]- .split(",")- .map((value) => Number(value.trim()))- .filter((value) => Number.isInteger(value));- break;- case "--profile":- profile = argv[++i];- break;- case "--max-tokens":- maxTokens = Number(argv[++i]);- break;- case "--temperature":- temperature = Number(argv[++i]);- break;- case "--max-turns":- maxTurns = Number(argv[++i]);- break;- case "--max-idle-turns":- maxIdleTurns = Number(argv[++i]);- break;- case "--timeout-seconds":- timeoutSeconds = Number(argv[++i]);- break;- case "--turn-delay-ms":- turnDelayMs = Number(argv[++i]);- break;- case "--report-file":- reportFile = argv[++i];- break;- case "--skip-init":- skipInit = true;- break;- case "--no-prompt-caching":- noPromptCaching = true;- break;- case "--finalize":- finalize = true;- break;- case "-h":- case "--help":- case "help":- usage(0);- break;- default:- throw new Error(`Unknown argument: ${arg}`);- }- }-- if (lane !== "agent" && lane !== "agent-browser") {- throw new Error("--lane must be 'agent' or 'agent-browser'");- }-- return {- lane,- provider,- model,- groups,- profile,- maxTokens,- temperature,- maxTurns,- maxIdleTurns,- timeoutSeconds,- turnDelayMs,- reportFile,- skipInit,- noPromptCaching,- finalize,- };-}--function resolveGroups(args: Args): number[] | undefined {- if (args.groups && args.groups.length > 0) {- return [...new Set(args.groups)].sort((a, b) => a - b);- }- if (!args.profile) {- return undefined;- }- switch (args.profile) {- case "common10":- return [0, 1, 2, 3];- default:- throw new Error(`Unknown profile: ${args.profile}`);- }-}--function formatGroups(groups?: number[]): string {- if (!groups || groups.length === 0) {- return "all";- }- return groups.join(",");-}--function extractGroupSections(markdownPath: string, groups: number[]): string {- const lines = readText(markdownPath).split("\n");- const wanted = new Set(groups);- const collected: string[] = [];- let capture = false;-- for (const line of lines) {- const match = line.match(/^## Group (\d+):/);- if (match) {- const group = Number(match[1]);- capture = wanted.has(group);- }- if (capture) {- collected.push(line);- }- }-- return collected.join("\n").trim();-}--function extractSingleGroupSection(markdownPath: string, group: number): string {- const lines = readText(markdownPath).split("\n");- const collected: string[] = [];- let capture = false;-- for (const line of lines) {- const match = line.match(/^## Group (\d+):/);- if (match) {- const currentGroup = Number(match[1]);- if (capture && currentGroup !== group) {- break;- }- capture = currentGroup === group;- }- if (capture) {- collected.push(line);- }- }-- return collected.join("\n").trim();-}--function readText(path: string): string {- return readFileSync(path, "utf8").trim();-}--function readJsonFile(path: string): any {- return JSON.parse(readFileSync(path, "utf8"));-}--function resolveReportPath(lane: Lane): string {- const ptr = lane === "agent" ? CURRENT_AGENT_PTR : CURRENT_AGENT_BROWSER_PTR;- return readText(ptr);-}--function runChecked(cmd: string[], env?: NodeJS.ProcessEnv): void {- const result = spawnSync(cmd[0], cmd.slice(1), {- cwd: REPO_ROOT,- env: env ?? process.env,- stdio: "inherit",- });- if (result.status !== 0) {- throw new Error(`command failed: ${cmd.join(" ")}`);- }-}--function resolveProvider(args: Args): Provider {- if (args.provider) {- return args.provider;- }- const hasOpenAI = (process.env.OPENAI_API_KEY ?? "").trim().length > 0;- const hasAnthropic = (process.env.ANTHROPIC_API_KEY ?? "").trim().length > 0;- const configured = [hasOpenAI, hasAnthropic].filter(Boolean).length;- if (configured > 1) {- throw new Error("Multiple providers appear configured. Pass --provider openai|anthropic explicitly.");- }- if (hasOpenAI) {- return "openai";- }- if (hasAnthropic) {- return "anthropic";- }- throw new Error("Set OPENAI_API_KEY or ANTHROPIC_API_KEY, or pass --provider");-}--function resolveModel(provider: Provider, explicit?: string): string {- if (explicit) {- return explicit;- }- if (provider === "openai") {- return process.env.OPENAI_MODEL?.trim() || "gpt-5";- }- return process.env.ANTHROPIC_MODEL?.trim() || "claude-haiku-4-5-20251001";-}--async function listAnthropicModels(apiKey: string): Promise<string[]> {- const response = await fetch("https://api.anthropic.com/v1/models", {- method: "GET",- headers: {- "x-api-key": apiKey,- "anthropic-version": "2023-06-01",- },- });-- const payload = await response.json();- if (!response.ok) {- throw new Error(`Anthropic models API error ${response.status}: ${JSON.stringify(payload)}`);- }-- return Array.isArray(payload.data)- ? payload.data- .map((item: any) => String(item.id ?? "").trim())- .filter((id: string) => id.length > 0)- : [];-}--async function resolveAnthropicModel(explicit?: string): Promise<string> {- if (explicit) {- return explicit;- }-- const envModel = process.env.ANTHROPIC_MODEL?.trim();- if (envModel) {- return envModel;- }-- const apiKey = (process.env.ANTHROPIC_API_KEY ?? "").trim();- if (!apiKey) {- return "claude-haiku-4-5-20251001";- }-- try {- const available = await listAnthropicModels(apiKey);- const preferred = [- "claude-haiku-4-5-20251001",- "claude-3-5-haiku-20241022",- "claude-3-5-haiku-latest",- "claude-3-haiku-20240307",- "claude-sonnet-4-6",- "claude-sonnet-4-20250514",- "claude-sonnet-4-5-20250929",- "claude-3-7-sonnet-20250219",- "claude-3-7-sonnet-latest",- "claude-3-5-sonnet-20241022",- "claude-3-5-sonnet-latest",- "claude-3-5-sonnet-20240620",- "claude-opus-4-7",- "claude-opus-4-6",- "claude-opus-4-5-20251101",- "claude-opus-4-1-20250805",- ];-- for (const candidate of preferred) {- if (available.includes(candidate)) {- return candidate;- }- }-- if (available.length > 0) {- return available[0];- }- } catch {- // Fall back to a documented default if model discovery fails.- }-- return "claude-haiku-4-5-20251001";-}--function runnerSource(provider: Provider): string {- if (provider === "openai") {- return "openai-responses";- }- return "anthropic-messages";-}--function initializeLane(args: Args, provider: Provider, model: string): string {- const env = { ...process.env, BENCHMARK_MODEL: model, BENCHMARK_RUNNER: runnerSource(provider) };- if (args.lane === "agent") {- runChecked([resolve(BENCH_DIR, "scripts", "run-optimization.sh")], env);- } else {- runChecked([resolve(BENCH_DIR, "scripts", "run-agent-browser-benchmark.sh")], env);- }- return resolveReportPath(args.lane);-}--class PersistentShell {- private proc = spawn("/bin/bash", [], {- cwd: BENCH_DIR,- stdio: ["pipe", "pipe", "pipe"],- });-- constructor() {- this.proc.stderr?.pipe(process.stderr);- }-- async run(command: string, timeoutSeconds: number): Promise<{ output: string; exitCode: number }> {- const marker = `__BENCH_DONE__${Math.random().toString(16).slice(2)}`;- const wrapped = `${command}\nprintf '\\n${marker}:%s\\n' "$?"\n`;- const stdout = this.proc.stdout;- const stdin = this.proc.stdin;- if (!stdout || !stdin) {- throw new Error("persistent shell unavailable");- }-- stdin.write(wrapped);- const deadline = Date.now() + timeoutSeconds * 1000;- let buffer = "";-- while (Date.now() < deadline) {- const chunk = await new Promise<string>((resolveChunk, rejectChunk) => {- const onData = (data: Buffer) => {- cleanup();- resolveChunk(data.toString("utf8"));- };- const onExit = () => {- cleanup();- rejectChunk(new Error("persistent shell exited unexpectedly"));- };- const cleanup = () => {- clearTimeout(timer);- stdout.off("data", onData);- this.proc.off("exit", onExit);- };- const timer = setTimeout(() => {- cleanup();- resolveChunk("");- }, 100);- stdout.once("data", onData);- this.proc.once("exit", onExit);- });-- if (chunk) {- buffer += chunk;- const match = buffer.match(new RegExp(`\\n${marker}:(\\d+)\\r?\\n`));- if (match) {- const output = buffer.slice(0, match.index).replace(/^\n/, "");- return { output, exitCode: Number(match[1]) };- }- }- }-- this.close(true);- throw new Error(`shell command timed out after ${timeoutSeconds}s: ${command}`);- }-- close(force = false): void {- if (this.proc.exitCode !== null) {- return;- }- if (force) {- this.proc.kill("SIGKILL");- return;- }- this.proc.stdin?.end("exit\n");- }-}--abstract class ApiRunner {- requestCount = 0;- inputTokens = 0;- outputTokens = 0;- cacheCreationInputTokens = 0;- cacheReadInputTokens = 0;- provider: Provider;- source: string;- apiKey: string;- model: string;- maxTokens: number;- temperature: number;- promptCaching: boolean;-- constructor(provider: Provider, source: string, apiKey: string, model: string, maxTokens: number, temperature: number, promptCaching: boolean) {- this.provider = provider;- this.source = source;- this.apiKey = apiKey;- this.model = model;- this.maxTokens = maxTokens;- this.temperature = temperature;- this.promptCaching = promptCaching;- }-- abstract toolDefinitions(): unknown[];- abstract initialConversation(userPrompt: string): unknown[];- abstract send(systemPrompt: string, conversation: unknown[]): Promise<any>;- abstract extractToolCalls(response: any, defaultTimeout: number): ToolCall[];- abstract appendToolResults(conversation: unknown[], response: any, results: ToolExecutionResult[]): void;- abstract extractFinalText(response: any): string;-}--class AnthropicRunner extends ApiRunner {- constructor(apiKey: string, model: string, maxTokens: number, temperature: number, promptCaching: boolean) {- super("anthropic", "anthropic-messages", apiKey, model, maxTokens, temperature, promptCaching);- }-- toolDefinitions(): unknown[] {- return [- {- name: "run_command",- description:- "Run a shell command in a persistent bash session rooted at tests/benchmark. Environment changes such as cd/export persist across calls.",- input_schema: {- type: "object",- properties: {- command: { type: "string" },- timeout_seconds: { type: "integer", minimum: 1, maximum: 600 },- },- required: ["command"],- },- },- ];- }-- initialConversation(userPrompt: string): unknown[] {- return [{ role: "user", content: userPrompt }];- }-- async send(systemPrompt: string, conversation: unknown[]): Promise<any> {- const body: Record<string, unknown> = {- model: this.model,- max_tokens: this.maxTokens,- temperature: this.temperature,- system: systemPrompt,- tools: this.toolDefinitions(),- messages: conversation,- };- if (this.promptCaching) {- body.cache_control = { type: "ephemeral" };- }-- const response = await fetchWithRetry("https://api.anthropic.com/v1/messages", {- method: "POST",- headers: {- "content-type": "application/json",- "x-api-key": this.apiKey,- "anthropic-version": "2023-06-01",- },- body: JSON.stringify(body),- });-- const payload = await response.json();- if (!response.ok) {- throw new Error(`Anthropic API error ${response.status}: ${JSON.stringify(payload)}`);- }-- const usage = payload.usage ?? {};- this.requestCount += 1;- this.inputTokens += Number(usage.input_tokens ?? 0);- this.outputTokens += Number(usage.output_tokens ?? 0);- this.cacheCreationInputTokens += Number(usage.cache_creation_input_tokens ?? 0);- this.cacheReadInputTokens += Number(usage.cache_read_input_tokens ?? 0);- return payload;- }-- extractToolCalls(response: any, defaultTimeout: number): ToolCall[] {- return (response.content ?? [])- .filter((item: any) => item.type === "tool_use")- .map((item: any) => ({- id: String(item.id),- command: String(item.input?.command ?? "").trim(),- timeoutSeconds: Number(item.input?.timeout_seconds ?? defaultTimeout),- }))- .filter((call: ToolCall) => call.command.length > 0);- }-- appendToolResults(conversation: unknown[], response: any, results: ToolExecutionResult[]): void {- conversation.push({ role: "assistant", content: response.content });- if (results.length > 0) {- conversation.push({- role: "user",- content: results.map((result) => ({- type: "tool_result",- tool_use_id: result.id,- is_error: result.isError,- content: result.content,- })),- });- }- }-- extractFinalText(response: any): string {- return (response.content ?? [])- .filter((item: any) => item.type === "text" && item.text)- .map((item: any) => String(item.text))- .join("\n")- .trim();- }-}--class OpenAIRunner extends ApiRunner {- private baseUrl: string;-- constructor(provider: "openai", source: string, baseUrl: string, apiKey: string, model: string, maxTokens: number, temperature: number, promptCaching: boolean) {- super(provider, source, apiKey, model, maxTokens, temperature, promptCaching);- this.baseUrl = baseUrl;- }-- toolDefinitions(): unknown[] {- return [- {- type: "function",- name: "run_command",- description:- "Run a shell command in a persistent bash session rooted at tests/benchmark. Environment changes such as cd/export persist across calls.",- parameters: {- type: "object",- properties: {- command: { type: "string" },- timeout_seconds: { type: "integer", minimum: 1, maximum: 600 },- },- required: ["command"],- },- },- ];- }-- initialConversation(userPrompt: string): unknown[] {- return [{ role: "user", content: [{ type: "input_text", text: userPrompt }] }];- }-- async send(systemPrompt: string, conversation: unknown[]): Promise<any> {- const body: Record<string, unknown> = {- model: this.model,- instructions: systemPrompt,- tools: this.toolDefinitions(),- input: conversation,- max_output_tokens: this.maxTokens,- };- if (this.promptCaching) {- body.prompt_cache_retention = "24h";- }-- const response = await fetchWithRetry(this.baseUrl, {- method: "POST",- headers: {- "content-type": "application/json",- authorization: `Bearer ${this.apiKey}`,- },- body: JSON.stringify(body),- });-- const payload = await response.json();- if (!response.ok) {- throw new Error(`OpenAI API error ${response.status}: ${JSON.stringify(payload)}`);- }-- const usage = payload.usage ?? {};- const totalInput = Number(usage.input_tokens ?? 0);- const cached = Number(usage.input_tokens_details?.cached_tokens ?? 0);- this.requestCount += 1;- this.inputTokens += Math.max(0, totalInput - cached);- this.outputTokens += Number(usage.output_tokens ?? 0);- this.cacheReadInputTokens += cached;- return payload;- }-- extractToolCalls(response: any, defaultTimeout: number): ToolCall[] {- return (response.output ?? [])- .filter((item: any) => item.type === "function_call")- .map((item: any) => {- const parsed = item.arguments ? JSON.parse(item.arguments) : {};- return {- id: String(item.call_id ?? item.id),- command: String(parsed.command ?? "").trim(),- timeoutSeconds: Number(parsed.timeout_seconds ?? defaultTimeout),- };- })- .filter((call: ToolCall) => call.command.length > 0);- }-- appendToolResults(conversation: unknown[], response: any, results: ToolExecutionResult[]): void {- conversation.push(...(response.output ?? []));- for (const result of results) {- conversation.push({- type: "function_call_output",- call_id: result.id,- output: result.content,- });- }- }-- extractFinalText(response: any): string {- if (typeof response.output_text === "string" && response.output_text.trim()) {- return response.output_text.trim();- }- return (response.output ?? [])- .flatMap((item: any) => (item.type === "message" ? item.content ?? [] : []))- .filter((part: any) => (part.type === "output_text" || part.type === "text") && part.text)- .map((part: any) => String(part.text))- .join("\n")- .trim();- }-}--function trimToolOutput(text: string): string {- if (text.length <= MAX_TOOL_OUTPUT_CHARS) {- return text;- }- return `${text.slice(0, MAX_TOOL_OUTPUT_CHARS)}\n\n[output truncated: ${text.length - MAX_TOOL_OUTPUT_CHARS} more chars]`;-}--function parseRetryAfterMs(value: string | null): number {- if (!value) {- return 0;- }- const seconds = Number(value);- if (Number.isFinite(seconds) && seconds > 0) {- return Math.ceil(seconds * 1000);- }- const dateMs = Date.parse(value);- if (!Number.isNaN(dateMs)) {- return Math.max(0, dateMs - Date.now());- }- return 0;-}--function sleep(ms: number): Promise<void> {- return new Promise((resolvePromise) => setTimeout(resolvePromise, ms));-}--async function fetchWithRetry(url: string, init: RequestInit, maxRetries = 3): Promise<Response> {- let attempt = 0;- while (true) {- const response = await fetch(url, init);- if (response.status !== 429 || attempt >= maxRetries) {- return response;- }-- const retryAfterMs = parseRetryAfterMs(response.headers.get("retry-after"));- const backoffMs = retryAfterMs > 0 ? retryAfterMs : Math.min(15000, 2000 * (attempt + 1));- await sleep(backoffMs);- attempt += 1;- }-}--async function executeToolCalls(shell: PersistentShell, calls: ToolCall[]): Promise<ToolExecutionResult[]> {- const results: ToolExecutionResult[] = [];- for (const call of calls) {- try {- const { output, exitCode } = await shell.run(call.command, call.timeoutSeconds);- const result = {- id: call.id,- isError: exitCode !== 0,- content: `$ ${call.command}\n[exit_code=${exitCode}]\n${trimToolOutput(output)}`,- };- results.push(result);- appendCommandLog(call.command, exitCode, trimToolOutput(output));- } catch (error) {- const message = error instanceof Error ? error.message : String(error);- const result = {- id: call.id,- isError: true,- content: `$ ${call.command}\n[runner_error]\n${message}`,- };- results.push(result);- appendCommandLog(call.command, -1, message);- }- }- return results;-}--function appendCommandLog(command: string, exitCode: number, output: string): void {- const entry = {- timestamp: new Date().toISOString(),- command,- exit_code: exitCode,- output,- };- appendFileSync(AGENT_COMMANDS_LOG, `${JSON.stringify(entry)}\n`, "utf8");-}--function recordUsage(reportFile: string, runner: ApiRunner): void {- runChecked([- resolve(BENCH_DIR, "scripts", "record-run-usage.sh"),- "--report-file",- reportFile,- "--provider",- runner.provider,- "--source",- runner.source,- "--request-count",- String(runner.requestCount),- "--input-tokens",- String(runner.inputTokens),- "--output-tokens",- String(runner.outputTokens),- "--cache-creation-input-tokens",- String(runner.cacheCreationInputTokens),- "--cache-read-input-tokens",- String(runner.cacheReadInputTokens),- ]);-}--function shorten(text: string, maxChars = MAX_SUMMARY_TEXT_CHARS): string {- const normalized = text.replace(/\s+/g, " ").trim();- if (normalized.length <= maxChars) {- return normalized;- }- return `${normalized.slice(0, maxChars - 3)}...`;-}--function buildProgressSummary(reportFile: string): string {- if (!existsSync(reportFile)) {- return "No benchmark report file exists yet.";- }-- const report = JSON.parse(readText(reportFile));- const totals = report.totals ?? {};- const steps = Array.isArray(report.steps) ? report.steps : [];- const recentSteps = steps.slice(-MAX_SUMMARY_STEPS).map((step: any) => {- const verification = step.verification?.status ? ` / verify=${step.verification.status}` : "";- const answer = step.answer || step.notes || "";- return `- ${step.id}: status=${step.status}${verification}; ${shorten(String(answer))}`;- });-- return [- "Benchmark progress summary from the external report.",- `- answered=${totals.steps_answered ?? 0}`,- `- execution_failed=${totals.steps_failed ?? 0}`,- `- execution_skipped=${totals.steps_skipped ?? 0}`,- `- verified_passed=${totals.steps_verified_passed ?? 0}`,- `- verified_failed=${totals.steps_verified_failed ?? 0}`,- `- verified_skipped=${totals.steps_verified_skipped ?? 0}`,- `- pending_verification=${totals.steps_pending_verification ?? 0}`,- recentSteps.length > 0 ? "Recent recorded steps:" : "No steps recorded yet.",- ...recentSteps,- "Use the report file and benchmark wrappers as the source of truth.",- "Do not re-read large documentation files unless you are blocked on a specific syntax detail.",- ].join("\n");-}--function readProgress(reportFile: string): { answered: number; failed: number; skipped: number; verifiedPassed: number } {- if (!existsSync(reportFile)) {- return { answered: 0, failed: 0, skipped: 0, verifiedPassed: 0 };- }- const report = readJsonFile(reportFile);- const totals = report.totals ?? {};- return {- answered: Number(totals.steps_answered ?? 0),- failed: Number(totals.steps_failed ?? 0),- skipped: Number(totals.steps_skipped ?? 0),- verifiedPassed: Number(totals.steps_verified_passed ?? 0),- };-}--function compactAnthropicConversation(conversation: unknown[], reportFile: string): void {- if (conversation.length <= COMPACT_AFTER_ANTHROPIC_MESSAGES) {- return;- }- const head = conversation[0];- const recent = conversation.slice(-KEEP_RECENT_ANTHROPIC_MESSAGES);- conversation.splice(- 0,- conversation.length,- head,- {- role: "user",- content: buildProgressSummary(reportFile),- },- ...recent,- );-}--function compactOpenAIConversation(conversation: unknown[], reportFile: string): void {- if (conversation.length <= COMPACT_AFTER_OPENAI_ITEMS) {- return;- }- const head = conversation[0];- const recent = conversation.slice(-KEEP_RECENT_OPENAI_ITEMS);- conversation.splice(- 0,- conversation.length,- head,- {- role: "user",- content: [{ type: "input_text", text: buildProgressSummary(reportFile) }],- },- ...recent,- );-}--function compactConversation(provider: Provider, conversation: unknown[], reportFile: string): void {- if (provider === "anthropic") {- compactAnthropicConversation(conversation, reportFile);- return;- }- compactOpenAIConversation(conversation, reportFile);-}--function laneSubsetInstructions(groups?: number[]): string {- if (!groups || groups.length === 0) {- return "Execute the full benchmark task set.";- }-- return [- `Execute only these benchmark groups: ${groups.join(", ")}.`,- "Do not attempt groups outside this subset.",- "Treat all other groups as out of scope for this run rather than as failures.",- "For the selected groups, execute every step in the group unless blocked.",- ].join("\n");-}--function laneTaskSourceInstructions(lane: Lane, groups?: number[]): string {- const fullTaskFile = lane === "agent" ? AGENT_TASKS_FILE : AGENT_BROWSER_TASKS_FILE;- if (!groups || groups.length === 0) {- return [- `Read the full task file at ${fullTaskFile}.`,- "Execute the full benchmark task set.",- ].join("\n");- }-- if (lane === "agent") {- const subset = extractGroupSections(AGENT_TASKS_FILE, groups);- return [- `Use only this selected task subset from ${AGENT_TASKS_FILE}:`,- "",- subset,- ].join("\n");- }-- const sections: string[] = [];- for (const group of groups) {- if (group === 0) {- sections.push([- "## Group 0: Setup & Diagnosis (agent-browser lane)",- "",- "### 0.1 Open fixtures home",- "Run `./scripts/ab open http://fixtures/`.",- "",- "**Verify**: Open succeeds and the home page loads.",- "",- "### 0.2 Snapshot interactive refs",- "Run `./scripts/ab snapshot -i -c` on the home page.",- "",- "**Verify**: Interactive refs are returned without error.",- "",- "### 0.3 Session persists across commands",- "Use the same wrapper session across multiple commands and confirm browser state persists.",- "",- "**Verify**: A follow-up action can use the existing page/session state.",- ].join("\n"));- continue;- }-- const section = extractSingleGroupSection(AGENT_TASKS_FILE, group);- if (section) {- sections.push(section);- }- }-- return [- `Use only this selected task subset for agent-browser: Group 0 from ${AGENT_BROWSER_TASKS_FILE}, Groups 1+ from ${AGENT_TASKS_FILE}.`,- "",- sections.join("\n\n"),- ].join("\n");-}--type LanePromptConfig = {- label: string;- skillInstruction: string;- wrapper: string;- recordType: string;- workflowSummary: string[];- adapterNotes: string[];- bootstrapCommands: string[];-};--function lanePromptConfig(lane: Lane): LanePromptConfig {- if (lane === "agent") {- return {- label: "PinchTab agent",- skillInstruction: `Read ${resolve(REPO_ROOT, "skills", "pinchtab", "SKILL.md")} exactly once before acting.`,- wrapper: "./scripts/pt",- recordType: "agent",- workflowSummary: [- "use only ./scripts/pt for browser actions",- "keep one shared tab via PINCHTAB_TAB",- "use snap -i -c for actionable refs",- "use text or text --full for reading content",- "refresh refs after any navigation or DOM change",- ],- adapterNotes: [- 'a navigation-expected click can return "Error 409: unexpected page navigation ..."; treat that as likely success and verify with a fresh snapshot/text read',- "do not assume every command returns JSON; only parse JSON when the command actually returned JSON",- 'the download endpoint returns JSON with base64 content in "data", not a local file path',- 'this environment uses BSD/macOS userland tools; avoid GNU-only flags such as "head -n -1"',- ],- bootstrapCommands: [- "./scripts/pt health",- "./scripts/pt tab",- "export PINCHTAB_TAB=$(./scripts/pt nav http://fixtures/)",- "./scripts/pt snap -i -c",- ],- };- }-- return {- label: "agent-browser",- skillInstruction: "Load the official agent-browser skill exactly once with `./scripts/ab skills get agent-browser --full` before acting.",- wrapper: "./scripts/ab",- recordType: "agent-browser",- workflowSummary: [- "use only ./scripts/ab for browser actions",- "keep the shared benchmark session across commands",- "use snapshot -i -c for actionable refs",- "use fresh refs after any navigation or DOM change",- "reuse the current browser state instead of reopening pages unless needed",- ],- adapterNotes: [- "after navigation-triggering clicks, verify the resulting page with a fresh snapshot/text read instead of trusting the click response alone",- "do not assume every command returns JSON; parse only when the command actually returned JSON",- "for downloads, inspect returned content instead of assuming a file already exists locally",- 'this environment uses BSD/macOS userland tools; avoid GNU-only flags such as "head -n -1"',- ],- bootstrapCommands: [- "./scripts/ab skills get agent-browser --full",- "./scripts/ab open http://fixtures/",- "./scripts/ab snapshot -i -c",- ],- };-}--function laneUserPrompt(lane: Lane, reportFile: string, groups?: number[]): string {- const config = lanePromptConfig(lane);- const firstGroup = groups && groups.length > 0 ? groups[0] : 0;- const bootstrap = config.bootstrapCommands.map((command, index) => `${index + 1}. ${command}`).join("\n");-- return `-Work in this repo: ${REPO_ROOT}--Benchmark lane: ${config.label} execution.--Requirements:-- Follow a linear execution flow: skill once, selected groups, execute, record, verify, continue.-- Do not read README, browse directories, or inspect unrelated files unless a command path is missing.-- ${config.skillInstruction}-- Tool wrapper:- - ${config.wrapper}-- Workflow summary:- - ${config.workflowSummary.join("\n - ")}-- Adapter notes:- - ${config.adapterNotes.join("\n - ")}-- Task scope:- ${laneTaskSourceInstructions(lane, groups).replace(/\n/g, "\n ")}-- For each completed step:- 1. record the observed answer with:- - ./scripts/record-step.sh --report-file ${reportFile} --type ${config.recordType} <group> <step> answer "<what you saw>" "notes"- 2. immediately verify it with:- - ./scripts/verify-step.sh --report-file ${reportFile} --type ${config.recordType} <group> <step> <pass|fail|skip> "verification notes"-- If a step cannot be completed, record fail or skip in the same report.-- Do not leave answered steps pending verification.-- Keep commands concise. Prefer rg/sed/cat only when you must inspect a specific file.-- After the skill step, begin actual benchmark execution immediately.-- Start with this bootstrap command sequence before attempting the selected steps:-${bootstrap}-- After the bootstrap, immediately execute Group ${firstGroup} step 1.-- Subset selection:- - ${laneSubsetInstructions(groups).replace(/\n/g, "\n - ")}-- Finish when all selected steps are executed or when you are blocked.--Your final answer should briefly summarize completion status and the main blockers, if any.-`.trim();-}--function systemPrompt(): string {- return `-You are a precise benchmark execution agent. Use tools to inspect the repo and run the benchmark lane exactly as instructed.--Rules:-- Never fabricate command output or task results.-- Use the shell tool for all file reads and command execution.-- Do not use destructive commands such as rm -rf, git reset, or checkout.-- After recording an answer, verify it immediately against the task oracle.-- Prefer factual command output over long reasoning.-`.trim();-}--function createRunner(provider: Provider, model: string, args: Args): ApiRunner {- if (provider === "openai") {- const apiKey = (process.env.OPENAI_API_KEY ?? "").trim();- if (!apiKey) {- throw new Error("OPENAI_API_KEY is required for provider=openai");- }- return new OpenAIRunner(- "openai",- "openai-responses",- "https://api.openai.com/v1/responses",- apiKey,- model,- args.maxTokens,- args.temperature,- !args.noPromptCaching,- );- }-- const apiKey = (process.env.ANTHROPIC_API_KEY ?? "").trim();- if (!apiKey) {- throw new Error("ANTHROPIC_API_KEY is required for provider=anthropic");- }- return new AnthropicRunner(apiKey, model, args.maxTokens, args.temperature, !args.noPromptCaching);-}--async function main(): Promise<number> {- try {- const args = parseArgs(process.argv.slice(2));- const provider = resolveProvider(args);- const model = provider === "anthropic" ? await resolveAnthropicModel(args.model) : resolveModel(provider, args.model);- const groups = resolveGroups(args);- const reportFile = args.reportFile || (args.skipInit ? resolveReportPath(args.lane) : initializeLane(args, provider, model));- writeFileSync(AGENT_COMMANDS_LOG, "", "utf8");-- console.log(`[benchmark-runner] provider=${provider} model=${model} lane=${args.lane} groups=${formatGroups(groups)} report=${reportFile}`);-- const runner = createRunner(provider, model, args);- const shell = new PersistentShell();- const conversation = runner.initialConversation(laneUserPrompt(args.lane, reportFile, groups));-- let finalText = "";- let exitCode = 0;- let idleTurns = 0;- let lastAnswered = readProgress(reportFile).answered;-- try {- for (let turn = 1; turn <= args.maxTurns; turn += 1) {- if (turn > 1 && args.turnDelayMs > 0) {- await sleep(args.turnDelayMs);- }- const response = await runner.send(systemPrompt(), conversation);- const toolCalls = runner.extractToolCalls(response, args.timeoutSeconds);- if (toolCalls.length > 0) {- const results = await executeToolCalls(shell, toolCalls);- runner.appendToolResults(conversation, response, results);- compactConversation(runner.provider, conversation, reportFile);- const progress = readProgress(reportFile);- if (progress.answered > lastAnswered) {- idleTurns = 0;- lastAnswered = progress.answered;- } else {- idleTurns += 1;- }- if (idleTurns >= args.maxIdleTurns) {- finalText = `Stopped after ${idleTurns} consecutive turns without recording a benchmark step. Check ${AGENT_COMMANDS_LOG} for the command trace.`;- exitCode = 3;- break;- }- continue;- }-- finalText = runner.extractFinalText(response);- break;- }-- if (!finalText) {- finalText = `Stopped after reaching max turns (${args.maxTurns}).`;- exitCode = 2;- }- } finally {- shell.close();- if (existsSync(reportFile)) {- recordUsage(reportFile, runner);- }- if (args.finalize && existsSync(reportFile)) {- runChecked([resolve(BENCH_DIR, "scripts", "finalize-report.sh"), reportFile]);- }- }-- if (finalText) {- console.log(finalText);- }- console.log(- `\n[run-usage] provider=${runner.provider} requests=${runner.requestCount} input=${runner.inputTokens} cache_create=${runner.cacheCreationInputTokens} cache_read=${runner.cacheReadInputTokens} output=${runner.outputTokens} total=${runner.inputTokens + runner.cacheCreationInputTokens + runner.cacheReadInputTokens + runner.outputTokens}`,- );- return exitCode;- } catch (error) {- console.error(error instanceof Error ? error.message : String(error));- return 1;- }-}--process.exitCode = await main();diff --git a/tests/benchmark/scripts/run-optimization.sh b/tests/benchmark/scripts/run-optimization.shindex 43016e2e..8beeade3 100755--- a/tests/benchmark/scripts/run-optimization.sh+++ b/tests/benchmark/scripts/run-optimization.sh@@ -82,7 +82,7 @@ EOFcat > "${AGENT_REPORT}" << EOF{"benchmark": {- "type": "agent",+ "type": "pinchtab","run_number": ${RUN_NUMBER},"timestamp": "${TIMESTAMP}","model": "${BENCHMARK_MODEL:-unknown}",diff --git a/tests/benchmark/scripts/verify-answers.sh b/tests/benchmark/scripts/verify-answers.shindex 11ced7d1..67470774 100755--- a/tests/benchmark/scripts/verify-answers.sh+++ b/tests/benchmark/scripts/verify-answers.sh@@ -3,7 +3,7 @@# One-off verifier for Loop #31 answers. Reads the agent report, matches each# step's `.answer` against an expected marker/substring set, and invokes# verify-step.sh with pass/fail. The marker map below is hand-derived from-# BASELINE_TASKS.md "Pass if" criteria.+# the baseline benchmark oracle implemented in scripts/baseline.sh.## Usage:# ./verify_answers_for_loop31.sh results/agent_benchmark_20260415_233421.jsondiff --git a/tests/benchmark/scripts/verify-step.sh b/tests/benchmark/scripts/verify-step.shindex d6439d40..86668394 100755--- a/tests/benchmark/scripts/verify-step.sh+++ b/tests/benchmark/scripts/verify-step.sh@@ -4,7 +4,7 @@# step.## Usage:-# ./verify-step.sh [--type baseline|agent|agent-browser] [--report-file <path>] <group> <step> <pass|fail|skip> "notes"+# ./verify-step.sh [--type baseline|pinchtab|agent-browser] [--report-file <path>] <group> <step> <pass|fail|skip> "notes"set -euo pipefail@@ -35,7 +35,7 @@ while [[ $# -gt 0 && "$1" == --* ]]; dodoneif [[ $# -lt 3 ]]; then- echo "Usage: $0 [--type baseline|agent|agent-browser] [--report-file <path>] <group> <step> <pass|fail|skip> [notes]"+ echo "Usage: $0 [--type baseline|pinchtab|agent-browser] [--report-file <path>] <group> <step> <pass|fail|skip> [notes]"exit 1fi@@ -70,7 +70,7 @@ if [[ -z "${REPORT_FILE}" ]]; thenREPORT_FILE="$(resolve_current_report "${CURRENT_BASELINE_PTR}" || true)"[[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/baseline_*.json 2>/dev/null | head -1);;- agent)+ pinchtab|agent)REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_PTR}" || true)"[[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_benchmark_*.json 2>/dev/null | head -1);;@@ -79,7 +79,7 @@ if [[ -z "${REPORT_FILE}" ]]; then[[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_browser_benchmark_*.json 2>/dev/null | head -1);;*)- echo "ERROR: --type must be 'baseline', 'agent', or 'agent-browser'"+ echo "ERROR: --type must be 'baseline', 'pinchtab', or 'agent-browser'"exit 1;;esacdiff --git a/tests/benchmark/AGENT_BROWSER_INSTRUCTIONS.md b/tests/benchmark/setup-agent-browser.mdsimilarity index 56%rename from tests/benchmark/AGENT_BROWSER_INSTRUCTIONS.mdrename to tests/benchmark/setup-agent-browser.mdindex d35e46af..ebf279df 100644--- a/tests/benchmark/AGENT_BROWSER_INSTRUCTIONS.md+++ b/tests/benchmark/setup-agent-browser.md@@ -1,13 +1,31 @@-# Agent Browser Instructions+# Agent Browser Setup-These are benchmark-specific instructions for running the `agent-browser` lane.-They are not the official `agent-browser` skill. For the live CLI skill, load it-through the benchmark wrapper:+Lane-specific setup and operating guidance for running the benchmark with+`agent-browser`.++## Read Order++1. Read `./benchmark-run/index.md`+2. Load the live skill through the wrapper:```bash./scripts/ab skills get agent-browser --full```+This file covers setup, wrapper usage, and Group 0 differences. Shared task+groups live in `benchmark-run/`.++## Recording++Record each completed step as factual `answer`, `fail`, or `skip`, then verify:++```bash+./scripts/record-step.sh --type agent-browser <group> <step> answer "<what you saw>" "notes"+./scripts/verify-step.sh --type agent-browser <group> <step> <pass|fail|skip> "verification notes"+```++Do not self-grade inside the answer payload. Keep the answer factual.+## WrapperFor benchmark runs in this repo, do not call `agent-browser` directly. Use:@@ -65,6 +83,8 @@ cd tests/benchmark- Fixtures: `http://fixtures/`- Session name: `benchmark` by default (`AGENT_BROWSER_SESSION` overrides)- Browser driver: Docker service `agent-browser`+- Pages: `/`, `/wiki.html`, `/wiki-go.html`, `/articles.html`, `/search.html`,+ `/form.html`, `/dashboard.html`, `/ecommerce.html`, `/spa.html`, `/login.html`## Operating Guidance@@ -73,3 +93,13 @@ cd tests/benchmark- Prefer refs from `snapshot -i -c` over brittle selectors- Re-snapshot after navigation or any DOM-changing action- Keep one session for the whole benchmark lane unless a task explicitly needs a reset++## Group 0 Override++For the `agent-browser` lane, Group 0 differs from `benchmark-run/group-00.md`:++- 0.1 `./scripts/ab open http://fixtures/` succeeds+- 0.2 `./scripts/ab snapshot -i -c` returns interactive refs+- 0.3 session state persists across multiple `./scripts/ab ...` commands++From Group 1 onward, use `benchmark-run/group-01.md` through `benchmark-run/group-38.md`.diff --git a/tests/benchmark/setup-pinchtab.md b/tests/benchmark/setup-pinchtab.mdnew file mode 100644index 00000000..0acc8cdd--- /dev/null+++ b/tests/benchmark/setup-pinchtab.md@@ -0,0 +1,73 @@+# PinchTab Setup++Lane-specific setup and operating guidance for running the benchmark with+PinchTab.++## Read Order++1. Read `../../skills/pinchtab/SKILL.md`+2. Read `./benchmark-run/index.md`++`benchmark-run/index.md` contains the shared task map. This file only covers+setup, wrapper usage, and PinchTab-specific expectations.++## Recording++Record each completed step as factual `answer`, `fail`, or `skip`, then verify:++```bash+./scripts/record-step.sh --type pinchtab <group> <step> answer "<what you saw>" "notes"+./scripts/verify-step.sh --type pinchtab <group> <step> <pass|fail|skip> "verification notes"+```++Do not self-grade inside the answer payload. Keep the answer factual.++## Environment++- PinchTab: `http://localhost:9867`, token: `benchmark-token`+- Fixtures: `http://fixtures/` (running in Docker as `fixtures` hostname)+- Pages: `/`, `/wiki.html`, `/wiki-go.html`, `/articles.html`, `/search.html`,+ `/form.html`, `/dashboard.html`, `/ecommerce.html`, `/spa.html`, `/login.html`++## Wrapper++Use only:++```bash+./scripts/pt ...+```++Do not call `pinchtab` directly for benchmark runs.++The wrapper:++- executes the CLI inside the benchmark Docker container+- forwards `PINCHTAB_TOKEN`, `PINCHTAB_SERVER`, and `PINCHTAB_TAB`+- preserves one shared tab across commands when `PINCHTAB_TAB` is set++## Recommended Flow++```bash+./scripts/pt health+./scripts/pt tab+export PINCHTAB_TAB=$(./scripts/pt nav http://fixtures/)+./scripts/pt snap -i -c+```++After that:++- use `./scripts/pt snap -i -c` for actionable refs+- use `./scripts/pt text` or `./scripts/pt text --full` for reading+- re-snapshot after navigation or DOM changes+- record and verify each step as you go++## PinchTab-Specific Notes++- A navigation-triggering click can return `Error 409: unexpected page navigation ...` and still have succeeded. Verify with a fresh snapshot or text read.+- Do not assume every command returns JSON. Parse only when the output is actually JSON.+- The download endpoint returns JSON with base64 content in `data`, not a local file path.+- `PINCHTAB_TAB` should be reused for the whole lane unless a task explicitly needs a new tab.++## Group 0++For the PinchTab lane, run Group 0 from `benchmark-run/group-00.md`.diff --git a/tests/benchmark/TEST_CASES.md b/tests/benchmark/test-cases-summary.mdsimilarity index 100%rename from tests/benchmark/TEST_CASES.mdrename to tests/benchmark/test-cases-summary.md