Skip to content

High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.

chore: extending optimization set

Bosch52fe381c25d8
diff --git a/tests/benchmark/AGENT_SCRIPT.md b/tests/benchmark/AGENT_SCRIPT.md
deleted file mode 100644
index 4358a811..00000000
--- a/tests/benchmark/AGENT_SCRIPT.md
+++ /dev/null
@@ -1,945 +0,0 @@
-# PinchTab Agent Benchmark Script
-
-A structured execution plan for AI agents to benchmark PinchTab capabilities through real-world browser tasks.
-
----
-
-## Token & Cost Tracking
-
-Every step must track token usage to measure real-world agent costs.
-
-### What to Track
-
-For each step, record:
-- **Input tokens**: Tokens sent to the model (prompt + context)
-- **Output tokens**: Tokens generated by the model (response)
-- **Tool calls**: Number of tool invocations
-- **PinchTab output size**: Characters/bytes returned by PinchTab commands
-
-### How to Track
-
-**Option A: Model API tracking**
-If your model provider exposes usage in responses, capture `prompt_tokens` and `completion_tokens` per step.
-
-**Option B: Estimation**
-Use these approximations:
-- 1 token ≈ 4 characters (English text)
-- 1 token ≈ 3 characters (code/structured output)
-
-**Option C: Session status (OpenClaw)**
-If running under OpenClaw, call `📊 session_status` periodically to get cumulative usage.
-
-### Cost Calculation
-
-Use these per-1M-token rates (adjust for your provider):
-
-| Model | Input $/1M | Output $/1M |
-|-------|-----------|-------------|
-| Claude Haiku | $0.25 | $1.25 |
-| Claude Sonnet | $3.00 | $15.00 |
-| Claude Opus | $15.00 | $75.00 |
-| GPT-4o | $2.50 | $10.00 |
-| GPT-4o-mini | $0.15 | $0.60 |
-| Gemini 1.5 Pro | $1.25 | $5.00 |
-| Gemini 1.5 Flash | $0.075 | $0.30 |
-
-Formula:
-```
-cost = (input_tokens / 1_000_000 * input_rate) + (output_tokens / 1_000_000 * output_rate)
-```
-
-### Recording Format (Extended)
-
-```
-Step X.Y: [PASS/FAIL/SLOW]
-- Duration: Xms
-- Input tokens: X
-- Output tokens: X
-- PinchTab output: X chars
-- Tool calls: X
-- Est. cost: $X.XXXX
-- Notes: (any observations)
-- Error: (if failed)
-```
-
----
-
-## Rules
-
-### Execution Rules
-
-1. **Follow steps sequentially.** Do not skip steps unless explicitly marked as optional.
-2. **Record outcomes.** After each step, note: success/failure, time taken, any errors encountered.
-3. **Stop on blockers.** If a step fails and is marked `[BLOCKING]`, halt the group and record the failure.
-4. **Retry once.** If a step fails, retry exactly once before marking as failed.
-5. **Fresh state per group.** Each task group assumes no prior state unless it explicitly references a previous group.
-6. **Use CLI first.** Prefer `pinchtab` CLI commands. Use HTTP API only when the step specifies it.
-7. **Capture evidence.** When a step says "verify", use `pinchtab text` or `pinchtab snap -i -c` to confirm.
-
-### Timing Rules
-
-1. **Wait for navigation.** Use `--wait-nav` when clicking links that navigate.
-2. **Re-snapshot after changes.** Always get fresh refs after any click, submit, or navigation.
-3. **Timeout threshold.** If any command takes longer than 30 seconds, mark as slow and continue.
-
-### Recording Format
-
-For each step, record:
-```
-Step X.Y: [PASS/FAIL/SLOW]
-- Duration: Xms
-- Notes: (any observations)
-- Error: (if failed)
-```
-
----
-
-## Pre-Flight Checks
-
-Before starting the benchmark, verify the environment is ready.
-
-### P.1 Verify PinchTab is installed
-Run `pinchtab version` and confirm it returns a version string.
-
-### P.2 Start the PinchTab server
-Run `pinchtab server` in background or verify it's already running.
-
-### P.3 Verify server health
-Run `pinchtab health` and confirm status is healthy.
-
-### P.4 Verify Chrome is available
-Run `pinchtab instances` to confirm no errors about missing Chrome.
-
-### P.5 Clean slate
-Stop any running instances with `pinchtab instance stop --all` if needed.
-
----
-
-## Task Group 1: Basic Navigation
-
-Test fundamental navigation and page observation.
-
-### 1.1 Navigate to a simple page `[BLOCKING]`
-Run `pinchtab nav https://github.com`.
-Verify: Command completes without error.
-
-### 1.2 Extract page text
-Run `pinchtab text`.
-Verify: Output contains "GitHub".
-
-### 1.3 Get interactive snapshot
-Run `pinchtab snap -i -c`.
-Verify: Output shows refs for interactive elements (links, buttons, inputs).
-
-### 1.4 Navigate to a second page
-Run `pinchtab nav https://wikipedia.org`.
-Verify: Command completes without error.
-
-### 1.5 Verify navigation changed the page
-Run `pinchtab text`.
-Verify: Output contains "Wikipedia" and NOT "GitHub".
-
-### 1.6 Navigate back
-Run `pinchtab back`.
-Verify: `pinchtab text` now shows "GitHub" again.
-
-### 1.7 Navigate forward
-Run `pinchtab forward`.
-Verify: `pinchtab text` shows "Wikipedia" content.
-
----
-
-## Task Group 2: Element Interaction
-
-Test clicking, typing, and form operations.
-
-### 2.1 Navigate to Wikipedia search `[BLOCKING]`
-Run `pinchtab nav https://www.wikipedia.org`.
-Verify: Command completes and main page loads.
-
-### 2.2 Snapshot the search form
-Run `pinchtab snap -i -c`.
-Verify: Output shows refs for the search input field.
-
-### 2.3 Fill search field using ref
-Identify the ref for the search input from the snapshot.
-Run `pinchtab fill <ref> "Artificial intelligence"`.
-Verify: Command completes without error.
-
-### 2.4 Fill search field using CSS selector
-Run `pinchtab fill "#searchInput" "Machine learning"`.
-Verify: Command completes without error.
-
-### 2.5 Click a language link
-Identify a language link ref (e.g., English, Español).
-Run `pinchtab click --wait-nav <lang-ref>`.
-Verify: Page navigates to that language Wikipedia.
-
-### 2.6 Navigate to GitHub
-Run `pinchtab nav https://github.com`.
-Verify: GitHub homepage loads.
-
-### 2.7 Use GitHub search
-Run `pinchtab snap -i -c`.
-Identify the search input ref.
-Run `pinchtab fill <search-ref> "pinchtab"`.
-Run `pinchtab press Enter`.
-Verify: `pinchtab text` shows search results.
-
----
-
-## Task Group 3: Multi-Tab Workflow
-
-Test tab management and parallel browsing.
-
-### 3.1 Open first tab `[BLOCKING]`
-Run `pinchtab nav https://news.ycombinator.com`.
-Verify: Page loads successfully.
-
-### 3.2 List tabs
-Run `pinchtab tab`.
-Verify: Output shows at least one tab with Hacker News URL.
-
-### 3.3 Open a second tab
-Run `pinchtab tab new https://lobste.rs`.
-Verify: `pinchtab tab` now shows two tabs.
-
-### 3.4 Verify second tab content
-Run `pinchtab text`.
-Verify: Output contains "Lobsters" or lobste.rs content.
-
-### 3.5 Switch back to first tab
-Identify the Hacker News tab ID from `pinchtab tab`.
-Run `pinchtab tab <tab-id>` to focus it.
-Verify: `pinchtab text` now shows Hacker News content.
-
-### 3.6 Close second tab
-Identify the Lobsters tab ID.
-Run `pinchtab tab close <tab-id>`.
-Verify: `pinchtab tab` shows only one tab remaining.
-
----
-
-## Task Group 4: Search Flow
-
-Test a realistic search workflow.
-
-### 4.1 Navigate to DuckDuckGo `[BLOCKING]`
-Run `pinchtab nav https://duckduckgo.com`.
-Verify: Page loads.
-
-### 4.2 Snapshot search form
-Run `pinchtab snap -i -c`.
-Verify: A search input ref is visible.
-
-### 4.3 Enter search query
-Run `pinchtab fill <search-ref> "pinchtab browser automation"`.
-Verify: Command completes.
-
-### 4.4 Submit search
-Run `pinchtab press Enter`.
-Wait 2 seconds for results to load.
-
-### 4.5 Verify search results
-Run `pinchtab text`.
-Verify: Output contains search result text (links, descriptions).
-
-### 4.6 Click first result
-Run `pinchtab snap -i -c` to get fresh refs.
-Identify the first result link ref.
-Run `pinchtab click --wait-nav <first-result-ref>`.
-Verify: `pinchtab text` shows content from the clicked result page.
-
----
-
-## Task Group 5: Screenshot and PDF Export
-
-Test visual output capabilities.
-
-### 5.1 Navigate to a visual page `[BLOCKING]`
-Run `pinchtab nav https://github.com`.
-Verify: Page loads.
-
-### 5.2 Take a screenshot
-Run `pinchtab screenshot -o /tmp/pinchtab-bench-screenshot.png`.
-Verify: File exists at that path and is a valid PNG.
-
-### 5.3 Take a JPEG screenshot with quality
-Run `pinchtab screenshot -o /tmp/pinchtab-bench-screenshot.jpg -q 80`.
-Verify: File exists and is smaller than the PNG.
-
-### 5.4 Export PDF
-Run `pinchtab pdf -o /tmp/pinchtab-bench-export.pdf`.
-Verify: File exists and is a valid PDF.
-
-### 5.5 Export landscape PDF
-Run `pinchtab pdf -o /tmp/pinchtab-bench-landscape.pdf --landscape`.
-Verify: File exists.
-
-### 5.6 Cleanup
-Remove the test files from /tmp.
-
----
-
-## Task Group 6: Profile Management
-
-Test persistent profile creation and reuse.
-
-### 6.1 List existing profiles
-Run `pinchtab profiles`.
-Record: Number of existing profiles.
-
-### 6.2 Create a test profile `[BLOCKING]`
-Run via HTTP API:
-```bash
-curl -X POST http://localhost:9867/profiles \
- -H "Content-Type: application/json" \
- -d '{"name":"bench-test","description":"Benchmark test profile"}'
-```
-Verify: Response contains the new profile ID.
-
-### 6.3 Verify profile appears in list
-Run `pinchtab profiles`.
-Verify: "bench-test" appears in the list.
-
-### 6.4 Start instance with profile
-Run `pinchtab instance start --profile bench-test --mode headless`.
-Record: The instance ID and port returned.
-
-### 6.5 Use the profiled instance
-Run `pinchtab --server http://localhost:<port> nav https://github.com`.
-Verify: Navigation succeeds.
-
-### 6.6 Stop the instance
-Run `pinchtab instance stop <instance-id>`.
-Verify: Instance no longer appears in `pinchtab instances`.
-
-### 6.7 Delete test profile
-Run via HTTP API:
-```bash
-curl -X DELETE http://localhost:9867/profiles/bench-test
-```
-Verify: Profile no longer appears in `pinchtab profiles`.
-
----
-
-## Task Group 7: Instance Management
-
-Test multi-instance workflows.
-
-### 7.1 Start first instance `[BLOCKING]`
-Run `pinchtab instance start --mode headless`.
-Record: Instance ID and port (e.g., inst_abc123, port 9868).
-
-### 7.2 Start second instance
-Run `pinchtab instance start --mode headless`.
-Record: Instance ID and port (e.g., inst_def456, port 9869).
-
-### 7.3 Verify both instances running
-Run `pinchtab instances`.
-Verify: Both instances appear in the list.
-
-### 7.4 Navigate instances to different sites
-Run `pinchtab --server http://localhost:9868 nav https://github.com`.
-Run `pinchtab --server http://localhost:9869 nav https://wikipedia.org`.
-Verify: Both complete without error.
-
-### 7.5 Verify instance isolation
-Run `pinchtab --server http://localhost:9868 text`.
-Verify: Shows "GitHub".
-Run `pinchtab --server http://localhost:9869 text`.
-Verify: Shows "Wikipedia".
-
-### 7.6 Stop all instances
-Run `pinchtab instance stop --all`.
-Verify: `pinchtab instances` shows no running instances.
-
----
-
-## Task Group 8: Diff Snapshots
-
-Test incremental snapshot diffing.
-
-### 8.1 Navigate to interactive page `[BLOCKING]`
-Run `pinchtab nav https://www.wikipedia.org`.
-Verify: Page loads.
-
-### 8.2 Take baseline snapshot
-Run `pinchtab snap -i -c`.
-Record: The refs visible.
-
-### 8.3 Make a change
-Identify the search input ref.
-Run `pinchtab fill <search-ref> "Test query"`.
-Verify: Command completes.
-
-### 8.4 Take diff snapshot
-Run `pinchtab snap -d -i -c`.
-Verify: Output indicates the changed element or shows delta.
-
-### 8.5 Click a language link
-Click one of the language links.
-Run `pinchtab snap -d -i -c` again.
-Verify: Diff shows the page change.
-
----
-
-## Task Group 9: Error Handling
-
-Test graceful handling of errors.
-
-### 9.1 Navigate to non-existent page
-Run `pinchtab nav https://this-domain-does-not-exist-12345.com`.
-Verify: Command returns an error (not a crash).
-Record: The error message.
-
-### 9.2 Click non-existent ref
-Run `pinchtab nav https://github.com`.
-Run `pinchtab click e999`.
-Verify: Command returns an error about missing ref.
-
-### 9.3 Fill non-existent selector
-Run `pinchtab fill "#nonexistent-element-12345" "test"`.
-Verify: Command returns a selector not found error.
-
-### 9.4 Invalid command syntax
-Run `pinchtab nav` (missing URL).
-Verify: Command returns a usage error.
-
-### 9.5 Server not running (optional)
-Stop the server if safe to do so.
-Run `pinchtab nav https://github.com`.
-Verify: Command returns connection refused or similar error.
-Restart the server.
-
----
-
-## Task Group 10: Performance Baselines
-
-Test timing under normal conditions. Run each 3 times and record average.
-
-### 10.1 Navigation speed
-Time: `pinchtab nav https://github.com`
-Record: Average duration across 3 runs.
-
-### 10.2 Snapshot speed
-Navigate to github.com first.
-Time: `pinchtab snap -i -c`
-Record: Average duration.
-
-### 10.3 Text extraction speed
-Time: `pinchtab text`
-Record: Average duration.
-
-### 10.4 Click speed
-Get a ref from snapshot.
-Time: `pinchtab click <ref>`
-Record: Average duration.
-
-### 10.5 Screenshot speed
-Time: `pinchtab screenshot -o /tmp/bench-perf.png`
-Record: Average duration.
-
----
-
-## Task Group 11: Token Efficiency (Optional)
-
-Measure token usage for agent context optimization.
-
-### 11.1 Full snapshot token count
-Run `pinchtab nav https://news.ycombinator.com`.
-Run `pinchtab snap` and count output tokens/characters.
-Record: Character count of full snapshot.
-
-### 11.2 Interactive-only snapshot
-Run `pinchtab snap -i` and count output.
-Record: Character count, calculate reduction %.
-
-### 11.3 Interactive + compact snapshot
-Run `pinchtab snap -i -c` and count output.
-Record: Character count, calculate reduction %.
-
-### 11.4 Text-only extraction
-Run `pinchtab text` and count output.
-Record: Character count vs snapshot.
-
-### 11.5 Token-limited snapshot
-Run `pinchtab snap --max-tokens 500`.
-Verify: Output is truncated to approximately that budget.
-
----
-
-## Task Group 12: Login Flow Simulation
-
-Test authentication workflows with a practice site.
-Note: Using dedicated test automation sites that are designed for this purpose.
-
-### 12.1 Navigate to practice login page `[BLOCKING]`
-Run `pinchtab nav https://practicetestautomation.com/practice-test-login/`.
-Alternative: `pinchtab nav https://the-internet.herokuapp.com/login`
-Verify: Page loads with login form.
-
-### 12.2 Snapshot login form
-Run `pinchtab snap -i -c`.
-Verify: Username and password fields visible with refs.
-
-### 12.3 Enter valid credentials
-Run `pinchtab fill "#username" "student"`.
-Run `pinchtab fill "#password" "Password123"`.
-Verify: Both commands complete.
-
-### 12.4 Submit login
-Identify the submit button ref.
-Run `pinchtab click --wait-nav <submit-ref>`.
-Verify: Page navigates.
-
-### 12.5 Verify successful login
-Run `pinchtab text`.
-Verify: Output contains "Logged In Successfully" or similar success message.
-
-### 12.6 Test invalid credentials
-Navigate back to the login page.
-Fill with invalid credentials ("wrong", "wrong").
-Submit and verify error message appears.
-
----
-
-## Task Group 13: Dynamic Content
-
-Test handling of JavaScript-rendered content.
-Note: the-internet.herokuapp.com is a test automation practice site, purpose-built for these tests.
-
-### 13.1 Navigate to dynamic page `[BLOCKING]`
-Run `pinchtab nav https://the-internet.herokuapp.com/dynamic_loading/1`.
-Verify: Page loads.
-
-### 13.2 Snapshot initial state
-Run `pinchtab snap -i -c`.
-Verify: "Start" button is visible.
-
-### 13.3 Click to trigger loading
-Click the Start button.
-Wait 2 seconds.
-
-### 13.4 Verify dynamic content loaded
-Run `pinchtab text`.
-Verify: Output contains "Hello World!" (the dynamically loaded text).
-
-### 13.5 Handle loading states
-Run `pinchtab snap -d -i -c`.
-Record: Whether the diff shows the new content correctly.
-
----
-
-## Task Group 14: Dropdown and Select
-
-Test complex form controls.
-
-### 14.1 Navigate to dropdown page `[BLOCKING]`
-Run `pinchtab nav https://the-internet.herokuapp.com/dropdown`.
-Verify: Page loads with dropdown.
-
-### 14.2 Snapshot dropdown
-Run `pinchtab snap -i -c`.
-Verify: Dropdown element has a ref.
-
-### 14.3 Select option by value
-Run `pinchtab select <dropdown-ref> "1"`.
-Verify: Command completes.
-
-### 14.4 Verify selection
-Run `pinchtab snap -i -c`.
-Verify: Option 1 is now selected.
-
-### 14.5 Select different option
-Run `pinchtab select <dropdown-ref> "2"`.
-Verify: Selection changed to Option 2.
-
----
-
-## Task Group 15: Hover and Tooltips
-
-Test hover interactions.
-
-### 15.1 Navigate to hover page `[BLOCKING]`
-Run `pinchtab nav https://the-internet.herokuapp.com/hovers`.
-Verify: Page loads with user avatars.
-
-### 15.2 Snapshot before hover
-Run `pinchtab snap -i -c`.
-Record: Initial state refs.
-
-### 15.3 Hover over first avatar
-Identify the first avatar image ref.
-Run `pinchtab hover <avatar-ref>`.
-Verify: Command completes.
-
-### 15.4 Verify hover content appeared
-Run `pinchtab snap -d -i -c`.
-Verify: New content visible (user name, "View profile" link).
-
-### 15.5 Hover over second avatar
-Hover over the second avatar.
-Verify: Different user info appears.
-
----
-
-## Task Group 16: Scrolling
-
-Test page scrolling capabilities.
-
-### 16.1 Navigate to long page `[BLOCKING]`
-Run `pinchtab nav https://the-internet.herokuapp.com/infinite_scroll`.
-Verify: Page loads.
-
-### 16.2 Get initial content
-Run `pinchtab text`.
-Record: Initial content length.
-
-### 16.3 Scroll down by pixels
-Run `pinchtab scroll 500`.
-Verify: Command completes.
-
-### 16.4 Verify more content loaded
-Run `pinchtab text`.
-Verify: Content length increased (infinite scroll loaded more).
-
-### 16.5 Scroll to element
-Navigate to a page with a footer.
-Run `pinchtab scroll "text:Footer"` or equivalent selector.
-Verify: Footer is now in view.
-
----
-
-## Task Group 17: File Download
-
-Test download capabilities.
-
-### 17.1 Navigate to download page `[BLOCKING]`
-Run `pinchtab nav https://the-internet.herokuapp.com/download`.
-Verify: Page loads with download links.
-
-### 17.2 Identify a download link
-Run `pinchtab snap -i -c`.
-Identify a file link (e.g., .txt or .pdf file).
-
-### 17.3 Download the file
-Run `pinchtab download <file-url> -o /tmp/pinchtab-bench-download.txt`.
-Verify: File exists at that path.
-
-### 17.4 Verify file content
-Check that the downloaded file has content.
-Record: File size.
-
-### 17.5 Cleanup
-Remove the test file.
-
----
-
-## Task Group 18: iFrame Handling
-
-Test interaction with embedded frames.
-
-### 18.1 Navigate to iframe page `[BLOCKING]`
-Run `pinchtab nav https://the-internet.herokuapp.com/iframe`.
-Verify: Page loads with TinyMCE editor in iframe.
-
-### 18.2 Snapshot main frame
-Run `pinchtab snap -i -c`.
-Record: What refs are visible in main frame.
-
-### 18.3 Access iframe content
-Use appropriate command to interact with iframe content.
-Record: Method used and whether it succeeded.
-
-### 18.4 Type in iframe editor
-If iframe is accessible, type "Benchmark test content" in the editor.
-Verify: Text appears in the editor.
-
----
-
-## Task Group 19: Alerts and Dialogs
-
-Test JavaScript alert/confirm/prompt handling.
-
-### 19.1 Navigate to alerts page `[BLOCKING]`
-Run `pinchtab nav https://the-internet.herokuapp.com/javascript_alerts`.
-Verify: Page loads with alert buttons.
-
-### 19.2 Trigger alert
-Click the "Click for JS Alert" button.
-Record: How PinchTab handles the alert.
-
-### 19.3 Accept alert
-Dismiss or accept the alert.
-Verify: `pinchtab text` shows "You successfully clicked an alert".
-
-### 19.4 Handle confirm dialog
-Click "Click for JS Confirm".
-Accept the confirm.
-Verify: Result text shows "Ok".
-
-### 19.5 Handle confirm cancel
-Trigger confirm again.
-Cancel it.
-Verify: Result text shows "Cancel".
-
----
-
-## Task Group 20: Semantic Search (find command)
-
-Test natural language element finding.
-
-### 20.1 Navigate to complex page `[BLOCKING]`
-Run `pinchtab nav https://news.ycombinator.com`.
-Verify: Page loads.
-
-### 20.2 Find login link semantically
-Run `pinchtab find "login link"`.
-Verify: Returns a ref or selector for the login link.
-
-### 20.3 Find search box
-Run `pinchtab find --ref-only "search input"`.
-Verify: Returns a ref.
-
-### 20.4 Find navigation element
-Run `pinchtab find "link to submit new story"`.
-Verify: Returns relevant element.
-
-### 20.5 Compare to manual snapshot
-Run `pinchtab snap -i -c`.
-Verify: The semantic find results match elements in the snapshot.
-
----
-
-## Task Group 21: Challenge/CAPTCHA Detection
-
-Test challenge solver capabilities (requires appropriate site).
-
-### 21.1 Check available solvers
-Run `curl http://localhost:9867/solvers`.
-Record: Which solvers are registered.
-
-### 21.2 Navigate to protected page (if available)
-If a Cloudflare-protected test site is available, navigate to it.
-Otherwise mark this group as SKIPPED.
-
-### 21.3 Detect challenge
-Run `pinchtab text`.
-Check if output indicates a challenge page ("Just a moment...", etc.).
-
-### 21.4 Attempt solve
-Run:
-```bash
-curl -X POST http://localhost:9867/solve \
- -H 'Content-Type: application/json' \
- -d '{"maxAttempts": 3}'
-```
-Record: Response (solved, attempts, solver used).
-
-### 21.5 Verify page loaded
-Run `pinchtab text`.
-Verify: Real page content now visible.
-
----
-
-## Task Group 22: Agent Identity
-
-Test agent attribution features.
-
-### 22.1 Set agent identity via CLI
-Run `pinchtab --agent-id "bench-agent-1" nav https://github.com`.
-Verify: Command completes.
-
-### 22.2 Verify activity recorded
-Run `curl "http://localhost:9867/api/activity?agentId=bench-agent-1"`.
-Verify: Activity entries show for that agent.
-
-### 22.3 Use different agent ID
-Run `pinchtab --agent-id "bench-agent-2" nav https://wikipedia.org`.
-Verify: Command completes.
-
-### 22.4 Verify separate activity streams
-Query activity for both agents.
-Verify: Each has separate activity entries.
-
-### 22.5 Test header-based identity
-Run:
-```bash
-curl -X POST http://localhost:9867/navigate \
- -H "X-Agent-Id: bench-agent-http" \
- -H "Content-Type: application/json" \
- -d '{"url":"https://github.com"}'
-```
-Verify: Activity recorded under "bench-agent-http".
-
----
-
-## Task Group 23: Block Images/Ads Mode
-
-Test bandwidth-saving navigation modes.
-
-### 23.1 Navigate with images blocked `[BLOCKING]`
-Run `pinchtab nav https://news.ycombinator.com --block-images`.
-Verify: Page loads successfully.
-
-### 23.2 Verify page still functional
-Run `pinchtab snap -i -c`.
-Verify: Interactive elements are present.
-
-### 23.3 Compare loading time
-Time navigation with and without `--block-images`.
-Record: Time difference.
-
-### 23.4 Navigate with ads blocked
-Run `pinchtab nav https://github.com --block-ads`.
-Verify: Page loads successfully.
-
----
-
-## Summary Template
-
-After completing all groups, compile results using this template:
-
----
-
-## Results Template
-
-Copy this section to a new file `BENCHMARK_RESULTS_<date>.md`:
-
-```markdown
-# PinchTab Benchmark Results
-
-**Date**: YYYY-MM-DD
-**Agent/Model**: (e.g., Claude Sonnet 3.5)
-**PinchTab Version**: (output of `pinchtab version`)
-**Environment**: (OS, Chrome version)
-
----
-
-## Cost Summary
-
-| Metric | Value |
-|--------|-------|
-| Total input tokens | X |
-| Total output tokens | X |
-| Total tool calls | X |
-| Total estimated cost | $X.XX |
-| Cost per task group (avg) | $X.XX |
-
----
-
-## Pass/Fail Summary
-
-| Group | Name | Pass | Fail | Skip | Blocking? |
-|-------|------|------|------|------|-----------|
-| 1 | Basic Navigation | X/X | X | X | No |
-| 2 | Element Interaction | X/X | X | X | No |
-| 3 | Multi-Tab Workflow | X/X | X | X | No |
-| 4 | Search Flow | X/X | X | X | No |
-| 5 | Screenshot/PDF Export | X/X | X | X | No |
-| 6 | Profile Management | X/X | X | X | No |
-| 7 | Instance Management | X/X | X | X | No |
-| 8 | Diff Snapshots | X/X | X | X | No |
-| 9 | Error Handling | X/X | X | X | No |
-| 10 | Performance Baselines | X/X | X | X | No |
-| 11 | Token Efficiency | X/X | X | X | No |
-| 12 | Login Flow | X/X | X | X | No |
-| 13 | Dynamic Content | X/X | X | X | No |
-| 14 | Dropdown/Select | X/X | X | X | No |
-| 15 | Hover/Tooltips | X/X | X | X | No |
-| 16 | Scrolling | X/X | X | X | No |
-| 17 | File Download | X/X | X | X | No |
-| 18 | iFrame Handling | X/X | X | X | No |
-| 19 | Alerts/Dialogs | X/X | X | X | No |
-| 20 | Semantic Search | X/X | X | X | No |
-| 21 | Challenge/CAPTCHA | X/X | X | X | No |
-| 22 | Agent Identity | X/X | X | X | No |
-| 23 | Block Images/Ads | X/X | X | X | No |
-
-**Total**: X/X passed (X%)
-
----
-
-## Performance Baselines
-
-| Operation | Avg (ms) | Min (ms) | Max (ms) |
-|-----------|----------|----------|----------|
-| Navigation | X | X | X |
-| Snapshot (-i -c) | X | X | X |
-| Text extraction | X | X | X |
-| Click | X | X | X |
-| Fill | X | X | X |
-| Screenshot | X | X | X |
-| PDF export | X | X | X |
-
----
-
-## Token Efficiency
-
-| Mode | Output Size | vs Full Snapshot |
-|------|-------------|------------------|
-| Full snapshot | X chars | 100% |
-| Interactive only (-i) | X chars | X% |
-| Interactive + compact (-i -c) | X chars | X% |
-| Text only | X chars | X% |
-| Token-limited (500) | X chars | X% |
-
----
-
-## Blocking Failures
-
-List any steps marked `[BLOCKING]` that failed:
-
-1. Group X, Step X.Y: (reason)
-2. ...
-
----
-
-## Notable Observations
-
-- (Any interesting findings, edge cases, or suggestions)
-- (Performance anomalies)
-- (Feature requests based on benchmark)
-
----
-
-## Error Messages Quality
-
-Rate 1-5 for each error type encountered:
-
-| Error Type | Clarity (1-5) | Actionable (1-5) | Example |
-|------------|---------------|------------------|---------|
-| Missing ref | X | X | "..." |
-| Connection refused | X | X | "..." |
-| Timeout | X | X | "..." |
-| Invalid selector | X | X | "..." |
-
----
-
-## Detailed Step Results
-
-(Paste step-by-step results here)
-```
-
----
-
-## Benchmark Scope
-
-This benchmark covers:
-- **Navigation**: forward, back, reload, block-images, block-ads
-- **Observation**: snap, text, diff, find (semantic)
-- **Interaction**: click, fill, type, press, select, hover, scroll
-- **Tabs and instances**: multi-tab, multi-instance isolation
-- **Profiles**: create, use, delete
-- **Export**: screenshot (PNG/JPEG), PDF (portrait/landscape)
-- **Downloads**: file download
-- **Dynamic content**: JS-rendered, loading states
-- **Form controls**: dropdowns, radio buttons, checkboxes
-- **Frames**: iframe interaction
-- **Dialogs**: alert, confirm, prompt
-- **Error resilience**: invalid refs, missing elements, network errors
-- **Performance**: timing baselines
-- **Token efficiency**: output size comparisons
-- **Challenge solving**: CAPTCHA/Cloudflare detection
-- **Agent identity**: attribution and activity tracking
diff --git a/tests/benchmark/AGENT_TASKS.md b/tests/benchmark/AGENT_TASKS.md
index a36f3bfe..b483ac87 100644
--- a/tests/benchmark/AGENT_TASKS.md
+++ b/tests/benchmark/AGENT_TASKS.md
@@ -18,17 +18,47 @@ Natural language tasks to test how well an agent uses PinchTab from skill docs a
---
-## Group 0: Setup Verification
+## Group 0: Setup & Diagnosis
-### 0.1 Confirm PinchTab is running
-Check that the PinchTab server is healthy and a Chrome instance is active.
+### 0.1 Server reachable
+Check that the PinchTab server is healthy.
-**Verify**: Server responds with `status: ok` and at least one running instance.
+**Verify**: Server responds with `status: ok`.
-### 0.2 Navigate to fixtures and confirm reachable
-Navigate to `http://fixtures/` and confirm the benchmark fixtures server is reachable.
+### 0.2 Auth is required
+Make a request to the server WITHOUT an auth token and confirm it is rejected.
-**Verify**: Page title contains "Benchmark" or "PinchTab".
+**Verify**: Response is HTTP 401 (or other auth-error status).
+
+### 0.3 Auth works with token
+Repeat the same request WITH the bearer token and confirm it succeeds.
+
+**Verify**: Response is HTTP 200.
+
+### 0.4 Instance available
+Confirm at least one Chrome instance is running. If none exist, start one.
+
+**Verify**: Health response shows `defaultInstance.status == "running"` (or after starting one, the new instance is running).
+
+### 0.5 List existing tabs
+Get the current list of open tabs.
+
+**Verify**: A list (possibly empty) is returned without error.
+
+### 0.6 Clean stale tabs
+If any tabs from previous runs are open, close them so the benchmark starts from a known state.
+
+**Verify**: After cleanup, the tab list is empty (or only contains a single about:blank tab).
+
+### 0.7 Network reach to target
+Navigate to `http://fixtures/` and confirm the fixtures server is reachable from PinchTab.
+
+**Verify**: Navigate returns successfully and the page contains benchmark content.
+
+### 0.8 Capture initial tab ID
+Save the tab ID returned by the navigate in 0.7. Use this tab ID for all subsequent tasks to avoid creating new tabs.
+
+**Verify**: A tab ID was captured and matches what `GET /tabs` reports.
---
@@ -111,7 +141,7 @@ After submitting, if the form is still accessible (or navigate back), verify you
## Group 4: SPA State Management
### 4.1 Read initial app state
-Navigate to `http://fixtures/spa.html` (always navigate fresh — do not reuse a cached page). Read the current task list — how many tasks exist, how many are active vs done?
+Navigate to `http://fixtures/spa.html?reset=1` (the `?reset=1` query param clears any previous localStorage state so the SPA starts with its default 3 tasks). Read the current task list — how many tasks exist, how many are active vs done?
**Verify**: Found 3 total, 2 active, 1 done (verify `TASK_STATS_TOTAL_3_ACTIVE_2_DONE_1`).
@@ -219,7 +249,7 @@ In the modal, select "Dark" from the theme dropdown (`#theme-select`), then clic
## Group 11: State Persistence & Page Reload
### 11.1 Add an item and verify after page reload
-Navigate to `http://fixtures/spa.html`. Add a task titled exactly "Persistent Task Test". Then reload by navigating to `http://fixtures/spa.html` again.
+Navigate to `http://fixtures/spa.html?reset=1` (starts with clean state). Add a task titled exactly "Persistent Task Test". Then navigate away to `http://fixtures/` and back to `http://fixtures/spa.html` (WITHOUT the reset param) to verify localStorage persistence.
**Verify**: After reload, the task still appears in the list (`TASK_PERSISTENT_TEST_FOUND_AFTER_RELOAD`).
@@ -291,11 +321,76 @@ Report which language has the most features and name 1 feature unique to each.
---
+## Group 16: Hover & Tooltips
+
+### 16.1 Hover reveals hidden content
+Navigate to `http://fixtures/hovers.html`. Hover over the first avatar to reveal its hidden info.
+
+**Verify**: After hovering, snapshot contains `HOVER_REVEALED_USER_1`.
+
+### 16.2 Hover swap
+Hover over the second avatar.
+
+**Verify**: Snapshot contains `HOVER_REVEALED_USER_2`.
+
+---
+
+## Group 17: Scrolling
+
+### 17.1 Scroll by pixels
+Navigate to `http://fixtures/scroll.html`. Scroll down approximately 1500 pixels.
+
+**Verify**: Snapshot or text contains `SCROLL_MIDDLE_MARKER` (the mid-page marker is now in view).
+
+### 17.2 Scroll to footer
+Continue scrolling until the footer is visible (or scroll directly to the footer element).
+
+**Verify**: Snapshot contains `SCROLL_REACHED_FOOTER`.
+
+---
+
+## Group 18: File Download
+
+### 18.1 Download a file
+Download `http://fixtures/download-sample.txt` to a local file.
+
+**Verify**: The downloaded file exists and its content includes `DOWNLOAD_FILE_CONTENT_VERIFIED`.
+
+---
+
+## Group 19: iFrame
+
+### 19.1 Read iframe content
+Navigate to `http://fixtures/iframe.html` and extract content from inside the embedded iframe.
+
+**Verify**: The iframe's inner content includes `IFRAME_INNER_CONTENT_LOADED`.
+
+### 19.2 Type into iframe input
+Inside the iframe, fill the input with the text "Hello World" and click the Save button.
+
+**Verify**: The iframe shows `IFRAME_INPUT_RECEIVED_HELLO_WORLD`.
+
+---
+
+## Group 20: Dialogs
+
+### 20.1 Accept alert
+Navigate to `http://fixtures/alerts.html`. Click the "Click for Alert" button and dismiss the alert.
+
+**Verify**: The page result contains `DIALOG_ALERT_DISMISSED`.
+
+### 20.2 Cancel confirm
+Click the "Click for Confirm" button and cancel the confirm dialog.
+
+**Verify**: The page result contains `DIALOG_CONFIRM_CANCELLED`.
+
+---
+
## Summary
| Group | Tasks | Description |
|-------|-------|-------------|
-| 0 | 2 | Setup Verification |
+| 0 | 8 | Setup & Diagnosis |
| 1 | 6 | Reading & Extracting Content |
| 2 | 3 | Search & Dynamic Interaction |
| 3 | 2 | Complex Form |
@@ -311,8 +406,13 @@ Report which language has the most features and name 1 feature unique to each.
| 13 | 2 | Form State & Multi-Step Submission |
| 14 | 2 | Dynamic Content Loading |
| 15 | 2 | Complex Data Extraction & Aggregation |
+| 16 | 2 | Hover & Tooltips |
+| 17 | 2 | Scrolling |
+| 18 | 1 | File Download |
+| 19 | 2 | iFrame |
+| 20 | 2 | Dialogs |
-**Total: 39 tasks** (expanded from 27)
+**Total: 54 tasks**
## Key Differences from Baseline
diff --git a/tests/benchmark/BENCHMARK_TASKS.md b/tests/benchmark/BASELINE_TASKS.md
similarity index 80%
rename from tests/benchmark/BENCHMARK_TASKS.md
rename to tests/benchmark/BASELINE_TASKS.md
index 9931d2e6..88e1007b 100644
--- a/tests/benchmark/BENCHMARK_TASKS.md
+++ b/tests/benchmark/BASELINE_TASKS.md
@@ -53,16 +53,63 @@ tab ID captured in step 0.2.
---
-## Group 0: Setup
+## Group 0: Setup & Diagnosis
-### 0.1 Health check
+### 0.1 Server reachable
```bash
curl -sf http://localhost:9867/health \
-H "Authorization: Bearer benchmark-token"
```
-**Pass if**: `status == "ok"`, `authRequired == true`, `instances >= 1`, `defaultInstance.status == "running"`.
+**Pass if**: `status == "ok"` in response body.
-### 0.2 Fixtures reachable
+### 0.2 Auth required
+```bash
+curl -s -o /dev/null -w "%{http_code}" http://localhost:9867/health
+```
+**Pass if**: HTTP status is `401` (auth rejected without token).
+
+### 0.3 Auth works with token
+```bash
+curl -s -o /dev/null -w "%{http_code}" http://localhost:9867/health \
+ -H "Authorization: Bearer benchmark-token"
+```
+**Pass if**: HTTP status is `200`.
+
+### 0.4 Instance available
+```bash
+curl -sf http://localhost:9867/health \
+ -H "Authorization: Bearer benchmark-token"
+```
+**Pass if**: Response contains `defaultInstance.status == "running"`. If not, run:
+```bash
+curl -X POST http://localhost:9867/instances/start \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{}'
+```
+And re-verify the new instance is running.
+
+### 0.5 List existing tabs
+```bash
+curl -sf http://localhost:9867/tabs \
+ -H "Authorization: Bearer benchmark-token"
+```
+**Pass if**: Returns a JSON array (possibly empty) without error.
+
+### 0.6 Clean stale tabs
+For each tab returned by step 0.5, close it:
+```bash
+curl -X POST http://localhost:9867/tabs/TAB_ID/close \
+ -H "Authorization: Bearer benchmark-token"
+```
+Then verify cleanup:
+```bash
+curl -sf http://localhost:9867/tabs \
+ -H "Authorization: Bearer benchmark-token"
+```
+**Pass if**: After cleanup, the tab list is empty or contains only an about:blank tab.
+
+### 0.7 Network reach to target
```bash
curl -sf -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
@@ -76,6 +123,13 @@ curl -sf "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=50
```
**Pass if**: Navigate returns HTTP 200 AND snapshot contains `VERIFY_HOME_LOADED_12345`.
+### 0.8 Capture initial tab ID
+```bash
+curl -sf http://localhost:9867/tabs \
+ -H "Authorization: Bearer benchmark-token"
+```
+**Pass if**: The captured `TAB_ID` from step 0.7 appears in the tabs list.
+
---
## Group 1: Reading & Extracting
@@ -286,7 +340,7 @@ curl "http://localhost:9867/tabs/TAB_ID/snapshot?filter=interactive&format=compa
curl -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/spa.html"}'
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/spa.html?reset=1"}'
curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \
-H "Authorization: Bearer benchmark-token"
@@ -574,11 +628,13 @@ curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
### 11.1 State after reload
```bash
+# Start fresh (reset param clears localStorage on this load only)
curl -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"tabId":"TAB_ID","url":"http://fixtures/spa.html"}'
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/spa.html?reset=1"}'
+# Add the persistent task
curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
@@ -589,6 +645,12 @@ curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Content-Type: application/json" \
-d '{"kind":"click","selector":"#add-task-btn"}'
+# Navigate away then back WITHOUT reset param — state should persist in localStorage
+curl -X POST http://localhost:9867/navigate \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/"}'
+
curl -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
@@ -840,11 +902,151 @@ curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000"
---
+## Group 16: Hover & Tooltips
+
+### 16.1 Hover reveals info
+```bash
+curl -X POST http://localhost:9867/navigate \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/hovers.html"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"hover","selector":"#avatar-1"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
+ -H "Authorization: Bearer benchmark-token"
+```
+**Pass if**: Snapshot contains `HOVER_REVEALED_USER_1`.
+
+### 16.2 Hover swap
+```bash
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"hover","selector":"#avatar-2"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
+ -H "Authorization: Bearer benchmark-token"
+```
+**Pass if**: Snapshot contains `HOVER_REVEALED_USER_2`.
+
+---
+
+## Group 17: Scrolling
+
+### 17.1 Scroll by pixels
+```bash
+curl -X POST http://localhost:9867/navigate \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/scroll.html"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/evaluate \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"expression":"window.scrollTo(0, 1500); document.querySelector(\"#middle-block\").innerText;"}'
+```
+**Pass if**: Response value contains `SCROLL_MIDDLE_MARKER`.
+
+### 17.2 Scroll to footer
+```bash
+curl -X POST http://localhost:9867/tabs/TAB_ID/evaluate \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"expression":"document.querySelector(\"#footer\").scrollIntoView(); document.querySelector(\"#footer\").innerText;"}'
+```
+**Pass if**: Response value contains `SCROLL_REACHED_FOOTER`.
+
+---
+
+## Group 18: File Download
+
+### 18.1 Download a file
+```bash
+curl -X POST http://localhost:9867/navigate \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/download-sample.txt"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/text" \
+ -H "Authorization: Bearer benchmark-token"
+```
+**Pass if**: Response text contains `DOWNLOAD_FILE_CONTENT_VERIFIED`.
+
+**Note**: PinchTab's `/download` endpoint blocks private/internal IPs as a safety measure. For local fixture downloads, navigate to the file URL and read body text instead.
+
+---
+
+## Group 19: iFrame
+
+### 19.1 Read iframe content
+```bash
+curl -X POST http://localhost:9867/navigate \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/iframe.html"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
+ -H "Authorization: Bearer benchmark-token"
+```
+**Pass if**: Snapshot contains `IFRAME_INNER_CONTENT_LOADED`.
+
+### 19.2 Type into iframe input
+```bash
+curl -X POST http://localhost:9867/tabs/TAB_ID/evaluate \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"expression":"const f=document.getElementById(\"content-frame\").contentDocument; f.getElementById(\"iframe-input\").value=\"Hello World\"; f.getElementById(\"iframe-submit\").click(); f.getElementById(\"iframe-result\").textContent;"}'
+```
+**Pass if**: Response value contains `IFRAME_INPUT_RECEIVED_HELLO_WORLD`.
+
+**Note**: PinchTab's `/action` selectors don't reach inside iframes. Use `/evaluate` with `contentDocument` to interact with iframe content.
+
+---
+
+## Group 20: Dialogs
+
+### 20.1 Accept alert
+```bash
+curl -X POST http://localhost:9867/navigate \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/alerts.html"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"click","selector":"#alert-btn","dialogAction":"accept"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
+ -H "Authorization: Bearer benchmark-token"
+```
+**Pass if**: Snapshot contains `DIALOG_ALERT_DISMISSED`.
+
+### 20.2 Cancel confirm
+```bash
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"click","selector":"#confirm-btn","dialogAction":"dismiss"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
+ -H "Authorization: Bearer benchmark-token"
+```
+**Pass if**: Snapshot contains `DIALOG_CONFIRM_CANCELLED`.
+
+**Note**: Use `dialogAction` on the click action to auto-accept or auto-dismiss a JS dialog that the click opens. Without it, the click would hang until `/dialog` is called from a separate request.
+
+---
+
## Summary
| Group | Tasks | Description |
|-------|-------|-------------|
-| 0 | 2 | Setup |
+| 0 | 8 | Setup & Diagnosis |
| 1 | 6 | Reading & Extracting |
| 2 | 3 | Search & Dynamic |
| 3 | 2 | Form |
@@ -860,8 +1062,13 @@ curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000"
| 13 | 2 | Form Validation |
| 14 | 2 | Dynamic Content |
| 15 | 2 | Data Aggregation |
+| 16 | 2 | Hover & Tooltips |
+| 17 | 2 | Scrolling |
+| 18 | 1 | File Download |
+| 19 | 2 | iFrame |
+| 20 | 2 | Dialogs |
-**Total: 39 tasks**
+**Total: 54 tasks**
## Verification Strings
diff --git a/tests/benchmark/CRON_TASK.md b/tests/benchmark/CRON_TASK.md
index d165e42e..00eaf9c6 100644
--- a/tests/benchmark/CRON_TASK.md
+++ b/tests/benchmark/CRON_TASK.md
@@ -17,14 +17,14 @@ git checkout feat/benchmark && git pull --rebase origin feat/benchmark
```
### Step 2 — Run Baseline Benchmark
-Execute BENCHMARK_TASKS.md exactly as written.
+Execute BASELINE_TASKS.md exactly as written.
- Token: `benchmark-token`
- Record each step: `./record-step.sh <group> <step> <pass|fail> <in> <out> "notes"`
- For failures: include HTTP status, expected string, actual response in notes
### Step 3 — Run Agent Benchmark
Execute AGENT_TASKS.md using SKILL.md as your only guide.
-- **Do NOT look at BENCHMARK_TASKS.md**
+- **Do NOT look at BASELINE_TASKS.md**
- Figure out commands from skill documentation
- Record: `./record-agent-step.sh <group> <step> <pass|fail> <in> <out> "commands used" "result"`
- Log every curl/command executed
@@ -46,7 +46,7 @@ Compare results and classify each agent failure:
Priority order:
1. **API Bug** → Fix PinchTab Go code, commit as `fix: <description>`
2. **Skill Gap** → Improve SKILL.md with clearer guidance or example, commit as `docs(skill): <description>`
-3. **Test Ambiguity** → Fix fixture HTML or BENCHMARK_TASKS.md verification, commit as `test: <description>`
+3. **Test Ambiguity** → Fix fixture HTML or BASELINE_TASKS.md verification, commit as `test: <description>`
4. **Agent Task Clarity** → Improve AGENT_TASKS.md phrasing, commit as `test(agent): <description>`
5. **No Gaps Found** → Add 2-3 new test cases covering uncovered scenarios, commit as `test: add cases for <scenario>`
diff --git a/tests/benchmark/OPTIMIZATION_LOOP.md b/tests/benchmark/OPTIMIZATION_LOOP.md
index 054401d8..2544ebd6 100644
--- a/tests/benchmark/OPTIMIZATION_LOOP.md
+++ b/tests/benchmark/OPTIMIZATION_LOOP.md
@@ -15,24 +15,12 @@ curl -sf -H "Authorization: Bearer benchmark-token" http://localhost:9867/health
```
┌─────────────────────────────────────────────────────────────┐
-│ 1a. RUN BASELINE BENCHMARK [Sub-agent A] │
-│ - Only runs when: │
-│ • No previous baseline exists, OR │
-│ • Changes affect: server code, fixtures, or │
-│ BENCHMARK_TASKS.md │
-│ - Otherwise: reuse last baseline results │
-│ - Load BENCHMARK_TASKS.md │
-│ - Execute 68 curl commands sequentially │
-│ - Check pass/fail per documented conditions │
-│ - Record via: record-step.sh --type baseline │
-│ - Output: results/baseline_<timestamp>.json │
-├─────────────────────────────────────────────────────────────┤
-│ 1b. RUN AGENT BENCHMARK [Sub-agent B] │
-│ - Runs every iteration │
-│ - Load AGENT_TASKS.md + skills/pinchtab/SKILL.md │
-│ - Execute 39 tasks using only skill docs │
-│ - Record via: record-step.sh --type agent --tokens │
-│ - Output: results/agent_benchmark_<timestamp>.json │
+│ 1. RUN AGENT BENCHMARK [Sub-agent] │
+│ - Runs every iteration │
+│ - Load AGENT_TASKS.md + skills/pinchtab/SKILL.md │
+│ - Execute tasks using only skill docs │
+│ - Record via: record-step.sh --type agent │
+│ - Output: results/agent_benchmark_<timestamp>.json │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
@@ -79,14 +67,36 @@ curl -sf -H "Authorization: Bearer benchmark-token" http://localhost:9867/health
| Step | Who | What |
|------|-----|------|
| Pre-req | **Human** | Start Docker, verify health |
-| 1a Baseline | **Sub-agent A** | Execute 68 curl commands (skip if baseline exists and no server/fixture changes) |
-| 1b Agent | **Sub-agent B** | Execute 39 tasks from skill docs, record results |
+| 1 Agent | **Sub-agent** | Execute agent tasks from skill docs, record results |
| 2 Regression | **Main agent** | Compare against best_score.txt |
| 3 Analyze | **Main agent** | Diff baseline vs agent, find patterns |
| 4 Propose | **Main agent → Human** | Suggest 1 fix, get approval |
| 5 Implement | **Main agent** | Make the change, leave uncommitted |
| 6 Log | **Main agent** | Append to optimization_log.md |
+## Baseline is a Prerequisite, Not Part of the Loop
+
+The baseline suite (`BASELINE_TASKS.md`) validates that the benchmark infrastructure itself works — fixtures serve correctly, PinchTab APIs behave as expected, and the test conditions are reachable. It's deterministic (no LLM involved), so once it's at 100% it stays at 100% unless something changes underneath it.
+
+**The loop does NOT re-run the baseline each iteration.** Instead, run the baseline manually when any of these change:
+
+- PinchTab server code (anything in `internal/` that affects the HTTP API)
+- Fixtures (anything in `tests/benchmark/fixtures/`)
+- `BASELINE_TASKS.md` itself (adding/modifying tests)
+- `docker-compose.yml` or the Docker image
+
+**Workflow for adding a new test case:**
+
+1. Write the new test in `BASELINE_TASKS.md` (curl commands + pass condition)
+2. Write the matching task in `AGENT_TASKS.md` (natural language)
+3. Write the summary row in `TEST_CASES.md`
+4. Add any new fixture files needed
+5. **Run the baseline** — it must reach 100% before the case is considered valid
+6. If baseline fails: fix the test/fixture/PinchTab until it passes
+7. Commit the new case — now the agent loop can use it
+
+Once baseline is green, the optimization loop only runs the agent suite.
+
## Regression Detection
Each run compares against `results/best_score.txt`, which tracks the highest pass rates achieved:
diff --git a/tests/benchmark/README.md b/tests/benchmark/README.md
index 5e770eb4..88bc3779 100644
--- a/tests/benchmark/README.md
+++ b/tests/benchmark/README.md
@@ -31,8 +31,7 @@ If Docker build fails or is skipped, the benchmark is **INVALID**.
| File | Purpose |
|------|---------|
| `../../skills/pinchtab/SKILL.md` | PinchTab skill (same as shipped product) |
-| `BENCHMARK_TASKS.md` | Standalone task list (same as skill) |
-| `AGENT_SCRIPT.md` | Extended: 100+ comprehensive tasks |
+| `BASELINE_TASKS.md` | Standalone task list (same as skill) |
| `run-agent-benchmark.sh` | **Recommended**: Start Docker + spawn subagent |
| `run-benchmark.sh` | Start Docker environment only |
| `record-step.sh` | Record step results with token counts |
@@ -114,7 +113,7 @@ Reports are generated in `results/`:
For automated benchmarks, you can:
-1. Parse `BENCHMARK_TASKS.md` for curl commands
+1. Parse `BASELINE_TASKS.md` for curl commands
2. Execute each command
3. Parse responses for pass/fail
4. Call `record-step.sh` with results
diff --git a/tests/benchmark/TEST_CASES.md b/tests/benchmark/TEST_CASES.md
index 31b32ad9..a1c08a67 100644
--- a/tests/benchmark/TEST_CASES.md
+++ b/tests/benchmark/TEST_CASES.md
@@ -1,12 +1,18 @@
# PinchTab Benchmark Test Cases
-39 test cases covering realistic browser automation scenarios against local fixture pages.
+54 test cases covering realistic browser automation scenarios against local fixture pages.
| # | Task | Description |
|---|------|-------------|
-| **Group 0: Setup** | | |
-| 0.1 | Health check | Confirm PinchTab is running with active instance |
-| 0.2 | Fixtures reachable | Navigate to fixtures root, confirm accessible |
+| **Group 0: Setup & Diagnosis** | | |
+| 0.1 | Server reachable | `GET /health` returns `status: ok` |
+| 0.2 | Auth required | Request without token returns 401 |
+| 0.3 | Auth works with token | Request with bearer token returns 200 |
+| 0.4 | Instance available | At least one instance running, or start one |
+| 0.5 | List existing tabs | `GET /tabs` returns array without error |
+| 0.6 | Clean stale tabs | Close any leftover tabs from previous runs |
+| 0.7 | Network reach to target | Navigate to fixtures root, get success |
+| 0.8 | Capture initial tab ID | Save tab ID from 0.7 for subsequent tasks |
| **Group 1: Reading & Extracting** | | |
| 1.1 | Wiki categories | Extract category names + article counts from wiki index |
| 1.2 | Click a link | From wiki, click through to Go article |
@@ -59,3 +65,17 @@
| **Group 15: Data Aggregation** | | |
| 15.1 | Financial calc | Extract revenue + profit, calculate margin |
| 15.2 | Multi-page comparison | Compare features across 3 wiki language pages |
+| **Group 16: Hover & Tooltips** | | |
+| 16.1 | Hover reveals info | Hover first avatar, verify hidden content appears |
+| 16.2 | Hover swap | Hover second avatar, verify different content appears |
+| **Group 17: Scrolling** | | |
+| 17.1 | Scroll by pixels | Scroll down 1500px, verify mid-page marker visible |
+| 17.2 | Scroll to footer | Scroll to bottom, verify footer marker visible |
+| **Group 18: File Download** | | |
+| 18.1 | Download a file | Download sample.txt, verify content marker present |
+| **Group 19: iFrame** | | |
+| 19.1 | Read iframe content | Verify content from inside iframe is accessible |
+| 19.2 | Type into iframe | Fill input inside iframe, verify saved value |
+| **Group 20: Dialogs** | | |
+| 20.1 | Accept alert | Trigger alert, accept, verify result marker |
+| 20.2 | Cancel confirm | Trigger confirm dialog, cancel, verify cancelled marker |
diff --git a/tests/benchmark/fixtures/alerts.html b/tests/benchmark/fixtures/alerts.html
new file mode 100644
index 00000000..03bad9fc
--- /dev/null
+++ b/tests/benchmark/fixtures/alerts.html
@@ -0,0 +1,37 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="UTF-8">
+ <title>Alerts — PinchTab Benchmark</title>
+</head>
+<body>
+ <h1>JavaScript Dialogs — VERIFY_ALERTS_PAGE_44441</h1>
+
+ <button id="alert-btn">Click for Alert</button>
+ <button id="confirm-btn">Click for Confirm</button>
+ <button id="prompt-btn">Click for Prompt</button>
+
+ <p id="result"></p>
+
+ <script>
+ document.getElementById('alert-btn').addEventListener('click', function() {
+ alert('This is an alert');
+ document.getElementById('result').textContent = 'DIALOG_ALERT_DISMISSED';
+ });
+
+ document.getElementById('confirm-btn').addEventListener('click', function() {
+ const ok = confirm('Are you sure?');
+ document.getElementById('result').textContent = ok ? 'DIALOG_CONFIRM_ACCEPTED' : 'DIALOG_CONFIRM_CANCELLED';
+ });
+
+ document.getElementById('prompt-btn').addEventListener('click', function() {
+ const value = prompt('Enter your name:');
+ if (value === null) {
+ document.getElementById('result').textContent = 'DIALOG_PROMPT_CANCELLED';
+ } else {
+ document.getElementById('result').textContent = 'DIALOG_PROMPT_VALUE_' + value.replace(/\s/g, '_').toUpperCase();
+ }
+ });
+ </script>
+</body>
+</html>
diff --git a/tests/benchmark/fixtures/download-sample.txt b/tests/benchmark/fixtures/download-sample.txt
new file mode 100644
index 00000000..c787f775
--- /dev/null
+++ b/tests/benchmark/fixtures/download-sample.txt
@@ -0,0 +1,5 @@
+DOWNLOAD_FILE_CONTENT_VERIFIED
+This is a sample file used by the PinchTab benchmark to test file downloads.
+Line 3.
+Line 4.
+END_OF_FILE_MARKER
diff --git a/tests/benchmark/fixtures/hovers.html b/tests/benchmark/fixtures/hovers.html
new file mode 100644
index 00000000..f4e5e51c
--- /dev/null
+++ b/tests/benchmark/fixtures/hovers.html
@@ -0,0 +1,49 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="UTF-8">
+ <title>Hovers — PinchTab Benchmark</title>
+ <style>
+ .avatar { display: inline-block; margin: 20px; }
+ .avatar-img { width: 100px; height: 100px; background: #ccc; border-radius: 50%; cursor: pointer; }
+ .info { display: none; padding: 10px; background: #f0f0f0; margin-top: 5px; }
+ </style>
+</head>
+<body>
+ <h1>Hovers — VERIFY_HOVERS_PAGE_66666</h1>
+
+ <div class="avatar" id="avatar-1" onmouseenter="showInfo(1)" onmouseleave="hideInfo(1)">
+ <div class="avatar-img"></div>
+ <div class="info" id="info-1"></div>
+ </div>
+
+ <div class="avatar" id="avatar-2" onmouseenter="showInfo(2)" onmouseleave="hideInfo(2)">
+ <div class="avatar-img"></div>
+ <div class="info" id="info-2"></div>
+ </div>
+
+ <div class="avatar" id="avatar-3" onmouseenter="showInfo(3)" onmouseleave="hideInfo(3)">
+ <div class="avatar-img"></div>
+ <div class="info" id="info-3"></div>
+ </div>
+
+ <script>
+ const users = {
+ 1: { name: 'Alice Anderson', marker: 'HOVER_REVEALED_USER_1' },
+ 2: { name: 'Bob Brown', marker: 'HOVER_REVEALED_USER_2' },
+ 3: { name: 'Carol Chen', marker: 'HOVER_REVEALED_USER_3' },
+ };
+
+ function showInfo(n) {
+ const info = document.getElementById('info-' + n);
+ info.innerHTML = '<p>name: ' + users[n].name + '</p><p>' + users[n].marker + '</p>';
+ info.style.display = 'block';
+ }
+
+ function hideInfo(n) {
+ // Keep content visible after hover so snapshots and verification can read it
+ // (don't actually hide; the marker should persist for the next snapshot)
+ }
+ </script>
+</body>
+</html>
diff --git a/tests/benchmark/fixtures/iframe-content.html b/tests/benchmark/fixtures/iframe-content.html
new file mode 100644
index 00000000..b17afe2c
--- /dev/null
+++ b/tests/benchmark/fixtures/iframe-content.html
@@ -0,0 +1,23 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="UTF-8">
+ <title>iFrame Inner</title>
+</head>
+<body>
+ <h2>Inner Frame — IFRAME_INNER_CONTENT_LOADED</h2>
+ <form id="iframe-form">
+ <label>Comment:</label>
+ <input type="text" id="iframe-input" name="comment">
+ <button type="button" id="iframe-submit">Save</button>
+ </form>
+ <p id="iframe-result"></p>
+
+ <script>
+ document.getElementById('iframe-submit').addEventListener('click', function() {
+ const value = document.getElementById('iframe-input').value;
+ document.getElementById('iframe-result').textContent = 'IFRAME_INPUT_RECEIVED_' + value.replace(/\s/g, '_').toUpperCase();
+ });
+ </script>
+</body>
+</html>
diff --git a/tests/benchmark/fixtures/iframe.html b/tests/benchmark/fixtures/iframe.html
new file mode 100644
index 00000000..22cebd06
--- /dev/null
+++ b/tests/benchmark/fixtures/iframe.html
@@ -0,0 +1,12 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="UTF-8">
+ <title>iFrame — PinchTab Benchmark</title>
+</head>
+<body>
+ <h1>iFrame Test — VERIFY_IFRAME_PAGE_33331</h1>
+ <p>The iframe below contains an editable form.</p>
+ <iframe id="content-frame" src="iframe-content.html" width="600" height="400"></iframe>
+</body>
+</html>
diff --git a/tests/benchmark/fixtures/scroll.html b/tests/benchmark/fixtures/scroll.html
new file mode 100644
index 00000000..1705a94e
--- /dev/null
+++ b/tests/benchmark/fixtures/scroll.html
@@ -0,0 +1,30 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="UTF-8">
+ <title>Scroll — PinchTab Benchmark</title>
+ <style>
+ .block { padding: 100px; margin: 50px 0; background: #eef; }
+ #footer { padding: 50px; background: #cce; margin-top: 200px; }
+ </style>
+</head>
+<body>
+ <h1>Scroll — VERIFY_SCROLL_PAGE_22221</h1>
+ <p>SCROLL_TOP_MARKER</p>
+
+ <div class="block">Block 1 — content</div>
+ <div class="block">Block 2 — content</div>
+ <div style="height: 800px;"></div>
+ <div class="block" id="middle-block">
+ <p>SCROLL_MIDDLE_MARKER</p>
+ </div>
+ <div style="height: 1500px;"></div>
+ <div class="block">Block 4 — content</div>
+ <div style="height: 800px;"></div>
+
+ <footer id="footer">
+ <h2>Page Footer</h2>
+ <p>SCROLL_REACHED_FOOTER</p>
+ </footer>
+</body>
+</html>
diff --git a/tests/benchmark/fixtures/spa.html b/tests/benchmark/fixtures/spa.html
index e226c07f..b5b4bdd3 100644
--- a/tests/benchmark/fixtures/spa.html
+++ b/tests/benchmark/fixtures/spa.html
@@ -55,6 +55,11 @@
let taskCount = 3;
let tasks = [];
+ // Support ?reset=1 to clear state between benchmark runs
+ if (new URLSearchParams(window.location.search).has('reset')) {
+ localStorage.removeItem('tasks');
+ }
+
// Load tasks from localStorage on page load
function loadTasks() {
const saved = localStorage.getItem('tasks');