High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.
chore: extending optimization set
Bosch52fe381c25d8
diff --git a/tests/benchmark/AGENT_SCRIPT.md b/tests/benchmark/AGENT_SCRIPT.mddeleted file mode 100644index 4358a811..00000000--- a/tests/benchmark/AGENT_SCRIPT.md+++ /dev/null@@ -1,945 +0,0 @@-# PinchTab Agent Benchmark Script--A structured execution plan for AI agents to benchmark PinchTab capabilities through real-world browser tasks.-------## Token & Cost Tracking--Every step must track token usage to measure real-world agent costs.--### What to Track--For each step, record:-- **Input tokens**: Tokens sent to the model (prompt + context)-- **Output tokens**: Tokens generated by the model (response)-- **Tool calls**: Number of tool invocations-- **PinchTab output size**: Characters/bytes returned by PinchTab commands--### How to Track--**Option A: Model API tracking**-If your model provider exposes usage in responses, capture `prompt_tokens` and `completion_tokens` per step.--**Option B: Estimation**-Use these approximations:-- 1 token ≈ 4 characters (English text)-- 1 token ≈ 3 characters (code/structured output)--**Option C: Session status (OpenClaw)**-If running under OpenClaw, call `📊 session_status` periodically to get cumulative usage.--### Cost Calculation--Use these per-1M-token rates (adjust for your provider):--| Model | Input $/1M | Output $/1M |-|-------|-----------|-------------|-| Claude Haiku | $0.25 | $1.25 |-| Claude Sonnet | $3.00 | $15.00 |-| Claude Opus | $15.00 | $75.00 |-| GPT-4o | $2.50 | $10.00 |-| GPT-4o-mini | $0.15 | $0.60 |-| Gemini 1.5 Pro | $1.25 | $5.00 |-| Gemini 1.5 Flash | $0.075 | $0.30 |--Formula:-```-cost = (input_tokens / 1_000_000 * input_rate) + (output_tokens / 1_000_000 * output_rate)-```--### Recording Format (Extended)--```-Step X.Y: [PASS/FAIL/SLOW]-- Duration: Xms-- Input tokens: X-- Output tokens: X-- PinchTab output: X chars-- Tool calls: X-- Est. cost: $X.XXXX-- Notes: (any observations)-- Error: (if failed)-```-------## Rules--### Execution Rules--1. **Follow steps sequentially.** Do not skip steps unless explicitly marked as optional.-2. **Record outcomes.** After each step, note: success/failure, time taken, any errors encountered.-3. **Stop on blockers.** If a step fails and is marked `[BLOCKING]`, halt the group and record the failure.-4. **Retry once.** If a step fails, retry exactly once before marking as failed.-5. **Fresh state per group.** Each task group assumes no prior state unless it explicitly references a previous group.-6. **Use CLI first.** Prefer `pinchtab` CLI commands. Use HTTP API only when the step specifies it.-7. **Capture evidence.** When a step says "verify", use `pinchtab text` or `pinchtab snap -i -c` to confirm.--### Timing Rules--1. **Wait for navigation.** Use `--wait-nav` when clicking links that navigate.-2. **Re-snapshot after changes.** Always get fresh refs after any click, submit, or navigation.-3. **Timeout threshold.** If any command takes longer than 30 seconds, mark as slow and continue.--### Recording Format--For each step, record:-```-Step X.Y: [PASS/FAIL/SLOW]-- Duration: Xms-- Notes: (any observations)-- Error: (if failed)-```-------## Pre-Flight Checks--Before starting the benchmark, verify the environment is ready.--### P.1 Verify PinchTab is installed-Run `pinchtab version` and confirm it returns a version string.--### P.2 Start the PinchTab server-Run `pinchtab server` in background or verify it's already running.--### P.3 Verify server health-Run `pinchtab health` and confirm status is healthy.--### P.4 Verify Chrome is available-Run `pinchtab instances` to confirm no errors about missing Chrome.--### P.5 Clean slate-Stop any running instances with `pinchtab instance stop --all` if needed.-------## Task Group 1: Basic Navigation--Test fundamental navigation and page observation.--### 1.1 Navigate to a simple page `[BLOCKING]`-Run `pinchtab nav https://github.com`.-Verify: Command completes without error.--### 1.2 Extract page text-Run `pinchtab text`.-Verify: Output contains "GitHub".--### 1.3 Get interactive snapshot-Run `pinchtab snap -i -c`.-Verify: Output shows refs for interactive elements (links, buttons, inputs).--### 1.4 Navigate to a second page-Run `pinchtab nav https://wikipedia.org`.-Verify: Command completes without error.--### 1.5 Verify navigation changed the page-Run `pinchtab text`.-Verify: Output contains "Wikipedia" and NOT "GitHub".--### 1.6 Navigate back-Run `pinchtab back`.-Verify: `pinchtab text` now shows "GitHub" again.--### 1.7 Navigate forward-Run `pinchtab forward`.-Verify: `pinchtab text` shows "Wikipedia" content.-------## Task Group 2: Element Interaction--Test clicking, typing, and form operations.--### 2.1 Navigate to Wikipedia search `[BLOCKING]`-Run `pinchtab nav https://www.wikipedia.org`.-Verify: Command completes and main page loads.--### 2.2 Snapshot the search form-Run `pinchtab snap -i -c`.-Verify: Output shows refs for the search input field.--### 2.3 Fill search field using ref-Identify the ref for the search input from the snapshot.-Run `pinchtab fill <ref> "Artificial intelligence"`.-Verify: Command completes without error.--### 2.4 Fill search field using CSS selector-Run `pinchtab fill "#searchInput" "Machine learning"`.-Verify: Command completes without error.--### 2.5 Click a language link-Identify a language link ref (e.g., English, Español).-Run `pinchtab click --wait-nav <lang-ref>`.-Verify: Page navigates to that language Wikipedia.--### 2.6 Navigate to GitHub-Run `pinchtab nav https://github.com`.-Verify: GitHub homepage loads.--### 2.7 Use GitHub search-Run `pinchtab snap -i -c`.-Identify the search input ref.-Run `pinchtab fill <search-ref> "pinchtab"`.-Run `pinchtab press Enter`.-Verify: `pinchtab text` shows search results.-------## Task Group 3: Multi-Tab Workflow--Test tab management and parallel browsing.--### 3.1 Open first tab `[BLOCKING]`-Run `pinchtab nav https://news.ycombinator.com`.-Verify: Page loads successfully.--### 3.2 List tabs-Run `pinchtab tab`.-Verify: Output shows at least one tab with Hacker News URL.--### 3.3 Open a second tab-Run `pinchtab tab new https://lobste.rs`.-Verify: `pinchtab tab` now shows two tabs.--### 3.4 Verify second tab content-Run `pinchtab text`.-Verify: Output contains "Lobsters" or lobste.rs content.--### 3.5 Switch back to first tab-Identify the Hacker News tab ID from `pinchtab tab`.-Run `pinchtab tab <tab-id>` to focus it.-Verify: `pinchtab text` now shows Hacker News content.--### 3.6 Close second tab-Identify the Lobsters tab ID.-Run `pinchtab tab close <tab-id>`.-Verify: `pinchtab tab` shows only one tab remaining.-------## Task Group 4: Search Flow--Test a realistic search workflow.--### 4.1 Navigate to DuckDuckGo `[BLOCKING]`-Run `pinchtab nav https://duckduckgo.com`.-Verify: Page loads.--### 4.2 Snapshot search form-Run `pinchtab snap -i -c`.-Verify: A search input ref is visible.--### 4.3 Enter search query-Run `pinchtab fill <search-ref> "pinchtab browser automation"`.-Verify: Command completes.--### 4.4 Submit search-Run `pinchtab press Enter`.-Wait 2 seconds for results to load.--### 4.5 Verify search results-Run `pinchtab text`.-Verify: Output contains search result text (links, descriptions).--### 4.6 Click first result-Run `pinchtab snap -i -c` to get fresh refs.-Identify the first result link ref.-Run `pinchtab click --wait-nav <first-result-ref>`.-Verify: `pinchtab text` shows content from the clicked result page.-------## Task Group 5: Screenshot and PDF Export--Test visual output capabilities.--### 5.1 Navigate to a visual page `[BLOCKING]`-Run `pinchtab nav https://github.com`.-Verify: Page loads.--### 5.2 Take a screenshot-Run `pinchtab screenshot -o /tmp/pinchtab-bench-screenshot.png`.-Verify: File exists at that path and is a valid PNG.--### 5.3 Take a JPEG screenshot with quality-Run `pinchtab screenshot -o /tmp/pinchtab-bench-screenshot.jpg -q 80`.-Verify: File exists and is smaller than the PNG.--### 5.4 Export PDF-Run `pinchtab pdf -o /tmp/pinchtab-bench-export.pdf`.-Verify: File exists and is a valid PDF.--### 5.5 Export landscape PDF-Run `pinchtab pdf -o /tmp/pinchtab-bench-landscape.pdf --landscape`.-Verify: File exists.--### 5.6 Cleanup-Remove the test files from /tmp.-------## Task Group 6: Profile Management--Test persistent profile creation and reuse.--### 6.1 List existing profiles-Run `pinchtab profiles`.-Record: Number of existing profiles.--### 6.2 Create a test profile `[BLOCKING]`-Run via HTTP API:-```bash-curl -X POST http://localhost:9867/profiles \- -H "Content-Type: application/json" \- -d '{"name":"bench-test","description":"Benchmark test profile"}'-```-Verify: Response contains the new profile ID.--### 6.3 Verify profile appears in list-Run `pinchtab profiles`.-Verify: "bench-test" appears in the list.--### 6.4 Start instance with profile-Run `pinchtab instance start --profile bench-test --mode headless`.-Record: The instance ID and port returned.--### 6.5 Use the profiled instance-Run `pinchtab --server http://localhost:<port> nav https://github.com`.-Verify: Navigation succeeds.--### 6.6 Stop the instance-Run `pinchtab instance stop <instance-id>`.-Verify: Instance no longer appears in `pinchtab instances`.--### 6.7 Delete test profile-Run via HTTP API:-```bash-curl -X DELETE http://localhost:9867/profiles/bench-test-```-Verify: Profile no longer appears in `pinchtab profiles`.-------## Task Group 7: Instance Management--Test multi-instance workflows.--### 7.1 Start first instance `[BLOCKING]`-Run `pinchtab instance start --mode headless`.-Record: Instance ID and port (e.g., inst_abc123, port 9868).--### 7.2 Start second instance-Run `pinchtab instance start --mode headless`.-Record: Instance ID and port (e.g., inst_def456, port 9869).--### 7.3 Verify both instances running-Run `pinchtab instances`.-Verify: Both instances appear in the list.--### 7.4 Navigate instances to different sites-Run `pinchtab --server http://localhost:9868 nav https://github.com`.-Run `pinchtab --server http://localhost:9869 nav https://wikipedia.org`.-Verify: Both complete without error.--### 7.5 Verify instance isolation-Run `pinchtab --server http://localhost:9868 text`.-Verify: Shows "GitHub".-Run `pinchtab --server http://localhost:9869 text`.-Verify: Shows "Wikipedia".--### 7.6 Stop all instances-Run `pinchtab instance stop --all`.-Verify: `pinchtab instances` shows no running instances.-------## Task Group 8: Diff Snapshots--Test incremental snapshot diffing.--### 8.1 Navigate to interactive page `[BLOCKING]`-Run `pinchtab nav https://www.wikipedia.org`.-Verify: Page loads.--### 8.2 Take baseline snapshot-Run `pinchtab snap -i -c`.-Record: The refs visible.--### 8.3 Make a change-Identify the search input ref.-Run `pinchtab fill <search-ref> "Test query"`.-Verify: Command completes.--### 8.4 Take diff snapshot-Run `pinchtab snap -d -i -c`.-Verify: Output indicates the changed element or shows delta.--### 8.5 Click a language link-Click one of the language links.-Run `pinchtab snap -d -i -c` again.-Verify: Diff shows the page change.-------## Task Group 9: Error Handling--Test graceful handling of errors.--### 9.1 Navigate to non-existent page-Run `pinchtab nav https://this-domain-does-not-exist-12345.com`.-Verify: Command returns an error (not a crash).-Record: The error message.--### 9.2 Click non-existent ref-Run `pinchtab nav https://github.com`.-Run `pinchtab click e999`.-Verify: Command returns an error about missing ref.--### 9.3 Fill non-existent selector-Run `pinchtab fill "#nonexistent-element-12345" "test"`.-Verify: Command returns a selector not found error.--### 9.4 Invalid command syntax-Run `pinchtab nav` (missing URL).-Verify: Command returns a usage error.--### 9.5 Server not running (optional)-Stop the server if safe to do so.-Run `pinchtab nav https://github.com`.-Verify: Command returns connection refused or similar error.-Restart the server.-------## Task Group 10: Performance Baselines--Test timing under normal conditions. Run each 3 times and record average.--### 10.1 Navigation speed-Time: `pinchtab nav https://github.com`-Record: Average duration across 3 runs.--### 10.2 Snapshot speed-Navigate to github.com first.-Time: `pinchtab snap -i -c`-Record: Average duration.--### 10.3 Text extraction speed-Time: `pinchtab text`-Record: Average duration.--### 10.4 Click speed-Get a ref from snapshot.-Time: `pinchtab click <ref>`-Record: Average duration.--### 10.5 Screenshot speed-Time: `pinchtab screenshot -o /tmp/bench-perf.png`-Record: Average duration.-------## Task Group 11: Token Efficiency (Optional)--Measure token usage for agent context optimization.--### 11.1 Full snapshot token count-Run `pinchtab nav https://news.ycombinator.com`.-Run `pinchtab snap` and count output tokens/characters.-Record: Character count of full snapshot.--### 11.2 Interactive-only snapshot-Run `pinchtab snap -i` and count output.-Record: Character count, calculate reduction %.--### 11.3 Interactive + compact snapshot-Run `pinchtab snap -i -c` and count output.-Record: Character count, calculate reduction %.--### 11.4 Text-only extraction-Run `pinchtab text` and count output.-Record: Character count vs snapshot.--### 11.5 Token-limited snapshot-Run `pinchtab snap --max-tokens 500`.-Verify: Output is truncated to approximately that budget.-------## Task Group 12: Login Flow Simulation--Test authentication workflows with a practice site.-Note: Using dedicated test automation sites that are designed for this purpose.--### 12.1 Navigate to practice login page `[BLOCKING]`-Run `pinchtab nav https://practicetestautomation.com/practice-test-login/`.-Alternative: `pinchtab nav https://the-internet.herokuapp.com/login`-Verify: Page loads with login form.--### 12.2 Snapshot login form-Run `pinchtab snap -i -c`.-Verify: Username and password fields visible with refs.--### 12.3 Enter valid credentials-Run `pinchtab fill "#username" "student"`.-Run `pinchtab fill "#password" "Password123"`.-Verify: Both commands complete.--### 12.4 Submit login-Identify the submit button ref.-Run `pinchtab click --wait-nav <submit-ref>`.-Verify: Page navigates.--### 12.5 Verify successful login-Run `pinchtab text`.-Verify: Output contains "Logged In Successfully" or similar success message.--### 12.6 Test invalid credentials-Navigate back to the login page.-Fill with invalid credentials ("wrong", "wrong").-Submit and verify error message appears.-------## Task Group 13: Dynamic Content--Test handling of JavaScript-rendered content.-Note: the-internet.herokuapp.com is a test automation practice site, purpose-built for these tests.--### 13.1 Navigate to dynamic page `[BLOCKING]`-Run `pinchtab nav https://the-internet.herokuapp.com/dynamic_loading/1`.-Verify: Page loads.--### 13.2 Snapshot initial state-Run `pinchtab snap -i -c`.-Verify: "Start" button is visible.--### 13.3 Click to trigger loading-Click the Start button.-Wait 2 seconds.--### 13.4 Verify dynamic content loaded-Run `pinchtab text`.-Verify: Output contains "Hello World!" (the dynamically loaded text).--### 13.5 Handle loading states-Run `pinchtab snap -d -i -c`.-Record: Whether the diff shows the new content correctly.-------## Task Group 14: Dropdown and Select--Test complex form controls.--### 14.1 Navigate to dropdown page `[BLOCKING]`-Run `pinchtab nav https://the-internet.herokuapp.com/dropdown`.-Verify: Page loads with dropdown.--### 14.2 Snapshot dropdown-Run `pinchtab snap -i -c`.-Verify: Dropdown element has a ref.--### 14.3 Select option by value-Run `pinchtab select <dropdown-ref> "1"`.-Verify: Command completes.--### 14.4 Verify selection-Run `pinchtab snap -i -c`.-Verify: Option 1 is now selected.--### 14.5 Select different option-Run `pinchtab select <dropdown-ref> "2"`.-Verify: Selection changed to Option 2.-------## Task Group 15: Hover and Tooltips--Test hover interactions.--### 15.1 Navigate to hover page `[BLOCKING]`-Run `pinchtab nav https://the-internet.herokuapp.com/hovers`.-Verify: Page loads with user avatars.--### 15.2 Snapshot before hover-Run `pinchtab snap -i -c`.-Record: Initial state refs.--### 15.3 Hover over first avatar-Identify the first avatar image ref.-Run `pinchtab hover <avatar-ref>`.-Verify: Command completes.--### 15.4 Verify hover content appeared-Run `pinchtab snap -d -i -c`.-Verify: New content visible (user name, "View profile" link).--### 15.5 Hover over second avatar-Hover over the second avatar.-Verify: Different user info appears.-------## Task Group 16: Scrolling--Test page scrolling capabilities.--### 16.1 Navigate to long page `[BLOCKING]`-Run `pinchtab nav https://the-internet.herokuapp.com/infinite_scroll`.-Verify: Page loads.--### 16.2 Get initial content-Run `pinchtab text`.-Record: Initial content length.--### 16.3 Scroll down by pixels-Run `pinchtab scroll 500`.-Verify: Command completes.--### 16.4 Verify more content loaded-Run `pinchtab text`.-Verify: Content length increased (infinite scroll loaded more).--### 16.5 Scroll to element-Navigate to a page with a footer.-Run `pinchtab scroll "text:Footer"` or equivalent selector.-Verify: Footer is now in view.-------## Task Group 17: File Download--Test download capabilities.--### 17.1 Navigate to download page `[BLOCKING]`-Run `pinchtab nav https://the-internet.herokuapp.com/download`.-Verify: Page loads with download links.--### 17.2 Identify a download link-Run `pinchtab snap -i -c`.-Identify a file link (e.g., .txt or .pdf file).--### 17.3 Download the file-Run `pinchtab download <file-url> -o /tmp/pinchtab-bench-download.txt`.-Verify: File exists at that path.--### 17.4 Verify file content-Check that the downloaded file has content.-Record: File size.--### 17.5 Cleanup-Remove the test file.-------## Task Group 18: iFrame Handling--Test interaction with embedded frames.--### 18.1 Navigate to iframe page `[BLOCKING]`-Run `pinchtab nav https://the-internet.herokuapp.com/iframe`.-Verify: Page loads with TinyMCE editor in iframe.--### 18.2 Snapshot main frame-Run `pinchtab snap -i -c`.-Record: What refs are visible in main frame.--### 18.3 Access iframe content-Use appropriate command to interact with iframe content.-Record: Method used and whether it succeeded.--### 18.4 Type in iframe editor-If iframe is accessible, type "Benchmark test content" in the editor.-Verify: Text appears in the editor.-------## Task Group 19: Alerts and Dialogs--Test JavaScript alert/confirm/prompt handling.--### 19.1 Navigate to alerts page `[BLOCKING]`-Run `pinchtab nav https://the-internet.herokuapp.com/javascript_alerts`.-Verify: Page loads with alert buttons.--### 19.2 Trigger alert-Click the "Click for JS Alert" button.-Record: How PinchTab handles the alert.--### 19.3 Accept alert-Dismiss or accept the alert.-Verify: `pinchtab text` shows "You successfully clicked an alert".--### 19.4 Handle confirm dialog-Click "Click for JS Confirm".-Accept the confirm.-Verify: Result text shows "Ok".--### 19.5 Handle confirm cancel-Trigger confirm again.-Cancel it.-Verify: Result text shows "Cancel".-------## Task Group 20: Semantic Search (find command)--Test natural language element finding.--### 20.1 Navigate to complex page `[BLOCKING]`-Run `pinchtab nav https://news.ycombinator.com`.-Verify: Page loads.--### 20.2 Find login link semantically-Run `pinchtab find "login link"`.-Verify: Returns a ref or selector for the login link.--### 20.3 Find search box-Run `pinchtab find --ref-only "search input"`.-Verify: Returns a ref.--### 20.4 Find navigation element-Run `pinchtab find "link to submit new story"`.-Verify: Returns relevant element.--### 20.5 Compare to manual snapshot-Run `pinchtab snap -i -c`.-Verify: The semantic find results match elements in the snapshot.-------## Task Group 21: Challenge/CAPTCHA Detection--Test challenge solver capabilities (requires appropriate site).--### 21.1 Check available solvers-Run `curl http://localhost:9867/solvers`.-Record: Which solvers are registered.--### 21.2 Navigate to protected page (if available)-If a Cloudflare-protected test site is available, navigate to it.-Otherwise mark this group as SKIPPED.--### 21.3 Detect challenge-Run `pinchtab text`.-Check if output indicates a challenge page ("Just a moment...", etc.).--### 21.4 Attempt solve-Run:-```bash-curl -X POST http://localhost:9867/solve \- -H 'Content-Type: application/json' \- -d '{"maxAttempts": 3}'-```-Record: Response (solved, attempts, solver used).--### 21.5 Verify page loaded-Run `pinchtab text`.-Verify: Real page content now visible.-------## Task Group 22: Agent Identity--Test agent attribution features.--### 22.1 Set agent identity via CLI-Run `pinchtab --agent-id "bench-agent-1" nav https://github.com`.-Verify: Command completes.--### 22.2 Verify activity recorded-Run `curl "http://localhost:9867/api/activity?agentId=bench-agent-1"`.-Verify: Activity entries show for that agent.--### 22.3 Use different agent ID-Run `pinchtab --agent-id "bench-agent-2" nav https://wikipedia.org`.-Verify: Command completes.--### 22.4 Verify separate activity streams-Query activity for both agents.-Verify: Each has separate activity entries.--### 22.5 Test header-based identity-Run:-```bash-curl -X POST http://localhost:9867/navigate \- -H "X-Agent-Id: bench-agent-http" \- -H "Content-Type: application/json" \- -d '{"url":"https://github.com"}'-```-Verify: Activity recorded under "bench-agent-http".-------## Task Group 23: Block Images/Ads Mode--Test bandwidth-saving navigation modes.--### 23.1 Navigate with images blocked `[BLOCKING]`-Run `pinchtab nav https://news.ycombinator.com --block-images`.-Verify: Page loads successfully.--### 23.2 Verify page still functional-Run `pinchtab snap -i -c`.-Verify: Interactive elements are present.--### 23.3 Compare loading time-Time navigation with and without `--block-images`.-Record: Time difference.--### 23.4 Navigate with ads blocked-Run `pinchtab nav https://github.com --block-ads`.-Verify: Page loads successfully.-------## Summary Template--After completing all groups, compile results using this template:-------## Results Template--Copy this section to a new file `BENCHMARK_RESULTS_<date>.md`:--```markdown-# PinchTab Benchmark Results--**Date**: YYYY-MM-DD-**Agent/Model**: (e.g., Claude Sonnet 3.5)-**PinchTab Version**: (output of `pinchtab version`)-**Environment**: (OS, Chrome version)-------## Cost Summary--| Metric | Value |-|--------|-------|-| Total input tokens | X |-| Total output tokens | X |-| Total tool calls | X |-| Total estimated cost | $X.XX |-| Cost per task group (avg) | $X.XX |-------## Pass/Fail Summary--| Group | Name | Pass | Fail | Skip | Blocking? |-|-------|------|------|------|------|-----------|-| 1 | Basic Navigation | X/X | X | X | No |-| 2 | Element Interaction | X/X | X | X | No |-| 3 | Multi-Tab Workflow | X/X | X | X | No |-| 4 | Search Flow | X/X | X | X | No |-| 5 | Screenshot/PDF Export | X/X | X | X | No |-| 6 | Profile Management | X/X | X | X | No |-| 7 | Instance Management | X/X | X | X | No |-| 8 | Diff Snapshots | X/X | X | X | No |-| 9 | Error Handling | X/X | X | X | No |-| 10 | Performance Baselines | X/X | X | X | No |-| 11 | Token Efficiency | X/X | X | X | No |-| 12 | Login Flow | X/X | X | X | No |-| 13 | Dynamic Content | X/X | X | X | No |-| 14 | Dropdown/Select | X/X | X | X | No |-| 15 | Hover/Tooltips | X/X | X | X | No |-| 16 | Scrolling | X/X | X | X | No |-| 17 | File Download | X/X | X | X | No |-| 18 | iFrame Handling | X/X | X | X | No |-| 19 | Alerts/Dialogs | X/X | X | X | No |-| 20 | Semantic Search | X/X | X | X | No |-| 21 | Challenge/CAPTCHA | X/X | X | X | No |-| 22 | Agent Identity | X/X | X | X | No |-| 23 | Block Images/Ads | X/X | X | X | No |--**Total**: X/X passed (X%)-------## Performance Baselines--| Operation | Avg (ms) | Min (ms) | Max (ms) |-|-----------|----------|----------|----------|-| Navigation | X | X | X |-| Snapshot (-i -c) | X | X | X |-| Text extraction | X | X | X |-| Click | X | X | X |-| Fill | X | X | X |-| Screenshot | X | X | X |-| PDF export | X | X | X |-------## Token Efficiency--| Mode | Output Size | vs Full Snapshot |-|------|-------------|------------------|-| Full snapshot | X chars | 100% |-| Interactive only (-i) | X chars | X% |-| Interactive + compact (-i -c) | X chars | X% |-| Text only | X chars | X% |-| Token-limited (500) | X chars | X% |-------## Blocking Failures--List any steps marked `[BLOCKING]` that failed:--1. Group X, Step X.Y: (reason)-2. ...-------## Notable Observations--- (Any interesting findings, edge cases, or suggestions)-- (Performance anomalies)-- (Feature requests based on benchmark)-------## Error Messages Quality--Rate 1-5 for each error type encountered:--| Error Type | Clarity (1-5) | Actionable (1-5) | Example |-|------------|---------------|------------------|---------|-| Missing ref | X | X | "..." |-| Connection refused | X | X | "..." |-| Timeout | X | X | "..." |-| Invalid selector | X | X | "..." |-------## Detailed Step Results--(Paste step-by-step results here)-```-------## Benchmark Scope--This benchmark covers:-- **Navigation**: forward, back, reload, block-images, block-ads-- **Observation**: snap, text, diff, find (semantic)-- **Interaction**: click, fill, type, press, select, hover, scroll-- **Tabs and instances**: multi-tab, multi-instance isolation-- **Profiles**: create, use, delete-- **Export**: screenshot (PNG/JPEG), PDF (portrait/landscape)-- **Downloads**: file download-- **Dynamic content**: JS-rendered, loading states-- **Form controls**: dropdowns, radio buttons, checkboxes-- **Frames**: iframe interaction-- **Dialogs**: alert, confirm, prompt-- **Error resilience**: invalid refs, missing elements, network errors-- **Performance**: timing baselines-- **Token efficiency**: output size comparisons-- **Challenge solving**: CAPTCHA/Cloudflare detection-- **Agent identity**: attribution and activity trackingdiff --git a/tests/benchmark/AGENT_TASKS.md b/tests/benchmark/AGENT_TASKS.mdindex a36f3bfe..b483ac87 100644--- a/tests/benchmark/AGENT_TASKS.md+++ b/tests/benchmark/AGENT_TASKS.md@@ -18,17 +18,47 @@ Natural language tasks to test how well an agent uses PinchTab from skill docs a----## Group 0: Setup Verification+## Group 0: Setup & Diagnosis-### 0.1 Confirm PinchTab is running-Check that the PinchTab server is healthy and a Chrome instance is active.+### 0.1 Server reachable+Check that the PinchTab server is healthy.-**Verify**: Server responds with `status: ok` and at least one running instance.+**Verify**: Server responds with `status: ok`.-### 0.2 Navigate to fixtures and confirm reachable-Navigate to `http://fixtures/` and confirm the benchmark fixtures server is reachable.+### 0.2 Auth is required+Make a request to the server WITHOUT an auth token and confirm it is rejected.-**Verify**: Page title contains "Benchmark" or "PinchTab".+**Verify**: Response is HTTP 401 (or other auth-error status).++### 0.3 Auth works with token+Repeat the same request WITH the bearer token and confirm it succeeds.++**Verify**: Response is HTTP 200.++### 0.4 Instance available+Confirm at least one Chrome instance is running. If none exist, start one.++**Verify**: Health response shows `defaultInstance.status == "running"` (or after starting one, the new instance is running).++### 0.5 List existing tabs+Get the current list of open tabs.++**Verify**: A list (possibly empty) is returned without error.++### 0.6 Clean stale tabs+If any tabs from previous runs are open, close them so the benchmark starts from a known state.++**Verify**: After cleanup, the tab list is empty (or only contains a single about:blank tab).++### 0.7 Network reach to target+Navigate to `http://fixtures/` and confirm the fixtures server is reachable from PinchTab.++**Verify**: Navigate returns successfully and the page contains benchmark content.++### 0.8 Capture initial tab ID+Save the tab ID returned by the navigate in 0.7. Use this tab ID for all subsequent tasks to avoid creating new tabs.++**Verify**: A tab ID was captured and matches what `GET /tabs` reports.---@@ -111,7 +141,7 @@ After submitting, if the form is still accessible (or navigate back), verify you## Group 4: SPA State Management### 4.1 Read initial app state-Navigate to `http://fixtures/spa.html` (always navigate fresh — do not reuse a cached page). Read the current task list — how many tasks exist, how many are active vs done?+Navigate to `http://fixtures/spa.html?reset=1` (the `?reset=1` query param clears any previous localStorage state so the SPA starts with its default 3 tasks). Read the current task list — how many tasks exist, how many are active vs done?**Verify**: Found 3 total, 2 active, 1 done (verify `TASK_STATS_TOTAL_3_ACTIVE_2_DONE_1`).@@ -219,7 +249,7 @@ In the modal, select "Dark" from the theme dropdown (`#theme-select`), then clic## Group 11: State Persistence & Page Reload### 11.1 Add an item and verify after page reload-Navigate to `http://fixtures/spa.html`. Add a task titled exactly "Persistent Task Test". Then reload by navigating to `http://fixtures/spa.html` again.+Navigate to `http://fixtures/spa.html?reset=1` (starts with clean state). Add a task titled exactly "Persistent Task Test". Then navigate away to `http://fixtures/` and back to `http://fixtures/spa.html` (WITHOUT the reset param) to verify localStorage persistence.**Verify**: After reload, the task still appears in the list (`TASK_PERSISTENT_TEST_FOUND_AFTER_RELOAD`).@@ -291,11 +321,76 @@ Report which language has the most features and name 1 feature unique to each.---+## Group 16: Hover & Tooltips++### 16.1 Hover reveals hidden content+Navigate to `http://fixtures/hovers.html`. Hover over the first avatar to reveal its hidden info.++**Verify**: After hovering, snapshot contains `HOVER_REVEALED_USER_1`.++### 16.2 Hover swap+Hover over the second avatar.++**Verify**: Snapshot contains `HOVER_REVEALED_USER_2`.++---++## Group 17: Scrolling++### 17.1 Scroll by pixels+Navigate to `http://fixtures/scroll.html`. Scroll down approximately 1500 pixels.++**Verify**: Snapshot or text contains `SCROLL_MIDDLE_MARKER` (the mid-page marker is now in view).++### 17.2 Scroll to footer+Continue scrolling until the footer is visible (or scroll directly to the footer element).++**Verify**: Snapshot contains `SCROLL_REACHED_FOOTER`.++---++## Group 18: File Download++### 18.1 Download a file+Download `http://fixtures/download-sample.txt` to a local file.++**Verify**: The downloaded file exists and its content includes `DOWNLOAD_FILE_CONTENT_VERIFIED`.++---++## Group 19: iFrame++### 19.1 Read iframe content+Navigate to `http://fixtures/iframe.html` and extract content from inside the embedded iframe.++**Verify**: The iframe's inner content includes `IFRAME_INNER_CONTENT_LOADED`.++### 19.2 Type into iframe input+Inside the iframe, fill the input with the text "Hello World" and click the Save button.++**Verify**: The iframe shows `IFRAME_INPUT_RECEIVED_HELLO_WORLD`.++---++## Group 20: Dialogs++### 20.1 Accept alert+Navigate to `http://fixtures/alerts.html`. Click the "Click for Alert" button and dismiss the alert.++**Verify**: The page result contains `DIALOG_ALERT_DISMISSED`.++### 20.2 Cancel confirm+Click the "Click for Confirm" button and cancel the confirm dialog.++**Verify**: The page result contains `DIALOG_CONFIRM_CANCELLED`.++---+## Summary| Group | Tasks | Description ||-------|-------|-------------|-| 0 | 2 | Setup Verification |+| 0 | 8 | Setup & Diagnosis || 1 | 6 | Reading & Extracting Content || 2 | 3 | Search & Dynamic Interaction || 3 | 2 | Complex Form |@@ -311,8 +406,13 @@ Report which language has the most features and name 1 feature unique to each.| 13 | 2 | Form State & Multi-Step Submission || 14 | 2 | Dynamic Content Loading || 15 | 2 | Complex Data Extraction & Aggregation |+| 16 | 2 | Hover & Tooltips |+| 17 | 2 | Scrolling |+| 18 | 1 | File Download |+| 19 | 2 | iFrame |+| 20 | 2 | Dialogs |-**Total: 39 tasks** (expanded from 27)+**Total: 54 tasks**## Key Differences from Baselinediff --git a/tests/benchmark/BENCHMARK_TASKS.md b/tests/benchmark/BASELINE_TASKS.mdsimilarity index 80%rename from tests/benchmark/BENCHMARK_TASKS.mdrename to tests/benchmark/BASELINE_TASKS.mdindex 9931d2e6..88e1007b 100644--- a/tests/benchmark/BENCHMARK_TASKS.md+++ b/tests/benchmark/BASELINE_TASKS.md@@ -53,16 +53,63 @@ tab ID captured in step 0.2.----## Group 0: Setup+## Group 0: Setup & Diagnosis-### 0.1 Health check+### 0.1 Server reachable```bashcurl -sf http://localhost:9867/health \-H "Authorization: Bearer benchmark-token"```-**Pass if**: `status == "ok"`, `authRequired == true`, `instances >= 1`, `defaultInstance.status == "running"`.+**Pass if**: `status == "ok"` in response body.-### 0.2 Fixtures reachable+### 0.2 Auth required+```bash+curl -s -o /dev/null -w "%{http_code}" http://localhost:9867/health+```+**Pass if**: HTTP status is `401` (auth rejected without token).++### 0.3 Auth works with token+```bash+curl -s -o /dev/null -w "%{http_code}" http://localhost:9867/health \+ -H "Authorization: Bearer benchmark-token"+```+**Pass if**: HTTP status is `200`.++### 0.4 Instance available+```bash+curl -sf http://localhost:9867/health \+ -H "Authorization: Bearer benchmark-token"+```+**Pass if**: Response contains `defaultInstance.status == "running"`. If not, run:+```bash+curl -X POST http://localhost:9867/instances/start \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{}'+```+And re-verify the new instance is running.++### 0.5 List existing tabs+```bash+curl -sf http://localhost:9867/tabs \+ -H "Authorization: Bearer benchmark-token"+```+**Pass if**: Returns a JSON array (possibly empty) without error.++### 0.6 Clean stale tabs+For each tab returned by step 0.5, close it:+```bash+curl -X POST http://localhost:9867/tabs/TAB_ID/close \+ -H "Authorization: Bearer benchmark-token"+```+Then verify cleanup:+```bash+curl -sf http://localhost:9867/tabs \+ -H "Authorization: Bearer benchmark-token"+```+**Pass if**: After cleanup, the tab list is empty or contains only an about:blank tab.++### 0.7 Network reach to target```bashcurl -sf -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \@@ -76,6 +123,13 @@ curl -sf "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=50```**Pass if**: Navigate returns HTTP 200 AND snapshot contains `VERIFY_HOME_LOADED_12345`.+### 0.8 Capture initial tab ID+```bash+curl -sf http://localhost:9867/tabs \+ -H "Authorization: Bearer benchmark-token"+```+**Pass if**: The captured `TAB_ID` from step 0.7 appears in the tabs list.+---## Group 1: Reading & Extracting@@ -286,7 +340,7 @@ curl "http://localhost:9867/tabs/TAB_ID/snapshot?filter=interactive&format=compacurl -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/spa.html"}'+ -d '{"tabId":"TAB_ID","url":"http://fixtures/spa.html?reset=1"}'curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \-H "Authorization: Bearer benchmark-token"@@ -574,11 +628,13 @@ curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \### 11.1 State after reload```bash+# Start fresh (reset param clears localStorage on this load only)curl -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"tabId":"TAB_ID","url":"http://fixtures/spa.html"}'+ -d '{"tabId":"TAB_ID","url":"http://fixtures/spa.html?reset=1"}'+# Add the persistent taskcurl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \@@ -589,6 +645,12 @@ curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Content-Type: application/json" \-d '{"kind":"click","selector":"#add-task-btn"}'+# Navigate away then back WITHOUT reset param — state should persist in localStorage+curl -X POST http://localhost:9867/navigate \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"tabId":"TAB_ID","url":"http://fixtures/"}'+curl -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \@@ -840,11 +902,151 @@ curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000"---+## Group 16: Hover & Tooltips++### 16.1 Hover reveals info+```bash+curl -X POST http://localhost:9867/navigate \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"tabId":"TAB_ID","url":"http://fixtures/hovers.html"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"hover","selector":"#avatar-1"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \+ -H "Authorization: Bearer benchmark-token"+```+**Pass if**: Snapshot contains `HOVER_REVEALED_USER_1`.++### 16.2 Hover swap+```bash+curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"hover","selector":"#avatar-2"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \+ -H "Authorization: Bearer benchmark-token"+```+**Pass if**: Snapshot contains `HOVER_REVEALED_USER_2`.++---++## Group 17: Scrolling++### 17.1 Scroll by pixels+```bash+curl -X POST http://localhost:9867/navigate \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"tabId":"TAB_ID","url":"http://fixtures/scroll.html"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/evaluate \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"expression":"window.scrollTo(0, 1500); document.querySelector(\"#middle-block\").innerText;"}'+```+**Pass if**: Response value contains `SCROLL_MIDDLE_MARKER`.++### 17.2 Scroll to footer+```bash+curl -X POST http://localhost:9867/tabs/TAB_ID/evaluate \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"expression":"document.querySelector(\"#footer\").scrollIntoView(); document.querySelector(\"#footer\").innerText;"}'+```+**Pass if**: Response value contains `SCROLL_REACHED_FOOTER`.++---++## Group 18: File Download++### 18.1 Download a file+```bash+curl -X POST http://localhost:9867/navigate \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"tabId":"TAB_ID","url":"http://fixtures/download-sample.txt"}'++curl "http://localhost:9867/tabs/TAB_ID/text" \+ -H "Authorization: Bearer benchmark-token"+```+**Pass if**: Response text contains `DOWNLOAD_FILE_CONTENT_VERIFIED`.++**Note**: PinchTab's `/download` endpoint blocks private/internal IPs as a safety measure. For local fixture downloads, navigate to the file URL and read body text instead.++---++## Group 19: iFrame++### 19.1 Read iframe content+```bash+curl -X POST http://localhost:9867/navigate \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"tabId":"TAB_ID","url":"http://fixtures/iframe.html"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \+ -H "Authorization: Bearer benchmark-token"+```+**Pass if**: Snapshot contains `IFRAME_INNER_CONTENT_LOADED`.++### 19.2 Type into iframe input+```bash+curl -X POST http://localhost:9867/tabs/TAB_ID/evaluate \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"expression":"const f=document.getElementById(\"content-frame\").contentDocument; f.getElementById(\"iframe-input\").value=\"Hello World\"; f.getElementById(\"iframe-submit\").click(); f.getElementById(\"iframe-result\").textContent;"}'+```+**Pass if**: Response value contains `IFRAME_INPUT_RECEIVED_HELLO_WORLD`.++**Note**: PinchTab's `/action` selectors don't reach inside iframes. Use `/evaluate` with `contentDocument` to interact with iframe content.++---++## Group 20: Dialogs++### 20.1 Accept alert+```bash+curl -X POST http://localhost:9867/navigate \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"tabId":"TAB_ID","url":"http://fixtures/alerts.html"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"click","selector":"#alert-btn","dialogAction":"accept"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \+ -H "Authorization: Bearer benchmark-token"+```+**Pass if**: Snapshot contains `DIALOG_ALERT_DISMISSED`.++### 20.2 Cancel confirm+```bash+curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"click","selector":"#confirm-btn","dialogAction":"dismiss"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \+ -H "Authorization: Bearer benchmark-token"+```+**Pass if**: Snapshot contains `DIALOG_CONFIRM_CANCELLED`.++**Note**: Use `dialogAction` on the click action to auto-accept or auto-dismiss a JS dialog that the click opens. Without it, the click would hang until `/dialog` is called from a separate request.++---+## Summary| Group | Tasks | Description ||-------|-------|-------------|-| 0 | 2 | Setup |+| 0 | 8 | Setup & Diagnosis || 1 | 6 | Reading & Extracting || 2 | 3 | Search & Dynamic || 3 | 2 | Form |@@ -860,8 +1062,13 @@ curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000"| 13 | 2 | Form Validation || 14 | 2 | Dynamic Content || 15 | 2 | Data Aggregation |+| 16 | 2 | Hover & Tooltips |+| 17 | 2 | Scrolling |+| 18 | 1 | File Download |+| 19 | 2 | iFrame |+| 20 | 2 | Dialogs |-**Total: 39 tasks**+**Total: 54 tasks**## Verification Stringsdiff --git a/tests/benchmark/CRON_TASK.md b/tests/benchmark/CRON_TASK.mdindex d165e42e..00eaf9c6 100644--- a/tests/benchmark/CRON_TASK.md+++ b/tests/benchmark/CRON_TASK.md@@ -17,14 +17,14 @@ git checkout feat/benchmark && git pull --rebase origin feat/benchmark```### Step 2 — Run Baseline Benchmark-Execute BENCHMARK_TASKS.md exactly as written.+Execute BASELINE_TASKS.md exactly as written.- Token: `benchmark-token`- Record each step: `./record-step.sh <group> <step> <pass|fail> <in> <out> "notes"`- For failures: include HTTP status, expected string, actual response in notes### Step 3 — Run Agent BenchmarkExecute AGENT_TASKS.md using SKILL.md as your only guide.-- **Do NOT look at BENCHMARK_TASKS.md**+- **Do NOT look at BASELINE_TASKS.md**- Figure out commands from skill documentation- Record: `./record-agent-step.sh <group> <step> <pass|fail> <in> <out> "commands used" "result"`- Log every curl/command executed@@ -46,7 +46,7 @@ Compare results and classify each agent failure:Priority order:1. **API Bug** → Fix PinchTab Go code, commit as `fix: <description>`2. **Skill Gap** → Improve SKILL.md with clearer guidance or example, commit as `docs(skill): <description>`-3. **Test Ambiguity** → Fix fixture HTML or BENCHMARK_TASKS.md verification, commit as `test: <description>`+3. **Test Ambiguity** → Fix fixture HTML or BASELINE_TASKS.md verification, commit as `test: <description>`4. **Agent Task Clarity** → Improve AGENT_TASKS.md phrasing, commit as `test(agent): <description>`5. **No Gaps Found** → Add 2-3 new test cases covering uncovered scenarios, commit as `test: add cases for <scenario>`diff --git a/tests/benchmark/OPTIMIZATION_LOOP.md b/tests/benchmark/OPTIMIZATION_LOOP.mdindex 054401d8..2544ebd6 100644--- a/tests/benchmark/OPTIMIZATION_LOOP.md+++ b/tests/benchmark/OPTIMIZATION_LOOP.md@@ -15,24 +15,12 @@ curl -sf -H "Authorization: Bearer benchmark-token" http://localhost:9867/health```┌─────────────────────────────────────────────────────────────┐-│ 1a. RUN BASELINE BENCHMARK [Sub-agent A] │-│ - Only runs when: │-│ • No previous baseline exists, OR │-│ • Changes affect: server code, fixtures, or │-│ BENCHMARK_TASKS.md │-│ - Otherwise: reuse last baseline results │-│ - Load BENCHMARK_TASKS.md │-│ - Execute 68 curl commands sequentially │-│ - Check pass/fail per documented conditions │-│ - Record via: record-step.sh --type baseline │-│ - Output: results/baseline_<timestamp>.json │-├─────────────────────────────────────────────────────────────┤-│ 1b. RUN AGENT BENCHMARK [Sub-agent B] │-│ - Runs every iteration │-│ - Load AGENT_TASKS.md + skills/pinchtab/SKILL.md │-│ - Execute 39 tasks using only skill docs │-│ - Record via: record-step.sh --type agent --tokens │-│ - Output: results/agent_benchmark_<timestamp>.json │+│ 1. RUN AGENT BENCHMARK [Sub-agent] │+│ - Runs every iteration │+│ - Load AGENT_TASKS.md + skills/pinchtab/SKILL.md │+│ - Execute tasks using only skill docs │+│ - Record via: record-step.sh --type agent │+│ - Output: results/agent_benchmark_<timestamp>.json │└─────────────────────────────────────────────────────────────┘↓┌─────────────────────────────────────────────────────────────┐@@ -79,14 +67,36 @@ curl -sf -H "Authorization: Bearer benchmark-token" http://localhost:9867/health| Step | Who | What ||------|-----|------|| Pre-req | **Human** | Start Docker, verify health |-| 1a Baseline | **Sub-agent A** | Execute 68 curl commands (skip if baseline exists and no server/fixture changes) |-| 1b Agent | **Sub-agent B** | Execute 39 tasks from skill docs, record results |+| 1 Agent | **Sub-agent** | Execute agent tasks from skill docs, record results || 2 Regression | **Main agent** | Compare against best_score.txt || 3 Analyze | **Main agent** | Diff baseline vs agent, find patterns || 4 Propose | **Main agent → Human** | Suggest 1 fix, get approval || 5 Implement | **Main agent** | Make the change, leave uncommitted || 6 Log | **Main agent** | Append to optimization_log.md |+## Baseline is a Prerequisite, Not Part of the Loop++The baseline suite (`BASELINE_TASKS.md`) validates that the benchmark infrastructure itself works — fixtures serve correctly, PinchTab APIs behave as expected, and the test conditions are reachable. It's deterministic (no LLM involved), so once it's at 100% it stays at 100% unless something changes underneath it.++**The loop does NOT re-run the baseline each iteration.** Instead, run the baseline manually when any of these change:++- PinchTab server code (anything in `internal/` that affects the HTTP API)+- Fixtures (anything in `tests/benchmark/fixtures/`)+- `BASELINE_TASKS.md` itself (adding/modifying tests)+- `docker-compose.yml` or the Docker image++**Workflow for adding a new test case:**++1. Write the new test in `BASELINE_TASKS.md` (curl commands + pass condition)+2. Write the matching task in `AGENT_TASKS.md` (natural language)+3. Write the summary row in `TEST_CASES.md`+4. Add any new fixture files needed+5. **Run the baseline** — it must reach 100% before the case is considered valid+6. If baseline fails: fix the test/fixture/PinchTab until it passes+7. Commit the new case — now the agent loop can use it++Once baseline is green, the optimization loop only runs the agent suite.+## Regression DetectionEach run compares against `results/best_score.txt`, which tracks the highest pass rates achieved:diff --git a/tests/benchmark/README.md b/tests/benchmark/README.mdindex 5e770eb4..88bc3779 100644--- a/tests/benchmark/README.md+++ b/tests/benchmark/README.md@@ -31,8 +31,7 @@ If Docker build fails or is skipped, the benchmark is **INVALID**.| File | Purpose ||------|---------|| `../../skills/pinchtab/SKILL.md` | PinchTab skill (same as shipped product) |-| `BENCHMARK_TASKS.md` | Standalone task list (same as skill) |-| `AGENT_SCRIPT.md` | Extended: 100+ comprehensive tasks |+| `BASELINE_TASKS.md` | Standalone task list (same as skill) || `run-agent-benchmark.sh` | **Recommended**: Start Docker + spawn subagent || `run-benchmark.sh` | Start Docker environment only || `record-step.sh` | Record step results with token counts |@@ -114,7 +113,7 @@ Reports are generated in `results/`:For automated benchmarks, you can:-1. Parse `BENCHMARK_TASKS.md` for curl commands+1. Parse `BASELINE_TASKS.md` for curl commands2. Execute each command3. Parse responses for pass/fail4. Call `record-step.sh` with resultsdiff --git a/tests/benchmark/TEST_CASES.md b/tests/benchmark/TEST_CASES.mdindex 31b32ad9..a1c08a67 100644--- a/tests/benchmark/TEST_CASES.md+++ b/tests/benchmark/TEST_CASES.md@@ -1,12 +1,18 @@# PinchTab Benchmark Test Cases-39 test cases covering realistic browser automation scenarios against local fixture pages.+54 test cases covering realistic browser automation scenarios against local fixture pages.| # | Task | Description ||---|------|-------------|-| **Group 0: Setup** | | |-| 0.1 | Health check | Confirm PinchTab is running with active instance |-| 0.2 | Fixtures reachable | Navigate to fixtures root, confirm accessible |+| **Group 0: Setup & Diagnosis** | | |+| 0.1 | Server reachable | `GET /health` returns `status: ok` |+| 0.2 | Auth required | Request without token returns 401 |+| 0.3 | Auth works with token | Request with bearer token returns 200 |+| 0.4 | Instance available | At least one instance running, or start one |+| 0.5 | List existing tabs | `GET /tabs` returns array without error |+| 0.6 | Clean stale tabs | Close any leftover tabs from previous runs |+| 0.7 | Network reach to target | Navigate to fixtures root, get success |+| 0.8 | Capture initial tab ID | Save tab ID from 0.7 for subsequent tasks || **Group 1: Reading & Extracting** | | || 1.1 | Wiki categories | Extract category names + article counts from wiki index || 1.2 | Click a link | From wiki, click through to Go article |@@ -59,3 +65,17 @@| **Group 15: Data Aggregation** | | || 15.1 | Financial calc | Extract revenue + profit, calculate margin || 15.2 | Multi-page comparison | Compare features across 3 wiki language pages |+| **Group 16: Hover & Tooltips** | | |+| 16.1 | Hover reveals info | Hover first avatar, verify hidden content appears |+| 16.2 | Hover swap | Hover second avatar, verify different content appears |+| **Group 17: Scrolling** | | |+| 17.1 | Scroll by pixels | Scroll down 1500px, verify mid-page marker visible |+| 17.2 | Scroll to footer | Scroll to bottom, verify footer marker visible |+| **Group 18: File Download** | | |+| 18.1 | Download a file | Download sample.txt, verify content marker present |+| **Group 19: iFrame** | | |+| 19.1 | Read iframe content | Verify content from inside iframe is accessible |+| 19.2 | Type into iframe | Fill input inside iframe, verify saved value |+| **Group 20: Dialogs** | | |+| 20.1 | Accept alert | Trigger alert, accept, verify result marker |+| 20.2 | Cancel confirm | Trigger confirm dialog, cancel, verify cancelled marker |diff --git a/tests/benchmark/fixtures/alerts.html b/tests/benchmark/fixtures/alerts.htmlnew file mode 100644index 00000000..03bad9fc--- /dev/null+++ b/tests/benchmark/fixtures/alerts.html@@ -0,0 +1,37 @@+<!DOCTYPE html>+<html lang="en">+<head>+ <meta charset="UTF-8">+ <title>Alerts — PinchTab Benchmark</title>+</head>+<body>+ <h1>JavaScript Dialogs — VERIFY_ALERTS_PAGE_44441</h1>++ <button id="alert-btn">Click for Alert</button>+ <button id="confirm-btn">Click for Confirm</button>+ <button id="prompt-btn">Click for Prompt</button>++ <p id="result"></p>++ <script>+ document.getElementById('alert-btn').addEventListener('click', function() {+ alert('This is an alert');+ document.getElementById('result').textContent = 'DIALOG_ALERT_DISMISSED';+ });++ document.getElementById('confirm-btn').addEventListener('click', function() {+ const ok = confirm('Are you sure?');+ document.getElementById('result').textContent = ok ? 'DIALOG_CONFIRM_ACCEPTED' : 'DIALOG_CONFIRM_CANCELLED';+ });++ document.getElementById('prompt-btn').addEventListener('click', function() {+ const value = prompt('Enter your name:');+ if (value === null) {+ document.getElementById('result').textContent = 'DIALOG_PROMPT_CANCELLED';+ } else {+ document.getElementById('result').textContent = 'DIALOG_PROMPT_VALUE_' + value.replace(/\s/g, '_').toUpperCase();+ }+ });+ </script>+</body>+</html>diff --git a/tests/benchmark/fixtures/download-sample.txt b/tests/benchmark/fixtures/download-sample.txtnew file mode 100644index 00000000..c787f775--- /dev/null+++ b/tests/benchmark/fixtures/download-sample.txt@@ -0,0 +1,5 @@+DOWNLOAD_FILE_CONTENT_VERIFIED+This is a sample file used by the PinchTab benchmark to test file downloads.+Line 3.+Line 4.+END_OF_FILE_MARKERdiff --git a/tests/benchmark/fixtures/hovers.html b/tests/benchmark/fixtures/hovers.htmlnew file mode 100644index 00000000..f4e5e51c--- /dev/null+++ b/tests/benchmark/fixtures/hovers.html@@ -0,0 +1,49 @@+<!DOCTYPE html>+<html lang="en">+<head>+ <meta charset="UTF-8">+ <title>Hovers — PinchTab Benchmark</title>+ <style>+ .avatar { display: inline-block; margin: 20px; }+ .avatar-img { width: 100px; height: 100px; background: #ccc; border-radius: 50%; cursor: pointer; }+ .info { display: none; padding: 10px; background: #f0f0f0; margin-top: 5px; }+ </style>+</head>+<body>+ <h1>Hovers — VERIFY_HOVERS_PAGE_66666</h1>++ <div class="avatar" id="avatar-1" onmouseenter="showInfo(1)" onmouseleave="hideInfo(1)">+ <div class="avatar-img"></div>+ <div class="info" id="info-1"></div>+ </div>++ <div class="avatar" id="avatar-2" onmouseenter="showInfo(2)" onmouseleave="hideInfo(2)">+ <div class="avatar-img"></div>+ <div class="info" id="info-2"></div>+ </div>++ <div class="avatar" id="avatar-3" onmouseenter="showInfo(3)" onmouseleave="hideInfo(3)">+ <div class="avatar-img"></div>+ <div class="info" id="info-3"></div>+ </div>++ <script>+ const users = {+ 1: { name: 'Alice Anderson', marker: 'HOVER_REVEALED_USER_1' },+ 2: { name: 'Bob Brown', marker: 'HOVER_REVEALED_USER_2' },+ 3: { name: 'Carol Chen', marker: 'HOVER_REVEALED_USER_3' },+ };++ function showInfo(n) {+ const info = document.getElementById('info-' + n);+ info.innerHTML = '<p>name: ' + users[n].name + '</p><p>' + users[n].marker + '</p>';+ info.style.display = 'block';+ }++ function hideInfo(n) {+ // Keep content visible after hover so snapshots and verification can read it+ // (don't actually hide; the marker should persist for the next snapshot)+ }+ </script>+</body>+</html>diff --git a/tests/benchmark/fixtures/iframe-content.html b/tests/benchmark/fixtures/iframe-content.htmlnew file mode 100644index 00000000..b17afe2c--- /dev/null+++ b/tests/benchmark/fixtures/iframe-content.html@@ -0,0 +1,23 @@+<!DOCTYPE html>+<html lang="en">+<head>+ <meta charset="UTF-8">+ <title>iFrame Inner</title>+</head>+<body>+ <h2>Inner Frame — IFRAME_INNER_CONTENT_LOADED</h2>+ <form id="iframe-form">+ <label>Comment:</label>+ <input type="text" id="iframe-input" name="comment">+ <button type="button" id="iframe-submit">Save</button>+ </form>+ <p id="iframe-result"></p>++ <script>+ document.getElementById('iframe-submit').addEventListener('click', function() {+ const value = document.getElementById('iframe-input').value;+ document.getElementById('iframe-result').textContent = 'IFRAME_INPUT_RECEIVED_' + value.replace(/\s/g, '_').toUpperCase();+ });+ </script>+</body>+</html>diff --git a/tests/benchmark/fixtures/iframe.html b/tests/benchmark/fixtures/iframe.htmlnew file mode 100644index 00000000..22cebd06--- /dev/null+++ b/tests/benchmark/fixtures/iframe.html@@ -0,0 +1,12 @@+<!DOCTYPE html>+<html lang="en">+<head>+ <meta charset="UTF-8">+ <title>iFrame — PinchTab Benchmark</title>+</head>+<body>+ <h1>iFrame Test — VERIFY_IFRAME_PAGE_33331</h1>+ <p>The iframe below contains an editable form.</p>+ <iframe id="content-frame" src="iframe-content.html" width="600" height="400"></iframe>+</body>+</html>diff --git a/tests/benchmark/fixtures/scroll.html b/tests/benchmark/fixtures/scroll.htmlnew file mode 100644index 00000000..1705a94e--- /dev/null+++ b/tests/benchmark/fixtures/scroll.html@@ -0,0 +1,30 @@+<!DOCTYPE html>+<html lang="en">+<head>+ <meta charset="UTF-8">+ <title>Scroll — PinchTab Benchmark</title>+ <style>+ .block { padding: 100px; margin: 50px 0; background: #eef; }+ #footer { padding: 50px; background: #cce; margin-top: 200px; }+ </style>+</head>+<body>+ <h1>Scroll — VERIFY_SCROLL_PAGE_22221</h1>+ <p>SCROLL_TOP_MARKER</p>++ <div class="block">Block 1 — content</div>+ <div class="block">Block 2 — content</div>+ <div style="height: 800px;"></div>+ <div class="block" id="middle-block">+ <p>SCROLL_MIDDLE_MARKER</p>+ </div>+ <div style="height: 1500px;"></div>+ <div class="block">Block 4 — content</div>+ <div style="height: 800px;"></div>++ <footer id="footer">+ <h2>Page Footer</h2>+ <p>SCROLL_REACHED_FOOTER</p>+ </footer>+</body>+</html>diff --git a/tests/benchmark/fixtures/spa.html b/tests/benchmark/fixtures/spa.htmlindex e226c07f..b5b4bdd3 100644--- a/tests/benchmark/fixtures/spa.html+++ b/tests/benchmark/fixtures/spa.html@@ -55,6 +55,11 @@let taskCount = 3;let tasks = [];+ // Support ?reset=1 to clear state between benchmark runs+ if (new URLSearchParams(window.location.search).has('reset')) {+ localStorage.removeItem('tasks');+ }+// Load tasks from localStorage on page loadfunction loadTasks() {const saved = localStorage.getItem('tasks');