High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.
feat: align benchmark suites and improve SKILL.md from optimization loop findings
Bosch94dfe6c4cbf4
diff --git a/skills/pinchtab/SKILL.md b/skills/pinchtab/SKILL.mdindex 534ff93f..3c38525f 100644--- a/skills/pinchtab/SKILL.md+++ b/skills/pinchtab/SKILL.md@@ -129,143 +129,21 @@ pinchtab snap -i -c## Challenge Solving-PinchTab includes a pluggable solver framework that auto-detects and resolves browser challenges (Cloudflare Turnstile, CAPTCHAs, interstitials). Use this **after navigation** when the page shows a challenge instead of the expected content.--**Important:** Solvers work best with `stealthLevel: "full"` in the PinchTab config (or `instanceDefaults.stealthLevel: "full"`). Full stealth mode patches CDP detection vectors, rotates fingerprints, and masks automation signals — all of which challenge providers like Cloudflare check before and after the checkbox click. Without full stealth, the solver may click correctly but the challenge can still fail fingerprint verification.--```bash-# Auto-detect and solve any challenge on the current page-curl -X POST http://localhost:9867/solve \- -H 'Content-Type: application/json' \- -d '{"maxAttempts": 3, "timeout": 30000}'--# Use a specific solver-curl -X POST http://localhost:9867/solve/cloudflare \- -H 'Content-Type: application/json' \- -d '{"maxAttempts": 3}'--# Tab-scoped solve-curl -X POST http://localhost:9867/tabs/TAB_ID/solve \- -H 'Content-Type: application/json' \- -d '{}'--# List available solvers-curl http://localhost:9867/solvers-```--**When to use solve:**--- Page title is "Just a moment..." or similar challenge indicator-- `pinchtab text` returns empty or challenge-page text after navigation-- A Cloudflare Turnstile widget blocks the target content--**Workflow pattern:**--```bash-pinchtab nav https://protected-site.com-pinchtab text # Check if page loaded or shows challenge-# If challenge detected:-curl -X POST http://localhost:9867/solve \- -H 'Content-Type: application/json' -d '{}'-pinchtab text # Verify: should now show real page content-```--**Response fields:** `solver` (which solver handled it), `solved` (bool), `challengeType` (e.g. "managed"), `attempts`, `title` (final page title).--The auto-detect mode (`POST /solve` without specifying a solver) tries each registered solver in order and returns immediately with `solved: true, attempts: 0` if no challenge is present. This makes it safe to call speculatively after any navigation.+If a page shows a challenge instead of content (e.g., "Just a moment..."), call `POST /solve` with `{"maxAttempts": 3}` to auto-detect and resolve it. Use `POST /tabs/TAB_ID/solve` for tab-scoped. Works best with `stealthLevel: "full"` in config. Safe to call speculatively — returns immediately if no challenge is present. See [api.md](./references/api.md) for full solver options.## Handling Authentication and State-Pick one of these five patterns before you start interacting with the site.--### 1. One-off public browsing--Use a temporary instance for public pages, scraping, or tasks that do not need login persistence.--```bash-pinchtab instance start-pinchtab instances-# Point CLI commands at the instance port you want to use.-pinchtab --server http://localhost:9868 nav https://pinchtab.com-pinchtab --server http://localhost:9868 text-```--### 2. Reuse an existing named profile+Pick a pattern before interacting with the site:-Use this for recurring tasks against the same authenticated site.--```bash-pinchtab profiles-pinchtab instance start --profile work --mode headed-pinchtab --server http://localhost:9868 nav https://mail.google.com-```--If the login is already stored in that profile, you can switch to headless later:--```bash-pinchtab instance stop inst_ea2e747f-pinchtab instance start --profile work --mode headless-```+1. **One-off browsing**: `pinchtab instance start` → use `--server http://localhost:<port>` for commands.+2. **Reuse a profile**: `pinchtab instance start --profile work --mode headed` → switch to `--mode headless` after login is stored.+3. **Create profile via HTTP**: `POST /profiles` with `{"name":"..."}`, then `POST /profiles/<name>/start`.+4. **Human-assisted login**: Start headed, human signs in, agent reuses the profile headless.+5. **HTTP-only agent**: Use `POST /instances/start`, then target the instance port with curl. Send `X-Agent-Id` for attribution.-### 3. Create a dedicated auth profile over HTTP--Use this when you need a durable profile and it does not exist yet.--```bash-curl -X POST http://localhost:9867/profiles \- -H "Content-Type: application/json" \- -d '{"name":"billing","description":"Billing portal automation","useWhen":"Use for billing tasks"}'--curl -X POST http://localhost:9867/profiles/billing/start \- -H "Content-Type: application/json" \- -d '{"headless":false}'-```--Then target the returned port with `--server`.--### 4. Human-assisted headed login, then agent reuse--Use this for CAPTCHA, MFA, or first-time setup.--```bash-pinchtab instance start --profile work --mode headed-# Human completes login in the visible Chrome window.-pinchtab --server http://localhost:9868 nav https://app.example.com/dashboard-pinchtab --server http://localhost:9868 snap -i -c-```--Once the session is stored, reuse the same profile for later tasks.--### 5. Remote or non-shell agent with tokenized HTTP API--Use this when the agent cannot call the CLI directly.--```bash-curl http://localhost:9867/health-curl -X POST http://localhost:9867/profiles \- -H "Content-Type: application/json" \- -d '{"name":"work"}'-curl -X POST http://localhost:9867/instances/start \- -H "Content-Type: application/json" \- -d '{"profileId":"work","mode":"headless"}'-curl -X POST http://localhost:9868/action \- -H "X-Agent-Id: agent-main" \- -H "Content-Type: application/json" \- -d '{"kind":"click","selector":"e5"}'-```--If the server is exposed beyond localhost, require a token and use a dedicated automation profile. See [TRUST.md](./TRUST.md).--**Agent sessions**: Instead of sharing the server bearer token, each agent can get its own revocable session token. Set `PINCHTAB_SESSION=ses_...` or send `Authorization: Session ses_...`. Sessions have idle timeout (default 30m) and max lifetime (default 24h). Use them only in trusted, controlled environments. They are not a multi-tenant isolation boundary, and session-authenticated callers are still intended for operator-controlled automation only.--```bash-pinchtab session create --agent-id <agent-id> [--label <label>] # Create; prints sessionToken-pinchtab session list # List all sessions-pinchtab session info # Show session for current token-pinchtab session revoke <session-id> # Revoke session-```+If the server is exposed beyond localhost, require a token. See [TRUST.md](./TRUST.md).-HTTP equivalents: `POST /sessions`, `GET /sessions`, `GET /sessions/me`, `POST /sessions/{id}/revoke`.+**Agent sessions**: Each agent can get its own revocable session token via `pinchtab session create --agent-id <id>` or `POST /sessions`. Set `PINCHTAB_SESSION=ses_...` or send `Authorization: Session ses_...`. Sessions have idle timeout (default 30m) and max lifetime (default 24h).## Essential Commands@@ -319,6 +197,7 @@ Guidance:- `snap -d` is the default follow-up snapshot for multi-step flows.- `text` is the default for reading articles, dashboards, reports, or confirmation messages.- `find --ref-only` is useful when the page is large and you already know the semantic target.+- Refs from `snap -i` and full `snap` use different numbering. Do not mix them — if you snapshot with `-i`, use those refs. If you re-snapshot without `-i`, get fresh refs before acting.### Interaction@@ -343,6 +222,7 @@ Rules:- Prefer `type` only when the site depends on keystroke events.- Prefer `click --wait-nav` when a click is expected to navigate.- Re-snapshot immediately after `click`, `press Enter`, `select`, or `scroll` if the UI can change.+- To discover valid dropdown values, snapshot with `filter=interactive` first — the output shows `<option>` elements with their `value` attributes. Then use `select` with the exact value.### Export, debug, and verification@@ -400,6 +280,80 @@ Use the API when:- profile creation or mutation is required,- or you need explicit instance- and tab-scoped routes.+### Tab-scoped HTTP API++**Important:** Each `POST /navigate` creates a new tab by default. The default (non-tab-scoped) endpoints like `/snapshot`, `/action`, `/text` operate on the *active* tab, which may not be the one you just navigated. In multi-tab workflows, always use tab-scoped routes to avoid acting on the wrong page.++Get the tab ID from the navigate response or from `GET /tabs`.++```bash+# List all tabs+curl http://localhost:9867/tabs \+ -H "Authorization: Bearer <token>"++# Navigate in a specific tab (does not create a new tab)+curl -X POST http://localhost:9867/tabs/TAB_ID/navigate \+ -H "Authorization: Bearer <token>" \+ -H "Content-Type: application/json" \+ -d '{"url":"https://example.com"}'++# Snapshot a specific tab+curl "http://localhost:9867/tabs/TAB_ID/snapshot?filter=interactive&format=compact" \+ -H "Authorization: Bearer <token>"++# Get text from a specific tab+curl http://localhost:9867/tabs/TAB_ID/text \+ -H "Authorization: Bearer <token>"++# Perform action on a specific tab+curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer <token>" \+ -H "Content-Type: application/json" \+ -d '{"kind":"click","selector":"#submit-btn"}'++# Navigate back/forward in a specific tab+curl -X POST http://localhost:9867/tabs/TAB_ID/back \+ -H "Authorization: Bearer <token>"+curl -X POST http://localhost:9867/tabs/TAB_ID/forward \+ -H "Authorization: Bearer <token>"++# Screenshot (GET, not POST)+curl http://localhost:9867/tabs/TAB_ID/screenshot \+ -H "Authorization: Bearer <token>" \+ --output screenshot.png++# PDF export (GET or POST)+curl http://localhost:9867/tabs/TAB_ID/pdf \+ -H "Authorization: Bearer <token>" \+ --output page.pdf++# Close a tab+curl -X POST http://localhost:9867/tabs/TAB_ID/close \+ -H "Authorization: Bearer <token>"+```++The default (non-tab-scoped) endpoints also support screenshots and PDF:++```bash+# Screenshot of active tab (GET)+curl http://localhost:9867/screenshot \+ -H "Authorization: Bearer <token>" \+ --output screenshot.png++# PDF of active tab (GET or POST)+curl http://localhost:9867/pdf \+ -H "Authorization: Bearer <token>" \+ --output page.pdf+```++**Navigation with `waitNav`:** When clicking a link or button that triggers page navigation, include `"waitNav": true` in the action body. Without it, PinchTab returns a `navigation_changed` error to protect against unexpected navigation during form interactions.++```json+{"kind": "click", "selector": "#search-btn", "waitNav": true}+```++All tab-scoped routes follow the pattern `/tabs/{TAB_ID}/...` and mirror the default endpoints. The full list includes: `navigate`, `back`, `forward`, `reload`, `snapshot`, `screenshot`, `text`, `pdf`, `action`, `actions`, `dialog`, `wait`, `find`, `lock`, `unlock`, `cookies`, `metrics`, `network`, `solve`, `close`, `storage`, `evaluate`, `download`, `upload`.+## Common Patterns### Open a page and inspect actions@@ -429,34 +383,7 @@ pinchtab click e3 # Click the Search buttonpinchtab text```-**Form submission rules:**-- ❌ **NEVER use `press Enter` on regular form inputs.** It does NOT submit standard HTML forms.-- ✅ **ALWAYS click the submit button** to trigger form submission handlers.-- **Why?** HTML5 forms only auto-submit on Enter if they have explicit JavaScript `onkeypress` or `onkeyup` handlers. The `<form onsubmit>` handler only fires when you click the button, not on Enter.--**Real-world example from benchmark:**-```bash-# WRONG: This fails-pinchtab nav http://fixtures/wiki.html-pinchtab snap -i -c-pinchtab fill "#wiki-search-input" "go"-pinchtab press Enter # ❌ Does NOT submit the form--# RIGHT: This works-pinchtab nav http://fixtures/wiki.html-pinchtab snap -i -c-pinchtab fill "#wiki-search-input" "go"-pinchtab click "#wiki-search-btn" # ✅ Triggers onsubmit handler-pinchtab text # Now on the results page-```--**Always follow this pattern:**-```bash-# Template: fill, then click-pinchtab fill "<selector>" "value"-pinchtab click "<button-selector>" # Always click, never press Enter-pinchtab text # Verify the form was submitted-```+**Form submission:** Always click the submit button — never use `press Enter`. Most HTML forms only fire their submission handler on button click, not on Enter keypress.### Use diff snapshots in a multi-step flow@@ -477,36 +404,6 @@ pinchtab fill "#search" "quarterly report"pinchtab click "xpath://button[@type='submit']"```-### Navigate through a Cloudflare-protected site--```bash-pinchtab nav https://protected-site.com-# Page may show CF challenge ("Just a moment...")-curl -X POST http://localhost:9867/solve \- -H 'Content-Type: application/json' -d '{"maxAttempts": 3}'-# Now the real page is loaded — proceed normally-pinchtab snap -i -c-pinchtab text-```--### Bootstrap an authenticated profile--```bash-pinchtab profiles-pinchtab instance start --profile work --mode headed-# Human signs in once.-pinchtab --server http://localhost:9868 text-```--### Run separate instances for separate sites--```bash-pinchtab instance start --profile work --mode headless-pinchtab instance start --profile staging --mode headless-pinchtab instances-```--Then point each command stream at its own port using `--server`.## Security and Token Economy@@ -523,16 +420,6 @@ Then point each command stream at its own port using `--server`.- Use `pinchtab screenshot` only when visual regressions, CAPTCHA, or layout-specific confirmation matters.- If a ref disappears after a change, treat that as expected and fetch fresh refs instead of retrying the stale one.-## Privacy and Security--PinchTab is a fully open-source, local-only browser automation tool:--- **Runs on localhost only.** The server binds to `127.0.0.1` by default. No external network calls are made by PinchTab itself.-- **No telemetry or analytics.** The binary makes zero outbound connections.-- **Single Go binary (~16 MB).** Fully verifiable — anyone can build from source at [github.com/pinchtab/pinchtab](https://github.com/pinchtab/pinchtab).-- **Local Chrome profiles.** Persistent profiles store cookies and sessions on your machine only. This enables agents to reuse authenticated sessions without re-entering credentials, similar to how a human reuses their browser profile.-- **Token-efficient by design.** Uses the accessibility tree (structured text) instead of screenshots, keeping agent context windows small. Comparable to Playwright but purpose-built for AI agents.-- **Multi-instance isolation.** Each browser instance runs in its own profile directory with tab-level locking for safe multi-agent use.## References@@ -543,36 +430,4 @@ PinchTab is a fully open-source, local-only browser automation tool:- MCP: [mcp.md](./references/mcp.md)- Security model: [TRUST.md](./TRUST.md)-## Content Extraction: text vs snapshot--Choose the right extraction method:--| Use Case | Recommended | Why |-|----------|-------------|-----|-| Article body, paragraphs | `text` | Clean prose extraction |-| Prices, numbers in cards | `snapshot` | Text strips structured data |-| Form field values | `snapshot` | See current input values |-| Verify element exists | `snapshot` with selector | Text won't show headings |-| JS-rendered content | `snapshot` after wait | Text may miss dynamic content |--**Common pitfall**: `/text` extracts readable prose but strips headings, prices, and structured UI elements. If you need to verify a heading, price, or button label, use `/snapshot` instead.--```bash-# Wrong: looking for "$149.99" in text output-pinchtab text | grep "149.99" # May fail - prices often stripped--# Right: snapshot includes all visible text-pinchtab snap -c | grep "149.99" # Works-```-## Fixture Selector Quick Reference--| Page | Key selectors |-|------|---------------|-| ecommerce | `.add-to-cart`, `#checkout-btn`, `.price` (use `snapshot` not `-c` for price values) |-| search | `#search-input`, `#search-btn` (click button — never press Enter) |-| form | `#fullname`, `#email`, `#phone`, `#country`, `#subject`, `#message`, `#submit-btn` |-| wiki | `#wiki-search-input`, `#wiki-search-btn` |-| spa | `#new-task-input`, `#priority-select`, `#add-task-btn` |-| login | `#username`, `#password`, `#login-btn`, `#logout-btn` |-| dashboard | `#settings-btn`, `#theme-select`, `#modal-save` |diff --git a/tests/benchmark/AGENT_TASKS.md b/tests/benchmark/AGENT_TASKS.mdindex 43c2aeaf..a36f3bfe 100644--- a/tests/benchmark/AGENT_TASKS.md+++ b/tests/benchmark/AGENT_TASKS.md@@ -7,7 +7,7 @@ Natural language tasks to test how well an agent uses PinchTab from skill docs a1. Read `../../skills/pinchtab/SKILL.md` — this is your only guide2. For each task, figure out which commands to use3. **Log every command executed**-4. Record: `./record-agent-step.sh <group> <step> <pass|fail> <in> <out> "commands" "notes"`+4. Record: `./scripts/record-step.sh --type agent <group> <step> <pass|fail> --tokens <in> <out> "notes"`## Environment@@ -121,7 +121,7 @@ Add a task called "Automate deployment" with high priority.**Verify**: Task appeared in the list (`TASK_ADDED_AUTOMATE_DEPLOYMENT_PRIORITY_HIGH`).### 4.3 Delete a task-Delete the first existing task ("Write benchmark tests").+Delete the task titled "Write benchmark tests".**Verify**: Task count changed (went from 4 to 3).diff --git a/tests/benchmark/BENCHMARK_TASKS.md b/tests/benchmark/BENCHMARK_TASKS.mdindex 84de1491..9931d2e6 100644--- a/tests/benchmark/BENCHMARK_TASKS.md+++ b/tests/benchmark/BENCHMARK_TASKS.md@@ -1,11 +1,12 @@-# PinchTab Benchmark Tasks+# PinchTab Benchmark Tasks (Baseline)-Reproducible benchmark tasks using controlled fixture pages with verifiable content.+Reproducible benchmark using explicit curl commands against controlled fixture pages.+Each task maps 1:1 to an agent task in AGENT_TASKS.md for direct comparison.## Environment- **PinchTab Server**: `http://localhost:9867`-- **Fixtures Server**: `http://fixtures/`+- **Fixtures Server**: `http://fixtures/` (inside Docker network)- **Token**: `benchmark-token`- **Auth Header**: `Authorization: Bearer benchmark-token`@@ -20,7 +21,11 @@ docker compose up -d --build## Recording```bash-./record-step.sh <group> <step> <pass|fail> <in_tokens> <out_tokens> "notes"+# Baseline (no tokens, optional response bytes):+./scripts/record-step.sh --type baseline <group> <step> <pass|fail> "notes"++# Minimal (just pass/fail):+./scripts/record-step.sh <group> <step> <pass|fail> "notes"```**On failure, include in notes:**@@ -28,675 +33,835 @@ docker compose up -d --build- What was actually returned- HTTP status code / error message----+## Tab Reuse-## Group 0: Setup & Configuration Verification+`POST /navigate` creates a new tab by default. To avoid multi-tab issues, the+runner must:-Verify the environment is correctly configured before running tests.+1. Capture `tabId` from the first navigate response (step 0.2).+2. Pass `"tabId":"TAB_ID"` in every subsequent navigate request body.+3. Use tab-scoped endpoints for actions and snapshots:+ - `POST /tabs/TAB_ID/action` instead of `POST /action`+ - `GET /tabs/TAB_ID/snapshot?...` instead of `GET /snapshot?...`+ - `GET /tabs/TAB_ID/text` instead of `GET /text`+ - `POST /tabs/TAB_ID/back` instead of `POST /back`+ - `GET /tabs/TAB_ID/screenshot` instead of `GET /screenshot`+ - `GET /tabs/TAB_ID/pdf` instead of `GET /pdf`-### 0.1 Load PinchTab skill-Read `../../skills/pinchtab/SKILL.md`.-```bash-./record-step.sh 0 1 pass <in> <out> "Skill loaded"-```-**Pass if**: Skill loaded successfully.+All curl examples below use `TAB_ID` as a placeholder. Replace with the actual+tab ID captured in step 0.2.-### 0.2 Verify PinchTab health-```bash-curl -sf http://localhost:9867/health \- -H "Authorization: Bearer benchmark-token" | jq .-```-**Pass if**: `status == "ok"`, `authRequired == true`, `instances >= 1`.+----### 0.3 Verify fixtures server-```bash-curl -sf http://localhost:9867/navigate \- -X POST -H "Authorization: Bearer benchmark-token" \- -H "Content-Type: application/json" \- -d '{"url":"http://fixtures/"}' | jq .-```-**Pass if**: Response contains `"url": "http://fixtures/"`.+## Group 0: Setup-### 0.4 Verify Chrome instance running+### 0.1 Health check```bashcurl -sf http://localhost:9867/health \- -H "Authorization: Bearer benchmark-token" | jq '.defaultInstance.status'+ -H "Authorization: Bearer benchmark-token"```-**Pass if**: Value is `"running"`.-------## Group 1: Navigation & Content Extraction+**Pass if**: `status == "ok"`, `authRequired == true`, `instances >= 1`, `defaultInstance.status == "running"`.-### 1.1 Navigate to fixtures home+### 0.2 Fixtures reachable```bash-curl -X POST http://localhost:9867/navigate \+curl -sf -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"url":"http://fixtures/"}'```-**Pass if**: HTTP 200.--### 1.2 Verify home content via snapshot+**Capture**: Save the `tabId` from the JSON response. All subsequent commands use this tab ID.```bash-curl "http://localhost:9867/snapshot?format=compact&maxTokens=1000" \+curl -sf "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \-H "Authorization: Bearer benchmark-token"```-**Pass if**: Contains `VERIFY_HOME_LOADED_12345`.+**Pass if**: Navigate returns HTTP 200 AND snapshot contains `VERIFY_HOME_LOADED_12345`.++---++## Group 1: Reading & Extracting-### 1.3 Navigate to wiki index+### 1.1 Wiki categories```bashcurl -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"url":"http://fixtures/wiki.html"}'-```-**Pass if**: HTTP 200.+ -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki.html"}'-### 1.4 Verify wiki content and count categories-```bash-curl "http://localhost:9867/snapshot?format=compact&maxTokens=1500" \+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \-H "Authorization: Bearer benchmark-token"```**Pass if**: Contains `VERIFY_WIKI_INDEX_55555` AND `COUNT_LANGUAGES_12` AND `COUNT_TOOLS_15`.-### 1.5 Click through to Go article+### 1.2 Click a link```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"click","selector":"#link-go","waitNav":true}'```**Pass if**: HTTP 200 with `{"success":true}`.-**Note**: Use `waitNav: true` when clicking a link that causes page navigation. Without it, PinchTab returns a 409 "navigation_changed" error to protect against unexpected navigation during form interactions.+### 1.3 Table extraction+```bash+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \+ -H "Authorization: Bearer benchmark-token"+```+**Pass if**: Contains `VERIFY_WIKI_GO_LANG_88888` AND `Robert Griesemer` AND `2009`.-### 1.6 Verify Go article loaded and extract key facts+### 1.4 Count list items```bash-curl "http://localhost:9867/snapshot?format=compact&maxTokens=2000" \+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \-H "Authorization: Bearer benchmark-token"```-**Pass if**: Contains `VERIFY_WIKI_GO_LANG_88888` AND `Robert Griesemer` AND `2009` AND `FEATURE_COUNT_6`.+**Pass if**: Contains `FEATURE_COUNT_6`.-### 1.7 Extract specific table data (designer)+### 1.5 Article headlines```bash-curl http://localhost:9867/text \+curl -X POST http://localhost:9867/navigate \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"tabId":"TAB_ID","url":"http://fixtures/articles.html"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \-H "Authorization: Bearer benchmark-token"```-**Pass if**: Text contains `Google LLC` AND `Ken Thompson`.+**Pass if**: Contains `The Future of Artificial Intelligence` AND `Climate Action in 2026` AND `Mars Colony`.-### 1.8 Navigate to articles and extract all headlines+### 1.6 Dashboard metrics```bashcurl -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"url":"http://fixtures/articles.html"}'+ -d '{"tabId":"TAB_ID","url":"http://fixtures/dashboard.html"}'-curl "http://localhost:9867/snapshot?format=compact&maxTokens=2000" \+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \-H "Authorization: Bearer benchmark-token"```-**Pass if**: Snapshot contains all 3 article titles: `The Future of Artificial Intelligence`, `Climate Action in 2026`, `Mars Colony`.+**Pass if**: Contains `24,582` AND `$1,284,930` AND `4.28%`.----## Group 2: Search & Dynamic Content+## Group 2: Search & Dynamic-### 2.1 Navigate to wiki search+### 2.1 Wiki search```bashcurl -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"url":"http://fixtures/wiki.html"}'-```-**Pass if**: HTTP 200.+ -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki.html"}'-### 2.2 Get snapshot to find search input ref-```bash-curl "http://localhost:9867/snapshot?filter=interactive&format=compact" \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"fill","selector":"#wiki-search-input","text":"golang"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"click","selector":"#wiki-search-btn","waitNav":true}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \-H "Authorization: Bearer benchmark-token"```-**Pass if**: Contains search input element. Note the ref.+**Pass if**: Snapshot contains `VERIFY_WIKI_GO_LANG_88888` (search redirected to Go page).-### 2.3 Fill search query+### 2.2 No results search```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#wiki-search-input","text":"golang"}'-```-**Pass if**: HTTP 200.+ -d '{"tabId":"TAB_ID","url":"http://fixtures/search.html"}'-### 2.4 Submit search-```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#wiki-search-btn"}'-```-**Pass if**: HTTP 200.+ -d '{"kind":"fill","selector":"#search-input","text":"xyznonexistent"}'-### 2.5 Verify navigation to Go article-```bash-curl "http://localhost:9867/snapshot?format=compact&maxTokens=2000" \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"click","selector":"#search-btn"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \-H "Authorization: Bearer benchmark-token"```-**Pass if**: Contains `VERIFY_WIKI_GO_LANG_88888` (search redirected to Go page).+**Pass if**: All curls return HTTP 200 AND snapshot shows no-results message.-### 2.6 Search for something with no results+### 2.3 AI content search```bashcurl -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"url":"http://fixtures/search.html"}'+ -d '{"tabId":"TAB_ID","url":"http://fixtures/search.html"}'-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#search-input","text":"xyznonexistent"}'+ -d '{"kind":"fill","selector":"#search-input","text":"artificial intelligence"}'-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"click","selector":"#search-btn"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \+ -H "Authorization: Bearer benchmark-token"```-**Pass if**: HTTP 200 (graceful no-results state).+**Pass if**: Snapshot contains `The Future of Artificial Intelligence`.----## Group 3: Complex Form Interaction+## Group 3: Form-### 3.1 Navigate to contact form+### 3.1 Complete form```bashcurl -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"url":"http://fixtures/form.html"}'-```-**Pass if**: HTTP 200.--### 3.2 Get snapshot of interactive elements-```bash-curl "http://localhost:9867/snapshot?filter=interactive&format=compact" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains all expected fields: fullname, email, country, subject, message.+ -d '{"tabId":"TAB_ID","url":"http://fixtures/form.html"}'-### 3.3 Fill full name-```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"fill","selector":"#fullname","text":"John Benchmark"}'-```-**Pass if**: HTTP 200.-### 3.4 Fill email-```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"fill","selector":"#email","text":"[email protected]"}'-```-**Pass if**: HTTP 200.-### 3.5 Fill phone-```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"fill","selector":"#phone","text":"+44 20 1234 5678"}'-```-**Pass if**: HTTP 200.-### 3.6 Select country-```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"select","selector":"#country","value":"uk"}'-```-**Pass if**: HTTP 200.-### 3.7 Select subject-```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"select","selector":"#subject","value":"support"}'-```-**Pass if**: HTTP 200.-### 3.8 Fill message textarea-```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"fill","selector":"#message","text":"This is a benchmark test message for PinchTab automation."}'-```-**Pass if**: HTTP 200.-### 3.9 Check newsletter checkbox-```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"click","selector":"#newsletter"}'-```-**Pass if**: HTTP 200.-### 3.10 Select high priority radio-```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"click","selector":"input[name=priority][value=high]"}'-```-**Pass if**: HTTP 200.-### 3.11 Submit form-```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"click","selector":"#submit-btn"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \+ -H "Authorization: Bearer benchmark-token"```-**Pass if**: HTTP 200.+**Pass if**: Contains `VERIFY_FORM_SUBMITTED_SUCCESS` AND `SUBMISSION_DATA_NAME_JOHN_BENCHMARK`.-### 3.12 Verify submission with data check+### 3.2 Reset/refill```bash-curl "http://localhost:9867/snapshot?format=compact&maxTokens=1000" \+curl "http://localhost:9867/tabs/TAB_ID/snapshot?filter=interactive&format=compact" \-H "Authorization: Bearer benchmark-token"```-**Pass if**: Contains `VERIFY_FORM_SUBMITTED_SUCCESS` AND `SUBMISSION_DATA_NAME_JOHN_BENCHMARK`.+**Pass if**: Contains reset button or form element (after submission, page still has interactive elements).----## Group 4: SPA & Dynamic State+## Group 4: SPA-### 4.1 Navigate to task manager SPA+### 4.1 Read app state```bashcurl -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"url":"http://fixtures/spa.html"}'-```-**Pass if**: HTTP 200.+ -d '{"tabId":"TAB_ID","url":"http://fixtures/spa.html"}'-### 4.2 Verify initial state-```bash-curl "http://localhost:9867/snapshot?format=compact&maxTokens=1500" \+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \-H "Authorization: Bearer benchmark-token"```**Pass if**: Contains `VERIFY_SPA_PAGE_99999` AND `TASK_STATS_TOTAL_3_ACTIVE_2_DONE_1`.-### 4.3 Add a new task+### 4.2 Add task```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"fill","selector":"#new-task-input","text":"Deploy to production"}'-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"select","selector":"#priority-select","value":"high"}'-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"click","selector":"#add-task-btn"}'-```-**Pass if**: All HTTP 200.-### 4.4 Verify task was added-```bash-curl "http://localhost:9867/snapshot?format=compact&maxTokens=1500" \+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \-H "Authorization: Bearer benchmark-token"```**Pass if**: Contains `TASK_ADDED_DEPLOY_TO_PRODUCTION_PRIORITY_HIGH`.-### 4.5 Delete existing task+### 4.3 Delete task```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"kind":"click","selector":".delete-task[data-id=\"1\"]"}'-```-**Pass if**: HTTP 200.+ -d '{"kind":"click","selector":"#task-1 .delete-task"}'-### 4.6 Verify task count updated-```bash-curl "http://localhost:9867/snapshot?format=compact&maxTokens=500" \+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \-H "Authorization: Bearer benchmark-token"```-**Pass if**: Contains `3` in total count (started with 3, deleted 1, added 1 = 3).+**Pass if**: Total count is `3` (started with 3, added 1, deleted 1 = 3).----## Group 5: Login & Auth Flow+## Group 5: Login-### 5.1 Navigate to login page+### 5.1 Invalid login```bashcurl -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"url":"http://fixtures/login.html"}'-```-**Pass if**: HTTP 200.--### 5.2 Verify login page-```bash-curl "http://localhost:9867/snapshot?format=compact&maxTokens=500" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Contains `VERIFY_LOGIN_PAGE_77777`.+ -d '{"tabId":"TAB_ID","url":"http://fixtures/login.html"}'-### 5.3 Try invalid credentials-```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"fill","selector":"#username","text":"baduser"}'-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"fill","selector":"#password","text":"wrongpassword"}'-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"click","selector":"#login-btn"}'-```-**Pass if**: All HTTP 200.-### 5.4 Verify error shown-```bash-curl "http://localhost:9867/snapshot?format=compact&maxTokens=500" \+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \-H "Authorization: Bearer benchmark-token"```**Pass if**: Contains `INVALID_CREDENTIALS_ERROR`.-### 5.5 Login with valid credentials+### 5.2 Valid login```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"fill","selector":"#username","text":"benchmark"}'-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"fill","selector":"#password","text":"test456"}'-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"click","selector":"#login-btn"}'-```-**Pass if**: All HTTP 200.-### 5.6 Verify logged in-```bash-curl "http://localhost:9867/snapshot?format=compact&maxTokens=500" \+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \-H "Authorization: Bearer benchmark-token"```**Pass if**: Contains `VERIFY_LOGIN_SUCCESS_DASHBOARD` AND `SESSION_TOKEN_ACTIVE_TRUE`.----## Group 6: E-commerce Flow+## Group 6: E-commerce-### 6.1 Navigate to shop+### 6.1 Research products```bashcurl -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"url":"http://fixtures/ecommerce.html"}'-```-**Pass if**: HTTP 200.+ -d '{"tabId":"TAB_ID","url":"http://fixtures/ecommerce.html"}'-### 6.2 Verify shop page and read prices-```bash-curl "http://localhost:9867/snapshot?format=compact&maxTokens=2000" \+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \-H "Authorization: Bearer benchmark-token"```-**Pass if**: Contains `VERIFY_SHOP_PAGE_44444` AND `$149.99` AND `$299.99` AND `Out of Stock`.+**Pass if**: Contains `VERIFY_SHOP_PAGE_44444` AND `$149.99` AND `$299.99` AND `$49.99` AND `Out of Stock`.-### 6.3 Verify out-of-stock button is disabled+### 6.2 Add to cart```bash-curl "http://localhost:9867/snapshot?filter=interactive&format=compact" \- -H "Authorization: Bearer benchmark-token"-```-**Pass if**: Mechanical Keyboard add-to-cart button is shown as disabled.--### 6.4 Add Wireless Headphones-```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"click","selector":"#product-1 .add-to-cart"}'-```-**Pass if**: HTTP 200.-### 6.5 Add Smart Watch-```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"click","selector":"#product-2 .add-to-cart"}'-```-**Pass if**: HTTP 200.-### 6.6 Verify cart total-```bash-curl "http://localhost:9867/snapshot?format=compact&maxTokens=1000" \+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \-H "Authorization: Bearer benchmark-token"```-**Pass if**: Contains `CART_ITEM_WIRELESS_HEADPHONES` AND `CART_ITEM_SMART_WATCH_PRO` AND `449.98` (total).+**Pass if**: Contains `CART_ITEM_WIRELESS_HEADPHONES` AND `CART_ITEM_SMART_WATCH_PRO` AND `449.98`.-### 6.7 Checkout+### 6.3 Checkout```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \-d '{"kind":"click","selector":"#checkout-btn"}'-```-**Pass if**: HTTP 200.-### 6.8 Verify checkout complete-```bash-curl "http://localhost:9867/snapshot?format=compact&maxTokens=500" \+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \-H "Authorization: Bearer benchmark-token"```**Pass if**: Contains `VERIFY_CHECKOUT_SUCCESS_ORDER` AND `ORDER_TOTAL_449_98`.----## Group 7: Wiki Article + Comment Flow+## Group 7: Content + Interaction-### 7.1 Navigate to Go article+### 7.1 Read & comment```bashcurl -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"url":"http://fixtures/wiki-go.html"}'+ -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki-go.html"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \+ -H "Authorization: Bearer benchmark-token"++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"fill","selector":"#comment-text","text":"Great article on Go! Very comprehensive."}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"select","selector":"#comment-rating","value":"5"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"click","selector":"#submit-comment"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \+ -H "Authorization: Bearer benchmark-token"```-**Pass if**: HTTP 200.+**Pass if**: First snapshot contains `VERIFY_WIKI_GO_LANG_88888` AND final snapshot contains `COMMENT_POSTED_RATING_5_TEXT_RECEIVED`.-### 7.2 Read and verify article facts+### 7.2 Cross-page research```bash-curl "http://localhost:9867/snapshot?format=compact&maxTokens=2000" \+curl -X POST http://localhost:9867/navigate \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki.html"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \+ -H "Authorization: Bearer benchmark-token"++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"click","selector":"#link-go","waitNav":true}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \-H "Authorization: Bearer benchmark-token"```-**Pass if**: Contains `VERIFY_WIKI_GO_LANG_88888` AND `BSD-style` AND `go.dev`.+**Pass if**: First snapshot contains `COUNT_LANGUAGES_12` AND second snapshot contains `VERIFY_WIKI_GO_LANG_88888`.++---++## Group 8: Error Handling-### 7.3 Fill comment+### 8.1 404 handling```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#comment-text","text":"Great article on Go! Very comprehensive."}'+ -d '{"tabId":"TAB_ID","url":"http://fixtures/nonexistent-page-xyz.html"}'```-**Pass if**: HTTP 200.+**Pass if**: Returns response without crash (HTTP 200 with error page, or structured error).-### 7.4 Rate the article+### 8.2 Missing element```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"kind":"select","selector":"#comment-rating","value":"5"}'+ -d '{"kind":"click","selector":"#element-that-does-not-exist"}'```-**Pass if**: HTTP 200.+**Pass if**: Error response with clear message (not crash).++---++## Group 9: Export-### 7.5 Submit comment+### 9.1 Screenshot```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#submit-comment"}'+ -d '{"tabId":"TAB_ID","url":"http://fixtures/dashboard.html"}'++curl http://localhost:9867/tabs/TAB_ID/screenshot \+ -H "Authorization: Bearer benchmark-token" \+ --output /tmp/benchmark-screenshot.png++ls -la /tmp/benchmark-screenshot.png```-**Pass if**: HTTP 200.+**Pass if**: File exists and size > 10240 bytes.-### 7.6 Verify comment posted+### 9.2 PDF export```bash-curl "http://localhost:9867/snapshot?format=compact&maxTokens=500" \- -H "Authorization: Bearer benchmark-token"+curl http://localhost:9867/tabs/TAB_ID/pdf \+ -H "Authorization: Bearer benchmark-token" \+ --output /tmp/benchmark-dashboard.pdf++ls -la /tmp/benchmark-dashboard.pdf```-**Pass if**: Contains `COMMENT_POSTED_RATING_5_TEXT_RECEIVED`.+**Pass if**: File exists and size > 10240 bytes.----## Group 8: Error Handling+## Group 10: Modals-### 8.1 Navigate to non-existent page+### 10.1 Open modal```bashcurl -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"url":"http://fixtures/nonexistent-page-xyz.html"}'+ -d '{"tabId":"TAB_ID","url":"http://fixtures/dashboard.html"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"click","selector":"#settings-btn"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \+ -H "Authorization: Bearer benchmark-token"```-**Pass if**: Returns response without crash (404 or error).+**Pass if**: Snapshot contains `Dashboard Settings`.-### 8.2 Click non-existent element+### 10.2 Modal interaction```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"kind":"click","selector":"#element-that-does-not-exist"}'+ -d '{"kind":"select","selector":"#theme-select","value":"dark"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"click","selector":"#modal-save"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \+ -H "Authorization: Bearer benchmark-token"```-**Pass if**: Error response with message (not crash).+**Pass if**: Contains `THEME_DARK_APPLIED`.++----### 8.3 Fill with invalid selector+## Group 11: Persistence++### 11.1 State after reload```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/navigate \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"tabId":"TAB_ID","url":"http://fixtures/spa.html"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"kind":"fill","selector":"#nonexistent-input","text":"test"}'+ -d '{"kind":"fill","selector":"#new-task-input","text":"Persistent Task Test"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"click","selector":"#add-task-btn"}'++curl -X POST http://localhost:9867/navigate \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"tabId":"TAB_ID","url":"http://fixtures/spa.html"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \+ -H "Authorization: Bearer benchmark-token"```-**Pass if**: Error response with message.+**Pass if**: Contains `TASK_PERSISTENT_TEST_FOUND_AFTER_RELOAD`.-### 8.4 Invalid action kind+### 11.2 Logout/re-login```bash-curl -X POST http://localhost:9867/action \+curl -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"kind":"teleport","selector":"#btn"}'+ -d '{"tabId":"TAB_ID","url":"http://fixtures/login.html"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"fill","selector":"#username","text":"benchmark"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"fill","selector":"#password","text":"test456"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"click","selector":"#login-btn"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"click","selector":"#logout-btn"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"fill","selector":"#username","text":"benchmark"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"fill","selector":"#password","text":"test456"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"click","selector":"#login-btn"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \+ -H "Authorization: Bearer benchmark-token"```-**Pass if**: Returns 400/422 error with clear message.+**Pass if**: Contains `VERIFY_LOGIN_SUCCESS_DASHBOARD` AND `SESSION_RENEWED`.----## Group 9: Screenshot & Export+## Group 12: Multi-page Nav-### 9.1 Navigate to dashboard+### 12.1 Navigate & return```bashcurl -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \-H "Content-Type: application/json" \- -d '{"url":"http://fixtures/dashboard.html"}'+ -d '{"tabId":"TAB_ID","url":"http://fixtures/"}'++curl -X POST http://localhost:9867/navigate \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki.html"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"click","selector":"#link-go","waitNav":true}'++curl -X POST http://localhost:9867/tabs/TAB_ID/back \+ -H "Authorization: Bearer benchmark-token"++curl -X POST http://localhost:9867/tabs/TAB_ID/back \+ -H "Authorization: Bearer benchmark-token"++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \+ -H "Authorization: Bearer benchmark-token"```-**Pass if**: HTTP 200.+**Pass if**: Final snapshot contains `VERIFY_HOME_LOADED_12345` (returned to home).-### 9.2 Take screenshot+### 12.2 Cross-page compare```bash-curl http://localhost:9867/screenshot \+curl -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \- --output /tmp/benchmark-screenshot.png+ -H "Content-Type: application/json" \+ -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki.html"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \+ -H "Authorization: Bearer benchmark-token"++curl -X POST http://localhost:9867/navigate \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"tabId":"TAB_ID","url":"http://fixtures/articles.html"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \+ -H "Authorization: Bearer benchmark-token"```-**Pass if**: File created, size > 10KB.+**Pass if**: Wiki snapshot contains `COUNT_LANGUAGES_12` AND articles snapshot contains article titles.++----### 9.3 Verify screenshot has content+## Group 13: Form Validation++### 13.1 Required field```bash-ls -la /tmp/benchmark-screenshot.png+curl -X POST http://localhost:9867/navigate \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"tabId":"TAB_ID","url":"http://fixtures/form.html"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"fill","selector":"#fullname","text":"Validator Test"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"click","selector":"#submit-btn"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \+ -H "Authorization: Bearer benchmark-token"```-**Pass if**: File exists and size > 10240 bytes.+**Pass if**: Snapshot does NOT contain `VERIFY_FORM_SUBMITTED_SUCCESS` (submission blocked by validation).-### 9.4 Export PDF+### 13.2 Optional field```bash-curl -X POST http://localhost:9867/pdf \+curl -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \- --output /tmp/benchmark-dashboard.pdf+ -H "Content-Type: application/json" \+ -d '{"tabId":"TAB_ID","url":"http://fixtures/form.html"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"fill","selector":"#fullname","text":"No Phone User"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"fill","selector":"#email","text":"[email protected]"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"select","selector":"#country","value":"de"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"select","selector":"#subject","value":"feedback"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"click","selector":"#submit-btn"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \+ -H "Authorization: Bearer benchmark-token"```-**Pass if**: File created, size > 10KB.+**Pass if**: Contains `VERIFY_FORM_SUBMITTED_SUCCESS` AND `OPTIONAL_FIELD_SKIPPED_SUCCESS`.----## Group 10: Agent Identity+## Group 14: Dynamic Content-### 10.1 Navigate as agent alpha+### 14.1 Load more```bashcurl -X POST http://localhost:9867/navigate \-H "Authorization: Bearer benchmark-token" \- -H "X-Agent-Id: bench-alpha" \-H "Content-Type: application/json" \- -d '{"url":"http://fixtures/wiki.html"}'+ -d '{"tabId":"TAB_ID","url":"http://fixtures/ecommerce.html"}'++curl -X POST http://localhost:9867/tabs/TAB_ID/action \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"kind":"click","selector":"#load-more-btn"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \+ -H "Authorization: Bearer benchmark-token"```-**Pass if**: HTTP 200.+**Pass if**: Contains `ADDITIONAL_PRODUCTS_LOADED`.-### 10.2 Navigate as agent beta+### 14.2 Lazy-loaded item```bash-curl -X POST http://localhost:9867/navigate \+curl -X POST http://localhost:9867/tabs/TAB_ID/action \-H "Authorization: Bearer benchmark-token" \- -H "X-Agent-Id: bench-beta" \-H "Content-Type: application/json" \- -d '{"url":"http://fixtures/dashboard.html"}'+ -d '{"kind":"click","selector":"#product-5 .add-to-cart"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \+ -H "Authorization: Bearer benchmark-token"```-**Pass if**: HTTP 200.+**Pass if**: Contains `CART_UPDATED_WITH_LAZY_PRODUCT`.-### 10.3 Check alpha activity+---++## Group 15: Data Aggregation++### 15.1 Financial calc```bash-curl "http://localhost:9867/api/activity?agentId=bench-alpha" \+curl -X POST http://localhost:9867/navigate \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"tabId":"TAB_ID","url":"http://fixtures/dashboard.html"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \-H "Authorization: Bearer benchmark-token"```-**Pass if**: Response contains activity for `bench-alpha`.+**Pass if**: Contains `$1,284,930` (revenue) AND `$384,930` (profit) AND `PROFIT_MARGIN_CALCULATED`.-### 10.4 Check beta activity+### 15.2 Multi-page comparison```bash-curl "http://localhost:9867/api/activity?agentId=bench-beta" \+curl -X POST http://localhost:9867/navigate \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki-go.html"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \+ -H "Authorization: Bearer benchmark-token"++curl -X POST http://localhost:9867/navigate \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki-python.html"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \+ -H "Authorization: Bearer benchmark-token"++curl -X POST http://localhost:9867/navigate \+ -H "Authorization: Bearer benchmark-token" \+ -H "Content-Type: application/json" \+ -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki-rust.html"}'++curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \-H "Authorization: Bearer benchmark-token"```-**Pass if**: Response contains activity for `bench-beta`.+**Pass if**: Go snapshot contains `FEATURE_COUNT_6` AND Python snapshot contains `FEATURE_COUNT_7` AND `COMPARISON_TABLE_BUILT` AND Rust snapshot contains `FEATURE_COUNT_5`.---## Summary-| Group | Steps | Description |+| Group | Tasks | Description ||-------|-------|-------------|-| 0 | 4 | Setup & Configuration |-| 1 | 8 | Navigation & Content Extraction |-| 2 | 6 | Search & Dynamic Content |-| 3 | 12 | Complex Form Interaction |-| 4 | 6 | SPA & Dynamic State |-| 5 | 6 | Login & Auth Flow |-| 6 | 8 | E-commerce Flow |-| 7 | 6 | Wiki Article + Comment |-| 8 | 4 | Error Handling |-| 9 | 4 | Screenshot & Export |-| 10 | 4 | Agent Identity |--**Total: 68 steps**+| 0 | 2 | Setup |+| 1 | 6 | Reading & Extracting |+| 2 | 3 | Search & Dynamic |+| 3 | 2 | Form |+| 4 | 3 | SPA |+| 5 | 2 | Login |+| 6 | 3 | E-commerce |+| 7 | 2 | Content + Interaction |+| 8 | 2 | Error Handling |+| 9 | 2 | Export |+| 10 | 2 | Modals |+| 11 | 2 | Persistence |+| 12 | 2 | Multi-page Nav |+| 13 | 2 | Form Validation |+| 14 | 2 | Dynamic Content |+| 15 | 2 | Data Aggregation |++**Total: 39 tasks**## Verification Strings@@ -712,3 +877,5 @@ curl "http://localhost:9867/api/activity?agentId=bench-beta" \| Login | `VERIFY_LOGIN_PAGE_77777` || Go Article | `VERIFY_WIKI_GO_LANG_88888` || SPA | `VERIFY_SPA_PAGE_99999` |+| Python Article | `VERIFY_WIKI_PYTHON_LANG` |+| Rust Article | `VERIFY_WIKI_RUST_LANG` |diff --git a/tests/benchmark/OPTIMIZATION_LOOP.md b/tests/benchmark/OPTIMIZATION_LOOP.mdindex a46566cc..054401d8 100644--- a/tests/benchmark/OPTIMIZATION_LOOP.md+++ b/tests/benchmark/OPTIMIZATION_LOOP.md@@ -1,50 +1,104 @@# PinchTab Auto-Optimization Loop-Automated improvement cycle that runs every 25 minutes.+Automated improvement cycle for PinchTab benchmarks.++## Pre-requisite (Human, once)++```bash+cd ~/dev/pinchtab/tests/benchmark+docker compose up -d --build+# Wait for healthy:+curl -sf -H "Authorization: Bearer benchmark-token" http://localhost:9867/health+```## Loop Steps```┌─────────────────────────────────────────────────────────────┐-│ 1. RUN BENCHMARKS │-│ - Baseline (explicit curl commands) │-│ - Agent (natural language tasks) │+│ 1a. RUN BASELINE BENCHMARK [Sub-agent A] │+│ - Only runs when: │+│ • No previous baseline exists, OR │+│ • Changes affect: server code, fixtures, or │+│ BENCHMARK_TASKS.md │+│ - Otherwise: reuse last baseline results │+│ - Load BENCHMARK_TASKS.md │+│ - Execute 68 curl commands sequentially │+│ - Check pass/fail per documented conditions │+│ - Record via: record-step.sh --type baseline │+│ - Output: results/baseline_<timestamp>.json │+├─────────────────────────────────────────────────────────────┤+│ 1b. RUN AGENT BENCHMARK [Sub-agent B] │+│ - Runs every iteration │+│ - Load AGENT_TASKS.md + skills/pinchtab/SKILL.md │+│ - Execute 39 tasks using only skill docs │+│ - Record via: record-step.sh --type agent --tokens │+│ - Output: results/agent_benchmark_<timestamp>.json │└─────────────────────────────────────────────────────────────┘↓┌─────────────────────────────────────────────────────────────┐-│ 2. ANALYZE DIFFERENCES │-│ - Compare pass rates │-│ - Identify step failures │-│ - Find patterns (text vs snapshot, selectors, etc.) │+│ 2. CHECK FOR REGRESSION [Main agent] │+│ - Read both JSON reports for pass/fail counts │+│ - Compare against results/best_score.txt │+│ - If worse: flag regression, prioritize fix/revert │+│ - If equal or better: update best_score.txt │└─────────────────────────────────────────────────────────────┘↓┌─────────────────────────────────────────────────────────────┐-│ 3. PROPOSE 1 IMPROVEMENT │+│ 3. ANALYZE DIFFERENCES [Main agent] │+│ - Diff baseline vs agent results side by side │+│ - Identify which steps agent failed but baseline passed │+│ - Find patterns (wrong endpoint, selectors, etc.) │+└─────────────────────────────────────────────────────────────┘+ ↓+┌─────────────────────────────────────────────────────────────┐+│ 4. PROPOSE 1 IMPROVEMENT [Main agent → Human]││ Priority order: ││ a) Fix PinchTab CLI/REST bug if found ││ b) Improve skill documentation if agent confused ││ c) Add verification to existing test case ││ d) Add new test case for uncovered scenario │+│ Present proposal to human for approval. │└─────────────────────────────────────────────────────────────┘↓┌─────────────────────────────────────────────────────────────┐-│ 4. IMPLEMENT & COMMIT │-│ - Make the change │-│ - Commit with descriptive message │-│ - Push to feat/benchmark branch │+│ 5. IMPLEMENT (no commit) [Main agent] │+│ - Make the single change │+│ - Leave changes uncommitted for manual review │└─────────────────────────────────────────────────────────────┘↓┌─────────────────────────────────────────────────────────────┐-│ 5. LOG RUN │-│ - Append to optimization_log.md │-│ - Record: timestamp, pass rates, change made │+│ 6. LOG RUN [Main agent] │+│ - Append to results/optimization_log.md │+│ - Record: timestamp, pass rates, regression check, │+│ change made, next focus │└─────────────────────────────────────────────────────────────┘- ↓- [Wait 25 minutes]- ↓- [Repeat]```+## Roles++| Step | Who | What |+|------|-----|------|+| Pre-req | **Human** | Start Docker, verify health |+| 1a Baseline | **Sub-agent A** | Execute 68 curl commands (skip if baseline exists and no server/fixture changes) |+| 1b Agent | **Sub-agent B** | Execute 39 tasks from skill docs, record results |+| 2 Regression | **Main agent** | Compare against best_score.txt |+| 3 Analyze | **Main agent** | Diff baseline vs agent, find patterns |+| 4 Propose | **Main agent → Human** | Suggest 1 fix, get approval |+| 5 Implement | **Main agent** | Make the change, leave uncommitted |+| 6 Log | **Main agent** | Append to optimization_log.md |++## Regression Detection++Each run compares against `results/best_score.txt`, which tracks the highest pass rates achieved:++```+baseline=62/68+agent=33/39+```++- **If current run is worse**: Regression detected. The improvement priority shifts to fixing or reverting the last change before proposing anything new.+- **If equal or better**: Update `best_score.txt` with the new high-water mark.+## Improvement Priority1. **API/CLI Bug** — If a curl command returns unexpected error, investigate PinchTab code@@ -63,6 +117,10 @@ Each run appends to `results/optimization_log.md`:- Baseline: X/Y (Z%)- Agent: X/Y (Z%)+**Regression Check:**+- Best: baseline X/Y, agent X/Y+- This run: [better | same | REGRESSION]+**Analysis:**- [What differed between runs]- [Root cause identified]@@ -70,7 +128,7 @@ Each run appends to `results/optimization_log.md`:**Change Made:**- [Type: api|skill|benchmark]- [Description]-- [Commit: abc1234]+- [Status: uncommitted — pending manual review]**Next Focus:**- [What to look at next run]@@ -80,4 +138,5 @@ Each run appends to `results/optimization_log.md`:- `run-optimization.sh` — Main loop script- `results/optimization_log.md` — Run history-- All changes committed to `feat/benchmark` branch+- `results/best_score.txt` — High-water mark for regression detection+- Changes left uncommitted for manual review and commitdiff --git a/tests/benchmark/TEST_CASES.md b/tests/benchmark/TEST_CASES.mdnew file mode 100644index 00000000..31b32ad9--- /dev/null+++ b/tests/benchmark/TEST_CASES.md@@ -0,0 +1,61 @@+# PinchTab Benchmark Test Cases++39 test cases covering realistic browser automation scenarios against local fixture pages.++| # | Task | Description |+|---|------|-------------|+| **Group 0: Setup** | | |+| 0.1 | Health check | Confirm PinchTab is running with active instance |+| 0.2 | Fixtures reachable | Navigate to fixtures root, confirm accessible |+| **Group 1: Reading & Extracting** | | |+| 1.1 | Wiki categories | Extract category names + article counts from wiki index |+| 1.2 | Click a link | From wiki, click through to Go article |+| 1.3 | Table extraction | Read infobox — who designed Go, what year |+| 1.4 | Count list items | Count key features on Go article (expect 6) |+| 1.5 | Article headlines | Navigate to articles page, list all titles |+| 1.6 | Dashboard metrics | Extract Total Users, Revenue, Conversion Rate |+| **Group 2: Search & Dynamic** | | |+| 2.1 | Wiki search | Use search form to find "golang" |+| 2.2 | No results search | Search for nonexistent term, verify graceful |+| 2.3 | AI content search | Search for "artificial intelligence" |+| **Group 3: Form** | | |+| 3.1 | Complete form | Fill all fields + submit (name, email, phone, country, subject, message, checkbox, radio) |+| 3.2 | Reset/refill | After submit, verify reset button exists |+| **Group 4: SPA** | | |+| 4.1 | Read app state | Count tasks: 3 total, 2 active, 1 done |+| 4.2 | Add task | Add "Automate deployment" with high priority |+| 4.3 | Delete task | Delete first task, verify count changed |+| **Group 5: Login** | | |+| 5.1 | Invalid login | Try wrong credentials, verify error |+| 5.2 | Valid login | Login with correct creds, verify dashboard |+| **Group 6: E-commerce** | | |+| 6.1 | Research products | List products + prices, identify out-of-stock |+| 6.2 | Add to cart | Add 2 items, verify $449.98 total |+| 6.3 | Checkout | Complete order, verify confirmation |+| **Group 7: Content + Interaction** | | |+| 7.1 | Read & comment | Read Go article, post comment with 5-star rating |+| 7.2 | Cross-page research | Find biggest category on wiki, navigate to an article in it |+| **Group 8: Error Handling** | | |+| 8.1 | 404 handling | Navigate to missing page, verify no crash |+| 8.2 | Missing element | Click nonexistent selector, verify clear error |+| **Group 9: Export** | | |+| 9.1 | Screenshot | Take screenshot of dashboard |+| 9.2 | PDF export | Export dashboard as PDF |+| **Group 10: Modals** | | |+| 10.1 | Open modal | Click Settings on dashboard, verify modal |+| 10.2 | Modal interaction | Change theme to Dark, save, verify applied |+| **Group 11: Persistence** | | |+| 11.1 | State after reload | Add task, reload SPA, verify task persists |+| 11.2 | Logout/re-login | Sign out, log back in, verify session renewed |+| **Group 12: Multi-page Nav** | | |+| 12.1 | Navigate & return | Home -> wiki -> Go -> back -> back -> home |+| 12.2 | Cross-page compare | Compare article counts between wiki and articles page |+| **Group 13: Form Validation** | | |+| 13.1 | Required field | Submit without email, verify blocked |+| 13.2 | Optional field | Submit without phone, verify success |+| **Group 14: Dynamic Content** | | |+| 14.1 | Load more | Click "Load More Products" on ecommerce |+| 14.2 | Lazy-loaded item | Add a lazy-loaded product to cart |+| **Group 15: Data Aggregation** | | |+| 15.1 | Financial calc | Extract revenue + profit, calculate margin |+| 15.2 | Multi-page comparison | Compare features across 3 wiki language pages |diff --git a/tests/benchmark/scripts/record-agent-step.sh b/tests/benchmark/scripts/record-agent-step.shdeleted file mode 100755index 34def848..00000000--- a/tests/benchmark/scripts/record-agent-step.sh+++ /dev/null@@ -1,100 +0,0 @@-#!/bin/bash-#-# Record a benchmark step result-#-# Usage:-# ./record-step.sh <group> <step> <pass|fail|skip> <input_tokens> <output_tokens> [notes]-#-# Example:-# ./record-step.sh 1 1 pass 150 45 "Navigation completed"-# ./record-step.sh 2 3 fail 200 80 "Element not found"--set -euo pipefail--SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"-RESULTS_DIR="${SCRIPT_DIR}/../results"--# Find most recent report file-REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_benchmark_*.json 2>/dev/null | head -1)--if [[ -z "${REPORT_FILE}" ]]; then- echo "ERROR: No benchmark report found. Run ./run-benchmark.sh first."- exit 1-fi--if [[ $# -lt 5 ]]; then- echo "Usage: $0 <group> <step> <pass|fail|skip> <input_tokens> <output_tokens> [notes]"- exit 1-fi--GROUP="$1"-STEP="$2"-STATUS="$3"-INPUT_TOKENS="$4"-OUTPUT_TOKENS="$5"-NOTES="${6:-}"-TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ)-TOTAL_TOKENS=$((INPUT_TOKENS + OUTPUT_TOKENS))--# Calculate cost based on model (from report)-MODEL=$(jq -r '.benchmark.model' "${REPORT_FILE}")-PROVIDER=$(jq -r '.benchmark.provider' "${REPORT_FILE}")--# Cost per 1M tokens-case "${MODEL}" in- *haiku*) INPUT_RATE=0.25; OUTPUT_RATE=1.25 ;;- *sonnet*) INPUT_RATE=3.0; OUTPUT_RATE=15.0 ;;- *opus*) INPUT_RATE=15.0; OUTPUT_RATE=75.0 ;;- *gpt-4o-mini*) INPUT_RATE=0.15; OUTPUT_RATE=0.60 ;;- *gpt-4o*) INPUT_RATE=2.50; OUTPUT_RATE=10.0 ;;- *gpt-4*) INPUT_RATE=10.0; OUTPUT_RATE=30.0 ;;- *gemini*flash*) INPUT_RATE=0.075; OUTPUT_RATE=0.30 ;;- *gemini*pro*) INPUT_RATE=1.25; OUTPUT_RATE=5.0 ;;- *) INPUT_RATE=1.0; OUTPUT_RATE=3.0 ;;-esac--# Calculate cost (using bc for floating point)-COST=$(echo "scale=6; (${INPUT_TOKENS} / 1000000 * ${INPUT_RATE}) + (${OUTPUT_TOKENS} / 1000000 * ${OUTPUT_RATE})" | bc)--# Create step entry-STEP_JSON=$(cat << EOF-{- "group": ${GROUP},- "step": ${STEP},- "id": "${GROUP}.${STEP}",- "status": "${STATUS}",- "input_tokens": ${INPUT_TOKENS},- "output_tokens": ${OUTPUT_TOKENS},- "total_tokens": ${TOTAL_TOKENS},- "cost_usd": ${COST},- "notes": "${NOTES}",- "timestamp": "${TIMESTAMP}"-}-EOF-)--# Append to report-TMP_FILE=$(mktemp)-jq --argjson step "${STEP_JSON}" '.steps += [$step]' "${REPORT_FILE}" > "${TMP_FILE}"--# Update totals-jq --argjson in "${INPUT_TOKENS}" --argjson out "${OUTPUT_TOKENS}" --argjson cost "${COST}" \- --arg status "${STATUS}" \- '.totals.input_tokens += $in |- .totals.output_tokens += $out |- .totals.total_tokens += ($in + $out) |- .totals.estimated_cost_usd += $cost |- if $status == "pass" then .totals.steps_passed += 1- elif $status == "fail" then .totals.steps_failed += 1- else .totals.steps_skipped += 1 end' \- "${TMP_FILE}" > "${REPORT_FILE}"--rm -f "${TMP_FILE}"--# If failed, also log to error log with details-if [[ "${STATUS}" == "fail" ]]; then- ERROR_LOG="${RESULTS_DIR}/errors.log"- echo "[${TIMESTAMP}] Step ${GROUP}.${STEP} FAILED: ${NOTES}" >> "${ERROR_LOG}"-fi--echo "Recorded: Step ${GROUP}.${STEP} = ${STATUS} (${TOTAL_TOKENS} tokens, \$${COST})"diff --git a/tests/benchmark/scripts/record-step.sh b/tests/benchmark/scripts/record-step.shindex 4702da88..ec5fddf9 100755--- a/tests/benchmark/scripts/record-step.sh+++ b/tests/benchmark/scripts/record-step.sh@@ -3,98 +3,149 @@# Record a benchmark step result## Usage:-# ./record-step.sh <group> <step> <pass|fail|skip> <input_tokens> <output_tokens> [notes]+# ./record-step.sh [--type baseline|agent] <group> <step> <pass|fail|skip> [options] "notes"#-# Example:-# ./record-step.sh 1 1 pass 150 45 "Navigation completed"-# ./record-step.sh 2 3 fail 200 80 "Element not found"+# Options:+# --type baseline|agent Report type (default: auto-detect most recent)+# --tokens <in> <out> Token usage (agent runs only, default: 0 0)+# --bytes <n> HTTP response size in bytes (baseline runs)+#+# Examples:+# ./record-step.sh 1 1 pass "Navigation completed"+# ./record-step.sh --type agent 2 3 fail --tokens 200 80 "Element not found"+# ./record-step.sh --type baseline 1 2 pass --bytes 4520 "Snapshot returned"set -euo pipefailSCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"RESULTS_DIR="${SCRIPT_DIR}/../results"+mkdir -p "${RESULTS_DIR}"-# Find most recent report file-REPORT_FILE=$(ls -t "${RESULTS_DIR}"/baseline_*.json "${RESULTS_DIR}"/benchmark_*.json 2>/dev/null | head -1)+# Parse flags+REPORT_TYPE=""+INPUT_TOKENS=0+OUTPUT_TOKENS=0+RESPONSE_BYTES=0-if [[ -z "${REPORT_FILE}" ]]; then- echo "ERROR: No benchmark report found. Run ./run-benchmark.sh first."- exit 1-fi+while [[ $# -gt 0 && "$1" == --* ]]; do+ case "$1" in+ --type)+ REPORT_TYPE="$2"+ shift 2+ ;;+ --tokens)+ INPUT_TOKENS="$2"+ OUTPUT_TOKENS="$3"+ shift 3+ ;;+ --bytes)+ RESPONSE_BYTES="$2"+ shift 2+ ;;+ *)+ echo "Unknown option: $1"+ exit 1+ ;;+ esac+done-if [[ $# -lt 5 ]]; then- echo "Usage: $0 <group> <step> <pass|fail|skip> <input_tokens> <output_tokens> [notes]"+if [[ $# -lt 3 ]]; then+ echo "Usage: $0 [--type baseline|agent] <group> <step> <pass|fail|skip> [--tokens <in> <out>] [--bytes <n>] [notes]"exit 1fiGROUP="$1"STEP="$2"STATUS="$3"-INPUT_TOKENS="$4"-OUTPUT_TOKENS="$5"-NOTES="${6:-}"+NOTES="${4:-}"TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ)TOTAL_TOKENS=$((INPUT_TOKENS + OUTPUT_TOKENS))-# Calculate cost based on model (from report)-MODEL=$(jq -r '.benchmark.model' "${REPORT_FILE}")-PROVIDER=$(jq -r '.benchmark.provider' "${REPORT_FILE}")--# Cost per 1M tokens-case "${MODEL}" in- *haiku*) INPUT_RATE=0.25; OUTPUT_RATE=1.25 ;;- *sonnet*) INPUT_RATE=3.0; OUTPUT_RATE=15.0 ;;- *opus*) INPUT_RATE=15.0; OUTPUT_RATE=75.0 ;;- *gpt-4o-mini*) INPUT_RATE=0.15; OUTPUT_RATE=0.60 ;;- *gpt-4o*) INPUT_RATE=2.50; OUTPUT_RATE=10.0 ;;- *gpt-4*) INPUT_RATE=10.0; OUTPUT_RATE=30.0 ;;- *gemini*flash*) INPUT_RATE=0.075; OUTPUT_RATE=0.30 ;;- *gemini*pro*) INPUT_RATE=1.25; OUTPUT_RATE=5.0 ;;- *) INPUT_RATE=1.0; OUTPUT_RATE=3.0 ;;-esac--# Calculate cost (using bc for floating point)-COST=$(echo "scale=6; (${INPUT_TOKENS} / 1000000 * ${INPUT_RATE}) + (${OUTPUT_TOKENS} / 1000000 * ${OUTPUT_RATE})" | bc)+# Find report file+if [[ -n "${REPORT_TYPE}" ]]; then+ case "${REPORT_TYPE}" in+ baseline)+ REPORT_FILE=$(ls -t "${RESULTS_DIR}"/baseline_*.json 2>/dev/null | head -1)+ ;;+ agent)+ REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_benchmark_*.json 2>/dev/null | head -1)+ ;;+ *)+ echo "ERROR: --type must be 'baseline' or 'agent'"+ exit 1+ ;;+ esac+else+ # Auto-detect: find most recent report of any type+ REPORT_FILE=$(ls -t "${RESULTS_DIR}"/baseline_*.json "${RESULTS_DIR}"/agent_benchmark_*.json 2>/dev/null | head -1)+fi++if [[ -z "${REPORT_FILE}" ]]; then+ echo "ERROR: No benchmark report found. Run ./run-optimization.sh first."+ exit 1+fi++# Calculate cost (only meaningful for agent runs with token data)+COST=0+if [[ ${TOTAL_TOKENS} -gt 0 ]]; then+ MODEL=$(jq -r '.benchmark.model' "${REPORT_FILE}")++ # Cost per 1M tokens+ case "${MODEL}" in+ *haiku*) INPUT_RATE=0.25; OUTPUT_RATE=1.25 ;;+ *sonnet*) INPUT_RATE=3.0; OUTPUT_RATE=15.0 ;;+ *opus*) INPUT_RATE=15.0; OUTPUT_RATE=75.0 ;;+ *gpt-4o-mini*) INPUT_RATE=0.15; OUTPUT_RATE=0.60 ;;+ *gpt-4o*) INPUT_RATE=2.50; OUTPUT_RATE=10.0 ;;+ *gpt-4*) INPUT_RATE=10.0; OUTPUT_RATE=30.0 ;;+ *gemini*flash*) INPUT_RATE=0.075; OUTPUT_RATE=0.30 ;;+ *gemini*pro*) INPUT_RATE=1.25; OUTPUT_RATE=5.0 ;;+ *) INPUT_RATE=1.0; OUTPUT_RATE=3.0 ;;+ esac++ COST=$(echo "scale=6; (${INPUT_TOKENS} / 1000000 * ${INPUT_RATE}) + (${OUTPUT_TOKENS} / 1000000 * ${OUTPUT_RATE})" | bc)+fi# Create step entry-STEP_JSON=$(cat << EOF-{- "group": ${GROUP},- "step": ${STEP},- "id": "${GROUP}.${STEP}",- "status": "${STATUS}",- "input_tokens": ${INPUT_TOKENS},- "output_tokens": ${OUTPUT_TOKENS},- "total_tokens": ${TOTAL_TOKENS},- "cost_usd": ${COST},- "notes": "${NOTES}",- "timestamp": "${TIMESTAMP}"-}-EOF-)--# Append to report-TMP_FILE=$(mktemp)-jq --argjson step "${STEP_JSON}" '.steps += [$step]' "${REPORT_FILE}" > "${TMP_FILE}"+STEP_JSON=$(jq -n \+ --argjson group "${GROUP}" \+ --argjson step "${STEP}" \+ --arg id "${GROUP}.${STEP}" \+ --arg status "${STATUS}" \+ --argjson in_tokens "${INPUT_TOKENS}" \+ --argjson out_tokens "${OUTPUT_TOKENS}" \+ --argjson total_tokens "${TOTAL_TOKENS}" \+ --argjson cost "${COST}" \+ --argjson bytes "${RESPONSE_BYTES}" \+ --arg notes "${NOTES}" \+ --arg ts "${TIMESTAMP}" \+ '{group: $group, step: $step, id: $id, status: $status,+ input_tokens: $in_tokens, output_tokens: $out_tokens,+ total_tokens: $total_tokens, cost_usd: $cost,+ response_bytes: $bytes, notes: $notes, timestamp: $ts}')-# Update totals-jq --argjson in "${INPUT_TOKENS}" --argjson out "${OUTPUT_TOKENS}" --argjson cost "${COST}" \+# Append to report and update totals+TMP_FILE=$(mktemp)+jq --argjson step "${STEP_JSON}" \+ --argjson in "${INPUT_TOKENS}" \+ --argjson out "${OUTPUT_TOKENS}" \+ --argjson cost "${COST}" \--arg status "${STATUS}" \- '.totals.input_tokens += $in |+ '.steps += [$step] |+ .totals.input_tokens += $in |.totals.output_tokens += $out |.totals.total_tokens += ($in + $out) |.totals.estimated_cost_usd += $cost |if $status == "pass" then .totals.steps_passed += 1elif $status == "fail" then .totals.steps_failed += 1else .totals.steps_skipped += 1 end' \- "${TMP_FILE}" > "${REPORT_FILE}"+ "${REPORT_FILE}" > "${TMP_FILE}"-rm -f "${TMP_FILE}"+mv "${TMP_FILE}" "${REPORT_FILE}"-# If failed, also log to error log with details+# Log failuresif [[ "${STATUS}" == "fail" ]]; then- ERROR_LOG="${RESULTS_DIR}/errors.log"- echo "[${TIMESTAMP}] Step ${GROUP}.${STEP} FAILED: ${NOTES}" >> "${ERROR_LOG}"+ echo "[${TIMESTAMP}] Step ${GROUP}.${STEP} FAILED: ${NOTES}" >> "${RESULTS_DIR}/errors.log"fi-echo "Recorded: Step ${GROUP}.${STEP} = ${STATUS} (${TOTAL_TOKENS} tokens, \$${COST})"+echo "Recorded: Step ${GROUP}.${STEP} = ${STATUS} (tokens: ${TOTAL_TOKENS}, bytes: ${RESPONSE_BYTES}, \$${COST})"diff --git a/tests/benchmark/scripts/run-optimization.sh b/tests/benchmark/scripts/run-optimization.shindex f96e0dcd..caf512d3 100755--- a/tests/benchmark/scripts/run-optimization.sh+++ b/tests/benchmark/scripts/run-optimization.sh@@ -7,7 +7,8 @@set -euo pipefailSCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"-RESULTS_DIR="${SCRIPT_DIR}/results"+RESULTS_DIR="${SCRIPT_DIR}/../results"+mkdir -p "${RESULTS_DIR}"LOG_FILE="${RESULTS_DIR}/optimization_log.md"TIMESTAMP=$(date +%Y%m%d_%H%M%S)RUN_NUMBER=$(grep -c "^## Run #" "${LOG_FILE}" 2>/dev/null || echo 0)@@ -16,14 +17,8 @@ RUN_NUMBER=$((RUN_NUMBER + 1))echo "=== PinchTab Optimization Run #${RUN_NUMBER} ==="echo "Timestamp: ${TIMESTAMP}"-# Ensure we're on feat/benchmark branch+# Run on current branchcd ~/dev/pinchtab-CURRENT_BRANCH=$(git branch --show-current)-if [[ "${CURRENT_BRANCH}" != "feat/benchmark" ]]; then- echo "Switching to feat/benchmark branch..."- git checkout feat/benchmark-fi-git pull --rebase origin feat/benchmark 2>/dev/null || true# Ensure Docker is runningcd "${SCRIPT_DIR}"