Skip to content

High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.

feat: align benchmark suites and improve SKILL.md from optimization loop findings

Bosch94dfe6c4cbf4
diff --git a/skills/pinchtab/SKILL.md b/skills/pinchtab/SKILL.md
index 534ff93f..3c38525f 100644
--- a/skills/pinchtab/SKILL.md
+++ b/skills/pinchtab/SKILL.md
@@ -129,143 +129,21 @@ pinchtab snap -i -c
## Challenge Solving
-PinchTab includes a pluggable solver framework that auto-detects and resolves browser challenges (Cloudflare Turnstile, CAPTCHAs, interstitials). Use this **after navigation** when the page shows a challenge instead of the expected content.
-
-**Important:** Solvers work best with `stealthLevel: "full"` in the PinchTab config (or `instanceDefaults.stealthLevel: "full"`). Full stealth mode patches CDP detection vectors, rotates fingerprints, and masks automation signals — all of which challenge providers like Cloudflare check before and after the checkbox click. Without full stealth, the solver may click correctly but the challenge can still fail fingerprint verification.
-
-```bash
-# Auto-detect and solve any challenge on the current page
-curl -X POST http://localhost:9867/solve \
- -H 'Content-Type: application/json' \
- -d '{"maxAttempts": 3, "timeout": 30000}'
-
-# Use a specific solver
-curl -X POST http://localhost:9867/solve/cloudflare \
- -H 'Content-Type: application/json' \
- -d '{"maxAttempts": 3}'
-
-# Tab-scoped solve
-curl -X POST http://localhost:9867/tabs/TAB_ID/solve \
- -H 'Content-Type: application/json' \
- -d '{}'
-
-# List available solvers
-curl http://localhost:9867/solvers
-```
-
-**When to use solve:**
-
-- Page title is "Just a moment..." or similar challenge indicator
-- `pinchtab text` returns empty or challenge-page text after navigation
-- A Cloudflare Turnstile widget blocks the target content
-
-**Workflow pattern:**
-
-```bash
-pinchtab nav https://protected-site.com
-pinchtab text # Check if page loaded or shows challenge
-# If challenge detected:
-curl -X POST http://localhost:9867/solve \
- -H 'Content-Type: application/json' -d '{}'
-pinchtab text # Verify: should now show real page content
-```
-
-**Response fields:** `solver` (which solver handled it), `solved` (bool), `challengeType` (e.g. "managed"), `attempts`, `title` (final page title).
-
-The auto-detect mode (`POST /solve` without specifying a solver) tries each registered solver in order and returns immediately with `solved: true, attempts: 0` if no challenge is present. This makes it safe to call speculatively after any navigation.
+If a page shows a challenge instead of content (e.g., "Just a moment..."), call `POST /solve` with `{"maxAttempts": 3}` to auto-detect and resolve it. Use `POST /tabs/TAB_ID/solve` for tab-scoped. Works best with `stealthLevel: "full"` in config. Safe to call speculatively — returns immediately if no challenge is present. See [api.md](./references/api.md) for full solver options.
## Handling Authentication and State
-Pick one of these five patterns before you start interacting with the site.
-
-### 1. One-off public browsing
-
-Use a temporary instance for public pages, scraping, or tasks that do not need login persistence.
-
-```bash
-pinchtab instance start
-pinchtab instances
-# Point CLI commands at the instance port you want to use.
-pinchtab --server http://localhost:9868 nav https://pinchtab.com
-pinchtab --server http://localhost:9868 text
-```
-
-### 2. Reuse an existing named profile
+Pick a pattern before interacting with the site:
-Use this for recurring tasks against the same authenticated site.
-
-```bash
-pinchtab profiles
-pinchtab instance start --profile work --mode headed
-pinchtab --server http://localhost:9868 nav https://mail.google.com
-```
-
-If the login is already stored in that profile, you can switch to headless later:
-
-```bash
-pinchtab instance stop inst_ea2e747f
-pinchtab instance start --profile work --mode headless
-```
+1. **One-off browsing**: `pinchtab instance start` → use `--server http://localhost:<port>` for commands.
+2. **Reuse a profile**: `pinchtab instance start --profile work --mode headed` → switch to `--mode headless` after login is stored.
+3. **Create profile via HTTP**: `POST /profiles` with `{"name":"..."}`, then `POST /profiles/<name>/start`.
+4. **Human-assisted login**: Start headed, human signs in, agent reuses the profile headless.
+5. **HTTP-only agent**: Use `POST /instances/start`, then target the instance port with curl. Send `X-Agent-Id` for attribution.
-### 3. Create a dedicated auth profile over HTTP
-
-Use this when you need a durable profile and it does not exist yet.
-
-```bash
-curl -X POST http://localhost:9867/profiles \
- -H "Content-Type: application/json" \
- -d '{"name":"billing","description":"Billing portal automation","useWhen":"Use for billing tasks"}'
-
-curl -X POST http://localhost:9867/profiles/billing/start \
- -H "Content-Type: application/json" \
- -d '{"headless":false}'
-```
-
-Then target the returned port with `--server`.
-
-### 4. Human-assisted headed login, then agent reuse
-
-Use this for CAPTCHA, MFA, or first-time setup.
-
-```bash
-pinchtab instance start --profile work --mode headed
-# Human completes login in the visible Chrome window.
-pinchtab --server http://localhost:9868 nav https://app.example.com/dashboard
-pinchtab --server http://localhost:9868 snap -i -c
-```
-
-Once the session is stored, reuse the same profile for later tasks.
-
-### 5. Remote or non-shell agent with tokenized HTTP API
-
-Use this when the agent cannot call the CLI directly.
-
-```bash
-curl http://localhost:9867/health
-curl -X POST http://localhost:9867/profiles \
- -H "Content-Type: application/json" \
- -d '{"name":"work"}'
-curl -X POST http://localhost:9867/instances/start \
- -H "Content-Type: application/json" \
- -d '{"profileId":"work","mode":"headless"}'
-curl -X POST http://localhost:9868/action \
- -H "X-Agent-Id: agent-main" \
- -H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"e5"}'
-```
-
-If the server is exposed beyond localhost, require a token and use a dedicated automation profile. See [TRUST.md](./TRUST.md).
-
-**Agent sessions**: Instead of sharing the server bearer token, each agent can get its own revocable session token. Set `PINCHTAB_SESSION=ses_...` or send `Authorization: Session ses_...`. Sessions have idle timeout (default 30m) and max lifetime (default 24h). Use them only in trusted, controlled environments. They are not a multi-tenant isolation boundary, and session-authenticated callers are still intended for operator-controlled automation only.
-
-```bash
-pinchtab session create --agent-id <agent-id> [--label <label>] # Create; prints sessionToken
-pinchtab session list # List all sessions
-pinchtab session info # Show session for current token
-pinchtab session revoke <session-id> # Revoke session
-```
+If the server is exposed beyond localhost, require a token. See [TRUST.md](./TRUST.md).
-HTTP equivalents: `POST /sessions`, `GET /sessions`, `GET /sessions/me`, `POST /sessions/{id}/revoke`.
+**Agent sessions**: Each agent can get its own revocable session token via `pinchtab session create --agent-id <id>` or `POST /sessions`. Set `PINCHTAB_SESSION=ses_...` or send `Authorization: Session ses_...`. Sessions have idle timeout (default 30m) and max lifetime (default 24h).
## Essential Commands
@@ -319,6 +197,7 @@ Guidance:
- `snap -d` is the default follow-up snapshot for multi-step flows.
- `text` is the default for reading articles, dashboards, reports, or confirmation messages.
- `find --ref-only` is useful when the page is large and you already know the semantic target.
+- Refs from `snap -i` and full `snap` use different numbering. Do not mix them — if you snapshot with `-i`, use those refs. If you re-snapshot without `-i`, get fresh refs before acting.
### Interaction
@@ -343,6 +222,7 @@ Rules:
- Prefer `type` only when the site depends on keystroke events.
- Prefer `click --wait-nav` when a click is expected to navigate.
- Re-snapshot immediately after `click`, `press Enter`, `select`, or `scroll` if the UI can change.
+- To discover valid dropdown values, snapshot with `filter=interactive` first — the output shows `<option>` elements with their `value` attributes. Then use `select` with the exact value.
### Export, debug, and verification
@@ -400,6 +280,80 @@ Use the API when:
- profile creation or mutation is required,
- or you need explicit instance- and tab-scoped routes.
+### Tab-scoped HTTP API
+
+**Important:** Each `POST /navigate` creates a new tab by default. The default (non-tab-scoped) endpoints like `/snapshot`, `/action`, `/text` operate on the *active* tab, which may not be the one you just navigated. In multi-tab workflows, always use tab-scoped routes to avoid acting on the wrong page.
+
+Get the tab ID from the navigate response or from `GET /tabs`.
+
+```bash
+# List all tabs
+curl http://localhost:9867/tabs \
+ -H "Authorization: Bearer <token>"
+
+# Navigate in a specific tab (does not create a new tab)
+curl -X POST http://localhost:9867/tabs/TAB_ID/navigate \
+ -H "Authorization: Bearer <token>" \
+ -H "Content-Type: application/json" \
+ -d '{"url":"https://example.com"}'
+
+# Snapshot a specific tab
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?filter=interactive&format=compact" \
+ -H "Authorization: Bearer <token>"
+
+# Get text from a specific tab
+curl http://localhost:9867/tabs/TAB_ID/text \
+ -H "Authorization: Bearer <token>"
+
+# Perform action on a specific tab
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer <token>" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"click","selector":"#submit-btn"}'
+
+# Navigate back/forward in a specific tab
+curl -X POST http://localhost:9867/tabs/TAB_ID/back \
+ -H "Authorization: Bearer <token>"
+curl -X POST http://localhost:9867/tabs/TAB_ID/forward \
+ -H "Authorization: Bearer <token>"
+
+# Screenshot (GET, not POST)
+curl http://localhost:9867/tabs/TAB_ID/screenshot \
+ -H "Authorization: Bearer <token>" \
+ --output screenshot.png
+
+# PDF export (GET or POST)
+curl http://localhost:9867/tabs/TAB_ID/pdf \
+ -H "Authorization: Bearer <token>" \
+ --output page.pdf
+
+# Close a tab
+curl -X POST http://localhost:9867/tabs/TAB_ID/close \
+ -H "Authorization: Bearer <token>"
+```
+
+The default (non-tab-scoped) endpoints also support screenshots and PDF:
+
+```bash
+# Screenshot of active tab (GET)
+curl http://localhost:9867/screenshot \
+ -H "Authorization: Bearer <token>" \
+ --output screenshot.png
+
+# PDF of active tab (GET or POST)
+curl http://localhost:9867/pdf \
+ -H "Authorization: Bearer <token>" \
+ --output page.pdf
+```
+
+**Navigation with `waitNav`:** When clicking a link or button that triggers page navigation, include `"waitNav": true` in the action body. Without it, PinchTab returns a `navigation_changed` error to protect against unexpected navigation during form interactions.
+
+```json
+{"kind": "click", "selector": "#search-btn", "waitNav": true}
+```
+
+All tab-scoped routes follow the pattern `/tabs/{TAB_ID}/...` and mirror the default endpoints. The full list includes: `navigate`, `back`, `forward`, `reload`, `snapshot`, `screenshot`, `text`, `pdf`, `action`, `actions`, `dialog`, `wait`, `find`, `lock`, `unlock`, `cookies`, `metrics`, `network`, `solve`, `close`, `storage`, `evaluate`, `download`, `upload`.
+
## Common Patterns
### Open a page and inspect actions
@@ -429,34 +383,7 @@ pinchtab click e3 # Click the Search button
pinchtab text
```
-**Form submission rules:**
-- ❌ **NEVER use `press Enter` on regular form inputs.** It does NOT submit standard HTML forms.
-- ✅ **ALWAYS click the submit button** to trigger form submission handlers.
-- **Why?** HTML5 forms only auto-submit on Enter if they have explicit JavaScript `onkeypress` or `onkeyup` handlers. The `<form onsubmit>` handler only fires when you click the button, not on Enter.
-
-**Real-world example from benchmark:**
-```bash
-# WRONG: This fails
-pinchtab nav http://fixtures/wiki.html
-pinchtab snap -i -c
-pinchtab fill "#wiki-search-input" "go"
-pinchtab press Enter # ❌ Does NOT submit the form
-
-# RIGHT: This works
-pinchtab nav http://fixtures/wiki.html
-pinchtab snap -i -c
-pinchtab fill "#wiki-search-input" "go"
-pinchtab click "#wiki-search-btn" # ✅ Triggers onsubmit handler
-pinchtab text # Now on the results page
-```
-
-**Always follow this pattern:**
-```bash
-# Template: fill, then click
-pinchtab fill "<selector>" "value"
-pinchtab click "<button-selector>" # Always click, never press Enter
-pinchtab text # Verify the form was submitted
-```
+**Form submission:** Always click the submit button — never use `press Enter`. Most HTML forms only fire their submission handler on button click, not on Enter keypress.
### Use diff snapshots in a multi-step flow
@@ -477,36 +404,6 @@ pinchtab fill "#search" "quarterly report"
pinchtab click "xpath://button[@type='submit']"
```
-### Navigate through a Cloudflare-protected site
-
-```bash
-pinchtab nav https://protected-site.com
-# Page may show CF challenge ("Just a moment...")
-curl -X POST http://localhost:9867/solve \
- -H 'Content-Type: application/json' -d '{"maxAttempts": 3}'
-# Now the real page is loaded — proceed normally
-pinchtab snap -i -c
-pinchtab text
-```
-
-### Bootstrap an authenticated profile
-
-```bash
-pinchtab profiles
-pinchtab instance start --profile work --mode headed
-# Human signs in once.
-pinchtab --server http://localhost:9868 text
-```
-
-### Run separate instances for separate sites
-
-```bash
-pinchtab instance start --profile work --mode headless
-pinchtab instance start --profile staging --mode headless
-pinchtab instances
-```
-
-Then point each command stream at its own port using `--server`.
## Security and Token Economy
@@ -523,16 +420,6 @@ Then point each command stream at its own port using `--server`.
- Use `pinchtab screenshot` only when visual regressions, CAPTCHA, or layout-specific confirmation matters.
- If a ref disappears after a change, treat that as expected and fetch fresh refs instead of retrying the stale one.
-## Privacy and Security
-
-PinchTab is a fully open-source, local-only browser automation tool:
-
-- **Runs on localhost only.** The server binds to `127.0.0.1` by default. No external network calls are made by PinchTab itself.
-- **No telemetry or analytics.** The binary makes zero outbound connections.
-- **Single Go binary (~16 MB).** Fully verifiable — anyone can build from source at [github.com/pinchtab/pinchtab](https://github.com/pinchtab/pinchtab).
-- **Local Chrome profiles.** Persistent profiles store cookies and sessions on your machine only. This enables agents to reuse authenticated sessions without re-entering credentials, similar to how a human reuses their browser profile.
-- **Token-efficient by design.** Uses the accessibility tree (structured text) instead of screenshots, keeping agent context windows small. Comparable to Playwright but purpose-built for AI agents.
-- **Multi-instance isolation.** Each browser instance runs in its own profile directory with tab-level locking for safe multi-agent use.
## References
@@ -543,36 +430,4 @@ PinchTab is a fully open-source, local-only browser automation tool:
- MCP: [mcp.md](./references/mcp.md)
- Security model: [TRUST.md](./TRUST.md)
-## Content Extraction: text vs snapshot
-
-Choose the right extraction method:
-
-| Use Case | Recommended | Why |
-|----------|-------------|-----|
-| Article body, paragraphs | `text` | Clean prose extraction |
-| Prices, numbers in cards | `snapshot` | Text strips structured data |
-| Form field values | `snapshot` | See current input values |
-| Verify element exists | `snapshot` with selector | Text won't show headings |
-| JS-rendered content | `snapshot` after wait | Text may miss dynamic content |
-
-**Common pitfall**: `/text` extracts readable prose but strips headings, prices, and structured UI elements. If you need to verify a heading, price, or button label, use `/snapshot` instead.
-
-```bash
-# Wrong: looking for "$149.99" in text output
-pinchtab text | grep "149.99" # May fail - prices often stripped
-
-# Right: snapshot includes all visible text
-pinchtab snap -c | grep "149.99" # Works
-```
-## Fixture Selector Quick Reference
-
-| Page | Key selectors |
-|------|---------------|
-| ecommerce | `.add-to-cart`, `#checkout-btn`, `.price` (use `snapshot` not `-c` for price values) |
-| search | `#search-input`, `#search-btn` (click button — never press Enter) |
-| form | `#fullname`, `#email`, `#phone`, `#country`, `#subject`, `#message`, `#submit-btn` |
-| wiki | `#wiki-search-input`, `#wiki-search-btn` |
-| spa | `#new-task-input`, `#priority-select`, `#add-task-btn` |
-| login | `#username`, `#password`, `#login-btn`, `#logout-btn` |
-| dashboard | `#settings-btn`, `#theme-select`, `#modal-save` |
diff --git a/tests/benchmark/AGENT_TASKS.md b/tests/benchmark/AGENT_TASKS.md
index 43c2aeaf..a36f3bfe 100644
--- a/tests/benchmark/AGENT_TASKS.md
+++ b/tests/benchmark/AGENT_TASKS.md
@@ -7,7 +7,7 @@ Natural language tasks to test how well an agent uses PinchTab from skill docs a
1. Read `../../skills/pinchtab/SKILL.md` — this is your only guide
2. For each task, figure out which commands to use
3. **Log every command executed**
-4. Record: `./record-agent-step.sh <group> <step> <pass|fail> <in> <out> "commands" "notes"`
+4. Record: `./scripts/record-step.sh --type agent <group> <step> <pass|fail> --tokens <in> <out> "notes"`
## Environment
@@ -121,7 +121,7 @@ Add a task called "Automate deployment" with high priority.
**Verify**: Task appeared in the list (`TASK_ADDED_AUTOMATE_DEPLOYMENT_PRIORITY_HIGH`).
### 4.3 Delete a task
-Delete the first existing task ("Write benchmark tests").
+Delete the task titled "Write benchmark tests".
**Verify**: Task count changed (went from 4 to 3).
diff --git a/tests/benchmark/BENCHMARK_TASKS.md b/tests/benchmark/BENCHMARK_TASKS.md
index 84de1491..9931d2e6 100644
--- a/tests/benchmark/BENCHMARK_TASKS.md
+++ b/tests/benchmark/BENCHMARK_TASKS.md
@@ -1,11 +1,12 @@
-# PinchTab Benchmark Tasks
+# PinchTab Benchmark Tasks (Baseline)
-Reproducible benchmark tasks using controlled fixture pages with verifiable content.
+Reproducible benchmark using explicit curl commands against controlled fixture pages.
+Each task maps 1:1 to an agent task in AGENT_TASKS.md for direct comparison.
## Environment
- **PinchTab Server**: `http://localhost:9867`
-- **Fixtures Server**: `http://fixtures/`
+- **Fixtures Server**: `http://fixtures/` (inside Docker network)
- **Token**: `benchmark-token`
- **Auth Header**: `Authorization: Bearer benchmark-token`
@@ -20,7 +21,11 @@ docker compose up -d --build
## Recording
```bash
-./record-step.sh <group> <step> <pass|fail> <in_tokens> <out_tokens> "notes"
+# Baseline (no tokens, optional response bytes):
+./scripts/record-step.sh --type baseline <group> <step> <pass|fail> "notes"
+
+# Minimal (just pass/fail):
+./scripts/record-step.sh <group> <step> <pass|fail> "notes"
```
**On failure, include in notes:**
@@ -28,675 +33,835 @@ docker compose up -d --build
- What was actually returned
- HTTP status code / error message
----
+## Tab Reuse
-## Group 0: Setup & Configuration Verification
+`POST /navigate` creates a new tab by default. To avoid multi-tab issues, the
+runner must:
-Verify the environment is correctly configured before running tests.
+1. Capture `tabId` from the first navigate response (step 0.2).
+2. Pass `"tabId":"TAB_ID"` in every subsequent navigate request body.
+3. Use tab-scoped endpoints for actions and snapshots:
+ - `POST /tabs/TAB_ID/action` instead of `POST /action`
+ - `GET /tabs/TAB_ID/snapshot?...` instead of `GET /snapshot?...`
+ - `GET /tabs/TAB_ID/text` instead of `GET /text`
+ - `POST /tabs/TAB_ID/back` instead of `POST /back`
+ - `GET /tabs/TAB_ID/screenshot` instead of `GET /screenshot`
+ - `GET /tabs/TAB_ID/pdf` instead of `GET /pdf`
-### 0.1 Load PinchTab skill
-Read `../../skills/pinchtab/SKILL.md`.
-```bash
-./record-step.sh 0 1 pass <in> <out> "Skill loaded"
-```
-**Pass if**: Skill loaded successfully.
+All curl examples below use `TAB_ID` as a placeholder. Replace with the actual
+tab ID captured in step 0.2.
-### 0.2 Verify PinchTab health
-```bash
-curl -sf http://localhost:9867/health \
- -H "Authorization: Bearer benchmark-token" | jq .
-```
-**Pass if**: `status == "ok"`, `authRequired == true`, `instances >= 1`.
+---
-### 0.3 Verify fixtures server
-```bash
-curl -sf http://localhost:9867/navigate \
- -X POST -H "Authorization: Bearer benchmark-token" \
- -H "Content-Type: application/json" \
- -d '{"url":"http://fixtures/"}' | jq .
-```
-**Pass if**: Response contains `"url": "http://fixtures/"`.
+## Group 0: Setup
-### 0.4 Verify Chrome instance running
+### 0.1 Health check
```bash
curl -sf http://localhost:9867/health \
- -H "Authorization: Bearer benchmark-token" | jq '.defaultInstance.status'
+ -H "Authorization: Bearer benchmark-token"
```
-**Pass if**: Value is `"running"`.
-
----
-
-## Group 1: Navigation & Content Extraction
+**Pass if**: `status == "ok"`, `authRequired == true`, `instances >= 1`, `defaultInstance.status == "running"`.
-### 1.1 Navigate to fixtures home
+### 0.2 Fixtures reachable
```bash
-curl -X POST http://localhost:9867/navigate \
+curl -sf -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"url":"http://fixtures/"}'
```
-**Pass if**: HTTP 200.
-
-### 1.2 Verify home content via snapshot
+**Capture**: Save the `tabId` from the JSON response. All subsequent commands use this tab ID.
```bash
-curl "http://localhost:9867/snapshot?format=compact&maxTokens=1000" \
+curl -sf "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
-H "Authorization: Bearer benchmark-token"
```
-**Pass if**: Contains `VERIFY_HOME_LOADED_12345`.
+**Pass if**: Navigate returns HTTP 200 AND snapshot contains `VERIFY_HOME_LOADED_12345`.
+
+---
+
+## Group 1: Reading & Extracting
-### 1.3 Navigate to wiki index
+### 1.1 Wiki categories
```bash
curl -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"url":"http://fixtures/wiki.html"}'
-```
-**Pass if**: HTTP 200.
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki.html"}'
-### 1.4 Verify wiki content and count categories
-```bash
-curl "http://localhost:9867/snapshot?format=compact&maxTokens=1500" \
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \
-H "Authorization: Bearer benchmark-token"
```
**Pass if**: Contains `VERIFY_WIKI_INDEX_55555` AND `COUNT_LANGUAGES_12` AND `COUNT_TOOLS_15`.
-### 1.5 Click through to Go article
+### 1.2 Click a link
```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"click","selector":"#link-go","waitNav":true}'
```
**Pass if**: HTTP 200 with `{"success":true}`.
-**Note**: Use `waitNav: true` when clicking a link that causes page navigation. Without it, PinchTab returns a 409 "navigation_changed" error to protect against unexpected navigation during form interactions.
+### 1.3 Table extraction
+```bash
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
+ -H "Authorization: Bearer benchmark-token"
+```
+**Pass if**: Contains `VERIFY_WIKI_GO_LANG_88888` AND `Robert Griesemer` AND `2009`.
-### 1.6 Verify Go article loaded and extract key facts
+### 1.4 Count list items
```bash
-curl "http://localhost:9867/snapshot?format=compact&maxTokens=2000" \
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
-H "Authorization: Bearer benchmark-token"
```
-**Pass if**: Contains `VERIFY_WIKI_GO_LANG_88888` AND `Robert Griesemer` AND `2009` AND `FEATURE_COUNT_6`.
+**Pass if**: Contains `FEATURE_COUNT_6`.
-### 1.7 Extract specific table data (designer)
+### 1.5 Article headlines
```bash
-curl http://localhost:9867/text \
+curl -X POST http://localhost:9867/navigate \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/articles.html"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
-H "Authorization: Bearer benchmark-token"
```
-**Pass if**: Text contains `Google LLC` AND `Ken Thompson`.
+**Pass if**: Contains `The Future of Artificial Intelligence` AND `Climate Action in 2026` AND `Mars Colony`.
-### 1.8 Navigate to articles and extract all headlines
+### 1.6 Dashboard metrics
```bash
curl -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"url":"http://fixtures/articles.html"}'
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/dashboard.html"}'
-curl "http://localhost:9867/snapshot?format=compact&maxTokens=2000" \
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
-H "Authorization: Bearer benchmark-token"
```
-**Pass if**: Snapshot contains all 3 article titles: `The Future of Artificial Intelligence`, `Climate Action in 2026`, `Mars Colony`.
+**Pass if**: Contains `24,582` AND `$1,284,930` AND `4.28%`.
---
-## Group 2: Search & Dynamic Content
+## Group 2: Search & Dynamic
-### 2.1 Navigate to wiki search
+### 2.1 Wiki search
```bash
curl -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"url":"http://fixtures/wiki.html"}'
-```
-**Pass if**: HTTP 200.
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki.html"}'
-### 2.2 Get snapshot to find search input ref
-```bash
-curl "http://localhost:9867/snapshot?filter=interactive&format=compact" \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"fill","selector":"#wiki-search-input","text":"golang"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"click","selector":"#wiki-search-btn","waitNav":true}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
-H "Authorization: Bearer benchmark-token"
```
-**Pass if**: Contains search input element. Note the ref.
+**Pass if**: Snapshot contains `VERIFY_WIKI_GO_LANG_88888` (search redirected to Go page).
-### 2.3 Fill search query
+### 2.2 No results search
```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#wiki-search-input","text":"golang"}'
-```
-**Pass if**: HTTP 200.
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/search.html"}'
-### 2.4 Submit search
-```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#wiki-search-btn"}'
-```
-**Pass if**: HTTP 200.
+ -d '{"kind":"fill","selector":"#search-input","text":"xyznonexistent"}'
-### 2.5 Verify navigation to Go article
-```bash
-curl "http://localhost:9867/snapshot?format=compact&maxTokens=2000" \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"click","selector":"#search-btn"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
-H "Authorization: Bearer benchmark-token"
```
-**Pass if**: Contains `VERIFY_WIKI_GO_LANG_88888` (search redirected to Go page).
+**Pass if**: All curls return HTTP 200 AND snapshot shows no-results message.
-### 2.6 Search for something with no results
+### 2.3 AI content search
```bash
curl -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"url":"http://fixtures/search.html"}'
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/search.html"}'
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#search-input","text":"xyznonexistent"}'
+ -d '{"kind":"fill","selector":"#search-input","text":"artificial intelligence"}'
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"click","selector":"#search-btn"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
+ -H "Authorization: Bearer benchmark-token"
```
-**Pass if**: HTTP 200 (graceful no-results state).
+**Pass if**: Snapshot contains `The Future of Artificial Intelligence`.
---
-## Group 3: Complex Form Interaction
+## Group 3: Form
-### 3.1 Navigate to contact form
+### 3.1 Complete form
```bash
curl -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"url":"http://fixtures/form.html"}'
-```
-**Pass if**: HTTP 200.
-
-### 3.2 Get snapshot of interactive elements
-```bash
-curl "http://localhost:9867/snapshot?filter=interactive&format=compact" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains all expected fields: fullname, email, country, subject, message.
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/form.html"}'
-### 3.3 Fill full name
-```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"fill","selector":"#fullname","text":"John Benchmark"}'
-```
-**Pass if**: HTTP 200.
-### 3.4 Fill email
-```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"fill","selector":"#email","text":"[email protected]"}'
-```
-**Pass if**: HTTP 200.
-### 3.5 Fill phone
-```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"fill","selector":"#phone","text":"+44 20 1234 5678"}'
-```
-**Pass if**: HTTP 200.
-### 3.6 Select country
-```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"select","selector":"#country","value":"uk"}'
-```
-**Pass if**: HTTP 200.
-### 3.7 Select subject
-```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"select","selector":"#subject","value":"support"}'
-```
-**Pass if**: HTTP 200.
-### 3.8 Fill message textarea
-```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"fill","selector":"#message","text":"This is a benchmark test message for PinchTab automation."}'
-```
-**Pass if**: HTTP 200.
-### 3.9 Check newsletter checkbox
-```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"click","selector":"#newsletter"}'
-```
-**Pass if**: HTTP 200.
-### 3.10 Select high priority radio
-```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"click","selector":"input[name=priority][value=high]"}'
-```
-**Pass if**: HTTP 200.
-### 3.11 Submit form
-```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"click","selector":"#submit-btn"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
+ -H "Authorization: Bearer benchmark-token"
```
-**Pass if**: HTTP 200.
+**Pass if**: Contains `VERIFY_FORM_SUBMITTED_SUCCESS` AND `SUBMISSION_DATA_NAME_JOHN_BENCHMARK`.
-### 3.12 Verify submission with data check
+### 3.2 Reset/refill
```bash
-curl "http://localhost:9867/snapshot?format=compact&maxTokens=1000" \
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?filter=interactive&format=compact" \
-H "Authorization: Bearer benchmark-token"
```
-**Pass if**: Contains `VERIFY_FORM_SUBMITTED_SUCCESS` AND `SUBMISSION_DATA_NAME_JOHN_BENCHMARK`.
+**Pass if**: Contains reset button or form element (after submission, page still has interactive elements).
---
-## Group 4: SPA & Dynamic State
+## Group 4: SPA
-### 4.1 Navigate to task manager SPA
+### 4.1 Read app state
```bash
curl -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"url":"http://fixtures/spa.html"}'
-```
-**Pass if**: HTTP 200.
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/spa.html"}'
-### 4.2 Verify initial state
-```bash
-curl "http://localhost:9867/snapshot?format=compact&maxTokens=1500" \
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \
-H "Authorization: Bearer benchmark-token"
```
**Pass if**: Contains `VERIFY_SPA_PAGE_99999` AND `TASK_STATS_TOTAL_3_ACTIVE_2_DONE_1`.
-### 4.3 Add a new task
+### 4.2 Add task
```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"fill","selector":"#new-task-input","text":"Deploy to production"}'
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"select","selector":"#priority-select","value":"high"}'
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"click","selector":"#add-task-btn"}'
-```
-**Pass if**: All HTTP 200.
-### 4.4 Verify task was added
-```bash
-curl "http://localhost:9867/snapshot?format=compact&maxTokens=1500" \
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \
-H "Authorization: Bearer benchmark-token"
```
**Pass if**: Contains `TASK_ADDED_DEPLOY_TO_PRODUCTION_PRIORITY_HIGH`.
-### 4.5 Delete existing task
+### 4.3 Delete task
```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"kind":"click","selector":".delete-task[data-id=\"1\"]"}'
-```
-**Pass if**: HTTP 200.
+ -d '{"kind":"click","selector":"#task-1 .delete-task"}'
-### 4.6 Verify task count updated
-```bash
-curl "http://localhost:9867/snapshot?format=compact&maxTokens=500" \
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
-H "Authorization: Bearer benchmark-token"
```
-**Pass if**: Contains `3` in total count (started with 3, deleted 1, added 1 = 3).
+**Pass if**: Total count is `3` (started with 3, added 1, deleted 1 = 3).
---
-## Group 5: Login & Auth Flow
+## Group 5: Login
-### 5.1 Navigate to login page
+### 5.1 Invalid login
```bash
curl -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"url":"http://fixtures/login.html"}'
-```
-**Pass if**: HTTP 200.
-
-### 5.2 Verify login page
-```bash
-curl "http://localhost:9867/snapshot?format=compact&maxTokens=500" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Contains `VERIFY_LOGIN_PAGE_77777`.
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/login.html"}'
-### 5.3 Try invalid credentials
-```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"fill","selector":"#username","text":"baduser"}'
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"fill","selector":"#password","text":"wrongpassword"}'
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"click","selector":"#login-btn"}'
-```
-**Pass if**: All HTTP 200.
-### 5.4 Verify error shown
-```bash
-curl "http://localhost:9867/snapshot?format=compact&maxTokens=500" \
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
-H "Authorization: Bearer benchmark-token"
```
**Pass if**: Contains `INVALID_CREDENTIALS_ERROR`.
-### 5.5 Login with valid credentials
+### 5.2 Valid login
```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"fill","selector":"#username","text":"benchmark"}'
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"fill","selector":"#password","text":"test456"}'
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"click","selector":"#login-btn"}'
-```
-**Pass if**: All HTTP 200.
-### 5.6 Verify logged in
-```bash
-curl "http://localhost:9867/snapshot?format=compact&maxTokens=500" \
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
-H "Authorization: Bearer benchmark-token"
```
**Pass if**: Contains `VERIFY_LOGIN_SUCCESS_DASHBOARD` AND `SESSION_TOKEN_ACTIVE_TRUE`.
---
-## Group 6: E-commerce Flow
+## Group 6: E-commerce
-### 6.1 Navigate to shop
+### 6.1 Research products
```bash
curl -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"url":"http://fixtures/ecommerce.html"}'
-```
-**Pass if**: HTTP 200.
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/ecommerce.html"}'
-### 6.2 Verify shop page and read prices
-```bash
-curl "http://localhost:9867/snapshot?format=compact&maxTokens=2000" \
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
-H "Authorization: Bearer benchmark-token"
```
-**Pass if**: Contains `VERIFY_SHOP_PAGE_44444` AND `$149.99` AND `$299.99` AND `Out of Stock`.
+**Pass if**: Contains `VERIFY_SHOP_PAGE_44444` AND `$149.99` AND `$299.99` AND `$49.99` AND `Out of Stock`.
-### 6.3 Verify out-of-stock button is disabled
+### 6.2 Add to cart
```bash
-curl "http://localhost:9867/snapshot?filter=interactive&format=compact" \
- -H "Authorization: Bearer benchmark-token"
-```
-**Pass if**: Mechanical Keyboard add-to-cart button is shown as disabled.
-
-### 6.4 Add Wireless Headphones
-```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"click","selector":"#product-1 .add-to-cart"}'
-```
-**Pass if**: HTTP 200.
-### 6.5 Add Smart Watch
-```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"click","selector":"#product-2 .add-to-cart"}'
-```
-**Pass if**: HTTP 200.
-### 6.6 Verify cart total
-```bash
-curl "http://localhost:9867/snapshot?format=compact&maxTokens=1000" \
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
-H "Authorization: Bearer benchmark-token"
```
-**Pass if**: Contains `CART_ITEM_WIRELESS_HEADPHONES` AND `CART_ITEM_SMART_WATCH_PRO` AND `449.98` (total).
+**Pass if**: Contains `CART_ITEM_WIRELESS_HEADPHONES` AND `CART_ITEM_SMART_WATCH_PRO` AND `449.98`.
-### 6.7 Checkout
+### 6.3 Checkout
```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
-d '{"kind":"click","selector":"#checkout-btn"}'
-```
-**Pass if**: HTTP 200.
-### 6.8 Verify checkout complete
-```bash
-curl "http://localhost:9867/snapshot?format=compact&maxTokens=500" \
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
-H "Authorization: Bearer benchmark-token"
```
**Pass if**: Contains `VERIFY_CHECKOUT_SUCCESS_ORDER` AND `ORDER_TOTAL_449_98`.
---
-## Group 7: Wiki Article + Comment Flow
+## Group 7: Content + Interaction
-### 7.1 Navigate to Go article
+### 7.1 Read & comment
```bash
curl -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"url":"http://fixtures/wiki-go.html"}'
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki-go.html"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
+ -H "Authorization: Bearer benchmark-token"
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"fill","selector":"#comment-text","text":"Great article on Go! Very comprehensive."}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"select","selector":"#comment-rating","value":"5"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"click","selector":"#submit-comment"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
+ -H "Authorization: Bearer benchmark-token"
```
-**Pass if**: HTTP 200.
+**Pass if**: First snapshot contains `VERIFY_WIKI_GO_LANG_88888` AND final snapshot contains `COMMENT_POSTED_RATING_5_TEXT_RECEIVED`.
-### 7.2 Read and verify article facts
+### 7.2 Cross-page research
```bash
-curl "http://localhost:9867/snapshot?format=compact&maxTokens=2000" \
+curl -X POST http://localhost:9867/navigate \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki.html"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \
+ -H "Authorization: Bearer benchmark-token"
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"click","selector":"#link-go","waitNav":true}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
-H "Authorization: Bearer benchmark-token"
```
-**Pass if**: Contains `VERIFY_WIKI_GO_LANG_88888` AND `BSD-style` AND `go.dev`.
+**Pass if**: First snapshot contains `COUNT_LANGUAGES_12` AND second snapshot contains `VERIFY_WIKI_GO_LANG_88888`.
+
+---
+
+## Group 8: Error Handling
-### 7.3 Fill comment
+### 8.1 404 handling
```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#comment-text","text":"Great article on Go! Very comprehensive."}'
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/nonexistent-page-xyz.html"}'
```
-**Pass if**: HTTP 200.
+**Pass if**: Returns response without crash (HTTP 200 with error page, or structured error).
-### 7.4 Rate the article
+### 8.2 Missing element
```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"kind":"select","selector":"#comment-rating","value":"5"}'
+ -d '{"kind":"click","selector":"#element-that-does-not-exist"}'
```
-**Pass if**: HTTP 200.
+**Pass if**: Error response with clear message (not crash).
+
+---
+
+## Group 9: Export
-### 7.5 Submit comment
+### 9.1 Screenshot
```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#submit-comment"}'
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/dashboard.html"}'
+
+curl http://localhost:9867/tabs/TAB_ID/screenshot \
+ -H "Authorization: Bearer benchmark-token" \
+ --output /tmp/benchmark-screenshot.png
+
+ls -la /tmp/benchmark-screenshot.png
```
-**Pass if**: HTTP 200.
+**Pass if**: File exists and size > 10240 bytes.
-### 7.6 Verify comment posted
+### 9.2 PDF export
```bash
-curl "http://localhost:9867/snapshot?format=compact&maxTokens=500" \
- -H "Authorization: Bearer benchmark-token"
+curl http://localhost:9867/tabs/TAB_ID/pdf \
+ -H "Authorization: Bearer benchmark-token" \
+ --output /tmp/benchmark-dashboard.pdf
+
+ls -la /tmp/benchmark-dashboard.pdf
```
-**Pass if**: Contains `COMMENT_POSTED_RATING_5_TEXT_RECEIVED`.
+**Pass if**: File exists and size > 10240 bytes.
---
-## Group 8: Error Handling
+## Group 10: Modals
-### 8.1 Navigate to non-existent page
+### 10.1 Open modal
```bash
curl -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"url":"http://fixtures/nonexistent-page-xyz.html"}'
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/dashboard.html"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"click","selector":"#settings-btn"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
+ -H "Authorization: Bearer benchmark-token"
```
-**Pass if**: Returns response without crash (404 or error).
+**Pass if**: Snapshot contains `Dashboard Settings`.
-### 8.2 Click non-existent element
+### 10.2 Modal interaction
```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"kind":"click","selector":"#element-that-does-not-exist"}'
+ -d '{"kind":"select","selector":"#theme-select","value":"dark"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"click","selector":"#modal-save"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
+ -H "Authorization: Bearer benchmark-token"
```
-**Pass if**: Error response with message (not crash).
+**Pass if**: Contains `THEME_DARK_APPLIED`.
+
+---
-### 8.3 Fill with invalid selector
+## Group 11: Persistence
+
+### 11.1 State after reload
```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/navigate \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/spa.html"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"kind":"fill","selector":"#nonexistent-input","text":"test"}'
+ -d '{"kind":"fill","selector":"#new-task-input","text":"Persistent Task Test"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"click","selector":"#add-task-btn"}'
+
+curl -X POST http://localhost:9867/navigate \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/spa.html"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \
+ -H "Authorization: Bearer benchmark-token"
```
-**Pass if**: Error response with message.
+**Pass if**: Contains `TASK_PERSISTENT_TEST_FOUND_AFTER_RELOAD`.
-### 8.4 Invalid action kind
+### 11.2 Logout/re-login
```bash
-curl -X POST http://localhost:9867/action \
+curl -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"kind":"teleport","selector":"#btn"}'
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/login.html"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"fill","selector":"#username","text":"benchmark"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"fill","selector":"#password","text":"test456"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"click","selector":"#login-btn"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"click","selector":"#logout-btn"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"fill","selector":"#username","text":"benchmark"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"fill","selector":"#password","text":"test456"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"click","selector":"#login-btn"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
+ -H "Authorization: Bearer benchmark-token"
```
-**Pass if**: Returns 400/422 error with clear message.
+**Pass if**: Contains `VERIFY_LOGIN_SUCCESS_DASHBOARD` AND `SESSION_RENEWED`.
---
-## Group 9: Screenshot & Export
+## Group 12: Multi-page Nav
-### 9.1 Navigate to dashboard
+### 12.1 Navigate & return
```bash
curl -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
-H "Content-Type: application/json" \
- -d '{"url":"http://fixtures/dashboard.html"}'
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/"}'
+
+curl -X POST http://localhost:9867/navigate \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki.html"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"click","selector":"#link-go","waitNav":true}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/back \
+ -H "Authorization: Bearer benchmark-token"
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/back \
+ -H "Authorization: Bearer benchmark-token"
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=500" \
+ -H "Authorization: Bearer benchmark-token"
```
-**Pass if**: HTTP 200.
+**Pass if**: Final snapshot contains `VERIFY_HOME_LOADED_12345` (returned to home).
-### 9.2 Take screenshot
+### 12.2 Cross-page compare
```bash
-curl http://localhost:9867/screenshot \
+curl -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
- --output /tmp/benchmark-screenshot.png
+ -H "Content-Type: application/json" \
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki.html"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \
+ -H "Authorization: Bearer benchmark-token"
+
+curl -X POST http://localhost:9867/navigate \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/articles.html"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1500" \
+ -H "Authorization: Bearer benchmark-token"
```
-**Pass if**: File created, size > 10KB.
+**Pass if**: Wiki snapshot contains `COUNT_LANGUAGES_12` AND articles snapshot contains article titles.
+
+---
-### 9.3 Verify screenshot has content
+## Group 13: Form Validation
+
+### 13.1 Required field
```bash
-ls -la /tmp/benchmark-screenshot.png
+curl -X POST http://localhost:9867/navigate \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/form.html"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"fill","selector":"#fullname","text":"Validator Test"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"click","selector":"#submit-btn"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
+ -H "Authorization: Bearer benchmark-token"
```
-**Pass if**: File exists and size > 10240 bytes.
+**Pass if**: Snapshot does NOT contain `VERIFY_FORM_SUBMITTED_SUCCESS` (submission blocked by validation).
-### 9.4 Export PDF
+### 13.2 Optional field
```bash
-curl -X POST http://localhost:9867/pdf \
+curl -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
- --output /tmp/benchmark-dashboard.pdf
+ -H "Content-Type: application/json" \
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/form.html"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"fill","selector":"#fullname","text":"No Phone User"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"fill","selector":"#email","text":"[email protected]"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"select","selector":"#country","value":"de"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"select","selector":"#subject","value":"feedback"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"click","selector":"#submit-btn"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
+ -H "Authorization: Bearer benchmark-token"
```
-**Pass if**: File created, size > 10KB.
+**Pass if**: Contains `VERIFY_FORM_SUBMITTED_SUCCESS` AND `OPTIONAL_FIELD_SKIPPED_SUCCESS`.
---
-## Group 10: Agent Identity
+## Group 14: Dynamic Content
-### 10.1 Navigate as agent alpha
+### 14.1 Load more
```bash
curl -X POST http://localhost:9867/navigate \
-H "Authorization: Bearer benchmark-token" \
- -H "X-Agent-Id: bench-alpha" \
-H "Content-Type: application/json" \
- -d '{"url":"http://fixtures/wiki.html"}'
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/ecommerce.html"}'
+
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"click","selector":"#load-more-btn"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
+ -H "Authorization: Bearer benchmark-token"
```
-**Pass if**: HTTP 200.
+**Pass if**: Contains `ADDITIONAL_PRODUCTS_LOADED`.
-### 10.2 Navigate as agent beta
+### 14.2 Lazy-loaded item
```bash
-curl -X POST http://localhost:9867/navigate \
+curl -X POST http://localhost:9867/tabs/TAB_ID/action \
-H "Authorization: Bearer benchmark-token" \
- -H "X-Agent-Id: bench-beta" \
-H "Content-Type: application/json" \
- -d '{"url":"http://fixtures/dashboard.html"}'
+ -d '{"kind":"click","selector":"#product-5 .add-to-cart"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
+ -H "Authorization: Bearer benchmark-token"
```
-**Pass if**: HTTP 200.
+**Pass if**: Contains `CART_UPDATED_WITH_LAZY_PRODUCT`.
-### 10.3 Check alpha activity
+---
+
+## Group 15: Data Aggregation
+
+### 15.1 Financial calc
```bash
-curl "http://localhost:9867/api/activity?agentId=bench-alpha" \
+curl -X POST http://localhost:9867/navigate \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/dashboard.html"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=2000" \
-H "Authorization: Bearer benchmark-token"
```
-**Pass if**: Response contains activity for `bench-alpha`.
+**Pass if**: Contains `$1,284,930` (revenue) AND `$384,930` (profit) AND `PROFIT_MARGIN_CALCULATED`.
-### 10.4 Check beta activity
+### 15.2 Multi-page comparison
```bash
-curl "http://localhost:9867/api/activity?agentId=bench-beta" \
+curl -X POST http://localhost:9867/navigate \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki-go.html"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
+ -H "Authorization: Bearer benchmark-token"
+
+curl -X POST http://localhost:9867/navigate \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki-python.html"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
+ -H "Authorization: Bearer benchmark-token"
+
+curl -X POST http://localhost:9867/navigate \
+ -H "Authorization: Bearer benchmark-token" \
+ -H "Content-Type: application/json" \
+ -d '{"tabId":"TAB_ID","url":"http://fixtures/wiki-rust.html"}'
+
+curl "http://localhost:9867/tabs/TAB_ID/snapshot?format=compact&maxTokens=1000" \
-H "Authorization: Bearer benchmark-token"
```
-**Pass if**: Response contains activity for `bench-beta`.
+**Pass if**: Go snapshot contains `FEATURE_COUNT_6` AND Python snapshot contains `FEATURE_COUNT_7` AND `COMPARISON_TABLE_BUILT` AND Rust snapshot contains `FEATURE_COUNT_5`.
---
## Summary
-| Group | Steps | Description |
+| Group | Tasks | Description |
|-------|-------|-------------|
-| 0 | 4 | Setup & Configuration |
-| 1 | 8 | Navigation & Content Extraction |
-| 2 | 6 | Search & Dynamic Content |
-| 3 | 12 | Complex Form Interaction |
-| 4 | 6 | SPA & Dynamic State |
-| 5 | 6 | Login & Auth Flow |
-| 6 | 8 | E-commerce Flow |
-| 7 | 6 | Wiki Article + Comment |
-| 8 | 4 | Error Handling |
-| 9 | 4 | Screenshot & Export |
-| 10 | 4 | Agent Identity |
-
-**Total: 68 steps**
+| 0 | 2 | Setup |
+| 1 | 6 | Reading & Extracting |
+| 2 | 3 | Search & Dynamic |
+| 3 | 2 | Form |
+| 4 | 3 | SPA |
+| 5 | 2 | Login |
+| 6 | 3 | E-commerce |
+| 7 | 2 | Content + Interaction |
+| 8 | 2 | Error Handling |
+| 9 | 2 | Export |
+| 10 | 2 | Modals |
+| 11 | 2 | Persistence |
+| 12 | 2 | Multi-page Nav |
+| 13 | 2 | Form Validation |
+| 14 | 2 | Dynamic Content |
+| 15 | 2 | Data Aggregation |
+
+**Total: 39 tasks**
## Verification Strings
@@ -712,3 +877,5 @@ curl "http://localhost:9867/api/activity?agentId=bench-beta" \
| Login | `VERIFY_LOGIN_PAGE_77777` |
| Go Article | `VERIFY_WIKI_GO_LANG_88888` |
| SPA | `VERIFY_SPA_PAGE_99999` |
+| Python Article | `VERIFY_WIKI_PYTHON_LANG` |
+| Rust Article | `VERIFY_WIKI_RUST_LANG` |
diff --git a/tests/benchmark/OPTIMIZATION_LOOP.md b/tests/benchmark/OPTIMIZATION_LOOP.md
index a46566cc..054401d8 100644
--- a/tests/benchmark/OPTIMIZATION_LOOP.md
+++ b/tests/benchmark/OPTIMIZATION_LOOP.md
@@ -1,50 +1,104 @@
# PinchTab Auto-Optimization Loop
-Automated improvement cycle that runs every 25 minutes.
+Automated improvement cycle for PinchTab benchmarks.
+
+## Pre-requisite (Human, once)
+
+```bash
+cd ~/dev/pinchtab/tests/benchmark
+docker compose up -d --build
+# Wait for healthy:
+curl -sf -H "Authorization: Bearer benchmark-token" http://localhost:9867/health
+```
## Loop Steps
```
┌─────────────────────────────────────────────────────────────┐
-│ 1. RUN BENCHMARKS │
-│ - Baseline (explicit curl commands) │
-│ - Agent (natural language tasks) │
+│ 1a. RUN BASELINE BENCHMARK [Sub-agent A] │
+│ - Only runs when: │
+│ • No previous baseline exists, OR │
+│ • Changes affect: server code, fixtures, or │
+│ BENCHMARK_TASKS.md │
+│ - Otherwise: reuse last baseline results │
+│ - Load BENCHMARK_TASKS.md │
+│ - Execute 68 curl commands sequentially │
+│ - Check pass/fail per documented conditions │
+│ - Record via: record-step.sh --type baseline │
+│ - Output: results/baseline_<timestamp>.json │
+├─────────────────────────────────────────────────────────────┤
+│ 1b. RUN AGENT BENCHMARK [Sub-agent B] │
+│ - Runs every iteration │
+│ - Load AGENT_TASKS.md + skills/pinchtab/SKILL.md │
+│ - Execute 39 tasks using only skill docs │
+│ - Record via: record-step.sh --type agent --tokens │
+│ - Output: results/agent_benchmark_<timestamp>.json │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
-│ 2. ANALYZE DIFFERENCES │
-│ - Compare pass rates │
-│ - Identify step failures │
-│ - Find patterns (text vs snapshot, selectors, etc.) │
+│ 2. CHECK FOR REGRESSION [Main agent] │
+│ - Read both JSON reports for pass/fail counts │
+│ - Compare against results/best_score.txt │
+│ - If worse: flag regression, prioritize fix/revert │
+│ - If equal or better: update best_score.txt │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
-│ 3. PROPOSE 1 IMPROVEMENT │
+│ 3. ANALYZE DIFFERENCES [Main agent] │
+│ - Diff baseline vs agent results side by side │
+│ - Identify which steps agent failed but baseline passed │
+│ - Find patterns (wrong endpoint, selectors, etc.) │
+└─────────────────────────────────────────────────────────────┘
+ ↓
+┌─────────────────────────────────────────────────────────────┐
+│ 4. PROPOSE 1 IMPROVEMENT [Main agent → Human]│
│ Priority order: │
│ a) Fix PinchTab CLI/REST bug if found │
│ b) Improve skill documentation if agent confused │
│ c) Add verification to existing test case │
│ d) Add new test case for uncovered scenario │
+│ Present proposal to human for approval. │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
-│ 4. IMPLEMENT & COMMIT │
-│ - Make the change │
-│ - Commit with descriptive message │
-│ - Push to feat/benchmark branch │
+│ 5. IMPLEMENT (no commit) [Main agent] │
+│ - Make the single change │
+│ - Leave changes uncommitted for manual review │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
-│ 5. LOG RUN │
-│ - Append to optimization_log.md │
-│ - Record: timestamp, pass rates, change made │
+│ 6. LOG RUN [Main agent] │
+│ - Append to results/optimization_log.md │
+│ - Record: timestamp, pass rates, regression check, │
+│ change made, next focus │
└─────────────────────────────────────────────────────────────┘
- ↓
- [Wait 25 minutes]
- ↓
- [Repeat]
```
+## Roles
+
+| Step | Who | What |
+|------|-----|------|
+| Pre-req | **Human** | Start Docker, verify health |
+| 1a Baseline | **Sub-agent A** | Execute 68 curl commands (skip if baseline exists and no server/fixture changes) |
+| 1b Agent | **Sub-agent B** | Execute 39 tasks from skill docs, record results |
+| 2 Regression | **Main agent** | Compare against best_score.txt |
+| 3 Analyze | **Main agent** | Diff baseline vs agent, find patterns |
+| 4 Propose | **Main agent → Human** | Suggest 1 fix, get approval |
+| 5 Implement | **Main agent** | Make the change, leave uncommitted |
+| 6 Log | **Main agent** | Append to optimization_log.md |
+
+## Regression Detection
+
+Each run compares against `results/best_score.txt`, which tracks the highest pass rates achieved:
+
+```
+baseline=62/68
+agent=33/39
+```
+
+- **If current run is worse**: Regression detected. The improvement priority shifts to fixing or reverting the last change before proposing anything new.
+- **If equal or better**: Update `best_score.txt` with the new high-water mark.
+
## Improvement Priority
1. **API/CLI Bug** — If a curl command returns unexpected error, investigate PinchTab code
@@ -63,6 +117,10 @@ Each run appends to `results/optimization_log.md`:
- Baseline: X/Y (Z%)
- Agent: X/Y (Z%)
+**Regression Check:**
+- Best: baseline X/Y, agent X/Y
+- This run: [better | same | REGRESSION]
+
**Analysis:**
- [What differed between runs]
- [Root cause identified]
@@ -70,7 +128,7 @@ Each run appends to `results/optimization_log.md`:
**Change Made:**
- [Type: api|skill|benchmark]
- [Description]
-- [Commit: abc1234]
+- [Status: uncommitted — pending manual review]
**Next Focus:**
- [What to look at next run]
@@ -80,4 +138,5 @@ Each run appends to `results/optimization_log.md`:
- `run-optimization.sh` — Main loop script
- `results/optimization_log.md` — Run history
-- All changes committed to `feat/benchmark` branch
+- `results/best_score.txt` — High-water mark for regression detection
+- Changes left uncommitted for manual review and commit
diff --git a/tests/benchmark/TEST_CASES.md b/tests/benchmark/TEST_CASES.md
new file mode 100644
index 00000000..31b32ad9
--- /dev/null
+++ b/tests/benchmark/TEST_CASES.md
@@ -0,0 +1,61 @@
+# PinchTab Benchmark Test Cases
+
+39 test cases covering realistic browser automation scenarios against local fixture pages.
+
+| # | Task | Description |
+|---|------|-------------|
+| **Group 0: Setup** | | |
+| 0.1 | Health check | Confirm PinchTab is running with active instance |
+| 0.2 | Fixtures reachable | Navigate to fixtures root, confirm accessible |
+| **Group 1: Reading & Extracting** | | |
+| 1.1 | Wiki categories | Extract category names + article counts from wiki index |
+| 1.2 | Click a link | From wiki, click through to Go article |
+| 1.3 | Table extraction | Read infobox — who designed Go, what year |
+| 1.4 | Count list items | Count key features on Go article (expect 6) |
+| 1.5 | Article headlines | Navigate to articles page, list all titles |
+| 1.6 | Dashboard metrics | Extract Total Users, Revenue, Conversion Rate |
+| **Group 2: Search & Dynamic** | | |
+| 2.1 | Wiki search | Use search form to find "golang" |
+| 2.2 | No results search | Search for nonexistent term, verify graceful |
+| 2.3 | AI content search | Search for "artificial intelligence" |
+| **Group 3: Form** | | |
+| 3.1 | Complete form | Fill all fields + submit (name, email, phone, country, subject, message, checkbox, radio) |
+| 3.2 | Reset/refill | After submit, verify reset button exists |
+| **Group 4: SPA** | | |
+| 4.1 | Read app state | Count tasks: 3 total, 2 active, 1 done |
+| 4.2 | Add task | Add "Automate deployment" with high priority |
+| 4.3 | Delete task | Delete first task, verify count changed |
+| **Group 5: Login** | | |
+| 5.1 | Invalid login | Try wrong credentials, verify error |
+| 5.2 | Valid login | Login with correct creds, verify dashboard |
+| **Group 6: E-commerce** | | |
+| 6.1 | Research products | List products + prices, identify out-of-stock |
+| 6.2 | Add to cart | Add 2 items, verify $449.98 total |
+| 6.3 | Checkout | Complete order, verify confirmation |
+| **Group 7: Content + Interaction** | | |
+| 7.1 | Read & comment | Read Go article, post comment with 5-star rating |
+| 7.2 | Cross-page research | Find biggest category on wiki, navigate to an article in it |
+| **Group 8: Error Handling** | | |
+| 8.1 | 404 handling | Navigate to missing page, verify no crash |
+| 8.2 | Missing element | Click nonexistent selector, verify clear error |
+| **Group 9: Export** | | |
+| 9.1 | Screenshot | Take screenshot of dashboard |
+| 9.2 | PDF export | Export dashboard as PDF |
+| **Group 10: Modals** | | |
+| 10.1 | Open modal | Click Settings on dashboard, verify modal |
+| 10.2 | Modal interaction | Change theme to Dark, save, verify applied |
+| **Group 11: Persistence** | | |
+| 11.1 | State after reload | Add task, reload SPA, verify task persists |
+| 11.2 | Logout/re-login | Sign out, log back in, verify session renewed |
+| **Group 12: Multi-page Nav** | | |
+| 12.1 | Navigate & return | Home -> wiki -> Go -> back -> back -> home |
+| 12.2 | Cross-page compare | Compare article counts between wiki and articles page |
+| **Group 13: Form Validation** | | |
+| 13.1 | Required field | Submit without email, verify blocked |
+| 13.2 | Optional field | Submit without phone, verify success |
+| **Group 14: Dynamic Content** | | |
+| 14.1 | Load more | Click "Load More Products" on ecommerce |
+| 14.2 | Lazy-loaded item | Add a lazy-loaded product to cart |
+| **Group 15: Data Aggregation** | | |
+| 15.1 | Financial calc | Extract revenue + profit, calculate margin |
+| 15.2 | Multi-page comparison | Compare features across 3 wiki language pages |
diff --git a/tests/benchmark/scripts/record-agent-step.sh b/tests/benchmark/scripts/record-agent-step.sh
deleted file mode 100755
index 34def848..00000000
--- a/tests/benchmark/scripts/record-agent-step.sh
+++ /dev/null
@@ -1,100 +0,0 @@
-#!/bin/bash
-#
-# Record a benchmark step result
-#
-# Usage:
-# ./record-step.sh <group> <step> <pass|fail|skip> <input_tokens> <output_tokens> [notes]
-#
-# Example:
-# ./record-step.sh 1 1 pass 150 45 "Navigation completed"
-# ./record-step.sh 2 3 fail 200 80 "Element not found"
-
-set -euo pipefail
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-RESULTS_DIR="${SCRIPT_DIR}/../results"
-
-# Find most recent report file
-REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_benchmark_*.json 2>/dev/null | head -1)
-
-if [[ -z "${REPORT_FILE}" ]]; then
- echo "ERROR: No benchmark report found. Run ./run-benchmark.sh first."
- exit 1
-fi
-
-if [[ $# -lt 5 ]]; then
- echo "Usage: $0 <group> <step> <pass|fail|skip> <input_tokens> <output_tokens> [notes]"
- exit 1
-fi
-
-GROUP="$1"
-STEP="$2"
-STATUS="$3"
-INPUT_TOKENS="$4"
-OUTPUT_TOKENS="$5"
-NOTES="${6:-}"
-TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ)
-TOTAL_TOKENS=$((INPUT_TOKENS + OUTPUT_TOKENS))
-
-# Calculate cost based on model (from report)
-MODEL=$(jq -r '.benchmark.model' "${REPORT_FILE}")
-PROVIDER=$(jq -r '.benchmark.provider' "${REPORT_FILE}")
-
-# Cost per 1M tokens
-case "${MODEL}" in
- *haiku*) INPUT_RATE=0.25; OUTPUT_RATE=1.25 ;;
- *sonnet*) INPUT_RATE=3.0; OUTPUT_RATE=15.0 ;;
- *opus*) INPUT_RATE=15.0; OUTPUT_RATE=75.0 ;;
- *gpt-4o-mini*) INPUT_RATE=0.15; OUTPUT_RATE=0.60 ;;
- *gpt-4o*) INPUT_RATE=2.50; OUTPUT_RATE=10.0 ;;
- *gpt-4*) INPUT_RATE=10.0; OUTPUT_RATE=30.0 ;;
- *gemini*flash*) INPUT_RATE=0.075; OUTPUT_RATE=0.30 ;;
- *gemini*pro*) INPUT_RATE=1.25; OUTPUT_RATE=5.0 ;;
- *) INPUT_RATE=1.0; OUTPUT_RATE=3.0 ;;
-esac
-
-# Calculate cost (using bc for floating point)
-COST=$(echo "scale=6; (${INPUT_TOKENS} / 1000000 * ${INPUT_RATE}) + (${OUTPUT_TOKENS} / 1000000 * ${OUTPUT_RATE})" | bc)
-
-# Create step entry
-STEP_JSON=$(cat << EOF
-{
- "group": ${GROUP},
- "step": ${STEP},
- "id": "${GROUP}.${STEP}",
- "status": "${STATUS}",
- "input_tokens": ${INPUT_TOKENS},
- "output_tokens": ${OUTPUT_TOKENS},
- "total_tokens": ${TOTAL_TOKENS},
- "cost_usd": ${COST},
- "notes": "${NOTES}",
- "timestamp": "${TIMESTAMP}"
-}
-EOF
-)
-
-# Append to report
-TMP_FILE=$(mktemp)
-jq --argjson step "${STEP_JSON}" '.steps += [$step]' "${REPORT_FILE}" > "${TMP_FILE}"
-
-# Update totals
-jq --argjson in "${INPUT_TOKENS}" --argjson out "${OUTPUT_TOKENS}" --argjson cost "${COST}" \
- --arg status "${STATUS}" \
- '.totals.input_tokens += $in |
- .totals.output_tokens += $out |
- .totals.total_tokens += ($in + $out) |
- .totals.estimated_cost_usd += $cost |
- if $status == "pass" then .totals.steps_passed += 1
- elif $status == "fail" then .totals.steps_failed += 1
- else .totals.steps_skipped += 1 end' \
- "${TMP_FILE}" > "${REPORT_FILE}"
-
-rm -f "${TMP_FILE}"
-
-# If failed, also log to error log with details
-if [[ "${STATUS}" == "fail" ]]; then
- ERROR_LOG="${RESULTS_DIR}/errors.log"
- echo "[${TIMESTAMP}] Step ${GROUP}.${STEP} FAILED: ${NOTES}" >> "${ERROR_LOG}"
-fi
-
-echo "Recorded: Step ${GROUP}.${STEP} = ${STATUS} (${TOTAL_TOKENS} tokens, \$${COST})"
diff --git a/tests/benchmark/scripts/record-step.sh b/tests/benchmark/scripts/record-step.sh
index 4702da88..ec5fddf9 100755
--- a/tests/benchmark/scripts/record-step.sh
+++ b/tests/benchmark/scripts/record-step.sh
@@ -3,98 +3,149 @@
# Record a benchmark step result
#
# Usage:
-# ./record-step.sh <group> <step> <pass|fail|skip> <input_tokens> <output_tokens> [notes]
+# ./record-step.sh [--type baseline|agent] <group> <step> <pass|fail|skip> [options] "notes"
#
-# Example:
-# ./record-step.sh 1 1 pass 150 45 "Navigation completed"
-# ./record-step.sh 2 3 fail 200 80 "Element not found"
+# Options:
+# --type baseline|agent Report type (default: auto-detect most recent)
+# --tokens <in> <out> Token usage (agent runs only, default: 0 0)
+# --bytes <n> HTTP response size in bytes (baseline runs)
+#
+# Examples:
+# ./record-step.sh 1 1 pass "Navigation completed"
+# ./record-step.sh --type agent 2 3 fail --tokens 200 80 "Element not found"
+# ./record-step.sh --type baseline 1 2 pass --bytes 4520 "Snapshot returned"
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
RESULTS_DIR="${SCRIPT_DIR}/../results"
+mkdir -p "${RESULTS_DIR}"
-# Find most recent report file
-REPORT_FILE=$(ls -t "${RESULTS_DIR}"/baseline_*.json "${RESULTS_DIR}"/benchmark_*.json 2>/dev/null | head -1)
+# Parse flags
+REPORT_TYPE=""
+INPUT_TOKENS=0
+OUTPUT_TOKENS=0
+RESPONSE_BYTES=0
-if [[ -z "${REPORT_FILE}" ]]; then
- echo "ERROR: No benchmark report found. Run ./run-benchmark.sh first."
- exit 1
-fi
+while [[ $# -gt 0 && "$1" == --* ]]; do
+ case "$1" in
+ --type)
+ REPORT_TYPE="$2"
+ shift 2
+ ;;
+ --tokens)
+ INPUT_TOKENS="$2"
+ OUTPUT_TOKENS="$3"
+ shift 3
+ ;;
+ --bytes)
+ RESPONSE_BYTES="$2"
+ shift 2
+ ;;
+ *)
+ echo "Unknown option: $1"
+ exit 1
+ ;;
+ esac
+done
-if [[ $# -lt 5 ]]; then
- echo "Usage: $0 <group> <step> <pass|fail|skip> <input_tokens> <output_tokens> [notes]"
+if [[ $# -lt 3 ]]; then
+ echo "Usage: $0 [--type baseline|agent] <group> <step> <pass|fail|skip> [--tokens <in> <out>] [--bytes <n>] [notes]"
exit 1
fi
GROUP="$1"
STEP="$2"
STATUS="$3"
-INPUT_TOKENS="$4"
-OUTPUT_TOKENS="$5"
-NOTES="${6:-}"
+NOTES="${4:-}"
TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ)
TOTAL_TOKENS=$((INPUT_TOKENS + OUTPUT_TOKENS))
-# Calculate cost based on model (from report)
-MODEL=$(jq -r '.benchmark.model' "${REPORT_FILE}")
-PROVIDER=$(jq -r '.benchmark.provider' "${REPORT_FILE}")
-
-# Cost per 1M tokens
-case "${MODEL}" in
- *haiku*) INPUT_RATE=0.25; OUTPUT_RATE=1.25 ;;
- *sonnet*) INPUT_RATE=3.0; OUTPUT_RATE=15.0 ;;
- *opus*) INPUT_RATE=15.0; OUTPUT_RATE=75.0 ;;
- *gpt-4o-mini*) INPUT_RATE=0.15; OUTPUT_RATE=0.60 ;;
- *gpt-4o*) INPUT_RATE=2.50; OUTPUT_RATE=10.0 ;;
- *gpt-4*) INPUT_RATE=10.0; OUTPUT_RATE=30.0 ;;
- *gemini*flash*) INPUT_RATE=0.075; OUTPUT_RATE=0.30 ;;
- *gemini*pro*) INPUT_RATE=1.25; OUTPUT_RATE=5.0 ;;
- *) INPUT_RATE=1.0; OUTPUT_RATE=3.0 ;;
-esac
-
-# Calculate cost (using bc for floating point)
-COST=$(echo "scale=6; (${INPUT_TOKENS} / 1000000 * ${INPUT_RATE}) + (${OUTPUT_TOKENS} / 1000000 * ${OUTPUT_RATE})" | bc)
+# Find report file
+if [[ -n "${REPORT_TYPE}" ]]; then
+ case "${REPORT_TYPE}" in
+ baseline)
+ REPORT_FILE=$(ls -t "${RESULTS_DIR}"/baseline_*.json 2>/dev/null | head -1)
+ ;;
+ agent)
+ REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_benchmark_*.json 2>/dev/null | head -1)
+ ;;
+ *)
+ echo "ERROR: --type must be 'baseline' or 'agent'"
+ exit 1
+ ;;
+ esac
+else
+ # Auto-detect: find most recent report of any type
+ REPORT_FILE=$(ls -t "${RESULTS_DIR}"/baseline_*.json "${RESULTS_DIR}"/agent_benchmark_*.json 2>/dev/null | head -1)
+fi
+
+if [[ -z "${REPORT_FILE}" ]]; then
+ echo "ERROR: No benchmark report found. Run ./run-optimization.sh first."
+ exit 1
+fi
+
+# Calculate cost (only meaningful for agent runs with token data)
+COST=0
+if [[ ${TOTAL_TOKENS} -gt 0 ]]; then
+ MODEL=$(jq -r '.benchmark.model' "${REPORT_FILE}")
+
+ # Cost per 1M tokens
+ case "${MODEL}" in
+ *haiku*) INPUT_RATE=0.25; OUTPUT_RATE=1.25 ;;
+ *sonnet*) INPUT_RATE=3.0; OUTPUT_RATE=15.0 ;;
+ *opus*) INPUT_RATE=15.0; OUTPUT_RATE=75.0 ;;
+ *gpt-4o-mini*) INPUT_RATE=0.15; OUTPUT_RATE=0.60 ;;
+ *gpt-4o*) INPUT_RATE=2.50; OUTPUT_RATE=10.0 ;;
+ *gpt-4*) INPUT_RATE=10.0; OUTPUT_RATE=30.0 ;;
+ *gemini*flash*) INPUT_RATE=0.075; OUTPUT_RATE=0.30 ;;
+ *gemini*pro*) INPUT_RATE=1.25; OUTPUT_RATE=5.0 ;;
+ *) INPUT_RATE=1.0; OUTPUT_RATE=3.0 ;;
+ esac
+
+ COST=$(echo "scale=6; (${INPUT_TOKENS} / 1000000 * ${INPUT_RATE}) + (${OUTPUT_TOKENS} / 1000000 * ${OUTPUT_RATE})" | bc)
+fi
# Create step entry
-STEP_JSON=$(cat << EOF
-{
- "group": ${GROUP},
- "step": ${STEP},
- "id": "${GROUP}.${STEP}",
- "status": "${STATUS}",
- "input_tokens": ${INPUT_TOKENS},
- "output_tokens": ${OUTPUT_TOKENS},
- "total_tokens": ${TOTAL_TOKENS},
- "cost_usd": ${COST},
- "notes": "${NOTES}",
- "timestamp": "${TIMESTAMP}"
-}
-EOF
-)
-
-# Append to report
-TMP_FILE=$(mktemp)
-jq --argjson step "${STEP_JSON}" '.steps += [$step]' "${REPORT_FILE}" > "${TMP_FILE}"
+STEP_JSON=$(jq -n \
+ --argjson group "${GROUP}" \
+ --argjson step "${STEP}" \
+ --arg id "${GROUP}.${STEP}" \
+ --arg status "${STATUS}" \
+ --argjson in_tokens "${INPUT_TOKENS}" \
+ --argjson out_tokens "${OUTPUT_TOKENS}" \
+ --argjson total_tokens "${TOTAL_TOKENS}" \
+ --argjson cost "${COST}" \
+ --argjson bytes "${RESPONSE_BYTES}" \
+ --arg notes "${NOTES}" \
+ --arg ts "${TIMESTAMP}" \
+ '{group: $group, step: $step, id: $id, status: $status,
+ input_tokens: $in_tokens, output_tokens: $out_tokens,
+ total_tokens: $total_tokens, cost_usd: $cost,
+ response_bytes: $bytes, notes: $notes, timestamp: $ts}')
-# Update totals
-jq --argjson in "${INPUT_TOKENS}" --argjson out "${OUTPUT_TOKENS}" --argjson cost "${COST}" \
+# Append to report and update totals
+TMP_FILE=$(mktemp)
+jq --argjson step "${STEP_JSON}" \
+ --argjson in "${INPUT_TOKENS}" \
+ --argjson out "${OUTPUT_TOKENS}" \
+ --argjson cost "${COST}" \
--arg status "${STATUS}" \
- '.totals.input_tokens += $in |
+ '.steps += [$step] |
+ .totals.input_tokens += $in |
.totals.output_tokens += $out |
.totals.total_tokens += ($in + $out) |
.totals.estimated_cost_usd += $cost |
if $status == "pass" then .totals.steps_passed += 1
elif $status == "fail" then .totals.steps_failed += 1
else .totals.steps_skipped += 1 end' \
- "${TMP_FILE}" > "${REPORT_FILE}"
+ "${REPORT_FILE}" > "${TMP_FILE}"
-rm -f "${TMP_FILE}"
+mv "${TMP_FILE}" "${REPORT_FILE}"
-# If failed, also log to error log with details
+# Log failures
if [[ "${STATUS}" == "fail" ]]; then
- ERROR_LOG="${RESULTS_DIR}/errors.log"
- echo "[${TIMESTAMP}] Step ${GROUP}.${STEP} FAILED: ${NOTES}" >> "${ERROR_LOG}"
+ echo "[${TIMESTAMP}] Step ${GROUP}.${STEP} FAILED: ${NOTES}" >> "${RESULTS_DIR}/errors.log"
fi
-echo "Recorded: Step ${GROUP}.${STEP} = ${STATUS} (${TOTAL_TOKENS} tokens, \$${COST})"
+echo "Recorded: Step ${GROUP}.${STEP} = ${STATUS} (tokens: ${TOTAL_TOKENS}, bytes: ${RESPONSE_BYTES}, \$${COST})"
diff --git a/tests/benchmark/scripts/run-optimization.sh b/tests/benchmark/scripts/run-optimization.sh
index f96e0dcd..caf512d3 100755
--- a/tests/benchmark/scripts/run-optimization.sh
+++ b/tests/benchmark/scripts/run-optimization.sh
@@ -7,7 +7,8 @@
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-RESULTS_DIR="${SCRIPT_DIR}/results"
+RESULTS_DIR="${SCRIPT_DIR}/../results"
+mkdir -p "${RESULTS_DIR}"
LOG_FILE="${RESULTS_DIR}/optimization_log.md"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
RUN_NUMBER=$(grep -c "^## Run #" "${LOG_FILE}" 2>/dev/null || echo 0)
@@ -16,14 +17,8 @@ RUN_NUMBER=$((RUN_NUMBER + 1))
echo "=== PinchTab Optimization Run #${RUN_NUMBER} ==="
echo "Timestamp: ${TIMESTAMP}"
-# Ensure we're on feat/benchmark branch
+# Run on current branch
cd ~/dev/pinchtab
-CURRENT_BRANCH=$(git branch --show-current)
-if [[ "${CURRENT_BRANCH}" != "feat/benchmark" ]]; then
- echo "Switching to feat/benchmark branch..."
- git checkout feat/benchmark
-fi
-git pull --rebase origin feat/benchmark 2>/dev/null || true
# Ensure Docker is running
cd "${SCRIPT_DIR}"