High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.
chore: benchmark improvements
Luigi Agosti90c446e0c6ee
diff --git a/skills/agent-browser/SKILL.md b/skills/agent-browser/SKILL.mddeleted file mode 100644index e58a9ef5..00000000--- a/skills/agent-browser/SKILL.md+++ /dev/null@@ -1,132 +0,0 @@-----name: agent-browser-description: "Use this skill when a benchmark or local automation task should drive the browser through the agent-browser CLI instead of PinchTab. In this repo, prefer the benchmark wrapper `./tests/benchmark/scripts/ab` so commands execute inside the shared Docker benchmark environment and tool calls are logged."------# Browser Automation with Agent Browser--For benchmark runs in this repo, do not call `agent-browser` directly. Use the-wrapper:--```bash-./tests/benchmark/scripts/ab ...-```--That wrapper:--- executes `agent-browser` inside the benchmark Docker service-- preserves the shared browser session across commands-- logs every tool call into `tests/benchmark/results/agent_browser_commands.ndjson`--Use this skill when the benchmark should run the same fixture tasks as the-PinchTab lane, but with `agent-browser` as the browser-control surface.--## Benchmark Environment--- Shared fixture site: `http://fixtures/`-- Wrapper command: `./tests/benchmark/scripts/ab`-- Default session name: `benchmark`-- Report recorder: `./tests/benchmark/scripts/record-step.sh --type agent-browser ...`--Before running tasks, initialize the benchmark lane:--```bash-cd tests/benchmark-./scripts/run-agent-browser-benchmark.sh-```--## Core Workflow--Follow this pattern:--1. Open or navigate the target page with `./tests/benchmark/scripts/ab open <url>`-2. Inspect the page with `./tests/benchmark/scripts/ab snapshot -i -c`-3. Use fresh refs such as `@e2`, `@e7`, `@e11` for actions-4. Re-snapshot after any action that changes the DOM-5. Record the finished benchmark step with `record-step.sh`--Rules:--- Prefer `snapshot -i -c` to find actionable elements-- Prefer `@eN` refs from the latest snapshot over brittle selectors-- Re-snapshot after navigation, form submit, modal open, tab change, accordion expand, or SPA update-- Use the same session for the whole benchmark run unless a task explicitly needs a reset--## Essential Commands--### Navigation and observation--```bash-./tests/benchmark/scripts/ab open http://fixtures/-./tests/benchmark/scripts/ab back-./tests/benchmark/scripts/ab forward-./tests/benchmark/scripts/ab reload-./tests/benchmark/scripts/ab snapshot-./tests/benchmark/scripts/ab snapshot -i -c-./tests/benchmark/scripts/ab snapshot -i --urls-./tests/benchmark/scripts/ab get text-./tests/benchmark/scripts/ab get title-./tests/benchmark/scripts/ab get url-```--### Interaction--```bash-./tests/benchmark/scripts/ab click @e2-./tests/benchmark/scripts/ab fill @e4 "[email protected]"-./tests/benchmark/scripts/ab type @e5 "hello"-./tests/benchmark/scripts/ab press Enter-./tests/benchmark/scripts/ab hover @e7-./tests/benchmark/scripts/ab check @e8-./tests/benchmark/scripts/ab uncheck @e8-./tests/benchmark/scripts/ab select @e9 "uk"-./tests/benchmark/scripts/ab drag @e10 @e11-./tests/benchmark/scripts/ab scroll down 800-./tests/benchmark/scripts/ab scrollintoview @e12-./tests/benchmark/scripts/ab wait @e13-./tests/benchmark/scripts/ab wait 1000-```--### Exports and debugging--```bash-./tests/benchmark/scripts/ab screenshot /tmp/page.png-./tests/benchmark/scripts/ab pdf /tmp/page.pdf-./tests/benchmark/scripts/ab console-./tests/benchmark/scripts/ab errors-./tests/benchmark/scripts/ab eval "document.title"-```--## Selector Guidance--`agent-browser` accepts:--- refs from snapshot output, like `@e2`-- CSS selectors, like `#submit` or `.btn-primary`-- XPath selectors, like `//button[@type="submit"]`--Prefer refs first. They are the most stable option inside the benchmark.--## Benchmark Reporting--After completing a benchmark step, record it:--```bash-./tests/benchmark/scripts/record-step.sh \- --type agent-browser \- --tokens 120 48 \- 1 1 pass \- "opened page and extracted content"-```--`record-step.sh` automatically derives `tool_calls` for the step from the-wrapper log unless you override it with `--tool-calls <n>`.--## Task Entry Point--Use this skill together with:--- `tests/benchmark/AGENT_BROWSER_TASKS.md`--That file contains the benchmark tasks; this skill tells you how to operate the-browser tool surface correctly.diff --git a/tests/benchmark/AGENT_BROWSER_INSTRUCTIONS.md b/tests/benchmark/AGENT_BROWSER_INSTRUCTIONS.mdnew file mode 100644index 00000000..d35e46af--- /dev/null+++ b/tests/benchmark/AGENT_BROWSER_INSTRUCTIONS.md@@ -0,0 +1,75 @@+# Agent Browser Instructions++These are benchmark-specific instructions for running the `agent-browser` lane.+They are not the official `agent-browser` skill. For the live CLI skill, load it+through the benchmark wrapper:++```bash+./scripts/ab skills get agent-browser --full+```++## Wrapper++For benchmark runs in this repo, do not call `agent-browser` directly. Use:++```bash+./scripts/ab ...+```++That wrapper:++- executes `agent-browser` inside the benchmark Docker service+- preserves the shared browser session across commands+- logs tool calls to `results/agent_browser_commands.ndjson`++## Benchmark Workflow++1. Start the lane:++```bash+cd tests/benchmark+./scripts/run-agent-browser-benchmark.sh+```++2. Load the live CLI skill:++```bash+./scripts/ab skills get agent-browser --full+```++3. Run browser actions through the wrapper:++```bash+./scripts/ab open http://fixtures/+./scripts/ab snapshot -i -c+./scripts/ab click @e2+./scripts/ab fill @e3 "[email protected]"+```++4. Record execution results:++```bash+./scripts/record-step.sh --type agent-browser 1 1 answer \+ "opened fixtures home and got refs e1-e13" "completed"+```++5. Verify later in a separate pass:++```bash+./scripts/verify-step.sh --type agent-browser 1 1 pass \+ "matched expected homepage state"+```++## Environment++- Fixtures: `http://fixtures/`+- Session name: `benchmark` by default (`AGENT_BROWSER_SESSION` overrides)+- Browser driver: Docker service `agent-browser`++## Operating Guidance++- Treat `./scripts/ab skills get agent-browser --full` as the source of truth for+ current command syntax and workflows+- Prefer refs from `snapshot -i -c` over brittle selectors+- Re-snapshot after navigation or any DOM-changing action+- Keep one session for the whole benchmark lane unless a task explicitly needs a resetdiff --git a/tests/benchmark/AGENT_BROWSER_TASKS.md b/tests/benchmark/AGENT_BROWSER_TASKS.mdindex 00e6ae39..cdd891c3 100644--- a/tests/benchmark/AGENT_BROWSER_TASKS.md+++ b/tests/benchmark/AGENT_BROWSER_TASKS.md@@ -6,17 +6,23 @@ PinchTab CLI.## Setup-1. Read `../../skills/agent-browser/SKILL.md` first. This is your guide to the- benchmark wrapper and command workflow.+1. Read `./AGENT_BROWSER_INSTRUCTIONS.md` first. This is your benchmark guide to+ the wrapper and command workflow.-2. Start the Docker benchmark environment and initialize a report:+2. Load the live CLI skill content before using browser commands:++```bash+./scripts/ab skills get agent-browser --full+```++3. Start the Docker benchmark environment and initialize a report:```bashcd tests/benchmark./scripts/run-agent-browser-benchmark.sh```-3. Use the Docker-backed wrapper for every browser action:+4. Use the Docker-backed wrapper for every browser action:```bash./scripts/ab open http://fixtures/@@ -25,16 +31,23 @@ cd tests/benchmark./scripts/ab fill @e3 "[email protected]"```-4. Record each completed step:+5. Record each completed step as an answer/result first:```bash-./scripts/record-step.sh --type agent-browser 1 1 pass --tokens 120 48 "completed"+./scripts/record-step.sh --type agent-browser 1 1 answer \+ "opened fixtures home and got refs e1-e13" "completed"````record-step.sh` will automatically calculate the number of `agent-browser`tool calls used since the previous recorded step by reading`results/agent_browser_commands.ndjson`.+6. After the execution lane is done, run a separate verification pass with:++```bash+./scripts/verify-step.sh --type agent-browser 1 1 pass "matched expected homepage state"+```+## Environment- Fixtures: `http://fixtures/`@@ -43,7 +56,7 @@ tool calls used since the previous recorded step by reading## Tooling Guidance-Use `../../skills/agent-browser/SKILL.md` as the primary operating guide.+Use `./AGENT_BROWSER_INSTRUCTIONS.md` as the primary benchmark operating guide.Reach for `./scripts/ab --help` only when the skill does not already answer thequestion.diff --git a/tests/benchmark/AGENT_TASKS.md b/tests/benchmark/AGENT_TASKS.mdindex 31d00032..18a16e28 100644--- a/tests/benchmark/AGENT_TASKS.md+++ b/tests/benchmark/AGENT_TASKS.md@@ -7,7 +7,11 @@ Natural language tasks to test how well an agent uses PinchTab from skill docs a1. Read `../../skills/pinchtab/SKILL.md` — this is your only guide2. For each task, figure out which commands to use3. **Log every command executed**-4. Record: `./scripts/record-step.sh --type agent <group> <step> <pass|fail> --tokens <in> <out> "notes"`+4. Do not score your own work as pass/fail unless a task is trivially objective.+5. Prefer recording each step as `answer` with the actual response/result:+ `./scripts/record-step.sh --type agent <group> <step> answer "<what you saw>" "notes"`+6. A separate verification pass should later stamp the step with:+ `./scripts/verify-step.sh --type agent <group> <step> <pass|fail|skip> "verification notes"`### Recommended setup: env vars + `./scripts/pt` wrapper@@ -40,8 +44,8 @@ export PINCHTAB_TAB=$(./scripts/pt nav http://fixtures/)./scripts/pt click '#submit'./scripts/pt drag '#piece' --drag-x 12 --drag-y -158-# Record step results:-./scripts/record-step.sh --type agent 1 2 pass "worked"+# Record answer/result for later verification:+./scripts/record-step.sh --type agent 1 2 answer "clicked submit and saw success banner" "worked"```The wrapper is a ~60-line shell script; read `scripts/pt` for the exact@@ -50,6 +54,10 @@ anything that works with `pinchtab ...` works with `./scripts/pt ...`.Explicit `--tab <id>` on any command still wins over `PINCHTAB_TAB`.+For deferred verification, keep the answer payload factual. Record the+actual page text, count, answer, or state transition you observed, not your own+judgment about whether that matches the benchmark oracle.+## Environment- PinchTab: `http://localhost:9867`, token: `benchmark-token`diff --git a/tests/benchmark/README.md b/tests/benchmark/README.mdindex 943ce790..b653f212 100644--- a/tests/benchmark/README.md+++ b/tests/benchmark/README.md@@ -13,7 +13,7 @@ cd tests/benchmark# agent-browser lane./scripts/run-agent-browser-benchmark.sh-# Then run tasks from AGENT_BROWSER_TASKS.md with ./scripts/ab+# Then read AGENT_BROWSER_INSTRUCTIONS.md and run tasks from AGENT_BROWSER_TASKS.md with ./scripts/ab./scripts/finalize-report.sh```@@ -34,13 +34,13 @@ If Docker build fails or is skipped, the benchmark is **INVALID**.| File | Purpose ||------|---------|| `../../skills/pinchtab/SKILL.md` | PinchTab skill (same as shipped product) |-| `../../skills/agent-browser/SKILL.md` | `agent-browser` skill for the benchmark wrapper |+| `AGENT_BROWSER_INSTRUCTIONS.md` | Benchmark-local instructions for the `agent-browser` lane || `BASELINE_TASKS.md` | Standalone task list (same as skill) || `AGENT_BROWSER_TASKS.md` | Equivalent task lane for `agent-browser` || `scripts/run-optimization.sh` | Initialize PinchTab benchmark reports || `scripts/run-agent-browser-benchmark.sh` | Start fixtures + `agent-browser` and initialize a fresh report || `scripts/ab` | Docker-backed `agent-browser` wrapper with tool-call logging |-| `scripts/record-step.sh` | Record step results with token counts and tool-call counts |+| `scripts/record-step.sh` | Record step results and tool-call counts || `scripts/finalize-report.sh` | Generate final summary report || `config/pinchtab.json` | PinchTab configuration || `agent-browser/Dockerfile` | `agent-browser` benchmark image |@@ -53,15 +53,20 @@ The benchmark is designed to run in a fresh agent context:1. Initialize the relevant benchmark lane2. Execute the natural-language tasks with the target browser surface-3. Record each step's input/output tokens-4. Let the harness count browser/tool calls where possible+3. Record each step's raw answer/result+4. Verify observed steps in a separate pass+5. Let the harness count browser/tool calls where possibleThis measures the **real cost** of using a browser tool with an agent, including:- Context loading overhead-- Per-task token usage- Browser/tool-call count- Total benchmark cost+For agent lanes, the recommended flow is two-phase:++1. execution agent records each step as `answer`+2. verifier agent reads the report and stamps each step with `verify-step.sh`+## EnvironmentThe benchmark runs PinchTab in Docker with:@@ -72,24 +77,29 @@ The benchmark runs PinchTab in Docker with:- **Headless**: Yes- **Multi-instance**: Enabled (2 instances)-## Token Tracking+## Step Recording-Every step must track token usage:+Every step must record the answer/result:```bash-./scripts/record-step.sh <group> <step> <pass|fail|skip> <input_tokens> <output_tokens> "notes"+./scripts/record-step.sh <group> <step> <pass|fail|skip|answer> "notes"```Example:```bash-./scripts/record-step.sh 1 1 pass 150 45 "Navigation completed in 1.2s"-./scripts/record-step.sh 2 3 fail 200 80 "Element not found"+./scripts/record-step.sh 1 1 pass "Navigation completed in 1.2s"+./scripts/record-step.sh 2 3 fail "Element not found"```-Token counts should come from your model's API response:-- **Anthropic**: `usage.input_tokens`, `usage.output_tokens`-- **OpenAI**: `usage.prompt_tokens`, `usage.completion_tokens`-- **Gemini**: `usageMetadata.promptTokenCount`, `usageMetadata.candidatesTokenCount`+Deferred-verification example:++```bash+./scripts/record-step.sh --type agent 1 1 answer \+ "Found categories Programming Languages: 12, Databases: 8" \+ "raw answer"+./scripts/verify-step.sh --type agent 1 1 pass \+ "Answer satisfies the benchmark oracle"+```## Reports@@ -111,11 +121,10 @@ Reports are generated in `results/`:| Steps Failed | 2 || Pass Rate | 93.7% |-## Token Usage+## Tooling| Metric | Value ||--------|-------|-| Total Tokens | 4,523 |-| Estimated Cost | $0.0158 |+| Tool Calls | 128 |```## Running Programmaticallydiff --git a/tests/benchmark/scripts/finalize-report.sh b/tests/benchmark/scripts/finalize-report.shindex 0ba93690..c126f8ed 100755--- a/tests/benchmark/scripts/finalize-report.sh+++ b/tests/benchmark/scripts/finalize-report.sh@@ -7,19 +7,36 @@ set -euo pipefailSCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"BENCH_DIR="${SCRIPT_DIR}/.."RESULTS_DIR="${BENCH_DIR}/results"+CURRENT_BASELINE_PTR="${RESULTS_DIR}/current_baseline_report.txt"+CURRENT_AGENT_PTR="${RESULTS_DIR}/current_agent_report.txt"+CURRENT_AGENT_BROWSER_PTR="${RESULTS_DIR}/current_agent_browser_report.txt"++resolve_current_report() {+ local ptr="$1"+ if [[ -f "${ptr}" ]]; then+ tr -d '[:space:]' < "${ptr}"+ return 0+ fi+ return 1+}REPORT_FILE="${1:-}"if [[ -z "${REPORT_FILE}" ]]; then- shopt -s nullglob- candidates=(- "${RESULTS_DIR}"/agent_browser_benchmark_*.json- "${RESULTS_DIR}"/agent_benchmark_*.json- "${RESULTS_DIR}"/baseline_*.json- )- shopt -u nullglob-- if [[ ${#candidates[@]} -gt 0 ]]; then- REPORT_FILE=$(ls -t "${candidates[@]}" | head -1)+ REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_BROWSER_PTR}" || true)"+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_PTR}" || true)"+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE="$(resolve_current_report "${CURRENT_BASELINE_PTR}" || true)"+ if [[ -z "${REPORT_FILE}" ]]; then+ shopt -s nullglob+ candidates=(+ "${RESULTS_DIR}"/agent_browser_benchmark_*.json+ "${RESULTS_DIR}"/agent_benchmark_*.json+ "${RESULTS_DIR}"/baseline_*.json+ )+ shopt -u nullglob++ if [[ ${#candidates[@]} -gt 0 ]]; then+ REPORT_FILE=$(ls -t "${candidates[@]}" | head -1)+ fififi@@ -34,28 +51,66 @@ jq -r 'def pct($a; $b):if $b == 0 then "0.0%" else (((1000 * $a) / $b | round) / 10 | tostring) + "%" end;. as $root |- ($root.totals.steps_passed + $root.totals.steps_failed + $root.totals.steps_skipped) as $step_count |- "# Benchmark Summary",+ ($root.totals.steps_passed + $root.totals.steps_failed + $root.totals.steps_skipped) as $baseline_total |+ (($root.totals.steps_answered // 0) + $root.totals.steps_failed + $root.totals.steps_skipped) as $agent_total |+ ($root.totals.steps_verified_passed // 0) as $verified_passed |+ ($root.totals.steps_verified_failed // 0) as $verified_failed |+ ($root.totals.steps_verified_skipped // 0) as $verified_skipped |+ ($root.totals.steps_pending_verification // 0) as $pending |+ (if $root.benchmark.type == "baseline" then+ [+ "# Benchmark Summary",+ "",+ "| Metric | Value |",+ "|--------|-------|",+ "| Type | \($root.benchmark.type) |",+ "| Model | \($root.benchmark.model) |",+ "| Steps Passed | \($root.totals.steps_passed) |",+ "| Steps Failed | \($root.totals.steps_failed) |",+ "| Steps Skipped | \($root.totals.steps_skipped) |",+ "| Pass Rate | \(pct($root.totals.steps_passed; $baseline_total)) |",+ "| Tool Calls | \($root.totals.tool_calls // 0) |"+ ]+ else+ [+ "# Benchmark Summary",+ "",+ "| Metric | Value |",+ "|--------|-------|",+ "| Type | \($root.benchmark.type) |",+ "| Model | \($root.benchmark.model) |",+ "| Steps Answered | \($root.totals.steps_answered // 0) |",+ "| Steps Failed | \($root.totals.steps_failed) |",+ "| Steps Skipped | \($root.totals.steps_skipped) |",+ "| Answer Rate | \(pct($root.totals.steps_answered // 0; $agent_total)) |",+ "| Verified Passed | \($verified_passed) |",+ "| Verified Failed | \($verified_failed) |",+ "| Verified Skipped | \($verified_skipped) |",+ "| Pending Verification | \($pending) |",+ "| Verification Pass Rate | \(pct($verified_passed; $agent_total)) |",+ "| Tool Calls | \($root.totals.tool_calls // 0) |"+ ]+ end)[],"",- "| Metric | Value |",- "|--------|-------|",- "| Type | \($root.benchmark.type) |",- "| Model | \($root.benchmark.model) |",- "| Steps Passed | \($root.totals.steps_passed) |",- "| Steps Failed | \($root.totals.steps_failed) |",- "| Steps Skipped | \($root.totals.steps_skipped) |",- "| Pass Rate | \(pct($root.totals.steps_passed; $step_count)) |",- "| Input Tokens | \($root.totals.input_tokens) |",- "| Output Tokens | \($root.totals.output_tokens) |",- "| Total Tokens | \($root.totals.total_tokens) |",- "| Tool Calls | \($root.totals.tool_calls // 0) |",- "| Estimated Cost (USD) | \($root.totals.estimated_cost_usd) |",+ "## Pending Verification",+ "",+ (+ [ $root.steps[] | select(.status == "answer" and ((.verification.status // "pending") == "pending")) | "- \(.id): \(.answer // .notes // "")" ] |+ if length == 0 then ["- none"] else . end+ )[],"","## Failed Steps","",([ $root.steps[] | select(.status == "fail") | "- \(.id): \(.notes)" ] |if length == 0 then ["- none"] else . end+ )[],+ "",+ "## Verification Failures",+ "",+ (+ [ $root.steps[] | select(.status == "answer" and ((.verification.status // "") == "fail")) | "- \(.id): \(.verification.notes // "")" ] |+ if length == 0 then ["- none"] else . end)[]' "${REPORT_FILE}" > "${SUMMARY_FILE}"diff --git a/tests/benchmark/scripts/record-step.sh b/tests/benchmark/scripts/record-step.shindex d8cf70d8..7f44ff31 100755--- a/tests/benchmark/scripts/record-step.sh+++ b/tests/benchmark/scripts/record-step.sh@@ -3,17 +3,20 @@# Record a benchmark step result## Usage:-# ./record-step.sh [--type baseline|agent|agent-browser] <group> <step> <pass|fail|skip> [options] "notes"+# ./record-step.sh [--type baseline|agent|agent-browser] <group> <step> <pass|fail|skip|answer> [answer] [notes]## Options:# --type baseline|agent|agent-browser Report type (default: auto-detect most recent)-# --tokens <in> <out> Token usage (agent runs only, default: 0 0)+# --tokens <in> <out> Deprecated and ignored (kept for compatibility)# --bytes <n> HTTP response size in bytes (baseline runs)# --tool-calls <n> Override auto-counted tool calls+# --observed <text> Deprecated alias for answer text+# --answer <text> Optional explicit answer text## Examples:# ./record-step.sh 1 1 pass "Navigation completed"-# ./record-step.sh --type agent 2 3 fail --tokens 200 80 "Element not found"+# ./record-step.sh --type agent 2 3 fail "Element not found"+# ./record-step.sh --type agent 2 4 answer "Robert Griesemer, 2009" "read infobox"# ./record-step.sh --type baseline 1 2 pass --bytes 4520 "Snapshot returned"set -euo pipefail@@ -24,11 +27,16 @@ mkdir -p "${RESULTS_DIR}"# Parse flagsREPORT_TYPE=""+REPORT_FILE=""INPUT_TOKENS=0OUTPUT_TOKENS=0RESPONSE_BYTES=0TOOL_CALLS=""+ANSWER=""AGENT_BROWSER_LOG="${RESULTS_DIR}/agent_browser_commands.ndjson"+CURRENT_BASELINE_PTR="${RESULTS_DIR}/current_baseline_report.txt"+CURRENT_AGENT_PTR="${RESULTS_DIR}/current_agent_report.txt"+CURRENT_AGENT_BROWSER_PTR="${RESULTS_DIR}/current_agent_browser_report.txt"while [[ $# -gt 0 && "$1" == --* ]]; docase "$1" in@@ -36,6 +44,10 @@ while [[ $# -gt 0 && "$1" == --* ]]; doREPORT_TYPE="$2"shift 2;;+ --report-file)+ REPORT_FILE="$2"+ shift 2+ ;;--tokens)INPUT_TOKENS="$2"OUTPUT_TOKENS="$3"@@ -49,6 +61,14 @@ while [[ $# -gt 0 && "$1" == --* ]]; doTOOL_CALLS="$2"shift 2;;+ --observed)+ ANSWER="$2"+ shift 2+ ;;+ --answer)+ ANSWER="$2"+ shift 2+ ;;*)echo "Unknown option: $1"exit 1@@ -57,64 +77,110 @@ while [[ $# -gt 0 && "$1" == --* ]]; dodoneif [[ $# -lt 3 ]]; then- echo "Usage: $0 [--type baseline|agent|agent-browser] <group> <step> <pass|fail|skip> [--tokens <in> <out>] [--bytes <n>] [--tool-calls <n>] [notes]"+ echo "Usage: $0 [--type baseline|agent|agent-browser] [--report-file <path>] <group> <step> <pass|fail|skip|answer> [answer] [notes]"exit 1fiGROUP="$1"STEP="$2"STATUS="$3"-NOTES="${4:-}"TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ)TOTAL_TOKENS=$((INPUT_TOKENS + OUTPUT_TOKENS))-# Find report file-if [[ -n "${REPORT_TYPE}" ]]; then- case "${REPORT_TYPE}" in- baseline)- REPORT_FILE=$(ls -t "${RESULTS_DIR}"/baseline_*.json 2>/dev/null | head -1)- ;;- agent)- REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_benchmark_*.json 2>/dev/null | head -1)- ;;- agent-browser|agent_browser)- REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_browser_benchmark_*.json 2>/dev/null | head -1)- ;;- *)- echo "ERROR: --type must be 'baseline', 'agent', or 'agent-browser'"- exit 1- ;;- esac+case "${STATUS}" in+ pass|fail|skip|answer|observed) ;;+ *)+ echo "ERROR: status must be one of pass, fail, skip, answer"+ exit 1+ ;;+esac++if [[ "${STATUS}" == "observed" ]]; then+ STATUS="answer"+fi++shift 3++if [[ "${STATUS}" == "answer" ]]; then+ if [[ -z "${ANSWER}" && $# -ge 1 ]]; then+ ANSWER="$1"+ shift+ fi+ NOTES="${1:-}"else- # Auto-detect: find most recent report of any type- REPORT_FILE=$(ls -t "${RESULTS_DIR}"/baseline_*.json "${RESULTS_DIR}"/agent_benchmark_*.json "${RESULTS_DIR}"/agent_browser_benchmark_*.json 2>/dev/null | head -1)+ NOTES="${1:-}"fi+resolve_current_report() {+ local ptr="$1"+ if [[ -f "${ptr}" ]]; then+ tr -d '[:space:]' < "${ptr}"+ return 0+ fi+ return 1+}+if [[ -z "${REPORT_FILE}" ]]; then+ if [[ -n "${REPORT_TYPE}" ]]; then+ case "${REPORT_TYPE}" in+ baseline)+ REPORT_FILE="$(resolve_current_report "${CURRENT_BASELINE_PTR}" || true)"+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/baseline_*.json 2>/dev/null | head -1)+ ;;+ agent)+ REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_PTR}" || true)"+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_benchmark_*.json 2>/dev/null | head -1)+ ;;+ agent-browser|agent_browser)+ REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_BROWSER_PTR}" || true)"+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_browser_benchmark_*.json 2>/dev/null | head -1)+ ;;+ *)+ echo "ERROR: --type must be 'baseline', 'agent', or 'agent-browser'"+ exit 1+ ;;+ esac+ else+ REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_BROWSER_PTR}" || true)"+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_PTR}" || true)"+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE="$(resolve_current_report "${CURRENT_BASELINE_PTR}" || true)"+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/baseline_*.json "${RESULTS_DIR}"/agent_benchmark_*.json "${RESULTS_DIR}"/agent_browser_benchmark_*.json 2>/dev/null | head -1)+ fi+fi++if [[ -z "${REPORT_FILE}" || ! -f "${REPORT_FILE}" ]]; thenecho "ERROR: No benchmark report found. Run ./run-optimization.sh first."exit 1fi-STEP_TOOL_CALLS=0-if [[ -n "${TOOL_CALLS}" ]]; then- STEP_TOOL_CALLS="${TOOL_CALLS}"-elif [[ "$(jq -r '.benchmark.type' "${REPORT_FILE}")" == "agent-browser" ]]; then- CURRENT_TOOL_CALLS=0- PREV_TOOL_CALLS=$(jq -r '.totals.tool_calls // 0' "${REPORT_FILE}")- if [[ -f "${AGENT_BROWSER_LOG}" ]]; then- CURRENT_TOOL_CALLS=$(jq -Rsc 'split("\n") | map(select(length > 0) | try fromjson catch empty) | length' "${AGENT_BROWSER_LOG}")- fi- if [[ "${CURRENT_TOOL_CALLS}" -ge "${PREV_TOOL_CALLS}" ]]; then- STEP_TOOL_CALLS=$((CURRENT_TOOL_CALLS - PREV_TOOL_CALLS))- fi-fi+BENCHMARK_TYPE=$(jq -r '.benchmark.type' "${REPORT_FILE}")++case "${BENCHMARK_TYPE}" in+ baseline)+ case "${STATUS}" in+ pass|fail|skip) ;;+ *)+ echo "ERROR: baseline steps must use pass, fail, or skip"+ exit 1+ ;;+ esac+ ;;+ agent|agent-browser)+ case "${STATUS}" in+ answer|fail|skip) ;;+ *)+ echo "ERROR: agent steps must use answer, fail, or skip"+ exit 1+ ;;+ esac+ ;;+esac-# Calculate cost (only meaningful for agent runs with token data)+# Optional token metadata for runners that can provide trustworthy numbers.COST=0if [[ ${TOTAL_TOKENS} -gt 0 ]]; then- MODEL=$(jq -r '.benchmark.model' "${REPORT_FILE}")+ MODEL=$(jq -r '.benchmark.model // "unknown"' "${REPORT_FILE}")- # Cost per 1M tokenscase "${MODEL}" in*haiku*) INPUT_RATE=0.25; OUTPUT_RATE=1.25 ;;*sonnet*) INPUT_RATE=3.0; OUTPUT_RATE=15.0 ;;@@ -130,6 +196,20 @@ if [[ ${TOTAL_TOKENS} -gt 0 ]]; thenCOST=$(echo "scale=6; (${INPUT_TOKENS} / 1000000 * ${INPUT_RATE}) + (${OUTPUT_TOKENS} / 1000000 * ${OUTPUT_RATE})" | bc)fi+STEP_TOOL_CALLS=0+if [[ -n "${TOOL_CALLS}" ]]; then+ STEP_TOOL_CALLS="${TOOL_CALLS}"+elif [[ "${BENCHMARK_TYPE}" == "agent-browser" ]]; then+ CURRENT_TOOL_CALLS=0+ PREV_TOOL_CALLS=$(jq -r '.totals.tool_calls // 0' "${REPORT_FILE}")+ if [[ -f "${AGENT_BROWSER_LOG}" ]]; then+ CURRENT_TOOL_CALLS=$(jq -Rsc 'split("\n") | map(select(length > 0) | try fromjson catch empty) | length' "${AGENT_BROWSER_LOG}")+ fi+ if [[ "${CURRENT_TOOL_CALLS}" -ge "${PREV_TOOL_CALLS}" ]]; then+ STEP_TOOL_CALLS=$((CURRENT_TOOL_CALLS - PREV_TOOL_CALLS))+ fi+fi+# Create step entrySTEP_JSON=$(jq -n \--argjson group "${GROUP}" \@@ -142,13 +222,19 @@ STEP_JSON=$(jq -n \--argjson tool_calls "${STEP_TOOL_CALLS}" \--argjson cost "${COST}" \--argjson bytes "${RESPONSE_BYTES}" \+ --arg answer "${ANSWER}" \--arg notes "${NOTES}" \--arg ts "${TIMESTAMP}" \'{group: $group, step: $step, id: $id, status: $status,input_tokens: $in_tokens, output_tokens: $out_tokens,- total_tokens: $total_tokens, tool_calls: $tool_calls,+ total_tokens: $total_tokens,+ tool_calls: $tool_calls,cost_usd: $cost,- response_bytes: $bytes, notes: $notes, timestamp: $ts}')+ response_bytes: $bytes, answer: $answer,+ notes: $notes, timestamp: $ts} |+ if $status == "answer" then+ .verification = {status: "pending"}+ else . end')# Append to report and update totalsTMP_FILE=$(mktemp)@@ -157,17 +243,20 @@ jq --argjson step "${STEP_JSON}" \--argjson out "${OUTPUT_TOKENS}" \--argjson tool_calls "${STEP_TOOL_CALLS}" \--argjson cost "${COST}" \- --arg status "${STATUS}" \- '(.totals.tool_calls //= 0) |- .steps += [$step] |- .totals.input_tokens += $in |- .totals.output_tokens += $out |- .totals.total_tokens += ($in + $out) |- .totals.tool_calls += $tool_calls |- .totals.estimated_cost_usd += $cost |- if $status == "pass" then .totals.steps_passed += 1- elif $status == "fail" then .totals.steps_failed += 1- else .totals.steps_skipped += 1 end' \+ '.steps += [$step] |+ .totals.input_tokens = ((.totals.input_tokens // 0) + $in) |+ .totals.output_tokens = ((.totals.output_tokens // 0) + $out) |+ .totals.total_tokens = ((.totals.total_tokens // 0) + $in + $out) |+ .totals.tool_calls = ((.totals.tool_calls // 0) + $tool_calls) |+ .totals.estimated_cost_usd = ((.totals.estimated_cost_usd // 0) + $cost) |+ .totals.steps_passed = ([.steps[] | select(.status == "pass")] | length) |+ .totals.steps_failed = ([.steps[] | select(.status == "fail")] | length) |+ .totals.steps_skipped = ([.steps[] | select(.status == "skip")] | length) |+ .totals.steps_answered = ([.steps[] | select(.status == "answer")] | length) |+ .totals.steps_verified_passed = ([.steps[] | select(.status == "answer" and ((.verification.status // "") == "pass"))] | length) |+ .totals.steps_verified_failed = ([.steps[] | select(.status == "answer" and ((.verification.status // "") == "fail"))] | length) |+ .totals.steps_verified_skipped = ([.steps[] | select(.status == "answer" and ((.verification.status // "") == "skip"))] | length) |+ .totals.steps_pending_verification = ([.steps[] | select(.status == "answer" and ((.verification.status // "pending") == "pending"))] | length)' \"${REPORT_FILE}" > "${TMP_FILE}"mv "${TMP_FILE}" "${REPORT_FILE}"@@ -177,4 +266,4 @@ if [[ "${STATUS}" == "fail" ]]; thenecho "[${TIMESTAMP}] Step ${GROUP}.${STEP} FAILED: ${NOTES}" >> "${RESULTS_DIR}/errors.log"fi-echo "Recorded: Step ${GROUP}.${STEP} = ${STATUS} (tokens: ${TOTAL_TOKENS}, bytes: ${RESPONSE_BYTES}, \$${COST})"+echo "Recorded: Step ${GROUP}.${STEP} = ${STATUS} (bytes: ${RESPONSE_BYTES}, tool_calls: ${STEP_TOOL_CALLS})"diff --git a/tests/benchmark/scripts/run-agent-browser-benchmark.sh b/tests/benchmark/scripts/run-agent-browser-benchmark.shindex 11e8e586..3de99c43 100755--- a/tests/benchmark/scripts/run-agent-browser-benchmark.sh+++ b/tests/benchmark/scripts/run-agent-browser-benchmark.sh@@ -8,6 +8,7 @@ set -euo pipefailSCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"BENCH_DIR="${SCRIPT_DIR}/.."RESULTS_DIR="${BENCH_DIR}/results"+CURRENT_REPORT_PTR="${RESULTS_DIR}/current_agent_browser_report.txt"mkdir -p "${RESULTS_DIR}"TIMESTAMP=$(date +%Y%m%d_%H%M%S)@@ -56,12 +57,15 @@ cat > "${REPORT_FILE}" << EOF}EOF+printf '%s\n' "${REPORT_FILE}" > "${CURRENT_REPORT_PTR}"+echo "Initialized agent-browser benchmark report:"echo " ${REPORT_FILE}"echo ""echo "Next steps:"-echo " 1. Read ../../skills/agent-browser/SKILL.md"-echo " 2. Use ./scripts/ab ... to drive agent-browser inside Docker"-echo " 3. Record each completed benchmark step with:"-echo " ./scripts/record-step.sh --type agent-browser <group> <step> <pass|fail|skip> --tokens <in> <out> \"notes\""-echo " 4. Summarize the report with ./scripts/finalize-report.sh"+echo " 1. Read ./AGENT_BROWSER_INSTRUCTIONS.md"+echo " 2. Load the live CLI skill with ./scripts/ab skills get agent-browser --full"+echo " 3. Use ./scripts/ab ... to drive agent-browser inside Docker"+echo " 4. Record each completed benchmark step with:"+echo " ./scripts/record-step.sh --type agent-browser <group> <step> <pass|fail|skip|answer> \"what you saw\" \"notes\""+echo " 5. Summarize the report with ./scripts/finalize-report.sh"diff --git a/tests/benchmark/scripts/run-optimization.sh b/tests/benchmark/scripts/run-optimization.shindex caf512d3..41d5ef99 100755--- a/tests/benchmark/scripts/run-optimization.sh+++ b/tests/benchmark/scripts/run-optimization.sh@@ -10,6 +10,8 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"RESULTS_DIR="${SCRIPT_DIR}/../results"mkdir -p "${RESULTS_DIR}"LOG_FILE="${RESULTS_DIR}/optimization_log.md"+CURRENT_BASELINE_PTR="${RESULTS_DIR}/current_baseline_report.txt"+CURRENT_AGENT_PTR="${RESULTS_DIR}/current_agent_report.txt"TIMESTAMP=$(date +%Y%m%d_%H%M%S)RUN_NUMBER=$(grep -c "^## Run #" "${LOG_FILE}" 2>/dev/null || echo 0)RUN_NUMBER=$((RUN_NUMBER + 1))@@ -82,6 +84,9 @@ cat > "${AGENT_REPORT}" << EOF}EOF+printf '%s\n' "${BASELINE_REPORT}" > "${CURRENT_BASELINE_PTR}"+printf '%s\n' "${AGENT_REPORT}" > "${CURRENT_AGENT_PTR}"+# Clear previous agent commands logrm -f "${RESULTS_DIR}/agent_commands.log"diff --git a/tests/benchmark/scripts/verify-step.sh b/tests/benchmark/scripts/verify-step.shnew file mode 100755index 00000000..1e1b9dd3--- /dev/null+++ b/tests/benchmark/scripts/verify-step.sh@@ -0,0 +1,122 @@+#!/usr/bin/env bash+#+# Record a deferred verification decision for a previously observed benchmark+# step.+#+# Usage:+# ./verify-step.sh [--type agent|agent-browser] [--report-file <path>] <group> <step> <pass|fail|skip> "notes"++set -euo pipefail++SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+RESULTS_DIR="${SCRIPT_DIR}/../results"+CURRENT_AGENT_PTR="${RESULTS_DIR}/current_agent_report.txt"+CURRENT_AGENT_BROWSER_PTR="${RESULTS_DIR}/current_agent_browser_report.txt"++REPORT_TYPE=""+REPORT_FILE=""++while [[ $# -gt 0 && "$1" == --* ]]; do+ case "$1" in+ --type)+ REPORT_TYPE="$2"+ shift 2+ ;;+ --report-file)+ REPORT_FILE="$2"+ shift 2+ ;;+ *)+ echo "Unknown option: $1"+ exit 1+ ;;+ esac+done++if [[ $# -lt 3 ]]; then+ echo "Usage: $0 [--type agent|agent-browser] [--report-file <path>] <group> <step> <pass|fail|skip> [notes]"+ exit 1+fi++GROUP="$1"+STEP="$2"+STATUS="$3"+NOTES="${4:-}"+STEP_ID="${GROUP}.${STEP}"+TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ)++case "${STATUS}" in+ pass|fail|skip) ;;+ *)+ echo "ERROR: verification status must be one of pass, fail, skip"+ exit 1+ ;;+esac++resolve_current_report() {+ local ptr="$1"+ if [[ -f "${ptr}" ]]; then+ tr -d '[:space:]' < "${ptr}"+ return 0+ fi+ return 1+}++if [[ -z "${REPORT_FILE}" ]]; then+ if [[ -n "${REPORT_TYPE}" ]]; then+ case "${REPORT_TYPE}" in+ agent)+ REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_PTR}" || true)"+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_benchmark_*.json 2>/dev/null | head -1)+ ;;+ agent-browser|agent_browser)+ REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_BROWSER_PTR}" || true)"+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_browser_benchmark_*.json 2>/dev/null | head -1)+ ;;+ *)+ echo "ERROR: --type must be 'agent' or 'agent-browser'"+ exit 1+ ;;+ esac+ else+ REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_BROWSER_PTR}" || true)"+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_PTR}" || true)"+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_browser_benchmark_*.json "${RESULTS_DIR}"/agent_benchmark_*.json 2>/dev/null | head -1)+ fi+fi++if [[ -z "${REPORT_FILE:-}" || ! -f "${REPORT_FILE}" ]]; then+ echo "ERROR: no agent benchmark report found"+ exit 1+fi++TMP_FILE=$(mktemp)+jq --arg id "${STEP_ID}" \+ --arg status "${STATUS}" \+ --arg notes "${NOTES}" \+ --arg ts "${TIMESTAMP}" \+ '+ if any(.steps[]; .id == $id) | not then+ error("step not found: " + $id)+ elif any(.steps[]; .id == $id and .status == "answer") | not then+ error("step is not answer-status and cannot be verified: " + $id)+ else+ .+ end |+ .steps |= map(+ if .id == $id then+ .verification = {+ status: $status,+ notes: $notes,+ timestamp: $ts+ }+ else . end+ ) |+ .totals.steps_verified_passed = ([.steps[] | select(.status == "answer" and ((.verification.status // "") == "pass"))] | length) |+ .totals.steps_verified_failed = ([.steps[] | select(.status == "answer" and ((.verification.status // "") == "fail"))] | length) |+ .totals.steps_verified_skipped = ([.steps[] | select(.status == "answer" and ((.verification.status // "") == "skip"))] | length) |+ .totals.steps_pending_verification = ([.steps[] | select(.status == "answer" and ((.verification.status // "pending") == "pending"))] | length)+ ' "${REPORT_FILE}" > "${TMP_FILE}"++mv "${TMP_FILE}" "${REPORT_FILE}"+echo "Verified: Step ${STEP_ID} = ${STATUS}"