Skip to content

High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.

chore: benchmark improvements

Luigi Agosti90c446e0c6ee
diff --git a/skills/agent-browser/SKILL.md b/skills/agent-browser/SKILL.md
deleted file mode 100644
index e58a9ef5..00000000
--- a/skills/agent-browser/SKILL.md
+++ /dev/null
@@ -1,132 +0,0 @@
----
-name: agent-browser
-description: "Use this skill when a benchmark or local automation task should drive the browser through the agent-browser CLI instead of PinchTab. In this repo, prefer the benchmark wrapper `./tests/benchmark/scripts/ab` so commands execute inside the shared Docker benchmark environment and tool calls are logged."
----
-
-# Browser Automation with Agent Browser
-
-For benchmark runs in this repo, do not call `agent-browser` directly. Use the
-wrapper:
-
-```bash
-./tests/benchmark/scripts/ab ...
-```
-
-That wrapper:
-
-- executes `agent-browser` inside the benchmark Docker service
-- preserves the shared browser session across commands
-- logs every tool call into `tests/benchmark/results/agent_browser_commands.ndjson`
-
-Use this skill when the benchmark should run the same fixture tasks as the
-PinchTab lane, but with `agent-browser` as the browser-control surface.
-
-## Benchmark Environment
-
-- Shared fixture site: `http://fixtures/`
-- Wrapper command: `./tests/benchmark/scripts/ab`
-- Default session name: `benchmark`
-- Report recorder: `./tests/benchmark/scripts/record-step.sh --type agent-browser ...`
-
-Before running tasks, initialize the benchmark lane:
-
-```bash
-cd tests/benchmark
-./scripts/run-agent-browser-benchmark.sh
-```
-
-## Core Workflow
-
-Follow this pattern:
-
-1. Open or navigate the target page with `./tests/benchmark/scripts/ab open <url>`
-2. Inspect the page with `./tests/benchmark/scripts/ab snapshot -i -c`
-3. Use fresh refs such as `@e2`, `@e7`, `@e11` for actions
-4. Re-snapshot after any action that changes the DOM
-5. Record the finished benchmark step with `record-step.sh`
-
-Rules:
-
-- Prefer `snapshot -i -c` to find actionable elements
-- Prefer `@eN` refs from the latest snapshot over brittle selectors
-- Re-snapshot after navigation, form submit, modal open, tab change, accordion expand, or SPA update
-- Use the same session for the whole benchmark run unless a task explicitly needs a reset
-
-## Essential Commands
-
-### Navigation and observation
-
-```bash
-./tests/benchmark/scripts/ab open http://fixtures/
-./tests/benchmark/scripts/ab back
-./tests/benchmark/scripts/ab forward
-./tests/benchmark/scripts/ab reload
-./tests/benchmark/scripts/ab snapshot
-./tests/benchmark/scripts/ab snapshot -i -c
-./tests/benchmark/scripts/ab snapshot -i --urls
-./tests/benchmark/scripts/ab get text
-./tests/benchmark/scripts/ab get title
-./tests/benchmark/scripts/ab get url
-```
-
-### Interaction
-
-```bash
-./tests/benchmark/scripts/ab click @e2
-./tests/benchmark/scripts/ab fill @e4 "[email protected]"
-./tests/benchmark/scripts/ab type @e5 "hello"
-./tests/benchmark/scripts/ab press Enter
-./tests/benchmark/scripts/ab hover @e7
-./tests/benchmark/scripts/ab check @e8
-./tests/benchmark/scripts/ab uncheck @e8
-./tests/benchmark/scripts/ab select @e9 "uk"
-./tests/benchmark/scripts/ab drag @e10 @e11
-./tests/benchmark/scripts/ab scroll down 800
-./tests/benchmark/scripts/ab scrollintoview @e12
-./tests/benchmark/scripts/ab wait @e13
-./tests/benchmark/scripts/ab wait 1000
-```
-
-### Exports and debugging
-
-```bash
-./tests/benchmark/scripts/ab screenshot /tmp/page.png
-./tests/benchmark/scripts/ab pdf /tmp/page.pdf
-./tests/benchmark/scripts/ab console
-./tests/benchmark/scripts/ab errors
-./tests/benchmark/scripts/ab eval "document.title"
-```
-
-## Selector Guidance
-
-`agent-browser` accepts:
-
-- refs from snapshot output, like `@e2`
-- CSS selectors, like `#submit` or `.btn-primary`
-- XPath selectors, like `//button[@type="submit"]`
-
-Prefer refs first. They are the most stable option inside the benchmark.
-
-## Benchmark Reporting
-
-After completing a benchmark step, record it:
-
-```bash
-./tests/benchmark/scripts/record-step.sh \
- --type agent-browser \
- --tokens 120 48 \
- 1 1 pass \
- "opened page and extracted content"
-```
-
-`record-step.sh` automatically derives `tool_calls` for the step from the
-wrapper log unless you override it with `--tool-calls <n>`.
-
-## Task Entry Point
-
-Use this skill together with:
-
-- `tests/benchmark/AGENT_BROWSER_TASKS.md`
-
-That file contains the benchmark tasks; this skill tells you how to operate the
-browser tool surface correctly.
diff --git a/tests/benchmark/AGENT_BROWSER_INSTRUCTIONS.md b/tests/benchmark/AGENT_BROWSER_INSTRUCTIONS.md
new file mode 100644
index 00000000..d35e46af
--- /dev/null
+++ b/tests/benchmark/AGENT_BROWSER_INSTRUCTIONS.md
@@ -0,0 +1,75 @@
+# Agent Browser Instructions
+
+These are benchmark-specific instructions for running the `agent-browser` lane.
+They are not the official `agent-browser` skill. For the live CLI skill, load it
+through the benchmark wrapper:
+
+```bash
+./scripts/ab skills get agent-browser --full
+```
+
+## Wrapper
+
+For benchmark runs in this repo, do not call `agent-browser` directly. Use:
+
+```bash
+./scripts/ab ...
+```
+
+That wrapper:
+
+- executes `agent-browser` inside the benchmark Docker service
+- preserves the shared browser session across commands
+- logs tool calls to `results/agent_browser_commands.ndjson`
+
+## Benchmark Workflow
+
+1. Start the lane:
+
+```bash
+cd tests/benchmark
+./scripts/run-agent-browser-benchmark.sh
+```
+
+2. Load the live CLI skill:
+
+```bash
+./scripts/ab skills get agent-browser --full
+```
+
+3. Run browser actions through the wrapper:
+
+```bash
+./scripts/ab open http://fixtures/
+./scripts/ab snapshot -i -c
+./scripts/ab click @e2
+./scripts/ab fill @e3 "[email protected]"
+```
+
+4. Record execution results:
+
+```bash
+./scripts/record-step.sh --type agent-browser 1 1 answer \
+ "opened fixtures home and got refs e1-e13" "completed"
+```
+
+5. Verify later in a separate pass:
+
+```bash
+./scripts/verify-step.sh --type agent-browser 1 1 pass \
+ "matched expected homepage state"
+```
+
+## Environment
+
+- Fixtures: `http://fixtures/`
+- Session name: `benchmark` by default (`AGENT_BROWSER_SESSION` overrides)
+- Browser driver: Docker service `agent-browser`
+
+## Operating Guidance
+
+- Treat `./scripts/ab skills get agent-browser --full` as the source of truth for
+ current command syntax and workflows
+- Prefer refs from `snapshot -i -c` over brittle selectors
+- Re-snapshot after navigation or any DOM-changing action
+- Keep one session for the whole benchmark lane unless a task explicitly needs a reset
diff --git a/tests/benchmark/AGENT_BROWSER_TASKS.md b/tests/benchmark/AGENT_BROWSER_TASKS.md
index 00e6ae39..cdd891c3 100644
--- a/tests/benchmark/AGENT_BROWSER_TASKS.md
+++ b/tests/benchmark/AGENT_BROWSER_TASKS.md
@@ -6,17 +6,23 @@ PinchTab CLI.
## Setup
-1. Read `../../skills/agent-browser/SKILL.md` first. This is your guide to the
- benchmark wrapper and command workflow.
+1. Read `./AGENT_BROWSER_INSTRUCTIONS.md` first. This is your benchmark guide to
+ the wrapper and command workflow.
-2. Start the Docker benchmark environment and initialize a report:
+2. Load the live CLI skill content before using browser commands:
+
+```bash
+./scripts/ab skills get agent-browser --full
+```
+
+3. Start the Docker benchmark environment and initialize a report:
```bash
cd tests/benchmark
./scripts/run-agent-browser-benchmark.sh
```
-3. Use the Docker-backed wrapper for every browser action:
+4. Use the Docker-backed wrapper for every browser action:
```bash
./scripts/ab open http://fixtures/
@@ -25,16 +31,23 @@ cd tests/benchmark
./scripts/ab fill @e3 "[email protected]"
```
-4. Record each completed step:
+5. Record each completed step as an answer/result first:
```bash
-./scripts/record-step.sh --type agent-browser 1 1 pass --tokens 120 48 "completed"
+./scripts/record-step.sh --type agent-browser 1 1 answer \
+ "opened fixtures home and got refs e1-e13" "completed"
```
`record-step.sh` will automatically calculate the number of `agent-browser`
tool calls used since the previous recorded step by reading
`results/agent_browser_commands.ndjson`.
+6. After the execution lane is done, run a separate verification pass with:
+
+```bash
+./scripts/verify-step.sh --type agent-browser 1 1 pass "matched expected homepage state"
+```
+
## Environment
- Fixtures: `http://fixtures/`
@@ -43,7 +56,7 @@ tool calls used since the previous recorded step by reading
## Tooling Guidance
-Use `../../skills/agent-browser/SKILL.md` as the primary operating guide.
+Use `./AGENT_BROWSER_INSTRUCTIONS.md` as the primary benchmark operating guide.
Reach for `./scripts/ab --help` only when the skill does not already answer the
question.
diff --git a/tests/benchmark/AGENT_TASKS.md b/tests/benchmark/AGENT_TASKS.md
index 31d00032..18a16e28 100644
--- a/tests/benchmark/AGENT_TASKS.md
+++ b/tests/benchmark/AGENT_TASKS.md
@@ -7,7 +7,11 @@ Natural language tasks to test how well an agent uses PinchTab from skill docs a
1. Read `../../skills/pinchtab/SKILL.md` — this is your only guide
2. For each task, figure out which commands to use
3. **Log every command executed**
-4. Record: `./scripts/record-step.sh --type agent <group> <step> <pass|fail> --tokens <in> <out> "notes"`
+4. Do not score your own work as pass/fail unless a task is trivially objective.
+5. Prefer recording each step as `answer` with the actual response/result:
+ `./scripts/record-step.sh --type agent <group> <step> answer "<what you saw>" "notes"`
+6. A separate verification pass should later stamp the step with:
+ `./scripts/verify-step.sh --type agent <group> <step> <pass|fail|skip> "verification notes"`
### Recommended setup: env vars + `./scripts/pt` wrapper
@@ -40,8 +44,8 @@ export PINCHTAB_TAB=$(./scripts/pt nav http://fixtures/)
./scripts/pt click '#submit'
./scripts/pt drag '#piece' --drag-x 12 --drag-y -158
-# Record step results:
-./scripts/record-step.sh --type agent 1 2 pass "worked"
+# Record answer/result for later verification:
+./scripts/record-step.sh --type agent 1 2 answer "clicked submit and saw success banner" "worked"
```
The wrapper is a ~60-line shell script; read `scripts/pt` for the exact
@@ -50,6 +54,10 @@ anything that works with `pinchtab ...` works with `./scripts/pt ...`.
Explicit `--tab <id>` on any command still wins over `PINCHTAB_TAB`.
+For deferred verification, keep the answer payload factual. Record the
+actual page text, count, answer, or state transition you observed, not your own
+judgment about whether that matches the benchmark oracle.
+
## Environment
- PinchTab: `http://localhost:9867`, token: `benchmark-token`
diff --git a/tests/benchmark/README.md b/tests/benchmark/README.md
index 943ce790..b653f212 100644
--- a/tests/benchmark/README.md
+++ b/tests/benchmark/README.md
@@ -13,7 +13,7 @@ cd tests/benchmark
# agent-browser lane
./scripts/run-agent-browser-benchmark.sh
-# Then run tasks from AGENT_BROWSER_TASKS.md with ./scripts/ab
+# Then read AGENT_BROWSER_INSTRUCTIONS.md and run tasks from AGENT_BROWSER_TASKS.md with ./scripts/ab
./scripts/finalize-report.sh
```
@@ -34,13 +34,13 @@ If Docker build fails or is skipped, the benchmark is **INVALID**.
| File | Purpose |
|------|---------|
| `../../skills/pinchtab/SKILL.md` | PinchTab skill (same as shipped product) |
-| `../../skills/agent-browser/SKILL.md` | `agent-browser` skill for the benchmark wrapper |
+| `AGENT_BROWSER_INSTRUCTIONS.md` | Benchmark-local instructions for the `agent-browser` lane |
| `BASELINE_TASKS.md` | Standalone task list (same as skill) |
| `AGENT_BROWSER_TASKS.md` | Equivalent task lane for `agent-browser` |
| `scripts/run-optimization.sh` | Initialize PinchTab benchmark reports |
| `scripts/run-agent-browser-benchmark.sh` | Start fixtures + `agent-browser` and initialize a fresh report |
| `scripts/ab` | Docker-backed `agent-browser` wrapper with tool-call logging |
-| `scripts/record-step.sh` | Record step results with token counts and tool-call counts |
+| `scripts/record-step.sh` | Record step results and tool-call counts |
| `scripts/finalize-report.sh` | Generate final summary report |
| `config/pinchtab.json` | PinchTab configuration |
| `agent-browser/Dockerfile` | `agent-browser` benchmark image |
@@ -53,15 +53,20 @@ The benchmark is designed to run in a fresh agent context:
1. Initialize the relevant benchmark lane
2. Execute the natural-language tasks with the target browser surface
-3. Record each step's input/output tokens
-4. Let the harness count browser/tool calls where possible
+3. Record each step's raw answer/result
+4. Verify observed steps in a separate pass
+5. Let the harness count browser/tool calls where possible
This measures the **real cost** of using a browser tool with an agent, including:
- Context loading overhead
-- Per-task token usage
- Browser/tool-call count
- Total benchmark cost
+For agent lanes, the recommended flow is two-phase:
+
+1. execution agent records each step as `answer`
+2. verifier agent reads the report and stamps each step with `verify-step.sh`
+
## Environment
The benchmark runs PinchTab in Docker with:
@@ -72,24 +77,29 @@ The benchmark runs PinchTab in Docker with:
- **Headless**: Yes
- **Multi-instance**: Enabled (2 instances)
-## Token Tracking
+## Step Recording
-Every step must track token usage:
+Every step must record the answer/result:
```bash
-./scripts/record-step.sh <group> <step> <pass|fail|skip> <input_tokens> <output_tokens> "notes"
+./scripts/record-step.sh <group> <step> <pass|fail|skip|answer> "notes"
```
Example:
```bash
-./scripts/record-step.sh 1 1 pass 150 45 "Navigation completed in 1.2s"
-./scripts/record-step.sh 2 3 fail 200 80 "Element not found"
+./scripts/record-step.sh 1 1 pass "Navigation completed in 1.2s"
+./scripts/record-step.sh 2 3 fail "Element not found"
```
-Token counts should come from your model's API response:
-- **Anthropic**: `usage.input_tokens`, `usage.output_tokens`
-- **OpenAI**: `usage.prompt_tokens`, `usage.completion_tokens`
-- **Gemini**: `usageMetadata.promptTokenCount`, `usageMetadata.candidatesTokenCount`
+Deferred-verification example:
+
+```bash
+./scripts/record-step.sh --type agent 1 1 answer \
+ "Found categories Programming Languages: 12, Databases: 8" \
+ "raw answer"
+./scripts/verify-step.sh --type agent 1 1 pass \
+ "Answer satisfies the benchmark oracle"
+```
## Reports
@@ -111,11 +121,10 @@ Reports are generated in `results/`:
| Steps Failed | 2 |
| Pass Rate | 93.7% |
-## Token Usage
+## Tooling
| Metric | Value |
|--------|-------|
-| Total Tokens | 4,523 |
-| Estimated Cost | $0.0158 |
+| Tool Calls | 128 |
```
## Running Programmatically
diff --git a/tests/benchmark/scripts/finalize-report.sh b/tests/benchmark/scripts/finalize-report.sh
index 0ba93690..c126f8ed 100755
--- a/tests/benchmark/scripts/finalize-report.sh
+++ b/tests/benchmark/scripts/finalize-report.sh
@@ -7,19 +7,36 @@ set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
BENCH_DIR="${SCRIPT_DIR}/.."
RESULTS_DIR="${BENCH_DIR}/results"
+CURRENT_BASELINE_PTR="${RESULTS_DIR}/current_baseline_report.txt"
+CURRENT_AGENT_PTR="${RESULTS_DIR}/current_agent_report.txt"
+CURRENT_AGENT_BROWSER_PTR="${RESULTS_DIR}/current_agent_browser_report.txt"
+
+resolve_current_report() {
+ local ptr="$1"
+ if [[ -f "${ptr}" ]]; then
+ tr -d '[:space:]' < "${ptr}"
+ return 0
+ fi
+ return 1
+}
REPORT_FILE="${1:-}"
if [[ -z "${REPORT_FILE}" ]]; then
- shopt -s nullglob
- candidates=(
- "${RESULTS_DIR}"/agent_browser_benchmark_*.json
- "${RESULTS_DIR}"/agent_benchmark_*.json
- "${RESULTS_DIR}"/baseline_*.json
- )
- shopt -u nullglob
-
- if [[ ${#candidates[@]} -gt 0 ]]; then
- REPORT_FILE=$(ls -t "${candidates[@]}" | head -1)
+ REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_BROWSER_PTR}" || true)"
+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_PTR}" || true)"
+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE="$(resolve_current_report "${CURRENT_BASELINE_PTR}" || true)"
+ if [[ -z "${REPORT_FILE}" ]]; then
+ shopt -s nullglob
+ candidates=(
+ "${RESULTS_DIR}"/agent_browser_benchmark_*.json
+ "${RESULTS_DIR}"/agent_benchmark_*.json
+ "${RESULTS_DIR}"/baseline_*.json
+ )
+ shopt -u nullglob
+
+ if [[ ${#candidates[@]} -gt 0 ]]; then
+ REPORT_FILE=$(ls -t "${candidates[@]}" | head -1)
+ fi
fi
fi
@@ -34,28 +51,66 @@ jq -r '
def pct($a; $b):
if $b == 0 then "0.0%" else (((1000 * $a) / $b | round) / 10 | tostring) + "%" end;
. as $root |
- ($root.totals.steps_passed + $root.totals.steps_failed + $root.totals.steps_skipped) as $step_count |
- "# Benchmark Summary",
+ ($root.totals.steps_passed + $root.totals.steps_failed + $root.totals.steps_skipped) as $baseline_total |
+ (($root.totals.steps_answered // 0) + $root.totals.steps_failed + $root.totals.steps_skipped) as $agent_total |
+ ($root.totals.steps_verified_passed // 0) as $verified_passed |
+ ($root.totals.steps_verified_failed // 0) as $verified_failed |
+ ($root.totals.steps_verified_skipped // 0) as $verified_skipped |
+ ($root.totals.steps_pending_verification // 0) as $pending |
+ (if $root.benchmark.type == "baseline" then
+ [
+ "# Benchmark Summary",
+ "",
+ "| Metric | Value |",
+ "|--------|-------|",
+ "| Type | \($root.benchmark.type) |",
+ "| Model | \($root.benchmark.model) |",
+ "| Steps Passed | \($root.totals.steps_passed) |",
+ "| Steps Failed | \($root.totals.steps_failed) |",
+ "| Steps Skipped | \($root.totals.steps_skipped) |",
+ "| Pass Rate | \(pct($root.totals.steps_passed; $baseline_total)) |",
+ "| Tool Calls | \($root.totals.tool_calls // 0) |"
+ ]
+ else
+ [
+ "# Benchmark Summary",
+ "",
+ "| Metric | Value |",
+ "|--------|-------|",
+ "| Type | \($root.benchmark.type) |",
+ "| Model | \($root.benchmark.model) |",
+ "| Steps Answered | \($root.totals.steps_answered // 0) |",
+ "| Steps Failed | \($root.totals.steps_failed) |",
+ "| Steps Skipped | \($root.totals.steps_skipped) |",
+ "| Answer Rate | \(pct($root.totals.steps_answered // 0; $agent_total)) |",
+ "| Verified Passed | \($verified_passed) |",
+ "| Verified Failed | \($verified_failed) |",
+ "| Verified Skipped | \($verified_skipped) |",
+ "| Pending Verification | \($pending) |",
+ "| Verification Pass Rate | \(pct($verified_passed; $agent_total)) |",
+ "| Tool Calls | \($root.totals.tool_calls // 0) |"
+ ]
+ end)[],
"",
- "| Metric | Value |",
- "|--------|-------|",
- "| Type | \($root.benchmark.type) |",
- "| Model | \($root.benchmark.model) |",
- "| Steps Passed | \($root.totals.steps_passed) |",
- "| Steps Failed | \($root.totals.steps_failed) |",
- "| Steps Skipped | \($root.totals.steps_skipped) |",
- "| Pass Rate | \(pct($root.totals.steps_passed; $step_count)) |",
- "| Input Tokens | \($root.totals.input_tokens) |",
- "| Output Tokens | \($root.totals.output_tokens) |",
- "| Total Tokens | \($root.totals.total_tokens) |",
- "| Tool Calls | \($root.totals.tool_calls // 0) |",
- "| Estimated Cost (USD) | \($root.totals.estimated_cost_usd) |",
+ "## Pending Verification",
+ "",
+ (
+ [ $root.steps[] | select(.status == "answer" and ((.verification.status // "pending") == "pending")) | "- \(.id): \(.answer // .notes // "")" ] |
+ if length == 0 then ["- none"] else . end
+ )[],
"",
"## Failed Steps",
"",
(
[ $root.steps[] | select(.status == "fail") | "- \(.id): \(.notes)" ] |
if length == 0 then ["- none"] else . end
+ )[],
+ "",
+ "## Verification Failures",
+ "",
+ (
+ [ $root.steps[] | select(.status == "answer" and ((.verification.status // "") == "fail")) | "- \(.id): \(.verification.notes // "")" ] |
+ if length == 0 then ["- none"] else . end
)[]
' "${REPORT_FILE}" > "${SUMMARY_FILE}"
diff --git a/tests/benchmark/scripts/record-step.sh b/tests/benchmark/scripts/record-step.sh
index d8cf70d8..7f44ff31 100755
--- a/tests/benchmark/scripts/record-step.sh
+++ b/tests/benchmark/scripts/record-step.sh
@@ -3,17 +3,20 @@
# Record a benchmark step result
#
# Usage:
-# ./record-step.sh [--type baseline|agent|agent-browser] <group> <step> <pass|fail|skip> [options] "notes"
+# ./record-step.sh [--type baseline|agent|agent-browser] <group> <step> <pass|fail|skip|answer> [answer] [notes]
#
# Options:
# --type baseline|agent|agent-browser Report type (default: auto-detect most recent)
-# --tokens <in> <out> Token usage (agent runs only, default: 0 0)
+# --tokens <in> <out> Deprecated and ignored (kept for compatibility)
# --bytes <n> HTTP response size in bytes (baseline runs)
# --tool-calls <n> Override auto-counted tool calls
+# --observed <text> Deprecated alias for answer text
+# --answer <text> Optional explicit answer text
#
# Examples:
# ./record-step.sh 1 1 pass "Navigation completed"
-# ./record-step.sh --type agent 2 3 fail --tokens 200 80 "Element not found"
+# ./record-step.sh --type agent 2 3 fail "Element not found"
+# ./record-step.sh --type agent 2 4 answer "Robert Griesemer, 2009" "read infobox"
# ./record-step.sh --type baseline 1 2 pass --bytes 4520 "Snapshot returned"
set -euo pipefail
@@ -24,11 +27,16 @@ mkdir -p "${RESULTS_DIR}"
# Parse flags
REPORT_TYPE=""
+REPORT_FILE=""
INPUT_TOKENS=0
OUTPUT_TOKENS=0
RESPONSE_BYTES=0
TOOL_CALLS=""
+ANSWER=""
AGENT_BROWSER_LOG="${RESULTS_DIR}/agent_browser_commands.ndjson"
+CURRENT_BASELINE_PTR="${RESULTS_DIR}/current_baseline_report.txt"
+CURRENT_AGENT_PTR="${RESULTS_DIR}/current_agent_report.txt"
+CURRENT_AGENT_BROWSER_PTR="${RESULTS_DIR}/current_agent_browser_report.txt"
while [[ $# -gt 0 && "$1" == --* ]]; do
case "$1" in
@@ -36,6 +44,10 @@ while [[ $# -gt 0 && "$1" == --* ]]; do
REPORT_TYPE="$2"
shift 2
;;
+ --report-file)
+ REPORT_FILE="$2"
+ shift 2
+ ;;
--tokens)
INPUT_TOKENS="$2"
OUTPUT_TOKENS="$3"
@@ -49,6 +61,14 @@ while [[ $# -gt 0 && "$1" == --* ]]; do
TOOL_CALLS="$2"
shift 2
;;
+ --observed)
+ ANSWER="$2"
+ shift 2
+ ;;
+ --answer)
+ ANSWER="$2"
+ shift 2
+ ;;
*)
echo "Unknown option: $1"
exit 1
@@ -57,64 +77,110 @@ while [[ $# -gt 0 && "$1" == --* ]]; do
done
if [[ $# -lt 3 ]]; then
- echo "Usage: $0 [--type baseline|agent|agent-browser] <group> <step> <pass|fail|skip> [--tokens <in> <out>] [--bytes <n>] [--tool-calls <n>] [notes]"
+ echo "Usage: $0 [--type baseline|agent|agent-browser] [--report-file <path>] <group> <step> <pass|fail|skip|answer> [answer] [notes]"
exit 1
fi
GROUP="$1"
STEP="$2"
STATUS="$3"
-NOTES="${4:-}"
TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ)
TOTAL_TOKENS=$((INPUT_TOKENS + OUTPUT_TOKENS))
-# Find report file
-if [[ -n "${REPORT_TYPE}" ]]; then
- case "${REPORT_TYPE}" in
- baseline)
- REPORT_FILE=$(ls -t "${RESULTS_DIR}"/baseline_*.json 2>/dev/null | head -1)
- ;;
- agent)
- REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_benchmark_*.json 2>/dev/null | head -1)
- ;;
- agent-browser|agent_browser)
- REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_browser_benchmark_*.json 2>/dev/null | head -1)
- ;;
- *)
- echo "ERROR: --type must be 'baseline', 'agent', or 'agent-browser'"
- exit 1
- ;;
- esac
+case "${STATUS}" in
+ pass|fail|skip|answer|observed) ;;
+ *)
+ echo "ERROR: status must be one of pass, fail, skip, answer"
+ exit 1
+ ;;
+esac
+
+if [[ "${STATUS}" == "observed" ]]; then
+ STATUS="answer"
+fi
+
+shift 3
+
+if [[ "${STATUS}" == "answer" ]]; then
+ if [[ -z "${ANSWER}" && $# -ge 1 ]]; then
+ ANSWER="$1"
+ shift
+ fi
+ NOTES="${1:-}"
else
- # Auto-detect: find most recent report of any type
- REPORT_FILE=$(ls -t "${RESULTS_DIR}"/baseline_*.json "${RESULTS_DIR}"/agent_benchmark_*.json "${RESULTS_DIR}"/agent_browser_benchmark_*.json 2>/dev/null | head -1)
+ NOTES="${1:-}"
fi
+resolve_current_report() {
+ local ptr="$1"
+ if [[ -f "${ptr}" ]]; then
+ tr -d '[:space:]' < "${ptr}"
+ return 0
+ fi
+ return 1
+}
+
if [[ -z "${REPORT_FILE}" ]]; then
+ if [[ -n "${REPORT_TYPE}" ]]; then
+ case "${REPORT_TYPE}" in
+ baseline)
+ REPORT_FILE="$(resolve_current_report "${CURRENT_BASELINE_PTR}" || true)"
+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/baseline_*.json 2>/dev/null | head -1)
+ ;;
+ agent)
+ REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_PTR}" || true)"
+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_benchmark_*.json 2>/dev/null | head -1)
+ ;;
+ agent-browser|agent_browser)
+ REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_BROWSER_PTR}" || true)"
+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_browser_benchmark_*.json 2>/dev/null | head -1)
+ ;;
+ *)
+ echo "ERROR: --type must be 'baseline', 'agent', or 'agent-browser'"
+ exit 1
+ ;;
+ esac
+ else
+ REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_BROWSER_PTR}" || true)"
+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_PTR}" || true)"
+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE="$(resolve_current_report "${CURRENT_BASELINE_PTR}" || true)"
+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/baseline_*.json "${RESULTS_DIR}"/agent_benchmark_*.json "${RESULTS_DIR}"/agent_browser_benchmark_*.json 2>/dev/null | head -1)
+ fi
+fi
+
+if [[ -z "${REPORT_FILE}" || ! -f "${REPORT_FILE}" ]]; then
echo "ERROR: No benchmark report found. Run ./run-optimization.sh first."
exit 1
fi
-STEP_TOOL_CALLS=0
-if [[ -n "${TOOL_CALLS}" ]]; then
- STEP_TOOL_CALLS="${TOOL_CALLS}"
-elif [[ "$(jq -r '.benchmark.type' "${REPORT_FILE}")" == "agent-browser" ]]; then
- CURRENT_TOOL_CALLS=0
- PREV_TOOL_CALLS=$(jq -r '.totals.tool_calls // 0' "${REPORT_FILE}")
- if [[ -f "${AGENT_BROWSER_LOG}" ]]; then
- CURRENT_TOOL_CALLS=$(jq -Rsc 'split("\n") | map(select(length > 0) | try fromjson catch empty) | length' "${AGENT_BROWSER_LOG}")
- fi
- if [[ "${CURRENT_TOOL_CALLS}" -ge "${PREV_TOOL_CALLS}" ]]; then
- STEP_TOOL_CALLS=$((CURRENT_TOOL_CALLS - PREV_TOOL_CALLS))
- fi
-fi
+BENCHMARK_TYPE=$(jq -r '.benchmark.type' "${REPORT_FILE}")
+
+case "${BENCHMARK_TYPE}" in
+ baseline)
+ case "${STATUS}" in
+ pass|fail|skip) ;;
+ *)
+ echo "ERROR: baseline steps must use pass, fail, or skip"
+ exit 1
+ ;;
+ esac
+ ;;
+ agent|agent-browser)
+ case "${STATUS}" in
+ answer|fail|skip) ;;
+ *)
+ echo "ERROR: agent steps must use answer, fail, or skip"
+ exit 1
+ ;;
+ esac
+ ;;
+esac
-# Calculate cost (only meaningful for agent runs with token data)
+# Optional token metadata for runners that can provide trustworthy numbers.
COST=0
if [[ ${TOTAL_TOKENS} -gt 0 ]]; then
- MODEL=$(jq -r '.benchmark.model' "${REPORT_FILE}")
+ MODEL=$(jq -r '.benchmark.model // "unknown"' "${REPORT_FILE}")
- # Cost per 1M tokens
case "${MODEL}" in
*haiku*) INPUT_RATE=0.25; OUTPUT_RATE=1.25 ;;
*sonnet*) INPUT_RATE=3.0; OUTPUT_RATE=15.0 ;;
@@ -130,6 +196,20 @@ if [[ ${TOTAL_TOKENS} -gt 0 ]]; then
COST=$(echo "scale=6; (${INPUT_TOKENS} / 1000000 * ${INPUT_RATE}) + (${OUTPUT_TOKENS} / 1000000 * ${OUTPUT_RATE})" | bc)
fi
+STEP_TOOL_CALLS=0
+if [[ -n "${TOOL_CALLS}" ]]; then
+ STEP_TOOL_CALLS="${TOOL_CALLS}"
+elif [[ "${BENCHMARK_TYPE}" == "agent-browser" ]]; then
+ CURRENT_TOOL_CALLS=0
+ PREV_TOOL_CALLS=$(jq -r '.totals.tool_calls // 0' "${REPORT_FILE}")
+ if [[ -f "${AGENT_BROWSER_LOG}" ]]; then
+ CURRENT_TOOL_CALLS=$(jq -Rsc 'split("\n") | map(select(length > 0) | try fromjson catch empty) | length' "${AGENT_BROWSER_LOG}")
+ fi
+ if [[ "${CURRENT_TOOL_CALLS}" -ge "${PREV_TOOL_CALLS}" ]]; then
+ STEP_TOOL_CALLS=$((CURRENT_TOOL_CALLS - PREV_TOOL_CALLS))
+ fi
+fi
+
# Create step entry
STEP_JSON=$(jq -n \
--argjson group "${GROUP}" \
@@ -142,13 +222,19 @@ STEP_JSON=$(jq -n \
--argjson tool_calls "${STEP_TOOL_CALLS}" \
--argjson cost "${COST}" \
--argjson bytes "${RESPONSE_BYTES}" \
+ --arg answer "${ANSWER}" \
--arg notes "${NOTES}" \
--arg ts "${TIMESTAMP}" \
'{group: $group, step: $step, id: $id, status: $status,
input_tokens: $in_tokens, output_tokens: $out_tokens,
- total_tokens: $total_tokens, tool_calls: $tool_calls,
+ total_tokens: $total_tokens,
+ tool_calls: $tool_calls,
cost_usd: $cost,
- response_bytes: $bytes, notes: $notes, timestamp: $ts}')
+ response_bytes: $bytes, answer: $answer,
+ notes: $notes, timestamp: $ts} |
+ if $status == "answer" then
+ .verification = {status: "pending"}
+ else . end')
# Append to report and update totals
TMP_FILE=$(mktemp)
@@ -157,17 +243,20 @@ jq --argjson step "${STEP_JSON}" \
--argjson out "${OUTPUT_TOKENS}" \
--argjson tool_calls "${STEP_TOOL_CALLS}" \
--argjson cost "${COST}" \
- --arg status "${STATUS}" \
- '(.totals.tool_calls //= 0) |
- .steps += [$step] |
- .totals.input_tokens += $in |
- .totals.output_tokens += $out |
- .totals.total_tokens += ($in + $out) |
- .totals.tool_calls += $tool_calls |
- .totals.estimated_cost_usd += $cost |
- if $status == "pass" then .totals.steps_passed += 1
- elif $status == "fail" then .totals.steps_failed += 1
- else .totals.steps_skipped += 1 end' \
+ '.steps += [$step] |
+ .totals.input_tokens = ((.totals.input_tokens // 0) + $in) |
+ .totals.output_tokens = ((.totals.output_tokens // 0) + $out) |
+ .totals.total_tokens = ((.totals.total_tokens // 0) + $in + $out) |
+ .totals.tool_calls = ((.totals.tool_calls // 0) + $tool_calls) |
+ .totals.estimated_cost_usd = ((.totals.estimated_cost_usd // 0) + $cost) |
+ .totals.steps_passed = ([.steps[] | select(.status == "pass")] | length) |
+ .totals.steps_failed = ([.steps[] | select(.status == "fail")] | length) |
+ .totals.steps_skipped = ([.steps[] | select(.status == "skip")] | length) |
+ .totals.steps_answered = ([.steps[] | select(.status == "answer")] | length) |
+ .totals.steps_verified_passed = ([.steps[] | select(.status == "answer" and ((.verification.status // "") == "pass"))] | length) |
+ .totals.steps_verified_failed = ([.steps[] | select(.status == "answer" and ((.verification.status // "") == "fail"))] | length) |
+ .totals.steps_verified_skipped = ([.steps[] | select(.status == "answer" and ((.verification.status // "") == "skip"))] | length) |
+ .totals.steps_pending_verification = ([.steps[] | select(.status == "answer" and ((.verification.status // "pending") == "pending"))] | length)' \
"${REPORT_FILE}" > "${TMP_FILE}"
mv "${TMP_FILE}" "${REPORT_FILE}"
@@ -177,4 +266,4 @@ if [[ "${STATUS}" == "fail" ]]; then
echo "[${TIMESTAMP}] Step ${GROUP}.${STEP} FAILED: ${NOTES}" >> "${RESULTS_DIR}/errors.log"
fi
-echo "Recorded: Step ${GROUP}.${STEP} = ${STATUS} (tokens: ${TOTAL_TOKENS}, bytes: ${RESPONSE_BYTES}, \$${COST})"
+echo "Recorded: Step ${GROUP}.${STEP} = ${STATUS} (bytes: ${RESPONSE_BYTES}, tool_calls: ${STEP_TOOL_CALLS})"
diff --git a/tests/benchmark/scripts/run-agent-browser-benchmark.sh b/tests/benchmark/scripts/run-agent-browser-benchmark.sh
index 11e8e586..3de99c43 100755
--- a/tests/benchmark/scripts/run-agent-browser-benchmark.sh
+++ b/tests/benchmark/scripts/run-agent-browser-benchmark.sh
@@ -8,6 +8,7 @@ set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
BENCH_DIR="${SCRIPT_DIR}/.."
RESULTS_DIR="${BENCH_DIR}/results"
+CURRENT_REPORT_PTR="${RESULTS_DIR}/current_agent_browser_report.txt"
mkdir -p "${RESULTS_DIR}"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
@@ -56,12 +57,15 @@ cat > "${REPORT_FILE}" << EOF
}
EOF
+printf '%s\n' "${REPORT_FILE}" > "${CURRENT_REPORT_PTR}"
+
echo "Initialized agent-browser benchmark report:"
echo " ${REPORT_FILE}"
echo ""
echo "Next steps:"
-echo " 1. Read ../../skills/agent-browser/SKILL.md"
-echo " 2. Use ./scripts/ab ... to drive agent-browser inside Docker"
-echo " 3. Record each completed benchmark step with:"
-echo " ./scripts/record-step.sh --type agent-browser <group> <step> <pass|fail|skip> --tokens <in> <out> \"notes\""
-echo " 4. Summarize the report with ./scripts/finalize-report.sh"
+echo " 1. Read ./AGENT_BROWSER_INSTRUCTIONS.md"
+echo " 2. Load the live CLI skill with ./scripts/ab skills get agent-browser --full"
+echo " 3. Use ./scripts/ab ... to drive agent-browser inside Docker"
+echo " 4. Record each completed benchmark step with:"
+echo " ./scripts/record-step.sh --type agent-browser <group> <step> <pass|fail|skip|answer> \"what you saw\" \"notes\""
+echo " 5. Summarize the report with ./scripts/finalize-report.sh"
diff --git a/tests/benchmark/scripts/run-optimization.sh b/tests/benchmark/scripts/run-optimization.sh
index caf512d3..41d5ef99 100755
--- a/tests/benchmark/scripts/run-optimization.sh
+++ b/tests/benchmark/scripts/run-optimization.sh
@@ -10,6 +10,8 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
RESULTS_DIR="${SCRIPT_DIR}/../results"
mkdir -p "${RESULTS_DIR}"
LOG_FILE="${RESULTS_DIR}/optimization_log.md"
+CURRENT_BASELINE_PTR="${RESULTS_DIR}/current_baseline_report.txt"
+CURRENT_AGENT_PTR="${RESULTS_DIR}/current_agent_report.txt"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
RUN_NUMBER=$(grep -c "^## Run #" "${LOG_FILE}" 2>/dev/null || echo 0)
RUN_NUMBER=$((RUN_NUMBER + 1))
@@ -82,6 +84,9 @@ cat > "${AGENT_REPORT}" << EOF
}
EOF
+printf '%s\n' "${BASELINE_REPORT}" > "${CURRENT_BASELINE_PTR}"
+printf '%s\n' "${AGENT_REPORT}" > "${CURRENT_AGENT_PTR}"
+
# Clear previous agent commands log
rm -f "${RESULTS_DIR}/agent_commands.log"
diff --git a/tests/benchmark/scripts/verify-step.sh b/tests/benchmark/scripts/verify-step.sh
new file mode 100755
index 00000000..1e1b9dd3
--- /dev/null
+++ b/tests/benchmark/scripts/verify-step.sh
@@ -0,0 +1,122 @@
+#!/usr/bin/env bash
+#
+# Record a deferred verification decision for a previously observed benchmark
+# step.
+#
+# Usage:
+# ./verify-step.sh [--type agent|agent-browser] [--report-file <path>] <group> <step> <pass|fail|skip> "notes"
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+RESULTS_DIR="${SCRIPT_DIR}/../results"
+CURRENT_AGENT_PTR="${RESULTS_DIR}/current_agent_report.txt"
+CURRENT_AGENT_BROWSER_PTR="${RESULTS_DIR}/current_agent_browser_report.txt"
+
+REPORT_TYPE=""
+REPORT_FILE=""
+
+while [[ $# -gt 0 && "$1" == --* ]]; do
+ case "$1" in
+ --type)
+ REPORT_TYPE="$2"
+ shift 2
+ ;;
+ --report-file)
+ REPORT_FILE="$2"
+ shift 2
+ ;;
+ *)
+ echo "Unknown option: $1"
+ exit 1
+ ;;
+ esac
+done
+
+if [[ $# -lt 3 ]]; then
+ echo "Usage: $0 [--type agent|agent-browser] [--report-file <path>] <group> <step> <pass|fail|skip> [notes]"
+ exit 1
+fi
+
+GROUP="$1"
+STEP="$2"
+STATUS="$3"
+NOTES="${4:-}"
+STEP_ID="${GROUP}.${STEP}"
+TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ)
+
+case "${STATUS}" in
+ pass|fail|skip) ;;
+ *)
+ echo "ERROR: verification status must be one of pass, fail, skip"
+ exit 1
+ ;;
+esac
+
+resolve_current_report() {
+ local ptr="$1"
+ if [[ -f "${ptr}" ]]; then
+ tr -d '[:space:]' < "${ptr}"
+ return 0
+ fi
+ return 1
+}
+
+if [[ -z "${REPORT_FILE}" ]]; then
+ if [[ -n "${REPORT_TYPE}" ]]; then
+ case "${REPORT_TYPE}" in
+ agent)
+ REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_PTR}" || true)"
+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_benchmark_*.json 2>/dev/null | head -1)
+ ;;
+ agent-browser|agent_browser)
+ REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_BROWSER_PTR}" || true)"
+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_browser_benchmark_*.json 2>/dev/null | head -1)
+ ;;
+ *)
+ echo "ERROR: --type must be 'agent' or 'agent-browser'"
+ exit 1
+ ;;
+ esac
+ else
+ REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_BROWSER_PTR}" || true)"
+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE="$(resolve_current_report "${CURRENT_AGENT_PTR}" || true)"
+ [[ -n "${REPORT_FILE}" ]] || REPORT_FILE=$(ls -t "${RESULTS_DIR}"/agent_browser_benchmark_*.json "${RESULTS_DIR}"/agent_benchmark_*.json 2>/dev/null | head -1)
+ fi
+fi
+
+if [[ -z "${REPORT_FILE:-}" || ! -f "${REPORT_FILE}" ]]; then
+ echo "ERROR: no agent benchmark report found"
+ exit 1
+fi
+
+TMP_FILE=$(mktemp)
+jq --arg id "${STEP_ID}" \
+ --arg status "${STATUS}" \
+ --arg notes "${NOTES}" \
+ --arg ts "${TIMESTAMP}" \
+ '
+ if any(.steps[]; .id == $id) | not then
+ error("step not found: " + $id)
+ elif any(.steps[]; .id == $id and .status == "answer") | not then
+ error("step is not answer-status and cannot be verified: " + $id)
+ else
+ .
+ end |
+ .steps |= map(
+ if .id == $id then
+ .verification = {
+ status: $status,
+ notes: $notes,
+ timestamp: $ts
+ }
+ else . end
+ ) |
+ .totals.steps_verified_passed = ([.steps[] | select(.status == "answer" and ((.verification.status // "") == "pass"))] | length) |
+ .totals.steps_verified_failed = ([.steps[] | select(.status == "answer" and ((.verification.status // "") == "fail"))] | length) |
+ .totals.steps_verified_skipped = ([.steps[] | select(.status == "answer" and ((.verification.status // "") == "skip"))] | length) |
+ .totals.steps_pending_verification = ([.steps[] | select(.status == "answer" and ((.verification.status // "pending") == "pending"))] | length)
+ ' "${REPORT_FILE}" > "${TMP_FILE}"
+
+mv "${TMP_FILE}" "${REPORT_FILE}"
+echo "Verified: Step ${STEP_ID} = ${STATUS}"