Skip to content

armgit / axuse

Public Imported from GitHub
0

Accessibility-first browser & computer use for AI agents — one MCP server for every harness (Codex, Claude Code, OpenCode, Antigravity). Instruments the DOM/accessibility tree via CDP instead of screenshot loops: diffed snapshots, stable element refs, trusted input, typed errors. Zero dependencies

5.9 KB
Raw Blame

axuse benchmark suite v2

A head-to-head benchmark of browser-use tool layers for coding-agent harnesses. Same harness (Codex CLI), same model (gpt-5.6-luna, reasoning medium), same 16 tasks, same machine. The only variable is the browser layer:

arm what the model drives
codex-chrome Codex's own browser use: the Codex Chrome extension in the user's real Chrome (Codex's built-in plugin + skill)
axuse the axuse MCP server (its own headless Chromium)
browser-use browser-use 0.13 CLI + its official SKILL.md, attached to a dedicated headless Chrome via BU_CDP_URL
playwright Microsoft's @playwright/mcp (pinned local install via --playwright-cli), headless Chrome, isolated profile, --caps vision

Tasks

Deterministic, per-run seeded pages (server.mjs) covering what real browser agents trip over:

id capability pass condition (server-side)
t1 click → reveal claim endpoint hit + code reported
t2 form fill (text, email, select, checkbox) exact values submitted
t3 150-row table lookup price + stock reported
t4 4-step wizard with server validation all steps + token
t5 cross-origin iframe (OOPIF) click inside iframe
t6 nested shadow DOM switches + radios exact settings saved
t7 infinite scroll feed (300 posts, lazy pages) exactly the target post liked
t8 hover-only menu export clicked, never "Delete account"
t9 confirm() → prompt() dialog chain only the target file deleted, reason "cleanup"
t10 target=_blank new tab round trip help tab visited + code verified
t11 async job (5 s) then download download after ready
t12 pointer-events drag & drop (kanban) card in Done
t13 contenteditable editor exact text published
t14 ARIA autocomplete combobox + custom listbox Zurich / Switzerland saved
t15 debounced search + pagination right order's details opened
t16 canvas (no DOM) — needs vision click inside the red circle

Fairness and anti-cheat

  • Public HTTPS fixtures. The Codex extension refuses localhost / private IPs (ERR_BLOCKED_BY_CLIENT), so the fixtures are served through two Cloudflare quick tunnels (two origins → the t5 iframe is genuinely cross-origin). Every arm loads the same public URLs.
  • Server-side scoring. Every state change POSTs to the fixture server; success = server ground truth and the reported value. Ground truth is only readable with a random bench key.
  • Browser-only endpoints. Scored endpoints require Sec-Fetch-* headers; the runner also audits every tool call and disqualifies runs that hit the task host with a shell HTTP client.
  • Isolation. Each run gets an empty working directory outside the repo (no peeking at the fixture source). The axuse and browser-use arms run with Codex's browser/computer-use features and plugins disabled; the codex-chrome and browser-use arms run with axuse disabled; any call into another arm's browser layer disqualifies the run.
  • Same instructions. All arms get the same task text plus the same "unattended, pre-approved, no HTTP clients" rules; each arm gets one line naming its tools (browser-use additionally gets its official SKILL.md, the content Codex would load when that skill triggers).
  • Frozen subject. The axuse arm runs a frozen copy (--axuse-cli), so development edits cannot leak into a benchmark in flight.
  • Order rotation. Arm order rotates per task and repetition.

Metrics

From the Codex session log of each run: wall time, turns, tool calls, and thread token totals (total, and fresh = uncached input + output). capability success excludes runs where the agent stopped to ask the absent user for confirmation (a safety-policy outcome — reported separately, never hidden).

Reproduce

# once: browser-use CLI + its skill text, cloudflared
uv tool install browser-use==0.13.11
browser-use skill show > bench/suite/browser-use-SKILL.md
winget install Cloudflare.cloudflared

# sanity: every task is solvable and scoring is correct (scripted, no LLM)
node bench/suite/oracle.mjs

# the benchmark (freeze axuse first)
cp -r src package.json bridge /path/to/axuse-frozen/
node bench/suite/run.mjs --reps 3 --tag final --axuse-cli /path/to/axuse-frozen/src/cli.js
node bench/suite/summarize.mjs bench/suite/results/final.jsonl

# portability: the same axuse server in other harnesses
node bench/suite/portability.mjs --harness claude,opencode

Results land in results/*.jsonl (one line per run, with the session file path for audit).

Comparing against another MCP browser tool, in a clean agent home

# pinned Playwright MCP, installed once
mkdir -p ~/.axbench-tmp/pw-mcp && (cd ~/.axbench-tmp/pw-mcp && npm init -y && npm i @playwright/[email protected])

# a Codex home with the same login but none of Codex's bundled browser /
# computer-use plugins (the --disable flags do not remove its built-in `cua`
# and `node_repl` servers, so without this an arm can still reach another layer)
mkdir -p ~/.axbench-tmp/codex-home && cp ~/.codex/auth.json ~/.axbench-tmp/codex-home/ && : > ~/.axbench-tmp/codex-home/config.toml

node bench/suite/run.mjs --arms axuse,playwright --reps 3 --tag compare   --codex-home ~/.axbench-tmp/codex-home   --axuse-cli /path/to/axuse-frozen/src/cli.js   --playwright-cli ~/.axbench-tmp/pw-mcp/node_modules/@playwright/mcp/cli.js
node bench/suite/summarize.mjs bench/suite/results/compare.jsonl
node bench/suite/rescore.mjs bench/suite/results/compare.jsonl   # strict vs "touched the task site" rule

auth.json holds your login: delete the benchmark home when you are done. A run that produces no agent session prints the agent's own error (NO-SESSION) instead of a bare FAIL. The result of one such comparison, with its caveats, is in PLAYWRIGHT-COMPARISON.md.