Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

8.7 KB
Raw Blame

LLM tiny v2: reasoning-heavy browser evaluation

Independent of test/llm-tiny: this directory has its own copied/pinned native adapters, fixtures, runner, frozen manifest, and ignored results. It imports no code from the original suite. No root package scripts were changed. There is no RunPod controller, provisioning, teardown, scheduling, or process-management code.

Creation and local validation do not call a model endpoint. No v2 model scores exist yet. The existing paid run must finish under its original hard $30 cap; this suite does not extend that approval or budget.

Workload

Held-out category Tasks What differs from basic control tests
Constraint planning 15 Eligibility, exclusions, total cost, whole-package rounding, scheduling and optimization
Calculations 15 Split-page evidence, tax, deductions, weighted rates, invoice deduplication and rounding
Evidence reconciliation 10 Effective dates, source precedence, regional exceptions, revocations and same-name identities
Long workflows 10 Six saved stages, original requirements, distinct quantity amendments, derived totals and draft-only authority
Recovery 5 Changed validation, expired coupon, stock change, stale revision and uncertain commit outcome
Contextual/visual controls 5 Repeated Choose labels, spatial relationships, status and utilization

There are 60 held-out task definitions, each with three reproducible seeded variants (17,43,89): 180 held-out episodes per model. Ten separate pilot task definitions have their own data and scenarios: 30 pilot episodes per model. Seeds change task data and/or layout, not the underlying task definition; they are not 180 independent task types. Full held-out coverage across six models is 1,080 episodes, so budget it before launching anything.

The fixtures are deliberately synthetic and reasoning-heavy. A stronger reasoning model may do better, but there is no preferred winner in scoring and no promised ranking. This does not measure open-web robustness or overall product capability. Workflow pages can be revisited; this tests retaining and reconciling requirements, not a strict inaccessible-memory experiment.

What a pass means

The browser submits values to an isolated, random local session. The Node server owns expected answers and authoritative state. No answer key, hidden expected attributes, evaluator code, or future workflow stages are sent to the browser. The model has no JavaScript execution or unrestricted navigation tool. Browser requests are restricted to that exact session, not arbitrary localhost.

A strict pass requires all expected fields/stages, required recovery events, terminal completion, no forbidden side effects, and completion within limits. Judging uses exact trimmed strings; tasks state decimal formatting explicitly. Wrong submitted answers are terminal and not corrected by the fixture. There is no correctness feedback that allows trial-and-error guessing. “Done” alone never passes. Deletion, publication and duplicate submissions remain sticky failures. The unknown-commit task requires checking the ledger and acknowledging the already-saved request instead of retrying it.

Partial credit is the fraction of individual field/recovery checks met; it is reported separately and never substitutes for a strict pass. UI receipts only confirm that a submission was recorded, not that it was correct.

Native interfaces and fairness

Adapter Models Observation
compass WebBrain Compass Tiny v2 Visible viewport text + accessibility controls
browser-use BU-30B-A3B Preview Visible viewport text + indexed controls + screenshot
fara Fara1.5 4B / 9B / 27B Screenshot only, 1000×1000 action coordinates
openai Any explicitly supplied OpenAI-compatible baseline Visible viewport text + indexed controls + screenshot

All task facts are visibly rendered. DOM observations include table/document text, not just interactive labels, and never include hidden form routing fields. Spatial cards also display row/column and utilization text, making the facts accessible to the text-only participant. Closed select options are enumerated for DOM participants; Fara can open the in-page picker. These native access differences are explicit: this is not an identical-input modality experiment.

Defaults for every participant: 80 model turns, 120 individual attempted actions, 15 minutes per task, 90 seconds per request, 4,096 output tokens per request, latest three screenshots, 1440×900 viewport. All actions in a batch count, and stale DOM handles cannot silently target a new control. Native temperature/top-p and prompts remain adapter-specific and are recorded. No text-only OpenAI override is allowed in this frozen protocol.

Local validation — safe while another suite runs

Use the repository's installed Node 22+ and Playwright/Chromium dependencies. No dependency installation or global browser profile is needed.

node test/llm-tiny-v2/run.mjs --validate-only
node --test --test-concurrency=1 --test-reporter=spec test/llm-tiny-v2/*.test.mjs test/llm-tiny-v2/lib/*.test.mjs

Unit checks cover all 210 variants, positive and negative outcomes, sticky violations, unknown commits, action limits, paired reporting and adapters. Browser checks use a single test process and disposable contexts. They perform golden scripted interactions for all 70 task definitions on seed 17, test native dropdown coordinates, stale references, blocked external navigation, and an offline mocked model loop. Those are harness tests, not model capability results. They do not contact any model endpoint or touch the existing run.

Later model runs — not started automatically

First review the separate pilot, verify remaining authorized budget, and obtain approval for paid execution. A localhost model endpoint may still incur GPU rental costs. --allow-paid is a deliberate launch gate, not a billing cap. This runner cannot enforce a shared hard cloud-spend cap and must not be used without a separately budgeted/protected deployment. It does not start or stop model servers. The example ports are placeholders, not discovered services.

Create a private config from participants.example.json. Keep secrets in environment variables, never in this repository or command arguments. Remove apiKeyEnv only for a deliberately unauthenticated local endpoint. The openai adapter can use another explicitly supplied model instead of DeepSeek; the five specialized models remain allow-listed.

After separate approval and endpoint preparation:

node test/llm-tiny-v2/run.mjs --allow-paid \
  --config /private/tiny-v2-participants.json \
  --split pilot --seeds 17 --tag pilot-01

# Only after pilot review, freezing, and a budget decision:
node test/llm-tiny-v2/run.mjs --allow-paid \
  --config /private/tiny-v2-participants.json \
  --split heldout --tag heldout-01

Results go exclusively under this directory's ignored results/<unique-tag>. Existing tags are rejected. Per-episode JSON retains every returned action, outcome, usage and error; PNGs retain the screenshot observations. Partial summaries are written after every episode, marked incomplete until coverage is complete. Never copy incomplete or cherry-picked runs into a final comparison.

Compare and freeze

node test/llm-tiny-v2/compare.mjs \
  test/llm-tiny-v2/results/heldout-01/deepseek/summary.json \
  test/llm-tiny-v2/results/heldout-01/fara-4b/summary.json

The report prints pass-rate and category tables, latency, attempted actions, reported API cost and paired 95% task-cluster bootstrap intervals. It rejects different manifest hashes, limits, splits or coverage. Variants stay grouped under their task; intervals overlapping zero do not establish a winner. Costs missing from provider usage are null/unknown, never zero; GPU rental and storage are not included. Hosting differences remain separate from accuracy.

manifest.json pins every task variant, adapter/evaluator/runner source hash, default seed and limit. Live runs verify it before any endpoint call. Calibrate on the pilot only. If adapter or corpus changes are warranted, review them, update the suite version and intentionally re-freeze:

node test/llm-tiny-v2/freeze.mjs --write --replace-existing

This never runs automatically. Results under different hashes cannot be ranked together; rerun all affected participants after a genuine harness fix. Do not change held-out tasks or weights because a particular model won or lost.

The separate pilot/holdout and deterministic outcome checks follow the dataset and grader principles in OpenAI's evaluation guide.