Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

22.5 KB
Raw Blame

Website tests

This self-contained suite lives in the repository's root-level website-tests/ directory. Its .env, account configuration, harness, scenarios, tests, guide, profiles and report artifacts live here.

The opt-in runner tests WebBrain using an explicitly selected local browser or WebBrain Cloud session. Account A performs the task; account B independently observes receipts or shared access. B may use a different backend and needs no WebBrain extension/provider when it is a local DOM observer.

The harness changes no product guards. Local tasks enter the installed extension's normal chat_start/continue_start protocol; Cloud tasks use the documented run API. Helpers only prepare observation, navigate an explicitly selected observer, read evidence, or perform configured human interventions.

The directory is organized as follows (private files and generated directories are ignored):

website-tests/
  .env                         # private settings
  .env.example                 # tracked placeholders
  .env.authenticated.json      # private accounts and mutation scope
  config.example.json          # tracked configuration template
  package.json                 # commands for this suite
  README.md
  cli.mjs, config.mjs, ...      # harness, scenarios and harness tests
  backends/                    # local browser and Cloud adapters
  artifacts/                   # private reports, ledgers and fixture evidence
  profiles/                    # optional dedicated logged-in test profiles

Install and configure

Use Node 22+ and install the repository's existing Playwright dependencies once. From the repository root:

npm install
npx playwright install chromium
Set-Location website-tests
if (-not (Test-Path -LiteralPath .env)) { Copy-Item .env.example .env }
if (-not (Test-Path -LiteralPath .env.authenticated.json)) { Copy-Item config.example.json .env.authenticated.json }

Both populated files are ignored by this directory's .gitignore. The current worktree already has placeholder copies ready to edit. Set backend/session/provider choices in .env, and account identities, exact recipients, test folders, and permitted actions in .env.authenticated.json. Credentials in JSON must use env:NAME references. Process environment values take precedence over .env values.

Leave WBTEST_TARGET=fixtures for deterministic development. A default run executes only the owned installed-extension file-transfer fixture. It neither selects a logged-in browser nor sends anything to a real website.

For live X, for example, set WBTEST_TARGET=live, WBTEST_SCENARIOS=x-post-lifecycle, WBTEST_LIVE_MUTATIONS=true, and WBTEST_ALLOWED_ACTIONS=compose,edit,upload,post,navigation,delete. Replace the X account placeholder in private JSON. Authorization covers exactly the chosen account and uniquely marked disposable resources; it is not an account-wide permission. Once configured, the harness does not repeatedly ask for that grant. WebBrain's own host/capability permissions still apply: grant the required test host/actions through its normal UI during setup. Pending approvals are reported as pending input; the harness never disables or automatically approves them.

allowedRecipients must contain the actual delivery identity: email address, E.164 phone, or Telegram peer ID. A contact display label alone is insufficient. allowedDestinations must contain the exact disposable Drive folder URL.

Commands

Run these from website-tests/. All relative configuration/report/profile paths resolve from this directory.

npm run cli -- help
npm run list -- --target live
npm run dry-run -- --target live --sites x,gmail
npm run setup -- --role A
npm run setup -- --role B
npm run doctor -- --role A
npm run doctor -- --role B
npm run run -- --target live --scenario x-post-lifecycle
npm run run -- --target fixtures --scenario fixture-shared-guards
npm run summarize -- --report artifacts/RUN_DIRECTORY
npm run reconcile -- --report artifacts/RUN_DIRECTORY
npm run cleanup -- --report artifacts/RUN_DIRECTORY

Use --env FILE/--config FILE to select another private configuration in this directory. The repository-root aliases npm run authenticated -- COMMAND, npm run test:authenticated and npm run test:authenticated:installed delegate to this suite and use the same .env and artifact paths. --backend local|cloud overrides the role selected by --role (A by default); configure B independently. --sites or --scenario select cases without rewriting scripts. Runs are serial because accounts, drafts, and conversations are shared mutable state. Repetitions, per-request/run time, polling and run count are bounded. The suite deadline stops new dispatches; an in-flight bounded observation/request or owned-run Stop may finish afterward. Cloud polling timeouts do not imply that the remote run has stopped: reconcile that existing run before further work. Exit codes are 0 for verified success, 1 for failure, and 2 for blocked, unsupported or pending-input outcomes. A dry run performs no network requests.

Local browsers and manual login

Set WBTEST_ACTOR_PROFILE=profiles/actor and WBTEST_OBSERVER_PROFILE=profiles/observer for ignored dedicated profiles here, or set WBTEST_ACTOR_CDP_URL to an explicitly enabled debugging endpoint. Do not configure both. A dedicated profile retains logins between runs. Setup opens WebBrain settings and configured sites; log in manually, including QR, OTP and MFA, and press Enter in the terminal when finished. No password/login automation or challenge bypass is implemented.

An arbitrary already-open personal browser cannot always be attached. Chrome 136+ restricts debugging its default user-data directory; use a dedicated non-default profile with debugging intentionally enabled. See Chrome's debugging changes. For automated unpacked loading, use Playwright's bundled Chromium: current branded Chrome/Edge have removed the needed side-loading switches. See Playwright extension guidance. WBTEST_ACTOR_EXECUTABLE is available for compatible explicitly chosen builds; unsupported loading is reported rather than bypassed.

The runner refuses standard Chrome/Edge profile directory launches and does not copy cookies, delete profile locks, restart unrelated browsers or change security settings. Dedicated profiles launched by the runner close on exit; CDP sessions detach and existing pages stay open. Remote CDP requires an explicit trustRemoteCdp:true private role configuration. CDP endpoints must not contain embedded credentials.

Doctor reports actual installed version, selected provider/model and a runtime fingerprint of four named installed files. This fingerprint is not a Git commit attestation or a claim that every extension file was hashed. Local expectedVersion pins the extension version; expectedBuild pins that reported fingerprint. A DOM-only observer reports no extension build.

Account identity must be visible or mechanically readable from exact native account UI. For WhatsApp/Telegram, expose your own profile phone and the intended peer details before preflight. Operator-reviewed identity/recipient selector bindings can accommodate another UI layout; their values still must match. Missing identity metadata blocks the case instead of guessing the account.

WebBrain Cloud

Set A's backend to cloud, WEBBRAIN_API_KEY in the private environment, and WBTEST_ACTOR_SESSION_ID to the exact normal persistent session. B can use a local observer or independently configured Cloud credentials/session. No arbitrary ready session is selected.

To explicitly create a named persistent test session:

npm run setup -- --backend cloud --role A --name webbrain-qa-actor

Omit A's session ID only for that explicit setup command. Selection is saved in session-A.private.json in private artifacts; put its ID in .env. Setup and close preserve the persistent session. Sessions and runs are billable; limits apply, and ambiguous creation is never retried automatically. If setup loses its response, the private setup ledger retains its request ID/name/time. Select the matching session from your Cloud dashboard, then reconcile it without creating another session:

npm run reconcile-setup -- --backend cloud --role A --request-id SAVED_REQUEST_ID --session-id CONFIRMED_SESSION_ID

This checks the original name, creation time and normal profile. Run doctor with that explicitly configured session afterward. Both selected and harness-created persistent sessions remain available; session deletion is an explicit operator action through the Cloud dashboard. Workflow cleanup never deletes a session.

Readiness requires both status=ready and runtime_ready=true. The public API does not attest a runtime build/version; doctor reports unknown and configured strict Cloud build pins block. Existing-session provider/model selection and introspection are not exposed here: configure them through normal Cloud browser Settings and leave the role provider/model pins empty. Requested Cloud pins report unsupported rather than being silently ignored. Repository HEAD is never reported as the Cloud build. Configure manualActorIdentityVerified:true only after checking A's actual login in the chosen browser; the string "false" is not confirmation.

Cloud REST has no documented DOM/control endpoint. A separate logged-in local B can independently prove incoming mail/messages and shared access in the separate receiver stages. Every A stage requires explicit actor-tab manual evidence, including sends, replies and cleanup, so captured message/thread IDs belong to A's account. Without it, preflight reports unsupported before dispatch. Automated active steering/checkpoints are also unsupported on REST alone. Finished-run /messages continuation is not active-run steering.

For manual verification set manualEvidenceFile to an ignored private JSON file and run in an interactive terminal. A noninteractive CLI with no manual observer callback is blocked before any actor task. Each checkpoint writes a private requirement file with the exact run marker, stage, expected body/account/resource. Observe the actual browser and append evidence to { "observations": [...] }. Rows use the same webbrain-authenticated-evidence/1 schema as DOM observations, source:"manual-confirmed-private", confirmedByOperator:true, a timestamp in confirmedAt, the exact site, stage, marker, url, account identity:{matched:true,actual:...}, and phase-specific evidence. Composer proof needs one exact marked unsent composer; item proof needs exact body, author, recipient and canonical ID; deletion needs exact-resource removal proof. A placeholder-only template is in manual-evidence.example.json. The schema and graders are in sites.mjs and scenarios.mjs. Model output is not manual evidence. Reports identify this verification layer.

Cloud upload verifies dummy-file bytes and uses a browser path only when browser_ready is exactly true. Downloads are verified by independent bytes and SHA-256, not by filenames or the agent's claimed checksum.

Scenario matrix

Scenario Actions and independent checks Known boundaries
x-post-lifecycle Compose, atomic replacement, saved draft, PNG, post, refresh, exact deletion Saved state needs native evidence; live selectors unverified
bluesky-post-lifecycle Compose/replace, PNG, publish, refresh, exact deletion Saved drafts not claimed for the current web client
mastodon-post-lifecycle Compose/replace, PNG, publish, refresh, exact deletion Explicit instance; current account setting controls confirmation/direct delete
whatsapp-conversation Bound contact, exact text/file, B incoming receipts, individual cleanup Positive own-account/peer identity and stable message IDs required
telegram-conversation Bound peer, exact text/file, B incoming receipts Row disappearance cannot prove deletion; cleanup stays pending without exact proof
gmail-test-thread Draft/replacement/reopen, attachment, B receipt, reply, forward, individual Trash Full From/To and immutable message/thread IDs required
protonmail-test-thread Same test-thread lifecycle through native UI UI/editor layout may require explicit selector bindings
drive-file-lifecycle Own folder, upload, rename, Viewer share, B access, A download/hash, exact Trash B download is not claimed; inaccessible file/resource bytes block verification

All live selectors are provisional until exercised against the selected real accounts. The driver never swaps a live failure for a fixture success. An unsupported feature is not an empty successful test.

DOM observers open only an exact marked mail subject and expose its own header details. They refuse wrong accounts, multiple candidates and prefix matches; they do not click send/delete controls. {marker} in an explicit observer URL is substituted for the current unique run marker, useful for a marked search.

Monitoring and recovery

Configure interventions by scenario-id:stage-id in private JSON. Each needs a selector/state checkpoint and the explicit operator action. For example:

{
  "interventions": {
    "x-post-lifecycle:replace-body": {
      "kind": "scroll",
      "checkpoint": { "selector": "[data-testid=tweetTextarea_0]", "state": "visible" },
      "selector": "[data-testid=tweetTextarea_0]"
    }
  }
}

This example is a configuration pattern, not a verified current X selector. Available actions include native click/type/scroll/selection/navigation, draft changes, steering/narrowing, Stop, draft-only Stop/Continue, upload cancel through its explicit cancel control, and waiting for a prearranged passive update via updateCheckpoint. No arbitrary timed sleep triggers the intervention. Steer/narrow/cancel needs an explicit expectedBody containing {marker}. Corrections update the body expected by the paired receipt. Cancellation is limited to an unsent compose/edit stage, with expectedPhase:"composer" and finishAfterStage:true; consequential stages cannot be downgraded to draft success. Post-publication correction/replacement is covered by the owned shared-guard fixture; a general live replacement intervention is not claimed. Recorded checkpoints show when input occurred; an unchanged expected-body workflow must still end with the originally authorized content.

Descriptors from list document broader recovery scenarios. Provider/observation fault injection and uncertain post-dispatch recovery are exercised in owned fixtures, not injected into personal-account traffic. A configured Cloud intervention without a controller blocks before starting a run. Post-dispatch Stop/Continue remains a fixture case; live automatic Continue is restricted to unsent draft stages.

Reconciliation, cleanup and privacy

Every attempt is written before dispatch. Independently owned canonical IDs remain journaled when attachment quality or duplicate-effect checks fail. IDs and exact independently observed resource ownership are saved immediately. A lost acknowledgement, failed proof, or unfinished cleanup leaves a pending ledger and blocks another suite for the same configured scope. Shared private execution locks also bind the selected profile/CDP endpoint/Cloud session independently of permissions, account scope, worktree and artifact directory. A changed allowlist cannot acquire the same browser concurrently. Keep the original configuration for recovery. Session locks live in the primary checkout's website-tests/artifacts/.env.session-locks/ so linked worktrees also coordinate pending effects; individual reports stay in this checkout's website-tests/artifacts/. The suite writes no durable session-lock files to OS temporary storage. Do not solve that condition by rerunning or changing the account configuration.

Inspect the report and run reconcile. If a Cloud creation response was lost, select the matching dashboard run explicitly:

npm run reconcile -- --report RUN_DIRECTORY --request-id RECORDED_REQUEST_ID --run-id CONFIRMED_RUN_ID

Local shutdown saves exact terminal extension receipts in the private ledger before closing actor tabs. Recovery can use those receipts after the live tab journal has been deleted, and still requires independent effect evidence. If a receipt cannot be saved or the run is still active, shutdown retains the browser and its tabs for reconciliation.

Reconciliation reads existing runs/effects and never resends. An ambiguous Cloud upload can be reconciled by its recorded byte hash and exact server filename; use --remote-name CONFIRMED_SERVER_FILENAME if its upload response was lost. Cleanup operates only on recorded immutable IDs, markers, accounts and conversation/thread scope. It cannot empty an inbox/chat/folder, delete by position, or touch pre-existing data. Missing ownership/absence proof leaves visible pending cleanup.

Artifacts default to ignored artifacts/. Summary JSON/Markdown contain statuses, stages, runtime/observation timings, dispatch counts, and content-free diagnostic events where the runtime exposes them; failure categories distinguish authentication/authorization, setup/backend, verification/site layout and behavioral evidence. A missing selector is not conclusive proof of a product regression. Reports contain no task bodies, credentials or unrelated messages. Private ledger/checkpoint/fixture logs have restricted file modes where the OS supports them. Windows directory ACLs still govern access. Raw live-page screenshots/traces are not automatically saved because they may include unrelated private content. The backend's explicit trace-export method is available to a caller handling those private artifacts deliberately.

Verification

npm test
npm run test:installed
npm run run -- --target fixtures --scenario fixture-file-transfer
npm run run -- --target fixtures --scenario fixture-stop-continue-recovery

Deterministic tests cover configuration/scope, ownership, ambiguity, strict Cloud readiness, bounded polling, journals, continuation, checksums, privacy and cleanup. DOM fixtures test the eight real observers and the complete planner/runner/grader pipeline. Installed smoke exercises normal MV3 Ask through an actual accessibility read and proves that CDP detachment preserves Chrome. Installed fixture packs use scripted loopback providers and owned websites; they do not certify live accounts. Doctor separately reports local native account-identity checks for the selected sites. Cloud doctor cannot inspect login identities and reports the manual observation requirement. Noninteractive local setup returns pending input. Live cases are intentionally absent from both this directory's npm test and the repository's ordinary test/CI runs.

Results verified during implementation

After relocation, the root alias passed 230 harness/layout tests with zero failures (one opt-in smoke skipped). Running npm run test:installed inside this directory passed all 34 checks, and npm run run -- --target fixtures --scenario fixture-file-transfer passed with its report under artifacts/.

Before the directory move on 2026-10-12, the harness test command passed 226 tests with zero failures and one opt-in installed smoke skipped. The installed check command passed all 34 checks, including that real smoke. The six CLI fixture packs listed below passed. The installed extension reported version 40.4.1 and the four-file runtime fingerprint e516ebbc51e7780e0da851348b35539253ceb4df88343fe8ba7c7c830dab6581. Repository source revision was 6c451849f5bff6be88a92c3fd442375537ad8f84; harness changes were uncommitted.

Path Evidence Status
Harness configuration, backends, journals, grading and recovery Deterministic tests including mocked documented Cloud responses Verified locally
Eight website observers and planner/runner integration Owned Chromium DOM fixtures; seven complete cleanups, Telegram intentionally pending Verified fixture behavior
Installed local WebBrain Real MV3 Ask, real accessibility read, installed four-file fingerprint, CDP detach preserving Chrome Verified locally, version 40.4.1
Installed shared guards, chat intervention, Stop/Continue, transfers, Mastodon confirmation/direct delete Six existing owned MV3 fixture packs through CLI Verified locally
Authenticated local website flows Needs explicitly selected logged-in test accounts and scopes Unverified; no external mutations made
Deployed Cloud flows, mixed actor/observer accounts Needs API keys, explicit ready persistent sessions and manual actor evidence Unverified; API contract tested with mocks
Firefox attachment, REST-only active intervention, opaque mail iframes Not supported by this driver Unsupported

Example sanitized report shape (illustrative, not live-site evidence):

{
  "schema": "authenticated-summary/1",
  "target": "live",
  "counts": { "passed": 0, "failed": 0, "blocked": 0, "unsupported": 1, "needs_user_input": 0 },
  "results": [{ "scenario": "x-post-lifecycle", "backend": "cloud", "status": "unsupported", "stage": "preflight", "code": "cloud_private_phase_requires_manual_evidence", "cleanup": "not_started" }]
}

Observer copies of received mail/messages are read-only in this initial suite. Actor cleanup targets its individually recorded test items; receiver-side cleanup needs a separately authorized WebBrain actor/controller and is not claimed. Telegram does not expose dependable exact-message deletion proof to this observer, so a row disappearing leaves cleanup pending rather than passing. Drive uses an existing explicitly allowlisted test folder, renames the created file, and verifies its bytes; creating folders or editing document contents is not part of this initial file scenario.

Adding a website

Add one site definition/read-only identity and evidence extraction in sites.mjs, and concrete natural-language stages in scenarios.mjs. Give destructive stages a captured immutable resource key. Add owned DOM examples in sites.test.mjs and a complete pipeline example in runner-integration.test.mjs; then add placeholders and a row to this matrix. Re-use these two execution adapters and the shared repository ../ci/lib/webbrain-client.mjs. Owned installed-extension fixture packs remain shared under ../test/quality/; the launcher supplies repository source paths and directs their evidence into this suite's ignored artifacts/. A site-specific observer may prepare a read-only view, but it must never perform the task's send/post/share/upload/delete action. Missing identity, native IDs, receipts or browser capabilities must remain explicit blocked/unsupported cases.