Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) đź§ 

16.6 KB
Raw Blame

Social publication contracts

X and Bluesky publication use the agent's selected provider for language understanding. The runtime validates a structured contract and compares app-observed values before dispatch and completion. There is no separate intent model or provider setting.

Intent and authorization

social-publish-contract.js defines the schema and prompts in both browser builds. The agent caches a contract per task, with at most one repair attempt for invalid output. A direct human clarification reply refreshes an unattempted contract, using the original request, task anchor, authentic prior user turns for referenced follow-ups, and the exact question/answer pairs. Timeouts, automatic choices, and model-authored tool results cannot supply these replies. The approved plan, assistant drafts, and unanswered proposals are reference data. Plans, drafts, and page content do not independently authorize publication. The app-observed platform at request start can resolve an omitted destination for an explicit publication command; later navigation cannot change that default or override an explicit destination.

Prior user context includes at most four recent whole turns, within the remaining 120,000-character serialized source budget. Current instructions and clarification pairs take priority. Selection stops at a turn that cannot fit; it never truncates a correction or skips back to an older instruction. Refreshes also bound previously cached history. Oversized current instructions still block compilation rather than being silently truncated.

Assistant draft references include ordinary assistant text and exact done(summary) arguments paired with a completed tool result. The tool result proves completion only; its summary and page-derived verification fields cannot replace the authored draft. This keeps references such as “publish the second option” intact across Ask-to-Act turns. An unresolved contract reports the missing detail and asks for complete text when needed, rather than another permission-only confirmation.

The contract records destinations, accounts, ordered post bodies, media constraints, reply/quote targets, prohibited destinations, and all, any, or conditional fallback requirements. Exact text uses source anchors so the runtime copies the original text instead of accepting a model-reconstructed excerpt. Repeated anchors accept optional startOccurrence and endOccurrence positive, one-based ordinals, each counted from the beginning of the named source (including overlapping matches). For a short literal with identical start/end anchors, startOccurrence alone selects that copy. Omitted ordinals still require unambiguous anchors; invalid or reversed selections are rejected. Unsupported or ambiguous requirements produce clarify; read-only or narrative requests produce none. A requested action that publishes nothing and comes after the publication, such as deleting the post afterwards, is outside the contract: the publication it accompanies compiles as usual, its reason notes the other action, and the audit does not count that action as missing. The action is checked when it happens; a deletion goes through the deletion checks. A condition on whether to publish at all still produces clarify, and the audit still denies a contract that leaves one out. That includes an action the user wants done before publishing ("delete my last post, then post this") and a condition on a later action ("post it only if you can delete it right after"). Invalid contracts cannot fall back to lexical intent rules.

A ready contract normally also requires an independent audit by the selected provider against the concrete composer snapshot. Approval is cached only for that contract, action, payload, account, page, and document/frame. The runtime rereads the composer after the audit and immediately before execution. A changed snapshot needs a new audit. Rejections retain the checker's reason; invalid/mismatched output and request failures are separate technical failures. Arbitrary JavaScript, bundled editing/submission, keyboard submission, and opaque callbacks cannot substitute for the observed publish control.

Both checks can run before the click instead of during it. When the agent writes into X's or Bluesky's own post editor (X's tweetTextarea_N, Bluesky's ProseMirror editor) in a run whose plan submits something and is not bound to another site job, the runtime starts compiling the contract in the background as the write begins. After the write succeeds it runs the click's own probe against that composer's publish control, audits the observed draft for the exact binding key the click would compute, and reads the control captions the click's checks will ask about. An upload into the composer last written repeats the audit. Background work changes nothing in the run: a contract compiled ahead is installed only by the next caller that would compile exactly the same input (sources, context, task, prior state), and a correction drops it. At the click, a verdict stored for the key computed from the click's fresh probe decides once, exactly as the click's own audit would (a denial blocks and offers the same Retry). An audit still running for that key is awaited. Any other key, or no verdict, runs the audit at the click. The post-audit reread, Retry exemption and every other check are unchanged. Stop, steering, a changed task and navigation discard background results; the prefetch writes no transcript, emits only trace notes, and its model calls go through the run's cost allowance.

Compilation and audit use 2,048 output tokens initially and 4,096 for one format repair, capped by the selected provider's limit. Supported provider controls reduce reasoning and request structured JSON; an explicit unsupported-control rejection gets a portable prompt-only repair. Repairs keep the original user input and binding frozen and add format instructions to the system message. The audit accepts one JSON object, optionally fenced, with exact key/action identifiers, a boolean verdict, and a string reason; extra metadata is harmless and displayed reasons are limited to 600 characters. Empty, malformed, conflicting, or mismatched results cannot authorize dispatch. Trace exports include each request/reply, output budget, finish/token diagnostics, and the precise validation failure.

An exhausted publication check stops the run with a clear failure message and app-owned Give up / Retry controls. Give up leaves the draft untouched. A direct user Retry grants one audit exemption for this unchanged task, action, account, payload, page, and document/frame. It skips the independent model audit for one dispatch attempt while keeping intent compilation, deterministic preflight, fresh observations, capability permissions, and duplicate-delivery checks. Model-authored arguments, alternate click strategies, and ordinary Continue cannot grant this exemption. Changes or steering invalidate it; it is consumed immediately before the actual guarded tool invocation and cannot be replayed, including after an uncertain or failed invocation. The frozen contract and user grant survive trusted Continue without being counted as successful execution.

Deterministic enforcement

  • Preserve exact body and alt text, accepting only NFC equivalence and CRLF normalization. Compose requests bind completion to the audited draft.
  • App-observed non-submit controls such as X's “Add post” remain repeatable while composing a thread. Duplicate-submit protection, for a caption a model read as a commit, still applies to unresolved and actual publish controls.
  • Chrome text replacement sends the browser's explicit Select All editing command before Delete, including on macOS, and still requires a proven empty field before inserting the replacement.
  • Read ProseMirror paragraph boundaries and hard breaks as document text, excluding its trailing caret placeholder. Field verification, field digests, Chrome CDP verification, and publication snapshots preserve the same spaces and line breaks.
  • Require complete composer/account/media/context observations and a pre-dispatch permalink baseline. Unassigned or shared thread media makes the observation incomplete.
  • Failed preflight reports publicationValidation.issues, distinguishing missing account/composer/baseline evidence from body, media, or context mismatches. Body mismatches include lengths and the first differing character position/code points without copying the observed draft into diagnostics.
  • Verify a new, account-bound permalink with complete authored body and media evidence. Reply, quote, and ordered thread relationships need their own observed proof.
  • Bluesky detail pages can bind their current permalink to one visible matching-author thread card without a self-link. Conflicting permalinks, duplicate candidates, hidden cards, feeds, and embedded posts cannot supply that binding. Read native word-wrapped rich text within the owning card, excluding previews and controls, and preserve exact paragraphs and link destinations.
  • Include the current resource URL in the pre-dispatch baseline even when a composer hides the existing post. Keep the focused detail record ahead of neighboring records when applying completion limits, so long threads retain its verification evidence.
  • Exclude native Bluesky quote containers (div[role="link"]) from the owning post's body and media. A quote without an observed target, or a focused thread item with preceding cards but no verified parent, retains incomplete context and cannot satisfy publication. An empty parent URL alone is not proof of a standalone post.
  • Track each contract action as pending, failed, or verified. Uncertain delivery never unlocks an alternative or fallback, and cannot authorize another publish attempt.
  • Preserve the contract, authentic clarification pairs, and outcomes only across the existing trusted Continue boundary. A new user task gets a new contract. Clarification after an attempted publication retains its outcomes and blocks further dispatch under the stale instructions; it cannot reset delivery uncertainty or authorize a duplicate.
  • Preserve bound upload evidence when clarification recompiles an unchanged, unattempted action in the same workflow. Changed action constraints or workflow bindings reset it; every refreshed contract still requires a fresh authorization audit.
  • Recognize Bluesky desktop composer launchers as ordinary controls when a model reads their caption as opening an editor, in any language. Actual publish buttons retain the full contract checks.
  • Ordinary X/Bluesky controls (menus, tabs, settings toggles, reply openers, composer tools with non-publish ids) pass without a contract; unidentified controls are reported as unidentified and stay blocked. An unmodified Enter in the app-marked post editor edits the draft.
  • Repeating an identical post under a revised task is blocked and reported as already published (verified) or already attempted under the earlier instruction.
  • Deleting a post whose record shows its permalink (X, Bluesky, LinkedIn, GitHub release or commit links) or a native Mastodon status keeps the exact URL rule. The visible-text binding for requested deletions of ordinary rows (see architecture.md) never applies to records with permalink evidence.
  • Markdown blockquote markers copied from a chat answer are formatting, not post text: an exact body whose continuation lines all carry > markers is published as the rendered text. Greentext (every line quoted, no empty quoted line) keeps its characters.
  • Recognize the modal Cancel, Keep editing and Close controls as non-submitting recovery actions when a model reads their captions as only changing the view, in any language; publish test IDs, captions read as a publication, unrecognized app test IDs on typeless buttons, and form ownership keep the checks.
  • A corrected replacement after an uncertain text write can proceed only after a read-only digest proves the original replacement landed in the same document and field. Changed, incomplete, or unavailable readbacks retain the block; generic page reads never clear it.
  • Convert screenshot clicks to CSS coordinates before target/iframe preflight, and use that same point for dispatch.

Malformed output or failed model calls block publication until repaired or directly retried by the user through the bound audit exemption. Schema validation establishes structure, not correct language understanding; the independent audit is also fallible. Missing site evidence blocks dispatch/completion rather than supplying inferred proof.

Validation

LinkedIn public-post controls with their own composer and audience selector are distinguished from private-message sends. The public Post button still uses the normal submission checks; adjacent messaging panels still require recipient verification. Local navigation fixtures cover both browser engines and reject ambiguous or conflicting composer evidence.

npm test includes the deterministic contract/runtime tests, using mocked responses to verify selected-provider routing, validation, caching, conditional progress, and dispatch/completion enforcement. npm run test:social-contract:dom runs both browser implementations against local Playwright fixtures, including their actual injected completion probes. It requires installed Playwright Chromium; every fixture request is fulfilled locally.

The historical language cases from PR #340 are retained in test/llm/fixtures/social-publication-intent.json. They are evaluation inputs, not a claim of live-model accuracy. Evaluate the compiler and independent audit against the selected provider before drawing conclusions about multilingual understanding, false authorization rates, or latency. Some historical inputs omit the payload or parent URL and should legitimately require clarification.

Live benchmark

test/llm/run-social-publication.mjs uses the production provider class, compiler/repair loop, and audit prompt. Its reviewed set contains 59 compiler cases and 23 deliberately incorrect audit proposals, plus positive audits of initially eligible reviewed actions. The historical corpus is a separate exploratory suite because some old destination labels treat opening a composer as publication intent.

node test/llm/run-social-publication.mjs --validate-only
node --test test/llm/lib/social-publication-score.test.mjs
node test/llm/run-social-publication.mjs --config /private/provider.json
node test/llm/run-social-publication.mjs --config /private/provider.json --suite legacy

The config accepts the existing provider fields (providerName, baseUrl, model, apiKey, and optional provider settings). Alternatively supply --base, --model, --provider, and an API-key environment variable named by --api-key-env. Use the same provider/model/settings as the agent; the runner does not silently select a replacement. Anthropic native and OpenAI-compatible transports are supported.

Calls run sequentially. Reports distinguish first-response validity, repairs, semantic contract errors, audit false accepts/rejects, token usage, and median/p95 latency. First-call latency is reported separately without assuming a cold server. Default results are ignored local artifacts under test/llm/results-social-publication/, with a Markdown report, per-case JSONL, and prompt/fixture hashes. The runner never invokes a browser action or publishes a post.

For X direct messages, a mounted empty log is treated as loading until the same non-loading log remains empty across repeated observations. That establishes a first-message baseline; completion accepts exactly one resulting matching row and rejects history that appears after dispatch.

X direct messages use their own message workflow, not the public publication contract. On /i/chat/<conversation>, the conversation header binds the recipient. An exact display-name or bare-username match resolves to that header's canonical handle during planning or before dispatch; subsequent verification requires the same handle, never a replacement account with the same display name. A send binds the exact body and the identities of all mounted message rows, including hidden and pending ones. An explicitly supplied body is required by the message-field classifier and is recovered from the approved task when a classifier response omits it, including route-local commands such as Send Alex: Hello there. Completion requires a new matching outgoing row after the prior conversation tail with data-send-status="sent", an empty composer, and the same conversation. The open X chat composer does not override that delivery evidence; other adapters retain their dialog checks. Workflow field requirements are loaded even when repeated-item tracking is disabled. Incoming, pending, failed, or pre-existing messages cannot satisfy it.