Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Mark hover, drag and checkbox input dispatched at the transport boundary

Claude0bfd09242139
diff --git a/CHANGELOG.md b/CHANGELOG.md
index fb926a200..9e7249128 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -25,7 +25,7 @@ This changelog was generated from the repository Git history and release tags. V
- A Delete or Remove you ask for on an ordinary row without its own link (a cart item, a mail row, a table row) is no longer always refused. It proceeds when your latest request asks for the deletion and names that row by a word only that row shows, including the confirmation dialog the click opens. The same row is never deleted twice; toolbar and bulk deletions, menus without a linked button, and posts with a permalink keep their previous checks.
- Ordinary controls (filter chips, folder links, menus, tabs, toggles, draft attachment and media removal, composer tools) work before and after a correction; X and Bluesky no longer reject them as unidentified publication actions. After a correction, identical resubmissions are reported as already done, a corrected in-place save or a requested post on another platform is allowed once the earlier result is known, and requests to remove characters or hashtags are no longer treated as deleting the post. A submit-probe exception that left real publish controls unresolved is fixed.
- A correction sent while a run is working is no longer lost when the extension background restarts before the run uses it. The resumed run applies it first, once, in the order it was sent; if the run does not resume, the correction returns to the queue marked as paused, as after Stop. Clearing the conversation or closing the tab discards it.
-- Buttons that animate when the pointer reaches them (a hover lift, scale or border transition) are no longer refused once before working: the click waits up to 0.6 seconds, without sending anything, until the button stops moving, then presses it once at its new centre. A button that keeps moving, is covered or replaced, or any input or Stop during that wait still cancels the click.
+- Buttons that animate when the pointer reaches them (a hover lift, scale or border transition) are no longer refused once before working: the click waits up to 0.6 seconds, without sending anything, until the button stops moving, then presses it once at its new centre. A button that keeps moving, is covered or replaced, or any input or Stop during that wait still cancels the click. Hover, drag and drop, and checkbox actions refused before any input are reported as not sent.
## [40.3.0] - 2026-10-12
diff --git a/docs/page-feedback-monitor.md b/docs/page-feedback-monitor.md
index 44f20aafc..89cf3d4ab 100644
--- a/docs/page-feedback-monitor.md
+++ b/docs/page-feedback-monitor.md
@@ -110,12 +110,19 @@ Steer or grant authorization. Enter and Alt+Enter retain their existing behavior
the receiving-node and page-feedback fences pass. A rejection before the
protocol call means nothing was sent; any later failure means the input was
delivered with an unknown outcome. Such input is never resent and never
- followed by a DOM, script or content-script fallback. For `click_ax`, only
- the browser's missing-receiver error proves that the click was not
- delivered. A possibly delivered, unverified trusted click locks its exact
- document and ref for the rest of the task, including Continue and Retry; a
- verified effect or a new task releases it. Stop between a field write and
- its submit or Enter prevents the submit and reports the write as it happened.
+ followed by a DOM, script or content-script fallback. This covers hover
+ (from its first pointer move) and drag_drop and set_checked (from their
+ press; the no-button move before a press is pointer preparation, as for a
+ click). In Firefox, a trusted BiDi action is dispatched when its native
+ request is posted after the preparation probe and page-feedback fence, and
+ a content-script action when its message is sent; a refusal before either,
+ a content-side fence or a companion refusal before input reports no
+ dispatch. For `click_ax`, only the browser's missing-receiver error proves
+ that the click was not delivered. A possibly delivered, unverified trusted
+ click locks its exact document and ref for the rest of the task, including
+ Continue and Retry; a verified effect or a new task releases it. Stop
+ between a field write and its submit or Enter prevents the submit and
+ reports the write as it happened.
- Meaningful visible content, control state, visibility and geometry changes are
coalesced. Visible subtree identity changes invalidate targets even when the
replacement has identical text and geometry. CSS animation and extension
diff --git a/package.json b/package.json
index 750ca53ac..85ff10f71 100644
--- a/package.json
+++ b/package.json
@@ -60,7 +60,7 @@
"test:social-contract": "node --test test/social-publish-contract.mjs test/publication-quote-markup.mjs",
"test:social-contract:dom": "node test/social-publish-contract-dom.mjs",
"test:runtime-lifecycle": "node test/bot-memory-dom.mjs && node --test test/agent-lifecycle.mjs test/runtime-startup.mjs test/run-ui-scheduled-lifecycle.mjs test/transport-deadlines.mjs test/cdp-worker-restart.mjs test/bot-memory.mjs test/bot-memory-registries.mjs test/bot-memory-cloud-controller.mjs test/cdp-clear-receipt.mjs test/cdp-field-submit-recovery.mjs test/conversation-persistence.mjs test/context-live-tool-group.mjs test/integration-races.mjs test/watch-completion-checkpoint.mjs test/runtime-model-identity.mjs test/provider-message-privacy.mjs test/runtime-task-authority.mjs test/planner-repair-authority.mjs test/runtime-progress-continuation.mjs test/chat-resume-chain.mjs test/chat-resume-completion.mjs test/native-chat-execution-ledger.mjs test/chat-send-dispatch-result.mjs test/chat-reply-dedup-race.mjs test/memcode-lifecycle.mjs test/runtime-peer-review.mjs test/read-continuation-schema.mjs test/completion-submit-acknowledgement.mjs test/completion-live-status.mjs test/completion-iframe-submit.mjs test/completion-semantic-commit.mjs test/completion-diagnostics.mjs test/mastodon-publication.mjs test/calculator.mjs test/chat-recovery-protocol.mjs test/chat-pending-reconciliation.mjs test/semantic-page-retry.mjs test/pdf-detection.mjs test/runtime-lifecycle.mjs test/collection-read-progress.mjs test/network-lifecycle.mjs test/text-write-cancellation.mjs test/trusted-field-preparation.mjs test/draftjs-text.mjs test/rich-text-frameworks.mjs test/cdp-managed-text-selection.mjs test/firefox-rich-text-readback.mjs test/contenteditable-beforeinput.mjs test/trusted-field-preparation-ownership.mjs test/typing-preparation-geometry.mjs test/chat-steering.mjs test/lifecycle-steering-restart.mjs test/resource-mutation-guards.mjs test/completion-steering-receipt-authority.mjs test/steering-revision-recovery.mjs test/pending-clarification-hydration.mjs test/permission-steering-wakeup.mjs test/pure-calculation-reply.mjs test/pure-calculation-tools.mjs test/page-feedback.mjs test/page-feedback-progress.mjs test/ask-response-scope.mjs test/ask-completion-churn.mjs test/page-feedback-policy.mjs test/page-feedback-transport-races.mjs test/page-feedback-recovery.mjs test/telegram-read-preparation-agent.mjs test/media-download-agent-handler.mjs test/code-scanning-url-recipient.mjs test/apocalypse-comm-parsing.mjs test/apocalypse-scalar-decoding.mjs test/bitgpu-chat-security.mjs test/social-media-url-upgrades.mjs test/downloaded-file-saved-bytes.mjs test/downloaded-file-local-probe.mjs test/formless-verification-recovery.mjs test/file-upload-accept.mjs test/file-upload-accept-agent.mjs test/workflow-route-generic-completion.mjs test/release-publication-completion.mjs test/workflow-implicit-metadata.mjs test/completion-current-execution-evidence.mjs test/monitor-intervention-contract.mjs test/monitor-owned-geometry.mjs test/monitor-viewport-remainder.mjs test/monitor-local-attribution.mjs test/monitor-lifecycle-leaks.mjs test/feedback-automatic-churn.mjs test/feedback-navigation-idle.mjs test/feedback-run-boundaries.mjs test/feedback-ask-viewing.mjs test/lifecycle-terminal-done.mjs test/lifecycle-planner-digest.mjs test/lifecycle-vision-fallback.mjs test/guards-control-classification.mjs test/guards-unbound-deletion.mjs test/guards-steering-receipts.mjs && node test/ui-content-lifecycle.mjs && node --test test/native-telegram-chat.mjs test/native-telegram-recipient.mjs test/telegram-native-baseline.mjs test/telegram-native-latest.mjs test/native-chat-pre-enter-draft.mjs test/chat-draft-dispatch-ownership.mjs test/native-telegram-alias-preparation.mjs test/telegram-read-preparation-content.mjs test/queued-composer-ownership.mjs test/onboarding-provider-ready.mjs test/workflow-native-targets.mjs test/native-whatsapp-chat.mjs test/native-chat-context-identity.mjs test/native-whatsapp-contact.mjs test/whatsapp-native-bootstrap.mjs test/whatsapp-drawer-identity.mjs test/whatsapp-read-latest.mjs test/whatsapp-read-preparation.mjs test/whatsapp-chat-completion.mjs test/whatsapp-chat-restart.mjs test/chat-goal-lifecycle.mjs test/chat-completion.mjs test/chat-generic-dispatch.mjs test/completion-live-status-browser.mjs test/native-confirmation-visibility.mjs test/read-continuation-browser.mjs test/iframe-preparation-focus.mjs test/contenteditable-literal.mjs test/native-chat-empty-editor.mjs test/native-chat-trusted-clear.mjs test/chat-steering-ui.mjs test/assistant-message-visibility-ui.mjs test/page-monitor-ui.mjs test/page-monitor-dialog.mjs test/page-monitor-dialog-native.mjs test/page-monitor-inline-post.mjs test/bluesky-composer-context.mjs test/page-monitor-anchored.mjs test/page-monitor-upload-owner.mjs test/page-monitor-upload-release.mjs test/page-monitor-native-controls.mjs test/page-monitor-native-preparation.mjs test/native-preparation-cold.mjs test/page-monitor-navigation.mjs test/page-monitor-main-owner.mjs test/page-monitor-collateral-controls.mjs test/accessibility-tree-scoped-budget.mjs test/ax-reference-origin.mjs test/page-monitor-decorative-hit.mjs test/social-ui-navigation.mjs test/native-resource-deletion-binding.mjs test/dropdown-menu-inventory.mjs test/page-monitor-passive-progress.mjs test/media-download-binding.mjs test/media-download-capture-integration.mjs test/social-media-downloader-reinjection.mjs test/upload-file-local-state-guidance.mjs && node test/message-recipient-upload.mjs && node --test test/message-recipient-whatsapp.mjs && node test/quality/integration-mv3.mjs --full-form && node test/quality/literal-calculate-midrun-mv3.mjs && node test/quality/ask-completion-churn-mv3.mjs && node test/quality/telegram-cold-preparation-mv3.mjs && node test/quality/native-chat-mv3.mjs && node test/quality/native-chat-presubmit-mv3.mjs && node test/quality/cloud-bot-memory-mv3.mjs && node test/quality/whatsapp-contact-matrix.mjs && node test/quality/pdf-detection-mv3.mjs && node test/quality/download-upload-mv3.mjs && node test/quality/upload-release-binding-mv3.mjs && node test/quality/release-review-world-mv3.mjs && node test/quality/downloaded-file-saved-bytes-mv3.mjs && node test/quality/file-upload-accept-mv3.mjs && node test/quality/bulk-resume-mv3.mjs && node test/quality/literal-frame-mv3.mjs && node test/quality/mastodon-publication-mv3.mjs && node test/quality/mastodon-deletion-mv3.mjs && node test/quality/unbound-deletion-mv3.mjs && node test/quality/anchored-action-mv3.mjs && node test/quality/inline-social-composer-mv3.mjs && node test/quality/bluesky-composer-mv3.mjs && node test/quality/dropdown-menu-matrix.mjs && node test/quality/mastodon-deletion-navigation-mv3.mjs && node test/quality/semantic-navigation-matrix.mjs && node test/quality/workflow-replay-mv3.mjs && node test/quality/generative-media-mv3.mjs && node test/quality/cdp-worker-restart-mv3.mjs && node test/quality/steering-worker-restart-mv3.mjs",
- "test:page-feedback": "node --test test/trusted-field-preparation.mjs test/draftjs-text.mjs test/rich-text-frameworks.mjs test/cdp-managed-text-selection.mjs test/firefox-rich-text-readback.mjs test/contenteditable-beforeinput.mjs test/trusted-field-preparation-ownership.mjs test/typing-preparation-geometry.mjs test/pure-calculation-reply.mjs test/pure-calculation-tools.mjs test/page-feedback.mjs test/page-feedback-progress.mjs test/ask-response-scope.mjs test/ask-completion-churn.mjs test/page-feedback-policy.mjs test/page-feedback-transport-races.mjs test/page-feedback-recovery.mjs test/telegram-read-preparation-agent.mjs test/media-download-agent-handler.mjs test/page-monitor-ui.mjs test/page-monitor-dialog.mjs test/page-monitor-dialog-native.mjs test/page-monitor-inline-post.mjs test/bluesky-composer-context.mjs test/page-monitor-anchored.mjs test/page-monitor-upload-owner.mjs test/page-monitor-upload-release.mjs test/page-monitor-native-controls.mjs test/page-monitor-passive-progress.mjs test/media-download-binding.mjs test/media-download-capture-integration.mjs test/monitor-intervention-contract.mjs test/monitor-owned-geometry.mjs test/monitor-viewport-remainder.mjs test/monitor-local-attribution.mjs test/monitor-lifecycle-leaks.mjs test/feedback-automatic-churn.mjs test/feedback-navigation-idle.mjs test/feedback-run-boundaries.mjs test/feedback-ask-viewing.mjs test/delivery-click-ax-response-loss.mjs test/delivery-stop-compound.mjs test/delivery-cdp-input.mjs test/delivery-click-locks.mjs test/delivery-native-button-geometry.mjs test/delivery-hover-transition.mjs && node test/message-recipient-upload.mjs && node --test test/message-recipient-whatsapp.mjs",
+ "test:page-feedback": "node --test test/trusted-field-preparation.mjs test/draftjs-text.mjs test/rich-text-frameworks.mjs test/cdp-managed-text-selection.mjs test/firefox-rich-text-readback.mjs test/contenteditable-beforeinput.mjs test/trusted-field-preparation-ownership.mjs test/typing-preparation-geometry.mjs test/pure-calculation-reply.mjs test/pure-calculation-tools.mjs test/page-feedback.mjs test/page-feedback-progress.mjs test/ask-response-scope.mjs test/ask-completion-churn.mjs test/page-feedback-policy.mjs test/page-feedback-transport-races.mjs test/page-feedback-recovery.mjs test/telegram-read-preparation-agent.mjs test/media-download-agent-handler.mjs test/page-monitor-ui.mjs test/page-monitor-dialog.mjs test/page-monitor-dialog-native.mjs test/page-monitor-inline-post.mjs test/bluesky-composer-context.mjs test/page-monitor-anchored.mjs test/page-monitor-upload-owner.mjs test/page-monitor-upload-release.mjs test/page-monitor-native-controls.mjs test/page-monitor-passive-progress.mjs test/media-download-binding.mjs test/media-download-capture-integration.mjs test/monitor-intervention-contract.mjs test/monitor-owned-geometry.mjs test/monitor-viewport-remainder.mjs test/monitor-local-attribution.mjs test/monitor-lifecycle-leaks.mjs test/feedback-automatic-churn.mjs test/feedback-navigation-idle.mjs test/feedback-run-boundaries.mjs test/feedback-ask-viewing.mjs test/delivery-click-ax-response-loss.mjs test/delivery-stop-compound.mjs test/delivery-cdp-input.mjs test/delivery-click-locks.mjs test/delivery-native-button-geometry.mjs test/delivery-hover-transition.mjs test/delivery-dispatch-boundary.mjs && node test/message-recipient-upload.mjs && node --test test/message-recipient-whatsapp.mjs",
"test:chat-steering": "node --test test/chat-steering.mjs test/chat-steering-ui.mjs test/lifecycle-steering-latency.mjs test/lifecycle-steering-restart.mjs test/lifecycle-run-ui.mjs test/lifecycle-menu-prompts-held.mjs",
"test:rich-text-dom": "node test/ui-content-lifecycle.mjs --dom",
"test:firefox-bidi": "node --test test/firefox-bidi.mjs test/firefox-full-page-screenshot.mjs",
@@ -92,7 +92,7 @@
"test:upload-owner": "node --test test/page-monitor-upload-owner.mjs test/page-monitor-upload-release.mjs",
"test:shard:quick": "npm run test:cms && npm run test:markdown && npm run test:generative-media && npm run test:systemone && npm run test:systemone:fast && node --test test/browser-dialogs.mjs test/capsolver-cloud-broker.mjs test/captcha-providers.mjs test/captcha-weighted-fallback.mjs test/captcha-hcaptcha-providers.mjs test/captcha-native-providers.mjs test/captcha-runtime-policy.mjs test/captcha-callback-bridge.mjs test/captcha-bridge-registration.mjs test/captcha-aws-recovery.mjs test/captcha-documented-contracts.mjs test/huggingface-signup-recovery.mjs test/trace-full-export.mjs test/feedback-traces.mjs && npm run test:firefox-bidi && npm run test:billing-notice && npm run test:compass-quota && npm run test:provider-limits && npm run test:provider-catalog && npm run test:provider-instances:ui && npm run test:openrouter-capabilities && npm run test:anthropic && npm run test:accessibility-tree-benchmark && npm run test:toolbar-guard && npm run test:cloud-bridge-approval && npm run test:discord && npm run test:pdf-read && npm run test:pdf-selection && npm run test:social-contract && npm run test:safesocial && npm run test:build-unpacked && npm run test:attachment-drop && npm run test:workflow-editor && node --test test/profile-sync-stats.mjs && node --test test/memcode-recall.mjs",
"test:shard:runtime-unit": "node --test test/agent-lifecycle.mjs test/runtime-startup.mjs test/run-ui-scheduled-lifecycle.mjs test/transport-deadlines.mjs test/cdp-worker-restart.mjs test/bot-memory.mjs test/bot-memory-click-guard.mjs test/bot-memory-registries.mjs test/bot-memory-cloud-controller.mjs test/cdp-clear-receipt.mjs test/cdp-field-submit-recovery.mjs test/conversation-persistence.mjs test/context-live-tool-group.mjs test/integration-races.mjs test/watch-completion-checkpoint.mjs test/runtime-model-identity.mjs test/provider-message-privacy.mjs test/runtime-task-authority.mjs test/planner-repair-authority.mjs test/runtime-progress-continuation.mjs test/chat-resume-chain.mjs test/chat-resume-completion.mjs test/native-chat-execution-ledger.mjs test/chat-send-dispatch-result.mjs test/chat-reply-dedup-race.mjs test/memcode-lifecycle.mjs test/runtime-peer-review.mjs test/read-continuation-schema.mjs test/completion-submit-acknowledgement.mjs test/completion-live-status.mjs test/completion-iframe-submit.mjs test/completion-semantic-commit.mjs test/completion-diagnostics.mjs test/mastodon-publication.mjs test/calculator.mjs test/chat-recovery-protocol.mjs test/chat-pending-reconciliation.mjs test/semantic-page-retry.mjs test/pdf-detection.mjs test/runtime-lifecycle.mjs test/collection-read-progress.mjs test/network-lifecycle.mjs test/text-write-cancellation.mjs test/trusted-field-preparation.mjs test/draftjs-text.mjs test/rich-text-frameworks.mjs test/cdp-managed-text-selection.mjs test/firefox-rich-text-readback.mjs test/contenteditable-beforeinput.mjs test/trusted-field-preparation-ownership.mjs test/typing-preparation-geometry.mjs test/chat-steering.mjs test/resource-mutation-guards.mjs test/completion-steering-receipt-authority.mjs test/steering-revision-recovery.mjs test/pending-clarification-hydration.mjs test/permission-steering-wakeup.mjs test/pure-calculation-reply.mjs test/pure-calculation-tools.mjs test/page-feedback.mjs test/page-feedback-progress.mjs test/ask-response-scope.mjs test/ask-completion-churn.mjs test/page-feedback-policy.mjs test/page-feedback-transport-races.mjs test/page-feedback-recovery.mjs test/telegram-read-preparation-agent.mjs test/media-download-agent-handler.mjs test/code-scanning-url-recipient.mjs test/apocalypse-comm-parsing.mjs test/apocalypse-scalar-decoding.mjs test/bitgpu-chat-security.mjs test/social-media-url-upgrades.mjs test/downloaded-file-saved-bytes.mjs test/downloaded-file-local-probe.mjs test/formless-verification-recovery.mjs test/file-upload-accept.mjs test/file-upload-accept-agent.mjs test/workflow-route-generic-completion.mjs test/release-publication-completion.mjs test/workflow-implicit-metadata.mjs test/completion-current-execution-evidence.mjs test/video-upload-attachment.mjs test/feedback-automatic-churn.mjs test/feedback-navigation-idle.mjs test/feedback-run-boundaries.mjs test/feedback-ask-viewing.mjs test/lifecycle-terminal-done.mjs test/lifecycle-planner-digest.mjs test/lifecycle-steering-latency.mjs test/lifecycle-steering-restart.mjs test/lifecycle-vision-fallback.mjs test/delivery-click-locks.mjs test/guards-steering-receipts.mjs test/terminal-user-messages.mjs",
- "test:shard:browser-a": "node test/bot-memory-dom.mjs && node test/bot-memory-click-dom.mjs && node test/ui-content-lifecycle.mjs && node --test test/native-telegram-chat.mjs test/native-telegram-recipient.mjs test/telegram-native-baseline.mjs test/telegram-native-latest.mjs test/native-chat-pre-enter-draft.mjs test/chat-draft-dispatch-ownership.mjs test/native-telegram-alias-preparation.mjs test/telegram-read-preparation-content.mjs test/queued-composer-ownership.mjs test/onboarding-provider-ready.mjs test/workflow-native-targets.mjs test/native-whatsapp-chat.mjs test/native-chat-context-identity.mjs test/native-whatsapp-contact.mjs test/whatsapp-native-bootstrap.mjs test/whatsapp-drawer-identity.mjs test/whatsapp-read-latest.mjs test/whatsapp-read-preparation.mjs test/whatsapp-chat-completion.mjs test/whatsapp-chat-restart.mjs test/chat-goal-lifecycle.mjs test/chat-completion.mjs test/chat-generic-dispatch.mjs test/completion-live-status-browser.mjs test/native-confirmation-visibility.mjs test/native-button-preparation.mjs test/read-continuation-browser.mjs test/iframe-preparation-focus.mjs test/contenteditable-literal.mjs test/native-chat-empty-editor.mjs test/native-chat-trusted-clear.mjs test/chat-steering-ui.mjs test/assistant-message-visibility-ui.mjs test/page-monitor-ui.mjs test/page-monitor-dialog.mjs test/page-monitor-dialog-native.mjs test/page-monitor-inline-post.mjs test/bluesky-composer-context.mjs test/page-monitor-anchored.mjs test/page-monitor-upload-owner.mjs test/page-monitor-upload-release.mjs test/page-monitor-native-controls.mjs test/page-monitor-native-preparation.mjs test/native-preparation-cold.mjs test/page-monitor-navigation.mjs test/page-monitor-main-owner.mjs test/page-monitor-collateral-controls.mjs test/accessibility-tree-scoped-budget.mjs test/ax-reference-origin.mjs test/page-monitor-decorative-hit.mjs test/social-ui-navigation.mjs test/native-resource-deletion-binding.mjs test/dropdown-menu-inventory.mjs test/page-monitor-passive-progress.mjs test/media-download-binding.mjs test/media-download-capture-integration.mjs test/social-media-downloader-reinjection.mjs test/upload-file-local-state-guidance.mjs test/monitor-intervention-contract.mjs test/monitor-owned-geometry.mjs test/monitor-viewport-remainder.mjs test/monitor-local-attribution.mjs test/monitor-lifecycle-leaks.mjs test/lifecycle-run-ui.mjs test/lifecycle-menu-prompts-held.mjs test/delivery-cdp-input.mjs test/delivery-click-ax-response-loss.mjs test/delivery-native-button-geometry.mjs test/delivery-hover-transition.mjs test/delivery-stop-compound.mjs test/guards-control-classification.mjs test/guards-unbound-deletion.mjs",
+ "test:shard:browser-a": "node test/bot-memory-dom.mjs && node test/bot-memory-click-dom.mjs && node test/ui-content-lifecycle.mjs && node --test test/native-telegram-chat.mjs test/native-telegram-recipient.mjs test/telegram-native-baseline.mjs test/telegram-native-latest.mjs test/native-chat-pre-enter-draft.mjs test/chat-draft-dispatch-ownership.mjs test/native-telegram-alias-preparation.mjs test/telegram-read-preparation-content.mjs test/queued-composer-ownership.mjs test/onboarding-provider-ready.mjs test/workflow-native-targets.mjs test/native-whatsapp-chat.mjs test/native-chat-context-identity.mjs test/native-whatsapp-contact.mjs test/whatsapp-native-bootstrap.mjs test/whatsapp-drawer-identity.mjs test/whatsapp-read-latest.mjs test/whatsapp-read-preparation.mjs test/whatsapp-chat-completion.mjs test/whatsapp-chat-restart.mjs test/chat-goal-lifecycle.mjs test/chat-completion.mjs test/chat-generic-dispatch.mjs test/completion-live-status-browser.mjs test/native-confirmation-visibility.mjs test/native-button-preparation.mjs test/read-continuation-browser.mjs test/iframe-preparation-focus.mjs test/contenteditable-literal.mjs test/native-chat-empty-editor.mjs test/native-chat-trusted-clear.mjs test/chat-steering-ui.mjs test/assistant-message-visibility-ui.mjs test/page-monitor-ui.mjs test/page-monitor-dialog.mjs test/page-monitor-dialog-native.mjs test/page-monitor-inline-post.mjs test/bluesky-composer-context.mjs test/page-monitor-anchored.mjs test/page-monitor-upload-owner.mjs test/page-monitor-upload-release.mjs test/page-monitor-native-controls.mjs test/page-monitor-native-preparation.mjs test/native-preparation-cold.mjs test/page-monitor-navigation.mjs test/page-monitor-main-owner.mjs test/page-monitor-collateral-controls.mjs test/accessibility-tree-scoped-budget.mjs test/ax-reference-origin.mjs test/page-monitor-decorative-hit.mjs test/social-ui-navigation.mjs test/native-resource-deletion-binding.mjs test/dropdown-menu-inventory.mjs test/page-monitor-passive-progress.mjs test/media-download-binding.mjs test/media-download-capture-integration.mjs test/social-media-downloader-reinjection.mjs test/upload-file-local-state-guidance.mjs test/monitor-intervention-contract.mjs test/monitor-owned-geometry.mjs test/monitor-viewport-remainder.mjs test/monitor-local-attribution.mjs test/monitor-lifecycle-leaks.mjs test/lifecycle-run-ui.mjs test/lifecycle-menu-prompts-held.mjs test/delivery-cdp-input.mjs test/delivery-click-ax-response-loss.mjs test/delivery-native-button-geometry.mjs test/delivery-hover-transition.mjs test/delivery-dispatch-boundary.mjs test/delivery-stop-compound.mjs test/guards-control-classification.mjs test/guards-unbound-deletion.mjs",
"test:shard:browser-b1": "node test/message-recipient-upload.mjs && node --test test/message-recipient-whatsapp.mjs && node test/quality/integration-mv3.mjs --full-form && node test/quality/literal-calculate-midrun-mv3.mjs && node test/quality/ask-completion-churn-mv3.mjs && node test/quality/telegram-cold-preparation-mv3.mjs && node test/quality/native-chat-mv3.mjs && node test/quality/native-chat-presubmit-mv3.mjs && node test/quality/cloud-bot-memory-mv3.mjs",
"test:shard:browser-b2": "node test/quality/whatsapp-contact-matrix.mjs && node test/quality/pdf-detection-mv3.mjs && node test/quality/download-upload-mv3.mjs && node test/quality/upload-release-binding-mv3.mjs && node test/quality/release-review-world-mv3.mjs && node test/quality/downloaded-file-saved-bytes-mv3.mjs && node test/quality/file-upload-accept-mv3.mjs && node test/quality/bulk-resume-mv3.mjs && node test/quality/shared-action-guards-mv3.mjs && node test/quality/video-upload-attachment-mv3.mjs",
"test:shard:browser-c": "node test/quality/literal-frame-mv3.mjs && node test/quality/mastodon-publication-mv3.mjs && node test/quality/mastodon-deletion-mv3.mjs && node test/quality/unbound-deletion-mv3.mjs && node test/quality/anchored-action-mv3.mjs && node test/quality/inline-social-composer-mv3.mjs && node test/quality/bluesky-composer-mv3.mjs && node test/quality/dropdown-menu-matrix.mjs && node test/quality/mastodon-deletion-navigation-mv3.mjs && node test/quality/semantic-navigation-matrix.mjs && node test/quality/workflow-replay-mv3.mjs && node test/quality/generative-media-mv3.mjs && node test/quality/cdp-worker-restart-mv3.mjs && node test/quality/steering-worker-restart-mv3.mjs",
diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.js
index 789eb72bf..f549524b4 100644
--- a/src/chrome/src/agent/agent.js
+++ b/src/chrome/src/agent/agent.js
@@ -39044,10 +39044,11 @@ If the user has already named or confirmed this exact recipient, do NOT ask agai
abortSignal,
deadlineAt: Number(CONTENT_ACTION_SIGNAL_DEADLINES.get(abortSignal)?.deadlineAt) || 0,
deadlineError: CONTENT_ACTION_SIGNAL_DEADLINES.get(abortSignal)?.error || null,
- beforeDispatch: () => {
+ // Marked at the transport boundary, once the press passed its fences
+ // (or for a rejection that may follow delivery), never before them.
+ onDispatch: () => {
recoveryState.clickStarted = true;
if (upstreamDispatchState) upstreamDispatchState.started = true;
- return { success: true };
},
});
recoveryState.clickResult = clickResult;
@@ -39063,7 +39064,10 @@ If the user has already named or confirmed this exact recipient, do NOT ask agai
}
this._throwIfAborted(abortSignal);
- const clickDispatched = clickResult?.dispatched === true || clickResult?.success === true;
+ // The transport boundary is the authority: a press that passed its fences
+ // may have been delivered even when its acknowledgement or result was lost.
+ const clickDispatched = recoveryState.clickStarted === true
+ || clickResult?.dispatched === true || clickResult?.success === true;
if (clickDispatched) {
await new Promise(resolve => setTimeout(resolve, SET_CHECKED_VERIFY_DELAY_MS));
this._throwIfAborted(abortSignal);
@@ -39174,6 +39178,9 @@ If the user has already named or confirmed this exact recipient, do NOT ask agai
success,
dispatched: clickDispatched,
noDispatch: clickDispatched ? undefined : true,
+ // Possibly delivered, never acknowledged: inspect, never repeat blindly.
+ ...(clickDispatched && !trustedClickSucceeded ? { outcomeUnknown: true, retryable: false } : {}),
+ ...(!clickDispatched && clickError?.code === 'page_feedback_pending' ? { pageFeedbackPending: true } : {}),
trusted: trustedClickSucceeded,
verified: success,
needsTrustedClick: false,
@@ -46341,6 +46348,11 @@ If the user has already named or confirmed this exact recipient, do NOT ask agai
if (typeof refId !== 'string') {
return { success: false, error: 'hover: ref_id (string, e.g. "ref_42") is required' };
}
+ // The pointer move is this tool's input: dispatch starts when the first
+ // move passes its fences at the transport boundary. A refusal before that
+ // sent nothing; after it, a failure has an unknown outcome and the moves
+ // are never resent.
+ let hoverInputSent = false;
try {
const rectResp = await chrome.tabs.sendMessage(tabId, {
target: 'content', action: 'ax_resolve_rect', params: { ref_id: refId },
@@ -46367,12 +46379,10 @@ If the user has already named or confirmed this exact recipient, do NOT ask agai
// listening to mouseenter/mouseover/pointerover. We move twice (once
// ~10px outside, once at center) so sites that only fire on the
// first crossing of the element boundary still see a transition.
- markEarlyCdpDispatched();
- await cdpClient.sendCommand(tabId, 'Input.dispatchMouseEvent', {
+ await sendTrustedCdpInput(options => cdpClient.sendCommand(tabId, 'Input.dispatchMouseEvent', {
type: 'mouseMoved', x: Math.max(0, x - 10), y, button: 'none', buttons: 0,
- });
+ }, '', options), { beforeSend: throwIfEarlyCdpAborted, onSent: () => { hoverInputSent = true; markEarlyCdpDispatched(); } });
throwIfEarlyCdpAborted();
- markEarlyCdpDispatched();
await cdpClient.sendCommand(tabId, 'Input.dispatchMouseEvent', {
type: 'mouseMoved', x, y, button: 'none', buttons: 0,
});
@@ -46387,7 +46397,14 @@ If the user has already named or confirmed this exact recipient, do NOT ask agai
hint: 'A hover-revealed menu/tooltip typically renders in a React portal at the end of <body>. Re-read the tree with get_accessibility_tree({filter:"visible"}) β€” do NOT pass a ref_id (subtree filter will miss the portal).',
};
} catch (e) {
- return { success: false, error: `hover failed: ${e.message || e}` };
+ // Stop and the action deadline keep their lifecycle reporting, which
+ // reads the dispatch state; a fence veto is reported by the
+ // page-feedback wrapper (undispatched before any move, uncertain after).
+ if (e?.name === 'AbortError' || e?.code === 'content_action_timeout' || e?.code === 'page_feedback_pending') throw e;
+ return hoverInputSent
+ ? { success: false, dispatched: true, outcomeUnknown: true, retryable: false, method: 'cdp-hover', ref_id: refId,
+ error: `hover input was sent but did not complete (${e.message || e}). Inspect the page before hovering again.` }
+ : { success: false, dispatched: false, noDispatch: true, outcomeUnknown: false, error: `hover failed: ${e.message || e}` };
}
}
@@ -46399,7 +46416,11 @@ If the user has already named or confirmed this exact recipient, do NOT ask agai
}
const stepsRaw = Number(args.steps ?? 10);
const steps = Math.max(2, Math.min(40, Number.isFinite(stepsRaw) ? Math.floor(stepsRaw) : 10));
- let dragPressed = false;
+ // Like a click, the no-button move onto the source is pointer
+ // preparation; the drag is dispatched when its press passes its fences at
+ // the transport boundary. After that a failure has an unknown outcome,
+ // any held button is released, and nothing is resent.
+ let dragPressSent = false, dragReleased = false;
try {
// Single round-trip: content.js scrolls source-then-dest and
// measures BOTH rects in the same viewport snapshot. The previous
@@ -46441,15 +46462,13 @@ If the user has already named or confirmed this exact recipient, do NOT ask agai
// mid-drag waypoints are what HTML5 dnd dragenter/dragover handlers
// listen to; Trello/Linear/Notion need at least a handful so the
// drop indicator can settle before the release.
- markEarlyCdpDispatched();
await cdpClient.sendCommand(tabId, 'Input.dispatchMouseEvent', {
type: 'mouseMoved', x: x1, y: y1, button: 'none', buttons: 0,
});
throwIfEarlyCdpAborted();
- await cdpClient.sendCommand(tabId, 'Input.dispatchMouseEvent', {
+ await sendTrustedCdpInput(options => cdpClient.sendCommand(tabId, 'Input.dispatchMouseEvent', {
type: 'mousePressed', x: x1, y: y1, button: 'left', buttons: 1, clickCount: 1,
- });
- dragPressed = true;
+ }, '', options), { beforeSend: throwIfEarlyCdpAborted, onSent: () => { dragPressSent = true; markEarlyCdpDispatched(); } });
throwIfEarlyCdpAborted();
for (let i = 1; i <= steps; i++) {
const t = i / steps;
@@ -46467,7 +46486,7 @@ If the user has already named or confirmed this exact recipient, do NOT ask agai
await cdpClient.sendCommand(tabId, 'Input.dispatchMouseEvent', {
type: 'mouseReleased', x: x2, y: y2, button: 'left', buttons: 0, clickCount: 1,
});
- dragPressed = false;
+ dragReleased = true;
throwIfEarlyCdpAborted();
return {
success: true,
@@ -46478,8 +46497,13 @@ If the user has already named or confirmed this exact recipient, do NOT ask agai
hint: 'Re-read the accessibility tree to confirm the order/position changed. If nothing moved, the site may use HTML5 drag-and-drop with custom DataTransfer payloads that CDP cannot fully emulate β€” try the drag again with higher `steps` (15–20) or fall back to keyboard-based reorder controls if the site exposes them.',
};
} catch (e) {
- if (dragPressed) await this._cancelTrustedClickPress(tabId);
- return { success: false, error: `drag_drop failed: ${e.message || e}` };
+ // A press whose release was not acknowledged may still be held.
+ if (dragPressSent && !dragReleased) await this._cancelTrustedClickPress(tabId);
+ if (e?.name === 'AbortError' || e?.code === 'content_action_timeout' || e?.code === 'page_feedback_pending') throw e;
+ return dragPressSent
+ ? { success: false, dispatched: true, outcomeUnknown: true, retryable: false, method: 'cdp-drag',
+ error: `drag_drop input was sent but did not complete (${e.message || e}). Inspect the page before dragging again.` }
+ : { success: false, dispatched: false, noDispatch: true, outcomeUnknown: false, error: `drag_drop failed: ${e.message || e}` };
}
}
diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.js
index 8bbd79058..8ad72e4b2 100644
--- a/src/firefox/src/agent/agent.js
+++ b/src/firefox/src/agent/agent.js
@@ -40685,18 +40685,27 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
}
throwIfContentPipelineAborted();
this._throwIfAborted(stageAbortSignal);
- if (Agent.STATE_CHANGE_TOOLS.has(name)) markContentPipelineDispatched();
- this._throwIfAborted(stageAbortSignal);
const deadlines = [contentPipelineAbortSignal, stageAbortSignal]
.map(signal => Number(CONTENT_ACTION_SIGNAL_DEADLINES.get(signal)?.deadlineAt))
.filter(value => Number.isFinite(value) && value > 0);
const actionDeadlineAt = deadlines.length ? Math.min(...deadlines) : 0;
- return firefoxBidi.sendContent(tabId, {
+ // Dispatch starts at the transport boundary of the route actually taken:
+ // the content message, or a trusted BiDi request after its preparation
+ // probe and page-feedback fence. A refusal before either sent nothing.
+ const response = await firefoxBidi.sendContent(tabId, {
target: 'content',
action,
params: contentArgs,
...(actionDeadlineAt > 0 ? { actionDeadlineAt } : {}),
- }, messageOptions);
+ }, messageOptions, { onDispatch: Agent.STATE_CHANGE_TOOLS.has(name) ? () => {
+ throwIfContentPipelineAborted();
+ this._throwIfAborted(stageAbortSignal);
+ markContentPipelineDispatched();
+ } : null });
+ // A content-side fence or a companion refusal before input proves that
+ // nothing was dispatched; the provisional transport state ends here.
+ if (response?.dispatched === false && response?.noDispatch === true) contentPipelineDispatchState.started = false;
+ return response;
};
const dispatchContentAction = () => runContentActionStage(sendContentAction);
const provenNoDispatchDeadline = response => {
diff --git a/src/firefox/src/bidi/client.js b/src/firefox/src/bidi/client.js
index 127941793..a5a7188b5 100644
--- a/src/firefox/src/bidi/client.js
+++ b/src/firefox/src/bidi/client.js
@@ -1,11 +1,18 @@
/** Optional native-messaging transport. No raw BiDi commands are exposed to pages. */
import { beforePageAgentDispatch, validateNativePageDispatch } from '../agent/page-feedback.js';
+// Failures raised before a native request could reach the companion (run
+// binding, the page-feedback fence, the caller's last check, an unavailable
+// port). Nothing was sent; any later failure may follow delivered input.
+const REQUEST_NOT_SENT = new WeakSet();
+const notSent = error => { if (error && typeof error === 'object') REQUEST_NOT_SENT.add(error); return error; };
+export const isBidiRequestNotSentError = error => !!error && typeof error === 'object' && REQUEST_NOT_SENT.has(error);
export class FirefoxBidiClient {
constructor(api) { this.apiOverride = api; this.pending = new Map(); this.runs = new Map(); this.captures = new Map(); this.sequence = 0; this.connectionEpoch = 0; }
get api() { return this.apiOverride || globalThis.browser; }
request(command, args = {}) {
if (!this.port) {
- this.port = this.api.runtime.connectNative('one.webbrain.bidi');
+ try { this.port = this.api.runtime.connectNative('one.webbrain.bidi'); }
+ catch (error) { return Promise.reject(notSent(error)); }
const port = this.port;
port.onMessage.addListener(message => {
if (message?.command === 'validatePageDispatch' && typeof message.id === 'string') {
@@ -65,7 +72,7 @@ export class FirefoxBidiClient {
const timer = setTimeout(() => { this.pending.delete(id); reject(new Error('Firefox companion timed out; action outcome may be unknown')); }, 15000);
this.pending.set(id, { resolve: value => { clearTimeout(timer); resolve(value); }, reject: error => { clearTimeout(timer); reject(error); } });
try { this.port.postMessage({ id, command, ...args }); }
- catch (error) { this.pending.get(id).reject(error); this.pending.delete(id); }
+ catch (error) { this.pending.get(id).reject(notSent(error)); this.pending.delete(id); }
});
}
async connect() {
@@ -148,34 +155,43 @@ export class FirefoxBidiClient {
if (this.captures.get(tabId) === owner) this.captures.delete(tabId);
}
}
- async perform(tabId, action, payload) {
- const owner = this.runs.get(tabId);
- if (!owner || owner.signal?.aborted || owner.disconnected) throw new Error('No connected Firefox BiDi run; restart the task after reconnecting');
- await this.bindRun(tabId, owner);
- if (this.runs.get(tabId) !== owner || owner.disconnected || owner.signal?.aborted) throw new Error('Run stopped');
- const pageFeedbackGuard = await beforePageAgentDispatch(this.api, tabId, {
- kind: action === 'navigate' ? 'navigate' : action === 'scroll' ? 'scroll'
- : ['type', 'field', 'key', 'upload'].includes(action) ? 'input' : 'click',
- ...(payload.point ? { x: payload.point.x, y: payload.point.y } : {}),
- selector: payload.selector, ref_id: payload.ref_id, frameId: payload.frameId || 0,
- tool: action, prepareMonitor: true,
- navigationCandidate: ['click', 'checked'].includes(action),
- fenceOnly: true,
- url: action === 'navigate' ? payload.url : undefined,
- });
+ // `onDispatch` runs once every pre-send check passed, immediately before the
+ // native request (or deferred navigation) can deliver input. It may throw
+ // as the caller's last check; that, like any earlier failure, sent nothing.
+ async perform(tabId, action, payload, { onDispatch = null } = {}) {
+ let owner, pageFeedbackGuard;
+ try {
+ owner = this.runs.get(tabId);
+ if (!owner || owner.signal?.aborted || owner.disconnected) throw new Error('No connected Firefox BiDi run; restart the task after reconnecting');
+ await this.bindRun(tabId, owner);
+ if (this.runs.get(tabId) !== owner || owner.disconnected || owner.signal?.aborted) throw new Error('Run stopped');
+ pageFeedbackGuard = await beforePageAgentDispatch(this.api, tabId, {
+ kind: action === 'navigate' ? 'navigate' : action === 'scroll' ? 'scroll'
+ : ['type', 'field', 'key', 'upload'].includes(action) ? 'input' : 'click',
+ ...(payload.point ? { x: payload.point.x, y: payload.point.y } : {}),
+ selector: payload.selector, ref_id: payload.ref_id, frameId: payload.frameId || 0,
+ tool: action, prepareMonitor: true,
+ navigationCandidate: ['click', 'checked'].includes(action),
+ fenceOnly: true,
+ url: action === 'navigate' ? payload.url : undefined,
+ });
+ if (!owner.bound && (action !== 'navigate' || !/^https?:\/\//.test(payload.url || ''))) throw new Error('Navigate to a web page before trusted input');
+ onDispatch?.();
+ } catch (error) { throw notSent(error); }
if (!owner.bound) {
- if (action !== 'navigate' || !/^https?:\/\//.test(payload.url || '')) throw new Error('Navigate to a web page before trusted input');
await this.api.tabs.update(tabId, { url: payload.url });
return { success: true, dispatched: true, bindingDeferred: true };
}
return this.request('perform', { runId: owner.runId, action, payload: { ...payload, pageFeedbackGuard } });
}
- async sendContent(tabId, message, options) {
+ // `onDispatch` marks the transport boundary of whichever route is taken: the
+ // content message, or the native request after its preparation and fence.
+ async sendContent(tabId, message, options, { onDispatch = null } = {}) {
const actions = { click: 'click', click_ax: 'click', type: 'type', type_ax: 'type', set_field: 'field', press_keys: 'key', hover: 'hover', bidi_prepare_upload: 'upload' };
const action = actions[message.action];
const owner = this.runs.get(tabId);
if (owner?.disconnected || owner?.signal?.aborted) return { success: false, dispatched: false, noDispatch: true, retryable: false, error: 'Firefox connection stopped; restart the task.' };
- if (!owner || !action) return this.api.tabs.sendMessage(tabId, message, options);
+ if (!owner || !action) { onDispatch?.(); return this.api.tabs.sendMessage(tabId, message, options); }
await this.bindRun(tabId, owner);
if (!owner.bound) return { success: false, dispatched: false, noDispatch: true, error: 'Navigate to a web page before trusted input.' };
const token = crypto.randomUUID();
@@ -192,11 +208,15 @@ export class FirefoxBidiClient {
point: prepared.point || null,
checkable: prepared.checkable || null,
deadlineAt: message.actionDeadlineAt || 0,
- }), ...(prepared.rect ? { rect: prepared.rect } : {}), ...(prepared._filePickerGuardId ? { _filePickerGuardId: prepared._filePickerGuardId } : {}) };
+ }, { onDispatch }), ...(prepared.rect ? { rect: prepared.rect } : {}), ...(prepared._filePickerGuardId ? { _filePickerGuardId: prepared._filePickerGuardId } : {}) };
} catch (error) {
- // A transport failure can occur after trusted input was delivered.
+ if (error?.name === 'AbortError' && isBidiRequestNotSentError(error)) throw error;
+ // A refusal before the native request was posted sent nothing. A
+ // transport failure after it can follow delivered trusted input.
if (error?.code === 'page_feedback_pending') return { ...metadata, success: false, dispatched: false,
noDispatch: true, pageFeedbackPending: true, error: error.message };
+ if (isBidiRequestNotSentError(error)) return { ...metadata, success: false, dispatched: false, noDispatch: true,
+ outcomeUnknown: false, retryable: false, error: error.message };
return { ...metadata, success: false, dispatched: true, outcomeUnknown: true, retryable: false, ...(error.dispatchState || {}), error: error.message };
}
}
diff --git a/test/delivery-dispatch-boundary.mjs b/test/delivery-dispatch-boundary.mjs
new file mode 100644
index 000000000..8c8624de0
--- /dev/null
+++ b/test/delivery-dispatch-boundary.mjs
@@ -0,0 +1,433 @@
+// Delivery regression: hover, drag_drop and set_checked mark dispatch at the
+// input transport boundary, never before their fences.
+// - Chrome: the trusted CDP command that is the tool's first consequential
+// input (hover: the pointer move; drag_drop and set_checked: the press, the
+// no-button move before it being pointer preparation as for every click).
+// - Firefox: the native BiDi request after its preparation probe and
+// page-feedback fence, or the content message for content-script tools.
+// A refusal before that boundary reports noDispatch; a failure after it
+// reports dispatched with an unknown outcome, and nothing is resent.
+//
+// [mocked transport] The real Agent of each build. Chrome runs the real
+// cdpClient.sendCommand: its pre-protocol check (_beforeCommand, where the
+// page-feedback fence runs) is wrapped to refuse, and chrome.debugger is a
+// recorder that can lose an acknowledgement after the command arrived.
+// Firefox runs the real FirefoxBidiClient with a mocked native request and
+// content messaging; its page-feedback fences use the real page-feedback run.
+import assert from 'node:assert/strict';
+import { test } from 'node:test';
+
+const area = { get: async () => ({}), set: async () => {}, remove: async () => {} };
+const api = {
+ storage: { local: area, session: area },
+ runtime: { getURL: value => `extension://test/${value}`, getManifest: () => ({ version: 'test' }), lastError: null },
+ tabs: { get: async () => ({ id: 17, url: 'https://boundary.test/start' }), query: async () => [{ id: 17 }],
+ sendMessage: async () => ({}), update: async () => ({}) },
+ scripting: { executeScript: async () => [{}] },
+ webNavigation: { getAllFrames: async () => [{ frameId: 0, documentId: 'boundary-document', url: 'https://boundary.test/start' }] },
+ debugger: { sendCommand: () => { throw new Error('debugger recorder not installed'); },
+ onEvent: { addListener() {}, removeListener() {} }, onDetach: { addListener() {}, removeListener() {} } },
+};
+globalThis.chrome = api;
+globalThis.browser = api;
+const { Agent: ChromeAgent } = await import('../src/chrome/src/agent/agent.js');
+const { cdpClient } = await import('../src/chrome/src/cdp/cdp-client.js');
+const { Agent: FirefoxAgent } = await import('../src/firefox/src/agent/agent.js');
+const { firefoxBidi } = await import('../src/firefox/src/bidi/client.js');
+
+const pageFeedbackVeto = () => Object.assign(
+ new Error('The browser changed during action preparation. Re-observe the page before acting.'),
+ { code: 'page_feedback_pending' },
+);
+const notAttached = () => new Error('Not attached to tab 17');
+const lostAck = 'Detached while handling command.';
+
+// ── Chrome ─────────────────────────────────────────────────────────────────
+const original = { attach: cdpClient.attach, beforeCommand: cdpClient._beforeCommand,
+ resolveSelector: cdpClient.resolveSelector, arm: cdpClient.armFileInputClickGuard, consume: cdpClient.consumeFileInputClickGuard };
+
+// `refuse(params, index)` rejects an Input command before the protocol call
+// (the fence or the session check); `lose(params, index)` loses the
+// acknowledgement of an Input command that reached the page. Returns the
+// Input commands that reached chrome.debugger.
+function chromeTransport({ refuse = () => null, lose = () => false } = {}) {
+ const delivered = [];
+ let index = 0;
+ cdpClient.sessions.set(17, { tabId: 17, attached: true });
+ cdpClient.attach = async () => {};
+ cdpClient.armFileInputClickGuard = async () => {};
+ cdpClient.consumeFileInputClickGuard = async () => ({ blocked: false });
+ cdpClient._beforeCommand = async function (tabId, method, params, sessionId) {
+ if (method === 'Input.dispatchMouseEvent') {
+ const refusal = refuse(params, index);
+ if (refusal) { index++; throw refusal(); }
+ }
+ return original.beforeCommand.call(this, tabId, method, params, sessionId);
+ };
+ api.debugger.sendCommand = (_target, method, params, callback) => {
+ let lost = false;
+ if (method === 'Input.dispatchMouseEvent') {
+ delivered.push({ type: params.type, x: params.x, y: params.y, buttons: params.buttons });
+ lost = lose(params, index++);
+ }
+ Promise.resolve().then(() => {
+ if (lost) api.runtime.lastError = { message: lostAck };
+ try { callback({}); } finally { api.runtime.lastError = null; }
+ });
+ };
+ return delivered;
+}
+
+function restoreChrome() {
+ Object.assign(cdpClient, { attach: original.attach, _beforeCommand: original.beforeCommand,
+ resolveSelector: original.resolveSelector, armFileInputClickGuard: original.arm, consumeFileInputClickGuard: original.consume });
+ cdpClient.sessions.delete(17);
+ api.debugger.sendCommand = () => { throw new Error('debugger recorder not installed'); };
+ api.tabs.sendMessage = async () => ({});
+}
+
+function chromeAgent() {
+ const agent = new ChromeAgent({ getActive: () => ({ model: 'boundary-test' }) });
+ agent._currentUrl = async () => 'https://boundary.test/start';
+ agent._clickProgressSnapshot = async () => '';
+ agent._annotateClickProgress = async (_t, _n, _a, response) => response;
+ agent._recordInteractionRect = () => {};
+ agent._isPdfTab = async () => false;
+ agent._getDevDocumentIdentity = async () => ({ documentId: 'boundary-document', pageUrl: 'https://boundary.test/start' });
+ return agent;
+}
+
+function chromeContent(extra = {}) {
+ const messages = [];
+ api.tabs.sendMessage = async (_tab, message) => {
+ messages.push(message.action);
+ if (message.action === 'ax_resolve_rect') {
+ return { success: true, ref_id: 'ref_1', tag: 'button', name: 'Menu', x: 60, y: 40, inViewport: true, rect: { x: 20, y: 20, w: 80, h: 40 } };
+ }
+ if (message.action === 'ax_resolve_two_rects') {
+ return { success: true, from: { ref_id: 'ref_1', x: 60, y: 40, inViewport: true, name: 'Card A' },
+ to: { ref_id: 'ref_2', x: 60, y: 240, inViewport: true, name: 'Lane B' } };
+ }
+ if (message.action === 'set_checked') return extra.setChecked?.(message.params) ?? { success: true };
+ return { success: true };
+ };
+ return messages;
+}
+
+const presses = delivered => delivered.filter(event => event.type === 'mousePressed');
+const run = (agent, name, args) => {
+ const state = { started: false };
+ return agent.executeTool(17, name, args, null, { _contentActionDispatchState: state }).then(result => ({ result, state }));
+};
+
+function assertNotDispatched({ result, state }, label) {
+ assert.equal(result.success, false, `${label}: ${JSON.stringify(result)}`);
+ assert.equal(result.dispatched, false, `${label}: ${JSON.stringify(result)}`);
+ assert.equal(result.noDispatch, true, label);
+ assert.notEqual(result.outcomeUnknown, true, label);
+ assert.equal(state.started, false, `${label}: the call never entered dispatch`);
+}
+
+function assertUncertain({ result, state }, label) {
+ assert.equal(result.success, false, `${label}: ${JSON.stringify(result)}`);
+ assert.equal(result.dispatched, true, `${label}: ${JSON.stringify(result)}`);
+ assert.notEqual(result.noDispatch, true, label);
+ assert.equal(result.outcomeUnknown, true, label);
+ assert.equal(result.retryable, false, label);
+ assert.equal(state.started, true, label);
+}
+
+test('Chrome hover: dispatch starts at the first pointer move\'s transport boundary', async t => {
+ await t.test('a fence veto before the first move is not dispatched', async () => {
+ const delivered = chromeTransport({ refuse: () => pageFeedbackVeto });
+ chromeContent();
+ try {
+ const outcome = await run(chromeAgent(), 'hover', { ref_id: 'ref_1' });
+ assertNotDispatched(outcome, 'hover fence veto');
+ assert.equal(outcome.result.pageFeedbackPending, true);
+ assert.deepEqual(delivered, [], 'no pointer input reached the page');
+ } finally { restoreChrome(); }
+ });
+
+ await t.test('a transport refusal before the protocol call is not dispatched', async () => {
+ const delivered = chromeTransport({ refuse: () => notAttached });
+ chromeContent();
+ try {
+ assertNotDispatched(await run(chromeAgent(), 'hover', { ref_id: 'ref_1' }), 'hover transport refusal');
+ assert.deepEqual(delivered, []);
+ } finally { restoreChrome(); }
+ });
+
+ await t.test('a lost acknowledgement after the first move is dispatched with an unknown outcome and not resent', async () => {
+ const delivered = chromeTransport({ lose: (_params, index) => index === 1 });
+ chromeContent();
+ try {
+ assertUncertain(await run(chromeAgent(), 'hover', { ref_id: 'ref_1' }), 'hover lost acknowledgement');
+ assert.deepEqual(delivered.map(event => event.type), ['mouseMoved', 'mouseMoved'], 'each move was sent once');
+ } finally { restoreChrome(); }
+ });
+
+ await t.test('a refusal of the second move after the first was sent stays uncertain', async () => {
+ const delivered = chromeTransport({ refuse: (_params, index) => index === 1 ? pageFeedbackVeto : null });
+ chromeContent();
+ try {
+ const outcome = await run(chromeAgent(), 'hover', { ref_id: 'ref_1' });
+ assertUncertain(outcome, 'hover second-move veto');
+ assert.deepEqual(delivered.map(event => event.type), ['mouseMoved']);
+ } finally { restoreChrome(); }
+ });
+
+ await t.test('a normal hover sends two moves once and is dispatched', async () => {
+ const delivered = chromeTransport();
+ chromeContent();
+ try {
+ const { result, state } = await run(chromeAgent(), 'hover', { ref_id: 'ref_1' });
+ assert.equal(result.success, true, JSON.stringify(result));
+ assert.equal(state.started, true);
+ assert.deepEqual(delivered.map(event => [event.type, event.x, event.y]), [['mouseMoved', 50, 40], ['mouseMoved', 60, 40]]);
+ } finally { restoreChrome(); }
+ });
+});
+
+test('Chrome drag_drop: dispatch starts when the press passes its fences', async t => {
+ for (const [label, refuse] of [
+ ['a fence veto on the pointer move before the press', (_params, index) => index === 0 ? pageFeedbackVeto : null],
+ ['a fence veto on the press', params => params.type === 'mousePressed' ? pageFeedbackVeto : null],
+ ]) await t.test(`${label} is not dispatched`, async () => {
+ const delivered = chromeTransport({ refuse });
+ chromeContent();
+ try {
+ const outcome = await run(chromeAgent(), 'drag_drop', { fromRefId: 'ref_1', toRefId: 'ref_2', steps: 3 });
+ assertNotDispatched(outcome, label);
+ assert.equal(outcome.result.pageFeedbackPending, true);
+ assert.equal(presses(delivered).length, 0, 'no press reached the page');
+ } finally { restoreChrome(); }
+ });
+
+ await t.test('a transport refusal before the press protocol call is not dispatched', async () => {
+ const delivered = chromeTransport({ refuse: params => params.type === 'mousePressed' ? notAttached : null });
+ chromeContent();
+ try {
+ assertNotDispatched(await run(chromeAgent(), 'drag_drop', { fromRefId: 'ref_1', toRefId: 'ref_2', steps: 3 }), 'drag transport refusal');
+ assert.equal(presses(delivered).length, 0);
+ } finally { restoreChrome(); }
+ });
+
+ for (const [label, lose] of [
+ ['a lost press acknowledgement', params => params.type === 'mousePressed'],
+ ['a lost waypoint acknowledgement after the press', params => params.type === 'mouseMoved' && params.buttons === 1 && params.x >= 0],
+ ]) await t.test(`${label} is dispatched with an unknown outcome, released, and never resent`, async () => {
+ const delivered = chromeTransport({ lose });
+ chromeContent();
+ try {
+ assertUncertain(await run(chromeAgent(), 'drag_drop', { fromRefId: 'ref_1', toRefId: 'ref_2', steps: 3 }), label);
+ assert.equal(presses(delivered).length, 1, 'exactly one press');
+ assert.equal(delivered.at(-1).type, 'mouseReleased', 'the possibly held button was released');
+ assert.equal(delivered.filter(event => event.type === 'mouseMoved' && event.buttons === 1 && event.x >= 0).length,
+ label.includes('waypoint') ? 1 : 0, 'no waypoint was resent');
+ } finally { restoreChrome(); }
+ });
+
+ await t.test('a normal drag presses once and is dispatched', async () => {
+ const delivered = chromeTransport();
+ chromeContent();
+ try {
+ const { result, state } = await run(chromeAgent(), 'drag_drop', { fromRefId: 'ref_1', toRefId: 'ref_2', steps: 3 });
+ assert.equal(result.success, true, JSON.stringify(result));
+ assert.equal(state.started, true);
+ assert.deepEqual(delivered.map(event => event.type),
+ ['mouseMoved', 'mousePressed', 'mouseMoved', 'mouseMoved', 'mouseMoved', 'mouseReleased']);
+ } finally { restoreChrome(); }
+ });
+});
+
+test('Chrome set_checked: the trusted click is dispatched at its press transport boundary', async t => {
+ const checkbox = { setChecked: params => params.cleanupMarker
+ ? { success: true, method: 'set_checked', tag: 'input', type: 'checkbox', checkboxIdentity: 'id:agree', checkedBefore: false, checkedAfter: false }
+ : { success: true, dispatched: false, noDispatch: true, needsTrustedClick: true, marker: 'boundary-marker',
+ trustedSelector: '[data-webbrain-set-checked-target="boundary-marker"]', method: 'set_checked', tag: 'input', type: 'checkbox',
+ checkboxIdentity: 'id:agree', checkedBefore: false, checkedAfter: false, desiredChecked: true, rect: { x: 20, y: 20, w: 16, h: 16 } } };
+ const resolveCheckbox = () => {
+ cdpClient.resolveSelector = async () => ({ inViewport: true, hitOk: true, x: 28, y: 28, width: 16, height: 16, tag: 'INPUT', text: '' });
+ };
+
+ for (const [label, refuse] of [
+ ['a fence veto on the press', params => params.type === 'mousePressed' ? pageFeedbackVeto : null],
+ ['a transport refusal before the press protocol call', params => params.type === 'mousePressed' ? notAttached : null],
+ ]) await t.test(`${label} is not dispatched`, async () => {
+ const delivered = chromeTransport({ refuse });
+ const messages = chromeContent(checkbox);
+ resolveCheckbox();
+ try {
+ const outcome = await run(chromeAgent(), 'set_checked', { ref_id: 'ref_3', checked: true });
+ assertNotDispatched(outcome, label);
+ assert.equal(presses(delivered).length, 0, 'no press reached the page');
+ assert.deepEqual(messages, ['set_checked', 'set_checked'], 'probe and read-only verification only');
+ } finally { restoreChrome(); }
+ });
+
+ await t.test('a lost release acknowledgement after the press is dispatched with an unknown outcome and never resent', async () => {
+ const delivered = chromeTransport({ lose: params => params.type === 'mouseReleased' && params.x >= 0 });
+ chromeContent(checkbox);
+ resolveCheckbox();
+ try {
+ assertUncertain(await run(chromeAgent(), 'set_checked', { ref_id: 'ref_3', checked: true }), 'set_checked lost release');
+ assert.equal(presses(delivered).length, 1, 'exactly one trusted press, no fallback click');
+ } finally { restoreChrome(); }
+ });
+});
+
+// ── Firefox ────────────────────────────────────────────────────────────────
+function firefoxAgent() {
+ const agent = new FirefoxAgent({ getActive: () => ({ model: 'boundary-test' }) });
+ agent._runningTabs.add(17);
+ agent._currentUrl = async () => 'https://boundary.test/start';
+ agent._isPdfTab = async () => false;
+ agent._injectCoreContentScripts = async () => {};
+ return agent;
+}
+
+// `monitor(action, params)` answers page-monitor fences; `content(message)`
+// answers content messages (or throws a browser messaging error).
+function firefoxMessaging({ monitor = () => ({ ready: true }), content = () => ({ success: true }) } = {}) {
+ const contentMessages = [];
+ api.tabs.sendMessage = async (_tab, message) => {
+ if (message.action?.startsWith('page_monitor_')) return monitor(message.action, message.params || {});
+ contentMessages.push({ action: message.action, prepare: !!message.params?._bidiPrepare });
+ return content(message);
+ };
+ return contentMessages;
+}
+
+function bidiRun({ request } = {}) {
+ const requests = [];
+ firefoxBidi.runs.set(17, { runId: 'boundary-run', bound: true });
+ firefoxBidi.request = async (command, args) => { requests.push({ command, action: args.action }); return request?.(command, args) ?? { success: true, dispatched: true, method: 'firefox-bidi' }; };
+ return requests;
+}
+
+function restoreFirefox() {
+ firefoxBidi.runs.delete(17);
+ delete firefoxBidi.request;
+ api.tabs.sendMessage = async () => ({});
+}
+
+const bidiPrepared = message => message.params?._bidiPrepare
+ ? { bidiPrepared: true, success: false, dispatched: false, noDispatch: true, url: 'https://boundary.test/start', rect: { x: 20, y: 20, w: 80, h: 40 } }
+ : { success: true, method: 'synthetic-hover' };
+
+async function withFeedbackRun(agent, body) {
+ await agent._beginPageFeedbackRun(17, 'interactive');
+ try { return await body(); } finally { agent._finishPageFeedbackRun(17); }
+}
+
+test('Firefox hover through BiDi: dispatch starts when the native request is posted', async t => {
+ await t.test('a fence veto at the native request is not dispatched', async () => {
+ let prepared = false;
+ const contentMessages = firefoxMessaging({ content: bidiPrepared, monitor: (action, params) => {
+ if (action === 'page_monitor_prepare' && params.fenceOnly) { prepared = true; return { ready: true, modelBindingValid: true }; }
+ // The native request's own fence (after its preparation probe) refuses.
+ return action === 'page_monitor_dispatch' && prepared
+ ? { ready: false, pageFeedbackPending: true, preparationVeto: { predicate: 'geometry_changed' } } : { ready: true };
+ } });
+ const requests = bidiRun();
+ const agent = firefoxAgent();
+ try {
+ const outcome = await withFeedbackRun(agent, () => run(agent, 'hover', { ref_id: 'ref_1' }));
+ assertNotDispatched(outcome, 'BiDi hover fence veto');
+ assert.equal(outcome.result.pageFeedbackPending, true);
+ assert.deepEqual(requests, [], 'no native input request was posted');
+ assert.deepEqual(contentMessages, [{ action: 'hover', prepare: true }], 'only the read-only preparation probe');
+ } finally { restoreFirefox(); }
+ });
+
+ await t.test('a native port that cannot accept the request is not dispatched', async () => {
+ firefoxMessaging({ content: bidiPrepared });
+ firefoxBidi.runs.set(17, { runId: 'boundary-run', bound: true });
+ const previous = firefoxBidi.apiOverride;
+ firefoxBidi.apiOverride = { ...api, runtime: { connectNative: () => ({
+ onMessage: { addListener() {} }, onDisconnect: { addListener() {} }, disconnect() {},
+ postMessage() { throw new Error('Attempt to postMessage on disconnected port'); },
+ }) } };
+ try {
+ assertNotDispatched(await run(firefoxAgent(), 'hover', { ref_id: 'ref_1' }), 'BiDi hover port refusal');
+ } finally { firefoxBidi.apiOverride = previous; firefoxBidi.port = null; restoreFirefox(); }
+ });
+
+ await t.test('a companion refusal before any input is not dispatched', async () => {
+ firefoxMessaging({ content: bidiPrepared });
+ bidiRun({ request: () => { throw Object.assign(new Error('Page changed during native preparation; no further input sent'), {
+ code: 'page_feedback_pending', dispatchState: { dispatched: false, noDispatch: true, outcomeUnknown: false, retryable: true } }); } });
+ try {
+ const outcome = await run(firefoxAgent(), 'hover', { ref_id: 'ref_1' });
+ assertNotDispatched(outcome, 'BiDi hover companion refusal');
+ assert.equal(outcome.result.pageFeedbackPending, true);
+ } finally { restoreFirefox(); }
+ });
+
+ await t.test('a lost native reply after the request was posted is dispatched with an unknown outcome and never resent', async () => {
+ const contentMessages = firefoxMessaging({ content: bidiPrepared });
+ const requests = bidiRun({ request: () => { throw new Error('Firefox companion disconnected; do not retry an uncertain action'); } });
+ try {
+ assertUncertain(await run(firefoxAgent(), 'hover', { ref_id: 'ref_1' }), 'BiDi hover lost reply');
+ assert.deepEqual(requests.map(request => request.action), ['hover'], 'one native request');
+ assert.equal(contentMessages.filter(message => !message.prepare).length, 0, 'no synthetic content-script fallback');
+ } finally { restoreFirefox(); }
+ });
+
+ await t.test('a delivered native hover is dispatched', async () => {
+ firefoxMessaging({ content: bidiPrepared });
+ const requests = bidiRun();
+ try {
+ const { result, state } = await run(firefoxAgent(), 'hover', { ref_id: 'ref_1' });
+ assert.equal(result.success, true, JSON.stringify(result));
+ assert.equal(state.started, true);
+ assert.equal(requests.length, 1);
+ } finally { restoreFirefox(); }
+ });
+});
+
+test('Firefox content-script hover, drag_drop and set_checked: dispatch starts at the content message', async t => {
+ const lostResponse = "Actor 'Conduits' destroyed before query 'RuntimeMessage' was resolved";
+ const missingReceiver = 'Could not establish connection. Receiving end does not exist.';
+ for (const [name, args] of [
+ ['hover', { ref_id: 'ref_1' }],
+ ['drag_drop', { fromRefId: 'ref_1', toRefId: 'ref_2' }],
+ ['set_checked', { ref_id: 'ref_3', checked: true }],
+ ]) {
+ await t.test(`${name}: a fence veto before the content message is not dispatched`, async () => {
+ const contentMessages = firefoxMessaging({ monitor: action => action === 'page_monitor_dispatch'
+ ? { ready: false, pageFeedbackPending: true, preparationVeto: { predicate: 'pending_feedback' } } : { ready: true } });
+ const agent = firefoxAgent();
+ try {
+ const outcome = await withFeedbackRun(agent, () => run(agent, name, args));
+ assertNotDispatched(outcome, `${name} fence veto`);
+ assert.deepEqual(contentMessages, [], 'no content message was sent');
+ } finally { restoreFirefox(); }
+ });
+
+ await t.test(`${name}: a content-side fence veto is not dispatched`, async () => {
+ firefoxMessaging({ content: () => ({ success: false, dispatched: false, noDispatch: true, pageFeedbackPending: true,
+ error: `${name} failed: Browser changed during action preparation. Re-observe before acting.` }) });
+ try {
+ assertNotDispatched(await run(firefoxAgent(), name, args), `${name} content veto`);
+ } finally { restoreFirefox(); }
+ });
+
+ await t.test(`${name}: a missing receiver is not dispatched`, async () => {
+ const contentMessages = firefoxMessaging({ content: () => { throw new Error(missingReceiver); } });
+ try {
+ assertNotDispatched(await run(firefoxAgent(), name, args), `${name} missing receiver`);
+ assert.equal(contentMessages.length, 2, 'one reinjected attempt; nothing was delivered');
+ } finally { restoreFirefox(); }
+ });
+
+ await t.test(`${name}: a lost content response is dispatched with an unknown outcome and never resent`, async () => {
+ const contentMessages = firefoxMessaging({ content: () => { throw new Error(lostResponse); } });
+ try {
+ assertUncertain(await run(firefoxAgent(), name, args), `${name} lost response`);
+ assert.equal(contentMessages.length, 1, 'the content action was sent once');
+ } finally { restoreFirefox(); }
+ });
+ }
+});
diff --git a/test/run.js b/test/run.js
index ff8ba2fe0..f81cbd430 100644
--- a/test/run.js
+++ b/test/run.js
@@ -3345,7 +3345,8 @@ test('Chrome set_checked completes one selector-backed trusted click and verifie
assert.equal(options.trustedOnly, true);
assert.equal(options.requireUnique, true);
assert.ok(options.abortSignal instanceof AbortSignal);
- assert.equal(typeof options.beforeDispatch, 'function');
+ // Dispatch is recorded at the press transport boundary, never before its fence.
+ assert.equal(typeof options.onDispatch, 'function');
trustedClickCompletedAt = Date.now();
return { success: true, method: 'cdp-mouse', rect: { x: 1, y: 2, w: 20, h: 20 } };
};
@@ -3588,7 +3589,7 @@ test('Chrome set_checked fails closed when marker verification resolves another
assert.equal(options.trustedOnly, true);
assert.equal(options.requireUnique, true);
assert.ok(options.abortSignal instanceof AbortSignal);
- assert.equal(typeof options.beforeDispatch, 'function');
+ assert.equal(typeof options.onDispatch, 'function');
return { success: true, method: 'cdp-mouse' };
};
@@ -114874,7 +114875,8 @@ test('Chrome bounds trusted checkbox recovery before and during the CDP click',
};
cdpClientCh.attach = async () => ({ attached: true });
cdpClientCh.clickElement = async (_tabId, _selector, options) => {
- options.beforeDispatch();
+ // The press reached its transport boundary (after its fences).
+ options.onDispatch();
markClickStarted();
return delayedClick;
};