Skip to content

armgit / orca

Public Imported from GitHub
0

Orca is the place for working with a fleet of agents (e.g. Grok Bot, Claude Code). Run any coding agent with your own subscription. Available on desktop, mobile and remote runtime.

fix(browser): agent fill, select, upload and type change the page or fail with a reason (#36263)

Kelvin Amoabab67c5254f044
diff --git a/skill-guides/orca-cli/references/browser.md b/skill-guides/orca-cli/references/browser.md
index a8435f8a8ce..cc84303d277 100644
--- a/skill-guides/orca-cli/references/browser.md
+++ b/skill-guides/orca-cli/references/browser.md
@@ -60,12 +60,14 @@ Browser rules:
- Use `ORCA set device` or `ORCA viewport --mobile`, not `ORCA exec --command "set device ..."`, so the page also gets touch input and `pointer: coarse`. Each page load (reload, `goto`, link click) resets it, so run it again after navigating.
- Prefer `wait --text`, `--url`, `--selector`, or `--load` after async page changes instead of bare timeouts.
- Anything not listed above goes through `ORCA exec --page <browserPageId> --command "<agent-browser command>"`.
-- If `fill` or `type` fails on a custom input, try `ORCA focus --page <browserPageId> --element @e1 --json` then `ORCA inserttext --page <browserPageId> --text "text" --json`.
+- `fill` sets a field's whole value in one step, at any size. `type` presses real keys into the focused element, so `minlength`/`maxlength`, key handlers, Enter and Tab act as they would for a person; `ORCA focus` the element first. Use `type` when the page reacts to keystrokes.
+- On server-hosted pages `fill` and `type` still insert text the engine's own way. If either does nothing on a custom input there, try `ORCA focus --page <browserPageId> --element @e1 --json` then `ORCA inserttext --page <browserPageId> --text "text" --json`.
- A client-hosted page renders in the paired desktop's browser engine, so every command against it needs that desktop online and returns `browser_host_unavailable` while it is closed, asleep, or disconnected. Server-hosted pages run with no desktop attached; prefer them for long or unattended automation.
Common recoveries:
- `browser_no_tab`: open a tab with `ORCA tab create --url <url> --json`.
- `browser_stale_ref`: run `ORCA snapshot --page <browserPageId> --json` and retry on the same page with fresh refs.
+- `browser_element_not_interactable`: the element can't take that command, and the message says why (not a text field, disabled, read-only, not a `<select>`, not a file input). Re-snapshot and pick the right ref; for a custom dropdown, click it and then click the option.
- `browser_tab_not_found`: run `ORCA tab list --json` before switching or closing.
- `browser_host_unavailable`: the desktop hosting the page is offline. Bring it back, or recreate the page with server placement if the work must outlive the desktop session.
diff --git a/src/cli/bundled-skill-guides.ts b/src/cli/bundled-skill-guides.ts
index 2b649c28076..ad8a7298850 100644
--- a/src/cli/bundled-skill-guides.ts
+++ b/src/cli/bundled-skill-guides.ts
@@ -871,13 +871,15 @@ const ORCA_CLI_BROWSER_REFERENCE_MARKDOWN = [
"- Use `ORCA set device` or `ORCA viewport --mobile`, not `ORCA exec --command \"set device ...\"`, so the page also gets touch input and `pointer: coarse`. Each page load (reload, `goto`, link click) resets it, so run it again after navigating.",
"- Prefer `wait --text`, `--url`, `--selector`, or `--load` after async page changes instead of bare timeouts.",
"- Anything not listed above goes through `ORCA exec --page <browserPageId> --command \"<agent-browser command>\"`.",
- "- If `fill` or `type` fails on a custom input, try `ORCA focus --page <browserPageId> --element @e1 --json` then `ORCA inserttext --page <browserPageId> --text \"text\" --json`.",
+ "- `fill` sets a field's whole value in one step, at any size. `type` presses real keys into the focused element, so `minlength`/`maxlength`, key handlers, Enter and Tab act as they would for a person; `ORCA focus` the element first. Use `type` when the page reacts to keystrokes.",
+ "- On server-hosted pages `fill` and `type` still insert text the engine's own way. If either does nothing on a custom input there, try `ORCA focus --page <browserPageId> --element @e1 --json` then `ORCA inserttext --page <browserPageId> --text \"text\" --json`.",
"- A client-hosted page renders in the paired desktop's browser engine, so every command against it needs that desktop online and returns `browser_host_unavailable` while it is closed, asleep, or disconnected. Server-hosted pages run with no desktop attached; prefer them for long or unattended automation.",
"",
"Common recoveries:",
"",
"- `browser_no_tab`: open a tab with `ORCA tab create --url <url> --json`.",
"- `browser_stale_ref`: run `ORCA snapshot --page <browserPageId> --json` and retry on the same page with fresh refs.",
+ "- `browser_element_not_interactable`: the element can't take that command, and the message says why (not a text field, disabled, read-only, not a `<select>`, not a file input). Re-snapshot and pick the right ref; for a custom dropdown, click it and then click the option.",
"- `browser_tab_not_found`: run `ORCA tab list --json` before switching or closing.",
"- `browser_host_unavailable`: the desktop hosting the page is offline. Bring it back, or recreate the page with server placement if the work must outlive the desktop session.",
"",
diff --git a/src/main/browser/agent-browser-bridge-core-commands.ts b/src/main/browser/agent-browser-bridge-core-commands.ts
index 53f3185c1b4..a2a1ce407bb 100644
--- a/src/main/browser/agent-browser-bridge-core-commands.ts
+++ b/src/main/browser/agent-browser-bridge-core-commands.ts
@@ -8,7 +8,7 @@ import { assertClipboardTextWriteWithinLimitWithYield } from '../../shared/clipb
import { normalizeBrowserNavigationUrl } from '../../shared/browser-url'
import { BrowserError } from './cdp-bridge'
import { ORCA_TAB_SESSION_PREFIX } from './agent-browser-orphan-sweep'
-import { focusedValueSetExpression } from './agent-browser-bridge-input'
+import { FILL_ELEMENT_FUNCTION } from './browser-element-functions'
import { EMBEDDED_NAVIGATION_TIMEOUT_MS } from './agent-browser-bridge-types'
import {
isAbortedNavigationError,
@@ -159,22 +159,14 @@ export abstract class AgentBrowserBridgeCoreCommands extends AgentBrowserBridgeQ
browserPageId?: string
): Promise<BrowserFillResult> {
await assertClipboardTextWriteWithinLimitWithYield(value)
- // agent-browser's CDP insertion loses focus in Electron guests; use the browser input pipeline.
+ // agent-browser's fill types into whatever holds focus and sends text per character; set the
+ // resolved element's value in one call instead.
return this.enqueueTargetedCommand(
worktreeId,
browserPageId,
async (sessionName) => {
- if (!(await this.isExplicitContentEditableTarget(sessionName, element))) {
- await this.execAgentBrowser(sessionName, ['focus', element])
- // One stdin edit avoids argv limits and repeated copying of the growing field value.
- await this.execAgentBrowser(sessionName, ['eval', '--stdin'], {
- stdinText: focusedValueSetExpression(JSON.stringify(value), { dispatchEvents: true })
- })
- return { filled: element } as BrowserFillResult
- }
-
- await this.fillExplicitContentEditable(sessionName, element, value)
- return { filled: element } as BrowserFillResult
+ await this.actOnElement(sessionName, element, 'fill', FILL_ELEMENT_FUNCTION, [value])
+ return { filled: element }
},
{ requireScopedTarget: true, elements: [element] }
)
diff --git a/src/main/browser/agent-browser-bridge-execution.ts b/src/main/browser/agent-browser-bridge-execution.ts
index 7faee8ef652..c77d32425ed 100644
--- a/src/main/browser/agent-browser-bridge-execution.ts
+++ b/src/main/browser/agent-browser-bridge-execution.ts
@@ -1,10 +1,9 @@
import { execFile } from 'node:child_process'
import type { WebContents } from 'electron'
import { BrowserError } from './cdp-bridge'
-import {
- focusedRichTextEditExpression,
- isExplicitContentEditableResult
-} from './agent-browser-bridge-input'
+import type { CapturedElement } from './cdp-element-capture'
+import { sendGuestCdpCommand } from './guest-cdp-command'
+import { readRecord, readString } from './guest-frame-input-sessions'
import {
isTabClosedTransportError,
pageUnavailableMessageForSession
@@ -170,29 +169,73 @@ export abstract class AgentBrowserBridgeExecution extends AgentBrowserBridgeTabs
return this.runAgentBrowserRaw(sessionName, args, execOptions)
}
- protected async isExplicitContentEditableTarget(
- sessionName: string,
- element: string
- ): Promise<boolean> {
- const result = await this.execAgentBrowser(sessionName, [
- 'get',
- 'attr',
- element,
- 'contenteditable'
- ])
- return isExplicitContentEditableResult(result)
- }
-
- protected async fillExplicitContentEditable(
+ /** Runs a browser-element-functions action on the element the helper resolves, in its own frame. */
+ protected async actOnElement(
sessionName: string,
element: string,
- value: string
- ): Promise<void> {
- await this.execAgentBrowser(sessionName, ['focus', element])
- // Send one stdin edit to avoid argv limits and focus changes between chunks.
- await this.execAgentBrowser(sessionName, ['eval', '--stdin'], {
- stdinText: focusedRichTextEditExpression(JSON.stringify(value), { selectAll: true })
+ action: string,
+ functionDeclaration: string,
+ args: unknown[] = []
+ ): Promise<unknown> {
+ const session = this.sessions.get(sessionName)
+ if (!session) {
+ throw this.createPageUnavailableError(sessionName)
+ }
+ session.proxy.elementCapture.start()
+ session.proxy.pointAtNextNode(true)
+ let captured: CapturedElement | null
+ try {
+ // Any element read makes the helper resolve the ref or selector, in whichever frame owns it.
+ await this.execAgentBrowser(sessionName, ['get', 'attr', element, 'id'])
+ } finally {
+ session.proxy.pointAtNextNode(false)
+ captured = session.proxy.elementCapture.stop()
+ }
+ const wc = this.getWebContents(session.webContentsId)
+ if (!wc) {
+ throw this.createPageUnavailableError(sessionName)
+ }
+ if (!captured) {
+ throw new BrowserError('browser_error', `Could not resolve ${element} in the page`)
+ }
+ const response = await sendGuestCdpCommand(
+ wc,
+ 'Runtime.callFunctionOn',
+ {
+ objectId: captured.objectId,
+ functionDeclaration,
+ arguments: args.map((value) => ({ value })),
+ returnByValue: true
+ },
+ captured.sessionId
+ ).catch((error: unknown) => {
+ if (error instanceof BrowserError) {
+ throw error
+ }
+ if (!this.getWebContents(session.webContentsId)) {
+ throw this.createPageUnavailableError(sessionName)
+ }
+ throw new BrowserError(
+ 'browser_cdp_error',
+ error instanceof Error ? error.message : String(error)
+ )
})
+ const exception = readRecord(response, 'exceptionDetails')
+ if (exception) {
+ const description = readString(readRecord(exception, 'exception'), 'description')
+ throw new BrowserError(
+ 'browser_eval_error',
+ description ?? readString(exception, 'text') ?? 'Page function failed'
+ )
+ }
+ const outcome = readRecord(response, 'result')?.value
+ if (typeof outcome === 'string') {
+ throw new BrowserError(
+ 'browser_element_not_interactable',
+ `Cannot ${action} ${element}: ${outcome}`
+ )
+ }
+ return outcome
}
protected createPageUnavailableError(sessionName: string): BrowserError {
diff --git a/src/main/browser/agent-browser-bridge-host-revocation.test.ts b/src/main/browser/agent-browser-bridge-host-revocation.test.ts
index d889b1a8fe0..c96a976a8d0 100644
--- a/src/main/browser/agent-browser-bridge-host-revocation.test.ts
+++ b/src/main/browser/agent-browser-bridge-host-revocation.test.ts
@@ -85,6 +85,42 @@ describe('AgentBrowserBridge under a host revocation', () => {
bridge.setActiveTab(100)
})
+ it('stops a long type at the next run once the host is revoked', async () => {
+ const wc = mockWebContents(100)
+ wc.getType.mockReturnValue('webview')
+ let resumeTyping = (): void => {}
+ const paused = new Promise<void>((resolve) => {
+ resumeTyping = resolve
+ })
+ let evaluations = 0
+ wc.debugger.sendCommand.mockImplementation(async (method: string) => {
+ if (method === 'Runtime.evaluate') {
+ evaluations += 1
+ // The first run's settle wait: hold it while the host is revoked.
+ if (evaluations === 2) {
+ await paused
+ }
+ return { result: { type: 'object', subtype: 'null' } }
+ }
+ return {}
+ })
+ webContentsFromIdMock.mockReturnValue(wc)
+ const revoked = new AbortController()
+
+ const typing = runInRemoteCliBrowserScope(
+ WORKSPACE,
+ () => bridge.type('x'.repeat(2000), undefined, 'tab-1'),
+ revoked.signal
+ )
+ await vi.waitFor(() => expect(evaluations).toBe(2))
+ const sentBeforeRevocation = wc.sendInputEvent.mock.calls.length
+ revoked.abort()
+ resumeTyping()
+
+ await expect(typing).rejects.toMatchObject({ code: 'desktop_browser_bridge_unavailable' })
+ expect(wc.sendInputEvent.mock.calls.length).toBe(sentBeforeRevocation)
+ })
+
it('never presses the mouse once the host is revoked during a running click', async () => {
const wc = mockWebContents(100)
let answerHitTest = (): void => {}
diff --git a/src/main/browser/agent-browser-bridge-input-commands.ts b/src/main/browser/agent-browser-bridge-input-commands.ts
index 242fdbc7b99..c623655cd3a 100644
--- a/src/main/browser/agent-browser-bridge-input-commands.ts
+++ b/src/main/browser/agent-browser-bridge-input-commands.ts
@@ -1,36 +1,11 @@
-import type {
- BrowserTypeResult,
- BrowserSelectResult,
- BrowserScrollResult
-} from '../../shared/runtime-types'
+import type { BrowserSelectResult, BrowserScrollResult } from '../../shared/runtime-types'
import { assertClipboardTextWriteWithinLimitWithYield } from '../../shared/clipboard-text'
import { iterateBrowserTextInsertionChunks } from '../../shared/browser-text-insertion-chunks'
import { AGENT_BROWSER_TEXT_ARGUMENT_MAX_BYTES } from './agent-browser-bridge-types'
import { AgentBrowserBridgeCoreCommands } from './agent-browser-bridge-core-commands'
+import { SELECT_OPTION_TARGET_FUNCTION } from './browser-element-functions'
export abstract class AgentBrowserBridgeInputCommands extends AgentBrowserBridgeCoreCommands {
- async type(
- input: string,
- worktreeId?: string,
- browserPageId?: string
- ): Promise<BrowserTypeResult> {
- await assertClipboardTextWriteWithinLimitWithYield(input)
- return this.enqueueTargetedCommand(
- worktreeId,
- browserPageId,
- async (sessionName) => {
- for (const chunk of iterateBrowserTextInsertionChunks(
- input,
- AGENT_BROWSER_TEXT_ARGUMENT_MAX_BYTES
- )) {
- await this.execAgentBrowser(sessionName, ['keyboard', 'type', chunk])
- }
- return { typed: true } as BrowserTypeResult
- },
- { requireScopedTarget: true }
- )
- }
-
async select(
element: string,
value: string,
@@ -41,6 +16,13 @@ export abstract class AgentBrowserBridgeInputCommands extends AgentBrowserBridge
worktreeId,
browserPageId,
async (sessionName) => {
+ // The helper reports success for any element; only a <select> has options to choose.
+ await this.actOnElement(
+ sessionName,
+ element,
+ 'select an option in',
+ SELECT_OPTION_TARGET_FUNCTION
+ )
return (await this.execAgentBrowser(sessionName, [
'select',
element,
diff --git a/src/main/browser/agent-browser-bridge-input.ts b/src/main/browser/agent-browser-bridge-input.ts
deleted file mode 100644
index e7338dc53f8..00000000000
--- a/src/main/browser/agent-browser-bridge-input.ts
+++ /dev/null
@@ -1,67 +0,0 @@
-export function focusedValueSetExpression(
- valueExpression: string,
- options?: { append?: boolean; dispatchEvents?: boolean }
-): string {
- const nextValue = options?.append
- ? ["String(target.value ?? '') + ", valueExpression].join('')
- : valueExpression
- const dispatchEvents = options?.dispatchEvents
- ? " target.dispatchEvent(new Event('input', { bubbles: true })); target.dispatchEvent(new Event('change', { bubbles: true }));"
- : ''
- return [
- '(() => { const el = document.activeElement; if (el) {',
- // Spinbutton wrappers can hold focus while another input owns the value.
- " const editableSelector = \"input:not([type='hidden']):not([type='button']):not([type='checkbox']):not([type='radio']):not([type='file']):not([type='image']):not([type='reset']):not([type='submit']), textarea\";",
- " const isEditable = (node) => !!node && (node.matches?.(editableSelector) ?? (node.tagName === 'TEXTAREA' || (node.tagName === 'INPUT' && !/^(hidden|button|checkbox|radio|file|image|reset|submit)$/i.test(node.getAttribute?.('type') ?? ''))));",
- ' const findEditable = (root) => root?.querySelector?.(editableSelector) ?? null;',
- ' let target = el;',
- " if (!isEditable(target) && target.getAttribute?.('role') === 'spinbutton') {",
- " const controls = target.getAttribute('aria-controls');",
- ' if (controls) { for (const id of controls.split(/\\s+/)) { if (!id) continue; const controlled = document.getElementById(id); if (isEditable(controlled)) { target = controlled; break; } const descendant = findEditable(controlled); if (descendant) { target = descendant; break; } } }',
- ' if (target === el) { const descendant = findEditable(target); if (descendant) target = descendant; }',
- ' }',
- " const nativeSetter = Object.getOwnPropertyDescriptor(Object.getPrototypeOf(target), 'value')?.set;",
- ' const nextValue = ',
- nextValue,
- '; if (nativeSetter) { nativeSetter.call(target, nextValue); } else { target.value = nextValue; }',
- dispatchEvents,
- ' } })()'
- ].join('')
-}
-
-// Rich editors require browser edit transactions to keep their models synchronized.
-export function focusedRichTextEditExpression(
- valueExpression: string,
- options?: { selectAll?: boolean }
-): string {
- const selectAll = options?.selectAll ? 'true' : 'false'
- return [
- '(() => {',
- ' const target = document.activeElement;',
- ' const value = ',
- valueExpression,
- ';',
- ` const selectAll = ${selectAll};`,
- " const isEditable = target?.isContentEditable === true || /^(|true|plaintext-only)$/i.test(target?.getAttribute?.('contenteditable') ?? 'false');",
- " if (!target || target === document.body || !isEditable) { throw new Error('Focused rich-text target is unavailable'); }",
- ' if (selectAll) {',
- " if (typeof window.getSelection !== 'function') { throw new Error('Rich-text selection is unavailable'); }",
- ' const selection = window.getSelection();',
- " if (!selection) { throw new Error('Rich-text selection is unavailable'); }",
- ' selection.selectAllChildren(target);',
- ' }',
- " const editCommand = selectAll && value.length === 0 ? 'delete' : 'insertText';",
- ' let edited = false;',
- ' try {',
- ' edited = document.execCommand(editCommand, false, value) === true;',
- ' } catch { edited = false; }',
- " if (!edited) { throw new Error('Browser rich-text editing command failed'); }",
- ' })()'
- ].join('')
-}
-
-export function isExplicitContentEditableResult(result: unknown): boolean {
- const value =
- result && typeof result === 'object' ? (result as { value?: unknown }).value : undefined
- return typeof value === 'string' && /^(|true|plaintext-only)$/i.test(value)
-}
diff --git a/src/main/browser/agent-browser-bridge-interaction-commands.ts b/src/main/browser/agent-browser-bridge-interaction-commands.ts
index 53c5cf4dc74..12b0e61c4b7 100644
--- a/src/main/browser/agent-browser-bridge-interaction-commands.ts
+++ b/src/main/browser/agent-browser-bridge-interaction-commands.ts
@@ -7,22 +7,24 @@ import type {
BrowserFocusResult,
BrowserClearResult,
BrowserSelectAllResult,
- BrowserKeypressResult,
BrowserPdfResult
} from '../../shared/runtime-types'
import { BrowserError } from './cdp-bridge'
+import {
+ FILE_INPUT_TARGET_FUNCTION,
+ FILL_ELEMENT_FUNCTION,
+ SELECT_ALL_ELEMENT_FUNCTION
+} from './browser-element-functions'
+import { assertUploadFilesExist } from './browser-upload-files'
import {
agentBrowserWaitArguments,
type AgentBrowserWaitOptions
} from './agent-browser-wait-arguments'
-import { acquireElectronDebugger } from './electron-debugger-lease'
-import { parseCdpKeyEvent, imeFallbackKeyEvent } from './cdp-keyboard-us-layout'
-import { dispatchPageKeyEvent, type GuestCdpSender } from './guest-direct-key-input'
-import { sendGuestCdpCommand } from './guest-cdp-command'
+import { parseCdpKeyEvent } from './cdp-keyboard-us-layout'
import { readRemoteCliAdmittedPage } from './remote-cli-page-admission'
-import { AgentBrowserBridgeCaptureCommands } from './agent-browser-bridge-capture-commands'
+import { AgentBrowserBridgeKeyCommands } from './agent-browser-bridge-key-commands'
-export abstract class AgentBrowserBridgeInteractionCommands extends AgentBrowserBridgeCaptureCommands {
+export abstract class AgentBrowserBridgeInteractionCommands extends AgentBrowserBridgeKeyCommands {
async hover(
element: string,
worktreeId?: string,
@@ -64,6 +66,8 @@ export abstract class AgentBrowserBridgeInteractionCommands extends AgentBrowser
worktreeId,
browserPageId,
async (sessionName) => {
+ await assertUploadFilesExist(filePaths)
+ await this.actOnElement(sessionName, element, 'upload to', FILE_INPUT_TARGET_FUNCTION)
return (await this.execAgentBrowser(sessionName, [
'upload',
element,
@@ -143,13 +147,7 @@ export abstract class AgentBrowserBridgeInteractionCommands extends AgentBrowser
worktreeId,
browserPageId,
async (sessionName) => {
- if (!(await this.isExplicitContentEditableTarget(sessionName, element))) {
- // Let the helper resolve ordinary refs inside iframes and shadow roots, including unfocusable fields.
- await this.execAgentBrowser(sessionName, ['fill', element, ''])
- return { cleared: element }
- }
-
- await this.fillExplicitContentEditable(sessionName, element, '')
+ await this.actOnElement(sessionName, element, 'clear', FILL_ELEMENT_FUNCTION, [''])
return { cleared: element }
},
{ requireScopedTarget: true, elements: [element] }
@@ -161,98 +159,25 @@ export abstract class AgentBrowserBridgeInteractionCommands extends AgentBrowser
worktreeId?: string,
browserPageId?: string
): Promise<BrowserSelectAllResult> {
- return this.enqueueTargetedCommand(
- worktreeId,
- browserPageId,
- async (sessionName) => {
- await this.execAgentBrowser(sessionName, ['focus', element])
- return (await this.execAgentBrowser(sessionName, [
- 'press',
- 'Control+a'
- ])) as BrowserSelectAllResult
- },
- { elements: [element] }
- )
- }
-
- async keypress(
- key: string,
- worktreeId?: string,
- browserPageId?: string
- ): Promise<BrowserKeypressResult> {
return this.enqueueTargetedCommand(
worktreeId,
browserPageId,
async (sessionName, target) => {
- const parsed = parseCdpKeyEvent(key) ?? imeFallbackKeyEvent(key)
- if (!parsed) {
- // Unknown keys need the helper; create its session only on this fallback path.
- await this.ensureSession(sessionName, target.browserPageId, target.webContentsId)
- return (await this.execAgentBrowser(sessionName, ['press', key])) as BrowserKeypressResult
- }
- const wc = this.getWebContents(target.webContentsId)
- if (!wc || wc.isDestroyed()) {
- throw new BrowserError(
- 'browser_tab_not_found',
- `Browser page ${target.browserPageId} is no longer available`
- )
- }
- const event = {
- windowsVirtualKeyCode: parsed.keyCode,
- nativeVirtualKeyCode: parsed.keyCode,
- key: parsed.key,
- code: parsed.code,
- modifiers: parsed.modifiers,
- location: parsed.location
- }
- let releaseDebugger = (): void => {}
- try {
- releaseDebugger = acquireElectronDebugger(wc).release
- await this.waitForPagePaint(wc, target.webContentsId)
- const send: GuestCdpSender = (method, params, sessionId) =>
- sendGuestCdpCommand(wc, method, params, sessionId)
- const guardFocus = (): Promise<() => void> =>
- this.browserManager.guardUserFocusForAgentInput(target.webContentsId)
- await dispatchPageKeyEvent(
- wc,
- {
- // keyDown without text makes Blink synthesize empty input; use rawKeyDown for editing keys.
- type: parsed.text === null ? 'rawKeyDown' : 'keyDown',
- ...event,
- ...(parsed.text === null ? {} : { text: parsed.text, unmodifiedText: parsed.text })
- },
- send,
- guardFocus
- )
- await dispatchPageKeyEvent(
- wc,
- {
- type: 'keyUp',
- ...event,
- // Blink clears a modifier's own bit on keyup.
- modifiers: parsed.modifiers & ~parsed.selfModifier
- },
- send,
- guardFocus
- )
- return { pressed: key }
- } catch (error) {
- // Browser error codes let the pane recognize and reclaim a dead page.
- if (error instanceof BrowserError) {
- throw error
- }
- if (!this.getWebContents(target.webContentsId)) {
- throw this.createPageUnavailableError(sessionName)
- }
- throw new BrowserError(
- 'browser_error',
- `Failed to press ${key} in browser page ${target.browserPageId}: ${error instanceof Error ? error.message : String(error)}`
- )
- } finally {
- releaseDebugger()
+ const focused = await this.actOnElement(
+ sessionName,
+ element,
+ 'select all in',
+ SELECT_ALL_ELEMENT_FUNCTION
+ )
+ // Code editors select their whole document from the shortcut, beyond the DOM they render;
+ // an element that can't take focus keeps the DOM selection instead.
+ const shortcut = parseCdpKeyEvent(process.platform === 'darwin' ? 'Meta+a' : 'Control+a')
+ if (focused === true && shortcut) {
+ await this.pressKeysInPage(sessionName, target, [shortcut])
}
+ return { selected: element }
},
- { ensureSession: false, requireScopedTarget: true }
+ { elements: [element] }
)
}
diff --git a/src/main/browser/agent-browser-bridge-key-commands.ts b/src/main/browser/agent-browser-bridge-key-commands.ts
new file mode 100644
index 00000000000..0f9c3b692ea
--- /dev/null
+++ b/src/main/browser/agent-browser-bridge-key-commands.ts
@@ -0,0 +1,138 @@
+import type { BrowserKeypressResult, BrowserTypeResult } from '../../shared/runtime-types'
+import { assertClipboardTextWriteWithinLimitWithYield } from '../../shared/clipboard-text'
+import { BrowserError } from './cdp-bridge'
+import { acquireElectronDebugger } from './electron-debugger-lease'
+import {
+ parseCdpKeyEvent,
+ imeFallbackKeyEvent,
+ typedInputSteps,
+ type TypedInputStep
+} from './cdp-keyboard-us-layout'
+import type { ResolvedBrowserCommandTarget } from './agent-browser-bridge-types'
+import {
+ dispatchPageKeyEvents,
+ routePageTextInsertion,
+ type GuestCdpSender
+} from './guest-direct-key-input'
+import { sendGuestCdpCommand } from './guest-cdp-command'
+import { AgentBrowserBridgeCaptureCommands } from './agent-browser-bridge-capture-commands'
+
+// Bounds the events a typed run holds and how long it keeps the main loop busy.
+const MAX_KEY_EVENTS_PER_RUN = 512
+
+export abstract class AgentBrowserBridgeKeyCommands extends AgentBrowserBridgeCaptureCommands {
+ async keypress(
+ key: string,
+ worktreeId?: string,
+ browserPageId?: string
+ ): Promise<BrowserKeypressResult> {
+ return this.enqueueTargetedCommand(
+ worktreeId,
+ browserPageId,
+ async (sessionName, target) => {
+ const parsed = parseCdpKeyEvent(key) ?? imeFallbackKeyEvent(key)
+ if (!parsed) {
+ // Unknown keys need the helper; create its session only on this fallback path.
+ await this.ensureSession(sessionName, target.browserPageId, target.webContentsId)
+ return (await this.execAgentBrowser(sessionName, ['press', key])) as BrowserKeypressResult
+ }
+ await this.pressKeysInPage(sessionName, target, [parsed])
+ return { pressed: key }
+ },
+ { ensureSession: false, requireScopedTarget: true }
+ )
+ }
+
+ async type(
+ input: string,
+ worktreeId?: string,
+ browserPageId?: string
+ ): Promise<BrowserTypeResult> {
+ await assertClipboardTextWriteWithinLimitWithYield(input)
+ return this.enqueueTargetedCommand(
+ worktreeId,
+ browserPageId,
+ async (sessionName, target) => {
+ // The helper inserts printable text without key events, which key-driven editors ignore.
+ await this.pressKeysInPage(sessionName, target, typedInputSteps(input))
+ return { typed: true }
+ },
+ { ensureSession: false, requireScopedTarget: true }
+ )
+ }
+
+ /** Sends each key as keydown and keyup to the page's focused element, as a person typing would. */
+ protected async pressKeysInPage(
+ sessionName: string,
+ target: ResolvedBrowserCommandTarget,
+ steps: Iterable<TypedInputStep>
+ ): Promise<void> {
+ const wc = this.requireTargetWebContents(target)
+ let releaseDebugger = (): void => {}
+ try {
+ releaseDebugger = acquireElectronDebugger(wc).release
+ await this.waitForPagePaint(wc, target.webContentsId)
+ const send: GuestCdpSender = (method, params, sessionId) =>
+ sendGuestCdpCommand(wc, method, params, sessionId)
+ const guardFocus = (): Promise<() => void> =>
+ this.browserManager.guardUserFocusForAgentInput(target.webContentsId)
+ let run: Record<string, unknown>[] = []
+ const flush = async (): Promise<void> => {
+ if (run.length > 0) {
+ await dispatchPageKeyEvents(wc, run, send, guardFocus)
+ run = []
+ }
+ }
+ for (const step of steps) {
+ if (typeof step === 'string') {
+ // Text no key types (emoji, controls) goes in like a paste, between the keys around it.
+ await flush()
+ const route = await routePageTextInsertion(wc, { text: step }, send, guardFocus)
+ try {
+ await route.deliver({ text: step })
+ } finally {
+ route.release()
+ }
+ continue
+ }
+ const event = {
+ windowsVirtualKeyCode: step.keyCode,
+ nativeVirtualKeyCode: step.keyCode,
+ key: step.key,
+ code: step.code,
+ modifiers: step.modifiers,
+ location: step.location
+ }
+ run.push(
+ {
+ // keyDown without text makes Blink synthesize empty input; use rawKeyDown for editing keys.
+ type: step.text === null ? 'rawKeyDown' : 'keyDown',
+ ...event,
+ ...(step.text === null ? {} : { text: step.text, unmodifiedText: step.text })
+ },
+ // Blink clears a modifier's own bit on keyup.
+ { type: 'keyUp', ...event, modifiers: step.modifiers & ~step.selfModifier }
+ )
+ // Tab and Enter can move focus into another frame; long lines yield to the main loop.
+ if (step.key === 'Tab' || step.key === 'Enter' || run.length >= MAX_KEY_EVENTS_PER_RUN) {
+ await flush()
+ }
+ }
+ await flush()
+ } catch (error) {
+ // Browser error codes let the pane recognize and reclaim a dead page.
+ if (error instanceof BrowserError) {
+ throw error
+ }
+ if (!this.getWebContents(target.webContentsId)) {
+ throw this.createPageUnavailableError(sessionName)
+ }
+ throw new BrowserError(
+ 'browser_error',
+ `Failed to send keys to browser page ${target.browserPageId}: ${error instanceof Error ? error.message : String(error)}`
+ )
+ } finally {
+ releaseDebugger()
+ }
+ }
+}
diff --git a/src/main/browser/agent-browser-bridge-keypress-input.test.ts b/src/main/browser/agent-browser-bridge-keypress-input.test.ts
index dec53b45be5..b37e08f0946 100644
--- a/src/main/browser/agent-browser-bridge-keypress-input.test.ts
+++ b/src/main/browser/agent-browser-bridge-keypress-input.test.ts
@@ -291,6 +291,38 @@ describe('AgentBrowserBridge keypress input', () => {
})
expect(keyEventCalls(wc)).toHaveLength(0)
})
+
+ it('types text as a keydown and keyup per character, which key-driven editors need', async () => {
+ await expect(bridge.type('Hi\n', undefined, 'tab-1')).resolves.toEqual({ typed: true })
+
+ expect(execFileMock).not.toHaveBeenCalled()
+ expect(keyEventCalls(wc).map(({ type, key, text }) => [type, key, text])).toEqual([
+ ['keyDown', 'H', 'H'],
+ ['keyUp', 'H', undefined],
+ ['keyDown', 'i', 'i'],
+ ['keyUp', 'i', undefined],
+ ['keyDown', 'Enter', '\r'],
+ ['keyUp', 'Enter', undefined]
+ ])
+ })
+
+ it('inserts characters no key types, in order with the typed ones', async () => {
+ const order: string[] = []
+ wc.debugger.sendCommand.mockImplementation(async (method: string, params: unknown) => {
+ if (isRecord(params) && params.type === 'keyDown') {
+ order.push(`key ${String(params.key)}`)
+ } else if (method === 'Input.insertText' && isRecord(params)) {
+ order.push(`text ${String(params.text)}`)
+ }
+ return {}
+ })
+
+ await bridge.type('a😀b', undefined, 'tab-1')
+
+ expect(order).toEqual(['key a', 'text 😀', 'key b'])
+ expect(execFileMock).not.toHaveBeenCalled()
+ })
+
describe.each(['a', 'MediaPlayPause'])('target admission for %s', (key) => {
function registerTwoWorkspaces(): MockWebContents {
const otherPage = mockWebContents(200)
diff --git a/src/main/browser/agent-browser-bridge-text-input.test.ts b/src/main/browser/agent-browser-bridge-text-input.test.ts
index 00dcef8799a..ad7c5b1e087 100644
--- a/src/main/browser/agent-browser-bridge-text-input.test.ts
+++ b/src/main/browser/agent-browser-bridge-text-input.test.ts
@@ -24,6 +24,12 @@ vi.mock('electron', () => {
webContents: { fromId: webContentsFromIdMock }
}
})
+const { capturedElement } = vi.hoisted(() => {
+ const capturedElement: { current: { objectId: string; sessionId?: string } | null } = {
+ current: null
+ }
+ return { capturedElement }
+})
const { CdpWsProxyMock } = vi.hoisted(() => {
const instances: unknown[] = []
const MockClass = vi.fn().mockImplementation(function (
@@ -35,6 +41,7 @@ const { CdpWsProxyMock } = vi.hoisted(() => {
this.stop = vi.fn(async () => {})
this.getPort = vi.fn(() => 9222)
this.pointAtNextNode = vi.fn()
+ this.elementCapture = { start: vi.fn(), stop: vi.fn(() => capturedElement.current) }
instances.push(this)
})
return { CdpWsProxyMock: Object.assign(MockClass, { instances }) }
@@ -64,180 +71,31 @@ import {
} from '../../shared/clipboard-text'
import {
agentBrowserCommandsIn,
- answerAgentBrowserCommands,
createSucceedWith,
mockBrowserManager,
mockWebContents,
overrideBridgeWebContentsLookup,
- resetAgentBrowserBridgeMocks
+ resetAgentBrowserBridgeMocks,
+ type MockWebContents
} from './agent-browser-bridge-test-harness'
+import { FILL_ELEMENT_FUNCTION } from './browser-element-functions'
overrideBridgeWebContentsLookup(AgentBrowserBridge.prototype, webContentsFromIdMock)
const succeedWith = createSucceedWith(execFileMock, stdinWrites)
-function succeedForContentEditable(data: unknown = { ok: true }): void {
- answerAgentBrowserCommands(execFileMock, stdinWrites, (command, cb) => {
- const result =
- command[0] === 'get' && command[1] === 'attr' && command.includes('contenteditable')
- ? { value: 'true' }
- : data
- cb(null, JSON.stringify({ success: true, data: result }), '')
- })
-}
-
-// Fill scripts go to `eval --stdin`; every other command arrives as batch JSON.
-function evalScripts(stdin: string[]): string[] {
- return stdin.filter((text) => !text.startsWith('[['))
-}
-
-class TestEvent {
- type: string
- bubbles: boolean
-
- constructor(type: string, init?: { bubbles?: boolean }) {
- this.type = type
- this.bubbles = init?.bubbles ?? false
- }
-}
-
-type FillEvalNode = {
- tagName: string
- getAttribute: (name: string) => string | null
- matches: (selector: string) => boolean
- querySelector?: (selector: string) => FillEvalNode | null
- dispatchEvent: (event: TestEvent) => boolean
- value: string
-}
-
-function matchesFillEvalSelector(node: FillEvalNode, selector: string): boolean {
- return selector.split(',').some((candidate) => {
- const trimmed = candidate.trim()
- if (trimmed === 'textarea') {
- return node.tagName === 'TEXTAREA'
- }
- if (!trimmed.startsWith('input') || node.tagName !== 'INPUT') {
- return false
- }
- const excludedTypes = [...trimmed.matchAll(/:not\(\[type='([^']+)'\]\)/g)].map((match) =>
- match[1].toLowerCase()
- )
- const inputType = node.getAttribute('type')?.toLowerCase() ?? ''
- return !excludedTypes.includes(inputType)
- })
-}
-
-function createFillEvalNode(options: {
- tagName: string
- role?: string
- ariaControls?: string
- descendant?: FillEvalNode | null
- descendants?: FillEvalNode[]
- type?: string
-}) {
- const events: TestEvent[] = []
- let value = ''
- const proto = {
- get value() {
- return value
- },
- set value(next: string) {
- value = next
- }
- }
- const node = Object.create(proto) as FillEvalNode
-
- node.tagName = options.tagName
- node.getAttribute = (name: string) => {
- if (name === 'role') {
- return options.role ?? null
- }
- if (name === 'aria-controls') {
- return options.ariaControls ?? null
- }
- if (name === 'type') {
- return options.type ?? null
- }
- return null
- }
- node.matches = vi.fn((selector: string) => matchesFillEvalSelector(node, selector))
- const descendants = options.descendants ?? (options.descendant ? [options.descendant] : [])
- node.querySelector = vi.fn(
- (selector: string) => descendants.find((descendant) => descendant.matches(selector)) ?? null
+function elementCalls(wc: MockWebContents): unknown[][] {
+ return wc.debugger.sendCommand.mock.calls.filter(
+ ([method]) => method === 'Runtime.callFunctionOn'
)
- node.dispatchEvent = vi.fn((event: TestEvent) => {
- events.push(event)
- return true
- })
-
- return {
- node,
- events,
- get value() {
- return value
- }
- }
-}
-
-function runFillEvalExpressions(
- stdin: string[],
- document: { activeElement: unknown; getElementById: (id: string) => unknown },
- windowObject: Record<string, unknown> = {}
-): void {
- for (const expression of evalScripts(stdin)) {
- new Function('document', 'Event', 'window', `return (${expression})`)(
- document,
- TestEvent,
- windowObject
- )
- }
-}
-
-function createContentEditableEvalEnvironment(initialText: string) {
- const editor = {
- tagName: 'DIV',
- isContentEditable: true,
- textContent: initialText,
- matches: vi.fn(() => false),
- getAttribute: vi.fn((name: string) => (name === 'contenteditable' ? 'true' : null)),
- dispatchEvent: vi.fn()
- }
- let selected = false
- const selection = {
- selectAllChildren: vi.fn(() => {
- selected = true
- })
- }
- const execCommand = vi.fn((command: string, _showUi: boolean, value: string) => {
- // Empty insertText succeeds without clearing a selection; simulate Chromium requiring delete.
- if (command === 'delete') {
- if (!selected) {
- return false
- }
- editor.textContent = ''
- } else if (value.length > 0) {
- editor.textContent = selected ? value : editor.textContent + value
- }
- selected = false
- return true
- })
- return {
- editor,
- execCommand,
- document: {
- activeElement: editor,
- body: {},
- getElementById: () => null,
- execCommand
- },
- windowObject: { getSelection: () => selection }
- }
}
describe('AgentBrowserBridge', () => {
let bridge: AgentBrowserBridge
+ let wc: MockWebContents
beforeEach(() => {
+ capturedElement.current = { objectId: 'element-1', sessionId: 'frame-session' }
resetAgentBrowserBridgeMocks({
webContentsFromIdMock,
existsSyncMock,
@@ -247,6 +105,8 @@ describe('AgentBrowserBridge', () => {
})
bridge = new AgentBrowserBridge(mockBrowserManager())
bridge.setActiveTab(100)
+ wc = mockWebContents(100)
+ webContentsFromIdMock.mockImplementation((id: number) => (id === 100 ? wc : null))
})
it('rejects browser clipboard writes that exceed the safe agent-browser argument size', async () => {
@@ -259,196 +119,81 @@ describe('AgentBrowserBridge', () => {
expect(execFileMock).not.toHaveBeenCalled()
})
- it('replaces contenteditable text through the browser editing pipeline', async () => {
- succeedForContentEditable()
- const environment = createContentEditableEvalEnvironment('existing text')
-
- await bridge.fill('@editor', 'replacement text')
-
- runFillEvalExpressions(stdinWrites, environment.document, environment.windowObject)
-
- expect(environment.editor.textContent).toBe('replacement text')
- expect(environment.execCommand).toHaveBeenCalledWith('insertText', false, 'replacement text')
- expect(environment.editor.dispatchEvent).not.toHaveBeenCalled()
- })
-
- it('clears selected contenteditable text with a browser delete command', async () => {
- succeedForContentEditable()
- const environment = createContentEditableEvalEnvironment('existing text')
-
- const result = await bridge.clear('@editor')
-
- runFillEvalExpressions(stdinWrites, environment.document, environment.windowObject)
-
- expect(environment.editor.textContent).toBe('')
- expect(environment.execCommand).toHaveBeenCalledWith('delete', false, '')
- expect(environment.editor.dispatchEvent).not.toHaveBeenCalled()
- expect(result).toEqual({ cleared: '@editor' })
- })
-
- it('fails contenteditable fill when the browser editing command is unavailable', async () => {
- succeedForContentEditable()
- const environment = createContentEditableEvalEnvironment('existing text')
- environment.execCommand.mockReturnValue(false)
-
- await bridge.fill('@editor', 'replacement text')
-
- expect(() =>
- runFillEvalExpressions(stdinWrites, environment.document, environment.windowObject)
- ).toThrow('Browser rich-text editing command failed')
- expect(environment.editor.textContent).toBe('existing text')
- expect(environment.editor.dispatchEvent).not.toHaveBeenCalled()
+ it('fills the element the helper resolved, in its own frame, with one page call', async () => {
+ const text = `${'é\n'.repeat(512 * 1024)}tail'\\`
+ succeedWith({ value: null })
+ wc.debugger.sendCommand.mockResolvedValue({ result: { type: 'object', value: null } })
+
+ await expect(bridge.fill('@e3', text)).resolves.toEqual({ filled: '@e3' })
+
+ expect(agentBrowserCommandsIn(stdinWrites)).toEqual([['get', 'attr', '@e3', 'id']])
+ expect(execFileMock.mock.calls.flatMap((call) => call[1]).join('')).not.toContain('tail')
+ expect(elementCalls(wc)).toEqual([
+ [
+ 'Runtime.callFunctionOn',
+ {
+ objectId: 'element-1',
+ functionDeclaration: FILL_ELEMENT_FUNCTION,
+ arguments: [{ value: text }],
+ returnByValue: true
+ },
+ 'frame-session'
+ ]
+ ])
})
- it('inserts paste-sized contenteditable text as one stdin editing transaction', async () => {
- const firstChunk = 'x'.repeat(AGENT_BROWSER_TEXT_ARGUMENT_MAX_BYTES)
- succeedForContentEditable()
- const environment = createContentEditableEvalEnvironment('existing text')
-
- await bridge.fill('@editor', `${firstChunk}tail`)
+ it('clears by filling the resolved element with an empty value', async () => {
+ succeedWith({ value: null })
+ wc.debugger.sendCommand.mockResolvedValue({ result: { type: 'object', value: null } })
- const evalCalls = execFileMock.mock.calls.filter((call: unknown[]) =>
- (call[1] as string[]).includes('eval')
- )
- runFillEvalExpressions(stdinWrites, environment.document, environment.windowObject)
+ await expect(bridge.clear('@e3')).resolves.toEqual({ cleared: '@e3' })
- expect(evalCalls).toHaveLength(1)
- expect(evalCalls[0][1]).toContain('--stdin')
- expect(evalScripts(stdinWrites)).toHaveLength(1)
- expect(environment.editor.textContent).toBe(`${firstChunk}tail`)
+ expect(elementCalls(wc)[0]?.[1]).toMatchObject({ arguments: [{ value: '' }] })
})
- it('routes focused spinbutton wrappers to editable descendants before filling', async () => {
- succeedWith({ ok: true })
-
- await bridge.fill('@spinbutton', '200')
-
- const expressions = stdinWrites
-
- const input = createFillEvalNode({ tagName: 'INPUT' })
- const wrapper = createFillEvalNode({
- tagName: 'DIV',
- role: 'spinbutton',
- descendant: input.node
+ it('refuses an element the page function rejects as not interactable', async () => {
+ succeedWith({ value: null })
+ wc.debugger.sendCommand.mockResolvedValue({
+ result: { type: 'string', value: 'it is not a text field or editable element' }
})
- runFillEvalExpressions(expressions, {
- activeElement: wrapper.node,
- getElementById: () => null
+ await expect(bridge.fill('@e1', 'x')).rejects.toMatchObject({
+ code: 'browser_element_not_interactable',
+ message: 'Cannot fill @e1: it is not a text field or editable element'
})
-
- expect(input.value).toBe('200')
- expect(wrapper.value).toBe('')
- expect(input.events.map((event) => event.type)).toEqual(['input', 'change'])
- expect(wrapper.events).toHaveLength(0)
})
- it('routes aria-controlled spinbutton wrappers to editable inputs before filling', async () => {
- succeedWith({ ok: true })
-
- await bridge.fill('@spinbutton', '200')
-
- const expressions = stdinWrites
+ it('fails instead of writing anywhere when the helper resolves no element', async () => {
+ capturedElement.current = null
+ succeedWith({ value: null })
- const input = createFillEvalNode({ tagName: 'INPUT' })
- const wrapper = createFillEvalNode({
- tagName: 'DIV',
- role: 'spinbutton',
- ariaControls: 'target-id'
- })
-
- runFillEvalExpressions(expressions, {
- activeElement: wrapper.node,
- getElementById: (id: string) => (id === 'target-id' ? input.node : null)
- })
-
- expect(input.value).toBe('200')
- expect(wrapper.value).toBe('')
- expect(input.events.map((event) => event.type)).toEqual(['input', 'change'])
- expect(wrapper.events).toHaveLength(0)
+ await expect(bridge.fill('@e1', 'x')).rejects.toMatchObject({ code: 'browser_error' })
+ expect(elementCalls(wc)).toEqual([])
})
- it('routes aria-controlled spinbutton containers to editable descendants before filling', async () => {
- succeedWith({ ok: true })
-
- await bridge.fill('@spinbutton', '200')
-
- const expressions = stdinWrites
-
- const input = createFillEvalNode({ tagName: 'INPUT' })
- const controlled = createFillEvalNode({ tagName: 'DIV', descendant: input.node })
- const wrapper = createFillEvalNode({
- tagName: 'DIV',
- role: 'spinbutton',
- ariaControls: 'target-id'
- })
-
- runFillEvalExpressions(expressions, {
- activeElement: wrapper.node,
- getElementById: (id: string) => (id === 'target-id' ? controlled.node : null)
- })
-
- expect(input.value).toBe('200')
- expect(controlled.value).toBe('')
- expect(wrapper.value).toBe('')
- expect(input.events.map((event) => event.type)).toEqual(['input', 'change'])
- expect(controlled.events).toHaveLength(0)
- expect(wrapper.events).toHaveLength(0)
- })
-
- it('skips non-text spinbutton descendant inputs before filling', async () => {
- succeedWith({ ok: true })
-
- await bridge.fill('@spinbutton', '200')
-
- const expressions = stdinWrites
-
- const hiddenInput = createFillEvalNode({ tagName: 'INPUT', type: 'hidden' })
- const numberInput = createFillEvalNode({ tagName: 'INPUT', type: 'number' })
- const wrapper = createFillEvalNode({
- tagName: 'DIV',
- role: 'spinbutton',
- descendants: [hiddenInput.node, numberInput.node]
+ it('refuses select on an element that is not a <select> before the helper reports success', async () => {
+ succeedWith({ selected: ['Mango'] })
+ wc.debugger.sendCommand.mockResolvedValue({
+ result: { type: 'string', value: 'it is a combobox, not a <select>' }
})
- runFillEvalExpressions(expressions, {
- activeElement: wrapper.node,
- getElementById: () => null
+ await expect(bridge.select('@e9', 'Mango')).rejects.toMatchObject({
+ code: 'browser_element_not_interactable'
})
-
- expect(numberInput.value).toBe('200')
- expect(hiddenInput.value).toBe('')
- expect(wrapper.value).toBe('')
- expect(numberInput.events.map((event) => event.type)).toEqual(['input', 'change'])
- expect(hiddenInput.events).toHaveLength(0)
- expect(wrapper.events).toHaveLength(0)
+ expect(agentBrowserCommandsIn(stdinWrites)).toEqual([['get', 'attr', '@e9', 'id']])
})
- it('fills large plain fields with one stdin edit and one event pair', async () => {
- const text = `${'é\n'.repeat(512 * 1024)}tail'\\`
- succeedWith({ ok: true })
-
- await bridge.fill('@textarea', text)
+ it('refuses to upload a path that is not an existing file', async () => {
+ succeedWith({ uploaded: 1 })
- const evalCalls = execFileMock.mock.calls.filter((call: unknown[]) =>
- (call[1] as string[]).includes('eval')
- )
- expect(evalCalls).toHaveLength(1)
- expect(evalCalls[0][1]).toContain('--stdin')
- expect(evalScripts(stdinWrites)).toHaveLength(1)
- expect((evalCalls[0][1] as string[]).join('')).not.toContain(text)
- const input = createFillEvalNode({ tagName: 'TEXTAREA' })
- runFillEvalExpressions(stdinWrites, {
- activeElement: input.node,
- getElementById: () => null
+ await expect(bridge.upload('@e19', ['/nonexistent/orca-upload.txt'])).rejects.toMatchObject({
+ code: 'invalid_argument'
})
- expect(input.value).toBe(text)
- expect(input.events.map((event) => event.type)).toEqual(['input', 'change'])
+ expect(agentBrowserCommandsIn(stdinWrites).map(([name]) => name)).not.toContain('upload')
})
it.each([
['fill', (b: AgentBrowserBridge, text: string) => b.fill('@textarea', text)],
- ['type', (b: AgentBrowserBridge, text: string) => b.type(text)],
['keyboard insert', (b: AgentBrowserBridge, text: string) => b.keyboardInsertText(text)]
])('yields before spawning agent-browser for accepted large %s text', async (_name, run) => {
vi.useFakeTimers()
@@ -479,19 +224,6 @@ describe('AgentBrowserBridge', () => {
expect(agentBrowserCommandsIn(stdinWrites)).toEqual([['keyboard', 'inserttext', '--help']])
})
- it('chunks large agent-browser type text before keyboard transport', async () => {
- const text = ['y'.repeat(AGENT_BROWSER_TEXT_ARGUMENT_MAX_BYTES), 'zz'].join('')
- succeedWith({ typed: true })
-
- await bridge.type(text)
-
- const chunks = agentBrowserCommandsIn(stdinWrites)
- .filter(([name, subcommand]) => name === 'keyboard' && subcommand === 'type')
- .map(([, , chunk]) => chunk)
-
- expect(chunks).toEqual(['y'.repeat(AGENT_BROWSER_TEXT_ARGUMENT_MAX_BYTES), 'zz'])
- })
-
it('chunks large agent-browser keyboard insert text before transport', async () => {
const text = ['z'.repeat(AGENT_BROWSER_TEXT_ARGUMENT_MAX_BYTES), 'qq'].join('')
succeedWith({ inserted: true })
diff --git a/src/main/browser/browser-element-functions.test.ts b/src/main/browser/browser-element-functions.test.ts
new file mode 100644
index 00000000000..9c7cc504f75
--- /dev/null
+++ b/src/main/browser/browser-element-functions.test.ts
@@ -0,0 +1,155 @@
+// @vitest-environment happy-dom
+import { describe, it, expect, vi, beforeEach } from 'vitest'
+import {
+ FILE_INPUT_TARGET_FUNCTION,
+ FILL_ELEMENT_FUNCTION,
+ SELECT_ALL_ELEMENT_FUNCTION,
+ SELECT_OPTION_TARGET_FUNCTION
+} from './browser-element-functions'
+
+// CDP's Runtime.callFunctionOn calls the declaration with the resolved element as `this`.
+function callOn(element: Element, declaration: string, ...args: unknown[]): unknown {
+ return new Function(`return (${declaration})`)().apply(element, args)
+}
+
+function mount(html: string): HTMLElement {
+ document.body.innerHTML = html
+ return document.body
+}
+
+describe('FILL_ELEMENT_FUNCTION', () => {
+ beforeEach(() => {
+ document.body.innerHTML = ''
+ })
+
+ it('sets an input inside an open shadow root and fires one input and change event', () => {
+ const host = mount('<div id="host"></div>').querySelector('#host')!
+ const input = host.attachShadow({ mode: 'open' }).appendChild(document.createElement('input'))
+ const events: string[] = []
+ input.addEventListener('input', () => events.push('input'))
+ input.addEventListener('change', () => events.push('change'))
+ // Like a person's input, it crosses the shadow boundary for page-level listeners.
+ document.addEventListener('input', () => events.push('document input'))
+
+ expect(callOn(input, FILL_ELEMENT_FUNCTION, 'hello')).toBeNull()
+
+ expect(input.value).toBe('hello')
+ expect(events).toEqual(['input', 'document input', 'change'])
+ })
+
+ it.each([
+ ['<h1>Title</h1>', 'h1', 'it is not a text field or editable element'],
+ ['<input disabled>', 'input', 'it is disabled'],
+ ['<textarea readonly></textarea>', 'textarea', 'it is read-only'],
+ ['<input type="checkbox">', 'input', 'it is not a text field or editable element'],
+ ['<select><option>a</option></select>', 'select', 'it is a <select>; use select instead']
+ ])('refuses %s', (html, selector, reason) => {
+ const element = mount(html).querySelector(selector)!
+
+ expect(callOn(element, FILL_ELEMENT_FUNCTION, 'x')).toBe(reason)
+ })
+
+ it('keeps the previous value when a number field sanitizes the new one away', () => {
+ const input = mount('<input type="number" value="42">').querySelector('input')!
+ const events: string[] = []
+ input.addEventListener('input', () => events.push('input'))
+
+ expect(callOn(input, FILL_ELEMENT_FUNCTION, 'abc')).toBe('the field rejected this value')
+
+ expect(input.value).toBe('42')
+ expect(events).toEqual([])
+ })
+
+ it('fills the input an aria-controlled spinbutton points at', () => {
+ const body = mount('<div role="spinbutton" aria-controls="qty"></div><input id="qty">')
+
+ callOn(body.querySelector('[role=spinbutton]')!, FILL_ELEMENT_FUNCTION, '200')
+
+ expect(body.querySelector<HTMLInputElement>('#qty')!.value).toBe('200')
+ })
+
+ it('fills the text descendant of a spinbutton wrapper, skipping non-text inputs', () => {
+ const body = mount(
+ '<div role="spinbutton"><input type="hidden" id="h"><input type="number" id="n"></div>'
+ )
+
+ callOn(body.querySelector('[role=spinbutton]')!, FILL_ELEMENT_FUNCTION, '7')
+
+ expect(body.querySelector<HTMLInputElement>('#n')!.value).toBe('7')
+ expect(body.querySelector<HTMLInputElement>('#h')!.value).toBe('')
+ })
+
+ it.each([
+ ['replaces', 'new text', 'insertText'],
+ ['clears', '', 'delete']
+ ])('%s contenteditable text through a browser edit command', (_name, value, command) => {
+ const editor = mount('<div contenteditable="true">old</div>').querySelector('div')!
+ const execCommand = vi.fn(() => true)
+ Object.defineProperty(document, 'execCommand', { configurable: true, value: execCommand })
+
+ expect(callOn(editor, FILL_ELEMENT_FUNCTION, value)).toBeNull()
+
+ expect(execCommand).toHaveBeenCalledWith(command, false, value)
+ expect(window.getSelection()?.toString()).toBe('old')
+ })
+
+ it('refuses a rich-text edit when a focus handler moves focus to another field', () => {
+ const body = mount('<div contenteditable="true">old</div><input value="keep">')
+ const execCommand = vi.fn(() => true)
+ Object.defineProperty(document, 'execCommand', { configurable: true, value: execCommand })
+ body.querySelector('div')!.addEventListener('focus', () => body.querySelector('input')!.focus())
+
+ expect(callOn(body.querySelector('div')!, FILL_ELEMENT_FUNCTION, 'new')).toBe(
+ 'it gave its focus to another element'
+ )
+ expect(execCommand).not.toHaveBeenCalled()
+ })
+
+ it('refuses an input the page replaces when it takes focus', () => {
+ const input = mount('<input value="old">').querySelector('input')!
+ input.addEventListener('focus', () => input.replaceWith(document.createElement('input')))
+
+ expect(callOn(input, FILL_ELEMENT_FUNCTION, 'new')).toMatch(/replaced it/)
+ })
+
+ it('reports a rich-text edit the browser refused', () => {
+ const editor = mount('<div contenteditable="true">old</div>').querySelector('div')!
+ Object.defineProperty(document, 'execCommand', { configurable: true, value: () => false })
+
+ expect(callOn(editor, FILL_ELEMENT_FUNCTION, 'new')).toBe('the browser editing command failed')
+ })
+})
+
+describe('element target checks', () => {
+ it.each([
+ ['<div role="combobox">Pick</div>', 'div', 'it is a combobox, not a <select>'],
+ ['<h1>x</h1>', 'h1', 'it is <h1>, not a <select>'],
+ ['<select disabled></select>', 'select', 'it is disabled']
+ ])('refuses select on %s', (html, selector, reason) => {
+ const element = mount(html).querySelector(selector)!
+
+ expect(callOn(element, SELECT_OPTION_TARGET_FUNCTION)).toMatch(reason)
+ })
+
+ it('accepts select on an enabled <select> and upload on a file input', () => {
+ const body = mount('<select></select><input type="file">')
+
+ expect(callOn(body.querySelector('select')!, SELECT_OPTION_TARGET_FUNCTION)).toBeNull()
+ expect(callOn(body.querySelector('input')!, FILE_INPUT_TARGET_FUNCTION)).toBeNull()
+ })
+
+ it('refuses upload on an element that is not a file input', () => {
+ const button = mount('<button>Attach</button>').querySelector('button')!
+
+ expect(callOn(button, FILE_INPUT_TARGET_FUNCTION)).toBe('it is not a file input')
+ })
+
+ it('selects all of an input value and reports that it holds focus', () => {
+ const input = mount('<input value="abcdef">').querySelector('input')!
+ input.setSelectionRange(3, 3)
+
+ expect(callOn(input, SELECT_ALL_ELEMENT_FUNCTION)).toBe(true)
+
+ expect([input.selectionStart, input.selectionEnd]).toEqual([0, 6])
+ })
+})
diff --git a/src/main/browser/browser-element-functions.ts b/src/main/browser/browser-element-functions.ts
new file mode 100644
index 00000000000..48efd8467b4
--- /dev/null
+++ b/src/main/browser/browser-element-functions.ts
@@ -0,0 +1,81 @@
+// Page functions for Runtime.callFunctionOn; `this` is the element the agent's ref resolved to.
+// A returned string is the reason the element cannot take the action.
+
+export const FILL_ELEMENT_FUNCTION = `function (value) {
+ const isTextField = (node) =>
+ node?.localName === 'textarea' ||
+ (node?.localName === 'input' && !/^(hidden|button|checkbox|radio|file|image|reset|submit)$/i.test(node.type));
+ const findTextField = (root) => Array.from(root?.querySelectorAll?.('input, textarea') ?? []).find(isTextField) ?? null;
+ let target = this;
+ // Spinbutton wrappers can take the ref while another input owns the value.
+ if (!isTextField(target) && target.getAttribute?.('role') === 'spinbutton') {
+ const root = target.getRootNode();
+ for (const id of (target.getAttribute('aria-controls') ?? '').split(/\\s+/).filter(Boolean)) {
+ const controlled = root.getElementById?.(id) ?? null;
+ const field = isTextField(controlled) ? controlled : findTextField(controlled);
+ if (field) { target = field; break; }
+ }
+ if (target === this) target = findTextField(this) ?? this;
+ }
+ if (target.matches?.(':disabled')) return 'it is disabled';
+ if (target.isContentEditable && !isTextField(target)) {
+ // Rich editors revert direct DOM writes; browser edit transactions keep their models in sync.
+ target.focus();
+ // The edit lands wherever focus is, so a focus handler that moved it must stop the write.
+ const active = target.getRootNode().activeElement;
+ if (!active || !(active === target || active.contains(target) || target.contains(active))) {
+ return 'it gave its focus to another element';
+ }
+ const selection = target.ownerDocument.defaultView?.getSelection?.();
+ if (!selection) return 'its text selection is unavailable';
+ selection.selectAllChildren(target);
+ let edited = false;
+ try {
+ edited = target.ownerDocument.execCommand(value.length === 0 ? 'delete' : 'insertText', false, value) === true;
+ } catch {}
+ return edited ? null : 'the browser editing command failed';
+ }
+ if (target.localName === 'select') return 'it is a <select>; use select instead';
+ if (!isTextField(target)) return 'it is not a text field or editable element';
+ if (target.readOnly) return 'it is read-only';
+ target.focus();
+ if (!target.isConnected) return 'the page replaced it when it took focus; take a new snapshot';
+ const nativeSetter = Object.getOwnPropertyDescriptor(Object.getPrototypeOf(target), 'value')?.set;
+ const setValue = (next) => (nativeSetter ? nativeSetter.call(target, next) : (target.value = next));
+ const previous = target.value;
+ setValue(value);
+ // Sanitizing fields (number, email, date) empty a value they cannot hold; handlers may clear it later.
+ if (value !== '' && target.value === '') {
+ setValue(previous);
+ return 'the field rejected this value';
+ }
+ target.dispatchEvent(new Event('input', { bubbles: true, composed: true }));
+ target.dispatchEvent(new Event('change', { bubbles: true }));
+ return null;
+}`
+
+// Returns whether an editable element holds focus, so a select-all shortcut would stay inside it.
+export const SELECT_ALL_ELEMENT_FUNCTION = `function () {
+ this.focus();
+ const active = this.getRootNode().activeElement;
+ const selectable = typeof this.select === 'function';
+ const focused = (selectable || this.isContentEditable) && (active === this || this.contains(active));
+ if (selectable) { this.select(); return focused; }
+ const selection = this.ownerDocument.defaultView?.getSelection?.();
+ if (!selection) return 'its text selection is unavailable';
+ selection.selectAllChildren(this);
+ return focused;
+}`
+
+export const SELECT_OPTION_TARGET_FUNCTION = `function () {
+ if (this.localName !== 'select') {
+ const role = this.getAttribute?.('role');
+ return 'it is ' + (role ? 'a ' + role : '<' + this.localName + '>') + ', not a <select>; click it and choose the option instead';
+ }
+ return this.matches(':disabled') ? 'it is disabled' : null;
+}`
+
+export const FILE_INPUT_TARGET_FUNCTION = `function () {
+ if (this.localName !== 'input' || this.type !== 'file') return 'it is not a file input';
+ return this.matches(':disabled') ? 'it is disabled' : null;
+}`
diff --git a/src/main/browser/browser-upload-files.ts b/src/main/browser/browser-upload-files.ts
new file mode 100644
index 00000000000..d5e68250230
--- /dev/null
+++ b/src/main/browser/browser-upload-files.ts
@@ -0,0 +1,16 @@
+import { stat } from 'node:fs/promises'
+import { BrowserError } from './browser-error'
+
+/** Chromium attaches a missing path as an empty file, so check the paths before handing them over. */
+export async function assertUploadFilesExist(filePaths: readonly string[]): Promise<void> {
+ for (const filePath of filePaths) {
+ // Directories stay allowed: webkitdirectory inputs take them.
+ const exists = await stat(filePath).then(
+ () => true,
+ () => false
+ )
+ if (!exists) {
+ throw new BrowserError('invalid_argument', `Cannot upload ${filePath}: it does not exist`)
+ }
+ }
+}
diff --git a/src/main/browser/cdp-element-capture.ts b/src/main/browser/cdp-element-capture.ts
new file mode 100644
index 00000000000..760ea1eaf89
--- /dev/null
+++ b/src/main/browser/cdp-element-capture.ts
@@ -0,0 +1,29 @@
+// A helper-resolved remote object; its session is the debugger's, undefined for the page itself.
+export type CapturedElement = { objectId: string; sessionId: string | undefined }
+
+/** Records the element the helper's calls run on, so Orca can act on that exact node. */
+export class CdpElementCapture {
+ // Undefined while not capturing.
+ private captured: CapturedElement | null | undefined
+
+ start(): void {
+ this.captured = null
+ }
+
+ stop(): CapturedElement | null {
+ const captured = this.captured ?? null
+ this.captured = undefined
+ return captured
+ }
+
+ observe(method: string, params: Record<string, unknown> | undefined, sessionId?: string): void {
+ const objectId = params?.objectId
+ if (
+ this.captured !== undefined &&
+ method === 'Runtime.callFunctionOn' &&
+ typeof objectId === 'string'
+ ) {
+ this.captured = { objectId, sessionId }
+ }
+ }
+}
diff --git a/src/main/browser/cdp-keyboard-us-layout.ts b/src/main/browser/cdp-keyboard-us-layout.ts
index 0bbab9946f7..5210c075525 100644
--- a/src/main/browser/cdp-keyboard-us-layout.ts
+++ b/src/main/browser/cdp-keyboard-us-layout.ts
@@ -237,3 +237,44 @@ export function parseCdpKeyEvent(raw: string): CdpKeyEvent | null {
return { keyCode, key, code, modifiers, location, selfModifier, text }
}
+
+/** The key a person presses to type one character, or null when no key types it. */
+function typedCharacterKeyEvent(char: string): CdpKeyEvent | null {
+ if (char === '\n') {
+ return parseCdpKeyEvent('Enter')
+ }
+ if (char === '\t') {
+ return parseCdpKeyEvent('Tab')
+ }
+ return parseCdpKeyEvent(char) ?? imeFallbackKeyEvent(char)
+}
+
+const MAX_UNTYPABLE_RUN_LENGTH = 512
+
+/** A key to press, or a run of text that no key types. */
+export type TypedInputStep = CdpKeyEvent | string
+
+/** The keys that type `input`, yielded lazily so long text is never held as events. */
+export function* typedInputSteps(input: string): Generator<TypedInputStep> {
+ let untypable = ''
+ for (const char of input.replace(/\r\n?/g, '\n')) {
+ const key = typedCharacterKeyEvent(char)
+ if (!key) {
+ untypable += char
+ // Bounded like key runs, since a widget page receives this text one event per character.
+ if (untypable.length >= MAX_UNTYPABLE_RUN_LENGTH) {
+ yield untypable
+ untypable = ''
+ }
+ continue
+ }
+ if (untypable) {
+ yield untypable
+ untypable = ''
+ }
+ yield key
+ }
+ if (untypable) {
+ yield untypable
+ }
+}
diff --git a/src/main/browser/cdp-ws-proxy.ts b/src/main/browser/cdp-ws-proxy.ts
index 0d3934d09e0..b5d84a5ee2c 100644
--- a/src/main/browser/cdp-ws-proxy.ts
+++ b/src/main/browser/cdp-ws-proxy.ts
@@ -7,6 +7,7 @@ import { CdpTargetDiscovery } from './cdp-target-discovery'
import { CdpDebuggerChannel } from './cdp-debugger-channel'
import { CdpPageNavigationCommands } from './cdp-page-navigation-commands'
import { CdpDomFocusReplay } from './cdp-dom-focus-replay'
+import { CdpElementCapture } from './cdp-element-capture'
import { CdpPageCaptureCommands } from './cdp-page-capture-commands'
import type { CapturePaintHold } from './cdp-screenshot'
import type { AgentInputFocusGuard } from './agent-input-focus-guard'
@@ -34,6 +35,7 @@ export class CdpWsProxy {
private readonly pageCapture: CdpPageCaptureCommands
private readonly input: CdpPageInputCommands
private pointAtNextResolvedNode = false
+ readonly elementCapture = new CdpElementCapture()
constructor(
private readonly webContents: WebContents,
@@ -248,6 +250,7 @@ export class CdpWsProxy {
)
return
}
+ this.elementCapture.observe(msg.method, msg.params, effectiveSessionId)
if (msg.method === 'DOM.focus') {
this.domFocusReplay.forwardDomFocus(client, clientId, msg.params ?? {}, effectiveSessionId)
return
diff --git a/src/main/browser/guest-direct-key-input.test.ts b/src/main/browser/guest-direct-key-input.test.ts
index 8bf5c9f9228..0249a91439e 100644
--- a/src/main/browser/guest-direct-key-input.test.ts
+++ b/src/main/browser/guest-direct-key-input.test.ts
@@ -2,6 +2,7 @@ import { describe, expect, it, vi } from 'vitest'
import type { WebContents } from 'electron'
import {
dispatchPageKeyEvent,
+ dispatchPageKeyEvents,
routePageTextInsertion,
type GuestCdpSender
} from './guest-direct-key-input'
@@ -225,6 +226,23 @@ describe('dispatchPageKeyEvent', () => {
expect(page.selectAll).toHaveBeenCalledTimes(1)
})
+ it('settles a run of widget keys with one focus probe and one page round trip', async () => {
+ const { page, webContents } = fakePage('webview')
+ const send = fakeSender()
+ const keys = ['a', 'b', 'c'].flatMap((key) => [
+ { type: 'keyDown', key, text: key },
+ { type: 'keyUp', key }
+ ])
+
+ await dispatchPageKeyEvents(webContents, keys, send)
+
+ expect(page.sendInputEvent).toHaveBeenCalledTimes(9)
+ expect(send.mock.calls.map(([method]) => method)).toEqual([
+ 'Runtime.evaluate',
+ 'Runtime.evaluate'
+ ])
+ })
+
it('types into a focused cross-site frame on its own session, without focusing the guest', async () => {
const { page, webContents } = fakePage('webview')
const send = crossSiteFrameSender({ attachable: true })
@@ -238,6 +256,31 @@ describe('dispatchPageKeyEvent', () => {
expect(send).toHaveBeenLastCalledWith('Input.dispatchKeyEvent', params, 'S1')
})
+ it('follows focus that a frame key handler moves into the parent page mid-run', async () => {
+ const { page, webContents } = fakePage('webview')
+ const inFrame = crossSiteFrameSender({ attachable: true })
+ let focusInFrame = true
+ const send = vi.fn<GuestCdpSender>(async (method, params, sessionId) => {
+ if (method === 'Input.dispatchKeyEvent' && params.type === 'keyUp') {
+ // The frame's handler sends focus to the parent after the first key.
+ focusInFrame = false
+ }
+ return focusInFrame || sessionId
+ ? inFrame(method, params, sessionId)
+ : fakeSender()(method, params, sessionId)
+ })
+ const keys = ['a', 'b'].flatMap((key) => [
+ { type: 'keyDown', key, text: key },
+ { type: 'keyUp', key }
+ ])
+
+ await dispatchPageKeyEvents(webContents, keys, send)
+
+ const frameKeys = send.mock.calls.filter(([method]) => method === 'Input.dispatchKeyEvent')
+ expect(frameKeys.map(([, params]) => params.key)).toEqual(['a', 'a'])
+ expect(page.sendInputEvent).toHaveBeenCalledWith(expect.objectContaining({ keyCode: 'b' }))
+ })
+
it('borrows focus for CDP when the focused frame cannot be reached, then gives it back', async () => {
const { page, webContents } = fakePage('webview')
const send = crossSiteFrameSender({ attachable: false })
diff --git a/src/main/browser/guest-direct-key-input.ts b/src/main/browser/guest-direct-key-input.ts
index d2f0b34896d..ff1a49aec5c 100644
--- a/src/main/browser/guest-direct-key-input.ts
+++ b/src/main/browser/guest-direct-key-input.ts
@@ -101,48 +101,88 @@ export async function dispatchPageKeyEvent(
send: GuestCdpSender,
guardFocus: AgentInputFocusGuard = NO_FOCUS_GUARD
): Promise<unknown> {
+ return await dispatchPageKeyEvents(webContents, [params], send, guardFocus)
+}
+
+const isTabDown = (params: Record<string, unknown>): boolean =>
+ params.key === 'Tab' && (params.type === 'keyDown' || params.type === 'rawKeyDown')
+
+/** Deliver a run of key events along one focus route; callers split runs where focus may move. */
+export async function dispatchPageKeyEvents(
+ webContents: WebContents,
+ events: readonly Record<string, unknown>[],
+ send: GuestCdpSender,
+ guardFocus: AgentInputFocusGuard = NO_FOCUS_GUARD
+): Promise<unknown> {
+ let result: unknown = {}
if (!isEmbeddedGuest(webContents)) {
- return await send('Input.dispatchKeyEvent', params)
+ for (const params of events) {
+ result = await send('Input.dispatchKeyEvent', params)
+ }
+ return result
}
- const route = await resolveFocusedFrameRoute(webContents, send)
- if (route.kind === 'session') {
+ let start = 0
+ let route = await resolveFocusedFrameRoute(webContents, send)
+ // A frame's key handler can move focus to another frame, so each key finds its route again.
+ while (route.kind === 'session') {
+ const keyUp = events.findIndex((params, index) => index >= start && params.type === 'keyUp')
+ const end = keyUp === -1 ? events.length : keyUp + 1
+ const key = events.slice(start, end)
// Tab from the last frame field can escape into Orca's window.
- const isTabDown =
- params.key === 'Tab' && (params.type === 'keyDown' || params.type === 'rawKeyDown')
- const release = isTabDown ? await guardFocus() : () => {}
+ const release = key.some(isTabDown) ? await guardFocus() : () => {}
try {
- return await sendToFrameSession(
- webContents,
- send,
- route.sessionId,
- 'Input.dispatchKeyEvent',
- params
- )
+ for (const params of key) {
+ result = await sendToFrameSession(
+ webContents,
+ send,
+ route.sessionId,
+ 'Input.dispatchKeyEvent',
+ params
+ )
+ }
} finally {
release()
}
+ if (end >= events.length) {
+ return result
+ }
+ start = end
+ route = await resolveFocusedFrameRoute(webContents, send)
}
- const plan = planCdpKeyEvent(params)
- if (plan && route.kind === 'widget') {
+ const remaining = events.slice(start)
+ const plans = remaining.map((params) => planCdpKeyEvent(params))
+ if (route.kind === 'widget' && plans.every((plan) => plan !== null)) {
// Tab from the last page field can escape into Orca's window.
- const release = plan.events.some((event) => event.type === 'keyDown' && event.keyCode === 'Tab')
- ? await guardFocus()
- : () => {}
+ const release = remaining.some(isTabDown) ? await guardFocus() : () => {}
try {
- await withAgentInputInjection(webContents, () =>
- // Widget events bypass CDP send, so track held keys here.
- trackRemoteCliInput(webContents.id, 'Input.dispatchKeyEvent', params, async () => {
- for (const event of plan.events) {
- webContents.sendInputEvent(event)
- }
- if (plan.commands.length > 0 || plan.events.length > 0) {
- await waitForPageToHandleInput(send)
- }
- for (const command of plan.commands) {
- runEditCommand(webContents, command)
- }
- })
- )
+ await withAgentInputInjection(webContents, async () => {
+ // One page round trip settles the whole run; edit commands must see the keys before them.
+ let unsettled = false
+ for (const [index, plan] of plans.entries()) {
+ // Widget events bypass CDP send, so track held keys here.
+ await trackRemoteCliInput(
+ webContents.id,
+ 'Input.dispatchKeyEvent',
+ remaining[index],
+ async () => {
+ for (const event of plan.events) {
+ webContents.sendInputEvent(event)
+ }
+ unsettled ||= plan.events.length > 0
+ if (plan.commands.length > 0) {
+ await waitForPageToHandleInput(send)
+ unsettled = false
+ for (const command of plan.commands) {
+ runEditCommand(webContents, command)
+ }
+ }
+ }
+ )
+ }
+ if (unsettled) {
+ await waitForPageToHandleInput(send)
+ }
+ })
} finally {
release()
}
@@ -150,7 +190,12 @@ export async function dispatchPageKeyEvent(
}
const release = await focusGuestForCdpInput(webContents, send, guardFocus)
try {
- return await withAgentInputInjection(webContents, () => send('Input.dispatchKeyEvent', params))
+ for (const params of remaining) {
+ result = await withAgentInputInjection(webContents, () =>
+ send('Input.dispatchKeyEvent', params)
+ )
+ }
+ return result
} finally {
release()
}
diff --git a/src/main/orcad/external-chromium-browser-process.ts b/src/main/orcad/external-chromium-browser-process.ts
index f5965afa60a..7b8d25a8b65 100644
--- a/src/main/orcad/external-chromium-browser-process.ts
+++ b/src/main/orcad/external-chromium-browser-process.ts
@@ -2,6 +2,7 @@ import { z } from 'zod'
import type { RuntimeBrowserCommandHost } from '../runtime/browser/browser-command-contracts'
import type { RuntimeBrowserCommands } from '../runtime/browser/browser-commands'
import { BrowserError } from '../browser/browser-error'
+import { assertUploadFilesExist } from '../browser/browser-upload-files'
import {
ExternalChromiumBrowserSession,
type ExternalChromiumLaunch
@@ -179,6 +180,10 @@ export class ExternalChromiumBrowserProcess {
return { url: tab.url, title: tab.title }
}
+ if (method === 'browserUpload') {
+ const files: unknown[] = Array.isArray(params.files) ? params.files : []
+ await assertUploadFilesExist(files.map(String))
+ }
const command = externalChromiumCommandArguments(method, params)
if (!command) {
throw new BrowserError(
diff --git a/tests/e2e/helpers/browser-agent-focus-scenarios.ts b/tests/e2e/helpers/browser-agent-focus-scenarios.ts
index 037a888f91e..5e136767c1a 100644
--- a/tests/e2e/helpers/browser-agent-focus-scenarios.ts
+++ b/tests/e2e/helpers/browser-agent-focus-scenarios.ts
@@ -216,8 +216,8 @@ export async function runAgentKeysScenario(
`JSON.stringify({ submits: window.__submits, keys: window.__keys })`
)
)
- // Why: `type` inserts text without key events, so only the keypresses log keydowns.
- .toBe(JSON.stringify({ submits: 1, keys: ['Z', 'Enter'] }))
+ // Why: `type` presses real keys, so its characters log keydowns like the keypresses.
+ .toBe(JSON.stringify({ submits: 1, keys: ['Z', 'q', 'r', 'Enter'] }))
await afterAgentInput?.(pageId)
} finally {
await server.close()
@@ -257,7 +257,7 @@ export async function runAgentClicksScenario(
.toBe(JSON.stringify({ active: 'probe', focused: true, blurs: 0 }))
expect((await readHost(page)).terminalFocused).toBe(true)
- // Why: text with a newline cannot be typed as key events; CDP delivers it with focus borrowed.
+ // Why: fill writes multiline values straight into the field, never through the user's focus.
await rpc(page, 'browser.fill', { page: pageId, element: '#notes', value: 'a\nb' })
await expect
.poll(() => evalInPage(page, pageId, `document.getElementById('notes').value`))