Orca is the place for working with a fleet of agents (e.g. Grok Bot, Claude Code). Run any coding agent with your own subscription. Available on desktop, mobile and remote runtime.
TypeScript adeagent-ideai-agentsautomationbotclaude-codecodexghosttygrok-bothermes-agentmobile-appmuseopenclawopencodeorchestrationpiruntimeterminalworktreesyc-backed
fix(browser): agent fill, select, upload and type change the page or fail with a reason (#36263)
Kelvin Amoabab67c5254f044
diff --git a/skill-guides/orca-cli/references/browser.md b/skill-guides/orca-cli/references/browser.mdindex a8435f8a8ce..cc84303d277 100644--- a/skill-guides/orca-cli/references/browser.md+++ b/skill-guides/orca-cli/references/browser.md@@ -60,12 +60,14 @@ Browser rules:- Use `ORCA set device` or `ORCA viewport --mobile`, not `ORCA exec --command "set device ..."`, so the page also gets touch input and `pointer: coarse`. Each page load (reload, `goto`, link click) resets it, so run it again after navigating.- Prefer `wait --text`, `--url`, `--selector`, or `--load` after async page changes instead of bare timeouts.- Anything not listed above goes through `ORCA exec --page <browserPageId> --command "<agent-browser command>"`.-- If `fill` or `type` fails on a custom input, try `ORCA focus --page <browserPageId> --element @e1 --json` then `ORCA inserttext --page <browserPageId> --text "text" --json`.+- `fill` sets a field's whole value in one step, at any size. `type` presses real keys into the focused element, so `minlength`/`maxlength`, key handlers, Enter and Tab act as they would for a person; `ORCA focus` the element first. Use `type` when the page reacts to keystrokes.+- On server-hosted pages `fill` and `type` still insert text the engine's own way. If either does nothing on a custom input there, try `ORCA focus --page <browserPageId> --element @e1 --json` then `ORCA inserttext --page <browserPageId> --text "text" --json`.- A client-hosted page renders in the paired desktop's browser engine, so every command against it needs that desktop online and returns `browser_host_unavailable` while it is closed, asleep, or disconnected. Server-hosted pages run with no desktop attached; prefer them for long or unattended automation.Common recoveries:- `browser_no_tab`: open a tab with `ORCA tab create --url <url> --json`.- `browser_stale_ref`: run `ORCA snapshot --page <browserPageId> --json` and retry on the same page with fresh refs.+- `browser_element_not_interactable`: the element can't take that command, and the message says why (not a text field, disabled, read-only, not a `<select>`, not a file input). Re-snapshot and pick the right ref; for a custom dropdown, click it and then click the option.- `browser_tab_not_found`: run `ORCA tab list --json` before switching or closing.- `browser_host_unavailable`: the desktop hosting the page is offline. Bring it back, or recreate the page with server placement if the work must outlive the desktop session.diff --git a/src/cli/bundled-skill-guides.ts b/src/cli/bundled-skill-guides.tsindex 2b649c28076..ad8a7298850 100644--- a/src/cli/bundled-skill-guides.ts+++ b/src/cli/bundled-skill-guides.ts@@ -871,13 +871,15 @@ const ORCA_CLI_BROWSER_REFERENCE_MARKDOWN = ["- Use `ORCA set device` or `ORCA viewport --mobile`, not `ORCA exec --command \"set device ...\"`, so the page also gets touch input and `pointer: coarse`. Each page load (reload, `goto`, link click) resets it, so run it again after navigating.","- Prefer `wait --text`, `--url`, `--selector`, or `--load` after async page changes instead of bare timeouts.","- Anything not listed above goes through `ORCA exec --page <browserPageId> --command \"<agent-browser command>\"`.",- "- If `fill` or `type` fails on a custom input, try `ORCA focus --page <browserPageId> --element @e1 --json` then `ORCA inserttext --page <browserPageId> --text \"text\" --json`.",+ "- `fill` sets a field's whole value in one step, at any size. `type` presses real keys into the focused element, so `minlength`/`maxlength`, key handlers, Enter and Tab act as they would for a person; `ORCA focus` the element first. Use `type` when the page reacts to keystrokes.",+ "- On server-hosted pages `fill` and `type` still insert text the engine's own way. If either does nothing on a custom input there, try `ORCA focus --page <browserPageId> --element @e1 --json` then `ORCA inserttext --page <browserPageId> --text \"text\" --json`.","- A client-hosted page renders in the paired desktop's browser engine, so every command against it needs that desktop online and returns `browser_host_unavailable` while it is closed, asleep, or disconnected. Server-hosted pages run with no desktop attached; prefer them for long or unattended automation.","","Common recoveries:","","- `browser_no_tab`: open a tab with `ORCA tab create --url <url> --json`.","- `browser_stale_ref`: run `ORCA snapshot --page <browserPageId> --json` and retry on the same page with fresh refs.",+ "- `browser_element_not_interactable`: the element can't take that command, and the message says why (not a text field, disabled, read-only, not a `<select>`, not a file input). Re-snapshot and pick the right ref; for a custom dropdown, click it and then click the option.","- `browser_tab_not_found`: run `ORCA tab list --json` before switching or closing.","- `browser_host_unavailable`: the desktop hosting the page is offline. Bring it back, or recreate the page with server placement if the work must outlive the desktop session.","",diff --git a/src/main/browser/agent-browser-bridge-core-commands.ts b/src/main/browser/agent-browser-bridge-core-commands.tsindex 53f3185c1b4..a2a1ce407bb 100644--- a/src/main/browser/agent-browser-bridge-core-commands.ts+++ b/src/main/browser/agent-browser-bridge-core-commands.ts@@ -8,7 +8,7 @@ import { assertClipboardTextWriteWithinLimitWithYield } from '../../shared/clipbimport { normalizeBrowserNavigationUrl } from '../../shared/browser-url'import { BrowserError } from './cdp-bridge'import { ORCA_TAB_SESSION_PREFIX } from './agent-browser-orphan-sweep'-import { focusedValueSetExpression } from './agent-browser-bridge-input'+import { FILL_ELEMENT_FUNCTION } from './browser-element-functions'import { EMBEDDED_NAVIGATION_TIMEOUT_MS } from './agent-browser-bridge-types'import {isAbortedNavigationError,@@ -159,22 +159,14 @@ export abstract class AgentBrowserBridgeCoreCommands extends AgentBrowserBridgeQbrowserPageId?: string): Promise<BrowserFillResult> {await assertClipboardTextWriteWithinLimitWithYield(value)- // agent-browser's CDP insertion loses focus in Electron guests; use the browser input pipeline.+ // agent-browser's fill types into whatever holds focus and sends text per character; set the+ // resolved element's value in one call instead.return this.enqueueTargetedCommand(worktreeId,browserPageId,async (sessionName) => {- if (!(await this.isExplicitContentEditableTarget(sessionName, element))) {- await this.execAgentBrowser(sessionName, ['focus', element])- // One stdin edit avoids argv limits and repeated copying of the growing field value.- await this.execAgentBrowser(sessionName, ['eval', '--stdin'], {- stdinText: focusedValueSetExpression(JSON.stringify(value), { dispatchEvents: true })- })- return { filled: element } as BrowserFillResult- }-- await this.fillExplicitContentEditable(sessionName, element, value)- return { filled: element } as BrowserFillResult+ await this.actOnElement(sessionName, element, 'fill', FILL_ELEMENT_FUNCTION, [value])+ return { filled: element }},{ requireScopedTarget: true, elements: [element] })diff --git a/src/main/browser/agent-browser-bridge-execution.ts b/src/main/browser/agent-browser-bridge-execution.tsindex 7faee8ef652..c77d32425ed 100644--- a/src/main/browser/agent-browser-bridge-execution.ts+++ b/src/main/browser/agent-browser-bridge-execution.ts@@ -1,10 +1,9 @@import { execFile } from 'node:child_process'import type { WebContents } from 'electron'import { BrowserError } from './cdp-bridge'-import {- focusedRichTextEditExpression,- isExplicitContentEditableResult-} from './agent-browser-bridge-input'+import type { CapturedElement } from './cdp-element-capture'+import { sendGuestCdpCommand } from './guest-cdp-command'+import { readRecord, readString } from './guest-frame-input-sessions'import {isTabClosedTransportError,pageUnavailableMessageForSession@@ -170,29 +169,73 @@ export abstract class AgentBrowserBridgeExecution extends AgentBrowserBridgeTabsreturn this.runAgentBrowserRaw(sessionName, args, execOptions)}- protected async isExplicitContentEditableTarget(- sessionName: string,- element: string- ): Promise<boolean> {- const result = await this.execAgentBrowser(sessionName, [- 'get',- 'attr',- element,- 'contenteditable'- ])- return isExplicitContentEditableResult(result)- }-- protected async fillExplicitContentEditable(+ /** Runs a browser-element-functions action on the element the helper resolves, in its own frame. */+ protected async actOnElement(sessionName: string,element: string,- value: string- ): Promise<void> {- await this.execAgentBrowser(sessionName, ['focus', element])- // Send one stdin edit to avoid argv limits and focus changes between chunks.- await this.execAgentBrowser(sessionName, ['eval', '--stdin'], {- stdinText: focusedRichTextEditExpression(JSON.stringify(value), { selectAll: true })+ action: string,+ functionDeclaration: string,+ args: unknown[] = []+ ): Promise<unknown> {+ const session = this.sessions.get(sessionName)+ if (!session) {+ throw this.createPageUnavailableError(sessionName)+ }+ session.proxy.elementCapture.start()+ session.proxy.pointAtNextNode(true)+ let captured: CapturedElement | null+ try {+ // Any element read makes the helper resolve the ref or selector, in whichever frame owns it.+ await this.execAgentBrowser(sessionName, ['get', 'attr', element, 'id'])+ } finally {+ session.proxy.pointAtNextNode(false)+ captured = session.proxy.elementCapture.stop()+ }+ const wc = this.getWebContents(session.webContentsId)+ if (!wc) {+ throw this.createPageUnavailableError(sessionName)+ }+ if (!captured) {+ throw new BrowserError('browser_error', `Could not resolve ${element} in the page`)+ }+ const response = await sendGuestCdpCommand(+ wc,+ 'Runtime.callFunctionOn',+ {+ objectId: captured.objectId,+ functionDeclaration,+ arguments: args.map((value) => ({ value })),+ returnByValue: true+ },+ captured.sessionId+ ).catch((error: unknown) => {+ if (error instanceof BrowserError) {+ throw error+ }+ if (!this.getWebContents(session.webContentsId)) {+ throw this.createPageUnavailableError(sessionName)+ }+ throw new BrowserError(+ 'browser_cdp_error',+ error instanceof Error ? error.message : String(error)+ )})+ const exception = readRecord(response, 'exceptionDetails')+ if (exception) {+ const description = readString(readRecord(exception, 'exception'), 'description')+ throw new BrowserError(+ 'browser_eval_error',+ description ?? readString(exception, 'text') ?? 'Page function failed'+ )+ }+ const outcome = readRecord(response, 'result')?.value+ if (typeof outcome === 'string') {+ throw new BrowserError(+ 'browser_element_not_interactable',+ `Cannot ${action} ${element}: ${outcome}`+ )+ }+ return outcome}protected createPageUnavailableError(sessionName: string): BrowserError {diff --git a/src/main/browser/agent-browser-bridge-host-revocation.test.ts b/src/main/browser/agent-browser-bridge-host-revocation.test.tsindex d889b1a8fe0..c96a976a8d0 100644--- a/src/main/browser/agent-browser-bridge-host-revocation.test.ts+++ b/src/main/browser/agent-browser-bridge-host-revocation.test.ts@@ -85,6 +85,42 @@ describe('AgentBrowserBridge under a host revocation', () => {bridge.setActiveTab(100)})+ it('stops a long type at the next run once the host is revoked', async () => {+ const wc = mockWebContents(100)+ wc.getType.mockReturnValue('webview')+ let resumeTyping = (): void => {}+ const paused = new Promise<void>((resolve) => {+ resumeTyping = resolve+ })+ let evaluations = 0+ wc.debugger.sendCommand.mockImplementation(async (method: string) => {+ if (method === 'Runtime.evaluate') {+ evaluations += 1+ // The first run's settle wait: hold it while the host is revoked.+ if (evaluations === 2) {+ await paused+ }+ return { result: { type: 'object', subtype: 'null' } }+ }+ return {}+ })+ webContentsFromIdMock.mockReturnValue(wc)+ const revoked = new AbortController()++ const typing = runInRemoteCliBrowserScope(+ WORKSPACE,+ () => bridge.type('x'.repeat(2000), undefined, 'tab-1'),+ revoked.signal+ )+ await vi.waitFor(() => expect(evaluations).toBe(2))+ const sentBeforeRevocation = wc.sendInputEvent.mock.calls.length+ revoked.abort()+ resumeTyping()++ await expect(typing).rejects.toMatchObject({ code: 'desktop_browser_bridge_unavailable' })+ expect(wc.sendInputEvent.mock.calls.length).toBe(sentBeforeRevocation)+ })+it('never presses the mouse once the host is revoked during a running click', async () => {const wc = mockWebContents(100)let answerHitTest = (): void => {}diff --git a/src/main/browser/agent-browser-bridge-input-commands.ts b/src/main/browser/agent-browser-bridge-input-commands.tsindex 242fdbc7b99..c623655cd3a 100644--- a/src/main/browser/agent-browser-bridge-input-commands.ts+++ b/src/main/browser/agent-browser-bridge-input-commands.ts@@ -1,36 +1,11 @@-import type {- BrowserTypeResult,- BrowserSelectResult,- BrowserScrollResult-} from '../../shared/runtime-types'+import type { BrowserSelectResult, BrowserScrollResult } from '../../shared/runtime-types'import { assertClipboardTextWriteWithinLimitWithYield } from '../../shared/clipboard-text'import { iterateBrowserTextInsertionChunks } from '../../shared/browser-text-insertion-chunks'import { AGENT_BROWSER_TEXT_ARGUMENT_MAX_BYTES } from './agent-browser-bridge-types'import { AgentBrowserBridgeCoreCommands } from './agent-browser-bridge-core-commands'+import { SELECT_OPTION_TARGET_FUNCTION } from './browser-element-functions'export abstract class AgentBrowserBridgeInputCommands extends AgentBrowserBridgeCoreCommands {- async type(- input: string,- worktreeId?: string,- browserPageId?: string- ): Promise<BrowserTypeResult> {- await assertClipboardTextWriteWithinLimitWithYield(input)- return this.enqueueTargetedCommand(- worktreeId,- browserPageId,- async (sessionName) => {- for (const chunk of iterateBrowserTextInsertionChunks(- input,- AGENT_BROWSER_TEXT_ARGUMENT_MAX_BYTES- )) {- await this.execAgentBrowser(sessionName, ['keyboard', 'type', chunk])- }- return { typed: true } as BrowserTypeResult- },- { requireScopedTarget: true }- )- }-async select(element: string,value: string,@@ -41,6 +16,13 @@ export abstract class AgentBrowserBridgeInputCommands extends AgentBrowserBridgeworktreeId,browserPageId,async (sessionName) => {+ // The helper reports success for any element; only a <select> has options to choose.+ await this.actOnElement(+ sessionName,+ element,+ 'select an option in',+ SELECT_OPTION_TARGET_FUNCTION+ )return (await this.execAgentBrowser(sessionName, ['select',element,diff --git a/src/main/browser/agent-browser-bridge-input.ts b/src/main/browser/agent-browser-bridge-input.tsdeleted file mode 100644index e7338dc53f8..00000000000--- a/src/main/browser/agent-browser-bridge-input.ts+++ /dev/null@@ -1,67 +0,0 @@-export function focusedValueSetExpression(- valueExpression: string,- options?: { append?: boolean; dispatchEvents?: boolean }-): string {- const nextValue = options?.append- ? ["String(target.value ?? '') + ", valueExpression].join('')- : valueExpression- const dispatchEvents = options?.dispatchEvents- ? " target.dispatchEvent(new Event('input', { bubbles: true })); target.dispatchEvent(new Event('change', { bubbles: true }));"- : ''- return [- '(() => { const el = document.activeElement; if (el) {',- // Spinbutton wrappers can hold focus while another input owns the value.- " const editableSelector = \"input:not([type='hidden']):not([type='button']):not([type='checkbox']):not([type='radio']):not([type='file']):not([type='image']):not([type='reset']):not([type='submit']), textarea\";",- " const isEditable = (node) => !!node && (node.matches?.(editableSelector) ?? (node.tagName === 'TEXTAREA' || (node.tagName === 'INPUT' && !/^(hidden|button|checkbox|radio|file|image|reset|submit)$/i.test(node.getAttribute?.('type') ?? ''))));",- ' const findEditable = (root) => root?.querySelector?.(editableSelector) ?? null;',- ' let target = el;',- " if (!isEditable(target) && target.getAttribute?.('role') === 'spinbutton') {",- " const controls = target.getAttribute('aria-controls');",- ' if (controls) { for (const id of controls.split(/\\s+/)) { if (!id) continue; const controlled = document.getElementById(id); if (isEditable(controlled)) { target = controlled; break; } const descendant = findEditable(controlled); if (descendant) { target = descendant; break; } } }',- ' if (target === el) { const descendant = findEditable(target); if (descendant) target = descendant; }',- ' }',- " const nativeSetter = Object.getOwnPropertyDescriptor(Object.getPrototypeOf(target), 'value')?.set;",- ' const nextValue = ',- nextValue,- '; if (nativeSetter) { nativeSetter.call(target, nextValue); } else { target.value = nextValue; }',- dispatchEvents,- ' } })()'- ].join('')-}--// Rich editors require browser edit transactions to keep their models synchronized.-export function focusedRichTextEditExpression(- valueExpression: string,- options?: { selectAll?: boolean }-): string {- const selectAll = options?.selectAll ? 'true' : 'false'- return [- '(() => {',- ' const target = document.activeElement;',- ' const value = ',- valueExpression,- ';',- ` const selectAll = ${selectAll};`,- " const isEditable = target?.isContentEditable === true || /^(|true|plaintext-only)$/i.test(target?.getAttribute?.('contenteditable') ?? 'false');",- " if (!target || target === document.body || !isEditable) { throw new Error('Focused rich-text target is unavailable'); }",- ' if (selectAll) {',- " if (typeof window.getSelection !== 'function') { throw new Error('Rich-text selection is unavailable'); }",- ' const selection = window.getSelection();',- " if (!selection) { throw new Error('Rich-text selection is unavailable'); }",- ' selection.selectAllChildren(target);',- ' }',- " const editCommand = selectAll && value.length === 0 ? 'delete' : 'insertText';",- ' let edited = false;',- ' try {',- ' edited = document.execCommand(editCommand, false, value) === true;',- ' } catch { edited = false; }',- " if (!edited) { throw new Error('Browser rich-text editing command failed'); }",- ' })()'- ].join('')-}--export function isExplicitContentEditableResult(result: unknown): boolean {- const value =- result && typeof result === 'object' ? (result as { value?: unknown }).value : undefined- return typeof value === 'string' && /^(|true|plaintext-only)$/i.test(value)-}diff --git a/src/main/browser/agent-browser-bridge-interaction-commands.ts b/src/main/browser/agent-browser-bridge-interaction-commands.tsindex 53c5cf4dc74..12b0e61c4b7 100644--- a/src/main/browser/agent-browser-bridge-interaction-commands.ts+++ b/src/main/browser/agent-browser-bridge-interaction-commands.ts@@ -7,22 +7,24 @@ import type {BrowserFocusResult,BrowserClearResult,BrowserSelectAllResult,- BrowserKeypressResult,BrowserPdfResult} from '../../shared/runtime-types'import { BrowserError } from './cdp-bridge'+import {+ FILE_INPUT_TARGET_FUNCTION,+ FILL_ELEMENT_FUNCTION,+ SELECT_ALL_ELEMENT_FUNCTION+} from './browser-element-functions'+import { assertUploadFilesExist } from './browser-upload-files'import {agentBrowserWaitArguments,type AgentBrowserWaitOptions} from './agent-browser-wait-arguments'-import { acquireElectronDebugger } from './electron-debugger-lease'-import { parseCdpKeyEvent, imeFallbackKeyEvent } from './cdp-keyboard-us-layout'-import { dispatchPageKeyEvent, type GuestCdpSender } from './guest-direct-key-input'-import { sendGuestCdpCommand } from './guest-cdp-command'+import { parseCdpKeyEvent } from './cdp-keyboard-us-layout'import { readRemoteCliAdmittedPage } from './remote-cli-page-admission'-import { AgentBrowserBridgeCaptureCommands } from './agent-browser-bridge-capture-commands'+import { AgentBrowserBridgeKeyCommands } from './agent-browser-bridge-key-commands'-export abstract class AgentBrowserBridgeInteractionCommands extends AgentBrowserBridgeCaptureCommands {+export abstract class AgentBrowserBridgeInteractionCommands extends AgentBrowserBridgeKeyCommands {async hover(element: string,worktreeId?: string,@@ -64,6 +66,8 @@ export abstract class AgentBrowserBridgeInteractionCommands extends AgentBrowserworktreeId,browserPageId,async (sessionName) => {+ await assertUploadFilesExist(filePaths)+ await this.actOnElement(sessionName, element, 'upload to', FILE_INPUT_TARGET_FUNCTION)return (await this.execAgentBrowser(sessionName, ['upload',element,@@ -143,13 +147,7 @@ export abstract class AgentBrowserBridgeInteractionCommands extends AgentBrowserworktreeId,browserPageId,async (sessionName) => {- if (!(await this.isExplicitContentEditableTarget(sessionName, element))) {- // Let the helper resolve ordinary refs inside iframes and shadow roots, including unfocusable fields.- await this.execAgentBrowser(sessionName, ['fill', element, ''])- return { cleared: element }- }-- await this.fillExplicitContentEditable(sessionName, element, '')+ await this.actOnElement(sessionName, element, 'clear', FILL_ELEMENT_FUNCTION, [''])return { cleared: element }},{ requireScopedTarget: true, elements: [element] }@@ -161,98 +159,25 @@ export abstract class AgentBrowserBridgeInteractionCommands extends AgentBrowserworktreeId?: string,browserPageId?: string): Promise<BrowserSelectAllResult> {- return this.enqueueTargetedCommand(- worktreeId,- browserPageId,- async (sessionName) => {- await this.execAgentBrowser(sessionName, ['focus', element])- return (await this.execAgentBrowser(sessionName, [- 'press',- 'Control+a'- ])) as BrowserSelectAllResult- },- { elements: [element] }- )- }-- async keypress(- key: string,- worktreeId?: string,- browserPageId?: string- ): Promise<BrowserKeypressResult> {return this.enqueueTargetedCommand(worktreeId,browserPageId,async (sessionName, target) => {- const parsed = parseCdpKeyEvent(key) ?? imeFallbackKeyEvent(key)- if (!parsed) {- // Unknown keys need the helper; create its session only on this fallback path.- await this.ensureSession(sessionName, target.browserPageId, target.webContentsId)- return (await this.execAgentBrowser(sessionName, ['press', key])) as BrowserKeypressResult- }- const wc = this.getWebContents(target.webContentsId)- if (!wc || wc.isDestroyed()) {- throw new BrowserError(- 'browser_tab_not_found',- `Browser page ${target.browserPageId} is no longer available`- )- }- const event = {- windowsVirtualKeyCode: parsed.keyCode,- nativeVirtualKeyCode: parsed.keyCode,- key: parsed.key,- code: parsed.code,- modifiers: parsed.modifiers,- location: parsed.location- }- let releaseDebugger = (): void => {}- try {- releaseDebugger = acquireElectronDebugger(wc).release- await this.waitForPagePaint(wc, target.webContentsId)- const send: GuestCdpSender = (method, params, sessionId) =>- sendGuestCdpCommand(wc, method, params, sessionId)- const guardFocus = (): Promise<() => void> =>- this.browserManager.guardUserFocusForAgentInput(target.webContentsId)- await dispatchPageKeyEvent(- wc,- {- // keyDown without text makes Blink synthesize empty input; use rawKeyDown for editing keys.- type: parsed.text === null ? 'rawKeyDown' : 'keyDown',- ...event,- ...(parsed.text === null ? {} : { text: parsed.text, unmodifiedText: parsed.text })- },- send,- guardFocus- )- await dispatchPageKeyEvent(- wc,- {- type: 'keyUp',- ...event,- // Blink clears a modifier's own bit on keyup.- modifiers: parsed.modifiers & ~parsed.selfModifier- },- send,- guardFocus- )- return { pressed: key }- } catch (error) {- // Browser error codes let the pane recognize and reclaim a dead page.- if (error instanceof BrowserError) {- throw error- }- if (!this.getWebContents(target.webContentsId)) {- throw this.createPageUnavailableError(sessionName)- }- throw new BrowserError(- 'browser_error',- `Failed to press ${key} in browser page ${target.browserPageId}: ${error instanceof Error ? error.message : String(error)}`- )- } finally {- releaseDebugger()+ const focused = await this.actOnElement(+ sessionName,+ element,+ 'select all in',+ SELECT_ALL_ELEMENT_FUNCTION+ )+ // Code editors select their whole document from the shortcut, beyond the DOM they render;+ // an element that can't take focus keeps the DOM selection instead.+ const shortcut = parseCdpKeyEvent(process.platform === 'darwin' ? 'Meta+a' : 'Control+a')+ if (focused === true && shortcut) {+ await this.pressKeysInPage(sessionName, target, [shortcut])}+ return { selected: element }},- { ensureSession: false, requireScopedTarget: true }+ { elements: [element] })}diff --git a/src/main/browser/agent-browser-bridge-key-commands.ts b/src/main/browser/agent-browser-bridge-key-commands.tsnew file mode 100644index 00000000000..0f9c3b692ea--- /dev/null+++ b/src/main/browser/agent-browser-bridge-key-commands.ts@@ -0,0 +1,138 @@+import type { BrowserKeypressResult, BrowserTypeResult } from '../../shared/runtime-types'+import { assertClipboardTextWriteWithinLimitWithYield } from '../../shared/clipboard-text'+import { BrowserError } from './cdp-bridge'+import { acquireElectronDebugger } from './electron-debugger-lease'+import {+ parseCdpKeyEvent,+ imeFallbackKeyEvent,+ typedInputSteps,+ type TypedInputStep+} from './cdp-keyboard-us-layout'+import type { ResolvedBrowserCommandTarget } from './agent-browser-bridge-types'+import {+ dispatchPageKeyEvents,+ routePageTextInsertion,+ type GuestCdpSender+} from './guest-direct-key-input'+import { sendGuestCdpCommand } from './guest-cdp-command'+import { AgentBrowserBridgeCaptureCommands } from './agent-browser-bridge-capture-commands'++// Bounds the events a typed run holds and how long it keeps the main loop busy.+const MAX_KEY_EVENTS_PER_RUN = 512++export abstract class AgentBrowserBridgeKeyCommands extends AgentBrowserBridgeCaptureCommands {+ async keypress(+ key: string,+ worktreeId?: string,+ browserPageId?: string+ ): Promise<BrowserKeypressResult> {+ return this.enqueueTargetedCommand(+ worktreeId,+ browserPageId,+ async (sessionName, target) => {+ const parsed = parseCdpKeyEvent(key) ?? imeFallbackKeyEvent(key)+ if (!parsed) {+ // Unknown keys need the helper; create its session only on this fallback path.+ await this.ensureSession(sessionName, target.browserPageId, target.webContentsId)+ return (await this.execAgentBrowser(sessionName, ['press', key])) as BrowserKeypressResult+ }+ await this.pressKeysInPage(sessionName, target, [parsed])+ return { pressed: key }+ },+ { ensureSession: false, requireScopedTarget: true }+ )+ }++ async type(+ input: string,+ worktreeId?: string,+ browserPageId?: string+ ): Promise<BrowserTypeResult> {+ await assertClipboardTextWriteWithinLimitWithYield(input)+ return this.enqueueTargetedCommand(+ worktreeId,+ browserPageId,+ async (sessionName, target) => {+ // The helper inserts printable text without key events, which key-driven editors ignore.+ await this.pressKeysInPage(sessionName, target, typedInputSteps(input))+ return { typed: true }+ },+ { ensureSession: false, requireScopedTarget: true }+ )+ }++ /** Sends each key as keydown and keyup to the page's focused element, as a person typing would. */+ protected async pressKeysInPage(+ sessionName: string,+ target: ResolvedBrowserCommandTarget,+ steps: Iterable<TypedInputStep>+ ): Promise<void> {+ const wc = this.requireTargetWebContents(target)+ let releaseDebugger = (): void => {}+ try {+ releaseDebugger = acquireElectronDebugger(wc).release+ await this.waitForPagePaint(wc, target.webContentsId)+ const send: GuestCdpSender = (method, params, sessionId) =>+ sendGuestCdpCommand(wc, method, params, sessionId)+ const guardFocus = (): Promise<() => void> =>+ this.browserManager.guardUserFocusForAgentInput(target.webContentsId)+ let run: Record<string, unknown>[] = []+ const flush = async (): Promise<void> => {+ if (run.length > 0) {+ await dispatchPageKeyEvents(wc, run, send, guardFocus)+ run = []+ }+ }+ for (const step of steps) {+ if (typeof step === 'string') {+ // Text no key types (emoji, controls) goes in like a paste, between the keys around it.+ await flush()+ const route = await routePageTextInsertion(wc, { text: step }, send, guardFocus)+ try {+ await route.deliver({ text: step })+ } finally {+ route.release()+ }+ continue+ }+ const event = {+ windowsVirtualKeyCode: step.keyCode,+ nativeVirtualKeyCode: step.keyCode,+ key: step.key,+ code: step.code,+ modifiers: step.modifiers,+ location: step.location+ }+ run.push(+ {+ // keyDown without text makes Blink synthesize empty input; use rawKeyDown for editing keys.+ type: step.text === null ? 'rawKeyDown' : 'keyDown',+ ...event,+ ...(step.text === null ? {} : { text: step.text, unmodifiedText: step.text })+ },+ // Blink clears a modifier's own bit on keyup.+ { type: 'keyUp', ...event, modifiers: step.modifiers & ~step.selfModifier }+ )+ // Tab and Enter can move focus into another frame; long lines yield to the main loop.+ if (step.key === 'Tab' || step.key === 'Enter' || run.length >= MAX_KEY_EVENTS_PER_RUN) {+ await flush()+ }+ }+ await flush()+ } catch (error) {+ // Browser error codes let the pane recognize and reclaim a dead page.+ if (error instanceof BrowserError) {+ throw error+ }+ if (!this.getWebContents(target.webContentsId)) {+ throw this.createPageUnavailableError(sessionName)+ }+ throw new BrowserError(+ 'browser_error',+ `Failed to send keys to browser page ${target.browserPageId}: ${error instanceof Error ? error.message : String(error)}`+ )+ } finally {+ releaseDebugger()+ }+ }+}diff --git a/src/main/browser/agent-browser-bridge-keypress-input.test.ts b/src/main/browser/agent-browser-bridge-keypress-input.test.tsindex dec53b45be5..b37e08f0946 100644--- a/src/main/browser/agent-browser-bridge-keypress-input.test.ts+++ b/src/main/browser/agent-browser-bridge-keypress-input.test.ts@@ -291,6 +291,38 @@ describe('AgentBrowserBridge keypress input', () => {})expect(keyEventCalls(wc)).toHaveLength(0)})++ it('types text as a keydown and keyup per character, which key-driven editors need', async () => {+ await expect(bridge.type('Hi\n', undefined, 'tab-1')).resolves.toEqual({ typed: true })++ expect(execFileMock).not.toHaveBeenCalled()+ expect(keyEventCalls(wc).map(({ type, key, text }) => [type, key, text])).toEqual([+ ['keyDown', 'H', 'H'],+ ['keyUp', 'H', undefined],+ ['keyDown', 'i', 'i'],+ ['keyUp', 'i', undefined],+ ['keyDown', 'Enter', '\r'],+ ['keyUp', 'Enter', undefined]+ ])+ })++ it('inserts characters no key types, in order with the typed ones', async () => {+ const order: string[] = []+ wc.debugger.sendCommand.mockImplementation(async (method: string, params: unknown) => {+ if (isRecord(params) && params.type === 'keyDown') {+ order.push(`key ${String(params.key)}`)+ } else if (method === 'Input.insertText' && isRecord(params)) {+ order.push(`text ${String(params.text)}`)+ }+ return {}+ })++ await bridge.type('a😀b', undefined, 'tab-1')++ expect(order).toEqual(['key a', 'text 😀', 'key b'])+ expect(execFileMock).not.toHaveBeenCalled()+ })+describe.each(['a', 'MediaPlayPause'])('target admission for %s', (key) => {function registerTwoWorkspaces(): MockWebContents {const otherPage = mockWebContents(200)diff --git a/src/main/browser/agent-browser-bridge-text-input.test.ts b/src/main/browser/agent-browser-bridge-text-input.test.tsindex 00dcef8799a..ad7c5b1e087 100644--- a/src/main/browser/agent-browser-bridge-text-input.test.ts+++ b/src/main/browser/agent-browser-bridge-text-input.test.ts@@ -24,6 +24,12 @@ vi.mock('electron', () => {webContents: { fromId: webContentsFromIdMock }}})+const { capturedElement } = vi.hoisted(() => {+ const capturedElement: { current: { objectId: string; sessionId?: string } | null } = {+ current: null+ }+ return { capturedElement }+})const { CdpWsProxyMock } = vi.hoisted(() => {const instances: unknown[] = []const MockClass = vi.fn().mockImplementation(function (@@ -35,6 +41,7 @@ const { CdpWsProxyMock } = vi.hoisted(() => {this.stop = vi.fn(async () => {})this.getPort = vi.fn(() => 9222)this.pointAtNextNode = vi.fn()+ this.elementCapture = { start: vi.fn(), stop: vi.fn(() => capturedElement.current) }instances.push(this)})return { CdpWsProxyMock: Object.assign(MockClass, { instances }) }@@ -64,180 +71,31 @@ import {} from '../../shared/clipboard-text'import {agentBrowserCommandsIn,- answerAgentBrowserCommands,createSucceedWith,mockBrowserManager,mockWebContents,overrideBridgeWebContentsLookup,- resetAgentBrowserBridgeMocks+ resetAgentBrowserBridgeMocks,+ type MockWebContents} from './agent-browser-bridge-test-harness'+import { FILL_ELEMENT_FUNCTION } from './browser-element-functions'overrideBridgeWebContentsLookup(AgentBrowserBridge.prototype, webContentsFromIdMock)const succeedWith = createSucceedWith(execFileMock, stdinWrites)-function succeedForContentEditable(data: unknown = { ok: true }): void {- answerAgentBrowserCommands(execFileMock, stdinWrites, (command, cb) => {- const result =- command[0] === 'get' && command[1] === 'attr' && command.includes('contenteditable')- ? { value: 'true' }- : data- cb(null, JSON.stringify({ success: true, data: result }), '')- })-}--// Fill scripts go to `eval --stdin`; every other command arrives as batch JSON.-function evalScripts(stdin: string[]): string[] {- return stdin.filter((text) => !text.startsWith('[['))-}--class TestEvent {- type: string- bubbles: boolean-- constructor(type: string, init?: { bubbles?: boolean }) {- this.type = type- this.bubbles = init?.bubbles ?? false- }-}--type FillEvalNode = {- tagName: string- getAttribute: (name: string) => string | null- matches: (selector: string) => boolean- querySelector?: (selector: string) => FillEvalNode | null- dispatchEvent: (event: TestEvent) => boolean- value: string-}--function matchesFillEvalSelector(node: FillEvalNode, selector: string): boolean {- return selector.split(',').some((candidate) => {- const trimmed = candidate.trim()- if (trimmed === 'textarea') {- return node.tagName === 'TEXTAREA'- }- if (!trimmed.startsWith('input') || node.tagName !== 'INPUT') {- return false- }- const excludedTypes = [...trimmed.matchAll(/:not\(\[type='([^']+)'\]\)/g)].map((match) =>- match[1].toLowerCase()- )- const inputType = node.getAttribute('type')?.toLowerCase() ?? ''- return !excludedTypes.includes(inputType)- })-}--function createFillEvalNode(options: {- tagName: string- role?: string- ariaControls?: string- descendant?: FillEvalNode | null- descendants?: FillEvalNode[]- type?: string-}) {- const events: TestEvent[] = []- let value = ''- const proto = {- get value() {- return value- },- set value(next: string) {- value = next- }- }- const node = Object.create(proto) as FillEvalNode-- node.tagName = options.tagName- node.getAttribute = (name: string) => {- if (name === 'role') {- return options.role ?? null- }- if (name === 'aria-controls') {- return options.ariaControls ?? null- }- if (name === 'type') {- return options.type ?? null- }- return null- }- node.matches = vi.fn((selector: string) => matchesFillEvalSelector(node, selector))- const descendants = options.descendants ?? (options.descendant ? [options.descendant] : [])- node.querySelector = vi.fn(- (selector: string) => descendants.find((descendant) => descendant.matches(selector)) ?? null+function elementCalls(wc: MockWebContents): unknown[][] {+ return wc.debugger.sendCommand.mock.calls.filter(+ ([method]) => method === 'Runtime.callFunctionOn')- node.dispatchEvent = vi.fn((event: TestEvent) => {- events.push(event)- return true- })-- return {- node,- events,- get value() {- return value- }- }-}--function runFillEvalExpressions(- stdin: string[],- document: { activeElement: unknown; getElementById: (id: string) => unknown },- windowObject: Record<string, unknown> = {}-): void {- for (const expression of evalScripts(stdin)) {- new Function('document', 'Event', 'window', `return (${expression})`)(- document,- TestEvent,- windowObject- )- }-}--function createContentEditableEvalEnvironment(initialText: string) {- const editor = {- tagName: 'DIV',- isContentEditable: true,- textContent: initialText,- matches: vi.fn(() => false),- getAttribute: vi.fn((name: string) => (name === 'contenteditable' ? 'true' : null)),- dispatchEvent: vi.fn()- }- let selected = false- const selection = {- selectAllChildren: vi.fn(() => {- selected = true- })- }- const execCommand = vi.fn((command: string, _showUi: boolean, value: string) => {- // Empty insertText succeeds without clearing a selection; simulate Chromium requiring delete.- if (command === 'delete') {- if (!selected) {- return false- }- editor.textContent = ''- } else if (value.length > 0) {- editor.textContent = selected ? value : editor.textContent + value- }- selected = false- return true- })- return {- editor,- execCommand,- document: {- activeElement: editor,- body: {},- getElementById: () => null,- execCommand- },- windowObject: { getSelection: () => selection }- }}describe('AgentBrowserBridge', () => {let bridge: AgentBrowserBridge+ let wc: MockWebContentsbeforeEach(() => {+ capturedElement.current = { objectId: 'element-1', sessionId: 'frame-session' }resetAgentBrowserBridgeMocks({webContentsFromIdMock,existsSyncMock,@@ -247,6 +105,8 @@ describe('AgentBrowserBridge', () => {})bridge = new AgentBrowserBridge(mockBrowserManager())bridge.setActiveTab(100)+ wc = mockWebContents(100)+ webContentsFromIdMock.mockImplementation((id: number) => (id === 100 ? wc : null))})it('rejects browser clipboard writes that exceed the safe agent-browser argument size', async () => {@@ -259,196 +119,81 @@ describe('AgentBrowserBridge', () => {expect(execFileMock).not.toHaveBeenCalled()})- it('replaces contenteditable text through the browser editing pipeline', async () => {- succeedForContentEditable()- const environment = createContentEditableEvalEnvironment('existing text')-- await bridge.fill('@editor', 'replacement text')-- runFillEvalExpressions(stdinWrites, environment.document, environment.windowObject)-- expect(environment.editor.textContent).toBe('replacement text')- expect(environment.execCommand).toHaveBeenCalledWith('insertText', false, 'replacement text')- expect(environment.editor.dispatchEvent).not.toHaveBeenCalled()- })-- it('clears selected contenteditable text with a browser delete command', async () => {- succeedForContentEditable()- const environment = createContentEditableEvalEnvironment('existing text')-- const result = await bridge.clear('@editor')-- runFillEvalExpressions(stdinWrites, environment.document, environment.windowObject)-- expect(environment.editor.textContent).toBe('')- expect(environment.execCommand).toHaveBeenCalledWith('delete', false, '')- expect(environment.editor.dispatchEvent).not.toHaveBeenCalled()- expect(result).toEqual({ cleared: '@editor' })- })-- it('fails contenteditable fill when the browser editing command is unavailable', async () => {- succeedForContentEditable()- const environment = createContentEditableEvalEnvironment('existing text')- environment.execCommand.mockReturnValue(false)-- await bridge.fill('@editor', 'replacement text')-- expect(() =>- runFillEvalExpressions(stdinWrites, environment.document, environment.windowObject)- ).toThrow('Browser rich-text editing command failed')- expect(environment.editor.textContent).toBe('existing text')- expect(environment.editor.dispatchEvent).not.toHaveBeenCalled()+ it('fills the element the helper resolved, in its own frame, with one page call', async () => {+ const text = `${'é\n'.repeat(512 * 1024)}tail'\\`+ succeedWith({ value: null })+ wc.debugger.sendCommand.mockResolvedValue({ result: { type: 'object', value: null } })++ await expect(bridge.fill('@e3', text)).resolves.toEqual({ filled: '@e3' })++ expect(agentBrowserCommandsIn(stdinWrites)).toEqual([['get', 'attr', '@e3', 'id']])+ expect(execFileMock.mock.calls.flatMap((call) => call[1]).join('')).not.toContain('tail')+ expect(elementCalls(wc)).toEqual([+ [+ 'Runtime.callFunctionOn',+ {+ objectId: 'element-1',+ functionDeclaration: FILL_ELEMENT_FUNCTION,+ arguments: [{ value: text }],+ returnByValue: true+ },+ 'frame-session'+ ]+ ])})- it('inserts paste-sized contenteditable text as one stdin editing transaction', async () => {- const firstChunk = 'x'.repeat(AGENT_BROWSER_TEXT_ARGUMENT_MAX_BYTES)- succeedForContentEditable()- const environment = createContentEditableEvalEnvironment('existing text')-- await bridge.fill('@editor', `${firstChunk}tail`)+ it('clears by filling the resolved element with an empty value', async () => {+ succeedWith({ value: null })+ wc.debugger.sendCommand.mockResolvedValue({ result: { type: 'object', value: null } })- const evalCalls = execFileMock.mock.calls.filter((call: unknown[]) =>- (call[1] as string[]).includes('eval')- )- runFillEvalExpressions(stdinWrites, environment.document, environment.windowObject)+ await expect(bridge.clear('@e3')).resolves.toEqual({ cleared: '@e3' })- expect(evalCalls).toHaveLength(1)- expect(evalCalls[0][1]).toContain('--stdin')- expect(evalScripts(stdinWrites)).toHaveLength(1)- expect(environment.editor.textContent).toBe(`${firstChunk}tail`)+ expect(elementCalls(wc)[0]?.[1]).toMatchObject({ arguments: [{ value: '' }] })})- it('routes focused spinbutton wrappers to editable descendants before filling', async () => {- succeedWith({ ok: true })-- await bridge.fill('@spinbutton', '200')-- const expressions = stdinWrites-- const input = createFillEvalNode({ tagName: 'INPUT' })- const wrapper = createFillEvalNode({- tagName: 'DIV',- role: 'spinbutton',- descendant: input.node+ it('refuses an element the page function rejects as not interactable', async () => {+ succeedWith({ value: null })+ wc.debugger.sendCommand.mockResolvedValue({+ result: { type: 'string', value: 'it is not a text field or editable element' }})- runFillEvalExpressions(expressions, {- activeElement: wrapper.node,- getElementById: () => null+ await expect(bridge.fill('@e1', 'x')).rejects.toMatchObject({+ code: 'browser_element_not_interactable',+ message: 'Cannot fill @e1: it is not a text field or editable element'})-- expect(input.value).toBe('200')- expect(wrapper.value).toBe('')- expect(input.events.map((event) => event.type)).toEqual(['input', 'change'])- expect(wrapper.events).toHaveLength(0)})- it('routes aria-controlled spinbutton wrappers to editable inputs before filling', async () => {- succeedWith({ ok: true })-- await bridge.fill('@spinbutton', '200')-- const expressions = stdinWrites+ it('fails instead of writing anywhere when the helper resolves no element', async () => {+ capturedElement.current = null+ succeedWith({ value: null })- const input = createFillEvalNode({ tagName: 'INPUT' })- const wrapper = createFillEvalNode({- tagName: 'DIV',- role: 'spinbutton',- ariaControls: 'target-id'- })-- runFillEvalExpressions(expressions, {- activeElement: wrapper.node,- getElementById: (id: string) => (id === 'target-id' ? input.node : null)- })-- expect(input.value).toBe('200')- expect(wrapper.value).toBe('')- expect(input.events.map((event) => event.type)).toEqual(['input', 'change'])- expect(wrapper.events).toHaveLength(0)+ await expect(bridge.fill('@e1', 'x')).rejects.toMatchObject({ code: 'browser_error' })+ expect(elementCalls(wc)).toEqual([])})- it('routes aria-controlled spinbutton containers to editable descendants before filling', async () => {- succeedWith({ ok: true })-- await bridge.fill('@spinbutton', '200')-- const expressions = stdinWrites-- const input = createFillEvalNode({ tagName: 'INPUT' })- const controlled = createFillEvalNode({ tagName: 'DIV', descendant: input.node })- const wrapper = createFillEvalNode({- tagName: 'DIV',- role: 'spinbutton',- ariaControls: 'target-id'- })-- runFillEvalExpressions(expressions, {- activeElement: wrapper.node,- getElementById: (id: string) => (id === 'target-id' ? controlled.node : null)- })-- expect(input.value).toBe('200')- expect(controlled.value).toBe('')- expect(wrapper.value).toBe('')- expect(input.events.map((event) => event.type)).toEqual(['input', 'change'])- expect(controlled.events).toHaveLength(0)- expect(wrapper.events).toHaveLength(0)- })-- it('skips non-text spinbutton descendant inputs before filling', async () => {- succeedWith({ ok: true })-- await bridge.fill('@spinbutton', '200')-- const expressions = stdinWrites-- const hiddenInput = createFillEvalNode({ tagName: 'INPUT', type: 'hidden' })- const numberInput = createFillEvalNode({ tagName: 'INPUT', type: 'number' })- const wrapper = createFillEvalNode({- tagName: 'DIV',- role: 'spinbutton',- descendants: [hiddenInput.node, numberInput.node]+ it('refuses select on an element that is not a <select> before the helper reports success', async () => {+ succeedWith({ selected: ['Mango'] })+ wc.debugger.sendCommand.mockResolvedValue({+ result: { type: 'string', value: 'it is a combobox, not a <select>' }})- runFillEvalExpressions(expressions, {- activeElement: wrapper.node,- getElementById: () => null+ await expect(bridge.select('@e9', 'Mango')).rejects.toMatchObject({+ code: 'browser_element_not_interactable'})-- expect(numberInput.value).toBe('200')- expect(hiddenInput.value).toBe('')- expect(wrapper.value).toBe('')- expect(numberInput.events.map((event) => event.type)).toEqual(['input', 'change'])- expect(hiddenInput.events).toHaveLength(0)- expect(wrapper.events).toHaveLength(0)+ expect(agentBrowserCommandsIn(stdinWrites)).toEqual([['get', 'attr', '@e9', 'id']])})- it('fills large plain fields with one stdin edit and one event pair', async () => {- const text = `${'é\n'.repeat(512 * 1024)}tail'\\`- succeedWith({ ok: true })-- await bridge.fill('@textarea', text)+ it('refuses to upload a path that is not an existing file', async () => {+ succeedWith({ uploaded: 1 })- const evalCalls = execFileMock.mock.calls.filter((call: unknown[]) =>- (call[1] as string[]).includes('eval')- )- expect(evalCalls).toHaveLength(1)- expect(evalCalls[0][1]).toContain('--stdin')- expect(evalScripts(stdinWrites)).toHaveLength(1)- expect((evalCalls[0][1] as string[]).join('')).not.toContain(text)- const input = createFillEvalNode({ tagName: 'TEXTAREA' })- runFillEvalExpressions(stdinWrites, {- activeElement: input.node,- getElementById: () => null+ await expect(bridge.upload('@e19', ['/nonexistent/orca-upload.txt'])).rejects.toMatchObject({+ code: 'invalid_argument'})- expect(input.value).toBe(text)- expect(input.events.map((event) => event.type)).toEqual(['input', 'change'])+ expect(agentBrowserCommandsIn(stdinWrites).map(([name]) => name)).not.toContain('upload')})it.each([['fill', (b: AgentBrowserBridge, text: string) => b.fill('@textarea', text)],- ['type', (b: AgentBrowserBridge, text: string) => b.type(text)],['keyboard insert', (b: AgentBrowserBridge, text: string) => b.keyboardInsertText(text)]])('yields before spawning agent-browser for accepted large %s text', async (_name, run) => {vi.useFakeTimers()@@ -479,19 +224,6 @@ describe('AgentBrowserBridge', () => {expect(agentBrowserCommandsIn(stdinWrites)).toEqual([['keyboard', 'inserttext', '--help']])})- it('chunks large agent-browser type text before keyboard transport', async () => {- const text = ['y'.repeat(AGENT_BROWSER_TEXT_ARGUMENT_MAX_BYTES), 'zz'].join('')- succeedWith({ typed: true })-- await bridge.type(text)-- const chunks = agentBrowserCommandsIn(stdinWrites)- .filter(([name, subcommand]) => name === 'keyboard' && subcommand === 'type')- .map(([, , chunk]) => chunk)-- expect(chunks).toEqual(['y'.repeat(AGENT_BROWSER_TEXT_ARGUMENT_MAX_BYTES), 'zz'])- })-it('chunks large agent-browser keyboard insert text before transport', async () => {const text = ['z'.repeat(AGENT_BROWSER_TEXT_ARGUMENT_MAX_BYTES), 'qq'].join('')succeedWith({ inserted: true })diff --git a/src/main/browser/browser-element-functions.test.ts b/src/main/browser/browser-element-functions.test.tsnew file mode 100644index 00000000000..9c7cc504f75--- /dev/null+++ b/src/main/browser/browser-element-functions.test.ts@@ -0,0 +1,155 @@+// @vitest-environment happy-dom+import { describe, it, expect, vi, beforeEach } from 'vitest'+import {+ FILE_INPUT_TARGET_FUNCTION,+ FILL_ELEMENT_FUNCTION,+ SELECT_ALL_ELEMENT_FUNCTION,+ SELECT_OPTION_TARGET_FUNCTION+} from './browser-element-functions'++// CDP's Runtime.callFunctionOn calls the declaration with the resolved element as `this`.+function callOn(element: Element, declaration: string, ...args: unknown[]): unknown {+ return new Function(`return (${declaration})`)().apply(element, args)+}++function mount(html: string): HTMLElement {+ document.body.innerHTML = html+ return document.body+}++describe('FILL_ELEMENT_FUNCTION', () => {+ beforeEach(() => {+ document.body.innerHTML = ''+ })++ it('sets an input inside an open shadow root and fires one input and change event', () => {+ const host = mount('<div id="host"></div>').querySelector('#host')!+ const input = host.attachShadow({ mode: 'open' }).appendChild(document.createElement('input'))+ const events: string[] = []+ input.addEventListener('input', () => events.push('input'))+ input.addEventListener('change', () => events.push('change'))+ // Like a person's input, it crosses the shadow boundary for page-level listeners.+ document.addEventListener('input', () => events.push('document input'))++ expect(callOn(input, FILL_ELEMENT_FUNCTION, 'hello')).toBeNull()++ expect(input.value).toBe('hello')+ expect(events).toEqual(['input', 'document input', 'change'])+ })++ it.each([+ ['<h1>Title</h1>', 'h1', 'it is not a text field or editable element'],+ ['<input disabled>', 'input', 'it is disabled'],+ ['<textarea readonly></textarea>', 'textarea', 'it is read-only'],+ ['<input type="checkbox">', 'input', 'it is not a text field or editable element'],+ ['<select><option>a</option></select>', 'select', 'it is a <select>; use select instead']+ ])('refuses %s', (html, selector, reason) => {+ const element = mount(html).querySelector(selector)!++ expect(callOn(element, FILL_ELEMENT_FUNCTION, 'x')).toBe(reason)+ })++ it('keeps the previous value when a number field sanitizes the new one away', () => {+ const input = mount('<input type="number" value="42">').querySelector('input')!+ const events: string[] = []+ input.addEventListener('input', () => events.push('input'))++ expect(callOn(input, FILL_ELEMENT_FUNCTION, 'abc')).toBe('the field rejected this value')++ expect(input.value).toBe('42')+ expect(events).toEqual([])+ })++ it('fills the input an aria-controlled spinbutton points at', () => {+ const body = mount('<div role="spinbutton" aria-controls="qty"></div><input id="qty">')++ callOn(body.querySelector('[role=spinbutton]')!, FILL_ELEMENT_FUNCTION, '200')++ expect(body.querySelector<HTMLInputElement>('#qty')!.value).toBe('200')+ })++ it('fills the text descendant of a spinbutton wrapper, skipping non-text inputs', () => {+ const body = mount(+ '<div role="spinbutton"><input type="hidden" id="h"><input type="number" id="n"></div>'+ )++ callOn(body.querySelector('[role=spinbutton]')!, FILL_ELEMENT_FUNCTION, '7')++ expect(body.querySelector<HTMLInputElement>('#n')!.value).toBe('7')+ expect(body.querySelector<HTMLInputElement>('#h')!.value).toBe('')+ })++ it.each([+ ['replaces', 'new text', 'insertText'],+ ['clears', '', 'delete']+ ])('%s contenteditable text through a browser edit command', (_name, value, command) => {+ const editor = mount('<div contenteditable="true">old</div>').querySelector('div')!+ const execCommand = vi.fn(() => true)+ Object.defineProperty(document, 'execCommand', { configurable: true, value: execCommand })++ expect(callOn(editor, FILL_ELEMENT_FUNCTION, value)).toBeNull()++ expect(execCommand).toHaveBeenCalledWith(command, false, value)+ expect(window.getSelection()?.toString()).toBe('old')+ })++ it('refuses a rich-text edit when a focus handler moves focus to another field', () => {+ const body = mount('<div contenteditable="true">old</div><input value="keep">')+ const execCommand = vi.fn(() => true)+ Object.defineProperty(document, 'execCommand', { configurable: true, value: execCommand })+ body.querySelector('div')!.addEventListener('focus', () => body.querySelector('input')!.focus())++ expect(callOn(body.querySelector('div')!, FILL_ELEMENT_FUNCTION, 'new')).toBe(+ 'it gave its focus to another element'+ )+ expect(execCommand).not.toHaveBeenCalled()+ })++ it('refuses an input the page replaces when it takes focus', () => {+ const input = mount('<input value="old">').querySelector('input')!+ input.addEventListener('focus', () => input.replaceWith(document.createElement('input')))++ expect(callOn(input, FILL_ELEMENT_FUNCTION, 'new')).toMatch(/replaced it/)+ })++ it('reports a rich-text edit the browser refused', () => {+ const editor = mount('<div contenteditable="true">old</div>').querySelector('div')!+ Object.defineProperty(document, 'execCommand', { configurable: true, value: () => false })++ expect(callOn(editor, FILL_ELEMENT_FUNCTION, 'new')).toBe('the browser editing command failed')+ })+})++describe('element target checks', () => {+ it.each([+ ['<div role="combobox">Pick</div>', 'div', 'it is a combobox, not a <select>'],+ ['<h1>x</h1>', 'h1', 'it is <h1>, not a <select>'],+ ['<select disabled></select>', 'select', 'it is disabled']+ ])('refuses select on %s', (html, selector, reason) => {+ const element = mount(html).querySelector(selector)!++ expect(callOn(element, SELECT_OPTION_TARGET_FUNCTION)).toMatch(reason)+ })++ it('accepts select on an enabled <select> and upload on a file input', () => {+ const body = mount('<select></select><input type="file">')++ expect(callOn(body.querySelector('select')!, SELECT_OPTION_TARGET_FUNCTION)).toBeNull()+ expect(callOn(body.querySelector('input')!, FILE_INPUT_TARGET_FUNCTION)).toBeNull()+ })++ it('refuses upload on an element that is not a file input', () => {+ const button = mount('<button>Attach</button>').querySelector('button')!++ expect(callOn(button, FILE_INPUT_TARGET_FUNCTION)).toBe('it is not a file input')+ })++ it('selects all of an input value and reports that it holds focus', () => {+ const input = mount('<input value="abcdef">').querySelector('input')!+ input.setSelectionRange(3, 3)++ expect(callOn(input, SELECT_ALL_ELEMENT_FUNCTION)).toBe(true)++ expect([input.selectionStart, input.selectionEnd]).toEqual([0, 6])+ })+})diff --git a/src/main/browser/browser-element-functions.ts b/src/main/browser/browser-element-functions.tsnew file mode 100644index 00000000000..48efd8467b4--- /dev/null+++ b/src/main/browser/browser-element-functions.ts@@ -0,0 +1,81 @@+// Page functions for Runtime.callFunctionOn; `this` is the element the agent's ref resolved to.+// A returned string is the reason the element cannot take the action.++export const FILL_ELEMENT_FUNCTION = `function (value) {+ const isTextField = (node) =>+ node?.localName === 'textarea' ||+ (node?.localName === 'input' && !/^(hidden|button|checkbox|radio|file|image|reset|submit)$/i.test(node.type));+ const findTextField = (root) => Array.from(root?.querySelectorAll?.('input, textarea') ?? []).find(isTextField) ?? null;+ let target = this;+ // Spinbutton wrappers can take the ref while another input owns the value.+ if (!isTextField(target) && target.getAttribute?.('role') === 'spinbutton') {+ const root = target.getRootNode();+ for (const id of (target.getAttribute('aria-controls') ?? '').split(/\\s+/).filter(Boolean)) {+ const controlled = root.getElementById?.(id) ?? null;+ const field = isTextField(controlled) ? controlled : findTextField(controlled);+ if (field) { target = field; break; }+ }+ if (target === this) target = findTextField(this) ?? this;+ }+ if (target.matches?.(':disabled')) return 'it is disabled';+ if (target.isContentEditable && !isTextField(target)) {+ // Rich editors revert direct DOM writes; browser edit transactions keep their models in sync.+ target.focus();+ // The edit lands wherever focus is, so a focus handler that moved it must stop the write.+ const active = target.getRootNode().activeElement;+ if (!active || !(active === target || active.contains(target) || target.contains(active))) {+ return 'it gave its focus to another element';+ }+ const selection = target.ownerDocument.defaultView?.getSelection?.();+ if (!selection) return 'its text selection is unavailable';+ selection.selectAllChildren(target);+ let edited = false;+ try {+ edited = target.ownerDocument.execCommand(value.length === 0 ? 'delete' : 'insertText', false, value) === true;+ } catch {}+ return edited ? null : 'the browser editing command failed';+ }+ if (target.localName === 'select') return 'it is a <select>; use select instead';+ if (!isTextField(target)) return 'it is not a text field or editable element';+ if (target.readOnly) return 'it is read-only';+ target.focus();+ if (!target.isConnected) return 'the page replaced it when it took focus; take a new snapshot';+ const nativeSetter = Object.getOwnPropertyDescriptor(Object.getPrototypeOf(target), 'value')?.set;+ const setValue = (next) => (nativeSetter ? nativeSetter.call(target, next) : (target.value = next));+ const previous = target.value;+ setValue(value);+ // Sanitizing fields (number, email, date) empty a value they cannot hold; handlers may clear it later.+ if (value !== '' && target.value === '') {+ setValue(previous);+ return 'the field rejected this value';+ }+ target.dispatchEvent(new Event('input', { bubbles: true, composed: true }));+ target.dispatchEvent(new Event('change', { bubbles: true }));+ return null;+}`++// Returns whether an editable element holds focus, so a select-all shortcut would stay inside it.+export const SELECT_ALL_ELEMENT_FUNCTION = `function () {+ this.focus();+ const active = this.getRootNode().activeElement;+ const selectable = typeof this.select === 'function';+ const focused = (selectable || this.isContentEditable) && (active === this || this.contains(active));+ if (selectable) { this.select(); return focused; }+ const selection = this.ownerDocument.defaultView?.getSelection?.();+ if (!selection) return 'its text selection is unavailable';+ selection.selectAllChildren(this);+ return focused;+}`++export const SELECT_OPTION_TARGET_FUNCTION = `function () {+ if (this.localName !== 'select') {+ const role = this.getAttribute?.('role');+ return 'it is ' + (role ? 'a ' + role : '<' + this.localName + '>') + ', not a <select>; click it and choose the option instead';+ }+ return this.matches(':disabled') ? 'it is disabled' : null;+}`++export const FILE_INPUT_TARGET_FUNCTION = `function () {+ if (this.localName !== 'input' || this.type !== 'file') return 'it is not a file input';+ return this.matches(':disabled') ? 'it is disabled' : null;+}`diff --git a/src/main/browser/browser-upload-files.ts b/src/main/browser/browser-upload-files.tsnew file mode 100644index 00000000000..d5e68250230--- /dev/null+++ b/src/main/browser/browser-upload-files.ts@@ -0,0 +1,16 @@+import { stat } from 'node:fs/promises'+import { BrowserError } from './browser-error'++/** Chromium attaches a missing path as an empty file, so check the paths before handing them over. */+export async function assertUploadFilesExist(filePaths: readonly string[]): Promise<void> {+ for (const filePath of filePaths) {+ // Directories stay allowed: webkitdirectory inputs take them.+ const exists = await stat(filePath).then(+ () => true,+ () => false+ )+ if (!exists) {+ throw new BrowserError('invalid_argument', `Cannot upload ${filePath}: it does not exist`)+ }+ }+}diff --git a/src/main/browser/cdp-element-capture.ts b/src/main/browser/cdp-element-capture.tsnew file mode 100644index 00000000000..760ea1eaf89--- /dev/null+++ b/src/main/browser/cdp-element-capture.ts@@ -0,0 +1,29 @@+// A helper-resolved remote object; its session is the debugger's, undefined for the page itself.+export type CapturedElement = { objectId: string; sessionId: string | undefined }++/** Records the element the helper's calls run on, so Orca can act on that exact node. */+export class CdpElementCapture {+ // Undefined while not capturing.+ private captured: CapturedElement | null | undefined++ start(): void {+ this.captured = null+ }++ stop(): CapturedElement | null {+ const captured = this.captured ?? null+ this.captured = undefined+ return captured+ }++ observe(method: string, params: Record<string, unknown> | undefined, sessionId?: string): void {+ const objectId = params?.objectId+ if (+ this.captured !== undefined &&+ method === 'Runtime.callFunctionOn' &&+ typeof objectId === 'string'+ ) {+ this.captured = { objectId, sessionId }+ }+ }+}diff --git a/src/main/browser/cdp-keyboard-us-layout.ts b/src/main/browser/cdp-keyboard-us-layout.tsindex 0bbab9946f7..5210c075525 100644--- a/src/main/browser/cdp-keyboard-us-layout.ts+++ b/src/main/browser/cdp-keyboard-us-layout.ts@@ -237,3 +237,44 @@ export function parseCdpKeyEvent(raw: string): CdpKeyEvent | null {return { keyCode, key, code, modifiers, location, selfModifier, text }}++/** The key a person presses to type one character, or null when no key types it. */+function typedCharacterKeyEvent(char: string): CdpKeyEvent | null {+ if (char === '\n') {+ return parseCdpKeyEvent('Enter')+ }+ if (char === '\t') {+ return parseCdpKeyEvent('Tab')+ }+ return parseCdpKeyEvent(char) ?? imeFallbackKeyEvent(char)+}++const MAX_UNTYPABLE_RUN_LENGTH = 512++/** A key to press, or a run of text that no key types. */+export type TypedInputStep = CdpKeyEvent | string++/** The keys that type `input`, yielded lazily so long text is never held as events. */+export function* typedInputSteps(input: string): Generator<TypedInputStep> {+ let untypable = ''+ for (const char of input.replace(/\r\n?/g, '\n')) {+ const key = typedCharacterKeyEvent(char)+ if (!key) {+ untypable += char+ // Bounded like key runs, since a widget page receives this text one event per character.+ if (untypable.length >= MAX_UNTYPABLE_RUN_LENGTH) {+ yield untypable+ untypable = ''+ }+ continue+ }+ if (untypable) {+ yield untypable+ untypable = ''+ }+ yield key+ }+ if (untypable) {+ yield untypable+ }+}diff --git a/src/main/browser/cdp-ws-proxy.ts b/src/main/browser/cdp-ws-proxy.tsindex 0d3934d09e0..b5d84a5ee2c 100644--- a/src/main/browser/cdp-ws-proxy.ts+++ b/src/main/browser/cdp-ws-proxy.ts@@ -7,6 +7,7 @@ import { CdpTargetDiscovery } from './cdp-target-discovery'import { CdpDebuggerChannel } from './cdp-debugger-channel'import { CdpPageNavigationCommands } from './cdp-page-navigation-commands'import { CdpDomFocusReplay } from './cdp-dom-focus-replay'+import { CdpElementCapture } from './cdp-element-capture'import { CdpPageCaptureCommands } from './cdp-page-capture-commands'import type { CapturePaintHold } from './cdp-screenshot'import type { AgentInputFocusGuard } from './agent-input-focus-guard'@@ -34,6 +35,7 @@ export class CdpWsProxy {private readonly pageCapture: CdpPageCaptureCommandsprivate readonly input: CdpPageInputCommandsprivate pointAtNextResolvedNode = false+ readonly elementCapture = new CdpElementCapture()constructor(private readonly webContents: WebContents,@@ -248,6 +250,7 @@ export class CdpWsProxy {)return}+ this.elementCapture.observe(msg.method, msg.params, effectiveSessionId)if (msg.method === 'DOM.focus') {this.domFocusReplay.forwardDomFocus(client, clientId, msg.params ?? {}, effectiveSessionId)returndiff --git a/src/main/browser/guest-direct-key-input.test.ts b/src/main/browser/guest-direct-key-input.test.tsindex 8bf5c9f9228..0249a91439e 100644--- a/src/main/browser/guest-direct-key-input.test.ts+++ b/src/main/browser/guest-direct-key-input.test.ts@@ -2,6 +2,7 @@ import { describe, expect, it, vi } from 'vitest'import type { WebContents } from 'electron'import {dispatchPageKeyEvent,+ dispatchPageKeyEvents,routePageTextInsertion,type GuestCdpSender} from './guest-direct-key-input'@@ -225,6 +226,23 @@ describe('dispatchPageKeyEvent', () => {expect(page.selectAll).toHaveBeenCalledTimes(1)})+ it('settles a run of widget keys with one focus probe and one page round trip', async () => {+ const { page, webContents } = fakePage('webview')+ const send = fakeSender()+ const keys = ['a', 'b', 'c'].flatMap((key) => [+ { type: 'keyDown', key, text: key },+ { type: 'keyUp', key }+ ])++ await dispatchPageKeyEvents(webContents, keys, send)++ expect(page.sendInputEvent).toHaveBeenCalledTimes(9)+ expect(send.mock.calls.map(([method]) => method)).toEqual([+ 'Runtime.evaluate',+ 'Runtime.evaluate'+ ])+ })+it('types into a focused cross-site frame on its own session, without focusing the guest', async () => {const { page, webContents } = fakePage('webview')const send = crossSiteFrameSender({ attachable: true })@@ -238,6 +256,31 @@ describe('dispatchPageKeyEvent', () => {expect(send).toHaveBeenLastCalledWith('Input.dispatchKeyEvent', params, 'S1')})+ it('follows focus that a frame key handler moves into the parent page mid-run', async () => {+ const { page, webContents } = fakePage('webview')+ const inFrame = crossSiteFrameSender({ attachable: true })+ let focusInFrame = true+ const send = vi.fn<GuestCdpSender>(async (method, params, sessionId) => {+ if (method === 'Input.dispatchKeyEvent' && params.type === 'keyUp') {+ // The frame's handler sends focus to the parent after the first key.+ focusInFrame = false+ }+ return focusInFrame || sessionId+ ? inFrame(method, params, sessionId)+ : fakeSender()(method, params, sessionId)+ })+ const keys = ['a', 'b'].flatMap((key) => [+ { type: 'keyDown', key, text: key },+ { type: 'keyUp', key }+ ])++ await dispatchPageKeyEvents(webContents, keys, send)++ const frameKeys = send.mock.calls.filter(([method]) => method === 'Input.dispatchKeyEvent')+ expect(frameKeys.map(([, params]) => params.key)).toEqual(['a', 'a'])+ expect(page.sendInputEvent).toHaveBeenCalledWith(expect.objectContaining({ keyCode: 'b' }))+ })+it('borrows focus for CDP when the focused frame cannot be reached, then gives it back', async () => {const { page, webContents } = fakePage('webview')const send = crossSiteFrameSender({ attachable: false })diff --git a/src/main/browser/guest-direct-key-input.ts b/src/main/browser/guest-direct-key-input.tsindex d2f0b34896d..ff1a49aec5c 100644--- a/src/main/browser/guest-direct-key-input.ts+++ b/src/main/browser/guest-direct-key-input.ts@@ -101,48 +101,88 @@ export async function dispatchPageKeyEvent(send: GuestCdpSender,guardFocus: AgentInputFocusGuard = NO_FOCUS_GUARD): Promise<unknown> {+ return await dispatchPageKeyEvents(webContents, [params], send, guardFocus)+}++const isTabDown = (params: Record<string, unknown>): boolean =>+ params.key === 'Tab' && (params.type === 'keyDown' || params.type === 'rawKeyDown')++/** Deliver a run of key events along one focus route; callers split runs where focus may move. */+export async function dispatchPageKeyEvents(+ webContents: WebContents,+ events: readonly Record<string, unknown>[],+ send: GuestCdpSender,+ guardFocus: AgentInputFocusGuard = NO_FOCUS_GUARD+): Promise<unknown> {+ let result: unknown = {}if (!isEmbeddedGuest(webContents)) {- return await send('Input.dispatchKeyEvent', params)+ for (const params of events) {+ result = await send('Input.dispatchKeyEvent', params)+ }+ return result}- const route = await resolveFocusedFrameRoute(webContents, send)- if (route.kind === 'session') {+ let start = 0+ let route = await resolveFocusedFrameRoute(webContents, send)+ // A frame's key handler can move focus to another frame, so each key finds its route again.+ while (route.kind === 'session') {+ const keyUp = events.findIndex((params, index) => index >= start && params.type === 'keyUp')+ const end = keyUp === -1 ? events.length : keyUp + 1+ const key = events.slice(start, end)// Tab from the last frame field can escape into Orca's window.- const isTabDown =- params.key === 'Tab' && (params.type === 'keyDown' || params.type === 'rawKeyDown')- const release = isTabDown ? await guardFocus() : () => {}+ const release = key.some(isTabDown) ? await guardFocus() : () => {}try {- return await sendToFrameSession(- webContents,- send,- route.sessionId,- 'Input.dispatchKeyEvent',- params- )+ for (const params of key) {+ result = await sendToFrameSession(+ webContents,+ send,+ route.sessionId,+ 'Input.dispatchKeyEvent',+ params+ )+ }} finally {release()}+ if (end >= events.length) {+ return result+ }+ start = end+ route = await resolveFocusedFrameRoute(webContents, send)}- const plan = planCdpKeyEvent(params)- if (plan && route.kind === 'widget') {+ const remaining = events.slice(start)+ const plans = remaining.map((params) => planCdpKeyEvent(params))+ if (route.kind === 'widget' && plans.every((plan) => plan !== null)) {// Tab from the last page field can escape into Orca's window.- const release = plan.events.some((event) => event.type === 'keyDown' && event.keyCode === 'Tab')- ? await guardFocus()- : () => {}+ const release = remaining.some(isTabDown) ? await guardFocus() : () => {}try {- await withAgentInputInjection(webContents, () =>- // Widget events bypass CDP send, so track held keys here.- trackRemoteCliInput(webContents.id, 'Input.dispatchKeyEvent', params, async () => {- for (const event of plan.events) {- webContents.sendInputEvent(event)- }- if (plan.commands.length > 0 || plan.events.length > 0) {- await waitForPageToHandleInput(send)- }- for (const command of plan.commands) {- runEditCommand(webContents, command)- }- })- )+ await withAgentInputInjection(webContents, async () => {+ // One page round trip settles the whole run; edit commands must see the keys before them.+ let unsettled = false+ for (const [index, plan] of plans.entries()) {+ // Widget events bypass CDP send, so track held keys here.+ await trackRemoteCliInput(+ webContents.id,+ 'Input.dispatchKeyEvent',+ remaining[index],+ async () => {+ for (const event of plan.events) {+ webContents.sendInputEvent(event)+ }+ unsettled ||= plan.events.length > 0+ if (plan.commands.length > 0) {+ await waitForPageToHandleInput(send)+ unsettled = false+ for (const command of plan.commands) {+ runEditCommand(webContents, command)+ }+ }+ }+ )+ }+ if (unsettled) {+ await waitForPageToHandleInput(send)+ }+ })} finally {release()}@@ -150,7 +190,12 @@ export async function dispatchPageKeyEvent(}const release = await focusGuestForCdpInput(webContents, send, guardFocus)try {- return await withAgentInputInjection(webContents, () => send('Input.dispatchKeyEvent', params))+ for (const params of remaining) {+ result = await withAgentInputInjection(webContents, () =>+ send('Input.dispatchKeyEvent', params)+ )+ }+ return result} finally {release()}diff --git a/src/main/orcad/external-chromium-browser-process.ts b/src/main/orcad/external-chromium-browser-process.tsindex f5965afa60a..7b8d25a8b65 100644--- a/src/main/orcad/external-chromium-browser-process.ts+++ b/src/main/orcad/external-chromium-browser-process.ts@@ -2,6 +2,7 @@ import { z } from 'zod'import type { RuntimeBrowserCommandHost } from '../runtime/browser/browser-command-contracts'import type { RuntimeBrowserCommands } from '../runtime/browser/browser-commands'import { BrowserError } from '../browser/browser-error'+import { assertUploadFilesExist } from '../browser/browser-upload-files'import {ExternalChromiumBrowserSession,type ExternalChromiumLaunch@@ -179,6 +180,10 @@ export class ExternalChromiumBrowserProcess {return { url: tab.url, title: tab.title }}+ if (method === 'browserUpload') {+ const files: unknown[] = Array.isArray(params.files) ? params.files : []+ await assertUploadFilesExist(files.map(String))+ }const command = externalChromiumCommandArguments(method, params)if (!command) {throw new BrowserError(diff --git a/tests/e2e/helpers/browser-agent-focus-scenarios.ts b/tests/e2e/helpers/browser-agent-focus-scenarios.tsindex 037a888f91e..5e136767c1a 100644--- a/tests/e2e/helpers/browser-agent-focus-scenarios.ts+++ b/tests/e2e/helpers/browser-agent-focus-scenarios.ts@@ -216,8 +216,8 @@ export async function runAgentKeysScenario(`JSON.stringify({ submits: window.__submits, keys: window.__keys })`))- // Why: `type` inserts text without key events, so only the keypresses log keydowns.- .toBe(JSON.stringify({ submits: 1, keys: ['Z', 'Enter'] }))+ // Why: `type` presses real keys, so its characters log keydowns like the keypresses.+ .toBe(JSON.stringify({ submits: 1, keys: ['Z', 'q', 'r', 'Enter'] }))await afterAgentInput?.(pageId)} finally {await server.close()@@ -257,7 +257,7 @@ export async function runAgentClicksScenario(.toBe(JSON.stringify({ active: 'probe', focused: true, blurs: 0 }))expect((await readHost(page)).terminalFocused).toBe(true)- // Why: text with a newline cannot be typed as key events; CDP delivers it with focus borrowed.+ // Why: fill writes multiline values straight into the field, never through the user's focus.await rpc(page, 'browser.fill', { page: pageId, element: '#notes', value: 'a\nb' })await expect.poll(() => evalInPage(page, pageId, `document.getElementById('notes').value`))