Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cua-driver): browser JS primitives + menu-bar ghost click fix (#1386)

Francesco Bonacci7d75c4115a8d
diff --git a/libs/cua-driver/Skills/cua-driver/SKILL.md b/libs/cua-driver/Skills/cua-driver/SKILL.md
index 996c6aab9f..7dbcd67391 100644
--- a/libs/cua-driver/Skills/cua-driver/SKILL.md
+++ b/libs/cua-driver/Skills/cua-driver/SKILL.md
@@ -747,6 +747,69 @@ Chromium web content specifically also coerces `right_click` back to
left — use `element_index` for AX-addressable targets and accept the
limit otherwise.
+### Browser JS primitives — `page` tool and `get_window_state(javascript=)`
+
+When the AX tree doesn't expose the data you need (common in
+Chromium/Electron — the tree is sparse for web content), use the
+`page` tool or the `javascript` param on `get_window_state` to query
+the DOM directly via Apple Events. Requires "Allow JavaScript from
+Apple Events" to be enabled — see `WEB_APPS.md` for the setup path.
+
+**Three actions on the `page` tool:**
+
+- `page({pid, window_id, action: "get_text"})` — returns
+ `document.body.innerText`. Fastest way to read page content, prices,
+ article text, or any raw text the AX tree truncates or omits.
+
+- `page({pid, window_id, action: "query_dom", css_selector: "a[href]",
+ attributes: ["href"]})` — runs `querySelectorAll` and returns each
+ match's tag, text, and requested attributes as a JSON array. Use for
+ table rows, link hrefs, data attributes, structured page data.
+
+- `page({pid, window_id, action: "execute_javascript", javascript:
+ "..."})` — raw JS. Wrap in an IIFE with try-catch. Don't use this for
+ elements already indexed by `get_window_state` — `click` and
+ `set_value` are more reliable there.
+
+**Co-located read — `get_window_state` with `javascript`:**
+
+```
+get_window_state({pid, window_id, javascript: "document.title"})
+```
+
+Runs the JS and appends the result as a `## JavaScript result` section
+alongside the AX snapshot — one round-trip instead of two. Use this
+when you need both the element tree (for subsequent clicks) and some
+page data in the same turn.
+
+**Decision rule — AX vs JS:**
+
+| Need | Use |
+|---|---|
+| Click / type into an element | `get_window_state` → `click` / `set_value` (AX, works backgrounded) |
+| Read text the AX tree drops | `page(get_text)` or `get_window_state(javascript=)` |
+| Scrape structured data (tables, hrefs) | `page(query_dom)` |
+| Trigger JS events / mutations | `page(execute_javascript)` |
+
+Supported backends:
+
+| App type | How | Context |
+|---|---|---|
+| Chrome / Brave / Edge | Apple Events `execute javascript` | Full DOM ✅ |
+| Safari | Apple Events `do JavaScript` | Full DOM ✅ |
+| Electron (VS Code, Cursor…) | SIGUSR1 → V8 inspector → CDP | Main process only: `process`, `Buffer` — no `document`, no `require` in sandboxed apps |
+| Electron (with `--remote-debugging-port`) | CDP page target | Full DOM ✅ |
+
+**Electron sandbox note:** SIGUSR1 connects to the Node.js *main* process.
+Sandboxed Electron apps (VS Code, Cursor) strip `require` and Electron
+APIs there. Useful for: `process.env`, `process.versions`, `process.cwd()`,
+`process.pid`. For full DOM/renderer access, launch the app with
+`--remote-debugging-port=9222` — cua-driver will detect and prefer the
+page target automatically.
+
+Arc returns no values; Firefox has no JS-via-AppleEvents support — see
+`WEB_APPS.md` for the full matrix.
+
### 3. Re-snapshot and verify — mandatory
**Always** call `get_window_state({pid, window_id})` after the action.
diff --git a/libs/cua-driver/Skills/cua-driver/WEB_APPS.md b/libs/cua-driver/Skills/cua-driver/WEB_APPS.md
index 5ea0cc2e4f..6fbaef9e21 100644
--- a/libs/cua-driver/Skills/cua-driver/WEB_APPS.md
+++ b/libs/cua-driver/Skills/cua-driver/WEB_APPS.md
@@ -239,6 +239,225 @@ coerces the event back to a left-click — this appears to affect
every non-HID-tap synthesis path. Prefer `element_index` whenever
the target is AX-addressable.
+## Enable "Allow JavaScript from Apple Events" — browser support matrix
+
+| Browser | `execute javascript` supported | Setting needed | Programmatic path |
+|---|---|---|---|
+| Chrome | ✅ Full | ✅ Yes | Edit Preferences JSON (see below) |
+| Brave | ✅ Full | ✅ Yes | Edit Preferences JSON (same key, different path) |
+| Edge | ✅ Full | ✅ Yes | Edit Preferences JSON (same key, different path) |
+| Safari | ✅ Full (`do JavaScript`) | ✅ Yes | UI automation only — `defaults write` broken |
+| Arc | ⚠️ No return values | No toggle | No reliable path |
+| Firefox | ❌ Not supported | N/A | N/A |
+
+### Chrome / Brave / Edge — Preferences JSON
+
+Required for `osascript execute javascript` calls. All three are
+Chromium-based and share the same preference key and mechanism.
+Each browser stores preferences per-profile.
+
+### Why menu clicks don't work
+
+The menu item (`View → Developer → Allow JavaScript from Apple Events`)
+is a security-sensitive toggle. Verified experimentally:
+
+- `AXPress` — advertised actions are `[AXCancel, AXPick]`, not
+ `AXPress`; Chrome's command dispatch silently discards it.
+- `AXPick` on a leaf item — opens submenus correctly but does NOT
+ commit a leaf toggle; the item is "selected" but not activated.
+- System Events `click theItem` / `click at {x, y}` — returns the
+ menu item reference (found it) but Chrome requires a genuine
+ trusted user event to flip this flag; synthetic AppleEvent-routed
+ clicks are rejected.
+- `CGEvent.post(tap: .cghidEventTap)` while the menu is open —
+ Chrome's event loop is occupied processing the menu; the event
+ either races or Chrome treats it as untrusted for this toggle.
+
+Additionally, when Chrome is **backgrounded**, the Developer submenu
+items appear with `AXEnabled = false` (Chrome's `commandDispatch`
+marks them DISABLED) — any action dispatched returns `.success` at
+the AX layer but is silently discarded. This is the root cause of the
+"ghost click" pattern: the driver reports ✅ but nothing changes.
+
+### Correct path — write the Preferences JSON directly
+
+Quit Chrome first, then write the flag, then relaunch:
+
+```bash
+# 1. Quit Chrome
+osascript -e 'quit app "Google Chrome"'
+sleep 1
+
+# 2. Write the flag into the active profile's Preferences.
+# Chrome stores this in TWO places — both must be set.
+python3 -c "
+import json, os
+prefs_path = os.path.expanduser(
+ '~/Library/Application Support/Google/Chrome/Default/Preferences')
+data = json.load(open(prefs_path))
+data.setdefault('browser', {})['allow_javascript_apple_events'] = True
+data.setdefault('account_values', {}).setdefault('browser', {})['allow_javascript_apple_events'] = True
+json.dump(data, open(prefs_path, 'w'))
+print('allow_javascript_apple_events enabled in Default profile')
+"
+
+# 3. Relaunch Chrome and wait for sync to stabilise.
+# Chrome sync fires ~1-2 s after launch and may briefly pull
+# an older value from the server before Chrome pushes our local
+# True back. Either test before sync fires (<1 s) or after it
+# settles (>4 s). Waiting exactly ~2 s lands in the race window
+# and is the most likely way to see a false negative.
+open -a "Google Chrome"
+sleep 5
+
+# 4. Verify
+osascript -e 'tell application "Google Chrome"
+ tell active tab of front window
+ execute javascript "1+1"
+ end tell
+end tell'
+# → 2
+```
+
+**Which profile?** Chrome writes to whichever profile is active.
+If you're unsure, write to all non-system profiles:
+
+```bash
+python3 -c "
+import json, glob, os
+for p in glob.glob(os.path.expanduser(
+ '~/Library/Application Support/Google/Chrome/*/Preferences')):
+ profile = p.split('/')[-2]
+ if 'System' in profile or 'Guest' in profile:
+ continue
+ try:
+ data = json.load(open(p))
+ data.setdefault('browser', {})['allow_javascript_apple_events'] = True
+ data.setdefault('account_values', {}).setdefault('browser', {})['allow_javascript_apple_events'] = True
+ json.dump(data, open(p, 'w'))
+ print(f'wrote to {profile}')
+ except Exception as e:
+ print(f'skipped {profile}: {e}')
+"
+```
+
+Chrome overwrites its Preferences file on every clean exit, so the
+write must happen while Chrome is **not running** — otherwise Chrome
+will stomp the change when it quits.
+
+**Sync note (Chrome only):** Chrome syncs `browser.allow_javascript_apple_events`
+via Google account (confirmed in `chrome_syncable_prefs_database.cc`).
+Writing both `browser` and `account_values.browser` to the local file
+causes Chrome to push `true` to the sync server on next launch,
+making the change durable. Brave and Edge use their own sync systems
+and likely do NOT sync this Mac-only pref — treat as local-only for
+those browsers.
+
+### Brave
+
+Same Chromium pref key, different profile directory:
+
+```bash
+osascript -e 'quit app "Brave Browser"' && sleep 1
+python3 -c "
+import json, glob, os
+for p in glob.glob(os.path.expanduser(
+ '~/Library/Application Support/BraveSoftware/Brave-Browser/*/Preferences')):
+ profile = p.split('/')[-2]
+ if 'System' in profile or 'Guest' in profile:
+ continue
+ try:
+ data = json.load(open(p))
+ data.setdefault('browser', {})['allow_javascript_apple_events'] = True
+ json.dump(data, open(p, 'w'))
+ print(f'wrote to {profile}')
+ except Exception as e:
+ print(f'skipped {profile}: {e}')
+"
+open -a "Brave Browser" && sleep 5
+```
+
+### Edge
+
+Same Chromium pref key, different profile directory:
+
+```bash
+osascript -e 'quit app "Microsoft Edge"' && sleep 1
+python3 -c "
+import json, glob, os
+for p in glob.glob(os.path.expanduser(
+ '~/Library/Application Support/Microsoft Edge/*/Preferences')):
+ profile = p.split('/')[-2]
+ if 'System' in profile or 'Guest' in profile:
+ continue
+ try:
+ data = json.load(open(p))
+ data.setdefault('browser', {})['allow_javascript_apple_events'] = True
+ json.dump(data, open(p, 'w'))
+ print(f'wrote to {profile}')
+ except Exception as e:
+ print(f'skipped {profile}: {e}')
+"
+open -a "Microsoft Edge" && sleep 5
+```
+
+### Safari
+
+Safari uses `do JavaScript "..." in document 1` (different AppleScript
+verb from Chrome's `execute javascript`). The setting is under
+`Develop → Allow JavaScript from Apple Events` (requires the Develop
+menu to be enabled first via Settings → Advanced → "Show features for
+web developers").
+
+`defaults write -app Safari AllowJavaScriptFromAppleEvents 1` **does
+not work** — Safari ignores the defaults key and routes this toggle
+through macOS's security framework, which shows a password-confirmation
+dialog. The only working programmatic path is UI automation:
+
+```applescript
+-- Requires Accessibility permission for the calling process.
+-- Safari must already be running with the Develop menu visible.
+tell application "Safari" to activate
+delay 0.3
+tell application "System Events"
+ tell process "Safari"
+ click menu item "Allow JavaScript from Apple Events" of menu 1 ¬
+ of menu bar item "Develop" of menu bar 1
+ delay 0.3
+ -- Safari shows a confirmation dialog — click Allow
+ click button "Allow" of window 1
+ end tell
+end tell
+```
+
+### Arc
+
+Arc has an AppleScript dictionary and accepts `execute javascript`, but
+**never returns a value** — the call always returns `missing value`.
+There is no "Allow JavaScript from Apple Events" toggle. The only
+workaround is to write results to the clipboard inside the JS and read
+it back:
+
+```applescript
+tell application "Arc"
+ tell active tab of front window
+ execute javascript "navigator.clipboard.writeText(document.title)"
+ end tell
+end tell
+delay 0.3
+set theTitle to the clipboard
+```
+
+Arc's AppleScript JS support is confirmed broken for return values
+(as of 2025). If you need JS results from Arc, use a WebExtension
+with Native Messaging instead.
+
+### Firefox
+
+Firefox has no `execute javascript` capability. Bugzilla #287447
+(filed 2004) tracks this and remains unresolved. Use WebDriver /
+Playwright / a WebExtension with Native Messaging for Firefox.
+
## Typing into a web input
```
diff --git a/libs/cua-driver/Sources/CuaDriverCore/Browser/BrowserJS.swift b/libs/cua-driver/Sources/CuaDriverCore/Browser/BrowserJS.swift
new file mode 100644
index 0000000000..6daff59ebb
--- /dev/null
+++ b/libs/cua-driver/Sources/CuaDriverCore/Browser/BrowserJS.swift
@@ -0,0 +1,263 @@
+import CoreGraphics
+import Foundation
+import os
+
+/// Execute JavaScript in a browser tab via Apple Events.
+///
+/// Supports the Chromium family (Chrome, Brave, Edge) which share the same
+/// `execute javascript` AppleScript verb, and Safari which uses `do JavaScript`.
+/// Firefox and Arc are not supported — see WEB_APPS.md for the full matrix.
+///
+/// **Prerequisite:** "Allow JavaScript from Apple Events" must be enabled in
+/// the target browser. See WEB_APPS.md → "Enable Allow JavaScript from Apple
+/// Events" for the programmatic setup path.
+public enum BrowserJS {
+
+ // MARK: - Public API
+
+ public enum Error: Swift.Error, CustomStringConvertible {
+ case unsupportedBrowser(String)
+ case windowNotFound(UInt32)
+ case javascriptNotEnabled(String)
+ case executionFailed(String)
+
+ public var description: String {
+ switch self {
+ case .unsupportedBrowser(let id):
+ return "Browser '\(id)' does not support JavaScript via Apple Events. "
+ + "Supported: Chrome (com.google.Chrome), Brave (com.brave.Browser), "
+ + "Edge (com.microsoft.edgemac), Safari (com.apple.Safari)."
+ case .windowNotFound(let wid):
+ return "Could not find a browser window matching window_id \(wid). "
+ + "Ensure the browser is running and the window is on the current Space."
+ case .javascriptNotEnabled(let appName):
+ return "'\(appName)' rejected the JavaScript execution — "
+ + "'Allow JavaScript from Apple Events' is not enabled. "
+ + "See WEB_APPS.md → 'Enable Allow JavaScript from Apple Events' "
+ + "for the programmatic setup path (quit browser → edit Preferences JSON → relaunch)."
+ case .executionFailed(let detail):
+ return "JavaScript execution failed: \(detail)"
+ }
+ }
+ }
+
+ /// Run `javascript` in the active tab of the browser window identified by
+ /// `windowId` (CGWindowID) and return the result as a string.
+ public static func execute(
+ javascript: String,
+ bundleId: String,
+ windowId: UInt32
+ ) async throws -> String {
+ guard let spec = browserSpec(for: bundleId) else {
+ throw Error.unsupportedBrowser(bundleId)
+ }
+
+ // Find the CGWindow entry for this window_id to get its title,
+ // which we use to locate the matching AppleScript window.
+ guard let windowTitle = cgWindowTitle(for: windowId) else {
+ throw Error.windowNotFound(windowId)
+ }
+
+ let script = spec.buildScript(javascript, windowTitle)
+ return try await runAppleScript(script, appName: spec.appName)
+ }
+
+ /// True when this bundle ID is handled via Apple Events (Chrome/Brave/Edge/Safari).
+ public static func supports(bundleId: String) -> Bool {
+ browserSpec(for: bundleId) != nil
+ }
+
+ // MARK: - Browser specs
+
+ private struct BrowserSpec {
+ let appName: String
+ let buildScript: (_ javascript: String, _ windowTitle: String) -> String
+ }
+
+ private static func browserSpec(for bundleId: String) -> BrowserSpec? {
+ switch bundleId {
+ case "com.google.Chrome":
+ return chromiumSpec(appName: "Google Chrome")
+ case "com.brave.Browser":
+ return chromiumSpec(appName: "Brave Browser")
+ case "com.microsoft.edgemac":
+ return chromiumSpec(appName: "Microsoft Edge")
+ case "com.apple.Safari":
+ return safariSpec()
+ default:
+ return nil
+ }
+ }
+
+ private static func chromiumSpec(appName: String) -> BrowserSpec {
+ BrowserSpec(appName: appName) { javascript, windowTitle in
+ // Escape the window title for AppleScript string comparison.
+ // We use `whose name contains` to handle truncated titles.
+ let escapedTitle = windowTitle
+ .replacingOccurrences(of: "\\", with: "\\\\")
+ .replacingOccurrences(of: "\"", with: "\\\"")
+ return """
+ tell application "\(appName)"
+ set matchedWindow to missing value
+ repeat with w in windows
+ if name of w contains "\(escapedTitle)" then
+ set matchedWindow to w
+ exit repeat
+ end if
+ end repeat
+ if matchedWindow is missing value then
+ set matchedWindow to front window
+ end if
+ tell active tab of matchedWindow
+ execute javascript \(appleScriptString(javascript))
+ end tell
+ end tell
+ """
+ }
+ }
+
+ private static func safariSpec() -> BrowserSpec {
+ BrowserSpec(appName: "Safari") { javascript, windowTitle in
+ let escapedTitle = windowTitle
+ .replacingOccurrences(of: "\\", with: "\\\\")
+ .replacingOccurrences(of: "\"", with: "\\\"")
+ return """
+ tell application "Safari"
+ set matchedDoc to missing value
+ repeat with d in documents
+ if name of d contains "\(escapedTitle)" then
+ set matchedDoc to d
+ exit repeat
+ end if
+ end repeat
+ if matchedDoc is missing value then
+ set matchedDoc to document 1
+ end if
+ do JavaScript \(appleScriptString(javascript)) in matchedDoc
+ end tell
+ """
+ }
+ }
+
+ // MARK: - AppleScript runner
+
+ private static func runAppleScript(
+ _ script: String,
+ appName: String
+ ) async throws -> String {
+ // Write to a temp file to avoid shell-quoting the script entirely.
+ let tmpURL = URL(fileURLWithPath: NSTemporaryDirectory())
+ .appendingPathComponent(UUID().uuidString + ".applescript")
+ try script.write(to: tmpURL, atomically: true, encoding: .utf8)
+ defer { try? FileManager.default.removeItem(at: tmpURL) }
+
+ return try await withCheckedThrowingContinuation { continuation in
+ let proc = Process()
+ proc.executableURL = URL(fileURLWithPath: "/usr/bin/osascript")
+ proc.arguments = [tmpURL.path]
+ let stdout = Pipe()
+ let stderr = Pipe()
+ proc.standardOutput = stdout
+ proc.standardError = stderr
+
+ let once = OSAllocatedUnfairLock(initialState: false)
+ let resumeOnce: @Sendable (Result<String, Swift.Error>) -> Void = { result in
+ let shouldResume = once.withLock { (fired: inout Bool) -> Bool in
+ guard !fired else { return false }
+ fired = true; return true
+ }
+ guard shouldResume else { return }
+ switch result {
+ case .success(let s): continuation.resume(returning: s)
+ case .failure(let e): continuation.resume(throwing: e)
+ }
+ }
+
+ let timeoutItem = DispatchWorkItem {
+ proc.terminate()
+ resumeOnce(.failure(Error.executionFailed(
+ "osascript timed out after 15 s — browser may be showing a permission dialog")))
+ }
+ DispatchQueue.global().asyncAfter(deadline: .now() + 15, execute: timeoutItem)
+
+ proc.terminationHandler = { p in
+ timeoutItem.cancel()
+ let out = String(
+ data: stdout.fileHandleForReading.readDataToEndOfFile(),
+ encoding: .utf8
+ )?.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
+ let err = String(
+ data: stderr.fileHandleForReading.readDataToEndOfFile(),
+ encoding: .utf8
+ )?.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
+ if p.terminationStatus == 0 {
+ resumeOnce(.success(out))
+ } else if err.contains("turned off") || err.contains("AppleScript is turned off") {
+ resumeOnce(.failure(Error.javascriptNotEnabled(appName)))
+ } else {
+ resumeOnce(.failure(Error.executionFailed(err.isEmpty ? out : err)))
+ }
+ }
+ do {
+ try proc.run()
+ } catch {
+ timeoutItem.cancel()
+ resumeOnce(.failure(error))
+ }
+ }
+ }
+
+ // MARK: - CGWindow helpers
+
+ private static func cgWindowTitle(for windowId: UInt32) -> String? {
+ // Enumerate all on-screen windows and find the one with matching ID.
+ // CGWindowListCopyWindowInfo([.optionIncludingWindow], id) is unreliable
+ // for windows not yet in the compositing list — enumerating all is safer.
+ //
+ // Return "" (empty string) when the window exists but has no title
+ // (new tab, about:blank, freshly-opened window). The AppleScript
+ // `name contains ""` expression matches every string, so an empty
+ // title falls through to the `front window` fallback — correct behavior.
+ // Only return nil when the window ID is genuinely not in any list.
+ let list = CGWindowListCopyWindowInfo(
+ [.optionOnScreenOnly, .excludeDesktopElements], kCGNullWindowID
+ ) as? [[String: Any]] ?? []
+ for entry in list {
+ guard let id = entry[kCGWindowNumber as String] as? Int,
+ UInt32(id) == windowId else { continue }
+ return entry[kCGWindowName as String] as? String ?? ""
+ }
+ // Fall back to off-screen windows (minimized, hidden).
+ let all = CGWindowListCopyWindowInfo([], kCGNullWindowID) as? [[String: Any]] ?? []
+ for entry in all {
+ guard let id = entry[kCGWindowNumber as String] as? Int,
+ UInt32(id) == windowId else { continue }
+ return entry[kCGWindowName as String] as? String ?? ""
+ }
+ return nil
+ }
+
+ // MARK: - AppleScript string literal builder
+
+ /// Wraps `value` in a multi-line AppleScript string literal using
+ /// quoted form syntax to avoid any quoting/escaping issues with the
+ /// JavaScript payload.
+ private static func appleScriptString(_ value: String) -> String {
+ // AppleScript doesn't support heredocs; we use string concatenation
+ // for newlines and escape only the characters AppleScript requires.
+ if !value.contains("\n") && !value.contains("\"") && !value.contains("\\") {
+ return "\"\(value)\""
+ }
+ // Build the string by concatenating segments split on newlines,
+ // joined with (ASCII character 10) for newlines.
+ let lines = value.split(separator: "\n", omittingEmptySubsequences: false)
+ let escaped = lines.map { line -> String in
+ let s = String(line)
+ .replacingOccurrences(of: "\\", with: "\\\\")
+ .replacingOccurrences(of: "\"", with: "\\\"")
+ return "\"\(s)\""
+ }
+ if escaped.count == 1 { return escaped[0] }
+ return escaped.joined(separator: " & (ASCII character 10) & ")
+ }
+}
diff --git a/libs/cua-driver/Sources/CuaDriverCore/Browser/ElectronJS.swift b/libs/cua-driver/Sources/CuaDriverCore/Browser/ElectronJS.swift
new file mode 100644
index 0000000000..9657196a33
--- /dev/null
+++ b/libs/cua-driver/Sources/CuaDriverCore/Browser/ElectronJS.swift
@@ -0,0 +1,302 @@
+import AppKit
+import Foundation
+
+/// Execute JavaScript in a running Electron app's renderer via the V8 inspector
+/// (SIGUSR1 → CDP WebSocket → Runtime.evaluate).
+///
+/// **How it works:**
+/// 1. Send SIGUSR1 to the Electron main process — Node activates the V8 inspector
+/// on a random localhost port (same as `node --inspect`).
+/// 2. Discover the new port by diffing the pid's TCP LISTEN sockets before/after.
+/// 3. GET `http://localhost:{port}/json` for the CDP WebSocket URL.
+/// 4. Open a WebSocket, send `Runtime.evaluate`, return the result.
+///
+/// **Requirement:** The app must have the `EnableNodeCliInspectArguments` Electron
+/// fuse enabled (the default in development builds; some production apps disable it
+/// post-CVE-2024-23738). Check with:
+/// `npx @electron/fuses read --app /Applications/MyApp.app`
+///
+/// VS Code and Cursor both have this fuse ON. Slack/Discord likely have it OFF.
+public enum ElectronJS {
+
+ // MARK: - Public API
+
+ public enum Error: Swift.Error, CustomStringConvertible {
+ case notElectron(String)
+ case inspectorNotAvailable(String)
+ case cdpConnectionFailed(String)
+ case evaluationFailed(String)
+
+ public var description: String {
+ switch self {
+ case .notElectron(let detail):
+ return "Not an Electron app: \(detail). "
+ + "Use the `page` tool only with Electron apps "
+ + "(bundle must contain Electron Framework.framework)."
+ case .inspectorNotAvailable(let detail):
+ return "V8 inspector did not activate after SIGUSR1: \(detail)\n"
+ + "The app may have the EnableNodeCliInspectArguments fuse disabled. "
+ + "Verify with: npx @electron/fuses read --app /path/to/App.app"
+ case .cdpConnectionFailed(let detail):
+ return "CDP connection failed: \(detail)"
+ case .evaluationFailed(let detail):
+ return "JavaScript evaluation failed: \(detail)"
+ }
+ }
+ }
+
+ /// True if the app running at `pid` is Electron-based — detected by the
+ /// presence of `Electron Framework.framework` inside its bundle.
+ public static func isElectron(pid: Int32) -> Bool {
+ guard let app = NSWorkspace.shared.runningApplications
+ .first(where: { $0.processIdentifier == pid }),
+ let bundleURL = app.bundleURL else { return false }
+ let frameworkPath = bundleURL
+ .appendingPathComponent("Contents/Frameworks/Electron Framework.framework")
+ return FileManager.default.fileExists(atPath: frameworkPath.path)
+ }
+
+ /// Execute `javascript` in the Electron app running at `pid`.
+ /// Returns the result serialised as a string.
+ ///
+ /// **Context available depends on how the inspector was activated:**
+ ///
+ /// - `--remote-debugging-port=N` at launch → prefers "page" targets (full DOM,
+ /// `window`, `document`, renderer globals). Best for UI automation.
+ /// - SIGUSR1 (no relaunch) → activates the Node.js main process inspector.
+ /// `process`, `Buffer`, `console` available; `require`, `document`, and
+ /// Electron APIs are NOT available in sandboxed apps (VS Code, Cursor).
+ /// Useful for process-level reads: `process.env`, `process.versions`,
+ /// `process.cwd()`, `process.pid`.
+ public static func execute(javascript: String, pid: Int32) async throws -> String {
+ guard isElectron(pid: pid) else {
+ let name = NSWorkspace.shared.runningApplications
+ .first(where: { $0.processIdentifier == pid })?.localizedName ?? "pid \(pid)"
+ throw Error.notElectron(name)
+ }
+
+ // Prefer a "page" target (renderer with DOM) if the app was launched with
+ // --remote-debugging-port. Page targets expose document/window/DOM APIs.
+ if let pagePort = await pageTarget() {
+ return try await cdpEvaluate(port: pagePort, javascript: javascript)
+ }
+
+ // If the inspector is already active (from a prior call in this session),
+ // skip SIGUSR1 and reuse the port directly.
+ if let existingPort = await activeInspectorPort(pid: pid) {
+ return try await cdpEvaluate(port: existingPort, javascript: javascript)
+ }
+
+ // Snapshot TCP LISTEN ports before signalling so we can diff afterwards.
+ let portsBefore = await listeningPorts(pid: pid)
+
+ // Activate the V8 inspector — same mechanism as `node --inspect`.
+ kill(pid, SIGUSR1)
+
+ // Poll up to 2 s for a new LISTEN port to appear.
+ var inspectorPort: Int?
+ for _ in 0..<10 {
+ try await Task.sleep(for: .milliseconds(200))
+ let portsAfter = await listeningPorts(pid: pid)
+ let newPorts = portsAfter.subtracting(portsBefore)
+ for port in newPorts {
+ if await isInspectorPort(port) {
+ inspectorPort = port
+ break
+ }
+ }
+ if inspectorPort != nil { break }
+
+ // Fallback: scan the common Node inspector range in case lsof
+ // didn't resolve the pid association in time.
+ if let port = await scanInspectorPorts() {
+ inspectorPort = port
+ break
+ }
+ }
+
+ guard let port = inspectorPort else {
+ throw Error.inspectorNotAvailable(
+ "no CDP endpoint appeared on localhost within 2 s after SIGUSR1"
+ )
+ }
+
+ return try await cdpEvaluate(port: port, javascript: javascript)
+ }
+
+ // MARK: - Port discovery
+
+ /// Scan common remote-debugging ports for a CDP "page" target (renderer with
+ /// DOM access). Returns the port if found. Apps launched with
+ /// `--remote-debugging-port=N` expose page targets here; apps activated via
+ /// SIGUSR1 only expose a "node" main-process target with no DOM.
+ private static func pageTarget() async -> Int? {
+ for port in [9222, 9223, 9224, 9225, 9230] {
+ guard let url = URL(string: "http://127.0.0.1:\(port)/json") else { continue }
+ var req = URLRequest(url: url); req.timeoutInterval = 0.3
+ guard let (data, _) = try? await URLSession.shared.data(for: req),
+ let targets = try? JSONSerialization.jsonObject(with: data) as? [[String: Any]],
+ targets.contains(where: { ($0["type"] as? String) == "page" })
+ else { continue }
+ return port
+ }
+ return nil
+ }
+
+ /// Check if the pid already has an active V8 inspector port.
+ private static func activeInspectorPort(pid: Int32) async -> Int? {
+ let ports = await listeningPorts(pid: pid)
+ for port in ports {
+ if await isInspectorPort(port) { return port }
+ }
+ return nil
+ }
+
+ /// TCP ports the pid is currently listening on, via `lsof -p <pid> -iTCP:LISTEN`.
+ private static func listeningPorts(pid: Int32) async -> Set<Int> {
+ let output = await runProcess(
+ "/usr/sbin/lsof",
+ args: ["-p", "\(pid)", "-iTCP", "-sTCP:LISTEN", "-Fn", "-P"]
+ )
+ var ports = Set<Int>()
+ for line in output.split(separator: "\n") {
+ // lsof -Fn lines: "n*:9229" or "n127.0.0.1:9229"
+ guard line.hasPrefix("n"), let colon = line.lastIndex(of: ":") else { continue }
+ let portStr = String(line[line.index(after: colon)...])
+ if let port = Int(portStr) { ports.insert(port) }
+ }
+ return ports
+ }
+
+ /// Returns true if `port` responds to the CDP `/json` HTTP endpoint.
+ private static func isInspectorPort(_ port: Int) async -> Bool {
+ guard let url = URL(string: "http://127.0.0.1:\(port)/json") else { return false }
+ var req = URLRequest(url: url)
+ req.timeoutInterval = 0.5
+ do {
+ let (_, resp) = try await URLSession.shared.data(for: req)
+ return (resp as? HTTPURLResponse)?.statusCode == 200
+ } catch { return false }
+ }
+
+ /// Scan the standard Node inspector port range (9229-9249) for an active endpoint.
+ private static func scanInspectorPorts() async -> Int? {
+ for port in 9229...9249 {
+ if await isInspectorPort(port) { return port }
+ }
+ return nil
+ }
+
+ // MARK: - CDP evaluation
+
+ private static func cdpEvaluate(port: Int, javascript: String) async throws -> String {
+ // 1. Fetch the list of debuggable targets.
+ guard let jsonURL = URL(string: "http://127.0.0.1:\(port)/json") else {
+ throw Error.cdpConnectionFailed("bad port \(port)")
+ }
+ var req = URLRequest(url: jsonURL)
+ req.timeoutInterval = 5
+ let (data, _) = try await URLSession.shared.data(for: req)
+
+ guard let targets = try? JSONSerialization.jsonObject(with: data) as? [[String: Any]] else {
+ throw Error.cdpConnectionFailed("could not parse /json response from port \(port)")
+ }
+ // Prefer the page target; fall back to the first target with a WS URL.
+ let target = targets.first(where: { ($0["type"] as? String) == "page" })
+ ?? targets.first(where: { $0["webSocketDebuggerUrl"] != nil })
+ guard let wsURLStr = target?["webSocketDebuggerUrl"] as? String,
+ let wsURL = URL(string: wsURLStr) else {
+ throw Error.cdpConnectionFailed("no debuggable target found at port \(port)")
+ }
+
+ // 2. Send Runtime.evaluate over WebSocket.
+ let payload: [String: Any] = [
+ "id": 1,
+ "method": "Runtime.evaluate",
+ "params": [
+ "expression": javascript,
+ "returnByValue": true,
+ "awaitPromise": true,
+ ],
+ ]
+ let payloadStr = String(
+ data: try JSONSerialization.data(withJSONObject: payload),
+ encoding: .utf8
+ )!
+
+ return try await withCheckedThrowingContinuation { continuation in
+ let ws = URLSession.shared.webSocketTask(with: wsURL)
+ ws.resume()
+ ws.send(.string(payloadStr)) { sendError in
+ if let err = sendError {
+ ws.cancel()
+ continuation.resume(throwing: Error.cdpConnectionFailed(err.localizedDescription))
+ return
+ }
+ ws.receive { result in
+ ws.cancel()
+ switch result {
+ case .failure(let err):
+ continuation.resume(throwing: Error.cdpConnectionFailed(err.localizedDescription))
+ case .success(let message):
+ guard case .string(let str) = message else {
+ continuation.resume(throwing: Error.cdpConnectionFailed("binary WebSocket frame"))
+ return
+ }
+ continuation.resume(with: Result { try Self.parseCDPResult(str) })
+ }
+ }
+ }
+ }
+ }
+
+ /// Extract the return value from a `Runtime.evaluate` CDP response.
+ private static func parseCDPResult(_ json: String) throws -> String {
+ guard let data = json.data(using: .utf8),
+ let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any] else {
+ return json
+ }
+ if let error = obj["error"] as? [String: Any] {
+ let msg = error["message"] as? String ?? json
+ throw Error.evaluationFailed(msg)
+ }
+ if let result = obj["result"] as? [String: Any],
+ let inner = result["result"] as? [String: Any] {
+ // Exception thrown inside JS
+ if let exDesc = (result["exceptionDetails"] as? [String: Any])?["text"] as? String {
+ throw Error.evaluationFailed(exDesc)
+ }
+ // Serialise the return value
+ if let value = inner["value"] {
+ if let str = value as? String { return str }
+ if let num = value as? NSNumber { return num.stringValue }
+ if let data = try? JSONSerialization.data(withJSONObject: value),
+ let str = String(data: data, encoding: .utf8) { return str }
+ return "\(value)"
+ }
+ return inner["description"] as? String ?? "undefined"
+ }
+ return json
+ }
+
+ // MARK: - Subprocess helper
+
+ private static func runProcess(_ executable: String, args: [String]) async -> String {
+ await withCheckedContinuation { continuation in
+ let proc = Process()
+ proc.executableURL = URL(fileURLWithPath: executable)
+ proc.arguments = args
+ let pipe = Pipe()
+ proc.standardOutput = pipe
+ proc.standardError = Pipe()
+ proc.terminationHandler = { _ in
+ let out = String(
+ data: pipe.fileHandleForReading.readDataToEndOfFile(),
+ encoding: .utf8
+ ) ?? ""
+ continuation.resume(returning: out)
+ }
+ do { try proc.run() } catch { continuation.resume(returning: "") }
+ }
+ }
+}
diff --git a/libs/cua-driver/Sources/CuaDriverCore/Input/AXInput.swift b/libs/cua-driver/Sources/CuaDriverCore/Input/AXInput.swift
index 4dc764ba7f..e552bbcb0c 100644
--- a/libs/cua-driver/Sources/CuaDriverCore/Input/AXInput.swift
+++ b/libs/cua-driver/Sources/CuaDriverCore/Input/AXInput.swift
@@ -166,6 +166,16 @@ public enum AXInput {
return attributeString(element, name)
}
+ /// Read a boolean attribute from an element. Returns nil when the attribute
+ /// is absent or unreadable.
+ public static func boolAttribute(_ name: String, of element: AXUIElement) -> Bool? {
+ var value: CFTypeRef?
+ guard AXUIElementCopyAttributeValue(element, name as CFString, &value) == .success,
+ let v = value else { return nil }
+ guard CFGetTypeID(v) == CFBooleanGetTypeID() else { return nil }
+ return CFBooleanGetValue((v as! CFBoolean))
+ }
+
/// The element's on-screen center in screen-point coordinates
/// (top-left origin). Returns nil when the element has no
/// position / size (menus, offscreen elements, hidden rows).
diff --git a/libs/cua-driver/Sources/CuaDriverServer/ToolRegistry.swift b/libs/cua-driver/Sources/CuaDriverServer/ToolRegistry.swift
index fbab2b4e52..81b3270bcb 100644
--- a/libs/cua-driver/Sources/CuaDriverServer/ToolRegistry.swift
+++ b/libs/cua-driver/Sources/CuaDriverServer/ToolRegistry.swift
@@ -40,6 +40,7 @@ public struct ToolRegistry: Sendable {
"press_key",
"hotkey",
"set_value",
+ "page",
]
/// Click-family tools — their argument set carries a click point we
@@ -239,5 +240,6 @@ public struct ToolRegistry: Sendable {
GetConfigTool.handler,
SetConfigTool.handler,
ZoomTool.handler,
+ PageTool.handler,
])
}
diff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/ClickTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/ClickTool.swift
index 1e6ac10446..e0a9f55f06 100644
--- a/libs/cua-driver/Sources/CuaDriverServer/Tools/ClickTool.swift
+++ b/libs/cua-driver/Sources/CuaDriverServer/Tools/ClickTool.swift
@@ -249,6 +249,23 @@ public enum ClickTool {
// without this we have no way to detect a silent no-op.
// See the post-dispatch warning assembled below.
let advertisedActions = AXInput.advertisedActionNames(of: element)
+ // Bail early if the element is disabled (AXEnabled = false).
+ // AXUIElementPerformAction returns .success even on disabled
+ // elements — the call reaches the app's AX layer but the app's
+ // command-dispatch silently discards it (Chrome marks Developer
+ // submenu items as disabled via commandDispatch when it isn't
+ // frontmost). Surfacing this as an error rather than a silent
+ // no-op lets the caller decide whether to activate the app first.
+ if AXInput.boolAttribute("AXEnabled", of: element) == false {
+ let target = AXInput.describe(element)
+ let appName = NSRunningApplication(processIdentifier: pid)?
+ .localizedName ?? "the app"
+ var msg = "❌ [\(index)] \(target.role ?? "element") \"\(target.title ?? "")\" is disabled (AXEnabled = false) — action would be a silent no-op."
+ msg += "\n\nThis usually means the target app is not frontmost."
+ msg += " Activate it first, then re-snapshot and retry:"
+ msg += "\n osascript -e 'tell application \"\(appName)\" to activate'"
+ return errorResult(msg)
+ }
// Animate the visual agent cursor to the target before
// firing the AX action. `animateAndWait` is a no-op
// when the cursor is disabled (the default) or when
diff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift
index 022807fe36..018d43ebc2 100644
--- a/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift
+++ b/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift
@@ -94,6 +94,19 @@ public enum GetWindowStateTool {
"description":
"Optional case-insensitive substring. When set, `tree_markdown` only contains lines that match plus their ancestor chain; element indices and `element_count` are unchanged.",
],
+ "javascript": [
+ "type": "string",
+ "description": """
+ Optional JavaScript to execute in the browser tab and return \
+ alongside the AX snapshot — one round-trip instead of two. \
+ Only works for Chromium-family browsers (Chrome, Brave, Edge) \
+ and Safari; requires 'Allow JavaScript from Apple Events' to be \
+ enabled first (see WEB_APPS.md). The result is appended to the \
+ response as a `## JavaScript result` section. Use for read-only \
+ queries (document.title, innerText, querySelectorAll, etc.). \
+ For mutations or side effects use the `page` tool instead.
+ """,
+ ],
],
"additionalProperties": false,
],
@@ -123,6 +136,7 @@ public enum GetWindowStateTool {
"window_id \(rawWindowId) is outside the supported UInt32 range.")
}
let query = arguments?["query"]?.stringValue
+ let javascript = arguments?["javascript"]?.stringValue
// Validate that the window belongs to this pid. The driver
// never guesses which window to snapshot — the caller names
@@ -218,6 +232,28 @@ public enum GetWindowStateTool {
if captureMode != .vision && !snapshot.treeMarkdown.isEmpty {
textContent += "\n\n" + snapshot.treeMarkdown
}
+ // If the caller passed a javascript snippet, run it in the
+ // browser tab and append the result — one round-trip instead
+ // of two separate tool calls.
+ if let js = javascript {
+ let bundleId = snapshot.bundleId ?? ""
+ do {
+ let jsResult: String
+ if BrowserJS.supports(bundleId: bundleId) {
+ jsResult = try await BrowserJS.execute(
+ javascript: js, bundleId: bundleId, windowId: windowId)
+ } else if ElectronJS.isElectron(pid: pid) {
+ jsResult = try await ElectronJS.execute(
+ javascript: js, pid: pid)
+ } else {
+ throw BrowserJS.Error.unsupportedBrowser(bundleId)
+ }
+ textContent += "\n\n## JavaScript result\n\n```\n\(jsResult)\n```"
+ } catch {
+ textContent += "\n\n## JavaScript result\n\n❌ \(error)"
+ }
+ }
+
var content: [Tool.Content] = []
if let b64 = snapshot.screenshotPngBase64 {
content.append(
diff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/PageTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/PageTool.swift
new file mode 100644
index 0000000000..e9a737fcb1
--- /dev/null
+++ b/libs/cua-driver/Sources/CuaDriverServer/Tools/PageTool.swift
@@ -0,0 +1,221 @@
+import AppKit
+import CuaDriverCore
+import Foundation
+import MCP
+
+/// Browser page primitives — execute JavaScript, extract page text, query DOM.
+///
+/// Use `get_window_state` with its `javascript` param for **read-only** queries
+/// co-located with a snapshot. Use `page` when you need a standalone JS call
+/// (mutations, navigation side-effects, or results without a full AX walk).
+///
+/// Requires "Allow JavaScript from Apple Events" to be enabled in the target
+/// browser — see WEB_APPS.md for the setup path.
+public enum PageTool {
+
+ public static let handler = ToolHandler(
+ tool: Tool(
+ name: "page",
+ description: """
+ Browser page primitives — execute JavaScript, extract page \
+ text, or query DOM elements via CSS selector.
+
+ Three actions:
+
+ • `execute_javascript` — run arbitrary JS in the active tab \
+ and return the result. Wrap in an IIFE with try-catch for \
+ safety. Don't use this for elements already indexed by \
+ `get_window_state` — prefer `click` / `set_value` there.
+
+ • `get_text` — returns `document.body.innerText`. Faster than \
+ walking the AX tree for plain-text content (prices, article \
+ body, table values) that the AX tree drops or truncates.
+
+ • `query_dom` — runs `querySelectorAll(css_selector)` and \
+ returns each matching element's tag, innerText, and any \
+ requested attributes as a JSON array. Useful for structured \
+ data (table rows, link hrefs, data-* attributes) that the AX \
+ tree flattens.
+
+ Supported browsers: Chrome (com.google.Chrome), Brave \
+ (com.brave.Browser), Edge (com.microsoft.edgemac), Safari \
+ (com.apple.Safari). Requires "Allow JavaScript from Apple \
+ Events" enabled — see WEB_APPS.md.
+
+ `pid` and `window_id` identify the target window. The browser \
+ does NOT need to be frontmost for read actions. For mutations \
+ that trigger UI changes (form submits, navigation) the browser \
+ should be frontmost so the results are visible.
+ """,
+ inputSchema: [
+ "type": "object",
+ "required": ["pid", "window_id", "action"],
+ "properties": [
+ "pid": [
+ "type": "integer",
+ "description": "Browser process ID.",
+ ],
+ "window_id": [
+ "type": "integer",
+ "description": "CGWindowID of the target browser window.",
+ ],
+ "action": [
+ "type": "string",
+ "enum": ["execute_javascript", "get_text", "query_dom"],
+ "description": "Which page primitive to run.",
+ ],
+ "javascript": [
+ "type": "string",
+ "description": "JS to execute (action=execute_javascript only). Should return a serialisable value. Wrap in IIFE: `(() => { … })()`.",
+ ],
+ "css_selector": [
+ "type": "string",
+ "description": "CSS selector (action=query_dom only). Passed to querySelectorAll — standard CSS syntax.",
+ ],
+ "attributes": [
+ "type": "array",
+ "items": ["type": "string"],
+ "description": "Element attributes to include per match (action=query_dom only). E.g. [\"href\", \"src\", \"data-id\"]. tag and innerText are always included.",
+ ],
+ ],
+ "additionalProperties": false,
+ ],
+ annotations: .init(
+ readOnlyHint: false,
+ destructiveHint: false,
+ idempotentHint: false,
+ openWorldHint: false
+ )
+ ),
+ invoke: { arguments in
+ guard let rawPid = arguments?["pid"]?.intValue,
+ let pid = Int32(exactly: rawPid)
+ else {
+ return errorResult("Missing or invalid pid.")
+ }
+ guard let rawWindowId = arguments?["window_id"]?.intValue,
+ let windowId = UInt32(exactly: rawWindowId)
+ else {
+ return errorResult("Missing or invalid window_id.")
+ }
+ guard let action = arguments?["action"]?.stringValue else {
+ return errorResult("Missing required field action.")
+ }
+
+ // Resolve bundle ID from the running app (NSWorkspace requires main thread).
+ let bundleId = await MainActor.run {
+ NSWorkspace.shared.runningApplications
+ .first(where: { $0.processIdentifier == pid })?
+ .bundleIdentifier ?? ""
+ }
+
+ switch action {
+
+ case "execute_javascript":
+ guard let js = arguments?["javascript"]?.stringValue, !js.isEmpty else {
+ return errorResult(
+ "action=execute_javascript requires a non-empty javascript field.")
+ }
+ do {
+ let result = try await executeJS(js, bundleId: bundleId, pid: pid, windowId: windowId)
+ return okResult("## Result\n\n```\n\(result)\n```")
+ } catch {
+ return errorResult("\(error)")
+ }
+
+ case "get_text":
+ do {
+ let result = try await executeJS(
+ "document.body.innerText",
+ bundleId: bundleId, pid: pid, windowId: windowId)
+ return okResult(result)
+ } catch {
+ return errorResult("\(error)")
+ }
+
+ case "query_dom":
+ guard let selector = arguments?["css_selector"]?.stringValue,
+ !selector.isEmpty
+ else {
+ return errorResult(
+ "action=query_dom requires a non-empty css_selector field.")
+ }
+ let attrs: [String]
+ if let rawAttrs = arguments?["attributes"]?.arrayValue {
+ attrs = rawAttrs.compactMap { $0.stringValue }
+ } else {
+ attrs = []
+ }
+ let attrJS = attrs.isEmpty
+ ? "[]"
+ : "[\(attrs.map { "\"\($0)\"" }.joined(separator: ", "))]"
+ let js = """
+ (() => {
+ const attrs = \(attrJS);
+ return JSON.stringify(
+ Array.from(document.querySelectorAll(\(jsonString(selector)))).map(el => {
+ const obj = { tag: el.tagName.toLowerCase(), text: el.innerText?.trim() };
+ for (const a of attrs) obj[a] = el.getAttribute(a);
+ return obj;
+ })
+ );
+ })()
+ """
+ do {
+ let result = try await executeJS(js, bundleId: bundleId, pid: pid, windowId: windowId)
+ return okResult("## DOM query: `\(selector)`\n\n```json\n\(result)\n```")
+ } catch {
+ return errorResult("\(error)")
+ }
+
+ default:
+ return errorResult(
+ "Unknown action '\(action)'. Valid values: execute_javascript, get_text, query_dom.")
+ }
+ }
+ )
+
+ private static func okResult(_ text: String) -> CallTool.Result {
+ CallTool.Result(content: [.text(text: text, annotations: nil, _meta: nil)])
+ }
+
+ private static func errorResult(_ message: String) -> CallTool.Result {
+ CallTool.Result(
+ content: [.text(text: message, annotations: nil, _meta: nil)],
+ isError: true
+ )
+ }
+
+ /// Route JS execution to the right backend:
+ /// 1. Apple Events (BrowserJS) — Chrome, Brave, Edge, Safari by bundle ID.
+ /// 2. Electron CDP (ElectronJS) — any Electron app detected by bundle presence.
+ /// 3. Error — unsupported app type.
+ private static func executeJS(
+ _ javascript: String,
+ bundleId: String,
+ pid: Int32,
+ windowId: UInt32
+ ) async throws -> String {
+ // Apple Events path — covers all Chromium browsers and Safari.
+ if BrowserJS.supports(bundleId: bundleId) {
+ return try await BrowserJS.execute(
+ javascript: javascript, bundleId: bundleId, windowId: windowId)
+ }
+ // Electron CDP path — SIGUSR1 → V8 inspector → Runtime.evaluate.
+ if ElectronJS.isElectron(pid: pid) {
+ return try await ElectronJS.execute(javascript: javascript, pid: pid)
+ }
+ throw BrowserJS.Error.unsupportedBrowser(bundleId)
+ }
+
+ /// JSON-safe string literal for embedding in JS source.
+ private static func jsonString(_ value: String) -> String {
+ let escaped = value
+ .replacingOccurrences(of: "\\", with: "\\\\")
+ .replacingOccurrences(of: "\"", with: "\\\"")
+ .replacingOccurrences(of: "\n", with: "\\n")
+ .replacingOccurrences(of: "\r", with: "\\r")
+ .replacingOccurrences(of: "\t", with: "\\t")
+ return "\"\(escaped)\""
+ }
+}
diff --git a/libs/cua-driver/Tests/integration/test_background_focus.py b/libs/cua-driver/Tests/integration/test_background_focus.py
index 171bd28a8d..dd1674411b 100644
--- a/libs/cua-driver/Tests/integration/test_background_focus.py
+++ b/libs/cua-driver/Tests/integration/test_background_focus.py
@@ -52,6 +52,13 @@ FOCUS_MONITOR_BUNDLE = "com.trycua.FocusMonitorApp"
# Helpers
# ---------------------------------------------------------------------------
+def _tool_text(result: dict) -> str:
+ """Extract text from a tool call result's content array."""
+ for item in result.get("content", []):
+ if item.get("type") == "text":
+ return item.get("text", "")
+ return ""
+
def _build_focus_app() -> None:
if not os.path.exists(_FOCUS_APP_EXE):
subprocess.run(
@@ -86,17 +93,23 @@ def _read_focus_losses() -> int:
def _open_safari_to_html(client: DriverClient) -> int:
- """Open the test HTML page in Safari and return its pid."""
- result = client.call_tool("launch_app", {"bundle_id": SAFARI_BUNDLE})
- safari_pid = result["structuredContent"]["pid"]
+ """Open the test HTML page in Safari (backgrounded) and return its pid.
+ Uses launch_app with urls so Safari never becomes frontmost — the
+ driver's FocusRestoreGuard clobbers any activate Safari fires during
+ document load back to whatever was frontmost before the call.
+ """
file_url = f"file://{_HTML_PAGE}"
- subprocess.run(
- ["osascript", "-e", f'tell application "Safari" to open location "{file_url}"'],
- check=True,
- )
+ result = client.call_tool("launch_app", {
+ "bundle_id": SAFARI_BUNDLE,
+ "urls": [file_url],
+ })
+ text = _tool_text(result)
+ m = re.search(r'pid[=:\s]+(\d+)', text, re.IGNORECASE)
+ if not m:
+ raise RuntimeError(f"launch_app did not return a pid: {text[:200]}")
time.sleep(2.0) # let page load
- return safari_pid
+ return int(m.group(1))
def _get_page_text(client: DriverClient, pid: int) -> str:
diff --git a/libs/cua-driver/Tests/integration/test_browser_js.py b/libs/cua-driver/Tests/integration/test_browser_js.py
new file mode 100644
index 0000000000..528469894e
--- /dev/null
+++ b/libs/cua-driver/Tests/integration/test_browser_js.py
@@ -0,0 +1,336 @@
+"""Integration tests: browser JS primitives (page tool + get_window_state javascript param).
+
+Tests the three new browser primitives:
+ - page(action=get_text) — document.body.innerText
+ - page(action=query_dom) — querySelectorAll as JSON
+ - page(action=execute_javascript)— raw JS with return value
+ - get_window_state(javascript=…) — JS result alongside AX snapshot
+
+Setup:
+ Requires Chrome to be running with "Allow JavaScript from Apple Events"
+ enabled. The test enables it automatically using the Preferences JSON path
+ documented in WEB_APPS.md (quits Chrome, writes pref, relaunches).
+
+Run:
+ scripts/test.sh test_browser_js
+"""
+
+from __future__ import annotations
+
+import glob
+import json
+import os
+import re
+import subprocess
+import sys
+import time
+import unittest
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+from driver_client import DriverClient, default_binary_path
+
+CHROME_BUNDLE = "com.google.Chrome"
+# Simple, stable public page with predictable DOM: one <h1>, one <a>.
+TEST_URL = "https://example.com"
+# Expected content from example.com.
+EXPECTED_H1 = "Example Domain"
+EXPECTED_LINK_TEXT = "Learn more"
+EXPECTED_HREF_FRAGMENT = "iana.org/domains/example"
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+def _tool_text(result: dict) -> str:
+ """Extract the text string from a tool call result."""
+ for item in result.get("content", []):
+ if item.get("type") == "text":
+ return item.get("text", "")
+ return ""
+
+
+def _enable_chrome_apple_events() -> None:
+ """Quit Chrome, write allow_javascript_apple_events to all profiles, relaunch."""
+ subprocess.run(
+ ["osascript", "-e", 'quit app "Google Chrome"'],
+ check=False, timeout=10,
+ )
+ time.sleep(1.5)
+
+ for prefs_path in glob.glob(
+ os.path.expanduser(
+ "~/Library/Application Support/Google/Chrome/*/Preferences"
+ )
+ ):
+ profile = prefs_path.split("/")[-2]
+ if "System" in profile or "Guest" in profile:
+ continue
+ try:
+ with open(prefs_path) as f:
+ data = json.load(f)
+ data.setdefault("browser", {})["allow_javascript_apple_events"] = True
+ data.setdefault("account_values", {}).setdefault("browser", {})[
+ "allow_javascript_apple_events"
+ ] = True
+ with open(prefs_path, "w") as f:
+ json.dump(data, f)
+ except Exception as e:
+ print(f" skipped {profile}: {e}")
+
+ subprocess.run(["open", "-a", "Google Chrome"], check=True)
+ # 0.8s: test before sync fires so local True is still in effect.
+ time.sleep(0.8)
+
+
+def _find_chrome(client: DriverClient) -> int:
+ """Return Chrome's pid from list_apps text output."""
+ text = _tool_text(client.call_tool("list_apps"))
+ m = re.search(r"Google Chrome \(pid (\d+)\)", text)
+ if not m:
+ raise RuntimeError("Chrome not found in list_apps output")
+ return int(m.group(1))
+
+
+def _main_window(client: DriverClient, pid: int) -> int:
+ """Return window_id of the Chrome window that contains example.com."""
+ text = _tool_text(client.call_tool("list_windows", {"pid": pid}))
+ # Lines look like: - Google Chrome (pid N) "Title" [window_id: NNNN]
+ # Prefer a window whose title contains "Example Domain".
+ for line in text.splitlines():
+ if "Example Domain" in line or "example.com" in line.lower():
+ m = re.search(r'\[window_id:\s*(\d+)\]', line)
+ if m:
+ return int(m.group(1))
+ # Fall back to first titled window.
+ titled = re.findall(r'"[^"]+"\s+\[window_id:\s*(\d+)\]', text)
+ if titled:
+ return int(titled[0])
+ m = re.search(r'\[window_id:\s*(\d+)\]', text)
+ if m:
+ return int(m.group(1))
+ raise RuntimeError(f"No window found for Chrome pid {pid}")
+
+
+# ---------------------------------------------------------------------------
+# Test class
+# ---------------------------------------------------------------------------
+
+class BrowserJSTests(unittest.TestCase):
+ """Browser JS primitives work end-to-end against a live Chrome window."""
+
+ _chrome_pid: int
+ _window_id: int
+
+ @classmethod
+ def setUpClass(cls) -> None:
+ cls.binary = default_binary_path()
+
+ _enable_chrome_apple_events()
+
+ # launch_app opens the URL in a new window WITHOUT stealing focus —
+ # the driver's FocusRestoreGuard catches Chrome's activate call.
+ with DriverClient(cls.binary) as c:
+ result = c.call_tool("launch_app", {
+ "bundle_id": CHROME_BUNDLE,
+ "urls": [TEST_URL],
+ })
+ text = _tool_text(result)
+ # Extract pid from launch_app response.
+ m = re.search(r'pid[=:\s]+(\d+)', text, re.IGNORECASE)
+ cls._chrome_pid = int(m.group(1)) if m else _find_chrome(c)
+
+ # Wait for Chrome to load the page and update the window title.
+ # Retry up to 8 s — network load time varies.
+ deadline = time.monotonic() + 8.0
+ cls._window_id = None
+ while time.monotonic() < deadline:
+ with DriverClient(cls.binary) as c:
+ try:
+ cls._window_id = _main_window(c, cls._chrome_pid)
+ break
+ except RuntimeError:
+ pass
+ time.sleep(0.5)
+
+ if cls._window_id is None:
+ raise RuntimeError(
+ f"example.com window never appeared for Chrome pid {cls._chrome_pid}"
+ )
+
+ print(f"\n Chrome pid={cls._chrome_pid} window_id={cls._window_id}")
+
+ # -----------------------------------------------------------------------
+ # page action=execute_javascript
+ # -----------------------------------------------------------------------
+
+ def test_01_execute_javascript_arithmetic(self) -> None:
+ """execute_javascript returns the JS evaluation result."""
+ with DriverClient(self.binary) as c:
+ result = c.call_tool("page", {
+ "pid": self._chrome_pid,
+ "window_id": self._window_id,
+ "action": "execute_javascript",
+ "javascript": "1 + 1",
+ })
+ self.assertFalse(result.get("isError"))
+ self.assertIn("2", _tool_text(result))
+
+ def test_02_execute_javascript_dom_read(self) -> None:
+ """execute_javascript can read DOM content."""
+ with DriverClient(self.binary) as c:
+ result = c.call_tool("page", {
+ "pid": self._chrome_pid,
+ "window_id": self._window_id,
+ "action": "execute_javascript",
+ "javascript": "document.querySelector('h1').innerText",
+ })
+ self.assertFalse(result.get("isError"))
+ self.assertIn(EXPECTED_H1, _tool_text(result))
+
+ def test_03_execute_javascript_iife(self) -> None:
+ """execute_javascript handles IIFE with try-catch."""
+ with DriverClient(self.binary) as c:
+ result = c.call_tool("page", {
+ "pid": self._chrome_pid,
+ "window_id": self._window_id,
+ "action": "execute_javascript",
+ "javascript": (
+ "(() => { try { return document.title; } "
+ "catch(e) { return 'error: ' + e; } })()"
+ ),
+ })
+ self.assertFalse(result.get("isError"))
+
+ # -----------------------------------------------------------------------
+ # page action=get_text
+ # -----------------------------------------------------------------------
+
+ def test_04_get_text_returns_body_text(self) -> None:
+ """get_text returns document.body.innerText containing the H1."""
+ with DriverClient(self.binary) as c:
+ result = c.call_tool("page", {
+ "pid": self._chrome_pid,
+ "window_id": self._window_id,
+ "action": "get_text",
+ })
+ self.assertFalse(result.get("isError"))
+ self.assertIn(EXPECTED_H1, _tool_text(result))
+
+ def test_05_get_text_includes_link_text(self) -> None:
+ """get_text includes anchor text from the page."""
+ with DriverClient(self.binary) as c:
+ result = c.call_tool("page", {
+ "pid": self._chrome_pid,
+ "window_id": self._window_id,
+ "action": "get_text",
+ })
+ text = _tool_text(result)
+ self.assertIn(EXPECTED_LINK_TEXT, text,
+ f"Expected link text in body text, got: {text[:200]!r}")
+
+ # -----------------------------------------------------------------------
+ # page action=query_dom
+ # -----------------------------------------------------------------------
+
+ def test_06_query_dom_returns_json_array(self) -> None:
+ """query_dom returns a JSON array of matching elements with hrefs."""
+ with DriverClient(self.binary) as c:
+ result = c.call_tool("page", {
+ "pid": self._chrome_pid,
+ "window_id": self._window_id,
+ "action": "query_dom",
+ "css_selector": "a[href]",
+ "attributes": ["href"],
+ })
+ self.assertFalse(result.get("isError"))
+ text = _tool_text(result)
+ self.assertIn("```json", text)
+ json_block = text.split("```json")[-1].split("```")[0].strip()
+ items = json.loads(json_block)
+ self.assertIsInstance(items, list)
+ self.assertGreaterEqual(len(items), 1)
+ hrefs = {item.get("href") or "" for item in items}
+ self.assertTrue(
+ any(EXPECTED_HREF_FRAGMENT in h for h in hrefs),
+ f"Expected href containing '{EXPECTED_HREF_FRAGMENT}', got: {hrefs}",
+ )
+
+ def test_07_query_dom_h1_no_attributes(self) -> None:
+ """query_dom without attributes returns tag and text."""
+ with DriverClient(self.binary) as c:
+ result = c.call_tool("page", {
+ "pid": self._chrome_pid,
+ "window_id": self._window_id,
+ "action": "query_dom",
+ "css_selector": "h1",
+ })
+ self.assertFalse(result.get("isError"))
+ text = _tool_text(result)
+ json_block = text.split("```json")[-1].split("```")[0].strip()
+ items = json.loads(json_block)
+ self.assertEqual(len(items), 1)
+ self.assertEqual(items[0]["tag"], "h1")
+ self.assertIn(EXPECTED_H1, items[0]["text"])
+
+ # -----------------------------------------------------------------------
+ # get_window_state javascript param
+ # -----------------------------------------------------------------------
+
+ def test_08_get_window_state_javascript_co_located(self) -> None:
+ """get_window_state with javascript= returns JS result alongside AX tree."""
+ with DriverClient(self.binary) as c:
+ result = c.call_tool("get_window_state", {
+ "pid": self._chrome_pid,
+ "window_id": self._window_id,
+ "javascript": "document.title",
+ })
+ self.assertFalse(result.get("isError"))
+ text = _tool_text(result)
+ self.assertIn("## JavaScript result", text)
+ # AX tree content is also present.
+ self.assertRegex(text, r"elements.*turn", re.IGNORECASE)
+
+ def test_09_get_window_state_javascript_error_is_inline(self) -> None:
+ """JS runtime errors in get_window_state appear inline, tool does not isError."""
+ # Chrome returns "missing value" for JS that throws — osascript exits 0.
+ # The result section appears but may contain "missing value" or an error msg.
+ with DriverClient(self.binary) as c:
+ result = c.call_tool("get_window_state", {
+ "pid": self._chrome_pid,
+ "window_id": self._window_id,
+ "javascript": "undefined_var_that_does_not_exist",
+ })
+ # isError must be False — JS failures must not kill the whole tool call.
+ self.assertFalse(result.get("isError"))
+ text = _tool_text(result)
+ # The JS result section must always appear.
+ self.assertIn("## JavaScript result", text)
+
+ # -----------------------------------------------------------------------
+ # Error cases
+ # -----------------------------------------------------------------------
+
+ def test_10_page_execute_missing_javascript_field(self) -> None:
+ """page execute_javascript returns isError when javascript field is absent."""
+ with DriverClient(self.binary) as c:
+ result = c.call_tool("page", {
+ "pid": self._chrome_pid,
+ "window_id": self._window_id,
+ "action": "execute_javascript",
+ })
+ self.assertTrue(result.get("isError"))
+
+ def test_11_page_query_dom_missing_selector(self) -> None:
+ """page query_dom returns isError when css_selector is absent."""
+ with DriverClient(self.binary) as c:
+ result = c.call_tool("page", {
+ "pid": self._chrome_pid,
+ "window_id": self._window_id,
+ "action": "query_dom",
+ })
+ self.assertTrue(result.get("isError"))
+
+
+if __name__ == "__main__":
+ unittest.main(verbosity=2)
diff --git a/libs/cua-driver/Tests/integration/test_chrome_minimized_nav.py b/libs/cua-driver/Tests/integration/test_chrome_minimized_nav.py
index 7f849daad5..0a5816ea2b 100644
--- a/libs/cua-driver/Tests/integration/test_chrome_minimized_nav.py
+++ b/libs/cua-driver/Tests/integration/test_chrome_minimized_nav.py
@@ -149,18 +149,12 @@ class ChromeMinimizedNavTests(unittest.TestCase):
time.sleep(2.0)
print(f"\n Chrome pid: {cls._chrome_pid}")
- # Ensure Chrome is visible with a window.
- subprocess.run(
- ["osascript", "-e",
- 'tell application "Google Chrome"\n'
- ' activate\n'
- ' if (count of windows) is 0 then\n'
- ' make new window\n'
- ' set URL of active tab of window 1 to "about:blank"\n'
- ' end if\n'
- 'end tell'],
- check=False, timeout=10,
- )
+ # Ensure Chrome has an about:blank window via launch_app (no focus steal).
+ with DriverClient(cls.binary) as c:
+ c.call_tool("launch_app", {
+ "bundle_id": CHROME_BUNDLE,
+ "urls": ["about:blank"],
+ })
time.sleep(2.0)
# Snapshot AX tree BEFORE minimizing to cache the omnibox index.
@@ -219,17 +213,13 @@ class ChromeMinimizedNavTests(unittest.TestCase):
pass
def setUp(self) -> None:
- # Reliably minimize Chrome: activate, wait, Cmd+M, wait.
- subprocess.run(
- ["osascript", "-e", 'tell application "Google Chrome" to activate'],
- check=False, timeout=5,
- )
- time.sleep(1.0)
- subprocess.run(
- ["osascript", "-e",
- 'tell application "System Events" to keystroke "m" using command down'],
- check=False, timeout=5,
- )
+ # Minimize Chrome via hotkey — cua-driver delivers Cmd+M to the pid
+ # via CGEvent.postToPid without stealing focus.
+ with DriverClient(self.binary) as c:
+ c.call_tool("hotkey", {
+ "pid": self._chrome_pid,
+ "keys": ["cmd", "m"],
+ })
time.sleep(1.5)
_activate_focus_monitor()
self._losses_before = _read_focus_losses()