Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cua-driver): browser JS primitives + menu-bar ghost click fix (#1386)
Francesco Bonacci7d75c4115a8d
diff --git a/libs/cua-driver/Skills/cua-driver/SKILL.md b/libs/cua-driver/Skills/cua-driver/SKILL.mdindex 996c6aab9f..7dbcd67391 100644--- a/libs/cua-driver/Skills/cua-driver/SKILL.md+++ b/libs/cua-driver/Skills/cua-driver/SKILL.md@@ -747,6 +747,69 @@ Chromium web content specifically also coerces `right_click` back toleft — use `element_index` for AX-addressable targets and accept thelimit otherwise.+### Browser JS primitives — `page` tool and `get_window_state(javascript=)`++When the AX tree doesn't expose the data you need (common in+Chromium/Electron — the tree is sparse for web content), use the+`page` tool or the `javascript` param on `get_window_state` to query+the DOM directly via Apple Events. Requires "Allow JavaScript from+Apple Events" to be enabled — see `WEB_APPS.md` for the setup path.++**Three actions on the `page` tool:**++- `page({pid, window_id, action: "get_text"})` — returns+ `document.body.innerText`. Fastest way to read page content, prices,+ article text, or any raw text the AX tree truncates or omits.++- `page({pid, window_id, action: "query_dom", css_selector: "a[href]",+ attributes: ["href"]})` — runs `querySelectorAll` and returns each+ match's tag, text, and requested attributes as a JSON array. Use for+ table rows, link hrefs, data attributes, structured page data.++- `page({pid, window_id, action: "execute_javascript", javascript:+ "..."})` — raw JS. Wrap in an IIFE with try-catch. Don't use this for+ elements already indexed by `get_window_state` — `click` and+ `set_value` are more reliable there.++**Co-located read — `get_window_state` with `javascript`:**++```+get_window_state({pid, window_id, javascript: "document.title"})+```++Runs the JS and appends the result as a `## JavaScript result` section+alongside the AX snapshot — one round-trip instead of two. Use this+when you need both the element tree (for subsequent clicks) and some+page data in the same turn.++**Decision rule — AX vs JS:**++| Need | Use |+|---|---|+| Click / type into an element | `get_window_state` → `click` / `set_value` (AX, works backgrounded) |+| Read text the AX tree drops | `page(get_text)` or `get_window_state(javascript=)` |+| Scrape structured data (tables, hrefs) | `page(query_dom)` |+| Trigger JS events / mutations | `page(execute_javascript)` |++Supported backends:++| App type | How | Context |+|---|---|---|+| Chrome / Brave / Edge | Apple Events `execute javascript` | Full DOM ✅ |+| Safari | Apple Events `do JavaScript` | Full DOM ✅ |+| Electron (VS Code, Cursor…) | SIGUSR1 → V8 inspector → CDP | Main process only: `process`, `Buffer` — no `document`, no `require` in sandboxed apps |+| Electron (with `--remote-debugging-port`) | CDP page target | Full DOM ✅ |++**Electron sandbox note:** SIGUSR1 connects to the Node.js *main* process.+Sandboxed Electron apps (VS Code, Cursor) strip `require` and Electron+APIs there. Useful for: `process.env`, `process.versions`, `process.cwd()`,+`process.pid`. For full DOM/renderer access, launch the app with+`--remote-debugging-port=9222` — cua-driver will detect and prefer the+page target automatically.++Arc returns no values; Firefox has no JS-via-AppleEvents support — see+`WEB_APPS.md` for the full matrix.+### 3. Re-snapshot and verify — mandatory**Always** call `get_window_state({pid, window_id})` after the action.diff --git a/libs/cua-driver/Skills/cua-driver/WEB_APPS.md b/libs/cua-driver/Skills/cua-driver/WEB_APPS.mdindex 5ea0cc2e4f..6fbaef9e21 100644--- a/libs/cua-driver/Skills/cua-driver/WEB_APPS.md+++ b/libs/cua-driver/Skills/cua-driver/WEB_APPS.md@@ -239,6 +239,225 @@ coerces the event back to a left-click — this appears to affectevery non-HID-tap synthesis path. Prefer `element_index` wheneverthe target is AX-addressable.+## Enable "Allow JavaScript from Apple Events" — browser support matrix++| Browser | `execute javascript` supported | Setting needed | Programmatic path |+|---|---|---|---|+| Chrome | ✅ Full | ✅ Yes | Edit Preferences JSON (see below) |+| Brave | ✅ Full | ✅ Yes | Edit Preferences JSON (same key, different path) |+| Edge | ✅ Full | ✅ Yes | Edit Preferences JSON (same key, different path) |+| Safari | ✅ Full (`do JavaScript`) | ✅ Yes | UI automation only — `defaults write` broken |+| Arc | ⚠️ No return values | No toggle | No reliable path |+| Firefox | ❌ Not supported | N/A | N/A |++### Chrome / Brave / Edge — Preferences JSON++Required for `osascript execute javascript` calls. All three are+Chromium-based and share the same preference key and mechanism.+Each browser stores preferences per-profile.++### Why menu clicks don't work++The menu item (`View → Developer → Allow JavaScript from Apple Events`)+is a security-sensitive toggle. Verified experimentally:++- `AXPress` — advertised actions are `[AXCancel, AXPick]`, not+ `AXPress`; Chrome's command dispatch silently discards it.+- `AXPick` on a leaf item — opens submenus correctly but does NOT+ commit a leaf toggle; the item is "selected" but not activated.+- System Events `click theItem` / `click at {x, y}` — returns the+ menu item reference (found it) but Chrome requires a genuine+ trusted user event to flip this flag; synthetic AppleEvent-routed+ clicks are rejected.+- `CGEvent.post(tap: .cghidEventTap)` while the menu is open —+ Chrome's event loop is occupied processing the menu; the event+ either races or Chrome treats it as untrusted for this toggle.++Additionally, when Chrome is **backgrounded**, the Developer submenu+items appear with `AXEnabled = false` (Chrome's `commandDispatch`+marks them DISABLED) — any action dispatched returns `.success` at+the AX layer but is silently discarded. This is the root cause of the+"ghost click" pattern: the driver reports ✅ but nothing changes.++### Correct path — write the Preferences JSON directly++Quit Chrome first, then write the flag, then relaunch:++```bash+# 1. Quit Chrome+osascript -e 'quit app "Google Chrome"'+sleep 1++# 2. Write the flag into the active profile's Preferences.+# Chrome stores this in TWO places — both must be set.+python3 -c "+import json, os+prefs_path = os.path.expanduser(+ '~/Library/Application Support/Google/Chrome/Default/Preferences')+data = json.load(open(prefs_path))+data.setdefault('browser', {})['allow_javascript_apple_events'] = True+data.setdefault('account_values', {}).setdefault('browser', {})['allow_javascript_apple_events'] = True+json.dump(data, open(prefs_path, 'w'))+print('allow_javascript_apple_events enabled in Default profile')+"++# 3. Relaunch Chrome and wait for sync to stabilise.+# Chrome sync fires ~1-2 s after launch and may briefly pull+# an older value from the server before Chrome pushes our local+# True back. Either test before sync fires (<1 s) or after it+# settles (>4 s). Waiting exactly ~2 s lands in the race window+# and is the most likely way to see a false negative.+open -a "Google Chrome"+sleep 5++# 4. Verify+osascript -e 'tell application "Google Chrome"+ tell active tab of front window+ execute javascript "1+1"+ end tell+end tell'+# → 2+```++**Which profile?** Chrome writes to whichever profile is active.+If you're unsure, write to all non-system profiles:++```bash+python3 -c "+import json, glob, os+for p in glob.glob(os.path.expanduser(+ '~/Library/Application Support/Google/Chrome/*/Preferences')):+ profile = p.split('/')[-2]+ if 'System' in profile or 'Guest' in profile:+ continue+ try:+ data = json.load(open(p))+ data.setdefault('browser', {})['allow_javascript_apple_events'] = True+ data.setdefault('account_values', {}).setdefault('browser', {})['allow_javascript_apple_events'] = True+ json.dump(data, open(p, 'w'))+ print(f'wrote to {profile}')+ except Exception as e:+ print(f'skipped {profile}: {e}')+"+```++Chrome overwrites its Preferences file on every clean exit, so the+write must happen while Chrome is **not running** — otherwise Chrome+will stomp the change when it quits.++**Sync note (Chrome only):** Chrome syncs `browser.allow_javascript_apple_events`+via Google account (confirmed in `chrome_syncable_prefs_database.cc`).+Writing both `browser` and `account_values.browser` to the local file+causes Chrome to push `true` to the sync server on next launch,+making the change durable. Brave and Edge use their own sync systems+and likely do NOT sync this Mac-only pref — treat as local-only for+those browsers.++### Brave++Same Chromium pref key, different profile directory:++```bash+osascript -e 'quit app "Brave Browser"' && sleep 1+python3 -c "+import json, glob, os+for p in glob.glob(os.path.expanduser(+ '~/Library/Application Support/BraveSoftware/Brave-Browser/*/Preferences')):+ profile = p.split('/')[-2]+ if 'System' in profile or 'Guest' in profile:+ continue+ try:+ data = json.load(open(p))+ data.setdefault('browser', {})['allow_javascript_apple_events'] = True+ json.dump(data, open(p, 'w'))+ print(f'wrote to {profile}')+ except Exception as e:+ print(f'skipped {profile}: {e}')+"+open -a "Brave Browser" && sleep 5+```++### Edge++Same Chromium pref key, different profile directory:++```bash+osascript -e 'quit app "Microsoft Edge"' && sleep 1+python3 -c "+import json, glob, os+for p in glob.glob(os.path.expanduser(+ '~/Library/Application Support/Microsoft Edge/*/Preferences')):+ profile = p.split('/')[-2]+ if 'System' in profile or 'Guest' in profile:+ continue+ try:+ data = json.load(open(p))+ data.setdefault('browser', {})['allow_javascript_apple_events'] = True+ json.dump(data, open(p, 'w'))+ print(f'wrote to {profile}')+ except Exception as e:+ print(f'skipped {profile}: {e}')+"+open -a "Microsoft Edge" && sleep 5+```++### Safari++Safari uses `do JavaScript "..." in document 1` (different AppleScript+verb from Chrome's `execute javascript`). The setting is under+`Develop → Allow JavaScript from Apple Events` (requires the Develop+menu to be enabled first via Settings → Advanced → "Show features for+web developers").++`defaults write -app Safari AllowJavaScriptFromAppleEvents 1` **does+not work** — Safari ignores the defaults key and routes this toggle+through macOS's security framework, which shows a password-confirmation+dialog. The only working programmatic path is UI automation:++```applescript+-- Requires Accessibility permission for the calling process.+-- Safari must already be running with the Develop menu visible.+tell application "Safari" to activate+delay 0.3+tell application "System Events"+ tell process "Safari"+ click menu item "Allow JavaScript from Apple Events" of menu 1 ¬+ of menu bar item "Develop" of menu bar 1+ delay 0.3+ -- Safari shows a confirmation dialog — click Allow+ click button "Allow" of window 1+ end tell+end tell+```++### Arc++Arc has an AppleScript dictionary and accepts `execute javascript`, but+**never returns a value** — the call always returns `missing value`.+There is no "Allow JavaScript from Apple Events" toggle. The only+workaround is to write results to the clipboard inside the JS and read+it back:++```applescript+tell application "Arc"+ tell active tab of front window+ execute javascript "navigator.clipboard.writeText(document.title)"+ end tell+end tell+delay 0.3+set theTitle to the clipboard+```++Arc's AppleScript JS support is confirmed broken for return values+(as of 2025). If you need JS results from Arc, use a WebExtension+with Native Messaging instead.++### Firefox++Firefox has no `execute javascript` capability. Bugzilla #287447+(filed 2004) tracks this and remains unresolved. Use WebDriver /+Playwright / a WebExtension with Native Messaging for Firefox.+## Typing into a web input```diff --git a/libs/cua-driver/Sources/CuaDriverCore/Browser/BrowserJS.swift b/libs/cua-driver/Sources/CuaDriverCore/Browser/BrowserJS.swiftnew file mode 100644index 0000000000..6daff59ebb--- /dev/null+++ b/libs/cua-driver/Sources/CuaDriverCore/Browser/BrowserJS.swift@@ -0,0 +1,263 @@+import CoreGraphics+import Foundation+import os++/// Execute JavaScript in a browser tab via Apple Events.+///+/// Supports the Chromium family (Chrome, Brave, Edge) which share the same+/// `execute javascript` AppleScript verb, and Safari which uses `do JavaScript`.+/// Firefox and Arc are not supported — see WEB_APPS.md for the full matrix.+///+/// **Prerequisite:** "Allow JavaScript from Apple Events" must be enabled in+/// the target browser. See WEB_APPS.md → "Enable Allow JavaScript from Apple+/// Events" for the programmatic setup path.+public enum BrowserJS {++ // MARK: - Public API++ public enum Error: Swift.Error, CustomStringConvertible {+ case unsupportedBrowser(String)+ case windowNotFound(UInt32)+ case javascriptNotEnabled(String)+ case executionFailed(String)++ public var description: String {+ switch self {+ case .unsupportedBrowser(let id):+ return "Browser '\(id)' does not support JavaScript via Apple Events. "+ + "Supported: Chrome (com.google.Chrome), Brave (com.brave.Browser), "+ + "Edge (com.microsoft.edgemac), Safari (com.apple.Safari)."+ case .windowNotFound(let wid):+ return "Could not find a browser window matching window_id \(wid). "+ + "Ensure the browser is running and the window is on the current Space."+ case .javascriptNotEnabled(let appName):+ return "'\(appName)' rejected the JavaScript execution — "+ + "'Allow JavaScript from Apple Events' is not enabled. "+ + "See WEB_APPS.md → 'Enable Allow JavaScript from Apple Events' "+ + "for the programmatic setup path (quit browser → edit Preferences JSON → relaunch)."+ case .executionFailed(let detail):+ return "JavaScript execution failed: \(detail)"+ }+ }+ }++ /// Run `javascript` in the active tab of the browser window identified by+ /// `windowId` (CGWindowID) and return the result as a string.+ public static func execute(+ javascript: String,+ bundleId: String,+ windowId: UInt32+ ) async throws -> String {+ guard let spec = browserSpec(for: bundleId) else {+ throw Error.unsupportedBrowser(bundleId)+ }++ // Find the CGWindow entry for this window_id to get its title,+ // which we use to locate the matching AppleScript window.+ guard let windowTitle = cgWindowTitle(for: windowId) else {+ throw Error.windowNotFound(windowId)+ }++ let script = spec.buildScript(javascript, windowTitle)+ return try await runAppleScript(script, appName: spec.appName)+ }++ /// True when this bundle ID is handled via Apple Events (Chrome/Brave/Edge/Safari).+ public static func supports(bundleId: String) -> Bool {+ browserSpec(for: bundleId) != nil+ }++ // MARK: - Browser specs++ private struct BrowserSpec {+ let appName: String+ let buildScript: (_ javascript: String, _ windowTitle: String) -> String+ }++ private static func browserSpec(for bundleId: String) -> BrowserSpec? {+ switch bundleId {+ case "com.google.Chrome":+ return chromiumSpec(appName: "Google Chrome")+ case "com.brave.Browser":+ return chromiumSpec(appName: "Brave Browser")+ case "com.microsoft.edgemac":+ return chromiumSpec(appName: "Microsoft Edge")+ case "com.apple.Safari":+ return safariSpec()+ default:+ return nil+ }+ }++ private static func chromiumSpec(appName: String) -> BrowserSpec {+ BrowserSpec(appName: appName) { javascript, windowTitle in+ // Escape the window title for AppleScript string comparison.+ // We use `whose name contains` to handle truncated titles.+ let escapedTitle = windowTitle+ .replacingOccurrences(of: "\\", with: "\\\\")+ .replacingOccurrences(of: "\"", with: "\\\"")+ return """+ tell application "\(appName)"+ set matchedWindow to missing value+ repeat with w in windows+ if name of w contains "\(escapedTitle)" then+ set matchedWindow to w+ exit repeat+ end if+ end repeat+ if matchedWindow is missing value then+ set matchedWindow to front window+ end if+ tell active tab of matchedWindow+ execute javascript \(appleScriptString(javascript))+ end tell+ end tell+ """+ }+ }++ private static func safariSpec() -> BrowserSpec {+ BrowserSpec(appName: "Safari") { javascript, windowTitle in+ let escapedTitle = windowTitle+ .replacingOccurrences(of: "\\", with: "\\\\")+ .replacingOccurrences(of: "\"", with: "\\\"")+ return """+ tell application "Safari"+ set matchedDoc to missing value+ repeat with d in documents+ if name of d contains "\(escapedTitle)" then+ set matchedDoc to d+ exit repeat+ end if+ end repeat+ if matchedDoc is missing value then+ set matchedDoc to document 1+ end if+ do JavaScript \(appleScriptString(javascript)) in matchedDoc+ end tell+ """+ }+ }++ // MARK: - AppleScript runner++ private static func runAppleScript(+ _ script: String,+ appName: String+ ) async throws -> String {+ // Write to a temp file to avoid shell-quoting the script entirely.+ let tmpURL = URL(fileURLWithPath: NSTemporaryDirectory())+ .appendingPathComponent(UUID().uuidString + ".applescript")+ try script.write(to: tmpURL, atomically: true, encoding: .utf8)+ defer { try? FileManager.default.removeItem(at: tmpURL) }++ return try await withCheckedThrowingContinuation { continuation in+ let proc = Process()+ proc.executableURL = URL(fileURLWithPath: "/usr/bin/osascript")+ proc.arguments = [tmpURL.path]+ let stdout = Pipe()+ let stderr = Pipe()+ proc.standardOutput = stdout+ proc.standardError = stderr++ let once = OSAllocatedUnfairLock(initialState: false)+ let resumeOnce: @Sendable (Result<String, Swift.Error>) -> Void = { result in+ let shouldResume = once.withLock { (fired: inout Bool) -> Bool in+ guard !fired else { return false }+ fired = true; return true+ }+ guard shouldResume else { return }+ switch result {+ case .success(let s): continuation.resume(returning: s)+ case .failure(let e): continuation.resume(throwing: e)+ }+ }++ let timeoutItem = DispatchWorkItem {+ proc.terminate()+ resumeOnce(.failure(Error.executionFailed(+ "osascript timed out after 15 s — browser may be showing a permission dialog")))+ }+ DispatchQueue.global().asyncAfter(deadline: .now() + 15, execute: timeoutItem)++ proc.terminationHandler = { p in+ timeoutItem.cancel()+ let out = String(+ data: stdout.fileHandleForReading.readDataToEndOfFile(),+ encoding: .utf8+ )?.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""+ let err = String(+ data: stderr.fileHandleForReading.readDataToEndOfFile(),+ encoding: .utf8+ )?.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""+ if p.terminationStatus == 0 {+ resumeOnce(.success(out))+ } else if err.contains("turned off") || err.contains("AppleScript is turned off") {+ resumeOnce(.failure(Error.javascriptNotEnabled(appName)))+ } else {+ resumeOnce(.failure(Error.executionFailed(err.isEmpty ? out : err)))+ }+ }+ do {+ try proc.run()+ } catch {+ timeoutItem.cancel()+ resumeOnce(.failure(error))+ }+ }+ }++ // MARK: - CGWindow helpers++ private static func cgWindowTitle(for windowId: UInt32) -> String? {+ // Enumerate all on-screen windows and find the one with matching ID.+ // CGWindowListCopyWindowInfo([.optionIncludingWindow], id) is unreliable+ // for windows not yet in the compositing list — enumerating all is safer.+ //+ // Return "" (empty string) when the window exists but has no title+ // (new tab, about:blank, freshly-opened window). The AppleScript+ // `name contains ""` expression matches every string, so an empty+ // title falls through to the `front window` fallback — correct behavior.+ // Only return nil when the window ID is genuinely not in any list.+ let list = CGWindowListCopyWindowInfo(+ [.optionOnScreenOnly, .excludeDesktopElements], kCGNullWindowID+ ) as? [[String: Any]] ?? []+ for entry in list {+ guard let id = entry[kCGWindowNumber as String] as? Int,+ UInt32(id) == windowId else { continue }+ return entry[kCGWindowName as String] as? String ?? ""+ }+ // Fall back to off-screen windows (minimized, hidden).+ let all = CGWindowListCopyWindowInfo([], kCGNullWindowID) as? [[String: Any]] ?? []+ for entry in all {+ guard let id = entry[kCGWindowNumber as String] as? Int,+ UInt32(id) == windowId else { continue }+ return entry[kCGWindowName as String] as? String ?? ""+ }+ return nil+ }++ // MARK: - AppleScript string literal builder++ /// Wraps `value` in a multi-line AppleScript string literal using+ /// quoted form syntax to avoid any quoting/escaping issues with the+ /// JavaScript payload.+ private static func appleScriptString(_ value: String) -> String {+ // AppleScript doesn't support heredocs; we use string concatenation+ // for newlines and escape only the characters AppleScript requires.+ if !value.contains("\n") && !value.contains("\"") && !value.contains("\\") {+ return "\"\(value)\""+ }+ // Build the string by concatenating segments split on newlines,+ // joined with (ASCII character 10) for newlines.+ let lines = value.split(separator: "\n", omittingEmptySubsequences: false)+ let escaped = lines.map { line -> String in+ let s = String(line)+ .replacingOccurrences(of: "\\", with: "\\\\")+ .replacingOccurrences(of: "\"", with: "\\\"")+ return "\"\(s)\""+ }+ if escaped.count == 1 { return escaped[0] }+ return escaped.joined(separator: " & (ASCII character 10) & ")+ }+}diff --git a/libs/cua-driver/Sources/CuaDriverCore/Browser/ElectronJS.swift b/libs/cua-driver/Sources/CuaDriverCore/Browser/ElectronJS.swiftnew file mode 100644index 0000000000..9657196a33--- /dev/null+++ b/libs/cua-driver/Sources/CuaDriverCore/Browser/ElectronJS.swift@@ -0,0 +1,302 @@+import AppKit+import Foundation++/// Execute JavaScript in a running Electron app's renderer via the V8 inspector+/// (SIGUSR1 → CDP WebSocket → Runtime.evaluate).+///+/// **How it works:**+/// 1. Send SIGUSR1 to the Electron main process — Node activates the V8 inspector+/// on a random localhost port (same as `node --inspect`).+/// 2. Discover the new port by diffing the pid's TCP LISTEN sockets before/after.+/// 3. GET `http://localhost:{port}/json` for the CDP WebSocket URL.+/// 4. Open a WebSocket, send `Runtime.evaluate`, return the result.+///+/// **Requirement:** The app must have the `EnableNodeCliInspectArguments` Electron+/// fuse enabled (the default in development builds; some production apps disable it+/// post-CVE-2024-23738). Check with:+/// `npx @electron/fuses read --app /Applications/MyApp.app`+///+/// VS Code and Cursor both have this fuse ON. Slack/Discord likely have it OFF.+public enum ElectronJS {++ // MARK: - Public API++ public enum Error: Swift.Error, CustomStringConvertible {+ case notElectron(String)+ case inspectorNotAvailable(String)+ case cdpConnectionFailed(String)+ case evaluationFailed(String)++ public var description: String {+ switch self {+ case .notElectron(let detail):+ return "Not an Electron app: \(detail). "+ + "Use the `page` tool only with Electron apps "+ + "(bundle must contain Electron Framework.framework)."+ case .inspectorNotAvailable(let detail):+ return "V8 inspector did not activate after SIGUSR1: \(detail)\n"+ + "The app may have the EnableNodeCliInspectArguments fuse disabled. "+ + "Verify with: npx @electron/fuses read --app /path/to/App.app"+ case .cdpConnectionFailed(let detail):+ return "CDP connection failed: \(detail)"+ case .evaluationFailed(let detail):+ return "JavaScript evaluation failed: \(detail)"+ }+ }+ }++ /// True if the app running at `pid` is Electron-based — detected by the+ /// presence of `Electron Framework.framework` inside its bundle.+ public static func isElectron(pid: Int32) -> Bool {+ guard let app = NSWorkspace.shared.runningApplications+ .first(where: { $0.processIdentifier == pid }),+ let bundleURL = app.bundleURL else { return false }+ let frameworkPath = bundleURL+ .appendingPathComponent("Contents/Frameworks/Electron Framework.framework")+ return FileManager.default.fileExists(atPath: frameworkPath.path)+ }++ /// Execute `javascript` in the Electron app running at `pid`.+ /// Returns the result serialised as a string.+ ///+ /// **Context available depends on how the inspector was activated:**+ ///+ /// - `--remote-debugging-port=N` at launch → prefers "page" targets (full DOM,+ /// `window`, `document`, renderer globals). Best for UI automation.+ /// - SIGUSR1 (no relaunch) → activates the Node.js main process inspector.+ /// `process`, `Buffer`, `console` available; `require`, `document`, and+ /// Electron APIs are NOT available in sandboxed apps (VS Code, Cursor).+ /// Useful for process-level reads: `process.env`, `process.versions`,+ /// `process.cwd()`, `process.pid`.+ public static func execute(javascript: String, pid: Int32) async throws -> String {+ guard isElectron(pid: pid) else {+ let name = NSWorkspace.shared.runningApplications+ .first(where: { $0.processIdentifier == pid })?.localizedName ?? "pid \(pid)"+ throw Error.notElectron(name)+ }++ // Prefer a "page" target (renderer with DOM) if the app was launched with+ // --remote-debugging-port. Page targets expose document/window/DOM APIs.+ if let pagePort = await pageTarget() {+ return try await cdpEvaluate(port: pagePort, javascript: javascript)+ }++ // If the inspector is already active (from a prior call in this session),+ // skip SIGUSR1 and reuse the port directly.+ if let existingPort = await activeInspectorPort(pid: pid) {+ return try await cdpEvaluate(port: existingPort, javascript: javascript)+ }++ // Snapshot TCP LISTEN ports before signalling so we can diff afterwards.+ let portsBefore = await listeningPorts(pid: pid)++ // Activate the V8 inspector — same mechanism as `node --inspect`.+ kill(pid, SIGUSR1)++ // Poll up to 2 s for a new LISTEN port to appear.+ var inspectorPort: Int?+ for _ in 0..<10 {+ try await Task.sleep(for: .milliseconds(200))+ let portsAfter = await listeningPorts(pid: pid)+ let newPorts = portsAfter.subtracting(portsBefore)+ for port in newPorts {+ if await isInspectorPort(port) {+ inspectorPort = port+ break+ }+ }+ if inspectorPort != nil { break }++ // Fallback: scan the common Node inspector range in case lsof+ // didn't resolve the pid association in time.+ if let port = await scanInspectorPorts() {+ inspectorPort = port+ break+ }+ }++ guard let port = inspectorPort else {+ throw Error.inspectorNotAvailable(+ "no CDP endpoint appeared on localhost within 2 s after SIGUSR1"+ )+ }++ return try await cdpEvaluate(port: port, javascript: javascript)+ }++ // MARK: - Port discovery++ /// Scan common remote-debugging ports for a CDP "page" target (renderer with+ /// DOM access). Returns the port if found. Apps launched with+ /// `--remote-debugging-port=N` expose page targets here; apps activated via+ /// SIGUSR1 only expose a "node" main-process target with no DOM.+ private static func pageTarget() async -> Int? {+ for port in [9222, 9223, 9224, 9225, 9230] {+ guard let url = URL(string: "http://127.0.0.1:\(port)/json") else { continue }+ var req = URLRequest(url: url); req.timeoutInterval = 0.3+ guard let (data, _) = try? await URLSession.shared.data(for: req),+ let targets = try? JSONSerialization.jsonObject(with: data) as? [[String: Any]],+ targets.contains(where: { ($0["type"] as? String) == "page" })+ else { continue }+ return port+ }+ return nil+ }++ /// Check if the pid already has an active V8 inspector port.+ private static func activeInspectorPort(pid: Int32) async -> Int? {+ let ports = await listeningPorts(pid: pid)+ for port in ports {+ if await isInspectorPort(port) { return port }+ }+ return nil+ }++ /// TCP ports the pid is currently listening on, via `lsof -p <pid> -iTCP:LISTEN`.+ private static func listeningPorts(pid: Int32) async -> Set<Int> {+ let output = await runProcess(+ "/usr/sbin/lsof",+ args: ["-p", "\(pid)", "-iTCP", "-sTCP:LISTEN", "-Fn", "-P"]+ )+ var ports = Set<Int>()+ for line in output.split(separator: "\n") {+ // lsof -Fn lines: "n*:9229" or "n127.0.0.1:9229"+ guard line.hasPrefix("n"), let colon = line.lastIndex(of: ":") else { continue }+ let portStr = String(line[line.index(after: colon)...])+ if let port = Int(portStr) { ports.insert(port) }+ }+ return ports+ }++ /// Returns true if `port` responds to the CDP `/json` HTTP endpoint.+ private static func isInspectorPort(_ port: Int) async -> Bool {+ guard let url = URL(string: "http://127.0.0.1:\(port)/json") else { return false }+ var req = URLRequest(url: url)+ req.timeoutInterval = 0.5+ do {+ let (_, resp) = try await URLSession.shared.data(for: req)+ return (resp as? HTTPURLResponse)?.statusCode == 200+ } catch { return false }+ }++ /// Scan the standard Node inspector port range (9229-9249) for an active endpoint.+ private static func scanInspectorPorts() async -> Int? {+ for port in 9229...9249 {+ if await isInspectorPort(port) { return port }+ }+ return nil+ }++ // MARK: - CDP evaluation++ private static func cdpEvaluate(port: Int, javascript: String) async throws -> String {+ // 1. Fetch the list of debuggable targets.+ guard let jsonURL = URL(string: "http://127.0.0.1:\(port)/json") else {+ throw Error.cdpConnectionFailed("bad port \(port)")+ }+ var req = URLRequest(url: jsonURL)+ req.timeoutInterval = 5+ let (data, _) = try await URLSession.shared.data(for: req)++ guard let targets = try? JSONSerialization.jsonObject(with: data) as? [[String: Any]] else {+ throw Error.cdpConnectionFailed("could not parse /json response from port \(port)")+ }+ // Prefer the page target; fall back to the first target with a WS URL.+ let target = targets.first(where: { ($0["type"] as? String) == "page" })+ ?? targets.first(where: { $0["webSocketDebuggerUrl"] != nil })+ guard let wsURLStr = target?["webSocketDebuggerUrl"] as? String,+ let wsURL = URL(string: wsURLStr) else {+ throw Error.cdpConnectionFailed("no debuggable target found at port \(port)")+ }++ // 2. Send Runtime.evaluate over WebSocket.+ let payload: [String: Any] = [+ "id": 1,+ "method": "Runtime.evaluate",+ "params": [+ "expression": javascript,+ "returnByValue": true,+ "awaitPromise": true,+ ],+ ]+ let payloadStr = String(+ data: try JSONSerialization.data(withJSONObject: payload),+ encoding: .utf8+ )!++ return try await withCheckedThrowingContinuation { continuation in+ let ws = URLSession.shared.webSocketTask(with: wsURL)+ ws.resume()+ ws.send(.string(payloadStr)) { sendError in+ if let err = sendError {+ ws.cancel()+ continuation.resume(throwing: Error.cdpConnectionFailed(err.localizedDescription))+ return+ }+ ws.receive { result in+ ws.cancel()+ switch result {+ case .failure(let err):+ continuation.resume(throwing: Error.cdpConnectionFailed(err.localizedDescription))+ case .success(let message):+ guard case .string(let str) = message else {+ continuation.resume(throwing: Error.cdpConnectionFailed("binary WebSocket frame"))+ return+ }+ continuation.resume(with: Result { try Self.parseCDPResult(str) })+ }+ }+ }+ }+ }++ /// Extract the return value from a `Runtime.evaluate` CDP response.+ private static func parseCDPResult(_ json: String) throws -> String {+ guard let data = json.data(using: .utf8),+ let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any] else {+ return json+ }+ if let error = obj["error"] as? [String: Any] {+ let msg = error["message"] as? String ?? json+ throw Error.evaluationFailed(msg)+ }+ if let result = obj["result"] as? [String: Any],+ let inner = result["result"] as? [String: Any] {+ // Exception thrown inside JS+ if let exDesc = (result["exceptionDetails"] as? [String: Any])?["text"] as? String {+ throw Error.evaluationFailed(exDesc)+ }+ // Serialise the return value+ if let value = inner["value"] {+ if let str = value as? String { return str }+ if let num = value as? NSNumber { return num.stringValue }+ if let data = try? JSONSerialization.data(withJSONObject: value),+ let str = String(data: data, encoding: .utf8) { return str }+ return "\(value)"+ }+ return inner["description"] as? String ?? "undefined"+ }+ return json+ }++ // MARK: - Subprocess helper++ private static func runProcess(_ executable: String, args: [String]) async -> String {+ await withCheckedContinuation { continuation in+ let proc = Process()+ proc.executableURL = URL(fileURLWithPath: executable)+ proc.arguments = args+ let pipe = Pipe()+ proc.standardOutput = pipe+ proc.standardError = Pipe()+ proc.terminationHandler = { _ in+ let out = String(+ data: pipe.fileHandleForReading.readDataToEndOfFile(),+ encoding: .utf8+ ) ?? ""+ continuation.resume(returning: out)+ }+ do { try proc.run() } catch { continuation.resume(returning: "") }+ }+ }+}diff --git a/libs/cua-driver/Sources/CuaDriverCore/Input/AXInput.swift b/libs/cua-driver/Sources/CuaDriverCore/Input/AXInput.swiftindex 4dc764ba7f..e552bbcb0c 100644--- a/libs/cua-driver/Sources/CuaDriverCore/Input/AXInput.swift+++ b/libs/cua-driver/Sources/CuaDriverCore/Input/AXInput.swift@@ -166,6 +166,16 @@ public enum AXInput {return attributeString(element, name)}+ /// Read a boolean attribute from an element. Returns nil when the attribute+ /// is absent or unreadable.+ public static func boolAttribute(_ name: String, of element: AXUIElement) -> Bool? {+ var value: CFTypeRef?+ guard AXUIElementCopyAttributeValue(element, name as CFString, &value) == .success,+ let v = value else { return nil }+ guard CFGetTypeID(v) == CFBooleanGetTypeID() else { return nil }+ return CFBooleanGetValue((v as! CFBoolean))+ }+/// The element's on-screen center in screen-point coordinates/// (top-left origin). Returns nil when the element has no/// position / size (menus, offscreen elements, hidden rows).diff --git a/libs/cua-driver/Sources/CuaDriverServer/ToolRegistry.swift b/libs/cua-driver/Sources/CuaDriverServer/ToolRegistry.swiftindex fbab2b4e52..81b3270bcb 100644--- a/libs/cua-driver/Sources/CuaDriverServer/ToolRegistry.swift+++ b/libs/cua-driver/Sources/CuaDriverServer/ToolRegistry.swift@@ -40,6 +40,7 @@ public struct ToolRegistry: Sendable {"press_key","hotkey","set_value",+ "page",]/// Click-family tools — their argument set carries a click point we@@ -239,5 +240,6 @@ public struct ToolRegistry: Sendable {GetConfigTool.handler,SetConfigTool.handler,ZoomTool.handler,+ PageTool.handler,])}diff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/ClickTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/ClickTool.swiftindex 1e6ac10446..e0a9f55f06 100644--- a/libs/cua-driver/Sources/CuaDriverServer/Tools/ClickTool.swift+++ b/libs/cua-driver/Sources/CuaDriverServer/Tools/ClickTool.swift@@ -249,6 +249,23 @@ public enum ClickTool {// without this we have no way to detect a silent no-op.// See the post-dispatch warning assembled below.let advertisedActions = AXInput.advertisedActionNames(of: element)+ // Bail early if the element is disabled (AXEnabled = false).+ // AXUIElementPerformAction returns .success even on disabled+ // elements — the call reaches the app's AX layer but the app's+ // command-dispatch silently discards it (Chrome marks Developer+ // submenu items as disabled via commandDispatch when it isn't+ // frontmost). Surfacing this as an error rather than a silent+ // no-op lets the caller decide whether to activate the app first.+ if AXInput.boolAttribute("AXEnabled", of: element) == false {+ let target = AXInput.describe(element)+ let appName = NSRunningApplication(processIdentifier: pid)?+ .localizedName ?? "the app"+ var msg = "❌ [\(index)] \(target.role ?? "element") \"\(target.title ?? "")\" is disabled (AXEnabled = false) — action would be a silent no-op."+ msg += "\n\nThis usually means the target app is not frontmost."+ msg += " Activate it first, then re-snapshot and retry:"+ msg += "\n osascript -e 'tell application \"\(appName)\" to activate'"+ return errorResult(msg)+ }// Animate the visual agent cursor to the target before// firing the AX action. `animateAndWait` is a no-op// when the cursor is disabled (the default) or whendiff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swiftindex 022807fe36..018d43ebc2 100644--- a/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift+++ b/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift@@ -94,6 +94,19 @@ public enum GetWindowStateTool {"description":"Optional case-insensitive substring. When set, `tree_markdown` only contains lines that match plus their ancestor chain; element indices and `element_count` are unchanged.",],+ "javascript": [+ "type": "string",+ "description": """+ Optional JavaScript to execute in the browser tab and return \+ alongside the AX snapshot — one round-trip instead of two. \+ Only works for Chromium-family browsers (Chrome, Brave, Edge) \+ and Safari; requires 'Allow JavaScript from Apple Events' to be \+ enabled first (see WEB_APPS.md). The result is appended to the \+ response as a `## JavaScript result` section. Use for read-only \+ queries (document.title, innerText, querySelectorAll, etc.). \+ For mutations or side effects use the `page` tool instead.+ """,+ ],],"additionalProperties": false,],@@ -123,6 +136,7 @@ public enum GetWindowStateTool {"window_id \(rawWindowId) is outside the supported UInt32 range.")}let query = arguments?["query"]?.stringValue+ let javascript = arguments?["javascript"]?.stringValue// Validate that the window belongs to this pid. The driver// never guesses which window to snapshot — the caller names@@ -218,6 +232,28 @@ public enum GetWindowStateTool {if captureMode != .vision && !snapshot.treeMarkdown.isEmpty {textContent += "\n\n" + snapshot.treeMarkdown}+ // If the caller passed a javascript snippet, run it in the+ // browser tab and append the result — one round-trip instead+ // of two separate tool calls.+ if let js = javascript {+ let bundleId = snapshot.bundleId ?? ""+ do {+ let jsResult: String+ if BrowserJS.supports(bundleId: bundleId) {+ jsResult = try await BrowserJS.execute(+ javascript: js, bundleId: bundleId, windowId: windowId)+ } else if ElectronJS.isElectron(pid: pid) {+ jsResult = try await ElectronJS.execute(+ javascript: js, pid: pid)+ } else {+ throw BrowserJS.Error.unsupportedBrowser(bundleId)+ }+ textContent += "\n\n## JavaScript result\n\n```\n\(jsResult)\n```"+ } catch {+ textContent += "\n\n## JavaScript result\n\n❌ \(error)"+ }+ }+var content: [Tool.Content] = []if let b64 = snapshot.screenshotPngBase64 {content.append(diff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/PageTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/PageTool.swiftnew file mode 100644index 0000000000..e9a737fcb1--- /dev/null+++ b/libs/cua-driver/Sources/CuaDriverServer/Tools/PageTool.swift@@ -0,0 +1,221 @@+import AppKit+import CuaDriverCore+import Foundation+import MCP++/// Browser page primitives — execute JavaScript, extract page text, query DOM.+///+/// Use `get_window_state` with its `javascript` param for **read-only** queries+/// co-located with a snapshot. Use `page` when you need a standalone JS call+/// (mutations, navigation side-effects, or results without a full AX walk).+///+/// Requires "Allow JavaScript from Apple Events" to be enabled in the target+/// browser — see WEB_APPS.md for the setup path.+public enum PageTool {++ public static let handler = ToolHandler(+ tool: Tool(+ name: "page",+ description: """+ Browser page primitives — execute JavaScript, extract page \+ text, or query DOM elements via CSS selector.++ Three actions:++ • `execute_javascript` — run arbitrary JS in the active tab \+ and return the result. Wrap in an IIFE with try-catch for \+ safety. Don't use this for elements already indexed by \+ `get_window_state` — prefer `click` / `set_value` there.++ • `get_text` — returns `document.body.innerText`. Faster than \+ walking the AX tree for plain-text content (prices, article \+ body, table values) that the AX tree drops or truncates.++ • `query_dom` — runs `querySelectorAll(css_selector)` and \+ returns each matching element's tag, innerText, and any \+ requested attributes as a JSON array. Useful for structured \+ data (table rows, link hrefs, data-* attributes) that the AX \+ tree flattens.++ Supported browsers: Chrome (com.google.Chrome), Brave \+ (com.brave.Browser), Edge (com.microsoft.edgemac), Safari \+ (com.apple.Safari). Requires "Allow JavaScript from Apple \+ Events" enabled — see WEB_APPS.md.++ `pid` and `window_id` identify the target window. The browser \+ does NOT need to be frontmost for read actions. For mutations \+ that trigger UI changes (form submits, navigation) the browser \+ should be frontmost so the results are visible.+ """,+ inputSchema: [+ "type": "object",+ "required": ["pid", "window_id", "action"],+ "properties": [+ "pid": [+ "type": "integer",+ "description": "Browser process ID.",+ ],+ "window_id": [+ "type": "integer",+ "description": "CGWindowID of the target browser window.",+ ],+ "action": [+ "type": "string",+ "enum": ["execute_javascript", "get_text", "query_dom"],+ "description": "Which page primitive to run.",+ ],+ "javascript": [+ "type": "string",+ "description": "JS to execute (action=execute_javascript only). Should return a serialisable value. Wrap in IIFE: `(() => { … })()`.",+ ],+ "css_selector": [+ "type": "string",+ "description": "CSS selector (action=query_dom only). Passed to querySelectorAll — standard CSS syntax.",+ ],+ "attributes": [+ "type": "array",+ "items": ["type": "string"],+ "description": "Element attributes to include per match (action=query_dom only). E.g. [\"href\", \"src\", \"data-id\"]. tag and innerText are always included.",+ ],+ ],+ "additionalProperties": false,+ ],+ annotations: .init(+ readOnlyHint: false,+ destructiveHint: false,+ idempotentHint: false,+ openWorldHint: false+ )+ ),+ invoke: { arguments in+ guard let rawPid = arguments?["pid"]?.intValue,+ let pid = Int32(exactly: rawPid)+ else {+ return errorResult("Missing or invalid pid.")+ }+ guard let rawWindowId = arguments?["window_id"]?.intValue,+ let windowId = UInt32(exactly: rawWindowId)+ else {+ return errorResult("Missing or invalid window_id.")+ }+ guard let action = arguments?["action"]?.stringValue else {+ return errorResult("Missing required field action.")+ }++ // Resolve bundle ID from the running app (NSWorkspace requires main thread).+ let bundleId = await MainActor.run {+ NSWorkspace.shared.runningApplications+ .first(where: { $0.processIdentifier == pid })?+ .bundleIdentifier ?? ""+ }++ switch action {++ case "execute_javascript":+ guard let js = arguments?["javascript"]?.stringValue, !js.isEmpty else {+ return errorResult(+ "action=execute_javascript requires a non-empty javascript field.")+ }+ do {+ let result = try await executeJS(js, bundleId: bundleId, pid: pid, windowId: windowId)+ return okResult("## Result\n\n```\n\(result)\n```")+ } catch {+ return errorResult("\(error)")+ }++ case "get_text":+ do {+ let result = try await executeJS(+ "document.body.innerText",+ bundleId: bundleId, pid: pid, windowId: windowId)+ return okResult(result)+ } catch {+ return errorResult("\(error)")+ }++ case "query_dom":+ guard let selector = arguments?["css_selector"]?.stringValue,+ !selector.isEmpty+ else {+ return errorResult(+ "action=query_dom requires a non-empty css_selector field.")+ }+ let attrs: [String]+ if let rawAttrs = arguments?["attributes"]?.arrayValue {+ attrs = rawAttrs.compactMap { $0.stringValue }+ } else {+ attrs = []+ }+ let attrJS = attrs.isEmpty+ ? "[]"+ : "[\(attrs.map { "\"\($0)\"" }.joined(separator: ", "))]"+ let js = """+ (() => {+ const attrs = \(attrJS);+ return JSON.stringify(+ Array.from(document.querySelectorAll(\(jsonString(selector)))).map(el => {+ const obj = { tag: el.tagName.toLowerCase(), text: el.innerText?.trim() };+ for (const a of attrs) obj[a] = el.getAttribute(a);+ return obj;+ })+ );+ })()+ """+ do {+ let result = try await executeJS(js, bundleId: bundleId, pid: pid, windowId: windowId)+ return okResult("## DOM query: `\(selector)`\n\n```json\n\(result)\n```")+ } catch {+ return errorResult("\(error)")+ }++ default:+ return errorResult(+ "Unknown action '\(action)'. Valid values: execute_javascript, get_text, query_dom.")+ }+ }+ )++ private static func okResult(_ text: String) -> CallTool.Result {+ CallTool.Result(content: [.text(text: text, annotations: nil, _meta: nil)])+ }++ private static func errorResult(_ message: String) -> CallTool.Result {+ CallTool.Result(+ content: [.text(text: message, annotations: nil, _meta: nil)],+ isError: true+ )+ }++ /// Route JS execution to the right backend:+ /// 1. Apple Events (BrowserJS) — Chrome, Brave, Edge, Safari by bundle ID.+ /// 2. Electron CDP (ElectronJS) — any Electron app detected by bundle presence.+ /// 3. Error — unsupported app type.+ private static func executeJS(+ _ javascript: String,+ bundleId: String,+ pid: Int32,+ windowId: UInt32+ ) async throws -> String {+ // Apple Events path — covers all Chromium browsers and Safari.+ if BrowserJS.supports(bundleId: bundleId) {+ return try await BrowserJS.execute(+ javascript: javascript, bundleId: bundleId, windowId: windowId)+ }+ // Electron CDP path — SIGUSR1 → V8 inspector → Runtime.evaluate.+ if ElectronJS.isElectron(pid: pid) {+ return try await ElectronJS.execute(javascript: javascript, pid: pid)+ }+ throw BrowserJS.Error.unsupportedBrowser(bundleId)+ }++ /// JSON-safe string literal for embedding in JS source.+ private static func jsonString(_ value: String) -> String {+ let escaped = value+ .replacingOccurrences(of: "\\", with: "\\\\")+ .replacingOccurrences(of: "\"", with: "\\\"")+ .replacingOccurrences(of: "\n", with: "\\n")+ .replacingOccurrences(of: "\r", with: "\\r")+ .replacingOccurrences(of: "\t", with: "\\t")+ return "\"\(escaped)\""+ }+}diff --git a/libs/cua-driver/Tests/integration/test_background_focus.py b/libs/cua-driver/Tests/integration/test_background_focus.pyindex 171bd28a8d..dd1674411b 100644--- a/libs/cua-driver/Tests/integration/test_background_focus.py+++ b/libs/cua-driver/Tests/integration/test_background_focus.py@@ -52,6 +52,13 @@ FOCUS_MONITOR_BUNDLE = "com.trycua.FocusMonitorApp"# Helpers# ---------------------------------------------------------------------------+def _tool_text(result: dict) -> str:+ """Extract text from a tool call result's content array."""+ for item in result.get("content", []):+ if item.get("type") == "text":+ return item.get("text", "")+ return ""+def _build_focus_app() -> None:if not os.path.exists(_FOCUS_APP_EXE):subprocess.run(@@ -86,17 +93,23 @@ def _read_focus_losses() -> int:def _open_safari_to_html(client: DriverClient) -> int:- """Open the test HTML page in Safari and return its pid."""- result = client.call_tool("launch_app", {"bundle_id": SAFARI_BUNDLE})- safari_pid = result["structuredContent"]["pid"]+ """Open the test HTML page in Safari (backgrounded) and return its pid.+ Uses launch_app with urls so Safari never becomes frontmost — the+ driver's FocusRestoreGuard clobbers any activate Safari fires during+ document load back to whatever was frontmost before the call.+ """file_url = f"file://{_HTML_PAGE}"- subprocess.run(- ["osascript", "-e", f'tell application "Safari" to open location "{file_url}"'],- check=True,- )+ result = client.call_tool("launch_app", {+ "bundle_id": SAFARI_BUNDLE,+ "urls": [file_url],+ })+ text = _tool_text(result)+ m = re.search(r'pid[=:\s]+(\d+)', text, re.IGNORECASE)+ if not m:+ raise RuntimeError(f"launch_app did not return a pid: {text[:200]}")time.sleep(2.0) # let page load- return safari_pid+ return int(m.group(1))def _get_page_text(client: DriverClient, pid: int) -> str:diff --git a/libs/cua-driver/Tests/integration/test_browser_js.py b/libs/cua-driver/Tests/integration/test_browser_js.pynew file mode 100644index 0000000000..528469894e--- /dev/null+++ b/libs/cua-driver/Tests/integration/test_browser_js.py@@ -0,0 +1,336 @@+"""Integration tests: browser JS primitives (page tool + get_window_state javascript param).++Tests the three new browser primitives:+ - page(action=get_text) — document.body.innerText+ - page(action=query_dom) — querySelectorAll as JSON+ - page(action=execute_javascript)— raw JS with return value+ - get_window_state(javascript=…) — JS result alongside AX snapshot++Setup:+ Requires Chrome to be running with "Allow JavaScript from Apple Events"+ enabled. The test enables it automatically using the Preferences JSON path+ documented in WEB_APPS.md (quits Chrome, writes pref, relaunches).++Run:+ scripts/test.sh test_browser_js+"""++from __future__ import annotations++import glob+import json+import os+import re+import subprocess+import sys+import time+import unittest++sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))+from driver_client import DriverClient, default_binary_path++CHROME_BUNDLE = "com.google.Chrome"+# Simple, stable public page with predictable DOM: one <h1>, one <a>.+TEST_URL = "https://example.com"+# Expected content from example.com.+EXPECTED_H1 = "Example Domain"+EXPECTED_LINK_TEXT = "Learn more"+EXPECTED_HREF_FRAGMENT = "iana.org/domains/example"+++# ---------------------------------------------------------------------------+# Helpers+# ---------------------------------------------------------------------------++def _tool_text(result: dict) -> str:+ """Extract the text string from a tool call result."""+ for item in result.get("content", []):+ if item.get("type") == "text":+ return item.get("text", "")+ return ""+++def _enable_chrome_apple_events() -> None:+ """Quit Chrome, write allow_javascript_apple_events to all profiles, relaunch."""+ subprocess.run(+ ["osascript", "-e", 'quit app "Google Chrome"'],+ check=False, timeout=10,+ )+ time.sleep(1.5)++ for prefs_path in glob.glob(+ os.path.expanduser(+ "~/Library/Application Support/Google/Chrome/*/Preferences"+ )+ ):+ profile = prefs_path.split("/")[-2]+ if "System" in profile or "Guest" in profile:+ continue+ try:+ with open(prefs_path) as f:+ data = json.load(f)+ data.setdefault("browser", {})["allow_javascript_apple_events"] = True+ data.setdefault("account_values", {}).setdefault("browser", {})[+ "allow_javascript_apple_events"+ ] = True+ with open(prefs_path, "w") as f:+ json.dump(data, f)+ except Exception as e:+ print(f" skipped {profile}: {e}")++ subprocess.run(["open", "-a", "Google Chrome"], check=True)+ # 0.8s: test before sync fires so local True is still in effect.+ time.sleep(0.8)+++def _find_chrome(client: DriverClient) -> int:+ """Return Chrome's pid from list_apps text output."""+ text = _tool_text(client.call_tool("list_apps"))+ m = re.search(r"Google Chrome \(pid (\d+)\)", text)+ if not m:+ raise RuntimeError("Chrome not found in list_apps output")+ return int(m.group(1))+++def _main_window(client: DriverClient, pid: int) -> int:+ """Return window_id of the Chrome window that contains example.com."""+ text = _tool_text(client.call_tool("list_windows", {"pid": pid}))+ # Lines look like: - Google Chrome (pid N) "Title" [window_id: NNNN]+ # Prefer a window whose title contains "Example Domain".+ for line in text.splitlines():+ if "Example Domain" in line or "example.com" in line.lower():+ m = re.search(r'\[window_id:\s*(\d+)\]', line)+ if m:+ return int(m.group(1))+ # Fall back to first titled window.+ titled = re.findall(r'"[^"]+"\s+\[window_id:\s*(\d+)\]', text)+ if titled:+ return int(titled[0])+ m = re.search(r'\[window_id:\s*(\d+)\]', text)+ if m:+ return int(m.group(1))+ raise RuntimeError(f"No window found for Chrome pid {pid}")+++# ---------------------------------------------------------------------------+# Test class+# ---------------------------------------------------------------------------++class BrowserJSTests(unittest.TestCase):+ """Browser JS primitives work end-to-end against a live Chrome window."""++ _chrome_pid: int+ _window_id: int++ @classmethod+ def setUpClass(cls) -> None:+ cls.binary = default_binary_path()++ _enable_chrome_apple_events()++ # launch_app opens the URL in a new window WITHOUT stealing focus —+ # the driver's FocusRestoreGuard catches Chrome's activate call.+ with DriverClient(cls.binary) as c:+ result = c.call_tool("launch_app", {+ "bundle_id": CHROME_BUNDLE,+ "urls": [TEST_URL],+ })+ text = _tool_text(result)+ # Extract pid from launch_app response.+ m = re.search(r'pid[=:\s]+(\d+)', text, re.IGNORECASE)+ cls._chrome_pid = int(m.group(1)) if m else _find_chrome(c)++ # Wait for Chrome to load the page and update the window title.+ # Retry up to 8 s — network load time varies.+ deadline = time.monotonic() + 8.0+ cls._window_id = None+ while time.monotonic() < deadline:+ with DriverClient(cls.binary) as c:+ try:+ cls._window_id = _main_window(c, cls._chrome_pid)+ break+ except RuntimeError:+ pass+ time.sleep(0.5)++ if cls._window_id is None:+ raise RuntimeError(+ f"example.com window never appeared for Chrome pid {cls._chrome_pid}"+ )++ print(f"\n Chrome pid={cls._chrome_pid} window_id={cls._window_id}")++ # -----------------------------------------------------------------------+ # page action=execute_javascript+ # -----------------------------------------------------------------------++ def test_01_execute_javascript_arithmetic(self) -> None:+ """execute_javascript returns the JS evaluation result."""+ with DriverClient(self.binary) as c:+ result = c.call_tool("page", {+ "pid": self._chrome_pid,+ "window_id": self._window_id,+ "action": "execute_javascript",+ "javascript": "1 + 1",+ })+ self.assertFalse(result.get("isError"))+ self.assertIn("2", _tool_text(result))++ def test_02_execute_javascript_dom_read(self) -> None:+ """execute_javascript can read DOM content."""+ with DriverClient(self.binary) as c:+ result = c.call_tool("page", {+ "pid": self._chrome_pid,+ "window_id": self._window_id,+ "action": "execute_javascript",+ "javascript": "document.querySelector('h1').innerText",+ })+ self.assertFalse(result.get("isError"))+ self.assertIn(EXPECTED_H1, _tool_text(result))++ def test_03_execute_javascript_iife(self) -> None:+ """execute_javascript handles IIFE with try-catch."""+ with DriverClient(self.binary) as c:+ result = c.call_tool("page", {+ "pid": self._chrome_pid,+ "window_id": self._window_id,+ "action": "execute_javascript",+ "javascript": (+ "(() => { try { return document.title; } "+ "catch(e) { return 'error: ' + e; } })()"+ ),+ })+ self.assertFalse(result.get("isError"))++ # -----------------------------------------------------------------------+ # page action=get_text+ # -----------------------------------------------------------------------++ def test_04_get_text_returns_body_text(self) -> None:+ """get_text returns document.body.innerText containing the H1."""+ with DriverClient(self.binary) as c:+ result = c.call_tool("page", {+ "pid": self._chrome_pid,+ "window_id": self._window_id,+ "action": "get_text",+ })+ self.assertFalse(result.get("isError"))+ self.assertIn(EXPECTED_H1, _tool_text(result))++ def test_05_get_text_includes_link_text(self) -> None:+ """get_text includes anchor text from the page."""+ with DriverClient(self.binary) as c:+ result = c.call_tool("page", {+ "pid": self._chrome_pid,+ "window_id": self._window_id,+ "action": "get_text",+ })+ text = _tool_text(result)+ self.assertIn(EXPECTED_LINK_TEXT, text,+ f"Expected link text in body text, got: {text[:200]!r}")++ # -----------------------------------------------------------------------+ # page action=query_dom+ # -----------------------------------------------------------------------++ def test_06_query_dom_returns_json_array(self) -> None:+ """query_dom returns a JSON array of matching elements with hrefs."""+ with DriverClient(self.binary) as c:+ result = c.call_tool("page", {+ "pid": self._chrome_pid,+ "window_id": self._window_id,+ "action": "query_dom",+ "css_selector": "a[href]",+ "attributes": ["href"],+ })+ self.assertFalse(result.get("isError"))+ text = _tool_text(result)+ self.assertIn("```json", text)+ json_block = text.split("```json")[-1].split("```")[0].strip()+ items = json.loads(json_block)+ self.assertIsInstance(items, list)+ self.assertGreaterEqual(len(items), 1)+ hrefs = {item.get("href") or "" for item in items}+ self.assertTrue(+ any(EXPECTED_HREF_FRAGMENT in h for h in hrefs),+ f"Expected href containing '{EXPECTED_HREF_FRAGMENT}', got: {hrefs}",+ )++ def test_07_query_dom_h1_no_attributes(self) -> None:+ """query_dom without attributes returns tag and text."""+ with DriverClient(self.binary) as c:+ result = c.call_tool("page", {+ "pid": self._chrome_pid,+ "window_id": self._window_id,+ "action": "query_dom",+ "css_selector": "h1",+ })+ self.assertFalse(result.get("isError"))+ text = _tool_text(result)+ json_block = text.split("```json")[-1].split("```")[0].strip()+ items = json.loads(json_block)+ self.assertEqual(len(items), 1)+ self.assertEqual(items[0]["tag"], "h1")+ self.assertIn(EXPECTED_H1, items[0]["text"])++ # -----------------------------------------------------------------------+ # get_window_state javascript param+ # -----------------------------------------------------------------------++ def test_08_get_window_state_javascript_co_located(self) -> None:+ """get_window_state with javascript= returns JS result alongside AX tree."""+ with DriverClient(self.binary) as c:+ result = c.call_tool("get_window_state", {+ "pid": self._chrome_pid,+ "window_id": self._window_id,+ "javascript": "document.title",+ })+ self.assertFalse(result.get("isError"))+ text = _tool_text(result)+ self.assertIn("## JavaScript result", text)+ # AX tree content is also present.+ self.assertRegex(text, r"elements.*turn", re.IGNORECASE)++ def test_09_get_window_state_javascript_error_is_inline(self) -> None:+ """JS runtime errors in get_window_state appear inline, tool does not isError."""+ # Chrome returns "missing value" for JS that throws — osascript exits 0.+ # The result section appears but may contain "missing value" or an error msg.+ with DriverClient(self.binary) as c:+ result = c.call_tool("get_window_state", {+ "pid": self._chrome_pid,+ "window_id": self._window_id,+ "javascript": "undefined_var_that_does_not_exist",+ })+ # isError must be False — JS failures must not kill the whole tool call.+ self.assertFalse(result.get("isError"))+ text = _tool_text(result)+ # The JS result section must always appear.+ self.assertIn("## JavaScript result", text)++ # -----------------------------------------------------------------------+ # Error cases+ # -----------------------------------------------------------------------++ def test_10_page_execute_missing_javascript_field(self) -> None:+ """page execute_javascript returns isError when javascript field is absent."""+ with DriverClient(self.binary) as c:+ result = c.call_tool("page", {+ "pid": self._chrome_pid,+ "window_id": self._window_id,+ "action": "execute_javascript",+ })+ self.assertTrue(result.get("isError"))++ def test_11_page_query_dom_missing_selector(self) -> None:+ """page query_dom returns isError when css_selector is absent."""+ with DriverClient(self.binary) as c:+ result = c.call_tool("page", {+ "pid": self._chrome_pid,+ "window_id": self._window_id,+ "action": "query_dom",+ })+ self.assertTrue(result.get("isError"))+++if __name__ == "__main__":+ unittest.main(verbosity=2)diff --git a/libs/cua-driver/Tests/integration/test_chrome_minimized_nav.py b/libs/cua-driver/Tests/integration/test_chrome_minimized_nav.pyindex 7f849daad5..0a5816ea2b 100644--- a/libs/cua-driver/Tests/integration/test_chrome_minimized_nav.py+++ b/libs/cua-driver/Tests/integration/test_chrome_minimized_nav.py@@ -149,18 +149,12 @@ class ChromeMinimizedNavTests(unittest.TestCase):time.sleep(2.0)print(f"\n Chrome pid: {cls._chrome_pid}")- # Ensure Chrome is visible with a window.- subprocess.run(- ["osascript", "-e",- 'tell application "Google Chrome"\n'- ' activate\n'- ' if (count of windows) is 0 then\n'- ' make new window\n'- ' set URL of active tab of window 1 to "about:blank"\n'- ' end if\n'- 'end tell'],- check=False, timeout=10,- )+ # Ensure Chrome has an about:blank window via launch_app (no focus steal).+ with DriverClient(cls.binary) as c:+ c.call_tool("launch_app", {+ "bundle_id": CHROME_BUNDLE,+ "urls": ["about:blank"],+ })time.sleep(2.0)# Snapshot AX tree BEFORE minimizing to cache the omnibox index.@@ -219,17 +213,13 @@ class ChromeMinimizedNavTests(unittest.TestCase):passdef setUp(self) -> None:- # Reliably minimize Chrome: activate, wait, Cmd+M, wait.- subprocess.run(- ["osascript", "-e", 'tell application "Google Chrome" to activate'],- check=False, timeout=5,- )- time.sleep(1.0)- subprocess.run(- ["osascript", "-e",- 'tell application "System Events" to keystroke "m" using command down'],- check=False, timeout=5,- )+ # Minimize Chrome via hotkey — cua-driver delivers Cmd+M to the pid+ # via CGEvent.postToPid without stealing focus.+ with DriverClient(self.binary) as c:+ c.call_tool("hotkey", {+ "pid": self._chrome_pid,+ "keys": ["cmd", "m"],+ })time.sleep(1.5)_activate_focus_monitor()self._losses_before = _read_focus_losses()