Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
docs(cua-driver-rs)(skill)(macos): verify with `som` + Catalyst/Electron clipboard paste workaround (#1717)
Francesco Bonaccid680f12159ae
diff --git a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md b/libs/cua-driver/rust/Skills/cua-driver/MACOS.mdindex 7be50ebf03..6048621abc 100644--- a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md+++ b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md@@ -172,6 +172,33 @@ genuinely needs frontmost state, fall through to the activatefallback. Always name the focus steal in your response ("I'llbriefly bring Chrome to the front because …").+### Verifying actions: prefer `som` over `ax`++When verifying that an action **landed** (a click registered, a text+input took, a menu pick changed selection), default to `get_window_state({+capture_mode: "som"})`. `"ax"` is cheaper (no PNG in the response), but+the AX tree **lies** for two common surface types:++- **Canvas-backed editors** — Monaco (VSCode, Cursor), xterm, Figma,+ WebGL apps. The AX tree shows the surrounding chrome but reports+ nothing for the canvas content. A snapshot can look unchanged after a+ successful text edit.+- **Catalyst / iOS-on-Mac text views** — see "Known text-input limits"+ above. The AX `AXValue` can lag the rendered pixels, or report the+ placeholder string when the field is actually populated.++`"som"` returns both the AX tree AND a PNG screenshot in a single call.+Pay the extra tokens; the screenshot is the ground truth for both of+the cases above, and you avoid the "type → AX-check succeeds → think+the action landed → discover the lie three calls later" loop.++Rule of thumb:+- **`som`** — verifying an action landed (default).+- **`ax`** — cheap element lookup before a click (you don't need the+ pixels, you need the `[N]` index).+- **`vision`** — pure visual inspection (e.g. asking the model to read+ a chart). Skip the AX walk entirely.+### Self-check patternBefore every `Bash` call whose command line touches any macOS app@@ -293,6 +320,38 @@ There is no backgrounded path that reaches these apps today.AX-addressable elements (links, buttons, toolbar items), use`right_click({pid, element_index})` instead.+### Known text-input limits (Catalyst + Electron)++`type_text` returns `✅ Inserted N char(s)` against **Catalyst** apps+(WhatsApp, Reminders, Notes-via-Catalyst, anything in `/Applications`+that's actually `iOSAppOnMac.app`) and **Electron** apps (VSCode's+Monaco editor, Slack composer, Discord, Linear) but the characters+don't reach the rendered text view. The AX path (`AXSetAttribute` on+`kAXSelectedText`) reports success because the API call succeeds on+the AX shim, but the UIKit/Chromium text view that owns the actual+input doesn't observe the AX value change. The CGEvent fallback path+runs, but the renderer's focused-process check drops the keystrokes+when the app isn't WindowServer-frontmost.++Workaround — clipboard + windowed `Cmd+V`:++1. Put the string on the macOS clipboard externally (e.g. shell+ `pbcopy`, or `NSPasteboard.general.setString:forType:` from your+ own helper).+2. Dispatch `hotkey({pid, window_id, keys: ["cmd", "v"]})`. **The+ `window_id` matters**: the NSMenu key-equivalent path briefly+ foregrounds the app via `SLPSSetFrontProcessWithOptions(kCPSNoWindows)`+ for ~1 ms — enough for `paste:` to deliver — then restores prior+ frontmost. Without `window_id`, the auth-message envelope path+ runs and hits the same drop as `type_text`.+3. Verify with a `get_window_state({capture_mode: "som"})` snapshot.+ Both the AX value and the rendered pixels should show the pasted+ text. (See the verify-with-som pattern below.)++This costs one round-trip of clipboard pollution. Restore the user's+prior clipboard if your workflow cares — store via `pbpaste` before,+write back via `pbcopy` after.+## Navigating native menu bars (AXMenuBar)**Only drive the menu bar when the target app is frontmost.** This