Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver): prefer semantic routes before GUI fallback
Francesco Bonacci6f2faf9f9c0e
diff --git a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md b/libs/cua-driver/rust/Skills/cua-driver/SKILL.mdindex 5b2407c459..4f080fac95 100644--- a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md+++ b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md@@ -65,14 +65,27 @@ Use whichever combination matches the host. When in doubt, run`cua-driver doctor` — it reports the platform and the right entrypoint.-## Choose a semantic system operation before a GUI path--Before opening or operating an application, classify the requested outcome. If-the caller already has a purpose-built API, CLI, or filesystem operation that-can complete a non-GUI outcome directly, use it before Cua Driver. File moves,-renames, copies, directory creation, archive extraction, data conversion, and-process inspection should not be performed by opening Finder, Explorer, or a-Linux file manager and imitating a user.+## Start with the narrowest semantic route++Before opening or operating an application, name the desired postcondition and+use the first applicable route below. Verify the result in the same domain+before stopping or advancing:++0. **Caller-provided headless/background operation for a non-GUI outcome.**+ Prefer an exact application API/SDK, service or database client, CLI, or+ filesystem operation over imitating a user. This includes batch-safe file+ moves, renames, copies, directory creation, archive extraction, data+ conversion, and process inspection. Read the resulting semantic state back;+ a zero exit status alone is not proof.+1. **Typed Cua operation for an application or window outcome.** Use+ `set_window_frame` for exact geometry, typed browser tools for supported page+ content, and clipboard tools for clipboard state. Verify with+ `list_windows`, `get_browser_state`, or `clipboard_read`, respectively.+2. **Background accessibility action.** Use a fresh AX/UIA/AT-SPI target.+3. **Background pixel action.** Use the pixels from the same state snapshot.+4. **Foreground delivery.** Retry only the action that evidence says could not+ land in the background.+5. **Desktop fallback.** Enter this explicit, one-way session phase last.Use Cua Driver when the outcome lives in an application's UI or window state,or when the user explicitly asks to operate that GUI. Once the task crosses@@ -458,7 +471,13 @@ the tree against the pixels you already have, and only change_dispatch rung_ on a real signal. Walk the rungs:```-# Rung 1 — element ax action, backgrounded (the cheap default)+# Routes 0–1 — resolve non-GUI, exact geometry, and supported page outcomes first+# Use a caller-provided semantic operation for a non-GUI outcome, then read it back.+# For exact window geometry: set_window_frame(...), then list_windows(...) readback.+# For supported page content: get_browser_state(...), typed browser action, refresh refs.+# Continue below only when the postcondition actually requires native UI interaction.++# Route 2 — element AX/UIA/AT-SPI action, backgroundedget_window_state(pid, window_id) # tree + screenshot, both, alwaysresp = click(pid, window_id, element_index) # or type_text / set_value / press_keycheck = verify_state( # bounded structured read-back@@ -481,20 +500,13 @@ if resp.effect == "suspected_noop"or check.status != "satisfied"or the tree looks wrong vs the screenshot: # e.g. an h:1 / off-viewport row- # Rung 2 — element px action off the SAME screenshot+ # Route 3 — element px action off the SAME screenshotpick the target pixel from the screenshot already in the responseclick(pid, x, y) # background pixel — still no foregroundverify_state(..., include_screenshot=true)if it landed: done-# Rung 2b — exact browser page tools, when get_browser_state can bind this window-# Use typed browser refs for page content; native window tools still handle chrome.-get_browser_state(session, pid, window_id)-browser_click(session, target_id, tab_id, ref) # or browser_type; see BROWSER.md-get_browser_state(session, pid, window_id) # verify with fresh refs-if it landed: done--# Rung 3 — background delivery was dropped (insert/click never arrived)+# Route 4 — background delivery was dropped (insert/click never arrived)if resp.escalation.target == "foreground"or the px action still did nothing:re-call the same action with delivery_mode:"foreground"@@ -502,9 +514,9 @@ if resp.escalation.target == "foreground"# unfocused window there; see LINUX.mdverify again-# Rung 4 — desktop fallback (auto sessions only, explicit and one-way)-# Reach this only after AX, window-pixel, browser-page (when available), and-# foreground-window delivery have all been exhausted and verified ineffective.+# Route 5 — desktop fallback (auto sessions only, explicit and one-way)+# Reach this only after semantic, AX, window-pixel, and foreground-window+# delivery have all been exhausted and verified ineffective.escalate_session(session,reason="foreground_ineffective", # or another advertised reasondetail="bounded non-sensitive summary")diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/protocol.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/protocol.rsindex 36efce1be7..8f5f1c72ae 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/protocol.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/protocol.rs@@ -369,18 +369,17 @@ fn agent_instructions() -> String {format!(r#"cua-driver: cross-platform background computer-use automation.-Tools operate apps without stealing focus or moving the real pointer. Prefer `element_index` ({tree_kind}) over pixels; it works on backgrounded or hidden windows.+Before starting UI work, classify the desired postcondition. For a non-GUI outcome, prefer a client-provided app API/SDK, headless/background interface, CLI, or filesystem operation and read the result back in that semantic domain. This server has no shell.-Prefer any client-provided API, CLI, or filesystem operation that completes a non-GUI outcome directly; do not open a file manager for ordinary file operations. This MCP server has no shell.+For an app or window outcome, use the narrowest semantic Cua route first: `set_window_frame` plus `list_windows` readback for geometry, typed browser tools for supported page content, and clipboard tools for clipboard state. Then climb through background `element_index` ({tree_kind}), background pixels, foreground delivery, and desktop fallback. Never advance on transport success alone.Workflow per turn:-0. `start_session(session)` once per run; reuse that id on actions. Concurrent runs use distinct ids. End it when done.-1. `launch_app` returns pid and windows. Use `creates_new_application_instance:true` when another run may touch the app.-2. `get_window_state(pid, window_id)` refreshes the tree and element indices.-3. Act with the fresh index.-4. `verify_state(pid, window_id, expect)` checks bounded structured postconditions. `unknown` is not success. Set `include_screenshot:true` when the multimodal agent should also read visual evidence and decide whether to stop, retry, or advance the ladder.+0. `start_session(session)` once; reuse that id and end it when done.+1. `launch_app`, then `get_window_state(pid, window_id)` to refresh element indices.+2. Act with the fresh index.+3. `verify_state(pid, window_id, expect)` checks bounded postconditions. `unknown` is not success; `include_screenshot:true` lets the multimodal agent judge visual evidence.-If a `cua-driver` skill is loaded in your harness (Claude Code / Codex / OpenClaw / OpenCode dirs), prefer its detailed workflow — SKILL.md plus {platform_skill_pointer}. Install with `cua-driver skills install` if not yet present."#+If the `cua-driver` skill is loaded, follow SKILL.md plus {platform_skill_pointer}."#)}@@ -474,8 +473,18 @@ mod agent_instruction_tests {assert!(instructions.contains("verify_state"));assert!(instructions.contains("`unknown` is not success"));assert!(instructions.contains("multimodal agent"));- assert!(instructions.contains("client-provided API, CLI, or filesystem operation"));+ assert!(instructions.contains("client-provided app API/SDK"));+ assert!(instructions.contains("headless/background interface"));+ assert!(instructions.contains("read the result back in that semantic domain"));+ assert!(instructions.contains("narrowest semantic Cua route first"));+ assert!(instructions.contains("`set_window_frame` plus `list_windows` readback"));+ assert!(instructions.contains("typed browser tools for supported page content"));assert!(instructions.contains("has no shell"));+ assert!(+ instructions.find("client-provided app API/SDK")+ < instructions.find("background `element_index`"),+ "semantic/headless operations must precede native UI dispatch"+ );assert!(instructions.split_whitespace().count() <= 200,"initialize instructions should stay within the documented context budget"