Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): prefer semantic routes before GUI fallback

Francesco Bonacci6f2faf9f9c0e
diff --git a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
index 5b2407c459..4f080fac95 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
@@ -65,14 +65,27 @@ Use whichever combination matches the host. When in doubt, run
`cua-driver doctor` — it reports the platform and the right entry
point.
-## Choose a semantic system operation before a GUI path
-
-Before opening or operating an application, classify the requested outcome. If
-the caller already has a purpose-built API, CLI, or filesystem operation that
-can complete a non-GUI outcome directly, use it before Cua Driver. File moves,
-renames, copies, directory creation, archive extraction, data conversion, and
-process inspection should not be performed by opening Finder, Explorer, or a
-Linux file manager and imitating a user.
+## Start with the narrowest semantic route
+
+Before opening or operating an application, name the desired postcondition and
+use the first applicable route below. Verify the result in the same domain
+before stopping or advancing:
+
+0. **Caller-provided headless/background operation for a non-GUI outcome.**
+ Prefer an exact application API/SDK, service or database client, CLI, or
+ filesystem operation over imitating a user. This includes batch-safe file
+ moves, renames, copies, directory creation, archive extraction, data
+ conversion, and process inspection. Read the resulting semantic state back;
+ a zero exit status alone is not proof.
+1. **Typed Cua operation for an application or window outcome.** Use
+ `set_window_frame` for exact geometry, typed browser tools for supported page
+ content, and clipboard tools for clipboard state. Verify with
+ `list_windows`, `get_browser_state`, or `clipboard_read`, respectively.
+2. **Background accessibility action.** Use a fresh AX/UIA/AT-SPI target.
+3. **Background pixel action.** Use the pixels from the same state snapshot.
+4. **Foreground delivery.** Retry only the action that evidence says could not
+ land in the background.
+5. **Desktop fallback.** Enter this explicit, one-way session phase last.
Use Cua Driver when the outcome lives in an application's UI or window state,
or when the user explicitly asks to operate that GUI. Once the task crosses
@@ -458,7 +471,13 @@ the tree against the pixels you already have, and only change
_dispatch rung_ on a real signal. Walk the rungs:
```
-# Rung 1 — element ax action, backgrounded (the cheap default)
+# Routes 0–1 — resolve non-GUI, exact geometry, and supported page outcomes first
+# Use a caller-provided semantic operation for a non-GUI outcome, then read it back.
+# For exact window geometry: set_window_frame(...), then list_windows(...) readback.
+# For supported page content: get_browser_state(...), typed browser action, refresh refs.
+# Continue below only when the postcondition actually requires native UI interaction.
+
+# Route 2 — element AX/UIA/AT-SPI action, backgrounded
get_window_state(pid, window_id) # tree + screenshot, both, always
resp = click(pid, window_id, element_index) # or type_text / set_value / press_key
check = verify_state( # bounded structured read-back
@@ -481,20 +500,13 @@ if resp.effect == "suspected_noop"
or check.status != "satisfied"
or the tree looks wrong vs the screenshot: # e.g. an h:1 / off-viewport row
- # Rung 2 — element px action off the SAME screenshot
+ # Route 3 — element px action off the SAME screenshot
pick the target pixel from the screenshot already in the response
click(pid, x, y) # background pixel — still no foreground
verify_state(..., include_screenshot=true)
if it landed: done
-# Rung 2b — exact browser page tools, when get_browser_state can bind this window
-# Use typed browser refs for page content; native window tools still handle chrome.
-get_browser_state(session, pid, window_id)
-browser_click(session, target_id, tab_id, ref) # or browser_type; see BROWSER.md
-get_browser_state(session, pid, window_id) # verify with fresh refs
-if it landed: done
-
-# Rung 3 — background delivery was dropped (insert/click never arrived)
+# Route 4 — background delivery was dropped (insert/click never arrived)
if resp.escalation.target == "foreground"
or the px action still did nothing:
re-call the same action with delivery_mode:"foreground"
@@ -502,9 +514,9 @@ if resp.escalation.target == "foreground"
# unfocused window there; see LINUX.md
verify again
-# Rung 4 — desktop fallback (auto sessions only, explicit and one-way)
-# Reach this only after AX, window-pixel, browser-page (when available), and
-# foreground-window delivery have all been exhausted and verified ineffective.
+# Route 5 — desktop fallback (auto sessions only, explicit and one-way)
+# Reach this only after semantic, AX, window-pixel, and foreground-window
+# delivery have all been exhausted and verified ineffective.
escalate_session(session,
reason="foreground_ineffective", # or another advertised reason
detail="bounded non-sensitive summary")
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/protocol.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/protocol.rs
index 36efce1be7..8f5f1c72ae 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/protocol.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/protocol.rs
@@ -369,18 +369,17 @@ fn agent_instructions() -> String {
format!(
r#"cua-driver: cross-platform background computer-use automation.
-Tools operate apps without stealing focus or moving the real pointer. Prefer `element_index` ({tree_kind}) over pixels; it works on backgrounded or hidden windows.
+Before starting UI work, classify the desired postcondition. For a non-GUI outcome, prefer a client-provided app API/SDK, headless/background interface, CLI, or filesystem operation and read the result back in that semantic domain. This server has no shell.
-Prefer any client-provided API, CLI, or filesystem operation that completes a non-GUI outcome directly; do not open a file manager for ordinary file operations. This MCP server has no shell.
+For an app or window outcome, use the narrowest semantic Cua route first: `set_window_frame` plus `list_windows` readback for geometry, typed browser tools for supported page content, and clipboard tools for clipboard state. Then climb through background `element_index` ({tree_kind}), background pixels, foreground delivery, and desktop fallback. Never advance on transport success alone.
Workflow per turn:
-0. `start_session(session)` once per run; reuse that id on actions. Concurrent runs use distinct ids. End it when done.
-1. `launch_app` returns pid and windows. Use `creates_new_application_instance:true` when another run may touch the app.
-2. `get_window_state(pid, window_id)` refreshes the tree and element indices.
-3. Act with the fresh index.
-4. `verify_state(pid, window_id, expect)` checks bounded structured postconditions. `unknown` is not success. Set `include_screenshot:true` when the multimodal agent should also read visual evidence and decide whether to stop, retry, or advance the ladder.
+0. `start_session(session)` once; reuse that id and end it when done.
+1. `launch_app`, then `get_window_state(pid, window_id)` to refresh element indices.
+2. Act with the fresh index.
+3. `verify_state(pid, window_id, expect)` checks bounded postconditions. `unknown` is not success; `include_screenshot:true` lets the multimodal agent judge visual evidence.
-If a `cua-driver` skill is loaded in your harness (Claude Code / Codex / OpenClaw / OpenCode dirs), prefer its detailed workflow — SKILL.md plus {platform_skill_pointer}. Install with `cua-driver skills install` if not yet present."#
+If the `cua-driver` skill is loaded, follow SKILL.md plus {platform_skill_pointer}."#
)
}
@@ -474,8 +473,18 @@ mod agent_instruction_tests {
assert!(instructions.contains("verify_state"));
assert!(instructions.contains("`unknown` is not success"));
assert!(instructions.contains("multimodal agent"));
- assert!(instructions.contains("client-provided API, CLI, or filesystem operation"));
+ assert!(instructions.contains("client-provided app API/SDK"));
+ assert!(instructions.contains("headless/background interface"));
+ assert!(instructions.contains("read the result back in that semantic domain"));
+ assert!(instructions.contains("narrowest semantic Cua route first"));
+ assert!(instructions.contains("`set_window_frame` plus `list_windows` readback"));
+ assert!(instructions.contains("typed browser tools for supported page content"));
assert!(instructions.contains("has no shell"));
+ assert!(
+ instructions.find("client-provided app API/SDK")
+ < instructions.find("background `element_index`"),
+ "semantic/headless operations must precede native UI dispatch"
+ );
assert!(
instructions.split_whitespace().count() <= 200,
"initialize instructions should stay within the documented context budget"