Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver): keep a failed run_actions report readable within client error limits (#4897)
Francesco Bonacci5a364bbe60e1
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/batch.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/batch.mdxindex 7e23b2a488..f10fcda7ff 100644--- a/docs/content/docs/cua-driver/reference/mcp-tools/batch.mdx+++ b/docs/content/docs/cua-driver/reference/mcp-tools/batch.mdx@@ -16,7 +16,7 @@ Served by `cua-driver mcp`; see [MCP tools](/cua-driver/reference/mcp-tools) forThe default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag, move_cursor and `args` are exactly that tool's arguments. `pid` and `window_id` (or `app`/`window`) on run_actions itself are the default window of every step that names none. Reads (get_window_state, zoom) do not run between steps: one there is skipped; a get_window_state last is the `observe`, a zoom last runs after it. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or target later steps by role/name. A token that went stale is retried once on the element with the same role and label, if exactly one matches.-Returns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step (after a failure caused by a dialog holding focus, that dialog is read, even without `observe`). A step refused because its app ignores background input says how to re-send it; `foreground_fallback:true` retries such steps in the foreground once (the window is activated). That read is a `since:"latest"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.+Returns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step (after a failure caused by a dialog holding focus, that dialog is read, even without `observe`). A step refused because its app ignores background input says how to re-send it; `foreground_fallback:true` retries such steps in the foreground once (the window is activated). That read is a `since:"latest"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 64 steps; `delivery_mode` on run_actions is the default for its steps.TARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{"click": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.@@ -31,11 +31,12 @@ Effect: destructive. Platforms: macOS, Linux, Windows.| --- | --- | --- | --- || <span id="run_actions--app"></span>`app` | `string` | | Default app (name or bundle id) for steps that name no window. || <span id="run_actions--delay_ms"></span>`delay_ms` | `integer` | | Pause between steps in milliseconds (not after the last). Default 0. Range: 0 to 2000. |+| <span id="run_actions--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | | Default delivery_mode for steps whose tool takes one and that set none. || <span id="run_actions--foreground_fallback"></span>`foreground_fallback` | `boolean` | | When a step is refused because its target ignores background input (Electron, LibreOffice and other toolkits), retry it once with delivery_mode:"foreground": the window is activated and the pointer may move. Default false. || <span id="run_actions--observe"></span>`observe` | `object \| boolean` | | Optional end-of-batch observation: `true`, or arguments for ONE get_window_state call. `pid` and `window_id` default to those of the last step that names both. Defaults to since="latest" (only what changed since your last read of that window with the same query/max_elements/max_depth; a full read when there is none), include_screenshot=false and max_elements=250; pass include_screenshot=true to see the window, or since=null for a full read. Omit to read nothing. || <span id="run_actions--pid"></span>`pid` | `integer` | | Default pid for steps that name no window (a step's own pid/window_id/app/window wins). || <span id="run_actions--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |-| <span id="run_actions--steps"></span>`steps` | `object[]` | required | Ordered actions. Execution stops at the first failing step. Each step is {tool, args} or {<tool>: args}, plus optional wait_for / expect / timeout_ms; a step may also be only a wait_for or expect check. Items: 1 to 32. |+| <span id="run_actions--steps"></span>`steps` | `object[]` | required | Ordered actions. Execution stops at the first failing step. Each step is {tool, args} or {<tool>: args}, plus optional wait_for / expect / timeout_ms; a step may also be only a wait_for or expect check. Items: 1 to 64. || <span id="run_actions--window"></span>`window` | `string` | | Default window title substring for steps that name no window. || <span id="run_actions--window_id"></span>`window_id` | `integer` | | Default window_id for steps that name no window, with `pid`. |diff --git a/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md b/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.mdindex a3a16754bc..c691be6439 100644--- a/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md+++ b/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md@@ -90,7 +90,7 @@ cua-driver run_actions '{"session":"run-1","steps":[],"delay_ms":100,"observe":true}'```-- `tool` is one of `click`, `double_click`, `right_click`, `set_value`, `type_text`, `press_key`, `hotkey`, `scroll`, `drag`, `move_cursor` (`press` and `type` are accepted for `press_key` and `type_text`, `triple_click` is a click with `count:3`, and a key name such as `down` as the tool is a `press_key`); `args` are exactly that tool's arguments. Observation tools do not run inside a batch: a read between actions is skipped (it could not change the later steps), a `get_window_state` as the last step is the observation (merged with `observe`), and a trailing `zoom` runs after it. Run `describe run_actions` and `describe <tool>` for schemas. Up to 32 steps.+- `tool` is one of `click`, `double_click`, `right_click`, `set_value`, `type_text`, `press_key`, `hotkey`, `scroll`, `drag`, `move_cursor` (`press` and `type` are accepted for `press_key` and `type_text`, `triple_click` is a click with `count:3`, and a key name such as `down` as the tool is a `press_key`); `args` are exactly that tool's arguments. Observation tools do not run inside a batch: a read between actions is skipped (it could not change the later steps), a `get_window_state` as the last step is the observation (merged with `observe`), and a trailing `zoom` runs after it. Run `describe run_actions` and `describe <tool>` for schemas. Up to 64 steps. `delivery_mode` on `run_actions` is the default for steps that set none.- Name the window once: `pid` and `window_id` (or `app`/`window`) on `run_actions` itself are the default for every step that names none.- A step whose `element_token` went stale because an earlier step re-rendered the window is retried once on the element with the same role and label in a fresh read, when exactly one matches; the step's line says so.- Every step is validated before the first runs, so a malformed step changes nothing. Each step then passes the same session, permission, capability-manifest and approval checks as a direct call; a batch grants nothing a single call lacks, and a refused step ends the batch like any other failure.diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rsindex b82d69f90d..9420dc7558 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs@@ -43,7 +43,7 @@ pub const BATCHABLE_TOOLS: &[&str] = &[];/// Most steps one call may carry. Keeps a batch inside client call timeouts.-pub const MAX_STEPS: usize = 32;+pub const MAX_STEPS: usize = 64;/// Longest pause between two steps.pub const MAX_DELAY_MS: u64 = 2_000;/// Default `max_elements` for the end-of-batch observation: the same budget as@@ -189,7 +189,7 @@ impl Tool for RunActionsTool {since your last read of the window, plus a new snapshot_id. Without \`observe` nothing is read. The batch uses one session: steps may \omit `session` or repeat the batch's. `delay_ms` pauses between steps (max \- 2000). At most 32 steps.\n\n\+ 2000). At most 64 steps; `delivery_mode` on run_actions is the default for its steps.\n\n\TARGET BY NAME (no earlier read needed, so one batch can cross screens): \a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and \its args may name the target instead of carrying a token: `role` (button, \@@ -277,6 +277,11 @@ impl Tool for RunActionsTool {"type": "string","description": "Default window title substring for steps that name no window."},+ "delivery_mode": {+ "type": "string",+ "enum": ["background", "foreground"],+ "description": "Default delivery_mode for steps whose tool takes one and that set none."+ },"foreground_fallback": {"type": "boolean","description": "When a step is refused because its target ignores background input (Electron, LibreOffice and other toolkits), retry it once with delivery_mode:\"foreground\": the window is activated and the pointer may move. Default false."@@ -458,6 +463,17 @@ impl Plan {.map(str::to_owned);let default_window = batch_window(object)?;+ let default_delivery = match object.get("delivery_mode") {+ None | Some(Value::Null) => None,+ Some(Value::String(mode)) if matches!(mode.as_str(), "background" | "foreground") => {+ Some(mode.clone())+ }+ Some(_) => {+ return Err(PlanError::batch(+ "`delivery_mode` must be \"background\" or \"foreground\"",+ ))+ }+ };let foreground_fallback = match object.get("foreground_fallback") {None | Some(Value::Null) => false,Some(Value::Bool(flag)) => *flag,@@ -537,7 +553,14 @@ impl Plan {});continue;}- let step = parse_step(registry, index, raw, session.as_deref(), has_window)?;+ let mut step = parse_step(registry, index, raw, session.as_deref(), has_window)?;+ if let (Some(mode), Some(action)) = (&default_delivery, step.action.as_mut()) {+ if accepts(registry, action.tool, "delivery_mode") {+ if let Some(object) = action.args.as_object_mut() {+ object.entry("delivery_mode").or_insert_with(|| json!(mode));+ }+ }+ }let names_window = step.action.as_ref()@@ -801,6 +824,9 @@ impl Plan {let mut content = vec![Content::text(lines.join("\n"))];content.append(&mut observation_content);content.append(&mut tail_content);+ if failed_step.is_some() {+ fit_error_text(&mut content, ERROR_TEXT_BUDGET);+ }ToolResult {content,is_error: failed_step.map(|_| true),@@ -1278,6 +1304,41 @@ fn focus_holder(message: &str) -> Option<u64> {digits.parse().ok()}+/// Most text characters a failed batch returns. MCP clients cut long error+/// results to their first line or so (Claude Code keeps ~100 characters of+/// an error over ~4 000), which hid why a step failed behind the read that+/// followed it. The step report goes first and whole; the read is trimmed.+const ERROR_TEXT_BUDGET: usize = 3_500;++/// Keep the text of a failed batch within `budget` characters: the first+/// part (the step report) up to the budget, then later text parts trimmed+/// to what is left, with a note. Images are kept.+fn fit_error_text(content: &mut Vec<Content>, budget: usize) {+ let mut left = budget;+ let mut trimmed = false;+ for part in content.iter_mut() {+ let Content::Text { text, .. } = part else {+ continue;+ };+ let length = text.chars().count();+ if length <= left {+ left -= length;+ continue;+ }+ let keep: String = text.chars().take(left).collect();+ *text = keep;+ left = 0;+ trimmed = true;+ }+ content.retain(|part| !matches!(part, Content::Text { text, .. } if text.is_empty()));+ if trimmed {+ content.push(Content::text(+ "[the rest of this read was cut so the step report stays readable; call \+ get_window_state for the full window]",+ ));+ }+}+/// Whether a refusal names foreground delivery as the route that works:/// the target ignores background input (Electron, LibreOffice VCL, ...).fn wants_foreground(result: &ToolResult) -> bool {diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/reliability_tests.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/reliability_tests.rsindex faca14d92f..508ea4bb39 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/reliability_tests.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/reliability_tests.rs@@ -152,6 +152,69 @@ async fn a_step_refused_for_background_delivery_says_how_or_falls_back() {assert_eq!(harness.last("click")["delivery_mode"], "foreground");}+#[test]+fn a_failed_batch_keeps_the_step_report_readable() {+ // v037c: a failed batch returned its step report plus a full read+ // (~4 100 characters); the client cut it to "1. click ok: ... AXCh",+ // hiding why step 2 failed.+ let report =+ "run_actions: step 2 of 2 failed\n1. click ok\n2. click ERROR (action): why".to_owned();+ let mut content = vec![+ crate::protocol::Content::text(report.clone()),+ crate::protocol::Content::text("x".repeat(6_000)),+ ];+ super::super::fit_error_text(&mut content, super::super::ERROR_TEXT_BUDGET);+ let text: String = content+ .iter()+ .filter_map(|part| match part {+ crate::protocol::Content::Text { text, .. } => Some(text.as_str()),+ _ => None,+ })+ .collect();+ assert!(text.starts_with(&report));+ assert!(text.chars().count() <= super::super::ERROR_TEXT_BUDGET + 200);+ assert!(text.contains("call get_window_state for the full window"));+}++#[tokio::test]+async fn a_failed_batch_with_a_long_read_stays_within_the_budget() {+ let harness = Harness::new();+ let result = harness+ .run(json!({"steps": [+ {"tool": "click", "args": {"pid": 7, "window_id": 3}},+ {"tool": "click", "args": {"pid": 7, "window_id": 3, "fail": true}}+ ], "observe": true}))+ .await;+ assert_eq!(result.is_error, Some(true));+ assert!(+ text(&result).contains("2. click ERROR"),+ "{}",+ text(&result)+ );+ assert!(text(&result).chars().count() <= super::super::ERROR_TEXT_BUDGET + 200);+}++#[tokio::test]+async fn batch_level_delivery_mode_is_the_steps_default() {+ let harness = Harness::new();+ // v037c: `delivery_mode` on run_actions itself was refused.+ let result = harness+ .run(+ json!({"pid": 7, "window_id": 3, "delivery_mode": "foreground", "steps": [+ {"tool": "click", "args": {"x": 1, "y": 2}},+ {"tool": "click", "args": {"x": 1, "y": 2, "delivery_mode": "background"}}+ ]}),+ )+ .await;+ assert_ne!(result.is_error, Some(true), "{}", text(&result));+ let calls = harness.calls["click"].lock().unwrap().clone();+ assert_eq!(calls[0]["delivery_mode"], "foreground");+ assert_eq!(+ calls[1]["delivery_mode"], "background",+ "a step's own mode wins"+ );+}+// ── read steps ───────────────────────────────────────────────────────────#[tokio::test]diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/protocol_schema_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/protocol_schema_test.rsindex 5e3d5de306..8d37775678 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/protocol_schema_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/protocol_schema_test.rs@@ -190,6 +190,8 @@ fn tools_list_schema_shape() {// Chromium); foreground accepts that."browser_click","browser_pointer",+ // The batch's default for its steps' own delivery_mode.+ "run_actions",// macOS set_value has no delivery ladder.#[cfg(any(target_os = "linux", target_os = "windows"))]"set_value",diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.jsonindex 0171de6428..d9fe4f9a47 100644--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json@@ -3471,7 +3471,7 @@"read_only": false},{- "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag, move_cursor and `args` are exactly that tool's arguments. `pid` and `window_id` (or `app`/`window`) on run_actions itself are the default window of every step that names none. Reads (get_window_state, zoom) do not run between steps: one there is skipped; a get_window_state last is the `observe`, a zoom last runs after it. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or target later steps by role/name. A token that went stale is retried once on the element with the same role and label, if exactly one matches.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step (after a failure caused by a dialog holding focus, that dialog is read, even without `observe`). A step refused because its app ignores background input says how to re-send it; `foreground_fallback:true` retries such steps in the foreground once (the window is activated). That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.\n\nTARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.\n\nWAITS AND CHECKS: a step may add `wait_for` (checked before its action) and `expect` (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the tree, static text included; `gone:true` waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.\n\nExample (fill a form, submit, confirm the dialog, check the result):\n{\"steps\":[{\"set_value\":{\"app\":\"Safari\",\"role\":\"textfield\",\"name\":\"Email\",\"value\":\"[email protected]\"}},{\"click\":{\"role\":\"button\",\"name\":\"Submit\"},\"expect\":{\"role\":\"button\",\"name\":\"Confirm\"}},{\"click\":{\"role\":\"button\",\"name\":\"Confirm\"},\"expect\":[{\"name\":\"Confirm\",\"gone\":true},{\"text\":\"Thanks\"}]}],\"observe\":true}",+ "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag, move_cursor and `args` are exactly that tool's arguments. `pid` and `window_id` (or `app`/`window`) on run_actions itself are the default window of every step that names none. Reads (get_window_state, zoom) do not run between steps: one there is skipped; a get_window_state last is the `observe`, a zoom last runs after it. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or target later steps by role/name. A token that went stale is retried once on the element with the same role and label, if exactly one matches.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step (after a failure caused by a dialog holding focus, that dialog is read, even without `observe`). A step refused because its app ignores background input says how to re-send it; `foreground_fallback:true` retries such steps in the foreground once (the window is activated). That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 64 steps; `delivery_mode` on run_actions is the default for its steps.\n\nTARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.\n\nWAITS AND CHECKS: a step may add `wait_for` (checked before its action) and `expect` (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the tree, static text included; `gone:true` waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.\n\nExample (fill a form, submit, confirm the dialog, check the result):\n{\"steps\":[{\"set_value\":{\"app\":\"Safari\",\"role\":\"textfield\",\"name\":\"Email\",\"value\":\"[email protected]\"}},{\"click\":{\"role\":\"button\",\"name\":\"Submit\"},\"expect\":{\"role\":\"button\",\"name\":\"Confirm\"}},{\"click\":{\"role\":\"button\",\"name\":\"Confirm\"},\"expect\":[{\"name\":\"Confirm\",\"gone\":true},{\"text\":\"Thanks\"}]}],\"observe\":true}","destructive": true,"idempotent": false,"input_schema": {@@ -3487,6 +3487,14 @@"minimum": 0,"type": "integer"},+ "delivery_mode": {+ "description": "Default delivery_mode for steps whose tool takes one and that set none.",+ "enum": [+ "background",+ "foreground"+ ],+ "type": "string"+ },"foreground_fallback": {"description": "When a step is refused because its target ignores background input (Electron, LibreOffice and other toolkits), retry it once with delivery_mode:\"foreground\": the window is activated and the pointer may move. Default false.","type": "boolean"@@ -3579,7 +3587,7 @@}]},- "maxItems": 32,+ "maxItems": 64,"minItems": 1,"type": "array"},diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.jsonindex 5063f93bca..ba180f0bf5 100644--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json@@ -3107,7 +3107,7 @@"read_only": false},{- "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag, move_cursor and `args` are exactly that tool's arguments. `pid` and `window_id` (or `app`/`window`) on run_actions itself are the default window of every step that names none. Reads (get_window_state, zoom) do not run between steps: one there is skipped; a get_window_state last is the `observe`, a zoom last runs after it. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or target later steps by role/name. A token that went stale is retried once on the element with the same role and label, if exactly one matches.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step (after a failure caused by a dialog holding focus, that dialog is read, even without `observe`). A step refused because its app ignores background input says how to re-send it; `foreground_fallback:true` retries such steps in the foreground once (the window is activated). That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.\n\nTARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.\n\nWAITS AND CHECKS: a step may add `wait_for` (checked before its action) and `expect` (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the tree, static text included; `gone:true` waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.\n\nExample (fill a form, submit, confirm the dialog, check the result):\n{\"steps\":[{\"set_value\":{\"app\":\"Safari\",\"role\":\"textfield\",\"name\":\"Email\",\"value\":\"[email protected]\"}},{\"click\":{\"role\":\"button\",\"name\":\"Submit\"},\"expect\":{\"role\":\"button\",\"name\":\"Confirm\"}},{\"click\":{\"role\":\"button\",\"name\":\"Confirm\"},\"expect\":[{\"name\":\"Confirm\",\"gone\":true},{\"text\":\"Thanks\"}]}],\"observe\":true}",+ "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag, move_cursor and `args` are exactly that tool's arguments. `pid` and `window_id` (or `app`/`window`) on run_actions itself are the default window of every step that names none. Reads (get_window_state, zoom) do not run between steps: one there is skipped; a get_window_state last is the `observe`, a zoom last runs after it. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or target later steps by role/name. A token that went stale is retried once on the element with the same role and label, if exactly one matches.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step (after a failure caused by a dialog holding focus, that dialog is read, even without `observe`). A step refused because its app ignores background input says how to re-send it; `foreground_fallback:true` retries such steps in the foreground once (the window is activated). That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 64 steps; `delivery_mode` on run_actions is the default for its steps.\n\nTARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.\n\nWAITS AND CHECKS: a step may add `wait_for` (checked before its action) and `expect` (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the tree, static text included; `gone:true` waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.\n\nExample (fill a form, submit, confirm the dialog, check the result):\n{\"steps\":[{\"set_value\":{\"app\":\"Safari\",\"role\":\"textfield\",\"name\":\"Email\",\"value\":\"[email protected]\"}},{\"click\":{\"role\":\"button\",\"name\":\"Submit\"},\"expect\":{\"role\":\"button\",\"name\":\"Confirm\"}},{\"click\":{\"role\":\"button\",\"name\":\"Confirm\"},\"expect\":[{\"name\":\"Confirm\",\"gone\":true},{\"text\":\"Thanks\"}]}],\"observe\":true}","destructive": true,"idempotent": false,"input_schema": {@@ -3123,6 +3123,14 @@"minimum": 0,"type": "integer"},+ "delivery_mode": {+ "description": "Default delivery_mode for steps whose tool takes one and that set none.",+ "enum": [+ "background",+ "foreground"+ ],+ "type": "string"+ },"foreground_fallback": {"description": "When a step is refused because its target ignores background input (Electron, LibreOffice and other toolkits), retry it once with delivery_mode:\"foreground\": the window is activated and the pointer may move. Default false.","type": "boolean"@@ -3215,7 +3223,7 @@}]},- "maxItems": 32,+ "maxItems": 64,"minItems": 1,"type": "array"},diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.jsonindex 9d9c40ac02..b3a37d06ea 100644--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json@@ -3163,7 +3163,7 @@"read_only": false},{- "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag, move_cursor and `args` are exactly that tool's arguments. `pid` and `window_id` (or `app`/`window`) on run_actions itself are the default window of every step that names none. Reads (get_window_state, zoom) do not run between steps: one there is skipped; a get_window_state last is the `observe`, a zoom last runs after it. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or target later steps by role/name. A token that went stale is retried once on the element with the same role and label, if exactly one matches.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step (after a failure caused by a dialog holding focus, that dialog is read, even without `observe`). A step refused because its app ignores background input says how to re-send it; `foreground_fallback:true` retries such steps in the foreground once (the window is activated). That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.\n\nTARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.\n\nWAITS AND CHECKS: a step may add `wait_for` (checked before its action) and `expect` (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the tree, static text included; `gone:true` waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.\n\nExample (fill a form, submit, confirm the dialog, check the result):\n{\"steps\":[{\"set_value\":{\"app\":\"Safari\",\"role\":\"textfield\",\"name\":\"Email\",\"value\":\"[email protected]\"}},{\"click\":{\"role\":\"button\",\"name\":\"Submit\"},\"expect\":{\"role\":\"button\",\"name\":\"Confirm\"}},{\"click\":{\"role\":\"button\",\"name\":\"Confirm\"},\"expect\":[{\"name\":\"Confirm\",\"gone\":true},{\"text\":\"Thanks\"}]}],\"observe\":true}",+ "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag, move_cursor and `args` are exactly that tool's arguments. `pid` and `window_id` (or `app`/`window`) on run_actions itself are the default window of every step that names none. Reads (get_window_state, zoom) do not run between steps: one there is skipped; a get_window_state last is the `observe`, a zoom last runs after it. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or target later steps by role/name. A token that went stale is retried once on the element with the same role and label, if exactly one matches.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step (after a failure caused by a dialog holding focus, that dialog is read, even without `observe`). A step refused because its app ignores background input says how to re-send it; `foreground_fallback:true` retries such steps in the foreground once (the window is activated). That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 64 steps; `delivery_mode` on run_actions is the default for its steps.\n\nTARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.\n\nWAITS AND CHECKS: a step may add `wait_for` (checked before its action) and `expect` (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the tree, static text included; `gone:true` waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.\n\nExample (fill a form, submit, confirm the dialog, check the result):\n{\"steps\":[{\"set_value\":{\"app\":\"Safari\",\"role\":\"textfield\",\"name\":\"Email\",\"value\":\"[email protected]\"}},{\"click\":{\"role\":\"button\",\"name\":\"Submit\"},\"expect\":{\"role\":\"button\",\"name\":\"Confirm\"}},{\"click\":{\"role\":\"button\",\"name\":\"Confirm\"},\"expect\":[{\"name\":\"Confirm\",\"gone\":true},{\"text\":\"Thanks\"}]}],\"observe\":true}","destructive": true,"idempotent": false,"input_schema": {@@ -3179,6 +3179,14 @@"minimum": 0,"type": "integer"},+ "delivery_mode": {+ "description": "Default delivery_mode for steps whose tool takes one and that set none.",+ "enum": [+ "background",+ "foreground"+ ],+ "type": "string"+ },"foreground_fallback": {"description": "When a step is refused because its target ignores background input (Electron, LibreOffice and other toolkits), retry it once with delivery_mode:\"foreground\": the window is activated and the pointer may move. Default false.","type": "boolean"@@ -3271,7 +3279,7 @@}]},- "maxItems": 32,+ "maxItems": 64,"minItems": 1,"type": "array"},