Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): keep a failed run_actions report readable within client error limits (#4897)

Francesco Bonacci5a364bbe60e1
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/batch.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/batch.mdx
index 7e23b2a488..f10fcda7ff 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/batch.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/batch.mdx
@@ -16,7 +16,7 @@ Served by `cua-driver mcp`; see [MCP tools](/cua-driver/reference/mcp-tools) for
The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag, move_cursor and `args` are exactly that tool's arguments. `pid` and `window_id` (or `app`/`window`) on run_actions itself are the default window of every step that names none. Reads (get_window_state, zoom) do not run between steps: one there is skipped; a get_window_state last is the `observe`, a zoom last runs after it. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or target later steps by role/name. A token that went stale is retried once on the element with the same role and label, if exactly one matches.
-Returns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step (after a failure caused by a dialog holding focus, that dialog is read, even without `observe`). A step refused because its app ignores background input says how to re-send it; `foreground_fallback:true` retries such steps in the foreground once (the window is activated). That read is a `since:"latest"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.
+Returns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step (after a failure caused by a dialog holding focus, that dialog is read, even without `observe`). A step refused because its app ignores background input says how to re-send it; `foreground_fallback:true` retries such steps in the foreground once (the window is activated). That read is a `since:"latest"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 64 steps; `delivery_mode` on run_actions is the default for its steps.
TARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{"click": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.
@@ -31,11 +31,12 @@ Effect: destructive. Platforms: macOS, Linux, Windows.
| --- | --- | --- | --- |
| <span id="run_actions--app"></span>`app` | `string` | | Default app (name or bundle id) for steps that name no window. |
| <span id="run_actions--delay_ms"></span>`delay_ms` | `integer` | | Pause between steps in milliseconds (not after the last). Default 0. Range: 0 to 2000. |
+| <span id="run_actions--delivery_mode"></span>`delivery_mode` | `"background" \| "foreground"` | | Default delivery_mode for steps whose tool takes one and that set none. |
| <span id="run_actions--foreground_fallback"></span>`foreground_fallback` | `boolean` | | When a step is refused because its target ignores background input (Electron, LibreOffice and other toolkits), retry it once with delivery_mode:"foreground": the window is activated and the pointer may move. Default false. |
| <span id="run_actions--observe"></span>`observe` | `object \| boolean` | | Optional end-of-batch observation: `true`, or arguments for ONE get_window_state call. `pid` and `window_id` default to those of the last step that names both. Defaults to since="latest" (only what changed since your last read of that window with the same query/max_elements/max_depth; a full read when there is none), include_screenshot=false and max_elements=250; pass include_screenshot=true to see the window, or since=null for a full read. Omit to read nothing. |
| <span id="run_actions--pid"></span>`pid` | `integer` | | Default pid for steps that name no window (a step's own pid/window_id/app/window wins). |
| <span id="run_actions--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
-| <span id="run_actions--steps"></span>`steps` | `object[]` | required | Ordered actions. Execution stops at the first failing step. Each step is &#123;tool, args&#125; or &#123;&lt;tool&gt;: args&#125;, plus optional wait_for / expect / timeout_ms; a step may also be only a wait_for or expect check. Items: 1 to 32. |
+| <span id="run_actions--steps"></span>`steps` | `object[]` | required | Ordered actions. Execution stops at the first failing step. Each step is &#123;tool, args&#125; or &#123;&lt;tool&gt;: args&#125;, plus optional wait_for / expect / timeout_ms; a step may also be only a wait_for or expect check. Items: 1 to 64. |
| <span id="run_actions--window"></span>`window` | `string` | | Default window title substring for steps that name no window. |
| <span id="run_actions--window_id"></span>`window_id` | `integer` | | Default window_id for steps that name no window, with `pid`. |
diff --git a/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md b/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md
index a3a16754bc..c691be6439 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md
@@ -90,7 +90,7 @@ cua-driver run_actions '{"session":"run-1","steps":[
],"delay_ms":100,"observe":true}'
```
-- `tool` is one of `click`, `double_click`, `right_click`, `set_value`, `type_text`, `press_key`, `hotkey`, `scroll`, `drag`, `move_cursor` (`press` and `type` are accepted for `press_key` and `type_text`, `triple_click` is a click with `count:3`, and a key name such as `down` as the tool is a `press_key`); `args` are exactly that tool's arguments. Observation tools do not run inside a batch: a read between actions is skipped (it could not change the later steps), a `get_window_state` as the last step is the observation (merged with `observe`), and a trailing `zoom` runs after it. Run `describe run_actions` and `describe <tool>` for schemas. Up to 32 steps.
+- `tool` is one of `click`, `double_click`, `right_click`, `set_value`, `type_text`, `press_key`, `hotkey`, `scroll`, `drag`, `move_cursor` (`press` and `type` are accepted for `press_key` and `type_text`, `triple_click` is a click with `count:3`, and a key name such as `down` as the tool is a `press_key`); `args` are exactly that tool's arguments. Observation tools do not run inside a batch: a read between actions is skipped (it could not change the later steps), a `get_window_state` as the last step is the observation (merged with `observe`), and a trailing `zoom` runs after it. Run `describe run_actions` and `describe <tool>` for schemas. Up to 64 steps. `delivery_mode` on `run_actions` is the default for steps that set none.
- Name the window once: `pid` and `window_id` (or `app`/`window`) on `run_actions` itself are the default for every step that names none.
- A step whose `element_token` went stale because an earlier step re-rendered the window is retried once on the element with the same role and label in a fresh read, when exactly one matches; the step's line says so.
- Every step is validated before the first runs, so a malformed step changes nothing. Each step then passes the same session, permission, capability-manifest and approval checks as a direct call; a batch grants nothing a single call lacks, and a refused step ends the batch like any other failure.
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs
index b82d69f90d..9420dc7558 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs
@@ -43,7 +43,7 @@ pub const BATCHABLE_TOOLS: &[&str] = &[
];
/// Most steps one call may carry. Keeps a batch inside client call timeouts.
-pub const MAX_STEPS: usize = 32;
+pub const MAX_STEPS: usize = 64;
/// Longest pause between two steps.
pub const MAX_DELAY_MS: u64 = 2_000;
/// Default `max_elements` for the end-of-batch observation: the same budget as
@@ -189,7 +189,7 @@ impl Tool for RunActionsTool {
since your last read of the window, plus a new snapshot_id. Without \
`observe` nothing is read. The batch uses one session: steps may \
omit `session` or repeat the batch's. `delay_ms` pauses between steps (max \
- 2000). At most 32 steps.\n\n\
+ 2000). At most 64 steps; `delivery_mode` on run_actions is the default for its steps.\n\n\
TARGET BY NAME (no earlier read needed, so one batch can cross screens): \
a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and \
its args may name the target instead of carrying a token: `role` (button, \
@@ -277,6 +277,11 @@ impl Tool for RunActionsTool {
"type": "string",
"description": "Default window title substring for steps that name no window."
},
+ "delivery_mode": {
+ "type": "string",
+ "enum": ["background", "foreground"],
+ "description": "Default delivery_mode for steps whose tool takes one and that set none."
+ },
"foreground_fallback": {
"type": "boolean",
"description": "When a step is refused because its target ignores background input (Electron, LibreOffice and other toolkits), retry it once with delivery_mode:\"foreground\": the window is activated and the pointer may move. Default false."
@@ -458,6 +463,17 @@ impl Plan {
.map(str::to_owned);
let default_window = batch_window(object)?;
+ let default_delivery = match object.get("delivery_mode") {
+ None | Some(Value::Null) => None,
+ Some(Value::String(mode)) if matches!(mode.as_str(), "background" | "foreground") => {
+ Some(mode.clone())
+ }
+ Some(_) => {
+ return Err(PlanError::batch(
+ "`delivery_mode` must be \"background\" or \"foreground\"",
+ ))
+ }
+ };
let foreground_fallback = match object.get("foreground_fallback") {
None | Some(Value::Null) => false,
Some(Value::Bool(flag)) => *flag,
@@ -537,7 +553,14 @@ impl Plan {
});
continue;
}
- let step = parse_step(registry, index, raw, session.as_deref(), has_window)?;
+ let mut step = parse_step(registry, index, raw, session.as_deref(), has_window)?;
+ if let (Some(mode), Some(action)) = (&default_delivery, step.action.as_mut()) {
+ if accepts(registry, action.tool, "delivery_mode") {
+ if let Some(object) = action.args.as_object_mut() {
+ object.entry("delivery_mode").or_insert_with(|| json!(mode));
+ }
+ }
+ }
let names_window = step
.action
.as_ref()
@@ -801,6 +824,9 @@ impl Plan {
let mut content = vec![Content::text(lines.join("\n"))];
content.append(&mut observation_content);
content.append(&mut tail_content);
+ if failed_step.is_some() {
+ fit_error_text(&mut content, ERROR_TEXT_BUDGET);
+ }
ToolResult {
content,
is_error: failed_step.map(|_| true),
@@ -1278,6 +1304,41 @@ fn focus_holder(message: &str) -> Option<u64> {
digits.parse().ok()
}
+/// Most text characters a failed batch returns. MCP clients cut long error
+/// results to their first line or so (Claude Code keeps ~100 characters of
+/// an error over ~4 000), which hid why a step failed behind the read that
+/// followed it. The step report goes first and whole; the read is trimmed.
+const ERROR_TEXT_BUDGET: usize = 3_500;
+
+/// Keep the text of a failed batch within `budget` characters: the first
+/// part (the step report) up to the budget, then later text parts trimmed
+/// to what is left, with a note. Images are kept.
+fn fit_error_text(content: &mut Vec<Content>, budget: usize) {
+ let mut left = budget;
+ let mut trimmed = false;
+ for part in content.iter_mut() {
+ let Content::Text { text, .. } = part else {
+ continue;
+ };
+ let length = text.chars().count();
+ if length <= left {
+ left -= length;
+ continue;
+ }
+ let keep: String = text.chars().take(left).collect();
+ *text = keep;
+ left = 0;
+ trimmed = true;
+ }
+ content.retain(|part| !matches!(part, Content::Text { text, .. } if text.is_empty()));
+ if trimmed {
+ content.push(Content::text(
+ "[the rest of this read was cut so the step report stays readable; call \
+ get_window_state for the full window]",
+ ));
+ }
+}
+
/// Whether a refusal names foreground delivery as the route that works:
/// the target ignores background input (Electron, LibreOffice VCL, ...).
fn wants_foreground(result: &ToolResult) -> bool {
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/reliability_tests.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/reliability_tests.rs
index faca14d92f..508ea4bb39 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/reliability_tests.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/reliability_tests.rs
@@ -152,6 +152,69 @@ async fn a_step_refused_for_background_delivery_says_how_or_falls_back() {
assert_eq!(harness.last("click")["delivery_mode"], "foreground");
}
+#[test]
+fn a_failed_batch_keeps_the_step_report_readable() {
+ // v037c: a failed batch returned its step report plus a full read
+ // (~4 100 characters); the client cut it to "1. click ok: ... AXCh",
+ // hiding why step 2 failed.
+ let report =
+ "run_actions: step 2 of 2 failed\n1. click ok\n2. click ERROR (action): why".to_owned();
+ let mut content = vec![
+ crate::protocol::Content::text(report.clone()),
+ crate::protocol::Content::text("x".repeat(6_000)),
+ ];
+ super::super::fit_error_text(&mut content, super::super::ERROR_TEXT_BUDGET);
+ let text: String = content
+ .iter()
+ .filter_map(|part| match part {
+ crate::protocol::Content::Text { text, .. } => Some(text.as_str()),
+ _ => None,
+ })
+ .collect();
+ assert!(text.starts_with(&report));
+ assert!(text.chars().count() <= super::super::ERROR_TEXT_BUDGET + 200);
+ assert!(text.contains("call get_window_state for the full window"));
+}
+
+#[tokio::test]
+async fn a_failed_batch_with_a_long_read_stays_within_the_budget() {
+ let harness = Harness::new();
+ let result = harness
+ .run(json!({"steps": [
+ {"tool": "click", "args": {"pid": 7, "window_id": 3}},
+ {"tool": "click", "args": {"pid": 7, "window_id": 3, "fail": true}}
+ ], "observe": true}))
+ .await;
+ assert_eq!(result.is_error, Some(true));
+ assert!(
+ text(&result).contains("2. click ERROR"),
+ "{}",
+ text(&result)
+ );
+ assert!(text(&result).chars().count() <= super::super::ERROR_TEXT_BUDGET + 200);
+}
+
+#[tokio::test]
+async fn batch_level_delivery_mode_is_the_steps_default() {
+ let harness = Harness::new();
+ // v037c: `delivery_mode` on run_actions itself was refused.
+ let result = harness
+ .run(
+ json!({"pid": 7, "window_id": 3, "delivery_mode": "foreground", "steps": [
+ {"tool": "click", "args": {"x": 1, "y": 2}},
+ {"tool": "click", "args": {"x": 1, "y": 2, "delivery_mode": "background"}}
+ ]}),
+ )
+ .await;
+ assert_ne!(result.is_error, Some(true), "{}", text(&result));
+ let calls = harness.calls["click"].lock().unwrap().clone();
+ assert_eq!(calls[0]["delivery_mode"], "foreground");
+ assert_eq!(
+ calls[1]["delivery_mode"], "background",
+ "a step's own mode wins"
+ );
+}
+
// ── read steps ───────────────────────────────────────────────────────────
#[tokio::test]
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/protocol_schema_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/protocol_schema_test.rs
index 5e3d5de306..8d37775678 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/protocol_schema_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/protocol_schema_test.rs
@@ -190,6 +190,8 @@ fn tools_list_schema_shape() {
// Chromium); foreground accepts that.
"browser_click",
"browser_pointer",
+ // The batch's default for its steps' own delivery_mode.
+ "run_actions",
// macOS set_value has no delivery ladder.
#[cfg(any(target_os = "linux", target_os = "windows"))]
"set_value",
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json
index 0171de6428..d9fe4f9a47 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json
@@ -3471,7 +3471,7 @@
"read_only": false
},
{
- "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag, move_cursor and `args` are exactly that tool's arguments. `pid` and `window_id` (or `app`/`window`) on run_actions itself are the default window of every step that names none. Reads (get_window_state, zoom) do not run between steps: one there is skipped; a get_window_state last is the `observe`, a zoom last runs after it. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or target later steps by role/name. A token that went stale is retried once on the element with the same role and label, if exactly one matches.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step (after a failure caused by a dialog holding focus, that dialog is read, even without `observe`). A step refused because its app ignores background input says how to re-send it; `foreground_fallback:true` retries such steps in the foreground once (the window is activated). That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.\n\nTARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.\n\nWAITS AND CHECKS: a step may add `wait_for` (checked before its action) and `expect` (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the tree, static text included; `gone:true` waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.\n\nExample (fill a form, submit, confirm the dialog, check the result):\n{\"steps\":[{\"set_value\":{\"app\":\"Safari\",\"role\":\"textfield\",\"name\":\"Email\",\"value\":\"[email protected]\"}},{\"click\":{\"role\":\"button\",\"name\":\"Submit\"},\"expect\":{\"role\":\"button\",\"name\":\"Confirm\"}},{\"click\":{\"role\":\"button\",\"name\":\"Confirm\"},\"expect\":[{\"name\":\"Confirm\",\"gone\":true},{\"text\":\"Thanks\"}]}],\"observe\":true}",
+ "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag, move_cursor and `args` are exactly that tool's arguments. `pid` and `window_id` (or `app`/`window`) on run_actions itself are the default window of every step that names none. Reads (get_window_state, zoom) do not run between steps: one there is skipped; a get_window_state last is the `observe`, a zoom last runs after it. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or target later steps by role/name. A token that went stale is retried once on the element with the same role and label, if exactly one matches.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step (after a failure caused by a dialog holding focus, that dialog is read, even without `observe`). A step refused because its app ignores background input says how to re-send it; `foreground_fallback:true` retries such steps in the foreground once (the window is activated). That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 64 steps; `delivery_mode` on run_actions is the default for its steps.\n\nTARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.\n\nWAITS AND CHECKS: a step may add `wait_for` (checked before its action) and `expect` (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the tree, static text included; `gone:true` waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.\n\nExample (fill a form, submit, confirm the dialog, check the result):\n{\"steps\":[{\"set_value\":{\"app\":\"Safari\",\"role\":\"textfield\",\"name\":\"Email\",\"value\":\"[email protected]\"}},{\"click\":{\"role\":\"button\",\"name\":\"Submit\"},\"expect\":{\"role\":\"button\",\"name\":\"Confirm\"}},{\"click\":{\"role\":\"button\",\"name\":\"Confirm\"},\"expect\":[{\"name\":\"Confirm\",\"gone\":true},{\"text\":\"Thanks\"}]}],\"observe\":true}",
"destructive": true,
"idempotent": false,
"input_schema": {
@@ -3487,6 +3487,14 @@
"minimum": 0,
"type": "integer"
},
+ "delivery_mode": {
+ "description": "Default delivery_mode for steps whose tool takes one and that set none.",
+ "enum": [
+ "background",
+ "foreground"
+ ],
+ "type": "string"
+ },
"foreground_fallback": {
"description": "When a step is refused because its target ignores background input (Electron, LibreOffice and other toolkits), retry it once with delivery_mode:\"foreground\": the window is activated and the pointer may move. Default false.",
"type": "boolean"
@@ -3579,7 +3587,7 @@
}
]
},
- "maxItems": 32,
+ "maxItems": 64,
"minItems": 1,
"type": "array"
},
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
index 5063f93bca..ba180f0bf5 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
@@ -3107,7 +3107,7 @@
"read_only": false
},
{
- "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag, move_cursor and `args` are exactly that tool's arguments. `pid` and `window_id` (or `app`/`window`) on run_actions itself are the default window of every step that names none. Reads (get_window_state, zoom) do not run between steps: one there is skipped; a get_window_state last is the `observe`, a zoom last runs after it. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or target later steps by role/name. A token that went stale is retried once on the element with the same role and label, if exactly one matches.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step (after a failure caused by a dialog holding focus, that dialog is read, even without `observe`). A step refused because its app ignores background input says how to re-send it; `foreground_fallback:true` retries such steps in the foreground once (the window is activated). That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.\n\nTARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.\n\nWAITS AND CHECKS: a step may add `wait_for` (checked before its action) and `expect` (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the tree, static text included; `gone:true` waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.\n\nExample (fill a form, submit, confirm the dialog, check the result):\n{\"steps\":[{\"set_value\":{\"app\":\"Safari\",\"role\":\"textfield\",\"name\":\"Email\",\"value\":\"[email protected]\"}},{\"click\":{\"role\":\"button\",\"name\":\"Submit\"},\"expect\":{\"role\":\"button\",\"name\":\"Confirm\"}},{\"click\":{\"role\":\"button\",\"name\":\"Confirm\"},\"expect\":[{\"name\":\"Confirm\",\"gone\":true},{\"text\":\"Thanks\"}]}],\"observe\":true}",
+ "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag, move_cursor and `args` are exactly that tool's arguments. `pid` and `window_id` (or `app`/`window`) on run_actions itself are the default window of every step that names none. Reads (get_window_state, zoom) do not run between steps: one there is skipped; a get_window_state last is the `observe`, a zoom last runs after it. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or target later steps by role/name. A token that went stale is retried once on the element with the same role and label, if exactly one matches.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step (after a failure caused by a dialog holding focus, that dialog is read, even without `observe`). A step refused because its app ignores background input says how to re-send it; `foreground_fallback:true` retries such steps in the foreground once (the window is activated). That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 64 steps; `delivery_mode` on run_actions is the default for its steps.\n\nTARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.\n\nWAITS AND CHECKS: a step may add `wait_for` (checked before its action) and `expect` (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the tree, static text included; `gone:true` waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.\n\nExample (fill a form, submit, confirm the dialog, check the result):\n{\"steps\":[{\"set_value\":{\"app\":\"Safari\",\"role\":\"textfield\",\"name\":\"Email\",\"value\":\"[email protected]\"}},{\"click\":{\"role\":\"button\",\"name\":\"Submit\"},\"expect\":{\"role\":\"button\",\"name\":\"Confirm\"}},{\"click\":{\"role\":\"button\",\"name\":\"Confirm\"},\"expect\":[{\"name\":\"Confirm\",\"gone\":true},{\"text\":\"Thanks\"}]}],\"observe\":true}",
"destructive": true,
"idempotent": false,
"input_schema": {
@@ -3123,6 +3123,14 @@
"minimum": 0,
"type": "integer"
},
+ "delivery_mode": {
+ "description": "Default delivery_mode for steps whose tool takes one and that set none.",
+ "enum": [
+ "background",
+ "foreground"
+ ],
+ "type": "string"
+ },
"foreground_fallback": {
"description": "When a step is refused because its target ignores background input (Electron, LibreOffice and other toolkits), retry it once with delivery_mode:\"foreground\": the window is activated and the pointer may move. Default false.",
"type": "boolean"
@@ -3215,7 +3223,7 @@
}
]
},
- "maxItems": 32,
+ "maxItems": 64,
"minItems": 1,
"type": "array"
},
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json
index 9d9c40ac02..b3a37d06ea 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json
@@ -3163,7 +3163,7 @@
"read_only": false
},
{
- "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag, move_cursor and `args` are exactly that tool's arguments. `pid` and `window_id` (or `app`/`window`) on run_actions itself are the default window of every step that names none. Reads (get_window_state, zoom) do not run between steps: one there is skipped; a get_window_state last is the `observe`, a zoom last runs after it. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or target later steps by role/name. A token that went stale is retried once on the element with the same role and label, if exactly one matches.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step (after a failure caused by a dialog holding focus, that dialog is read, even without `observe`). A step refused because its app ignores background input says how to re-send it; `foreground_fallback:true` retries such steps in the foreground once (the window is activated). That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.\n\nTARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.\n\nWAITS AND CHECKS: a step may add `wait_for` (checked before its action) and `expect` (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the tree, static text included; `gone:true` waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.\n\nExample (fill a form, submit, confirm the dialog, check the result):\n{\"steps\":[{\"set_value\":{\"app\":\"Safari\",\"role\":\"textfield\",\"name\":\"Email\",\"value\":\"[email protected]\"}},{\"click\":{\"role\":\"button\",\"name\":\"Submit\"},\"expect\":{\"role\":\"button\",\"name\":\"Confirm\"}},{\"click\":{\"role\":\"button\",\"name\":\"Confirm\"},\"expect\":[{\"name\":\"Confirm\",\"gone\":true},{\"text\":\"Thanks\"}]}],\"observe\":true}",
+ "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag, move_cursor and `args` are exactly that tool's arguments. `pid` and `window_id` (or `app`/`window`) on run_actions itself are the default window of every step that names none. Reads (get_window_state, zoom) do not run between steps: one there is skipped; a get_window_state last is the `observe`, a zoom last runs after it. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or target later steps by role/name. A token that went stale is retried once on the element with the same role and label, if exactly one matches.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step (after a failure caused by a dialog holding focus, that dialog is read, even without `observe`). A step refused because its app ignores background input says how to re-send it; `foreground_fallback:true` retries such steps in the foreground once (the window is activated). That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 64 steps; `delivery_mode` on run_actions is the default for its steps.\n\nTARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.\n\nWAITS AND CHECKS: a step may add `wait_for` (checked before its action) and `expect` (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the tree, static text included; `gone:true` waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.\n\nExample (fill a form, submit, confirm the dialog, check the result):\n{\"steps\":[{\"set_value\":{\"app\":\"Safari\",\"role\":\"textfield\",\"name\":\"Email\",\"value\":\"[email protected]\"}},{\"click\":{\"role\":\"button\",\"name\":\"Submit\"},\"expect\":{\"role\":\"button\",\"name\":\"Confirm\"}},{\"click\":{\"role\":\"button\",\"name\":\"Confirm\"},\"expect\":[{\"name\":\"Confirm\",\"gone\":true},{\"text\":\"Thanks\"}]}],\"observe\":true}",
"destructive": true,
"idempotent": false,
"input_schema": {
@@ -3179,6 +3179,14 @@
"minimum": 0,
"type": "integer"
},
+ "delivery_mode": {
+ "description": "Default delivery_mode for steps whose tool takes one and that set none.",
+ "enum": [
+ "background",
+ "foreground"
+ ],
+ "type": "string"
+ },
"foreground_fallback": {
"description": "When a step is refused because its target ignores background input (Electron, LibreOffice and other toolkits), retry it once with delivery_mode:\"foreground\": the window is activated and the pointer may move. Default false.",
"type": "boolean"
@@ -3271,7 +3279,7 @@
}
]
},
- "maxItems": 32,
+ "maxItems": 64,
"minItems": 1,
"type": "array"
},