Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cua-driver): run_actions steps that find, wait for and check by name (#4820)

Francesco Bonacciccaacd8fda02
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/batch.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/batch.mdx
index d41ba3faed..1223f2a4e8 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/batch.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/batch.mdx
@@ -18,6 +18,13 @@ The default way to act: run one or more action tools in ONE call, in order, stop
Returns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. That read is a `since:"latest"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.
+TARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{"click": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.
+
+WAITS AND CHECKS: a step may add `wait_for` (checked before its action) and `expect` (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the tree, static text included; `gone:true` waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.
+
+Example (fill a form, submit, confirm the dialog, check the result):
+{"steps":[{"set_value":{"app":"Safari","role":"textfield","name":"Email","value":"[email protected]"}},{"click":{"role":"button","name":"Submit"},"expect":{"role":"button","name":"Confirm"}},{"click":{"role":"button","name":"Confirm"},"expect":[{"name":"Confirm","gone":true},{"text":"Thanks"}]}],"observe":true}.
+
Effect: destructive. Platforms: macOS, Linux, Windows.
| Parameter | Type | Default | Description |
@@ -25,7 +32,7 @@ Effect: destructive. Platforms: macOS, Linux, Windows.
| <span id="run_actions--delay_ms"></span>`delay_ms` | `integer` | | Pause between steps in milliseconds (not after the last). Default 0. Range: 0 to 2000. |
| <span id="run_actions--observe"></span>`observe` | `object \| boolean` | | Optional end-of-batch observation: `true`, or arguments for ONE get_window_state call. `pid` and `window_id` default to those of the last step that names both. Defaults to since="latest" (only what changed since your last read of that window with the same query/max_elements/max_depth; a full read when there is none), include_screenshot=false and max_elements=250; pass include_screenshot=true to see the window, or since=null for a full read. Omit to read nothing. |
| <span id="run_actions--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
-| <span id="run_actions--steps"></span>`steps` | `object[]` | required | Ordered actions. Execution stops at the first failing step. Items: 1 to 32. |
+| <span id="run_actions--steps"></span>`steps` | `object[]` | required | Ordered actions. Execution stops at the first failing step. Each step is &#123;tool, args&#125; or &#123;&lt;tool&gt;: args&#125;, plus optional wait_for / expect / timeout_ms; a step may also be only a wait_for or expect check. Items: 1 to 32. |
**Example arguments**
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs
index 368b164a17..dc86c03745 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs
@@ -9,7 +9,7 @@
//! not. The batch itself is only a thin ordered loop plus up-front validation.
use std::sync::OnceLock;
-use std::time::Duration;
+use std::time::{Duration, Instant};
use async_trait::async_trait;
use serde_json::{json, Map, Value};
@@ -20,6 +20,10 @@ use crate::{
tool::{Tool, ToolDef, ToolRegistry},
};
+pub(crate) mod locate;
+
+use locate::{Check, ElementSpec, Locator, Window, WindowSpec};
+
/// Public name of the batch tool.
pub const RUN_ACTIONS_TOOL: &str = "run_actions";
@@ -37,17 +41,6 @@ pub const BATCHABLE_TOOLS: &[&str] = &[
"drag",
];
-/// Read-only tools models put in a batch; refused with a pointer to `observe`.
-const OBSERVATION_TOOLS: &[&str] = &[
- OBSERVE_TOOL,
- "get_desktop_state",
- "get_accessibility_tree",
- "get_browser_state",
- "list_windows",
- "verify_state",
- "zoom",
-];
-
/// Most steps one call may carry. Keeps a batch inside client call timeouts.
pub const MAX_STEPS: usize = 32;
/// Longest pause between two steps.
@@ -64,6 +57,23 @@ const TOOL_ALIASES: &[(&str, &str)] = &[
("set", "set_value"),
];
const MESSAGE_LIMIT: usize = 300;
+/// Read-only tools models put in a batch; refused with a pointer to `observe`.
+const OBSERVATION_TOOLS: &[&str] = &[
+ OBSERVE_TOOL,
+ "get_desktop_state",
+ "get_accessibility_tree",
+ "get_browser_state",
+ "list_windows",
+ "verify_state",
+ "zoom",
+];
+/// Most `expect` checks one step may carry.
+pub const MAX_EXPECTS: usize = 4;
+/// Wall-time budget for one batch, waits included. Keeps a batch inside
+/// client call timeouts.
+pub const MAX_BATCH_MS: u64 = 60_000;
+/// Fields a step object may carry besides one action.
+const STEP_KEYS: &[&str] = &["tool", "args", "wait_for", "expect", "timeout_ms"];
const OBSERVE_TOOL: &str = "get_window_state";
pub struct RunActionsTool {
@@ -104,7 +114,35 @@ impl Tool for RunActionsTool {
since your last read of the window, plus a new snapshot_id. Without \
`observe` nothing is read. The batch uses one session: steps may \
omit `session` or repeat the batch's. `delay_ms` pauses between steps (max \
- 2000). At most 32 steps."
+ 2000). At most 32 steps.\n\n\
+ TARGET BY NAME (no earlier read needed, so one batch can cross screens): \
+ a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and \
+ its args may name the target instead of carrying a token: `role` (button, \
+ textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), \
+ `name` (the element's label; exact match wins over substring, case-insensitive), \
+ `nth` (0-based, when several match), and `app` (app name or bundle id) and/or \
+ `window` (title substring) instead of pid/window_id. The batch reads the window \
+ fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to \
+ appear, and acts on it; with only `app` it uses that app's frontmost window, so \
+ a dialog that opened on top is found. Later steps inherit the window, so name \
+ `app` once. The first lookup in a window also reads it once for `observe`, so \
+ the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a \
+ fuller name.\n\n\
+ WAITS AND CHECKS: a step may add `wait_for` (checked before its action) and \
+ `expect` (one check or up to 4, checked after it); a step may also be only a \
+ check. A check is {role?, name?, text?, gone?, value?, value_contains?, \
+ enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the \
+ tree, static text included; `gone:true` waits for it to disappear. Checks poll \
+ until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check \
+ that does not hold fails the step with what was found instead. The report says \
+ which step failed, in which phase (wait_for, find, action, expect) and why. \
+ Whole batch: at most 60 s.\n\n\
+ Example (fill a form, submit, confirm the dialog, check the result):\n\
+ {\"steps\":[\
+ {\"set_value\":{\"app\":\"Safari\",\"role\":\"textfield\",\"name\":\"Email\",\"value\":\"[email protected]\"}},\
+ {\"click\":{\"role\":\"button\",\"name\":\"Submit\"},\"expect\":{\"role\":\"button\",\"name\":\"Confirm\"}},\
+ {\"click\":{\"role\":\"button\",\"name\":\"Confirm\"},\"expect\":[{\"name\":\"Confirm\",\"gone\":true},{\"text\":\"Thanks\"}]}\
+ ],\"observe\":true}"
.into(),
input_schema: json!({
"type": "object",
@@ -114,15 +152,32 @@ impl Tool for RunActionsTool {
"type": "array",
"minItems": 1,
"maxItems": MAX_STEPS,
- "description": "Ordered actions. Execution stops at the first failing step.",
+ "description": "Ordered actions. Execution stops at the first failing step. Each step is {tool, args} or {<tool>: args}, plus optional wait_for / expect / timeout_ms; a step may also be only a wait_for or expect check.",
"items": {
- "type": "object",
- "required": ["tool"],
- "properties": {
- "tool": { "type": "string", "enum": BATCHABLE_TOOLS, "description": "Action tool to run." },
- "args": { "type": "object", "description": "Arguments for that tool, as in a direct call." }
- },
- "additionalProperties": false
+ "anyOf": [
+ {
+ "type": "object",
+ "required": ["tool"],
+ "properties": {
+ "tool": { "type": "string", "enum": BATCHABLE_TOOLS, "description": "Action tool to run." },
+ "args": { "type": "object", "description": "Arguments for that tool, as in a direct call, optionally with role/name/nth/app/window to name the target instead of pid/window_id/element_token." },
+ "wait_for": { "type": "object", "description": "Check that must hold before the action: {role?, name?, text?, gone?, app?, window?, timeout_ms?}. Default timeout 5000 ms." },
+ "expect": { "type": ["object", "array"], "description": "Check (or up to 4) that must hold after the action: {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}. Default timeout 2000 ms." },
+ "timeout_ms": { "type": "integer", "minimum": 0, "maximum": 10000, "description": "How long a named target may take to appear. Default 3000." }
+ },
+ "additionalProperties": false
+ },
+ {
+ "type": "object",
+ "description": "Shorthand {<tool>: args} with optional wait_for / expect / timeout_ms, or a step that is only a wait_for or expect check.",
+ "properties": {
+ "wait_for": { "type": "object" },
+ "expect": { "type": ["object", "array"] },
+ "timeout_ms": { "type": "integer", "minimum": 0, "maximum": 10000 }
+ },
+ "additionalProperties": true
+ }
+ ]
}
},
"delay_ms": {
@@ -163,15 +218,40 @@ impl Tool for RunActionsTool {
}
}
-struct Step {
+/// One action of a step.
+struct Action {
tool: &'static str,
+ /// The tool's own arguments, without the selector keys.
args: Value,
+ /// Window named by the step itself (pid/window_id/app/window).
+ window: WindowSpec,
+ /// Element named by description; empty when the step targets by token,
+ /// pixels or focus.
+ element: ElementSpec,
+ find_timeout: Duration,
+}
+
+struct Step {
+ action: Option<Action>,
+ wait_for: Option<Check>,
+ expect: Vec<Check>,
+}
+
+impl Step {
+ fn label(&self) -> &'static str {
+ match &self.action {
+ Some(action) => action.tool,
+ None if self.wait_for.is_some() => "wait_for",
+ None => "expect",
+ }
+ }
}
struct Plan {
steps: Vec<Step>,
delay: Duration,
observe: Option<Value>,
+ session: Option<String>,
}
struct PlanError {
@@ -213,6 +293,35 @@ impl PlanError {
}
}
+/// How one step ended.
+struct StepOutcome {
+ report: Value,
+ line: String,
+ ok: bool,
+}
+
+/// Where in a step a failure happened.
+#[derive(Clone, Copy)]
+enum Phase {
+ WaitFor,
+ Find,
+ Action,
+ Expect,
+ Budget,
+}
+
+impl Phase {
+ fn as_str(self) -> &'static str {
+ match self {
+ Phase::WaitFor => "wait_for",
+ Phase::Find => "find",
+ Phase::Action => "action",
+ Phase::Expect => "expect",
+ Phase::Budget => "time_budget",
+ }
+ }
+}
+
impl Plan {
fn parse(registry: &ToolRegistry, args: &Value) -> Result<Self, PlanError> {
let object = args
@@ -253,81 +362,129 @@ impl Plan {
object.get("observe"),
None | Some(Value::Null) | Some(Value::Bool(false))
);
- if !actions.is_empty()
- && !observe_given
- && last.get("tool").and_then(Value::as_str) == Some(OBSERVE_TOOL)
- {
- trailing_observe = Some(
- last.get("args")
- .cloned()
- .filter(Value::is_object)
- .unwrap_or_else(|| Value::Object(Map::new())),
- );
- raw_steps = actions;
+ if !actions.is_empty() && !observe_given {
+ if let Some(args) = observation_step_args(last) {
+ trailing_observe = Some(args);
+ raw_steps = actions;
+ }
}
}
let mut steps = Vec::with_capacity(raw_steps.len());
+ // Whether some earlier step names a window a later step can inherit.
+ let mut has_window = false;
for (index, raw) in raw_steps.iter().enumerate() {
- steps.push(parse_step(registry, index, raw, session.as_deref())?);
+ let step = parse_step(registry, index, raw, session.as_deref())?;
+ let names_window = step
+ .action
+ .as_ref()
+ .is_some_and(|action| !action.window.is_empty())
+ || step
+ .wait_for
+ .iter()
+ .chain(step.expect.iter())
+ .any(|check| !check.window.is_empty());
+ let needs_window = step
+ .action
+ .as_ref()
+ .is_some_and(|action| !action.element.is_empty() && action.window.is_empty())
+ || step
+ .wait_for
+ .iter()
+ .chain(step.expect.iter())
+ .any(|check| check.window.is_empty());
+ if needs_window && !names_window && !has_window {
+ return Err(PlanError::step(
+ index,
+ "names an element but no window: add `app` (or pid and window_id) to this step or an earlier one",
+ ));
+ }
+ has_window |= names_window;
+ steps.push(step);
}
let observe = match (trailing_observe.as_ref(), object.get("observe")) {
- (Some(value), _) => Some(parse_observe(registry, value, &steps, session.as_deref())?),
+ (Some(value), _) => Some(parse_observe(
+ registry,
+ value,
+ has_window,
+ session.as_deref(),
+ )?),
(None, None | Some(Value::Null) | Some(Value::Bool(false))) => None,
- (None, Some(value)) => {
- Some(parse_observe(registry, value, &steps, session.as_deref())?)
- }
+ (None, Some(value)) => Some(parse_observe(
+ registry,
+ value,
+ has_window,
+ session.as_deref(),
+ )?),
};
Ok(Self {
steps,
delay: Duration::from_millis(delay_ms),
observe,
+ session,
})
}
async fn run(self, registry: &ToolRegistry) -> ToolResult {
let total = self.steps.len();
+ let started = Instant::now();
+ let mut locator = Locator::new(
+ registry,
+ self.session.clone(),
+ started + Duration::from_millis(MAX_BATCH_MS),
+ );
+ // A default diff observation gets a baseline from the batch's first
+ // read of each window it looks in.
+ if let Some(observe) = &self.observe {
+ let plain_view = observe.get("query").is_none()
+ && observe.get("max_depth").is_none()
+ && observe.get("since").and_then(Value::as_str)
+ == Some(crate::window_state_view::SINCE_LATEST);
+ if let (true, Some(max_elements)) = (
+ plain_view,
+ observe.get("max_elements").and_then(Value::as_u64),
+ ) {
+ locator = locator.with_observe_baseline(max_elements);
+ }
+ }
let mut reports = Vec::with_capacity(total);
let mut lines = Vec::with_capacity(total + 2);
let mut failed_step = None;
+ // The window later steps inherit, and the last concrete window used.
+ let mut context = WindowSpec::default();
+ let mut last_window: Option<Window> = None;
for (index, step) in self.steps.into_iter().enumerate() {
if index > 0 && !self.delay.is_zero() {
tokio::time::sleep(self.delay).await;
}
- let result = registry.invoke(step.tool, step.args).await;
- let ok = result.is_error != Some(true);
- let message = bounded_message(&result);
- lines.push(format!(
- "{}. {} {}{}",
- index + 1,
- step.tool,
- if ok { "ok" } else { "ERROR" },
- if message.is_empty() {
- String::new()
- } else {
- format!(": {message}")
- }
- ));
- let mut report = json!({
- "index": index,
- "tool": step.tool,
- "ok": ok,
- "message": message,
- });
- if !ok {
- if let Some(code) = result
- .structured_content
- .as_ref()
- .and_then(|value| value.get("code"))
- .and_then(Value::as_str)
- {
- report["code"] = Value::String(code.to_owned());
- }
- }
- reports.push(report);
- if !ok {
+ let outcome = if started.elapsed() >= Duration::from_millis(MAX_BATCH_MS) {
+ failure(
+ index,
+ step.label(),
+ Phase::Budget,
+ "time_budget",
+ format!(
+ "the batch used its {} s budget before this step",
+ MAX_BATCH_MS / 1000
+ ),
+ json!({}),
+ )
+ } else {
+ run_step(
+ registry,
+ &locator,
+ index,
+ step,
+ &mut context,
+ &mut last_window,
+ )
+ .await
+ };
+ lines.push(outcome.line);
+ reports.push(outcome.report);
+ if !outcome.ok {
failed_step = Some(index);
break;
}
@@ -355,16 +512,33 @@ impl Plan {
// One observation at the end, also after a failure so the caller can
// see where the app was left.
let mut observation_content = Vec::new();
- if let Some(args) = self.observe {
- let result = registry.invoke(OBSERVE_TOOL, args).await;
+ if let Some(mut args) = self.observe {
+ let filled = fill_observed_window(&mut args, last_window.as_ref());
+ let result = if filled {
+ registry.invoke(OBSERVE_TOOL, args).await
+ } else {
+ ToolResult::error("no window to observe: no step resolved one")
+ };
let ok = result.is_error != Some(true);
let mut report = json!({ "ok": ok, "tool": OBSERVE_TOOL });
if ok {
+ let summary = result
+ .structured_content
+ .as_ref()
+ .map(observation_summary)
+ .unwrap_or_default();
if let Some(state) = result.structured_content.clone() {
report["state"] = state;
}
+ if !summary.is_empty() {
+ report["summary"] = Value::String(summary.clone());
+ }
observation_content = result.content;
- lines.push("observation: get_window_state ok (below)".to_owned());
+ lines.push(if summary.is_empty() {
+ "observation: get_window_state ok (below)".to_owned()
+ } else {
+ format!("observation: {summary} (below)")
+ });
} else {
let message = bounded_message(&result);
lines.push(format!("observation: get_window_state ERROR: {message}"));
@@ -384,28 +558,458 @@ impl Plan {
}
}
+async fn run_step(
+ registry: &ToolRegistry,
+ locator: &Locator<'_>,
+ index: usize,
+ step: Step,
+ context: &mut WindowSpec,
+ last_window: &mut Option<Window>,
+) -> StepOutcome {
+ let label = step.label();
+ let step_has_no_action = step.action.is_none();
+ let mut report = json!({ "index": index, "tool": label });
+ let mut notes: Vec<String> = Vec::new();
+
+ if let Some(check) = &step.wait_for {
+ match locator.check(check, context).await {
+ Ok(checked) => {
+ if !check.window.is_empty() {
+ *context = check.window.clone();
+ }
+ if let Some(window) = &checked.window {
+ *last_window = Some(window.clone());
+ }
+ report["wait_for"] = json!({
+ "ok": true,
+ "waited_ms": checked.waited.as_millis() as u64,
+ });
+ notes.push(format!(
+ "waited {} ms for {}",
+ checked.waited.as_millis(),
+ check.describe()
+ ));
+ }
+ Err(miss) => {
+ if let Some(window) = &miss.window {
+ *last_window = Some(window.clone());
+ }
+ return failure(
+ index,
+ label,
+ Phase::WaitFor,
+ miss.code,
+ format!("{} did not hold: {}", check.describe(), miss.message),
+ report,
+ );
+ }
+ }
+ }
+
+ if let Some(action) = step.action {
+ let Action {
+ tool,
+ mut args,
+ window,
+ element,
+ find_timeout,
+ } = action;
+ if !window.is_empty() {
+ *context = window.clone();
+ }
+ let window_spec = if window.is_empty() {
+ context.clone()
+ } else {
+ window
+ };
+ if !element.is_empty() {
+ match locator.find(&window_spec, &element, find_timeout).await {
+ Ok(found) => {
+ let object = args.as_object_mut().expect("object");
+ object.insert("pid".into(), json!(found.window.pid));
+ object.insert("window_id".into(), json!(found.window.window_id));
+ object.insert(
+ "element_token".into(),
+ json!(found.token().unwrap_or_default()),
+ );
+ report["target"] = json!({
+ "window": found.window.to_json(),
+ "element": found.describe(),
+ });
+ notes.push(format!(
+ "on {} in {}",
+ found.describe(),
+ found.window.describe()
+ ));
+ *last_window = Some(found.window);
+ }
+ Err(miss) => {
+ if let Some(window) = &miss.window {
+ *last_window = Some(window.clone());
+ }
+ return failure(index, label, Phase::Find, miss.code, miss.message, report);
+ }
+ }
+ } else if needs_lookup(&window_spec, &args, tool, registry) {
+ match locator.window(&window_spec).await {
+ Ok(resolved) => {
+ let object = args.as_object_mut().expect("object");
+ object.insert("pid".into(), json!(resolved.pid));
+ if accepts(registry, tool, "window_id") {
+ object
+ .entry("window_id")
+ .or_insert_with(|| json!(resolved.window_id));
+ }
+ report["target"] = json!({ "window": resolved.to_json() });
+ *last_window = Some(resolved);
+ }
+ Err(miss) => {
+ return failure(index, label, Phase::Find, miss.code, miss.message, report)
+ }
+ }
+ } else if let (Some(pid), Some(window_id)) = (
+ args.get("pid").and_then(Value::as_i64),
+ args.get("window_id").and_then(Value::as_u64),
+ ) {
+ *last_window = Some(Window {
+ pid,
+ window_id,
+ app: None,
+ title: None,
+ on_screen: true,
+ });
+ }
+
+ let result = registry.invoke(tool, args).await;
+ let ok = result.is_error != Some(true);
+ let message = bounded_message(&result);
+ report["ok"] = json!(ok);
+ report["message"] = json!(message);
+ if !ok {
+ if let Some(code) = result
+ .structured_content
+ .as_ref()
+ .and_then(|value| value.get("code"))
+ .and_then(Value::as_str)
+ {
+ report["code"] = Value::String(code.to_owned());
+ }
+ return StepOutcome {
+ line: format!("{}. {label} ERROR (action): {message}", index + 1),
+ report: with_phase(report, Phase::Action),
+ ok: false,
+ };
+ }
+ if !message.is_empty() {
+ notes.insert(0, message);
+ }
+ }
+
+ let fallback = last_window
+ .as_ref()
+ .map(|window| WindowSpec {
+ pid: Some(window.pid),
+ window_id: Some(window.window_id),
+ ..Default::default()
+ })
+ .filter(|_| context.is_empty())
+ .unwrap_or_else(|| context.clone());
+ let mut checked = Vec::new();
+ for (nth, check) in step.expect.iter().enumerate() {
+ match locator.check(check, &fallback).await {
+ Ok(outcome) => {
+ if let Some(window) = &outcome.window {
+ if last_window.is_none() || step_has_no_action {
+ *last_window = Some(window.clone());
+ }
+ }
+ if !check.window.is_empty() && step_has_no_action {
+ *context = check.window.clone();
+ }
+ checked.push(json!({
+ "ok": true,
+ "check": check.describe(),
+ "observed": outcome.observed,
+ "waited_ms": outcome.waited.as_millis() as u64,
+ }));
+ }
+ Err(miss) => {
+ if let Some(window) = &miss.window {
+ *last_window = Some(window.clone());
+ }
+ report["expect"] = Value::Array(checked);
+ let which = if step.expect.len() > 1 {
+ format!("expect {} of {}: ", nth + 1, step.expect.len())
+ } else {
+ String::new()
+ };
+ return failure(
+ index,
+ label,
+ Phase::Expect,
+ miss.code,
+ format!("{which}{}", miss.message),
+ report,
+ );
+ }
+ }
+ }
+ if !checked.is_empty() {
+ notes.push(format!("{} check(s) held", checked.len()));
+ report["expect"] = Value::Array(checked);
+ }
+ report["ok"] = json!(true);
+ if report.get("message").is_none() {
+ report["message"] = json!(notes.join("; "));
+ }
+ let detail = notes.join("; ");
+ StepOutcome {
+ line: format!(
+ "{}. {label} ok{}",
+ index + 1,
+ if detail.is_empty() {
+ String::new()
+ } else {
+ format!(": {detail}")
+ }
+ ),
+ report,
+ ok: true,
+ }
+}
+
+fn failure(
+ index: usize,
+ label: &str,
+ phase: Phase,
+ code: &str,
+ message: String,
+ mut report: Value,
+) -> StepOutcome {
+ report["ok"] = json!(false);
+ report["code"] = json!(code);
+ report["message"] = json!(message);
+ StepOutcome {
+ line: format!(
+ "{}. {label} ERROR ({}): {message}",
+ index + 1,
+ phase.as_str()
+ ),
+ report: with_phase(report, phase),
+ ok: false,
+ }
+}
+
+fn with_phase(mut report: Value, phase: Phase) -> Value {
+ report["phase"] = json!(phase.as_str());
+ report
+}
+
+/// A step with a window by name, or one that relies on an inherited window
+/// and does not carry its own pid, needs a window lookup.
+fn needs_lookup(spec: &WindowSpec, args: &Value, tool: &str, registry: &ToolRegistry) -> bool {
+ if spec.app.is_some() || spec.title.is_some() {
+ return true;
+ }
+ let has_pid = args.get("pid").is_some();
+ !has_pid && !spec.is_empty() && accepts(registry, tool, "pid")
+}
+
+fn accepts(registry: &ToolRegistry, tool: &str, field: &str) -> bool {
+ registry
+ .get_def(tool)
+ .and_then(|def| def.input_schema.get("properties"))
+ .and_then(|properties| properties.get(field))
+ .is_some()
+}
+
+/// Fill the observation's window from the last window the batch used.
+fn fill_observed_window(args: &mut Value, last: Option<&Window>) -> bool {
+ let object = args.as_object_mut().expect("object");
+ if object.contains_key("pid") && object.contains_key("window_id") {
+ return true;
+ }
+ let Some(window) = last else {
+ return false;
+ };
+ object.entry("pid").or_insert_with(|| json!(window.pid));
+ object
+ .entry("window_id")
+ .or_insert_with(|| json!(window.window_id));
+ true
+}
+
+/// One line saying what the end-of-batch read found.
+fn observation_summary(state: &Value) -> String {
+ let status = state.get("since_status").and_then(Value::as_str);
+ let since = state.get("since").and_then(Value::as_str).unwrap_or("");
+ match status {
+ Some("diff") => {
+ let counts = &state["diff_counts"];
+ format!(
+ "changed since {since}: {} added, {} changed, {} removed",
+ counts["added"].as_u64().unwrap_or(0),
+ counts["changed"].as_u64().unwrap_or(0),
+ counts["removed"].as_u64().unwrap_or(0)
+ )
+ }
+ Some("no_change") => format!("no change since {since}"),
+ Some(_) => "full read (no usable earlier read of this window to diff against)".to_owned(),
+ None => String::new(),
+ }
+}
+
+/// The arguments of a get_window_state step, in either step form.
+fn observation_step_args(step: &Value) -> Option<Value> {
+ let object = step.as_object()?;
+ let args = if object.get("tool").and_then(Value::as_str) == Some(OBSERVE_TOOL) {
+ object.get("args")
+ } else if object.len() == 1 && object.contains_key(OBSERVE_TOOL) {
+ object.get(OBSERVE_TOOL)
+ } else {
+ return None;
+ };
+ Some(
+ args.cloned()
+ .filter(Value::is_object)
+ .unwrap_or_else(|| Value::Object(Map::new())),
+ )
+}
+
fn parse_step(
registry: &ToolRegistry,
index: usize,
raw: &Value,
session: Option<&str>,
) -> Result<Step, PlanError> {
- let object = raw
- .as_object()
- .ok_or_else(|| PlanError::step(index, "must be an object {tool, args}"))?;
- if let Some(extra) = object
+ let object = raw.as_object().ok_or_else(|| {
+ PlanError::step(index, "must be an object: {tool, args} or {<tool>: args}")
+ })?;
+ // The action: `{tool, args}`, or the shorthand `{<tool>: args}`.
+ let shorthand: Vec<&String> = object
.keys()
- .find(|key| !matches!(key.as_str(), "tool" | "args"))
- {
- return Err(PlanError::step(
+ .filter(|key| !STEP_KEYS.contains(&key.as_str()))
+ .collect();
+ let (name, raw_args) = match (object.get("tool"), shorthand.as_slice()) {
+ (Some(_), [extra, ..]) => {
+ return Err(PlanError::step(
+ index,
+ format!("unknown field `{extra}`; a step has `tool` and `args`, or one `{{<tool>: args}}` key, plus optional wait_for/expect/timeout_ms"),
+ ))
+ }
+ (Some(tool), []) => (
+ Some(
+ tool.as_str()
+ .ok_or_else(|| PlanError::step(index, "`tool` must be a string"))?,
+ ),
+ object.get("args"),
+ ),
+ (None, [key]) => {
+ if object.contains_key("args") {
+ return Err(PlanError::step(
+ index,
+ format!("`{key}` already carries the arguments; drop `args`"),
+ ));
+ }
+ (Some(key.as_str()), object.get(key.as_str()))
+ }
+ (None, []) => {
+ if object.contains_key("args") {
+ return Err(PlanError::step(index, "`args` without `tool`"));
+ }
+ (None, None)
+ }
+ (None, keys) => {
+ return Err(PlanError::step(
+ index,
+ format!(
+ "one action per step, got {}",
+ keys.iter()
+ .map(|key| format!("`{key}`"))
+ .collect::<Vec<_>>()
+ .join(", ")
+ ),
+ ))
+ }
+ };
+
+ let find_timeout = match object.get("timeout_ms") {
+ None | Some(Value::Null) => locate::DEFAULT_FIND_TIMEOUT_MS,
+ Some(value) => value
+ .as_u64()
+ .filter(|ms| *ms <= locate::MAX_CHECK_TIMEOUT_MS)
+ .ok_or_else(|| {
+ PlanError::step(
+ index,
+ format!(
+ "`timeout_ms` must be an integer 0 to {}",
+ locate::MAX_CHECK_TIMEOUT_MS
+ ),
+ )
+ })?,
+ };
+ let wait_for = match object.get("wait_for") {
+ None | Some(Value::Null) => None,
+ Some(raw) => Some(
+ Check::parse(raw, locate::DEFAULT_WAIT_TIMEOUT_MS)
+ .map_err(|message| PlanError::step(index, format!("wait_for: {message}")))?,
+ ),
+ };
+ let expect = match object.get("expect") {
+ None | Some(Value::Null) => Vec::new(),
+ Some(Value::Array(items)) if items.is_empty() || items.len() > MAX_EXPECTS => {
+ return Err(PlanError::step(
+ index,
+ format!("`expect` must hold 1 to {MAX_EXPECTS} checks"),
+ ))
+ }
+ Some(Value::Array(items)) => items
+ .iter()
+ .enumerate()
+ .map(|(nth, raw)| {
+ Check::parse(raw, locate::DEFAULT_EXPECT_TIMEOUT_MS).map_err(|message| {
+ PlanError::step(index, format!("expect {}: {message}", nth + 1))
+ })
+ })
+ .collect::<Result<_, _>>()?,
+ Some(raw) => vec![Check::parse(raw, locate::DEFAULT_EXPECT_TIMEOUT_MS)
+ .map_err(|message| PlanError::step(index, format!("expect: {message}")))?],
+ };
+
+ let action = match name {
+ None => {
+ if wait_for.is_none() && expect.is_empty() {
+ return Err(PlanError::step(
+ index,
+ "empty step: give an action ({tool, args} or {<tool>: args}) or a wait_for/expect check",
+ ));
+ }
+ None
+ }
+ Some(name) => Some(parse_action(
+ registry,
index,
- format!("unknown field `{extra}`; a step has only `tool` and `args`"),
- ));
- }
- let name = object
- .get("tool")
- .and_then(Value::as_str)
- .ok_or_else(|| PlanError::step(index, "`tool` must be a string"))?;
+ name,
+ raw_args,
+ find_timeout,
+ session,
+ )?),
+ };
+ Ok(Step {
+ action,
+ wait_for,
+ expect,
+ })
+}
+
+fn parse_action(
+ registry: &ToolRegistry,
+ index: usize,
+ name: &str,
+ raw_args: Option<&Value>,
+ find_timeout: u64,
+ session: Option<&str>,
+) -> Result<Action, PlanError> {
let name = TOOL_ALIASES
.iter()
.find(|(alias, _)| *alias == name)
@@ -419,8 +1023,8 @@ fn parse_step(
format!(
"`{name}` reads state and a batch only acts. Put get_window_state arguments \
in `observe` (one read after the last step; a get_window_state as the \
- last step is taken as `observe`), or split the batch where you need to look. \
- Batchable tools: {}",
+ last step is taken as `observe`), or use a wait_for/expect check, or split \
+ the batch where you need to look. Batchable tools: {}",
BATCHABLE_TOOLS.join(", ")
)
} else {
@@ -431,23 +1035,62 @@ fn parse_step(
};
PlanError::step(index, message)
})?;
- let mut args = match object.get("args") {
+ let mut args = match raw_args {
None | Some(Value::Null) => Value::Object(Map::new()),
Some(value @ Value::Object(_)) => value.clone(),
Some(_) => return Err(PlanError::step(index, "`args` must be an object")),
};
prepare_args(&mut args, session).map_err(|message| PlanError::step(index, message))?;
+ let (window, element) = locate::take_selector(args.as_object_mut().expect("object"))
+ .map_err(|message| PlanError::step(index, format!("{tool}: {message}")))?;
crate::tool::normalize_argument_aliases(tool, &mut args)
.map_err(|message| PlanError::step(index, format!("{tool}: {message}")))?;
- validate_against_schema(registry, tool, &args)
+ if !element.is_empty() {
+ if tool == "drag" {
+ return Err(PlanError::step(
+ index,
+ "drag takes coordinates; it cannot target an element by name",
+ ));
+ }
+ if let Some(conflict) = ["element_token", "element_index", "x", "y"]
+ .iter()
+ .find(|key| args.get(**key).is_some())
+ {
+ return Err(PlanError::step(
+ index,
+ format!("{tool}: target by role/name OR by `{conflict}`, not both"),
+ ));
+ }
+ }
+ // Check the arguments now, with stand-ins for what the lookup fills.
+ let mut probe = args.clone();
+ if let Some(object) = probe.as_object_mut() {
+ let needs_window = !element.is_empty() || window.app.is_some() || window.title.is_some();
+ if needs_window {
+ object.entry("pid").or_insert(json!(1));
+ if accepts(registry, tool, "window_id") {
+ object.entry("window_id").or_insert(json!(1));
+ }
+ }
+ if !element.is_empty() {
+ object.insert("element_token".into(), json!("s00000001:1"));
+ }
+ }
+ validate_against_schema(registry, tool, &probe)
.map_err(|message| PlanError::step(index, format!("{tool}: {message}")))?;
- Ok(Step { tool, args })
+ Ok(Action {
+ tool,
+ args,
+ window,
+ element,
+ find_timeout: Duration::from_millis(find_timeout),
+ })
}
fn parse_observe(
registry: &ToolRegistry,
value: &Value,
- steps: &[Step],
+ has_window: bool,
session: Option<&str>,
) -> Result<Value, PlanError> {
let mut args = match value {
@@ -462,22 +1105,12 @@ fn parse_observe(
prepare_args(&mut args, session)
.map_err(|message| PlanError::batch(format!("observe: {message}")))?;
let object = args.as_object_mut().expect("object");
- // Inherit the window from the last step that names one, so the common
+ // The window comes from the last window the batch used, so the common
// "act, then look at the same window" case needs no repetition.
- let inherited = steps.iter().rev().find_map(|step| {
- let pid = step.args.get("pid").and_then(Value::as_i64)?;
- let window_id = step.args.get("window_id").and_then(Value::as_i64)?;
- Some((pid, window_id))
- });
- if let Some((pid, window_id)) = inherited {
- object.entry("pid").or_insert_with(|| json!(pid));
- object
- .entry("window_id")
- .or_insert_with(|| json!(window_id));
- }
- if !object.contains_key("pid") || !object.contains_key("window_id") {
+ let explicit = object.contains_key("pid") && object.contains_key("window_id");
+ if !explicit && !has_window {
return Err(PlanError::batch(
- "observe: `pid` and `window_id` are required (no step names both to inherit from)",
+ "observe: `pid` and `window_id` are required (no step names a window to inherit from)",
));
}
object
@@ -501,7 +1134,12 @@ fn parse_observe(
}
_ => {}
}
- validate_against_schema(registry, OBSERVE_TOOL, &args)
+ let mut probe = args.clone();
+ if let Some(object) = probe.as_object_mut() {
+ object.entry("pid").or_insert(json!(1));
+ object.entry("window_id").or_insert(json!(1));
+ }
+ validate_against_schema(registry, OBSERVE_TOOL, &probe)
.map_err(|message| PlanError::batch(format!("observe: {message}")))?;
Ok(args)
}
@@ -571,3 +1209,6 @@ fn bounded_message(result: &ToolResult) -> String {
#[cfg(test)]
mod tests;
+
+#[cfg(test)]
+mod flow_tests;
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/flow_tests.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/flow_tests.rs
new file mode 100644
index 0000000000..c3f03a1d01
--- /dev/null
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/flow_tests.rs
@@ -0,0 +1,546 @@
+//! `run_actions` v2 against a fake two-screen app: steps that name their
+//! target, waits, checks and the end-of-batch summary.
+
+use std::sync::{Arc, Mutex};
+use std::time::Duration;
+
+use async_trait::async_trait;
+use serde_json::{json, Value};
+
+use crate::{
+ protocol::{Content, ToolResult},
+ tool::{Tool, ToolDef, ToolRegistry},
+};
+
+/// What the fake app shows.
+#[derive(Default)]
+struct Ui {
+ screen: u8,
+ email: String,
+ dialog_open: bool,
+ /// A transient popup (no useful controls) stacked on top.
+ bubble_open: bool,
+ /// Reads left before the "Finish" button appears on screen 2.
+ finish_delay_reads: u32,
+ snapshot: u32,
+ /// Element labels of the latest read, by element index.
+ last_labels: Vec<(String, String)>,
+ calls: Vec<(String, Value)>,
+}
+
+impl Ui {
+ fn elements(&mut self, window_id: u64) -> Vec<(&'static str, String, Option<String>)> {
+ if window_id == 9 {
+ return vec![("AXButton", "Done".into(), None)];
+ }
+ if window_id == 8 {
+ return vec![
+ ("AXButton", "Discard".into(), None),
+ ("AXButton", "Keep".into(), None),
+ ];
+ }
+ match self.screen {
+ 0 => vec![
+ ("AXTextField", "Email".into(), Some(self.email.clone())),
+ ("AXButton", "Next".into(), None),
+ ("AXButton", "OK".into(), None),
+ ("AXButton", "OK".into(), None),
+ ],
+ _ => {
+ let mut rows = vec![("AXButton", "Back".into(), None)];
+ if self.finish_delay_reads == 0 {
+ rows.push(("AXButton", "Finish".into(), None));
+ } else {
+ self.finish_delay_reads -= 1;
+ }
+ rows
+ }
+ }
+ }
+}
+
+struct Fake {
+ def: ToolDef,
+ ui: Arc<Mutex<Ui>>,
+}
+
+#[async_trait]
+impl Tool for Fake {
+ fn def(&self) -> &ToolDef {
+ &self.def
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let mut ui = self.ui.lock().unwrap();
+ let name = self.def.name.clone();
+ ui.calls.push((name.clone(), args.clone()));
+ match name.as_str() {
+ "list_windows" => {
+ let mut windows = vec![json!({
+ "pid": 42, "window_id": 7, "app_name": "Demo", "title": "Main",
+ "z_index": 1, "is_on_screen": true,
+ "bounds": {"x": 0.0, "y": 0.0, "width": 800.0, "height": 600.0}
+ })];
+ if ui.dialog_open {
+ windows.push(json!({
+ "pid": 42, "window_id": 8, "app_name": "Demo", "title": "Unsaved changes",
+ "z_index": 2, "is_on_screen": true,
+ "bounds": {"x": 100.0, "y": 100.0, "width": 300.0, "height": 200.0}
+ }));
+ }
+ if ui.bubble_open {
+ windows.push(json!({
+ "pid": 42, "window_id": 9, "app_name": "Demo", "title": "Bookmark added",
+ "z_index": 3, "is_on_screen": true,
+ "bounds": {"x": 500.0, "y": 0.0, "width": 200.0, "height": 100.0}
+ }));
+ }
+ ToolResult::text("windows").with_structured(json!({ "windows": windows }))
+ }
+ "list_apps" => ToolResult::text("apps").with_structured(json!({
+ "apps": [{"pid": 42, "name": "Demo", "bundle_id": "com.example.demo", "running": true, "active": false}]
+ })),
+ "get_window_state" => {
+ let window_id = args["window_id"].as_u64().unwrap_or(7);
+ ui.snapshot += 1;
+ let snapshot = ui.snapshot;
+ let rows = ui.elements(window_id);
+ let mut markdown = String::from("- [0] AXWindow \"Demo\"\n");
+ let mut elements = Vec::new();
+ ui.last_labels.clear();
+ for (offset, (role, label, value)) in rows.iter().enumerate() {
+ let index = offset + 1;
+ markdown.push_str(&format!(" - [{index}] {role} \"{label}\"\n"));
+ let mut element = json!({
+ "element_index": index,
+ "element_token": format!("s{snapshot:08x}:{index}"),
+ "role": role,
+ "depth": 1,
+ "label": label,
+ "frame": {"x": 0, "y": 0, "w": 10, "h": 10},
+ });
+ if let Some(value) = value.as_ref().filter(|value| !value.is_empty()) {
+ element["value"] = json!(value);
+ }
+ elements.push(element);
+ ui.last_labels
+ .push((format!("s{snapshot:08x}:{index}"), label.clone()));
+ }
+ if ui.screen == 1 && window_id == 7 {
+ markdown.push_str(" - AXStaticText \"Step 2 of 2\"\n");
+ }
+ ToolResult::text(markdown.clone()).with_structured(json!({
+ "pid": 42,
+ "window_id": window_id,
+ "snapshot_id": format!("s{snapshot:08x}"),
+ "elements": elements,
+ "tree_markdown": markdown,
+ }))
+ }
+ "click" | "set_value" => {
+ let token = args["element_token"].as_str().unwrap_or_default();
+ let label = ui
+ .last_labels
+ .iter()
+ .find(|(candidate, _)| candidate == token)
+ .map(|(_, label)| label.clone());
+ let Some(label) = label else {
+ return ToolResult::error(format!("{name}: stale or unknown element_token {token}"));
+ };
+ match (name.as_str(), label.as_str()) {
+ ("set_value", "Email") => {
+ ui.email = args["value"].as_str().unwrap_or_default().to_owned()
+ }
+ ("click", "Next") => ui.screen = 1,
+ ("click", "Back") => ui.dialog_open = true,
+ ("click", "Keep") | ("click", "Discard") => ui.dialog_open = false,
+ _ => {}
+ }
+ ToolResult::text(format!("{name} {label} done"))
+ }
+ other => ToolResult::text(format!("{other} done")),
+ }
+ }
+}
+
+struct Harness {
+ registry: Arc<ToolRegistry>,
+ ui: Arc<Mutex<Ui>>,
+}
+
+impl Harness {
+ fn new() -> Self {
+ let ui = Arc::new(Mutex::new(Ui::default()));
+ let mut registry = ToolRegistry::new();
+ for name in super::BATCHABLE_TOOLS.iter().copied().chain([
+ "get_window_state",
+ "list_windows",
+ "list_apps",
+ ]) {
+ registry.register(Box::new(Fake {
+ def: ToolDef {
+ name: name.into(),
+ description: "fake".into(),
+ input_schema: json!({
+ "type": "object",
+ "properties": {
+ "pid": {"type": "integer"},
+ "window_id": {"type": "integer"},
+ "element_token": {"type": "string"},
+ "value": {"type": "string"},
+ "key": {"type": "string"},
+ "x": {"type": "number"},
+ "y": {"type": "number"}
+ }
+ }),
+ read_only: matches!(name, "get_window_state" | "list_windows" | "list_apps"),
+ destructive: false,
+ idempotent: false,
+ open_world: false,
+ },
+ ui: ui.clone(),
+ }));
+ }
+ registry.register_session_tools();
+ let registry = Arc::new(registry);
+ registry.init_self_weak();
+ Self { registry, ui }
+ }
+
+ async fn run(&self, args: Value) -> ToolResult {
+ self.registry.invoke(super::RUN_ACTIONS_TOOL, args).await
+ }
+
+ fn calls(&self, tool: &str) -> Vec<Value> {
+ self.ui
+ .lock()
+ .unwrap()
+ .calls
+ .iter()
+ .filter(|(name, _)| name == tool)
+ .map(|(_, args)| args.clone())
+ .collect()
+ }
+}
+
+fn text(result: &ToolResult) -> String {
+ result
+ .content
+ .iter()
+ .filter_map(|content| match content {
+ Content::Text { text, .. } => Some(text.as_str()),
+ _ => None,
+ })
+ .collect::<Vec<_>>()
+ .join("\n")
+}
+
+#[tokio::test]
+async fn one_batch_crosses_screens_by_name() {
+ let harness = Harness::new();
+ let result = harness
+ .run(json!({
+ "steps": [
+ {"set_value": {"app": "Demo", "role": "textfield", "name": "email", "value": "[email protected]"},
+ "expect": {"role": "textfield", "name": "Email", "value": "[email protected]"}},
+ {"click": {"role": "button", "name": "Next"}, "expect": {"text": "Step 2"}},
+ {"tool": "click", "args": {"role": "button", "name": "Back"},
+ "expect": {"app": "Demo", "window": "Unsaved", "role": "button", "name": "Keep"}},
+ {"click": {"role": "button", "name": "Keep"},
+ "expect": {"name": "Keep", "gone": true}}
+ ],
+ "observe": true
+ }))
+ .await;
+ assert_ne!(result.is_error, Some(true), "{}", text(&result));
+ let structured = result.structured_content.clone().unwrap();
+ assert_eq!(structured["executed"], 4);
+ // The Keep button lives in the dialog window that opened on top: the
+ // app-level target found it there without a window id.
+ assert_eq!(structured["steps"][3]["target"]["window"]["window_id"], 8);
+ assert!(
+ text(&result).contains("on [2] AXButton \"Keep\""),
+ "{}",
+ text(&result)
+ );
+
+ let clicks = harness.calls("click");
+ assert_eq!(clicks.len(), 3);
+ assert_eq!(clicks[0]["pid"], 42);
+ assert_eq!(clicks[0]["window_id"], 7);
+ assert!(
+ clicks[0].get("role").is_none(),
+ "selector keys never reach the tool"
+ );
+ assert!(clicks[0]["element_token"].as_str().unwrap().ends_with(":2"));
+ assert_eq!(harness.ui.lock().unwrap().email, "[email protected]");
+
+ // Lookup reads ask for full_output, which `since` never remembers; the
+ // end observation is the one diff read, of the last window used.
+ let reads = harness.calls("get_window_state");
+ let observed = reads.last().unwrap();
+ assert_eq!(observed["since"], "latest");
+ // Lookup reads are full_output, which `since` never remembers; the only
+ // other reads are one plain baseline per window the batch looked in.
+ let baselines: Vec<u64> = reads[..reads.len() - 1]
+ .iter()
+ .filter(|read| read["full_output"] != true)
+ .map(|read| read["window_id"].as_u64().unwrap())
+ .collect();
+ assert_eq!(baselines, [7, 8], "{reads:?}");
+}
+
+#[tokio::test]
+async fn several_matches_fail_with_the_candidates_and_nth_picks_one() {
+ let harness = Harness::new();
+ let result = harness
+ .run(json!({"steps": [
+ {"click": {"app": "Demo", "role": "button", "name": "OK"}},
+ {"click": {"role": "button", "name": "Next"}}
+ ]}))
+ .await;
+ assert_eq!(result.is_error, Some(true));
+ let structured = result.structured_content.clone().unwrap();
+ assert_eq!(structured["failed_step"], 0);
+ assert_eq!(structured["steps"][0]["phase"], "find");
+ assert_eq!(structured["steps"][0]["code"], "ambiguous");
+ let message = text(&result);
+ assert!(message.contains("2 elements match"), "{message}");
+ assert!(message.contains("nth 1: [4] AXButton \"OK\""), "{message}");
+ assert!(harness.calls("click").is_empty(), "nothing clicked");
+
+ let result = harness
+ .run(json!({"steps": [{"click": {"app": "Demo", "role": "button", "name": "OK", "nth": 1}}]}))
+ .await;
+ assert_ne!(result.is_error, Some(true), "{}", text(&result));
+ assert!(harness.calls("click")[0]["element_token"]
+ .as_str()
+ .unwrap()
+ .ends_with(":4"));
+}
+
+#[tokio::test]
+async fn a_missing_target_waits_then_names_the_nearest_elements() {
+ let harness = Harness::new();
+ let started = std::time::Instant::now();
+ let result = harness
+ .run(json!({"steps": [
+ {"click": {"app": "com.example.demo", "role": "button", "name": "Submit"}, "timeout_ms": 600}
+ ]}))
+ .await;
+ let elapsed = started.elapsed();
+ assert_eq!(result.is_error, Some(true));
+ let message = text(&result);
+ assert!(
+ message.contains("1. click ERROR (find): no button \"Submit\""),
+ "{message}"
+ );
+ assert!(
+ message.contains("nearest: [2] AXButton \"Next\""),
+ "{message}"
+ );
+ assert!(message.contains("waited"), "{message}");
+ assert!(elapsed >= Duration::from_millis(500), "{elapsed:?}");
+ assert!(elapsed < Duration::from_millis(2_000), "{elapsed:?}");
+}
+
+#[tokio::test]
+async fn a_named_target_that_appears_late_is_waited_for() {
+ let harness = Harness::new();
+ harness.ui.lock().unwrap().screen = 1;
+ harness.ui.lock().unwrap().finish_delay_reads = 2;
+ let result = harness
+ .run(json!({"steps": [
+ {"wait_for": {"app": "Demo", "text": "Step 2"}},
+ {"click": {"role": "button", "name": "Finish"}}
+ ]}))
+ .await;
+ assert_ne!(result.is_error, Some(true), "{}", text(&result));
+ assert_eq!(harness.calls("click").len(), 1);
+}
+
+#[tokio::test]
+async fn a_failed_expect_stops_the_batch_and_says_what_was_found() {
+ let harness = Harness::new();
+ let result = harness
+ .run(json!({"steps": [
+ {"set_value": {"app": "Demo", "role": "textfield", "name": "Email", "value": "bob"},
+ "expect": {"name": "Email", "value": "ada", "timeout_ms": 300}},
+ {"click": {"role": "button", "name": "Next"}}
+ ], "observe": true}))
+ .await;
+ assert_eq!(result.is_error, Some(true));
+ let structured = result.structured_content.clone().unwrap();
+ assert_eq!(structured["failed_step"], 0);
+ assert_eq!(structured["steps"][0]["phase"], "expect");
+ assert_eq!(structured["steps"][0]["code"], "unexpected_state");
+ let message = text(&result);
+ assert!(message.contains("value=\"bob\""), "{message}");
+ assert!(harness.calls("click").is_empty(), "later steps never run");
+ // The failed batch is still observed, in the window it was working in.
+ assert_eq!(structured["observation"]["ok"], true);
+ assert_eq!(
+ harness.calls("get_window_state").last().unwrap()["window_id"],
+ 7
+ );
+}
+
+#[tokio::test]
+async fn a_check_only_step_and_gone_on_a_closed_window() {
+ let harness = Harness::new();
+ let result = harness
+ .run(json!({"steps": [
+ {"expect": {"app": "Demo", "window": "Unsaved", "name": "Keep", "gone": true, "timeout_ms": 0}}
+ ]}))
+ .await;
+ assert_ne!(result.is_error, Some(true), "{}", text(&result));
+ assert!(harness.calls("click").is_empty());
+}
+
+#[tokio::test]
+async fn bad_v2_steps_are_rejected_before_anything_runs() {
+ let harness = Harness::new();
+ for (step, needle) in [
+ (json!({"click": {"pid": 1}, "args": {}}), "drop `args`"),
+ (json!({"click": {}, "press_key": {}}), "one action per step"),
+ (
+ json!({"drag": {"app": "Demo", "name": "x"}}),
+ "drag takes coordinates",
+ ),
+ (
+ json!({"click": {"app": "Demo", "name": "x", "element_token": "s1:1"}}),
+ "not both",
+ ),
+ (json!({"click": {"name": "Next"}}), "no window"),
+ (
+ json!({"wait_for": {"app": "Demo"}}),
+ "needs `role`, `name` or `text`",
+ ),
+ (json!({"expect": []}), "1 to 4 checks"),
+ (json!({}), "empty step"),
+ (
+ json!({"click": {"app": "Demo", "name": "x"}, "timeout_ms": 99999}),
+ "timeout_ms",
+ ),
+ (json!({"wait": {"app": "Demo"}}), "cannot run in a batch"),
+ ] {
+ let result = harness
+ .run(json!({"steps": [{"press_key": {"key": "a"}}, step.clone()]}))
+ .await;
+ assert_eq!(result.is_error, Some(true), "{step}");
+ let message = text(&result);
+ assert!(message.contains("step 2"), "{step}: {message}");
+ assert!(message.contains(needle), "{step}: {message}");
+ }
+ assert!(harness.calls("press_key").is_empty());
+ assert!(harness.calls("list_windows").is_empty());
+}
+
+#[tokio::test]
+async fn later_steps_inherit_the_window() {
+ let harness = Harness::new();
+ let result = harness
+ .run(json!({"steps": [
+ {"click": {"app": "Demo", "role": "textfield", "name": "Email"}},
+ {"press_key": {"key": "tab"}}
+ ]}))
+ .await;
+ assert_ne!(result.is_error, Some(true), "{}", text(&result));
+ let pressed = &harness.calls("press_key")[0];
+ assert_eq!(pressed["pid"], 42);
+ assert_eq!(pressed["window_id"], 7);
+}
+
+#[tokio::test]
+async fn a_popup_on_top_does_not_hide_the_main_window() {
+ let harness = Harness::new();
+ harness.ui.lock().unwrap().bubble_open = true;
+ let result = harness
+ .run(json!({"steps": [
+ {"click": {"app": "Demo", "role": "button", "name": "Next"}, "expect": {"text": "Step 2"}}
+ ], "observe": true}))
+ .await;
+ assert_ne!(result.is_error, Some(true), "{}", text(&result));
+ let structured = result.structured_content.clone().unwrap();
+ assert_eq!(structured["steps"][0]["target"]["window"]["window_id"], 7);
+ // The observation follows the window the batch acted in, not the popup.
+ assert_eq!(
+ harness.calls("get_window_state").last().unwrap()["window_id"],
+ 7
+ );
+
+ // A miss names the window with the most elements, and says how many
+ // windows were searched.
+ let result = harness
+ .run(json!({"steps": [
+ {"click": {"app": "Demo", "role": "button", "name": "Publish"}, "timeout_ms": 0}
+ ]}))
+ .await;
+ let message = text(&result);
+ assert!(message.contains("searched 2 windows"), "{message}");
+ assert!(message.contains("window 7"), "{message}");
+}
+
+#[tokio::test]
+async fn a_check_only_step_names_the_window_to_observe() {
+ let harness = Harness::new();
+ let result = harness
+ .run(json!({"steps": [{"expect": {"app": "Demo", "text": "Email", "timeout_ms": 0}}], "observe": true}))
+ .await;
+ assert_ne!(result.is_error, Some(true), "{}", text(&result));
+ assert_eq!(
+ result.structured_content.unwrap()["observation"]["ok"],
+ true
+ );
+ assert_eq!(
+ harness.calls("get_window_state").last().unwrap()["window_id"],
+ 7
+ );
+}
+
+#[tokio::test]
+async fn the_end_observation_diffs_against_the_batch_start() {
+ let harness = Harness::new();
+ let result = harness
+ .run(json!({"steps": [
+ {"set_value": {"app": "Demo", "role": "textfield", "name": "Email", "value": "[email protected]"}}
+ ], "observe": true}))
+ .await;
+ assert_ne!(result.is_error, Some(true), "{}", text(&result));
+ let reads = harness.calls("get_window_state");
+ // First a plain read with the observation's budget (remembered by
+ // `since`), then the full lookup read, then the end diff.
+ assert_eq!(reads[0]["max_elements"], 250);
+ assert!(reads[0].get("full_output").is_none());
+ assert_eq!(reads[1]["full_output"], true);
+ let last = reads.last().unwrap();
+ assert_eq!(last["since"], "latest");
+ assert_eq!(last["max_elements"], 250);
+ assert_eq!(reads.len(), 3, "one baseline per window, not per poll");
+
+ // Without observe there is no baseline read.
+ let before = harness.calls("get_window_state").len();
+ harness
+ .run(json!({"steps": [{"click": {"app": "Demo", "role": "button", "name": "Next"}}]}))
+ .await;
+ assert_eq!(harness.calls("get_window_state").len(), before + 1);
+}
+
+#[tokio::test]
+async fn a_trailing_get_window_state_in_shorthand_is_the_observation() {
+ let harness = Harness::new();
+ let result = harness
+ .run(json!({"steps": [
+ {"click": {"app": "Demo", "role": "button", "name": "Next"}},
+ {"get_window_state": {}}
+ ]}))
+ .await;
+ assert_ne!(result.is_error, Some(true), "{}", text(&result));
+ let structured = result.structured_content.unwrap();
+ assert_eq!(structured["total"], 1, "the read is not a step");
+ assert_eq!(structured["observation"]["ok"], true);
+ assert_eq!(
+ harness.calls("get_window_state").last().unwrap()["window_id"],
+ 7
+ );
+}
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/locate.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/locate.rs
new file mode 100644
index 0000000000..1c2287fbb6
--- /dev/null
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/locate.rs
@@ -0,0 +1,1561 @@
+//! Find windows and elements by description, wait for them, and check them.
+//!
+//! Used by `run_actions` steps that name their target (`role`, `name`, `app`,
+//! `window`) instead of carrying a token from an earlier read, and by the
+//! per-step `wait_for` / `expect` checks. Every read goes through the same
+//! [`ToolRegistry`] entry point a direct call uses (`list_windows`,
+//! `list_apps`, `get_window_state`), so a lookup is held to the same session,
+//! policy and approval checks as the reads a model would make itself.
+//!
+//! Lookup reads ask for `full_output`, which the `since` snapshot memory does
+//! not remember, so they never become the baseline of the caller's next
+//! `since:"latest"` diff.
+
+use std::time::{Duration, Instant};
+
+use serde_json::{json, Map, Value};
+
+use crate::{
+ protocol::{Content, ToolResult},
+ tool::ToolRegistry,
+};
+
+/// Keys a step may add to an action's arguments to name its target. They are
+/// removed before the action tool runs.
+pub(crate) const SELECTOR_KEYS: &[&str] = &["role", "name", "label", "nth", "app", "window"];
+/// Keys a `wait_for` / `expect` check accepts.
+pub(crate) const CHECK_KEYS: &[&str] = &[
+ "role",
+ "name",
+ "label",
+ "text",
+ "nth",
+ "app",
+ "window",
+ "pid",
+ "window_id",
+ "gone",
+ "value",
+ "value_contains",
+ "enabled",
+ "selected",
+ "timeout_ms",
+];
+
+/// Default wait for a named target to appear before its action runs.
+pub(crate) const DEFAULT_FIND_TIMEOUT_MS: u64 = 3_000;
+/// Default wait for a `wait_for` check.
+pub(crate) const DEFAULT_WAIT_TIMEOUT_MS: u64 = 5_000;
+/// Default wait for an `expect` check.
+pub(crate) const DEFAULT_EXPECT_TIMEOUT_MS: u64 = 2_000;
+/// Longest wait any single check may ask for.
+pub(crate) const MAX_CHECK_TIMEOUT_MS: u64 = 10_000;
+const POLL_INTERVAL: Duration = Duration::from_millis(250);
+const CANDIDATES_SHOWN: usize = 5;
+/// Most windows of one app a lookup reads per poll.
+const MAX_WINDOWS_SEARCHED: usize = 4;
+
+/// Which window a step or check addresses.
+#[derive(Clone, Debug, Default, PartialEq, Eq)]
+pub(crate) struct WindowSpec {
+ pub pid: Option<i64>,
+ pub window_id: Option<u64>,
+ /// App name (case-insensitive, exact or substring) or bundle id.
+ pub app: Option<String>,
+ /// Case-insensitive substring of the window title.
+ pub title: Option<String>,
+}
+
+impl WindowSpec {
+ pub fn is_empty(&self) -> bool {
+ self.pid.is_none() && self.window_id.is_none() && self.app.is_none() && self.title.is_none()
+ }
+
+ /// A spec that names one exact window needs no lookup.
+ fn exact(&self) -> Option<(i64, u64)> {
+ match (self.pid, self.window_id, &self.app, &self.title) {
+ (Some(pid), Some(window_id), None, None) => Some((pid, window_id)),
+ _ => None,
+ }
+ }
+
+ pub fn describe(&self) -> String {
+ let mut parts = Vec::new();
+ if let Some(app) = &self.app {
+ parts.push(format!("app \"{app}\""));
+ }
+ if let Some(title) = &self.title {
+ parts.push(format!("window \"{title}\""));
+ }
+ if let Some(pid) = self.pid {
+ parts.push(format!("pid {pid}"));
+ }
+ if let Some(window_id) = self.window_id {
+ parts.push(format!("window_id {window_id}"));
+ }
+ if parts.is_empty() {
+ "no window".to_owned()
+ } else {
+ parts.join(", ")
+ }
+ }
+}
+
+/// Which element inside the window.
+#[derive(Clone, Debug, Default, PartialEq, Eq)]
+pub(crate) struct ElementSpec {
+ pub role: Option<String>,
+ pub name: Option<String>,
+ /// Any row of the tree containing this text, display-only rows included.
+ pub text: Option<String>,
+ pub nth: Option<usize>,
+}
+
+impl ElementSpec {
+ pub fn is_empty(&self) -> bool {
+ self.role.is_none() && self.name.is_none() && self.text.is_none()
+ }
+
+ pub fn describe(&self) -> String {
+ let mut out = String::new();
+ if let Some(role) = &self.role {
+ out.push_str(role);
+ } else {
+ out.push_str("element");
+ }
+ if let Some(name) = &self.name {
+ out.push_str(&format!(" \"{name}\""));
+ }
+ if let Some(text) = &self.text {
+ out.push_str(&format!(" with text \"{text}\""));
+ }
+ if let Some(nth) = self.nth {
+ out.push_str(&format!(" (nth {nth})"));
+ }
+ out
+ }
+}
+
+/// A `wait_for` or `expect` predicate.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub(crate) struct Check {
+ pub window: WindowSpec,
+ pub element: ElementSpec,
+ pub gone: bool,
+ pub value: Option<String>,
+ pub value_contains: Option<String>,
+ pub enabled: Option<bool>,
+ pub selected: Option<bool>,
+ pub timeout: Duration,
+}
+
+impl Check {
+ pub fn parse(raw: &Value, default_timeout_ms: u64) -> Result<Self, String> {
+ let object = raw
+ .as_object()
+ .ok_or("must be an object such as {\"role\":\"button\",\"name\":\"Save\"}")?;
+ if let Some(key) = object
+ .keys()
+ .find(|key| !CHECK_KEYS.contains(&key.as_str()))
+ {
+ return Err(format!(
+ "unknown field `{key}`; a check takes {}",
+ CHECK_KEYS.join(", ")
+ ));
+ }
+ let element = element_spec(object)?;
+ if element.is_empty() {
+ return Err("needs `role`, `name` or `text` to say which element".to_owned());
+ }
+ let window = window_spec(object)?;
+ let gone = opt_bool(object, "gone")?.unwrap_or(false);
+ let value = opt_string(object, "value")?;
+ let value_contains = opt_string(object, "value_contains")?;
+ let enabled = opt_bool(object, "enabled")?;
+ let selected = opt_bool(object, "selected")?;
+ if gone
+ && (value.is_some()
+ || value_contains.is_some()
+ || enabled.is_some()
+ || selected.is_some())
+ {
+ return Err("`gone:true` cannot be combined with value/enabled/selected".to_owned());
+ }
+ if element.text.is_some()
+ && element.role.is_none()
+ && element.name.is_none()
+ && (value.is_some()
+ || value_contains.is_some()
+ || enabled.is_some()
+ || selected.is_some())
+ {
+ return Err(
+ "value/enabled/selected need `role` or `name`; `text` alone only checks that the text is shown"
+ .to_owned(),
+ );
+ }
+ let timeout_ms = match object.get("timeout_ms") {
+ None | Some(Value::Null) => default_timeout_ms,
+ Some(value) => value
+ .as_u64()
+ .filter(|ms| *ms <= MAX_CHECK_TIMEOUT_MS)
+ .ok_or_else(|| {
+ format!("`timeout_ms` must be an integer 0 to {MAX_CHECK_TIMEOUT_MS}")
+ })?,
+ };
+ Ok(Self {
+ window,
+ element,
+ gone,
+ value,
+ value_contains,
+ enabled,
+ selected,
+ timeout: Duration::from_millis(timeout_ms),
+ })
+ }
+
+ pub fn describe(&self) -> String {
+ let mut out = self.element.describe();
+ if self.gone {
+ out.push_str(" is gone");
+ } else {
+ out.push_str(" is shown");
+ }
+ if let Some(value) = &self.value {
+ out.push_str(&format!(", value \"{value}\""));
+ }
+ if let Some(value) = &self.value_contains {
+ out.push_str(&format!(", value contains \"{value}\""));
+ }
+ if let Some(enabled) = self.enabled {
+ out.push_str(if enabled { ", enabled" } else { ", disabled" });
+ }
+ if let Some(selected) = self.selected {
+ out.push_str(if selected {
+ ", selected"
+ } else {
+ ", not selected"
+ });
+ }
+ out
+ }
+}
+
+/// Split the selector keys out of an action's arguments.
+pub(crate) fn take_selector(
+ args: &mut Map<String, Value>,
+) -> Result<(WindowSpec, ElementSpec), String> {
+ let mut picked = Map::new();
+ for key in SELECTOR_KEYS {
+ if let Some(value) = args.remove(*key) {
+ picked.insert((*key).to_owned(), value);
+ }
+ }
+ let element = element_spec(&picked)?;
+ if element.text.is_some() {
+ return Err(
+ "`text` only works in wait_for/expect; target an element by role and name".to_owned(),
+ );
+ }
+ let mut window = window_spec(&picked)?;
+ // A malformed pid/window_id is left to the tool's schema check, which
+ // names the field the same way a direct call would.
+ window.pid = args.get("pid").and_then(Value::as_i64);
+ window.window_id = args.get("window_id").and_then(Value::as_u64);
+ Ok((window, element))
+}
+
+fn element_spec(object: &Map<String, Value>) -> Result<ElementSpec, String> {
+ let name = match (opt_string(object, "name")?, opt_string(object, "label")?) {
+ (Some(_), Some(_)) => return Err("give `name` or `label`, not both".to_owned()),
+ (name, label) => name.or(label),
+ };
+ let nth = match object.get("nth") {
+ None | Some(Value::Null) => None,
+ Some(value) => Some(
+ value
+ .as_u64()
+ .ok_or("`nth` must be a non-negative integer (0 is the first match)")?
+ as usize,
+ ),
+ };
+ Ok(ElementSpec {
+ role: opt_string(object, "role")?,
+ name,
+ text: opt_string(object, "text")?,
+ nth,
+ })
+}
+
+fn window_spec(object: &Map<String, Value>) -> Result<WindowSpec, String> {
+ Ok(WindowSpec {
+ pid: opt_i64(object, "pid")?,
+ window_id: opt_u64(object, "window_id")?,
+ app: opt_string(object, "app")?,
+ title: opt_string(object, "window")?,
+ })
+}
+
+fn opt_string(object: &Map<String, Value>, key: &str) -> Result<Option<String>, String> {
+ match object.get(key) {
+ None | Some(Value::Null) => Ok(None),
+ Some(Value::String(text)) if !text.trim().is_empty() => Ok(Some(text.trim().to_owned())),
+ Some(_) => Err(format!("`{key}` must be a non-empty string")),
+ }
+}
+
+fn opt_bool(object: &Map<String, Value>, key: &str) -> Result<Option<bool>, String> {
+ match object.get(key) {
+ None | Some(Value::Null) => Ok(None),
+ Some(Value::Bool(flag)) => Ok(Some(*flag)),
+ Some(_) => Err(format!("`{key}` must be true or false")),
+ }
+}
+
+fn opt_i64(object: &Map<String, Value>, key: &str) -> Result<Option<i64>, String> {
+ match object.get(key) {
+ None | Some(Value::Null) => Ok(None),
+ Some(value) => value
+ .as_i64()
+ .map(Some)
+ .ok_or_else(|| format!("`{key}` must be an integer")),
+ }
+}
+
+fn opt_u64(object: &Map<String, Value>, key: &str) -> Result<Option<u64>, String> {
+ match object.get(key) {
+ None | Some(Value::Null) => Ok(None),
+ Some(value) => value
+ .as_u64()
+ .map(Some)
+ .ok_or_else(|| format!("`{key}` must be a non-negative integer")),
+ }
+}
+
+/// One concrete window.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub(crate) struct Window {
+ pub pid: i64,
+ pub window_id: u64,
+ pub app: Option<String>,
+ pub title: Option<String>,
+ pub on_screen: bool,
+}
+
+impl Window {
+ pub fn describe(&self) -> String {
+ let mut out = String::new();
+ if let Some(app) = &self.app {
+ out.push_str(app);
+ out.push(' ');
+ }
+ out.push_str(&format!("window {}", self.window_id));
+ if let Some(title) = self.title.as_deref().filter(|title| !title.is_empty()) {
+ out.push_str(&format!(" \"{}\"", clip(title, 60)));
+ }
+ out
+ }
+
+ pub fn to_json(&self) -> Value {
+ json!({
+ "pid": self.pid,
+ "window_id": self.window_id,
+ "app": self.app,
+ "title": self.title,
+ })
+ }
+}
+
+/// One accessibility read of a window.
+pub(crate) struct Read {
+ pub window: Window,
+ pub elements: Vec<Value>,
+ pub markdown: String,
+}
+
+/// An element found by description.
+#[derive(Clone, Debug)]
+pub(crate) struct Found {
+ pub window: Window,
+ pub element: Value,
+}
+
+impl Found {
+ pub fn token(&self) -> Option<&str> {
+ self.element.get("element_token").and_then(Value::as_str)
+ }
+
+ pub fn describe(&self) -> String {
+ describe_element(&self.element)
+ }
+}
+
+/// Why a lookup or check did not succeed.
+#[derive(Clone, Debug)]
+pub(crate) struct Miss {
+ /// Stable machine-readable reason.
+ pub code: &'static str,
+ pub message: String,
+ pub window: Option<Window>,
+}
+
+impl Miss {
+ fn new(code: &'static str, message: impl Into<String>) -> Self {
+ Self {
+ code,
+ message: message.into(),
+ window: None,
+ }
+ }
+
+ /// Whether polling again can change the answer. A refused or failed
+ /// read, or an ambiguous match, will not fix itself, and retrying a
+ /// refused read would only repeat the refusal.
+ fn worth_waiting(&self) -> bool {
+ matches!(
+ self.code,
+ "not_found"
+ | "app_not_found"
+ | "window_not_found"
+ | "still_present"
+ | "unexpected_state"
+ )
+ }
+
+ fn in_window(mut self, window: &Window) -> Self {
+ self.window = Some(window.clone());
+ self
+ }
+}
+
+/// Outcome of a satisfied check.
+#[derive(Clone, Debug)]
+pub(crate) struct Checked {
+ pub window: Option<Window>,
+ pub observed: Option<String>,
+ pub waited: Duration,
+}
+
+/// Window and element lookup through the registry.
+pub(crate) struct Locator<'a> {
+ registry: &'a ToolRegistry,
+ session: Option<String>,
+ /// Hard stop for every poll loop (the batch's own time budget).
+ deadline: Instant,
+ /// When the batch ends with a default `observe`, the `max_elements` of
+ /// that read: the first lookup of each window also makes one plain
+ /// read with that budget, which `since:"latest"` remembers, so the end
+ /// observation is a diff of what the batch changed rather than a full
+ /// read of a window the caller never read.
+ baseline_max_elements: Option<u64>,
+ baselined: std::sync::Mutex<std::collections::HashSet<(i64, u64)>>,
+}
+
+impl<'a> Locator<'a> {
+ pub fn new(registry: &'a ToolRegistry, session: Option<String>, deadline: Instant) -> Self {
+ Self {
+ registry,
+ session,
+ deadline,
+ baseline_max_elements: None,
+ baselined: Default::default(),
+ }
+ }
+
+ pub fn with_observe_baseline(mut self, max_elements: u64) -> Self {
+ self.baseline_max_elements = Some(max_elements);
+ self
+ }
+
+ fn with_session(&self, mut args: Value) -> Value {
+ if let (Some(session), Some(object)) = (&self.session, args.as_object_mut()) {
+ object.insert("session".to_owned(), Value::String(session.clone()));
+ }
+ args
+ }
+
+ async fn call(&self, tool: &str, args: Value) -> Result<Value, String> {
+ let result = self.registry.invoke(tool, self.with_session(args)).await;
+ if result.is_error == Some(true) {
+ return Err(format!("{tool}: {}", first_text(&result)));
+ }
+ Ok(result.structured_content.unwrap_or(Value::Null))
+ }
+
+ /// Resolve a spec to one window. With only `app` or `pid`, the app's
+ /// frontmost window wins, so a dialog that opens on top is found.
+ pub async fn window(&self, spec: &WindowSpec) -> Result<Window, Miss> {
+ if let Some((pid, window_id)) = spec.exact() {
+ return Ok(Window {
+ pid,
+ window_id,
+ app: None,
+ title: None,
+ on_screen: true,
+ });
+ }
+ self.listed_window(spec).await
+ }
+
+ /// Like [`Self::window`], but always confirms the window through
+ /// `list_windows`, so the result carries its app name and title.
+ pub async fn listed_window(&self, spec: &WindowSpec) -> Result<Window, Miss> {
+ let mut windows = self.listed_windows(spec).await?;
+ Ok(windows.remove(0))
+ }
+
+ /// Every window matching `spec`, best first: on-screen before
+ /// off-screen, then topmost (largest z_index) first. Never empty.
+ pub async fn listed_windows(&self, spec: &WindowSpec) -> Result<Vec<Window>, Miss> {
+ if spec.is_empty() {
+ return Err(Miss::new(
+ "no_window",
+ "no window to look in: give `app` (or pid and window_id) on this step or an earlier one",
+ ));
+ }
+ let mut list_args = json!({"on_screen_only": false});
+ if let Some(pid) = spec.pid {
+ list_args["pid"] = json!(pid);
+ }
+ let listed = self
+ .call("list_windows", list_args)
+ .await
+ .map_err(|error| Miss::new("lookup_failed", error))?;
+ let windows: Vec<Value> = listed
+ .get("windows")
+ .and_then(Value::as_array)
+ .cloned()
+ .unwrap_or_default();
+ let mut candidates: Vec<&Value> = windows
+ .iter()
+ .filter(|window| window.get("pid").and_then(Value::as_i64).is_some())
+ .collect();
+ if let Some(app) = &spec.app {
+ let by_name: Vec<&Value> = candidates
+ .iter()
+ .copied()
+ .filter(|window| text_matches(window.get("app_name"), app))
+ .collect();
+ candidates = if by_name.is_empty() {
+ let pids = self.app_pids(app).await;
+ candidates
+ .into_iter()
+ .filter(|window| {
+ window
+ .get("pid")
+ .and_then(Value::as_i64)
+ .is_some_and(|pid| pids.contains(&pid))
+ })
+ .collect()
+ } else {
+ by_name
+ };
+ if candidates.is_empty() {
+ let mut apps: Vec<&str> = windows
+ .iter()
+ .filter_map(|window| window.get("app_name").and_then(Value::as_str))
+ .collect();
+ apps.sort_unstable();
+ apps.dedup();
+ return Err(Miss::new(
+ "app_not_found",
+ format!(
+ "no window of app \"{app}\" (apps with windows: {})",
+ if apps.is_empty() {
+ "none".to_owned()
+ } else {
+ apps.join(", ")
+ }
+ ),
+ ));
+ }
+ }
+ if let Some(window_id) = spec.window_id {
+ candidates.retain(|window| {
+ window.get("window_id").and_then(Value::as_u64) == Some(window_id)
+ });
+ }
+ if let Some(title) = &spec.title {
+ let before: Vec<String> = candidates
+ .iter()
+ .filter_map(|window| window.get("title").and_then(Value::as_str))
+ .map(|title| format!("\"{}\"", clip(title, 40)))
+ .collect();
+ candidates.retain(|window| text_matches(window.get("title"), title));
+ if candidates.is_empty() {
+ return Err(Miss::new(
+ "window_not_found",
+ format!(
+ "no window titled like \"{title}\" for {} (titles: {})",
+ spec.describe(),
+ if before.is_empty() {
+ "none".to_owned()
+ } else {
+ before.join(", ")
+ }
+ ),
+ ));
+ }
+ }
+ if candidates.is_empty() {
+ return Err(Miss::new(
+ "window_not_found",
+ format!("no window for {}", spec.describe()),
+ ));
+ }
+ // On-screen windows first, then the topmost (largest z_index).
+ let mut ranked: Vec<(usize, &Value)> = candidates.into_iter().enumerate().collect();
+ ranked.sort_by_key(|(order, window)| {
+ let on_screen = window.get("is_on_screen").and_then(Value::as_bool) != Some(false);
+ let z = window
+ .get("z_index")
+ .and_then(Value::as_i64)
+ .unwrap_or(i64::MIN);
+ std::cmp::Reverse((on_screen, z, std::cmp::Reverse(*order)))
+ });
+ Ok(ranked
+ .into_iter()
+ .map(|(_, window)| Window {
+ pid: window
+ .get("pid")
+ .and_then(Value::as_i64)
+ .unwrap_or_default(),
+ window_id: window
+ .get("window_id")
+ .and_then(Value::as_u64)
+ .unwrap_or_default(),
+ app: window
+ .get("app_name")
+ .and_then(Value::as_str)
+ .map(str::to_owned),
+ title: window
+ .get("title")
+ .and_then(Value::as_str)
+ .map(str::to_owned),
+ on_screen: window.get("is_on_screen").and_then(Value::as_bool) != Some(false),
+ })
+ .collect())
+ }
+
+ /// The windows a lookup searches, best first. An exact pid/window_id is
+ /// the only one; otherwise the app's on-screen windows in stacking
+ /// order, so an element in a dialog on top is found first and one in
+ /// the main window is still found when a transient popup is on top.
+ pub async fn search_windows(&self, spec: &WindowSpec) -> Result<Vec<Window>, Miss> {
+ if spec.exact().is_some() {
+ return Ok(vec![self.window(spec).await?]);
+ }
+ let windows = self.listed_windows(spec).await?;
+ let on_screen: Vec<Window> = windows.iter().filter(|w| w.on_screen).cloned().collect();
+ let mut pool = if on_screen.is_empty() {
+ windows
+ } else {
+ on_screen
+ };
+ pool.truncate(MAX_WINDOWS_SEARCHED);
+ Ok(pool)
+ }
+
+ /// Pids of running apps whose name or bundle id matches.
+ async fn app_pids(&self, app: &str) -> Vec<i64> {
+ let Ok(listed) = self.call("list_apps", json!({})).await else {
+ return Vec::new();
+ };
+ listed
+ .get("apps")
+ .and_then(Value::as_array)
+ .map(|apps| {
+ apps.iter()
+ .filter(|entry| {
+ entry
+ .get("bundle_id")
+ .and_then(Value::as_str)
+ .is_some_and(|bundle| bundle.eq_ignore_ascii_case(app))
+ || text_matches(entry.get("name"), app)
+ })
+ .filter_map(|entry| entry.get("pid").and_then(Value::as_i64))
+ .filter(|pid| *pid > 0)
+ .collect()
+ })
+ .unwrap_or_default()
+ }
+
+ /// One fresh accessibility read. It refreshes the window's element
+ /// tokens, so the tokens it returns are the ones to act with.
+ pub async fn read(&self, window: &Window) -> Result<Read, Miss> {
+ if let Some(max_elements) = self.baseline_max_elements {
+ let first = self
+ .baselined
+ .lock()
+ .unwrap_or_else(|e| e.into_inner())
+ .insert((window.pid, window.window_id));
+ if first {
+ // Best effort: a failed baseline only means a full end read.
+ let _ = self
+ .call(
+ "get_window_state",
+ json!({
+ "pid": window.pid,
+ "window_id": window.window_id,
+ "max_elements": max_elements,
+ "include_screenshot": false,
+ }),
+ )
+ .await;
+ }
+ }
+ let state = self
+ .call(
+ "get_window_state",
+ json!({
+ "pid": window.pid,
+ "window_id": window.window_id,
+ "full_output": true,
+ "include_screenshot": false,
+ }),
+ )
+ .await
+ .map_err(|error| Miss::new("read_failed", error).in_window(window))?;
+ let elements = state
+ .get("elements")
+ .and_then(Value::as_array)
+ .cloned()
+ .unwrap_or_default();
+ let markdown = state
+ .get("tree_markdown")
+ .and_then(Value::as_str)
+ .unwrap_or_default()
+ .to_owned();
+ Ok(Read {
+ window: window.clone(),
+ elements,
+ markdown,
+ })
+ }
+
+ /// Find exactly one element, waiting up to `timeout` for it to appear.
+ pub async fn find(
+ &self,
+ window: &WindowSpec,
+ element: &ElementSpec,
+ timeout: Duration,
+ ) -> Result<Found, Miss> {
+ let started = Instant::now();
+ let deadline = (started + timeout).min(self.deadline);
+ loop {
+ let attempt = self.find_once(window, element).await;
+ match attempt {
+ Ok(found) => return Ok(found),
+ Err(miss) if !miss.worth_waiting() => return Err(miss),
+ Err(miss) => {
+ if Instant::now() + POLL_INTERVAL > deadline {
+ let waited = started.elapsed().as_millis();
+ let mut miss = miss;
+ if waited > 0 {
+ miss.message = format!("{} (waited {waited} ms)", miss.message);
+ }
+ return Err(miss);
+ }
+ tokio::time::sleep(POLL_INTERVAL).await;
+ }
+ }
+ }
+ }
+
+ /// One pass over the windows `spec` names: the first window holding a
+ /// unique match wins. When none does, the miss comes from the window
+ /// with the most elements, whose nearest matches are the useful ones.
+ async fn find_once(&self, spec: &WindowSpec, element: &ElementSpec) -> Result<Found, Miss> {
+ let windows = self.search_windows(spec).await?;
+ let mut best_miss: Option<(usize, Miss)> = None;
+ for window in &windows {
+ let read = self.read(window).await?;
+ match pick(&read, element) {
+ Ok(found) => return Ok(found),
+ Err(miss) if miss.code == "ambiguous" || miss.code == "not_addressable" => {
+ return Err(miss.in_window(window))
+ }
+ Err(miss) => {
+ let size = read.elements.len();
+ if best_miss.as_ref().is_none_or(|(best, _)| size > *best) {
+ best_miss = Some((size, miss.in_window(window)));
+ }
+ }
+ }
+ }
+ let (_, mut miss) = best_miss.expect("search_windows is never empty");
+ if windows.len() > 1 {
+ miss.message = format!(
+ "{} (searched {} windows of the app)",
+ miss.message,
+ windows.len()
+ );
+ }
+ Err(miss)
+ }
+
+ /// One pass of a check over the windows it names. A presence check
+ /// holds when any window satisfies it; `gone` holds when none shows it.
+ async fn check_once(
+ &self,
+ check: &Check,
+ spec: &WindowSpec,
+ started: Instant,
+ ) -> Result<Checked, Miss> {
+ let windows = match self.search_windows(spec).await {
+ Ok(windows) => windows,
+ Err(miss)
+ if check.gone && matches!(miss.code, "window_not_found" | "app_not_found") =>
+ {
+ return Ok(Checked {
+ window: None,
+ observed: Some(format!("{} is closed", spec.describe())),
+ waited: started.elapsed(),
+ })
+ }
+ Err(miss) => return Err(miss),
+ };
+ let mut best_miss: Option<(usize, Miss)> = None;
+ for window in &windows {
+ let read = match self.read(window).await {
+ Ok(read) => read,
+ // A window that closed between the lookup and the read has
+ // no elements left.
+ Err(_) if check.gone => continue,
+ Err(miss) => return Err(miss),
+ };
+ match evaluate(check, &read) {
+ Ok(observed) if !check.gone => {
+ return Ok(Checked {
+ window: Some(window.clone()),
+ observed,
+ waited: started.elapsed(),
+ })
+ }
+ Ok(_) => {}
+ Err(miss) if check.gone => return Err(miss),
+ Err(miss) => {
+ let size = read.elements.len();
+ if best_miss.as_ref().is_none_or(|(best, _)| size > *best) {
+ best_miss = Some((size, miss));
+ }
+ }
+ }
+ }
+ match best_miss {
+ None => Ok(Checked {
+ window: windows.first().cloned(),
+ observed: None,
+ waited: started.elapsed(),
+ }),
+ Some((_, mut miss)) => {
+ if windows.len() > 1 {
+ miss.message = format!(
+ "{} (searched {} windows of the app)",
+ miss.message,
+ windows.len()
+ );
+ }
+ Err(miss)
+ }
+ }
+ }
+
+ /// Poll until the check holds or its timeout passes.
+ pub async fn check(&self, check: &Check, fallback: &WindowSpec) -> Result<Checked, Miss> {
+ let window_spec = if check.window.is_empty() {
+ fallback.clone()
+ } else {
+ check.window.clone()
+ };
+ let started = Instant::now();
+ let deadline = (started + check.timeout).min(self.deadline);
+ loop {
+ let outcome = self.check_once(check, &window_spec, started).await;
+ match outcome {
+ Ok(checked) => return Ok(checked),
+ Err(miss) if !miss.worth_waiting() => return Err(miss),
+ Err(miss) => {
+ if Instant::now() + POLL_INTERVAL > deadline {
+ let mut miss = miss;
+ miss.message = format!(
+ "{} (checked for {} ms)",
+ miss.message,
+ started.elapsed().as_millis()
+ );
+ return Err(miss);
+ }
+ tokio::time::sleep(POLL_INTERVAL).await;
+ }
+ }
+ }
+ }
+}
+
+/// Elements of a read that match `spec`. Exact name matches win over
+/// substring matches; zero-size elements only count when nothing visible
+/// matches.
+pub(crate) fn matches<'r>(read: &'r Read, spec: &ElementSpec) -> Vec<&'r Value> {
+ let by_role: Vec<&Value> = read
+ .elements
+ .iter()
+ .filter(|element| element.get("display_only").and_then(Value::as_bool) != Some(true))
+ .filter(|element| {
+ spec.role.as_deref().is_none_or(|role| {
+ element
+ .get("role")
+ .and_then(Value::as_str)
+ .is_some_and(|actual| role_matches(role, actual))
+ })
+ })
+ .collect();
+ let named: Vec<&Value> = match &spec.name {
+ None => by_role,
+ Some(name) => {
+ let needle = fold(name);
+ let exact: Vec<&Value> = by_role
+ .iter()
+ .copied()
+ .filter(|element| label_of(element).is_some_and(|label| fold(label) == needle))
+ .collect();
+ if exact.is_empty() {
+ by_role
+ .into_iter()
+ .filter(|element| {
+ label_of(element).is_some_and(|label| has_word_start(&fold(label), &needle))
+ })
+ .collect()
+ } else {
+ exact
+ }
+ }
+ };
+ let visible: Vec<&Value> = named
+ .iter()
+ .copied()
+ .filter(|element| has_area(element))
+ .collect();
+ if visible.is_empty() {
+ named
+ } else {
+ visible
+ }
+}
+
+fn pick(read: &Read, spec: &ElementSpec) -> Result<Found, Miss> {
+ let found = matches(read, spec);
+ let chosen = match (found.len(), spec.nth) {
+ (0, _) => {
+ return Err(Miss::new(
+ "not_found",
+ format!(
+ "no {} in {}{}",
+ spec.describe(),
+ read.window.describe(),
+ near_misses(read, spec)
+ ),
+ ))
+ }
+ (count, Some(nth)) if nth >= count => {
+ return Err(Miss::new(
+ "not_found",
+ format!(
+ "nth {nth} asked but only {count} {} in {}",
+ spec.describe(),
+ read.window.describe()
+ ),
+ ))
+ }
+ (_, Some(nth)) => found[nth],
+ (1, None) => found[0],
+ (count, None) => {
+ let listed: Vec<String> = found
+ .iter()
+ .take(CANDIDATES_SHOWN)
+ .enumerate()
+ .map(|(nth, element)| format!("nth {nth}: {}", describe_element(element)))
+ .collect();
+ return Err(Miss::new(
+ "ambiguous",
+ format!(
+ "{count} elements match {} in {}; add `nth` or a more exact name: {}",
+ spec.describe(),
+ read.window.describe(),
+ listed.join("; ")
+ ),
+ ));
+ }
+ };
+ if chosen
+ .get("element_token")
+ .and_then(Value::as_str)
+ .is_none()
+ {
+ return Err(Miss::new(
+ "not_addressable",
+ format!(
+ "{} in {} has no element_token to act on",
+ describe_element(chosen),
+ read.window.describe()
+ ),
+ ));
+ }
+ Ok(Found {
+ window: read.window.clone(),
+ element: chosen.clone(),
+ })
+}
+
+/// `Ok(observed)` when the check holds against this read.
+fn evaluate(check: &Check, read: &Read) -> Result<Option<String>, Miss> {
+ let element_spec = ElementSpec {
+ nth: None,
+ ..check.element.clone()
+ };
+ let structured = if element_spec.role.is_some() || element_spec.name.is_some() {
+ matches(read, &element_spec)
+ } else {
+ Vec::new()
+ };
+ // Static text and other display-only rows are only in the markdown.
+ let rows: Vec<&str> = read
+ .markdown
+ .lines()
+ .map(str::trim)
+ .filter(|line| row_matches(line, &check.element))
+ .collect();
+ let by_text_only = element_spec.role.is_none() && element_spec.name.is_none();
+ let present = if by_text_only {
+ !rows.is_empty()
+ } else {
+ !structured.is_empty() || (check.element.text.is_none() && !rows.is_empty())
+ };
+ let window = &read.window;
+ if check.gone {
+ return if present {
+ let shown = structured
+ .first()
+ .map(|element| describe_element(element))
+ .or_else(|| rows.first().map(|row| clip(row, 120)))
+ .unwrap_or_default();
+ Err(Miss::new(
+ "still_present",
+ format!(
+ "{} is still shown in {}: {shown}",
+ check.element.describe(),
+ window.describe()
+ ),
+ )
+ .in_window(window))
+ } else {
+ Ok(None)
+ };
+ }
+ if !present {
+ return Err(Miss::new(
+ "not_found",
+ format!(
+ "no {} in {}{}",
+ check.element.describe(),
+ window.describe(),
+ near_misses(read, &check.element)
+ ),
+ )
+ .in_window(window));
+ }
+ let needs_fields = check.value.is_some()
+ || check.value_contains.is_some()
+ || check.enabled.is_some()
+ || check.selected.is_some();
+ if !needs_fields {
+ let observed = structured
+ .first()
+ .map(|element| describe_element(element))
+ .or_else(|| rows.first().map(|row| clip(row, 120)));
+ return Ok(observed);
+ }
+ let pool: Vec<&Value> = match check.element.nth {
+ Some(nth) => structured.get(nth).copied().into_iter().collect(),
+ None => structured.clone(),
+ };
+ if let Some(element) = pool.iter().find(|element| fields_hold(check, element)) {
+ return Ok(Some(describe_element(element)));
+ }
+ let shown = pool
+ .iter()
+ .take(CANDIDATES_SHOWN)
+ .map(|element| describe_element(element))
+ .collect::<Vec<_>>()
+ .join("; ");
+ Err(Miss::new(
+ "unexpected_state",
+ format!(
+ "{} expected in {}, found {}",
+ check.describe(),
+ window.describe(),
+ if shown.is_empty() {
+ "no addressable match".to_owned()
+ } else {
+ shown
+ }
+ ),
+ )
+ .in_window(window))
+}
+
+fn fields_hold(check: &Check, element: &Value) -> bool {
+ let value = element
+ .get("value")
+ .and_then(Value::as_str)
+ .unwrap_or_default();
+ if let Some(expected) = &check.value {
+ if value.trim() != expected.trim() {
+ return false;
+ }
+ }
+ if let Some(expected) = &check.value_contains {
+ if !fold(value).contains(&fold(expected)) {
+ return false;
+ }
+ }
+ if let Some(expected) = check.enabled {
+ // Elements report `enabled` only when the platform knows it.
+ if element
+ .get("enabled")
+ .and_then(Value::as_bool)
+ .unwrap_or(true)
+ != expected
+ {
+ return false;
+ }
+ }
+ if let Some(expected) = check.selected {
+ if selected_state(element).unwrap_or(false) != expected {
+ return false;
+ }
+ }
+ true
+}
+
+/// An element's on/off state. For a toggle control (checkbox, radio,
+/// switch) its checked value ("1"/"0", "true"/"false", "on"/"off") wins:
+/// Chromium reports web checkboxes as value "1" with AXSelected false,
+/// because AXSelected means list selection, not checked. Otherwise the
+/// platform's `selected` flag, else that value.
+fn selected_state(element: &Value) -> Option<bool> {
+ let checked = element
+ .get("value")
+ .and_then(Value::as_str)
+ .and_then(|value| match value.trim().to_ascii_lowercase().as_str() {
+ "1" | "true" | "on" | "checked" => Some(true),
+ "0" | "false" | "off" | "unchecked" => Some(false),
+ _ => None,
+ });
+ let toggle = element
+ .get("role")
+ .and_then(Value::as_str)
+ .map(role_family)
+ .is_some_and(|family| matches!(family.as_str(), "checkbox" | "radiobutton" | "switch"));
+ if toggle && checked.is_some() {
+ return checked;
+ }
+ element.get("selected").and_then(Value::as_bool).or(checked)
+}
+
+fn row_matches(row: &str, spec: &ElementSpec) -> bool {
+ if !row.starts_with("- ") {
+ return false;
+ }
+ let folded = fold(row);
+ if let Some(text) = &spec.text {
+ if !folded.contains(&fold(text)) {
+ return false;
+ }
+ }
+ if let Some(name) = &spec.name {
+ if !has_word_start(&folded, &fold(name)) {
+ return false;
+ }
+ }
+ if let Some(role) = &spec.role {
+ let row_role = row
+ .trim_start_matches("- ")
+ .trim_start_matches(|c: char| c == '[' || c.is_ascii_digit() || c == ']')
+ .split_whitespace()
+ .next()
+ .unwrap_or("");
+ if !role_matches(role, row_role) {
+ return false;
+ }
+ }
+ spec.text.is_some() || spec.name.is_some()
+}
+
+fn near_misses(read: &Read, spec: &ElementSpec) -> String {
+ // Same role, any name: the usual slip is a near-miss label.
+ let relaxed = ElementSpec {
+ name: None,
+ text: None,
+ nth: None,
+ role: spec.role.clone(),
+ };
+ let pool: Vec<&Value> = if spec.role.is_some() {
+ matches(read, &relaxed)
+ } else if let Some(name) = &spec.name {
+ let first_word = fold(name.split_whitespace().next().unwrap_or(name));
+ read.elements
+ .iter()
+ .filter(|element| {
+ label_of(element).is_some_and(|label| fold(label).contains(&first_word))
+ })
+ .collect()
+ } else {
+ Vec::new()
+ };
+ let named: Vec<String> = pool
+ .iter()
+ .filter(|element| label_of(element).is_some())
+ .take(CANDIDATES_SHOWN)
+ .map(|element| describe_element(element))
+ .collect();
+ if named.is_empty() {
+ if read.elements.is_empty() {
+ "; the window has no readable elements".to_owned()
+ } else {
+ String::new()
+ }
+ } else {
+ format!("; nearest: {}", named.join("; "))
+ }
+}
+
+pub(crate) fn describe_element(element: &Value) -> String {
+ let role = element
+ .get("role")
+ .and_then(Value::as_str)
+ .unwrap_or("element");
+ let mut out = match element.get("element_index").and_then(Value::as_u64) {
+ Some(index) => format!("[{index}] {role}"),
+ None => role.to_owned(),
+ };
+ if let Some(label) = label_of(element) {
+ out.push_str(&format!(" \"{}\"", clip(label, 60)));
+ }
+ if let Some(value) = element.get("value").and_then(Value::as_str) {
+ if Some(value) != label_of(element) {
+ out.push_str(&format!(" value=\"{}\"", clip(value, 60)));
+ }
+ }
+ if element.get("enabled").and_then(Value::as_bool) == Some(false) {
+ out.push_str(" disabled");
+ }
+ if element.get("selected").and_then(Value::as_bool) == Some(true) {
+ out.push_str(" selected");
+ }
+ out
+}
+
+fn label_of(element: &Value) -> Option<&str> {
+ element
+ .get("label")
+ .and_then(Value::as_str)
+ .filter(|label| !label.trim().is_empty())
+}
+
+fn has_area(element: &Value) -> bool {
+ let Some(frame) = element.get("frame") else {
+ return true;
+ };
+ let dimension = |a: &str, b: &str| {
+ frame
+ .get(a)
+ .or_else(|| frame.get(b))
+ .and_then(Value::as_f64)
+ .unwrap_or(1.0)
+ };
+ dimension("w", "width") > 0.0 && dimension("h", "height") > 0.0
+}
+
+fn text_matches(value: Option<&Value>, needle: &str) -> bool {
+ value
+ .and_then(Value::as_str)
+ .is_some_and(|text| fold(text).contains(&fold(needle)))
+}
+
+/// Whether `needle` occurs in `haystack` starting at a word boundary, so
+/// "ok" matches "OK" and "Ok, continue" but not "Bookmark".
+fn has_word_start(haystack: &str, needle: &str) -> bool {
+ if needle.is_empty() {
+ return false;
+ }
+ haystack.match_indices(needle).any(|(at, _)| {
+ haystack[..at]
+ .chars()
+ .next_back()
+ .is_none_or(|before| !before.is_alphanumeric())
+ })
+}
+
+fn fold(text: &str) -> String {
+ text.trim().to_lowercase()
+}
+
+fn clip(text: &str, max: usize) -> String {
+ let flat = text.replace('\n', " ");
+ if flat.chars().count() <= max {
+ flat
+ } else {
+ let mut out: String = flat.chars().take(max).collect();
+ out.push('…');
+ out
+ }
+}
+
+fn first_text(result: &ToolResult) -> String {
+ let text = result
+ .content
+ .iter()
+ .find_map(|content| match content {
+ Content::Text { text, .. } => Some(text.trim()),
+ _ => None,
+ })
+ .unwrap_or("failed");
+ clip(text.lines().next().unwrap_or(text), 300)
+}
+
+/// Role names models use, folded to one family per control kind, across the
+/// macOS (AX), Windows (UIA) and Linux (AT-SPI) vocabularies.
+pub(crate) fn role_family(role: &str) -> String {
+ let normalized: String = role
+ .chars()
+ .filter(char::is_ascii_alphanumeric)
+ .flat_map(|c| c.to_lowercase())
+ .collect();
+ let normalized = normalized
+ .strip_prefix("ax")
+ .unwrap_or(&normalized)
+ .to_owned();
+ let family = match normalized.as_str() {
+ "button" | "pushbutton" | "togglebutton" => "button",
+ "textfield" | "textarea" | "textbox" | "edit" | "entry" | "input" | "searchfield"
+ | "securetextfield" | "passwordtext" | "text" => "textfield",
+ "statictext" | "label" | "static" => "statictext",
+ "link" | "hyperlink" => "link",
+ "checkbox" | "checkbutton" | "check" => "checkbox",
+ "radiobutton" | "radio" => "radiobutton",
+ "popupbutton" | "combobox" | "menubutton" | "dropdown" | "select" => "popupbutton",
+ "menuitem" | "menuitemcheckbox" | "menuitemradio" => "menuitem",
+ "pagetab" | "tabitem" | "tab" => "tab",
+ "image" | "img" | "icon" => "image",
+ "row" | "listitem" | "tablerow" | "outlinerow" => "row",
+ "cell" | "tablecell" | "gridcell" => "cell",
+ other => return other.to_owned(),
+ };
+ family.to_owned()
+}
+
+fn role_matches(wanted: &str, actual: &str) -> bool {
+ role_family(wanted) == role_family(actual)
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn read(elements: Vec<Value>, markdown: &str) -> Read {
+ Read {
+ window: Window {
+ pid: 1,
+ window_id: 2,
+ app: Some("Demo".into()),
+ title: Some("Doc".into()),
+ on_screen: true,
+ },
+ elements,
+ markdown: markdown.to_owned(),
+ }
+ }
+
+ fn element(index: u64, role: &str, label: &str) -> Value {
+ json!({
+ "element_index": index,
+ "element_token": format!("s00000001:{index}"),
+ "role": role,
+ "label": label,
+ "frame": {"x": 0, "y": 0, "w": 10, "h": 10},
+ })
+ }
+
+ #[test]
+ fn exact_name_beats_substring_and_roles_fold_across_platforms() {
+ let read = read(
+ vec![
+ element(1, "AXButton", "Save As…"),
+ element(2, "AXButton", "Save"),
+ element(3, "AXTextField", "Save"),
+ ],
+ "",
+ );
+ let spec = ElementSpec {
+ role: Some("button".into()),
+ name: Some("save".into()),
+ ..Default::default()
+ };
+ let found = pick(&read, &spec).unwrap();
+ assert_eq!(found.element["element_index"], 2);
+ let field = ElementSpec {
+ role: Some("textbox".into()),
+ name: Some("Save".into()),
+ ..Default::default()
+ };
+ assert_eq!(pick(&read, &field).unwrap().element["element_index"], 3);
+ assert_eq!(role_family("Push Button"), "button");
+ assert_eq!(role_family("Hyperlink"), "link");
+ }
+
+ #[test]
+ fn a_short_name_matches_whole_words_only() {
+ let read = read(
+ vec![
+ element(1, "AXButton", "Bookmark this tab"),
+ element(2, "AXButton", "Ok, continue"),
+ ],
+ "",
+ );
+ let spec = ElementSpec {
+ role: Some("button".into()),
+ name: Some("ok".into()),
+ ..Default::default()
+ };
+ assert_eq!(pick(&read, &spec).unwrap().element["element_index"], 2);
+ assert!(has_word_start("save as…", "as"));
+ assert!(!has_word_start("bookmark", "ok"));
+ }
+
+ #[test]
+ fn ambiguity_lists_candidates_and_nth_picks_one() {
+ let read = read(
+ vec![element(4, "AXButton", "OK"), element(9, "AXButton", "OK")],
+ "",
+ );
+ let spec = ElementSpec {
+ role: Some("button".into()),
+ name: Some("OK".into()),
+ ..Default::default()
+ };
+ let miss = pick(&read, &spec).unwrap_err();
+ assert_eq!(miss.code, "ambiguous");
+ assert!(
+ miss.message.contains("nth 1: [9] AXButton \"OK\""),
+ "{}",
+ miss.message
+ );
+ let second = ElementSpec {
+ nth: Some(1),
+ ..spec
+ };
+ assert_eq!(pick(&read, &second).unwrap().element["element_index"], 9);
+ }
+
+ #[test]
+ fn a_miss_names_the_nearest_elements() {
+ let read = read(vec![element(1, "AXButton", "Save Draft")], "");
+ let spec = ElementSpec {
+ role: Some("button".into()),
+ name: Some("Submit".into()),
+ ..Default::default()
+ };
+ let miss = pick(&read, &spec).unwrap_err();
+ assert_eq!(miss.code, "not_found");
+ assert!(
+ miss.message
+ .contains("nearest: [1] AXButton \"Save Draft\""),
+ "{}",
+ miss.message
+ );
+ }
+
+ #[test]
+ fn checks_read_values_static_text_and_absence() {
+ let mut field = element(5, "AXTextField", "Email");
+ field["value"] = json!("[email protected]");
+ let read = read(
+ vec![field],
+ "- [0] AXWindow \"Doc\"\n - AXStaticText \"Saved to disk\"\n - [5] AXTextField \"Email\"",
+ );
+ let check = |raw: Value| Check::parse(&raw, 0).unwrap();
+ assert!(evaluate(
+ &check(json!({"role": "textfield", "name": "email", "value": "[email protected]"})),
+ &read
+ )
+ .is_ok());
+ let wrong = evaluate(&check(json!({"name": "Email", "value": "bob"})), &read).unwrap_err();
+ assert_eq!(wrong.code, "unexpected_state");
+ assert!(
+ wrong.message.contains("value=\"[email protected]\""),
+ "{}",
+ wrong.message
+ );
+ assert!(evaluate(&check(json!({"text": "saved to disk"})), &read).is_ok());
+ assert!(evaluate(&check(json!({"text": "Error", "gone": true})), &read).is_ok());
+ let still = evaluate(&check(json!({"text": "Saved", "gone": true})), &read).unwrap_err();
+ assert_eq!(still.code, "still_present");
+ }
+
+ #[test]
+ fn selected_falls_back_to_a_checkbox_value() {
+ let mut checkbox = element(3, "AXCheckBox", "Accept terms");
+ checkbox["value"] = json!("1");
+ // As Chromium reports it: checked, but not list-selected.
+ checkbox["selected"] = json!(false);
+ let read = read(vec![checkbox], "");
+ let check = Check::parse(
+ &json!({"role": "checkbox", "name": "Accept terms", "selected": true}),
+ 0,
+ )
+ .unwrap();
+ assert!(evaluate(&check, &read).is_ok());
+ let check = Check::parse(
+ &json!({"role": "checkbox", "name": "Accept terms", "selected": false}),
+ 0,
+ )
+ .unwrap();
+ assert_eq!(
+ evaluate(&check, &read).unwrap_err().code,
+ "unexpected_state"
+ );
+ }
+
+ #[test]
+ fn check_parsing_rejects_unclear_predicates() {
+ for (raw, needle) in [
+ (json!({}), "needs `role`, `name` or `text`"),
+ (json!({"name": "x", "bogus": 1}), "unknown field `bogus`"),
+ (
+ json!({"name": "x", "gone": true, "value": "y"}),
+ "cannot be combined",
+ ),
+ (json!({"text": "x", "value": "y"}), "need `role` or `name`"),
+ (json!({"name": "x", "timeout_ms": 60000}), "timeout_ms"),
+ (json!({"name": "x", "label": "y"}), "not both"),
+ ] {
+ let error = Check::parse(&raw, 0).unwrap_err();
+ assert!(error.contains(needle), "{raw}: {error}");
+ }
+ }
+
+ #[test]
+ fn selector_keys_leave_the_action_arguments() {
+ let mut args =
+ json!({"role": "button", "name": "Go", "app": "Safari", "pid": 3, "count": 2})
+ .as_object()
+ .cloned()
+ .unwrap();
+ let (window, element) = take_selector(&mut args).unwrap();
+ assert_eq!(window.app.as_deref(), Some("Safari"));
+ assert_eq!(window.pid, Some(3));
+ assert_eq!(element.name.as_deref(), Some("Go"));
+ assert_eq!(args.keys().collect::<Vec<_>>(), ["count", "pid"]);
+ }
+}
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/tests.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/tests.rs
index 962fd4c3a6..b7dc14c8b6 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/tests.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/tests.rs
@@ -398,8 +398,8 @@ async fn the_batch_is_advertised_with_its_schema() {
.get_def(super::RUN_ACTIONS_TOOL)
.expect("registered");
assert_eq!(def.input_schema["required"], json!(["steps"]));
- let listed: Vec<_> = def.input_schema["properties"]["steps"]["items"]["properties"]["tool"]
- ["enum"]
+ let listed: Vec<_> = def.input_schema["properties"]["steps"]["items"]["anyOf"][0]["properties"]
+ ["tool"]["enum"]
.as_array()
.unwrap()
.iter()
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json
index 7b2a64f569..747298151c 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json
@@ -3467,7 +3467,7 @@
"read_only": false
},
{
- "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag and `args` are exactly that tool's arguments. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or use pixel targets after them.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.",
+ "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag and `args` are exactly that tool's arguments. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or use pixel targets after them.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.\n\nTARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.\n\nWAITS AND CHECKS: a step may add `wait_for` (checked before its action) and `expect` (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the tree, static text included; `gone:true` waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.\n\nExample (fill a form, submit, confirm the dialog, check the result):\n{\"steps\":[{\"set_value\":{\"app\":\"Safari\",\"role\":\"textfield\",\"name\":\"Email\",\"value\":\"[email protected]\"}},{\"click\":{\"role\":\"button\",\"name\":\"Submit\"},\"expect\":{\"role\":\"button\",\"name\":\"Confirm\"}},{\"click\":{\"role\":\"button\",\"name\":\"Confirm\"},\"expect\":[{\"name\":\"Confirm\",\"gone\":true},{\"text\":\"Thanks\"}]}],\"observe\":true}",
"destructive": true,
"idempotent": false,
"input_schema": {
@@ -3491,34 +3491,76 @@
"type": "string"
},
"steps": {
- "description": "Ordered actions. Execution stops at the first failing step.",
+ "description": "Ordered actions. Execution stops at the first failing step. Each step is {tool, args} or {<tool>: args}, plus optional wait_for / expect / timeout_ms; a step may also be only a wait_for or expect check.",
"items": {
- "additionalProperties": false,
- "properties": {
- "args": {
- "description": "Arguments for that tool, as in a direct call.",
+ "anyOf": [
+ {
+ "additionalProperties": false,
+ "properties": {
+ "args": {
+ "description": "Arguments for that tool, as in a direct call, optionally with role/name/nth/app/window to name the target instead of pid/window_id/element_token.",
+ "type": "object"
+ },
+ "expect": {
+ "description": "Check (or up to 4) that must hold after the action: {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}. Default timeout 2000 ms.",
+ "type": [
+ "object",
+ "array"
+ ]
+ },
+ "timeout_ms": {
+ "description": "How long a named target may take to appear. Default 3000.",
+ "maximum": 10000,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "tool": {
+ "description": "Action tool to run.",
+ "enum": [
+ "click",
+ "double_click",
+ "right_click",
+ "set_value",
+ "type_text",
+ "press_key",
+ "hotkey",
+ "scroll",
+ "drag"
+ ],
+ "type": "string"
+ },
+ "wait_for": {
+ "description": "Check that must hold before the action: {role?, name?, text?, gone?, app?, window?, timeout_ms?}. Default timeout 5000 ms.",
+ "type": "object"
+ }
+ },
+ "required": [
+ "tool"
+ ],
"type": "object"
},
- "tool": {
- "description": "Action tool to run.",
- "enum": [
- "click",
- "double_click",
- "right_click",
- "set_value",
- "type_text",
- "press_key",
- "hotkey",
- "scroll",
- "drag"
- ],
- "type": "string"
+ {
+ "additionalProperties": true,
+ "description": "Shorthand {<tool>: args} with optional wait_for / expect / timeout_ms, or a step that is only a wait_for or expect check.",
+ "properties": {
+ "expect": {
+ "type": [
+ "object",
+ "array"
+ ]
+ },
+ "timeout_ms": {
+ "maximum": 10000,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "wait_for": {
+ "type": "object"
+ }
+ },
+ "type": "object"
}
- },
- "required": [
- "tool"
- ],
- "type": "object"
+ ]
},
"maxItems": 32,
"minItems": 1,
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
index c967bfaf1e..00c24e6a87 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
@@ -3103,7 +3103,7 @@
"read_only": false
},
{
- "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag and `args` are exactly that tool's arguments. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or use pixel targets after them.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.",
+ "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag and `args` are exactly that tool's arguments. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or use pixel targets after them.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.\n\nTARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.\n\nWAITS AND CHECKS: a step may add `wait_for` (checked before its action) and `expect` (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the tree, static text included; `gone:true` waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.\n\nExample (fill a form, submit, confirm the dialog, check the result):\n{\"steps\":[{\"set_value\":{\"app\":\"Safari\",\"role\":\"textfield\",\"name\":\"Email\",\"value\":\"[email protected]\"}},{\"click\":{\"role\":\"button\",\"name\":\"Submit\"},\"expect\":{\"role\":\"button\",\"name\":\"Confirm\"}},{\"click\":{\"role\":\"button\",\"name\":\"Confirm\"},\"expect\":[{\"name\":\"Confirm\",\"gone\":true},{\"text\":\"Thanks\"}]}],\"observe\":true}",
"destructive": true,
"idempotent": false,
"input_schema": {
@@ -3127,34 +3127,76 @@
"type": "string"
},
"steps": {
- "description": "Ordered actions. Execution stops at the first failing step.",
+ "description": "Ordered actions. Execution stops at the first failing step. Each step is {tool, args} or {<tool>: args}, plus optional wait_for / expect / timeout_ms; a step may also be only a wait_for or expect check.",
"items": {
- "additionalProperties": false,
- "properties": {
- "args": {
- "description": "Arguments for that tool, as in a direct call.",
+ "anyOf": [
+ {
+ "additionalProperties": false,
+ "properties": {
+ "args": {
+ "description": "Arguments for that tool, as in a direct call, optionally with role/name/nth/app/window to name the target instead of pid/window_id/element_token.",
+ "type": "object"
+ },
+ "expect": {
+ "description": "Check (or up to 4) that must hold after the action: {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}. Default timeout 2000 ms.",
+ "type": [
+ "object",
+ "array"
+ ]
+ },
+ "timeout_ms": {
+ "description": "How long a named target may take to appear. Default 3000.",
+ "maximum": 10000,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "tool": {
+ "description": "Action tool to run.",
+ "enum": [
+ "click",
+ "double_click",
+ "right_click",
+ "set_value",
+ "type_text",
+ "press_key",
+ "hotkey",
+ "scroll",
+ "drag"
+ ],
+ "type": "string"
+ },
+ "wait_for": {
+ "description": "Check that must hold before the action: {role?, name?, text?, gone?, app?, window?, timeout_ms?}. Default timeout 5000 ms.",
+ "type": "object"
+ }
+ },
+ "required": [
+ "tool"
+ ],
"type": "object"
},
- "tool": {
- "description": "Action tool to run.",
- "enum": [
- "click",
- "double_click",
- "right_click",
- "set_value",
- "type_text",
- "press_key",
- "hotkey",
- "scroll",
- "drag"
- ],
- "type": "string"
+ {
+ "additionalProperties": true,
+ "description": "Shorthand {<tool>: args} with optional wait_for / expect / timeout_ms, or a step that is only a wait_for or expect check.",
+ "properties": {
+ "expect": {
+ "type": [
+ "object",
+ "array"
+ ]
+ },
+ "timeout_ms": {
+ "maximum": 10000,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "wait_for": {
+ "type": "object"
+ }
+ },
+ "type": "object"
}
- },
- "required": [
- "tool"
- ],
- "type": "object"
+ ]
},
"maxItems": 32,
"minItems": 1,
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json
index 8b81bd6161..65227c254d 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json
@@ -3159,7 +3159,7 @@
"read_only": false
},
{
- "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag and `args` are exactly that tool's arguments. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or use pixel targets after them.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.",
+ "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag and `args` are exactly that tool's arguments. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or use pixel targets after them.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.\n\nTARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.\n\nWAITS AND CHECKS: a step may add `wait_for` (checked before its action) and `expect` (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the tree, static text included; `gone:true` waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.\n\nExample (fill a form, submit, confirm the dialog, check the result):\n{\"steps\":[{\"set_value\":{\"app\":\"Safari\",\"role\":\"textfield\",\"name\":\"Email\",\"value\":\"[email protected]\"}},{\"click\":{\"role\":\"button\",\"name\":\"Submit\"},\"expect\":{\"role\":\"button\",\"name\":\"Confirm\"}},{\"click\":{\"role\":\"button\",\"name\":\"Confirm\"},\"expect\":[{\"name\":\"Confirm\",\"gone\":true},{\"text\":\"Thanks\"}]}],\"observe\":true}",
"destructive": true,
"idempotent": false,
"input_schema": {
@@ -3183,34 +3183,76 @@
"type": "string"
},
"steps": {
- "description": "Ordered actions. Execution stops at the first failing step.",
+ "description": "Ordered actions. Execution stops at the first failing step. Each step is {tool, args} or {<tool>: args}, plus optional wait_for / expect / timeout_ms; a step may also be only a wait_for or expect check.",
"items": {
- "additionalProperties": false,
- "properties": {
- "args": {
- "description": "Arguments for that tool, as in a direct call.",
+ "anyOf": [
+ {
+ "additionalProperties": false,
+ "properties": {
+ "args": {
+ "description": "Arguments for that tool, as in a direct call, optionally with role/name/nth/app/window to name the target instead of pid/window_id/element_token.",
+ "type": "object"
+ },
+ "expect": {
+ "description": "Check (or up to 4) that must hold after the action: {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}. Default timeout 2000 ms.",
+ "type": [
+ "object",
+ "array"
+ ]
+ },
+ "timeout_ms": {
+ "description": "How long a named target may take to appear. Default 3000.",
+ "maximum": 10000,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "tool": {
+ "description": "Action tool to run.",
+ "enum": [
+ "click",
+ "double_click",
+ "right_click",
+ "set_value",
+ "type_text",
+ "press_key",
+ "hotkey",
+ "scroll",
+ "drag"
+ ],
+ "type": "string"
+ },
+ "wait_for": {
+ "description": "Check that must hold before the action: {role?, name?, text?, gone?, app?, window?, timeout_ms?}. Default timeout 5000 ms.",
+ "type": "object"
+ }
+ },
+ "required": [
+ "tool"
+ ],
"type": "object"
},
- "tool": {
- "description": "Action tool to run.",
- "enum": [
- "click",
- "double_click",
- "right_click",
- "set_value",
- "type_text",
- "press_key",
- "hotkey",
- "scroll",
- "drag"
- ],
- "type": "string"
+ {
+ "additionalProperties": true,
+ "description": "Shorthand {<tool>: args} with optional wait_for / expect / timeout_ms, or a step that is only a wait_for or expect check.",
+ "properties": {
+ "expect": {
+ "type": [
+ "object",
+ "array"
+ ]
+ },
+ "timeout_ms": {
+ "maximum": 10000,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "wait_for": {
+ "type": "object"
+ }
+ },
+ "type": "object"
}
- },
- "required": [
- "tool"
- ],
- "type": "object"
+ ]
},
"maxItems": 32,
"minItems": 1,