Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cua-driver): run_actions steps that find, wait for and check by name (#4820)
Francesco Bonacciccaacd8fda02
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/batch.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/batch.mdxindex d41ba3faed..1223f2a4e8 100644--- a/docs/content/docs/cua-driver/reference/mcp-tools/batch.mdx+++ b/docs/content/docs/cua-driver/reference/mcp-tools/batch.mdx@@ -18,6 +18,13 @@ The default way to act: run one or more action tools in ONE call, in order, stopReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. That read is a `since:"latest"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.+TARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{"click": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.++WAITS AND CHECKS: a step may add `wait_for` (checked before its action) and `expect` (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the tree, static text included; `gone:true` waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.++Example (fill a form, submit, confirm the dialog, check the result):+{"steps":[{"set_value":{"app":"Safari","role":"textfield","name":"Email","value":"[email protected]"}},{"click":{"role":"button","name":"Submit"},"expect":{"role":"button","name":"Confirm"}},{"click":{"role":"button","name":"Confirm"},"expect":[{"name":"Confirm","gone":true},{"text":"Thanks"}]}],"observe":true}.+Effect: destructive. Platforms: macOS, Linux, Windows.| Parameter | Type | Default | Description |@@ -25,7 +32,7 @@ Effect: destructive. Platforms: macOS, Linux, Windows.| <span id="run_actions--delay_ms"></span>`delay_ms` | `integer` | | Pause between steps in milliseconds (not after the last). Default 0. Range: 0 to 2000. || <span id="run_actions--observe"></span>`observe` | `object \| boolean` | | Optional end-of-batch observation: `true`, or arguments for ONE get_window_state call. `pid` and `window_id` default to those of the last step that names both. Defaults to since="latest" (only what changed since your last read of that window with the same query/max_elements/max_depth; a full read when there is none), include_screenshot=false and max_elements=250; pass include_screenshot=true to see the window, or since=null for a full read. Omit to read nothing. || <span id="run_actions--session"></span>`session` | `string` | | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |-| <span id="run_actions--steps"></span>`steps` | `object[]` | required | Ordered actions. Execution stops at the first failing step. Items: 1 to 32. |+| <span id="run_actions--steps"></span>`steps` | `object[]` | required | Ordered actions. Execution stops at the first failing step. Each step is {tool, args} or {<tool>: args}, plus optional wait_for / expect / timeout_ms; a step may also be only a wait_for or expect check. Items: 1 to 32. |**Example arguments**diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rsindex 368b164a17..dc86c03745 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs@@ -9,7 +9,7 @@//! not. The batch itself is only a thin ordered loop plus up-front validation.use std::sync::OnceLock;-use std::time::Duration;+use std::time::{Duration, Instant};use async_trait::async_trait;use serde_json::{json, Map, Value};@@ -20,6 +20,10 @@ use crate::{tool::{Tool, ToolDef, ToolRegistry},};+pub(crate) mod locate;++use locate::{Check, ElementSpec, Locator, Window, WindowSpec};+/// Public name of the batch tool.pub const RUN_ACTIONS_TOOL: &str = "run_actions";@@ -37,17 +41,6 @@ pub const BATCHABLE_TOOLS: &[&str] = &["drag",];-/// Read-only tools models put in a batch; refused with a pointer to `observe`.-const OBSERVATION_TOOLS: &[&str] = &[- OBSERVE_TOOL,- "get_desktop_state",- "get_accessibility_tree",- "get_browser_state",- "list_windows",- "verify_state",- "zoom",-];-/// Most steps one call may carry. Keeps a batch inside client call timeouts.pub const MAX_STEPS: usize = 32;/// Longest pause between two steps.@@ -64,6 +57,23 @@ const TOOL_ALIASES: &[(&str, &str)] = &[("set", "set_value"),];const MESSAGE_LIMIT: usize = 300;+/// Read-only tools models put in a batch; refused with a pointer to `observe`.+const OBSERVATION_TOOLS: &[&str] = &[+ OBSERVE_TOOL,+ "get_desktop_state",+ "get_accessibility_tree",+ "get_browser_state",+ "list_windows",+ "verify_state",+ "zoom",+];+/// Most `expect` checks one step may carry.+pub const MAX_EXPECTS: usize = 4;+/// Wall-time budget for one batch, waits included. Keeps a batch inside+/// client call timeouts.+pub const MAX_BATCH_MS: u64 = 60_000;+/// Fields a step object may carry besides one action.+const STEP_KEYS: &[&str] = &["tool", "args", "wait_for", "expect", "timeout_ms"];const OBSERVE_TOOL: &str = "get_window_state";pub struct RunActionsTool {@@ -104,7 +114,35 @@ impl Tool for RunActionsTool {since your last read of the window, plus a new snapshot_id. Without \`observe` nothing is read. The batch uses one session: steps may \omit `session` or repeat the batch's. `delay_ms` pauses between steps (max \- 2000). At most 32 steps."+ 2000). At most 32 steps.\n\n\+ TARGET BY NAME (no earlier read needed, so one batch can cross screens): \+ a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and \+ its args may name the target instead of carrying a token: `role` (button, \+ textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), \+ `name` (the element's label; exact match wins over substring, case-insensitive), \+ `nth` (0-based, when several match), and `app` (app name or bundle id) and/or \+ `window` (title substring) instead of pid/window_id. The batch reads the window \+ fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to \+ appear, and acts on it; with only `app` it uses that app's frontmost window, so \+ a dialog that opened on top is found. Later steps inherit the window, so name \+ `app` once. The first lookup in a window also reads it once for `observe`, so \+ the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a \+ fuller name.\n\n\+ WAITS AND CHECKS: a step may add `wait_for` (checked before its action) and \+ `expect` (one check or up to 4, checked after it); a step may also be only a \+ check. A check is {role?, name?, text?, gone?, value?, value_contains?, \+ enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the \+ tree, static text included; `gone:true` waits for it to disappear. Checks poll \+ until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check \+ that does not hold fails the step with what was found instead. The report says \+ which step failed, in which phase (wait_for, find, action, expect) and why. \+ Whole batch: at most 60 s.\n\n\+ Example (fill a form, submit, confirm the dialog, check the result):\n\+ {\"steps\":[\+ {\"set_value\":{\"app\":\"Safari\",\"role\":\"textfield\",\"name\":\"Email\",\"value\":\"[email protected]\"}},\+ {\"click\":{\"role\":\"button\",\"name\":\"Submit\"},\"expect\":{\"role\":\"button\",\"name\":\"Confirm\"}},\+ {\"click\":{\"role\":\"button\",\"name\":\"Confirm\"},\"expect\":[{\"name\":\"Confirm\",\"gone\":true},{\"text\":\"Thanks\"}]}\+ ],\"observe\":true}".into(),input_schema: json!({"type": "object",@@ -114,15 +152,32 @@ impl Tool for RunActionsTool {"type": "array","minItems": 1,"maxItems": MAX_STEPS,- "description": "Ordered actions. Execution stops at the first failing step.",+ "description": "Ordered actions. Execution stops at the first failing step. Each step is {tool, args} or {<tool>: args}, plus optional wait_for / expect / timeout_ms; a step may also be only a wait_for or expect check.","items": {- "type": "object",- "required": ["tool"],- "properties": {- "tool": { "type": "string", "enum": BATCHABLE_TOOLS, "description": "Action tool to run." },- "args": { "type": "object", "description": "Arguments for that tool, as in a direct call." }- },- "additionalProperties": false+ "anyOf": [+ {+ "type": "object",+ "required": ["tool"],+ "properties": {+ "tool": { "type": "string", "enum": BATCHABLE_TOOLS, "description": "Action tool to run." },+ "args": { "type": "object", "description": "Arguments for that tool, as in a direct call, optionally with role/name/nth/app/window to name the target instead of pid/window_id/element_token." },+ "wait_for": { "type": "object", "description": "Check that must hold before the action: {role?, name?, text?, gone?, app?, window?, timeout_ms?}. Default timeout 5000 ms." },+ "expect": { "type": ["object", "array"], "description": "Check (or up to 4) that must hold after the action: {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}. Default timeout 2000 ms." },+ "timeout_ms": { "type": "integer", "minimum": 0, "maximum": 10000, "description": "How long a named target may take to appear. Default 3000." }+ },+ "additionalProperties": false+ },+ {+ "type": "object",+ "description": "Shorthand {<tool>: args} with optional wait_for / expect / timeout_ms, or a step that is only a wait_for or expect check.",+ "properties": {+ "wait_for": { "type": "object" },+ "expect": { "type": ["object", "array"] },+ "timeout_ms": { "type": "integer", "minimum": 0, "maximum": 10000 }+ },+ "additionalProperties": true+ }+ ]}},"delay_ms": {@@ -163,15 +218,40 @@ impl Tool for RunActionsTool {}}-struct Step {+/// One action of a step.+struct Action {tool: &'static str,+ /// The tool's own arguments, without the selector keys.args: Value,+ /// Window named by the step itself (pid/window_id/app/window).+ window: WindowSpec,+ /// Element named by description; empty when the step targets by token,+ /// pixels or focus.+ element: ElementSpec,+ find_timeout: Duration,+}++struct Step {+ action: Option<Action>,+ wait_for: Option<Check>,+ expect: Vec<Check>,+}++impl Step {+ fn label(&self) -> &'static str {+ match &self.action {+ Some(action) => action.tool,+ None if self.wait_for.is_some() => "wait_for",+ None => "expect",+ }+ }}struct Plan {steps: Vec<Step>,delay: Duration,observe: Option<Value>,+ session: Option<String>,}struct PlanError {@@ -213,6 +293,35 @@ impl PlanError {}}+/// How one step ended.+struct StepOutcome {+ report: Value,+ line: String,+ ok: bool,+}++/// Where in a step a failure happened.+#[derive(Clone, Copy)]+enum Phase {+ WaitFor,+ Find,+ Action,+ Expect,+ Budget,+}++impl Phase {+ fn as_str(self) -> &'static str {+ match self {+ Phase::WaitFor => "wait_for",+ Phase::Find => "find",+ Phase::Action => "action",+ Phase::Expect => "expect",+ Phase::Budget => "time_budget",+ }+ }+}+impl Plan {fn parse(registry: &ToolRegistry, args: &Value) -> Result<Self, PlanError> {let object = args@@ -253,81 +362,129 @@ impl Plan {object.get("observe"),None | Some(Value::Null) | Some(Value::Bool(false)));- if !actions.is_empty()- && !observe_given- && last.get("tool").and_then(Value::as_str) == Some(OBSERVE_TOOL)- {- trailing_observe = Some(- last.get("args")- .cloned()- .filter(Value::is_object)- .unwrap_or_else(|| Value::Object(Map::new())),- );- raw_steps = actions;+ if !actions.is_empty() && !observe_given {+ if let Some(args) = observation_step_args(last) {+ trailing_observe = Some(args);+ raw_steps = actions;+ }}}let mut steps = Vec::with_capacity(raw_steps.len());+ // Whether some earlier step names a window a later step can inherit.+ let mut has_window = false;for (index, raw) in raw_steps.iter().enumerate() {- steps.push(parse_step(registry, index, raw, session.as_deref())?);+ let step = parse_step(registry, index, raw, session.as_deref())?;+ let names_window = step+ .action+ .as_ref()+ .is_some_and(|action| !action.window.is_empty())+ || step+ .wait_for+ .iter()+ .chain(step.expect.iter())+ .any(|check| !check.window.is_empty());+ let needs_window = step+ .action+ .as_ref()+ .is_some_and(|action| !action.element.is_empty() && action.window.is_empty())+ || step+ .wait_for+ .iter()+ .chain(step.expect.iter())+ .any(|check| check.window.is_empty());+ if needs_window && !names_window && !has_window {+ return Err(PlanError::step(+ index,+ "names an element but no window: add `app` (or pid and window_id) to this step or an earlier one",+ ));+ }+ has_window |= names_window;+ steps.push(step);}let observe = match (trailing_observe.as_ref(), object.get("observe")) {- (Some(value), _) => Some(parse_observe(registry, value, &steps, session.as_deref())?),+ (Some(value), _) => Some(parse_observe(+ registry,+ value,+ has_window,+ session.as_deref(),+ )?),(None, None | Some(Value::Null) | Some(Value::Bool(false))) => None,- (None, Some(value)) => {- Some(parse_observe(registry, value, &steps, session.as_deref())?)- }+ (None, Some(value)) => Some(parse_observe(+ registry,+ value,+ has_window,+ session.as_deref(),+ )?),};Ok(Self {steps,delay: Duration::from_millis(delay_ms),observe,+ session,})}async fn run(self, registry: &ToolRegistry) -> ToolResult {let total = self.steps.len();+ let started = Instant::now();+ let mut locator = Locator::new(+ registry,+ self.session.clone(),+ started + Duration::from_millis(MAX_BATCH_MS),+ );+ // A default diff observation gets a baseline from the batch's first+ // read of each window it looks in.+ if let Some(observe) = &self.observe {+ let plain_view = observe.get("query").is_none()+ && observe.get("max_depth").is_none()+ && observe.get("since").and_then(Value::as_str)+ == Some(crate::window_state_view::SINCE_LATEST);+ if let (true, Some(max_elements)) = (+ plain_view,+ observe.get("max_elements").and_then(Value::as_u64),+ ) {+ locator = locator.with_observe_baseline(max_elements);+ }+ }let mut reports = Vec::with_capacity(total);let mut lines = Vec::with_capacity(total + 2);let mut failed_step = None;+ // The window later steps inherit, and the last concrete window used.+ let mut context = WindowSpec::default();+ let mut last_window: Option<Window> = None;for (index, step) in self.steps.into_iter().enumerate() {if index > 0 && !self.delay.is_zero() {tokio::time::sleep(self.delay).await;}- let result = registry.invoke(step.tool, step.args).await;- let ok = result.is_error != Some(true);- let message = bounded_message(&result);- lines.push(format!(- "{}. {} {}{}",- index + 1,- step.tool,- if ok { "ok" } else { "ERROR" },- if message.is_empty() {- String::new()- } else {- format!(": {message}")- }- ));- let mut report = json!({- "index": index,- "tool": step.tool,- "ok": ok,- "message": message,- });- if !ok {- if let Some(code) = result- .structured_content- .as_ref()- .and_then(|value| value.get("code"))- .and_then(Value::as_str)- {- report["code"] = Value::String(code.to_owned());- }- }- reports.push(report);- if !ok {+ let outcome = if started.elapsed() >= Duration::from_millis(MAX_BATCH_MS) {+ failure(+ index,+ step.label(),+ Phase::Budget,+ "time_budget",+ format!(+ "the batch used its {} s budget before this step",+ MAX_BATCH_MS / 1000+ ),+ json!({}),+ )+ } else {+ run_step(+ registry,+ &locator,+ index,+ step,+ &mut context,+ &mut last_window,+ )+ .await+ };+ lines.push(outcome.line);+ reports.push(outcome.report);+ if !outcome.ok {failed_step = Some(index);break;}@@ -355,16 +512,33 @@ impl Plan {// One observation at the end, also after a failure so the caller can// see where the app was left.let mut observation_content = Vec::new();- if let Some(args) = self.observe {- let result = registry.invoke(OBSERVE_TOOL, args).await;+ if let Some(mut args) = self.observe {+ let filled = fill_observed_window(&mut args, last_window.as_ref());+ let result = if filled {+ registry.invoke(OBSERVE_TOOL, args).await+ } else {+ ToolResult::error("no window to observe: no step resolved one")+ };let ok = result.is_error != Some(true);let mut report = json!({ "ok": ok, "tool": OBSERVE_TOOL });if ok {+ let summary = result+ .structured_content+ .as_ref()+ .map(observation_summary)+ .unwrap_or_default();if let Some(state) = result.structured_content.clone() {report["state"] = state;}+ if !summary.is_empty() {+ report["summary"] = Value::String(summary.clone());+ }observation_content = result.content;- lines.push("observation: get_window_state ok (below)".to_owned());+ lines.push(if summary.is_empty() {+ "observation: get_window_state ok (below)".to_owned()+ } else {+ format!("observation: {summary} (below)")+ });} else {let message = bounded_message(&result);lines.push(format!("observation: get_window_state ERROR: {message}"));@@ -384,28 +558,458 @@ impl Plan {}}+async fn run_step(+ registry: &ToolRegistry,+ locator: &Locator<'_>,+ index: usize,+ step: Step,+ context: &mut WindowSpec,+ last_window: &mut Option<Window>,+) -> StepOutcome {+ let label = step.label();+ let step_has_no_action = step.action.is_none();+ let mut report = json!({ "index": index, "tool": label });+ let mut notes: Vec<String> = Vec::new();++ if let Some(check) = &step.wait_for {+ match locator.check(check, context).await {+ Ok(checked) => {+ if !check.window.is_empty() {+ *context = check.window.clone();+ }+ if let Some(window) = &checked.window {+ *last_window = Some(window.clone());+ }+ report["wait_for"] = json!({+ "ok": true,+ "waited_ms": checked.waited.as_millis() as u64,+ });+ notes.push(format!(+ "waited {} ms for {}",+ checked.waited.as_millis(),+ check.describe()+ ));+ }+ Err(miss) => {+ if let Some(window) = &miss.window {+ *last_window = Some(window.clone());+ }+ return failure(+ index,+ label,+ Phase::WaitFor,+ miss.code,+ format!("{} did not hold: {}", check.describe(), miss.message),+ report,+ );+ }+ }+ }++ if let Some(action) = step.action {+ let Action {+ tool,+ mut args,+ window,+ element,+ find_timeout,+ } = action;+ if !window.is_empty() {+ *context = window.clone();+ }+ let window_spec = if window.is_empty() {+ context.clone()+ } else {+ window+ };+ if !element.is_empty() {+ match locator.find(&window_spec, &element, find_timeout).await {+ Ok(found) => {+ let object = args.as_object_mut().expect("object");+ object.insert("pid".into(), json!(found.window.pid));+ object.insert("window_id".into(), json!(found.window.window_id));+ object.insert(+ "element_token".into(),+ json!(found.token().unwrap_or_default()),+ );+ report["target"] = json!({+ "window": found.window.to_json(),+ "element": found.describe(),+ });+ notes.push(format!(+ "on {} in {}",+ found.describe(),+ found.window.describe()+ ));+ *last_window = Some(found.window);+ }+ Err(miss) => {+ if let Some(window) = &miss.window {+ *last_window = Some(window.clone());+ }+ return failure(index, label, Phase::Find, miss.code, miss.message, report);+ }+ }+ } else if needs_lookup(&window_spec, &args, tool, registry) {+ match locator.window(&window_spec).await {+ Ok(resolved) => {+ let object = args.as_object_mut().expect("object");+ object.insert("pid".into(), json!(resolved.pid));+ if accepts(registry, tool, "window_id") {+ object+ .entry("window_id")+ .or_insert_with(|| json!(resolved.window_id));+ }+ report["target"] = json!({ "window": resolved.to_json() });+ *last_window = Some(resolved);+ }+ Err(miss) => {+ return failure(index, label, Phase::Find, miss.code, miss.message, report)+ }+ }+ } else if let (Some(pid), Some(window_id)) = (+ args.get("pid").and_then(Value::as_i64),+ args.get("window_id").and_then(Value::as_u64),+ ) {+ *last_window = Some(Window {+ pid,+ window_id,+ app: None,+ title: None,+ on_screen: true,+ });+ }++ let result = registry.invoke(tool, args).await;+ let ok = result.is_error != Some(true);+ let message = bounded_message(&result);+ report["ok"] = json!(ok);+ report["message"] = json!(message);+ if !ok {+ if let Some(code) = result+ .structured_content+ .as_ref()+ .and_then(|value| value.get("code"))+ .and_then(Value::as_str)+ {+ report["code"] = Value::String(code.to_owned());+ }+ return StepOutcome {+ line: format!("{}. {label} ERROR (action): {message}", index + 1),+ report: with_phase(report, Phase::Action),+ ok: false,+ };+ }+ if !message.is_empty() {+ notes.insert(0, message);+ }+ }++ let fallback = last_window+ .as_ref()+ .map(|window| WindowSpec {+ pid: Some(window.pid),+ window_id: Some(window.window_id),+ ..Default::default()+ })+ .filter(|_| context.is_empty())+ .unwrap_or_else(|| context.clone());+ let mut checked = Vec::new();+ for (nth, check) in step.expect.iter().enumerate() {+ match locator.check(check, &fallback).await {+ Ok(outcome) => {+ if let Some(window) = &outcome.window {+ if last_window.is_none() || step_has_no_action {+ *last_window = Some(window.clone());+ }+ }+ if !check.window.is_empty() && step_has_no_action {+ *context = check.window.clone();+ }+ checked.push(json!({+ "ok": true,+ "check": check.describe(),+ "observed": outcome.observed,+ "waited_ms": outcome.waited.as_millis() as u64,+ }));+ }+ Err(miss) => {+ if let Some(window) = &miss.window {+ *last_window = Some(window.clone());+ }+ report["expect"] = Value::Array(checked);+ let which = if step.expect.len() > 1 {+ format!("expect {} of {}: ", nth + 1, step.expect.len())+ } else {+ String::new()+ };+ return failure(+ index,+ label,+ Phase::Expect,+ miss.code,+ format!("{which}{}", miss.message),+ report,+ );+ }+ }+ }+ if !checked.is_empty() {+ notes.push(format!("{} check(s) held", checked.len()));+ report["expect"] = Value::Array(checked);+ }+ report["ok"] = json!(true);+ if report.get("message").is_none() {+ report["message"] = json!(notes.join("; "));+ }+ let detail = notes.join("; ");+ StepOutcome {+ line: format!(+ "{}. {label} ok{}",+ index + 1,+ if detail.is_empty() {+ String::new()+ } else {+ format!(": {detail}")+ }+ ),+ report,+ ok: true,+ }+}++fn failure(+ index: usize,+ label: &str,+ phase: Phase,+ code: &str,+ message: String,+ mut report: Value,+) -> StepOutcome {+ report["ok"] = json!(false);+ report["code"] = json!(code);+ report["message"] = json!(message);+ StepOutcome {+ line: format!(+ "{}. {label} ERROR ({}): {message}",+ index + 1,+ phase.as_str()+ ),+ report: with_phase(report, phase),+ ok: false,+ }+}++fn with_phase(mut report: Value, phase: Phase) -> Value {+ report["phase"] = json!(phase.as_str());+ report+}++/// A step with a window by name, or one that relies on an inherited window+/// and does not carry its own pid, needs a window lookup.+fn needs_lookup(spec: &WindowSpec, args: &Value, tool: &str, registry: &ToolRegistry) -> bool {+ if spec.app.is_some() || spec.title.is_some() {+ return true;+ }+ let has_pid = args.get("pid").is_some();+ !has_pid && !spec.is_empty() && accepts(registry, tool, "pid")+}++fn accepts(registry: &ToolRegistry, tool: &str, field: &str) -> bool {+ registry+ .get_def(tool)+ .and_then(|def| def.input_schema.get("properties"))+ .and_then(|properties| properties.get(field))+ .is_some()+}++/// Fill the observation's window from the last window the batch used.+fn fill_observed_window(args: &mut Value, last: Option<&Window>) -> bool {+ let object = args.as_object_mut().expect("object");+ if object.contains_key("pid") && object.contains_key("window_id") {+ return true;+ }+ let Some(window) = last else {+ return false;+ };+ object.entry("pid").or_insert_with(|| json!(window.pid));+ object+ .entry("window_id")+ .or_insert_with(|| json!(window.window_id));+ true+}++/// One line saying what the end-of-batch read found.+fn observation_summary(state: &Value) -> String {+ let status = state.get("since_status").and_then(Value::as_str);+ let since = state.get("since").and_then(Value::as_str).unwrap_or("");+ match status {+ Some("diff") => {+ let counts = &state["diff_counts"];+ format!(+ "changed since {since}: {} added, {} changed, {} removed",+ counts["added"].as_u64().unwrap_or(0),+ counts["changed"].as_u64().unwrap_or(0),+ counts["removed"].as_u64().unwrap_or(0)+ )+ }+ Some("no_change") => format!("no change since {since}"),+ Some(_) => "full read (no usable earlier read of this window to diff against)".to_owned(),+ None => String::new(),+ }+}++/// The arguments of a get_window_state step, in either step form.+fn observation_step_args(step: &Value) -> Option<Value> {+ let object = step.as_object()?;+ let args = if object.get("tool").and_then(Value::as_str) == Some(OBSERVE_TOOL) {+ object.get("args")+ } else if object.len() == 1 && object.contains_key(OBSERVE_TOOL) {+ object.get(OBSERVE_TOOL)+ } else {+ return None;+ };+ Some(+ args.cloned()+ .filter(Value::is_object)+ .unwrap_or_else(|| Value::Object(Map::new())),+ )+}+fn parse_step(registry: &ToolRegistry,index: usize,raw: &Value,session: Option<&str>,) -> Result<Step, PlanError> {- let object = raw- .as_object()- .ok_or_else(|| PlanError::step(index, "must be an object {tool, args}"))?;- if let Some(extra) = object+ let object = raw.as_object().ok_or_else(|| {+ PlanError::step(index, "must be an object: {tool, args} or {<tool>: args}")+ })?;+ // The action: `{tool, args}`, or the shorthand `{<tool>: args}`.+ let shorthand: Vec<&String> = object.keys()- .find(|key| !matches!(key.as_str(), "tool" | "args"))- {- return Err(PlanError::step(+ .filter(|key| !STEP_KEYS.contains(&key.as_str()))+ .collect();+ let (name, raw_args) = match (object.get("tool"), shorthand.as_slice()) {+ (Some(_), [extra, ..]) => {+ return Err(PlanError::step(+ index,+ format!("unknown field `{extra}`; a step has `tool` and `args`, or one `{{<tool>: args}}` key, plus optional wait_for/expect/timeout_ms"),+ ))+ }+ (Some(tool), []) => (+ Some(+ tool.as_str()+ .ok_or_else(|| PlanError::step(index, "`tool` must be a string"))?,+ ),+ object.get("args"),+ ),+ (None, [key]) => {+ if object.contains_key("args") {+ return Err(PlanError::step(+ index,+ format!("`{key}` already carries the arguments; drop `args`"),+ ));+ }+ (Some(key.as_str()), object.get(key.as_str()))+ }+ (None, []) => {+ if object.contains_key("args") {+ return Err(PlanError::step(index, "`args` without `tool`"));+ }+ (None, None)+ }+ (None, keys) => {+ return Err(PlanError::step(+ index,+ format!(+ "one action per step, got {}",+ keys.iter()+ .map(|key| format!("`{key}`"))+ .collect::<Vec<_>>()+ .join(", ")+ ),+ ))+ }+ };++ let find_timeout = match object.get("timeout_ms") {+ None | Some(Value::Null) => locate::DEFAULT_FIND_TIMEOUT_MS,+ Some(value) => value+ .as_u64()+ .filter(|ms| *ms <= locate::MAX_CHECK_TIMEOUT_MS)+ .ok_or_else(|| {+ PlanError::step(+ index,+ format!(+ "`timeout_ms` must be an integer 0 to {}",+ locate::MAX_CHECK_TIMEOUT_MS+ ),+ )+ })?,+ };+ let wait_for = match object.get("wait_for") {+ None | Some(Value::Null) => None,+ Some(raw) => Some(+ Check::parse(raw, locate::DEFAULT_WAIT_TIMEOUT_MS)+ .map_err(|message| PlanError::step(index, format!("wait_for: {message}")))?,+ ),+ };+ let expect = match object.get("expect") {+ None | Some(Value::Null) => Vec::new(),+ Some(Value::Array(items)) if items.is_empty() || items.len() > MAX_EXPECTS => {+ return Err(PlanError::step(+ index,+ format!("`expect` must hold 1 to {MAX_EXPECTS} checks"),+ ))+ }+ Some(Value::Array(items)) => items+ .iter()+ .enumerate()+ .map(|(nth, raw)| {+ Check::parse(raw, locate::DEFAULT_EXPECT_TIMEOUT_MS).map_err(|message| {+ PlanError::step(index, format!("expect {}: {message}", nth + 1))+ })+ })+ .collect::<Result<_, _>>()?,+ Some(raw) => vec![Check::parse(raw, locate::DEFAULT_EXPECT_TIMEOUT_MS)+ .map_err(|message| PlanError::step(index, format!("expect: {message}")))?],+ };++ let action = match name {+ None => {+ if wait_for.is_none() && expect.is_empty() {+ return Err(PlanError::step(+ index,+ "empty step: give an action ({tool, args} or {<tool>: args}) or a wait_for/expect check",+ ));+ }+ None+ }+ Some(name) => Some(parse_action(+ registry,index,- format!("unknown field `{extra}`; a step has only `tool` and `args`"),- ));- }- let name = object- .get("tool")- .and_then(Value::as_str)- .ok_or_else(|| PlanError::step(index, "`tool` must be a string"))?;+ name,+ raw_args,+ find_timeout,+ session,+ )?),+ };+ Ok(Step {+ action,+ wait_for,+ expect,+ })+}++fn parse_action(+ registry: &ToolRegistry,+ index: usize,+ name: &str,+ raw_args: Option<&Value>,+ find_timeout: u64,+ session: Option<&str>,+) -> Result<Action, PlanError> {let name = TOOL_ALIASES.iter().find(|(alias, _)| *alias == name)@@ -419,8 +1023,8 @@ fn parse_step(format!("`{name}` reads state and a batch only acts. Put get_window_state arguments \in `observe` (one read after the last step; a get_window_state as the \- last step is taken as `observe`), or split the batch where you need to look. \- Batchable tools: {}",+ last step is taken as `observe`), or use a wait_for/expect check, or split \+ the batch where you need to look. Batchable tools: {}",BATCHABLE_TOOLS.join(", "))} else {@@ -431,23 +1035,62 @@ fn parse_step(};PlanError::step(index, message)})?;- let mut args = match object.get("args") {+ let mut args = match raw_args {None | Some(Value::Null) => Value::Object(Map::new()),Some(value @ Value::Object(_)) => value.clone(),Some(_) => return Err(PlanError::step(index, "`args` must be an object")),};prepare_args(&mut args, session).map_err(|message| PlanError::step(index, message))?;+ let (window, element) = locate::take_selector(args.as_object_mut().expect("object"))+ .map_err(|message| PlanError::step(index, format!("{tool}: {message}")))?;crate::tool::normalize_argument_aliases(tool, &mut args).map_err(|message| PlanError::step(index, format!("{tool}: {message}")))?;- validate_against_schema(registry, tool, &args)+ if !element.is_empty() {+ if tool == "drag" {+ return Err(PlanError::step(+ index,+ "drag takes coordinates; it cannot target an element by name",+ ));+ }+ if let Some(conflict) = ["element_token", "element_index", "x", "y"]+ .iter()+ .find(|key| args.get(**key).is_some())+ {+ return Err(PlanError::step(+ index,+ format!("{tool}: target by role/name OR by `{conflict}`, not both"),+ ));+ }+ }+ // Check the arguments now, with stand-ins for what the lookup fills.+ let mut probe = args.clone();+ if let Some(object) = probe.as_object_mut() {+ let needs_window = !element.is_empty() || window.app.is_some() || window.title.is_some();+ if needs_window {+ object.entry("pid").or_insert(json!(1));+ if accepts(registry, tool, "window_id") {+ object.entry("window_id").or_insert(json!(1));+ }+ }+ if !element.is_empty() {+ object.insert("element_token".into(), json!("s00000001:1"));+ }+ }+ validate_against_schema(registry, tool, &probe).map_err(|message| PlanError::step(index, format!("{tool}: {message}")))?;- Ok(Step { tool, args })+ Ok(Action {+ tool,+ args,+ window,+ element,+ find_timeout: Duration::from_millis(find_timeout),+ })}fn parse_observe(registry: &ToolRegistry,value: &Value,- steps: &[Step],+ has_window: bool,session: Option<&str>,) -> Result<Value, PlanError> {let mut args = match value {@@ -462,22 +1105,12 @@ fn parse_observe(prepare_args(&mut args, session).map_err(|message| PlanError::batch(format!("observe: {message}")))?;let object = args.as_object_mut().expect("object");- // Inherit the window from the last step that names one, so the common+ // The window comes from the last window the batch used, so the common// "act, then look at the same window" case needs no repetition.- let inherited = steps.iter().rev().find_map(|step| {- let pid = step.args.get("pid").and_then(Value::as_i64)?;- let window_id = step.args.get("window_id").and_then(Value::as_i64)?;- Some((pid, window_id))- });- if let Some((pid, window_id)) = inherited {- object.entry("pid").or_insert_with(|| json!(pid));- object- .entry("window_id")- .or_insert_with(|| json!(window_id));- }- if !object.contains_key("pid") || !object.contains_key("window_id") {+ let explicit = object.contains_key("pid") && object.contains_key("window_id");+ if !explicit && !has_window {return Err(PlanError::batch(- "observe: `pid` and `window_id` are required (no step names both to inherit from)",+ "observe: `pid` and `window_id` are required (no step names a window to inherit from)",));}object@@ -501,7 +1134,12 @@ fn parse_observe(}_ => {}}- validate_against_schema(registry, OBSERVE_TOOL, &args)+ let mut probe = args.clone();+ if let Some(object) = probe.as_object_mut() {+ object.entry("pid").or_insert(json!(1));+ object.entry("window_id").or_insert(json!(1));+ }+ validate_against_schema(registry, OBSERVE_TOOL, &probe).map_err(|message| PlanError::batch(format!("observe: {message}")))?;Ok(args)}@@ -571,3 +1209,6 @@ fn bounded_message(result: &ToolResult) -> String {#[cfg(test)]mod tests;++#[cfg(test)]+mod flow_tests;diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/flow_tests.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/flow_tests.rsnew file mode 100644index 0000000000..c3f03a1d01--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/flow_tests.rs@@ -0,0 +1,546 @@+//! `run_actions` v2 against a fake two-screen app: steps that name their+//! target, waits, checks and the end-of-batch summary.++use std::sync::{Arc, Mutex};+use std::time::Duration;++use async_trait::async_trait;+use serde_json::{json, Value};++use crate::{+ protocol::{Content, ToolResult},+ tool::{Tool, ToolDef, ToolRegistry},+};++/// What the fake app shows.+#[derive(Default)]+struct Ui {+ screen: u8,+ email: String,+ dialog_open: bool,+ /// A transient popup (no useful controls) stacked on top.+ bubble_open: bool,+ /// Reads left before the "Finish" button appears on screen 2.+ finish_delay_reads: u32,+ snapshot: u32,+ /// Element labels of the latest read, by element index.+ last_labels: Vec<(String, String)>,+ calls: Vec<(String, Value)>,+}++impl Ui {+ fn elements(&mut self, window_id: u64) -> Vec<(&'static str, String, Option<String>)> {+ if window_id == 9 {+ return vec![("AXButton", "Done".into(), None)];+ }+ if window_id == 8 {+ return vec![+ ("AXButton", "Discard".into(), None),+ ("AXButton", "Keep".into(), None),+ ];+ }+ match self.screen {+ 0 => vec![+ ("AXTextField", "Email".into(), Some(self.email.clone())),+ ("AXButton", "Next".into(), None),+ ("AXButton", "OK".into(), None),+ ("AXButton", "OK".into(), None),+ ],+ _ => {+ let mut rows = vec![("AXButton", "Back".into(), None)];+ if self.finish_delay_reads == 0 {+ rows.push(("AXButton", "Finish".into(), None));+ } else {+ self.finish_delay_reads -= 1;+ }+ rows+ }+ }+ }+}++struct Fake {+ def: ToolDef,+ ui: Arc<Mutex<Ui>>,+}++#[async_trait]+impl Tool for Fake {+ fn def(&self) -> &ToolDef {+ &self.def+ }++ async fn invoke(&self, args: Value) -> ToolResult {+ let mut ui = self.ui.lock().unwrap();+ let name = self.def.name.clone();+ ui.calls.push((name.clone(), args.clone()));+ match name.as_str() {+ "list_windows" => {+ let mut windows = vec![json!({+ "pid": 42, "window_id": 7, "app_name": "Demo", "title": "Main",+ "z_index": 1, "is_on_screen": true,+ "bounds": {"x": 0.0, "y": 0.0, "width": 800.0, "height": 600.0}+ })];+ if ui.dialog_open {+ windows.push(json!({+ "pid": 42, "window_id": 8, "app_name": "Demo", "title": "Unsaved changes",+ "z_index": 2, "is_on_screen": true,+ "bounds": {"x": 100.0, "y": 100.0, "width": 300.0, "height": 200.0}+ }));+ }+ if ui.bubble_open {+ windows.push(json!({+ "pid": 42, "window_id": 9, "app_name": "Demo", "title": "Bookmark added",+ "z_index": 3, "is_on_screen": true,+ "bounds": {"x": 500.0, "y": 0.0, "width": 200.0, "height": 100.0}+ }));+ }+ ToolResult::text("windows").with_structured(json!({ "windows": windows }))+ }+ "list_apps" => ToolResult::text("apps").with_structured(json!({+ "apps": [{"pid": 42, "name": "Demo", "bundle_id": "com.example.demo", "running": true, "active": false}]+ })),+ "get_window_state" => {+ let window_id = args["window_id"].as_u64().unwrap_or(7);+ ui.snapshot += 1;+ let snapshot = ui.snapshot;+ let rows = ui.elements(window_id);+ let mut markdown = String::from("- [0] AXWindow \"Demo\"\n");+ let mut elements = Vec::new();+ ui.last_labels.clear();+ for (offset, (role, label, value)) in rows.iter().enumerate() {+ let index = offset + 1;+ markdown.push_str(&format!(" - [{index}] {role} \"{label}\"\n"));+ let mut element = json!({+ "element_index": index,+ "element_token": format!("s{snapshot:08x}:{index}"),+ "role": role,+ "depth": 1,+ "label": label,+ "frame": {"x": 0, "y": 0, "w": 10, "h": 10},+ });+ if let Some(value) = value.as_ref().filter(|value| !value.is_empty()) {+ element["value"] = json!(value);+ }+ elements.push(element);+ ui.last_labels+ .push((format!("s{snapshot:08x}:{index}"), label.clone()));+ }+ if ui.screen == 1 && window_id == 7 {+ markdown.push_str(" - AXStaticText \"Step 2 of 2\"\n");+ }+ ToolResult::text(markdown.clone()).with_structured(json!({+ "pid": 42,+ "window_id": window_id,+ "snapshot_id": format!("s{snapshot:08x}"),+ "elements": elements,+ "tree_markdown": markdown,+ }))+ }+ "click" | "set_value" => {+ let token = args["element_token"].as_str().unwrap_or_default();+ let label = ui+ .last_labels+ .iter()+ .find(|(candidate, _)| candidate == token)+ .map(|(_, label)| label.clone());+ let Some(label) = label else {+ return ToolResult::error(format!("{name}: stale or unknown element_token {token}"));+ };+ match (name.as_str(), label.as_str()) {+ ("set_value", "Email") => {+ ui.email = args["value"].as_str().unwrap_or_default().to_owned()+ }+ ("click", "Next") => ui.screen = 1,+ ("click", "Back") => ui.dialog_open = true,+ ("click", "Keep") | ("click", "Discard") => ui.dialog_open = false,+ _ => {}+ }+ ToolResult::text(format!("{name} {label} done"))+ }+ other => ToolResult::text(format!("{other} done")),+ }+ }+}++struct Harness {+ registry: Arc<ToolRegistry>,+ ui: Arc<Mutex<Ui>>,+}++impl Harness {+ fn new() -> Self {+ let ui = Arc::new(Mutex::new(Ui::default()));+ let mut registry = ToolRegistry::new();+ for name in super::BATCHABLE_TOOLS.iter().copied().chain([+ "get_window_state",+ "list_windows",+ "list_apps",+ ]) {+ registry.register(Box::new(Fake {+ def: ToolDef {+ name: name.into(),+ description: "fake".into(),+ input_schema: json!({+ "type": "object",+ "properties": {+ "pid": {"type": "integer"},+ "window_id": {"type": "integer"},+ "element_token": {"type": "string"},+ "value": {"type": "string"},+ "key": {"type": "string"},+ "x": {"type": "number"},+ "y": {"type": "number"}+ }+ }),+ read_only: matches!(name, "get_window_state" | "list_windows" | "list_apps"),+ destructive: false,+ idempotent: false,+ open_world: false,+ },+ ui: ui.clone(),+ }));+ }+ registry.register_session_tools();+ let registry = Arc::new(registry);+ registry.init_self_weak();+ Self { registry, ui }+ }++ async fn run(&self, args: Value) -> ToolResult {+ self.registry.invoke(super::RUN_ACTIONS_TOOL, args).await+ }++ fn calls(&self, tool: &str) -> Vec<Value> {+ self.ui+ .lock()+ .unwrap()+ .calls+ .iter()+ .filter(|(name, _)| name == tool)+ .map(|(_, args)| args.clone())+ .collect()+ }+}++fn text(result: &ToolResult) -> String {+ result+ .content+ .iter()+ .filter_map(|content| match content {+ Content::Text { text, .. } => Some(text.as_str()),+ _ => None,+ })+ .collect::<Vec<_>>()+ .join("\n")+}++#[tokio::test]+async fn one_batch_crosses_screens_by_name() {+ let harness = Harness::new();+ let result = harness+ .run(json!({+ "steps": [+ {"set_value": {"app": "Demo", "role": "textfield", "name": "email", "value": "[email protected]"},+ "expect": {"role": "textfield", "name": "Email", "value": "[email protected]"}},+ {"click": {"role": "button", "name": "Next"}, "expect": {"text": "Step 2"}},+ {"tool": "click", "args": {"role": "button", "name": "Back"},+ "expect": {"app": "Demo", "window": "Unsaved", "role": "button", "name": "Keep"}},+ {"click": {"role": "button", "name": "Keep"},+ "expect": {"name": "Keep", "gone": true}}+ ],+ "observe": true+ }))+ .await;+ assert_ne!(result.is_error, Some(true), "{}", text(&result));+ let structured = result.structured_content.clone().unwrap();+ assert_eq!(structured["executed"], 4);+ // The Keep button lives in the dialog window that opened on top: the+ // app-level target found it there without a window id.+ assert_eq!(structured["steps"][3]["target"]["window"]["window_id"], 8);+ assert!(+ text(&result).contains("on [2] AXButton \"Keep\""),+ "{}",+ text(&result)+ );++ let clicks = harness.calls("click");+ assert_eq!(clicks.len(), 3);+ assert_eq!(clicks[0]["pid"], 42);+ assert_eq!(clicks[0]["window_id"], 7);+ assert!(+ clicks[0].get("role").is_none(),+ "selector keys never reach the tool"+ );+ assert!(clicks[0]["element_token"].as_str().unwrap().ends_with(":2"));+ assert_eq!(harness.ui.lock().unwrap().email, "[email protected]");++ // Lookup reads ask for full_output, which `since` never remembers; the+ // end observation is the one diff read, of the last window used.+ let reads = harness.calls("get_window_state");+ let observed = reads.last().unwrap();+ assert_eq!(observed["since"], "latest");+ // Lookup reads are full_output, which `since` never remembers; the only+ // other reads are one plain baseline per window the batch looked in.+ let baselines: Vec<u64> = reads[..reads.len() - 1]+ .iter()+ .filter(|read| read["full_output"] != true)+ .map(|read| read["window_id"].as_u64().unwrap())+ .collect();+ assert_eq!(baselines, [7, 8], "{reads:?}");+}++#[tokio::test]+async fn several_matches_fail_with_the_candidates_and_nth_picks_one() {+ let harness = Harness::new();+ let result = harness+ .run(json!({"steps": [+ {"click": {"app": "Demo", "role": "button", "name": "OK"}},+ {"click": {"role": "button", "name": "Next"}}+ ]}))+ .await;+ assert_eq!(result.is_error, Some(true));+ let structured = result.structured_content.clone().unwrap();+ assert_eq!(structured["failed_step"], 0);+ assert_eq!(structured["steps"][0]["phase"], "find");+ assert_eq!(structured["steps"][0]["code"], "ambiguous");+ let message = text(&result);+ assert!(message.contains("2 elements match"), "{message}");+ assert!(message.contains("nth 1: [4] AXButton \"OK\""), "{message}");+ assert!(harness.calls("click").is_empty(), "nothing clicked");++ let result = harness+ .run(json!({"steps": [{"click": {"app": "Demo", "role": "button", "name": "OK", "nth": 1}}]}))+ .await;+ assert_ne!(result.is_error, Some(true), "{}", text(&result));+ assert!(harness.calls("click")[0]["element_token"]+ .as_str()+ .unwrap()+ .ends_with(":4"));+}++#[tokio::test]+async fn a_missing_target_waits_then_names_the_nearest_elements() {+ let harness = Harness::new();+ let started = std::time::Instant::now();+ let result = harness+ .run(json!({"steps": [+ {"click": {"app": "com.example.demo", "role": "button", "name": "Submit"}, "timeout_ms": 600}+ ]}))+ .await;+ let elapsed = started.elapsed();+ assert_eq!(result.is_error, Some(true));+ let message = text(&result);+ assert!(+ message.contains("1. click ERROR (find): no button \"Submit\""),+ "{message}"+ );+ assert!(+ message.contains("nearest: [2] AXButton \"Next\""),+ "{message}"+ );+ assert!(message.contains("waited"), "{message}");+ assert!(elapsed >= Duration::from_millis(500), "{elapsed:?}");+ assert!(elapsed < Duration::from_millis(2_000), "{elapsed:?}");+}++#[tokio::test]+async fn a_named_target_that_appears_late_is_waited_for() {+ let harness = Harness::new();+ harness.ui.lock().unwrap().screen = 1;+ harness.ui.lock().unwrap().finish_delay_reads = 2;+ let result = harness+ .run(json!({"steps": [+ {"wait_for": {"app": "Demo", "text": "Step 2"}},+ {"click": {"role": "button", "name": "Finish"}}+ ]}))+ .await;+ assert_ne!(result.is_error, Some(true), "{}", text(&result));+ assert_eq!(harness.calls("click").len(), 1);+}++#[tokio::test]+async fn a_failed_expect_stops_the_batch_and_says_what_was_found() {+ let harness = Harness::new();+ let result = harness+ .run(json!({"steps": [+ {"set_value": {"app": "Demo", "role": "textfield", "name": "Email", "value": "bob"},+ "expect": {"name": "Email", "value": "ada", "timeout_ms": 300}},+ {"click": {"role": "button", "name": "Next"}}+ ], "observe": true}))+ .await;+ assert_eq!(result.is_error, Some(true));+ let structured = result.structured_content.clone().unwrap();+ assert_eq!(structured["failed_step"], 0);+ assert_eq!(structured["steps"][0]["phase"], "expect");+ assert_eq!(structured["steps"][0]["code"], "unexpected_state");+ let message = text(&result);+ assert!(message.contains("value=\"bob\""), "{message}");+ assert!(harness.calls("click").is_empty(), "later steps never run");+ // The failed batch is still observed, in the window it was working in.+ assert_eq!(structured["observation"]["ok"], true);+ assert_eq!(+ harness.calls("get_window_state").last().unwrap()["window_id"],+ 7+ );+}++#[tokio::test]+async fn a_check_only_step_and_gone_on_a_closed_window() {+ let harness = Harness::new();+ let result = harness+ .run(json!({"steps": [+ {"expect": {"app": "Demo", "window": "Unsaved", "name": "Keep", "gone": true, "timeout_ms": 0}}+ ]}))+ .await;+ assert_ne!(result.is_error, Some(true), "{}", text(&result));+ assert!(harness.calls("click").is_empty());+}++#[tokio::test]+async fn bad_v2_steps_are_rejected_before_anything_runs() {+ let harness = Harness::new();+ for (step, needle) in [+ (json!({"click": {"pid": 1}, "args": {}}), "drop `args`"),+ (json!({"click": {}, "press_key": {}}), "one action per step"),+ (+ json!({"drag": {"app": "Demo", "name": "x"}}),+ "drag takes coordinates",+ ),+ (+ json!({"click": {"app": "Demo", "name": "x", "element_token": "s1:1"}}),+ "not both",+ ),+ (json!({"click": {"name": "Next"}}), "no window"),+ (+ json!({"wait_for": {"app": "Demo"}}),+ "needs `role`, `name` or `text`",+ ),+ (json!({"expect": []}), "1 to 4 checks"),+ (json!({}), "empty step"),+ (+ json!({"click": {"app": "Demo", "name": "x"}, "timeout_ms": 99999}),+ "timeout_ms",+ ),+ (json!({"wait": {"app": "Demo"}}), "cannot run in a batch"),+ ] {+ let result = harness+ .run(json!({"steps": [{"press_key": {"key": "a"}}, step.clone()]}))+ .await;+ assert_eq!(result.is_error, Some(true), "{step}");+ let message = text(&result);+ assert!(message.contains("step 2"), "{step}: {message}");+ assert!(message.contains(needle), "{step}: {message}");+ }+ assert!(harness.calls("press_key").is_empty());+ assert!(harness.calls("list_windows").is_empty());+}++#[tokio::test]+async fn later_steps_inherit_the_window() {+ let harness = Harness::new();+ let result = harness+ .run(json!({"steps": [+ {"click": {"app": "Demo", "role": "textfield", "name": "Email"}},+ {"press_key": {"key": "tab"}}+ ]}))+ .await;+ assert_ne!(result.is_error, Some(true), "{}", text(&result));+ let pressed = &harness.calls("press_key")[0];+ assert_eq!(pressed["pid"], 42);+ assert_eq!(pressed["window_id"], 7);+}++#[tokio::test]+async fn a_popup_on_top_does_not_hide_the_main_window() {+ let harness = Harness::new();+ harness.ui.lock().unwrap().bubble_open = true;+ let result = harness+ .run(json!({"steps": [+ {"click": {"app": "Demo", "role": "button", "name": "Next"}, "expect": {"text": "Step 2"}}+ ], "observe": true}))+ .await;+ assert_ne!(result.is_error, Some(true), "{}", text(&result));+ let structured = result.structured_content.clone().unwrap();+ assert_eq!(structured["steps"][0]["target"]["window"]["window_id"], 7);+ // The observation follows the window the batch acted in, not the popup.+ assert_eq!(+ harness.calls("get_window_state").last().unwrap()["window_id"],+ 7+ );++ // A miss names the window with the most elements, and says how many+ // windows were searched.+ let result = harness+ .run(json!({"steps": [+ {"click": {"app": "Demo", "role": "button", "name": "Publish"}, "timeout_ms": 0}+ ]}))+ .await;+ let message = text(&result);+ assert!(message.contains("searched 2 windows"), "{message}");+ assert!(message.contains("window 7"), "{message}");+}++#[tokio::test]+async fn a_check_only_step_names_the_window_to_observe() {+ let harness = Harness::new();+ let result = harness+ .run(json!({"steps": [{"expect": {"app": "Demo", "text": "Email", "timeout_ms": 0}}], "observe": true}))+ .await;+ assert_ne!(result.is_error, Some(true), "{}", text(&result));+ assert_eq!(+ result.structured_content.unwrap()["observation"]["ok"],+ true+ );+ assert_eq!(+ harness.calls("get_window_state").last().unwrap()["window_id"],+ 7+ );+}++#[tokio::test]+async fn the_end_observation_diffs_against_the_batch_start() {+ let harness = Harness::new();+ let result = harness+ .run(json!({"steps": [+ {"set_value": {"app": "Demo", "role": "textfield", "name": "Email", "value": "[email protected]"}}+ ], "observe": true}))+ .await;+ assert_ne!(result.is_error, Some(true), "{}", text(&result));+ let reads = harness.calls("get_window_state");+ // First a plain read with the observation's budget (remembered by+ // `since`), then the full lookup read, then the end diff.+ assert_eq!(reads[0]["max_elements"], 250);+ assert!(reads[0].get("full_output").is_none());+ assert_eq!(reads[1]["full_output"], true);+ let last = reads.last().unwrap();+ assert_eq!(last["since"], "latest");+ assert_eq!(last["max_elements"], 250);+ assert_eq!(reads.len(), 3, "one baseline per window, not per poll");++ // Without observe there is no baseline read.+ let before = harness.calls("get_window_state").len();+ harness+ .run(json!({"steps": [{"click": {"app": "Demo", "role": "button", "name": "Next"}}]}))+ .await;+ assert_eq!(harness.calls("get_window_state").len(), before + 1);+}++#[tokio::test]+async fn a_trailing_get_window_state_in_shorthand_is_the_observation() {+ let harness = Harness::new();+ let result = harness+ .run(json!({"steps": [+ {"click": {"app": "Demo", "role": "button", "name": "Next"}},+ {"get_window_state": {}}+ ]}))+ .await;+ assert_ne!(result.is_error, Some(true), "{}", text(&result));+ let structured = result.structured_content.unwrap();+ assert_eq!(structured["total"], 1, "the read is not a step");+ assert_eq!(structured["observation"]["ok"], true);+ assert_eq!(+ harness.calls("get_window_state").last().unwrap()["window_id"],+ 7+ );+}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/locate.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/locate.rsnew file mode 100644index 0000000000..1c2287fbb6--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/locate.rs@@ -0,0 +1,1561 @@+//! Find windows and elements by description, wait for them, and check them.+//!+//! Used by `run_actions` steps that name their target (`role`, `name`, `app`,+//! `window`) instead of carrying a token from an earlier read, and by the+//! per-step `wait_for` / `expect` checks. Every read goes through the same+//! [`ToolRegistry`] entry point a direct call uses (`list_windows`,+//! `list_apps`, `get_window_state`), so a lookup is held to the same session,+//! policy and approval checks as the reads a model would make itself.+//!+//! Lookup reads ask for `full_output`, which the `since` snapshot memory does+//! not remember, so they never become the baseline of the caller's next+//! `since:"latest"` diff.++use std::time::{Duration, Instant};++use serde_json::{json, Map, Value};++use crate::{+ protocol::{Content, ToolResult},+ tool::ToolRegistry,+};++/// Keys a step may add to an action's arguments to name its target. They are+/// removed before the action tool runs.+pub(crate) const SELECTOR_KEYS: &[&str] = &["role", "name", "label", "nth", "app", "window"];+/// Keys a `wait_for` / `expect` check accepts.+pub(crate) const CHECK_KEYS: &[&str] = &[+ "role",+ "name",+ "label",+ "text",+ "nth",+ "app",+ "window",+ "pid",+ "window_id",+ "gone",+ "value",+ "value_contains",+ "enabled",+ "selected",+ "timeout_ms",+];++/// Default wait for a named target to appear before its action runs.+pub(crate) const DEFAULT_FIND_TIMEOUT_MS: u64 = 3_000;+/// Default wait for a `wait_for` check.+pub(crate) const DEFAULT_WAIT_TIMEOUT_MS: u64 = 5_000;+/// Default wait for an `expect` check.+pub(crate) const DEFAULT_EXPECT_TIMEOUT_MS: u64 = 2_000;+/// Longest wait any single check may ask for.+pub(crate) const MAX_CHECK_TIMEOUT_MS: u64 = 10_000;+const POLL_INTERVAL: Duration = Duration::from_millis(250);+const CANDIDATES_SHOWN: usize = 5;+/// Most windows of one app a lookup reads per poll.+const MAX_WINDOWS_SEARCHED: usize = 4;++/// Which window a step or check addresses.+#[derive(Clone, Debug, Default, PartialEq, Eq)]+pub(crate) struct WindowSpec {+ pub pid: Option<i64>,+ pub window_id: Option<u64>,+ /// App name (case-insensitive, exact or substring) or bundle id.+ pub app: Option<String>,+ /// Case-insensitive substring of the window title.+ pub title: Option<String>,+}++impl WindowSpec {+ pub fn is_empty(&self) -> bool {+ self.pid.is_none() && self.window_id.is_none() && self.app.is_none() && self.title.is_none()+ }++ /// A spec that names one exact window needs no lookup.+ fn exact(&self) -> Option<(i64, u64)> {+ match (self.pid, self.window_id, &self.app, &self.title) {+ (Some(pid), Some(window_id), None, None) => Some((pid, window_id)),+ _ => None,+ }+ }++ pub fn describe(&self) -> String {+ let mut parts = Vec::new();+ if let Some(app) = &self.app {+ parts.push(format!("app \"{app}\""));+ }+ if let Some(title) = &self.title {+ parts.push(format!("window \"{title}\""));+ }+ if let Some(pid) = self.pid {+ parts.push(format!("pid {pid}"));+ }+ if let Some(window_id) = self.window_id {+ parts.push(format!("window_id {window_id}"));+ }+ if parts.is_empty() {+ "no window".to_owned()+ } else {+ parts.join(", ")+ }+ }+}++/// Which element inside the window.+#[derive(Clone, Debug, Default, PartialEq, Eq)]+pub(crate) struct ElementSpec {+ pub role: Option<String>,+ pub name: Option<String>,+ /// Any row of the tree containing this text, display-only rows included.+ pub text: Option<String>,+ pub nth: Option<usize>,+}++impl ElementSpec {+ pub fn is_empty(&self) -> bool {+ self.role.is_none() && self.name.is_none() && self.text.is_none()+ }++ pub fn describe(&self) -> String {+ let mut out = String::new();+ if let Some(role) = &self.role {+ out.push_str(role);+ } else {+ out.push_str("element");+ }+ if let Some(name) = &self.name {+ out.push_str(&format!(" \"{name}\""));+ }+ if let Some(text) = &self.text {+ out.push_str(&format!(" with text \"{text}\""));+ }+ if let Some(nth) = self.nth {+ out.push_str(&format!(" (nth {nth})"));+ }+ out+ }+}++/// A `wait_for` or `expect` predicate.+#[derive(Clone, Debug, PartialEq, Eq)]+pub(crate) struct Check {+ pub window: WindowSpec,+ pub element: ElementSpec,+ pub gone: bool,+ pub value: Option<String>,+ pub value_contains: Option<String>,+ pub enabled: Option<bool>,+ pub selected: Option<bool>,+ pub timeout: Duration,+}++impl Check {+ pub fn parse(raw: &Value, default_timeout_ms: u64) -> Result<Self, String> {+ let object = raw+ .as_object()+ .ok_or("must be an object such as {\"role\":\"button\",\"name\":\"Save\"}")?;+ if let Some(key) = object+ .keys()+ .find(|key| !CHECK_KEYS.contains(&key.as_str()))+ {+ return Err(format!(+ "unknown field `{key}`; a check takes {}",+ CHECK_KEYS.join(", ")+ ));+ }+ let element = element_spec(object)?;+ if element.is_empty() {+ return Err("needs `role`, `name` or `text` to say which element".to_owned());+ }+ let window = window_spec(object)?;+ let gone = opt_bool(object, "gone")?.unwrap_or(false);+ let value = opt_string(object, "value")?;+ let value_contains = opt_string(object, "value_contains")?;+ let enabled = opt_bool(object, "enabled")?;+ let selected = opt_bool(object, "selected")?;+ if gone+ && (value.is_some()+ || value_contains.is_some()+ || enabled.is_some()+ || selected.is_some())+ {+ return Err("`gone:true` cannot be combined with value/enabled/selected".to_owned());+ }+ if element.text.is_some()+ && element.role.is_none()+ && element.name.is_none()+ && (value.is_some()+ || value_contains.is_some()+ || enabled.is_some()+ || selected.is_some())+ {+ return Err(+ "value/enabled/selected need `role` or `name`; `text` alone only checks that the text is shown"+ .to_owned(),+ );+ }+ let timeout_ms = match object.get("timeout_ms") {+ None | Some(Value::Null) => default_timeout_ms,+ Some(value) => value+ .as_u64()+ .filter(|ms| *ms <= MAX_CHECK_TIMEOUT_MS)+ .ok_or_else(|| {+ format!("`timeout_ms` must be an integer 0 to {MAX_CHECK_TIMEOUT_MS}")+ })?,+ };+ Ok(Self {+ window,+ element,+ gone,+ value,+ value_contains,+ enabled,+ selected,+ timeout: Duration::from_millis(timeout_ms),+ })+ }++ pub fn describe(&self) -> String {+ let mut out = self.element.describe();+ if self.gone {+ out.push_str(" is gone");+ } else {+ out.push_str(" is shown");+ }+ if let Some(value) = &self.value {+ out.push_str(&format!(", value \"{value}\""));+ }+ if let Some(value) = &self.value_contains {+ out.push_str(&format!(", value contains \"{value}\""));+ }+ if let Some(enabled) = self.enabled {+ out.push_str(if enabled { ", enabled" } else { ", disabled" });+ }+ if let Some(selected) = self.selected {+ out.push_str(if selected {+ ", selected"+ } else {+ ", not selected"+ });+ }+ out+ }+}++/// Split the selector keys out of an action's arguments.+pub(crate) fn take_selector(+ args: &mut Map<String, Value>,+) -> Result<(WindowSpec, ElementSpec), String> {+ let mut picked = Map::new();+ for key in SELECTOR_KEYS {+ if let Some(value) = args.remove(*key) {+ picked.insert((*key).to_owned(), value);+ }+ }+ let element = element_spec(&picked)?;+ if element.text.is_some() {+ return Err(+ "`text` only works in wait_for/expect; target an element by role and name".to_owned(),+ );+ }+ let mut window = window_spec(&picked)?;+ // A malformed pid/window_id is left to the tool's schema check, which+ // names the field the same way a direct call would.+ window.pid = args.get("pid").and_then(Value::as_i64);+ window.window_id = args.get("window_id").and_then(Value::as_u64);+ Ok((window, element))+}++fn element_spec(object: &Map<String, Value>) -> Result<ElementSpec, String> {+ let name = match (opt_string(object, "name")?, opt_string(object, "label")?) {+ (Some(_), Some(_)) => return Err("give `name` or `label`, not both".to_owned()),+ (name, label) => name.or(label),+ };+ let nth = match object.get("nth") {+ None | Some(Value::Null) => None,+ Some(value) => Some(+ value+ .as_u64()+ .ok_or("`nth` must be a non-negative integer (0 is the first match)")?+ as usize,+ ),+ };+ Ok(ElementSpec {+ role: opt_string(object, "role")?,+ name,+ text: opt_string(object, "text")?,+ nth,+ })+}++fn window_spec(object: &Map<String, Value>) -> Result<WindowSpec, String> {+ Ok(WindowSpec {+ pid: opt_i64(object, "pid")?,+ window_id: opt_u64(object, "window_id")?,+ app: opt_string(object, "app")?,+ title: opt_string(object, "window")?,+ })+}++fn opt_string(object: &Map<String, Value>, key: &str) -> Result<Option<String>, String> {+ match object.get(key) {+ None | Some(Value::Null) => Ok(None),+ Some(Value::String(text)) if !text.trim().is_empty() => Ok(Some(text.trim().to_owned())),+ Some(_) => Err(format!("`{key}` must be a non-empty string")),+ }+}++fn opt_bool(object: &Map<String, Value>, key: &str) -> Result<Option<bool>, String> {+ match object.get(key) {+ None | Some(Value::Null) => Ok(None),+ Some(Value::Bool(flag)) => Ok(Some(*flag)),+ Some(_) => Err(format!("`{key}` must be true or false")),+ }+}++fn opt_i64(object: &Map<String, Value>, key: &str) -> Result<Option<i64>, String> {+ match object.get(key) {+ None | Some(Value::Null) => Ok(None),+ Some(value) => value+ .as_i64()+ .map(Some)+ .ok_or_else(|| format!("`{key}` must be an integer")),+ }+}++fn opt_u64(object: &Map<String, Value>, key: &str) -> Result<Option<u64>, String> {+ match object.get(key) {+ None | Some(Value::Null) => Ok(None),+ Some(value) => value+ .as_u64()+ .map(Some)+ .ok_or_else(|| format!("`{key}` must be a non-negative integer")),+ }+}++/// One concrete window.+#[derive(Clone, Debug, PartialEq, Eq)]+pub(crate) struct Window {+ pub pid: i64,+ pub window_id: u64,+ pub app: Option<String>,+ pub title: Option<String>,+ pub on_screen: bool,+}++impl Window {+ pub fn describe(&self) -> String {+ let mut out = String::new();+ if let Some(app) = &self.app {+ out.push_str(app);+ out.push(' ');+ }+ out.push_str(&format!("window {}", self.window_id));+ if let Some(title) = self.title.as_deref().filter(|title| !title.is_empty()) {+ out.push_str(&format!(" \"{}\"", clip(title, 60)));+ }+ out+ }++ pub fn to_json(&self) -> Value {+ json!({+ "pid": self.pid,+ "window_id": self.window_id,+ "app": self.app,+ "title": self.title,+ })+ }+}++/// One accessibility read of a window.+pub(crate) struct Read {+ pub window: Window,+ pub elements: Vec<Value>,+ pub markdown: String,+}++/// An element found by description.+#[derive(Clone, Debug)]+pub(crate) struct Found {+ pub window: Window,+ pub element: Value,+}++impl Found {+ pub fn token(&self) -> Option<&str> {+ self.element.get("element_token").and_then(Value::as_str)+ }++ pub fn describe(&self) -> String {+ describe_element(&self.element)+ }+}++/// Why a lookup or check did not succeed.+#[derive(Clone, Debug)]+pub(crate) struct Miss {+ /// Stable machine-readable reason.+ pub code: &'static str,+ pub message: String,+ pub window: Option<Window>,+}++impl Miss {+ fn new(code: &'static str, message: impl Into<String>) -> Self {+ Self {+ code,+ message: message.into(),+ window: None,+ }+ }++ /// Whether polling again can change the answer. A refused or failed+ /// read, or an ambiguous match, will not fix itself, and retrying a+ /// refused read would only repeat the refusal.+ fn worth_waiting(&self) -> bool {+ matches!(+ self.code,+ "not_found"+ | "app_not_found"+ | "window_not_found"+ | "still_present"+ | "unexpected_state"+ )+ }++ fn in_window(mut self, window: &Window) -> Self {+ self.window = Some(window.clone());+ self+ }+}++/// Outcome of a satisfied check.+#[derive(Clone, Debug)]+pub(crate) struct Checked {+ pub window: Option<Window>,+ pub observed: Option<String>,+ pub waited: Duration,+}++/// Window and element lookup through the registry.+pub(crate) struct Locator<'a> {+ registry: &'a ToolRegistry,+ session: Option<String>,+ /// Hard stop for every poll loop (the batch's own time budget).+ deadline: Instant,+ /// When the batch ends with a default `observe`, the `max_elements` of+ /// that read: the first lookup of each window also makes one plain+ /// read with that budget, which `since:"latest"` remembers, so the end+ /// observation is a diff of what the batch changed rather than a full+ /// read of a window the caller never read.+ baseline_max_elements: Option<u64>,+ baselined: std::sync::Mutex<std::collections::HashSet<(i64, u64)>>,+}++impl<'a> Locator<'a> {+ pub fn new(registry: &'a ToolRegistry, session: Option<String>, deadline: Instant) -> Self {+ Self {+ registry,+ session,+ deadline,+ baseline_max_elements: None,+ baselined: Default::default(),+ }+ }++ pub fn with_observe_baseline(mut self, max_elements: u64) -> Self {+ self.baseline_max_elements = Some(max_elements);+ self+ }++ fn with_session(&self, mut args: Value) -> Value {+ if let (Some(session), Some(object)) = (&self.session, args.as_object_mut()) {+ object.insert("session".to_owned(), Value::String(session.clone()));+ }+ args+ }++ async fn call(&self, tool: &str, args: Value) -> Result<Value, String> {+ let result = self.registry.invoke(tool, self.with_session(args)).await;+ if result.is_error == Some(true) {+ return Err(format!("{tool}: {}", first_text(&result)));+ }+ Ok(result.structured_content.unwrap_or(Value::Null))+ }++ /// Resolve a spec to one window. With only `app` or `pid`, the app's+ /// frontmost window wins, so a dialog that opens on top is found.+ pub async fn window(&self, spec: &WindowSpec) -> Result<Window, Miss> {+ if let Some((pid, window_id)) = spec.exact() {+ return Ok(Window {+ pid,+ window_id,+ app: None,+ title: None,+ on_screen: true,+ });+ }+ self.listed_window(spec).await+ }++ /// Like [`Self::window`], but always confirms the window through+ /// `list_windows`, so the result carries its app name and title.+ pub async fn listed_window(&self, spec: &WindowSpec) -> Result<Window, Miss> {+ let mut windows = self.listed_windows(spec).await?;+ Ok(windows.remove(0))+ }++ /// Every window matching `spec`, best first: on-screen before+ /// off-screen, then topmost (largest z_index) first. Never empty.+ pub async fn listed_windows(&self, spec: &WindowSpec) -> Result<Vec<Window>, Miss> {+ if spec.is_empty() {+ return Err(Miss::new(+ "no_window",+ "no window to look in: give `app` (or pid and window_id) on this step or an earlier one",+ ));+ }+ let mut list_args = json!({"on_screen_only": false});+ if let Some(pid) = spec.pid {+ list_args["pid"] = json!(pid);+ }+ let listed = self+ .call("list_windows", list_args)+ .await+ .map_err(|error| Miss::new("lookup_failed", error))?;+ let windows: Vec<Value> = listed+ .get("windows")+ .and_then(Value::as_array)+ .cloned()+ .unwrap_or_default();+ let mut candidates: Vec<&Value> = windows+ .iter()+ .filter(|window| window.get("pid").and_then(Value::as_i64).is_some())+ .collect();+ if let Some(app) = &spec.app {+ let by_name: Vec<&Value> = candidates+ .iter()+ .copied()+ .filter(|window| text_matches(window.get("app_name"), app))+ .collect();+ candidates = if by_name.is_empty() {+ let pids = self.app_pids(app).await;+ candidates+ .into_iter()+ .filter(|window| {+ window+ .get("pid")+ .and_then(Value::as_i64)+ .is_some_and(|pid| pids.contains(&pid))+ })+ .collect()+ } else {+ by_name+ };+ if candidates.is_empty() {+ let mut apps: Vec<&str> = windows+ .iter()+ .filter_map(|window| window.get("app_name").and_then(Value::as_str))+ .collect();+ apps.sort_unstable();+ apps.dedup();+ return Err(Miss::new(+ "app_not_found",+ format!(+ "no window of app \"{app}\" (apps with windows: {})",+ if apps.is_empty() {+ "none".to_owned()+ } else {+ apps.join(", ")+ }+ ),+ ));+ }+ }+ if let Some(window_id) = spec.window_id {+ candidates.retain(|window| {+ window.get("window_id").and_then(Value::as_u64) == Some(window_id)+ });+ }+ if let Some(title) = &spec.title {+ let before: Vec<String> = candidates+ .iter()+ .filter_map(|window| window.get("title").and_then(Value::as_str))+ .map(|title| format!("\"{}\"", clip(title, 40)))+ .collect();+ candidates.retain(|window| text_matches(window.get("title"), title));+ if candidates.is_empty() {+ return Err(Miss::new(+ "window_not_found",+ format!(+ "no window titled like \"{title}\" for {} (titles: {})",+ spec.describe(),+ if before.is_empty() {+ "none".to_owned()+ } else {+ before.join(", ")+ }+ ),+ ));+ }+ }+ if candidates.is_empty() {+ return Err(Miss::new(+ "window_not_found",+ format!("no window for {}", spec.describe()),+ ));+ }+ // On-screen windows first, then the topmost (largest z_index).+ let mut ranked: Vec<(usize, &Value)> = candidates.into_iter().enumerate().collect();+ ranked.sort_by_key(|(order, window)| {+ let on_screen = window.get("is_on_screen").and_then(Value::as_bool) != Some(false);+ let z = window+ .get("z_index")+ .and_then(Value::as_i64)+ .unwrap_or(i64::MIN);+ std::cmp::Reverse((on_screen, z, std::cmp::Reverse(*order)))+ });+ Ok(ranked+ .into_iter()+ .map(|(_, window)| Window {+ pid: window+ .get("pid")+ .and_then(Value::as_i64)+ .unwrap_or_default(),+ window_id: window+ .get("window_id")+ .and_then(Value::as_u64)+ .unwrap_or_default(),+ app: window+ .get("app_name")+ .and_then(Value::as_str)+ .map(str::to_owned),+ title: window+ .get("title")+ .and_then(Value::as_str)+ .map(str::to_owned),+ on_screen: window.get("is_on_screen").and_then(Value::as_bool) != Some(false),+ })+ .collect())+ }++ /// The windows a lookup searches, best first. An exact pid/window_id is+ /// the only one; otherwise the app's on-screen windows in stacking+ /// order, so an element in a dialog on top is found first and one in+ /// the main window is still found when a transient popup is on top.+ pub async fn search_windows(&self, spec: &WindowSpec) -> Result<Vec<Window>, Miss> {+ if spec.exact().is_some() {+ return Ok(vec![self.window(spec).await?]);+ }+ let windows = self.listed_windows(spec).await?;+ let on_screen: Vec<Window> = windows.iter().filter(|w| w.on_screen).cloned().collect();+ let mut pool = if on_screen.is_empty() {+ windows+ } else {+ on_screen+ };+ pool.truncate(MAX_WINDOWS_SEARCHED);+ Ok(pool)+ }++ /// Pids of running apps whose name or bundle id matches.+ async fn app_pids(&self, app: &str) -> Vec<i64> {+ let Ok(listed) = self.call("list_apps", json!({})).await else {+ return Vec::new();+ };+ listed+ .get("apps")+ .and_then(Value::as_array)+ .map(|apps| {+ apps.iter()+ .filter(|entry| {+ entry+ .get("bundle_id")+ .and_then(Value::as_str)+ .is_some_and(|bundle| bundle.eq_ignore_ascii_case(app))+ || text_matches(entry.get("name"), app)+ })+ .filter_map(|entry| entry.get("pid").and_then(Value::as_i64))+ .filter(|pid| *pid > 0)+ .collect()+ })+ .unwrap_or_default()+ }++ /// One fresh accessibility read. It refreshes the window's element+ /// tokens, so the tokens it returns are the ones to act with.+ pub async fn read(&self, window: &Window) -> Result<Read, Miss> {+ if let Some(max_elements) = self.baseline_max_elements {+ let first = self+ .baselined+ .lock()+ .unwrap_or_else(|e| e.into_inner())+ .insert((window.pid, window.window_id));+ if first {+ // Best effort: a failed baseline only means a full end read.+ let _ = self+ .call(+ "get_window_state",+ json!({+ "pid": window.pid,+ "window_id": window.window_id,+ "max_elements": max_elements,+ "include_screenshot": false,+ }),+ )+ .await;+ }+ }+ let state = self+ .call(+ "get_window_state",+ json!({+ "pid": window.pid,+ "window_id": window.window_id,+ "full_output": true,+ "include_screenshot": false,+ }),+ )+ .await+ .map_err(|error| Miss::new("read_failed", error).in_window(window))?;+ let elements = state+ .get("elements")+ .and_then(Value::as_array)+ .cloned()+ .unwrap_or_default();+ let markdown = state+ .get("tree_markdown")+ .and_then(Value::as_str)+ .unwrap_or_default()+ .to_owned();+ Ok(Read {+ window: window.clone(),+ elements,+ markdown,+ })+ }++ /// Find exactly one element, waiting up to `timeout` for it to appear.+ pub async fn find(+ &self,+ window: &WindowSpec,+ element: &ElementSpec,+ timeout: Duration,+ ) -> Result<Found, Miss> {+ let started = Instant::now();+ let deadline = (started + timeout).min(self.deadline);+ loop {+ let attempt = self.find_once(window, element).await;+ match attempt {+ Ok(found) => return Ok(found),+ Err(miss) if !miss.worth_waiting() => return Err(miss),+ Err(miss) => {+ if Instant::now() + POLL_INTERVAL > deadline {+ let waited = started.elapsed().as_millis();+ let mut miss = miss;+ if waited > 0 {+ miss.message = format!("{} (waited {waited} ms)", miss.message);+ }+ return Err(miss);+ }+ tokio::time::sleep(POLL_INTERVAL).await;+ }+ }+ }+ }++ /// One pass over the windows `spec` names: the first window holding a+ /// unique match wins. When none does, the miss comes from the window+ /// with the most elements, whose nearest matches are the useful ones.+ async fn find_once(&self, spec: &WindowSpec, element: &ElementSpec) -> Result<Found, Miss> {+ let windows = self.search_windows(spec).await?;+ let mut best_miss: Option<(usize, Miss)> = None;+ for window in &windows {+ let read = self.read(window).await?;+ match pick(&read, element) {+ Ok(found) => return Ok(found),+ Err(miss) if miss.code == "ambiguous" || miss.code == "not_addressable" => {+ return Err(miss.in_window(window))+ }+ Err(miss) => {+ let size = read.elements.len();+ if best_miss.as_ref().is_none_or(|(best, _)| size > *best) {+ best_miss = Some((size, miss.in_window(window)));+ }+ }+ }+ }+ let (_, mut miss) = best_miss.expect("search_windows is never empty");+ if windows.len() > 1 {+ miss.message = format!(+ "{} (searched {} windows of the app)",+ miss.message,+ windows.len()+ );+ }+ Err(miss)+ }++ /// One pass of a check over the windows it names. A presence check+ /// holds when any window satisfies it; `gone` holds when none shows it.+ async fn check_once(+ &self,+ check: &Check,+ spec: &WindowSpec,+ started: Instant,+ ) -> Result<Checked, Miss> {+ let windows = match self.search_windows(spec).await {+ Ok(windows) => windows,+ Err(miss)+ if check.gone && matches!(miss.code, "window_not_found" | "app_not_found") =>+ {+ return Ok(Checked {+ window: None,+ observed: Some(format!("{} is closed", spec.describe())),+ waited: started.elapsed(),+ })+ }+ Err(miss) => return Err(miss),+ };+ let mut best_miss: Option<(usize, Miss)> = None;+ for window in &windows {+ let read = match self.read(window).await {+ Ok(read) => read,+ // A window that closed between the lookup and the read has+ // no elements left.+ Err(_) if check.gone => continue,+ Err(miss) => return Err(miss),+ };+ match evaluate(check, &read) {+ Ok(observed) if !check.gone => {+ return Ok(Checked {+ window: Some(window.clone()),+ observed,+ waited: started.elapsed(),+ })+ }+ Ok(_) => {}+ Err(miss) if check.gone => return Err(miss),+ Err(miss) => {+ let size = read.elements.len();+ if best_miss.as_ref().is_none_or(|(best, _)| size > *best) {+ best_miss = Some((size, miss));+ }+ }+ }+ }+ match best_miss {+ None => Ok(Checked {+ window: windows.first().cloned(),+ observed: None,+ waited: started.elapsed(),+ }),+ Some((_, mut miss)) => {+ if windows.len() > 1 {+ miss.message = format!(+ "{} (searched {} windows of the app)",+ miss.message,+ windows.len()+ );+ }+ Err(miss)+ }+ }+ }++ /// Poll until the check holds or its timeout passes.+ pub async fn check(&self, check: &Check, fallback: &WindowSpec) -> Result<Checked, Miss> {+ let window_spec = if check.window.is_empty() {+ fallback.clone()+ } else {+ check.window.clone()+ };+ let started = Instant::now();+ let deadline = (started + check.timeout).min(self.deadline);+ loop {+ let outcome = self.check_once(check, &window_spec, started).await;+ match outcome {+ Ok(checked) => return Ok(checked),+ Err(miss) if !miss.worth_waiting() => return Err(miss),+ Err(miss) => {+ if Instant::now() + POLL_INTERVAL > deadline {+ let mut miss = miss;+ miss.message = format!(+ "{} (checked for {} ms)",+ miss.message,+ started.elapsed().as_millis()+ );+ return Err(miss);+ }+ tokio::time::sleep(POLL_INTERVAL).await;+ }+ }+ }+ }+}++/// Elements of a read that match `spec`. Exact name matches win over+/// substring matches; zero-size elements only count when nothing visible+/// matches.+pub(crate) fn matches<'r>(read: &'r Read, spec: &ElementSpec) -> Vec<&'r Value> {+ let by_role: Vec<&Value> = read+ .elements+ .iter()+ .filter(|element| element.get("display_only").and_then(Value::as_bool) != Some(true))+ .filter(|element| {+ spec.role.as_deref().is_none_or(|role| {+ element+ .get("role")+ .and_then(Value::as_str)+ .is_some_and(|actual| role_matches(role, actual))+ })+ })+ .collect();+ let named: Vec<&Value> = match &spec.name {+ None => by_role,+ Some(name) => {+ let needle = fold(name);+ let exact: Vec<&Value> = by_role+ .iter()+ .copied()+ .filter(|element| label_of(element).is_some_and(|label| fold(label) == needle))+ .collect();+ if exact.is_empty() {+ by_role+ .into_iter()+ .filter(|element| {+ label_of(element).is_some_and(|label| has_word_start(&fold(label), &needle))+ })+ .collect()+ } else {+ exact+ }+ }+ };+ let visible: Vec<&Value> = named+ .iter()+ .copied()+ .filter(|element| has_area(element))+ .collect();+ if visible.is_empty() {+ named+ } else {+ visible+ }+}++fn pick(read: &Read, spec: &ElementSpec) -> Result<Found, Miss> {+ let found = matches(read, spec);+ let chosen = match (found.len(), spec.nth) {+ (0, _) => {+ return Err(Miss::new(+ "not_found",+ format!(+ "no {} in {}{}",+ spec.describe(),+ read.window.describe(),+ near_misses(read, spec)+ ),+ ))+ }+ (count, Some(nth)) if nth >= count => {+ return Err(Miss::new(+ "not_found",+ format!(+ "nth {nth} asked but only {count} {} in {}",+ spec.describe(),+ read.window.describe()+ ),+ ))+ }+ (_, Some(nth)) => found[nth],+ (1, None) => found[0],+ (count, None) => {+ let listed: Vec<String> = found+ .iter()+ .take(CANDIDATES_SHOWN)+ .enumerate()+ .map(|(nth, element)| format!("nth {nth}: {}", describe_element(element)))+ .collect();+ return Err(Miss::new(+ "ambiguous",+ format!(+ "{count} elements match {} in {}; add `nth` or a more exact name: {}",+ spec.describe(),+ read.window.describe(),+ listed.join("; ")+ ),+ ));+ }+ };+ if chosen+ .get("element_token")+ .and_then(Value::as_str)+ .is_none()+ {+ return Err(Miss::new(+ "not_addressable",+ format!(+ "{} in {} has no element_token to act on",+ describe_element(chosen),+ read.window.describe()+ ),+ ));+ }+ Ok(Found {+ window: read.window.clone(),+ element: chosen.clone(),+ })+}++/// `Ok(observed)` when the check holds against this read.+fn evaluate(check: &Check, read: &Read) -> Result<Option<String>, Miss> {+ let element_spec = ElementSpec {+ nth: None,+ ..check.element.clone()+ };+ let structured = if element_spec.role.is_some() || element_spec.name.is_some() {+ matches(read, &element_spec)+ } else {+ Vec::new()+ };+ // Static text and other display-only rows are only in the markdown.+ let rows: Vec<&str> = read+ .markdown+ .lines()+ .map(str::trim)+ .filter(|line| row_matches(line, &check.element))+ .collect();+ let by_text_only = element_spec.role.is_none() && element_spec.name.is_none();+ let present = if by_text_only {+ !rows.is_empty()+ } else {+ !structured.is_empty() || (check.element.text.is_none() && !rows.is_empty())+ };+ let window = &read.window;+ if check.gone {+ return if present {+ let shown = structured+ .first()+ .map(|element| describe_element(element))+ .or_else(|| rows.first().map(|row| clip(row, 120)))+ .unwrap_or_default();+ Err(Miss::new(+ "still_present",+ format!(+ "{} is still shown in {}: {shown}",+ check.element.describe(),+ window.describe()+ ),+ )+ .in_window(window))+ } else {+ Ok(None)+ };+ }+ if !present {+ return Err(Miss::new(+ "not_found",+ format!(+ "no {} in {}{}",+ check.element.describe(),+ window.describe(),+ near_misses(read, &check.element)+ ),+ )+ .in_window(window));+ }+ let needs_fields = check.value.is_some()+ || check.value_contains.is_some()+ || check.enabled.is_some()+ || check.selected.is_some();+ if !needs_fields {+ let observed = structured+ .first()+ .map(|element| describe_element(element))+ .or_else(|| rows.first().map(|row| clip(row, 120)));+ return Ok(observed);+ }+ let pool: Vec<&Value> = match check.element.nth {+ Some(nth) => structured.get(nth).copied().into_iter().collect(),+ None => structured.clone(),+ };+ if let Some(element) = pool.iter().find(|element| fields_hold(check, element)) {+ return Ok(Some(describe_element(element)));+ }+ let shown = pool+ .iter()+ .take(CANDIDATES_SHOWN)+ .map(|element| describe_element(element))+ .collect::<Vec<_>>()+ .join("; ");+ Err(Miss::new(+ "unexpected_state",+ format!(+ "{} expected in {}, found {}",+ check.describe(),+ window.describe(),+ if shown.is_empty() {+ "no addressable match".to_owned()+ } else {+ shown+ }+ ),+ )+ .in_window(window))+}++fn fields_hold(check: &Check, element: &Value) -> bool {+ let value = element+ .get("value")+ .and_then(Value::as_str)+ .unwrap_or_default();+ if let Some(expected) = &check.value {+ if value.trim() != expected.trim() {+ return false;+ }+ }+ if let Some(expected) = &check.value_contains {+ if !fold(value).contains(&fold(expected)) {+ return false;+ }+ }+ if let Some(expected) = check.enabled {+ // Elements report `enabled` only when the platform knows it.+ if element+ .get("enabled")+ .and_then(Value::as_bool)+ .unwrap_or(true)+ != expected+ {+ return false;+ }+ }+ if let Some(expected) = check.selected {+ if selected_state(element).unwrap_or(false) != expected {+ return false;+ }+ }+ true+}++/// An element's on/off state. For a toggle control (checkbox, radio,+/// switch) its checked value ("1"/"0", "true"/"false", "on"/"off") wins:+/// Chromium reports web checkboxes as value "1" with AXSelected false,+/// because AXSelected means list selection, not checked. Otherwise the+/// platform's `selected` flag, else that value.+fn selected_state(element: &Value) -> Option<bool> {+ let checked = element+ .get("value")+ .and_then(Value::as_str)+ .and_then(|value| match value.trim().to_ascii_lowercase().as_str() {+ "1" | "true" | "on" | "checked" => Some(true),+ "0" | "false" | "off" | "unchecked" => Some(false),+ _ => None,+ });+ let toggle = element+ .get("role")+ .and_then(Value::as_str)+ .map(role_family)+ .is_some_and(|family| matches!(family.as_str(), "checkbox" | "radiobutton" | "switch"));+ if toggle && checked.is_some() {+ return checked;+ }+ element.get("selected").and_then(Value::as_bool).or(checked)+}++fn row_matches(row: &str, spec: &ElementSpec) -> bool {+ if !row.starts_with("- ") {+ return false;+ }+ let folded = fold(row);+ if let Some(text) = &spec.text {+ if !folded.contains(&fold(text)) {+ return false;+ }+ }+ if let Some(name) = &spec.name {+ if !has_word_start(&folded, &fold(name)) {+ return false;+ }+ }+ if let Some(role) = &spec.role {+ let row_role = row+ .trim_start_matches("- ")+ .trim_start_matches(|c: char| c == '[' || c.is_ascii_digit() || c == ']')+ .split_whitespace()+ .next()+ .unwrap_or("");+ if !role_matches(role, row_role) {+ return false;+ }+ }+ spec.text.is_some() || spec.name.is_some()+}++fn near_misses(read: &Read, spec: &ElementSpec) -> String {+ // Same role, any name: the usual slip is a near-miss label.+ let relaxed = ElementSpec {+ name: None,+ text: None,+ nth: None,+ role: spec.role.clone(),+ };+ let pool: Vec<&Value> = if spec.role.is_some() {+ matches(read, &relaxed)+ } else if let Some(name) = &spec.name {+ let first_word = fold(name.split_whitespace().next().unwrap_or(name));+ read.elements+ .iter()+ .filter(|element| {+ label_of(element).is_some_and(|label| fold(label).contains(&first_word))+ })+ .collect()+ } else {+ Vec::new()+ };+ let named: Vec<String> = pool+ .iter()+ .filter(|element| label_of(element).is_some())+ .take(CANDIDATES_SHOWN)+ .map(|element| describe_element(element))+ .collect();+ if named.is_empty() {+ if read.elements.is_empty() {+ "; the window has no readable elements".to_owned()+ } else {+ String::new()+ }+ } else {+ format!("; nearest: {}", named.join("; "))+ }+}++pub(crate) fn describe_element(element: &Value) -> String {+ let role = element+ .get("role")+ .and_then(Value::as_str)+ .unwrap_or("element");+ let mut out = match element.get("element_index").and_then(Value::as_u64) {+ Some(index) => format!("[{index}] {role}"),+ None => role.to_owned(),+ };+ if let Some(label) = label_of(element) {+ out.push_str(&format!(" \"{}\"", clip(label, 60)));+ }+ if let Some(value) = element.get("value").and_then(Value::as_str) {+ if Some(value) != label_of(element) {+ out.push_str(&format!(" value=\"{}\"", clip(value, 60)));+ }+ }+ if element.get("enabled").and_then(Value::as_bool) == Some(false) {+ out.push_str(" disabled");+ }+ if element.get("selected").and_then(Value::as_bool) == Some(true) {+ out.push_str(" selected");+ }+ out+}++fn label_of(element: &Value) -> Option<&str> {+ element+ .get("label")+ .and_then(Value::as_str)+ .filter(|label| !label.trim().is_empty())+}++fn has_area(element: &Value) -> bool {+ let Some(frame) = element.get("frame") else {+ return true;+ };+ let dimension = |a: &str, b: &str| {+ frame+ .get(a)+ .or_else(|| frame.get(b))+ .and_then(Value::as_f64)+ .unwrap_or(1.0)+ };+ dimension("w", "width") > 0.0 && dimension("h", "height") > 0.0+}++fn text_matches(value: Option<&Value>, needle: &str) -> bool {+ value+ .and_then(Value::as_str)+ .is_some_and(|text| fold(text).contains(&fold(needle)))+}++/// Whether `needle` occurs in `haystack` starting at a word boundary, so+/// "ok" matches "OK" and "Ok, continue" but not "Bookmark".+fn has_word_start(haystack: &str, needle: &str) -> bool {+ if needle.is_empty() {+ return false;+ }+ haystack.match_indices(needle).any(|(at, _)| {+ haystack[..at]+ .chars()+ .next_back()+ .is_none_or(|before| !before.is_alphanumeric())+ })+}++fn fold(text: &str) -> String {+ text.trim().to_lowercase()+}++fn clip(text: &str, max: usize) -> String {+ let flat = text.replace('\n', " ");+ if flat.chars().count() <= max {+ flat+ } else {+ let mut out: String = flat.chars().take(max).collect();+ out.push('…');+ out+ }+}++fn first_text(result: &ToolResult) -> String {+ let text = result+ .content+ .iter()+ .find_map(|content| match content {+ Content::Text { text, .. } => Some(text.trim()),+ _ => None,+ })+ .unwrap_or("failed");+ clip(text.lines().next().unwrap_or(text), 300)+}++/// Role names models use, folded to one family per control kind, across the+/// macOS (AX), Windows (UIA) and Linux (AT-SPI) vocabularies.+pub(crate) fn role_family(role: &str) -> String {+ let normalized: String = role+ .chars()+ .filter(char::is_ascii_alphanumeric)+ .flat_map(|c| c.to_lowercase())+ .collect();+ let normalized = normalized+ .strip_prefix("ax")+ .unwrap_or(&normalized)+ .to_owned();+ let family = match normalized.as_str() {+ "button" | "pushbutton" | "togglebutton" => "button",+ "textfield" | "textarea" | "textbox" | "edit" | "entry" | "input" | "searchfield"+ | "securetextfield" | "passwordtext" | "text" => "textfield",+ "statictext" | "label" | "static" => "statictext",+ "link" | "hyperlink" => "link",+ "checkbox" | "checkbutton" | "check" => "checkbox",+ "radiobutton" | "radio" => "radiobutton",+ "popupbutton" | "combobox" | "menubutton" | "dropdown" | "select" => "popupbutton",+ "menuitem" | "menuitemcheckbox" | "menuitemradio" => "menuitem",+ "pagetab" | "tabitem" | "tab" => "tab",+ "image" | "img" | "icon" => "image",+ "row" | "listitem" | "tablerow" | "outlinerow" => "row",+ "cell" | "tablecell" | "gridcell" => "cell",+ other => return other.to_owned(),+ };+ family.to_owned()+}++fn role_matches(wanted: &str, actual: &str) -> bool {+ role_family(wanted) == role_family(actual)+}++#[cfg(test)]+mod tests {+ use super::*;++ fn read(elements: Vec<Value>, markdown: &str) -> Read {+ Read {+ window: Window {+ pid: 1,+ window_id: 2,+ app: Some("Demo".into()),+ title: Some("Doc".into()),+ on_screen: true,+ },+ elements,+ markdown: markdown.to_owned(),+ }+ }++ fn element(index: u64, role: &str, label: &str) -> Value {+ json!({+ "element_index": index,+ "element_token": format!("s00000001:{index}"),+ "role": role,+ "label": label,+ "frame": {"x": 0, "y": 0, "w": 10, "h": 10},+ })+ }++ #[test]+ fn exact_name_beats_substring_and_roles_fold_across_platforms() {+ let read = read(+ vec![+ element(1, "AXButton", "Save As…"),+ element(2, "AXButton", "Save"),+ element(3, "AXTextField", "Save"),+ ],+ "",+ );+ let spec = ElementSpec {+ role: Some("button".into()),+ name: Some("save".into()),+ ..Default::default()+ };+ let found = pick(&read, &spec).unwrap();+ assert_eq!(found.element["element_index"], 2);+ let field = ElementSpec {+ role: Some("textbox".into()),+ name: Some("Save".into()),+ ..Default::default()+ };+ assert_eq!(pick(&read, &field).unwrap().element["element_index"], 3);+ assert_eq!(role_family("Push Button"), "button");+ assert_eq!(role_family("Hyperlink"), "link");+ }++ #[test]+ fn a_short_name_matches_whole_words_only() {+ let read = read(+ vec![+ element(1, "AXButton", "Bookmark this tab"),+ element(2, "AXButton", "Ok, continue"),+ ],+ "",+ );+ let spec = ElementSpec {+ role: Some("button".into()),+ name: Some("ok".into()),+ ..Default::default()+ };+ assert_eq!(pick(&read, &spec).unwrap().element["element_index"], 2);+ assert!(has_word_start("save as…", "as"));+ assert!(!has_word_start("bookmark", "ok"));+ }++ #[test]+ fn ambiguity_lists_candidates_and_nth_picks_one() {+ let read = read(+ vec![element(4, "AXButton", "OK"), element(9, "AXButton", "OK")],+ "",+ );+ let spec = ElementSpec {+ role: Some("button".into()),+ name: Some("OK".into()),+ ..Default::default()+ };+ let miss = pick(&read, &spec).unwrap_err();+ assert_eq!(miss.code, "ambiguous");+ assert!(+ miss.message.contains("nth 1: [9] AXButton \"OK\""),+ "{}",+ miss.message+ );+ let second = ElementSpec {+ nth: Some(1),+ ..spec+ };+ assert_eq!(pick(&read, &second).unwrap().element["element_index"], 9);+ }++ #[test]+ fn a_miss_names_the_nearest_elements() {+ let read = read(vec![element(1, "AXButton", "Save Draft")], "");+ let spec = ElementSpec {+ role: Some("button".into()),+ name: Some("Submit".into()),+ ..Default::default()+ };+ let miss = pick(&read, &spec).unwrap_err();+ assert_eq!(miss.code, "not_found");+ assert!(+ miss.message+ .contains("nearest: [1] AXButton \"Save Draft\""),+ "{}",+ miss.message+ );+ }++ #[test]+ fn checks_read_values_static_text_and_absence() {+ let mut field = element(5, "AXTextField", "Email");+ field["value"] = json!("[email protected]");+ let read = read(+ vec![field],+ "- [0] AXWindow \"Doc\"\n - AXStaticText \"Saved to disk\"\n - [5] AXTextField \"Email\"",+ );+ let check = |raw: Value| Check::parse(&raw, 0).unwrap();+ assert!(evaluate(+ &check(json!({"role": "textfield", "name": "email", "value": "[email protected]"})),+ &read+ )+ .is_ok());+ let wrong = evaluate(&check(json!({"name": "Email", "value": "bob"})), &read).unwrap_err();+ assert_eq!(wrong.code, "unexpected_state");+ assert!(+ wrong.message.contains("value=\"[email protected]\""),+ "{}",+ wrong.message+ );+ assert!(evaluate(&check(json!({"text": "saved to disk"})), &read).is_ok());+ assert!(evaluate(&check(json!({"text": "Error", "gone": true})), &read).is_ok());+ let still = evaluate(&check(json!({"text": "Saved", "gone": true})), &read).unwrap_err();+ assert_eq!(still.code, "still_present");+ }++ #[test]+ fn selected_falls_back_to_a_checkbox_value() {+ let mut checkbox = element(3, "AXCheckBox", "Accept terms");+ checkbox["value"] = json!("1");+ // As Chromium reports it: checked, but not list-selected.+ checkbox["selected"] = json!(false);+ let read = read(vec![checkbox], "");+ let check = Check::parse(+ &json!({"role": "checkbox", "name": "Accept terms", "selected": true}),+ 0,+ )+ .unwrap();+ assert!(evaluate(&check, &read).is_ok());+ let check = Check::parse(+ &json!({"role": "checkbox", "name": "Accept terms", "selected": false}),+ 0,+ )+ .unwrap();+ assert_eq!(+ evaluate(&check, &read).unwrap_err().code,+ "unexpected_state"+ );+ }++ #[test]+ fn check_parsing_rejects_unclear_predicates() {+ for (raw, needle) in [+ (json!({}), "needs `role`, `name` or `text`"),+ (json!({"name": "x", "bogus": 1}), "unknown field `bogus`"),+ (+ json!({"name": "x", "gone": true, "value": "y"}),+ "cannot be combined",+ ),+ (json!({"text": "x", "value": "y"}), "need `role` or `name`"),+ (json!({"name": "x", "timeout_ms": 60000}), "timeout_ms"),+ (json!({"name": "x", "label": "y"}), "not both"),+ ] {+ let error = Check::parse(&raw, 0).unwrap_err();+ assert!(error.contains(needle), "{raw}: {error}");+ }+ }++ #[test]+ fn selector_keys_leave_the_action_arguments() {+ let mut args =+ json!({"role": "button", "name": "Go", "app": "Safari", "pid": 3, "count": 2})+ .as_object()+ .cloned()+ .unwrap();+ let (window, element) = take_selector(&mut args).unwrap();+ assert_eq!(window.app.as_deref(), Some("Safari"));+ assert_eq!(window.pid, Some(3));+ assert_eq!(element.name.as_deref(), Some("Go"));+ assert_eq!(args.keys().collect::<Vec<_>>(), ["count", "pid"]);+ }+}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/tests.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/tests.rsindex 962fd4c3a6..b7dc14c8b6 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/tests.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/tests.rs@@ -398,8 +398,8 @@ async fn the_batch_is_advertised_with_its_schema() {.get_def(super::RUN_ACTIONS_TOOL).expect("registered");assert_eq!(def.input_schema["required"], json!(["steps"]));- let listed: Vec<_> = def.input_schema["properties"]["steps"]["items"]["properties"]["tool"]- ["enum"]+ let listed: Vec<_> = def.input_schema["properties"]["steps"]["items"]["anyOf"][0]["properties"]+ ["tool"]["enum"].as_array().unwrap().iter()diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.jsonindex 7b2a64f569..747298151c 100644--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-linux.json@@ -3467,7 +3467,7 @@"read_only": false},{- "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag and `args` are exactly that tool's arguments. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or use pixel targets after them.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.",+ "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag and `args` are exactly that tool's arguments. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or use pixel targets after them.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.\n\nTARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.\n\nWAITS AND CHECKS: a step may add `wait_for` (checked before its action) and `expect` (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the tree, static text included; `gone:true` waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.\n\nExample (fill a form, submit, confirm the dialog, check the result):\n{\"steps\":[{\"set_value\":{\"app\":\"Safari\",\"role\":\"textfield\",\"name\":\"Email\",\"value\":\"[email protected]\"}},{\"click\":{\"role\":\"button\",\"name\":\"Submit\"},\"expect\":{\"role\":\"button\",\"name\":\"Confirm\"}},{\"click\":{\"role\":\"button\",\"name\":\"Confirm\"},\"expect\":[{\"name\":\"Confirm\",\"gone\":true},{\"text\":\"Thanks\"}]}],\"observe\":true}","destructive": true,"idempotent": false,"input_schema": {@@ -3491,34 +3491,76 @@"type": "string"},"steps": {- "description": "Ordered actions. Execution stops at the first failing step.",+ "description": "Ordered actions. Execution stops at the first failing step. Each step is {tool, args} or {<tool>: args}, plus optional wait_for / expect / timeout_ms; a step may also be only a wait_for or expect check.","items": {- "additionalProperties": false,- "properties": {- "args": {- "description": "Arguments for that tool, as in a direct call.",+ "anyOf": [+ {+ "additionalProperties": false,+ "properties": {+ "args": {+ "description": "Arguments for that tool, as in a direct call, optionally with role/name/nth/app/window to name the target instead of pid/window_id/element_token.",+ "type": "object"+ },+ "expect": {+ "description": "Check (or up to 4) that must hold after the action: {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}. Default timeout 2000 ms.",+ "type": [+ "object",+ "array"+ ]+ },+ "timeout_ms": {+ "description": "How long a named target may take to appear. Default 3000.",+ "maximum": 10000,+ "minimum": 0,+ "type": "integer"+ },+ "tool": {+ "description": "Action tool to run.",+ "enum": [+ "click",+ "double_click",+ "right_click",+ "set_value",+ "type_text",+ "press_key",+ "hotkey",+ "scroll",+ "drag"+ ],+ "type": "string"+ },+ "wait_for": {+ "description": "Check that must hold before the action: {role?, name?, text?, gone?, app?, window?, timeout_ms?}. Default timeout 5000 ms.",+ "type": "object"+ }+ },+ "required": [+ "tool"+ ],"type": "object"},- "tool": {- "description": "Action tool to run.",- "enum": [- "click",- "double_click",- "right_click",- "set_value",- "type_text",- "press_key",- "hotkey",- "scroll",- "drag"- ],- "type": "string"+ {+ "additionalProperties": true,+ "description": "Shorthand {<tool>: args} with optional wait_for / expect / timeout_ms, or a step that is only a wait_for or expect check.",+ "properties": {+ "expect": {+ "type": [+ "object",+ "array"+ ]+ },+ "timeout_ms": {+ "maximum": 10000,+ "minimum": 0,+ "type": "integer"+ },+ "wait_for": {+ "type": "object"+ }+ },+ "type": "object"}- },- "required": [- "tool"- ],- "type": "object"+ ]},"maxItems": 32,"minItems": 1,diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.jsonindex c967bfaf1e..00c24e6a87 100644--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json@@ -3103,7 +3103,7 @@"read_only": false},{- "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag and `args` are exactly that tool's arguments. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or use pixel targets after them.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.",+ "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag and `args` are exactly that tool's arguments. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or use pixel targets after them.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.\n\nTARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.\n\nWAITS AND CHECKS: a step may add `wait_for` (checked before its action) and `expect` (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the tree, static text included; `gone:true` waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.\n\nExample (fill a form, submit, confirm the dialog, check the result):\n{\"steps\":[{\"set_value\":{\"app\":\"Safari\",\"role\":\"textfield\",\"name\":\"Email\",\"value\":\"[email protected]\"}},{\"click\":{\"role\":\"button\",\"name\":\"Submit\"},\"expect\":{\"role\":\"button\",\"name\":\"Confirm\"}},{\"click\":{\"role\":\"button\",\"name\":\"Confirm\"},\"expect\":[{\"name\":\"Confirm\",\"gone\":true},{\"text\":\"Thanks\"}]}],\"observe\":true}","destructive": true,"idempotent": false,"input_schema": {@@ -3127,34 +3127,76 @@"type": "string"},"steps": {- "description": "Ordered actions. Execution stops at the first failing step.",+ "description": "Ordered actions. Execution stops at the first failing step. Each step is {tool, args} or {<tool>: args}, plus optional wait_for / expect / timeout_ms; a step may also be only a wait_for or expect check.","items": {- "additionalProperties": false,- "properties": {- "args": {- "description": "Arguments for that tool, as in a direct call.",+ "anyOf": [+ {+ "additionalProperties": false,+ "properties": {+ "args": {+ "description": "Arguments for that tool, as in a direct call, optionally with role/name/nth/app/window to name the target instead of pid/window_id/element_token.",+ "type": "object"+ },+ "expect": {+ "description": "Check (or up to 4) that must hold after the action: {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}. Default timeout 2000 ms.",+ "type": [+ "object",+ "array"+ ]+ },+ "timeout_ms": {+ "description": "How long a named target may take to appear. Default 3000.",+ "maximum": 10000,+ "minimum": 0,+ "type": "integer"+ },+ "tool": {+ "description": "Action tool to run.",+ "enum": [+ "click",+ "double_click",+ "right_click",+ "set_value",+ "type_text",+ "press_key",+ "hotkey",+ "scroll",+ "drag"+ ],+ "type": "string"+ },+ "wait_for": {+ "description": "Check that must hold before the action: {role?, name?, text?, gone?, app?, window?, timeout_ms?}. Default timeout 5000 ms.",+ "type": "object"+ }+ },+ "required": [+ "tool"+ ],"type": "object"},- "tool": {- "description": "Action tool to run.",- "enum": [- "click",- "double_click",- "right_click",- "set_value",- "type_text",- "press_key",- "hotkey",- "scroll",- "drag"- ],- "type": "string"+ {+ "additionalProperties": true,+ "description": "Shorthand {<tool>: args} with optional wait_for / expect / timeout_ms, or a step that is only a wait_for or expect check.",+ "properties": {+ "expect": {+ "type": [+ "object",+ "array"+ ]+ },+ "timeout_ms": {+ "maximum": 10000,+ "minimum": 0,+ "type": "integer"+ },+ "wait_for": {+ "type": "object"+ }+ },+ "type": "object"}- },- "required": [- "tool"- ],- "type": "object"+ ]},"maxItems": 32,"minItems": 1,diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.jsonindex 8b81bd6161..65227c254d 100644--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-windows.json@@ -3159,7 +3159,7 @@"read_only": false},{- "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag and `args` are exactly that tool's arguments. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or use pixel targets after them.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.",+ "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:true` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag and `args` are exactly that tool's arguments. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or use pixel targets after them.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step. That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 32 steps.\n\nTARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.\n\nWAITS AND CHECKS: a step may add `wait_for` (checked before its action) and `expect` (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the tree, static text included; `gone:true` waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.\n\nExample (fill a form, submit, confirm the dialog, check the result):\n{\"steps\":[{\"set_value\":{\"app\":\"Safari\",\"role\":\"textfield\",\"name\":\"Email\",\"value\":\"[email protected]\"}},{\"click\":{\"role\":\"button\",\"name\":\"Submit\"},\"expect\":{\"role\":\"button\",\"name\":\"Confirm\"}},{\"click\":{\"role\":\"button\",\"name\":\"Confirm\"},\"expect\":[{\"name\":\"Confirm\",\"gone\":true},{\"text\":\"Thanks\"}]}],\"observe\":true}","destructive": true,"idempotent": false,"input_schema": {@@ -3183,34 +3183,76 @@"type": "string"},"steps": {- "description": "Ordered actions. Execution stops at the first failing step.",+ "description": "Ordered actions. Execution stops at the first failing step. Each step is {tool, args} or {<tool>: args}, plus optional wait_for / expect / timeout_ms; a step may also be only a wait_for or expect check.","items": {- "additionalProperties": false,- "properties": {- "args": {- "description": "Arguments for that tool, as in a direct call.",+ "anyOf": [+ {+ "additionalProperties": false,+ "properties": {+ "args": {+ "description": "Arguments for that tool, as in a direct call, optionally with role/name/nth/app/window to name the target instead of pid/window_id/element_token.",+ "type": "object"+ },+ "expect": {+ "description": "Check (or up to 4) that must hold after the action: {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}. Default timeout 2000 ms.",+ "type": [+ "object",+ "array"+ ]+ },+ "timeout_ms": {+ "description": "How long a named target may take to appear. Default 3000.",+ "maximum": 10000,+ "minimum": 0,+ "type": "integer"+ },+ "tool": {+ "description": "Action tool to run.",+ "enum": [+ "click",+ "double_click",+ "right_click",+ "set_value",+ "type_text",+ "press_key",+ "hotkey",+ "scroll",+ "drag"+ ],+ "type": "string"+ },+ "wait_for": {+ "description": "Check that must hold before the action: {role?, name?, text?, gone?, app?, window?, timeout_ms?}. Default timeout 5000 ms.",+ "type": "object"+ }+ },+ "required": [+ "tool"+ ],"type": "object"},- "tool": {- "description": "Action tool to run.",- "enum": [- "click",- "double_click",- "right_click",- "set_value",- "type_text",- "press_key",- "hotkey",- "scroll",- "drag"- ],- "type": "string"+ {+ "additionalProperties": true,+ "description": "Shorthand {<tool>: args} with optional wait_for / expect / timeout_ms, or a step that is only a wait_for or expect check.",+ "properties": {+ "expect": {+ "type": [+ "object",+ "array"+ ]+ },+ "timeout_ms": {+ "maximum": 10000,+ "minimum": 0,+ "type": "integer"+ },+ "wait_for": {+ "type": "object"+ }+ },+ "type": "object"}- },- "required": [- "tool"- ],- "type": "object"+ ]},"maxItems": 32,"minItems": 1,