Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
perf(cua-driver): faster batched steps and window reads, and a nudge to batch (#5487)
Francesco Bonacci26330f7edd04
diff --git a/docs/content/docs/cua-driver/guides/use-the-sdk.mdx b/docs/content/docs/cua-driver/guides/use-the-sdk.mdxindex b707577d18..7182934392 100644--- a/docs/content/docs/cua-driver/guides/use-the-sdk.mdx+++ b/docs/content/docs/cua-driver/guides/use-the-sdk.mdx@@ -313,6 +313,14 @@ Python exposes the same objects:`environment` option. On macOS the wait also holds the focus-restorelease, so a shorter wait shortens that protection. Windows ignores both.+- **Keep the full cursor glide inside batches.** On macOS a step of+ `run_steps` or `run_script` waits at most 120 ms for the agent cursor and+ 100 ms for a window change, because another step follows at once. The last+ step of a `run_steps` batch and every direct call keep the full waits. Set+ `CUA_DRIVER_BATCH_PACING=full` in the daemon's launch environment to give+ every step the full waits, for example when you record a demo;+ `EmbeddedCuaDriverHost` admits it.+- **Shorten macOS keystroke pacing.** The macOS adapter waits 8 ms betweensynthesized key events. Set `CUA_DRIVER_KEY_GAP_MS` (2 to 100) in the samelaunch environment to change it; `EmbeddedCuaDriverHost` admits it. Onlydiff --git a/libs/cua-driver/docs/action-result-contract.md b/libs/cua-driver/docs/action-result-contract.mdindex a1a5fc8e75..d382d8c490 100644--- a/libs/cua-driver/docs/action-result-contract.md+++ b/libs/cua-driver/docs/action-result-contract.md@@ -112,6 +112,17 @@ JSON text.The codes with a hint are listed in the `RULES` table in`rust/crates/cua-driver-core/src/recovery_hint.rs`; its tests cover each one.+### Batching suggestion++An agent that acts one call at a time pays a model turn per action. After three+direct action calls in a row in one session (reads do not count, and a pause of+over 60 seconds starts the count again), the third result also carries a `hint`+that suggests chaining the next steps with `run_steps`, and so does every third+after it. On an `unverifiable` result the suggestion follows the recovery hint+in the same string. It is advice only: it has no `next_call`, it never rides on+an error, and it is never sent for a step that already runs inside `run_steps`+or `run_script`. The logic is in `rust/crates/cua-driver-core/src/batch_nudge.rs`.+## Window target resolutionWindow-scoped actions accept a PID without `window_id` only when that processdiff --git a/libs/cua-driver/rust/Cargo.lock b/libs/cua-driver/rust/Cargo.lockindex a8eb393714..d16cef1b0f 100644--- a/libs/cua-driver/rust/Cargo.lock+++ b/libs/cua-driver/rust/Cargo.lock@@ -3586,6 +3586,7 @@ dependencies = ["clipboard-rs","core-foundation","core-graphics",+ "cua-cursor-motion","cua-driver-contract","cua-driver-core","cursor-overlay",diff --git a/libs/cua-driver/rust/Skills/cua-driver/EMBEDDING.md b/libs/cua-driver/rust/Skills/cua-driver/EMBEDDING.mdindex 428d32cbab..f2aac08389 100644--- a/libs/cua-driver/rust/Skills/cua-driver/EMBEDDING.md+++ b/libs/cua-driver/rust/Skills/cua-driver/EMBEDDING.md@@ -262,6 +262,25 @@ Measured on macOS with Calculator in the background and Terminal frontmost, abackground AX click took 1153 ms with the default, 325 ms with `200`, and 78 mswith `0`.+### Steps inside `run_steps` and `run_script` (macOS)++Two waits make up most of a macOS action's latency: the agent cursor's glide to+the target (600 ms or more with the default `steady` timing) and the first 300 ms+of the window watch above. A call that is one step of a `run_steps` batch or a+`run_script` program is followed at once by another step, so the driver shortens+both for it: the step waits at most 120 ms for the cursor to arrive and 100 ms+for a window change. The cursor keeps gliding and the watch keeps running+detached, as above, so a window that opens late is still reported on the next+action's result. The last step of a `run_steps` batch, and every direct call,+keep the full waits, so the batch's final result still reports what it left+open. A script has no last step the driver can know in advance; read the window+with `getState()` or `waitFor()` after the action that opens something.++An embedding host that wants every step to play the full glide and watch, for+example to record a demo, sets `CUA_DRIVER_BATCH_PACING=full` in the same+environment as the variables above. Any other value, or none, applies the+shortened waits. No tool argument can change it.+## Pacing synthesized keystrokes (macOS)The macOS keyboard adapter waits a fixed gap between each key down and up anddiff --git a/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md b/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.mdindex 18a809d204..41f39f8314 100644--- a/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md+++ b/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md@@ -173,6 +173,8 @@ A foreground retry is never a `next_call`. When the `hint` names `delivery_mode:The fields sit next to `code` on an error result, next to `refusal` on a browser refusal, and on the `ActionResult` itself. In `run_steps` a failed step's report carries them; in `run_script` the hint is appended to the failed call's error message. A tool that already set its own `hint` or `next_call` keeps it. `next_call` is advice, not an automatic retry: never replay a partial or unknown action without a fresh read.+After three single action calls in a row in one session (not counting reads), the result of the third also carries a `hint` to chain the next ones with `run_steps`, and again after every third. It is advice with no `next_call`, and it is never sent for a step that already runs inside `run_steps` or `run_script`. Every model turn costs seconds, so chain actions whenever you already know the next few steps.+When the postcondition is visibly satisfied, stop acting. For media, selection alone is not playback: check the title, playing/pause indicator, and elapsed progress after preroll/buffering. For closure, verify the requested window disappeared. State only what the evidence proves.Never replay canceled, partial, or unknown actions automatically. An interrupted transport may have delivered input before losing its response. Report the interruption accurately and obtain fresh state before any further action.diff --git a/libs/cua-driver/rust/crates/cua-driver-contract/src/outputs.rs b/libs/cua-driver/rust/crates/cua-driver-contract/src/outputs.rsindex 4e25c31d06..de182f9142 100644--- a/libs/cua-driver/rust/crates/cua-driver-contract/src/outputs.rs+++ b/libs/cua-driver/rust/crates/cua-driver-contract/src/outputs.rs@@ -698,7 +698,9 @@ pub struct ActionResult {pub error: Option<ActionError>,/// What to do next, in one short sentence. Present when the effect is/// `unverifiable` or `suspected_noop`, so an unconfirmed action points at- /// the next step instead of leaving the agent to guess.+ /// the next step instead of leaving the agent to guess. Also carries a+ /// suggestion to chain steps with `run_steps` after several direct actions+ /// in a row.#[serde(default, skip_serializing_if = "Option::is_none")]pub hint: Option<String>,/// The call that moves the task forward, always safe to run as is (a read,diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_nudge.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_nudge.rsnew file mode 100644index 0000000000..d9d549a3d2--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_nudge.rs@@ -0,0 +1,319 @@+//! A nudge toward `run_steps` for agents that act one call at a time.+//!+//! Every single action call costs the agent a model turn, and in recorded runs+//! a turn took 3 to 5 s while the action itself took a fraction of that.+//! `run_steps` chains the actions, stops at the first failure and returns what+//! changed. This module notices a streak of direct action calls in one session+//! and puts a one-line `hint` on the third result, and on every third after it.+//!+//! The nudge is a `hint` on a successful action result, the same field the+//! recovery hints use ([`crate::recovery_hint`]). Most clicks and keypresses+//! come back `unverifiable` with a recovery hint already, so the nudge is added+//! after it instead of replacing it. It never rides on an error, and is never+//! sent for a step that already runs inside `run_steps` or `run_script`.++use std::collections::HashMap;+use std::sync::Mutex;+use std::time::{Duration, Instant};++use serde_json::Value;++use crate::protocol::ToolResult;++/// Direct actions in a row before the first nudge, and between nudges.+pub const STREAK_LEN: u32 = 3;++/// A pause longer than this ends a streak: the agent has stopped and started+/// again, so the next action is not one of an unbroken run.+pub const STREAK_GAP: Duration = Duration::from_secs(60);++/// Sessions tracked at once. The oldest is dropped past this, so a long-lived+/// daemon that sees many short sessions keeps a bounded table.+const MAX_SESSIONS: usize = 256;++/// The hint text. One sentence, so it costs few tokens.+pub const NUDGE: &str = "Three single actions in a row: send the next ones as one run_steps \+ call. It runs them in order, stops at the first failure and returns what changed.";++/// Action tools that count toward a streak: the ones `run_steps` can chain.+/// `move_cursor` only moves the agent cursor, so it is left out.+fn counts(tool: &str) -> bool {+ crate::batch_tools::BATCHABLE_TOOLS.contains(&tool) && tool != "move_cursor"+}++/// Tools that mean the agent already batches.+fn is_batch_tool(tool: &str) -> bool {+ #[cfg(feature = "script")]+ if tool == crate::script_tool::RUN_SCRIPT_TOOL {+ return true;+ }+ tool == crate::batch_tools::RUN_STEPS_TOOL+}++#[derive(Clone, Copy)]+struct Streak {+ count: u32,+ last: Instant,+}++/// Streaks per public session label (`""` for the implicit session).+#[derive(Default)]+struct Tracker {+ streaks: HashMap<String, Streak>,+}++impl Tracker {+ /// Record one successful direct call. Returns whether it should carry the+ /// nudge.+ fn note(&mut self, session: &str, tool: &str, now: Instant) -> bool {+ if is_batch_tool(tool) {+ self.streaks.remove(session);+ return false;+ }+ if !counts(tool) {+ return false;+ }+ let streak = self.streaks.get(session).copied();+ let count = match streak {+ Some(streak) if now.saturating_duration_since(streak.last) <= STREAK_GAP => {+ streak.count + 1+ }+ _ => 1,+ };+ if self.streaks.len() >= MAX_SESSIONS && !self.streaks.contains_key(session) {+ if let Some(oldest) = self+ .streaks+ .iter()+ .min_by_key(|(_, streak)| streak.last)+ .map(|(session, _)| session.clone())+ {+ self.streaks.remove(&oldest);+ }+ }+ self.streaks+ .insert(session.to_owned(), Streak { count, last: now });+ count % STREAK_LEN == 0+ }+}++static TRACKER: Mutex<Option<Tracker>> = Mutex::new(None);++fn note(session: &str, tool: &str, now: Instant) -> bool {+ let mut guard = TRACKER.lock().unwrap_or_else(|e| e.into_inner());+ guard+ .get_or_insert_with(Tracker::default)+ .note(session, tool, now)+}++/// Add the nudge to `result` when `tool`, called directly in `session`, makes+/// a streak. Call once per tool call at the dispatch seam, with the call's+/// public `session` label and whether the call is a step of a batch.+pub fn annotate(tool: &str, session: Option<&str>, in_batch: bool, result: &mut ToolResult) {+ if in_batch || result.is_error == Some(true) {+ return;+ }+ if !note(session.unwrap_or(""), tool, Instant::now()) {+ return;+ }+ let Some(Value::Object(structured)) = result.structured_content.as_mut() else {+ return;+ };+ // Only an action result carries a hint field; a read does not.+ if !structured.contains_key("effect") {+ return;+ }+ match structured.get_mut("hint") {+ Some(Value::String(existing)) => {+ existing.push(' ');+ existing.push_str(NUDGE);+ }+ Some(_) => {}+ None => {+ structured.insert("hint".to_owned(), Value::String(NUDGE.to_owned()));+ }+ }+}++#[cfg(test)]+mod tests {+ use super::*;+ use serde_json::json;++ fn action_result() -> ToolResult {+ ToolResult::text("done")+ .with_structured(json!({ "effect": "confirmed", "route": "accessibility" }))+ }++ #[test]+ fn the_third_direct_action_in_a_row_is_nudged_and_so_is_every_third_after() {+ let mut tracker = Tracker::default();+ let t0 = Instant::now();+ let nudged: Vec<bool> = (0..7)+ .map(|i| tracker.note("s", "click", t0 + Duration::from_secs(i)))+ .collect();+ assert_eq!(nudged, [false, false, true, false, false, true, false]);+ }++ #[test]+ fn different_action_tools_share_a_streak() {+ let mut tracker = Tracker::default();+ let t0 = Instant::now();+ assert!(!tracker.note("s", "click", t0));+ assert!(!tracker.note("s", "type_text", t0));+ assert!(tracker.note("s", "press_key", t0));+ }++ #[test]+ fn reads_and_cursor_moves_neither_count_nor_break_a_streak() {+ let mut tracker = Tracker::default();+ let t0 = Instant::now();+ assert!(!tracker.note("s", "click", t0));+ for tool in ["get_window_state", "list_windows", "move_cursor", "zoom"] {+ assert!(!tracker.note("s", tool, t0));+ }+ assert!(!tracker.note("s", "click", t0));+ assert!(tracker.note("s", "click", t0));+ }++ #[test]+ fn using_run_steps_or_a_script_ends_the_streak() {+ let mut tracker = Tracker::default();+ let t0 = Instant::now();+ assert!(!tracker.note("s", "click", t0));+ assert!(!tracker.note("s", "click", t0));+ assert!(!tracker.note("s", "run_steps", t0));+ assert!(!tracker.note("s", "click", t0));+ assert!(!tracker.note("s", "click", t0));+ assert!(tracker.note("s", "click", t0));+ #[cfg(feature = "script")]+ {+ assert!(!tracker.note("s", "run_script", t0));+ assert!(!tracker.note("s", "click", t0));+ }+ }++ #[test]+ fn a_long_pause_starts_a_new_streak() {+ let mut tracker = Tracker::default();+ let t0 = Instant::now();+ assert!(!tracker.note("s", "click", t0));+ assert!(!tracker.note("s", "click", t0 + Duration::from_secs(1)));+ // The agent comes back after the gap: this is the first of a new run.+ let later = t0 + STREAK_GAP + Duration::from_secs(5);+ assert!(!tracker.note("s", "click", later));+ assert!(!tracker.note("s", "click", later));+ assert!(tracker.note("s", "click", later));+ }++ #[test]+ fn sessions_keep_separate_streaks() {+ let mut tracker = Tracker::default();+ let t0 = Instant::now();+ assert!(!tracker.note("a", "click", t0));+ assert!(!tracker.note("b", "click", t0));+ assert!(!tracker.note("a", "click", t0));+ assert!(!tracker.note("b", "click", t0));+ assert!(tracker.note("a", "click", t0));+ assert!(tracker.note("b", "click", t0));+ }++ #[test]+ fn the_session_table_stays_bounded() {+ let mut tracker = Tracker::default();+ let t0 = Instant::now();+ for i in 0..(MAX_SESSIONS + 50) {+ tracker.note(+ &format!("s{i}"),+ "click",+ t0 + Duration::from_millis(i as u64),+ );+ }+ assert_eq!(tracker.streaks.len(), MAX_SESSIONS);+ // The newest sessions survive; the oldest were dropped.+ assert!(tracker+ .streaks+ .contains_key(&format!("s{}", MAX_SESSIONS + 49)));+ assert!(!tracker.streaks.contains_key("s0"));+ }++ #[test]+ fn the_nudge_rides_on_a_successful_action_result() {+ let session = "nudge-annotate-success";+ let mut results: Vec<ToolResult> = (0..3).map(|_| action_result()).collect();+ for result in &mut results {+ annotate("click", Some(session), false, result);+ }+ let hint = |result: &ToolResult| {+ result+ .structured_content+ .as_ref()+ .and_then(|value| value.get("hint"))+ .and_then(Value::as_str)+ .map(str::to_owned)+ };+ assert_eq!(hint(&results[0]), None);+ assert_eq!(hint(&results[1]), None);+ assert_eq!(hint(&results[2]).as_deref(), Some(NUDGE));+ }++ #[test]+ fn a_step_of_a_batch_is_never_nudged_or_counted() {+ let session = "nudge-annotate-batch";+ for _ in 0..6 {+ let mut result = action_result();+ annotate("click", Some(session), true, &mut result);+ assert!(result.structured_content.unwrap().get("hint").is_none());+ }+ // The six batched steps did not count: two direct calls are not yet a streak.+ for _ in 0..2 {+ let mut result = action_result();+ annotate("click", Some(session), false, &mut result);+ assert!(result.structured_content.unwrap().get("hint").is_none());+ }+ }++ #[test]+ fn an_error_is_never_nudged() {+ let session = "nudge-annotate-error";+ for _ in 0..2 {+ annotate("click", Some(session), false, &mut action_result());+ }+ let mut refused = ToolResult::error("refused")+ .with_structured(json!({ "code": "background_unavailable", "effect": "refused" }));+ annotate("click", Some(session), false, &mut refused);+ assert!(refused.structured_content.unwrap().get("hint").is_none());+ }++ #[test]+ fn the_nudge_follows_a_recovery_hint_instead_of_replacing_it() {+ let session = "nudge-annotate-append";+ let unverifiable = || {+ ToolResult::text("done").with_structured(json!({+ "effect": "unverifiable",+ "route": "synthetic_events",+ "hint": "Read what changed before repeating.",+ }))+ };+ for _ in 0..2 {+ annotate("click", Some(session), false, &mut unverifiable());+ }+ let mut third = unverifiable();+ annotate("click", Some(session), false, &mut third);+ assert_eq!(+ third.structured_content.unwrap()["hint"],+ format!("Read what changed before repeating. {NUDGE}")+ );+ }++ #[test]+ fn a_result_that_is_not_an_action_result_is_left_alone() {+ let session = "nudge-annotate-read";+ for _ in 0..2 {+ annotate("click", Some(session), false, &mut action_result());+ }+ // The third direct action has no `effect`: nothing to attach a hint to.+ let mut odd = ToolResult::text("done").with_structured(json!({ "ok": true }));+ annotate("click", Some(session), false, &mut odd);+ assert_eq!(odd.structured_content.unwrap(), json!({ "ok": true }));+ }+}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rsindex eec03f0971..286d429153 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools.rs@@ -675,14 +675,26 @@ impl Plan {json!({}),)} else {- run_step(- registry,- &locator,- index,- step,- &mut context,- &mut last_window,- self.foreground_fallback,+ // Every step but the last is followed by another, so it does+ // not hold up the batch for the cursor glide or the full+ // window-change report. The last step keeps them: it reports+ // what the batch left open.+ let pacing = if index + 1 < total {+ crate::step_pacing::StepPacing::Batched+ } else {+ crate::step_pacing::StepPacing::Single+ };+ crate::step_pacing::scope(+ pacing,+ run_step(+ registry,+ &locator,+ index,+ step,+ &mut context,+ &mut last_window,+ self.foreground_fallback,+ ),).await};diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/tests.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/tests.rsindex 9783adc48b..07a79e8c3b 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/tests.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/batch_tools/tests.rs@@ -19,6 +19,7 @@ use crate::{struct Probe {def: ToolDef,calls: Arc<Mutex<Vec<Value>>>,+ pacings: Arc<Mutex<Vec<(String, crate::step_pacing::StepPacing)>>>,}#[async_trait]@@ -29,6 +30,10 @@ impl Tool for Probe {async fn invoke(&self, args: Value) -> ToolResult {self.calls.lock().unwrap().push(args.clone());+ self.pacings+ .lock()+ .unwrap()+ .push((self.def.name.clone(), crate::step_pacing::current()));if let Some(message) = args["fail_message"].as_str() {return ToolResult::error(message.to_owned());}@@ -43,12 +48,14 @@ impl Tool for Probe {struct Harness {registry: Arc<ToolRegistry>,calls: std::collections::HashMap<&'static str, Arc<Mutex<Vec<Value>>>>,+ pacings: Arc<Mutex<Vec<(String, crate::step_pacing::StepPacing)>>>,}impl Harness {fn new() -> Self {let mut registry = ToolRegistry::new();let mut calls = std::collections::HashMap::new();+ let pacings: Arc<Mutex<Vec<(String, crate::step_pacing::StepPacing)>>> = Arc::default();for name in super::BATCHABLE_TOOLS.iter().copied().chain(["get_window_state","zoom",@@ -98,12 +105,17 @@ impl Harness {open_world: false,},calls: log,+ pacings: pacings.clone(),}));}registry.register_session_tools();let registry = Arc::new(registry);registry.init_self_weak();- Self { registry, calls }+ Self {+ registry,+ calls,+ pacings,+ }}fn hits(&self, tool: &str) -> usize {@@ -161,6 +173,49 @@ fn text(result: &ToolResult) -> String {.join("\n")}+#[tokio::test]+async fn every_step_but_the_last_is_paced_as_a_batched_step() {+ use crate::step_pacing::StepPacing::{Batched, Single};+ let harness = Harness::new();+ let result = harness+ .run(json!({"steps": [+ {"tool": "click", "args": {"pid": 42, "window_id": 7}},+ {"tool": "type_text", "args": {"pid": 42, "text": "b"}},+ {"tool": "press_key", "args": {"pid": 42}},+ ]}))+ .await;+ assert_ne!(result.is_error, Some(true), "{}", text(&result));+ let pacings: Vec<_> = harness+ .pacings+ .lock()+ .unwrap()+ .iter()+ .map(|(tool, pacing)| (tool.clone(), *pacing))+ .collect();+ assert_eq!(+ pacings,+ vec![+ ("click".to_owned(), Batched),+ ("type_text".to_owned(), Batched),+ ("press_key".to_owned(), Single),+ ]+ );+}++#[tokio::test]+async fn a_one_step_batch_keeps_the_full_waits() {+ use crate::step_pacing::StepPacing::Single;+ let harness = Harness::new();+ let result = harness+ .run(json!({"steps": [+ {"tool": "click", "args": {"pid": 42, "window_id": 7}},+ ]}))+ .await;+ assert_ne!(result.is_error, Some(true), "{}", text(&result));+ let pacings = harness.pacings.lock().unwrap().clone();+ assert_eq!(pacings, vec![("click".to_owned(), Single)]);+}+#[tokio::test]async fn runs_steps_in_order_and_reports_each() {let harness = Harness::new();diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rsindex 5b739537ff..f34132090a 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs@@ -54,6 +54,7 @@ pub mod action_target;pub mod agent_cursor;pub mod authorization;pub mod background_input;+pub mod batch_nudge;pub mod batch_tools;pub mod browser;pub mod build_info;@@ -108,6 +109,7 @@ pub mod snapshot_store;#[cfg(test)]pub(crate) mod snapshot_test_support;pub mod socket_io;+pub mod step_pacing;pub mod text_sanitize;pub mod timestamp;pub mod tool;diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/script_tool.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/script_tool.rsindex 3757bc9569..fff364837d 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/script_tool.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/script_tool.rs@@ -483,7 +483,17 @@ impl<'a> Api<'a> {if let Some(session) = &self.session {args.insert("session".into(), json!(session));}- let result = self.registry.invoke(tool, Value::Object(args)).await;+ // A script chains its actions, so each one returns without waiting out+ // the cursor glide or the full window-change report; the script's own+ // reads (`getState`, `waitFor`) see what an action opened.+ let pacing = if BATCHABLE_TOOLS.contains(&tool) {+ crate::step_pacing::StepPacing::Batched+ } else {+ crate::step_pacing::StepPacing::Single+ };+ let result =+ crate::step_pacing::scope(pacing, self.registry.invoke(tool, Value::Object(args)))+ .await;let mut text = String::new();for content in &result.content {match content {diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/script_tool/tests.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/script_tool/tests.rsindex 5aeb48387a..47b0d49136 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/script_tool/tests.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/script_tool/tests.rs@@ -13,11 +13,13 @@ use crate::{};type Calls = Arc<Mutex<Vec<(String, Value)>>>;+type Pacings = Arc<Mutex<Vec<(String, crate::step_pacing::StepPacing)>>>;/// Fake app: pid 42, window 7, a "Name" field and a "Save" button.struct Fake {def: ToolDef,calls: Calls,+ pacings: Pacings,}#[async_trait]@@ -28,6 +30,10 @@ impl Tool for Fake {async fn invoke(&self, args: Value) -> ToolResult {let name = self.def.name.clone();+ self.pacings+ .lock()+ .unwrap()+ .push((name.clone(), crate::step_pacing::current()));self.calls.lock().unwrap()@@ -61,11 +67,13 @@ impl Tool for Fake {struct Harness {registry: Arc<ToolRegistry>,calls: Calls,+ pacings: Pacings,}impl Harness {fn new() -> Self {let calls: Calls = Arc::default();+ let pacings: Pacings = Arc::default();let mut registry = ToolRegistry::new();for name in crate::batch_tools::BATCHABLE_TOOLS.iter()@@ -84,13 +92,18 @@ impl Harness {open_world: false,},calls: calls.clone(),+ pacings: pacings.clone(),}));}registry.register_session_tools();registry.register_script_tool();let registry = Arc::new(registry);registry.init_self_weak();- Self { registry, calls }+ Self {+ registry,+ calls,+ pacings,+ }}async fn run(&self, script: &str) -> ToolResult {@@ -542,3 +555,36 @@ async fn query_returns_static_text_rows_with_their_value() {assert_eq!(value["rows"][0]["window"]["window_id"], 7);assert_eq!(value["kinds"], json!(["AXButton", "text"]));}++#[tokio::test]+async fn script_actions_are_paced_as_batched_steps_and_reads_are_not() {+ let harness = Harness::new();+ let result = harness+ .run(+ "const app = await cua.getApp({pid: 42, windowId: 7});\n\+ await app.getState();\n\+ await app.click({role: 'AXButton', name: 'Save'});\n\+ await app.pressKey('return');\n\+ return 'ok';",+ )+ .await;+ assert_ne!(result.is_error, Some(true), "{result:?}");+ let pacings = harness.pacings.lock().unwrap().clone();+ let of = |tool: &str| -> Vec<crate::step_pacing::StepPacing> {+ pacings+ .iter()+ .filter(|(name, _)| name == tool)+ .map(|(_, pacing)| *pacing)+ .collect()+ };+ use crate::step_pacing::StepPacing::{Batched, Single};+ assert_eq!(of("click"), vec![Batched]);+ assert_eq!(of("press_key"), vec![Batched]);+ assert!(+ of("get_window_state")+ .iter()+ .all(|pacing| *pacing == Single),+ "reads keep their own timing: {:?}",+ of("get_window_state")+ );+}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/step_pacing.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/step_pacing.rsnew file mode 100644index 0000000000..4ada4bdd26--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/step_pacing.rs@@ -0,0 +1,180 @@+//! How an action is paced when it is one step of a batch.+//!+//! A single tool call waits for two things that exist for the person watching+//! and for the next read: the agent cursor's glide to the target, and the+//! post-action window-change observation that reports a menu, dialog or new+//! window the action opened. Measured on one agent run, those two waits are+//! most of an action's latency: a click took about 1.4 s, of which the glide+//! floor and the observation were roughly 0.9 s.+//!+//! Inside `run_steps` and `run_script` the caller has already decided to chain+//! actions, and the next step starts right away. The batch executors mark each+//! step with [`StepPacing::Batched`], and the platform adapters then shorten+//! both waits (see [`BATCH_GLIDE_WAIT`] and [`BATCH_REPORT_WINDOW`]). The+//! cursor keeps gliding and the window watch keeps running detached, so only+//! the time the step is held up changes. A direct call keeps the full waits,+//! and so does the last step of a `run_steps` batch, which reports what the+//! batch left open.+//!+//! The marker is a task-local set by the batch executors. Callers cannot set it+//! through arguments, and a daemon operator can turn the shortening off with+//! [`BATCH_PACING_ENV`]. See `Skills/cua-driver/EMBEDDING.md`.++use std::future::Future;+use std::time::Duration;++/// Daemon environment switch. `full` keeps the full cursor glide and the full+/// window-change report for every step of a batch, as for a direct call. Any+/// other value, or none, applies the batch pacing.+pub const BATCH_PACING_ENV: &str = "CUA_DRIVER_BATCH_PACING";++/// Longest a batched step waits for the agent cursor to arrive. The glide+/// itself still plays to the end.+pub const BATCH_GLIDE_WAIT: Duration = Duration::from_millis(120);++/// How long a batched step waits for a window change to show before it returns.+/// The watch continues detached for the rest of its deadline, and a late change+/// is reported on the next action's result.+pub const BATCH_REPORT_WINDOW: Duration = Duration::from_millis(100);++/// The role of the action being dispatched within the current tool call.+#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]+pub enum StepPacing {+ /// A direct call, or the last step of a batch: full waits.+ #[default]+ Single,+ /// A step that another step follows: shortened waits.+ Batched,+}++tokio::task_local! {+ static CURRENT: StepPacing;+}++/// Run `future` with `pacing` as the pacing of every action it dispatches.+pub async fn scope<F: Future>(pacing: StepPacing, future: F) -> F::Output {+ CURRENT.scope(pacing, future).await+}++/// Whether the action being dispatched is a step of a `run_steps` batch or a+/// `run_script` program, as opposed to a call the agent made directly.+pub fn in_batch() -> bool {+ CURRENT.try_with(|_| ()).is_ok()+}++/// The pacing of the action being dispatched. [`StepPacing::Single`] outside a+/// batch, and whenever the operator set [`BATCH_PACING_ENV`] to `full`.+pub fn current() -> StepPacing {+ let pacing = CURRENT.try_with(|pacing| *pacing).unwrap_or_default();+ resolve(pacing, std::env::var(BATCH_PACING_ENV).ok().as_deref())+}++/// Apply the operator switch to a step's pacing. Pure, so tests do not touch+/// the process environment.+fn resolve(pacing: StepPacing, env: Option<&str>) -> StepPacing {+ match env {+ Some(value) if value.trim().eq_ignore_ascii_case("full") => StepPacing::Single,+ _ => pacing,+ }+}++/// The longest a step waits for the cursor to arrive, given the arrival limit+/// of a direct call.+pub fn glide_wait_limit(direct: Duration) -> Duration {+ match current() {+ StepPacing::Single => direct,+ StepPacing::Batched => direct.min(BATCH_GLIDE_WAIT),+ }+}++/// How long the step's result waits for a window change, given the report+/// window of a direct call.+pub fn report_window(direct: Duration) -> Duration {+ match current() {+ StepPacing::Single => direct,+ StepPacing::Batched => direct.min(BATCH_REPORT_WINDOW),+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ #[tokio::test]+ async fn a_direct_call_keeps_the_full_waits() {+ assert_eq!(current(), StepPacing::Single);+ let direct = Duration::from_secs(8);+ assert_eq!(glide_wait_limit(direct), direct);+ assert_eq!(+ report_window(Duration::from_millis(300)),+ Duration::from_millis(300)+ );+ }++ #[tokio::test]+ async fn a_batched_step_caps_both_waits() {+ scope(StepPacing::Batched, async {+ assert_eq!(current(), StepPacing::Batched);+ assert_eq!(glide_wait_limit(Duration::from_secs(8)), BATCH_GLIDE_WAIT);+ assert_eq!(+ report_window(Duration::from_millis(300)),+ BATCH_REPORT_WINDOW+ );+ })+ .await;+ }++ #[tokio::test]+ async fn the_caps_never_lengthen_a_shorter_wait() {+ scope(StepPacing::Batched, async {+ let short = Duration::from_millis(40);+ assert_eq!(glide_wait_limit(short), short);+ assert_eq!(report_window(short), short);+ })+ .await;+ }++ #[tokio::test]+ async fn a_scope_marks_its_calls_as_inside_a_batch() {+ assert!(!in_batch());+ scope(StepPacing::Single, async {+ assert!(in_batch(), "the last step of a batch is still a batch step");+ })+ .await;+ scope(StepPacing::Batched, async {+ assert!(in_batch());+ })+ .await;+ assert!(!in_batch());+ }++ #[tokio::test]+ async fn a_nested_single_scope_restores_full_waits() {+ scope(StepPacing::Batched, async {+ scope(StepPacing::Single, async {+ assert_eq!(current(), StepPacing::Single);+ })+ .await;+ assert_eq!(current(), StepPacing::Batched);+ })+ .await;+ }++ #[test]+ fn the_operator_switch_turns_the_shortening_off() {+ assert_eq!(resolve(StepPacing::Batched, None), StepPacing::Batched);+ assert_eq!(resolve(StepPacing::Batched, Some("")), StepPacing::Batched);+ assert_eq!(+ resolve(StepPacing::Batched, Some("fast")),+ StepPacing::Batched+ );+ assert_eq!(+ resolve(StepPacing::Batched, Some(" Full ")),+ StepPacing::Single+ );+ assert_eq!(+ resolve(StepPacing::Single, Some("full")),+ StepPacing::Single+ );+ }+}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rsindex 103ebcc5d7..5493c6c386 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs@@ -1688,10 +1688,17 @@ impl ToolRegistry {evidence: &TrustedInvocationEvidence,) -> ToolResult {let seed = crate::recovery_hint::Seed::from_args(&args);+ let session = args+ .get("session")+ .and_then(Value::as_str)+ .map(str::to_owned);+ let in_batch = crate::step_pacing::in_batch();let mut result = self.invoke_authorized_unannotated(name, args, context, evidence).await;- crate::recovery_hint::annotate(renamed_tool(name).unwrap_or(name), &seed, &mut result);+ let tool = renamed_tool(name).unwrap_or(name);+ crate::recovery_hint::annotate(tool, &seed, &mut result);+ crate::batch_nudge::annotate(tool, session.as_deref(), in_batch, &mut result);result}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/tests/batch_nudge_dispatch.rs b/libs/cua-driver/rust/crates/cua-driver-core/tests/batch_nudge_dispatch.rsnew file mode 100644index 0000000000..46a434f6cd--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-core/tests/batch_nudge_dispatch.rs@@ -0,0 +1,138 @@+//! An agent that sends action after action as single calls is told, at the+//! dispatch seam, to chain them with `run_steps`. Steps inside `run_steps` are+//! never nudged.++use cua_driver_core::authorization::PermissionMode;+use cua_driver_core::batch_nudge::NUDGE;+use cua_driver_core::protocol::ToolResult;+use cua_driver_core::session_authorization::{+ EffectiveAuthorizationContext, SessionAuthorizationRegistry, SessionModeCeiling,+};+use cua_driver_core::tool::{Tool, ToolDef, ToolRegistry};+use serde_json::{json, Value};+use std::sync::Arc;+use std::time::Duration;++/// A stand-in action tool whose text-only result the registry turns into an+/// `unverifiable` action result, as a click is in practice.+struct Stub(ToolDef);++#[async_trait::async_trait]+impl Tool for Stub {+ fn def(&self) -> &ToolDef {+ &self.0+ }++ async fn invoke(&self, _args: Value) -> ToolResult {+ ToolResult::text("Posted")+ }+}++fn unrestricted_context() -> Arc<EffectiveAuthorizationContext> {+ let ceiling = SessionModeCeiling::for_trusted_sessions(+ [PermissionMode::Unrestricted],+ true,+ Duration::from_secs(60),+ Duration::from_secs(30),+ )+ .unwrap();+ SessionAuthorizationRegistry::with_ceiling(ceiling)+ .compatibility_context(PermissionMode::Unrestricted, None)+ .unwrap()+}++fn registry() -> Arc<ToolRegistry> {+ let mut registry = ToolRegistry::new();+ registry.register_session_tools();+ for name in ["click", "press_key", "get_window_state"] {+ registry.register(Box::new(Stub(ToolDef {+ name: name.to_owned(),+ description: "stub".to_owned(),+ input_schema: json!({+ "type": "object",+ "properties": {+ "pid": {"type": "integer"},+ "window_id": {"type": "integer"},+ "session": {"type": "string"},+ "key": {"type": "string"},+ },+ }),+ read_only: name == "get_window_state",+ destructive: false,+ idempotent: false,+ open_world: false,+ })));+ }+ let registry = Arc::new(registry);+ registry.init_self_weak();+ registry+}++async fn call(registry: &ToolRegistry, tool: &str, args: Value) -> ToolResult {+ registry+ .invoke_with_context(tool, args, unrestricted_context())+ .await+}++fn hint(result: &ToolResult) -> Option<String> {+ result+ .structured_content+ .as_ref()+ .and_then(|value| value.get("hint"))+ .and_then(Value::as_str)+ .map(str::to_owned)+}++#[tokio::test]+async fn the_third_direct_action_carries_the_nudge_after_its_own_hint() {+ let registry = registry();+ let args = || json!({"pid": 42, "window_id": 7, "session": "nudge-direct"});+ let first = call(®istry, "click", args()).await;+ let second = call(+ ®istry,+ "press_key",+ json!({"pid": 42, "key": "tab", "session": "nudge-direct"}),+ )+ .await;+ // A read between actions neither counts nor breaks the streak.+ call(®istry, "get_window_state", args()).await;+ let third = call(®istry, "click", args()).await;++ let own = hint(&first).expect("an unverifiable click carries its recovery hint");+ assert!(!own.contains(NUDGE));+ assert!(!hint(&second).unwrap_or_default().contains(NUDGE));+ let third = hint(&third).expect("hint");+ assert!(third.ends_with(NUDGE), "{third}");+ assert!(third.starts_with(&own), "{third}");++ // The nudge is advice only: it adds no call to make.+ let fourth = call(®istry, "click", args()).await;+ assert!(!hint(&fourth).unwrap_or_default().contains(NUDGE));+}++#[tokio::test]+async fn steps_inside_run_steps_are_not_nudged_and_do_not_count() {+ let registry = registry();+ let batch = call(+ ®istry,+ "run_steps",+ json!({"session": "nudge-batch", "steps": [+ {"tool": "click", "args": {"pid": 42, "window_id": 7}},+ {"tool": "click", "args": {"pid": 42, "window_id": 7}},+ {"tool": "click", "args": {"pid": 42, "window_id": 7}},+ {"tool": "click", "args": {"pid": 42, "window_id": 7}},+ ]}),+ )+ .await;+ let rendered = serde_json::to_string(&batch.structured_content).unwrap();+ assert!(!rendered.contains(NUDGE), "{rendered}");++ // Four batched steps did not make a streak: the next direct call is the first.+ let direct = call(+ ®istry,+ "click",+ json!({"pid": 42, "window_id": 7, "session": "nudge-batch"}),+ )+ .await;+ assert!(!hint(&direct).unwrap_or_default().contains(NUDGE));+}diff --git a/libs/cua-driver/rust/crates/cua-driver-sdk/src/embedded.rs b/libs/cua-driver/rust/crates/cua-driver-sdk/src/embedded.rsindex abd692c362..99075b3b9b 100644--- a/libs/cua-driver/rust/crates/cua-driver-sdk/src/embedded.rs+++ b/libs/cua-driver/rust/crates/cua-driver-sdk/src/embedded.rs@@ -8,6 +8,7 @@ use cua_driver_contract::{};use cua_driver_core::daemon::{request_daemon_metadata, DaemonMetadata};use cua_driver_core::key_pacing::KEY_GAP_ENV;+use cua_driver_core::step_pacing::BATCH_PACING_ENV;use cua_driver_core::window_observation::{WINDOW_CHANGE_POLL_ENV, WINDOW_CHANGE_TIMEOUT_ENV};use std::collections::BTreeMap;use std::process::Stdio;@@ -842,6 +843,7 @@ pub(crate) fn allowed_environment_name(name: &str) -> bool {|| upper == WINDOW_CHANGE_TIMEOUT_ENV|| upper == WINDOW_CHANGE_POLL_ENV|| upper == KEY_GAP_ENV+ || upper == BATCH_PACING_ENV|| matches!(upper.as_str(),"PATH"@@ -1593,6 +1595,20 @@ mod tests {.any(|v| v.name.eq_ignore_ascii_case("CUA_DRIVER_KEY_GAP_MS") && v.value == "3"));}+ /// `CUA_DRIVER_BATCH_PACING` must reach an embedded child, or a host that+ /// sets it to record full-speed cursor motion is silently overridden.+ #[test]+ fn batch_pacing_env_propagates() {+ assert!(allowed_environment_name("CUA_DRIVER_BATCH_PACING"));+ let merged = merge_safe_environment(+ [("CUA_DRIVER_BATCH_PACING".to_owned(), "full".to_owned())],+ &[],+ );+ assert!(merged+ .iter()+ .any(|v| v.name.eq_ignore_ascii_case("CUA_DRIVER_BATCH_PACING") && v.value == "full"));+ }+#[cfg(target_os = "windows")]#[tokio::test]async fn daemon_console_window_follows_the_host_console() {diff --git a/libs/cua-driver/rust/crates/platform-macos/Cargo.toml b/libs/cua-driver/rust/crates/platform-macos/Cargo.tomlindex 8d429484aa..b457f1342b 100644--- a/libs/cua-driver/rust/crates/platform-macos/Cargo.toml+++ b/libs/cua-driver/rust/crates/platform-macos/Cargo.toml@@ -78,6 +78,7 @@ futures-util = "0.3"[dev-dependencies]tempfile = "3"+cua-cursor-motion = { path = "../cua-cursor-motion" }[[test]]name = "embedded_menu_restore"diff --git a/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rs b/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rsindex 358037ea5d..b3fb832086 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rs@@ -449,12 +449,32 @@ pub async fn animate_cursor_to_target(key: CursorKey, x: f64, y: f64, target: Op// thread never entered `run_on_main_thread`): the command stays queued// for a loop that starts later, and the action proceeds now. Bounded// either way.- wait_for_arrival(- rx,- RENDER_LOOP_RUNNING.load(Ordering::Acquire),- ARRIVAL_TIMEOUT,- )- .await;+ await_arrival(rx, RENDER_LOOP_RUNNING.load(Ordering::Acquire)).await;+}++/// Waits for the cursor's arrival signal for as long as the current step's+/// pacing allows. A step of a batch waits only briefly for the glide (it plays+/// on), so a chain of actions is not held up by one cursor move per step.+pub(crate) async fn await_arrival(+ rx: tokio::sync::oneshot::Receiver<()>,+ render_loop_running: bool,+) -> bool {+ let limit = cua_driver_core::step_pacing::glide_wait_limit(ARRIVAL_TIMEOUT);+ if limit < ARRIVAL_TIMEOUT {+ wait_for_arrival_briefly(rx, render_loop_running, limit).await+ } else {+ wait_for_arrival(rx, render_loop_running, ARRIVAL_TIMEOUT).await+ }+}++/// [`wait_for_arrival`] for a deliberately short wait: the cursor is still+/// gliding when the limit passes, which is expected and not logged.+async fn wait_for_arrival_briefly(+ rx: tokio::sync::oneshot::Receiver<()>,+ render_loop_running: bool,+ limit: Duration,+) -> bool {+ render_loop_running && tokio::time::timeout(limit, rx).await.is_ok()}/// Waits for `rx` only while a render loop runs, and never longer than@@ -1445,6 +1465,29 @@ mod tests {assert!(rt.block_on(wait_for_arrival(rx, true, Duration::from_secs(60))));}+ #[test]+ fn a_batched_step_waits_for_the_glide_only_up_to_the_batch_cap() {+ use cua_driver_core::step_pacing::{glide_wait_limit, scope, StepPacing, BATCH_GLIDE_WAIT};+ let rt = runtime();+ // The sender stays alive and never fires: the glide is still playing.+ let (_tx, rx) = tokio::sync::oneshot::channel::<()>();+ let started = Instant::now();+ let arrived = rt.block_on(scope(StepPacing::Batched, async {+ let limit = glide_wait_limit(ARRIVAL_TIMEOUT);+ assert_eq!(limit, BATCH_GLIDE_WAIT);+ wait_for_arrival_briefly(rx, true, limit).await+ }));+ assert!(!arrived);+ assert!(started.elapsed() >= BATCH_GLIDE_WAIT);+ assert!(started.elapsed() < ARRIVAL_TIMEOUT / 4);+ // A direct call keeps the full arrival limit.+ assert_eq!(glide_wait_limit(ARRIVAL_TIMEOUT), ARRIVAL_TIMEOUT);+ // An arrival inside the cap is still seen.+ let (tx, rx) = tokio::sync::oneshot::channel::<()>();+ tx.send(()).unwrap();+ assert!(rt.block_on(wait_for_arrival_briefly(rx, true, BATCH_GLIDE_WAIT)));+ }+fn window(window_id: u32, pid: i32, z_index: usize) -> crate::windows::WindowInfo {crate::windows::WindowInfo {window_id,diff --git a/libs/cua-driver/rust/crates/platform-macos/src/lib.rs b/libs/cua-driver/rust/crates/platform-macos/src/lib.rsindex 399a636aca..6b847b452c 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/lib.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/lib.rs@@ -55,6 +55,8 @@ pub mod pointer_shape_map;pub mod recording_hooks;#[cfg(target_os = "macos")]pub mod session;+#[cfg(all(test, target_os = "macos"))]+mod step_latency_bench;#[cfg(target_os = "macos")]pub mod terminal;#[cfg(target_os = "macos")]diff --git a/libs/cua-driver/rust/crates/platform-macos/src/step_latency_bench.rs b/libs/cua-driver/rust/crates/platform-macos/src/step_latency_bench.rsnew file mode 100644index 0000000000..c009f830ea--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-macos/src/step_latency_bench.rs@@ -0,0 +1,139 @@+//! Micro-benchmark: how long one step of a batch is held up by the agent+//! cursor's glide and by the post-action window watch, before and after the+//! batch pacing in `cua_driver_core::step_pacing`.+//!+//! It runs the driver's own waits with no accessibility or screen-recording+//! permission and no window: the glide time comes from the cursor planner with+//! the default `steady` timing, the arrival is delivered when that glide ends,+//! and the window watch polls a synthetic window list that never changes (an+//! action that opens nothing). Run it on any macOS host:+//!+//! ```sh+//! cd libs/cua-driver/rust+//! cargo test --release -p platform-macos --lib step_latency_bench \+//! -- --ignored --nocapture+//! ```+//!+//! Each row is one JSON line (`distance_px`, `glide_ms`, `direct_ms`,+//! `batched_ms`), followed by the totals for a typical pointer step and for a+//! key step. It measures only the two waits. It does not time the action+//! itself (an AX press takes about 80 ms) or a real glide on a live overlay.++use std::time::{Duration, Instant};++use cua_cursor_motion::{plan_move, MotionParams, MoveRequest, Pt};+use cua_driver_core::step_pacing::{+ report_window, scope, StepPacing, BATCH_GLIDE_WAIT, BATCH_REPORT_WINDOW,+};+use cua_driver_core::window_observation::WindowObservationBounds;++use crate::cursor::overlay::await_arrival;+use crate::window_change_detector::{observe_quiet_window_list, DIRECT_REPORT_WINDOW};++const DISTANCES_PX: [f64; 4] = [120.0, 320.0, 640.0, 1100.0];+const RUNS: usize = 5;++/// How long one default-timing cursor move of `distance` points takes to+/// arrive, in ms. The overlay releases the waiting action at arrival; the+/// settle after it plays during the action.+fn glide_ms(distance: f64) -> f64 {+ plan_move(+ &MotionParams::default(),+ &MoveRequest::new(Pt::new(100.0, 400.0), Pt::new(100.0 + distance, 400.0)),+ )+ .arrival_t+ * 1000.0+}++/// Median of `RUNS` timings of `once`, in ms.+fn median_ms(mut once: impl FnMut() -> Duration) -> f64 {+ let mut runs: Vec<f64> = (0..RUNS).map(|_| once().as_secs_f64() * 1000.0).collect();+ runs.sort_by(|a, b| a.total_cmp(b));+ runs[RUNS / 2]+}++/// How long a step with this pacing waits for a glide of `glide` to arrive.+fn glide_hold_up(pacing: StepPacing, glide: Duration) -> Duration {+ let runtime = tokio::runtime::Builder::new_current_thread()+ .enable_all()+ .build()+ .unwrap();+ runtime.block_on(scope(pacing, async move {+ let (tx, rx) = tokio::sync::oneshot::channel::<()>();+ std::thread::spawn(move || {+ std::thread::sleep(glide);+ let _ = tx.send(());+ });+ let started = Instant::now();+ await_arrival(rx, true).await;+ started.elapsed()+ }))+}++/// How long a step with this pacing waits for the window watch to report a+/// quiet window list.+fn watch_hold_up(pacing: StepPacing) -> Duration {+ let runtime = tokio::runtime::Builder::new_current_thread()+ .enable_all()+ .build()+ .unwrap();+ let report = runtime.block_on(scope(pacing, async { report_window(DIRECT_REPORT_WINDOW) }));+ let bounds = WindowObservationBounds::from_raw(+ None,+ None,+ Duration::from_millis(1000),+ Duration::from_millis(25),+ );+ let started = Instant::now();+ observe_quiet_window_list(bounds, report);+ started.elapsed()+}++#[test]+#[ignore = "benchmark: run with --ignored --nocapture"]+fn batched_steps_are_held_up_less_by_the_glide_and_the_window_watch() {+ println!(+ "step pacing: batch glide cap {} ms, batch report window {} ms, direct report window {} ms",+ BATCH_GLIDE_WAIT.as_millis(),+ BATCH_REPORT_WINDOW.as_millis(),+ DIRECT_REPORT_WINDOW.as_millis()+ );++ let mut glide_direct_total = 0.0;+ let mut glide_batched_total = 0.0;+ for distance in DISTANCES_PX {+ let glide = glide_ms(distance);+ let glide_duration = Duration::from_secs_f64(glide / 1000.0);+ let direct = median_ms(|| glide_hold_up(StepPacing::Single, glide_duration));+ let batched = median_ms(|| glide_hold_up(StepPacing::Batched, glide_duration));+ println!(+ "{{\"wait\":\"cursor_glide\",\"distance_px\":{distance},\"glide_ms\":{glide:.0},\+ \"direct_ms\":{direct:.0},\"batched_ms\":{batched:.0}}}"+ );+ assert!(batched <= direct + 5.0);+ glide_direct_total += direct;+ glide_batched_total += batched;+ }+ let glide_direct = glide_direct_total / DISTANCES_PX.len() as f64;+ let glide_batched = glide_batched_total / DISTANCES_PX.len() as f64;++ let watch_direct = median_ms(|| watch_hold_up(StepPacing::Single));+ let watch_batched = median_ms(|| watch_hold_up(StepPacing::Batched));+ println!(+ "{{\"wait\":\"window_watch\",\"direct_ms\":{watch_direct:.0},\"batched_ms\":{watch_batched:.0}}}"+ );+ assert!(watch_batched < watch_direct);++ let pointer_direct = glide_direct + watch_direct;+ let pointer_batched = glide_batched + watch_batched;+ println!(+ "{{\"step\":\"pointer (click, type_text, set_value, scroll)\",\"direct_ms\":{pointer_direct:.0},\+ \"batched_ms\":{pointer_batched:.0},\"saved_ms\":{:.0}}}",+ pointer_direct - pointer_batched+ );+ println!(+ "{{\"step\":\"key (press_key, hotkey)\",\"direct_ms\":{watch_direct:.0},\+ \"batched_ms\":{watch_batched:.0},\"saved_ms\":{:.0}}}",+ watch_direct - watch_batched+ );+}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rsindex bb9d8e5ac4..8503db326f 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs@@ -5,7 +5,8 @@//! * **AX path** (`element_token`): performs AXAction on the cached//! element. Fires via AX RPC — the target app never needs to be frontmost.//! Extra behaviors vs. the naive dispatch:-//! - AXTextField / AXTextArea: 800 ms post-click delay for WebKit DOM focus settle.+//! - AXTextField / AXTextArea inside web content (AXWebArea): 800 ms post-click delay+//! for DOM focus settle. Native text inputs return without it.//! - AXPopUpButton: appends the list of available options and redirects to set_value.//! - Advertised-action warning if the element didn't list the requested action.//!@@ -1518,6 +1519,52 @@ fn prime_key_window_for_background_press(pid: i32, window_id: u32, action: &str)}}+/// Whether an AX press on a text input must wait for the page to move DOM+/// focus (800 ms). Only text inputs inside web content (an `AXWebArea`+/// ancestor) have a DOM focus to settle; waiting on a native field only+/// delays the result. `ancestor_roles` is read only for a text input pressed+/// with `AXPress`.+fn needs_web_focus_settle(+ ax_action: &str,+ role: &str,+ ancestor_roles: impl FnOnce() -> Vec<String>,+) -> bool {+ ax_action == "AXPress"+ && matches!(role, "AXTextField" | "AXTextArea")+ && ancestor_roles().iter().any(|role| role == "AXWebArea")+}++/// Roles of `element`'s ancestors, nearest first, up to the window. Bounded so+/// a cyclic or very deep tree cannot stall the click.+///+/// # Safety+///+/// `element` must be a valid, live `AXUIElementRef`.+unsafe fn ancestor_roles(element: AXUIElementRef) -> Vec<String> {+ const MAX_DEPTH: usize = 40;+ let mut roles = Vec::new();+ let mut current = copy_element_attr(element, "AXParent");+ while let Some(parent) = current {+ if roles.len() >= MAX_DEPTH {+ CFRelease(parent as CFTypeRef);+ break;+ }+ let role = copy_string_attr(parent, "AXRole").unwrap_or_default();+ let next = copy_element_attr(parent, "AXParent");+ CFRelease(parent as CFTypeRef);+ let is_window = role == "AXWindow";+ roles.push(role);+ if is_window {+ if let Some(next) = next {+ CFRelease(next as CFTypeRef);+ }+ break;+ }+ current = next;+ }+ roles+}+fn perform_ax_click(element: (usize, usize),window: (i32, u32),@@ -1774,9 +1821,10 @@ fn perform_ax_click());}- // WebKit DOM focus settle: 800 ms for text inputs (returned to async caller).+ // DOM focus settle: 800 ms for text inputs in web content (returned to the+ // async caller). A native text field has no DOM focus to wait for.let needs_webkit_delay =- ax_action == "AXPress" && (role == "AXTextField" || role == "AXTextArea");+ needs_web_focus_settle(ax_action, &role, || unsafe { ancestor_roles(element) });// Show focus-rect highlight around the element (matches Swift showFocusRect).// Also move the cursor to the element center so the glide animation plays.@@ -1802,6 +1850,53 @@ fn perform_ax_click(Ok((summary, needs_webkit_delay, suspected_noop, false, false))}+#[cfg(test)]+mod web_focus_settle_tests {+ use super::needs_web_focus_settle;++ fn roles(list: &[&str]) -> impl FnOnce() -> Vec<String> {+ let list: Vec<String> = list.iter().map(|role| (*role).to_owned()).collect();+ move || list+ }++ #[test]+ fn a_text_input_in_web_content_waits_for_dom_focus() {+ assert!(needs_web_focus_settle(+ "AXPress",+ "AXTextField",+ roles(&["AXGroup", "AXWebArea", "AXScrollArea", "AXWindow"])+ ));+ assert!(needs_web_focus_settle(+ "AXPress",+ "AXTextArea",+ roles(&["AXWebArea", "AXWindow"])+ ));+ }++ #[test]+ fn a_native_text_input_does_not_wait() {+ assert!(!needs_web_focus_settle(+ "AXPress",+ "AXTextField",+ roles(&["AXGroup", "AXScrollArea", "AXWindow"])+ ));+ assert!(!needs_web_focus_settle("AXPress", "AXTextArea", roles(&[])));+ }++ #[test]+ fn only_a_press_on_a_text_input_is_considered() {+ for (action, role) in [+ ("AXShowMenu", "AXTextField"),+ ("AXPress", "AXButton"),+ ("AXPress", "AXCheckBox"),+ ] {+ assert!(!needs_web_focus_settle(action, role, || {+ panic!("ancestors are not read for {action} on {role}")+ }));+ }+ }+}+#[cfg(test)]mod selection_fallback_tests {use super::{selection_pixel_target, selection_readback_confirms};diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rsindex a1a03ed93b..29d9a97149 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs@@ -88,6 +88,17 @@ fn def() -> &'static ToolDef {})}+/// Start `capture` on a blocking thread now, so it overlaps the AX walk of a+/// read that wants both. `None` when nothing would overlap: no screenshot is+/// wanted, or no tree is walked (the caller then captures in its own turn).+fn prefetch_capture<T: Send + 'static>(+ should_capture: bool,+ want_tree: bool,+ capture: impl FnOnce() -> T + Send + 'static,+) -> Option<tokio::task::JoinHandle<T>> {+ (should_capture && want_tree).then(|| tokio::task::spawn_blocking(capture))+}+/// Chromium stops painting and stops updating its accessibility tree while its/// window is not visible, which includes sitting on another Space (CUA-1351,/// cdb-v2-40: a click delivered off-Space only showed once the Space was shown).@@ -274,6 +285,12 @@ impl Tool for GetWindowStateTool {let collapse_menus = query.as_deref().is_none_or(|q| q.trim().is_empty())&& !view.full_output&& !observation_only;+ // Start the screenshot grab before the tree walk so the two overlap:+ // ScreenCaptureKit and the accessibility walk use independent system+ // services, and the read otherwise pays for them one after the other.+ let prefetched_capture = prefetch_capture(should_capture, want_tree, move || {+ crate::capture::screenshot_window_bytes(window_id)+ });let (tree_result, prepared_snapshot) = if want_tree {let q = query.clone();// `timeout_ms` bounds the walk itself: it returns the partial tree@@ -350,6 +367,10 @@ impl Tool for GetWindowStateTool {// against, and the raw capture's backing scale.let mut screenshot_frame_error = None;let mut screenshot_resize_scale = None;+ let prefetched_raw = match prefetched_capture {+ Some(handle) => Some(handle.await.unwrap_or_else(|e| Err(anyhow::anyhow!(e)))),+ None => None,+ };let screenshot = if should_capture {let out_file = screenshot_out_file.clone();let res = tokio::task::spawn_blocking(move || -> Result<@@ -368,11 +389,13 @@ impl Tool for GetWindowStateTool {let bounds = crate::windows::window_bounds_by_id(window_id).filter(|b| b.width > 0.0 && b.height > 0.0).ok_or(super::px_frame::PxFrameError::WindowNotFound { window_id })?;- let raw = crate::capture::screenshot_window_bytes(window_id).map_err(|e| {- super::px_frame::PxFrameError::CaptureUnavailable {- window_id,- reason: e.to_string(),- }+ let raw = match prefetched_raw {+ Some(raw) => raw,+ None => crate::capture::screenshot_window_bytes(window_id),+ }+ .map_err(|e| super::px_frame::PxFrameError::CaptureUnavailable {+ window_id,+ reason: e.to_string(),})?;let (orig_w, orig_h) = crate::capture::png_dimensions(&raw).map_err(|e| {super::px_frame::PxFrameError::CaptureUnavailable {@@ -1387,6 +1410,53 @@ mod window_scope_contract_tests {mod tests {use super::*;+ #[test]+ fn a_capture_is_prefetched_only_when_a_tree_walk_can_overlap_it() {+ let rt = tokio::runtime::Builder::new_current_thread()+ .enable_all()+ .build()+ .unwrap();+ rt.block_on(async {+ assert!(prefetch_capture(true, false, || 1).is_none());+ assert!(prefetch_capture(false, true, || 1).is_none());+ assert!(prefetch_capture(false, false, || 1).is_none());+ let handle = prefetch_capture(true, true, || 7).expect("prefetched");+ assert_eq!(handle.await.unwrap(), 7);+ });+ }++ /// Each side waits for the other to have started. They can only both+ /// finish if the capture is running while the walk runs; a capture that+ /// follows the walk would leave both waiting out the timeout.+ #[test]+ fn a_prefetched_capture_overlaps_the_tree_walk() {+ use std::sync::mpsc;+ use std::time::Duration;+ const LIMIT: Duration = Duration::from_secs(10);+ let rt = tokio::runtime::Builder::new_current_thread()+ .enable_all()+ .build()+ .unwrap();+ let (capture_started, capture_seen) = mpsc::channel::<()>();+ let (walk_started, walk_seen) = mpsc::channel::<()>();+ let (capture_ok, walk_ok) = rt.block_on(async {+ let capture = prefetch_capture(true, true, move || {+ capture_started.send(()).unwrap();+ walk_seen.recv_timeout(LIMIT).is_ok()+ })+ .unwrap();+ let walk = tokio::task::spawn_blocking(move || {+ walk_started.send(()).unwrap();+ capture_seen.recv_timeout(LIMIT).is_ok()+ })+ .await+ .unwrap();+ (capture.await.unwrap(), walk)+ });+ assert!(capture_ok, "the capture never saw the walk start");+ assert!(walk_ok, "the walk never saw the capture start");+ }+#[test]fn only_off_space_chromium_windows_get_the_staleness_note() {assert!(off_space_staleness_note(Some(false), true).is_some());diff --git a/libs/cua-driver/rust/crates/platform-macos/src/window_change_detector.rs b/libs/cua-driver/rust/crates/platform-macos/src/window_change_detector.rsindex becd8b6f19..0e5c693edd 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/window_change_detector.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/window_change_detector.rs@@ -423,8 +423,20 @@ impl Snapshot {/// `detect_bounded` over an explicit observer of the current layer-0/// windows and frontmost pid.fn detect_bounded_with(+ self,+ bounds: WindowObservationBounds,+ observe: impl FnMut() -> (Vec<WindowInfo>, Option<i32>) + Send + 'static,+ ) -> Changes {+ self.detect_bounded_reporting(bounds, REPORT_WINDOW, observe)+ }++ /// [`detect_bounded_with`](Self::detect_bounded_with) with an explicit+ /// report window: how long the result waits for a change before the rest+ /// of the deadline runs detached.+ fn detect_bounded_reporting(mut self,bounds: WindowObservationBounds,+ report_window: Duration,observe: impl FnMut() -> (Vec<WindowInfo>, Option<i32>) + Send + 'static,) -> Changes {let earlier = self.earlier.take();@@ -433,7 +445,7 @@ impl Snapshot {return Changes::not_polled().with_earlier(earlier);}let started = Instant::now();- let report = REPORT_WINDOW.min(bounds.timeout);+ let report = report_window.min(bounds.timeout);let mut observe = observe;let changes = match self.poll_until(started + report, bounds.poll, &mut observe) {Ok(changes) => changes,@@ -450,11 +462,17 @@ impl Snapshot {/// tokio runtime. Most action-tool call sites should prefer this/// over the blocking `detect()`.pub async fn detect_async(self) -> Changes {+ // A step of a batch returns after a shorter report window, because+ // the next step follows at once. The pacing is read here: a blocking+ // thread does not inherit the task's batch marker.+ let report_window = cua_driver_core::step_pacing::report_window(REPORT_WINDOW);// Move the Snapshot (and its embedded lease) onto the blocking// thread; the lease's Drop runs there when detect_with returns.- tokio::task::spawn_blocking(move || self.detect())- .await- .unwrap_or_else(|_| Changes::not_polled())+ tokio::task::spawn_blocking(move || {+ self.detect_bounded_reporting(host_observation_bounds(), report_window, observe_host)+ })+ .await+ .unwrap_or_else(|_| Changes::not_polled())}/// One observation against the snapshot: the change it shows, if any.@@ -603,6 +621,19 @@ fn observe_host() -> (Vec<WindowInfo>, Option<i32>) {// ── Tests ────────────────────────────────────────────────────────────────────+/// Time-boxed observation of a window list that never changes, as for an+/// action that opens nothing. Support for the step-latency benchmark.+#[cfg(test)]+pub(crate) fn observe_quiet_window_list(bounds: WindowObservationBounds, report_window: Duration) {+ let snap = WindowChangeDetector::capture_from(HashSet::new(), Some(7), false, None);+ let _ = snap.detect_bounded_reporting(bounds, report_window, || (Vec::new(), Some(7)));+ end_tail();+}++/// The report window of a direct call. Support for the step-latency benchmark.+#[cfg(test)]+pub(crate) const DIRECT_REPORT_WINDOW: Duration = REPORT_WINDOW;+#[cfg(test)]mod tests {use super::*;@@ -893,6 +924,66 @@ mod tests {assert!(lock(&TAIL).is_none());}+ /// A step of a batch returns after the shorter batch report window, and+ /// the rest of the deadline still runs detached with the lease held.+ ///+ /// The upper bound is a wall-clock comparison, so a loaded runner can+ /// overshoot it once; a few attempts keep the test from flaking.+ #[test]+ fn a_batched_quiet_poll_returns_after_the_batch_report_window() {+ use cua_driver_core::step_pacing::BATCH_REPORT_WINDOW;+ let _serial = lock(&TAIL_TEST_LOCK);+ let mut elapsed = Duration::ZERO;+ for _ in 0..5 {+ end_tail();+ let snap = WindowChangeDetector::capture_from(HashSet::from([1]), Some(7), false, None);+ let started = Instant::now();+ let changes = snap.detect_bounded_reporting(+ observation_bounds_from(Some("3000"), Some("10")),+ BATCH_REPORT_WINDOW,+ || (vec![win(1, 7, "App", "Main")], Some(7)),+ );+ elapsed = started.elapsed();+ assert!(changes.polled);+ assert!(!changes.needs_restore());+ assert!(elapsed >= BATCH_REPORT_WINDOW, "returned after {elapsed:?}");+ assert!(lock(&TAIL).is_some(), "the remainder runs detached");+ if elapsed < REPORT_WINDOW {+ break;+ }+ }+ end_tail();+ assert!(+ elapsed < REPORT_WINDOW,+ "a batched step waited as long as a direct call: {elapsed:?}"+ );+ }++ /// The shorter window still reports a window that opens inside it.+ #[test]+ fn a_batched_poll_still_reports_a_window_that_opens_quickly() {+ use cua_driver_core::step_pacing::BATCH_REPORT_WINDOW;+ let _serial = lock(&TAIL_TEST_LOCK);+ end_tail();+ let started = Instant::now();+ let snap = WindowChangeDetector::capture_from(HashSet::from([1]), Some(7), false, None);+ let changes = snap.detect_bounded_reporting(+ observation_bounds_from(Some("3000"), Some("10")),+ BATCH_REPORT_WINDOW,+ move || {+ if started.elapsed() > Duration::from_millis(40) {+ (+ vec![win(1, 7, "App", "Main"), win(2, 7, "App", "Menu")],+ Some(7),+ )+ } else {+ (vec![win(1, 7, "App", "Main")], Some(7))+ }+ },+ );+ assert_eq!(changes.new_windows.len(), 1);+ }+/// A deadline inside the report window behaves as before: no tail.#[test]fn a_short_deadline_leaves_no_detached_tail() {diff --git a/libs/cua-driver/typescript/src/native/cua_driver_contract.ts b/libs/cua-driver/typescript/src/native/cua_driver_contract.tsindex 686e97bfe9..e4bcf4b4a2 100644--- a/libs/cua-driver/typescript/src/native/cua_driver_contract.ts+++ b/libs/cua-driver/typescript/src/native/cua_driver_contract.ts@@ -505,7 +505,9 @@ export type ActionResult = {/*** What to do next, in one short sentence. Present when the effect is* `unverifiable` or `suspected_noop`, so an unconfirmed action points at- * the next step instead of leaving the agent to guess.+ * the next step instead of leaving the agent to guess. Also carries a+ * suggestion to chain steps with `run_steps` after several direct actions+ * in a row.*/hint?: string,/**