Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver): let SDK hosts bound the post-action window poll (#3946)
Will Bogusz990a2d1f17b2
diff --git a/docs/content/docs/reference/cua-driver/embedding.mdx b/docs/content/docs/reference/cua-driver/embedding.mdxindex 24ec9041c3..a0e78266ae 100644--- a/docs/content/docs/reference/cua-driver/embedding.mdx+++ b/docs/content/docs/reference/cua-driver/embedding.mdx@@ -144,7 +144,8 @@ After each input action, the driver watches the window list to report a menu,dialog, or new window the action opened. For an action that opens nothing, thewait lasts 1000 ms on macOS and 800 ms for Linux X11 foreground delivery. A hostthat observes its target continuously can shorten it by setting these variables-in the daemon's launch environment. Tool arguments cannot change them.+in the daemon's launch environment. `EmbeddedCuaDriverHost` admits both in its+`environment` option. Tool arguments cannot change them.| Variable | Meaning | Accepted values || ------------------------------------- | ----------------------------------------------- | ------------------------------------------------------ |diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdxindex f1ef83fe47..a063ab19d8 100644--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx@@ -491,7 +491,7 @@ Mapping: by='page' → larger step; by='line' → smaller step; amount = number**Arguments:**-- `amount` (integer, optional): Pixel-wheel path: number of wheel notches. Keystroke path: number of keystroke repetitions. Default: 3. range: 1–50+- `amount` (integer, optional): Pixel-wheel path: number of wheel notches. Keystroke path: number of keystroke repetitions. Larger requests are clamped to the maximum. Default: 3. range: 1–50- `by` (string, optional): Scroll granularity. Default: line.- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target — no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it.- `direction` (string, required): Scroll direction.diff --git a/libs/cua-driver/rust/Skills/cua-driver/EMBEDDING.md b/libs/cua-driver/rust/Skills/cua-driver/EMBEDDING.mdindex c79548c271..b2cc243892 100644--- a/libs/cua-driver/rust/Skills/cua-driver/EMBEDDING.md+++ b/libs/cua-driver/rust/Skills/cua-driver/EMBEDDING.md@@ -208,7 +208,9 @@ that watch lasts up to 1000 ms for an action that opens nothing, which makes itthe largest part of a background click's latency. A host that already observesits target continuously can shorten it through two variables set at trustedlaunch, in the environment of the `serve --embedded` child (or of the host-process when you use the same-process runtime):+process when you use the same-process runtime). `EmbeddedCuaDriverHost` starts+that child from an allowlisted environment that admits both variables, so pass+them in its `environment` option or set them in the host process:| Variable | Meaning | Default | Accepted range || ------------------------------------- | ---------------------------------------------------- | ------------------------------- | ----------------------------------------- |diff --git a/libs/cua-driver/rust/crates/cua-driver-sdk/src/embedded.rs b/libs/cua-driver/rust/crates/cua-driver-sdk/src/embedded.rsindex 3445e64f34..f17daccc3b 100644--- a/libs/cua-driver/rust/crates/cua-driver-sdk/src/embedded.rs+++ b/libs/cua-driver/rust/crates/cua-driver-sdk/src/embedded.rs@@ -7,6 +7,7 @@ use cua_driver_contract::{CAPABILITY_VERSION, CONTRACT_VERSION, MCP_PROTOCOL_VERSION, TOOLS_LIST_SCHEMA_VERSION,};use cua_driver_core::daemon::{request_daemon_metadata, DaemonMetadata};+use cua_driver_core::window_observation::{WINDOW_CHANGE_POLL_ENV, WINDOW_CHANGE_TIMEOUT_ENV};use std::collections::BTreeMap;use std::process::Stdio;use std::sync::atomic::{AtomicBool, Ordering};@@ -835,6 +836,8 @@ fn configuration_error<T>(reason: impl Into<String>) -> Result<T, EmbeddedDriverpub(crate) fn allowed_environment_name(name: &str) -> bool {let upper = name.to_ascii_uppercase();upper.starts_with("LC_")+ || upper == WINDOW_CHANGE_TIMEOUT_ENV+ || upper == WINDOW_CHANGE_POLL_ENV|| matches!(upper.as_str(),"PATH"@@ -1221,6 +1224,10 @@ mod tests {assert!(!allowed_environment_name("CUA_DRIVER_PERMISSION_MODE"));assert!(!allowed_environment_name("LD_PRELOAD"));assert!(!allowed_environment_name("NODE_OPTIONS"));+ assert!(allowed_environment_name(+ "CUA_DRIVER_WINDOW_CHANGE_TIMEOUT_MS"+ ));+ assert!(allowed_environment_name("CUA_DRIVER_WINDOW_CHANGE_POLL_MS"));}#[test]diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rsindex a4826ed3ef..aeee63cc5e 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs@@ -46,6 +46,15 @@ impl ScrollTool {static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();+/// Wheel notches or keystroke repetitions accepted in one call. The input+/// schema advertises this range and both delivery paths enforce it.+const AMOUNT_MIN: u64 = 1;+const AMOUNT_MAX: u64 = 50;++fn clamp_amount(requested: u64) -> usize {+ requested.clamp(AMOUNT_MIN, AMOUNT_MAX) as usize+}+fn def() -> &'static ToolDef {DEF.get_or_init(|| ToolDef {name: "scroll".into(),@@ -85,9 +94,9 @@ fn def() -> &'static ToolDef {},"amount": {"type": "integer",- "minimum": 1,- "maximum": 50,- "description": "Pixel-wheel path: number of wheel notches. Keystroke path: number of keystroke repetitions. Default: 3."+ "minimum": AMOUNT_MIN,+ "maximum": AMOUNT_MAX,+ "description": "Pixel-wheel path: number of wheel notches. Keystroke path: number of keystroke repetitions. Larger requests are clamped to the maximum. Default: 3."},"window_id": { "type": "integer" },"element_index": cua_driver_core::tool_schema::element_index_schema(),@@ -126,7 +135,7 @@ impl Tool for ScrollTool {let (x, y) = (input.x, input.y);let direction = input.direction.as_str();let by = input.by.unwrap_or(ScrollBy::Line).as_str();- let amount = input.amount.unwrap_or(3).clamp(1, 50) as usize;+ let amount = clamp_amount(input.amount.unwrap_or(3));let step = if input.by == Some(ScrollBy::Page) {WHEEL_STEP_PAGE_PX} else {@@ -177,7 +186,7 @@ impl Tool for ScrollTool {Err(e) => return e,};let by = args.str_or("by", "line");- let amount = args.u64_or("amount", 3) as usize;+ let amount = clamp_amount(args.u64_or("amount", 3));// Surface 6: element_token / element_index precedence.let element_token_arg = args.opt_str("element_token");let window_id_arg = args.opt_u64("window_id");@@ -733,3 +742,18 @@ unsafe fn collect_ax_buttons(}}}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn amount_is_clamped_to_the_advertised_range() {+ let amount = &def().input_schema["properties"]["amount"];+ assert_eq!(amount["minimum"], serde_json::json!(AMOUNT_MIN));+ assert_eq!(amount["maximum"], serde_json::json!(AMOUNT_MAX));+ assert_eq!(clamp_amount(1100), AMOUNT_MAX as usize);+ assert_eq!(clamp_amount(0), AMOUNT_MIN as usize);+ assert_eq!(clamp_amount(3), 3);+ }+}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/window_change_detector.rs b/libs/cua-driver/rust/crates/platform-macos/src/window_change_detector.rsindex 57b44875ad..3826a61e03 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/window_change_detector.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/window_change_detector.rs@@ -78,6 +78,10 @@ pub struct Snapshot {pub struct Changes {pub new_windows: Vec<WindowEvent>,pub foreground_changed: bool,+ /// Whether the post-action window poll ran. `false` when the host bound+ /// skipped it or the poll task was lost: an empty `new_windows` then means+ /// nothing was watched, not that nothing opened.+ pub polled: bool,}impl Changes {@@ -85,6 +89,14 @@ impl Changes {Self {new_windows: Vec::new(),foreground_changed: false,+ polled: true,+ }+ }++ pub fn not_polled() -> Self {+ Self {+ polled: false,+ ..Self::no_change()}}@@ -285,7 +297,7 @@ impl Snapshot {pub(crate) fn detect_bounded(self, bounds: WindowObservationBounds) -> Changes {if bounds.skips_observation() {drop(self);- return Changes::no_change();+ return Changes::not_polled();}self.detect_with(bounds.timeout, bounds.poll)}@@ -299,7 +311,7 @@ impl Snapshot {// thread; the lease's Drop runs there when detect_with returns.tokio::task::spawn_blocking(move || self.detect()).await- .unwrap_or_else(|_| Changes::no_change())+ .unwrap_or_else(|_| Changes::not_polled())}/// Same as `detect()` but with configurable timing.@@ -333,6 +345,7 @@ impl Snapshot {return Changes {new_windows,foreground_changed,+ polled: true,};}if Instant::now() >= deadline {@@ -358,6 +371,7 @@ mod tests {#[test]fn changes_result_suffix_single_new_window_with_title() {let c = Changes {+ polled: true,new_windows: vec![WindowEvent {window_id: 99,pid: 100,@@ -376,6 +390,7 @@ mod tests {#[test]fn changes_result_suffix_groups_windows_by_app() {let c = Changes {+ polled: true,new_windows: vec![WindowEvent {window_id: 1,@@ -409,6 +424,7 @@ mod tests {#[test]fn changes_result_suffix_foreground_change_only() {let c = Changes {+ polled: true,new_windows: vec![],foreground_changed: true,};@@ -422,6 +438,7 @@ mod tests {#[test]fn changes_result_suffix_empty_title_is_dropped() {let c = Changes {+ polled: true,new_windows: vec![WindowEvent {window_id: 1,pid: 100,@@ -487,10 +504,21 @@ mod tests {let started = Instant::now();let changes = snap.detect_bounded(observation_bounds_from(Some("0"), None));assert!(started.elapsed() < DEFAULT_POLL_INTERVAL);+ assert!(!changes.polled);assert!(!changes.needs_restore());assert_eq!(changes.result_suffix(), "");}+ /// The other half of the pair above: a poll that ran is `polled` whether+ /// or not anything opened before its deadline, so a skipped poll never+ /// reads as a quiet one.+ #[test]+ fn a_poll_that_ran_is_polled_even_when_it_times_out() {+ let snap = WindowChangeDetector::snapshot(None);+ let changes = snap.detect_bounded(observation_bounds_from(Some("30"), Some("10")));+ assert!(changes.polled);+ }+/// Regression: `snapshot(prior_front)` must store the caller's/// captured front pid verbatim (rather than re-reading it inside/// the function and racing with concurrent activations).