Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver): report every layer when list_windows is given a pid (#5162)
injaneitye6bc5f93afe5
diff --git a/.github/release-attribution-config.json b/.github/release-attribution-config.jsonindex c7d71fd599..748d85ea0e 100644--- a/.github/release-attribution-config.json+++ b/.github/release-attribution-config.json@@ -22,6 +22,7 @@"identityOverrides": {"[email protected]": "abonneth","[email protected]": "AnvitDevadiga",+ "[email protected]": "bitomule","[email protected]": "ddupont808","[email protected]": "ddupont808","[email protected]": "ngnichtel",@@ -39,8 +40,8 @@"[email protected]": "madhavajay","[email protected]": "methanoya","[email protected]": "enchanted-koala",- "[email protected]": "r33drichards","[email protected]": "rsyuzyov",+ "[email protected]": "r33drichards","[email protected]": "sarinali","[email protected]": "steipete","[email protected]": "SunflowersLwtech",@@ -49,6 +50,10 @@"[email protected]": "injaneity","[email protected]": "z-x-yang"},- "internalHandles": ["ddupont808", "enchanted-koala", "f-trycua"],+ "internalHandles": [+ "ddupont808",+ "enchanted-koala",+ "f-trycua"+ ],"optOutHandles": []}diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/apps-and-windows.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/apps-and-windows.mdxindex 483b77dca6..9de012cdc3 100644--- a/docs/content/docs/cua-driver/reference/mcp-tools/apps-and-windows.mdx+++ b/docs/content/docs/cua-driver/reference/mcp-tools/apps-and-windows.mdx@@ -15,7 +15,7 @@ import { Tab, Tabs } from 'fumadocs-ui/components/tabs';| Tool | Description | Platforms || --- | --- | --- || [`list_apps`](#list_apps) | List macOS apps, both currently running and installed-but-not-running, with per-app state flags: - running: is a process for this app live? | macOS, Linux, Windows |-| [`list_windows`](#list_windows) | List layer-0 top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden). | macOS, Linux, Windows |+| [`list_windows`](#list_windows) | List top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden): layer-0 only without `pid`, every layer with `pid`. | macOS, Linux, Windows || [`launch_app`](#launch_app) | Launch a macOS app in the background: the target does NOT come to the foreground. | macOS, Linux, Windows || [`kill_app`](#kill_app) | Force-terminate a process by pid (kill -9 equivalent on macOS / Linux; taskkill /F equivalent on Windows). | macOS, Linux, Windows || [`bring_to_front`](#bring_to_front) | Persistently activate an app and leave it in the foreground. | macOS, Linux, Windows |@@ -96,16 +96,16 @@ Effect: read-only, idempotent. Platforms: macOS, Linux, Windows.<Tabs items={['macOS', 'Linux', 'Windows']} groupId="platform" persist><Tab value="macOS">-List layer-0 top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden). Use it to find a `pid` and `window_id` for get_window_state; filter with `app` or `pid`, because an unfiltered call lists every window on the host. AppKit-internal helper windows and the driver's own overlay are omitted.+List top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden): layer-0 only without `pid`, every layer with `pid`. Use it to find a `pid` and `window_id` for get_window_state; filter with `app` or `pid`, because an unfiltered call lists every window on the host. AppKit-internal helper windows and the driver's own overlay are omitted.-Per record: window_id, pid, app_name, title, bounds, z_index, is_on_screen, space_ids, current_space_id, on_current_space. For the frontmost window take the maximum integer z_index; null means stacking is unavailable, so do not infer it from array order.+Per record: window_id, pid, app_name, title, bounds, layer, z_index, is_on_screen, space_ids, current_space_id, on_current_space. For the frontmost window take the maximum integer z_index; null means stacking is unavailable, so do not infer it from array order.**macOS parameters**| Parameter | Type | Default | Description || --- | --- | --- | --- || <span id="list_windows--macos--on_screen_only"></span>`on_screen_only` | `boolean` | | Drop windows not on the current Space. |-| <span id="list_windows--macos--pid"></span>`pid` | `integer` | | Only this pid's windows. |+| <span id="list_windows--macos--pid"></span>`pid` | `integer` | | Only this pid's windows, on every layer. |</Tab><Tab value="Linux">diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/index.mdxindex 4d632b7209..7b259aba4c 100644--- a/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx+++ b/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx@@ -55,7 +55,7 @@ List, launch, quit, front and arrange apps and windows. [Reference](/cua-driver/| Tool | Description | Platforms || --- | --- | --- || <span id="list_apps"></span>[`list_apps`](/cua-driver/reference/mcp-tools/apps-and-windows#list_apps) | List macOS apps, both currently running and installed-but-not-running, with per-app state flags: - running: is a process for this app live? | macOS, Linux, Windows |-| <span id="list_windows"></span>[`list_windows`](/cua-driver/reference/mcp-tools/apps-and-windows#list_windows) | List layer-0 top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden). | macOS, Linux, Windows |+| <span id="list_windows"></span>[`list_windows`](/cua-driver/reference/mcp-tools/apps-and-windows#list_windows) | List top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden): layer-0 only without `pid`, every layer with `pid`. | macOS, Linux, Windows || <span id="launch_app"></span>[`launch_app`](/cua-driver/reference/mcp-tools/apps-and-windows#launch_app) | Launch a macOS app in the background: the target does NOT come to the foreground. | macOS, Linux, Windows || <span id="kill_app"></span>[`kill_app`](/cua-driver/reference/mcp-tools/apps-and-windows#kill_app) | Force-terminate a process by pid (kill -9 equivalent on macOS / Linux; taskkill /F equivalent on Windows). | macOS, Linux, Windows || <span id="bring_to_front"></span>[`bring_to_front`](/cua-driver/reference/mcp-tools/apps-and-windows#bring_to_front) | Persistently activate an app and leave it in the foreground. | macOS, Linux, Windows |diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/launch_app.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/launch_app.rsindex 648a0adb05..dd879312fd 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/launch_app.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/launch_app.rs@@ -514,15 +514,11 @@ fn protected_host_launch_refusal() -> ToolResult {// ── Blocking helpers ──────────────────────────────────────────────────────────-/// Poll for the pid's layer-0 windows, retrying up to 5x100ms to absorb-/// LaunchServices → WindowServer latency (mirrors the Swift reference).fn resolve_windows_for_pid(pid: i32) -> Vec<crate::windows::WindowInfo> {for attempt in 0..5 {- let mut found: Vec<_> = crate::windows::all_windows()- .into_iter()- .filter(|w| w.pid == pid && w.layer == 0)- .filter(|w| w.bounds.width > 1.0 && w.bounds.height > 1.0)- .collect();+ let mut found = crate::windows::pid_windows_with_space_snapshot(pid, false).windows;+ found.retain(|w| w.bounds.width > 1.0 && w.bounds.height > 1.0);+ crate::windows::prefer_layer0(&mut found);if !found.is_empty() {rank_launch_windows(&mut found);return found;diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/list_windows.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/list_windows.rsindex 5d0798576a..4b016c81b5 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/list_windows.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/list_windows.rs@@ -12,15 +12,15 @@ static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();fn def() -> &'static ToolDef {DEF.get_or_init(|| ToolDef {name: "list_windows".into(),- description: "List layer-0 top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden). Use it to find a `pid` and `window_id` for get_window_state; filter with `app` or `pid`, because an unfiltered call lists every window on the host. AppKit-internal helper windows and the driver's own overlay are omitted.\n\+ description: "List top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden): layer-0 only without `pid`, every layer with `pid`. Use it to find a `pid` and `window_id` for get_window_state; filter with `app` or `pid`, because an unfiltered call lists every window on the host. AppKit-internal helper windows and the driver's own overlay are omitted.\n\\n\- Per record: window_id, pid, app_name, title, bounds, z_index, is_on_screen, space_ids, current_space_id, on_current_space. For the frontmost window take the maximum integer z_index; null means stacking is unavailable, so do not infer it from array order.".into(),+ Per record: window_id, pid, app_name, title, bounds, layer, z_index, is_on_screen, space_ids, current_space_id, on_current_space. For the frontmost window take the maximum integer z_index; null means stacking is unavailable, so do not infer it from array order.".into(),input_schema: serde_json::json!({"type": "object","properties": {"pid": {"type": "integer",- "description": "Only this pid's windows."+ "description": "Only this pid's windows, on every layer."},"app": {"type": "string",@@ -51,17 +51,14 @@ impl Tool for ListWindowsTool {let pid_filter: Option<i32> = args.opt_i64("pid").map(|v| v as i32);let on_screen_only = args.bool_or("on_screen_only", false);- let enumeration = if on_screen_only {- crate::windows::visible_windows_with_space_snapshot()- } else {- crate::windows::all_windows_with_space_snapshot()+ let enumeration = match pid_filter {+ Some(pid) => crate::windows::pid_windows_with_space_snapshot(pid, on_screen_only),+ None if on_screen_only => crate::windows::visible_windows_with_space_snapshot(),+ None => crate::windows::all_windows_with_space_snapshot(),};let current_space_id = enumeration.current_space_id;let mut windows = enumeration.windows;- if let Some(pid) = pid_filter {- windows.retain(|w| w.pid == pid);- }// The driver's own windows (the agent cursor overlay) are not targets.let own_pid = std::process::id() as i32;windows.retain(|w| w.pid != own_pid);diff --git a/libs/cua-driver/rust/crates/platform-macos/src/windows.rs b/libs/cua-driver/rust/crates/platform-macos/src/windows.rsindex ba6f3943ea..d598429241 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/windows.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/windows.rs@@ -70,7 +70,12 @@ pub fn all_windows() -> Vec<WindowInfo> {}pub(crate) fn all_windows_with_space_snapshot() -> WindowEnumeration {- enumerate_windows(kCGWindowListExcludeDesktopElements, LayerFilter::ZeroOnly)+ enumerate_windows(+ kCGWindowListExcludeDesktopElements,+ LayerFilter::ZeroOnly,+ None,+ true,+ )}/// Enumerate only on-screen windows.@@ -82,9 +87,31 @@ pub(crate) fn visible_windows_with_space_snapshot() -> WindowEnumeration {enumerate_windows(kCGWindowListOptionOnScreenOnly | kCGWindowListExcludeDesktopElements,LayerFilter::ZeroOnly,+ None,+ true,+ )+}++pub(crate) fn pid_windows_with_space_snapshot(pid: i32, on_screen_only: bool) -> WindowEnumeration {+ let on_screen = if on_screen_only {+ kCGWindowListOptionOnScreenOnly+ } else {+ 0+ };+ enumerate_windows(+ on_screen | kCGWindowListExcludeDesktopElements,+ LayerFilter::AnyLayer,+ Some(pid),+ true,)}+pub(crate) fn prefer_layer0(windows: &mut Vec<WindowInfo>) {+ if windows.iter().any(|w| w.layer == 0) {+ windows.retain(|w| w.layer == 0);+ }+}+/// Enumerate windows on every CGWindow layer, including the accessory layers/// (`layer != 0`) that [`all_windows`] hides.///@@ -95,7 +122,13 @@ pub(crate) fn visible_windows_with_space_snapshot() -> WindowEnumeration {/// `get_window_state` tell "no such window" apart from "exists, but is not a/// layer-0 window" (issue #2237).pub(crate) fn all_windows_any_layer() -> Vec<WindowInfo> {- enumerate_windows(kCGWindowListExcludeDesktopElements, LayerFilter::AnyLayer).windows+ enumerate_windows(+ kCGWindowListExcludeDesktopElements,+ LayerFilter::AnyLayer,+ None,+ false,+ )+ .windows}/// Which CGWindow layers an enumeration admits.@@ -107,7 +140,12 @@ enum LayerFilter {AnyLayer,}-fn enumerate_windows(options: u32, layers: LayerFilter) -> WindowEnumeration {+fn enumerate_windows(+ options: u32,+ layers: LayerFilter,+ pid_filter: Option<i32>,+ spaces: bool,+) -> WindowEnumeration {use core_foundation::{array::CFArray,base::{CFGetTypeID, CFTypeRef, TCFType},@@ -118,7 +156,7 @@ fn enumerate_windows(options: u32, layers: LayerFilter) -> WindowEnumeration {};use std::os::raw::c_void;- let space_query = (layers == LayerFilter::ZeroOnly)+ let space_query = spaces.then(crate::input::skylight::SpaceQuery::new).flatten();let current_space_id = space_query@@ -202,6 +240,9 @@ fn enumerate_windows(options: u32, layers: LayerFilter) -> WindowEnumeration {if layer != 0 && layers == LayerFilter::ZeroOnly {continue;}+ if pid_filter.is_some_and(|wanted| wanted != pid) {+ continue;+ }// Parse bounds dict.let bounds = {@@ -252,13 +293,7 @@ fn enumerate_windows(options: u32, layers: LayerFilter) -> WindowEnumeration {});}- if layers == LayerFilter::ZeroOnly {- let Some(query) = &space_query else {- return WindowEnumeration {- windows: results,- current_space_id,- };- };+ if let Some(query) = &space_query {for window in &mut results {let space_ids = query.window_space_ids(window.window_id);let display_space_id = space_ids@@ -452,12 +487,18 @@ pub fn resolve_window_owner(pid: i32, window_id: u32) -> WindowOwner {/// Select the best window_id for a pid.pub fn resolve_main_window_id(pid: i32) -> anyhow::Result<u32> {- let windows = all_windows();- let pid_windows: Vec<&WindowInfo> = windows.iter().filter(|w| w.pid == pid).collect();+ let mut pid_windows = enumerate_windows(+ kCGWindowListExcludeDesktopElements,+ LayerFilter::AnyLayer,+ Some(pid),+ false,+ )+ .windows;+ prefer_layer0(&mut pid_windows);if pid_windows.is_empty() {anyhow::bail!("pid {pid} has no windows");}- let mut on_screen: Vec<&&WindowInfo> = pid_windows.iter().filter(|w| w.is_on_screen).collect();+ let mut on_screen: Vec<&WindowInfo> = pid_windows.iter().filter(|w| w.is_on_screen).collect();if !on_screen.is_empty() {on_screen.sort_by_key(|window| std::cmp::Reverse(window.z_index));return Ok(on_screen[0].window_id);@@ -608,6 +649,33 @@ mod tests {}}+ #[test]+ fn layer0_windows_win_over_accessory_windows() {+ let accessory = WindowInfo {+ layer: 3,+ ..window(41, 800, "TextEdit")+ };+ let mut windows = vec![accessory, window(42, 800, "TextEdit")];+ prefer_layer0(&mut windows);+ assert_eq!(+ windows.iter().map(|w| w.window_id).collect::<Vec<_>>(),+ [42]+ );+ }++ #[test]+ fn accessory_only_app_keeps_its_accessory_windows() {+ let mut windows = vec![WindowInfo {+ layer: 3,+ ..window(194, 800, "OnboardingApp")+ }];+ prefer_layer0(&mut windows);+ assert_eq!(+ windows.iter().map(|w| w.window_id).collect::<Vec<_>>(),+ [194]+ );+ }+#[test]fn menu_close_polling_keeps_an_unreadable_window_list_unknown() {let timeout = std::time::Duration::from_millis(100);diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.jsonindex c0c6ebe9a5..212bf676ce 100644--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json@@ -18,7 +18,7 @@"read_only": true},{- "description": "List layer-0 top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden). Use it to find a `pid` and `window_id` for get_window_state; filter with `app` or `pid`, because an unfiltered call lists every window on the host. AppKit-internal helper windows and the driver's own overlay are omitted.\n\nPer record: window_id, pid, app_name, title, bounds, z_index, is_on_screen, space_ids, current_space_id, on_current_space. For the frontmost window take the maximum integer z_index; null means stacking is unavailable, so do not infer it from array order.",+ "description": "List top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden): layer-0 only without `pid`, every layer with `pid`. Use it to find a `pid` and `window_id` for get_window_state; filter with `app` or `pid`, because an unfiltered call lists every window on the host. AppKit-internal helper windows and the driver's own overlay are omitted.\n\nPer record: window_id, pid, app_name, title, bounds, layer, z_index, is_on_screen, space_ids, current_space_id, on_current_space. For the frontmost window take the maximum integer z_index; null means stacking is unavailable, so do not infer it from array order.","destructive": false,"idempotent": true,"input_schema": {@@ -33,7 +33,7 @@"type": "boolean"},"pid": {- "description": "Only this pid's windows.",+ "description": "Only this pid's windows, on every layer.","type": "integer"},"session": {