Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): report every layer when list_windows is given a pid (#5162)

injaneitye6bc5f93afe5
diff --git a/.github/release-attribution-config.json b/.github/release-attribution-config.json
index c7d71fd599..748d85ea0e 100644
--- a/.github/release-attribution-config.json
+++ b/.github/release-attribution-config.json
@@ -22,6 +22,7 @@
"identityOverrides": {
"[email protected]": "abonneth",
"[email protected]": "AnvitDevadiga",
+ "[email protected]": "bitomule",
"[email protected]": "ddupont808",
"[email protected]": "ddupont808",
"[email protected]": "ngnichtel",
@@ -39,8 +40,8 @@
"[email protected]": "madhavajay",
"[email protected]": "methanoya",
"[email protected]": "enchanted-koala",
- "[email protected]": "r33drichards",
"[email protected]": "rsyuzyov",
+ "[email protected]": "r33drichards",
"[email protected]": "sarinali",
"[email protected]": "steipete",
"[email protected]": "SunflowersLwtech",
@@ -49,6 +50,10 @@
"[email protected]": "injaneity",
"[email protected]": "z-x-yang"
},
- "internalHandles": ["ddupont808", "enchanted-koala", "f-trycua"],
+ "internalHandles": [
+ "ddupont808",
+ "enchanted-koala",
+ "f-trycua"
+ ],
"optOutHandles": []
}
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/apps-and-windows.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/apps-and-windows.mdx
index 483b77dca6..9de012cdc3 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/apps-and-windows.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/apps-and-windows.mdx
@@ -15,7 +15,7 @@ import { Tab, Tabs } from 'fumadocs-ui/components/tabs';
| Tool | Description | Platforms |
| --- | --- | --- |
| [`list_apps`](#list_apps) | List macOS apps, both currently running and installed-but-not-running, with per-app state flags: - running: is a process for this app live? | macOS, Linux, Windows |
-| [`list_windows`](#list_windows) | List layer-0 top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden). | macOS, Linux, Windows |
+| [`list_windows`](#list_windows) | List top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden): layer-0 only without `pid`, every layer with `pid`. | macOS, Linux, Windows |
| [`launch_app`](#launch_app) | Launch a macOS app in the background: the target does NOT come to the foreground. | macOS, Linux, Windows |
| [`kill_app`](#kill_app) | Force-terminate a process by pid (kill -9 equivalent on macOS / Linux; taskkill /F equivalent on Windows). | macOS, Linux, Windows |
| [`bring_to_front`](#bring_to_front) | Persistently activate an app and leave it in the foreground. | macOS, Linux, Windows |
@@ -96,16 +96,16 @@ Effect: read-only, idempotent. Platforms: macOS, Linux, Windows.
<Tabs items={['macOS', 'Linux', 'Windows']} groupId="platform" persist>
<Tab value="macOS">
-List layer-0 top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden). Use it to find a `pid` and `window_id` for get_window_state; filter with `app` or `pid`, because an unfiltered call lists every window on the host. AppKit-internal helper windows and the driver's own overlay are omitted.
+List top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden): layer-0 only without `pid`, every layer with `pid`. Use it to find a `pid` and `window_id` for get_window_state; filter with `app` or `pid`, because an unfiltered call lists every window on the host. AppKit-internal helper windows and the driver's own overlay are omitted.
-Per record: window_id, pid, app_name, title, bounds, z_index, is_on_screen, space_ids, current_space_id, on_current_space. For the frontmost window take the maximum integer z_index; null means stacking is unavailable, so do not infer it from array order.
+Per record: window_id, pid, app_name, title, bounds, layer, z_index, is_on_screen, space_ids, current_space_id, on_current_space. For the frontmost window take the maximum integer z_index; null means stacking is unavailable, so do not infer it from array order.
**macOS parameters**
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| <span id="list_windows--macos--on_screen_only"></span>`on_screen_only` | `boolean` | | Drop windows not on the current Space. |
-| <span id="list_windows--macos--pid"></span>`pid` | `integer` | | Only this pid's windows. |
+| <span id="list_windows--macos--pid"></span>`pid` | `integer` | | Only this pid's windows, on every layer. |
</Tab>
<Tab value="Linux">
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx b/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx
index 4d632b7209..7b259aba4c 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools/index.mdx
@@ -55,7 +55,7 @@ List, launch, quit, front and arrange apps and windows. [Reference](/cua-driver/
| Tool | Description | Platforms |
| --- | --- | --- |
| <span id="list_apps"></span>[`list_apps`](/cua-driver/reference/mcp-tools/apps-and-windows#list_apps) | List macOS apps, both currently running and installed-but-not-running, with per-app state flags: - running: is a process for this app live? | macOS, Linux, Windows |
-| <span id="list_windows"></span>[`list_windows`](/cua-driver/reference/mcp-tools/apps-and-windows#list_windows) | List layer-0 top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden). | macOS, Linux, Windows |
+| <span id="list_windows"></span>[`list_windows`](/cua-driver/reference/mcp-tools/apps-and-windows#list_windows) | List top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden): layer-0 only without `pid`, every layer with `pid`. | macOS, Linux, Windows |
| <span id="launch_app"></span>[`launch_app`](/cua-driver/reference/mcp-tools/apps-and-windows#launch_app) | Launch a macOS app in the background: the target does NOT come to the foreground. | macOS, Linux, Windows |
| <span id="kill_app"></span>[`kill_app`](/cua-driver/reference/mcp-tools/apps-and-windows#kill_app) | Force-terminate a process by pid (kill -9 equivalent on macOS / Linux; taskkill /F equivalent on Windows). | macOS, Linux, Windows |
| <span id="bring_to_front"></span>[`bring_to_front`](/cua-driver/reference/mcp-tools/apps-and-windows#bring_to_front) | Persistently activate an app and leave it in the foreground. | macOS, Linux, Windows |
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/launch_app.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/launch_app.rs
index 648a0adb05..dd879312fd 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/launch_app.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/launch_app.rs
@@ -514,15 +514,11 @@ fn protected_host_launch_refusal() -> ToolResult {
// ── Blocking helpers ──────────────────────────────────────────────────────────
-/// Poll for the pid's layer-0 windows, retrying up to 5x100ms to absorb
-/// LaunchServices → WindowServer latency (mirrors the Swift reference).
fn resolve_windows_for_pid(pid: i32) -> Vec<crate::windows::WindowInfo> {
for attempt in 0..5 {
- let mut found: Vec<_> = crate::windows::all_windows()
- .into_iter()
- .filter(|w| w.pid == pid && w.layer == 0)
- .filter(|w| w.bounds.width > 1.0 && w.bounds.height > 1.0)
- .collect();
+ let mut found = crate::windows::pid_windows_with_space_snapshot(pid, false).windows;
+ found.retain(|w| w.bounds.width > 1.0 && w.bounds.height > 1.0);
+ crate::windows::prefer_layer0(&mut found);
if !found.is_empty() {
rank_launch_windows(&mut found);
return found;
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/list_windows.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/list_windows.rs
index 5d0798576a..4b016c81b5 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/list_windows.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/list_windows.rs
@@ -12,15 +12,15 @@ static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
fn def() -> &'static ToolDef {
DEF.get_or_init(|| ToolDef {
name: "list_windows".into(),
- description: "List layer-0 top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden). Use it to find a `pid` and `window_id` for get_window_state; filter with `app` or `pid`, because an unfiltered call lists every window on the host. AppKit-internal helper windows and the driver's own overlay are omitted.\n\
+ description: "List top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden): layer-0 only without `pid`, every layer with `pid`. Use it to find a `pid` and `window_id` for get_window_state; filter with `app` or `pid`, because an unfiltered call lists every window on the host. AppKit-internal helper windows and the driver's own overlay are omitted.\n\
\n\
- Per record: window_id, pid, app_name, title, bounds, z_index, is_on_screen, space_ids, current_space_id, on_current_space. For the frontmost window take the maximum integer z_index; null means stacking is unavailable, so do not infer it from array order.".into(),
+ Per record: window_id, pid, app_name, title, bounds, layer, z_index, is_on_screen, space_ids, current_space_id, on_current_space. For the frontmost window take the maximum integer z_index; null means stacking is unavailable, so do not infer it from array order.".into(),
input_schema: serde_json::json!({
"type": "object",
"properties": {
"pid": {
"type": "integer",
- "description": "Only this pid's windows."
+ "description": "Only this pid's windows, on every layer."
},
"app": {
"type": "string",
@@ -51,17 +51,14 @@ impl Tool for ListWindowsTool {
let pid_filter: Option<i32> = args.opt_i64("pid").map(|v| v as i32);
let on_screen_only = args.bool_or("on_screen_only", false);
- let enumeration = if on_screen_only {
- crate::windows::visible_windows_with_space_snapshot()
- } else {
- crate::windows::all_windows_with_space_snapshot()
+ let enumeration = match pid_filter {
+ Some(pid) => crate::windows::pid_windows_with_space_snapshot(pid, on_screen_only),
+ None if on_screen_only => crate::windows::visible_windows_with_space_snapshot(),
+ None => crate::windows::all_windows_with_space_snapshot(),
};
let current_space_id = enumeration.current_space_id;
let mut windows = enumeration.windows;
- if let Some(pid) = pid_filter {
- windows.retain(|w| w.pid == pid);
- }
// The driver's own windows (the agent cursor overlay) are not targets.
let own_pid = std::process::id() as i32;
windows.retain(|w| w.pid != own_pid);
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/windows.rs b/libs/cua-driver/rust/crates/platform-macos/src/windows.rs
index ba6f3943ea..d598429241 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/windows.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/windows.rs
@@ -70,7 +70,12 @@ pub fn all_windows() -> Vec<WindowInfo> {
}
pub(crate) fn all_windows_with_space_snapshot() -> WindowEnumeration {
- enumerate_windows(kCGWindowListExcludeDesktopElements, LayerFilter::ZeroOnly)
+ enumerate_windows(
+ kCGWindowListExcludeDesktopElements,
+ LayerFilter::ZeroOnly,
+ None,
+ true,
+ )
}
/// Enumerate only on-screen windows.
@@ -82,9 +87,31 @@ pub(crate) fn visible_windows_with_space_snapshot() -> WindowEnumeration {
enumerate_windows(
kCGWindowListOptionOnScreenOnly | kCGWindowListExcludeDesktopElements,
LayerFilter::ZeroOnly,
+ None,
+ true,
+ )
+}
+
+pub(crate) fn pid_windows_with_space_snapshot(pid: i32, on_screen_only: bool) -> WindowEnumeration {
+ let on_screen = if on_screen_only {
+ kCGWindowListOptionOnScreenOnly
+ } else {
+ 0
+ };
+ enumerate_windows(
+ on_screen | kCGWindowListExcludeDesktopElements,
+ LayerFilter::AnyLayer,
+ Some(pid),
+ true,
)
}
+pub(crate) fn prefer_layer0(windows: &mut Vec<WindowInfo>) {
+ if windows.iter().any(|w| w.layer == 0) {
+ windows.retain(|w| w.layer == 0);
+ }
+}
+
/// Enumerate windows on every CGWindow layer, including the accessory layers
/// (`layer != 0`) that [`all_windows`] hides.
///
@@ -95,7 +122,13 @@ pub(crate) fn visible_windows_with_space_snapshot() -> WindowEnumeration {
/// `get_window_state` tell "no such window" apart from "exists, but is not a
/// layer-0 window" (issue #2237).
pub(crate) fn all_windows_any_layer() -> Vec<WindowInfo> {
- enumerate_windows(kCGWindowListExcludeDesktopElements, LayerFilter::AnyLayer).windows
+ enumerate_windows(
+ kCGWindowListExcludeDesktopElements,
+ LayerFilter::AnyLayer,
+ None,
+ false,
+ )
+ .windows
}
/// Which CGWindow layers an enumeration admits.
@@ -107,7 +140,12 @@ enum LayerFilter {
AnyLayer,
}
-fn enumerate_windows(options: u32, layers: LayerFilter) -> WindowEnumeration {
+fn enumerate_windows(
+ options: u32,
+ layers: LayerFilter,
+ pid_filter: Option<i32>,
+ spaces: bool,
+) -> WindowEnumeration {
use core_foundation::{
array::CFArray,
base::{CFGetTypeID, CFTypeRef, TCFType},
@@ -118,7 +156,7 @@ fn enumerate_windows(options: u32, layers: LayerFilter) -> WindowEnumeration {
};
use std::os::raw::c_void;
- let space_query = (layers == LayerFilter::ZeroOnly)
+ let space_query = spaces
.then(crate::input::skylight::SpaceQuery::new)
.flatten();
let current_space_id = space_query
@@ -202,6 +240,9 @@ fn enumerate_windows(options: u32, layers: LayerFilter) -> WindowEnumeration {
if layer != 0 && layers == LayerFilter::ZeroOnly {
continue;
}
+ if pid_filter.is_some_and(|wanted| wanted != pid) {
+ continue;
+ }
// Parse bounds dict.
let bounds = {
@@ -252,13 +293,7 @@ fn enumerate_windows(options: u32, layers: LayerFilter) -> WindowEnumeration {
});
}
- if layers == LayerFilter::ZeroOnly {
- let Some(query) = &space_query else {
- return WindowEnumeration {
- windows: results,
- current_space_id,
- };
- };
+ if let Some(query) = &space_query {
for window in &mut results {
let space_ids = query.window_space_ids(window.window_id);
let display_space_id = space_ids
@@ -452,12 +487,18 @@ pub fn resolve_window_owner(pid: i32, window_id: u32) -> WindowOwner {
/// Select the best window_id for a pid.
pub fn resolve_main_window_id(pid: i32) -> anyhow::Result<u32> {
- let windows = all_windows();
- let pid_windows: Vec<&WindowInfo> = windows.iter().filter(|w| w.pid == pid).collect();
+ let mut pid_windows = enumerate_windows(
+ kCGWindowListExcludeDesktopElements,
+ LayerFilter::AnyLayer,
+ Some(pid),
+ false,
+ )
+ .windows;
+ prefer_layer0(&mut pid_windows);
if pid_windows.is_empty() {
anyhow::bail!("pid {pid} has no windows");
}
- let mut on_screen: Vec<&&WindowInfo> = pid_windows.iter().filter(|w| w.is_on_screen).collect();
+ let mut on_screen: Vec<&WindowInfo> = pid_windows.iter().filter(|w| w.is_on_screen).collect();
if !on_screen.is_empty() {
on_screen.sort_by_key(|window| std::cmp::Reverse(window.z_index));
return Ok(on_screen[0].window_id);
@@ -608,6 +649,33 @@ mod tests {
}
}
+ #[test]
+ fn layer0_windows_win_over_accessory_windows() {
+ let accessory = WindowInfo {
+ layer: 3,
+ ..window(41, 800, "TextEdit")
+ };
+ let mut windows = vec![accessory, window(42, 800, "TextEdit")];
+ prefer_layer0(&mut windows);
+ assert_eq!(
+ windows.iter().map(|w| w.window_id).collect::<Vec<_>>(),
+ [42]
+ );
+ }
+
+ #[test]
+ fn accessory_only_app_keeps_its_accessory_windows() {
+ let mut windows = vec![WindowInfo {
+ layer: 3,
+ ..window(194, 800, "OnboardingApp")
+ }];
+ prefer_layer0(&mut windows);
+ assert_eq!(
+ windows.iter().map(|w| w.window_id).collect::<Vec<_>>(),
+ [194]
+ );
+ }
+
#[test]
fn menu_close_polling_keeps_an_unreadable_window_list_unknown() {
let timeout = std::time::Duration::from_millis(100);
diff --git a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
index c0c6ebe9a5..212bf676ce 100644
--- a/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
+++ b/scripts/docs-generators/cli-specs/cua-driver-mcp-macos.json
@@ -18,7 +18,7 @@
"read_only": true
},
{
- "description": "List layer-0 top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden). Use it to find a `pid` and `window_id` for get_window_state; filter with `app` or `pid`, because an unfiltered call lists every window on the host. AppKit-internal helper windows and the driver's own overlay are omitted.\n\nPer record: window_id, pid, app_name, title, bounds, z_index, is_on_screen, space_ids, current_space_id, on_current_space. For the frontmost window take the maximum integer z_index; null means stacking is unavailable, so do not infer it from array order.",
+ "description": "List top-level windows known to WindowServer, including off-screen ones (minimized, other Space, hidden): layer-0 only without `pid`, every layer with `pid`. Use it to find a `pid` and `window_id` for get_window_state; filter with `app` or `pid`, because an unfiltered call lists every window on the host. AppKit-internal helper windows and the driver's own overlay are omitted.\n\nPer record: window_id, pid, app_name, title, bounds, layer, z_index, is_on_screen, space_ids, current_space_id, on_current_space. For the frontmost window take the maximum integer z_index; null means stacking is unavailable, so do not infer it from array order.",
"destructive": false,
"idempotent": true,
"input_schema": {
@@ -33,7 +33,7 @@
"type": "boolean"
},
"pid": {
- "description": "Only this pid's windows.",
+ "description": "Only this pid's windows, on every layer.",
"type": "integer"
},
"session": {