Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver): bind macOS state and actions to requested windows (#2645)
Francesco Bonacci729c4da91fcb
diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdxindex 7ba4265573..9a4c4d713b 100644--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx@@ -65,6 +65,8 @@ PREFERRED CONSUMERS read `structuredContent.elements` (one entry per indexed rowAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces: Electron echo-confirms, Catalyst null values, virtualized off-viewport rows with `h:1` frames). You choose the modality at ACTION time, not here: an element ax action (pass `element_index`/`element_token` → the accessibility rung) or an element px action (pass `x`,`y` → the pixel rung, read straight off this screenshot). `capture_mode` is deprecated and ignored. Pass `include_screenshot:false` to skip the grab and get the tree only — the cheap path when you're just re-indexing before an element ax action.+The snapshot is SCOPED to `window_id`: a window_id that no longer exists is refused with `window_id_not_found`, and one owned by another process is refused with `window_owner_pid_mismatch` naming the real `owner_pid` to retry with (macOS hosts a sandboxed app's Open/Save panel out-of-process, so its window belongs to the panel service, not the app). If the window is live under this pid but its accessibility surface can't be resolved, the tree comes back EMPTY with `degraded_reason: ax_window_unresolved` and the screenshot of the requested window — act by pixel there. This tool never returns another surface's elements under your window_id. Before exposing a screenshot, its raw dimensions are validated as a coherent 1x/2x representation of the requested WindowServer bounds. `px_frame_mismatch` or `px_capture_unavailable` omits an unprovable screenshot/pixel frame instead of guessing a transform; the truthful AX payload remains available.+Optional `query` filters the tree_markdown to matching lines plus their ancestor chain (case-insensitive substring). The element_index values are unchanged — filtering only trims the rendered Markdown.Optional `max_elements` / `max_depth` bound the AX walk to mitigate context-window blow-up on Electron / Obsidian / large web apps that produce 10k+ element trees. When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour (≤2 000 elements, depth ≤25).diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/mod.rsindex fb8edc866d..817c8dc7eb 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/ax/mod.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/mod.rs@@ -19,8 +19,10 @@pub mod bindings;pub mod cache;pub mod tree;+pub mod window_scope;pub use cache::ElementCache;pub use tree::{walk_tree, walk_tree_bounded, AXNode, TreeWalkResult, DEFAULT_MAX_DEPTH, DEFAULT_MAX_ELEMENTS,};+pub use window_scope::WindowScope;diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rsindex fe1c053b9f..60a238a4fe 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs@@ -11,6 +11,7 @@//! - Tree is walked depth-first; element_index is assigned in DFS order.use super::bindings::*;+use super::window_scope::{decide_window_scope, TopLevelCandidate, WindowScope};use core_foundation::base::{CFEqual, CFRelease, CFRetain, CFTypeRef};use std::collections::HashSet;use std::sync::{Mutex, OnceLock};@@ -132,6 +133,15 @@ pub struct TreeWalkResult {pub nodes: Vec<AXNode>,/// True when the walk was cut short by the MAX_ELEMENTS cap.pub truncated: bool,+ /// Whether the requested `window_id` actually resolved to an AX surface,+ /// and if not, why. `None` when no `window_id` was requested.+ ///+ /// Issue #2237: without this, an unresolvable id was indistinguishable+ /// from a clean snapshot — callers had no way to tell that the tree they+ /// were handed belonged to a different surface. Any variant other than+ /// [`WindowScope::Matched`] comes with an EMPTY walk, so `nodes` never+ /// describes a window other than the requested one.+ pub window_scope: Option<WindowScope>,}/// Walk the AX tree of `pid`, optionally filtered to a specific window.@@ -181,6 +191,7 @@ pub fn walk_tree_bounded(// Set to true only when walk_element actually stops early due to the cap —// avoids a false-positive when the tree naturally ends on exactly the cap.let mut truncated = false;+ let mut window_scope: Option<WindowScope> = None;unsafe {let app_elem = AXUIElementCreateApplication(pid);@@ -189,6 +200,9 @@ pub fn walk_tree_bounded(tree_markdown: String::new(),nodes,truncated: false,+ // No application AX element at all, so a requested window+ // certainly did not resolve.+ window_scope: window_id.map(|_| WindowScope::AxUnresolved { ax_window_count: 0 }),};}set_messaging_timeout(app_elem);@@ -236,21 +250,44 @@ pub fn walk_tree_bounded(}}- // Filter: keep non-window children (menu bar) + the target window.+ // Scope: keep non-window children (menu bar) + the target window —+ // but ONLY once the target window has actually been identified. When+ // nothing claims the requested id, `decide_window_scope` reports why+ // and walks nothing; it must never fall back to "everything that isn't+ // a window", which is how issue #2237 returned menu bars as panels.let walk_these: Vec<AXUIElementRef> = if let Some(wid) = window_id {- top_level+ let candidates: Vec<TopLevelCandidate> = top_level.iter()- .copied()- .filter(|&child| {+ .map(|&child| {set_messaging_timeout(child);let role = copy_string_attr(child, "AXRole").unwrap_or_default();- if role != "AXWindow" {- return true; // always keep menu bar and other non-window items- }+ let subrole = copy_string_attr(child, "AXSubrole");+ let identifier = copy_string_attr(child, "AXIdentifier");// Match AX window element → CGWindowID via private SPI.- ax_get_window_id(child) == Some(wid)+ // Only windows carry one, so skip the round-trip elsewhere.+ let ax_window_id = if role == "AXWindow" {+ ax_get_window_id(child)+ } else {+ None+ };+ TopLevelCandidate {+ role,+ subrole,+ identifier,+ ax_window_id,+ }})- .collect()+ .collect();+ let decision = decide_window_scope(&candidates, wid, || {+ crate::windows::resolve_window_owner(pid, wid)+ });+ let walk = decision+ .walk+ .iter()+ .map(|&index| top_level[index])+ .collect();+ window_scope = Some(decision.scope);+ walk} else {top_level.to_vec()};@@ -300,6 +337,7 @@ pub fn walk_tree_bounded(tree_markdown,nodes,truncated: truncated_flag,+ window_scope,}}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/window_scope.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/window_scope.rsnew file mode 100644index 0000000000..f23b53b530--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/window_scope.rs@@ -0,0 +1,369 @@+//! Pure decision logic for scoping an AX walk to one requested CGWindowID.+//!+//! Split out of [`super::tree`] so the decision — which is where issue #2237's+//! silent menu-bar-only "success" came from — is testable without a+//! WindowServer.+//!+//! **Structural invariant:** the walk set is non-empty ONLY for+//! [`WindowScope::Matched`]. Every failure variant walks nothing, so no+//! unresolved scope can produce a healthy-looking tree of some *other* surface+//! (the app's menu bar) that a caller would then click by `element_index`.++use crate::windows::WindowOwner;++/// What the requested `window_id` turned out to be.+#[derive(Debug, Clone, PartialEq, Eq)]+pub enum WindowScope {+ /// A top-level AXWindow reported the requested CGWindowID.+ Matched,+ /// WindowServer has no record of the requested CGWindowID — closed,+ /// stale, or fabricated.+ NotFound,+ /// The CGWindowID exists but another process owns it. macOS hosts every+ /// sandboxed app's Open/Save panel out-of-process, so this is a routine+ /// shape, not an exotic race.+ OwnerPidMismatch {+ owner_pid: i32,+ owner_app_name: String,+ },+ /// The CGWindowID exists under the requested pid, but no top-level+ /// AXWindow claims it. The screenshot is still the requested window; the+ /// AX surface is not, so callers report a degraded (empty-tree) snapshot+ /// rather than an error.+ AxUnresolved { ax_window_count: usize },+}++impl WindowScope {+ /// True only when the requested window's own AX surface was found.+ /// Element caches and element-token snapshots may be written only then.+ pub fn is_matched(&self) -> bool {+ matches!(self, WindowScope::Matched)+ }+}++/// A top-level child of the application AX element, reduced to the two facts+/// the scoping decision needs.+#[derive(Debug, Clone)]+pub struct TopLevelCandidate {+ pub role: String,+ /// AXSubrole for top-level windows. Dialog-like windows must not inherit+ /// the application's menu bar into their window-scoped snapshot.+ pub subrole: Option<String>,+ /// Stable AppKit AX identifier, when exposed. Tahoe's Open/Save panels+ /// report `AXStandardWindow` rather than a dialog subrole, but identify+ /// themselves as `open-panel` / `save-panel`.+ pub identifier: Option<String>,+ /// `_AXUIElementGetWindow` result, when the SPI resolved one. Only read+ /// for `AXWindow` roles.+ pub ax_window_id: Option<u32>,+}++impl TopLevelCandidate {+ pub fn new(role: impl Into<String>, ax_window_id: Option<u32>) -> Self {+ Self {+ role: role.into(),+ subrole: None,+ identifier: None,+ ax_window_id,+ }+ }++ pub fn with_subrole(mut self, subrole: impl Into<String>) -> Self {+ self.subrole = Some(subrole.into());+ self+ }++ pub fn with_identifier(mut self, identifier: impl Into<String>) -> Self {+ self.identifier = Some(identifier.into());+ self+ }++ fn is_dialog_like(&self) -> bool {+ self.role == "AXSheet"+ || self+ .subrole+ .as_deref()+ .is_some_and(|s| matches!(s, "AXDialog" | "AXSystemDialog" | "AXSheet"))+ || self+ .identifier+ .as_deref()+ .is_some_and(|id| matches!(id, "open-panel" | "save-panel"))+ }+}++/// The scope conclusion plus which candidates to walk at depth 0.+#[derive(Debug, Clone, PartialEq)]+pub struct ScopeDecision {+ pub scope: WindowScope,+ /// Indices into the candidate slice. Empty for every non-`Matched` scope.+ pub walk: Vec<usize>,+}++/// The scope conclusion reachable from CGWindowList alone, before any AX work.+///+/// `None` means the window exists under the requested pid, so only the AX walk+/// can decide between [`WindowScope::Matched`] and+/// [`WindowScope::AxUnresolved`].+pub fn scope_from_owner(owner: &WindowOwner) -> Option<WindowScope> {+ match owner {+ WindowOwner::SamePid => None,+ WindowOwner::Unknown => Some(WindowScope::NotFound),+ WindowOwner::ForeignPid {+ owner_pid,+ owner_app_name,+ } => Some(WindowScope::OwnerPidMismatch {+ owner_pid: *owner_pid,+ owner_app_name: owner_app_name.clone(),+ }),+ }+}++/// Decide what a window-scoped walk of `candidates` should cover.+///+/// `resolve_owner` is only invoked when no candidate claims `requested` — the+/// CGWindowList enumeration it performs is wasted work on the happy path.+pub fn decide_window_scope<F>(+ candidates: &[TopLevelCandidate],+ requested: u32,+ resolve_owner: F,+) -> ScopeDecision+where+ F: FnOnce() -> WindowOwner,+{+ let matched: Vec<usize> = candidates+ .iter()+ .enumerate()+ .filter(|(_, c)| c.role == "AXWindow" && c.ax_window_id == Some(requested))+ .map(|(i, _)| i)+ .collect();++ if matched.is_empty() {+ // Nothing claims the id. Ask WindowServer *why*: a fabricated/stale id+ // and a live id owned by another process are different refusals, and a+ // live same-pid id whose AX surface never materialised is not a refusal+ // at all.+ let scope = scope_from_owner(&resolve_owner()).unwrap_or(WindowScope::AxUnresolved {+ ax_window_count: candidates.iter().filter(|c| c.role == "AXWindow").count(),+ });+ return ScopeDecision {+ scope,+ walk: Vec::new(),+ };+ }++ // Matched: walk the requested window plus the non-window top-level children.+ //+ // The menu bar is deliberately among them. MACOS.md documents a+ // two-snapshot menu flow whose first step reads `AXMenuBarItem` rows+ // straight out of this window-scoped tree, and `get_window_state` is the+ // only tool that returns a subtree — dropping AXMenuBar here would remove+ // menu navigation with no replacement path. Keeping other non-window+ // children is also what `browser/consent_ui.rs` relies on to reach a+ // top-level `AXSheet` consent prompt.+ let dialog_scope = matched.iter().any(|&i| candidates[i].is_dialog_like());+ let walk = candidates+ .iter()+ .enumerate()+ .filter(|(i, c)| {+ (c.role != "AXWindow" || matched.contains(i))+ && !(dialog_scope && c.role == "AXMenuBar")+ })+ .map(|(i, _)| i)+ .collect();+ ScopeDecision {+ scope: WindowScope::Matched,+ walk,+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ fn panel_service_owner() -> WindowOwner {+ WindowOwner::ForeignPid {+ owner_pid: 900,+ owner_app_name: "Open and Save Panel Service".into(),+ }+ }++ fn never_called() -> WindowOwner {+ panic!("owner lookup must be skipped when a candidate claims the id");+ }++ /// The MACOS.md contract: a resolved normal window keeps the menu bar in+ /// its window-scoped tree, so the documented two-snapshot menu flow+ /// (`AXMenuBarItem` → menu item) still works.+ #[test]+ fn matched_window_walks_window_and_menu_bar() {+ let candidates = [+ TopLevelCandidate::new("AXMenuBar", None),+ TopLevelCandidate::new("AXWindow", Some(11)),+ TopLevelCandidate::new("AXWindow", Some(22)),+ ];+ let d = decide_window_scope(&candidates, 22, never_called);+ assert_eq!(d.scope, WindowScope::Matched);+ assert_eq!(d.walk, vec![0, 2], "menu bar + requested window only");+ }++ #[test]+ fn matched_dialog_excludes_the_application_menu_bar() {+ let candidates = [+ TopLevelCandidate::new("AXMenuBar", None),+ TopLevelCandidate::new("AXWindow", Some(11)).with_subrole("AXStandardWindow"),+ TopLevelCandidate::new("AXWindow", Some(22)).with_subrole("AXDialog"),+ ];+ let d = decide_window_scope(&candidates, 22, never_called);+ assert_eq!(d.scope, WindowScope::Matched);+ assert_eq!(d.walk, vec![2], "dialog snapshot must not carry AXMenuBar");+ }++ #[test]+ fn matched_appkit_file_panel_excludes_the_application_menu_bar() {+ let candidates = [+ TopLevelCandidate::new("AXMenuBar", None),+ TopLevelCandidate::new("AXWindow", Some(11)).with_subrole("AXStandardWindow"),+ TopLevelCandidate::new("AXWindow", Some(22))+ .with_subrole("AXStandardWindow")+ .with_identifier("open-panel"),+ ];+ let d = decide_window_scope(&candidates, 22, never_called);+ assert_eq!(d.scope, WindowScope::Matched);+ assert_eq!(+ d.walk,+ vec![2],+ "AppKit file-panel snapshot must not carry AXMenuBar"+ );+ }++ /// Issue #2237's headline defect: an id nothing claims used to fall through+ /// to `[AXMenuBar, ...]` and return it as a healthy snapshot.+ #[test]+ fn unresolved_id_is_not_a_menu_bar_success() {+ let candidates = [+ TopLevelCandidate::new("AXMenuBar", None),+ TopLevelCandidate::new("AXWindow", Some(11)),+ ];+ let d = decide_window_scope(&candidates, 67340, || WindowOwner::SamePid);+ assert_eq!(d.scope, WindowScope::AxUnresolved { ax_window_count: 1 });+ assert!(d.walk.is_empty(), "must not walk the menu bar instead");+ }++ #[test]+ fn foreign_id_reports_the_owner_pid() {+ let candidates = [+ TopLevelCandidate::new("AXMenuBar", None),+ TopLevelCandidate::new("AXWindow", Some(11)),+ ];+ let d = decide_window_scope(&candidates, 67340, panel_service_owner);+ assert_eq!(+ d.scope,+ WindowScope::OwnerPidMismatch {+ owner_pid: 900,+ owner_app_name: "Open and Save Panel Service".into(),+ }+ );+ assert!(d.walk.is_empty());+ }++ #[test]+ fn stale_id_is_not_found() {+ let candidates = [TopLevelCandidate::new("AXWindow", Some(11))];+ let d = decide_window_scope(&candidates, 67340, || WindowOwner::Unknown);+ assert_eq!(d.scope, WindowScope::NotFound);+ assert!(d.walk.is_empty());+ }++ /// A top-level `AXSheet` (or any non-window child) must not act as a+ /// wildcard that makes an unmatched id look resolved.+ #[test]+ fn top_level_sheet_without_the_id_is_not_a_wildcard() {+ let candidates = [+ TopLevelCandidate::new("AXSheet", None),+ TopLevelCandidate::new("AXMenuBar", None),+ ];+ let d = decide_window_scope(&candidates, 67340, || WindowOwner::SamePid);+ assert_eq!(d.scope, WindowScope::AxUnresolved { ax_window_count: 0 });+ assert!(d.walk.is_empty());+ }++ /// A resolved window still carries its sibling sheets — `consent_ui.rs`+ /// walks Chrome's other window ids expecting the consent `AXSheet` as a+ /// top-level non-window child.+ #[test]+ fn matched_window_keeps_sibling_sheets() {+ let candidates = [+ TopLevelCandidate::new("AXSheet", None),+ TopLevelCandidate::new("AXWindow", Some(11)),+ ];+ let d = decide_window_scope(&candidates, 11, never_called);+ assert_eq!(d.scope, WindowScope::Matched);+ assert_eq!(d.walk, vec![0, 1]);+ }++ #[test]+ fn menu_bar_only_app_has_no_ax_windows() {+ let candidates = [TopLevelCandidate::new("AXMenuBar", None)];+ let d = decide_window_scope(&candidates, 11, || WindowOwner::SamePid);+ assert_eq!(d.scope, WindowScope::AxUnresolved { ax_window_count: 0 });+ assert!(d.walk.is_empty());+ }++ #[test]+ fn empty_candidate_list_walks_nothing() {+ for owner in [+ WindowOwner::SamePid,+ WindowOwner::Unknown,+ panel_service_owner(),+ ] {+ let d = decide_window_scope(&[], 11, || owner.clone());+ assert!(d.walk.is_empty());+ assert!(!d.scope.is_matched());+ }+ }++ /// The invariant that makes the reported bug unregressable: no failure+ /// variant may ever hand back a walk set.+ #[test]+ fn menu_bar_is_never_walked_alone() {+ let candidates = [+ TopLevelCandidate::new("AXMenuBar", None),+ TopLevelCandidate::new("AXSheet", None),+ TopLevelCandidate::new("AXWindow", Some(11)),+ // An AXWindow whose window-id SPI failed — the ViewBridge-remoted+ // panel shape.+ TopLevelCandidate::new("AXWindow", None),+ ];+ for owner in [+ WindowOwner::SamePid,+ WindowOwner::Unknown,+ panel_service_owner(),+ ] {+ let d = decide_window_scope(&candidates, 67340, || owner.clone());+ assert!(+ !d.scope.is_matched(),+ "id 67340 is claimed by nothing; scope was {:?}",+ d.scope+ );+ assert!(+ d.walk.is_empty(),+ "failure scope {:?} must walk nothing, got {:?}",+ d.scope,+ d.walk+ );+ }+ }++ #[test]+ fn scope_from_owner_defers_only_for_same_pid() {+ assert_eq!(scope_from_owner(&WindowOwner::SamePid), None);+ assert_eq!(+ scope_from_owner(&WindowOwner::Unknown),+ Some(WindowScope::NotFound)+ );+ assert!(matches!(+ scope_from_owner(&panel_service_owner()),+ Some(WindowScope::OwnerPidMismatch { owner_pid: 900, .. })+ ));+ }+}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/browser/setup_ui.rs b/libs/cua-driver/rust/crates/platform-macos/src/browser/setup_ui.rsindex 0cdf5dea1e..2f50d420be 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/browser/setup_ui.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/browser/setup_ui.rs@@ -1723,6 +1723,7 @@ mod tests {tree_markdown: String::new(),nodes,truncated: false,+ window_scope: Some(crate::ax::WindowScope::Matched),}}@@ -1850,6 +1851,7 @@ mod tests {tree_markdown: String::new(),nodes: Vec::new(),truncated: true,+ window_scope: Some(crate::ax::WindowScope::Matched),};assert!(exact_pixel_setup_checkbox(0, &truncated, 0, chrome(), false)diff --git a/libs/cua-driver/rust/crates/platform-macos/src/recording_hooks.rs b/libs/cua-driver/rust/crates/platform-macos/src/recording_hooks.rsindex b1a28e7d85..51e18fe674 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/recording_hooks.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/recording_hooks.rs@@ -75,25 +75,12 @@ pub fn element_window_local_xy(window_id: u64, pid: i64, element_index: u32) ->let element = cache.get_element_retained(pid_i32, window_id_u32, element_index as usize)?;let (sx, sy) = unsafe { element_screen_center(element.as_ptr() as AXUIElementRef)? };- let bounds = crate::windows::window_bounds_by_id(window_id_u32)?;- // Probe the captured PNG's width to derive the Retina scale — the- // screenshot is in physical pixels, the window bounds are in points.- let scale = if let Ok(png) = crate::capture::screenshot_window_bytes(window_id_u32) {- if png.len() >= 24 {- let pw = u32::from_be_bytes([png[16], png[17], png[18], png[19]]) as f64;- if bounds.width > 0.0 && pw > bounds.width {- pw / bounds.width- } else {- 1.0- }- } else {- 1.0- }- } else {- 1.0- };-- let wx = (sx - bounds.x) * scale;- let wy = (sy - bounds.y) * scale;+ // Same frame resolution the pixel action rungs use (origin + Retina scale+ // measured off the capture), so a recording marker can never disagree with+ // the click it is annotating. `None` when the window has no live frame —+ // this hook is best-effort and simply records no coordinates then.+ let frame = crate::tools::px_frame::resolve_window_px_frame(window_id_u32).ok()?;+ let wx = (sx - frame.bounds.x) * frame.scale;+ let wy = (sy - frame.bounds.y) * frame.scale;Some((wx, wy))}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rsindex d027093586..b40532d629 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs@@ -616,7 +616,7 @@ impl Tool for ClickTool {))}}- } else if let Some(ratio) = self.state.resize_registry.ratio(pid) {+ } else if let Some(ratio) = self.state.resize_registry.ratio(pid, window_id) {// Coordinates are in the downscaled image space; scale back to native pixels.cx *= ratio;cy *= ratio;@@ -624,53 +624,18 @@ impl Tool for ClickTool {// ── Window-local → screen coordinate translation ──────────────────// `click_at_xy` accepts screen-space coordinates (top-left origin).- // Callers supply window-local screenshot pixels; we add the window's- // screen-origin to produce the final screen position.+ // Callers supply window-local screenshot pixels; `px_frame` adds the+ // window's screen-origin (and divides out the Retina backing scale)+ // to produce the final screen position, or refuses when the window+ // has no live frame — see px_frame's module docs for why there is+ // no screen-absolute fallback.//- // Backing scale: screencapture captures at physical pixels, so on a- // Retina display the screenshot is 2× the logical window size.- // We detect the scale by comparing the live screenshot dimensions to- // the window's logical bounds from WindowServer.- //- // win_local_x/y: window-local logical-pixel coords (= cx/scale, cy/scale)- // needed for CGEventSetWindowLocation in the Chromium recipe.+ // win_local_x/y: window-local logical-pixel coords needed for+ // CGEventSetWindowLocation in the Chromium recipe.let (screen_x, screen_y, win_local_x, win_local_y) = if let Some(wid) = window_id {- let result = tokio::task::spawn_blocking(move || {- let bounds = crate::windows::window_bounds_by_id(wid);- let scale: f64 = if let Some(ref b) = bounds {- // Detect Retina scale from the window screenshot.- // We take a tiny peek at the PNG dimensions to compare- // against the logical bounds.- if let Ok(png) = crate::capture::screenshot_window_bytes(wid) {- if png.len() >= 24 {- let pw =- u32::from_be_bytes([png[16], png[17], png[18], png[19]]) as f64;- let lw = b.width;- if lw > 0.0 && pw > lw {- pw / lw- } else {- 1.0- }- } else {- 1.0- }- } else {- 1.0- }- } else {- 1.0- };- (bounds, scale)- })- .await- .unwrap_or((None, 1.0));- if let (Some(b), scale) = result {- let wx = cx / scale;- let wy = cy / scale;- (b.x + wx, b.y + wy, wx, wy)- } else {- // window_id not found — fall back to treating x,y as screen coords.- (cx, cy, cx, cy)+ match super::px_frame::resolve_or_refuse(wid).await {+ Ok(frame) => frame.to_screen(cx, cy),+ Err(refusal) => return refusal,}} else {// No window_id → treat x,y as screen coordinates (legacy behaviour).diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rsindex f9fdafc0be..6ae4f53313 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs@@ -141,46 +141,19 @@ impl Tool for DoubleClickTool {};// Scale back from downscaled-image space to native pixels when needed.- if let Some(ratio) = self.state.resize_registry.ratio(pid) {+ if let Some(ratio) = self.state.resize_registry.ratio(pid, window_id) {cx *= ratio;cy *= ratio;}// Window-local → screen coordinate translation + win-local logical coords- // for CGEventSetWindowLocation (matches click.rs enhancement).+ // for CGEventSetWindowLocation (shared with click.rs via px_frame, which+ // refuses a window with no live frame instead of silently treating the+ // local point as screen-absolute).let (screen_x, screen_y, win_local_x, win_local_y) = if let Some(wid) = window_id {- let result = tokio::task::spawn_blocking(move || {- let bounds = crate::windows::window_bounds_by_id(wid);- let scale: f64 = if let Some(ref b) = bounds {- if let Ok(png) = crate::capture::screenshot_window_bytes(wid) {- if png.len() >= 24 {- let pw =- u32::from_be_bytes([png[16], png[17], png[18], png[19]]) as f64;- let lw = b.width;- if lw > 0.0 && pw > lw {- pw / lw- } else {- 1.0- }- } else {- 1.0- }- } else {- 1.0- }- } else {- 1.0- };- (bounds, scale)- })- .await- .unwrap_or((None, 1.0));- if let (Some(b), scale) = result {- let wx = cx / scale;- let wy = cy / scale;- (b.x + wx, b.y + wy, wx, wy)- } else {- (cx, cy, cx, cy)+ match super::px_frame::resolve_or_refuse(wid).await {+ Ok(frame) => frame.to_screen(cx, cy),+ Err(refusal) => return refusal,}} else {(cx, cy, cx, cy)diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rsindex ba1fa203bd..950a4f876a 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs@@ -234,61 +234,26 @@ impl Tool for DragTool {))}}- } else if let Some(ratio) = self.state.resize_registry.ratio(pid) {+ } else if let Some(ratio) = self.state.resize_registry.ratio(pid, window_id) {from_x *= ratio;from_y *= ratio;to_x *= ratio;to_y *= ratio;}- // Translate window-local screenshot pixels → screen coordinates.- // Also compute window-local logical coords for CGEventSetWindowLocation.+ // Translate window-local screenshot pixels → screen coordinates, and+ // window-local logical coords for CGEventSetWindowLocation. Both ends of+ // the drag resolve against ONE frame, so a window that closed mid-call+ // refuses rather than dragging across the desktop behind it.let (from_sx, from_sy, from_lx, from_ly, to_sx, to_sy, to_lx, to_ly) =if let Some(wid) = window_id {- let result = tokio::task::spawn_blocking(move || {- let bounds = crate::windows::window_bounds_by_id(wid);- let scale: f64 = if let Some(ref b) = bounds {- if let Ok(png) = crate::capture::screenshot_window_bytes(wid) {- if png.len() >= 24 {- let pw =- u32::from_be_bytes([png[16], png[17], png[18], png[19]]) as f64;- let lw = b.width;- if lw > 0.0 && pw > lw {- pw / lw- } else {- 1.0- }- } else {- 1.0- }- } else {- 1.0- }- } else {- 1.0- };- (bounds, scale)- })- .await- .unwrap_or((None, 1.0));-- if let (Some(b), scale) = result {- let flx = from_x / scale;- let fly = from_y / scale;- let tlx = to_x / scale;- let tly = to_y / scale;- (- b.x + flx,- b.y + fly,- flx,- fly,- b.x + tlx,- b.y + tly,- tlx,- tly,- )- } else {- (from_x, from_y, from_x, from_y, to_x, to_y, to_x, to_y)+ match super::px_frame::resolve_or_refuse(wid).await {+ Ok(frame) => {+ let (fsx, fsy, flx, fly) = frame.to_screen(from_x, from_y);+ let (tsx, tsy, tlx, tly) = frame.to_screen(to_x, to_y);+ (fsx, fsy, flx, fly, tsx, tsy, tlx, tly)+ }+ Err(refusal) => return refusal,}} else {(from_x, from_y, from_x, from_y, to_x, to_y, to_x, to_y)diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rsindex 025f66eb8b..93f7e63bed 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs@@ -47,6 +47,19 @@ fn def() -> &'static ToolDef {and ignored. Pass `include_screenshot:false` to skip the grab and get \the tree only — the cheap path when you're just re-indexing before an \element ax action.\n\n\+ The snapshot is SCOPED to `window_id`: a window_id that no longer exists is \+ refused with `window_id_not_found`, and one owned by another process is \+ refused with `window_owner_pid_mismatch` naming the real `owner_pid` to retry \+ with (macOS hosts a sandboxed app's Open/Save panel out-of-process, so its \+ window belongs to the panel service, not the app). If the window is live under \+ this pid but its accessibility surface can't be resolved, the tree comes back \+ EMPTY with `degraded_reason: ax_window_unresolved` and the screenshot of the \+ requested window — act by pixel there. This tool never returns another \+ surface's elements under your window_id. Before exposing a screenshot, \+ its raw dimensions are validated as a coherent 1x/2x representation of \+ the requested WindowServer bounds. `px_frame_mismatch` or \+ `px_capture_unavailable` omits an unprovable screenshot/pixel frame \+ instead of guessing a transform; the truthful AX payload remains available.\n\n\Optional `query` filters the tree_markdown to matching lines plus their ancestor \chain (case-insensitive substring). The element_index values are unchanged — \filtering only trims the rendered Markdown.\n\n\@@ -108,6 +121,35 @@ impl Tool for GetWindowStateTool {Ok(v) => v,Err(e) => return e,};++ // Issue #2237: pre-flight the requested window against WindowServer+ // BEFORE the (up to 20 s) AX walk. An id that no window carries, or+ // that another process owns, used to fall through the scoped filter and+ // return the app's MENU BAR as a healthy snapshot of the requested+ // window — with a screenshot of the requested window beside it. macOS+ // hosts every sandboxed app's Open/Save panel in+ // `com.apple.appkit.xpc.openAndSavePanelService`, so the owner-mismatch+ // shape is routine, and the caller must be told the real owner pid.+ {+ let owner = match tokio::task::spawn_blocking(move || {+ crate::windows::resolve_window_owner(pid, window_id)+ })+ .await+ {+ Ok(owner) => owner,+ Err(e) => {+ return ToolResult::error(format!(+ "window ownership lookup for window_id {window_id} failed: {e}"+ ))+ }+ };+ if let Some(scope) = crate::ax::window_scope::scope_from_owner(&owner) {+ if let Some(refusal) = window_scope_refusal(pid, window_id, &scope) {+ return refusal;+ }+ }+ }+let query = args.opt_str("query");let screenshot_out_file = args.opt_str("screenshot_out_file").map(|s| {// Expand ~ prefix.@@ -188,9 +230,31 @@ impl Tool for GetWindowStateTool {}};- // Update element cache.+ // The window can close, or its CGWindow can be re-parented onto another+ // process, between the pre-flight and the walk. Re-apply the same+ // refusals against what the walk actually observed.+ let window_scope = tree_result.as_ref().and_then(|r| r.window_scope.clone());+ if let Some(ref scope) = window_scope {+ if let Some(refusal) = window_scope_refusal(pid, window_id, scope) {+ return refusal;+ }+ }+ // `window_scope` is None only when no window_id was requested, which+ // this tool never does — so treat that as resolved.+ let scope_matched = window_scope.as_ref().is_none_or(|s| s.is_matched());++ // Update element cache — ONLY for a resolved window scope. Caching an+ // unresolved scope's nodes under (pid, window_id) is what turned a+ // wrong-surface snapshot into a wrong-surface *action*: a follow-up+ // click(element_index=N) picked whatever the walk happened to return.+ // For an unresolved scope, replace any prior entry with an empty+ // snapshot so a stale index map cannot be clicked through either.if let Some(ref r) = tree_result {- self.state.element_cache.update(pid, window_id, &r.nodes);+ if scope_matched {+ self.state.element_cache.update(pid, window_id, &r.nodes);+ } else {+ self.state.element_cache.update(pid, window_id, &[]);+ }}// Capture the screenshot and deliver it alongside the tree — the@@ -199,39 +263,109 @@ impl Tool for GetWindowStateTool {// screenshot_out_file). With `screenshot_out_file` set, write to disk and// surface the path instead of embedding base64; otherwise embed base64.let max_dim = effective_max_dim;- // Returns (b64_or_path, final_w, final_h, Option<original_w>, is_file_path)+ // Returns the encoded/file capture, delivered dimensions, optional+ // downscale source width, the WindowServer bounds it was validated+ // against, and the raw capture's backing scale.+ let mut screenshot_frame_error = None;let screenshot = if should_capture {let out_file = screenshot_out_file.clone();- let res = tokio::task::spawn_blocking(move || -> anyhow::Result<(Option<String>, Option<String>, u32, u32, Option<u32>)> {+ let res = tokio::task::spawn_blocking(move || -> Result<+ (+ Option<String>,+ Option<String>,+ u32,+ u32,+ Option<u32>,+ crate::windows::WindowBounds,+ f64,+ ),+ super::px_frame::PxFrameError,+ > {use base64::{engine::general_purpose::STANDARD as BASE64, Engine};- let raw = crate::capture::screenshot_window_bytes(window_id)?;- let (orig_w, _orig_h) = crate::capture::png_dimensions(&raw)?;- let png = crate::capture::resize_png_if_needed(&raw, max_dim)?;- let (w, h) = crate::capture::png_dimensions(&png)?;+ let bounds = crate::windows::window_bounds_by_id(window_id)+ .filter(|b| b.width > 0.0 && b.height > 0.0)+ .ok_or(super::px_frame::PxFrameError::WindowNotFound { window_id })?;+ let raw = crate::capture::screenshot_window_bytes(window_id).map_err(|e| {+ super::px_frame::PxFrameError::CaptureUnavailable {+ window_id,+ reason: e.to_string(),+ }+ })?;+ let (orig_w, orig_h) = crate::capture::png_dimensions(&raw).map_err(|e| {+ super::px_frame::PxFrameError::CaptureUnavailable {+ window_id,+ reason: e.to_string(),+ }+ })?;+ let scale =+ super::px_frame::validate_capture_frame(window_id, &bounds, orig_w, orig_h)?;+ let png = crate::capture::resize_png_if_needed(&raw, max_dim).map_err(|e| {+ super::px_frame::PxFrameError::CaptureUnavailable {+ window_id,+ reason: e.to_string(),+ }+ })?;+ let (w, h) = crate::capture::png_dimensions(&png).map_err(|e| {+ super::px_frame::PxFrameError::CaptureUnavailable {+ window_id,+ reason: e.to_string(),+ }+ })?;let original_w = if w < orig_w { Some(orig_w) } else { None };if let Some(ref path) = out_file {- std::fs::write(path, &png)?;- Ok((None, Some(path.clone()), w, h, original_w))+ std::fs::write(path, &png).map_err(|e| {+ super::px_frame::PxFrameError::CaptureUnavailable {+ window_id,+ reason: e.to_string(),+ }+ })?;+ Ok((+ None,+ Some(path.clone()),+ w,+ h,+ original_w,+ bounds,+ scale,+ ))} else {- Ok((Some(BASE64.encode(&png)), None, w, h, original_w))+ Ok((+ Some(BASE64.encode(&png)),+ None,+ w,+ h,+ original_w,+ bounds,+ scale,+ ))}}).await;match res {- Ok(Ok((b64, file_path, w, h, orig_w))) => {- // Record resize ratio so ClickTool can scale coordinates back up.+ Ok(Ok((b64, file_path, w, h, orig_w, bounds, scale))) => {+ // Record resize ratio so ClickTool can scale coordinates back+ // up. Keyed per window: two windows of one pid can carry+ // different ratios (only the large one downscales), and a+ // pid-only key leaked one window's ratio into the other's+ // pixel clicks.if let Some(ow) = orig_w {if w > 0 {- self.state- .resize_registry- .set_ratio(pid, ow as f64 / w as f64);+ self.state.resize_registry.set_ratio(+ pid,+ window_id,+ ow as f64 / w as f64,+ );}} else {- self.state.resize_registry.clear_ratio(pid);+ self.state.resize_registry.clear_ratio(pid, window_id);}- Some((b64, file_path, w, h))+ Some((b64, file_path, w, h, bounds, scale))}Ok(Err(e)) => {- tracing::warn!("Screenshot failed for window {window_id}: {e}");+ tracing::warn!(+ "Screenshot frame could not be verified for window {window_id}: {e:?}"+ );+ self.state.resize_registry.clear_ratio(pid, window_id);+ screenshot_frame_error = Some(e);None}Err(e) => {@@ -244,13 +378,18 @@ impl Tool for GetWindowStateTool {};// Capture screenshot dimensions before consuming.- let screenshot_dims = screenshot.as_ref().map(|(_, _, w, h)| (*w, *h));- let screenshot_file_path = screenshot.as_ref().and_then(|(_, fp, _, _)| fp.clone());+ let screenshot_dims = screenshot.as_ref().map(|(_, _, w, h, _, _)| (*w, *h));+ let screenshot_file_path = screenshot+ .as_ref()+ .and_then(|(_, fp, _, _, _, _)| fp.clone());+ let screenshot_frame = screenshot+ .as_ref()+ .map(|(_, _, _, _, bounds, scale)| (bounds.clone(), *scale));// Build response.let mut content: Vec<Content> = Vec::new();- if let Some((b64_opt, _file_path, w, h)) = screenshot {+ if let Some((b64_opt, _file_path, w, h, _bounds, _scale)) = screenshot {if let Some(b64) = b64_opt {content.push(Content::image_png(b64));}@@ -293,15 +432,23 @@ impl Tool for GetWindowStateTool {// generated even when the walk returned no elements so consumers// calling `get_window_state` and then immediately re-snapshotting// get a clean LRU step every time.+ //+ // Skipped entirely for an unresolved window scope: an element_token is+ // a promise that index N addresses a row of THIS window, and there is+ // no such row to promise (issue #2237).let elem_count_for_snapshot = tree_result.as_ref().map(|r| r.nodes.iter().filter(|n| n.element_index.is_some()).count()).unwrap_or(0);- let snapshot_id = cua_driver_core::element_token::global().register_snapshot(- pid,- window_id,- elem_count_for_snapshot,- );+ let snapshot_id = if scope_matched {+ Some(cua_driver_core::element_token::global().register_snapshot(+ pid,+ window_id,+ elem_count_for_snapshot,+ ))+ } else {+ None+ };// Build the structured `elements` array — one entry per actionable// node, matching the order (and indices) of the markdown rendering.@@ -309,10 +456,10 @@ impl Tool for GetWindowStateTool {// alongside for back-compat with existing text-parsing callers// (Hermes' regex parser, Codex, Claude Code) and is signalled as// preferred-for-back-compat-only via the `_note` field below.- let elements_json: Vec<serde_json::Value> = tree_result- .as_ref()- .map(|r| build_elements_array_with_token(&r.nodes, snapshot_id))- .unwrap_or_default();+ let elements_json: Vec<serde_json::Value> = match (snapshot_id, tree_result.as_ref()) {+ (Some(sid), Some(r)) => build_elements_array_with_token(&r.nodes, sid),+ _ => Vec::new(),+ };let mut structured = serde_json::json!({"window_id": window_id,@@ -320,45 +467,64 @@ impl Tool for GetWindowStateTool {"element_count": element_count,"tree_markdown": tree_md,"elements": elements_json,- // Surface 6: an opaque snapshot identifier consumers can log- // alongside the per-element tokens for debug correlation.- // Same value embedded in every `element_token` emitted in- // `elements[]` above. Additive — old consumers ignore it.- "snapshot_id": cua_driver_core::element_token::token_for(snapshot_id, 0)- .trim_end_matches(":0")- .to_string(),"_note": "Prefer `elements` — `tree_markdown` will continue to work \but new fields will only be added to the structured side. \Issue #22865: use `max_elements` / `max_depth` to bound the \AX walk on apps with very large trees."});- // Best-effort-background ladder, rung (2): an AX walk that ran but found- // zero actionable elements is NOT a clean snapshot — the window may be a- // non-AX surface (canvas/WebGL) or its tree wasn't ready (Chromium needs- // an enable+settle). Mark it degraded so callers don't read `elements: []`- // as "this window has no controls". Only applies when a walk was actually- // attempted (tree_result is None in capture_mode=vision, where empty is- // expected, not degraded).- if tree_result.is_some() && element_count == 0 {- structured["degraded"] = serde_json::json!(true);- structured["degraded_reason"] = serde_json::json!(- "ax_tree_empty: the AX walk returned no actionable elements. The \- window may be a non-AX surface (canvas/WebGL/custom-drawn) or its \- accessibility tree was not ready (Chromium/Electron require an \- AX-enable + settle). Do not treat element data as authoritative — \- re-snapshot if the app just launched, otherwise switch to the \- visual path."- );- // Point the agent at the next rung: an empty AX tree means- // element_index has nothing to bind to, so the deliberate move is an- // element px action — read the screenshot already in this response and- // click by pixel (x,y). macOS can pixel-target in the background, so- // the recommendation is `px`, not `foreground`.- structured["escalation"] = serde_json::json!({- "recommended": "px",- "reason": "non-AX surface — act by pixel (x,y) off the screenshot \- in this response (an element px action)."- });+ // Surface 6: an opaque snapshot identifier consumers can log+ // alongside the per-element tokens for debug correlation. Same value+ // embedded in every `element_token` emitted in `elements[]` above.+ // Additive — old consumers ignore it. Absent when no snapshot was+ // registered (unresolved window scope).+ if let Some(sid) = snapshot_id {+ structured["snapshot_id"] =+ serde_json::json!(cua_driver_core::element_token::token_for(sid, 0)+ .trim_end_matches(":0")+ .to_string());+ }+ // Best-effort-background ladder, rung (2). Both rungs point the agent at+ // the same next move: an empty AX tree means element_index has nothing+ // to bind to, so the deliberate action is an element px action — read+ // the screenshot already in this response and click by pixel (x,y).+ // macOS can pixel-target in the background, so the recommendation is+ // `px`, not `foreground`.+ match degradation_for(tree_result.is_some(), element_count, window_scope.as_ref()) {+ Degradation::None => {}+ Degradation::AxTreeEmpty => {+ structured["degraded"] = serde_json::json!(true);+ structured["degraded_reason"] = serde_json::json!(+ "ax_tree_empty: the AX walk returned no actionable elements. The \+ window may be a non-AX surface (canvas/WebGL/custom-drawn) or its \+ accessibility tree was not ready (Chromium/Electron require an \+ AX-enable + settle). Do not treat element data as authoritative — \+ re-snapshot if the app just launched, otherwise switch to the \+ visual path."+ );+ structured["escalation"] = serde_json::json!({+ "recommended": "px",+ "reason": "non-AX surface — act by pixel (x,y) off the screenshot \+ in this response (an element px action)."+ });+ }+ Degradation::AxWindowUnresolved { ax_window_count } => {+ structured["degraded"] = serde_json::json!(true);+ structured["degraded_reason"] = serde_json::json!(format!(+ "ax_window_unresolved: window_id {window_id} exists and is owned by \+ pid {pid}, but none of the {ax_window_count} AXWindow element(s) \+ under that pid reports this CGWindowID. The tree is returned EMPTY \+ on purpose: the accessibility elements reachable under this pid \+ belong to other surfaces (the menu bar, other windows), not to the \+ requested window, so presenting them would misground the next \+ action."+ ));+ structured["escalation"] = serde_json::json!({+ "recommended": "px",+ "reason": "act by pixel (x,y) off the screenshot in this response — \+ the frame IS the requested window even though its AX \+ surface is unresolved."+ });+ }}if let Some((sw, sh)) = screenshot_dims {structured["screenshot_width"] = serde_json::json!(sw);@@ -371,6 +537,20 @@ impl Tool for GetWindowStateTool {// the structured side. Additive: keeps every existing field.structured["screenshot_mime_type"] = serde_json::json!("image/png");}+ if let Some((bounds, scale)) = screenshot_frame {+ structured["window_bounds"] = serde_json::json!({+ "x": bounds.x,+ "y": bounds.y,+ "width": bounds.width,+ "height": bounds.height+ });+ structured["screenshot_scale"] = serde_json::json!(scale);+ structured["screenshot_frame_valid"] = serde_json::json!(true);+ }+ if let Some(error) = screenshot_frame_error {+ structured["screenshot_frame_valid"] = serde_json::json!(false);+ structured["screenshot_error"] = super::px_frame::error_structured(&error);+ }if let Some(ref fp) = screenshot_file_path {structured["screenshot_file_path"] = serde_json::json!(fp);}@@ -382,6 +562,105 @@ impl Tool for GetWindowStateTool {}}+/// Turn an unresolvable window scope into a structured refusal, or `None` when+/// the scope is one the caller can still be served (issue #2237).+///+/// Refusing is the point: the pre-fix behaviour returned the app's menu bar+/// under the requested `window_id`, which reads as a healthy snapshot and gets+/// clicked by `element_index`. Both refusals name the exact retry, matching the+/// remedy-in-the-refusal shape the rest of the driver uses.+///+/// The owner pid is REPORTED, not followed: `element_cache`, the element-token+/// registry and `ResizeRegistry` are all keyed on the caller-supplied pid, so+/// walking under `owner_pid` while echoing the requested pid would hand back+/// indices the caller replays against the wrong key. One retry with the named+/// pid is correct and cheap.+fn window_scope_refusal(+ pid: i32,+ window_id: u32,+ scope: &crate::ax::WindowScope,+) -> Option<ToolResult> {+ use crate::ax::WindowScope;+ match scope {+ // Resolved, or resolvable-as-degraded — the caller gets a response.+ WindowScope::Matched | WindowScope::AxUnresolved { .. } => None,+ WindowScope::NotFound => Some(+ ToolResult::error(format!(+ "window_id {window_id} is not a live window (closed, or the id is stale). \+ Refusing to return an accessibility tree, because the elements reachable \+ under pid {pid} belong to other surfaces — not to the window you asked \+ for. Call list_windows for current window_ids."+ ))+ .with_structured(serde_json::json!({+ "code": "window_id_not_found",+ "pid": pid,+ "window_id": window_id,+ "suggestion": "call list_windows for current window_ids; the window may have closed"+ })),+ ),+ WindowScope::OwnerPidMismatch {+ owner_pid,+ owner_app_name,+ } => Some(+ ToolResult::error(format!(+ "window_id {window_id} is owned by pid {owner_pid} (\"{owner_app_name}\"), \+ not pid {pid}. macOS hosts a sandboxed app's Open/Save panel \+ out-of-process, so the panel's CGWindowID belongs to the panel service \+ rather than the app that opened it. Refusing to return pid {pid}'s \+ accessibility tree for it. Re-call get_window_state with pid={owner_pid} \+ and the same window_id."+ ))+ .with_structured(serde_json::json!({+ "code": "window_owner_pid_mismatch",+ "pid": pid,+ "window_id": window_id,+ "owner_pid": owner_pid,+ "owner_app_name": owner_app_name,+ "suggestion": format!(+ "window_id {window_id} is owned by pid {owner_pid}, not pid {pid} \+ (macOS hosts sandboxed Open/Save panels out-of-process). Re-call \+ get_window_state with pid={owner_pid} and the same window_id."+ )+ })),+ ),+ }+}++/// Which degradation rung a snapshot lands on.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+enum Degradation {+ /// Clean snapshot — no `degraded` field is emitted.+ None,+ /// A walk ran and produced no actionable elements.+ AxTreeEmpty,+ /// The requested window is live and owned by this pid, but no AXWindow+ /// claims its CGWindowID, so the walk deliberately covered nothing.+ AxWindowUnresolved { ax_window_count: usize },+}++/// Decide the degradation rung. Pure: `walk_attempted` is false in the+/// screenshot-only path (an empty tree is expected there, not degraded), and+/// the unresolved-scope rung outranks the generic empty-tree rung because it+/// explains *why* the tree is empty.+fn degradation_for(+ walk_attempted: bool,+ element_count: usize,+ scope: Option<&crate::ax::WindowScope>,+) -> Degradation {+ if !walk_attempted {+ return Degradation::None;+ }+ if let Some(crate::ax::WindowScope::AxUnresolved { ax_window_count }) = scope {+ return Degradation::AxWindowUnresolved {+ ax_window_count: *ax_window_count,+ };+ }+ if element_count == 0 {+ return Degradation::AxTreeEmpty;+ }+ Degradation::None+}+/// Render the actionable nodes from the AX walk into the/// `structuredContent.elements` array shape described on the tool: one entry/// per node with an `element_index`, carrying role, label (built from@@ -496,6 +775,123 @@ pub(crate) fn build_elements_array(nodes: &[crate::ax::tree::AXNode]) -> Vec<serout}+#[cfg(test)]+mod window_scope_contract_tests {+ use super::*;+ use crate::ax::WindowScope;++ fn panel_mismatch() -> WindowScope {+ WindowScope::OwnerPidMismatch {+ owner_pid: 900,+ owner_app_name: "Open and Save Panel Service".into(),+ }+ }++ fn structured(result: ToolResult) -> serde_json::Value {+ assert_eq!(result.is_error, Some(true), "must be an error result");+ result+ .structured_content+ .expect("refusals carry structured content")+ }++ #[test]+ fn stale_window_id_is_a_structured_not_found() {+ let s = structured(+ window_scope_refusal(800, 67340, &WindowScope::NotFound).expect("must refuse"),+ );+ assert_eq!(s["code"], "window_id_not_found");+ assert_eq!(s["pid"], 800);+ assert_eq!(s["window_id"], 67340);+ assert!(s["suggestion"].as_str().unwrap().contains("list_windows"));+ }++ /// Issue #2237's reported case: TextEdit's Open panel window belongs to the+ /// out-of-process panel service. The refusal must name the real owner pid+ /// so the caller can retry, and must NOT redirect on its own (the element+ /// caches are keyed on the caller-supplied pid).+ #[test]+ fn owner_pid_mismatch_names_the_owner_and_the_retry() {+ let refusal = window_scope_refusal(800, 67340, &panel_mismatch()).expect("must refuse");+ let text = format!("{:?}", refusal.content);+ let s = structured(refusal);+ assert_eq!(s["code"], "window_owner_pid_mismatch");+ assert_eq!(s["owner_pid"], 900);+ assert_eq!(s["owner_app_name"], "Open and Save Panel Service");+ assert_eq!(s["pid"], 800, "the requested pid is echoed, not replaced");+ assert!(+ s["suggestion"].as_str().unwrap().contains("pid=900"),+ "the retry must name the owner pid: {}",+ s["suggestion"]+ );+ assert!(+ !text.contains("AXMenuBar"),+ "the refusal must never carry menu-bar content"+ );+ }++ #[test]+ fn resolvable_scopes_are_not_refused() {+ assert!(window_scope_refusal(800, 11, &WindowScope::Matched).is_none());+ assert!(+ window_scope_refusal(800, 11, &WindowScope::AxUnresolved { ax_window_count: 2 })+ .is_none(),+ "a live same-pid window degrades; it does not error"+ );+ }++ /// The reported failure signature: a wrong-surface walk returns a healthy+ /// non-zero element count, so the pre-fix `element_count == 0` rung stayed+ /// silent. An unresolved scope now degrades on its own evidence.+ #[test]+ fn unresolved_scope_degrades_with_its_own_reason() {+ assert_eq!(+ degradation_for(+ true,+ 0,+ Some(&WindowScope::AxUnresolved { ax_window_count: 3 })+ ),+ Degradation::AxWindowUnresolved { ax_window_count: 3 }+ );+ }++ #[test]+ fn empty_tree_still_degrades_as_ax_tree_empty() {+ // Back-compat with the pre-existing rung.+ assert_eq!(+ degradation_for(true, 0, Some(&WindowScope::Matched)),+ Degradation::AxTreeEmpty+ );+ }++ #[test]+ fn resolved_window_with_elements_is_not_degraded() {+ assert_eq!(+ degradation_for(true, 42, Some(&WindowScope::Matched)),+ Degradation::None+ );+ }++ #[test]+ fn screenshot_only_path_does_not_degrade() {+ assert_eq!(degradation_for(false, 0, None), Degradation::None);+ }++ #[test]+ fn schema_advertises_the_window_scope_error_codes() {+ let description = def().description.clone();+ for code in [+ "window_id_not_found",+ "window_owner_pid_mismatch",+ "ax_window_unresolved",+ ] {+ assert!(+ description.contains(code),+ "tool description must advertise {code}"+ );+ }+ }+}+#[cfg(test)]mod tests {use super::*;diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rsindex 4c9c1ed1b3..f5c5ca2277 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs@@ -30,6 +30,7 @@ pub(crate) mod get_screen_size;mod health_report;mod move_cursor;mod page;+pub(crate) mod px_frame;mod set_config;mod type_text_chars;mod zoom;@@ -247,13 +248,19 @@ impl ZoomRegistry {}}-/// Tracks the per-pid ratio applied by `max_image_dimension` downscaling.+/// Tracks the per-(pid, window_id) ratio applied by `max_image_dimension`+/// downscaling.////// `ratio = original_dim / resized_dim` — multiply resized image coordinates/// by this to recover original (native) window-local pixel coordinates./// Mirrors Swift's `ImageResizeRegistry`.+///+/// Keyed per window, matching the element cache and the element-token+/// registry. A pid-only key leaked the ratio recorded while snapshotting+/// window A into pixel clicks aimed at window B of the same pid, sending them+/// off-target (issue #2237).pub struct ResizeRegistry {- inner: std::sync::Mutex<HashMap<i32, f64>>,+ inner: std::sync::Mutex<HashMap<(i32, u32), f64>>,}impl Default for ResizeRegistry {@@ -269,19 +276,39 @@ impl ResizeRegistry {}}- /// Record that pid's screenshot was downscaled by `ratio`.- pub fn set_ratio(&self, pid: i32, ratio: f64) {- self.inner.lock().unwrap().insert(pid, ratio);- }-- /// Remove the ratio entry (no active downscale).- pub fn clear_ratio(&self, pid: i32) {- self.inner.lock().unwrap().remove(&pid);- }-- /// Returns the most recent ratio, or `None` if no downscale happened.- pub fn ratio(&self, pid: i32) -> Option<f64> {- self.inner.lock().unwrap().get(&pid).copied()+ /// Record that (pid, window_id)'s screenshot was downscaled by `ratio`.+ pub fn set_ratio(&self, pid: i32, window_id: u32, ratio: f64) {+ self.inner.lock().unwrap().insert((pid, window_id), ratio);+ }++ /// Remove the ratio entry for one window (no active downscale).+ pub fn clear_ratio(&self, pid: i32, window_id: u32) {+ self.inner.lock().unwrap().remove(&(pid, window_id));+ }++ /// The ratio for a window, or `None` if no downscale happened.+ ///+ /// `window_id: None` is the screen-scope (legacy) path: it returns a ratio+ /// only when every window recorded for `pid` agrees on one, so a+ /// window-less caller can never inherit some other window's scale. That+ /// preserves today's behaviour for the single-window case without guessing+ /// across windows.+ pub fn ratio(&self, pid: i32, window_id: Option<u32>) -> Option<f64> {+ let inner = self.inner.lock().unwrap();+ match window_id {+ Some(wid) => inner.get(&(pid, wid)).copied(),+ None => {+ let mut agreed: Option<f64> = None;+ for (_, ratio) in inner.iter().filter(|((p, _), _)| *p == pid) {+ match agreed {+ None => agreed = Some(*ratio),+ Some(seen) if (seen - *ratio).abs() < 1e-9 => {}+ Some(_) => return None,+ }+ }+ agreed+ }+ }}}@@ -738,6 +765,75 @@ mod session_config_guard_tests {}}+#[cfg(test)]+mod resize_registry_tests {+ use super::ResizeRegistry;++ /// Issue #2237: the registry was keyed by pid alone, so the downscale+ /// ratio recorded while snapshotting one window was applied to pixel+ /// clicks aimed at another window of the same app.+ #[test]+ fn resize_ratio_is_keyed_per_window() {+ let reg = ResizeRegistry::new();+ reg.set_ratio(800, 11, 2.0);+ reg.set_ratio(800, 22, 1.25);+ assert_eq!(reg.ratio(800, Some(11)), Some(2.0));+ assert_eq!(reg.ratio(800, Some(22)), Some(1.25));+ }++ #[test]+ fn undownscaled_window_reports_no_ratio() {+ let reg = ResizeRegistry::new();+ reg.set_ratio(800, 11, 2.0);+ assert_eq!(+ reg.ratio(800, Some(22)),+ None,+ "window 22 was never downscaled; it must not inherit window 11's ratio"+ );+ }++ #[test]+ fn clearing_one_window_keeps_the_other() {+ let reg = ResizeRegistry::new();+ reg.set_ratio(800, 11, 2.0);+ reg.set_ratio(800, 22, 1.25);+ reg.clear_ratio(800, 11);+ assert_eq!(reg.ratio(800, Some(11)), None);+ assert_eq!(reg.ratio(800, Some(22)), Some(1.25));+ }++ #[test]+ fn distinct_pids_with_the_same_window_id_do_not_collide() {+ let reg = ResizeRegistry::new();+ reg.set_ratio(800, 11, 2.0);+ reg.set_ratio(900, 11, 3.0);+ assert_eq!(reg.ratio(800, Some(11)), Some(2.0));+ assert_eq!(reg.ratio(900, Some(11)), Some(3.0));+ }++ /// Screen-scope callers pass no window_id. One window (the common case)+ /// keeps working; disagreeing windows refuse to guess.+ #[test]+ fn screen_scope_lookup_only_answers_when_windows_agree() {+ let reg = ResizeRegistry::new();+ assert_eq!(reg.ratio(800, None), None, "nothing recorded yet");+ reg.set_ratio(800, 11, 2.0);+ assert_eq!(+ reg.ratio(800, None),+ Some(2.0),+ "single window is unambiguous"+ );+ reg.set_ratio(800, 22, 2.0);+ assert_eq!(reg.ratio(800, None), Some(2.0), "agreeing windows answer");+ reg.set_ratio(800, 33, 1.25);+ assert_eq!(+ reg.ratio(800, None),+ None,+ "disagreeing windows must not pick one arbitrarily"+ );+ }+}+#[cfg(test)]mod cursor_overlay_facility_tests {use super::*;diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/px_frame.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/px_frame.rsnew file mode 100644index 0000000000..19d54df23b--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/px_frame.rs@@ -0,0 +1,312 @@+//! The one place that turns window-local screenshot pixels into screen+//! coordinates for the pixel rungs (`click`, `double_click`, `right_click`,+//! `drag`, `scroll`).+//!+//! Issue #2237: each of those tools carried a byte-identical copy of this+//! translation, and every copy silently reinterpreted window-local pixels as+//! screen-absolute when `window_bounds_by_id` returned `None`. A dismissed+//! Open/Save panel's coordinates then landed on whatever was behind it — a+//! misclick reported as success. There is no safe fallback here: without the+//! window's origin there is nothing to add the local point to, so a missing or+//! stale window is a refusal.++use cua_driver_core::protocol::ToolResult;++use crate::windows::WindowBounds;++/// A window's screen origin plus the physical-pixels-per-logical-point scale+/// of its `screencapture` output.+#[derive(Debug, Clone)]+pub struct WindowPxFrame {+ pub bounds: WindowBounds,+ /// Physical capture pixels per logical point (1.0 non-Retina, 2.0 Retina).+ pub scale: f64,+}++impl WindowPxFrame {+ /// Window-local capture pixels → `(screen_x, screen_y, local_pt_x,+ /// local_pt_y)`. The local-point pair is what+ /// `CGEventSetWindowLocation` needs for background delivery.+ pub fn to_screen(&self, cx: f64, cy: f64) -> (f64, f64, f64, f64) {+ let lx = cx / self.scale;+ let ly = cy / self.scale;+ (self.bounds.x + lx, self.bounds.y + ly, lx, ly)+ }+}++/// Why a window-local pixel action cannot be translated.+#[derive(Debug, Clone, PartialEq)]+pub enum PxFrameError {+ /// WindowServer has no usable frame for this id: it was closed, the id is+ /// stale or fabricated, or the record carries degenerate `0×0` bounds+ /// (`windows.rs`'s missing-`kCGWindowBounds` default), which would make+ /// the window origin indistinguishable from the screen origin.+ WindowNotFound { window_id: u32 },+ /// The current capture could not be obtained or decoded, so its pixel+ /// coordinate system cannot be proven against the WindowServer frame.+ CaptureUnavailable { window_id: u32, reason: String },+ /// The capture dimensions are not a coherent 1×/2× representation of the+ /// requested WindowServer frame. Dispatching with this transform would+ /// recreate #2237's wrong-surface misclick.+ FrameMismatch {+ window_id: u32,+ bounds_width: f64,+ bounds_height: f64,+ capture_width: u32,+ capture_height: u32,+ scale_x: f64,+ scale_y: f64,+ },+}++/// Validate that a raw window capture and its WindowServer bounds describe one+/// coordinate frame, returning the physical-pixels-per-point backing scale.+///+/// macOS window captures are 1× or 2×. Small rounding differences are allowed,+/// but arbitrary ratios are not: the original #2237 failure paired a parent+/// window screenshot with panel bounds and would otherwise look like a+/// plausible-but-wrong ~2.2× scale.+pub fn validate_capture_frame(+ window_id: u32,+ bounds: &WindowBounds,+ capture_px_w: u32,+ capture_px_h: u32,+) -> Result<f64, PxFrameError> {+ if bounds.width <= 0.0 || bounds.height <= 0.0 {+ return Err(PxFrameError::WindowNotFound { window_id });+ }+ let scale_x = capture_px_w as f64 / bounds.width;+ let scale_y = capture_px_h as f64 / bounds.height;+ let nearest = if (scale_x - 1.0).abs() <= (scale_x - 2.0).abs() {+ 1.0+ } else {+ 2.0+ };+ let coherent = scale_x.is_finite()+ && scale_y.is_finite()+ && (scale_x - scale_y).abs() <= 0.03+ && (scale_x - nearest).abs() <= 0.03+ && (scale_y - nearest).abs() <= 0.03;+ if coherent {+ Ok(nearest)+ } else {+ Err(PxFrameError::FrameMismatch {+ window_id,+ bounds_width: bounds.width,+ bounds_height: bounds.height,+ capture_width: capture_px_w,+ capture_height: capture_px_h,+ scale_x,+ scale_y,+ })+ }+}++/// Resolve `window_id`'s screen origin and capture scale. Blocking: enumerates+/// CGWindowList and takes one window capture to measure the scale.+pub fn resolve_window_px_frame(window_id: u32) -> Result<WindowPxFrame, PxFrameError> {+ let bounds = crate::windows::window_bounds_by_id(window_id)+ .filter(|b| b.width > 0.0 && b.height > 0.0)+ .ok_or(PxFrameError::WindowNotFound { window_id })?;+ // A window-local point is meaningful only when the current capture proves+ // the same frame and scale. Never guess 1× on capture failure.+ let png = crate::capture::screenshot_window_bytes(window_id).map_err(|e| {+ PxFrameError::CaptureUnavailable {+ window_id,+ reason: e.to_string(),+ }+ })?;+ let (pw, ph) =+ crate::capture::png_dimensions(&png).map_err(|e| PxFrameError::CaptureUnavailable {+ window_id,+ reason: e.to_string(),+ })?;+ let scale = validate_capture_frame(window_id, &bounds, pw, ph)?;+ Ok(WindowPxFrame { bounds, scale })+}++/// The shared refusal for a pixel action whose window cannot be framed.+pub fn refusal(error: &PxFrameError) -> ToolResult {+ match error {+ PxFrameError::WindowNotFound { window_id } => ToolResult::error(format!(+ "window_id {window_id} has no live frame, so window-local pixels cannot be \+ translated to screen coordinates. Refusing to dispatch — treating them as \+ screen-absolute would act on whatever is behind the closed window. Call \+ list_windows for a current window_id, then re-snapshot with get_window_state."+ ))+ .with_structured(error_structured(error)),+ PxFrameError::CaptureUnavailable { window_id, reason } => ToolResult::error(format!(+ "window_id {window_id}'s current capture is unavailable ({reason}), so its \+ pixel coordinate frame cannot be verified. Refusing to dispatch."+ ))+ .with_structured(error_structured(error)),+ PxFrameError::FrameMismatch {+ window_id,+ bounds_width,+ bounds_height,+ capture_width,+ capture_height,+ scale_x,+ scale_y,+ } => ToolResult::error(format!(+ "window_id {window_id}'s capture is {capture_width}x{capture_height}, but its \+ WindowServer bounds are {bounds_width:.2}x{bounds_height:.2} points \+ (scale_x={scale_x:.4}, scale_y={scale_y:.4}). These are not one coherent \+ 1x/2x frame. Refusing to dispatch."+ ))+ .with_structured(error_structured(error)),+ }+}++/// Structured form shared by action refusals and a successful state snapshot+/// whose optional screenshot could not be proven. The latter must keep its AX+/// payload rather than turning a capture problem into a total perception+/// failure.+pub fn error_structured(error: &PxFrameError) -> serde_json::Value {+ match error {+ PxFrameError::WindowNotFound { window_id } => serde_json::json!({+ "code": "px_window_not_found",+ "window_id": window_id,+ "suggestion": "refusing to interpret window-local pixels as screen-absolute; \+ call list_windows for a current window_id"+ }),+ PxFrameError::CaptureUnavailable { window_id, reason } => serde_json::json!({+ "code": "px_capture_unavailable",+ "window_id": window_id,+ "reason": reason,+ "suggestion": "re-snapshot the window after capture permission/state recovers"+ }),+ PxFrameError::FrameMismatch {+ window_id,+ bounds_width,+ bounds_height,+ capture_width,+ capture_height,+ scale_x,+ scale_y,+ } => serde_json::json!({+ "code": "px_frame_mismatch",+ "window_id": window_id,+ "window_bounds": {+ "width": bounds_width,+ "height": bounds_height+ },+ "capture_size": {+ "width": capture_width,+ "height": capture_height+ },+ "scale_x": scale_x,+ "scale_y": scale_y,+ "suggestion": "re-list and re-snapshot the exact window; do not reuse coordinates \+ from a different surface"+ }),+ }+}++/// Resolve the frame off the async path, mapping both the task error and the+/// refusal onto a `ToolResult` the caller can return directly.+pub async fn resolve_or_refuse(window_id: u32) -> Result<WindowPxFrame, ToolResult> {+ match tokio::task::spawn_blocking(move || resolve_window_px_frame(window_id)).await {+ Ok(Ok(frame)) => Ok(frame),+ Ok(Err(e)) => Err(refusal(&e)),+ Err(e) => Err(ToolResult::error(format!(+ "window frame lookup for window_id {window_id} failed: {e}. Not dispatching."+ ))),+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ fn frame(scale: f64) -> WindowPxFrame {+ WindowPxFrame {+ bounds: WindowBounds {+ x: 100.0,+ y: 580.0,+ width: 500.0,+ height: 500.0,+ },+ scale,+ }+ }++ #[test]+ fn resolves_1x_window_local_to_screen() {+ let (sx, sy, lx, ly) = frame(1.0).to_screen(30.0, 40.0);+ assert_eq!((sx, sy), (130.0, 620.0));+ assert_eq!((lx, ly), (30.0, 40.0));+ }++ #[test]+ fn resolves_2x_retina_scale() {+ // A point 60 capture px in is 30 pt in.+ let (sx, sy, lx, ly) = frame(2.0).to_screen(60.0, 80.0);+ assert_eq!((sx, sy), (130.0, 620.0));+ assert_eq!((lx, ly), (30.0, 40.0));+ }++ #[test]+ fn backing_scale_detects_retina_capture() {+ let b = frame(1.0).bounds;+ assert_eq!(validate_capture_frame(11, &b, 1000, 1000).unwrap(), 2.0);+ }++ #[test]+ fn backing_scale_is_one_for_point_sized_capture() {+ let b = frame(1.0).bounds;+ assert_eq!(validate_capture_frame(11, &b, 500, 500).unwrap(), 1.0);+ }++ #[test]+ fn backing_scale_never_divides_by_zero_bounds() {+ // windows.rs defaults absent kCGWindowBounds to 0×0.+ let mut b = frame(1.0).bounds;+ b.width = 0.0;+ assert_eq!(+ validate_capture_frame(11, &b, 1000, 1000),+ Err(PxFrameError::WindowNotFound { window_id: 11 })+ );+ }++ #[test]+ fn capture_narrower_than_bounds_is_a_mismatch() {+ let b = frame(1.0).bounds;+ assert!(matches!(+ validate_capture_frame(11, &b, 250, 250),+ Err(PxFrameError::FrameMismatch { .. })+ ));+ }++ #[test]+ fn parent_capture_cannot_masquerade_as_a_panel_scale() {+ let bounds = WindowBounds {+ x: 72.0,+ y: 88.0,+ width: 880.0,+ height: 448.0,+ };+ assert!(matches!(+ validate_capture_frame(55, &bounds, 1920, 960),+ Err(PxFrameError::FrameMismatch { .. })+ ));+ }++ /// Locks the exact regression: a stale window_id used to fall through to+ /// "treat x,y as screen coordinates".+ #[test]+ fn missing_window_refuses_instead_of_treating_local_as_screen_px() {+ let refused = refusal(&PxFrameError::WindowNotFound { window_id: 67340 });+ assert_eq!(refused.is_error, Some(true));+ let structured = refused.structured_content.expect("structured refusal");+ assert_eq!(structured["code"], "px_window_not_found");+ assert_eq!(structured["window_id"], 67340);+ assert!(+ structured["suggestion"]+ .as_str()+ .unwrap()+ .contains("list_windows"),+ "the refusal must name the retry path"+ );+ }+}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rsindex 338cda20d9..6abbe8f9fd 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs@@ -170,46 +170,18 @@ impl Tool for RightClickTool {// ── Pixel path ───────────────────────────────────────────────────────let (mut cx, mut cy) = (x.unwrap(), y.unwrap());// Scale back from downscaled-image space to native pixels when needed.- if let Some(ratio) = self.state.resize_registry.ratio(pid) {+ if let Some(ratio) = self.state.resize_registry.ratio(pid, window_id) {cx *= ratio;cy *= ratio;}// Window-local → screen coordinate translation + win-local logical coords- // for CGEventSetWindowLocation (matches click.rs enhancement).+ // for CGEventSetWindowLocation (shared with click.rs via px_frame, which+ // refuses a window with no live frame).let (screen_x, screen_y, win_local_x, win_local_y) = if let Some(wid) = window_id {- let result = tokio::task::spawn_blocking(move || {- let bounds = crate::windows::window_bounds_by_id(wid);- let scale: f64 = if let Some(ref b) = bounds {- if let Ok(png) = crate::capture::screenshot_window_bytes(wid) {- if png.len() >= 24 {- let pw =- u32::from_be_bytes([png[16], png[17], png[18], png[19]]) as f64;- let lw = b.width;- if lw > 0.0 && pw > lw {- pw / lw- } else {- 1.0- }- } else {- 1.0- }- } else {- 1.0- }- } else {- 1.0- };- (bounds, scale)- })- .await- .unwrap_or((None, 1.0));- if let (Some(b), scale) = result {- let wx = cx / scale;- let wy = cy / scale;- (b.x + wx, b.y + wy, wx, wy)- } else {- (cx, cy, cx, cy)+ match super::px_frame::resolve_or_refuse(wid).await {+ Ok(frame) => frame.to_screen(cx, cy),+ Err(refusal) => return refusal,}} else {(cx, cy, cx, cy)diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rsindex 29e9ba2f04..986dbed249 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs@@ -373,55 +373,32 @@ impl Tool for ScrollTool {);}// Pixel path: x,y are window-local screenshot pixels. Mirror the- // click pixel path — undo any session downscale, then add the- // window origin and divide out the Retina backing scale.- if let Some(ratio) = self.state.resize_registry.ratio(pid) {+ // click pixel path — undo any session downscale, then translate+ // through the shared window frame (which refuses a window with no+ // live frame rather than scrolling at screen-absolute coords).+ if let Some(ratio) = self.state.resize_registry.ratio(pid, window_id) {cx *= ratio;cy *= ratio;}- let wid = window_id;- tokio::task::spawn_blocking(move || {- if let Some(wid) = wid {- let bounds = crate::windows::window_bounds_by_id(wid);- let scale: f64 = if let Some(ref b) = bounds {- if let Ok(png) = crate::capture::screenshot_window_bytes(wid) {- if png.len() >= 24 {- let pw =- u32::from_be_bytes([png[16], png[17], png[18], png[19]]) as f64;- if b.width > 0.0 && pw > b.width {- pw / b.width- } else {- 1.0- }- } else {- 1.0- }- } else {- 1.0- }- } else {- 1.0- };- if let Some(b) = bounds {- let (wx, wy) = (cx / scale, cy / scale);- return WheelTarget {- screen_x: b.x + wx,- screen_y: b.y + wy,- win_local: Some((wx, wy)),- wid: Some(wid),- };- }- }- // No window_id → treat x,y as screen coordinates.- WheelTarget {- screen_x: cx,- screen_y: cy,- win_local: None,- wid: None,+ let Some(wid) = window_id else {+ // Unreachable: the None case refused above. Kept explicit so a+ // future edit cannot reintroduce the screen-absolute fallback.+ return ToolResult::error(+ "window_id is required when scrolling by window-local x,y pixels.".to_string(),+ );+ };+ match super::px_frame::resolve_or_refuse(wid).await {+ Ok(frame) => {+ let (sx, sy, lx, ly) = frame.to_screen(cx, cy);+ Some(WheelTarget {+ screen_x: sx,+ screen_y: sy,+ win_local: Some((lx, ly)),+ wid: Some(wid),+ })}- })- .await- .ok()+ Err(refusal) => return refusal,+ }} else {None};diff --git a/libs/cua-driver/rust/crates/platform-macos/src/windows.rs b/libs/cua-driver/rust/crates/platform-macos/src/windows.rsindex 4b52345851..740b8a9dfa 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/windows.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/windows.rs@@ -58,15 +58,40 @@ extern "C" {/// Enumerate all windows (including off-screen).pub fn all_windows() -> Vec<WindowInfo> {- enumerate_windows(kCGWindowListExcludeDesktopElements)+ enumerate_windows(kCGWindowListExcludeDesktopElements, LayerFilter::ZeroOnly)}/// Enumerate only on-screen windows.pub fn visible_windows() -> Vec<WindowInfo> {- enumerate_windows(kCGWindowListOptionOnScreenOnly | kCGWindowListExcludeDesktopElements)+ enumerate_windows(+ kCGWindowListOptionOnScreenOnly | kCGWindowListExcludeDesktopElements,+ LayerFilter::ZeroOnly,+ )}-fn enumerate_windows(options: u32) -> Vec<WindowInfo> {+/// Enumerate windows on every CGWindow layer, including the accessory layers+/// (`layer != 0`) that [`all_windows`] hides.+///+/// Only used to answer "does this CGWindowID exist, and who owns it?" — the+/// question `list_windows` must NOT answer, because surfacing tooltips,+/// popovers, the Dock and every NSMenu window would swamp callers. Keeping the+/// layer filter on enumeration and off identity lookup is what lets+/// `get_window_state` tell "no such window" apart from "exists, but is not a+/// layer-0 window" (issue #2237).+fn all_windows_any_layer() -> Vec<WindowInfo> {+ enumerate_windows(kCGWindowListExcludeDesktopElements, LayerFilter::AnyLayer)+}++/// Which CGWindow layers an enumeration admits.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+enum LayerFilter {+ /// Normal application windows only — what `list_windows` reports.+ ZeroOnly,+ /// Every layer, accessory windows included.+ AnyLayer,+}++fn enumerate_windows(options: u32, layers: LayerFilter) -> Vec<WindowInfo> {use core_foundation::{array::CFArray,base::{CFGetTypeID, CFTypeRef, TCFType},@@ -147,8 +172,8 @@ fn enumerate_windows(options: u32) -> Vec<WindowInfo> {let layer = get_num("kCGWindowLayer") as i32;let is_on_screen = get_bool("kCGWindowIsOnscreen");- // Only include layer-0 windows.- if layer != 0 {+ // Only include layer-0 windows, unless the caller asked for every layer.+ if layer != 0 && layers == LayerFilter::ZeroOnly {continue;}@@ -230,15 +255,59 @@ fn get_bounds_num(.unwrap_or(0.0)}+/// Look up a window by its CGWindowID across every layer.+///+/// Returns `None` only when WindowServer has no record of the id at all —+/// which is precisely the "closed or fabricated window_id" signal callers need.+pub fn window_info_by_id(window_id: u32) -> Option<WindowInfo> {+ all_windows_any_layer()+ .into_iter()+ .find(|w| w.window_id == window_id)+}+/// Look up a window's bounds by its CGWindowID.////// Returns `None` if the window is not currently known to WindowServer/// (e.g. it was closed or the window_id is stale).pub fn window_bounds_by_id(window_id: u32) -> Option<WindowBounds> {- all_windows()- .into_iter()- .find(|w| w.window_id == window_id)- .map(|w| w.bounds)+ window_info_by_id(window_id).map(|w| w.bounds)+}++/// Who owns a requested CGWindowID, as seen by a caller that asked about `pid`.+#[derive(Debug, Clone, PartialEq, Eq)]+pub enum WindowOwner {+ /// The window exists and `pid` owns it.+ SamePid,+ /// The window exists, but a different process owns it. macOS hosts a+ /// sandboxed app's Open/Save panel in+ /// `com.apple.appkit.xpc.openAndSavePanelService`, so the panel's+ /// CGWindowID belongs to that service and not to the app that opened it+ /// (issue #2237).+ ForeignPid {+ owner_pid: i32,+ owner_app_name: String,+ },+ /// WindowServer has no record of the id — closed, stale, or fabricated.+ Unknown,+}++/// Pure form of [`resolve_window_owner`] over an already-enumerated window+/// list, so the ownership decision is testable without a WindowServer.+pub fn resolve_window_owner_in(windows: &[WindowInfo], pid: i32, window_id: u32) -> WindowOwner {+ match windows.iter().find(|w| w.window_id == window_id) {+ None => WindowOwner::Unknown,+ Some(w) if w.pid == pid => WindowOwner::SamePid,+ Some(w) => WindowOwner::ForeignPid {+ owner_pid: w.pid,+ owner_app_name: w.app_name.clone(),+ },+ }+}++/// Resolve whether `pid` really owns `window_id`. Blocking (one CGWindowList+/// enumeration).+pub fn resolve_window_owner(pid: i32, window_id: u32) -> WindowOwner {+ resolve_window_owner_in(&all_windows_any_layer(), pid, window_id)}/// Select the best window_id for a pid.@@ -262,3 +331,85 @@ pub fn resolve_main_window_id(pid: i32) -> anyhow::Result<u32> {});Ok(largest.unwrap().window_id)}++#[cfg(test)]+mod tests {+ use super::*;++ fn window(window_id: u32, pid: i32, app_name: &str) -> WindowInfo {+ WindowInfo {+ window_id,+ pid,+ app_name: app_name.into(),+ title: String::new(),+ bounds: WindowBounds {+ x: 0.,+ y: 580.,+ width: 500.,+ height: 500.,+ },+ layer: 0,+ z_index: 1,+ is_on_screen: true,+ on_current_space: None,+ space_ids: None,+ }+ }++ #[test]+ fn owner_resolves_same_pid() {+ let windows = vec![window(42, 800, "TextEdit")];+ assert_eq!(+ resolve_window_owner_in(&windows, 800, 42),+ WindowOwner::SamePid+ );+ }++ /// Issue #2237: TextEdit's Open panel is a layer-0 CGWindow owned by+ /// `com.apple.appkit.xpc.openAndSavePanelService`, not by TextEdit. The+ /// caller must be told the real owner pid, not handed TextEdit's menu bar.+ #[test]+ fn owner_detects_out_of_process_panel_host() {+ let windows = vec![+ window(41, 800, "TextEdit"),+ window(42, 900, "Open and Save Panel Service"),+ ];+ assert_eq!(+ resolve_window_owner_in(&windows, 800, 42),+ WindowOwner::ForeignPid {+ owner_pid: 900,+ owner_app_name: "Open and Save Panel Service".into(),+ }+ );+ }++ #[test]+ fn owner_is_unknown_for_fabricated_id() {+ let windows = vec![window(42, 800, "TextEdit")];+ assert_eq!(+ resolve_window_owner_in(&windows, 800, 0xFFFF_FFF0),+ WindowOwner::Unknown+ );+ }++ #[test]+ fn owner_is_unknown_for_zero_id() {+ // kCGNullWindowID is never a real window number.+ let windows = vec![window(42, 800, "TextEdit")];+ assert_eq!(+ resolve_window_owner_in(&windows, 800, 0),+ WindowOwner::Unknown+ );+ }++ #[test]+ fn owner_is_unknown_after_the_window_closes() {+ // Stale id: it was enumerated once, then the panel was dismissed.+ let before = vec![window(42, 900, "Open and Save Panel Service")];+ assert_eq!(+ resolve_window_owner_in(&before, 900, 42),+ WindowOwner::SamePid+ );+ assert_eq!(resolve_window_owner_in(&[], 900, 42), WindowOwner::Unknown);+ }+}