Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cua-driver): land background and foreground input on X11 desktops (OSWorld) and budget get_window_state walks on every platform (#3882)
ddupont7ffcdf13e635
diff --git a/docs/content/docs/reference/cua-driver/mcp-tools-linux.mdx b/docs/content/docs/reference/cua-driver/mcp-tools-linux.mdxindex 1c6ba3e80c..d1b72b2996 100644--- a/docs/content/docs/reference/cua-driver/mcp-tools-linux.mdx+++ b/docs/content/docs/reference/cua-driver/mcp-tools-linux.mdx@@ -42,8 +42,9 @@ List Linux apps — both currently running and installed-but-not-running — wit- launch_path: the launcher command from `Exec=` (field codes stripped). Pass to `launch_app(launch_path=...)`.- bundle_id: the XDG "desktop file id" — the `.desktop` file's path relative to its XDG `applications/` root with the `.desktop` suffix stripped and path separators replaced with `-` (e.g. `kde4/konqbrowser.desktop` → `kde4-konqbrowser`).- last_used: RFC3339 mtime of the `.desktop` file, when readable.+- windows: the app's current top-level windows (same records as list_windows); empty for a process without one.-Running apps come from `/proc`. Installed apps come from XDG Desktop Entry files in $XDG_DATA_HOME/applications and each $XDG_DATA_DIRS entry's applications/ subdir. Entries with `NoDisplay=true` or `Hidden=true` are filtered. A `.desktop` file whose launcher matches a running process (by basename) is merged into a single entry with `running: true`.+Running apps come from `/proc`. Installed apps come from XDG Desktop Entry files in $XDG_DATA_HOME/applications and each $XDG_DATA_DIRS entry's applications/ subdir. Entries with `NoDisplay=true` or `Hidden=true` are filtered. A `.desktop` file whose launcher matches a running process (by basename), or whose WM_CLASS a top-level window carries, is merged into a single entry with `running: true`.Use this for "is X installed?" as well as "is X running?". For per-window state — visibility, geometry, titles — call list_windows instead.@@ -70,6 +71,12 @@ The mirror image: pass `include_accessibility_tree:false` to SKIP the AT-SPI walOptional `max_elements` / `max_depth` bound the AT-SPI walk to mitigate context-window blow-up on Electron / large web apps that produce 10k+ element trees. When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour.+TIME BUDGET: `timeout_ms` (default 1000) bounds the whole AT-SPI walk. Large apps (LibreOffice, GIMP, file managers) can exceed it; the call then returns the PARTIAL tree with `truncated: true`, `truncation_reason`, `nodes_visited`/`nodes_pending` and `elements_complete: false`. Every element listed is real and clickable; elements after the cut are simply missing. Retry with a larger `timeout_ms` (e.g. 5000) or narrow with `query`/`max_depth` when the element you need is absent.++SCREENSHOT SCALE: the screenshot is delivered at or below 1.15 megapixels (long edge <= max_image_dimension, 1568 by default), because larger images are downsized before a model reads them and its pixel coordinates would then be uniformly short. Element `screenshot_frame`s and x/y for the pointer tools are pixels of the delivered screenshot (`frame` stays in screen coordinates, as on every platform); `frame_scale` < 1 reports the downsizing. An explicit per-call `max_image_dimension` (0 = native) replaces this cap.++POPUP MENUS: a context menu / popover / combo list is an override-redirect window that list_windows never shows. A click or right_click that opened one names it in its result (`popup: {window_id, bounds, title}`); pass that window_id here to walk the popup's own AT-SPI toplevel so its menu items get element indices (then click them by element_index). Omitting window_id while a popup of this pid is open walks that popup.+**Arguments:**- `capture_mode` (string, optional): DEPRECATED and ignored. get_window_state always returns BOTH the element tree and a screenshot — ground on both. The modality is chosen at action time by how you address the target: an element ax action (element_index/element_token) or an element px action (x,y). Any value (including the old "som"/"screenshot" aliases) is accepted but has no effect.@@ -83,10 +90,11 @@ Optional `max_elements` / `max_depth` bound the AT-SPI walk to mitigate context-- `query` (string, optional): Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count.- `screenshot_out_file` (string, optional): When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output carries screenshot_file_path instead.- `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.-- `window_id` (integer, required): Native window identifier from list_windows.+- `timeout_ms` (integer, optional): Wall-clock budget in milliseconds for the accessibility-tree walk (default 1000, min 100, max 120000). Bounds the WHOLE walk. When the budget runs out the tool returns the PARTIAL tree it has, flagged with `truncated: true`, `truncation_reason`, `nodes_visited`, `nodes_pending` and `elements_complete: false`; retry with a larger value (e.g. 5000) or narrow with `query` / `max_depth`. default: `1000`; range: 100–120000+- `window_id` (integer, optional): Native window identifier from list_windows, or the `popup.window_id` a click / right_click result named (an open context menu / popover; its menu items then get element indices). Omitted: the pid's open popup menu when one is mapped, else its focused / active / largest window.```json-{"pid":844,"window_id":10725}+{"pid":844}```### `get_accessibility_tree`@@ -103,6 +111,7 @@ Capture the full display in the desktop action coordinate frame. Use the returne**Arguments:**+- `max_image_dimension` (integer, optional): Optional long-edge cap for the returned PNG, in pixels (aspect ratio preserved). Omitted or 0 returns the full-size capture. When the cap downsizes the image, the response reports `screenshot_original_width/height`, and x/y read off it for this session's later scope:"desktop" actions (or passed with its `capture_id`) are mapped back to the full-size frame automatically. range: 0–unbounded- `screenshot_out_file` (string, optional): Write PNG here instead of base64.- `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.@@ -210,17 +219,20 @@ accepts it. Omit it to use the authenticated transport's implicit lifecycle sessClick against a target pid. **Prefer `element_index` over pixel coordinates** — element_index works on backgrounded / hidden windows, surfaces a stable handle, and tells you what you're clicking via the cached AT-SPI element's role + label. Reach for `x, y` only when the target is a canvas / custom-drawn surface that doesn't appear in the AT-SPI tree.-Provide either (window_id + x/y) or (pid + element_index). Routes via XSendEvent (no focus steal). element_index cache is scoped per (pid, window_id) and is replaced by the next get_window_state of the same window — re-snapshot every turn before clicking.+Provide either (window_id + x/y) or (pid + element_index). x/y are WINDOW-LOCAL pixels of window_id (the get_window_state screenshot frame); a point outside the window is refused (point_outside_window) and every pixel result reports the resolved window_point / screen_point. Background delivery first resolves the accessible under the point and fires its action (the result says `hit: <role> "<name>" action=...`; a file / list item is selected through its container: `selected: ...`), else presses the real virtual pointer there. A click that opens a popup menu names it (`popup: {window_id, bounds}`): pass that window_id to get_window_state to index its items. element_index cache is scoped per (pid, window_id) and is replaced by the next get_window_state of the same window — re-snapshot every turn before clicking.After a zoom call, pass from_zoom=true to auto-translate zoom-image coords back to full-window space.+`count`: 1 (default), 2 for a double-click, 3 for a triple-click (selects a line / paragraph in editors) — one press train with real double-click cadence, in background and foreground alike; there is no separate triple_click tool.+button: "left" (default), "right", or "middle". Defaults to left so the field is fully back-compat. X11: routes through XSendEvent ButtonPress/Release with the matching button code. Native Wayland: only left-button is supported via the virtual-pointer protocol — right/middle return an error rather than silently degrading to left. `modifier` holds ctrl/shift/alt/super for the click on X11. Native Wayland refuses modified pointer clicks until its input protocol can carry keyboard modifier state.**Arguments:**- `button` (string, optional): Mouse button. Default: "left" (legacy back-compat). X11: routed via ButtonPress/Release with the matching evdev code. Native Wayland: only left-button is supported via the virtual-pointer protocol; right/middle return an error.- `capture_id` (string, optional): Optional one-shot binding to the exact PNG returned by get_window_state or get_desktop_state. When present, x/y are interpreted in that capture and admitted before native dispatch.-- `count` (integer, optional)+- `coordinate_frame` (string, optional): Frame of x/y (and from_x/from_y/to_x/to_y). Default "window": window-local screenshot pixels as returned by get_window_state. "desktop": full-screen pixels as returned by get_desktop_state, translated to the target window. Passing scope:"desktop" together with pid/window_id means the same thing.+- `count` (integer, optional): Number of clicks in one press train: 1 (default), 2 = double-click, 3 = triple-click (line/paragraph selection). There is no separate triple_click tool — pass count: 3 here. range: 1–unbounded- `cursor_id` (string, optional): Optional multi-cursor instance id. Default: 'default'.- `delivery_mode` (string, optional): Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface. default: `"background"`- `element_index` (integer, optional): Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.@@ -237,15 +249,16 @@ button: "left" (default), "right", or "middle". Defaults to left so the field isPlatforms that cannot address another display reject it explicitly ratherthan silently changing coordinate spaces.- `window_id` (integer, optional)-- `x` (number, optional)-- `y` (number, optional)+- `x` (number, optional): Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:"desktop" (or coordinate_frame:"desktop").+- `y` (number, optional): Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x.### `double_click`-Double-click at (x,y) or an element_index (AT-SPI bounds) via XSendEvent. No focus steal. Provide either (window_id + x/y) or (pid + element_index). After a zoom call, pass from_zoom=true to auto-translate zoom-image coords.+Double-click at (x,y) or an element_index (AT-SPI bounds). Background delivery on X11 sends real button events from the session's virtual master pointer (path mpx_pointer); a point covered by another application's window is refused (target_occluded), one covered by this app's own dialog is retargeted to it (retargeted_to). No focus steal. Provide either (window_id + x/y) or (pid + element_index). After a zoom call, pass from_zoom=true to auto-translate zoom-image coords.**Arguments:**+- `coordinate_frame` (string, optional): Frame of x/y (and from_x/from_y/to_x/to_y). Default "window": window-local screenshot pixels as returned by get_window_state. "desktop": full-screen pixels as returned by get_desktop_state, translated to the target window. Passing scope:"desktop" together with pid/window_id means the same thing.- `cursor_id` (string, optional): Optional multi-cursor instance id. Default: 'default'.- `delivery_mode` (string, optional): Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface. default: `"background"`- `element_index` (integer, optional): Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.@@ -255,8 +268,8 @@ Double-click at (x,y) or an element_index (AT-SPI bounds) via XSendEvent. No foc- `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.- `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed.- `window_id` (integer, optional)-- `x` (number, optional)-- `y` (number, optional)+- `x` (number, optional): Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:"desktop" (or coordinate_frame:"desktop").+- `y` (number, optional): Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x.```json{"pid":844}@@ -264,10 +277,11 @@ Double-click at (x,y) or an element_index (AT-SPI bounds) via XSendEvent. No foc### `right_click`-Right-click at (x,y) or an element_index (AT-SPI bounds) via XSendEvent. No focus steal. Provide either (window_id + x/y) or (pid + element_index). After a zoom call, pass from_zoom=true to auto-translate zoom-image coords.+Right-click at (x,y) or an element_index (AT-SPI bounds). Background delivery on X11 sends real button events from the session's virtual master pointer (path mpx_pointer); a point covered by another application's window is refused (target_occluded), one covered by this app's own dialog is retargeted to it (retargeted_to), one outside the window is refused (point_outside_window; x/y are window-local pixels, the result reports window_point / screen_point). No focus steal. When a context menu opens, the result names it (`popup: {window_id, bounds, title}`): call get_window_state(pid, window_id=<that id>) to index its menu items and click them by element_index. Provide either (window_id + x/y) or (pid + element_index). After a zoom call, pass from_zoom=true to auto-translate zoom-image coords.**Arguments:**+- `coordinate_frame` (string, optional): Frame of x/y (and from_x/from_y/to_x/to_y). Default "window": window-local screenshot pixels as returned by get_window_state. "desktop": full-screen pixels as returned by get_desktop_state, translated to the target window. Passing scope:"desktop" together with pid/window_id means the same thing.- `cursor_id` (string, optional): Optional multi-cursor instance id. Default: 'default'.- `delivery_mode` (string, optional): Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface. default: `"background"`- `element_index` (integer, optional): Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.@@ -278,8 +292,8 @@ Right-click at (x,y) or an element_index (AT-SPI bounds) via XSendEvent. No focu- `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.- `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed.- `window_id` (integer, optional)-- `x` (number, optional)-- `y` (number, optional)+- `x` (number, optional): Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:"desktop" (or coordinate_frame:"desktop").+- `y` (number, optional): Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x.```json{"pid":844}@@ -287,16 +301,17 @@ Right-click at (x,y) or an element_index (AT-SPI bounds) via XSendEvent. No focu### `drag`-Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels via XSendEvent (ButtonPress + MotionNotify × steps + ButtonRelease). duration_ms (default 500), steps (default 20). No focus steal.+Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels. Background delivery on X11 is one real held gesture on the session's virtual master pointer (path mpx_pointer: press, interpolated motion, release) — it lands on GTK/VCL/Qt/Chromium; a point covered by another application's window is refused (target_occluded), one covered by this app's own dialog is retargeted to it. Headless servers fall back to XSendEvent. duration_ms (default 500), steps (default 20). No focus steal.**Arguments:**- `button` (string, optional): Mouse button. Default "left".+- `coordinate_frame` (string, optional): Frame of x/y (and from_x/from_y/to_x/to_y). Default "window": window-local screenshot pixels as returned by get_window_state. "desktop": full-screen pixels as returned by get_desktop_state, translated to the target window. Passing scope:"desktop" together with pid/window_id means the same thing.- `cursor_id` (string, optional): Optional multi-cursor instance id. Default: 'default'.- `delivery_mode` (string, optional): Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface. default: `"background"`- `duration_ms` (integer, optional): Total drag duration. Default: 500. range: 0–10000-- `from_x` (number, required)-- `from_y` (number, required)+- `from_x` (number, required): Drag start X: window-local pixels of the target window's own get_window_state screenshot; pass scope:"desktop" for get_desktop_state pixels.+- `from_y` (number, required): Drag start Y (same frame as from_x).- `from_zoom` (boolean, optional)- `modifier` (array of string, optional): Modifier keys held during the action: cmd, shift, option/alt, ctrl.- `pid` (integer, optional)@@ -308,8 +323,8 @@ Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local`display_id="primary"` is the portable desktop target in this release.Platforms that cannot address another display reject it explicitly ratherthan silently changing coordinate spaces.-- `to_x` (number, required)-- `to_y` (number, required)+- `to_x` (number, required): Drag end X (same frame as from_x).+- `to_y` (number, required): Drag end Y (same frame as from_x).- `window_id` (integer, optional): Target window XID. Required.```json@@ -322,6 +337,7 @@ Type text to a window via XSendEvent (KeyPress/KeyRelease). No focus steal.**Arguments:**+- `coordinate_frame` (string, optional): Frame of x/y (and from_x/from_y/to_x/to_y). Default "window": window-local screenshot pixels as returned by get_window_state. "desktop": full-screen pixels as returned by get_desktop_state, translated to the target window. Passing scope:"desktop" together with pid/window_id means the same thing.- `delivery_mode` (string, optional): Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface. default: `"background"`- `element_index` (integer, optional): Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it.@@ -336,8 +352,8 @@ Platforms that cannot address another display reject it explicitly ratherthan silently changing coordinate spaces.- `text` (string, required)- `window_id` (integer, optional)-- `x` (number, optional): Screenshot-pixel X of the field to type into — the element px action form. Pass x,y (no element_index) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the AX path can't reach. Read straight off the get_window_state PNG, same convention as click.-- `y` (number, optional): Screenshot-pixel Y of the field (see x).+- `x` (number, optional): Pixel X of the field to type into — the element px action form. Pass x,y (no element_index) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the AX path can't reach. Window-local pixels of the target window's own get_window_state screenshot by default (same convention as click); for get_desktop_state pixels pass scope:"desktop" (or coordinate_frame:"desktop") — a bare pid+x/y is otherwise misinterpreted as window-local and can focus the wrong widget.+- `y` (number, optional): Pixel Y of the field (see x).```json{"text":"hello"}@@ -349,6 +365,7 @@ Press a key via XSendEvent to a window. No focus steal.**Arguments:**+- `coordinate_frame` (string, optional): Frame of x/y (and from_x/from_y/to_x/to_y). Default "window": window-local screenshot pixels as returned by get_window_state. "desktop": full-screen pixels as returned by get_desktop_state, translated to the target window. Passing scope:"desktop" together with pid/window_id means the same thing.- `delivery_mode` (string, optional): Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface. default: `"background"`- `element_index` (integer, optional): Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it.@@ -364,8 +381,8 @@ Press a key via XSendEvent to a window. No focus steal.Platforms that cannot address another display reject it explicitly ratherthan silently changing coordinate spaces.- `window_id` (integer, optional)-- `x` (number, optional): Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_index.-- `y` (number, optional): Screenshot-pixel Y (see x).+- `x` (number, optional): Pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_index. Window-local pixels by default (same convention as click); for get_desktop_state pixels pass scope:"desktop" (or coordinate_frame:"desktop").+- `y` (number, optional): Pixel Y (see x).```json{"key":"return"}@@ -373,7 +390,7 @@ than silently changing coordinate spaces.### `hotkey`-Press a combination of keys simultaneously, e.g. ["ctrl","c"] for Copy. Sent via XSendEvent directly to the target pid; target does NOT need to be frontmost.+Press a combination of keys simultaneously, e.g. ["ctrl","c"] for Copy. Delivered to the target window without focus steal (real key events from a virtual master keyboard on X11); target does NOT need to be frontmost. Window-manager chords cannot reach the WM that way: alt+F4 on a window target closes that window through the WM (_NET_CLOSE_WINDOW, effect confirmed when it unmaps); super+*, alt+tab, ctrl+alt+* and other WM bindings are refused with code wm_chord_unavailable — use bring_to_front / list_windows / kill_app or delivery_mode:"foreground" instead.**Arguments:**@@ -409,11 +426,11 @@ Set value of an AT-SPI element via SetValue action.- `pid` (integer, required)- `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.- `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed.-- `value` (string, required)+- `value` (string or number, required): New value. Written through AT-SPI EditableText/Value when the element exposes them; otherwise (GTK2 spin scales, VCL spin buttons) the field is clicked with the session's real pointer, its text selected and replaced through the virtual keyboard and committed with Tab, then read back.- `window_id` (integer, optional): Required when element_index is used; optional when element_token is supplied (the token carries it).```json-{"pid":844,"value":"42"}+{"pid":844,"value":"value"}```### `scroll`@@ -424,6 +441,7 @@ Scroll the target pid's focused region via XSendEvent Button4/5. direction requi- `amount` (integer, optional): range: 1–50- `by` (string, optional)+- `coordinate_frame` (string, optional): Frame of x/y (and from_x/from_y/to_x/to_y). Default "window": window-local screenshot pixels as returned by get_window_state. "desktop": full-screen pixels as returned by get_desktop_state, translated to the target window. Passing scope:"desktop" together with pid/window_id means the same thing.- `cursor_id` (string, optional): Optional multi-cursor instance id. Default: 'default'.- `delivery_mode` (string, optional): Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface. default: `"background"`- `direction` (string, required)@@ -797,13 +815,14 @@ Press and hold a mouse button at (x,y) via background X11 delivery. Does not rel**Arguments:**- `button` (string, optional): Mouse button. Default "left".+- `coordinate_frame` (string, optional): Frame of x/y (and from_x/from_y/to_x/to_y). Default "window": window-local screenshot pixels as returned by get_window_state. "desktop": full-screen pixels as returned by get_desktop_state, translated to the target window. Passing scope:"desktop" together with pid/window_id means the same thing.- `cursor_id` (string, optional): Optional multi-cursor instance id. Default: 'default'.- `from_zoom` (boolean, optional): Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space.- `pid` (integer, required)- `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. When both are present, session takes precedence over cursor_id.- `window_id` (integer, required)-- `x` (number, required)-- `y` (number, required)+- `x` (number, required): Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:"desktop" (or coordinate_frame:"desktop").+- `y` (number, required): Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x.```json{"pid":844,"window_id":10725,"x":100,"y":200}@@ -822,8 +841,8 @@ Move a previously-held mouse button to a new point via background X11 delivery.- `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. When both are present, session takes precedence over cursor_id.- `steps` (integer, optional): Intermediate MotionNotify events. Default: 20. range: 1–200- `window_id` (integer, optional)-- `x` (number, required)-- `y` (number, required)+- `x` (number, required): Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:"desktop" (or coordinate_frame:"desktop").+- `y` (number, required): Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x.```json{"x":100,"y":200}@@ -835,13 +854,14 @@ Release a previously-held mouse button via background X11 delivery. If x/y are o**Arguments:**+- `coordinate_frame` (string, optional): Frame of x/y (and from_x/from_y/to_x/to_y). Default "window": window-local screenshot pixels as returned by get_window_state. "desktop": full-screen pixels as returned by get_desktop_state, translated to the target window. Passing scope:"desktop" together with pid/window_id means the same thing.- `cursor_id` (string, optional): Optional multi-cursor instance id. Default: 'default'.- `from_zoom` (boolean, optional): Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space.- `pid` (integer, optional)- `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. When both are present, session takes precedence over cursor_id.- `window_id` (integer, optional)-- `x` (number, optional)-- `y` (number, optional)+- `x` (number, optional): Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:"desktop" (or coordinate_frame:"desktop").+- `y` (number, optional): Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x.### `parallel_mouse_drag`diff --git a/docs/content/docs/reference/cua-driver/mcp-tools-windows.mdx b/docs/content/docs/reference/cua-driver/mcp-tools-windows.mdxindex e3ec87497f..305785aafb 100644--- a/docs/content/docs/reference/cua-driver/mcp-tools-windows.mdx+++ b/docs/content/docs/reference/cua-driver/mcp-tools-windows.mdx@@ -104,6 +104,7 @@ Windows requires no special permissions.- `query` (string, optional): Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count.- `screenshot_out_file` (string, optional): When set, write the PNG to this file path instead of embedding base64 in the response. The structured output will contain `screenshot_file_path` instead.- `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.+- `timeout_ms` (integer, optional): Wall-clock budget in milliseconds for the accessibility-tree walk (default 1000, min 100, max 120000). Bounds the WHOLE walk. When the budget runs out the tool returns the PARTIAL tree it has, flagged with `truncated: true`, `truncation_reason`, `nodes_visited`, `nodes_pending` and `elements_complete: false`; retry with a larger value (e.g. 5000) or narrow with `query` / `max_depth`. default: `1000`; range: 100–120000- `window_id` (integer, required): HWND of the target window. Must belong to `pid`. Enumerate via `list_windows` or read from `launch_app`'s `windows` array.```json@@ -120,10 +121,11 @@ For the full UIA subtree of a single window (with interactive element indices yo### `get_desktop_state`-Capture the full display in true screen pixels with no downscale. Use its native-size PNG as the coordinate source for actions whose target is {kind:"desktop",display_id:"primary"}. Returns the true screen size. Vision-only: no UIA tree walk.+Capture the full display in true screen pixels, full size unless `max_image_dimension` caps it. Use its PNG as the coordinate source for actions whose target is {kind:"desktop",display_id:"primary"}. Returns the true screen size. Vision-only: no UIA tree walk.**Arguments:**+- `max_image_dimension` (integer, optional): Optional long-edge cap for the returned PNG, in pixels (aspect ratio preserved). Omitted or 0 returns the full-size capture. When the cap downsizes the image, the response reports `screenshot_original_width/height`, and x/y read off it for this session's later scope:"desktop" actions (or passed with its `capture_id`) are mapped back to the full-size frame automatically. range: 0–unbounded- `screenshot_out_file` (string, optional): Write PNG here instead of base64.- `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdxindex eafe2902c1..c3abd79af0 100644--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx@@ -69,7 +69,7 @@ PREFERRED CONSUMERS read `structuredContent.elements` (one entry per indexed rowAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces: Electron echo-confirms, Catalyst null values, virtualized off-viewport rows with `h:1` frames). You choose the modality at ACTION time, not here: an element ax action (pass `element_index`/`element_token` → the accessibility rung) or an element px action (pass `x`,`y` → the pixel rung, read straight off this screenshot). `capture_mode` is deprecated and ignored. Pass `include_screenshot:false` to skip the grab and get the tree only — the cheap path when you're just re-indexing before an element ax action.-The mirror image: pass `include_accessibility_tree:false` to SKIP the AX walk entirely (the expensive part, up to 20 s) and return just the screenshot plus window metadata — `window_bounds`, `screenshot_scale`, `screenshot_width`/`screenshot_height`, `app_name`, and `window_title` — the capture-only path for rendering a live window preview / picture-in-picture without paying for perception. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error (nothing to return). Optional `max_image_dimension` overrides the configured screenshot long-edge limit for this call; use 0 for native resolution. The legacy `max_dimension` remains a tighter cap for compatibility.+The mirror image: pass `include_accessibility_tree:false` to SKIP the AX walk entirely (the expensive part, bounded by timeout_ms) and return just the screenshot plus window metadata — `window_bounds`, `screenshot_scale`, `screenshot_width`/`screenshot_height`, `app_name`, and `window_title` — the capture-only path for rendering a live window preview / picture-in-picture without paying for perception. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error (nothing to return). Optional `max_image_dimension` overrides the configured screenshot long-edge limit for this call; use 0 for native resolution. The legacy `max_dimension` remains a tighter cap for compatibility.The snapshot is SCOPED to `window_id`: a window_id that no longer exists is refused with `window_id_not_found`, and one owned by another process is refused with `window_owner_pid_mismatch` naming the real `owner_pid` to retry with (macOS hosts a sandboxed app's Open/Save panel out-of-process, so its window belongs to the panel service, not the app). If the window is live under this pid but its accessibility surface can't be resolved, the tree comes back EMPTY with `degraded_reason: ax_window_unresolved` and the screenshot of the requested window — act by pixel there. This tool never returns another surface's elements under your window_id. Before exposing a screenshot, its raw dimensions are validated as a coherent 1x/2x representation of the requested WindowServer bounds. `px_frame_mismatch` or `px_capture_unavailable` omits an unprovable screenshot/pixel frame instead of guessing a transform; the truthful AX payload remains available.@@ -80,7 +80,7 @@ Optional `max_elements` / `max_depth` bound the AX walk to mitigate context-wind**Arguments:**- `capture_mode` (string, optional): DEPRECATED and ignored. get_window_state always returns BOTH the element tree and a screenshot — ground on both. The modality is chosen at action time by how you address the target: an element ax action (element_index/element_token) or an element px action (x,y). Any value (including the old "som"/"screenshot" aliases) is accepted but has no effect.-- `include_accessibility_tree` (boolean, optional): Default true — walk the AX tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AX walk entirely (the expensive part, up to 20 s) and return just the screenshot plus window metadata (bounds, scale, app_name, window_title) — the capture-only path for rendering a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return).+- `include_accessibility_tree` (boolean, optional): Default true — walk the AX tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AX walk entirely (the expensive part, bounded by timeout_ms) and return just the screenshot plus window metadata (bounds, scale, app_name, window_title) — the capture-only path for rendering a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return).- `include_screenshot` (boolean, optional): Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return the tree only (the cheap path when you're just re-indexing before an element ax action; saves the image tokens + screen-grab latency). screenshot_out_file still forces a capture to disk.- `max_depth` (integer, optional): Cap on the AX-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower this for deep menu/Electron trees. range: 1–unbounded- `max_dimension` (integer, optional): Optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved) — the cheap path for a small preview / thumbnail. Applied on top of the session/global max_image_dimension ceiling; the tighter of the two wins. Omit for the configured default. range: 1–unbounded@@ -90,6 +90,7 @@ Optional `max_elements` / `max_depth` bound the AX walk to mitigate context-wind- `query` (string, optional): Case-insensitive filter for tree_markdown and structured elements. Returns matching actionable rows plus their actionable ancestors without renumbering element_index values.- `screenshot_out_file` (string, optional): When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output will contain screenshot_file_path instead.- `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.+- `timeout_ms` (integer, optional): Wall-clock budget in milliseconds for the accessibility-tree walk (default 1000, min 100, max 120000). Bounds the WHOLE walk. When the budget runs out the tool returns the PARTIAL tree it has, flagged with `truncated: true`, `truncation_reason`, `nodes_visited`, `nodes_pending` and `elements_complete: false`; retry with a larger value (e.g. 5000) or narrow with `query` / `max_depth`. default: `1000`; range: 100–120000- `window_id` (integer, required): Target window ID from list_windows.```json@@ -106,10 +107,11 @@ For the full AX subtree of a single window (with interactive element indices you### `get_desktop_state`-Capture the full display in true screen pixels with no downscale. Use its native-size PNG as the coordinate source for actions whose target is {kind:"desktop",display_id:"primary"}. Returns the true screen size and backing scale factor. Vision-only: no AX tree walk.+Capture the full display in true screen pixels, full size unless `max_image_dimension` caps it. Use its PNG as the coordinate source for actions whose target is {kind:"desktop",display_id:"primary"}. Returns the true screen size and backing scale factor. Vision-only: no AX tree walk.**Arguments:**+- `max_image_dimension` (integer, optional): Optional long-edge cap for the returned PNG, in pixels (aspect ratio preserved). Omitted or 0 returns the full-size capture. When the cap downsizes the image, the response reports `screenshot_original_width/height`, and x/y read off it for this session's later scope:"desktop" actions (or passed with its `capture_id`) are mapped back to the full-size frame automatically. range: 0–unbounded- `screenshot_out_file` (string, optional): Write PNG here instead of base64.- `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.diff --git a/libs/cua-driver/contract/manifest.json b/libs/cua-driver/contract/manifest.jsonindex 09e8c46a2b..9bfe1d4aa6 100644--- a/libs/cua-driver/contract/manifest.json+++ b/libs/cua-driver/contract/manifest.json@@ -211,6 +211,27 @@],"type": "string"},+ "error": {+ "additionalProperties": false,+ "properties": {+ "code": {+ "type": "string"+ },+ "hint": {+ "type": [+ "string",+ "null"+ ]+ }+ },+ "required": [+ "code"+ ],+ "type": [+ "object",+ "null"+ ]+ },"escalation": {"additionalProperties": false,"properties": {@@ -247,6 +268,12 @@"items": {"additionalProperties": false,"properties": {+ "detail": {+ "type": [+ "string",+ "null"+ ]+ },"kind": {"enum": ["value_readback",@@ -275,6 +302,12 @@"trusted_input"],"type": "string"+ },+ "summary": {+ "type": [+ "string",+ "null"+ ]}},"required": [@@ -614,6 +647,27 @@],"type": "string"},+ "error": {+ "additionalProperties": false,+ "properties": {+ "code": {+ "type": "string"+ },+ "hint": {+ "type": [+ "string",+ "null"+ ]+ }+ },+ "required": [+ "code"+ ],+ "type": [+ "object",+ "null"+ ]+ },"escalation": {"additionalProperties": false,"properties": {@@ -650,6 +704,12 @@"items": {"additionalProperties": false,"properties": {+ "detail": {+ "type": [+ "string",+ "null"+ ]+ },"kind": {"enum": ["value_readback",@@ -678,6 +738,12 @@"trusted_input"],"type": "string"+ },+ "summary": {+ "type": [+ "string",+ "null"+ ]}},"required": [@@ -1153,6 +1219,11 @@"input_schema": {"additionalProperties": false,"properties": {+ "max_image_dimension": {+ "description": "Optional long-edge cap for the returned PNG, in pixels. Omitted or 0\nreturns the full-size capture. When the cap downsizes the image,\ndesktop-scope x/y taken from it are mapped back automatically.",+ "minimum": 0,+ "type": "integer"+ },"screenshot_out_file": {"description": "Write the PNG here instead of returning base64.","type": "string"@@ -1538,6 +1609,12 @@"session": {"type": "string"},+ "timeout_ms": {+ "description": "Wall-clock budget for the accessibility walk in milliseconds\n(default 1000 on every platform). A walk that runs out returns a\npartial tree flagged `truncated` rather than failing.",+ "maximum": 120000,+ "minimum": 100,+ "type": "integer"+ },"window_id": {"minimum": 1,"type": "integer"@@ -2000,6 +2077,27 @@],"type": "string"},+ "error": {+ "additionalProperties": false,+ "properties": {+ "code": {+ "type": "string"+ },+ "hint": {+ "type": [+ "string",+ "null"+ ]+ }+ },+ "required": [+ "code"+ ],+ "type": [+ "object",+ "null"+ ]+ },"escalation": {"additionalProperties": false,"properties": {@@ -2036,6 +2134,12 @@"items": {"additionalProperties": false,"properties": {+ "detail": {+ "type": [+ "string",+ "null"+ ]+ },"kind": {"enum": ["value_readback",@@ -2064,6 +2168,12 @@"trusted_input"],"type": "string"+ },+ "summary": {+ "type": [+ "string",+ "null"+ ]}},"required": [@@ -2171,6 +2281,27 @@],"type": "string"},+ "error": {+ "additionalProperties": false,+ "properties": {+ "code": {+ "type": "string"+ },+ "hint": {+ "type": [+ "string",+ "null"+ ]+ }+ },+ "required": [+ "code"+ ],+ "type": [+ "object",+ "null"+ ]+ },"escalation": {"additionalProperties": false,"properties": {@@ -2207,6 +2338,12 @@"items": {"additionalProperties": false,"properties": {+ "detail": {+ "type": [+ "string",+ "null"+ ]+ },"kind": {"enum": ["value_readback",@@ -2235,6 +2372,12 @@"trusted_input"],"type": "string"+ },+ "summary": {+ "type": [+ "string",+ "null"+ ]}},"required": [@@ -2774,6 +2917,27 @@],"type": "string"},+ "error": {+ "additionalProperties": false,+ "properties": {+ "code": {+ "type": "string"+ },+ "hint": {+ "type": [+ "string",+ "null"+ ]+ }+ },+ "required": [+ "code"+ ],+ "type": [+ "object",+ "null"+ ]+ },"escalation": {"additionalProperties": false,"properties": {@@ -2810,6 +2974,12 @@"items": {"additionalProperties": false,"properties": {+ "detail": {+ "type": [+ "string",+ "null"+ ]+ },"kind": {"enum": ["value_readback",@@ -2838,6 +3008,12 @@"trusted_input"],"type": "string"+ },+ "summary": {+ "type": [+ "string",+ "null"+ ]}},"required": [@@ -3494,6 +3670,27 @@],"type": "string"},+ "error": {+ "additionalProperties": false,+ "properties": {+ "code": {+ "type": "string"+ },+ "hint": {+ "type": [+ "string",+ "null"+ ]+ }+ },+ "required": [+ "code"+ ],+ "type": [+ "object",+ "null"+ ]+ },"escalation": {"additionalProperties": false,"properties": {@@ -3530,6 +3727,12 @@"items": {"additionalProperties": false,"properties": {+ "detail": {+ "type": [+ "string",+ "null"+ ]+ },"kind": {"enum": ["value_readback",@@ -3558,6 +3761,12 @@"trusted_input"],"type": "string"+ },+ "summary": {+ "type": [+ "string",+ "null"+ ]}},"required": [@@ -3731,6 +3940,27 @@],"type": "string"},+ "error": {+ "additionalProperties": false,+ "properties": {+ "code": {+ "type": "string"+ },+ "hint": {+ "type": [+ "string",+ "null"+ ]+ }+ },+ "required": [+ "code"+ ],+ "type": [+ "object",+ "null"+ ]+ },"escalation": {"additionalProperties": false,"properties": {@@ -3767,6 +3997,12 @@"items": {"additionalProperties": false,"properties": {+ "detail": {+ "type": [+ "string",+ "null"+ ]+ },"kind": {"enum": ["value_readback",@@ -3795,6 +4031,12 @@"trusted_input"],"type": "string"+ },+ "summary": {+ "type": [+ "string",+ "null"+ ]}},"required": [@@ -4215,6 +4457,27 @@],"type": "string"},+ "error": {+ "additionalProperties": false,+ "properties": {+ "code": {+ "type": "string"+ },+ "hint": {+ "type": [+ "string",+ "null"+ ]+ }+ },+ "required": [+ "code"+ ],+ "type": [+ "object",+ "null"+ ]+ },"escalation": {"additionalProperties": false,"properties": {@@ -4251,6 +4514,12 @@"items": {"additionalProperties": false,"properties": {+ "detail": {+ "type": [+ "string",+ "null"+ ]+ },"kind": {"enum": ["value_readback",@@ -4279,6 +4548,12 @@"trusted_input"],"type": "string"+ },+ "summary": {+ "type": [+ "string",+ "null"+ ]}},"required": [@@ -4568,6 +4843,27 @@],"type": "string"},+ "error": {+ "additionalProperties": false,+ "properties": {+ "code": {+ "type": "string"+ },+ "hint": {+ "type": [+ "string",+ "null"+ ]+ }+ },+ "required": [+ "code"+ ],+ "type": [+ "object",+ "null"+ ]+ },"escalation": {"additionalProperties": false,"properties": {@@ -4604,6 +4900,12 @@"items": {"additionalProperties": false,"properties": {+ "detail": {+ "type": [+ "string",+ "null"+ ]+ },"kind": {"enum": ["value_readback",@@ -4632,6 +4934,12 @@"trusted_input"],"type": "string"+ },+ "summary": {+ "type": [+ "string",+ "null"+ ]}},"required": [diff --git a/libs/cua-driver/docs/action-result-contract.md b/libs/cua-driver/docs/action-result-contract.mdindex 409dbfcc7e..3acb0093c7 100644--- a/libs/cua-driver/docs/action-result-contract.md+++ b/libs/cua-driver/docs/action-result-contract.md@@ -19,11 +19,17 @@ Every successful action returns a closed `structuredContent` object:"effect": "confirmed","route": "accessibility","delivery": {"mode": "background"},- "evidence": [{"kind": "value_readback"}]+ "evidence": [{"kind": "value_readback", "detail": "value read back as 42"}],+ "summary": "Set value to 42 on spin button 'Width' (delivery_mode=background, path=x11_atspi)"}```-`effect` and `route` are required.+`effect` and `route` are required. `summary` repeats the result's text+content (resolved `window_point` / `screen_point`, the element hit, popups+that opened and the call that targets them, focus outcome, retargeting) so a+client that hands the model only `structuredContent` still sees everything a+raw MCP client reads from `content`. `evidence[].detail` is the readback the+evidence rests on. A `refused` result adds `error: {code, hint}`.| Field | Values || --- | --- |@@ -31,6 +37,9 @@ Every successful action returns a closed `structuredContent` object:| `route` | `accessibility`, `synthetic_events`, `global_input`, `dom`, `trusted_input` || `delivery.mode` | `background`, `foreground`, `not_applicable`, `unknown` || `evidence[].kind` | `value_readback`, `window_change` |+| `evidence[].detail` | optional human readback behind the evidence |+| `summary` | optional; the result text, verbatim |+| `error.code` / `error.hint` | only with `effect: refused` || `escalation.target` | `pixel`, `foreground`, `page`, `session` || `escalation.reason` | `route_unavailable`, `delivery_failed`, `effect_unconfirmed`, `suspected_noop`, `permission_required` |@@ -45,15 +54,17 @@ The action-result tools are:Other mutating tools such as application launch, window activation, browsernavigation, dialogs, uploads, and downloads retain their own typed results.-The contract is deliberately closed. It does not echo selectors, coordinates,-scope, targets, platform transport names, diagnostic pointers, or the old-`verified` boolean.+The contract is deliberately closed. It does not echo selectors, request+coordinates, scope, targets, platform transport names, diagnostic pointers, or+the old `verified` boolean as fields; `summary` is prose for the agent, not a+machine-readable echo of the request.The invariants are:- `confirmed` has publishable readback or window-change evidence;- `partial` has `delivery.delivered_count`;-- `refused` has neither delivery nor evidence.+- `refused` has neither delivery nor evidence;+- `error` appears only on `refused`.## Window target resolutiondiff --git a/libs/cua-driver/docs/research/2026-09-osworld/01-eval-stock-driver-vs-pyautogui.md b/libs/cua-driver/docs/research/2026-09-osworld/01-eval-stock-driver-vs-pyautogui.mdnew file mode 100644index 0000000000..ede17afec7--- /dev/null+++ b/libs/cua-driver/docs/research/2026-09-osworld/01-eval-stock-driver-vs-pyautogui.md@@ -0,0 +1,66 @@+# Does driving OSWorld through cua-driver drop the score? (Haiku 4.5, 2026-09-09)++Claim under test: "cua-driver dropped OSWorld scores from ~80% to under 10%."++## Setup++- **Tasks:** 15 tasks from OSWorld `evaluation_examples/test_small.json` (upstream `xlang-ai/OSWorld` at HEAD on 2026-09-09): `os` (1), `gimp` (2), `vlc` (2), `vs_code` (3), `libreoffice_calc` (3), `libreoffice_writer` (2), `libreoffice_impress` (2). Excluded: `chrome`, `thunderbird`, `multi_apps` (need accounts, proxies, or internet) and `os/5812b315` (its evaluator `apt-get install`s inside the guest, and the local VM has no outbound network).+- **Environment:** the OSWorld golden image patched with cua-driver 0.25.0 (see README), one fresh qcow2 overlay per episode, bare-metal QEMU/KVM under WSL2, 4 vCPU / 8 GB. Task setup runs through OSWorld's own `SetupController`, scoring through OSWorld's own `metrics`/`getters` (a 40-line port of `DesktopEnv.evaluate`).+- **Policy:** Claude Haiku 4.5 (`claude-haiku-4-5-20251001`) via the Claude Agent SDK (bundled Claude Code CLI, OAuth token), `permission_mode=bypassPermissions`, no built-in tools, same system prompt skeleton, **25-action cap** and 15-minute episode timeout in every condition. Each task run once per condition (n=1; this is a smoke-scale eval, not a benchmark number).+- **Conditions:**+ - `noop`: task setup, no agent, evaluate. Control for evaluator leniency.+ - `pyautogui`: the classic OSWorld loop. Tools: `screenshot` (1280x720 JPEG of the 1920x1080 screen), `pyautogui(code)` (runs in the guest via the OSWorld server's `/execute`, coordinates scaled 1.5x), `done`, `fail`.+ - `cuadriver`: the stock cua-driver MCP catalog (60 tools) over Streamable HTTP, plus `done`/`fail`.+ - `cuadriver_curated`: same, but 42 of the 60 tools hidden (`disallowed_tools`), leaving 18 desktop tools: `get_desktop_state`, `get_screen_size`, `list_windows`, `list_apps`, `get_window_state`, `launch_app`, `bring_to_front`, `click`, `double_click`, `right_click`, `type_text`, `press_key`, `hotkey`, `scroll`, `drag`, `zoom`, `invoke_menu`, `set_value`.++Harness: `eval/harness.py` in the session scratchpad (copied to `~/osworld-eval` in WSL); results in `~/osworld-eval/results/*.jsonl`.++## Results++| Condition | Tasks solved | Mean score | Avg actions | Haiku spend |+| --- | --- | --- | --- | --- |+| `noop` (control) | 0 / 15 (0 %) | 0.00 | 0 | $0 |+| `pyautogui` (classic OSWorld loop) | **12 / 15 (80 %)** | 0.87 | 9.9 | $1.26 |+| `cuadriver` (stock 60-tool catalog) | **2 / 15 (13 %)** | 0.13 | 22.8 | $3.71 |+| `cuadriver_curated` (18 desktop tools) | **0 / 15 (0 %)** | 0.07 | 21.3 | $3.28 |++Per task (score, actions, how the episode ended; `!` = episode timeout or evaluator timeout because the guest froze):++```text+domain task | pyautogui | cuadriver | cuadriver_curated+gimp 554785e9 | 1.00 6 DONE | 1.00 28 (cap) | 0.00 28 (cap)+gimp 7a4deb26 | 1.00 8 DONE | 0.00 28 (cap) | 0.00 8 timeout !+libreoffice_calc 357ef137 | 1.00 5 DONE | 0.00 13 timeout ! | 0.00 14 eval-timeout !+libreoffice_calc 42e0a640 | 1.00 10 DONE | 0.00 18 FAIL | 0.00 16 timeout !+libreoffice_calc abed40dc | 1.00 11 DONE | 0.00 14 FAIL ! | 0.00 12 FAIL+libreoffice_impress 550ce7e7 | 1.00 9 DONE | 0.00 28 (cap) | 0.00 28 (cap)+libreoffice_impress 5d901039 | 0.00 25 (cap) | 0.00 28 (cap) | 0.00 28 (cap)+libreoffice_writer 0810415c | 1.00 7 DONE | 0.00 28 (cap) | 0.00 28 (cap)+libreoffice_writer 0a0faba3 | 0.00 25 DONE | 0.00 15 FAIL | 0.00 28 (cap)+os 5ea617a3 | 1.00 4 DONE | 0.00 8 timeout ! | 0.00 15 timeout !+vlc 59f21cfb | 1.00 8 DONE | 0.00 28 (cap) | 0.00 18 DONE+vlc 8f080098 | 0.99 14 DONE | 0.00 28 (cap) | 0.99 19 (cap)+vs_code 0ed39f63 | 1.00 6 DONE | 0.00 28 (cap) | 0.00 28 (cap)+vs_code 276cc624 | 1.00 7 DONE | 1.00 22 DONE | 0.00 23 DONE+vs_code 53ad5833 | 1.00 3 DONE | 0.00 28 (cap) | 0.00 27 DONE+```++Raw records: `eval/*.jsonl`; harness and diagnostics: `eval/harness.py`, `eval/diag_click.py`, `eval/diag_typetext.py`.++**So the claim reproduces.** Same model, same tasks, same image, same step budget: 80 % through the classic screenshot + pyautogui loop, 13 % through the stock cua-driver MCP, 0 % through a trimmed catalog. The driver conditions also cost about 3x more per task (long tool schemas, 2.3x more actions per episode, and turns spent recovering from stalls).++## Why the driver conditions lose++Three mechanisms, each verified outside the eval with a scripted test on a fresh VM (`diag_click.py`, `diag_typetext.py`; screenshots in `1_cua_click_bg.png` ... `4_pyautogui_click.png` and `diag-typetext/`):++1. **Default clicks do not operate GTK menus.** In GIMP, `click {pid, x, y}` (background XSendEvent delivery) and `click {..., delivery_mode: "foreground"}` both report "✅ Clicked" and open nothing; only `click {target: {kind: "desktop"}}` or OSWorld's `pyautogui.click` open the Colors menu (pixel-diff 0.30 % / 0.04 % vs 0.54 % / 0.51 %). Haiku's default is the pid form, so in the GIMP episode it clicked the menubar five times, then abandoned menus.+2. **No working text-input route into LibreOffice, and the foreground route freezes the X session.** `type_text {pid}` is refused after 13-26 s ("Background delivery is not available ... no focus-free input backend"), `type_text {target: desktop}` claims success but types nothing, and `type_text {..., delivery_mode: "foreground"}` hangs and freezes the guest display (top-bar clock stops; OSWorld server and MCP endpoint stop answering). Both eval VMs froze this way on the first Calc task (`diag-hang/*.png`), which is why every LibreOffice episode under the driver scores 0.+3. **Stalls and per-call latency.** An AT-SPI element click on Files took 5.5 min to return; after a stalled action, every following call on that window took ~60 s until `health_report` was called. With a 25-action budget and a 15-minute episode cap, a single stall removes a third of the episode.++Secondary, not decisive: the stock catalog is 60 tools (~200 KB of schema) and Haiku spent calls on `page`, `health_report`, `check_permissions`, `list_sessions`, `end_session`/`start_session` while trying to recover; the curated 18-tool variant did not score better, so the catalog size is not what loses the tasks. The model's first click of every episode also targeted `pid: 1` because `get_desktop_state` does not say which pid owns what.++## Caveats++- n=1 per task per condition, 15 tasks, one model. Enough to see a large effect, not to estimate a rate precisely.+- `test_small` skews easy for the pyautogui loop (12/15 is well above OSWorld leaderboard numbers for Haiku-class models); the comparison is within this harness, not against published scores.+- The cua-driver prompt was written once and not tuned; a prompt that says "always use `target: {kind: desktop}` for clicks and `foreground` delivery" would likely recover part of the gap. That would also mean giving up the no-focus contract the driver is designed around.diff --git a/libs/cua-driver/docs/research/2026-09-osworld/02-optimization-log.md b/libs/cua-driver/docs/research/2026-09-osworld/02-optimization-log.mdnew file mode 100644index 0000000000..484e4e720a--- /dev/null+++ b/libs/cua-driver/docs/research/2026-09-osworld/02-optimization-log.md@@ -0,0 +1,91 @@+# cua-driver optimization for OSWorld (branch `feat/osworld-cuadriver-optimization`)++Worktree `O:\projects\cua-osworld\cua-opt`, head `f16420178` (13 commits over `origin/main` at `b4e3caecd`, cua-driver 0.26.1). Not pushed. All test binaries green in the Ubuntu 22.04 build container (`platform-linux` 453, `cua-driver-core` 612, `cua-driver-contract` 40, `schema_consistency_test`, `protocol_schema_test`, `contract_parity`).++Sub-branches merged in: `feat/opt-foreground` (074c5776f..3d5fb913f), `feat/opt-atspi-background` (d8788d348..d68b8356d), then 245bd4517 (merge), dfbe3c93e, 54e276fdd, 197ba7096, f16420178.++## What changed++**Foreground delivery** (`crates/platform-linux/src/input/foreground.rs`, new): x11rb-based `with_x11_foreground_opts`; activate (`_NET_ACTIVE_WINDOW` + `XSetInputFocus`), confirm focus on the active window and the core-focus tree, one re-activation at 40 % of the budget (pointer 800 ms, keyboard 1500 ms), watchdog threads around confirm and post-check, **no restore afterwards**; structured `foreground_unavailable` / `foreground_timeout` codes and `focus_after: target|same_pid|elsewhere`. `type_text` foreground goes straight to activate → optional AT-SPI focus → XTest; no more `focused_is_editable`/Qt5 probing. Element clicks in foreground: activate + XTest at the element centre. Pointer results say "not verified — confirm with a screenshot" and carry `effect: suspected_noop` when focus left the process.++**AT-SPI query engine** (`crates/platform-linux/src/atspi/{native,mod,cache}.rs`, `tools/impl_.rs`): `get_window_state.timeout_ms` (default 1000, 100..120000) bounds retries + walk + bounds under one task-local deadline; every D-Bus call is clamped to the remaining budget; partial trees return `truncated`, `truncation_reason` (`timeout|node_budget|app_unresponsive|app_lookup_timeout|huge_container`), `nodes_visited`, `nodes_pending`, `bounds_complete`, `walk_elapsed_ms`, and a one-line note telling the model to retry with a larger `timeout_ms` or narrow with `query`/`max_depth`; `elements_complete` is truthful. Containers above 2048 children are never expanded (LibreOffice Calc's sheet reports 2^30). Snapshot cache stores element identity + frame, so element clicks/actions/focus reuse it (`perform_action_in`, `focus_element_ref`, `element_bounds_ref`). Input-time AT-SPI budgets: 1.5 s query, 6 s index resolve, 30 s AX action (`ax_timeout`). Focus-event tracker makes the background keyboard refusal instant (0.1 s, was 13-34 s). `list_windows` / `get_desktop_state` text carries `pid=N window_id=M` per window.++**Daemon deadlock fix** (dfbe3c93e): the cursor-overlay arrival await is capped at 5.5 s with a degraded latch; this was the cause of the multi-minute daemon-wide hangs (see PAPERCUTS.md #37).++**Coordinate frames** (a3bccdcea, 4006e4d15, 54e276fdd, 197ba7096, f16420178): `scope: "desktop"` / `coordinate_frame: "desktop"` with a pid means desktop pixels (translated once); `get_desktop_state` now returns a ≤1280-px screenshot with `frame_scale` and scales desktop-frame actions back (the model's vision downsizes 1920-px images, so raw coordinates were ~20 % short); an ambiguous pid resolves to the topmost window under the target point; pid-only keyboard actions fall back to the pid's active window.++## Verification++Swap test (fresh VM, binary hot-swapped, `opt/swaptest.sh` + `swaptest_fg.sh`):++| Probe | Stock 0.25.0 | Branch |+| --- | --- | --- |+| GIMP menubar pixel click, background | nondeterministic | opens menu, 1.6 s |+| GIMP menubar pixel click, foreground | no effect | opens menu (window-local and desktop-scope) |+| GIMP element click (cached ref) | 5.5 min hang, daemon wedged | opens menu, 1.1 s, both modes |+| Calc `type_text` foreground | **X session freeze** | text in A1, 0.4 s, guest alive |+| Calc `type_text` background | refusal after 13-34 s | same refusal in 0.1 s |+| Calc `get_window_state` | 25 s timeout → guest death | 1.2 s (default) / 2.5 s (`timeout_ms: 5000`), labeled partial |++OSWorld subset (5 tasks: gimp x2, vlc, vs_code, libreoffice_calc; Haiku 4.5, 25 actions, harness forces one delivery mode; classic pyautogui baseline 5/5, stock cua-driver 1/5):++| Run | Score | Notes |+| --- | --- | --- |+| foreground, first merged binary | 0 / 5 | no stalls, coordinates still 20 % short |+| foreground, after screenshot scaling | 3 / 5 | GIMP x2 and VLC pass; VS Code wrote the wrong setting, Calc wrong formulas; no driver errors, no call over 10 s on this 5-task probe subset (gimp x2, vlc, vs_code, libreoffice_calc x1) — the larger 15-task run below found `get_window_state` on LibreOffice taking 13-43 s, so this does not hold as a general property of the fix |+| foreground, final binary (fg3) | 3 / 5 | same |+| background, merged | 0 / 5 | no hangs; 42 of 50 errors are the by-design keyboard refusal |++Full 15-task run, foreground mode, final binary (`merged-fg-full`, same tasks/model/budget as EVAL.md):++| Condition | Solved | By domain (solved/total) |+| --- | --- | --- |+| classic pyautogui (baseline) | 12 / 15 (80 %) | os 1/1, gimp 2/2, vlc 2/2, vs_code 3/3, calc 3/3, writer 1/2, impress 1/2 |+| stock cua-driver 0.25.0 | 2 / 15 (13 %) | |+| **branch, foreground** | **8 / 15 (53 %)** | os 1/1, gimp 2/2, vlc 2/2 (one at 0.99), vs_code 3/3, calc 1/3, writer 0/2, impress 0/2 |+| branch, background-only, MPX + uinput (full 15) | 6 / 15 (40 %) | see the MPX section |++Everything outside LibreOffice now matches the pyautogui baseline (8/8). LibreOffice is 1/7 vs 5/7. From the per-call log of the six failures: no hangs, but `get_window_state` on LibreOffice windows still took 13-43 s five times (budget overrun on the bounds/retry path, worth a follow-up), the model burned budget re-calling `get_window_state` (up to 10 times in one episode), invented tools (`triple_click`) and passed numbers as strings (`InputValidationError` in four episodes), and typed wrong formulas; one `pid owns more than one eligible top-level window` refusal remained in the Impress run. Foreground clicks and typing themselves landed.+++## Background delivery via MPX + uinput (commits 17444e862 .. 4c40daa49)++The X11 equivalent of the Hyprland isolated seat: each session lazily creates its own XI2 master pointer+keyboard pair, a uinput keyboard (and pointer) is hot-added by the X server and attached to that pair, `XISetFocus` points the virtual keyboard at the target window, and real evdev events are written through uinput. Result path `mpx_uinput`; the honest refusal remains when `/dev/uinput` is not writable. Lifecycle: retained per session with an idle TTL, removed on `end_session` and by the startup reaper. Fixes found in the loop: thaw the retained virtual pointer after a shield grab, deliver clicks into override-redirect popups without the shield grab, choose a mapped toplevel for pid-only keys, prefer the virtual keyboard over AT-SPI `insert_text`.++Verified in the swap test: Calc typing lands (A1), ctrl+z undo, click-then-type, VS Code typing and hotkeys, GIMP menubar and menu-item clicks, with `XGetInputFocus` and the real pointer position unchanged and no stray `CUA` devices after `end_session`.++Background-only subset (same 5 tasks): 0/5 → 2/5 → 3/5 (`mpx-bg5`: GIMP x2, VS Code). Full 15-task background-only run (`mpx-bg-full`, commit 4c40daa49): **6 / 15 (40 %)** — os 1/1, gimp 2/2, vlc 1/2 (other at 0.99), vs_code 1/3, calc 2/3, writer 0/2, impress 0/2. Stock driver was 2/15 (13 %) with the model free to choose modes; the foreground-only run on this branch is 8/15 (53 %); the pyautogui baseline is 12/15 (80 %).++Image requirement: `build/99-cua-uinput.rules` + `build/uinput.conf` + `usermod -aG input user` (added to `build/build-osworld-cua.sh`).++## Remaining gaps+- Background-only keyboard input into GTK/VCL apps has no focus-free route on this image (PAPERCUTS #41); GIMP spin scales are not AT-SPI editable.+- Why the overlay renderer stops reporting arrival above a GIMP menu is undiagnosed (the cap makes it harmless).+- One stale-window-id `BadWindow` in `desktop_to_window_local` is surfaced, not retried.+- Windows/macOS accept `timeout_ms` in the schema but do not honor it yet.++## Round 8: Qt popups, labels, screenshot scale (branch `feat/opt-r8-qt-keys`)++From the round-7 background traces (VLC 59f21cfb, GIMP 554785e9) and the Calc+regression analysis. Probe: `part8_qt.py` (results `part8-qt*.md`).++| Finding | Root cause | Change |+| --- | --- | --- |+| Keys dropped while a Qt popup (combo list, completer) is open, reported "focus untouched" | The X server drops core key events from any *other* master keyboard to a client holding an active keyboard grab (`IsInterferingGrab`); the virtual master keyboard route is silently lost | `popup_of_pid` + `send_keys_under_popup_grab`: XTest on the core keyboard (the grab routes it to the target), core focus / active window verified after (`path=xtest_core_grab`); refused with `popup_keyboard_grab` when another pid owns the core focus |+| `get_window_state(pid, popup)` returned the main menubar (Audio/Video/…) instead of the combo rows | Frames stayed in screen pixels when the override-redirect window's origin was unavailable, so the menubar "fit" the popup box; only `menu` roles were kept | Popup rectangle is the origin fallback; container-subtree scoping; list/tree/table rows win over menubar entries |+| AT-SPI `Description` never read | — | Read for controls; rendered `(description "Pause")`, `label` falls back to it |+| Four GIMP spin scales all `label:"0.0"` | GtkSpinButton implements AtkText with its number; the text became the name | Value controls are never named after text; LABELLED_BY, description (GIMP keeps "Hue"/"Lightness"/"Saturation" there), sibling label, else `unlabelled` + "3rd of 4 spin buttons in this window" |+| Drop onto VLC reported "3.6% changed" | Guard watches the target pid only | `foreign_window`: title before/after + new windows of the pid under the point, as window_change evidence |+| `parent_index` pointed at a preceding push button | "nearest preceding actionable at lower depth" | Real pre-order ancestor chain |+| Model called `triple_click` | — | `click` documents `count: 2/3`; verified: count 3 selects a whole line in gedit, background and foreground |+| Model's Calc coordinates 0.94x short | The API downsizes images above ~1.15 MP; a 1568x861 window shot reached the model as ~1447x795 | Window screenshots capped at 1.15 MP, element frames scaled to the delivered image, `frame_scale` reported; verified: a click at screenshot pixels selects the same cell as a real click |+| One-row Calc misses found six values later | — | After a pointer press the focused cell is named (`focus: cell D2`) from `object:active-descendant-changed` |++Probe (`part8-qt.md`, `part8-qt-b.md`, `part8-qt-c.md`): VLC Open File end to end+by set_value + accessibility Press (title → "tone.wav - VLC media player"),+combo popup rows listed, Escape under the popup grab delivered with focus+verified unchanged, transport buttons described; GIMP Hue-Saturation spin+buttons named; gedit triple click; Calc 1337x860 (1.15 MP) screenshot with+`frame_scale` 0.955 and cell-accurate clicks with `focus: cell`.++Subset run (`r8-bg-subset`, background, tasks 59f21cfb 554785e9 357ef137 abed40dc): see below.diff --git a/libs/cua-driver/docs/research/2026-09-osworld/03-papercuts.md b/libs/cua-driver/docs/research/2026-09-osworld/03-papercuts.mdnew file mode 100644index 0000000000..400cb32d12--- /dev/null+++ b/libs/cua-driver/docs/research/2026-09-osworld/03-papercuts.md@@ -0,0 +1,65 @@+# Cua SDK papercuts hit while building the OSWorld-on-Fleet guide and eval (2026-09-09)++Running log. "Fixed in #3686" means the fix is in the PR from this session.++## cua-sandbox (Python)++1. **OSWorld disks on Fleet had no transport.** `Image.from_registry` had no `agent_type`, and Fleet claims always spoke computer-server (`/cmd` on 8000), so `sb.screenshot()`/`sb.shell.run()` failed against an OSWorld image even though `from_file(..., agent_type="osworld")` worked locally. Fixed in #3686.+2. **`sb.shell.run()` raised `Unknown action: run_command` on the local `OSWorldTransport`.** The transport only mapped `execute`/`run_bash_script`/`run_python`. Fixed in #3686 (maps onto `/execute`, which every OSWorld server generation has; the golden image predates `/run_bash_script`).+3. **`cpu=` / `memory_mb=` / `disk_gb=` were silently ignored by local runtimes.** `Sandbox.ephemeral(..., cpu=4, memory_mb=8192, local=True)` booted a 2-vCPU/4 GB VM. Fixed in #3686.+4. **Local boot timeout is hard-coded to 120 s** in `QEMUBaremetalRuntime.start()`; `time_to_start=` is a Fleet-only knob. A slow first boot (I/O contention, second NIC) fails with `TimeoutError` and there is no way to raise it without monkeypatching `is_ready`.+5. **A timed-out local VM leaks.** After the `TimeoutError` above the `qemu-system-x86_64` process keeps running and holds the overlay; nothing in the SDK stops it. Had to `pkill` by hand twice.+6. **`Image.from_file()` writes to the file you pass.** Ephemeral sessions only get an auto-overlay when the image has build work; with a plain qcow2 the VM mutates it in place, and `stop()` only unlinks paths containing `sessions`. The guide has to tell users to `qemu-img create -b` their own overlay.+7. **Local QEMU guest has no outbound network and it is not configurable.** `-netdev user,...,restrict=on` is baked into the command; `extra_args` can add a second NIC but cannot change the first. Anything that must install software at runtime has to be baked into the disk.+8. **`sb.services.request()` has no `timeout=`** even though the underlying `FleetTransport.request_service()` accepts one. Long MCP calls through the Fleet proxy can only use the 30 s default.+9. **`QEMUWSL2Runtime` ignores `agent_type`, `expose()` ports, and cpu/memory opts**; it always forwards guest 8000 and polls `/status`, so it cannot boot an OSWorld disk from Windows even though the disk boots fine under WSL with the bare-metal runtime.+10. **Private ECR allowlist failure is reported as a name collision.** A pool on a non-allowlisted private repo fails at template creation with HTTP 403 `{"error":"k8s request is not allowed"}`, and the SDK wraps it as "Pool names are globally unique across accounts, so this name may already be taken". Nothing in the docs mentions the per-repo allowlist.+11. **Claim timeouts carry no diagnostics.** `fleet_sdk.SdkError.ClaimTimeout` after the SDK's poll limit, with the pool stuck at `ready_replicas=0` and the claim `phase=Failed`; there is no SDK surface for VM/pod events, image pull status, or readiness-probe state, so pull failure vs boot failure vs networking has to be inferred by elimination (I mirrored a known-good image into the same repo to isolate it).+12. **Docs pin `cua-sandbox==0.4.3` while `main` is 0.5.0**, and the inline-script examples pin `cua-fleet==0.1.14`; a reader following the docs against a `main` checkout gets a different `Pool.claim`/`Sandbox.ephemeral` surface than the reference describes.+13. **Telemetry logs at INFO** (`core.telemetry Recording event ... Sent event to PostHog`) on every create/destroy, which drowns application logs unless you silence the logger by name.+14. **`ClaimTimeout` cleanup left the pool for a while** in the first probe (the `finally: pool.delete()` ran, but the namespace stayed visible for minutes); a `wait_deleted` helper would make scripts' cleanup verifiable.++## Fleet / images++15. **Cross-repo blob mount did not happen on ECR push**; retagging a 12.7 GB containerDisk into a second repo re-uploaded the whole layer (about 35 min each time).+16. **The containerDisk contract (`/disk/disk.img`, KubeVirt, readiness on the `server` port) is documented, but not the ECR admission allowlist**, so "publish your own image" only works for public registries or a repo someone has allowlisted.++## cua-driver++17. **The stock MCP catalog is 60 tools (~200 KB of schema).** A small model pays that on every turn and spends its context on browser/recording/cursor tools it will never use for OSWorld. `cua-driver serve --capability-manifest` narrows it for the daemon route, but the `cua-driver mcp` stdio route (what supergateway wraps) has no equivalent flag; the Minecraft guide works around it by filtering client-side.+18. **`get_desktop_state` returns a full-resolution PNG**; with the Claude Agent SDK's default 1 MB stdio message buffer a single screenshot can exceed the limit (`CLIJSONDecodeError: JSON message exceeded maximum buffer size`). Worth documenting `max_buffer_size` next to the driver's Agent SDK example, or letting `set_config max_image_dimension` be a launch flag.+19. **`launch_app` in background mode + `bring_to_front` leaves the window invisible to the OSWorld screenshot**, while `list_windows` reports it mapped at 412x541+66+36. Both processes share `DISPLAY=:0`. Unclear whether that is intended background-delivery behavior or a compositor quirk.+20. **The built-in HTTP MCP listener is loopback-only and token-gated**, which is right for a workstation but means every VM image needs Node + supergateway (about 100 MB) to expose MCP on a NIC. A `--bind 0.0.0.0` option for in-VM use would remove that dependency.+21. **`cua-driver mcp --help` prints the global help**, not the subcommand's options, so the transport/socket flags for `mcp` have to be found in the docs.++## Claude Agent SDK interplay (not Cua, but affects the recipes)++22. The SDK's `mcp_servers` HTTP config connected to supergateway in about 1 s, so the driver works out of the box there; only the buffer size (18) needed a change.++## Found during the eval runs++23. **An element-targeted `click` on a Nautilus (Files) window stalled for ~5.5 minutes before returning** (`click {"pid": 1951, "element_index": 29, "snapshot_id": "s00000001"}` at 17:21:23, next call 17:26:54; the curated run hit the identical stall with `element_token`). Coordinate clicks on the same desktop returned in 2-4 s. With a 25-action budget one such stall eats a third of a 15-minute episode; a per-action deadline (or a documented AT-SPI timeout knob) would make this predictable.+24. **First `click` attempts used `{"pid": 1, "x": .., "y": ..}`** (pid 1 = systemd) after `get_desktop_state`, then retried with `target: {"kind": "desktop"}`. The screenshot tool's result does not tell the model which pid/target to click with, so the first action is wasted every episode; returning the owning pid per window in `get_desktop_state`, or accepting desktop coordinates without a target, would remove that.+25. **The default `click` (pid-targeted, background XSendEvent delivery) does not operate GTK menus in GIMP, and `delivery_mode: "foreground"` does not either.** Controlled test on the OSWorld image (GIMP 2.10, GNOME/Xorg), clicking the "Colors" menubar item at (371,76) and measuring the screen pixel diff 2 s later: `click {pid, x, y}` 0.30 % (only the agent-cursor overlay moved), `click {pid, x, y, delivery_mode: "foreground"}` 0.04 %, `click {target: {kind: "desktop"}, x, y}` 0.54 % (menu open), OSWorld `pyautogui.click` 0.51 % (menu open). Screenshots: `1_cua_click_bg.png`, `2_cua_click_fg.png`, `3_cua_click_desktop.png`, `4_pyautogui_click.png`. Every tool response said "✅ Clicked", so the model has no signal that nothing happened; in the eval traces Haiku clicked GIMP's menubar with `{pid, x, y}` five times in a row and then gave up on menus. Either make `desktop` scope the fallback when XSendEvent delivery is rejected by the toolkit, report the delivery outcome honestly, or document that GTK2/GTK3 menus need `target: desktop`.+26. **`type_text` into LibreOffice Calc stalled for ~2.5 minutes** (`type_text {"pid": 1825, "text": "=C3*24*E3"}` at 17:54:06, next call 17:56:33), then the model retried with `delivery_mode: "foreground"` and finally fell back to the legacy `page` tool with `selector: "*"`. Together with #23 this suggests background delivery to GTK/VCL windows can block on the toolkit side for minutes with no timeout and no error surfaced to the caller.+27. **After a stalled action, every subsequent tool call on that window took ~62 s** (17:57:35 → 18:04:39 in the `cuadriver/357ef137` trace: `page`, `get_window_state`, `hotkey`, `bring_to_front`, `click`, `get_desktop_state` each returned after about a minute), until `health_report` was called and latency dropped back to 1-2 s. Looks like a per-call 60 s timeout draining a wedged delivery/AT-SPI queue; it silently burns most of an episode's wall clock.+28. **The guest X session froze during LibreOffice Calc episodes in both driver runs.** QMP screendumps of the two eval VMs taken at guest time 09:41 show the GNOME top-bar clock stopped at 09:20 (cuadriver VM, right after `type_text {"pid", "window_id", "text": "=C3*24*E3"}`) and 09:30 (curated VM); the agent-cursor overlay is still painted mid-glide on one of them (`diag-hang/faint-quetzal.png`, `lithe-xerus.png`). From the host, the OSWorld server (`:5000`) and supergateway (`:3000/healthz`) stopped answering, which is also what hung the harness in `evaluate()`. The pyautogui condition ran the same tasks on the same image without a single freeze. This looks like the driver's XSendEvent/XTest delivery (or the cursor overlay's X grabs) deadlocking the Xorg/gnome-shell session when targeting LibreOffice; it is the single biggest contributor to the cua-driver scores here.+29. **Reproduction of #26/#28 in isolation** (`diag-typetext/`, fresh VM, LibreOffice Calc open, A1 selected via pyautogui): `type_text {pid, text}` returned after **13.3 s and 25.6 s** with `Background delivery is not available: the requested target has no focus-free input backend; the remaining XTest/X11 route can only deliver to the globally focused...` and typed nothing; `click {pid, x, y}` took 7.7 s and changed nothing on screen; `type_text {target: desktop}` returned in 0.3 s claiming `Typed text into the focused desktop application.` but A1 stayed empty; `type_text {pid, delivery_mode: "foreground"}` never returned and the guest's X session froze (screendump clock stuck at 09:32, OSWorld server and supergateway unreachable from then on). So on this stock Ubuntu 22.04 / GNOME Xorg image the driver has no working text-input route into LibreOffice: background is refused (slowly), desktop-scope is a silent no-op, foreground hangs the display server. The refusal message itself is good; the 13-26 s it takes to arrive and the foreground hang are the problems.++## Found during the optimization work (branch `feat/osworld-cuadriver-optimization`)++30. **`click {pid, x, y}` takes window-local coordinates, but the model reads desktop coordinates off `get_desktop_state`.** Nothing in the tool description or the screenshot result says so, and the driver accepted pid + desktop pixels without complaint, so every pid-targeted pixel click in the eval landed offset by the window origin (GIMP menubar at screen (371,76) hit the canvas at (443,141)). The only screen-coordinate route was `target: {kind: desktop}` (no pid). Both optimization branches now accept `scope: "desktop"` / `coordinate_frame: "desktop"` together with a pid and translate once; the default should arguably be desktop pixels whenever the coordinates came from `get_desktop_state`.+31. **LibreOffice Calc's sheet reports 2^30 AT-SPI children;** `get_window_state` enumerated them, which is the real cause of the multi-minute Calc stalls (#26/#27) and the eventual guest death. Fixed on the branch by never expanding containers above a child-count cap and labelling the tree `truncation_reason: huge_container`.+32. **AT-SPI walks had no caller-visible budget:** 4 retries × 25 s worst case, truncation reasons only logged, `elements_complete` hard-coded false. The branch adds `get_window_state.timeout_ms` (default 1000) with `truncated`, `truncation_reason`, `nodes_visited`, `nodes_pending`, `bounds_complete`, `walk_elapsed_ms`.+33. **Daemon-wide deadlock on element clicks whose snapshot has no cached frame** (GIMP submenu items, VLC controls): the click hangs ~5.5 min and every later tool call on the daemon hangs too; all runtime threads idle in futex_wait/epoll, tokio timeouts around the task never fire. A one-shot `cua-driver mcp` process does not hang. Still open at the time of writing (being fixed on the branch).+34. **Foreground delivery restored the previously-active window after every action**, which pops GTK menus closed and made the settle budget (fixed 400 ms) fail on slow WMs; keyboard actions confirmed focus by stacking order only. Rewritten on the branch: x11rb-based, watchdogged, 800/1500 ms budgets with one re-activation, no restore, `focus_after` reported, `foreground_unavailable`/`foreground_timeout` codes.+35. **Background `type_text` into LibreOffice probed `focused_is_editable`/`type_into_editable`/Qt5 FocusIn for 13-34 s before refusing.** With a focus-event tracker the refusal now takes 0.1 s.+36. **The result text always claimed success ("✅ Clicked").** On the branch pointer results say "not verified — confirm with a screenshot" and carry `effect: suspected_noop` when focus left the process; background pixel clicks that the backend will drop are refused with `path: background_unavailable_pointer` and a hint.+37. **Root cause of #33 (daemon-wide hang): an unbounded await on the cursor-overlay "arrival" oneshot.** Every pointer/element action awaits `overlay::animate_cursor_to_for`; when the X11 overlay renderer never reports arrival (seen with the overlay pinned above an open GIMP menu) the tool future parks forever, and because the stdio MCP loop serves one request at a time every later call queues behind it. gdb/strace on the wedged daemon: all tokio workers parked, only the overlay thread polling QueryPointer at 60 Hz, no D-Bus, request never answered. Fixed on the branch with a 5.5 s arrival cap plus a degraded latch (fire-and-forget until the renderer reports again); regression test added. Why the renderer stops reporting in that case is still open.+38. **`get_desktop_state` returned a 1920x1080 PNG, which the model's vision pipeline downsizes to ~1568 px before answering,** so every desktop-grounded coordinate came back ~20 % short (GIMP menubar at (371,76) was reported as (281,57)). The pyautogui baseline avoided this by showing 1280x720 and scaling by 1.5. Fixed on the branch: `get_desktop_state` emits a ≤1280-px screenshot, records `frame_scale`, reports window rects in screenshot pixels, and every desktop-frame consumer scales back. Any screenshot tool for LLM agents needs this by default.+39. **"pid owns more than one eligible top-level window" refused 18 actions in one run** (GIMP: docks + image window). Fixed on the branch by resolving an ambiguous pid to the topmost on-screen window under the target point, and pid-only keyboard actions falling back to the pid's `_NET_ACTIVE_WINDOW`.+40. **Element clicks re-walked the whole app when the cached snapshot had no frame** (GIMP docks numbered differently in an app-wide walk vs the window-scoped one, so index spaces disagreed). Fixed: one `GetExtents` on the cached reference, else a walk of the caller's window first, same index space as the snapshot.+41. **Background keyboard delivery on these apps is a dead end by design:** XTest reaches only the focused widget, XSendEvent is ignored by GTK/VCL, and GIMP's spin-scale widgets expose neither `EditableText` nor `Value` over AT-SPI (`GrabFocus` returns false). 42 of 50 errors in the final background run were the (now instant, honest) `background_unavailable` refusal. A background-only OSWorld agent cannot reach 80 % on this image without a different input route (e.g. XTest with focus save/restore, or an in-app CDP/UNO bridge).+42. **`get_window_state` on LibreOffice still overran its budget in the full run** (13-43 s on five calls with the default 1 s `timeout_ms`), so the deadline is not yet covering every phase (bounds lookups or the window-scoped retry). Everywhere else the walk returned in 1-3 s.+43. **Schema strictness vs small models:** Haiku sent `"pid": "1874"` / `"x": "537"` as strings and invented `triple_click`, each costing a turn as `InputValidationError`. Coercing numeric strings and aliasing `triple_click` → `click count:3` would be cheap wins for weaker models.+44. **Background keyboard delivery on X11 is solvable with MPX + uinput (#41 was wrong to call it a dead end).** A per-session XI2 master pair (`XIChangeHierarchy` XIAddMaster) with a uinput keyboard hot-added by the X server and attached as its slave (`XIAttachSlave`; XTEST slaves cannot be re-parented), `XISetFocus` on the target window, real EV_KEY events through uinput: LibreOffice Calc typing lands in 1.1 s, VS Code/Electron typing and hotkeys land, GIMP menus open via the virtual pointer, and `XGetInputFocus` plus the real pointer position are unchanged before/after. Requirements the image must provide: `/dev/uinput` writable by the desktop user (udev rule + `input` group + `uinput` module). Findings on the way: the retained pointer half of the pair must be thawed after a shield grab, clicks into override-redirect popup menus must bypass the shield grab, and pid-only keys need a mapped toplevel (not the first window of the pid). Implemented on `feat/osworld-cuadriver-optimization` (17444e862 .. 4c40daa49), result path reported as `path: "mpx_uinput"`.diff --git a/libs/cua-driver/docs/research/2026-09-osworld/README.md b/libs/cua-driver/docs/research/2026-09-osworld/README.mdnew file mode 100644index 0000000000..1cd5723a0b--- /dev/null+++ b/libs/cua-driver/docs/research/2026-09-osworld/README.md@@ -0,0 +1,20 @@+# OSWorld dogfooding, September 2026++Research notes behind the Linux delivery changes in this branch. They were+written during the work, so they read as a log rather than a spec.++1. [Eval: stock driver vs the classic pyautogui loop](01-eval-stock-driver-vs-pyautogui.md).+ Same model, tasks, image and step budget: 80 % vs 13 %. Each cause was+ reproduced in isolation on a fresh VM.+2. [Optimization log](02-optimization-log.md). Foreground rewrite, AT-SPI+ budgets and partial-tree labels, the daemon deadlock, coordinate frames,+ and the MPX + uinput background route, with every run's numbers.+3. [Papercuts](03-papercuts.md). 44 numbered findings across cua-sandbox,+ Fleet, and cua-driver, each with the exact input, timing, and screenshot+ evidence where it existed.++Harness and diagnostics referenced by these notes are in the session's+scratchpad; the guide for running OSWorld on Fleet or locally is+`docs/content/docs/how-to-guides/sandbox/run-osworld-on-cloud-fleet.mdx`.+Paths such as `osworld-on-cua/...` and `diag-*/` in the notes refer to the+author's working folder, not to this repository.diff --git a/libs/cua-driver/python/src/cua_driver/_native_contract.py b/libs/cua-driver/python/src/cua_driver/_native_contract.pyindex 8773cdaaa4..2cbdb0ec60 100644--- a/libs/cua-driver/python/src/cua_driver/_native_contract.py+++ b/libs/cua-driver/python/src/cua_driver/_native_contract.py@@ -884,6 +884,102 @@ class _UniffiFfiConverterTypeActionDelivery(_UniffiConverterRustBuffer):_UniffiFfiConverterTypeActionDeliveryMode.write(value.mode, buf)_UniffiFfiConverterOptionalUInt32.write(value.delivered_count, buf)+class _UniffiFfiConverterString:+ @staticmethod+ def check_lower(value):+ if not isinstance(value, str):+ raise TypeError("argument must be str, not {}".format(type(value).__name__))+ return value++ @staticmethod+ def read(buf):+ size = buf.read_i32()+ if size < 0:+ raise InternalError("Unexpected negative string length")+ utf8_bytes = buf.read(size)+ return utf8_bytes.decode("utf-8")++ @staticmethod+ def write(value, buf):+ utf8_bytes = value.encode("utf-8")+ buf.write_i32(len(utf8_bytes))+ buf.write(utf8_bytes)++ @staticmethod+ def lift(buf):+ with buf.consume_with_stream() as stream:+ return stream.read(stream.remaining()).decode("utf-8")++ @staticmethod+ def lower(value):+ with _UniffiRustBuffer.alloc_with_builder() as builder:+ builder.write(value.encode("utf-8"))+ return builder.finalize()++class _UniffiFfiConverterOptionalString(_UniffiConverterRustBuffer):+ @classmethod+ def check_lower(cls, value):+ if value is not None:+ _UniffiFfiConverterString.check_lower(value)++ @classmethod+ def write(cls, value, buf):+ if value is None:+ buf.write_u8(0)+ return++ buf.write_u8(1)+ _UniffiFfiConverterString.write(value, buf)++ @classmethod+ def read(cls, buf):+ flag = buf.read_u8()+ if flag == 0:+ return None+ elif flag == 1:+ return _UniffiFfiConverterString.read(buf)+ else:+ raise InternalError("Unexpected flag byte for optional type")++@dataclass+class ActionError:+ """+ Why a `refused` action sent no input, and what to do instead.+"""+ def __init__(self, *, code:str, hint:typing.Optional[str]):+ self.code = code+ self.hint = hint+++++ def __str__(self):+ return "ActionError(code={}, hint={})".format(self.code, self.hint)+ def __eq__(self, other):+ if self.code != other.code:+ return False+ if self.hint != other.hint:+ return False+ return True++class _UniffiFfiConverterTypeActionError(_UniffiConverterRustBuffer):+ @staticmethod+ def read(buf):+ return ActionError(+ code=_UniffiFfiConverterString.read(buf),+ hint=_UniffiFfiConverterOptionalString.read(buf),+ )++ @staticmethod+ def check_lower(value):+ _UniffiFfiConverterString.check_lower(value.code)+ _UniffiFfiConverterOptionalString.check_lower(value.hint)++ @staticmethod+ def write(value, buf):+ _UniffiFfiConverterString.write(value.code, buf)+ _UniffiFfiConverterOptionalString.write(value.hint, buf)+@@ -1082,17 +1178,20 @@ class _UniffiFfiConverterTypeActionEvidenceKind(_UniffiConverterRustBuffer):@dataclassclass ActionEvidence:- def __init__(self, *, kind:ActionEvidenceKind):+ def __init__(self, *, kind:ActionEvidenceKind, detail:typing.Optional[str]):self.kind = kind+ self.detail = detaildef __str__(self):- return "ActionEvidence(kind={})".format(self.kind)+ return "ActionEvidence(kind={}, detail={})".format(self.kind, self.detail)def __eq__(self, other):if self.kind != other.kind:return False+ if self.detail != other.detail:+ return Falsereturn Trueclass _UniffiFfiConverterTypeActionEvidence(_UniffiConverterRustBuffer):@@ -1100,15 +1199,18 @@ class _UniffiFfiConverterTypeActionEvidence(_UniffiConverterRustBuffer):def read(buf):return ActionEvidence(kind=_UniffiFfiConverterTypeActionEvidenceKind.read(buf),+ detail=_UniffiFfiConverterOptionalString.read(buf),)@staticmethoddef check_lower(value):_UniffiFfiConverterTypeActionEvidenceKind.check_lower(value.kind)+ _UniffiFfiConverterOptionalString.check_lower(value.detail)@staticmethoddef write(value, buf):_UniffiFfiConverterTypeActionEvidenceKind.write(value.kind, buf)+ _UniffiFfiConverterOptionalString.write(value.detail, buf)@@ -1344,20 +1446,47 @@ class _UniffiFfiConverterOptionalTypeActionEscalation(_UniffiConverterRustBufferelse:raise InternalError("Unexpected flag byte for optional type")+class _UniffiFfiConverterOptionalTypeActionError(_UniffiConverterRustBuffer):+ @classmethod+ def check_lower(cls, value):+ if value is not None:+ _UniffiFfiConverterTypeActionError.check_lower(value)++ @classmethod+ def write(cls, value, buf):+ if value is None:+ buf.write_u8(0)+ return++ buf.write_u8(1)+ _UniffiFfiConverterTypeActionError.write(value, buf)++ @classmethod+ def read(cls, buf):+ flag = buf.read_u8()+ if flag == 0:+ return None+ elif flag == 1:+ return _UniffiFfiConverterTypeActionError.read(buf)+ else:+ raise InternalError("Unexpected flag byte for optional type")+@dataclassclass ActionResult:- def __init__(self, *, effect:ActionEffect, route:ActionRoute, delivery:typing.Optional[ActionDelivery], evidence:typing.Optional[typing.List[ActionEvidence]], escalation:typing.Optional[ActionEscalation]):+ def __init__(self, *, effect:ActionEffect, route:ActionRoute, delivery:typing.Optional[ActionDelivery], evidence:typing.Optional[typing.List[ActionEvidence]], escalation:typing.Optional[ActionEscalation], summary:typing.Optional[str], error:typing.Optional[ActionError]):self.effect = effectself.route = routeself.delivery = deliveryself.evidence = evidenceself.escalation = escalation+ self.summary = summary+ self.error = errordef __str__(self):- return "ActionResult(effect={}, route={}, delivery={}, evidence={}, escalation={})".format(self.effect, self.route, self.delivery, self.evidence, self.escalation)+ return "ActionResult(effect={}, route={}, delivery={}, evidence={}, escalation={}, summary={}, error={})".format(self.effect, self.route, self.delivery, self.evidence, self.escalation, self.summary, self.error)def __eq__(self, other):if self.effect != other.effect:return False@@ -1369,6 +1498,10 @@ class ActionResult:return Falseif self.escalation != other.escalation:return False+ if self.summary != other.summary:+ return False+ if self.error != other.error:+ return Falsereturn Trueclass _UniffiFfiConverterTypeActionResult(_UniffiConverterRustBuffer):@@ -1380,6 +1513,8 @@ class _UniffiFfiConverterTypeActionResult(_UniffiConverterRustBuffer):delivery=_UniffiFfiConverterOptionalTypeActionDelivery.read(buf),evidence=_UniffiFfiConverterOptionalSequenceTypeActionEvidence.read(buf),escalation=_UniffiFfiConverterOptionalTypeActionEscalation.read(buf),+ summary=_UniffiFfiConverterOptionalString.read(buf),+ error=_UniffiFfiConverterOptionalTypeActionError.read(buf),)@staticmethod@@ -1389,6 +1524,8 @@ class _UniffiFfiConverterTypeActionResult(_UniffiConverterRustBuffer):_UniffiFfiConverterOptionalTypeActionDelivery.check_lower(value.delivery)_UniffiFfiConverterOptionalSequenceTypeActionEvidence.check_lower(value.evidence)_UniffiFfiConverterOptionalTypeActionEscalation.check_lower(value.escalation)+ _UniffiFfiConverterOptionalString.check_lower(value.summary)+ _UniffiFfiConverterOptionalTypeActionError.check_lower(value.error)@staticmethoddef write(value, buf):@@ -1397,38 +1534,8 @@ class _UniffiFfiConverterTypeActionResult(_UniffiConverterRustBuffer):_UniffiFfiConverterOptionalTypeActionDelivery.write(value.delivery, buf)_UniffiFfiConverterOptionalSequenceTypeActionEvidence.write(value.evidence, buf)_UniffiFfiConverterOptionalTypeActionEscalation.write(value.escalation, buf)--class _UniffiFfiConverterString:- @staticmethod- def check_lower(value):- if not isinstance(value, str):- raise TypeError("argument must be str, not {}".format(type(value).__name__))- return value-- @staticmethod- def read(buf):- size = buf.read_i32()- if size < 0:- raise InternalError("Unexpected negative string length")- utf8_bytes = buf.read(size)- return utf8_bytes.decode("utf-8")-- @staticmethod- def write(value, buf):- utf8_bytes = value.encode("utf-8")- buf.write_i32(len(utf8_bytes))- buf.write(utf8_bytes)-- @staticmethod- def lift(buf):- with buf.consume_with_stream() as stream:- return stream.read(stream.remaining()).decode("utf-8")-- @staticmethod- def lower(value):- with _UniffiRustBuffer.alloc_with_builder() as builder:- builder.write(value.encode("utf-8"))- return builder.finalize()+ _UniffiFfiConverterOptionalString.write(value.summary, buf)+ _UniffiFfiConverterOptionalTypeActionError.write(value.error, buf)class _UniffiFfiConverterBoolean:@classmethod@@ -1451,31 +1558,6 @@ class _UniffiFfiConverterBoolean:def write(cls, value, buf):buf.write_u8(value)-class _UniffiFfiConverterOptionalString(_UniffiConverterRustBuffer):- @classmethod- def check_lower(cls, value):- if value is not None:- _UniffiFfiConverterString.check_lower(value)-- @classmethod- def write(cls, value, buf):- if value is None:- buf.write_u8(0)- return-- buf.write_u8(1)- _UniffiFfiConverterString.write(value, buf)-- @classmethod- def read(cls, buf):- flag = buf.read_u8()- if flag == 0:- return None- elif flag == 1:- return _UniffiFfiConverterString.read(buf)- else:- raise InternalError("Unexpected flag byte for optional type")-@dataclassclass AppInfo:def __init__(self, *, pid:int, name:str, running:bool, active:bool, bundle_id:typing.Optional[str], launch_path:typing.Optional[str], kind:typing.Optional[str], last_used:typing.Optional[str]):@@ -3486,20 +3568,26 @@ class _UniffiFfiConverterTypeGetCursorPositionInput(_UniffiConverterRustBuffer):@dataclassclass GetDesktopStateInput:- def __init__(self, *, session:typing.Optional[str], screenshot_out_file:typing.Optional[str]):+ def __init__(self, *, session:typing.Optional[str], screenshot_out_file:typing.Optional[str], max_image_dimension:typing.Optional[int] = _DEFAULT):self.session = sessionself.screenshot_out_file = screenshot_out_file+ if max_image_dimension is _DEFAULT:+ self.max_image_dimension = None+ else:+ self.max_image_dimension = max_image_dimensiondef __str__(self):- return "GetDesktopStateInput(session={}, screenshot_out_file={})".format(self.session, self.screenshot_out_file)+ return "GetDesktopStateInput(session={}, screenshot_out_file={}, max_image_dimension={})".format(self.session, self.screenshot_out_file, self.max_image_dimension)def __eq__(self, other):if self.session != other.session:return Falseif self.screenshot_out_file != other.screenshot_out_file:return False+ if self.max_image_dimension != other.max_image_dimension:+ return Falsereturn Trueclass _UniffiFfiConverterTypeGetDesktopStateInput(_UniffiConverterRustBuffer):@@ -3508,17 +3596,20 @@ class _UniffiFfiConverterTypeGetDesktopStateInput(_UniffiConverterRustBuffer):return GetDesktopStateInput(session=_UniffiFfiConverterOptionalString.read(buf),screenshot_out_file=_UniffiFfiConverterOptionalString.read(buf),+ max_image_dimension=_UniffiFfiConverterOptionalUInt32.read(buf),)@staticmethoddef check_lower(value):_UniffiFfiConverterOptionalString.check_lower(value.session)_UniffiFfiConverterOptionalString.check_lower(value.screenshot_out_file)+ _UniffiFfiConverterOptionalUInt32.check_lower(value.max_image_dimension)@staticmethoddef write(value, buf):_UniffiFfiConverterOptionalString.write(value.session, buf)_UniffiFfiConverterOptionalString.write(value.screenshot_out_file, buf)+ _UniffiFfiConverterOptionalUInt32.write(value.max_image_dimension, buf)@dataclassclass GetScreenSizeInput:@@ -3612,7 +3703,7 @@ class _UniffiFfiConverterTypeGetSessionStateInput(_UniffiConverterRustBuffer):@dataclassclass GetWindowStateInput:- def __init__(self, *, pid:int, window_id:int, session:typing.Optional[str], query:typing.Optional[str], include_accessibility_tree:typing.Optional[bool], include_screenshot:typing.Optional[bool], screenshot_out_file:typing.Optional[str], max_elements:typing.Optional[int], max_depth:typing.Optional[int], max_dimension:typing.Optional[int], max_image_dimension:typing.Optional[int]):+ def __init__(self, *, pid:int, window_id:int, session:typing.Optional[str], query:typing.Optional[str], include_accessibility_tree:typing.Optional[bool], include_screenshot:typing.Optional[bool], screenshot_out_file:typing.Optional[str], max_elements:typing.Optional[int], max_depth:typing.Optional[int], max_dimension:typing.Optional[int], max_image_dimension:typing.Optional[int], timeout_ms:typing.Optional[int] = _DEFAULT):self.pid = pidself.window_id = window_idself.session = session@@ -3624,12 +3715,16 @@ class GetWindowStateInput:self.max_depth = max_depthself.max_dimension = max_dimensionself.max_image_dimension = max_image_dimension+ if timeout_ms is _DEFAULT:+ self.timeout_ms = None+ else:+ self.timeout_ms = timeout_msdef __str__(self):- return "GetWindowStateInput(pid={}, window_id={}, session={}, query={}, include_accessibility_tree={}, include_screenshot={}, screenshot_out_file={}, max_elements={}, max_depth={}, max_dimension={}, max_image_dimension={})".format(self.pid, self.window_id, self.session, self.query, self.include_accessibility_tree, self.include_screenshot, self.screenshot_out_file, self.max_elements, self.max_depth, self.max_dimension, self.max_image_dimension)+ return "GetWindowStateInput(pid={}, window_id={}, session={}, query={}, include_accessibility_tree={}, include_screenshot={}, screenshot_out_file={}, max_elements={}, max_depth={}, max_dimension={}, max_image_dimension={}, timeout_ms={})".format(self.pid, self.window_id, self.session, self.query, self.include_accessibility_tree, self.include_screenshot, self.screenshot_out_file, self.max_elements, self.max_depth, self.max_dimension, self.max_image_dimension, self.timeout_ms)def __eq__(self, other):if self.pid != other.pid:return False@@ -3653,6 +3748,8 @@ class GetWindowStateInput:return Falseif self.max_image_dimension != other.max_image_dimension:return False+ if self.timeout_ms != other.timeout_ms:+ return Falsereturn Trueclass _UniffiFfiConverterTypeGetWindowStateInput(_UniffiConverterRustBuffer):@@ -3670,6 +3767,7 @@ class _UniffiFfiConverterTypeGetWindowStateInput(_UniffiConverterRustBuffer):max_depth=_UniffiFfiConverterOptionalUInt32.read(buf),max_dimension=_UniffiFfiConverterOptionalUInt32.read(buf),max_image_dimension=_UniffiFfiConverterOptionalUInt32.read(buf),+ timeout_ms=_UniffiFfiConverterOptionalUInt32.read(buf),)@staticmethod@@ -3685,6 +3783,7 @@ class _UniffiFfiConverterTypeGetWindowStateInput(_UniffiConverterRustBuffer):_UniffiFfiConverterOptionalUInt32.check_lower(value.max_depth)_UniffiFfiConverterOptionalUInt32.check_lower(value.max_dimension)_UniffiFfiConverterOptionalUInt32.check_lower(value.max_image_dimension)+ _UniffiFfiConverterOptionalUInt32.check_lower(value.timeout_ms)@staticmethoddef write(value, buf):@@ -3699,6 +3798,7 @@ class _UniffiFfiConverterTypeGetWindowStateInput(_UniffiConverterRustBuffer):_UniffiFfiConverterOptionalUInt32.write(value.max_depth, buf)_UniffiFfiConverterOptionalUInt32.write(value.max_dimension, buf)_UniffiFfiConverterOptionalUInt32.write(value.max_image_dimension, buf)+ _UniffiFfiConverterOptionalUInt32.write(value.timeout_ms, buf)@dataclassclass HotkeyInput:@@ -7840,6 +7940,7 @@ __all__ = ["VisualParseErrorCode","Platform","ActionDelivery",+ "ActionError","ActionEscalation","ActionEvidence","ActionResult",diff --git a/libs/cua-driver/python/tests/test_remote_channel.py b/libs/cua-driver/python/tests/test_remote_channel.pyindex dd40d7bea5..d07f2d7a00 100644--- a/libs/cua-driver/python/tests/test_remote_channel.py+++ b/libs/cua-driver/python/tests/test_remote_channel.py@@ -325,6 +325,7 @@ class NativeWindowTests(unittest.IsolatedAsyncioTestCase):max_depth=3,max_dimension=800,max_image_dimension=1600,+ timeout_ms=None,))self.assertIsInstance(state, sdk.WindowStateOutput)diff --git a/libs/cua-driver/rust/Skills/cua-driver/LINUX.md b/libs/cua-driver/rust/Skills/cua-driver/LINUX.mdindex e046a637f2..fc782f8ad4 100644--- a/libs/cua-driver/rust/Skills/cua-driver/LINUX.md+++ b/libs/cua-driver/rust/Skills/cua-driver/LINUX.md@@ -68,6 +68,74 @@ desktop route after user screenshot approval. The window route returned`unverifiable`, so screenshots and user confirmation established the result.This is not certification of background targeting, other compositors, or video.+## X11 observation and input details++**What is open over the window.** `get_window_state` lists the pid's transient+dialogs (`dialogs: [{window_id, title, transient_for, modal, bounds}]`) and+override-redirect popups (`popups: [...]`) that are mapped, with a `follow_up`+sentence naming the call that targets each one+(`get_window_state(pid, window_id=<that id>)`). When one overlaps the window+the screenshot is taken from the screen (`screenshot_composited:true`) so the+menu or dialog is visible; the window's own drawable never shows them. The+payload states its frame (`coordinate_frame:"window"`, `frame_note`): `x`/`y`+of the pointer tools are pixels of THIS screenshot. Closed menus are listed+with a `description` ("closed menu with N items…") and are not walked —+click the menu (a real press) and read the popup by its window_id.+`press_key` / `hotkey` that map a new top-level report it as `window_opened`+with `window_change` evidence. Action results carry the same text as+`summary` inside `structuredContent`, for clients that only show that.++**Screenshot scale.** The window screenshot is delivered at or below 1.15+megapixels (long edge ≤ `max_image_dimension`, 1568 by default): larger+images are downsized before a model reads them, and its pixel coordinates+would then be uniformly short. Pointer `x`/`y` and each element's+`screenshot_frame` are pixels of the delivered image (the element's `frame`+stays in screen coordinates, as on every platform); `frame_scale` < 1 and+`screenshot_original_width` report the downsizing, and the driver scales your+pixels back to the window.+An explicit per-call `max_image_dimension` (0 = native) replaces this cap.++**Keys while the app's own popup is open.** A Qt combo list / completer or a+GTK/VCL menu holds a keyboard grab that makes the X server drop keys from the+virtual keyboard. `press_key` / `hotkey` / `type_text` / `set_value` then go+through the core keyboard (`path: "xtest_core_grab"`, the result names the+popup and states that the core focus and active window were verified+unchanged), or — when another application holds the core focus — are refused+with `code: "popup_keyboard_grab"` and a hint: dismiss the popup (click+outside it, or click one of its rows by element_index after+`get_window_state(pid, window_id=<popup>)`) and retry. Typing an absolute path+into a Qt file dialog opens its completer after the first `/`; prefer+`set_value` on the "File name" field, which writes the path in one go.++**Popup walks, labels, descriptions.** `get_window_state(pid, window_id=<popup>)`+returns the popup's own rows (list / tree / menu items), never the main+window's menubar. Elements carry `description` when the toolkit publishes one+(Qt keeps a button's tooltip there: `push button "" (description "Pause")`;+`label` falls back to it). A control's `label` is never its value: a spin+button or slider nobody names is `unlabelled: true` with its place in+`description` (`unlabelled spin button; 3rd of 4 spin buttons in this panel`,+in visual order). `parent_index` is the nearest indexed real ancestor.++**Grid presses.** After a click that lands on a table cell (LibreOffice Calc),+the result appends `focus: cell D2` (`focused_cell`, evidence) read from the+focus log, so a one-row miss is visible before you type.++**Multi-click.** `click` takes `count: 2` (double) or `count: 3` (triple, a+line/paragraph selection in editors) as one press train with real double-click+cadence in both delivery modes; there is no `triple_click` tool.++**Typing throughput.** Key-event typing (XTest in foreground, the virtual+keyboard in background) runs at roughly 45-60 characters per second, so a+1,300-character script takes about 30 s: scale a client-side `type_text`+deadline with the text length instead of using a flat one (0.04 s per+character leaves margin over those rates). A terminal can still be echoing when the call returns; read the+result again before retyping text that looks truncated.++**Cross-application drops.** A pointer action whose point lies over another+application's window (a drag dropped onto VLC) reports that window's title+change and the windows its pid opened as `foreign_window` / `window_change`+evidence; the focus guard alone only watches the target pid.+## Native application menusUse `invoke_menu({pid, window_id, path:[...]})` for a known GTK/Qt applicationdiff --git a/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md b/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.mdindex 2bad6b31ab..a1ab002277 100644--- a/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md+++ b/libs/cua-driver/rust/Skills/cua-driver/WORKFLOW.md@@ -37,7 +37,7 @@ These objects are argument fragments for tools advertising `target`, not standal`get_window_state({pid, window_id})` requests the accessibility tree and a grounding screenshot by default. Check what actually came back: permission, backing-store, or surface-identity failures can leave usable tree data without an image. `screenshot_error` and `screenshot_frame_valid:false` are not empty-tree signals.-Prefer `structuredContent.elements` in MCP (the CLI prints structured fields directly) over parsing `tree_markdown`. Rows may contain `element_token`, role, label, value, actions, parent, depth, enabled/selected state, and frame. Missing fields are unknown.+Prefer `structuredContent.elements` in MCP (the CLI prints structured fields directly) over parsing `tree_markdown`. Rows may contain `element_token`, role, label, value, actions, parent, depth, enabled/selected state, `frame` (screen coordinates, the space of `scope:"desktop"` actions), and `screenshot_frame` (pixels of the screenshot in the same response, the space of window-local pointer `x`/`y`). Missing fields are unknown.- Use `query` to project matching rows plus ancestors without renumbering their indices.- Use `max_elements` / `max_depth` to bound the walk, and compare returned/total counts. Truncation does not prove absence.diff --git a/libs/cua-driver/rust/crates/cua-driver-contract/src/inputs.rs b/libs/cua-driver/rust/crates/cua-driver-contract/src/inputs.rsindex 9afc4ff687..9adb6ec7fa 100644--- a/libs/cua-driver/rust/crates/cua-driver-contract/src/inputs.rs+++ b/libs/cua-driver/rust/crates/cua-driver-contract/src/inputs.rs@@ -460,6 +460,17 @@ pub struct GetDesktopStateInput {#[serde(default, skip_serializing_if = "Option::is_none")]#[schemars(schema_with = "string_schema")]pub screenshot_out_file: Option<String>,+ /// Optional long-edge cap for the returned PNG, in pixels. Omitted or 0+ /// returns the full-size capture. When the cap downsizes the image,+ /// desktop-scope x/y taken from it are mapped back automatically.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ #[schemars(schema_with = "nonnegative_integer_schema")]+ #[uniffi(default = None)]+ pub max_image_dimension: Option<u32>,+}++fn nonnegative_integer_schema(_: &mut SchemaGenerator) -> Schema {+ json_schema!({ "type": "integer", "minimum": 0 })}impl ToolInput for GetDesktopStateInput {diff --git a/libs/cua-driver/rust/crates/cua-driver-contract/src/lib.rs b/libs/cua-driver/rust/crates/cua-driver-contract/src/lib.rsindex 964baf39ac..1336622507 100644--- a/libs/cua-driver/rust/crates/cua-driver-contract/src/lib.rs+++ b/libs/cua-driver/rust/crates/cua-driver-contract/src/lib.rs@@ -41,15 +41,16 @@ pub use inputs::{};pub use outputs::{advertised_output_schema, conforming_error_envelope, is_refusal_envelope,- refusal_envelope_schema, ActionDelivery, ActionDeliveryMode, ActionEffect, ActionEscalation,- ActionEscalationReason, ActionEscalationTarget, ActionEvidence, ActionEvidenceKind,- ActionResult, ActionResultValidationError, ActionRoute, ClipboardReadOutput,- ClipboardWriteOutput, CursorMotionOutput, CursorPointOutput, CursorPositionOutput,- CursorThemeOutput, CursorVisualOutput, DesktopStateOutput, EffectiveScope, EndSessionOutput,- GetAgentCursorStateOutput, ListSessionsOutput, ScreenSizeOutput, SessionClientKindOutput,- SessionLifecycleState, SessionOutput, SessionStateOutput, SessionTransportOutput,- SetAgentCursorEnabledOutput, SetAgentCursorMotionOutput, SetAgentCursorThemeOutput,- StartSessionOutput, ToolOutput, TOOL_INVOCATION_FAILED_CODE,+ refusal_envelope_schema, ActionDelivery, ActionDeliveryMode, ActionEffect, ActionError,+ ActionEscalation, ActionEscalationReason, ActionEscalationTarget, ActionEvidence,+ ActionEvidenceKind, ActionResult, ActionResultValidationError, ActionRoute,+ ClipboardReadOutput, ClipboardWriteOutput, CursorMotionOutput, CursorPointOutput,+ CursorPositionOutput, CursorThemeOutput, CursorVisualOutput, DesktopStateOutput,+ EffectiveScope, EndSessionOutput, GetAgentCursorStateOutput, ListSessionsOutput,+ ScreenSizeOutput, SessionClientKindOutput, SessionLifecycleState, SessionOutput,+ SessionStateOutput, SessionTransportOutput, SetAgentCursorEnabledOutput,+ SetAgentCursorMotionOutput, SetAgentCursorThemeOutput, StartSessionOutput, ToolOutput,+ TOOL_INVOCATION_FAILED_CODE,};pub use verification::{BoundsExpectation, ElementPredicate, ElementSelector, PredicateOutcome, StatePredicate,diff --git a/libs/cua-driver/rust/crates/cua-driver-contract/src/outputs.rs b/libs/cua-driver/rust/crates/cua-driver-contract/src/outputs.rsindex 0933fb5666..eb517eb601 100644--- a/libs/cua-driver/rust/crates/cua-driver-contract/src/outputs.rs+++ b/libs/cua-driver/rust/crates/cua-driver-contract/src/outputs.rs@@ -510,6 +510,19 @@ pub enum ActionEvidenceKind {#[serde(deny_unknown_fields)]pub struct ActionEvidence {pub kind: ActionEvidenceKind,+ /// Human-readable readback the evidence rests on (what changed, which+ /// popup or window appeared and how to target it). Never request data.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub detail: Option<String>,+}++/// Why a `refused` action sent no input, and what to do instead.+#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, PartialEq, Eq, uniffi::Record)]+#[serde(deny_unknown_fields)]+pub struct ActionError {+ pub code: String,+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub hint: Option<String>,}#[derive(Debug, Clone, Copy, Serialize, Deserialize, JsonSchema, PartialEq, Eq, uniffi::Enum)]@@ -549,6 +562,15 @@ pub struct ActionResult {pub evidence: Option<Vec<ActionEvidence>>,#[serde(default, skip_serializing_if = "Option::is_none")]pub escalation: Option<ActionEscalation>,+ /// The producer's human summary of what happened (resolved points, the+ /// element hit, popups that opened, focus outcome, follow-up calls).+ /// Clients that read only `structuredContent` still get everything the+ /// text content says.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub summary: Option<String>,+ /// Present only with `effect: refused`.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub error: Option<ActionError>,}#[derive(Debug, Clone, Copy, PartialEq, Eq)]@@ -557,6 +579,7 @@ pub enum ActionResultValidationError {PartialRequiresDeliveredCount,RefusedCannotHaveDelivery,RefusedCannotHaveEvidence,+ ErrorRequiresRefused,}impl std::fmt::Display for ActionResultValidationError {@@ -566,6 +589,7 @@ impl std::fmt::Display for ActionResultValidationError {Self::PartialRequiresDeliveredCount => "partial effect requires delivered_count",Self::RefusedCannotHaveDelivery => "refused effect cannot include delivery",Self::RefusedCannotHaveEvidence => "refused effect cannot include evidence",+ Self::ErrorRequiresRefused => "error is only reported with a refused effect",})}}@@ -598,6 +622,9 @@ impl ActionResult {ActionEffect::Refused if self.evidence.is_some() => {Err(ActionResultValidationError::RefusedCannotHaveEvidence)}+ effect if effect != ActionEffect::Refused && self.error.is_some() => {+ Err(ActionResultValidationError::ErrorRequiresRefused)+ }_ => Ok(()),}}@@ -686,8 +713,11 @@ mod tests {}),evidence: Some(vec![ActionEvidence {kind: ActionEvidenceKind::ValueReadback,+ detail: None,}]),escalation: None,+ summary: None,+ error: None,}}@@ -700,7 +730,15 @@ mod tests {let properties = schema["properties"].as_object().expect("properties");assert_eq!(properties.keys().map(String::as_str).collect::<Vec<_>>(),- ["delivery", "effect", "escalation", "evidence", "route"]+ [+ "delivery",+ "effect",+ "error",+ "escalation",+ "evidence",+ "route",+ "summary"+ ]);assert_eq!(properties["effect"]["enum"],@@ -814,10 +852,40 @@ mod tests {delivery_extension["delivery"]["requested"] = json!("background");assert!(serde_json::from_value::<ActionResult>(delivery_extension).is_err());+ // Evidence carries its human detail so a client that reads only+ // structuredContent still sees what the readback said; other+ // evidence extensions stay closed.+ let mut evidence_detail = serde_json::to_value(&result).expect("serialize");+ evidence_detail["evidence"][0]["detail"] = json!("value read back as 42");+ assert_eq!(+ serde_json::from_value::<ActionResult>(evidence_detail)+ .expect("detail")+ .evidence+ .expect("evidence")[0]+ .detail+ .as_deref(),+ Some("value read back as 42")+ );let mut evidence_extension = serde_json::to_value(&result).expect("serialize");- evidence_extension["evidence"][0]["detail"] = json!("private readback");+ evidence_extension["evidence"][0]["raw"] = json!("private readback");assert!(serde_json::from_value::<ActionResult>(evidence_extension).is_err());+ let refusal = json!({+ "effect": "refused",+ "route": "synthetic_events",+ "summary": "no input was sent",+ "error": {"code": "point_outside_window", "hint": "use window-local pixels"}+ });+ let refusal = serde_json::from_value::<ActionResult>(refusal).expect("refusal");+ assert_eq!(refusal.validate_invariants(), Ok(()));+ assert_eq!(+ refusal.error.as_ref().map(|error| error.code.as_str()),+ Some("point_outside_window")+ );+ let mut error_extension = serde_json::to_value(&refusal).expect("serialize");+ error_extension["error"]["detail"] = json!({"x": 1});+ assert!(serde_json::from_value::<ActionResult>(error_extension).is_err());+let escalation_extension = json!({"effect": "unverifiable","route": "synthetic_events",@@ -863,6 +931,7 @@ mod tests {result.delivery = None;result.evidence = Some(vec![ActionEvidence {kind: ActionEvidenceKind::WindowChange,+ detail: None,}]);assert_eq!(result.validate_invariants(),@@ -870,6 +939,17 @@ mod tests {);result.evidence = None;assert_eq!(result.validate_invariants(), Ok(()));++ result.error = Some(ActionError {+ code: "window_target_not_found".into(),+ hint: None,+ });+ assert_eq!(result.validate_invariants(), Ok(()));+ result.effect = ActionEffect::Unverifiable;+ assert_eq!(+ result.validate_invariants(),+ Err(ActionResultValidationError::ErrorRequiresRefused)+ );}}diff --git a/libs/cua-driver/rust/crates/cua-driver-contract/src/windows.rs b/libs/cua-driver/rust/crates/cua-driver-contract/src/windows.rsindex 70394d03dd..ee76c55116 100644--- a/libs/cua-driver/rust/crates/cua-driver-contract/src/windows.rs+++ b/libs/cua-driver/rust/crates/cua-driver-contract/src/windows.rs@@ -97,6 +97,26 @@ pub struct GetWindowStateInput {#[serde(default, skip_serializing_if = "Option::is_none")]#[schemars(schema_with = "nonnegative_integer_schema")]pub max_image_dimension: Option<u32>,+ /// Wall-clock budget for the accessibility walk in milliseconds+ /// (default 1000 on every platform). A walk that runs out returns a+ /// partial tree flagged `truncated` rather than failing.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ #[schemars(schema_with = "timeout_ms_schema")]+ #[uniffi(default = None)]+ pub timeout_ms: Option<u32>,+}++/// Bounds of the accessibility-walk budget, shared with every live backend+/// schema (`cua_driver_core::tool_schema::timeout_ms_schema`).+pub const TIMEOUT_MS_MIN: u32 = 100;+pub const TIMEOUT_MS_MAX: u32 = 120_000;++fn timeout_ms_schema(_: &mut schemars::SchemaGenerator) -> schemars::Schema {+ schemars::json_schema!({+ "type": "integer",+ "minimum": TIMEOUT_MS_MIN,+ "maximum": TIMEOUT_MS_MAX+ })}impl ToolInput for GetWindowStateInput {@@ -108,6 +128,14 @@ impl ToolInput for GetWindowStateInput {if [self.max_elements, self.max_depth, self.max_dimension].contains(&Some(0)) {return Err("window observation limits must be positive".into());}+ if self+ .timeout_ms+ .is_some_and(|ms| !(TIMEOUT_MS_MIN..=TIMEOUT_MS_MAX).contains(&ms))+ {+ return Err(format!(+ "timeout_ms must be between {TIMEOUT_MS_MIN} and {TIMEOUT_MS_MAX} milliseconds"+ ));+ }if self.include_accessibility_tree == Some(false) && self.include_screenshot == Some(false){return Err("window observation requires accessibility or screenshot capture".into());diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/action_record.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/action_record.rsindex e08a7a2a81..54eab02f37 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/action_record.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/action_record.rs@@ -83,6 +83,10 @@ pub enum ActionTransport {LinuxPty,LinuxXSendEvent,LinuxXTest,+ /// X11 XInput2 MPX virtual master pointer/keyboard fed by a uinput slave:+ /// real (non-synthetic) events aimed at one window without touching the+ /// user's core focus or pointer.+ LinuxX11MpxUinput,LinuxLibei,LinuxWaylandVirtualPointer,LinuxCuaCompositorInject,@@ -121,6 +125,7 @@ impl ActionTransport {Self::LinuxPty,Self::LinuxXSendEvent,Self::LinuxXTest,+ Self::LinuxX11MpxUinput,Self::LinuxLibei,Self::LinuxWaylandVirtualPointer,Self::LinuxCuaCompositorInject,@@ -159,6 +164,7 @@ impl ActionTransport {| Self::WindowsSetCursorPos| Self::WindowsShellExecute| Self::LinuxXTest+ | Self::LinuxX11MpxUinput| Self::LinuxLibei| Self::LinuxWaylandVirtualPointer| Self::LinuxHyprlandForegroundInput@@ -236,6 +242,13 @@ pub enum EscalationKind {RetryWithForegroundDelivery,}+/// Why a refused action sent no input, in the words the caller can act on.+#[derive(Clone, Debug, Eq, PartialEq)]+pub struct ActionRefusal {+ pub code: String,+ pub hint: Option<String>,+}+/// Complete internal accounting for one action execution.#[derive(Clone, Debug, Eq, PartialEq)]pub struct ActionExecutionRecord {@@ -249,6 +262,7 @@ pub struct ActionExecutionRecord {pub escalation: Option<ActionEscalation>,pub delivered_count: Option<u32>,pub detail: Option<String>,+ pub refusal: Option<ActionRefusal>,}impl ActionExecutionRecord {@@ -268,6 +282,7 @@ impl ActionExecutionRecord {escalation: None,delivered_count: None,detail: None,+ refusal: None,}}@@ -360,6 +375,7 @@ impl ActionExecutionRecord {evidence.into_iter().map(|evidence| cua_driver_contract::ActionEvidence {+ detail: (!evidence.detail.is_empty()).then(|| evidence.detail.clone()),kind: match evidence.kind {ProjectedEvidenceKind::AccessibilityReadback| ProjectedEvidenceKind::BrowserReadback@@ -420,6 +436,21 @@ impl ActionExecutionRecord {},}}),+ // The dispatch seam attaches the producer's text as the summary;+ // the record itself never carries prose that could echo request+ // data.+ summary: None,+ error: match projection.effect {+ ActionEffect::Refused => {+ self.refusal+ .as_ref()+ .map(|refusal| cua_driver_contract::ActionError {+ code: refusal.code.clone(),+ hint: refusal.hint.clone(),+ })+ }+ _ => None,+ },})}@@ -471,7 +502,7 @@ impl ActionExecutionRecord {detail: structured.get("verify").and_then(serde_json::Value::as_str)- .unwrap_or("confirmed")+ .unwrap_or_default().to_owned(),});}@@ -501,6 +532,38 @@ impl ActionExecutionRecord {{record.escalation = browser_refusal_escalation(code);}+ // Producers may declare explicit evidence items (`{kind, detail}`);+ // only the kinds the public contract publishes can carry a+ // `confirmed` effect (screenshot comparisons stay internal).+ if let Some(items) = structured+ .get("evidence")+ .and_then(serde_json::Value::as_array)+ {+ for item in items {+ let kind = match item.get("kind").and_then(serde_json::Value::as_str) {+ Some("accessibility_readback") => EvidenceKind::AccessibilityReadback,+ Some("browser_readback") => EvidenceKind::BrowserReadback,+ Some("value_readback") => EvidenceKind::ValueReadback,+ Some("window_change") => EvidenceKind::WindowChange,+ Some("native_api_result") => EvidenceKind::NativeApiResult,+ Some("screenshot_comparison") => EvidenceKind::ScreenshotComparison,+ Some("event_receipt") => EvidenceKind::EventReceipt,+ Some("operator_observation") => EvidenceKind::OperatorObservation,+ _ => continue,+ };+ let detail = item+ .get("detail")+ .and_then(serde_json::Value::as_str)+ .unwrap_or_default()+ .to_owned();+ if !record.evidence.iter().any(|existing| existing.kind == kind) {+ record.evidence.push(ActionEvidence { kind, detail });+ }+ }+ }+ if effect == ActionEffect::Refused {+ record.refusal = legacy_refusal(structured);+ }if effect == ActionEffect::Partial && record.delivered_count.is_none() {return None;}@@ -543,6 +606,10 @@ impl ActionExecutionRecord {"detail": escalation.detail,})),"detail": self.detail,+ "refusal": self.refusal.as_ref().map(|refusal| serde_json::json!({+ "code": refusal.code,+ "hint": refusal.hint,+ })),})}}@@ -572,6 +639,27 @@ fn requested_delivery(tool_name: &str, args: &serde_json::Value) -> RequestedDel}}+/// The refusal code and hint a legacy payload carries, in either envelope+/// (`{"refusal":{code,message}}` or `{"code":…, "hint"|"message":…}`).+fn legacy_refusal(structured: &serde_json::Value) -> Option<ActionRefusal> {+ let code = structured+ .pointer("/refusal/code")+ .or_else(|| structured.get("code"))+ .and_then(serde_json::Value::as_str)?;+ let hint = ["/refusal/hint", "/refusal/message", "/hint", "/message"]+ .iter()+ .find_map(|pointer| {+ structured+ .pointer(pointer)+ .and_then(serde_json::Value::as_str)+ })+ .map(str::to_owned);+ Some(ActionRefusal {+ code: code.to_owned(),+ hint,+ })+}+fn legacy_effect(structured: &serde_json::Value) -> ActionEffect {if structured.get("status").and_then(serde_json::Value::as_str) == Some("refused") {let is_partial = structured@@ -686,9 +774,10 @@ fn transport_from_legacy("SetCursorPos" => ActionTransport::WindowsSetCursorPos,"atspi" | "wayland_atspi" | "x11_atspi" => ActionTransport::LinuxAtSpiAction,"pty" => ActionTransport::LinuxPty,- "x11_pixel" | "x11_pixel_fg" | "x11_xtest_fg" | "xtest" | "xtest_desktop" => {- ActionTransport::LinuxXTest- }+ "mpx_uinput" | "mpx_pointer" => ActionTransport::LinuxX11MpxUinput,+ "x11_xsendevent" => ActionTransport::LinuxXSendEvent,+ "x11_pixel" | "x11_pixel_fg" | "x11_xtest_fg" | "xtest" | "xtest_desktop"+ | "xtest_core_grab" => ActionTransport::LinuxXTest,"wayland_activate" | "wayland_focused" => ActionTransport::LinuxLibei,"wayland_desktop" => ActionTransport::LinuxWaylandVirtualPointer,"cua_compositor_inject" | "wayland_cua_compositor" => {@@ -1029,6 +1118,7 @@ fn transport_name(transport: ActionTransport) -> &'static str {ActionTransport::LinuxPty => "linux_pty",ActionTransport::LinuxXSendEvent => "linux_x_send_event",ActionTransport::LinuxXTest => "linux_x_test",+ ActionTransport::LinuxX11MpxUinput => "linux_x11_mpx_uinput",ActionTransport::LinuxLibei => "linux_libei",ActionTransport::LinuxWaylandVirtualPointer => "linux_wayland_virtual_pointer",ActionTransport::LinuxCuaCompositorInject => "linux_cua_compositor_inject",@@ -1088,6 +1178,14 @@ impl ActionExecutionRecordBuilder {self}+ pub fn refusal(mut self, code: impl Into<String>, hint: Option<String>) -> Self {+ self.0.refusal = Some(ActionRefusal {+ code: code.into(),+ hint,+ });+ self+ }+pub fn detail(mut self, detail: impl Into<String>) -> Self {self.0.detail = Some(detail.into());self@@ -1430,7 +1528,7 @@ mod tests {.actual_delivery(ActualDelivery::Background).evidence(ActionEvidence {kind: EvidenceKind::AccessibilityReadback,- detail: "secret request-adjacent diagnostic".to_owned(),+ detail: "value read back as 42".to_owned(),}).detail("pid=42 window_id=7 x=10 y=20").build()@@ -1445,12 +1543,11 @@ mod tests {"effect": "confirmed","route": "accessibility","delivery": {"mode": "background"},- "evidence": [{"kind": "value_readback"}]+ "evidence": [{"kind": "value_readback", "detail": "value read back as 42"}]}));let rendered = value.to_string();for forbidden in [- "secret","pid","window_id","\"x\"",@@ -1488,7 +1585,7 @@ mod tests {"effect": "confirmed","route": "system_api","delivery": {"mode": "not_applicable"},- "evidence": [{"kind": "value_readback"}]+ "evidence": [{"kind": "value_readback", "detail": "GetWindowRect matched exact HWND geometry"}]}));}@@ -1574,6 +1671,60 @@ mod tests {}}+ #[test]+ fn mpx_pointer_window_change_evidence_publishes_confirmed_effect() {+ // Linux MPX real-pointer click that opened a context menu: the popup+ // appearing is window-change evidence, so `confirmed` survives.+ let record = ActionExecutionRecord::from_legacy(+ "right_click",+ &serde_json::json!({"delivery_mode": "background"}),+ &serde_json::json!({+ "path": "mpx_pointer",+ "verified": true,+ "effect": "confirmed",+ "evidence": [+ {"kind": "window_change", "detail": "1 popup window(s) appeared"},+ {"kind": "screenshot_comparison", "detail": "4.4% of the region changed"}+ ],+ }),+ )+ .expect("mpx pointer click should normalize");+ assert_eq!(record.transport, ActionTransport::LinuxX11MpxUinput);+ assert_eq!(record.effect, ActionEffect::Confirmed);+ assert_eq!(record.evidence.len(), 2);+ let public = record.public_result().expect("public result");+ assert_eq!(public.effect, cua_driver_contract::ActionEffect::Confirmed);+ // Only the publishable kind is projected.+ assert_eq!(public.evidence.as_ref().map(Vec::len), Some(1));+ }++ #[test]+ fn mpx_pointer_screenshot_evidence_alone_stays_unverifiable() {+ let record = ActionExecutionRecord::from_legacy(+ "click",+ &serde_json::json!({"delivery_mode": "background"}),+ &serde_json::json!({+ "path": "mpx_pointer",+ "verified": true,+ "effect": "confirmed",+ "evidence": [{"kind": "screenshot_comparison", "detail": "2.8% changed"}],+ }),+ )+ .expect("mpx pointer click should normalize");+ assert_eq!(record.effect, ActionEffect::Unverifiable);+ assert_eq!(record.evidence.len(), 1);+ assert_eq!(+ ActionExecutionRecord::from_legacy(+ "drag",+ &serde_json::json!({"delivery_mode": "background"}),+ &serde_json::json!({"path": "x11_xsendevent", "verified": false, "effect": "unverifiable"}),+ )+ .expect("legacy xsendevent drag should normalize")+ .transport,+ ActionTransport::LinuxXSendEvent+ );+ }+#[test]fn click_selection_readback_preserves_confirmed_effect() {let record = ActionExecutionRecord::from_legacy(@@ -1675,6 +1826,13 @@ mod tests {let refused = refused.public_result().expect("refusal should project");assert_eq!(refused.effect, cua_driver_contract::ActionEffect::Refused);assert!(refused.delivery.is_none());+ assert_eq!(+ refused.error,+ Some(cua_driver_contract::ActionError {+ code: "browser_ref_stale".into(),+ hint: Some("refresh browser state".into()),+ })+ );assert_eq!(refused.escalation,Some(cua_driver_contract::ActionEscalation {@@ -1822,6 +1980,7 @@ mod tests {"x11_xtest_fg","xtest","xtest_desktop",+ "xtest_core_grab","wayland_activate","wayland_focused","wayland_desktop",diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/action_target.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/action_target.rsindex 794ee7d8ca..ce27b9736a 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/action_target.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/action_target.rs@@ -32,12 +32,14 @@ pub fn normalize_action_target(tool_name: &str, args: &mut Value) -> Result<(),return Ok(());};let Some(target) = object.remove("target") else {+ // Legacy `scope:"desktop"` together with a pid/window_id means+ // "desktop-frame (get_desktop_state) coordinates against this named+ // window". It is rewritten to `coordinate_frame:"desktop"` where the+ // adapter translates the point (Linux) and refused elsewhere.if object.get("scope").and_then(Value::as_str) == Some("desktop")&& (object.contains_key("pid") || object.contains_key("window_id")){- return Err(invalid_target(- "desktop scope cannot be combined with pid or window_id",- ));+ return normalize_desktop_frame_for_window(object);}return Ok(());};@@ -102,6 +104,32 @@ pub fn normalize_action_target(tool_name: &str, args: &mut Value) -> Result<(),Ok(())}+/// `scope: "desktop"` together with `pid` / `window_id` means "these x/y are+/// desktop (full-screen) pixels — the ones `get_desktop_state` returns — but+/// deliver to THIS window". Agents that ground on a desktop screenshot and+/// then target the window that owns the pixel need exactly this; refusing it+/// made every such click fail. The Linux adapter translates the point into+/// the window's local frame (`coordinate_frame: "desktop"`); backends that do+/// not implement the translation keep the explicit refusal rather than+/// silently reading desktop pixels as window-local ones.+fn normalize_desktop_frame_for_window(+ object: &mut serde_json::Map<String, Value>,+) -> Result<(), ToolResult> {+ if !desktop_frame_for_window_supported() {+ return Err(invalid_target(+ "desktop scope cannot be combined with pid or window_id",+ ));+ }+ object.remove("scope");+ object.insert("coordinate_frame".into(), Value::String("desktop".into()));+ Ok(())+}++/// Which backends translate desktop-frame pixels for a window target.+pub fn desktop_frame_for_window_supported() -> bool {+ cfg!(target_os = "linux")+}+/// Reject the delivery combination that cannot preserve background posture./// Call this only after delivery aliases and typed targets have been normalized.pub fn enforce_delivery_target(tool_name: &str, args: &Value) -> Result<(), ToolResult> {@@ -128,6 +156,25 @@ pub fn enforce_delivery_target(tool_name: &str, args: &Value) -> Result<(), Toolmod tests {use super::*;+ #[test]+ fn desktop_scope_with_pid_becomes_a_desktop_coordinate_frame_where_supported() {+ let mut args = json!({"pid": 7, "x": 100, "y": 200, "scope": "desktop"});+ let result = normalize_action_target("click", &mut args);+ if desktop_frame_for_window_supported() {+ result.unwrap();+ assert!(args.get("scope").is_none());+ assert_eq!(args["coordinate_frame"], "desktop");+ assert_eq!(args["pid"], 7);+ } else {+ assert!(result.is_err());+ }+ // Windowless desktop scope is untouched.+ let mut bare = json!({"x": 1, "y": 2, "scope": "desktop"});+ normalize_action_target("click", &mut bare).unwrap();+ assert_eq!(bare["scope"], "desktop");+ assert!(bare.get("coordinate_frame").is_none());+ }+#[test]fn exact_targets_normalize_to_one_unambiguous_legacy_shape() {let mut window = json!({@@ -158,12 +205,35 @@ mod tests {"scope": "desktop","target": {"kind": "desktop", "display_id": "primary"}}),- json!({"scope": "desktop", "pid": 7}),json!({"target": {"kind": "desktop", "display_id": "secondary"}}),json!({"target": {"kind": "window", "pid": 7, "window_id": 0}}),] {assert!(normalize_action_target("click", &mut args).is_err());}+ // desktop scope with a pid is a desktop-frame window target where the+ // backend translates it, and a refusal everywhere else.+ let mut args = json!({"scope": "desktop", "pid": 7});+ assert_eq!(+ normalize_action_target("click", &mut args).is_ok(),+ desktop_frame_for_window_supported()+ );+ }++ #[test]+ fn desktop_scope_with_pid_is_platform_gated() {+ let mut args = json!({"scope": "desktop", "pid": 7, "x": 1, "y": 2});+ let result = normalize_action_target("click", &mut args);+ if cfg!(target_os = "linux") {+ assert!(+ result.is_ok(),+ "Linux maps desktop-frame points into the window"+ );+ assert!(args.get("scope").is_none());+ assert_eq!(args["coordinate_frame"], "desktop");+ assert_eq!(args["pid"], 7);+ } else {+ assert!(result.is_err());+ }}#[test]diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/desktop_capture_scale.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/desktop_capture_scale.rsnew file mode 100644index 0000000000..5c1dd9627e--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/desktop_capture_scale.rs@@ -0,0 +1,148 @@+//! Opt-in downsizing of `get_desktop_state` screenshots, made transparent to+//! desktop-scope actions on every platform.+//!+//! `get_desktop_state` returns its full-size capture unless the caller passes+//! `max_image_dimension`. When that cap downsizes the PNG, the response+//! reports the uncapped size as `screenshot_original_width/height`, and this+//! module remembers the ratio for the calling session. A later desktop-scope+//! action (`scope:"desktop"` or `coordinate_frame:"desktop"`) from the same+//! session has its `x`/`y` (and drag endpoints) scaled back to the uncapped+//! image before the platform tool interprets them, exactly as if the caller+//! had read them off the full-size capture. `capture_id` actions are left+//! alone: their capture binding already maps the exact published image.++use serde_json::Value;+use std::collections::HashMap;+use std::sync::{Mutex, OnceLock};++/// Per-session `(x, y)` factors from capped-image pixels to uncapped pixels.+fn scales() -> &'static Mutex<HashMap<String, (f64, f64)>> {+ static SCALES: OnceLock<Mutex<HashMap<String, (f64, f64)>>> = OnceLock::new();+ SCALES.get_or_init(|| Mutex::new(HashMap::new()))+}++/// The runtime session key a dispatched call belongs to.+fn session_key(args: &Value) -> Option<&str> {+ args.get("_session_id").and_then(Value::as_str)+}++/// Record (or clear) the calling session's cap after a successful+/// `get_desktop_state`: the latest capture defines the frame of the pixels+/// the caller reads next.+pub fn record_desktop_state(args: &Value, structured: Option<&Value>) {+ let Some(key) = session_key(args) else {+ return;+ };+ let factor = structured.and_then(|s| {+ let axis = |delivered: &str, original: &str| {+ let delivered = s.get(delivered)?.as_f64()?;+ let original = s.get(original)?.as_f64()?;+ (delivered > 0.0 && original > 0.0).then_some(original / delivered)+ };+ Some((+ axis("screenshot_width", "screenshot_original_width")?,+ axis("screenshot_height", "screenshot_original_height")?,+ ))+ });+ let mut scales = scales().lock().unwrap();+ match factor {+ Some(factor) if factor != (1.0, 1.0) => {+ scales.insert(key.to_owned(), factor);+ }+ _ => {+ scales.remove(key);+ }+ }+}++/// Forget a session's cap (the session ended).+pub fn forget_session(session: &str) {+ scales().lock().unwrap().remove(session);+}++const X_KEYS: [&str; 3] = ["x", "from_x", "to_x"];+const Y_KEYS: [&str; 3] = ["y", "from_y", "to_y"];++/// Scale a desktop-scope action's coordinates from the session's capped+/// desktop image back to the uncapped one. No-op without a recorded cap, for+/// window-scoped pixels, and for `capture_id` actions.+pub fn map_desktop_args(args: &mut Value) {+ let desktop = args.get("scope").and_then(Value::as_str) == Some("desktop")+ || args.get("coordinate_frame").and_then(Value::as_str) == Some("desktop");+ if !desktop || args.get("capture_id").is_some() {+ return;+ }+ let Some(key) = session_key(args) else {+ return;+ };+ let Some((fx, fy)) = scales().lock().unwrap().get(key).copied() else {+ return;+ };+ for (keys, factor) in [(X_KEYS, fx), (Y_KEYS, fy)] {+ for key in keys {+ if let Some(value) = args.get(key).and_then(Value::as_f64) {+ args[key] = serde_json::json!(value * factor);+ }+ }+ }+}++#[cfg(test)]+mod tests {+ use super::*;+ use serde_json::json;++ #[test]+ fn a_capped_capture_scales_later_desktop_pixels_back() {+ let session = json!({"_session_id": "cap-test-a"});+ record_desktop_state(+ &session,+ Some(&json!({+ "screenshot_width": 960, "screenshot_height": 540,+ "screenshot_original_width": 1920, "screenshot_original_height": 1080+ })),+ );+ let mut click = json!({"_session_id": "cap-test-a", "scope": "desktop", "x": 100, "y": 50});+ map_desktop_args(&mut click);+ assert_eq!(+ (click["x"].as_f64(), click["y"].as_f64()),+ (Some(200.0), Some(100.0))+ );++ let mut drag = json!({"_session_id": "cap-test-a", "coordinate_frame": "desktop",+ "from_x": 10, "from_y": 20, "to_x": 30, "to_y": 40});+ map_desktop_args(&mut drag);+ assert_eq!(drag["to_y"].as_f64(), Some(80.0));++ // Window pixels, capture-bound pixels, and other sessions are untouched.+ let mut window = json!({"_session_id": "cap-test-a", "x": 100, "y": 50});+ map_desktop_args(&mut window);+ assert_eq!(window["x"], 100);+ let mut bound =+ json!({"_session_id": "cap-test-a", "scope": "desktop", "capture_id": "c", "x": 100});+ map_desktop_args(&mut bound);+ assert_eq!(bound["x"], 100);+ let mut other = json!({"_session_id": "cap-test-b", "scope": "desktop", "x": 100});+ map_desktop_args(&mut other);+ assert_eq!(other["x"], 100);+ }++ #[test]+ fn a_full_size_capture_clears_the_cap() {+ let session = json!({"_session_id": "cap-test-c"});+ record_desktop_state(+ &session,+ Some(&json!({+ "screenshot_width": 960, "screenshot_height": 540,+ "screenshot_original_width": 1920, "screenshot_original_height": 1080+ })),+ );+ record_desktop_state(+ &session,+ Some(&json!({"screenshot_width": 1920, "screenshot_height": 1080})),+ );+ let mut click = json!({"_session_id": "cap-test-c", "scope": "desktop", "x": 100, "y": 50});+ map_desktop_args(&mut click);+ assert_eq!(click["x"], 100);+ }+}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/element_cache.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/element_cache.rsindex 77777dbd6d..fec5c7c208 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/element_cache.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/element_cache.rs@@ -506,6 +506,19 @@ impl<S: SnapshotPayload> ElementCacheCore<S> {count}+ /// The window a published snapshot describes, so a pid-only action that+ /// names a `snapshot_id` (a popup's or a dialog's) is routed to that+ /// window instead of the pid's default window.+ pub fn window_for_snapshot(&self, pid: i32, snapshot_id: u32) -> Option<u64> {+ let inner = self.inner.lock().unwrap();+ inner+ .snapshots+ .get(&pid)?+ .iter()+ .find(|entry| entry.id == snapshot_id)+ .map(|entry| entry.window_id)+ }+pub fn resolve_element_args(&self,pid: i32,diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/element_frame.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/element_frame.rsnew file mode 100644index 0000000000..f046d16406--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/element_frame.rs@@ -0,0 +1,85 @@+//! `get_window_state` element rectangles in the two coordinate spaces a+//! caller uses.+//!+//! `frame` is the element's rectangle in screen coordinates on every platform+//! (the space of `scope:"desktop"` actions). `screenshot_frame` is the same+//! rectangle in pixels of the window screenshot returned by the same response+//! — the space of window-local pointer `x`/`y` — so a caller that grounds on+//! the screenshot can click an element's centre without knowing the window+//! origin or the screenshot's downsizing. Each backend supplies the window+//! origin and the delivered-pixels-per-screen-unit scale of its own capture;+//! the arithmetic is shared here.++use serde_json::{json, Value};++/// Add `screenshot_frame` to every element that has a screen `frame`.+///+/// `origin` is the screen position of the screenshot's top-left pixel and+/// `scale` the number of delivered screenshot pixels per screen unit+/// (`< 1.0` when the capture was downsized, `2.0` for a full-size Retina+/// capture of a point-based frame).+pub fn with_screenshot_frames(elements: Vec<Value>, origin: (f64, f64), scale: f64) -> Vec<Value> {+ if !(scale.is_finite() && scale > 0.0) {+ return elements;+ }+ elements+ .into_iter()+ .map(|mut entry| {+ if let Some(frame) = screenshot_frame(&entry["frame"], origin, scale) {+ entry["screenshot_frame"] = frame;+ }+ entry+ })+ .collect()+}++fn screenshot_frame(frame: &Value, (ox, oy): (f64, f64), scale: f64) -> Option<Value> {+ let x = frame.get("x")?.as_f64()?;+ let y = frame.get("y")?.as_f64()?;+ let w = frame.get("w")?.as_f64()?;+ let h = frame.get("h")?.as_f64()?;+ Some(json!({+ "x": ((x - ox) * scale).round() as i64,+ "y": ((y - oy) * scale).round() as i64,+ "w": (w * scale).round() as i64,+ "h": (h * scale).round() as i64,+ }))+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn screen_frame_maps_into_the_delivered_screenshot() {+ let elements = vec![+ json!({"element_index": 0, "frame": {"x": 144, "y": 150, "w": 100, "h": 40}}),+ json!({"element_index": 1}),+ ];+ let out = with_screenshot_frames(elements, (44.0, 40.0), 0.5);+ assert_eq!(+ out[0]["screenshot_frame"],+ json!({"x": 50, "y": 55, "w": 50, "h": 20})+ );+ // Unchanged screen frame; no frame, no screenshot_frame.+ assert_eq!(out[0]["frame"]["x"], 144);+ assert!(out[1].get("screenshot_frame").is_none());+ }++ #[test]+ fn point_frames_scale_to_retina_pixels() {+ let elements = vec![json!({"frame": {"x": 110.5, "y": 220.0, "w": 30.0, "h": 10.0}})];+ let out = with_screenshot_frames(elements, (100.0, 200.0), 2.0);+ assert_eq!(+ out[0]["screenshot_frame"],+ json!({"x": 21, "y": 40, "w": 60, "h": 20})+ );+ }++ #[test]+ fn a_degenerate_scale_adds_nothing() {+ let elements = vec![json!({"frame": {"x": 1, "y": 2, "w": 3, "h": 4}})];+ let out = with_screenshot_frames(elements, (0.0, 0.0), 0.0);+ assert!(out[0].get("screenshot_frame").is_none());+ }+}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rsindex cc82578a74..b1ac83baf2 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs@@ -58,7 +58,9 @@ pub mod cursor_hook;pub mod cursor_sampler;pub mod cursor_shape;pub mod daemon;+pub mod desktop_capture_scale;pub mod element_cache;+pub mod element_frame;pub mod element_query;pub mod element_token;pub mod expectation;@@ -92,6 +94,7 @@ pub mod tool_args;pub mod tool_schema;pub mod video;pub mod video_ffmpeg;+pub mod walk_budget;pub mod window_inspection;pub mod window_target;diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rsindex 79c7ded69f..06c5878e46 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs@@ -1567,11 +1567,28 @@ impl ToolRegistry {}).flatten();+ // Desktop pixels read off a capped get_desktop_state image are mapped+ // back to the uncapped capture before any platform interprets them.+ crate::desktop_capture_scale::map_desktop_args(&mut args);let mut result = crate::recording::scope_dispatch_click_capture(pending_turn.as_ref(),tool.invoke(args.clone()),).await;+ match resolved_name {+ "get_desktop_state" if result.is_error != Some(true) => {+ crate::desktop_capture_scale::record_desktop_state(+ &args,+ result.structured_content.as_ref(),+ );+ }+ "end_session" => {+ if let Some(session) = args.get("_session_id").and_then(Value::as_str) {+ crate::desktop_capture_scale::forget_session(session);+ }+ }+ _ => {}+ }drop(lifecycle_dispatch);// The platform worker has exited, so another text operation for this// pid may now start even while result projection and evidence capture@@ -2763,9 +2780,18 @@ fn publish_action_result(result: &mut ToolResult) -> Result<(), String> {.action_record.as_ref().ok_or_else(|| "successful action omitted its internal execution record".to_owned())?;- let public = action+ let mut public = action.public_result().map_err(|error| format!("invalid internal execution record: {error:?}"))?;+ // Some MCP clients (Claude Code among them) hand the model only+ // `structuredContent` when it is present and drop the text blocks. The+ // producer's text is the only place the resolved points, the element hit,+ // popups, focus outcome and follow-up calls are spelled out, so the+ // closed contract carries it as `summary`.+ public.summary = result.content.iter().find_map(|content| match content {+ Content::Text { text, .. } if !text.trim().is_empty() => Some(text.clone()),+ _ => None,+ });public.validate_invariants().map_err(|error| format!("invalid public projection: {error}"))?;@@ -2837,7 +2863,7 @@ mod runtime_isolation_tests {use crate::{authorization::PermissionMode,consent::{ConsentAction, ConsentRequest, ProtectedConsentProvider, ProviderDecision},- protocol::ToolResult,+ protocol::{Content, ToolResult},session_authorization::{SessionAuthorizationRegistry, SessionModeCeiling},};use std::io::Write;@@ -4604,10 +4630,17 @@ resources:let object = structured.as_object().expect("ActionResult is an object");assert_eq!(object.keys().map(String::as_str).collect::<Vec<_>>(),- ["delivery", "effect", "route"]+ ["delivery", "effect", "route", "summary"]);assert_eq!(structured["effect"], "unverifiable");assert_eq!(structured["delivery"]["mode"], "unknown");+ // The producer's text travels inside the closed contract for clients+ // that surface only structuredContent.+ let text = result.content.iter().find_map(|content| match content {+ Content::Text { text, .. } => Some(text.as_str()),+ _ => None,+ });+ assert_eq!(structured["summary"].as_str(), text);assert!(matches!(structured["route"].as_str(),Some("accessibility" | "synthetic_events" | "global_input" | "dom" | "trusted_input")diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/tool_args.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/tool_args.rsindex c045a8ebaa..be38fee372 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/tool_args.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/tool_args.rs@@ -104,6 +104,22 @@ pub fn sanitize_reserved_args(args: &mut Value) {}}+/// The session key an isolation-scoped state store should key on for this+/// call: the public `session` label, else the proxy-minted `_session_id`,+/// else the legacy `cursor_id`, else a shared `"default"` bucket for+/// anonymous calls. Mirrors the cursor/mouse-hold isolation convention so new+/// per-call state (e.g. desktop screenshot scale) doesn't invent its own.+pub fn resolve_session_key(args: &Value) -> String {+ for key in ["session", "_session_id", "cursor_id"] {+ if let Some(v) = args.get(key).and_then(Value::as_str) {+ if !v.is_empty() {+ return v.to_owned();+ }+ }+ }+ "default".to_owned()+}+#[cfg(test)]mod reserved_args_tests {use super::{parse_legacy_click_input, parse_typed_input, sanitize_reserved_args};diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/tool_schema.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/tool_schema.rsindex 4f7f7c2800..39e9266257 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/tool_schema.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/tool_schema.rs@@ -122,6 +122,56 @@ pub fn element_token_schema() -> Value {})}+/// Default and bounds for the per-call accessibility-walk budget+/// (`timeout_ms`), the same on every platform.+pub const TIMEOUT_MS_DEFAULT: u64 = 1000;+pub const TIMEOUT_MS_MIN: u64 = 100;+pub const TIMEOUT_MS_MAX: u64 = 120_000;++/// Shared `timeout_ms` parameter: wall-clock budget for the accessibility walk+/// behind an observation tool, with the same default, bounds, and partial-tree+/// semantics on every platform (see [`crate::walk_budget`]).+pub fn timeout_ms_schema() -> Value {+ json!({+ "type": "integer",+ "minimum": TIMEOUT_MS_MIN,+ "maximum": TIMEOUT_MS_MAX,+ "default": TIMEOUT_MS_DEFAULT,+ "description": format!(+ "Wall-clock budget in milliseconds for the accessibility-tree walk \+ (default {TIMEOUT_MS_DEFAULT}, min {TIMEOUT_MS_MIN}, max {TIMEOUT_MS_MAX}). \+ Bounds the WHOLE walk. When the budget runs out the tool returns the PARTIAL tree \+ it has, flagged with `truncated: true`, `truncation_reason`, `nodes_visited`, \+ `nodes_pending` and `elements_complete: false`; retry with a larger value \+ (e.g. 5000) or narrow with `query` / `max_depth`."+ )+ })+}++/// Shared `get_desktop_state.max_image_dimension`: an opt-in long-edge cap+/// whose downsizing is mapped back for later desktop-scope actions+/// (`crate::desktop_capture_scale`) and by the capture's `capture_id`.+pub fn desktop_max_image_dimension_schema() -> Value {+ json!({+ "type": "integer",+ "minimum": 0,+ "description": "Optional long-edge cap for the returned PNG, in pixels (aspect ratio \+ preserved). Omitted or 0 returns the full-size capture. When the cap downsizes \+ the image, the response reports `screenshot_original_width/height`, and x/y read \+ off it for this session's later scope:\"desktop\" actions (or passed with its \+ `capture_id`) are mapped back to the full-size frame automatically."+ })+}++/// Clamp a caller-supplied `timeout_ms` to the shared bounds, or apply the+/// default when absent / not an integer.+pub fn resolve_timeout_ms(value: Option<&Value>) -> u64 {+ value+ .and_then(Value::as_u64)+ .map(|v| v.clamp(TIMEOUT_MS_MIN, TIMEOUT_MS_MAX))+ .unwrap_or(TIMEOUT_MS_DEFAULT)+}+// ── The gate ─────────────────────────────────────────────────────────────────/// The canonical *shape* of each shared param (description stripped). `None` for@@ -139,6 +189,7 @@ fn shared_param_canonical(name: &str) -> Option<Value> {"element_token" => element_token_schema(),"snapshot_id" => snapshot_id_schema(),"capture_mode" => crate::capture_mode::capture_mode_schema(),+ "timeout_ms" => timeout_ms_schema(),_ => return None,};Some(structural(&v))@@ -264,6 +315,22 @@ pub fn shared_schema_violations(tool_name: &str, input_schema: &Value) -> Vec<Stmod tests {use super::*;+ #[test]+ fn timeout_ms_resolves_default_and_clamps() {+ assert_eq!(resolve_timeout_ms(None), TIMEOUT_MS_DEFAULT);+ assert_eq!(resolve_timeout_ms(Some(&json!("fast"))), TIMEOUT_MS_DEFAULT);+ assert_eq!(resolve_timeout_ms(Some(&json!(5))), TIMEOUT_MS_MIN);+ assert_eq!(resolve_timeout_ms(Some(&json!(5_000))), 5_000);+ assert_eq!(resolve_timeout_ms(Some(&json!(10_000_000))), TIMEOUT_MS_MAX);+ let schema = timeout_ms_schema();+ assert_eq!(schema["default"], TIMEOUT_MS_DEFAULT);+ assert!(shared_schema_violations(+ "get_window_state",+ &json!({"type":"object","properties":{"timeout_ms": schema}})+ )+ .is_empty());+ }+#[test]fn structural_strips_description_keeps_type_and_enum() {let with_prose = json!({ "type": "string", "enum": ["a", "b"], "description": "x" });diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/walk_budget.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/walk_budget.rsnew file mode 100644index 0000000000..d1e36619ee--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/walk_budget.rs@@ -0,0 +1,283 @@+//! Wall-clock and node budget for the accessibility walk behind+//! `get_window_state` (`timeout_ms` / `max_elements`).+//!+//! Every backend walks its own accessibility API (AT-SPI, AX, UIA/MSAA), but+//! the contract is shared: the walk stops at whichever budget runs out first,+//! the tool returns the PARTIAL tree it has, and the response says so with the+//! same structured fields and the same note on every platform.++use serde_json::{json, Value};+use std::time::{Duration, Instant};++/// Why a walk stopped before it had visited every node it discovered.+#[derive(Clone, Copy, Debug, PartialEq, Eq)]+pub enum WalkStop {+ /// The `timeout_ms` wall-clock budget ran out.+ Timeout,+ /// The `max_elements` node budget ran out.+ NodeBudget,+}++impl WalkStop {+ pub fn as_str(self) -> &'static str {+ match self {+ WalkStop::Timeout => "timeout",+ WalkStop::NodeBudget => "node_budget",+ }+ }+}++/// Budget for one walk. Call [`WalkBudget::admit`] before visiting each node;+/// a refused node is counted as discovered but not visited.+///+/// The clock starts at the first admitted node, so `timeout_ms` bounds the+/// walk itself rather than the backend's setup (resolving the application and+/// the target window, which on a freshly launched app can take most of a+/// second on a slow host). Each backend bounds that setup separately with its+/// own backstop past `timeout_ms`.+#[derive(Debug)]+pub struct WalkBudget {+ /// Set by the first [`WalkBudget::admit`].+ started: Option<Instant>,+ timeout: Duration,+ timeout_ms: u64,+ max_elements: usize,+ visited: usize,+ pending: usize,+ stop: Option<WalkStop>,+}++impl WalkBudget {+ /// A walk bounded by `timeout_ms` of wall-clock time and `max_elements`+ /// nodes.+ pub fn new(timeout_ms: u64, max_elements: usize) -> Self {+ Self {+ started: None,+ timeout: Duration::from_millis(timeout_ms),+ timeout_ms,+ max_elements,+ visited: 0,+ pending: 0,+ stop: None,+ }+ }++ /// A walk bounded only by `max_elements` (internal callers with no+ /// caller-supplied time budget).+ pub fn nodes_only(max_elements: usize) -> Self {+ // A year: never fires, without overflowing Instant.+ Self::new(365 * 24 * 60 * 60 * 1000, max_elements)+ }++ /// Admit one node for visiting. `false` means the walk must not visit it:+ /// a budget has run out, and the node is counted as pending.+ pub fn admit(&mut self) -> bool {+ let started = *self.started.get_or_insert_with(Instant::now);+ if self.stop.is_none() {+ if self.visited >= self.max_elements {+ self.stop = Some(WalkStop::NodeBudget);+ } else if started.elapsed() >= self.timeout {+ self.stop = Some(WalkStop::Timeout);+ }+ }+ if self.stop.is_some() {+ self.pending += 1;+ return false;+ }+ self.visited += 1;+ true+ }++ /// Whether the wall-clock budget has run out (for phases that cannot be+ /// interrupted per node). Never true before the walk has started.+ pub fn expired(&self) -> bool {+ self.started+ .is_some_and(|started| started.elapsed() >= self.timeout)+ }++ /// Record that the walk ran out of time outside [`WalkBudget::admit`]+ /// (a bulk fetch that returned after the deadline).+ pub fn stop_for_timeout(&mut self) {+ self.stop.get_or_insert(WalkStop::Timeout);+ }++ pub fn outcome(&self) -> WalkOutcome {+ WalkOutcome {+ timeout_ms: self.timeout_ms,+ stop: self.stop,+ nodes_visited: self.visited,+ nodes_pending: self.pending,+ elapsed_ms: self+ .started+ .map_or(0, |started| started.elapsed().as_millis() as u64),+ }+ }+}++/// What a finished walk reports.+#[derive(Clone, Copy, Debug, PartialEq, Eq)]+pub struct WalkOutcome {+ pub timeout_ms: u64,+ pub stop: Option<WalkStop>,+ pub nodes_visited: usize,+ pub nodes_pending: usize,+ pub elapsed_ms: u64,+}++impl WalkOutcome {+ /// A walk that produced nothing because the whole budget ran out before+ /// the backend returned (an uninterruptible provider call).+ pub fn timed_out(timeout_ms: u64, elapsed: Duration) -> Self {+ Self {+ timeout_ms,+ stop: Some(WalkStop::Timeout),+ nodes_visited: 0,+ nodes_pending: 0,+ elapsed_ms: elapsed.as_millis() as u64,+ }+ }++ pub fn truncated(&self) -> bool {+ self.stop.is_some()+ }++ pub fn reason(&self) -> Option<&'static str> {+ self.stop.map(WalkStop::as_str)+ }++ /// Write the shared walk fields into a `get_window_state` payload.+ pub fn apply(&self, structured: &mut Value) {+ structured["truncated"] = json!(self.truncated());+ if let Some(reason) = self.reason() {+ structured["truncation_reason"] = json!(reason);+ }+ structured["nodes_visited"] = json!(self.nodes_visited);+ structured["nodes_pending"] = json!(self.nodes_pending);+ structured["walk_elapsed_ms"] = json!(self.elapsed_ms);+ structured["timeout_ms"] = json!(self.timeout_ms);+ }++ /// The PARTIAL TREE note for the tree text, when the walk was cut short.+ pub fn note(&self) -> Option<String> {+ self.truncated().then(|| {+ truncation_note(+ self.reason(),+ self.timeout_ms,+ self.nodes_visited,+ self.nodes_pending,+ )+ })+ }+}++/// The note that heads a partial tree, naming why the walk stopped.+pub fn truncation_note(+ reason: Option<&str>,+ timeout_ms: u64,+ visited: usize,+ pending: usize,+) -> String {+ let why = match reason {+ Some("timeout") => format!("the {timeout_ms} ms timeout_ms budget ran out"),+ Some("node_budget") => "the max_elements node budget ran out".to_owned(),+ Some("app_unresponsive") => {+ "the application stopped answering its accessibility API".to_owned()+ }+ Some("app_lookup_timeout") => {+ format!("the application did not register with AT-SPI within {timeout_ms} ms")+ }+ Some("huge_container") => "a container with more children than can be enumerated \+ (e.g. a spreadsheet's cell grid) was not expanded"+ .to_owned(),+ Some(other) => other.to_owned(),+ None => "the walk stopped early".to_owned(),+ };+ format!(+ "⚠️ PARTIAL TREE: {why} after {visited} node(s) ({pending} discovered but not visited). \+ Every element listed is real; elements after the cut are missing. If the element you \+ need is absent, retry with a larger timeout_ms (e.g. 5000) or narrow with query / max_depth."+ )+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn node_budget_stops_and_counts_pending() {+ let mut budget = WalkBudget::nodes_only(2);+ assert!(budget.admit());+ assert!(budget.admit());+ assert!(!budget.admit());+ assert!(!budget.admit());+ let outcome = budget.outcome();+ assert_eq!(outcome.reason(), Some("node_budget"));+ assert_eq!((outcome.nodes_visited, outcome.nodes_pending), (2, 2));+ }++ #[test]+ fn expired_deadline_stops_with_timeout() {+ let mut budget = WalkBudget::new(0, 100);+ // Setup before the first node does not count against the budget.+ assert!(!budget.expired());+ assert!(!budget.admit());+ assert!(budget.expired());+ let outcome = budget.outcome();+ assert_eq!(outcome.reason(), Some("timeout"));+ assert_eq!((outcome.nodes_visited, outcome.nodes_pending), (0, 1));+ }++ #[test]+ fn the_clock_starts_at_the_first_node() {+ let mut budget = WalkBudget::new(50, 100);+ std::thread::sleep(Duration::from_millis(80));+ assert!(budget.admit(), "setup time must not exhaust the budget");+ }++ #[test]+ fn a_complete_walk_is_not_truncated() {+ let mut budget = WalkBudget::new(60_000, 10);+ for _ in 0..10 {+ assert!(budget.admit());+ }+ let outcome = budget.outcome();+ assert!(!outcome.truncated());+ assert_eq!(outcome.note(), None);+ let mut structured = json!({});+ outcome.apply(&mut structured);+ assert_eq!(structured["truncated"], false);+ assert!(structured.get("truncation_reason").is_none());+ assert_eq!(structured["nodes_visited"], 10);+ assert_eq!(structured["timeout_ms"], 60_000);+ }++ #[test]+ fn a_timed_out_walk_reports_the_budget() {+ let outcome = WalkOutcome::timed_out(1000, Duration::from_millis(1200));+ let mut structured = json!({});+ outcome.apply(&mut structured);+ assert_eq!(structured["truncated"], true);+ assert_eq!(structured["truncation_reason"], "timeout");+ let note = outcome.note().unwrap();+ assert!(note.starts_with("⚠️ PARTIAL TREE: the 1000 ms timeout_ms budget ran out"));+ }++ #[test]+ fn truncation_note_names_the_budget_and_the_remedy() {+ let note = truncation_note(Some("timeout"), 1000, 240, 88);+ assert!(note.contains("PARTIAL TREE"));+ assert!(note.contains("1000 ms"));+ assert!(note.contains("240 node(s)"));+ assert!(note.contains("88 discovered"));+ assert!(note.contains("timeout_ms"));+ assert!(note.contains("query"));+ let note = truncation_note(Some("node_budget"), 1000, 5000, 3);+ assert!(note.contains("max_elements"));+ let note = truncation_note(Some("app_unresponsive"), 1000, 3, 0);+ assert!(note.contains("stopped answering"));+ let note = truncation_note(Some("app_lookup_timeout"), 250, 0, 0);+ assert!(note.contains("250 ms"));+ let note = truncation_note(Some("huge_container"), 1000, 1918, 0);+ assert!(note.contains("not expanded"));+ }+}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/window_target.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/window_target.rsindex 823bfcfdcf..f5e0702c18 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/window_target.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/window_target.rs@@ -14,6 +14,10 @@ use crate::tool::{ProtectedResourceOwnership, Tool, ToolDef};#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]pub struct WindowTargetCandidate {pub window_id: u64,+ /// The window this one is a dialog of (`WM_TRANSIENT_FOR` on X11), when+ /// the platform reports it.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub transient_for: Option<u64>,pub title: String,#[serde(skip_serializing_if = "Option::is_none")]pub app_name: Option<String>,@@ -49,16 +53,73 @@ pub fn resolve_pid_window_target(pub type WindowTargetCandidates =Arc<dyn Fn(i64) -> Vec<WindowTargetCandidate> + Send + Sync + 'static>;+/// `(pid, x, y)` in the desktop action frame -> the pid's topmost on-screen+/// window covering that point. Lets a desktop-grounded pixel action name a+/// multi-window app by pid alone. Pixels from a capped `get_desktop_state`+/// image were already mapped back by `desktop_capture_scale` at dispatch.+pub type DesktopPointWindowResolver =+ Arc<dyn Fn(i64, f64, f64) -> Option<u64> + Send + Sync + 'static>;++/// `pid` -> the window a pid-only action without a point should mean: the+/// pid's currently active (focused) window, else its topmost on-screen one.+/// Keyboard actions (`type_text`, `hotkey`, `press_key`) name no point, and a+/// multi-window app (GIMP + docks) otherwise refused every one of them.+pub type PidFallbackWindowResolver = Arc<dyn Fn(i64) -> Option<u64> + Send + Sync + 'static>;++/// `(pid, snapshot handle)` -> the window that snapshot was taken of.+pub type SnapshotWindowResolver = Arc<dyn Fn(i64, &str) -> Option<u64> + Send + Sync + 'static>;++/// Desktop-frame `x`/`y` of an action, when the call carries them.+fn desktop_frame_point(args: &Value) -> Option<(f64, f64)> {+ let frame_is_desktop = args.get("coordinate_frame").and_then(Value::as_str) == Some("desktop")+ || args.get("scope").and_then(Value::as_str) == Some("desktop");+ if !frame_is_desktop {+ return None;+ }+ Some((+ args.get("x").and_then(Value::as_f64)?,+ args.get("y").and_then(Value::as_f64)?,+ ))+}+/// Tool decorator that resolves PID-only calls before an action can send input./// Explicit `window_id` and `element_token` targets pass through unchanged.pub struct PidOnlyWindowTargetGuard {inner: Box<dyn Tool>,candidates: WindowTargetCandidates,+ point_resolver: Option<DesktopPointWindowResolver>,+ fallback_resolver: Option<PidFallbackWindowResolver>,+ snapshot_resolver: Option<SnapshotWindowResolver>,}impl PidOnlyWindowTargetGuard {pub fn new(inner: Box<dyn Tool>, candidates: WindowTargetCandidates) -> Self {- Self { inner, candidates }+ Self {+ inner,+ candidates,+ point_resolver: None,+ fallback_resolver: None,+ snapshot_resolver: None,+ }+ }++ /// Resolve an otherwise ambiguous pid-only call that carries no point+ /// (keyboard actions) to the pid's active / topmost window.+ pub fn with_fallback_resolver(mut self, resolver: PidFallbackWindowResolver) -> Self {+ self.fallback_resolver = Some(resolver);+ self+ }++ /// Resolve an otherwise ambiguous pid by the desktop-frame point the+ /// action targets (only for calls that carry desktop-frame `x`/`y`).+ pub fn with_snapshot_resolver(mut self, resolver: SnapshotWindowResolver) -> Self {+ self.snapshot_resolver = Some(resolver);+ self+ }++ pub fn with_point_resolver(mut self, resolver: DesktopPointWindowResolver) -> Self {+ self.point_resolver = Some(resolver);+ self}}@@ -102,7 +163,11 @@ impl Tool for PidOnlyWindowTargetGuard {}async fn invoke(&self, mut args: Value) -> ToolResult {- if args.get("scope").and_then(Value::as_str) == Some("desktop")+ // A windowless desktop-scope action needs no pid window. A desktop-+ // scope action that names a pid still resolves that pid's window (the+ // coordinates are desktop-frame, the target is the window).+ let has_pid = args.get("pid").is_some_and(|value| !value.is_null());+ if (args.get("scope").and_then(Value::as_str) == Some("desktop") && !has_pid)|| args.get("window_id").is_some_and(|value| !value.is_null())|| args.get("element_token")@@ -114,6 +179,23 @@ impl Tool for PidOnlyWindowTargetGuard {let Some(pid) = args.get("pid").and_then(Value::as_i64) else {return self.inner.invoke(args).await;};+ // An element addressed through a snapshot belongs to the window that+ // snapshot was taken of (a popup menu, a dialog): route there rather+ // than to the pid's default window, which would only refuse with+ // "snapshot belongs to window_id X".+ if let (Some(resolver), Some(handle)) = (+ self.snapshot_resolver.clone(),+ args.get("snapshot_id")+ .and_then(Value::as_str)+ .map(str::to_owned),+ ) {+ if let Some(window_id) = resolver(pid, &handle) {+ if let Some(object) = args.as_object_mut() {+ object.insert("window_id".to_owned(), window_id.into());+ }+ return self.inner.invoke(args).await;+ }+ }let candidates = self.candidates.clone();let candidates = match tokio::task::spawn_blocking(move || candidates(pid)).await {Ok(candidates) => candidates,@@ -144,19 +226,137 @@ impl Tool for PidOnlyWindowTargetGuard {}self.inner.invoke(args).await}- PidWindowTargetResolution::Ambiguous(candidates) => ToolResult::error(format!(- "pid {pid} owns more than one eligible top-level window; provide window_id."- ))- .with_structured(serde_json::json!({- "code": "ambiguous_window_target",- "effect": "refused",- "pid": pid,- "candidates": candidates- })),+ PidWindowTargetResolution::Ambiguous(candidates) => {+ if let (Some(resolver), Some((x, y))) =+ (self.point_resolver.clone(), desktop_frame_point(&args))+ {+ let hit = tokio::task::spawn_blocking(move || resolver(pid, x, y))+ .await+ .ok()+ .flatten();+ if let Some(window_id) = hit {+ if let Some(candidate) =+ candidates.iter().find(|c| c.window_id == window_id)+ {+ if let Some(object) = args.as_object_mut() {+ object.insert("window_id".to_owned(), window_id.into());+ }+ let result = self.inner.invoke(args).await;+ return note_resolved_window(result, candidate);+ }+ }+ }+ // A desktop-frame point that the point resolver could not place+ // stays ambiguous: it names a screen location, not a window.+ // A window-local point (or no point at all) is relative to+ // whichever window the call resolves to, and a pid-only call+ // means the pid's focused toplevel — the open dialog when+ // there is one, exactly as a pid-only key resolves.+ let carries_desktop_point = desktop_frame_point(&args).is_some();+ if let (Some(resolver), false) =+ (self.fallback_resolver.clone(), carries_desktop_point)+ {+ let hit = tokio::task::spawn_blocking(move || resolver(pid))+ .await+ .ok()+ .flatten();+ if let Some(window_id) = hit {+ if let Some(candidate) =+ candidates.iter().find(|c| c.window_id == window_id)+ {+ if let Some(object) = args.as_object_mut() {+ object.insert("window_id".to_owned(), window_id.into());+ }+ let result = self.inner.invoke(args).await;+ return note_resolved_window(result, candidate);+ }+ }+ }+ ToolResult::error(format!(+ "pid {pid} owns more than one eligible top-level window: {}. Provide \+ window_id (an open dialog receives keys and window-local x/y; pass its \+ window_id to act on it), or pass desktop-frame x/y over the window you mean.",+ describe_candidates(&candidates)+ ))+ .with_structured(serde_json::json!({+ "code": "ambiguous_window_target",+ "effect": "refused",+ "pid": pid,+ "candidates": candidates+ }))+ }}}}+/// `window_id 7 "Position and Size" (dialog, transient of window 3); window_id 3 "doc - Writer"`+fn describe_candidates(candidates: &[WindowTargetCandidate]) -> String {+ candidates+ .iter()+ .map(|candidate| {+ let kind = match candidate.transient_for {+ Some(owner) => format!("dialog, transient of window {owner}"),+ None => "top-level".to_owned(),+ };+ format!(+ "window_id {} \"{}\" ({kind}{})",+ candidate.window_id,+ candidate.title,+ if candidate.is_on_screen {+ ""+ } else {+ ", off-screen"+ }+ )+ })+ .collect::<Vec<_>>()+ .join("; ")+}++/// A pid-only action was routed to one of several windows: say which one in+/// the text (the model's summary) and in the payload.+fn note_resolved_window(mut result: ToolResult, candidate: &WindowTargetCandidate) -> ToolResult {+ use crate::protocol::Content;+ let note = format!(+ " [pid-only target resolved to window {} \"{}\"{}]",+ candidate.window_id,+ candidate.title,+ if candidate.transient_for.is_some() {+ " (an open dialog)"+ } else {+ ""+ }+ );+ match result.content.iter_mut().find_map(|content| match content {+ Content::Text { text, .. } => Some(text),+ _ => None,+ }) {+ Some(text) => text.push_str(¬e),+ None => result.content.push(Content::text(note.trim().to_owned())),+ }+ let resolved_window = serde_json::json!({+ "window_id": candidate.window_id,+ "title": candidate.title,+ "transient_for": candidate.transient_for,+ });+ match result+ .structured_content+ .as_mut()+ .and_then(Value::as_object_mut)+ {+ Some(object) => {+ object.insert("resolved_window".to_owned(), resolved_window);+ }+ // The inner tool did not attach structured content of its own: give+ // the resolution note somewhere to live rather than dropping it.+ None => {+ result.structured_content =+ Some(serde_json::json!({ "resolved_window": resolved_window }));+ }+ }+ result+}+#[cfg(test)]mod tests {use super::*;@@ -191,6 +391,7 @@ mod tests {fn candidate(window_id: u64) -> WindowTargetCandidate {WindowTargetCandidate {window_id,+ transient_for: None,title: format!("Window {window_id}"),app_name: Some("Editor".into()),is_on_screen: true,@@ -247,6 +448,44 @@ mod tests {assert_eq!(result.structured_content.unwrap()["window_id"], 7);}+ #[tokio::test]+ async fn desktop_scope_with_pid_still_resolves_the_window() {+ let calls = Arc::new(AtomicUsize::new(0));+ let guard = PidOnlyWindowTargetGuard::new(+ Box::new(EchoTool {+ calls: calls.clone(),+ }),+ Arc::new(|_| vec![candidate(7)]),+ );+ let result = guard+ .invoke(serde_json::json!({"pid": 42, "scope": "desktop", "x": 1, "y": 2}))+ .await;+ assert_eq!(calls.load(Ordering::SeqCst), 1);+ let structured = result.structured_content.unwrap();+ assert_eq!(structured["window_id"], 7);+ assert_eq!(structured["scope"], "desktop");+ }++ #[tokio::test]+ async fn windowless_desktop_scope_passes_through() {+ let calls = Arc::new(AtomicUsize::new(0));+ let guard = PidOnlyWindowTargetGuard::new(+ Box::new(EchoTool {+ calls: calls.clone(),+ }),+ Arc::new(|_| vec![candidate(7), candidate(8)]),+ );+ let result = guard+ .invoke(serde_json::json!({"scope": "desktop", "x": 1, "y": 2}))+ .await;+ assert_eq!(calls.load(Ordering::SeqCst), 1);+ assert!(result+ .structured_content+ .unwrap()+ .get("window_id")+ .is_none());+ }+#[tokio::test]async fn explicit_window_and_element_token_targets_pass_through() {for args in [@@ -266,4 +505,247 @@ mod tests {assert_eq!(result.structured_content.unwrap(), expected);}}++ #[tokio::test]+ async fn ambiguous_pid_is_resolved_by_the_desktop_point_when_the_call_carries_one() {+ let calls = Arc::new(AtomicUsize::new(0));+ let candidates: WindowTargetCandidates = Arc::new(|_| {+ vec![+ WindowTargetCandidate {+ window_id: 11,+ transient_for: None,+ title: "a".into(),+ app_name: None,+ is_on_screen: true,+ },+ WindowTargetCandidate {+ window_id: 22,+ transient_for: None,+ title: "b".into(),+ app_name: None,+ is_on_screen: true,+ },+ ]+ });+ let resolver: DesktopPointWindowResolver =+ Arc::new(|_pid, x, _y| if x > 100.0 { Some(22) } else { Some(11) });+ let guard = PidOnlyWindowTargetGuard::new(+ Box::new(EchoTool {+ calls: calls.clone(),+ }),+ candidates,+ )+ .with_point_resolver(resolver);+ // Window-frame pixels: still ambiguous.+ let result = guard+ .invoke(serde_json::json!({"pid": 7, "x": 150, "y": 5}))+ .await;+ assert_eq!(result.is_error, Some(true));+ assert_eq!(calls.load(Ordering::SeqCst), 0);+ // Desktop-frame pixels: resolved to the covering window.+ let result = guard+ .invoke(serde_json::json!({"pid": 7, "x": 150, "y": 5, "coordinate_frame": "desktop"}))+ .await;+ assert_ne!(result.is_error, Some(true), "{result:?}");+ assert_eq!(calls.load(Ordering::SeqCst), 1);+ }++ #[tokio::test]+ async fn ambiguous_pid_without_a_point_falls_back_to_the_active_window() {+ let calls = Arc::new(AtomicUsize::new(0));+ let candidates: WindowTargetCandidates = Arc::new(|_| {+ vec![+ WindowTargetCandidate {+ window_id: 11,+ transient_for: None,+ title: "a".into(),+ app_name: None,+ is_on_screen: true,+ },+ WindowTargetCandidate {+ window_id: 22,+ transient_for: None,+ title: "b".into(),+ app_name: None,+ is_on_screen: true,+ },+ ]+ });+ let fallback: PidFallbackWindowResolver = Arc::new(|_pid| Some(22));+ let guard = PidOnlyWindowTargetGuard::new(+ Box::new(EchoTool {+ calls: calls.clone(),+ }),+ candidates,+ )+ .with_fallback_resolver(fallback);+ // Keyboard-style call (no point): resolved to the active window.+ let result = guard+ .invoke(serde_json::json!({"pid": 7, "text": "hi"}))+ .await;+ assert_ne!(result.is_error, Some(true), "{result:?}");+ assert_eq!(calls.load(Ordering::SeqCst), 1);+ // Window-local pixels resolve the same way: they are relative to the+ // window the call resolves to, and the pid's focused toplevel is it.+ let result = guard+ .invoke(serde_json::json!({"pid": 7, "x": 1, "y": 2}))+ .await;+ assert_ne!(result.is_error, Some(true), "{result:?}");+ assert_eq!(calls.load(Ordering::SeqCst), 2);+ // A desktop-frame point that no window claimed stays ambiguous: it+ // names a screen location, and the fallback window may not be under it.+ let result = guard+ .invoke(serde_json::json!({"pid": 7, "x": 1, "y": 2, "coordinate_frame": "desktop"}))+ .await;+ assert_eq!(result.is_error, Some(true));+ assert_eq!(calls.load(Ordering::SeqCst), 2);+ }++ #[tokio::test]+ async fn pid_only_window_local_point_resolves_to_the_open_dialog_like_a_key() {+ use crate::protocol::Content;+ use std::sync::atomic::{AtomicU64, Ordering};+ struct Echo(Arc<AtomicU64>);+ #[async_trait]+ impl Tool for Echo {+ fn def(&self) -> &ToolDef {+ unreachable!()+ }+ async fn invoke(&self, args: Value) -> ToolResult {+ self.0+ .store(args["window_id"].as_u64().unwrap_or(0), Ordering::SeqCst);+ ToolResult::text("Typed 'hi'.")+ }+ }+ let seen = Arc::new(AtomicU64::new(0));+ let candidates: WindowTargetCandidates = Arc::new(|_| {+ vec![+ WindowTargetCandidate {+ window_id: 3,+ transient_for: None,+ title: "doc - LibreOffice Writer".into(),+ app_name: Some("soffice".into()),+ is_on_screen: true,+ },+ WindowTargetCandidate {+ window_id: 7,+ transient_for: Some(3),+ title: "Position and Size".into(),+ app_name: Some("soffice".into()),+ is_on_screen: true,+ },+ ]+ });+ // The dialog is the pid's focused toplevel.+ let fallback: PidFallbackWindowResolver = Arc::new(|_pid| Some(7));+ let guard = PidOnlyWindowTargetGuard::new(Box::new(Echo(seen.clone())), candidates)+ .with_fallback_resolver(fallback);+ for args in [+ serde_json::json!({"pid": 5, "text": "hi", "x": 120, "y": 80}),+ serde_json::json!({"pid": 5, "x": 120, "y": 80, "coordinate_frame": "window"}),+ ] {+ let result = guard.invoke(args).await;+ assert_ne!(result.is_error, Some(true), "{result:?}");+ assert_eq!(seen.load(Ordering::SeqCst), 7);+ let text = match &result.content[0] {+ Content::Text { text, .. } => text.clone(),+ _ => panic!("text"),+ };+ assert!(+ text.ends_with(+ "[pid-only target resolved to window 7 \"Position and Size\" (an open dialog)]"+ ),+ "{text}"+ );+ let structured = result.structured_content.unwrap();+ assert_eq!(structured["resolved_window"]["window_id"], 7);+ }+ }+}++#[cfg(test)]+mod visibility_tests {+ use super::*;+ use crate::protocol::Content;++ fn candidates() -> Vec<WindowTargetCandidate> {+ vec![+ WindowTargetCandidate {+ window_id: 3,+ transient_for: None,+ title: "doc - LibreOffice Writer".into(),+ app_name: Some("soffice".into()),+ is_on_screen: true,+ },+ WindowTargetCandidate {+ window_id: 7,+ transient_for: Some(3),+ title: "Position and Size".into(),+ app_name: Some("soffice".into()),+ is_on_screen: true,+ },+ ]+ }++ #[test]+ fn ambiguity_names_every_candidate_and_its_transient_owner() {+ let text = describe_candidates(&candidates());+ assert_eq!(+ text,+ "window_id 3 \"doc - LibreOffice Writer\" (top-level); window_id 7 \"Position and Size\" (dialog, transient of window 3)"+ );+ let json = serde_json::to_value(&candidates()[1]).unwrap();+ assert_eq!(json["transient_for"], 3);+ assert!(serde_json::to_value(&candidates()[0])+ .unwrap()+ .get("transient_for")+ .is_none());+ }++ #[tokio::test]+ async fn a_snapshot_id_routes_the_pid_only_action_to_the_snapshot_window() {+ use std::sync::atomic::{AtomicU64, Ordering};+ struct Echo(Arc<AtomicU64>);+ #[async_trait]+ impl Tool for Echo {+ fn def(&self) -> &ToolDef {+ unreachable!()+ }+ async fn invoke(&self, args: Value) -> ToolResult {+ self.0+ .store(args["window_id"].as_u64().unwrap_or(0), Ordering::SeqCst);+ ToolResult::text("ok")+ }+ }+ let seen = Arc::new(AtomicU64::new(0));+ let candidates: WindowTargetCandidates = Arc::new(|_| vec![]);+ let resolver: SnapshotWindowResolver =+ Arc::new(|pid, handle| (pid == 5 && handle == "s00000004").then_some(8395475));+ let guard = PidOnlyWindowTargetGuard::new(Box::new(Echo(seen.clone())), candidates)+ .with_snapshot_resolver(resolver);+ let result = guard+ .invoke(serde_json::json!({"pid": 5, "element_index": 12, "snapshot_id": "s00000004"}))+ .await;+ assert_ne!(result.is_error, Some(true));+ assert_eq!(seen.load(Ordering::SeqCst), 8395475);+ }++ #[test]+ fn auto_resolved_pid_only_actions_say_which_window_received_them() {+ let result =+ ToolResult::text("Pressed Escape.").with_structured(serde_json::json!({"path": "mpx"}));+ let result = note_resolved_window(result, &candidates()[1]);+ let text = match &result.content[0] {+ Content::Text { text, .. } => text.clone(),+ _ => panic!("text"),+ };+ assert_eq!(+ text,+ "Pressed Escape. [pid-only target resolved to window 7 \"Position and Size\" (an open dialog)]"+ );+ let structured = result.structured_content.unwrap();+ assert_eq!(structured["resolved_window"]["window_id"], 7);+ assert_eq!(structured["resolved_window"]["transient_for"], 3);+ assert_eq!(structured["path"], "mpx");+ }}diff --git a/libs/cua-driver/rust/crates/cua-driver-sdk/src/lib.rs b/libs/cua-driver/rust/crates/cua-driver-sdk/src/lib.rsindex 0b531e00b0..f84bab2fff 100644--- a/libs/cua-driver/rust/crates/cua-driver-sdk/src/lib.rs+++ b/libs/cua-driver/rust/crates/cua-driver-sdk/src/lib.rs@@ -2789,6 +2789,7 @@ mod tests {.get_desktop_state(GetDesktopStateInput {session: Some("run-1".into()),screenshot_out_file: None,+ max_image_dimension: None,}).await.unwrap();diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rsindex d323066087..4da4c7f4fe 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rs@@ -875,6 +875,7 @@ fn sdk_window_input(fixture: &Fixture) -> GetWindowStateInput {include_screenshot: Some(false),screenshot_out_file: None,max_elements: None,+ timeout_ms: None,max_depth: None,max_dimension: None,max_image_dimension: None,diff --git a/libs/cua-driver/rust/crates/platform-linux/Cargo.toml b/libs/cua-driver/rust/crates/platform-linux/Cargo.tomlindex f1a3d305b4..ba3c7a15bb 100644--- a/libs/cua-driver/rust/crates/platform-linux/Cargo.toml+++ b/libs/cua-driver/rust/crates/platform-linux/Cargo.toml@@ -38,7 +38,12 @@ libc = "0.2"# subprocess bridge so accessibility input/readback needs no Python or GI# typelibs at runtime. `tokio` matches the driver's async runtime; `zbus`# re-exports the bus types we need (fdo::DBusProxy for pid resolution).-atspi = { version = "0.30", features = ["tokio", "zbus"] }+# `p2p` (a default feature) opens a private socket to every application that+# advertises one; LibreOffice advertises a socket its VCL main loop never+# services, and the crate's peer bookkeeping then wedges every call on the+# shared connection once LibreOffice registers. The session a11y bus is what+# gdbus/pyatspi use and is the only route this crate takes.+atspi = { version = "0.30", default-features = false, features = ["connection", "proxies", "wrappers", "tokio", "zbus"] }evdev = "0.12"# Math-expression evaluator for parallel_mouse_drag `fn` paths (y = f(x)).meval = "0.2"diff --git a/libs/cua-driver/rust/crates/platform-linux/src/atspi/cache.rs b/libs/cua-driver/rust/crates/platform-linux/src/atspi/cache.rsindex af2d263c13..defdcc8475 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/atspi/cache.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/atspi/cache.rs@@ -1,42 +1,200 @@+//! AT-SPI element cache for Linux.+//!+//! Two views of the same `get_window_state` snapshot live here:+//!+//! 1. [`CachedSnapshot`], the [`SnapshotPayload`] published into the runtime+//! element cache (`cua_driver_core::element_cache`). An `element_token` /+//! `element_index` resolves through it to a [`CachedElement`]: the+//! element's proven AT-SPI identity ([`AtspiIdentity`]) plus what a later+//! per-index action needs to act WITHOUT re-walking the application (its+//! D-Bus address ([`native::ObjectRef`]), role, and the screen frame the+//! snapshot reported). Re-walking large trees (LibreOffice, GIMP, Nautilus)+//! was the multi-second stall behind every element-index click.+//!+//! 2. A process-global (pid, xid) side index over the same elements, so the+//! free functions in `atspi` (perform_action_in, get_element_bounds,+//! perform_action_at_point_in, ...) can consult the last snapshot by window+//! without the tool state threaded through every input path.+//!+//! Membership follows the runtime cache rule: only nodes with an+//! `element_index` AND a proven identity are addressable. An unproven+//! identity (a well-known bus name whose owner could not be pinned) is+//! discovery-only.++use super::native::{self, ObjectRef};use super::{AtspiIdentity, AtspiNode};use cua_driver_core::element_cache::{ElementCacheCore, SnapshotPayload};use std::collections::HashMap;+use std::sync::{Arc, Mutex, OnceLock};++#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]+pub struct CacheKey {+ pub pid: u32,+ pub xid: u64,+}++/// One indexed element as the last `get_window_state` snapshot saw it.+#[derive(Debug, Clone)]+pub struct CachedElement {+ /// Opaque AT-SPI path hash (legacy `element_key`).+ pub key: u64,+ /// Proven identity (unique bus owner + owning frame): what an indexed+ /// click acts on; the live ordinal is never used to retarget.+ pub identity: AtspiIdentity,+ /// D-Bus address for the cached fast paths (same object as `identity`).+ pub object_ref: Option<ObjectRef>,+ pub role: String,+ /// The node lives inside an embedded web document (bounds need the+ /// document origin, which only a walk resolves).+ pub in_web_content: bool,+ /// Screen-space frame `(x, y, w, h)` when the snapshot resolved one.+ pub bounds: Option<(i32, i32, u32, u32)>,+}pub struct CachedSnapshot {- elements: HashMap<usize, AtspiIdentity>,+ elements: Arc<HashMap<usize, CachedElement>>,}impl CachedSnapshot {pub fn from_nodes(nodes: &[AtspiNode]) -> Self {- let elements = nodes+ Self::from_nodes_with_bounds(nodes, &[])+ }++ /// `bounds` are the screen frames the same walk produced, keyed by+ /// element_index.+ pub fn from_nodes_with_bounds(+ nodes: &[AtspiNode],+ bounds: &[(usize, i32, i32, u32, u32)],+ ) -> Self {+ let mut elements: HashMap<usize, CachedElement> = nodes.iter()- .filter_map(|node| Some((node.element_index?, node.identity.clone()?)))+ .filter_map(|node| {+ let index = node.element_index?;+ let identity = node.identity.clone()?;+ Some((+ index,+ CachedElement {+ key: node.element_key,+ identity,+ object_ref: node.object_ref.clone(),+ role: node.role.clone(),+ in_web_content: node.in_web_content,+ bounds: None,+ },+ ))+ }).collect();- Self { elements }+ for &(idx, x, y, w, h) in bounds {+ if let Some(element) = elements.get_mut(&idx) {+ element.bounds = Some((x, y, w, h));+ }+ }+ Self {+ elements: Arc::new(elements),+ }}}impl SnapshotPayload for CachedSnapshot {- type Element = AtspiIdentity;+ type Element = CachedElement;fn len(&self) -> usize {self.elements.len()}- fn retain(&self, index: usize) -> Option<AtspiIdentity> {+ fn retain(&self, index: usize) -> Option<CachedElement> {self.elements.get(&index).cloned()}}pub type ElementCache = ElementCacheCore<CachedSnapshot>;+// -- (pid, xid) side index ----------------------------------------------------++type SideIndex = HashMap<CacheKey, Arc<HashMap<usize, CachedElement>>>;++fn store() -> &'static Mutex<SideIndex> {+ static STORE: OnceLock<Mutex<SideIndex>> = OnceLock::new();+ STORE.get_or_init(|| Mutex::new(HashMap::new()))+}++/// Build the payload for (pid, xid) and record its elements in the side index.+/// Returns the payload for the caller to publish into the runtime cache.+pub(crate) fn update_snapshot(+ pid: u32,+ xid: u64,+ nodes: &[AtspiNode],+ bounds: &[(usize, i32, i32, u32, u32)],+) -> CachedSnapshot {+ let snapshot = CachedSnapshot::from_nodes_with_bounds(nodes, bounds);+ store()+ .lock()+ .unwrap()+ .insert(CacheKey { pid, xid }, snapshot.elements.clone());+ snapshot+}++/// Forget the side-index entry for (pid, xid) (the runtime cache is retired+/// separately by the owning `ElementCache`).+pub(crate) fn forget_window(pid: u32, xid: u64) {+ store().lock().unwrap().remove(&CacheKey { pid, xid });+}++fn snapshot_for(pid: u32, xid: Option<u64>) -> Option<Arc<HashMap<usize, CachedElement>>> {+ let store = store().lock().unwrap();+ match xid {+ Some(xid) => store.get(&CacheKey { pid, xid }).cloned(),+ // Callers that lack an xid get the first snapshot found for the pid.+ None => store+ .iter()+ .find(|(key, _)| key.pid == pid)+ .map(|(_, elements)| elements.clone()),+ }+}++/// The cached element at `idx` for (pid, xid); with `xid == None`, the first+/// snapshot recorded for `pid`.+pub(crate) fn cached_element(pid: u32, xid: Option<u64>, idx: usize) -> Option<CachedElement> {+ snapshot_for(pid, xid)?.get(&idx).cloned()+}++/// Hit-test a *screen* point against the cached frames of (pid, xid): the+/// smallest covering real actuator wins, then a passive label. Returns the+/// element index and its cached identity.+pub(crate) fn hit_test(pid: u32, xid: u64, sx: i32, sy: i32) -> Option<(usize, CachedElement)> {+ let elements = snapshot_for(pid, Some(xid))?;+ let mut indexed: Vec<(&usize, &CachedElement)> = elements.iter().collect();+ indexed.sort_by_key(|(idx, _)| **idx);+ let frames: Vec<(usize, i32, i32, u32, u32, bool)> = indexed+ .iter()+ .enumerate()+ .filter_map(|(pos, (_, e))| {+ let (x, y, w, h) = e.bounds?;+ (w > 0 && h > 0).then_some((pos, x, y, w, h, native::is_passive_role(&e.role)))+ })+ .collect();+ let pos = native::select_click_target(&frames, sx, sy)?;+ let (idx, element) = indexed[pos];+ Some((*idx, element.clone()))+}++/// Drop every side-index snapshot for `pid` (e.g. after the process exited).+#[allow(dead_code)]+pub(crate) fn forget_pid(pid: u32) {+ store().lock().unwrap().retain(|key, _| key.pid != pid);+}+#[cfg(test)]mod tests {use super::*;use cua_driver_core::element_token::{token_for, ResolvedElement};fn node(index: usize) -> AtspiNode {+ node_with_role(index, "button")+ }++ fn node_with_role(index: usize, role: &str) -> AtspiNode {AtspiNode {element_index: Some(index),- role: "button".into(),+ role: role.into(),name: None,value: None,checked: None,@@ -54,6 +212,10 @@ mod tests {depth: 0,parent_element_index: None,in_web_content: false,+ object_ref: Some(ObjectRef {+ bus: ":1.1".into(),+ path: format!("/node/{index}"),+ }),}}@@ -66,7 +228,7 @@ mod tests {.resolve_element_args(42, None, Some(&token_for(id, index)), None, None, "click").unwrap();assert!(- matches!(resolved, ResolvedElement::Element { element, .. } if element.path == format!("/node/{index}"))+ matches!(resolved, ResolvedElement::Element { element, .. } if element.identity.path == format!("/node/{index}")));}for index in [0, 1, 8, 999] {@@ -82,10 +244,19 @@ mod tests {unindexed.element_index = None;let payload = CachedSnapshot::from_nodes(&[node(11), unindexed, node(7), node(11)]);assert_eq!(payload.len(), 2);- assert_eq!(payload.retain(7).unwrap().path, "/node/7");- assert_eq!(payload.retain(11).unwrap().path, "/node/11");- assert_eq!(payload.retain(8), None);- assert_eq!(payload.retain(0), None);+ assert_eq!(payload.retain(7).unwrap().identity.path, "/node/7");+ assert_eq!(payload.retain(11).unwrap().identity.path, "/node/11");+ assert!(payload.retain(8).is_none());+ assert!(payload.retain(0).is_none());+ }++ #[test]+ fn unproven_identity_is_discovery_only() {+ let mut unproven = node(3);+ unproven.identity = None;+ let payload = CachedSnapshot::from_nodes(&[unproven, node(4)]);+ assert_eq!(payload.len(), 1);+ assert!(payload.retain(3).is_none());}#[test]@@ -106,7 +277,7 @@ mod tests {.resolve_element_args(42, None, Some(&token_for(current, 5)), None, None, "click").unwrap();assert!(- matches!(current, ResolvedElement::Element { element, .. } if element.path == "/node/replacement")+ matches!(current, ResolvedElement::Element { element, .. } if element.identity.path == "/node/replacement"));}@@ -143,7 +314,7 @@ mod tests {window_id: Some(7),element,..- } if element.path == "/node/3"+ } if element.identity.path == "/node/3"));}@@ -164,7 +335,7 @@ mod tests {).unwrap();assert!(- matches!(target, ResolvedElement::Element { window_id: Some(id), element, .. } if id == window && element.path == "/node/11")+ matches!(target, ResolvedElement::Element { window_id: Some(id), element, .. } if id == window && element.identity.path == "/node/11"));assert!(cache.resolve_element_args(42, None, Some(&token_for(high, 11)), None, Some(7), "click")@@ -183,4 +354,40 @@ mod tests {.resolve_element_args(42, None, Some(&token_for(id, 0)), None, None, "click").is_err());}++ #[test]+ fn snapshot_hit_test_prefers_smallest_real_actuator_and_keeps_refs() {+ let pid = 424_242;+ let nodes = vec![+ node_with_role(0, "panel"),+ node_with_role(1, "push button"),+ node_with_role(2, "label"),+ ];+ let snapshot = update_snapshot(+ pid,+ 77,+ &nodes,+ &[+ (0, 0, 0, 500, 500),+ (1, 100, 100, 50, 20),+ (2, 105, 102, 40, 16),+ ],+ );+ assert_eq!(snapshot.retain(1).unwrap().bounds, Some((100, 100, 50, 20)));+ let (idx, element) = hit_test(pid, 77, 110, 110).expect("hit");+ assert_eq!(idx, 1);+ assert_eq!(+ element.object_ref.as_ref().map(|r| r.path.as_str()),+ Some("/node/1")+ );+ assert!(hit_test(pid, 77, 900, 900).is_none());+ // Index lookup without an xid falls back to any snapshot for the pid.+ assert_eq!(cached_element(pid, None, 2).unwrap().role, "label");+ assert!(cached_element(pid, Some(78), 2).is_none());+ forget_window(pid, 77);+ assert!(cached_element(pid, None, 0).is_none());+ update_snapshot(pid, 77, &nodes, &[]);+ forget_pid(pid);+ assert!(cached_element(pid, None, 0).is_none());+ }}diff --git a/libs/cua-driver/rust/crates/platform-linux/src/atspi/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/atspi/mod.rsindex 89d3d9fe78..4d74d0039c 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/atspi/mod.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/atspi/mod.rs@@ -14,29 +14,7 @@ use anyhow::Result;pub mod cache;pub mod native;pub use cache::ElementCache;-pub use native::{ensure_listener_active, resolve_observed_click_target, ObservedClickTarget};--/// No input has been delivered; this control needs a real pointer click.-#[derive(Debug)]-pub(crate) struct ElementClickNeedsForeground;-impl std::fmt::Display for ElementClickNeedsForeground {- fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {- f.write_str("editable text or table cell selection requires real foreground pointer input")- }-}-impl std::error::Error for ElementClickNeedsForeground {}--/// The requested indexed click has no supported AX activation route. This-/// classification is made before submitting any action, so pointer fallback-/// is safe. Other action errors must not be treated as this pre-input result.-#[derive(Debug)]-pub(crate) struct ClickActionUnavailable(pub String);-impl std::fmt::Display for ClickActionUnavailable {- fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {- f.write_str(&self.0)- }-}-impl std::error::Error for ClickActionUnavailable {}+pub use native::ensure_listener_active;/// Stable address on one AT-SPI bus connection, including the owning frame./// Unique bus names prevent a restarted process from reusing an observed path.@@ -75,6 +53,10 @@ pub struct AtspiNode {/// True when the native AT-SPI walker observed this node below renderer/// web content. Browser-owned consent UI must never match such nodes.pub in_web_content: bool,+ /// D-Bus identity of the node (bus name, object path) for a native AT-SPI+ /// walk; `None` for the X11 property fallback and test fixtures. Lets a+ /// later per-index action re-open the exact object from a cached snapshot.+ pub object_ref: Option<native::ObjectRef>,}pub struct AtspiTreeResult {@@ -93,6 +75,56 @@ pub struct AtspiTreeResult {/// snapshot of a multi-window app carries every window's controls; callers/// that act on behalf of an exact native window must require this.pub window_scoped: bool,+ /// True when the walk stopped before exhausting the application's tree+ /// (deadline, node budget, unresponsive app, application lookup timeout).+ /// `nodes` is then a pre-order *prefix* of the real tree: every index in it+ /// is valid, but elements after the cut are simply absent.+ pub truncated: bool,+ /// Machine-readable reason when `truncated`: `timeout`, `node_budget`,+ /// `app_unresponsive`, `app_lookup_timeout`, `huge_container`.+ pub truncation_reason: Option<String>,+ /// Nodes fully visited by the native walk (0 for the fallback tree).+ pub nodes_visited: usize,+ /// Nodes discovered but not visited when the walk stopped (lower bound).+ pub nodes_pending: usize,+ /// False when the bounds phase ran out of time; some indexed elements then+ /// carry no frame even though they exist.+ pub bounds_complete: bool,+ /// Wall time the native snapshot took (walk + bounds), in milliseconds.+ pub elapsed_ms: u128,+}++/// Total budget for callers that did not ask for one (browser flows, the+/// page tool, `walk_tree`). Bounds the WHOLE operation including retries —+/// previously the cold-registry retry loop could take 4 × 25 s.+pub const DEFAULT_WALK_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(25);++impl AtspiTreeResult {+ fn from_walked(walked: native::WalkedTree, query: Option<&str>) -> Self {+ let md = match query {+ Some(q) => filter_tree(&walked.markdown, q),+ None => walked.markdown,+ };+ let (truncated, truncation_reason, nodes_visited, nodes_pending) = match &walked.truncation+ {+ Some(t) => (true, Some(t.reason.to_owned()), t.visited, t.pending),+ None => (false, None, walked.nodes.len(), 0),+ };+ AtspiTreeResult {+ tree_markdown: md,+ nodes: walked.nodes,+ bounds: walked.bounds,+ trusted: true,+ degraded_reason: None,+ window_scoped: walked.window_scoped,+ truncated,+ truncation_reason,+ nodes_visited,+ nodes_pending,+ bounds_complete: walked.bounds_complete,+ elapsed_ms: walked.elapsed.as_millis(),+ }+ }}/// Walk the AT-SPI tree for a window identified by (pid, xid).@@ -115,14 +147,7 @@ pub(crate) fn walk_tree_for_recording(if let Ok(Some(walked)) = native::walk_tree_bounded_with_timeout(pid, xid, None, None, timeout){if !walked.markdown.is_empty() {- return AtspiTreeResult {- tree_markdown: walked.markdown,- nodes: walked.nodes,- bounds: walked.bounds,- trusted: true,- degraded_reason: None,- window_scoped: walked.window_scoped,- };+ return AtspiTreeResult::from_walked(walked, None);}}walk_via_x11_properties(xid, None)@@ -138,6 +163,28 @@ pub fn walk_tree_bounded(query: Option<&str>,max_elements: Option<usize>,max_depth: Option<usize>,+) -> AtspiTreeResult {+ walk_tree_bounded_within(+ pid,+ xid,+ query,+ max_elements,+ max_depth,+ DEFAULT_WALK_TIMEOUT,+ )+}++/// [`walk_tree_bounded`] with an explicit wall-clock budget for the WHOLE+/// operation: the cold-registry retry loop, the tree walk and the bounds+/// phase all share `timeout`. A walk that runs out of budget returns the+/// partial tree with `truncated = true` rather than nothing.+pub fn walk_tree_bounded_within(+ pid: u32,+ xid: u64,+ query: Option<&str>,+ max_elements: Option<usize>,+ max_depth: Option<usize>,+ timeout: std::time::Duration,) -> AtspiTreeResult {// Native AT-SPI (most complete). On a COLD launch the Qt6 (and some GTK)// AT-SPI bridge registers lazily — the first walk against a freshly@@ -146,58 +193,143 @@ pub fn walk_tree_bounded(// enumerating the app's tree yet. Retry a few times with a short backoff// while the tree is suspiciously root-only, so the first get_window_state// after launch returns the real tree instead of an empty one. See #1927.+ // Every retry runs against what is LEFT of the caller's budget.const MAX_ATTEMPTS: usize = 4;+ const RETRY_BACKOFF: std::time::Duration = std::time::Duration::from_millis(150);+ let started = std::time::Instant::now();let mut native_failure = None;+ let mut last_partial: Option<native::WalkedTree> = None;for attempt in 0..MAX_ATTEMPTS {- match native::walk_tree_bounded(pid, xid, max_elements, max_depth) {+ let remaining = timeout.saturating_sub(started.elapsed());+ if remaining.is_zero() {+ break;+ }+ match native::walk_tree_bounded_with_timeout(pid, xid, max_elements, max_depth, remaining) {Ok(Some(walked)) => {// `nodes.len() <= 1` == only the root window resolved: the// cold-registry symptom. Accept any real tree immediately; only// keep waiting on the degenerate case, and accept it anyway on the// final attempt rather than discarding a (minimal) valid result.+ // A truncated walk is never the cold-registry symptom — retrying+ // it would only burn the budget again.+ let is_last = attempt == MAX_ATTEMPTS - 1+ || timeout.saturating_sub(started.elapsed()) <= RETRY_BACKOFF;if !walked.markdown.is_empty()- && (walked.nodes.len() > 1 || attempt == MAX_ATTEMPTS - 1)+ && (walked.nodes.len() > 1 || walked.truncation.is_some() || is_last){- let md = if let Some(q) = query {- filter_tree(&walked.markdown, q)- } else {- walked.markdown- };- return AtspiTreeResult {- tree_markdown: md,- nodes: walked.nodes,- bounds: walked.bounds,- trusted: true,- degraded_reason: None,- window_scoped: walked.window_scoped,- };+ return AtspiTreeResult::from_walked(walked, query);+ }+ if walked.truncation.is_some() {+ // Out of time before the application even answered.+ last_partial = Some(walked);+ break;+ }+ if !walked.markdown.is_empty() {+ last_partial = Some(walked);}}Ok(None) => {}Err(error) => native_failure = Some(error.to_string()),}if attempt < MAX_ATTEMPTS - 1 {- std::thread::sleep(std::time::Duration::from_millis(150));+ std::thread::sleep(RETRY_BACKOFF);+ }+ }+ if let Some(walked) = last_partial {+ if !walked.markdown.is_empty() || walked.truncation.is_some() {+ let mut result = AtspiTreeResult::from_walked(walked, query);+ if result.nodes.is_empty() {+ // Nothing came back in time: the X11 property tree is the+ // best discovery aid, but say WHY it is all we have.+ let mut fallback = walk_via_x11_properties(xid, query);+ fallback.truncated = true;+ fallback.truncation_reason = result.truncation_reason.take();+ fallback.elapsed_ms = started.elapsed().as_millis();+ fallback.degraded_reason = Some(match fallback.truncation_reason.as_deref() {+ Some("app_unresponsive") => "atspi_app_unresponsive: the application is \+ registered with AT-SPI but did not answer GetChildren; its accessibility \+ bridge may still be initialising (LibreOffice does this on first launch) \+ - retry after a moment or with a larger timeout_ms"+ .to_owned(),+ _ => "atspi_walk_timed_out: the application did not answer AT-SPI within \+ timeout_ms; retry with a larger timeout_ms"+ .to_owned(),+ });+ return fallback;+ }+ return result;}}// Fallback: X11 window properties as minimal tree.let mut fallback = walk_via_x11_properties(xid, query);+ fallback.elapsed_ms = started.elapsed().as_millis();fallback.degraded_reason = native_failure.map(|error| format!("atspi_walk_failed: {error}"));fallback}/// Perform the first advertised action on element `idx` within pid's app tree.-/// Returns `Ok((action_name, suspected_noop))` on success — `suspected_noop`-/// is true when the actuated node looked like a silent no-op (a passive-/// display role, or no advertised action), so the caller can surface-/// `effect: "suspected_noop"`.-pub fn perform_action(pid: u32, idx: usize) -> Result<(String, bool)> {+/// Returns `Ok((action_name, suspected_noop, unacknowledged))` on success —+/// `suspected_noop` is true when the actuated node looked like a silent+/// no-op (a passive display role, or no advertised action), so the caller+/// can surface `effect: "suspected_noop"`. `unacknowledged` is true when the+/// `doAction` D-Bus reply never arrived in time: the toolkit may still be+/// processing the request (e.g. behind a nested dialog loop), so neither+/// acceptance nor effect is known — the caller must report this as+/// `unverifiable` with an explicit note, never as a plain confirmed/dispatched+/// success.+pub fn perform_action(pid: u32, idx: usize) -> Result<(String, bool, bool)> {+ perform_action_in(pid, None, idx)+}++/// [`perform_action`] that first tries the element identity cached by the last+/// `get_window_state` snapshot of (pid, xid) — no re-walk — and only resolves+/// the index against a fresh walk when the cached object is gone.+pub fn perform_action_in(pid: u32, xid: Option<u64>, idx: usize) -> Result<(String, bool, bool)> {+ if let Some(object_ref) = cache::cached_element(pid, xid, idx).and_then(|e| e.object_ref) {+ match native::perform_action_ref(&object_ref) {+ Ok(done) => return Ok(done),+ Err(error) => tracing::debug!(+ "cached element {idx} (pid {pid}) action failed, re-resolving: {error:#}"+ ),+ }+ }native::perform_action(pid, idx)}+/// [`perform_action`] on the exact object a snapshot observed. Never re-walks+/// or retargets by index: when the object is gone the error downcasts to+/// [`native::CachedElementGone`] so the caller can refuse as stale.+pub fn perform_action_observed(element: &cache::CachedElement) -> Result<(String, bool, bool)> {+ let Some(object_ref) = element.object_ref.as_ref() else {+ return Err(+ native::CachedElementGone("observed element has no D-Bus address".into()).into(),+ );+ };+ native::perform_action_ref(object_ref)+}++/// Whether the exact object a snapshot observed is on screen right now+/// (its own state set carries `Showing`). Never re-walks.+pub fn element_showing_observed(element: &cache::CachedElement) -> Result<bool> {+ let Some(object_ref) = element.object_ref.as_ref() else {+ return Err(+ native::CachedElementGone("observed element has no D-Bus address".into()).into(),+ );+ };+ native::element_showing_ref(object_ref)+}+/// Give an indexed AT-SPI element keyboard focus without activating its window.pub fn focus_element(pid: u32, idx: usize) -> Result<bool> {+ if let Some(object_ref) = cache::cached_element(pid, None, idx).and_then(|e| e.object_ref) {+ match native::focus_element_ref(&object_ref) {+ Ok(done) => return Ok(done),+ Err(error) => tracing::debug!(+ "cached element {idx} (pid {pid}) focus failed, re-resolving: {error:#}"+ ),+ }+ }native::focus_element(pid, idx)}@@ -227,8 +359,85 @@ pub fn list_windows(filter_pid: Option<u32>) -> Vec<crate::x11::WindowInfo> {/// click on toolkits (GTK) that drop synthetic X11 pointer events. Returns/// `Ok(Some(action))` when an element was actuated, `Ok(None)` when no/// actionable element covers the point (caller falls back to the X11 path).-pub fn perform_action_at_point(pid: u32, win_x: i32, win_y: i32) -> Result<Option<String>> {- native::perform_action_at_point(pid, win_x, win_y)+pub fn perform_action_at_point(pid: u32, win_x: i32, win_y: i32) -> Result<Option<AtPointHit>> {+ perform_action_at_point_in(pid, 0, win_x, win_y, false)+}++pub use native::is_focus_taking_role;+pub use native::AtPointHit;++/// [`perform_action_at_point`] for a known window, in three rungs that never+/// re-walk the tree first:+/// 1. the frames cached by the last `get_window_state` snapshot of (pid, xid);+/// 2. the toolkit's own `Component.GetAccessibleAtPoint` descent (O(depth));+/// 3. the historical bounded full-walk hit-test (last resort, short budget).+///+/// `skip_focus_roles`: the caller has a real pointer route to fall back to, so+/// an entry / spin button / table cell under the point (whose `doAction` does+/// not give it the keyboard focus a following `type_text` needs) is left+/// alone and `Ok(None)` is returned; see [`is_focus_taking_role`].+pub fn perform_action_at_point_in(+ pid: u32,+ xid: u64,+ win_x: i32,+ win_y: i32,+ skip_focus_roles: bool,+) -> Result<Option<AtPointHit>> {+ if xid != 0 {+ if let Some((ox, oy)) = native::x11_window_origin(xid) {+ if let Some(popup) = crate::input::popup_under_screen_point(win_x + ox, win_y + oy) {+ if popup.window != xid {+ // A popup menu / popover covers the point. The window's+ // own accessibles under it are not what the caller sees;+ // the real pointer press reaches the popup item (or the+ // caller names the popup as window_id and lands here+ // with popup.window == xid).+ tracing::debug!(+ "point ({}, {}) is under {}; skipping the AT-SPI at-point tier for window {xid}",+ win_x + ox,+ win_y + oy,+ popup.describe()+ );+ return Ok(None);+ }+ }+ if let Some((idx, element)) = cache::hit_test(pid, xid, win_x + ox, win_y + oy) {+ if skip_focus_roles && is_focus_taking_role(&element.role) {+ return Ok(None);+ }+ if native::is_container_role(&element.role) {+ // The cached snapshot's smallest covering element is a+ // container (list / icon view / pane): its activation+ // opens the current selection, not the item under the+ // point. Let the live hit-test find the item.+ tracing::debug!(+ "cached hit-test element {idx} (pid {pid}) is a {} container; skipping",+ element.role+ );+ } else if let Some(object_ref) = element.object_ref {+ match native::perform_action_ref(&object_ref) {+ Ok((action, _, _)) => {+ return Ok(Some(AtPointHit::fired(+ action,+ element.role.clone(),+ String::new(),+ object_ref.path.clone(),+ )))+ }+ Err(error) => tracing::debug!(+ "cached hit-test element {idx} (pid {pid}) failed: {error:#}"+ ),+ }+ }+ }+ }+ }+ match native::perform_action_at_point_in(pid, xid, win_x, win_y, skip_focus_roles) {+ Ok(Some(hit)) => return Ok(Some(hit)),+ Ok(None) => {}+ Err(error) => tracing::debug!("GetAccessibleAtPoint hit-test failed: {error:#}"),+ }+ native::perform_action_at_point(pid, win_x, win_y, skip_focus_roles)}/// Resolve a *screen* pixel to the indexable element whose reconstructed screen@@ -255,6 +464,14 @@ pub fn type_into_editable(pid: u32, text: &str) -> Result<()> {/// Type into the exact indexed editable from the caller's accessibility snapshot.pub fn type_into_editable_at(pid: u32, idx: usize, text: &str) -> Result<()> {+ if let Some(object_ref) = cache::cached_element(pid, None, idx).and_then(|e| e.object_ref) {+ match native::type_into_editable_ref(&object_ref, text) {+ Ok(()) => return Ok(()),+ Err(error) => tracing::debug!(+ "cached element {idx} (pid {pid}) editable write failed, re-resolving: {error:#}"+ ),+ }+ }native::type_into_editable_at(pid, idx, text)}@@ -265,6 +482,32 @@ pub fn set_value(pid: u32, idx: usize, value: &str) -> Result<()> {native::set_value(pid, idx, value)}+/// [`set_value`] on the exact object the caller's snapshot indexed, when the+/// snapshot cache still knows it; a fresh walk (same index space as the+/// snapshot of `xid`) otherwise. `Err` messages starting with+/// [`native::NO_VALUE_ROUTE`] mean the element has no accessibility write+/// route at all.+pub fn set_value_in(pid: u32, xid: Option<u64>, idx: usize, value: &str) -> Result<()> {+ if let Some(object_ref) = cache::cached_element(pid, xid, idx).and_then(|e| e.object_ref) {+ match native::set_value_ref(&object_ref, value) {+ Ok(()) => return Ok(()),+ Err(error) if native::is_no_value_route(&error) => return Err(error),+ Err(error) => tracing::debug!(+ "cached element {idx} (pid {pid}) set_value failed, re-resolving: {error:#}"+ ),+ }+ }+ native::set_value(pid, idx, value)+}++/// Read the current value/text of a snapshot-cached element (for the+/// read-back after `set_value`). `None` when the cache has no live object+/// for it or the element exposes neither `Value` nor `Text`.+pub fn read_value_in(pid: u32, xid: Option<u64>, idx: usize) -> Option<String> {+ let object_ref = cache::cached_element(pid, xid, idx)?.object_ref?;+ native::read_value_ref(&object_ref).ok().flatten()+}+/// Insert `text` into a GUI app's editable field via AT-SPI EditableText —/// focus-free and toolkit-agnostic, unlike X11 key injection which only reaches/// the *focused* toplevel's focused widget. Targets the focused editable element@@ -283,7 +526,16 @@ pub fn focused_is_editable(pid: u32) -> Result<Option<bool>> {native::focused_is_editable(pid)}+/// `(role, name)` of the accessible holding the widget focus, from the+/// focus-event log only (no tree walk). See `native::focused_control`.+pub fn focused_control(pid: u32) -> Option<(String, String)> {+ native::focused_control(pid)+}+pub fn get_element_bounds(pid: u32, idx: usize) -> Result<(i32, i32, u32, u32)> {+ if let Some(bounds) = cached_bounds(pid, None, idx) {+ return Ok(bounds);+ }native::get_element_bounds(pid, idx)}@@ -292,9 +544,34 @@ pub fn get_element_bounds_for_window(xid: u64,idx: usize,) -> Result<(i32, i32, u32, u32)> {+ if let Some(bounds) = cached_bounds(pid, Some(xid), idx) {+ return Ok(bounds);+ }native::get_element_bounds_for_window(pid, xid, idx)}+/// Bounds for element `idx` from the last snapshot of (pid, xid): the frame+/// the snapshot recorded, or — when the snapshot ran out of bounds budget —+/// one `GetExtents` on the cached object identity. `None` only when the+/// snapshot never saw the element (or the object is gone), in which case the+/// caller re-walks.+fn cached_bounds(pid: u32, xid: Option<u64>, idx: usize) -> Option<(i32, i32, u32, u32)> {+ let element = cache::cached_element(pid, xid, idx)?;+ if let Some(bounds) = element.bounds {+ return Some(bounds);+ }+ let object_ref = element.object_ref.as_ref()?;+ match native::element_bounds_ref(object_ref, pid, xid.unwrap_or(0), element.in_web_content) {+ Ok(bounds) => Some(bounds),+ Err(error) => {+ tracing::debug!(+ "cached element {idx} (pid {pid}) bounds lookup failed, re-resolving: {error:#}"+ );+ None+ }+ }+}+// ── Internal helpers ─────────────────────────────────────────────────────────/// Minimal X11 property-based tree (fallback when AT-SPI is unavailable).@@ -311,6 +588,12 @@ fn walk_via_x11_properties(xid: u64, query: Option<&str>) -> AtspiTreeResult {trusted: false,degraded_reason: None,window_scoped: false,+ truncated: false,+ truncation_reason: None,+ nodes_visited: 0,+ nodes_pending: 0,+ bounds_complete: true,+ elapsed_ms: 0,}}};@@ -349,6 +632,7 @@ fn walk_via_x11_properties(xid: u64, query: Option<&str>) -> AtspiTreeResult {depth: 0,parent_element_index: None,in_web_content: false,+ object_ref: None,};md.push_str(&format!("- [0] window \"{}\" [actions=[activate]]\n",@@ -373,6 +657,12 @@ fn walk_via_x11_properties(xid: u64, query: Option<&str>) -> AtspiTreeResult {// proving anything a caller acts on.window_scoped: true,degraded_reason: None,+ truncated: false,+ truncation_reason: None,+ nodes_visited: 0,+ nodes_pending: 0,+ bounds_complete: true,+ elapsed_ms: 0,}}@@ -459,3 +749,84 @@ fn filter_tree(markdown: &str, query: &str) -> String {r.push('\n');r}++#[cfg(test)]+mod budget_tests {+ use super::*;++ fn walked(truncation: Option<native::Truncation>) -> native::WalkedTree {+ let node =+ |idx: usize, role: &str, name: &str, depth: usize, actions: Vec<String>| AtspiNode {+ element_index: Some(idx),+ role: role.into(),+ name: Some(name.into()),+ value: None,+ checked: None,+ enabled: Some(true),+ selected: None,+ description: None,+ actions,+ element_key: idx as u64,+ depth,+ parent_element_index: (depth > 0).then_some(0),+ in_web_content: false,+ identity: None,+ object_ref: None,+ };+ native::WalkedTree {+ markdown: "- [0] frame \"Untitled\"\n - [1] push button \"OK\"\n".into(),+ nodes: vec![+ node(0, "frame", "Untitled", 0, vec![]),+ node(1, "push button", "OK", 1, vec!["click".into()]),+ ],+ bounds: vec![(1, 10, 10, 50, 20)],+ window_scoped: true,+ truncation,+ bounds_complete: false,+ elapsed: std::time::Duration::from_millis(1234),+ }+ }++ #[test]+ fn complete_walk_reports_no_truncation_and_counts_nodes() {+ let result = AtspiTreeResult::from_walked(walked(None), None);+ assert!(result.trusted);+ assert!(!result.truncated);+ assert_eq!(result.truncation_reason, None);+ assert_eq!(result.nodes_visited, 2);+ assert_eq!(result.nodes_pending, 0);+ assert!(!result.bounds_complete);+ assert_eq!(result.elapsed_ms, 1234);+ assert_eq!(result.bounds, vec![(1, 10, 10, 50, 20)]);+ }++ #[test]+ fn truncated_walk_carries_reason_and_pending_count_through_query_projection() {+ let result = AtspiTreeResult::from_walked(+ walked(Some(native::Truncation {+ reason: "timeout",+ visited: 2,+ pending: 17,+ })),+ Some("ok"),+ );+ assert!(result.truncated);+ assert_eq!(result.truncation_reason.as_deref(), Some("timeout"));+ assert_eq!(result.nodes_visited, 2);+ assert_eq!(result.nodes_pending, 17);+ // The query projection keeps the matching row plus its ancestor.+ assert!(result.tree_markdown.contains("push button \"OK\""));+ assert!(result.tree_markdown.contains("frame \"Untitled\""));+ // Nodes are the unfiltered prefix: indices stay valid for actuation.+ assert_eq!(result.nodes.len(), 2);+ }++ #[test]+ fn x11_fallback_is_never_truncated_or_trusted() {+ let result = walk_via_x11_properties(0, None);+ assert!(!result.trusted);+ assert!(!result.truncated);+ assert_eq!(result.nodes_visited, 0);+ assert!(result.bounds_complete);+ }+}diff --git a/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs b/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rsindex 660a9169e0..5423aef70e 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs@@ -16,13 +16,15 @@ use std::sync::OnceLock;use std::time::Duration;use anyhow::{anyhow, Context, Result};-use atspi::connection::{AccessibilityConnection, P2P};+use atspi::connection::AccessibilityConnection;use atspi::proxy::accessible::AccessibleProxy;use atspi::proxy::proxy_ext::ProxyExt;use atspi::{CoordType, Interface, State, StateSet};use super::{AtspiIdentity, AtspiNode};+pub(crate) mod hit_test;+/// Per-call D-Bus timeout: a single unresponsive accessible (common in large,/// lazily-built trees like Chromium's) must not stall the whole walk.const CALL_TIMEOUT: Duration = Duration::from_secs(3);@@ -35,10 +37,74 @@ const OP_TIMEOUT: Duration = Duration::from_secs(25);/// listener.const LISTENER_STARTUP_TIMEOUT: Duration = Duration::from_secs(3);-/// Run `fut` with [`CALL_TIMEOUT`]; `None` on timeout so the caller can skip-/// the node and keep walking rather than blocking forever.+tokio::task_local! {+ /// Wall-clock deadline for the AT-SPI operation currently running on the+ /// runtime. Every [`call`] clamps its per-call timeout to what is left of+ /// it, so a 1 s tool budget really is 1 s even when a single node would+ /// otherwise be allowed the full [`CALL_TIMEOUT`]. Set via [`bounded_for`]+ /// / [`walk_tree_bounded_with_timeout`]; absent for legacy callers.+ static OP_DEADLINE: tokio::time::Instant;+}++/// Time left on the current operation's deadline, or [`CALL_TIMEOUT`] when no+/// deadline is in scope.+fn remaining_budget() -> Duration {+ OP_DEADLINE+ .try_with(|deadline| deadline.saturating_duration_since(tokio::time::Instant::now()))+ .unwrap_or(CALL_TIMEOUT)+}++/// True once the current operation's deadline has passed (never for callers+/// that did not set one).+fn deadline_passed() -> bool {+ OP_DEADLINE+ .try_with(|deadline| tokio::time::Instant::now() >= *deadline)+ .unwrap_or(false)+}++/// Run `fut` with [`CALL_TIMEOUT`] (clamped to the operation deadline when one+/// is in scope); `None` on timeout so the caller can skip the node and keep+/// walking rather than blocking forever.async fn call<T>(fut: impl std::future::Future<Output = T>) -> Option<T> {- tokio::time::timeout(CALL_TIMEOUT, fut).await.ok()+ let budget = remaining_budget().min(CALL_TIMEOUT);+ if budget.is_zero() {+ return None;+ }+ tokio::time::timeout(budget, fut).await.ok()+}++/// Run a *synchronous, blocking* closure (typically a raw X11 round-trip via+/// `x11rb::RustConnection`, which has no built-in socket timeout) off the+/// current task and bound how long the caller waits for it.+///+/// This exists because `tokio::time::timeout` around an `.await` only helps+/// when the wrapped future actually yields control back to the runtime at+/// its await points; a plain synchronous call made directly inside an async+/// fn (no `.await` inside it) blocks the executor thread for its full+/// duration regardless of any `timeout`/`timeout_at` wrapped around the+/// *outer* future, because that outer future never gets polled again until+/// the blocking call returns. `get_window_state`'s bounds phase hit exactly+/// this on LibreOffice: `window_to_screen_offset`/`x11_window_origin` open a+/// fresh X11 connection and block on `GetGeometry`/`TranslateCoordinates`+/// replies with no timeout, and when the X server is slow to answer (busy+/// servicing LibreOffice's own heavy repaint traffic) that single call could+/// run for many seconds — well past `timeout_ms` — with none of the AT-SPI+/// D-Bus deadline machinery able to see or bound it (#42).+///+/// `spawn_blocking` moves the closure to a dedicated blocking-pool thread so+/// the timeout here can make the *caller* proceed with `None`; the orphaned+/// thread may still be blocked on the X server afterward (there is no way to+/// cancel a live X11 socket read), but that no longer holds up the walk or+/// consumes the operation's wall-clock budget.+async fn bounded_blocking<T: Send + 'static>(f: impl FnOnce() -> T + Send + 'static) -> Option<T> {+ let budget = remaining_budget().min(CALL_TIMEOUT);+ if budget.is_zero() {+ return None;+ }+ match tokio::time::timeout(budget, tokio::task::spawn_blocking(f)).await {+ Ok(Ok(value)) => Some(value),+ _ => None,+ }}async fn before_snapshot_deadline<T>(@@ -62,15 +128,39 @@ async fn before_snapshot_deadline<T>(fn bounded<T>(work: impl std::future::Future<Output = Result<T>>,on_timeout: impl FnOnce() -> Result<T>,+) -> Result<T> {+ bounded_for(OP_TIMEOUT, work, on_timeout)+}++/// [`bounded`] with an explicit budget. The budget is also installed as the+/// operation deadline, so every inner [`call`] shortens itself to what is+/// left and the walk returns *partial* results before the outer timeout fires+/// (the outer timeout is a backstop for un-wrapped awaits only).+fn bounded_for<T>(+ budget: Duration,+ work: impl std::future::Future<Output = Result<T>>,+ on_timeout: impl FnOnce() -> Result<T>,) -> Result<T> {runtime().block_on(async move {- match tokio::time::timeout(OP_TIMEOUT, work).await {+ let deadline = tokio::time::Instant::now() + budget;+ let backstop = deadline + Duration::from_millis(500);+ match tokio::time::timeout_at(backstop, OP_DEADLINE.scope(deadline, work)).await {Ok(r) => r,Err(_) => on_timeout(),}})}+/// Budget for AT-SPI queries that run *inside* an input action (hit-testing a+/// pixel click, classifying the focused widget, locating an editable). These+/// must never stall the action on a slow app: a miss simply falls through to+/// the next delivery rung.+pub(crate) const INPUT_QUERY_BUDGET: Duration = Duration::from_millis(1500);+/// Budget for resolving an element index against a fresh walk when no cached+/// snapshot answers it. Larger than [`INPUT_QUERY_BUDGET`] because the caller+/// named a specific element and a miss is an error, not a fallback.+pub(crate) const INDEX_RESOLVE_BUDGET: Duration = Duration::from_secs(6);+/// Emit a one-line diagnostic to stderr when `CUA_ATSPI_DEBUG` is set. The/// driver's stderr is surfaced in the test logs, so this is how we see what the/// native walk actually found in CI.@@ -111,9 +201,114 @@ async fn shared_connection() -> Result<&'static AccessibilityConnection> {if let Err(error) = conn.add_registry_event::<atspi::ObjectEvents>().await {dlog!("AT-SPI object-event registration failed: {error}");}+ // Subscribe to focus changes so `focused_is_editable` / typing can+ // find the focused widget in O(1) instead of walking the tree.+ if let Err(error) = conn+ .register_event::<atspi::events::object::StateChangedEvent>()+ .await+ {+ dlog!("AT-SPI state-changed subscription failed: {error}");+ }+ // Grids (LibreOffice Calc) keep the focus on the table and report+ // the current cell as its active descendant.+ if let Err(error) = conn+ .register_event::<atspi::events::object::ActiveDescendantChangedEvent>()+ .await+ {+ dlog!("AT-SPI active-descendant subscription failed: {error}");+ }Ok(conn)}).await+ .inspect(|conn| {+ static TRACKER: OnceLock<()> = OnceLock::new();+ TRACKER.get_or_init(|| spawn_focus_tracker(conn));+ })+}++/// Object path of the accessible that most recently reported `focused=true`,+/// keyed by the emitting application's bus name. Maintained by+/// [`spawn_focus_tracker`]; consulted by the focus-aware typing paths.+fn focus_map() -> &'static std::sync::Mutex<std::collections::HashMap<String, String>> {+ static MAP: OnceLock<std::sync::Mutex<std::collections::HashMap<String, String>>> =+ OnceLock::new();+ MAP.get_or_init(|| std::sync::Mutex::new(std::collections::HashMap::new()))+}++/// Object path of the active descendant most recently reported per bus+/// (`object:active-descendant-changed`): the current cell of a focused grid.+fn active_descendant_map() -> &'static std::sync::Mutex<std::collections::HashMap<String, String>> {+ static MAP: OnceLock<std::sync::Mutex<std::collections::HashMap<String, String>>> =+ OnceLock::new();+ MAP.get_or_init(|| std::sync::Mutex::new(std::collections::HashMap::new()))+}++/// Record the focused-object path for `bus` from one `object:state-changed:focused` event.+fn note_focus_event(bus: &str, path: &str, enabled: bool) {+ let mut map = focus_map().lock().unwrap();+ if enabled {+ map.insert(bus.to_owned(), path.to_owned());+ } else if map.get(bus).is_some_and(|current| current == path) {+ map.remove(bus);+ }+}++/// Consume the shared connection's event stream for the daemon lifetime,+/// remembering the last focused accessible per application bus.+fn spawn_focus_tracker(conn: &'static AccessibilityConnection) {+ runtime().spawn(async move {+ use futures_util::StreamExt;+ let stream = conn.event_stream();+ futures_util::pin_mut!(stream);+ while let Some(event) = stream.next().await {+ if let Ok(atspi::Event::Object(atspi::ObjectEvents::ActiveDescendantChanged(active))) =+ &event+ {+ if let Some(bus) = active.item.name_as_str() {+ active_descendant_map()+ .lock()+ .unwrap()+ .insert(bus.to_owned(), active.descendant.path_as_str().to_owned());+ }+ continue;+ }+ let Ok(atspi::Event::Object(atspi::ObjectEvents::StateChanged(changed))) = event else {+ continue;+ };+ if changed.state != State::Focused {+ continue;+ }+ let Some(bus) = changed.item.name_as_str() else {+ continue;+ };+ note_focus_event(bus, changed.item.path_as_str(), changed.enabled);+ }+ dlog!("AT-SPI focus tracker stream ended");+ });+}++/// The accessible that currently holds keyboard focus inside `pid`'s+/// application, resolved from the focus-event log (no tree walk). `None` when+/// no focus event was seen for the app or the recorded object no longer+/// reports `Focused`.+async fn focused_accessible_via_events<'a>(+ conn: &'a AccessibilityConnection,+ pid: u32,+) -> Option<(AccessibleProxy<'a>, ObjectRef)> {+ let app = app_for_pid(conn, pid).await.ok()??;+ let bus = app.inner().destination().to_string();+ let path = focus_map().lock().unwrap().get(&bus).cloned()?;+ let oref = RawObjectRef {+ name: bus.clone(),+ path: path.clone(),+ };+ let acc = call(accessible_for(conn, &oref)).await?.ok()?;+ let state = call(acc.get_state()).await?.ok()?;+ if !state.contains(State::Focused) {+ dlog!("focus log for {bus} is stale ({path} no longer focused)");+ return None;+ }+ Some((acc, ObjectRef { bus, path }))}/// Establish the process-lifetime listener before accessibility-aware apps are@@ -233,11 +428,28 @@ struct Visited<'a> {/// Display text: the accessible `name`, or — for editable/text widgets that/// expose no name — the Text-interface content (where typed text lives).name: String,+ /// AT-SPI `Description`, read for controls only: Qt publishes a control's+ /// tooltip there ("Play"/"Pause", "Look in"), GTK its tooltip text.+ description: String,value: Option<String>,checked: Option<bool>,enabled: Option<bool>,selected: Option<bool>,selectable: bool,+ /// False when the toolkit's state set is known and marks the widget+ /// neither `Showing` nor `Visible`: it exists in the tree but is hidden+ /// (a widget on an unmapped page). Unknown state sets count as showing.+ showing: bool,+ /// `Visible` but not `Showing`: listed and indexed, but not on screen+ /// right now (GTK3 drops `Showing` for a control scrolled out of its+ /// viewport, and for a revealer child such as Nautilus' hover-only eject+ /// buttons). The tree says so, so the caller can scroll it into view or+ /// prefer an on-screen control.+ offscreen: bool,+ /// Children this walk deliberately did not enumerate: the items of a+ /// menu that is not open (every closed LibreOffice menu would otherwise+ /// put ~500 frameless entries ahead of the document).+ collapsed_children: Option<usize>,actions: Vec<String>,has_editable: bool,has_value: bool,@@ -256,10 +468,41 @@ struct Visited<'a> {/// tree; this is what lets a caller that named an exact native window prove/// which of those windows a node actually lives in.frame_ordinal: usize,+ /// D-Bus identity (bus name + object path) of this accessible, so a later+ /// per-index action can re-open it directly from a cached snapshot instead+ /// of re-walking the application.+ object_ref: ObjectRef,identity: Option<AtspiIdentity>,acc: AccessibleProxy<'a>,}+/// Owned D-Bus identity of an accessible object: `(bus name, object path)`.+#[derive(Clone, Debug, PartialEq, Eq)]+pub struct ObjectRef {+ pub bus: String,+ pub path: String,+}++/// Why a tree walk stopped before exhausting the application's tree.+#[derive(Clone, Debug, PartialEq, Eq)]+pub struct Truncation {+ /// Machine-readable reason: `timeout`, `node_budget`, `app_unresponsive`,+ /// `app_lookup_timeout`.+ pub reason: &'static str,+ /// Nodes fully visited before the walk stopped.+ pub visited: usize,+ /// Nodes discovered but not yet visited when the walk stopped (lower bound+ /// on what is missing — their descendants are unknown).+ pub pending: usize,+}++/// Result of one bounded collection pass.+struct Collected<'a> {+ visited: Vec<Visited<'a>>,+ scoped_frame: Option<usize>,+ truncation: Option<Truncation>,+}+/// Role names that denote embedded web/document content. An editable beneath/// one of these is page content (the field a user means when typing into a/// background browser) rather than browser chrome like the address bar.@@ -292,29 +535,11 @@ fn is_web_process_bus(name: &str) -> bool {/// be walked and written without killing the app. Other toolkits are/// unaffected (they already tolerate `GetAll`), and the sub-interface proxies/// from `proxies()` already use `CacheProperties::No`.-async fn accessible_for<'a>(+/// Build the proxy on the shared bus connection, never the peer socket.+async fn accessible_via_bus<'a>(conn: &'a AccessibilityConnection,oref: &RawObjectRef,) -> Result<AccessibleProxy<'a>> {- // Keep the atspi crate's peer-to-peer path when this connection actually- // knows the peer. Late WebKit WebProcess children are not in the initial- // peer snapshot; object_as_accessible's bus fallback omits their destination- // and targets the Accessible interface name instead. Build an explicit bus- // proxy below for those late peers and for well-known references.- if oref.name.starts_with(':') {- let name = atspi::zbus::names::UniqueName::try_from(oref.name.clone())- .map_err(|e| anyhow!("bad a11y unique name: {e}"))?;- let bus_name = atspi::zbus::names::BusName::Unique(name.as_ref());- if conn.get_peer(&bus_name).is_some() {- let path = atspi::zbus::zvariant::ObjectPath::try_from(oref.path.clone())- .map_err(|e| anyhow!("bad a11y path: {e}"))?;- let object = atspi::ObjectRef::new_owned(name, path);- return conn- .object_as_accessible(&object)- .await- .map_err(|e| anyhow!("AccessibleProxy build failed: {e}"));- }- }AccessibleProxy::builder(conn.connection()).cache_properties(atspi::zbus::proxy::CacheProperties::No).destination(oref.name.clone())@@ -326,6 +551,16 @@ async fn accessible_for<'a>(.map_err(|e| anyhow!("AccessibleProxy build failed: {e}"))}+async fn accessible_for<'a>(+ conn: &'a AccessibilityConnection,+ oref: &RawObjectRef,+) -> Result<AccessibleProxy<'a>> {+ // Always the session a11y bus. WebKitGTK's late WebProcess children use a+ // well-known name that only the explicit bus builder can address, and the+ // crate's peer-to-peer route is disabled (see Cargo.toml).+ accessible_via_bus(conn, oref).await+}+/// AT-SPI's `(so)` object references are documented as unique bus names, but/// WebKitGTK uses its well-known WebProcess name for the embedded web tree./// Keep the wire values as strings while walking so zbus does not reject that@@ -345,6 +580,39 @@ impl RawObjectRef {}}+/// Containers with more children than this are not expanded. LibreOffice+/// Calc's spreadsheet accessible reports the whole sheet (rows x columns, in+/// the billions); a `GetChildren` on it makes the application enumerate every+/// cell, which stalls its accessibility bridge for minutes and wedges every+/// later call from this connection behind it. pyatspi/Orca never enumerate+/// such containers wholesale either. A caller-supplied `max_elements` above+/// this raises the cap (the walk cannot exceed the budget anyway).+pub(crate) const CHILD_ENUMERATION_CAP: usize = 2048;++/// Accessible.ChildCount: one cheap property read that tells a leaf (skip the+/// GetChildren round-trip entirely) from a container that must not be expanded.+async fn raw_child_count(conn: &atspi::zbus::Connection, oref: &RawObjectRef) -> Result<i32> {+ // `cache_properties(No)` is load-bearing (see `accessible_via_bus`): a+ // zbus proxy with the default lazy cache answers this one `ChildCount`+ // read with `Properties.GetAll`, which segfaults Qt5's AT-SPI bridge+ // (VLC died on the first get_window_state walk).+ let proxy = atspi::zbus::proxy::Builder::<atspi::zbus::Proxy<'_>>::new(conn)+ .cache_properties(atspi::zbus::proxy::CacheProperties::No)+ .destination(oref.name.as_str())+ .map_err(|e| anyhow!("bad a11y destination: {e}"))?+ .path(oref.path.as_str())+ .map_err(|e| anyhow!("bad a11y path: {e}"))?+ .interface("org.a11y.atspi.Accessible")+ .map_err(|e| anyhow!("bad a11y interface: {e}"))?+ .build()+ .await+ .map_err(|e| anyhow!("Accessible proxy unavailable: {e}"))?;+ proxy+ .get_property::<i32>("ChildCount")+ .await+ .map_err(|e| anyhow!("Accessible.ChildCount failed: {e}"))+}+/// Pin well-known names to their current unique owner. A missing owner is/// discovery-only: indexed clicks cannot use an unproven persistent identity.async fn identity_ref(@@ -459,11 +727,37 @@ impl<T> ApplicationSelection<T> {}}+/// An application accessible plus the top-level children its selection probe+/// already fetched (`None` when that probe timed out or failed), so callers do+/// not pay a second `GetChildren` round-trip — on an application that never+/// answers it, that was a second full CALL_TIMEOUT.+struct ResolvedApp<'a> {+ app: AccessibleProxy<'a>,+ children: Option<Vec<atspi::ObjectRefOwned>>,+}+/// Locate the application accessible whose backing process is `pid`.async fn app_for_pid<'a>(conn: &'a AccessibilityConnection,pid: u32,) -> Result<Option<AccessibleProxy<'a>>> {+ Ok(resolve_app_for_pid(conn, pid).await?.map(|r| r.app))+}++/// Best-effort executable/process name for `pid`, read from `/proc/<pid>/comm`+/// (Linux only; falls back to `None` on any error, e.g. the process already+/// exited or `/proc` is unavailable in a sandbox).+fn process_name(pid: u32) -> Option<String> {+ std::fs::read_to_string(format!("/proc/{pid}/comm"))+ .ok()+ .map(|s| s.trim().to_string())+ .filter(|s| !s.is_empty())+}++async fn resolve_app_for_pid<'a>(+ conn: &'a AccessibilityConnection,+ pid: u32,+) -> Result<Option<ResolvedApp<'a>>> {let zconn = conn.connection();// Every AT-SPI round-trip below can block on an app whose main loop isn't// servicing D-Bus — most commonly one holding a modal grab (an "Add/Edit/@@ -496,6 +790,13 @@ async fn app_for_pid<'a>(apps.len());let mut selection = ApplicationSelection::new(pid);+ // Retained for a name-based fallback pass below (#2706-adjacent gap): a+ // blind prelaunched app whose AT-SPI application object registered under+ // a *different* pid than the one the caller targets (e.g. a launcher+ // shim re-execs into the real binary, or a toolkit forks once before+ // registering with the a11y bus) is otherwise invisible even though its+ // process name matches exactly.+ let mut other_apps: Vec<(atspi::ObjectRefOwned, u32)> = Vec::new();for child in apps {// A modal-grabbed app can't answer the pid query; skip it after// CALL_TIMEOUT rather than blocking the whole walk on it.@@ -511,6 +812,9 @@ async fn app_for_pid<'a>(};dlog!(" app bus={:?} pid={:?}", child.name_as_str(), cpid);if !selection.matches_pid(cpid) {+ if let Some(cpid) = cpid {+ other_apps.push((child, cpid));+ }continue;}let child = match RawObjectRef::from_atspi(&child) {@@ -528,25 +832,61 @@ async fn app_for_pid<'a>(continue;}};- let has_children = match call(app.get_children()).await {- Some(Ok(children)) => !children.is_empty(),+ let children = match call(app.get_children()).await {+ Some(Ok(children)) => Some(children),Some(Err(error)) => {dlog!(" get_children failed for pid {pid}: {error:#}");- false+ None}None => {dlog!(" get_children timed out for pid {pid}");- false+ None}};+ let has_children = children.as_ref().is_some_and(|c| !c.is_empty());dlog!(" matching app has_children={has_children}");- selection.consider_matching(app, has_children);+ selection.consider_matching(ResolvedApp { app, children }, has_children);}match selection.into_selected() {Ok(Some(app)) => Ok(Some(app)),Ok(None) => {- dlog!("no application accessible matched pid {pid}");- Ok(None)+ dlog!("no application accessible matched pid {pid} by exact pid; trying WM_CLASS/process-name fallback");+ // Exact-pid match failed. Fall back to matching by process+ // executable name across every OTHER AT-SPI-registered+ // application; accept it only when exactly one candidate shares+ // the target's process name, so an ambiguous match still+ // refuses rather than guessing.+ let Some(target_name) = process_name(pid) else {+ return Ok(None);+ };+ let mut name_matches: Vec<(atspi::ObjectRefOwned, u32)> = Vec::new();+ for (child, cpid) in other_apps {+ if process_name(cpid).as_deref() == Some(target_name.as_str()) {+ name_matches.push((child, cpid));+ }+ }+ if name_matches.len() != 1 {+ dlog!(+ "process-name fallback for pid {pid} ({target_name}) found {} candidate(s); refusing",+ name_matches.len()+ );+ return Ok(None);+ }+ let (child, fallback_pid) = name_matches.into_iter().next().unwrap();+ dlog!("process-name fallback matched pid {pid} ({target_name}) to AT-SPI app pid {fallback_pid}");+ let child = match RawObjectRef::from_atspi(&child) {+ Some(child) => child,+ None => return Ok(None),+ };+ let app = match call(accessible_for(conn, &child)).await {+ Some(Ok(app)) => app,+ _ => return Ok(None),+ };+ let children = match call(app.get_children()).await {+ Some(Ok(children)) => Some(children),+ _ => None,+ };+ Ok(Some(ResolvedApp { app, children }))}Err(count) => Err(anyhow!("ambiguous AT-SPI application selection for pid {pid}: \@@ -564,7 +904,7 @@ async fn collect_visited<'a>() -> Result<Option<Vec<Visited<'a>>>> {collect_visited_bounded(conn, pid, 0, None, None).await- .map(|walked| walked.map(|(visited, _)| visited))+ .map(|walked| walked.map(|collected| collected.visited))}/// Screen-space distance between an AT-SPI frame's extents and a native@@ -669,24 +1009,37 @@ async fn resolve_window_frame(// that more certain.return Some(0);}- let window = crate::x11::list_windows(Some(pid))+ // A popup menu / popover is an override-redirect window the WM's client+ // list never carries: describe it from the server directly, and let it+ // correlate with the toolkit's popup toplevels (role `window` for GTK+ // menus, `menu` / `popup menu` elsewhere).+ let (window, popup) = match crate::x11::list_windows(Some(pid)).into_iter()- .find(|candidate| candidate.xid == xid)?;+ .find(|candidate| candidate.xid == xid)+ {+ Some(window) => (window, false),+ None => (+ crate::x11::window_info(xid).filter(|w| w.is_on_screen)?,+ true,+ ),+ };let mut candidates: Vec<(usize, (i32, i32, i32, i32))> = Vec::new();for (ordinal, oref) in seeds.iter().enumerate() {let Some(Ok(acc)) = call(accessible_for(conn, oref)).await else {continue;};// Menus, tooltips and other transients are top-level accessibles too;- // only real windows can correspond to a native window id.+ // only real windows can correspond to a managed native window id.let role = match call(acc.get_role_name()).await {Some(Ok(role)) => role,_ => continue,};- if !matches!(+ let real_window = matches!(role.as_str(),"frame" | "window" | "dialog" | "alert" | "file chooser"- ) {+ );+ let popup_toplevel = matches!(role.as_str(), "window" | "menu" | "popup menu");+ if !(real_window || (popup && popup_toplevel)) {continue;}let Some(Ok(proxies)) = call(acc.proxies()).await else {@@ -722,12 +1075,41 @@ async fn collect_visited_bounded<'a>(xid: u64,max_elements: Option<usize>,max_depth: Option<usize>,-) -> Result<Option<(Vec<Visited<'a>>, Option<usize>)>> {- let app = match app_for_pid(conn, pid).await? {+) -> Result<Option<Collected<'a>>> {+ collect_visited_bounded_opts(conn, pid, xid, max_elements, max_depth, false).await+}++/// [`collect_visited_bounded`] with `walk_closed_menus`: a targeted walk+/// (invoke_menu resolving a path segment by segment) descends into menus+/// the snapshot walk keeps collapsed, since a toolkit that never marks its+/// entries SHOWING (gail) cannot show the walk that a `doAction` opened one.+async fn collect_visited_bounded_opts<'a>(+ conn: &'a AccessibilityConnection,+ pid: u32,+ xid: u64,+ max_elements: Option<usize>,+ max_depth: Option<usize>,+ walk_closed_menus: bool,+) -> Result<Option<Collected<'a>>> {+ let ResolvedApp {+ app,+ children: probed,+ } = match resolve_app_for_pid(conn, pid).await? {Some(a) => a,- None => return Ok(None),+ None => {+ if deadline_passed() {+ // The registry / application did not answer inside the caller's+ // budget. Distinguish that from "no such application" so the+ // caller can report a timeout instead of an absent tree.+ return Err(anyhow!("app_lookup_timeout"));+ }+ return Ok(None);+ }};let zconn = conn.connection();+ // Mapped popups of this pid, read once and only when a menubar menu's+ // open state cannot be told from AT-SPI states alone (gail, focus elsewhere).+ let mut pid_popups: Option<Vec<crate::input::PopupWindow>> = None;// Stack of (object ref, depth, in_web_doc, frame_ordinal). Seed with the// app's windows; push children reversed so siblings pop left-to-right and@@ -736,13 +1118,41 @@ async fn collect_visited_bounded<'a>(// chrome. `frame_ordinal` is the seed's position in `get_children()` order// and is likewise inherited, so every node carries the identity of the// top-level window it belongs to.- let seeds: Vec<RawObjectRef> = match call(app.get_children()).await {- Some(Ok(children)) => children- .into_iter()- .filter_map(|child| RawObjectRef::from_atspi(&child))- .collect(),- _ => Vec::new(),+ let (seeds, app_answered): (Vec<RawObjectRef>, bool) = match probed {+ Some(children) => (+ children+ .iter()+ .filter_map(RawObjectRef::from_atspi)+ .collect(),+ true,+ ),+ None => match call(app.get_children()).await {+ Some(Ok(children)) => (+ children+ .iter()+ .filter_map(RawObjectRef::from_atspi)+ .collect(),+ true,+ ),+ _ => (Vec::new(), false),+ },};+ if !app_answered {+ // The application registered with AT-SPI but never answered+ // `GetChildren`: LibreOffice with its bridge half-initialised, or a+ // modal-grabbed app. Report it as unresponsive rather than as an+ // application without windows — the caller can retry later.+ dlog!("application for pid {pid} did not answer GetChildren");+ return Ok(Some(Collected {+ visited: Vec::new(),+ scoped_frame: None,+ truncation: Some(Truncation {+ reason: "app_unresponsive",+ visited: 0,+ pending: 0,+ }),+ }));+ }// Resolve which seed is the caller's window before walking, from the same// child list the walk is about to seed from. Re-reading `get_children()`@@ -754,13 +1164,32 @@ async fn collect_visited_bounded<'a>(resolve_window_frame(conn, pid, xid, &seeds).await};- let mut stack: Vec<(RawObjectRef, usize, bool, usize)> = seeds+ // Walk the caller's window FIRST when it was resolved: under a short+ // budget an application-wide pre-order (main window, then dialogs) could+ // exhaust the budget before ever reaching the dialog the caller asked+ // about. Ordinals stay the application's own, so `render` still scopes+ // by the same identity; only the visiting order (and thus the index+ // space of this snapshot) changes.+ let mut ordered: Vec<(RawObjectRef, usize, bool, usize)> = seeds.iter().cloned().enumerate().map(|(ordinal, r)| (r, 0usize, false, ordinal))- .rev().collect();+ if let Some(scope) = scoped_frame {+ if let Some(position) = ordered+ .iter()+ .position(|(_, _, _, ordinal)| *ordinal == scope)+ {+ let scoped = ordered.remove(position);+ ordered.insert(0, scoped);+ }+ }+ let mut stack: Vec<(RawObjectRef, usize, bool, usize)> = ordered.into_iter().rev().collect();+ // Direct children of a `menu bar`: a top-level menu whose popup is not+ // open keeps its child count but is not descended into.+ let mut menubar_children: std::collections::HashSet<(String, String)> =+ std::collections::HashSet::new();let mut visited: Vec<Visited<'a>> = Vec::new();let mut identity_owners = std::collections::HashMap::new();@@ -775,24 +1204,39 @@ async fn collect_visited_bounded<'a>(// never happening — bound it here so the walk returns partial within// OP_TIMEOUT for every caller, instead of hanging get_window_state/type_text// on modal dialogs (#1936).- let deadline = std::time::Instant::now() + OP_TIMEOUT;+ // The caller's operation deadline (when one is in scope) wins over the+ // historical OP_TIMEOUT so a 1 s tool budget bounds the whole walk.+ let deadline = {+ let legacy = tokio::time::Instant::now() + OP_TIMEOUT;+ OP_DEADLINE.try_with(|d| (*d).min(legacy)).unwrap_or(legacy)+ };// Fast bail for an app that has stopped answering AT-SPI entirely (modal// grab): if several consecutive nodes each burn the full CALL_TIMEOUT, the// app is unresponsive and the remaining ~OP_TIMEOUT of walking would all// time out too. Give up after a few so type_text falls back to XTEST in a// few seconds rather than ~25s.let mut consecutive_timeouts = 0u32;+ let mut truncation: Option<&'static str> = None;+ // Containers left unexpanded because their child count exceeded the cap:+ // (child count, position in `visited` of the container).+ let mut huge_containers: Vec<(usize, usize)> = Vec::new();+ let child_cap = max_elements+ .map(|cap| cap.max(CHILD_ENUMERATION_CAP))+ .unwrap_or(CHILD_ENUMERATION_CAP);while let Some((oref, depth, inherited_web_doc, frame_ordinal)) = stack.pop() {if budget == 0 {dlog!("node budget exhausted; truncating walk");+ truncation = Some("node_budget");+ stack.push((oref, depth, inherited_web_doc, frame_ordinal));break;}- if std::time::Instant::now() >= deadline {+ if tokio::time::Instant::now() >= deadline {dlog!("collect_visited time budget exhausted; returning partial walk");+ truncation = Some("timeout");+ stack.push((oref, depth, inherited_web_doc, frame_ordinal));break;}- budget -= 1;// WebKitGTK publishes its embedded page on a distinct WebProcess// D-Bus peer and can expose blank role names for the entire subtree.// The peer identity is therefore the reliable document boundary when@@ -819,12 +1263,19 @@ async fn collect_visited_bounded<'a>(continue;}None => {+ if tokio::time::Instant::now() >= deadline {+ truncation = Some("timeout");+ stack.push((oref, depth, inherited_web_doc, frame_ordinal));+ break;+ }consecutive_timeouts += 1;if consecutive_timeouts >= 3 {dlog!("{} consecutive AT-SPI timeouts (accessible_for); app unresponsive, bailing walk",consecutive_timeouts);+ truncation = Some("app_unresponsive");+ stack.push((oref, depth, inherited_web_doc, frame_ordinal));break;}continue;@@ -846,12 +1297,19 @@ async fn collect_visited_bounded<'a>(// A timeout means the app didn't answer in CALL_TIMEOUT. A run of// these means the whole app is wedged — bail so callers fall back.None => {+ if tokio::time::Instant::now() >= deadline {+ truncation = Some("timeout");+ stack.push((oref, depth, inherited_web_doc, frame_ordinal));+ break;+ }consecutive_timeouts += 1;if consecutive_timeouts >= 3 {dlog!("{} consecutive AT-SPI timeouts; app unresponsive, bailing walk",consecutive_timeouts);+ truncation = Some("app_unresponsive");+ stack.push((oref, depth, inherited_web_doc, frame_ordinal));break;}continue;@@ -863,15 +1321,46 @@ async fn collect_visited_bounded<'a>(let has_component = ifaces.contains(Interface::Component);let has_text = ifaces.contains(Interface::Text);+ // A toplevel transient of role `tool tip` is never read or entered:+ // gail (GTK2, e.g. GIMP 2.10) answers `name` on a tooltip-role popup+ // whose child is not a plain label with a `g_message` ("ATK_ROLE_TOOLTIP+ // object found, but doesn't look like a tooltip"), which GIMP shows as+ // a "GIMP Message" dialog. Such a popup exists only after the real+ // pointer hovered a widget (foreground delivery); it carries no+ // actionable element, so skipping it changes nothing else. One extra+ // round-trip per toplevel, not per node.+ if depth == 0 {+ if let Some(Ok(role)) = call(acc.get_role_name()).await {+ if is_tooltip_role(&role) {+ dlog!("skipping tooltip toplevel {}", oref.path);+ continue;+ }+ }+ }// These four are independent — issue them concurrently to cut the// per-node round-trip cost (large trees like Chromium's have hundreds// of nodes, so sequential reads dominate the walk time).- let (role_r, name_r, state_r, children_r) = tokio::join!(+ let (role_r, name_r, state_r, count_r) = tokio::join!(call(acc.get_role_name()),call(acc.name()),call(acc.get_state()),- call(raw_children(zconn, &oref)),+ call(raw_child_count(zconn, &oref)),);+ // Fetch the child list only for real containers within the cap: leaves+ // (the majority of nodes) save a round-trip, and a huge container is+ // never enumerated (see CHILD_ENUMERATION_CAP).+ let children_r: Option<Result<Vec<RawObjectRef>>> = match count_r {+ Some(Ok(count)) if count <= 0 => Some(Ok(Vec::new())),+ Some(Ok(count)) if count as usize > child_cap => {+ huge_containers.push((count as usize, visited.len()));+ Some(Ok(Vec::new()))+ }+ Some(Ok(_)) => call(raw_children(zconn, &oref)).await,+ // Count unavailable (older bridges, error): fall back to the+ // historical unconditional fetch.+ Some(Err(_)) => call(raw_children(zconn, &oref)).await,+ None => None,+ };let role = match role_r {Some(Ok(r)) => r,_ => String::new(),@@ -898,6 +1387,15 @@ async fn collect_visited_bounded<'a>(.as_ref().and_then(|state| state.as_ref().ok()).is_some_and(|state| state.contains(State::Selectable));+ let (showing, offscreen) = state_r+ .as_ref()+ .and_then(|state| state.as_ref().ok())+ .map_or((true, false), |state| listing_state(&role_lower, state));+ let expanded = state_r+ .as_ref()+ .and_then(|state| state.as_ref().ok())+ .is_some_and(|state| state.contains(State::Expanded));+ let under_menubar = menubar_children.remove(&(oref.name.clone(), oref.path.clone()));let selected = if role_lower.contains("check") {checked} else if role_lower.contains("radio")@@ -966,11 +1464,35 @@ async fn collect_visited_bounded<'a>(}}- // Surface Text content as the display name when the widget has no name.- if name.trim().is_empty() && !text_content.trim().is_empty() {+ // Surface Text content as the display name when the widget has no+ // name — for text widgets. A value control (GtkSpinButton implements+ // AtkText with its formatted number) would otherwise be named after+ // its own value, and four spin buttons then all read "0.0".+ if name.trim().is_empty() && !text_content.trim().is_empty() && !has_value {name = text_content;}+ // AT-SPI `Description` for controls (one `Get` per actionable node,+ // never for passive containers/labels): Qt reports tooltips there.+ let is_control = has_action+ || has_value+ || has_editable+ || (has_component && is_control_role(&role_lower));+ let mut description = String::new();+ if is_control {+ if let Some(Ok(text)) = call(acc.description()).await {+ description = text.trim().to_owned();+ }+ }+ // An unnamed value/editable control: its `LABELLED_BY` relation names+ // it (GTK grids, Qt forms with buddies). One relation read plus one+ // name read, only for the controls that need it.+ if name.trim().is_empty() && (has_value || has_editable || is_control_role(&role_lower)) {+ if let Some(label) = labelled_by_name(conn, &acc).await {+ name = label;+ }+ }+// Children inherit web-document context, plus this node's own role.let child_in_web_doc = in_web_doc || is_document_role(&role);@@ -978,9 +1500,98 @@ async fn collect_visited_bounded<'a>(// Honor max_depth (#22865): skip enqueueing descendants whose depth// would exceed the cap.let descend = max_depth.map(|d| depth + 1 <= d).unwrap_or(true);- if descend {+ // A menu that is not open (a menubar entry that is not expanded, or+ // any menu that is not showing) keeps its child count and is not+ // walked: its items are hidden, never indexed, and cost a round-trip+ // each. Open menus (expanded / showing popup contents) still descend.+ let mut closed_menu = !walk_closed_menus+ && role_lower == "menu"+ && (!showing || (under_menubar && !expanded));+ if closed_menu && showing && under_menubar {+ // LibreOffice VCL never sets EXPANDED on an open menubar menu; its+ // items are simply SHOWING. gail (GTK2, e.g. GIMP 2.10) never sets+ // EXPANDED either and never marks a menu entry SHOWING, open or+ // not (its entries are VISIBLE only, closed or open, and the+ // leading tearoff item carries no state at all); what it does+ // publish is FOCUSED on the menubar menu that is popped up. So+ // an open menu is one that is FOCUSED itself, or whose leading+ // children are SHOWING: a few extra state reads per menubar+ // menu. Everything else stays collapsed.+ if focused {+ closed_menu = false;+ } else if let Some(Ok(children)) = &children_r {+ let mut leading: Vec<AccessibleProxy<'_>> = Vec::new();+ for child_ref in children.iter().take(MENU_PEEK_CHILDREN) {+ if let Some(Ok(child)) = call(accessible_for(conn, child_ref)).await {+ if let Some(Ok(state)) = call(child.get_state()).await {+ if is_showing_state(&state) {+ closed_menu = false;+ break;+ }+ }+ leading.push(child);+ }+ }+ if closed_menu {+ // gail with the focus elsewhere (the caller's window is+ // not active, so the popped-up menu is not FOCUSED): the+ // open menu is a mapped override-redirect window of this+ // pid, and its entries' screen extents lie inside it.+ if pid_popups.is_none() {+ let popups =+ tokio::task::spawn_blocking(crate::input::mapped_popup_windows)+ .await+ .unwrap_or_default()+ .into_iter()+ .filter(|popup| popup.pid.is_none_or(|popup_pid| popup_pid == pid))+ .collect::<Vec<_>>();+ pid_popups = Some(popups);+ }+ if let Some(popups) = pid_popups.as_ref().filter(|popups| !popups.is_empty()) {+ for child in &leading {+ let Some(Ok(proxies)) = call(child.proxies()).await else {+ continue;+ };+ let Some(Ok(component)) = call(proxies.component()).await else {+ continue;+ };+ let Some(Ok((ex, ey, ew, eh))) =+ call(component.get_extents(CoordType::Screen)).await+ else {+ continue;+ };+ if ew <= 0 || eh <= 0 || ex < -100_000 || ey < -100_000 {+ continue;+ }+ let (cx, cy) = (ex + ew / 2, ey + eh / 2);+ if popups.iter().any(|popup| {+ cx >= popup.x+ && cy >= popup.y+ && cx < popup.x + popup.width as i32+ && cy < popup.y + popup.height as i32+ }) {+ closed_menu = false;+ break;+ }+ }+ }+ }+ }+ }+ let mut collapsed_children = None;+ if descend && closed_menu {+ if let Some(Ok(children)) = &children_r {+ collapsed_children = Some(children.len());+ }+ }+ if descend && !closed_menu {match children_r {Some(Ok(children)) => {+ if role_lower == "menu bar" {+ for c in &children {+ menubar_children.insert((c.name.clone(), c.path.clone()));+ }+ }for c in children.into_iter().rev() {stack.push((c, depth + 1, child_in_web_doc, frame_ordinal));}@@ -990,15 +1601,34 @@ async fn collect_visited_bounded<'a>(}}+ // Only nodes the caller can act on count against `max_elements`, so+ // a budget of N returns N elements rather than N visited containers.+ if showing+ && is_indexable_capabilities(+ &role,+ !actions.is_empty(),+ has_editable,+ has_value,+ selectable,+ has_component,+ enabled,+ )+ {+ budget -= 1;+ }visited.push(Visited {depth,role,name,+ description,value,checked,enabled,selected,selectable,+ showing,+ offscreen,+ collapsed_children,actions,has_editable,has_value,@@ -1007,6 +1637,10 @@ async fn collect_visited_bounded<'a>(in_web_doc,on_web_process_bus: is_web_process_bus(&oref.name),frame_ordinal,+ object_ref: ObjectRef {+ bus: oref.name.clone(),+ path: oref.path.clone(),+ },identity: object_identity.zip(frame_identity).map(|(object, frame)| AtspiIdentity {@@ -1019,8 +1653,169 @@ async fn collect_visited_bounded<'a>(});}- dlog!("walked pid {pid}: {} node(s)", visited.len());- Ok(Some((visited, scoped_frame)))+ if truncation.is_none() && !huge_containers.is_empty() {+ for (count, at) in &huge_containers {+ dlog!(+ "container {:?} at node {at} has {count} children; not expanded",+ visited.get(*at).map(|v| v.role.as_str()).unwrap_or("?")+ );+ }+ truncation = Some("huge_container");+ }+ let truncation = truncation.map(|reason| Truncation {+ reason,+ visited: visited.len(),+ pending: stack.len(),+ });+ dlog!(+ "walked pid {pid}: {} node(s), truncation={:?}",+ visited.len(),+ truncation+ );+ Ok(Some(Collected {+ visited,+ scoped_frame,+ truncation,+ }))+}++/// Roles whose Description / LABELLED_BY are worth a round-trip even when+/// the node exposes no Action / Value / EditableText interface.+fn is_control_role(role_lower: &str) -> bool {+ matches!(+ role_lower,+ "button"+ | "push button"+ | "toggle button"+ | "combo box"+ | "spin button"+ | "slider"+ | "check box"+ | "radio button"+ | "entry"+ | "text"+ | "password text"+ )+}++/// Value controls the model tells apart by their label; when nothing names+/// one, the tree says so instead of repeating the value as its name.+fn is_value_control_role(role_lower: &str) -> bool {+ matches!(+ role_lower,+ "spin button" | "slider" | "scroll bar" | "progress bar" | "combo box" | "entry" | "text"+ )+}++/// The name of the first `LABELLED_BY` target of `acc`, when it has one.+async fn labelled_by_name(+ conn: &AccessibilityConnection,+ acc: &AccessibleProxy<'_>,+) -> Option<String> {+ let relations = call(acc.get_relation_set()).await?.ok()?;+ for (kind, targets) in relations {+ if kind != atspi::RelationType::LabelledBy {+ continue;+ }+ for target in targets {+ let Some(raw) = RawObjectRef::from_atspi(&target) else {+ continue;+ };+ let Some(Ok(label)) = call(accessible_for(conn, &raw)).await else {+ continue;+ };+ if let Some(Ok(text)) = call(label.name()).await {+ let text = text.trim();+ if !text.is_empty() {+ return Some(text.to_owned());+ }+ }+ }+ }+ None+}++/// Prefix of the description note an unnamed value control carries; the+/// element builder turns it into `unlabelled: true`.+pub const UNLABELLED_NOTE_PREFIX: &str = "unlabelled";++fn ordinal_word(n: usize) -> String {+ let suffix = match (n % 10, n % 100) {+ (1, 11) | (2, 12) | (3, 13) => "th",+ (1, _) => "st",+ (2, _) => "nd",+ (3, _) => "rd",+ _ => "th",+ };+ format!("{n}{suffix}")+}++/// The real AT-SPI parent of every visited node (its position in `visited`),+/// from the pre-order depth sequence: a node at depth `d` ends every open+/// subtree at depth >= `d`. Ancestry never spans two top-levels.+fn true_parents(visited: &[Visited<'_>]) -> Vec<Option<usize>> {+ let mut parents = Vec::with_capacity(visited.len());+ let mut chain: Vec<usize> = Vec::new();+ let mut current_frame: Option<usize> = None;+ for (pos, v) in visited.iter().enumerate() {+ if current_frame != Some(v.frame_ordinal) {+ current_frame = Some(v.frame_ordinal);+ chain.clear();+ }+ chain.truncate(v.depth);+ parents.push(chain.last().copied());+ chain.push(pos);+ }+ parents+}++/// For an unnamed value control at `pos`: the nearest preceding sibling+/// label in the same container (a GTK grid row "Hue:" [spin]) — scanning+/// stops at another control of the same role, so the second row never+/// borrows the first row's label.+fn preceding_sibling_label(+ visited: &[Visited<'_>],+ parents: &[Option<usize>],+ pos: usize,+) -> Option<String> {+ let parent = parents[pos]?;+ let role = visited[pos].role.to_ascii_lowercase();+ for q in (0..pos).rev() {+ if visited[q].depth < visited[pos].depth {+ break;+ }+ if parents[q] != Some(parent) {+ continue;+ }+ let q_role = visited[q].role.to_ascii_lowercase();+ if q_role == role {+ return None;+ }+ if q_role == "label" && !visited[q].name.trim().is_empty() {+ return Some(visited[q].name.trim().to_owned());+ }+ }+ None+}++/// "3rd of 4 spin buttons in this window": the control's place among the+/// showing same-role controls of its top-level, in walk (= visual) order —+/// what a screenshot shows, so the model can map the rows it sees.+fn sibling_ordinal(visited: &[Visited<'_>], _parents: &[Option<usize>], pos: usize) -> String {+ let role = visited[pos].role.to_ascii_lowercase();+ let same_role = |q: usize| {+ visited[q].frame_ordinal == visited[pos].frame_ordinal+ && visited[q].showing+ && visited[q].role.eq_ignore_ascii_case(&role)+ };+ let total = (0..visited.len()).filter(|q| same_role(*q)).count();+ let nth = (0..=pos).filter(|q| same_role(*q)).count();+ let plural = if total == 1 {+ role.clone()+ } else {+ format!("{role}s")+ };+ format!("{} of {total} {plural} in this window", ordinal_word(nth))}/// Render visited nodes into the markdown + node list `walk_tree` returns.@@ -1041,85 +1836,123 @@ fn render(visited: &[Visited<'_>], only_frame: Option<usize>) -> (String, Vec<Atlet mut md = String::new();let mut nodes = Vec::new();let mut idx = 0usize;- let mut current_frame: Option<usize> = None;- // Sparse stack: parent_at_depth[d] = Some(idx) for the actionable node- // most recently emitted at depth d. When a new node appears at depth d,- // its parent_element_index is the closest ancestor at depth < d that has- // an entry. We invalidate deeper entries on each emit so stale siblings- // don't leak across subtrees.- let mut parent_at_depth: Vec<Option<usize>> = Vec::new();+ // The real AT-SPI parent of every node, so `parent_element_index` is the+ // nearest *indexed ancestor* — never a preceding cousin at a lower depth+ // (which parented GIMP's spin buttons to a push button).+ let parents = true_parents(visited);+ let mut index_of: Vec<Option<usize>> = vec![None; visited.len()];- for v in visited {- // Ancestry never spans two top-levels, so a frame change retires every- // recorded parent. Without this a window's first descendants could- // inherit a parent index from the previous window's subtree.- if current_frame != Some(v.frame_ordinal) {- current_frame = Some(v.frame_ordinal);- parent_at_depth.clear();- }+ for (pos, v) in visited.iter().enumerate() {let emit = only_frame.is_none_or(|frame| frame == v.frame_ordinal);let indent = " ".repeat(v.depth);- // Resolve parent: walk parent_at_depth from v.depth-1 down to 0.- let parent_element_index = if v.depth == 0 {- None- } else {- (0..v.depth)- .rev()- .find_map(|d| parent_at_depth.get(d).copied().flatten())+ let parent_element_index = {+ let mut cursor = parents[pos];+ let mut found = None;+ while let Some(p) = cursor {+ if let Some(i) = index_of[p] {+ found = Some(i);+ break;+ }+ cursor = parents[p];+ }+ found};if is_indexable(v) {+ index_of[pos] = Some(idx);if !emit {// Consume the index without emitting: indices stay aligned with// the application-wide walk the actuators perform.idx += 1;continue;}+ let role_lower = v.role.to_ascii_lowercase();+ // An unnamed value control: name it from the sibling label of its+ // row when there is one; otherwise say it is unlabelled and where+ // it sits among its same-role siblings, and never repeat its value+ // as its name.+ let mut name = v.name.clone();+ let mut unlabelled_note = None;+ if name.trim().is_empty() && is_value_control_role(&role_lower) {+ if !v.description.trim().is_empty() {+ // GIMP's spin scales draw their label inside the widget+ // and publish it as the description.+ name = v.description.trim().to_owned();+ } else if let Some(label) = preceding_sibling_label(visited, &parents, pos) {+ name = label;+ } else {+ unlabelled_note = Some(format!(+ "{UNLABELLED_NOTE_PREFIX} {}; {}",+ role_lower,+ sibling_ordinal(visited, &parents, pos)+ ));+ }+ }let act_str = v.actions.join(",");let val_part = match &v.value {Some(val) if !val.is_empty() => format!(" value=\"{val}\""),_ => String::new(),};+ let description_shown = (!v.description.is_empty() && v.description != name)+ .then(|| format!(" (description \"{}\")", v.description));+ let collapsed_note = v+ .collapsed_children+ .map(|n| format!(" (closed menu, {n} items; click it to open)"))+ .unwrap_or_default();+ let unlabelled_shown = unlabelled_note+ .as_ref()+ .map(|note| format!(" ({note})"))+ .unwrap_or_default();md.push_str(&format!(- "{indent}- [{idx}] {role} \"{name}\"{val_part} [actions=[{act_str}]]\n",+ "{indent}- [{idx}] {role} \"{name}\"{desc}{val_part} [actions=[{act_str}]]{collapsed_note}{unlabelled_shown}+",role = v.role,- name = v.name,+ desc = description_shown.as_deref().unwrap_or(""),));- nodes.push(AtspiNode {- element_index: Some(idx),- role: v.role.clone(),- name: if v.name.is_empty() {- None+ let mut description_parts: Vec<String> = Vec::new();+ if !v.description.is_empty() {+ description_parts.push(v.description.clone());+ }+ if let Some(n) = v.collapsed_children {+ description_parts.push(format!(+ "closed menu with {n} items; not expanded: click it to open"+ ));+ }+ if let Some(note) = unlabelled_note {+ description_parts.push(note);+ }+ if v.offscreen {+ description_parts.push(OFFSCREEN_NOTE.to_owned());+ }+ nodes.push(AtspiNode {+ element_index: Some(idx),+ role: v.role.clone(),+ name: if name.trim().is_empty() {+ None} else {- Some(v.name.clone())+ Some(name)},value: v.value.clone().filter(|s| !s.is_empty()),checked: v.checked,enabled: v.enabled,selected: v.selected,- description: None,+ description: (!description_parts.is_empty()).then(|| description_parts.join("; ")),actions: v.actions.clone(),element_key: idx as u64,identity: v.identity.clone(),depth: v.depth,parent_element_index,in_web_content: v.in_web_doc,+ object_ref: Some(v.object_ref.clone()),});- // Record this actionable index at its depth, and invalidate any- // deeper entries from a previous subtree.- while parent_at_depth.len() <= v.depth {- parent_at_depth.push(None);- }- parent_at_depth[v.depth] = Some(idx);- for deeper in (v.depth + 1)..parent_at_depth.len() {- parent_at_depth[deeper] = None;- }idx += 1;- } else if emit && !v.name.is_empty() {+ } else if emit && !v.name.is_empty() && v.showing {md.push_str(&format!(- "{indent}- {role} = \"{name}\"\n",+ "{indent}- {role} = \"{name}\"{marker}+",role = v.role,name = v.name,+ marker = passive_marker(&v.role, !v.actions.is_empty(), v.has_component, v.enabled),));}}@@ -1127,6 +1960,82 @@ fn render(visited: &[Visited<'_>], only_frame: Option<usize>) -> (String, Vec<At(md, nodes)}+/// Why a visible, named control is listed without an index. A disabled+/// button is rendered as `- push button = "Restore" (disabled)` so a caller+/// learns it must enable it (select something) rather than assume the tree+/// dropped it.+fn passive_marker(+ role: &str,+ has_action: bool,+ has_component: bool,+ enabled: Option<bool>,+) -> &'static str {+ let control = has_action+ || (has_component+ && matches!(+ role.trim().to_ascii_lowercase().as_str(),+ "button"+ | "push button"+ | "toggle button"+ | "menu item"+ | "check box"+ | "radio button"+ ));+ if control && enabled == Some(false) {+ " (disabled)"+ } else {+ ""+ }+}++/// GTK, Qt, VCL and the browser bridges publish `Showing` for a widget that+/// is mapped on screen; a hidden widget keeps `Visible` at most. A state set+/// that carries neither is a hidden widget, not an old bridge.+fn is_showing_state(state: &StateSet) -> bool {+ state.contains(State::Showing)+}++/// Menu entries are the exception to the `Showing` gate: gail (GTK2, e.g.+/// GIMP 2.10) never publishes `Showing` on a menu item, open menu or not,+/// and an item in a closed menu is still reachable through its AT-SPI+/// `click` action (that is how a background click opens a GIMP dialog). A+/// `Visible` menu entry therefore stays indexed; a foreground click on one+/// that is not actually on screen takes the action route instead of the+/// pointer (see the click tool), so listing it is safe.+fn is_menu_entry_role(role_lower: &str) -> bool {+ matches!(+ role_lower,+ "menu" | "menu item" | "check menu item" | "radio menu item" | "tear off menu item"+ )+}++fn counts_as_showing(role_lower: &str, state: &StateSet) -> bool {+ is_showing_state(state) || (is_menu_entry_role(role_lower) && state.contains(State::Visible))+}++/// `(listed, offscreen)` for a node with a known state set. A hidden widget+/// (neither showing nor `Visible`) is left out of the tree and the index. A+/// `Visible` widget that is not showing stays listed — an element action can+/// still reach a control scrolled out of view, as on every other platform —+/// and is marked off-screen.+fn listing_state(role_lower: &str, state: &StateSet) -> (bool, bool) {+ if counts_as_showing(role_lower, state) {+ (true, false)+ } else if state.contains(State::Visible) {+ (true, true)+ } else {+ (false, false)+ }+}++/// Description note for an indexed element that is not on screen.+const OFFSCREEN_NOTE: &str =+ "off-screen: scroll it into view before a pixel action; element actions still reach it";++/// How many leading children of a menubar menu the closed-menu peek reads+/// (gail puts a state-less tearoff item first).+const MENU_PEEK_CHILDREN: usize = 3;+/// Format an AT-SPI numeric value like the historical `str(currentValue)`/// (e.g. `1.0`), so `value="..."` fields stay byte-compatible.fn format_value(v: f64) -> String {@@ -1160,15 +2069,16 @@ fn is_enabled_state(state: &StateSet) -> bool {/// (`perform_action`, `set_value`, `get_element_bounds`, snapshot bounds);/// any divergence would desync indices between the snapshot and the operations.fn is_indexable(v: &Visited) -> bool {- is_indexable_capabilities(- &v.role,- !v.actions.is_empty(),- v.has_editable,- v.has_value,- v.selectable,- v.has_component,- v.enabled,- )+ v.showing+ && is_indexable_capabilities(+ &v.role,+ !v.actions.is_empty(),+ v.has_editable,+ v.has_value,+ v.selectable,+ v.has_component,+ v.enabled,+ )}fn is_indexable_capabilities(@@ -1210,6 +2120,13 @@ pub struct WalkedTree {/// top-level and the snapshot contains only that window's nodes. False/// means the snapshot spans every window the application publishes.pub window_scoped: bool,+ /// Set when the walk stopped before exhausting the tree.+ pub truncation: Option<Truncation>,+ /// False when the bounds phase ran out of time; some indexed elements+ /// then carry no frame even though they exist.+ pub bounds_complete: bool,+ /// Wall time of the walk + bounds phases.+ pub elapsed: Duration,}/// Walk the AT-SPI tree with caller-supplied node + depth caps.@@ -1234,44 +2151,95 @@ pub(super) fn walk_tree_bounded_with_timeout(runtime().block_on(async {// Tree traversal and bounds collection form one snapshot. Keep one// deadline for both phases so a dead AT-SPI peer cannot outlive the- // operation timeout while resolving geometry.- let deadline = tokio::time::Instant::now() + timeout;+ // operation timeout while resolving geometry. The deadline is installed+ // as the task-local operation deadline so every per-node call clamps+ // to it and the walk returns *partial* results instead of being+ // dropped wholesale by the outer timeout.let walk_started = std::time::Instant::now();- let walk = async {+ let deadline = tokio::time::Instant::now() + timeout;+ // Backstop only: the walk observes `deadline` itself and returns its+ // partial result; this guards the few un-wrapped awaits.+ let backstop = deadline + Duration::from_millis(500);+ let walk = OP_DEADLINE.scope(deadline, async {let conn = shared_connection().await?;collect_visited_bounded(conn, pid, xid, max_elements, max_depth).await- };- let walked = match before_snapshot_deadline(deadline, walk).await {- Ok(result) => result?,+ });+ let walked = match before_snapshot_deadline(backstop, walk).await {+ Ok(Ok(result)) => result,+ Ok(Err(error)) if error.to_string() == "app_lookup_timeout" => {+ dlog!("walk_tree: application lookup timed out for pid {pid}");+ return Ok(Some(WalkedTree {+ markdown: String::new(),+ nodes: Vec::new(),+ bounds: Vec::new(),+ window_scoped: false,+ truncation: Some(Truncation {+ reason: "app_lookup_timeout",+ visited: 0,+ pending: 0,+ }),+ bounds_complete: false,+ elapsed: walk_started.elapsed(),+ }));+ }+ Ok(Err(error)) => return Err(error),Err(_) => {dlog!("walk_tree timed out for pid {pid}");- return Ok(None);+ return Ok(Some(WalkedTree {+ markdown: String::new(),+ nodes: Vec::new(),+ bounds: Vec::new(),+ window_scoped: false,+ truncation: Some(Truncation {+ reason: "timeout",+ visited: 0,+ pending: 0,+ }),+ bounds_complete: false,+ elapsed: walk_started.elapsed(),+ }));}};- let Some((visited, scoped_frame)) = walked else {+ let Some(Collected {+ visited,+ scoped_frame,+ truncation,+ }) = walked+ else {return Ok(None);};let walk_elapsed = walk_started.elapsed();let (markdown, nodes) = render(&visited, scoped_frame);let bounds_started = std::time::Instant::now();- let bounds = match before_snapshot_deadline(- deadline,- element_bounds_for_visited(&visited, pid, xid, scoped_frame),+ // Bounds resolution shares the deadline, but a walk that consumed the+ // whole budget must still yield *some* frames (element-index clicks+ // resolve against them without a re-walk), so grant the bounds phase a+ // small grace of up to half the budget. The tool documents this.+ let bounds_deadline = deadline+ .max(tokio::time::Instant::now() + (timeout / 2).min(Duration::from_millis(1500)));+ let bounds_backstop = bounds_deadline + Duration::from_millis(500);+ let (bounds, bounds_complete) = match before_snapshot_deadline(+ bounds_backstop,+ OP_DEADLINE.scope(+ bounds_deadline,+ element_bounds_for_visited(&visited, pid, xid, scoped_frame),+ ),).await{- Ok(bounds) => bounds,+ Ok((bounds, complete)) => (bounds, complete),Err(_) => {dlog!("element bounds timed out for pid {pid}");- Vec::new()+ (Vec::new(), false)}};dlog!(- "snapshot phases pid {pid}: walk_ms={} bounds_ms={} visited={} bounds={}",+ "snapshot phases pid {pid}: walk_ms={} bounds_ms={} visited={} bounds={} complete={}",walk_elapsed.as_millis(),bounds_started.elapsed().as_millis(),visited.len(),- bounds.len()+ bounds.len(),+ bounds_complete);// Bounds are keyed by the application-wide element index, so drop the// entries for windows this snapshot no longer shows.@@ -1290,6 +2258,9 @@ pub(super) fn walk_tree_bounded_with_timeout(nodes,bounds,window_scoped: scoped_frame.is_some(),+ truncation,+ bounds_complete,+ elapsed: walk_started.elapsed(),}))})}@@ -1315,8 +2286,34 @@ pub fn list_windows(filter_pid: Option<u32>) -> Vec<crate::x11::WindowInfo> {list_windows_blocking(filter_pid)}+/// Stacking position (`_NET_CLIENT_LIST_STACKING`, bottom-to-top) for an+/// AT-SPI frame, joined to the X11 toplevels of the same pid by title (the+/// AT-SPI xid is synthetic). `None` when X11 does not know the window.+fn z_index_from_stacking(+ stacking: &[crate::x11::WindowInfo],+ pid: u32,+ title: &str,+) -> Option<usize> {+ let same_pid: Vec<&crate::x11::WindowInfo> =+ stacking.iter().filter(|w| w.pid == Some(pid)).collect();+ same_pid+ .iter()+ .find(|w| w.title == title)+ .copied()+ .or_else(|| (same_pid.len() == 1).then(|| same_pid[0]))+ .and_then(|w| w.z_index)+}+fn list_windows_blocking(filter_pid: Option<u32>) -> Vec<crate::x11::WindowInfo> {use crate::x11::WindowInfo;+ // X11 knows the stacking order the AT-SPI registry does not; on an X+ // session the frames get their `z_index` from it so callers can rank+ // an app's dialog above the main window it covers.+ let stacking: Vec<WindowInfo> = if crate::wayland::is_wayland() {+ Vec::new()+ } else {+ crate::x11::list_windows(filter_pid)+ };runtime().block_on(async {let work = async {let conn = shared_connection().await?;@@ -1413,13 +2410,14 @@ fn list_windows_blocking(filter_pid: Option<u32>) -> Vec<crate::x11::WindowInfo>.unwrap_or((observed_x, observed_y));// Stable, non-zero, unique per (pid, frame ordinal).let xid = (((cpid as u64) << 16) | (i as u64)).max(1);+ let z_index = z_index_from_stacking(&stacking, cpid, &title);out.push(WindowInfo {xid,pid: Some(cpid),app_name: app_name.clone(),title,is_on_screen: width > 0 && height > 0,- z_index: None,+ z_index,x,y,width,@@ -1467,19 +2465,28 @@ fn list_windows_blocking(filter_pid: Option<u32>) -> Vec<crate::x11::WindowInfo>/// chrome),/// 3. the first editable anywhere (covers single-field apps like a GTK dialog/// entry, or a GTK4 GtkEntry).-fn pick_editable<'v, 'a>(visited: &'v [Visited<'a>]) -> Option<&'v Visited<'a>> {+/// Choose the editable to write into. `complete` says whether `visited` is the+/// whole tree: on a *partial* walk the "first editable anywhere" fallback is+/// withheld, because the field the user means (the focused one) may simply not+/// have been reached yet and writing into an arbitrary earlier field (a name+/// box, a search entry) would be a silent misdelivery.+fn pick_editable<'v, 'a>(visited: &'v [Visited<'a>], complete: bool) -> Option<&'v Visited<'a>> {visited.iter().find(|v| v.has_editable && v.focused).or_else(|| visited.iter().find(|v| v.has_editable && v.in_web_doc))- .or_else(|| visited.iter().find(|v| v.has_editable))+ .or_else(|| {+ complete+ .then(|| visited.iter().find(|v| v.has_editable))+ .flatten()+ })}/// Try to write `text` into the best editable node in `visited` via AT-SPI/// EditableText. Returns `Ok(true)` if the write landed,/// `Ok(false)` if no editable was found / the EditableText write was rejected.-async fn write_into_editable(visited: &[Visited<'_>], text: &str) -> Result<bool> {- let target = match pick_editable(visited) {+async fn write_into_editable(visited: &[Visited<'_>], complete: bool, text: &str) -> Result<bool> {+ let target = match pick_editable(visited, complete) {Some(t) => t,None => return Ok(false),};@@ -1494,9 +2501,18 @@ async fn write_into_editable_target(target: &Visited<'_>, text: &str) -> Result<target.focused,target.has_component);+ write_into_editable_acc(&target.acc, target.has_component, text).await+}- let proxies = target- .acc+/// EditableText write against a bare accessible (from a cached snapshot ref or+/// the focus-event log). `has_component` gates the GrabFocus retry.+async fn write_into_editable_acc(+ acc: &AccessibleProxy<'_>,+ has_component: bool,+ text: &str,+) -> Result<bool> {+ let target_has_component = has_component;+ let proxies = acc.proxies().await.map_err(|e| anyhow!("interface proxies unavailable: {e}"))?;@@ -1512,13 +2528,13 @@ async fn write_into_editable_target(target: &Visited<'_>, text: &str) -> Result<// This should give the widget internal keyboard focus without activating// the window, allowing GTK4 (and similar toolkits) to expose EditableText// on an unfocused window's focused widget.- if target.has_component {+ if target_has_component {if let Ok(comp) = proxies.component().await {match call(comp.grab_focus()).await {- Some(Ok(true)) => dlog!("GrabFocus succeeded on {:?}", target.role),- Some(Ok(false)) => dlog!("GrabFocus returned false on {:?}", target.role),- Some(Err(e)) => dlog!("GrabFocus failed on {:?}: {}", target.role, e),- None => dlog!("GrabFocus timed out on {:?}", target.role),+ Some(Ok(true)) => dlog!("GrabFocus succeeded"),+ Some(Ok(false)) => dlog!("GrabFocus returned false"),+ Some(Err(e)) => dlog!("GrabFocus failed: {}", e),+ None => dlog!("GrabFocus timed out"),}} else {dlog!("Component interface unavailable despite has_component=true");@@ -1557,13 +2573,22 @@ async fn write_through_editable_proxies(/// Write into the best editable exposed by the current AT-SPI tree without/// falling through to synthetic X11 input.pub fn type_into_editable(pid: u32, text: &str) -> Result<()> {- bounded(+ bounded_for(+ INPUT_QUERY_BUDGET,async {let conn = shared_connection().await?;- let visited = collect_visited(conn, pid)+ // Focused editable known from the event log: write there directly.+ if let Some(true) = write_into_focused_editable(conn, pid, text).await {+ return Ok(());+ }+ let Collected {+ visited,+ truncation,+ ..+ } = collect_visited_bounded(conn, pid, 0, None, None).await?.ok_or_else(|| anyhow!("no AT-SPI application for pid {pid}"))?;- if write_into_editable(&visited, text).await? {+ if write_into_editable(&visited, truncation.is_none(), text).await? {Ok(())} else {Err(anyhow!("no writable AT-SPI element found for pid {pid}"))@@ -1575,7 +2600,8 @@ pub fn type_into_editable(pid: u32, text: &str) -> Result<()> {/// Write into the exact indexed editable exposed by the caller's snapshot.pub fn type_into_editable_at(pid: u32, idx: usize, text: &str) -> Result<()> {- bounded(+ bounded_for(+ INDEX_RESOLVE_BUDGET,async {let conn = shared_connection().await?;let visited = collect_visited(conn, pid)@@ -1620,13 +2646,22 @@ pub fn type_into_editable_at(pid: u32, idx: usize, text: &str) -> Result<()> {}pub fn insert_text(pid: u32, text: &str) -> Result<bool> {- bounded(+ bounded_for(+ INPUT_QUERY_BUDGET,async {let conn = shared_connection().await?;- let visited = match collect_visited(conn, pid).await? {+ if let Some(true) = write_into_focused_editable(conn, pid, text).await {+ return Ok(true);+ }+ let Collected {+ visited,+ truncation,+ ..+ } = match collect_visited_bounded(conn, pid, 0, None, None).await? {Some(v) => v,None => return Ok(false),};+ let complete = truncation.is_none();dlog!("insert_text: {} node(s), {} editable, {} entry/text-role",@@ -1643,9 +2678,16 @@ pub fn insert_text(pid: u32, text: &str) -> Result<bool> {// keyboard focus without activating the window, so toolkits that expose// EditableText on an unfocused window (Qt6, and GTK4 with GTK_A11Y=atspi)// accept the write here.- if write_into_editable(&visited, text).await? {+ if write_into_editable(&visited, complete, text).await? {return Ok(true);}+ if !complete {+ // A partial walk cannot prove there is no better field further+ // down the tree; the click+type guess below would be a silent+ // misdelivery. Let the caller refuse honestly instead.+ dlog!("insert_text: walk incomplete; skipping the entry-guess fallback");+ return Ok(false);+ }// GTK3 fallback: the toolkit exposes entry/text nodes in the tree (so// get_text reads work) but gates EditableText on focus/activation. Try@@ -1732,9 +2774,16 @@ pub fn insert_text(pid: u32, text: &str) -> Result<bool> {/// `None` — nothing is focused (or the app is unreachable): fall back to/// the focus-free "first editable" path for background typing.pub fn focused_is_editable(pid: u32) -> Result<Option<bool>> {- bounded(+ bounded_for(+ INPUT_QUERY_BUDGET,async {let conn = shared_connection().await?;+ // O(1) answer from the focus-event log when available.+ if let Some((acc, _)) = focused_accessible_via_events(conn, pid).await {+ if let Some(Ok(ifaces)) = call(acc.get_interfaces()).await {+ return Ok(Some(ifaces.contains(Interface::EditableText)));+ }+ }let visited = match collect_visited(conn, pid).await? {Some(v) => v,None => return Ok(None),@@ -1745,6 +2794,81 @@ pub fn focused_is_editable(pid: u32) -> Result<Option<bool>> {)}+/// Role and name of the accessible holding the widget focus in `pid`, from+/// the focus-event log only (no walk): cheap enough to run after every+/// pointer press, so a grid click can name the cell it landed on+/// (LibreOffice reports "D2"). `None` when the log has nothing current.+pub fn focused_control(pid: u32) -> Option<(String, String)> {+ bounded_for(+ Duration::from_millis(400),+ async {+ let conn = shared_connection().await?;+ let Some((acc, focused_ref)) = focused_accessible_via_events(conn, pid).await else {+ return Ok(None);+ };+ let (role, name) = tokio::join!(call(acc.get_role_name()), call(acc.name()));+ let (Some(Ok(role)), Some(Ok(name))) = (role, name) else {+ return Ok(None);+ };+ // LibreOffice keeps the focus on the sheet's `table`; the current+ // cell is its active descendant (event log), else its selected+ // child.+ if role.eq_ignore_ascii_case("table") {+ let active = active_descendant_map()+ .lock()+ .unwrap()+ .get(&focused_ref.bus)+ .cloned();+ if let Some(path) = active {+ let raw = RawObjectRef {+ name: focused_ref.bus.clone(),+ path,+ };+ if let Some(Ok(cell)) = call(accessible_for(conn, &raw)).await {+ let (cell_role, cell_name) =+ tokio::join!(call(cell.get_role_name()), call(cell.name()));+ if let (Some(Ok(cell_role)), Some(Ok(cell_name))) = (cell_role, cell_name) {+ if !cell_name.trim().is_empty() {+ return Ok(Some((cell_role, cell_name)));+ }+ }+ }+ }+ if let Some(Ok(proxies)) = call(acc.proxies()).await {+ if let Some(Ok(selection)) = call(proxies.selection()).await {+ if let Some(Ok(n)) = call(selection.n_selected_children()).await {+ if n > 0 {+ if let Some(Ok(child)) = call(selection.get_selected_child(0)).await+ {+ if let Some(raw) = RawObjectRef::from_atspi(&child) {+ if let Some(Ok(cell)) =+ call(accessible_for(conn, &raw)).await+ {+ let (cell_role, cell_name) = tokio::join!(+ call(cell.get_role_name()),+ call(cell.name())+ );+ if let (Some(Ok(cell_role)), Some(Ok(cell_name))) =+ (cell_role, cell_name)+ {+ return Ok(Some((cell_role, cell_name)));+ }+ }+ }+ }+ }+ }+ }+ }+ }+ Ok(Some((role, name)))+ },+ || Ok(None),+ )+ .ok()+ .flatten()+}+/// Find the window XID for a PID by listing its X11 windows.async fn entry_find_window_xid(pid: u32) -> Option<u64> {use crate::x11::list_windows;@@ -1839,6 +2963,118 @@ fn activation_index(role: &str, actions: &[String]) -> Option<usize> {})}+/// Container roles whose `activate` / `open` / `menu` action acts on the+/// container's *current selection* (Nautilus' `layered pane "Icon View"`+/// opens the selected file; a GTK `list` activates its selected row). A+/// bubbling at-point walk must never fire these: the user clicked an item,+/// not "open whatever is selected".+pub(crate) fn is_container_role(role: &str) -> bool {+ matches!(+ role.trim().to_ascii_lowercase().as_str(),+ "layered pane"+ | "list"+ | "list box"+ | "table"+ | "tree"+ | "tree table"+ | "icon view"+ | "panel"+ | "scroll pane"+ | "viewport"+ )+}++/// Item roles a click selects rather than activates when they expose no+/// Action of their own (Nautilus file `canvas` items, GTK list rows / cells).+/// A spreadsheet / grid cell. Selecting one through its table's `Selection`+/// interface marks it (LibreOffice Calc highlights the cell) but does not+/// move the cell cursor, so text typed afterwards still lands in the+/// previously active cell (A1 on a fresh sheet). With a real pointer+/// available, a click on a cell must be a real press.+fn is_cell_role(role: &str) -> bool {+ matches!(+ role.trim().to_ascii_lowercase().as_str(),+ "table cell" | "cell"+ )+}++fn is_selectable_item_role(role: &str) -> bool {+ matches!(+ role.trim().to_ascii_lowercase().as_str(),+ "canvas" | "list item" | "table cell" | "cell" | "icon" | "tree item" | "table row" | "row"+ )+}++/// The at-point activation rule. The deepest node under the point may fire+/// any activation it advertises unless it is a container (a click on a+/// container's empty area must not open its current selection); an ancestor+/// may only be fired when it is neither a container nor a `canvas` (a drawing+/// surface whose `activate` means "open the selection" once it has children).+fn at_point_activation_index(+ role: &str,+ actions: &[String],+ is_deepest_hit: bool,+) -> Option<usize> {+ if is_container_role(role) {+ return None;+ }+ if !is_deepest_hit && role.trim().eq_ignore_ascii_case("canvas") {+ return None;+ }+ activation_index(role, actions)+}++/// What an at-point accessibility click did, for the tool result.+#[derive(Debug, Clone, PartialEq, Eq)]+pub struct AtPointHit {+ /// AT-SPI action name fired (`activate`, `click`), or `select_child`+ /// when the item was selected through its container's Selection+ /// interface instead.+ pub action: String,+ pub role: String,+ pub name: String,+ /// D-Bus object path of the node acted on.+ pub path: String,+ /// Set on the Selection route: the item that is now selected.+ pub selected: Option<String>,+ /// Selection route only: `Selection.IsChildSelected` read back true.+ pub selection_verified: bool,+}++impl AtPointHit {+ pub fn fired(action: String, role: String, name: String, path: String) -> Self {+ Self {+ action,+ role,+ name,+ path,+ selected: None,+ selection_verified: false,+ }+ }++ /// `hit: push button "Restore" action=activate path=...` /+ /// `selected: canvas "file.txt" ...`.+ pub fn describe(&self) -> String {+ match &self.selected {+ Some(item) => format!(+ "selected: {} \"{}\" (Selection.SelectChild on its container{})",+ self.role,+ item,+ if self.selection_verified {+ ", read back as selected"+ } else {+ ", selection not read back"+ }+ ),+ None => format!(+ "hit: {} \"{}\" action={} path={}",+ self.role, self.name, self.action, self.path+ ),+ }+ }+}+fn is_menu_role(role: &str) -> bool {role.trim().to_ascii_lowercase().contains("menu")}@@ -1896,14 +3132,38 @@ fn exact_menu_path_matches(visited: &[Visited<'_>], path: &[String]) -> Vec<usiz/// Every hop re-walks AT-SPI after the preceding menu has materialized; no/// snapshot index is retained across mutations.pub fn invoke_menu_path(pid: u32, path: &[String]) -> Result<()> {+ invoke_menu_path_in(pid, 0, path)+}++/// [`invoke_menu_path`] scoped to the caller's native window: when the same+/// menu path exists under several top-levels of the process (LibreOffice+/// publishes every document window's menubar in one AT-SPI application),+/// the hop under `xid`'s frame wins instead of the path being ambiguous.+pub fn invoke_menu_path_in(pid: u32, xid: u64, path: &[String]) -> Result<()> {bounded(async {let conn = shared_connection().await?;for depth in 0..path.len() {- let visited = collect_visited(conn, pid)+ let collected = collect_visited_bounded_opts(conn, pid, xid, None, None, true).await?.ok_or_else(|| anyhow!("no AT-SPI application for pid {pid}"))?;- let matches = exact_menu_path_matches(&visited, &path[..=depth]);+ let visited = collected.visited;+ let mut matches = exact_menu_path_matches(&visited, &path[..=depth]);+ if matches.len() > 1 {+ if let Some(scope) = collected.scoped_frame {+ let scoped: Vec<usize> = matches+ .iter()+ .copied()+ .filter(|index| visited[*index].frame_ordinal == scope)+ .collect();+ if !scoped.is_empty() {+ matches = scoped;+ }+ }+ }+ if matches.len() > 1 {+ matches = dedupe_menu_matches(&visited, matches);+ }let target_index = match matches.as_slice() {[index] => *index,[] => anyhow::bail!("menu path segment {depth} was not found"),@@ -1942,8 +3202,39 @@ pub fn invoke_menu_path(pid: u32, path: &[String]) -> Result<()> {)}-pub fn perform_action(pid: u32, idx: usize) -> Result<(String, bool)> {- bounded(+/// Several nodes matched one menu path segment inside the scoped frame.+/// LibreOffice publishes its menubar twice per document frame (the VCL+/// menubar and its accessible mirror), so the same item shows up under two+/// parents: prefer the SHOWING instances, then collapse nodes that share one+/// D-Bus object path, and when what is left still carries one role and name+/// take the first (they are the same menu).+fn dedupe_menu_matches(visited: &[Visited<'_>], matches: Vec<usize>) -> Vec<usize> {+ let showing: Vec<usize> = matches+ .iter()+ .copied()+ .filter(|index| visited[*index].showing && !visited[*index].offscreen)+ .collect();+ let mut candidates = if showing.is_empty() { matches } else { showing };+ let mut seen = std::collections::HashSet::new();+ candidates.retain(|index| {+ let node = &visited[*index];+ seen.insert((node.object_ref.bus.clone(), node.object_ref.path.clone()))+ });+ if candidates.len() > 1 {+ let first = &visited[candidates[0]];+ let same = candidates+ .iter()+ .all(|index| visited[*index].role == first.role && visited[*index].name == first.name);+ if same {+ candidates.truncate(1);+ }+ }+ candidates+}++pub fn perform_action(pid: u32, idx: usize) -> Result<(String, bool, bool)> {+ bounded_for(+ INDEX_RESOLVE_BUDGET,async {let conn = shared_connection().await?;let visited = collect_visited(conn, pid)@@ -1982,18 +3273,34 @@ pub fn perform_action(pid: u32, idx: usize) -> Result<(String, bool)> {.await.map_err(|e| anyhow!("Action unavailable: {e}"))?;let action = target.actions.get(chosen).cloned().unwrap_or_default();- let accepted = ap- .do_action(chosen as i32)- .await- .map_err(|e| anyhow!("doAction failed: {e}"))?;- require_action_accepted(accepted)- .with_context(|| format!("element {idx} rejected its native action"))?;+ // `doAction` answers a bool: `false` is the toolkit declining the+ // request (LibreOffice VCL menus answer it for a closed menu), not+ // a delivered click, so it is reported as a suspected no-op.+ // `unacknowledged` is distinct from `rejected`: the D-Bus call+ // never got a reply in time (the toolkit may still be processing+ // it — e.g. a nested dialog main loop), so we know neither+ // acceptance nor effect. This must never read as a plain+ // dispatched/confirmed success; the caller reports it as+ // `unverifiable` with an explicit "no acknowledgement received"+ // note, and — unlike a genuine `suspected_noop` — does not by+ // itself imply the toolkit declined anything.+ let mut unacknowledged = false;+ let rejected = match call(ap.do_action(chosen as i32)).await {+ Some(Ok(accepted)) => !accepted,+ Some(Err(e)) => return Err(anyhow!("doAction failed: {e}")),+ None => {+ dlog!("doAction dispatched but not acknowledged in time");+ unacknowledged = true;+ false+ }+ };+ let suspected_noop = suspected_noop || rejected;// AT-SPI's doAction acknowledgement can precede the renderer's// queued DOM mutation. Give WebKit/Chromium one short event-loop// turn before returning success so a caller's immediate external// state read observes the action it was told was delivered.tokio::time::sleep(Duration::from_millis(50)).await;- Ok((action, suspected_noop))+ Ok((action, suspected_noop, unacknowledged))},|| {Err(anyhow!(@@ -2003,225 +3310,802 @@ pub fn perform_action(pid: u32, idx: usize) -> Result<(String, bool)> {)}-/// A click target resolved from the object address observed in the snapshot.-/// The public integer index remains an address within that snapshot only.-pub struct ObservedClickTarget {- pid: u32,- xid: u64,- index: usize,- bounds_index: usize,- target_position: usize,- frame_ordinal: usize,- visited: Vec<Visited<'static>>,-}--impl ObservedClickTarget {- pub fn verify_live(&self) -> Result<()> {- if !crate::x11::window_belongs_to_pid(self.xid, self.pid) {- anyhow::bail!("stale_element_token: window ownership changed");- }- bounded(- async {- let state = call(self.visited[self.target_position].acc.get_state())- .await- .and_then(|reply| reply.ok())- .ok_or_else(|| {- anyhow!("stale_element_token: observed object no longer responds")- })?;- if state.contains(State::Defunct) || !is_enabled_state(&state) {- anyhow::bail!("stale_element_token: observed object is defunct or disabled");- }- Ok(())- },- || {- Err(anyhow!(- "stale_element_token: observed object liveness check timed out"- ))- },- )+// ── Cached-reference fast paths ──────────────────────────────────────────────+//+// Every per-index entry point above re-walks the application to turn an+// `element_index` back into an accessible. On large trees (LibreOffice, GIMP,+// Nautilus) that walk is the multi-second stall the agent feels on every+// click. A `get_window_state` snapshot already knows each indexed node's D-Bus+// identity ([`AtspiNode::object_ref`]); these entry points act on that identity+// directly and let the caller fall back to the index walk only when the object+// is gone.++/// Budget for a single cached-object action: a handful of round-trips to one+/// live object, never a walk.+const REF_ACTION_BUDGET: Duration = Duration::from_secs(3);++fn raw_ref(object_ref: &ObjectRef) -> RawObjectRef {+ RawObjectRef {+ name: object_ref.bus.clone(),+ path: object_ref.path.clone(),}+}- pub fn screen_bounds(&self) -> Result<(i32, i32, u32, u32)> {- bounded(- async {- element_bounds_for_visited(- &self.visited,- self.pid,- self.xid,- Some(self.frame_ordinal),- )+/// Current action names of an Action proxy, index-aligned with `do_action`.+async fn action_names(ap: &atspi::proxy::action::ActionProxy<'_>) -> Vec<String> {+ let n = call(ap.n_actions()).await.and_then(|r| r.ok()).unwrap_or(0);+ let mut names = Vec::with_capacity(n.max(0) as usize);+ for i in 0..n {+ names.push(+ call(ap.get_name(i)).await- .into_iter()- .find(|(index, _, _, _, _)| *index == self.bounds_index)- .map(|(_, x, y, width, height)| (x, y, width, height))- .ok_or_else(|| anyhow!("element {} has no usable Component bounds", self.index))- },- || {- Err(anyhow!(- "indexed click bounds timed out for pid {}",- self.pid- ))- },- )+ .and_then(|r| r.ok())+ .unwrap_or_default(),+ );}+ names+}- pub fn needs_foreground_pointer(&self) -> bool {- let target = &self.visited[self.target_position];- target.has_editable || target.role == "table cell"+/// The snapshot-cached object no longer answers on the bus (or never did):+/// the observation is stale and must not be retargeted by index.+#[derive(Debug)]+pub struct CachedElementGone(pub String);+impl std::fmt::Display for CachedElementGone {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ f.write_str(&self.0)}+}+impl std::error::Error for CachedElementGone {}- pub fn perform_action(&self, allow_activation: bool) -> Result<(String, bool)> {- let target = &self.visited[self.target_position];- if self.needs_foreground_pointer() {- return Err(super::ElementClickNeedsForeground.into());+/// Open a cached accessible and prove it is still alive (one role read).+async fn live_accessible<'a>(+ conn: &'a AccessibilityConnection,+ object_ref: &ObjectRef,+) -> Result<(AccessibleProxy<'a>, String)> {+ let acc = match call(accessible_for(conn, &raw_ref(object_ref))).await {+ Some(Ok(acc)) => acc,+ Some(Err(error)) => {+ return Err(CachedElementGone(format!("cached element unreachable: {error}")).into())}- if !allow_activation {- return Err(super::ClickActionUnavailable(- "modified click requires pointer delivery".into(),- )- .into());+ None => return Err(anyhow!("cached element did not answer in time")),+ };+ let role = match call(acc.get_role_name()).await {+ Some(Ok(role)) => role,+ Some(Err(error)) => {+ return Err(CachedElementGone(format!("cached element is gone: {error}")).into())}- let chosen = activation_index(&target.role, &target.actions).ok_or_else(|| {- super::ClickActionUnavailable(format!(- "element {} does not advertise a safe activation action",- self.index- ))- })?;- let suspected_noop = target.actions.is_empty() || is_passive_role(&target.role);- bounded(- async {- let action = target.acc.proxies().await?.action().await?;- if !action.do_action(chosen as i32).await? {- anyhow::bail!("element {} rejected the accessibility action", self.index);- }- Ok((- target.actions.get(chosen).cloned().unwrap_or_default(),- suspected_noop,- ))- },- || {- Err(anyhow!(- "indexed click action timed out for pid {}",- self.pid- ))- },- )- }-}--/// Select a retained identity from a live walk. The supplied ordinal is only-/// the position in that live walk, never the snapshot's public index.-fn unique_observed_identity_position<'a>(- nodes: impl Iterator<Item = (usize, Option<&'a AtspiIdentity>, usize, bool)>,- identity: &AtspiIdentity,- expected_frame: usize,-) -> Result<usize> {- let mut matches = nodes.filter(|(_, candidate, _, _)| *candidate == Some(identity));- let Some((position, _, frame_ordinal, indexable)) = matches.next() else {- anyhow::bail!("stale_element_token: observed AT-SPI object is no longer present");+ None => return Err(anyhow!("cached element did not answer in time")),};- if matches.next().is_some() || frame_ordinal != expected_frame || !indexable {- anyhow::bail!(- "stale_element_token: observed object is ambiguous, disabled or outside the target window"- );- }- Ok(position)+ Ok((acc, role))}-/// Match a fresh walk against the object and owning frame observed in a-/// snapshot. A reordered ordinal must refuse rather than retarget the action.-pub fn resolve_observed_click_target(- pid: u32,- index: usize,- xid: u64,- identity: &AtspiIdentity,-) -> Result<ObservedClickTarget> {- bounded(+/// [`perform_action`] on a snapshot-cached element identity. Same contract:+/// `Ok((action_name, suspected_noop))`. Errors when the object no longer+/// exists so the caller can fall back to resolving the index afresh.+pub fn perform_action_ref(object_ref: &ObjectRef) -> Result<(String, bool, bool)> {+ bounded_for(+ REF_ACTION_BUDGET,async {let conn = shared_connection().await?;- let (visited, scoped_frame) = collect_visited_bounded(conn, pid, xid, None, None)- .await?- .ok_or_else(|| anyhow!("no AT-SPI application for pid {pid}"))?;- let frame_ordinal = scoped_frame- .ok_or_else(|| anyhow!("stale_element_token: target window frame is unproven"))?;- let target_position = unique_observed_identity_position(- visited.iter().enumerate().map(|(position, node)| {- (- position,- node.identity.as_ref(),- node.frame_ordinal,- is_indexable(node),- )- }),- identity,- frame_ordinal,- )?;- let bounds_index = visited[..target_position]- .iter()- .filter(|node| is_indexable(node))- .count();- Ok(ObservedClickTarget {- pid,- xid,- index,- bounds_index,- target_position,- frame_ordinal,- visited,- })- },- || {- Err(anyhow!(- "stale_element_token: indexed click resolution timed out"+ let (acc, role) = live_accessible(conn, object_ref).await?;+ let proxies = acc+ .proxies()+ .await+ .map_err(|e| anyhow!("interface proxies unavailable: {e}"))?;+ let ap = proxies+ .action()+ .await+ .map_err(|e| anyhow!("Action unavailable: {e}"))?;+ let actions = action_names(&ap).await;+ let suspected_noop = actions.is_empty() || is_passive_role(&role);+ let chosen = activation_index(&role, &actions)+ .ok_or_else(|| anyhow!("element does not advertise a safe activation action"))?;+ // See `perform_action`'s `unacknowledged`: an unanswered doAction+ // is unknown, not dispatched-implies-success, and must not be+ // folded into `rejected`/`suspected_noop` (which would claim the+ // toolkit declined it) nor silently reported as a plain success.+ let mut unacknowledged = false;+ let rejected = match call(ap.do_action(chosen as i32)).await {+ Some(Ok(accepted)) => !accepted,+ Some(Err(e)) => return Err(anyhow!("doAction failed: {e}")),+ // The request was delivered; a GTK item whose action opens a+ // dialog (nested main loop) only replies once that dialog+ // closes. Treat the timeout as dispatched-but-unconfirmed:+ // falling back to another route would fire the item twice.+ None => {+ dlog!("doAction dispatched but not acknowledged in time");+ unacknowledged = true;+ false+ }+ };+ tokio::time::sleep(Duration::from_millis(50)).await;+ Ok((+ actions.get(chosen).cloned().unwrap_or_default(),+ suspected_noop || rejected,+ unacknowledged,))},+ || Err(anyhow!("perform_action (cached element) timed out")),)}-#[cfg(test)]-mod observed_identity_tests {- use super::*;-- fn identity(path: &str) -> AtspiIdentity {- AtspiIdentity {- bus_name: ":1.1".into(),- path: path.into(),- frame_bus_name: ":1.1".into(),- frame_path: "/frame".into(),- }- }-- #[test]- fn live_reorder_matches_retained_identity_not_the_old_index() {- let observed = identity("/ok");- let replacement = identity("/cancel");- let live = vec![replacement, observed.clone()];- let target = unique_observed_identity_position(- live.iter()- .enumerate()- .map(|(position, item)| (position, Some(item), 0, true)),- &observed,- 0,- )- .unwrap();- assert_eq!(target, 1);- }-- #[test]- fn removed_observed_identity_refuses_before_click() {- let observed = identity("/ok");- let replacement = identity("/cancel");- assert!(unique_observed_identity_position(- std::iter::once((0, Some(&replacement), 0, true)),- &observed,- 0,- )- .is_err());+/// Screen bounds of a snapshot-cached element identity: one `GetExtents`+/// round-trip, no tree re-walk. `Err` when the object is gone or does not+/// answer within [`REF_ACTION_BUDGET`]; the caller decides whether to re-walk.+/// The X11 origin lookup runs on the caller's (blocking) thread before the+/// runtime is entered, so no synchronous X11 I/O happens under `block_on`.+pub fn element_bounds_ref(+ object_ref: &ObjectRef,+ pid: u32,+ xid: u64,+ in_web_content: bool,+) -> Result<(i32, i32, u32, u32)> {+ if in_web_content {+ return Err(anyhow!(+ "web-content bounds need the document origin; re-walk"+ ));}+ let offset = window_to_screen_offset(pid, xid, None);+ let coord = if offset.is_some() {+ CoordType::Window+ } else {+ CoordType::Screen+ };+ let display = (!crate::wayland::is_wayland())+ .then(x11_display_size)+ .flatten();+ bounded_for(+ REF_ACTION_BUDGET,+ async {+ let conn = shared_connection().await?;+ let (acc, _) = live_accessible(conn, object_ref).await?;+ let proxies = match call(acc.proxies()).await {+ Some(Ok(proxies)) => proxies,+ Some(Err(e)) => return Err(anyhow!("interface proxies unavailable: {e}")),+ None => return Err(anyhow!("cached element did not answer in time")),+ };+ let component = match call(proxies.component()).await {+ Some(Ok(component)) => component,+ Some(Err(e)) => return Err(anyhow!("Component unavailable: {e}")),+ None => return Err(anyhow!("cached element did not answer in time")),+ };+ if coord == CoordType::Window {+ if let Some(Ok(raw)) = call(component.get_extents(CoordType::Screen)).await {+ if screen_extents_trusted(raw, display) {+ return project_screen_extents(raw, (0, 0), None)+ .ok_or_else(|| anyhow!("cached element reports no on-screen extents"));+ }+ }+ }+ let extents = match call(component.get_extents(coord)).await {+ Some(Ok(extents)) => extents,+ Some(Err(e)) => return Err(anyhow!("getExtents failed: {e}")),+ None => return Err(anyhow!("getExtents did not answer in time")),+ };+ project_screen_extents(extents, offset.unwrap_or((0, 0)), None)+ .ok_or_else(|| anyhow!("cached element reports no on-screen extents"))+ },+ || Err(anyhow!("element_bounds (cached element) timed out")),+ )}-/// A mutation was attempted; callers must not replay through another route.-#[derive(Debug)]+/// Whether a snapshot-cached element is currently on screen: its own AT-SPI+/// state set carries `Showing` (a GTK menu item whose menu is closed, or a+/// widget on an unmapped notebook page, lacks it even though the object+/// still exists). `Ok(true)` when the toolkit exposes no state set.+pub fn element_showing_ref(object_ref: &ObjectRef) -> Result<bool> {+ bounded_for(+ REF_ACTION_BUDGET,+ async {+ let conn = shared_connection().await?;+ let (acc, _) = live_accessible(conn, object_ref).await?;+ match call(acc.get_state()).await {+ Some(Ok(state)) => Ok(is_showing_state(&state)),+ Some(Err(_)) => Ok(true),+ None => Err(anyhow!("cached element did not answer in time")),+ }+ },+ || Err(anyhow!("element_showing (cached element) timed out")),+ )+}++/// [`focus_element`] on a snapshot-cached element identity.+pub fn focus_element_ref(object_ref: &ObjectRef) -> Result<bool> {+ bounded_for(+ REF_ACTION_BUDGET,+ async {+ let conn = shared_connection().await?;+ let (acc, _) = live_accessible(conn, object_ref).await?;+ let proxies = acc+ .proxies()+ .await+ .map_err(|e| anyhow!("interface proxies unavailable: {e}"))?;+ let component = proxies+ .component()+ .await+ .map_err(|e| anyhow!("Component interface unavailable: {e}"))?;+ let accepted = match call(component.grab_focus()).await {+ Some(Ok(focused)) => focused,+ Some(Err(e)) => return Err(anyhow!("Component.GrabFocus failed: {e}")),+ None => return Err(anyhow!("Component.GrabFocus timed out")),+ };+ if !accepted {+ return Ok(false);+ }+ let settle_deadline = tokio::time::Instant::now() + Duration::from_millis(500);+ while tokio::time::Instant::now() < settle_deadline {+ match tokio::time::timeout(Duration::from_millis(100), acc.get_state()).await {+ Ok(Ok(state)) if state.contains(State::Focused) => return Ok(true),+ _ => tokio::time::sleep(Duration::from_millis(20)).await,+ }+ }+ Ok(false)+ },+ || Err(anyhow!("focus_element (cached element) timed out")),+ )+}++/// [`type_into_editable_at`] on a snapshot-cached element identity.+pub fn type_into_editable_ref(object_ref: &ObjectRef, text: &str) -> Result<()> {+ bounded_for(+ REF_ACTION_BUDGET,+ async {+ let conn = shared_connection().await?;+ let (acc, _) = live_accessible(conn, object_ref).await?;+ let ifaces = match call(acc.get_interfaces()).await {+ Some(Ok(ifaces)) => ifaces,+ _ => return Err(anyhow!("cached element interfaces unavailable")),+ };+ if !ifaces.contains(Interface::EditableText) {+ return Err(anyhow!("cached element is not editable"));+ }+ if write_into_editable_acc(&acc, ifaces.contains(Interface::Component), text).await? {+ Ok(())+ } else {+ Err(anyhow!("EditableText write was rejected"))+ }+ },+ || Err(anyhow!("type_into_editable (cached element) timed out")),+ )+}++/// Write `text` into the focused widget known from the focus-event log, if it+/// is editable. `Some(true)` = written, `Some(false)` = focused but not+/// editable (or write rejected), `None` = no usable focus record.+async fn write_into_focused_editable(+ conn: &AccessibilityConnection,+ pid: u32,+ text: &str,+) -> Option<bool> {+ let (acc, oref) = focused_accessible_via_events(conn, pid).await?;+ let ifaces = call(acc.get_interfaces()).await?.ok()?;+ if !ifaces.contains(Interface::EditableText) {+ dlog!("focused object {} is not editable", oref.path);+ return Some(false);+ }+ match write_into_editable_acc(&acc, ifaces.contains(Interface::Component), text).await {+ Ok(written) => Some(written),+ Err(error) => {+ dlog!("focused editable write failed: {error:#}");+ Some(false)+ }+ }+}++// ── Point hit-testing without a walk ─────────────────────────────────────────++/// One node of a `GetAccessibleAtPoint` descent.+struct HitNode<'a> {+ acc: AccessibleProxy<'a>,+ oref: RawObjectRef,+}++/// Descend from `start` through `Component.GetAccessibleAtPoint(Window)` until+/// the toolkit stops returning a deeper child. O(depth) round-trips instead of+/// a full tree walk — this is how AT-SPI clients are meant to hit-test.+async fn descend_at_point<'a>(+ conn: &'a AccessibilityConnection,+ start: RawObjectRef,+ x: i32,+ y: i32,+) -> Vec<HitNode<'a>> {+ let mut chain: Vec<HitNode<'a>> = Vec::new();+ let mut current = start;+ for _ in 0..64 {+ let Some(Ok(acc)) = call(accessible_for(conn, ¤t)).await else {+ break;+ };+ let next = match call(acc.proxies()).await {+ Some(Ok(proxies)) => match call(proxies.component()).await {+ Some(Ok(component)) => {+ call(component.get_accessible_at_point(x, y, CoordType::Window)).await+ }+ _ => None,+ },+ _ => None,+ };+ chain.push(HitNode {+ acc,+ oref: current.clone(),+ });+ let Some(Ok(next)) = next else { break };+ let Some(next) = RawObjectRef::from_atspi(&next) else {+ break;+ };+ if next.path.ends_with("/null") || next == current {+ break;+ }+ current = next;+ }+ chain+}++impl PartialEq for RawObjectRef {+ fn eq(&self, other: &Self) -> bool {+ self.name == other.name && self.path == other.path+ }+}++/// Hit-test `(win_x, win_y)` (window-local) inside `pid`'s application via+/// the toolkit's own `GetAccessibleAtPoint` and actuate the deepest node on+/// the chain that advertises a safe activation, preferring a real actuator+/// over a passive label. `xid` (when non-zero and resolvable) restricts the+/// descent to that window's frame. `Ok(None)` = nothing actionable there.+pub fn perform_action_at_point_in(+ pid: u32,+ xid: u64,+ win_x: i32,+ win_y: i32,+ skip_focus_roles: bool,+) -> Result<Option<AtPointHit>> {+ bounded_for(+ INPUT_QUERY_BUDGET,+ async {+ let conn = shared_connection().await?;+ let Some(app) = app_for_pid(conn, pid).await? else {+ return Ok(None);+ };+ let seeds: Vec<RawObjectRef> = match call(app.get_children()).await {+ Some(Ok(children)) => children+ .into_iter()+ .filter_map(|child| RawObjectRef::from_atspi(&child))+ .collect(),+ _ => Vec::new(),+ };+ let scoped = if xid == 0 {+ None+ } else {+ resolve_window_frame(conn, pid, xid, &seeds).await+ };+ // The toolkit's `Window` coordinate frame is not always the X11+ // client window the caller's pixels are relative to: VCL+ // (LibreOffice) measures from the WM frame, title bar included,+ // so a client-local point hit-tests one or two rows too high.+ // Re-base the point on the frame's own screen origin.+ let (hit_x, hit_y) = match scoped {+ Some(ordinal) => {+ let frame_origin = decoration_frame_origin(conn, &app, &seeds[ordinal]).await;+ let client_origin = x11_window_origin(xid);+ at_point_toolkit_coords((win_x, win_y), client_origin, frame_origin)+ }+ None => (win_x, win_y),+ };+ if (hit_x, hit_y) != (win_x, win_y) {+ dlog!(+ "at-point ({win_x},{win_y}) re-based to toolkit window coords ({hit_x},{hit_y})"+ );+ }+ let ordered: Vec<RawObjectRef> = match scoped {+ Some(ordinal) => seeds.get(ordinal).cloned().into_iter().collect(),+ None => seeds,+ };+ for seed in ordered {+ let chain = descend_at_point(conn, seed, hit_x, hit_y).await;+ if chain.len() <= 1 {+ // The frame itself, or nothing: this top-level does not+ // cover the point (or cannot hit-test). Try the next one.+ continue;+ }+ return actuate_chain(&chain, hit_x, hit_y, skip_focus_roles).await;+ }+ Ok(None)+ },+ || Ok(None),+ )+}++/// The frame origin to re-base an at-point query on: the top-level's screen+/// origin for VCL (LibreOffice), whose `Window` coordinates start at the WM+/// frame, and `None` for every other toolkit. GTK and Chromium measure+/// `Window` coordinates from their client window but report the top-level's+/// screen extents including server-side decorations (Openbox, XFCE), so+/// adding that inset would hit-test a title bar too low; under client-side+/// decorations (GNOME) the inset is zero anyway.+async fn decoration_frame_origin(+ conn: &AccessibilityConnection,+ app: &AccessibleProxy<'_>,+ frame: &RawObjectRef,+) -> Option<(i32, i32)> {+ if !toolkit_measures_from_frame(app).await {+ return None;+ }+ frame_screen_origin(conn, frame).await+}++async fn toolkit_measures_from_frame(app: &AccessibleProxy<'_>) -> bool {+ let Some(Ok(proxies)) = call(app.proxies()).await else {+ return false;+ };+ let Some(Ok(application)) = call(proxies.application()).await else {+ return false;+ };+ let Some(Ok(name)) = call(application.toolkit_name()).await else {+ return false;+ };+ is_frame_coordinate_toolkit(&name)+}++/// VCL (LibreOffice) measures `Window` coordinates from the WM frame, title+/// bar included; the other toolkits measure from the client window.+fn is_frame_coordinate_toolkit(toolkit_name: &str) -> bool {+ toolkit_name.trim().eq_ignore_ascii_case("vcl")+}++/// Screen origin of an application top-level as the toolkit reports it+/// (`Component.GetExtents(Screen)`), when that answer is trustworthy.+async fn frame_screen_origin(+ conn: &AccessibilityConnection,+ seed: &RawObjectRef,+) -> Option<(i32, i32)> {+ let acc = call(accessible_for(conn, seed)).await?.ok()?;+ let proxies = call(acc.proxies()).await?.ok()?;+ let component = call(proxies.component()).await?.ok()?;+ let raw = call(component.get_extents(CoordType::Screen)).await?.ok()?;+ let display = (!crate::wayland::is_wayland())+ .then(x11_display_size)+ .flatten();+ screen_extents_trusted(raw, display).then_some((raw.0, raw.1))+}++/// Largest offset between an X11 client origin and the toolkit's own frame+/// origin that still reads as decorations (a title bar, a border). Anything+/// larger is a mis-correlated frame, and the point is left alone.+const FRAME_DELTA_MAX_PX: i32 = 128;++/// Translate a client-window-local point into the toolkit's `Window`+/// coordinate frame for `GetAccessibleAtPoint`. The toolkit frame origin is+/// where the top-level accessible says it is on screen; the client origin is+/// the X11 window the caller's pixels are relative to. Their difference is+/// the decoration inset the toolkit folds into its frame (0 for GTK, the+/// title bar height for VCL). Unknown origins or an implausible delta leave+/// the point untouched.+pub(crate) fn at_point_toolkit_coords(+ (x, y): (i32, i32),+ client_origin: Option<(i32, i32)>,+ frame_origin: Option<(i32, i32)>,+) -> (i32, i32) {+ let (Some((cx, cy)), Some((fx, fy))) = (client_origin, frame_origin) else {+ return (x, y);+ };+ let (dx, dy) = (cx - fx, cy - fy);+ if dx.abs() > FRAME_DELTA_MAX_PX || dy.abs() > FRAME_DELTA_MAX_PX {+ return (x, y);+ }+ (x + dx, y + dy)+}++#[cfg(test)]+mod at_point_coords_tests {+ use super::at_point_toolkit_coords;++ #[test]+ fn only_vcl_measures_from_the_wm_frame() {+ assert!(super::is_frame_coordinate_toolkit("VCL"));+ assert!(!super::is_frame_coordinate_toolkit("GTK"));+ assert!(!super::is_frame_coordinate_toolkit("GAIL"));+ assert!(!super::is_frame_coordinate_toolkit("Chromium"));+ }++ #[test]+ fn vcl_title_bar_inset_is_added() {+ // Calc: X11 client at (70,64), VCL frame origin (70,27): +37 px.+ assert_eq!(+ at_point_toolkit_coords((315, 218), Some((70, 64)), Some((70, 27))),+ (315, 255)+ );+ }++ #[test]+ fn gtk_frame_equals_client_so_nothing_changes() {+ assert_eq!(+ at_point_toolkit_coords((315, 218), Some((70, 64)), Some((70, 64))),+ (315, 218)+ );+ }++ #[test]+ fn unknown_or_implausible_origins_leave_the_point() {+ assert_eq!(+ at_point_toolkit_coords((10, 20), None, Some((1, 2))),+ (10, 20)+ );+ assert_eq!(+ at_point_toolkit_coords((10, 20), Some((1, 2)), None),+ (10, 20)+ );+ assert_eq!(+ at_point_toolkit_coords((10, 20), Some((900, 64)), Some((70, 64))),+ (10, 20)+ );+ }+}++/// Fire the right thing on a `GetAccessibleAtPoint` chain (frame first,+/// deepest hit last). The deepest node with a safe activation wins; a+/// container ancestor is never fired for a click on one of its items (see+/// [`at_point_activation_index`]). An item that advertises no Action is+/// selected through its container's `Selection` interface, the way a plain+/// click selects a file in a file manager; when that is not possible either,+/// `Ok(None)` leaves the point to the caller's real pointer press.+async fn actuate_chain(+ chain: &[HitNode<'_>],+ win_x: i32,+ win_y: i32,+ skip_focus_roles: bool,+) -> Result<Option<AtPointHit>> {+ let deepest = chain.len() - 1;+ let mut deepest_role = String::new();+ let mut deepest_name = String::new();+ let mut deepest_has_action = false;+ let mut passive_fallback: Option<(usize, String, String, Vec<String>)> = None;+ for (depth, node) in chain.iter().enumerate().rev() {+ let Some(Ok(role)) = call(node.acc.get_role_name()).await else {+ continue;+ };+ let Some(Ok(ifaces)) = call(node.acc.get_interfaces()).await else {+ continue;+ };+ let has_action = ifaces.contains(Interface::Action);+ if depth == deepest {+ deepest_role = role.clone();+ deepest_name = call(node.acc.name())+ .await+ .and_then(|r| r.ok())+ .unwrap_or_default();+ deepest_has_action = has_action;+ }+ if depth != deepest && !deepest_has_action && is_selectable_item_role(&role) {+ // The point is on a label / icon inside a file or list item: a+ // plain click selects that item.+ if let Some(hit) =+ select_item_in_chain(chain, depth, &role, &deepest_name, skip_focus_roles).await+ {+ return Ok(Some(hit));+ }+ dlog!(+ "hit-test ({win_x},{win_y}) -> depth {depth} {role:?} is a selectable item that cannot be selected through Selection; leaving it to a real click"+ );+ return Ok(None);+ }+ if !has_action {+ continue;+ }+ let Some(Ok(proxies)) = call(node.acc.proxies()).await else {+ continue;+ };+ let Some(Ok(ap)) = call(proxies.action()).await else {+ continue;+ };+ let actions = action_names(&ap).await;+ if depth == deepest && is_selectable_item_role(&role) {+ // A file / list item under a plain click is SELECTED; its own+ // `open` / `activate` is the double-click (Nautilus canvas items+ // advertise `open`, which launched the file). Select it through+ // the container; only fall back to the item's action when the+ // container cannot select.+ if let Some(hit) =+ select_item_in_chain(chain, deepest, &role, &deepest_name, skip_focus_roles).await+ {+ return Ok(Some(hit));+ }+ if actions.iter().any(|a| normalized_action_verb(a) == "open") {+ dlog!(+ "hit-test ({win_x},{win_y}) -> {role:?} {deepest_name:?} only offers `open`; leaving the selection to a real click"+ );+ return Ok(None);+ }+ }+ let Some(chosen) = at_point_activation_index(&role, &actions, depth == deepest) else {+ if depth != deepest+ && !deepest_has_action+ && activation_index(&role, &actions).is_some()+ {+ // The item under the point has no Action and this ancestor's+ // activation is a container's "open the selection": select+ // the item instead, else leave the point to a real pointer+ // press, which is what selects it.+ dlog!(+ "hit-test ({win_x},{win_y}) -> depth {depth} role={role:?} is a container over {deepest_role:?}: not firing {:?}",+ activation_index(&role, &actions).and_then(|i| actions.get(i))+ );+ return Ok(select_item_in_chain(+ chain,+ deepest,+ &deepest_role,+ &deepest_name,+ skip_focus_roles,+ )+ .await);+ }+ continue;+ };+ let name = call(node.acc.name())+ .await+ .and_then(|r| r.ok())+ .unwrap_or_default();+ if is_passive_role(&role) {+ if passive_fallback.is_none() {+ passive_fallback = Some((depth, role.clone(), name, actions.clone()));+ }+ continue;+ }+ if skip_focus_roles && is_focus_taking_role(&role) {+ // The caller has a real pointer: an entry / spin button /+ // cell gets the focus from a real press, not from+ // `doAction`. Fall through to that route.+ dlog!(+ "hit-test ({win_x},{win_y}) -> depth {depth} role={role:?}: focus-taking, leaving it to a real click"+ );+ return Ok(None);+ }+ dlog!(+ "hit-test ({win_x},{win_y}) -> depth {depth} role={role:?} name={name:?} path={} action={:?}",+ node.oref.path,+ actions.get(chosen)+ );+ let hit = AtPointHit::fired(+ actions.get(chosen).cloned().unwrap_or_default(),+ role.clone(),+ name,+ node.oref.path.clone(),+ );+ match call(ap.do_action(chosen as i32)).await {+ Some(Ok(true)) => {+ tokio::time::sleep(Duration::from_millis(50)).await;+ return Ok(Some(hit));+ }+ // Declined (a closed VCL menu answers false): nothing fired, so+ // the caller falls through to the real pointer press.+ Some(Ok(false)) => {+ dlog!("hit-test doAction declined; falling through to the pointer route");+ return Ok(None);+ }+ Some(Err(e)) => return Err(anyhow!("doAction failed: {e}")),+ // Dispatched; the reply waits on a nested main loop.+ None => return Ok(Some(hit)),+ }+ }+ if !deepest_has_action && is_selectable_item_role(&deepest_role) {+ if let Some(hit) = select_item_in_chain(+ chain,+ deepest,+ &deepest_role,+ &deepest_name,+ skip_focus_roles,+ )+ .await+ {+ return Ok(Some(hit));+ }+ }+ if let Some((depth, role, name, actions)) = passive_fallback {+ let node = &chain[depth];+ let Some(chosen) = activation_index(&role, &actions) else {+ return Ok(None);+ };+ let Some(Ok(proxies)) = call(node.acc.proxies()).await else {+ return Ok(None);+ };+ let Some(Ok(ap)) = call(proxies.action()).await else {+ return Ok(None);+ };+ if let Some(Ok(true)) = call(ap.do_action(chosen as i32)).await {+ return Ok(Some(AtPointHit::fired(+ actions.get(chosen).cloned().unwrap_or_default(),+ role,+ name,+ node.oref.path.clone(),+ )));+ }+ }+ Ok(None)+}++/// Select the item under the point through its parent's `Selection`+/// interface: scanning the hit chain upward from the deepest node, the first+/// node with a selectable item role whose parent selects (Nautilus: `icon`+/// label -> `canvas` item -> `layered pane` with Selection) is cleared-and-+/// selected (`ClearSelection` + `SelectChild(index_in_parent)`) and read back+/// with `IsChildSelected`. `None` when nothing on the chain selects, the+/// toolkit refused, or the item is a grid cell and a real pointer is+/// available (`real_pointer_available`) — the caller then falls back to a+/// real pointer press.+async fn select_item_in_chain(+ chain: &[HitNode<'_>],+ from_depth: usize,+ _role_hint: &str,+ _name_hint: &str,+ real_pointer_available: bool,+) -> Option<AtPointHit> {+ for item_depth in (1..=from_depth.min(chain.len() - 1)).rev() {+ let item = &chain[item_depth];+ let parent = &chain[item_depth - 1];+ let Some(Ok(item_role)) = call(item.acc.get_role_name()).await else {+ continue;+ };+ if !is_selectable_item_role(&item_role) {+ continue;+ }+ if real_pointer_available && is_cell_role(&item_role) {+ // `Selection.SelectChild` marks a Calc cell without moving the+ // cell cursor: a following type_text would land in the old+ // active cell. The caller's real press both selects and+ // focuses it.+ dlog!(+ "{item_role:?} under the point is a grid cell; leaving it to the real pointer press instead of Selection.SelectChild"+ );+ return None;+ }+ let Some(Ok(ifaces)) = call(parent.acc.get_interfaces()).await else {+ continue;+ };+ if !ifaces.contains(Interface::Selection) {+ continue;+ }+ let item_name = call(item.acc.name())+ .await+ .and_then(|r| r.ok())+ .unwrap_or_default();+ let index = call(item.acc.get_index_in_parent()).await?.ok()?;+ if index < 0 {+ return None;+ }+ let proxies = call(parent.acc.proxies()).await?.ok()?;+ let selection = call(proxies.selection()).await?.ok()?;+ // Best effort: some containers refuse ClearSelection while still+ // honouring SelectChild (which then replaces a single selection).+ let _ = call(selection.clear_selection()).await;+ let accepted = call(selection.select_child(index)).await?.ok()?;+ if !accepted {+ dlog!(+ "Selection.SelectChild({index}) refused by {}",+ parent.oref.path+ );+ return None;+ }+ tokio::time::sleep(Duration::from_millis(50)).await;+ let verified = matches!(+ call(selection.is_child_selected(index)).await,+ Some(Ok(true))+ );+ dlog!(+ "selected child {index} ({item_role:?} {item_name:?}) of {} verified={verified}",+ parent.oref.path+ );+ return Some(AtPointHit {+ action: "select_child".to_owned(),+ role: item_role,+ name: item_name.clone(),+ path: item.oref.path.clone(),+ selected: Some(item_name),+ selection_verified: verified,+ });+ }+ None+}++/// A mutation was attempted; callers must not replay through another route.+#[derive(Debug)]pub struct ScrollProgress {pub acknowledged: u32,pub complete: bool,@@ -2742,7 +4626,8 @@ pub fn scroll_element(/// and new controls. Wait for the target's Focused state to become observable;/// an acknowledgement without read-back is not sufficient for global input.pub fn focus_element(pid: u32, idx: usize) -> Result<bool> {- bounded(+ bounded_for(+ INDEX_RESOLVE_BUDGET,async {let conn = shared_connection().await?;let visited = collect_visited(conn, pid)@@ -2811,8 +4696,14 @@ pub fn focus_element(pid: u32, idx: usize) -> Result<bool> {/// a click lands on the button, not its enclosing panel. Returns `Ok(Some(action))`/// when an element was actuated, `Ok(None)` when no actionable element covers the/// point (the caller then falls back to the synthetic X11 path).-pub fn perform_action_at_point(pid: u32, win_x: i32, win_y: i32) -> Result<Option<String>> {- bounded(+pub fn perform_action_at_point(+ pid: u32,+ win_x: i32,+ win_y: i32,+ skip_focus_roles: bool,+) -> Result<Option<AtPointHit>> {+ bounded_for(+ INPUT_QUERY_BUDGET,async {let conn = shared_connection().await?;let visited = match collect_visited(conn, pid).await? {@@ -2831,7 +4722,7 @@ pub fn perform_action_at_point(pid: u32, win_x: i32, win_y: i32) -> Result<Optio// children, but the area/role split is what actually disambiguates.let mut frames: Vec<(usize, i32, i32, u32, u32, bool)> = Vec::new();for (i, v) in visited.iter().enumerate() {- if v.actions.is_empty() || !v.has_component {+ if v.actions.is_empty() || !v.has_component || is_container_role(&v.role) {continue;}let Some(Ok(proxies)) = call(v.acc.proxies()).await else {@@ -2865,6 +4756,20 @@ pub fn perform_action_at_point(pid: u32, win_x: i32, win_y: i32) -> Result<Optioreturn Ok(None);};let target = &visited[idx];+ if skip_focus_roles && is_focus_taking_role(&target.role) {+ return Ok(None);+ }+ if is_selectable_item_role(&target.role) {+ // A plain click on a file / list item selects it; its own+ // `open` / `activate` is the double-click. Leave it to the+ // caller's real pointer press.+ dlog!(+ "full-walk hit {:?} {:?} is a selectable item; not firing",+ target.role,+ target.name+ );+ return Ok(None);+ }let Some(chosen) = activation_index(&target.role, &target.actions) else {return Ok(None);};@@ -2876,14 +4781,21 @@ pub fn perform_action_at_point(pid: u32, win_x: i32, win_y: i32) -> Result<Optio.action().await.map_err(|e| anyhow!("Action unavailable: {e}"))?;- let accepted = ap- .do_action(chosen as i32)- .await- .map_err(|e| anyhow!("doAction failed: {e}"))?;- Ok(accepted_action(- accepted,- target.actions.get(chosen).cloned(),- ))+ match call(ap.do_action(chosen as i32)).await {+ Some(Ok(true)) => {}+ Some(Ok(false)) => {+ dlog!("doAction declined at point; falling through to the pointer route");+ return Ok(None);+ }+ Some(Err(e)) => return Err(anyhow!("doAction failed: {e}")),+ None => dlog!("doAction dispatched but not acknowledged in time"),+ }+ Ok(Some(AtPointHit::fired(+ target.actions.get(chosen).cloned().unwrap_or_default(),+ target.role.clone(),+ target.name.clone(),+ target.object_ref.path.clone(),+ )))},|| Ok(None),)@@ -2917,14 +4829,18 @@ pub fn perform_action_at_screen_point(screen_x: i32,screen_y: i32,) -> Result<Option<String>> {- bounded(+ bounded_for(+ INPUT_QUERY_BUDGET,async {let conn = shared_connection().await?;- let (visited, scoped_frame) =- match collect_visited_bounded(conn, pid, xid, None, None).await? {- Some(walked) => walked,- None => return Ok(None),- };+ let Collected {+ visited,+ scoped_frame,+ ..+ } = match collect_visited_bounded(conn, pid, xid, None, None).await? {+ Some(walked) => walked,+ None => return Ok(None),+ };// Share snapshot correlation and geometry, including renderer-frame// rebasing and embedded WebProcess insets. Bounds retain application-@@ -2932,6 +4848,7 @@ pub fn perform_action_at_screen_point(let action_nodes: Vec<&Visited> = visited.iter().filter(|v| is_indexable(v)).collect();let frames: Vec<_> = element_bounds_for_visited(&visited, pid, xid, scoped_frame).await+ .0.into_iter().map(|(idx, x, y, w, h)| {(idx, x, y, w, h, is_passive_role(&action_nodes[idx].role))@@ -2953,41 +4870,58 @@ pub fn perform_action_at_screen_point(.action().await.map_err(|e| anyhow!("Action unavailable: {e}"))?;- let accepted = ap- .do_action(chosen as i32)- .await- .map_err(|e| anyhow!("doAction failed: {e}"))?;- Ok(accepted_action(- accepted,- target.actions.get(chosen).cloned(),- ))+ match call(ap.do_action(chosen as i32)).await {+ Some(Ok(true)) => {}+ // Declined by the toolkit (a closed VCL menu): no accessible+ // hit, so the caller falls through to the real pointer press.+ Some(Ok(false)) => {+ dlog!("doAction declined at point; falling through to the pointer route");+ return Ok(None);+ }+ Some(Err(e)) => return Err(anyhow!("doAction failed: {e}")),+ None => dlog!("doAction dispatched but not acknowledged in time"),+ }+ Ok(target.actions.get(chosen).cloned())},|| Ok(None),)}-fn accepted_action(accepted: bool, action: Option<String>) -> Option<String> {- if accepted {- action- } else {- None- }-}--fn require_action_accepted(accepted: bool) -> Result<()> {- if accepted {- Ok(())- } else {- Err(anyhow!("AT-SPI action was rejected"))- }-}-/// Roles that draw text/graphics but don't *do* anything when actuated. GTK4/// nests a `label` inside every `button` with a near-identical (slightly/// smaller) frame, so an area-only hit-test lands on the inert label —/// `do_action` is a silent no-op (the "false success"). Treat these as/// last-resort click targets.-fn is_passive_role(role: &str) -> bool {+/// AT-SPI roles whose `activate`/`press` action does not move the widget+/// focus into them (Chromium and GTK entries, spin buttons, sliders, table+/// cells), so a following keystroke would land elsewhere; a real pointer+/// click at the point is what a user does.+pub fn is_focus_taking_role(role: &str) -> bool {+ matches!(+ role.trim().to_ascii_lowercase().as_str(),+ "table cell"+ | "spin button"+ | "text"+ | "entry"+ | "password text"+ | "slider"+ | "combo box"+ | "editbar"+ | "search box"+ | "textbox"+ )+}++/// AT-SPI role names a toolkit gives a tooltip popup (`tool tip` is the+/// canonical spelling; `tooltip` appears in some bridges).+pub(crate) fn is_tooltip_role(role: &str) -> bool {+ matches!(+ role.trim().to_ascii_lowercase().as_str(),+ "tool tip" | "tooltip"+ )+}++pub(crate) fn is_passive_role(role: &str) -> bool {matches!(role,"label" | "static" | "static text" | "separator" | "filler" | "image" | "icon"@@ -3000,7 +4934,7 @@ fn is_passive_role(role: &str) -> bool {/// to a passive label if nothing else covers the point. Smallest-area within a/// class wins so the click lands on the button, not its enclosing panel./// Right/bottom edges are exclusive (`px < x + w`). `None` if nothing covers it.-fn select_click_target(+pub(crate) fn select_click_target(frames: &[(usize, i32, i32, u32, u32, bool)],px: i32,py: i32,@@ -3024,8 +4958,84 @@ fn select_click_target(best_active.or(best_passive).map(|(_, idx)| idx)}+/// Error-message prefix when an element offers neither `EditableText` nor+/// `Value`: the tool layer branches to the keyboard fallback on it.+pub const NO_VALUE_ROUTE: &str = "no_value_route";++pub fn is_no_value_route(error: &anyhow::Error) -> bool {+ error.to_string().starts_with(NO_VALUE_ROUTE)+}++/// A spin button / spin scale only parses its entry text on activate (Return+/// or focus-out); after an EditableText write, fire the element's `activate`+/// action when it exposes one so the numeric value commits. Best effort.+async fn commit_editable_write(proxies: &atspi::proxy::proxy_ext::Proxies<'_>) {+ let Some(Ok(ap)) = call(proxies.action()).await else {+ return;+ };+ let names = action_names(&ap).await;+ if let Some(index) = names+ .iter()+ .position(|name| normalized_action_verb(name) == "activate")+ {+ let _ = call(ap.do_action(index as i32)).await;+ }+}++/// The focus-free write shared by the index and cached-ref entry points.+/// SetValue never calls Component.GrabFocus: GTK may activate and raise the+/// toplevel in response, violating the background contract. Toolkits that+/// expose EditableText only while focused get an honest `NO_VALUE_ROUTE`.+async fn set_value_on(+ acc: &AccessibleProxy<'_>,+ has_value: bool,+ value: &str,+ label: &str,+) -> Result<()> {+ let proxies = acc+ .proxies()+ .await+ .map_err(|e| anyhow!("interface proxies unavailable: {e}"))?;+ if let Ok(et) = proxies.editable_text().await {+ // Replace whole contents (parity with the Windows/macOS set_value,+ // which overwrite rather than insert at the caret).+ if et.set_text_contents(value).await.unwrap_or(false) {+ commit_editable_write(&proxies).await;+ return Ok(());+ }+ // Some toolkits reject SetTextContents but accept an insert at the+ // caret offset; clear-then-insert as a fallback.+ let off = match proxies.text().await {+ Ok(tp) => tp.caret_offset().await.unwrap_or(0),+ Err(_) => 0,+ };+ let len = value.chars().count() as i32;+ if et.insert_text(off, value, len).await.unwrap_or(false) {+ commit_editable_write(&proxies).await;+ return Ok(());+ }+ }+ if has_value {+ let v: f64 = value+ .parse()+ .map_err(|_| anyhow!("value '{value}' is not numeric for a Value element"))?;+ proxies+ .value()+ .await+ .map_err(|e| anyhow!("Value unavailable: {e}"))?+ .set_current_value(v)+ .await+ .map_err(|e| anyhow!("setCurrentValue failed: {e}"))?;+ return Ok(());+ }+ Err(anyhow!(+ "{NO_VALUE_ROUTE}: {label} exposes neither EditableText nor Value"+ ))+}+pub fn set_value(pid: u32, idx: usize, value: &str) -> Result<()> {- bounded(+ bounded_for(+ INDEX_RESOLVE_BUDGET,async {let conn = shared_connection().await?;let visited = collect_visited(conn, pid)@@ -3035,57 +5045,84 @@ pub fn set_value(pid: u32, idx: usize, value: &str) -> Result<()> {let target = action_nodes.get(idx).ok_or_else(|| {anyhow!("element {idx} not found (total: {})", action_nodes.len())})?;+ set_value_on(+ &target.acc,+ target.has_value,+ value,+ &format!("element {idx}"),+ )+ .await+ },+ || {+ Err(anyhow!(+ "set_value timed out for pid {pid} (app unresponsive to AT-SPI)"+ ))+ },+ )+}- let proxies = target- .acc- .proxies()- .await- .map_err(|e| anyhow!("interface proxies unavailable: {e}"))?;+/// [`set_value`] on a snapshot-cached element identity (the exact object the+/// caller's `element_index` named, not the same ordinal in a fresh walk).+pub fn set_value_ref(object_ref: &ObjectRef, value: &str) -> Result<()> {+ bounded_for(+ REF_ACTION_BUDGET,+ async {+ let conn = shared_connection().await?;+ let (acc, _) = live_accessible(conn, object_ref).await?;+ let ifaces = match call(acc.get_interfaces()).await {+ Some(Ok(ifaces)) => ifaces,+ _ => return Err(anyhow!("cached element interfaces unavailable")),+ };+ set_value_on(+ &acc,+ ifaces.contains(Interface::Value),+ value,+ "cached element",+ )+ .await+ },+ || Err(anyhow!("set_value (cached element) timed out")),+ )+}- // SetValue is a focus-free accessibility operation. Do not call- // Component.GrabFocus here: GTK may activate and raise the entire- // toplevel in response, violating the background contract. Toolkits- // that expose EditableText only while focused must return an honest- // unsupported error rather than changing desktop focus implicitly.- if let Ok(et) = proxies.editable_text().await {- // Replace whole contents (parity with the Windows/macOS set_value,- // which overwrite rather than insert at the caret).- if et.set_text_contents(value).await.unwrap_or(false) {- return Ok(());- }- // Some toolkits reject SetTextContents but accept an insert at the- // caret offset; clear-then-insert as a fallback.- let off = match proxies.text().await {- Ok(tp) => tp.caret_offset().await.unwrap_or(0),- Err(_) => 0,- };- let len = value.chars().count() as i32;- if et.insert_text(off, value, len).await.unwrap_or(false) {- return Ok(());+/// Current value of a cached element for read-back after a write: the+/// `Value` interface's number, else the `Text` content. `None` when the+/// element exposes neither (or does not answer in time).+pub fn read_value_ref(object_ref: &ObjectRef) -> Result<Option<String>> {+ bounded_for(+ REF_ACTION_BUDGET,+ async {+ let conn = shared_connection().await?;+ let (acc, _) = live_accessible(conn, object_ref).await?;+ let ifaces = match call(acc.get_interfaces()).await {+ Some(Ok(ifaces)) => ifaces,+ _ => return Ok(None),+ };+ let proxies = match call(acc.proxies()).await {+ Some(Ok(proxies)) => proxies,+ _ => return Ok(None),+ };+ if ifaces.contains(Interface::Value) {+ if let Some(Ok(vp)) = call(proxies.value()).await {+ if let Some(Ok(v)) = call(vp.current_value()).await {+ return Ok(Some(format_value(v)));+ }}}- if target.has_value {- let v: f64 = value- .parse()- .map_err(|_| anyhow!("value '{value}' is not numeric for a Value element"))?;- proxies- .value()- .await- .map_err(|e| anyhow!("Value unavailable: {e}"))?- .set_current_value(v)- .await- .map_err(|e| anyhow!("setCurrentValue failed: {e}"))?;- return Ok(());+ if ifaces.contains(Interface::Text) {+ if let Some(Ok(tp)) = call(proxies.text()).await {+ let count = call(tp.character_count())+ .await+ .and_then(|r| r.ok())+ .unwrap_or(0);+ if let Some(Ok(t)) = call(tp.get_text(0, count.min(4096))).await {+ return Ok(Some(t));+ }+ }}- Err(anyhow!(- "element {idx} exposes neither EditableText nor Value"- ))- },- || {- Err(anyhow!(- "set_value timed out for pid {pid} (app unresponsive to AT-SPI)"- ))+ Ok(None)},+ || Err(anyhow!("read_value (cached element) timed out")),)}@@ -3095,13 +5132,51 @@ pub fn get_element_bounds(pid: u32, idx: usize) -> Result<(i32, i32, u32, u32)>.context("Hyprland bounds require an exact window for multi-window apps")?;return get_element_bounds_for_window(pid, window.address, idx);}- bounded(+ element_bounds_by_walk(pid, 0, idx)+}++/// Resolve element `idx` against a fresh walk and read its screen bounds.+/// `xid != 0` walks the caller's window FIRST — the same visiting order (and+/// therefore the same index space) `get_window_state` used for the snapshot+/// the index came from; an application-wide pre-order would number a+/// multi-window app (GIMP's docks, LibreOffice dialogs) differently.+/// Screen bounds a walk resolved are usable for a pointer action only when+/// they have a real size and intersect the display. `display` is `None` when+/// the display size is unknown (Wayland); the size check still applies.+pub(crate) fn walk_bounds_usable(+ bounds: Option<(i32, i32, u32, u32)>,+ display: Option<(u32, u32)>,+) -> Option<(i32, i32, u32, u32)> {+ let (x, y, w, h) = bounds?;+ if w == 0 || h == 0 {+ return None;+ }+ if let Some((dw, dh)) = display {+ let right = i64::from(x) + i64::from(w);+ let bottom = i64::from(y) + i64::from(h);+ if x >= dw as i32 || y >= dh as i32 || right <= 0 || bottom <= 0 {+ return None;+ }+ }+ Some((x, y, w, h))+}++fn element_bounds_by_walk(pid: u32, xid: u64, idx: usize) -> Result<(i32, i32, u32, u32)> {+ // Synchronous X11 lookups stay on the caller's blocking thread, outside+ // the AT-SPI runtime.+ let offset = window_to_screen_offset(pid, xid, None);+ let display = (!crate::wayland::is_wayland())+ .then(x11_display_size)+ .flatten();+ bounded_for(+ INDEX_RESOLVE_BUDGET,async {let conn = shared_connection().await?;- let visited = collect_visited(conn, pid)+ let visited = collect_visited_bounded(conn, pid, xid, None, None).await?- .ok_or_else(|| anyhow!("no AT-SPI application for pid {pid}"))?;- let web_document_origin = web_document_origin_for_visited(&visited, pid, 0, None)+ .ok_or_else(|| anyhow!("no AT-SPI application for pid {pid}"))?+ .visited;+ let web_document_origin = web_document_origin_for_visited(&visited, pid, xid, None).await.unwrap_or((0, 0));let action_nodes: Vec<&Visited> = visited.iter().filter(|v| is_indexable(v)).collect();@@ -3122,30 +5197,41 @@ pub fn get_element_bounds(pid: u32, idx: usize) -> Result<(i32, i32, u32, u32)>// Prefer WINDOW coords + a deterministic screen offset — fixes GTK4,// whose CoordType::Screen collapses every element to (0,0). Fall back to// Screen on Wayland / when no X11 window resolves (offset is None).- match window_to_screen_offset(pid, 0, None) {+ match offset {Some((ox, oy)) => {- let (x, y, w, h) = comp+ let raw = comp.get_extents(CoordType::Window).await.map_err(|e| anyhow!("getExtents failed: {e}"))?;- let (document_x, document_y) = if target.in_web_doc {- web_document_origin- } else {- (0, 0)- };- Ok((- x + ox + document_x,- y + oy + document_y,- w.max(0) as u32,- h.max(0) as u32,- ))+ let document_origin = target.in_web_doc.then_some(web_document_origin);+ // An unrealized widget (a menu item whose menu is closed)+ // answers (0,0,0,0) in Window coordinates; projected onto+ // the window origin that would be a "valid" centre at the+ // window's top-left corner. Refuse it like the cached+ // path does.+ walk_bounds_usable(+ project_screen_extents(raw, (ox, oy), document_origin),+ display,+ )+ .ok_or_else(|| {+ anyhow!(+ "element {idx} ({}) reports no usable on-screen extents (raw window extents {raw:?}); it may be inside a closed menu or an unmapped page",+ target.role+ )+ })}None => {- let (x, y, w, h) = comp+ let raw = comp.get_extents(CoordType::Screen).await.map_err(|e| anyhow!("getExtents failed: {e}"))?;- Ok((x, y, w.max(0) as u32, h.max(0) as u32))+ walk_bounds_usable(project_screen_extents(raw, (0, 0), None), display)+ .ok_or_else(|| {+ anyhow!(+ "element {idx} ({}) reports no usable on-screen extents (raw screen extents {raw:?})",+ target.role+ )+ })}}},@@ -3163,12 +5249,17 @@ pub fn get_element_bounds_for_window(idx: usize,) -> Result<(i32, i32, u32, u32)> {if !crate::wayland::is_wayland() || !crate::wayland::hyprland::is_session() {- return get_element_bounds(pid, idx);+ return element_bounds_by_walk(pid, xid, idx);}- bounded(+ bounded_for(+ INDEX_RESOLVE_BUDGET,async {let conn = shared_connection().await?;- let (visited, scoped_frame) = collect_visited_bounded(conn, pid, xid, None, None)+ let Collected {+ visited,+ scoped_frame,+ ..+ } = collect_visited_bounded(conn, pid, xid, None, None).await?.context("no AT-SPI application")?;let scope =@@ -3183,6 +5274,7 @@ pub fn get_element_bounds_for_window(}element_bounds_for_visited(&visited, pid, xid, Some(scope)).await+ .0.into_iter().find(|(index, ..)| *index == idx).map(|(_, x, y, w, h)| (x, y, w, h))@@ -3194,7 +5286,39 @@ pub fn get_element_bounds_for_window(/// Real on-screen origin (root-relative top-left) of an X11 window, or `None`/// if it can't be resolved. Mirrors `list_windows`' geometry path.-fn x11_window_origin(xid: u64) -> Option<(i32, i32)> {+/// Size of the X11 display, for sanity-checking screen extents.+fn x11_display_size() -> Option<(u32, u32)> {+ use x11rb::connection::Connection as _;+ use x11rb::rust_connection::RustConnection;+ let (conn, screen_num) = RustConnection::connect(None).ok()?;+ let screen = &conn.setup().roots[screen_num];+ Some((+ u32::from(screen.width_in_pixels),+ u32::from(screen.height_in_pixels),+ ))+}++/// Whether a `CoordType::Screen` answer can be used as is. GTK 4.6 (Ubuntu+/// 22.04) reports correct Screen extents while its `CoordType::Window`+/// extents are relative to the *outer* X11 window (shadow included), so the+/// `_GTK_FRAME_EXTENTS` reconstruction lands (left, top) px too far; newer+/// GTK4 instead collapses Screen extents to (0,0). Trust a Screen answer when+/// it is neither that sentinel nor off the display.+pub(crate) fn screen_extents_trusted(+ raw: (i32, i32, i32, i32),+ display: Option<(u32, u32)>,+) -> bool {+ let (x, y, w, h) = raw;+ if !crate::snapshot_queries::plausible_raw_extents(raw) || (x == 0 && y == 0) {+ return false;+ }+ match display {+ Some((dw, dh)) => x < dw as i32 && y < dh as i32 && x + w > 0 && y + h > 0,+ None => true,+ }+}++pub(crate) fn x11_window_origin(xid: u64) -> Option<(i32, i32)> {use x11rb::protocol::xproto::*;use x11rb::rust_connection::RustConnection;@@ -3578,7 +5702,7 @@ async fn element_bounds_for_visited(pid: u32,xid: u64,scoped_frame: Option<usize>,-) -> Vec<(usize, i32, i32, u32, u32)> {+) -> (Vec<(usize, i32, i32, u32, u32)>, bool) {// Query WINDOW-relative extents and add a deterministic screen offset// (X11 window origin + GTK4 CSD inset). This fixes GTK4 — whose// CoordType::Screen reports every element at (0,0) — by using the@@ -3593,18 +5717,31 @@ async fn element_bounds_for_visited()).then_some(node.name.as_str())});- let offset = window_to_screen_offset(pid, xid, window_title);+ // Wayland's `window_to_screen_offset` path never touches X11 (it reads+ // compositor IPC / AT-SPI state instead), so only the X11 branch risks an+ // un-timed blocking round-trip; still route both through `bounded_blocking`+ // so this single call site can't stall the walk past its budget (#42).+ let window_title_owned = window_title.map(str::to_owned);+ let offset =+ bounded_blocking(move || window_to_screen_offset(pid, xid, window_title_owned.as_deref()))+ .await+ .flatten();if crate::wayland::is_wayland()&& crate::wayland::hyprland::is_session()&& (offset.is_none() || scoped_frame.is_none()){- return Vec::new();+ return (Vec::new(), true);}let coord = if offset.is_some() {CoordType::Window} else {CoordType::Screen};+ let display = if crate::wayland::is_wayland() {+ None+ } else {+ bounded_blocking(x11_display_size).await.flatten()+ };// Chromium on X11 labels its component extents as Screen while// returning coordinates relative to the renderer frame. Rebase// those values by comparing the top-level accessible frame with@@ -3613,7 +5750,9 @@ async fn element_bounds_for_visited(// required window-origin delta. GTK's explicit Window-coordinate// path above remains authoritative when available.let screen_rebase = if offset.is_none() && !crate::wayland::is_wayland() && xid != 0 {- let x11_origin = x11_window_origin(xid);+ let x11_origin = bounded_blocking(move || x11_window_origin(xid))+ .await+ .flatten();let frame = visited.iter().find(|node| {scoped_frame.is_none_or(|scope| node.frame_ordinal == scope)&& node.has_component@@ -3700,7 +5839,7 @@ async fn element_bounds_for_visited(// in time, but do not impose an index-based node cap: a cap silently// stripped frames from valid controls later in renderer trees and// made PX targeting depend on DOM order.- let deadline = tokio::time::Instant::now() + Duration::from_secs(20);+ let deadline = tokio::time::Instant::now() + Duration::from_secs(20).min(remaining_budget());// Bounds are independent read-only queries. Overlap a bounded number of// calls instead of serializing thousands of unrealized menu components.// Preserve original indices, all extents checks, and per-call timeouts.@@ -3710,6 +5849,16 @@ async fn element_bounds_for_visited(.map(|(idx, node)| async move {let proxies = call(node.acc.proxies()).await?.ok()?;let comp = call(proxies.component()).await?.ok()?;+ // A trustworthy Screen answer wins over the Window reconstruction+ // (see `screen_extents_trusted`); web content keeps its document+ // origin path.+ if coord == CoordType::Window && !node.in_web_doc {+ if let Some(Ok(raw)) = call(comp.get_extents(CoordType::Screen)).await {+ if screen_extents_trusted(raw, display) {+ return project_screen_extents(raw, (0, 0), None).map(|bounds| (idx, bounds));+ }+ }+ }if let Some(Ok((x, y, w, h))) = call(comp.get_extents(coord)).await {let document_origin = if node.in_web_doc {web_document_origin@@ -3722,16 +5871,140 @@ async fn element_bounds_for_visited(None});let collected = crate::snapshot_queries::collect_indexed(queries, deadline).await;- if tokio::time::Instant::now() >= deadline {+ let complete = tokio::time::Instant::now() < deadline;+ if !complete {dlog!(- "snapshot bounds: 20s budget exhausted; returning {} bound(s)",+ "snapshot bounds: time budget exhausted; returning {} bound(s)",collected.len());}- collected- .into_iter()- .map(|(idx, (x, y, w, h))| (idx, x, y, w, h))- .collect()+ (+ collected+ .into_iter()+ .map(|(idx, (x, y, w, h))| (idx, x, y, w, h))+ .collect(),+ complete,+ )+}++#[cfg(test)]+mod walk_bounds_tests {+ use super::walk_bounds_usable;++ #[test]+ fn degenerate_window_origin_is_rejected() {+ // A closed menu item projects (0,0,0,0) onto the window origin.+ assert_eq!(+ walk_bounds_usable(Some((40, 60, 0, 0)), Some((1920, 1080))),+ None+ );+ assert_eq!(walk_bounds_usable(None, Some((1920, 1080))), None);+ }++ #[test]+ fn off_screen_bounds_are_rejected() {+ assert_eq!(+ walk_bounds_usable(Some((2000, 10, 50, 20)), Some((1920, 1080))),+ None+ );+ assert_eq!(+ walk_bounds_usable(Some((-80, 10, 50, 20)), Some((1920, 1080))),+ None+ );+ }++ #[test]+ fn real_bounds_pass_through() {+ assert_eq!(+ walk_bounds_usable(Some((100, 200, 80, 24)), Some((1920, 1080))),+ Some((100, 200, 80, 24))+ );+ assert_eq!(+ walk_bounds_usable(Some((100, 200, 80, 24)), None),+ Some((100, 200, 80, 24))+ );+ }+}++#[cfg(test)]+mod screen_extents_tests {+ use super::screen_extents_trusted;++ #[test]+ fn trusts_plausible_on_screen_extents() {+ assert!(screen_extents_trusted(+ (70, 110, 848, 433),+ Some((1920, 1080))+ ));+ assert!(screen_extents_trusted(+ (76, 1045, 43, 32),+ Some((1920, 1080))+ ));+ }++ #[test]+ fn rejects_sentinels_and_off_screen_answers() {+ // GTK4 (newer a11y) collapses every element to the origin.+ assert!(!screen_extents_trusted((0, 0, 100, 20), Some((1920, 1080))));+ // Unrealized widgets report INT_MIN.+ assert!(!screen_extents_trusted(+ (i32::MIN, i32::MIN, 1, 1),+ Some((1920, 1080))+ ));+ // Entirely off the display.+ assert!(!screen_extents_trusted(+ (2000, 10, 40, 20),+ Some((1920, 1080))+ ));+ assert!(!screen_extents_trusted(+ (-500, 10, 40, 20),+ Some((1920, 1080))+ ));+ }+}++#[cfg(test)]+mod stacking_z_index_tests {+ use super::z_index_from_stacking;+ use crate::x11::WindowInfo;++ fn window(pid: u32, title: &str, z: usize) -> WindowInfo {+ WindowInfo {+ xid: 100 + z as u64,+ pid: Some(pid),+ app_name: "app".into(),+ title: title.into(),+ is_on_screen: true,+ z_index: Some(z),+ x: 0,+ y: 0,+ width: 10,+ height: 10,+ }+ }++ #[test]+ fn frames_take_the_stacking_position_of_the_same_pid_and_title() {+ let stacking = vec![+ window(7, "GNU Image Manipulation Program", 2),+ window(9, "Other", 3),+ window(7, "Brightness-Contrast", 4),+ ];+ assert_eq!(+ z_index_from_stacking(&stacking, 7, "Brightness-Contrast"),+ Some(4)+ );+ assert_eq!(+ z_index_from_stacking(&stacking, 7, "GNU Image Manipulation Program"),+ Some(2)+ );+ // Unknown title with several candidates: no guess.+ assert_eq!(z_index_from_stacking(&stacking, 7, "Untitled"), None);+ // A pid with exactly one X toplevel maps regardless of the title.+ assert_eq!(z_index_from_stacking(&stacking, 9, "anything"), Some(3));+ assert_eq!(z_index_from_stacking(&stacking, 11, "anything"), None);+ assert_eq!(z_index_from_stacking(&[], 7, "anything"), None);+ }}#[cfg(test)]@@ -3840,13 +6113,15 @@ mod frame_correlation_tests {#[cfg(test)]mod coord_tests {use super::parse_gtk_frame_extents;+ use super::OP_DEADLINE;use super::{- accepted_action, activation_index, before_snapshot_deadline,- combine_wayland_content_offsets, hyprland_document_top_inset, is_activation_action,- is_enabled_state, is_indexable_capabilities, is_passive_role, is_web_process_bus,+ activation_index, before_snapshot_deadline, bounded_blocking,+ combine_wayland_content_offsets, counts_as_showing, hyprland_document_top_inset,+ is_activation_action, is_enabled_state, is_indexable_capabilities, is_passive_role,+ is_showing_state, is_web_process_bus, listing_state, passive_marker,prefer_authoritative_wayland_origin, project_screen_extents, rebase_renderer_window_offset,- require_action_accepted, scoped_component_nodes, screen_extent_rebase, select_click_target,- select_web_document, ApplicationSelection,+ scoped_component_nodes, screen_extent_rebase, select_click_target, select_web_document,+ ApplicationSelection,};use atspi::{State, StateSet};use std::time::Duration;@@ -3919,6 +6194,69 @@ mod coord_tests {assert_eq!(tokio::time::Instant::now(), deadline);}+ // Regression coverage for #42: a synchronous, un-timed blocking call+ // (the shape of `window_to_screen_offset`/`x11_window_origin`'s raw X11+ // round-trips) executed directly inside the bounds phase must not be+ // able to hold the walk hostage past the operation's deadline, even+ // though the blocking work itself cannot be cancelled once started.+ #[tokio::test]+ async fn bounded_blocking_returns_none_once_the_operation_deadline_passes() {+ let deadline = tokio::time::Instant::now() + Duration::from_millis(20);+ let started = std::time::Instant::now();++ let result = OP_DEADLINE+ .scope(deadline, async {+ // Simulates a wedged X11 socket read: a plain blocking sleep+ // with no cooperative await points of its own, far longer+ // than the operation's deadline.+ bounded_blocking(|| {+ std::thread::sleep(Duration::from_secs(2));+ "would-be X11 reply"+ })+ .await+ })+ .await;++ assert_eq!(+ result, None,+ "bounded_blocking must give up once the deadline passes rather than \+ waiting for the blocking closure to finish"+ );+ assert!(+ started.elapsed() < Duration::from_secs(1),+ "the caller must not be held up by the still-running blocking thread; \+ took {:?}",+ started.elapsed()+ );+ }++ #[tokio::test]+ async fn bounded_blocking_returns_none_immediately_once_budget_is_exhausted() {+ // A deadline already in the past means `remaining_budget()` is zero,+ // so `bounded_blocking` must refuse to even spawn the closure —+ // exercising the fast-bail branch the per-node `call()` helper shares.+ let deadline = tokio::time::Instant::now();+ tokio::time::sleep(Duration::from_millis(1)).await;++ let ran = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false));+ let ran_in_closure = ran.clone();+ let result = OP_DEADLINE+ .scope(deadline, async {+ bounded_blocking(move || {+ ran_in_closure.store(true, std::sync::atomic::Ordering::SeqCst);+ 42+ })+ .await+ })+ .await;++ assert_eq!(result, None);+ assert!(+ !ran.load(std::sync::atomic::Ordering::SeqCst),+ "an already-exhausted budget must skip the closure entirely"+ );+ }+#[test]fn operable_nodes_are_addressable() {assert!(is_indexable_capabilities(@@ -3989,6 +6327,66 @@ mod coord_tests {));}+ #[test]+ fn hidden_and_disabled_controls_are_not_indexed_but_are_named() {+ // `Showing` is the on-screen state.+ assert!(!is_showing_state(&StateSet::new(+ State::Enabled | State::Visible+ )));+ assert!(is_showing_state(&StateSet::new(+ State::Enabled | State::Visible | State::Showing+ )));+ // A Visible-but-not-Showing control (scrolled out of its viewport, a+ // hover-only eject button) stays listed, marked off-screen; a hidden+ // one (not Visible) is left out.+ assert_eq!(+ listing_state(+ "push button",+ &StateSet::new(State::Enabled | State::Visible)+ ),+ (true, true)+ );+ assert_eq!(+ listing_state("push button", &StateSet::new(State::Enabled)),+ (false, false)+ );+ assert_eq!(+ listing_state(+ "push button",+ &StateSet::new(State::Enabled | State::Visible | State::Showing)+ ),+ (true, false)+ );+ // ...but a GTK2 menu item is never `Showing`, and stays indexed while `Visible`.+ assert!(!counts_as_showing(+ "push button",+ &StateSet::new(State::Enabled | State::Visible)+ ));+ assert!(counts_as_showing(+ "menu item",+ &StateSet::new(State::Enabled | State::Visible)+ ));+ assert!(counts_as_showing(+ "check menu item",+ &StateSet::new(State::Visible)+ ));+ assert!(!counts_as_showing(+ "menu item",+ &StateSet::new(State::Enabled)+ ));+ // A visible but insensitive button keeps a marker in the markdown.+ assert_eq!(+ passive_marker("push button", true, true, Some(false)),+ " (disabled)"+ );+ assert_eq!(+ passive_marker("push button", false, true, Some(false)),+ " (disabled)"+ );+ assert_eq!(passive_marker("push button", true, true, Some(true)), "");+ assert_eq!(passive_marker("label", false, true, Some(false)), "");+ }+#[test]fn gtk_state_sets_establish_operability_from_enabled_or_sensitive() {assert!(is_enabled_state(&StateSet::new(State::Enabled)));@@ -4274,25 +6672,6 @@ mod coord_tests {assert_eq!(select_click_target(&[], 0, 0), None); // no frames}- #[test]- fn rejected_action_acknowledgement_does_not_report_success() {- assert_eq!(accepted_action(false, Some("click".to_owned())), None);- }-- #[test]- fn accepted_action_acknowledgement_reports_the_selected_action() {- assert_eq!(- accepted_action(true, Some("click".to_owned())),- Some("click".to_owned())- );- }-- #[test]- fn indexed_action_rejection_is_an_error() {- assert!(require_action_accepted(false).is_err());- assert!(require_action_accepted(true).is_ok());- }-#[test]fn passive_roles_classified() {assert!(is_passive_role("label"));@@ -4424,3 +6803,210 @@ mod coord_tests {assert_eq!(activation_index("button", &sparse), Some(2));}}++#[cfg(test)]+mod budget_tests {+ use super::*;++ #[test]+ fn focus_log_tracks_latest_focused_object_per_bus_and_clears_on_blur() {+ let bus = ":9.test-focus-log";+ note_focus_event(bus, "/a", true);+ assert_eq!(+ focus_map().lock().unwrap().get(bus).map(String::as_str),+ Some("/a")+ );+ note_focus_event(bus, "/b", true);+ assert_eq!(+ focus_map().lock().unwrap().get(bus).map(String::as_str),+ Some("/b")+ );+ // A blur for a stale object must not erase the newer focus.+ note_focus_event(bus, "/a", false);+ assert_eq!(+ focus_map().lock().unwrap().get(bus).map(String::as_str),+ Some("/b")+ );+ note_focus_event(bus, "/b", false);+ assert!(focus_map().lock().unwrap().get(bus).is_none());+ }++ #[test]+ fn per_call_timeout_clamps_to_operation_deadline() {+ let rt = tokio::runtime::Builder::new_current_thread()+ .enable_all()+ .build()+ .unwrap();+ rt.block_on(async {+ // No deadline in scope: the historical per-call timeout applies.+ assert_eq!(remaining_budget(), CALL_TIMEOUT);+ assert!(!deadline_passed());+ let deadline = tokio::time::Instant::now() + Duration::from_millis(200);+ OP_DEADLINE+ .scope(deadline, async {+ assert!(remaining_budget() <= Duration::from_millis(200));+ // A call that would outlive the remaining budget is cut at+ // the deadline, not at CALL_TIMEOUT.+ let started = std::time::Instant::now();+ let got = call(tokio::time::sleep(Duration::from_secs(10))).await;+ assert!(got.is_none());+ assert!(started.elapsed() < Duration::from_secs(2));+ assert!(deadline_passed());+ // Once the deadline has passed, calls short-circuit.+ let started = std::time::Instant::now();+ assert!(call(tokio::time::sleep(Duration::from_secs(10)))+ .await+ .is_none());+ assert!(started.elapsed() < Duration::from_millis(50));+ })+ .await;+ });+ }++ #[test]+ fn bounded_for_returns_partial_via_on_timeout_when_work_ignores_deadline() {+ let started = std::time::Instant::now();+ let result: Result<&'static str> = bounded_for(+ Duration::from_millis(100),+ async {+ tokio::time::sleep(Duration::from_secs(30)).await;+ Ok("finished")+ },+ || Ok("partial"),+ );+ assert_eq!(result.unwrap(), "partial");+ // deadline + 500 ms backstop, well under the old 25 s.+ assert!(started.elapsed() < Duration::from_secs(3));+ }+}++#[cfg(test)]+mod at_point_rules_tests {+ use super::*;++ fn acts(names: &[&str]) -> Vec<String> {+ names.iter().map(|n| n.to_string()).collect()+ }++ #[test]+ fn container_activation_never_bubbles() {+ // Nautilus: canvas item (no Action) under `layered pane "Icon View"`+ // [activate, menu]: the container is an ancestor -> not fired.+ assert_eq!(+ at_point_activation_index("layered pane", &acts(&["activate", "menu"]), false),+ None+ );+ for role in [+ "list",+ "table",+ "tree",+ "tree table",+ "icon view",+ "panel",+ "list box",+ ] {+ assert_eq!(+ at_point_activation_index(role, &acts(&["activate"]), false),+ None+ );+ // A click on the container's own empty area must not open its+ // current selection either.+ assert_eq!(+ at_point_activation_index(role, &acts(&["activate"]), true),+ None+ );+ }+ // A canvas is a surface: as an ancestor its activate is the+ // container's; as the deepest hit it is the widget itself.+ assert_eq!(+ at_point_activation_index("canvas", &acts(&["activate"]), false),+ None+ );+ assert_eq!(+ at_point_activation_index("canvas", &acts(&["activate"]), true),+ Some(0)+ );+ }++ #[test]+ fn real_actuators_still_fire_from_any_depth() {+ assert_eq!(+ at_point_activation_index("push button", &acts(&["click"]), false),+ Some(0)+ );+ assert_eq!(+ at_point_activation_index("menu item", &acts(&["click"]), true),+ Some(0)+ );+ assert_eq!(+ at_point_activation_index("check box", &acts(&["check"]), true),+ Some(0)+ );+ // Non-activation verbs never fire, container or not.+ assert_eq!(+ at_point_activation_index("push button", &acts(&["buffer.delete-line"]), true),+ None+ );+ }++ #[test]+ fn cell_roles_take_a_real_press() {+ for role in ["table cell", "cell", "Table Cell"] {+ assert!(is_cell_role(role), "{role}");+ assert!(is_selectable_item_role(role), "{role}");+ }+ for role in [+ "canvas",+ "list item",+ "icon",+ "tree item",+ "row",+ "table row",+ ] {+ assert!(!is_cell_role(role), "{role}");+ }+ }++ #[test]+ fn selectable_item_roles() {+ for role in [+ "canvas",+ "list item",+ "table cell",+ "tree item",+ "icon",+ "row",+ ] {+ assert!(is_selectable_item_role(role), "{role}");+ }+ for role in ["push button", "layered pane", "label", "menu item"] {+ assert!(!is_selectable_item_role(role), "{role}");+ }+ }++ #[test]+ fn hit_description_names_what_was_fired() {+ let hit = AtPointHit::fired(+ "activate".into(),+ "push button".into(),+ "Restore".into(),+ "/org/a11y/atspi/accessible/42".into(),+ );+ assert_eq!(+ hit.describe(),+ "hit: push button \"Restore\" action=activate path=/org/a11y/atspi/accessible/42"+ );+ let selected = AtPointHit {+ action: "select_child".into(),+ role: "canvas".into(),+ name: "file.txt".into(),+ path: "/x".into(),+ selected: Some("file.txt".into()),+ selection_verified: true,+ };+ assert!(selected+ .describe()+ .starts_with("selected: canvas \"file.txt\""));+ assert!(selected.describe().contains("read back as selected"));+ }+}diff --git a/libs/cua-driver/rust/crates/platform-linux/src/atspi/native/hit_test.rs b/libs/cua-driver/rust/crates/platform-linux/src/atspi/native/hit_test.rsnew file mode 100644index 0000000000..ec699ffcf0--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-linux/src/atspi/native/hit_test.rs@@ -0,0 +1,318 @@+//! Point ownership for element clicks.+//!+//! A snapshot frame is where the toolkit *said* an element is; a real pointer+//! press lands on whatever the toolkit *draws* there. The two disagree when a+//! view that is not on screen still reports its rows with frames: Qt's+//! `QAccessibleTableCell::state()` never derives `invisible` from its view, so+//! the QListView of a QFileDialog in Detail mode (a QStackedWidget page that is+//! not current) lists every row over the visible QTreeView. Pressing the+//! list-row centre hit the tree's "Name" column header and flipped the sort+//! order instead of selecting the file (VLC "Convert / Save", round 7).+//!+//! Before a press, ask the toolkit's own `GetAccessibleAtPoint` who owns the+//! resolved point. The observed object (or an ancestor of the deepest hit)+//! owning it means the press is honest. Otherwise look for the same-named+//! selectable item of the table-like container the point really belongs to+//! (the visible tree row) and press that; failing that the caller falls back+//! to the element's AT-SPI action or refuses with the owner named.++use super::*;++/// The accessible that owns a screen point, as the toolkit answers it.+#[derive(Clone, Debug, PartialEq, Eq)]+pub struct PointOwner {+ pub object_ref: ObjectRef,+ pub role: String,+ pub name: String,+}++impl PointOwner {+ /// `tree item "Desktop"` / `column header` (unnamed).+ pub fn describe(&self) -> String {+ if self.name.is_empty() {+ format!("{} (unnamed)", self.role)+ } else {+ format!("{} \"{}\"", self.role, self.name)+ }+ }+}++/// What a hit-test of an element's press point found.+#[derive(Clone, Debug, PartialEq, Eq)]+pub enum PointOwnership {+ /// The element itself, or one of its descendants, owns the point.+ Owned,+ /// Another object owns the point, but a same-named selectable item of a+ /// table-like container on the owner's chain has a point of its own.+ Redirect {+ owner: PointOwner,+ target: PointOwner,+ /// Screen centre of `target`.+ screen_point: (i32, i32),+ },+ /// Another object owns the point and no equivalent item was found.+ Mismatch { owner: PointOwner },+ /// The toolkit could not be asked (Wayland, no application, no+ /// `GetAccessibleAtPoint` answer, timeout): the press proceeds unchecked.+ Unknown,+}++/// Rows of a table-like container the name lookup may redirect to.+fn is_row_container_role(role_lower: &str) -> bool {+ matches!(role_lower, "tree" | "tree table" | "table" | "list")+}++/// Children a redirect lookup reads from one container at most.+const REDIRECT_CHILDREN_CAP: usize = 512;++async fn describe_node(node: &HitNode<'_>) -> PointOwner {+ let role = call(node.acc.get_role_name())+ .await+ .and_then(Result::ok)+ .unwrap_or_default();+ let name = call(node.acc.name())+ .await+ .and_then(Result::ok)+ .unwrap_or_default();+ PointOwner {+ object_ref: ObjectRef {+ bus: node.oref.name.clone(),+ path: node.oref.path.clone(),+ },+ role,+ name,+ }+}++/// The same-named selectable item among the children of the table-like+/// containers on `chain` (deepest first), with its trusted screen centre.+async fn same_named_item_on_chain(+ conn: &AccessibilityConnection,+ chain: &[HitNode<'_>],+ name: &str,+ display: Option<(u32, u32)>,+) -> Option<(PointOwner, (i32, i32))> {+ for node in chain.iter().rev().skip(1) {+ let Some(Ok(role)) = call(node.acc.get_role_name()).await else {+ continue;+ };+ if !is_row_container_role(&role.trim().to_ascii_lowercase()) {+ continue;+ }+ let Some(Ok(children)) = call(raw_children(conn.connection(), &node.oref)).await else {+ continue;+ };+ for child in children.iter().take(REDIRECT_CHILDREN_CAP) {+ let Some(Ok(acc)) = call(accessible_for(conn, child)).await else {+ continue;+ };+ let Some(Ok(child_name)) = call(acc.name()).await else {+ continue;+ };+ if child_name != name {+ continue;+ }+ let Some(Ok(child_role)) = call(acc.get_role_name()).await else {+ continue;+ };+ if !is_selectable_item_role(&child_role) {+ continue;+ }+ let Some(Ok(proxies)) = call(acc.proxies()).await else {+ continue;+ };+ let Some(Ok(component)) = call(proxies.component()).await else {+ continue;+ };+ let Some(Ok(raw)) = call(component.get_extents(CoordType::Screen)).await else {+ continue;+ };+ if !screen_extents_trusted(raw, display) || raw.2 <= 0 || raw.3 <= 0 {+ continue;+ }+ let centre = (raw.0 + raw.2 / 2, raw.1 + raw.3 / 2);+ return Some((+ PointOwner {+ object_ref: ObjectRef {+ bus: child.name.clone(),+ path: child.path.clone(),+ },+ role: child_role,+ name: child_name,+ },+ centre,+ ));+ }+ }+ None+}++/// Who owns screen point `(sx, sy)` of window `xid` in `pid`'s application,+/// relative to the snapshot object `observed` a click is about to press.+/// Blocking; bounded by [`INPUT_QUERY_BUDGET`]. Never errors: an unanswered+/// question is [`PointOwnership::Unknown`].+pub fn point_ownership(+ pid: u32,+ xid: u64,+ observed: &ObjectRef,+ sx: i32,+ sy: i32,+) -> PointOwnership {+ if crate::wayland::is_wayland() {+ return PointOwnership::Unknown;+ }+ let Some(client_origin) = x11_window_origin(xid) else {+ return PointOwnership::Unknown;+ };+ let (win_x, win_y) = (sx - client_origin.0, sy - client_origin.1);+ let display = x11_display_size();+ let observed_raw = raw_ref(observed);+ let result: Result<PointOwnership> = bounded_for(+ INPUT_QUERY_BUDGET,+ async {+ let conn = shared_connection().await?;+ let Some(app) = app_for_pid(conn, pid).await? else {+ return Ok(PointOwnership::Unknown);+ };+ let seeds: Vec<RawObjectRef> = match call(app.get_children()).await {+ Some(Ok(children)) => children+ .into_iter()+ .filter_map(|child| RawObjectRef::from_atspi(&child))+ .collect(),+ _ => return Ok(PointOwnership::Unknown),+ };+ let scoped = resolve_window_frame(conn, pid, xid, &seeds).await;+ let (hit_x, hit_y) = match scoped {+ Some(ordinal) => {+ let frame_origin = decoration_frame_origin(conn, &app, &seeds[ordinal]).await;+ at_point_toolkit_coords((win_x, win_y), Some(client_origin), frame_origin)+ }+ None => (win_x, win_y),+ };+ let ordered: Vec<RawObjectRef> = match scoped {+ Some(ordinal) => seeds.get(ordinal).cloned().into_iter().collect(),+ None => seeds,+ };+ for seed in ordered {+ let chain = descend_at_point(conn, seed, hit_x, hit_y).await;+ if chain.len() <= 1 {+ continue;+ }+ if chain.iter().any(|node| node.oref == observed_raw) {+ return Ok(PointOwnership::Owned);+ }+ let deepest = chain.last().expect("chain has more than one node");+ // The toolkit stopped at one of the target's own containers+ // (a web document answers with its `panel` / `filler`): it+ // cannot resolve the point any deeper, which is not evidence+ // that a different control owns it.+ if is_ancestor_of(conn, &deepest.oref, &observed_raw).await {+ return Ok(PointOwnership::Unknown);+ }+ let owner = describe_node(deepest).await;+ let observed_name = match call(accessible_for(conn, &observed_raw)).await {+ Some(Ok(acc)) => call(acc.name())+ .await+ .and_then(Result::ok)+ .unwrap_or_default(),+ _ => String::new(),+ };+ if !observed_name.is_empty() {+ if let Some((target, screen_point)) =+ same_named_item_on_chain(conn, &chain, &observed_name, display).await+ {+ if target.object_ref != observed.clone() {+ return Ok(PointOwnership::Redirect {+ owner,+ target,+ screen_point,+ });+ }+ }+ }+ return Ok(PointOwnership::Mismatch { owner });+ }+ Ok(PointOwnership::Unknown)+ },+ || Ok(PointOwnership::Unknown),+ );+ result.unwrap_or(PointOwnership::Unknown)+}++/// Whether `ancestor` is on `node`'s parent chain (bounded).+async fn is_ancestor_of(+ conn: &AccessibilityConnection,+ ancestor: &RawObjectRef,+ node: &RawObjectRef,+) -> bool {+ let same = |a: &RawObjectRef, b: &RawObjectRef| a.name == b.name && a.path == b.path;+ let mut current = node.clone();+ for _ in 0..32 {+ let Some(Ok(acc)) = call(accessible_for(conn, ¤t)).await else {+ return false;+ };+ let Some(Ok(parent)) = call(acc.parent()).await else {+ return false;+ };+ let Some(parent) = RawObjectRef::from_atspi(&parent) else {+ return false;+ };+ if same(&parent, ancestor) {+ return true;+ }+ if same(&parent, ¤t) {+ return false;+ }+ current = parent;+ }+ false+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn owner_description_names_role_and_name() {+ let owner = PointOwner {+ object_ref: ObjectRef {+ bus: ":1.9".into(),+ path: "/a".into(),+ },+ role: "column header".into(),+ name: String::new(),+ };+ assert_eq!(owner.describe(), "column header (unnamed)");+ let named = PointOwner {+ name: "Desktop".into(),+ role: "tree item".into(),+ ..owner+ };+ assert_eq!(named.describe(), "tree item \"Desktop\"");+ }++ #[test]+ fn row_containers_are_the_table_like_roles() {+ assert!(is_row_container_role("tree"));+ assert!(is_row_container_role("tree table"));+ assert!(is_row_container_role("list"));+ assert!(!is_row_container_role("panel"));+ assert!(!is_row_container_role("menu"));+ }++ #[test]+ fn no_display_is_unknown_not_an_error() {+ let prior = std::env::var_os("DISPLAY");+ std::env::set_var("DISPLAY", ":9999999");+ let observed = ObjectRef {+ bus: ":1.9".into(),+ path: "/org/a11y/atspi/accessible/1".into(),+ };+ let ownership = point_ownership(1, 0x1234, &observed, 10, 10);+ match prior {+ Some(v) => std::env::set_var("DISPLAY", v),+ None => std::env::remove_var("DISPLAY"),+ }+ assert_eq!(ownership, PointOwnership::Unknown);+ }+}diff --git a/libs/cua-driver/rust/crates/platform-linux/src/browser_consent_ui.rs b/libs/cua-driver/rust/crates/platform-linux/src/browser_consent_ui.rsindex bf61b4ca7a..665bdb078d 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/browser_consent_ui.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/browser_consent_ui.rs@@ -369,6 +369,7 @@ mod tests {depth: 0,parent_element_index: None,in_web_content: false,+ object_ref: None,}}diff --git a/libs/cua-driver/rust/crates/platform-linux/src/browser_setup_ui.rs b/libs/cua-driver/rust/crates/platform-linux/src/browser_setup_ui.rsindex d9fd19f924..aca7cd20ad 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/browser_setup_ui.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/browser_setup_ui.rs@@ -689,6 +689,7 @@ mod tests {depth: 0,parent_element_index: None,in_web_content: false,+ object_ref: None,}}diff --git a/libs/cua-driver/rust/crates/platform-linux/src/capture.rs b/libs/cua-driver/rust/crates/platform-linux/src/capture.rsindex bf6cd24fc4..f63cd17059 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/capture.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/capture.rs@@ -86,11 +86,38 @@ fn capture_window_with_backends())}+/// Longest an `import` run may take before it is killed.+///+/// Given a window id the server no longer knows, ImageMagick falls back to+/// its interactive `XSelectWindow`: it grabs the X server and waits for a+/// mouse click that never comes, and every other X client on the display+/// hangs with it until the process dies. Killing it releases the grab.+const IMPORT_TIMEOUT: Duration = Duration::from_secs(5);+fn capture_via_import(xid: u64) -> Result<Vec<u8>> {- let out = Command::new("import")+ if !crate::x11::window_exists(xid) {+ bail!("window {xid} no longer exists");+ }+ let child = Command::new("import").args(["-window", &xid.to_string(), "png:-"])- .output()+ .stdin(std::process::Stdio::null())+ .stdout(std::process::Stdio::piped())+ .stderr(std::process::Stdio::piped())+ .spawn().map_err(|e| anyhow!("failed to launch ImageMagick import: {e}"))?;+ let pid = child.id();+ let (tx, rx) = std::sync::mpsc::channel();+ std::thread::spawn(move || {+ let _ = tx.send(child.wait_with_output());+ });+ let out = match rx.recv_timeout(IMPORT_TIMEOUT) {+ Ok(out) => out.map_err(|e| anyhow!("failed to wait for ImageMagick import: {e}"))?,+ Err(_) => {+ // SAFETY: plain libc call on a pid this process spawned and has not reaped.+ unsafe { libc::kill(pid as libc::pid_t, libc::SIGKILL) };+ bail!("ImageMagick import did not exit within {IMPORT_TIMEOUT:?} and was killed");+ }+ };if !out.status.success() {let stderr = String::from_utf8_lossy(&out.stderr);let detail = stderr.trim().chars().take(512).collect::<String>();@@ -921,6 +948,52 @@ pub(crate) fn screenshot_display_bytes_x11() -> Result<Vec<u8>> {cua_driver_core::image_utils::encode_rgba_to_png(&rgba, w, h)}+/// Capture a rectangle of the root window (what is on the screen there,+/// override-redirect popups and transient dialogs included) as PNG. Used+/// for a window whose own drawable would hide the menu or dialog open over+/// it. The rectangle is clipped to the screen.+pub fn screenshot_root_region_png(x: i32, y: i32, width: u32, height: u32) -> Result<Vec<u8>> {+ use x11rb::connection::Connection;+ use x11rb::protocol::xproto::*;+ use x11rb::rust_connection::RustConnection;+ let (conn, screen_num) = RustConnection::connect(None)+ .map_err(|e| anyhow::anyhow!("{e}{}", crate::no_display_hint()))?;+ let root = conn.setup().roots[screen_num].root;+ let geom = conn.get_geometry(root)?.reply()?;+ let (sw, sh) = (i32::from(geom.width), i32::from(geom.height));+ let x0 = x.clamp(0, sw);+ let y0 = y.clamp(0, sh);+ let x1 = (x.saturating_add(width as i32)).clamp(0, sw);+ let y1 = (y.saturating_add(height as i32)).clamp(0, sh);+ let (w, h) = ((x1 - x0) as u32, (y1 - y0) as u32);+ if w == 0 || h == 0 {+ anyhow::bail!(+ "window rectangle {x},{y} {width}x{height} lies outside the {sw}x{sh} screen"+ );+ }+ let img = conn+ .get_image(+ ImageFormat::Z_PIXMAP,+ root,+ x0 as i16,+ y0 as i16,+ w as u16,+ h as u16,+ !0u32,+ )?+ .reply()?;+ let bpp = match img.depth {+ 32 | 24 => 4usize,+ _ => anyhow::bail!("Unsupported depth"),+ };+ let mut rgba = Vec::with_capacity((w * h * 4) as usize);+ for chunk in img.data.chunks_exact(bpp) {+ let (b, g, r) = (chunk[0], chunk[1], chunk[2]);+ rgba.extend_from_slice(&[r, g, b, 255]);+ }+ cua_driver_core::image_utils::encode_rgba_to_png(&rgba, w, h)+}+/// Capture the primary display, returning (base64_png, width, height).pub fn screenshot_display() -> Result<(String, u32, u32)> {let png_bytes = screenshot_display_bytes()?;diff --git a/libs/cua-driver/rust/crates/platform-linux/src/capture_action_frame.rs b/libs/cua-driver/rust/crates/platform-linux/src/capture_action_frame.rsindex 0770189a34..af6cb35e03 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/capture_action_frame.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/capture_action_frame.rs@@ -58,13 +58,6 @@ pub fn publish_window(encoded_dimensions: (u32, u32),native_action_dimensions: (u32, u32),) -> anyhow::Result<String> {- // The resizer preserves aspect ratio before rounding each encoded axis.- // Derive both ratios independently so a one-pixel rounded height does not- // skew Y coordinates in the native window frame.- let scale_x = f64::from(native_action_dimensions.0) / f64::from(encoded_dimensions.0);- let scale_y = f64::from(native_action_dimensions.1) / f64::from(encoded_dimensions.1);- let screenshot_to_action =- ScreenshotToActionTransform::new(scale_x, 0.0, 0.0, scale_y, 0.0, 0.0)?;publish(service,args,@@ -72,7 +65,7 @@ pub fn publish_window(window_target(pid, window_id),encoded_dimensions,native_action_dimensions,- screenshot_to_action,+ scaled_transform(encoded_dimensions, native_action_dimensions)?,)}@@ -80,19 +73,42 @@ pub fn publish_desktop(service: &CaptureService,args: &Value,png_bytes: &[u8],- dimensions: (u32, u32),+ encoded_dimensions: (u32, u32),+ native_action_dimensions: (u32, u32),) -> anyhow::Result<String> {+ // The desktop screenshot can be downsized below the action frame.publish(service,args,png_bytes,CaptureTarget::PrimaryDesktop,- dimensions,- dimensions,- ScreenshotToActionTransform::identity(),+ encoded_dimensions,+ native_action_dimensions,+ scaled_transform(encoded_dimensions, native_action_dimensions)?,)}+/// Screenshot-to-action scaling for a capture encoded at `encoded` pixels of+/// a `native` action frame. The resizer preserves aspect ratio before rounding+/// each encoded axis, so both ratios are derived independently: a one-pixel+/// rounded height must not skew Y coordinates.+fn scaled_transform(+ encoded: (u32, u32),+ native: (u32, u32),+) -> anyhow::Result<ScreenshotToActionTransform> {+ anyhow::ensure!(+ encoded.0 > 0 && encoded.1 > 0,+ "capture has an empty encoded frame: {}x{}",+ encoded.0,+ encoded.1+ );+ let scale_x = f64::from(native.0) / f64::from(encoded.0);+ let scale_y = f64::from(native.1) / f64::from(encoded.1);+ Ok(ScreenshotToActionTransform::new(+ scale_x, 0.0, 0.0, scale_y, 0.0, 0.0,+ )?)+}+fn admit(service: &CaptureService,args: &Value,@@ -355,7 +371,7 @@ mod tests {fn retired_session_capture_is_stale() {let service = CaptureService::default();let call_args = args("retired");- let id = publish_desktop(&service, &call_args, &png(3, 2, 0x77), (3, 2)).unwrap();+ let id = publish_desktop(&service, &call_args, &png(3, 2, 0x77), (3, 2), (3, 2)).unwrap();let binding = service.binding_from_args(&call_args).unwrap();service.retire_session(&binding);@@ -387,7 +403,7 @@ mod tests {fn another_session_cannot_admit_or_consume_a_capture() {let service = CaptureService::default();let owner = args("owner");- let id = publish_desktop(&service, &owner, &png(4, 3, 0x21), (4, 3)).unwrap();+ let id = publish_desktop(&service, &owner, &png(4, 3, 0x21), (4, 3), (4, 3)).unwrap();let refusal = admit_desktop(&service, &args("other"), &id, (2.0, 1.0), (4, 3)).expect_err("cross-session admission must fail");@@ -414,6 +430,7 @@ mod tests {&call_args,&png(logical.0, logical.1, 0x31),logical,+ logical,).unwrap();assert_eq!(@@ -422,6 +439,17 @@ mod tests {);}+ #[test]+ fn downsized_desktop_capture_maps_back_to_the_action_frame() {+ let service = CaptureService::default();+ let call_args = args("downsized");+ let id = publish_desktop(&service, &call_args, &png(4, 3, 0x42), (4, 3), (8, 6)).unwrap();+ assert_eq!(+ admit_desktop(&service, &call_args, &id, (2.0, 1.5), (8, 6)).unwrap(),+ (4.0, 3.0)+ );+ }+#[test]fn capture_action_refusals_have_stable_specific_codes() {for (error, code) in [@@ -462,7 +490,7 @@ mod tests {fn desktop_publication_uses_the_post_normalization_bytes() {let service = CaptureService::default();let normalized = png(4, 3, 0x18);- let id = publish_desktop(&service, &args("desktop"), &normalized, (4, 3)).unwrap();+ let id = publish_desktop(&service, &args("desktop"), &normalized, (4, 3), (4, 3)).unwrap();let binding = service.binding_from_args(&args("desktop")).unwrap();let capture = service.read_for_perception(service.parse_capture_id(&id).unwrap(), &binding)diff --git a/libs/cua-driver/rust/crates/platform-linux/src/input/delivery.rs b/libs/cua-driver/rust/crates/platform-linux/src/input/delivery.rsindex c01d0621aa..35edfb0f6a 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/input/delivery.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/input/delivery.rs@@ -37,7 +37,7 @@ pub enum DeliveryMode {/// Inject without activating the target (X11 XTEST/MPX; Wayland libei-to-focus).#[default]Background,- /// Activate the target, inject, restore prior active window.+ /// Activate the target, confirm focus, inject real input, leave it active.Foreground,}@@ -146,8 +146,8 @@ pub fn background_unavailable_error(let detail = reason.detail();cua_driver_core::protocol::ToolResult::error(format!("Background delivery is not available: {detail}. Retry this action with \- delivery_mode:\"foreground\"; Cua Driver will activate the target for \- the action and restore the previous foreground afterward."+ delivery_mode:\"foreground\"; Cua Driver will activate the target window, \+ confirm it holds the input focus, and deliver real input to it.")).with_structured(serde_json::json!({"code": reason.code(),diff --git a/libs/cua-driver/rust/crates/platform-linux/src/input/focus_guard.rs b/libs/cua-driver/rust/crates/platform-linux/src/input/focus_guard.rsnew file mode 100644index 0000000000..78d2959874--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-linux/src/input/focus_guard.rs@@ -0,0 +1,986 @@+//! Background non-invasiveness safety net for X11.+//!+//! A `delivery_mode:"background"` action must leave the user's desktop as it+//! found it: the core keyboard focus (`XGetInputFocus`), the window manager's+//! active window (`_NET_ACTIVE_WINDOW`) and the top of the stacking order.+//! The focus-free routes (MPX virtual keyboard, AT-SPI actions) never touch+//! those themselves, but the *application* may: GTK/VCL activate their+//! toplevel when a menu opens, mutter focuses a freshly mapped transient+//! dialog, Nautilus presents its window after a navigation shortcut.+//!+//! [`FocusSnapshot::capture`] records the three signals before an action and+//! [`FocusSnapshot::restore_if_changed`] re-asserts them afterwards when they+//! moved *away from the user*: the previously active window is re-activated+//! through EWMH (so the WM's own bookkeeping and stacking follow) and the+//! previous core focus is re-set.+//!+//! A move that stays inside the target application is not an intrusion: when+//! the target app already owned the focus and it opens a dialog that mutter+//! focuses, the user (or the agent) asked for that dialog. Re-activating the+//! main window there would undo the app's own behaviour and send the next+//! pid-only keystrokes to the wrong window, so the guard leaves it and reports+//! `focus_outcome: same_app_dialog` instead. The outcome is reported, never+//! hidden: a toolkit keyboard grab held by an open menu cannot be undone+//! without closing the menu, so it is surfaced as `grab_held_by`.++use anyhow::{anyhow, Result};+use std::collections::HashSet;+use std::time::{Duration, Instant};+use x11rb::connection::Connection as _;+use x11rb::protocol::xproto::{+ AtomEnum, ClientMessageEvent, ConnectionExt as _, CreateWindowAux, EventMask, InputFocus,+ MapState, PropMode, Window, WindowClass,+};+use x11rb::protocol::Event;+use x11rb::rust_connection::RustConnection;+use x11rb::COPY_DEPTH_FROM_PARENT;++/// How long the guard watches for a late focus change after delivery (a+/// transient dialog is mapped and focused by the WM a beat after the action).+const SETTLE_WATCH: Duration = Duration::from_millis(220);+/// Longer watch once a new top-level appeared during the short one while the+/// focus belonged to *another* application: a dialog (LibreOffice's take+/// ~1 s to build) is focused by the WM only when mapped, and that steal must+/// be undone. When the target app already owns the focus the new window is+/// its own and there is nothing to wait for.+const SETTLE_WATCH_NEW_WINDOW: Duration = Duration::from_millis(900);+const SETTLE_POLL: Duration = Duration::from_millis(30);+/// Bound on the restore loop: re-activation, verification, one re-send.+// Together with the extended settle watch this bounds the guard at+// ~1.5 s worst case (menubar clicks were spending 4-9 s here while a popup's+// keyboard grab kept every re-activation from sticking).+const RESTORE_BUDGET: Duration = Duration::from_millis(600);+const RESTORE_POLL: Duration = Duration::from_millis(50);+/// Consecutive stable polls before the restore is called done.+const STABLE_POLLS: u32 = 3;+/// Bound on the ancestor / transient walk when attributing a window to a pid.+const OWNER_WALK_LIMIT: usize = 16;++struct Atoms {+ net_active_window: u32,+ net_client_list_stacking: u32,+ net_wm_pid: u32,+ net_wm_name: u32,+ utf8_string: u32,+}++struct X {+ conn: RustConnection,+ root: Window,+ atoms: Atoms,+}++impl X {+ fn open() -> Result<Self> {+ let (conn, screen) = RustConnection::connect(None)+ .map_err(|e| anyhow!("focus guard: cannot open DISPLAY: {e}"))?;+ let root = conn.setup().roots[screen].root;+ let intern =+ |name: &[u8]| -> Result<u32> { Ok(conn.intern_atom(false, name)?.reply()?.atom) };+ let atoms = Atoms {+ net_active_window: intern(b"_NET_ACTIVE_WINDOW")?,+ net_client_list_stacking: intern(b"_NET_CLIENT_LIST_STACKING")?,+ net_wm_pid: intern(b"_NET_WM_PID")?,+ net_wm_name: intern(b"_NET_WM_NAME")?,+ utf8_string: intern(b"UTF8_STRING")?,+ };+ Ok(Self { conn, root, atoms })+ }++ fn window_property(&self, window: Window, atom: u32, len: u32) -> Vec<u32> {+ self.conn+ .get_property(false, window, atom, AtomEnum::ANY, 0, len)+ .ok()+ .and_then(|c| c.reply().ok())+ .and_then(|r| r.value32().map(|v| v.collect()))+ .unwrap_or_default()+ }++ fn active_window(&self) -> Option<Window> {+ self.window_property(self.root, self.atoms.net_active_window, 1)+ .first()+ .copied()+ .filter(|w| *w != 0)+ }++ fn client_list(&self) -> Vec<Window> {+ self.window_property(self.root, self.atoms.net_client_list_stacking, u32::MAX)+ }++ fn stacking_top(&self) -> Option<Window> {+ self.client_list().last().copied().filter(|w| *w != 0)+ }++ fn core_focus(&self) -> (Window, InputFocus) {+ match self+ .conn+ .get_input_focus()+ .ok()+ .and_then(|c| c.reply().ok())+ {+ Some(reply) => (reply.focus, reply.revert_to),+ None => (0, InputFocus::NONE),+ }+ }++ fn window_pid(&self, window: Window) -> Option<u32> {+ self.window_property(window, self.atoms.net_wm_pid, 1)+ .first()+ .copied()+ .filter(|p| *p != 0)+ }++ fn transient_for(&self, window: Window) -> Option<Window> {+ self.window_property(window, AtomEnum::WM_TRANSIENT_FOR.into(), 1)+ .first()+ .copied()+ .filter(|w| *w != 0 && *w != self.root)+ }++ /// The window still exists and is mapped (a destroyed dialog fails both).+ fn window_viewable(&self, window: Window) -> bool {+ self.conn+ .get_window_attributes(window)+ .ok()+ .and_then(|c| c.reply().ok())+ .is_some_and(|a| a.map_state == MapState::VIEWABLE)+ }++ fn parent(&self, window: Window) -> Option<Window> {+ let tree = self.conn.query_tree(window).ok()?.reply().ok()?;+ (tree.parent != 0 && tree.parent != self.root).then_some(tree.parent)+ }++ /// The pid that owns `window`: its own `_NET_WM_PID`, else the one of+ /// the window it is transient for, else its ancestors' (the core focus+ /// often sits on a child of the client toplevel).+ fn owner_pid(&self, window: Window) -> Option<u32> {+ let mut current = window;+ for _ in 0..OWNER_WALK_LIMIT {+ if current == 0 || current == self.root {+ return None;+ }+ if let Some(pid) = self.window_pid(current) {+ return Some(pid);+ }+ match self.transient_for(current).or_else(|| self.parent(current)) {+ Some(next) if next != current => current = next,+ _ => return None,+ }+ }+ None+ }++ fn window_title(&self, window: Window) -> String {+ let read = |atom: u32, ty: u32| -> Option<String> {+ let reply = self+ .conn+ .get_property(false, window, atom, ty, 0, 256)+ .ok()?+ .reply()+ .ok()?;+ (!reply.value.is_empty()).then(|| String::from_utf8_lossy(&reply.value).into_owned())+ };+ read(self.atoms.net_wm_name, self.atoms.utf8_string)+ .or_else(|| read(AtomEnum::WM_NAME.into(), AtomEnum::STRING.into()))+ .unwrap_or_default()+ }++ /// Mapped override-redirect children of the root: menus, combo popups,+ /// tooltips. Requests are pipelined so a busy desktop costs one round trip.+ fn mapped_popups(&self) -> HashSet<Window> {+ let tree = match self.conn.query_tree(self.root).map(|c| c.reply()) {+ Ok(Ok(tree)) => tree,+ _ => return HashSet::new(),+ };+ let cookies: Vec<_> = tree+ .children+ .iter()+ .filter_map(|w| self.conn.get_window_attributes(*w).ok().map(|c| (*w, c)))+ .collect();+ cookies+ .into_iter()+ .filter_map(|(w, c)| c.reply().ok().map(|a| (w, a)))+ .filter(|(_, a)| a.override_redirect && a.map_state == MapState::VIEWABLE)+ .map(|(w, _)| w)+ .collect()+ }++ /// Current server time (PropertyNotify round trip), bounded at 300 ms;+ /// `CurrentTime` (0) would lose to focus-stealing prevention.+ fn server_time(&self) -> u32 {+ let Ok(probe) = self.conn.generate_id() else {+ return 0;+ };+ let aux = CreateWindowAux::new().event_mask(EventMask::PROPERTY_CHANGE);+ if self+ .conn+ .create_window(+ COPY_DEPTH_FROM_PARENT,+ probe,+ self.root,+ -1,+ -1,+ 1,+ 1,+ 0,+ WindowClass::INPUT_OUTPUT,+ 0,+ &aux,+ )+ .is_err()+ {+ return 0;+ }+ let _ = self.conn.change_property(+ PropMode::REPLACE,+ probe,+ self.atoms.net_wm_pid,+ AtomEnum::STRING,+ 8,+ 1,+ &[0u8],+ );+ let _ = self.conn.flush();+ let deadline = Instant::now() + Duration::from_millis(300);+ let mut time = 0;+ while Instant::now() < deadline {+ match self.conn.poll_for_event() {+ Ok(Some(Event::PropertyNotify(e))) if e.window == probe => {+ time = e.time;+ break;+ }+ Ok(Some(_)) => continue,+ Ok(None) => std::thread::sleep(Duration::from_millis(5)),+ Err(_) => break,+ }+ }+ let _ = self.conn.destroy_window(probe);+ let _ = self.conn.flush();+ time+ }++ fn activate(&self, target: Window, current: Window) {+ let event = ClientMessageEvent::new(+ 32,+ target,+ self.atoms.net_active_window,+ [2, self.server_time(), current, 0, 0],+ );+ let _ = self.conn.send_event(+ false,+ self.root,+ EventMask::SUBSTRUCTURE_REDIRECT | EventMask::SUBSTRUCTURE_NOTIFY,+ event,+ );+ let _ = self.conn.flush();+ }++ fn set_focus(&self, window: Window, revert_to: InputFocus) {+ // BadWindow / BadMatch (destroyed or unmapped meanwhile) are expected+ // and swallowed: the verification loop decides what to do next.+ if let Ok(cookie) = self+ .conn+ .set_input_focus(revert_to, window, x11rb::CURRENT_TIME)+ {+ let _ = cookie.check();+ }+ let _ = self.conn.flush();+ }+}++/// Where a focus move went, relative to the action's target application.+#[derive(Clone, Copy, Debug, PartialEq, Eq)]+pub enum FocusMove {+ /// The target app owned the focus before and still does: it opened one+ /// of its own windows (a dialog). Not an intrusion; leave it.+ SameAppDialog,+ /// The focus left the user's application (or arrived in the target from+ /// another app): restore it.+ OtherApp,+}++/// The same-app rule: a move is the app's own dialog only when the target+/// pid owned the focus before the action *and* owns it now. A move from a+/// decoy (another pid) into the target's dialog is a steal, as is a move+/// from the target to anything else.+pub fn classify_focus_move(+ target_pid: Option<u32>,+ previous_owner: Option<u32>,+ new_owner: Option<u32>,+) -> FocusMove {+ match (target_pid, previous_owner, new_owner) {+ (Some(target), Some(previous), Some(new)) if previous == target && new == target => {+ FocusMove::SameAppDialog+ }+ _ => FocusMove::OtherApp,+ }+}++/// What a background action must not change.+#[derive(Clone, Debug, PartialEq, Eq)]+pub struct FocusSnapshot {+ core_focus: Window,+ revert_to: InputFocus,+ active: Option<Window>,+ stacking_top: Option<Window>,+ popups: HashSet<Window>,+ clients: HashSet<Window>,+ /// Pid that owned the active window (else the core focus) before the+ /// action: the application the user was working in.+ previous_owner: Option<u32>,+}++/// A window of the target application that appeared or took the focus.+#[derive(Clone, Debug, Default, PartialEq, Eq)]+pub struct SameAppWindow {+ pub window: u64,+ pub title: String,+ /// It received the desktop focus (else it was merely mapped).+ pub focused: bool,+}++/// Outcome of the post-action check and restore.+#[derive(Clone, Debug, Default, PartialEq, Eq)]+pub struct FocusGuardReport {+ /// The core focus, active window or stacking top moved after the action.+ pub changed: bool,+ /// They were moved back (verified stable) before the tool returned.+ pub restored: bool,+ /// Human-readable list of what moved (`focus 0x..->0x..`, ...).+ pub changes: Vec<String>,+ /// A new override-redirect popup (menu, combo list) appeared, so its+ /// toolkit now holds a keyboard grab that cannot be released without+ /// closing it; the pid that owns it (falls back to the action's target).+ pub grab_held_by: Option<u32>,+ /// The target application opened (and possibly focused) one of its own+ /// windows; the guard left it alone.+ pub same_app_window: Option<SameAppWindow>,+ /// The window that held the focus before the action no longer exists+ /// (the action closed a dialog); there is nothing to restore to, the+ /// focus went to `same_app_window` / wherever the WM put it.+ pub closed_window: Option<u64>,+ /// Milliseconds spent watching and restoring after delivery.+ pub elapsed_ms: u64,+}++impl FocusGuardReport {+ /// `window_closed` / `same_app_dialog` / `restored` / `not_restored`+ /// once something moved.+ pub fn outcome(&self) -> Option<&'static str> {+ if !self.changed {+ return None;+ }+ Some(if self.closed_window.is_some() {+ "window_closed"+ } else if self.same_app_window.as_ref().is_some_and(|w| w.focused) {+ "same_app_dialog"+ } else if self.restored {+ "restored"+ } else if self.grab_held_by.is_some() {+ "grab_held"+ } else {+ "not_restored"+ })+ }++ pub fn to_json(&self) -> serde_json::Value {+ let mut v = serde_json::json!({+ "focus_changed": self.changed,+ "focus_restored": self.restored,+ });+ if let Some(outcome) = self.outcome() {+ v["focus_outcome"] = serde_json::json!(outcome);+ }+ if !self.changes.is_empty() {+ v["focus_changes"] = serde_json::json!(self.changes);+ }+ if let Some(pid) = self.grab_held_by {+ v["grab_held_by"] = serde_json::json!(pid);+ }+ if let Some(closed) = self.closed_window {+ v["focus_window_closed"] = serde_json::json!(closed);+ }+ if let Some(window) = &self.same_app_window {+ v["app_window_opened"] = serde_json::json!({+ "window_id": window.window,+ "title": window.title,+ "focused": window.focused,+ });+ }+ v+ }++ /// A `window_change` evidence item when the target application mapped or+ /// focused one of its own windows, or closed the one holding the focus:+ /// the public action contract keeps `evidence[]`, while the flat guard+ /// fields are reduced away.+ pub fn evidence_item(&self) -> Option<serde_json::Value> {+ if let Some(closed) = self.closed_window {+ return Some(serde_json::json!({+ "kind": "window_change",+ "detail": format!(+ "window {closed}, which held the focus, was closed by the action{}",+ self.same_app_window+ .as_ref()+ .map(|w| format!("; the focus moved to {} \"{}\"", w.window, w.title))+ .unwrap_or_default()+ ),+ }));+ }+ let window = self.same_app_window.as_ref()?;+ Some(serde_json::json!({+ "kind": "window_change",+ "detail": format!(+ "the application {} its own window {} \"{}\"",+ if window.focused { "opened and focused" } else { "opened" },+ window.window,+ window.title+ ),+ }))+ }++ /// One sentence for the tool's text content; empty when nothing moved.+ pub fn summary(&self) -> String {+ let mut s = String::new();+ match (&self.same_app_window, self.changed) {+ _ if self.closed_window.is_some() => s.push_str(&format!(+ " The window that held the focus ({}) was closed by this action \+ (focus_outcome=window_closed); the focus moved to {}.",+ self.closed_window.unwrap_or(0),+ self.same_app_window+ .as_ref()+ .map(|w| format!("{} \"{}\"", w.window, w.title))+ .unwrap_or_else(|| "another window".to_owned())+ )),+ (Some(window), _) if window.focused => s.push_str(&format!(+ " The application opened its own window {} \"{}\" and it now holds the \+ focus (focus_outcome=same_app_dialog, left in place); pid-only \+ type_text/press_key go to that window.",+ window.window, window.title+ )),+ (Some(window), false) => s.push_str(&format!(+ " The application opened its own window {} \"{}\".",+ window.window, window.title+ )),+ (_, true) if self.grab_held_by.is_some() && !self.restored => s.push_str(&format!(+ " The application moved the desktop focus ({}) and its open popup holds \+ the keyboard grab, so it was left in place (focus_outcome=grab_held).",+ self.changes.join(", ")+ )),+ (_, true) => s.push_str(&format!(+ " The application moved the desktop focus ({}); {}.",+ self.changes.join(", "),+ if self.restored {+ "it was restored to the previous window (focus_outcome=restored)"+ } else {+ "restoring it did not hold (focus_outcome=not_restored)"+ }+ )),+ _ => {}+ }+ if let Some(pid) = self.grab_held_by {+ s.push_str(&format!(+ " A popup menu is open, so pid {pid} holds a keyboard grab until it closes."+ ));+ }+ s+ }+}++impl FocusSnapshot {+ /// Record the current state. `None` when there is no usable X display+ /// (the guard then degrades to a no-op rather than failing the action).+ pub fn capture() -> Option<Self> {+ let x = X::open().ok()?;+ let (core_focus, revert_to) = x.core_focus();+ let active = x.active_window();+ let previous_owner = active+ .or_else(|| (core_focus > 1).then_some(core_focus))+ .and_then(|w| x.owner_pid(w));+ Some(Self {+ core_focus,+ revert_to,+ active,+ stacking_top: x.stacking_top(),+ popups: x.mapped_popups(),+ clients: x.client_list().into_iter().collect(),+ previous_owner,+ })+ }++ fn diff(&self, x: &X) -> Vec<String> {+ let mut changes = Vec::new();+ let (focus, _) = x.core_focus();+ // PointerRoot (1) / None (0) are transient states while the WM moves+ // focus; the settle loop sees the final value.+ if focus > 1 && focus != self.core_focus {+ changes.push(format!("focus 0x{:x}->0x{:x}", self.core_focus, focus));+ }+ let active = x.active_window();+ if active != self.active && active.is_some() {+ changes.push(format!(+ "active 0x{:x}->0x{:x}",+ self.active.unwrap_or(0),+ active.unwrap_or(0)+ ));+ }+ let top = x.stacking_top();+ if top != self.stacking_top && top.is_some() {+ changes.push(format!(+ "stacking_top 0x{:x}->0x{:x}",+ self.stacking_top.unwrap_or(0),+ top.unwrap_or(0)+ ));+ }+ changes+ }++ /// Managed toplevels that were not in the client list at capture time,+ /// topmost last.+ fn new_clients(&self, x: &X) -> Vec<Window> {+ x.client_list()+ .into_iter()+ .filter(|w| *w != 0 && !self.clients.contains(w))+ .collect()+ }++ /// The window the desktop focus sits on now (the WM's active window when+ /// it still exists — `_NET_ACTIVE_WINDOW` lags a destroyed dialog by a+ /// beat — else the core focus).+ fn current_focus_window(&self, x: &X) -> Option<Window> {+ x.active_window()+ .filter(|w| x.window_viewable(*w))+ .or_else(|| {+ let (focus, _) = x.core_focus();+ (focus > 1).then_some(focus)+ })+ }++ /// The window that held the focus at capture time (active, else core).+ fn previous_focus_window(&self) -> Option<Window> {+ self.active+ .or_else(|| (self.core_focus > 1).then_some(self.core_focus))+ }++ /// Watch briefly for a change, restore if one happened, and report.+ /// `target_pid` names the action's application for the grab attribution+ /// and the same-app rule.+ pub fn restore_if_changed(&self, target_pid: Option<u32>) -> FocusGuardReport {+ self.restore_if_changed_opts(target_pid, true)+ }++ /// [`Self::restore_if_changed`] with the settle watch optional: a body that+ /// already observed the desktop for longer than the watch (the MPX pointer+ /// route samples the screen ~250 ms after the release and has its own+ /// fast-path restore) checks once and returns unless a new top-level is+ /// still being mapped.+ pub fn restore_if_changed_opts(+ &self,+ target_pid: Option<u32>,+ settle_watch: bool,+ ) -> FocusGuardReport {+ let started = Instant::now();+ let Ok(x) = X::open() else {+ return FocusGuardReport::default();+ };+ let mut report = FocusGuardReport::default();+ // The target app owns the focus: whatever it maps next is its own.+ let target_owns_focus = target_pid.is_some() && self.previous_owner == target_pid;++ // Settle watch: stop at the first observed change. A new top-level+ // (dialog being mapped) extends the watch when the focus belongs to+ // another application, since the WM focuses it only once mapped and+ // that steal must be undone; when the target app already owns the+ // focus the new window is its own and the watch ends at once.+ let mut watch_until = if settle_watch {+ started + SETTLE_WATCH+ } else {+ started+ };+ let mut extended = false;+ let mut changes = self.diff(&x);+ let mut new_clients = self.new_clients(&x);+ let mut own_window = self.own_new_window(&x, target_pid, &new_clients);+ if !settle_watch && !new_clients.is_empty() && own_window.is_none() {+ extended = true;+ watch_until = started + SETTLE_WATCH_NEW_WINDOW;+ }+ while changes.is_empty() && own_window.is_none() && Instant::now() < watch_until {+ std::thread::sleep(SETTLE_POLL);+ changes = self.diff(&x);+ new_clients = self.new_clients(&x);+ own_window = self.own_new_window(&x, target_pid, &new_clients);+ if !extended && !new_clients.is_empty() && own_window.is_none() {+ extended = true;+ watch_until = started + SETTLE_WATCH_NEW_WINDOW;+ }+ }++ // Popups opened by the action: an open GTK/VCL menu grabs the keyboard.+ let new_popups: Vec<Window> = x+ .mapped_popups()+ .difference(&self.popups)+ .copied()+ .collect();+ if !new_popups.is_empty() {+ report.grab_held_by = new_popups+ .iter()+ .find_map(|w| x.window_pid(*w))+ .or(target_pid);+ }++ if changes.is_empty() {+ report.same_app_window = own_window;+ report.elapsed_ms = started.elapsed().as_millis() as u64;+ return report;+ }+ report.changed = true;+ report.changes = changes;+ // The target application mapped one of its own top-levels (a dialog+ // opened by a key or a menu item) while another application owned the+ // focus: the focus may still be restored below, but the new window is+ // the action's effect and is reported as such.+ if report.same_app_window.is_none() {+ report.same_app_window = self.any_new_window_of(&x, target_pid, &new_clients);+ }++ // The action closed the window that held the focus (Escape / OK on+ // a dialog): nothing to restore to, the WM already moved on.+ let focus_window = self.current_focus_window(&x);+ if let Some(previous) = self+ .previous_focus_window()+ .filter(|w| !x.window_viewable(*w))+ {+ report.closed_window = Some(u64::from(previous));+ report.same_app_window = focus_window.map(|w| SameAppWindow {+ window: u64::from(w),+ title: x.window_title(w),+ focused: true,+ });+ report.elapsed_ms = started.elapsed().as_millis() as u64;+ return report;+ }++ // Same-app rule: the focus stayed inside the target application+ // (it opened a dialog). Leave it and say so.+ let new_owner = focus_window.and_then(|w| x.owner_pid(w));+ if target_owns_focus+ && classify_focus_move(target_pid, self.previous_owner, new_owner)+ == FocusMove::SameAppDialog+ {+ let window = focus_window.unwrap_or(0);+ report.same_app_window = Some(SameAppWindow {+ window: u64::from(window),+ title: x.window_title(window),+ focused: true,+ });+ report.elapsed_ms = started.elapsed().as_millis() as u64;+ return report;+ }++ // A popup (menu, combo list) that the action opened holds a keyboard+ // grab: re-activating the previous window cannot take effect until it+ // closes, and closing it would undo the action. Report and return.+ if report.grab_held_by.is_some() {+ report.elapsed_ms = started.elapsed().as_millis() as u64;+ return report;+ }++ // Restore. EWMH re-activation makes the WM restore stacking and its+ // own focus bookkeeping; the explicit XSetInputFocus covers WMs that+ // treat _NET_ACTIVE_WINDOW as raise-only and the no-WM case.+ let mut stable = 0;+ let mut resent = 0;+ let deadline = started + RESTORE_BUDGET;+ self.reassert(&x);+ while Instant::now() < deadline {+ std::thread::sleep(RESTORE_POLL);+ let now = self.diff(&x);+ if now.is_empty() {+ stable += 1;+ if stable >= STABLE_POLLS {+ report.restored = true;+ break;+ }+ continue;+ }+ stable = 0;+ if resent < 2 {+ resent += 1;+ self.reassert(&x);+ }+ }+ if !report.restored {+ tracing::warn!(+ "background focus guard: could not restore {:?}",+ report.changes+ );+ }+ report.elapsed_ms = started.elapsed().as_millis() as u64;+ report+ }++ /// The topmost freshly mapped toplevel of the target pid, when the target+ /// already owned the focus (so the window is the app's own and the watch+ /// need not wait for the WM to focus it).+ fn own_new_window(+ &self,+ x: &X,+ target_pid: Option<u32>,+ new_clients: &[Window],+ ) -> Option<SameAppWindow> {+ if target_pid.is_none() || self.previous_owner != target_pid {+ return None;+ }+ let window = new_clients+ .iter()+ .rev()+ .copied()+ .find(|w| x.owner_pid(*w) == target_pid)?;+ Some(SameAppWindow {+ window: u64::from(window),+ title: x.window_title(window),+ focused: false,+ })+ }++ /// The topmost freshly mapped toplevel owned by the target pid, whoever+ /// held the focus before; `focused` says whether it holds it now.+ fn any_new_window_of(+ &self,+ x: &X,+ target_pid: Option<u32>,+ new_clients: &[Window],+ ) -> Option<SameAppWindow> {+ target_pid?;+ let window = new_clients+ .iter()+ .rev()+ .copied()+ .find(|w| x.owner_pid(*w) == target_pid)?;+ Some(SameAppWindow {+ window: u64::from(window),+ title: x.window_title(window),+ focused: self.current_focus_window(x) == Some(window),+ })+ }++ fn reassert(&self, x: &X) {+ if let Some(prev) = self.active {+ x.activate(prev, x.active_window().unwrap_or(0));+ }+ if self.core_focus > 1 {+ x.set_focus(self.core_focus, self.revert_to);+ }+ }+}++/// Run `body` (a background delivery) between a snapshot and a restore.+/// The body's error still propagates; the guard only adds its report.+pub fn guarded<T>(+ target_pid: Option<u32>,+ body: impl FnOnce() -> Result<T>,+) -> Result<(T, Option<FocusGuardReport>)> {+ let snapshot = FocusSnapshot::capture();+ let value = body()?;+ let report = snapshot.map(|s| s.restore_if_changed(target_pid));+ Ok((value, report))+}++/// [`guarded`] for a body that has already let the desktop settle (the MPX+/// pointer press train and drag gesture): one post-check, no extra settle+/// watch, so the pointer branch's fast path keeps its latency.+pub fn guarded_settled<T>(+ target_pid: Option<u32>,+ body: impl FnOnce() -> Result<T>,+) -> Result<(T, Option<FocusGuardReport>)> {+ let snapshot = FocusSnapshot::capture();+ let value = body()?;+ let report = snapshot.map(|s| s.restore_if_changed_opts(target_pid, false));+ Ok((value, report))+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn report_json_and_summary_name_what_moved() {+ let report = FocusGuardReport {+ changed: true,+ restored: true,+ changes: vec!["focus 0x1->0x2".into()],+ grab_held_by: Some(42),+ same_app_window: None,+ closed_window: None,+ elapsed_ms: 7,+ };+ let json = report.to_json();+ assert_eq!(json["focus_changed"], true);+ assert_eq!(json["focus_restored"], true);+ assert_eq!(json["focus_outcome"], "restored");+ assert_eq!(json["grab_held_by"], 42);+ assert_eq!(json["focus_changes"][0], "focus 0x1->0x2");+ assert!(report.evidence_item().is_none());+ let summary = report.summary();+ assert!(summary.contains("restored"), "{summary}");+ assert!(summary.contains("pid 42"), "{summary}");+ }++ #[test]+ fn quiet_report_is_silent() {+ let report = FocusGuardReport::default();+ assert_eq!(report.summary(), "");+ let json = report.to_json();+ assert_eq!(json["focus_changed"], false);+ assert!(json.get("focus_outcome").is_none());+ assert!(json.get("grab_held_by").is_none());+ assert!(json.get("focus_changes").is_none());+ }++ #[test]+ fn same_app_dialog_is_reported_not_restored() {+ let report = FocusGuardReport {+ changed: true,+ restored: false,+ changes: vec!["active 0x1->0x2".into()],+ grab_held_by: None,+ same_app_window: Some(SameAppWindow {+ window: 2,+ title: "Brightness-Contrast".into(),+ focused: true,+ }),+ closed_window: None,+ elapsed_ms: 3,+ };+ let json = report.to_json();+ assert_eq!(json["focus_outcome"], "same_app_dialog");+ assert_eq!(json["app_window_opened"]["window_id"], 2);+ assert_eq!(json["app_window_opened"]["focused"], true);+ let evidence = report.evidence_item().expect("window_change evidence");+ assert_eq!(evidence["kind"], "window_change");+ assert!(evidence["detail"]+ .as_str()+ .unwrap()+ .contains("Brightness-Contrast"));+ let summary = report.summary();+ assert!(summary.contains("same_app_dialog"), "{summary}");+ assert!(summary.contains("Brightness-Contrast"), "{summary}");+ assert!(!summary.contains("restored"), "{summary}");+ }++ #[test]+ fn mapped_but_unfocused_own_window_is_mentioned_without_a_focus_outcome() {+ let report = FocusGuardReport {+ same_app_window: Some(SameAppWindow {+ window: 9,+ title: "Format Cells".into(),+ focused: false,+ }),+ ..FocusGuardReport::default()+ };+ assert!(report.outcome().is_none());+ assert!(report.summary().contains("Format Cells"));+ assert!(report.evidence_item().is_some());+ }++ #[test]+ fn a_closed_focus_holder_is_window_closed_evidence_not_a_failed_restore() {+ let report = FocusGuardReport {+ changed: true,+ restored: false,+ changes: vec!["focus 0x2->0x1".into()],+ closed_window: Some(2),+ same_app_window: Some(SameAppWindow {+ window: 1,+ title: "GNU Image Manipulation Program".into(),+ focused: true,+ }),+ ..FocusGuardReport::default()+ };+ assert_eq!(report.outcome(), Some("window_closed"));+ assert_eq!(report.to_json()["focus_outcome"], "window_closed");+ assert_eq!(report.to_json()["focus_window_closed"], 2);+ let evidence = report.evidence_item().unwrap();+ assert!(evidence["detail"].as_str().unwrap().contains("closed"));+ assert!(report.summary().contains("window_closed"));+ assert!(!report.summary().contains("not_restored"));+ }++ #[test]+ fn same_app_rule_only_when_the_target_owned_the_focus_before_and_after() {+ // The target app was active and opened its own dialog: leave it.+ assert_eq!(+ classify_focus_move(Some(7), Some(7), Some(7)),+ FocusMove::SameAppDialog+ );+ // Decoy case: another app owned the focus and the target's dialog+ // stole it -> restore.+ assert_eq!(+ classify_focus_move(Some(7), Some(3), Some(7)),+ FocusMove::OtherApp+ );+ // The target was active and something else took the focus -> restore.+ assert_eq!(+ classify_focus_move(Some(7), Some(7), Some(3)),+ FocusMove::OtherApp+ );+ // Unknown owners never qualify.+ assert_eq!(+ classify_focus_move(Some(7), None, Some(7)),+ FocusMove::OtherApp+ );+ assert_eq!(+ classify_focus_move(Some(7), Some(7), None),+ FocusMove::OtherApp+ );+ assert_eq!(+ classify_focus_move(None, Some(7), Some(7)),+ FocusMove::OtherApp+ );+ }++ #[test]+ fn capture_without_display_degrades_to_none() {+ let prior = std::env::var_os("DISPLAY");+ std::env::set_var("DISPLAY", ":9999999");+ let snapshot = FocusSnapshot::capture();+ match prior {+ Some(v) => std::env::set_var("DISPLAY", v),+ None => std::env::remove_var("DISPLAY"),+ }+ assert!(snapshot.is_none());+ }+}++#[cfg(test)]+mod grab_tests {+ use super::*;++ #[test]+ fn a_popup_grab_is_reported_instead_of_a_failed_restore() {+ let report = FocusGuardReport {+ changed: true,+ restored: false,+ changes: vec!["active 0x1->0x2".into()],+ grab_held_by: Some(77),+ same_app_window: None,+ closed_window: None,+ elapsed_ms: 40,+ };+ assert_eq!(report.outcome(), Some("grab_held"));+ assert_eq!(report.to_json()["focus_outcome"], "grab_held");+ let summary = report.summary();+ assert!(summary.contains("focus_outcome=grab_held"), "{summary}");+ assert!(+ summary.contains("pid 77 holds a keyboard grab"),+ "{summary}"+ );+ }++ #[test]+ fn restore_and_settle_budgets_bound_the_guard_under_1500ms() {+ assert!(SETTLE_WATCH_NEW_WINDOW + RESTORE_BUDGET <= Duration::from_millis(1500));+ }+}diff --git a/libs/cua-driver/rust/crates/platform-linux/src/input/foreground.rs b/libs/cua-driver/rust/crates/platform-linux/src/input/foreground.rsnew file mode 100644index 0000000000..3762da696a--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-linux/src/input/foreground.rs@@ -0,0 +1,835 @@+//! X11 foreground transaction (`delivery_mode:"foreground"`).+//!+//! A foreground action behaves like a user at the keyboard: activate + raise the+//! target toplevel, wait until the window manager confirms that it is the+//! active window AND that the core X input focus sits inside it, then inject+//! real input (XTest) and report where focus ended up. The target is left+//! active afterwards — restoring the previously active window would pop down+//! any menu the click just opened and would move keyboard focus away from the+//! widget a following `type_text` needs.+//!+//! Everything here is bounded: the activation/confirmation phase and the+//! post-check run on watchdog threads with hard deadlines, so a wedged X+//! server, a client holding a server grab, or a window that never takes focus+//! yields a structured `foreground_timeout` / `foreground_unavailable` error+//! instead of a hung tool call. Only x11rb (a pure-Rust, thread-safe client)+//! is used: protocol errors come back as `Result`s rather than going through+//! Xlib's process-wide error handler.++use anyhow::{anyhow, Result};+use std::sync::mpsc;+use std::time::{Duration, Instant};+use x11rb::connection::Connection as _;+use x11rb::protocol::xproto::{+ AtomEnum, ClientMessageEvent, ConnectionExt as _, CreateWindowAux, EventMask, InputFocus,+ PropMode, Window, WindowClass, CLIENT_MESSAGE_EVENT,+};+use x11rb::protocol::Event;+use x11rb::rust_connection::RustConnection;+use x11rb::COPY_DEPTH_FROM_PARENT;++/// Error-code prefixes carried in the `anyhow` message (`<code>: <detail>`)+/// so tool handlers can attach a structured `code` without a bespoke error type.+pub const CODE_UNAVAILABLE: &str = "foreground_unavailable";+pub const CODE_TIMEOUT: &str = "foreground_timeout";++/// Tunables for one foreground transaction.+#[derive(Clone, Copy, Debug)]+pub struct ForegroundOptions {+ /// How long to wait for the WM to confirm activation + input focus.+ pub settle: Duration,+ /// Whether the body injects keyboard input. Keyboard delivery is routed by+ /// the X input focus, so the post-check is reported for it.+ pub keyboard: bool,+ /// The process the caller resolved the target window to. The post-check+ /// otherwise reads `_NET_WM_PID` off the window itself, which an+ /// override-redirect popup (a LibreOffice VCL menu) never carries — and+ /// which is gone by the post-check when the click closed it.+ pub target_pid: Option<u32>,+}++impl ForegroundOptions {+ /// Pointer actions land by stacking; a short confirmation suffices.+ pub fn pointer() -> Self {+ Self {+ settle: Duration::from_millis(800),+ keyboard: false,+ target_pid: None,+ }+ }++ /// Keyboard actions need the input focus, which GTK/VCL toplevels accept a+ /// beat after being raised; give them a real budget.+ pub fn keyboard() -> Self {+ Self {+ settle: Duration::from_millis(1500),+ keyboard: true,+ target_pid: None,+ }+ }++ /// Legacy `settle_ms` hint: folded into the pointer budget (never shorter).+ pub fn from_settle_hint(settle_ms: u64) -> Self {+ let mut opts = Self::pointer();+ opts.settle = opts.settle.max(Duration::from_millis(settle_ms));+ opts+ }++ /// Name the process the target window belongs to (see `target_pid`).+ pub fn for_pid(mut self, pid: u32) -> Self {+ self.target_pid = Some(pid);+ self+ }+}++/// Where the X input focus was after the body ran.+#[derive(Clone, Copy, Debug, PartialEq, Eq)]+pub enum FocusAfter {+ /// Inside the target toplevel.+ Target,+ /// Inside another toplevel owned by the same process (a dialog opened or+ /// closed, a new document window, ...).+ SamePid,+ /// Somewhere else entirely.+ Elsewhere,+ /// Could not be determined before the post-check deadline.+ Unknown,+}++impl FocusAfter {+ pub fn as_str(self) -> &'static str {+ match self {+ Self::Target => "target",+ Self::SamePid => "same_pid",+ Self::Elsewhere => "elsewhere",+ Self::Unknown => "unknown",+ }+ }+}++/// What the transaction observed; surfaced in tool structured content.+#[derive(Clone, Debug)]+pub struct ForegroundReport {+ /// The target was already active and focused; no activation was sent.+ pub already_active: bool,+ /// Activation had to be re-sent once before the WM confirmed it.+ pub retried_activation: bool,+ /// Milliseconds from start until activation + focus were confirmed.+ pub confirm_ms: u64,+ /// Focus location after the body.+ pub focus_after: FocusAfter,+ /// Toplevels of the target's process (managed windows and popup menus)+ /// that appeared or vanished between the activation and the post-check:+ /// a menu opened, a dialog mapped or closed. `None` when nothing changed+ /// or the observation was unavailable.+ pub window_change: Option<String>,+}++impl ForegroundReport {+ pub fn to_json(&self) -> serde_json::Value {+ let mut v = serde_json::json!({+ "activated": !self.already_active,+ "retried_activation": self.retried_activation,+ "confirm_ms": self.confirm_ms,+ "focus_after": self.focus_after.as_str(),+ });+ if let Some(change) = &self.window_change {+ v["window_change"] = serde_json::json!(change);+ }+ v+ }++ /// Focus is proven to have stayed inside the target process after the+ /// body ran (the target toplevel itself, or a dialog/popup it owns).+ pub fn focus_kept(&self) -> bool {+ matches!(self.focus_after, FocusAfter::Target | FocusAfter::SamePid)+ }+}++/// The on-screen toplevels of `pid` (managed windows plus override-redirect+/// popups), as `(window, description)` pairs, for a before/after diff.+fn pid_window_set(pid: Option<u32>) -> Vec<(u64, String)> {+ let mut set: Vec<(u64, String)> = crate::x11::list_windows(pid)+ .into_iter()+ .filter(|w| w.is_on_screen)+ .map(|w| {+ let title = if w.title.is_empty() {+ String::new()+ } else {+ format!(" \"{}\"", w.title)+ };+ (w.xid, format!("window {}{title}", w.xid))+ })+ .collect();+ set.extend(+ super::mapped_popup_windows()+ .into_iter()+ .filter(|p| pid.is_none() || p.pid.is_none() || p.pid == pid)+ .map(|p| (p.window, p.describe())),+ );+ set+}++/// Describe what changed between two window sets, or `None` when nothing did.+pub(crate) fn describe_window_change(+ before: &[(u64, String)],+ after: &[(u64, String)],+) -> Option<String> {+ let appeared: Vec<&str> = after+ .iter()+ .filter(|(id, _)| !before.iter().any(|(b, _)| b == id))+ .map(|(_, d)| d.as_str())+ .collect();+ let vanished: Vec<&str> = before+ .iter()+ .filter(|(id, _)| !after.iter().any(|(a, _)| a == id))+ .map(|(_, d)| d.as_str())+ .collect();+ if appeared.is_empty() && vanished.is_empty() {+ return None;+ }+ let mut parts = Vec::new();+ if !appeared.is_empty() {+ parts.push(format!("appeared: {}", appeared.join(", ")));+ }+ if !vanished.is_empty() {+ parts.push(format!("closed: {}", vanished.join(", ")));+ }+ Some(parts.join("; "))+}++/// How often the target process's window set is re-read after the body.+const WINDOW_CHANGE_POLL: Duration = Duration::from_millis(50);++/// How long a toolkit gets to map a menu / dialog after the body before the+/// window set is declared unchanged. GTK / VCL menus and dialogs map within+/// ~100 ms of the click; a Qt file dialog (VLC "Add...", "Convert / Save")+/// takes 300-600 ms, which a fixed 150 ms settle reported as no change.+const WINDOW_CHANGE_DEADLINE: Duration = Duration::from_millis(800);++/// After the first change is seen, a beat for the new window's title+/// (`_NET_WM_NAME` arrives a moment after the map) so the summary names it.+const WINDOW_CHANGE_TITLE_GRACE: Duration = Duration::from_millis(60);++/// How long the post-check retries an empty core focus (`None` /+/// `PointerRoot`): a popup destroyed by the click leaves the focus unset for+/// a beat before the toolkit re-focuses its toplevel.+const FOCUS_RETRY: Duration = Duration::from_millis(250);++/// Poll `pid`'s window set until it differs from `before` or the deadline+/// passes. Returns the last set read and the change description, if any.+fn wait_for_window_change(+ pid: Option<u32>,+ before: &[(u64, String)],+) -> (Vec<(u64, String)>, Option<String>) {+ let deadline = Instant::now() + WINDOW_CHANGE_DEADLINE;+ loop {+ let after = pid_window_set(pid);+ if let Some(change) = describe_window_change(before, &after) {+ std::thread::sleep(WINDOW_CHANGE_TITLE_GRACE);+ let settled = pid_window_set(pid);+ return match describe_window_change(before, &settled) {+ Some(change) => (settled, Some(change)),+ None => (after, Some(change)),+ };+ }+ if Instant::now() >= deadline {+ return (after, None);+ }+ std::thread::sleep(WINDOW_CHANGE_POLL);+ }+}++/// Extract the structured code prefix from a foreground error message.+pub fn error_code(error: &anyhow::Error) -> Option<&'static str> {+ let text = error.to_string();+ if text.starts_with(CODE_TIMEOUT) {+ Some(CODE_TIMEOUT)+ } else if text.starts_with(CODE_UNAVAILABLE) {+ Some(CODE_UNAVAILABLE)+ } else {+ None+ }+}++/// Run `f` on a detached thread and wait at most `deadline` for its result.+///+/// All current callers (`confirm_phase`, `wait_for_window_change`,+/// `post_check`) already bound themselves cooperatively with their own+/// internal deadline loop, so in the common case the worker returns well+/// inside `deadline` and this wrapper's timeout never fires. The residual+/// leak this guards against is a single blocking X11 socket call (connect,+/// or a request/reply round trip inside `X11::open`/`activate`) hanging past+/// `deadline` because the X server itself has stopped answering — a+/// synchronous X11 read cannot be interrupted or polled from the outside+/// without switching the whole call chain to non-blocking I/O, which is out+/// of scope here. In that rare case the thread is intentionally leaked: it+/// is bounded (it exits as soon as the stalled X call returns, or the+/// process exits), and its completion is harmless because `tx.send` below+/// ignores a disconnected receiver (`let _ =`) rather than panicking, so a+/// late result from a leaked thread is silently dropped instead of crashing+/// the caller that already moved on.+fn run_with_deadline<T: Send + 'static>(+ deadline: Duration,+ f: impl FnOnce() -> T + Send + 'static,+) -> Option<T> {+ let (tx, rx) = mpsc::channel();+ std::thread::Builder::new()+ .name("cua-x11-foreground".into())+ .spawn(move || {+ let _ = tx.send(f());+ })+ .ok()?;+ rx.recv_timeout(deadline).ok()+}++struct X11 {+ conn: RustConnection,+ root: Window,+ net_active_window: u32,+ net_wm_pid: u32,+}++impl X11 {+ fn open() -> Result<Self> {+ let (conn, screen) = RustConnection::connect(None).map_err(|e| {+ anyhow!("{CODE_UNAVAILABLE}: cannot open DISPLAY to verify X11 input focus: {e}")+ })?;+ let root = conn.setup().roots[screen].root;+ let net_active_window = conn+ .intern_atom(false, b"_NET_ACTIVE_WINDOW")?+ .reply()?+ .atom;+ let net_wm_pid = conn.intern_atom(false, b"_NET_WM_PID")?.reply()?.atom;+ Ok(Self {+ conn,+ root,+ net_active_window,+ net_wm_pid,+ })+ }++ fn active_window(&self) -> Option<Window> {+ let reply = self+ .conn+ .get_property(+ false,+ self.root,+ self.net_active_window,+ AtomEnum::WINDOW,+ 0,+ 1,+ )+ .ok()?+ .reply()+ .ok()?;+ let value = reply.value32()?.next();+ value.filter(|w| *w != 0)+ }++ fn window_pid(&self, window: Window) -> Option<u32> {+ let reply = self+ .conn+ .get_property(false, window, self.net_wm_pid, AtomEnum::CARDINAL, 0, 1)+ .ok()?+ .reply()+ .ok()?;+ let value = reply.value32()?.next();+ value.filter(|p| *p != 0)+ }++ /// Current server time via the PropertyNotify round-trip. `_NET_ACTIVE_WINDOW`+ /// stamped `CurrentTime` (0) loses to focus-stealing prevention whenever any+ /// newer user input exists. Bounded: falls back to 0 after 300 ms.+ fn server_time(&self) -> u32 {+ let probe = match self.conn.generate_id() {+ Ok(id) => id,+ Err(_) => return 0,+ };+ let aux = CreateWindowAux::new().event_mask(EventMask::PROPERTY_CHANGE);+ if self+ .conn+ .create_window(+ COPY_DEPTH_FROM_PARENT,+ probe,+ self.root,+ -1,+ -1,+ 1,+ 1,+ 0,+ WindowClass::INPUT_OUTPUT,+ 0,+ &aux,+ )+ .is_err()+ {+ return 0;+ }+ let _ = x11rb::protocol::xproto::change_property(+ &self.conn,+ PropMode::REPLACE,+ probe,+ self.net_wm_pid,+ AtomEnum::STRING,+ 8,+ 1,+ &[0u8],+ );+ let _ = self.conn.flush();+ let deadline = Instant::now() + Duration::from_millis(300);+ let mut time = 0;+ while Instant::now() < deadline {+ match self.conn.poll_for_event() {+ Ok(Some(Event::PropertyNotify(e))) if e.window == probe => {+ time = e.time;+ break;+ }+ Ok(Some(_)) => continue,+ Ok(None) => std::thread::sleep(Duration::from_millis(5)),+ Err(_) => break,+ }+ }+ let _ = self.conn.destroy_window(probe);+ let _ = self.conn.flush();+ time+ }++ fn activate(&self, target: Window, prior: Window) -> Result<()> {+ let time = self.server_time();+ let event =+ ClientMessageEvent::new(32, target, self.net_active_window, [2, time, prior, 0, 0]);+ debug_assert_eq!(event.response_type, CLIENT_MESSAGE_EVENT);+ self.conn.send_event(+ false,+ self.root,+ EventMask::SUBSTRUCTURE_REDIRECT | EventMask::SUBSTRUCTURE_NOTIFY,+ event,+ )?;+ // Set the core focus explicitly as well (what `xdotool windowactivate`+ // does): WMs with focus-stealing prevention honour `_NET_ACTIVE_WINDOW`+ // as raise-only. BadMatch on a not-yet-viewable window is expected and+ // ignored; the confirmation loop decides.+ if let Ok(cookie) =+ self.conn+ .set_input_focus(InputFocus::PARENT, target, x11rb::CURRENT_TIME)+ {+ let _ = cookie.check();+ }+ self.conn.flush()?;+ Ok(())+ }++ fn focused(&self) -> Option<Window> {+ self.conn+ .get_input_focus()+ .ok()?+ .reply()+ .ok()+ .map(|r| r.focus)+ .filter(|f| *f > 1) // 0 = None, 1 = PointerRoot+ }++ /// Whether `focused` is `target` or one of its descendants.+ fn is_within(&self, mut focused: Window, target: Window) -> bool {+ for _ in 0..64 {+ if focused == target {+ return true;+ }+ if focused == 0 || focused == self.root {+ return false;+ }+ let Ok(cookie) = self.conn.query_tree(focused) else {+ return false;+ };+ let Ok(reply) = cookie.reply() else {+ return false;+ };+ if reply.parent == 0 || reply.parent == focused {+ return false;+ }+ focused = reply.parent;+ }+ false+ }++ /// Walk up from `window` to the first ancestor carrying `_NET_WM_PID`.+ fn owning_pid(&self, window: Window) -> Option<u32> {+ self.ancestors(window)+ .find_map(|window| self.window_pid(window))+ }++ /// `window` and its ancestors below the root (bounded).+ fn ancestors(&self, window: Window) -> impl Iterator<Item = Window> + '_ {+ std::iter::successors(Some(window), move |&window| {+ let parent = self.conn.query_tree(window).ok()?.reply().ok()?.parent;+ (parent != 0 && parent != self.root && parent != window).then_some(parent)+ })+ .take(64)+ }++ /// First `WM_CLASS` class (else instance) name on `window` or one of its+ /// ancestors.+ fn wm_class(&self, window: Window) -> Option<String> {+ self.ancestors(window).find_map(|window| {+ let (instance, class) = crate::x11::get_window_class(&self.conn, window)?;+ Some(if class.is_empty() { instance } else { class })+ })+ }++ /// Who holds the X input focus, for a refusal that has to explain why the+ /// target could not take it.+ fn focus_holder(&self) -> String {+ let focus = self+ .conn+ .get_input_focus()+ .ok()+ .and_then(|cookie| cookie.reply().ok())+ .map(|reply| reply.focus);+ let Some(focus) = focus else {+ return "unknown".into();+ };+ let pid = (focus > 1).then(|| self.owning_pid(focus)).flatten();+ let comm = pid.and_then(|pid| {+ std::fs::read_to_string(format!("/proc/{pid}/comm"))+ .ok()+ .map(|comm| comm.trim().to_owned())+ });+ let class = (focus > 1).then(|| self.wm_class(focus)).flatten();+ describe_focus_holder(focus, class.as_deref(), pid, comm.as_deref())+ }++ fn focus_is_within(&self, target: Window) -> bool {+ match self.focused() {+ Some(f) => self.is_within(f, target),+ None => false,+ }+ }+}++struct ConfirmOutcome {+ confirmed: bool,+ already_active: bool,+ retried: bool,+ elapsed: Duration,+ active_after: Window,+ focus_within: bool,+ /// Set when the activation was not confirmed: see [`X11::focus_holder`].+ focus_holder: Option<String>,+}++fn confirm_phase(target: Window, settle: Duration) -> Result<ConfirmOutcome> {+ let x = X11::open()?;+ let start = Instant::now();+ let prior = x.active_window().unwrap_or(0);+ let mut retried = false;+ // Already active and focused: leave the window (and any open menu / grab)+ // untouched — re-activating is what popped GTK menus back down.+ if prior == target && x.focus_is_within(target) {+ return Ok(ConfirmOutcome {+ confirmed: true,+ already_active: true,+ retried: false,+ elapsed: start.elapsed(),+ active_after: target,+ focus_within: true,+ focus_holder: None,+ });+ }+ x.activate(target, prior)?;+ let deadline = start + settle;+ let retry_at = start + settle.mul_f32(0.4);+ let mut retry_pending = true;+ loop {+ let active = x.active_window() == Some(target);+ let focus_within = x.focus_is_within(target);+ if active && focus_within {+ return Ok(ConfirmOutcome {+ confirmed: true,+ already_active: false,+ retried,+ elapsed: start.elapsed(),+ active_after: target,+ focus_within: true,+ focus_holder: None,+ });+ }+ let now = Instant::now();+ if now >= deadline {+ return Ok(ConfirmOutcome {+ confirmed: false,+ already_active: false,+ retried,+ elapsed: start.elapsed(),+ active_after: x.active_window().unwrap_or(0),+ focus_within,+ focus_holder: Some(x.focus_holder()),+ });+ }+ if retry_pending && now >= retry_at {+ retry_pending = false;+ retried = true;+ x.activate(target, x.active_window().unwrap_or(0))?;+ }+ std::thread::sleep(Duration::from_millis(15));+ }+}++/// Where the focus sits after the body. `pid_windows` is the target+/// process's window set read after the body (managed windows and popups):+/// focus inside any of them is `SamePid` even when neither the target nor+/// the focused window carries `_NET_WM_PID` (a VCL popup menu closed by the+/// click, the popup that the click opened).+fn post_check(+ target: Window,+ target_pid: Option<u32>,+ pid_windows: &[(u64, String)],+) -> FocusAfter {+ let Ok(x) = X11::open() else {+ return FocusAfter::Unknown;+ };+ let retry_until = Instant::now() + FOCUS_RETRY;+ let focused = loop {+ if let Some(focused) = x.focused() {+ break focused;+ }+ if Instant::now() >= retry_until {+ return FocusAfter::Elsewhere;+ }+ std::thread::sleep(Duration::from_millis(25));+ };+ if x.is_within(focused, target) {+ return FocusAfter::Target;+ }+ if pid_windows+ .iter()+ .any(|(window, _)| x.is_within(focused, *window as Window))+ {+ return FocusAfter::SamePid;+ }+ // The body may have closed the target itself (Escape on a dialog, alt+F4,+ // a popup item click): its pid was read (or named by the caller) before+ // the body, so the comparison still works once the window is gone.+ match (+ target_pid.or_else(|| x.owning_pid(target)),+ x.owning_pid(focused),+ ) {+ (Some(a), Some(b)) if a == b => FocusAfter::SamePid,+ _ => FocusAfter::Elsewhere,+ }+}++/// Activate `xid`, confirm it holds the active window + input focus within+/// `opts.settle`, run `body`, then (for keyboard actions) report where focus+/// ended up. Returns `foreground_unavailable` (no input sent) when the WM never+/// confirms, and `foreground_timeout` when the X server stops answering.+pub fn with_x11_foreground_opts<T>(+ xid: u64,+ opts: ForegroundOptions,+ body: impl FnOnce() -> Result<T>,+) -> Result<(T, ForegroundReport)> {+ let target = xid as Window;+ let phase_budget = opts.settle + Duration::from_millis(1500);+ let outcome = run_with_deadline(phase_budget, move || confirm_phase(target, opts.settle))+ .ok_or_else(|| {+ anyhow!(+ "{CODE_TIMEOUT}: the X server did not answer the activation of window \+ 0x{xid:x} within {phase_budget:?}; no input was sent"+ )+ })??;+ if !outcome.confirmed {+ let holder = outcome.focus_holder.as_deref().unwrap_or("unknown");+ let cause = if outcome.active_after == 0 {+ "No window is active at all: the window manager has not set \+ _NET_ACTIVE_WINDOW (it may be restarting, or showing a modal surface of \+ its own that no window list includes)"+ } else {+ "The window may be minimized, on another workspace, or blocked by a modal \+ dialog — bring_to_front it or target that dialog instead"+ };+ return Err(anyhow!(+ "{CODE_UNAVAILABLE}: window 0x{xid:x} did not become the active, focused \+ window within {:?} (active=0x{:x}, focus_within_target={}, \+ input focus: {holder}); no input was sent. {cause}",+ opts.settle,+ outcome.active_after,+ outcome.focus_within+ ));+ }+ let target_pid = crate::x11::window_pid(xid).or(opts.target_pid);+ let windows_before = pid_window_set(target_pid);+ let value = body()?;+ // The post-check is what the tool reports as evidence for both keyboard+ // and pointer bodies: a real click that opened a menu or a dialog moves+ // the window set / focus within the target's process, and one that+ // landed elsewhere moves focus out of it. The window set is polled+ // rather than read after a fixed settle: it returns on the first change.+ let (windows_after, window_change) = run_with_deadline(+ WINDOW_CHANGE_DEADLINE + Duration::from_millis(1500),+ move || wait_for_window_change(target_pid, &windows_before),+ )+ .unwrap_or((Vec::new(), None));+ let focus_after = run_with_deadline(Duration::from_millis(1500), move || {+ post_check(target, target_pid, &windows_after)+ })+ .unwrap_or(FocusAfter::Unknown);+ Ok((+ value,+ ForegroundReport {+ already_active: outcome.already_active,+ retried_activation: outcome.retried,+ confirm_ms: outcome.elapsed.as_millis() as u64,+ focus_after,+ window_change,+ },+ ))+}++/// `0x…` plus what owns it: the class and process of a client window, or a+/// note that the focus sits on a window no client claims (the window+/// manager's own, or the root).+fn describe_focus_holder(+ focus: Window,+ class: Option<&str>,+ pid: Option<u32>,+ comm: Option<&str>,+) -> String {+ match focus {+ 0 => return "none".into(),+ 1 => return "PointerRoot".into(),+ _ => {}+ }+ let owner = match (pid, comm) {+ (Some(pid), Some(comm)) => format!("pid {pid} {comm}"),+ (Some(pid), None) => format!("pid {pid}"),+ _ => "no _NET_WM_PID".into(),+ };+ match class {+ Some(class) => format!("0x{focus:x} ({class}, {owner})"),+ None => format!("0x{focus:x} (no WM_CLASS, {owner})"),+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn focus_holder_names_the_owner() {+ assert_eq!(describe_focus_holder(0, None, None, None), "none");+ assert_eq!(describe_focus_holder(1, None, None, None), "PointerRoot");+ assert_eq!(+ describe_focus_holder(+ 0x520000a,+ Some("Gnome-terminal"),+ Some(1865),+ Some("gnome-terminal-")+ ),+ "0x520000a (Gnome-terminal, pid 1865 gnome-terminal-)"+ );+ assert_eq!(+ describe_focus_holder(0x800003, None, None, None),+ "0x800003 (no WM_CLASS, no _NET_WM_PID)"+ );+ }++ #[test]+ fn error_code_recognises_prefixes() {+ assert_eq!(+ error_code(&anyhow!("{CODE_TIMEOUT}: x")),+ Some(CODE_TIMEOUT)+ );+ assert_eq!(+ error_code(&anyhow!("{CODE_UNAVAILABLE}: y")),+ Some(CODE_UNAVAILABLE)+ );+ assert_eq!(error_code(&anyhow!("other")), None);+ }++ #[test]+ fn deadline_returns_none_when_work_hangs() {+ let started = Instant::now();+ let result = run_with_deadline(Duration::from_millis(50), || {+ std::thread::sleep(Duration::from_secs(5));+ 1+ });+ assert!(result.is_none());+ assert!(started.elapsed() < Duration::from_secs(2));+ }++ #[test]+ fn deadline_returns_value_when_work_finishes() {+ assert_eq!(run_with_deadline(Duration::from_secs(2), || 7), Some(7));+ }++ #[test]+ fn keyboard_budget_exceeds_pointer_budget() {+ assert!(ForegroundOptions::keyboard().settle > ForegroundOptions::pointer().settle);+ assert!(ForegroundOptions::from_settle_hint(80).settle >= Duration::from_millis(400));+ assert_eq!(+ ForegroundOptions::from_settle_hint(5000).settle,+ Duration::from_millis(5000)+ );+ }++ #[test]+ fn options_carry_the_caller_resolved_pid() {+ assert_eq!(ForegroundOptions::pointer().target_pid, None);+ assert_eq!(+ ForegroundOptions::keyboard().for_pid(42).target_pid,+ Some(42)+ );+ assert!(!ForegroundOptions::from_settle_hint(80).for_pid(1).keyboard);+ }++ #[test]+ fn report_json_shape() {+ let report = ForegroundReport {+ already_active: true,+ retried_activation: false,+ confirm_ms: 3,+ focus_after: FocusAfter::SamePid,+ window_change: Some("appeared: popup window 7 (200x300 at 1,2)".into()),+ };+ let json = report.to_json();+ assert_eq!(json["activated"], false);+ assert_eq!(json["focus_after"], "same_pid");+ assert_eq!(json["confirm_ms"], 3);+ assert_eq!(+ json["window_change"],+ "appeared: popup window 7 (200x300 at 1,2)"+ );+ assert!(report.focus_kept());+ }++ #[test]+ fn window_change_diff_names_appeared_and_closed() {+ let before = vec![+ (1u64, "window 1 \"GIMP\"".to_string()),+ (2, "window 2".to_string()),+ ];+ let after = vec![+ (1u64, "window 1 \"GIMP\"".to_string()),+ (9, "popup window 9".to_string()),+ ];+ assert_eq!(+ describe_window_change(&before, &after).as_deref(),+ Some("appeared: popup window 9; closed: window 2")+ );+ assert_eq!(describe_window_change(&before, &before), None);+ }++ #[test]+ fn unavailable_display_is_structured() {+ // Force a failing connect regardless of the host environment.+ let prior = std::env::var_os("DISPLAY");+ std::env::set_var("DISPLAY", ":9999999");+ let result = with_x11_foreground_opts(0x1234, ForegroundOptions::pointer(), || Ok(()));+ match prior {+ Some(v) => std::env::set_var("DISPLAY", v),+ None => std::env::remove_var("DISPLAY"),+ }+ let error = result.err().expect("connect must fail");+ assert!(error_code(&error).is_some(), "{error}");+ }+}diff --git a/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rsindex 6d24eb40f9..180ebdba4b 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs@@ -14,8 +14,18 @@/// Shared `delivery_mode` contract (background|foreground) — mirrors macOS/// `tools::DeliveryMode` and Windows `input::delivery`.pub mod delivery;+pub mod focus_guard;+pub mod foreground;+mod mpx_keyboard;mod mpx_owner;+pub use focus_guard::{FocusGuardReport, FocusSnapshot, SameAppWindow};+pub use foreground::{with_x11_foreground_opts, FocusAfter, ForegroundOptions, ForegroundReport};+pub use mpx_keyboard::{+ real_keyboard_input_available, send_virtual_keyboard_key, send_virtual_keyboard_text,+ KeyboardDeliveryReport, MPX_UINPUT_PATH,+};+use anyhow::{anyhow, bail, Context, Result};use evdev::uinput::VirtualDevice;use evdev::{AttributeSet, EventType, InputEvent, Key, RelativeAxisType};@@ -116,26 +126,49 @@ fn point_on_path(path: &[(i32, i32)], cum: &[f64], total: f64, t: f64) -> (i32,(x.round() as i32, y.round() as i32)}+/// One session's XI2 master pair (XIAddMaster always creates a pointer AND a+/// keyboard) plus the uinput slaves attached to it. The keyboard slave is+/// created lazily by `mpx_keyboard::ensure_master_keyboard`, so a pointer-only+/// click never pays for a second device hotplug.#[derive(Clone, Copy, Debug)]struct MasterPointerIds {pointer_id: i32,- _keyboard_id: i32,+ keyboard_id: i32,_slave_pointer_id: i32,+ slave_keyboard_id: Option<i32>,}static MPX_POINTERS: OnceLock<Mutex<HashMap<String, MasterPointerIds>>> = OnceLock::new();static UINPUT_POINTERS: OnceLock<Mutex<HashMap<String, Arc<Mutex<VirtualDevice>>>>> =OnceLock::new();static XLIB_THREADS_READY: OnceLock<Result<(), String>> = OnceLock::new();+/// Serialises every MPX operation against the idle reaper (and each other),+/// so a retained master pair is never torn down while a call is using it.+static MPX_OP_LOCK: Mutex<()> = Mutex::new(());+static MPX_LAST_USE: OnceLock<Mutex<HashMap<String, std::time::Instant>>> = OnceLock::new();+static MPX_IDLE_REAPER: std::sync::Once = std::sync::Once::new();+/// A session's retained master pair is removed after this much inactivity;+/// `end_session` and the startup reaper cover the explicit and crash cases.+/// `XIRemoveMaster` churns the XInput hierarchy, and mutter 42.x segfaults when+/// it processes that churn (the same reason per-call teardown crashes LibreOffice+/// VCL), so the pair must outlive the inter-action idle of a live session; a+/// long terminal-only stretch leaves the pointer unused far past a few minutes.+/// The window is only a leak backstop for a client that dies without+/// `end_session`; the startup reaper reclaims anything it misses.+const MPX_IDLE_TTL: Duration = Duration::from_secs(1800);+const MPX_IDLE_REAPER_PERIOD: Duration = Duration::from_secs(60);static MPX_NAME_COUNTER: AtomicU64 = AtomicU64::new(1);// evdev 0.12.2 asserts `name.len() + 1 < UINPUT_MAX_NAME_SIZE` while building// a device. Linux defines UINPUT_MAX_NAME_SIZE as 80, leaving 78 usable bytes.const EVDEV_UINPUT_NAME_MAX_BYTES: usize = 78;const UINPUT_POINTER_SUFFIX: &str = " uinput pointer";pub const UINPUT_UNAVAILABLE_CODE: &str = "uinput_unavailable";+/// Result `path` for pointer actions delivered as real button events from the+/// session's MPX virtual master pointer.+pub const MPX_POINTER_PATH: &str = "mpx_pointer";#[derive(Debug, thiserror::Error)]-#[error("Linux uinput pointer unavailable: {reason}")]+#[error("Linux uinput device unavailable: {reason}")]struct UinputUnavailable {reason: String,}@@ -144,6 +177,56 @@ fn mpx_pointers() -> &'static Mutex<HashMap<String, MasterPointerIds>> {MPX_POINTERS.get_or_init(|| Mutex::new(HashMap::new()))}+fn mpx_last_use() -> &'static Mutex<HashMap<String, std::time::Instant>> {+ MPX_LAST_USE.get_or_init(|| Mutex::new(HashMap::new()))+}++/// Take the MPX operation lock for one call on `cursor_id`, stamp its last+/// use, and make sure the idle reaper is running. Hold the guard for the+/// whole operation.+fn mpx_op_guard(cursor_id: &str) -> std::sync::MutexGuard<'static, ()> {+ let guard = MPX_OP_LOCK+ .lock()+ .unwrap_or_else(|poisoned| poisoned.into_inner());+ mpx_last_use()+ .lock()+ .unwrap()+ .insert(cursor_id.to_owned(), std::time::Instant::now());+ MPX_IDLE_REAPER.call_once(|| {+ std::thread::Builder::new()+ .name("cua-mpx-idle-reaper".into())+ .spawn(|| loop {+ sleep(MPX_IDLE_REAPER_PERIOD);+ let _op = MPX_OP_LOCK+ .lock()+ .unwrap_or_else(|poisoned| poisoned.into_inner());+ let stale = {+ let last_use = mpx_last_use().lock().unwrap();+ stale_cursor_ids(&last_use, std::time::Instant::now(), MPX_IDLE_TTL)+ };+ for cursor_id in stale {+ tracing::info!(cursor_id, "removing idle MPX master pair");+ forget_master_pointer(&cursor_id);+ }+ })+ .ok();+ });+ guard+}++/// Sessions whose last MPX use is older than `ttl`.+fn stale_cursor_ids(+ last_use: &HashMap<String, std::time::Instant>,+ now: std::time::Instant,+ ttl: Duration,+) -> Vec<String> {+ last_use+ .iter()+ .filter(|(_, at)| now.saturating_duration_since(**at) >= ttl)+ .map(|(id, _)| id.clone())+ .collect()+}+fn uinput_pointers() -> &'static Mutex<HashMap<String, Arc<Mutex<VirtualDevice>>>> {UINPUT_POINTERS.get_or_init(|| Mutex::new(HashMap::new()))}@@ -469,7 +552,7 @@ fn kde_x11_uinput_hotplug_is_unsafe_from_env() -> bool {)}-fn uinput_accessible() -> bool {+pub(crate) fn uinput_accessible() -> bool {fs::OpenOptions::new().read(true).write(true)@@ -586,15 +669,31 @@ fn ensure_master_pointer_for_session(let keyboard_id = keyboard_id.ok_or_else(|| anyhow!("failed to locate created master keyboard for '{cursor_id}'"))?;- let slave_pointer_id = wait_for_slave_pointer_id(display, &device_name)?;+ let slave_pointer_id = match wait_for_slave_id(+ display,+ &device_name,+ x11::xinput2::XISlavePointer,+ Duration::from_secs(5),+ ) {+ Ok(id) => id,+ Err(error) => {+ // Never leave the fresh master pair behind when the slave never+ // hotplugs (no udev/libinput on this server): a stray master with+ // nothing attached is exactly the leak the startup reaper exists for.+ let _ = remove_master_pointer(display, pointer_id);+ unsafe { x11::xlib::XCloseDisplay(display) };+ return Err(error);+ }+ };attach_slave_to_master(display, slave_pointer_id, pointer_id)?;set_flat_pointer_accel(display, slave_pointer_id);unsafe { x11::xlib::XCloseDisplay(display) };let ids = MasterPointerIds {pointer_id,- _keyboard_id: keyboard_id,+ keyboard_id,_slave_pointer_id: slave_pointer_id,+ slave_keyboard_id: None,};mpx_pointers().lock()@@ -608,7 +707,12 @@ fn ensure_master_pointer_for_session(}pub fn forget_master_pointer(cursor_id: &str) {+ mpx_last_use().lock().unwrap().remove(cursor_id);+ // Drop the uinput slaves first: closing the fds unplugs them, so the master+ // removal below never has to hand a live slave back to the user's core+ // devices (XIAttachToMaster only re-homes slaves that still exist).uinput_pointers().lock().unwrap().remove(cursor_id);+ mpx_keyboard::forget_uinput_keyboard(cursor_id);let Some(ids) = mpx_pointers().lock().unwrap().remove(cursor_id) else {return;};@@ -674,7 +778,10 @@ pub(crate) fn reap_orphaned_master_pointers() {unsafe {x11::xlib::XGrabServer(display);}- let result = (|| -> Result<()> {+ // A panic while the server is grabbed would freeze every other X client+ // (the whole desktop) until this process dies. Catch it so the ungrab+ // below always runs, and report it like any other incomplete recovery.+ let result = catch_unwind(AssertUnwindSafe(|| -> Result<()> {for (id, use_, name) in xi2_query_devices(display)? {if use_ != x11::xinput2::XIMasterPointer {continue;@@ -688,14 +795,19 @@ pub(crate) fn reap_orphaned_master_pointers() {}}Ok(())- })();+ }));unsafe {x11::xlib::XUngrabServer(display);x11::xlib::XSync(display, 0);x11::xlib::XCloseDisplay(display);}- if let Err(error) = result {- tracing::warn!("MPX orphan recovery incomplete: {error}");+ match result {+ Ok(Ok(())) => {}+ Ok(Err(error)) => tracing::warn!("MPX orphan recovery incomplete: {error}"),+ Err(payload) => tracing::warn!(+ "MPX orphan recovery panicked (server grab released): {}",+ panic_payload_message(payload.as_ref())+ ),}}@@ -724,16 +836,25 @@ fn create_uinput_pointer(name: &str) -> Result<VirtualDevice> {})}-fn wait_for_slave_pointer_id(display: *mut x11::xlib::Display, device_name: &str) -> Result<i32> {- let deadline = std::time::Instant::now() + Duration::from_secs(5);+/// Poll `XIQueryDevice` until the X server has hot-added the uinput device+/// named `device_name` as a slave of kind `use_` (`XISlavePointer` /+/// `XISlaveKeyboard`). udev + xf86-input-libinput do the hotplug on a real+/// Xorg; Xvfb/Xtigervnc never will, which is what the timeout covers.+fn wait_for_slave_id(+ display: *mut x11::xlib::Display,+ device_name: &str,+ use_: i32,+ timeout: Duration,+) -> Result<i32> {+ let deadline = std::time::Instant::now() + timeout;loop {- for (device_id, use_, seen_name) in xi2_query_devices(display)? {- if use_ == x11::xinput2::XISlavePointer && seen_name == device_name {+ for (device_id, seen_use, seen_name) in xi2_query_devices(display)? {+ if seen_use == use_ && seen_name == device_name {return Ok(device_id);}}if std::time::Instant::now() >= deadline {- bail!("timed out waiting for X input slave pointer '{device_name}'");+ bail!("timed out waiting for X input slave device '{device_name}'");}sleep(Duration::from_millis(50));}@@ -847,10 +968,79 @@ fn warp_master_pointer(Ok(())}-/// XIAnyModifier (1u32 << 31). The x11 crate doesn't export it.-const XI_ANY_MODIFIER: std::os::raw::c_int = 0x8000_0000u32 as std::os::raw::c_int;+/// True when `window` is override-redirect — a menu, tooltip, or other popup+/// the WM does not manage. Such a window takes its own active pointer grab, so+/// the shield-grab-and-replay dance cannot deliver into it; a plain warp+press+/// on the virtual master is what reaches it (and there is no WM focus to steal).+fn is_override_redirect(display: *mut x11::xlib::Display, window: x11::xlib::Window) -> bool {+ let mut attrs: x11::xlib::XWindowAttributes = unsafe { std::mem::zeroed() };+ let previous_handler = unsafe { x11::xlib::XSetErrorHandler(Some(ignore_x_error)) };+ let rc = unsafe { x11::xlib::XGetWindowAttributes(display, window, &mut attrs) };+ unsafe { x11::xlib::XSetErrorHandler(previous_handler) };+ rc != 0 && attrs.override_redirect != 0+}++/// Direct child of the root window under screen point `(x, y)`: the WM frame+/// of a managed toplevel, or an override-redirect popup (menu, tooltip).+fn root_child_under_point(+ display: *mut x11::xlib::Display,+ x: i32,+ y: i32,+) -> Option<x11::xlib::Window> {+ let root = unsafe { x11::xlib::XDefaultRootWindow(display) };+ let mut child: x11::xlib::Window = 0;+ let mut dx = 0;+ let mut dy = 0;+ let rc = unsafe {+ x11::xlib::XTranslateCoordinates(display, root, root, x, y, &mut dx, &mut dy, &mut child)+ };+ (rc != 0 && child != 0).then_some(child)+}++/// The root child (WM frame or the window itself) that contains `window`.+fn root_child_of(+ display: *mut x11::xlib::Display,+ window: x11::xlib::Window,+) -> Option<x11::xlib::Window> {+ let root = unsafe { x11::xlib::XDefaultRootWindow(display) };+ let previous_handler = unsafe { x11::xlib::XSetErrorHandler(Some(ignore_x_error)) };+ let mut current = window;+ let mut result = None;+ for _ in 0..64 {+ let mut root_ret: x11::xlib::Window = 0;+ let mut parent: x11::xlib::Window = 0;+ let mut children: *mut x11::xlib::Window = ptr::null_mut();+ let mut count: std::os::raw::c_uint = 0;+ let rc = unsafe {+ x11::xlib::XQueryTree(+ display,+ current,+ &mut root_ret,+ &mut parent,+ &mut children,+ &mut count,+ )+ };+ if !children.is_null() {+ unsafe { x11::xlib::XFree(children as *mut _) };+ }+ if rc == 0 || parent == 0 {+ break;+ }+ if parent == root {+ result = Some(current);+ break;+ }+ current = parent;+ }+ unsafe {+ x11::xlib::XSync(display, 0);+ x11::xlib::XSetErrorHandler(previous_handler);+ }+ result+}-fn xi_mask_len() -> usize {+pub(super) fn xi_mask_len() -> usize {(x11::xinput2::XI_LASTEVENT as usize >> 3) + 1}@@ -874,128 +1064,130 @@ fn xinput_opcode(display: *mut x11::xlib::Display) -> Option<std::os::raw::c_int}}-/// Install a device-specific XI2 synchronous passive button grab on `window`-/// for `device_id`. This shields the drag: the grab is newer than (and thus-/// checked before) the window manager's click-to-focus grab on the same-/// window, and being device-specific it does not conflict with the WM's-/// core/all-master grabs. The matching press freezes the device and is-/// delivered to us; replaying it (XIReplayDevice) re-checks grabs only-/// *below* this window and then delivers the event normally to the app, so-/// the WM never sees the press and never steals focus.-fn install_shield_grab(+/// Modifier evdev codes the virtual keyboard may hold mid-chord.+const MODIFIER_EVDEV_CODES: [u16; 8] = [42, 54, 29, 97, 56, 100, 125, 126];++/// Buttons (1..=3) currently held on the master pointer `pointer_id`, plus the+/// paired master keyboard's modifier state, via `XIQueryPointer`.+fn virtual_master_input_state(display: *mut x11::xlib::Display,- device_id: i32,- window: x11::xlib::Window,- button: u8,-) -> Result<()> {- let mut mask_bits = vec![0u8; xi_mask_len()];- x11::xinput2::XISetMask(&mut mask_bits, x11::xinput2::XI_ButtonPress);- let mut evmask = x11::xinput2::XIEventMask {- deviceid: device_id,- mask_len: mask_bits.len() as std::os::raw::c_int,- mask: mask_bits.as_mut_ptr(),- };- let mut mods = x11::xinput2::XIGrabModifiers {- modifiers: XI_ANY_MODIFIER,- status: 0,- };+ pointer_id: i32,+) -> (Vec<u8>, x11::xinput2::XIModifierState) {+ let root = unsafe { x11::xlib::XDefaultRootWindow(display) };+ let mut root_ret = 0;+ let mut child_ret = 0;+ let (mut rx, mut ry, mut wx, mut wy) = (0f64, 0f64, 0f64, 0f64);+ let mut buttons = x11::xinput2::XIButtonState::default();+ let mut mods = x11::xinput2::XIModifierState::default();+ let mut group = x11::xinput2::XIModifierState::default();+ let prev = unsafe { x11::xlib::XSetErrorHandler(Some(ignore_x_error)) };let rc = unsafe {- x11::xinput2::XIGrabButton(+ x11::xinput2::XIQueryPointer(display,- device_id,- button as std::os::raw::c_int,- window,- 0, // cursor: None- x11::xinput2::XIGrabModeSync, // freeze the pointer on press- x11::xinput2::XIGrabModeAsync, // leave the paired keyboard alone- x11::xlib::False, // owner_events: deliver to us- &mut evmask,- 1,+ pointer_id,+ root,+ &mut root_ret,+ &mut child_ret,+ &mut rx,+ &mut ry,+ &mut wx,+ &mut wy,+ &mut buttons,&mut mods,+ &mut group,)};- unsafe { x11::xlib::XSync(display, 0) };- if rc != 0 {- bail!("XIGrabButton(shield) failed with status {rc}");+ unsafe { x11::xlib::XSetErrorHandler(prev) };+ let mut held = Vec::new();+ if rc != 0 && !buttons.mask.is_null() && buttons.mask_len > 0 {+ let mask = unsafe { std::slice::from_raw_parts(buttons.mask, buttons.mask_len as usize) };+ for button in 1u8..=3 {+ let byte = (button / 8) as usize;+ if mask.get(byte).is_some_and(|b| b & (1 << (button % 8)) != 0) {+ held.push(button);+ }+ }+ unsafe { x11::xlib::XFree(buttons.mask as *mut _) };}- Ok(())+ (held, mods)}-fn remove_shield_grab(+/// Release anything the session's virtual master still holds from an earlier+/// aborted action: pointer buttons (a shield replay that timed out returned+/// before the release) and keyboard modifiers (a chord interrupted mid-way).+///+/// A held button is not cosmetic: every key event from the paired virtual+/// keyboard then carries `Button1Mask` (bit 8) in its core `state`, which+/// at-spi2 forwards verbatim and Orca reads as its `ORCA_MODIFIER_MASK`+/// (`1 << 8`) — a plain space becomes "Orca+space" and opens the Screen+/// Reader Preferences. Returns what was released, for the tool's report.+fn release_stuck_virtual_input(+ cursor_id: &str,display: *mut x11::xlib::Display,- device_id: i32,- window: x11::xlib::Window,- button: u8,-) {- let mut mods = x11::xinput2::XIGrabModifiers {- modifiers: XI_ANY_MODIFIER,- status: 0,- };+ ids: MasterPointerIds,+ keyboard: &Mutex<VirtualDevice>,+) -> Vec<String> {+ let mut released = Vec::new();+ let (held, mods) = virtual_master_input_state(display, ids.pointer_id);+ if !held.is_empty() {+ if let Some(pointer) = uinput_pointers().lock().unwrap().get(cursor_id).cloned() {+ let mut pointer = pointer.lock().unwrap();+ for &button in &held {+ if emit_button(&mut pointer, button, false).is_ok() {+ released.push(format!("button{button}"));+ }+ }+ }+ }+ if mods.base != 0 {+ let mut keyboard = keyboard.lock().unwrap();+ let mut any = false;+ for code in MODIFIER_EVDEV_CODES {+ any |= keyboard+ .emit(&[InputEvent::new(EventType::KEY, code, 0)])+ .is_ok();+ }+ if any {+ released.push(format!("modifiers(base=0x{:x})", mods.base));+ }+ }+ if mods.locked != 0 || mods.latched != 0 {+ // A locked/latched modifier on the virtual master (CapsLock leaked+ // from an interrupted chord) would shift every later character.+ unsafe {+ let prev = x11::xlib::XSetErrorHandler(Some(ignore_x_error));+ x11::xlib::XkbLockModifiers(display, ids.keyboard_id as u32, 0xff, 0);+ x11::xlib::XkbLatchModifiers(display, ids.keyboard_id as u32, 0xff, 0);+ x11::xlib::XSetErrorHandler(prev);+ }+ released.push(format!(+ "locked/latched modifiers (0x{:x}/0x{:x})",+ mods.locked, mods.latched+ ));+ }+ if !released.is_empty() {+ unsafe { x11::xlib::XSync(display, 0) };+ tracing::warn!(cursor_id, ?released, "released stuck virtual input state");+ sleep(Duration::from_millis(20));+ }+ released+}++/// Release a device frozen by a synchronous grab; harmless when not frozen.+fn thaw_device(display: *mut x11::xlib::Display, device_id: i32) {unsafe {let prev = x11::xlib::XSetErrorHandler(Some(ignore_x_error));- x11::xinput2::XIUngrabButton(+ x11::xinput2::XIAllowEvents(display,device_id,- button as std::os::raw::c_int,- window,- 1,- &mut mods,+ x11::xinput2::XIAsyncDevice,+ x11::xlib::CurrentTime,);x11::xlib::XSync(display, 0);x11::xlib::XSetErrorHandler(prev);}}-/// Drain the frozen shield presses for `pending_devices` and replay each so-/// it continues to the application. Returns the set of device ids we failed-/// to see within the timeout (their drags still proceed; the focus-restore-/// safety net covers any leak).-fn replay_shielded_presses(- display: *mut x11::xlib::Display,- xi_opcode: std::os::raw::c_int,- pending_devices: &mut std::collections::HashSet<i32>,- timeout: Duration,-) {- let deadline = std::time::Instant::now() + timeout;- while !pending_devices.is_empty() && std::time::Instant::now() < deadline {- // Only block on XNextEvent when something is queued, so a missing- // press can't hang us past the deadline.- if unsafe { x11::xlib::XPending(display) } == 0 {- sleep(Duration::from_millis(2));- continue;- }- let mut ev: x11::xlib::XEvent = unsafe { std::mem::zeroed() };- unsafe { x11::xlib::XNextEvent(display, &mut ev) };- if unsafe { ev.type_ } != x11::xlib::GenericEvent {- continue;- }- let mut cookie = unsafe { ev.generic_event_cookie };- if cookie.extension != xi_opcode || cookie.evtype != x11::xinput2::XI_ButtonPress {- continue;- }- if unsafe { x11::xlib::XGetEventData(display, &mut cookie) } == 0 {- continue;- }- let de = cookie.data as *const x11::xinput2::XIDeviceEvent;- if !de.is_null() {- let device_id = unsafe { (*de).deviceid };- let time = unsafe { (*de).time };- if pending_devices.remove(&device_id) {- unsafe {- x11::xinput2::XIAllowEvents(- display,- device_id,- x11::xinput2::XIReplayDevice,- time,- );- x11::xlib::XSync(display, 0);- }- }- }- unsafe { x11::xlib::XFreeEventData(display, &mut cookie) };- }-}-fn ewmh_active_window(display: *mut x11::xlib::Display) -> Option<x11::xlib::Window> {unsafe {let atom = x11::xlib::XInternAtom(display, c"_NET_ACTIVE_WINDOW".as_ptr(), x11::xlib::True);@@ -1101,144 +1293,19 @@ fn ewmh_activate_window(}}-/// Foreground rung for X11 (`delivery_mode:"foreground"`): briefly activate-/// `xid` via EWMH `_NET_ACTIVE_WINDOW`, run `body` (which injects the input-/// while the window holds focus), then restore the prior active window. The-/// Linux analogue of macOS `with_foreground_assist` / the Windows foreground-/// swap, reusing the existing [`ewmh_active_window`] / [`ewmh_activate_window`]-/// primitives (proper `x_server_time` stamping beats the WM's focus-stealing-/// prevention).-///-/// The transition is confirmed from both EWMH active-window state and the X11-/// core input-focus tree before `body` runs. A fixed delay or a successful-/// `XSetInputFocus` return is not evidence that global XTest input is safe.-/// `settle_ms` is retained as a compatibility hint and folded into the bounded-/// confirmation timeout; it is no longer an unconditional sleep.+/// Foreground rung for X11 (`delivery_mode:"foreground"`): activate `xid`+/// (EWMH `_NET_ACTIVE_WINDOW` + core input focus), confirm the transition+/// against both the WM's active window and the X input-focus tree, then run+/// `body` (which injects the input while the window holds focus). The target+/// is left active afterwards; see [`foreground`] for the rationale and the+/// deadline/watchdog behaviour. `settle_ms` is a minimum confirmation budget.pub fn with_x11_foreground<T>(xid: u64,settle_ms: u64,body: impl FnOnce() -> Result<T>,) -> Result<T> {- let display = unsafe { x11::xlib::XOpenDisplay(ptr::null()) };- if display.is_null() {- bail!("foreground_unavailable: cannot open DISPLAY to verify exact X11 input focus");- }- let prior = ewmh_active_window(display);- let mut prior_core_focus: x11::xlib::Window = 0;- let mut prior_revert = 0;- unsafe {- x11::xlib::XGetInputFocus(display, &mut prior_core_focus, &mut prior_revert);- }- ewmh_activate_window(display, xid as x11::xlib::Window, prior.unwrap_or(0));- unsafe {- x11::xlib::XSync(display, 0);- }- // EWMH `_NET_ACTIVE_WINDOW` is honored as *raise-only* by WMs with- // focus-stealing prevention (e.g. KWin): the window reaches the top of the- // stack — enough for a coordinate click, which lands by stacking — but the X- // *input focus* never transfers, so XTest key events (which the server routes- // to the focused window) go to whatever was focused before. Set the input- // focus explicitly too, exactly as `xdotool windowactivate` does, so the- // foreground key/hotkey rung actually reaches the target. Guarded against- // BadMatch on a not-yet-viewable window — Xlib's default handler would exit- // the process — reusing the scoped `ignore_x_error` pattern used elsewhere.- unsafe {- let prev_handler = x11::xlib::XSetErrorHandler(Some(ignore_x_error));- x11::xlib::XSetInputFocus(- display,- xid as x11::xlib::Window,- x11::xlib::RevertToParent,- x11::xlib::CurrentTime,- );- x11::xlib::XSync(display, 0);- x11::xlib::XSetErrorHandler(prev_handler);- }- let timeout = std::time::Duration::from_millis(settle_ms.max(400));- let deadline = std::time::Instant::now() + timeout;- let target = xid as x11::xlib::Window;- let focused = loop {- let active = ewmh_active_window(display) == Some(target);- if active && x11_focus_is_within(display, target) {- break true;- }- if std::time::Instant::now() >= deadline {- break false;- }- std::thread::sleep(std::time::Duration::from_millis(10));- };- let result = if focused {- body()- } else {- let active = ewmh_active_window(display).unwrap_or(0);- Err(anyhow::anyhow!(- "foreground_unavailable: X11 did not confirm active window and input focus within \- exact target 0x{xid:x} before the {:?} deadline (active=0x{active:x}); no input was sent",- timeout- ))- };-- // Restore both the EWMH active toplevel and the exact prior core focus.- if let Some(p) = prior {- ewmh_activate_window(display, p, xid as x11::xlib::Window);- }- if prior_core_focus != 0 {- unsafe {- let previous_handler = x11::xlib::XSetErrorHandler(Some(ignore_x_error));- x11::xlib::XSetInputFocus(- display,- prior_core_focus,- prior_revert,- x11::xlib::CurrentTime,- );- x11::xlib::XSync(display, 0);- x11::xlib::XSetErrorHandler(previous_handler);- }- }- unsafe {- x11::xlib::XCloseDisplay(display);- }- result-}--fn x11_focus_is_within(display: *mut x11::xlib::Display, target: x11::xlib::Window) -> bool {- let mut focused: x11::xlib::Window = 0;- let mut revert_to = 0;- unsafe {- x11::xlib::XGetInputFocus(display, &mut focused, &mut revert_to);- }- if focused == target {- return true;- }- let root = unsafe { x11::xlib::XDefaultRootWindow(display) };- while focused != 0 && focused != root {- let mut query_root = 0;- let mut parent = 0;- let mut children: *mut x11::xlib::Window = ptr::null_mut();- let mut child_count = 0;- let status = unsafe {- x11::xlib::XQueryTree(- display,- focused,- &mut query_root,- &mut parent,- &mut children,- &mut child_count,- )- };- if !children.is_null() {- unsafe {- x11::xlib::XFree(children.cast());- }- }- if status == 0 || parent == 0 || parent == focused {- return false;- }- if parent == target {- return true;- }- focused = parent;- }- false+ with_x11_foreground_opts(xid, ForegroundOptions::from_settle_hint(settle_ms), body)+ .map(|(value, _report)| value)}/// Activate `xid` and LEAVE it active (no restore) — the persistent foreground@@ -1246,6 +1313,16 @@ fn x11_focus_is_within(display: *mut x11::xlib::Display, target: x11::xlib::Wind/// the caller can report/inspect it. Best-effort; returns `None` prior on a/// headless display.pub fn x11_activate_window_persistent(xid: u64) -> Result<Option<u64>> {+ // Called directly from the tool layer (bring_to_front / foreground+ // activation), not through `send_virtual_pointer_*`/`deliver`, so without+ // this guard its `XSetErrorHandler` swap below can race theirs: whichever+ // thread restores its saved "previous" handler last can clobber the+ // other's still-in-flight `ignore_x_error` installation, briefly leaving+ // the process default handler active (which can abort the process on an+ // unrelated async X11 error) — or vice versa. `MPX_OP_LOCK` is a leaf+ // lock here (this function calls nothing that itself takes it), so no+ // reentrancy risk.+ let _op = mpx_op_guard("x11_activate_window_persistent");let display = unsafe { x11::xlib::XOpenDisplay(ptr::null()) };if display.is_null() {bail!(@@ -1297,6 +1374,17 @@ fn emit_button(device: &mut VirtualDevice, button: u8, press: bool) -> Result<()Ok(())}+/// Emit a release for `button` regardless of state; the kernel drops a+/// release for a button that is not held, so this is safe to call on every+/// exit path of a press train or gesture.+fn release_button_best_effort(device: &Arc<Mutex<VirtualDevice>>, button: u8) {+ if let Ok(mut device) = device.lock() {+ if let Err(error) = emit_button(&mut device, button, false) {+ tracing::warn!("virtual pointer button {button} release failed: {error:#}");+ }+ }+}+fn emit_relative_motion(device: &mut VirtualDevice, dx: i32, dy: i32) -> Result<()> {let mut events = Vec::with_capacity(2);if dx != 0 {@@ -1334,9 +1422,20 @@ fn emit_scroll(device: &mut VirtualDevice, horizontal: bool, value: i32) -> Resu}pub fn send_parallel_virtual_pointer_drags(drags: &[(String, VirtualPointerDrag)]) -> Result<()> {+ let _op = mpx_op_guard(+ drags+ .first()+ .map(|(id, _)| id.as_str())+ .unwrap_or("default"),+ );+ {+ let mut last_use = mpx_last_use().lock().unwrap();+ for (cursor_id, _) in drags {+ last_use.insert(cursor_id.clone(), std::time::Instant::now());+ }+ }let display = open_display()?;supports_parallel_pointer_injection(display)?;- let xi_opcode = xinput_opcode(display);struct ActiveDrag {cursor_id: String,@@ -1356,207 +1455,1116 @@ pub fn send_parallel_virtual_pointer_drags(drags: &[(String, VirtualPointerDrag)let start_at = std::time::Instant::now() + Duration::from_millis(120);let mut active = Vec::with_capacity(drags.len());- // Click-to-focus WMs grab buttons for XIAllMasterDevices, so the drag's- // press activates the target window exactly like a user click would.- // Remember the focus state and hand it back afterwards so parallel- // drags don't steal it.- let saved_focus = save_focus_state(display);+ // A click-to-focus WM that grabs buttons for every master device would+ // activate the target on the press. Remember the focus state and hand it+ // back afterwards so parallel drags don't steal it (mutter grabs for the+ // Virtual Core Pointer only, so there the restore is a no-op).+ let saved_focus = save_focus_state(display);++ let result = (|| -> Result<()> {+ for (cursor_id, drag) in drags {+ let ids = ensure_master_pointer(cursor_id)?;+ let device = uinput_pointers()+ .lock()+ .unwrap()+ .get(cursor_id)+ .cloned()+ .ok_or_else(|| anyhow!("missing uinput pointer for '{cursor_id}'"))?;+ let (cum, total) = path_cumulative(&drag.path);+ let start = *drag.path.first().unwrap_or(&(0, 0));+ active.push(ActiveDrag {+ cursor_id: cursor_id.clone(),+ ids,+ device,+ drag: drag.clone(),+ cum,+ total,+ steps: drag.steps.max(1),+ step_delay: if drag.steps.max(1) > 1 {+ Duration::from_millis(drag.duration_ms / drag.steps.max(1) as u64)+ } else {+ Duration::from_millis(drag.duration_ms)+ },+ current_step: 0,+ next_at: start_at,+ last_x: start.0,+ last_y: start.1,+ });+ }++ let now = std::time::Instant::now();+ if start_at > now {+ std::thread::sleep(start_at - now);+ }++ // Press each drag straight from its virtual master (see+ // `send_virtual_pointer_click` for why there is no XI2 shield grab:+ // on this server the grab-and-replay swallowed the press). The press+ // point must not be covered by another toplevel.+ for item in &active {+ let start = *item.drag.path.first().unwrap_or(&(0, 0));+ if let PointCover::Occluded(occluded) = occluding_window(+ display,+ item.drag.target_window as x11::xlib::Window,+ start.0,+ start.1,+ )? {+ return Err(occluded.into());+ }+ thaw_device(display, item.ids.pointer_id);+ warp_master_pointer(display, item.ids, start.0, start.1)?;+ {+ let mut device = item.device.lock().unwrap();+ emit_button(&mut device, item.drag.button, true)?;+ }+ }+ sleep(Duration::from_millis(40));++ while active.iter().any(|item| item.current_step < item.steps) {+ let now = std::time::Instant::now();+ let mut advanced = false;+ let mut next_deadline = None;++ for item in &mut active {+ if item.current_step >= item.steps {+ continue;+ }+ if now >= item.next_at {+ item.current_step += 1;+ let t = item.current_step as f64 / item.steps as f64;+ let (ix, iy) = point_on_path(&item.drag.path, &item.cum, item.total, t);+ let dx = ix - item.last_x;+ let dy = iy - item.last_y;+ if dx != 0 || dy != 0 {+ let mut device = item.device.lock().unwrap();+ emit_relative_motion(&mut device, dx, dy)?;+ // Keep the agent cursor overlay tracking the drag so+ // the gesture is visible, not just its endpoints.+ crate::overlay::send_command_for(+ item.cursor_id.clone(),+ cursor_overlay::OverlayCommand::SnapTo {+ x: ix as f64,+ y: iy as f64,+ heading_radians: Some((dy as f64).atan2(dx as f64)),+ },+ );+ }+ item.last_x = ix;+ item.last_y = iy;+ item.next_at = now + item.step_delay;+ advanced = true;+ }+ if item.current_step < item.steps {+ next_deadline = Some(match next_deadline {+ Some(deadline) => std::cmp::min(deadline, item.next_at),+ None => item.next_at,+ });+ }+ }++ if !advanced {+ if let Some(deadline) = next_deadline {+ let now = std::time::Instant::now();+ if deadline > now {+ std::thread::sleep(deadline - now);+ }+ }+ }+ }++ for item in &active {+ let mut device = item.device.lock().unwrap();+ emit_button(&mut device, item.drag.button, false)?;+ }+ Ok(())+ })();+ // Whatever happened above, no virtual master may keep a button held+ // (a stuck Button1 poisons the core modifier state for later chords).+ for (cursor_id, drag) in drags {+ if let Some(device) = uinput_pointers().lock().unwrap().get(cursor_id).cloned() {+ release_button_best_effort(&device, drag.button);+ }+ }+ // The per-session master pair is retained for reuse (torn down on+ // end_session / idle / startup reap). Creating and destroying an XI2 master+ // plus hot-plugging a uinput slave on every call churns the XInput+ // hierarchy hard enough to crash fragile toolkits (LibreOffice VCL); one+ // long-lived pair per session avoids that and is cheaper. The focus is+ // still saved and restored around each gesture.+ let _ = drags;+ restore_focus_state(display, &saved_focus);+ unsafe {+ x11::xlib::XCloseDisplay(display);+ }+ result+}++/// A discrete no-focus-steal pointer click driven through the same real-input+/// pipeline as [`send_parallel_virtual_pointer_drags`] — MPX master pointer ++/// uinput slave — reduced to a press/release (or a short press/release train+/// for `count` > 1) at one screen point.+///+/// This is what lands **right / middle / double** clicks (and any left click+/// the AT-SPI path can't actuate) on XInput2 toolkits: GTK3/4, VCL, Qt and+/// Chromium silently drop synthetic `XSendEvent` pointer events, so those+/// clicks are otherwise no-ops. Coordinates are screen-absolute;+/// `target_window` is the X11 toplevel the caller means to hit. The press is+/// delivered straight from the virtual master: no XI2 shield grab. On GNOME+/// (mutter) the click-to-focus passive grab is installed for the Virtual Core+/// Pointer only, so a second master's press never activates the window; the+/// grab-and-replay "shield" that was meant to hide the press from the WM+/// instead swallowed it on this server (the replayed press never reached the+/// application) and left the device frozen. Focus is still saved and restored+/// around the click for WMs that do grab every master device. `button` is an+/// X button number (1=left, 2=middle, 3=right); `count` >= 1 (2 = double-click).+#[derive(Clone, Debug)]+pub struct VirtualPointerClick {+ pub target_window: u64,+ pub x: i32,+ pub y: i32,+ pub button: u8,+ pub count: usize,+}++/// The screen point the caller aimed at is covered by another toplevel, so a+/// real pointer press there would land on the covering window, not the+/// target. Refused before any input is sent.+#[derive(Debug, Clone)]+pub struct TargetOccluded {+ pub target_window: u64,+ pub covering_window: u64,+ pub covering_title: String,+ pub covering_pid: Option<u32>,+ pub x: i32,+ pub y: i32,+}++impl std::fmt::Display for TargetOccluded {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ write!(+ f,+ "screen point ({}, {}) over window {} is covered by window {}{}{}; a real pointer \+ press there would land on the covering window",+ self.x,+ self.y,+ self.target_window,+ self.covering_window,+ if self.covering_title.is_empty() {+ String::new()+ } else {+ format!(" \"{}\"", self.covering_title)+ },+ self.covering_pid+ .map(|pid| format!(" (pid {pid})"))+ .unwrap_or_default(),+ )+ }+}+impl std::error::Error for TargetOccluded {}++/// A mapped override-redirect toplevel: a popup menu, popover, combo list+/// or tooltip. The WM does not manage it, so it is absent from+/// `list_windows`; the pointer tools name it so a caller can walk it with+/// `get_window_state(pid, window_id=<window>)`.+#[derive(Clone, Debug, Default, PartialEq, Eq)]+pub struct PopupWindow {+ pub window: u64,+ pub x: i32,+ pub y: i32,+ pub width: u32,+ pub height: u32,+ pub title: String,+ pub pid: Option<u32>,+}++impl PopupWindow {+ pub fn to_json(&self) -> serde_json::Value {+ serde_json::json!({+ "window_id": self.window,+ "bounds": { "x": self.x, "y": self.y, "width": self.width, "height": self.height },+ "title": self.title,+ "pid": self.pid,+ })+ }++ /// `popup window 12345678 "Edit" (220x340 at 410,220)`.+ pub fn describe(&self) -> String {+ format!(+ "popup window {}{} ({}x{} at {},{})",+ self.window,+ if self.title.is_empty() {+ String::new()+ } else {+ format!(" \"{}\"", self.title)+ },+ self.width,+ self.height,+ self.x,+ self.y+ )+ }+}++/// The caller's window-local point does not lie inside the window: a+/// coordinate-frame mistake (screen pixels passed as window pixels, or the+/// wrong window_id), refused before any input is sent.+#[derive(Debug, Clone, PartialEq, Eq)]+pub struct PointOutsideWindow {+ pub target_window: u64,+ pub screen_x: i32,+ pub screen_y: i32,+ pub window_x: i32,+ pub window_y: i32,+ /// Screen-space `(x, y, width, height)` of the window.+ pub bounds: (i32, i32, u32, u32),+}++impl std::fmt::Display for PointOutsideWindow {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ let (bx, by, bw, bh) = self.bounds;+ write!(+ f,+ "window-local point ({}, {}) of window {} resolves to screen ({}, {}), outside \+ the window's bounds (x={bx}, y={by}, {bw}x{bh}); no input was sent",+ self.window_x, self.window_y, self.target_window, self.screen_x, self.screen_y+ )+ }+}+impl std::error::Error for PointOutsideWindow {}++/// Screen-space `(x, y, width, height)` of `window`, or `None` when it is+/// gone.+fn window_screen_bounds(+ display: *mut x11::xlib::Display,+ window: x11::xlib::Window,+) -> Option<(i32, i32, u32, u32)> {+ let previous_handler = unsafe { x11::xlib::XSetErrorHandler(Some(ignore_x_error)) };+ let mut root: x11::xlib::Window = 0;+ let (mut gx, mut gy) = (0i32, 0i32);+ let (mut width, mut height, mut border, mut depth) = (0u32, 0u32, 0u32, 0u32);+ let rc = unsafe {+ x11::xlib::XGetGeometry(+ display,+ window,+ &mut root,+ &mut gx,+ &mut gy,+ &mut width,+ &mut height,+ &mut border,+ &mut depth,+ )+ };+ let mut result = None;+ if rc != 0 && root != 0 {+ let (mut dx, mut dy) = (0i32, 0i32);+ let mut child: x11::xlib::Window = 0;+ let ok = unsafe {+ x11::xlib::XTranslateCoordinates(+ display, window, root, 0, 0, &mut dx, &mut dy, &mut child,+ )+ };+ if ok != 0 {+ result = Some((dx, dy, width, height));+ }+ }+ unsafe {+ x11::xlib::XSync(display, 0);+ x11::xlib::XSetErrorHandler(previous_handler);+ }+ result+}++fn bounds_contain(bounds: (i32, i32, u32, u32), x: i32, y: i32) -> bool {+ let (bx, by, bw, bh) = bounds;+ x >= bx && y >= by && x < bx.saturating_add(bw as i32) && y < by.saturating_add(bh as i32)+}++/// Describe a mapped override-redirect root child as a [`PopupWindow`].+fn popup_info(display: *mut x11::xlib::Display, child: x11::xlib::Window) -> Option<PopupWindow> {+ let mut attrs: x11::xlib::XWindowAttributes = unsafe { std::mem::zeroed() };+ let previous_handler = unsafe { x11::xlib::XSetErrorHandler(Some(ignore_x_error)) };+ let rc = unsafe { x11::xlib::XGetWindowAttributes(display, child, &mut attrs) };+ unsafe {+ x11::xlib::XSync(display, 0);+ x11::xlib::XSetErrorHandler(previous_handler);+ }+ if rc == 0 || attrs.override_redirect == 0 || attrs.map_state != x11::xlib::IsViewable {+ return None;+ }+ if attrs.width <= 1 || attrs.height <= 1 {+ return None;+ }+ let pid = crate::x11::window_pid(child as u64).or_else(|| {+ window_children(display, child)+ .into_iter()+ .find_map(|kid| crate::x11::window_pid(kid as u64))+ });+ Some(PopupWindow {+ window: child as u64,+ x: attrs.x,+ y: attrs.y,+ width: attrs.width as u32,+ height: attrs.height as u32,+ title: window_title_for_report(display, child),+ pid,+ })+}++/// Every mapped override-redirect child of the root, bottom to top.+fn mapped_popups(display: *mut x11::xlib::Display) -> Vec<PopupWindow> {+ let root = unsafe { x11::xlib::XDefaultRootWindow(display) };+ window_children(display, root)+ .into_iter()+ .filter_map(|child| popup_info(display, child))+ .collect()+}++/// The mapped override-redirect popup (menu, popover, combo list) under the+/// screen point, if any.+pub fn popup_under_screen_point(x: i32, y: i32) -> Option<PopupWindow> {+ let display = open_display().ok()?;+ let popup = root_child_under_point(display, x, y).and_then(|child| popup_info(display, child));+ unsafe {+ x11::xlib::XCloseDisplay(display);+ }+ popup+}++/// `xid` described as a popup when it is a mapped override-redirect window.+pub fn popup_window_info(xid: u64) -> Option<PopupWindow> {+ let display = open_display().ok()?;+ let popup = popup_info(display, xid as x11::xlib::Window);+ unsafe {+ x11::xlib::XCloseDisplay(display);+ }+ popup+}++/// Mapped popups on the screen right now (bottom to top), for a caller that+/// wants "the menu that is open" without a window id.+pub fn mapped_popup_windows() -> Vec<PopupWindow> {+ let Ok(display) = open_display() else {+ return Vec::new();+ };+ let popups = mapped_popups(display);+ unsafe {+ x11::xlib::XCloseDisplay(display);+ }+ popups+}++/// `_NET_WM_WINDOW_TYPE` says the popup is a tooltip / notification / DND+/// icon: mapped and override-redirect, but nobody's grab.+fn popup_is_passive(display: *mut x11::xlib::Display, window: x11::xlib::Window) -> bool {+ let type_atom = intern_atom(display, "_NET_WM_WINDOW_TYPE");+ let passive: Vec<x11::xlib::Atom> = [+ "_NET_WM_WINDOW_TYPE_TOOLTIP",+ "_NET_WM_WINDOW_TYPE_NOTIFICATION",+ "_NET_WM_WINDOW_TYPE_DND",+ ]+ .iter()+ .map(|name| intern_atom(display, name))+ .collect();+ let mut actual_type: x11::xlib::Atom = 0;+ let mut actual_format: std::os::raw::c_int = 0;+ let mut nitems: std::os::raw::c_ulong = 0;+ let mut bytes_after: std::os::raw::c_ulong = 0;+ let mut prop: *mut std::os::raw::c_uchar = ptr::null_mut();+ let previous_handler = unsafe { x11::xlib::XSetErrorHandler(Some(ignore_x_error)) };+ let rc = unsafe {+ x11::xlib::XGetWindowProperty(+ display,+ window,+ type_atom,+ 0,+ 8,+ 0,+ x11::xlib::XA_ATOM,+ &mut actual_type,+ &mut actual_format,+ &mut nitems,+ &mut bytes_after,+ &mut prop,+ )+ };+ unsafe {+ x11::xlib::XSync(display, 0);+ x11::xlib::XSetErrorHandler(previous_handler);+ }+ let mut result = false;+ if rc == 0 && !prop.is_null() {+ if actual_format == 32 && nitems > 0 {+ let atoms = unsafe {+ std::slice::from_raw_parts(prop as *const std::os::raw::c_ulong, nitems as usize)+ };+ result = atoms+ .iter()+ .any(|atom| passive.contains(&(*atom as x11::xlib::Atom)));+ }+ unsafe { x11::xlib::XFree(prop as *mut _) };+ }+ result+}++fn intern_atom(display: *mut x11::xlib::Display, name: &str) -> x11::xlib::Atom {+ let cname = CString::new(name).unwrap_or_default();+ unsafe { x11::xlib::XInternAtom(display, cname.as_ptr(), 0) }+}++/// The topmost mapped override-redirect popup that `pid` owns (a Qt combo+/// list or completer, a GTK/VCL menu) — the toolkit behind it holds an+/// active keyboard grab that makes the X server drop core key events from+/// any *other* master keyboard aimed at that client (`IsInterferingGrab`),+/// so the virtual master keyboard route is silently lost while it is up.+/// Tooltips / notifications are not grabs and do not count.+pub fn popup_of_pid(pid: u32) -> Option<PopupWindow> {+ let display = open_display().ok()?;+ let popup = mapped_popups(display)+ .into_iter()+ .rev()+ .filter(|popup| popup.pid == Some(pid))+ .find(|popup| !popup_is_passive(display, popup.window as x11::xlib::Window));+ unsafe {+ x11::xlib::XCloseDisplay(display);+ }+ popup+}++/// `_NET_WM_PID` of the window holding the core keyboard focus, walking up+/// its X parents (the focus often sits on a child of the client toplevel).+pub fn core_focus_owner_pid() -> Option<u32> {+ let display = open_display().ok()?;+ let mut focus: x11::xlib::Window = 0;+ let mut revert: std::os::raw::c_int = 0;+ unsafe { x11::xlib::XGetInputFocus(display, &mut focus, &mut revert) };+ let root = unsafe { x11::xlib::XDefaultRootWindow(display) };+ let mut owner = None;+ let mut current = focus;+ for _ in 0..8 {+ if current == 0 || current == root || current == x11::xlib::PointerRoot as x11::xlib::Window+ {+ break;+ }+ if let Some(pid) = crate::x11::window_pid(current as u64) {+ owner = Some(pid);+ break;+ }+ let mut parent: x11::xlib::Window = 0;+ let mut qroot: x11::xlib::Window = 0;+ let mut children: *mut x11::xlib::Window = ptr::null_mut();+ let mut n: std::os::raw::c_uint = 0;+ let previous_handler = unsafe { x11::xlib::XSetErrorHandler(Some(ignore_x_error)) };+ let rc = unsafe {+ x11::xlib::XQueryTree(+ display,+ current,+ &mut qroot,+ &mut parent,+ &mut children,+ &mut n,+ )+ };+ unsafe {+ x11::xlib::XSync(display, 0);+ x11::xlib::XSetErrorHandler(previous_handler);+ }+ if !children.is_null() {+ unsafe { x11::xlib::XFree(children as *mut _) };+ }+ if rc == 0 || parent == current {+ break;+ }+ current = parent;+ }+ unsafe {+ x11::xlib::XCloseDisplay(display);+ }+ owner+}++/// Delivery path name for keys sent on the core keyboard while the target's+/// own popup holds the keyboard grab.+pub const XTEST_CORE_GRAB_PATH: &str = "xtest_core_grab";++/// A popup of the target pid holds the keyboard grab and the core focus is+/// not the target's, so no route reaches the target without stealing input+/// from the focused application; refused before any key was sent.+#[derive(Debug, Clone)]+pub struct PopupKeyboardGrab {+ pub pid: u32,+ pub popup: PopupWindow,+ pub focus_owner: Option<u32>,+}++impl std::fmt::Display for PopupKeyboardGrab {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ write!(+ f,+ "pid {} has {} open, which holds a keyboard grab that drops keys from the virtual \+ keyboard, and the core focus belongs to {}; no key was sent",+ self.pid,+ self.popup.describe(),+ match self.focus_owner {+ Some(pid) => format!("pid {pid}"),+ None => "no window".to_owned(),+ }+ )+ }+}+impl std::error::Error for PopupKeyboardGrab {}++/// Keys for a target whose own popup (`popup`) holds the keyboard grab: the+/// X server routes the *core* keyboard to the grab holder, which is the+/// target itself, so XTest on the core keyboard reaches it without any+/// focus change — verified afterwards on the core focus and the active+/// window. Refused (`PopupKeyboardGrab`) when the core focus belongs to+/// another pid: then a missing grab would send the keys elsewhere.+pub fn send_keys_under_popup_grab(+ pid: u32,+ popup: PopupWindow,+ send: impl FnOnce() -> Result<()>,+) -> Result<KeyboardDeliveryReport> {+ let focus_owner = core_focus_owner_pid();+ if focus_owner != Some(pid) {+ return Err(PopupKeyboardGrab {+ pid,+ popup,+ focus_owner,+ }+ .into());+ }+ let display = open_display()?;+ let saved = save_focus_state(display);+ let sent = send();+ unsafe { x11::xlib::XSync(display, 0) };+ let unchanged = focus_state_unchanged(display, &saved);+ unsafe {+ x11::xlib::XCloseDisplay(display);+ }+ sent?;+ Ok(KeyboardDeliveryReport {+ virtual_focus_held: true,+ core_focus_unchanged: unchanged,+ delivery_confirmed: true,+ key_events: 0,+ skipped_characters: Vec::new(),+ focus_guard: None,+ released_stuck: Vec::new(),+ path: XTEST_CORE_GRAB_PATH,+ grab_popup: Some(popup),+ })+}++/// Cheap post-checks a real-pointer action can make without touching the+/// application: whether the WM's active window / core focus stayed put, and+/// whether the screen region around the action point changed.+#[derive(Clone, Debug, Default)]+pub struct PointerEffect {+ /// `_NET_ACTIVE_WINDOW` and the core input focus were the same after the+ /// action as before it (sampled before the safety-net restore).+ pub focus_unchanged: bool,+ /// Percentage of pixels (0..100) that changed in a bounded region around+ /// the action point, comparing right before the press with ~250 ms after+ /// the release; `None` when the capture failed.+ pub region_diff_pct: Option<f64>,+ /// Real screen point the pointer acted at.+ pub x: i32,+ pub y: i32,+ /// Mapped override-redirect toplevels (menus, popovers, combo lists,+ /// tooltips) that appeared between the press and the post-check. A+ /// context menu opening is a window change the contract accepts as+ /// evidence of a landed click.+ pub popups_appeared: usize,+ /// Background focus-guard outcome when the tool ran the action under+ /// [`focus_guard`]: whether the application moved the desktop focus after+ /// the press (a menu grab, a dialog) and whether it was restored.+ pub focus_guard: Option<FocusGuardReport>,+ /// The point was under another window of the target's own pid (its+ /// dialog over its main window) and the press went to that window.+ pub retargeted_to: Option<SameAppCover>,+ /// The popups behind `popups_appeared`, so the tool can name them.+ pub popups: Vec<PopupWindow>,+ /// The window-local point the caller asked for, when it had one.+ pub window_point: Option<(i32, i32)>,+ /// A toplevel of ANOTHER pid under the action point (a drop target, the+ /// window a drag ended on): what it did during the action.+ pub foreign_window: Option<ForeignWindowEffect>,+}++/// What a toplevel of another pid under the action point did during the+/// action: its title before/after and the windows its pid mapped meanwhile.+/// The focus guard only watches the target pid, so a file dropped onto VLC+/// (auto-played, title changed) would otherwise be invisible.+#[derive(Clone, Debug, Default, PartialEq, Eq)]+pub struct ForeignWindowEffect {+ pub window: u64,+ pub pid: u32,+ pub title_before: String,+ pub title_after: String,+ pub new_windows: Vec<(u64, String)>,+}++impl ForeignWindowEffect {+ pub fn changed(&self) -> bool {+ self.title_before != self.title_after || !self.new_windows.is_empty()+ }++ pub fn to_json(&self) -> serde_json::Value {+ serde_json::json!({+ "window_id": self.window,+ "pid": self.pid,+ "title_before": self.title_before,+ "title_after": self.title_after,+ "new_windows": self.new_windows.iter().map(|(id, title)| serde_json::json!({"window_id": id, "title": title})).collect::<Vec<_>>(),+ })+ }++ /// One sentence for the tool text.+ pub fn describe(&self) -> String {+ let mut parts = Vec::new();+ if self.title_before != self.title_after {+ parts.push(format!(+ "its title changed from \"{}\" to \"{}\"",+ self.title_before, self.title_after+ ));+ }+ if !self.new_windows.is_empty() {+ parts.push(format!(+ "it opened {}",+ self.new_windows+ .iter()+ .map(|(id, title)| format!("window {id} \"{title}\""))+ .collect::<Vec<_>>()+ .join(", ")+ ));+ }+ format!(+ "The point was over window {} of another application (pid {}); {}.",+ self.window,+ self.pid,+ if parts.is_empty() {+ "it did not visibly react (title unchanged, no new window)".to_owned()+ } else {+ parts.join(" and ")+ }+ )+ }+}++/// The topmost managed toplevel under the screen point that belongs to a+/// pid other than `target_pid`, with that pid's toplevels at this moment.+fn foreign_toplevel_under(+ target_pid: Option<u32>,+ x: i32,+ y: i32,+) -> Option<(crate::x11::WindowInfo, Vec<u64>)> {+ let windows = crate::x11::list_windows(None);+ let hit = windows+ .iter()+ .rev()+ .find(|w| {+ w.is_on_screen+ && w.width > 0+ && w.height > 0+ && x >= w.x+ && y >= w.y+ && x < w.x + w.width as i32+ && y < w.y + w.height as i32+ })?+ .clone();+ let pid = hit.pid?;+ if target_pid == Some(pid) {+ return None;+ }+ let owned = windows+ .iter()+ .filter(|w| w.pid == Some(pid))+ .map(|w| w.xid)+ .collect();+ Some((hit, owned))+}++/// Re-read the foreign toplevel after the action.+fn foreign_window_effect(+ before: Option<(crate::x11::WindowInfo, Vec<u64>)>,+) -> Option<ForeignWindowEffect> {+ let (hit, owned) = before?;+ let pid = hit.pid?;+ let title_after = crate::x11::window_info(hit.xid)+ .map(|w| w.title)+ .unwrap_or_default();+ let new_windows = crate::x11::list_windows(Some(pid))+ .into_iter()+ .filter(|w| !owned.contains(&w.xid) && w.is_on_screen)+ .map(|w| (w.xid, w.title))+ .collect();+ Some(ForeignWindowEffect {+ window: hit.xid,+ pid,+ title_before: hit.title,+ title_after,+ new_windows,+ })+}++impl PointerEffect {+ /// A region that changed more than this has plainly reacted to the click+ /// (a menu, a caret, a selection, a pressed button). Below it the click+ /// may still have landed without a visible reaction.+ pub const LANDED_THRESHOLD_PCT: f64 = 0.4;++ pub fn landed(&self) -> bool {+ self.popups_appeared > 0+ || self+ .region_diff_pct+ .is_some_and(|pct| pct >= Self::LANDED_THRESHOLD_PCT)+ }+}++/// Mapped override-redirect children of the root window: popup menus and+/// popovers, but also tooltips and the agent-cursor overlay, so only the+/// *new* ones across an action are meaningful (see [`mapped_popups`]).+fn new_popups(before: &[PopupWindow], after: Vec<PopupWindow>) -> Vec<PopupWindow> {+ after+ .into_iter()+ .filter(|popup| !before.iter().any(|old| old.window == popup.window))+ .collect()+}- let result = (|| -> Result<()> {- for (cursor_id, drag) in drags {- let ids = ensure_master_pointer(cursor_id)?;- let device = uinput_pointers()- .lock()- .unwrap()- .get(cursor_id)- .cloned()- .ok_or_else(|| anyhow!("missing uinput pointer for '{cursor_id}'"))?;- let (cum, total) = path_cumulative(&drag.path);- let start = *drag.path.first().unwrap_or(&(0, 0));- active.push(ActiveDrag {- cursor_id: cursor_id.clone(),- ids,- device,- drag: drag.clone(),- cum,- total,- steps: drag.steps.max(1),- step_delay: if drag.steps.max(1) > 1 {- Duration::from_millis(drag.duration_ms / drag.steps.max(1) as u64)- } else {- Duration::from_millis(drag.duration_ms)- },- current_step: 0,- next_at: start_at,- last_x: start.0,- last_y: start.1,- });- }+/// Debug overrides for the multi-click cadence (milliseconds), read once.+fn click_cadence() -> (u64, u64) {+ fn env_ms(name: &str, default: u64) -> u64 {+ std::env::var(name)+ .ok()+ .and_then(|v| v.parse().ok())+ .unwrap_or(default)+ }+ (+ env_ms("CUA_MPX_PRESS_MS", 12),+ env_ms("CUA_MPX_CLICK_GAP_MS", CLICK_DELAY_MS),+ )+}- let now = std::time::Instant::now();- if start_at > now {- std::thread::sleep(start_at - now);- }+/// Half-size of the square screen region compared around the action point.+const EFFECT_REGION_HALF: i32 = 120;+/// Half-size of the central square masked out of the comparison: the agent+/// cursor overlay pulses there and would count as a change.+const EFFECT_MASK_HALF: i32 = 32;+/// Wait for the application to react before comparing the region.+const EFFECT_SETTLE: Duration = Duration::from_millis(250);++/// Raw ZPixmap bytes of a root-window region (clamped to the screen).+fn root_region_pixels(cx: i32, cy: i32) -> Option<(Vec<u8>, usize, usize)> {+ let (conn, screen_num) = RustConnection::connect(None).ok()?;+ let screen = &conn.setup().roots[screen_num];+ let sw = i32::from(screen.width_in_pixels);+ let sh = i32::from(screen.height_in_pixels);+ let x0 = (cx - EFFECT_REGION_HALF).clamp(0, sw.max(1) - 1);+ let y0 = (cy - EFFECT_REGION_HALF).clamp(0, sh.max(1) - 1);+ let x1 = (cx + EFFECT_REGION_HALF).clamp(x0 + 1, sw);+ let y1 = (cy + EFFECT_REGION_HALF).clamp(y0 + 1, sh);+ let (w, h) = ((x1 - x0) as usize, (y1 - y0) as usize);+ let reply = conn+ .get_image(+ ImageFormat::Z_PIXMAP,+ screen.root,+ x0 as i16,+ y0 as i16,+ w as u16,+ h as u16,+ u32::MAX,+ )+ .ok()?+ .reply()+ .ok()?;+ if reply.depth < 24 || reply.data.len() < w * h * 4 {+ return None;+ }+ Some((reply.data, w, h))+}- // Shield each drag from the WM's click-to-focus grab, then press.- // Per item: install a device-specific sync grab on the target window,- // warp, press, and immediately replay the frozen press so it reaches- // the app while the WM stays blind to it. We replay each press before- // emitting the next so only ONE device is ever frozen at a time — the- // X server drops replayed presses when several devices are frozen on- // the same window and replayed together. The few-ms stagger this adds- // to the presses is invisible; the concurrency that matters is motion.- // Shielding is mandatory: if install/replay fails, abort instead of- // continuing with a drag that could steal focus and rely on restore.- let mut shielded = std::collections::HashSet::new();- for item in &active {- let opcode = xi_opcode.ok_or_else(|| {- anyhow!("parallel_mouse_drag requires XInput/XI2 shield grabs for no-focus-steal operation")- })?;- install_shield_grab(- display,- item.ids.pointer_id,- item.drag.target_window as x11::xlib::Window,- item.drag.button,- )- .with_context(|| format!("shield grab failed for '{}'", item.cursor_id))?;- shielded.insert(item.ids.pointer_id);- let start = *item.drag.path.first().unwrap_or(&(0, 0));- warp_master_pointer(display, item.ids, start.0, start.1)?;+/// Percentage of pixels outside the central mask whose 8-bit channels differ+/// by more than a small tolerance between two captures of the same region.+pub(crate) fn region_diff_pct(+ before: &(Vec<u8>, usize, usize),+ after: &(Vec<u8>, usize, usize),+) -> Option<f64> {+ let (a, w, h) = before;+ let (b, w2, h2) = after;+ if w != w2 || h != h2 || a.len() != b.len() || *w == 0 || *h == 0 {+ return None;+ }+ let (cx, cy) = (*w as i32 / 2, *h as i32 / 2);+ let mut changed = 0usize;+ let mut counted = 0usize;+ for y in 0..*h {+ for x in 0..*w {+ if (x as i32 - cx).abs() <= EFFECT_MASK_HALF+ && (y as i32 - cy).abs() <= EFFECT_MASK_HALF{- let mut device = item.device.lock().unwrap();- emit_button(&mut device, item.drag.button, true)?;+ continue;}- let mut pending = std::collections::HashSet::from([item.ids.pointer_id]);- replay_shielded_presses(display, opcode, &mut pending, Duration::from_millis(1000));- if !pending.is_empty() {- return Err(anyhow!(- "shield replay timed out before XI_ButtonPress arrived for '{}'",- item.cursor_id- ));+ counted += 1;+ let i = (y * w + x) * 4;+ let differs = (0..3).any(|c| {+ let pa = i32::from(a[i + c]);+ let pb = i32::from(b[i + c]);+ (pa - pb).abs() > 24+ });+ if differs {+ changed += 1;}}+ }+ (counted > 0).then(|| 100.0 * changed as f64 / counted as f64)+}- while active.iter().any(|item| item.current_step < item.steps) {- let now = std::time::Instant::now();- let mut advanced = false;- let mut next_deadline = None;-- for item in &mut active {- if item.current_step >= item.steps {+/// Best-effort `_NET_WM_NAME` / `WM_NAME` of a window (or of a child, for a+/// WM frame), for naming a covering window in a refusal.+fn window_title_for_report(display: *mut x11::xlib::Display, window: x11::xlib::Window) -> String {+ fn name_of(display: *mut x11::xlib::Display, window: x11::xlib::Window) -> Option<String> {+ unsafe {+ let net_name =+ x11::xlib::XInternAtom(display, c"_NET_WM_NAME".as_ptr(), x11::xlib::True);+ for prop in [net_name, x11::xlib::XA_WM_NAME] {+ if prop == 0 {continue;}- if now >= item.next_at {- item.current_step += 1;- let t = item.current_step as f64 / item.steps as f64;- let (ix, iy) = point_on_path(&item.drag.path, &item.cum, item.total, t);- let dx = ix - item.last_x;- let dy = iy - item.last_y;- if dx != 0 || dy != 0 {- let mut device = item.device.lock().unwrap();- emit_relative_motion(&mut device, dx, dy)?;- // Keep the agent cursor overlay tracking the drag so- // the gesture is visible, not just its endpoints.- crate::overlay::send_command_for(- item.cursor_id.clone(),- cursor_overlay::OverlayCommand::SnapTo {- x: ix as f64,- y: iy as f64,- heading_radians: Some((dy as f64).atan2(dx as f64)),- },- );- }- item.last_x = ix;- item.last_y = iy;- item.next_at = now + item.step_delay;- advanced = true;- }- if item.current_step < item.steps {- next_deadline = Some(match next_deadline {- Some(deadline) => std::cmp::min(deadline, item.next_at),- None => item.next_at,- });- }- }-- if !advanced {- if let Some(deadline) = next_deadline {- let now = std::time::Instant::now();- if deadline > now {- std::thread::sleep(deadline - now);+ let mut type_ret: x11::xlib::Atom = 0;+ let mut format_ret = 0;+ let mut nitems: std::os::raw::c_ulong = 0;+ let mut bytes_after: std::os::raw::c_ulong = 0;+ let mut data: *mut std::os::raw::c_uchar = std::ptr::null_mut();+ let rc = x11::xlib::XGetWindowProperty(+ display,+ window,+ prop,+ 0,+ 256,+ x11::xlib::False,+ x11::xlib::AnyPropertyType as x11::xlib::Atom,+ &mut type_ret,+ &mut format_ret,+ &mut nitems,+ &mut bytes_after,+ &mut data,+ );+ if rc == x11::xlib::Success as i32 && !data.is_null() {+ let text = if format_ret == 8 && nitems > 0 {+ Some(+ String::from_utf8_lossy(std::slice::from_raw_parts(+ data,+ nitems as usize,+ ))+ .into_owned(),+ )+ } else {+ None+ };+ x11::xlib::XFree(data as *mut _);+ if let Some(text) = text.filter(|t| !t.trim().is_empty()) {+ return Some(text);}}}}+ None+ }+ let previous_handler = unsafe { x11::xlib::XSetErrorHandler(Some(ignore_x_error)) };+ let mut title = name_of(display, window);+ if title.is_none() {+ title = window_children(display, window)+ .into_iter()+ .rev()+ .find_map(|kid| name_of(display, kid));+ }+ unsafe {+ x11::xlib::XSync(display, 0);+ x11::xlib::XSetErrorHandler(previous_handler);+ }+ title.unwrap_or_default()+}- for item in &active {- let mut device = item.device.lock().unwrap();- emit_button(&mut device, item.drag.button, false)?;- }+/// Direct children of `window` (empty on error). Caller installs the error+/// handler.+fn window_children(+ display: *mut x11::xlib::Display,+ window: x11::xlib::Window,+) -> Vec<x11::xlib::Window> {+ let mut root_ret: x11::xlib::Window = 0;+ let mut parent: x11::xlib::Window = 0;+ let mut children: *mut x11::xlib::Window = ptr::null_mut();+ let mut count: std::os::raw::c_uint = 0;+ let rc = unsafe {+ x11::xlib::XQueryTree(+ display,+ window,+ &mut root_ret,+ &mut parent,+ &mut children,+ &mut count,+ )+ };+ if rc == 0 || children.is_null() {+ return Vec::new();+ }+ let kids = unsafe { std::slice::from_raw_parts(children, count as usize) }.to_vec();+ unsafe { x11::xlib::XFree(children as *mut _) };+ kids+}- // Remove the shields now that the drag is done. The button is only- // grabbed for ButtonPress, so the shield is dormant during motion and- // release; this just stops it matching the next gesture's press.- for item in &active {- if shielded.contains(&item.ids.pointer_id) {- remove_shield_grab(- display,- item.ids.pointer_id,- item.drag.target_window as x11::xlib::Window,- item.drag.button,- );+/// The toplevel (root child) the virtual pointer would press on at `(x, y)`+/// when the caller means `window`. `Ok(None)` when the point is over the+/// target itself (its WM frame) or over an override-redirect popup (a menu or+/// combo list the click is meant to reach, or the agent-cursor overlay);+/// `Ok(Some(_))` names the window that covers the point instead.+fn occluding_window(+ display: *mut x11::xlib::Display,+ window: x11::xlib::Window,+ x: i32,+ y: i32,+) -> Result<PointCover> {+ if let Some(bounds) = window_screen_bounds(display, window) {+ if !bounds_contain(bounds, x, y) {+ return Ok(PointCover::Outside(PointOutsideWindow {+ target_window: window as u64,+ screen_x: x,+ screen_y: y,+ window_x: x - bounds.0,+ window_y: y - bounds.1,+ bounds,+ }));+ }+ }+ let under = root_child_under_point(display, x, y);+ let frame = root_child_of(display, window);+ match (under, frame) {+ (Some(under), Some(frame)) if under != frame && !is_override_redirect(display, under) => {+ let previous_handler = unsafe { x11::xlib::XSetErrorHandler(Some(ignore_x_error)) };+ let covering_client = window_children(display, under)+ .into_iter()+ .find(|kid| crate::x11::window_pid(*kid as u64).is_some());+ let covering_pid = crate::x11::window_pid(under as u64)+ .or_else(|| covering_client.and_then(|kid| crate::x11::window_pid(kid as u64)));+ unsafe {+ x11::xlib::XSync(display, 0);+ x11::xlib::XSetErrorHandler(previous_handler);}+ let target_pid = crate::x11::window_pid(window as u64);+ let title = window_title_for_report(display, under);+ // The app's own window (a dialog, a file chooser) sits over the+ // point: the press goes where the caller can see it, on that+ // window, and the effect reports the retarget.+ if target_pid.is_some() && covering_pid == target_pid {+ return Ok(PointCover::SameApp(SameAppCover {+ window: covering_client.map(u64::from).unwrap_or(under as u64),+ title,+ }));+ }+ Ok(PointCover::Occluded(TargetOccluded {+ target_window: window as u64,+ covering_window: under as u64,+ covering_title: title,+ covering_pid,+ x,+ y,+ }))}- Ok(())- })();- // Remove the per-session masters before handing focus back: non-MPX-aware- // WMs (xfwm4, openbox) desync their focus bookkeeping while foreign- // master keyboards linger, and the next call recreates masters cheaply.- for (cursor_id, _) in drags {- forget_master_pointer(cursor_id);+ (Some(_), Some(_)) | (None, _) => Ok(PointCover::Clear),+ (Some(_), None) => bail!("target window {window} is not mapped on this screen"),}- restore_focus_state(display, &saved_focus);+}++/// What sits over the screen point a background pointer action aims at.+#[derive(Debug, Clone)]+enum PointCover {+ /// The target (or a popup meant to receive the press).+ Clear,+ /// A window of the target's own pid; the press is retargeted to it.+ SameApp(SameAppCover),+ /// Another application's window; refused.+ Occluded(TargetOccluded),+ /// The point is not inside the target window at all; refused.+ Outside(PointOutsideWindow),+}++/// The target application's own window that received a press aimed at a+/// point of another of its windows (its dialog over its main window).+#[derive(Debug, Clone, Default, PartialEq, Eq)]+pub struct SameAppCover {+ /// Client window id (the `list_windows` id, not the WM frame).+ pub window: u64,+ pub title: String,+}++/// True when neither the EWMH active window nor the core focus moved since+/// `saved` was taken.+fn focus_state_unchanged(display: *mut x11::xlib::Display, saved: &SavedFocus) -> bool {+ unsafe { x11::xlib::XSync(display, 0) };+ let mut core_focus: x11::xlib::Window = 0;+ let mut revert_to: std::os::raw::c_int = 0;unsafe {- x11::xlib::XCloseDisplay(display);+ x11::xlib::XGetInputFocus(display, &mut core_focus, &mut revert_to);}- result+ ewmh_active_window(display) == saved.ewmh_active && core_focus == saved.core_focus}-/// A discrete no-focus-steal pointer click driven through the same real-input-/// pipeline as [`send_parallel_virtual_pointer_drags`] — MPX master pointer +-/// uinput slave + XI2 shield grab — reduced to a press/release (or a short-/// press/release train for `count` > 1) at one screen point.-///-/// This is what lands **right / middle / double** clicks (and any left click-/// the AT-SPI path can't actuate) on XInput2 toolkits: GTK3/4 silently drop-/// synthetic `XSendEvent` pointer events and never see XTEST core events, so-/// those clicks are otherwise no-ops. Coordinates are screen-absolute;-/// `target_window` is the X11 window the shield grab is installed on so the WM-/// never sees the press and never steals focus. `button` is an X button number-/// (1=left, 2=middle, 3=right); `count` >= 1 (2 = double-click).-#[derive(Clone, Debug)]-pub struct VirtualPointerClick {- pub target_window: u64,- pub x: i32,- pub y: i32,- pub button: u8,- pub count: usize,+fn pointer_effect(+ display: *mut x11::xlib::Display,+ saved: &SavedFocus,+ before: Option<(Vec<u8>, usize, usize)>,+ popups_before: &[PopupWindow],+ foreign_before: Option<(crate::x11::WindowInfo, Vec<u64>)>,+ x: i32,+ y: i32,+) -> PointerEffect {+ unsafe { x11::xlib::XSync(display, 0) };+ sleep(EFFECT_SETTLE);+ let after = root_region_pixels(x, y);+ let popups = new_popups(popups_before, mapped_popups(display));+ let foreign_window = foreign_window_effect(foreign_before);+ PointerEffect {+ focus_unchanged: focus_state_unchanged(display, saved),+ region_diff_pct: match (before, after) {+ (Some(b), Some(a)) => region_diff_pct(&b, &a),+ _ => None,+ },+ x,+ y,+ popups_appeared: popups.len(),+ focus_guard: None,+ retargeted_to: None,+ popups,+ window_point: None,+ foreign_window,+ }}/// Land a discrete click via the MPX real-input pipeline (see-/// [`VirtualPointerClick`]). Mirrors the per-item press/replay logic of-/// `send_parallel_virtual_pointer_drags`: install a device-specific synchronous-/// XI2 shield grab on the target window, warp the master pointer, then for each-/// press freeze→replay it so the application receives a real button event while-/// the WM stays blind to it. The master is torn down and focus restored on exit-/// (matching the drag) to keep non-MPX WMs' focus bookkeeping consistent.-pub fn send_virtual_pointer_click(cursor_id: &str, click: &VirtualPointerClick) -> Result<()> {+/// [`VirtualPointerClick`]). Refuses with [`TargetOccluded`] when another+/// toplevel covers the point. Returns the post-checks the tool reports.+pub fn send_virtual_pointer_click(+ cursor_id: &str,+ click: &VirtualPointerClick,+) -> Result<PointerEffect> {+ let _op = mpx_op_guard(cursor_id);+ mpx_last_use()+ .lock()+ .unwrap()+ .insert(cursor_id.to_owned(), std::time::Instant::now());let display = open_display()?;supports_parallel_pointer_injection(display)?;- let xi_opcode = xinput_opcode(display);let saved_focus = save_focus_state(display);- let result = (|| -> Result<()> {+ let result = (|| -> Result<PointerEffect> {+ let window = click.target_window as x11::xlib::Window;+ let retargeted_to = match occluding_window(display, window, click.x, click.y)? {+ PointCover::Clear => None,+ PointCover::SameApp(cover) => Some(cover),+ PointCover::Occluded(occluded) => return Err(occluded.into()),+ PointCover::Outside(outside) => return Err(outside.into()),+ };let ids = ensure_master_pointer(cursor_id)?;let device = uinput_pointers().lock()@@ -1564,50 +2572,162 @@ pub fn send_virtual_pointer_click(cursor_id: &str, click: &VirtualPointerClick).get(cursor_id).cloned().ok_or_else(|| anyhow!("missing uinput pointer for '{cursor_id}'"))?;- let opcode = xi_opcode- .ok_or_else(|| anyhow!("no-focus-steal click requires XInput/XI2 shield grabs"))?;-- let window = click.target_window as x11::xlib::Window;- install_shield_grab(display, ids.pointer_id, window, click.button)- .with_context(|| format!("shield grab failed for '{cursor_id}'"))?;- // Run the press train under a guard so the shield is always removed,- // even on an early error mid-train.- let click_result = (|| -> Result<()> {- warp_master_pointer(display, ids, click.x, click.y)?;- let count = click.count.max(1);+ let count = click.count.max(1);+ // A device left frozen by an earlier synchronous grab (a WM that never+ // replayed) would queue this press forever.+ thaw_device(display, ids.pointer_id);+ warp_master_pointer(display, ids, click.x, click.y)?;+ let popups_before = mapped_popups(display);+ let foreign_before = foreign_toplevel_under(+ crate::x11::window_pid(click.target_window),+ click.x,+ click.y,+ );+ let before = root_region_pixels(click.x, click.y);+ let (press_ms, gap_ms) = click_cadence();+ let train = (|| -> Result<()> {for i in 0..count {{let mut device = device.lock().unwrap();emit_button(&mut device, click.button, true)?;- }- // The shield grab freezes the device on every press; drain and- // replay this one so it reaches the app (and re-arms for the- // next press in a multi-click train).- let mut pending = std::collections::HashSet::from([ids.pointer_id]);- replay_shielded_presses(display, opcode, &mut pending, Duration::from_millis(1000));- if !pending.is_empty() {- return Err(anyhow!(- "shield replay timed out before XI_ButtonPress arrived for '{cursor_id}'"- ));- }- {- let mut device = device.lock().unwrap();+ // A real press and release are separate evdev frames; give+ // the toolkit a press it can see before the release lands.+ sleep(Duration::from_millis(press_ms));emit_button(&mut device, click.button, false)?;}// Multi-click cadence: keep press→press well under the toolkit// double-click threshold (GTK default 250 ms) so count=2 lands// as a real double-click, not two singles.if count > 1 && i + 1 < count {- sleep(Duration::from_millis(CLICK_DELAY_MS));+ sleep(Duration::from_millis(gap_ms));+ }+ }+ Ok(())+ })();+ // Never leave the button held: a stuck Button1 on the virtual master+ // sets bit 8 of the core modifier state (Orca's modifier), so every+ // later virtual-keyboard chord would open Orca's preferences.+ release_button_best_effort(&device, click.button);+ train?;+ let mut effect = pointer_effect(+ display,+ &saved_focus,+ before,+ &popups_before,+ foreign_before,+ click.x,+ click.y,+ );+ effect.retargeted_to = retargeted_to;+ Ok(effect)+ })();++ restore_focus_state(display, &saved_focus);+ unsafe {+ x11::xlib::XCloseDisplay(display);+ }+ result+}++/// One held drag on the session's virtual master pointer: press at+/// `path[0]`, glide through the waypoints, release at the end. Same delivery+/// rules as [`send_virtual_pointer_click`] (no shield grab, occlusion refusal+/// at the press point, focus saved and restored).+pub fn send_virtual_pointer_drag(+ cursor_id: &str,+ drag: &VirtualPointerDrag,+) -> Result<PointerEffect> {+ let _op = mpx_op_guard(cursor_id);+ mpx_last_use()+ .lock()+ .unwrap()+ .insert(cursor_id.to_owned(), std::time::Instant::now());+ let display = open_display()?;+ supports_parallel_pointer_injection(display)?;+ let saved_focus = save_focus_state(display);++ let result = (|| -> Result<PointerEffect> {+ if drag.path.len() < 2 {+ bail!("drag path needs at least 2 points");+ }+ let window = drag.target_window as x11::xlib::Window;+ let start = drag.path[0];+ let end = drag.path[drag.path.len() - 1];+ let retargeted_to = match occluding_window(display, window, start.0, start.1)? {+ PointCover::Clear => None,+ PointCover::SameApp(cover) => Some(cover),+ PointCover::Occluded(occluded) => return Err(occluded.into()),+ PointCover::Outside(outside) => return Err(outside.into()),+ };+ let ids = ensure_master_pointer(cursor_id)?;+ let device = uinput_pointers()+ .lock()+ .unwrap()+ .get(cursor_id)+ .cloned()+ .ok_or_else(|| anyhow!("missing uinput pointer for '{cursor_id}'"))?;+ let (cum, total) = path_cumulative(&drag.path);+ let steps = drag.steps.max(1);+ let step_delay = Duration::from_millis(drag.duration_ms / steps as u64);+ thaw_device(display, ids.pointer_id);+ warp_master_pointer(display, ids, start.0, start.1)?;+ let popups_before = mapped_popups(display);+ let foreign_before =+ foreign_toplevel_under(crate::x11::window_pid(drag.target_window), end.0, end.1);+ let before = root_region_pixels(end.0, end.1);+ let gesture = (|| -> Result<()> {+ {+ let mut device = device.lock().unwrap();+ emit_button(&mut device, drag.button, true)?;+ }+ // Let the toolkit register the press (and arm its drag threshold)+ // before the first motion.+ sleep(Duration::from_millis(40));+ let (mut last_x, mut last_y) = start;+ for step in 1..=steps {+ let t = step as f64 / steps as f64;+ let (ix, iy) = point_on_path(&drag.path, &cum, total, t);+ let (dx, dy) = (ix - last_x, iy - last_y);+ if dx != 0 || dy != 0 {+ let mut device = device.lock().unwrap();+ emit_relative_motion(&mut device, dx, dy)?;+ crate::overlay::send_command_for(+ cursor_id.to_owned(),+ cursor_overlay::OverlayCommand::SnapTo {+ x: ix as f64,+ y: iy as f64,+ heading_radians: Some((dy as f64).atan2(dx as f64)),+ },+ );}+ last_x = ix;+ last_y = iy;+ sleep(step_delay);}+ // Relative motion accumulates libinput rounding; pin the release+ // to the exact end point before letting go.+ warp_master_pointer(display, ids, end.0, end.1)?;+ sleep(Duration::from_millis(20));+ let mut device = device.lock().unwrap();+ emit_button(&mut device, drag.button, false)?;Ok(())})();- remove_shield_grab(display, ids.pointer_id, window, click.button);- click_result+ // Release on every exit path (see `send_virtual_pointer_click`).+ release_button_best_effort(&device, drag.button);+ gesture?;+ let mut effect = pointer_effect(+ display,+ &saved_focus,+ before,+ &popups_before,+ foreign_before,+ end.0,+ end.1,+ );+ effect.retargeted_to = retargeted_to;+ Ok(effect)})();- forget_master_pointer(cursor_id);restore_focus_state(display, &saved_focus);unsafe {x11::xlib::XCloseDisplay(display);@@ -1636,6 +2756,7 @@ pub struct VirtualPointerScroll {/// point, then emits `|ticks|` wheel detents on the uinput slave. The master is/// torn down and focus restored on exit, matching the click/drag paths.pub fn send_virtual_pointer_scroll(cursor_id: &str, scroll: &VirtualPointerScroll) -> Result<()> {+ let _op = mpx_op_guard(cursor_id);let display = open_display()?;supports_parallel_pointer_injection(display)?;let saved_focus = save_focus_state(display);@@ -1665,7 +2786,7 @@ pub fn send_virtual_pointer_scroll(cursor_id: &str, scroll: &VirtualPointerScrolOk(())})();- forget_master_pointer(cursor_id);+ let _ = cursor_id; // master pair retained for reuse (see the drag path).restore_focus_state(display, &saved_focus);unsafe {x11::xlib::XCloseDisplay(display);@@ -1707,6 +2828,24 @@ fn restore_focus_state(display: *mut x11::xlib::Display, saved: &SavedFocus) {unsafe { x11::xlib::XSync(display, 0) };if let Some(prev) = saved.ewmh_active {+ // Fast path: on WMs whose click-to-focus grab ignores the virtual+ // master (mutter grabs for the Virtual Core Pointer only) nothing+ // moved, and a 300 ms settle per click would be pure latency.+ if ewmh_active_window(display) == Some(prev) {+ let mut stable = 0;+ for _ in 0..4 {+ sleep(Duration::from_millis(25));+ if ewmh_active_window(display) == Some(prev) {+ stable += 1;+ } else {+ stable = 0;+ break;+ }+ }+ if stable >= 3 {+ return;+ }+ }// EWMH path: ask the WM to re-activate, so its active-window// bookkeeping (decorations, stacking) stays consistent. The WM// processes its own click-to-focus for the drag asynchronously and@@ -2228,14 +3367,32 @@ pub fn send_type_text_with_delay(xid: u64, text: &str, inter_char_ms: u64) -> Relet window = xid as u32;let root = conn.setup().roots[0].root;let mapping = conn.get_keyboard_mapping(8, 248)?.reply()?;+ // Remap guards live until the function returns: a character outside the+ // keymap (CJK, emoji, ...) borrows a spare keycode rather than being+ // silently dropped — see keycode_for_keysym's doc and the analogous fix+ // in mpx_keyboard::plan_text_with_fallback for the background path.+ let mut remap_guards = Vec::new();for ch in text.chars() {// Resolve the keycode and whether Shift must be held — without it,// uppercase and shifted symbols would otherwise type their unshifted// form (e.g. "A" arriving as "a").- let Some((keycode, needs_shift)) = char_to_keycode_shift(&mapping, ch as u32) else {- continue;- };+ let (keycode, needs_shift) =+ match char_to_keycode_shift(&mapping, mpx_keyboard::keysym_for_char(ch)) {+ Some(found) => found,+ None => match keycode_for_keysym(+ &conn,+ &mapping,+ mpx_keyboard::keysym_for_char(ch),+ &ch.to_string(),+ ) {+ Ok((keycode, guard)) => {+ remap_guards.extend(guard);+ (keycode, false)+ }+ Err(_) => continue,+ },+ };let state = if needs_shift {KeyButMask::SHIFT} else {@@ -2307,14 +3464,19 @@ pub fn send_type_text_xtest(text: &str) -> Result<()> {.get(..kpm).and_then(|s| s.iter().copied().find(|&k| k != 0)).unwrap_or(50);+ // Remap guards live until the function returns — see send_type_text_with_delay.+ let mut remap_guards = Vec::new();for ch in text.chars() {- let cp = match ch {- '\n' => 0xff0d, // XK_Return- '\t' => 0xff09, // XK_Tab- c => c as u32,- };- let Some((keycode, needs_shift)) = char_to_keycode_shift(&mapping, cp) else {- continue;+ let cp = mpx_keyboard::keysym_for_char(ch);+ let (keycode, needs_shift) = match char_to_keycode_shift(&mapping, cp) {+ Some(found) => found,+ None => match keycode_for_keysym(&conn, &mapping, cp, &ch.to_string()) {+ Ok((keycode, guard)) => {+ remap_guards.extend(guard);+ (keycode, false)+ }+ Err(_) => continue,+ },};if needs_shift {conn.xtest_fake_input(KEY_PRESS_EVENT, shift_kc, 0, x11rb::NONE, 0, 0, 0)?;@@ -2776,8 +3938,8 @@ fn key_name_to_keysym(key: &str) -> Result<u32> {"insert" | "ins" => 0xFF63,"home" => 0xFF50,"end" => 0xFF57,- "pageup" | "pgup" => 0xFF55,- "pagedown" | "pgdn" => 0xFF56,+ "pageup" | "pgup" | "page_up" => 0xFF55,+ "pagedown" | "pgdn" | "page_down" => 0xFF56,"up" => 0xFF52,"down" => 0xFF54,"left" => 0xFF51,@@ -2797,7 +3959,7 @@ fn key_name_to_keysym(key: &str) -> Result<u32> {"shift" => 0xFFE1,"ctrl" | "control" => 0xFFE3,"alt" => 0xFFE9,- "super" | "meta" | "win" => 0xFFEB,+ "super" | "meta" | "win" | "cmd" => 0xFFEB,"capslock" => 0xFFE5,"numlock" => 0xFF7F,// Common X keysym names for punctuation. The single-char branch below@@ -2823,6 +3985,36 @@ fn key_name_to_keysym(key: &str) -> Result<u32> {Ok(keysym)}+#[cfg(test)]+mod key_name_alias_tests {+ use super::key_name_to_keysym;++ #[test]+ fn common_key_name_aliases_resolve_to_the_expected_keysym() {+ // Page_Up / Page_Down: the underscore form is the literal X11 keysym+ // name and a common cross-platform prompt spelling; only the+ // no-underscore "pageup"/"pgup" aliases existed before.+ assert_eq!(key_name_to_keysym("Page_Up").unwrap(), 0xFF55);+ assert_eq!(key_name_to_keysym("page_up").unwrap(), 0xFF55);+ assert_eq!(+ key_name_to_keysym("Page_Up").unwrap(),+ key_name_to_keysym("pageup").unwrap()+ );+ assert_eq!(key_name_to_keysym("Page_Down").unwrap(), 0xFF56);+ assert_eq!(+ key_name_to_keysym("Page_Down").unwrap(),+ key_name_to_keysym("pgdn").unwrap()+ );+ // "cmd": common cross-platform prompt name for the Super/Meta/Windows+ // key; must alias to the same keysym as "super"/"meta"/"win".+ assert_eq!(key_name_to_keysym("cmd").unwrap(), 0xFFEB);+ assert_eq!(+ key_name_to_keysym("cmd").unwrap(),+ key_name_to_keysym("super").unwrap()+ );+ }+}+/// A keycode we have *temporarily* rebound to host a keysym that is absent from/// the current X keyboard map (sparse/headless keymaps such as a minimal/// Xwayland). On drop it reinstates the keycode's original keysyms so the@@ -3042,12 +4234,44 @@ mod path_tests {use super::{click_event_target, create_uinput_pointer, ensure_master_pointer_for_session,guarded_uinput_creation, is_uinput_unavailable, kde_x11_uinput_hotplug_is_unsafe,- master_pointer_name, modifiers_to_state, normalize_uinput_device_name, path_cumulative,- point_on_path, real_pointer_capabilities_available, sample_function, slave_pointer_name,- EVDEV_UINPUT_NAME_MAX_BYTES, UINPUT_POINTER_SUFFIX,+ master_pointer_name, modifiers_to_state, mpx_op_guard, normalize_uinput_device_name,+ path_cumulative, point_on_path, real_pointer_capabilities_available, sample_function,+ slave_pointer_name, EVDEV_UINPUT_NAME_MAX_BYTES, UINPUT_POINTER_SUFFIX,};use x11rb::protocol::xproto::{EventMask, KeyButMask};+ /// Regression for the `XSetErrorHandler` swap race: every caller that+ /// could swap the process-global X11 error handler while an MPX+ /// operation is mid-swap (`send_virtual_pointer_*`, `deliver`, and now+ /// `x11_activate_window_persistent`) must take `MPX_OP_LOCK` first. This+ /// doesn't touch X11 (no display in CI), but proves the lock itself is+ /// exclusive and not reentrant-by-accident: a second `mpx_op_guard` call+ /// blocks until the first is dropped, exactly the property the handler+ /// swap needs to be race-free.+ #[test]+ fn mpx_op_guard_is_mutually_exclusive() {+ use std::sync::atomic::{AtomicBool, Ordering};+ use std::sync::Arc;++ let entered_second = Arc::new(AtomicBool::new(false));+ let first = mpx_op_guard("test-a");+ let entered_second_clone = entered_second.clone();+ let handle = std::thread::spawn(move || {+ let _second = mpx_op_guard("test-b");+ entered_second_clone.store(true, Ordering::SeqCst);+ });+ // The second guard cannot have been acquired yet — it is blocked on+ // the lock we are still holding.+ std::thread::sleep(std::time::Duration::from_millis(50));+ assert!(+ !entered_second.load(Ordering::SeqCst),+ "second guard acquired while the first was still held"+ );+ drop(first);+ handle.join().unwrap();+ assert!(entered_second.load(Ordering::SeqCst));+ }+#[test]fn synthetic_click_keeps_release_on_press_recipient() {let path = [@@ -3110,6 +4334,23 @@ mod path_tests {);}+ #[test]+ fn idle_reaper_selects_only_sessions_past_the_ttl() {+ use super::{stale_cursor_ids, MPX_IDLE_TTL};+ let now = std::time::Instant::now();+ let mut last_use = std::collections::HashMap::new();+ last_use.insert("fresh".to_owned(), now);+ last_use.insert(+ "old".to_owned(),+ now - (MPX_IDLE_TTL + std::time::Duration::from_secs(60)),+ );+ last_use.insert("edge".to_owned(), now - MPX_IDLE_TTL);+ let mut stale = stale_cursor_ids(&last_use, now, MPX_IDLE_TTL);+ stale.sort();+ assert_eq!(stale, vec!["edge".to_owned(), "old".to_owned()]);+ assert!(stale_cursor_ids(&std::collections::HashMap::new(), now, MPX_IDLE_TTL).is_empty());+ }+#[test]fn slave_pointer_name_fits_evdev_uinput_limit() {for cursor_id in ["m".repeat(200), "cursor-鼠".repeat(50)] {@@ -3186,7 +4427,7 @@ mod path_tests {assert!(is_uinput_unavailable(&error));assert_eq!(error.to_string(),- "Linux uinput pointer unavailable: permission denied"+ "Linux uinput device unavailable: permission denied");}diff --git a/libs/cua-driver/rust/crates/platform-linux/src/input/mpx_keyboard.rs b/libs/cua-driver/rust/crates/platform-linux/src/input/mpx_keyboard.rsnew file mode 100644index 0000000000..2aba7cd014--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-linux/src/input/mpx_keyboard.rs@@ -0,0 +1,956 @@+//! Focus-free keyboard delivery on X11 via XInput2 MPX: a second master+//! keyboard with its own focus, fed by a uinput slave.+//!+//! X11 has exactly one *core* keyboard focus, which is why XTest (real input,+//! but always to the core focus) cannot type into a background window, and+//! why the toolkits that ignore `send_event` (GTK, LibreOffice VCL, Qt,+//! Chromium) drop the XSendEvent alternative. MPX lifts the single-focus+//! limit: every master keyboard carries its own focus window+//! (`XISetFocus`), and key events from a slave attached to that master are+//! delivered — as real, non-synthetic XI2 *and* core events — to that master's+//! focus, while the user's core keyboard, its focus and the pointer stay+//! untouched. XTEST slaves cannot be re-attached to another master, so the+//! slave is a uinput keyboard hot-added through udev/libinput.+//!+//! Lifecycle mirrors the MPX pointer click: the master pair is created (or+//! reused) for the session, the keyboard slave is attached, the target window+//! is focused for that master only, keys are emitted, delivery is confirmed+//! through the server's raw key events, and the pair is torn down again so a+//! non-MPX-aware WM never sees a lingering foreign master.++use super::*;+use x11rb::protocol::xproto::{ConnectionExt as _, GetKeyboardMappingReply};++const UINPUT_KEYBOARD_SUFFIX: &str = " uinput keyboard";+/// Structured `path` reported for a delivery through the virtual master keyboard.+pub const MPX_UINPUT_PATH: &str = "mpx_uinput";+/// evdev keycodes are X keycodes minus this offset (X reserves 0..=7).+const X_KEYCODE_OFFSET: u16 = 8;+/// Highest evdev key code the uinput keyboard advertises (KEY_* range below+/// the BTN_* block at 0x100, so libinput classifies it as a keyboard only).+const MAX_EVDEV_KEY: u16 = 0xff;+/// Unicode keysyms are the codepoint with this bit set (X11 keysym encoding).+const UNICODE_KEYSYM_BASE: u32 = 0x0100_0000;+/// How long the server gets to report the last emitted key through a raw+/// event before the pair is torn down.+const DELIVERY_CONFIRM_TIMEOUT: Duration = Duration::from_millis(1500);++static UINPUT_KEYBOARDS: OnceLock<Mutex<HashMap<String, Arc<Mutex<VirtualDevice>>>>> =+ OnceLock::new();++fn uinput_keyboards() -> &'static Mutex<HashMap<String, Arc<Mutex<VirtualDevice>>>> {+ UINPUT_KEYBOARDS.get_or_init(|| Mutex::new(HashMap::new()))+}++pub(super) fn forget_uinput_keyboard(cursor_id: &str) {+ uinput_keyboards().lock().unwrap().remove(cursor_id);+}++fn slave_keyboard_name(master_name: &str) -> String {+ format!("{master_name}{UINPUT_KEYBOARD_SUFFIX}")+}++/// The keyboard route needs exactly what the MPX pointer route needs: a real+/// Xorg with udev/libinput hotplug and a writable `/dev/uinput`.+pub fn real_keyboard_input_available() -> bool {+ real_pointer_input_available()+}++/// One evdev key transition for the uinput keyboard.+#[derive(Clone, Copy, Debug, PartialEq, Eq)]+pub(super) struct KeyStep {+ pub evdev_code: u16,+ pub press: bool,+}++impl KeyStep {+ fn x_keycode(self) -> u8 {+ (self.evdev_code + X_KEYCODE_OFFSET) as u8+ }+}++/// X keycode → evdev code. X keycodes below 8 have no evdev counterpart.+pub(super) fn evdev_code_for_x_keycode(keycode: u8) -> Option<u16> {+ let keycode = u16::from(keycode);+ (keycode >= X_KEYCODE_OFFSET && keycode - X_KEYCODE_OFFSET <= MAX_EVDEV_KEY)+ .then(|| keycode - X_KEYCODE_OFFSET)+}++/// Keysym for a typed character: Latin-1 keysyms are the codepoint, everything+/// above is the Unicode keysym; `\n` and `\t` become Return and Tab.+pub(super) fn keysym_for_char(ch: char) -> u32 {+ match ch {+ '\n' | '\r' => 0xff0d,+ '\t' => 0xff09,+ c if (c as u32) < 0x100 => c as u32,+ c => UNICODE_KEYSYM_BASE | c as u32,+ }+}++/// Press/release steps for one key at `keycode`, wrapped in Shift when the+/// keysym lives in the shifted column and Shift is not already held.+fn tap_steps(keycode: u8, needs_shift: bool, shift: Option<u16>, steps: &mut Vec<KeyStep>) {+ let Some(code) = evdev_code_for_x_keycode(keycode) else {+ return;+ };+ let shift = if needs_shift { shift } else { None };+ if let Some(shift) = shift {+ steps.push(KeyStep {+ evdev_code: shift,+ press: true,+ });+ }+ steps.push(KeyStep {+ evdev_code: code,+ press: true,+ });+ steps.push(KeyStep {+ evdev_code: code,+ press: false,+ });+ if let Some(shift) = shift {+ steps.push(KeyStep {+ evdev_code: shift,+ press: false,+ });+ }+}++/// Plan the evdev transitions that type `text` under `mapping`. Characters+/// with no keycode in the map are returned in the second tuple element (the+/// caller may remap a spare keycode for them or report them as skipped).+pub(super) fn plan_text(+ mapping: &GetKeyboardMappingReply,+ shift_x_keycode: Option<u8>,+ text: &str,+) -> (Vec<KeyStep>, Vec<char>) {+ let shift = shift_x_keycode.and_then(evdev_code_for_x_keycode);+ let mut steps = Vec::with_capacity(text.len() * 2);+ let mut missing = Vec::new();+ for ch in text.chars() {+ match char_to_keycode_shift(mapping, keysym_for_char(ch)) {+ Some((keycode, needs_shift)) => tap_steps(keycode, needs_shift, shift, &mut steps),+ None => missing.push(ch),+ }+ }+ (steps, missing)+}++/// Plan the evdev transitions that type `text` under `mapping`, resolving+/// characters missing from the keymap through `remap` (a spare-keycode+/// allocator) INLINE, one character at a time, so a mixed-script string+/// (an in-map prefix, a character needing remap, more in-map text) keeps its+/// original order in `steps` — remapping a character never defers it behind+/// the characters that came after it in `text`. `remap` returns `None` when+/// the character could neither be found in the keymap nor hosted on a spare+/// keycode; such characters are returned in the second tuple element.+pub(super) fn plan_text_with_fallback<F>(+ mapping: &GetKeyboardMappingReply,+ shift_x_keycode: Option<u8>,+ text: &str,+ mut remap: F,+) -> (Vec<KeyStep>, Vec<char>)+where+ F: FnMut(char) -> Option<u8>,+{+ let shift = shift_x_keycode.and_then(evdev_code_for_x_keycode);+ let mut steps = Vec::with_capacity(text.len() * 2);+ let mut skipped = Vec::new();+ for ch in text.chars() {+ match char_to_keycode_shift(mapping, keysym_for_char(ch)) {+ Some((keycode, needs_shift)) => tap_steps(keycode, needs_shift, shift, &mut steps),+ None => match remap(ch) {+ Some(keycode) => tap_steps(keycode, false, shift, &mut steps),+ None => skipped.push(ch),+ },+ }+ }+ (steps, skipped)+}++/// Plan a chord: modifiers pressed in order, the key tapped (with Shift when+/// its keysym is shifted and Shift was not requested), modifiers released in+/// reverse order — the same ordering physical input produces.+pub(super) fn plan_chord(+ modifier_x_keycodes: &[u8],+ key_x_keycode: u8,+ key_needs_shift: bool,+ shift_x_keycode: Option<u8>,+) -> Vec<KeyStep> {+ let modifiers: Vec<u16> = modifier_x_keycodes+ .iter()+ .filter_map(|&keycode| evdev_code_for_x_keycode(keycode))+ .collect();+ let shift = shift_x_keycode.and_then(evdev_code_for_x_keycode);+ let shift_requested = shift.is_some_and(|shift| modifiers.contains(&shift));+ let mut steps = Vec::with_capacity(modifiers.len() * 2 + 4);+ for &code in &modifiers {+ steps.push(KeyStep {+ evdev_code: code,+ press: true,+ });+ }+ tap_steps(+ key_x_keycode,+ key_needs_shift && !shift_requested,+ shift,+ &mut steps,+ );+ for &code in modifiers.iter().rev() {+ steps.push(KeyStep {+ evdev_code: code,+ press: false,+ });+ }+ steps+}++/// What the virtual-keyboard route could observe about its own delivery.+/// Nothing here reads the application's state back (that stays the caller's+/// screenshot/AT-SPI job); it only says the events went where they were aimed.+#[derive(Clone, Debug, PartialEq, Eq)]+pub struct KeyboardDeliveryReport {+ /// The target window still owned the virtual master keyboard's focus+ /// after the last key event.+ pub virtual_focus_held: bool,+ /// `XGetInputFocus` (the user's core focus) was identical before and after.+ pub core_focus_unchanged: bool,+ /// The server reported the last emitted key transition as a raw event.+ pub delivery_confirmed: bool,+ /// Key transitions emitted through the uinput slave.+ pub key_events: usize,+ /// Characters that have no keycode in the current keymap and could not be+ /// hosted on a spare keycode either.+ pub skipped_characters: Vec<char>,+ /// What the background focus guard observed and restored around the+ /// delivery (attached by the tool layer; `None` when it did not run).+ pub focus_guard: Option<super::focus_guard::FocusGuardReport>,+ /// Stuck input the virtual master still held from an earlier aborted+ /// action and that was released before this delivery (`button1`,+ /// `modifiers(...)`); see `release_stuck_virtual_input`.+ pub released_stuck: Vec<String>,+ /// Route name: [`MPX_UINPUT_PATH`], or [`super::XTEST_CORE_GRAB_PATH`]+ /// when the target's own popup held the keyboard grab and the keys went+ /// through the core keyboard instead.+ pub path: &'static str,+ /// The popup whose grab chose the core-keyboard route.+ pub grab_popup: Option<super::PopupWindow>,+}++impl KeyboardDeliveryReport {+ pub fn to_json(&self) -> serde_json::Value {+ let mut json = serde_json::json!({+ "path": self.path,+ "virtual_focus_held": self.virtual_focus_held,+ "core_focus_unchanged": self.core_focus_unchanged,+ "delivery_confirmed": self.delivery_confirmed,+ "key_events": self.key_events,+ "skipped_characters": self.skipped_characters.iter().collect::<String>(),+ });+ if let Some(guard) = &self.focus_guard {+ for (key, value) in guard.to_json().as_object().into_iter().flatten() {+ json[key] = value.clone();+ }+ if let Some(item) = guard.evidence_item() {+ json["evidence"] = serde_json::json!([item]);+ }+ }+ if !self.released_stuck.is_empty() {+ json["released_stuck_input"] = serde_json::json!(self.released_stuck);+ }+ if let Some(popup) = &self.grab_popup {+ json["grab_popup"] = popup.to_json();+ }+ json+ }++ /// How the keys travelled, for the tool text: the virtual master+ /// keyboard, or the core keyboard under the target's own popup grab.+ pub fn route_phrase(&self) -> String {+ match &self.grab_popup {+ Some(popup) => format!(+ "on the core keyboard, which the target's own {} holds grabbed (path={}; \+ core focus and active window verified {})",+ popup.describe(),+ self.path,+ if self.core_focus_unchanged {+ "unchanged"+ } else {+ "CHANGED — check with a screenshot"+ }+ ),+ None => format!(+ "through a virtual master keyboard (path={}, focus untouched)",+ self.path+ ),+ }+ }++ /// Sentences for the tool text: any focus change the guard saw, and any+ /// stuck input released before delivery (empty when nothing to say).+ pub fn delivery_notes(&self) -> String {+ let mut notes = self+ .focus_guard+ .as_ref()+ .map(|guard| guard.summary())+ .unwrap_or_default();+ if !self.released_stuck.is_empty() {+ notes.push_str(&format!(+ " Released stuck virtual input left by an earlier action before delivering: {}.",+ self.released_stuck.join(", ")+ ));+ }+ notes+ }+}++fn create_uinput_keyboard(name: &str) -> Result<VirtualDevice> {+ guarded_uinput_creation(name, |name| {+ let mut keys = AttributeSet::<Key>::new();+ // Every KEY_* code the X keycode range can address, so any keycode the+ // server's keymap resolves to can be emitted, including the modifiers.+ for code in 1..=MAX_EVDEV_KEY {+ keys.insert(Key::new(code));+ }+ Ok(evdev::uinput::VirtualDeviceBuilder::new()?+ .name(name)+ .with_keys(&keys)?+ .build()?)+ })+}++/// Make sure the session's master pair carries an attached uinput keyboard+/// slave; creates the pair (pointer + keyboard master, uinput pointer) through+/// `ensure_master_pointer` first when needed.+fn ensure_master_keyboard(+ cursor_id: &str,+) -> Result<(MasterPointerIds, Arc<Mutex<VirtualDevice>>)> {+ let ids = ensure_master_pointer(cursor_id)?;+ if let (Some(_), Some(device)) = (+ ids.slave_keyboard_id,+ uinput_keyboards().lock().unwrap().get(cursor_id).cloned(),+ ) {+ return Ok((ids, device));+ }++ let display = open_display()?;+ let result = (|| -> Result<(MasterPointerIds, Arc<Mutex<VirtualDevice>>)> {+ // The master keyboard's device name is "<base> keyboard"; recover the+ // base to derive the slave's name so the reaper can pair them by eye.+ let base = xi2_query_devices(display)?+ .into_iter()+ .find(|(id, _, _)| *id == ids.keyboard_id)+ .and_then(|(_, _, name)| name.strip_suffix(" keyboard").map(str::to_owned))+ .ok_or_else(|| anyhow!("master keyboard {} vanished", ids.keyboard_id))?;+ let device_name = slave_keyboard_name(&base);+ let mut device = create_uinput_keyboard(&device_name)?;+ let slave_keyboard_id = wait_for_slave_id(+ display,+ &device_name,+ x11::xinput2::XISlaveKeyboard,+ Duration::from_secs(5),+ )?;+ attach_slave_to_master(display, slave_keyboard_id, ids.keyboard_id)?;+ warm_up_keyboard(display, &mut device, slave_keyboard_id, ids.keyboard_id);+ let ids = MasterPointerIds {+ slave_keyboard_id: Some(slave_keyboard_id),+ ..ids+ };+ mpx_pointers()+ .lock()+ .unwrap()+ .insert(cursor_id.to_owned(), ids);+ let device = Arc::new(Mutex::new(device));+ uinput_keyboards()+ .lock()+ .unwrap()+ .insert(cursor_id.to_owned(), device.clone());+ Ok((ids, device))+ })();+ unsafe { x11::xlib::XCloseDisplay(display) };+ result+}++/// evdev `KEY_UNKNOWN`: X keycode 248, `NoSymbol` in every evdev keymap, so+/// it types nothing anywhere.+const WARM_UP_EVDEV_CODE: u16 = 240;++/// The first key events of a freshly hot-plugged keyboard were observed to+/// vanish (the master switches to the slave's keymap on its first event and+/// clients only learn about the new master from the hierarchy event queued+/// ahead of the keys). Tap a symbol-less key through the whole pipeline and+/// wait until the server reports it back, so the real text starts on a warm+/// device. Best effort: a failure here only costs the warm-up.+fn warm_up_keyboard(+ display: *mut x11::xlib::Display,+ device: &mut VirtualDevice,+ slave_keyboard_id: i32,+ master_keyboard_id: i32,+) {+ let Some(xi_opcode) = xinput_opcode(display) else {+ return;+ };+ if select_raw_key_events(display, slave_keyboard_id).is_err() {+ return;+ }+ let steps = [+ KeyStep {+ evdev_code: WARM_UP_EVDEV_CODE,+ press: true,+ },+ KeyStep {+ evdev_code: WARM_UP_EVDEV_CODE,+ press: false,+ },+ ];+ for step in steps {+ let _ = device.emit(&[InputEvent::new(+ EventType::KEY,+ step.evdev_code,+ if step.press { 1 } else { 0 },+ )]);+ sleep(Duration::from_millis(KEY_DELAY_MS));+ }+ let seen = wait_for_raw_key(+ display,+ xi_opcode,+ &[slave_keyboard_id, master_keyboard_id],+ steps[1],+ Duration::from_millis(1500),+ );+ if !seen {+ tracing::warn!("virtual keyboard warm-up: server did not report the probe key");+ }+ // Give clients a beat to process the hierarchy/keymap change.+ sleep(Duration::from_millis(120));+}++/// Shift keycode from the server's modifier map (modifier index 0).+fn shift_keycode(conn: &RustConnection) -> Option<u8> {+ let modmap = conn.get_modifier_mapping().ok()?.reply().ok()?;+ let per = modmap.keycodes_per_modifier() as usize;+ modmap+ .keycodes+ .get(..per)+ .and_then(|shift| shift.iter().copied().find(|&keycode| keycode != 0))+}++fn xi_get_focus(display: *mut x11::xlib::Display, device_id: i32) -> Option<x11::xlib::Window> {+ let mut focus: x11::xlib::Window = 0;+ let rc = unsafe { x11::xinput2::XIGetFocus(display, device_id, &mut focus) };+ (rc == 0).then_some(focus)+}++fn core_focus(display: *mut x11::xlib::Display) -> x11::xlib::Window {+ let mut focus: x11::xlib::Window = 0;+ let mut revert_to = 0;+ unsafe { x11::xlib::XGetInputFocus(display, &mut focus, &mut revert_to) };+ focus+}++/// Select raw key events on the root window for `device_id` so the server's+/// processing of our own key transitions can be observed on `display`.+fn select_raw_key_events(display: *mut x11::xlib::Display, device_id: i32) -> Result<()> {+ let root = unsafe { x11::xlib::XDefaultRootWindow(display) };+ let mut mask_bits = vec![0u8; xi_mask_len()];+ x11::xinput2::XISetMask(&mut mask_bits, x11::xinput2::XI_RawKeyPress);+ x11::xinput2::XISetMask(&mut mask_bits, x11::xinput2::XI_RawKeyRelease);+ let mut evmask = x11::xinput2::XIEventMask {+ deviceid: device_id,+ mask_len: mask_bits.len() as std::os::raw::c_int,+ mask: mask_bits.as_mut_ptr(),+ };+ let rc = unsafe { x11::xinput2::XISelectEvents(display, root, &mut evmask, 1) };+ unsafe { x11::xlib::XSync(display, 0) };+ if rc != 0 {+ bail!("XISelectEvents(raw keys) failed with status {rc}");+ }+ Ok(())+}++/// Block (bounded) until the server reports `last` as a raw key event from+/// one of `device_ids`. Returns false on timeout.+fn wait_for_raw_key(+ display: *mut x11::xlib::Display,+ xi_opcode: std::os::raw::c_int,+ device_ids: &[i32],+ last: KeyStep,+ timeout: Duration,+) -> bool {+ let want_type = if last.press {+ x11::xinput2::XI_RawKeyPress+ } else {+ x11::xinput2::XI_RawKeyRelease+ };+ let deadline = std::time::Instant::now() + timeout;+ while std::time::Instant::now() < deadline {+ if unsafe { x11::xlib::XPending(display) } == 0 {+ sleep(Duration::from_millis(2));+ continue;+ }+ let mut ev: x11::xlib::XEvent = unsafe { std::mem::zeroed() };+ unsafe { x11::xlib::XNextEvent(display, &mut ev) };+ if unsafe { ev.type_ } != x11::xlib::GenericEvent {+ continue;+ }+ let mut cookie = unsafe { ev.generic_event_cookie };+ if cookie.extension != xi_opcode+ || !matches!(+ cookie.evtype,+ x11::xinput2::XI_RawKeyPress | x11::xinput2::XI_RawKeyRelease+ )+ {+ continue;+ }+ if unsafe { x11::xlib::XGetEventData(display, &mut cookie) } == 0 {+ continue;+ }+ let raw = cookie.data as *const x11::xinput2::XIRawEvent;+ let matched = !raw.is_null() && {+ let raw = unsafe { &*raw };+ cookie.evtype == want_type+ && raw.detail == i32::from(last.x_keycode())+ && (device_ids.contains(&raw.deviceid) || device_ids.contains(&raw.sourceid))+ };+ unsafe { x11::xlib::XFreeEventData(display, &mut cookie) };+ if matched {+ return true;+ }+ }+ false+}++/// Everything the planners need from the server's keymap, resolved once per+/// call on a short-lived x11rb connection (the same helpers the XSendEvent and+/// XTest routes use, so keycode choice is identical across routes).+struct Keymap {+ conn: RustConnection,+ mapping: GetKeyboardMappingReply,+ shift: Option<u8>,+}++impl Keymap {+ fn load() -> Result<Self> {+ let (conn, _) = connect_x11_for_input()?;+ let mapping = conn.get_keyboard_mapping(8, 248)?.reply()?;+ let shift = shift_keycode(&conn);+ Ok(Self {+ conn,+ mapping,+ shift,+ })+ }+}++/// Deliver `steps` to `target_window` through the session's virtual master+/// keyboard. `remap_guards` keeps any spare-keycode remaps alive until the+/// server has processed the events.+fn deliver(+ cursor_id: &str,+ target_window: u64,+ steps: &[KeyStep],+ skipped_characters: Vec<char>,+ remap_guards: Vec<RemappedKeycode<'_>>,+ keymap_conn: &RustConnection,+) -> Result<KeyboardDeliveryReport> {+ let _op = mpx_op_guard(cursor_id);+ let display = open_display()?;+ let result = (|| -> Result<KeyboardDeliveryReport> {+ supports_parallel_pointer_injection(display)?;+ let mut major = 2;+ let mut minor = 3;+ let rc = unsafe { x11::xinput2::XIQueryVersion(display, &mut major, &mut minor) };+ if rc != 0 {+ bail!("XIQueryVersion failed with status {rc}");+ }+ let xi_opcode = xinput_opcode(display)+ .ok_or_else(|| anyhow!("virtual keyboard delivery requires XInput2"))?;+ let (ids, device) = ensure_master_keyboard(cursor_id)?;+ let slave_keyboard_id = ids+ .slave_keyboard_id+ .ok_or_else(|| anyhow!("uinput keyboard for '{cursor_id}' is not attached"))?;++ let window = target_window as x11::xlib::Window;+ let core_before = core_focus(display);+ let released_stuck = release_stuck_virtual_input(cursor_id, display, ids, &device);+ select_raw_key_events(display, slave_keyboard_id)?;+ // A BadWindow/BadMatch from XISetFocus (target unmapped meanwhile) must+ // surface as an error, not take the daemon down through Xlib's default+ // handler.+ let previous_handler = unsafe { x11::xlib::XSetErrorHandler(Some(ignore_x_error)) };+ // XISetFocus reports async BadWindow/BadMatch (window not viewable)+ // through the error handler, so a stale GetFocus is the real signal.+ // Retry once — a freshly mapped/raised toplevel can miss the first set.+ let mut focus_taken = false;+ for attempt in 0..2 {+ let rc = unsafe {+ x11::xinput2::XISetFocus(display, ids.keyboard_id, window, x11::xlib::CurrentTime)+ };+ unsafe { x11::xlib::XSync(display, 0) };+ if rc == 0 && xi_get_focus(display, ids.keyboard_id) == Some(window) {+ focus_taken = true;+ break;+ }+ if attempt == 0 {+ sleep(Duration::from_millis(30));+ }+ }+ unsafe { x11::xlib::XSetErrorHandler(previous_handler) };+ if !focus_taken {+ bail!(+ "virtual master keyboard could not take focus on window 0x{target_window:x} \+ (not viewable, or another client owns the master focus)"+ );+ }++ let mut last = None;+ let mut held: Vec<u16> = Vec::new();+ for step in steps {+ let emitted = {+ let mut device = device.lock().unwrap();+ device.emit(&[InputEvent::new(+ EventType::KEY,+ step.evdev_code,+ if step.press { 1 } else { 0 },+ )])+ };+ if let Err(error) = emitted {+ // Never abort with keys down: a held modifier would chord+ // every later key of this session.+ let mut device = device.lock().unwrap();+ for code in held.iter().rev() {+ let _ = device.emit(&[InputEvent::new(EventType::KEY, *code, 0)]);+ }+ return Err(error.into());+ }+ if step.press {+ held.push(step.evdev_code);+ } else {+ held.retain(|code| *code != step.evdev_code);+ }+ last = Some(*step);+ sleep(Duration::from_millis(KEY_DELAY_MS));+ }+ let delivery_confirmed = match last {+ Some(last) => wait_for_raw_key(+ display,+ xi_opcode,+ &[slave_keyboard_id, ids.keyboard_id],+ last,+ DELIVERY_CONFIRM_TIMEOUT,+ ),+ None => true,+ };+ // The raw event proves the server consumed the transition; give the+ // toolkit a beat to translate it under any temporary remap before the+ // guards restore the keymap and the slave disappears.+ sleep(Duration::from_millis(30));+ let _ = keymap_conn.get_input_focus()?.reply();+ let virtual_focus_held = xi_get_focus(display, ids.keyboard_id) == Some(window);+ let core_focus_unchanged = core_focus(display) == core_before;+ if !delivery_confirmed {+ tracing::warn!(+ cursor_id,+ "virtual keyboard: server did not report the last key event within {:?}",+ DELIVERY_CONFIRM_TIMEOUT+ );+ }+ Ok(KeyboardDeliveryReport {+ virtual_focus_held,+ core_focus_unchanged,+ delivery_confirmed,+ key_events: steps.len(),+ skipped_characters,+ focus_guard: None,+ released_stuck,+ path: MPX_UINPUT_PATH,+ grab_popup: None,+ })+ })();+ drop(remap_guards);+ // The master pair is retained for reuse. Per-call teardown would hot-plug+ // a uinput keyboard and churn the XInput hierarchy on every keystroke+ // batch — enough to crash LibreOffice VCL. The pair is cleaned up on+ // end_session / idle / startup reap instead.+ let _ = cursor_id;+ unsafe { x11::xlib::XCloseDisplay(display) };+ result+}++/// Type `text` into `target_window` without touching the core focus or the+/// user's pointer. See the module docs for the mechanism.+pub fn send_virtual_keyboard_text(+ cursor_id: &str,+ target_window: u64,+ text: &str,+) -> Result<KeyboardDeliveryReport> {+ let keymap = Keymap::load()?;+ // Characters absent from the keymap (a sparse layout, or a non-Latin+ // glyph) are hosted on spare keycodes, xdotool-style, with a guard that+ // restores the map after delivery. plan_text_with_fallback resolves each+ // character in ORIGINAL text order — remapping a character never defers+ // it behind the characters that came after it, or a mixed-script string+ // (e.g. an ASCII prefix, a character needing remap, then more ASCII)+ // would come out with all the in-map characters first, reordering the+ // typed text relative to what was asked for.+ let mut guards = Vec::new();+ let (steps, skipped) =+ plan_text_with_fallback(+ &keymap.mapping,+ keymap.shift,+ text,+ |ch| match remap_spare_keycode(&keymap.conn, &keymap.mapping, keysym_for_char(ch)) {+ Ok(guard) => {+ let keycode = guard.keycode;+ guards.push(guard);+ Some(keycode)+ }+ Err(_) => None,+ },+ );+ deliver(+ cursor_id,+ target_window,+ &steps,+ skipped,+ guards,+ &keymap.conn,+ )+}++/// Press `key` with `modifiers` held, into `target_window`, without touching+/// the core focus or the user's pointer.+pub fn send_virtual_keyboard_key(+ cursor_id: &str,+ target_window: u64,+ key: &str,+ modifiers: &[&str],+) -> Result<KeyboardDeliveryReport> {+ let keymap = Keymap::load()?;+ let mut guards = Vec::new();+ let mut modifier_keycodes = Vec::with_capacity(modifiers.len());+ for modifier in modifiers {+ let keysym = key_name_to_keysym(modifier)?;+ let (keycode, guard) = keycode_for_keysym(&keymap.conn, &keymap.mapping, keysym, modifier)?;+ guards.extend(guard);+ modifier_keycodes.push(keycode);+ }+ let keysym = key_name_to_keysym(key)?;+ let (keycode, needs_shift) = match char_to_keycode_shift(&keymap.mapping, keysym) {+ Some(found) => found,+ None => {+ let (keycode, guard) = keycode_for_keysym(&keymap.conn, &keymap.mapping, keysym, key)?;+ guards.extend(guard);+ (keycode, false)+ }+ };+ let steps = plan_chord(&modifier_keycodes, keycode, needs_shift, keymap.shift);+ deliver(+ cursor_id,+ target_window,+ &steps,+ Vec::new(),+ guards,+ &keymap.conn,+ )+}++#[cfg(test)]+mod tests {+ use super::*;++ /// A tiny US-like keymap: keycode 38 = a/A, 10 = 1/!, 36 = Return,+ /// 50 = Shift_L, 37 = Control_L, 64 = Alt_L, 250 = spare (NoSymbol).+ fn mapping() -> GetKeyboardMappingReply {+ let per = 2usize;+ let mut keysyms = vec![0u32; (248 - 8 + 1) * per];+ let mut set = |keycode: usize, syms: [u32; 2]| {+ let idx = (keycode - 8) * per;+ keysyms[idx..idx + per].copy_from_slice(&syms);+ };+ set(38, [0x61, 0x41]);+ set(10, [0x31, 0x21]);+ set(36, [0xff0d, 0xff0d]);+ set(50, [0xffe1, 0xffe1]);+ set(37, [0xffe3, 0xffe3]);+ set(64, [0xffe9, 0xffe9]);+ GetKeyboardMappingReply {+ keysyms_per_keycode: per as u8,+ sequence: 0,+ keysyms,+ }+ }++ fn step(evdev_code: u16, press: bool) -> KeyStep {+ KeyStep { evdev_code, press }+ }++ #[test]+ fn x_keycodes_map_to_evdev_by_subtracting_eight() {+ assert_eq!(evdev_code_for_x_keycode(38), Some(30)); // KEY_A+ assert_eq!(evdev_code_for_x_keycode(50), Some(42)); // KEY_LEFTSHIFT+ assert_eq!(evdev_code_for_x_keycode(37), Some(29)); // KEY_LEFTCTRL+ assert_eq!(evdev_code_for_x_keycode(8), Some(0));+ assert_eq!(evdev_code_for_x_keycode(7), None);+ assert_eq!(evdev_code_for_x_keycode(0), None);+ assert_eq!(step(30, true).x_keycode(), 38);+ }++ #[test]+ fn characters_resolve_to_latin1_unicode_or_named_keysyms() {+ assert_eq!(keysym_for_char('a'), 0x61);+ assert_eq!(keysym_for_char('é'), 0xe9);+ assert_eq!(keysym_for_char('€'), 0x0100_0000 | 0x20ac);+ assert_eq!(keysym_for_char('\n'), 0xff0d);+ assert_eq!(keysym_for_char('\t'), 0xff09);+ }++ #[test]+ fn text_plan_taps_keys_and_wraps_shifted_glyphs_in_shift() {+ let (steps, missing) = plan_text(&mapping(), Some(50), "a!\n");+ assert!(missing.is_empty());+ assert_eq!(+ steps,+ vec![+ step(30, true),+ step(30, false),+ step(42, true),+ step(2, true),+ step(2, false),+ step(42, false),+ step(28, true),+ step(28, false),+ ]+ );+ }++ #[test]+ fn text_plan_reports_characters_the_keymap_lacks() {+ let (steps, missing) = plan_text(&mapping(), Some(50), "aé€");+ assert_eq!(steps.len(), 2);+ assert_eq!(missing, vec!['é', '€']);+ }++ #[test]+ fn text_plan_without_a_shift_keycode_types_the_unshifted_level() {+ // No Shift in the modifier map: the keycode is still tapped (the+ // server types the base glyph) rather than the character being lost.+ let (steps, _) = plan_text(&mapping(), None, "A");+ assert_eq!(steps, vec![step(30, true), step(30, false)]);+ }++ #[test]+ fn text_plan_with_fallback_preserves_original_character_order() {+ // "a" + a char requiring remap (é, missing from this tiny keymap) ++ // "1" — a stand-in for a mixed-script string where an ASCII prefix+ // is followed by a character needing a spare-keycode remap and then+ // more ASCII. The remapped character must land in the MIDDLE of the+ // key-event sequence, not be deferred to the end.+ const REMAPPED_KEYCODE: u8 = 250; // spare, per the mapping() doc comment+ let (steps, skipped) = plan_text_with_fallback(&mapping(), Some(50), "aé1", |ch| {+ assert_eq!(+ ch, 'é',+ "only the out-of-keymap character should hit the fallback"+ );+ Some(REMAPPED_KEYCODE)+ });+ assert!(skipped.is_empty());+ let remapped_evdev = evdev_code_for_x_keycode(REMAPPED_KEYCODE).unwrap();+ assert_eq!(+ steps,+ vec![+ // 'a'+ step(30, true),+ step(30, false),+ // 'é' via the remapped spare keycode — must come BEFORE '1',+ // matching its position in the original string.+ step(remapped_evdev, true),+ step(remapped_evdev, false),+ // '1'+ step(2, true),+ step(2, false),+ ]+ );+ }++ #[test]+ fn text_plan_with_fallback_reports_characters_the_remap_also_misses() {+ let (steps, skipped) = plan_text_with_fallback(&mapping(), Some(50), "a€a", |_| None);+ assert_eq!(steps.len(), 4); // both 'a's still tap+ assert_eq!(skipped, vec!['€']);+ }++ #[test]+ fn chord_plan_nests_modifiers_around_the_key_in_physical_order() {+ // ctrl+alt+a+ assert_eq!(+ plan_chord(&[37, 64], 38, false, Some(50)),+ vec![+ step(29, true),+ step(56, true),+ step(30, true),+ step(30, false),+ step(56, false),+ step(29, false),+ ]+ );+ }++ #[test]+ fn chord_plan_adds_shift_only_when_the_key_is_shifted_and_shift_not_held() {+ // ctrl+A where 'A' is the shifted level of keycode 38: auto-Shift.+ assert_eq!(+ plan_chord(&[37], 38, true, Some(50)),+ vec![+ step(29, true),+ step(42, true),+ step(30, true),+ step(30, false),+ step(42, false),+ step(29, false),+ ]+ );+ // ctrl+shift+A: Shift already requested, never doubled.+ assert_eq!(+ plan_chord(&[37, 50], 38, true, Some(50)),+ vec![+ step(29, true),+ step(42, true),+ step(30, true),+ step(30, false),+ step(42, false),+ step(29, false),+ ]+ );+ }++ #[test]+ fn report_json_names_the_route() {+ let report = KeyboardDeliveryReport {+ virtual_focus_held: true,+ core_focus_unchanged: true,+ delivery_confirmed: false,+ key_events: 4,+ skipped_characters: vec!['€'],+ focus_guard: None,+ released_stuck: vec!["button1".into()],+ path: MPX_UINPUT_PATH,+ grab_popup: None,+ };+ let json = report.to_json();+ assert_eq!(json["path"], MPX_UINPUT_PATH);+ assert_eq!(json["key_events"], 4);+ assert_eq!(json["delivery_confirmed"], false);+ assert_eq!(json["skipped_characters"], "€");+ assert_eq!(json["released_stuck_input"][0], "button1");+ assert!(report.delivery_notes().contains("button1"));+ }++ #[test]+ fn slave_keyboard_name_fits_the_uinput_limit_for_versioned_masters() {+ let owner = mpx_owner::Owner::current().expect("procfs identity");+ let name = slave_keyboard_name(&owner.master_name(u64::MAX));+ assert!(name.len() <= EVDEV_UINPUT_NAME_MAX_BYTES, "{name}");+ assert!(name.ends_with(UINPUT_KEYBOARD_SUFFIX));+ }+}diff --git a/libs/cua-driver/rust/crates/platform-linux/src/installed_apps.rs b/libs/cua-driver/rust/crates/platform-linux/src/installed_apps.rsindex 4268e0ad8f..03dc4732c7 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/installed_apps.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/installed_apps.rs@@ -25,6 +25,34 @@ pub struct InstalledApp {/// RFC3339 timestamp from the `.desktop` file's filesystem mtime, or/// `None` if the metadata could not be read.pub last_used: Option<String>,+ /// `StartupWMClass=`: the WM_CLASS the entry's windows carry when it+ /// differs from the launcher (`libreoffice --calc` → `libreoffice-calc`).+ pub startup_wm_class: Option<String>,+}++impl InstalledApp {+ /// Whether a top-level window with this WM_CLASS belongs to the app.+ ///+ /// Mirrors the shell's own window-to-app matching: `StartupWMClass`+ /// first, then the desktop file id (`org.gnome.Nautilus`), its last+ /// segment (`Nautilus`), and the launcher basename (`gnome-terminal`).+ pub fn owns_window_class(&self, wm_class: &str) -> bool {+ if wm_class.is_empty() {+ return false;+ }+ let id_stem = self.bundle_id.rsplit('.').next().unwrap_or(&self.bundle_id);+ let exec = self+ .launch_path+ .split_whitespace()+ .next()+ .and_then(|token| token.rsplit('/').next())+ .unwrap_or("");+ self.startup_wm_class+ .as_deref()+ .into_iter()+ .chain([self.bundle_id.as_str(), id_stem, exec])+ .any(|key| !key.is_empty() && key.eq_ignore_ascii_case(wm_class))+ }}/// Return every visible application installed on the system.@@ -154,6 +182,7 @@ fn parse_desktop_file(path: &Path, bundle_id: &str) -> Option<InstalledApp> {bundle_id,launch_path,last_used,+ startup_wm_class: string_key(&entry, "StartupWMClass").filter(|class| !class.is_empty()),})}@@ -314,6 +343,29 @@ Exec=/opt/demo/bin/demo %Ulet _ = std::fs::remove_file(&path);}+ #[test]+ fn owns_window_class_by_startup_wm_class_id_or_launcher() {+ let entry = |bundle_id: &str, launch_path: &str, wm_class: Option<&str>| InstalledApp {+ name: "x".into(),+ bundle_id: bundle_id.into(),+ launch_path: launch_path.into(),+ last_used: None,+ startup_wm_class: wm_class.map(str::to_owned),+ };+ let calc = entry("libreoffice-calc", "libreoffice", Some("libreoffice-calc"));+ assert!(calc.owns_window_class("libreoffice-calc"));+ assert!(!calc.owns_window_class("libreoffice-writer"));+ assert!(!calc.owns_window_class(""));+ let chrome = entry("google-chrome", "chrome-stable", Some("Google-chrome"));+ assert!(chrome.owns_window_class("google-chrome"));+ let files = entry("org.gnome.Nautilus", "nautilus --new-window", None);+ assert!(files.owns_window_class("Org.gnome.Nautilus"));+ assert!(files.owns_window_class("nautilus"));+ let terminal = entry("org.gnome.Terminal", "gnome-terminal", None);+ assert!(terminal.owns_window_class("Gnome-terminal"));+ assert!(!terminal.owns_window_class("Terminal-server"));+ }+#[test]fn desktop_file_id_flat() {let root = Path::new("/usr/share/applications");diff --git a/libs/cua-driver/rust/crates/platform-linux/src/overlay.rs b/libs/cua-driver/rust/crates/platform-linux/src/overlay.rsindex 69262af890..59903f1754 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/overlay.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/overlay.rs@@ -82,6 +82,7 @@ fn arrival_fire(key: &CursorKey) {if let Some(map) = guard.as_mut() {if let Some(tx) = map.remove(key) {let _ = tx.send(());+ ARRIVAL_DEGRADED.store(false, std::sync::atomic::Ordering::Relaxed);}}}@@ -581,9 +582,39 @@ pub async fn animate_cursor_to_for(key: CursorKey, x: f64, y: f64) {return;}- let _ = rx.await;+ if ARRIVAL_DEGRADED.load(std::sync::atomic::Ordering::Relaxed) {+ // The renderer already failed to report one arrival. Keep the glide+ // fire-and-forget until it proves itself again rather than charging+ // every action the full cap.+ arrival_cancel(&key);+ return;+ }+ match tokio::time::timeout(ARRIVAL_WAIT_CAP, rx).await {+ Ok(_) => {}+ Err(_elapsed) => {+ arrival_cancel(&key);+ ARRIVAL_DEGRADED.store(true, std::sync::atomic::Ordering::Relaxed);+ tracing::warn!(+ key = %key,+ cap_ms = ARRIVAL_WAIT_CAP.as_millis() as u64,+ "overlay: cursor glide did not report arrival in time; continuing without waiting (further glides are fire-and-forget until the renderer reports an arrival again)"+ );+ }+ }}+/// Upper bound on how long an input action waits for its agent-cursor glide+/// to land. The glide is cosmetic: a renderer that never reports arrival+/// (deferred paints, a dropped MoveTo, a stalled X11 overlay thread) must not+/// hold the tool call — and, since the stdio transport handles requests one+/// at a time, every later tool call — open indefinitely. Sized above the+/// longest legal glide (`glide_duration_ms` ≤ 5000; speed-based glides cross+/// a 4K diagonal in ≈5 s at 900 px/s).+const ARRIVAL_WAIT_CAP: Duration = Duration::from_millis(5_500);++/// Latched once an arrival wait expired; cleared by the next real arrival.+static ARRIVAL_DEGRADED: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);+pub fn remove_cursor(key: CursorKey) {if key.is_empty() {return;@@ -3308,6 +3339,52 @@ mod tests {));}+ /// Regression: a MoveTo whose arrival the renderer never reports (the+ /// GIMP menu-item element click wedged the whole stdio daemon this way)+ /// must not hold the awaiting tool call open; the wait is capped and the+ /// registration is dropped.+ #[tokio::test(start_paused = true)]+ async fn unreported_arrival_releases_the_waiter_after_the_cap() {+ init(CursorConfig::default());+ let key = "arrival-cap-regression".to_owned();+ {+ let mut guard = RENDER.lock().unwrap();+ let map = guard.get_or_insert_with(default_render_map);+ let template = map.template.clone();+ let rs = map+ .cursors+ .entry(key.clone())+ .or_insert_with(|| render_state_for_key(&template, &key));+ rs.core.cfg.enabled = true;+ rs.core.visible = true;+ rs.core.pos = (10.0, 10.0);+ }+ ARRIVAL_DEGRADED.store(false, std::sync::atomic::Ordering::Relaxed);+ let started = tokio::time::Instant::now();+ // Nobody drains CMD_TX here, so no arrival can ever fire.+ tokio::time::timeout(+ ARRIVAL_WAIT_CAP + Duration::from_secs(5),+ animate_cursor_to_for(key.clone(), 60.0, 60.0),+ )+ .await+ .expect("glide wait must be bounded");+ assert!(started.elapsed() >= ARRIVAL_WAIT_CAP);+ assert!(ARRIVAL_TX+ .lock()+ .unwrap()+ .as_ref()+ .is_none_or(|map| !map.contains_key(&key)));+ // Degraded: the next glide returns without waiting at all.+ let started = tokio::time::Instant::now();+ animate_cursor_to_for(key.clone(), 20.0, 20.0).await;+ assert!(started.elapsed() < Duration::from_millis(100));+ // A real arrival re-arms the wait.+ let (tx, _rx) = tokio::sync::oneshot::channel();+ arrival_register(key.clone(), tx);+ arrival_fire(&key);+ assert!(!ARRIVAL_DEGRADED.load(std::sync::atomic::Ordering::Relaxed));+ }+#[test]fn clearing_arrivals_releases_waiters() {let (tx, mut rx) = tokio::sync::oneshot::channel();diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rsindex c2fbb95488..0ec20ad3b3 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs@@ -28,6 +28,9 @@ fn window_target_candidates_for_pid(.filter(|window| window.pid == Some(pid)).map(|window| WindowTargetCandidate {window_id: window.xid,+ transient_for: (!crate::wayland::is_wayland())+ .then(|| crate::x11::transient_for(window.xid))+ .flatten(),title: window.title,app_name: Some(window.app_name),is_on_screen: window.is_on_screen,@@ -42,14 +45,196 @@ fn pid_window_target_candidates(pid: i64) -> Vec<WindowTargetCandidate> {window_target_candidates_for_pid(crate::wayland::list_windows_dispatch(Some(pid)), pid)}+/// Topmost on-screen window of `pid` covering a desktop-frame point.+fn desktop_point_window_resolver() -> cua_driver_core::window_target::DesktopPointWindowResolver {+ Arc::new(move |pid, x, y| {+ let pid = u32::try_from(pid).ok()?;+ topmost_window_at(&crate::wayland::list_windows_dispatch(Some(pid)), pid, x, y)+ })+}++/// The pid's topmost on-screen window containing screen point `(sx, sy)`:+/// the highest `z_index` (bottom-to-top stacking position) wins, so an+/// app's own dialog beats the main window it covers. Windows without a+/// stacking position rank lowest.+fn topmost_window_at(+ windows: &[crate::x11::WindowInfo],+ pid: u32,+ sx: f64,+ sy: f64,+) -> Option<u64> {+ windows+ .iter()+ .filter(|w| w.pid == Some(pid) && w.is_on_screen && w.width > 0 && w.height > 0)+ .filter(|w| {+ sx >= f64::from(w.x)+ && sy >= f64::from(w.y)+ && sx < f64::from(w.x) + f64::from(w.width)+ && sy < f64::from(w.y) + f64::from(w.height)+ })+ .max_by_key(|w| w.z_index.map(|z| z as i64).unwrap_or(-1))+ .map(|w| w.xid)+}++/// The window a pid-only keyboard action means in a multi-window app: the+/// pid's window that holds the core focus, else its topmost transient dialog,+/// else the WM's active window when it is the pid's, else the largest mapped+/// toplevel (see [`crate::x11::pick_pid_window`]). A dialog the app just+/// opened therefore receives the following `type_text` / `press_key` even+/// while the main window is still the WM's active window.+fn pid_fallback_window_resolver() -> cua_driver_core::window_target::PidFallbackWindowResolver {+ Arc::new(|pid| {+ let pid = u32::try_from(pid).ok()?;+ let windows: Vec<crate::x11::WindowInfo> = crate::wayland::list_windows_dispatch(Some(pid))+ .into_iter()+ .filter(|w| w.pid == Some(pid))+ .collect();+ if crate::wayland::is_wayland() {+ return windows+ .into_iter()+ .filter(|w| w.is_on_screen)+ .max_by_key(|w| w.z_index.unwrap_or(0))+ .map(|w| w.xid);+ }+ let ids: Vec<u64> = windows.iter().map(|w| w.xid).collect();+ crate::x11::pick_pid_window(+ &windows,+ crate::x11::focused_window_among(&ids),+ crate::x11::transient_for,+ crate::x11::active_window(),+ )+ })+}++#[cfg(test)]+mod pid_window_resolver_tests {+ use super::*;++ fn win(xid: u64, z: Option<usize>, on_screen: bool, w: u32, h: u32) -> crate::x11::WindowInfo {+ crate::x11::WindowInfo {+ xid,+ pid: Some(7),+ app_name: "gimp".into(),+ title: format!("w{xid}"),+ is_on_screen: on_screen,+ z_index: z,+ x: 0,+ y: 0,+ width: w,+ height: h,+ }+ }++ #[test]+ fn point_resolver_prefers_the_topmost_window_by_stacking_position() {+ // Main window listed last (as `_NET_CLIENT_LIST_STACKING` bottom-up+ // enumeration does when a dialog sits above it); the dialog wins.+ let dialog = crate::x11::WindowInfo {+ z_index: Some(5),+ ..win(2, None, true, 400, 300)+ };+ let main = crate::x11::WindowInfo {+ z_index: Some(3),+ ..win(1, None, true, 1920, 1080)+ };+ assert_eq!(+ topmost_window_at(&[main.clone(), dialog.clone()], 7, 100.0, 100.0),+ Some(2)+ );+ assert_eq!(topmost_window_at(&[dialog, main], 7, 100.0, 100.0), Some(2));+ }++ #[test]+ fn point_resolver_ignores_windows_without_a_stacking_position_when_one_has_it() {+ let ranked = win(2, Some(0), true, 400, 300);+ let unranked = win(1, None, true, 1920, 1080);+ assert_eq!(+ topmost_window_at(&[ranked, unranked], 7, 10.0, 10.0),+ Some(2)+ );+ }++ #[test]+ fn pid_only_resolver_orders_focus_then_dialog_then_active_then_largest() {+ let main = win(1, Some(1), true, 1920, 1080);+ let dialog = win(2, Some(2), true, 400, 300);+ let dock = win(3, Some(0), true, 200, 900);+ let hidden = win(4, None, false, 3000, 3000);+ let windows = vec![main, dialog, dock, hidden];+ let transient = |xid: u64| (xid == 2).then_some(1u64);+ // 1. core focus inside the pid wins outright (even over a dialog).+ assert_eq!(+ crate::x11::pick_pid_window(&windows, Some(3), transient, Some(1)),+ Some(3)+ );+ // 2. no focus in the pid: its topmost transient dialog.+ assert_eq!(+ crate::x11::pick_pid_window(&windows, None, transient, Some(1)),+ Some(2)+ );+ // 3. no dialog: the WM's active window when it is the pid's.+ assert_eq!(+ crate::x11::pick_pid_window(&windows, None, |_| None, Some(3)),+ Some(3)+ );+ // 4. otherwise the largest mapped toplevel (never the unmapped one).+ assert_eq!(+ crate::x11::pick_pid_window(&windows, None, |_| None, None),+ Some(1)+ );+ assert_eq!(+ crate::x11::pick_pid_window(&windows, None, |_| None, Some(99)),+ Some(1)+ );+ // A focused window that is not the pid's is ignored.+ assert_eq!(+ crate::x11::pick_pid_window(&windows, Some(42), |_| None, None),+ Some(1)+ );+ }++ #[test]+ fn pid_only_resolver_skips_unmapped_dialogs() {+ let main = win(1, Some(1), true, 800, 600);+ let unmapped_dialog = win(2, Some(2), false, 400, 300);+ let windows = vec![main, unmapped_dialog];+ assert_eq!(+ crate::x11::pick_pid_window(&windows, None, |xid| (xid == 2).then_some(1u64), None),+ Some(1)+ );+ }+}++type PidWindowGuardParts = (+ WindowTargetCandidates,+ cua_driver_core::window_target::DesktopPointWindowResolver,+ cua_driver_core::window_target::PidFallbackWindowResolver,+ cua_driver_core::window_target::SnapshotWindowResolver,+);+fn pid_window_guarded<T: Tool + 'static>(tool: T,- candidates: &WindowTargetCandidates,+ (candidates, point_resolver, fallback_resolver, snapshot_resolver): &PidWindowGuardParts,) -> Box<dyn Tool> {- Box::new(PidOnlyWindowTargetGuard::new(- Box::new(tool),- candidates.clone(),- ))+ Box::new(+ PidOnlyWindowTargetGuard::new(Box::new(tool), candidates.clone())+ .with_point_resolver(point_resolver.clone())+ .with_fallback_resolver(fallback_resolver.clone())+ .with_snapshot_resolver(snapshot_resolver.clone()),+ )+}++/// The window a `snapshot_id` was published for (the element cache lane of+/// that pid), so pid-only element actions follow the snapshot to its popup+/// or dialog.+fn snapshot_window_resolver(+ state: Arc<ToolState>,+) -> cua_driver_core::window_target::SnapshotWindowResolver {+ Arc::new(move |pid, handle| {+ let pid = i32::try_from(pid).ok()?;+ let id = cua_driver_core::element_token::parse_snapshot_handle(handle)?;+ state.element_cache.window_for_snapshot(pid, id)+ })}// ── DriverConfig + ZoomRegistry ─────────────────────────────────────────────@@ -103,6 +288,8 @@ pub struct ToolState {pub config: Arc<RwLock<DriverConfig>>,}+impl ToolState {}+#[derive(Clone, Debug)]pub struct MouseHoldState {pub pid: u32,@@ -185,12 +372,15 @@ impl Tool for ListAppsTool {relative to its XDG `applications/` root with the `.desktop` suffix \stripped and path separators replaced with `-` \(e.g. `kde4/konqbrowser.desktop` → `kde4-konqbrowser`).\n\- - last_used: RFC3339 mtime of the `.desktop` file, when readable.\n\n\+ - last_used: RFC3339 mtime of the `.desktop` file, when readable.\n\+ - windows: the app's current top-level windows (same records as list_windows); \+ empty for a process without one.\n\n\Running apps come from `/proc`. Installed apps come from XDG Desktop Entry \files in $XDG_DATA_HOME/applications and each $XDG_DATA_DIRS entry's \applications/ subdir. Entries with `NoDisplay=true` or `Hidden=true` are \filtered. A `.desktop` file whose launcher matches a running process \- (by basename) is merged into a single entry with `running: true`.\n\n\+ (by basename), or whose WM_CLASS a top-level window carries, is merged \+ into a single entry with `running: true`.\n\n\Use this for \"is X installed?\" as well as \"is X running?\". For per-window \state — visibility, geometry, titles — call list_windows instead.".into(),@@ -206,6 +396,24 @@ impl Tool for ListAppsTool {let apps = tokio::task::spawn_blocking(|| -> Vec<serde_json::Value> {let procs = crate::proc_fs::list_processes();let installed = crate::installed_apps::list_installed_apps();+ let windows = crate::wayland::list_windows_dispatch(None);++ // An app whose launcher is not its process (`libreoffice --calc`+ // runs `soffice.bin`, `google-chrome` runs `chrome`, `gnome-terminal`+ // hands off to `gnome-terminal-server`) never matches by+ // basename. Its top-level windows still carry its WM_CLASS, so+ // owning one is what makes it running.+ let by_window: std::collections::HashMap<usize, u32> = installed+ .iter()+ .enumerate()+ .filter_map(|(i, app)| {+ windows+ .iter()+ .find(|w| w.pid.is_some() && app.owns_window_class(&w.app_name))+ .and_then(|w| w.pid)+ .map(|pid| (i, pid))+ })+ .collect();// Match running processes to installed apps by executable// basename (Exec=firefox %u → "firefox"; cmdline /usr/bin/firefox@@ -236,7 +444,14 @@ impl Tool for ListAppsTool {continue;}let candidates = by_exe.get(&basename).map(|v| v.as_slice()).unwrap_or(&[]);- let merged = disambiguate_installed_match(candidates, &installed, key_source);+ let merged = disambiguate_installed_match(candidates, &installed, key_source)+ .or_else(|| {+ by_window+ .iter()+ .filter(|(_, pid)| **pid == p.pid)+ .map(|(idx, _)| *idx)+ .min()+ });if let Some(idx) = merged {consumed.insert(idx);}@@ -263,6 +478,14 @@ impl Tool for ListAppsTool {None,),};+ let owned: Vec<Value> = windows+ .iter()+ .filter(|w| w.pid == Some(p.pid))+ .filter(|w| {+ merged.is_none_or(|idx| installed[idx].owns_window_class(&w.app_name))+ })+ .map(window_record_json)+ .collect();out.push(json!({"pid": p.pid,"bundle_id": bundle_id,@@ -272,23 +495,33 @@ impl Tool for ListAppsTool {"kind": kind,"launch_path": launch_path,"last_used": last_used,- "windows": Vec::<serde_json::Value>::new(),+ "windows": owned,}));}for (i, app) in installed.iter().enumerate() {if consumed.contains(&i) {continue;}+ // One process can own windows of several entries (a single+ // `soffice.bin` showing Calc and Writer): each entry with a+ // window is running under its own id.+ let window_pid = by_window.get(&i).copied();+ let owned: Vec<Value> = windows+ .iter()+ .filter(|w| w.pid.is_some() && w.pid == window_pid)+ .filter(|w| app.owns_window_class(&w.app_name))+ .map(window_record_json)+ .collect();out.push(json!({- "pid": 0,+ "pid": window_pid.unwrap_or(0),"bundle_id": app.bundle_id.clone(),"name": app.name.clone(),- "running": false,+ "running": window_pid.is_some(),"active": false,"kind": "desktop","launch_path": app.launch_path.clone(),"last_used": app.last_used.clone(),- "windows": Vec::<serde_json::Value>::new(),+ "windows": owned,}));}out@@ -436,8 +669,19 @@ impl Tool for ListWindowsTool {let mut lines = vec![format!("Found {} windows:", windows.len())];for w in &windows {lines.push(format!(- " [xid={}] pid={:?} \"{}\" {}x{}+{}+{}",- w.xid, w.pid, w.title, w.width, w.height, w.x, w.y+ " window_id={} pid={} \"{}\" {}x{}+{}+{}{}",+ w.xid,+ w.pid.map(|p| p.to_string()).unwrap_or_else(|| "?".into()),+ w.title,+ w.width,+ w.height,+ w.x,+ w.y,+ if w.app_name.is_empty() {+ String::new()+ } else {+ format!(" app={}", w.app_name)+ }));}let structured =@@ -543,6 +787,65 @@ mod list_windows_tests {// ── get_window_state ─────────────────────────────────────────────────────────+/// The largest window screenshot delivered as-is. The Anthropic API downsizes+/// any image above ~1.15 megapixels before the model sees it, so a 1568x861+/// PNG reached the model as ~1447x795 while the driver still mapped its x/y+/// as 1568x861: every pixel click landed a uniform 0.94x short. Capping here+/// keeps "pixels of THIS screenshot" true for the image the model reads.+const WINDOW_SCREENSHOT_MAX_PIXELS: u64 = 1_150_000;++/// The long edge a `w`x`h` image must shrink to so that it holds at most+/// `max_pixels` pixels; `None` when it already fits.+fn megapixel_long_edge_cap(w: u32, h: u32, max_pixels: u64) -> Option<u32> {+ let pixels = u64::from(w) * u64::from(h);+ if w == 0 || h == 0 || pixels <= max_pixels {+ return None;+ }+ let long = w.max(h) as f64;+ let short = w.min(h) as f64;+ let mut edge = (long * (max_pixels as f64 / pixels as f64).sqrt()).floor() as u32;+ // The resizer rounds both sides; step down until the rounded image fits.+ while edge > 1 {+ let scale = edge as f64 / long;+ let fits = ((long * scale).round() as u64) * ((short * scale).round() as u64) <= max_pixels;+ if fits {+ break;+ }+ edge -= 1;+ }+ Some(edge)+}++#[cfg(test)]+mod megapixel_cap_tests {+ use super::{megapixel_long_edge_cap, WINDOW_SCREENSHOT_MAX_PIXELS};++ #[test]+ fn a_window_shot_above_the_api_threshold_is_capped_below_it() {+ let edge =+ megapixel_long_edge_cap(1568, 861, WINDOW_SCREENSHOT_MAX_PIXELS).expect("capped");+ let scale = edge as f64 / 1568.0;+ let (w, h) = (+ (1568.0 * scale).round() as u64,+ (861.0 * scale).round() as u64,+ );+ assert!(w * h <= WINDOW_SCREENSHOT_MAX_PIXELS, "{w}x{h}");+ assert!(edge >= 1440 && edge < 1568, "{edge}");+ }++ #[test]+ fn a_shot_within_the_threshold_is_left_alone() {+ assert_eq!(+ megapixel_long_edge_cap(1280, 800, WINDOW_SCREENSHOT_MAX_PIXELS),+ None+ );+ assert_eq!(+ megapixel_long_edge_cap(0, 800, WINDOW_SCREENSHOT_MAX_PIXELS),+ None+ );+ }+}+/// Build a single structured element entry for `get_window_state`./// Returns `None` when the node has no `element_index` (non-actionable rows).fn build_element_entry(@@ -552,12 +855,18 @@ fn build_element_entry() -> Option<serde_json::Value> {let idx = n.element_index?;// `label` mirrors what a human reading the markdown row would call this- // element: name first, then value, then description.+ // element: its name, else its description (Qt keeps a button's tooltip+ // there: "Play"/"Pause"). Never its value: four spin buttons all labelled+ // "0.0" cannot be told apart — such a control is `unlabelled` instead,+ // with its place among its siblings in `description`.+ let unlabelled = n.name.is_none()+ && n.description+ .as_deref()+ .is_some_and(|d| d.starts_with(crate::atspi::native::UNLABELLED_NOTE_PREFIX));let label = n.name.clone()- .or_else(|| n.value.clone())- .or_else(|| n.description.clone());+ .or_else(|| n.description.clone().filter(|_| !unlabelled));let mut entry = json!({"element_index": idx,"role": n.role,@@ -572,6 +881,9 @@ fn build_element_entry(if let Some(label) = label {entry["label"] = json!(label);}+ if unlabelled {+ entry["unlabelled"] = json!(true);+ }// Surface the element's value separately from `label` (which collapses// name→value→description): a field with both a name AND typed text would// otherwise hide the text from a caller reading the structured side,@@ -601,9 +913,277 @@ fn build_element_entry(if let Some((x, y, w, h)) = bounds {entry["frame"] = json!({ "x": x, "y": y, "w": w, "h": h });}+ if let Some(description) = n.description.clone().filter(|d| !d.is_empty()) {+ entry["description"] = json!(description);+ }Some(entry)}+/// Roles a popup's rows carry: menu entries, and the list / tree / table+/// rows of a combo list, a completer or a chooser popover.+fn popup_item_role(role: &str) -> bool {+ let role = role.trim().to_ascii_lowercase();+ role.contains("menu item")+ || role == "menu"+ || role.contains("list item")+ || role.contains("tree item")+ || role.contains("table cell")+ || role == "cell"+ || role == "item"+ || role == "option"+}++/// The entries drawn inside a popup of `width`x`height` (frames are+/// popup-local here). When an indexed container (Qt exposes the combo list+/// under the combo box inside the dialog) fills the popup, its subtree is+/// the popup's content; otherwise the item-role elements inside the box.+/// Falls back to the input when nothing matches, so a popup that is not a+/// menu (a GTK popover without its own frame) still returns what the walk+/// found.+fn popup_menu_elements(+ elements: Vec<serde_json::Value>,+ (origin_x, origin_y): (i32, i32),+ width: u32,+ height: u32,+) -> Vec<serde_json::Value> {+ // `frame` is in screen coordinates; compare it with the popup's screen+ // rectangle.+ let frame_inside = |entry: &serde_json::Value| {+ let frame = &entry["frame"];+ let x = frame["x"].as_i64().unwrap_or(i64::MIN) - i64::from(origin_x);+ let y = frame["y"].as_i64().unwrap_or(i64::MIN) - i64::from(origin_y);+ frame.is_object()+ && x >= -2+ && y >= -2+ && x + frame["w"].as_i64().unwrap_or(0) <= width as i64 + 2+ && y + frame["h"].as_i64().unwrap_or(0) <= height as i64 + 2+ };+ let role_of =+ |entry: &serde_json::Value| entry["role"].as_str().unwrap_or("").to_ascii_lowercase();+ // A container whose frame covers most of the popup: the list behind a+ // combo box / completer, the menu behind a context menu.+ let popup_area = u64::from(width) * u64::from(height);+ let container = elements+ .iter()+ .filter(|entry| frame_inside(entry))+ .filter(|entry| {+ let role = role_of(entry);+ matches!(+ role.as_str(),+ "list"+ | "list box"+ | "menu"+ | "tree"+ | "tree table"+ | "table"+ | "panel"+ | "scroll pane"+ | "filler"+ )+ })+ .filter(|entry| {+ let frame = &entry["frame"];+ let area = frame["w"].as_u64().unwrap_or(0) * frame["h"].as_u64().unwrap_or(0);+ popup_area > 0 && area * 10 >= popup_area * 6+ })+ .max_by_key(|entry| {+ let frame = &entry["frame"];+ frame["w"].as_u64().unwrap_or(0) * frame["h"].as_u64().unwrap_or(0)+ })+ .and_then(|entry| entry["element_index"].as_u64());+ if let Some(container) = container {+ let parent_of: std::collections::HashMap<u64, u64> = elements+ .iter()+ .filter_map(|entry| {+ Some((+ entry["element_index"].as_u64()?,+ entry["parent_index"].as_u64()?,+ ))+ })+ .collect();+ let descends = |mut idx: u64| {+ for _ in 0..64 {+ match parent_of.get(&idx) {+ Some(parent) if *parent == container => return true,+ Some(parent) => idx = *parent,+ None => return false,+ }+ }+ false+ };+ let subtree: Vec<serde_json::Value> = elements+ .iter()+ .filter(|entry| {+ entry["element_index"]+ .as_u64()+ .is_some_and(|idx| idx != container && descends(idx))+ })+ .cloned()+ .collect();+ if !subtree.is_empty() {+ return subtree;+ }+ }+ let items: Vec<serde_json::Value> = elements+ .iter()+ .filter(|entry| popup_item_role(&role_of(entry)) && frame_inside(entry))+ .cloned()+ .collect();+ // A popup is either a menu or a list: when list / tree / table rows are+ // drawn inside it, menubar entries whose extents happen to fit the box+ // (Qt reports the main window's menubar items inside a chooser popup's+ // rectangle) are not its content.+ let rows: Vec<serde_json::Value> = items+ .iter()+ .filter(|entry| !role_of(entry).contains("menu"))+ .cloned()+ .collect();+ if !rows.is_empty() {+ rows+ } else if items.is_empty() {+ elements+ } else {+ items+ }+}++/// Elements that have a frame (visible, hit-testable on the screenshot)+/// come first, in walk order; frameless ones follow. A client that caps the+/// result then still shows the document, toolbar and sidebar controls.+fn framed_elements_first(elements: Vec<serde_json::Value>) -> Vec<serde_json::Value> {+ let (framed, frameless): (Vec<_>, Vec<_>) = elements+ .into_iter()+ .partition(|entry| entry.get("frame").is_some());+ framed.into_iter().chain(frameless).collect()+}++/// Same-pid windows mapped over a target window: transient dialogs (with+/// their modality) and override-redirect popups, plus whether any of them+/// overlaps the target's rectangle (then its own drawable is not what the+/// user sees).+#[derive(Default)]+struct WindowOverlays {+ dialogs: Vec<serde_json::Value>,+ popups: Vec<serde_json::Value>,+ covers_window: bool,+ window_rect: Option<(i32, i32, u32, u32)>,+}++impl WindowOverlays {+ /// One sentence per overlay naming the call that acts on it.+ fn follow_up(&self, pid: u32) -> Option<String> {+ let mut parts = Vec::new();+ for dialog in &self.dialogs {+ let id = dialog["window_id"].as_u64().unwrap_or(0);+ let owning_pid = dialog.get("owning_pid").and_then(|v| v.as_u64());+ parts.push(format!(+ "dialog \"{}\" (window_id {id}, transient of window {}{}{}) is open over this \+ window: call get_window_state(pid={pid}, window_id={id}) to index it and act \+ there (pid-only keys go to it).",+ dialog["title"].as_str().unwrap_or(""),+ dialog["transient_for"].as_u64().unwrap_or(0),+ if dialog["modal"].as_bool() == Some(true) {+ ", modal"+ } else {+ ""+ },+ owning_pid+ .map(|owner| format!(+ ", owned by a different process pid {owner}, not pid {pid}"+ ))+ .unwrap_or_default(),+ ));+ }+ for popup in &self.popups {+ let id = popup["window_id"].as_u64().unwrap_or(0);+ parts.push(format!(+ "popup (window_id {id}, bounds x={} y={} {}x{}) is open: call \+ get_window_state(pid={pid}, window_id={id}) to index its items and click them \+ by element_index.",+ popup["bounds"]["x"],+ popup["bounds"]["y"],+ popup["bounds"]["width"],+ popup["bounds"]["height"]+ ));+ }+ (!parts.is_empty()).then(|| parts.join(" "))+ }+}++fn rects_intersect(a: (i32, i32, u32, u32), b: (i32, i32, u32, u32)) -> bool {+ a.0 < b.0 + b.2 as i32+ && a.0 + a.2 as i32 > b.0+ && a.1 < b.1 + b.3 as i32+ && a.1 + a.3 as i32 > b.1+}++fn window_overlays(pid: u32, xid: u64) -> WindowOverlays {+ let mut out = WindowOverlays::default();+ out.window_rect = crate::x11::window_info(xid).map(|w| (w.x, w.y, w.width, w.height));+ let over = |rect: (i32, i32, u32, u32)| {+ out.window_rect+ .is_some_and(|target| rects_intersect(rect, target))+ };+ let mut covers = false;+ for window in crate::x11::list_windows(Some(pid)) {+ if window.xid == xid || !window.is_on_screen || window.width == 0 || window.height == 0 {+ continue;+ }+ let Some(owner) = crate::x11::transient_for(window.xid) else {+ continue;+ };+ covers |= over((window.x, window.y, window.width, window.height));+ out.dialogs.push(json!({+ "window_id": window.xid,+ "title": window.title,+ "transient_for": owner,+ "modal": crate::x11::window_is_modal(window.xid),+ "bounds": { "x": window.x, "y": window.y, "width": window.width, "height": window.height },+ }));+ }+ // A dialog/plugin window can legitimately run as a DIFFERENT process than+ // the application it belongs to (GIMP's separate-process export dialogs,+ // LibreOffice's Document Recovery dialog under a distinct soffice.bin).+ // Exact-pid matching above would make such a window invisible even though+ // it is unambiguously this application's own popup. WM_TRANSIENT_FOR+ // resolving to one of pid's own windows is the correlation signal (see+ // `list_cross_pid_transient_windows`); such entries are tagged with+ // `owning_pid` so callers can tell it apart from a same-process dialog.+ for window in crate::x11::list_cross_pid_transient_windows(pid) {+ if window.xid == xid || !window.is_on_screen || window.width == 0 || window.height == 0 {+ continue;+ }+ let Some(owner) = crate::x11::transient_for(window.xid) else {+ continue;+ };+ covers |= over((window.x, window.y, window.width, window.height));+ out.dialogs.push(json!({+ "window_id": window.xid,+ "title": window.title,+ "transient_for": owner,+ "modal": crate::x11::window_is_modal(window.xid),+ "bounds": { "x": window.x, "y": window.y, "width": window.width, "height": window.height },+ "owning_pid": window.pid,+ }));+ }+ for popup in crate::input::mapped_popup_windows() {+ if popup.window == xid || popup.pid.is_some_and(|owner| owner != pid) {+ continue;+ }+ // Desktop-wide override-redirect windows that are nobody's menu:+ // mutter's guard window and the driver's own cursor overlay.+ if popup.pid.is_none()+ && (popup.title.contains("guard window") || popup.title.starts_with("Cua."))+ {+ continue;+ }+ covers |= over((popup.x, popup.y, popup.width, popup.height));+ out.popups.push(popup.to_json());+ }+ out.covers_window = covers;+ out+}+pub struct GetWindowStateTool {state: Arc<ToolState>,}@@ -653,11 +1233,33 @@ impl Tool for GetWindowStateTool {mitigate context-window blow-up on Electron / large web apps \that produce 10k+ element trees. When applied, BOTH \the markdown and the structured elements are truncated \- identically. Omit both for current default behaviour.".into(),- input_schema: json!({"type":"object","required":["pid","window_id"],"properties":{+ identically. Omit both for current default behaviour.\n\n\+ TIME BUDGET: `timeout_ms` (default 1000) bounds the whole AT-SPI \+ walk. Large apps (LibreOffice, GIMP, file managers) can exceed it; \+ the call then returns the PARTIAL tree with `truncated: true`, \+ `truncation_reason`, `nodes_visited`/`nodes_pending` and \+ `elements_complete: false`. Every element listed is real and \+ clickable; elements after the cut are simply missing. Retry with a \+ larger `timeout_ms` (e.g. 5000) or narrow with `query`/`max_depth` \+ when the element you need is absent.\n\n\+ SCREENSHOT SCALE: the screenshot is delivered at or below 1.15 megapixels \+ (long edge <= max_image_dimension, 1568 by default), because larger images \+ are downsized before a model reads them and its pixel coordinates would then \+ be uniformly short. Element `screenshot_frame`s and x/y for the pointer tools \+ are pixels of the delivered screenshot (`frame` stays in screen coordinates, \+ as on every platform); `frame_scale` < 1 reports the downsizing. An \+ explicit per-call `max_image_dimension` (0 = native) replaces this cap.\n\n\+ POPUP MENUS: a context menu / popover / combo list is an \+ override-redirect window that list_windows never shows. A click or \+ right_click that opened one names it in its result (`popup: \+ {window_id, bounds, title}`); pass that window_id here to walk the \+ popup's own AT-SPI toplevel so its menu items get element indices \+ (then click them by element_index). Omitting window_id while a popup \+ of this pid is open walks that popup.".into(),+ input_schema: json!({"type":"object","required":["pid"],"properties":{"session": cua_driver_core::tool_schema::session_schema(),"pid":{"type":"integer"},- "window_id":{"type":"integer","description":"Native window identifier from list_windows."},+ "window_id":{"type":"integer","description":"Native window identifier from list_windows, or the `popup.window_id` a click / right_click result named (an open context menu / popover; its menu items then get element indices). Omitted: the pid's open popup menu when one is mapped, else its focused / active / largest window."},"capture_mode": cua_driver_core::capture_mode::capture_mode_schema(),"include_accessibility_tree":{"type":"boolean","description":"Default true — walk the AT-SPI tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)."},@@ -668,6 +1270,7 @@ impl Tool for GetWindowStateTool {"query":{"type":"string","description":"Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count."},"max_elements":{"type":"integer","minimum":1,"description":"Cap on total AT-SPI nodes walked. Omit for the default (5 000). Lower for huge web/Electron trees."},"max_depth":{"type":"integer","minimum":1,"description":"Cap on the AT-SPI tree walk depth. Omit for the default (uncapped). Lower for deeply nested apps."},+ "timeout_ms": cua_driver_core::tool_schema::timeout_ms_schema(),"max_dimension":{"type":"integer","minimum":1,"description":"Legacy optional cap on the returned screenshot's long edge. Applied on top of the configured max_image_dimension ceiling when max_image_dimension is omitted."},"max_image_dimension":{"type":"integer","minimum":0,"description":"Per-call long-edge override. This value wins over configured and legacy limits; 0 returns native-resolution PNG bytes. Omit to preserve configured behavior."}},"additionalProperties":false}),@@ -681,9 +1284,41 @@ impl Tool for GetWindowStateTool {Ok(v) => v,Err(e) => return e,};- let xid = match args.require_u64("window_id") {- Ok(v) => v,- Err(e) => return e,+ // `window_id` omitted: the open popup menu of this pid when one is+ // mapped (a context menu the caller wants to read), else the pid's+ // focused / dialog / active / largest window.+ let xid = match args.opt_u64("window_id") {+ Some(v) => v,+ None => {+ let chosen = tokio::task::spawn_blocking(move || {+ if let Some(popup) = crate::input::mapped_popup_windows()+ .into_iter()+ .rev()+ .find(|p| p.pid == Some(pid))+ {+ return Some(popup.window);+ }+ let windows = crate::x11::list_windows(Some(pid));+ let candidates: Vec<u64> = windows.iter().map(|w| w.xid).collect();+ crate::x11::pick_pid_window(+ &windows,+ crate::x11::focused_window_among(&candidates),+ crate::x11::transient_for,+ crate::x11::active_window(),+ )+ })+ .await+ .ok()+ .flatten();+ match chosen {+ Some(v) => v,+ None => {+ return ToolResult::error(format!(+ "No windows found for pid {pid}. Provide window_id."+ ))+ }+ }+ }};// Retain the legacy additive cap, but let the canonical per-call field// replace the configured ceiling entirely (including 0 = native size).@@ -703,6 +1338,7 @@ impl Tool for GetWindowStateTool {}},};+ let explicit_max_image_dimension = max_image_dimension.is_some();let max_dim = {let cfg = self.state.config.read().unwrap();crate::capture_action_frame::resolve_max_image_dimension(@@ -752,6 +1388,8 @@ impl Tool for GetWindowStateTool {.get("max_depth").and_then(|v| v.as_u64()).map(|v| v.max(1) as usize);+ let timeout_ms = cua_driver_core::tool_schema::resolve_timeout_ms(args.get("timeout_ms"));+ let walk_timeout = std::time::Duration::from_millis(timeout_ms);let process_is_live = crate::proc_fs::is_process_live(pid);// Enumerate the pid's windows ONCE and reuse the result for both the@@ -759,14 +1397,42 @@ impl Tool for GetWindowStateTool {// below, instead of paying for the compositor/X11 enumeration twice.// `window_meta` also names the surface + its on-screen rectangle on the// capture-only path, where no AT-SPI tree identifies it.+ let popup_meta = (!crate::wayland::is_wayland())+ .then(|| crate::input::popup_window_info(xid))+ .flatten();let window_meta = crate::wayland::list_windows_dispatch(Some(pid)).into_iter()- .find(|w| w.xid == xid);+ .find(|w| w.xid == xid)+ .or_else(|| {+ popup_meta.as_ref().map(|popup| crate::x11::WindowInfo {+ xid: popup.window,+ pid: popup.pid,+ app_name: String::new(),+ title: popup.title.clone(),+ is_on_screen: true,+ z_index: None,+ x: popup.x,+ y: popup.y,+ width: popup.width,+ height: popup.height,+ })+ });let window_matches = if crate::wayland::is_wayland() {window_meta.as_ref().is_some_and(|w| w.pid == Some(pid))|| crate::wayland::window_was_listed_for_pid(pid, xid)} else {crate::x11::window_belongs_to_pid(xid, pid)+ || crate::input::popup_window_info(xid)+ .is_some_and(|popup| popup.pid.is_none_or(|owner| owner == pid))+ // A cross-process dialog/plugin window transient-for one of+ // pid's own windows (GIMP export dialogs, LibreOffice's+ // Document Recovery) legitimately belongs to a different pid+ // than the caller's target; accept it here too so the+ // get_window_state(window_id=...) follow-up named in the+ // dialogs[] overlay actually works.+ || crate::x11::list_cross_pid_transient_windows(pid)+ .iter()+ .any(|w| w.xid == xid)};if !process_is_live || !window_matches {return ToolResult::error(format!(@@ -799,12 +1465,13 @@ impl Tool for GetWindowStateTool {// Skip the AT-SPI walk on the capture-only path// (include_accessibility_tree:false).let tree_result = if want_tree {- Some(crate::atspi::walk_tree_bounded(+ Some(crate::atspi::walk_tree_bounded_within(pid,xid,query_for_walk.as_deref(),max_elements,max_depth,+ walk_timeout,))} else {None@@ -825,12 +1492,44 @@ impl Tool for GetWindowStateTool {// binding must identify the bytes returned to the caller, including// any max-dimension resize, rather than the backend's raw frame.let mut screenshot_error = None;+ // Same-pid popups (menus, combo lists) and transient dialogs mapped+ // over this window: listed for the caller, and when one overlaps+ // the window the screenshot is taken from the screen, since the+ // window's own drawable never shows them.+ let overlays = if crate::wayland::is_wayland() {+ WindowOverlays::default()+ } else {+ window_overlays(pid, xid)+ };let screenshot = if should_capture {- match crate::wayland::screenshot_dispatch_with_pid(xid, pid) {+ let captured = if overlays.covers_window {+ overlays+ .window_rect+ .ok_or_else(|| anyhow::anyhow!("window geometry unavailable"))+ .and_then(|(x, y, w, h)| {+ crate::capture::screenshot_root_region_png(x, y, w, h)+ })+ .or_else(|_| crate::wayland::screenshot_dispatch_with_pid(xid, pid))+ } else {+ crate::wayland::screenshot_dispatch_with_pid(xid, pid)+ };+ match captured {Ok(raw) => {let (orig_w, orig_h) = crate::capture::png_dimensions_pub(&raw)?;let png = crate::capture::resize_png_if_needed(&raw, max_dim)?;let (w, h) = crate::capture::png_dimensions_pub(&png)?;+ // An explicit per-call max_image_dimension (0 = native)+ // is authoritative; the model-safe cap applies otherwise.+ let cap = if explicit_max_image_dimension {+ None+ } else {+ megapixel_long_edge_cap(w, h, WINDOW_SCREENSHOT_MAX_PIXELS)+ };+ let png = match cap {+ Some(edge) => crate::capture::resize_png_if_needed(&png, edge)?,+ None => png,+ };+ let (w, h) = crate::capture::png_dimensions_pub(&png)?;let original_w = if w < orig_w { Some(orig_w) } else { None };let (b64, file_path) = if let Some(ref path) = screenshot_out_file {std::fs::write(path, &png)?;@@ -858,6 +1557,15 @@ impl Tool for GetWindowStateTool {screenshot_error = Some(error.to_string());None}+ // The window passed the ownership check above and went+ // away during the AT-SPI walk (a document closed by the+ // action being observed): a stale target, not a capture+ // failure.+ Err(_) if !crate::wayland::is_wayland() && !crate::x11::window_exists(xid) => {+ return Err(anyhow::anyhow!(+ "Window target pid {pid}, window_id {xid} is stale or no longer running; refresh list_windows."+ ));+ }Err(error) => {return Err(anyhow::anyhow!("window screenshot failed for window {xid}: {error}"@@ -867,12 +1575,12 @@ impl Tool for GetWindowStateTool {} else {None};- Ok((tree_result, screenshot, bounds, screenshot_error))+ Ok((tree_result, screenshot, bounds, screenshot_error, overlays))}).await;match result {- Ok(Ok((tree_opt, shot_opt, bounds, screenshot_error))) => {+ Ok(Ok((tree_opt, shot_opt, bounds, screenshot_error, overlays))) => {let mut content = Vec::new();let mut structured = json!({ "window_id": xid, "pid": pid });let screenshot_scale = shot_opt.as_ref().map(|(_, _, w, _, original_w, _)| {@@ -888,14 +1596,44 @@ impl Tool for GetWindowStateTool {.iter().filter(|n| n.element_index.is_some()).count();- let header = format!("window_id={xid} pid={pid} elements={count}\n\n");+ let mut header = format!(+ "window_id={xid} pid={pid} elements={count} walk_ms={}\n",+ tr.elapsed_ms+ );+ if tr.truncated {+ header.push_str(&cua_driver_core::walk_budget::truncation_note(+ tr.truncation_reason.as_deref(),+ timeout_ms,+ tr.nodes_visited,+ tr.nodes_pending,+ ));+ header.push('\n');+ } else if !tr.bounds_complete {+ header.push_str(+ "⚠️ bounds phase ran out of time: some elements have no frame \+ (element_index clicks still work; pixel targeting may not). \+ Retry with a larger timeout_ms if you need frames.\n",+ );+ }+ header.push('\n');content.push(cua_driver_core::protocol::Content::text(header + &tr.tree_markdown,));structured["element_count"] = json!(count);- // AT-SPI's current bounded walker does not surface an- // exhaustive-walk proof. Keep negative existence unknown.- structured["elements_complete"] = json!(false);+ // `elements_complete` is a real claim now: a native walk that+ // finished without hitting the deadline / node budget saw+ // every node the application publishes. Truncated or+ // fallback trees keep negative existence unknown.+ structured["elements_complete"] = json!(tr.trusted && !tr.truncated);+ structured["truncated"] = json!(tr.truncated);+ if let Some(reason) = &tr.truncation_reason {+ structured["truncation_reason"] = json!(reason);+ }+ structured["nodes_visited"] = json!(tr.nodes_visited);+ structured["nodes_pending"] = json!(tr.nodes_pending);+ structured["bounds_complete"] = json!(tr.bounds_complete);+ structured["walk_elapsed_ms"] = json!(tr.elapsed_ms as u64);+ structured["timeout_ms"] = json!(timeout_ms);structured["tree_markdown"] = json!(tr.tree_markdown);let target_scoped = !(crate::wayland::is_wayland()@@ -903,13 +1641,19 @@ impl Tool for GetWindowStateTool {|| tr.window_scoped;if !observation_only && !target_scoped {state.element_cache.remove(pid as i32, xid);+ crate::atspi::cache::forget_window(pid, xid);}+ // The payload carries each element's proven identity plus+ // the frames this walk produced, so later per-index+ // actions act on the observed object without re-walking.let snapshot_id = (!observation_only && target_scoped).then(|| {state.element_cache.publish_for_session(pid as i32,xid,- crate::atspi::cache::CachedSnapshot::from_nodes(&tr.nodes),+ crate::atspi::cache::update_snapshot(+ pid, xid, &tr.nodes, &tr.bounds,+ ),session_id.as_deref(),screenshot_scale,)@@ -930,6 +1674,22 @@ impl Tool for GetWindowStateTool {// toolkits leave bounds unset on hidden / virtual// elements).use std::collections::HashMap;+ // The walk produces screen extents (what the element+ // cache and hit-tests compare against). The published+ // `frame` is the element's screen rectangle, as on macOS and+ // Windows; `screenshot_frame` (added below) is the same+ // rectangle in pixels of the screenshot in this response,+ // the space of the pointer tools' window-local x/y. The+ // window-local origin is the X11 window's root-relative+ // origin, the one `window_bounds` and+ // `coordinate_frame:"desktop"` translation use. A popup+ // (override-redirect) window is not in the WM's client+ // list; when its origin cannot be read, the popup's own+ // screen rectangle is the origin.+ let local_origin = (!crate::wayland::is_wayland())+ .then(|| crate::atspi::native::x11_window_origin(xid))+ .flatten()+ .or_else(|| popup_meta.as_ref().map(|popup| (popup.x, popup.y)));let bounds_by_idx: HashMap<usize, (i32, i32, u32, u32)> = bounds.into_iter().map(|(i, x, y, w, h)| (i, (x, y, w, h)))@@ -950,6 +1710,33 @@ impl Tool for GetWindowStateTool {query.as_deref(),&tr.tree_markdown,);+ let elements = framed_elements_first(elements);+ // Screenshot pixels: the capture is the window's own X11+ // drawable (screen pixels), downsized by delivered/native.+ let elements = match (local_origin, shot_opt.as_ref()) {+ (Some((ox, oy)), Some((_, _, w, _, orig_w, _))) => {+ let scale = orig_w.map_or(1.0, |ow| *w as f64 / ow as f64);+ cua_driver_core::element_frame::with_screenshot_frames(+ elements,+ (f64::from(ox), f64::from(oy)),+ scale,+ )+ }+ _ => elements,+ };+ // A popup that has no AT-SPI frame of its own (LibreOffice+ // VCL menus live under the menubar's `menu` node): return+ // the open menu's items, the ones drawn inside the popup,+ // instead of the whole application.+ let elements = match (&popup_meta, tr.window_scoped) {+ (Some(popup), false) => popup_menu_elements(+ elements,+ (popup.x, popup.y),+ popup.width,+ popup.height,+ ),+ _ => elements,+ };structured["total_element_count"] = json!(count);structured["returned_element_count"] = json!(elements.len());structured["elements"] = json!(elements);@@ -985,6 +1772,15 @@ impl Tool for GetWindowStateTool {discovery evidence; it cannot prove checked state.".to_owned()}));+ } else if count == 0 && tr.truncated {+ structured["degraded"] = json!(true);+ structured["degraded_reason"] =+ json!(cua_driver_core::walk_budget::truncation_note(+ tr.truncation_reason.as_deref(),+ timeout_ms,+ tr.nodes_visited,+ tr.nodes_pending,+ ));} else if count == 0 {structured["degraded"] = json!(true);structured["degraded_reason"] = json!(@@ -1020,7 +1816,7 @@ impl Tool for GetWindowStateTool {}}- if let Some((b64_opt, file_path, w, h, _orig_w, capture_id)) = shot_opt {+ if let Some((b64_opt, file_path, w, h, orig_w, capture_id)) = shot_opt {// ax mode + screenshot_out_file writes the PNG to disk and// returns b64=None — never embed the image bytes in that case.// Keep a text content part when the image went to disk so the@@ -1035,6 +1831,12 @@ impl Tool for GetWindowStateTool {}structured["screenshot_width"] = json!(w);structured["screenshot_height"] = json!(h);+ if let Some(ow) = orig_w {+ if ow > 0 {+ structured["frame_scale"] = json!(w as f64 / ow as f64);+ structured["screenshot_original_width"] = json!(ow);+ }+ }// Surface 7: mirror the MCP image part's `mimeType` onto// the structured payload so consumers don't have to sniff// magic bytes off the base64 to know the format.@@ -1054,6 +1856,9 @@ impl Tool for GetWindowStateTool {// Window identity metadata (additive): app + title + on-screen// rectangle for the requested window_id, useful on the// capture-only path where no AT-SPI tree names the surface.+ if popup_meta.is_some() {+ structured["popup"] = json!(true);+ }if let Some(meta) = &window_meta {if !meta.app_name.is_empty() {structured["app_name"] = json!(meta.app_name);@@ -1065,6 +1870,31 @@ impl Tool for GetWindowStateTool {"x": meta.x, "y": meta.y, "width": meta.width, "height": meta.height});}+ // Transient dialogs and popups of this pid that are open over+ // the window, each with the call that targets it.+ if !overlays.dialogs.is_empty() {+ structured["dialogs"] = json!(overlays.dialogs);+ }+ if !overlays.popups.is_empty() {+ structured["popups"] = json!(overlays.popups);+ }+ if overlays.covers_window {+ structured["screenshot_composited"] = json!(true);+ }+ if let Some(note) = overlays.follow_up(pid) {+ structured["follow_up"] = json!(note);+ content.push(cua_driver_core::protocol::Content::text(note));+ }+ structured["coordinate_frame"] = json!("window");+ structured["frame_note"] = json!(+ "x/y for click / double_click / right_click / drag / scroll on this \+ window are pixels of THIS screenshot (window-local, 0..screenshot_width \+ x 0..screenshot_height), as are the element frames; the screenshot is \+ kept at or below 1.15 megapixels so the image you read is the image these \+ pixels index (frame_scale < 1 says the window was downsized to it). \+ window_bounds is where it sits on the screen. Pass scope:\"desktop\" only \+ for get_desktop_state pixels."+ );// The capture-only path (include_accessibility_tree:false) leaves// `content` empty when the screenshot was also unavailable — most@@ -1096,6 +1926,8 @@ impl Tool for GetWindowStateTool {}}+/// One-line, model-facing explanation of a partial tree and what to do about+/// it. Shared by the text header and the structured `degraded_reason`.fn surface_identity_unproven_error(xid: u64, reason: String) -> Value {json!({"code": "surface_identity_unproven",@@ -1147,6 +1979,7 @@ mod get_window_state_actions_tests {depth: 0,parent_element_index: None,in_web_content: false,+ object_ref: None,}}@@ -1196,6 +2029,13 @@ fn spawn_launch_command(cmd: &str, additional_arguments: &[String]) -> std::io::let mut launch = std::process::Command::new(prog);launch.args(&rest)+ // The child must not inherit the driver's stdio: in private-worker mode+ // stdout is the JSON-RPC stream to the SDK, and a chatty app (Chromium's+ // zygote logs, GTK warnings) writing there corrupts a response and shuts+ // the worker down.+ .stdin(std::process::Stdio::null())+ .stdout(std::process::Stdio::null())+ .stderr(std::process::Stdio::null())// Enable accessibility for this child without toggling GNOME's global// ScreenReaderEnabled setting (which can launch Orca). Native// toolkits ignore these when they do not need them.@@ -1309,6 +2149,7 @@ mod launch_app_tests {bundle_id: bundle_id.to_owned(),launch_path: launch_path.to_owned(),last_used: None,+ startup_wm_class: None,}}@@ -1466,6 +2307,15 @@ impl Tool for LaunchAppTool {return ToolResult::error("Provide at least one of: launch_path, name, or urls.");}+ // Snapshot the top-levels before spawning so a window created by an+ // already-running service (D-Bus activation) can be told apart.+ let windows_before: std::collections::HashSet<u64> =+ tokio::task::spawn_blocking(|| crate::wayland::list_windows_dispatch(None))+ .await+ .unwrap_or_default()+ .into_iter()+ .map(|w| w.xid)+ .collect();let result = tokio::task::spawn_blocking(move || -> anyhow::Result<(String, Option<u32>, String)> {// Open URLs via xdg-open.@@ -1524,7 +2374,7 @@ impl Tool for LaunchAppTool {app.name, app.launch_path),Some(pid),- app.name.clone(),+ format!("{}\u{0}{cmd}\u{0}{}", app.name, app.launch_path),));}// xdg-open handles URLs and file paths, not app@@ -1560,28 +2410,54 @@ impl Tool for LaunchAppTool {match result {Ok(Ok((message, pid_opt, name))) => {- if let Some(pid) = pid_opt {- let windows = tokio::task::spawn_blocking(move || {- let deadline =- std::time::Instant::now() + std::time::Duration::from_secs(3);- loop {- let windows = crate::wayland::list_windows_dispatch(Some(pid));- if !windows.is_empty() || std::time::Instant::now() >= deadline {- return windows.iter().map(window_record_json).collect::<Vec<_>>();- }- std::thread::sleep(std::time::Duration::from_millis(100));- }+ if let Some(launcher_pid) = pid_opt {+ // `name` may carry NUL-separated match keys (display name,+ // query, Exec=) for a desktop-entry launch.+ let query = name.clone();+ let name = name.split('\u{0}').next().unwrap_or("").to_owned();+ let resolved = tokio::task::spawn_blocking(move || {+ resolve_launched_windows(launcher_pid, &query, &windows_before)}).await- .unwrap_or_default();- ToolResult::text(message).with_structured(json!({- "pid": pid,+ .unwrap_or_else(|_| LaunchedWindows::default());+ let windows: Vec<Value> =+ resolved.windows.iter().map(window_record_json).collect();+ let mut text = message;+ let mut structured = json!({+ "pid": resolved.pid,"bundle_id": Value::Null,"name": name,- "running": true,+ "running": resolved.pid.is_some(),"active": false,"windows": windows,- }))+ "launcher_pid": launcher_pid,+ });+ if resolved.handed_off {+ match resolved.pid {+ Some(pid) => {+ structured["handoff"] = json!("dbus_activation");+ text.push_str(&format!(+ " The launcher handed the request to another process \+ (a wrapper or D-Bus activation); the window belongs to pid {pid}."+ ));+ }+ None => {+ // The launcher exited (D-Bus activation or a failed+ // start) and the hand-off budget in+ // `resolve_launched_windows` passed with no new+ // window ever appearing. This is not a success: we+ // have no pid, no window, and no way to know the app+ // actually started. Report a typed refusal instead+ // of a `pid: null` shape that reads as success.+ return launch_handoff_timeout_result(+ &name,+ launcher_pid,+ LAUNCH_HANDOFF_BUDGET.as_secs(),+ );+ }+ }+ }+ ToolResult::text(text).with_structured(structured)} else {ToolResult::text(message).with_structured(json!({"pid": Value::Null,@@ -1599,20 +2475,241 @@ impl Tool for LaunchAppTool {}}-fn chromium_family_program(program: &str) -> bool {- let basename = std::path::Path::new(program)- .file_name()- .and_then(|name| name.to_str())- .unwrap_or(program)- .to_ascii_lowercase();- ["chrome", "chromium", "electron", "brave", "edge"]- .iter()- .any(|needle| basename.contains(needle))+/// The `launch_app` D-Bus-activation hand-off timed out: the launcher exited+/// (handing the request to a running service) but no new window of the app+/// ever appeared within the deadline. This has no pid and no confirmed+/// window, so it is a refusal (`code: "launch_handoff_timeout"`), never a+/// `pid: null` shape that a caller could mistake for success.+fn launch_handoff_timeout_result(name: &str, launcher_pid: u32, waited_secs: u64) -> ToolResult {+ ToolResult::error(format!(+ "'{name}' was handed off for D-Bus activation but no new window appeared within \+ {waited_secs}s; the launch could not be confirmed. Call list_windows to check \+ whether it started anyway, or retry."+ ))+ .with_structured(json!({+ "code": "launch_handoff_timeout",+ "effect": "refused",+ "handoff": "dbus_activation",+ "name": name,+ "launcher_pid": launcher_pid,+ "pid": Value::Null,+ "running": Value::Null,+ "waited_secs": waited_secs,+ }))}-// ── shared helpers ────────────────────────────────────────────────────────────+/// What `launch_app` could attribute to the launch after the spawn.+#[derive(Default)]+struct LaunchedWindows {+ /// The process that owns the app's window: the launcher itself, or the+ /// wrapper target / running service it handed off to.+ pid: Option<u32>,+ windows: Vec<crate::x11::WindowInfo>,+ /// The window belongs to another process than the launcher, or the+ /// launcher exited without one.+ handed_off: bool,+}-/// Resolve an AT-SPI element's center in window-local coordinates.+/// Whether the launched process is gone or a zombie (already reaped or being+/// reaped by the launch thread).+fn process_exited(pid: u32) -> bool {+ match std::fs::read_to_string(format!("/proc/{pid}/stat")) {+ Ok(stat) => stat+ .rsplit_once(')')+ .and_then(|(_, rest)| rest.split_whitespace().next())+ .is_some_and(|state| state == "Z" || state == "X"),+ Err(_) => true,+ }+}++/// Case-insensitive identity match between a launch query+/// (`nautilus`, `org.gnome.Nautilus`, `gnome-control-center`, a display+/// name) and a window's WM_CLASS / app id.+fn window_matches_launch(window: &crate::x11::WindowInfo, query: &str) -> bool {+ query+ .split('\u{0}')+ .filter(|key| !key.trim().is_empty())+ .any(|key| window_matches_launch_key(window, key))+}++fn window_matches_launch_key(window: &crate::x11::WindowInfo, query: &str) -> bool {+ let class = window.app_name.to_ascii_lowercase();+ if class.is_empty() {+ return false;+ }+ let query = query.to_ascii_lowercase();+ let stem = query+ .rsplit('/')+ .next()+ .unwrap_or(&query)+ .trim_end_matches(".desktop")+ .to_owned();+ let last = stem.rsplit('.').next().unwrap_or(&stem).to_owned();+ let class_last = class.rsplit('.').next().unwrap_or(&class).to_owned();+ class == stem+ || class_last == last+ || class.contains(&last)+ || last.contains(&class_last)+ || stem+ .split_whitespace()+ .next()+ .is_some_and(|word| class.contains(word))+}++/// How long a launch may take to show a window while the launcher process is+/// still alive (the app is starting; LibreOffice on a small VM needs well over+/// ten seconds), and once it has exited (a wrapper or D-Bus activation has+/// taken over, so the window is either imminent or never coming).+const LAUNCH_STARTING_BUDGET: std::time::Duration = std::time::Duration::from_secs(20);+const LAUNCH_HANDOFF_BUDGET: std::time::Duration = std::time::Duration::from_secs(8);++/// Resolve the window(s) of a launch. Prefers the launcher's own window;+/// otherwise adopts a newly mapped top-level whose WM_CLASS matches the+/// launch, whoever owns it: a wrapper script that exec'd the real binary+/// (`libreoffice` → `soffice.bin`), or the running D-Bus service GNOME apps+/// hand their request to.+fn resolve_launched_windows(+ launcher_pid: u32,+ query: &str,+ before: &std::collections::HashSet<u64>,+) -> LaunchedWindows {+ let started = std::time::Instant::now();+ loop {+ let all = crate::wayland::list_windows_dispatch(None);+ let own: Vec<_> = all+ .iter()+ .filter(|w| w.pid == Some(launcher_pid))+ .cloned()+ .collect();+ if !own.is_empty() {+ return LaunchedWindows {+ pid: Some(launcher_pid),+ windows: own,+ handed_off: false,+ };+ }+ let fresh: Vec<_> = all+ .iter()+ .filter(|w| !before.contains(&w.xid) && w.pid.is_some())+ .filter(|w| window_matches_launch(w, query))+ .cloned()+ .collect();+ if let Some(pid) = fresh.first().and_then(|w| w.pid) {+ let windows = fresh.into_iter().filter(|w| w.pid == Some(pid)).collect();+ return LaunchedWindows {+ pid: Some(pid),+ windows,+ handed_off: true,+ };+ }+ let launcher_exited = process_exited(launcher_pid);+ let budget = if launcher_exited {+ LAUNCH_HANDOFF_BUDGET+ } else {+ LAUNCH_STARTING_BUDGET+ };+ if started.elapsed() >= budget {+ return LaunchedWindows {+ pid: (!launcher_exited).then_some(launcher_pid),+ windows: Vec::new(),+ handed_off: launcher_exited,+ };+ }+ std::thread::sleep(std::time::Duration::from_millis(100));+ }+}++#[cfg(test)]+mod launch_resolution_tests {+ use super::*;++ fn window(class: &str) -> crate::x11::WindowInfo {+ crate::x11::WindowInfo {+ xid: 1,+ pid: Some(7),+ app_name: class.into(),+ title: "t".into(),+ is_on_screen: true,+ z_index: None,+ x: 0,+ y: 0,+ width: 1,+ height: 1,+ }+ }++ #[test]+ fn launch_queries_match_window_classes() {+ assert!(window_matches_launch(+ &window("org.gnome.Nautilus"),+ "nautilus"+ ));+ assert!(window_matches_launch(+ &window("org.gnome.Nautilus"),+ "org.gnome.Nautilus"+ ));+ assert!(window_matches_launch(+ &window("Gnome-terminal"),+ "gnome-terminal"+ ));+ assert!(window_matches_launch(+ &window("Gnome-control-center"),+ "gnome-control-center"+ ));+ assert!(window_matches_launch(+ &window("Org.gnome.Nautilus"),+ "org.gnome.Nautilus.desktop"+ ));+ assert!(window_matches_launch(+ &window("Org.gnome.Nautilus"),+ "Files\u{0}org.gnome.Nautilus\u{0}nautilus --new-window"+ ));+ assert!(window_matches_launch(+ &window("Gnome-control-center"),+ "Settings\u{0}Settings\u{0}gnome-control-center"+ ));+ assert!(!window_matches_launch(&window("Gedit"), "nautilus"));+ assert!(!window_matches_launch(&window(""), "nautilus"));+ }++ #[test]+ fn a_missing_process_counts_as_exited() {+ assert!(process_exited(u32::MAX - 1));+ assert!(!process_exited(std::process::id()));+ }++ #[test]+ fn a_handoff_timeout_is_a_refusal_not_a_null_pid_success() {+ let result = launch_handoff_timeout_result("Nautilus", 4242, 8);+ assert_eq!(+ result.is_error,+ Some(true),+ "a hand-off timeout must be reported as an error, not success"+ );+ let s = result.structured_content.unwrap();+ assert_eq!(s["code"], "launch_handoff_timeout");+ assert_eq!(s["effect"], "refused");+ assert_eq!(s["pid"], Value::Null);+ assert_eq!(s["running"], Value::Null);+ assert_eq!(s["launcher_pid"], 4242);+ assert_eq!(s["waited_secs"], 8);+ }+}++fn chromium_family_program(program: &str) -> bool {+ let basename = std::path::Path::new(program)+ .file_name()+ .and_then(|name| name.to_str())+ .unwrap_or(program)+ .to_ascii_lowercase();+ ["chrome", "chromium", "electron", "brave", "edge"]+ .iter()+ .any(|needle| basename.contains(needle))+}++// ── shared helpers ────────────────────────────────────────────────────────────++/// Resolve an AT-SPI element's center in window-local coordinates.////// Returns `(xid, window_local_x, window_local_y)`./// Looks up element bounds via native AT-SPI, finds the owning window@@ -1675,6 +2772,56 @@ fn resolve_element_local_coords(Ok((xid, local_x, local_y))}+/// An element whose AT-SPI extents do not fall inside its own toplevel: GTK4+/// reports empty extents for some widgets (a text view centre resolves to+/// (0,0), the GNOME hot corner), and a real pointer click there would hit+/// whatever is under that screen point. Refused before any input is sent.+#[derive(Debug)]+struct ElementBoundsUnusable {+ idx: usize,+ local: (f64, f64),+ window: (u32, u32),+}++impl std::fmt::Display for ElementBoundsUnusable {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ write!(+ f,+ "element [{}] resolves to window-local ({:.0}, {:.0}), outside its {}x{} window; \+ the toolkit reported unusable AT-SPI extents, so no pointer input was sent",+ self.idx, self.local.0, self.local.1, self.window.0, self.window.1+ )+ }+}+impl std::error::Error for ElementBoundsUnusable {}++/// `resolve_element_local_coords` plus the sanity check that the centre lies+/// inside the window it belongs to. Blocking.+fn checked_element_local_coords(+ pid: u32,+ idx: usize,+ xid_hint: Option<u64>,+) -> anyhow::Result<(u64, f64, f64)> {+ let (xid, lx, ly) = resolve_element_local_coords(pid, idx, xid_hint)?;+ if crate::wayland::wayland_input_enabled() {+ return Ok((xid, lx, ly));+ }+ use x11rb::protocol::xproto::ConnectionExt as _;+ use x11rb::rust_connection::RustConnection;+ let (conn, _) = RustConnection::connect(None)?;+ let geom = conn.get_geometry(xid as u32)?.reply()?;+ let (w, h) = (u32::from(geom.width), u32::from(geom.height));+ if lx < 0.0 || ly < 0.0 || lx >= f64::from(w) || ly >= f64::from(h) {+ return Err(ElementBoundsUnusable {+ idx,+ local: (lx, ly),+ window: (w, h),+ }+ .into());+ }+ Ok((xid, lx, ly))+}+fn element_screen_center(pid: u32, idx: usize, xid: Option<u64>) -> anyhow::Result<(f64, f64)> {let (bx, by, bw, bh) = match xid {Some(xid) => crate::atspi::get_element_bounds_for_window(pid, xid, idx)?,@@ -1683,6 +2830,22 @@ fn element_screen_center(pid: u32, idx: usize, xid: Option<u64>) -> anyhow::ResuOk((bx as f64 + bw as f64 / 2.0, by as f64 + bh as f64 / 2.0))}+/// Shared schema for the optional `coordinate_frame` of pointer tools.+fn coordinate_frame_schema() -> Value {+ json!({+ "type": "string",+ "enum": ["window", "desktop"],+ "description": "Frame of x/y (and from_x/from_y/to_x/to_y). Default \"window\": \+ window-local screenshot pixels as returned by get_window_state. \"desktop\": \+ full-screen pixels as returned by get_desktop_state, translated to the target \+ window. Passing scope:\"desktop\" together with pid/window_id means the same thing."+ })+}++fn coordinate_frame_is_desktop(args: &Value) -> bool {+ args.get("coordinate_frame").and_then(Value::as_str) == Some("desktop")+}+fn window_local_to_screen(xid: u64, x: f64, y: f64) -> anyhow::Result<(f64, f64)> {use x11rb::connection::Connection;use x11rb::protocol::xproto::ConnectionExt as _;@@ -1696,6 +2859,248 @@ fn window_local_to_screen(xid: u64, x: f64, y: f64) -> anyhow::Result<(f64, f64)Ok((reply.dst_x as f64 + x, reply.dst_y as f64 + y))}+/// `scope:"desktop"` alongside a pid/window target: the caller read `x,y` off+/// `get_desktop_state` (screen pixels) but still names the window to act on.+/// Upper bound for one AT-SPI element operation (bounds lookup / action).+/// The AT-SPI layer bounds single bus calls, but a full tree walk over a large+/// GTK/VCL window chains many of them; observed stalls ran to minutes.+const ELEMENT_AX_BUDGET: std::time::Duration = std::time::Duration::from_secs(30);++/// Either `coordinate_frame:"desktop"` or `scope:"desktop"` marks `x,y` as+/// desktop (screen) pixels; both are translated exactly once by the caller.+fn desktop_frame_requested(args: &Value) -> bool {+ coordinate_frame_is_desktop(args)+ || args.get("scope").and_then(Value::as_str) == Some("desktop")+}++/// Convert desktop (screen) coordinates to `xid`'s window-local frame so the+/// rest of the pointer pipeline (which is window-local by contract) is reused.+fn desktop_to_window_local(xid: u64, x: f64, y: f64) -> anyhow::Result<(f64, f64)> {+ let (ox, oy) = window_local_to_screen(xid, 0.0, 0.0)?;+ Ok((x - ox, y - oy))+}++/// Upper bound for one foreground input transaction: focus confirmation plus+/// the injection itself. Typing sleeps per character, so scale with `chars`.+fn foreground_budget(chars: usize) -> std::time::Duration {+ std::time::Duration::from_millis(12_000 + (chars as u64) * 40)+}++/// `spawn_blocking` with a hard deadline. A wedged X server (or a client+/// holding a server grab) must surface as a structured `foreground_timeout`+/// error instead of a tool call that never returns. The blocking task is not+/// cancellable; it is left to finish (or leak) on the blocking pool.+async fn spawn_blocking_bounded<T: Send + 'static>(+ label: &'static str,+ budget: std::time::Duration,+ f: impl FnOnce() -> anyhow::Result<T> + Send + 'static,+) -> Result<anyhow::Result<T>, tokio::task::JoinError> {+ match tokio::time::timeout(budget, tokio::task::spawn_blocking(f)).await {+ Ok(joined) => joined,+ Err(_elapsed) => Ok(Err(anyhow::anyhow!(+ "{}: {label} did not complete within {budget:?}; the X server or the target \+ application may be wedged. Take a screenshot before retrying",+ crate::input::foreground::CODE_TIMEOUT+ ))),+ }+}++/// Map an input-path error to a tool error, attaching the structured+/// foreground `code` (`foreground_unavailable` / `foreground_timeout`) when the+/// message carries one so the caller can branch instead of parsing prose.+fn input_error_result(e: anyhow::Error) -> ToolResult {+ let text = e.to_string();+ if crate::input::is_uinput_unavailable(&e) {+ // The focus-free real-input route exists but this process cannot open+ // /dev/uinput: an honest refusal the operator can act on, not a bare+ // I/O error.+ let mut refusal = with_uinput_hint(crate::input::delivery::background_unavailable_error(+ crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,+ ));+ if let Some(structured) = refusal.structured_content.as_mut() {+ structured["cause"] = json!(crate::input::UINPUT_UNAVAILABLE_CODE);+ structured["detail"] = json!(text);+ }+ return refusal;+ }+ match crate::input::foreground::error_code(&e) {+ Some(code) => ToolResult::error(text.clone()).with_structured(json!({+ "code": code,+ "detail": text,+ "effect": "none",+ "suggestion": if code == crate::input::foreground::CODE_TIMEOUT {+ "Take a screenshot (get_desktop_state) to confirm the desktop is responsive, then retry once."+ } else {+ "Call bring_to_front on the window (or its modal dialog), confirm with get_window_state, then retry."+ },+ })),+ None => linux_input_error(e),+ }+}++/// Structured payload for a completed foreground transaction.+fn foreground_structured(+ path: &str,+ report: crate::input::ForegroundReport,+ extra: serde_json::Map<String, Value>,+) -> Value {+ let mut v = json!({+ "path": path,+ "verified": false,+ "delivery_mode": "foreground",+ "effect": "unverifiable",+ });+ apply_foreground_report(&mut v, &report);+ for (k, val) in extra {+ v[k] = val;+ }+ v+}++/// Fold a foreground transaction's post-check into a result: the report+/// itself, `focus_after` at the top level, and an `evidence` item so the+/// observation survives the public-record reduction. Focus that stayed in+/// the target process AND a window that appeared / closed in it is+/// `window_change` evidence and an `effect: confirmed`; focus that left the+/// process is a suspected no-op; anything else stays unverifiable with a+/// `native_api_result` item.+fn apply_foreground_report(v: &mut Value, report: &crate::input::ForegroundReport) {+ v["foreground"] = report.to_json();+ v["focus_after"] = json!(report.focus_after.as_str());+ if report.focus_after == crate::input::FocusAfter::Elsewhere {+ v["effect"] = json!("suspected_noop");+ v["warning"] = json!(+ "input focus left the target process before the post-check; the input \+ may have reached another window. Verify with a screenshot."+ );+ return;+ }+ let evidence = match (&report.window_change, report.focus_kept()) {+ (Some(change), true) => {+ v["effect"] = json!("confirmed");+ v["verified"] = json!(true);+ json!({+ "kind": "window_change",+ "detail": format!("focus_after={}; {change}", report.focus_after.as_str()),+ })+ }+ _ => json!({+ "kind": "native_api_result",+ "detail": format!(+ "real input delivered to the activated window (focus_after={}); no window \+ change observed",+ report.focus_after.as_str()+ ),+ }),+ };+ let mut items = v+ .get("evidence")+ .and_then(Value::as_array)+ .cloned()+ .unwrap_or_default();+ items.push(evidence);+ v["evidence"] = json!(items);+}++/// Typed refusal for a pointer tool whose arguments name neither an element+/// nor a complete point. Before this check a call with only a pid was+/// executed at window-local (0, 0) — the top-left corner, i.e. the File menu.+fn invalid_pointer_arguments(tool: &str, detail: &str) -> ToolResult {+ let accepted = [+ "pid + element_index (or element_token) from get_window_state",+ "pid/window_id + x + y (window-local pixels of window_id; coordinate_frame:\"desktop\" for screen pixels)",+ "x + y + scope:\"desktop\" (screen pixels, no pid)",+ ];+ ToolResult::error(format!(+ "{tool}: invalid arguments: {detail}. Accepted forms: {}. No input was sent.",+ accepted.join("; ")+ ))+ .with_structured(json!({+ "code": "invalid_arguments",+ "tool": tool,+ "effect": "refused",+ "detail": detail,+ "accepted_forms": accepted,+ }))+}++/// `Some(refusal)` when a pointer tool reached its coordinate path without+/// both `x` and `y` (and, by construction, without an element).+fn require_point_args(tool: &str, args: &Value) -> Option<ToolResult> {+ let present = |name: &str| args.get(name).is_some_and(|v| v.is_number());+ match (present("x"), present("y")) {+ (true, true) => None,+ (false, false) => Some(invalid_pointer_arguments(+ tool,+ "no element_index/element_token and no x/y were given (a `target_id` is not an \+ element handle)",+ )),+ (true, false) => Some(invalid_pointer_arguments(tool, "x was given without y")),+ (false, true) => Some(invalid_pointer_arguments(tool, "y was given without x")),+ }+}++#[cfg(test)]+mod pointer_argument_tests {+ use super::*;++ #[test]+ fn pid_only_pointer_call_is_refused_not_clicked_at_origin() {+ let refusal = require_point_args("click", &json!({"target_id": "s1:147", "pid": 1867}))+ .expect("refused");+ assert_eq!(refusal.is_error, Some(true));+ let structured = refusal.structured_content.expect("structured");+ assert_eq!(structured["code"], "invalid_arguments");+ assert_eq!(structured["effect"], "refused");+ assert!(structured["accepted_forms"].as_array().unwrap().len() == 3);+ }++ #[test]+ fn complete_point_passes() {+ assert!(require_point_args(+ "click",+ &json!({"pid": 1, "window_id": 2, "x": 3, "y": 4.5})+ )+ .is_none());+ assert!(require_point_args("click", &json!({"x": 1, "y": null})).is_some());+ assert!(require_point_args("click", &json!({"y": 1})).is_some());+ }++ #[test]+ fn foreground_report_becomes_evidence() {+ let report = crate::input::ForegroundReport {+ already_active: false,+ retried_activation: false,+ confirm_ms: 12,+ focus_after: crate::input::FocusAfter::SamePid,+ window_change: Some("appeared: window 9 \"Brightness-Contrast\"".into()),+ };+ let v = foreground_structured("x11_xtest_fg", report, serde_json::Map::new());+ assert_eq!(v["effect"], "confirmed");+ assert_eq!(v["focus_after"], "same_pid");+ assert_eq!(v["evidence"][0]["kind"], "window_change");+ let quiet = crate::input::ForegroundReport {+ already_active: true,+ retried_activation: false,+ confirm_ms: 0,+ focus_after: crate::input::FocusAfter::Target,+ window_change: None,+ };+ let v = foreground_structured("x11_xtest_fg", quiet, serde_json::Map::new());+ assert_eq!(v["effect"], "unverifiable");+ assert_eq!(v["evidence"][0]["kind"], "native_api_result");+ let lost = crate::input::ForegroundReport {+ already_active: true,+ retried_activation: false,+ confirm_ms: 0,+ focus_after: crate::input::FocusAfter::Elsewhere,+ window_change: Some("closed: window 3".into()),+ };+ let v = foreground_structured("x11_xtest_fg", lost, serde_json::Map::new());+ assert_eq!(v["effect"], "suspected_noop");+ }+}+fn parse_mouse_button(name: &str) -> u8 {match name {"right" => 3,@@ -1736,26 +3141,21 @@ fn non_ax_escalation() -> Value {/// `effect` tri-state. Linux's AT-SPI `insertText` return value acknowledges/// the method call but does not read the widget value back, so it and every/// keystroke / XSendEvent / XTest / Wayland rung are `"unverifiable"` (the-/// caller confirms through a separate observation) and-/// carries a `foreground` escalation, because the field IS in the AT-SPI tree —-/// it's a delivery/focus problem, not a missing element. The foreground rung-/// itself (`key_events_fg`) is already the last resort, so it emits no-/// escalation. Mirrors the macOS `type_text` contract.+/// caller confirms through a separate observation). No escalation is+/// attached to a successful delivery: the public projection renders one as+/// `delivery_failed`, which is wrong for text that landed but was not read+/// back. Mirrors the macOS `type_text` contract.fn type_text_structured(path: &str, characters: usize, verified: bool) -> Value {- let mut s = json!({+ let s = json!({"path": path,"characters": characters,"verified": verified,"effect": if verified { "confirmed" } else { "unverifiable" },});- if !verified && path != "key_events_fg" {- s["escalation"] = json!({- "recommended": "foreground",- "reason": "background insert could not be confirmed — re-call with \- delivery_mode:\"foreground\" if a screenshot shows the text \- didn't land."- });- }+ // No escalation on a successful-but-unverified delivery: the public+ // projection renders a foreground escalation as `delivery_failed`, which+ // made models re-type text that had already landed. Refusals carry their own.+ let _ = verified;s}@@ -1782,6 +3182,101 @@ fn type_text_structured_electron(text_len: usize) -> Value {})}+/// [`type_text_ax_result`] for an element-addressed insert: the snapshot+/// object is read back (as `set_value` does) and, when the typed text is now+/// part of its value, the result is `effect: confirmed` with a+/// `value_readback` evidence item instead of `unverifiable`.+async fn type_text_ax_result_verified(+ pid: u32,+ xid_opt: Option<u64>,+ idx: usize,+ text: &str,+ route: &str,+) -> ToolResult {+ let text_len = text.chars().count();+ if is_chromium_embedder(pid) {+ return type_text_ax_result(pid, text_len, route);+ }+ let readback =+ tokio::task::spawn_blocking(move || crate::atspi::read_value_in(pid, xid_opt, idx))+ .await+ .ok()+ .flatten();+ type_text_readback_result(pid, idx, text, route, readback)+}++/// The typed text is confirmed when the element's fresh value contains it+/// (an insert at the caret) or agrees with it numerically (a spin button+/// that reformats "-40" as "-40.0").+fn type_text_readback_result(+ pid: u32,+ idx: usize,+ text: &str,+ route: &str,+ readback: Option<String>,+) -> ToolResult {+ let text_len = text.chars().count();+ match readback {+ Some(seen)+ if !text.trim().is_empty() && (seen.contains(text) || values_agree(text, &seen)) =>+ {+ let shown: String = seen.chars().take(200).collect();+ ToolResult::text(format!(+ "Typed {text_len} character(s) into element [{idx}] ({route}); it reads back '{shown}'."+ ))+ .with_structured(json!({+ "path": "ax",+ "characters": text_len,+ "verified": true,+ "effect": "confirmed",+ "readback": shown,+ "evidence": [{+ "kind": "value_readback",+ "detail": format!("element [{idx}] reads back the typed text"),+ }],+ }))+ }+ Some(seen) => {+ let mut result = type_text_ax_result(pid, text_len, route);+ let shown: String = seen.chars().take(200).collect();+ if let Some(structured) = result.structured_content.as_mut() {+ structured["readback"] = json!(shown);+ }+ result+ .content+ .push(cua_driver_core::protocol::Content::text(format!(+ "The element reads back '{shown}', which does not contain the typed text; \+ confirm with get_window_state."+ )));+ result+ }+ None => type_text_ax_result(pid, text_len, route),+ }+}++#[cfg(test)]+mod type_text_readback_tests {+ use super::*;++ #[test]+ fn typed_text_is_confirmed_only_when_read_back() {+ let confirmed =+ type_text_readback_result(1, 3, "-40", "via targeted AT-SPI", Some("-40.0".into()));+ let s = confirmed.structured_content.clone().unwrap();+ assert_eq!(s["effect"], "confirmed");+ assert_eq!(s["verified"], true);+ assert_eq!(s["evidence"][0]["kind"], "value_readback");+ let contained = type_text_readback_result(1, 3, "ab", "r", Some("xxabyy".into()));+ assert_eq!(contained.structured_content.unwrap()["effect"], "confirmed");+ let mismatch = type_text_readback_result(1, 3, "ab", "r", Some("zz".into()));+ let s = mismatch.structured_content.unwrap();+ assert_eq!(s["effect"], "unverifiable");+ assert_eq!(s["readback"], "zz");+ let blind = type_text_readback_result(1, 3, "ab", "r", None);+ assert_eq!(blind.structured_content.unwrap()["effect"], "unverifiable");+ }+}+/// Build the success `ToolResult` for an AT-SPI insert. The EditableText/// method's boolean is a delivery acknowledgement, not a fresh value readback,/// so native widgets remain `unverifiable`. Chromium embedders additionally@@ -1921,6 +3416,30 @@ fn maps_indicate_gtk(maps: &str) -> bool {maps.contains("libgtk-3.so") || maps.contains("libgtk-4.so")}+/// Toolkits known to discard synthetic (`send_event`) X11 pointer events:+/// GTK3/4 (XInput2 only), LibreOffice VCL (all plugins), Qt5/6 (xcb, XI2).+fn maps_indicate_synthetic_pointer_dropped(maps: &str) -> bool {+ maps_indicate_gtk(maps)+ || maps.contains("libvcl")+ || maps.contains("libmergedlo")+ || maps.contains("libQt5Gui")+ || maps.contains("libQt6Gui")+}++fn synthetic_pointer_is_dropped(pid: u32) -> bool {+ fs::read_to_string(format!("/proc/{pid}/maps"))+ .map(|maps| maps_indicate_synthetic_pointer_dropped(&maps))+ .unwrap_or(false)+}++/// Qt5 maps its own GUI library; only then is the synthetic-FocusIn bridge+/// workaround in `type_text` worth two extra AT-SPI walks.+fn is_qt5_process(pid: u32) -> bool {+ fs::read_to_string(format!("/proc/{pid}/maps"))+ .map(|maps| maps.contains("libQt5Gui") || maps.contains("libQt5Widgets"))+ .unwrap_or(false)+}+fn is_gtk_process(pid: u32) -> bool {fs::read_to_string(format!("/proc/{pid}/maps")).map(|maps| maps_indicate_gtk(&maps))@@ -1954,17 +3473,333 @@ fn unavailable_webkit_keyboard_background(})}+/// Operator-facing hint attached to a background keyboard refusal on X11: the+/// focus-free route exists (XI2 MPX virtual master keyboard) but needs a+/// writable `/dev/uinput`, which is a one-time host setup step.+const UINPUT_KEYBOARD_HINT: &str = "Background keyboard delivery on X11 needs a virtual master \+ keyboard fed by a uinput device. Grant this process access to /dev/uinput \+ (modprobe uinput; chmod 0666 /dev/uinput, or add the driver's user to the \+ 'input' group and restart it) and retry. Xvfb/Xtigervnc cannot hot-add \+ input devices, so the route is unavailable there.";++fn with_uinput_hint(mut refusal: ToolResult) -> ToolResult {+ refusal+ .content+ .push(cua_driver_core::protocol::Content::text(+ UINPUT_KEYBOARD_HINT,+ ));+ if let Some(structured) = refusal.structured_content.as_mut() {+ structured["hint"] = json!(UINPUT_KEYBOARD_HINT);+ }+ refusal+}++/// Honest refusal for a background keyboard action that has no focus-free+/// actuator. Carries the uinput hint whenever the MPX route is missing only+/// because `/dev/uinput` is not writable (the fixable case).+fn background_keyboard_refusal() -> ToolResult {+ let refusal = crate::input::delivery::background_unavailable_error(+ crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,+ );+ if !crate::wayland::is_wayland() && !crate::input::uinput_accessible() {+ with_uinput_hint(refusal)+ } else {+ refusal+ }+}+fn unavailable_gtk_keyboard_background(pid: u32,delivery: crate::input::delivery::DeliveryMode,) -> Option<ToolResult> {- (!delivery.is_foreground() && is_gtk_process(pid) && !crate::wayland::is_inject_mode()).then(- || {- crate::input::delivery::background_unavailable_error(- crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,- )- },- )+ (!delivery.is_foreground()+ && is_gtk_process(pid)+ && !crate::wayland::is_inject_mode()+ && !crate::input::real_keyboard_input_available())+ .then(background_keyboard_refusal)+}++/// Outcome of one background/foreground keyboard delivery on X11, funnelled+/// through `spawn_blocking`'s success type so the tool can report the route.+enum KeyRoute {+ /// Activated the window and injected via XTest.+ Foreground(crate::input::ForegroundReport),+ /// Real key events through the session's XI2 virtual master keyboard.+ Mpx(crate::input::KeyboardDeliveryReport),+ /// Legacy synthetic XSendEvent (core-only toolkits, Tk, xterm).+ Synthetic,+ /// Written to the terminal's pty master.+ Terminal,+}++/// Background key delivery that honours a keyboard grab held by the target's+/// own popup (a Qt combo list / completer, a GTK or VCL menu): the X server+/// drops core key events from any other master keyboard to a client that+/// holds an active keyboard grab, so the virtual master keyboard route would+/// be silently lost; the core keyboard (XTest) reaches the grab holder, which+/// is the target itself, without any focus change.+fn background_virtual_key(+ cursor_id: &str,+ pid: u32,+ xid: u64,+ key: &str,+ modifiers: &[&str],+) -> anyhow::Result<crate::input::KeyboardDeliveryReport> {+ if let Some(popup) = crate::input::popup_of_pid(pid) {+ return crate::input::send_keys_under_popup_grab(pid, popup, || {+ crate::input::send_key_xtest(key, modifiers)+ });+ }+ crate::input::send_virtual_keyboard_key(cursor_id, xid, key, modifiers)+}++/// Text counterpart of [`background_virtual_key`].+fn background_virtual_text(+ cursor_id: &str,+ pid: u32,+ xid: u64,+ text: &str,+) -> anyhow::Result<crate::input::KeyboardDeliveryReport> {+ if let Some(popup) = crate::input::popup_of_pid(pid) {+ return crate::input::send_keys_under_popup_grab(pid, popup, || {+ crate::input::send_type_text_xtest(text)+ });+ }+ crate::input::send_virtual_keyboard_text(cursor_id, xid, text)+}++fn is_popup_keyboard_grab(error: &anyhow::Error) -> bool {+ error+ .downcast_ref::<crate::input::PopupKeyboardGrab>()+ .is_some()+}++/// Deliver a background key/chord: prefer the focus-free real-input route+/// (MPX virtual master keyboard) when the host supports it, otherwise the+/// legacy synthetic XSendEvent. A uinput permission failure is surfaced as is+/// (the tool turns it into the refusal + hint); any other MPX failure falls+/// back to XSendEvent for the toolkits that still accept it, and is reported+/// as an error for the ones known to drop it.+fn background_key_route(+ cursor_id: &str,+ pid: u32,+ xid: u64,+ px_target: Option<(i32, i32)>,+ key: &str,+ modifiers: &[&str],+ element_index: Option<usize>,+) -> anyhow::Result<KeyRoute> {+ if crate::input::real_keyboard_input_available() {+ // The element GrabFocus and the delivery run under one focus guard:+ // GTK can raise/activate the toplevel for either, and the guard+ // restores the user's focus and reports it (see `focus_guard`).+ let delivered = crate::input::focus_guard::guarded(Some(pid), || {+ grab_focus_for_background(pid, element_index)?;+ background_virtual_key(cursor_id, pid, xid, key, modifiers)+ });+ match delivered {+ Ok((mut report, guard)) => {+ report.focus_guard = guard;+ return Ok(KeyRoute::Mpx(report));+ }+ Err(error) if crate::input::is_uinput_unavailable(&error) => return Err(error),+ Err(error) if is_popup_keyboard_grab(&error) => return Err(error),+ Err(error)+ if is_grab_focus_failure(&error)+ || is_gtk_process(pid)+ || synthetic_pointer_is_dropped(pid) =>+ {+ return Err(error.context("virtual master keyboard delivery failed"));+ }+ Err(error) => tracing::warn!("MPX keyboard fell back to XSendEvent: {error}"),+ }+ }+ if let Some(index) = element_index {+ if !crate::atspi::focus_element(pid, index)? {+ anyhow::bail!("AT-SPI Component.GrabFocus returned false for element {index}");+ }+ }+ match px_target {+ Some((x, y)) => crate::input::send_key_at(xid, x, y, key, modifiers),+ None => crate::input::send_key(xid, key, modifiers),+ }+ .map(|()| KeyRoute::Synthetic)+}++const GRAB_FOCUS_FAILURE_PREFIX: &str = "grab_focus_failed";++/// AT-SPI `Component.GrabFocus` on the addressed element, for the background+/// keyboard routes. Failures carry a prefix so the caller never mistakes them+/// for a delivery failure worth a synthetic-event fallback.+fn grab_focus_for_background(pid: u32, element_index: Option<usize>) -> anyhow::Result<()> {+ let Some(index) = element_index else {+ return Ok(());+ };+ match crate::atspi::focus_element(pid, index) {+ Ok(true) => Ok(()),+ Ok(false) => anyhow::bail!(+ "{GRAB_FOCUS_FAILURE_PREFIX}: AT-SPI Component.GrabFocus returned false for element {index}"+ ),+ Err(error) => Err(error.context(format!(+ "{GRAB_FOCUS_FAILURE_PREFIX}: AT-SPI Component.GrabFocus for element {index}"+ ))),+ }+}++fn is_grab_focus_failure(error: &anyhow::Error) -> bool {+ error.to_string().starts_with(GRAB_FOCUS_FAILURE_PREFIX)+}++/// Merge a background focus-guard report into a finished tool result: the+/// structured `focus_changed` / `focus_restored` / `grab_held_by` fields plus+/// one sentence of text when anything moved.+fn attach_focus_guard(+ mut result: ToolResult,+ report: Option<&crate::input::FocusGuardReport>,+) -> ToolResult {+ let Some(report) = report else {+ return result;+ };+ let summary = report.summary();+ if !summary.is_empty() {+ result+ .content+ .push(cua_driver_core::protocol::Content::text(+ summary.trim().to_owned(),+ ));+ }+ let mut structured = result+ .structured_content+ .take()+ .unwrap_or_else(|| json!({}));+ for (key, value) in report.to_json().as_object().into_iter().flatten() {+ structured[key] = value.clone();+ }+ if let Some(item) = report.evidence_item() {+ push_evidence(&mut structured, item);+ }+ result.structured_content = Some(structured);+ result+}++/// After a pointer press: name the grid cell that holds the widget focus now+/// (LibreOffice reports "D2"), so a one-row miss shows in the click result+/// instead of after six typed values. Read from the focus-event log only,+/// so it costs nothing when the press did not land on a grid.+async fn attach_focused_cell(mut result: ToolResult, pid: u32) -> ToolResult {+ if result.is_error == Some(true) {+ return result;+ }+ let focused = tokio::task::spawn_blocking(move || crate::atspi::focused_control(pid))+ .await+ .ok()+ .flatten();+ let Some((role, name)) = focused.filter(|(role, name)| {+ let role = role.to_ascii_lowercase();+ (role.contains("cell") || role == "table cell") && !name.trim().is_empty()+ }) else {+ return result;+ };+ result+ .content+ .push(cua_driver_core::protocol::Content::text(format!(+ "focus: cell {name}"+ )));+ let mut structured = result+ .structured_content+ .take()+ .unwrap_or_else(|| json!({}));+ structured["focused_cell"] = json!(name);+ push_evidence(+ &mut structured,+ json!({+ "kind": "native_api_result",+ "detail": format!("the accessible holding the focus after the press is {role} \"{name}\""),+ }),+ );+ result.structured_content = Some(structured);+ result+}++/// Append an `evidence[]` item to a structured payload (the public action+/// contract keeps `evidence`, while flat diagnostic fields are reduced away).+fn push_evidence(structured: &mut Value, item: Value) {+ match structured.get_mut("evidence").and_then(Value::as_array_mut) {+ Some(items) => items.push(item),+ None => structured["evidence"] = json!([item]),+ }+}++/// Tool result for a completed keyboard route (`press_key` / `hotkey`).+fn key_route_result(action: &str, route: KeyRoute, mode_label: &str) -> ToolResult {+ match route {+ KeyRoute::Foreground(report) => ToolResult::text(format!(+ "{action} as a real key event (delivery_mode=foreground, focus_after={}); \+ not verified — confirm with a screenshot.",+ report.focus_after.as_str()+ ))+ .with_structured(foreground_structured(+ "key_events_fg",+ report,+ serde_json::Map::new(),+ )),+ KeyRoute::Mpx(report) => {+ let mut structured = report.to_json();+ structured["verified"] = json!(false);+ structured["delivery_mode"] = json!(mode_label);+ structured["effect"] = json!("unverifiable");+ let mut text = format!(+ "{action} as real key events {} (delivery_mode={mode_label}); not verified — \+ confirm with a screenshot.",+ report.route_phrase()+ );+ let closed_target = report+ .focus_guard+ .as_ref()+ .is_some_and(|guard| guard.closed_window.is_some());+ let opened_window = report+ .focus_guard+ .as_ref()+ .and_then(|guard| guard.same_app_window.as_ref())+ .filter(|_| report.delivery_confirmed);+ if closed_target && report.delivery_confirmed {+ // The key closed the window that held the focus (Escape /+ // Return on a dialog): the lost virtual focus is the effect,+ // and the window change is evidence the contract publishes.+ structured["effect"] = json!("confirmed");+ structured["verified"] = json!(true);+ text.push_str(" The window that held the focus closed after the key.");+ } else if let Some(window) = opened_window {+ // The key mapped a new top-level of the target (F4 -> Position+ // and Size, ctrl+1 -> Format Cells): the window change is the+ // effect; name it and the call that targets it.+ structured["effect"] = json!("confirmed");+ structured["verified"] = json!(true);+ structured["window_opened"] = json!({+ "window_id": window.window,+ "title": window.title,+ "focused": window.focused,+ });+ text.push_str(&format!(+ " The key opened window {} \"{}\"; call get_window_state(pid, window_id={}) \+ to index it and act there.",+ window.window, window.title, window.window+ ));+ } else if !report.virtual_focus_held || !report.delivery_confirmed {+ structured["effect"] = json!("suspected_noop");+ text.push_str(+ " Warning: the target lost the virtual keyboard's focus or the server \+ did not confirm the last key; the keys may not have landed.",+ );+ }+ text.push_str(&report.delivery_notes());+ ToolResult::text(text).with_structured(structured)+ }+ KeyRoute::Synthetic | KeyRoute::Terminal => {+ ToolResult::text(format!("{action} (delivery_mode={mode_label})."))+ .with_structured(json!({ "verified": false, "delivery_mode": mode_label }))+ }+ }}fn unavailable_gtk_pointer_background(@@ -2030,11 +3865,15 @@ fn unavailable_chromium_background(pid: u32,delivery: crate::input::delivery::DeliveryMode,) -> Option<ToolResult> {- if chromium_background_must_refuse(- delivery.is_foreground(),- crate::wayland::is_inject_mode(),- is_chromium_embedder(pid),- ) {+ if delivery.is_foreground() || !is_chromium_embedder(pid) {+ return None;+ }+ // Chromium is an XInput2 client: real (non-synthetic) events from the MPX+ // virtual master pointer/keyboard reach its renderer like the private+ // compositor's do, so only the synthetic-XSendEvent-only host must refuse.+ let focus_free_real_input =+ crate::wayland::is_inject_mode() || crate::input::real_pointer_input_available();+ if chromium_background_must_refuse(delivery.is_foreground(), focus_free_real_input, true) {Some(crate::input::delivery::background_unavailable_error(crate::input::delivery::BackgroundUnavailable::ChromiumInput,))@@ -2043,43 +3882,369 @@ fn unavailable_chromium_background(}}-fn chromium_background_must_refuse(- foreground: bool,- focus_free_inject_mode: bool,- chromium: bool,-) -> bool {- chromium && !foreground && !focus_free_inject_mode+fn chromium_background_must_refuse(+ foreground: bool,+ focus_free_inject_mode: bool,+ chromium: bool,+) -> bool {+ chromium && !foreground && !focus_free_inject_mode+}++/// Screen-absolute center of a window (top-left from translate_coordinates plus+/// half its geometry). Used to position the no-focus-steal scroll over the+/// window's content. Blocking — call inside spawn_blocking.+fn window_screen_center(xid: u64) -> anyhow::Result<(i32, i32)> {+ use x11rb::connection::Connection;+ use x11rb::protocol::xproto::ConnectionExt as _;+ use x11rb::rust_connection::RustConnection;++ let (conn, screen_num) = RustConnection::connect(None)?;+ let root = conn.setup().roots[screen_num].root;+ let geom = conn.get_geometry(xid as u32)?.reply()?;+ let trans = conn+ .translate_coordinates(xid as u32, root, 0, 0)?+ .reply()?;+ Ok((+ trans.dst_x as i32 + geom.width as i32 / 2,+ trans.dst_y as i32 + geom.height as i32 / 2,+ ))+}++/// How a background pointer action was delivered on X11, with the cheap+/// post-checks the real-pointer route can make.+#[derive(Debug)]+enum PointerRoute {+ /// Real button events from the session's MPX virtual master pointer.+ Mpx(crate::input::PointerEffect),+ /// A background pixel click resolved to an accessible under the point and+ /// fired its action (or selected it through its container).+ Atspi(crate::atspi::AtPointHit),+ /// Target-addressed synthetic `XSendEvent` (core-only toolkits).+ Synthetic,+ /// Activated the window first and used XTest (delivery_mode=foreground).+ Foreground,+ /// Native Wayland pointer route.+ Wayland,+}++impl PointerRoute {+ /// Result `path`; the Wayland routes keep their own tool-level labels.+ fn path(&self) -> Option<&'static str> {+ match self {+ Self::Mpx(_) => Some(crate::input::MPX_POINTER_PATH),+ Self::Atspi(_) => Some("x11_atspi"),+ Self::Synthetic => Some("x11_xsendevent"),+ Self::Foreground => Some("x11_xtest_fg"),+ Self::Wayland => None,+ }+ }++ /// Structured fields shared by click / double_click / right_click / drag:+ /// `path`, `verified`, `effect`, and for the MPX route the post-checks+ /// (`focus_unchanged`, `region_diff_pct`). `verified:true` + `effect:+ /// "landed"` only when the screen around the point visibly reacted.+ fn structured(&self, mode_label: &str) -> Value {+ let mut v = json!({+ "verified": false,+ "effect": "unverifiable",+ "delivery_mode": mode_label,+ });+ if let Some(path) = self.path() {+ v["path"] = json!(path);+ }+ if let Self::Atspi(hit) = self {+ v["hit"] = json!({+ "role": hit.role,+ "name": hit.name,+ "action": hit.action,+ "path": hit.path,+ });+ if let Some(selected) = &hit.selected {+ v["selected"] = json!(selected);+ }+ // The read-back of the container's selection state is real+ // accessibility evidence; a fired action is only a native call+ // whose effect the caller still confirms.+ let evidence = if hit.selection_verified {+ v["verified"] = json!(true);+ v["effect"] = json!("confirmed");+ json!({+ "kind": "accessibility_readback",+ "detail": hit.describe(),+ })+ } else {+ json!({+ "kind": "native_api_result",+ "detail": hit.describe(),+ })+ };+ v["evidence"] = json!([evidence]);+ }+ if let Self::Mpx(effect) = self {+ v["focus_unchanged"] = json!(effect.focus_unchanged);+ v["screen_point"] = json!([effect.x, effect.y]);+ if let Some((wx, wy)) = effect.window_point {+ v["window_point"] = json!([wx, wy]);+ }+ if let Some(pct) = effect.region_diff_pct {+ v["region_diff_pct"] = json!((pct * 100.0).round() / 100.0);+ }+ v["popups_appeared"] = json!(effect.popups_appeared);+ if let Some(popup) = effect.popups.last() {+ v["popup"] = popup.to_json();+ }+ if !effect.popups.is_empty() {+ v["popups"] = json!(effect+ .popups+ .iter()+ .map(|p| p.to_json())+ .collect::<Vec<_>>());+ }+ // Evidence the action record can publish. A popup (menu, popover,+ // combo list) appearing is a window change the public contract+ // accepts for `effect: confirmed`; a screen-region change is+ // recorded but, per the contract, cannot promote the effect on+ // its own — the text still reports it.+ let mut evidence = Vec::new();+ if effect.popups_appeared > 0 {+ evidence.push(json!({+ "kind": "window_change",+ "detail": format!(+ "{} appeared after the {} at ({}, {}); get_window_state(pid, window_id=<popup window_id>) lists its items",+ popup_list(&effect.popups),+ self.path().unwrap_or("pointer"),+ effect.x,+ effect.y+ ),+ }));+ }+ if let Some(pct) = effect+ .region_diff_pct+ .filter(|pct| *pct >= crate::input::PointerEffect::LANDED_THRESHOLD_PCT)+ {+ evidence.push(json!({+ "kind": "screenshot_comparison",+ "detail": format!("{pct:.2}% of the screen region around ({}, {}) changed", effect.x, effect.y),+ }));+ }+ // The background focus guard ran around the press train: report+ // what the application moved and whether it was restored. An own+ // window it opened is a window change the contract publishes.+ if let Some(guard) = &effect.focus_guard {+ for (key, value) in guard.to_json().as_object().into_iter().flatten() {+ v[key] = value.clone();+ }+ evidence.extend(guard.evidence_item());+ }+ if let Some(foreign) = &effect.foreign_window {+ v["foreign_window"] = foreign.to_json();+ if foreign.changed() {+ evidence.push(json!({+ "kind": "window_change",+ "detail": foreign.describe(),+ }));+ }+ }+ if let Some(cover) = &effect.retargeted_to {+ v["retargeted_to"] = json!({+ "window_id": cover.window,+ "title": cover.title,+ });+ evidence.push(json!({+ "kind": "window_change",+ "detail": format!(+ "the point was under the application's own window {} \"{}\"; the press went to it",+ cover.window, cover.title+ ),+ }));+ }+ if !evidence.is_empty() {+ v["evidence"] = json!(evidence);+ }+ if effect.landed() || effect.foreign_window.as_ref().is_some_and(|f| f.changed()) {+ v["verified"] = json!(true);+ v["effect"] = json!("confirmed");+ }+ }+ v+ }++ /// Human-readable tail for the result text.+ fn text_suffix(&self, mode_label: &str) -> String {+ let mut text = self.text_suffix_inner(mode_label);+ if let Self::Mpx(effect) = self {+ if let Some(popup) = effect.popups.last() {+ text.push_str(&format!(+ " {} opened (popup: window_id={}, bounds x={} y={} {}x{}); call \+ get_window_state(pid={}, window_id={}) to index its items and click them \+ by element_index, or click(window_id={}, x, y) with popup-local pixels.",+ popup.describe(),+ popup.window,+ popup.x,+ popup.y,+ popup.width,+ popup.height,+ popup+ .pid+ .map(|p| p.to_string())+ .unwrap_or_else(|| "<pid>".to_owned()),+ popup.window,+ popup.window+ ));+ }+ if let Some(cover) = &effect.retargeted_to {+ text.push_str(&format!(+ " The point was under this application's own window {} \"{}\" \+ (retargeted_to), so the press went to that window.",+ cover.window, cover.title+ ));+ }+ if let Some(foreign) = &effect.foreign_window {+ text.push(' ');+ text.push_str(&foreign.describe());+ }+ if let Some(guard) = &effect.focus_guard {+ text.push_str(&guard.summary());+ }+ }+ text+ }++ fn text_suffix_inner(&self, mode_label: &str) -> String {+ match self {+ Self::Atspi(hit) if hit.selected.is_some() => format!(+ "(delivery_mode={mode_label}, path=x11_atspi); {}{}",+ hit.describe(),+ if hit.selection_verified {+ " — the item is selected."+ } else {+ "; confirm with a screenshot."+ }+ ),+ Self::Atspi(hit) => format!(+ "(delivery_mode={mode_label}, path=x11_atspi); {}; the AT-SPI action was \+ dispatched, not verified — confirm with a screenshot.",+ hit.describe()+ ),+ Self::Mpx(effect) if effect.landed() => format!(+ "(delivery_mode={mode_label}, path={}, focus {}); {}the screen around the \+ point changed ({:.1}% of the region) — the action landed.",+ crate::input::MPX_POINTER_PATH,+ if effect.focus_unchanged {+ "untouched"+ } else {+ "restored"+ },+ if effect.popups_appeared > 0 {+ format!("{} popup window(s) appeared and ", effect.popups_appeared)+ } else {+ String::new()+ },+ effect.region_diff_pct.unwrap_or(0.0)+ ),+ Self::Mpx(effect) => format!(+ "(delivery_mode={mode_label}, path={}, focus {}); real button events were \+ delivered but the screen around the point did not visibly change \+ ({}) — confirm with a screenshot.",+ crate::input::MPX_POINTER_PATH,+ if effect.focus_unchanged {+ "untouched"+ } else {+ "restored"+ },+ effect+ .region_diff_pct+ .map(|pct| format!("{pct:.2}% of the region"))+ .unwrap_or_else(|| "region capture unavailable".to_owned())+ ),+ _ => format!(+ "(delivery_mode={mode_label}{}); not verified — confirm with a screenshot.",+ self.path()+ .map(|p| format!(", path={p}"))+ .unwrap_or_default()+ ),+ }+ }+}++/// The focus-free real-pointer route existed but the press did not go+/// through, on a toolkit that would silently drop the synthetic fallback.+#[derive(Debug)]+struct BackgroundPointerFailed {+ reason: String,+}++impl std::fmt::Display for BackgroundPointerFailed {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ write!(+ f,+ "background pointer delivery failed: the virtual master pointer route did not \+ go through ({}) and this toolkit drops synthetic pointer events, so no click \+ was delivered",+ self.reason+ )+ }}--/// Screen-absolute center of a window (top-left from translate_coordinates plus-/// half its geometry). Used to position the no-focus-steal scroll over the-/// window's content. Blocking — call inside spawn_blocking.-fn window_screen_center(xid: u64) -> anyhow::Result<(i32, i32)> {- use x11rb::connection::Connection;- use x11rb::protocol::xproto::ConnectionExt as _;- use x11rb::rust_connection::RustConnection;-- let (conn, screen_num) = RustConnection::connect(None)?;- let root = conn.setup().roots[screen_num].root;- let geom = conn.get_geometry(xid as u32)?.reply()?;- let trans = conn- .translate_coordinates(xid as u32, root, 0, 0)?- .reply()?;- Ok((- trans.dst_x as i32 + geom.width as i32 / 2,- trans.dst_y as i32 + geom.height as i32 / 2,- ))+impl std::error::Error for BackgroundPointerFailed {}++/// X11 no-focus-steal pixel click. On a real Xorg host the MPX uinput pointer+/// lands a *true* button event on XInput2 toolkits (GTK3/4, VCL, Qt,+/// Chromium) that silently drop synthetic `XSendEvent` pointers — so right /+/// middle / double clicks actually register. It refuses (typed+/// [`crate::input::TargetOccluded`]) when another toplevel covers the point.+/// On Xvfb / Xtigervnc / unsupported servers (`real_pointer_input_available()`+/// returns false) it falls back to the legacy `XSendEvent` path so headless+/// tests and core-only toolkits keep working; when the MPX attempt itself+/// fails, the fallback is only taken for toolkits that accept synthetic+/// events, otherwise a typed [`BackgroundPointerFailed`] is returned. `lx`,`ly`+/// are window-local; screen-absolute coords for the warp are derived here.+/// Blocking — call inside spawn_blocking.+/// `popup window A "x" (..), popup window B (..)` for evidence text.+fn popup_list(popups: &[crate::input::PopupWindow]) -> String {+ if popups.is_empty() {+ return "a popup window".to_owned();+ }+ popups+ .iter()+ .map(crate::input::PopupWindow::describe)+ .collect::<Vec<_>>()+ .join(", ")+}++/// Every pixel action reports the point it resolved, success or refusal, so+/// a coordinate-frame mistake is visible: `window_point` is what the caller+/// passed (window-local pixels after zoom / resize translation),+/// `screen_point` is where that lands on the screen.+fn with_pixel_points(+ mut result: ToolResult,+ screen: Option<(i32, i32)>,+ window: (i32, i32),+) -> ToolResult {+ let mut structured = result+ .structured_content+ .take()+ .unwrap_or_else(|| json!({}));+ structured["window_point"] = json!([window.0, window.1]);+ let mut note = format!(" [window_point=({}, {})", window.0, window.1);+ if let Some((sx, sy)) = screen {+ structured["screen_point"] = json!([sx, sy]);+ note.push_str(&format!(" screen_point=({sx}, {sy})"));+ }+ note.push(']');+ result.structured_content = Some(structured);+ if let Some(cua_driver_core::protocol::Content::Text { text, .. }) = result.content.first_mut()+ {+ text.push_str(¬e);+ } else {+ result+ .content+ .push(cua_driver_core::protocol::Content::text(+ note.trim().to_owned(),+ ));+ }+ result}-/// X11 no-focus-steal pixel click with graceful fallback. On a real Xorg host-/// the MPX uinput pointer + XI2 shield grab lands a *true* button event on-/// XInput2 toolkits (GTK3/4) that silently drop synthetic `XSendEvent` pointers-/// — so right / middle / double clicks actually register. On Xvfb / Xtigervnc /-/// unsupported servers (`real_pointer_input_available()` returns false) or if-/// the MPX attempt fails, it falls back to the legacy `XSendEvent` path so-/// headless tests and core-only toolkits keep working. `lx`,`ly` are-/// window-local; screen-absolute coords for the warp are derived here. Blocking-/// — call inside spawn_blocking.fn x11_pixel_click_no_focus_steal(cursor_id: &str,xid: u64,@@ -2087,36 +4252,168 @@ fn x11_pixel_click_no_focus_steal(ly: i32,button: u8,count: usize,-) -> anyhow::Result<()> {+) -> anyhow::Result<PointerRoute> {if crate::input::real_pointer_input_available() {if let Ok((sx, sy)) = window_local_to_screen(xid, lx as f64, ly as f64) {- match crate::input::send_virtual_pointer_click(- cursor_id,- &crate::input::VirtualPointerClick {- target_window: xid,- x: sx.round() as i32,- y: sy.round() as i32,- button,- count,- },- ) {- Ok(()) => return Ok(()),+ // The press train has its own save/restore fast path; the focus+ // guard around it catches what the application does afterwards+ // (a menu grab, a dialog it maps) and reports it.+ let target_pid = crate::x11::window_pid(xid);+ let (outcome, guard) = crate::input::focus_guard::guarded_settled(target_pid, || {+ Ok(crate::input::send_virtual_pointer_click(+ cursor_id,+ &crate::input::VirtualPointerClick {+ target_window: xid,+ x: sx.round() as i32,+ y: sy.round() as i32,+ button,+ count,+ },+ ))+ })?;+ match outcome {+ Ok(mut effect) => {+ effect.focus_guard = guard;+ effect.window_point = Some((lx, ly));+ return Ok(PointerRoute::Mpx(effect));+ }Err(error) if crate::input::is_uinput_unavailable(&error) => return Err(error),- Err(e) => tracing::warn!("MPX click fell back to XSendEvent: {e}"),+ Err(error)+ if error+ .downcast_ref::<crate::input::TargetOccluded>()+ .is_some()+ || error+ .downcast_ref::<crate::input::PointOutsideWindow>()+ .is_some() =>+ {+ return Err(error)+ }+ // The synthetic fallback is a silent no-op on GTK/VCL/Qt:+ // report the real-input failure instead of a click that+ // changed nothing.+ Err(error)+ if crate::x11::window_pid(xid).is_some_and(synthetic_pointer_is_dropped) =>+ {+ tracing::warn!("MPX click failed on a toolkit that drops synthetic pointer events: {error:#}");+ return Err(BackgroundPointerFailed {+ reason: format!("{error:#}"),+ }+ .into());+ }+ Err(e) => tracing::warn!("MPX click fell back to XSendEvent: {e:#}"),}}}- crate::input::send_click(xid, lx, ly, count, button)+ crate::input::send_click(xid, lx, ly, count, button).map(|()| PointerRoute::Synthetic)}+/// Typed tool error for a Linux input failure: uinput permission, an occluded+/// target, or a real-pointer delivery that did not go through. Everything+/// else keeps its full error chain in the text.fn linux_input_error(error: anyhow::Error) -> ToolResult {if crate::input::is_uinput_unavailable(&error) {- ToolResult::error(error.to_string()).with_structured(json!({+ return ToolResult::error(error.to_string()).with_structured(json!({"code": crate::input::UINPUT_UNAVAILABLE_CODE,- }))- } else {- ToolResult::error(error.to_string())- }+ }));+ }+ if let Some(occluded) = error.downcast_ref::<crate::input::TargetOccluded>() {+ let hint = format!(+ "The point is under window {}{}{}, which belongs to a different application, so \+ a background pointer press there would hit that window instead of the target; \+ no input was sent. Either act on that window (pass its window_id{} and click / \+ press keys there, or dismiss it), click the target by element_index (an AT-SPI \+ action that needs no pointer), or pick a point of the target that is not \+ covered (list_windows gives bounds and z_index).",+ occluded.covering_window,+ if occluded.covering_title.is_empty() {+ String::new()+ } else {+ format!(" \"{}\"", occluded.covering_title)+ },+ occluded+ .covering_pid+ .map(|pid| format!(" of pid {pid}"))+ .unwrap_or_default(),+ occluded+ .covering_pid+ .map(|pid| format!(" with pid {pid}"))+ .unwrap_or_default(),+ );+ return ToolResult::error(format!("{occluded}. {hint}")).with_structured(json!({+ "code": "target_occluded",+ "effect": "refused",+ "path": crate::input::MPX_POINTER_PATH,+ "target_window": occluded.target_window,+ "covering_window": occluded.covering_window,+ "covering_title": occluded.covering_title,+ "covering_pid": occluded.covering_pid,+ "screen_point": [occluded.x, occluded.y],+ "hint": hint,+ }));+ }+ if let Some(grab) = error.downcast_ref::<crate::input::PopupKeyboardGrab>() {+ let hint = format!(+ "Keys for pid {} cannot be delivered while its {} is open: the popup's keyboard \+ grab drops keys from the virtual keyboard, and the core keyboard would go to the \+ focused application instead. Dismiss the popup first (click outside it, or \+ click an item in it with pid/window_id={}), or click the popup's item by \+ element_index after get_window_state(pid={}, window_id={}); then retry.",+ grab.pid,+ grab.popup.describe(),+ grab.popup.window,+ grab.pid,+ grab.popup.window+ );+ return ToolResult::error(format!("{grab}. {hint}")).with_structured(json!({+ "code": "popup_keyboard_grab",+ "effect": "refused",+ "verified": false,+ "popup": grab.popup.to_json(),+ "focus_owner_pid": grab.focus_owner,+ "hint": hint,+ }));+ }+ if let Some(outside) = error.downcast_ref::<crate::input::PointOutsideWindow>() {+ let (bx, by, bw, bh) = outside.bounds;+ let hint = format!(+ "x/y are window-local pixels of window {} (0..{bw} x 0..{bh}, its top-left is at \+ screen ({bx}, {by})). Pass window-local coordinates from get_window_state's \+ screenshot, or screen pixels with coordinate_frame:\"desktop\".",+ outside.target_window+ );+ return ToolResult::error(format!("{outside}. {hint}")).with_structured(json!({+ "code": "point_outside_window",+ "effect": "refused",+ "path": crate::input::MPX_POINTER_PATH,+ "target_window": outside.target_window,+ "screen_point": [outside.screen_x, outside.screen_y],+ "window_point": [outside.window_x, outside.window_y],+ "window_bounds": { "x": bx, "y": by, "width": bw, "height": bh },+ "hint": hint,+ }));+ }+ if let Some(unusable) = error.downcast_ref::<ElementBoundsUnusable>() {+ let hint = "Re-snapshot with get_window_state and click by pixel (x/y) from the \+ screenshot, or pick a child element that reports a frame.";+ return ToolResult::error(format!("{unusable}. {hint}")).with_structured(json!({+ "code": "element_bounds_unavailable",+ "effect": "none",+ "element_index": unusable.idx,+ "hint": hint,+ }));+ }+ if let Some(failed) = error.downcast_ref::<BackgroundPointerFailed>() {+ let hint = "Click by element_index (AT-SPI action) or retry with \+ delivery_mode:\"foreground\".";+ return ToolResult::error(format!("{failed}. {hint}")).with_structured(json!({+ "code": "background_pointer_failed",+ "effect": "none",+ "path": crate::input::MPX_POINTER_PATH,+ "reason": failed.reason,+ "hint": hint,+ }));+ }+ ToolResult::error(format!("{error:#}"))}fn isolated_hyprland_background(delivery: crate::input::delivery::DeliveryMode) -> bool {@@ -2129,6 +4426,246 @@ fn hyprland_foreground(delivery: crate::input::delivery::DeliveryMode) -> bool {&& crate::wayland::hyprland::is_session()}+/// AT-SPI roles whose `activate`/`press` action does not move the widget+/// focus into them, so a following keystroke would land elsewhere; a real+/// pointer click at their centre is what a user does.+fn element_needs_real_click(role: &str) -> bool {+ crate::atspi::is_focus_taking_role(role)+}++/// Menubar entries and the items of an open menu.+fn element_is_menu_role(role: &str) -> bool {+ matches!(+ role.trim().to_ascii_lowercase().as_str(),+ "menu" | "menu item" | "check menu item" | "radio menu item"+ )+}++/// A chord the window manager, not the application, acts on.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+enum WmChord {+ /// Alt+F4: close the target window (`_NET_CLOSE_WINDOW`).+ CloseWindow,+ /// Super+*, Alt+Tab, Ctrl+Alt+*, Alt+F7/F8/F10/space/Escape: a WM binding+ /// with no window-addressed equivalent.+ Unavailable,+}++/// mutter's key bindings are passive grabs on the core keyboard. A chord from+/// the session's virtual master keyboard (background delivery) reaches the+/// client window, never the WM, so `alt+F4` / `super+a` / `alt+tab` silently+/// do nothing. Recognise them so the tool can act (close the window through+/// EWMH) or refuse with a typed reason instead of reporting `unverifiable`.+fn wm_chord_kind(key: &str, modifiers: &[String]) -> Option<WmChord> {+ let mods: Vec<String> = modifiers+ .iter()+ .map(|m| m.trim().to_ascii_lowercase())+ .collect();+ let has = |names: &[&str]| mods.iter().any(|m| names.contains(&m.as_str()));+ let alt = has(&["alt", "alt_l", "alt_r", "option"]);+ let ctrl = has(&["ctrl", "control", "ctrl_l", "ctrl_r"]);+ let shift = has(&["shift", "shift_l", "shift_r"]);+ let super_ = has(&[+ "super", "super_l", "super_r", "meta", "win", "cmd", "command", "hyper",+ ]);+ let key_lower = key.trim().to_ascii_lowercase();+ if super_ {+ return Some(WmChord::Unavailable);+ }+ if alt && ctrl {+ return Some(WmChord::Unavailable);+ }+ if alt && !ctrl && key_lower == "f4" {+ return Some(WmChord::CloseWindow);+ }+ if alt+ && !ctrl+ && !shift+ && matches!(+ key_lower.as_str(),+ "tab" | "f5" | "f7" | "f8" | "f10" | "space" | "escape" | "esc"+ )+ {+ return Some(WmChord::Unavailable);+ }+ if alt && shift && key_lower == "tab" {+ return Some(WmChord::Unavailable);+ }+ None+}++/// Background delivery of a WM-level chord: close the window for Alt+F4,+/// refuse the rest (`wm_chord_unavailable`). `None` for ordinary chords.+async fn wm_chord_background(+ pid: u32,+ xid: u64,+ key: &str,+ modifiers: &[String],+ delivery: crate::input::delivery::DeliveryMode,+) -> Option<ToolResult> {+ if delivery.is_foreground()+ || crate::wayland::wayland_input_enabled()+ || crate::wayland::is_inject_mode()+ {+ return None;+ }+ let chord = wm_chord_kind(key, modifiers)?;+ let display = format!("{}+{}", modifiers.join("+"), key);+ match chord {+ WmChord::CloseWindow => Some(close_window_background(pid, xid, &display).await),+ WmChord::Unavailable => {+ let hint = format!(+ "{display} is a window-manager binding (mutter's passive grab on the core \+ keyboard); a virtual-keyboard chord in background mode reaches the window, \+ not the WM, so it would do nothing. Use the driver instead: launch_app / \+ list_windows / bring_to_front to switch windows, hotkey alt+F4 (or \+ kill_app) to close one, or retry with delivery_mode:\"foreground\"."+ );+ Some(+ ToolResult::error(format!("Refused {display} on pid {pid}: {hint}"))+ .with_structured(json!({+ "code": "wm_chord_unavailable",+ "effect": "refused",+ "verified": false,+ "delivery_mode": "background",+ "chord": display,+ "hint": hint,+ })),+ )+ }+ }+}++/// Alt+F4 in background mode: ask the WM to close the window+/// (`_NET_CLOSE_WINDOW` -> `WM_DELETE_WINDOW`) and watch it go.+async fn close_window_background(pid: u32, xid: u64, display: &str) -> ToolResult {+ let before: Vec<crate::x11::WindowInfo> =+ tokio::task::spawn_blocking(move || crate::x11::list_windows(Some(pid)))+ .await+ .unwrap_or_default();+ let title = before+ .iter()+ .find(|w| w.xid == xid)+ .map(|w| w.title.clone())+ .unwrap_or_default();+ let sent = tokio::task::spawn_blocking(move || crate::x11::close_window(xid, pid)).await;+ match sent {+ Ok(Ok(())) => {}+ Ok(Err(error)) => {+ return ToolResult::error(format!(+ "{display} on pid {pid}: could not request the window manager to close \+ window {xid}: {error:#}"+ ))+ .with_structured(json!({+ "code": "close_window_failed",+ "effect": "none",+ "detail": error.to_string(),+ }))+ }+ Err(error) => return ToolResult::error(format!("Task error: {error}")),+ }+ let deadline = std::time::Instant::now() + std::time::Duration::from_millis(2000);+ let mut closed = false;+ while std::time::Instant::now() < deadline {+ tokio::time::sleep(std::time::Duration::from_millis(100)).await;+ let viewable =+ tokio::task::spawn_blocking(move || crate::x11::window_is_viewable(xid)).await;+ if matches!(viewable, Ok(false)) {+ closed = true;+ break;+ }+ }+ let after: Vec<crate::x11::WindowInfo> =+ tokio::task::spawn_blocking(move || crate::x11::list_windows(Some(pid)))+ .await+ .unwrap_or_default();+ let new_windows: Vec<&crate::x11::WindowInfo> = after+ .iter()+ .filter(|w| w.is_on_screen && !before.iter().any(|old| old.xid == w.xid))+ .collect();+ let quoted = if title.is_empty() {+ String::new()+ } else {+ format!(" \"{title}\"")+ };+ // The close request is an XSendEvent client message to the root (the+ // transport the action contract knows as x11_xsendevent).+ let mut structured = json!({+ "path": "x11_xsendevent",+ "mechanism": "_NET_CLOSE_WINDOW",+ "delivery_mode": "background",+ "chord": display,+ "target_window": xid,+ "window_closed": closed,+ });+ if closed {+ structured["verified"] = json!(true);+ structured["effect"] = json!("confirmed");+ structured["closed_window"] = json!(xid);+ structured["evidence"] = json!([{+ "kind": "window_change",+ "detail": format!("window {xid}{quoted} of pid {pid} closed after _NET_CLOSE_WINDOW"),+ }]);+ let mut text = format!(+ "Pressed {display} on pid {pid} as a window-manager close request \+ (_NET_CLOSE_WINDOW -> WM_DELETE_WINDOW; a virtual-keyboard Alt+F4 never reaches \+ mutter): window {xid}{quoted} closed."+ );+ if let Some(w) = new_windows.first() {+ text.push_str(&format!(+ " The application then opened window {} \"{}\".",+ w.xid, w.title+ ));+ structured["opened_window"] = json!({ "window_id": w.xid, "title": w.title });+ }+ ToolResult::text(text).with_structured(structured)+ } else {+ structured["verified"] = json!(false);+ structured["effect"] = json!("unverifiable");+ let mut text = format!(+ "Pressed {display} on pid {pid} as a window-manager close request \+ (_NET_CLOSE_WINDOW -> WM_DELETE_WINDOW): window {xid}{quoted} is still mapped \+ after 2 s."+ );+ if let Some(w) = new_windows.first() {+ text.push_str(&format!(+ " The application opened window {} \"{}\" instead (a save / confirm \+ dialog?); act on it with get_window_state / click.",+ w.xid, w.title+ ));+ structured["opened_window"] = json!({ "window_id": w.xid, "title": w.title });+ structured["evidence"] = json!([{+ "kind": "window_change",+ "detail": format!("the application opened window {} \"{}\" after the close request", w.xid, w.title),+ }]);+ } else {+ text.push_str(" Confirm with list_windows; kill_app force-closes the application.");+ }+ ToolResult::text(text).with_structured(structured)+ }+}++/// `press_key` accepts `"alt+F4"` / `"ctrl+shift+t"` style keys: everything+/// before the last `+` is a modifier. A bare `"+"` stays the plus key.+fn split_key_combo(key: &str) -> (Vec<String>, String) {+ if key.len() <= 1 || !key.contains('+') {+ return (Vec::new(), key.to_owned());+ }+ let mut parts: Vec<&str> = key.split('+').collect();+ let last = parts.pop().unwrap_or("");+ let last = if last.is_empty() { "+" } else { last };+ let modifiers: Vec<String> = parts+ .into_iter()+ .filter(|part| !part.is_empty())+ .map(str::to_owned)+ .collect();+ if !modifiers.is_empty() && modifiers.iter().all(|m| is_modifier(m)) {+ (modifiers, last.to_owned())+ } else {+ (Vec::new(), key.to_owned())+ }+}+fn element_click_prefers_ax(hyprland_foreground: bool,button: u8,@@ -3084,14 +5621,7 @@ fn mouse_button_name(button: u8) -> &'static str {}fn resolve_cursor_key(args: &Value) -> String {- for key in ["session", "_session_id", "cursor_id"] {- if let Some(v) = args.get(key).and_then(|v| v.as_str()) {- if !v.is_empty() {- return v.to_owned();- }- }- }- "default".to_owned()+ cua_driver_core::tool_args::resolve_session_key(args)}/// Return the cursor key only for a lifecycle-owned session. Cursor positioning@@ -3641,123 +6171,377 @@ fn capture_admission_error(error: anyhow::Error) -> ToolResult {}static CLICK_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();+/// Where a foreground element click may put the real pointer.+enum ForegroundElementPlacement {+ /// The element centre lies inside the target window (or a popup of the+ /// same process): XTest-click it there.+ Point { xid: u64, sx: f64, sy: f64 },+ /// The element has no usable on-screen bounds (its menu is closed, its+ /// page is unmapped, its centre is outside every window of the process):+ /// only the AT-SPI action can reach it.+ NoPoint { reason: String },+}+impl ClickTool {- /// Resolve the snapshot's retained object identity once. The current- /// ordinal is never used to select a target after the observation.- async fn click_indexed_x11(+ /// X11 foreground element click. Behave like a user when the element is+ /// really on screen: activate the window and XTest-click the element's+ /// centre. A degenerate centre — the walk answering (0,0) for a menu item+ /// whose menu is closed used to glide the real pointer into the GNOME hot+ /// corner — is never clicked; the observed object's AT-SPI action is fired+ /// instead (`path: "ax_fg"`, a label the core's action record maps to+ /// the AT-SPI transport), with the foreground activation kept, and when+ /// that is not possible either the call is refused+ /// (`element_bounds_unavailable`).+ #[allow(clippy::too_many_arguments)]+ async fn foreground_element_click(&self,+ cursor_id: &str,pid: u32,idx: usize,+ observed: &crate::atspi::cache::CachedElement,xid_hint: Option<u64>,- identity: crate::atspi::AtspiIdentity,+ placement: Option<(u64, f64, f64)>,button: u8,count: usize,- modifiers: Vec<String>,- delivery: crate::input::delivery::DeliveryMode,- cursor_id: String,+ modifiers: &[String],) -> ToolResult {- let Some(xid) = xid_hint.filter(|xid| *xid != 0) else {- return ToolResult::error("Indexed click requires an observed exact X11 window");- };- let resolved = tokio::task::spawn_blocking(move || -> anyhow::Result<_> {- let target = crate::atspi::resolve_observed_click_target(pid, idx, xid, &identity)?;- let center = target- .screen_bounds()- .ok()- .map(|(x, y, w, h)| (x as f64 + w as f64 / 2.0, y as f64 + h as f64 / 2.0));- Ok((xid, target, center))+ let observed_for_state = observed.clone();+ let decided = tokio::task::spawn_blocking(move || {+ decide_foreground_element_placement(pid, &observed_for_state, xid_hint, placement)}).await;- let (xid, target, center) = match resolved {- Ok(Ok(target)) => target,- Ok(Err(error)) => {- return ToolResult::error(format!("AT-SPI element resolution failed: {error}"))- }- Err(error) => return ToolResult::error(format!("Task error: {error}")),+ let (xid, decision) = match decided {+ Ok(v) => v,+ Err(e) => return ToolResult::error(format!("Task error: {e}")),};- if let Some((sx, sy)) = center {- crate::overlay::send_command_for(- cursor_id.clone(),- cursor_overlay::OverlayCommand::PinAbove(xid),- );- reveal_pointer_action_for(&self.state, &cursor_id, sx, sy, true).await;- }- let result = tokio::task::spawn_blocking(move || -> anyhow::Result<ToolResult> {- target.verify_live()?;- let action = target.perform_action(modifiers.is_empty() && button == 1 && count == 1);- let (path, suspected_noop) = match action {- Ok((_, suspected_noop)) => ("ax", suspected_noop),- Err(error)- if error.is::<crate::atspi::ClickActionUnavailable>()- || error.is::<crate::atspi::ElementClickNeedsForeground>() =>- {- if let Some(refusal) = unavailable_chromium_background(pid, delivery) {- return Ok(refusal);+ match decision {+ ForegroundElementPlacement::Point { xid, sx, sy } => {+ // Ask the toolkit who actually owns this screen point before+ // pressing it: a snapshot frame is where the toolkit *said*+ // the element is, and a hidden sibling view (a QFileDialog's+ // off-page QListView mirroring the visible QTreeView) can+ // report a frame nothing is drawn at. See atspi::native::hit_test.+ // Web content is exempt: Chromium's GetAccessibleAtPoint+ // answers with an unrelated `panel` for web elements, so it+ // cannot vouch for (or against) the point.+ let owned_ref = observed+ .object_ref+ .clone()+ .filter(|_| !observed.in_web_content);+ let (sx, sy, redirect_note) = if let Some(oref) = owned_ref {+ let (pid_c, oref_c, sx_i, sy_i) =+ (pid, oref, sx.round() as i32, sy.round() as i32);+ let ownership = tokio::task::spawn_blocking(move || {+ crate::atspi::native::hit_test::point_ownership(+ pid_c, xid, &oref_c, sx_i, sy_i,+ )+ })+ .await+ .unwrap_or(crate::atspi::native::hit_test::PointOwnership::Unknown);+ match ownership {+ crate::atspi::native::hit_test::PointOwnership::Redirect {+ target,+ screen_point,+ ..+ } => (+ screen_point.0 as f64,+ screen_point.1 as f64,+ Some(format!("redirected the press to {}", target.describe())),+ ),+ crate::atspi::native::hit_test::PointOwnership::Mismatch { owner } => {+ return ToolResult::error(format!(+ "click: element [{idx}] (pid {pid}) has a snapshot frame at \+ ({sx_i}, {sy_i}), but that screen point is owned by {} rather \+ than the observed element; a real pointer click there would hit \+ the wrong control. Re-snapshot with get_window_state, or click by \+ pixel from a screenshot.",+ owner.describe()+ ))+ .with_structured(json!({+ "code": "element_bounds_unavailable",+ "effect": "none",+ "element_index": idx,+ "reason": "point_owned_by_another_element",+ "owner": owner.describe(),+ }));+ }+ _ => (sx, sy, None),}- let local_center = || -> anyhow::Result<(f64, f64)> {- let (x, y, w, h) = target.screen_bounds()?;- let (ox, oy) = window_local_to_screen(xid, 0.0, 0.0)?;- Ok((- x as f64 + w as f64 / 2.0 - ox,- y as f64 + h as f64 / 2.0 - oy,+ } else {+ (sx, sy, None)+ };+ crate::overlay::send_command_for(+ cursor_id.to_owned(),+ cursor_overlay::OverlayCommand::PinAbove(xid),+ );+ reveal_pointer_action_for(&self.state, cursor_id, sx, sy, true).await;+ let modifier_owned = modifiers.to_vec();+ let result = spawn_blocking_bounded(+ "foreground element click",+ foreground_budget(0),+ move || {+ let modifier_refs: Vec<&str> =+ modifier_owned.iter().map(String::as_str).collect();+ crate::input::with_x11_foreground_opts(+ xid,+ crate::input::ForegroundOptions::pointer(),+ || {+ crate::input::send_click_xtest_desktop_with_modifiers(+ sx.round() as i32,+ sy.round() as i32,+ button,+ count,+ &modifier_refs,+ )+ },+ )+ },+ )+ .await;+ let window_point = crate::x11::window_info(xid)+ .map(|w| (sx.round() as i32 - w.x, sy.round() as i32 - w.y))+ .unwrap_or((sx.round() as i32, sy.round() as i32));+ match result {+ Ok(Ok(((), report))) => {+ let confirmed = report.focus_kept() && report.window_change.is_some();+ let result = ToolResult::text(format!(+ "Clicked element [{idx}] (pid {pid}) with a real pointer click \+ (delivery_mode=foreground, focus_after={}){}{}",+ report.focus_after.as_str(),+ if confirmed {+ format!("; {}.", report.window_change.as_deref().unwrap_or(""))+ } else {+ "; not verified — confirm with a screenshot.".to_owned()+ },+ redirect_note+ .as_deref()+ .map(|n| format!(" ({n})."))+ .unwrap_or_default(),))- };- let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();- let (lx, ly) = local_center()?;- let path = if !delivery.is_foreground() && target.needs_foreground_pointer() {- return Ok(crate::input::delivery::background_unavailable_error(- crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,+ .with_structured(foreground_structured(+ "x11_xtest_fg",+ report,+ serde_json::Map::new(),));- } else if delivery.is_foreground() {- crate::input::with_x11_foreground(xid, 80, || {- let (lx, ly) = local_center()?;- let (sx, sy) = window_local_to_screen(xid, lx, ly)?;- crate::input::send_click_xtest_desktop_with_modifiers(- sx.round() as i32,- sy.round() as i32,- button,- count,- &modifier_refs,- )- })?;- "x11_xtest_fg"- } else {- crate::input::send_click_with_modifiers(+ with_pixel_points(+ result,+ Some((sx.round() as i32, sy.round() as i32)),+ window_point,+ )+ }+ Ok(Err(e)) => input_error_result(e),+ Err(e) => ToolResult::error(format!("Task error: {e}")),+ }+ }+ ForegroundElementPlacement::NoPoint { reason } => {+ let Some(xid) = xid else {+ return ToolResult::error(format!(+ "click: element [{idx}] (pid {pid}) has no usable on-screen bounds \+ ({reason}) and pid {pid} has no window to activate; no input was sent."+ ))+ .with_structured(json!({+ "code": "element_bounds_unavailable",+ "effect": "none",+ "element_index": idx,+ "reason": reason,+ }));+ };+ if observed.object_ref.is_none()+ || button != 1+ || count != 1+ || !modifiers.is_empty()+ {+ return ToolResult::error(format!(+ "click: element [{idx}] (pid {pid}) has no usable on-screen bounds \+ ({reason}); a real pointer click cannot reach it and this click \+ shape (button {button} × {count}{}) has no AT-SPI action form. No \+ input was sent. Open the containing menu first, re-snapshot with \+ get_window_state, or click by pixel from a screenshot.",+ if modifiers.is_empty() { String::new() } else { format!(", modifiers {modifiers:?}") }+ ))+ .with_structured(json!({+ "code": "element_bounds_unavailable",+ "effect": "none",+ "element_index": idx,+ "reason": reason,+ "suggestion": "open the containing menu first, or re-snapshot with get_window_state and click by x/y",+ }));+ }+ crate::overlay::send_command_for(+ cursor_id.to_owned(),+ cursor_overlay::OverlayCommand::PinAbove(xid),+ );+ let observed_for_ax = observed.clone();+ let result = spawn_blocking_bounded(+ "foreground element action",+ foreground_budget(0),+ move || {+ crate::input::with_x11_foreground_opts(xid,- lx.round() as i32,- ly.round() as i32,- count,- button,- &modifier_refs,- )?;- "x11_pixel"- };- (path, false)+ crate::input::ForegroundOptions::pointer(),+ || crate::atspi::perform_action_observed(&observed_for_ax),+ )+ },+ )+ .await;+ match result {+ Ok(Ok(((action, suspected_noop, unacknowledged), report))) => {+ let mut extra = serde_json::Map::new();+ extra.insert("ax_action".into(), json!(action));+ extra.insert("bounds_reason".into(), json!(reason));+ if suspected_noop {+ extra.insert("effect".into(), json!("suspected_noop"));+ extra.insert("escalation".into(), non_ax_escalation());+ } else if unacknowledged {+ // No doAction reply arrived in time: never let a+ // coincidental window-change observation promote+ // this to `confirmed` — the AT-SPI layer itself+ // never confirmed dispatch, let alone effect.+ extra.insert("effect".into(), json!("unverifiable"));+ extra.insert("verified".into(), json!(false));+ extra.insert(+ "note".into(),+ json!(+ "no acknowledgement received for the doAction call within \+ the AT-SPI timeout; its effect is unknown"+ ),+ );+ }+ ToolResult::text(format!(+ "Clicked element [{idx}] (pid {pid}) through its AT-SPI action \+ \"{action}\" with the window activated (delivery_mode=foreground, \+ path=ax_fg, focus_after={}): the element had no usable \+ on-screen bounds ({reason}), so no pointer was moved.{}",+ report.focus_after.as_str(),+ match &report.window_change {+ Some(change) => format!(" {change}."),+ None => " Not verified — confirm with a screenshot.".to_owned(),+ }+ ))+ .with_structured(foreground_structured("ax_fg", report, extra))+ }+ Ok(Err(error)) => {+ if error.is::<crate::atspi::native::CachedElementGone>() {+ return ToolResult::error(format!(+ "click: stale_element_token: observed AT-SPI object [{idx}] is no \+ longer present ({error}); re-snapshot with get_window_state"+ ))+ .with_structured(json!({+ "code": "stale_element_token",+ "effect": "none",+ }));+ }+ if crate::input::foreground::error_code(&error).is_some() {+ return input_error_result(error);+ }+ ToolResult::error(format!(+ "click: element [{idx}] (pid {pid}) has no usable on-screen bounds \+ ({reason}) and its AT-SPI action failed ({error:#}); no input was \+ sent. Open the containing menu first, re-snapshot with \+ get_window_state, or click by pixel from a screenshot."+ ))+ .with_structured(json!({+ "code": "element_bounds_unavailable",+ "effect": "none",+ "element_index": idx,+ "reason": reason,+ "ax_error": format!("{error:#}"),+ }))+ }+ Err(e) => ToolResult::error(format!("Task error: {e}")),}- Err(error) => return Err(error),- };- let mut structured = json!({- "path": path,- "verified": false,- "effect": if suspected_noop { "suspected_noop" } else { "unverifiable" },- });- if suspected_noop {- structured["escalation"] = non_ax_escalation();}- Ok(- ToolResult::text(format!("Clicked element [{idx}] (pid {pid})."))- .with_structured(structured),- )- })- .await;- match result {- Ok(Ok(result)) => result,- Ok(Err(error)) => ToolResult::error(format!("AT-SPI element click failed: {error}")),- Err(error) => ToolResult::error(format!("Task error: {error}")),}}}+/// Decide whether a foreground element click has a real point to press.+/// Returns the window to activate (when the process has one) and the+/// placement decision. Runs blocking X11 / AT-SPI queries.+fn decide_foreground_element_placement(+ pid: u32,+ observed: &crate::atspi::cache::CachedElement,+ xid_hint: Option<u64>,+ placement: Option<(u64, f64, f64)>,+) -> (Option<u64>, ForegroundElementPlacement) {+ let xid = xid_hint+ .or_else(|| placement.map(|(xid, ..)| xid).filter(|xid| *xid != 0))+ .or_else(|| {+ crate::x11::list_windows(Some(pid))+ .into_iter()+ .next()+ .map(|w| w.xid)+ });+ // The object's own state decides whether it is on screen at all: a menu+ // item in a closed menu still answers a (stale or zero) extent.+ let showing = match crate::atspi::element_showing_observed(observed) {+ Ok(showing) => Some(showing),+ Err(error) => {+ tracing::debug!(+ "foreground element [{}] showing-state unavailable: {error:#}",+ observed.key+ );+ None+ }+ };+ if showing == Some(false) {+ return (+ xid,+ ForegroundElementPlacement::NoPoint {+ reason: format!(+ "the {} is not showing (its menu or page is closed)",+ observed.role+ ),+ },+ );+ }+ let Some((placed_xid, sx, sy)) = placement.filter(|(xid, ..)| *xid != 0) else {+ return (+ xid,+ ForegroundElementPlacement::NoPoint {+ reason: "no usable on-screen bounds were resolved for it".to_owned(),+ },+ );+ };+ let (px, py) = (sx.round() as i32, sy.round() as i32);+ let inside = |w: &crate::x11::WindowInfo| {+ px >= w.x && py >= w.y && px < w.x + w.width as i32 && py < w.y + w.height as i32+ };+ if crate::x11::window_info(placed_xid).is_some_and(|w| inside(&w)) {+ return (+ xid,+ ForegroundElementPlacement::Point {+ xid: placed_xid,+ sx,+ sy,+ },+ );+ }+ // The centre is outside the named window: a popup menu of the same+ // process (an open menu's item) is the one legitimate case.+ if let Some(popup) = crate::input::popup_under_screen_point(px, py) {+ if popup.pid.is_none_or(|p| p == pid) {+ return (+ xid,+ ForegroundElementPlacement::Point {+ xid: placed_xid,+ sx,+ sy,+ },+ );+ }+ }+ (+ xid,+ ForegroundElementPlacement::NoPoint {+ reason: format!(+ "its centre ({px}, {py}) lies outside window {placed_xid} and every popup of \+ pid {pid}"+ ),+ },+ )+}+#[async_trait]impl Tool for ClickTool {fn def(&self) -> &ToolDef {@@ -3768,12 +6552,23 @@ impl Tool for ClickTool {a stable handle, and tells you what you're clicking via the cached AT-SPI \element's role + label. Reach for `x, y` only when the target is a canvas / \custom-drawn surface that doesn't appear in the AT-SPI tree.\n\n\- Provide either (window_id + x/y) or (pid + element_index). Routes via \- XSendEvent (no focus steal). element_index cache is scoped per (pid, \+ Provide either (window_id + x/y) or (pid + element_index). x/y are \+ WINDOW-LOCAL pixels of window_id (the get_window_state screenshot frame); \+ a point outside the window is refused (point_outside_window) and every \+ pixel result reports the resolved window_point / screen_point. Background \+ delivery first resolves the accessible under the point and fires its action \+ (the result says `hit: <role> \"<name>\" action=...`; a file / list item is \+ selected through its container: `selected: ...`), else presses the real \+ virtual pointer there. A click that opens a popup menu names it (`popup: \+ {window_id, bounds}`): pass that window_id to get_window_state to index its \+ items. element_index cache is scoped per (pid, \window_id) and is replaced by the next get_window_state of the same window — \re-snapshot every turn before clicking.\n\n\After a zoom call, pass from_zoom=true to auto-translate zoom-image coords \back to full-window space.\n\n\+ `count`: 1 (default), 2 for a double-click, 3 for a triple-click (selects a \+ line / paragraph in editors) — one press train with real double-click cadence, \+ in background and foreground alike; there is no separate triple_click tool.\n\n\button: \"left\" (default), \"right\", or \"middle\". Defaults to left so the \field is fully back-compat. X11: routes through XSendEvent ButtonPress/Release \with the matching button code. Native Wayland: only left-button is supported \@@ -3791,8 +6586,8 @@ impl Tool for ClickTool {"cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},"pid":{"type":"integer"},"window_id":{"type":"integer"},- "x":{"type":"number"},- "y":{"type":"number"},+ "x":{"type":"number","description":"Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\")."},+ "y":{"type":"number","description":"Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x."},"element_index": cua_driver_core::tool_schema::element_index_schema(),"element_token": cua_driver_core::tool_schema::element_token_schema(),"snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),@@ -3801,10 +6596,11 @@ impl Tool for ClickTool {// [left,right,middle]); kept inline to carry the Linux/Wayland// back-compat prose the click button-schema test asserts on."button":{"type":"string","enum":["left","right","middle"],"description":"Mouse button. Default: \"left\" (legacy back-compat). X11: routed via ButtonPress/Release with the matching evdev code. Native Wayland: only left-button is supported via the virtual-pointer protocol; right/middle return an error."},- "count":{"type":"integer"},+ "count":{"type":"integer","minimum":1,"description":"Number of clicks in one press train: 1 (default), 2 = double-click, 3 = triple-click (line/paragraph selection). There is no separate triple_click tool — pass count: 3 here."},"modifier": cua_driver_core::tool_schema::modifier_schema(),"from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."},"scope":{"type":"string","enum":["window","desktop"],"default":"window"},+ "coordinate_frame": coordinate_frame_schema(),"delivery_mode": crate::input::delivery::delivery_mode_schema()},"additionalProperties":false}),@@ -3865,8 +6661,8 @@ impl Tool for ClickTool {} else {(input.x, input.y)};- let sx = action_x as i32;- let sy = action_y as i32;+ let sx = action_x.round() as i32;+ let sy = action_y.round() as i32;// Glide the agent-cursor overlay to the click point first (the macOS// / Windows desktop paths already do this). Without it the overlay// sits idle elsewhere while only the real pointer warps, so a viewer@@ -3969,36 +6765,27 @@ impl Tool for ClickTool {}if let Some(idx) = elem_idx_resolved {- if !crate::wayland::is_wayland() {- let identity = match resolved {- cua_driver_core::element_token::ResolvedElement::Element {- element, ..- } => element,- cua_driver_core::element_token::ResolvedElement::None => {- unreachable!("indexed click resolved without an element")- }- };- return self- .click_indexed_x11(- pid,- idx,- window_id_resolved,- identity,- button,- count,- modifiers,- delivery,- cursor_id,- )- .await;- }+ // The observed element: its proven AT-SPI identity and the frame+ // the snapshot recorded. Every route below acts on that identity+ // (or its cached frame); the live ordinal `idx` is only an+ // address within the snapshot, never used to retarget.+ let observed = match resolved {+ cua_driver_core::element_token::ResolvedElement::Element { element, .. } => element,+ cua_driver_core::element_token::ResolvedElement::None => {+ unreachable!("indexed click resolved without an element")+ }+ };let xid_hint = window_id_resolved;// Resolve the element's screen center + its window FIRST, so the// agent cursor glides to the target *before* the click fires —// matching the coordinate path below and the macOS/Windows backends.// Previously perform_action ran inside this spawn_blocking, so the// app updated before the cursor visibly arrived.- let placement =+ // Bounded: resolving bounds re-walks the AT-SPI tree, which on a+ // large GTK/VCL tree can stall for minutes; a missing placement+ // degrades to the AX action (background) or a refusal (foreground).+ let placement = tokio::time::timeout(+ ELEMENT_AX_BUDGET,tokio::task::spawn_blocking(move || -> anyhow::Result<(u64, f64, f64)> {let (cx, cy) = element_screen_center(pid, idx, xid_hint)?;let xid = xid_hint@@ -4010,10 +6797,27 @@ impl Tool for ClickTool {}).unwrap_or(0);Ok((xid, cx, cy))- })- .await- .ok()- .and_then(Result::ok);+ }),+ )+ .await+ .ok()+ .and_then(Result::ok)+ .and_then(Result::ok);+ // X11 foreground element click: behave like a user — activate the+ // window and XTest-click the element's screen centre. No AT-SPI+ // action, no second tree walk. Element bounds come from the+ // (bounded) placement above.+ if delivery.is_foreground()+ && !crate::wayland::wayland_input_enabled()+ && !hyprland_foreground(delivery)+ {+ return self+ .foreground_element_click(+ &cursor_id, pid, idx, &observed, xid_hint, placement, button, count,+ &modifiers,+ )+ .await;+ }if let Some((xid, sx, sy)) = placement {if xid != 0 {crate::overlay::send_command_for(@@ -4034,11 +6838,90 @@ impl Tool for ClickTool {"an exact window_id or window-bound element token is required",);}- if element_click_prefers_ax(foreground_hyprland, button, count, !modifiers.is_empty()) {- let ax_result =- tokio::task::spawn_blocking(move || crate::atspi::perform_action(pid, idx))- .await;- if let Ok(Ok((_action, suspected_noop))) = ax_result {+ // An AT-SPI action can open a menu or press a button, but it cannot+ // give an entry / spin button / slider the widget focus a following+ // type_text needs (GIMP spin scales, VS Code settings inputs). With+ // the MPX real pointer available, click those like a user would.+ let needs_real_click = element_needs_real_click(&observed.role);+ // A menu / menu item's `doAction` is declined or silently ignored+ // by LibreOffice VCL (and opens GTK menus without the pointer+ // grab a following item click expects): with a real pointer, press+ // it like a user would and let the popup be listed. That holds+ // for an entry that is on screen (SHOWING: a menubar entry, an+ // item of an open GTK3 / VCL popup). A menu item that is not+ // showing has no point to press: gail (GTK2, GIMP) never marks+ // menu items SHOWING, open or closed, and an item of a closed+ // menu is still reached through its AT-SPI `click` action (the+ // same rule the foreground route applies), so those keep the+ // action route.+ let real_pointer = !delivery.is_foreground()+ && !crate::wayland::wayland_input_enabled()+ && crate::input::real_pointer_input_available();+ let menu_entry_on_screen = real_pointer && element_is_menu_role(&observed.role) && {+ // A bounded D-Bus round-trip: off the runtime thread.+ let observed_for_state = observed.clone();+ let showing = tokio::task::spawn_blocking(move || {+ crate::atspi::element_showing_observed(&observed_for_state)+ })+ .await;+ !matches!(showing, Ok(Ok(false)))+ };+ let real_click_role = real_pointer && (needs_real_click || menu_entry_on_screen);+ // Without a real pointer, an editable / table cell cannot take a+ // background click that gives it focus: refuse instead of firing+ // an AT-SPI action whose effect is not a click.+ if !delivery.is_foreground()+ && !crate::wayland::is_wayland()+ && needs_real_click+ && !crate::input::real_pointer_input_available()+ {+ if let Some(refusal) = unavailable_chromium_background(pid, delivery) {+ return refusal;+ }+ return crate::input::delivery::background_unavailable_error(+ crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,+ );+ }+ if !real_click_role+ && element_click_prefers_ax(+ foreground_hyprland,+ button,+ count,+ !modifiers.is_empty(),+ )+ {+ let observed_for_ax = observed.clone();+ let guard_pid = (!delivery.is_foreground()).then_some(pid);+ let ax_result = match tokio::time::timeout(+ ELEMENT_AX_BUDGET,+ tokio::task::spawn_blocking(move || {+ // Background: an AT-SPI action can open a menu (GTK/VCL+ // grab the keyboard and activate the toplevel) or map a+ // dialog mutter then focuses. Snapshot, act, restore.+ match guard_pid {+ Some(pid) => crate::input::focus_guard::guarded(Some(pid), || {+ crate::atspi::perform_action_observed(&observed_for_ax)+ }),+ None => crate::atspi::perform_action_observed(&observed_for_ax)+ .map(|value| (value, None)),+ }+ }),+ )+ .await+ {+ Ok(joined) => joined,+ Err(_elapsed) => {+ return ToolResult::error(format!(+ "click: the AT-SPI action for element [{idx}] did not complete within {ELEMENT_AX_BUDGET:?}; its effect is unknown. Take a screenshot, then retry by pixel (x/y) or with delivery_mode:\"foreground\"."+ ))+ .with_structured(json!({+ "code": "ax_timeout",+ "effect": "unknown",+ "escalation": non_ax_escalation(),+ }));+ }+ };+ if let Ok(Ok(((_action, suspected_noop, unacknowledged), guard))) = ax_result {let mut structured = json!({"path": "ax","verified": false,@@ -4046,9 +6929,32 @@ impl Tool for ClickTool {});if suspected_noop {structured["escalation"] = non_ax_escalation();+ } else if unacknowledged {+ structured["note"] = json!(+ "no acknowledgement received for the doAction call within the \+ AT-SPI timeout; its effect is unknown"+ );+ }+ return attach_focus_guard(+ ToolResult::text(format!("Clicked element [{idx}] (pid {pid})."))+ .with_structured(structured),+ guard.as_ref(),+ );+ }+ // The observed object is gone: the snapshot is stale. Refuse+ // rather than retarget whatever now lives at that index.+ if let Ok(Err(error)) = &ax_result {+ if error.is::<crate::atspi::native::CachedElementGone>() {+ return ToolResult::error(format!(+ "AT-SPI element click failed: stale_element_token: observed \+ AT-SPI object [{idx}] is no longer present ({error}); \+ re-snapshot with get_window_state"+ ))+ .with_structured(json!({+ "code": "stale_element_token",+ "effect": "none",+ }));}- return ToolResult::text(format!("Clicked element [{idx}] (pid {pid})."))- .with_structured(structured);}// AT-SPI errors can arrive after dispatch. Do not introduce a// primary-seat replay when its semantic outcome is uncertain.@@ -4131,10 +7037,12 @@ impl Tool for ClickTool {return refusal;}- // The AX route was unavailable. Fall back to a target-addressed+ // The AX route was unavailable. Fall back to a real MPX pointer+ // click at the element (no focus steal), then to a target-addressed// X11 event for toolkits that accept it.- let result = tokio::task::spawn_blocking(move || -> anyhow::Result<()> {- let (xid2, lx, ly) = resolve_element_local_coords(pid, idx, xid_hint)?;+ let cursor_id_for_fallback = cursor_id.clone();+ let result = tokio::task::spawn_blocking(move || -> anyhow::Result<PointerRoute> {+ let (xid2, lx, ly) = checked_element_local_coords(pid, idx, xid_hint)?;let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();if crate::wayland::wayland_input_enabled() && !modifier_refs.is_empty() {anyhow::bail!(@@ -4159,6 +7067,16 @@ impl Tool for ClickTool {&modifier_refs,)})+ .map(|()| PointerRoute::Foreground)+ } else if modifier_refs.is_empty() && !crate::wayland::wayland_input_enabled() {+ x11_pixel_click_no_focus_steal(+ &cursor_id_for_fallback,+ xid2,+ lx as i32,+ ly as i32,+ button,+ count,+ )} else {crate::input::send_click_with_modifiers(xid2,@@ -4168,25 +7086,31 @@ impl Tool for ClickTool {button,&modifier_refs,)+ .map(|()| PointerRoute::Synthetic)}}).await;+ let mode_label = if delivery.is_foreground() {+ "foreground"+ } else {+ "background"+ };return match result {- // An element click is never driver-verifiable (no read-back) —- // verified:false; the caller confirms via screenshot. `effect` is- // the richer signal: a passive/role-mismatched AT-SPI actuation is- // a likely no-op (→ cross to vision/pixel), otherwise the dispatch- // was fine but unconfirmable.- Ok(Ok(())) => {- let structured = json!({- "path": "x11_pixel",- "verified": false,- "effect": "unverifiable",- });- ToolResult::text(format!("Clicked element [{idx}] (pid {pid})."))- .with_structured(structured)+ // The MPX route reports its cheap post-checks (screen region+ // around the element, focus untouched); the other routes stay+ // unverifiable and the caller confirms via screenshot.+ Ok(Ok(route)) => {+ attach_focused_cell(+ ToolResult::text(format!(+ "Clicked element [{idx}] (pid {pid}) with a real pointer click {}",+ route.text_suffix(mode_label)+ ))+ .with_structured(route.structured(mode_label)),+ pid,+ )+ .await}- Ok(Err(e)) => ToolResult::error(format!("AT-SPI element click failed: {e}")),+ Ok(Err(e)) => input_error_result(e.context("AT-SPI element click failed")),Err(e) => ToolResult::error(format!("Task error: {e}")),};}@@ -4198,10 +7122,14 @@ impl Tool for ClickTool {}// Coordinate-based path.+ if let Some(refusal) = require_point_args("click", &args) {+ return refusal;+ }let xid = match args.opt_u64("window_id") {Some(v) => v,None => return ToolResult::error("Provide either element_index or window_id + x/y."),};+ let desktop_frame = desktop_frame_requested(&args);let from_zoom = args.bool_or("from_zoom", false);let mut x = args.f64_or("x", 0.0);let mut y = args.f64_or("y", 0.0);@@ -4211,10 +7139,21 @@ impl Tool for ClickTool {).with_structured(json!({ "code": "invalid_arguments" }));}+ if capture_id_arg.is_some() && desktop_frame {+ return ToolResult::error(+ "click.capture_id cannot be combined with desktop-frame coordinates; use coordinates from the published capture.",+ )+ .with_structured(json!({ "code": "invalid_arguments" }));+ }let native_refusal = (!isolated_background).then(|| unavailable_wayland_focused_input_background(delivery, true)).flatten();- if let Some(capture_id) = capture_id_arg {+ if desktop_frame {+ // Desktop pixels: no zoom/screenshot scaling; translated once below.+ if let Some(refusal) = native_refusal {+ return refusal;+ }+ } else if let Some(capture_id) = capture_id_arg {if let Some(refusal) = native_refusal {return refusal;}@@ -4254,6 +7193,20 @@ impl Tool for ClickTool {}}}+ if desktop_frame {+ match tokio::task::spawn_blocking(move || desktop_to_window_local(xid, x, y)).await {+ Ok(Ok((lx, ly))) => {+ x = lx;+ y = ly;+ }+ Ok(Err(e)) => {+ return ToolResult::error(format!(+ "desktop-frame coordinates could not be mapped into window {xid}: {e}"+ ))+ }+ Err(e) => return ToolResult::error(format!("Task error: {e}")),+ }+ }crate::overlay::send_command_for(cursor_id.clone(),@@ -4284,6 +7237,8 @@ impl Tool for ClickTool {}let (xi, yi) = (x as i32, y as i32);+ let pixel_screen_point =+ glide_target.map(|(sx, sy)| (sx.round() as i32, sy.round() as i32));let (output_x, output_y) = wayland_output_point.unwrap_or((xi, yi));if hyprland_foreground(delivery) {if !modifiers.is_empty() {@@ -4340,7 +7295,8 @@ impl Tool for ClickTool {// delivery_mode: background (default) = no-focus-steal injection;// foreground = activate the target window (EWMH) first, then inject,// then restore prior active. Mirrors macOS/Windows.- let result = tokio::task::spawn_blocking(move || -> anyhow::Result<&'static str> {+ let fg_budget = foreground_budget(0);+ let result = spawn_blocking_bounded("foreground click", fg_budget, move || -> anyhow::Result<(&'static str, Option<crate::input::ForegroundReport>, Option<PointerRoute>, Option<crate::input::FocusGuardReport>)> {if crate::wayland::wayland_input_enabled() {if !modifiers_for_task.is_empty() {anyhow::bail!(@@ -4359,15 +7315,15 @@ impl Tool for ClickTool {if let Ok(Some(_)) =crate::atspi::perform_action_at_screen_point(pid, xid, output_x, output_y){- return Ok("wayland_atspi");+ return Ok(("wayland_atspi", None, None, None));}}if crate::wayland::is_inject_mode() {crate::wayland::inject_click(pid, xid, x, y, count as u32, button)?;- return Ok("wayland_cua_compositor");+ return Ok(("wayland_cua_compositor", None, None, None));}if !delivery.is_foreground() {- return Ok("background_unavailable");+ return Ok(("background_unavailable", None, None, None));}// Native Wayland: focus+raise the target toplevel// (foreign-toplevel `activate`), then drive `count` virtual-pointer@@ -4375,7 +7331,7 @@ impl Tool for ClickTool {crate::wayland::with_target_foreground(pid, xid, || {crate::wayland::click_focused(output_x, output_y, count as u32, button)})?;- return Ok("wayland_activate");+ return Ok(("wayland_activate", None, None, None));}// X11 injection. Tiered no-focus-steal delivery (background):// 1. Plain left single-click → AT-SPI doAction at that point.@@ -4384,12 +7340,40 @@ impl Tool for ClickTool {// 3. Fallback → synthetic XSendEvent.// Foreground skips the AT-SPI shortcut and does a real activated pixel// click (the agent's escalation when background didn't land).- let inject = |fg: bool| -> anyhow::Result<&'static str> {+ let inject = |fg: bool| -> anyhow::Result<(&'static str, Option<PointerRoute>, Option<crate::input::FocusGuardReport>)> {if !fg && button == 1 && count == 1 && modifiers_for_task.is_empty() {- if let Ok(Some(_)) = crate::atspi::perform_action_at_point(pid, xi, yi) {- return Ok("x11_atspi");+ // The accessible action under the point may open a menu or+ // a dialog that takes the focus: guard and restore. With a+ // real pointer available, an entry / spin button / cell+ // under the point is left to the MPX click below (its+ // `doAction` would not focus it for a following type_text).+ let real_click_for_focus_roles =+ crate::input::real_pointer_input_available();+ let (hit, guard) = crate::input::focus_guard::guarded(Some(pid), || {+ Ok(crate::atspi::perform_action_at_point_in(+ pid,+ xid,+ xi,+ yi,+ real_click_for_focus_roles,+ )+ .ok()+ .flatten())+ })?;+ if let Some(hit) = hit {+ return Ok(("x11_atspi", Some(PointerRoute::Atspi(hit)), guard));}}+ if !fg+ && !crate::input::real_pointer_input_available()+ && synthetic_pointer_is_dropped(pid)+ {+ // No accessible actuator covers the point and the only+ // route left is a synthetic XSendEvent that this toolkit+ // (GTK3/4 XInput2, LibreOffice VCL) discards. Say so+ // instead of reporting a click that changed nothing.+ return Ok(("background_unavailable_pointer", None, None));+ }if fg {// Foreground: the window is already activated. Deliver a REAL// XTest warp+button click. Synthetic XSendEvent button events@@ -4408,11 +7392,13 @@ impl Tool for ClickTool {count,&modifier_refs,)?;- return Ok("x11_xtest_fg");+ return Ok(("x11_xtest_fg", None, None));}}if modifiers_for_task.is_empty() {- x11_pixel_click_no_focus_steal(+ // The MPX press train runs under its own (settled) focus+ // guard inside the helper; its report rides on the route.+ let route = x11_pixel_click_no_focus_steal(&cursor_id_for_task,xid,xi,@@ -4420,6 +7406,7 @@ impl Tool for ClickTool {button,count,)?;+ return Ok((route.path().unwrap_or("x11_pixel"), Some(route), None));} else {let modifier_refs: Vec<&str> =modifiers_for_task.iter().map(String::as_str).collect();@@ -4432,12 +7419,17 @@ impl Tool for ClickTool {&modifier_refs,)?;}- Ok(if fg { "x11_pixel_fg" } else { "x11_pixel" })+ Ok((if fg { "x11_pixel_fg" } else { "x11_pixel" }, None, None))};if delivery.is_foreground() {- crate::input::with_x11_foreground(xid, 80, || inject(true))+ crate::input::with_x11_foreground_opts(+ xid,+ crate::input::ForegroundOptions::pointer(),+ || inject(true),+ )+ .map(|((path, route, guard), report)| (path, Some(report), route, guard))} else {- inject(false)+ inject(false).map(|(path, route, guard)| (path, None, route, guard))}}).await;@@ -4446,22 +7438,64 @@ impl Tool for ClickTool {} else {"background"};- match result {- Ok(Ok("background_unavailable")) => {+ let outcome = match result {+ Ok(Ok(("background_unavailable", _, _, _))) => {crate::input::delivery::background_unavailable_error(crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,)}+ Ok(Ok(("background_unavailable_pointer", _, _, _))) => {+ let mut refusal = crate::input::delivery::background_unavailable_error(+ crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,+ );+ let hint = "No accessible control covers this point and the toolkit drops \+ synthetic pointer events, so a background pixel click here would \+ change nothing. Use get_window_state and click by element_index \+ (AT-SPI action), or retry with delivery_mode='foreground'.";+ refusal+ .content+ .push(cua_driver_core::protocol::Content::text(hint));+ if let Some(structured) = refusal.structured_content.as_mut() {+ structured["hint"] = json!(hint);+ structured["path"] = json!("background_unavailable_pointer");+ }+ refusal+ }// A pixel/coordinate click is never driver-verifiable (no read-back) —// verified:false, effect:"unverifiable"; the caller confirms via- // screenshot. path reports the rung taken.- Ok(Ok(path)) => ToolResult::text(format!(- "✅ Clicked at ({x:.1}, {y:.1}) × {count} (delivery_mode={mode_label})."- ))- .with_structured(json!({ "path": path, "verified": false, "effect": "unverifiable" })),- Ok(Err(e)) => linux_input_error(e),+ // screenshot. path reports the rung taken; a foreground click also+ // reports the activation/focus transaction it confirmed first.+ Ok(Ok((path, report, route, guard))) => {+ let mut structured = match &route {+ Some(route) => route.structured(mode_label),+ None => json!({+ "path": path,+ "verified": false,+ "effect": "unverifiable",+ "delivery_mode": mode_label,+ }),+ };+ if let Some(report) = report {+ apply_foreground_report(&mut structured, &report);+ }+ let suffix = match &route {+ Some(route) => route.text_suffix(mode_label),+ None => format!(+ "(delivery_mode={mode_label}, path={path}); not verified — confirm \+ with a screenshot."+ ),+ };+ attach_focus_guard(+ ToolResult::text(format!("Clicked at ({x:.1}, {y:.1}) × {count} {suffix}"))+ .with_structured(structured),+ guard.as_ref(),+ )+ }+ Ok(Err(e)) => input_error_result(e),Err(e) => ToolResult::error(format!("Task error: {e}")),- }+ };+ let outcome = attach_focused_cell(outcome, pid).await;+ with_pixel_points(outcome, pixel_screen_point, (xi, yi))}}@@ -4496,6 +7530,8 @@ async fn focus_by_pixel("_session_id","_transport_session_id","cursor_id",+ "scope",+ "coordinate_frame",] {if let Some(value) = parent_args.get(field) {click_args[field] = value.clone();@@ -4574,9 +7610,10 @@ impl Tool for TypeTextTool {"element_index": cua_driver_core::tool_schema::element_index_schema(),"element_token": cua_driver_core::tool_schema::element_token_schema(),"snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),- "x":{"type":"number","description":"Screenshot-pixel X of the field to type into — the element px action form. Pass x,y (no element_index) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the AX path can't reach. Read straight off the get_window_state PNG, same convention as click."},- "y":{"type":"number","description":"Screenshot-pixel Y of the field (see x)."},+ "x":{"type":"number","description":"Pixel X of the field to type into — the element px action form. Pass x,y (no element_index) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the AX path can't reach. Window-local pixels of the target window's own get_window_state screenshot by default (same convention as click); for get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\") — a bare pid+x/y is otherwise misinterpreted as window-local and can focus the wrong widget."},+ "y":{"type":"number","description":"Pixel Y of the field (see x)."},"scope":{"type":"string","enum":["window","desktop"],"default":"window"},+ "coordinate_frame": coordinate_frame_schema(),"delivery_mode": crate::input::delivery::delivery_mode_schema()},"additionalProperties":false}),@@ -4586,7 +7623,10 @@ impl Tool for TypeTextTool {async fn invoke(&self, args: Value) -> ToolResult {use cua_driver_core::tool_args::ArgsExt;- if args.opt_str("scope").as_deref() == Some("desktop") && args.get("pid").is_none() {+ if (args.opt_str("scope").as_deref() == Some("desktop")+ || coordinate_frame_is_desktop(&args))+ && args.get("pid").is_none()+ {let input = match parse_typed_projection::<TypeTextInput>("type_text", &args) {Ok(input) => input,Err(result) => return result,@@ -4615,7 +7655,7 @@ impl Tool for TypeTextTool {Err(error) => ToolResult::error(format!("Task error: {error}")),};}- let pid = args.u64_or("pid", 0) as u32;+ let mut pid = args.u64_or("pid", 0) as u32;let text_raw = match args.require_str("text") {Ok(v) => v,Err(e) => return e,@@ -4652,12 +7692,43 @@ impl Tool for TypeTextTool {let xid = match xid_opt {Some(x) => x,None => {- let windows =- tokio::task::spawn_blocking(move || crate::x11::list_windows(Some(pid)))- .await- .unwrap_or_default();- match windows.first() {- Some(w) => w.xid,+ let windows = tokio::task::spawn_blocking(move || {+ crate::x11::list_windows(if pid == 0 { None } else { Some(pid) })+ })+ .await+ .unwrap_or_default();+ // pid omitted: the keys go to the active window, like a+ // physical keyboard would, and the action adopts its pid.+ let chosen = if pid == 0 {+ let active = crate::x11::active_window();+ windows+ .iter()+ .find(|w| Some(w.xid) == active && w.pid.is_some())+ .or_else(|| windows.iter().find(|w| w.is_on_screen && w.pid.is_some()))+ } else {+ // The pid's active window, else its largest on-screen+ // toplevel: LibreOffice and GIMP own hidden/utility+ // toplevels that a plain `first()` could pick, and an+ // unmapped window cannot take the virtual keyboard focus.+ let active = crate::x11::active_window();+ windows+ .iter()+ .find(|w| Some(w.xid) == active)+ .or_else(|| {+ windows+ .iter()+ .filter(|w| w.is_on_screen)+ .max_by_key(|w| u64::from(w.width) * u64::from(w.height))+ })+ .or_else(|| windows.first())+ };+ match chosen {+ Some(w) => {+ if pid == 0 {+ pid = w.pid.unwrap_or(0);+ }+ w.xid+ }None => {return ToolResult::error(format!("No windows found for pid {pid}. Provide window_id."@@ -4713,7 +7784,10 @@ impl Tool for TypeTextTool {let px = args.get("x").and_then(|value| value.as_f64());let py = args.get("y").and_then(|value| value.as_f64());if px.is_some() != py.is_some() {- return ToolResult::error("Pass both x and y to type_text, or neither.");+ return invalid_pointer_arguments(+ "type_text",+ "x and y must be passed together (a pixel focus point), or neither",+ );}if px.is_some() && resolved_elem_idx.is_some() {return ToolResult::error(@@ -4819,7 +7893,14 @@ impl Tool for TypeTextTool {}).await;if let Ok(Ok(())) = targeted {- return type_text_ax_result(pid, text_len, "via targeted AT-SPI");+ return type_text_ax_result_verified(+ pid,+ xid_opt,+ idx,+ &text,+ "via targeted AT-SPI",+ )+ .await;}}// The private nested compositor can target the owning Wayland client@@ -4932,7 +8013,14 @@ impl Tool for TypeTextTool {.await;match targeted {Ok(Ok(())) => {- return type_text_ax_result(pid, text_len, "via targeted AT-SPI");+ return type_text_ax_result_verified(+ pid,+ xid_opt,+ idx,+ &text,+ "via targeted AT-SPI",+ )+ .await;}Ok(Err(_)) | Err(_)if !delivery.is_foreground() && crate::wayland::wayland_input_enabled() =>@@ -5000,71 +8088,123 @@ impl Tool for TypeTextTool {let text_len = text.chars().count();let text_t = text.clone();let foreground = delivery.is_foreground();- let result = tokio::task::spawn_blocking(move || -> anyhow::Result<&'static str> {+ let cursor_id = resolve_cursor_key(&args);+ let result = spawn_blocking_bounded(+ "type_text terminal",+ foreground_budget(text_len),+ move || -> anyhow::Result<(&'static str, Option<crate::input::KeyboardDeliveryReport>)> {// pty-master injection is preferred — it skips the X event- // queue entirely. Falls through to XTest if the terminal+ // queue entirely. Falls through to key events if the terminal// isn't reachable that way (descendant pty unresolvable).if inject_terminal_input(pid, xid, &text_t)? {- return Ok("pty");+ return Ok(("pty", None));}if foreground {crate::input::with_x11_foreground(xid, 80, || {crate::input::send_type_text_xtest(&text_t)})?;- Ok("key_events_fg")- } else {- Ok("background_unavailable")+ return Ok(("key_events_fg", None));+ }+ // Background: real key events through the XI2 virtual master+ // keyboard (xterm/gnome-terminal accept them like physical input).+ match background_text_route(&cursor_id, pid, xid, &text_t)? {+ Some(KeyRoute::Mpx(report)) => Ok((crate::input::MPX_UINPUT_PATH, Some(report))),+ _ => Ok(("background_unavailable", None)),}}).await;return match result {- Ok(Ok("background_unavailable")) => {- crate::input::delivery::background_unavailable_error(- crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,- )- }- Ok(Ok(path)) => ToolResult::text(format!(+ Ok(Ok(("background_unavailable", _))) => background_keyboard_refusal(),+ Ok(Ok((_, Some(report)))) => type_text_mpx_result(text_len, report),+ Ok(Ok((path, None))) => ToolResult::text(format!("Typed {text_len} character(s) (terminal emulator: pty/XTest key events).")).with_structured(type_text_structured(path, text_len, false)),- Ok(Err(e)) => ToolResult::error(e.to_string()),+ Ok(Err(e)) => input_error_result(e),Err(e) => ToolResult::error(format!("Task error: {e}")),};}- // Foreground means the caller explicitly permits activation. Chromium- // and WebKitGTK can acknowledge an accessibility write without- // producing the renderer input event, so web embedders use real XTest- // key events. Native toolkits keep their verifiable AT-SPI path below.- if delivery.is_foreground() && (is_chromium_embedder(pid) || is_webkitgtk_embedder(pid)) {+ // Foreground means the caller explicitly permits activation: behave like+ // a user. Activate the window and confirm it holds the input focus,+ // GrabFocus the indexed element when one was given, then type REAL key+ // events via XTest. The AT-SPI EditableText probes below are skipped on+ // purpose: on large trees (LibreOffice) they stall for the full bus+ // timeout, Chromium/WebKitGTK echo them without the renderer seeing the+ // input, and for a spreadsheet cell / canvas they would target the+ // wrong widget anyway. The whole transaction is deadline-bounded.+ if delivery.is_foreground() {let text_f = text.clone();let idx = resolved_elem_idx;- let result = tokio::task::spawn_blocking(move || {- crate::input::with_x11_foreground(xid, 80, || {- if let Some(idx) = idx {- if !crate::atspi::focus_element(pid, idx)? {- anyhow::bail!(- "AT-SPI Component.GrabFocus returned false for element {idx}"- );- }- }- crate::input::send_type_text_xtest(&text_f)- })- })+ let result = spawn_blocking_bounded(+ "foreground type_text",+ foreground_budget(text_len),+ move || {+ crate::input::with_x11_foreground_opts(+ xid,+ crate::input::ForegroundOptions::keyboard(),+ || {+ if let Some(idx) = idx {+ if !crate::atspi::focus_element(pid, idx)? {+ anyhow::bail!(+ "AT-SPI Component.GrabFocus returned false for element {idx}"+ );+ }+ }+ crate::input::send_type_text_xtest(&text_f)+ },+ )+ },+ ).await;return match result {- Ok(Ok(())) => ToolResult::text(format!(- "Typed {text_len} character(s) (via X11, delivery_mode=foreground)."- ))- .with_structured(type_text_structured(- "key_events_fg",- text_len,- false,- )),- Ok(Err(e)) => ToolResult::error(e.to_string()),+ Ok(Ok(((), report))) => {+ let mut extra = serde_json::Map::new();+ extra.insert("characters".into(), json!(text_len));+ ToolResult::text(format!(+ "Typed {text_len} character(s) as real key events \+ (delivery_mode=foreground, focus_after={}); not verified — \+ confirm with a screenshot.",+ report.focus_after.as_str()+ ))+ .with_structured(foreground_structured(+ "key_events_fg",+ report,+ extra,+ ))+ }+ Ok(Err(e)) => input_error_result(e),Err(e) => ToolResult::error(format!("Task error: {e}")),};}+ // With a focus-free real keyboard available, type where the widget+ // focus is — exactly what a physical keyboard does after the click that+ // focused the field. The blind AT-SPI editable search below can pick a+ // different editable of the app (it "succeeded" into GIMP's wrong entry+ // and VS Code's search box); it stays as the fallback for hosts without+ // the MPX route and for explicit element targets.+ if !delivery.is_foreground()+ && resolved_elem_idx.is_none()+ && crate::input::real_keyboard_input_available()+ {+ let text_b = text.clone();+ let cursor_id = resolve_cursor_key(&args);+ let result = spawn_blocking_bounded(+ "background type_text",+ foreground_budget(text_len),+ move || background_text_route(&cursor_id, pid, xid, &text_b),+ )+ .await;+ match result {+ Ok(Ok(Some(KeyRoute::Mpx(report)))) => {+ return type_text_mpx_result(text_len, report)+ }+ Ok(Err(e)) => return input_error_result(e),+ Err(e) => return ToolResult::error(format!("Task error: {e}")),+ Ok(Ok(_)) => {}+ }+ }+// Prefer the focused widget — the element the user just clicked. If a// NON-editable input holds keyboard focus (a spreadsheet cell, a// terminal, a canvas), the focus-free AT-SPI editable search below would@@ -5081,9 +8221,33 @@ impl Tool for TypeTextTool {.flatten();if focus_kind == Some(false) {if !delivery.is_foreground() {- return crate::input::delivery::background_unavailable_error(- crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,- );+ // A spreadsheet cell / canvas / terminal holds the widget focus:+ // real key events aimed at this window are the only thing that+ // types there. The XI2 virtual master keyboard delivers them+ // without touching the user's focus; terminals take the pty.+ let text_b = text.clone();+ let cursor_id = resolve_cursor_key(&args);+ let result = spawn_blocking_bounded(+ "background type_text",+ foreground_budget(text_len),+ move || -> anyhow::Result<Option<KeyRoute>> {+ if inject_terminal_input(pid, xid, &text_b)? {+ return Ok(Some(KeyRoute::Terminal));+ }+ background_text_route(&cursor_id, pid, xid, &text_b)+ },+ )+ .await;+ return match result {+ Ok(Ok(Some(KeyRoute::Terminal))) => ToolResult::text(format!(+ "Typed {text_len} character(s) into the focused terminal (pty)."+ ))+ .with_structured(type_text_structured("key_events", text_len, false)),+ Ok(Ok(Some(KeyRoute::Mpx(report)))) => type_text_mpx_result(text_len, report),+ Ok(Ok(_)) => background_keyboard_refusal(),+ Ok(Err(e)) => input_error_result(e),+ Err(e) => ToolResult::error(format!("Task error: {e}")),+ };}let text_f = text.clone();let result = tokio::task::spawn_blocking(move || {@@ -5108,7 +8272,7 @@ impl Tool for TypeTextTool {text_len,false,)),- Ok(Err(e)) => ToolResult::error(e.to_string()),+ Ok(Err(e)) => input_error_result(e),Err(e) => ToolResult::error(format!("Task error: {e}")),};}@@ -5137,6 +8301,9 @@ impl Tool for TypeTextTool {// This doesn't change the X11 active window, so the test's focus check passes.let text_clone2 = text.clone();let qt5_result = tokio::task::spawn_blocking(move || {+ if !is_qt5_process(pid) {+ anyhow::bail!("not a Qt5 process; synthetic-FocusIn bridge workaround skipped");+ }// Send FocusIn to trigger Qt5's bridgecrate::input::send_focus_in(xid)?;std::thread::sleep(std::time::Duration::from_millis(100));@@ -5168,30 +8335,38 @@ impl Tool for TypeTextTool {// foreground = activate the window (EWMH), then synthesize REAL key// events to it via XTest — the escalation when background didn't land// (e.g. a GTK dialog whose widget ignores synthetic XSendEvent keys).- let result = tokio::task::spawn_blocking(move || -> anyhow::Result<&'static str> {+ let cursor_id = resolve_cursor_key(&args);+ let result = spawn_blocking_bounded(+ "type_text",+ foreground_budget(text_len),+ move || -> anyhow::Result<(&'static str, Option<crate::input::KeyboardDeliveryReport>)> {// Terminals: write to the pty master (focus-free, below the toolkit).if inject_terminal_input(pid, xid, &text)? {- return Ok("key_events");+ return Ok(("key_events", None));}// GUI apps: X11 only routes keystrokes to the *focused* toplevel's// focused widget, so background XSendEvent typing doesn't land. Fill// the editable field via AT-SPI instead — focus-free and toolkit-// agnostic. Fall back to Tk send or XSendEvent when no a11y field is exposed.if crate::atspi::insert_text(pid, &text).unwrap_or(false) {- return Ok("ax");+ return Ok(("ax", None));}// Tk apps: use Tk's `send` command (no AT-SPI bridge, so AT-SPI above// returned false). This is the Tk-specific override, like CDP for Chromium.if crate::input::inject_tk_send(&text).unwrap_or(false) {- return Ok("key_events");+ return Ok(("key_events", None));}if delivery.is_foreground() {crate::input::with_x11_foreground(xid, 80, || {crate::input::send_type_text_xtest(&text)})?;- Ok("key_events_fg")- } else {- Ok("background_unavailable")+ return Ok(("key_events_fg", None));+ }+ // No a11y editable, no pty: real key events through the XI2 virtual+ // master keyboard are the last focus-free route.+ match background_text_route(&cursor_id, pid, xid, &text)? {+ Some(KeyRoute::Mpx(report)) => Ok((crate::input::MPX_UINPUT_PATH, Some(report))),+ _ => Ok(("background_unavailable", None)),}}).await;@@ -5204,26 +8379,84 @@ impl Tool for TypeTextTool {// AT-SPI's boolean acknowledges the EditableText call; it is not a// fresh value readback. Keep the result unverifiable and apply the// stricter Chromium escalation where appropriate.- Ok(Ok("ax")) => type_text_ax_result(+ Ok(Ok(("ax", _))) => type_text_ax_result(pid,text_len,&format!("via X11, delivery_mode={mode_label}"),),- Ok(Ok("background_unavailable")) => {- crate::input::delivery::background_unavailable_error(- crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,- )- }- Ok(Ok(path)) => ToolResult::text(format!(+ Ok(Ok(("background_unavailable", _))) => background_keyboard_refusal(),+ Ok(Ok((_, Some(report)))) => type_text_mpx_result(text_len, report),+ Ok(Ok((path, None))) => ToolResult::text(format!("Typed {text_len} character(s) (via X11, delivery_mode={mode_label}).")).with_structured(type_text_structured(path, text_len, false)),- Ok(Err(e)) => ToolResult::error(e.to_string()),+ Ok(Err(e)) => input_error_result(e),Err(e) => ToolResult::error(format!("Task error: {e}")),}}}+/// Background text delivery through the XI2 virtual master keyboard. `None`+/// when the host has no such route (Xvfb, no `/dev/uinput`); a uinput+/// permission error propagates so the tool can attach the operator hint.+fn background_text_route(+ cursor_id: &str,+ pid: u32,+ xid: u64,+ text: &str,+) -> anyhow::Result<Option<KeyRoute>> {+ if !crate::input::real_keyboard_input_available() {+ return Ok(None);+ }+ match crate::input::focus_guard::guarded(Some(pid), || {+ background_virtual_text(cursor_id, pid, xid, text)+ }) {+ Ok((mut report, guard)) => {+ report.focus_guard = guard;+ Ok(Some(KeyRoute::Mpx(report)))+ }+ Err(error) if crate::input::is_uinput_unavailable(&error) => Err(error),+ Err(error) if is_popup_keyboard_grab(&error) => Err(error),+ Err(error) => {+ tracing::warn!(pid, "MPX keyboard text delivery failed: {error}");+ Err(error.context("virtual master keyboard delivery failed"))+ }+ }+}++/// `type_text` result for a delivery through the virtual master keyboard.+fn type_text_mpx_result(+ text_len: usize,+ report: crate::input::KeyboardDeliveryReport,+) -> ToolResult {+ let mut structured = type_text_structured(report.path, text_len, false);+ for (key, value) in report.to_json().as_object().into_iter().flatten() {+ structured[key] = value.clone();+ }+ structured["delivery_mode"] = json!("background");+ let mut text = format!(+ "Typed {text_len} character(s) as real key events {} (delivery_mode=background); \+ not verified — confirm with a screenshot.",+ report.route_phrase()+ );+ if !report.skipped_characters.is_empty() {+ text.push_str(&format!(+ " {} character(s) had no keycode in the current keymap and were skipped: {:?}.",+ report.skipped_characters.len(),+ report.skipped_characters.iter().collect::<String>()+ ));+ }+ if !report.virtual_focus_held || !report.delivery_confirmed {+ structured["effect"] = json!("suspected_noop");+ text.push_str(+ " Warning: the target lost the virtual keyboard's focus or the server did \+ not confirm the last key; the text may not have landed.",+ );+ }+ text.push_str(&report.delivery_notes());+ ToolResult::text(text).with_structured(structured)+}+// ── press_key ─────────────────────────────────────────────────────────────────pub struct PressKeyTool {@@ -5292,9 +8525,10 @@ impl Tool for PressKeyTool {"element_index": cua_driver_core::tool_schema::element_index_schema(),"element_token": cua_driver_core::tool_schema::element_token_schema(),"snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),- "x":{"type":"number","description":"Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_index."},- "y":{"type":"number","description":"Screenshot-pixel Y (see x)."},+ "x":{"type":"number","description":"Pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_index. Window-local pixels by default (same convention as click); for get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\")."},+ "y":{"type":"number","description":"Pixel Y (see x)."},"scope":{"type":"string","enum":["window","desktop"],"default":"window"},+ "coordinate_frame": coordinate_frame_schema(),"delivery_mode": crate::input::delivery::delivery_mode_schema()},"additionalProperties":false}),@@ -5304,7 +8538,10 @@ impl Tool for PressKeyTool {async fn invoke(&self, args: Value) -> ToolResult {use cua_driver_core::tool_args::ArgsExt;- if args.opt_str("scope").as_deref() == Some("desktop") && args.get("pid").is_none() {+ if (args.opt_str("scope").as_deref() == Some("desktop")+ || coordinate_frame_is_desktop(&args))+ && args.get("pid").is_none()+ {let input = match parse_typed_projection::<PressKeyInput>("press_key", &args) {Ok(input) => input,Err(result) => return result,@@ -5338,12 +8575,14 @@ impl Tool for PressKeyTool {Err(error) => ToolResult::error(format!("Task error: {error}")),};}- let pid = args.u64_or("pid", 0) as u32;+ let mut pid = args.u64_or("pid", 0) as u32;let key = match args.require_str("key") {Ok(v) => v,Err(e) => return e,};- let mods: Vec<String> = args.str_array("modifiers");+ let mut mods: Vec<String> = args.str_array("modifiers");+ let (combo_mods, key) = split_key_combo(&key);+ mods.extend(combo_mods);// Surface 6: resolve the element token/index into both its owning// window and exact child. Foreground delivery establishes child focus@@ -5378,12 +8617,43 @@ impl Tool for PressKeyTool {let xid = match xid_opt {Some(x) => x,None => {- let windows =- tokio::task::spawn_blocking(move || crate::x11::list_windows(Some(pid)))- .await- .unwrap_or_default();- match windows.first() {- Some(w) => w.xid,+ let windows = tokio::task::spawn_blocking(move || {+ crate::x11::list_windows(if pid == 0 { None } else { Some(pid) })+ })+ .await+ .unwrap_or_default();+ // pid omitted: the keys go to the active window, like a+ // physical keyboard would, and the action adopts its pid.+ let chosen = if pid == 0 {+ let active = crate::x11::active_window();+ windows+ .iter()+ .find(|w| Some(w.xid) == active && w.pid.is_some())+ .or_else(|| windows.iter().find(|w| w.is_on_screen && w.pid.is_some()))+ } else {+ // The pid's active window, else its largest on-screen+ // toplevel: LibreOffice and GIMP own hidden/utility+ // toplevels that a plain `first()` could pick, and an+ // unmapped window cannot take the virtual keyboard focus.+ let active = crate::x11::active_window();+ windows+ .iter()+ .find(|w| Some(w.xid) == active)+ .or_else(|| {+ windows+ .iter()+ .filter(|w| w.is_on_screen)+ .max_by_key(|w| u64::from(w.width) * u64::from(w.height))+ })+ .or_else(|| windows.first())+ };+ match chosen {+ Some(w) => {+ if pid == 0 {+ pid = w.pid.unwrap_or(0);+ }+ w.xid+ }None => {return ToolResult::error(format!("No windows found for pid {pid}. Provide window_id."@@ -5436,11 +8706,17 @@ impl Tool for PressKeyTool {if let Some(refusal) = unavailable_wayland_focused_input_background(delivery, true) {return refusal;}+ if let Some(result) = wm_chord_background(pid, xid, &key, &mods, delivery).await {+ return result;+ }let px = args.get("x").and_then(|value| value.as_f64());let py = args.get("y").and_then(|value| value.as_f64());if px.is_some() != py.is_some() {- return ToolResult::error("Pass both x and y to press_key, or neither.");+ return invalid_pointer_arguments(+ "press_key",+ "x and y must be passed together (a pixel focus point), or neither",+ );}if px.is_some() && resolved_element_index.is_some() {return ToolResult::error(@@ -5571,49 +8847,56 @@ impl Tool for PressKeyTool {}let key_for_task = key.clone();+ let cursor_id = resolve_cursor_key(&args);// Foreground delivery is one atomic activate-and-XTest transaction.// A preceding PX click establishes internal widget focus, but that// click restores the prior top-level before returning.let deliver_fg = delivery.is_foreground();- let result = tokio::task::spawn_blocking(move || -> anyhow::Result<()> {+ let result = spawn_blocking_bounded(+ "press_key",+ foreground_budget(1),+ move || -> anyhow::Result<KeyRoute> {if resolved_element_index.is_none()&& mods.is_empty()&& key_for_task.eq_ignore_ascii_case("enter"){if inject_terminal_input(pid, xid, "\n")? {- return Ok(());+ return Ok(KeyRoute::Terminal);}}let m: Vec<&str> = mods.iter().map(String::as_str).collect();// foreground: activate the window first, then inject a REAL key via// XTest. Synthetic XSendEvent keys (`send_key`) are dropped by// GTK/Qt/Chromium/Firefox, so the foreground rung must use XTest —- // it delivers to the now-focused window. background = direct- // XSendEvent (no focus steal) for apps that accept it.+ // it delivers to the now-focused window. background = real key+ // events through the XI2 virtual master keyboard (no focus steal),+ // else direct XSendEvent for apps that accept it.if deliver_fg {- return crate::input::with_x11_foreground(xid, 80, || {- if let Some(element_index) = resolved_element_index {- if !crate::atspi::focus_element(pid, element_index)? {- anyhow::bail!(- "AT-SPI Component.GrabFocus returned false for element {element_index}"- );+ return crate::input::with_x11_foreground_opts(+ xid,+ crate::input::ForegroundOptions::keyboard(),+ || {+ if let Some(element_index) = resolved_element_index {+ if !crate::atspi::focus_element(pid, element_index)? {+ anyhow::bail!(+ "AT-SPI Component.GrabFocus returned false for element {element_index}"+ );+ }}- }- crate::input::send_key_xtest(&key_for_task, &m)- });- }- if let Some(element_index) = resolved_element_index {- if !crate::atspi::focus_element(pid, element_index)? {- anyhow::bail!(- "AT-SPI Component.GrabFocus returned false for element {element_index}"- );- }- }- if let Some((x, y)) = px_target {- crate::input::send_key_at(xid, x, y, &key_for_task, &m)- } else {- crate::input::send_key(xid, &key_for_task, &m)+ crate::input::send_key_xtest(&key_for_task, &m)+ },+ )+ .map(|((), report)| KeyRoute::Foreground(report));}+ background_key_route(+ &cursor_id,+ pid,+ xid,+ px_target,+ &key_for_task,+ &m,+ resolved_element_index,+ )}).await;let mode_label = if deliver_fg {@@ -5622,11 +8905,8 @@ impl Tool for PressKeyTool {"background"};match result {- Ok(Ok(())) => {- ToolResult::text(format!("Pressed key '{key}' (delivery_mode={mode_label})."))- .with_structured(json!({ "verified": false, "delivery_mode": mode_label }))- }- Ok(Err(e)) => ToolResult::error(e.to_string()),+ Ok(Ok(route)) => key_route_result(&format!("Pressed key '{key}'"), route, mode_label),+ Ok(Err(e)) => input_error_result(e),Err(e) => ToolResult::error(format!("Task error: {e}")),}}@@ -5661,7 +8941,12 @@ impl Tool for HotkeyTool {HOTKEY_DEF.get_or_init(|| ToolDef {name: "hotkey".into(),description: "Press a combination of keys simultaneously, e.g. [\"ctrl\",\"c\"] for Copy. \- Sent via XSendEvent directly to the target pid; target does NOT need to be frontmost.".into(),+ Delivered to the target window without focus steal (real key events from a virtual \+ master keyboard on X11); target does NOT need to be frontmost. Window-manager chords \+ cannot reach the WM that way: alt+F4 on a window target closes that window through \+ the WM (_NET_CLOSE_WINDOW, effect confirmed when it unmaps); super+*, alt+tab, \+ ctrl+alt+* and other WM bindings are refused with code wm_chord_unavailable — use \+ bring_to_front / list_windows / kill_app or delivery_mode:\"foreground\" instead.".into(),input_schema: json!({"type":"object","required":["keys"],"properties":{"session": cua_driver_core::tool_schema::session_schema(),@@ -5725,7 +9010,7 @@ impl Tool for HotkeyTool {Err(error) => ToolResult::error(format!("Task error: {error}")),};}- let pid = args.u64_or("pid", 0) as u32;+ let mut pid = args.u64_or("pid", 0) as u32;let window_id_arg = args.opt_u64("window_id");let element_index_arg = args.opt_u64("element_index").map(|value| value as usize);let resolved = match self.state.element_cache.resolve_element_args(@@ -5756,12 +9041,43 @@ impl Tool for HotkeyTool {let xid = match xid_opt {Some(x) => x,None => {- let windows =- tokio::task::spawn_blocking(move || crate::x11::list_windows(Some(pid)))- .await- .unwrap_or_default();- match windows.first() {- Some(w) => w.xid,+ let windows = tokio::task::spawn_blocking(move || {+ crate::x11::list_windows(if pid == 0 { None } else { Some(pid) })+ })+ .await+ .unwrap_or_default();+ // pid omitted: the keys go to the active window, like a+ // physical keyboard would, and the action adopts its pid.+ let chosen = if pid == 0 {+ let active = crate::x11::active_window();+ windows+ .iter()+ .find(|w| Some(w.xid) == active && w.pid.is_some())+ .or_else(|| windows.iter().find(|w| w.is_on_screen && w.pid.is_some()))+ } else {+ // The pid's active window, else its largest on-screen+ // toplevel: LibreOffice and GIMP own hidden/utility+ // toplevels that a plain `first()` could pick, and an+ // unmapped window cannot take the virtual keyboard focus.+ let active = crate::x11::active_window();+ windows+ .iter()+ .find(|w| Some(w.xid) == active)+ .or_else(|| {+ windows+ .iter()+ .filter(|w| w.is_on_screen)+ .max_by_key(|w| u64::from(w.width) * u64::from(w.height))+ })+ .or_else(|| windows.first())+ };+ match chosen {+ Some(w) => {+ if pid == 0 {+ pid = w.pid.unwrap_or(0);+ }+ w.xid+ }None => {return ToolResult::error(format!("No windows found for pid {pid}. Provide window_id."@@ -5845,11 +9161,17 @@ impl Tool for HotkeyTool {if let Some(refusal) = unavailable_wayland_focused_input_background(delivery, true) {return refusal;}+ if let Some(result) = wm_chord_background(pid, xid, &key, &mods, delivery).await {+ return result;+ }let px = args.get("x").and_then(|value| value.as_f64());let py = args.get("y").and_then(|value| value.as_f64());if px.is_some() != py.is_some() {- return ToolResult::error("Pass both x and y to hotkey, or neither.");+ return invalid_pointer_arguments(+ "hotkey",+ "x and y must be passed together (a pixel focus point), or neither",+ );}if px.is_some() && resolved_element_index.is_some() {return ToolResult::error(@@ -5950,7 +9272,9 @@ impl Tool for HotkeyTool {}}- if let Some(element_index) = resolved_element_index {+ // Foreground: GrabFocus the addressed element up front. Background+ // does it inside `background_key_route`, under the focus guard.+ if let Some(element_index) = resolved_element_index.filter(|_| delivery.is_foreground()) {let focused = tokio::task::spawn_blocking(move || {crate::atspi::focus_element(pid, element_index)})@@ -5994,32 +9318,44 @@ impl Tool for HotkeyTool {}};let deliver_fg = delivery.is_foreground();+ let cursor_id = resolve_cursor_key(&args);- let result = tokio::task::spawn_blocking(move || -> anyhow::Result<()> {- if crate::wayland::wayland_input_enabled() {- // Native Wayland: route the modifier combo through wtype's- // -M/-k/-m sequence — the closest equivalent to the X11- // state-mask path. window_id is irrelevant once focused.- let mut combo: Vec<String> = mods_for_wayland.clone();- combo.push(key_for_wayland.clone());- return crate::wayland::hotkey(xid, &combo);- }- let m: Vec<&str> = mods.iter().map(String::as_str).collect();- // foreground: activate the target first, then inject the accelerator- // as REAL key events via XTest. Synthetic XSendEvent keys- // (`send_key`) are dropped by GTK/Qt/Chromium/Firefox, so the- // foreground rung must use XTest, which reaches the focused window.- if deliver_fg {- return crate::input::with_x11_foreground(xid, 80, || {- crate::input::send_key_xtest(&key, &m)- });- }- if let Some((x, y)) = px_target {- crate::input::send_key_at(xid, x, y, &key, &m)- } else {- crate::input::send_key(xid, &key, &m)- }- })+ let result = spawn_blocking_bounded(+ "hotkey",+ foreground_budget(1),+ move || -> anyhow::Result<KeyRoute> {+ if crate::wayland::wayland_input_enabled() {+ // Native Wayland: route the modifier combo through wtype's+ // -M/-k/-m sequence — the closest equivalent to the X11+ // state-mask path. window_id is irrelevant once focused.+ let mut combo: Vec<String> = mods_for_wayland.clone();+ combo.push(key_for_wayland.clone());+ return crate::wayland::hotkey(xid, &combo).map(|()| KeyRoute::Synthetic);+ }+ let m: Vec<&str> = mods.iter().map(String::as_str).collect();+ // foreground: activate the target first, then inject the accelerator+ // as REAL key events via XTest. Synthetic XSendEvent keys+ // (`send_key`) are dropped by GTK/Qt/Chromium/Firefox, so the+ // foreground rung must use XTest, which reaches the focused window.+ if deliver_fg {+ return crate::input::with_x11_foreground_opts(+ xid,+ crate::input::ForegroundOptions::keyboard(),+ || crate::input::send_key_xtest(&key, &m),+ )+ .map(|((), report)| KeyRoute::Foreground(report));+ }+ background_key_route(+ &cursor_id,+ pid,+ xid,+ px_target,+ &key,+ &m,+ resolved_element_index,+ )+ },+ ).await;let mode_label = if deliver_fg {"foreground"@@ -6027,11 +9363,12 @@ impl Tool for HotkeyTool {"background"};match result {- Ok(Ok(())) => ToolResult::text(format!(- "Pressed {key_display} on pid {pid} (delivery_mode={mode_label})."- ))- .with_structured(json!({ "verified": false, "delivery_mode": mode_label })),- Ok(Err(e)) => ToolResult::error(e.to_string()),+ Ok(Ok(route)) => key_route_result(+ &format!("Pressed {key_display} on pid {pid}"),+ route,+ mode_label,+ ),+ Ok(Err(e)) => input_error_result(e),Err(e) => ToolResult::error(format!("Task error: {e}")),}}@@ -6058,7 +9395,7 @@ impl Tool for SetValueTool {"element_index": cua_driver_core::tool_schema::element_index_schema(),"element_token": cua_driver_core::tool_schema::element_token_schema(),"snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),- "value":{"type":"string"}+ "value":{"type":["string","number"],"description":"New value. Written through AT-SPI EditableText/Value when the element exposes them; otherwise (GTK2 spin scales, VCL spin buttons) the field is clicked with the session's real pointer, its text selected and replaced through the virtual keyboard and committed with Tab, then read back."}},"additionalProperties":false}),read_only: false, destructive: true, idempotent: false, open_world: true,@@ -6071,9 +9408,13 @@ impl Tool for SetValueTool {Ok(v) => v,Err(e) => return e,};- let value = match args.require_str("value") {- Ok(v) => v,- Err(e) => return e,+ let value = match args.get("value") {+ Some(Value::String(text)) => text.clone(),+ Some(Value::Number(number)) => number.to_string(),+ _ => {+ return ToolResult::error("set_value: 'value' must be a string or a number.")+ .with_structured(json!({ "code": "invalid_arguments" }))+ }};// Surface 6: element_token / element_index precedence resolution.let resolved = match self.state.element_cache.resolve_element_args(@@ -6098,23 +9439,253 @@ impl Tool for SetValueTool {),};let value_for_task = value.clone();- let xid = args- .opt_u64("window_id")- .or(resolved_window_id)- .unwrap_or(0);+ let xid_opt = args.opt_u64("window_id").or(resolved_window_id);+ let xid = xid_opt.unwrap_or(0);+ let delivery = crate::input::delivery::DeliveryMode::from_args(&args);+ let cursor_id = resolve_cursor_key(&args);position_named_session_keyboard_cursor(&self.state, &args, pid, xid, Some(idx), None, true).await;- let result =- tokio::task::spawn_blocking(move || crate::atspi::set_value(pid, idx, &value_for_task))- .await;- match result {- Ok(Ok(())) => ToolResult::text(format!("Set value of element [{idx}] to '{value}'.")),- Ok(Err(e)) => ToolResult::error(e.to_string()),- Err(e) => ToolResult::error(format!("Task error: {e}")),+ // 1. Focus-free accessibility write on the exact snapshot object.+ let ax = spawn_blocking_bounded("set_value", foreground_budget(0), move || {+ crate::atspi::set_value_in(pid, xid_opt, idx, &value_for_task)+ })+ .await;+ let ax_error = match ax {+ Ok(Ok(())) => {+ let readback = tokio::task::spawn_blocking(move || {+ crate::atspi::read_value_in(pid, xid_opt, idx)+ })+ .await+ .ok()+ .flatten();+ return set_value_result(idx, &value, "ax", readback, None);+ }+ Ok(Err(error)) if crate::atspi::native::is_no_value_route(&error) => error,+ Ok(Err(error)) => return input_error_result(error),+ Err(error) => return ToolResult::error(format!("Task error: {error}")),+ };+ // 2. No accessibility write route: type it like a user would. Click+ // the field (real pointer), select all, type, commit with Tab.+ let wayland = crate::wayland::wayland_input_enabled();+ let background_route = !delivery.is_foreground()+ && !wayland+ && crate::input::real_pointer_input_available()+ && crate::input::real_keyboard_input_available();+ let foreground_route = delivery.is_foreground() && !wayland;+ if !background_route && !foreground_route {+ let detail = format!(+ "{ax_error}; the keyboard fallback needs {}",+ if wayland {+ "an X11 session"+ } else if delivery.is_foreground() {+ "an X11 session"+ } else {+ "the focus-free real-input route (a writable /dev/uinput), or delivery_mode:\"foreground\""+ }+ );+ return ToolResult::error(format!(+ "set_value: element [{idx}] has no accessibility write route and no keyboard \+ fallback is available: {detail}."+ ))+ .with_structured(json!({+ "code": "set_value_unavailable",+ "detail": detail,+ "effect": "none",+ "escalation": { "recommended": "foreground", "reason": "click the field and type the value with delivery_mode:\"foreground\"" },+ }));+ }+ let value_for_keys = value.clone();+ let typed = spawn_blocking_bounded(+ "set_value keyboard fallback",+ foreground_budget(value.len()),+ move || -> anyhow::Result<Option<crate::input::FocusGuardReport>> {+ let (win, lx, ly) = resolve_element_local_coords(pid, idx, xid_opt)?;+ if foreground_route {+ let (sx, sy) = window_local_to_screen(win, lx, ly)?;+ crate::input::with_x11_foreground_opts(+ win,+ crate::input::ForegroundOptions::pointer(),+ || {+ crate::input::send_click_xtest_desktop(+ sx.round() as i32,+ sy.round() as i32,+ 1,+ 1,+ )?;+ std::thread::sleep(std::time::Duration::from_millis(80));+ crate::input::send_key_xtest("a", &["ctrl"])?;+ crate::input::send_type_text_xtest(&value_for_keys)?;+ crate::input::send_key_xtest("Tab", &[])+ },+ )?;+ return Ok(None);+ }+ let (_, guard) = crate::input::focus_guard::guarded(Some(pid), || {+ x11_pixel_click_no_focus_steal(&cursor_id, win, lx as i32, ly as i32, 1, 1)?;+ std::thread::sleep(std::time::Duration::from_millis(80));+ background_virtual_key(&cursor_id, pid, win, "a", &["ctrl"])?;+ background_virtual_text(&cursor_id, pid, win, &value_for_keys)?;+ background_virtual_key(&cursor_id, pid, win, "Tab", &[])?;+ Ok(())+ })?;+ Ok(guard)+ },+ )+ .await;+ match typed {+ Ok(Ok(guard)) => {+ // Let the toolkit commit on focus-out before reading back.+ tokio::time::sleep(std::time::Duration::from_millis(150)).await;+ let readback = tokio::task::spawn_blocking(move || {+ crate::atspi::read_value_in(pid, xid_opt, idx)+ })+ .await+ .ok()+ .flatten();+ set_value_result(+ idx,+ &value,+ if foreground_route {+ "click_type_fg"+ } else {+ "click_type_mpx"+ },+ readback,+ guard.as_ref(),+ )+ }+ Ok(Err(error)) => input_error_result(error),+ Err(error) => ToolResult::error(format!("Task error: {error}")),}}}+/// Numeric-aware equality for the `set_value` read-back ("40" == "40.0").+fn values_agree(wanted: &str, observed: &str) -> bool {+ let wanted = wanted.trim();+ let observed = observed.trim();+ if wanted == observed {+ return true;+ }+ match (wanted.parse::<f64>(), observed.parse::<f64>()) {+ (Ok(a), Ok(b)) => (a - b).abs() <= 1e-6 * a.abs().max(1.0),+ _ => false,+ }+}++/// `set_value` result with an honest verification: `verified` only when the+/// element read back the requested value.+fn set_value_result(+ idx: usize,+ value: &str,+ path: &str,+ readback: Option<String>,+ guard: Option<&crate::input::FocusGuardReport>,+) -> ToolResult {+ let normalised_path = readback+ .as_deref()+ .is_some_and(|seen| path_normalised_to_basename(value, seen));+ let verified = readback+ .as_deref()+ .is_some_and(|seen| values_agree(value, seen))+ || normalised_path;+ let mut structured = json!({+ "path": path,+ "verified": verified,+ "effect": if verified { "confirmed" } else if readback.is_some() { "suspected_noop" } else { "unverifiable" },+ });+ let mut text = format!("Set value of element [{idx}] to '{value}' (path={path})");+ match readback {+ Some(seen) if normalised_path => {+ structured["readback"] = json!(seen);+ text.push_str(&format!(+ "; the field reads back '{seen}': a file chooser navigated to the path's \+ directory and kept the file name (the write committed)."+ ));+ }+ Some(seen) if verified => text.push_str(&format!("; read back '{seen}'.")),+ Some(seen) => {+ structured["readback"] = json!(seen);+ structured["escalation"] = non_ax_escalation();+ text.push_str(&format!(+ "; but the element reads back '{seen}' — the write may not have committed. \+ Confirm with get_window_state or a screenshot."+ ));+ }+ None => {+ structured["readback"] = Value::Null;+ text.push_str("; not verified — confirm with get_window_state.");+ }+ }+ attach_focus_guard(ToolResult::text(text).with_structured(structured), guard)+}++/// A Qt/GTK file chooser given an absolute path in its "File name" field+/// moves to that directory and shows only the final component: the readback+/// is the basename, and the write committed.+fn path_normalised_to_basename(value: &str, seen: &str) -> bool {+ let value = value.trim();+ let seen = seen.trim();+ value.contains('/')+ && !seen.is_empty()+ && value+ .rsplit('/')+ .next()+ .is_some_and(|base| !base.is_empty() && base == seen)+}++#[cfg(test)]+mod set_value_tests {+ use super::*;++ #[test]+ fn a_file_chooser_keeping_the_basename_counts_as_committed() {+ assert!(path_normalised_to_basename(+ "/home/user/Desktop/tone.wav",+ "tone.wav"+ ));+ assert!(!path_normalised_to_basename(+ "/home/user/Desktop/tone.wav",+ "other.wav"+ ));+ assert!(!path_normalised_to_basename("tone.wav", "tone.wav"));+ let ok = set_value_result(+ 3,+ "/home/user/Desktop/tone.wav",+ "ax",+ Some("tone.wav".into()),+ None,+ );+ let s = ok.structured_content.unwrap();+ assert_eq!(s["verified"], true);+ assert_eq!(s["effect"], "confirmed");+ }++ #[test]+ fn readback_agreement_is_numeric_aware() {+ assert!(values_agree("40", "40.0"));+ assert!(values_agree("40", " 40 "));+ assert!(values_agree("abc", "abc"));+ assert!(!values_agree("40", "41"));+ assert!(!values_agree("40", "forty"));+ }++ #[test]+ fn set_value_result_is_verified_only_on_matching_readback() {+ let ok = set_value_result(3, "40", "click_type_mpx", Some("40.0".into()), None);+ let s = ok.structured_content.unwrap();+ assert_eq!(s["verified"], true);+ assert_eq!(s["effect"], "confirmed");+ let noop = set_value_result(3, "40", "ax", Some("10.0".into()), None);+ let s = noop.structured_content.unwrap();+ assert_eq!(s["verified"], false);+ assert_eq!(s["effect"], "suspected_noop");+ assert_eq!(s["readback"], "10.0");+ let blind = set_value_result(3, "40", "ax", None, None);+ assert_eq!(blind.structured_content.unwrap()["effect"], "unverifiable");+ }+}+// ── scroll ────────────────────────────────────────────────────────────────────pub struct ScrollTool {@@ -6230,6 +9801,7 @@ impl Tool for ScrollTool {"x":{"type":"number","description":"Window-local screenshot-pixel X of the scroll target. Pass with y and without element_index."},"y":{"type":"number","description":"Window-local screenshot-pixel Y of the scroll target. Pass with x and without element_index."},"scope":{"type":"string","enum":["window","desktop"],"default":"window"},+ "coordinate_frame": coordinate_frame_schema(),"delivery_mode": crate::input::delivery::delivery_mode_schema()},"additionalProperties":false}),@@ -6248,8 +9820,9 @@ impl Tool for ScrollTool {Err(result) => return result,};let direction = input.direction.as_str().to_owned();- let x = input.x.round() as i32;- let y = input.y.round() as i32;+ let (x, y) = (input.x, input.y);+ let x = x.round() as i32;+ let y = y.round() as i32;let amount = input.amount.unwrap_or(3).clamp(1, 50) as usize;let display = direction.clone();let wayland = crate::wayland::wayland_input_enabled();@@ -6281,7 +9854,7 @@ impl Tool for ScrollTool {Err(error) => ToolResult::error(format!("Task error: {error}")),};}- let pid = match args.require_u32("pid") {+ let mut pid = match args.require_u32("pid") {Ok(v) => v,Err(e) => return e,};@@ -6323,12 +9896,43 @@ impl Tool for ScrollTool {let xid = match xid_opt {Some(x) => x,None => {- let windows =- tokio::task::spawn_blocking(move || crate::x11::list_windows(Some(pid)))- .await- .unwrap_or_default();- match windows.first() {- Some(w) => w.xid,+ let windows = tokio::task::spawn_blocking(move || {+ crate::x11::list_windows(if pid == 0 { None } else { Some(pid) })+ })+ .await+ .unwrap_or_default();+ // pid omitted: the keys go to the active window, like a+ // physical keyboard would, and the action adopts its pid.+ let chosen = if pid == 0 {+ let active = crate::x11::active_window();+ windows+ .iter()+ .find(|w| Some(w.xid) == active && w.pid.is_some())+ .or_else(|| windows.iter().find(|w| w.is_on_screen && w.pid.is_some()))+ } else {+ // The pid's active window, else its largest on-screen+ // toplevel: LibreOffice and GIMP own hidden/utility+ // toplevels that a plain `first()` could pick, and an+ // unmapped window cannot take the virtual keyboard focus.+ let active = crate::x11::active_window();+ windows+ .iter()+ .find(|w| Some(w.xid) == active)+ .or_else(|| {+ windows+ .iter()+ .filter(|w| w.is_on_screen)+ .max_by_key(|w| u64::from(w.width) * u64::from(w.height))+ })+ .or_else(|| windows.first())+ };+ match chosen {+ Some(w) => {+ if pid == 0 {+ pid = w.pid.unwrap_or(0);+ }+ w.xid+ }None => {return ToolResult::error(format!("No windows found for pid {pid}. Provide window_id."@@ -6342,6 +9946,20 @@ impl Tool for ScrollTool {args.get("x").and_then(|value| value.as_f64()),args.get("y").and_then(|value| value.as_f64()),) {+ (Some(x), Some(y)) if desktop_frame_requested(&args) => {+ // Desktop-frame pixels (from get_desktop_state) against a named+ // window: map into the window-local frame the pipeline expects.+ match tokio::task::spawn_blocking(move || desktop_to_window_local(xid, x, y)).await+ {+ Ok(Ok(local)) => Some(local),+ Ok(Err(e)) => {+ return ToolResult::error(format!(+ "desktop-frame coordinates could not be mapped into window {xid}: {e}"+ ))+ }+ Err(e) => return ToolResult::error(format!("Task error: {e}")),+ }+ }(Some(x), Some(y)) => {// Pixel targets use the latest screenshot's coordinate frame.// Apply the same buffer-to-window ratio as click/drag before@@ -6353,7 +9971,8 @@ impl Tool for ScrollTool {Some((x * ratio, y * ratio))}(None, None) => None,- _ => return ToolResult::error("Pass both x and y to pixel-target scroll."),+ (Some(_), None) => return invalid_pointer_arguments("scroll", "x was given without y"),+ (None, Some(_)) => return invalid_pointer_arguments("scroll", "y was given without x"),};let resolved_element_index = match &resolved {cua_driver_core::element_token::ResolvedElement::Element { element_index, .. } => {@@ -6701,7 +10320,7 @@ impl Tool for ScrollTool {"Scrolled {direction} {amount} ticks (delivery_mode={mode_label}).")).with_structured(json!({ "verified": false, "delivery_mode": mode_label })),- Ok(Err(e)) => linux_input_error(e),+ Ok(Err(e)) => input_error_result(e),Err(e) => ToolResult::error(format!("Task error: {e}")),}}@@ -6725,16 +10344,20 @@ impl Tool for DoubleClickTool {fn def(&self) -> &ToolDef {DCLICK_DEF.get_or_init(|| ToolDef {name: "double_click".into(),- description: "Double-click at (x,y) or an element_index (AT-SPI bounds) via XSendEvent. \- No focus steal. Provide either (window_id + x/y) or (pid + element_index). \+ description: "Double-click at (x,y) or an element_index (AT-SPI bounds). Background \+ delivery on X11 sends real button events from the session's virtual master pointer \+ (path mpx_pointer); a point covered by another application's window is refused \+ (target_occluded), one covered by this app's own dialog is retargeted to it \+ (retargeted_to). No focus steal. Provide either (window_id + x/y) or (pid + element_index). \After a zoom call, pass from_zoom=true to auto-translate zoom-image coords.".into(),input_schema: json!({"type":"object","required":["pid"],"properties":{"session": cua_driver_core::tool_schema::session_schema(),"cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},"pid":{"type":"integer"},"window_id":{"type":"integer"},- "x":{"type":"number"},- "y":{"type":"number"},+ "x":{"type":"number","description":"Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\")."},+ "y":{"type":"number","description":"Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x."},+ "coordinate_frame": coordinate_frame_schema(),"element_index": cua_driver_core::tool_schema::element_index_schema(),"element_token": cua_driver_core::tool_schema::element_token_schema(),"snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),@@ -6801,7 +10424,7 @@ impl Tool for DoubleClickTool {if let Some(idx) = elem_idx_resolved {let xid_hint = window_id_resolved;let result = tokio::task::spawn_blocking(move || -> anyhow::Result<(u64, f64, f64)> {- resolve_element_local_coords(pid, idx, xid_hint)+ checked_element_local_coords(pid, idx, xid_hint)}).await;return match result {@@ -6822,48 +10445,66 @@ impl Tool for DoubleClickTool {let wayland_point = crate::wayland::wayland_input_enabled().then(|| crate::wayland::window_local_to_output(xid, lxi, lyi));let cursor_id_for_task = cursor_id.clone();- let click_result = tokio::task::spawn_blocking(move || {- if crate::wayland::is_inject_mode() {- return crate::wayland::inject_click(pid, xid, lx, ly, 2, 1);- }- if crate::wayland::wayland_input_enabled() {- let (output_x, output_y) = wayland_point.unwrap_or((lxi, lyi));- return crate::wayland::click(xid, output_x, output_y, 2, 1);- }- if delivery.is_foreground() {- return crate::input::with_x11_foreground(xid, 80, || {- let (sx, sy) = window_local_to_screen(xid, lxi as f64, lyi as f64)?;- crate::input::send_click_xtest_desktop(- sx.round() as i32,- sy.round() as i32,- 1,- 2,- )- });- }- x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, lxi, lyi, 1, 2)- })- .await;+ let click_result =+ tokio::task::spawn_blocking(move || -> anyhow::Result<PointerRoute> {+ if crate::wayland::is_inject_mode() {+ return crate::wayland::inject_click(pid, xid, lx, ly, 2, 1)+ .map(|()| PointerRoute::Wayland);+ }+ if crate::wayland::wayland_input_enabled() {+ let (output_x, output_y) = wayland_point.unwrap_or((lxi, lyi));+ return crate::wayland::click(xid, output_x, output_y, 2, 1)+ .map(|()| PointerRoute::Wayland);+ }+ if delivery.is_foreground() {+ return crate::input::with_x11_foreground(xid, 80, || {+ let (sx, sy) =+ window_local_to_screen(xid, lxi as f64, lyi as f64)?;+ crate::input::send_click_xtest_desktop(+ sx.round() as i32,+ sy.round() as i32,+ 1,+ 2,+ )+ })+ .map(|()| PointerRoute::Foreground);+ }+ x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, lxi, lyi, 1, 2)+ })+ .await;+ let mode_label = if delivery.is_foreground() {+ "foreground"+ } else {+ "background"+ };match click_result {- Ok(Ok(())) => {- ToolResult::text(format!("✅ Double-clicked element [{idx}]."))- }- Ok(Err(e)) => linux_input_error(e),+ Ok(Ok(route)) => ToolResult::text(format!(+ "Double-clicked element [{idx}] {}",+ route.text_suffix(mode_label)+ ))+ .with_structured(route.structured(mode_label)),+ Ok(Err(e)) => input_error_result(e),Err(e) => ToolResult::error(format!("Task error: {e}")),}}- Ok(Err(e)) => ToolResult::error(format!("AT-SPI bounds failed: {e}")),+ Ok(Err(e)) => input_error_result(e.context("AT-SPI bounds failed")),Err(e) => ToolResult::error(format!("Task error: {e}")),};}+ if let Some(refusal) = require_point_args("double_click", &args) {+ return refusal;+ }let xid = match window_id_resolved {Some(v) => v,None => return ToolResult::error("Provide either element_index or window_id + x/y."),};+ let desktop_frame = desktop_frame_requested(&args);let from_zoom = args.bool_or("from_zoom", false);let mut x = args.f64_or("x", 0.0);let mut y = args.f64_or("y", 0.0);- if from_zoom {+ if desktop_frame {+ // Desktop pixels: no zoom/screenshot scaling; translated once below.+ } else if from_zoom {match self.state.zoom_context(&args, pid, Some(xid)) {Ok(ctx) => {let (wx, wy) = ctx.zoom_to_window(x, y);@@ -6880,6 +10521,20 @@ impl Tool for DoubleClickTool {x *= ratio;y *= ratio;}+ if desktop_frame {+ match tokio::task::spawn_blocking(move || desktop_to_window_local(xid, x, y)).await {+ Ok(Ok((lx, ly))) => {+ x = lx;+ y = ly;+ }+ Ok(Err(e)) => {+ return ToolResult::error(format!(+ "desktop-frame coordinates could not be mapped into window {xid}: {e}"+ ))+ }+ Err(e) => return ToolResult::error(format!("Task error: {e}")),+ }+ }crate::overlay::send_command_for(cursor_id.clone(),cursor_overlay::OverlayCommand::PinAbove(xid),@@ -6906,13 +10561,15 @@ impl Tool for DoubleClickTool {}let (xi, yi) = (x as i32, y as i32);let cursor_id_for_task = cursor_id.clone();- let result = tokio::task::spawn_blocking(move || -> anyhow::Result<()> {+ let result = tokio::task::spawn_blocking(move || -> anyhow::Result<PointerRoute> {if crate::wayland::is_inject_mode() {- return crate::wayland::inject_click(pid, xid, x, y, 2, 1);+ return crate::wayland::inject_click(pid, xid, x, y, 2, 1)+ .map(|()| PointerRoute::Wayland);}if crate::wayland::wayland_input_enabled() {let (output_x, output_y) = wayland_output_point.unwrap_or((xi, yi));- return crate::wayland::click(xid, output_x, output_y, 2, 1);+ return crate::wayland::click(xid, output_x, output_y, 2, 1)+ .map(|()| PointerRoute::Wayland);}if delivery.is_foreground() {return crate::input::with_x11_foreground(xid, 80, || {@@ -6930,7 +10587,9 @@ impl Tool for DoubleClickTool {return Ok(());}x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, xi, yi, 1, 2)- });+ .map(|_| ())+ })+ .map(|()| PointerRoute::Foreground);}x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, xi, yi, 1, 2)})@@ -6940,14 +10599,20 @@ impl Tool for DoubleClickTool {} else {"background"};- match result {- Ok(Ok(())) => ToolResult::text(format!(- "✅ Double-clicked at ({x:.1}, {y:.1}) (delivery_mode={mode_label})."+ let outcome = match result {+ Ok(Ok(route)) => ToolResult::text(format!(+ "Double-clicked at ({x:.1}, {y:.1}) {}",+ route.text_suffix(mode_label)))- .with_structured(json!({ "verified": false, "delivery_mode": mode_label })),- Ok(Err(e)) => linux_input_error(e),+ .with_structured(route.structured(mode_label)),+ Ok(Err(e)) => input_error_result(e),Err(e) => ToolResult::error(format!("Task error: {e}")),- }+ };+ with_pixel_points(+ outcome,+ glide_target.map(|(sx, sy)| (sx.round() as i32, sy.round() as i32)),+ (xi, yi),+ )}}@@ -6963,16 +10628,24 @@ impl Tool for RightClickTool {fn def(&self) -> &ToolDef {RCLICK_DEF.get_or_init(|| ToolDef {name: "right_click".into(),- description: "Right-click at (x,y) or an element_index (AT-SPI bounds) via XSendEvent. \- No focus steal. Provide either (window_id + x/y) or (pid + element_index). \+ description: "Right-click at (x,y) or an element_index (AT-SPI bounds). Background \+ delivery on X11 sends real button events from the session's virtual master pointer \+ (path mpx_pointer); a point covered by another application's window is refused \+ (target_occluded), one covered by this app's own dialog is retargeted to it \+ (retargeted_to), one outside the window is refused (point_outside_window; x/y are \+ window-local pixels, the result reports window_point / screen_point). No focus \+ steal. When a context menu opens, the result names it (`popup: {window_id, bounds, \+ title}`): call get_window_state(pid, window_id=<that id>) to index its menu items \+ and click them by element_index. Provide either (window_id + x/y) or (pid + element_index). \After a zoom call, pass from_zoom=true to auto-translate zoom-image coords.".into(),input_schema: json!({"type":"object","required":["pid"],"properties":{"session": cua_driver_core::tool_schema::session_schema(),"cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},"pid":{"type":"integer"},"window_id":{"type":"integer"},- "x":{"type":"number"},- "y":{"type":"number"},+ "x":{"type":"number","description":"Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\")."},+ "y":{"type":"number","description":"Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x."},+ "coordinate_frame": coordinate_frame_schema(),"element_index": cua_driver_core::tool_schema::element_index_schema(),"element_token": cua_driver_core::tool_schema::element_token_schema(),"snapshot_id": cua_driver_core::tool_schema::snapshot_id_schema(),@@ -7040,7 +10713,7 @@ impl Tool for RightClickTool {if let Some(idx) = elem_idx_resolved {let xid_hint = window_id_resolved;let result = tokio::task::spawn_blocking(move || -> anyhow::Result<(u64, f64, f64)> {- resolve_element_local_coords(pid, idx, xid_hint)+ checked_element_local_coords(pid, idx, xid_hint)}).await;return match result {@@ -7061,48 +10734,66 @@ impl Tool for RightClickTool {let wayland_point = crate::wayland::wayland_input_enabled().then(|| crate::wayland::window_local_to_output(xid, lxi, lyi));let cursor_id_for_task = cursor_id.clone();- let click_result = tokio::task::spawn_blocking(move || {- if crate::wayland::is_inject_mode() {- return crate::wayland::inject_click(pid, xid, lx, ly, 1, 3);- }- if crate::wayland::wayland_input_enabled() {- let (output_x, output_y) = wayland_point.unwrap_or((lxi, lyi));- return crate::wayland::click(xid, output_x, output_y, 1, 3);- }- if delivery.is_foreground() {- return crate::input::with_x11_foreground(xid, 80, || {- let (sx, sy) = window_local_to_screen(xid, lxi as f64, lyi as f64)?;- crate::input::send_click_xtest_desktop(- sx.round() as i32,- sy.round() as i32,- 3,- 1,- )- });- }- x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, lxi, lyi, 3, 1)- })- .await;+ let click_result =+ tokio::task::spawn_blocking(move || -> anyhow::Result<PointerRoute> {+ if crate::wayland::is_inject_mode() {+ return crate::wayland::inject_click(pid, xid, lx, ly, 1, 3)+ .map(|()| PointerRoute::Wayland);+ }+ if crate::wayland::wayland_input_enabled() {+ let (output_x, output_y) = wayland_point.unwrap_or((lxi, lyi));+ return crate::wayland::click(xid, output_x, output_y, 1, 3)+ .map(|()| PointerRoute::Wayland);+ }+ if delivery.is_foreground() {+ return crate::input::with_x11_foreground(xid, 80, || {+ let (sx, sy) =+ window_local_to_screen(xid, lxi as f64, lyi as f64)?;+ crate::input::send_click_xtest_desktop(+ sx.round() as i32,+ sy.round() as i32,+ 3,+ 1,+ )+ })+ .map(|()| PointerRoute::Foreground);+ }+ x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, lxi, lyi, 3, 1)+ })+ .await;+ let mode_label = if delivery.is_foreground() {+ "foreground"+ } else {+ "background"+ };match click_result {- Ok(Ok(())) => {- ToolResult::text(format!("✅ Right-clicked element [{idx}]."))- }- Ok(Err(e)) => linux_input_error(e),+ Ok(Ok(route)) => ToolResult::text(format!(+ "Right-clicked element [{idx}] {}",+ route.text_suffix(mode_label)+ ))+ .with_structured(route.structured(mode_label)),+ Ok(Err(e)) => input_error_result(e),Err(e) => ToolResult::error(format!("Task error: {e}")),}}- Ok(Err(e)) => ToolResult::error(format!("AT-SPI bounds failed: {e}")),+ Ok(Err(e)) => input_error_result(e.context("AT-SPI bounds failed")),Err(e) => ToolResult::error(format!("Task error: {e}")),};}+ if let Some(refusal) = require_point_args("right_click", &args) {+ return refusal;+ }let xid = match window_id_resolved {Some(v) => v,None => return ToolResult::error("Provide either element_index or window_id + x/y."),};+ let desktop_frame = desktop_frame_requested(&args);let from_zoom = args.bool_or("from_zoom", false);let mut x = args.f64_or("x", 0.0);let mut y = args.f64_or("y", 0.0);- if from_zoom {+ if desktop_frame {+ // Desktop pixels: no zoom/screenshot scaling; translated once below.+ } else if from_zoom {match self.state.zoom_context(&args, pid, Some(xid)) {Ok(ctx) => {let (wx, wy) = ctx.zoom_to_window(x, y);@@ -7119,6 +10810,20 @@ impl Tool for RightClickTool {x *= ratio;y *= ratio;}+ if desktop_frame {+ match tokio::task::spawn_blocking(move || desktop_to_window_local(xid, x, y)).await {+ Ok(Ok((lx, ly))) => {+ x = lx;+ y = ly;+ }+ Ok(Err(e)) => {+ return ToolResult::error(format!(+ "desktop-frame coordinates could not be mapped into window {xid}: {e}"+ ))+ }+ Err(e) => return ToolResult::error(format!("Task error: {e}")),+ }+ }crate::overlay::send_command_for(cursor_id.clone(),cursor_overlay::OverlayCommand::PinAbove(xid),@@ -7145,13 +10850,15 @@ impl Tool for RightClickTool {}let (xi, yi) = (x as i32, y as i32);let cursor_id_for_task = cursor_id.clone();- let result = tokio::task::spawn_blocking(move || -> anyhow::Result<()> {+ let result = tokio::task::spawn_blocking(move || -> anyhow::Result<PointerRoute> {if crate::wayland::is_inject_mode() {- return crate::wayland::inject_click(pid, xid, x, y, 1, 3);+ return crate::wayland::inject_click(pid, xid, x, y, 1, 3)+ .map(|()| PointerRoute::Wayland);}if crate::wayland::wayland_input_enabled() {let (output_x, output_y) = wayland_output_point.unwrap_or((xi, yi));- return crate::wayland::click(xid, output_x, output_y, 1, 3);+ return crate::wayland::click(xid, output_x, output_y, 1, 3)+ .map(|()| PointerRoute::Wayland);}if delivery.is_foreground() {return crate::input::with_x11_foreground(xid, 80, || {@@ -7168,7 +10875,9 @@ impl Tool for RightClickTool {return Ok(());}x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, xi, yi, 3, 1)- });+ .map(|_| ())+ })+ .map(|()| PointerRoute::Foreground);}x11_pixel_click_no_focus_steal(&cursor_id_for_task, xid, xi, yi, 3, 1)})@@ -7178,14 +10887,20 @@ impl Tool for RightClickTool {} else {"background"};- match result {- Ok(Ok(())) => ToolResult::text(format!(- "✅ Right-clicked at ({x:.1}, {y:.1}) (delivery_mode={mode_label})."+ let outcome = match result {+ Ok(Ok(route)) => ToolResult::text(format!(+ "Right-clicked at ({x:.1}, {y:.1}) {}",+ route.text_suffix(mode_label)))- .with_structured(json!({ "verified": false, "delivery_mode": mode_label })),- Ok(Err(e)) => linux_input_error(e),+ .with_structured(route.structured(mode_label)),+ Ok(Err(e)) => input_error_result(e),Err(e) => ToolResult::error(format!("Task error: {e}")),- }+ };+ with_pixel_points(+ outcome,+ glide_target.map(|(sx, sy)| (sx.round() as i32, sy.round() as i32)),+ (xi, yi),+ )}}@@ -7214,23 +10929,28 @@ impl Tool for DragTool {DRAG_DEF.get_or_init(|| ToolDef {name: "drag".into(),description: "Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in \- window-local screenshot pixels via XSendEvent (ButtonPress + MotionNotify × steps + ButtonRelease). \- duration_ms (default 500), steps (default 20). No focus steal.".into(),+ window-local screenshot pixels. Background delivery on X11 is one real held \+ gesture on the session's virtual master pointer (path mpx_pointer: press, \+ interpolated motion, release) — it lands on GTK/VCL/Qt/Chromium; a point \+ covered by another application's window is refused (target_occluded), one \+ covered by this app's own dialog is retargeted to it. Headless servers fall \+ back to XSendEvent. duration_ms (default 500), steps (default 20). No focus steal.".into(),input_schema: json!({"type":"object","required":["from_x","from_y","to_x","to_y"],"properties":{"session": cua_driver_core::tool_schema::session_schema(),"cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},"pid":{"type":"integer"},"window_id":{"type":"integer","description":"Target window XID. Required."},- "from_x":{"type":"number"},- "from_y":{"type":"number"},- "to_x":{"type":"number"},- "to_y":{"type":"number"},+ "from_x":{"type":"number","description":"Drag start X: window-local pixels of the target window's own get_window_state screenshot; pass scope:\"desktop\" for get_desktop_state pixels."},+ "from_y":{"type":"number","description":"Drag start Y (same frame as from_x)."},+ "to_x":{"type":"number","description":"Drag end X (same frame as from_x)."},+ "to_y":{"type":"number","description":"Drag end Y (same frame as from_x)."},"duration_ms":{"type":"integer","minimum":0,"maximum":10000,"description":"Total drag duration. Default: 500."},"steps":{"type":"integer","minimum":1,"maximum":200,"description":"Intermediate MotionNotify events. Default: 20."},"modifier": cua_driver_core::tool_schema::modifier_schema(),"button": cua_driver_core::tool_schema::button_schema(),"from_zoom":{"type":"boolean"},"scope":{"type":"string","enum":["window","desktop"],"default":"window"},+ "coordinate_frame": coordinate_frame_schema(),"delivery_mode": crate::input::delivery::delivery_mode_schema()},"additionalProperties":false}),read_only: false, destructive: true, idempotent: false, open_world: true,@@ -7246,7 +10966,8 @@ impl Tool for DragTool {Ok(input) => input,Err(result) => return result,};- let (from_x, from_y, to_x, to_y) = (input.from_x, input.from_y, input.to_x, input.to_y);+ let (from_x, from_y) = (input.from_x, input.from_y);+ let (to_x, to_y) = (input.to_x, input.to_y);let button = parse_mouse_button(input.button.unwrap_or(ClickButton::Left).as_str());let duration_ms = input.duration_ms.unwrap_or(500).min(10_000);let steps = input.steps.unwrap_or(20).clamp(1, 200) as usize;@@ -7322,8 +11043,12 @@ impl Tool for DragTool {};let from_zoom = args.bool_or("from_zoom", false);+ let desktop_frame = desktop_frame_requested(&args);let context = match coordinate_drag_context(native_refusal, || {- if from_zoom {+ if desktop_frame {+ // Desktop pixels: no zoom/screenshot scaling; translated below.+ Ok(CoordinateContext::Screenshot(1.0))+ } else if from_zoom {self.state.zoom_context(&args, pid, Some(xid)).map(CoordinateContext::Zoom)@@ -7342,11 +11067,21 @@ impl Tool for DragTool {};let mut from_x = match coerce("from_x") {Some(v) => v,- None => return ToolResult::error("Missing: from_x"),+ None => {+ return invalid_pointer_arguments(+ "drag",+ "from_x is missing (from_x/from_y/to_x/to_y are all required)",+ )+ }};let mut from_y = match coerce("from_y") {Some(v) => v,- None => return ToolResult::error("Missing: from_y"),+ None => {+ return invalid_pointer_arguments(+ "drag",+ "from_y is missing (from_x/from_y/to_x/to_y are all required)",+ )+ }};let mut to_x = match coerce("to_x") {Some(v) => v,@@ -7373,6 +11108,29 @@ impl Tool for DragTool {to_y *= scale;}}+ if desktop_frame {+ let mapped = tokio::task::spawn_blocking(move || {+ Ok::<_, anyhow::Error>((+ desktop_to_window_local(xid, from_x, from_y)?,+ desktop_to_window_local(xid, to_x, to_y)?,+ ))+ })+ .await;+ match mapped {+ Ok(Ok(((fx, fy), (tx, ty)))) => {+ from_x = fx;+ from_y = fy;+ to_x = tx;+ to_y = ty;+ }+ Ok(Err(e)) => {+ return ToolResult::error(format!(+ "desktop-frame coordinates could not be mapped into window {xid}: {e}"+ ))+ }+ Err(e) => return ToolResult::error(format!("Task error: {e}")),+ }+ }if hyprland_foreground(delivery) {if button_str != "left" || args.get("modifier").is_some_and(|value| !value.is_null()) {@@ -7588,7 +11346,7 @@ impl Tool for DragTool {from ({from_x:.0}, {from_y:.0}) → ({to_x:.0}, {to_y:.0}) \in {duration_ms}ms / {steps} steps.")),- Ok(Err(e)) => ToolResult::error(e.to_string()),+ Ok(Err(e)) => input_error_result(e),Err(e) => ToolResult::error(format!("Task error: {e}")),};}@@ -7643,11 +11401,93 @@ impl Tool for DragTool {"verified": false,"delivery_mode": "foreground"})),- Ok(Err(e)) => ToolResult::error(e.to_string()),+ Ok(Err(e)) => input_error_result(e),+ Err(e) => ToolResult::error(format!("Task error: {e}")),+ };+ }++ // Background X11 drag. With a real Xorg server and /dev/uinput the+ // gesture is a held press + interpolated motion + release on the+ // session's MPX virtual master pointer (GTK/VCL/Qt/Chromium drop the+ // synthetic XSendEvent gesture below); the target's screen points are+ // resolved once and the agent cursor tracks the glide.+ if crate::input::real_pointer_input_available() {+ let screen_points = tokio::task::spawn_blocking(move || {+ Ok::<_, anyhow::Error>((+ window_local_to_screen(xid, from_x, from_y)?,+ window_local_to_screen(xid, to_x, to_y)?,+ ))+ })+ .await;+ let ((sfx, sfy), (stx, sty)) = match screen_points {+ Ok(Ok(points)) => points,+ Ok(Err(e)) => return ToolResult::error(format!("{e:#}")),+ Err(e) => return ToolResult::error(format!("Task error: {e}")),+ };+ crate::overlay::send_command_for(+ cursor_id.clone(),+ cursor_overlay::OverlayCommand::SetPressed(true),+ );+ let cursor_id_for_task = cursor_id.clone();+ let drag_result = tokio::task::spawn_blocking(move || {+ let (outcome, guard) =+ crate::input::focus_guard::guarded_settled(Some(pid), || {+ Ok(crate::input::send_virtual_pointer_drag(+ &cursor_id_for_task,+ &crate::input::VirtualPointerDrag {+ target_window: xid,+ button,+ path: vec![+ (sfx.round() as i32, sfy.round() as i32),+ (stx.round() as i32, sty.round() as i32),+ ],+ duration_ms,+ steps,+ },+ ))+ })?;+ outcome.map(|mut effect| {+ effect.focus_guard = guard;+ effect+ })+ })+ .await;+ crate::overlay::send_command_for(+ cursor_id.clone(),+ cursor_overlay::OverlayCommand::SetPressed(false),+ );+ if matches!(&drag_result, Ok(Ok(_))) {+ crate::overlay::send_command_for(+ cursor_id.clone(),+ cursor_overlay::track_pointer_command(stx, sty),+ );+ self.state+ .cursor_registry+ .update_position(&cursor_id, stx, sty);+ }+ return match drag_result {+ Ok(Ok(effect)) => {+ let route = PointerRoute::Mpx(effect);+ ToolResult::text(format!(+ "Dragged ({button_str}) pid {pid} from ({from_x:.0}, {from_y:.0}) to \+ ({to_x:.0}, {to_y:.0}) in {duration_ms}ms / {steps} steps with a real \+ held pointer gesture {}",+ route.text_suffix("background")+ ))+ .with_structured(route.structured("background"))+ }+ Ok(Err(e)) => input_error_result(e),Err(e) => ToolResult::error(format!("Task error: {e}")),};}+ // Legacy target-addressed XSendEvent gesture (headless servers / no+ // uinput): only core-protocol toolkits accept it.+ if synthetic_pointer_is_dropped(pid) {+ return crate::input::delivery::background_unavailable_error(+ crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,+ );+ }crate::overlay::send_command_for(cursor_id.clone(),cursor_overlay::OverlayCommand::SetPressed(true),@@ -7747,13 +11587,16 @@ impl Tool for DragTool {}}+ let route = PointerRoute::Synthetic;match result {Ok(()) => ToolResult::text(format!(- "✅ Posted drag ({button_str}) to pid {pid} \+ "Posted drag ({button_str}) to pid {pid} \from ({from_x:.0}, {from_y:.0}) → ({to_x:.0}, {to_y:.0}) \- in {duration_ms}ms / {steps} steps."- )),- Err(e) => ToolResult::error(e.to_string()),+ in {duration_ms}ms / {steps} steps {}",+ route.text_suffix("background")+ ))+ .with_structured(route.structured("background")),+ Err(e) => input_error_result(e),}}}@@ -7780,8 +11623,9 @@ impl Tool for MouseButtonDownTool {"cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},"pid":{"type":"integer"},"window_id":{"type":"integer"},- "x":{"type":"number"},- "y":{"type":"number"},+ "x":{"type":"number","description":"Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\")."},+ "y":{"type":"number","description":"Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x."},+ "coordinate_frame": coordinate_frame_schema(),"button": cua_driver_core::tool_schema::button_schema(),"from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."}},"additionalProperties":false}),@@ -7817,7 +11661,10 @@ impl Tool for MouseButtonDownTool {let button = parse_mouse_button(button_name.as_str());let mut x = args.f64_or("x", 0.0);let mut y = args.f64_or("y", 0.0);- if args.bool_or("from_zoom", false) {+ let desktop_frame = desktop_frame_requested(&args);+ if desktop_frame {+ // Desktop pixels: no zoom/screenshot scaling; translated once below.+ } else if args.bool_or("from_zoom", false) {match self.state.zoom_context(&args, pid, Some(xid)) {Ok(ctx) => {let (wx, wy) = ctx.zoom_to_window(x, y);@@ -7834,6 +11681,20 @@ impl Tool for MouseButtonDownTool {x *= ratio;y *= ratio;}+ if desktop_frame {+ match tokio::task::spawn_blocking(move || desktop_to_window_local(xid, x, y)).await {+ Ok(Ok((lx, ly))) => {+ x = lx;+ y = ly;+ }+ Ok(Err(e)) => {+ return ToolResult::error(format!(+ "desktop-frame coordinates could not be mapped into window {xid}: {e}"+ ))+ }+ Err(e) => return ToolResult::error(format!("Task error: {e}")),+ }+ }crate::overlay::send_command_for(cursor_id.clone(),@@ -7938,8 +11799,8 @@ impl Tool for MouseDragTool {"cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},"pid":{"type":"integer"},"window_id":{"type":"integer"},- "x":{"type":"number"},- "y":{"type":"number"},+ "x":{"type":"number","description":"Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\")."},+ "y":{"type":"number","description":"Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x."},"duration_ms":{"type":"integer","minimum":0,"maximum":10000,"description":"Total drag duration. Default: 500."},"steps":{"type":"integer","minimum":1,"maximum":200,"description":"Intermediate MotionNotify events. Default: 20."},"from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."}@@ -7967,6 +11828,9 @@ impl Tool for MouseDragTool {return err;}+ if let Some(refusal) = require_point_args("mouse_drag", &args) {+ return refusal;+ }let mut to_x = args.f64_or("x", 0.0);let mut to_y = args.f64_or("y", 0.0);if args.bool_or("from_zoom", false) {@@ -8145,8 +12009,9 @@ impl Tool for MouseButtonUpTool {"cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},"pid":{"type":"integer"},"window_id":{"type":"integer"},- "x":{"type":"number"},- "y":{"type":"number"},+ "x":{"type":"number","description":"Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\")."},+ "y":{"type":"number","description":"Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x."},+ "coordinate_frame": coordinate_frame_schema(),"from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."}},"additionalProperties":false}),read_only: false, destructive: true, idempotent: false, open_world: true,@@ -8174,10 +12039,36 @@ impl Tool for MouseButtonUpTool {let xid = hold.xid;- let (x, y) = match mouse_button_up_coordinates(&self.state, &args, &hold) {- Ok(point) => point,- Err(refusal) => return refusal,+ let desktop_frame = args.opt_f64("x").is_some()+ && args.opt_f64("y").is_some()+ && desktop_frame_requested(&args);+ let (mut x, mut y) = if desktop_frame {+ // Desktop pixels: no zoom/screenshot scaling; translated below.+ (args.f64_or("x", hold.x), args.f64_or("y", hold.y))+ } else {+ match mouse_button_up_coordinates(&self.state, &args, &hold) {+ Ok(point) => point,+ Err(refusal) => return refusal,+ }};+ if desktop_frame {+ match tokio::task::spawn_blocking(move || desktop_to_window_local(xid, x, y)).await {+ Ok(Ok((lx, ly))) => {+ x = lx;+ y = ly;+ }+ Ok(Err(e)) => {+ return ToolResult::error(format!(+ "desktop-frame coordinates could not be mapped into window {xid}: {e}"+ ))+ .with_structured(mouse_hold_json(&cursor_id, Some(&hold)))+ }+ Err(e) => {+ return ToolResult::error(format!("Task error: {e}"))+ .with_structured(mouse_hold_json(&cursor_id, Some(&hold)))+ }+ }+ }crate::overlay::send_command_for(cursor_id.clone(),@@ -8699,7 +12590,7 @@ fn normalize_desktop_capture_for_action_frame(// ── get_desktop_state ─────────────────────────────────────────────────────────pub struct GetDesktopStateTool {- capture_service: Arc<cua_driver_core::capture_runtime::CaptureService>,+ state: Arc<ToolState>,}static GDS_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();@@ -8713,7 +12604,8 @@ impl Tool for GetDesktopStateTool {{kind:\"desktop\",display_id:\"primary\"}. No AT-SPI walk.".into(),input_schema: json!({"type":"object","properties":{"session":{"type":"string","description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session."},- "screenshot_out_file":{"type":"string","description":"Write PNG here instead of base64."}+ "screenshot_out_file":{"type":"string","description":"Write PNG here instead of base64."},+ "max_image_dimension": cua_driver_core::tool_schema::desktop_max_image_dimension_schema()},"additionalProperties":false}),read_only: true, destructive: false, idempotent: false, open_world: false,})@@ -8726,7 +12618,8 @@ impl Tool for GetDesktopStateTool {Err(result) => return result,};let out_file = input.screenshot_out_file;- let capture_service = self.capture_service.clone();+ let max_image_dimension = input.max_image_dimension;+ let capture_service = self.state.capture_service.clone();let result = tokio::task::spawn_blocking(move || -> anyhow::Result<_> {// Capture the full display at native size first. When the@@ -8750,6 +12643,17 @@ impl Tool for GetDesktopStateTool {};let (png, shot_w, shot_h, scale_factor) =normalize_desktop_capture_for_action_frame(native_png, screen_w, screen_h)?;+ // Opt-in long-edge cap: the full-size capture is the desktop+ // action frame; a capped image is mapped back at dispatch+ // (cua_driver_core::desktop_capture_scale) and by its capture_id.+ let (png, shot_w, shot_h) = match max_image_dimension.filter(|cap| *cap > 0) {+ Some(cap) if shot_w.max(shot_h) > cap => {+ let png = crate::capture::resize_png_if_needed(&png, cap)?;+ let (w, h) = crate::capture::png_dimensions_pub(&png)?;+ (png, w, h)+ }+ _ => (png, shot_w, shot_h),+ };// Optional: write PNG to disk instead of returning base64.let written = if let Some(path) = out_file.as_deref() {std::fs::write(path, &png)?;@@ -8763,11 +12667,17 @@ impl Tool for GetDesktopStateTool {} else {Some(B64.encode(&png))};+ // Which pid owns which window is what the agent needs before its+ // first click; without it the first action guesses `pid: 1`.+ let mut windows = crate::wayland::list_windows_dispatch(None);+ windows+ .retain(|w| w.is_on_screen && w.pid.map_or(false, crate::proc_fs::is_process_live));let capture_id = crate::capture_action_frame::publish_desktop(&capture_service,&capture_args,&png,(shot_w, shot_h),+ (screen_w, screen_h),)?;Ok((b64,@@ -8777,6 +12687,7 @@ impl Tool for GetDesktopStateTool {screen_h,scale_factor,written,+ windows,capture_id,))})@@ -8791,8 +12702,14 @@ impl Tool for GetDesktopStateTool {screen_h,scale_factor,written,+ windows,capture_id,))) => {+ let frame_scale = if shot_w > 0 {+ f64::from(screen_w) / f64::from(shot_w)+ } else {+ 1.0+ };let mut content = Vec::new();let mut structured = json!({"platform": "linux",@@ -8802,20 +12719,36 @@ impl Tool for GetDesktopStateTool {"screen_width": screen_w,"screen_height": screen_h,"scale_factor": scale_factor,+ "frame_scale": frame_scale,"screenshot_mime_type": "image/png",+ "windows": windows.iter().map(window_record_json).collect::<Vec<_>>(),"capture_id": capture_id,});+ if (frame_scale - 1.0).abs() > 0.001 {+ // Capped: the uncapped capture is the action frame.+ structured["screenshot_original_width"] = json!(screen_w);+ structured["screenshot_original_height"] = json!(screen_h);+ }if let Some(b64) = b64_opt {content.push(cua_driver_core::protocol::Content::image_png(b64));}+ let window_lines = desktop_window_lines(&windows, frame_scale);+ let frame_note = if (frame_scale - 1.0).abs() > 0.001 {+ format!(+ "; x/y for scope:\"desktop\" actions are pixels of THIS screenshot \+ (mapped ×{frame_scale:.2} back to the screen automatically)"+ )+ } else {+ String::new()+ };if let Some(path) = written {structured["screenshot_file_path"] = json!(path);content.push(cua_driver_core::protocol::Content::text(format!(- "✅ Desktop screenshot {shot_w}x{shot_h} written to {path} (screen {screen_w}x{screen_h})"+ "✅ Desktop screenshot {shot_w}x{shot_h} written to {path} (screen {screen_w}x{screen_h}{frame_note}){window_lines}")));} else {content.push(cua_driver_core::protocol::Content::text(format!(- "✅ Desktop screenshot {shot_w}x{shot_h} (screen {screen_w}x{screen_h})"+ "✅ Desktop screenshot {shot_w}x{shot_h} (screen {screen_w}x{screen_h}{frame_note}){window_lines}")));}ToolResult {@@ -8831,6 +12764,43 @@ impl Tool for GetDesktopStateTool {}}+/// Text block naming every visible window with the `pid` / `window_id` pair+/// that every other tool takes, so the model's first action targets a real+/// process instead of guessing.+fn desktop_window_lines(windows: &[crate::x11::WindowInfo], frame_scale: f64) -> String {+ let px = |v: i32| (f64::from(v) / frame_scale).round() as i32;+ let sz = |v: u32| (f64::from(v) / frame_scale).round() as u32;+ if windows.is_empty() {+ return "\nVisible windows: none (use list_windows / launch_app).".to_owned();+ }+ let mut out =+ String::from("\nVisible windows (use these pid + window_id values in every tool call):");+ for w in windows {+ let title = if w.title.is_empty() {+ "(no title)".to_owned()+ } else {+ format!("\"{}\"", w.title)+ };+ out.push_str(&format!(+ "\n- pid={} window_id={} {} {}x{} at ({},{}){}",+ w.pid.map(|p| p.to_string()).unwrap_or_else(|| "?".into()),+ w.xid,+ title,+ sz(w.width),+ sz(w.height),+ px(w.x),+ px(w.y),+ if w.app_name.is_empty() {+ String::new()+ } else {+ format!(" app={}", w.app_name)+ }+ ));+ }+ out.push_str("\n→ get_window_state(pid, window_id) lists clickable elements; click/type_text take the same pid (+ window_id for x,y).");+ out+}+// ── get_cursor_position ───────────────────────────────────────────────────────pub struct GetCursorPositionTool;@@ -9528,8 +13498,14 @@ impl Tool for GetAccessibilityTreeTool {format!("\"{}\"", w.title)};lines.push(format!(- "- pid={:?} {} [window_id: {}] {}x{}+{}+{}",- w.pid, title, w.xid, w.width, w.height, w.x, w.y+ "- pid={} window_id={} {} {}x{}+{}+{}",+ w.pid.map(|p| p.to_string()).unwrap_or_else(|| "?".into()),+ w.xid,+ title,+ w.width,+ w.height,+ w.x,+ w.y));}lines.push(@@ -9710,7 +13686,8 @@ impl Tool for TypeTextCharsTool {async fn invoke(&self, args: Value) -> ToolResult {use cua_driver_core::tool_args::ArgsExt;- let pid = args.u64_or("pid", 0) as u32;+ #[allow(unused_assignments)]+ let mut pid = args.u64_or("pid", 0) as u32;let text_raw = match args.require_str("text") {Ok(v) => v,Err(e) => return e,@@ -9724,12 +13701,43 @@ impl Tool for TypeTextCharsTool {let xid = match xid_opt {Some(x) => x,None => {- let windows =- tokio::task::spawn_blocking(move || crate::x11::list_windows(Some(pid)))- .await- .unwrap_or_default();- match windows.first() {- Some(w) => w.xid,+ let windows = tokio::task::spawn_blocking(move || {+ crate::x11::list_windows(if pid == 0 { None } else { Some(pid) })+ })+ .await+ .unwrap_or_default();+ // pid omitted: the keys go to the active window, like a+ // physical keyboard would, and the action adopts its pid.+ let chosen = if pid == 0 {+ let active = crate::x11::active_window();+ windows+ .iter()+ .find(|w| Some(w.xid) == active && w.pid.is_some())+ .or_else(|| windows.iter().find(|w| w.is_on_screen && w.pid.is_some()))+ } else {+ // The pid's active window, else its largest on-screen+ // toplevel: LibreOffice and GIMP own hidden/utility+ // toplevels that a plain `first()` could pick, and an+ // unmapped window cannot take the virtual keyboard focus.+ let active = crate::x11::active_window();+ windows+ .iter()+ .find(|w| Some(w.xid) == active)+ .or_else(|| {+ windows+ .iter()+ .filter(|w| w.is_on_screen)+ .max_by_key(|w| u64::from(w.width) * u64::from(w.height))+ })+ .or_else(|| windows.first())+ };+ match chosen {+ Some(w) => {+ if pid == 0 {+ pid = w.pid.unwrap_or(0);+ }+ w.xid+ }None => {return ToolResult::error(format!("No windows found for pid {pid}. Provide window_id."@@ -9920,6 +13928,16 @@ impl Tool for InvokeMenuTool {EvidenceKind, RequestedDelivery,};+ // Menu paths are resolved and fired through AT-SPI, which needs no+ // window activation on X11: the default is background (focus-free,+ // under the focus guard). `delivery_mode:"foreground"` is the explicit+ // escalation that activates the window first, as every other tool.+ let delivery = crate::input::delivery::DeliveryMode::from_args(&args);+ let mut args = args;+ if let Some(object) = args.as_object_mut() {+ // Not part of the closed InvokeMenuInput contract; consumed above.+ object.remove("delivery_mode");+ }let input: InvokeMenuInput = match parse_typed_input("invoke_menu", args) {Ok(input) => input,Err(result) => return result,@@ -9937,6 +13955,46 @@ impl Tool for InvokeMenuTool {return menu_refusal("invoke_menu: window_id does not belong to pid".into());}+ if !crate::wayland::is_wayland() && !delivery.is_foreground() {+ let outcome = tokio::task::spawn_blocking(move || {+ crate::input::focus_guard::guarded(Some(pid), || {+ crate::atspi::native::invoke_menu_path_in(pid, window_id, &path)+ })+ })+ .await;+ return match outcome {+ Ok(Ok(((), guard))) => attach_focus_guard(+ ToolResult::text(+ "Resolved the live native menu path and dispatched its final accessibility action (delivery_mode=background, focus untouched); verify the command's semantic effect from fresh state.",+ )+ .with_structured(json!({+ "path": "ax",+ "verified": false,+ "effect": "unverifiable",+ "delivery_mode": "background",+ }))+ .with_action_record(+ ActionExecutionRecord::builder(+ ActionEffect::Unverifiable,+ ActionTransport::LinuxAtSpiAction,+ RequestedDelivery::Background,+ )+ .actual_delivery(ActualDelivery::Background)+ .evidence(ActionEvidence {+ kind: EvidenceKind::NativeApiResult,+ detail: "Every menu hop resolved uniquely and AT-SPI accepted the final action"+ .into(),+ })+ .build()+ .expect("invoke_menu record is valid"),+ ),+ guard.as_ref(),+ ),+ Ok(Err(error)) => menu_refusal(format!("invoke_menu: {error}")),+ Err(error) => menu_refusal(format!("invoke_menu: blocking task failed: {error}")),+ };+ }+let activation = if crate::wayland::is_wayland() {let result = tokio::task::spawn_blocking(move || {crate::wayland::activate_window_for_input_target(window_id, Some(pid))@@ -9972,7 +14030,7 @@ impl Tool for InvokeMenuTool {};let outcome = tokio::task::spawn_blocking(move || {- let result = crate::atspi::native::invoke_menu_path(pid, &path);+ let result = crate::atspi::native::invoke_menu_path_in(pid, window_id, &path);if let Some(Some(prior_window)) = activation {let _ = crate::input::x11_activate_window_persistent(prior_window);}@@ -9982,8 +14040,14 @@ impl Tool for InvokeMenuTool {match outcome {Ok(Ok(())) => ToolResult::text(- "Resolved the live native menu path and dispatched its final accessibility action; verify the command's semantic effect from fresh state.",+ "Resolved the live native menu path and dispatched its final accessibility action (delivery_mode=foreground); verify the command's semantic effect from fresh state.",)+ .with_structured(json!({+ "path": "ax",+ "verified": false,+ "effect": "unverifiable",+ "delivery_mode": "foreground",+ })).with_action_record(ActionExecutionRecord::builder(ActionEffect::Unverifiable,@@ -10407,7 +14471,12 @@ pub fn build_registry_with_provider());r.register(Box::new(LaunchAppTool));r.register(Box::new(KillAppTool));- let pid_window_candidates: WindowTargetCandidates = Arc::new(pid_window_target_candidates);+ let pid_window_candidates: PidWindowGuardParts = (+ Arc::new(pid_window_target_candidates),+ desktop_point_window_resolver(),+ pid_fallback_window_resolver(),+ snapshot_window_resolver(state.clone()),+ );r.register(pid_window_guarded(BringToFrontTool, &pid_window_candidates));r.register(Box::new(SetWindowFrameTool));r.register(Box::new(InvokeMenuTool));@@ -10496,7 +14565,7 @@ pub fn build_registry_with_provider(let _ = compat;r.register(Box::new(GetScreenSizeTool));r.register(Box::new(GetDesktopStateTool {- capture_service: state.capture_service.clone(),+ state: state.clone(),}));r.register(Box::new(GetCursorPositionTool));r.register(Box::new(MoveCursorTool {@@ -10801,3 +14870,273 @@ mod desktop_capture_frame_tests {assert_eq!(structured["effect"], "refused");}}++#[cfg(test)]+mod background_budget_tests {+ use super::*;++ #[test]+ fn synthetic_pointer_drop_detection_covers_gtk_vcl_and_qt() {+ assert!(maps_indicate_synthetic_pointer_dropped(+ "7f /usr/lib/libgtk-3.so.0\n"+ ));+ assert!(maps_indicate_synthetic_pointer_dropped(+ "7f /usr/lib/libgtk-4.so.1\n"+ ));+ assert!(maps_indicate_synthetic_pointer_dropped(+ "7f /opt/libreoffice/program/libmergedlo.so\n"+ ));+ assert!(maps_indicate_synthetic_pointer_dropped(+ "7f /usr/lib/libvcllo.so\n"+ ));+ assert!(maps_indicate_synthetic_pointer_dropped(+ "7f /usr/lib/libQt5Gui.so.5\n"+ ));+ assert!(!maps_indicate_synthetic_pointer_dropped(+ "7f /usr/lib/libgtk-x11-2.0.so.0\n"+ ));+ assert!(!maps_indicate_synthetic_pointer_dropped(+ "7f /usr/lib/libX11.so.6\n"+ ));+ }++ #[test]+ fn desktop_window_lines_expose_pid_and_window_id() {+ let windows = vec![crate::x11::WindowInfo {+ xid: 0x2e00003,+ pid: Some(4321),+ app_name: "libreoffice".into(),+ title: "Untitled 1 - LibreOffice Calc".into(),+ is_on_screen: true,+ z_index: None,+ x: 0,+ y: 27,+ width: 1920,+ height: 1053,+ }];+ let text = desktop_window_lines(&windows, 1.0);+ assert!(text.contains("pid=4321 window_id=48234499"));+ assert!(text.contains("LibreOffice Calc"));+ assert!(text.contains("app=libreoffice"));+ assert!(text.contains("get_window_state(pid, window_id)"));+ assert!(desktop_window_lines(&[], 1.0).contains("none"));+ // A downsized desktop screenshot reports window rects in ITS pixels.+ let scaled = desktop_window_lines(&windows, 1.5);+ assert!(scaled.contains("1280x702 at (0,18)"), "{scaled}");+ }+}++#[cfg(test)]+mod background_keyboard_route_tests {+ use super::{element_needs_real_click, split_key_combo, wm_chord_kind, WmChord};++ #[test]+ fn key_combos_split_into_modifiers_and_key() {+ assert_eq!(+ split_key_combo("alt+F4"),+ (vec!["alt".to_owned()], "F4".to_owned())+ );+ assert_eq!(+ split_key_combo("ctrl+shift+t"),+ (vec!["ctrl".to_owned(), "shift".to_owned()], "t".to_owned())+ );+ assert_eq!(split_key_combo("+"), (vec![], "+".to_owned()));+ assert_eq!(+ split_key_combo("ctrl++"),+ (vec!["ctrl".to_owned()], "+".to_owned())+ );+ assert_eq!(split_key_combo("Return"), (vec![], "Return".to_owned()));+ // Not a modifier prefix: left untouched for the keysym resolver.+ assert_eq!(split_key_combo("a+b"), (vec![], "a+b".to_owned()));+ }++ #[test]+ fn wm_chords_are_recognised() {+ let m = |xs: &[&str]| xs.iter().map(|x| x.to_string()).collect::<Vec<_>>();+ assert_eq!(+ wm_chord_kind("F4", &m(&["alt"])),+ Some(WmChord::CloseWindow)+ );+ assert_eq!(+ wm_chord_kind("f4", &m(&["Alt_L"])),+ Some(WmChord::CloseWindow)+ );+ assert_eq!(+ wm_chord_kind("Tab", &m(&["alt"])),+ Some(WmChord::Unavailable)+ );+ assert_eq!(+ wm_chord_kind("Tab", &m(&["alt", "shift"])),+ Some(WmChord::Unavailable)+ );+ assert_eq!(+ wm_chord_kind("a", &m(&["super"])),+ Some(WmChord::Unavailable)+ );+ assert_eq!(+ wm_chord_kind("t", &m(&["ctrl", "alt"])),+ Some(WmChord::Unavailable)+ );+ assert_eq!(+ wm_chord_kind("F4", &m(&["ctrl", "alt"])),+ Some(WmChord::Unavailable)+ );+ // Application chords stay with the application.+ assert_eq!(wm_chord_kind("c", &m(&["ctrl"])), None);+ assert_eq!(wm_chord_kind("F4", &m(&["ctrl"])), None);+ assert_eq!(wm_chord_kind("f", &m(&["alt"])), None);+ assert_eq!(wm_chord_kind("F4", &m(&[])), None);+ assert_eq!(wm_chord_kind("s", &m(&["ctrl", "shift"])), None);+ }++ #[test]+ fn focus_taking_roles_need_a_real_click() {+ for role in ["spin button", "Text", "entry", "slider", "combo box"] {+ assert!(element_needs_real_click(role), "{role}");+ }+ for role in ["menu item", "push button", "menu", "check box", "page tab"] {+ assert!(!element_needs_real_click(role), "{role}");+ }+ }+}++#[cfg(test)]+mod visibility_tests {+ use super::*;++ #[test]+ fn framed_elements_lead_and_keep_their_relative_order() {+ let elements = vec![+ json!({"element_index": 0, "role": "menu"}),+ json!({"element_index": 1, "role": "push button", "frame": {"x": 1, "y": 1, "w": 2, "h": 2}}),+ json!({"element_index": 2, "role": "menu"}),+ json!({"element_index": 3, "role": "paragraph", "frame": {"x": 1, "y": 9, "w": 2, "h": 2}}),+ ];+ let ordered: Vec<u64> = framed_elements_first(elements)+ .iter()+ .map(|e| e["element_index"].as_u64().unwrap())+ .collect();+ assert_eq!(ordered, vec![1, 3, 0, 2]);+ }++ #[test]+ fn a_frameless_popup_returns_only_the_menu_entries_drawn_inside_it() {+ let elements = vec![+ json!({"element_index": 1, "role": "push button", "frame": {"x": 5, "y": 5, "w": 20, "h": 20}}),+ json!({"element_index": 2, "role": "menu item", "label": "Paragraph...", "frame": {"x": 2, "y": 30, "w": 180, "h": 22}}),+ json!({"element_index": 3, "role": "menu item", "label": "elsewhere", "frame": {"x": 400, "y": 30, "w": 180, "h": 22}}),+ json!({"element_index": 4, "role": "menu", "label": "Format"}),+ ];+ let kept: Vec<u64> = popup_menu_elements(elements.clone(), (0, 0), 200, 400)+ .iter()+ .map(|e| e["element_index"].as_u64().unwrap())+ .collect();+ assert_eq!(kept, vec![2]);+ assert_eq!(+ popup_menu_elements(elements[..1].to_vec(), (0, 0), 200, 400).len(),+ 1+ );+ }++ #[test]+ fn a_combo_popup_returns_the_rows_of_the_list_that_fills_it() {+ // Qt: the combo list lives under the combo box inside the dialog; the+ // main window's menubar entries are elsewhere on the screen.+ let elements = vec![+ json!({"element_index": 1, "role": "menu", "label": "Audio", "frame": {"x": -600, "y": -300, "w": 50, "h": 20}}),+ json!({"element_index": 7, "role": "list", "frame": {"x": 1, "y": 1, "w": 198, "h": 118}}),+ json!({"element_index": 8, "role": "list item", "label": "Desktop", "parent_index": 7, "frame": {"x": 2, "y": 2, "w": 190, "h": 22}}),+ json!({"element_index": 9, "role": "list item", "label": "user", "parent_index": 7, "frame": {"x": 2, "y": 26, "w": 190, "h": 22}}),+ json!({"element_index": 12, "role": "push button", "label": "Open", "frame": {"x": 20, "y": 20, "w": 60, "h": 20}}),+ ];+ let kept: Vec<u64> = popup_menu_elements(elements, (0, 0), 200, 120)+ .iter()+ .map(|e| e["element_index"].as_u64().unwrap())+ .collect();+ assert_eq!(kept, vec![8, 9]);+ }++ #[test]+ fn a_list_popup_without_an_indexed_container_keeps_its_item_rows_only() {+ let elements = vec![+ json!({"element_index": 1, "role": "menu", "label": "Audio", "frame": {"x": 1, "y": 1, "w": 50, "h": 20}}),+ json!({"element_index": 8, "role": "list item", "label": "Desktop", "frame": {"x": 2, "y": 2, "w": 190, "h": 22}}),+ json!({"element_index": 9, "role": "tree item", "label": "user", "frame": {"x": 2, "y": 26, "w": 190, "h": 22}}),+ json!({"element_index": 12, "role": "push button", "label": "Open", "frame": {"x": 20, "y": 20, "w": 60, "h": 20}}),+ ];+ let kept: Vec<u64> = popup_menu_elements(elements, (0, 0), 200, 120)+ .iter()+ .map(|e| e["element_index"].as_u64().unwrap())+ .collect();+ assert_eq!(kept, vec![8, 9]);+ }++ #[test]+ fn menu_roles_take_the_real_press() {+ for role in ["menu", "Menu Item", "check menu item", "radio menu item"] {+ assert!(element_is_menu_role(role), "{role}");+ }+ for role in ["menu bar", "push button", "text"] {+ assert!(!element_is_menu_role(role), "{role}");+ }+ }++ #[test]+ fn overlays_name_the_follow_up_call_per_dialog_and_popup() {+ let overlays = WindowOverlays {+ dialogs: vec![json!({+ "window_id": 71, "title": "Position and Size", "transient_for": 3, "modal": true,+ "bounds": {"x": 100, "y": 100, "width": 400, "height": 300}+ })],+ popups: vec![json!({+ "window_id": 72, "title": "", "pid": 5,+ "bounds": {"x": 10, "y": 30, "width": 200, "height": 400}+ })],+ covers_window: true,+ window_rect: Some((0, 0, 800, 600)),+ };+ let note = overlays.follow_up(5).unwrap();+ assert!(+ note.contains(+ "dialog \"Position and Size\" (window_id 71, transient of window 3, modal)"+ ),+ "{note}"+ );+ assert!(+ note.contains("get_window_state(pid=5, window_id=71)"),+ "{note}"+ );+ assert!(+ note.contains("popup (window_id 72, bounds x=10 y=30 200x400)"),+ "{note}"+ );+ assert!(rects_intersect((0, 0, 800, 600), (100, 100, 400, 300)));+ assert!(!rects_intersect((0, 0, 800, 600), (800, 0, 10, 10)));+ }++ #[test]+ fn a_cross_process_dialog_names_its_real_owning_pid_in_the_follow_up() {+ // GIMP's "Export Image as JPEG" dialog spawned as a separate process+ // (owning_pid 999) but is transient-for the target pid (5)'s window.+ let overlays = WindowOverlays {+ dialogs: vec![json!({+ "window_id": 90, "title": "Export Image as JPEG", "transient_for": 10,+ "modal": false,+ "bounds": {"x": 0, "y": 0, "width": 300, "height": 200},+ "owning_pid": 999,+ })],+ popups: vec![],+ covers_window: false,+ window_rect: Some((0, 0, 800, 600)),+ };+ let note = overlays.follow_up(5).unwrap();+ assert!(+ note.contains("owned by a different process pid 999, not pid 5"),+ "{note}"+ );+ assert!(+ note.contains("get_window_state(pid=5, window_id=90)"),+ "{note}"+ );+ }+}diff --git a/libs/cua-driver/rust/crates/platform-linux/src/x11/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/x11/mod.rsindex 11f3b83588..c94c24ba48 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/x11/mod.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/x11/mod.rs@@ -23,6 +23,29 @@ pub struct WindowInfo {pub height: u32,}+/// The window the window manager reports as active (`_NET_ACTIVE_WINDOW`+/// on the root), if any.+pub fn active_window() -> Option<u64> {+ let (conn, screen_num) = RustConnection::connect(None).ok()?;+ let root = conn.setup().roots[screen_num].root;+ let atom = conn+ .intern_atom(true, b"_NET_ACTIVE_WINDOW")+ .ok()?+ .reply()+ .ok()?+ .atom;+ if atom == 0 {+ return None;+ }+ let reply = conn+ .get_property(false, root, atom, AtomEnum::WINDOW, 0, 1)+ .ok()?+ .reply()+ .ok()?;+ let id = reply.value32()?.next()?;+ (id != 0).then_some(u64::from(id))+}+/// List top-level windows, optionally filtered by pid.pub fn list_windows(filter_pid: Option<u32>) -> Vec<WindowInfo> {match list_windows_inner(filter_pid) {@@ -31,6 +54,20 @@ pub fn list_windows(filter_pid: Option<u32>) -> Vec<WindowInfo> {}}+/// Whether the X server still knows a window by this id.+pub fn window_exists(xid: u64) -> bool {+ let Ok(xid) = u32::try_from(xid) else {+ return false;+ };+ let Ok((conn, _)) = RustConnection::connect(None) else {+ return false;+ };+ conn.get_window_attributes(xid)+ .ok()+ .and_then(|cookie| cookie.reply().ok())+ .is_some()+}+/// Verify that an X11 window still exists and belongs to the requested process.////// Checking `/proc/<pid>` alone is insufficient because Linux may recycle the@@ -241,6 +278,245 @@ fn moveresize_window_flags() -> u32 {STATIC_GRAVITY | X_PRESENT | Y_PRESENT | WIDTH_PRESENT | HEIGHT_PRESENT}+/// The listed toplevel that holds the core keyboard focus (`XGetInputFocus`),+/// if any: the focus usually sits on a child of the client window, so the+/// ancestors are walked until one of `candidates` is met.+pub fn focused_window_among(candidates: &[u64]) -> Option<u64> {+ let (conn, screen_num) = RustConnection::connect(None).ok()?;+ let root = conn.setup().roots[screen_num].root;+ let focus = conn.get_input_focus().ok()?.reply().ok()?.focus;+ if focus <= 1 {+ return None;+ }+ let mut current = focus;+ for _ in 0..32 {+ if candidates.contains(&u64::from(current)) {+ return Some(u64::from(current));+ }+ let tree = conn.query_tree(current).ok()?.reply().ok()?;+ if tree.parent == 0 || tree.parent == root {+ return None;+ }+ current = tree.parent;+ }+ None+}++/// `WM_TRANSIENT_FOR` of a toplevel: the window it is a dialog of. `None`+/// when unset or pointing at the root (group-transient utility windows).+pub fn transient_for(xid: u64) -> Option<u64> {+ let xid = u32::try_from(xid).ok()?;+ let (conn, screen_num) = RustConnection::connect(None).ok()?;+ let root = conn.setup().roots[screen_num].root;+ let reply = conn+ .get_property(+ false,+ xid,+ AtomEnum::WM_TRANSIENT_FOR,+ AtomEnum::WINDOW,+ 0,+ 1,+ )+ .ok()?+ .reply()+ .ok()?;+ let owner = reply.value32()?.next()?;+ (owner != 0 && owner != root).then_some(u64::from(owner))+}++/// Windows NOT owned by `pid` whose `WM_TRANSIENT_FOR` chain (bounded to+/// avoid cycles) resolves to one of `pid`'s own top-level windows.+///+/// A dialog or plugin window can legitimately run as a *different* process+/// than the application it belongs to — e.g. GIMP's separate-process export+/// option dialogs, or LibreOffice's "Document Recovery" dialog surfacing+/// under a distinct `soffice.bin` instance. Exact-pid matching alone makes+/// `get_window_state`'s `dialogs[]` invisible to such windows even though+/// they are clearly the target application's own popup.+///+/// `WM_TRANSIENT_FOR` pointing (directly or transitively) at a window this+/// pid actually owns is used as the sole correlation signal because it is an+/// explicit, spoofing-resistant relationship the window manager enforces —+/// unlike `WM_CLASS`/process-name similarity, which many unrelated+/// applications built on the same toolkit share and which a hostile window+/// could set to anything.+pub fn list_cross_pid_transient_windows(pid: u32) -> Vec<WindowInfo> {+ list_cross_pid_transient_windows_with(pid, list_windows(None), transient_for)+}++fn list_cross_pid_transient_windows_with(+ pid: u32,+ all_windows: Vec<WindowInfo>,+ transient_for: impl Fn(u64) -> Option<u64>,+) -> Vec<WindowInfo> {+ let own_xids: std::collections::HashSet<u64> = all_windows+ .iter()+ .filter(|w| w.pid == Some(pid))+ .map(|w| w.xid)+ .collect();+ if own_xids.is_empty() {+ return Vec::new();+ }+ all_windows+ .into_iter()+ .filter(|w| w.pid != Some(pid))+ .filter(|w| transient_chain_reaches(w.xid, &own_xids, &transient_for))+ .collect()+}++/// Walks `WM_TRANSIENT_FOR` from `start`, bounded to guard against a cycle a+/// misbehaving client could create, until it lands on a member of `targets`.+fn transient_chain_reaches(+ start: u64,+ targets: &std::collections::HashSet<u64>,+ transient_for: &impl Fn(u64) -> Option<u64>,+) -> bool {+ let mut current = start;+ for _ in 0..8 {+ match transient_for(current) {+ Some(owner) if targets.contains(&owner) => return true,+ Some(owner) if owner == current => return false,+ Some(owner) => current = owner,+ None => return false,+ }+ }+ false+}++/// The window a pid-only keyboard action means in a multi-window app, in+/// order: the pid's window holding the core focus; its topmost on-screen+/// transient dialog (a file chooser, a filter dialog); the WM's active+/// window when it is the pid's; the largest mapped toplevel. `transient_for`+/// answers `WM_TRANSIENT_FOR` for a window id.+pub fn pick_pid_window(+ windows: &[WindowInfo],+ focused: Option<u64>,+ transient_for: impl Fn(u64) -> Option<u64>,+ active: Option<u64>,+) -> Option<u64> {+ if let Some(focused) = focused.filter(|f| windows.iter().any(|w| w.xid == *f)) {+ return Some(focused);+ }+ let on_screen: Vec<&WindowInfo> = windows.iter().filter(|w| w.is_on_screen).collect();+ if let Some(dialog) = on_screen+ .iter()+ .filter(|w| w.width > 0 && w.height > 0 && transient_for(w.xid).is_some())+ .max_by_key(|w| w.z_index.unwrap_or(0))+ {+ return Some(dialog.xid);+ }+ if let Some(active) = active.filter(|a| windows.iter().any(|w| w.xid == *a)) {+ return Some(active);+ }+ on_screen+ .iter()+ .max_by_key(|w| {+ (+ u64::from(w.width) * u64::from(w.height),+ w.z_index.unwrap_or(0),+ )+ })+ .map(|w| w.xid)+}++/// Geometry, title and owner of ANY mapped X window (an override-redirect+/// popup menu included), unlike [`list_windows`], which only enumerates the+/// WM's client list. `None` when the window does not exist.+pub fn window_info(xid: u64) -> Option<WindowInfo> {+ let window = u32::try_from(xid).ok()?;+ let (conn, screen_num) = RustConnection::connect(None).ok()?;+ let root = conn.setup().roots[screen_num].root;+ let attributes = conn.get_window_attributes(window).ok()?.reply().ok()?;+ let geom = conn.get_geometry(window).ok()?.reply().ok()?;+ let trans = conn+ .translate_coordinates(window, root, 0, 0)+ .ok()?+ .reply()+ .ok()?;+ let pid = get_window_pid(&conn, window).ok().flatten();+ let title = get_window_title(&conn, window).unwrap_or_default();+ let app_name = get_window_class(&conn, window)+ .map(|(instance, class)| if class.is_empty() { instance } else { class })+ .unwrap_or_default();+ Some(WindowInfo {+ xid,+ pid,+ app_name,+ title,+ is_on_screen: attributes.map_state == MapState::VIEWABLE,+ z_index: None,+ x: i32::from(trans.dst_x),+ y: i32::from(trans.dst_y),+ width: u32::from(geom.width),+ height: u32::from(geom.height),+ })+}++/// Ask the window manager to close `xid` (EWMH `_NET_CLOSE_WINDOW`, which+/// the WM turns into `WM_DELETE_WINDOW` for a cooperating client): what+/// Alt+F4 does through mutter's passive grab, which a virtual keyboard cannot+/// reach. Only a window of `pid` is accepted.+pub fn close_window(xid: u64, pid: u32) -> Result<()> {+ let window =+ u32::try_from(xid).map_err(|_| anyhow::anyhow!("window_id is out of X11 range"))?;+ let (conn, screen_num) = RustConnection::connect(None)?;+ let root = conn.setup().roots[screen_num].root;+ match get_window_pid(&conn, window)? {+ Some(owner) if owner == pid => {}+ Some(owner) => anyhow::bail!("window_id {xid} belongs to pid {owner}, not pid {pid}"),+ None => anyhow::bail!("window_id {xid} has no verifiable _NET_WM_PID owner"),+ }+ let atom = get_atom(&conn, "_NET_CLOSE_WINDOW")?;+ let event = ClientMessageEvent::new(+ 32,+ window,+ atom,+ ClientMessageData::from([0u32, 1u32, 0, 0, 0]),+ );+ conn.send_event(+ false,+ root,+ EventMask::SUBSTRUCTURE_REDIRECT | EventMask::SUBSTRUCTURE_NOTIFY,+ event,+ )?;+ conn.flush()?;+ Ok(())+}++/// `_NET_WM_STATE` carries `_NET_WM_STATE_MODAL`: the dialog blocks input to+/// the window it is transient for.+pub fn window_is_modal(xid: u64) -> bool {+ let Ok(xid) = u32::try_from(xid) else {+ return false;+ };+ let Ok((conn, _)) = RustConnection::connect(None) else {+ return false;+ };+ let (Ok(state_atom), Ok(modal_atom)) = (+ get_atom(&conn, "_NET_WM_STATE"),+ get_atom(&conn, "_NET_WM_STATE_MODAL"),+ ) else {+ return false;+ };+ conn.get_property(false, xid, state_atom, AtomEnum::ATOM, 0, 64)+ .ok()+ .and_then(|cookie| cookie.reply().ok())+ .and_then(|reply| reply.value32().map(|atoms| atoms.collect::<Vec<_>>()))+ .is_some_and(|atoms| atoms.contains(&modal_atom))+}++/// True while `xid` exists on the server and is viewable.+pub fn window_is_viewable(xid: u64) -> bool {+ window_info(xid).is_some_and(|w| w.is_on_screen)+}++/// `_NET_WM_PID` of a toplevel, when the window advertises one.+pub fn window_pid(xid: u64) -> Option<u32> {+ let xid = u32::try_from(xid).ok()?;+ let (conn, _) = RustConnection::connect(None).ok()?;+ get_window_pid(&conn, xid).ok().flatten()+}+fn get_window_pid(conn: &RustConnection, window: Window) -> Result<Option<u32>> {let atom = get_atom(conn, "_NET_WM_PID")?;let reply = conn@@ -285,7 +561,7 @@ pub fn wm_class_for_window(xid: u64) -> Option<(String, String)> {get_window_class(&conn, xid as u32)}-fn get_window_class(conn: &RustConnection, xid: Window) -> Option<(String, String)> {+pub(crate) fn get_window_class(conn: &RustConnection, xid: Window) -> Option<(String, String)> {let reply = conn.get_property(false,@@ -298,8 +574,10 @@ fn get_window_class(conn: &RustConnection, xid: Window) -> Option<(String, Strin.ok()?.reply().ok()?;+ // Two NUL-terminated strings, instance then class; either may be empty+ // (`\0Foo\0`), so the position decides which is which.let raw = reply.value;- let mut parts = raw.split(|&b| b == 0).filter(|s| !s.is_empty());+ let mut parts = raw.split(|&b| b == 0);let instance = parts.next().map(|s| String::from_utf8_lossy(s).into_owned())@@ -356,4 +634,84 @@ mod tests {assert_eq!(flags & 0x0f00, 0x0f00);assert_eq!(flags & !0x0fff, 0);}++ fn win(xid: u64, pid: Option<u32>, title: &str) -> WindowInfo {+ WindowInfo {+ xid,+ pid,+ app_name: String::new(),+ title: title.into(),+ is_on_screen: true,+ z_index: None,+ x: 0,+ y: 0,+ width: 400,+ height: 300,+ }+ }++ #[test]+ fn a_different_process_dialog_transient_to_the_target_pid_is_surfaced() {+ // GIMP-style separate-process export dialog: xid 90 belongs to pid 999+ // (not the target pid 7) but is WM_TRANSIENT_FOR the target's own+ // window (10).+ let windows = vec![+ win(10, Some(7), "GIMP"),+ win(90, Some(999), "Export Image as JPEG"),+ ];+ let cross = list_cross_pid_transient_windows_with(7, windows, |xid| match xid {+ 90 => Some(10),+ _ => None,+ });+ assert_eq!(cross.len(), 1);+ assert_eq!(cross[0].xid, 90);+ assert_eq!(cross[0].pid, Some(999));+ }++ #[test]+ fn a_transitive_transient_chain_through_another_cross_pid_window_still_resolves() {+ // xid 91 is transient-for xid 90, which is transient-for the target's+ // own window 10: both 90 and 91 should be attributed to pid 7.+ let windows = vec![+ win(10, Some(7), "LibreOffice"),+ win(90, Some(999), "Recovery helper"),+ win(91, Some(999), "Document Recovery"),+ ];+ let cross = list_cross_pid_transient_windows_with(7, windows, |xid| match xid {+ 90 => Some(10),+ 91 => Some(90),+ _ => None,+ });+ let mut xids: Vec<u64> = cross.iter().map(|w| w.xid).collect();+ xids.sort();+ assert_eq!(xids, vec![90, 91]);+ }++ #[test]+ fn an_unrelated_window_of_a_totally_different_app_is_not_attributed() {+ // No transient_for relationship at all to the target's windows: this+ // must never be surfaced, no matter how similar its title/class.+ let windows = vec![win(10, Some(7), "GIMP"), win(50, Some(555), "Firefox")];+ let cross = list_cross_pid_transient_windows_with(7, windows, |_| None);+ assert!(cross.is_empty());+ }++ #[test]+ fn a_transient_cycle_fails_closed_instead_of_looping_forever() {+ let windows = vec![win(10, Some(7), "target"), win(90, Some(999), "cyclic")];+ let cross = list_cross_pid_transient_windows_with(7, windows, |xid| match xid {+ 90 => Some(91),+ 91 => Some(90),+ _ => None,+ });+ assert!(cross.is_empty());+ }++ #[test]+ fn no_own_windows_for_the_pid_yields_nothing_to_correlate_against() {+ let windows = vec![win(90, Some(999), "orphan dialog")];+ let cross =+ list_cross_pid_transient_windows_with(7, windows, |xid| (xid == 90).then_some(10));+ assert!(cross.is_empty());+ }}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rsindex ee1f5d3a1d..ab41962bdb 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs@@ -14,6 +14,7 @@use super::bindings::*;use super::window_scope::{decide_window_scope, TopLevelCandidate, WindowScope};use core_foundation::base::{CFEqual, CFRelease, CFRetain, CFTypeRef};+use cua_driver_core::walk_budget::{WalkBudget, WalkOutcome};/// Default maximum depth for AX tree walks. Deep menus and complex web views/// can nest deeply; 25 covers realistic app chrome without exploding on@@ -135,8 +136,10 @@ fn is_addressable(actions_present: bool, value_settable: bool, enabled: Option<bpub struct TreeWalkResult {pub tree_markdown: String,pub nodes: Vec<AXNode>,- /// True when the walk was cut short by the MAX_ELEMENTS cap.+ /// True when the walk was cut short by its node or time budget.pub truncated: bool,+ /// Why and where the walk stopped (see [`cua_driver_core::walk_budget`]).+ pub walk: WalkOutcome,/// Whether the requested `window_id` actually resolved to an AX surface,/// and if not, why. `None` when no `window_id` was requested.///@@ -186,15 +189,30 @@ pub fn walk_tree_bounded(query: Option<&str>,max_elements: usize,max_depth: usize,+) -> TreeWalkResult {+ walk_tree_budgeted(+ pid,+ window_id,+ query,+ max_depth,+ WalkBudget::nodes_only(max_elements),+ )+}++/// [`walk_tree_bounded`] under a caller's [`WalkBudget`]: the walk also stops+/// when the budget's `timeout_ms` runs out, returning the partial tree. Each+/// AX call is bounded by the per-element messaging timeout, so the walk+/// overruns the deadline by at most one attribute read.+pub fn walk_tree_budgeted(+ pid: i32,+ window_id: Option<u32>,+ query: Option<&str>,+ max_depth: usize,+ mut budget: WalkBudget,) -> TreeWalkResult {let mut nodes: Vec<AXNode> = Vec::new();let mut lines: Vec<(usize, String)> = Vec::new(); // (depth, line)let mut index_counter = 0usize;- // Shared visited-node counter passed into walk_element to enforce the cap.- let mut visited_count = 0usize;- // Set to true only when walk_element actually stops early due to the cap —- // avoids a false-positive when the tree naturally ends on exactly the cap.- let mut truncated = false;let mut window_scope: Option<WindowScope> = None;unsafe {@@ -204,6 +222,7 @@ pub fn walk_tree_bounded(tree_markdown: String::new(),nodes,truncated: false,+ walk: budget.outcome(),// No application AX element at all, so a requested window// certainly did not resolve.window_scope: window_id.map(|_| WindowScope::AxUnresolved { ax_window_count: 0 }),@@ -297,9 +316,7 @@ pub fn walk_tree_bounded(&mut nodes,&mut lines,&mut index_counter,- &mut visited_count,- &mut truncated,- max_elements,+ &mut budget,max_depth,);}@@ -312,7 +329,7 @@ pub fn walk_tree_bounded(CFRelease(app_elem as CFTypeRef);}- let truncated_flag = truncated;+ let walk = budget.outcome();let raw_markdown = render_lines(&lines);let mut tree_markdown = if let Some(q) = query {filter_tree(&raw_markdown, q)@@ -320,19 +337,16 @@ pub fn walk_tree_bounded(raw_markdown};- if truncated_flag {- tree_markdown.push_str(&format!(- "\n⚠️ AX tree truncated at {max_elements} nodes \- (app has a very large accessibility tree — Arc, Electron, or similar). \- Element indices above are still valid. Use pixel clicks for elements \- not visible in this partial tree."- ));+ if let Some(note) = walk.note() {+ tree_markdown.push('\n');+ tree_markdown.push_str(¬e);}TreeWalkResult {tree_markdown,nodes,- truncated: truncated_flag,+ truncated: walk.truncated(),+ walk,window_scope,}}@@ -346,21 +360,17 @@ unsafe fn walk_element(nodes: &mut Vec<AXNode>,lines: &mut Vec<(usize, String)>,counter: &mut usize,- visited_count: &mut usize,- truncated: &mut bool,- max_elements: usize,+ budget: &mut WalkBudget,max_depth: usize,) {if depth > max_depth {return;}- // Enforce total-node cap — mirrors Swift's maxElements guard.- // Set the truncated flag only when we actually stop early.- if *visited_count >= max_elements {- *truncated = true;+ // Node and time budget: a refused node is counted as discovered but not+ // visited, and the walk unwinds with the partial tree it has.+ if !budget.admit() {return;}- *visited_count += 1;// Messaging timeouts are per AX object, not inherited from the application// element, so every descendant must be bounded before any attribute read.@@ -385,9 +395,7 @@ unsafe fn walk_element(nodes,lines,counter,- visited_count,- truncated,- max_elements,+ budget,max_depth,);CFRelease(child as CFTypeRef);@@ -453,9 +461,7 @@ unsafe fn walk_element(nodes,lines,counter,- visited_count,- truncated,- max_elements,+ budget,max_depth,);CFRelease(child as CFTypeRef);@@ -576,9 +582,7 @@ unsafe fn walk_element(nodes,lines,counter,- visited_count,- truncated,- max_elements,+ budget,max_depth,);CFRelease(child as CFTypeRef);diff --git a/libs/cua-driver/rust/crates/platform-macos/src/browser/setup_ui.rs b/libs/cua-driver/rust/crates/platform-macos/src/browser/setup_ui.rsindex 0ba2abd7ff..ea376a6f20 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/browser/setup_ui.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/browser/setup_ui.rs@@ -1747,6 +1747,7 @@ mod tests {tree_markdown: String::new(),nodes,truncated: false,+ walk: cua_driver_core::walk_budget::WalkBudget::nodes_only(0).outcome(),window_scope: Some(crate::ax::WindowScope::Matched),}}@@ -1875,6 +1876,10 @@ mod tests {tree_markdown: String::new(),nodes: Vec::new(),truncated: true,+ walk: cua_driver_core::walk_budget::WalkOutcome::timed_out(+ 1000,+ std::time::Duration::from_millis(1000),+ ),window_scope: Some(crate::ax::WindowScope::Matched),};assert!(diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/background_input_regression_tests.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/background_input_regression_tests.rsindex dc58908dbb..2b39f35333 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/background_input_regression_tests.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/background_input_regression_tests.rs@@ -114,6 +114,7 @@ impl Tool for KeyboardProbe {fn candidate(window_id: u64) -> WindowTargetCandidate {WindowTargetCandidate {window_id,+ transient_for: None,title: format!("Document {window_id}"),app_name: Some("Editor".into()),is_on_screen: true,diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_desktop_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_desktop_state.rsindex 40b0e8da85..99cbe2a3a5 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_desktop_state.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_desktop_state.rs@@ -37,16 +37,17 @@ static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();fn def() -> &'static ToolDef {DEF.get_or_init(|| ToolDef {name: "get_desktop_state".into(),- description: "Capture the full display in true screen pixels with no downscale. \- Use its native-size PNG as the coordinate source for actions whose target is \- {kind:\"desktop\",display_id:\"primary\"}. Returns the true screen size and \- backing scale factor. Vision-only: no AX tree walk."+ description: "Capture the full display in true screen pixels, full size unless \+ `max_image_dimension` caps it. Use its PNG as the coordinate source for actions \+ whose target is {kind:\"desktop\",display_id:\"primary\"}. Returns the true \+ screen size and backing scale factor. Vision-only: no AX tree walk.".into(),input_schema: serde_json::json!({"type": "object","properties": {"session": { "type": "string", "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session." },- "screenshot_out_file": { "type": "string", "description": "Write PNG here instead of base64." }+ "screenshot_out_file": { "type": "string", "description": "Write PNG here instead of base64." },+ "max_image_dimension": cua_driver_core::tool_schema::desktop_max_image_dimension_schema()},"additionalProperties": false}),@@ -69,6 +70,7 @@ impl Tool for GetDesktopStateTool {Ok(input) => input,Err(result) => return result,};+ let max_image_dimension = input.max_image_dimension.filter(|cap| *cap > 0);let screenshot_out_file = input.screenshot_out_file.map(|s| {// Expand ~ prefix (mirrors get_window_state).if let Some(relative) = s.strip_prefix("~/") {@@ -89,20 +91,29 @@ impl Tool for GetDesktopStateTool {// blocking screencapture subprocess off the async runtime.let out_file = screenshot_out_file.clone();let res = tokio::task::spawn_blocking(- move || -> anyhow::Result<(Vec<u8>, Option<String>, u32, u32)> {+ move || -> anyhow::Result<(Vec<u8>, Option<String>, u32, u32, (u32, u32))> {let png = crate::capture::screenshot_display_bytes()?;+ let full = crate::capture::png_dimensions(&png)?;+ // Opt-in cap; later desktop-scope pixels from the capped+ // image are mapped back at dispatch and by its capture_id.+ let png = match max_image_dimension {+ Some(cap) if full.0.max(full.1) > cap => {+ crate::capture::resize_png_if_needed(&png, cap)?+ }+ _ => png,+ };let (w, h) = crate::capture::png_dimensions(&png)?;if let Some(ref path) = out_file {std::fs::write(path, &png)?;- Ok((png, Some(path.clone()), w, h))+ Ok((png, Some(path.clone()), w, h, full))} else {- Ok((png, None, w, h))+ Ok((png, None, w, h, full))}},).await;- let (png, file_path, screenshot_width, screenshot_height) = match res {+ let (png, file_path, screenshot_width, screenshot_height, full_size) = match res {Ok(Ok(v)) => v,Ok(Err(e)) => return ToolResult::error(format!("Desktop screenshot failed: {e}")),Err(e) => return ToolResult::error(format!("Desktop screenshot task error: {e}")),@@ -148,6 +159,10 @@ impl Tool for GetDesktopStateTool {"screenshot_mime_type": "image/png","capture_id": capture_id,});+ if full_size != (screenshot_width, screenshot_height) {+ structured["screenshot_original_width"] = serde_json::json!(full_size.0);+ structured["screenshot_original_height"] = serde_json::json!(full_size.1);+ }if let Some(ref fp) = file_path {structured["screenshot_file_path"] = serde_json::json!(fp);}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rsindex 6208d50287..945efa9a65 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs@@ -20,6 +20,10 @@ impl GetWindowStateTool {static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();+/// Slack past `timeout_ms` before the walk task is abandoned: one in-flight AX+/// call may still be waiting on its messaging timeout.+const AX_WALK_BACKSTOP_GRACE: std::time::Duration = std::time::Duration::from_secs(5);+fn def() -> &'static ToolDef {DEF.get_or_init(|| ToolDef {name: "get_window_state".into(),@@ -49,7 +53,7 @@ fn def() -> &'static ToolDef {the tree only — the cheap path when you're just re-indexing before an \element ax action.\n\n\The mirror image: pass `include_accessibility_tree:false` to SKIP the \- AX walk entirely (the expensive part, up to 20 s) and return just the \+ AX walk entirely (the expensive part, bounded by timeout_ms) and return just the \screenshot plus window metadata — `window_bounds`, `screenshot_scale`, \`screenshot_width`/`screenshot_height`, `app_name`, and `window_title` \— the capture-only path for rendering a live window preview / \@@ -93,7 +97,7 @@ fn def() -> &'static ToolDef {"capture_mode": cua_driver_core::capture_mode::capture_mode_schema(),"include_accessibility_tree": {"type": "boolean",- "description": "Default true — walk the AX tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AX walk entirely (the expensive part, up to 20 s) and return just the screenshot plus window metadata (bounds, scale, app_name, window_title) — the capture-only path for rendering a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)."+ "description": "Default true — walk the AX tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AX walk entirely (the expensive part, bounded by timeout_ms) and return just the screenshot plus window metadata (bounds, scale, app_name, window_title) — the capture-only path for rendering a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)."},"include_screenshot": {"type": "boolean",@@ -113,6 +117,7 @@ fn def() -> &'static ToolDef {"minimum": 1,"description": "Cap on the AX-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower this for deep menu/Electron trees."},+ "timeout_ms": cua_driver_core::tool_schema::timeout_ms_schema(),"max_dimension": {"type": "integer","minimum": 1,@@ -198,7 +203,7 @@ impl Tool for GetWindowStateTool {};// Issue #2237: pre-flight the requested window against WindowServer- // BEFORE the (up to 20 s) AX walk. An id that no window carries, or+ // BEFORE the (timeout_ms-bounded) AX walk. An id that no window carries, or// that another process owns, used to fall through the scoped filter and// return the app's MENU BAR as a healthy snapshot of the requested// window — with a screenshot of the requested window beside it. macOS@@ -306,35 +311,36 @@ impl Tool for GetWindowStateTool {.and_then(|v| v.as_u64()).map(|v| v.max(1) as usize).unwrap_or(crate::ax::tree::DEFAULT_MAX_DEPTH);+ let timeout_ms = cua_driver_core::tool_schema::resolve_timeout_ms(args.get("timeout_ms"));let (tree_result, prepared_snapshot) = if want_tree {let q = query.clone();- // Keep the product deadline below the public client's 25-second- // deadline so callers receive a structured driver error. The AX- // walker also applies a native per-element messaging timeout because- // dropping a spawn_blocking JoinHandle cannot cancel a blocked AX call.+ // `timeout_ms` bounds the walk itself: it returns the partial tree+ // when the budget runs out. The outer deadline is only a backstop+ // for an AX call that ignores the per-element messaging timeout+ // (dropping a spawn_blocking JoinHandle cannot cancel it).let walk_future = tokio::task::spawn_blocking(move || {- let tree = crate::ax::tree::walk_tree_bounded(+ let tree = crate::ax::tree::walk_tree_budgeted(pid,Some(window_id),q.as_deref(),- max_elements,max_depth,+ cua_driver_core::walk_budget::WalkBudget::new(timeout_ms, max_elements),);let payload = crate::ax::cache::CachedSnapshot::from_nodes(&tree.nodes);(tree, payload)});- match tokio::time::timeout(std::time::Duration::from_secs(20), walk_future).await {+ let backstop = std::time::Duration::from_millis(timeout_ms) + AX_WALK_BACKSTOP_GRACE;+ match tokio::time::timeout(backstop, walk_future).await {Ok(Ok((tree, payload))) => (Some(tree), Some(payload)),Ok(Err(e)) => return ToolResult::error(format!("AX tree walk failed: {e}")),Err(_elapsed) => {return ToolResult::error(format!(- "AX tree walk for pid={pid} timed out after 20 s. \- The app (likely Arc, Electron, or Safari with many tabs) has a \- pathologically large accessibility tree. \- Workaround: re-call with a depth-limited scan \- (max_elements / max_depth), then act by pixel (x,y) off \- the screenshot if the tree stays unusable."+ "AX tree walk for pid={pid} did not return within {} s: an \+ accessibility call stopped answering past the {timeout_ms} ms \+ timeout_ms budget. Retry, or act by pixel (x,y) off a \+ screenshot-only get_window_state (include_accessibility_tree:false).",+ backstop.as_secs()));}}@@ -579,6 +585,18 @@ impl Tool for GetWindowStateTool {query.as_deref(),&tree_md,);+ // Screenshot pixels of the delivered capture: window origin in screen+ // points, delivered pixels per point (backing scale x downsizing).+ let elements_json = match (screenshot_frame.as_ref(), screenshot_dims) {+ (Some((bounds, _)), Some((width, _))) if bounds.width > 0.0 => {+ cua_driver_core::element_frame::with_screenshot_frames(+ elements_json,+ (bounds.x, bounds.y),+ f64::from(width) / bounds.width,+ )+ }+ _ => elements_json,+ };let filtered_element_count = elements_json.len();// The structured array intentionally contains only actionable nodes,// and AX child reads can fail independently of the element/depth caps.@@ -603,6 +621,9 @@ impl Tool for GetWindowStateTool {if query.is_some() {structured["filtered_element_count"] = serde_json::json!(filtered_element_count);}+ if let Some(r) = tree_result.as_ref() {+ r.walk.apply(&mut structured);+ }// Surface 6: an opaque snapshot identifier consumers can log// alongside the per-element tokens for debug correlation. Same value// embedded in every `element_token` emitted in `elements[]` above.diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rsindex 0c1231e577..e3fb405aae 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs@@ -85,6 +85,7 @@ fn window_target_candidates_for_pid(.filter(|window| window.pid == pid).map(|window| WindowTargetCandidate {window_id: u64::from(window.window_id),+ transient_for: None,title: window.title,app_name: Some(window.app_name),is_on_screen: window.is_on_screen,diff --git a/libs/cua-driver/rust/crates/platform-windows/src/msaa.rs b/libs/cua-driver/rust/crates/platform-windows/src/msaa.rsindex a1ed48da86..36339073be 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/msaa.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/msaa.rs@@ -25,6 +25,7 @@ use windows::Win32::Foundation::HWND;use windows::Win32::UI::Accessibility::{AccessibleObjectFromWindow, IAccessible};use crate::uia::{UiaNode, UiaTreeResult};+use cua_driver_core::walk_budget::WalkBudget;const OBJID_CLIENT: u32 = 0xFFFFFFFC;@@ -66,10 +67,15 @@ const MAX_TOTAL_ELEMENTS: usize = 5000;/// Walk the MSAA tree for the window with the given HWND. Used as fallback/// for SAL/VCL targets where the UIA walker would hang.pub fn walk_msaa_tree(hwnd: u64) -> UiaTreeResult {- unsafe { walk_unsafe(hwnd) }+ walk_msaa_tree_budgeted(hwnd, &mut WalkBudget::nodes_only(MAX_TOTAL_ELEMENTS))}-unsafe fn walk_unsafe(hwnd: u64) -> UiaTreeResult {+/// [`walk_msaa_tree`] under the caller's node and time budget.+pub fn walk_msaa_tree_budgeted(hwnd: u64, budget: &mut WalkBudget) -> UiaTreeResult {+ unsafe { walk_unsafe(hwnd, budget) }+}++unsafe fn walk_unsafe(hwnd: u64, budget: &mut WalkBudget) -> UiaTreeResult {let hwnd_win = HWND(hwnd as *mut _);let mut raw_root: *mut std::ffi::c_void = null_mut();// AccessibleObjectFromWindow returns IAccessible (as `*mut c_void` typed@@ -94,17 +100,8 @@ unsafe fn walk_unsafe(hwnd: u64) -> UiaTreeResult {let mut nodes: Vec<UiaNode> = Vec::new();let mut lines: Vec<(usize, String)> = Vec::new();let mut counter = 0usize;- let mut total = 0usize;- walk(- &root,- 0,- None,- &mut nodes,- &mut lines,- &mut counter,- &mut total,- );+ walk(&root, 0, None, &mut nodes, &mut lines, &mut counter, budget);let tree_markdown = render_lines(&lines);UiaTreeResult {@@ -121,12 +118,11 @@ unsafe fn walk(nodes: &mut Vec<UiaNode>,lines: &mut Vec<(usize, String)>,counter: &mut usize,- total: &mut usize,+ budget: &mut WalkBudget,) {- if depth >= MAX_DEPTH || *total >= MAX_TOTAL_ELEMENTS {+ if depth >= MAX_DEPTH || !budget.admit() {return;}- *total += 1;let self_var: VARIANT = VARIANT::from(0i32); // CHILDID_SELF@@ -245,7 +241,7 @@ unsafe fn walk(nodes,lines,counter,- total,+ budget,);}}@@ -267,7 +263,7 @@ unsafe fn walk(nodes,lines,counter,- total,+ budget,);}}diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rsindex 883d6d499c..e5a238d90f 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs@@ -342,6 +342,7 @@ fn window_target_candidates_for_pid(.filter(|window| window.pid == pid).map(|window| WindowTargetCandidate {window_id: window.hwnd,+ transient_for: None,title: window.title,app_name: None,is_on_screen: window.is_on_screen,@@ -1213,6 +1214,11 @@ mod get_window_state_max_image_dimension_tests {}}+/// Slack past `timeout_ms` before the walk task is abandoned.+const UIA_WALK_BACKSTOP_GRACE: std::time::Duration = std::time::Duration::from_millis(500);+/// Deadline for the window capture, separate from the walk budget.+const UIA_SCREENSHOT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(4);+#[async_trait]impl Tool for GetWindowStateTool {fn def(&self) -> &ToolDef {@@ -1288,6 +1294,7 @@ impl Tool for GetWindowStateTool {"query":{"type":"string","description":"Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count."},"max_elements":{"type":"integer","minimum":1,"description":"Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (5 000). Lower for Electron / large web apps that produce 10k+ element trees."},"max_depth":{"type":"integer","minimum":1,"description":"Cap on the UIA-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower for deep menu / Electron trees."},+ "timeout_ms": cua_driver_core::tool_schema::timeout_ms_schema(),"max_image_dimension":{"type":"integer","minimum":0,"description":"Per-call long-edge limit for the returned screenshot, in pixels (aspect ratio preserved). Explicit values override configured behavior; 0 returns native resolution. Omit to preserve the configured default."},"max_dimension":{"type":"integer","minimum":1,"description":"Legacy optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved). Applied on top of the configured max_image_dimension ceiling when max_image_dimension is omitted; the tighter wins."}},"additionalProperties":false}),@@ -1396,6 +1403,7 @@ impl Tool for GetWindowStateTool {.and_then(|v| v.as_u64()).map(|v| v.max(1) as usize).unwrap_or(crate::uia::DEFAULT_MAX_DEPTH);+ let timeout_ms = cua_driver_core::tool_schema::resolve_timeout_ms(args.get("timeout_ms"));// Always walk the UIA tree. The screenshot is returned by DEFAULT — the// grounding frame the agent cross-checks the (sometimes-lying) tree@@ -1428,26 +1436,55 @@ impl Tool for GetWindowStateTool {let state = self.state.clone();let q = query.clone();let out_file = screenshot_out_file.clone();- let blocking = tokio::task::spawn_blocking(move || -> anyhow::Result<_> {- let tree_result = if do_tree {- Some(crate::uia::walk_tree_bounded(- hwnd,- q.as_deref(),- max_elements,- max_depth,- ))- } else {- None- };- let tree_result = tree_result.map(|tree| {+ // The tree walk runs under `timeout_ms`. UIA's bulk BuildUpdatedCache+ // cannot be interrupted (Chrome's provider can block indefinitely on+ // property reads), so when the walk task itself outlives the budget+ // the response carries an empty tree flagged `truncated` instead of an+ // error, and the screenshot below still comes back.+ let tree_result = if do_tree {+ let tree_task = tokio::task::spawn_blocking(move || {+ let mut budget =+ cua_driver_core::walk_budget::WalkBudget::new(timeout_ms, max_elements);+ let tree =+ crate::uia::walk_tree_budgeted(hwnd, q.as_deref(), max_depth, &mut budget);let kind = if tree.nodes.iter().any(|node| node.msaa_role.is_some()) {crate::uia::cache::SnapshotKind::Msaa} else {crate::uia::cache::SnapshotKind::Uia};let payload = crate::uia::cache::CachedSnapshot::from_nodes(&tree.nodes, kind);- (tree, payload)+ (tree, payload, budget.outcome())});+ let started = std::time::Instant::now();+ let deadline = std::time::Duration::from_millis(timeout_ms) + UIA_WALK_BACKSTOP_GRACE;+ match tokio::time::timeout(deadline, tree_task).await {+ Ok(Ok(tree)) => Some(tree),+ Ok(Err(e)) => return ToolResult::error(format!("UIA walk task panicked: {e}")),+ Err(_elapsed) => {+ let class = crate::input::delivery::read_class_name(hwnd);+ let walk = cua_driver_core::walk_budget::WalkOutcome::timed_out(+ timeout_ms,+ started.elapsed(),+ );+ let tree = crate::uia::UiaTreeResult {+ tree_markdown: format!(+ "(The UIA provider for hwnd 0x{hwnd:x}, class '{class}', did not \+ answer. Act by pixel off the screenshot, or send press_key with \+ delivery_mode:\"foreground\" for a dialog's default accelerator.)"+ ),+ nodes: Vec::new(),+ };+ let payload = crate::uia::cache::CachedSnapshot::from_nodes(+ &[],+ crate::uia::cache::SnapshotKind::Uia,+ );+ Some((tree, payload, walk))+ }+ }+ } else {+ None+ };+ let blocking = tokio::task::spawn_blocking(move || -> anyhow::Result<_> {// Capture screenshot AND any error message so the response can// surface *why* there's no image (the iconic-window guard from// #1973 / PR #1974 is the load-bearing case: minimized windows@@ -1483,33 +1520,23 @@ impl Tool for GetWindowStateTool {} else {(None, None)};- Ok((tree_result, screenshot, screenshot_err))+ Ok((screenshot, screenshot_err))});- // Timeout: Chrome's UIA provider can block indefinitely on property reads.- let result: Result<anyhow::Result<_>, _> =- match tokio::time::timeout(std::time::Duration::from_secs(4), blocking).await {- Ok(join_result) => join_result.map_err(|e| anyhow::anyhow!("task panic: {e}")),- Err(_elapsed) => {- // Surface the target's window class + an actionable hint- // instead of just "UIA provider unresponsive". The class- // points the caller at the right workaround (e.g. SALFRAME- // → screenshot + pixel coords + delivery_mode:"foreground"; UWP- // class → re-call with a depth-limited scan and act by pixel- // off the screenshot if the tree stays unusable).- let class = crate::input::delivery::read_class_name(hwnd);- Err(anyhow::anyhow!(- "get_window_state timed out after 4s (UIA provider unresponsive on \- hwnd 0x{hwnd:x}, class '{class}'). Fallback options: \- (a) re-call this tool with a depth-limited scan \- (`max_elements` / `max_depth`) — if the tree stays unusable, act \- by pixel `click(x, y)` off the screenshot in the response; \- (b) if the target is a transient VCL / message-box dialog, send \- `press_key` with `delivery_mode:\"foreground\"` (SendInput) to fire the \- default accelerator (Esc / Enter / Y / N) without needing the tree."- ))- }+ let result: anyhow::Result<_> =+ match tokio::time::timeout(UIA_SCREENSHOT_TIMEOUT, blocking).await {+ Ok(join_result) => join_result+ .map_err(|e| anyhow::anyhow!("task panic: {e}"))+ .and_then(|r| r),+ Err(_elapsed) => Ok((+ None,+ Some(format!(+ "window capture did not finish within {}s",+ UIA_SCREENSHOT_TIMEOUT.as_secs()+ )),+ )),};- let result = result.and_then(|r| r);+ let result =+ result.map(|(screenshot, screenshot_err)| (tree_result, screenshot, screenshot_err));match result {Ok((tree_opt, screenshot_opt, screenshot_err)) => {@@ -1520,14 +1547,19 @@ impl Tool for GetWindowStateTool {.map(|(_, _, w, _, native_w, _, _)| *native_w as f64 / *w as f64);let mut published_snapshot = false;- if let Some((tr, payload)) = tree_opt {+ if let Some((tr, payload, walk)) = tree_opt {let is_msaa = tr.nodes.iter().any(|node| node.msaa_role.is_some());let count = tr.nodes.iter().filter(|n| n.element_index.is_some()).count();- let header = format!("window_id={hwnd} pid={pid} elements={count}\n\n");+ let mut header = format!("window_id={hwnd} pid={pid} elements={count}\n");+ if let Some(note) = walk.note() {+ header.push_str(¬e);+ header.push('\n');+ }+ header.push('\n');content.push(cua_driver_core::protocol::Content::text(header + &tr.tree_markdown,));@@ -1537,6 +1569,7 @@ impl Tool for GetWindowStateTool {// conservative until that proof is available.structured["elements_complete"] = json!(false);structured["tree_markdown"] = json!(tr.tree_markdown);+ walk.apply(&mut structured);let snapshot_id = (!observation_only).then(|| {@@ -1571,6 +1604,20 @@ impl Tool for GetWindowStateTool {query.as_deref(),&tr.tree_markdown,);+ // Screenshot pixels: the capture's top-left is the DWM+ // bitmap origin (the inverse of the pixel tools'+ // `bitmap_to_screen`), downsized by delivered/native.+ let elements = match screenshot_opt.as_ref() {+ Some((_, _, w, _, native_w, _, _)) if *native_w > 0 => {+ let (ox, oy) = bitmap_to_screen(hwnd, 0, 0);+ cua_driver_core::element_frame::with_screenshot_frames(+ elements,+ (f64::from(ox), f64::from(oy)),+ f64::from(*w) / f64::from(*native_w),+ )+ }+ _ => elements,+ };structured["total_element_count"] = json!(count);structured["returned_element_count"] = json!(elements.len());structured["elements"] = json!(elements);@@ -1603,6 +1650,9 @@ impl Tool for GetWindowStateTool {Cua Driver used a partial MSAA tree. Treat it as discovery \evidence only; it cannot prove checked state.");+ } else if let (0, Some(note)) = (count, walk.note()) {+ structured["degraded"] = json!(true);+ structured["degraded_reason"] = json!(note);} else if count == 0 {structured["degraded"] = json!(true);structured["degraded_reason"] = json!(@@ -8109,15 +8159,16 @@ impl Tool for GetDesktopStateTool {fn def(&self) -> &ToolDef {GDS_DEF.get_or_init(|| ToolDef {name: "get_desktop_state".into(),- description: "Capture the full display in true screen pixels with no downscale. \- Use its native-size PNG as the coordinate source for actions whose target is \- {kind:\"desktop\",display_id:\"primary\"}. Returns the true screen size. \- Vision-only: no UIA tree walk.".into(),+ description: "Capture the full display in true screen pixels, full size unless \+ `max_image_dimension` caps it. Use its PNG as the coordinate source for actions \+ whose target is {kind:\"desktop\",display_id:\"primary\"}. Returns the true \+ screen size. Vision-only: no UIA tree walk.".into(),input_schema: json!({"type": "object","properties": {"session": { "type": "string", "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session." },- "screenshot_out_file": { "type": "string", "description": "Write PNG here instead of base64." }+ "screenshot_out_file": { "type": "string", "description": "Write PNG here instead of base64." },+ "max_image_dimension": cua_driver_core::tool_schema::desktop_max_image_dimension_schema()},"additionalProperties": false}),@@ -8133,6 +8184,7 @@ impl Tool for GetDesktopStateTool {Err(error) => return error,};let screenshot_out_file = input.screenshot_out_file;+ let max_image_dimension = input.max_image_dimension.filter(|cap| *cap > 0);// True screen geometry in physical pixels (same space as the capture).let (screen_width, screen_height) = physical_screen_size();@@ -8141,21 +8193,30 @@ impl Tool for GetDesktopStateTool {// blocking GDI capture off the async runtime.let out_file = screenshot_out_file.clone();let res = tokio::task::spawn_blocking(- move || -> anyhow::Result<(Option<String>, Option<String>, u32, u32, Vec<u8>)> {+ move || -> anyhow::Result<(Option<String>, Option<String>, u32, u32, Vec<u8>, (u32, u32))> {use base64::{engine::general_purpose::STANDARD as BASE64, Engine};let png = crate::capture::screenshot_display_bytes()?;+ let full = crate::capture::png_dimensions_pub(&png)?;+ // Opt-in cap; later desktop-scope pixels from the capped+ // image are mapped back at dispatch and by its capture_id.+ let png = match max_image_dimension {+ Some(cap) if full.0.max(full.1) > cap => {+ crate::capture::resize_png_if_needed(&png, cap)?+ }+ _ => png,+ };let (w, h) = crate::capture::png_dimensions_pub(&png)?;if let Some(ref path) = out_file {std::fs::write(path, &png)?;- Ok((None, Some(path.clone()), w, h, png))+ Ok((None, Some(path.clone()), w, h, png, full))} else {- Ok((Some(BASE64.encode(&png)), None, w, h, png))+ Ok((Some(BASE64.encode(&png)), None, w, h, png, full))}},).await;- let (b64_opt, file_path, screenshot_width, screenshot_height, png) = match res {+ let (b64_opt, file_path, screenshot_width, screenshot_height, png, full_size) = match res {Ok(Ok(v)) => v,Ok(Err(e)) => return ToolResult::error(format!("Desktop screenshot failed: {e}")),Err(e) => return ToolResult::error(format!("Desktop screenshot task error: {e}")),@@ -8184,12 +8245,16 @@ impl Tool for GetDesktopStateTool {},"screenshot_mime_type": "image/png",});+ if full_size != (screenshot_width, screenshot_height) {+ structured["screenshot_original_width"] = json!(full_size.0);+ structured["screenshot_original_height"] = json!(full_size.1);+ }if let Some(bridge) = &self.state.capture_bridge {let geometry = match crate::capture_admission::CaptureGeometry::new(screenshot_width,screenshot_height,- screenshot_width,- screenshot_height,+ full_size.0,+ full_size.1,) {Ok(geometry) => geometry,Err(error) => {diff --git a/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rs b/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rsindex 193d89809d..24e2217e48 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/uia/mod.rs@@ -9,6 +9,7 @@//! (avoids per-property cross-process calls that make Chrome's 5000-node tree//! take >4s when reading each property individually).+use cua_driver_core::walk_budget::WalkBudget;use windows::core::{Interface, BSTR};use windows::Win32::System::Com::{CoCreateInstance, CoInitializeEx, CLSCTX_INPROC_SERVER, COINIT_MULTITHREADED,@@ -239,14 +240,28 @@ pub fn walk_tree_bounded(max_elements: usize,max_depth: usize,) -> UiaTreeResult {- unsafe { walk_tree_unsafe(hwnd, query, max_elements, max_depth) }+ let mut budget = WalkBudget::nodes_only(max_elements);+ walk_tree_budgeted(hwnd, query, max_depth, &mut budget)+}++/// [`walk_tree_bounded`] under a caller's [`WalkBudget`], checked per node of+/// the cached (and MSAA) walk. The bulk `BuildUpdatedCache` fetch before it is+/// setup the budget does not cover; the caller's backstop bounds it. Read the+/// outcome from `budget` afterwards.+pub fn walk_tree_budgeted(+ hwnd: u64,+ query: Option<&str>,+ max_depth: usize,+ budget: &mut WalkBudget,+) -> UiaTreeResult {+ unsafe { walk_tree_unsafe(hwnd, query, max_depth, budget) }}unsafe fn walk_tree_unsafe(hwnd: u64,query: Option<&str>,- max_elements: usize,max_depth: usize,+ budget: &mut WalkBudget,) -> UiaTreeResult {let _ = CoInitializeEx(None, COINIT_MULTITHREADED);@@ -318,7 +333,7 @@ unsafe fn walk_tree_unsafe(// Two reasons:// 1. **Hang avoidance** for SALSUBFRAME / SALMENU / SALTMPSUBFRAME —// VCL's UIA provider hangs on `BuildUpdatedCache(TreeScope.Subtree)`- // under the daemon's MTA pool, wasting the 4 s outer timeout.+ // under the daemon's MTA pool, wasting the timeout_ms walk budget.// 2. **Role fidelity** for SALFRAME (main document window, Recovery// dialog) — UIA technically walks fine, but the built-in// MSAA→UIA proxy collapses `ROLE_SYSTEM_BUTTONDROPDOWN` (0x38) to@@ -338,7 +353,7 @@ unsafe fn walk_tree_unsafe(n > 0 && String::from_utf16_lossy(&buf[..n as usize]).starts_with("SAL")};if sal_class {- return crate::msaa::walk_msaa_tree(hwnd);+ return crate::msaa::walk_msaa_tree_budgeted(hwnd, budget);}// Two-call sequence (ElementFromHandle + BuildUpdatedCache) instead of@@ -387,7 +402,6 @@ unsafe fn walk_tree_unsafe(let mut nodes: Vec<UiaNode> = Vec::new();let mut lines: Vec<(usize, String)> = Vec::new();let mut counter = 0usize;- let mut total = 0usize;walk_cached_bounded(&root_elem,@@ -397,8 +411,7 @@ unsafe fn walk_tree_unsafe(&mut nodes,&mut lines,&mut counter,- &mut total,- max_elements,+ budget,max_depth,);@@ -418,7 +431,7 @@ unsafe fn walk_tree_unsafe(// wrapper-only node from the primary walk stays the result — better than// erasing it AND leaving the consumed `MAX_TOTAL_ELEMENTS` budget intact// for the fallback (which would then truncate large trees prematurely).- if nodes.iter().filter(|n| n.element_index.is_some()).count() == 0 {+ if nodes.iter().filter(|n| n.element_index.is_some()).count() == 0 && !budget.expired() {// Skip the desktop-root walk-by-pid fallback for VCL / SAL// targets (LibreOffice, OpenOffice). The fallback does its own// `BuildUpdatedCache(TreeScope.Subtree)` per matched top-level@@ -450,7 +463,6 @@ unsafe fn walk_tree_unsafe(let mut fallback_nodes: Vec<UiaNode> = Vec::new();let mut fallback_lines: Vec<(usize, String)> = Vec::new();let mut fallback_counter = 0usize;- let mut fallback_total = 0usize;tracing::debug!(target: "uia",@@ -464,17 +476,16 @@ unsafe fn walk_tree_unsafe(&mut fallback_nodes,&mut fallback_lines,&mut fallback_counter,- &mut fallback_total,- max_elements,+ budget,max_depth,);if fallback_nodes.iter().any(|n| n.element_index.is_some()) {nodes = fallback_nodes;lines = fallback_lines;- // counter/total aren't read after this point — they're- // only used by walk_cached's &mut params for element- // indexing inside that call.+ // counter isn't read after this point — it's only used+ // by walk_cached's &mut params for element indexing+ // inside that call. The budget spans both walks.}}} else {@@ -487,7 +498,7 @@ unsafe fn walk_tree_unsafe(// Return a tree_markdown that mirrors the get_window_state// timeout diagnostic so callers get the same actionable// fallback options even though the walk itself didn't hit- // the 4 s outer timeout (because we skipped the hang-prone+ // the timeout_ms walk budget (because we skipped the hang-prone// fallback). Without this the caller sees an empty tree// and no error, which is less actionable.let stub = format!(@@ -550,8 +561,7 @@ unsafe fn walk_root_by_pid(nodes: &mut Vec<UiaNode>,lines: &mut Vec<(usize, String)>,counter: &mut usize,- total: &mut usize,- max_elements: usize,+ budget: &mut WalkBudget,max_depth: usize,) {let root = match automation.GetRootElement() {@@ -611,16 +621,7 @@ unsafe fn walk_root_by_pid(}};walk_cached_bounded(- &cached,- 0,- None,- false,- nodes,- lines,- counter,- total,- max_elements,- max_depth,+ &cached, 0, None, false, nodes, lines, counter, budget, max_depth,);}}@@ -632,7 +633,6 @@ unsafe fn walk_cached(nodes: &mut Vec<UiaNode>,lines: &mut Vec<(usize, String)>,counter: &mut usize,- total: &mut usize,) {walk_cached_bounded(element,@@ -642,8 +642,7 @@ unsafe fn walk_cached(nodes,lines,counter,- total,- MAX_TOTAL_ELEMENTS,+ &mut WalkBudget::nodes_only(MAX_TOTAL_ELEMENTS),MAX_DEPTH,);}@@ -657,14 +656,14 @@ unsafe fn walk_cached_bounded(nodes: &mut Vec<UiaNode>,lines: &mut Vec<(usize, String)>,counter: &mut usize,- total: &mut usize,- max_elements: usize,+ budget: &mut WalkBudget,max_depth: usize,) {- if depth > max_depth || *total >= max_elements {+ // Node and time budget: a refused node is counted as discovered but not+ // visited, and the walk unwinds with the partial tree it has.+ if depth > max_depth || !budget.admit() {return;}- *total += 1;let control_type = read_cached_control_type(element);let name = read_cached_bstr_name(element);@@ -756,8 +755,7 @@ unsafe fn walk_cached_bounded(nodes,lines,counter,- total,- max_elements,+ budget,max_depth,);}diff --git a/libs/cua-driver/typescript/src/native/cua_driver_contract.ts b/libs/cua-driver/typescript/src/native/cua_driver_contract.tsindex eba4dc48cc..604c832d08 100644--- a/libs/cua-driver/typescript/src/native/cua_driver_contract.ts+++ b/libs/cua-driver/typescript/src/native/cua_driver_contract.ts@@ -97,6 +97,73 @@ const FfiConverterTypeActionDelivery = (() => {return new FFIConverter();})();+const stringConverter = (() => {+ const encoder = new TextEncoder();+ const decoder = new TextDecoder();+ return {+ stringToBytes: (s: string) => encoder.encode(s),+ bytesToString: (ab: UniffiByteArray) => decoder.decode(ab),+ stringByteLength: (s: string) => encoder.encode(s).byteLength,+ writeStringIntoBuffer: (s: string, buf: any, offset: number): number => {+ const view = new Uint8Array(+ buf.arrayBuffer,+ offset,+ buf.arrayBuffer.byteLength - offset,+ );+ return encoder.encodeInto(s, view).written;+ },+ readStringFromBuffer: (buf: any, offset: number, length: number): string =>+ decoder.decode(new Uint8Array(buf.arrayBuffer, offset, length)),+ };+})();+const FfiConverterString = uniffiCreateFfiConverterString(stringConverter);++/**+ * Why a `refused` action sent no input, and what to do instead.+ */+export type ActionError = {+ code: string,+ hint?: string+}++/**+ * Generated factory for {@link ActionError} record objects.+ */+export const ActionError = (() => {+ const defaults = () => ({+ });+ const create = (() => {+ return uniffiCreateRecord<ActionError, ReturnType<typeof defaults>>(defaults);+ })();+ return Object.freeze({+ create,+ new: create,+ defaults: () => Object.freeze(defaults()) as Partial<ActionError>,+ });+})();++const FfiConverterTypeActionError = (() => {+ type TypeName = ActionError;+ class FFIConverter extends AbstractFfiConverterByteArray<TypeName> {+ read(from: RustBuffer): TypeName {+ return {+ code: FfiConverterString.read(from),+ hint: FfiConverterOptionalString.read(from)+ };+ }+ write(value: TypeName, into: RustBuffer): void {+ FfiConverterString.write(value.code, into);+ FfiConverterOptionalString.write(value.hint, into);+ }+ allocationSize(value: TypeName): number {+ return FfiConverterString.allocationSize(value.code) ++ FfiConverterOptionalString.allocationSize(value.hint);++ }+ };+ return new FFIConverter();+})();+export enum ActionEscalationTarget {Pixel,Foreground,@@ -243,7 +310,12 @@ const FfiConverterTypeActionEvidenceKind = (() => {})();export type ActionEvidence = {- kind: ActionEvidenceKind+ kind: ActionEvidenceKind,+ /**+ * Human-readable readback the evidence rests on (what changed, which+ * popup or window appeared and how to target it). Never request data.+ */+ detail?: string}/**@@ -267,14 +339,17 @@ const FfiConverterTypeActionEvidence = (() => {class FFIConverter extends AbstractFfiConverterByteArray<TypeName> {read(from: RustBuffer): TypeName {return {- kind: FfiConverterTypeActionEvidenceKind.read(from)+ kind: FfiConverterTypeActionEvidenceKind.read(from),+ detail: FfiConverterOptionalString.read(from)};}write(value: TypeName, into: RustBuffer): void {FfiConverterTypeActionEvidenceKind.write(value.kind, into);+ FfiConverterOptionalString.write(value.detail, into);}allocationSize(value: TypeName): number {- return FfiConverterTypeActionEvidenceKind.allocationSize(value.kind);+ return FfiConverterTypeActionEvidenceKind.allocationSize(value.kind) ++ FfiConverterOptionalString.allocationSize(value.detail);}};@@ -365,7 +440,18 @@ export type ActionResult = {route: ActionRoute,delivery?: ActionDelivery,evidence?: Array<ActionEvidence>,- escalation?: ActionEscalation+ escalation?: ActionEscalation,+ /**+ * The producer's human summary of what happened (resolved points, the+ * element hit, popups that opened, focus outcome, follow-up calls).+ * Clients that read only `structuredContent` still get everything the+ * text content says.+ */+ summary?: string,+ /**+ * Present only with `effect: refused`.+ */+ error?: ActionError}/**@@ -393,7 +479,9 @@ const FfiConverterTypeActionResult = (() => {route: FfiConverterTypeActionRoute.read(from),delivery: FfiConverterOptionalTypeActionDelivery.read(from),evidence: FfiConverterOptionalSequenceTypeActionEvidence.read(from),- escalation: FfiConverterOptionalTypeActionEscalation.read(from)+ escalation: FfiConverterOptionalTypeActionEscalation.read(from),+ summary: FfiConverterOptionalString.read(from),+ error: FfiConverterOptionalTypeActionError.read(from)};}write(value: TypeName, into: RustBuffer): void {@@ -402,40 +490,23 @@ const FfiConverterTypeActionResult = (() => {FfiConverterOptionalTypeActionDelivery.write(value.delivery, into);FfiConverterOptionalSequenceTypeActionEvidence.write(value.evidence, into);FfiConverterOptionalTypeActionEscalation.write(value.escalation, into);+ FfiConverterOptionalString.write(value.summary, into);+ FfiConverterOptionalTypeActionError.write(value.error, into);}allocationSize(value: TypeName): number {return FfiConverterTypeActionEffect.allocationSize(value.effect) +FfiConverterTypeActionRoute.allocationSize(value.route) +FfiConverterOptionalTypeActionDelivery.allocationSize(value.delivery) +FfiConverterOptionalSequenceTypeActionEvidence.allocationSize(value.evidence) +- FfiConverterOptionalTypeActionEscalation.allocationSize(value.escalation);+ FfiConverterOptionalTypeActionEscalation.allocationSize(value.escalation) ++ FfiConverterOptionalString.allocationSize(value.summary) ++ FfiConverterOptionalTypeActionError.allocationSize(value.error);}};return new FFIConverter();})();-const stringConverter = (() => {- const encoder = new TextEncoder();- const decoder = new TextDecoder();- return {- stringToBytes: (s: string) => encoder.encode(s),- bytesToString: (ab: UniffiByteArray) => decoder.decode(ab),- stringByteLength: (s: string) => encoder.encode(s).byteLength,- writeStringIntoBuffer: (s: string, buf: any, offset: number): number => {- const view = new Uint8Array(- buf.arrayBuffer,- offset,- buf.arrayBuffer.byteLength - offset,- );- return encoder.encodeInto(s, view).written;- },- readStringFromBuffer: (buf: any, offset: number, length: number): string =>- decoder.decode(new Uint8Array(buf.arrayBuffer, offset, length)),- };-})();-const FfiConverterString = uniffiCreateFfiConverterString(stringConverter);-export type AppInfo = {pid: number,name: string,@@ -2202,7 +2273,13 @@ export type GetDesktopStateInput = {/*** Write the PNG here instead of returning base64.*/- screenshotOutFile?: string+ screenshotOutFile?: string,+ /**+ * Optional long-edge cap for the returned PNG, in pixels. Omitted or 0+ * returns the full-size capture. When the cap downsizes the image,+ * desktop-scope x/y taken from it are mapped back automatically.+ */+ maxImageDimension?: number}/**@@ -2210,6 +2287,7 @@ export type GetDesktopStateInput = {*/export const GetDesktopStateInput = (() => {const defaults = () => ({+ maxImageDimension: undefined});const create = (() => {return uniffiCreateRecord<GetDesktopStateInput, ReturnType<typeof defaults>>(defaults);@@ -2227,16 +2305,19 @@ const FfiConverterTypeGetDesktopStateInput = (() => {read(from: RustBuffer): TypeName {return {session: FfiConverterOptionalString.read(from),- screenshotOutFile: FfiConverterOptionalString.read(from)+ screenshotOutFile: FfiConverterOptionalString.read(from),+ maxImageDimension: FfiConverterOptionalUInt32.read(from)};}write(value: TypeName, into: RustBuffer): void {FfiConverterOptionalString.write(value.session, into);FfiConverterOptionalString.write(value.screenshotOutFile, into);+ FfiConverterOptionalUInt32.write(value.maxImageDimension, into);}allocationSize(value: TypeName): number {return FfiConverterOptionalString.allocationSize(value.session) +- FfiConverterOptionalString.allocationSize(value.screenshotOutFile);+ FfiConverterOptionalString.allocationSize(value.screenshotOutFile) ++ FfiConverterOptionalUInt32.allocationSize(value.maxImageDimension);}};@@ -2387,7 +2468,13 @@ export type GetWindowStateInput = {* Optional per-call long-edge ceiling. Zero requests native resolution;* omit it to preserve the configured session or global behavior.*/- maxImageDimension?: number+ maxImageDimension?: number,+ /**+ * Wall-clock budget for the accessibility walk in milliseconds+ * (default 1000 on every platform). A walk that runs out returns a+ * partial tree flagged `truncated` rather than failing.+ */+ timeoutMs?: number}/**@@ -2395,6 +2482,7 @@ export type GetWindowStateInput = {*/export const GetWindowStateInput = (() => {const defaults = () => ({+ timeoutMs: undefined});const create = (() => {return uniffiCreateRecord<GetWindowStateInput, ReturnType<typeof defaults>>(defaults);@@ -2421,7 +2509,8 @@ const FfiConverterTypeGetWindowStateInput = (() => {maxElements: FfiConverterOptionalUInt32.read(from),maxDepth: FfiConverterOptionalUInt32.read(from),maxDimension: FfiConverterOptionalUInt32.read(from),- maxImageDimension: FfiConverterOptionalUInt32.read(from)+ maxImageDimension: FfiConverterOptionalUInt32.read(from),+ timeoutMs: FfiConverterOptionalUInt32.read(from)};}write(value: TypeName, into: RustBuffer): void {@@ -2436,6 +2525,7 @@ const FfiConverterTypeGetWindowStateInput = (() => {FfiConverterOptionalUInt32.write(value.maxDepth, into);FfiConverterOptionalUInt32.write(value.maxDimension, into);FfiConverterOptionalUInt32.write(value.maxImageDimension, into);+ FfiConverterOptionalUInt32.write(value.timeoutMs, into);}allocationSize(value: TypeName): number {return FfiConverterUInt32.allocationSize(value.pid) +@@ -2448,7 +2538,8 @@ const FfiConverterTypeGetWindowStateInput = (() => {FfiConverterOptionalUInt32.allocationSize(value.maxElements) +FfiConverterOptionalUInt32.allocationSize(value.maxDepth) +FfiConverterOptionalUInt32.allocationSize(value.maxDimension) +- FfiConverterOptionalUInt32.allocationSize(value.maxImageDimension);+ FfiConverterOptionalUInt32.allocationSize(value.maxImageDimension) ++ FfiConverterOptionalUInt32.allocationSize(value.timeoutMs);}};@@ -5791,6 +5882,9 @@ const FfiConverterTypePlatform = (() => {// FfiConverter for number | undefinedconst FfiConverterOptionalUInt32 = new FfiConverterOptional(FfiConverterUInt32);+// FfiConverter for string | undefined+const FfiConverterOptionalString = new FfiConverterOptional(FfiConverterString);+// FfiConverter for ActionDelivery | undefinedconst FfiConverterOptionalTypeActionDelivery = new FfiConverterOptional(FfiConverterTypeActionDelivery);@@ -5803,8 +5897,8 @@ const FfiConverterOptionalSequenceTypeActionEvidence = new FfiConverterOptional(// FfiConverter for ActionEscalation | undefinedconst FfiConverterOptionalTypeActionEscalation = new FfiConverterOptional(FfiConverterTypeActionEscalation);-// FfiConverter for string | undefined-const FfiConverterOptionalString = new FfiConverterOptional(FfiConverterString);+// FfiConverter for ActionError | undefined+const FfiConverterOptionalTypeActionError = new FfiConverterOptional(FfiConverterTypeActionError);// FfiConverter for number | undefinedconst FfiConverterOptionalFloat64 = new FfiConverterOptional(FfiConverterFloat64);@@ -5939,6 +6033,7 @@ export default Object.freeze({FfiConverterTypeActionDelivery,FfiConverterTypeActionDeliveryMode,FfiConverterTypeActionEffect,+ FfiConverterTypeActionError,FfiConverterTypeActionEscalation,FfiConverterTypeActionEscalationReason,FfiConverterTypeActionEscalationTarget,