Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

docs(cua-driver): document elements[].frame coordinate space (#2816)

webdevdd9628a7dea2
diff --git a/docs/content/docs/reference/cua-driver/limits.mdx b/docs/content/docs/reference/cua-driver/limits.mdx
index b9f48bd2bc..9ddde7bdf4 100644
--- a/docs/content/docs/reference/cua-driver/limits.mdx
+++ b/docs/content/docs/reference/cua-driver/limits.mdx
@@ -200,6 +200,79 @@ client on Sway, GNOME, or KDE, and its capabilities are documented separately.
---
+## `elements[].frame` is screen-absolute, but pixel actions use screenshot pixels
+
+**Symptom:** a caller reads an element's `frame` from `get_window_state` and
+passes its center straight to `click({pid, window_id, x, y})`. The click lands
+somewhere other than the element — displaced by the window's screen origin, and
+possibly scaled by the capture ratio. Nothing refuses, because the coordinates
+were structurally valid; they simply addressed a different point.
+
+**Cause:** element geometry and pixel action arguments are expressed in two
+different coordinate spaces, and neither field restates its own space.
+
+| Surface | Space | Units |
+| ----------------------------------------------------------- | ---------------------------------------------------- | ------------------------------------- |
+| `elements[].frame` on macOS | Screen-absolute, top-left origin | Logical points |
+| `elements[].frame` on Windows and Linux | Screen-absolute, top-left origin | Physical pixels |
+| `x`, `y` with `pid` and `window_id` | Window-local, top-left of the `get_window_state` PNG | Screenshot pixels |
+| `x`, `y` with `scope:"desktop"` and no `pid` or `window_id` | Screen-absolute | Pixels in the `get_desktop_state` PNG |
+
+With `pid` and `window_id`, the pixel forms of `click`, `right_click`,
+`double_click`, `drag`, `scroll`, `type_text`, `press_key`, and `hotkey` read
+the window-local screenshot space. `double_click` currently labels these as
+screen coordinates in its generated argument reference, but its implementation
+uses the same window-local conversion as the other pixel actions.
+
+**What to do:** prefer `element_token`, or `element_index` with the matching
+`snapshot_id`. The accessibility path carries no coordinates, so it is
+unaffected by this mismatch, and it also works on backgrounded windows. Reach
+for a pixel action only on canvas, WebGL, or custom-drawn surfaces that expose
+no element.
+
+When a pixel action is unavoidable, match the scope to the space:
+
+- **Window scope.** Rebase onto the window origin, then rescale to the capture.
+ Take the origin and size from `list_windows` (`windows[].bounds`), which is
+ reported in the same units as `frame`, and take the capture size from the
+ snapshot's `screenshot_width` and `screenshot_height`:
+
+ ```text
+ scale_x = screenshot_width / bounds.width
+ scale_y = screenshot_height / bounds.height
+
+ x = (frame.x + frame.w / 2 - bounds.x) * scale_x
+ y = (frame.y + frame.h / 2 - bounds.y) * scale_y
+ ```
+
+- **Desktop scope.** If the element is on the captured primary display, rescale
+ the frame center into the full-display PNG returned by `get_desktop_state`,
+ then call `click` with `scope:"desktop"` and no `pid` or `window_id`:
+
+ ```text
+ scale_x = desktop.screenshot_width / desktop.screen_width
+ scale_y = desktop.screenshot_height / desktop.screen_height
+
+ x = (frame.x + frame.w / 2) * scale_x
+ y = (frame.y + frame.h / 2) * scale_y
+ ```
+
+ These ratios are normally 1 on Windows and Linux. On macOS Retina displays,
+ the frame is in logical points while the desktop PNG is in native pixels, so
+ the ratio is normally 2.
+
+<Callout type="info">
+ Do not assume a window-scope scale is 1. It is normally 2 on a Retina display, and may be a
+ fraction whenever a large window was downscaled to `max_image_dimension`. Re-derive it from each
+ snapshot rather than caching it across calls.
+</Callout>
+
+On macOS, `get_window_state` also reports `window_bounds`, which may be used in
+place of `list_windows` there. Windows and Linux report `screenshot_width` and
+`screenshot_height` only.
+
+---
+
## Permission boundaries
Cua Driver is constrained by the macOS permission model. Two relevant grants: