Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(driver): add per-session capture scope (#2329)

Francesco Bonaccidb0ee870c200
diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
index 726273d5ef..f6afe0c5bc 100644
--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx
+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
@@ -12,7 +12,7 @@ description: Reference for every MCP tool Cua Driver exposes
import { Callout } from 'fumadocs-ui/components/callout';
-`cua-driver` exposes 47 MCP tools through a single stdio server (`cua-driver mcp`). Every tool is also callable from the shell as `cua-driver <name> '<JSON-args>'`.
+`cua-driver` exposes 49 MCP tools through a single stdio server (`cua-driver mcp`). Every tool is also callable from the shell as `cua-driver <name> '<JSON-args>'`.
Tool names are `snake_case`. Responses are MCP `CallTool.Result` envelopes: a text content block prefixed with a `✅` summary (or the error reason on failure), plus optional image or structured-content blocks on tools that produce them. See the [CLI reference](/reference/cua-driver/cli-reference) for CLI-specific options like `--socket` and `--screenshot-out-file`.
@@ -292,7 +292,8 @@ When `from_zoom` is true, coordinates are in the last zoom image for this pid; t
- `from_y` (number, required): Drag-start Y in window-local screenshot pixels. Top-left origin.
- `from_zoom` (boolean, optional): When true, coordinates are in the last zoom image for this pid; driver maps back to window coordinates.
- `modifier` (array of string, optional): Modifier keys held across the entire gesture: cmd/shift/option/ctrl.
-- `pid` (integer, required): Target process ID.
+- `pid` (integer, optional): Target process ID.
+- `scope` (string, optional): Use desktop with no pid/window_id for native get_desktop_state screenshot coordinates.
- `session` (string, optional): Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less.
- `steps` (integer, optional): Number of intermediate mouseDragged events linearly interpolated along the path. Default: 20.
- `to_x` (number, required): Drag-end X in window-local screenshot pixels.
@@ -300,7 +301,7 @@ When `from_zoom` is true, coordinates are in the last zoom image for this pid; t
- `window_id` (integer, optional): CGWindowID for the window the pixel coordinates were measured against. Optional — when omitted the driver picks the frontmost window of pid.
```json
-{"from_x":100,"from_y":200,"pid":844,"to_x":100,"to_y":200}
+{"from_x":100,"from_y":200,"to_x":100,"to_y":200}
```
### `type_text`
@@ -317,7 +318,8 @@ WEB CONTENT (Chromium/WebKit/Electron — browser tabs, Slack, VS Code, X's comp
- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": AX insert, then CGEvent keystrokes if needed — no focus steal; the driver verifies via an AXValue read-back and reports `verified`. "foreground": briefly front the window, type, restore the prior frontmost — the explicit last resort for focus-sensitive surfaces (e.g. WhatsApp/Catalyst) where background keystrokes don't land. Re-call with "foreground" when a background attempt returns `verified:false` and a screenshot shows the text didn't appear.
- `element_index` (integer, optional): Element index from last get_window_state. Directs the write to a specific field. REQUIRES `pid` and `window_id` to be passed alongside it — element_index alone (no pid) fails fast with "Missing required integer field: pid"; it is not a silent no-op.
- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit "stale" error if the snapshot has been superseded.
-- `pid` (integer, required): Target process ID.
+- `pid` (integer, optional): Target process ID.
+- `scope` (string, optional): Use desktop with no pid/window_id to type into the frontmost application.
- `session` (string, optional): Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less.
- `text` (string, required): Text to insert at the target's cursor.
- `window_id` (integer, optional): CGWindowID. Required when element_index is used. Optional when element_token is supplied (the token carries it).
@@ -325,7 +327,7 @@ WEB CONTENT (Chromium/WebKit/Electron — browser tabs, Slack, VS Code, X's comp
- `y` (number, optional): Screenshot-pixel Y of the field (see x).
```json
-{"pid":844,"text":"hello"}
+{"text":"hello"}
```
### `press_key`
@@ -343,14 +345,15 @@ A key press is never driver-verifiable → effect:"unverifiable"; confirm via sc
- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit "stale" error if the snapshot has been superseded.
- `key` (string, required): Key name: return, tab, escape, up, down, etc.
- `modifiers` (array of string, optional): Modifier keys: cmd, shift, option/alt, ctrl, fn.
-- `pid` (integer, required)
+- `pid` (integer, optional)
+- `scope` (string, optional): Use desktop with no pid/window_id to send the key to the frontmost application.
- `session` (string, optional): Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less.
- `window_id` (integer, optional): Target window. Required for delivery_mode:"foreground". Does NOT itself raise the window — raising is gated on delivery_mode.
- `x` (number, optional): Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_index.
- `y` (number, optional): Screenshot-pixel Y (see x).
```json
-{"key":"return","pid":844}
+{"key":"return"}
```
### `hotkey`
@@ -367,14 +370,15 @@ Recognized modifiers: cmd/command, shift, option/alt, ctrl/control, fn. Non-modi
- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target — no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it.
- `keys` (array of string, required): Modifier(s) and one non-modifier key, e.g. ["cmd", "c"].
-- `pid` (integer, required): Target process ID.
+- `pid` (integer, optional): Target process ID.
+- `scope` (string, optional): Use desktop with no pid/window_id to send the chord to the frontmost application.
- `session` (string, optional): Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less.
- `window_id` (integer, optional): Target window. Required for delivery_mode:"foreground" (the NSMenu activation needs a window). Does NOT itself raise the window — raising is gated on delivery_mode.
- `x` (number, optional): Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the combo (so e.g. Cmd+V pastes into that field). Pass with y. Use for Chromium/Electron surfaces the background combo can't reach.
- `y` (number, optional): Screenshot-pixel Y (see x).
```json
-{"keys":["cmd","c"],"pid":844}
+{"keys":["cmd","c"]}
```
### `set_value`
@@ -419,6 +423,7 @@ Mapping: by='page' → larger step; by='line' → smaller step; amount = number
- `element_index` (integer, optional): Element from last get_window_state. Routes through the pixel-wheel path AT this element's center — use it to scroll a nested overflow region you located in the AX tree.
- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit "stale" error if the snapshot has been superseded. Routes through the pixel-wheel path at the element's center.
- `pid` (integer, optional)
+- `scope` (string, optional): Use desktop with x,y and no pid/window_id for native get_desktop_state screenshot coordinates.
- `session` (string, optional): Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less.
- `window_id` (integer, optional)
- `x` (number, optional): Window-local screenshot X (top-left origin of the PNG from get_window_state). With `y`, routes through the pixel-wheel path at this point — use for a scrollable surface that isn't in the AX tree. Requires window_id to anchor the window→screen conversion.
@@ -430,11 +435,12 @@ Mapping: by='page' → larger step; by='line' → smaller step; amount = number
### `move_cursor`
-Move the agent cursor overlay to (x, y). Does NOT move the real mouse cursor — the user's cursor stays where it is. Useful for showing the agent's attention without interrupting the user.
+Move a cursor to (x, y). In window scope (default), moves only the agent cursor overlay. With scope=desktop, moves the real OS pointer in native get_desktop_state screenshot coordinates.
**Arguments:**
- `cursor_id` (string, optional): Cursor instance to move. Default: 'default'.
+- `scope` (string, optional)
- `session` (string, optional): Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less.
- `x` (number, required)
- `y` (number, required)
@@ -560,13 +566,12 @@ Caveats:
### `set_config`
-Update cua-driver-rs configuration. Changes to max_image_dimension and capture_scope take effect immediately. The experimental_pip keys are persisted to ~/.cua-driver/config.json and take effect on the next daemon restart (the PiP backend is initialised once at startup).
+Update cua-driver-rs configuration. Changes to max_image_dimension take effect immediately. The experimental_pip keys are persisted to ~/.cua-driver/config.json and take effect on the next daemon restart (the PiP backend is initialised once at startup).
-Note: capture_mode is a per-call param (on get_window_state / click), not a stored setting. capture_scope IS a global setting: it gates get_desktop_state (full-display capture requires capture_scope=desktop).
+Note: capture_mode is a per-call param (on get_window_state / click), not a stored setting. Capture scope is selected by start_session, not set_config.
**Arguments:**
-- `capture_scope` (string, optional): Capture scope: "window" (default) or "desktop". Desktop scope enables get_desktop_state (full-display capture) and window-less screen-absolute click/scroll. Global setting; takes effect immediately.
- `experimental_pip` (boolean, optional): Enable the experimental picture-in-picture preview window. Applies on next daemon restart.
- `experimental_pip_geometry` (string, optional): PiP window size + optional position in `WxH` or `WxH+X+Y` form (e.g. `320x200+24+24`). Applies on next daemon restart.
- `key` (string, optional): Name of a single config field to write (&#123;key, value&#125; shape, matching the CLI `config set` and the Windows/Linux tools). Pair with `value`. Equivalent to passing the field directly.
@@ -575,10 +580,11 @@ Note: capture_mode is a per-call param (on get_window_state / click), not a stor
### `start_session`
-Declare a session — a named, color-coded identity for THIS agent run. Pass a stable `session` id; the agent cursor, per-session config, and recording all key on it, and it follows the run across any apps/windows. The cursor's color is derived from the id, so distinct runs are visually distinct. A cursor is shown only for a declared session — call this (or pass `session` on your first action) to opt in. Idempotent: re-calling with the same id just refreshes its idle-TTL. End it with `end_session` (or let the idle-TTL reclaim it). Concurrent runs/subagents each pass their own `session` to get their own cursor.
+Declare a session — a named, color-coded identity for THIS agent run. Pass a stable `session` id and choose capture_scope=auto|window|desktop; the agent cursor, capture policy, per-session config, and recording all key on it, and it follows the run across any apps/windows. The cursor's color is derived from the id, so distinct runs are visually distinct. A cursor is shown only for a declared session — call this (or pass `session` on your first action) to opt in. Idempotent: re-calling with the same id just refreshes its idle-TTL. End it with `end_session` (or let the idle-TTL reclaim it). Concurrent runs/subagents each pass their own `session` to get their own cursor.
**Arguments:**
+- `capture_scope` (string, optional): Per-session perception/action modality. auto starts window-only and requires explicit escalation before desktop tools; window and desktop are strict. Immutable for the live session.
- `session` (string, required): Stable session id for this run (e.g. "research-run-1").
```json
@@ -901,3 +907,29 @@ Perform hover, right-click, double-click, scroll, or drag in an exactly-bound br
```json
{"action":"hover","session":"example","tab_id":"example","target_id":"example"}
```
+
+### `escalate_session`
+
+Unlock the desktop phase of an auto capture-scope session after the window action ladder has been exhausted and verified. This is a one-way transition for the live session and records a bounded reason.
+
+**Arguments:**
+
+- `detail` (string, optional): Optional bounded diagnostic detail. Never use secrets or page content.
+- `reason` (string, required)
+- `session` (string, required)
+
+```json
+{"reason":"ax_tree_pixel_mismatch","session":"example"}
+```
+
+### `get_session_state`
+
+Read the live session's capture policy and effective scope.
+
+**Arguments:**
+
+- `session` (string, required)
+
+```json
+{"session":"example"}
+```
diff --git a/libs/cua-driver/docs/test-matrix.md b/libs/cua-driver/docs/test-matrix.md
index b574985a5f..039f02d818 100644
--- a/libs/cua-driver/docs/test-matrix.md
+++ b/libs/cua-driver/docs/test-matrix.md
@@ -45,6 +45,7 @@ These run without the repo-local GUI applications:
| --- | --- | --- |
| Core driver logic | `rust/crates/*/src/**` | Protocol values, sessions, schemas, image helpers, input helpers, configuration, telemetry, CLI behavior |
| MCP and CLI boundary | `rust/crates/cua-driver/tests/protocol_*` | Handshake, tool registration, tool calls, media, sessions, and errors |
+| Session capture scope | `session_capture_scope_test.rs` plus core `capture_scope` tests | Per-session isolation, immutable live policy, explicit auto escalation, end/revive race safety, transport-mirror refusal, and retired persistent key |
| Schema gate | `schema_consistency_test.rs` | Shared tool schema parity across OS backends |
| Configuration transport | `transport_config_persistence_test.rs` | CLI and MCP configuration persistence |
| Token and protocol surfaces | `protocol_element_token_test.rs`, related tests | JSON-RPC-visible contract behavior |
@@ -106,6 +107,7 @@ Windows native harnesses are repo-local applications built from source:
| WPF | `harness_wpf_test.rs` | UIA controls, text, keys, pointer actions, scroll, drag, popups, menus, modal windows |
| WinUI3 | `harness_winui3_test.rs` | XAML controls, text, checkbox/radio, slider, combo, popup |
| WebView2 | `harness_web_test.rs` | Window discovery, CDP page access, JavaScript, DOM click path |
+| Desktop scope | `desktop_scope_windows_test.rs` | Concurrent strict window/desktop sessions, full-display capture, screen-absolute click/scroll, and strict rejection |
| Desktop invariants | Testkit `DesktopObserver` plus typed launch/capture/cursor owners | Cross-cutting focus, z-order, minimized-launch, screenshot, cursor, and desktop checks |
Native controls use AX/UIA state as their oracle. Pointer actions also use PX
@@ -119,6 +121,7 @@ background delivery and attach desktop-side-effect oracles.
| AppKit | `harness_appkit_test.rs` | AX tree/capture, AX value/text, AX scroll, PX clicks, and foreground slider drag across the proven delivery modes; background drag is an exact refusal |
| SwiftUI | `harness_swiftui_test.rs` | AX tree/capture, background click/value, and foreground popover-trigger state |
| WKWebView | `cross_platform_behavior_test.rs` | Dedicated native host running the full 40-cell shared web catalog |
+| Desktop scope | `desktop_scope_macos_test.rs` | Concurrent strict window/desktop sessions, screen-absolute action delivery, and strict rejection |
| Installed-app launch/focus | `installed_app_launch_macos_test.rs` | Real Calculator/TextEdit launch and focus behavior in the canonical logged-in lane |
| Installed-app text | `installed_app_textedit_macos_test.rs` | Real TextEdit AX background write and verification in the canonical logged-in lane |
@@ -138,7 +141,7 @@ transient-panel AX discovery gap.
| Electron | `cross_platform_behavior_test.rs` | X11 and hosted Sway | Shared web action matrix |
| Tauri | `cross_platform_behavior_test.rs` | X11 and hosted Sway | Shared web action matrix |
| GTK3 | `harness_gtk3_test.rs` | X11/AT-SPI and Wayland/AT-SPI where configured | Native GTK controls and input |
-| Desktop scope | `desktop_scope_linux_test.rs` | X11/Wayland | Desktop versus window scope |
+| Desktop scope | `desktop_scope_linux_test.rs` | X11/Wayland | Concurrent strict window/desktop sessions, screen-absolute action delivery, and strict rejection |
Nix provides the Linux build and desktop environment. X11 and Wayland are
separate matrix dimensions because their capture and input contracts differ.
diff --git a/libs/cua-driver/rust/Skills/cua-driver/LINUX.md b/libs/cua-driver/rust/Skills/cua-driver/LINUX.md
index ec83f3a8fb..d950968c22 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/LINUX.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/LINUX.md
@@ -113,12 +113,11 @@ CI so the three surfaces can't drift. Linux-relevant notes:
Linux builds rejected it via `additionalProperties:false` (it was
effectively macOS-only); it is now uniformly schema-accepted — Linux
glides a per-session cursor on X11 where the overlay is available.
-- **Windowless screen-absolute clicks** are supported on Linux, but Linux
- has **no per-call `scope` param** (that form is macOS-only). Linux gates
- the windowless path on the persisted `capture_scope` config: opt in once
- with `set_config capture_scope=desktop`, then send `x,y` with no
- pid/window_id. Under the default `capture_scope=window` a windowless
- action is rejected with a structured `desktop_scope_disabled` error.
+- **Windowless screen-absolute actions** pass `scope:"desktop"` with no
+ pid/window_id, uniformly with macOS and Windows. The session must have
+ effective desktop scope (`start_session(..., capture_scope:"desktop")`, or
+ an explicitly escalated `auto` session). Strict window sessions receive
+ `desktop_scope_disabled`; no persistent config is read or written.
## AT-SPI needs the session bus (headless / containers / `runuser`)
diff --git a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
index efd7f16024..06ed894644 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
@@ -307,12 +307,11 @@ addressing params are a shared cross-platform contract — see `SKILL.md` →
- **`session` always worked on macOS;** the cross-platform change is that
Windows/Linux stopped *rejecting* it. No macOS-side change to how you
pass it.
-- **`scope`** (`window` default / `desktop`) is a **macOS-specific per-call
- param** on the pointer tools — pass `scope:"desktop"` with `x,y` and no
- pid/window_id for a screen-absolute click. Windows and Linux support the
- same windowless capability but gate it on the persisted `capture_scope`
- config instead (`set_config capture_scope=desktop`), so they have no
- per-call `scope` param. Unifying the knob is a tracked follow-up.
+- **`scope`** (`window` / `desktop`) selects the action form uniformly on all
+ platforms. Pass `scope:"desktop"` with no pid/window_id for screen-absolute
+ pointer actions or foreground keyboard actions. The session's immutable
+ `capture_scope` policy must permit that form; set it with `start_session`,
+ never persistent config.
### Canvases, viewports, games (Blender, Unity, GHOST, Qt, wxWidgets)
diff --git a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
index 3c6a474bd4..8ea15095fe 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
@@ -62,10 +62,11 @@ Use whichever combination matches the host. When in doubt, run
`cua-driver doctor` — it reports the platform and the right entry
point.
-## The no-foreground principle (cross-platform)
+## The no-foreground principle (window phase)
-**The user's frontmost app MUST NOT change.** Every platform has its
-own list of forbidden commands; the principle is universal:
+In a strict `window` session, and during the initial window phase of an
+`auto` session, **the user's frontmost app MUST NOT change.** Every platform
+has its own list of forbidden commands:
- macOS: any `open` invocation, any `osascript` that mutates GUI
state, `cliclick`, `cghidEventTap` writes targeting another app's
@@ -78,6 +79,12 @@ If you reach for a command that says "activate", "foreground",
"raise", or "make key", stop and translate to the cua-driver tool
that does the same intent without focus-stealing.
+A strict `desktop` session is an explicit user choice to operate the visible
+desktop and therefore uses foreground/system input. An `auto` session may enter
+that phase only after the complete window ladder below has been attempted and
+verified, followed by `escalate_session`. Never infer desktop permission from a
+failed action or a proxy/transport session id.
+
## Defaults — always prefer cua-driver over shell shims
**Default transport is the `cua-driver` CLI** — `Bash` shelling out
@@ -184,7 +191,9 @@ Requires the daemon process's UI runloop, which `cua-driver serve` /
## The core invariant — snapshot before AND after every action
-**Every action MUST be bracketed by `get_window_state(pid, window_id)`**:
+**Every action MUST be bracketed by the state tool for the session's effective
+scope**: `get_window_state(pid, window_id)` in window scope, or
+`get_desktop_state(session)` in desktop scope.
- **Before** — the pre-action snapshot resolves the `element_index`
you're about to use. Indices from previous turns are stale; the
@@ -200,8 +209,39 @@ Requires the daemon process's UI runloop, which `cua-driver serve` /
the action fired. If nothing changed, the action probably failed
silently — say so, don't assume success.
-This applies to pixel clicks too — re-snapshot after to confirm the
-click landed on the intended target.
+This applies to pixel clicks and desktop actions too — re-snapshot after to
+confirm the action landed on the intended target.
+
+## Choose capture scope when the session starts
+
+`capture_scope` is a per-session policy, not persistent configuration. Declare
+it with `start_session`; it is immutable until that session ends. Concurrent
+sessions may choose different policies safely.
+
+- `auto` (default): begins with effective scope `window`. Desktop perception
+ and actions are locked until the window ladder is exhausted, each attempted
+ action is verified, and the caller explicitly invokes `escalate_session`.
+ Escalation is one-way for the live session.
+- `window`: strict window-only perception and actions. Desktop tools are always
+ rejected with `desktop_scope_disabled`.
+- `desktop`: strict full-desktop perception and foreground/system actions.
+ Window-scoped perception and actions are rejected with
+ `window_scope_disabled`.
+
+```bash
+cua-driver start_session '{"session":"research-1","capture_scope":"auto"}'
+cua-driver get_session_state '{"session":"research-1"}'
+```
+
+Do not use `config set capture_scope` or `set_config`; that key is retired and
+stale values on disk are ignored. Always pass the public `session` field on
+state and action calls. Reserved fields such as `_session_id` are transport
+metadata and cannot create or change policy.
+
+During a mixed-version rollout, require `tools/list` to advertise
+`session.capture_scope` (and `session.capture_scope.escalate` for `auto`). If an
+older daemon does not advertise them, fail closed and ask for an upgrade; never
+fall back to the retired global config key.
### Why window selection is the caller's job now
@@ -362,6 +402,13 @@ if resp.effect == "suspected_noop"
get_window_state(pid, window_id) # re-snapshot, eyeball the result
if it landed: done
+# Rung 2b — exact browser page tools, when get_browser_state can bind this window
+# Use typed browser refs for page content; native window tools still handle chrome.
+get_browser_state(session, pid, window_id)
+browser_click(session, target_id, tab_id, ref) # or browser_type; see BROWSER.md
+get_browser_state(session, pid, window_id) # verify with fresh refs
+if it landed: done
+
# Rung 3 — background delivery was dropped (insert/click never arrived)
if resp.escalation.recommended == "foreground"
or the px action still did nothing:
@@ -369,6 +416,16 @@ if resp.escalation.recommended == "foreground"
# on Wayland this is the ONLY escalation — px-bg can't target an
# unfocused window there; see LINUX.md
verify again
+
+# Rung 4 — desktop fallback (auto sessions only, explicit and one-way)
+# Reach this only after AX, window-pixel, browser-page (when available), and
+# foreground-window delivery have all been exhausted and verified ineffective.
+escalate_session(session,
+ reason="foreground_ineffective", # or another advertised reason
+ detail="bounded non-sensitive summary")
+get_desktop_state(session) # full primary display
+desktop_action(session, scope="desktop", ...) # no pid/window_id
+get_desktop_state(session) # verify in the same coordinate frame
```
The two ideas to hold onto: (1) the AX tree **lies** on canvas / web /
@@ -399,7 +456,7 @@ last resort.
## The canonical loop
```
-start_session(session) # once per run: declares this run's identity
+start_session(session, capture_scope="auto") # once per run; policy is immutable
launch_app(target)
→ pick window_id from the returned `windows` array
(or call list_windows(pid) separately)
@@ -409,19 +466,28 @@ launch_app(target)
end_session(session) # when the run finishes
```
+For strict desktop sessions, replace the window portion with
+`get_desktop_state(session) → action(session, scope="desktop", ...) →
+get_desktop_state(session)`. Desktop actions use screen-absolute coordinates
+from that exact full-display image and omit `pid`/`window_id`. The global
+`get_screen_size` and `get_cursor_position` helpers are desktop-scoped too.
+
`launch_app` now returns a `windows` array alongside the pid, so the
common case collapses to two calls (`launch_app` → `get_window_state`)
without a separate `list_windows` hop.
**Declare a session.** A session is _your run's_ identity — a stable id
you choose (`"research-1"`), declared with `start_session` and passed as
-`session` on every action. It owns your agent cursor (a distinct colour
-per id), follows the run across any apps/windows, and is the same whether
+`session` on every action. It owns your agent cursor and capture policy (a
+distinct colour and one immutable policy per id), follows the run across any
+apps/windows, and is the same whether
you drive over MCP, the CLI, or the socket. Declaring the session creates the
cursor; anonymous actions remain cursor-less.
End with `end_session` (or the idle-TTL reclaims it).
-**Concurrent runs/subagents:** `launch_app` is idempotent — two runs that
+**Concurrent runs/subagents:** each run may independently choose `auto`,
+`window`, or `desktop`; one session's escalation never changes another. Also,
+`launch_app` is idempotent — two runs that
launch the same app get the **same** instance (and on single-instance apps
like Calculator, the same window), so they clobber each other. Give each run
its **own `session`** (→ its own cursor) AND pass
@@ -551,10 +617,15 @@ against a specific window.
| Send key to pid | `press_key({pid, key, modifiers})` | no focus change; key goes to pid's current focus |
| Modifier combo | `hotkey({pid, keys})` (no focus) or `hotkey({pid, x, y, keys})` (px) | e.g. `["cmd","c"]` / `["ctrl","c"]`; posted per-pid, not HID tap. **px** pixel-clicks `(x,y)` to focus a field first, e.g. `["cmd","v"]` to paste into it |
-**All keyboard/text primitives require `pid`.** There is no
-frontmost-routed variant — every key goes to the named target via
-the platform's per-pid event-post path, so the driver cannot leak
-keystrokes into the user's foreground app.
+In effective desktop scope, the foreground/system equivalents omit
+`pid`/`window_id` and pass `scope:"desktop"`: `click`, `scroll`, `drag`,
+`move_cursor`, `type_text`, `press_key`, and `hotkey`. Coordinates are
+screen-absolute pixels from the latest `get_desktop_state` image.
+
+**Window-scope keyboard/text primitives require `pid`.** They use the named
+target's per-pid event-post path. Only a strict/effective desktop session may
+omit `pid`, and it intentionally routes keyboard input to the current
+foreground application.
**Why `element_index` is the primary path:** works on hidden /
occluded / off-desktop windows, no focus steal, stable across
diff --git a/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md b/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md
index 8d9d7a2ee7..dd89080938 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md
@@ -404,11 +404,11 @@ When a cua-driver call surprises you, diagnose cua-driver first:
regardless of what you pass.
- **`get_desktop_state` returns `desktop_scope_disabled`?** That's
intended: full-display capture is a **desktop-scope** operation, gated
- on the global `capture_scope`. It's `"window"` by default — so to verify
- a specific window use `get_window_state(pid, window_id)` (works
- backgrounded), and only use `get_desktop_state` after
- `set_config capture_scope=desktop` (the same opt-in that enables
- window-less screen-absolute `click`/`scroll`). Don't reach for
+ by the caller-declared session policy. To verify a specific window use
+ `get_window_state(pid, window_id)` (works backgrounded). Use
+ `get_desktop_state` only in a strict desktop session or after explicitly
+ escalating an `auto` session once the window ladder is exhausted. Desktop
+ actions pass `scope:"desktop"` with no pid/window_id. Don't reach for
`get_desktop_state` as a casual screenshot — it's the capture surface for
desktop-scope coordinate loops, not window inspection.
- **`Calc display stuck at 0 after my clicks`?** Almost always
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/capture_scope.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/capture_scope.rs
new file mode 100644
index 0000000000..860eb0523f
--- /dev/null
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/capture_scope.rs
@@ -0,0 +1,511 @@
+//! Per-session capture/action scope policy.
+//!
+//! The policy is keyed exclusively by the public, caller-declared `session`
+//! argument. Reserved transport mirrors such as `_session_id` must never mint
+//! policy state: doing so would let an anonymous proxy connection bypass the
+//! per-session contract.
+
+use serde::{Deserialize, Serialize};
+use serde_json::{json, Value};
+use std::collections::HashMap;
+use std::fmt;
+use std::sync::{Mutex, OnceLock};
+
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum CaptureScopePolicy {
+ #[default]
+ Auto,
+ Window,
+ Desktop,
+}
+
+impl CaptureScopePolicy {
+ pub fn parse(value: &str) -> Option<Self> {
+ match value {
+ "auto" => Some(Self::Auto),
+ "window" => Some(Self::Window),
+ "desktop" => Some(Self::Desktop),
+ _ => None,
+ }
+ }
+
+ pub fn as_str(self) -> &'static str {
+ match self {
+ Self::Auto => "auto",
+ Self::Window => "window",
+ Self::Desktop => "desktop",
+ }
+ }
+}
+
+impl fmt::Display for CaptureScopePolicy {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ f.write_str(self.as_str())
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum EffectiveCaptureScope {
+ Window,
+ Desktop,
+}
+
+impl EffectiveCaptureScope {
+ pub fn as_str(self) -> &'static str {
+ match self {
+ Self::Window => "window",
+ Self::Desktop => "desktop",
+ }
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum EscalationReason {
+ AxTreePixelMismatch,
+ BackgroundDeliveryFailed,
+ ForegroundIneffective,
+ NoWindowTarget,
+ Other,
+}
+
+impl EscalationReason {
+ pub fn parse(value: &str) -> Option<Self> {
+ match value {
+ "ax_tree_pixel_mismatch" => Some(Self::AxTreePixelMismatch),
+ "background_delivery_failed" => Some(Self::BackgroundDeliveryFailed),
+ "foreground_ineffective" => Some(Self::ForegroundIneffective),
+ "no_window_target" => Some(Self::NoWindowTarget),
+ "other" => Some(Self::Other),
+ _ => None,
+ }
+ }
+
+ pub fn as_str(self) -> &'static str {
+ match self {
+ Self::AxTreePixelMismatch => "ax_tree_pixel_mismatch",
+ Self::BackgroundDeliveryFailed => "background_delivery_failed",
+ Self::ForegroundIneffective => "foreground_ineffective",
+ Self::NoWindowTarget => "no_window_target",
+ Self::Other => "other",
+ }
+ }
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct SessionCaptureScope {
+ pub policy: CaptureScopePolicy,
+ pub desktop_unlocked: bool,
+ pub escalation_reason: Option<EscalationReason>,
+ pub escalation_detail: Option<String>,
+}
+
+impl SessionCaptureScope {
+ pub fn new(policy: CaptureScopePolicy) -> Self {
+ Self {
+ policy,
+ desktop_unlocked: policy == CaptureScopePolicy::Desktop,
+ escalation_reason: None,
+ escalation_detail: None,
+ }
+ }
+
+ pub fn effective_scope(&self) -> EffectiveCaptureScope {
+ match self.policy {
+ CaptureScopePolicy::Desktop => EffectiveCaptureScope::Desktop,
+ CaptureScopePolicy::Window => EffectiveCaptureScope::Window,
+ CaptureScopePolicy::Auto if self.desktop_unlocked => EffectiveCaptureScope::Desktop,
+ CaptureScopePolicy::Auto => EffectiveCaptureScope::Window,
+ }
+ }
+
+ pub fn as_json(&self, session: &str) -> Value {
+ json!({
+ "session": session,
+ "capture_scope": self.policy.as_str(),
+ "effective_scope": self.effective_scope().as_str(),
+ "desktop_unlocked": self.desktop_unlocked,
+ "escalation_reason": self.escalation_reason.map(EscalationReason::as_str),
+ "escalation_detail": self.escalation_detail,
+ })
+ }
+}
+
+static SESSION_SCOPES: OnceLock<Mutex<HashMap<String, SessionCaptureScope>>> = OnceLock::new();
+
+fn scopes() -> &'static Mutex<HashMap<String, SessionCaptureScope>> {
+ SESSION_SCOPES.get_or_init(|| Mutex::new(HashMap::new()))
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum BindError {
+ Ended,
+ Conflict {
+ existing: CaptureScopePolicy,
+ requested: CaptureScopePolicy,
+ },
+}
+
+/// Bind a policy to a live session. Omitted policy means "keep the existing
+/// policy, or use auto for a new session" so repeated `start_session` calls
+/// remain idempotent.
+pub fn bind_session(
+ session: &str,
+ requested: Option<CaptureScopePolicy>,
+) -> Result<(SessionCaptureScope, bool), BindError> {
+ let mut registry = scopes().lock().unwrap();
+ if crate::session::is_session_ended(session) {
+ return Err(BindError::Ended);
+ }
+ if let Some(existing) = registry.get(session) {
+ if let Some(requested) = requested {
+ if existing.policy != requested {
+ return Err(BindError::Conflict {
+ existing: existing.policy,
+ requested,
+ });
+ }
+ }
+ return Ok((existing.clone(), false));
+ }
+ let state = SessionCaptureScope::new(requested.unwrap_or_default());
+ registry.insert(session.to_owned(), state.clone());
+ Ok((state, true))
+}
+
+pub fn get_session(session: &str) -> Option<SessionCaptureScope> {
+ scopes().lock().unwrap().get(session).cloned()
+}
+
+pub fn clear_session(session: &str) {
+ scopes().lock().unwrap().remove(session);
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum EscalateError {
+ Ended,
+ NotStarted,
+ WindowStrict,
+ DesktopAlreadyActive,
+}
+
+pub fn escalate_session(
+ session: &str,
+ reason: EscalationReason,
+ detail: Option<&str>,
+) -> Result<SessionCaptureScope, EscalateError> {
+ let mut registry = scopes().lock().unwrap();
+ // This check deliberately happens while holding the scope lock. If an end
+ // races with escalation, either this write wins first and end removes it,
+ // or the tombstone wins first and this write is rejected. A late action
+ // can therefore never resurrect an unlocked session.
+ if crate::session::is_session_ended(session) {
+ return Err(EscalateError::Ended);
+ }
+ let Some(state) = registry.get_mut(session) else {
+ return Err(EscalateError::NotStarted);
+ };
+ if state.desktop_unlocked {
+ return Err(EscalateError::DesktopAlreadyActive);
+ }
+ match state.policy {
+ CaptureScopePolicy::Window => return Err(EscalateError::WindowStrict),
+ CaptureScopePolicy::Desktop => return Err(EscalateError::DesktopAlreadyActive),
+ CaptureScopePolicy::Auto => {}
+ }
+ state.desktop_unlocked = true;
+ state.escalation_reason = Some(reason);
+ state.escalation_detail = detail.map(|value| value.chars().take(200).collect());
+ Ok(state.clone())
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum ToolScope {
+ Window,
+ Desktop,
+ Unscoped,
+}
+
+fn is_scoped_action(tool_name: &str) -> bool {
+ matches!(
+ tool_name,
+ "click"
+ | "double_click"
+ | "right_click"
+ | "scroll"
+ | "drag"
+ | "mouse_drag"
+ | "parallel_mouse_drag"
+ | "move_cursor"
+ | "type_text"
+ | "type_text_chars"
+ | "press_key"
+ | "hotkey"
+ | "set_value"
+ )
+}
+
+fn tool_scope(tool_name: &str, args: &Value) -> ToolScope {
+ if matches!(
+ tool_name,
+ "get_desktop_state" | "get_screen_size" | "get_cursor_position"
+ ) {
+ return ToolScope::Desktop;
+ }
+ if matches!(
+ tool_name,
+ "get_window_state"
+ | "get_accessibility_tree"
+ | "get_app_state"
+ | "screenshot"
+ | "screenshot_compat"
+ | "zoom"
+ | "page"
+ )
+ || tool_name == "get_browser_state"
+ || (tool_name.starts_with("browser_")
+ && !matches!(tool_name, "browser_prepare" | "browser_download"))
+ {
+ return ToolScope::Window;
+ }
+ if is_scoped_action(tool_name) {
+ let explicit_desktop = args.get("scope").and_then(Value::as_str) == Some("desktop");
+ let has_window_target = args.get("pid").is_some()
+ || args.get("window_id").is_some()
+ || args.get("element_index").is_some()
+ || args.get("element_token").is_some();
+ if explicit_desktop && !has_window_target {
+ ToolScope::Desktop
+ } else {
+ ToolScope::Window
+ }
+ } else {
+ ToolScope::Unscoped
+ }
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct ScopeViolation {
+ pub code: &'static str,
+ pub message: String,
+ pub state: SessionCaptureScope,
+}
+
+impl ScopeViolation {
+ pub fn as_json(&self, session: &str) -> Value {
+ let mut value = self.state.as_json(session);
+ if let Some(object) = value.as_object_mut() {
+ object.insert("code".to_owned(), Value::String(self.code.to_owned()));
+ }
+ value
+ }
+}
+
+/// Enforce the modality for a public session. A first non-lifecycle action
+/// implicitly starts that session in `auto`, matching the existing cursor and
+/// recording lifecycle behavior.
+pub fn enforce_tool(tool_name: &str, args: &Value) -> Result<(), ScopeViolation> {
+ if matches!(
+ tool_name,
+ "start_session" | "end_session" | "escalate_session" | "get_session_state"
+ ) {
+ return Ok(());
+ }
+ let Some(session) = args
+ .get("session")
+ .and_then(Value::as_str)
+ .filter(|value| !value.is_empty() && *value != "default")
+ else {
+ return Ok(());
+ };
+ let (state, _) = bind_session(session, None).map_err(|_| ScopeViolation {
+ code: "session_ended",
+ message: format!("session '{session}' has ended"),
+ state: SessionCaptureScope::new(CaptureScopePolicy::Auto),
+ })?;
+ match (state.effective_scope(), tool_scope(tool_name, args)) {
+ (_, ToolScope::Unscoped)
+ | (EffectiveCaptureScope::Window, ToolScope::Window)
+ | (EffectiveCaptureScope::Desktop, ToolScope::Desktop) => Ok(()),
+ (EffectiveCaptureScope::Window, ToolScope::Desktop)
+ if state.policy == CaptureScopePolicy::Auto =>
+ {
+ Err(ScopeViolation {
+ code: "desktop_escalation_required",
+ message: format!(
+ "desktop-scope tool '{tool_name}' is locked for auto session '{session}'; exhaust the window action ladder, verify each attempt, then call escalate_session"
+ ),
+ state,
+ })
+ }
+ (EffectiveCaptureScope::Window, ToolScope::Desktop) => Err(ScopeViolation {
+ code: "desktop_scope_disabled",
+ message: format!(
+ "desktop-scope tool '{tool_name}' is disabled for strict window session '{session}'"
+ ),
+ state,
+ }),
+ (EffectiveCaptureScope::Desktop, ToolScope::Window) => Err(ScopeViolation {
+ code: "window_scope_disabled",
+ message: format!(
+ "window-scope tool '{tool_name}' is disabled while session '{session}' is in desktop scope"
+ ),
+ state,
+ }),
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn fresh(prefix: &str) -> String {
+ format!("capture-scope-{prefix}-{}", std::process::id())
+ }
+
+ #[test]
+ fn sessions_are_isolated_and_auto_escalates_one_way() {
+ let auto = fresh("auto");
+ let desktop = fresh("desktop");
+ let window = fresh("window");
+ bind_session(&auto, Some(CaptureScopePolicy::Auto)).unwrap();
+ bind_session(&desktop, Some(CaptureScopePolicy::Desktop)).unwrap();
+ bind_session(&window, Some(CaptureScopePolicy::Window)).unwrap();
+
+ assert_eq!(
+ enforce_tool("get_desktop_state", &json!({"session": auto}))
+ .unwrap_err()
+ .code,
+ "desktop_escalation_required"
+ );
+ assert!(enforce_tool("get_desktop_state", &json!({"session": desktop})).is_ok());
+ assert_eq!(
+ enforce_tool("get_desktop_state", &json!({"session": window}))
+ .unwrap_err()
+ .code,
+ "desktop_scope_disabled"
+ );
+
+ escalate_session(&auto, EscalationReason::ForegroundIneffective, None).unwrap();
+ assert!(enforce_tool("get_desktop_state", &json!({"session": auto})).is_ok());
+ assert_eq!(
+ enforce_tool("get_window_state", &json!({"session": auto, "pid": 1}))
+ .unwrap_err()
+ .code,
+ "window_scope_disabled"
+ );
+ assert_eq!(
+ get_session(&desktop).unwrap().effective_scope(),
+ EffectiveCaptureScope::Desktop
+ );
+ }
+
+ #[test]
+ fn conflicting_live_rebind_is_rejected() {
+ let session = fresh("conflict");
+ bind_session(&session, Some(CaptureScopePolicy::Window)).unwrap();
+ assert_eq!(
+ bind_session(&session, Some(CaptureScopePolicy::Desktop)),
+ Err(BindError::Conflict {
+ existing: CaptureScopePolicy::Window,
+ requested: CaptureScopePolicy::Desktop,
+ })
+ );
+ }
+
+ #[test]
+ fn transport_mirror_does_not_create_policy() {
+ let session = fresh("mirror");
+ assert!(enforce_tool("get_desktop_state", &json!({"_session_id": session})).is_ok());
+ assert!(get_session(&session).is_none());
+ }
+
+ #[test]
+ fn escalation_detail_is_bounded() {
+ let session = fresh("detail");
+ bind_session(&session, Some(CaptureScopePolicy::Auto)).unwrap();
+ let state =
+ escalate_session(&session, EscalationReason::Other, Some(&"x".repeat(500))).unwrap();
+ assert_eq!(state.escalation_detail.unwrap().chars().count(), 200);
+ }
+
+ #[test]
+ fn strict_modes_apply_symmetrically_to_perception_and_actions() {
+ let window = fresh("strict-window-actions");
+ let desktop = fresh("strict-desktop-actions");
+ bind_session(&window, Some(CaptureScopePolicy::Window)).unwrap();
+ bind_session(&desktop, Some(CaptureScopePolicy::Desktop)).unwrap();
+
+ assert_eq!(
+ enforce_tool(
+ "click",
+ &json!({"session": window, "scope": "desktop", "x": 4, "y": 5})
+ )
+ .unwrap_err()
+ .code,
+ "desktop_scope_disabled"
+ );
+ assert_eq!(
+ enforce_tool("get_cursor_position", &json!({"session": window}))
+ .unwrap_err()
+ .code,
+ "desktop_scope_disabled"
+ );
+ assert!(enforce_tool("get_screen_size", &json!({"session": desktop})).is_ok());
+ assert_eq!(
+ enforce_tool(
+ "click",
+ &json!({"session": desktop, "pid": 42, "x": 4, "y": 5})
+ )
+ .unwrap_err()
+ .code,
+ "window_scope_disabled"
+ );
+ }
+
+ #[test]
+ fn end_racing_escalation_never_resurrects_scope_state() {
+ use std::sync::{Arc, Barrier};
+
+ let session = fresh("end-escalate-race");
+ bind_session(&session, Some(CaptureScopePolicy::Auto)).unwrap();
+ let barrier = Arc::new(Barrier::new(3));
+ let escalation_session = session.clone();
+ let escalation_barrier = barrier.clone();
+ let escalation = std::thread::spawn(move || {
+ escalation_barrier.wait();
+ escalate_session(
+ &escalation_session,
+ EscalationReason::ForegroundIneffective,
+ None,
+ )
+ });
+ let end_session = session.clone();
+ let end_barrier = barrier.clone();
+ let end = std::thread::spawn(move || {
+ end_barrier.wait();
+ crate::session::fire_session_end(&end_session)
+ });
+ barrier.wait();
+ let _ = escalation.join().unwrap();
+ assert!(end.join().unwrap());
+
+ assert!(crate::session::is_session_ended(&session));
+ assert!(get_session(&session).is_none());
+ assert_eq!(
+ enforce_tool("get_window_state", &json!({"session": session}))
+ .unwrap_err()
+ .code,
+ "session_ended"
+ );
+
+ assert!(crate::session::revive_session(&session));
+ let (fresh_state, created) =
+ bind_session(&session, Some(CaptureScopePolicy::Window)).unwrap();
+ assert!(created);
+ assert_eq!(fresh_state.policy, CaptureScopePolicy::Window);
+ assert!(!fresh_state.desktop_unlocked);
+ }
+}
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs
index 52bd0bd6e6..e28bc9888e 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs
@@ -34,6 +34,7 @@ pub fn embedded_mode() -> bool {
pub mod browser;
pub mod capture_mode;
+pub mod capture_scope;
pub mod cdp;
pub mod element_cache;
pub mod element_token;
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/recording_render.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/recording_render.rs
index 325b84cace..f0165449a5 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/recording_render.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/recording_render.rs
@@ -252,4 +252,3 @@ fn probe_duration_ms(video_path: &Path) -> Option<f64> {
let secs: f64 = s.trim().parse().ok()?;
Some(secs * 1000.0)
}
-
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs
index f0e6bf29d0..5ae762837e 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs
@@ -306,6 +306,7 @@ pub fn fire_session_end(session_id: &str) -> bool {
return false; // already ended — idempotent no-op.
}
}
+ crate::capture_scope::clear_session(session_id);
for hook in hooks().lock().unwrap().iter() {
hook(session_id);
}
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/session_tools.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/session_tools.rs
index 20d51afc98..ac7c559d9a 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/session_tools.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/session_tools.rs
@@ -1,4 +1,4 @@
-//! `start_session` / `end_session` tools.
+//! Session lifecycle and per-session capture-scope tools.
//!
//! A session is a **caller-declared** identity for an agent run (see
//! [`crate::session`]). It owns the agent cursor and is the key for
@@ -7,26 +7,51 @@
//! identically over MCP, the CLI (`--session`), or the raw socket, and a run
//! can span any number of apps/windows.
//!
-//! The daemon mirrors an explicit `session` arg into the reserved `_session_id`
-//! key (see `serve.rs::apply_session_identity`), so these tools accept either —
-//! `session` is the public name.
+//! The daemon may mirror an explicit `session` arg into reserved transport
+//! fields, but lifecycle and policy tools accept only the public `session`
+//! name. Transport metadata can never mint or alter capture policy.
+use crate::capture_scope::{
+ bind_session, escalate_session, get_session, BindError, CaptureScopePolicy, EscalateError,
+ EscalationReason,
+};
use crate::protocol::ToolResult;
use crate::tool::{Tool, ToolDef};
use async_trait::async_trait;
use serde_json::{json, Value};
use std::sync::OnceLock;
-/// Read the explicit session id from a tool call (`session`, or its daemon
-/// mirror `_session_id`). Empty / missing → `None`.
+/// Read the public, caller-declared session id. Capture-scope policy must never
+/// bind to the daemon's reserved `_session_id` transport mirror.
fn session_id_of(args: &Value) -> Option<String> {
- let obj = args.as_object()?;
- ["session", "_session_id"]
- .into_iter()
- .find_map(|k| obj.get(k).and_then(|v| v.as_str()).filter(|s| !s.is_empty()))
+ args.as_object()?
+ .get("session")
+ .and_then(|v| v.as_str())
+ .filter(|s| !s.is_empty() && *s != "default")
.map(|s| s.to_owned())
}
+fn capture_scope_arg(args: &Value) -> Result<Option<CaptureScopePolicy>, ToolResult> {
+ let Some(raw) = args.get("capture_scope") else {
+ return Ok(None);
+ };
+ let Some(raw) = raw.as_str() else {
+ return Err(ToolResult::error(
+ "start_session.capture_scope must be one of: auto, window, desktop.",
+ )
+ .with_structured(json!({ "code": "invalid_capture_scope" })));
+ };
+ CaptureScopePolicy::parse(raw).map(Some).ok_or_else(|| {
+ ToolResult::error(format!(
+ "invalid capture_scope '{raw}'; expected auto, window, or desktop"
+ ))
+ .with_structured(json!({
+ "code": "invalid_capture_scope",
+ "capture_scope": raw,
+ }))
+ })
+}
+
// ── start_session ─────────────────────────────────────────────────────────────
pub struct StartSessionTool;
@@ -40,8 +65,9 @@ impl Tool for StartSessionTool {
name: "start_session".into(),
description:
"Declare a session — a named, color-coded identity for THIS agent run. \
- Pass a stable `session` id; the agent cursor, per-session config, and \
- recording all key on it, and it follows the run across any apps/windows. \
+ Pass a stable `session` id and choose capture_scope=auto|window|desktop; \
+ the agent cursor, capture policy, per-session config, and recording all \
+ key on it, and it follows the run across any apps/windows. \
The cursor's color is derived from the id, so distinct runs are visually \
distinct. A cursor is shown only for a declared session — call this (or \
pass `session` on your first action) to opt in. Idempotent: re-calling \
@@ -56,6 +82,12 @@ impl Tool for StartSessionTool {
"session": {
"type": "string",
"description": "Stable session id for this run (e.g. \"research-run-1\")."
+ },
+ "capture_scope": {
+ "type": "string",
+ "enum": ["auto", "window", "desktop"],
+ "default": "auto",
+ "description": "Per-session perception/action modality. auto starts window-only and requires explicit escalation before desktop tools; window and desktop are strict. Immutable for the live session."
}
},
"additionalProperties": true
@@ -69,21 +101,208 @@ impl Tool for StartSessionTool {
async fn invoke(&self, args: Value) -> ToolResult {
let Some(id) = session_id_of(&args) else {
- return ToolResult::error(
- "start_session requires a non-empty `session` id.",
- );
+ return ToolResult::error("start_session requires a non-empty `session` id.");
};
+ let requested = match capture_scope_arg(&args) {
+ Ok(scope) => scope,
+ Err(result) => return result,
+ };
+ let was_ended = crate::session::is_session_ended(&id);
+ if !was_ended {
+ match bind_session(&id, requested) {
+ Ok(_) => {}
+ Err(BindError::Conflict {
+ existing,
+ requested,
+ }) => {
+ return ToolResult::error(format!(
+ "session '{id}' already uses capture_scope='{existing}'; capture scope is immutable until the session ends"
+ ))
+ .with_structured(json!({
+ "code": "session_policy_conflict",
+ "session": id,
+ "capture_scope": existing.as_str(),
+ "requested_capture_scope": requested.as_str(),
+ }));
+ }
+ Err(BindError::Ended) => {
+ return ToolResult::error(format!(
+ "session '{id}' ended concurrently; retry start_session to revive it"
+ ))
+ .with_structured(json!({ "code": "session_ended", "session": id }));
+ }
+ }
+ }
// Revive a recycled id: if this session was previously ended (explicit
// `end_session` / idle-TTL / connection EOF), clear its tombstone so its
// actions stop being rejected by the daemon's resurrection guard.
// Re-declaring a session is the EXPLICIT, caller-driven way to reuse an
// id — a stray late action still can't silently resurrect a dead one.
let revived = crate::session::revive_session(&id);
+ let (scope, _) = match bind_session(&id, requested) {
+ Ok(bound) => bound,
+ Err(BindError::Conflict {
+ existing,
+ requested,
+ }) => {
+ return ToolResult::error(format!(
+ "session '{id}' already uses capture_scope='{existing}'; capture scope is immutable until the session ends"
+ ))
+ .with_structured(json!({
+ "code": "session_policy_conflict",
+ "session": id,
+ "capture_scope": existing.as_str(),
+ "requested_capture_scope": requested.as_str(),
+ }));
+ }
+ Err(BindError::Ended) => {
+ return ToolResult::error(format!("session '{id}' could not be revived"))
+ .with_structured(json!({ "code": "session_ended", "session": id }));
+ }
+ };
// Refresh (or begin) the session's idle-TTL clock. The cursor appears on
// the first action carrying this `session`.
crate::session::touch_session(&id);
- ToolResult::text(format!("✅ Session '{id}' is active."))
- .with_structured(json!({ "session": id, "active": true, "revived": revived }))
+ let mut structured = scope.as_json(&id);
+ if let Some(object) = structured.as_object_mut() {
+ object.insert("active".to_owned(), Value::Bool(true));
+ object.insert("revived".to_owned(), Value::Bool(revived));
+ }
+ ToolResult::text(format!(
+ "✅ Session '{id}' is active with capture_scope='{}'.",
+ scope.policy
+ ))
+ .with_structured(structured)
+ }
+}
+
+// ── escalate_session ─────────────────────────────────────────────────────────
+
+pub struct EscalateSessionTool;
+
+static ESCALATE_DEF: OnceLock<ToolDef> = OnceLock::new();
+
+#[async_trait]
+impl Tool for EscalateSessionTool {
+ fn def(&self) -> &ToolDef {
+ ESCALATE_DEF.get_or_init(|| ToolDef {
+ name: "escalate_session".into(),
+ description: "Unlock the desktop phase of an auto capture-scope session after the window action ladder has been exhausted and verified. This is a one-way transition for the live session and records a bounded reason.".into(),
+ input_schema: json!({
+ "type": "object",
+ "required": ["session", "reason"],
+ "properties": {
+ "session": { "type": "string" },
+ "reason": {
+ "type": "string",
+ "enum": [
+ "ax_tree_pixel_mismatch",
+ "background_delivery_failed",
+ "foreground_ineffective",
+ "no_window_target",
+ "other"
+ ]
+ },
+ "detail": {
+ "type": "string",
+ "maxLength": 200,
+ "description": "Optional bounded diagnostic detail. Never use secrets or page content."
+ }
+ },
+ "additionalProperties": true
+ }),
+ read_only: false,
+ destructive: false,
+ idempotent: false,
+ open_world: false,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let Some(id) = session_id_of(&args) else {
+ return ToolResult::error("escalate_session requires a public `session` id.")
+ .with_structured(json!({ "code": "session_required" }));
+ };
+ let Some(reason) = args
+ .get("reason")
+ .and_then(Value::as_str)
+ .and_then(EscalationReason::parse)
+ else {
+ return ToolResult::error("escalate_session.reason is invalid.")
+ .with_structured(json!({ "code": "invalid_escalation_reason" }));
+ };
+ let detail = args.get("detail").and_then(Value::as_str);
+ match escalate_session(&id, reason, detail) {
+ Ok(state) => ToolResult::text(format!("✅ Session '{id}' escalated to desktop scope."))
+ .with_structured(state.as_json(&id)),
+ Err(error) => {
+ let (code, message) = match error {
+ EscalateError::Ended => (
+ "session_ended",
+ format!("session '{id}' has ended; start it again before escalating"),
+ ),
+ EscalateError::NotStarted => (
+ "session_not_started",
+ format!("session '{id}' has no capture policy; call start_session first"),
+ ),
+ EscalateError::WindowStrict => (
+ "desktop_scope_disabled",
+ format!("session '{id}' is strict window scope and cannot escalate"),
+ ),
+ EscalateError::DesktopAlreadyActive => (
+ "desktop_already_active",
+ format!("session '{id}' already has effective desktop scope"),
+ ),
+ };
+ ToolResult::error(message).with_structured(json!({
+ "code": code,
+ "session": id,
+ }))
+ }
+ }
+ }
+}
+
+// ── get_session_state ────────────────────────────────────────────────────────
+
+pub struct GetSessionStateTool;
+
+static GET_STATE_DEF: OnceLock<ToolDef> = OnceLock::new();
+
+#[async_trait]
+impl Tool for GetSessionStateTool {
+ fn def(&self) -> &ToolDef {
+ GET_STATE_DEF.get_or_init(|| ToolDef {
+ name: "get_session_state".into(),
+ description: "Read the live session's capture policy and effective scope.".into(),
+ input_schema: json!({
+ "type": "object",
+ "required": ["session"],
+ "properties": { "session": { "type": "string" } },
+ "additionalProperties": true
+ }),
+ read_only: true,
+ destructive: false,
+ idempotent: true,
+ open_world: false,
+ })
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let Some(id) = session_id_of(&args) else {
+ return ToolResult::error("get_session_state requires a public `session` id.")
+ .with_structured(json!({ "code": "session_required" }));
+ };
+ let Some(state) = get_session(&id) else {
+ return ToolResult::error(format!("session '{id}' is not active"))
+ .with_structured(json!({ "code": "session_not_started", "session": id }));
+ };
+ ToolResult::text(format!(
+ "Session '{id}' uses capture_scope='{}' (effective_scope='{}').",
+ state.policy,
+ state.effective_scope().as_str()
+ ))
+ .with_structured(state.as_json(&id))
}
}
@@ -98,12 +317,11 @@ impl Tool for EndSessionTool {
fn def(&self) -> &ToolDef {
END_DEF.get_or_init(|| ToolDef {
name: "end_session".into(),
- description:
- "End a session declared with `start_session`: removes its agent cursor, \
+ description: "End a session declared with `start_session`: removes its agent cursor, \
stops any recording it owns, and clears its per-session config. Call this \
when a run finishes so its cursor doesn't linger (otherwise the idle-TTL \
reclaims it after a period of inactivity). Idempotent."
- .into(),
+ .into(),
input_schema: json!({
"type": "object",
"required": ["session"],
@@ -136,12 +354,85 @@ impl Tool for EndSessionTool {
mod tests {
use super::*;
+ fn result_code(result: &ToolResult) -> Option<&str> {
+ result
+ .structured_content
+ .as_ref()?
+ .get("code")?
+ .as_str()
+ }
+
#[test]
- fn session_id_of_reads_session_then_mirror() {
- assert_eq!(session_id_of(&json!({ "session": "a" })).as_deref(), Some("a"));
- assert_eq!(session_id_of(&json!({ "_session_id": "b" })).as_deref(), Some("b"));
- assert_eq!(session_id_of(&json!({ "session": "", "_session_id": "c" })).as_deref(), Some("c"));
+ fn session_id_of_reads_only_public_session() {
+ assert_eq!(
+ session_id_of(&json!({ "session": "a" })).as_deref(),
+ Some("a")
+ );
+ assert_eq!(session_id_of(&json!({ "_session_id": "b" })), None);
+ assert_eq!(
+ session_id_of(&json!({ "session": "", "_session_id": "c" })),
+ None
+ );
assert_eq!(session_id_of(&json!({})), None);
assert_eq!(session_id_of(&json!({ "session": "" })), None);
+ assert_eq!(session_id_of(&json!({ "session": "default" })), None);
+ }
+
+ #[tokio::test]
+ async fn live_policy_is_immutable_and_ended_id_gets_fresh_policy() {
+ let id = format!("session-tool-policy-{}", std::process::id());
+ let start = StartSessionTool;
+ let first = start
+ .invoke(json!({"session": id, "capture_scope": "window"}))
+ .await;
+ assert_ne!(first.is_error, Some(true));
+ assert_eq!(
+ first.structured_content.as_ref().unwrap()["capture_scope"],
+ "window"
+ );
+
+ let conflict = start
+ .invoke(json!({"session": id, "capture_scope": "desktop"}))
+ .await;
+ assert_eq!(conflict.is_error, Some(true));
+ assert_eq!(result_code(&conflict), Some("session_policy_conflict"));
+
+ EndSessionTool.invoke(json!({"session": id})).await;
+ assert!(get_session(&id).is_none());
+ let revived = start
+ .invoke(json!({"session": id, "capture_scope": "desktop"}))
+ .await;
+ assert_ne!(revived.is_error, Some(true));
+ let structured = revived.structured_content.as_ref().unwrap();
+ assert_eq!(structured["capture_scope"], "desktop");
+ assert_eq!(structured["effective_scope"], "desktop");
+ assert_eq!(structured["revived"], true);
+ }
+
+ #[tokio::test]
+ async fn auto_requires_explicit_bounded_escalation() {
+ let id = format!("session-tool-auto-{}", std::process::id());
+ let started = StartSessionTool.invoke(json!({"session": id})).await;
+ assert_eq!(
+ started.structured_content.as_ref().unwrap()["capture_scope"],
+ "auto"
+ );
+ let escalated = EscalateSessionTool
+ .invoke(json!({
+ "session": id,
+ "reason": "background_delivery_failed",
+ "detail": "window ladder exhausted"
+ }))
+ .await;
+ assert_ne!(escalated.is_error, Some(true));
+ let structured = escalated.structured_content.as_ref().unwrap();
+ assert_eq!(structured["effective_scope"], "desktop");
+ assert_eq!(structured["desktop_unlocked"], true);
+
+ let twice = EscalateSessionTool
+ .invoke(json!({"session": id, "reason": "other"}))
+ .await;
+ assert_eq!(twice.is_error, Some(true));
+ assert_eq!(result_code(&twice), Some("desktop_already_active"));
}
}
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs
index da0299c091..9b21b31ad0 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs
@@ -95,7 +95,9 @@ impl ToolDef {
/// `system.permissions.tcc.accessibility`,
/// `system.permissions.tcc.screen_recording`
/// - `system.config.read`, `system.config.write`
-/// - `session.lifecycle.start`, `session.lifecycle.end`
+/// - `session.lifecycle.start`, `session.lifecycle.end`,
+/// `session.capture_scope`, `session.capture_scope.read`,
+/// `session.capture_scope.escalate`
/// - `agent_cursor.move`, `agent_cursor.set_enabled`,
/// `agent_cursor.set_motion`, `agent_cursor.set_style`,
/// `agent_cursor.state`
@@ -141,13 +143,10 @@ pub fn default_capabilities_for(tool_name: &str) -> Vec<String> {
"mouse_button_up" => &["input.pointer.button"],
"scroll" => &["input.pointer.scroll", "accessibility.element_tokens"],
"move_cursor" => &[
- // Visual overlay move, not a real OS pointer move on
- // macOS/Windows — see SKILL.md. Surfaced as
- // `agent_cursor.move` because the canonical name on the
- // overlay side is "agent cursor"; `input.pointer.move` is
- // intentionally omitted to avoid claiming we shift the
- // real cursor.
+ // Window scope moves the agent overlay; explicit desktop scope
+ // moves the real OS pointer.
"agent_cursor.move",
+ "input.pointer.move",
],
// ── input.keyboard ───────────────────────────────────────────
@@ -237,7 +236,9 @@ pub fn default_capabilities_for(tool_name: &str) -> Vec<String> {
"set_config" => &["system.config.write"],
// ── sessions ─────────────────────────────────────────────────
- "start_session" => &["session.lifecycle.start"],
+ "start_session" => &["session.lifecycle.start", "session.capture_scope"],
+ "escalate_session" => &["session.capture_scope.escalate"],
+ "get_session_state" => &["session.capture_scope.read"],
"end_session" => &["session.lifecycle.end"],
// ── agent cursor ─────────────────────────────────────────────
@@ -327,8 +328,12 @@ impl ToolRegistry {
/// (`start_session` / `end_session`). Call alongside
/// `register_recording_tools` from each platform's `register_all`.
pub fn register_session_tools(&mut self) {
- use crate::session_tools::{EndSessionTool, StartSessionTool};
+ use crate::session_tools::{
+ EndSessionTool, EscalateSessionTool, GetSessionStateTool, StartSessionTool,
+ };
self.register(Box::new(StartSessionTool));
+ self.register(Box::new(EscalateSessionTool));
+ self.register(Box::new(GetSessionStateTool));
self.register(Box::new(EndSessionTool));
}
@@ -385,9 +390,6 @@ impl ToolRegistry {
/// Invoke a tool by name and (if recording is enabled) write its result to disk.
pub async fn invoke(&self, name: &str, args: Value) -> ToolResult {
- // Capture start time for recording timestamps.
- let start_ms = now_ms();
-
// Deprecated alias: `type_text_chars` → `type_text`. Swift's
// ToolRegistry.swift keeps the same alias (with stderr warning) for
// backwards compatibility with hermes-agent builds that still emit
@@ -401,13 +403,24 @@ impl ToolRegistry {
other => other,
};
+ let Some(tool) = self.tools.get(resolved_name) else {
+ return ToolResult::error(format!("Unknown tool: {name}"));
+ };
+ // Reject modality violations before reserving a recording turn. A
+ // rejected action has no before/after evidence and must not leave a
+ // pending recorder entry behind.
+ if let Err(violation) = crate::capture_scope::enforce_tool(resolved_name, &args) {
+ let structured =
+ violation.as_json(args.get("session").and_then(Value::as_str).unwrap_or(""));
+ return ToolResult::error(violation.message).with_structured(structured);
+ }
+
+ // Capture start time for recording timestamps only after validation.
+ let start_ms = now_ms();
+
// Reserve and capture the turn before dispatch so recorded evidence
// shows the application immediately before the action changed it.
- let should_record = self
- .tools
- .get(resolved_name)
- .map(|tool| !tool.def().read_only)
- .unwrap_or(false)
+ let should_record = !tool.def().read_only
&& !matches!(
resolved_name,
"start_recording" | "stop_recording" | "get_recording_state" | "replay_trajectory"
@@ -426,10 +439,7 @@ impl ToolRegistry {
})
.flatten();
- let result = match self.tools.get(resolved_name) {
- Some(tool) => tool.invoke(args.clone()).await,
- None => return ToolResult::error(format!("Unknown tool: {name}")),
- };
+ let result = tool.invoke(args.clone()).await;
// Use the original name for downstream code paths below so the
// exit-code matching and recording paths keep treating the alias
// as a distinct call site.
@@ -719,6 +729,8 @@ mod capability_tests {
"set_config",
// sessions
"start_session",
+ "escalate_session",
+ "get_session_state",
"end_session",
// agent cursor
"set_agent_cursor_enabled",
@@ -797,6 +809,9 @@ mod capability_tests {
// sessions
"session.lifecycle.start",
"session.lifecycle.end",
+ "session.capture_scope",
+ "session.capture_scope.read",
+ "session.capture_scope.escalate",
// agent cursor
"agent_cursor.move",
"agent_cursor.set_enabled",
diff --git a/libs/cua-driver/rust/crates/cua-driver/src/cli.rs b/libs/cua-driver/rust/crates/cua-driver/src/cli.rs
index c0a8498eb2..a6225f7653 100644
--- a/libs/cua-driver/rust/crates/cua-driver/src/cli.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/src/cli.rs
@@ -3314,6 +3314,12 @@ pub fn run_config_cmd(
process::exit(64);
}
};
+ if key == "capture_scope" {
+ eprintln!(
+ "config key 'capture_scope' is retired; use start_session(capture_scope=auto|window|desktop)"
+ );
+ process::exit(64);
+ }
// Try daemon first.
if crate::serve::is_daemon_listening(&socket_path) {
let req = crate::serve::DaemonRequest {
@@ -3353,7 +3359,9 @@ pub fn run_config_cmd(
(&mut sc, file_cfg)
{
for (k, v) in file_map {
- sc_map.insert(k, v);
+ if k != "capture_scope" {
+ sc_map.insert(k, v);
+ }
}
}
// Support dotted key paths like "agent_cursor.enabled".
@@ -3388,6 +3396,12 @@ pub fn run_config_cmd(
process::exit(64);
}
};
+ if key == "capture_scope" {
+ eprintln!(
+ "config key 'capture_scope' is retired; use start_session(capture_scope=auto|window|desktop)"
+ );
+ process::exit(64);
+ }
let value = match value {
Some(v) => v,
None => {
diff --git a/libs/cua-driver/rust/crates/cua-driver/src/serve.rs b/libs/cua-driver/rust/crates/cua-driver/src/serve.rs
index 45552e7bfd..90064a9b53 100644
--- a/libs/cua-driver/rust/crates/cua-driver/src/serve.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/src/serve.rs
@@ -1753,7 +1753,9 @@ mod gate_tests {
let start = DaemonRequest {
method: "call".into(),
name: Some("start_session".into()),
- args: Some(serde_json::json!({})),
+ // Session policy is caller-declared. The transport session id is
+ // cleanup/ownership metadata and must not grant policy authority.
+ args: Some(serde_json::json!({ "session": s3b.clone() })),
session_id: Some(s3b),
observation_origin: None,
};
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/README.md b/libs/cua-driver/rust/crates/cua-driver/tests/README.md
index 672b0ef54d..031bf26333 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/README.md
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/README.md
@@ -17,6 +17,7 @@ action, addressing mode, delivery mode, or OS-specific window-system case.
| Prefix | Runs by default | Purpose |
| --- | --- | --- |
| `protocol_*_test.rs` | yes | MCP/CLI protocol and schema behavior |
+| `session_capture_scope_test.rs` | yes | Per-session policy isolation, escalation, lifecycle, and retired config key |
| `schema_*_test.rs` | yes | Generated schema consistency |
| `harness_<toolkit>_test.rs` | no, `#[ignore]` | Toolkit-specific harness apps |
| `desktop_scope_<os>_test.rs` | no, `#[ignore]` | Platform window/desktop scope contract |
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_linux_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_linux_test.rs
index a7c887f167..ef05567943 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_linux_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_linux_test.rs
@@ -1,6 +1,6 @@
//! Linux **desktop-scope** (vision/foreground) modality through the SAME
-//! cua-driver interface as Windows/macOS: `set_config capture_scope=desktop` +
-//! a window-less screen-absolute `click` (no `pid`/`window_id`/`list_windows`).
+//! cua-driver interface as Windows/macOS: a strict desktop session plus a
+//! window-less screen-absolute `click` (no `pid`/`window_id`/`list_windows`).
//! The Linux actuator warps the pointer and injects a real button press via the
//! XTest extension — the peer of the Windows `WindowFromPoint` + macOS
//! global-HID desktop click. (XTest delivering to the under-pointer window is
@@ -11,16 +11,14 @@
//! `frame` (AT-SPI Component extents), asserts the counter advanced, plus the
//! window-scope rejection gate.
//!
-//! Linux config is global-only (no per-session override), so `set_config`
-//! capture_scope writes the on-disk default — the test resets it to `window`
-//! before asserting so a failure can't leave the sandbox in desktop scope.
+//! A separate strict-window session observes the fixture, proving two live
+//! sessions can use different capture scopes without shared mutable config.
//!
//! #[ignore] (needs an X11/Xwayland display + AT-SPI + the GTK3 harness). Run:
//! cargo test -p cua-driver --test desktop_scope_linux_test -- --ignored --nocapture --test-threads=1
#![cfg(target_os = "linux")]
-use std::panic::{self, AssertUnwindSafe};
use std::process::{Command, Stdio};
use std::time::{Duration, Instant};
@@ -87,11 +85,11 @@ fn launch(driver: &mut McpDriver) -> Option<(u32, u64)> {
None
}
-fn ax_snapshot(driver: &mut McpDriver, pid: u32, wid: u64) -> serde_json::Value {
+fn ax_snapshot(driver: &mut McpDriver, session: &str, pid: u32, wid: u64) -> serde_json::Value {
driver
.call(
"get_window_state",
- serde_json::json!({ "pid": pid as i64, "window_id": wid, "capture_mode": "ax" }),
+ serde_json::json!({ "session": session, "pid": pid as i64, "window_id": wid, "capture_mode": "ax" }),
)
.structured()
.clone()
@@ -129,28 +127,18 @@ fn counter(snap: &serde_json::Value) -> Option<u64> {
digits.parse().ok()
}
-fn set_scope(driver: &mut McpDriver, scope: &str) {
+fn start_scope(driver: &mut McpDriver, session: &str, scope: &str) {
let r = driver.call(
- "set_config",
- serde_json::json!({ "key": "capture_scope", "value": scope }),
+ "start_session",
+ serde_json::json!({ "session": session, "capture_scope": scope }),
);
assert!(
!r.is_error(),
- "set_config capture_scope={scope} failed: {}",
+ "start_session capture_scope={scope} failed: {}",
r.text()
);
}
-fn with_desktop_scope<R>(driver: &mut McpDriver, test: impl FnOnce(&mut McpDriver) -> R) -> R {
- set_scope(driver, "desktop");
- let result = panic::catch_unwind(AssertUnwindSafe(|| test(driver)));
- set_scope(driver, "window");
- match result {
- Ok(value) => value,
- Err(payload) => panic::resume_unwind(payload),
- }
-}
-
// ── tests ───────────────────────────────────────────────────────────────────────
/// In desktop scope, a window-less screen-absolute click (no pid/window_id)
@@ -177,15 +165,19 @@ fn desktop_scope_windowless_click_lands_on_control() {
execute_case(case, |evidence| {
let mut driver = McpDriver::spawn_named(cell_id).expect("start source-built Linux driver");
*evidence = recording_evidence(driver.recording_dir());
+ let window_session = format!("{cell_id}-window");
+ let desktop_session = format!("{cell_id}-desktop");
+ start_scope(&mut driver, &window_session, "window");
+ start_scope(&mut driver, &desktop_session, "desktop");
let (pid, wid) = launch(&mut driver).expect("required GTK3 harness did not launch");
// Settle for the AT-SPI tree to register the button + its extents.
- let mut snap = ax_snapshot(&mut driver, pid, wid);
+ let mut snap = ax_snapshot(&mut driver, &window_session, pid, wid);
let mut center = increment_center(&snap);
let deadline = Instant::now() + Duration::from_secs(8);
while center.is_none() && Instant::now() < deadline {
std::thread::sleep(Duration::from_millis(400));
- snap = ax_snapshot(&mut driver, pid, wid);
+ snap = ax_snapshot(&mut driver, &window_session, pid, wid);
center = increment_center(&snap);
}
let Some((cx, cy)) = center else {
@@ -195,7 +187,7 @@ fn desktop_scope_windowless_click_lands_on_control() {
println!("[desktop-linux] increment button screen-center=({cx},{cy}) pre-counter={pre}");
let posture = driver.call(
"bring_to_front",
- serde_json::json!({"pid": pid as i64, "window_id": wid}),
+ serde_json::json!({"session": window_session, "pid": pid as i64, "window_id": wid}),
);
assert!(!posture.is_error(), "could not foreground GTK3 fixture: {}", posture.text());
std::thread::sleep(Duration::from_millis(300));
@@ -209,11 +201,13 @@ fn desktop_scope_windowless_click_lands_on_control() {
// counter further, and `post > pre` still holds. We assert the click was
// *dispatched as desktop scope* on the first attempt, and that it eventually
// *lands* within the budget.
- let (post, first_text) = with_desktop_scope(&mut driver, |driver| {
+ let (post, first_text) = {
let mut post = pre;
let mut first_text = String::new();
for attempt in 0..12 {
- let clicked = driver.call("click", serde_json::json!({ "x": cx, "y": cy }));
+ let clicked = driver.call("click", serde_json::json!({
+ "session": desktop_session, "scope": "desktop", "x": cx, "y": cy
+ }));
if attempt == 0 {
first_text = clicked.text().to_string();
assert!(
@@ -223,13 +217,13 @@ fn desktop_scope_windowless_click_lands_on_control() {
);
}
std::thread::sleep(Duration::from_millis(500));
- post = counter(&ax_snapshot(driver, pid, wid)).unwrap_or(pre);
+ post = counter(&ax_snapshot(&mut driver, &window_session, pid, wid)).unwrap_or(pre);
if post > pre {
break;
}
}
(post, first_text)
- });
+ };
assert!(
first_text.to_lowercase().contains("desktop scope"),
@@ -263,9 +257,11 @@ fn window_scope_rejects_windowless_click() {
execute_case(case, |evidence| {
let mut driver = McpDriver::spawn_named(cell_id).expect("start source-built Linux driver");
*evidence = recording_evidence(driver.recording_dir());
- set_scope(&mut driver, "window");
+ start_scope(&mut driver, cell_id, "window");
driver.start_behavior_recording();
- let r = driver.call("click", serde_json::json!({ "x": 100, "y": 100 }));
+ let r = driver.call("click", serde_json::json!({
+ "session": cell_id, "scope": "desktop", "x": 100, "y": 100
+ }));
assert!(
r.is_error()
&& r.structured()["code"].as_str() == Some("desktop_scope_disabled"),
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_macos_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_macos_test.rs
index 9cf4a123fd..e508492937 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_macos_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_macos_test.rs
@@ -97,11 +97,11 @@ fn launch(driver: &mut McpDriver) -> Option<(u32, u64)> {
None
}
-fn ax_snapshot(driver: &mut McpDriver, pid: u32, wid: u64) -> serde_json::Value {
+fn ax_snapshot(driver: &mut McpDriver, session: &str, pid: u32, wid: u64) -> serde_json::Value {
driver
.call(
"get_window_state",
- serde_json::json!({ "pid": pid as i64, "window_id": wid, "capture_mode": "ax" }),
+ serde_json::json!({ "session": session, "pid": pid as i64, "window_id": wid, "capture_mode": "ax" }),
)
.structured()
.clone()
@@ -185,15 +185,27 @@ fn desktop_scope_windowless_click_lands_on_control() {
let mut driver = McpDriver::spawn_macos_daemon_proxy_named(cell_id)
.expect("start installed macOS daemon proxy");
*evidence = recording_evidence(driver.recording_dir());
+ let window_session = format!("{cell_id}-window");
+ let desktop_session = format!("{cell_id}-desktop");
+ for (session, capture_scope) in [
+ (window_session.as_str(), "window"),
+ (desktop_session.as_str(), "desktop"),
+ ] {
+ let started = driver.call(
+ "start_session",
+ serde_json::json!({"session": session, "capture_scope": capture_scope}),
+ );
+ assert!(!started.is_error(), "start_session failed: {}", started.text());
+ }
let (pid, wid) = launch(&mut driver).expect("required AppKit harness did not launch");
// Settle for the AppKit AX tree to register the button + its frame.
- let mut snap = ax_snapshot(&mut driver, pid, wid);
+ let mut snap = ax_snapshot(&mut driver, &window_session, pid, wid);
let mut center = increment_center(&snap);
let deadline = Instant::now() + Duration::from_secs(8);
while center.is_none() && Instant::now() < deadline {
std::thread::sleep(Duration::from_millis(400));
- snap = ax_snapshot(&mut driver, pid, wid);
+ snap = ax_snapshot(&mut driver, &window_session, pid, wid);
center = increment_center(&snap);
}
let Some((cx, cy)) = center else {
@@ -209,7 +221,7 @@ fn desktop_scope_windowless_click_lands_on_control() {
// Window-less screen-absolute click — no pid, no window_id; scope per-call.
let clicked = driver.call(
"click",
- serde_json::json!({ "x": cx, "y": cy, "scope": "desktop" }),
+ serde_json::json!({ "session": desktop_session, "x": cx, "y": cy, "scope": "desktop" }),
);
assert!(
!clicked.is_error(),
@@ -224,7 +236,7 @@ fn desktop_scope_windowless_click_lands_on_control() {
println!("[desktop-mac] {}", clicked.text());
std::thread::sleep(Duration::from_millis(600));
- let post = counter(&ax_snapshot(&mut driver, pid, wid)).unwrap_or(pre);
+ let post = counter(&ax_snapshot(&mut driver, &window_session, pid, wid)).unwrap_or(pre);
assert!(
post > pre,
"desktop click did not advance AppKit counter at ({cx},{cy}): {pre} -> {post}"
@@ -255,11 +267,15 @@ fn window_scope_rejects_windowless_click() {
let mut driver = McpDriver::spawn_macos_daemon_proxy_named(cell_id)
.expect("start installed macOS daemon proxy");
*evidence = recording_evidence(driver.recording_dir());
- // Default scope is "window" — a window-less click must be rejected.
+ let started = driver.call(
+ "start_session",
+ serde_json::json!({"session": cell_id, "capture_scope": "window"}),
+ );
+ assert!(!started.is_error(), "start_session failed: {}", started.text());
driver.start_behavior_recording();
let r = driver.call(
"click",
- serde_json::json!({ "x": 100, "y": 100, "scope": "window" }),
+ serde_json::json!({ "session": cell_id, "x": 100, "y": 100, "scope": "desktop" }),
);
assert!(
r.is_error() && r.structured()["code"].as_str() == Some("desktop_scope_disabled"),
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_windows_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_windows_test.rs
index 07d9d7de9f..b119e3de7f 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_windows_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_windows_test.rs
@@ -6,16 +6,15 @@
//! cover. This test exercises the Windows Phase-1 actuator end-to-end against a
//! real harness app:
//!
-//! 1. `set_config capture_scope=desktop` → `get_desktop_state` returns a
+//! 1. A strict desktop session → `get_desktop_state` returns a
//! full-display capture with true `screen_width/height` (no downscale).
//! 2. A **window-less** screen-absolute `click` / `scroll` (no pid/window_id)
//! lands via `WindowFromPoint` while in desktop scope.
//! 3. Negative gate: the same window-less `click` under `capture_scope=window`
//! is rejected with the structured `desktop_scope_disabled` error.
//!
-//! Note: `set_config` is a *session* override — it persists for the lifetime of
-//! the one MCP server we spawn here (not across separate `cua-driver call`
-//! processes), which is exactly why this test drives a single long-lived server.
+//! A separate strict-window session observes the WPF fixture. Keeping both live
+//! proves capture policy is isolated per session rather than process-global.
//!
//! All tests are `#[ignore]` (need a real desktop session). Run explicitly:
//! cargo test -p cua-driver --test desktop_scope_windows_test -- --ignored --nocapture --test-threads=1
@@ -88,10 +87,15 @@ fn launch_wpf(driver: &mut McpDriver) -> Option<(u32, u64)> {
None
}
-fn snapshot(driver: &mut McpDriver, pid: u32, wid: u64) -> cua_driver_testkit::ToolResponse {
+fn snapshot(
+ driver: &mut McpDriver,
+ session: &str,
+ pid: u32,
+ wid: u64,
+) -> cua_driver_testkit::ToolResponse {
driver.call(
"get_window_state",
- serde_json::json!({ "pid": pid as i64, "window_id": wid, "capture_mode": "ax" }),
+ serde_json::json!({ "session": session, "pid": pid as i64, "window_id": wid, "capture_mode": "ax" }),
)
}
@@ -115,20 +119,20 @@ fn element_center(state: &cua_driver_testkit::ToolResponse, id: &str) -> (i64, i
)
}
-fn set_scope(driver: &mut McpDriver, scope: &str) {
+fn start_scope(driver: &mut McpDriver, session: &str, scope: &str) {
let r = driver.call(
- "set_config",
- serde_json::json!({ "key": "capture_scope", "value": scope }),
+ "start_session",
+ serde_json::json!({ "session": session, "capture_scope": scope }),
);
assert!(
!r.is_error(),
- "set_config capture_scope={scope} failed: {}",
+ "start_session capture_scope={scope} failed: {}",
r.text()
);
assert_eq!(
r.structured()["capture_scope"].as_str(),
Some(scope),
- "set_config did not report capture_scope={scope}: {}",
+ "start_session did not report capture_scope={scope}: {}",
r.text()
);
}
@@ -136,7 +140,7 @@ fn set_scope(driver: &mut McpDriver, scope: &str) {
fn run_desktop_fixture_case(
action: &str,
route: DriverRoute,
- test: impl FnOnce(u32, u64, &mut McpDriver),
+ test: impl FnOnce(u32, u64, &str, &str, &mut McpDriver),
) {
let cell_id = format!("windows-wpf-desktop-{action}-px-foreground").replace('_', "-");
let case = CaseSpec::delivered(
@@ -154,16 +158,19 @@ fn run_desktop_fixture_case(
let mut driver =
McpDriver::spawn_named(&cell_id).expect("required source-built driver did not start");
*evidence = recording_evidence(driver.recording_dir());
+ let window_session = format!("{cell_id}-window");
+ let desktop_session = format!("{cell_id}-desktop");
+ start_scope(&mut driver, &window_session, "window");
+ start_scope(&mut driver, &desktop_session, "desktop");
let (pid, wid) = launch_wpf(&mut driver).expect("required WPF harness did not launch");
- set_scope(&mut driver, "desktop");
let posture = driver.call(
"bring_to_front",
- serde_json::json!({"pid": pid as i64, "window_id": wid}),
+ serde_json::json!({"session": window_session, "pid": pid as i64, "window_id": wid}),
);
assert!(!posture.is_error(), "could not foreground WPF fixture: {}", posture.text());
std::thread::sleep(Duration::from_millis(300));
driver.start_behavior_recording();
- test(pid, wid, &mut driver);
+ test(pid, wid, &window_session, &desktop_session, &mut driver);
Observation::delivered_with_fixture_state(Vec::new())
});
}
@@ -191,9 +198,13 @@ fn desktop_scope_capture_returns_screen_dims() {
let mut driver = McpDriver::spawn_named("windows-desktop-state-px-not-applicable")
.expect("required source-built driver did not start");
*evidence = recording_evidence(driver.recording_dir());
- set_scope(&mut driver, "desktop");
+ let session = "windows-desktop-state-px-not-applicable-desktop";
+ start_scope(&mut driver, session, "desktop");
driver.start_behavior_recording();
- let response = driver.call("get_desktop_state", serde_json::json!({}));
+ let response = driver.call(
+ "get_desktop_state",
+ serde_json::json!({"session": session}),
+ );
assert!(
!response.is_error(),
"get_desktop_state errored: {}",
@@ -217,17 +228,19 @@ fn desktop_scope_windowless_click_lands_on_control() {
run_desktop_fixture_case(
"left_click",
DriverRoute::WindowsSendInput,
- |pid, wid, driver| {
- let pre = snapshot(driver, pid, wid);
+ |pid, wid, window_session, desktop_session, driver| {
+ let pre = snapshot(driver, window_session, pid, wid);
let (x, y) = element_center(&pre, "border-click-target");
- let response = driver.call("click", serde_json::json!({ "x": x, "y": y }));
+ let response = driver.call("click", serde_json::json!({
+ "session": desktop_session, "scope": "desktop", "x": x, "y": y
+ }));
assert!(
!response.is_error(),
"desktop click failed: {}",
response.text()
);
std::thread::sleep(Duration::from_millis(400));
- let after = snapshot(driver, pid, wid);
+ let after = snapshot(driver, window_session, pid, wid);
assert!(
after.tree_text().contains("last_action=left_click"),
"desktop click did not update the WPF click oracle: {}",
@@ -243,8 +256,8 @@ fn desktop_scope_windowless_scroll_lands_on_control() {
run_desktop_fixture_case(
"scroll",
DriverRoute::WindowsSendInput,
- |pid, wid, driver| {
- let pre = snapshot(driver, pid, wid);
+ |pid, wid, window_session, desktop_session, driver| {
+ let pre = snapshot(driver, window_session, pid, wid);
// The fixture's outer ScrollViewer is visible while the nested
// scroll-tall region begins below a 768px CI desktop. Wheel over a
// visible child and verify the outer viewport moved by observing a
@@ -253,7 +266,10 @@ fn desktop_scope_windowless_scroll_lands_on_control() {
let (_, before_y) = element_center(&pre, "btn-increment");
let response = driver.call(
"scroll",
- serde_json::json!({ "x": x, "y": y, "direction": "down", "amount": 5 }),
+ serde_json::json!({
+ "session": desktop_session, "scope": "desktop", "x": x, "y": y,
+ "direction": "down", "amount": 5
+ }),
);
assert!(
!response.is_error(),
@@ -261,7 +277,7 @@ fn desktop_scope_windowless_scroll_lands_on_control() {
response.text()
);
std::thread::sleep(Duration::from_millis(500));
- let after = snapshot(driver, pid, wid);
+ let after = snapshot(driver, window_session, pid, wid);
let (_, after_y) = element_center(&after, "btn-increment");
assert!(
after_y < before_y,
@@ -291,9 +307,12 @@ fn window_scope_rejects_windowless_click() {
let mut driver = McpDriver::spawn_named("windows-window-scope-gate-px-not-applicable")
.expect("required source-built driver did not start");
*evidence = recording_evidence(driver.recording_dir());
- set_scope(&mut driver, "window");
+ let session = "windows-window-scope-gate-px-not-applicable";
+ start_scope(&mut driver, session, "window");
driver.start_behavior_recording();
- let response = driver.call("click", serde_json::json!({ "x": 100, "y": 100 }));
+ let response = driver.call("click", serde_json::json!({
+ "session": session, "scope": "desktop", "x": 100, "y": 100
+ }));
assert!(
response.is_error()
&& response.structured()["code"].as_str() == Some("desktop_scope_disabled"),
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/protocol_schema_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/protocol_schema_test.rs
index 18533635bb..abc14b49db 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/protocol_schema_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/protocol_schema_test.rs
@@ -3,7 +3,7 @@
//! These never invoke a tool — they only inspect the advertised inputSchemas:
//! that `type_text_chars` is hidden, the `list_windows.on_screen_only` knob, the
//! `set_agent_cursor_motion` Bezier knobs, delivery and scope enums, and the
-//! `set_config.capture_mode` enum.
+//! `set_config.capture_mode` enum and the per-session capture-scope contract.
#![cfg(any(target_os = "linux", target_os = "macos", target_os = "windows"))]
@@ -120,22 +120,35 @@ fn tools_list_schema_shape() {
"{tool}.delivery_mode should advertise background and foreground: {delivery:?}"
);
}
- // macOS selects desktop coordinates per click. Linux and Windows retain
- // the process-level capture_scope gate used by their desktop-state tools.
- #[cfg(target_os = "macos")]
- {
- let scope = &properties("click")["scope"];
+ // Capture scope belongs to the session lifecycle on every platform. The
+ // action-level `scope` selects a coordinate/transport form but cannot
+ // override the session policy enforced by the registry.
+ let capture_scope = &properties("start_session")["capture_scope"];
+ for expected in ["auto", "window", "desktop"] {
assert!(
- enum_contains(scope, "window") && enum_contains(scope, "desktop"),
- "click.scope should advertise window and desktop: {scope:?}"
+ enum_contains(capture_scope, expected),
+ "start_session.capture_scope should advertise {expected}: {capture_scope:?}"
);
}
- #[cfg(any(target_os = "linux", target_os = "windows"))]
- {
- let capture_scope = &properties("set_config")["capture_scope"];
+ assert!(
+ tools.iter().any(|tool| tool["name"] == "escalate_session"),
+ "escalate_session missing from tools/list"
+ );
+ assert!(
+ tools.iter().any(|tool| tool["name"] == "get_session_state"),
+ "get_session_state missing from tools/list"
+ );
+ assert!(
+ properties("set_config")["capture_scope"].is_null(),
+ "persistent set_config.capture_scope must remain retired"
+ );
+ for action in [
+ "click", "type_text", "press_key", "hotkey", "scroll", "drag", "move_cursor",
+ ] {
+ let scope = &properties(action)["scope"];
assert!(
- enum_contains(capture_scope, "window") && enum_contains(capture_scope, "desktop"),
- "set_config.capture_scope should advertise window and desktop: {capture_scope:?}"
+ enum_contains(scope, "window") && enum_contains(scope, "desktop"),
+ "{action}.scope should advertise window and desktop: {scope:?}"
);
}
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/protocol_tools_call_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/protocol_tools_call_test.rs
index c20e73955d..1bba2c470f 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/protocol_tools_call_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/protocol_tools_call_test.rs
@@ -85,8 +85,8 @@ fn get_config_and_check_permissions() {
}
// set_config — change max_image_dimension and verify get_config reflects it.
- // capture_mode / capture_scope are no longer settings on macOS (per-call
- // params now — modality-ladder refactor); max_image_dimension is the
+ // capture_mode / capture_scope are no longer persistent settings on macOS
+ // (capture_scope is per-session); max_image_dimension is the
// cross-platform persisted field this exercises.
d.send(&serde_json::json!({
"jsonrpc":"2.0","id":4,"method":"tools/call",
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/session_capture_scope_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/session_capture_scope_test.rs
new file mode 100644
index 0000000000..1feff78cff
--- /dev/null
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/session_capture_scope_test.rs
@@ -0,0 +1,154 @@
+//! Transport-level contract tests for per-session capture scope.
+
+#![cfg(any(target_os = "linux", target_os = "macos", target_os = "windows"))]
+
+use cua_driver_testkit::RawDriver;
+use serde_json::{json, Value};
+
+fn call(driver: &mut RawDriver, id: u64, name: &str, arguments: Value) -> Value {
+ driver.send(&json!({
+ "jsonrpc": "2.0",
+ "id": id,
+ "method": "tools/call",
+ "params": {"name": name, "arguments": arguments}
+ }));
+ driver.recv()
+}
+
+fn code(response: &Value) -> Option<&str> {
+ response["result"]["structuredContent"]["code"].as_str()
+}
+
+#[test]
+fn policies_are_isolated_immutable_and_enforced_over_mcp() {
+ let mut driver = RawDriver::spawn().expect("spawn source-built driver");
+ driver.send(&json!({"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}));
+ driver.recv();
+
+ for (id, session, scope) in [
+ (2, "scope-auto", "auto"),
+ (3, "scope-window", "window"),
+ (4, "scope-desktop", "desktop"),
+ ] {
+ let response = call(
+ &mut driver,
+ id,
+ "start_session",
+ json!({"session": session, "capture_scope": scope}),
+ );
+ assert_eq!(
+ response["result"]["structuredContent"]["capture_scope"],
+ scope,
+ "start_session failed: {response}"
+ );
+ }
+
+ let auto_desktop = call(
+ &mut driver,
+ 5,
+ "get_desktop_state",
+ json!({"session": "scope-auto"}),
+ );
+ assert_eq!(code(&auto_desktop), Some("desktop_escalation_required"));
+
+ let window_desktop = call(
+ &mut driver,
+ 6,
+ "get_desktop_state",
+ json!({"session": "scope-window"}),
+ );
+ assert_eq!(code(&window_desktop), Some("desktop_scope_disabled"));
+
+ let window_screen_size = call(
+ &mut driver,
+ 60,
+ "get_screen_size",
+ json!({"session": "scope-window"}),
+ );
+ assert_eq!(code(&window_screen_size), Some("desktop_scope_disabled"));
+ let desktop_screen_size = call(
+ &mut driver,
+ 61,
+ "get_screen_size",
+ json!({"session": "scope-desktop"}),
+ );
+ assert!(
+ code(&desktop_screen_size).is_none(),
+ "desktop scope should permit global screen geometry: {desktop_screen_size}"
+ );
+
+ let desktop_window = call(
+ &mut driver,
+ 7,
+ "get_window_state",
+ json!({"session": "scope-desktop"}),
+ );
+ assert_eq!(code(&desktop_window), Some("window_scope_disabled"));
+
+ let escalated = call(
+ &mut driver,
+ 8,
+ "escalate_session",
+ json!({
+ "session": "scope-auto",
+ "reason": "foreground_ineffective",
+ "detail": "window ladder exhausted"
+ }),
+ );
+ assert_eq!(
+ escalated["result"]["structuredContent"]["effective_scope"],
+ "desktop"
+ );
+ let auto_window = call(
+ &mut driver,
+ 9,
+ "get_window_state",
+ json!({"session": "scope-auto"}),
+ );
+ assert_eq!(code(&auto_window), Some("window_scope_disabled"));
+
+ let conflict = call(
+ &mut driver,
+ 10,
+ "start_session",
+ json!({"session": "scope-window", "capture_scope": "desktop"}),
+ );
+ assert_eq!(code(&conflict), Some("session_policy_conflict"));
+
+ let ended = call(
+ &mut driver,
+ 11,
+ "end_session",
+ json!({"session": "scope-window"}),
+ );
+ assert_eq!(ended["result"]["structuredContent"]["active"], false);
+ let revived = call(
+ &mut driver,
+ 12,
+ "start_session",
+ json!({"session": "scope-window", "capture_scope": "desktop"}),
+ );
+ assert_eq!(revived["result"]["structuredContent"]["revived"], true);
+ assert_eq!(
+ revived["result"]["structuredContent"]["capture_scope"],
+ "desktop"
+ );
+}
+
+#[test]
+fn persistent_capture_scope_key_is_retired() {
+ let mut driver = RawDriver::spawn().expect("spawn source-built driver");
+ driver.send(&json!({"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}));
+ driver.recv();
+ let response = call(
+ &mut driver,
+ 2,
+ "set_config",
+ json!({"key": "capture_scope", "value": "desktop"}),
+ );
+ assert_eq!(code(&response), Some("config_key_retired"));
+ assert_eq!(
+ response["result"]["structuredContent"]["replacement"],
+ "start_session.capture_scope"
+ );
+}
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/transport_config_persistence_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/transport_config_persistence_test.rs
index cdfd79c13b..dba07dfd53 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/transport_config_persistence_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/transport_config_persistence_test.rs
@@ -11,8 +11,8 @@
//! visible to later calls on the SAME driver within the session.
//!
//! Uses `max_image_dimension` as the persisted key. `capture_mode` /
-//! `capture_scope` are NO LONGER settings (they are per-call params now —
-//! the modality-ladder refactor), so `max_image_dimension` is the remaining
+//! `capture_scope` are NO LONGER settings (`capture_scope` is per-session), so
+//! `max_image_dimension` is the remaining
//! disk-persisted config field and the right probe for this transport behavior.
//!
//! Both tests are `#[ignore]`: they mutate the real on-disk config, so they
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs
index 1494da8fa2..8f2c823db8 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs
@@ -2052,6 +2052,40 @@ pub fn send_click_xtest_desktop(x: i32, y: i32, button: u8, count: usize) -> Res
Ok(())
}
+/// Move the real X11 pointer to a screen-absolute desktop coordinate.
+pub fn send_move_xtest_desktop(x: i32, y: i32) -> Result<()> {
+ use x11rb::protocol::xtest::ConnectionExt as _;
+ let (conn, screen_num) = connect_x11_for_input()?;
+ let root = conn.setup().roots[screen_num].root;
+ conn.xtest_fake_input(MOTION_NOTIFY_EVENT, 0, 0, root, x as i16, y as i16, 0)?;
+ conn.flush()?;
+ let _ = conn.get_input_focus()?.reply();
+ Ok(())
+}
+
+/// Scroll the window under a screen-absolute point via real XTest wheel-button
+/// events. X11 buttons 4/5 are vertical up/down and 6/7 horizontal left/right.
+pub fn send_scroll_xtest_desktop(x: i32, y: i32, direction: &str, amount: usize) -> Result<()> {
+ use x11rb::protocol::xtest::ConnectionExt as _;
+ let button = match direction {
+ "up" => 4,
+ "down" => 5,
+ "left" => 6,
+ "right" => 7,
+ other => anyhow::bail!("unknown desktop scroll direction: {other}"),
+ };
+ let (conn, screen_num) = connect_x11_for_input()?;
+ let root = conn.setup().roots[screen_num].root;
+ conn.xtest_fake_input(MOTION_NOTIFY_EVENT, 0, 0, root, x as i16, y as i16, 0)?;
+ for _ in 0..amount.max(1) {
+ conn.xtest_fake_input(BUTTON_PRESS_EVENT, button, 0, root, x as i16, y as i16, 0)?;
+ conn.xtest_fake_input(BUTTON_RELEASE_EVENT, button, 0, root, x as i16, y as i16, 0)?;
+ }
+ conn.flush()?;
+ let _ = conn.get_input_focus()?.reply();
+ Ok(())
+}
+
/// Screen-absolute drag via XTest. The caller activates the target first; XTest
/// then supplies one real press, interpolated pointer motion, and one release.
/// This is the foreground counterpart to the window-addressed XSendEvent drag.
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
index e325f4f484..49abf9805c 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
@@ -19,7 +19,6 @@ use cursor_overlay::CursorRegistry;
#[derive(Clone)]
pub struct DriverConfig {
pub capture_mode: String,
- pub capture_scope: String,
pub max_image_dimension: u32,
}
@@ -27,7 +26,6 @@ impl Default for DriverConfig {
fn default() -> Self {
Self {
capture_mode: "ax".into(),
- capture_scope: "window".into(),
max_image_dimension: 1568,
}
}
@@ -45,11 +43,6 @@ pub fn load_driver_config() -> DriverConfig {
{
cfg.capture_mode = v;
}
- if let Some(v) =
- pip_preview::read_config_value("capture_scope").and_then(|v| v.as_str().map(str::to_owned))
- {
- cfg.capture_scope = v;
- }
if let Some(v) = pip_preview::read_config_value("max_image_dimension").and_then(|v| v.as_u64())
{
if let Ok(v32) = u32::try_from(v) {
@@ -1750,6 +1743,7 @@ impl Tool for ClickTool {
"button":{"type":"string","enum":["left","right","middle"],"description":"Mouse button. Default: \"left\" (legacy back-compat). X11: routed via ButtonPress/Release with the matching evdev code. Native Wayland: only left-button is supported via the virtual-pointer protocol; right/middle return an error."},
"count":{"type":"integer"},
"from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."},
+ "scope":{"type":"string","enum":["window","desktop"],"default":"window"},
"delivery_mode": crate::input::delivery::delivery_mode_schema()
},"additionalProperties":false
}),
@@ -1762,28 +1756,23 @@ impl Tool for ClickTool {
// ── Window-less screen-absolute branch (capture_scope="desktop") ──────
// x,y with NO pid and NO window_id → TRUE SCREEN pixels. Foreground,
- // vision-driven desktop-scope click (the Linux peer of the Windows
- // WindowFromPoint / macOS global-HID path). Gate on the effective scope:
- // under "window" return the same `desktop_scope_disabled` contract.
+ // desktop-scope click (the Linux peer of the Windows WindowFromPoint /
+ // macOS global-HID path). The core registry has already enforced the
+ // session policy; this branch validates the explicit action form.
let has_pid = args.get("pid").map(|v| !v.is_null()).unwrap_or(false);
let has_window_id = args.get("window_id").map(|v| !v.is_null()).unwrap_or(false);
let has_xy = args.get("x").map(|v| v.is_number()).unwrap_or(false)
&& args.get("y").map(|v| v.is_number()).unwrap_or(false);
if has_xy && !has_pid && !has_window_id {
- // Linux config is global-only (no per-session override layer).
- let scope = self.state.config.read().unwrap().capture_scope.clone();
- if scope != "desktop" {
+ if args.get("scope").and_then(Value::as_str) != Some("desktop") {
return ToolResult::error(
- "click: x,y given with no pid/window_id, but capture_scope is \
- \"window\". Screen-absolute clicks require desktop scope. Call \
- set_config with capture_scope=desktop (and use get_desktop_state \
- to read true screen pixels) first."
+ "click: x,y given with no pid/window_id requires scope=\"desktop\"; \
+ use get_desktop_state to read true screen pixels first."
.to_string(),
)
.with_structured(json!({
- "code": "desktop_scope_disabled",
- "capture_scope": scope,
- "suggestion": "set_config capture_scope=desktop",
+ "code": "desktop_coordinate_scope_required",
+ "suggestion": "pass scope=desktop",
}));
}
let button_raw = args.str_or("button", "left").to_lowercase();
@@ -2218,7 +2207,7 @@ impl Tool for TypeTextTool {
name: "type_text".into(),
description: "Type text to a window via XSendEvent (KeyPress/KeyRelease). No focus steal.".into(),
input_schema: json!({
- "type":"object","required":["pid","text"],"properties":{
+ "type":"object","required":["text"],"properties":{
"session": cua_driver_core::tool_schema::session_schema(),
"pid":{"type":"integer"},
"window_id":{"type":"integer"},
@@ -2227,6 +2216,7 @@ impl Tool for TypeTextTool {
"element_token": cua_driver_core::tool_schema::element_token_schema(),
"x":{"type":"number","description":"Screenshot-pixel X of the field to type into — the element px action form. Pass x,y (no element_index) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the AX path can't reach. Read straight off the get_window_state PNG, same convention as click."},
"y":{"type":"number","description":"Screenshot-pixel Y of the field (see x)."},
+ "scope":{"type":"string","enum":["window","desktop"],"default":"window"},
"delivery_mode": crate::input::delivery::delivery_mode_schema()
},"additionalProperties":false
}),
@@ -2236,6 +2226,33 @@ impl Tool for TypeTextTool {
async fn invoke(&self, args: Value) -> ToolResult {
use cua_driver_core::tool_args::ArgsExt;
+ if args.opt_str("scope").as_deref() == Some("desktop") && args.get("pid").is_none() {
+ let text = match args.require_str("text") {
+ Ok(value) => {
+ cua_driver_core::text_sanitize::strip_trailing_agent_protocol_tags(&value)
+ .into_owned()
+ }
+ Err(error) => return error,
+ };
+ let wayland = crate::wayland::wayland_input_enabled();
+ let path = if wayland { "wayland_focused" } else { "xtest" };
+ let result = tokio::task::spawn_blocking(move || {
+ if wayland {
+ crate::wayland::type_text_focused(&text)
+ } else {
+ crate::input::send_type_text_xtest(&text)
+ }
+ })
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text("Typed text into the focused desktop application.")
+ .with_structured(
+ json!({"scope":"desktop","path":path,"effect":"unverifiable"}),
+ ),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
let pid = args.u64_or("pid", 0) as u32;
let text_raw = match args.require_str("text") {
Ok(v) => v,
@@ -2767,7 +2784,7 @@ impl Tool for PressKeyTool {
name: "press_key".into(),
description: "Press a key via XSendEvent to a window. No focus steal.".into(),
input_schema: json!({
- "type":"object","required":["pid","key"],"properties":{
+ "type":"object","required":["key"],"properties":{
"session": cua_driver_core::tool_schema::session_schema(),
"pid":{"type":"integer"},
"window_id":{"type":"integer"},
@@ -2777,6 +2794,7 @@ impl Tool for PressKeyTool {
"element_token": cua_driver_core::tool_schema::element_token_schema(),
"x":{"type":"number","description":"Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_index."},
"y":{"type":"number","description":"Screenshot-pixel Y (see x)."},
+ "scope":{"type":"string","enum":["window","desktop"],"default":"window"},
"delivery_mode": crate::input::delivery::delivery_mode_schema()
},"additionalProperties":false
}),
@@ -2786,6 +2804,37 @@ impl Tool for PressKeyTool {
async fn invoke(&self, args: Value) -> ToolResult {
use cua_driver_core::tool_args::ArgsExt;
+ if args.opt_str("scope").as_deref() == Some("desktop") && args.get("pid").is_none() {
+ let key = match args.require_str("key") {
+ Ok(value) => value,
+ Err(error) => return error,
+ };
+ let display = key.clone();
+ let modifiers: Vec<String> = args.str_array("modifiers");
+ let wayland = crate::wayland::wayland_input_enabled();
+ let path = if wayland { "wayland_focused" } else { "xtest" };
+ let result = tokio::task::spawn_blocking(move || {
+ if wayland && modifiers.is_empty() {
+ crate::wayland::press_key_focused(&key)
+ } else if wayland {
+ let mut keys = modifiers;
+ keys.push(key);
+ crate::wayland::hotkey_focused(&keys)
+ } else {
+ let refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
+ crate::input::send_key_xtest(&key, &refs)
+ }
+ })
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!("Pressed desktop key '{display}'."))
+ .with_structured(
+ json!({"scope":"desktop","path":path,"effect":"unverifiable"}),
+ ),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
let pid = args.u64_or("pid", 0) as u32;
let key = match args.require_str("key") {
Ok(v) => v,
@@ -3026,7 +3075,7 @@ impl Tool for HotkeyTool {
description: "Press a combination of keys simultaneously, e.g. [\"ctrl\",\"c\"] for Copy. \
Sent via XSendEvent directly to the target pid; target does NOT need to be frontmost.".into(),
input_schema: json!({
- "type":"object","required":["pid","keys"],"properties":{
+ "type":"object","required":["keys"],"properties":{
"session": cua_driver_core::tool_schema::session_schema(),
"pid":{"type":"integer"},
"window_id":{"type":"integer"},
@@ -3036,6 +3085,7 @@ impl Tool for HotkeyTool {
"element_token": cua_driver_core::tool_schema::element_token_schema(),
"x":{"type":"number","description":"Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the combo (so e.g. Ctrl+V pastes into that field). Pass with y. Use for Chromium/Electron surfaces the background combo can't reach."},
"y":{"type":"number","description":"Screenshot-pixel Y (see x)."},
+ "scope":{"type":"string","enum":["window","desktop"],"default":"window"},
"delivery_mode": crate::input::delivery::delivery_mode_schema()
},"additionalProperties":false
}),
@@ -3045,6 +3095,37 @@ impl Tool for HotkeyTool {
async fn invoke(&self, args: Value) -> ToolResult {
use cua_driver_core::tool_args::ArgsExt;
+ if args.opt_str("scope").as_deref() == Some("desktop") && args.get("pid").is_none() {
+ let keys: Vec<String> = args.str_array("keys");
+ let modifiers: Vec<String> = keys
+ .iter()
+ .filter(|key| is_modifier(key))
+ .cloned()
+ .collect();
+ let Some(key) = keys.iter().rev().find(|key| !is_modifier(key)).cloned() else {
+ return ToolResult::error("keys must include at least one non-modifier key.");
+ };
+ let display = keys.join("+");
+ let wayland = crate::wayland::wayland_input_enabled();
+ let path = if wayland { "wayland_focused" } else { "xtest" };
+ let result = tokio::task::spawn_blocking(move || {
+ if wayland {
+ crate::wayland::hotkey_focused(&keys)
+ } else {
+ let refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
+ crate::input::send_key_xtest(&key, &refs)
+ }
+ })
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!("Pressed desktop hotkey {display}."))
+ .with_structured(
+ json!({"scope":"desktop","path":path,"effect":"unverifiable"}),
+ ),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
let pid = args.u64_or("pid", 0) as u32;
let window_id_arg = args.opt_u64("window_id");
let element_index_arg = args.opt_u64("element_index").map(|value| value as usize);
@@ -3392,6 +3473,7 @@ impl Tool for ScrollTool {
"element_token": cua_driver_core::tool_schema::element_token_schema(),
"x":{"type":"number","description":"Window-local screenshot-pixel X of the scroll target. Pass with y and without element_index."},
"y":{"type":"number","description":"Window-local screenshot-pixel Y of the scroll target. Pass with x and without element_index."},
+ "scope":{"type":"string","enum":["window","desktop"],"default":"window"},
"delivery_mode": crate::input::delivery::delivery_mode_schema()
},"additionalProperties":false
}),
@@ -3401,6 +3483,43 @@ impl Tool for ScrollTool {
async fn invoke(&self, args: Value) -> ToolResult {
let cursor_id = resolve_cursor_key(&args);
+ if args.opt_str("scope").as_deref() == Some("desktop")
+ && args.get("pid").is_none()
+ && args.get("window_id").is_none()
+ {
+ let direction = match args.require_str("direction") {
+ Ok(value) => value,
+ Err(error) => return error,
+ };
+ let x = match args.require_f64("x") {
+ Ok(value) => value.round() as i32,
+ Err(error) => return error,
+ };
+ let y = match args.require_f64("y") {
+ Ok(value) => value.round() as i32,
+ Err(error) => return error,
+ };
+ let amount = args.u64_or("amount", 3).clamp(1, 50) as usize;
+ let display = direction.clone();
+ let wayland = crate::wayland::wayland_input_enabled();
+ let path = if wayland { "wayland_desktop" } else { "xtest" };
+ let result = tokio::task::spawn_blocking(move || {
+ if wayland {
+ crate::wayland::scroll_desktop(x, y, &direction, amount as u32)
+ } else {
+ crate::input::send_scroll_xtest_desktop(x, y, &direction, amount)
+ }
+ })
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!("Scrolled desktop {display} × {amount}."))
+ .with_structured(
+ json!({"scope":"desktop","path":path,"effect":"unverifiable"}),
+ ),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
let pid = match args.require_u32("pid") {
Ok(v) => v,
Err(e) => return e,
@@ -4199,7 +4318,7 @@ impl Tool for DragTool {
description: "Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in \
window-local screenshot pixels via XSendEvent (ButtonPress + MotionNotify × steps + ButtonRelease). \
duration_ms (default 500), steps (default 20). No focus steal.".into(),
- input_schema: json!({"type":"object","required":["pid","from_x","from_y","to_x","to_y"],"properties":{
+ input_schema: json!({"type":"object","required":["from_x","from_y","to_x","to_y"],"properties":{
"session": cua_driver_core::tool_schema::session_schema(),
"cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},
"pid":{"type":"integer"},
@@ -4213,6 +4332,7 @@ impl Tool for DragTool {
"modifier": cua_driver_core::tool_schema::modifier_schema(),
"button": cua_driver_core::tool_schema::button_schema(),
"from_zoom":{"type":"boolean"},
+ "scope":{"type":"string","enum":["window","desktop"],"default":"window"},
"delivery_mode": crate::input::delivery::delivery_mode_schema()
},"additionalProperties":false}),
read_only: false, destructive: true, idempotent: false, open_world: true,
@@ -4220,6 +4340,62 @@ impl Tool for DragTool {
}
async fn invoke(&self, args: Value) -> ToolResult {
let cursor_id = resolve_cursor_key(&args);
+ if args.opt_str("scope").as_deref() == Some("desktop")
+ && args.get("pid").is_none()
+ && args.get("window_id").is_none()
+ {
+ let coordinate = |key: &str| {
+ args.opt_f64(key)
+ .or_else(|| args.opt_i64(key).map(|value| value as f64))
+ };
+ let Some(from_x) = coordinate("from_x") else {
+ return ToolResult::error("Missing: from_x");
+ };
+ let Some(from_y) = coordinate("from_y") else {
+ return ToolResult::error("Missing: from_y");
+ };
+ let Some(to_x) = coordinate("to_x") else {
+ return ToolResult::error("Missing: to_x");
+ };
+ let Some(to_y) = coordinate("to_y") else {
+ return ToolResult::error("Missing: to_y");
+ };
+ let button = parse_mouse_button(&args.str_or("button", "left"));
+ let duration_ms = args.u64_or("duration_ms", 500).min(10_000);
+ let steps = args.u64_or("steps", 20).clamp(1, 200) as usize;
+ let wayland = crate::wayland::wayland_input_enabled();
+ let path = if wayland { "wayland_desktop" } else { "xtest" };
+ let result = tokio::task::spawn_blocking(move || {
+ if wayland {
+ crate::wayland::drag_desktop(
+ from_x.round() as i32,
+ from_y.round() as i32,
+ to_x.round() as i32,
+ to_y.round() as i32,
+ steps as u32,
+ button,
+ )
+ } else {
+ crate::input::send_drag_xtest_desktop(
+ from_x.round() as i32,
+ from_y.round() as i32,
+ to_x.round() as i32,
+ to_y.round() as i32,
+ button,
+ duration_ms,
+ steps,
+ )
+ }
+ })
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text("Dragged on the desktop.").with_structured(
+ json!({"scope":"desktop","path":path,"effect":"unverifiable"}),
+ ),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
let pid = match args.require_u32("pid") {
Ok(v) => v,
Err(e) => return e,
@@ -5456,28 +5632,6 @@ impl Tool for GetDesktopStateTool {
}
async fn invoke(&self, args: Value) -> ToolResult {
- // Gate on the global capture_scope: a full-display capture is a
- // desktop-scope operation, available only when capture_scope="desktop"
- // (same gate as window-less screen-absolute click/scroll). Read the
- // persisted value the same way load_config does.
- let scope = pip_preview::read_config_value("capture_scope")
- .and_then(|v| v.as_str().map(str::to_owned))
- .unwrap_or_else(|| "window".to_owned());
- if scope != "desktop" {
- return ToolResult::error(format!(
- "get_desktop_state requires capture_scope=\"desktop\" (current scope is \
- \"{scope}\"). Full-display capture is a desktop-scope operation; call \
- set_config with capture_scope=desktop first (it also enables window-less \
- screen-absolute click/scroll). For a single window, use \
- get_window_state(pid, window_id) instead."
- ))
- .with_structured(serde_json::json!({
- "code": "desktop_scope_disabled",
- "capture_scope": scope,
- "suggestion": "set_config capture_scope=desktop",
- }));
- }
-
let out_file = args.opt_str("screenshot_out_file");
let result = tokio::task::spawn_blocking(move || -> anyhow::Result<_> {
@@ -5521,10 +5675,12 @@ impl Tool for GetDesktopStateTool {
let mut content = Vec::new();
let mut structured = json!({
"platform": "linux",
+ "display": "primary",
"screenshot_width": shot_w,
"screenshot_height": shot_h,
"screen_width": screen_w,
"screen_height": screen_h,
+ "scale_factor": 1.0,
"screenshot_mime_type": "image/png",
});
if let Some(b64) = b64_opt {
@@ -5621,9 +5777,9 @@ impl Tool for MoveCursorTool {
fn def(&self) -> &ToolDef {
MCURSOR_DEF.get_or_init(|| ToolDef {
name: "move_cursor".into(),
- description: "Move the agent cursor overlay to (x, y). Does NOT move the real mouse cursor.".into(),
+ description: "Move the agent cursor overlay, or with scope=desktop move the real OS pointer in get_desktop_state coordinates.".into(),
input_schema: json!({"type":"object","required":["x","y"],"properties":{
- "x":{"type":"number"},"y":{"type":"number"},"session": cua_driver_core::tool_schema::session_schema(),"cursor_id":{"type":"string"}
+ "x":{"type":"number"},"y":{"type":"number"},"session": cua_driver_core::tool_schema::session_schema(),"cursor_id":{"type":"string"},"scope":{"type":"string","enum":["window","desktop"],"default":"window"}
},"additionalProperties":false}),
read_only: false, destructive: false, idempotent: true, open_world: false,
})
@@ -5632,6 +5788,29 @@ impl Tool for MoveCursorTool {
use cua_driver_core::tool_args::ArgsExt;
let x = args.f64_or("x", 0.0);
let y = args.f64_or("y", 0.0);
+ if args.opt_str("scope").as_deref() == Some("desktop") {
+ let xi = x.round() as i32;
+ let yi = y.round() as i32;
+ let result = if crate::wayland::wayland_input_enabled() {
+ tokio::task::spawn_blocking(move || {
+ crate::wayland::move_cursor_absolute(None, xi, yi)
+ })
+ .await
+ } else {
+ tokio::task::spawn_blocking(move || crate::input::send_move_xtest_desktop(xi, yi))
+ .await
+ };
+ return match result {
+ Ok(Ok(())) => {
+ ToolResult::text(format!("Moved the real desktop pointer to ({xi}, {yi})."))
+ .with_structured(
+ json!({"scope":"desktop","x":xi,"y":yi,"effect":"unverifiable"}),
+ )
+ }
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("Task error: {error}")),
+ };
+ }
let window_id = args.get("window_id").and_then(|v| v.as_u64());
let cursor_id = resolve_cursor_key(&args);
self.state.cursor_registry.update_position(&cursor_id, x, y);
@@ -6172,7 +6351,6 @@ impl Tool for GetConfigTool {
"source_sha": option_env!("CUA_DRIVER_SOURCE_SHA"),
"platform": "linux",
"capture_mode": cfg.capture_mode,
- "capture_scope": cfg.capture_scope,
"max_image_dimension": cfg.max_image_dimension,
"experimental_pip": pip_enabled,
"experimental_pip_geometry": pip_geometry
@@ -6205,7 +6383,6 @@ impl Tool for SetConfigTool {
"key":{"type":"string","description":"Name of a single config field to write ({key, value} shape). Pair with `value`."},
"value":{"description":"New value for `key`. JSON type depends on the key."},
"capture_mode":{"type":"string","enum":["ax","vision"],"description":"Legacy per-field shape. Default capture mode for get_window_state. (\"som\"/\"screenshot\" still decode as deprecated aliases.)"},
- "capture_scope":{"type":"string","enum":["window","desktop"],"description":"Capture scope: single window or whole desktop. Default window."},
"max_image_dimension":{"type":"integer","description":"Legacy per-field shape. Max dimension for screenshot resizing (0 = no limit)."},
"experimental_pip":{"type":"boolean","description":"Enable the experimental PiP preview window (applies next restart; Linux backend stubbed)."},
"experimental_pip_geometry":{"type":"string","description":"PiP window size + optional position in `WxH` or `WxH+X+Y` form."}
@@ -6215,6 +6392,18 @@ impl Tool for SetConfigTool {
}
async fn invoke(&self, args: Value) -> ToolResult {
use cua_driver_core::tool_args::ArgsExt;
+ if args.get("capture_scope").is_some()
+ || args.get("key").and_then(Value::as_str) == Some("capture_scope")
+ {
+ return ToolResult::error(
+ "config key 'capture_scope' is retired; pass capture_scope=auto|window|desktop to start_session",
+ )
+ .with_structured(json!({
+ "code": "config_key_retired",
+ "key": "capture_scope",
+ "replacement": "start_session.capture_scope",
+ }));
+ }
let mut cfg = self.state.config.write().unwrap();
let mut parts = Vec::new();
// {key, value} shape (what the Swift/macOS and Windows callers send).
@@ -6235,17 +6424,6 @@ impl Tool for SetConfigTool {
}
None => return ToolResult::error(format!("`capture_mode` must be a string, got {val}.")),
},
- "capture_scope" => match val.as_str() {
- Some(s @ ("window" | "desktop")) => {
- cfg.capture_scope = s.to_owned();
- if let Err(e) = pip_preview::write_config_key("capture_scope", Value::String(s.to_owned())) {
- tracing::warn!("set_config: failed to persist capture_scope: {e}");
- }
- parts.push(format!("capture_scope={s}"));
- }
- Some(other) => return ToolResult::error(format!("`capture_scope` must be \"window\" or \"desktop\", got \"{other}\".")),
- None => return ToolResult::error(format!("`capture_scope` must be a string, got {val}.")),
- },
"max_image_dimension" => match val.as_u64() {
Some(n) => {
cfg.max_image_dimension = n as u32;
@@ -6280,7 +6458,7 @@ impl Tool for SetConfigTool {
None => return ToolResult::error(format!("`experimental_pip_geometry` must be a string, got {val}.")),
},
other => return ToolResult::error(format!(
- "Unknown config key `{other}`. Known: capture_mode, capture_scope, max_image_dimension, experimental_pip, experimental_pip_geometry."
+ "Unknown config key `{other}`. Known: capture_mode, max_image_dimension, experimental_pip, experimental_pip_geometry."
)),
}
}
@@ -6294,20 +6472,6 @@ impl Tool for SetConfigTool {
parts.push(format!("capture_mode={mode}"));
cfg.capture_mode = mode;
}
- if let Some(scope) = args.opt_str("capture_scope") {
- if scope != "window" && scope != "desktop" {
- return ToolResult::error(format!(
- "`capture_scope` must be \"window\" or \"desktop\", got \"{scope}\"."
- ));
- }
- if let Err(e) =
- pip_preview::write_config_key("capture_scope", Value::String(scope.clone()))
- {
- tracing::warn!("set_config: failed to persist capture_scope: {e}");
- }
- parts.push(format!("capture_scope={scope}"));
- cfg.capture_scope = scope;
- }
if let Some(dim) = args.opt_u64("max_image_dimension") {
cfg.max_image_dimension = dim as u32;
if let Err(e) = pip_preview::write_config_key("max_image_dimension", Value::from(dim)) {
@@ -6346,7 +6510,6 @@ impl Tool for SetConfigTool {
let (pip_enabled, pip_geometry) = pip_preview::read_pip_keys_from_file();
ToolResult::text(msg).with_structured(json!({
"capture_mode": cfg.capture_mode,
- "capture_scope": cfg.capture_scope,
"max_image_dimension": cfg.max_image_dimension,
"experimental_pip": pip_enabled,
"experimental_pip_geometry": pip_geometry
@@ -6989,16 +7152,6 @@ mod click_button_schema_tests {
}
}
-#[cfg(test)]
-mod driver_config_tests {
- use super::DriverConfig;
-
- #[test]
- fn capture_scope_defaults_to_window() {
- assert_eq!(DriverConfig::default().capture_scope, "window");
- }
-}
-
#[cfg(test)]
mod browser_launch_guard_tests {
use super::contains_remote_debugging_flag;
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs
index b5dd150ca0..fcaa88e8c3 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs
@@ -1424,7 +1424,7 @@ pub fn scroll_at(
) -> anyhow::Result<()> {
let direction = direction.to_string();
with_libei_fallback(
- || scroll_vptr(window_id, point, &direction, amount),
+ || scroll_vptr(Some(window_id), point, &direction, amount),
|| {
libei_wait_scroll_ready()?;
activate_window_for_input(window_id)?;
@@ -1436,14 +1436,32 @@ pub fn scroll_at(
)
}
+/// Scroll at a desktop-absolute point without activating a named toplevel.
+pub fn scroll_desktop(
+ x: i32,
+ y: i32,
+ direction: &str,
+ amount: u32,
+) -> anyhow::Result<()> {
+ let direction = direction.to_string();
+ with_libei_fallback(
+ || scroll_vptr(None, Some((x, y)), &direction, amount),
+ || {
+ libei_wait_scroll_ready()?;
+ libei_move_absolute(x, y)?;
+ libei_scroll(&direction, amount)
+ },
+ )
+}
+
/// wlroots virtual-pointer implementation of [`scroll`].
fn scroll_vptr(
- window_id: u64,
+ window_id: Option<u64>,
point: Option<(i32, i32)>,
direction: &str,
amount: u32,
) -> anyhow::Result<()> {
- let mut sess = open_vptr_session(Some(window_id))?;
+ let mut sess = open_vptr_session(window_id)?;
if let Some((x, y)) = point {
let px = x.clamp(0, (sess.output_w as i32).saturating_sub(1)) as u32;
let py = y.clamp(0, (sess.output_h as i32).saturating_sub(1)) as u32;
@@ -1549,7 +1567,7 @@ pub fn drag(
button: u8,
) -> anyhow::Result<()> {
with_libei_fallback(
- || drag_vptr(window_id, from_x, from_y, to_x, to_y, steps, button),
+ || drag_vptr(Some(window_id), from_x, from_y, to_x, to_y, steps, button),
|| {
libei_wait_pointer_ready()?;
activate_window_for_input(window_id)?;
@@ -1558,10 +1576,28 @@ pub fn drag(
)
}
+/// Drag through desktop-absolute points without activating a named toplevel.
+pub fn drag_desktop(
+ from_x: i32,
+ from_y: i32,
+ to_x: i32,
+ to_y: i32,
+ steps: u32,
+ button: u8,
+) -> anyhow::Result<()> {
+ with_libei_fallback(
+ || drag_vptr(None, from_x, from_y, to_x, to_y, steps, button),
+ || {
+ libei_wait_pointer_ready()?;
+ libei_drag(from_x, from_y, to_x, to_y, steps, button)
+ },
+ )
+}
+
/// wlroots virtual-pointer implementation of [`drag`].
#[allow(clippy::too_many_arguments)]
fn drag_vptr(
- window_id: u64,
+ window_id: Option<u64>,
from_x: i32,
from_y: i32,
to_x: i32,
@@ -1569,7 +1605,7 @@ fn drag_vptr(
steps: u32,
button: u8,
) -> anyhow::Result<()> {
- let mut sess = open_vptr_session(Some(window_id))?;
+ let mut sess = open_vptr_session(window_id)?;
std::thread::sleep(std::time::Duration::from_millis(40));
let (w, h) = (sess.output_w, sess.output_h);
let btn = evdev_pointer_button(button);
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs
index 775203884d..f4aef93504 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs
@@ -146,6 +146,32 @@ pub fn press_key_global(key: &str, modifiers: &[&str]) -> anyhow::Result<()> {
Ok(())
}
+/// Type Unicode text into the frontmost application through the global HID
+/// queue. This is the desktop-scope counterpart to PID-routed `type_text` and
+/// mirrors computer-server's frontmost pynput typing behavior.
+pub fn type_text_global(text: &str, inter_char_delay_ms: u64) -> anyhow::Result<()> {
+ use core_graphics::event::CGEventTapLocation;
+
+ let source = CGEventSource::new(CGEventSourceStateID::HIDSystemState)
+ .map_err(|_| anyhow::anyhow!("CGEventSource::new failed"))?;
+ for ch in text.chars() {
+ let value = ch.to_string();
+ let down = CGEvent::new_keyboard_event(source.clone(), 0, true)
+ .map_err(|_| anyhow::anyhow!("CGEvent keyboard down failed"))?;
+ down.set_string(&value);
+ down.set_flags(CGEventFlags::CGEventFlagNull);
+ down.post(CGEventTapLocation::HID);
+ std::thread::sleep(std::time::Duration::from_millis(8));
+ let up = CGEvent::new_keyboard_event(source.clone(), 0, false)
+ .map_err(|_| anyhow::anyhow!("CGEvent keyboard up failed"))?;
+ up.set_string(&value);
+ up.set_flags(CGEventFlags::CGEventFlagNull);
+ up.post(CGEventTapLocation::HID);
+ std::thread::sleep(std::time::Duration::from_millis(inter_char_delay_ms.max(8)));
+ }
+ Ok(())
+}
+
/// Post a keyboard event to `pid` via SLEventPostToPid (with auth message for
/// Chromium/Electron support) or fall back to CGEvent::post_to_pid.
fn post_keyboard_event(pid: i32, event: &CGEvent) {
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs
index 0726951522..9a8fe09680 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs
@@ -87,6 +87,45 @@ pub fn click_at_xy_desktop(x: f64, y: f64, count: usize, button: &str) -> anyhow
Ok(())
}
+/// Move the real hardware cursor to a logical desktop point.
+pub fn move_cursor_desktop(x: f64, y: f64) -> anyhow::Result<()> {
+ use core_graphics::display::CGDisplay;
+ let point = CGPoint::new(x, y);
+ CGDisplay::warp_mouse_cursor_position(point)
+ .map_err(|error| anyhow::anyhow!("CGWarpMouseCursorPosition failed: {error:?}"))?;
+ unsafe { CGAssociateMouseAndMouseCursorPosition(true) };
+ Ok(())
+}
+
+/// Scroll the foreground desktop surface at a logical screen point through the
+/// global HID queue. Mirrors computer-server's pynput wheel behavior while
+/// preserving cua-driver's explicit direction/amount contract.
+pub fn scroll_wheel_desktop(
+ x: f64,
+ y: f64,
+ delta_y_per_tick: i32,
+ delta_x_per_tick: i32,
+ ticks: usize,
+) -> anyhow::Result<()> {
+ use core_graphics::event::{CGEventTapLocation, ScrollEventUnit};
+
+ move_cursor_desktop(x, y)?;
+ std::thread::sleep(std::time::Duration::from_millis(40));
+ for _ in 0..ticks.max(1) {
+ let source = CGEventSource::new(CGEventSourceStateID::HIDSystemState)
+ .map_err(|_| anyhow::anyhow!("CGEventSource::new failed"))?;
+ let wheel_y = (delta_y_per_tick / 120).clamp(-10, 10);
+ let wheel_x = (delta_x_per_tick / 120).clamp(-10, 10);
+ let event =
+ CGEvent::new_scroll_event(source, ScrollEventUnit::LINE, 2, wheel_y, wheel_x, 0)
+ .map_err(|_| anyhow::anyhow!("CGEvent::new_scroll_event failed"))?;
+ unsafe { CGEventSetLocation(event.as_ptr() as *mut std::ffi::c_void, x, y) };
+ event.post(CGEventTapLocation::HID);
+ std::thread::sleep(std::time::Duration::from_millis(30));
+ }
+ Ok(())
+}
+
/// Click one exact desktop point, then restore the user's cursor position.
///
/// This is intentionally narrower than the public desktop click path. It is
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs
index c07221adaf..b7ba4d1859 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs
@@ -52,7 +52,7 @@ fn def() -> &'static ToolDef {
.into(),
input_schema: serde_json::json!({
"type": "object",
- "required": ["pid", "from_x", "from_y", "to_x", "to_y"],
+ "required": ["from_x", "from_y", "to_x", "to_y"],
"properties": {
"session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." },
"pid": { "type": "integer", "description": "Target process ID." },
@@ -90,6 +90,7 @@ fn def() -> &'static ToolDef {
"type": "boolean",
"description": "When true, coordinates are in the last zoom image for this pid; driver maps back to window coordinates."
},
+ "scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "Use desktop with no pid/window_id for native get_desktop_state screenshot coordinates." },
"delivery_mode": cua_driver_core::tool_schema::delivery_mode_schema()
},
"additionalProperties": false
@@ -109,6 +110,68 @@ impl Tool for DragTool {
async fn invoke(&self, args: Value) -> ToolResult {
use cua_driver_core::tool_args::ArgsExt;
+ if args.opt_str("scope").as_deref() == Some("desktop")
+ && args.get("pid").is_none()
+ && args.get("window_id").is_none()
+ {
+ let coordinate = |key: &str| {
+ args.opt_f64(key)
+ .or_else(|| args.opt_i64(key).map(|value| value as f64))
+ };
+ let Some(from_x) = coordinate("from_x") else {
+ return ToolResult::error("Missing required parameter: from_x");
+ };
+ let Some(from_y) = coordinate("from_y") else {
+ return ToolResult::error("Missing required parameter: from_y");
+ };
+ let Some(to_x) = coordinate("to_x") else {
+ return ToolResult::error("Missing required parameter: to_x");
+ };
+ let Some(to_y) = coordinate("to_y") else {
+ return ToolResult::error("Missing required parameter: to_y");
+ };
+ let (from_x, from_y) = super::desktop_screenshot_point(from_x, from_y).await;
+ let (to_x, to_y) = super::desktop_screenshot_point(to_x, to_y).await;
+ let duration_ms = args.u64_or("duration_ms", 500).min(10_000);
+ let steps = args.u64_or("steps", 20).clamp(1, 200) as usize;
+ let modifiers: Vec<String> = args.str_array("modifier");
+ let button = match args.str_or("button", "left").to_lowercase().as_str() {
+ "left" => DragButton::Left,
+ "right" => DragButton::Right,
+ "middle" => DragButton::Middle,
+ other => {
+ return ToolResult::error(format!(
+ "Unknown button '{other}' — expected left, right, or middle."
+ ))
+ }
+ };
+ let result = tokio::task::spawn_blocking(move || {
+ let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
+ crate::input::mouse::drag_at_xy_foreground(
+ from_x,
+ from_y,
+ to_x,
+ to_y,
+ duration_ms,
+ steps,
+ &modifier_refs,
+ button,
+ )
+ })
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Dragged desktop from ({from_x:.1}, {from_y:.1}) to ({to_x:.1}, {to_y:.1})."
+ ))
+ .with_structured(serde_json::json!({
+ "scope": "desktop",
+ "path": "hid",
+ "effect": "unverifiable"
+ })),
+ Ok(Err(error)) => ToolResult::error(format!("desktop drag failed: {error}")),
+ Err(error) => ToolResult::error(format!("desktop drag task failed: {error}")),
+ };
+ }
let pid = match args.require_i32("pid") {
Ok(v) => v,
Err(e) => return e,
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_config.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_config.rs
index 79313eb528..a1b1474952 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_config.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_config.rs
@@ -68,7 +68,6 @@ impl Tool for GetConfigTool {
// response reflects whatever set_config (or a direct JSON edit)
// last wrote.
let (pip_enabled, pip_geometry) = pip_preview::read_pip_keys_from_file();
- let capture_scope = self.state.config.read().unwrap().capture_scope.clone();
ToolResult::text("cua-driver-rs configuration").with_structured(serde_json::json!({
"version": env!("CARGO_PKG_VERSION"),
// Maintainer E2E builds set this at compile time so the
@@ -77,10 +76,9 @@ impl Tool for GetConfigTool {
// sharing its package version.
"source_sha": option_env!("CUA_DRIVER_SOURCE_SHA"),
"platform": "macos",
- // capture_mode is a per-call param; capture_scope is a global
- // setting that gates get_desktop_state.
+ // capture_mode is per-call; capture_scope is per-session and is
+ // reported by get_session_state rather than persistent config.
"max_image_dimension": max_image_dimension,
- "capture_scope": capture_scope,
"agent_cursor": {
"enabled": cursor_enabled,
},
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_desktop_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_desktop_state.rs
index cf5558e8b7..fa2da56381 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_desktop_state.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_desktop_state.rs
@@ -52,25 +52,6 @@ impl Tool for GetDesktopStateTool {
async fn invoke(&self, args: Value) -> ToolResult {
use cua_driver_core::tool_args::ArgsExt;
- // Gate on the global capture_scope (re-read from the persisted config,
- // the same value set_config writes): a full-display capture is a
- // desktop-scope operation, available only under capture_scope="desktop".
- let scope = super::load_driver_config().capture_scope;
- if scope != "desktop" {
- return ToolResult::error(format!(
- "get_desktop_state requires capture_scope=\"desktop\" (current scope is \
- \"{scope}\"). Full-display capture is a desktop-scope operation; call \
- set_config with capture_scope=desktop first (it also enables window-less \
- screen-absolute click/scroll). For a single window, use \
- get_window_state(pid, window_id) instead."
- ))
- .with_structured(serde_json::json!({
- "code": "desktop_scope_disabled",
- "capture_scope": scope,
- "suggestion": "set_config capture_scope=desktop",
- }));
- }
-
let screenshot_out_file = args.opt_str("screenshot_out_file").map(|s| {
// Expand ~ prefix (mirrors get_window_state).
if s.starts_with("~/") {
@@ -123,6 +104,7 @@ impl Tool for GetDesktopStateTool {
let mut structured = serde_json::json!({
"platform": "macos",
+ "display": "primary",
"screenshot_width": screenshot_width,
"screenshot_height": screenshot_height,
"screen_width": screen_width,
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs
index c6aed03836..6a4398048d 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs
@@ -1,8 +1,11 @@
use async_trait::async_trait;
-use cua_driver_core::{protocol::ToolResult, tool::{Tool, ToolDef}};
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use libc;
use serde_json::Value;
use std::sync::Arc;
-use libc;
use crate::apps;
use crate::focus_guard;
@@ -15,7 +18,9 @@ pub struct HotkeyTool {
}
impl HotkeyTool {
- pub fn new(state: Arc<ToolState>) -> Self { Self { state } }
+ pub fn new(state: Arc<ToolState>) -> Self {
+ Self { state }
+ }
}
static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
@@ -48,7 +53,7 @@ fn def() -> &'static ToolDef {
.into(),
input_schema: serde_json::json!({
"type": "object",
- "required": ["pid", "keys"],
+ "required": ["keys"],
"properties": {
"session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." },
"pid": { "type": "integer", "description": "Target process ID." },
@@ -64,6 +69,7 @@ fn def() -> &'static ToolDef {
"type": "integer",
"description": "Target window. Required for delivery_mode:\"foreground\" (the NSMenu activation needs a window). Does NOT itself raise the window — raising is gated on delivery_mode."
},
+ "scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "Use desktop with no pid/window_id to send the chord to the frontmost application." },
"delivery_mode": cua_driver_core::tool_schema::delivery_mode_schema()
},
"additionalProperties": false
@@ -85,13 +91,51 @@ fn is_modifier(k: &str) -> bool {
#[async_trait]
impl Tool for HotkeyTool {
- fn def(&self) -> &ToolDef { def() }
+ fn def(&self) -> &ToolDef {
+ def()
+ }
async fn invoke(&self, args: Value) -> ToolResult {
use cua_driver_core::tool_args::ArgsExt;
let _ = &self.state;
- let pid = match args.require_i32("pid") { Ok(v) => v, Err(e) => return e };
+ if args.opt_str("scope").as_deref() == Some("desktop")
+ && args.get("pid").is_none()
+ && args.get("window_id").is_none()
+ {
+ let raw_keys = args.str_array("keys");
+ let modifiers: Vec<String> = raw_keys
+ .iter()
+ .filter(|key| is_modifier(key))
+ .cloned()
+ .collect();
+ let Some(key) = raw_keys.iter().rev().find(|key| !is_modifier(key)).cloned() else {
+ return ToolResult::error(
+ "keys must include at least one non-modifier key for desktop hotkey",
+ );
+ };
+ let display = raw_keys.join("+");
+ let result = tokio::task::spawn_blocking(move || {
+ let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
+ crate::input::keyboard::press_key_global(&key, &modifier_refs)
+ })
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!("Pressed desktop hotkey {display}."))
+ .with_structured(serde_json::json!({
+ "scope": "desktop",
+ "path": "hid",
+ "effect": "unverifiable"
+ })),
+ Ok(Err(error)) => ToolResult::error(format!("desktop hotkey failed: {error}")),
+ Err(error) => ToolResult::error(format!("desktop hotkey task failed: {error}")),
+ };
+ }
+
+ let pid = match args.require_i32("pid") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
if args.get("keys").and_then(|v| v.as_array()).is_none() {
return ToolResult::error("Missing required parameter: keys");
@@ -104,18 +148,20 @@ impl Tool for HotkeyTool {
// Split: modifiers are all entries that are modifier names; base key is everything else.
// Typically: last non-modifier is the key; all others are modifiers.
- let modifiers: Vec<String> = raw_keys.iter()
+ let modifiers: Vec<String> = raw_keys
+ .iter()
.filter(|k| is_modifier(k))
.cloned()
.collect();
- let non_modifiers: Vec<String> = raw_keys.iter()
+ let non_modifiers: Vec<String> = raw_keys
+ .iter()
.filter(|k| !is_modifier(k))
.cloned()
.collect();
if non_modifiers.is_empty() {
return ToolResult::error(
- "keys must include at least one non-modifier key (e.g. \"c\" in [\"cmd\", \"c\"])."
+ "keys must include at least one non-modifier key (e.g. \"c\" in [\"cmd\", \"c\"]).",
);
}
@@ -137,15 +183,29 @@ impl Tool for HotkeyTool {
let px = args.get("x").and_then(|v| v.as_f64());
let py = args.get("y").and_then(|v| v.as_f64());
if let (Some(cx), Some(cy)) = (px, py) {
- let from_zoom = args.get("from_zoom").and_then(|v| v.as_bool()).unwrap_or(false);
+ let from_zoom = args
+ .get("from_zoom")
+ .and_then(|v| v.as_bool())
+ .unwrap_or(false);
if let Err(e) = super::focus_by_pixel(
- &self.state, pid, window_id, cx, cy, fg,
- args.opt_str("session"), args.opt_str("_session_id"), from_zoom,
- ).await {
+ &self.state,
+ pid,
+ window_id,
+ cx,
+ cy,
+ fg,
+ args.opt_str("session"),
+ args.opt_str("_session_id"),
+ from_zoom,
+ )
+ .await
+ {
return e;
}
true
- } else { false }
+ } else {
+ false
+ }
};
// ── Focus-suppression wrap (Swift WindowChangeDetector + FocusGuard) ──
@@ -169,9 +229,11 @@ impl Tool for HotkeyTool {
// foreground rung: briefly front the window so NSMenu key
// equivalents dispatch, then restore prior frontmost.
(true, Some(wid)) => {
- crate::input::skylight::with_menu_shortcut_activation(pid as libc::pid_t, wid, || {
- crate::input::keyboard::hotkey_no_auth(pid, &key, &m)
- })?;
+ crate::input::skylight::with_menu_shortcut_activation(
+ pid as libc::pid_t,
+ wid,
+ || crate::input::keyboard::hotkey_no_auth(pid, &key, &m),
+ )?;
Ok(())
}
// background (default): auth-envelope post to the pid, no
@@ -188,7 +250,11 @@ impl Tool for HotkeyTool {
match result {
Ok(Ok(())) => {
- let label = if fg { " (delivery_mode:foreground)" } else { "" };
+ let label = if fg {
+ " (delivery_mode:foreground)"
+ } else {
+ ""
+ };
// A combo is never read-back-verifiable. On the background rung,
// point the agent at the foreground escalation for menu shortcuts
// an app drops in the background — same contract as type_text.
@@ -209,10 +275,11 @@ impl Tool for HotkeyTool {
ToolResult::text(format!(
"Pressed {key_display} on pid {pid}{label}.{}",
changes.result_suffix()
- )).with_structured(structured)
+ ))
+ .with_structured(structured)
}
Ok(Err(e)) => ToolResult::error(format!("hotkey failed: {e}")),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
}
}
}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs
index 5c28943d49..3618a9c1ef 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs
@@ -180,29 +180,17 @@ impl ResizeRegistry {
}
/// Runtime-mutable driver configuration persisted across calls within a session.
-///
-/// `capture_mode` is per-call now (the `capture_mode` param). `capture_scope` is
-/// re-introduced here as a GLOBAL setting: it gates `get_desktop_state` (a
-/// full-display capture has no pid/window_id and therefore no per-call scope to
-/// key on — the only coherent gate is a global one), matching the Windows/Linux
-/// `DriverConfig.capture_scope`. Per-call tools (`click`/`scroll`) still accept a
-/// per-call `scope`; this global is the baseline the no-args desktop-capture
-/// tool reads. Default `"window"`.
pub struct DriverConfig {
/// Max screenshot dimension (0 = no limit). Applied during screenshot/zoom.
/// Default 1568 matches Swift's `CuaDriverConfig.defaultMaxImageDimension` —
/// the long edge is downscaled to this before encoding.
pub max_image_dimension: u32,
- /// Capture scope: `"window"` (default) or `"desktop"`. Gates
- /// `get_desktop_state` (full-display capture requires `"desktop"`).
- pub capture_scope: String,
}
impl Default for DriverConfig {
fn default() -> Self {
Self {
max_image_dimension: 1568,
- capture_scope: "window".to_owned(),
}
}
}
@@ -228,22 +216,36 @@ pub fn load_driver_config() -> DriverConfig {
Ok(v) => v,
Err(_) => return cfg, // malformed file — use defaults
};
- // `capture_mode` is per-call now; an old on-disk `capture_mode` key is
- // silently ignored. `capture_scope` IS a global setting again (gates
- // get_desktop_state) — load it, accepting only window|desktop.
+ // `capture_mode` is per-call now; old on-disk `capture_mode` and
+ // `capture_scope` keys are intentionally inert.
if let Some(v) = json.get("max_image_dimension").and_then(|v| v.as_u64()) {
if let Ok(v32) = u32::try_from(v) {
cfg.max_image_dimension = v32;
}
}
- if let Some(s) = json.get("capture_scope").and_then(|v| v.as_str()) {
- if s == "window" || s == "desktop" {
- cfg.capture_scope = s.to_owned();
- }
- }
cfg
}
+/// Convert native pixels from `get_desktop_state` into the logical point space
+/// used by CoreGraphics input APIs. Retina scaled modes cannot rely on the
+/// nominal backing factor alone, so derive the ratio from the actual PNG.
+pub async fn desktop_screenshot_point(x: f64, y: f64) -> (f64, f64) {
+ let ratio = tokio::task::spawn_blocking(|| {
+ let logical_w = get_screen_size::main_screen_size().map(|(w, _, _)| w as f64);
+ let shot_w = crate::capture::screenshot_display_bytes()
+ .ok()
+ .and_then(|png| crate::capture::png_dimensions(&png).ok())
+ .map(|(w, _)| w as f64);
+ match (shot_w, logical_w) {
+ (Some(sw), Some(lw)) if lw > 0.0 && sw > lw => sw / lw,
+ _ => 1.0,
+ }
+ })
+ .await
+ .unwrap_or(1.0);
+ (x / ratio, y / ratio)
+}
+
/// Persist a single key/value pair to `~/.cua-driver/config.json`.
/// Merges with any existing file contents so other keys are preserved.
/// Returns `Err` if the directory cannot be created or the file cannot be written.
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs
index 3d14684046..dd7809d6ae 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs
@@ -1,5 +1,8 @@
use async_trait::async_trait;
-use cua_driver_core::{protocol::ToolResult, tool::{Tool, ToolDef}};
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
use serde_json::Value;
use std::sync::Arc;
@@ -10,7 +13,9 @@ pub struct MoveCursorTool {
}
impl MoveCursorTool {
- pub fn new(state: Arc<ToolState>) -> Self { Self { state } }
+ pub fn new(state: Arc<ToolState>) -> Self {
+ Self { state }
+ }
}
static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
@@ -18,9 +23,9 @@ static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
fn def() -> &'static ToolDef {
DEF.get_or_init(|| ToolDef {
name: "move_cursor".into(),
- description: "Move the agent cursor overlay to (x, y). Does NOT move the real mouse \
- cursor — the user's cursor stays where it is. Useful for showing the agent's \
- attention without interrupting the user.".into(),
+ description: "Move a cursor to (x, y). In window scope (default), moves only the \
+ agent cursor overlay. With scope=desktop, moves the real OS pointer in native \
+ get_desktop_state screenshot coordinates.".into(),
input_schema: serde_json::json!({
"type": "object",
"required": ["x", "y"],
@@ -28,15 +33,12 @@ fn def() -> &'static ToolDef {
"session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." },
"x": { "type": "number" },
"y": { "type": "number" },
+ "scope": { "type": "string", "enum": ["window", "desktop"], "default": "window" },
"cursor_id": { "type": "string", "description": "Cursor instance to move. Default: 'default'." }
},
"additionalProperties": false
}),
- // read-only: move_cursor only nudges the agent-cursor overlay, never the
- // target app — so it's safe to run concurrently. The `readOnlyHint` this
- // emits lets MCP clients (e.g. Claude Code's isConcurrencySafe) parallelize
- // cursor moves. (Mutating tools like click stay read_only:false on purpose.)
- read_only: true,
+ read_only: false,
destructive: false,
idempotent: true,
open_world: false,
@@ -45,12 +47,41 @@ fn def() -> &'static ToolDef {
#[async_trait]
impl Tool for MoveCursorTool {
- fn def(&self) -> &ToolDef { def() }
+ fn def(&self) -> &ToolDef {
+ def()
+ }
async fn invoke(&self, args: Value) -> ToolResult {
use cua_driver_core::tool_args::ArgsExt;
- let x = match args.require_f64("x") { Ok(v) => v, Err(e) => return e };
- let y = match args.require_f64("y") { Ok(v) => v, Err(e) => return e };
+ let x = match args.require_f64("x") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ let y = match args.require_f64("y") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ if args.opt_str("scope").as_deref() == Some("desktop") {
+ let (x, y) = super::desktop_screenshot_point(x, y).await;
+ let result =
+ tokio::task::spawn_blocking(move || crate::input::mouse::move_cursor_desktop(x, y))
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Moved the real desktop pointer to ({x:.1}, {y:.1})."
+ ))
+ .with_structured(serde_json::json!({
+ "scope": "desktop",
+ "x": x,
+ "y": y,
+ "effect": "unverifiable"
+ })),
+ Ok(Err(error)) => {
+ ToolResult::error(format!("desktop pointer move failed: {error}"))
+ }
+ Err(error) => ToolResult::error(format!("desktop pointer task failed: {error}")),
+ };
+ }
let cursor_id = super::cursor_tools::resolve_cursor_key(&args);
self.state.cursor_registry.update_position(&cursor_id, x, y);
@@ -62,6 +93,8 @@ impl Tool for MoveCursorTool {
// then glides in, identical to `click`. No-op for an empty (anonymous)
// key or when the overlay is disabled for this cursor.
crate::cursor::overlay::animate_cursor_to(cursor_id.clone(), x, y).await;
- ToolResult::text(format!("Agent cursor '{cursor_id}' moved to ({x:.1}, {y:.1})."))
+ ToolResult::text(format!(
+ "Agent cursor '{cursor_id}' moved to ({x:.1}, {y:.1})."
+ ))
}
}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs
index ebcf1ecd21..59b697de6a 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs
@@ -1,8 +1,11 @@
use async_trait::async_trait;
-use cua_driver_core::{protocol::ToolResult, tool::{Tool, ToolDef}};
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use libc;
use serde_json::Value;
use std::sync::Arc;
-use libc;
use crate::apps;
use crate::focus_guard;
@@ -15,7 +18,9 @@ pub struct PressKeyTool {
}
impl PressKeyTool {
- pub fn new(state: Arc<ToolState>) -> Self { Self { state } }
+ pub fn new(state: Arc<ToolState>) -> Self {
+ Self { state }
+ }
}
static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
@@ -39,7 +44,7 @@ fn def() -> &'static ToolDef {
Modifiers array: cmd, shift, option/alt, ctrl, fn.".into(),
input_schema: serde_json::json!({
"type": "object",
- "required": ["pid", "key"],
+ "required": ["key"],
"properties": {
"session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." },
"pid": { "type": "integer" },
@@ -54,6 +59,7 @@ fn def() -> &'static ToolDef {
"element_token": { "type": "string", "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit \"stale\" error if the snapshot has been superseded." },
"x": { "type": "number", "description": "Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_index." },
"y": { "type": "number", "description": "Screenshot-pixel Y (see x)." },
+ "scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "Use desktop with no pid/window_id to send the key to the frontmost application." },
"delivery_mode": cua_driver_core::tool_schema::delivery_mode_schema()
},
"additionalProperties": false
@@ -67,16 +73,50 @@ fn def() -> &'static ToolDef {
#[async_trait]
impl Tool for PressKeyTool {
- fn def(&self) -> &ToolDef { def() }
+ fn def(&self) -> &ToolDef {
+ def()
+ }
async fn invoke(&self, args: Value) -> ToolResult {
use cua_driver_core::tool_args::ArgsExt;
- let pid = match args.require_i32("pid") { Ok(v) => v, Err(e) => return e };
- let key_raw = match args.require_str("key") { Ok(v) => v, Err(e) => return e };
+ if args.opt_str("scope").as_deref() == Some("desktop")
+ && args.get("pid").is_none()
+ && args.get("window_id").is_none()
+ {
+ let key = match args.require_str("key") {
+ Ok(value) => value,
+ Err(error) => return error,
+ };
+ let modifiers: Vec<String> = args.str_array("modifiers");
+ let key_for_input = key.clone();
+ let result = tokio::task::spawn_blocking(move || {
+ let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
+ crate::input::keyboard::press_key_global(&key_for_input, &modifier_refs)
+ })
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!("Pressed '{key}' on the desktop."))
+ .with_structured(serde_json::json!({
+ "scope": "desktop",
+ "path": "hid",
+ "effect": "unverifiable"
+ })),
+ Ok(Err(error)) => ToolResult::error(format!("desktop press_key failed: {error}")),
+ Err(error) => ToolResult::error(format!("desktop press_key task failed: {error}")),
+ };
+ }
+ let pid = match args.require_i32("pid") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ let key_raw = match args.require_str("key") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
let mut modifiers: Vec<String> = args.str_array("modifiers");
// Surface 6: element_token / element_index precedence resolution.
let element_token_arg = args.opt_str("element_token");
- let window_id_arg = args.opt_u64("window_id").map(|v| v as u32);
+ let window_id_arg = args.opt_u64("window_id").map(|v| v as u32);
let element_index_arg = args.opt_u64("element_index").map(|v| v as usize);
let resolved = match cua_driver_core::element_token::resolve_element_args(
pid,
@@ -91,7 +131,9 @@ impl Tool for PressKeyTool {
let (element_index, window_id) = match resolved {
cua_driver_core::element_token::ResolvedElement::None => (None, window_id_arg),
cua_driver_core::element_token::ResolvedElement::Element {
- window_id: wid, element_index: idx, via_token: _,
+ window_id: wid,
+ element_index: idx,
+ via_token: _,
} => (Some(idx), wid),
};
@@ -120,18 +162,32 @@ impl Tool for PressKeyTool {
if let (Some(cx), Some(cy)) = (px, py) {
if element_index.is_some() {
return ToolResult::error(
- "Pass either element_index (ax) or x,y (px) to press_key, not both."
+ "Pass either element_index (ax) or x,y (px) to press_key, not both.",
);
}
- let from_zoom = args.get("from_zoom").and_then(|v| v.as_bool()).unwrap_or(false);
+ let from_zoom = args
+ .get("from_zoom")
+ .and_then(|v| v.as_bool())
+ .unwrap_or(false);
if let Err(e) = super::focus_by_pixel(
- &self.state, pid, window_id, cx, cy, fg,
- args.opt_str("session"), args.opt_str("_session_id"), from_zoom,
- ).await {
+ &self.state,
+ pid,
+ window_id,
+ cx,
+ cy,
+ fg,
+ args.opt_str("session"),
+ args.opt_str("_session_id"),
+ from_zoom,
+ )
+ .await
+ {
return e;
}
true
- } else { false }
+ } else {
+ false
+ }
};
// Resolve the pre-focus element pointer (if requested) outside
@@ -168,7 +224,8 @@ impl Tool for PressKeyTool {
if let Some(element_ptr) = pre_focus_ptr {
let _ = tokio::task::spawn_blocking(move || {
crate::input::ax_actions::focus_element(element_ptr)
- }).await;
+ })
+ .await;
tokio::time::sleep(std::time::Duration::from_millis(30)).await;
}
@@ -180,9 +237,11 @@ impl Tool for PressKeyTool {
if fg && !px_focus {
if let Some(wid) = window_id {
if element_index.is_none() {
- crate::input::skylight::with_menu_shortcut_activation(pid as libc::pid_t, wid, || {
- crate::input::keyboard::press_key_no_auth(pid, &key, &m)
- })?;
+ crate::input::skylight::with_menu_shortcut_activation(
+ pid as libc::pid_t,
+ wid,
+ || crate::input::keyboard::press_key_no_auth(pid, &key, &m),
+ )?;
return Ok(());
}
}
@@ -199,7 +258,11 @@ impl Tool for PressKeyTool {
match result {
Ok(Ok(())) => {
- let label = if fg { " (delivery_mode:foreground)" } else { "" };
+ let label = if fg {
+ " (delivery_mode:foreground)"
+ } else {
+ ""
+ };
let mut structured = serde_json::json!({
"path": if fg { "key_events_fg" } else { "key_events" },
"verified": false,
@@ -216,7 +279,8 @@ impl Tool for PressKeyTool {
ToolResult::text(format!(
"✅ Pressed {display_key} on pid {pid}{label}.{}",
changes.result_suffix()
- )).with_structured(structured)
+ ))
+ .with_structured(structured)
}
Ok(Err(e)) => ToolResult::error(format!("press_key failed: {e}")),
Err(e) => ToolResult::error(format!("Task error: {e}")),
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs
index ad69f68a41..beb7d0cb8a 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs
@@ -92,6 +92,7 @@ fn def() -> &'static ToolDef {
"element_token": { "type": "string", "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit \"stale\" error if the snapshot has been superseded. Routes through the pixel-wheel path at the element's center." },
"x": { "type": "number", "description": "Window-local screenshot X (top-left origin of the PNG from get_window_state). With `y`, routes through the pixel-wheel path at this point — use for a scrollable surface that isn't in the AX tree. Requires window_id to anchor the window→screen conversion." },
"y": { "type": "number", "description": "Window-local screenshot Y. See `x`." },
+ "scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "Use desktop with x,y and no pid/window_id for native get_desktop_state screenshot coordinates." },
"delivery_mode": cua_driver_core::tool_schema::delivery_mode_schema()
},
"additionalProperties": false
@@ -111,6 +112,58 @@ impl Tool for ScrollTool {
async fn invoke(&self, args: Value) -> ToolResult {
use cua_driver_core::tool_args::ArgsExt;
+ if args.opt_str("scope").as_deref() == Some("desktop")
+ && args.get("pid").is_none()
+ && args.get("window_id").is_none()
+ {
+ let direction = match args.require_str("direction") {
+ Ok(value) => value,
+ Err(error) => return error,
+ };
+ let x = match args.require_f64("x") {
+ Ok(value) => value,
+ Err(error) => return error,
+ };
+ let y = match args.require_f64("y") {
+ Ok(value) => value,
+ Err(error) => return error,
+ };
+ let by = args.str_or("by", "line");
+ let amount = args.u64_or("amount", 3).clamp(1, 50) as usize;
+ let step = if by == "page" {
+ WHEEL_STEP_PAGE_PX
+ } else {
+ WHEEL_STEP_LINE_PX
+ };
+ let (delta_y, delta_x) = match direction.as_str() {
+ "down" => (-step, 0),
+ "up" => (step, 0),
+ "right" => (0, -step),
+ "left" => (0, step),
+ other => {
+ return ToolResult::error(format!(
+ "unknown scroll direction '{other}'; expected up, down, left, or right"
+ ))
+ }
+ };
+ let (x, y) = super::desktop_screenshot_point(x, y).await;
+ let result = tokio::task::spawn_blocking(move || {
+ crate::input::mouse::scroll_wheel_desktop(x, y, delta_y, delta_x, amount)
+ })
+ .await;
+ return match result {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Scrolled desktop {direction} by {by} × {amount} at ({x:.1}, {y:.1})."
+ ))
+ .with_structured(serde_json::json!({
+ "scope": "desktop",
+ "path": "hid",
+ "effect": "unverifiable"
+ })),
+ Ok(Err(error)) => ToolResult::error(format!("desktop scroll failed: {error}")),
+ Err(error) => ToolResult::error(format!("desktop scroll task failed: {error}")),
+ };
+ }
let pid = match args.require_i32("pid") {
Ok(v) => v,
Err(e) => return e,
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_config.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_config.rs
index 52771d1b5d..22aa0138fb 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_config.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_config.rs
@@ -19,13 +19,12 @@ fn def() -> &'static ToolDef {
DEF.get_or_init(|| ToolDef {
name: "set_config".into(),
description: "Update cua-driver-rs configuration. Changes to \
- max_image_dimension and capture_scope take effect immediately. The \
+ max_image_dimension take effect immediately. The \
experimental_pip keys are persisted to ~/.cua-driver/config.json and \
take effect on the next daemon restart (the PiP backend is \
initialised once at startup).\n\nNote: capture_mode is a per-call \
- param (on get_window_state / click), not a stored setting. \
- capture_scope IS a global setting: it gates get_desktop_state \
- (full-display capture requires capture_scope=desktop).".into(),
+ param (on get_window_state / click), not a stored setting. Capture \
+ scope is selected by start_session, not set_config.".into(),
input_schema: serde_json::json!({
"type": "object",
"properties": {
@@ -42,13 +41,6 @@ fn def() -> &'static ToolDef {
"type": "integer",
"description": "Max dimension for screenshot resizing (0 = no limit)."
},
- "capture_scope": {
- "type": "string",
- "enum": ["window", "desktop"],
- "description": "Capture scope: \"window\" (default) or \"desktop\". Desktop \
- scope enables get_desktop_state (full-display capture) and window-less \
- screen-absolute click/scroll. Global setting; takes effect immediately."
- },
"experimental_pip": {
"type": "boolean",
"description": "Enable the experimental picture-in-picture preview window. \
@@ -75,6 +67,18 @@ impl Tool for SetConfigTool {
async fn invoke(&self, args: Value) -> ToolResult {
use cua_driver_core::tool_args::ArgsExt;
+ if args.get("capture_scope").is_some()
+ || args.get("key").and_then(Value::as_str) == Some("capture_scope")
+ {
+ return ToolResult::error(
+ "config key 'capture_scope' is retired; pass capture_scope=auto|window|desktop to start_session",
+ )
+ .with_structured(serde_json::json!({
+ "code": "config_key_retired",
+ "key": "capture_scope",
+ "replacement": "start_session.capture_scope",
+ }));
+ }
// The daemon injects `_session_id` for non-anonymous MCP sessions.
// Absent => anonymous/global session (CLI one-shot, legacy proxy) =>
// today's behavior: write the shared global DriverConfig + persist to
@@ -83,10 +87,7 @@ impl Tool for SetConfigTool {
// sessions don't clobber each other or the persisted default.
let session_id = args.opt_str("_session_id");
- // Accept BOTH shapes, matching Windows/Linux + the CLI `config set`:
- // - direct fields: {"capture_scope":"desktop"}
- // - {key, value}: {"key":"capture_scope","value":"desktop"}
- // A direct field wins if both are somehow present.
+ // Accept BOTH the direct field and {key,value} shapes.
let kv: Option<(String, Value)> = args
.opt_str("key")
.and_then(|k| args.get("value").map(|v| (k, v.clone())));
@@ -144,46 +145,19 @@ impl Tool for SetConfigTool {
pip_note = format!(" — restart cua-driver for experimental_pip_geometry={geom} to take effect");
}
}
- // capture_scope: GLOBAL setting (gates get_desktop_state). Accept the
- // direct field or {key,value} shape; validate window|desktop; write the
- // shared global config + persist (matches Windows/Linux — not
- // session-scoped, since it's the baseline a no-args capture tool reads).
- let scope_arg = args.opt_str("capture_scope").or_else(|| {
- kv.as_ref()
- .filter(|(k, _)| k == "capture_scope")
- .and_then(|(_, v)| v.as_str().map(str::to_owned))
- });
- let mut capture_scope_note = String::new();
- if let Some(sc) = scope_arg {
- if sc != "window" && sc != "desktop" {
- return ToolResult::error(format!(
- "`capture_scope` must be \"window\" or \"desktop\", got \"{sc}\"."
- ));
- }
- self.state.config.write().unwrap().capture_scope = sc.clone();
- if let Err(e) = write_driver_config_key("capture_scope", &Value::String(sc.clone())) {
- tracing::warn!("set_config: failed to persist capture_scope: {e}");
- }
- capture_scope_note = format!(", capture_scope={sc}");
- }
-
let scope_note = if session_id.is_some() {
" (session-scoped; persisted default unchanged)"
} else {
""
};
- // Echo the config back in structured content (matches Windows/Linux
- // set_config, which callers/tests read for the applied capture_scope).
- let capture_scope = self.state.config.read().unwrap().capture_scope.clone();
ToolResult::text(format!(
- "Config updated: max_image_dimension={}{}{}{}",
- effective_dim, capture_scope_note, scope_note, pip_note
+ "Config updated: max_image_dimension={}{}{}",
+ effective_dim, scope_note, pip_note
))
.with_structured(serde_json::json!({
"version": env!("CARGO_PKG_VERSION"),
"platform": "macos",
"max_image_dimension": effective_dim,
- "capture_scope": capture_scope,
}))
}
}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs
index 49def31da7..aa0fb4cca5 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs
@@ -22,15 +22,17 @@
//! (e.g., to trigger live-search debounce handlers).
use async_trait::async_trait;
-use cua_driver_core::{protocol::ToolResult, tool::{Tool, ToolDef}};
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
use serde_json::Value;
use std::sync::Arc;
+use crate::apps;
use crate::ax::bindings::{
- copy_string_attr, focused_element_of_pid, kAXErrorSuccess, set_string_attr,
- AXUIElementRef,
+ copy_string_attr, focused_element_of_pid, kAXErrorSuccess, set_string_attr, AXUIElementRef,
};
-use crate::apps;
use crate::focus_guard;
use crate::window_change_detector::WindowChangeDetector;
use core_foundation::base::CFRelease;
@@ -42,7 +44,9 @@ pub struct TypeTextTool {
}
impl TypeTextTool {
- pub fn new(state: Arc<ToolState>) -> Self { Self { state } }
+ pub fn new(state: Arc<ToolState>) -> Self {
+ Self { state }
+ }
}
static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
@@ -78,7 +82,7 @@ fn def() -> &'static ToolDef {
.into(),
input_schema: serde_json::json!({
"type": "object",
- "required": ["pid", "text"],
+ "required": ["text"],
"properties": {
"session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." },
"pid": { "type": "integer", "description": "Target process ID." },
@@ -103,6 +107,7 @@ fn def() -> &'static ToolDef {
"maximum": 200,
"description": "Milliseconds between characters in the CGEvent fallback path. Default 30. Ignored when the AX path succeeds."
},
+ "scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "Use desktop with no pid/window_id to type into the frontmost application." },
"delivery_mode": {
"type": "string",
"enum": ["background", "foreground"],
@@ -120,19 +125,56 @@ fn def() -> &'static ToolDef {
#[async_trait]
impl Tool for TypeTextTool {
- fn def(&self) -> &ToolDef { def() }
+ fn def(&self) -> &ToolDef {
+ def()
+ }
async fn invoke(&self, args: Value) -> ToolResult {
use cua_driver_core::tool_args::ArgsExt;
- let pid = match args.require_i32("pid") { Ok(v) => v, Err(e) => return e };
- let text_raw = match args.require_str("text") { Ok(v) => v, Err(e) => return e };
+ if args.opt_str("scope").as_deref() == Some("desktop")
+ && args.get("pid").is_none()
+ && args.get("window_id").is_none()
+ {
+ let text = match args.require_str("text") {
+ Ok(value) => {
+ cua_driver_core::text_sanitize::strip_trailing_agent_protocol_tags(&value)
+ .into_owned()
+ }
+ Err(error) => return error,
+ };
+ let delay_ms = args.u64_or("delay_ms", 30).min(200);
+ let result = tokio::task::spawn_blocking(move || {
+ crate::input::keyboard::type_text_global(&text, delay_ms)
+ })
+ .await;
+ return match result {
+ Ok(Ok(())) => {
+ ToolResult::text("Typed text into the frontmost desktop application.")
+ .with_structured(serde_json::json!({
+ "scope": "desktop",
+ "path": "hid",
+ "effect": "unverifiable"
+ }))
+ }
+ Ok(Err(error)) => ToolResult::error(format!("desktop type_text failed: {error}")),
+ Err(error) => ToolResult::error(format!("desktop type_text task failed: {error}")),
+ };
+ }
+ let pid = match args.require_i32("pid") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ let text_raw = match args.require_str("text") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
// Strip trailing agent-protocol closing tags — see
// cua_driver_core::text_sanitize docs for rationale.
let text = cua_driver_core::text_sanitize::strip_trailing_agent_protocol_tags(&text_raw)
.into_owned();
// Surface 6: element_token / element_index precedence resolution.
let element_token_arg = args.opt_str("element_token");
- let window_id_arg = args.opt_u64("window_id").map(|v| v as u32);
+ let window_id_arg = args.opt_u64("window_id").map(|v| v as u32);
let element_index_arg = args.opt_u64("element_index").map(|v| v as usize);
let resolved = match cua_driver_core::element_token::resolve_element_args(
pid,
@@ -147,10 +189,12 @@ impl Tool for TypeTextTool {
let (element_index, window_id) = match resolved {
cua_driver_core::element_token::ResolvedElement::None => (None, window_id_arg),
cua_driver_core::element_token::ResolvedElement::Element {
- window_id: wid, element_index: idx, via_token: _,
+ window_id: wid,
+ element_index: idx,
+ via_token: _,
} => (Some(idx), wid),
};
- let delay_ms = args.u64_or("delay_ms", 30);
+ let delay_ms = args.u64_or("delay_ms", 30);
let delivery_mode = super::DeliveryMode::parse(args.opt_str("delivery_mode").as_deref());
// ── px form: focus by pixel-click, then type into the focused element ──
@@ -165,14 +209,26 @@ impl Tool for TypeTextTool {
if let (Some(cx), Some(cy)) = (px, py) {
if element_index.is_some() {
return ToolResult::error(
- "Pass either element_index (ax) or x,y (px) to type_text, not both."
+ "Pass either element_index (ax) or x,y (px) to type_text, not both.",
);
}
- let from_zoom = args.get("from_zoom").and_then(|v| v.as_bool()).unwrap_or(false);
+ let from_zoom = args
+ .get("from_zoom")
+ .and_then(|v| v.as_bool())
+ .unwrap_or(false);
if let Err(e) = super::focus_by_pixel(
- &self.state, pid, window_id, cx, cy, delivery_mode.is_foreground(),
- args.opt_str("session"), args.opt_str("_session_id"), from_zoom,
- ).await {
+ &self.state,
+ pid,
+ window_id,
+ cx,
+ cy,
+ delivery_mode.is_foreground(),
+ args.opt_str("session"),
+ args.opt_str("_session_id"),
+ from_zoom,
+ )
+ .await
+ {
return e;
}
// element_index stays None → the type path below writes to the now-
@@ -182,9 +238,7 @@ impl Tool for TypeTextTool {
// Validate element_index requires window_id (still applies for
// the legacy integer path; token path already resolved window_id).
if element_index.is_some() && window_id.is_none() {
- return ToolResult::error(
- "window_id is required when element_index is used."
- );
+ return ToolResult::error("window_id is required when element_index is used.");
}
// Resolve the element pointer (if element_index given). Retain it out
@@ -194,17 +248,21 @@ impl Tool for TypeTextTool {
let element_guard = if let (Some(idx), Some(wid)) = (element_index, window_id) {
match self.state.element_cache.get_element_retained(pid, wid, idx) {
Some(e) => Some((e, idx)),
- None => return ToolResult::error(format!(
- "Element index {idx} not found. Call get_window_state first."
- )),
+ None => {
+ return ToolResult::error(format!(
+ "Element index {idx} not found. Call get_window_state first."
+ ))
+ }
}
} else {
None
};
- let element_ptr = element_guard.as_ref().map(|(g, idx)| (g.as_ptr(), Some(*idx)));
+ let element_ptr = element_guard
+ .as_ref()
+ .map(|(g, idx)| (g.as_ptr(), Some(*idx)));
- let text_clone = text.clone();
- let char_count = text.chars().count();
+ let text_clone = text.clone();
+ let char_count = text.chars().count();
// ── Focus-suppression wrap (Swift WindowChangeDetector + FocusGuard) ──
// Typing into a field can trigger autocomplete popovers or
@@ -256,9 +314,8 @@ impl Tool for TypeTextTool {
// browser's OWN native chrome (address bar, toolbar) stays trusted.
// Probe ONLY when it would otherwise confirm via the AX path, so
// native types pay nothing.
- let ax_echo_surface = verified
- && path == PATH_AX
- && target_in_web_area(pid, element_ptr);
+ let ax_echo_surface =
+ verified && path == PATH_AX && target_in_web_area(pid, element_ptr);
let verified = verified && !ax_echo_surface;
// `verified:false` means the driver could not confirm the text
@@ -269,20 +326,28 @@ impl Tool for TypeTextTool {
let (mark, note) = if verified {
("✅ Inserted", String::new())
} else if ax_echo_surface {
- ("📨 Sent (unverified)",
- " — web-content surface (Chromium / WebKit / Electron): the AX \
+ (
+ "📨 Sent (unverified)",
+ " — web-content surface (Chromium / WebKit / Electron): the AX \
layer accepts and echoes the write but the renderer/DOM may not \
have observed it, so the driver cannot confirm via AX. Verify \
via the screenshot. For a browser tab use the `page` tool (it \
drives the DOM); for an embedded web view, re-type with the px \
- form (x,y).".to_string())
+ form (x,y)."
+ .to_string(),
+ )
} else if path == PATH_KEY_EVENTS_FG {
- ("📨 Sent (unverified)",
- " — driver could not confirm; verify via screenshot.".to_string())
+ (
+ "📨 Sent (unverified)",
+ " — driver could not confirm; verify via screenshot.".to_string(),
+ )
} else {
- ("📨 Sent (unverified)",
- " — driver could not confirm the text landed; verify via screenshot, \
- and re-call with delivery_mode:\"foreground\" if it didn't.".to_string())
+ (
+ "📨 Sent (unverified)",
+ " — driver could not confirm the text landed; verify via screenshot, \
+ and re-call with delivery_mode:\"foreground\" if it didn't."
+ .to_string(),
+ )
};
ToolResult::text(format!(
"{mark} {char_count} char(s){detail}.{note}{}",
@@ -306,15 +371,18 @@ impl Tool for TypeTextTool {
// renderer/DOM-focus problem, never a foreground one.
let (recommended, reason) =
if crate::browser::electron_js::ElectronJs::is_electron(pid) {
- ("px",
- "Electron web view — the AX write was echoed but the \
+ (
+ "px",
+ "Electron web view — the AX write was echoed but the \
renderer may not have observed it. Confirm via the \
screenshot; if it didn't land, re-type with the \
element px action (x,y to pixel-focus the field, then \
- type).")
+ type).",
+ )
} else {
- ("page",
- "Browser web content — the AX write was echoed but the \
+ (
+ "page",
+ "Browser web content — the AX write was echoed but the \
DOM may not have observed it (and AX type_text on a \
contenteditable is racy). Drive the tab's DOM with the \
`page` tool: execute_javascript + el.value/innerText for a \
@@ -323,7 +391,8 @@ impl Tool for TypeTextTool {
a one-shot DOM write on their next render) try insert_text \
first (one CDP call, cheap), then type_keystrokes if that \
also gets discarded (real per-character keyboard events, \
- slower but most durable). Or confirm via the screenshot.")
+ slower but most durable). Or confirm via the screenshot.",
+ )
};
s["escalation"] = serde_json::json!({
"recommended": recommended,
@@ -341,7 +410,7 @@ impl Tool for TypeTextTool {
})
}
Ok(Err(e)) => ToolResult::error(format!("type_text failed: {e}")),
- Err(e) => ToolResult::error(format!("Task error: {e}")),
+ Err(e) => ToolResult::error(format!("Task error: {e}")),
}
}
}
@@ -367,10 +436,11 @@ const PATH_KEY_EVENTS_FG: &str = "key_events_fg";
/// substring/length test even though something landed — we report `false`
/// (unverified) rather than erroring, so the agent can still confirm.
fn verify_typed(before: Option<&str>, after: Option<&str>, text: &str) -> bool {
- if text.is_empty() { return true; }
+ if text.is_empty() {
+ return true;
+ }
let Some(after) = after else { return false };
- after.contains(text)
- || before.map_or(false, |b| after.chars().count() > b.chars().count())
+ after.contains(text) || before.map_or(false, |b| after.chars().count() > b.chars().count())
}
/// Read the focused/target field's `AXValue`, for before/after read-back.
@@ -381,7 +451,9 @@ fn read_axvalue(pid: i32, element_ptr_and_idx: Option<(usize, Option<usize>)>) -
unsafe { copy_string_attr(ptr as AXUIElementRef, "AXValue") }
} else if let Some(el) = unsafe { focused_element_of_pid(pid) } {
let v = unsafe { copy_string_attr(el, "AXValue") };
- unsafe { CFRelease(el as _); }
+ unsafe {
+ CFRelease(el as _);
+ }
v
} else {
None
@@ -416,16 +488,20 @@ fn target_in_web_area(pid: i32, element_ptr_and_idx: Option<(usize, Option<usize
let mut found = false;
for _ in 0..40 {
match copy_string_attr(cur, "AXRole").as_deref() {
- Some("AXWebArea") => { found = true; break; }
+ Some("AXWebArea") => {
+ found = true;
+ break;
+ }
// No web area lives above the window/app root — stop.
Some("AXWindow") | Some("AXApplication") | None => break,
_ => {}
}
let mut parent: CFTypeRef = std::ptr::null_mut();
- let err = AXUIElementCopyAttributeValue(
- cur, parent_attr.as_concrete_TypeRef(), &mut parent,
- );
- if cur_owned { CFRelease(cur as CFTypeRef); }
+ let err =
+ AXUIElementCopyAttributeValue(cur, parent_attr.as_concrete_TypeRef(), &mut parent);
+ if cur_owned {
+ CFRelease(cur as CFTypeRef);
+ }
if err != kAXErrorSuccess || parent.is_null() {
cur = std::ptr::null_mut();
cur_owned = false;
@@ -528,10 +604,17 @@ fn type_text_blocking(
// the remote host over hundreds of ms, so at 60ms every keystroke was
// dropped. 200ms covers that re-grab without being perceptible.
const FOREGROUND_SETTLE_MS: u64 = 200;
- let do_type = || cgevent_type_verified(
- pid, text, delay_ms, before.as_deref(), clear_first, element_ptr_and_idx,
- FOREGROUND_SETTLE_MS,
- );
+ let do_type = || {
+ cgevent_type_verified(
+ pid,
+ text,
+ delay_ms,
+ before.as_deref(),
+ clear_first,
+ element_ptr_and_idx,
+ FOREGROUND_SETTLE_MS,
+ )
+ };
let (verified, fronted) = match window_id {
Some(wid) => {
// Front → type → restore. The closure returns the read-back
@@ -557,7 +640,11 @@ fn type_text_blocking(
// background keystrokes and `path` must say so honestly.
return Ok((
format!(" via foreground keystrokes ({delay_ms}ms delay)"),
- if fronted { PATH_KEY_EVENTS_FG } else { PATH_KEY_EVENTS },
+ if fronted {
+ PATH_KEY_EVENTS_FG
+ } else {
+ PATH_KEY_EVENTS
+ },
verified,
));
}
@@ -569,7 +656,12 @@ fn type_text_blocking(
using CGEvent key-event synthesis"
);
let verified = cgevent_type_verified(
- pid, text, delay_ms, before.as_deref(), /*clear_first=*/ false, element_ptr_and_idx,
+ pid,
+ text,
+ delay_ms,
+ before.as_deref(),
+ /*clear_first=*/ false,
+ element_ptr_and_idx,
/*settle_ms=*/ 0,
)?;
return Ok((
@@ -585,7 +677,7 @@ fn type_text_blocking(
None => unsafe { focused_element_of_pid(pid) }.map(|el| (el, /*owns=*/ true, None)),
};
if let Some((element, owns, idx_opt)) = ax_target {
- let role = unsafe { copy_string_attr(element, "AXRole") }.unwrap_or_default();
+ let role = unsafe { copy_string_attr(element, "AXRole") }.unwrap_or_default();
let title = unsafe { copy_string_attr(element, "AXTitle") }.unwrap_or_default();
let err = unsafe { set_string_attr(element, "AXSelectedText", text) };
// Landed iff the API succeeded AND a read-back positively confirms the
@@ -595,9 +687,13 @@ fn type_text_blocking(
// its old (non-empty) value, which would otherwise report a false
// success. `verify_typed` compares the read-back against `before`/`text`.
let after = unsafe { copy_string_attr(element, "AXValue") };
- let ax_landed = err == kAXErrorSuccess
- && verify_typed(before.as_deref(), after.as_deref(), text);
- if owns { unsafe { CFRelease(element as _); } }
+ let ax_landed =
+ err == kAXErrorSuccess && verify_typed(before.as_deref(), after.as_deref(), text);
+ if owns {
+ unsafe {
+ CFRelease(element as _);
+ }
+ }
if ax_landed {
let idx_str = idx_opt.map(|i| format!(" [{i}]")).unwrap_or_default();
return Ok((format!(" into{idx_str} {role} \"{title}\""), PATH_AX, true));
@@ -615,7 +711,12 @@ fn type_text_blocking(
// background fallback would change insert-at-cursor semantics. The
// foreground rung owns the idempotent clear-then-type.
let verified = cgevent_type_verified(
- pid, text, delay_ms, before.as_deref(), /*clear_first=*/ false, element_ptr_and_idx,
+ pid,
+ text,
+ delay_ms,
+ before.as_deref(),
+ /*clear_first=*/ false,
+ element_ptr_and_idx,
/*settle_ms=*/ 0,
)?;
Ok((
@@ -648,8 +749,13 @@ mod tests {
// CGEvent. The fact that this returns an Err (without crashing)
// proves we routed through CGEvent-only and never touched AX.
let r = type_text_blocking(
- -1, "x", None, 0, /*is_terminal_target=*/ true,
- super::super::DeliveryMode::Background, None,
+ -1,
+ "x",
+ None,
+ 0,
+ /*is_terminal_target=*/ true,
+ super::super::DeliveryMode::Background,
+ None,
);
// We don't care whether r is Ok or Err — what matters is that
// calling it with is_terminal_target=true is safe and never
@@ -666,9 +772,9 @@ mod tests {
#[test]
fn verify_typed_contains_or_grew_is_verified() {
- assert!(verify_typed(Some(""), Some("hi"), "hi")); // contains
- assert!(verify_typed(Some("ab"), Some("ab hi"), "hi")); // contains, appended
- assert!(verify_typed(Some("ab"), Some("abXY"), "??")); // grew vs before
+ assert!(verify_typed(Some(""), Some("hi"), "hi")); // contains
+ assert!(verify_typed(Some("ab"), Some("ab hi"), "hi")); // contains, appended
+ assert!(verify_typed(Some("ab"), Some("abXY"), "??")); // grew vs before
}
#[test]
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs b/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs
index d370855657..320c528238 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs
@@ -834,6 +834,22 @@ pub fn send_wheel_synthesized(sx: i32, sy: i32, ticks: i32, horizontal: bool) ->
Ok(())
}
+/// Return the current foreground window for desktop-scope keyboard and drag
+/// operations, which intentionally target whatever the user can currently see.
+pub fn foreground_window() -> Result<u64> {
+ let hwnd = unsafe { GetForegroundWindow() };
+ if hwnd.0.is_null() {
+ bail!("no foreground window is available");
+ }
+ Ok(hwnd.0 as u64)
+}
+
+/// Move the real OS pointer to a desktop coordinate.
+pub fn move_cursor_desktop(x: i32, y: i32) -> Result<()> {
+ unsafe { SetCursorPos(x, y) }
+ .map_err(|error| anyhow::anyhow!("SetCursorPos({x}, {y}) failed: {error}"))
+}
+
#[cfg(test)]
mod wheel_tests {
use super::{posted_press_message, wheel_mouse_data, WHEEL_DELTA};
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
index 78551bad0b..2c41e7e684 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
@@ -247,14 +247,13 @@ pub(crate) fn resolve_cursor_key(args: &Value) -> String {
/// Returns `true` when a click/scroll invocation should take the **window-less
/// screen-absolute** branch: the caller gave numeric `x` AND `y`, gave NO `pid`
-/// and NO `window_id`, and the effective `capture_scope` is `"desktop"`.
+/// and NO `window_id`, and explicitly selected `scope="desktop"`.
///
/// Pure + arg-shape agnostic (works for both click and scroll args) so it's
/// unit-testable without Win32. When this returns `false` but `x,y` are present
-/// with no pid/window_id, the caller returns a `desktop_scope_disabled`
-/// structured error telling the agent to `set_config capture_scope=desktop`.
-fn is_windowless_desktop_action(args: &serde_json::Value, scope: &str) -> bool {
- if scope != "desktop" {
+/// with no pid/window_id, the caller returns a coordinate-scope error.
+fn is_windowless_desktop_action(args: &serde_json::Value) -> bool {
+ if args.get("scope").and_then(Value::as_str) != Some("desktop") {
return false;
}
let has_pid = args.get("pid").map(|v| !v.is_null()).unwrap_or(false);
@@ -271,10 +270,6 @@ fn is_windowless_desktop_action(args: &serde_json::Value, scope: &str) -> bool {
#[derive(Clone)]
pub struct DriverConfig {
pub capture_mode: String,
- /// Capture scope for vision loops: `"window"` (per-window, the default) or
- /// `"desktop"` (full-display). Gates the window-less screen-absolute
- /// click/scroll branches — those activate only under `"desktop"`.
- pub capture_scope: String,
pub max_image_dimension: u32,
}
@@ -282,7 +277,6 @@ impl Default for DriverConfig {
fn default() -> Self {
Self {
capture_mode: "ax".into(),
- capture_scope: "window".into(),
max_image_dimension: 1568,
}
}
@@ -302,11 +296,6 @@ pub fn load_driver_config() -> DriverConfig {
{
cfg.capture_mode = v;
}
- if let Some(v) =
- pip_preview::read_config_value("capture_scope").and_then(|v| v.as_str().map(str::to_owned))
- {
- cfg.capture_scope = v;
- }
if let Some(v) = pip_preview::read_config_value("max_image_dimension").and_then(|v| v.as_u64())
{
if let Ok(v32) = u32::try_from(v) {
@@ -2469,7 +2458,7 @@ impl Tool for ClickTool {
a visible on-screen window to anchor the coordinate conversion (errors \
with `pid X has no on-screen window` otherwise).\n\n\
Exactly one of `element_index` or (`x` AND `y`) must be provided. \
- `pid` is required in both modes. `window_id` is required when \
+ `pid` is required for window scope and omitted for desktop scope. `window_id` is required when \
`element_index` is used (scopes the cache lookup). After a `zoom` call, \
pass `from_zoom=true` to auto-translate zoom-image coords back to \
full-window space.\n\n\
@@ -2480,7 +2469,7 @@ impl Tool for ClickTool {
input_schema: json!({
"type":"object","properties":{
"session": cua_driver_core::tool_schema::session_schema(),
- "pid":{"type":"integer","description":"Target process ID. Required UNLESS using window-less desktop-scope clicks: with capture_scope=\"desktop\" and no pid/window_id, x/y are treated as TRUE SCREEN pixels (call get_desktop_state first)."},
+ "pid":{"type":"integer","description":"Target process ID for window scope. Omit with scope=desktop for screen-absolute coordinates from get_desktop_state."},
"window_id":{"type":"integer","description":"HWND for the window whose get_window_state produced the element_index. Required when element_index is used. Optional when element_token is supplied (the token carries it)."},
"element_index": cua_driver_core::tool_schema::element_index_schema(),
"element_token": cua_driver_core::tool_schema::element_token_schema(),
@@ -2490,6 +2479,7 @@ impl Tool for ClickTool {
"count":{"type":"integer","minimum":1,"maximum":3,"description":"Click count — 1 (single), 2 (double), 3 (triple). Default 1."},
"modifier": cua_driver_core::tool_schema::modifier_schema(),
"from_zoom":{"type":"boolean","description":"When true, x and y are pixel coordinates in the last `zoom` image for this pid. The driver maps them back to window coords."},
+ "scope":{"type":"string","enum":["window","desktop"],"default":"window"},
"delivery_mode": crate::input::delivery::delivery_mode_schema()
},"additionalProperties":false
}),
@@ -2505,26 +2495,21 @@ impl Tool for ClickTool {
// ── Window-less screen-absolute branch (capture_scope="desktop") ──────
// When the caller gives x,y with NO pid/window_id, treat x,y as TRUE
- // SCREEN pixels. Gate on effective capture_scope: only "desktop" enables
- // this; under "window" we return a structured `desktop_scope_disabled`
- // error pointing the caller at set_config.
+ // SCREEN pixels. The core registry has already enforced the session's
+ // effective capture scope; this local check validates the action form.
let has_pid = args.get("pid").map(|v| !v.is_null()).unwrap_or(false);
let has_window_id = args.get("window_id").map(|v| !v.is_null()).unwrap_or(false);
let has_xy = args.get("x").map(|v| v.is_number()).unwrap_or(false)
&& args.get("y").map(|v| v.is_number()).unwrap_or(false);
if !has_pid && !has_window_id && has_xy {
- let scope = self.state.config.read().unwrap().capture_scope.clone();
- if !is_windowless_desktop_action(&args, &scope) {
+ if !is_windowless_desktop_action(&args) {
return ToolResult::error(
- "click: x,y given with no pid/window_id, but capture_scope is \
- \"window\". Screen-absolute clicks require desktop scope. Call \
- set_config with capture_scope=desktop (and use get_desktop_state \
- to pick coordinates), or pass a pid/window_id.",
+ "click: x,y given with no pid/window_id requires scope=\"desktop\"; \
+ use get_desktop_state to pick coordinates, or pass a pid/window_id.",
)
.with_structured(json!({
- "code": "desktop_scope_disabled",
- "capture_scope": scope,
- "suggestion": "set_config capture_scope=desktop",
+ "code": "desktop_coordinate_scope_required",
+ "suggestion": "pass scope=desktop",
}));
}
// Resolve button (reuse the same validation as the pid path).
@@ -3465,8 +3450,9 @@ impl Tool for TypeTextTool {
PostMessage path so autocomplete and IME can keep up. Ignored on the \
UIA path (SetValue is atomic).".into(),
input_schema: json!({
- "type":"object","required":["pid","text"],"properties":{
+ "type":"object","required":["text"],"properties":{
"session": cua_driver_core::tool_schema::session_schema(),
+ "scope":{"type":"string","enum":["window","desktop"],"description":"Use desktop with no pid/window_id to type into the current foreground application."},
"pid":{"type":"integer","description":"Target process ID."},
"text":{"type":"string","description":"Text to insert at the focused element's cursor."},
"window_id":{"type":"integer","description":"HWND of the target window. Required when element_index is used. Optional when element_token is supplied (the token carries it)."},
@@ -3485,11 +3471,6 @@ impl Tool for TypeTextTool {
async fn invoke(&self, args: Value) -> ToolResult {
use crate::input::delivery::{DeliveryMode, EventKind};
use cua_driver_core::tool_args::ArgsExt;
- let raw_pid = match args.require_i64("pid") {
- Ok(v) => v,
- Err(e) => return e,
- };
- let pid = raw_pid as u32;
let text_raw = match args.require_str("text") {
Ok(v) => v,
Err(e) => return e,
@@ -3502,6 +3483,36 @@ impl Tool for TypeTextTool {
// case is allocation-free.
let text = cua_driver_core::text_sanitize::strip_trailing_agent_protocol_tags(&text_raw)
.into_owned();
+ if args.get("scope").and_then(Value::as_str) == Some("desktop")
+ && args.get("pid").is_none()
+ && args.get("window_id").is_none()
+ {
+ let hwnd = match crate::input::mouse::foreground_window() {
+ Ok(hwnd) => hwnd,
+ Err(error) => return ToolResult::error(error.to_string()),
+ };
+ let text_len = text.chars().count();
+ return match tokio::task::spawn_blocking(move || {
+ crate::input::send_text_synthesized(hwnd, &text)
+ })
+ .await
+ {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Typed {text_len} character(s) into the foreground application (scope:desktop)."
+ ))
+ .with_structured(json!({
+ "scope": "desktop", "path": "SendInput", "characters": text_len,
+ "effect": "unverifiable"
+ })),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("desktop type task failed: {error}")),
+ };
+ }
+ let raw_pid = match args.require_i64("pid") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ let pid = raw_pid as u32;
// Surface 6: element_token / element_index precedence resolution.
let resolved = match cua_driver_core::element_token::resolve_element_args(
pid as i32,
@@ -4101,8 +4112,9 @@ impl Tool for PressKeyTool {
`element_index` focuses the cached UIA element before sending the key; the \
top-level window remains backgrounded when delivery_mode is background.".into(),
input_schema: json!({
- "type":"object","required":["pid","key"],"properties":{
+ "type":"object","required":["key"],"properties":{
"session": cua_driver_core::tool_schema::session_schema(),
+ "scope":{"type":"string","enum":["window","desktop"],"description":"Use desktop with no pid/window_id to send the key to the current foreground application."},
"pid":{"type":"integer","description":"Target process ID."},
"key":{"type":"string","description":"Key name (return, tab, escape, up, down, left, right, space, delete, home, end, pageup, pagedown, f1-f12, letter, digit)."},
"modifiers":{"type":"array","items":{"type":"string"},"description":"Optional modifier names held while the key is pressed (ctrl/shift/alt/win)."},
@@ -4121,16 +4133,42 @@ impl Tool for PressKeyTool {
async fn invoke(&self, args: Value) -> ToolResult {
use crate::input::delivery::{DeliveryMode, EventKind};
use cua_driver_core::tool_args::ArgsExt;
- let raw_pid = match args.require_i64("pid") {
+ let key = match args.require_str("key") {
Ok(v) => v,
Err(e) => return e,
};
- let pid = raw_pid as u32;
- let key = match args.require_str("key") {
+ let mods: Vec<String> = args.str_array("modifiers");
+ if args.get("scope").and_then(Value::as_str) == Some("desktop")
+ && args.get("pid").is_none()
+ && args.get("window_id").is_none()
+ {
+ let hwnd = match crate::input::mouse::foreground_window() {
+ Ok(hwnd) => hwnd,
+ Err(error) => return ToolResult::error(error.to_string()),
+ };
+ let key_display = key.clone();
+ return match tokio::task::spawn_blocking(move || {
+ let modifiers: Vec<&str> = mods.iter().map(String::as_str).collect();
+ crate::input::send_key_synthesized(hwnd, &key, &modifiers)
+ })
+ .await
+ {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Sent {key_display} to the foreground application (scope:desktop)."
+ ))
+ .with_structured(json!({
+ "scope": "desktop", "path": "SendInput", "key": key_display,
+ "effect": "unverifiable"
+ })),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("desktop key task failed: {error}")),
+ };
+ }
+ let raw_pid = match args.require_i64("pid") {
Ok(v) => v,
Err(e) => return e,
};
- let mods: Vec<String> = args.str_array("modifiers");
+ let pid = raw_pid as u32;
// Surface 6: element_token / element_index precedence resolution.
let resolved = match cua_driver_core::element_token::resolve_element_args(
pid as i32,
@@ -4416,8 +4454,9 @@ impl Tool for HotkeyTool {
up/down/left/right, space, delete, home, end, pageup, pagedown, f1-f12, \
letters, digits). Order: modifiers first, one non-modifier last.".into(),
input_schema: json!({
- "type":"object","required":["pid","keys"],"properties":{
+ "type":"object","required":["keys"],"properties":{
"session": cua_driver_core::tool_schema::session_schema(),
+ "scope":{"type":"string","enum":["window","desktop"],"description":"Use desktop with no pid/window_id to send the hotkey to the current foreground application."},
"pid":{"type":"integer","description":"Target process ID."},
"keys":{"type":"array","items":{"type":"string"},"minItems":2,
"description":"Modifier(s) and one non-modifier key, e.g. [\"ctrl\", \"c\"]."},
@@ -4434,14 +4473,6 @@ impl Tool for HotkeyTool {
async fn invoke(&self, args: Value) -> ToolResult {
use crate::input::delivery::{DeliveryMode, EventKind};
use cua_driver_core::tool_args::ArgsExt;
- let hwnd_opt = args.opt_u64("window_id");
- let raw_pid = match args.require_i64("pid") {
- Ok(v) => v,
- Err(e) => return e,
- };
- let pid = raw_pid as u32;
- let delivery = DeliveryMode::from_args(&args);
-
// Parse keys array (Swift's only path). Legacy key+modifiers shape
// still accepted as a Rust-side convenience.
let (key, mods, full_keys) = if let Some(arr) = args.get("keys") {
@@ -4478,6 +4509,39 @@ impl Tool for HotkeyTool {
} else {
return ToolResult::error("Missing required array field keys.");
};
+ if args.get("scope").and_then(Value::as_str) == Some("desktop")
+ && args.get("pid").is_none()
+ && args.get("window_id").is_none()
+ {
+ let hwnd = match crate::input::mouse::foreground_window() {
+ Ok(hwnd) => hwnd,
+ Err(error) => return ToolResult::error(error.to_string()),
+ };
+ let key_display = full_keys.join("+");
+ return match tokio::task::spawn_blocking(move || {
+ let modifiers: Vec<&str> = mods.iter().map(String::as_str).collect();
+ crate::input::send_key_synthesized(hwnd, &key, &modifiers)
+ })
+ .await
+ {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Sent {key_display} to the foreground application (scope:desktop)."
+ ))
+ .with_structured(json!({
+ "scope": "desktop", "path": "SendInput", "keys": full_keys,
+ "effect": "unverifiable"
+ })),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("desktop hotkey task failed: {error}")),
+ };
+ }
+ let hwnd_opt = args.opt_u64("window_id");
+ let raw_pid = match args.require_i64("pid") {
+ Ok(v) => v,
+ Err(e) => return e,
+ };
+ let pid = raw_pid as u32;
+ let delivery = DeliveryMode::from_args(&args);
// Resolve HWND: use window_id if given, else pick first window for pid.
let hwnd = match hwnd_opt {
@@ -4890,7 +4954,7 @@ impl Tool for ScrollTool {
input_schema: json!({
"type":"object","required":["direction"],"properties":{
"session": cua_driver_core::tool_schema::session_schema(),
- "pid":{"type":"integer","description":"Target process ID. Required UNLESS using window-less desktop-scope scroll: with capture_scope=\"desktop\" and no pid/window_id, x/y are TRUE SCREEN pixels and the wheel routes to the window under that point."},
+ "pid":{"type":"integer","description":"Target process ID for window scope. Omit with scope=desktop for screen-absolute coordinates from get_desktop_state."},
"direction":{"type":"string","enum":["up","down","left","right"]},
"by":{"type":"string","enum":["line","page"],"description":"Scroll granularity. Default: line."},
"amount":{"type":"integer","minimum":1,"maximum":50,
@@ -4900,6 +4964,7 @@ impl Tool for ScrollTool {
"window_id":{"type":"integer","description":"HWND of the target window. Required when element_index is used; otherwise auto-resolves the pid's first visible window."},
"element_index":{"type":"integer","description":"Optional element_index. Accepted for parity; currently no-op on Windows."},
"element_token": cua_driver_core::tool_schema::element_token_schema(),
+ "scope":{"type":"string","enum":["window","desktop"],"default":"window"},
"delivery_mode": crate::input::delivery::delivery_mode_schema()
},"additionalProperties":false
}),
@@ -4928,18 +4993,14 @@ impl Tool for ScrollTool {
let has_xy = args.get("x").map(|v| v.is_number()).unwrap_or(false)
&& args.get("y").map(|v| v.is_number()).unwrap_or(false);
if !has_pid && !has_window_id && has_xy {
- let scope = self.state.config.read().unwrap().capture_scope.clone();
- if !is_windowless_desktop_action(&args, &scope) {
+ if !is_windowless_desktop_action(&args) {
return ToolResult::error(
- "scroll: x,y given with no pid/window_id, but capture_scope is \
- \"window\". Screen-absolute scroll requires desktop scope. Call \
- set_config with capture_scope=desktop (and use get_desktop_state \
- to pick coordinates), or pass a pid/window_id.",
+ "scroll: x,y given with no pid/window_id requires scope=\"desktop\"; \
+ use get_desktop_state to pick coordinates, or pass a pid/window_id.",
)
.with_structured(serde_json::json!({
- "code": "desktop_scope_disabled",
- "capture_scope": scope,
- "suggestion": "set_config capture_scope=desktop",
+ "code": "desktop_coordinate_scope_required",
+ "suggestion": "pass scope=desktop",
}));
}
let sx = args.f64_or("x", 0.0) as i32;
@@ -6115,8 +6176,9 @@ impl Tool for DragTool {
description: "Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels. \
duration_ms (default 500) is the wall-clock budget; steps (default 20) interpolates intermediate \
WM_MOUSEMOVE events along the path. No focus steal. After a zoom call, pass from_zoom=true.".into(),
- input_schema: json!({"type":"object","required":["pid","from_x","from_y","to_x","to_y"],"properties":{
+ input_schema: json!({"type":"object","required":["from_x","from_y","to_x","to_y"],"properties":{
"session": cua_driver_core::tool_schema::session_schema(),
+ "scope":{"type":"string","enum":["window","desktop"],"description":"Use desktop with no pid/window_id for screen-absolute coordinates."},
"pid":{"type":"integer","description":"Target process ID."},
"window_id":{"type":"integer","description":"Target window handle (HWND). Optional — driver picks frontmost window of pid when omitted."},
"from_x":{"type":"number","description":"Drag-start X in window-local screenshot pixels."},
@@ -6135,6 +6197,57 @@ impl Tool for DragTool {
}
async fn invoke(&self, args: Value) -> ToolResult {
use crate::input::delivery::{DeliveryMode, EventKind};
+ use cua_driver_core::tool_args::ArgsExt;
+ if args.get("scope").and_then(Value::as_str) == Some("desktop")
+ && args.get("pid").is_none()
+ && args.get("window_id").is_none()
+ {
+ let required = |key: &str| {
+ args.opt_f64(key)
+ .or_else(|| args.opt_i64(key).map(|value| value as f64))
+ };
+ let (from_x, from_y, to_x, to_y) = match (
+ required("from_x"),
+ required("from_y"),
+ required("to_x"),
+ required("to_y"),
+ ) {
+ (Some(a), Some(b), Some(c), Some(d)) => (a as i32, b as i32, c as i32, d as i32),
+ _ => return ToolResult::error("from_x, from_y, to_x, and to_y are required."),
+ };
+ let duration_ms = args.u64_or("duration_ms", 500);
+ let steps = args.u64_or("steps", 20) as usize;
+ let button = args.str_or("button", "left");
+ let hwnd = match crate::input::mouse::foreground_window() {
+ Ok(hwnd) => hwnd,
+ Err(error) => return ToolResult::error(error.to_string()),
+ };
+ return match tokio::task::spawn_blocking(move || {
+ crate::input::mouse::send_drag_synthesized(
+ hwnd,
+ from_x,
+ from_y,
+ to_x,
+ to_y,
+ duration_ms,
+ steps,
+ &button,
+ )
+ })
+ .await
+ {
+ Ok(Ok(())) => ToolResult::text(format!(
+ "Dragged from ({from_x}, {from_y}) to ({to_x}, {to_y}) (scope:desktop)."
+ ))
+ .with_structured(json!({
+ "scope": "desktop", "path": "SendInput",
+ "from": {"x": from_x, "y": from_y}, "to": {"x": to_x, "y": to_y},
+ "effect": "unverifiable"
+ })),
+ Ok(Err(error)) => ToolResult::error(error.to_string()),
+ Err(error) => ToolResult::error(format!("desktop drag task failed: {error}")),
+ };
+ }
// Swift error wording 1:1.
let raw_pid = match args.get("pid").and_then(|v| v.as_i64()) {
Some(p) => p,
@@ -6143,7 +6256,6 @@ impl Tool for DragTool {
let pid = raw_pid as u32;
let delivery = DeliveryMode::from_args(&args);
- use cua_driver_core::tool_args::ArgsExt;
let cursor_key = resolve_cursor_key(&args);
// Accepts numeric JSON as either float or integer — coerce both to f64.
let coerce = |key: &str| -> Option<f64> {
@@ -6491,29 +6603,6 @@ impl Tool for GetDesktopStateTool {
use cua_driver_core::protocol::Content;
use cua_driver_core::tool_args::ArgsExt;
- // Gate on the global capture_scope: a full-display capture is a
- // desktop-scope operation, so it's only available when the agent has
- // opted into capture_scope="desktop" (same gate as window-less
- // screen-absolute click/scroll — see is_windowless_desktop_action).
- // Read the persisted value the same way load_config does.
- let scope = pip_preview::read_config_value("capture_scope")
- .and_then(|v| v.as_str().map(str::to_owned))
- .unwrap_or_else(|| "window".to_owned());
- if scope != "desktop" {
- return ToolResult::error(format!(
- "get_desktop_state requires capture_scope=\"desktop\" (current scope is \
- \"{scope}\"). Full-display capture is a desktop-scope operation; call \
- set_config with capture_scope=desktop first (it also enables window-less \
- screen-absolute click/scroll). For a single window, use \
- get_window_state(pid, window_id) instead."
- ))
- .with_structured(serde_json::json!({
- "code": "desktop_scope_disabled",
- "capture_scope": scope,
- "suggestion": "set_config capture_scope=desktop",
- }));
- }
-
let screenshot_out_file = args.opt_str("screenshot_out_file");
// True screen geometry in physical pixels (same space as the capture).
@@ -6555,10 +6644,15 @@ impl Tool for GetDesktopStateTool {
let mut structured = json!({
"platform": "windows",
+ "display": "primary",
"screenshot_width": screenshot_width,
"screenshot_height": screenshot_height,
"screen_width": screen_width,
"screen_height": screen_height,
+ "scale_factor": unsafe {
+ let dpi = windows::Win32::UI::HiDpi::GetDpiForSystem();
+ if dpi == 0 { 1.0 } else { dpi as f64 / 96.0 }
+ },
"screenshot_mime_type": "image/png",
});
if let Some(ref fp) = file_path {
@@ -6624,6 +6718,7 @@ impl Tool for MoveCursorTool {
.into(),
input_schema: json!({"type":"object","required":["x","y"],"properties":{
"session": cua_driver_core::tool_schema::session_schema(),
+ "scope":{"type":"string","enum":["window","desktop"],"description":"desktop moves the real OS pointer; window moves only the agent overlay."},
"x":{"type":"number"},"y":{"type":"number"},"cursor_id":{"type":"string"}
},"additionalProperties":false}),
read_only: false,
@@ -6636,6 +6731,17 @@ impl Tool for MoveCursorTool {
use cua_driver_core::tool_args::ArgsExt;
let x = args.f64_or("x", 0.0);
let y = args.f64_or("y", 0.0);
+ if args.get("scope").and_then(Value::as_str) == Some("desktop") {
+ return match crate::input::mouse::move_cursor_desktop(x as i32, y as i32) {
+ Ok(()) => ToolResult::text(format!(
+ "Moved the OS pointer to ({x:.1}, {y:.1}) (scope:desktop)."
+ ))
+ .with_structured(json!({
+ "scope": "desktop", "path": "SetCursorPos", "x": x, "y": y
+ })),
+ Err(error) => ToolResult::error(error.to_string()),
+ };
+ }
// Cursor key precedence: caller-declared `session` > legacy `cursor_id`
// > NO_CURSOR. An anonymous run (no session) has no cursor to move.
let cursor_key = resolve_cursor_key(&args);
@@ -7322,7 +7428,6 @@ impl Tool for GetConfigTool {
"source_sha": option_env!("CUA_DRIVER_SOURCE_SHA"),
"platform": "windows",
"capture_mode": cfg.capture_mode,
- "capture_scope": cfg.capture_scope,
"max_image_dimension": cfg.max_image_dimension,
"agent_cursor": { "enabled": cursor_enabled },
"experimental_pip": pip_enabled,
@@ -7368,7 +7473,6 @@ impl Tool for SetConfigTool {
"key":{"type":"string","description":"Dotted snake_case path to a leaf config field (Swift-compatible shape). Pair with `value`."},
"value":{"description":"New value for `key`. JSON type depends on the key."},
"capture_mode":{"type":"string","enum":["ax","vision"],"description":"DEPRECATED and ignored — get_window_state always returns both the UIA tree and a screenshot. Still accepted/persisted for back-compat but has no effect. (\"som\"/\"screenshot\" still decode as deprecated aliases.)"},
- "capture_scope":{"type":"string","enum":["window","desktop"],"description":"Capture scope: single window (default) or whole desktop. Desktop scope enables window-less screen-absolute click/scroll (no pid/window_id). Accepted in both the {key,value} and legacy per-field shapes."},
"max_image_dimension":{"type":"integer","description":"Legacy per-field shape."},
"experimental_pip":{"type":"boolean","description":"Legacy per-field shape. Enables PiP preview (applies next restart)."},
"experimental_pip_geometry":{"type":"string","description":"Legacy per-field shape. PiP window size + optional position."}
@@ -7377,6 +7481,18 @@ impl Tool for SetConfigTool {
})
}
async fn invoke(&self, args: Value) -> ToolResult {
+ if args.get("capture_scope").is_some()
+ || args.get("key").and_then(Value::as_str) == Some("capture_scope")
+ {
+ return ToolResult::error(
+ "config key 'capture_scope' is retired; pass capture_scope=auto|window|desktop to start_session",
+ )
+ .with_structured(json!({
+ "code": "config_key_retired",
+ "key": "capture_scope",
+ "replacement": "start_session.capture_scope",
+ }));
+ }
let mut cfg = self.state.config.write().unwrap();
let mut applied = false;
// Swift-compatible {key, value} shape.
@@ -7394,17 +7510,6 @@ impl Tool for SetConfigTool {
}
None => return ToolResult::error(format!("`capture_mode` must be a string, got {val}.")),
},
- "capture_scope" => match val.as_str() {
- Some(s @ ("window" | "desktop")) => {
- cfg.capture_scope = s.to_owned();
- if let Err(e) = pip_preview::write_config_key("capture_scope", Value::String(s.to_owned())) {
- tracing::warn!("set_config: failed to persist capture_scope: {e}");
- }
- applied = true;
- }
- Some(other) => return ToolResult::error(format!("`capture_scope` must be \"window\" or \"desktop\", got \"{other}\".")),
- None => return ToolResult::error(format!("`capture_scope` must be a string, got {val}.")),
- },
"max_image_dimension" => match val.as_u64() {
Some(n) => {
cfg.max_image_dimension = n as u32;
@@ -7439,7 +7544,7 @@ impl Tool for SetConfigTool {
None => return ToolResult::error(format!("`experimental_pip_geometry` must be a string, got {val}.")),
},
other => return ToolResult::error(format!(
- "Unknown config key `{other}`. Known: capture_mode, capture_scope, max_image_dimension, experimental_pip, experimental_pip_geometry."
+ "Unknown config key `{other}`. Known: capture_mode, max_image_dimension, experimental_pip, experimental_pip_geometry."
)),
}
}
@@ -7453,20 +7558,6 @@ impl Tool for SetConfigTool {
}
applied = true;
}
- if let Some(scope) = args.get("capture_scope").and_then(|v| v.as_str()) {
- if !matches!(scope, "window" | "desktop") {
- return ToolResult::error(format!(
- "`capture_scope` must be \"window\" or \"desktop\", got \"{scope}\"."
- ));
- }
- cfg.capture_scope = scope.to_owned();
- if let Err(e) =
- pip_preview::write_config_key("capture_scope", Value::String(scope.to_owned()))
- {
- tracing::warn!("set_config: failed to persist capture_scope: {e}");
- }
- applied = true;
- }
if let Some(dim) = args.get("max_image_dimension").and_then(|v| v.as_u64()) {
cfg.max_image_dimension = dim as u32;
if let Err(e) = pip_preview::write_config_key("max_image_dimension", Value::from(dim)) {
@@ -7521,7 +7612,6 @@ impl Tool for SetConfigTool {
"source_sha": option_env!("CUA_DRIVER_SOURCE_SHA"),
"platform": "windows",
"capture_mode": cfg.capture_mode,
- "capture_scope": cfg.capture_scope,
"max_image_dimension": cfg.max_image_dimension,
"agent_cursor": { "enabled": cursor_enabled },
"experimental_pip": pip_enabled,
@@ -8888,7 +8978,7 @@ mod click_button_schema_tests {
#[cfg(test)]
mod desktop_scope_tests {
- use super::{is_windowless_desktop_action, DriverConfig, GetDesktopStateTool};
+ use super::{is_windowless_desktop_action, GetDesktopStateTool};
use cua_driver_core::tool::Tool;
use serde_json::json;
@@ -8898,61 +8988,52 @@ mod desktop_scope_tests {
fn windowless_true_for_xy_under_desktop_scope_click_shape() {
// Click arg shape: {x, y}.
assert!(is_windowless_desktop_action(
- &json!({"x": 10, "y": 20}),
- "desktop"
+ &json!({"x": 10, "y": 20, "scope": "desktop"})
));
}
#[test]
fn windowless_true_for_xy_under_desktop_scope_scroll_shape() {
// Scroll arg shape: {direction, x, y}.
- assert!(is_windowless_desktop_action(
- &json!({"direction": "down", "x": 10, "y": 20}),
- "desktop"
- ));
+ assert!(is_windowless_desktop_action(&json!({
+ "direction": "down", "x": 10, "y": 20, "scope": "desktop"
+ })));
}
#[test]
fn windowless_false_when_pid_present() {
- assert!(!is_windowless_desktop_action(
- &json!({"x": 10, "y": 20, "pid": 5}),
- "desktop"
- ));
+ assert!(!is_windowless_desktop_action(&json!({
+ "x": 10, "y": 20, "pid": 5, "scope": "desktop"
+ })));
}
#[test]
fn windowless_false_when_window_id_present() {
- assert!(!is_windowless_desktop_action(
- &json!({"x": 10, "y": 20, "window_id": 99}),
- "desktop"
- ));
+ assert!(!is_windowless_desktop_action(&json!({
+ "x": 10, "y": 20, "window_id": 99, "scope": "desktop"
+ })));
}
#[test]
fn windowless_false_under_window_scope() {
- assert!(!is_windowless_desktop_action(
- &json!({"x": 10, "y": 20}),
- "window"
- ));
+ assert!(!is_windowless_desktop_action(&json!({
+ "x": 10, "y": 20, "scope": "window"
+ })));
}
#[test]
fn windowless_false_when_xy_missing() {
- assert!(!is_windowless_desktop_action(&json!({"x": 10}), "desktop"));
- assert!(!is_windowless_desktop_action(&json!({"y": 20}), "desktop"));
- assert!(!is_windowless_desktop_action(&json!({}), "desktop"));
- // Non-numeric x/y must not qualify.
assert!(!is_windowless_desktop_action(
- &json!({"x": "10", "y": "20"}),
- "desktop"
+ &json!({"x": 10, "scope": "desktop"})
));
- }
-
- // ── DriverConfig default ──────────────────────────────────────────────────
-
- #[test]
- fn default_capture_scope_is_window() {
- assert_eq!(DriverConfig::default().capture_scope, "window");
+ assert!(!is_windowless_desktop_action(
+ &json!({"y": 20, "scope": "desktop"})
+ ));
+ assert!(!is_windowless_desktop_action(&json!({"scope": "desktop"})));
+ // Non-numeric x/y must not qualify.
+ assert!(!is_windowless_desktop_action(&json!({
+ "x": "10", "y": "20", "scope": "desktop"
+ })));
}
// ── get_desktop_state schema ──────────────────────────────────────────────
diff --git a/libs/cua-driver/tests/runners/macos-lume/README.md b/libs/cua-driver/tests/runners/macos-lume/README.md
index b4a9f71e74..97acb1999e 100644
--- a/libs/cua-driver/tests/runners/macos-lume/README.md
+++ b/libs/cua-driver/tests/runners/macos-lume/README.md
@@ -175,10 +175,12 @@ osascript -e \
~/.local/bin/cua-driver list_apps '{}'
# A desktop screenshot triggers Tahoe's direct-capture/private-window prompt.
-~/.local/bin/cua-driver call set_config '{"capture_scope":"desktop"}'
-~/.local/bin/cua-driver call get_desktop_state '{}' \
+~/.local/bin/cua-driver call start_session \
+ '{"session":"seed-desktop-consent","capture_scope":"desktop"}'
+~/.local/bin/cua-driver call get_desktop_state \
+ '{"session":"seed-desktop-consent"}' \
> /tmp/cua-driver-seed-desktop-state.json
-~/.local/bin/cua-driver call set_config '{"capture_scope":"window"}'
+~/.local/bin/cua-driver call end_session '{"session":"seed-desktop-consent"}'
jq -e '
.screenshot_mime_type == "image/png"
and .screenshot_width > 0