Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(driver): add per-session capture scope (#2329)
Francesco Bonaccidb0ee870c200
diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdxindex 726273d5ef..f6afe0c5bc 100644--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx@@ -12,7 +12,7 @@ description: Reference for every MCP tool Cua Driver exposesimport { Callout } from 'fumadocs-ui/components/callout';-`cua-driver` exposes 47 MCP tools through a single stdio server (`cua-driver mcp`). Every tool is also callable from the shell as `cua-driver <name> '<JSON-args>'`.+`cua-driver` exposes 49 MCP tools through a single stdio server (`cua-driver mcp`). Every tool is also callable from the shell as `cua-driver <name> '<JSON-args>'`.Tool names are `snake_case`. Responses are MCP `CallTool.Result` envelopes: a text content block prefixed with a `✅` summary (or the error reason on failure), plus optional image or structured-content blocks on tools that produce them. See the [CLI reference](/reference/cua-driver/cli-reference) for CLI-specific options like `--socket` and `--screenshot-out-file`.@@ -292,7 +292,8 @@ When `from_zoom` is true, coordinates are in the last zoom image for this pid; t- `from_y` (number, required): Drag-start Y in window-local screenshot pixels. Top-left origin.- `from_zoom` (boolean, optional): When true, coordinates are in the last zoom image for this pid; driver maps back to window coordinates.- `modifier` (array of string, optional): Modifier keys held across the entire gesture: cmd/shift/option/ctrl.-- `pid` (integer, required): Target process ID.+- `pid` (integer, optional): Target process ID.+- `scope` (string, optional): Use desktop with no pid/window_id for native get_desktop_state screenshot coordinates.- `session` (string, optional): Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less.- `steps` (integer, optional): Number of intermediate mouseDragged events linearly interpolated along the path. Default: 20.- `to_x` (number, required): Drag-end X in window-local screenshot pixels.@@ -300,7 +301,7 @@ When `from_zoom` is true, coordinates are in the last zoom image for this pid; t- `window_id` (integer, optional): CGWindowID for the window the pixel coordinates were measured against. Optional — when omitted the driver picks the frontmost window of pid.```json-{"from_x":100,"from_y":200,"pid":844,"to_x":100,"to_y":200}+{"from_x":100,"from_y":200,"to_x":100,"to_y":200}```### `type_text`@@ -317,7 +318,8 @@ WEB CONTENT (Chromium/WebKit/Electron — browser tabs, Slack, VS Code, X's comp- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": AX insert, then CGEvent keystrokes if needed — no focus steal; the driver verifies via an AXValue read-back and reports `verified`. "foreground": briefly front the window, type, restore the prior frontmost — the explicit last resort for focus-sensitive surfaces (e.g. WhatsApp/Catalyst) where background keystrokes don't land. Re-call with "foreground" when a background attempt returns `verified:false` and a screenshot shows the text didn't appear.- `element_index` (integer, optional): Element index from last get_window_state. Directs the write to a specific field. REQUIRES `pid` and `window_id` to be passed alongside it — element_index alone (no pid) fails fast with "Missing required integer field: pid"; it is not a silent no-op.- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit "stale" error if the snapshot has been superseded.-- `pid` (integer, required): Target process ID.+- `pid` (integer, optional): Target process ID.+- `scope` (string, optional): Use desktop with no pid/window_id to type into the frontmost application.- `session` (string, optional): Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less.- `text` (string, required): Text to insert at the target's cursor.- `window_id` (integer, optional): CGWindowID. Required when element_index is used. Optional when element_token is supplied (the token carries it).@@ -325,7 +327,7 @@ WEB CONTENT (Chromium/WebKit/Electron — browser tabs, Slack, VS Code, X's comp- `y` (number, optional): Screenshot-pixel Y of the field (see x).```json-{"pid":844,"text":"hello"}+{"text":"hello"}```### `press_key`@@ -343,14 +345,15 @@ A key press is never driver-verifiable → effect:"unverifiable"; confirm via sc- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit "stale" error if the snapshot has been superseded.- `key` (string, required): Key name: return, tab, escape, up, down, etc.- `modifiers` (array of string, optional): Modifier keys: cmd, shift, option/alt, ctrl, fn.-- `pid` (integer, required)+- `pid` (integer, optional)+- `scope` (string, optional): Use desktop with no pid/window_id to send the key to the frontmost application.- `session` (string, optional): Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less.- `window_id` (integer, optional): Target window. Required for delivery_mode:"foreground". Does NOT itself raise the window — raising is gated on delivery_mode.- `x` (number, optional): Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_index.- `y` (number, optional): Screenshot-pixel Y (see x).```json-{"key":"return","pid":844}+{"key":"return"}```### `hotkey`@@ -367,14 +370,15 @@ Recognized modifiers: cmd/command, shift, option/alt, ctrl/control, fn. Non-modi- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target — no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it.- `keys` (array of string, required): Modifier(s) and one non-modifier key, e.g. ["cmd", "c"].-- `pid` (integer, required): Target process ID.+- `pid` (integer, optional): Target process ID.+- `scope` (string, optional): Use desktop with no pid/window_id to send the chord to the frontmost application.- `session` (string, optional): Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less.- `window_id` (integer, optional): Target window. Required for delivery_mode:"foreground" (the NSMenu activation needs a window). Does NOT itself raise the window — raising is gated on delivery_mode.- `x` (number, optional): Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the combo (so e.g. Cmd+V pastes into that field). Pass with y. Use for Chromium/Electron surfaces the background combo can't reach.- `y` (number, optional): Screenshot-pixel Y (see x).```json-{"keys":["cmd","c"],"pid":844}+{"keys":["cmd","c"]}```### `set_value`@@ -419,6 +423,7 @@ Mapping: by='page' → larger step; by='line' → smaller step; amount = number- `element_index` (integer, optional): Element from last get_window_state. Routes through the pixel-wheel path AT this element's center — use it to scroll a nested overflow region you located in the AX tree.- `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit "stale" error if the snapshot has been superseded. Routes through the pixel-wheel path at the element's center.- `pid` (integer, optional)+- `scope` (string, optional): Use desktop with x,y and no pid/window_id for native get_desktop_state screenshot coordinates.- `session` (string, optional): Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less.- `window_id` (integer, optional)- `x` (number, optional): Window-local screenshot X (top-left origin of the PNG from get_window_state). With `y`, routes through the pixel-wheel path at this point — use for a scrollable surface that isn't in the AX tree. Requires window_id to anchor the window→screen conversion.@@ -430,11 +435,12 @@ Mapping: by='page' → larger step; by='line' → smaller step; amount = number### `move_cursor`-Move the agent cursor overlay to (x, y). Does NOT move the real mouse cursor — the user's cursor stays where it is. Useful for showing the agent's attention without interrupting the user.+Move a cursor to (x, y). In window scope (default), moves only the agent cursor overlay. With scope=desktop, moves the real OS pointer in native get_desktop_state screenshot coordinates.**Arguments:**- `cursor_id` (string, optional): Cursor instance to move. Default: 'default'.+- `scope` (string, optional)- `session` (string, optional): Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less.- `x` (number, required)- `y` (number, required)@@ -560,13 +566,12 @@ Caveats:### `set_config`-Update cua-driver-rs configuration. Changes to max_image_dimension and capture_scope take effect immediately. The experimental_pip keys are persisted to ~/.cua-driver/config.json and take effect on the next daemon restart (the PiP backend is initialised once at startup).+Update cua-driver-rs configuration. Changes to max_image_dimension take effect immediately. The experimental_pip keys are persisted to ~/.cua-driver/config.json and take effect on the next daemon restart (the PiP backend is initialised once at startup).-Note: capture_mode is a per-call param (on get_window_state / click), not a stored setting. capture_scope IS a global setting: it gates get_desktop_state (full-display capture requires capture_scope=desktop).+Note: capture_mode is a per-call param (on get_window_state / click), not a stored setting. Capture scope is selected by start_session, not set_config.**Arguments:**-- `capture_scope` (string, optional): Capture scope: "window" (default) or "desktop". Desktop scope enables get_desktop_state (full-display capture) and window-less screen-absolute click/scroll. Global setting; takes effect immediately.- `experimental_pip` (boolean, optional): Enable the experimental picture-in-picture preview window. Applies on next daemon restart.- `experimental_pip_geometry` (string, optional): PiP window size + optional position in `WxH` or `WxH+X+Y` form (e.g. `320x200+24+24`). Applies on next daemon restart.- `key` (string, optional): Name of a single config field to write ({key, value} shape, matching the CLI `config set` and the Windows/Linux tools). Pair with `value`. Equivalent to passing the field directly.@@ -575,10 +580,11 @@ Note: capture_mode is a per-call param (on get_window_state / click), not a stor### `start_session`-Declare a session — a named, color-coded identity for THIS agent run. Pass a stable `session` id; the agent cursor, per-session config, and recording all key on it, and it follows the run across any apps/windows. The cursor's color is derived from the id, so distinct runs are visually distinct. A cursor is shown only for a declared session — call this (or pass `session` on your first action) to opt in. Idempotent: re-calling with the same id just refreshes its idle-TTL. End it with `end_session` (or let the idle-TTL reclaim it). Concurrent runs/subagents each pass their own `session` to get their own cursor.+Declare a session — a named, color-coded identity for THIS agent run. Pass a stable `session` id and choose capture_scope=auto|window|desktop; the agent cursor, capture policy, per-session config, and recording all key on it, and it follows the run across any apps/windows. The cursor's color is derived from the id, so distinct runs are visually distinct. A cursor is shown only for a declared session — call this (or pass `session` on your first action) to opt in. Idempotent: re-calling with the same id just refreshes its idle-TTL. End it with `end_session` (or let the idle-TTL reclaim it). Concurrent runs/subagents each pass their own `session` to get their own cursor.**Arguments:**+- `capture_scope` (string, optional): Per-session perception/action modality. auto starts window-only and requires explicit escalation before desktop tools; window and desktop are strict. Immutable for the live session.- `session` (string, required): Stable session id for this run (e.g. "research-run-1").```json@@ -901,3 +907,29 @@ Perform hover, right-click, double-click, scroll, or drag in an exactly-bound br```json{"action":"hover","session":"example","tab_id":"example","target_id":"example"}```++### `escalate_session`++Unlock the desktop phase of an auto capture-scope session after the window action ladder has been exhausted and verified. This is a one-way transition for the live session and records a bounded reason.++**Arguments:**++- `detail` (string, optional): Optional bounded diagnostic detail. Never use secrets or page content.+- `reason` (string, required)+- `session` (string, required)++```json+{"reason":"ax_tree_pixel_mismatch","session":"example"}+```++### `get_session_state`++Read the live session's capture policy and effective scope.++**Arguments:**++- `session` (string, required)++```json+{"session":"example"}+```diff --git a/libs/cua-driver/docs/test-matrix.md b/libs/cua-driver/docs/test-matrix.mdindex b574985a5f..039f02d818 100644--- a/libs/cua-driver/docs/test-matrix.md+++ b/libs/cua-driver/docs/test-matrix.md@@ -45,6 +45,7 @@ These run without the repo-local GUI applications:| --- | --- | --- || Core driver logic | `rust/crates/*/src/**` | Protocol values, sessions, schemas, image helpers, input helpers, configuration, telemetry, CLI behavior || MCP and CLI boundary | `rust/crates/cua-driver/tests/protocol_*` | Handshake, tool registration, tool calls, media, sessions, and errors |+| Session capture scope | `session_capture_scope_test.rs` plus core `capture_scope` tests | Per-session isolation, immutable live policy, explicit auto escalation, end/revive race safety, transport-mirror refusal, and retired persistent key || Schema gate | `schema_consistency_test.rs` | Shared tool schema parity across OS backends || Configuration transport | `transport_config_persistence_test.rs` | CLI and MCP configuration persistence || Token and protocol surfaces | `protocol_element_token_test.rs`, related tests | JSON-RPC-visible contract behavior |@@ -106,6 +107,7 @@ Windows native harnesses are repo-local applications built from source:| WPF | `harness_wpf_test.rs` | UIA controls, text, keys, pointer actions, scroll, drag, popups, menus, modal windows || WinUI3 | `harness_winui3_test.rs` | XAML controls, text, checkbox/radio, slider, combo, popup || WebView2 | `harness_web_test.rs` | Window discovery, CDP page access, JavaScript, DOM click path |+| Desktop scope | `desktop_scope_windows_test.rs` | Concurrent strict window/desktop sessions, full-display capture, screen-absolute click/scroll, and strict rejection || Desktop invariants | Testkit `DesktopObserver` plus typed launch/capture/cursor owners | Cross-cutting focus, z-order, minimized-launch, screenshot, cursor, and desktop checks |Native controls use AX/UIA state as their oracle. Pointer actions also use PX@@ -119,6 +121,7 @@ background delivery and attach desktop-side-effect oracles.| AppKit | `harness_appkit_test.rs` | AX tree/capture, AX value/text, AX scroll, PX clicks, and foreground slider drag across the proven delivery modes; background drag is an exact refusal || SwiftUI | `harness_swiftui_test.rs` | AX tree/capture, background click/value, and foreground popover-trigger state || WKWebView | `cross_platform_behavior_test.rs` | Dedicated native host running the full 40-cell shared web catalog |+| Desktop scope | `desktop_scope_macos_test.rs` | Concurrent strict window/desktop sessions, screen-absolute action delivery, and strict rejection || Installed-app launch/focus | `installed_app_launch_macos_test.rs` | Real Calculator/TextEdit launch and focus behavior in the canonical logged-in lane || Installed-app text | `installed_app_textedit_macos_test.rs` | Real TextEdit AX background write and verification in the canonical logged-in lane |@@ -138,7 +141,7 @@ transient-panel AX discovery gap.| Electron | `cross_platform_behavior_test.rs` | X11 and hosted Sway | Shared web action matrix || Tauri | `cross_platform_behavior_test.rs` | X11 and hosted Sway | Shared web action matrix || GTK3 | `harness_gtk3_test.rs` | X11/AT-SPI and Wayland/AT-SPI where configured | Native GTK controls and input |-| Desktop scope | `desktop_scope_linux_test.rs` | X11/Wayland | Desktop versus window scope |+| Desktop scope | `desktop_scope_linux_test.rs` | X11/Wayland | Concurrent strict window/desktop sessions, screen-absolute action delivery, and strict rejection |Nix provides the Linux build and desktop environment. X11 and Wayland areseparate matrix dimensions because their capture and input contracts differ.diff --git a/libs/cua-driver/rust/Skills/cua-driver/LINUX.md b/libs/cua-driver/rust/Skills/cua-driver/LINUX.mdindex ec83f3a8fb..d950968c22 100644--- a/libs/cua-driver/rust/Skills/cua-driver/LINUX.md+++ b/libs/cua-driver/rust/Skills/cua-driver/LINUX.md@@ -113,12 +113,11 @@ CI so the three surfaces can't drift. Linux-relevant notes:Linux builds rejected it via `additionalProperties:false` (it waseffectively macOS-only); it is now uniformly schema-accepted — Linuxglides a per-session cursor on X11 where the overlay is available.-- **Windowless screen-absolute clicks** are supported on Linux, but Linux- has **no per-call `scope` param** (that form is macOS-only). Linux gates- the windowless path on the persisted `capture_scope` config: opt in once- with `set_config capture_scope=desktop`, then send `x,y` with no- pid/window_id. Under the default `capture_scope=window` a windowless- action is rejected with a structured `desktop_scope_disabled` error.+- **Windowless screen-absolute actions** pass `scope:"desktop"` with no+ pid/window_id, uniformly with macOS and Windows. The session must have+ effective desktop scope (`start_session(..., capture_scope:"desktop")`, or+ an explicitly escalated `auto` session). Strict window sessions receive+ `desktop_scope_disabled`; no persistent config is read or written.## AT-SPI needs the session bus (headless / containers / `runuser`)diff --git a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md b/libs/cua-driver/rust/Skills/cua-driver/MACOS.mdindex efd7f16024..06ed894644 100644--- a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md+++ b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md@@ -307,12 +307,11 @@ addressing params are a shared cross-platform contract — see `SKILL.md` →- **`session` always worked on macOS;** the cross-platform change is thatWindows/Linux stopped *rejecting* it. No macOS-side change to how youpass it.-- **`scope`** (`window` default / `desktop`) is a **macOS-specific per-call- param** on the pointer tools — pass `scope:"desktop"` with `x,y` and no- pid/window_id for a screen-absolute click. Windows and Linux support the- same windowless capability but gate it on the persisted `capture_scope`- config instead (`set_config capture_scope=desktop`), so they have no- per-call `scope` param. Unifying the knob is a tracked follow-up.+- **`scope`** (`window` / `desktop`) selects the action form uniformly on all+ platforms. Pass `scope:"desktop"` with no pid/window_id for screen-absolute+ pointer actions or foreground keyboard actions. The session's immutable+ `capture_scope` policy must permit that form; set it with `start_session`,+ never persistent config.### Canvases, viewports, games (Blender, Unity, GHOST, Qt, wxWidgets)diff --git a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md b/libs/cua-driver/rust/Skills/cua-driver/SKILL.mdindex 3c6a474bd4..8ea15095fe 100644--- a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md+++ b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md@@ -62,10 +62,11 @@ Use whichever combination matches the host. When in doubt, run`cua-driver doctor` — it reports the platform and the right entrypoint.-## The no-foreground principle (cross-platform)+## The no-foreground principle (window phase)-**The user's frontmost app MUST NOT change.** Every platform has its-own list of forbidden commands; the principle is universal:+In a strict `window` session, and during the initial window phase of an+`auto` session, **the user's frontmost app MUST NOT change.** Every platform+has its own list of forbidden commands:- macOS: any `open` invocation, any `osascript` that mutates GUIstate, `cliclick`, `cghidEventTap` writes targeting another app's@@ -78,6 +79,12 @@ If you reach for a command that says "activate", "foreground","raise", or "make key", stop and translate to the cua-driver toolthat does the same intent without focus-stealing.+A strict `desktop` session is an explicit user choice to operate the visible+desktop and therefore uses foreground/system input. An `auto` session may enter+that phase only after the complete window ladder below has been attempted and+verified, followed by `escalate_session`. Never infer desktop permission from a+failed action or a proxy/transport session id.+## Defaults — always prefer cua-driver over shell shims**Default transport is the `cua-driver` CLI** — `Bash` shelling out@@ -184,7 +191,9 @@ Requires the daemon process's UI runloop, which `cua-driver serve` /## The core invariant — snapshot before AND after every action-**Every action MUST be bracketed by `get_window_state(pid, window_id)`**:+**Every action MUST be bracketed by the state tool for the session's effective+scope**: `get_window_state(pid, window_id)` in window scope, or+`get_desktop_state(session)` in desktop scope.- **Before** — the pre-action snapshot resolves the `element_index`you're about to use. Indices from previous turns are stale; the@@ -200,8 +209,39 @@ Requires the daemon process's UI runloop, which `cua-driver serve` /the action fired. If nothing changed, the action probably failedsilently — say so, don't assume success.-This applies to pixel clicks too — re-snapshot after to confirm the-click landed on the intended target.+This applies to pixel clicks and desktop actions too — re-snapshot after to+confirm the action landed on the intended target.++## Choose capture scope when the session starts++`capture_scope` is a per-session policy, not persistent configuration. Declare+it with `start_session`; it is immutable until that session ends. Concurrent+sessions may choose different policies safely.++- `auto` (default): begins with effective scope `window`. Desktop perception+ and actions are locked until the window ladder is exhausted, each attempted+ action is verified, and the caller explicitly invokes `escalate_session`.+ Escalation is one-way for the live session.+- `window`: strict window-only perception and actions. Desktop tools are always+ rejected with `desktop_scope_disabled`.+- `desktop`: strict full-desktop perception and foreground/system actions.+ Window-scoped perception and actions are rejected with+ `window_scope_disabled`.++```bash+cua-driver start_session '{"session":"research-1","capture_scope":"auto"}'+cua-driver get_session_state '{"session":"research-1"}'+```++Do not use `config set capture_scope` or `set_config`; that key is retired and+stale values on disk are ignored. Always pass the public `session` field on+state and action calls. Reserved fields such as `_session_id` are transport+metadata and cannot create or change policy.++During a mixed-version rollout, require `tools/list` to advertise+`session.capture_scope` (and `session.capture_scope.escalate` for `auto`). If an+older daemon does not advertise them, fail closed and ask for an upgrade; never+fall back to the retired global config key.### Why window selection is the caller's job now@@ -362,6 +402,13 @@ if resp.effect == "suspected_noop"get_window_state(pid, window_id) # re-snapshot, eyeball the resultif it landed: done+# Rung 2b — exact browser page tools, when get_browser_state can bind this window+# Use typed browser refs for page content; native window tools still handle chrome.+get_browser_state(session, pid, window_id)+browser_click(session, target_id, tab_id, ref) # or browser_type; see BROWSER.md+get_browser_state(session, pid, window_id) # verify with fresh refs+if it landed: done+# Rung 3 — background delivery was dropped (insert/click never arrived)if resp.escalation.recommended == "foreground"or the px action still did nothing:@@ -369,6 +416,16 @@ if resp.escalation.recommended == "foreground"# on Wayland this is the ONLY escalation — px-bg can't target an# unfocused window there; see LINUX.mdverify again++# Rung 4 — desktop fallback (auto sessions only, explicit and one-way)+# Reach this only after AX, window-pixel, browser-page (when available), and+# foreground-window delivery have all been exhausted and verified ineffective.+escalate_session(session,+ reason="foreground_ineffective", # or another advertised reason+ detail="bounded non-sensitive summary")+get_desktop_state(session) # full primary display+desktop_action(session, scope="desktop", ...) # no pid/window_id+get_desktop_state(session) # verify in the same coordinate frame```The two ideas to hold onto: (1) the AX tree **lies** on canvas / web /@@ -399,7 +456,7 @@ last resort.## The canonical loop```-start_session(session) # once per run: declares this run's identity+start_session(session, capture_scope="auto") # once per run; policy is immutablelaunch_app(target)→ pick window_id from the returned `windows` array(or call list_windows(pid) separately)@@ -409,19 +466,28 @@ launch_app(target)end_session(session) # when the run finishes```+For strict desktop sessions, replace the window portion with+`get_desktop_state(session) → action(session, scope="desktop", ...) →+get_desktop_state(session)`. Desktop actions use screen-absolute coordinates+from that exact full-display image and omit `pid`/`window_id`. The global+`get_screen_size` and `get_cursor_position` helpers are desktop-scoped too.+`launch_app` now returns a `windows` array alongside the pid, so thecommon case collapses to two calls (`launch_app` → `get_window_state`)without a separate `list_windows` hop.**Declare a session.** A session is _your run's_ identity — a stable idyou choose (`"research-1"`), declared with `start_session` and passed as-`session` on every action. It owns your agent cursor (a distinct colour-per id), follows the run across any apps/windows, and is the same whether+`session` on every action. It owns your agent cursor and capture policy (a+distinct colour and one immutable policy per id), follows the run across any+apps/windows, and is the same whetheryou drive over MCP, the CLI, or the socket. Declaring the session creates thecursor; anonymous actions remain cursor-less.End with `end_session` (or the idle-TTL reclaims it).-**Concurrent runs/subagents:** `launch_app` is idempotent — two runs that+**Concurrent runs/subagents:** each run may independently choose `auto`,+`window`, or `desktop`; one session's escalation never changes another. Also,+`launch_app` is idempotent — two runs thatlaunch the same app get the **same** instance (and on single-instance appslike Calculator, the same window), so they clobber each other. Give each runits **own `session`** (→ its own cursor) AND pass@@ -551,10 +617,15 @@ against a specific window.| Send key to pid | `press_key({pid, key, modifiers})` | no focus change; key goes to pid's current focus || Modifier combo | `hotkey({pid, keys})` (no focus) or `hotkey({pid, x, y, keys})` (px) | e.g. `["cmd","c"]` / `["ctrl","c"]`; posted per-pid, not HID tap. **px** pixel-clicks `(x,y)` to focus a field first, e.g. `["cmd","v"]` to paste into it |-**All keyboard/text primitives require `pid`.** There is no-frontmost-routed variant — every key goes to the named target via-the platform's per-pid event-post path, so the driver cannot leak-keystrokes into the user's foreground app.+In effective desktop scope, the foreground/system equivalents omit+`pid`/`window_id` and pass `scope:"desktop"`: `click`, `scroll`, `drag`,+`move_cursor`, `type_text`, `press_key`, and `hotkey`. Coordinates are+screen-absolute pixels from the latest `get_desktop_state` image.++**Window-scope keyboard/text primitives require `pid`.** They use the named+target's per-pid event-post path. Only a strict/effective desktop session may+omit `pid`, and it intentionally routes keyboard input to the current+foreground application.**Why `element_index` is the primary path:** works on hidden /occluded / off-desktop windows, no focus steal, stable acrossdiff --git a/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md b/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.mdindex 8d9d7a2ee7..dd89080938 100644--- a/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md+++ b/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md@@ -404,11 +404,11 @@ When a cua-driver call surprises you, diagnose cua-driver first:regardless of what you pass.- **`get_desktop_state` returns `desktop_scope_disabled`?** That'sintended: full-display capture is a **desktop-scope** operation, gated- on the global `capture_scope`. It's `"window"` by default — so to verify- a specific window use `get_window_state(pid, window_id)` (works- backgrounded), and only use `get_desktop_state` after- `set_config capture_scope=desktop` (the same opt-in that enables- window-less screen-absolute `click`/`scroll`). Don't reach for+ by the caller-declared session policy. To verify a specific window use+ `get_window_state(pid, window_id)` (works backgrounded). Use+ `get_desktop_state` only in a strict desktop session or after explicitly+ escalating an `auto` session once the window ladder is exhausted. Desktop+ actions pass `scope:"desktop"` with no pid/window_id. Don't reach for`get_desktop_state` as a casual screenshot — it's the capture surface fordesktop-scope coordinate loops, not window inspection.- **`Calc display stuck at 0 after my clicks`?** Almost alwaysdiff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/capture_scope.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/capture_scope.rsnew file mode 100644index 0000000000..860eb0523f--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/capture_scope.rs@@ -0,0 +1,511 @@+//! Per-session capture/action scope policy.+//!+//! The policy is keyed exclusively by the public, caller-declared `session`+//! argument. Reserved transport mirrors such as `_session_id` must never mint+//! policy state: doing so would let an anonymous proxy connection bypass the+//! per-session contract.++use serde::{Deserialize, Serialize};+use serde_json::{json, Value};+use std::collections::HashMap;+use std::fmt;+use std::sync::{Mutex, OnceLock};++#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]+#[serde(rename_all = "snake_case")]+pub enum CaptureScopePolicy {+ #[default]+ Auto,+ Window,+ Desktop,+}++impl CaptureScopePolicy {+ pub fn parse(value: &str) -> Option<Self> {+ match value {+ "auto" => Some(Self::Auto),+ "window" => Some(Self::Window),+ "desktop" => Some(Self::Desktop),+ _ => None,+ }+ }++ pub fn as_str(self) -> &'static str {+ match self {+ Self::Auto => "auto",+ Self::Window => "window",+ Self::Desktop => "desktop",+ }+ }+}++impl fmt::Display for CaptureScopePolicy {+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {+ f.write_str(self.as_str())+ }+}++#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]+#[serde(rename_all = "snake_case")]+pub enum EffectiveCaptureScope {+ Window,+ Desktop,+}++impl EffectiveCaptureScope {+ pub fn as_str(self) -> &'static str {+ match self {+ Self::Window => "window",+ Self::Desktop => "desktop",+ }+ }+}++#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]+#[serde(rename_all = "snake_case")]+pub enum EscalationReason {+ AxTreePixelMismatch,+ BackgroundDeliveryFailed,+ ForegroundIneffective,+ NoWindowTarget,+ Other,+}++impl EscalationReason {+ pub fn parse(value: &str) -> Option<Self> {+ match value {+ "ax_tree_pixel_mismatch" => Some(Self::AxTreePixelMismatch),+ "background_delivery_failed" => Some(Self::BackgroundDeliveryFailed),+ "foreground_ineffective" => Some(Self::ForegroundIneffective),+ "no_window_target" => Some(Self::NoWindowTarget),+ "other" => Some(Self::Other),+ _ => None,+ }+ }++ pub fn as_str(self) -> &'static str {+ match self {+ Self::AxTreePixelMismatch => "ax_tree_pixel_mismatch",+ Self::BackgroundDeliveryFailed => "background_delivery_failed",+ Self::ForegroundIneffective => "foreground_ineffective",+ Self::NoWindowTarget => "no_window_target",+ Self::Other => "other",+ }+ }+}++#[derive(Debug, Clone, PartialEq, Eq)]+pub struct SessionCaptureScope {+ pub policy: CaptureScopePolicy,+ pub desktop_unlocked: bool,+ pub escalation_reason: Option<EscalationReason>,+ pub escalation_detail: Option<String>,+}++impl SessionCaptureScope {+ pub fn new(policy: CaptureScopePolicy) -> Self {+ Self {+ policy,+ desktop_unlocked: policy == CaptureScopePolicy::Desktop,+ escalation_reason: None,+ escalation_detail: None,+ }+ }++ pub fn effective_scope(&self) -> EffectiveCaptureScope {+ match self.policy {+ CaptureScopePolicy::Desktop => EffectiveCaptureScope::Desktop,+ CaptureScopePolicy::Window => EffectiveCaptureScope::Window,+ CaptureScopePolicy::Auto if self.desktop_unlocked => EffectiveCaptureScope::Desktop,+ CaptureScopePolicy::Auto => EffectiveCaptureScope::Window,+ }+ }++ pub fn as_json(&self, session: &str) -> Value {+ json!({+ "session": session,+ "capture_scope": self.policy.as_str(),+ "effective_scope": self.effective_scope().as_str(),+ "desktop_unlocked": self.desktop_unlocked,+ "escalation_reason": self.escalation_reason.map(EscalationReason::as_str),+ "escalation_detail": self.escalation_detail,+ })+ }+}++static SESSION_SCOPES: OnceLock<Mutex<HashMap<String, SessionCaptureScope>>> = OnceLock::new();++fn scopes() -> &'static Mutex<HashMap<String, SessionCaptureScope>> {+ SESSION_SCOPES.get_or_init(|| Mutex::new(HashMap::new()))+}++#[derive(Debug, Clone, PartialEq, Eq)]+pub enum BindError {+ Ended,+ Conflict {+ existing: CaptureScopePolicy,+ requested: CaptureScopePolicy,+ },+}++/// Bind a policy to a live session. Omitted policy means "keep the existing+/// policy, or use auto for a new session" so repeated `start_session` calls+/// remain idempotent.+pub fn bind_session(+ session: &str,+ requested: Option<CaptureScopePolicy>,+) -> Result<(SessionCaptureScope, bool), BindError> {+ let mut registry = scopes().lock().unwrap();+ if crate::session::is_session_ended(session) {+ return Err(BindError::Ended);+ }+ if let Some(existing) = registry.get(session) {+ if let Some(requested) = requested {+ if existing.policy != requested {+ return Err(BindError::Conflict {+ existing: existing.policy,+ requested,+ });+ }+ }+ return Ok((existing.clone(), false));+ }+ let state = SessionCaptureScope::new(requested.unwrap_or_default());+ registry.insert(session.to_owned(), state.clone());+ Ok((state, true))+}++pub fn get_session(session: &str) -> Option<SessionCaptureScope> {+ scopes().lock().unwrap().get(session).cloned()+}++pub fn clear_session(session: &str) {+ scopes().lock().unwrap().remove(session);+}++#[derive(Debug, Clone, PartialEq, Eq)]+pub enum EscalateError {+ Ended,+ NotStarted,+ WindowStrict,+ DesktopAlreadyActive,+}++pub fn escalate_session(+ session: &str,+ reason: EscalationReason,+ detail: Option<&str>,+) -> Result<SessionCaptureScope, EscalateError> {+ let mut registry = scopes().lock().unwrap();+ // This check deliberately happens while holding the scope lock. If an end+ // races with escalation, either this write wins first and end removes it,+ // or the tombstone wins first and this write is rejected. A late action+ // can therefore never resurrect an unlocked session.+ if crate::session::is_session_ended(session) {+ return Err(EscalateError::Ended);+ }+ let Some(state) = registry.get_mut(session) else {+ return Err(EscalateError::NotStarted);+ };+ if state.desktop_unlocked {+ return Err(EscalateError::DesktopAlreadyActive);+ }+ match state.policy {+ CaptureScopePolicy::Window => return Err(EscalateError::WindowStrict),+ CaptureScopePolicy::Desktop => return Err(EscalateError::DesktopAlreadyActive),+ CaptureScopePolicy::Auto => {}+ }+ state.desktop_unlocked = true;+ state.escalation_reason = Some(reason);+ state.escalation_detail = detail.map(|value| value.chars().take(200).collect());+ Ok(state.clone())+}++#[derive(Debug, Clone, Copy, PartialEq, Eq)]+enum ToolScope {+ Window,+ Desktop,+ Unscoped,+}++fn is_scoped_action(tool_name: &str) -> bool {+ matches!(+ tool_name,+ "click"+ | "double_click"+ | "right_click"+ | "scroll"+ | "drag"+ | "mouse_drag"+ | "parallel_mouse_drag"+ | "move_cursor"+ | "type_text"+ | "type_text_chars"+ | "press_key"+ | "hotkey"+ | "set_value"+ )+}++fn tool_scope(tool_name: &str, args: &Value) -> ToolScope {+ if matches!(+ tool_name,+ "get_desktop_state" | "get_screen_size" | "get_cursor_position"+ ) {+ return ToolScope::Desktop;+ }+ if matches!(+ tool_name,+ "get_window_state"+ | "get_accessibility_tree"+ | "get_app_state"+ | "screenshot"+ | "screenshot_compat"+ | "zoom"+ | "page"+ )+ || tool_name == "get_browser_state"+ || (tool_name.starts_with("browser_")+ && !matches!(tool_name, "browser_prepare" | "browser_download"))+ {+ return ToolScope::Window;+ }+ if is_scoped_action(tool_name) {+ let explicit_desktop = args.get("scope").and_then(Value::as_str) == Some("desktop");+ let has_window_target = args.get("pid").is_some()+ || args.get("window_id").is_some()+ || args.get("element_index").is_some()+ || args.get("element_token").is_some();+ if explicit_desktop && !has_window_target {+ ToolScope::Desktop+ } else {+ ToolScope::Window+ }+ } else {+ ToolScope::Unscoped+ }+}++#[derive(Debug, Clone, PartialEq, Eq)]+pub struct ScopeViolation {+ pub code: &'static str,+ pub message: String,+ pub state: SessionCaptureScope,+}++impl ScopeViolation {+ pub fn as_json(&self, session: &str) -> Value {+ let mut value = self.state.as_json(session);+ if let Some(object) = value.as_object_mut() {+ object.insert("code".to_owned(), Value::String(self.code.to_owned()));+ }+ value+ }+}++/// Enforce the modality for a public session. A first non-lifecycle action+/// implicitly starts that session in `auto`, matching the existing cursor and+/// recording lifecycle behavior.+pub fn enforce_tool(tool_name: &str, args: &Value) -> Result<(), ScopeViolation> {+ if matches!(+ tool_name,+ "start_session" | "end_session" | "escalate_session" | "get_session_state"+ ) {+ return Ok(());+ }+ let Some(session) = args+ .get("session")+ .and_then(Value::as_str)+ .filter(|value| !value.is_empty() && *value != "default")+ else {+ return Ok(());+ };+ let (state, _) = bind_session(session, None).map_err(|_| ScopeViolation {+ code: "session_ended",+ message: format!("session '{session}' has ended"),+ state: SessionCaptureScope::new(CaptureScopePolicy::Auto),+ })?;+ match (state.effective_scope(), tool_scope(tool_name, args)) {+ (_, ToolScope::Unscoped)+ | (EffectiveCaptureScope::Window, ToolScope::Window)+ | (EffectiveCaptureScope::Desktop, ToolScope::Desktop) => Ok(()),+ (EffectiveCaptureScope::Window, ToolScope::Desktop)+ if state.policy == CaptureScopePolicy::Auto =>+ {+ Err(ScopeViolation {+ code: "desktop_escalation_required",+ message: format!(+ "desktop-scope tool '{tool_name}' is locked for auto session '{session}'; exhaust the window action ladder, verify each attempt, then call escalate_session"+ ),+ state,+ })+ }+ (EffectiveCaptureScope::Window, ToolScope::Desktop) => Err(ScopeViolation {+ code: "desktop_scope_disabled",+ message: format!(+ "desktop-scope tool '{tool_name}' is disabled for strict window session '{session}'"+ ),+ state,+ }),+ (EffectiveCaptureScope::Desktop, ToolScope::Window) => Err(ScopeViolation {+ code: "window_scope_disabled",+ message: format!(+ "window-scope tool '{tool_name}' is disabled while session '{session}' is in desktop scope"+ ),+ state,+ }),+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ fn fresh(prefix: &str) -> String {+ format!("capture-scope-{prefix}-{}", std::process::id())+ }++ #[test]+ fn sessions_are_isolated_and_auto_escalates_one_way() {+ let auto = fresh("auto");+ let desktop = fresh("desktop");+ let window = fresh("window");+ bind_session(&auto, Some(CaptureScopePolicy::Auto)).unwrap();+ bind_session(&desktop, Some(CaptureScopePolicy::Desktop)).unwrap();+ bind_session(&window, Some(CaptureScopePolicy::Window)).unwrap();++ assert_eq!(+ enforce_tool("get_desktop_state", &json!({"session": auto}))+ .unwrap_err()+ .code,+ "desktop_escalation_required"+ );+ assert!(enforce_tool("get_desktop_state", &json!({"session": desktop})).is_ok());+ assert_eq!(+ enforce_tool("get_desktop_state", &json!({"session": window}))+ .unwrap_err()+ .code,+ "desktop_scope_disabled"+ );++ escalate_session(&auto, EscalationReason::ForegroundIneffective, None).unwrap();+ assert!(enforce_tool("get_desktop_state", &json!({"session": auto})).is_ok());+ assert_eq!(+ enforce_tool("get_window_state", &json!({"session": auto, "pid": 1}))+ .unwrap_err()+ .code,+ "window_scope_disabled"+ );+ assert_eq!(+ get_session(&desktop).unwrap().effective_scope(),+ EffectiveCaptureScope::Desktop+ );+ }++ #[test]+ fn conflicting_live_rebind_is_rejected() {+ let session = fresh("conflict");+ bind_session(&session, Some(CaptureScopePolicy::Window)).unwrap();+ assert_eq!(+ bind_session(&session, Some(CaptureScopePolicy::Desktop)),+ Err(BindError::Conflict {+ existing: CaptureScopePolicy::Window,+ requested: CaptureScopePolicy::Desktop,+ })+ );+ }++ #[test]+ fn transport_mirror_does_not_create_policy() {+ let session = fresh("mirror");+ assert!(enforce_tool("get_desktop_state", &json!({"_session_id": session})).is_ok());+ assert!(get_session(&session).is_none());+ }++ #[test]+ fn escalation_detail_is_bounded() {+ let session = fresh("detail");+ bind_session(&session, Some(CaptureScopePolicy::Auto)).unwrap();+ let state =+ escalate_session(&session, EscalationReason::Other, Some(&"x".repeat(500))).unwrap();+ assert_eq!(state.escalation_detail.unwrap().chars().count(), 200);+ }++ #[test]+ fn strict_modes_apply_symmetrically_to_perception_and_actions() {+ let window = fresh("strict-window-actions");+ let desktop = fresh("strict-desktop-actions");+ bind_session(&window, Some(CaptureScopePolicy::Window)).unwrap();+ bind_session(&desktop, Some(CaptureScopePolicy::Desktop)).unwrap();++ assert_eq!(+ enforce_tool(+ "click",+ &json!({"session": window, "scope": "desktop", "x": 4, "y": 5})+ )+ .unwrap_err()+ .code,+ "desktop_scope_disabled"+ );+ assert_eq!(+ enforce_tool("get_cursor_position", &json!({"session": window}))+ .unwrap_err()+ .code,+ "desktop_scope_disabled"+ );+ assert!(enforce_tool("get_screen_size", &json!({"session": desktop})).is_ok());+ assert_eq!(+ enforce_tool(+ "click",+ &json!({"session": desktop, "pid": 42, "x": 4, "y": 5})+ )+ .unwrap_err()+ .code,+ "window_scope_disabled"+ );+ }++ #[test]+ fn end_racing_escalation_never_resurrects_scope_state() {+ use std::sync::{Arc, Barrier};++ let session = fresh("end-escalate-race");+ bind_session(&session, Some(CaptureScopePolicy::Auto)).unwrap();+ let barrier = Arc::new(Barrier::new(3));+ let escalation_session = session.clone();+ let escalation_barrier = barrier.clone();+ let escalation = std::thread::spawn(move || {+ escalation_barrier.wait();+ escalate_session(+ &escalation_session,+ EscalationReason::ForegroundIneffective,+ None,+ )+ });+ let end_session = session.clone();+ let end_barrier = barrier.clone();+ let end = std::thread::spawn(move || {+ end_barrier.wait();+ crate::session::fire_session_end(&end_session)+ });+ barrier.wait();+ let _ = escalation.join().unwrap();+ assert!(end.join().unwrap());++ assert!(crate::session::is_session_ended(&session));+ assert!(get_session(&session).is_none());+ assert_eq!(+ enforce_tool("get_window_state", &json!({"session": session}))+ .unwrap_err()+ .code,+ "session_ended"+ );++ assert!(crate::session::revive_session(&session));+ let (fresh_state, created) =+ bind_session(&session, Some(CaptureScopePolicy::Window)).unwrap();+ assert!(created);+ assert_eq!(fresh_state.policy, CaptureScopePolicy::Window);+ assert!(!fresh_state.desktop_unlocked);+ }+}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rsindex 52bd0bd6e6..e28bc9888e 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs@@ -34,6 +34,7 @@ pub fn embedded_mode() -> bool {pub mod browser;pub mod capture_mode;+pub mod capture_scope;pub mod cdp;pub mod element_cache;pub mod element_token;diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/recording_render.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/recording_render.rsindex 325b84cace..f0165449a5 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/recording_render.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/recording_render.rs@@ -252,4 +252,3 @@ fn probe_duration_ms(video_path: &Path) -> Option<f64> {let secs: f64 = s.trim().parse().ok()?;Some(secs * 1000.0)}-diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/session.rsindex f0e6bf29d0..5ae762837e 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs@@ -306,6 +306,7 @@ pub fn fire_session_end(session_id: &str) -> bool {return false; // already ended — idempotent no-op.}}+ crate::capture_scope::clear_session(session_id);for hook in hooks().lock().unwrap().iter() {hook(session_id);}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/session_tools.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/session_tools.rsindex 20d51afc98..ac7c559d9a 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/session_tools.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/session_tools.rs@@ -1,4 +1,4 @@-//! `start_session` / `end_session` tools.+//! Session lifecycle and per-session capture-scope tools.//!//! A session is a **caller-declared** identity for an agent run (see//! [`crate::session`]). It owns the agent cursor and is the key for@@ -7,26 +7,51 @@//! identically over MCP, the CLI (`--session`), or the raw socket, and a run//! can span any number of apps/windows.//!-//! The daemon mirrors an explicit `session` arg into the reserved `_session_id`-//! key (see `serve.rs::apply_session_identity`), so these tools accept either —-//! `session` is the public name.+//! The daemon may mirror an explicit `session` arg into reserved transport+//! fields, but lifecycle and policy tools accept only the public `session`+//! name. Transport metadata can never mint or alter capture policy.+use crate::capture_scope::{+ bind_session, escalate_session, get_session, BindError, CaptureScopePolicy, EscalateError,+ EscalationReason,+};use crate::protocol::ToolResult;use crate::tool::{Tool, ToolDef};use async_trait::async_trait;use serde_json::{json, Value};use std::sync::OnceLock;-/// Read the explicit session id from a tool call (`session`, or its daemon-/// mirror `_session_id`). Empty / missing → `None`.+/// Read the public, caller-declared session id. Capture-scope policy must never+/// bind to the daemon's reserved `_session_id` transport mirror.fn session_id_of(args: &Value) -> Option<String> {- let obj = args.as_object()?;- ["session", "_session_id"]- .into_iter()- .find_map(|k| obj.get(k).and_then(|v| v.as_str()).filter(|s| !s.is_empty()))+ args.as_object()?+ .get("session")+ .and_then(|v| v.as_str())+ .filter(|s| !s.is_empty() && *s != "default").map(|s| s.to_owned())}+fn capture_scope_arg(args: &Value) -> Result<Option<CaptureScopePolicy>, ToolResult> {+ let Some(raw) = args.get("capture_scope") else {+ return Ok(None);+ };+ let Some(raw) = raw.as_str() else {+ return Err(ToolResult::error(+ "start_session.capture_scope must be one of: auto, window, desktop.",+ )+ .with_structured(json!({ "code": "invalid_capture_scope" })));+ };+ CaptureScopePolicy::parse(raw).map(Some).ok_or_else(|| {+ ToolResult::error(format!(+ "invalid capture_scope '{raw}'; expected auto, window, or desktop"+ ))+ .with_structured(json!({+ "code": "invalid_capture_scope",+ "capture_scope": raw,+ }))+ })+}+// ── start_session ─────────────────────────────────────────────────────────────pub struct StartSessionTool;@@ -40,8 +65,9 @@ impl Tool for StartSessionTool {name: "start_session".into(),description:"Declare a session — a named, color-coded identity for THIS agent run. \- Pass a stable `session` id; the agent cursor, per-session config, and \- recording all key on it, and it follows the run across any apps/windows. \+ Pass a stable `session` id and choose capture_scope=auto|window|desktop; \+ the agent cursor, capture policy, per-session config, and recording all \+ key on it, and it follows the run across any apps/windows. \The cursor's color is derived from the id, so distinct runs are visually \distinct. A cursor is shown only for a declared session — call this (or \pass `session` on your first action) to opt in. Idempotent: re-calling \@@ -56,6 +82,12 @@ impl Tool for StartSessionTool {"session": {"type": "string","description": "Stable session id for this run (e.g. \"research-run-1\")."+ },+ "capture_scope": {+ "type": "string",+ "enum": ["auto", "window", "desktop"],+ "default": "auto",+ "description": "Per-session perception/action modality. auto starts window-only and requires explicit escalation before desktop tools; window and desktop are strict. Immutable for the live session."}},"additionalProperties": true@@ -69,21 +101,208 @@ impl Tool for StartSessionTool {async fn invoke(&self, args: Value) -> ToolResult {let Some(id) = session_id_of(&args) else {- return ToolResult::error(- "start_session requires a non-empty `session` id.",- );+ return ToolResult::error("start_session requires a non-empty `session` id.");};+ let requested = match capture_scope_arg(&args) {+ Ok(scope) => scope,+ Err(result) => return result,+ };+ let was_ended = crate::session::is_session_ended(&id);+ if !was_ended {+ match bind_session(&id, requested) {+ Ok(_) => {}+ Err(BindError::Conflict {+ existing,+ requested,+ }) => {+ return ToolResult::error(format!(+ "session '{id}' already uses capture_scope='{existing}'; capture scope is immutable until the session ends"+ ))+ .with_structured(json!({+ "code": "session_policy_conflict",+ "session": id,+ "capture_scope": existing.as_str(),+ "requested_capture_scope": requested.as_str(),+ }));+ }+ Err(BindError::Ended) => {+ return ToolResult::error(format!(+ "session '{id}' ended concurrently; retry start_session to revive it"+ ))+ .with_structured(json!({ "code": "session_ended", "session": id }));+ }+ }+ }// Revive a recycled id: if this session was previously ended (explicit// `end_session` / idle-TTL / connection EOF), clear its tombstone so its// actions stop being rejected by the daemon's resurrection guard.// Re-declaring a session is the EXPLICIT, caller-driven way to reuse an// id — a stray late action still can't silently resurrect a dead one.let revived = crate::session::revive_session(&id);+ let (scope, _) = match bind_session(&id, requested) {+ Ok(bound) => bound,+ Err(BindError::Conflict {+ existing,+ requested,+ }) => {+ return ToolResult::error(format!(+ "session '{id}' already uses capture_scope='{existing}'; capture scope is immutable until the session ends"+ ))+ .with_structured(json!({+ "code": "session_policy_conflict",+ "session": id,+ "capture_scope": existing.as_str(),+ "requested_capture_scope": requested.as_str(),+ }));+ }+ Err(BindError::Ended) => {+ return ToolResult::error(format!("session '{id}' could not be revived"))+ .with_structured(json!({ "code": "session_ended", "session": id }));+ }+ };// Refresh (or begin) the session's idle-TTL clock. The cursor appears on// the first action carrying this `session`.crate::session::touch_session(&id);- ToolResult::text(format!("✅ Session '{id}' is active."))- .with_structured(json!({ "session": id, "active": true, "revived": revived }))+ let mut structured = scope.as_json(&id);+ if let Some(object) = structured.as_object_mut() {+ object.insert("active".to_owned(), Value::Bool(true));+ object.insert("revived".to_owned(), Value::Bool(revived));+ }+ ToolResult::text(format!(+ "✅ Session '{id}' is active with capture_scope='{}'.",+ scope.policy+ ))+ .with_structured(structured)+ }+}++// ── escalate_session ─────────────────────────────────────────────────────────++pub struct EscalateSessionTool;++static ESCALATE_DEF: OnceLock<ToolDef> = OnceLock::new();++#[async_trait]+impl Tool for EscalateSessionTool {+ fn def(&self) -> &ToolDef {+ ESCALATE_DEF.get_or_init(|| ToolDef {+ name: "escalate_session".into(),+ description: "Unlock the desktop phase of an auto capture-scope session after the window action ladder has been exhausted and verified. This is a one-way transition for the live session and records a bounded reason.".into(),+ input_schema: json!({+ "type": "object",+ "required": ["session", "reason"],+ "properties": {+ "session": { "type": "string" },+ "reason": {+ "type": "string",+ "enum": [+ "ax_tree_pixel_mismatch",+ "background_delivery_failed",+ "foreground_ineffective",+ "no_window_target",+ "other"+ ]+ },+ "detail": {+ "type": "string",+ "maxLength": 200,+ "description": "Optional bounded diagnostic detail. Never use secrets or page content."+ }+ },+ "additionalProperties": true+ }),+ read_only: false,+ destructive: false,+ idempotent: false,+ open_world: false,+ })+ }++ async fn invoke(&self, args: Value) -> ToolResult {+ let Some(id) = session_id_of(&args) else {+ return ToolResult::error("escalate_session requires a public `session` id.")+ .with_structured(json!({ "code": "session_required" }));+ };+ let Some(reason) = args+ .get("reason")+ .and_then(Value::as_str)+ .and_then(EscalationReason::parse)+ else {+ return ToolResult::error("escalate_session.reason is invalid.")+ .with_structured(json!({ "code": "invalid_escalation_reason" }));+ };+ let detail = args.get("detail").and_then(Value::as_str);+ match escalate_session(&id, reason, detail) {+ Ok(state) => ToolResult::text(format!("✅ Session '{id}' escalated to desktop scope."))+ .with_structured(state.as_json(&id)),+ Err(error) => {+ let (code, message) = match error {+ EscalateError::Ended => (+ "session_ended",+ format!("session '{id}' has ended; start it again before escalating"),+ ),+ EscalateError::NotStarted => (+ "session_not_started",+ format!("session '{id}' has no capture policy; call start_session first"),+ ),+ EscalateError::WindowStrict => (+ "desktop_scope_disabled",+ format!("session '{id}' is strict window scope and cannot escalate"),+ ),+ EscalateError::DesktopAlreadyActive => (+ "desktop_already_active",+ format!("session '{id}' already has effective desktop scope"),+ ),+ };+ ToolResult::error(message).with_structured(json!({+ "code": code,+ "session": id,+ }))+ }+ }+ }+}++// ── get_session_state ────────────────────────────────────────────────────────++pub struct GetSessionStateTool;++static GET_STATE_DEF: OnceLock<ToolDef> = OnceLock::new();++#[async_trait]+impl Tool for GetSessionStateTool {+ fn def(&self) -> &ToolDef {+ GET_STATE_DEF.get_or_init(|| ToolDef {+ name: "get_session_state".into(),+ description: "Read the live session's capture policy and effective scope.".into(),+ input_schema: json!({+ "type": "object",+ "required": ["session"],+ "properties": { "session": { "type": "string" } },+ "additionalProperties": true+ }),+ read_only: true,+ destructive: false,+ idempotent: true,+ open_world: false,+ })+ }++ async fn invoke(&self, args: Value) -> ToolResult {+ let Some(id) = session_id_of(&args) else {+ return ToolResult::error("get_session_state requires a public `session` id.")+ .with_structured(json!({ "code": "session_required" }));+ };+ let Some(state) = get_session(&id) else {+ return ToolResult::error(format!("session '{id}' is not active"))+ .with_structured(json!({ "code": "session_not_started", "session": id }));+ };+ ToolResult::text(format!(+ "Session '{id}' uses capture_scope='{}' (effective_scope='{}').",+ state.policy,+ state.effective_scope().as_str()+ ))+ .with_structured(state.as_json(&id))}}@@ -98,12 +317,11 @@ impl Tool for EndSessionTool {fn def(&self) -> &ToolDef {END_DEF.get_or_init(|| ToolDef {name: "end_session".into(),- description:- "End a session declared with `start_session`: removes its agent cursor, \+ description: "End a session declared with `start_session`: removes its agent cursor, \stops any recording it owns, and clears its per-session config. Call this \when a run finishes so its cursor doesn't linger (otherwise the idle-TTL \reclaims it after a period of inactivity). Idempotent."- .into(),+ .into(),input_schema: json!({"type": "object","required": ["session"],@@ -136,12 +354,85 @@ impl Tool for EndSessionTool {mod tests {use super::*;+ fn result_code(result: &ToolResult) -> Option<&str> {+ result+ .structured_content+ .as_ref()?+ .get("code")?+ .as_str()+ }+#[test]- fn session_id_of_reads_session_then_mirror() {- assert_eq!(session_id_of(&json!({ "session": "a" })).as_deref(), Some("a"));- assert_eq!(session_id_of(&json!({ "_session_id": "b" })).as_deref(), Some("b"));- assert_eq!(session_id_of(&json!({ "session": "", "_session_id": "c" })).as_deref(), Some("c"));+ fn session_id_of_reads_only_public_session() {+ assert_eq!(+ session_id_of(&json!({ "session": "a" })).as_deref(),+ Some("a")+ );+ assert_eq!(session_id_of(&json!({ "_session_id": "b" })), None);+ assert_eq!(+ session_id_of(&json!({ "session": "", "_session_id": "c" })),+ None+ );assert_eq!(session_id_of(&json!({})), None);assert_eq!(session_id_of(&json!({ "session": "" })), None);+ assert_eq!(session_id_of(&json!({ "session": "default" })), None);+ }++ #[tokio::test]+ async fn live_policy_is_immutable_and_ended_id_gets_fresh_policy() {+ let id = format!("session-tool-policy-{}", std::process::id());+ let start = StartSessionTool;+ let first = start+ .invoke(json!({"session": id, "capture_scope": "window"}))+ .await;+ assert_ne!(first.is_error, Some(true));+ assert_eq!(+ first.structured_content.as_ref().unwrap()["capture_scope"],+ "window"+ );++ let conflict = start+ .invoke(json!({"session": id, "capture_scope": "desktop"}))+ .await;+ assert_eq!(conflict.is_error, Some(true));+ assert_eq!(result_code(&conflict), Some("session_policy_conflict"));++ EndSessionTool.invoke(json!({"session": id})).await;+ assert!(get_session(&id).is_none());+ let revived = start+ .invoke(json!({"session": id, "capture_scope": "desktop"}))+ .await;+ assert_ne!(revived.is_error, Some(true));+ let structured = revived.structured_content.as_ref().unwrap();+ assert_eq!(structured["capture_scope"], "desktop");+ assert_eq!(structured["effective_scope"], "desktop");+ assert_eq!(structured["revived"], true);+ }++ #[tokio::test]+ async fn auto_requires_explicit_bounded_escalation() {+ let id = format!("session-tool-auto-{}", std::process::id());+ let started = StartSessionTool.invoke(json!({"session": id})).await;+ assert_eq!(+ started.structured_content.as_ref().unwrap()["capture_scope"],+ "auto"+ );+ let escalated = EscalateSessionTool+ .invoke(json!({+ "session": id,+ "reason": "background_delivery_failed",+ "detail": "window ladder exhausted"+ }))+ .await;+ assert_ne!(escalated.is_error, Some(true));+ let structured = escalated.structured_content.as_ref().unwrap();+ assert_eq!(structured["effective_scope"], "desktop");+ assert_eq!(structured["desktop_unlocked"], true);++ let twice = EscalateSessionTool+ .invoke(json!({"session": id, "reason": "other"}))+ .await;+ assert_eq!(twice.is_error, Some(true));+ assert_eq!(result_code(&twice), Some("desktop_already_active"));}}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rsindex da0299c091..9b21b31ad0 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs@@ -95,7 +95,9 @@ impl ToolDef {/// `system.permissions.tcc.accessibility`,/// `system.permissions.tcc.screen_recording`/// - `system.config.read`, `system.config.write`-/// - `session.lifecycle.start`, `session.lifecycle.end`+/// - `session.lifecycle.start`, `session.lifecycle.end`,+/// `session.capture_scope`, `session.capture_scope.read`,+/// `session.capture_scope.escalate`/// - `agent_cursor.move`, `agent_cursor.set_enabled`,/// `agent_cursor.set_motion`, `agent_cursor.set_style`,/// `agent_cursor.state`@@ -141,13 +143,10 @@ pub fn default_capabilities_for(tool_name: &str) -> Vec<String> {"mouse_button_up" => &["input.pointer.button"],"scroll" => &["input.pointer.scroll", "accessibility.element_tokens"],"move_cursor" => &[- // Visual overlay move, not a real OS pointer move on- // macOS/Windows — see SKILL.md. Surfaced as- // `agent_cursor.move` because the canonical name on the- // overlay side is "agent cursor"; `input.pointer.move` is- // intentionally omitted to avoid claiming we shift the- // real cursor.+ // Window scope moves the agent overlay; explicit desktop scope+ // moves the real OS pointer."agent_cursor.move",+ "input.pointer.move",],// ── input.keyboard ───────────────────────────────────────────@@ -237,7 +236,9 @@ pub fn default_capabilities_for(tool_name: &str) -> Vec<String> {"set_config" => &["system.config.write"],// ── sessions ─────────────────────────────────────────────────- "start_session" => &["session.lifecycle.start"],+ "start_session" => &["session.lifecycle.start", "session.capture_scope"],+ "escalate_session" => &["session.capture_scope.escalate"],+ "get_session_state" => &["session.capture_scope.read"],"end_session" => &["session.lifecycle.end"],// ── agent cursor ─────────────────────────────────────────────@@ -327,8 +328,12 @@ impl ToolRegistry {/// (`start_session` / `end_session`). Call alongside/// `register_recording_tools` from each platform's `register_all`.pub fn register_session_tools(&mut self) {- use crate::session_tools::{EndSessionTool, StartSessionTool};+ use crate::session_tools::{+ EndSessionTool, EscalateSessionTool, GetSessionStateTool, StartSessionTool,+ };self.register(Box::new(StartSessionTool));+ self.register(Box::new(EscalateSessionTool));+ self.register(Box::new(GetSessionStateTool));self.register(Box::new(EndSessionTool));}@@ -385,9 +390,6 @@ impl ToolRegistry {/// Invoke a tool by name and (if recording is enabled) write its result to disk.pub async fn invoke(&self, name: &str, args: Value) -> ToolResult {- // Capture start time for recording timestamps.- let start_ms = now_ms();-// Deprecated alias: `type_text_chars` → `type_text`. Swift's// ToolRegistry.swift keeps the same alias (with stderr warning) for// backwards compatibility with hermes-agent builds that still emit@@ -401,13 +403,24 @@ impl ToolRegistry {other => other,};+ let Some(tool) = self.tools.get(resolved_name) else {+ return ToolResult::error(format!("Unknown tool: {name}"));+ };+ // Reject modality violations before reserving a recording turn. A+ // rejected action has no before/after evidence and must not leave a+ // pending recorder entry behind.+ if let Err(violation) = crate::capture_scope::enforce_tool(resolved_name, &args) {+ let structured =+ violation.as_json(args.get("session").and_then(Value::as_str).unwrap_or(""));+ return ToolResult::error(violation.message).with_structured(structured);+ }++ // Capture start time for recording timestamps only after validation.+ let start_ms = now_ms();+// Reserve and capture the turn before dispatch so recorded evidence// shows the application immediately before the action changed it.- let should_record = self- .tools- .get(resolved_name)- .map(|tool| !tool.def().read_only)- .unwrap_or(false)+ let should_record = !tool.def().read_only&& !matches!(resolved_name,"start_recording" | "stop_recording" | "get_recording_state" | "replay_trajectory"@@ -426,10 +439,7 @@ impl ToolRegistry {}).flatten();- let result = match self.tools.get(resolved_name) {- Some(tool) => tool.invoke(args.clone()).await,- None => return ToolResult::error(format!("Unknown tool: {name}")),- };+ let result = tool.invoke(args.clone()).await;// Use the original name for downstream code paths below so the// exit-code matching and recording paths keep treating the alias// as a distinct call site.@@ -719,6 +729,8 @@ mod capability_tests {"set_config",// sessions"start_session",+ "escalate_session",+ "get_session_state","end_session",// agent cursor"set_agent_cursor_enabled",@@ -797,6 +809,9 @@ mod capability_tests {// sessions"session.lifecycle.start","session.lifecycle.end",+ "session.capture_scope",+ "session.capture_scope.read",+ "session.capture_scope.escalate",// agent cursor"agent_cursor.move","agent_cursor.set_enabled",diff --git a/libs/cua-driver/rust/crates/cua-driver/src/cli.rs b/libs/cua-driver/rust/crates/cua-driver/src/cli.rsindex c0a8498eb2..a6225f7653 100644--- a/libs/cua-driver/rust/crates/cua-driver/src/cli.rs+++ b/libs/cua-driver/rust/crates/cua-driver/src/cli.rs@@ -3314,6 +3314,12 @@ pub fn run_config_cmd(process::exit(64);}};+ if key == "capture_scope" {+ eprintln!(+ "config key 'capture_scope' is retired; use start_session(capture_scope=auto|window|desktop)"+ );+ process::exit(64);+ }// Try daemon first.if crate::serve::is_daemon_listening(&socket_path) {let req = crate::serve::DaemonRequest {@@ -3353,7 +3359,9 @@ pub fn run_config_cmd((&mut sc, file_cfg){for (k, v) in file_map {- sc_map.insert(k, v);+ if k != "capture_scope" {+ sc_map.insert(k, v);+ }}}// Support dotted key paths like "agent_cursor.enabled".@@ -3388,6 +3396,12 @@ pub fn run_config_cmd(process::exit(64);}};+ if key == "capture_scope" {+ eprintln!(+ "config key 'capture_scope' is retired; use start_session(capture_scope=auto|window|desktop)"+ );+ process::exit(64);+ }let value = match value {Some(v) => v,None => {diff --git a/libs/cua-driver/rust/crates/cua-driver/src/serve.rs b/libs/cua-driver/rust/crates/cua-driver/src/serve.rsindex 45552e7bfd..90064a9b53 100644--- a/libs/cua-driver/rust/crates/cua-driver/src/serve.rs+++ b/libs/cua-driver/rust/crates/cua-driver/src/serve.rs@@ -1753,7 +1753,9 @@ mod gate_tests {let start = DaemonRequest {method: "call".into(),name: Some("start_session".into()),- args: Some(serde_json::json!({})),+ // Session policy is caller-declared. The transport session id is+ // cleanup/ownership metadata and must not grant policy authority.+ args: Some(serde_json::json!({ "session": s3b.clone() })),session_id: Some(s3b),observation_origin: None,};diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/README.md b/libs/cua-driver/rust/crates/cua-driver/tests/README.mdindex 672b0ef54d..031bf26333 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/README.md+++ b/libs/cua-driver/rust/crates/cua-driver/tests/README.md@@ -17,6 +17,7 @@ action, addressing mode, delivery mode, or OS-specific window-system case.| Prefix | Runs by default | Purpose || --- | --- | --- || `protocol_*_test.rs` | yes | MCP/CLI protocol and schema behavior |+| `session_capture_scope_test.rs` | yes | Per-session policy isolation, escalation, lifecycle, and retired config key || `schema_*_test.rs` | yes | Generated schema consistency || `harness_<toolkit>_test.rs` | no, `#[ignore]` | Toolkit-specific harness apps || `desktop_scope_<os>_test.rs` | no, `#[ignore]` | Platform window/desktop scope contract |diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_linux_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_linux_test.rsindex a7c887f167..ef05567943 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_linux_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_linux_test.rs@@ -1,6 +1,6 @@//! Linux **desktop-scope** (vision/foreground) modality through the SAME-//! cua-driver interface as Windows/macOS: `set_config capture_scope=desktop` +-//! a window-less screen-absolute `click` (no `pid`/`window_id`/`list_windows`).+//! cua-driver interface as Windows/macOS: a strict desktop session plus a+//! window-less screen-absolute `click` (no `pid`/`window_id`/`list_windows`).//! The Linux actuator warps the pointer and injects a real button press via the//! XTest extension — the peer of the Windows `WindowFromPoint` + macOS//! global-HID desktop click. (XTest delivering to the under-pointer window is@@ -11,16 +11,14 @@//! `frame` (AT-SPI Component extents), asserts the counter advanced, plus the//! window-scope rejection gate.//!-//! Linux config is global-only (no per-session override), so `set_config`-//! capture_scope writes the on-disk default — the test resets it to `window`-//! before asserting so a failure can't leave the sandbox in desktop scope.+//! A separate strict-window session observes the fixture, proving two live+//! sessions can use different capture scopes without shared mutable config.//!//! #[ignore] (needs an X11/Xwayland display + AT-SPI + the GTK3 harness). Run://! cargo test -p cua-driver --test desktop_scope_linux_test -- --ignored --nocapture --test-threads=1#![cfg(target_os = "linux")]-use std::panic::{self, AssertUnwindSafe};use std::process::{Command, Stdio};use std::time::{Duration, Instant};@@ -87,11 +85,11 @@ fn launch(driver: &mut McpDriver) -> Option<(u32, u64)> {None}-fn ax_snapshot(driver: &mut McpDriver, pid: u32, wid: u64) -> serde_json::Value {+fn ax_snapshot(driver: &mut McpDriver, session: &str, pid: u32, wid: u64) -> serde_json::Value {driver.call("get_window_state",- serde_json::json!({ "pid": pid as i64, "window_id": wid, "capture_mode": "ax" }),+ serde_json::json!({ "session": session, "pid": pid as i64, "window_id": wid, "capture_mode": "ax" }),).structured().clone()@@ -129,28 +127,18 @@ fn counter(snap: &serde_json::Value) -> Option<u64> {digits.parse().ok()}-fn set_scope(driver: &mut McpDriver, scope: &str) {+fn start_scope(driver: &mut McpDriver, session: &str, scope: &str) {let r = driver.call(- "set_config",- serde_json::json!({ "key": "capture_scope", "value": scope }),+ "start_session",+ serde_json::json!({ "session": session, "capture_scope": scope }),);assert!(!r.is_error(),- "set_config capture_scope={scope} failed: {}",+ "start_session capture_scope={scope} failed: {}",r.text());}-fn with_desktop_scope<R>(driver: &mut McpDriver, test: impl FnOnce(&mut McpDriver) -> R) -> R {- set_scope(driver, "desktop");- let result = panic::catch_unwind(AssertUnwindSafe(|| test(driver)));- set_scope(driver, "window");- match result {- Ok(value) => value,- Err(payload) => panic::resume_unwind(payload),- }-}-// ── tests ───────────────────────────────────────────────────────────────────────/// In desktop scope, a window-less screen-absolute click (no pid/window_id)@@ -177,15 +165,19 @@ fn desktop_scope_windowless_click_lands_on_control() {execute_case(case, |evidence| {let mut driver = McpDriver::spawn_named(cell_id).expect("start source-built Linux driver");*evidence = recording_evidence(driver.recording_dir());+ let window_session = format!("{cell_id}-window");+ let desktop_session = format!("{cell_id}-desktop");+ start_scope(&mut driver, &window_session, "window");+ start_scope(&mut driver, &desktop_session, "desktop");let (pid, wid) = launch(&mut driver).expect("required GTK3 harness did not launch");// Settle for the AT-SPI tree to register the button + its extents.- let mut snap = ax_snapshot(&mut driver, pid, wid);+ let mut snap = ax_snapshot(&mut driver, &window_session, pid, wid);let mut center = increment_center(&snap);let deadline = Instant::now() + Duration::from_secs(8);while center.is_none() && Instant::now() < deadline {std::thread::sleep(Duration::from_millis(400));- snap = ax_snapshot(&mut driver, pid, wid);+ snap = ax_snapshot(&mut driver, &window_session, pid, wid);center = increment_center(&snap);}let Some((cx, cy)) = center else {@@ -195,7 +187,7 @@ fn desktop_scope_windowless_click_lands_on_control() {println!("[desktop-linux] increment button screen-center=({cx},{cy}) pre-counter={pre}");let posture = driver.call("bring_to_front",- serde_json::json!({"pid": pid as i64, "window_id": wid}),+ serde_json::json!({"session": window_session, "pid": pid as i64, "window_id": wid}),);assert!(!posture.is_error(), "could not foreground GTK3 fixture: {}", posture.text());std::thread::sleep(Duration::from_millis(300));@@ -209,11 +201,13 @@ fn desktop_scope_windowless_click_lands_on_control() {// counter further, and `post > pre` still holds. We assert the click was// *dispatched as desktop scope* on the first attempt, and that it eventually// *lands* within the budget.- let (post, first_text) = with_desktop_scope(&mut driver, |driver| {+ let (post, first_text) = {let mut post = pre;let mut first_text = String::new();for attempt in 0..12 {- let clicked = driver.call("click", serde_json::json!({ "x": cx, "y": cy }));+ let clicked = driver.call("click", serde_json::json!({+ "session": desktop_session, "scope": "desktop", "x": cx, "y": cy+ }));if attempt == 0 {first_text = clicked.text().to_string();assert!(@@ -223,13 +217,13 @@ fn desktop_scope_windowless_click_lands_on_control() {);}std::thread::sleep(Duration::from_millis(500));- post = counter(&ax_snapshot(driver, pid, wid)).unwrap_or(pre);+ post = counter(&ax_snapshot(&mut driver, &window_session, pid, wid)).unwrap_or(pre);if post > pre {break;}}(post, first_text)- });+ };assert!(first_text.to_lowercase().contains("desktop scope"),@@ -263,9 +257,11 @@ fn window_scope_rejects_windowless_click() {execute_case(case, |evidence| {let mut driver = McpDriver::spawn_named(cell_id).expect("start source-built Linux driver");*evidence = recording_evidence(driver.recording_dir());- set_scope(&mut driver, "window");+ start_scope(&mut driver, cell_id, "window");driver.start_behavior_recording();- let r = driver.call("click", serde_json::json!({ "x": 100, "y": 100 }));+ let r = driver.call("click", serde_json::json!({+ "session": cell_id, "scope": "desktop", "x": 100, "y": 100+ }));assert!(r.is_error()&& r.structured()["code"].as_str() == Some("desktop_scope_disabled"),diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_macos_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_macos_test.rsindex 9cf4a123fd..e508492937 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_macos_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_macos_test.rs@@ -97,11 +97,11 @@ fn launch(driver: &mut McpDriver) -> Option<(u32, u64)> {None}-fn ax_snapshot(driver: &mut McpDriver, pid: u32, wid: u64) -> serde_json::Value {+fn ax_snapshot(driver: &mut McpDriver, session: &str, pid: u32, wid: u64) -> serde_json::Value {driver.call("get_window_state",- serde_json::json!({ "pid": pid as i64, "window_id": wid, "capture_mode": "ax" }),+ serde_json::json!({ "session": session, "pid": pid as i64, "window_id": wid, "capture_mode": "ax" }),).structured().clone()@@ -185,15 +185,27 @@ fn desktop_scope_windowless_click_lands_on_control() {let mut driver = McpDriver::spawn_macos_daemon_proxy_named(cell_id).expect("start installed macOS daemon proxy");*evidence = recording_evidence(driver.recording_dir());+ let window_session = format!("{cell_id}-window");+ let desktop_session = format!("{cell_id}-desktop");+ for (session, capture_scope) in [+ (window_session.as_str(), "window"),+ (desktop_session.as_str(), "desktop"),+ ] {+ let started = driver.call(+ "start_session",+ serde_json::json!({"session": session, "capture_scope": capture_scope}),+ );+ assert!(!started.is_error(), "start_session failed: {}", started.text());+ }let (pid, wid) = launch(&mut driver).expect("required AppKit harness did not launch");// Settle for the AppKit AX tree to register the button + its frame.- let mut snap = ax_snapshot(&mut driver, pid, wid);+ let mut snap = ax_snapshot(&mut driver, &window_session, pid, wid);let mut center = increment_center(&snap);let deadline = Instant::now() + Duration::from_secs(8);while center.is_none() && Instant::now() < deadline {std::thread::sleep(Duration::from_millis(400));- snap = ax_snapshot(&mut driver, pid, wid);+ snap = ax_snapshot(&mut driver, &window_session, pid, wid);center = increment_center(&snap);}let Some((cx, cy)) = center else {@@ -209,7 +221,7 @@ fn desktop_scope_windowless_click_lands_on_control() {// Window-less screen-absolute click — no pid, no window_id; scope per-call.let clicked = driver.call("click",- serde_json::json!({ "x": cx, "y": cy, "scope": "desktop" }),+ serde_json::json!({ "session": desktop_session, "x": cx, "y": cy, "scope": "desktop" }),);assert!(!clicked.is_error(),@@ -224,7 +236,7 @@ fn desktop_scope_windowless_click_lands_on_control() {println!("[desktop-mac] {}", clicked.text());std::thread::sleep(Duration::from_millis(600));- let post = counter(&ax_snapshot(&mut driver, pid, wid)).unwrap_or(pre);+ let post = counter(&ax_snapshot(&mut driver, &window_session, pid, wid)).unwrap_or(pre);assert!(post > pre,"desktop click did not advance AppKit counter at ({cx},{cy}): {pre} -> {post}"@@ -255,11 +267,15 @@ fn window_scope_rejects_windowless_click() {let mut driver = McpDriver::spawn_macos_daemon_proxy_named(cell_id).expect("start installed macOS daemon proxy");*evidence = recording_evidence(driver.recording_dir());- // Default scope is "window" — a window-less click must be rejected.+ let started = driver.call(+ "start_session",+ serde_json::json!({"session": cell_id, "capture_scope": "window"}),+ );+ assert!(!started.is_error(), "start_session failed: {}", started.text());driver.start_behavior_recording();let r = driver.call("click",- serde_json::json!({ "x": 100, "y": 100, "scope": "window" }),+ serde_json::json!({ "session": cell_id, "x": 100, "y": 100, "scope": "desktop" }),);assert!(r.is_error() && r.structured()["code"].as_str() == Some("desktop_scope_disabled"),diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_windows_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_windows_test.rsindex 07d9d7de9f..b119e3de7f 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_windows_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/desktop_scope_windows_test.rs@@ -6,16 +6,15 @@//! cover. This test exercises the Windows Phase-1 actuator end-to-end against a//! real harness app://!-//! 1. `set_config capture_scope=desktop` → `get_desktop_state` returns a+//! 1. A strict desktop session → `get_desktop_state` returns a//! full-display capture with true `screen_width/height` (no downscale).//! 2. A **window-less** screen-absolute `click` / `scroll` (no pid/window_id)//! lands via `WindowFromPoint` while in desktop scope.//! 3. Negative gate: the same window-less `click` under `capture_scope=window`//! is rejected with the structured `desktop_scope_disabled` error.//!-//! Note: `set_config` is a *session* override — it persists for the lifetime of-//! the one MCP server we spawn here (not across separate `cua-driver call`-//! processes), which is exactly why this test drives a single long-lived server.+//! A separate strict-window session observes the WPF fixture. Keeping both live+//! proves capture policy is isolated per session rather than process-global.//!//! All tests are `#[ignore]` (need a real desktop session). Run explicitly://! cargo test -p cua-driver --test desktop_scope_windows_test -- --ignored --nocapture --test-threads=1@@ -88,10 +87,15 @@ fn launch_wpf(driver: &mut McpDriver) -> Option<(u32, u64)> {None}-fn snapshot(driver: &mut McpDriver, pid: u32, wid: u64) -> cua_driver_testkit::ToolResponse {+fn snapshot(+ driver: &mut McpDriver,+ session: &str,+ pid: u32,+ wid: u64,+) -> cua_driver_testkit::ToolResponse {driver.call("get_window_state",- serde_json::json!({ "pid": pid as i64, "window_id": wid, "capture_mode": "ax" }),+ serde_json::json!({ "session": session, "pid": pid as i64, "window_id": wid, "capture_mode": "ax" }),)}@@ -115,20 +119,20 @@ fn element_center(state: &cua_driver_testkit::ToolResponse, id: &str) -> (i64, i)}-fn set_scope(driver: &mut McpDriver, scope: &str) {+fn start_scope(driver: &mut McpDriver, session: &str, scope: &str) {let r = driver.call(- "set_config",- serde_json::json!({ "key": "capture_scope", "value": scope }),+ "start_session",+ serde_json::json!({ "session": session, "capture_scope": scope }),);assert!(!r.is_error(),- "set_config capture_scope={scope} failed: {}",+ "start_session capture_scope={scope} failed: {}",r.text());assert_eq!(r.structured()["capture_scope"].as_str(),Some(scope),- "set_config did not report capture_scope={scope}: {}",+ "start_session did not report capture_scope={scope}: {}",r.text());}@@ -136,7 +140,7 @@ fn set_scope(driver: &mut McpDriver, scope: &str) {fn run_desktop_fixture_case(action: &str,route: DriverRoute,- test: impl FnOnce(u32, u64, &mut McpDriver),+ test: impl FnOnce(u32, u64, &str, &str, &mut McpDriver),) {let cell_id = format!("windows-wpf-desktop-{action}-px-foreground").replace('_', "-");let case = CaseSpec::delivered(@@ -154,16 +158,19 @@ fn run_desktop_fixture_case(let mut driver =McpDriver::spawn_named(&cell_id).expect("required source-built driver did not start");*evidence = recording_evidence(driver.recording_dir());+ let window_session = format!("{cell_id}-window");+ let desktop_session = format!("{cell_id}-desktop");+ start_scope(&mut driver, &window_session, "window");+ start_scope(&mut driver, &desktop_session, "desktop");let (pid, wid) = launch_wpf(&mut driver).expect("required WPF harness did not launch");- set_scope(&mut driver, "desktop");let posture = driver.call("bring_to_front",- serde_json::json!({"pid": pid as i64, "window_id": wid}),+ serde_json::json!({"session": window_session, "pid": pid as i64, "window_id": wid}),);assert!(!posture.is_error(), "could not foreground WPF fixture: {}", posture.text());std::thread::sleep(Duration::from_millis(300));driver.start_behavior_recording();- test(pid, wid, &mut driver);+ test(pid, wid, &window_session, &desktop_session, &mut driver);Observation::delivered_with_fixture_state(Vec::new())});}@@ -191,9 +198,13 @@ fn desktop_scope_capture_returns_screen_dims() {let mut driver = McpDriver::spawn_named("windows-desktop-state-px-not-applicable").expect("required source-built driver did not start");*evidence = recording_evidence(driver.recording_dir());- set_scope(&mut driver, "desktop");+ let session = "windows-desktop-state-px-not-applicable-desktop";+ start_scope(&mut driver, session, "desktop");driver.start_behavior_recording();- let response = driver.call("get_desktop_state", serde_json::json!({}));+ let response = driver.call(+ "get_desktop_state",+ serde_json::json!({"session": session}),+ );assert!(!response.is_error(),"get_desktop_state errored: {}",@@ -217,17 +228,19 @@ fn desktop_scope_windowless_click_lands_on_control() {run_desktop_fixture_case("left_click",DriverRoute::WindowsSendInput,- |pid, wid, driver| {- let pre = snapshot(driver, pid, wid);+ |pid, wid, window_session, desktop_session, driver| {+ let pre = snapshot(driver, window_session, pid, wid);let (x, y) = element_center(&pre, "border-click-target");- let response = driver.call("click", serde_json::json!({ "x": x, "y": y }));+ let response = driver.call("click", serde_json::json!({+ "session": desktop_session, "scope": "desktop", "x": x, "y": y+ }));assert!(!response.is_error(),"desktop click failed: {}",response.text());std::thread::sleep(Duration::from_millis(400));- let after = snapshot(driver, pid, wid);+ let after = snapshot(driver, window_session, pid, wid);assert!(after.tree_text().contains("last_action=left_click"),"desktop click did not update the WPF click oracle: {}",@@ -243,8 +256,8 @@ fn desktop_scope_windowless_scroll_lands_on_control() {run_desktop_fixture_case("scroll",DriverRoute::WindowsSendInput,- |pid, wid, driver| {- let pre = snapshot(driver, pid, wid);+ |pid, wid, window_session, desktop_session, driver| {+ let pre = snapshot(driver, window_session, pid, wid);// The fixture's outer ScrollViewer is visible while the nested// scroll-tall region begins below a 768px CI desktop. Wheel over a// visible child and verify the outer viewport moved by observing a@@ -253,7 +266,10 @@ fn desktop_scope_windowless_scroll_lands_on_control() {let (_, before_y) = element_center(&pre, "btn-increment");let response = driver.call("scroll",- serde_json::json!({ "x": x, "y": y, "direction": "down", "amount": 5 }),+ serde_json::json!({+ "session": desktop_session, "scope": "desktop", "x": x, "y": y,+ "direction": "down", "amount": 5+ }),);assert!(!response.is_error(),@@ -261,7 +277,7 @@ fn desktop_scope_windowless_scroll_lands_on_control() {response.text());std::thread::sleep(Duration::from_millis(500));- let after = snapshot(driver, pid, wid);+ let after = snapshot(driver, window_session, pid, wid);let (_, after_y) = element_center(&after, "btn-increment");assert!(after_y < before_y,@@ -291,9 +307,12 @@ fn window_scope_rejects_windowless_click() {let mut driver = McpDriver::spawn_named("windows-window-scope-gate-px-not-applicable").expect("required source-built driver did not start");*evidence = recording_evidence(driver.recording_dir());- set_scope(&mut driver, "window");+ let session = "windows-window-scope-gate-px-not-applicable";+ start_scope(&mut driver, session, "window");driver.start_behavior_recording();- let response = driver.call("click", serde_json::json!({ "x": 100, "y": 100 }));+ let response = driver.call("click", serde_json::json!({+ "session": session, "scope": "desktop", "x": 100, "y": 100+ }));assert!(response.is_error()&& response.structured()["code"].as_str() == Some("desktop_scope_disabled"),diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/protocol_schema_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/protocol_schema_test.rsindex 18533635bb..abc14b49db 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/protocol_schema_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/protocol_schema_test.rs@@ -3,7 +3,7 @@//! These never invoke a tool — they only inspect the advertised inputSchemas://! that `type_text_chars` is hidden, the `list_windows.on_screen_only` knob, the//! `set_agent_cursor_motion` Bezier knobs, delivery and scope enums, and the-//! `set_config.capture_mode` enum.+//! `set_config.capture_mode` enum and the per-session capture-scope contract.#![cfg(any(target_os = "linux", target_os = "macos", target_os = "windows"))]@@ -120,22 +120,35 @@ fn tools_list_schema_shape() {"{tool}.delivery_mode should advertise background and foreground: {delivery:?}");}- // macOS selects desktop coordinates per click. Linux and Windows retain- // the process-level capture_scope gate used by their desktop-state tools.- #[cfg(target_os = "macos")]- {- let scope = &properties("click")["scope"];+ // Capture scope belongs to the session lifecycle on every platform. The+ // action-level `scope` selects a coordinate/transport form but cannot+ // override the session policy enforced by the registry.+ let capture_scope = &properties("start_session")["capture_scope"];+ for expected in ["auto", "window", "desktop"] {assert!(- enum_contains(scope, "window") && enum_contains(scope, "desktop"),- "click.scope should advertise window and desktop: {scope:?}"+ enum_contains(capture_scope, expected),+ "start_session.capture_scope should advertise {expected}: {capture_scope:?}");}- #[cfg(any(target_os = "linux", target_os = "windows"))]- {- let capture_scope = &properties("set_config")["capture_scope"];+ assert!(+ tools.iter().any(|tool| tool["name"] == "escalate_session"),+ "escalate_session missing from tools/list"+ );+ assert!(+ tools.iter().any(|tool| tool["name"] == "get_session_state"),+ "get_session_state missing from tools/list"+ );+ assert!(+ properties("set_config")["capture_scope"].is_null(),+ "persistent set_config.capture_scope must remain retired"+ );+ for action in [+ "click", "type_text", "press_key", "hotkey", "scroll", "drag", "move_cursor",+ ] {+ let scope = &properties(action)["scope"];assert!(- enum_contains(capture_scope, "window") && enum_contains(capture_scope, "desktop"),- "set_config.capture_scope should advertise window and desktop: {capture_scope:?}"+ enum_contains(scope, "window") && enum_contains(scope, "desktop"),+ "{action}.scope should advertise window and desktop: {scope:?}");}diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/protocol_tools_call_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/protocol_tools_call_test.rsindex c20e73955d..1bba2c470f 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/protocol_tools_call_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/protocol_tools_call_test.rs@@ -85,8 +85,8 @@ fn get_config_and_check_permissions() {}// set_config — change max_image_dimension and verify get_config reflects it.- // capture_mode / capture_scope are no longer settings on macOS (per-call- // params now — modality-ladder refactor); max_image_dimension is the+ // capture_mode / capture_scope are no longer persistent settings on macOS+ // (capture_scope is per-session); max_image_dimension is the// cross-platform persisted field this exercises.d.send(&serde_json::json!({"jsonrpc":"2.0","id":4,"method":"tools/call",diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/session_capture_scope_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/session_capture_scope_test.rsnew file mode 100644index 0000000000..1feff78cff--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver/tests/session_capture_scope_test.rs@@ -0,0 +1,154 @@+//! Transport-level contract tests for per-session capture scope.++#![cfg(any(target_os = "linux", target_os = "macos", target_os = "windows"))]++use cua_driver_testkit::RawDriver;+use serde_json::{json, Value};++fn call(driver: &mut RawDriver, id: u64, name: &str, arguments: Value) -> Value {+ driver.send(&json!({+ "jsonrpc": "2.0",+ "id": id,+ "method": "tools/call",+ "params": {"name": name, "arguments": arguments}+ }));+ driver.recv()+}++fn code(response: &Value) -> Option<&str> {+ response["result"]["structuredContent"]["code"].as_str()+}++#[test]+fn policies_are_isolated_immutable_and_enforced_over_mcp() {+ let mut driver = RawDriver::spawn().expect("spawn source-built driver");+ driver.send(&json!({"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}));+ driver.recv();++ for (id, session, scope) in [+ (2, "scope-auto", "auto"),+ (3, "scope-window", "window"),+ (4, "scope-desktop", "desktop"),+ ] {+ let response = call(+ &mut driver,+ id,+ "start_session",+ json!({"session": session, "capture_scope": scope}),+ );+ assert_eq!(+ response["result"]["structuredContent"]["capture_scope"],+ scope,+ "start_session failed: {response}"+ );+ }++ let auto_desktop = call(+ &mut driver,+ 5,+ "get_desktop_state",+ json!({"session": "scope-auto"}),+ );+ assert_eq!(code(&auto_desktop), Some("desktop_escalation_required"));++ let window_desktop = call(+ &mut driver,+ 6,+ "get_desktop_state",+ json!({"session": "scope-window"}),+ );+ assert_eq!(code(&window_desktop), Some("desktop_scope_disabled"));++ let window_screen_size = call(+ &mut driver,+ 60,+ "get_screen_size",+ json!({"session": "scope-window"}),+ );+ assert_eq!(code(&window_screen_size), Some("desktop_scope_disabled"));+ let desktop_screen_size = call(+ &mut driver,+ 61,+ "get_screen_size",+ json!({"session": "scope-desktop"}),+ );+ assert!(+ code(&desktop_screen_size).is_none(),+ "desktop scope should permit global screen geometry: {desktop_screen_size}"+ );++ let desktop_window = call(+ &mut driver,+ 7,+ "get_window_state",+ json!({"session": "scope-desktop"}),+ );+ assert_eq!(code(&desktop_window), Some("window_scope_disabled"));++ let escalated = call(+ &mut driver,+ 8,+ "escalate_session",+ json!({+ "session": "scope-auto",+ "reason": "foreground_ineffective",+ "detail": "window ladder exhausted"+ }),+ );+ assert_eq!(+ escalated["result"]["structuredContent"]["effective_scope"],+ "desktop"+ );+ let auto_window = call(+ &mut driver,+ 9,+ "get_window_state",+ json!({"session": "scope-auto"}),+ );+ assert_eq!(code(&auto_window), Some("window_scope_disabled"));++ let conflict = call(+ &mut driver,+ 10,+ "start_session",+ json!({"session": "scope-window", "capture_scope": "desktop"}),+ );+ assert_eq!(code(&conflict), Some("session_policy_conflict"));++ let ended = call(+ &mut driver,+ 11,+ "end_session",+ json!({"session": "scope-window"}),+ );+ assert_eq!(ended["result"]["structuredContent"]["active"], false);+ let revived = call(+ &mut driver,+ 12,+ "start_session",+ json!({"session": "scope-window", "capture_scope": "desktop"}),+ );+ assert_eq!(revived["result"]["structuredContent"]["revived"], true);+ assert_eq!(+ revived["result"]["structuredContent"]["capture_scope"],+ "desktop"+ );+}++#[test]+fn persistent_capture_scope_key_is_retired() {+ let mut driver = RawDriver::spawn().expect("spawn source-built driver");+ driver.send(&json!({"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}));+ driver.recv();+ let response = call(+ &mut driver,+ 2,+ "set_config",+ json!({"key": "capture_scope", "value": "desktop"}),+ );+ assert_eq!(code(&response), Some("config_key_retired"));+ assert_eq!(+ response["result"]["structuredContent"]["replacement"],+ "start_session.capture_scope"+ );+}diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/transport_config_persistence_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/transport_config_persistence_test.rsindex cdfd79c13b..dba07dfd53 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/transport_config_persistence_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/transport_config_persistence_test.rs@@ -11,8 +11,8 @@//! visible to later calls on the SAME driver within the session.//!//! Uses `max_image_dimension` as the persisted key. `capture_mode` /-//! `capture_scope` are NO LONGER settings (they are per-call params now —-//! the modality-ladder refactor), so `max_image_dimension` is the remaining+//! `capture_scope` are NO LONGER settings (`capture_scope` is per-session), so+//! `max_image_dimension` is the remaining//! disk-persisted config field and the right probe for this transport behavior.//!//! Both tests are `#[ignore]`: they mutate the real on-disk config, so theydiff --git a/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rsindex 1494da8fa2..8f2c823db8 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs@@ -2052,6 +2052,40 @@ pub fn send_click_xtest_desktop(x: i32, y: i32, button: u8, count: usize) -> ResOk(())}+/// Move the real X11 pointer to a screen-absolute desktop coordinate.+pub fn send_move_xtest_desktop(x: i32, y: i32) -> Result<()> {+ use x11rb::protocol::xtest::ConnectionExt as _;+ let (conn, screen_num) = connect_x11_for_input()?;+ let root = conn.setup().roots[screen_num].root;+ conn.xtest_fake_input(MOTION_NOTIFY_EVENT, 0, 0, root, x as i16, y as i16, 0)?;+ conn.flush()?;+ let _ = conn.get_input_focus()?.reply();+ Ok(())+}++/// Scroll the window under a screen-absolute point via real XTest wheel-button+/// events. X11 buttons 4/5 are vertical up/down and 6/7 horizontal left/right.+pub fn send_scroll_xtest_desktop(x: i32, y: i32, direction: &str, amount: usize) -> Result<()> {+ use x11rb::protocol::xtest::ConnectionExt as _;+ let button = match direction {+ "up" => 4,+ "down" => 5,+ "left" => 6,+ "right" => 7,+ other => anyhow::bail!("unknown desktop scroll direction: {other}"),+ };+ let (conn, screen_num) = connect_x11_for_input()?;+ let root = conn.setup().roots[screen_num].root;+ conn.xtest_fake_input(MOTION_NOTIFY_EVENT, 0, 0, root, x as i16, y as i16, 0)?;+ for _ in 0..amount.max(1) {+ conn.xtest_fake_input(BUTTON_PRESS_EVENT, button, 0, root, x as i16, y as i16, 0)?;+ conn.xtest_fake_input(BUTTON_RELEASE_EVENT, button, 0, root, x as i16, y as i16, 0)?;+ }+ conn.flush()?;+ let _ = conn.get_input_focus()?.reply();+ Ok(())+}+/// Screen-absolute drag via XTest. The caller activates the target first; XTest/// then supplies one real press, interpolated pointer motion, and one release./// This is the foreground counterpart to the window-addressed XSendEvent drag.diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rsindex e325f4f484..49abf9805c 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs@@ -19,7 +19,6 @@ use cursor_overlay::CursorRegistry;#[derive(Clone)]pub struct DriverConfig {pub capture_mode: String,- pub capture_scope: String,pub max_image_dimension: u32,}@@ -27,7 +26,6 @@ impl Default for DriverConfig {fn default() -> Self {Self {capture_mode: "ax".into(),- capture_scope: "window".into(),max_image_dimension: 1568,}}@@ -45,11 +43,6 @@ pub fn load_driver_config() -> DriverConfig {{cfg.capture_mode = v;}- if let Some(v) =- pip_preview::read_config_value("capture_scope").and_then(|v| v.as_str().map(str::to_owned))- {- cfg.capture_scope = v;- }if let Some(v) = pip_preview::read_config_value("max_image_dimension").and_then(|v| v.as_u64()){if let Ok(v32) = u32::try_from(v) {@@ -1750,6 +1743,7 @@ impl Tool for ClickTool {"button":{"type":"string","enum":["left","right","middle"],"description":"Mouse button. Default: \"left\" (legacy back-compat). X11: routed via ButtonPress/Release with the matching evdev code. Native Wayland: only left-button is supported via the virtual-pointer protocol; right/middle return an error."},"count":{"type":"integer"},"from_zoom":{"type":"boolean","description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space."},+ "scope":{"type":"string","enum":["window","desktop"],"default":"window"},"delivery_mode": crate::input::delivery::delivery_mode_schema()},"additionalProperties":false}),@@ -1762,28 +1756,23 @@ impl Tool for ClickTool {// ── Window-less screen-absolute branch (capture_scope="desktop") ──────// x,y with NO pid and NO window_id → TRUE SCREEN pixels. Foreground,- // vision-driven desktop-scope click (the Linux peer of the Windows- // WindowFromPoint / macOS global-HID path). Gate on the effective scope:- // under "window" return the same `desktop_scope_disabled` contract.+ // desktop-scope click (the Linux peer of the Windows WindowFromPoint /+ // macOS global-HID path). The core registry has already enforced the+ // session policy; this branch validates the explicit action form.let has_pid = args.get("pid").map(|v| !v.is_null()).unwrap_or(false);let has_window_id = args.get("window_id").map(|v| !v.is_null()).unwrap_or(false);let has_xy = args.get("x").map(|v| v.is_number()).unwrap_or(false)&& args.get("y").map(|v| v.is_number()).unwrap_or(false);if has_xy && !has_pid && !has_window_id {- // Linux config is global-only (no per-session override layer).- let scope = self.state.config.read().unwrap().capture_scope.clone();- if scope != "desktop" {+ if args.get("scope").and_then(Value::as_str) != Some("desktop") {return ToolResult::error(- "click: x,y given with no pid/window_id, but capture_scope is \- \"window\". Screen-absolute clicks require desktop scope. Call \- set_config with capture_scope=desktop (and use get_desktop_state \- to read true screen pixels) first."+ "click: x,y given with no pid/window_id requires scope=\"desktop\"; \+ use get_desktop_state to read true screen pixels first.".to_string(),).with_structured(json!({- "code": "desktop_scope_disabled",- "capture_scope": scope,- "suggestion": "set_config capture_scope=desktop",+ "code": "desktop_coordinate_scope_required",+ "suggestion": "pass scope=desktop",}));}let button_raw = args.str_or("button", "left").to_lowercase();@@ -2218,7 +2207,7 @@ impl Tool for TypeTextTool {name: "type_text".into(),description: "Type text to a window via XSendEvent (KeyPress/KeyRelease). No focus steal.".into(),input_schema: json!({- "type":"object","required":["pid","text"],"properties":{+ "type":"object","required":["text"],"properties":{"session": cua_driver_core::tool_schema::session_schema(),"pid":{"type":"integer"},"window_id":{"type":"integer"},@@ -2227,6 +2216,7 @@ impl Tool for TypeTextTool {"element_token": cua_driver_core::tool_schema::element_token_schema(),"x":{"type":"number","description":"Screenshot-pixel X of the field to type into — the element px action form. Pass x,y (no element_index) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the AX path can't reach. Read straight off the get_window_state PNG, same convention as click."},"y":{"type":"number","description":"Screenshot-pixel Y of the field (see x)."},+ "scope":{"type":"string","enum":["window","desktop"],"default":"window"},"delivery_mode": crate::input::delivery::delivery_mode_schema()},"additionalProperties":false}),@@ -2236,6 +2226,33 @@ impl Tool for TypeTextTool {async fn invoke(&self, args: Value) -> ToolResult {use cua_driver_core::tool_args::ArgsExt;+ if args.opt_str("scope").as_deref() == Some("desktop") && args.get("pid").is_none() {+ let text = match args.require_str("text") {+ Ok(value) => {+ cua_driver_core::text_sanitize::strip_trailing_agent_protocol_tags(&value)+ .into_owned()+ }+ Err(error) => return error,+ };+ let wayland = crate::wayland::wayland_input_enabled();+ let path = if wayland { "wayland_focused" } else { "xtest" };+ let result = tokio::task::spawn_blocking(move || {+ if wayland {+ crate::wayland::type_text_focused(&text)+ } else {+ crate::input::send_type_text_xtest(&text)+ }+ })+ .await;+ return match result {+ Ok(Ok(())) => ToolResult::text("Typed text into the focused desktop application.")+ .with_structured(+ json!({"scope":"desktop","path":path,"effect":"unverifiable"}),+ ),+ Ok(Err(error)) => ToolResult::error(error.to_string()),+ Err(error) => ToolResult::error(format!("Task error: {error}")),+ };+ }let pid = args.u64_or("pid", 0) as u32;let text_raw = match args.require_str("text") {Ok(v) => v,@@ -2767,7 +2784,7 @@ impl Tool for PressKeyTool {name: "press_key".into(),description: "Press a key via XSendEvent to a window. No focus steal.".into(),input_schema: json!({- "type":"object","required":["pid","key"],"properties":{+ "type":"object","required":["key"],"properties":{"session": cua_driver_core::tool_schema::session_schema(),"pid":{"type":"integer"},"window_id":{"type":"integer"},@@ -2777,6 +2794,7 @@ impl Tool for PressKeyTool {"element_token": cua_driver_core::tool_schema::element_token_schema(),"x":{"type":"number","description":"Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_index."},"y":{"type":"number","description":"Screenshot-pixel Y (see x)."},+ "scope":{"type":"string","enum":["window","desktop"],"default":"window"},"delivery_mode": crate::input::delivery::delivery_mode_schema()},"additionalProperties":false}),@@ -2786,6 +2804,37 @@ impl Tool for PressKeyTool {async fn invoke(&self, args: Value) -> ToolResult {use cua_driver_core::tool_args::ArgsExt;+ if args.opt_str("scope").as_deref() == Some("desktop") && args.get("pid").is_none() {+ let key = match args.require_str("key") {+ Ok(value) => value,+ Err(error) => return error,+ };+ let display = key.clone();+ let modifiers: Vec<String> = args.str_array("modifiers");+ let wayland = crate::wayland::wayland_input_enabled();+ let path = if wayland { "wayland_focused" } else { "xtest" };+ let result = tokio::task::spawn_blocking(move || {+ if wayland && modifiers.is_empty() {+ crate::wayland::press_key_focused(&key)+ } else if wayland {+ let mut keys = modifiers;+ keys.push(key);+ crate::wayland::hotkey_focused(&keys)+ } else {+ let refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();+ crate::input::send_key_xtest(&key, &refs)+ }+ })+ .await;+ return match result {+ Ok(Ok(())) => ToolResult::text(format!("Pressed desktop key '{display}'."))+ .with_structured(+ json!({"scope":"desktop","path":path,"effect":"unverifiable"}),+ ),+ Ok(Err(error)) => ToolResult::error(error.to_string()),+ Err(error) => ToolResult::error(format!("Task error: {error}")),+ };+ }let pid = args.u64_or("pid", 0) as u32;let key = match args.require_str("key") {Ok(v) => v,@@ -3026,7 +3075,7 @@ impl Tool for HotkeyTool {description: "Press a combination of keys simultaneously, e.g. [\"ctrl\",\"c\"] for Copy. \Sent via XSendEvent directly to the target pid; target does NOT need to be frontmost.".into(),input_schema: json!({- "type":"object","required":["pid","keys"],"properties":{+ "type":"object","required":["keys"],"properties":{"session": cua_driver_core::tool_schema::session_schema(),"pid":{"type":"integer"},"window_id":{"type":"integer"},@@ -3036,6 +3085,7 @@ impl Tool for HotkeyTool {"element_token": cua_driver_core::tool_schema::element_token_schema(),"x":{"type":"number","description":"Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the combo (so e.g. Ctrl+V pastes into that field). Pass with y. Use for Chromium/Electron surfaces the background combo can't reach."},"y":{"type":"number","description":"Screenshot-pixel Y (see x)."},+ "scope":{"type":"string","enum":["window","desktop"],"default":"window"},"delivery_mode": crate::input::delivery::delivery_mode_schema()},"additionalProperties":false}),@@ -3045,6 +3095,37 @@ impl Tool for HotkeyTool {async fn invoke(&self, args: Value) -> ToolResult {use cua_driver_core::tool_args::ArgsExt;+ if args.opt_str("scope").as_deref() == Some("desktop") && args.get("pid").is_none() {+ let keys: Vec<String> = args.str_array("keys");+ let modifiers: Vec<String> = keys+ .iter()+ .filter(|key| is_modifier(key))+ .cloned()+ .collect();+ let Some(key) = keys.iter().rev().find(|key| !is_modifier(key)).cloned() else {+ return ToolResult::error("keys must include at least one non-modifier key.");+ };+ let display = keys.join("+");+ let wayland = crate::wayland::wayland_input_enabled();+ let path = if wayland { "wayland_focused" } else { "xtest" };+ let result = tokio::task::spawn_blocking(move || {+ if wayland {+ crate::wayland::hotkey_focused(&keys)+ } else {+ let refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();+ crate::input::send_key_xtest(&key, &refs)+ }+ })+ .await;+ return match result {+ Ok(Ok(())) => ToolResult::text(format!("Pressed desktop hotkey {display}."))+ .with_structured(+ json!({"scope":"desktop","path":path,"effect":"unverifiable"}),+ ),+ Ok(Err(error)) => ToolResult::error(error.to_string()),+ Err(error) => ToolResult::error(format!("Task error: {error}")),+ };+ }let pid = args.u64_or("pid", 0) as u32;let window_id_arg = args.opt_u64("window_id");let element_index_arg = args.opt_u64("element_index").map(|value| value as usize);@@ -3392,6 +3473,7 @@ impl Tool for ScrollTool {"element_token": cua_driver_core::tool_schema::element_token_schema(),"x":{"type":"number","description":"Window-local screenshot-pixel X of the scroll target. Pass with y and without element_index."},"y":{"type":"number","description":"Window-local screenshot-pixel Y of the scroll target. Pass with x and without element_index."},+ "scope":{"type":"string","enum":["window","desktop"],"default":"window"},"delivery_mode": crate::input::delivery::delivery_mode_schema()},"additionalProperties":false}),@@ -3401,6 +3483,43 @@ impl Tool for ScrollTool {async fn invoke(&self, args: Value) -> ToolResult {let cursor_id = resolve_cursor_key(&args);+ if args.opt_str("scope").as_deref() == Some("desktop")+ && args.get("pid").is_none()+ && args.get("window_id").is_none()+ {+ let direction = match args.require_str("direction") {+ Ok(value) => value,+ Err(error) => return error,+ };+ let x = match args.require_f64("x") {+ Ok(value) => value.round() as i32,+ Err(error) => return error,+ };+ let y = match args.require_f64("y") {+ Ok(value) => value.round() as i32,+ Err(error) => return error,+ };+ let amount = args.u64_or("amount", 3).clamp(1, 50) as usize;+ let display = direction.clone();+ let wayland = crate::wayland::wayland_input_enabled();+ let path = if wayland { "wayland_desktop" } else { "xtest" };+ let result = tokio::task::spawn_blocking(move || {+ if wayland {+ crate::wayland::scroll_desktop(x, y, &direction, amount as u32)+ } else {+ crate::input::send_scroll_xtest_desktop(x, y, &direction, amount)+ }+ })+ .await;+ return match result {+ Ok(Ok(())) => ToolResult::text(format!("Scrolled desktop {display} × {amount}."))+ .with_structured(+ json!({"scope":"desktop","path":path,"effect":"unverifiable"}),+ ),+ Ok(Err(error)) => ToolResult::error(error.to_string()),+ Err(error) => ToolResult::error(format!("Task error: {error}")),+ };+ }let pid = match args.require_u32("pid") {Ok(v) => v,Err(e) => return e,@@ -4199,7 +4318,7 @@ impl Tool for DragTool {description: "Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in \window-local screenshot pixels via XSendEvent (ButtonPress + MotionNotify × steps + ButtonRelease). \duration_ms (default 500), steps (default 20). No focus steal.".into(),- input_schema: json!({"type":"object","required":["pid","from_x","from_y","to_x","to_y"],"properties":{+ input_schema: json!({"type":"object","required":["from_x","from_y","to_x","to_y"],"properties":{"session": cua_driver_core::tool_schema::session_schema(),"cursor_id":{"type":"string","description":"Optional multi-cursor instance id. Default: 'default'."},"pid":{"type":"integer"},@@ -4213,6 +4332,7 @@ impl Tool for DragTool {"modifier": cua_driver_core::tool_schema::modifier_schema(),"button": cua_driver_core::tool_schema::button_schema(),"from_zoom":{"type":"boolean"},+ "scope":{"type":"string","enum":["window","desktop"],"default":"window"},"delivery_mode": crate::input::delivery::delivery_mode_schema()},"additionalProperties":false}),read_only: false, destructive: true, idempotent: false, open_world: true,@@ -4220,6 +4340,62 @@ impl Tool for DragTool {}async fn invoke(&self, args: Value) -> ToolResult {let cursor_id = resolve_cursor_key(&args);+ if args.opt_str("scope").as_deref() == Some("desktop")+ && args.get("pid").is_none()+ && args.get("window_id").is_none()+ {+ let coordinate = |key: &str| {+ args.opt_f64(key)+ .or_else(|| args.opt_i64(key).map(|value| value as f64))+ };+ let Some(from_x) = coordinate("from_x") else {+ return ToolResult::error("Missing: from_x");+ };+ let Some(from_y) = coordinate("from_y") else {+ return ToolResult::error("Missing: from_y");+ };+ let Some(to_x) = coordinate("to_x") else {+ return ToolResult::error("Missing: to_x");+ };+ let Some(to_y) = coordinate("to_y") else {+ return ToolResult::error("Missing: to_y");+ };+ let button = parse_mouse_button(&args.str_or("button", "left"));+ let duration_ms = args.u64_or("duration_ms", 500).min(10_000);+ let steps = args.u64_or("steps", 20).clamp(1, 200) as usize;+ let wayland = crate::wayland::wayland_input_enabled();+ let path = if wayland { "wayland_desktop" } else { "xtest" };+ let result = tokio::task::spawn_blocking(move || {+ if wayland {+ crate::wayland::drag_desktop(+ from_x.round() as i32,+ from_y.round() as i32,+ to_x.round() as i32,+ to_y.round() as i32,+ steps as u32,+ button,+ )+ } else {+ crate::input::send_drag_xtest_desktop(+ from_x.round() as i32,+ from_y.round() as i32,+ to_x.round() as i32,+ to_y.round() as i32,+ button,+ duration_ms,+ steps,+ )+ }+ })+ .await;+ return match result {+ Ok(Ok(())) => ToolResult::text("Dragged on the desktop.").with_structured(+ json!({"scope":"desktop","path":path,"effect":"unverifiable"}),+ ),+ Ok(Err(error)) => ToolResult::error(error.to_string()),+ Err(error) => ToolResult::error(format!("Task error: {error}")),+ };+ }let pid = match args.require_u32("pid") {Ok(v) => v,Err(e) => return e,@@ -5456,28 +5632,6 @@ impl Tool for GetDesktopStateTool {}async fn invoke(&self, args: Value) -> ToolResult {- // Gate on the global capture_scope: a full-display capture is a- // desktop-scope operation, available only when capture_scope="desktop"- // (same gate as window-less screen-absolute click/scroll). Read the- // persisted value the same way load_config does.- let scope = pip_preview::read_config_value("capture_scope")- .and_then(|v| v.as_str().map(str::to_owned))- .unwrap_or_else(|| "window".to_owned());- if scope != "desktop" {- return ToolResult::error(format!(- "get_desktop_state requires capture_scope=\"desktop\" (current scope is \- \"{scope}\"). Full-display capture is a desktop-scope operation; call \- set_config with capture_scope=desktop first (it also enables window-less \- screen-absolute click/scroll). For a single window, use \- get_window_state(pid, window_id) instead."- ))- .with_structured(serde_json::json!({- "code": "desktop_scope_disabled",- "capture_scope": scope,- "suggestion": "set_config capture_scope=desktop",- }));- }-let out_file = args.opt_str("screenshot_out_file");let result = tokio::task::spawn_blocking(move || -> anyhow::Result<_> {@@ -5521,10 +5675,12 @@ impl Tool for GetDesktopStateTool {let mut content = Vec::new();let mut structured = json!({"platform": "linux",+ "display": "primary","screenshot_width": shot_w,"screenshot_height": shot_h,"screen_width": screen_w,"screen_height": screen_h,+ "scale_factor": 1.0,"screenshot_mime_type": "image/png",});if let Some(b64) = b64_opt {@@ -5621,9 +5777,9 @@ impl Tool for MoveCursorTool {fn def(&self) -> &ToolDef {MCURSOR_DEF.get_or_init(|| ToolDef {name: "move_cursor".into(),- description: "Move the agent cursor overlay to (x, y). Does NOT move the real mouse cursor.".into(),+ description: "Move the agent cursor overlay, or with scope=desktop move the real OS pointer in get_desktop_state coordinates.".into(),input_schema: json!({"type":"object","required":["x","y"],"properties":{- "x":{"type":"number"},"y":{"type":"number"},"session": cua_driver_core::tool_schema::session_schema(),"cursor_id":{"type":"string"}+ "x":{"type":"number"},"y":{"type":"number"},"session": cua_driver_core::tool_schema::session_schema(),"cursor_id":{"type":"string"},"scope":{"type":"string","enum":["window","desktop"],"default":"window"}},"additionalProperties":false}),read_only: false, destructive: false, idempotent: true, open_world: false,})@@ -5632,6 +5788,29 @@ impl Tool for MoveCursorTool {use cua_driver_core::tool_args::ArgsExt;let x = args.f64_or("x", 0.0);let y = args.f64_or("y", 0.0);+ if args.opt_str("scope").as_deref() == Some("desktop") {+ let xi = x.round() as i32;+ let yi = y.round() as i32;+ let result = if crate::wayland::wayland_input_enabled() {+ tokio::task::spawn_blocking(move || {+ crate::wayland::move_cursor_absolute(None, xi, yi)+ })+ .await+ } else {+ tokio::task::spawn_blocking(move || crate::input::send_move_xtest_desktop(xi, yi))+ .await+ };+ return match result {+ Ok(Ok(())) => {+ ToolResult::text(format!("Moved the real desktop pointer to ({xi}, {yi})."))+ .with_structured(+ json!({"scope":"desktop","x":xi,"y":yi,"effect":"unverifiable"}),+ )+ }+ Ok(Err(error)) => ToolResult::error(error.to_string()),+ Err(error) => ToolResult::error(format!("Task error: {error}")),+ };+ }let window_id = args.get("window_id").and_then(|v| v.as_u64());let cursor_id = resolve_cursor_key(&args);self.state.cursor_registry.update_position(&cursor_id, x, y);@@ -6172,7 +6351,6 @@ impl Tool for GetConfigTool {"source_sha": option_env!("CUA_DRIVER_SOURCE_SHA"),"platform": "linux","capture_mode": cfg.capture_mode,- "capture_scope": cfg.capture_scope,"max_image_dimension": cfg.max_image_dimension,"experimental_pip": pip_enabled,"experimental_pip_geometry": pip_geometry@@ -6205,7 +6383,6 @@ impl Tool for SetConfigTool {"key":{"type":"string","description":"Name of a single config field to write ({key, value} shape). Pair with `value`."},"value":{"description":"New value for `key`. JSON type depends on the key."},"capture_mode":{"type":"string","enum":["ax","vision"],"description":"Legacy per-field shape. Default capture mode for get_window_state. (\"som\"/\"screenshot\" still decode as deprecated aliases.)"},- "capture_scope":{"type":"string","enum":["window","desktop"],"description":"Capture scope: single window or whole desktop. Default window."},"max_image_dimension":{"type":"integer","description":"Legacy per-field shape. Max dimension for screenshot resizing (0 = no limit)."},"experimental_pip":{"type":"boolean","description":"Enable the experimental PiP preview window (applies next restart; Linux backend stubbed)."},"experimental_pip_geometry":{"type":"string","description":"PiP window size + optional position in `WxH` or `WxH+X+Y` form."}@@ -6215,6 +6392,18 @@ impl Tool for SetConfigTool {}async fn invoke(&self, args: Value) -> ToolResult {use cua_driver_core::tool_args::ArgsExt;+ if args.get("capture_scope").is_some()+ || args.get("key").and_then(Value::as_str) == Some("capture_scope")+ {+ return ToolResult::error(+ "config key 'capture_scope' is retired; pass capture_scope=auto|window|desktop to start_session",+ )+ .with_structured(json!({+ "code": "config_key_retired",+ "key": "capture_scope",+ "replacement": "start_session.capture_scope",+ }));+ }let mut cfg = self.state.config.write().unwrap();let mut parts = Vec::new();// {key, value} shape (what the Swift/macOS and Windows callers send).@@ -6235,17 +6424,6 @@ impl Tool for SetConfigTool {}None => return ToolResult::error(format!("`capture_mode` must be a string, got {val}.")),},- "capture_scope" => match val.as_str() {- Some(s @ ("window" | "desktop")) => {- cfg.capture_scope = s.to_owned();- if let Err(e) = pip_preview::write_config_key("capture_scope", Value::String(s.to_owned())) {- tracing::warn!("set_config: failed to persist capture_scope: {e}");- }- parts.push(format!("capture_scope={s}"));- }- Some(other) => return ToolResult::error(format!("`capture_scope` must be \"window\" or \"desktop\", got \"{other}\".")),- None => return ToolResult::error(format!("`capture_scope` must be a string, got {val}.")),- },"max_image_dimension" => match val.as_u64() {Some(n) => {cfg.max_image_dimension = n as u32;@@ -6280,7 +6458,7 @@ impl Tool for SetConfigTool {None => return ToolResult::error(format!("`experimental_pip_geometry` must be a string, got {val}.")),},other => return ToolResult::error(format!(- "Unknown config key `{other}`. Known: capture_mode, capture_scope, max_image_dimension, experimental_pip, experimental_pip_geometry."+ "Unknown config key `{other}`. Known: capture_mode, max_image_dimension, experimental_pip, experimental_pip_geometry.")),}}@@ -6294,20 +6472,6 @@ impl Tool for SetConfigTool {parts.push(format!("capture_mode={mode}"));cfg.capture_mode = mode;}- if let Some(scope) = args.opt_str("capture_scope") {- if scope != "window" && scope != "desktop" {- return ToolResult::error(format!(- "`capture_scope` must be \"window\" or \"desktop\", got \"{scope}\"."- ));- }- if let Err(e) =- pip_preview::write_config_key("capture_scope", Value::String(scope.clone()))- {- tracing::warn!("set_config: failed to persist capture_scope: {e}");- }- parts.push(format!("capture_scope={scope}"));- cfg.capture_scope = scope;- }if let Some(dim) = args.opt_u64("max_image_dimension") {cfg.max_image_dimension = dim as u32;if let Err(e) = pip_preview::write_config_key("max_image_dimension", Value::from(dim)) {@@ -6346,7 +6510,6 @@ impl Tool for SetConfigTool {let (pip_enabled, pip_geometry) = pip_preview::read_pip_keys_from_file();ToolResult::text(msg).with_structured(json!({"capture_mode": cfg.capture_mode,- "capture_scope": cfg.capture_scope,"max_image_dimension": cfg.max_image_dimension,"experimental_pip": pip_enabled,"experimental_pip_geometry": pip_geometry@@ -6989,16 +7152,6 @@ mod click_button_schema_tests {}}-#[cfg(test)]-mod driver_config_tests {- use super::DriverConfig;-- #[test]- fn capture_scope_defaults_to_window() {- assert_eq!(DriverConfig::default().capture_scope, "window");- }-}-#[cfg(test)]mod browser_launch_guard_tests {use super::contains_remote_debugging_flag;diff --git a/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rsindex b5dd150ca0..fcaa88e8c3 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs@@ -1424,7 +1424,7 @@ pub fn scroll_at() -> anyhow::Result<()> {let direction = direction.to_string();with_libei_fallback(- || scroll_vptr(window_id, point, &direction, amount),+ || scroll_vptr(Some(window_id), point, &direction, amount),|| {libei_wait_scroll_ready()?;activate_window_for_input(window_id)?;@@ -1436,14 +1436,32 @@ pub fn scroll_at()}+/// Scroll at a desktop-absolute point without activating a named toplevel.+pub fn scroll_desktop(+ x: i32,+ y: i32,+ direction: &str,+ amount: u32,+) -> anyhow::Result<()> {+ let direction = direction.to_string();+ with_libei_fallback(+ || scroll_vptr(None, Some((x, y)), &direction, amount),+ || {+ libei_wait_scroll_ready()?;+ libei_move_absolute(x, y)?;+ libei_scroll(&direction, amount)+ },+ )+}+/// wlroots virtual-pointer implementation of [`scroll`].fn scroll_vptr(- window_id: u64,+ window_id: Option<u64>,point: Option<(i32, i32)>,direction: &str,amount: u32,) -> anyhow::Result<()> {- let mut sess = open_vptr_session(Some(window_id))?;+ let mut sess = open_vptr_session(window_id)?;if let Some((x, y)) = point {let px = x.clamp(0, (sess.output_w as i32).saturating_sub(1)) as u32;let py = y.clamp(0, (sess.output_h as i32).saturating_sub(1)) as u32;@@ -1549,7 +1567,7 @@ pub fn drag(button: u8,) -> anyhow::Result<()> {with_libei_fallback(- || drag_vptr(window_id, from_x, from_y, to_x, to_y, steps, button),+ || drag_vptr(Some(window_id), from_x, from_y, to_x, to_y, steps, button),|| {libei_wait_pointer_ready()?;activate_window_for_input(window_id)?;@@ -1558,10 +1576,28 @@ pub fn drag()}+/// Drag through desktop-absolute points without activating a named toplevel.+pub fn drag_desktop(+ from_x: i32,+ from_y: i32,+ to_x: i32,+ to_y: i32,+ steps: u32,+ button: u8,+) -> anyhow::Result<()> {+ with_libei_fallback(+ || drag_vptr(None, from_x, from_y, to_x, to_y, steps, button),+ || {+ libei_wait_pointer_ready()?;+ libei_drag(from_x, from_y, to_x, to_y, steps, button)+ },+ )+}+/// wlroots virtual-pointer implementation of [`drag`].#[allow(clippy::too_many_arguments)]fn drag_vptr(- window_id: u64,+ window_id: Option<u64>,from_x: i32,from_y: i32,to_x: i32,@@ -1569,7 +1605,7 @@ fn drag_vptr(steps: u32,button: u8,) -> anyhow::Result<()> {- let mut sess = open_vptr_session(Some(window_id))?;+ let mut sess = open_vptr_session(window_id)?;std::thread::sleep(std::time::Duration::from_millis(40));let (w, h) = (sess.output_w, sess.output_h);let btn = evdev_pointer_button(button);diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rsindex 775203884d..f4aef93504 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs@@ -146,6 +146,32 @@ pub fn press_key_global(key: &str, modifiers: &[&str]) -> anyhow::Result<()> {Ok(())}+/// Type Unicode text into the frontmost application through the global HID+/// queue. This is the desktop-scope counterpart to PID-routed `type_text` and+/// mirrors computer-server's frontmost pynput typing behavior.+pub fn type_text_global(text: &str, inter_char_delay_ms: u64) -> anyhow::Result<()> {+ use core_graphics::event::CGEventTapLocation;++ let source = CGEventSource::new(CGEventSourceStateID::HIDSystemState)+ .map_err(|_| anyhow::anyhow!("CGEventSource::new failed"))?;+ for ch in text.chars() {+ let value = ch.to_string();+ let down = CGEvent::new_keyboard_event(source.clone(), 0, true)+ .map_err(|_| anyhow::anyhow!("CGEvent keyboard down failed"))?;+ down.set_string(&value);+ down.set_flags(CGEventFlags::CGEventFlagNull);+ down.post(CGEventTapLocation::HID);+ std::thread::sleep(std::time::Duration::from_millis(8));+ let up = CGEvent::new_keyboard_event(source.clone(), 0, false)+ .map_err(|_| anyhow::anyhow!("CGEvent keyboard up failed"))?;+ up.set_string(&value);+ up.set_flags(CGEventFlags::CGEventFlagNull);+ up.post(CGEventTapLocation::HID);+ std::thread::sleep(std::time::Duration::from_millis(inter_char_delay_ms.max(8)));+ }+ Ok(())+}+/// Post a keyboard event to `pid` via SLEventPostToPid (with auth message for/// Chromium/Electron support) or fall back to CGEvent::post_to_pid.fn post_keyboard_event(pid: i32, event: &CGEvent) {diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rsindex 0726951522..9a8fe09680 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs@@ -87,6 +87,45 @@ pub fn click_at_xy_desktop(x: f64, y: f64, count: usize, button: &str) -> anyhowOk(())}+/// Move the real hardware cursor to a logical desktop point.+pub fn move_cursor_desktop(x: f64, y: f64) -> anyhow::Result<()> {+ use core_graphics::display::CGDisplay;+ let point = CGPoint::new(x, y);+ CGDisplay::warp_mouse_cursor_position(point)+ .map_err(|error| anyhow::anyhow!("CGWarpMouseCursorPosition failed: {error:?}"))?;+ unsafe { CGAssociateMouseAndMouseCursorPosition(true) };+ Ok(())+}++/// Scroll the foreground desktop surface at a logical screen point through the+/// global HID queue. Mirrors computer-server's pynput wheel behavior while+/// preserving cua-driver's explicit direction/amount contract.+pub fn scroll_wheel_desktop(+ x: f64,+ y: f64,+ delta_y_per_tick: i32,+ delta_x_per_tick: i32,+ ticks: usize,+) -> anyhow::Result<()> {+ use core_graphics::event::{CGEventTapLocation, ScrollEventUnit};++ move_cursor_desktop(x, y)?;+ std::thread::sleep(std::time::Duration::from_millis(40));+ for _ in 0..ticks.max(1) {+ let source = CGEventSource::new(CGEventSourceStateID::HIDSystemState)+ .map_err(|_| anyhow::anyhow!("CGEventSource::new failed"))?;+ let wheel_y = (delta_y_per_tick / 120).clamp(-10, 10);+ let wheel_x = (delta_x_per_tick / 120).clamp(-10, 10);+ let event =+ CGEvent::new_scroll_event(source, ScrollEventUnit::LINE, 2, wheel_y, wheel_x, 0)+ .map_err(|_| anyhow::anyhow!("CGEvent::new_scroll_event failed"))?;+ unsafe { CGEventSetLocation(event.as_ptr() as *mut std::ffi::c_void, x, y) };+ event.post(CGEventTapLocation::HID);+ std::thread::sleep(std::time::Duration::from_millis(30));+ }+ Ok(())+}+/// Click one exact desktop point, then restore the user's cursor position.////// This is intentionally narrower than the public desktop click path. It isdiff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rsindex c07221adaf..b7ba4d1859 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs@@ -52,7 +52,7 @@ fn def() -> &'static ToolDef {.into(),input_schema: serde_json::json!({"type": "object",- "required": ["pid", "from_x", "from_y", "to_x", "to_y"],+ "required": ["from_x", "from_y", "to_x", "to_y"],"properties": {"session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." },"pid": { "type": "integer", "description": "Target process ID." },@@ -90,6 +90,7 @@ fn def() -> &'static ToolDef {"type": "boolean","description": "When true, coordinates are in the last zoom image for this pid; driver maps back to window coordinates."},+ "scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "Use desktop with no pid/window_id for native get_desktop_state screenshot coordinates." },"delivery_mode": cua_driver_core::tool_schema::delivery_mode_schema()},"additionalProperties": false@@ -109,6 +110,68 @@ impl Tool for DragTool {async fn invoke(&self, args: Value) -> ToolResult {use cua_driver_core::tool_args::ArgsExt;+ if args.opt_str("scope").as_deref() == Some("desktop")+ && args.get("pid").is_none()+ && args.get("window_id").is_none()+ {+ let coordinate = |key: &str| {+ args.opt_f64(key)+ .or_else(|| args.opt_i64(key).map(|value| value as f64))+ };+ let Some(from_x) = coordinate("from_x") else {+ return ToolResult::error("Missing required parameter: from_x");+ };+ let Some(from_y) = coordinate("from_y") else {+ return ToolResult::error("Missing required parameter: from_y");+ };+ let Some(to_x) = coordinate("to_x") else {+ return ToolResult::error("Missing required parameter: to_x");+ };+ let Some(to_y) = coordinate("to_y") else {+ return ToolResult::error("Missing required parameter: to_y");+ };+ let (from_x, from_y) = super::desktop_screenshot_point(from_x, from_y).await;+ let (to_x, to_y) = super::desktop_screenshot_point(to_x, to_y).await;+ let duration_ms = args.u64_or("duration_ms", 500).min(10_000);+ let steps = args.u64_or("steps", 20).clamp(1, 200) as usize;+ let modifiers: Vec<String> = args.str_array("modifier");+ let button = match args.str_or("button", "left").to_lowercase().as_str() {+ "left" => DragButton::Left,+ "right" => DragButton::Right,+ "middle" => DragButton::Middle,+ other => {+ return ToolResult::error(format!(+ "Unknown button '{other}' — expected left, right, or middle."+ ))+ }+ };+ let result = tokio::task::spawn_blocking(move || {+ let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();+ crate::input::mouse::drag_at_xy_foreground(+ from_x,+ from_y,+ to_x,+ to_y,+ duration_ms,+ steps,+ &modifier_refs,+ button,+ )+ })+ .await;+ return match result {+ Ok(Ok(())) => ToolResult::text(format!(+ "Dragged desktop from ({from_x:.1}, {from_y:.1}) to ({to_x:.1}, {to_y:.1})."+ ))+ .with_structured(serde_json::json!({+ "scope": "desktop",+ "path": "hid",+ "effect": "unverifiable"+ })),+ Ok(Err(error)) => ToolResult::error(format!("desktop drag failed: {error}")),+ Err(error) => ToolResult::error(format!("desktop drag task failed: {error}")),+ };+ }let pid = match args.require_i32("pid") {Ok(v) => v,Err(e) => return e,diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_config.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_config.rsindex 79313eb528..a1b1474952 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_config.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_config.rs@@ -68,7 +68,6 @@ impl Tool for GetConfigTool {// response reflects whatever set_config (or a direct JSON edit)// last wrote.let (pip_enabled, pip_geometry) = pip_preview::read_pip_keys_from_file();- let capture_scope = self.state.config.read().unwrap().capture_scope.clone();ToolResult::text("cua-driver-rs configuration").with_structured(serde_json::json!({"version": env!("CARGO_PKG_VERSION"),// Maintainer E2E builds set this at compile time so the@@ -77,10 +76,9 @@ impl Tool for GetConfigTool {// sharing its package version."source_sha": option_env!("CUA_DRIVER_SOURCE_SHA"),"platform": "macos",- // capture_mode is a per-call param; capture_scope is a global- // setting that gates get_desktop_state.+ // capture_mode is per-call; capture_scope is per-session and is+ // reported by get_session_state rather than persistent config."max_image_dimension": max_image_dimension,- "capture_scope": capture_scope,"agent_cursor": {"enabled": cursor_enabled,},diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_desktop_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_desktop_state.rsindex cf5558e8b7..fa2da56381 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_desktop_state.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_desktop_state.rs@@ -52,25 +52,6 @@ impl Tool for GetDesktopStateTool {async fn invoke(&self, args: Value) -> ToolResult {use cua_driver_core::tool_args::ArgsExt;- // Gate on the global capture_scope (re-read from the persisted config,- // the same value set_config writes): a full-display capture is a- // desktop-scope operation, available only under capture_scope="desktop".- let scope = super::load_driver_config().capture_scope;- if scope != "desktop" {- return ToolResult::error(format!(- "get_desktop_state requires capture_scope=\"desktop\" (current scope is \- \"{scope}\"). Full-display capture is a desktop-scope operation; call \- set_config with capture_scope=desktop first (it also enables window-less \- screen-absolute click/scroll). For a single window, use \- get_window_state(pid, window_id) instead."- ))- .with_structured(serde_json::json!({- "code": "desktop_scope_disabled",- "capture_scope": scope,- "suggestion": "set_config capture_scope=desktop",- }));- }-let screenshot_out_file = args.opt_str("screenshot_out_file").map(|s| {// Expand ~ prefix (mirrors get_window_state).if s.starts_with("~/") {@@ -123,6 +104,7 @@ impl Tool for GetDesktopStateTool {let mut structured = serde_json::json!({"platform": "macos",+ "display": "primary","screenshot_width": screenshot_width,"screenshot_height": screenshot_height,"screen_width": screen_width,diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rsindex c6aed03836..6a4398048d 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs@@ -1,8 +1,11 @@use async_trait::async_trait;-use cua_driver_core::{protocol::ToolResult, tool::{Tool, ToolDef}};+use cua_driver_core::{+ protocol::ToolResult,+ tool::{Tool, ToolDef},+};+use libc;use serde_json::Value;use std::sync::Arc;-use libc;use crate::apps;use crate::focus_guard;@@ -15,7 +18,9 @@ pub struct HotkeyTool {}impl HotkeyTool {- pub fn new(state: Arc<ToolState>) -> Self { Self { state } }+ pub fn new(state: Arc<ToolState>) -> Self {+ Self { state }+ }}static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();@@ -48,7 +53,7 @@ fn def() -> &'static ToolDef {.into(),input_schema: serde_json::json!({"type": "object",- "required": ["pid", "keys"],+ "required": ["keys"],"properties": {"session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." },"pid": { "type": "integer", "description": "Target process ID." },@@ -64,6 +69,7 @@ fn def() -> &'static ToolDef {"type": "integer","description": "Target window. Required for delivery_mode:\"foreground\" (the NSMenu activation needs a window). Does NOT itself raise the window — raising is gated on delivery_mode."},+ "scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "Use desktop with no pid/window_id to send the chord to the frontmost application." },"delivery_mode": cua_driver_core::tool_schema::delivery_mode_schema()},"additionalProperties": false@@ -85,13 +91,51 @@ fn is_modifier(k: &str) -> bool {#[async_trait]impl Tool for HotkeyTool {- fn def(&self) -> &ToolDef { def() }+ fn def(&self) -> &ToolDef {+ def()+ }async fn invoke(&self, args: Value) -> ToolResult {use cua_driver_core::tool_args::ArgsExt;let _ = &self.state;- let pid = match args.require_i32("pid") { Ok(v) => v, Err(e) => return e };+ if args.opt_str("scope").as_deref() == Some("desktop")+ && args.get("pid").is_none()+ && args.get("window_id").is_none()+ {+ let raw_keys = args.str_array("keys");+ let modifiers: Vec<String> = raw_keys+ .iter()+ .filter(|key| is_modifier(key))+ .cloned()+ .collect();+ let Some(key) = raw_keys.iter().rev().find(|key| !is_modifier(key)).cloned() else {+ return ToolResult::error(+ "keys must include at least one non-modifier key for desktop hotkey",+ );+ };+ let display = raw_keys.join("+");+ let result = tokio::task::spawn_blocking(move || {+ let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();+ crate::input::keyboard::press_key_global(&key, &modifier_refs)+ })+ .await;+ return match result {+ Ok(Ok(())) => ToolResult::text(format!("Pressed desktop hotkey {display}."))+ .with_structured(serde_json::json!({+ "scope": "desktop",+ "path": "hid",+ "effect": "unverifiable"+ })),+ Ok(Err(error)) => ToolResult::error(format!("desktop hotkey failed: {error}")),+ Err(error) => ToolResult::error(format!("desktop hotkey task failed: {error}")),+ };+ }++ let pid = match args.require_i32("pid") {+ Ok(v) => v,+ Err(e) => return e,+ };if args.get("keys").and_then(|v| v.as_array()).is_none() {return ToolResult::error("Missing required parameter: keys");@@ -104,18 +148,20 @@ impl Tool for HotkeyTool {// Split: modifiers are all entries that are modifier names; base key is everything else.// Typically: last non-modifier is the key; all others are modifiers.- let modifiers: Vec<String> = raw_keys.iter()+ let modifiers: Vec<String> = raw_keys+ .iter().filter(|k| is_modifier(k)).cloned().collect();- let non_modifiers: Vec<String> = raw_keys.iter()+ let non_modifiers: Vec<String> = raw_keys+ .iter().filter(|k| !is_modifier(k)).cloned().collect();if non_modifiers.is_empty() {return ToolResult::error(- "keys must include at least one non-modifier key (e.g. \"c\" in [\"cmd\", \"c\"])."+ "keys must include at least one non-modifier key (e.g. \"c\" in [\"cmd\", \"c\"]).",);}@@ -137,15 +183,29 @@ impl Tool for HotkeyTool {let px = args.get("x").and_then(|v| v.as_f64());let py = args.get("y").and_then(|v| v.as_f64());if let (Some(cx), Some(cy)) = (px, py) {- let from_zoom = args.get("from_zoom").and_then(|v| v.as_bool()).unwrap_or(false);+ let from_zoom = args+ .get("from_zoom")+ .and_then(|v| v.as_bool())+ .unwrap_or(false);if let Err(e) = super::focus_by_pixel(- &self.state, pid, window_id, cx, cy, fg,- args.opt_str("session"), args.opt_str("_session_id"), from_zoom,- ).await {+ &self.state,+ pid,+ window_id,+ cx,+ cy,+ fg,+ args.opt_str("session"),+ args.opt_str("_session_id"),+ from_zoom,+ )+ .await+ {return e;}true- } else { false }+ } else {+ false+ }};// ── Focus-suppression wrap (Swift WindowChangeDetector + FocusGuard) ──@@ -169,9 +229,11 @@ impl Tool for HotkeyTool {// foreground rung: briefly front the window so NSMenu key// equivalents dispatch, then restore prior frontmost.(true, Some(wid)) => {- crate::input::skylight::with_menu_shortcut_activation(pid as libc::pid_t, wid, || {- crate::input::keyboard::hotkey_no_auth(pid, &key, &m)- })?;+ crate::input::skylight::with_menu_shortcut_activation(+ pid as libc::pid_t,+ wid,+ || crate::input::keyboard::hotkey_no_auth(pid, &key, &m),+ )?;Ok(())}// background (default): auth-envelope post to the pid, no@@ -188,7 +250,11 @@ impl Tool for HotkeyTool {match result {Ok(Ok(())) => {- let label = if fg { " (delivery_mode:foreground)" } else { "" };+ let label = if fg {+ " (delivery_mode:foreground)"+ } else {+ ""+ };// A combo is never read-back-verifiable. On the background rung,// point the agent at the foreground escalation for menu shortcuts// an app drops in the background — same contract as type_text.@@ -209,10 +275,11 @@ impl Tool for HotkeyTool {ToolResult::text(format!("Pressed {key_display} on pid {pid}{label}.{}",changes.result_suffix()- )).with_structured(structured)+ ))+ .with_structured(structured)}Ok(Err(e)) => ToolResult::error(format!("hotkey failed: {e}")),- Err(e) => ToolResult::error(format!("Task error: {e}")),+ Err(e) => ToolResult::error(format!("Task error: {e}")),}}}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rsindex 5c28943d49..3618a9c1ef 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs@@ -180,29 +180,17 @@ impl ResizeRegistry {}/// Runtime-mutable driver configuration persisted across calls within a session.-///-/// `capture_mode` is per-call now (the `capture_mode` param). `capture_scope` is-/// re-introduced here as a GLOBAL setting: it gates `get_desktop_state` (a-/// full-display capture has no pid/window_id and therefore no per-call scope to-/// key on — the only coherent gate is a global one), matching the Windows/Linux-/// `DriverConfig.capture_scope`. Per-call tools (`click`/`scroll`) still accept a-/// per-call `scope`; this global is the baseline the no-args desktop-capture-/// tool reads. Default `"window"`.pub struct DriverConfig {/// Max screenshot dimension (0 = no limit). Applied during screenshot/zoom./// Default 1568 matches Swift's `CuaDriverConfig.defaultMaxImageDimension` —/// the long edge is downscaled to this before encoding.pub max_image_dimension: u32,- /// Capture scope: `"window"` (default) or `"desktop"`. Gates- /// `get_desktop_state` (full-display capture requires `"desktop"`).- pub capture_scope: String,}impl Default for DriverConfig {fn default() -> Self {Self {max_image_dimension: 1568,- capture_scope: "window".to_owned(),}}}@@ -228,22 +216,36 @@ pub fn load_driver_config() -> DriverConfig {Ok(v) => v,Err(_) => return cfg, // malformed file — use defaults};- // `capture_mode` is per-call now; an old on-disk `capture_mode` key is- // silently ignored. `capture_scope` IS a global setting again (gates- // get_desktop_state) — load it, accepting only window|desktop.+ // `capture_mode` is per-call now; old on-disk `capture_mode` and+ // `capture_scope` keys are intentionally inert.if let Some(v) = json.get("max_image_dimension").and_then(|v| v.as_u64()) {if let Ok(v32) = u32::try_from(v) {cfg.max_image_dimension = v32;}}- if let Some(s) = json.get("capture_scope").and_then(|v| v.as_str()) {- if s == "window" || s == "desktop" {- cfg.capture_scope = s.to_owned();- }- }cfg}+/// Convert native pixels from `get_desktop_state` into the logical point space+/// used by CoreGraphics input APIs. Retina scaled modes cannot rely on the+/// nominal backing factor alone, so derive the ratio from the actual PNG.+pub async fn desktop_screenshot_point(x: f64, y: f64) -> (f64, f64) {+ let ratio = tokio::task::spawn_blocking(|| {+ let logical_w = get_screen_size::main_screen_size().map(|(w, _, _)| w as f64);+ let shot_w = crate::capture::screenshot_display_bytes()+ .ok()+ .and_then(|png| crate::capture::png_dimensions(&png).ok())+ .map(|(w, _)| w as f64);+ match (shot_w, logical_w) {+ (Some(sw), Some(lw)) if lw > 0.0 && sw > lw => sw / lw,+ _ => 1.0,+ }+ })+ .await+ .unwrap_or(1.0);+ (x / ratio, y / ratio)+}+/// Persist a single key/value pair to `~/.cua-driver/config.json`./// Merges with any existing file contents so other keys are preserved./// Returns `Err` if the directory cannot be created or the file cannot be written.diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rsindex 3d14684046..dd7809d6ae 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs@@ -1,5 +1,8 @@use async_trait::async_trait;-use cua_driver_core::{protocol::ToolResult, tool::{Tool, ToolDef}};+use cua_driver_core::{+ protocol::ToolResult,+ tool::{Tool, ToolDef},+};use serde_json::Value;use std::sync::Arc;@@ -10,7 +13,9 @@ pub struct MoveCursorTool {}impl MoveCursorTool {- pub fn new(state: Arc<ToolState>) -> Self { Self { state } }+ pub fn new(state: Arc<ToolState>) -> Self {+ Self { state }+ }}static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();@@ -18,9 +23,9 @@ static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();fn def() -> &'static ToolDef {DEF.get_or_init(|| ToolDef {name: "move_cursor".into(),- description: "Move the agent cursor overlay to (x, y). Does NOT move the real mouse \- cursor — the user's cursor stays where it is. Useful for showing the agent's \- attention without interrupting the user.".into(),+ description: "Move a cursor to (x, y). In window scope (default), moves only the \+ agent cursor overlay. With scope=desktop, moves the real OS pointer in native \+ get_desktop_state screenshot coordinates.".into(),input_schema: serde_json::json!({"type": "object","required": ["x", "y"],@@ -28,15 +33,12 @@ fn def() -> &'static ToolDef {"session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." },"x": { "type": "number" },"y": { "type": "number" },+ "scope": { "type": "string", "enum": ["window", "desktop"], "default": "window" },"cursor_id": { "type": "string", "description": "Cursor instance to move. Default: 'default'." }},"additionalProperties": false}),- // read-only: move_cursor only nudges the agent-cursor overlay, never the- // target app — so it's safe to run concurrently. The `readOnlyHint` this- // emits lets MCP clients (e.g. Claude Code's isConcurrencySafe) parallelize- // cursor moves. (Mutating tools like click stay read_only:false on purpose.)- read_only: true,+ read_only: false,destructive: false,idempotent: true,open_world: false,@@ -45,12 +47,41 @@ fn def() -> &'static ToolDef {#[async_trait]impl Tool for MoveCursorTool {- fn def(&self) -> &ToolDef { def() }+ fn def(&self) -> &ToolDef {+ def()+ }async fn invoke(&self, args: Value) -> ToolResult {use cua_driver_core::tool_args::ArgsExt;- let x = match args.require_f64("x") { Ok(v) => v, Err(e) => return e };- let y = match args.require_f64("y") { Ok(v) => v, Err(e) => return e };+ let x = match args.require_f64("x") {+ Ok(v) => v,+ Err(e) => return e,+ };+ let y = match args.require_f64("y") {+ Ok(v) => v,+ Err(e) => return e,+ };+ if args.opt_str("scope").as_deref() == Some("desktop") {+ let (x, y) = super::desktop_screenshot_point(x, y).await;+ let result =+ tokio::task::spawn_blocking(move || crate::input::mouse::move_cursor_desktop(x, y))+ .await;+ return match result {+ Ok(Ok(())) => ToolResult::text(format!(+ "Moved the real desktop pointer to ({x:.1}, {y:.1})."+ ))+ .with_structured(serde_json::json!({+ "scope": "desktop",+ "x": x,+ "y": y,+ "effect": "unverifiable"+ })),+ Ok(Err(error)) => {+ ToolResult::error(format!("desktop pointer move failed: {error}"))+ }+ Err(error) => ToolResult::error(format!("desktop pointer task failed: {error}")),+ };+ }let cursor_id = super::cursor_tools::resolve_cursor_key(&args);self.state.cursor_registry.update_position(&cursor_id, x, y);@@ -62,6 +93,8 @@ impl Tool for MoveCursorTool {// then glides in, identical to `click`. No-op for an empty (anonymous)// key or when the overlay is disabled for this cursor.crate::cursor::overlay::animate_cursor_to(cursor_id.clone(), x, y).await;- ToolResult::text(format!("Agent cursor '{cursor_id}' moved to ({x:.1}, {y:.1})."))+ ToolResult::text(format!(+ "Agent cursor '{cursor_id}' moved to ({x:.1}, {y:.1})."+ ))}}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rsindex ebcf1ecd21..59b697de6a 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs@@ -1,8 +1,11 @@use async_trait::async_trait;-use cua_driver_core::{protocol::ToolResult, tool::{Tool, ToolDef}};+use cua_driver_core::{+ protocol::ToolResult,+ tool::{Tool, ToolDef},+};+use libc;use serde_json::Value;use std::sync::Arc;-use libc;use crate::apps;use crate::focus_guard;@@ -15,7 +18,9 @@ pub struct PressKeyTool {}impl PressKeyTool {- pub fn new(state: Arc<ToolState>) -> Self { Self { state } }+ pub fn new(state: Arc<ToolState>) -> Self {+ Self { state }+ }}static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();@@ -39,7 +44,7 @@ fn def() -> &'static ToolDef {Modifiers array: cmd, shift, option/alt, ctrl, fn.".into(),input_schema: serde_json::json!({"type": "object",- "required": ["pid", "key"],+ "required": ["key"],"properties": {"session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." },"pid": { "type": "integer" },@@ -54,6 +59,7 @@ fn def() -> &'static ToolDef {"element_token": { "type": "string", "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit \"stale\" error if the snapshot has been superseded." },"x": { "type": "number", "description": "Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_index." },"y": { "type": "number", "description": "Screenshot-pixel Y (see x)." },+ "scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "Use desktop with no pid/window_id to send the key to the frontmost application." },"delivery_mode": cua_driver_core::tool_schema::delivery_mode_schema()},"additionalProperties": false@@ -67,16 +73,50 @@ fn def() -> &'static ToolDef {#[async_trait]impl Tool for PressKeyTool {- fn def(&self) -> &ToolDef { def() }+ fn def(&self) -> &ToolDef {+ def()+ }async fn invoke(&self, args: Value) -> ToolResult {use cua_driver_core::tool_args::ArgsExt;- let pid = match args.require_i32("pid") { Ok(v) => v, Err(e) => return e };- let key_raw = match args.require_str("key") { Ok(v) => v, Err(e) => return e };+ if args.opt_str("scope").as_deref() == Some("desktop")+ && args.get("pid").is_none()+ && args.get("window_id").is_none()+ {+ let key = match args.require_str("key") {+ Ok(value) => value,+ Err(error) => return error,+ };+ let modifiers: Vec<String> = args.str_array("modifiers");+ let key_for_input = key.clone();+ let result = tokio::task::spawn_blocking(move || {+ let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();+ crate::input::keyboard::press_key_global(&key_for_input, &modifier_refs)+ })+ .await;+ return match result {+ Ok(Ok(())) => ToolResult::text(format!("Pressed '{key}' on the desktop."))+ .with_structured(serde_json::json!({+ "scope": "desktop",+ "path": "hid",+ "effect": "unverifiable"+ })),+ Ok(Err(error)) => ToolResult::error(format!("desktop press_key failed: {error}")),+ Err(error) => ToolResult::error(format!("desktop press_key task failed: {error}")),+ };+ }+ let pid = match args.require_i32("pid") {+ Ok(v) => v,+ Err(e) => return e,+ };+ let key_raw = match args.require_str("key") {+ Ok(v) => v,+ Err(e) => return e,+ };let mut modifiers: Vec<String> = args.str_array("modifiers");// Surface 6: element_token / element_index precedence resolution.let element_token_arg = args.opt_str("element_token");- let window_id_arg = args.opt_u64("window_id").map(|v| v as u32);+ let window_id_arg = args.opt_u64("window_id").map(|v| v as u32);let element_index_arg = args.opt_u64("element_index").map(|v| v as usize);let resolved = match cua_driver_core::element_token::resolve_element_args(pid,@@ -91,7 +131,9 @@ impl Tool for PressKeyTool {let (element_index, window_id) = match resolved {cua_driver_core::element_token::ResolvedElement::None => (None, window_id_arg),cua_driver_core::element_token::ResolvedElement::Element {- window_id: wid, element_index: idx, via_token: _,+ window_id: wid,+ element_index: idx,+ via_token: _,} => (Some(idx), wid),};@@ -120,18 +162,32 @@ impl Tool for PressKeyTool {if let (Some(cx), Some(cy)) = (px, py) {if element_index.is_some() {return ToolResult::error(- "Pass either element_index (ax) or x,y (px) to press_key, not both."+ "Pass either element_index (ax) or x,y (px) to press_key, not both.",);}- let from_zoom = args.get("from_zoom").and_then(|v| v.as_bool()).unwrap_or(false);+ let from_zoom = args+ .get("from_zoom")+ .and_then(|v| v.as_bool())+ .unwrap_or(false);if let Err(e) = super::focus_by_pixel(- &self.state, pid, window_id, cx, cy, fg,- args.opt_str("session"), args.opt_str("_session_id"), from_zoom,- ).await {+ &self.state,+ pid,+ window_id,+ cx,+ cy,+ fg,+ args.opt_str("session"),+ args.opt_str("_session_id"),+ from_zoom,+ )+ .await+ {return e;}true- } else { false }+ } else {+ false+ }};// Resolve the pre-focus element pointer (if requested) outside@@ -168,7 +224,8 @@ impl Tool for PressKeyTool {if let Some(element_ptr) = pre_focus_ptr {let _ = tokio::task::spawn_blocking(move || {crate::input::ax_actions::focus_element(element_ptr)- }).await;+ })+ .await;tokio::time::sleep(std::time::Duration::from_millis(30)).await;}@@ -180,9 +237,11 @@ impl Tool for PressKeyTool {if fg && !px_focus {if let Some(wid) = window_id {if element_index.is_none() {- crate::input::skylight::with_menu_shortcut_activation(pid as libc::pid_t, wid, || {- crate::input::keyboard::press_key_no_auth(pid, &key, &m)- })?;+ crate::input::skylight::with_menu_shortcut_activation(+ pid as libc::pid_t,+ wid,+ || crate::input::keyboard::press_key_no_auth(pid, &key, &m),+ )?;return Ok(());}}@@ -199,7 +258,11 @@ impl Tool for PressKeyTool {match result {Ok(Ok(())) => {- let label = if fg { " (delivery_mode:foreground)" } else { "" };+ let label = if fg {+ " (delivery_mode:foreground)"+ } else {+ ""+ };let mut structured = serde_json::json!({"path": if fg { "key_events_fg" } else { "key_events" },"verified": false,@@ -216,7 +279,8 @@ impl Tool for PressKeyTool {ToolResult::text(format!("✅ Pressed {display_key} on pid {pid}{label}.{}",changes.result_suffix()- )).with_structured(structured)+ ))+ .with_structured(structured)}Ok(Err(e)) => ToolResult::error(format!("press_key failed: {e}")),Err(e) => ToolResult::error(format!("Task error: {e}")),diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rsindex ad69f68a41..beb7d0cb8a 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs@@ -92,6 +92,7 @@ fn def() -> &'static ToolDef {"element_token": { "type": "string", "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Takes precedence over element_index when both supplied. Returns an explicit \"stale\" error if the snapshot has been superseded. Routes through the pixel-wheel path at the element's center." },"x": { "type": "number", "description": "Window-local screenshot X (top-left origin of the PNG from get_window_state). With `y`, routes through the pixel-wheel path at this point — use for a scrollable surface that isn't in the AX tree. Requires window_id to anchor the window→screen conversion." },"y": { "type": "number", "description": "Window-local screenshot Y. See `x`." },+ "scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "Use desktop with x,y and no pid/window_id for native get_desktop_state screenshot coordinates." },"delivery_mode": cua_driver_core::tool_schema::delivery_mode_schema()},"additionalProperties": false@@ -111,6 +112,58 @@ impl Tool for ScrollTool {async fn invoke(&self, args: Value) -> ToolResult {use cua_driver_core::tool_args::ArgsExt;+ if args.opt_str("scope").as_deref() == Some("desktop")+ && args.get("pid").is_none()+ && args.get("window_id").is_none()+ {+ let direction = match args.require_str("direction") {+ Ok(value) => value,+ Err(error) => return error,+ };+ let x = match args.require_f64("x") {+ Ok(value) => value,+ Err(error) => return error,+ };+ let y = match args.require_f64("y") {+ Ok(value) => value,+ Err(error) => return error,+ };+ let by = args.str_or("by", "line");+ let amount = args.u64_or("amount", 3).clamp(1, 50) as usize;+ let step = if by == "page" {+ WHEEL_STEP_PAGE_PX+ } else {+ WHEEL_STEP_LINE_PX+ };+ let (delta_y, delta_x) = match direction.as_str() {+ "down" => (-step, 0),+ "up" => (step, 0),+ "right" => (0, -step),+ "left" => (0, step),+ other => {+ return ToolResult::error(format!(+ "unknown scroll direction '{other}'; expected up, down, left, or right"+ ))+ }+ };+ let (x, y) = super::desktop_screenshot_point(x, y).await;+ let result = tokio::task::spawn_blocking(move || {+ crate::input::mouse::scroll_wheel_desktop(x, y, delta_y, delta_x, amount)+ })+ .await;+ return match result {+ Ok(Ok(())) => ToolResult::text(format!(+ "Scrolled desktop {direction} by {by} × {amount} at ({x:.1}, {y:.1})."+ ))+ .with_structured(serde_json::json!({+ "scope": "desktop",+ "path": "hid",+ "effect": "unverifiable"+ })),+ Ok(Err(error)) => ToolResult::error(format!("desktop scroll failed: {error}")),+ Err(error) => ToolResult::error(format!("desktop scroll task failed: {error}")),+ };+ }let pid = match args.require_i32("pid") {Ok(v) => v,Err(e) => return e,diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_config.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_config.rsindex 52771d1b5d..22aa0138fb 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_config.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_config.rs@@ -19,13 +19,12 @@ fn def() -> &'static ToolDef {DEF.get_or_init(|| ToolDef {name: "set_config".into(),description: "Update cua-driver-rs configuration. Changes to \- max_image_dimension and capture_scope take effect immediately. The \+ max_image_dimension take effect immediately. The \experimental_pip keys are persisted to ~/.cua-driver/config.json and \take effect on the next daemon restart (the PiP backend is \initialised once at startup).\n\nNote: capture_mode is a per-call \- param (on get_window_state / click), not a stored setting. \- capture_scope IS a global setting: it gates get_desktop_state \- (full-display capture requires capture_scope=desktop).".into(),+ param (on get_window_state / click), not a stored setting. Capture \+ scope is selected by start_session, not set_config.".into(),input_schema: serde_json::json!({"type": "object","properties": {@@ -42,13 +41,6 @@ fn def() -> &'static ToolDef {"type": "integer","description": "Max dimension for screenshot resizing (0 = no limit)."},- "capture_scope": {- "type": "string",- "enum": ["window", "desktop"],- "description": "Capture scope: \"window\" (default) or \"desktop\". Desktop \- scope enables get_desktop_state (full-display capture) and window-less \- screen-absolute click/scroll. Global setting; takes effect immediately."- },"experimental_pip": {"type": "boolean","description": "Enable the experimental picture-in-picture preview window. \@@ -75,6 +67,18 @@ impl Tool for SetConfigTool {async fn invoke(&self, args: Value) -> ToolResult {use cua_driver_core::tool_args::ArgsExt;+ if args.get("capture_scope").is_some()+ || args.get("key").and_then(Value::as_str) == Some("capture_scope")+ {+ return ToolResult::error(+ "config key 'capture_scope' is retired; pass capture_scope=auto|window|desktop to start_session",+ )+ .with_structured(serde_json::json!({+ "code": "config_key_retired",+ "key": "capture_scope",+ "replacement": "start_session.capture_scope",+ }));+ }// The daemon injects `_session_id` for non-anonymous MCP sessions.// Absent => anonymous/global session (CLI one-shot, legacy proxy) =>// today's behavior: write the shared global DriverConfig + persist to@@ -83,10 +87,7 @@ impl Tool for SetConfigTool {// sessions don't clobber each other or the persisted default.let session_id = args.opt_str("_session_id");- // Accept BOTH shapes, matching Windows/Linux + the CLI `config set`:- // - direct fields: {"capture_scope":"desktop"}- // - {key, value}: {"key":"capture_scope","value":"desktop"}- // A direct field wins if both are somehow present.+ // Accept BOTH the direct field and {key,value} shapes.let kv: Option<(String, Value)> = args.opt_str("key").and_then(|k| args.get("value").map(|v| (k, v.clone())));@@ -144,46 +145,19 @@ impl Tool for SetConfigTool {pip_note = format!(" — restart cua-driver for experimental_pip_geometry={geom} to take effect");}}- // capture_scope: GLOBAL setting (gates get_desktop_state). Accept the- // direct field or {key,value} shape; validate window|desktop; write the- // shared global config + persist (matches Windows/Linux — not- // session-scoped, since it's the baseline a no-args capture tool reads).- let scope_arg = args.opt_str("capture_scope").or_else(|| {- kv.as_ref()- .filter(|(k, _)| k == "capture_scope")- .and_then(|(_, v)| v.as_str().map(str::to_owned))- });- let mut capture_scope_note = String::new();- if let Some(sc) = scope_arg {- if sc != "window" && sc != "desktop" {- return ToolResult::error(format!(- "`capture_scope` must be \"window\" or \"desktop\", got \"{sc}\"."- ));- }- self.state.config.write().unwrap().capture_scope = sc.clone();- if let Err(e) = write_driver_config_key("capture_scope", &Value::String(sc.clone())) {- tracing::warn!("set_config: failed to persist capture_scope: {e}");- }- capture_scope_note = format!(", capture_scope={sc}");- }-let scope_note = if session_id.is_some() {" (session-scoped; persisted default unchanged)"} else {""};- // Echo the config back in structured content (matches Windows/Linux- // set_config, which callers/tests read for the applied capture_scope).- let capture_scope = self.state.config.read().unwrap().capture_scope.clone();ToolResult::text(format!(- "Config updated: max_image_dimension={}{}{}{}",- effective_dim, capture_scope_note, scope_note, pip_note+ "Config updated: max_image_dimension={}{}{}",+ effective_dim, scope_note, pip_note)).with_structured(serde_json::json!({"version": env!("CARGO_PKG_VERSION"),"platform": "macos","max_image_dimension": effective_dim,- "capture_scope": capture_scope,}))}}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rsindex 49def31da7..aa0fb4cca5 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs@@ -22,15 +22,17 @@//! (e.g., to trigger live-search debounce handlers).use async_trait::async_trait;-use cua_driver_core::{protocol::ToolResult, tool::{Tool, ToolDef}};+use cua_driver_core::{+ protocol::ToolResult,+ tool::{Tool, ToolDef},+};use serde_json::Value;use std::sync::Arc;+use crate::apps;use crate::ax::bindings::{- copy_string_attr, focused_element_of_pid, kAXErrorSuccess, set_string_attr,- AXUIElementRef,+ copy_string_attr, focused_element_of_pid, kAXErrorSuccess, set_string_attr, AXUIElementRef,};-use crate::apps;use crate::focus_guard;use crate::window_change_detector::WindowChangeDetector;use core_foundation::base::CFRelease;@@ -42,7 +44,9 @@ pub struct TypeTextTool {}impl TypeTextTool {- pub fn new(state: Arc<ToolState>) -> Self { Self { state } }+ pub fn new(state: Arc<ToolState>) -> Self {+ Self { state }+ }}static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();@@ -78,7 +82,7 @@ fn def() -> &'static ToolDef {.into(),input_schema: serde_json::json!({"type": "object",- "required": ["pid", "text"],+ "required": ["text"],"properties": {"session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." },"pid": { "type": "integer", "description": "Target process ID." },@@ -103,6 +107,7 @@ fn def() -> &'static ToolDef {"maximum": 200,"description": "Milliseconds between characters in the CGEvent fallback path. Default 30. Ignored when the AX path succeeds."},+ "scope": { "type": "string", "enum": ["window", "desktop"], "default": "window", "description": "Use desktop with no pid/window_id to type into the frontmost application." },"delivery_mode": {"type": "string","enum": ["background", "foreground"],@@ -120,19 +125,56 @@ fn def() -> &'static ToolDef {#[async_trait]impl Tool for TypeTextTool {- fn def(&self) -> &ToolDef { def() }+ fn def(&self) -> &ToolDef {+ def()+ }async fn invoke(&self, args: Value) -> ToolResult {use cua_driver_core::tool_args::ArgsExt;- let pid = match args.require_i32("pid") { Ok(v) => v, Err(e) => return e };- let text_raw = match args.require_str("text") { Ok(v) => v, Err(e) => return e };+ if args.opt_str("scope").as_deref() == Some("desktop")+ && args.get("pid").is_none()+ && args.get("window_id").is_none()+ {+ let text = match args.require_str("text") {+ Ok(value) => {+ cua_driver_core::text_sanitize::strip_trailing_agent_protocol_tags(&value)+ .into_owned()+ }+ Err(error) => return error,+ };+ let delay_ms = args.u64_or("delay_ms", 30).min(200);+ let result = tokio::task::spawn_blocking(move || {+ crate::input::keyboard::type_text_global(&text, delay_ms)+ })+ .await;+ return match result {+ Ok(Ok(())) => {+ ToolResult::text("Typed text into the frontmost desktop application.")+ .with_structured(serde_json::json!({+ "scope": "desktop",+ "path": "hid",+ "effect": "unverifiable"+ }))+ }+ Ok(Err(error)) => ToolResult::error(format!("desktop type_text failed: {error}")),+ Err(error) => ToolResult::error(format!("desktop type_text task failed: {error}")),+ };+ }+ let pid = match args.require_i32("pid") {+ Ok(v) => v,+ Err(e) => return e,+ };+ let text_raw = match args.require_str("text") {+ Ok(v) => v,+ Err(e) => return e,+ };// Strip trailing agent-protocol closing tags — see// cua_driver_core::text_sanitize docs for rationale.let text = cua_driver_core::text_sanitize::strip_trailing_agent_protocol_tags(&text_raw).into_owned();// Surface 6: element_token / element_index precedence resolution.let element_token_arg = args.opt_str("element_token");- let window_id_arg = args.opt_u64("window_id").map(|v| v as u32);+ let window_id_arg = args.opt_u64("window_id").map(|v| v as u32);let element_index_arg = args.opt_u64("element_index").map(|v| v as usize);let resolved = match cua_driver_core::element_token::resolve_element_args(pid,@@ -147,10 +189,12 @@ impl Tool for TypeTextTool {let (element_index, window_id) = match resolved {cua_driver_core::element_token::ResolvedElement::None => (None, window_id_arg),cua_driver_core::element_token::ResolvedElement::Element {- window_id: wid, element_index: idx, via_token: _,+ window_id: wid,+ element_index: idx,+ via_token: _,} => (Some(idx), wid),};- let delay_ms = args.u64_or("delay_ms", 30);+ let delay_ms = args.u64_or("delay_ms", 30);let delivery_mode = super::DeliveryMode::parse(args.opt_str("delivery_mode").as_deref());// ── px form: focus by pixel-click, then type into the focused element ──@@ -165,14 +209,26 @@ impl Tool for TypeTextTool {if let (Some(cx), Some(cy)) = (px, py) {if element_index.is_some() {return ToolResult::error(- "Pass either element_index (ax) or x,y (px) to type_text, not both."+ "Pass either element_index (ax) or x,y (px) to type_text, not both.",);}- let from_zoom = args.get("from_zoom").and_then(|v| v.as_bool()).unwrap_or(false);+ let from_zoom = args+ .get("from_zoom")+ .and_then(|v| v.as_bool())+ .unwrap_or(false);if let Err(e) = super::focus_by_pixel(- &self.state, pid, window_id, cx, cy, delivery_mode.is_foreground(),- args.opt_str("session"), args.opt_str("_session_id"), from_zoom,- ).await {+ &self.state,+ pid,+ window_id,+ cx,+ cy,+ delivery_mode.is_foreground(),+ args.opt_str("session"),+ args.opt_str("_session_id"),+ from_zoom,+ )+ .await+ {return e;}// element_index stays None → the type path below writes to the now-@@ -182,9 +238,7 @@ impl Tool for TypeTextTool {// Validate element_index requires window_id (still applies for// the legacy integer path; token path already resolved window_id).if element_index.is_some() && window_id.is_none() {- return ToolResult::error(- "window_id is required when element_index is used."- );+ return ToolResult::error("window_id is required when element_index is used.");}// Resolve the element pointer (if element_index given). Retain it out@@ -194,17 +248,21 @@ impl Tool for TypeTextTool {let element_guard = if let (Some(idx), Some(wid)) = (element_index, window_id) {match self.state.element_cache.get_element_retained(pid, wid, idx) {Some(e) => Some((e, idx)),- None => return ToolResult::error(format!(- "Element index {idx} not found. Call get_window_state first."- )),+ None => {+ return ToolResult::error(format!(+ "Element index {idx} not found. Call get_window_state first."+ ))+ }}} else {None};- let element_ptr = element_guard.as_ref().map(|(g, idx)| (g.as_ptr(), Some(*idx)));+ let element_ptr = element_guard+ .as_ref()+ .map(|(g, idx)| (g.as_ptr(), Some(*idx)));- let text_clone = text.clone();- let char_count = text.chars().count();+ let text_clone = text.clone();+ let char_count = text.chars().count();// ── Focus-suppression wrap (Swift WindowChangeDetector + FocusGuard) ──// Typing into a field can trigger autocomplete popovers or@@ -256,9 +314,8 @@ impl Tool for TypeTextTool {// browser's OWN native chrome (address bar, toolbar) stays trusted.// Probe ONLY when it would otherwise confirm via the AX path, so// native types pay nothing.- let ax_echo_surface = verified- && path == PATH_AX- && target_in_web_area(pid, element_ptr);+ let ax_echo_surface =+ verified && path == PATH_AX && target_in_web_area(pid, element_ptr);let verified = verified && !ax_echo_surface;// `verified:false` means the driver could not confirm the text@@ -269,20 +326,28 @@ impl Tool for TypeTextTool {let (mark, note) = if verified {("✅ Inserted", String::new())} else if ax_echo_surface {- ("📨 Sent (unverified)",- " — web-content surface (Chromium / WebKit / Electron): the AX \+ (+ "📨 Sent (unverified)",+ " — web-content surface (Chromium / WebKit / Electron): the AX \layer accepts and echoes the write but the renderer/DOM may not \have observed it, so the driver cannot confirm via AX. Verify \via the screenshot. For a browser tab use the `page` tool (it \drives the DOM); for an embedded web view, re-type with the px \- form (x,y).".to_string())+ form (x,y)."+ .to_string(),+ )} else if path == PATH_KEY_EVENTS_FG {- ("📨 Sent (unverified)",- " — driver could not confirm; verify via screenshot.".to_string())+ (+ "📨 Sent (unverified)",+ " — driver could not confirm; verify via screenshot.".to_string(),+ )} else {- ("📨 Sent (unverified)",- " — driver could not confirm the text landed; verify via screenshot, \- and re-call with delivery_mode:\"foreground\" if it didn't.".to_string())+ (+ "📨 Sent (unverified)",+ " — driver could not confirm the text landed; verify via screenshot, \+ and re-call with delivery_mode:\"foreground\" if it didn't."+ .to_string(),+ )};ToolResult::text(format!("{mark} {char_count} char(s){detail}.{note}{}",@@ -306,15 +371,18 @@ impl Tool for TypeTextTool {// renderer/DOM-focus problem, never a foreground one.let (recommended, reason) =if crate::browser::electron_js::ElectronJs::is_electron(pid) {- ("px",- "Electron web view — the AX write was echoed but the \+ (+ "px",+ "Electron web view — the AX write was echoed but the \renderer may not have observed it. Confirm via the \screenshot; if it didn't land, re-type with the \element px action (x,y to pixel-focus the field, then \- type).")+ type).",+ )} else {- ("page",- "Browser web content — the AX write was echoed but the \+ (+ "page",+ "Browser web content — the AX write was echoed but the \DOM may not have observed it (and AX type_text on a \contenteditable is racy). Drive the tab's DOM with the \`page` tool: execute_javascript + el.value/innerText for a \@@ -323,7 +391,8 @@ impl Tool for TypeTextTool {a one-shot DOM write on their next render) try insert_text \first (one CDP call, cheap), then type_keystrokes if that \also gets discarded (real per-character keyboard events, \- slower but most durable). Or confirm via the screenshot.")+ slower but most durable). Or confirm via the screenshot.",+ )};s["escalation"] = serde_json::json!({"recommended": recommended,@@ -341,7 +410,7 @@ impl Tool for TypeTextTool {})}Ok(Err(e)) => ToolResult::error(format!("type_text failed: {e}")),- Err(e) => ToolResult::error(format!("Task error: {e}")),+ Err(e) => ToolResult::error(format!("Task error: {e}")),}}}@@ -367,10 +436,11 @@ const PATH_KEY_EVENTS_FG: &str = "key_events_fg";/// substring/length test even though something landed — we report `false`/// (unverified) rather than erroring, so the agent can still confirm.fn verify_typed(before: Option<&str>, after: Option<&str>, text: &str) -> bool {- if text.is_empty() { return true; }+ if text.is_empty() {+ return true;+ }let Some(after) = after else { return false };- after.contains(text)- || before.map_or(false, |b| after.chars().count() > b.chars().count())+ after.contains(text) || before.map_or(false, |b| after.chars().count() > b.chars().count())}/// Read the focused/target field's `AXValue`, for before/after read-back.@@ -381,7 +451,9 @@ fn read_axvalue(pid: i32, element_ptr_and_idx: Option<(usize, Option<usize>)>) -unsafe { copy_string_attr(ptr as AXUIElementRef, "AXValue") }} else if let Some(el) = unsafe { focused_element_of_pid(pid) } {let v = unsafe { copy_string_attr(el, "AXValue") };- unsafe { CFRelease(el as _); }+ unsafe {+ CFRelease(el as _);+ }v} else {None@@ -416,16 +488,20 @@ fn target_in_web_area(pid: i32, element_ptr_and_idx: Option<(usize, Option<usizelet mut found = false;for _ in 0..40 {match copy_string_attr(cur, "AXRole").as_deref() {- Some("AXWebArea") => { found = true; break; }+ Some("AXWebArea") => {+ found = true;+ break;+ }// No web area lives above the window/app root — stop.Some("AXWindow") | Some("AXApplication") | None => break,_ => {}}let mut parent: CFTypeRef = std::ptr::null_mut();- let err = AXUIElementCopyAttributeValue(- cur, parent_attr.as_concrete_TypeRef(), &mut parent,- );- if cur_owned { CFRelease(cur as CFTypeRef); }+ let err =+ AXUIElementCopyAttributeValue(cur, parent_attr.as_concrete_TypeRef(), &mut parent);+ if cur_owned {+ CFRelease(cur as CFTypeRef);+ }if err != kAXErrorSuccess || parent.is_null() {cur = std::ptr::null_mut();cur_owned = false;@@ -528,10 +604,17 @@ fn type_text_blocking(// the remote host over hundreds of ms, so at 60ms every keystroke was// dropped. 200ms covers that re-grab without being perceptible.const FOREGROUND_SETTLE_MS: u64 = 200;- let do_type = || cgevent_type_verified(- pid, text, delay_ms, before.as_deref(), clear_first, element_ptr_and_idx,- FOREGROUND_SETTLE_MS,- );+ let do_type = || {+ cgevent_type_verified(+ pid,+ text,+ delay_ms,+ before.as_deref(),+ clear_first,+ element_ptr_and_idx,+ FOREGROUND_SETTLE_MS,+ )+ };let (verified, fronted) = match window_id {Some(wid) => {// Front → type → restore. The closure returns the read-back@@ -557,7 +640,11 @@ fn type_text_blocking(// background keystrokes and `path` must say so honestly.return Ok((format!(" via foreground keystrokes ({delay_ms}ms delay)"),- if fronted { PATH_KEY_EVENTS_FG } else { PATH_KEY_EVENTS },+ if fronted {+ PATH_KEY_EVENTS_FG+ } else {+ PATH_KEY_EVENTS+ },verified,));}@@ -569,7 +656,12 @@ fn type_text_blocking(using CGEvent key-event synthesis");let verified = cgevent_type_verified(- pid, text, delay_ms, before.as_deref(), /*clear_first=*/ false, element_ptr_and_idx,+ pid,+ text,+ delay_ms,+ before.as_deref(),+ /*clear_first=*/ false,+ element_ptr_and_idx,/*settle_ms=*/ 0,)?;return Ok((@@ -585,7 +677,7 @@ fn type_text_blocking(None => unsafe { focused_element_of_pid(pid) }.map(|el| (el, /*owns=*/ true, None)),};if let Some((element, owns, idx_opt)) = ax_target {- let role = unsafe { copy_string_attr(element, "AXRole") }.unwrap_or_default();+ let role = unsafe { copy_string_attr(element, "AXRole") }.unwrap_or_default();let title = unsafe { copy_string_attr(element, "AXTitle") }.unwrap_or_default();let err = unsafe { set_string_attr(element, "AXSelectedText", text) };// Landed iff the API succeeded AND a read-back positively confirms the@@ -595,9 +687,13 @@ fn type_text_blocking(// its old (non-empty) value, which would otherwise report a false// success. `verify_typed` compares the read-back against `before`/`text`.let after = unsafe { copy_string_attr(element, "AXValue") };- let ax_landed = err == kAXErrorSuccess- && verify_typed(before.as_deref(), after.as_deref(), text);- if owns { unsafe { CFRelease(element as _); } }+ let ax_landed =+ err == kAXErrorSuccess && verify_typed(before.as_deref(), after.as_deref(), text);+ if owns {+ unsafe {+ CFRelease(element as _);+ }+ }if ax_landed {let idx_str = idx_opt.map(|i| format!(" [{i}]")).unwrap_or_default();return Ok((format!(" into{idx_str} {role} \"{title}\""), PATH_AX, true));@@ -615,7 +711,12 @@ fn type_text_blocking(// background fallback would change insert-at-cursor semantics. The// foreground rung owns the idempotent clear-then-type.let verified = cgevent_type_verified(- pid, text, delay_ms, before.as_deref(), /*clear_first=*/ false, element_ptr_and_idx,+ pid,+ text,+ delay_ms,+ before.as_deref(),+ /*clear_first=*/ false,+ element_ptr_and_idx,/*settle_ms=*/ 0,)?;Ok((@@ -648,8 +749,13 @@ mod tests {// CGEvent. The fact that this returns an Err (without crashing)// proves we routed through CGEvent-only and never touched AX.let r = type_text_blocking(- -1, "x", None, 0, /*is_terminal_target=*/ true,- super::super::DeliveryMode::Background, None,+ -1,+ "x",+ None,+ 0,+ /*is_terminal_target=*/ true,+ super::super::DeliveryMode::Background,+ None,);// We don't care whether r is Ok or Err — what matters is that// calling it with is_terminal_target=true is safe and never@@ -666,9 +772,9 @@ mod tests {#[test]fn verify_typed_contains_or_grew_is_verified() {- assert!(verify_typed(Some(""), Some("hi"), "hi")); // contains- assert!(verify_typed(Some("ab"), Some("ab hi"), "hi")); // contains, appended- assert!(verify_typed(Some("ab"), Some("abXY"), "??")); // grew vs before+ assert!(verify_typed(Some(""), Some("hi"), "hi")); // contains+ assert!(verify_typed(Some("ab"), Some("ab hi"), "hi")); // contains, appended+ assert!(verify_typed(Some("ab"), Some("abXY"), "??")); // grew vs before}#[test]diff --git a/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs b/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rsindex d370855657..320c528238 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs@@ -834,6 +834,22 @@ pub fn send_wheel_synthesized(sx: i32, sy: i32, ticks: i32, horizontal: bool) ->Ok(())}+/// Return the current foreground window for desktop-scope keyboard and drag+/// operations, which intentionally target whatever the user can currently see.+pub fn foreground_window() -> Result<u64> {+ let hwnd = unsafe { GetForegroundWindow() };+ if hwnd.0.is_null() {+ bail!("no foreground window is available");+ }+ Ok(hwnd.0 as u64)+}++/// Move the real OS pointer to a desktop coordinate.+pub fn move_cursor_desktop(x: i32, y: i32) -> Result<()> {+ unsafe { SetCursorPos(x, y) }+ .map_err(|error| anyhow::anyhow!("SetCursorPos({x}, {y}) failed: {error}"))+}+#[cfg(test)]mod wheel_tests {use super::{posted_press_message, wheel_mouse_data, WHEEL_DELTA};diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rsindex 78551bad0b..2c41e7e684 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs@@ -247,14 +247,13 @@ pub(crate) fn resolve_cursor_key(args: &Value) -> String {/// Returns `true` when a click/scroll invocation should take the **window-less/// screen-absolute** branch: the caller gave numeric `x` AND `y`, gave NO `pid`-/// and NO `window_id`, and the effective `capture_scope` is `"desktop"`.+/// and NO `window_id`, and explicitly selected `scope="desktop"`.////// Pure + arg-shape agnostic (works for both click and scroll args) so it's/// unit-testable without Win32. When this returns `false` but `x,y` are present-/// with no pid/window_id, the caller returns a `desktop_scope_disabled`-/// structured error telling the agent to `set_config capture_scope=desktop`.-fn is_windowless_desktop_action(args: &serde_json::Value, scope: &str) -> bool {- if scope != "desktop" {+/// with no pid/window_id, the caller returns a coordinate-scope error.+fn is_windowless_desktop_action(args: &serde_json::Value) -> bool {+ if args.get("scope").and_then(Value::as_str) != Some("desktop") {return false;}let has_pid = args.get("pid").map(|v| !v.is_null()).unwrap_or(false);@@ -271,10 +270,6 @@ fn is_windowless_desktop_action(args: &serde_json::Value, scope: &str) -> bool {#[derive(Clone)]pub struct DriverConfig {pub capture_mode: String,- /// Capture scope for vision loops: `"window"` (per-window, the default) or- /// `"desktop"` (full-display). Gates the window-less screen-absolute- /// click/scroll branches — those activate only under `"desktop"`.- pub capture_scope: String,pub max_image_dimension: u32,}@@ -282,7 +277,6 @@ impl Default for DriverConfig {fn default() -> Self {Self {capture_mode: "ax".into(),- capture_scope: "window".into(),max_image_dimension: 1568,}}@@ -302,11 +296,6 @@ pub fn load_driver_config() -> DriverConfig {{cfg.capture_mode = v;}- if let Some(v) =- pip_preview::read_config_value("capture_scope").and_then(|v| v.as_str().map(str::to_owned))- {- cfg.capture_scope = v;- }if let Some(v) = pip_preview::read_config_value("max_image_dimension").and_then(|v| v.as_u64()){if let Ok(v32) = u32::try_from(v) {@@ -2469,7 +2458,7 @@ impl Tool for ClickTool {a visible on-screen window to anchor the coordinate conversion (errors \with `pid X has no on-screen window` otherwise).\n\n\Exactly one of `element_index` or (`x` AND `y`) must be provided. \- `pid` is required in both modes. `window_id` is required when \+ `pid` is required for window scope and omitted for desktop scope. `window_id` is required when \`element_index` is used (scopes the cache lookup). After a `zoom` call, \pass `from_zoom=true` to auto-translate zoom-image coords back to \full-window space.\n\n\@@ -2480,7 +2469,7 @@ impl Tool for ClickTool {input_schema: json!({"type":"object","properties":{"session": cua_driver_core::tool_schema::session_schema(),- "pid":{"type":"integer","description":"Target process ID. Required UNLESS using window-less desktop-scope clicks: with capture_scope=\"desktop\" and no pid/window_id, x/y are treated as TRUE SCREEN pixels (call get_desktop_state first)."},+ "pid":{"type":"integer","description":"Target process ID for window scope. Omit with scope=desktop for screen-absolute coordinates from get_desktop_state."},"window_id":{"type":"integer","description":"HWND for the window whose get_window_state produced the element_index. Required when element_index is used. Optional when element_token is supplied (the token carries it)."},"element_index": cua_driver_core::tool_schema::element_index_schema(),"element_token": cua_driver_core::tool_schema::element_token_schema(),@@ -2490,6 +2479,7 @@ impl Tool for ClickTool {"count":{"type":"integer","minimum":1,"maximum":3,"description":"Click count — 1 (single), 2 (double), 3 (triple). Default 1."},"modifier": cua_driver_core::tool_schema::modifier_schema(),"from_zoom":{"type":"boolean","description":"When true, x and y are pixel coordinates in the last `zoom` image for this pid. The driver maps them back to window coords."},+ "scope":{"type":"string","enum":["window","desktop"],"default":"window"},"delivery_mode": crate::input::delivery::delivery_mode_schema()},"additionalProperties":false}),@@ -2505,26 +2495,21 @@ impl Tool for ClickTool {// ── Window-less screen-absolute branch (capture_scope="desktop") ──────// When the caller gives x,y with NO pid/window_id, treat x,y as TRUE- // SCREEN pixels. Gate on effective capture_scope: only "desktop" enables- // this; under "window" we return a structured `desktop_scope_disabled`- // error pointing the caller at set_config.+ // SCREEN pixels. The core registry has already enforced the session's+ // effective capture scope; this local check validates the action form.let has_pid = args.get("pid").map(|v| !v.is_null()).unwrap_or(false);let has_window_id = args.get("window_id").map(|v| !v.is_null()).unwrap_or(false);let has_xy = args.get("x").map(|v| v.is_number()).unwrap_or(false)&& args.get("y").map(|v| v.is_number()).unwrap_or(false);if !has_pid && !has_window_id && has_xy {- let scope = self.state.config.read().unwrap().capture_scope.clone();- if !is_windowless_desktop_action(&args, &scope) {+ if !is_windowless_desktop_action(&args) {return ToolResult::error(- "click: x,y given with no pid/window_id, but capture_scope is \- \"window\". Screen-absolute clicks require desktop scope. Call \- set_config with capture_scope=desktop (and use get_desktop_state \- to pick coordinates), or pass a pid/window_id.",+ "click: x,y given with no pid/window_id requires scope=\"desktop\"; \+ use get_desktop_state to pick coordinates, or pass a pid/window_id.",).with_structured(json!({- "code": "desktop_scope_disabled",- "capture_scope": scope,- "suggestion": "set_config capture_scope=desktop",+ "code": "desktop_coordinate_scope_required",+ "suggestion": "pass scope=desktop",}));}// Resolve button (reuse the same validation as the pid path).@@ -3465,8 +3450,9 @@ impl Tool for TypeTextTool {PostMessage path so autocomplete and IME can keep up. Ignored on the \UIA path (SetValue is atomic).".into(),input_schema: json!({- "type":"object","required":["pid","text"],"properties":{+ "type":"object","required":["text"],"properties":{"session": cua_driver_core::tool_schema::session_schema(),+ "scope":{"type":"string","enum":["window","desktop"],"description":"Use desktop with no pid/window_id to type into the current foreground application."},"pid":{"type":"integer","description":"Target process ID."},"text":{"type":"string","description":"Text to insert at the focused element's cursor."},"window_id":{"type":"integer","description":"HWND of the target window. Required when element_index is used. Optional when element_token is supplied (the token carries it)."},@@ -3485,11 +3471,6 @@ impl Tool for TypeTextTool {async fn invoke(&self, args: Value) -> ToolResult {use crate::input::delivery::{DeliveryMode, EventKind};use cua_driver_core::tool_args::ArgsExt;- let raw_pid = match args.require_i64("pid") {- Ok(v) => v,- Err(e) => return e,- };- let pid = raw_pid as u32;let text_raw = match args.require_str("text") {Ok(v) => v,Err(e) => return e,@@ -3502,6 +3483,36 @@ impl Tool for TypeTextTool {// case is allocation-free.let text = cua_driver_core::text_sanitize::strip_trailing_agent_protocol_tags(&text_raw).into_owned();+ if args.get("scope").and_then(Value::as_str) == Some("desktop")+ && args.get("pid").is_none()+ && args.get("window_id").is_none()+ {+ let hwnd = match crate::input::mouse::foreground_window() {+ Ok(hwnd) => hwnd,+ Err(error) => return ToolResult::error(error.to_string()),+ };+ let text_len = text.chars().count();+ return match tokio::task::spawn_blocking(move || {+ crate::input::send_text_synthesized(hwnd, &text)+ })+ .await+ {+ Ok(Ok(())) => ToolResult::text(format!(+ "Typed {text_len} character(s) into the foreground application (scope:desktop)."+ ))+ .with_structured(json!({+ "scope": "desktop", "path": "SendInput", "characters": text_len,+ "effect": "unverifiable"+ })),+ Ok(Err(error)) => ToolResult::error(error.to_string()),+ Err(error) => ToolResult::error(format!("desktop type task failed: {error}")),+ };+ }+ let raw_pid = match args.require_i64("pid") {+ Ok(v) => v,+ Err(e) => return e,+ };+ let pid = raw_pid as u32;// Surface 6: element_token / element_index precedence resolution.let resolved = match cua_driver_core::element_token::resolve_element_args(pid as i32,@@ -4101,8 +4112,9 @@ impl Tool for PressKeyTool {`element_index` focuses the cached UIA element before sending the key; the \top-level window remains backgrounded when delivery_mode is background.".into(),input_schema: json!({- "type":"object","required":["pid","key"],"properties":{+ "type":"object","required":["key"],"properties":{"session": cua_driver_core::tool_schema::session_schema(),+ "scope":{"type":"string","enum":["window","desktop"],"description":"Use desktop with no pid/window_id to send the key to the current foreground application."},"pid":{"type":"integer","description":"Target process ID."},"key":{"type":"string","description":"Key name (return, tab, escape, up, down, left, right, space, delete, home, end, pageup, pagedown, f1-f12, letter, digit)."},"modifiers":{"type":"array","items":{"type":"string"},"description":"Optional modifier names held while the key is pressed (ctrl/shift/alt/win)."},@@ -4121,16 +4133,42 @@ impl Tool for PressKeyTool {async fn invoke(&self, args: Value) -> ToolResult {use crate::input::delivery::{DeliveryMode, EventKind};use cua_driver_core::tool_args::ArgsExt;- let raw_pid = match args.require_i64("pid") {+ let key = match args.require_str("key") {Ok(v) => v,Err(e) => return e,};- let pid = raw_pid as u32;- let key = match args.require_str("key") {+ let mods: Vec<String> = args.str_array("modifiers");+ if args.get("scope").and_then(Value::as_str) == Some("desktop")+ && args.get("pid").is_none()+ && args.get("window_id").is_none()+ {+ let hwnd = match crate::input::mouse::foreground_window() {+ Ok(hwnd) => hwnd,+ Err(error) => return ToolResult::error(error.to_string()),+ };+ let key_display = key.clone();+ return match tokio::task::spawn_blocking(move || {+ let modifiers: Vec<&str> = mods.iter().map(String::as_str).collect();+ crate::input::send_key_synthesized(hwnd, &key, &modifiers)+ })+ .await+ {+ Ok(Ok(())) => ToolResult::text(format!(+ "Sent {key_display} to the foreground application (scope:desktop)."+ ))+ .with_structured(json!({+ "scope": "desktop", "path": "SendInput", "key": key_display,+ "effect": "unverifiable"+ })),+ Ok(Err(error)) => ToolResult::error(error.to_string()),+ Err(error) => ToolResult::error(format!("desktop key task failed: {error}")),+ };+ }+ let raw_pid = match args.require_i64("pid") {Ok(v) => v,Err(e) => return e,};- let mods: Vec<String> = args.str_array("modifiers");+ let pid = raw_pid as u32;// Surface 6: element_token / element_index precedence resolution.let resolved = match cua_driver_core::element_token::resolve_element_args(pid as i32,@@ -4416,8 +4454,9 @@ impl Tool for HotkeyTool {up/down/left/right, space, delete, home, end, pageup, pagedown, f1-f12, \letters, digits). Order: modifiers first, one non-modifier last.".into(),input_schema: json!({- "type":"object","required":["pid","keys"],"properties":{+ "type":"object","required":["keys"],"properties":{"session": cua_driver_core::tool_schema::session_schema(),+ "scope":{"type":"string","enum":["window","desktop"],"description":"Use desktop with no pid/window_id to send the hotkey to the current foreground application."},"pid":{"type":"integer","description":"Target process ID."},"keys":{"type":"array","items":{"type":"string"},"minItems":2,"description":"Modifier(s) and one non-modifier key, e.g. [\"ctrl\", \"c\"]."},@@ -4434,14 +4473,6 @@ impl Tool for HotkeyTool {async fn invoke(&self, args: Value) -> ToolResult {use crate::input::delivery::{DeliveryMode, EventKind};use cua_driver_core::tool_args::ArgsExt;- let hwnd_opt = args.opt_u64("window_id");- let raw_pid = match args.require_i64("pid") {- Ok(v) => v,- Err(e) => return e,- };- let pid = raw_pid as u32;- let delivery = DeliveryMode::from_args(&args);-// Parse keys array (Swift's only path). Legacy key+modifiers shape// still accepted as a Rust-side convenience.let (key, mods, full_keys) = if let Some(arr) = args.get("keys") {@@ -4478,6 +4509,39 @@ impl Tool for HotkeyTool {} else {return ToolResult::error("Missing required array field keys.");};+ if args.get("scope").and_then(Value::as_str) == Some("desktop")+ && args.get("pid").is_none()+ && args.get("window_id").is_none()+ {+ let hwnd = match crate::input::mouse::foreground_window() {+ Ok(hwnd) => hwnd,+ Err(error) => return ToolResult::error(error.to_string()),+ };+ let key_display = full_keys.join("+");+ return match tokio::task::spawn_blocking(move || {+ let modifiers: Vec<&str> = mods.iter().map(String::as_str).collect();+ crate::input::send_key_synthesized(hwnd, &key, &modifiers)+ })+ .await+ {+ Ok(Ok(())) => ToolResult::text(format!(+ "Sent {key_display} to the foreground application (scope:desktop)."+ ))+ .with_structured(json!({+ "scope": "desktop", "path": "SendInput", "keys": full_keys,+ "effect": "unverifiable"+ })),+ Ok(Err(error)) => ToolResult::error(error.to_string()),+ Err(error) => ToolResult::error(format!("desktop hotkey task failed: {error}")),+ };+ }+ let hwnd_opt = args.opt_u64("window_id");+ let raw_pid = match args.require_i64("pid") {+ Ok(v) => v,+ Err(e) => return e,+ };+ let pid = raw_pid as u32;+ let delivery = DeliveryMode::from_args(&args);// Resolve HWND: use window_id if given, else pick first window for pid.let hwnd = match hwnd_opt {@@ -4890,7 +4954,7 @@ impl Tool for ScrollTool {input_schema: json!({"type":"object","required":["direction"],"properties":{"session": cua_driver_core::tool_schema::session_schema(),- "pid":{"type":"integer","description":"Target process ID. Required UNLESS using window-less desktop-scope scroll: with capture_scope=\"desktop\" and no pid/window_id, x/y are TRUE SCREEN pixels and the wheel routes to the window under that point."},+ "pid":{"type":"integer","description":"Target process ID for window scope. Omit with scope=desktop for screen-absolute coordinates from get_desktop_state."},"direction":{"type":"string","enum":["up","down","left","right"]},"by":{"type":"string","enum":["line","page"],"description":"Scroll granularity. Default: line."},"amount":{"type":"integer","minimum":1,"maximum":50,@@ -4900,6 +4964,7 @@ impl Tool for ScrollTool {"window_id":{"type":"integer","description":"HWND of the target window. Required when element_index is used; otherwise auto-resolves the pid's first visible window."},"element_index":{"type":"integer","description":"Optional element_index. Accepted for parity; currently no-op on Windows."},"element_token": cua_driver_core::tool_schema::element_token_schema(),+ "scope":{"type":"string","enum":["window","desktop"],"default":"window"},"delivery_mode": crate::input::delivery::delivery_mode_schema()},"additionalProperties":false}),@@ -4928,18 +4993,14 @@ impl Tool for ScrollTool {let has_xy = args.get("x").map(|v| v.is_number()).unwrap_or(false)&& args.get("y").map(|v| v.is_number()).unwrap_or(false);if !has_pid && !has_window_id && has_xy {- let scope = self.state.config.read().unwrap().capture_scope.clone();- if !is_windowless_desktop_action(&args, &scope) {+ if !is_windowless_desktop_action(&args) {return ToolResult::error(- "scroll: x,y given with no pid/window_id, but capture_scope is \- \"window\". Screen-absolute scroll requires desktop scope. Call \- set_config with capture_scope=desktop (and use get_desktop_state \- to pick coordinates), or pass a pid/window_id.",+ "scroll: x,y given with no pid/window_id requires scope=\"desktop\"; \+ use get_desktop_state to pick coordinates, or pass a pid/window_id.",).with_structured(serde_json::json!({- "code": "desktop_scope_disabled",- "capture_scope": scope,- "suggestion": "set_config capture_scope=desktop",+ "code": "desktop_coordinate_scope_required",+ "suggestion": "pass scope=desktop",}));}let sx = args.f64_or("x", 0.0) as i32;@@ -6115,8 +6176,9 @@ impl Tool for DragTool {description: "Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels. \duration_ms (default 500) is the wall-clock budget; steps (default 20) interpolates intermediate \WM_MOUSEMOVE events along the path. No focus steal. After a zoom call, pass from_zoom=true.".into(),- input_schema: json!({"type":"object","required":["pid","from_x","from_y","to_x","to_y"],"properties":{+ input_schema: json!({"type":"object","required":["from_x","from_y","to_x","to_y"],"properties":{"session": cua_driver_core::tool_schema::session_schema(),+ "scope":{"type":"string","enum":["window","desktop"],"description":"Use desktop with no pid/window_id for screen-absolute coordinates."},"pid":{"type":"integer","description":"Target process ID."},"window_id":{"type":"integer","description":"Target window handle (HWND). Optional — driver picks frontmost window of pid when omitted."},"from_x":{"type":"number","description":"Drag-start X in window-local screenshot pixels."},@@ -6135,6 +6197,57 @@ impl Tool for DragTool {}async fn invoke(&self, args: Value) -> ToolResult {use crate::input::delivery::{DeliveryMode, EventKind};+ use cua_driver_core::tool_args::ArgsExt;+ if args.get("scope").and_then(Value::as_str) == Some("desktop")+ && args.get("pid").is_none()+ && args.get("window_id").is_none()+ {+ let required = |key: &str| {+ args.opt_f64(key)+ .or_else(|| args.opt_i64(key).map(|value| value as f64))+ };+ let (from_x, from_y, to_x, to_y) = match (+ required("from_x"),+ required("from_y"),+ required("to_x"),+ required("to_y"),+ ) {+ (Some(a), Some(b), Some(c), Some(d)) => (a as i32, b as i32, c as i32, d as i32),+ _ => return ToolResult::error("from_x, from_y, to_x, and to_y are required."),+ };+ let duration_ms = args.u64_or("duration_ms", 500);+ let steps = args.u64_or("steps", 20) as usize;+ let button = args.str_or("button", "left");+ let hwnd = match crate::input::mouse::foreground_window() {+ Ok(hwnd) => hwnd,+ Err(error) => return ToolResult::error(error.to_string()),+ };+ return match tokio::task::spawn_blocking(move || {+ crate::input::mouse::send_drag_synthesized(+ hwnd,+ from_x,+ from_y,+ to_x,+ to_y,+ duration_ms,+ steps,+ &button,+ )+ })+ .await+ {+ Ok(Ok(())) => ToolResult::text(format!(+ "Dragged from ({from_x}, {from_y}) to ({to_x}, {to_y}) (scope:desktop)."+ ))+ .with_structured(json!({+ "scope": "desktop", "path": "SendInput",+ "from": {"x": from_x, "y": from_y}, "to": {"x": to_x, "y": to_y},+ "effect": "unverifiable"+ })),+ Ok(Err(error)) => ToolResult::error(error.to_string()),+ Err(error) => ToolResult::error(format!("desktop drag task failed: {error}")),+ };+ }// Swift error wording 1:1.let raw_pid = match args.get("pid").and_then(|v| v.as_i64()) {Some(p) => p,@@ -6143,7 +6256,6 @@ impl Tool for DragTool {let pid = raw_pid as u32;let delivery = DeliveryMode::from_args(&args);- use cua_driver_core::tool_args::ArgsExt;let cursor_key = resolve_cursor_key(&args);// Accepts numeric JSON as either float or integer — coerce both to f64.let coerce = |key: &str| -> Option<f64> {@@ -6491,29 +6603,6 @@ impl Tool for GetDesktopStateTool {use cua_driver_core::protocol::Content;use cua_driver_core::tool_args::ArgsExt;- // Gate on the global capture_scope: a full-display capture is a- // desktop-scope operation, so it's only available when the agent has- // opted into capture_scope="desktop" (same gate as window-less- // screen-absolute click/scroll — see is_windowless_desktop_action).- // Read the persisted value the same way load_config does.- let scope = pip_preview::read_config_value("capture_scope")- .and_then(|v| v.as_str().map(str::to_owned))- .unwrap_or_else(|| "window".to_owned());- if scope != "desktop" {- return ToolResult::error(format!(- "get_desktop_state requires capture_scope=\"desktop\" (current scope is \- \"{scope}\"). Full-display capture is a desktop-scope operation; call \- set_config with capture_scope=desktop first (it also enables window-less \- screen-absolute click/scroll). For a single window, use \- get_window_state(pid, window_id) instead."- ))- .with_structured(serde_json::json!({- "code": "desktop_scope_disabled",- "capture_scope": scope,- "suggestion": "set_config capture_scope=desktop",- }));- }-let screenshot_out_file = args.opt_str("screenshot_out_file");// True screen geometry in physical pixels (same space as the capture).@@ -6555,10 +6644,15 @@ impl Tool for GetDesktopStateTool {let mut structured = json!({"platform": "windows",+ "display": "primary","screenshot_width": screenshot_width,"screenshot_height": screenshot_height,"screen_width": screen_width,"screen_height": screen_height,+ "scale_factor": unsafe {+ let dpi = windows::Win32::UI::HiDpi::GetDpiForSystem();+ if dpi == 0 { 1.0 } else { dpi as f64 / 96.0 }+ },"screenshot_mime_type": "image/png",});if let Some(ref fp) = file_path {@@ -6624,6 +6718,7 @@ impl Tool for MoveCursorTool {.into(),input_schema: json!({"type":"object","required":["x","y"],"properties":{"session": cua_driver_core::tool_schema::session_schema(),+ "scope":{"type":"string","enum":["window","desktop"],"description":"desktop moves the real OS pointer; window moves only the agent overlay."},"x":{"type":"number"},"y":{"type":"number"},"cursor_id":{"type":"string"}},"additionalProperties":false}),read_only: false,@@ -6636,6 +6731,17 @@ impl Tool for MoveCursorTool {use cua_driver_core::tool_args::ArgsExt;let x = args.f64_or("x", 0.0);let y = args.f64_or("y", 0.0);+ if args.get("scope").and_then(Value::as_str) == Some("desktop") {+ return match crate::input::mouse::move_cursor_desktop(x as i32, y as i32) {+ Ok(()) => ToolResult::text(format!(+ "Moved the OS pointer to ({x:.1}, {y:.1}) (scope:desktop)."+ ))+ .with_structured(json!({+ "scope": "desktop", "path": "SetCursorPos", "x": x, "y": y+ })),+ Err(error) => ToolResult::error(error.to_string()),+ };+ }// Cursor key precedence: caller-declared `session` > legacy `cursor_id`// > NO_CURSOR. An anonymous run (no session) has no cursor to move.let cursor_key = resolve_cursor_key(&args);@@ -7322,7 +7428,6 @@ impl Tool for GetConfigTool {"source_sha": option_env!("CUA_DRIVER_SOURCE_SHA"),"platform": "windows","capture_mode": cfg.capture_mode,- "capture_scope": cfg.capture_scope,"max_image_dimension": cfg.max_image_dimension,"agent_cursor": { "enabled": cursor_enabled },"experimental_pip": pip_enabled,@@ -7368,7 +7473,6 @@ impl Tool for SetConfigTool {"key":{"type":"string","description":"Dotted snake_case path to a leaf config field (Swift-compatible shape). Pair with `value`."},"value":{"description":"New value for `key`. JSON type depends on the key."},"capture_mode":{"type":"string","enum":["ax","vision"],"description":"DEPRECATED and ignored — get_window_state always returns both the UIA tree and a screenshot. Still accepted/persisted for back-compat but has no effect. (\"som\"/\"screenshot\" still decode as deprecated aliases.)"},- "capture_scope":{"type":"string","enum":["window","desktop"],"description":"Capture scope: single window (default) or whole desktop. Desktop scope enables window-less screen-absolute click/scroll (no pid/window_id). Accepted in both the {key,value} and legacy per-field shapes."},"max_image_dimension":{"type":"integer","description":"Legacy per-field shape."},"experimental_pip":{"type":"boolean","description":"Legacy per-field shape. Enables PiP preview (applies next restart)."},"experimental_pip_geometry":{"type":"string","description":"Legacy per-field shape. PiP window size + optional position."}@@ -7377,6 +7481,18 @@ impl Tool for SetConfigTool {})}async fn invoke(&self, args: Value) -> ToolResult {+ if args.get("capture_scope").is_some()+ || args.get("key").and_then(Value::as_str) == Some("capture_scope")+ {+ return ToolResult::error(+ "config key 'capture_scope' is retired; pass capture_scope=auto|window|desktop to start_session",+ )+ .with_structured(json!({+ "code": "config_key_retired",+ "key": "capture_scope",+ "replacement": "start_session.capture_scope",+ }));+ }let mut cfg = self.state.config.write().unwrap();let mut applied = false;// Swift-compatible {key, value} shape.@@ -7394,17 +7510,6 @@ impl Tool for SetConfigTool {}None => return ToolResult::error(format!("`capture_mode` must be a string, got {val}.")),},- "capture_scope" => match val.as_str() {- Some(s @ ("window" | "desktop")) => {- cfg.capture_scope = s.to_owned();- if let Err(e) = pip_preview::write_config_key("capture_scope", Value::String(s.to_owned())) {- tracing::warn!("set_config: failed to persist capture_scope: {e}");- }- applied = true;- }- Some(other) => return ToolResult::error(format!("`capture_scope` must be \"window\" or \"desktop\", got \"{other}\".")),- None => return ToolResult::error(format!("`capture_scope` must be a string, got {val}.")),- },"max_image_dimension" => match val.as_u64() {Some(n) => {cfg.max_image_dimension = n as u32;@@ -7439,7 +7544,7 @@ impl Tool for SetConfigTool {None => return ToolResult::error(format!("`experimental_pip_geometry` must be a string, got {val}.")),},other => return ToolResult::error(format!(- "Unknown config key `{other}`. Known: capture_mode, capture_scope, max_image_dimension, experimental_pip, experimental_pip_geometry."+ "Unknown config key `{other}`. Known: capture_mode, max_image_dimension, experimental_pip, experimental_pip_geometry.")),}}@@ -7453,20 +7558,6 @@ impl Tool for SetConfigTool {}applied = true;}- if let Some(scope) = args.get("capture_scope").and_then(|v| v.as_str()) {- if !matches!(scope, "window" | "desktop") {- return ToolResult::error(format!(- "`capture_scope` must be \"window\" or \"desktop\", got \"{scope}\"."- ));- }- cfg.capture_scope = scope.to_owned();- if let Err(e) =- pip_preview::write_config_key("capture_scope", Value::String(scope.to_owned()))- {- tracing::warn!("set_config: failed to persist capture_scope: {e}");- }- applied = true;- }if let Some(dim) = args.get("max_image_dimension").and_then(|v| v.as_u64()) {cfg.max_image_dimension = dim as u32;if let Err(e) = pip_preview::write_config_key("max_image_dimension", Value::from(dim)) {@@ -7521,7 +7612,6 @@ impl Tool for SetConfigTool {"source_sha": option_env!("CUA_DRIVER_SOURCE_SHA"),"platform": "windows","capture_mode": cfg.capture_mode,- "capture_scope": cfg.capture_scope,"max_image_dimension": cfg.max_image_dimension,"agent_cursor": { "enabled": cursor_enabled },"experimental_pip": pip_enabled,@@ -8888,7 +8978,7 @@ mod click_button_schema_tests {#[cfg(test)]mod desktop_scope_tests {- use super::{is_windowless_desktop_action, DriverConfig, GetDesktopStateTool};+ use super::{is_windowless_desktop_action, GetDesktopStateTool};use cua_driver_core::tool::Tool;use serde_json::json;@@ -8898,61 +8988,52 @@ mod desktop_scope_tests {fn windowless_true_for_xy_under_desktop_scope_click_shape() {// Click arg shape: {x, y}.assert!(is_windowless_desktop_action(- &json!({"x": 10, "y": 20}),- "desktop"+ &json!({"x": 10, "y": 20, "scope": "desktop"})));}#[test]fn windowless_true_for_xy_under_desktop_scope_scroll_shape() {// Scroll arg shape: {direction, x, y}.- assert!(is_windowless_desktop_action(- &json!({"direction": "down", "x": 10, "y": 20}),- "desktop"- ));+ assert!(is_windowless_desktop_action(&json!({+ "direction": "down", "x": 10, "y": 20, "scope": "desktop"+ })));}#[test]fn windowless_false_when_pid_present() {- assert!(!is_windowless_desktop_action(- &json!({"x": 10, "y": 20, "pid": 5}),- "desktop"- ));+ assert!(!is_windowless_desktop_action(&json!({+ "x": 10, "y": 20, "pid": 5, "scope": "desktop"+ })));}#[test]fn windowless_false_when_window_id_present() {- assert!(!is_windowless_desktop_action(- &json!({"x": 10, "y": 20, "window_id": 99}),- "desktop"- ));+ assert!(!is_windowless_desktop_action(&json!({+ "x": 10, "y": 20, "window_id": 99, "scope": "desktop"+ })));}#[test]fn windowless_false_under_window_scope() {- assert!(!is_windowless_desktop_action(- &json!({"x": 10, "y": 20}),- "window"- ));+ assert!(!is_windowless_desktop_action(&json!({+ "x": 10, "y": 20, "scope": "window"+ })));}#[test]fn windowless_false_when_xy_missing() {- assert!(!is_windowless_desktop_action(&json!({"x": 10}), "desktop"));- assert!(!is_windowless_desktop_action(&json!({"y": 20}), "desktop"));- assert!(!is_windowless_desktop_action(&json!({}), "desktop"));- // Non-numeric x/y must not qualify.assert!(!is_windowless_desktop_action(- &json!({"x": "10", "y": "20"}),- "desktop"+ &json!({"x": 10, "scope": "desktop"})));- }-- // ── DriverConfig default ──────────────────────────────────────────────────-- #[test]- fn default_capture_scope_is_window() {- assert_eq!(DriverConfig::default().capture_scope, "window");+ assert!(!is_windowless_desktop_action(+ &json!({"y": 20, "scope": "desktop"})+ ));+ assert!(!is_windowless_desktop_action(&json!({"scope": "desktop"})));+ // Non-numeric x/y must not qualify.+ assert!(!is_windowless_desktop_action(&json!({+ "x": "10", "y": "20", "scope": "desktop"+ })));}// ── get_desktop_state schema ──────────────────────────────────────────────diff --git a/libs/cua-driver/tests/runners/macos-lume/README.md b/libs/cua-driver/tests/runners/macos-lume/README.mdindex b4a9f71e74..97acb1999e 100644--- a/libs/cua-driver/tests/runners/macos-lume/README.md+++ b/libs/cua-driver/tests/runners/macos-lume/README.md@@ -175,10 +175,12 @@ osascript -e \~/.local/bin/cua-driver list_apps '{}'# A desktop screenshot triggers Tahoe's direct-capture/private-window prompt.-~/.local/bin/cua-driver call set_config '{"capture_scope":"desktop"}'-~/.local/bin/cua-driver call get_desktop_state '{}' \+~/.local/bin/cua-driver call start_session \+ '{"session":"seed-desktop-consent","capture_scope":"desktop"}'+~/.local/bin/cua-driver call get_desktop_state \+ '{"session":"seed-desktop-consent"}' \> /tmp/cua-driver-seed-desktop-state.json-~/.local/bin/cua-driver call set_config '{"capture_scope":"window"}'+~/.local/bin/cua-driver call end_session '{"session":"seed-desktop-consent"}'jq -e '.screenshot_mime_type == "image/png"and .screenshot_width > 0