Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cua-driver): exact-target macOS background input v1 (#2837)

Hamish Bultitude1b2cb5a706c3
diff --git a/libs/cua-driver/docs/macos-background-input-v1-plan.md b/libs/cua-driver/docs/macos-background-input-v1-plan.md
new file mode 100644
index 0000000000..30ac341015
--- /dev/null
+++ b/libs/cua-driver/docs/macos-background-input-v1-plan.md
@@ -0,0 +1,920 @@
+# Exact macOS Background Input v1
+
+**Status:** Proposed implementation plan
+
+**Base:** `origin/main` at `d21e3447f9b08c761c090946648d5aca5e6c9cf1`
+
+**Date:** 2026-08-04
+
+## Goal
+
+Let an agent operate ordinary macOS applications in the background without
+moving the real pointer, changing the user's active application, or silently
+acting on another window owned by the same process.
+
+V1 is deliberately conservative. It expands the routes for which Cua Driver can
+prove an exact `(pid, CGWindowID)` destination and returns a structured refusal
+for every other route. A refused action is preferable to a process-scoped key
+event that appears successful after mutating a sibling window.
+
+## Success criteria
+
+V1 is complete when all of the following are true:
+
+1. Every window-scoped macOS mutation carries the requested `(pid,
+ CGWindowID)` from target resolution through route selection, dispatch, and
+ post-action verification.
+2. A result can be `confirmed` only from evidence belonging to that exact
+ window or to an exact browser target bound to it. State from another
+ same-process window is never accepted as evidence.
+3. Background routing is ordered as semantic Accessibility, exact browser/CDP,
+ exact window-local pointer, and finally PID keyboard with an independent
+ exact-delivery proof. If none is safe, the driver refuses before sending
+ input.
+4. Background mode never silently crosses to global HID, activates another
+ application, moves the real pointer, changes Space, or restores a minimized
+ window.
+5. Minimized and hidden windows retain safe semantic AX actions and current
+ one-shot capture. Raw key commits are refused; callers use `AXConfirm`, an
+ exact `AXPress`, or an exact browser action when one exists.
+6. An off-Space window that is absent from the process's `AXWindows` is
+ observation-only. Cua Driver does not use another window's AX tree or send
+ process-scoped input while the target is unresolved.
+7. Ordinary Electron applications get per-process AX enablement, exact
+ `BrowserWindow` mapping, bounded fresh-tree acquisition, delayed effect
+ verification, and explicit two-window safety coverage.
+8. Existing action request schemas and the closed `ActionResult` wire shape
+ remain compatible.
+9. Focused unit/contract tests run in ordinary CI. Exact-SHA native evidence is
+ collected separately on a signed macOS Namespace runner and in the
+ canonical Lume acceptance lane.
+
+## Non-goals
+
+V1 does not attempt to provide:
+
+- live minimized-window PiP or a window-scoped ScreenCaptureKit stream;
+- continuously changing Chromium pixels while the application is occluded,
+ hidden, or minimized;
+- full Accessibility parity across Spaces;
+- automatic Space switching, temporary Space membership, sticky windows, or
+ switch-and-restore input;
+- deminiaturize/order-behind/re-minimize transactions;
+- minimize-button interception or application-specific window swizzling;
+- a background raw-key guarantee for multi-window applications;
+- a global-HID fallback in background mode;
+- background control of games, Metal surfaces, protected video, secure input,
+ or other surfaces that do not expose an exact semantic or routed target;
+- generic WKWebView mutation, Safari JavaScript fallback expansion, or a new
+ browser attachment mechanism;
+- a redesign of the public action-result contract; or
+- changes to application-owned Chromium settings such as
+ `backgroundThrottling`.
+
+The existing caller-selected `delivery_mode: "foreground"` remains a separate
+last-resort behavior. This plan does not remove it, but the background router
+must never enter it implicitly.
+
+## Current source-grounded state
+
+### Exact AX scoping exists
+
+[`ax/window_scope.rs`](../rust/crates/platform-macos/src/ax/window_scope.rs)
+already defines `Matched`, `NotFound`, `OwnerPidMismatch`, and `AxUnresolved`.
+Its structural invariant is the right foundation: every non-`Matched` result
+walks no elements.
+
+[`ax/tree.rs`](../rust/crates/platform-macos/src/ax/tree.rs) unions
+`AXChildren` and `AXWindows`, maps top-level AX windows to CG window IDs through
+`_AXUIElementGetWindow`, and bounds the default walk at 2,000 nodes and depth
+25. [`get_window_state.rs`](../rust/crates/platform-macos/src/tools/get_window_state.rs)
+preflights WindowServer ownership, empties the element cache when the requested
+window does not resolve, and still returns an exact-window screenshot for the
+`AxUnresolved` degradation.
+
+These checks prevent an unresolved request from returning a sibling window's
+tree. Mutation needs to carry the same invariant farther than it does today.
+
+### Explicit `window_id` does not make PID keyboard delivery window-scoped
+
+[`window_target.rs`](../rust/crates/cua-driver-core/src/window_target.rs)
+correctly promotes a unique PID-only target and refuses a PID with multiple
+eligible top-level windows. Explicit `window_id` and element-token requests,
+however, pass through that cardinality guard unchanged.
+
+On macOS, [`keyboard.rs`](../rust/crates/platform-macos/src/input/keyboard.rs)
+posts through `SLEventPostToPid` with a public `CGEvent::post_to_pid` fallback.
+Both transports are process-scoped. [`press_key.rs`](../rust/crates/platform-macos/src/tools/press_key.rs)
+best-effort focuses an element and then posts the key to the PID. Its pixel form
+first invokes the click path to focus `(x, y)` and then posts the same
+process-scoped key, so that combined pointer-plus-key branch needs one exact
+route decision rather than bypassing keyboard safety after the focus click.
+
+[`type_text.rs`](../rust/crates/platform-macos/src/tools/type_text.rs) has a
+second exactness gap: when no explicit element pointer is supplied,
+`read_axvalue(pid, None)` reads `focused_element_of_pid(pid)`. Its before/after
+verification can therefore observe whichever same-process field is focused,
+not necessarily a descendant of the requested `window_id`. Passing an explicit
+window ID does not change that readback.
+
+The required same-PID regression test follows directly from these source
+semantics: unresolved target window A, focused sibling window B, process-routed
+key, and PID-global focused-element readback. V1 must refuse before the key is
+posted and prove from fixture journals that neither window changed.
+
+### Exact semantic and window-local primitives already exist
+
+[`ax_actions.rs`](../rust/crates/platform-macos/src/input/ax_actions.rs)
+supports `AXPress`, `AXConfirm`, selection, focus, and enabled-state checks.
+[`set_value.rs`](../rust/crates/platform-macos/src/tools/set_value.rs) already
+does native AX value readback and correctly distrusts AXValue echo from an
+`AXWebArea`.
+
+[`mouse.rs`](../rust/crates/platform-macos/src/input/mouse.rs) has a routed
+pointer primitive that stamps window-local location plus the target PID and
+CGWindowID fields before SkyLight/PID posting. It does not move the real
+pointer. The same file also contains private no-raise activation and
+Chromium-specific event recipes. These are implementation candidates, not by
+themselves proof that an action reached the requested window. V1 must gate them
+on exact target facts and exact post-action evidence.
+
+[`click.rs`](../rust/crates/platform-macos/src/tools/click.rs) already prefers
+AX, can perform window-local pointer fallback for specific selection controls,
+and distinguishes verified AX selection from an unverified dispatch. The new
+router should consolidate those decisions rather than add a second hidden
+fallback ladder.
+
+### Electron AX enablement exists but needs lifecycle and mutation discipline
+
+[`ax/bindings.rs`](../rust/crates/platform-macos/src/ax/bindings.rs) writes
+`AXManualAccessibility` and falls back to `AXEnhancedUserInterface` only when
+the modern attribute is unsupported. The tree walker enables it once per PID,
+waits 500 ms when accepted, and then walks the bounded tree.
+
+The current cache is a set of PIDs. V1 should associate enablement with the
+observed process lifetime so PID reuse cannot inherit a stale "enabled"
+decision. Electron mutation must reacquire an exact bounded tree rather than
+using title order, the first AX window, or stale process-global focus.
+
+### Exact browser mutation is already unique-or-refuse
+
+[`browser/binding.rs`](../rust/crates/cua-driver-core/src/browser/binding.rs)
+contains the pure native-window-to-CDP correlation. [`browser/engine.rs`](../rust/crates/cua-driver-core/src/browser/engine.rs)
+checks native ownership, process fingerprint, endpoint ownership, CDP window
+geometry/cardinality, tab identity, and generation again before every
+mutation. [`browser/tools.rs`](../rust/crates/cua-driver-core/src/browser/tools.rs)
+uses those exact capabilities for typed browser actions.
+
+V1 should reuse this rung rather than creating a macOS-specific browser
+shortcut. The bounded Electron fallback remains exact only when the endpoint
+exposes one page and the owner has exactly one native window. Two Electron
+`BrowserWindow`s must not use that fallback; they need a proven CDP-window
+mapping or a refusal. WKWebView mutation remains unsupported.
+
+### Capture and Spaces metadata are limited
+
+[`capture.rs`](../rust/crates/platform-macos/src/capture.rs) currently obtains
+a one-shot window image with `screencapture -l <CGWindowID> -x -o`. The
+ScreenCaptureKit implementation is display-scoped recording, not a
+window-scoped live stream. V1 keeps the current one-shot path and does not
+claim frame freshness from API success.
+
+[`list_windows.rs`](../rust/crates/platform-macos/src/tools/list_windows.rs)
+describes minimized, hidden, and off-Space enumeration, but the current macOS
+records do not reliably populate `current_space_id`, `on_current_space`, or
+`space_ids`. V1 must therefore use exact AX-window presence as its input safety
+gate instead of inventing a complete Spaces model.
+
+### The result contract is intentionally closed
+
+[`outputs.rs`](../rust/crates/cua-driver-contract/src/outputs.rs) defines the
+closed `ActionResult`: effect, route, delivery, evidence, and escalation.
+[`action_record.rs`](../rust/crates/cua-driver-core/src/action_record.rs) is the
+functional core that conservatively projects native attempts to that public
+shape. [`action-result-contract.md`](action-result-contract.md) explicitly
+excludes echoed selectors, coordinates, and request targets.
+
+V1 keeps that public result shape. Exact target facts belong to the request,
+the internal execution record, a read-only capability report, and structured
+refusals. Any future proposal to add optional result fields must update the
+contract and every generated strict SDK together; it is not part of this work.
+
+## Locked design decisions
+
+### 1. Exact-window invariant
+
+The canonical target key is the requested `(pid, CGWindowID)`. It is carried
+unchanged through planning, execution, and verification.
+
+The following rules are non-negotiable:
+
+- WindowServer must attribute the CGWindowID to the requested PID immediately
+ before mutation.
+- An AX element route must prove that the live element ascends to an AX window
+ whose `_AXUIElementGetWindow` is the requested CGWindowID.
+- A browser route must revalidate its existing exact native-window capability
+ immediately before mutation.
+- A pointer route must use the requested window's validated pixel frame and
+ stamped CGWindowID, then verify an effect on that same target.
+- A keyboard readback must read the addressed element or a fresh element
+ proven to descend from the requested AX window. It must never call a
+ PID-global focused-element reader as target evidence.
+- A sibling window's state may be retained as a negative canary, but it can
+ never confirm the requested action.
+- If the target disappears, changes owner, becomes AX-unresolved where the
+ chosen route needs AX, or becomes ambiguous between planning and dispatch,
+ no fallback actuator runs.
+
+An explicit `window_id` is an address supplied by the caller, not proof that a
+process-scoped actuator will honor it.
+
+### 2. Functional core and imperative shell
+
+Add a small pure background-input decision module in
+`cua-driver-core`. It owns no macOS objects and performs no I/O. Its inputs are
+facts collected by a platform shell:
+
+- exact target and WindowServer ownership;
+- AX scope (`matched`, `not_found`, `owner_mismatch`, `unresolved`);
+- requested action semantics and trust class;
+- target element ancestry and available semantic actions;
+- target state (`visible`, `occluded`, `offscreen`, `minimized`, `hidden`, or
+ unknown where macOS cannot distinguish safely);
+- same-PID top-level-window cardinality;
+- exact browser-binding eligibility;
+- exact window-local pointer eligibility;
+- keyboard destination and available verifier;
+- baseline foreground PID, real pointer position, and Space posture; and
+- permission/private-symbol readiness.
+
+The pure output is one route with prerequisites and a verification plan, or a
+typed refusal reason. Keep names internal, but model at least:
+
+```text
+ExactWindowTarget(pid, window_id)
+TargetResolution
+BackgroundRoute
+VerificationPlan
+BackgroundInputDecision = Execute(...) | Refuse(...)
+```
+
+Use a pure truth table for route choice and result classification. Extend the
+existing `ActionExecutionRecord` projection rather than adding macOS wire
+semantics. Platform adapters translate a chosen internal route to the existing
+public `ActionRoute` values.
+
+The macOS imperative shell:
+
+1. serializes mutation planning per PID;
+2. gathers fresh WindowServer, AX, browser, and foreground facts;
+3. asks the pure core for exactly one decision;
+4. revalidates route-specific facts immediately before dispatch;
+5. invokes only that actuator;
+6. polls only the planned exact-target verifier within a bounded deadline;
+7. checks foreground PID, real pointer, and Space posture; and
+8. emits the existing action result or a structured refusal.
+
+The core never says "try all routes." The shell cannot improvise a second
+actuator after a failed proof. A caller may take a newly reported route after
+fresh state, preserving the existing agent-owned action ladder.
+
+### 3. Background routing ladder
+
+The decision order is:
+
+| Order | Route | Required exactness | V1 behavior |
+| --- | --- | --- | --- |
+| 1 | AX semantic action/value | Exact AX window and exact element ancestry | Prefer `AXPress`, `AXConfirm`, selection, or value mutation with target-bound readback. AX API acceptance alone is unverified. |
+| 2 | Browser/CDP | Existing exact native-window, endpoint, CDP-window, tab, generation, and ref proof | Reuse typed browser tools. Do not guess or auto-convert an AX token into a DOM ref. |
+| 3 | Window-local routed pointer | Exact owner, AX-window presence, validated pixel frame, stamped CGWindowID, supported target state, and exact postcondition | Never move the real pointer. Private SPI absence or failed revalidation refuses. |
+| 4 | PID keyboard | Exact target element/window, no competing same-PID keyboard destination, and an exact verifier available before posting | Native single-window text insertion may qualify. Generic keys, multi-window apps, minimized/hidden windows, and unresolved off-Space windows refuse. |
+| 5 | Refusal | No safe route | Return the exact reason and available alternatives without sending input. |
+
+The ladder preserves action semantics. It must not silently replace a trusted
+browser click with a synthetic DOM event, infer a submit button from a label,
+or turn a desktop request into a browser request without an exact capability.
+Where request shapes cannot be translated exactly, the result advises the
+explicit tool the caller may use next.
+
+Global HID is not in this table. Only an explicit foreground request may use
+the existing guarded foreground/HID path.
+
+### 4. Semantic AX behavior and commit policy
+
+For a token/index action, revalidate the retained element's ancestry against
+the requested top-level AX window. A cached element under the right cache key
+is not sufficient after a lifecycle or Space transition.
+
+Safe commit behavior is explicit:
+
+- use an advertised `AXConfirm` on the exact field when available;
+- use `AXPress` on an explicitly addressed commit button;
+- use an exact browser action when the caller has a live page/ref capability;
+- otherwise refuse a background `Return` for minimized, hidden, unresolved,
+ or multi-window targets.
+
+Do not heuristically find the first default button or press another window's
+button. Do not turn `press_key(Return)` into `AXPress` unless the request itself
+identifies the semantic target.
+
+Native `AXValue` equality can confirm a value operation on the exact retained
+element. AX values under `AXWebArea` remain untrusted because the accessibility
+shim can echo a write that the renderer did not consume. Electron/web content
+requires browser/DOM readback or another exact application effect.
+
+### 5. Electron policy
+
+Electron support is part of v1, not a generic Chromium exception.
+
+For each observed process lifetime:
+
+1. Create the AX application element and try `AXManualAccessibility=true`.
+2. Fall back to `AXEnhancedUserInterface=true` only when the modern attribute
+ returns `kAXErrorAttributeUnsupported`; transient failures do not trigger a
+ second private write or a success claim.
+3. If enablement is accepted, wait the existing bounded settle and perform a
+ fresh exact-window walk. Key the enablement cache by process fingerprint or
+ generation, not an immortal numeric PID.
+4. Build the top-level `BrowserWindow` map only from `_AXUIElementGetWindow`.
+ Window title, array order, focused-window status, and first-match selection
+ are not identity proofs.
+5. Require the requested BrowserWindow to be in `AXWindows` before native
+ mutation. An absent target returns no elements and no input route.
+6. Before an element action, prove its ancestry to that BrowserWindow. After
+ the action, poll a fresh bounded exact-window tree or exact CDP/DOM state.
+ Never verify against process-global focus or a stale cached tree.
+7. Use a delayed bounded verifier because renderer and AX state publication are
+ asynchronous. Timeouts produce `unverifiable` or a proven delivery failure,
+ never a fabricated confirmation.
+8. Retain the existing distrust of web-area AXValue echo. Normal DOM text and
+ key semantics should use the exact browser rung when a binding is available.
+9. With two BrowserWindows, disable the single-page/single-native-window CDP
+ shortcut and every raw PID-key route. A route is available only when the
+ exact native/CDP mapping or exact semantic AX action survives revalidation.
+
+The signed fixture launches once without `--force-renderer-accessibility` to
+prove the driver's per-PID enablement works for a normal `BrowserWindow`, and
+once with the existing fixture flag as a diagnostic control. It also covers
+default `backgroundThrottling` and `backgroundThrottling:false`. The setting
+may affect changing pixels, but it must not weaken exact-target input policy or
+become a driver prerequisite.
+
+### 6. Minimized and hidden windows
+
+For minimized or application-hidden targets:
+
+- keep exact one-shot capture as observation when it succeeds;
+- label capture freshness as unknown rather than promising live frames;
+- allow exact semantic AX actions with target-bound verification;
+- allow an already-exact browser/CDP action only if all native and browser
+ binding checks still pass;
+- use semantic commit rather than a raw Return key;
+- refuse PID keyboard and native routed pointer in v1; and
+- do not deminiaturize, unhide, activate, reorder, move, or re-minimize the
+ window as an implementation detail.
+
+This preserves foreground state by construction. A no-activate restore
+transaction can be investigated as a later opt-in mode, but it is not a v1
+fallback.
+
+### 7. Other-Space policy
+
+Observation is allowed: `list_windows`, exact one-shot capture, and any honest
+degradation metadata may still be returned.
+
+For mutation, query a fresh application `AXWindows` and map every candidate via
+`_AXUIElementGetWindow`:
+
+- If the requested CGWindowID is absent, refuse all input with an
+ `off_space_or_ax_unresolved` reason. Do not inspect or act on a sibling
+ window, even if it is on the current Space.
+- If the exact target is present, v1 may use a semantic AX or exact browser
+ route whose own preconditions and postcondition pass. A native pixel or
+ PID-key route is not promoted merely because capture succeeded.
+- Unknown Space metadata remains unknown. Do not claim `on_current_space`
+ without a real source.
+
+Explicit switch/restore, temporarily adding a window to the current Space, and
+sticky-window behavior require a later opt-in design with its own visible-state
+and cleanup contract. V1 adds no SkyLight Spaces manipulation.
+
+### 8. Exact pointer route
+
+The existing window-local pointer implementation may be selected only when:
+
+- WindowServer ownership and AX-window identity both match the requested
+ target immediately before dispatch;
+- the current screenshot dimensions have a validated mapping to the requested
+ WindowServer bounds;
+- the local point falls inside that exact window frame;
+- the target is neither minimized, hidden, nor AX-unresolved/off-Space;
+- the required private symbols resolve on the running macOS version;
+- baseline frontmost PID and real pointer position have been captured; and
+- an exact postcondition exists for the selected control.
+
+Use the existing stamped local coordinate and CGWindowID fields. Treat private
+no-raise activation as part of the actuator, not as delivery proof. If it
+changes the user's frontmost application, pointer, or Space, mark the route
+failed, restore only state owned by the driver when that is safe, and disable
+the route for the remainder of the process session.
+
+The v1 router does not use unverified pixel input for a mutation whose only
+result is "event posted." It may remain available through current APIs as
+`unverifiable` outside the new exact capability, but it must not be advertised
+as exact background input.
+
+### 9. PID keyboard gate
+
+PID-routed keyboard is a last background rung because macOS accepts a process,
+not a CGWindowID.
+
+Before posting text, require all of:
+
+- a live explicit element proven to descend from the requested AX window;
+- that exact element is the PID's focused UI element and the PID's focused AX
+ window maps to the requested CGWindowID;
+- the requested target is the only eligible same-PID keyboard window at the
+ pre-dispatch revalidation point;
+- the target is visible or occluded on the current Space, not minimized or
+ hidden;
+- a target-bound value verifier is readable before dispatch; and
+- the value surface is native, not an `AXWebArea` echo surface.
+
+Post the event only after those facts pass, then poll the same retained target
+or a freshly reacquired equivalent inside the exact requested tree. If any
+precondition cannot be proven, refuse before input. Generic `press_key` and
+hotkeys normally lack a safe target-specific postcondition and therefore
+refuse in background exact mode unless a future action-specific verifier is
+added.
+
+The singleton rule is a conservative v1 proof, not a claim that
+`SLEventPostToPid` is window-addressed. It prevents the known class of
+same-process sibling delivery while semantic and browser routes cover most
+ordinary controls.
+
+### 10. Capability and result reporting
+
+Add an additive `background_input` section to the read-only exact-window state
+output. It should be produced from the same pure decision facts and contain no
+window titles, control values, or private symbol addresses. A representative
+shape is:
+
+```json
+{
+ "background_input": {
+ "exact_window": {
+ "status": "matched",
+ "pid": 123,
+ "window_id": 456,
+ "ax_window": "matched"
+ },
+ "routes": [
+ {"route": "accessibility", "status": "available"},
+ {
+ "route": "pid_keyboard",
+ "status": "refused",
+ "reason": "same_pid_sibling_windows"
+ }
+ ],
+ "observation": {
+ "one_shot_capture": "available",
+ "frame_freshness": "unknown"
+ }
+ }
+}
+```
+
+The capability route and status strings above are illustrative and are not new
+values for the closed public `ActionRoute` enum.
+
+The exact enums and generated types should be reviewed with the implementation,
+but the semantics are fixed:
+
+- `available` means route prerequisites are currently proven, not that a
+ future call is guaranteed to succeed;
+- every action revalidates instead of trusting this report;
+- unknown facts stay unknown;
+- route refusal includes one stable machine-readable reason; and
+- the report does not expose user content.
+
+Keep successful action outputs on the current `ActionResult` contract:
+
+- `route` uses the existing enum, including accessibility, synthetic events,
+ system API, DOM, and trusted input for relevant background actions;
+- `delivery.mode` reports what actually occurred;
+- `confirmed` requires publishable exact-target readback;
+- `unverifiable` means the route ran but exact effect is unknown;
+- `refused` means no actuator ran; and
+- escalation is advice for an explicit caller decision.
+
+Structured refusals should include `code`, `effect: "refused"`, the requested
+`pid` and `window_id`, the failed exactness fact, and the safe next route when
+one exists. Initial codes should distinguish at least ownership mismatch,
+AX-window unresolved/off-Space, same-PID keyboard ambiguity, unavailable
+verification, and unavailable private route.
+
+Do not report `delivery_failed` merely because the driver lacks evidence.
+Use it only when exact negative evidence proves delivery did not occur. A
+missing/unreadable postcondition is `unverifiable` with
+`effect_unconfirmed`; an unchanged readable exact target after the bounded
+delivery window may be `suspected_noop` or delivery failure according to the
+existing action's semantics. API acceptance is never confirmation.
+
+## State policy matrix
+
+This is the required decision policy, not a claim that every route already has
+native evidence:
+
+| Target state | Observe | AX semantic | Exact browser | Window-local pointer | PID keyboard |
+| --- | --- | --- | --- | --- | --- |
+| Frontmost | Current state/capture | Yes when exact | Yes when exact | Yes when exact and verified | Only with singleton exact element and verifier |
+| Background visible | Current state/capture | Yes when exact | Yes when exact | Yes when exact and verified | Same conservative gate |
+| Fully occluded | One-shot capture; freshness honest | Yes when exact | Yes when exact | Yes only after signed proof and exact verifier | Same conservative gate |
+| Fully offscreen on current Space | One-shot capture when valid | Yes when exact | Yes when exact | Signed-evidence gated; no geometry guess | Refuse in v1 |
+| Minimized | One-shot capture, freshness unknown | Yes when exact | Yes only if exact binding revalidates | Refuse | Refuse; semantic commit instead |
+| Application hidden | One-shot capture, freshness unknown | Yes when exact | Yes only if exact binding revalidates | Refuse | Refuse; semantic commit instead |
+| Other Space, exact AX window present | Observation allowed | Yes when exact | Yes when exact | Refuse in v1 | Refuse in v1 |
+| Other Space/AX window absent | Observation only | Refuse | Refuse | Refuse | Refuse |
+| Owner mismatch/stale window | Refuse stale target; report owner when safe | Refuse | Refuse | Refuse | Refuse |
+
+## Compatibility expectations
+
+These are route expectations to certify, not unsupported parity claims:
+
+| Surface | V1 expectation | Evidence required |
+| --- | --- | --- |
+| AppKit | Primary supported lane: exact AX actions/value, verified window-local pointer where semantic AX is unavailable, narrowly gated native text | Runtime AppKit fixture, including two windows under one PID |
+| SwiftUI | Exact AX on the current Space; truthful refusal when an off-Space window disappears from `AXWindows` | Runtime SwiftUI fixture on current and another Space |
+| Electron DOM | AX enablement and exact BrowserWindow tree; exact CDP route when bound; no multi-window PID keys | Runtime two-BrowserWindow fixture with per-window DOM journals |
+| Mac Catalyst | Semantic AX or verified pointer only; raw keyboard remains gated | Source-supported candidate; runtime row required before advertising |
+| Qt/Java | Use exposed exact AX semantics; refuse absent/custom AX and raw-key ambiguity | Source-based until dedicated signed fixtures exist |
+| Browser canvas/WebGL | Exact browser action when a supported ref/route exists; otherwise no semantic claim | Source-based browser binding plus route-specific runtime tests |
+| Games/Metal/custom event loops | No v1 background guarantee | Explicitly unsupported until an exact app-owned route exists |
+| Secure/protected surfaces and secure input | No capture or input bypass | Refusal/permission tests; never weaken platform protection |
+| System UI and out-of-process panels | Require actual WindowServer owner and exact AX window; never follow a PID silently | Existing owner-mismatch tests plus signed panel coverage |
+
+`backgroundThrottling:false` may preserve Electron painting in cases where the
+default renderer throttles. That is an application compatibility option, not a
+Cua Driver route guarantee. Canvas/WebGL, trusted-event filters, and custom
+event loops can still behave differently from ordinary DOM controls.
+
+## Staged implementation
+
+### Stage 0: Lock the safety regression
+
+- Extend the AppKit and Electron fixtures with two fixed-title windows, one
+ field and one commit control per window, and a run-owned per-window journal.
+- Reproduce process-scoped keyboard delivery with target A unresolved or
+ minimized and sibling B focused.
+- Add a test that requests A by exact `(pid, window_id)` and requires a refusal
+ before dispatch; both journals and both field values must remain unchanged.
+- Add a unit regression showing that a sibling's focused-element readback can
+ never satisfy A's verification plan.
+
+Do not refactor routes until this failing behavioral shape and pure regression
+are both captured.
+
+### Stage 1: Pure exact-target router
+
+- Add the target facts, route decision, verifier requirement, and refusal
+ reasons to common core.
+- Encode the complete state matrix as table-driven unit tests.
+- Make "explicit `window_id`" distinct from "exact delivery proven."
+- Extend internal action records with diagnostic exact-target proof state only
+ as needed; do not change the public `ActionResult` shape.
+- Add a per-PID mutation coordinator so two concurrent background decisions
+ cannot change process focus underneath one another.
+
+### Stage 2: Fresh macOS target acquisition
+
+- Centralize WindowServer owner preflight and fresh `AXWindows` mapping.
+- Add a bounded ancestry check from a retained element to its exact AX window.
+- Make Chromium/Electron AX enablement process-lifetime aware and preserve the
+ modern-attribute/fallback error distinction.
+- Reacquire bounded exact-window state for delayed postconditions and clear
+ stale caches on every unresolved result.
+- Change the `AxUnresolved` input advice from unconditional pixel mutation to
+ observation-only/refusal when exact input cannot be proven.
+
+### Stage 3: Semantic actions and exact verification
+
+- Route exact `AXPress`, `AXConfirm`, selection, and native value mutations
+ first.
+- Bind every readback to the requested target element/window.
+- Preserve web-area AX echo distrust and add browser/DOM verification where an
+ exact capability already exists.
+- Make process-level window changes diagnostic only unless the changed window
+ is the exact target postcondition.
+- Normalize `confirmed`, `unverifiable`, `suspected_noop`, and proven delivery
+ failure through the functional core.
+
+### Stage 4: Browser, pointer, and keyboard rungs
+
+- Reuse browser mutation revalidation without a second Electron-only binding
+ implementation.
+- Gate the existing window-local pointer route on exact target, state, pixel
+ frame, private-symbol readiness, and verifier facts.
+- Route `press_key`'s existing pixel-focus-then-key form through the same pure
+ decision. A successful focus click cannot waive the exact AX-window,
+ singleton destination, target-bound verifier, or state gates for the key.
+- Remove PID-global focused-element readback from window-scoped keyboard
+ verification.
+- Permit native PID text only under the singleton exact-element/verifier gate.
+- Refuse generic keys and every unsafe minimized/hidden/off-Space raw-key case.
+- Assert that the background router has no call edge to global HID or
+ foreground assist.
+
+### Stage 5: Capabilities, documentation, and rollout
+
+- Add the read-only per-window capability report and structured refusal codes.
+- Keep request and `ActionResult` schemas compatible; regenerate/check SDK
+ artifacts only if the read-only output has a generated typed surface.
+- Update the background-delivery and known-limit documentation to match the
+ certified matrix, not the intended matrix.
+- Ship the high-risk same-PID refusal before expanding routes. If a new route
+ regresses, disable that route while retaining the exact-target guard.
+- Run the full desktop acceptance matrix once against the final exact SHA, as
+ required by repository guidance.
+
+## Test and acceptance plan
+
+### Ordinary CI
+
+Run these on every implementation pull request without a GUI or TCC grants:
+
+- pure route-decision table for every state and route;
+- owner mismatch, stale ID, AX-unresolved, and exact matched scope;
+- explicit-window versus exact-delivery distinction;
+- same-PID sibling rejection before actuator invocation;
+- wrong-sibling readback ignored by verification;
+- pixel-focus-then-key cannot bypass keyboard refusal or dispatch a second
+ actuator after target facts change;
+- no background decision can produce a global-HID transport;
+- native value, missing readback, unchanged value, partial text, delayed
+ success, and negative-delivery result classification;
+- Electron PID-lifetime enablement cache and fallback behavior;
+- exact BrowserWindow map cardinality and no first/title-only match;
+- existing CDP binding/revalidation, including one-page/one-window fallback
+ refusal after a second native window appears;
+- minimized/hidden/off-Space policy truth tables;
+- read-only capability and structured-refusal schema snapshots;
+- existing closed `ActionResult` invariants and generated SDK parity; and
+- fixture build/lint tests.
+
+### Signed macOS behavior matrix
+
+The native harness must use only repository fixtures and a fixture-owned
+occlusion sentinel. Each action records pre/post foreground PID, real pointer
+position, requested CGWindowID, AX window IDs, route, result, exact fixture
+journal entry, and cleanup state.
+
+Required rows:
+
+1. **AppKit:** frontmost, background visible, fully occluded, offscreen,
+ minimized, and hidden. Cover AXPress, AXConfirm, AXValue, exact pointer, and
+ singleton native text.
+2. **AppKit same PID/two windows:** request each window independently; minimize
+ or make one AX-unresolved while the sibling is focused; raw key/text must
+ refuse and neither sibling journal may change.
+3. **SwiftUI:** background AX actions and value readback on the current Space;
+ move only the fixture window to another preconfigured Space, query from the
+ other Space, and require refusal whenever its CGWindowID is absent from
+ `AXWindows`.
+4. **Electron DOM/two BrowserWindows:** run without force-renderer AX first,
+ then the diagnostic flag; run default throttling and
+ `backgroundThrottling:false`; prove exact AX window mapping, AXPress,
+ semantic commit, exact browser type/click, delayed readback, and raw-key
+ refusal for minimized/unresolved targets. Every effect must appear in the
+ addressed window's own fixture journal and not its sibling's.
+5. **Browser routes:** exact bind and mutation, stale generation, geometry
+ mismatch, second native window, ambiguous tab/window, and unavailable
+ endpoint. Every ambiguous row refuses without mutation.
+6. **Foreground preservation:** for every background success and refusal, the
+ fixture sentinel remains frontmost, the real pointer is unchanged, and the
+ current Space is unchanged.
+7. **Capture:** record whether one-shot capture succeeds in each state and
+ never infer live animation from a successful PNG alone.
+8. **Permissions/private APIs:** missing Accessibility, missing Screen
+ Recording, and missing private symbols return an honest unavailable/refused
+ result without prompting or global fallback.
+9. **Cleanup:** close fixture windows, stop all fixture/driver processes,
+ restore the initial fixture foreground, pointer, and Space, and leave no
+ run-owned artifact outside the selected artifact directory.
+
+The other-Space setup belongs to the disposable runner image or a test-only
+harness. Product code must not create, join, or switch Spaces. The test records
+its starting Space and always restores it in a trap/finally path.
+
+### Canonical Lume gate
+
+The repository's canonical signed/TCC acceptance remains
+[`tests/runners/macos-lume/README.md`](../tests/runners/macos-lume/README.md)
+and [`run-all.sh`](../tests/runners/macos-lume/run-all.sh). It provides the
+logged-in Aqua session, certificate-backed `CuaDriverLocal.app`, normal TCC
+grant flow, and exact-source checks.
+
+After focused Namespace evidence passes, add the new fixture rows to the Lume
+entrypoint and run once on the final candidate SHA. Preserve its existing
+requirements:
+
+- use a disposable worker cloned from the stopped private seed;
+- install with `install-local.sh --release --autostart
+ --require-stable-signing`;
+- never edit `TCC.db`;
+- execute the GUI suite from the guest's logged-in Terminal, not SSH;
+- retain exact SHA, environment, code-signing requirement, permission status,
+ structured results, fixture journals, screenshots/recording where allowed,
+ and cleanup evidence; and
+- retrieve evidence, then delete the worker.
+
+Namespace validation complements this gate; it does not replace the Lume
+golden-image contract.
+
+## Namespace macOS runner workflow
+
+### Discovery recorded for this plan
+
+The planning checkout was inspected on 2026-08-04:
+
+- project `.amp/plugins` was absent;
+- the surrounding workspace `.amp/plugins` was absent;
+- user `~/.config/amp/plugins` contained
+ `namespace-mac-runners.ts` and `namespace-mac-runners.js`;
+- no repository-local Namespace runner instructions were found; and
+- the TypeScript source registered the exact tool name
+ `namespace_mac_runner`.
+
+The user plugin is not versioned with this repository, so a later agent must
+repeat discovery before use. Check project `.amp/plugins`, user
+`~/.config/amp/plugins`, the active workspace plugin location, Amp plugin
+documentation, and the loaded plugin source. If the tool is absent, record
+those checks and make plugin discovery/setup step one; do not guess an API.
+
+The discovered tool accepts:
+
+```text
+action: list | spawn | destroy
+runner_id: optional hostname-compatible ID for spawn
+repository_url: optional HTTPS github.com URL for spawn
+duration_minutes: optional integer from 5 through 720
+instance_id: required for destroy
+```
+
+The discovered implementation defaults to macOS `26.x`, arm64, 6 vCPU,
+14,336 MB, and a 60-minute TTL. It requires `nsc login`. Spawn creates billable
+infrastructure immediately without confirmation, clones into
+`$HOME/workspace/repository` through Amp's GitHub credential helper, starts
+`amp --no-tui --runner-id "$AMP_RUNNER_ID" --remote-control-terminal` in tmux,
+and waits for `Registered. Create threads`. Failed provisioning attempts to
+destroy the instance it created. Destroy accepts only plugin-managed instance
+IDs and asks for confirmation. These details must be re-read from the installed
+plugin because it can change independently of this plan.
+
+### Exact later validation workflow
+
+1. Re-run the plugin discovery above and record the resolved source/config.
+2. Call `namespace_mac_runner` with `{"action":"list"}`. Do not modify or
+ destroy unrelated runners.
+3. Obtain explicit approval for the billable spawn. The approval for this plan
+ PR does not authorize a later infrastructure charge.
+4. Push the implementation candidate and record its full SHA. Spawn with a
+ unique hostname-compatible ID and bounded TTL, for example:
+
+ ```json
+ {
+ "action": "spawn",
+ "runner_id": "cua-bg-input-<unique-suffix>",
+ "repository_url": "https://github.com/trycua/cua.git",
+ "duration_minutes": 180
+ }
+ ```
+
+5. Record the returned runner ID, Namespace instance ID, URL, resolved macOS
+ selector, repository URL, and deadline. TTL is a backstop, not cleanup.
+6. Call Amp's `list_runners` immediately before `create_thread`. Start a child
+ with `executor: "runner"` and that exact live runner ID; do not silently use
+ an orb. Instruct it to fetch and detach-checkout the exact candidate SHA,
+ because the plugin's shallow clone starts from repository default HEAD.
+7. On the Namespace Mac, record `sw_vers`, OS build, architecture, SIP status,
+ Xcode/Rust/Node versions, exact git SHA, and a clean worktree before testing.
+8. Use repository-supported `CuaDriverLocal.app` setup only. Follow
+ [`scripts/README.md`](../scripts/README.md), use an ephemeral
+ certificate-backed local signing identity, and install with
+ `--require-stable-signing`. Never copy a maintainer private key or signing
+ keychain to Namespace. Never edit `TCC.db` or bypass TCC.
+9. Grant Accessibility and Screen Recording only through normal macOS UI if the
+ disposable Namespace environment and explicit test approval support it.
+ Record `permissions status --json` and the `codesign -d -r-` designated
+ requirement. The Namespace plugin provisions neither a signing identity nor
+ TCC. If stable signing, interactive UI, or either grant is unavailable,
+ mark the signed native lane blocked and run only pure/build tests; do not
+ claim native behavior passed.
+10. Use only fixed-title repository fixtures and run-owned artifact paths.
+ Capture the matrix evidence listed above, including exact fixture journal
+ effects, requested and sibling window IDs, AX counts/mapping, result
+ payloads, foreground PID, real pointer, Space posture, and action timing.
+ Do not collect unrelated application state, window titles, or user data.
+11. In a finally/trap path, stop fixture and driver processes, close fixture
+ windows, restore fixture focus/pointer/Space state, remove run-owned
+ temporary files, and write a cleanup assertion. Retrieve only sanitized
+ evidence needed for review.
+12. From the controlling thread, call
+ `namespace_mac_runner` with
+ `{"action":"destroy","instance_id":"<recorded-instance-id>"}` and confirm
+ destruction. Call `{"action":"list"}` again and require the instance to be
+ absent.
+
+## Risk analysis
+
+### macOS and private APIs
+
+`_AXUIElementGetWindow`, `AXManualAccessibility`,
+`AXEnhancedUserInterface`, SkyLight event posting, private event fields, and
+no-raise activation may change between macOS releases. Every lookup must be
+bounded and fail closed. Missing symbols remove a route; they never cause a
+global or PID-only fallback.
+
+V1 adds no private Spaces API. Space membership remains incomplete, and the
+policy deliberately refuses when AX cannot prove the exact target.
+
+### TCC and signing identity
+
+Accessibility and capture evidence is valid only for the process identity that
+performed it. Ad-hoc rebuilds can invalidate grants. Native acceptance requires
+the certificate-backed `CuaDriverLocal.app` flow and normal permission UI.
+Neither CI mocks nor a successful build substitutes for this evidence.
+
+### Application semantic differences
+
+AX API success may be a no-op, Chromium AXValue may be an echo, and custom
+renderers may reject synthetic input. Confirmation always comes from an
+independent exact-target effect. Unsupported applications receive narrower
+capabilities, not a misleading parity claim.
+
+### Races and lifecycle
+
+Windows can close, change owner, add a modal, or restart between observation
+and input. Per-PID serialization and immediate revalidation reduce the window;
+they do not make stale capabilities permanent. Any changed fact invalidates
+the decision and requires fresh state.
+
+### Compatibility and rollout
+
+Request schemas and successful action results remain stable. Some calls that
+currently post best-effort PID keys will begin refusing when exact delivery is
+not provable. That is an intentional safety correction. Refusals include a
+recoverable semantic/browser/foreground alternative when one actually exists.
+
+Roll out one route at a time after its fixture lane passes. The rollback is to
+disable the new route decision while retaining the same-PID/unresolved-target
+guard. Do not restore the unsafe process-scoped fallback to improve success
+rates.
+
+Telemetry and retained artifacts may contain route, refusal reason, OS build,
+target PID/window ID, timing, and boolean invariants. They must not contain
+window titles, field values, typed text, screenshots of unrelated apps,
+browser URLs, or fixture logs outside the fixed test vocabulary.
+
+## Alternatives rejected for v1
+
+- **Post to the PID and verify any focused field.** This is the wrong-window
+ bug, not a verification strategy.
+- **Assume explicit `window_id` makes `post_to_pid` exact.** The keyboard
+ transport has no window parameter.
+- **Restore without activation, order behind, act, and re-minimize.** This
+ changes target state, can still change internal key-window routing, and adds
+ cleanup races. It belongs in a later opt-in design.
+- **Move the window offscreen or to another Space.** This mutates user desktop
+ state and does not solve AX or renderer suspension reliably.
+- **Temporarily make a window sticky or add it to the current Space.** This is
+ full Spaces manipulation, outside v1's contract.
+- **Treat capture success as input exactness.** A CGWindowID image proves the
+ capture target, not the destination of a process-scoped key.
+- **Treat AX API acceptance or event-post return as confirmation.** Both can
+ report success without the intended application effect.
+- **Silently use foreground HID.** It violates the background contract and can
+ act on the user's current application.
+- **Add request echoes to the closed `ActionResult`.** Callers already own the
+ request; a wire change would require coordinated strict-SDK migration without
+ improving route safety.
+
+## Open implementation questions
+
+Resolve these during Stage 0/1 review, before enabling a route:
+
+1. Which existing read-only output surface should own the additive
+ `background_input` capability in every SDK without duplicating native
+ window state?
+2. Which stable process fingerprint should replace the PID-only Chromium AX
+ enablement cache on macOS?
+3. Which native text-field properties are sufficient to predeclare a safe
+ postcondition for the singleton PID-key rung?
+4. Can the current private no-raise pointer preparation satisfy foreground,
+ pointer, and Space invariants on every supported macOS build? Builds without
+ signed evidence must keep the route disabled.
+5. What bounded delayed-verification deadline gives Electron enough time while
+ staying below the public tool timeout? Use evidence from the fixture rather
+ than reusing an unrelated settle constant blindly.
+6. How will the disposable Lume/Namespace image provide a deterministic second
+ Space for tests while guaranteeing restoration? This is a test-environment
+ decision, not a product Space API.
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/background_input.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/background_input.rs
new file mode 100644
index 0000000000..cfd35d2647
--- /dev/null
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/background_input.rs
@@ -0,0 +1,698 @@
+//! Pure exact-target decisions for macOS-style background input (v1).
+//!
+//! Background mutation must carry one exact `(pid, CGWindowID)` target from
+//! request to postcondition. The platform shell gathers fresh facts about that
+//! target immediately before dispatch and asks this module for exactly one
+//! decision: execute the requested route, or refuse with a stable
+//! machine-readable reason. The shell never improvises a second actuator after
+//! a refusal, and a sibling window's state can never confirm the requested
+//! action.
+//!
+//! This module owns no platform objects and performs no I/O, so the complete
+//! state-policy matrix is testable in ordinary CI. See
+//! `docs/macos-background-input-v1-plan.md` for the design contract.
+
+use serde_json::{json, Value};
+
+/// The canonical target key: the requested `(pid, CGWindowID)`.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub struct ExactWindowTarget {
+ pub pid: i32,
+ pub window_id: u32,
+}
+
+/// WindowServer's attribution of the requested CGWindowID, resolved
+/// immediately before the decision.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum WindowServerOwnership {
+ /// The window exists and the requested pid owns it.
+ SamePid,
+ /// WindowServer has no record of the id — closed, stale, or fabricated.
+ NotFound,
+ /// The window exists but another process owns it (out-of-process panels).
+ ForeignPid { owner_pid: i32 },
+}
+
+/// Whether an explicitly addressed element was proven to belong to the
+/// requested window.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum ElementAncestry {
+ /// The action does not address a retained element.
+ NotAddressed,
+ /// The live element ascends to an AX window whose CGWindowID is the
+ /// requested one.
+ ProvenDescendant,
+ /// The live element ascends to a different window of the same process.
+ OutsideTargetWindow,
+ /// Ancestry could not be resolved (dead element, SPI failure). An address
+ /// the shell cannot re-prove is not an exact target.
+ Unproven,
+}
+
+/// Fresh facts about one exact target, gathered by the platform shell.
+///
+/// Unknown facts must stay unknown: gatherers must not guess a value merely to
+/// unlock a route.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct BackgroundTargetFacts {
+ pub window_server: WindowServerOwnership,
+ /// The requested CGWindowID is claimed by one of the application's fresh
+ /// `AXWindows` (mapped through `_AXUIElementGetWindow`). Absent means
+ /// off-Space or AX-unresolved: observation-only.
+ pub ax_window_present: bool,
+ /// `AXMinimized` on the exact target window. `None` means the attribute
+ /// could not be read — an unproven fact, which fails closed for pointer
+ /// and keyboard routes (it never unlocks them).
+ pub target_minimized: Option<bool>,
+ /// `AXHidden` on the owning application. `None` fails closed like
+ /// `target_minimized`.
+ pub app_hidden: Option<bool>,
+ /// Same-pid top-level windows, other than the target, that could receive
+ /// process-scoped keyboard input (enumerated from WindowServer so an
+ /// off-Space sibling that AX cannot see still counts; proven-minimized
+ /// siblings are excluded because they cannot be the key window).
+ pub competing_keyboard_destinations: usize,
+ /// Ancestry proof for an explicitly addressed element, when one exists.
+ pub element: ElementAncestry,
+}
+
+/// The background action classes v1 distinguishes. Route order and trust are
+/// documented in the plan's routing ladder.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum BackgroundAction {
+ /// Semantic AX action or value mutation on an exact element
+ /// (`AXPress`, `AXConfirm`, selection, `AXValue`).
+ AxSemantic,
+ /// Window-local routed pointer (stamped CGWindowID; real pointer unmoved).
+ WindowPointer,
+ /// Process-scoped text insertion (AX selected-text write or CGEvent
+ /// keystrokes) with a target-bound value readback.
+ InsertText,
+ /// Process-scoped key/hotkey without an action-specific verifier.
+ GenericKey,
+}
+
+impl BackgroundAction {
+ fn is_pid_keyboard(self) -> bool {
+ matches!(
+ self,
+ BackgroundAction::InsertText | BackgroundAction::GenericKey
+ )
+ }
+}
+
+/// Stable machine-readable refusal codes. These appear in structured tool
+/// results; do not rename them without a compatibility review.
+pub mod refusal_codes {
+ pub const WINDOW_NOT_FOUND: &str = "window_not_found";
+ pub const OWNER_PID_MISMATCH: &str = "owner_pid_mismatch";
+ pub const OFF_SPACE_OR_AX_UNRESOLVED: &str = "off_space_or_ax_unresolved";
+ pub const MINIMIZED_OR_HIDDEN: &str = "minimized_or_hidden_window";
+ pub const SAME_PID_KEYBOARD_AMBIGUITY: &str = "same_pid_keyboard_ambiguity";
+ pub const ELEMENT_OUTSIDE_TARGET_WINDOW: &str = "element_outside_target_window";
+}
+
+/// A typed refusal: no actuator ran and none may run for this decision.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct BackgroundRefusal {
+ pub code: &'static str,
+ pub reason: String,
+ /// The safe next route when one actually exists (advice for an explicit
+ /// caller decision, never an implicit fallback).
+ pub advice: Option<&'static str>,
+}
+
+/// Verification binding for an executed route: evidence is acceptable only
+/// when it belongs to the exact requested window.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub struct TargetBoundVerification {
+ pub window_id: u32,
+}
+
+impl TargetBoundVerification {
+ /// Whether a readback observed on an element whose resolved window is
+ /// `evidence_window_id` may confirm the requested action. Unresolvable
+ /// evidence (`None`) is never accepted — it is unverifiable, not proof —
+ /// and a sibling window's state can never confirm the requested target.
+ pub fn accepts_evidence_from(&self, evidence_window_id: Option<u32>) -> bool {
+ evidence_window_id == Some(self.window_id)
+ }
+}
+
+/// One decision for one gathered-facts snapshot.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum BackgroundInputDecision {
+ Execute {
+ /// The only evidence source that may report `confirmed`.
+ verification: TargetBoundVerification,
+ },
+ Refuse(BackgroundRefusal),
+}
+
+impl BackgroundInputDecision {
+ pub fn is_execute(&self) -> bool {
+ matches!(self, BackgroundInputDecision::Execute { .. })
+ }
+}
+
+fn refuse(
+ code: &'static str,
+ reason: String,
+ advice: Option<&'static str>,
+) -> BackgroundInputDecision {
+ BackgroundInputDecision::Refuse(BackgroundRefusal {
+ code,
+ reason,
+ advice,
+ })
+}
+
+/// Decide whether one background action may run against one exact target.
+///
+/// The rules encode the v1 state-policy matrix:
+/// - a stale or foreign CGWindowID refuses every mutation;
+/// - a target absent from fresh `AXWindows` (off-Space or AX-unresolved) is
+/// observation-only;
+/// - an addressed element must prove ancestry to the requested window for
+/// every route, including semantic AX;
+/// - semantic AX actions remain available for minimized/hidden targets;
+/// - the routed pointer requires a visible (possibly occluded) target; and
+/// - process-scoped keyboard additionally requires that the target is the
+/// only eligible same-pid keyboard destination, because the transport
+/// addresses a process, not a window.
+pub fn decide_background_input(
+ target: ExactWindowTarget,
+ facts: &BackgroundTargetFacts,
+ action: BackgroundAction,
+) -> BackgroundInputDecision {
+ match &facts.window_server {
+ WindowServerOwnership::NotFound => {
+ return refuse(
+ refusal_codes::WINDOW_NOT_FOUND,
+ format!(
+ "WindowServer has no record of window {} (closed or stale); \
+ re-enumerate windows and re-target",
+ target.window_id
+ ),
+ None,
+ );
+ }
+ WindowServerOwnership::ForeignPid { owner_pid } => {
+ return refuse(
+ refusal_codes::OWNER_PID_MISMATCH,
+ format!(
+ "window {} is owned by pid {owner_pid}, not requested pid {}; \
+ re-target the actual owner",
+ target.window_id, target.pid
+ ),
+ None,
+ );
+ }
+ WindowServerOwnership::SamePid => {}
+ }
+
+ match facts.element {
+ ElementAncestry::NotAddressed | ElementAncestry::ProvenDescendant => {}
+ ElementAncestry::OutsideTargetWindow | ElementAncestry::Unproven => {
+ return refuse(
+ refusal_codes::ELEMENT_OUTSIDE_TARGET_WINDOW,
+ format!(
+ "the addressed element could not be proven to belong to window {}; \
+ take a fresh get_window_state snapshot and re-address it",
+ target.window_id
+ ),
+ Some("get_window_state"),
+ );
+ }
+ }
+
+ if !facts.ax_window_present {
+ return refuse(
+ refusal_codes::OFF_SPACE_OR_AX_UNRESOLVED,
+ format!(
+ "window {} is not among the process's current AXWindows (another Space, \
+ or its AX surface is unresolved); background input is refused so a \
+ sibling window cannot receive it. Observation (capture/list_windows) \
+ remains available",
+ target.window_id
+ ),
+ Some("foreground"),
+ );
+ }
+
+ // Pointer/keyboard require PROVEN not-minimized and not-hidden: an
+ // unreadable attribute (`None`) is an unknown fact and unknown facts
+ // never unlock a route. Semantic AX stays available either way.
+ let visibility_disproven =
+ !(facts.target_minimized == Some(false) && facts.app_hidden == Some(false));
+ match action {
+ BackgroundAction::AxSemantic => BackgroundInputDecision::Execute {
+ verification: TargetBoundVerification {
+ window_id: target.window_id,
+ },
+ },
+ BackgroundAction::WindowPointer => {
+ if visibility_disproven {
+ return refuse(
+ refusal_codes::MINIMIZED_OR_HIDDEN,
+ format!(
+ "window {} is minimized or its application is hidden (or that \
+ state could not be proven); the routed pointer is refused in \
+ v1. Use an exact element action (click/set_value by element) \
+ instead",
+ target.window_id
+ ),
+ Some("accessibility"),
+ );
+ }
+ BackgroundInputDecision::Execute {
+ verification: TargetBoundVerification {
+ window_id: target.window_id,
+ },
+ }
+ }
+ BackgroundAction::InsertText | BackgroundAction::GenericKey => {
+ if visibility_disproven {
+ return refuse(
+ refusal_codes::MINIMIZED_OR_HIDDEN,
+ format!(
+ "window {} is minimized or its application is hidden (or that \
+ state could not be proven); raw key input is refused in v1. \
+ Use an exact element action (set_value, click \
+ action:\"confirm\"/\"press\") instead",
+ target.window_id
+ ),
+ Some("accessibility"),
+ );
+ }
+ debug_assert!(action.is_pid_keyboard());
+ if facts.competing_keyboard_destinations > 0 {
+ return refuse(
+ refusal_codes::SAME_PID_KEYBOARD_AMBIGUITY,
+ format!(
+ "pid {} owns {} other eligible top-level window(s); process-scoped \
+ key events cannot be proven to reach window {} and could mutate a \
+ sibling window. Use an exact element action, the page tool for \
+ browser content, or delivery_mode:\"foreground\"",
+ target.pid, facts.competing_keyboard_destinations, target.window_id
+ ),
+ Some("accessibility"),
+ );
+ }
+ BackgroundInputDecision::Execute {
+ verification: TargetBoundVerification {
+ window_id: target.window_id,
+ },
+ }
+ }
+ }
+}
+
+/// The exact-window resolution string reported by the read-only capability
+/// section.
+fn exact_window_status(facts: &BackgroundTargetFacts) -> &'static str {
+ match &facts.window_server {
+ WindowServerOwnership::NotFound => "not_found",
+ WindowServerOwnership::ForeignPid { .. } => "owner_mismatch",
+ WindowServerOwnership::SamePid if facts.ax_window_present => "matched",
+ WindowServerOwnership::SamePid => "ax_unresolved",
+ }
+}
+
+/// Build the additive read-only `background_input` capability report from the
+/// same pure facts that gate every action.
+///
+/// Semantics: `available` means the route's prerequisites are currently
+/// proven, not that a future call is guaranteed to succeed; every action
+/// revalidates instead of trusting this report. The report contains no window
+/// titles, control values, or private symbol addresses.
+pub fn background_input_capability_report(
+ target: ExactWindowTarget,
+ facts: &BackgroundTargetFacts,
+ one_shot_capture_available: Option<bool>,
+) -> Value {
+ let route_entry = |route: &str, action: BackgroundAction| -> Value {
+ match decide_background_input(target, facts, action) {
+ BackgroundInputDecision::Execute { .. } => {
+ json!({ "route": route, "status": "available" })
+ }
+ BackgroundInputDecision::Refuse(refusal) => {
+ json!({ "route": route, "status": "refused", "reason": refusal.code })
+ }
+ }
+ };
+ json!({
+ "exact_window": {
+ "status": exact_window_status(facts),
+ "pid": target.pid,
+ "window_id": target.window_id,
+ },
+ "routes": [
+ route_entry("accessibility", BackgroundAction::AxSemantic),
+ route_entry("window_pointer", BackgroundAction::WindowPointer),
+ route_entry("pid_keyboard", BackgroundAction::GenericKey),
+ ],
+ "observation": {
+ "one_shot_capture": match one_shot_capture_available {
+ Some(true) => "available",
+ Some(false) => "unavailable",
+ None => "unknown",
+ },
+ "frame_freshness": "unknown",
+ },
+ })
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ const TARGET: ExactWindowTarget = ExactWindowTarget {
+ pid: 42,
+ window_id: 700,
+ };
+
+ fn matched_facts() -> BackgroundTargetFacts {
+ BackgroundTargetFacts {
+ window_server: WindowServerOwnership::SamePid,
+ ax_window_present: true,
+ target_minimized: Some(false),
+ app_hidden: Some(false),
+ competing_keyboard_destinations: 0,
+ element: ElementAncestry::NotAddressed,
+ }
+ }
+
+ const ALL_ACTIONS: [BackgroundAction; 4] = [
+ BackgroundAction::AxSemantic,
+ BackgroundAction::WindowPointer,
+ BackgroundAction::InsertText,
+ BackgroundAction::GenericKey,
+ ];
+
+ fn code_of(decision: BackgroundInputDecision) -> &'static str {
+ match decision {
+ BackgroundInputDecision::Refuse(refusal) => refusal.code,
+ BackgroundInputDecision::Execute { .. } => panic!("expected a refusal"),
+ }
+ }
+
+ /// Regression for the proven same-PID wrong-target case: target window A
+ /// requested while sibling window B is the process's focused/eligible
+ /// window. Process-scoped keyboard must refuse before dispatch — an
+ /// explicit `window_id` is an address, not delivery proof.
+ #[test]
+ fn two_window_process_refuses_background_keyboard_for_both_text_and_keys() {
+ let facts = BackgroundTargetFacts {
+ competing_keyboard_destinations: 1,
+ ..matched_facts()
+ };
+ for action in [BackgroundAction::InsertText, BackgroundAction::GenericKey] {
+ assert_eq!(
+ code_of(decide_background_input(TARGET, &facts, action)),
+ refusal_codes::SAME_PID_KEYBOARD_AMBIGUITY,
+ "{action:?} must refuse while a sibling keyboard destination exists"
+ );
+ }
+ // Semantic AX and the stamped window-local pointer remain available:
+ // they are window-addressed, not process-addressed.
+ for action in [
+ BackgroundAction::AxSemantic,
+ BackgroundAction::WindowPointer,
+ ] {
+ assert!(
+ decide_background_input(TARGET, &facts, action).is_execute(),
+ "{action:?} is window-addressed and stays available"
+ );
+ }
+ }
+
+ /// A sibling window's state can never satisfy the requested target's
+ /// verification plan, and unresolvable evidence is not proof.
+ #[test]
+ fn sibling_readback_never_confirms_the_requested_window() {
+ let verification = TargetBoundVerification { window_id: 700 };
+ assert!(verification.accepts_evidence_from(Some(700)));
+ assert!(!verification.accepts_evidence_from(Some(701)), "sibling");
+ assert!(!verification.accepts_evidence_from(None), "unresolvable");
+ }
+
+ #[test]
+ fn stale_window_id_refuses_every_mutation() {
+ let facts = BackgroundTargetFacts {
+ window_server: WindowServerOwnership::NotFound,
+ ax_window_present: false,
+ ..matched_facts()
+ };
+ for action in ALL_ACTIONS {
+ assert_eq!(
+ code_of(decide_background_input(TARGET, &facts, action)),
+ refusal_codes::WINDOW_NOT_FOUND,
+ "{action:?}"
+ );
+ }
+ }
+
+ #[test]
+ fn foreign_owner_refuses_every_mutation() {
+ let facts = BackgroundTargetFacts {
+ window_server: WindowServerOwnership::ForeignPid { owner_pid: 900 },
+ ..matched_facts()
+ };
+ for action in ALL_ACTIONS {
+ assert_eq!(
+ code_of(decide_background_input(TARGET, &facts, action)),
+ refusal_codes::OWNER_PID_MISMATCH,
+ "{action:?}"
+ );
+ }
+ }
+
+ /// Off-Space / AX-unresolved targets are observation-only: no route may
+ /// send input, and no sibling window may be used instead.
+ #[test]
+ fn ax_unresolved_target_is_observation_only() {
+ let facts = BackgroundTargetFacts {
+ ax_window_present: false,
+ ..matched_facts()
+ };
+ for action in ALL_ACTIONS {
+ assert_eq!(
+ code_of(decide_background_input(TARGET, &facts, action)),
+ refusal_codes::OFF_SPACE_OR_AX_UNRESOLVED,
+ "{action:?}"
+ );
+ }
+ }
+
+ /// Minimized/hidden targets retain exact semantic AX actions; raw keys and
+ /// the routed pointer refuse rather than restoring the window.
+ #[test]
+ fn minimized_or_hidden_keeps_semantic_ax_and_refuses_keys_and_pointer() {
+ for facts in [
+ BackgroundTargetFacts {
+ target_minimized: Some(true),
+ ..matched_facts()
+ },
+ BackgroundTargetFacts {
+ app_hidden: Some(true),
+ ..matched_facts()
+ },
+ ] {
+ assert!(
+ decide_background_input(TARGET, &facts, BackgroundAction::AxSemantic).is_execute()
+ );
+ for action in [
+ BackgroundAction::WindowPointer,
+ BackgroundAction::InsertText,
+ BackgroundAction::GenericKey,
+ ] {
+ assert_eq!(
+ code_of(decide_background_input(TARGET, &facts, action)),
+ refusal_codes::MINIMIZED_OR_HIDDEN,
+ "{action:?}"
+ );
+ }
+ }
+ }
+
+ /// Unknown visibility fails closed: when minimized/hidden state could not
+ /// be read, the routed pointer and raw keys refuse exactly as if the
+ /// window were minimized, while exact semantic AX actions remain allowed.
+ #[test]
+ fn unknown_visibility_fails_closed_for_pointer_and_keyboard() {
+ for facts in [
+ BackgroundTargetFacts {
+ target_minimized: None,
+ ..matched_facts()
+ },
+ BackgroundTargetFacts {
+ app_hidden: None,
+ ..matched_facts()
+ },
+ ] {
+ assert!(
+ decide_background_input(TARGET, &facts, BackgroundAction::AxSemantic).is_execute()
+ );
+ for action in [
+ BackgroundAction::WindowPointer,
+ BackgroundAction::InsertText,
+ BackgroundAction::GenericKey,
+ ] {
+ assert_eq!(
+ code_of(decide_background_input(TARGET, &facts, action)),
+ refusal_codes::MINIMIZED_OR_HIDDEN,
+ "{action:?}"
+ );
+ }
+ }
+ }
+
+ /// The singleton rule: with no competing keyboard destination, exact
+ /// background text/keys are permitted and verification is target-bound.
+ #[test]
+ fn singleton_window_permits_background_keyboard_with_target_bound_verification() {
+ let facts = matched_facts();
+ for action in [BackgroundAction::InsertText, BackgroundAction::GenericKey] {
+ match decide_background_input(TARGET, &facts, action) {
+ BackgroundInputDecision::Execute { verification } => {
+ assert_eq!(verification.window_id, TARGET.window_id);
+ }
+ other => panic!("{action:?} expected Execute, got {other:?}"),
+ }
+ }
+ }
+
+ /// An addressed element that cannot be proven to descend from the
+ /// requested window refuses every route — a cached element under the right
+ /// cache key is not sufficient after a lifecycle or Space transition.
+ #[test]
+ fn unproven_element_ancestry_refuses_all_routes() {
+ for element in [
+ ElementAncestry::OutsideTargetWindow,
+ ElementAncestry::Unproven,
+ ] {
+ let facts = BackgroundTargetFacts {
+ element,
+ ..matched_facts()
+ };
+ for action in ALL_ACTIONS {
+ assert_eq!(
+ code_of(decide_background_input(TARGET, &facts, action)),
+ refusal_codes::ELEMENT_OUTSIDE_TARGET_WINDOW,
+ "{action:?} with {element:?}"
+ );
+ }
+ }
+ }
+
+ #[test]
+ fn proven_element_ancestry_executes_semantic_ax_even_when_minimized() {
+ let facts = BackgroundTargetFacts {
+ element: ElementAncestry::ProvenDescendant,
+ target_minimized: Some(true),
+ ..matched_facts()
+ };
+ assert!(decide_background_input(TARGET, &facts, BackgroundAction::AxSemantic).is_execute());
+ }
+
+ /// Refusal precedence: exactness failures are reported before state or
+ /// cardinality failures so the caller fixes the most fundamental fact.
+ #[test]
+ fn refusal_precedence_owner_before_element_before_scope_before_state() {
+ let worst = BackgroundTargetFacts {
+ window_server: WindowServerOwnership::ForeignPid { owner_pid: 9 },
+ ax_window_present: false,
+ target_minimized: Some(true),
+ app_hidden: Some(true),
+ competing_keyboard_destinations: 3,
+ element: ElementAncestry::OutsideTargetWindow,
+ };
+ assert_eq!(
+ code_of(decide_background_input(
+ TARGET,
+ &worst,
+ BackgroundAction::InsertText
+ )),
+ refusal_codes::OWNER_PID_MISMATCH
+ );
+ let bad_element = BackgroundTargetFacts {
+ window_server: WindowServerOwnership::SamePid,
+ ..worst.clone()
+ };
+ assert_eq!(
+ code_of(decide_background_input(
+ TARGET,
+ &bad_element,
+ BackgroundAction::InsertText
+ )),
+ refusal_codes::ELEMENT_OUTSIDE_TARGET_WINDOW
+ );
+ let unresolved = BackgroundTargetFacts {
+ element: ElementAncestry::NotAddressed,
+ ..bad_element
+ };
+ assert_eq!(
+ code_of(decide_background_input(
+ TARGET,
+ &unresolved,
+ BackgroundAction::InsertText
+ )),
+ refusal_codes::OFF_SPACE_OR_AX_UNRESOLVED
+ );
+ let minimized = BackgroundTargetFacts {
+ ax_window_present: true,
+ ..unresolved
+ };
+ assert_eq!(
+ code_of(decide_background_input(
+ TARGET,
+ &minimized,
+ BackgroundAction::InsertText
+ )),
+ refusal_codes::MINIMIZED_OR_HIDDEN
+ );
+ }
+
+ #[test]
+ fn capability_report_shape_is_stable() {
+ let report = background_input_capability_report(TARGET, &matched_facts(), Some(true));
+ assert_eq!(report["exact_window"]["status"], "matched");
+ assert_eq!(report["exact_window"]["pid"], 42);
+ assert_eq!(report["exact_window"]["window_id"], 700);
+ assert_eq!(report["routes"][0]["route"], "accessibility");
+ assert_eq!(report["routes"][0]["status"], "available");
+ assert_eq!(report["routes"][2]["route"], "pid_keyboard");
+ assert_eq!(report["routes"][2]["status"], "available");
+ assert_eq!(report["observation"]["one_shot_capture"], "available");
+ assert_eq!(report["observation"]["frame_freshness"], "unknown");
+ }
+
+ #[test]
+ fn capability_report_refusals_carry_stable_reason_codes() {
+ let facts = BackgroundTargetFacts {
+ ax_window_present: false,
+ ..matched_facts()
+ };
+ let report = background_input_capability_report(TARGET, &facts, Some(false));
+ assert_eq!(report["exact_window"]["status"], "ax_unresolved");
+ for route in report["routes"].as_array().unwrap() {
+ assert_eq!(route["status"], "refused");
+ assert_eq!(route["reason"], refusal_codes::OFF_SPACE_OR_AX_UNRESOLVED);
+ }
+ assert_eq!(report["observation"]["one_shot_capture"], "unavailable");
+
+ let two_windows = BackgroundTargetFacts {
+ competing_keyboard_destinations: 1,
+ ..matched_facts()
+ };
+ let report = background_input_capability_report(TARGET, &two_windows, None);
+ assert_eq!(report["routes"][0]["status"], "available");
+ assert_eq!(report["routes"][2]["status"], "refused");
+ assert_eq!(
+ report["routes"][2]["reason"],
+ refusal_codes::SAME_PID_KEYBOARD_AMBIGUITY
+ );
+ assert_eq!(report["observation"]["one_shot_capture"], "unknown");
+ }
+}
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs
index c220f56df5..fca981562b 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs
@@ -47,6 +47,7 @@ pub fn parent_liveness_stdin_enabled() -> bool {
pub mod action_record;
pub mod authorization;
+pub mod background_input;
pub mod browser;
pub mod capture_mode;
pub mod capture_scope;
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/enablement.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/enablement.rs
new file mode 100644
index 0000000000..762acbb1f6
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/enablement.rs
@@ -0,0 +1,133 @@
+//! Process-lifetime Chromium/Electron accessibility enablement.
+//!
+//! Chromium-family apps (Arc, VS Code, Electron shells) ship their web-content
+//! AX tree OFF and only build it once an assistive client asks for it. The
+//! walker flips `AXManualAccessibility` (falling back to
+//! `AXEnhancedUserInterface` only when the modern attribute is unsupported —
+//! see [`super::bindings::enable_chromium_accessibility`]) and pays a one-time
+//! settle so the asynchronously-built tree exists before it is read.
+//!
+//! The "already enabled" cache is keyed by the observed process lifetime
+//! (pid + kernel start time), not by the numeric pid alone: pids are recycled,
+//! and a relaunched Electron app must not inherit a stale "enabled" decision
+//! that would skip enablement and return an empty web-content tree.
+
+use std::collections::HashMap;
+use std::sync::{Mutex, OnceLock};
+
+use super::bindings::{enable_chromium_accessibility, AXUIElementRef};
+
+/// How long to let a freshly-enabled Chromium/Electron app build its
+/// web-content AX tree before we read it. Paid at most once per process
+/// lifetime.
+const CHROMIUM_SETTLE_SECONDS: f64 = 0.5;
+
+type ProcessStartStamp = (u64, u64);
+
+/// Kernel start time of a process: `(pbi_start_tvsec, pbi_start_tvusec)`.
+/// `None` when the process is gone or proc info is unreadable.
+fn process_start_stamp(pid: i32) -> Option<ProcessStartStamp> {
+ // SAFETY: proc_pidinfo writes at most `size` bytes into `info` and returns
+ // the number of bytes filled (<= size) or <= 0 on failure.
+ unsafe {
+ let mut info: libc::proc_bsdinfo = std::mem::zeroed();
+ let size = std::mem::size_of::<libc::proc_bsdinfo>() as libc::c_int;
+ let filled = libc::proc_pidinfo(
+ pid,
+ libc::PROC_PIDTBSDINFO,
+ 0,
+ &mut info as *mut _ as *mut libc::c_void,
+ size,
+ );
+ if filled != size {
+ return None;
+ }
+ Some((info.pbi_start_tvsec, info.pbi_start_tvusec))
+ }
+}
+
+/// Process lifetimes for which enablement has already run and the settle has
+/// been paid. Values are the process start stamp observed at enablement time.
+fn enabled_processes() -> &'static Mutex<HashMap<i32, Option<ProcessStartStamp>>> {
+ static ENABLED: OnceLock<Mutex<HashMap<i32, Option<ProcessStartStamp>>>> = OnceLock::new();
+ ENABLED.get_or_init(|| Mutex::new(HashMap::new()))
+}
+
+fn cached_lifetime_is_current(
+ cached: Option<&Option<ProcessStartStamp>>,
+ observed: Option<ProcessStartStamp>,
+) -> bool {
+ cached.is_some_and(|cached| cached.is_some() && *cached == observed)
+}
+
+/// Flip Chromium/Electron accessibility on for `pid`'s application element and
+/// wait for the web-content tree to settle — once per observed process
+/// lifetime. Native Cocoa apps reject the attribute and pay no settle cost.
+///
+/// # Safety
+///
+/// `app_element` must be a valid application `AXUIElementRef` for `pid`.
+pub unsafe fn ensure_chromium_ax_enabled(pid: i32, app_element: AXUIElementRef) {
+ let stamp = process_start_stamp(pid);
+ let already_enabled = enabled_processes()
+ .lock()
+ .map(|cache| cached_lifetime_is_current(cache.get(&pid), stamp))
+ .unwrap_or(false);
+ if already_enabled {
+ return;
+ }
+ if enable_chromium_accessibility(app_element) {
+ crate::permissions::panel::pump_run_loop_briefly(CHROMIUM_SETTLE_SECONDS);
+ if let Ok(mut cache) = enabled_processes().lock() {
+ cache.insert(pid, stamp);
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn start_stamp_reads_the_current_process_and_rejects_dead_pids() {
+ let own = process_start_stamp(std::process::id() as i32);
+ assert!(own.is_some(), "own process start time must be readable");
+ // Pid 0 is the kernel idle task; proc info for it is not readable from
+ // user space, so lifetime keying must fail closed (None).
+ assert_eq!(process_start_stamp(0), None);
+ }
+
+ #[test]
+ fn distinct_lifetimes_do_not_alias() {
+ // Two different processes must not produce identical (pid, stamp)
+ // cache keys. Use launchd (pid 1) vs our own process.
+ let own_pid = std::process::id() as i32;
+ let own = process_start_stamp(own_pid);
+ let launchd = process_start_stamp(1);
+ if let (Some(own), Some(launchd)) = (own, launchd) {
+ assert_ne!(
+ (own_pid, own),
+ (1, launchd),
+ "cache keys must differ across processes"
+ );
+ }
+ }
+
+ #[test]
+ fn cache_hit_requires_the_same_readable_process_lifetime() {
+ let first_launch = Some((100, 10));
+ let relaunched = Some((101, 20));
+
+ assert!(cached_lifetime_is_current(
+ Some(&first_launch),
+ first_launch
+ ));
+ assert!(
+ !cached_lifetime_is_current(Some(&first_launch), relaunched),
+ "a relaunched process must not inherit the prior enablement cache entry"
+ );
+ assert!(!cached_lifetime_is_current(Some(&first_launch), None));
+ assert!(!cached_lifetime_is_current(Some(&None), first_launch));
+ assert!(!cached_lifetime_is_current(None, first_launch));
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/exact_target.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/exact_target.rs
new file mode 100644
index 0000000000..b2ed078893
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/exact_target.rs
@@ -0,0 +1,191 @@
+//! Fresh exact-target acquisition for background input decisions.
+//!
+//! Gathers the facts [`cua_driver_core::background_input`] needs about one
+//! requested `(pid, CGWindowID)` immediately before a background mutation:
+//! WindowServer ownership, fresh `AXWindows` membership (mapped through
+//! `_AXUIElementGetWindow`), minimized/hidden state, competing same-pid
+//! keyboard destinations, and addressed-element ancestry. All reads are
+//! bounded and fail closed — an unreadable fact never unlocks a route.
+
+use core_foundation::base::{CFRelease, CFTypeRef};
+use cua_driver_core::background_input::{
+ BackgroundTargetFacts, ElementAncestry, WindowServerOwnership,
+};
+
+use super::bindings::{
+ ax_get_window_id, copy_ax_windows, copy_bool_attr, copy_element_attr, copy_string_attr,
+ focused_element_of_pid, AXUIElementCreateApplication, AXUIElementRef,
+};
+use crate::windows::{all_windows, resolve_window_owner, WindowOwner};
+
+/// Bounded `AXParent` ascent used when an element does not expose `AXWindow`.
+const MAX_ANCESTRY_DEPTH: usize = 40;
+
+/// Resolve the CGWindowID of the top-level AX window that owns `element`.
+///
+/// Prefers the element's `AXWindow` attribute and falls back to a bounded
+/// `AXParent` walk. `None` means ancestry could not be proven — callers must
+/// treat that as "not the requested window", never as a wildcard.
+///
+/// # Safety
+///
+/// `element` must be a valid `AXUIElementRef` for the duration of the call.
+pub unsafe fn element_window_id(element: AXUIElementRef) -> Option<u32> {
+ if let Some(window) = copy_element_attr(element, "AXWindow") {
+ let window_id = ax_get_window_id(window);
+ CFRelease(window as CFTypeRef);
+ if window_id.is_some() {
+ return window_id;
+ }
+ }
+ // Fallback: ascend AXParent until a window role, then map it.
+ let mut current: AXUIElementRef = element;
+ let mut owned = false;
+ let mut resolved = None;
+ for _ in 0..MAX_ANCESTRY_DEPTH {
+ match copy_string_attr(current, "AXRole").as_deref() {
+ Some("AXWindow") | Some("AXSheet") => {
+ resolved = ax_get_window_id(current);
+ break;
+ }
+ Some("AXApplication") | None => break,
+ _ => {}
+ }
+ let parent = copy_element_attr(current, "AXParent");
+ if owned {
+ CFRelease(current as CFTypeRef);
+ }
+ match parent {
+ Some(parent) => {
+ current = parent;
+ owned = true;
+ }
+ None => return None,
+ }
+ }
+ if owned {
+ CFRelease(current as CFTypeRef);
+ }
+ resolved
+}
+
+/// The process's focused AX element, but only when it provably belongs to the
+/// requested window. Returns a retained element the caller must release.
+///
+/// This is the only focused-element reader background window-scoped keyboard
+/// paths may use: a PID-global focused element can belong to a sibling window,
+/// and sibling state must never address or confirm the requested target.
+///
+/// # Safety
+///
+/// Caller must `CFRelease` the returned element.
+pub unsafe fn focused_element_in_window(pid: i32, window_id: u32) -> Option<AXUIElementRef> {
+ let element = focused_element_of_pid(pid)?;
+ if element_window_id(element) == Some(window_id) {
+ Some(element)
+ } else {
+ CFRelease(element as CFTypeRef);
+ None
+ }
+}
+
+/// One fresh `AXWindows` row: the mapped CGWindowID plus its minimized state.
+/// `minimized: None` means the attribute could not be read — unknown, not
+/// "not minimized".
+struct AxWindowRecord {
+ window_id: u32,
+ minimized: Option<bool>,
+}
+
+/// Map the application's fresh `AXWindows` through `_AXUIElementGetWindow`.
+/// Windows whose id the SPI cannot resolve are omitted: an unmappable window
+/// can never satisfy an exact-target requirement.
+unsafe fn ax_window_records(app: AXUIElementRef) -> Vec<AxWindowRecord> {
+ copy_ax_windows(app)
+ .into_iter()
+ .filter_map(|window| {
+ let record = ax_get_window_id(window).map(|window_id| AxWindowRecord {
+ window_id,
+ minimized: copy_bool_attr(window, "AXMinimized"),
+ });
+ CFRelease(window as CFTypeRef);
+ record
+ })
+ .collect()
+}
+
+/// Gather fresh background-input facts for one `(pid, window_id)` target.
+///
+/// `element_ptr` is an optional retained `AXUIElementRef` (as `usize`) for an
+/// explicitly addressed element; the caller must keep it retained for the
+/// duration of this call. Blocking: performs one CGWindowList enumeration and
+/// bounded AX reads. Call from a blocking context immediately before deciding.
+pub fn gather_background_facts(
+ pid: i32,
+ window_id: u32,
+ element_ptr: Option<usize>,
+) -> BackgroundTargetFacts {
+ let window_server = match resolve_window_owner(pid, window_id) {
+ WindowOwner::SamePid => WindowServerOwnership::SamePid,
+ WindowOwner::Unknown => WindowServerOwnership::NotFound,
+ WindowOwner::ForeignPid { owner_pid, .. } => {
+ WindowServerOwnership::ForeignPid { owner_pid }
+ }
+ };
+
+ // SAFETY: the application element is created and released here; window
+ // elements are released inside ax_window_records; the caller guarantees
+ // element_ptr stays retained.
+ let (records, app_hidden, element) = unsafe {
+ let app = AXUIElementCreateApplication(pid);
+ if app.is_null() {
+ (
+ Vec::new(),
+ None,
+ element_ptr.map(|_| ElementAncestry::Unproven),
+ )
+ } else {
+ // Electron/Chromium apps may need per-process-lifetime enablement
+ // before their AX windows and subtrees are materialized.
+ super::enablement::ensure_chromium_ax_enabled(pid, app);
+ let records = ax_window_records(app);
+ let app_hidden = copy_bool_attr(app, "AXHidden");
+ let element = element_ptr.map(|ptr| match element_window_id(ptr as AXUIElementRef) {
+ Some(id) if id == window_id => ElementAncestry::ProvenDescendant,
+ Some(_) => ElementAncestry::OutsideTargetWindow,
+ None => ElementAncestry::Unproven,
+ });
+ CFRelease(app as CFTypeRef);
+ (records, app_hidden, element)
+ }
+ };
+
+ let target = records.iter().find(|record| record.window_id == window_id);
+ let minimized_ids: Vec<u32> = records
+ .iter()
+ .filter(|record| record.minimized == Some(true))
+ .map(|record| record.window_id)
+ .collect();
+
+ // Competing keyboard destinations come from WindowServer, not AX: an
+ // off-Space sibling that AXWindows cannot see is still a real key-window
+ // candidate for process-scoped events. Proven-minimized windows are
+ // excluded — a miniaturized window cannot be the key window.
+ let competing_keyboard_destinations = all_windows()
+ .iter()
+ .filter(|window| {
+ window.pid == pid
+ && window.window_id != window_id
+ && !minimized_ids.contains(&window.window_id)
+ })
+ .count();
+
+ BackgroundTargetFacts {
+ window_server,
+ ax_window_present: target.is_some(),
+ target_minimized: target.and_then(|record| record.minimized),
+ app_hidden,
+ competing_keyboard_destinations,
+ element: element.unwrap_or(ElementAncestry::NotAddressed),
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/mod.rs
index 817c8dc7eb..ee7b6c405a 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/ax/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/mod.rs
@@ -18,6 +18,8 @@
pub mod bindings;
pub mod cache;
+pub mod enablement;
+pub mod exact_target;
pub mod tree;
pub mod window_scope;
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs
index 822b3866cc..ee1f5d3a1d 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs
@@ -14,8 +14,6 @@
use super::bindings::*;
use super::window_scope::{decide_window_scope, TopLevelCandidate, WindowScope};
use core_foundation::base::{CFEqual, CFRelease, CFRetain, CFTypeRef};
-use std::collections::HashSet;
-use std::sync::{Mutex, OnceLock};
/// Default maximum depth for AX tree walks. Deep menus and complex web views
/// can nest deeply; 25 covers realistic app chrome without exploding on
@@ -36,14 +34,6 @@ pub const DEFAULT_MAX_DEPTH: usize = 25;
/// (issue #22865).
pub const DEFAULT_MAX_ELEMENTS: usize = 2_000;
-/// How long to let a freshly-enabled Chromium/Electron app build its
-/// web-content AX tree before we read it. The tree is materialized
-/// asynchronously over IPC once the app detects an assistive client, so a
-/// walk that starts immediately sees only the chrome (title bar, a handful
-/// of elements). This settle is paid at most once per pid — see
-/// `enabled_pids`.
-const CHROMIUM_SETTLE_SECONDS: f64 = 0.5;
-
/// Bound each native AX request. Tokio cannot cancel a blocked
/// `AXUIElementCopyAttributeValue` after `spawn_blocking` starts, so the native
/// messaging timeout is what keeps an unresponsive app from retaining a worker
@@ -54,14 +44,6 @@ unsafe fn set_messaging_timeout(element: AXUIElementRef) {
let _ = AXUIElementSetMessagingTimeout(element, AX_MESSAGING_TIMEOUT_SECONDS);
}
-/// Pids for which we have already flipped on accessibility and paid the
-/// one-time settle delay. Repeat snapshots of the same app skip the settle:
-/// the tree is already built and stays built for the life of the process.
-fn enabled_pids() -> &'static Mutex<HashSet<i32>> {
- static ENABLED_PIDS: OnceLock<Mutex<HashSet<i32>>> = OnceLock::new();
- ENABLED_PIDS.get_or_init(|| Mutex::new(HashSet::new()))
-}
-
/// A single node in the AX tree.
#[derive(Debug, Clone)]
pub struct AXNode {
@@ -234,20 +216,11 @@ pub fn walk_tree_bounded(
// asks for it. Without this, the first walk of such an app returns an
// empty/title-bar-only tree (#1616). Flip the enablement attribute,
// then — only when the flip actually took and only the first time we
- // see this pid — let the asynchronously-built tree settle before we
- // read it. Native Cocoa apps reject the attribute, so they pay no
- // settle cost. This relies on the MAX_ELEMENTS node cap to keep the
- // now-materialized (potentially large) tree bounded.
- let already_enabled = enabled_pids()
- .lock()
- .map(|s| s.contains(&pid))
- .unwrap_or(false);
- if !already_enabled && enable_chromium_accessibility(app_elem) {
- crate::permissions::panel::pump_run_loop_briefly(CHROMIUM_SETTLE_SECONDS);
- if let Ok(mut set) = enabled_pids().lock() {
- set.insert(pid);
- }
- }
+ // see this process lifetime — let the asynchronously-built tree settle
+ // before we read it. Native Cocoa apps reject the attribute, so they
+ // pay no settle cost. This relies on the MAX_ELEMENTS node cap to keep
+ // the now-materialized (potentially large) tree bounded.
+ super::enablement::ensure_chromium_ax_enabled(pid, app_elem);
// Union AXChildren + AXWindows — the only way to see background windows.
// AXChildren omits windows when the app isn't frontmost (AppKit limitation).
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/background_mutation.rs b/libs/cua-driver/rust/crates/platform-macos/src/background_mutation.rs
new file mode 100644
index 0000000000..4ca69d9ab5
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-macos/src/background_mutation.rs
@@ -0,0 +1,99 @@
+//! Per-process serialization for exact-target background mutations.
+//!
+//! macOS keyboard delivery and focus suppression are process-scoped. Two
+//! concurrent window-addressed mutations for one pid must not interleave their
+//! fresh target proof, focus changes, dispatch, restoration, or verification.
+//! Different processes remain independent.
+
+use std::collections::HashMap;
+use std::future::Future;
+use std::sync::{Arc, Mutex, OnceLock, Weak};
+use tokio::sync::{Mutex as AsyncMutex, OwnedMutexGuard};
+
+tokio::task_local! {
+ static HELD_PID: i32;
+}
+
+fn process_locks() -> &'static Mutex<HashMap<i32, Weak<AsyncMutex<()>>>> {
+ static LOCKS: OnceLock<Mutex<HashMap<i32, Weak<AsyncMutex<()>>>>> = OnceLock::new();
+ LOCKS.get_or_init(|| Mutex::new(HashMap::new()))
+}
+
+fn process_lock(pid: i32) -> Arc<AsyncMutex<()>> {
+ let mut locks = process_locks()
+ .lock()
+ .unwrap_or_else(|error| error.into_inner());
+ locks.retain(|_, lock| lock.strong_count() > 0);
+ if let Some(lock) = locks.get(&pid).and_then(Weak::upgrade) {
+ return lock;
+ }
+ let lock = Arc::new(AsyncMutex::new(()));
+ locks.insert(pid, Arc::downgrade(&lock));
+ lock
+}
+
+/// Acquire exclusive background-mutation ownership for one process.
+///
+/// The returned guard must live from immediately before fresh fact gathering
+/// until dispatch, focus restoration, and postcondition verification finish.
+pub(crate) async fn acquire(pid: i32) -> OwnedMutexGuard<()> {
+ process_lock(pid).lock_owned().await
+}
+
+/// Run one nested tool call with non-forgeable proof that its caller already
+/// owns this process's mutation lease.
+pub(crate) async fn with_held_lease<T>(pid: i32, future: impl Future<Output = T>) -> T {
+ HELD_PID.scope(pid, future).await
+}
+
+pub(crate) fn held_by_current_task(pid: i32) -> bool {
+ HELD_PID
+ .try_with(|held_pid| *held_pid == pid)
+ .unwrap_or(false)
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use std::time::Duration;
+
+ #[tokio::test]
+ async fn same_pid_mutations_serialize_until_the_first_guard_drops() {
+ let first = acquire(-91_001).await;
+ let mut waiting = tokio::spawn(acquire(-91_001));
+
+ assert!(
+ tokio::time::timeout(Duration::from_millis(30), &mut waiting)
+ .await
+ .is_err(),
+ "a second mutation for the same pid must remain queued"
+ );
+ drop(first);
+
+ let second = tokio::time::timeout(Duration::from_secs(1), waiting)
+ .await
+ .expect("same-pid waiter must proceed after the first guard drops")
+ .expect("same-pid waiter task must not panic");
+ drop(second);
+ }
+
+ #[tokio::test]
+ async fn different_pids_do_not_block_each_other() {
+ let first = acquire(-91_002).await;
+ let second = tokio::time::timeout(Duration::from_millis(100), acquire(-91_003))
+ .await
+ .expect("independent pids must acquire concurrently");
+ drop((first, second));
+ }
+
+ #[tokio::test]
+ async fn nested_lease_proof_is_task_local_and_pid_bound() {
+ assert!(!held_by_current_task(-91_004));
+ with_held_lease(-91_004, async {
+ assert!(held_by_current_task(-91_004));
+ assert!(!held_by_current_task(-91_005));
+ })
+ .await;
+ assert!(!held_by_current_task(-91_004));
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/lib.rs b/libs/cua-driver/rust/crates/platform-macos/src/lib.rs
index 196077bd9e..a75c1dc96f 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/lib.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/lib.rs
@@ -11,6 +11,8 @@ pub mod apps;
#[cfg(target_os = "macos")]
pub mod ax;
#[cfg(target_os = "macos")]
+mod background_mutation;
+#[cfg(target_os = "macos")]
pub mod browser;
#[cfg(target_os = "macos")]
pub mod capture;
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/background_input_regression_tests.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/background_input_regression_tests.rs
new file mode 100644
index 0000000000..dc58908dbb
--- /dev/null
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/background_input_regression_tests.rs
@@ -0,0 +1,149 @@
+use super::*;
+use async_trait::async_trait;
+use cua_driver_core::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+use serde_json::Value;
+use std::sync::{
+ atomic::{AtomicUsize, Ordering},
+ Arc, Mutex,
+};
+use std::time::Duration;
+
+#[tokio::test]
+async fn same_pid_lease_covers_proof_dispatch_restoration_and_verification() {
+ const PID: i32 = -92_101;
+ let events = Arc::new(Mutex::new(Vec::new()));
+ let (first_started_tx, first_started_rx) = tokio::sync::oneshot::channel();
+ let (finish_first_tx, finish_first_rx) = tokio::sync::oneshot::channel();
+
+ let first_events = events.clone();
+ let first = tokio::spawn(async move {
+ let _lease = acquire_background_mutation(PID).await;
+ first_events
+ .lock()
+ .unwrap()
+ .extend(["first:proof", "first:dispatch"]);
+ first_started_tx.send(()).unwrap();
+ finish_first_rx.await.unwrap();
+ first_events
+ .lock()
+ .unwrap()
+ .extend(["first:restore", "first:verify"]);
+ });
+ first_started_rx.await.unwrap();
+
+ let second_events = events.clone();
+ let mut second = tokio::spawn(async move {
+ let _lease = acquire_background_mutation(PID).await;
+ second_events.lock().unwrap().push("second:proof");
+ });
+
+ assert!(
+ tokio::time::timeout(Duration::from_millis(50), &mut second)
+ .await
+ .is_err(),
+ "the sibling-window mutation must wait through restoration and verification"
+ );
+ finish_first_tx.send(()).unwrap();
+ first.await.unwrap();
+ tokio::time::timeout(Duration::from_secs(1), second)
+ .await
+ .expect("same-pid mutation should proceed after the full lifecycle")
+ .unwrap();
+
+ assert_eq!(
+ *events.lock().unwrap(),
+ [
+ "first:proof",
+ "first:dispatch",
+ "first:restore",
+ "first:verify",
+ "second:proof",
+ ]
+ );
+}
+
+#[tokio::test]
+async fn different_pid_lease_enters_while_first_mutation_is_in_flight() {
+ const FIRST_PID: i32 = -92_102;
+ const SECOND_PID: i32 = -92_103;
+ let first = acquire_background_mutation(FIRST_PID).await;
+ let (entered_tx, entered_rx) = tokio::sync::oneshot::channel();
+
+ let second = tokio::spawn(async move {
+ let _lease = acquire_background_mutation(SECOND_PID).await;
+ entered_tx.send(()).unwrap();
+ });
+
+ tokio::time::timeout(Duration::from_secs(1), entered_rx)
+ .await
+ .expect("a different pid must not wait for the in-flight mutation")
+ .unwrap();
+ drop(first);
+ second.await.unwrap();
+}
+
+struct KeyboardProbe {
+ calls: Arc<AtomicUsize>,
+}
+
+static KEYBOARD_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+
+#[async_trait]
+impl Tool for KeyboardProbe {
+ fn def(&self) -> &ToolDef {
+ KEYBOARD_DEF.get_or_init(|| ToolDef {
+ name: "press_key".into(),
+ description: "keyboard ambiguity regression probe".into(),
+ input_schema: serde_json::json!({"type": "object"}),
+ read_only: false,
+ destructive: true,
+ idempotent: false,
+ open_world: false,
+ })
+ }
+
+ async fn invoke(&self, _args: Value) -> ToolResult {
+ self.calls.fetch_add(1, Ordering::SeqCst);
+ ToolResult::text("keyboard actuator invoked")
+ }
+}
+
+fn candidate(window_id: u64) -> WindowTargetCandidate {
+ WindowTargetCandidate {
+ window_id,
+ title: format!("Document {window_id}"),
+ app_name: Some("Editor".into()),
+ is_on_screen: true,
+ }
+}
+
+#[tokio::test]
+async fn pid_only_keyboard_refuses_same_pid_multi_window_ambiguity_before_dispatch() {
+ let calls = Arc::new(AtomicUsize::new(0));
+ let candidates: WindowTargetCandidates = Arc::new(|pid| {
+ (pid == 42)
+ .then(|| vec![candidate(7), candidate(8)])
+ .unwrap_or_default()
+ });
+ let keyboard = pid_window_guarded(
+ KeyboardProbe {
+ calls: calls.clone(),
+ },
+ &candidates,
+ );
+
+ let result = keyboard
+ .invoke(serde_json::json!({"pid": 42, "key": "return"}))
+ .await;
+
+ assert_eq!(calls.load(Ordering::SeqCst), 0, "no key may be posted");
+ assert_eq!(result.is_error, Some(true));
+ let refusal = result.structured_content.unwrap();
+ assert_eq!(refusal["code"], "ambiguous_window_target");
+ assert_eq!(refusal["effect"], "refused");
+ assert_eq!(refusal["candidates"][0]["window_id"], 7);
+ assert_eq!(refusal["candidates"][1]["window_id"], 8);
+}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
index 67f038d4dd..59298a27f4 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs
@@ -438,6 +438,28 @@ impl Tool for ClickTool {
};
let element_ptr = element_guard.as_ptr();
+ // ── Exact-target background gate (macOS background input v1) ──
+ // The element branch is semantic AX delivery, except button=middle
+ // which falls back to a routed pixel click at the element's center
+ // and is therefore held to the stricter WindowPointer rung. Gate
+ // BEFORE any cursor/dispatch work so a stale or sibling-owned
+ // target refuses instead of acting on the wrong window.
+ let _mutation_lease = if !delivery_mode.is_foreground() {
+ let gate_action = if button_str == "middle" {
+ cua_driver_core::background_input::BackgroundAction::WindowPointer
+ } else {
+ cua_driver_core::background_input::BackgroundAction::AxSemantic
+ };
+ match super::gate_background_window_action(pid, wid, Some(element_ptr), gate_action)
+ .await
+ {
+ Ok(lease) => Some(lease),
+ Err(refusal_result) => return refusal_result,
+ }
+ } else {
+ None
+ };
+
// Surface 5: button=right on the AX path → AXShowMenu (the same surface
// the dedicated `right_click` tool dispatches). Threads through the
// identical perform_ax_click code path with the action remapped.
@@ -541,7 +563,7 @@ impl Tool for ClickTool {
} else {
false
};
- let selection_pixel = if selection_candidate {
+ let mut selection_pixel = if selection_candidate {
if let Some((cx, cy)) = center {
super::px_frame::resolve_or_refuse(wid)
.await
@@ -558,6 +580,26 @@ impl Tool for ClickTool {
} else {
None
};
+ // The selection fallback delivers a routed window-local pixel
+ // click — a stricter (WindowPointer) rung than the semantic gate
+ // above. In background, drop the fallback rather than silently
+ // escalate when the pointer rung would refuse (e.g. a
+ // minimized/hidden target); the semantic path still runs.
+ if selection_pixel.is_some()
+ && !delivery_mode.is_foreground()
+ && _mutation_lease
+ .as_ref()
+ .expect("background element actions hold the per-pid lease")
+ .gate_again(
+ wid,
+ Some(element_ptr),
+ cua_driver_core::background_input::BackgroundAction::WindowPointer,
+ )
+ .await
+ .is_err()
+ {
+ selection_pixel = None;
+ }
// ── Focus-suppression wrap (Swift WindowChangeDetector + FocusGuard) ──
// Capture prior frontmost, arm the wildcard suppressor in the
@@ -787,7 +829,29 @@ impl Tool for ClickTool {
// CGEventSetWindowLocation in the Chromium recipe.
let (screen_x, screen_y, win_local_x, win_local_y) = if let Some(wid) = window_id {
match super::px_frame::resolve_or_refuse(wid).await {
- Ok(frame) => frame.to_screen(cx, cy),
+ Ok(frame) => {
+ let (sx, sy, lx, ly) = frame.to_screen(cx, cy);
+ // A window-local point outside the live frame would
+ // dispatch onto whatever occupies that screen point —
+ // the same wrong-surface misclick class as #2237.
+ // Refuse in background, where the caller cannot see
+ // what is actually under the translated point.
+ if !delivery_mode.is_foreground()
+ && (lx < 0.0
+ || ly < 0.0
+ || lx > frame.bounds.width
+ || ly > frame.bounds.height)
+ {
+ return ToolResult::error(format!(
+ "click: window-local point ({lx:.1}, {ly:.1}) pt lies outside \
+ window {wid}'s {:.0}×{:.0} pt frame; background delivery \
+ refused. Re-read coordinates from a fresh get_window_state \
+ screenshot.",
+ frame.bounds.width, frame.bounds.height
+ ));
+ }
+ (sx, sy, lx, ly)
+ }
Err(refusal) => return refusal,
}
} else {
@@ -795,6 +859,34 @@ impl Tool for ClickTool {
(cx, cy, cx, cy)
};
+ // ── Exact-target background gate (macOS background input v1) ──
+ // A window-addressed background pixel action targets coordinates,
+ // which only mean something while the exact window is current and
+ // not minimized/hidden: a stale target would let the pid-scoped
+ // hit-test or routed events land on a same-process sibling. Gate
+ // BEFORE the AX hit-test backend and any cursor/dispatch work.
+ // delivery_mode:"foreground" stays the explicit last resort.
+ let mutation_lease_held = crate::background_mutation::held_by_current_task(pid);
+ let _mutation_lease = if !delivery_mode.is_foreground() && !mutation_lease_held {
+ if let Some(wid) = window_id {
+ match super::gate_background_window_action(
+ pid,
+ wid,
+ None,
+ cua_driver_core::background_input::BackgroundAction::WindowPointer,
+ )
+ .await
+ {
+ Ok(lease) => Some(lease),
+ Err(refusal_result) => return refusal_result,
+ }
+ } else {
+ None
+ }
+ } else {
+ None
+ };
+
// A background PX action can still use an accessibility delivery
// backend after resolving the requested screen point. This keeps
// targeting (PX) orthogonal to delivery (AX) and avoids making a
@@ -806,10 +898,20 @@ impl Tool for ClickTool {
&& modifiers.is_empty()
{
let focus_only = action == "focus";
+ let hit_test_wid = window_id.expect("guarded by window_id.is_some() above");
let ax_result = tokio::task::spawn_blocking(move || unsafe {
let Some(element) = element_at_screen_position(pid, screen_x, screen_y) else {
return Ok::<bool, anyhow::Error>(false);
};
+ // The pid-scoped hit-test can resolve an element from a
+ // same-process sibling overlapping the requested point.
+ // Require proven ancestry in the requested window before
+ // acting; otherwise fall through to the routed pixel path
+ // (already gated for this exact window).
+ if crate::ax::exact_target::element_window_id(element) != Some(hit_test_wid) {
+ CFRelease(element as _);
+ return Ok(false);
+ }
let delivered = if focus_only {
crate::input::ax_actions::focus_element(element as usize).is_ok()
} else {
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs
index b47de84c48..26b03e06e8 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs
@@ -16,6 +16,16 @@ pub struct DoubleClickTool {
state: Arc<ToolState>,
}
+fn background_action_for_element(
+ has_ax_open: bool,
+) -> cua_driver_core::background_input::BackgroundAction {
+ if has_ax_open {
+ cua_driver_core::background_input::BackgroundAction::AxSemantic
+ } else {
+ cua_driver_core::background_input::BackgroundAction::WindowPointer
+ }
+}
+
impl DoubleClickTool {
pub fn new(state: Arc<ToolState>) -> Self {
Self { state }
@@ -113,11 +123,42 @@ impl Tool for DoubleClickTool {
};
let element_ptr = element_guard.as_ptr();
+ // Choose one background actuator before dispatch. An element that
+ // advertises AXOpen uses the exact semantic route; all other
+ // elements require the stricter routed-pointer proof. Do not let a
+ // failed AXOpen silently cross into an ungated pointer fallback.
+ let has_ax_open = tokio::task::spawn_blocking(move || unsafe {
+ copy_action_names(element_ptr as AXUIElementRef)
+ .iter()
+ .any(|action| action == "AXOpen")
+ })
+ .await
+ .unwrap_or(false);
+ let _mutation_lease = if !delivery_mode.is_foreground() {
+ let action = background_action_for_element(has_ax_open);
+ match super::gate_background_window_action(pid, wid, Some(element_ptr), action)
+ .await
+ {
+ Ok(lease) => Some(lease),
+ Err(refusal_result) => return refusal_result,
+ }
+ } else {
+ None
+ };
+
// Thread the resolved session cursor key into the blocking AX path
// so its ClickPulse lands on THIS session's cursor, not "default".
let ck = cursor_key.clone();
let result = tokio::task::spawn_blocking(move || {
- ax_double_click(pid, wid, element_ptr, idx, &ck)
+ ax_double_click(
+ pid,
+ wid,
+ element_ptr,
+ idx,
+ &ck,
+ has_ax_open,
+ delivery_mode.is_foreground(),
+ )
})
.await;
@@ -154,13 +195,48 @@ impl Tool for DoubleClickTool {
// local point as screen-absolute).
let (screen_x, screen_y, win_local_x, win_local_y) = if let Some(wid) = window_id {
match super::px_frame::resolve_or_refuse(wid).await {
- Ok(frame) => frame.to_screen(cx, cy),
+ Ok(frame) => {
+ let translated = frame.to_screen(cx, cy);
+ if !delivery_mode.is_foreground()
+ && (translated.2 < 0.0
+ || translated.3 < 0.0
+ || translated.2 > frame.bounds.width
+ || translated.3 > frame.bounds.height)
+ {
+ return ToolResult::error(format!(
+ "double_click: window-local point ({:.1}, {:.1}) pt lies outside \
+ window {wid}'s {:.0}×{:.0} pt frame; background delivery refused",
+ translated.2, translated.3, frame.bounds.width, frame.bounds.height
+ ));
+ }
+ translated
+ }
Err(refusal) => return refusal,
}
} else {
(cx, cy, cx, cy)
};
+ let _mutation_lease = if !delivery_mode.is_foreground() {
+ if let Some(wid) = window_id {
+ match super::gate_background_window_action(
+ pid,
+ wid,
+ None,
+ cua_driver_core::background_input::BackgroundAction::WindowPointer,
+ )
+ .await
+ {
+ Ok(lease) => Some(lease),
+ Err(refusal_result) => return refusal_result,
+ }
+ } else {
+ None
+ }
+ } else {
+ None
+ };
+
// Pin overlay above the target window before animating.
if let Some(wid) = window_id {
crate::cursor::overlay::send_command(
@@ -235,16 +311,23 @@ fn ax_double_click(
element_ptr: usize,
idx: usize,
cursor_key: &str,
+ has_ax_open: bool,
+ allow_pointer_fallback: bool,
) -> anyhow::Result<String> {
let element = element_ptr as AXUIElementRef;
// Try AXOpen first (Finder items, openable list rows, document cells).
- let actions = unsafe { copy_action_names(element) };
- if actions.iter().any(|a| a == "AXOpen") {
+ if has_ax_open {
let err = unsafe { perform_action(element, "AXOpen") };
if err == kAXErrorSuccess {
return Ok(format!("AXOpen performed on element [{idx}]."));
}
+ if !allow_pointer_fallback {
+ anyhow::bail!(
+ "AXOpen returned {err} for element [{idx}]; background delivery will not \
+ improvise a pointer fallback after choosing the semantic route"
+ );
+ }
tracing::debug!(
"AXOpen returned {err} for element [{idx}], falling back to pixel double-click"
);
@@ -281,3 +364,20 @@ fn ax_double_click(
"✅ Double-clicked element [{idx}] at ({cx:.1}, {cy:.1})."
))
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn background_element_route_does_not_guess_across_actuator_classes() {
+ assert_eq!(
+ background_action_for_element(true),
+ cua_driver_core::background_input::BackgroundAction::AxSemantic
+ );
+ assert_eq!(
+ background_action_for_element(false),
+ cua_driver_core::background_input::BackgroundAction::WindowPointer
+ );
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
index dc7231bbc1..61a0fe2bbc 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
@@ -584,13 +584,35 @@ impl Tool for GetWindowStateTool {
action."
));
structured["escalation"] = serde_json::json!({
- "recommended": "px",
- "reason": "act by pixel (x,y) off the screenshot in this response — \
- the frame IS the requested window even though its AX \
- surface is unresolved."
+ "recommended": "foreground",
+ "reason": "observation-only: the screenshot in this response IS the \
+ requested window, but background input (including px) is \
+ refused while its AX surface is unresolved — events could \
+ reach a same-process sibling window. Re-snapshot after the \
+ app settles, or act with delivery_mode:\"foreground\"."
});
}
}
+ // Additive read-only `background_input` capability section (macOS
+ // background input v1): the same fresh facts that gate every
+ // background mutation, reported per route so an agent can choose
+ // before acting. Every action still revalidates — this is advisory,
+ // not a promise. Old consumers ignore the extra field.
+ {
+ let capture_available = screenshot_dims.is_some();
+ let report = tokio::task::spawn_blocking(move || {
+ let facts = crate::ax::exact_target::gather_background_facts(pid, window_id, None);
+ cua_driver_core::background_input::background_input_capability_report(
+ cua_driver_core::background_input::ExactWindowTarget { pid, window_id },
+ &facts,
+ Some(capture_available),
+ )
+ })
+ .await;
+ if let Ok(report) = report {
+ structured["background_input"] = report;
+ }
+ }
if let Some((sw, sh)) = screenshot_dims {
structured["screenshot_width"] = serde_json::json!(sw);
structured["screenshot_height"] = serde_json::json!(sh);
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs
index ccaa6f8243..b2e8438f89 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs
@@ -226,6 +226,33 @@ impl Tool for HotkeyTool {
return error;
}
+ // ── Exact-target background gate (macOS background input v1) ──
+ // A window-addressed background combo is process-scoped transport: it
+ // must prove exact delivery to the requested window (fresh AXWindows
+ // membership, not minimized/hidden, no competing same-pid keyboard
+ // destination) BEFORE anything is sent — including the px focus
+ // click. delivery_mode:"foreground" stays the caller's explicit last
+ // resort and is not gated here.
+ let _mutation_lease = if !fg {
+ if let Some(wid) = window_id {
+ match super::gate_background_window_action(
+ pid,
+ wid,
+ None,
+ cua_driver_core::background_input::BackgroundAction::GenericKey,
+ )
+ .await
+ {
+ Ok(lease) => Some(lease),
+ Err(refusal_result) => return refusal_result,
+ }
+ } else {
+ None
+ }
+ } else {
+ None
+ };
+
// px form: focus the field before sending the combo. Foreground delivery
// still needs to front the target for the chord itself: the focus helper
// restores the previous app before returning.
@@ -247,6 +274,7 @@ impl Tool for HotkeyTool {
args.opt_str("session"),
args.opt_str("_session_id"),
from_zoom,
+ _mutation_lease.as_ref(),
)
.await
{
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs
index ba22178f91..7839f01daa 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs
@@ -107,6 +107,9 @@ mod pid_window_target_tests {
}
}
+#[cfg(test)]
+mod background_input_regression_tests;
+
fn pid_window_guarded<T: Tool + 'static>(
tool: T,
candidates: &WindowTargetCandidates,
@@ -177,6 +180,106 @@ impl DeliveryMode {
}
}
+/// Convert a pure background-input refusal into the structured refusal result
+/// shape shared by exact-target tools: `code`, `effect: "refused"`, the
+/// requested target, and the safe next route when one exists. No actuator ran.
+pub(crate) fn background_refusal_result(
+ pid: i32,
+ window_id: u32,
+ refusal: &cua_driver_core::background_input::BackgroundRefusal,
+) -> cua_driver_core::protocol::ToolResult {
+ let mut structured = serde_json::json!({
+ "code": refusal.code,
+ "effect": "refused",
+ "pid": pid,
+ "window_id": window_id,
+ "reason": refusal.reason,
+ });
+ if let Some(advice) = refusal.advice {
+ structured["escalation"] = serde_json::json!({
+ "recommended": advice,
+ "reason": refusal.reason,
+ });
+ }
+ cua_driver_core::protocol::ToolResult::error(format!(
+ "Background input refused ({}): {}",
+ refusal.code, refusal.reason
+ ))
+ .with_structured(structured)
+}
+
+/// Exclusive per-process ownership of one background mutation. Callers must
+/// keep this value alive through actuator dispatch, focus restoration, and
+/// target-bound verification.
+pub(crate) struct BackgroundMutationLease {
+ pid: i32,
+ _guard: tokio::sync::OwnedMutexGuard<()>,
+}
+
+impl BackgroundMutationLease {
+ /// Revalidate another actuator class while retaining the same per-PID
+ /// lease. This supports explicit ladders without deadlocking by attempting
+ /// to reacquire the coordinator recursively.
+ pub(crate) async fn gate_again(
+ &self,
+ window_id: u32,
+ element_ptr: Option<usize>,
+ action: cua_driver_core::background_input::BackgroundAction,
+ ) -> Result<(), cua_driver_core::protocol::ToolResult> {
+ decide_background_window_action(self.pid, window_id, element_ptr, action).await
+ }
+}
+
+async fn decide_background_window_action(
+ pid: i32,
+ window_id: u32,
+ element_ptr: Option<usize>,
+ action: cua_driver_core::background_input::BackgroundAction,
+) -> Result<(), cua_driver_core::protocol::ToolResult> {
+ use cua_driver_core::background_input::{
+ decide_background_input, BackgroundInputDecision, ExactWindowTarget,
+ };
+ let facts = match tokio::task::spawn_blocking(move || {
+ crate::ax::exact_target::gather_background_facts(pid, window_id, element_ptr)
+ })
+ .await
+ {
+ Ok(facts) => facts,
+ Err(error) => {
+ return Err(cua_driver_core::protocol::ToolResult::error(format!(
+ "Could not gather exact-target facts for pid {pid} window {window_id}: {error}"
+ )));
+ }
+ };
+ match decide_background_input(ExactWindowTarget { pid, window_id }, &facts, action) {
+ BackgroundInputDecision::Execute { .. } => Ok(()),
+ BackgroundInputDecision::Refuse(refusal) => {
+ Err(background_refusal_result(pid, window_id, &refusal))
+ }
+ }
+}
+
+/// Acquire the per-PID mutation coordinator, gather fresh exact-target facts,
+/// and ask the pure core for one background decision. `element_ptr` must stay
+/// retained by the caller until the returned lease is dropped.
+pub(crate) async fn gate_background_window_action(
+ pid: i32,
+ window_id: u32,
+ element_ptr: Option<usize>,
+ action: cua_driver_core::background_input::BackgroundAction,
+) -> Result<BackgroundMutationLease, cua_driver_core::protocol::ToolResult> {
+ let lease = acquire_background_mutation(pid).await;
+ lease.gate_again(window_id, element_ptr, action).await?;
+ Ok(lease)
+}
+
+pub(crate) async fn acquire_background_mutation(pid: i32) -> BackgroundMutationLease {
+ BackgroundMutationLease {
+ pid,
+ _guard: crate::background_mutation::acquire(pid).await,
+ }
+}
+
/// Finish the post-action observation window. Embedded interactive clients
/// that already observe the target continuously may opt out through the
/// private registry argument to avoid adding a one-second acknowledgement
@@ -230,6 +333,7 @@ pub(crate) async fn focus_by_pixel(
session: Option<String>,
session_id: Option<String>,
from_zoom: bool,
+ mutation_lease: Option<&BackgroundMutationLease>,
) -> Result<(), cua_driver_core::protocol::ToolResult> {
use cua_driver_core::tool::Tool;
let mut click_args = serde_json::json!({
@@ -239,6 +343,15 @@ pub(crate) async fn focus_by_pixel(
});
if let Some(wid) = window_id {
click_args["window_id"] = serde_json::json!(wid);
+ if let Some(lease) = mutation_lease {
+ lease
+ .gate_again(
+ wid,
+ None,
+ cua_driver_core::background_input::BackgroundAction::WindowPointer,
+ )
+ .await?;
+ }
}
if let Some(ref s) = session {
click_args["session"] = serde_json::json!(s);
@@ -249,9 +362,13 @@ pub(crate) async fn focus_by_pixel(
if from_zoom {
click_args["from_zoom"] = serde_json::json!(true);
}
- let focus = click::ClickTool::new(state.clone())
- .invoke(click_args)
- .await;
+ let click_tool = click::ClickTool::new(state.clone());
+ let click = click_tool.invoke(click_args);
+ let focus = if let Some(lease) = mutation_lease {
+ crate::background_mutation::with_held_lease(lease.pid, click).await
+ } else {
+ click.await
+ };
if focus.is_error != Some(true) {
// AXFocused is non-destructive: unlike a second real click, it keeps a
// Cmd+A selection intact before a follow-up type_text or Cmd+V.
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs
index d1ad65c245..228ff8848f 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs
@@ -76,8 +76,8 @@ fn validate_post_target(pid: i32) -> anyhow::Result<()> {
}
}
-fn read_ax_key_state(pid: i32, element_ptr: usize) -> Option<AxKeyState> {
- if super::type_text::target_in_web_area(pid, Some((element_ptr, None))) {
+fn read_ax_key_state(pid: i32, window_id: Option<u32>, element_ptr: usize) -> Option<AxKeyState> {
+ if super::type_text::target_in_web_area(pid, Some((element_ptr, None)), window_id) {
return None;
}
let element = element_ptr as AXUIElementRef;
@@ -90,16 +90,22 @@ fn read_ax_key_state(pid: i32, element_ptr: usize) -> Option<AxKeyState> {
fn dispatch_with_ax_oracle(
pid: i32,
+ window_id: Option<u32>,
explicit_element_ptr: Option<usize>,
dispatch: impl FnOnce() -> anyhow::Result<()>,
) -> anyhow::Result<bool> {
let (element_ptr, owns_element) = match explicit_element_ptr {
Some(ptr) => (Some(ptr), false),
- None => unsafe { focused_element_of_pid(pid) }
- .map(|element| (Some(element as usize), true))
- .unwrap_or((None, false)),
+ None => unsafe {
+ match window_id {
+ Some(wid) => crate::ax::exact_target::focused_element_in_window(pid, wid),
+ None => focused_element_of_pid(pid),
+ }
+ }
+ .map(|element| (Some(element as usize), true))
+ .unwrap_or((None, false)),
};
- let before = element_ptr.and_then(|ptr| read_ax_key_state(pid, ptr));
+ let before = element_ptr.and_then(|ptr| read_ax_key_state(pid, window_id, ptr));
let result = dispatch();
// Native controls normally publish their new value/selection on the next
// run-loop turn. Keep this bounded and reuse the exact retained element so
@@ -107,7 +113,7 @@ fn dispatch_with_ax_oracle(
if before.is_some() {
std::thread::sleep(std::time::Duration::from_millis(60));
}
- let after = element_ptr.and_then(|ptr| read_ax_key_state(pid, ptr));
+ let after = element_ptr.and_then(|ptr| read_ax_key_state(pid, window_id, ptr));
if owns_element {
if let Some(ptr) = element_ptr {
unsafe { CFRelease(ptr as _) };
@@ -308,18 +314,69 @@ impl Tool for PressKeyTool {
let delivery_mode = super::DeliveryMode::parse(args.opt_str("delivery_mode").as_deref());
let fg = delivery_mode.is_foreground();
+ // Argument-shape errors are reported before any gating or retained
+ // lookups: a malformed call must fail the same way regardless of
+ // background-target state.
+ let px = args.get("x").and_then(|v| v.as_f64());
+ let py = args.get("y").and_then(|v| v.as_f64());
+ if px.is_some() && py.is_some() && element_index.is_some() {
+ return ToolResult::error(
+ "Pass either element_index (ax) or x,y (px) to press_key, not both.",
+ );
+ }
+
+ // Resolve the pre-focus element pointer (if requested) outside
+ // the suppression closure — only the focus_element() write itself
+ // needs to run under suppression, the cache lookup does not.
+ // Retain out of the cache so a concurrent get_window_state can't free
+ // the element before the suppressed focus below dereferences it
+ // (use-after-free → daemon crash). Guard lives to method end.
+ let pre_focus_guard = if let (Some(idx), Some(wid)) = (element_index, window_id) {
+ match self.state.element_cache.get_element_retained(pid, wid, idx) {
+ Some(guard) => Some(guard),
+ None => {
+ return ToolResult::error(format!(
+ "Element index {idx} not found. Call get_window_state first."
+ ));
+ }
+ }
+ } else {
+ None
+ };
+ let pre_focus_ptr: Option<usize> = pre_focus_guard.as_ref().map(|g| g.as_ptr());
+
+ // ── Exact-target background gate (macOS background input v1) ──
+ // A window-addressed background key is process-scoped transport: it
+ // must prove exact delivery to the requested window (fresh AXWindows
+ // membership, not minimized/hidden, no competing same-pid keyboard
+ // destination, proven element ancestry) BEFORE anything is sent —
+ // including the px focus click. delivery_mode:"foreground" stays the
+ // caller's explicit last resort and is not gated here.
+ let _mutation_lease = if !fg {
+ if let Some(wid) = window_id {
+ match super::gate_background_window_action(
+ pid,
+ wid,
+ pre_focus_ptr,
+ cua_driver_core::background_input::BackgroundAction::GenericKey,
+ )
+ .await
+ {
+ Ok(lease) => Some(lease),
+ Err(refusal_result) => return refusal_result,
+ }
+ } else {
+ None
+ }
+ } else {
+ None
+ };
+
// px form: pixel-click to focus, then the key goes to the focused element.
// Reuses click's translation + delivery_mode; after it, deliver via the
// plain background path (the focus-click already handled fronting if fg).
let px_focus = {
- let px = args.get("x").and_then(|v| v.as_f64());
- let py = args.get("y").and_then(|v| v.as_f64());
if let (Some(cx), Some(cy)) = (px, py) {
- if element_index.is_some() {
- return ToolResult::error(
- "Pass either element_index (ax) or x,y (px) to press_key, not both.",
- );
- }
let from_zoom = args
.get("from_zoom")
.and_then(|v| v.as_bool())
@@ -334,6 +391,7 @@ impl Tool for PressKeyTool {
args.opt_str("session"),
args.opt_str("_session_id"),
from_zoom,
+ _mutation_lease.as_ref(),
)
.await
{
@@ -345,19 +403,6 @@ impl Tool for PressKeyTool {
}
};
- // Resolve the pre-focus element pointer (if requested) outside
- // the suppression closure — only the focus_element() write itself
- // needs to run under suppression, the cache lookup does not.
- // Retain out of the cache so a concurrent get_window_state can't free
- // the element before the suppressed focus below dereferences it
- // (use-after-free → daemon crash). Guard lives to method end.
- let pre_focus_guard = if let (Some(idx), Some(wid)) = (element_index, window_id) {
- self.state.element_cache.get_element_retained(pid, wid, idx)
- } else {
- None
- };
- let pre_focus_ptr: Option<usize> = pre_focus_guard.as_ref().map(|g| g.as_ptr());
-
// ── Focus-suppression wrap (Swift WindowChangeDetector + FocusGuard) ──
// Single-key presses can fire autocomplete (Return on a search
// box opens a results popover) or trigger menu shortcuts that
@@ -398,7 +443,7 @@ impl Tool for PressKeyTool {
"delivery_mode=foreground requires window_id for press_key"
)
})?;
- return dispatch_with_ax_oracle(pid, pre_focus_ptr, || {
+ return dispatch_with_ax_oracle(pid, window_id, pre_focus_ptr, || {
crate::input::skylight::with_foreground_hid_activation(
pid as libc::pid_t,
wid,
@@ -417,7 +462,7 @@ impl Tool for PressKeyTool {
});
}
// background (default): auth-envelope post, no raise.
- dispatch_with_ax_oracle(pid, pre_focus_ptr, || {
+ dispatch_with_ax_oracle(pid, window_id, pre_focus_ptr, || {
crate::input::keyboard::press_key(pid, &key, &m)
})
})
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs
index 28787bbaea..c32d343013 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs
@@ -153,6 +153,18 @@ impl Tool for RightClickTool {
};
let element_ptr = element_guard.as_ptr();
+ let _mutation_lease = match super::gate_background_window_action(
+ pid,
+ wid,
+ Some(element_ptr),
+ cua_driver_core::background_input::BackgroundAction::AxSemantic,
+ )
+ .await
+ {
+ Ok(lease) => lease,
+ Err(refusal_result) => return refusal_result,
+ };
+
let result =
tokio::task::spawn_blocking(move || ax_show_menu(element_ptr, idx, pid, wid)).await;
@@ -176,13 +188,48 @@ impl Tool for RightClickTool {
// refuses a window with no live frame).
let (screen_x, screen_y, win_local_x, win_local_y) = if let Some(wid) = window_id {
match super::px_frame::resolve_or_refuse(wid).await {
- Ok(frame) => frame.to_screen(cx, cy),
+ Ok(frame) => {
+ let translated = frame.to_screen(cx, cy);
+ if !delivery_mode.is_foreground()
+ && (translated.2 < 0.0
+ || translated.3 < 0.0
+ || translated.2 > frame.bounds.width
+ || translated.3 > frame.bounds.height)
+ {
+ return ToolResult::error(format!(
+ "right_click: window-local point ({:.1}, {:.1}) pt lies outside \
+ window {wid}'s {:.0}×{:.0} pt frame; background delivery refused",
+ translated.2, translated.3, frame.bounds.width, frame.bounds.height
+ ));
+ }
+ translated
+ }
Err(refusal) => return refusal,
}
} else {
(cx, cy, cx, cy)
};
+ let _mutation_lease = if !delivery_mode.is_foreground() {
+ if let Some(wid) = window_id {
+ match super::gate_background_window_action(
+ pid,
+ wid,
+ None,
+ cua_driver_core::background_input::BackgroundAction::WindowPointer,
+ )
+ .await
+ {
+ Ok(lease) => Some(lease),
+ Err(refusal_result) => return refusal_result,
+ }
+ } else {
+ None
+ }
+ } else {
+ None
+ };
+
// Pin overlay above the target window before animating.
if let Some(wid) = window_id {
crate::cursor::overlay::send_command(
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs
index 0930888257..c788ee7af0 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs
@@ -34,6 +34,14 @@ struct WheelTarget {
wid: Option<u32>,
}
+fn after_exact_target_gate<T>(
+ gate: Result<(), ToolResult>,
+ action: impl FnOnce() -> T,
+) -> Result<T, ToolResult> {
+ gate?;
+ Ok(action())
+}
+
pub struct ScrollTool {
state: Arc<ToolState>,
}
@@ -226,12 +234,39 @@ impl Tool for ScrollTool {
));
}
}
+ let mut _mutation_lease: Option<super::BackgroundMutationLease> = None;
// AppKit exposes vertical scroll-bar buttons beneath the text area's
// AXScrollArea parent. Pressing those controls is a true
// background-safe scroll: no activation, z-order change, or cursor move.
if matches!(direction.as_str(), "up" | "down") {
if let (Some(index), Some(wid)) = (element_index, window_id) {
+ if !delivery_mode.is_foreground() {
+ if let Some(lease) = _mutation_lease.as_ref() {
+ if let Err(refusal_result) = lease
+ .gate_again(
+ wid,
+ pre_focus_ptr,
+ cua_driver_core::background_input::BackgroundAction::AxSemantic,
+ )
+ .await
+ {
+ return refusal_result;
+ }
+ } else {
+ match super::gate_background_window_action(
+ pid,
+ wid,
+ pre_focus_ptr,
+ cua_driver_core::background_input::BackgroundAction::AxSemantic,
+ )
+ .await
+ {
+ Ok(lease) => _mutation_lease = Some(lease),
+ Err(refusal_result) => return refusal_result,
+ }
+ }
+ }
let native_element_guard = self
.state
.element_cache
@@ -332,25 +367,61 @@ impl Tool for ScrollTool {
// Resolve a screen-space wheel target, if a target was supplied.
let wheel_target: Option<WheelTarget> = if let Some(element_ptr) = pre_focus_ptr {
+ // Revealing an element is itself an AX mutation. Prove that the
+ // cached element still belongs to the exact requested window before
+ // AXScrollToVisible for every direction, then keep the lease for the
+ // stricter pointer revalidation below.
+ let semantic_gate = if !delivery_mode.is_foreground() {
+ if let Some(wid) = window_id {
+ if let Some(lease) = _mutation_lease.as_ref() {
+ lease
+ .gate_again(
+ wid,
+ Some(element_ptr),
+ cua_driver_core::background_input::BackgroundAction::AxSemantic,
+ )
+ .await
+ } else {
+ match super::gate_background_window_action(
+ pid,
+ wid,
+ Some(element_ptr),
+ cua_driver_core::background_input::BackgroundAction::AxSemantic,
+ )
+ .await
+ {
+ Ok(lease) => {
+ _mutation_lease = Some(lease);
+ Ok(())
+ }
+ Err(refusal) => Err(refusal),
+ }
+ }
+ } else {
+ Ok(())
+ }
+ } else {
+ Ok(())
+ };
// Element path: wheel at the element's screen-space center. Both AX
// coordinates and window bounds are logical top-left points, so no
// Retina scaling is needed here.
let wid = window_id;
- tokio::task::spawn_blocking(move || {
+ let target_task = tokio::task::spawn_blocking(move || {
// Web content can be present in AX while its frame is below
// the outer page viewport. Ask the accessibility hierarchy to
// reveal the target before taking the screen-space center;
// otherwise the wheel is posted outside the rendered window
// and nested overflow regions never receive it.
- unsafe {
- let _ = crate::ax::bindings::perform_action(
+ after_exact_target_gate(semantic_gate, || unsafe {
+ crate::ax::bindings::perform_action(
element_ptr as AXUIElementRef,
"AXScrollToVisible",
- );
- }
+ )
+ })?;
std::thread::sleep(std::time::Duration::from_millis(40));
let center = unsafe { element_screen_center(element_ptr as AXUIElementRef) };
- center.map(|(cx, cy)| {
+ Ok(center.map(|(cx, cy)| {
let win_local = wid
.and_then(crate::windows::window_bounds_by_id)
.map(|b| (cx - b.x, cy - b.y));
@@ -360,11 +431,13 @@ impl Tool for ScrollTool {
win_local,
wid,
}
- })
- })
- .await
- .ok()
- .flatten()
+ }))
+ });
+ match target_task.await {
+ Ok(Ok(target)) => target,
+ Ok(Err(refusal)) => return refusal,
+ Err(_) => None,
+ }
} else if let (Some(mut cx), Some(mut cy)) = (x_arg, y_arg) {
// Targeted x,y are window-local screenshot pixels and REQUIRE a
// window_id to anchor the window→screen conversion (schema contract).
@@ -406,6 +479,49 @@ impl Tool for ScrollTool {
};
if let Some(target) = wheel_target {
+ if !delivery_mode.is_foreground() {
+ if let Some(wid) = target.wid {
+ if let (Some((lx, ly)), Some(bounds)) =
+ (target.win_local, crate::windows::window_bounds_by_id(wid))
+ {
+ if lx < 0.0 || ly < 0.0 || lx > bounds.width || ly > bounds.height {
+ return ToolResult::error(format!(
+ "scroll: window-local point ({lx:.1}, {ly:.1}) pt lies outside \
+ window {wid}'s {:.0}×{:.0} pt frame; background delivery \
+ refused",
+ bounds.width, bounds.height
+ ));
+ }
+ }
+ // The semantic reveal gate does not authorize pointer
+ // delivery. Revalidate the stricter route immediately
+ // before proceeding to wheel dispatch.
+ if let Some(lease) = _mutation_lease.as_ref() {
+ if let Err(refusal_result) = lease
+ .gate_again(
+ wid,
+ pre_focus_ptr,
+ cua_driver_core::background_input::BackgroundAction::WindowPointer,
+ )
+ .await
+ {
+ return refusal_result;
+ }
+ } else {
+ match super::gate_background_window_action(
+ pid,
+ wid,
+ pre_focus_ptr,
+ cua_driver_core::background_input::BackgroundAction::WindowPointer,
+ )
+ .await
+ {
+ Ok(lease) => _mutation_lease = Some(lease),
+ Err(refusal_result) => return refusal_result,
+ }
+ }
+ }
+ }
let cursor_key = super::cursor_tools::resolve_cursor_key(&args);
// Pin + glide the agent-cursor overlay to the target for visibility
// (overlay only — does NOT move the hardware cursor). Mirrors click.
@@ -506,6 +622,35 @@ impl Tool for ScrollTool {
};
let key = key.to_owned();
+ if !delivery_mode.is_foreground() {
+ if let Some(wid) = window_id {
+ if let Some(lease) = _mutation_lease.as_ref() {
+ if let Err(refusal_result) = lease
+ .gate_again(
+ wid,
+ pre_focus_ptr,
+ cua_driver_core::background_input::BackgroundAction::GenericKey,
+ )
+ .await
+ {
+ return refusal_result;
+ }
+ } else {
+ match super::gate_background_window_action(
+ pid,
+ wid,
+ pre_focus_ptr,
+ cua_driver_core::background_input::BackgroundAction::GenericKey,
+ )
+ .await
+ {
+ Ok(lease) => _mutation_lease = Some(lease),
+ Err(refusal_result) => return refusal_result,
+ }
+ }
+ }
+ }
+
// ── Focus-suppression wrap (Swift WindowChangeDetector + FocusGuard) ──
// Scroll keystrokes (PageDown / arrow) into search-box autocomplete
// can spawn floating helper windows; rare but real. Wrap for parity
@@ -624,3 +769,43 @@ unsafe fn collect_ax_buttons(
}
}
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use cua_driver_core::background_input::{
+ decide_background_input, BackgroundAction, BackgroundInputDecision, BackgroundTargetFacts,
+ ElementAncestry, ExactWindowTarget, WindowServerOwnership,
+ };
+ use std::sync::atomic::{AtomicBool, Ordering};
+
+ #[test]
+ fn exact_target_refusal_prevents_ax_reveal() {
+ let action_ran = AtomicBool::new(false);
+ let target = ExactWindowTarget {
+ pid: 42,
+ window_id: 7,
+ };
+ let facts = BackgroundTargetFacts {
+ window_server: WindowServerOwnership::SamePid,
+ ax_window_present: true,
+ target_minimized: Some(false),
+ app_hidden: Some(false),
+ competing_keyboard_destinations: 0,
+ element: ElementAncestry::OutsideTargetWindow,
+ };
+ let refusal = match decide_background_input(target, &facts, BackgroundAction::AxSemantic) {
+ BackgroundInputDecision::Refuse(refusal) => Err(
+ super::super::background_refusal_result(target.pid, target.window_id, &refusal),
+ ),
+ BackgroundInputDecision::Execute { .. } => panic!("exact-target facts must refuse"),
+ };
+
+ let result = after_exact_target_gate(refusal, || {
+ action_ran.store(true, Ordering::SeqCst);
+ });
+
+ assert!(result.is_err());
+ assert!(!action_ran.load(Ordering::SeqCst));
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs
index 5d8c8595cb..340b15fd35 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs
@@ -160,6 +160,23 @@ impl Tool for SetValueTool {
}
};
let element_ptr = element_guard.as_ptr();
+
+ // set_value is an always-background semantic AX mutation. Re-prove
+ // that the retained element still belongs to the requested exact
+ // window immediately before any cursor or AX work; a cache hit alone
+ // is not delivery proof after a window lifecycle or Space change.
+ let _mutation_lease = match super::gate_background_window_action(
+ pid,
+ window_id,
+ Some(element_ptr),
+ cua_driver_core::background_input::BackgroundAction::AxSemantic,
+ )
+ .await
+ {
+ Ok(lease) => lease,
+ Err(refusal_result) => return refusal_result,
+ };
+
let cursor_key = super::cursor_tools::resolve_cursor_key(&args);
let center_ptr = element_ptr as usize;
if let Ok(Some((screen_x, screen_y))) = tokio::task::spawn_blocking(move || unsafe {
@@ -181,8 +198,11 @@ impl Tool for SetValueTool {
// the renderer never observes it. Reuse type_text's bounded ancestor
// check so native browser chrome stays trusted but rendered content is
// always reported as unverified.
- let ax_echo_surface =
- super::type_text::target_in_web_area(pid, Some((element_ptr, Some(element_index))));
+ let ax_echo_surface = super::type_text::target_in_web_area(
+ pid,
+ Some((element_ptr, Some(element_index))),
+ Some(window_id),
+ );
// ── Focus-suppression wrap (Swift WindowChangeDetector + FocusGuard) ──
// AXValue writes on popups / sliders can cause reflex activations
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs
index 893e03009e..98fe058c58 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs
@@ -38,6 +38,7 @@ use crate::ax::bindings::{
use crate::focus_guard;
use crate::window_change_detector::WindowChangeDetector;
use core_foundation::base::CFRelease;
+use cua_driver_core::background_input::BackgroundRefusal;
use super::ToolState;
@@ -228,6 +229,59 @@ impl Tool for TypeTextTool {
return error;
}
+ // Validate element_index requires window_id (still applies for
+ // the legacy integer path; token path already resolved window_id).
+ if element_index.is_some() && window_id.is_none() {
+ return ToolResult::error("window_id is required when element_index is used.");
+ }
+
+ // Argument-shape errors are reported before any gating or retained
+ // lookups: a malformed call must fail the same way regardless of
+ // background-target state.
+ let px = args.get("x").and_then(|v| v.as_f64());
+ let py = args.get("y").and_then(|v| v.as_f64());
+ if px.is_some() && py.is_some() && element_index.is_some() {
+ return ToolResult::error(
+ "Pass either element_index (ax) or x,y (px) to type_text, not both.",
+ );
+ }
+
+ // Resolve the element pointer (if element_index given). Retain it out
+ // of the cache so a concurrent get_window_state can't free it before
+ // the blocking type below dereferences it (use-after-free → daemon
+ // crash). The guard lives to method end, past type_text_blocking.
+ let element_guard = if let (Some(idx), Some(wid)) = (element_index, window_id) {
+ match self.state.element_cache.get_element_retained(pid, wid, idx) {
+ Some(e) => Some((e, idx)),
+ None => {
+ return ToolResult::error(format!(
+ "Element index {idx} not found. Call get_window_state first."
+ ))
+ }
+ }
+ } else {
+ None
+ };
+
+ // ── Exact-target background gate (macOS background input v1) ──
+ // A window-addressed background insert must prove exact delivery
+ // before any input — including the px focus click — is sent. The pure
+ // core decides once from fresh facts: full keyboard ladder, semantic
+ // AX write only (exact element, no CGEvent fallback), or a structured
+ // refusal. delivery_mode:"foreground" stays the caller's explicit
+ // last resort and is not gated here.
+ let (_mutation_lease, keyboard_policy) =
+ if !delivery_mode.is_foreground() && window_id.is_some() {
+ let wid = window_id.expect("checked above");
+ let gate_element_ptr = element_guard.as_ref().map(|(g, _)| g.as_ptr() as usize);
+ match background_keyboard_policy(pid, wid, gate_element_ptr).await {
+ Ok((lease, policy)) => (Some(lease), policy),
+ Err(refusal_result) => return refusal_result,
+ }
+ } else {
+ (None, BackgroundKeyboardPolicy::Allowed)
+ };
+
// ── px form: focus by pixel-click, then type into the focused element ──
// Pass x,y (no element_index) for an *element px action*: pixel-click the
// field to give the Chromium/Electron renderer the real keyboard focus the
@@ -235,14 +289,13 @@ impl Tool for TypeTextTool {
// escalates AX → CGEvent and lands once focused). Reuses ClickTool's exact
// coordinate translation + delivery_mode, so it lands on the same pixel a
// px-click would.
- let px = args.get("x").and_then(|v| v.as_f64());
- let py = args.get("y").and_then(|v| v.as_f64());
let used_pixel_focus = px.is_some() && py.is_some();
if let (Some(cx), Some(cy)) = (px, py) {
- if element_index.is_some() {
- return ToolResult::error(
- "Pass either element_index (ax) or x,y (px) to type_text, not both.",
- );
+ // The px form has no exact element for a semantic-only write; when
+ // the keyboard rung is refused, refuse before the focus click too.
+ if let BackgroundKeyboardPolicy::SemanticOnly(ref refusal) = keyboard_policy {
+ let wid = window_id.expect("gate ran only with window_id");
+ return super::background_refusal_result(pid, wid, refusal);
}
let from_zoom = args
.get("from_zoom")
@@ -258,6 +311,7 @@ impl Tool for TypeTextTool {
args.opt_str("session"),
args.opt_str("_session_id"),
from_zoom,
+ _mutation_lease.as_ref(),
)
.await
{
@@ -266,29 +320,6 @@ impl Tool for TypeTextTool {
// element_index stays None → the type path below writes to the now-
// focused element via the CGEvent (key_events) rung.
}
-
- // Validate element_index requires window_id (still applies for
- // the legacy integer path; token path already resolved window_id).
- if element_index.is_some() && window_id.is_none() {
- return ToolResult::error("window_id is required when element_index is used.");
- }
-
- // Resolve the element pointer (if element_index given). Retain it out
- // of the cache so a concurrent get_window_state can't free it before
- // the blocking type below dereferences it (use-after-free → daemon
- // crash). The guard lives to method end, past type_text_blocking.
- let element_guard = if let (Some(idx), Some(wid)) = (element_index, window_id) {
- match self.state.element_cache.get_element_retained(pid, wid, idx) {
- Some(e) => Some((e, idx)),
- None => {
- return ToolResult::error(format!(
- "Element index {idx} not found. Call get_window_state first."
- ))
- }
- }
- } else {
- None
- };
if let (Some((element, _)), Some(wid)) = (element_guard.as_ref(), window_id) {
let center_ptr = element.as_ptr() as usize;
if let Ok(Some((screen_x, screen_y))) = tokio::task::spawn_blocking(move || unsafe {
@@ -330,6 +361,7 @@ impl Tool for TypeTextTool {
// "success but nothing typed" symptom.
let is_terminal_target = crate::terminal::is_terminal_pid(pid);
+ let blocking_policy = keyboard_policy.clone();
let result = focus_guard::with_focus_suppressed(
Some(pid),
prior_front,
@@ -344,6 +376,7 @@ impl Tool for TypeTextTool {
is_terminal_target,
delivery_mode,
window_id,
+ blocking_policy,
)
})
.await
@@ -353,6 +386,18 @@ impl Tool for TypeTextTool {
let changes = super::finish_window_observation(snapshot, &args).await;
+ // Unwrap the delivery envelope: a structured refusal means no
+ // actuator ran and the caller gets the exact reason.
+ let result = match result {
+ Ok(Ok(TypeTextDelivery::Refused(refusal))) => {
+ let wid = window_id.expect("background refusals require a window target");
+ return super::background_refusal_result(pid, wid, &refusal);
+ }
+ Ok(Ok(TypeTextDelivery::Typed(outcome))) => Ok(Ok(outcome)),
+ Ok(Err(error)) => Ok(Err(error)),
+ Err(error) => Err(error),
+ };
+
match result {
Ok(Ok(outcome)) if outcome.delivered_chars.is_some_and(|n| n < char_count) => {
let delivered_chars = outcome.delivered_chars.unwrap_or_default();
@@ -389,7 +434,7 @@ impl Tool for TypeTextTool {
// native types and already-unverified deliveries pay nothing.
let target_is_web_content = verified
&& path_has_untrusted_web_readback(path)
- && target_in_web_area(pid, element_ptr);
+ && target_in_web_area(pid, element_ptr, window_id);
let verification = surface_verification(path, verified, target_is_web_content);
let verified = verification.verified;
let untrusted_web_readback = verification.untrusted_web_readback;
@@ -547,6 +592,73 @@ fn web_readback_next_rung(is_electron: bool, used_pixel_focus: bool) -> Option<&
const DELIVERY_DRAIN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(2);
const DELIVERY_DRAIN_POLL_INTERVAL: std::time::Duration = std::time::Duration::from_millis(10);
+/// Keyboard-rung policy for one window-addressed background insert, decided
+/// once by the pure exact-target core before any input is posted.
+#[derive(Clone, Debug)]
+enum BackgroundKeyboardPolicy {
+ /// The full ladder may run: foreground requests, pid-only targets, and
+ /// window targets whose exact keyboard delivery is proven (singleton
+ /// same-pid destination).
+ Allowed,
+ /// Only the semantic AX write on the proven exact element may run. The
+ /// process-scoped CGEvent rung is refused with this refusal — carried so
+ /// the exact reason is returned if the AX write does not land.
+ SemanticOnly(BackgroundRefusal),
+}
+
+/// Delivery envelope for `type_text_blocking`: either an actuator ran
+/// (`Typed`), or the exact-target decision refused before any event was
+/// posted (`Refused`) and the caller must return the structured refusal.
+enum TypeTextDelivery {
+ Typed(TypeTextOutcome),
+ Refused(BackgroundRefusal),
+}
+
+/// Decide the keyboard policy for a window-addressed background `type_text`.
+///
+/// Gathers fresh exact-target facts once and asks the pure core:
+/// - `InsertText` executes → the full ladder is `Allowed`;
+/// - `InsertText` refused but the caller addressed an exact element whose
+/// ancestry is proven and semantic AX executes → `SemanticOnly`;
+/// - otherwise the structured refusal result is returned and no input of any
+/// kind (including a px focus click) may be sent.
+async fn background_keyboard_policy(
+ pid: i32,
+ window_id: u32,
+ element_ptr: Option<usize>,
+) -> Result<(super::BackgroundMutationLease, BackgroundKeyboardPolicy), ToolResult> {
+ use cua_driver_core::background_input::{
+ decide_background_input, BackgroundAction, BackgroundInputDecision, ExactWindowTarget,
+ };
+ let lease = super::acquire_background_mutation(pid).await;
+ let facts = match tokio::task::spawn_blocking(move || {
+ crate::ax::exact_target::gather_background_facts(pid, window_id, element_ptr)
+ })
+ .await
+ {
+ Ok(facts) => facts,
+ Err(error) => {
+ return Err(ToolResult::error(format!(
+ "Could not gather exact-target facts for pid {pid} window {window_id}: {error}"
+ )));
+ }
+ };
+ let target = ExactWindowTarget { pid, window_id };
+ match decide_background_input(target, &facts, BackgroundAction::InsertText) {
+ BackgroundInputDecision::Execute { .. } => Ok((lease, BackgroundKeyboardPolicy::Allowed)),
+ BackgroundInputDecision::Refuse(refusal) => {
+ let semantic_available = element_ptr.is_some()
+ && decide_background_input(target, &facts, BackgroundAction::AxSemantic)
+ .is_execute();
+ if semantic_available {
+ Ok((lease, BackgroundKeyboardPolicy::SemanticOnly(refusal)))
+ } else {
+ Err(super::background_refusal_result(pid, window_id, &refusal))
+ }
+ }
+ }
+}
+
struct TypeTextOutcome {
detail: String,
path: &'static str,
@@ -633,6 +745,31 @@ fn read_axvalue(pid: i32, element_ptr_and_idx: Option<(usize, Option<usize>)>) -
}
}
+/// Window-bound variant of [`read_axvalue`]: when no explicit element is
+/// addressed and a `window_id` is known, the focused element is used ONLY when
+/// its ancestry provably resolves to that exact window. A sibling window's
+/// focused field must never supply before/after evidence for the requested
+/// target — an unprovable focus reads as `None` (unverifiable), never as
+/// sibling data. Without a window the legacy pid-global read applies.
+fn read_axvalue_bound(
+ pid: i32,
+ element_ptr_and_idx: Option<(usize, Option<usize>)>,
+ window_id: Option<u32>,
+) -> Option<String> {
+ if element_ptr_and_idx.is_some() {
+ return read_axvalue(pid, element_ptr_and_idx);
+ }
+ match window_id {
+ Some(wid) => unsafe {
+ let el = crate::ax::exact_target::focused_element_in_window(pid, wid)?;
+ let v = copy_string_attr(el, "AXValue");
+ CFRelease(el as _);
+ v
+ },
+ None => read_axvalue(pid, None),
+ }
+}
+
/// True when the addressed (or focused) AX element sits inside a web-content
/// subtree — an `AXWebArea` ancestor. That covers every Chromium / WebKit /
/// Electron rendered surface (Chrome, Safari, Slack, VS Code, X's compose box…),
@@ -644,18 +781,30 @@ fn read_axvalue(pid: i32, element_ptr_and_idx: Option<(usize, Option<usize>)>) -
pub(super) fn target_in_web_area(
pid: i32,
element_ptr_and_idx: Option<(usize, Option<usize>)>,
+ window_id: Option<u32>,
) -> bool {
use crate::ax::bindings::AXUIElementCopyAttributeValue;
use core_foundation::base::{CFTypeRef, TCFType};
use core_foundation::string::CFString;
unsafe {
- // Start element: the addressed one (borrowed — do NOT release) or the pid's
- // focused element (owned — must release when done).
+ // Start element: the addressed one (borrowed — do NOT release) or the
+ // focused element (owned — must release when done). A window-addressed
+ // request may only classify from the window's OWN focused element; a
+ // pid-global focused element can belong to a same-process sibling and
+ // sibling state must never vouch for the target. When window-bound
+ // reacquisition fails, fail closed: report web content (untrusted
+ // read-back) rather than trusting an unproven surface.
let (start, start_owned) = match element_ptr_and_idx {
Some((ptr, _)) => (ptr as AXUIElementRef, false),
- None => match focused_element_of_pid(pid) {
- Some(el) => (el, true),
- None => return false,
+ None => match window_id {
+ Some(wid) => match crate::ax::exact_target::focused_element_in_window(pid, wid) {
+ Some(el) => (el, true),
+ None => return true,
+ },
+ None => match focused_element_of_pid(pid) {
+ Some(el) => (el, true),
+ None => return false,
+ },
},
};
let parent_attr = CFString::new("AXParent");
@@ -703,6 +852,7 @@ fn cgevent_type_verified(
before: Option<&str>,
element_ptr_and_idx: Option<(usize, Option<usize>)>,
settle_ms: u64,
+ window_id: Option<u32>,
) -> anyhow::Result<(bool, Option<usize>)> {
// Focus the target element first so the keystrokes land in IT. Critical in
// foreground mode: a freshly-fronted window's keyboard focus may be on the
@@ -730,7 +880,7 @@ fn cgevent_type_verified(
// expires after observable growth, surface the exact partial count.
let deadline = std::time::Instant::now() + DELIVERY_DRAIN_TIMEOUT;
Ok(await_typed_delivery(before, text, deadline, || {
- read_axvalue(pid, element_ptr_and_idx)
+ read_axvalue_bound(pid, element_ptr_and_idx, window_id)
}))
}
@@ -781,10 +931,13 @@ fn type_text_blocking(
is_terminal_target: bool,
delivery_mode: super::DeliveryMode,
window_id: Option<u32>,
-) -> anyhow::Result<TypeTextOutcome> {
+ keyboard_policy: BackgroundKeyboardPolicy,
+) -> anyhow::Result<TypeTextDelivery> {
// Original field value before any rung drives read-back verification only.
- // An unreadable value is not evidence that the field is empty.
- let before = read_axvalue(pid, element_ptr_and_idx);
+ // An unreadable value is not evidence that the field is empty — and for a
+ // window-addressed request it must come from the exact target window,
+ // never a same-process sibling.
+ let before = read_axvalue_bound(pid, element_ptr_and_idx, window_id);
// --- Foreground rung: explicit agent request (skip AX/background ladder). ---
if delivery_mode.is_foreground() {
@@ -806,6 +959,7 @@ fn type_text_blocking(
before.as_deref(),
element_ptr_and_idx,
foreground_settle_ms,
+ window_id,
)
};
let ((verified, delivered_chars), fronted) = match window_id {
@@ -852,7 +1006,7 @@ fn type_text_blocking(
// Only claim the `_fg` path when a front actually happened; when no
// foregrounding occurred (no window, or SPIs unavailable) these were
// background keystrokes and `path` must say so honestly.
- return Ok(TypeTextOutcome {
+ return Ok(TypeTextDelivery::Typed(TypeTextOutcome {
detail: format!(" via foreground keystrokes ({delay_ms}ms delay)"),
path: if fronted {
PATH_KEY_EVENTS_FG
@@ -861,11 +1015,17 @@ fn type_text_blocking(
},
delivered_chars,
verified,
- });
+ }));
}
// --- Background rung 0: terminal emulator → CGEvent only (AX is dropped). ---
if is_terminal_target {
+ // A terminal insert has no semantic AX rung: when the exact-target
+ // decision restricted this request to semantic-only, there is nothing
+ // safe to run — refuse before posting anything.
+ if let BackgroundKeyboardPolicy::SemanticOnly(refusal) = keyboard_policy {
+ return Ok(TypeTextDelivery::Refused(refusal));
+ }
tracing::debug!(
"type_text: pid {pid} is a terminal emulator; skipping AX value-set, \
using CGEvent key-event synthesis"
@@ -877,19 +1037,29 @@ fn type_text_blocking(
before.as_deref(),
element_ptr_and_idx,
/*settle_ms=*/ 0,
+ window_id,
)?;
- return Ok(TypeTextOutcome {
+ return Ok(TypeTextDelivery::Typed(TypeTextOutcome {
detail: format!(" via CGEvent (terminal emulator, {delay_ms}ms delay)"),
path: PATH_KEY_EVENTS,
verified,
delivered_chars,
- });
+ }));
}
// --- Background rung 1: AX SelectedText write (element or focused). ---
+ // Without an explicit element, a window-addressed request may only write
+ // to the focused element when it provably belongs to the exact target
+ // window — a sibling window's focused field is not the requested target.
let ax_target: Option<(AXUIElementRef, bool, Option<usize>)> = match element_ptr_and_idx {
Some((ptr, idx)) => Some((ptr as AXUIElementRef, /*owns=*/ false, idx)),
- None => unsafe { focused_element_of_pid(pid) }.map(|el| (el, /*owns=*/ true, None)),
+ None => match window_id {
+ Some(wid) => unsafe { crate::ax::exact_target::focused_element_in_window(pid, wid) }
+ .map(|el| (el, /*owns=*/ true, None)),
+ None => {
+ unsafe { focused_element_of_pid(pid) }.map(|el| (el, /*owns=*/ true, None))
+ }
+ },
};
if let Some((element, owns, idx_opt)) = ax_target {
let role = unsafe { copy_string_attr(element, "AXRole") }.unwrap_or_default();
@@ -911,12 +1081,12 @@ fn type_text_blocking(
}
if ax_landed {
let idx_str = idx_opt.map(|i| format!(" [{i}]")).unwrap_or_default();
- return Ok(TypeTextOutcome {
+ return Ok(TypeTextDelivery::Typed(TypeTextOutcome {
detail: format!(" into{idx_str} {role} \"{title}\""),
path: PATH_AX,
verified: true,
delivered_chars: Some(text.chars().count()),
- });
+ }));
}
tracing::debug!(
"AX write did not land for {role} \"{title}\" (err={err}); \
@@ -926,6 +1096,13 @@ fn type_text_blocking(
tracing::debug!("No focused element for pid {pid}; using CGEvent keystrokes");
}
+ // The semantic AX rung did not land and this request is restricted to it:
+ // the process-scoped CGEvent rung could reach a sibling window, so return
+ // the structured refusal instead of escalating.
+ if let BackgroundKeyboardPolicy::SemanticOnly(refusal) = keyboard_policy {
+ return Ok(TypeTextDelivery::Refused(refusal));
+ }
+
// --- Background rung 2: CGEvent keystrokes with read-back. ---
// Never clear here: a partial AX write is rare, and clearing would violate
// insert-at-cursor semantics.
@@ -936,13 +1113,14 @@ fn type_text_blocking(
before.as_deref(),
element_ptr_and_idx,
/*settle_ms=*/ 0,
+ window_id,
)?;
- Ok(TypeTextOutcome {
+ Ok(TypeTextDelivery::Typed(TypeTextOutcome {
detail: format!(" via CGEvent ({delay_ms}ms delay)"),
path: PATH_KEY_EVENTS,
verified,
delivered_chars,
- })
+ }))
}
#[cfg(test)]
@@ -975,6 +1153,7 @@ mod tests {
/*is_terminal_target=*/ true,
super::super::DeliveryMode::Background,
None,
+ BackgroundKeyboardPolicy::Allowed,
);
// We don't care whether r is Ok or Err — what matters is that
// calling it with is_terminal_target=true is safe and never
@@ -982,6 +1161,32 @@ mod tests {
let _ = r;
}
+ /// A semantic-only policy must refuse the terminal short-circuit before
+ /// any CGEvent is posted: terminals have no semantic AX rung, so nothing
+ /// safe remains and the carried refusal comes back unchanged.
+ #[test]
+ fn semantic_only_policy_refuses_terminal_cgevent_rung() {
+ let refusal = BackgroundRefusal {
+ code: cua_driver_core::background_input::refusal_codes::SAME_PID_KEYBOARD_AMBIGUITY,
+ reason: "test".into(),
+ advice: None,
+ };
+ let r = type_text_blocking(
+ -1,
+ "x",
+ None,
+ 0,
+ /*is_terminal_target=*/ true,
+ super::super::DeliveryMode::Background,
+ Some(7),
+ BackgroundKeyboardPolicy::SemanticOnly(refusal.clone()),
+ );
+ match r {
+ Ok(TypeTextDelivery::Refused(returned)) => assert_eq!(returned, refusal),
+ other => panic!("expected a structured refusal, got {:?}", other.is_ok()),
+ }
+ }
+
#[test]
fn verify_typed_unreadable_after_is_unverified() {
// Catalyst: can't read AXValue back → cannot confirm → false.
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text_chars.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text_chars.rs
index 0f390c9878..342fe0ab84 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text_chars.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text_chars.rs
@@ -94,25 +94,55 @@ impl Tool for TypeTextCharsTool {
};
let type_chars_only = args.bool_or("type_chars_only", false);
+ // Retain the addressed element (if any) so a concurrent
+ // get_window_state can't free it during the gate/focus below
+ // (use-after-free → daemon crash). Guard lives past the focus call.
+ let element_guard = if let (Some(idx), Some(wid)) = (element_index, window_id) {
+ match self.state.element_cache.get_element_retained(pid, wid, idx) {
+ Some(guard) => Some(guard),
+ None => {
+ return ToolResult::error(format!(
+ "Element index {idx} not found. Call get_window_state first."
+ ));
+ }
+ }
+ } else {
+ None
+ };
+ let element_ptr: Option<usize> = element_guard.as_ref().map(|g| g.as_ptr());
+
+ // ── Exact-target background gate (macOS background input v1) ──
+ // This tool is always background CGEvent keystrokes — process-scoped
+ // transport with no semantic AX rung. A window-addressed request must
+ // prove exact delivery before any event is posted; there is no
+ // semantic fallback here, so a refusal is final.
+ let _mutation_lease = if let Some(wid) = window_id {
+ match super::gate_background_window_action(
+ pid,
+ wid,
+ element_ptr,
+ cua_driver_core::background_input::BackgroundAction::InsertText,
+ )
+ .await
+ {
+ Ok(lease) => Some(lease),
+ Err(refusal_result) => return refusal_result,
+ }
+ } else {
+ None
+ };
+
// Pre-focus element if requested.
if !type_chars_only {
- if let (Some(idx), Some(wid)) = (element_index, window_id) {
- // Retain so a concurrent get_window_state can't free the element
- // during the focus call (use-after-free → daemon crash). The
- // guard outlives the awaited spawn_blocking below.
- if let Some(element_guard) =
- self.state.element_cache.get_element_retained(pid, wid, idx)
- {
- let element_ptr = element_guard.as_ptr();
- let _ = tokio::task::spawn_blocking(move || {
- crate::input::ax_actions::focus_element(element_ptr)
- })
- .await;
- drop(element_guard);
- tokio::time::sleep(std::time::Duration::from_millis(50)).await;
- }
+ if let Some(element_ptr) = element_ptr {
+ let _ = tokio::task::spawn_blocking(move || {
+ crate::input::ax_actions::focus_element(element_ptr)
+ })
+ .await;
+ tokio::time::sleep(std::time::Duration::from_millis(50)).await;
}
}
+ drop(element_guard);
let text_len = text.chars().count();
let result = tokio::task::spawn_blocking(move || {