Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cua-driver): exact-target macOS background input v1 (#2837)
Hamish Bultitude1b2cb5a706c3
diff --git a/libs/cua-driver/docs/macos-background-input-v1-plan.md b/libs/cua-driver/docs/macos-background-input-v1-plan.mdnew file mode 100644index 0000000000..30ac341015--- /dev/null+++ b/libs/cua-driver/docs/macos-background-input-v1-plan.md@@ -0,0 +1,920 @@+# Exact macOS Background Input v1++**Status:** Proposed implementation plan++**Base:** `origin/main` at `d21e3447f9b08c761c090946648d5aca5e6c9cf1`++**Date:** 2026-08-04++## Goal++Let an agent operate ordinary macOS applications in the background without+moving the real pointer, changing the user's active application, or silently+acting on another window owned by the same process.++V1 is deliberately conservative. It expands the routes for which Cua Driver can+prove an exact `(pid, CGWindowID)` destination and returns a structured refusal+for every other route. A refused action is preferable to a process-scoped key+event that appears successful after mutating a sibling window.++## Success criteria++V1 is complete when all of the following are true:++1. Every window-scoped macOS mutation carries the requested `(pid,+ CGWindowID)` from target resolution through route selection, dispatch, and+ post-action verification.+2. A result can be `confirmed` only from evidence belonging to that exact+ window or to an exact browser target bound to it. State from another+ same-process window is never accepted as evidence.+3. Background routing is ordered as semantic Accessibility, exact browser/CDP,+ exact window-local pointer, and finally PID keyboard with an independent+ exact-delivery proof. If none is safe, the driver refuses before sending+ input.+4. Background mode never silently crosses to global HID, activates another+ application, moves the real pointer, changes Space, or restores a minimized+ window.+5. Minimized and hidden windows retain safe semantic AX actions and current+ one-shot capture. Raw key commits are refused; callers use `AXConfirm`, an+ exact `AXPress`, or an exact browser action when one exists.+6. An off-Space window that is absent from the process's `AXWindows` is+ observation-only. Cua Driver does not use another window's AX tree or send+ process-scoped input while the target is unresolved.+7. Ordinary Electron applications get per-process AX enablement, exact+ `BrowserWindow` mapping, bounded fresh-tree acquisition, delayed effect+ verification, and explicit two-window safety coverage.+8. Existing action request schemas and the closed `ActionResult` wire shape+ remain compatible.+9. Focused unit/contract tests run in ordinary CI. Exact-SHA native evidence is+ collected separately on a signed macOS Namespace runner and in the+ canonical Lume acceptance lane.++## Non-goals++V1 does not attempt to provide:++- live minimized-window PiP or a window-scoped ScreenCaptureKit stream;+- continuously changing Chromium pixels while the application is occluded,+ hidden, or minimized;+- full Accessibility parity across Spaces;+- automatic Space switching, temporary Space membership, sticky windows, or+ switch-and-restore input;+- deminiaturize/order-behind/re-minimize transactions;+- minimize-button interception or application-specific window swizzling;+- a background raw-key guarantee for multi-window applications;+- a global-HID fallback in background mode;+- background control of games, Metal surfaces, protected video, secure input,+ or other surfaces that do not expose an exact semantic or routed target;+- generic WKWebView mutation, Safari JavaScript fallback expansion, or a new+ browser attachment mechanism;+- a redesign of the public action-result contract; or+- changes to application-owned Chromium settings such as+ `backgroundThrottling`.++The existing caller-selected `delivery_mode: "foreground"` remains a separate+last-resort behavior. This plan does not remove it, but the background router+must never enter it implicitly.++## Current source-grounded state++### Exact AX scoping exists++[`ax/window_scope.rs`](../rust/crates/platform-macos/src/ax/window_scope.rs)+already defines `Matched`, `NotFound`, `OwnerPidMismatch`, and `AxUnresolved`.+Its structural invariant is the right foundation: every non-`Matched` result+walks no elements.++[`ax/tree.rs`](../rust/crates/platform-macos/src/ax/tree.rs) unions+`AXChildren` and `AXWindows`, maps top-level AX windows to CG window IDs through+`_AXUIElementGetWindow`, and bounds the default walk at 2,000 nodes and depth+25. [`get_window_state.rs`](../rust/crates/platform-macos/src/tools/get_window_state.rs)+preflights WindowServer ownership, empties the element cache when the requested+window does not resolve, and still returns an exact-window screenshot for the+`AxUnresolved` degradation.++These checks prevent an unresolved request from returning a sibling window's+tree. Mutation needs to carry the same invariant farther than it does today.++### Explicit `window_id` does not make PID keyboard delivery window-scoped++[`window_target.rs`](../rust/crates/cua-driver-core/src/window_target.rs)+correctly promotes a unique PID-only target and refuses a PID with multiple+eligible top-level windows. Explicit `window_id` and element-token requests,+however, pass through that cardinality guard unchanged.++On macOS, [`keyboard.rs`](../rust/crates/platform-macos/src/input/keyboard.rs)+posts through `SLEventPostToPid` with a public `CGEvent::post_to_pid` fallback.+Both transports are process-scoped. [`press_key.rs`](../rust/crates/platform-macos/src/tools/press_key.rs)+best-effort focuses an element and then posts the key to the PID. Its pixel form+first invokes the click path to focus `(x, y)` and then posts the same+process-scoped key, so that combined pointer-plus-key branch needs one exact+route decision rather than bypassing keyboard safety after the focus click.++[`type_text.rs`](../rust/crates/platform-macos/src/tools/type_text.rs) has a+second exactness gap: when no explicit element pointer is supplied,+`read_axvalue(pid, None)` reads `focused_element_of_pid(pid)`. Its before/after+verification can therefore observe whichever same-process field is focused,+not necessarily a descendant of the requested `window_id`. Passing an explicit+window ID does not change that readback.++The required same-PID regression test follows directly from these source+semantics: unresolved target window A, focused sibling window B, process-routed+key, and PID-global focused-element readback. V1 must refuse before the key is+posted and prove from fixture journals that neither window changed.++### Exact semantic and window-local primitives already exist++[`ax_actions.rs`](../rust/crates/platform-macos/src/input/ax_actions.rs)+supports `AXPress`, `AXConfirm`, selection, focus, and enabled-state checks.+[`set_value.rs`](../rust/crates/platform-macos/src/tools/set_value.rs) already+does native AX value readback and correctly distrusts AXValue echo from an+`AXWebArea`.++[`mouse.rs`](../rust/crates/platform-macos/src/input/mouse.rs) has a routed+pointer primitive that stamps window-local location plus the target PID and+CGWindowID fields before SkyLight/PID posting. It does not move the real+pointer. The same file also contains private no-raise activation and+Chromium-specific event recipes. These are implementation candidates, not by+themselves proof that an action reached the requested window. V1 must gate them+on exact target facts and exact post-action evidence.++[`click.rs`](../rust/crates/platform-macos/src/tools/click.rs) already prefers+AX, can perform window-local pointer fallback for specific selection controls,+and distinguishes verified AX selection from an unverified dispatch. The new+router should consolidate those decisions rather than add a second hidden+fallback ladder.++### Electron AX enablement exists but needs lifecycle and mutation discipline++[`ax/bindings.rs`](../rust/crates/platform-macos/src/ax/bindings.rs) writes+`AXManualAccessibility` and falls back to `AXEnhancedUserInterface` only when+the modern attribute is unsupported. The tree walker enables it once per PID,+waits 500 ms when accepted, and then walks the bounded tree.++The current cache is a set of PIDs. V1 should associate enablement with the+observed process lifetime so PID reuse cannot inherit a stale "enabled"+decision. Electron mutation must reacquire an exact bounded tree rather than+using title order, the first AX window, or stale process-global focus.++### Exact browser mutation is already unique-or-refuse++[`browser/binding.rs`](../rust/crates/cua-driver-core/src/browser/binding.rs)+contains the pure native-window-to-CDP correlation. [`browser/engine.rs`](../rust/crates/cua-driver-core/src/browser/engine.rs)+checks native ownership, process fingerprint, endpoint ownership, CDP window+geometry/cardinality, tab identity, and generation again before every+mutation. [`browser/tools.rs`](../rust/crates/cua-driver-core/src/browser/tools.rs)+uses those exact capabilities for typed browser actions.++V1 should reuse this rung rather than creating a macOS-specific browser+shortcut. The bounded Electron fallback remains exact only when the endpoint+exposes one page and the owner has exactly one native window. Two Electron+`BrowserWindow`s must not use that fallback; they need a proven CDP-window+mapping or a refusal. WKWebView mutation remains unsupported.++### Capture and Spaces metadata are limited++[`capture.rs`](../rust/crates/platform-macos/src/capture.rs) currently obtains+a one-shot window image with `screencapture -l <CGWindowID> -x -o`. The+ScreenCaptureKit implementation is display-scoped recording, not a+window-scoped live stream. V1 keeps the current one-shot path and does not+claim frame freshness from API success.++[`list_windows.rs`](../rust/crates/platform-macos/src/tools/list_windows.rs)+describes minimized, hidden, and off-Space enumeration, but the current macOS+records do not reliably populate `current_space_id`, `on_current_space`, or+`space_ids`. V1 must therefore use exact AX-window presence as its input safety+gate instead of inventing a complete Spaces model.++### The result contract is intentionally closed++[`outputs.rs`](../rust/crates/cua-driver-contract/src/outputs.rs) defines the+closed `ActionResult`: effect, route, delivery, evidence, and escalation.+[`action_record.rs`](../rust/crates/cua-driver-core/src/action_record.rs) is the+functional core that conservatively projects native attempts to that public+shape. [`action-result-contract.md`](action-result-contract.md) explicitly+excludes echoed selectors, coordinates, and request targets.++V1 keeps that public result shape. Exact target facts belong to the request,+the internal execution record, a read-only capability report, and structured+refusals. Any future proposal to add optional result fields must update the+contract and every generated strict SDK together; it is not part of this work.++## Locked design decisions++### 1. Exact-window invariant++The canonical target key is the requested `(pid, CGWindowID)`. It is carried+unchanged through planning, execution, and verification.++The following rules are non-negotiable:++- WindowServer must attribute the CGWindowID to the requested PID immediately+ before mutation.+- An AX element route must prove that the live element ascends to an AX window+ whose `_AXUIElementGetWindow` is the requested CGWindowID.+- A browser route must revalidate its existing exact native-window capability+ immediately before mutation.+- A pointer route must use the requested window's validated pixel frame and+ stamped CGWindowID, then verify an effect on that same target.+- A keyboard readback must read the addressed element or a fresh element+ proven to descend from the requested AX window. It must never call a+ PID-global focused-element reader as target evidence.+- A sibling window's state may be retained as a negative canary, but it can+ never confirm the requested action.+- If the target disappears, changes owner, becomes AX-unresolved where the+ chosen route needs AX, or becomes ambiguous between planning and dispatch,+ no fallback actuator runs.++An explicit `window_id` is an address supplied by the caller, not proof that a+process-scoped actuator will honor it.++### 2. Functional core and imperative shell++Add a small pure background-input decision module in+`cua-driver-core`. It owns no macOS objects and performs no I/O. Its inputs are+facts collected by a platform shell:++- exact target and WindowServer ownership;+- AX scope (`matched`, `not_found`, `owner_mismatch`, `unresolved`);+- requested action semantics and trust class;+- target element ancestry and available semantic actions;+- target state (`visible`, `occluded`, `offscreen`, `minimized`, `hidden`, or+ unknown where macOS cannot distinguish safely);+- same-PID top-level-window cardinality;+- exact browser-binding eligibility;+- exact window-local pointer eligibility;+- keyboard destination and available verifier;+- baseline foreground PID, real pointer position, and Space posture; and+- permission/private-symbol readiness.++The pure output is one route with prerequisites and a verification plan, or a+typed refusal reason. Keep names internal, but model at least:++```text+ExactWindowTarget(pid, window_id)+TargetResolution+BackgroundRoute+VerificationPlan+BackgroundInputDecision = Execute(...) | Refuse(...)+```++Use a pure truth table for route choice and result classification. Extend the+existing `ActionExecutionRecord` projection rather than adding macOS wire+semantics. Platform adapters translate a chosen internal route to the existing+public `ActionRoute` values.++The macOS imperative shell:++1. serializes mutation planning per PID;+2. gathers fresh WindowServer, AX, browser, and foreground facts;+3. asks the pure core for exactly one decision;+4. revalidates route-specific facts immediately before dispatch;+5. invokes only that actuator;+6. polls only the planned exact-target verifier within a bounded deadline;+7. checks foreground PID, real pointer, and Space posture; and+8. emits the existing action result or a structured refusal.++The core never says "try all routes." The shell cannot improvise a second+actuator after a failed proof. A caller may take a newly reported route after+fresh state, preserving the existing agent-owned action ladder.++### 3. Background routing ladder++The decision order is:++| Order | Route | Required exactness | V1 behavior |+| --- | --- | --- | --- |+| 1 | AX semantic action/value | Exact AX window and exact element ancestry | Prefer `AXPress`, `AXConfirm`, selection, or value mutation with target-bound readback. AX API acceptance alone is unverified. |+| 2 | Browser/CDP | Existing exact native-window, endpoint, CDP-window, tab, generation, and ref proof | Reuse typed browser tools. Do not guess or auto-convert an AX token into a DOM ref. |+| 3 | Window-local routed pointer | Exact owner, AX-window presence, validated pixel frame, stamped CGWindowID, supported target state, and exact postcondition | Never move the real pointer. Private SPI absence or failed revalidation refuses. |+| 4 | PID keyboard | Exact target element/window, no competing same-PID keyboard destination, and an exact verifier available before posting | Native single-window text insertion may qualify. Generic keys, multi-window apps, minimized/hidden windows, and unresolved off-Space windows refuse. |+| 5 | Refusal | No safe route | Return the exact reason and available alternatives without sending input. |++The ladder preserves action semantics. It must not silently replace a trusted+browser click with a synthetic DOM event, infer a submit button from a label,+or turn a desktop request into a browser request without an exact capability.+Where request shapes cannot be translated exactly, the result advises the+explicit tool the caller may use next.++Global HID is not in this table. Only an explicit foreground request may use+the existing guarded foreground/HID path.++### 4. Semantic AX behavior and commit policy++For a token/index action, revalidate the retained element's ancestry against+the requested top-level AX window. A cached element under the right cache key+is not sufficient after a lifecycle or Space transition.++Safe commit behavior is explicit:++- use an advertised `AXConfirm` on the exact field when available;+- use `AXPress` on an explicitly addressed commit button;+- use an exact browser action when the caller has a live page/ref capability;+- otherwise refuse a background `Return` for minimized, hidden, unresolved,+ or multi-window targets.++Do not heuristically find the first default button or press another window's+button. Do not turn `press_key(Return)` into `AXPress` unless the request itself+identifies the semantic target.++Native `AXValue` equality can confirm a value operation on the exact retained+element. AX values under `AXWebArea` remain untrusted because the accessibility+shim can echo a write that the renderer did not consume. Electron/web content+requires browser/DOM readback or another exact application effect.++### 5. Electron policy++Electron support is part of v1, not a generic Chromium exception.++For each observed process lifetime:++1. Create the AX application element and try `AXManualAccessibility=true`.+2. Fall back to `AXEnhancedUserInterface=true` only when the modern attribute+ returns `kAXErrorAttributeUnsupported`; transient failures do not trigger a+ second private write or a success claim.+3. If enablement is accepted, wait the existing bounded settle and perform a+ fresh exact-window walk. Key the enablement cache by process fingerprint or+ generation, not an immortal numeric PID.+4. Build the top-level `BrowserWindow` map only from `_AXUIElementGetWindow`.+ Window title, array order, focused-window status, and first-match selection+ are not identity proofs.+5. Require the requested BrowserWindow to be in `AXWindows` before native+ mutation. An absent target returns no elements and no input route.+6. Before an element action, prove its ancestry to that BrowserWindow. After+ the action, poll a fresh bounded exact-window tree or exact CDP/DOM state.+ Never verify against process-global focus or a stale cached tree.+7. Use a delayed bounded verifier because renderer and AX state publication are+ asynchronous. Timeouts produce `unverifiable` or a proven delivery failure,+ never a fabricated confirmation.+8. Retain the existing distrust of web-area AXValue echo. Normal DOM text and+ key semantics should use the exact browser rung when a binding is available.+9. With two BrowserWindows, disable the single-page/single-native-window CDP+ shortcut and every raw PID-key route. A route is available only when the+ exact native/CDP mapping or exact semantic AX action survives revalidation.++The signed fixture launches once without `--force-renderer-accessibility` to+prove the driver's per-PID enablement works for a normal `BrowserWindow`, and+once with the existing fixture flag as a diagnostic control. It also covers+default `backgroundThrottling` and `backgroundThrottling:false`. The setting+may affect changing pixels, but it must not weaken exact-target input policy or+become a driver prerequisite.++### 6. Minimized and hidden windows++For minimized or application-hidden targets:++- keep exact one-shot capture as observation when it succeeds;+- label capture freshness as unknown rather than promising live frames;+- allow exact semantic AX actions with target-bound verification;+- allow an already-exact browser/CDP action only if all native and browser+ binding checks still pass;+- use semantic commit rather than a raw Return key;+- refuse PID keyboard and native routed pointer in v1; and+- do not deminiaturize, unhide, activate, reorder, move, or re-minimize the+ window as an implementation detail.++This preserves foreground state by construction. A no-activate restore+transaction can be investigated as a later opt-in mode, but it is not a v1+fallback.++### 7. Other-Space policy++Observation is allowed: `list_windows`, exact one-shot capture, and any honest+degradation metadata may still be returned.++For mutation, query a fresh application `AXWindows` and map every candidate via+`_AXUIElementGetWindow`:++- If the requested CGWindowID is absent, refuse all input with an+ `off_space_or_ax_unresolved` reason. Do not inspect or act on a sibling+ window, even if it is on the current Space.+- If the exact target is present, v1 may use a semantic AX or exact browser+ route whose own preconditions and postcondition pass. A native pixel or+ PID-key route is not promoted merely because capture succeeded.+- Unknown Space metadata remains unknown. Do not claim `on_current_space`+ without a real source.++Explicit switch/restore, temporarily adding a window to the current Space, and+sticky-window behavior require a later opt-in design with its own visible-state+and cleanup contract. V1 adds no SkyLight Spaces manipulation.++### 8. Exact pointer route++The existing window-local pointer implementation may be selected only when:++- WindowServer ownership and AX-window identity both match the requested+ target immediately before dispatch;+- the current screenshot dimensions have a validated mapping to the requested+ WindowServer bounds;+- the local point falls inside that exact window frame;+- the target is neither minimized, hidden, nor AX-unresolved/off-Space;+- the required private symbols resolve on the running macOS version;+- baseline frontmost PID and real pointer position have been captured; and+- an exact postcondition exists for the selected control.++Use the existing stamped local coordinate and CGWindowID fields. Treat private+no-raise activation as part of the actuator, not as delivery proof. If it+changes the user's frontmost application, pointer, or Space, mark the route+failed, restore only state owned by the driver when that is safe, and disable+the route for the remainder of the process session.++The v1 router does not use unverified pixel input for a mutation whose only+result is "event posted." It may remain available through current APIs as+`unverifiable` outside the new exact capability, but it must not be advertised+as exact background input.++### 9. PID keyboard gate++PID-routed keyboard is a last background rung because macOS accepts a process,+not a CGWindowID.++Before posting text, require all of:++- a live explicit element proven to descend from the requested AX window;+- that exact element is the PID's focused UI element and the PID's focused AX+ window maps to the requested CGWindowID;+- the requested target is the only eligible same-PID keyboard window at the+ pre-dispatch revalidation point;+- the target is visible or occluded on the current Space, not minimized or+ hidden;+- a target-bound value verifier is readable before dispatch; and+- the value surface is native, not an `AXWebArea` echo surface.++Post the event only after those facts pass, then poll the same retained target+or a freshly reacquired equivalent inside the exact requested tree. If any+precondition cannot be proven, refuse before input. Generic `press_key` and+hotkeys normally lack a safe target-specific postcondition and therefore+refuse in background exact mode unless a future action-specific verifier is+added.++The singleton rule is a conservative v1 proof, not a claim that+`SLEventPostToPid` is window-addressed. It prevents the known class of+same-process sibling delivery while semantic and browser routes cover most+ordinary controls.++### 10. Capability and result reporting++Add an additive `background_input` section to the read-only exact-window state+output. It should be produced from the same pure decision facts and contain no+window titles, control values, or private symbol addresses. A representative+shape is:++```json+{+ "background_input": {+ "exact_window": {+ "status": "matched",+ "pid": 123,+ "window_id": 456,+ "ax_window": "matched"+ },+ "routes": [+ {"route": "accessibility", "status": "available"},+ {+ "route": "pid_keyboard",+ "status": "refused",+ "reason": "same_pid_sibling_windows"+ }+ ],+ "observation": {+ "one_shot_capture": "available",+ "frame_freshness": "unknown"+ }+ }+}+```++The capability route and status strings above are illustrative and are not new+values for the closed public `ActionRoute` enum.++The exact enums and generated types should be reviewed with the implementation,+but the semantics are fixed:++- `available` means route prerequisites are currently proven, not that a+ future call is guaranteed to succeed;+- every action revalidates instead of trusting this report;+- unknown facts stay unknown;+- route refusal includes one stable machine-readable reason; and+- the report does not expose user content.++Keep successful action outputs on the current `ActionResult` contract:++- `route` uses the existing enum, including accessibility, synthetic events,+ system API, DOM, and trusted input for relevant background actions;+- `delivery.mode` reports what actually occurred;+- `confirmed` requires publishable exact-target readback;+- `unverifiable` means the route ran but exact effect is unknown;+- `refused` means no actuator ran; and+- escalation is advice for an explicit caller decision.++Structured refusals should include `code`, `effect: "refused"`, the requested+`pid` and `window_id`, the failed exactness fact, and the safe next route when+one exists. Initial codes should distinguish at least ownership mismatch,+AX-window unresolved/off-Space, same-PID keyboard ambiguity, unavailable+verification, and unavailable private route.++Do not report `delivery_failed` merely because the driver lacks evidence.+Use it only when exact negative evidence proves delivery did not occur. A+missing/unreadable postcondition is `unverifiable` with+`effect_unconfirmed`; an unchanged readable exact target after the bounded+delivery window may be `suspected_noop` or delivery failure according to the+existing action's semantics. API acceptance is never confirmation.++## State policy matrix++This is the required decision policy, not a claim that every route already has+native evidence:++| Target state | Observe | AX semantic | Exact browser | Window-local pointer | PID keyboard |+| --- | --- | --- | --- | --- | --- |+| Frontmost | Current state/capture | Yes when exact | Yes when exact | Yes when exact and verified | Only with singleton exact element and verifier |+| Background visible | Current state/capture | Yes when exact | Yes when exact | Yes when exact and verified | Same conservative gate |+| Fully occluded | One-shot capture; freshness honest | Yes when exact | Yes when exact | Yes only after signed proof and exact verifier | Same conservative gate |+| Fully offscreen on current Space | One-shot capture when valid | Yes when exact | Yes when exact | Signed-evidence gated; no geometry guess | Refuse in v1 |+| Minimized | One-shot capture, freshness unknown | Yes when exact | Yes only if exact binding revalidates | Refuse | Refuse; semantic commit instead |+| Application hidden | One-shot capture, freshness unknown | Yes when exact | Yes only if exact binding revalidates | Refuse | Refuse; semantic commit instead |+| Other Space, exact AX window present | Observation allowed | Yes when exact | Yes when exact | Refuse in v1 | Refuse in v1 |+| Other Space/AX window absent | Observation only | Refuse | Refuse | Refuse | Refuse |+| Owner mismatch/stale window | Refuse stale target; report owner when safe | Refuse | Refuse | Refuse | Refuse |++## Compatibility expectations++These are route expectations to certify, not unsupported parity claims:++| Surface | V1 expectation | Evidence required |+| --- | --- | --- |+| AppKit | Primary supported lane: exact AX actions/value, verified window-local pointer where semantic AX is unavailable, narrowly gated native text | Runtime AppKit fixture, including two windows under one PID |+| SwiftUI | Exact AX on the current Space; truthful refusal when an off-Space window disappears from `AXWindows` | Runtime SwiftUI fixture on current and another Space |+| Electron DOM | AX enablement and exact BrowserWindow tree; exact CDP route when bound; no multi-window PID keys | Runtime two-BrowserWindow fixture with per-window DOM journals |+| Mac Catalyst | Semantic AX or verified pointer only; raw keyboard remains gated | Source-supported candidate; runtime row required before advertising |+| Qt/Java | Use exposed exact AX semantics; refuse absent/custom AX and raw-key ambiguity | Source-based until dedicated signed fixtures exist |+| Browser canvas/WebGL | Exact browser action when a supported ref/route exists; otherwise no semantic claim | Source-based browser binding plus route-specific runtime tests |+| Games/Metal/custom event loops | No v1 background guarantee | Explicitly unsupported until an exact app-owned route exists |+| Secure/protected surfaces and secure input | No capture or input bypass | Refusal/permission tests; never weaken platform protection |+| System UI and out-of-process panels | Require actual WindowServer owner and exact AX window; never follow a PID silently | Existing owner-mismatch tests plus signed panel coverage |++`backgroundThrottling:false` may preserve Electron painting in cases where the+default renderer throttles. That is an application compatibility option, not a+Cua Driver route guarantee. Canvas/WebGL, trusted-event filters, and custom+event loops can still behave differently from ordinary DOM controls.++## Staged implementation++### Stage 0: Lock the safety regression++- Extend the AppKit and Electron fixtures with two fixed-title windows, one+ field and one commit control per window, and a run-owned per-window journal.+- Reproduce process-scoped keyboard delivery with target A unresolved or+ minimized and sibling B focused.+- Add a test that requests A by exact `(pid, window_id)` and requires a refusal+ before dispatch; both journals and both field values must remain unchanged.+- Add a unit regression showing that a sibling's focused-element readback can+ never satisfy A's verification plan.++Do not refactor routes until this failing behavioral shape and pure regression+are both captured.++### Stage 1: Pure exact-target router++- Add the target facts, route decision, verifier requirement, and refusal+ reasons to common core.+- Encode the complete state matrix as table-driven unit tests.+- Make "explicit `window_id`" distinct from "exact delivery proven."+- Extend internal action records with diagnostic exact-target proof state only+ as needed; do not change the public `ActionResult` shape.+- Add a per-PID mutation coordinator so two concurrent background decisions+ cannot change process focus underneath one another.++### Stage 2: Fresh macOS target acquisition++- Centralize WindowServer owner preflight and fresh `AXWindows` mapping.+- Add a bounded ancestry check from a retained element to its exact AX window.+- Make Chromium/Electron AX enablement process-lifetime aware and preserve the+ modern-attribute/fallback error distinction.+- Reacquire bounded exact-window state for delayed postconditions and clear+ stale caches on every unresolved result.+- Change the `AxUnresolved` input advice from unconditional pixel mutation to+ observation-only/refusal when exact input cannot be proven.++### Stage 3: Semantic actions and exact verification++- Route exact `AXPress`, `AXConfirm`, selection, and native value mutations+ first.+- Bind every readback to the requested target element/window.+- Preserve web-area AX echo distrust and add browser/DOM verification where an+ exact capability already exists.+- Make process-level window changes diagnostic only unless the changed window+ is the exact target postcondition.+- Normalize `confirmed`, `unverifiable`, `suspected_noop`, and proven delivery+ failure through the functional core.++### Stage 4: Browser, pointer, and keyboard rungs++- Reuse browser mutation revalidation without a second Electron-only binding+ implementation.+- Gate the existing window-local pointer route on exact target, state, pixel+ frame, private-symbol readiness, and verifier facts.+- Route `press_key`'s existing pixel-focus-then-key form through the same pure+ decision. A successful focus click cannot waive the exact AX-window,+ singleton destination, target-bound verifier, or state gates for the key.+- Remove PID-global focused-element readback from window-scoped keyboard+ verification.+- Permit native PID text only under the singleton exact-element/verifier gate.+- Refuse generic keys and every unsafe minimized/hidden/off-Space raw-key case.+- Assert that the background router has no call edge to global HID or+ foreground assist.++### Stage 5: Capabilities, documentation, and rollout++- Add the read-only per-window capability report and structured refusal codes.+- Keep request and `ActionResult` schemas compatible; regenerate/check SDK+ artifacts only if the read-only output has a generated typed surface.+- Update the background-delivery and known-limit documentation to match the+ certified matrix, not the intended matrix.+- Ship the high-risk same-PID refusal before expanding routes. If a new route+ regresses, disable that route while retaining the exact-target guard.+- Run the full desktop acceptance matrix once against the final exact SHA, as+ required by repository guidance.++## Test and acceptance plan++### Ordinary CI++Run these on every implementation pull request without a GUI or TCC grants:++- pure route-decision table for every state and route;+- owner mismatch, stale ID, AX-unresolved, and exact matched scope;+- explicit-window versus exact-delivery distinction;+- same-PID sibling rejection before actuator invocation;+- wrong-sibling readback ignored by verification;+- pixel-focus-then-key cannot bypass keyboard refusal or dispatch a second+ actuator after target facts change;+- no background decision can produce a global-HID transport;+- native value, missing readback, unchanged value, partial text, delayed+ success, and negative-delivery result classification;+- Electron PID-lifetime enablement cache and fallback behavior;+- exact BrowserWindow map cardinality and no first/title-only match;+- existing CDP binding/revalidation, including one-page/one-window fallback+ refusal after a second native window appears;+- minimized/hidden/off-Space policy truth tables;+- read-only capability and structured-refusal schema snapshots;+- existing closed `ActionResult` invariants and generated SDK parity; and+- fixture build/lint tests.++### Signed macOS behavior matrix++The native harness must use only repository fixtures and a fixture-owned+occlusion sentinel. Each action records pre/post foreground PID, real pointer+position, requested CGWindowID, AX window IDs, route, result, exact fixture+journal entry, and cleanup state.++Required rows:++1. **AppKit:** frontmost, background visible, fully occluded, offscreen,+ minimized, and hidden. Cover AXPress, AXConfirm, AXValue, exact pointer, and+ singleton native text.+2. **AppKit same PID/two windows:** request each window independently; minimize+ or make one AX-unresolved while the sibling is focused; raw key/text must+ refuse and neither sibling journal may change.+3. **SwiftUI:** background AX actions and value readback on the current Space;+ move only the fixture window to another preconfigured Space, query from the+ other Space, and require refusal whenever its CGWindowID is absent from+ `AXWindows`.+4. **Electron DOM/two BrowserWindows:** run without force-renderer AX first,+ then the diagnostic flag; run default throttling and+ `backgroundThrottling:false`; prove exact AX window mapping, AXPress,+ semantic commit, exact browser type/click, delayed readback, and raw-key+ refusal for minimized/unresolved targets. Every effect must appear in the+ addressed window's own fixture journal and not its sibling's.+5. **Browser routes:** exact bind and mutation, stale generation, geometry+ mismatch, second native window, ambiguous tab/window, and unavailable+ endpoint. Every ambiguous row refuses without mutation.+6. **Foreground preservation:** for every background success and refusal, the+ fixture sentinel remains frontmost, the real pointer is unchanged, and the+ current Space is unchanged.+7. **Capture:** record whether one-shot capture succeeds in each state and+ never infer live animation from a successful PNG alone.+8. **Permissions/private APIs:** missing Accessibility, missing Screen+ Recording, and missing private symbols return an honest unavailable/refused+ result without prompting or global fallback.+9. **Cleanup:** close fixture windows, stop all fixture/driver processes,+ restore the initial fixture foreground, pointer, and Space, and leave no+ run-owned artifact outside the selected artifact directory.++The other-Space setup belongs to the disposable runner image or a test-only+harness. Product code must not create, join, or switch Spaces. The test records+its starting Space and always restores it in a trap/finally path.++### Canonical Lume gate++The repository's canonical signed/TCC acceptance remains+[`tests/runners/macos-lume/README.md`](../tests/runners/macos-lume/README.md)+and [`run-all.sh`](../tests/runners/macos-lume/run-all.sh). It provides the+logged-in Aqua session, certificate-backed `CuaDriverLocal.app`, normal TCC+grant flow, and exact-source checks.++After focused Namespace evidence passes, add the new fixture rows to the Lume+entrypoint and run once on the final candidate SHA. Preserve its existing+requirements:++- use a disposable worker cloned from the stopped private seed;+- install with `install-local.sh --release --autostart+ --require-stable-signing`;+- never edit `TCC.db`;+- execute the GUI suite from the guest's logged-in Terminal, not SSH;+- retain exact SHA, environment, code-signing requirement, permission status,+ structured results, fixture journals, screenshots/recording where allowed,+ and cleanup evidence; and+- retrieve evidence, then delete the worker.++Namespace validation complements this gate; it does not replace the Lume+golden-image contract.++## Namespace macOS runner workflow++### Discovery recorded for this plan++The planning checkout was inspected on 2026-08-04:++- project `.amp/plugins` was absent;+- the surrounding workspace `.amp/plugins` was absent;+- user `~/.config/amp/plugins` contained+ `namespace-mac-runners.ts` and `namespace-mac-runners.js`;+- no repository-local Namespace runner instructions were found; and+- the TypeScript source registered the exact tool name+ `namespace_mac_runner`.++The user plugin is not versioned with this repository, so a later agent must+repeat discovery before use. Check project `.amp/plugins`, user+`~/.config/amp/plugins`, the active workspace plugin location, Amp plugin+documentation, and the loaded plugin source. If the tool is absent, record+those checks and make plugin discovery/setup step one; do not guess an API.++The discovered tool accepts:++```text+action: list | spawn | destroy+runner_id: optional hostname-compatible ID for spawn+repository_url: optional HTTPS github.com URL for spawn+duration_minutes: optional integer from 5 through 720+instance_id: required for destroy+```++The discovered implementation defaults to macOS `26.x`, arm64, 6 vCPU,+14,336 MB, and a 60-minute TTL. It requires `nsc login`. Spawn creates billable+infrastructure immediately without confirmation, clones into+`$HOME/workspace/repository` through Amp's GitHub credential helper, starts+`amp --no-tui --runner-id "$AMP_RUNNER_ID" --remote-control-terminal` in tmux,+and waits for `Registered. Create threads`. Failed provisioning attempts to+destroy the instance it created. Destroy accepts only plugin-managed instance+IDs and asks for confirmation. These details must be re-read from the installed+plugin because it can change independently of this plan.++### Exact later validation workflow++1. Re-run the plugin discovery above and record the resolved source/config.+2. Call `namespace_mac_runner` with `{"action":"list"}`. Do not modify or+ destroy unrelated runners.+3. Obtain explicit approval for the billable spawn. The approval for this plan+ PR does not authorize a later infrastructure charge.+4. Push the implementation candidate and record its full SHA. Spawn with a+ unique hostname-compatible ID and bounded TTL, for example:++ ```json+ {+ "action": "spawn",+ "runner_id": "cua-bg-input-<unique-suffix>",+ "repository_url": "https://github.com/trycua/cua.git",+ "duration_minutes": 180+ }+ ```++5. Record the returned runner ID, Namespace instance ID, URL, resolved macOS+ selector, repository URL, and deadline. TTL is a backstop, not cleanup.+6. Call Amp's `list_runners` immediately before `create_thread`. Start a child+ with `executor: "runner"` and that exact live runner ID; do not silently use+ an orb. Instruct it to fetch and detach-checkout the exact candidate SHA,+ because the plugin's shallow clone starts from repository default HEAD.+7. On the Namespace Mac, record `sw_vers`, OS build, architecture, SIP status,+ Xcode/Rust/Node versions, exact git SHA, and a clean worktree before testing.+8. Use repository-supported `CuaDriverLocal.app` setup only. Follow+ [`scripts/README.md`](../scripts/README.md), use an ephemeral+ certificate-backed local signing identity, and install with+ `--require-stable-signing`. Never copy a maintainer private key or signing+ keychain to Namespace. Never edit `TCC.db` or bypass TCC.+9. Grant Accessibility and Screen Recording only through normal macOS UI if the+ disposable Namespace environment and explicit test approval support it.+ Record `permissions status --json` and the `codesign -d -r-` designated+ requirement. The Namespace plugin provisions neither a signing identity nor+ TCC. If stable signing, interactive UI, or either grant is unavailable,+ mark the signed native lane blocked and run only pure/build tests; do not+ claim native behavior passed.+10. Use only fixed-title repository fixtures and run-owned artifact paths.+ Capture the matrix evidence listed above, including exact fixture journal+ effects, requested and sibling window IDs, AX counts/mapping, result+ payloads, foreground PID, real pointer, Space posture, and action timing.+ Do not collect unrelated application state, window titles, or user data.+11. In a finally/trap path, stop fixture and driver processes, close fixture+ windows, restore fixture focus/pointer/Space state, remove run-owned+ temporary files, and write a cleanup assertion. Retrieve only sanitized+ evidence needed for review.+12. From the controlling thread, call+ `namespace_mac_runner` with+ `{"action":"destroy","instance_id":"<recorded-instance-id>"}` and confirm+ destruction. Call `{"action":"list"}` again and require the instance to be+ absent.++## Risk analysis++### macOS and private APIs++`_AXUIElementGetWindow`, `AXManualAccessibility`,+`AXEnhancedUserInterface`, SkyLight event posting, private event fields, and+no-raise activation may change between macOS releases. Every lookup must be+bounded and fail closed. Missing symbols remove a route; they never cause a+global or PID-only fallback.++V1 adds no private Spaces API. Space membership remains incomplete, and the+policy deliberately refuses when AX cannot prove the exact target.++### TCC and signing identity++Accessibility and capture evidence is valid only for the process identity that+performed it. Ad-hoc rebuilds can invalidate grants. Native acceptance requires+the certificate-backed `CuaDriverLocal.app` flow and normal permission UI.+Neither CI mocks nor a successful build substitutes for this evidence.++### Application semantic differences++AX API success may be a no-op, Chromium AXValue may be an echo, and custom+renderers may reject synthetic input. Confirmation always comes from an+independent exact-target effect. Unsupported applications receive narrower+capabilities, not a misleading parity claim.++### Races and lifecycle++Windows can close, change owner, add a modal, or restart between observation+and input. Per-PID serialization and immediate revalidation reduce the window;+they do not make stale capabilities permanent. Any changed fact invalidates+the decision and requires fresh state.++### Compatibility and rollout++Request schemas and successful action results remain stable. Some calls that+currently post best-effort PID keys will begin refusing when exact delivery is+not provable. That is an intentional safety correction. Refusals include a+recoverable semantic/browser/foreground alternative when one actually exists.++Roll out one route at a time after its fixture lane passes. The rollback is to+disable the new route decision while retaining the same-PID/unresolved-target+guard. Do not restore the unsafe process-scoped fallback to improve success+rates.++Telemetry and retained artifacts may contain route, refusal reason, OS build,+target PID/window ID, timing, and boolean invariants. They must not contain+window titles, field values, typed text, screenshots of unrelated apps,+browser URLs, or fixture logs outside the fixed test vocabulary.++## Alternatives rejected for v1++- **Post to the PID and verify any focused field.** This is the wrong-window+ bug, not a verification strategy.+- **Assume explicit `window_id` makes `post_to_pid` exact.** The keyboard+ transport has no window parameter.+- **Restore without activation, order behind, act, and re-minimize.** This+ changes target state, can still change internal key-window routing, and adds+ cleanup races. It belongs in a later opt-in design.+- **Move the window offscreen or to another Space.** This mutates user desktop+ state and does not solve AX or renderer suspension reliably.+- **Temporarily make a window sticky or add it to the current Space.** This is+ full Spaces manipulation, outside v1's contract.+- **Treat capture success as input exactness.** A CGWindowID image proves the+ capture target, not the destination of a process-scoped key.+- **Treat AX API acceptance or event-post return as confirmation.** Both can+ report success without the intended application effect.+- **Silently use foreground HID.** It violates the background contract and can+ act on the user's current application.+- **Add request echoes to the closed `ActionResult`.** Callers already own the+ request; a wire change would require coordinated strict-SDK migration without+ improving route safety.++## Open implementation questions++Resolve these during Stage 0/1 review, before enabling a route:++1. Which existing read-only output surface should own the additive+ `background_input` capability in every SDK without duplicating native+ window state?+2. Which stable process fingerprint should replace the PID-only Chromium AX+ enablement cache on macOS?+3. Which native text-field properties are sufficient to predeclare a safe+ postcondition for the singleton PID-key rung?+4. Can the current private no-raise pointer preparation satisfy foreground,+ pointer, and Space invariants on every supported macOS build? Builds without+ signed evidence must keep the route disabled.+5. What bounded delayed-verification deadline gives Electron enough time while+ staying below the public tool timeout? Use evidence from the fixture rather+ than reusing an unrelated settle constant blindly.+6. How will the disposable Lume/Namespace image provide a deterministic second+ Space for tests while guaranteeing restoration? This is a test-environment+ decision, not a product Space API.diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/background_input.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/background_input.rsnew file mode 100644index 0000000000..cfd35d2647--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/background_input.rs@@ -0,0 +1,698 @@+//! Pure exact-target decisions for macOS-style background input (v1).+//!+//! Background mutation must carry one exact `(pid, CGWindowID)` target from+//! request to postcondition. The platform shell gathers fresh facts about that+//! target immediately before dispatch and asks this module for exactly one+//! decision: execute the requested route, or refuse with a stable+//! machine-readable reason. The shell never improvises a second actuator after+//! a refusal, and a sibling window's state can never confirm the requested+//! action.+//!+//! This module owns no platform objects and performs no I/O, so the complete+//! state-policy matrix is testable in ordinary CI. See+//! `docs/macos-background-input-v1-plan.md` for the design contract.++use serde_json::{json, Value};++/// The canonical target key: the requested `(pid, CGWindowID)`.+#[derive(Clone, Copy, Debug, PartialEq, Eq)]+pub struct ExactWindowTarget {+ pub pid: i32,+ pub window_id: u32,+}++/// WindowServer's attribution of the requested CGWindowID, resolved+/// immediately before the decision.+#[derive(Clone, Debug, PartialEq, Eq)]+pub enum WindowServerOwnership {+ /// The window exists and the requested pid owns it.+ SamePid,+ /// WindowServer has no record of the id — closed, stale, or fabricated.+ NotFound,+ /// The window exists but another process owns it (out-of-process panels).+ ForeignPid { owner_pid: i32 },+}++/// Whether an explicitly addressed element was proven to belong to the+/// requested window.+#[derive(Clone, Copy, Debug, PartialEq, Eq)]+pub enum ElementAncestry {+ /// The action does not address a retained element.+ NotAddressed,+ /// The live element ascends to an AX window whose CGWindowID is the+ /// requested one.+ ProvenDescendant,+ /// The live element ascends to a different window of the same process.+ OutsideTargetWindow,+ /// Ancestry could not be resolved (dead element, SPI failure). An address+ /// the shell cannot re-prove is not an exact target.+ Unproven,+}++/// Fresh facts about one exact target, gathered by the platform shell.+///+/// Unknown facts must stay unknown: gatherers must not guess a value merely to+/// unlock a route.+#[derive(Clone, Debug, PartialEq, Eq)]+pub struct BackgroundTargetFacts {+ pub window_server: WindowServerOwnership,+ /// The requested CGWindowID is claimed by one of the application's fresh+ /// `AXWindows` (mapped through `_AXUIElementGetWindow`). Absent means+ /// off-Space or AX-unresolved: observation-only.+ pub ax_window_present: bool,+ /// `AXMinimized` on the exact target window. `None` means the attribute+ /// could not be read — an unproven fact, which fails closed for pointer+ /// and keyboard routes (it never unlocks them).+ pub target_minimized: Option<bool>,+ /// `AXHidden` on the owning application. `None` fails closed like+ /// `target_minimized`.+ pub app_hidden: Option<bool>,+ /// Same-pid top-level windows, other than the target, that could receive+ /// process-scoped keyboard input (enumerated from WindowServer so an+ /// off-Space sibling that AX cannot see still counts; proven-minimized+ /// siblings are excluded because they cannot be the key window).+ pub competing_keyboard_destinations: usize,+ /// Ancestry proof for an explicitly addressed element, when one exists.+ pub element: ElementAncestry,+}++/// The background action classes v1 distinguishes. Route order and trust are+/// documented in the plan's routing ladder.+#[derive(Clone, Copy, Debug, PartialEq, Eq)]+pub enum BackgroundAction {+ /// Semantic AX action or value mutation on an exact element+ /// (`AXPress`, `AXConfirm`, selection, `AXValue`).+ AxSemantic,+ /// Window-local routed pointer (stamped CGWindowID; real pointer unmoved).+ WindowPointer,+ /// Process-scoped text insertion (AX selected-text write or CGEvent+ /// keystrokes) with a target-bound value readback.+ InsertText,+ /// Process-scoped key/hotkey without an action-specific verifier.+ GenericKey,+}++impl BackgroundAction {+ fn is_pid_keyboard(self) -> bool {+ matches!(+ self,+ BackgroundAction::InsertText | BackgroundAction::GenericKey+ )+ }+}++/// Stable machine-readable refusal codes. These appear in structured tool+/// results; do not rename them without a compatibility review.+pub mod refusal_codes {+ pub const WINDOW_NOT_FOUND: &str = "window_not_found";+ pub const OWNER_PID_MISMATCH: &str = "owner_pid_mismatch";+ pub const OFF_SPACE_OR_AX_UNRESOLVED: &str = "off_space_or_ax_unresolved";+ pub const MINIMIZED_OR_HIDDEN: &str = "minimized_or_hidden_window";+ pub const SAME_PID_KEYBOARD_AMBIGUITY: &str = "same_pid_keyboard_ambiguity";+ pub const ELEMENT_OUTSIDE_TARGET_WINDOW: &str = "element_outside_target_window";+}++/// A typed refusal: no actuator ran and none may run for this decision.+#[derive(Clone, Debug, PartialEq, Eq)]+pub struct BackgroundRefusal {+ pub code: &'static str,+ pub reason: String,+ /// The safe next route when one actually exists (advice for an explicit+ /// caller decision, never an implicit fallback).+ pub advice: Option<&'static str>,+}++/// Verification binding for an executed route: evidence is acceptable only+/// when it belongs to the exact requested window.+#[derive(Clone, Copy, Debug, PartialEq, Eq)]+pub struct TargetBoundVerification {+ pub window_id: u32,+}++impl TargetBoundVerification {+ /// Whether a readback observed on an element whose resolved window is+ /// `evidence_window_id` may confirm the requested action. Unresolvable+ /// evidence (`None`) is never accepted — it is unverifiable, not proof —+ /// and a sibling window's state can never confirm the requested target.+ pub fn accepts_evidence_from(&self, evidence_window_id: Option<u32>) -> bool {+ evidence_window_id == Some(self.window_id)+ }+}++/// One decision for one gathered-facts snapshot.+#[derive(Clone, Debug, PartialEq, Eq)]+pub enum BackgroundInputDecision {+ Execute {+ /// The only evidence source that may report `confirmed`.+ verification: TargetBoundVerification,+ },+ Refuse(BackgroundRefusal),+}++impl BackgroundInputDecision {+ pub fn is_execute(&self) -> bool {+ matches!(self, BackgroundInputDecision::Execute { .. })+ }+}++fn refuse(+ code: &'static str,+ reason: String,+ advice: Option<&'static str>,+) -> BackgroundInputDecision {+ BackgroundInputDecision::Refuse(BackgroundRefusal {+ code,+ reason,+ advice,+ })+}++/// Decide whether one background action may run against one exact target.+///+/// The rules encode the v1 state-policy matrix:+/// - a stale or foreign CGWindowID refuses every mutation;+/// - a target absent from fresh `AXWindows` (off-Space or AX-unresolved) is+/// observation-only;+/// - an addressed element must prove ancestry to the requested window for+/// every route, including semantic AX;+/// - semantic AX actions remain available for minimized/hidden targets;+/// - the routed pointer requires a visible (possibly occluded) target; and+/// - process-scoped keyboard additionally requires that the target is the+/// only eligible same-pid keyboard destination, because the transport+/// addresses a process, not a window.+pub fn decide_background_input(+ target: ExactWindowTarget,+ facts: &BackgroundTargetFacts,+ action: BackgroundAction,+) -> BackgroundInputDecision {+ match &facts.window_server {+ WindowServerOwnership::NotFound => {+ return refuse(+ refusal_codes::WINDOW_NOT_FOUND,+ format!(+ "WindowServer has no record of window {} (closed or stale); \+ re-enumerate windows and re-target",+ target.window_id+ ),+ None,+ );+ }+ WindowServerOwnership::ForeignPid { owner_pid } => {+ return refuse(+ refusal_codes::OWNER_PID_MISMATCH,+ format!(+ "window {} is owned by pid {owner_pid}, not requested pid {}; \+ re-target the actual owner",+ target.window_id, target.pid+ ),+ None,+ );+ }+ WindowServerOwnership::SamePid => {}+ }++ match facts.element {+ ElementAncestry::NotAddressed | ElementAncestry::ProvenDescendant => {}+ ElementAncestry::OutsideTargetWindow | ElementAncestry::Unproven => {+ return refuse(+ refusal_codes::ELEMENT_OUTSIDE_TARGET_WINDOW,+ format!(+ "the addressed element could not be proven to belong to window {}; \+ take a fresh get_window_state snapshot and re-address it",+ target.window_id+ ),+ Some("get_window_state"),+ );+ }+ }++ if !facts.ax_window_present {+ return refuse(+ refusal_codes::OFF_SPACE_OR_AX_UNRESOLVED,+ format!(+ "window {} is not among the process's current AXWindows (another Space, \+ or its AX surface is unresolved); background input is refused so a \+ sibling window cannot receive it. Observation (capture/list_windows) \+ remains available",+ target.window_id+ ),+ Some("foreground"),+ );+ }++ // Pointer/keyboard require PROVEN not-minimized and not-hidden: an+ // unreadable attribute (`None`) is an unknown fact and unknown facts+ // never unlock a route. Semantic AX stays available either way.+ let visibility_disproven =+ !(facts.target_minimized == Some(false) && facts.app_hidden == Some(false));+ match action {+ BackgroundAction::AxSemantic => BackgroundInputDecision::Execute {+ verification: TargetBoundVerification {+ window_id: target.window_id,+ },+ },+ BackgroundAction::WindowPointer => {+ if visibility_disproven {+ return refuse(+ refusal_codes::MINIMIZED_OR_HIDDEN,+ format!(+ "window {} is minimized or its application is hidden (or that \+ state could not be proven); the routed pointer is refused in \+ v1. Use an exact element action (click/set_value by element) \+ instead",+ target.window_id+ ),+ Some("accessibility"),+ );+ }+ BackgroundInputDecision::Execute {+ verification: TargetBoundVerification {+ window_id: target.window_id,+ },+ }+ }+ BackgroundAction::InsertText | BackgroundAction::GenericKey => {+ if visibility_disproven {+ return refuse(+ refusal_codes::MINIMIZED_OR_HIDDEN,+ format!(+ "window {} is minimized or its application is hidden (or that \+ state could not be proven); raw key input is refused in v1. \+ Use an exact element action (set_value, click \+ action:\"confirm\"/\"press\") instead",+ target.window_id+ ),+ Some("accessibility"),+ );+ }+ debug_assert!(action.is_pid_keyboard());+ if facts.competing_keyboard_destinations > 0 {+ return refuse(+ refusal_codes::SAME_PID_KEYBOARD_AMBIGUITY,+ format!(+ "pid {} owns {} other eligible top-level window(s); process-scoped \+ key events cannot be proven to reach window {} and could mutate a \+ sibling window. Use an exact element action, the page tool for \+ browser content, or delivery_mode:\"foreground\"",+ target.pid, facts.competing_keyboard_destinations, target.window_id+ ),+ Some("accessibility"),+ );+ }+ BackgroundInputDecision::Execute {+ verification: TargetBoundVerification {+ window_id: target.window_id,+ },+ }+ }+ }+}++/// The exact-window resolution string reported by the read-only capability+/// section.+fn exact_window_status(facts: &BackgroundTargetFacts) -> &'static str {+ match &facts.window_server {+ WindowServerOwnership::NotFound => "not_found",+ WindowServerOwnership::ForeignPid { .. } => "owner_mismatch",+ WindowServerOwnership::SamePid if facts.ax_window_present => "matched",+ WindowServerOwnership::SamePid => "ax_unresolved",+ }+}++/// Build the additive read-only `background_input` capability report from the+/// same pure facts that gate every action.+///+/// Semantics: `available` means the route's prerequisites are currently+/// proven, not that a future call is guaranteed to succeed; every action+/// revalidates instead of trusting this report. The report contains no window+/// titles, control values, or private symbol addresses.+pub fn background_input_capability_report(+ target: ExactWindowTarget,+ facts: &BackgroundTargetFacts,+ one_shot_capture_available: Option<bool>,+) -> Value {+ let route_entry = |route: &str, action: BackgroundAction| -> Value {+ match decide_background_input(target, facts, action) {+ BackgroundInputDecision::Execute { .. } => {+ json!({ "route": route, "status": "available" })+ }+ BackgroundInputDecision::Refuse(refusal) => {+ json!({ "route": route, "status": "refused", "reason": refusal.code })+ }+ }+ };+ json!({+ "exact_window": {+ "status": exact_window_status(facts),+ "pid": target.pid,+ "window_id": target.window_id,+ },+ "routes": [+ route_entry("accessibility", BackgroundAction::AxSemantic),+ route_entry("window_pointer", BackgroundAction::WindowPointer),+ route_entry("pid_keyboard", BackgroundAction::GenericKey),+ ],+ "observation": {+ "one_shot_capture": match one_shot_capture_available {+ Some(true) => "available",+ Some(false) => "unavailable",+ None => "unknown",+ },+ "frame_freshness": "unknown",+ },+ })+}++#[cfg(test)]+mod tests {+ use super::*;++ const TARGET: ExactWindowTarget = ExactWindowTarget {+ pid: 42,+ window_id: 700,+ };++ fn matched_facts() -> BackgroundTargetFacts {+ BackgroundTargetFacts {+ window_server: WindowServerOwnership::SamePid,+ ax_window_present: true,+ target_minimized: Some(false),+ app_hidden: Some(false),+ competing_keyboard_destinations: 0,+ element: ElementAncestry::NotAddressed,+ }+ }++ const ALL_ACTIONS: [BackgroundAction; 4] = [+ BackgroundAction::AxSemantic,+ BackgroundAction::WindowPointer,+ BackgroundAction::InsertText,+ BackgroundAction::GenericKey,+ ];++ fn code_of(decision: BackgroundInputDecision) -> &'static str {+ match decision {+ BackgroundInputDecision::Refuse(refusal) => refusal.code,+ BackgroundInputDecision::Execute { .. } => panic!("expected a refusal"),+ }+ }++ /// Regression for the proven same-PID wrong-target case: target window A+ /// requested while sibling window B is the process's focused/eligible+ /// window. Process-scoped keyboard must refuse before dispatch — an+ /// explicit `window_id` is an address, not delivery proof.+ #[test]+ fn two_window_process_refuses_background_keyboard_for_both_text_and_keys() {+ let facts = BackgroundTargetFacts {+ competing_keyboard_destinations: 1,+ ..matched_facts()+ };+ for action in [BackgroundAction::InsertText, BackgroundAction::GenericKey] {+ assert_eq!(+ code_of(decide_background_input(TARGET, &facts, action)),+ refusal_codes::SAME_PID_KEYBOARD_AMBIGUITY,+ "{action:?} must refuse while a sibling keyboard destination exists"+ );+ }+ // Semantic AX and the stamped window-local pointer remain available:+ // they are window-addressed, not process-addressed.+ for action in [+ BackgroundAction::AxSemantic,+ BackgroundAction::WindowPointer,+ ] {+ assert!(+ decide_background_input(TARGET, &facts, action).is_execute(),+ "{action:?} is window-addressed and stays available"+ );+ }+ }++ /// A sibling window's state can never satisfy the requested target's+ /// verification plan, and unresolvable evidence is not proof.+ #[test]+ fn sibling_readback_never_confirms_the_requested_window() {+ let verification = TargetBoundVerification { window_id: 700 };+ assert!(verification.accepts_evidence_from(Some(700)));+ assert!(!verification.accepts_evidence_from(Some(701)), "sibling");+ assert!(!verification.accepts_evidence_from(None), "unresolvable");+ }++ #[test]+ fn stale_window_id_refuses_every_mutation() {+ let facts = BackgroundTargetFacts {+ window_server: WindowServerOwnership::NotFound,+ ax_window_present: false,+ ..matched_facts()+ };+ for action in ALL_ACTIONS {+ assert_eq!(+ code_of(decide_background_input(TARGET, &facts, action)),+ refusal_codes::WINDOW_NOT_FOUND,+ "{action:?}"+ );+ }+ }++ #[test]+ fn foreign_owner_refuses_every_mutation() {+ let facts = BackgroundTargetFacts {+ window_server: WindowServerOwnership::ForeignPid { owner_pid: 900 },+ ..matched_facts()+ };+ for action in ALL_ACTIONS {+ assert_eq!(+ code_of(decide_background_input(TARGET, &facts, action)),+ refusal_codes::OWNER_PID_MISMATCH,+ "{action:?}"+ );+ }+ }++ /// Off-Space / AX-unresolved targets are observation-only: no route may+ /// send input, and no sibling window may be used instead.+ #[test]+ fn ax_unresolved_target_is_observation_only() {+ let facts = BackgroundTargetFacts {+ ax_window_present: false,+ ..matched_facts()+ };+ for action in ALL_ACTIONS {+ assert_eq!(+ code_of(decide_background_input(TARGET, &facts, action)),+ refusal_codes::OFF_SPACE_OR_AX_UNRESOLVED,+ "{action:?}"+ );+ }+ }++ /// Minimized/hidden targets retain exact semantic AX actions; raw keys and+ /// the routed pointer refuse rather than restoring the window.+ #[test]+ fn minimized_or_hidden_keeps_semantic_ax_and_refuses_keys_and_pointer() {+ for facts in [+ BackgroundTargetFacts {+ target_minimized: Some(true),+ ..matched_facts()+ },+ BackgroundTargetFacts {+ app_hidden: Some(true),+ ..matched_facts()+ },+ ] {+ assert!(+ decide_background_input(TARGET, &facts, BackgroundAction::AxSemantic).is_execute()+ );+ for action in [+ BackgroundAction::WindowPointer,+ BackgroundAction::InsertText,+ BackgroundAction::GenericKey,+ ] {+ assert_eq!(+ code_of(decide_background_input(TARGET, &facts, action)),+ refusal_codes::MINIMIZED_OR_HIDDEN,+ "{action:?}"+ );+ }+ }+ }++ /// Unknown visibility fails closed: when minimized/hidden state could not+ /// be read, the routed pointer and raw keys refuse exactly as if the+ /// window were minimized, while exact semantic AX actions remain allowed.+ #[test]+ fn unknown_visibility_fails_closed_for_pointer_and_keyboard() {+ for facts in [+ BackgroundTargetFacts {+ target_minimized: None,+ ..matched_facts()+ },+ BackgroundTargetFacts {+ app_hidden: None,+ ..matched_facts()+ },+ ] {+ assert!(+ decide_background_input(TARGET, &facts, BackgroundAction::AxSemantic).is_execute()+ );+ for action in [+ BackgroundAction::WindowPointer,+ BackgroundAction::InsertText,+ BackgroundAction::GenericKey,+ ] {+ assert_eq!(+ code_of(decide_background_input(TARGET, &facts, action)),+ refusal_codes::MINIMIZED_OR_HIDDEN,+ "{action:?}"+ );+ }+ }+ }++ /// The singleton rule: with no competing keyboard destination, exact+ /// background text/keys are permitted and verification is target-bound.+ #[test]+ fn singleton_window_permits_background_keyboard_with_target_bound_verification() {+ let facts = matched_facts();+ for action in [BackgroundAction::InsertText, BackgroundAction::GenericKey] {+ match decide_background_input(TARGET, &facts, action) {+ BackgroundInputDecision::Execute { verification } => {+ assert_eq!(verification.window_id, TARGET.window_id);+ }+ other => panic!("{action:?} expected Execute, got {other:?}"),+ }+ }+ }++ /// An addressed element that cannot be proven to descend from the+ /// requested window refuses every route — a cached element under the right+ /// cache key is not sufficient after a lifecycle or Space transition.+ #[test]+ fn unproven_element_ancestry_refuses_all_routes() {+ for element in [+ ElementAncestry::OutsideTargetWindow,+ ElementAncestry::Unproven,+ ] {+ let facts = BackgroundTargetFacts {+ element,+ ..matched_facts()+ };+ for action in ALL_ACTIONS {+ assert_eq!(+ code_of(decide_background_input(TARGET, &facts, action)),+ refusal_codes::ELEMENT_OUTSIDE_TARGET_WINDOW,+ "{action:?} with {element:?}"+ );+ }+ }+ }++ #[test]+ fn proven_element_ancestry_executes_semantic_ax_even_when_minimized() {+ let facts = BackgroundTargetFacts {+ element: ElementAncestry::ProvenDescendant,+ target_minimized: Some(true),+ ..matched_facts()+ };+ assert!(decide_background_input(TARGET, &facts, BackgroundAction::AxSemantic).is_execute());+ }++ /// Refusal precedence: exactness failures are reported before state or+ /// cardinality failures so the caller fixes the most fundamental fact.+ #[test]+ fn refusal_precedence_owner_before_element_before_scope_before_state() {+ let worst = BackgroundTargetFacts {+ window_server: WindowServerOwnership::ForeignPid { owner_pid: 9 },+ ax_window_present: false,+ target_minimized: Some(true),+ app_hidden: Some(true),+ competing_keyboard_destinations: 3,+ element: ElementAncestry::OutsideTargetWindow,+ };+ assert_eq!(+ code_of(decide_background_input(+ TARGET,+ &worst,+ BackgroundAction::InsertText+ )),+ refusal_codes::OWNER_PID_MISMATCH+ );+ let bad_element = BackgroundTargetFacts {+ window_server: WindowServerOwnership::SamePid,+ ..worst.clone()+ };+ assert_eq!(+ code_of(decide_background_input(+ TARGET,+ &bad_element,+ BackgroundAction::InsertText+ )),+ refusal_codes::ELEMENT_OUTSIDE_TARGET_WINDOW+ );+ let unresolved = BackgroundTargetFacts {+ element: ElementAncestry::NotAddressed,+ ..bad_element+ };+ assert_eq!(+ code_of(decide_background_input(+ TARGET,+ &unresolved,+ BackgroundAction::InsertText+ )),+ refusal_codes::OFF_SPACE_OR_AX_UNRESOLVED+ );+ let minimized = BackgroundTargetFacts {+ ax_window_present: true,+ ..unresolved+ };+ assert_eq!(+ code_of(decide_background_input(+ TARGET,+ &minimized,+ BackgroundAction::InsertText+ )),+ refusal_codes::MINIMIZED_OR_HIDDEN+ );+ }++ #[test]+ fn capability_report_shape_is_stable() {+ let report = background_input_capability_report(TARGET, &matched_facts(), Some(true));+ assert_eq!(report["exact_window"]["status"], "matched");+ assert_eq!(report["exact_window"]["pid"], 42);+ assert_eq!(report["exact_window"]["window_id"], 700);+ assert_eq!(report["routes"][0]["route"], "accessibility");+ assert_eq!(report["routes"][0]["status"], "available");+ assert_eq!(report["routes"][2]["route"], "pid_keyboard");+ assert_eq!(report["routes"][2]["status"], "available");+ assert_eq!(report["observation"]["one_shot_capture"], "available");+ assert_eq!(report["observation"]["frame_freshness"], "unknown");+ }++ #[test]+ fn capability_report_refusals_carry_stable_reason_codes() {+ let facts = BackgroundTargetFacts {+ ax_window_present: false,+ ..matched_facts()+ };+ let report = background_input_capability_report(TARGET, &facts, Some(false));+ assert_eq!(report["exact_window"]["status"], "ax_unresolved");+ for route in report["routes"].as_array().unwrap() {+ assert_eq!(route["status"], "refused");+ assert_eq!(route["reason"], refusal_codes::OFF_SPACE_OR_AX_UNRESOLVED);+ }+ assert_eq!(report["observation"]["one_shot_capture"], "unavailable");++ let two_windows = BackgroundTargetFacts {+ competing_keyboard_destinations: 1,+ ..matched_facts()+ };+ let report = background_input_capability_report(TARGET, &two_windows, None);+ assert_eq!(report["routes"][0]["status"], "available");+ assert_eq!(report["routes"][2]["status"], "refused");+ assert_eq!(+ report["routes"][2]["reason"],+ refusal_codes::SAME_PID_KEYBOARD_AMBIGUITY+ );+ assert_eq!(report["observation"]["one_shot_capture"], "unknown");+ }+}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rsindex c220f56df5..fca981562b 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs@@ -47,6 +47,7 @@ pub fn parent_liveness_stdin_enabled() -> bool {pub mod action_record;pub mod authorization;+pub mod background_input;pub mod browser;pub mod capture_mode;pub mod capture_scope;diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/enablement.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/enablement.rsnew file mode 100644index 0000000000..762acbb1f6--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/enablement.rs@@ -0,0 +1,133 @@+//! Process-lifetime Chromium/Electron accessibility enablement.+//!+//! Chromium-family apps (Arc, VS Code, Electron shells) ship their web-content+//! AX tree OFF and only build it once an assistive client asks for it. The+//! walker flips `AXManualAccessibility` (falling back to+//! `AXEnhancedUserInterface` only when the modern attribute is unsupported —+//! see [`super::bindings::enable_chromium_accessibility`]) and pays a one-time+//! settle so the asynchronously-built tree exists before it is read.+//!+//! The "already enabled" cache is keyed by the observed process lifetime+//! (pid + kernel start time), not by the numeric pid alone: pids are recycled,+//! and a relaunched Electron app must not inherit a stale "enabled" decision+//! that would skip enablement and return an empty web-content tree.++use std::collections::HashMap;+use std::sync::{Mutex, OnceLock};++use super::bindings::{enable_chromium_accessibility, AXUIElementRef};++/// How long to let a freshly-enabled Chromium/Electron app build its+/// web-content AX tree before we read it. Paid at most once per process+/// lifetime.+const CHROMIUM_SETTLE_SECONDS: f64 = 0.5;++type ProcessStartStamp = (u64, u64);++/// Kernel start time of a process: `(pbi_start_tvsec, pbi_start_tvusec)`.+/// `None` when the process is gone or proc info is unreadable.+fn process_start_stamp(pid: i32) -> Option<ProcessStartStamp> {+ // SAFETY: proc_pidinfo writes at most `size` bytes into `info` and returns+ // the number of bytes filled (<= size) or <= 0 on failure.+ unsafe {+ let mut info: libc::proc_bsdinfo = std::mem::zeroed();+ let size = std::mem::size_of::<libc::proc_bsdinfo>() as libc::c_int;+ let filled = libc::proc_pidinfo(+ pid,+ libc::PROC_PIDTBSDINFO,+ 0,+ &mut info as *mut _ as *mut libc::c_void,+ size,+ );+ if filled != size {+ return None;+ }+ Some((info.pbi_start_tvsec, info.pbi_start_tvusec))+ }+}++/// Process lifetimes for which enablement has already run and the settle has+/// been paid. Values are the process start stamp observed at enablement time.+fn enabled_processes() -> &'static Mutex<HashMap<i32, Option<ProcessStartStamp>>> {+ static ENABLED: OnceLock<Mutex<HashMap<i32, Option<ProcessStartStamp>>>> = OnceLock::new();+ ENABLED.get_or_init(|| Mutex::new(HashMap::new()))+}++fn cached_lifetime_is_current(+ cached: Option<&Option<ProcessStartStamp>>,+ observed: Option<ProcessStartStamp>,+) -> bool {+ cached.is_some_and(|cached| cached.is_some() && *cached == observed)+}++/// Flip Chromium/Electron accessibility on for `pid`'s application element and+/// wait for the web-content tree to settle — once per observed process+/// lifetime. Native Cocoa apps reject the attribute and pay no settle cost.+///+/// # Safety+///+/// `app_element` must be a valid application `AXUIElementRef` for `pid`.+pub unsafe fn ensure_chromium_ax_enabled(pid: i32, app_element: AXUIElementRef) {+ let stamp = process_start_stamp(pid);+ let already_enabled = enabled_processes()+ .lock()+ .map(|cache| cached_lifetime_is_current(cache.get(&pid), stamp))+ .unwrap_or(false);+ if already_enabled {+ return;+ }+ if enable_chromium_accessibility(app_element) {+ crate::permissions::panel::pump_run_loop_briefly(CHROMIUM_SETTLE_SECONDS);+ if let Ok(mut cache) = enabled_processes().lock() {+ cache.insert(pid, stamp);+ }+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn start_stamp_reads_the_current_process_and_rejects_dead_pids() {+ let own = process_start_stamp(std::process::id() as i32);+ assert!(own.is_some(), "own process start time must be readable");+ // Pid 0 is the kernel idle task; proc info for it is not readable from+ // user space, so lifetime keying must fail closed (None).+ assert_eq!(process_start_stamp(0), None);+ }++ #[test]+ fn distinct_lifetimes_do_not_alias() {+ // Two different processes must not produce identical (pid, stamp)+ // cache keys. Use launchd (pid 1) vs our own process.+ let own_pid = std::process::id() as i32;+ let own = process_start_stamp(own_pid);+ let launchd = process_start_stamp(1);+ if let (Some(own), Some(launchd)) = (own, launchd) {+ assert_ne!(+ (own_pid, own),+ (1, launchd),+ "cache keys must differ across processes"+ );+ }+ }++ #[test]+ fn cache_hit_requires_the_same_readable_process_lifetime() {+ let first_launch = Some((100, 10));+ let relaunched = Some((101, 20));++ assert!(cached_lifetime_is_current(+ Some(&first_launch),+ first_launch+ ));+ assert!(+ !cached_lifetime_is_current(Some(&first_launch), relaunched),+ "a relaunched process must not inherit the prior enablement cache entry"+ );+ assert!(!cached_lifetime_is_current(Some(&first_launch), None));+ assert!(!cached_lifetime_is_current(Some(&None), first_launch));+ assert!(!cached_lifetime_is_current(None, first_launch));+ }+}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/exact_target.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/exact_target.rsnew file mode 100644index 0000000000..b2ed078893--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/exact_target.rs@@ -0,0 +1,191 @@+//! Fresh exact-target acquisition for background input decisions.+//!+//! Gathers the facts [`cua_driver_core::background_input`] needs about one+//! requested `(pid, CGWindowID)` immediately before a background mutation:+//! WindowServer ownership, fresh `AXWindows` membership (mapped through+//! `_AXUIElementGetWindow`), minimized/hidden state, competing same-pid+//! keyboard destinations, and addressed-element ancestry. All reads are+//! bounded and fail closed — an unreadable fact never unlocks a route.++use core_foundation::base::{CFRelease, CFTypeRef};+use cua_driver_core::background_input::{+ BackgroundTargetFacts, ElementAncestry, WindowServerOwnership,+};++use super::bindings::{+ ax_get_window_id, copy_ax_windows, copy_bool_attr, copy_element_attr, copy_string_attr,+ focused_element_of_pid, AXUIElementCreateApplication, AXUIElementRef,+};+use crate::windows::{all_windows, resolve_window_owner, WindowOwner};++/// Bounded `AXParent` ascent used when an element does not expose `AXWindow`.+const MAX_ANCESTRY_DEPTH: usize = 40;++/// Resolve the CGWindowID of the top-level AX window that owns `element`.+///+/// Prefers the element's `AXWindow` attribute and falls back to a bounded+/// `AXParent` walk. `None` means ancestry could not be proven — callers must+/// treat that as "not the requested window", never as a wildcard.+///+/// # Safety+///+/// `element` must be a valid `AXUIElementRef` for the duration of the call.+pub unsafe fn element_window_id(element: AXUIElementRef) -> Option<u32> {+ if let Some(window) = copy_element_attr(element, "AXWindow") {+ let window_id = ax_get_window_id(window);+ CFRelease(window as CFTypeRef);+ if window_id.is_some() {+ return window_id;+ }+ }+ // Fallback: ascend AXParent until a window role, then map it.+ let mut current: AXUIElementRef = element;+ let mut owned = false;+ let mut resolved = None;+ for _ in 0..MAX_ANCESTRY_DEPTH {+ match copy_string_attr(current, "AXRole").as_deref() {+ Some("AXWindow") | Some("AXSheet") => {+ resolved = ax_get_window_id(current);+ break;+ }+ Some("AXApplication") | None => break,+ _ => {}+ }+ let parent = copy_element_attr(current, "AXParent");+ if owned {+ CFRelease(current as CFTypeRef);+ }+ match parent {+ Some(parent) => {+ current = parent;+ owned = true;+ }+ None => return None,+ }+ }+ if owned {+ CFRelease(current as CFTypeRef);+ }+ resolved+}++/// The process's focused AX element, but only when it provably belongs to the+/// requested window. Returns a retained element the caller must release.+///+/// This is the only focused-element reader background window-scoped keyboard+/// paths may use: a PID-global focused element can belong to a sibling window,+/// and sibling state must never address or confirm the requested target.+///+/// # Safety+///+/// Caller must `CFRelease` the returned element.+pub unsafe fn focused_element_in_window(pid: i32, window_id: u32) -> Option<AXUIElementRef> {+ let element = focused_element_of_pid(pid)?;+ if element_window_id(element) == Some(window_id) {+ Some(element)+ } else {+ CFRelease(element as CFTypeRef);+ None+ }+}++/// One fresh `AXWindows` row: the mapped CGWindowID plus its minimized state.+/// `minimized: None` means the attribute could not be read — unknown, not+/// "not minimized".+struct AxWindowRecord {+ window_id: u32,+ minimized: Option<bool>,+}++/// Map the application's fresh `AXWindows` through `_AXUIElementGetWindow`.+/// Windows whose id the SPI cannot resolve are omitted: an unmappable window+/// can never satisfy an exact-target requirement.+unsafe fn ax_window_records(app: AXUIElementRef) -> Vec<AxWindowRecord> {+ copy_ax_windows(app)+ .into_iter()+ .filter_map(|window| {+ let record = ax_get_window_id(window).map(|window_id| AxWindowRecord {+ window_id,+ minimized: copy_bool_attr(window, "AXMinimized"),+ });+ CFRelease(window as CFTypeRef);+ record+ })+ .collect()+}++/// Gather fresh background-input facts for one `(pid, window_id)` target.+///+/// `element_ptr` is an optional retained `AXUIElementRef` (as `usize`) for an+/// explicitly addressed element; the caller must keep it retained for the+/// duration of this call. Blocking: performs one CGWindowList enumeration and+/// bounded AX reads. Call from a blocking context immediately before deciding.+pub fn gather_background_facts(+ pid: i32,+ window_id: u32,+ element_ptr: Option<usize>,+) -> BackgroundTargetFacts {+ let window_server = match resolve_window_owner(pid, window_id) {+ WindowOwner::SamePid => WindowServerOwnership::SamePid,+ WindowOwner::Unknown => WindowServerOwnership::NotFound,+ WindowOwner::ForeignPid { owner_pid, .. } => {+ WindowServerOwnership::ForeignPid { owner_pid }+ }+ };++ // SAFETY: the application element is created and released here; window+ // elements are released inside ax_window_records; the caller guarantees+ // element_ptr stays retained.+ let (records, app_hidden, element) = unsafe {+ let app = AXUIElementCreateApplication(pid);+ if app.is_null() {+ (+ Vec::new(),+ None,+ element_ptr.map(|_| ElementAncestry::Unproven),+ )+ } else {+ // Electron/Chromium apps may need per-process-lifetime enablement+ // before their AX windows and subtrees are materialized.+ super::enablement::ensure_chromium_ax_enabled(pid, app);+ let records = ax_window_records(app);+ let app_hidden = copy_bool_attr(app, "AXHidden");+ let element = element_ptr.map(|ptr| match element_window_id(ptr as AXUIElementRef) {+ Some(id) if id == window_id => ElementAncestry::ProvenDescendant,+ Some(_) => ElementAncestry::OutsideTargetWindow,+ None => ElementAncestry::Unproven,+ });+ CFRelease(app as CFTypeRef);+ (records, app_hidden, element)+ }+ };++ let target = records.iter().find(|record| record.window_id == window_id);+ let minimized_ids: Vec<u32> = records+ .iter()+ .filter(|record| record.minimized == Some(true))+ .map(|record| record.window_id)+ .collect();++ // Competing keyboard destinations come from WindowServer, not AX: an+ // off-Space sibling that AXWindows cannot see is still a real key-window+ // candidate for process-scoped events. Proven-minimized windows are+ // excluded — a miniaturized window cannot be the key window.+ let competing_keyboard_destinations = all_windows()+ .iter()+ .filter(|window| {+ window.pid == pid+ && window.window_id != window_id+ && !minimized_ids.contains(&window.window_id)+ })+ .count();++ BackgroundTargetFacts {+ window_server,+ ax_window_present: target.is_some(),+ target_minimized: target.and_then(|record| record.minimized),+ app_hidden,+ competing_keyboard_destinations,+ element: element.unwrap_or(ElementAncestry::NotAddressed),+ }+}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/mod.rsindex 817c8dc7eb..ee7b6c405a 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/ax/mod.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/mod.rs@@ -18,6 +18,8 @@pub mod bindings;pub mod cache;+pub mod enablement;+pub mod exact_target;pub mod tree;pub mod window_scope;diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rsindex 822b3866cc..ee1f5d3a1d 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs@@ -14,8 +14,6 @@use super::bindings::*;use super::window_scope::{decide_window_scope, TopLevelCandidate, WindowScope};use core_foundation::base::{CFEqual, CFRelease, CFRetain, CFTypeRef};-use std::collections::HashSet;-use std::sync::{Mutex, OnceLock};/// Default maximum depth for AX tree walks. Deep menus and complex web views/// can nest deeply; 25 covers realistic app chrome without exploding on@@ -36,14 +34,6 @@ pub const DEFAULT_MAX_DEPTH: usize = 25;/// (issue #22865).pub const DEFAULT_MAX_ELEMENTS: usize = 2_000;-/// How long to let a freshly-enabled Chromium/Electron app build its-/// web-content AX tree before we read it. The tree is materialized-/// asynchronously over IPC once the app detects an assistive client, so a-/// walk that starts immediately sees only the chrome (title bar, a handful-/// of elements). This settle is paid at most once per pid — see-/// `enabled_pids`.-const CHROMIUM_SETTLE_SECONDS: f64 = 0.5;-/// Bound each native AX request. Tokio cannot cancel a blocked/// `AXUIElementCopyAttributeValue` after `spawn_blocking` starts, so the native/// messaging timeout is what keeps an unresponsive app from retaining a worker@@ -54,14 +44,6 @@ unsafe fn set_messaging_timeout(element: AXUIElementRef) {let _ = AXUIElementSetMessagingTimeout(element, AX_MESSAGING_TIMEOUT_SECONDS);}-/// Pids for which we have already flipped on accessibility and paid the-/// one-time settle delay. Repeat snapshots of the same app skip the settle:-/// the tree is already built and stays built for the life of the process.-fn enabled_pids() -> &'static Mutex<HashSet<i32>> {- static ENABLED_PIDS: OnceLock<Mutex<HashSet<i32>>> = OnceLock::new();- ENABLED_PIDS.get_or_init(|| Mutex::new(HashSet::new()))-}-/// A single node in the AX tree.#[derive(Debug, Clone)]pub struct AXNode {@@ -234,20 +216,11 @@ pub fn walk_tree_bounded(// asks for it. Without this, the first walk of such an app returns an// empty/title-bar-only tree (#1616). Flip the enablement attribute,// then — only when the flip actually took and only the first time we- // see this pid — let the asynchronously-built tree settle before we- // read it. Native Cocoa apps reject the attribute, so they pay no- // settle cost. This relies on the MAX_ELEMENTS node cap to keep the- // now-materialized (potentially large) tree bounded.- let already_enabled = enabled_pids()- .lock()- .map(|s| s.contains(&pid))- .unwrap_or(false);- if !already_enabled && enable_chromium_accessibility(app_elem) {- crate::permissions::panel::pump_run_loop_briefly(CHROMIUM_SETTLE_SECONDS);- if let Ok(mut set) = enabled_pids().lock() {- set.insert(pid);- }- }+ // see this process lifetime — let the asynchronously-built tree settle+ // before we read it. Native Cocoa apps reject the attribute, so they+ // pay no settle cost. This relies on the MAX_ELEMENTS node cap to keep+ // the now-materialized (potentially large) tree bounded.+ super::enablement::ensure_chromium_ax_enabled(pid, app_elem);// Union AXChildren + AXWindows — the only way to see background windows.// AXChildren omits windows when the app isn't frontmost (AppKit limitation).diff --git a/libs/cua-driver/rust/crates/platform-macos/src/background_mutation.rs b/libs/cua-driver/rust/crates/platform-macos/src/background_mutation.rsnew file mode 100644index 0000000000..4ca69d9ab5--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-macos/src/background_mutation.rs@@ -0,0 +1,99 @@+//! Per-process serialization for exact-target background mutations.+//!+//! macOS keyboard delivery and focus suppression are process-scoped. Two+//! concurrent window-addressed mutations for one pid must not interleave their+//! fresh target proof, focus changes, dispatch, restoration, or verification.+//! Different processes remain independent.++use std::collections::HashMap;+use std::future::Future;+use std::sync::{Arc, Mutex, OnceLock, Weak};+use tokio::sync::{Mutex as AsyncMutex, OwnedMutexGuard};++tokio::task_local! {+ static HELD_PID: i32;+}++fn process_locks() -> &'static Mutex<HashMap<i32, Weak<AsyncMutex<()>>>> {+ static LOCKS: OnceLock<Mutex<HashMap<i32, Weak<AsyncMutex<()>>>>> = OnceLock::new();+ LOCKS.get_or_init(|| Mutex::new(HashMap::new()))+}++fn process_lock(pid: i32) -> Arc<AsyncMutex<()>> {+ let mut locks = process_locks()+ .lock()+ .unwrap_or_else(|error| error.into_inner());+ locks.retain(|_, lock| lock.strong_count() > 0);+ if let Some(lock) = locks.get(&pid).and_then(Weak::upgrade) {+ return lock;+ }+ let lock = Arc::new(AsyncMutex::new(()));+ locks.insert(pid, Arc::downgrade(&lock));+ lock+}++/// Acquire exclusive background-mutation ownership for one process.+///+/// The returned guard must live from immediately before fresh fact gathering+/// until dispatch, focus restoration, and postcondition verification finish.+pub(crate) async fn acquire(pid: i32) -> OwnedMutexGuard<()> {+ process_lock(pid).lock_owned().await+}++/// Run one nested tool call with non-forgeable proof that its caller already+/// owns this process's mutation lease.+pub(crate) async fn with_held_lease<T>(pid: i32, future: impl Future<Output = T>) -> T {+ HELD_PID.scope(pid, future).await+}++pub(crate) fn held_by_current_task(pid: i32) -> bool {+ HELD_PID+ .try_with(|held_pid| *held_pid == pid)+ .unwrap_or(false)+}++#[cfg(test)]+mod tests {+ use super::*;+ use std::time::Duration;++ #[tokio::test]+ async fn same_pid_mutations_serialize_until_the_first_guard_drops() {+ let first = acquire(-91_001).await;+ let mut waiting = tokio::spawn(acquire(-91_001));++ assert!(+ tokio::time::timeout(Duration::from_millis(30), &mut waiting)+ .await+ .is_err(),+ "a second mutation for the same pid must remain queued"+ );+ drop(first);++ let second = tokio::time::timeout(Duration::from_secs(1), waiting)+ .await+ .expect("same-pid waiter must proceed after the first guard drops")+ .expect("same-pid waiter task must not panic");+ drop(second);+ }++ #[tokio::test]+ async fn different_pids_do_not_block_each_other() {+ let first = acquire(-91_002).await;+ let second = tokio::time::timeout(Duration::from_millis(100), acquire(-91_003))+ .await+ .expect("independent pids must acquire concurrently");+ drop((first, second));+ }++ #[tokio::test]+ async fn nested_lease_proof_is_task_local_and_pid_bound() {+ assert!(!held_by_current_task(-91_004));+ with_held_lease(-91_004, async {+ assert!(held_by_current_task(-91_004));+ assert!(!held_by_current_task(-91_005));+ })+ .await;+ assert!(!held_by_current_task(-91_004));+ }+}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/lib.rs b/libs/cua-driver/rust/crates/platform-macos/src/lib.rsindex 196077bd9e..a75c1dc96f 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/lib.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/lib.rs@@ -11,6 +11,8 @@ pub mod apps;#[cfg(target_os = "macos")]pub mod ax;#[cfg(target_os = "macos")]+mod background_mutation;+#[cfg(target_os = "macos")]pub mod browser;#[cfg(target_os = "macos")]pub mod capture;diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/background_input_regression_tests.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/background_input_regression_tests.rsnew file mode 100644index 0000000000..dc58908dbb--- /dev/null+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/background_input_regression_tests.rs@@ -0,0 +1,149 @@+use super::*;+use async_trait::async_trait;+use cua_driver_core::{+ protocol::ToolResult,+ tool::{Tool, ToolDef},+};+use serde_json::Value;+use std::sync::{+ atomic::{AtomicUsize, Ordering},+ Arc, Mutex,+};+use std::time::Duration;++#[tokio::test]+async fn same_pid_lease_covers_proof_dispatch_restoration_and_verification() {+ const PID: i32 = -92_101;+ let events = Arc::new(Mutex::new(Vec::new()));+ let (first_started_tx, first_started_rx) = tokio::sync::oneshot::channel();+ let (finish_first_tx, finish_first_rx) = tokio::sync::oneshot::channel();++ let first_events = events.clone();+ let first = tokio::spawn(async move {+ let _lease = acquire_background_mutation(PID).await;+ first_events+ .lock()+ .unwrap()+ .extend(["first:proof", "first:dispatch"]);+ first_started_tx.send(()).unwrap();+ finish_first_rx.await.unwrap();+ first_events+ .lock()+ .unwrap()+ .extend(["first:restore", "first:verify"]);+ });+ first_started_rx.await.unwrap();++ let second_events = events.clone();+ let mut second = tokio::spawn(async move {+ let _lease = acquire_background_mutation(PID).await;+ second_events.lock().unwrap().push("second:proof");+ });++ assert!(+ tokio::time::timeout(Duration::from_millis(50), &mut second)+ .await+ .is_err(),+ "the sibling-window mutation must wait through restoration and verification"+ );+ finish_first_tx.send(()).unwrap();+ first.await.unwrap();+ tokio::time::timeout(Duration::from_secs(1), second)+ .await+ .expect("same-pid mutation should proceed after the full lifecycle")+ .unwrap();++ assert_eq!(+ *events.lock().unwrap(),+ [+ "first:proof",+ "first:dispatch",+ "first:restore",+ "first:verify",+ "second:proof",+ ]+ );+}++#[tokio::test]+async fn different_pid_lease_enters_while_first_mutation_is_in_flight() {+ const FIRST_PID: i32 = -92_102;+ const SECOND_PID: i32 = -92_103;+ let first = acquire_background_mutation(FIRST_PID).await;+ let (entered_tx, entered_rx) = tokio::sync::oneshot::channel();++ let second = tokio::spawn(async move {+ let _lease = acquire_background_mutation(SECOND_PID).await;+ entered_tx.send(()).unwrap();+ });++ tokio::time::timeout(Duration::from_secs(1), entered_rx)+ .await+ .expect("a different pid must not wait for the in-flight mutation")+ .unwrap();+ drop(first);+ second.await.unwrap();+}++struct KeyboardProbe {+ calls: Arc<AtomicUsize>,+}++static KEYBOARD_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();++#[async_trait]+impl Tool for KeyboardProbe {+ fn def(&self) -> &ToolDef {+ KEYBOARD_DEF.get_or_init(|| ToolDef {+ name: "press_key".into(),+ description: "keyboard ambiguity regression probe".into(),+ input_schema: serde_json::json!({"type": "object"}),+ read_only: false,+ destructive: true,+ idempotent: false,+ open_world: false,+ })+ }++ async fn invoke(&self, _args: Value) -> ToolResult {+ self.calls.fetch_add(1, Ordering::SeqCst);+ ToolResult::text("keyboard actuator invoked")+ }+}++fn candidate(window_id: u64) -> WindowTargetCandidate {+ WindowTargetCandidate {+ window_id,+ title: format!("Document {window_id}"),+ app_name: Some("Editor".into()),+ is_on_screen: true,+ }+}++#[tokio::test]+async fn pid_only_keyboard_refuses_same_pid_multi_window_ambiguity_before_dispatch() {+ let calls = Arc::new(AtomicUsize::new(0));+ let candidates: WindowTargetCandidates = Arc::new(|pid| {+ (pid == 42)+ .then(|| vec![candidate(7), candidate(8)])+ .unwrap_or_default()+ });+ let keyboard = pid_window_guarded(+ KeyboardProbe {+ calls: calls.clone(),+ },+ &candidates,+ );++ let result = keyboard+ .invoke(serde_json::json!({"pid": 42, "key": "return"}))+ .await;++ assert_eq!(calls.load(Ordering::SeqCst), 0, "no key may be posted");+ assert_eq!(result.is_error, Some(true));+ let refusal = result.structured_content.unwrap();+ assert_eq!(refusal["code"], "ambiguous_window_target");+ assert_eq!(refusal["effect"], "refused");+ assert_eq!(refusal["candidates"][0]["window_id"], 7);+ assert_eq!(refusal["candidates"][1]["window_id"], 8);+}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rsindex 67f038d4dd..59298a27f4 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs@@ -438,6 +438,28 @@ impl Tool for ClickTool {};let element_ptr = element_guard.as_ptr();+ // ── Exact-target background gate (macOS background input v1) ──+ // The element branch is semantic AX delivery, except button=middle+ // which falls back to a routed pixel click at the element's center+ // and is therefore held to the stricter WindowPointer rung. Gate+ // BEFORE any cursor/dispatch work so a stale or sibling-owned+ // target refuses instead of acting on the wrong window.+ let _mutation_lease = if !delivery_mode.is_foreground() {+ let gate_action = if button_str == "middle" {+ cua_driver_core::background_input::BackgroundAction::WindowPointer+ } else {+ cua_driver_core::background_input::BackgroundAction::AxSemantic+ };+ match super::gate_background_window_action(pid, wid, Some(element_ptr), gate_action)+ .await+ {+ Ok(lease) => Some(lease),+ Err(refusal_result) => return refusal_result,+ }+ } else {+ None+ };+// Surface 5: button=right on the AX path → AXShowMenu (the same surface// the dedicated `right_click` tool dispatches). Threads through the// identical perform_ax_click code path with the action remapped.@@ -541,7 +563,7 @@ impl Tool for ClickTool {} else {false};- let selection_pixel = if selection_candidate {+ let mut selection_pixel = if selection_candidate {if let Some((cx, cy)) = center {super::px_frame::resolve_or_refuse(wid).await@@ -558,6 +580,26 @@ impl Tool for ClickTool {} else {None};+ // The selection fallback delivers a routed window-local pixel+ // click — a stricter (WindowPointer) rung than the semantic gate+ // above. In background, drop the fallback rather than silently+ // escalate when the pointer rung would refuse (e.g. a+ // minimized/hidden target); the semantic path still runs.+ if selection_pixel.is_some()+ && !delivery_mode.is_foreground()+ && _mutation_lease+ .as_ref()+ .expect("background element actions hold the per-pid lease")+ .gate_again(+ wid,+ Some(element_ptr),+ cua_driver_core::background_input::BackgroundAction::WindowPointer,+ )+ .await+ .is_err()+ {+ selection_pixel = None;+ }// ── Focus-suppression wrap (Swift WindowChangeDetector + FocusGuard) ──// Capture prior frontmost, arm the wildcard suppressor in the@@ -787,7 +829,29 @@ impl Tool for ClickTool {// CGEventSetWindowLocation in the Chromium recipe.let (screen_x, screen_y, win_local_x, win_local_y) = if let Some(wid) = window_id {match super::px_frame::resolve_or_refuse(wid).await {- Ok(frame) => frame.to_screen(cx, cy),+ Ok(frame) => {+ let (sx, sy, lx, ly) = frame.to_screen(cx, cy);+ // A window-local point outside the live frame would+ // dispatch onto whatever occupies that screen point —+ // the same wrong-surface misclick class as #2237.+ // Refuse in background, where the caller cannot see+ // what is actually under the translated point.+ if !delivery_mode.is_foreground()+ && (lx < 0.0+ || ly < 0.0+ || lx > frame.bounds.width+ || ly > frame.bounds.height)+ {+ return ToolResult::error(format!(+ "click: window-local point ({lx:.1}, {ly:.1}) pt lies outside \+ window {wid}'s {:.0}×{:.0} pt frame; background delivery \+ refused. Re-read coordinates from a fresh get_window_state \+ screenshot.",+ frame.bounds.width, frame.bounds.height+ ));+ }+ (sx, sy, lx, ly)+ }Err(refusal) => return refusal,}} else {@@ -795,6 +859,34 @@ impl Tool for ClickTool {(cx, cy, cx, cy)};+ // ── Exact-target background gate (macOS background input v1) ──+ // A window-addressed background pixel action targets coordinates,+ // which only mean something while the exact window is current and+ // not minimized/hidden: a stale target would let the pid-scoped+ // hit-test or routed events land on a same-process sibling. Gate+ // BEFORE the AX hit-test backend and any cursor/dispatch work.+ // delivery_mode:"foreground" stays the explicit last resort.+ let mutation_lease_held = crate::background_mutation::held_by_current_task(pid);+ let _mutation_lease = if !delivery_mode.is_foreground() && !mutation_lease_held {+ if let Some(wid) = window_id {+ match super::gate_background_window_action(+ pid,+ wid,+ None,+ cua_driver_core::background_input::BackgroundAction::WindowPointer,+ )+ .await+ {+ Ok(lease) => Some(lease),+ Err(refusal_result) => return refusal_result,+ }+ } else {+ None+ }+ } else {+ None+ };+// A background PX action can still use an accessibility delivery// backend after resolving the requested screen point. This keeps// targeting (PX) orthogonal to delivery (AX) and avoids making a@@ -806,10 +898,20 @@ impl Tool for ClickTool {&& modifiers.is_empty(){let focus_only = action == "focus";+ let hit_test_wid = window_id.expect("guarded by window_id.is_some() above");let ax_result = tokio::task::spawn_blocking(move || unsafe {let Some(element) = element_at_screen_position(pid, screen_x, screen_y) else {return Ok::<bool, anyhow::Error>(false);};+ // The pid-scoped hit-test can resolve an element from a+ // same-process sibling overlapping the requested point.+ // Require proven ancestry in the requested window before+ // acting; otherwise fall through to the routed pixel path+ // (already gated for this exact window).+ if crate::ax::exact_target::element_window_id(element) != Some(hit_test_wid) {+ CFRelease(element as _);+ return Ok(false);+ }let delivered = if focus_only {crate::input::ax_actions::focus_element(element as usize).is_ok()} else {diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rsindex b47de84c48..26b03e06e8 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs@@ -16,6 +16,16 @@ pub struct DoubleClickTool {state: Arc<ToolState>,}+fn background_action_for_element(+ has_ax_open: bool,+) -> cua_driver_core::background_input::BackgroundAction {+ if has_ax_open {+ cua_driver_core::background_input::BackgroundAction::AxSemantic+ } else {+ cua_driver_core::background_input::BackgroundAction::WindowPointer+ }+}+impl DoubleClickTool {pub fn new(state: Arc<ToolState>) -> Self {Self { state }@@ -113,11 +123,42 @@ impl Tool for DoubleClickTool {};let element_ptr = element_guard.as_ptr();+ // Choose one background actuator before dispatch. An element that+ // advertises AXOpen uses the exact semantic route; all other+ // elements require the stricter routed-pointer proof. Do not let a+ // failed AXOpen silently cross into an ungated pointer fallback.+ let has_ax_open = tokio::task::spawn_blocking(move || unsafe {+ copy_action_names(element_ptr as AXUIElementRef)+ .iter()+ .any(|action| action == "AXOpen")+ })+ .await+ .unwrap_or(false);+ let _mutation_lease = if !delivery_mode.is_foreground() {+ let action = background_action_for_element(has_ax_open);+ match super::gate_background_window_action(pid, wid, Some(element_ptr), action)+ .await+ {+ Ok(lease) => Some(lease),+ Err(refusal_result) => return refusal_result,+ }+ } else {+ None+ };+// Thread the resolved session cursor key into the blocking AX path// so its ClickPulse lands on THIS session's cursor, not "default".let ck = cursor_key.clone();let result = tokio::task::spawn_blocking(move || {- ax_double_click(pid, wid, element_ptr, idx, &ck)+ ax_double_click(+ pid,+ wid,+ element_ptr,+ idx,+ &ck,+ has_ax_open,+ delivery_mode.is_foreground(),+ )}).await;@@ -154,13 +195,48 @@ impl Tool for DoubleClickTool {// local point as screen-absolute).let (screen_x, screen_y, win_local_x, win_local_y) = if let Some(wid) = window_id {match super::px_frame::resolve_or_refuse(wid).await {- Ok(frame) => frame.to_screen(cx, cy),+ Ok(frame) => {+ let translated = frame.to_screen(cx, cy);+ if !delivery_mode.is_foreground()+ && (translated.2 < 0.0+ || translated.3 < 0.0+ || translated.2 > frame.bounds.width+ || translated.3 > frame.bounds.height)+ {+ return ToolResult::error(format!(+ "double_click: window-local point ({:.1}, {:.1}) pt lies outside \+ window {wid}'s {:.0}×{:.0} pt frame; background delivery refused",+ translated.2, translated.3, frame.bounds.width, frame.bounds.height+ ));+ }+ translated+ }Err(refusal) => return refusal,}} else {(cx, cy, cx, cy)};+ let _mutation_lease = if !delivery_mode.is_foreground() {+ if let Some(wid) = window_id {+ match super::gate_background_window_action(+ pid,+ wid,+ None,+ cua_driver_core::background_input::BackgroundAction::WindowPointer,+ )+ .await+ {+ Ok(lease) => Some(lease),+ Err(refusal_result) => return refusal_result,+ }+ } else {+ None+ }+ } else {+ None+ };+// Pin overlay above the target window before animating.if let Some(wid) = window_id {crate::cursor::overlay::send_command(@@ -235,16 +311,23 @@ fn ax_double_click(element_ptr: usize,idx: usize,cursor_key: &str,+ has_ax_open: bool,+ allow_pointer_fallback: bool,) -> anyhow::Result<String> {let element = element_ptr as AXUIElementRef;// Try AXOpen first (Finder items, openable list rows, document cells).- let actions = unsafe { copy_action_names(element) };- if actions.iter().any(|a| a == "AXOpen") {+ if has_ax_open {let err = unsafe { perform_action(element, "AXOpen") };if err == kAXErrorSuccess {return Ok(format!("AXOpen performed on element [{idx}]."));}+ if !allow_pointer_fallback {+ anyhow::bail!(+ "AXOpen returned {err} for element [{idx}]; background delivery will not \+ improvise a pointer fallback after choosing the semantic route"+ );+ }tracing::debug!("AXOpen returned {err} for element [{idx}], falling back to pixel double-click");@@ -281,3 +364,20 @@ fn ax_double_click("✅ Double-clicked element [{idx}] at ({cx:.1}, {cy:.1})."))}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn background_element_route_does_not_guess_across_actuator_classes() {+ assert_eq!(+ background_action_for_element(true),+ cua_driver_core::background_input::BackgroundAction::AxSemantic+ );+ assert_eq!(+ background_action_for_element(false),+ cua_driver_core::background_input::BackgroundAction::WindowPointer+ );+ }+}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rsindex dc7231bbc1..61a0fe2bbc 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs@@ -584,13 +584,35 @@ impl Tool for GetWindowStateTool {action."));structured["escalation"] = serde_json::json!({- "recommended": "px",- "reason": "act by pixel (x,y) off the screenshot in this response — \- the frame IS the requested window even though its AX \- surface is unresolved."+ "recommended": "foreground",+ "reason": "observation-only: the screenshot in this response IS the \+ requested window, but background input (including px) is \+ refused while its AX surface is unresolved — events could \+ reach a same-process sibling window. Re-snapshot after the \+ app settles, or act with delivery_mode:\"foreground\"."});}}+ // Additive read-only `background_input` capability section (macOS+ // background input v1): the same fresh facts that gate every+ // background mutation, reported per route so an agent can choose+ // before acting. Every action still revalidates — this is advisory,+ // not a promise. Old consumers ignore the extra field.+ {+ let capture_available = screenshot_dims.is_some();+ let report = tokio::task::spawn_blocking(move || {+ let facts = crate::ax::exact_target::gather_background_facts(pid, window_id, None);+ cua_driver_core::background_input::background_input_capability_report(+ cua_driver_core::background_input::ExactWindowTarget { pid, window_id },+ &facts,+ Some(capture_available),+ )+ })+ .await;+ if let Ok(report) = report {+ structured["background_input"] = report;+ }+ }if let Some((sw, sh)) = screenshot_dims {structured["screenshot_width"] = serde_json::json!(sw);structured["screenshot_height"] = serde_json::json!(sh);diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rsindex ccaa6f8243..b2e8438f89 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs@@ -226,6 +226,33 @@ impl Tool for HotkeyTool {return error;}+ // ── Exact-target background gate (macOS background input v1) ──+ // A window-addressed background combo is process-scoped transport: it+ // must prove exact delivery to the requested window (fresh AXWindows+ // membership, not minimized/hidden, no competing same-pid keyboard+ // destination) BEFORE anything is sent — including the px focus+ // click. delivery_mode:"foreground" stays the caller's explicit last+ // resort and is not gated here.+ let _mutation_lease = if !fg {+ if let Some(wid) = window_id {+ match super::gate_background_window_action(+ pid,+ wid,+ None,+ cua_driver_core::background_input::BackgroundAction::GenericKey,+ )+ .await+ {+ Ok(lease) => Some(lease),+ Err(refusal_result) => return refusal_result,+ }+ } else {+ None+ }+ } else {+ None+ };+// px form: focus the field before sending the combo. Foreground delivery// still needs to front the target for the chord itself: the focus helper// restores the previous app before returning.@@ -247,6 +274,7 @@ impl Tool for HotkeyTool {args.opt_str("session"),args.opt_str("_session_id"),from_zoom,+ _mutation_lease.as_ref(),).await{diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rsindex ba22178f91..7839f01daa 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs@@ -107,6 +107,9 @@ mod pid_window_target_tests {}}+#[cfg(test)]+mod background_input_regression_tests;+fn pid_window_guarded<T: Tool + 'static>(tool: T,candidates: &WindowTargetCandidates,@@ -177,6 +180,106 @@ impl DeliveryMode {}}+/// Convert a pure background-input refusal into the structured refusal result+/// shape shared by exact-target tools: `code`, `effect: "refused"`, the+/// requested target, and the safe next route when one exists. No actuator ran.+pub(crate) fn background_refusal_result(+ pid: i32,+ window_id: u32,+ refusal: &cua_driver_core::background_input::BackgroundRefusal,+) -> cua_driver_core::protocol::ToolResult {+ let mut structured = serde_json::json!({+ "code": refusal.code,+ "effect": "refused",+ "pid": pid,+ "window_id": window_id,+ "reason": refusal.reason,+ });+ if let Some(advice) = refusal.advice {+ structured["escalation"] = serde_json::json!({+ "recommended": advice,+ "reason": refusal.reason,+ });+ }+ cua_driver_core::protocol::ToolResult::error(format!(+ "Background input refused ({}): {}",+ refusal.code, refusal.reason+ ))+ .with_structured(structured)+}++/// Exclusive per-process ownership of one background mutation. Callers must+/// keep this value alive through actuator dispatch, focus restoration, and+/// target-bound verification.+pub(crate) struct BackgroundMutationLease {+ pid: i32,+ _guard: tokio::sync::OwnedMutexGuard<()>,+}++impl BackgroundMutationLease {+ /// Revalidate another actuator class while retaining the same per-PID+ /// lease. This supports explicit ladders without deadlocking by attempting+ /// to reacquire the coordinator recursively.+ pub(crate) async fn gate_again(+ &self,+ window_id: u32,+ element_ptr: Option<usize>,+ action: cua_driver_core::background_input::BackgroundAction,+ ) -> Result<(), cua_driver_core::protocol::ToolResult> {+ decide_background_window_action(self.pid, window_id, element_ptr, action).await+ }+}++async fn decide_background_window_action(+ pid: i32,+ window_id: u32,+ element_ptr: Option<usize>,+ action: cua_driver_core::background_input::BackgroundAction,+) -> Result<(), cua_driver_core::protocol::ToolResult> {+ use cua_driver_core::background_input::{+ decide_background_input, BackgroundInputDecision, ExactWindowTarget,+ };+ let facts = match tokio::task::spawn_blocking(move || {+ crate::ax::exact_target::gather_background_facts(pid, window_id, element_ptr)+ })+ .await+ {+ Ok(facts) => facts,+ Err(error) => {+ return Err(cua_driver_core::protocol::ToolResult::error(format!(+ "Could not gather exact-target facts for pid {pid} window {window_id}: {error}"+ )));+ }+ };+ match decide_background_input(ExactWindowTarget { pid, window_id }, &facts, action) {+ BackgroundInputDecision::Execute { .. } => Ok(()),+ BackgroundInputDecision::Refuse(refusal) => {+ Err(background_refusal_result(pid, window_id, &refusal))+ }+ }+}++/// Acquire the per-PID mutation coordinator, gather fresh exact-target facts,+/// and ask the pure core for one background decision. `element_ptr` must stay+/// retained by the caller until the returned lease is dropped.+pub(crate) async fn gate_background_window_action(+ pid: i32,+ window_id: u32,+ element_ptr: Option<usize>,+ action: cua_driver_core::background_input::BackgroundAction,+) -> Result<BackgroundMutationLease, cua_driver_core::protocol::ToolResult> {+ let lease = acquire_background_mutation(pid).await;+ lease.gate_again(window_id, element_ptr, action).await?;+ Ok(lease)+}++pub(crate) async fn acquire_background_mutation(pid: i32) -> BackgroundMutationLease {+ BackgroundMutationLease {+ pid,+ _guard: crate::background_mutation::acquire(pid).await,+ }+}+/// Finish the post-action observation window. Embedded interactive clients/// that already observe the target continuously may opt out through the/// private registry argument to avoid adding a one-second acknowledgement@@ -230,6 +333,7 @@ pub(crate) async fn focus_by_pixel(session: Option<String>,session_id: Option<String>,from_zoom: bool,+ mutation_lease: Option<&BackgroundMutationLease>,) -> Result<(), cua_driver_core::protocol::ToolResult> {use cua_driver_core::tool::Tool;let mut click_args = serde_json::json!({@@ -239,6 +343,15 @@ pub(crate) async fn focus_by_pixel(});if let Some(wid) = window_id {click_args["window_id"] = serde_json::json!(wid);+ if let Some(lease) = mutation_lease {+ lease+ .gate_again(+ wid,+ None,+ cua_driver_core::background_input::BackgroundAction::WindowPointer,+ )+ .await?;+ }}if let Some(ref s) = session {click_args["session"] = serde_json::json!(s);@@ -249,9 +362,13 @@ pub(crate) async fn focus_by_pixel(if from_zoom {click_args["from_zoom"] = serde_json::json!(true);}- let focus = click::ClickTool::new(state.clone())- .invoke(click_args)- .await;+ let click_tool = click::ClickTool::new(state.clone());+ let click = click_tool.invoke(click_args);+ let focus = if let Some(lease) = mutation_lease {+ crate::background_mutation::with_held_lease(lease.pid, click).await+ } else {+ click.await+ };if focus.is_error != Some(true) {// AXFocused is non-destructive: unlike a second real click, it keeps a// Cmd+A selection intact before a follow-up type_text or Cmd+V.diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rsindex d1ad65c245..228ff8848f 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs@@ -76,8 +76,8 @@ fn validate_post_target(pid: i32) -> anyhow::Result<()> {}}-fn read_ax_key_state(pid: i32, element_ptr: usize) -> Option<AxKeyState> {- if super::type_text::target_in_web_area(pid, Some((element_ptr, None))) {+fn read_ax_key_state(pid: i32, window_id: Option<u32>, element_ptr: usize) -> Option<AxKeyState> {+ if super::type_text::target_in_web_area(pid, Some((element_ptr, None)), window_id) {return None;}let element = element_ptr as AXUIElementRef;@@ -90,16 +90,22 @@ fn read_ax_key_state(pid: i32, element_ptr: usize) -> Option<AxKeyState> {fn dispatch_with_ax_oracle(pid: i32,+ window_id: Option<u32>,explicit_element_ptr: Option<usize>,dispatch: impl FnOnce() -> anyhow::Result<()>,) -> anyhow::Result<bool> {let (element_ptr, owns_element) = match explicit_element_ptr {Some(ptr) => (Some(ptr), false),- None => unsafe { focused_element_of_pid(pid) }- .map(|element| (Some(element as usize), true))- .unwrap_or((None, false)),+ None => unsafe {+ match window_id {+ Some(wid) => crate::ax::exact_target::focused_element_in_window(pid, wid),+ None => focused_element_of_pid(pid),+ }+ }+ .map(|element| (Some(element as usize), true))+ .unwrap_or((None, false)),};- let before = element_ptr.and_then(|ptr| read_ax_key_state(pid, ptr));+ let before = element_ptr.and_then(|ptr| read_ax_key_state(pid, window_id, ptr));let result = dispatch();// Native controls normally publish their new value/selection on the next// run-loop turn. Keep this bounded and reuse the exact retained element so@@ -107,7 +113,7 @@ fn dispatch_with_ax_oracle(if before.is_some() {std::thread::sleep(std::time::Duration::from_millis(60));}- let after = element_ptr.and_then(|ptr| read_ax_key_state(pid, ptr));+ let after = element_ptr.and_then(|ptr| read_ax_key_state(pid, window_id, ptr));if owns_element {if let Some(ptr) = element_ptr {unsafe { CFRelease(ptr as _) };@@ -308,18 +314,69 @@ impl Tool for PressKeyTool {let delivery_mode = super::DeliveryMode::parse(args.opt_str("delivery_mode").as_deref());let fg = delivery_mode.is_foreground();+ // Argument-shape errors are reported before any gating or retained+ // lookups: a malformed call must fail the same way regardless of+ // background-target state.+ let px = args.get("x").and_then(|v| v.as_f64());+ let py = args.get("y").and_then(|v| v.as_f64());+ if px.is_some() && py.is_some() && element_index.is_some() {+ return ToolResult::error(+ "Pass either element_index (ax) or x,y (px) to press_key, not both.",+ );+ }++ // Resolve the pre-focus element pointer (if requested) outside+ // the suppression closure — only the focus_element() write itself+ // needs to run under suppression, the cache lookup does not.+ // Retain out of the cache so a concurrent get_window_state can't free+ // the element before the suppressed focus below dereferences it+ // (use-after-free → daemon crash). Guard lives to method end.+ let pre_focus_guard = if let (Some(idx), Some(wid)) = (element_index, window_id) {+ match self.state.element_cache.get_element_retained(pid, wid, idx) {+ Some(guard) => Some(guard),+ None => {+ return ToolResult::error(format!(+ "Element index {idx} not found. Call get_window_state first."+ ));+ }+ }+ } else {+ None+ };+ let pre_focus_ptr: Option<usize> = pre_focus_guard.as_ref().map(|g| g.as_ptr());++ // ── Exact-target background gate (macOS background input v1) ──+ // A window-addressed background key is process-scoped transport: it+ // must prove exact delivery to the requested window (fresh AXWindows+ // membership, not minimized/hidden, no competing same-pid keyboard+ // destination, proven element ancestry) BEFORE anything is sent —+ // including the px focus click. delivery_mode:"foreground" stays the+ // caller's explicit last resort and is not gated here.+ let _mutation_lease = if !fg {+ if let Some(wid) = window_id {+ match super::gate_background_window_action(+ pid,+ wid,+ pre_focus_ptr,+ cua_driver_core::background_input::BackgroundAction::GenericKey,+ )+ .await+ {+ Ok(lease) => Some(lease),+ Err(refusal_result) => return refusal_result,+ }+ } else {+ None+ }+ } else {+ None+ };+// px form: pixel-click to focus, then the key goes to the focused element.// Reuses click's translation + delivery_mode; after it, deliver via the// plain background path (the focus-click already handled fronting if fg).let px_focus = {- let px = args.get("x").and_then(|v| v.as_f64());- let py = args.get("y").and_then(|v| v.as_f64());if let (Some(cx), Some(cy)) = (px, py) {- if element_index.is_some() {- return ToolResult::error(- "Pass either element_index (ax) or x,y (px) to press_key, not both.",- );- }let from_zoom = args.get("from_zoom").and_then(|v| v.as_bool())@@ -334,6 +391,7 @@ impl Tool for PressKeyTool {args.opt_str("session"),args.opt_str("_session_id"),from_zoom,+ _mutation_lease.as_ref(),).await{@@ -345,19 +403,6 @@ impl Tool for PressKeyTool {}};- // Resolve the pre-focus element pointer (if requested) outside- // the suppression closure — only the focus_element() write itself- // needs to run under suppression, the cache lookup does not.- // Retain out of the cache so a concurrent get_window_state can't free- // the element before the suppressed focus below dereferences it- // (use-after-free → daemon crash). Guard lives to method end.- let pre_focus_guard = if let (Some(idx), Some(wid)) = (element_index, window_id) {- self.state.element_cache.get_element_retained(pid, wid, idx)- } else {- None- };- let pre_focus_ptr: Option<usize> = pre_focus_guard.as_ref().map(|g| g.as_ptr());-// ── Focus-suppression wrap (Swift WindowChangeDetector + FocusGuard) ──// Single-key presses can fire autocomplete (Return on a search// box opens a results popover) or trigger menu shortcuts that@@ -398,7 +443,7 @@ impl Tool for PressKeyTool {"delivery_mode=foreground requires window_id for press_key")})?;- return dispatch_with_ax_oracle(pid, pre_focus_ptr, || {+ return dispatch_with_ax_oracle(pid, window_id, pre_focus_ptr, || {crate::input::skylight::with_foreground_hid_activation(pid as libc::pid_t,wid,@@ -417,7 +462,7 @@ impl Tool for PressKeyTool {});}// background (default): auth-envelope post, no raise.- dispatch_with_ax_oracle(pid, pre_focus_ptr, || {+ dispatch_with_ax_oracle(pid, window_id, pre_focus_ptr, || {crate::input::keyboard::press_key(pid, &key, &m)})})diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rsindex 28787bbaea..c32d343013 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs@@ -153,6 +153,18 @@ impl Tool for RightClickTool {};let element_ptr = element_guard.as_ptr();+ let _mutation_lease = match super::gate_background_window_action(+ pid,+ wid,+ Some(element_ptr),+ cua_driver_core::background_input::BackgroundAction::AxSemantic,+ )+ .await+ {+ Ok(lease) => lease,+ Err(refusal_result) => return refusal_result,+ };+let result =tokio::task::spawn_blocking(move || ax_show_menu(element_ptr, idx, pid, wid)).await;@@ -176,13 +188,48 @@ impl Tool for RightClickTool {// refuses a window with no live frame).let (screen_x, screen_y, win_local_x, win_local_y) = if let Some(wid) = window_id {match super::px_frame::resolve_or_refuse(wid).await {- Ok(frame) => frame.to_screen(cx, cy),+ Ok(frame) => {+ let translated = frame.to_screen(cx, cy);+ if !delivery_mode.is_foreground()+ && (translated.2 < 0.0+ || translated.3 < 0.0+ || translated.2 > frame.bounds.width+ || translated.3 > frame.bounds.height)+ {+ return ToolResult::error(format!(+ "right_click: window-local point ({:.1}, {:.1}) pt lies outside \+ window {wid}'s {:.0}×{:.0} pt frame; background delivery refused",+ translated.2, translated.3, frame.bounds.width, frame.bounds.height+ ));+ }+ translated+ }Err(refusal) => return refusal,}} else {(cx, cy, cx, cy)};+ let _mutation_lease = if !delivery_mode.is_foreground() {+ if let Some(wid) = window_id {+ match super::gate_background_window_action(+ pid,+ wid,+ None,+ cua_driver_core::background_input::BackgroundAction::WindowPointer,+ )+ .await+ {+ Ok(lease) => Some(lease),+ Err(refusal_result) => return refusal_result,+ }+ } else {+ None+ }+ } else {+ None+ };+// Pin overlay above the target window before animating.if let Some(wid) = window_id {crate::cursor::overlay::send_command(diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rsindex 0930888257..c788ee7af0 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs@@ -34,6 +34,14 @@ struct WheelTarget {wid: Option<u32>,}+fn after_exact_target_gate<T>(+ gate: Result<(), ToolResult>,+ action: impl FnOnce() -> T,+) -> Result<T, ToolResult> {+ gate?;+ Ok(action())+}+pub struct ScrollTool {state: Arc<ToolState>,}@@ -226,12 +234,39 @@ impl Tool for ScrollTool {));}}+ let mut _mutation_lease: Option<super::BackgroundMutationLease> = None;// AppKit exposes vertical scroll-bar buttons beneath the text area's// AXScrollArea parent. Pressing those controls is a true// background-safe scroll: no activation, z-order change, or cursor move.if matches!(direction.as_str(), "up" | "down") {if let (Some(index), Some(wid)) = (element_index, window_id) {+ if !delivery_mode.is_foreground() {+ if let Some(lease) = _mutation_lease.as_ref() {+ if let Err(refusal_result) = lease+ .gate_again(+ wid,+ pre_focus_ptr,+ cua_driver_core::background_input::BackgroundAction::AxSemantic,+ )+ .await+ {+ return refusal_result;+ }+ } else {+ match super::gate_background_window_action(+ pid,+ wid,+ pre_focus_ptr,+ cua_driver_core::background_input::BackgroundAction::AxSemantic,+ )+ .await+ {+ Ok(lease) => _mutation_lease = Some(lease),+ Err(refusal_result) => return refusal_result,+ }+ }+ }let native_element_guard = self.state.element_cache@@ -332,25 +367,61 @@ impl Tool for ScrollTool {// Resolve a screen-space wheel target, if a target was supplied.let wheel_target: Option<WheelTarget> = if let Some(element_ptr) = pre_focus_ptr {+ // Revealing an element is itself an AX mutation. Prove that the+ // cached element still belongs to the exact requested window before+ // AXScrollToVisible for every direction, then keep the lease for the+ // stricter pointer revalidation below.+ let semantic_gate = if !delivery_mode.is_foreground() {+ if let Some(wid) = window_id {+ if let Some(lease) = _mutation_lease.as_ref() {+ lease+ .gate_again(+ wid,+ Some(element_ptr),+ cua_driver_core::background_input::BackgroundAction::AxSemantic,+ )+ .await+ } else {+ match super::gate_background_window_action(+ pid,+ wid,+ Some(element_ptr),+ cua_driver_core::background_input::BackgroundAction::AxSemantic,+ )+ .await+ {+ Ok(lease) => {+ _mutation_lease = Some(lease);+ Ok(())+ }+ Err(refusal) => Err(refusal),+ }+ }+ } else {+ Ok(())+ }+ } else {+ Ok(())+ };// Element path: wheel at the element's screen-space center. Both AX// coordinates and window bounds are logical top-left points, so no// Retina scaling is needed here.let wid = window_id;- tokio::task::spawn_blocking(move || {+ let target_task = tokio::task::spawn_blocking(move || {// Web content can be present in AX while its frame is below// the outer page viewport. Ask the accessibility hierarchy to// reveal the target before taking the screen-space center;// otherwise the wheel is posted outside the rendered window// and nested overflow regions never receive it.- unsafe {- let _ = crate::ax::bindings::perform_action(+ after_exact_target_gate(semantic_gate, || unsafe {+ crate::ax::bindings::perform_action(element_ptr as AXUIElementRef,"AXScrollToVisible",- );- }+ )+ })?;std::thread::sleep(std::time::Duration::from_millis(40));let center = unsafe { element_screen_center(element_ptr as AXUIElementRef) };- center.map(|(cx, cy)| {+ Ok(center.map(|(cx, cy)| {let win_local = wid.and_then(crate::windows::window_bounds_by_id).map(|b| (cx - b.x, cy - b.y));@@ -360,11 +431,13 @@ impl Tool for ScrollTool {win_local,wid,}- })- })- .await- .ok()- .flatten()+ }))+ });+ match target_task.await {+ Ok(Ok(target)) => target,+ Ok(Err(refusal)) => return refusal,+ Err(_) => None,+ }} else if let (Some(mut cx), Some(mut cy)) = (x_arg, y_arg) {// Targeted x,y are window-local screenshot pixels and REQUIRE a// window_id to anchor the window→screen conversion (schema contract).@@ -406,6 +479,49 @@ impl Tool for ScrollTool {};if let Some(target) = wheel_target {+ if !delivery_mode.is_foreground() {+ if let Some(wid) = target.wid {+ if let (Some((lx, ly)), Some(bounds)) =+ (target.win_local, crate::windows::window_bounds_by_id(wid))+ {+ if lx < 0.0 || ly < 0.0 || lx > bounds.width || ly > bounds.height {+ return ToolResult::error(format!(+ "scroll: window-local point ({lx:.1}, {ly:.1}) pt lies outside \+ window {wid}'s {:.0}×{:.0} pt frame; background delivery \+ refused",+ bounds.width, bounds.height+ ));+ }+ }+ // The semantic reveal gate does not authorize pointer+ // delivery. Revalidate the stricter route immediately+ // before proceeding to wheel dispatch.+ if let Some(lease) = _mutation_lease.as_ref() {+ if let Err(refusal_result) = lease+ .gate_again(+ wid,+ pre_focus_ptr,+ cua_driver_core::background_input::BackgroundAction::WindowPointer,+ )+ .await+ {+ return refusal_result;+ }+ } else {+ match super::gate_background_window_action(+ pid,+ wid,+ pre_focus_ptr,+ cua_driver_core::background_input::BackgroundAction::WindowPointer,+ )+ .await+ {+ Ok(lease) => _mutation_lease = Some(lease),+ Err(refusal_result) => return refusal_result,+ }+ }+ }+ }let cursor_key = super::cursor_tools::resolve_cursor_key(&args);// Pin + glide the agent-cursor overlay to the target for visibility// (overlay only — does NOT move the hardware cursor). Mirrors click.@@ -506,6 +622,35 @@ impl Tool for ScrollTool {};let key = key.to_owned();+ if !delivery_mode.is_foreground() {+ if let Some(wid) = window_id {+ if let Some(lease) = _mutation_lease.as_ref() {+ if let Err(refusal_result) = lease+ .gate_again(+ wid,+ pre_focus_ptr,+ cua_driver_core::background_input::BackgroundAction::GenericKey,+ )+ .await+ {+ return refusal_result;+ }+ } else {+ match super::gate_background_window_action(+ pid,+ wid,+ pre_focus_ptr,+ cua_driver_core::background_input::BackgroundAction::GenericKey,+ )+ .await+ {+ Ok(lease) => _mutation_lease = Some(lease),+ Err(refusal_result) => return refusal_result,+ }+ }+ }+ }+// ── Focus-suppression wrap (Swift WindowChangeDetector + FocusGuard) ──// Scroll keystrokes (PageDown / arrow) into search-box autocomplete// can spawn floating helper windows; rare but real. Wrap for parity@@ -624,3 +769,43 @@ unsafe fn collect_ax_buttons(}}}++#[cfg(test)]+mod tests {+ use super::*;+ use cua_driver_core::background_input::{+ decide_background_input, BackgroundAction, BackgroundInputDecision, BackgroundTargetFacts,+ ElementAncestry, ExactWindowTarget, WindowServerOwnership,+ };+ use std::sync::atomic::{AtomicBool, Ordering};++ #[test]+ fn exact_target_refusal_prevents_ax_reveal() {+ let action_ran = AtomicBool::new(false);+ let target = ExactWindowTarget {+ pid: 42,+ window_id: 7,+ };+ let facts = BackgroundTargetFacts {+ window_server: WindowServerOwnership::SamePid,+ ax_window_present: true,+ target_minimized: Some(false),+ app_hidden: Some(false),+ competing_keyboard_destinations: 0,+ element: ElementAncestry::OutsideTargetWindow,+ };+ let refusal = match decide_background_input(target, &facts, BackgroundAction::AxSemantic) {+ BackgroundInputDecision::Refuse(refusal) => Err(+ super::super::background_refusal_result(target.pid, target.window_id, &refusal),+ ),+ BackgroundInputDecision::Execute { .. } => panic!("exact-target facts must refuse"),+ };++ let result = after_exact_target_gate(refusal, || {+ action_ran.store(true, Ordering::SeqCst);+ });++ assert!(result.is_err());+ assert!(!action_ran.load(Ordering::SeqCst));+ }+}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rsindex 5d8c8595cb..340b15fd35 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs@@ -160,6 +160,23 @@ impl Tool for SetValueTool {}};let element_ptr = element_guard.as_ptr();++ // set_value is an always-background semantic AX mutation. Re-prove+ // that the retained element still belongs to the requested exact+ // window immediately before any cursor or AX work; a cache hit alone+ // is not delivery proof after a window lifecycle or Space change.+ let _mutation_lease = match super::gate_background_window_action(+ pid,+ window_id,+ Some(element_ptr),+ cua_driver_core::background_input::BackgroundAction::AxSemantic,+ )+ .await+ {+ Ok(lease) => lease,+ Err(refusal_result) => return refusal_result,+ };+let cursor_key = super::cursor_tools::resolve_cursor_key(&args);let center_ptr = element_ptr as usize;if let Ok(Some((screen_x, screen_y))) = tokio::task::spawn_blocking(move || unsafe {@@ -181,8 +198,11 @@ impl Tool for SetValueTool {// the renderer never observes it. Reuse type_text's bounded ancestor// check so native browser chrome stays trusted but rendered content is// always reported as unverified.- let ax_echo_surface =- super::type_text::target_in_web_area(pid, Some((element_ptr, Some(element_index))));+ let ax_echo_surface = super::type_text::target_in_web_area(+ pid,+ Some((element_ptr, Some(element_index))),+ Some(window_id),+ );// ── Focus-suppression wrap (Swift WindowChangeDetector + FocusGuard) ──// AXValue writes on popups / sliders can cause reflex activationsdiff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rsindex 893e03009e..98fe058c58 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs@@ -38,6 +38,7 @@ use crate::ax::bindings::{use crate::focus_guard;use crate::window_change_detector::WindowChangeDetector;use core_foundation::base::CFRelease;+use cua_driver_core::background_input::BackgroundRefusal;use super::ToolState;@@ -228,6 +229,59 @@ impl Tool for TypeTextTool {return error;}+ // Validate element_index requires window_id (still applies for+ // the legacy integer path; token path already resolved window_id).+ if element_index.is_some() && window_id.is_none() {+ return ToolResult::error("window_id is required when element_index is used.");+ }++ // Argument-shape errors are reported before any gating or retained+ // lookups: a malformed call must fail the same way regardless of+ // background-target state.+ let px = args.get("x").and_then(|v| v.as_f64());+ let py = args.get("y").and_then(|v| v.as_f64());+ if px.is_some() && py.is_some() && element_index.is_some() {+ return ToolResult::error(+ "Pass either element_index (ax) or x,y (px) to type_text, not both.",+ );+ }++ // Resolve the element pointer (if element_index given). Retain it out+ // of the cache so a concurrent get_window_state can't free it before+ // the blocking type below dereferences it (use-after-free → daemon+ // crash). The guard lives to method end, past type_text_blocking.+ let element_guard = if let (Some(idx), Some(wid)) = (element_index, window_id) {+ match self.state.element_cache.get_element_retained(pid, wid, idx) {+ Some(e) => Some((e, idx)),+ None => {+ return ToolResult::error(format!(+ "Element index {idx} not found. Call get_window_state first."+ ))+ }+ }+ } else {+ None+ };++ // ── Exact-target background gate (macOS background input v1) ──+ // A window-addressed background insert must prove exact delivery+ // before any input — including the px focus click — is sent. The pure+ // core decides once from fresh facts: full keyboard ladder, semantic+ // AX write only (exact element, no CGEvent fallback), or a structured+ // refusal. delivery_mode:"foreground" stays the caller's explicit+ // last resort and is not gated here.+ let (_mutation_lease, keyboard_policy) =+ if !delivery_mode.is_foreground() && window_id.is_some() {+ let wid = window_id.expect("checked above");+ let gate_element_ptr = element_guard.as_ref().map(|(g, _)| g.as_ptr() as usize);+ match background_keyboard_policy(pid, wid, gate_element_ptr).await {+ Ok((lease, policy)) => (Some(lease), policy),+ Err(refusal_result) => return refusal_result,+ }+ } else {+ (None, BackgroundKeyboardPolicy::Allowed)+ };+// ── px form: focus by pixel-click, then type into the focused element ──// Pass x,y (no element_index) for an *element px action*: pixel-click the// field to give the Chromium/Electron renderer the real keyboard focus the@@ -235,14 +289,13 @@ impl Tool for TypeTextTool {// escalates AX → CGEvent and lands once focused). Reuses ClickTool's exact// coordinate translation + delivery_mode, so it lands on the same pixel a// px-click would.- let px = args.get("x").and_then(|v| v.as_f64());- let py = args.get("y").and_then(|v| v.as_f64());let used_pixel_focus = px.is_some() && py.is_some();if let (Some(cx), Some(cy)) = (px, py) {- if element_index.is_some() {- return ToolResult::error(- "Pass either element_index (ax) or x,y (px) to type_text, not both.",- );+ // The px form has no exact element for a semantic-only write; when+ // the keyboard rung is refused, refuse before the focus click too.+ if let BackgroundKeyboardPolicy::SemanticOnly(ref refusal) = keyboard_policy {+ let wid = window_id.expect("gate ran only with window_id");+ return super::background_refusal_result(pid, wid, refusal);}let from_zoom = args.get("from_zoom")@@ -258,6 +311,7 @@ impl Tool for TypeTextTool {args.opt_str("session"),args.opt_str("_session_id"),from_zoom,+ _mutation_lease.as_ref(),).await{@@ -266,29 +320,6 @@ impl Tool for TypeTextTool {// element_index stays None → the type path below writes to the now-// focused element via the CGEvent (key_events) rung.}-- // Validate element_index requires window_id (still applies for- // the legacy integer path; token path already resolved window_id).- if element_index.is_some() && window_id.is_none() {- return ToolResult::error("window_id is required when element_index is used.");- }-- // Resolve the element pointer (if element_index given). Retain it out- // of the cache so a concurrent get_window_state can't free it before- // the blocking type below dereferences it (use-after-free → daemon- // crash). The guard lives to method end, past type_text_blocking.- let element_guard = if let (Some(idx), Some(wid)) = (element_index, window_id) {- match self.state.element_cache.get_element_retained(pid, wid, idx) {- Some(e) => Some((e, idx)),- None => {- return ToolResult::error(format!(- "Element index {idx} not found. Call get_window_state first."- ))- }- }- } else {- None- };if let (Some((element, _)), Some(wid)) = (element_guard.as_ref(), window_id) {let center_ptr = element.as_ptr() as usize;if let Ok(Some((screen_x, screen_y))) = tokio::task::spawn_blocking(move || unsafe {@@ -330,6 +361,7 @@ impl Tool for TypeTextTool {// "success but nothing typed" symptom.let is_terminal_target = crate::terminal::is_terminal_pid(pid);+ let blocking_policy = keyboard_policy.clone();let result = focus_guard::with_focus_suppressed(Some(pid),prior_front,@@ -344,6 +376,7 @@ impl Tool for TypeTextTool {is_terminal_target,delivery_mode,window_id,+ blocking_policy,)}).await@@ -353,6 +386,18 @@ impl Tool for TypeTextTool {let changes = super::finish_window_observation(snapshot, &args).await;+ // Unwrap the delivery envelope: a structured refusal means no+ // actuator ran and the caller gets the exact reason.+ let result = match result {+ Ok(Ok(TypeTextDelivery::Refused(refusal))) => {+ let wid = window_id.expect("background refusals require a window target");+ return super::background_refusal_result(pid, wid, &refusal);+ }+ Ok(Ok(TypeTextDelivery::Typed(outcome))) => Ok(Ok(outcome)),+ Ok(Err(error)) => Ok(Err(error)),+ Err(error) => Err(error),+ };+match result {Ok(Ok(outcome)) if outcome.delivered_chars.is_some_and(|n| n < char_count) => {let delivered_chars = outcome.delivered_chars.unwrap_or_default();@@ -389,7 +434,7 @@ impl Tool for TypeTextTool {// native types and already-unverified deliveries pay nothing.let target_is_web_content = verified&& path_has_untrusted_web_readback(path)- && target_in_web_area(pid, element_ptr);+ && target_in_web_area(pid, element_ptr, window_id);let verification = surface_verification(path, verified, target_is_web_content);let verified = verification.verified;let untrusted_web_readback = verification.untrusted_web_readback;@@ -547,6 +592,73 @@ fn web_readback_next_rung(is_electron: bool, used_pixel_focus: bool) -> Option<&const DELIVERY_DRAIN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(2);const DELIVERY_DRAIN_POLL_INTERVAL: std::time::Duration = std::time::Duration::from_millis(10);+/// Keyboard-rung policy for one window-addressed background insert, decided+/// once by the pure exact-target core before any input is posted.+#[derive(Clone, Debug)]+enum BackgroundKeyboardPolicy {+ /// The full ladder may run: foreground requests, pid-only targets, and+ /// window targets whose exact keyboard delivery is proven (singleton+ /// same-pid destination).+ Allowed,+ /// Only the semantic AX write on the proven exact element may run. The+ /// process-scoped CGEvent rung is refused with this refusal — carried so+ /// the exact reason is returned if the AX write does not land.+ SemanticOnly(BackgroundRefusal),+}++/// Delivery envelope for `type_text_blocking`: either an actuator ran+/// (`Typed`), or the exact-target decision refused before any event was+/// posted (`Refused`) and the caller must return the structured refusal.+enum TypeTextDelivery {+ Typed(TypeTextOutcome),+ Refused(BackgroundRefusal),+}++/// Decide the keyboard policy for a window-addressed background `type_text`.+///+/// Gathers fresh exact-target facts once and asks the pure core:+/// - `InsertText` executes → the full ladder is `Allowed`;+/// - `InsertText` refused but the caller addressed an exact element whose+/// ancestry is proven and semantic AX executes → `SemanticOnly`;+/// - otherwise the structured refusal result is returned and no input of any+/// kind (including a px focus click) may be sent.+async fn background_keyboard_policy(+ pid: i32,+ window_id: u32,+ element_ptr: Option<usize>,+) -> Result<(super::BackgroundMutationLease, BackgroundKeyboardPolicy), ToolResult> {+ use cua_driver_core::background_input::{+ decide_background_input, BackgroundAction, BackgroundInputDecision, ExactWindowTarget,+ };+ let lease = super::acquire_background_mutation(pid).await;+ let facts = match tokio::task::spawn_blocking(move || {+ crate::ax::exact_target::gather_background_facts(pid, window_id, element_ptr)+ })+ .await+ {+ Ok(facts) => facts,+ Err(error) => {+ return Err(ToolResult::error(format!(+ "Could not gather exact-target facts for pid {pid} window {window_id}: {error}"+ )));+ }+ };+ let target = ExactWindowTarget { pid, window_id };+ match decide_background_input(target, &facts, BackgroundAction::InsertText) {+ BackgroundInputDecision::Execute { .. } => Ok((lease, BackgroundKeyboardPolicy::Allowed)),+ BackgroundInputDecision::Refuse(refusal) => {+ let semantic_available = element_ptr.is_some()+ && decide_background_input(target, &facts, BackgroundAction::AxSemantic)+ .is_execute();+ if semantic_available {+ Ok((lease, BackgroundKeyboardPolicy::SemanticOnly(refusal)))+ } else {+ Err(super::background_refusal_result(pid, window_id, &refusal))+ }+ }+ }+}+struct TypeTextOutcome {detail: String,path: &'static str,@@ -633,6 +745,31 @@ fn read_axvalue(pid: i32, element_ptr_and_idx: Option<(usize, Option<usize>)>) -}}+/// Window-bound variant of [`read_axvalue`]: when no explicit element is+/// addressed and a `window_id` is known, the focused element is used ONLY when+/// its ancestry provably resolves to that exact window. A sibling window's+/// focused field must never supply before/after evidence for the requested+/// target — an unprovable focus reads as `None` (unverifiable), never as+/// sibling data. Without a window the legacy pid-global read applies.+fn read_axvalue_bound(+ pid: i32,+ element_ptr_and_idx: Option<(usize, Option<usize>)>,+ window_id: Option<u32>,+) -> Option<String> {+ if element_ptr_and_idx.is_some() {+ return read_axvalue(pid, element_ptr_and_idx);+ }+ match window_id {+ Some(wid) => unsafe {+ let el = crate::ax::exact_target::focused_element_in_window(pid, wid)?;+ let v = copy_string_attr(el, "AXValue");+ CFRelease(el as _);+ v+ },+ None => read_axvalue(pid, None),+ }+}+/// True when the addressed (or focused) AX element sits inside a web-content/// subtree — an `AXWebArea` ancestor. That covers every Chromium / WebKit //// Electron rendered surface (Chrome, Safari, Slack, VS Code, X's compose box…),@@ -644,18 +781,30 @@ fn read_axvalue(pid: i32, element_ptr_and_idx: Option<(usize, Option<usize>)>) -pub(super) fn target_in_web_area(pid: i32,element_ptr_and_idx: Option<(usize, Option<usize>)>,+ window_id: Option<u32>,) -> bool {use crate::ax::bindings::AXUIElementCopyAttributeValue;use core_foundation::base::{CFTypeRef, TCFType};use core_foundation::string::CFString;unsafe {- // Start element: the addressed one (borrowed — do NOT release) or the pid's- // focused element (owned — must release when done).+ // Start element: the addressed one (borrowed — do NOT release) or the+ // focused element (owned — must release when done). A window-addressed+ // request may only classify from the window's OWN focused element; a+ // pid-global focused element can belong to a same-process sibling and+ // sibling state must never vouch for the target. When window-bound+ // reacquisition fails, fail closed: report web content (untrusted+ // read-back) rather than trusting an unproven surface.let (start, start_owned) = match element_ptr_and_idx {Some((ptr, _)) => (ptr as AXUIElementRef, false),- None => match focused_element_of_pid(pid) {- Some(el) => (el, true),- None => return false,+ None => match window_id {+ Some(wid) => match crate::ax::exact_target::focused_element_in_window(pid, wid) {+ Some(el) => (el, true),+ None => return true,+ },+ None => match focused_element_of_pid(pid) {+ Some(el) => (el, true),+ None => return false,+ },},};let parent_attr = CFString::new("AXParent");@@ -703,6 +852,7 @@ fn cgevent_type_verified(before: Option<&str>,element_ptr_and_idx: Option<(usize, Option<usize>)>,settle_ms: u64,+ window_id: Option<u32>,) -> anyhow::Result<(bool, Option<usize>)> {// Focus the target element first so the keystrokes land in IT. Critical in// foreground mode: a freshly-fronted window's keyboard focus may be on the@@ -730,7 +880,7 @@ fn cgevent_type_verified(// expires after observable growth, surface the exact partial count.let deadline = std::time::Instant::now() + DELIVERY_DRAIN_TIMEOUT;Ok(await_typed_delivery(before, text, deadline, || {- read_axvalue(pid, element_ptr_and_idx)+ read_axvalue_bound(pid, element_ptr_and_idx, window_id)}))}@@ -781,10 +931,13 @@ fn type_text_blocking(is_terminal_target: bool,delivery_mode: super::DeliveryMode,window_id: Option<u32>,-) -> anyhow::Result<TypeTextOutcome> {+ keyboard_policy: BackgroundKeyboardPolicy,+) -> anyhow::Result<TypeTextDelivery> {// Original field value before any rung drives read-back verification only.- // An unreadable value is not evidence that the field is empty.- let before = read_axvalue(pid, element_ptr_and_idx);+ // An unreadable value is not evidence that the field is empty — and for a+ // window-addressed request it must come from the exact target window,+ // never a same-process sibling.+ let before = read_axvalue_bound(pid, element_ptr_and_idx, window_id);// --- Foreground rung: explicit agent request (skip AX/background ladder). ---if delivery_mode.is_foreground() {@@ -806,6 +959,7 @@ fn type_text_blocking(before.as_deref(),element_ptr_and_idx,foreground_settle_ms,+ window_id,)};let ((verified, delivered_chars), fronted) = match window_id {@@ -852,7 +1006,7 @@ fn type_text_blocking(// Only claim the `_fg` path when a front actually happened; when no// foregrounding occurred (no window, or SPIs unavailable) these were// background keystrokes and `path` must say so honestly.- return Ok(TypeTextOutcome {+ return Ok(TypeTextDelivery::Typed(TypeTextOutcome {detail: format!(" via foreground keystrokes ({delay_ms}ms delay)"),path: if fronted {PATH_KEY_EVENTS_FG@@ -861,11 +1015,17 @@ fn type_text_blocking(},delivered_chars,verified,- });+ }));}// --- Background rung 0: terminal emulator → CGEvent only (AX is dropped). ---if is_terminal_target {+ // A terminal insert has no semantic AX rung: when the exact-target+ // decision restricted this request to semantic-only, there is nothing+ // safe to run — refuse before posting anything.+ if let BackgroundKeyboardPolicy::SemanticOnly(refusal) = keyboard_policy {+ return Ok(TypeTextDelivery::Refused(refusal));+ }tracing::debug!("type_text: pid {pid} is a terminal emulator; skipping AX value-set, \using CGEvent key-event synthesis"@@ -877,19 +1037,29 @@ fn type_text_blocking(before.as_deref(),element_ptr_and_idx,/*settle_ms=*/ 0,+ window_id,)?;- return Ok(TypeTextOutcome {+ return Ok(TypeTextDelivery::Typed(TypeTextOutcome {detail: format!(" via CGEvent (terminal emulator, {delay_ms}ms delay)"),path: PATH_KEY_EVENTS,verified,delivered_chars,- });+ }));}// --- Background rung 1: AX SelectedText write (element or focused). ---+ // Without an explicit element, a window-addressed request may only write+ // to the focused element when it provably belongs to the exact target+ // window — a sibling window's focused field is not the requested target.let ax_target: Option<(AXUIElementRef, bool, Option<usize>)> = match element_ptr_and_idx {Some((ptr, idx)) => Some((ptr as AXUIElementRef, /*owns=*/ false, idx)),- None => unsafe { focused_element_of_pid(pid) }.map(|el| (el, /*owns=*/ true, None)),+ None => match window_id {+ Some(wid) => unsafe { crate::ax::exact_target::focused_element_in_window(pid, wid) }+ .map(|el| (el, /*owns=*/ true, None)),+ None => {+ unsafe { focused_element_of_pid(pid) }.map(|el| (el, /*owns=*/ true, None))+ }+ },};if let Some((element, owns, idx_opt)) = ax_target {let role = unsafe { copy_string_attr(element, "AXRole") }.unwrap_or_default();@@ -911,12 +1081,12 @@ fn type_text_blocking(}if ax_landed {let idx_str = idx_opt.map(|i| format!(" [{i}]")).unwrap_or_default();- return Ok(TypeTextOutcome {+ return Ok(TypeTextDelivery::Typed(TypeTextOutcome {detail: format!(" into{idx_str} {role} \"{title}\""),path: PATH_AX,verified: true,delivered_chars: Some(text.chars().count()),- });+ }));}tracing::debug!("AX write did not land for {role} \"{title}\" (err={err}); \@@ -926,6 +1096,13 @@ fn type_text_blocking(tracing::debug!("No focused element for pid {pid}; using CGEvent keystrokes");}+ // The semantic AX rung did not land and this request is restricted to it:+ // the process-scoped CGEvent rung could reach a sibling window, so return+ // the structured refusal instead of escalating.+ if let BackgroundKeyboardPolicy::SemanticOnly(refusal) = keyboard_policy {+ return Ok(TypeTextDelivery::Refused(refusal));+ }+// --- Background rung 2: CGEvent keystrokes with read-back. ---// Never clear here: a partial AX write is rare, and clearing would violate// insert-at-cursor semantics.@@ -936,13 +1113,14 @@ fn type_text_blocking(before.as_deref(),element_ptr_and_idx,/*settle_ms=*/ 0,+ window_id,)?;- Ok(TypeTextOutcome {+ Ok(TypeTextDelivery::Typed(TypeTextOutcome {detail: format!(" via CGEvent ({delay_ms}ms delay)"),path: PATH_KEY_EVENTS,verified,delivered_chars,- })+ }))}#[cfg(test)]@@ -975,6 +1153,7 @@ mod tests {/*is_terminal_target=*/ true,super::super::DeliveryMode::Background,None,+ BackgroundKeyboardPolicy::Allowed,);// We don't care whether r is Ok or Err — what matters is that// calling it with is_terminal_target=true is safe and never@@ -982,6 +1161,32 @@ mod tests {let _ = r;}+ /// A semantic-only policy must refuse the terminal short-circuit before+ /// any CGEvent is posted: terminals have no semantic AX rung, so nothing+ /// safe remains and the carried refusal comes back unchanged.+ #[test]+ fn semantic_only_policy_refuses_terminal_cgevent_rung() {+ let refusal = BackgroundRefusal {+ code: cua_driver_core::background_input::refusal_codes::SAME_PID_KEYBOARD_AMBIGUITY,+ reason: "test".into(),+ advice: None,+ };+ let r = type_text_blocking(+ -1,+ "x",+ None,+ 0,+ /*is_terminal_target=*/ true,+ super::super::DeliveryMode::Background,+ Some(7),+ BackgroundKeyboardPolicy::SemanticOnly(refusal.clone()),+ );+ match r {+ Ok(TypeTextDelivery::Refused(returned)) => assert_eq!(returned, refusal),+ other => panic!("expected a structured refusal, got {:?}", other.is_ok()),+ }+ }+#[test]fn verify_typed_unreadable_after_is_unverified() {// Catalyst: can't read AXValue back → cannot confirm → false.diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text_chars.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text_chars.rsindex 0f390c9878..342fe0ab84 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text_chars.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text_chars.rs@@ -94,25 +94,55 @@ impl Tool for TypeTextCharsTool {};let type_chars_only = args.bool_or("type_chars_only", false);+ // Retain the addressed element (if any) so a concurrent+ // get_window_state can't free it during the gate/focus below+ // (use-after-free → daemon crash). Guard lives past the focus call.+ let element_guard = if let (Some(idx), Some(wid)) = (element_index, window_id) {+ match self.state.element_cache.get_element_retained(pid, wid, idx) {+ Some(guard) => Some(guard),+ None => {+ return ToolResult::error(format!(+ "Element index {idx} not found. Call get_window_state first."+ ));+ }+ }+ } else {+ None+ };+ let element_ptr: Option<usize> = element_guard.as_ref().map(|g| g.as_ptr());++ // ── Exact-target background gate (macOS background input v1) ──+ // This tool is always background CGEvent keystrokes — process-scoped+ // transport with no semantic AX rung. A window-addressed request must+ // prove exact delivery before any event is posted; there is no+ // semantic fallback here, so a refusal is final.+ let _mutation_lease = if let Some(wid) = window_id {+ match super::gate_background_window_action(+ pid,+ wid,+ element_ptr,+ cua_driver_core::background_input::BackgroundAction::InsertText,+ )+ .await+ {+ Ok(lease) => Some(lease),+ Err(refusal_result) => return refusal_result,+ }+ } else {+ None+ };+// Pre-focus element if requested.if !type_chars_only {- if let (Some(idx), Some(wid)) = (element_index, window_id) {- // Retain so a concurrent get_window_state can't free the element- // during the focus call (use-after-free → daemon crash). The- // guard outlives the awaited spawn_blocking below.- if let Some(element_guard) =- self.state.element_cache.get_element_retained(pid, wid, idx)- {- let element_ptr = element_guard.as_ptr();- let _ = tokio::task::spawn_blocking(move || {- crate::input::ax_actions::focus_element(element_ptr)- })- .await;- drop(element_guard);- tokio::time::sleep(std::time::Duration::from_millis(50)).await;- }+ if let Some(element_ptr) = element_ptr {+ let _ = tokio::task::spawn_blocking(move || {+ crate::input::ax_actions::focus_element(element_ptr)+ })+ .await;+ tokio::time::sleep(std::time::Duration::from_millis(50)).await;}}+ drop(element_guard);let text_len = text.chars().count();let result = tokio::task::spawn_blocking(move || {