Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): background keyboard into editor surfaces and multi-window apps (CUA-1767) (#5306)

Francesco Bonacci9867c0ae9340
diff --git a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
index 888a8c6be2..bb3f39dea1 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/MACOS.md
@@ -618,7 +618,7 @@ starting point for new browser workflows.
| `Screen Recording permission not granted` | TCC not granted for capture | Screenshots and pixel actions are unavailable. If the task is AX-completable, use `get_window_state({include_screenshot:false})` and `element_token` actions; otherwise stop and ask the user to run `cua-driver permissions grant` |
| `AX action failed: ... returned -25200` (or -25205, -25206) | The app may have acted anyway: some AppKit controls return these after a press they performed | Re-read with `get_window_state` before retrying; a second press on a checkbox toggles it back. A radio button or checkbox whose AXValue settles on the pressed state is reported as performed, naming the error |
| `set_value refused (file_name_needs_rename)` | A file's name in Finder's Get Info window (AXIdentifier `Name`): an AXValue write changes only the field | `set_value` the full new name on the file's name cell in a Finder window that lists it (renamed through Finder in the background and read back from disk) |
-| `Background input refused (no_focused_text_input)` | Untargeted background `type_text` while the window's focused element is not a text input; keys would drive the view (Finder type-to-select) | pass the field's `element_token`, or `x,y` to pixel-focus it first; to rename a Finder file, `set_value` its name cell |
+| `Background input refused (no_focused_text_input)` | Untargeted background `type_text` while the window's focused element is not a text input or an editing surface (spreadsheet grid, document canvas: `AXLayoutArea`, `AXScrollArea`, `AXTable`, `AXCell`, `AXGroup`; not in Finder/Dock); keys would drive the view (Finder type-to-select) | pass the field's `element_token`, or `x,y` to pixel-focus it first; to rename a Finder file, `set_value` its name cell |
## Example end-to-end task (macOS)
@@ -637,3 +637,7 @@ starting point for new browser workflows.
If the user instead asks to navigate _within_ an already-open Finder
window, use the menu-bar flow from "Navigating native menu bars"
above (`invoke_menu` with a path such as `["Go", "Downloads"]`).
+
+### LibreOffice, Numbers and other editors in the background
+
+With `window_id`, `type_text`, `press_key` and `hotkey` post keys to that exact window: the driver makes it the app's key window through a focus record (no raise, no activation, real pointer untouched), posts, and hands key back. A second window of the same app (Find & Replace, another document) is no longer `same_pid_keyboard_ambiguity`. A command key whose menu route is refused (item disabled while the app is in the background) falls through to key events in that window. Recipe: Name Box navigation (`type_text` `A1\n`), type with `\n` and `\t`, then read the cell back.
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/background_input.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/background_input.rs
index cfd35d2647..7f390feb75 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/background_input.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/background_input.rs
@@ -90,13 +90,23 @@ pub enum BackgroundAction {
InsertText,
/// Process-scoped key/hotkey without an action-specific verifier.
GenericKey,
+ /// Process-scoped keys whose transport first makes the exact target window
+ /// the app's key window (a window focus record: no activation, no raise,
+ /// the user's front window keeps key status) and hands key back after. A
+ /// second same-pid window is then not a competing destination, because
+ /// the keys are routed to the requested window explicitly. Callers use
+ /// this only where that transport is available; otherwise they ask for
+ /// `InsertText` / `GenericKey` and keep the fail-closed ambiguity guard.
+ ExactWindowKeys,
}
impl BackgroundAction {
fn is_pid_keyboard(self) -> bool {
matches!(
self,
- BackgroundAction::InsertText | BackgroundAction::GenericKey
+ BackgroundAction::InsertText
+ | BackgroundAction::GenericKey
+ | BackgroundAction::ExactWindowKeys
)
}
}
@@ -271,7 +281,9 @@ pub fn decide_background_input(
},
}
}
- BackgroundAction::InsertText | BackgroundAction::GenericKey => {
+ BackgroundAction::InsertText
+ | BackgroundAction::GenericKey
+ | BackgroundAction::ExactWindowKeys => {
if visibility_disproven {
return refuse(
refusal_codes::MINIMIZED_OR_HIDDEN,
@@ -286,7 +298,9 @@ pub fn decide_background_input(
);
}
debug_assert!(action.is_pid_keyboard());
- if facts.competing_keyboard_destinations > 0 {
+ if facts.competing_keyboard_destinations > 0
+ && action != BackgroundAction::ExactWindowKeys
+ {
return refuse(
refusal_codes::SAME_PID_KEYBOARD_AMBIGUITY,
format!(
@@ -548,6 +562,43 @@ mod tests {
}
}
+ /// CUA-1767: a second window of the same process (LibreOffice's Find &
+ /// Replace beside the sheet) is not an ambiguity when the transport makes
+ /// the exact window key first. Every other guard still applies.
+ #[test]
+ fn exact_window_keys_ignore_sibling_windows_but_keep_other_guards() {
+ let facts = BackgroundTargetFacts {
+ competing_keyboard_destinations: 2,
+ ..matched_facts()
+ };
+ assert!(
+ decide_background_input(TARGET, &facts, BackgroundAction::ExactWindowKeys).is_execute()
+ );
+ for broken in [
+ BackgroundTargetFacts {
+ target_minimized: Some(true),
+ ..facts.clone()
+ },
+ BackgroundTargetFacts {
+ app_hidden: None,
+ ..facts.clone()
+ },
+ BackgroundTargetFacts {
+ ax_window_present: false,
+ ..facts.clone()
+ },
+ BackgroundTargetFacts {
+ element: ElementAncestry::OutsideTargetWindow,
+ ..facts.clone()
+ },
+ ] {
+ assert!(
+ !decide_background_input(TARGET, &broken, BackgroundAction::ExactWindowKeys)
+ .is_execute()
+ );
+ }
+ }
+
/// The singleton rule: with no competing keyboard destination, exact
/// background text/keys are permitted and verification is target-bound.
#[test]
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/skylight.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/skylight.rs
index 9da32128df..e9e28d4dac 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/input/skylight.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/skylight.rs
@@ -53,6 +53,141 @@ type CopyManagedDisplayForWindowFn = unsafe extern "C" fn(u32, u32) -> *mut c_vo
/// `uint64_t SLSManagedDisplayGetCurrentSpace(uint32_t cid, CFStringRef display)`
type ManagedDisplayGetCurrentSpaceFn = unsafe extern "C" fn(u32, *const c_void) -> u64;
+/// How the exact window became the app's key window.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum KeyPrime {
+ AlreadyKey,
+ FocusRecord,
+ AxFocusedWindow,
+}
+
+/// Make `target` the app's key window and PROVE it before any key is posted.
+///
+/// `focused` re-reads the app's `AXFocusedWindow`; `focus_record` sends the
+/// SkyLight focus record; `ax_focus` writes `AXFocusedWindow` on the app. Each
+/// step is followed by a re-read; keys may only be posted when the re-read
+/// names `target`. Returns `Err` otherwise: refusing is safe, a key that lands
+/// in a sibling window (a Find & Replace field) is not.
+pub fn prime_exact_window_key(
+ target: u32,
+ mut focused: impl FnMut() -> Option<u32>,
+ mut focus_record: impl FnMut() -> bool,
+ mut ax_focus: impl FnMut() -> bool,
+) -> Result<KeyPrime, String> {
+ if focused() == Some(target) {
+ return Ok(KeyPrime::AlreadyKey);
+ }
+ if focus_record() && focused() == Some(target) {
+ return Ok(KeyPrime::FocusRecord);
+ }
+ if ax_focus() && focused() == Some(target) {
+ return Ok(KeyPrime::AxFocusedWindow);
+ }
+ Err(format!(
+ "window {target} could not be confirmed as the app's key window (the app's focused \
+ window is {:?}), so no keys were sent: they could have gone to another window of \
+ the same app. Close the other window, or use delivery_mode:\"foreground\"",
+ focused()
+ ))
+}
+
+/// Holds an exact window as the app's key window; hands key back on drop.
+pub struct WindowKeyScope {
+ pid: pid_t,
+ previous: Option<u32>,
+ primed: KeyPrime,
+}
+
+impl Drop for WindowKeyScope {
+ fn drop(&mut self) {
+ if self.primed == KeyPrime::AlreadyKey {
+ return;
+ }
+ if let Some(previous) = self.previous {
+ match self.primed {
+ KeyPrime::FocusRecord => {
+ activate_without_raise_with(self.pid, previous, false);
+ }
+ _ => {
+ set_app_focused_window(self.pid, previous);
+ }
+ }
+ }
+ }
+}
+
+fn set_app_focused_window(pid: pid_t, wid: u32) -> bool {
+ use crate::ax::bindings::{
+ ax_get_window_id, copy_ax_windows, AXUIElementCreateApplication,
+ AXUIElementSetAttributeValue,
+ };
+ use core_foundation::{
+ base::{CFRelease, TCFType},
+ string::CFString,
+ };
+ unsafe {
+ let app = AXUIElementCreateApplication(pid);
+ if app.is_null() {
+ return false;
+ }
+ let attr = CFString::new("AXFocusedWindow");
+ let mut ok = false;
+ for window in copy_ax_windows(app) {
+ if !ok && ax_get_window_id(window) == Some(wid) {
+ ok = AXUIElementSetAttributeValue(
+ app,
+ attr.as_concrete_TypeRef(),
+ window as core_foundation::base::CFTypeRef,
+ ) == 0;
+ }
+ CFRelease(window as _);
+ }
+ CFRelease(app as _);
+ ok
+ }
+}
+
+/// Make `target_wid` the app's key window (no activation, no raise, no
+/// pointer), confirmed by re-reading the app's focused window, and return a
+/// guard that hands key back. `Err` when it cannot be confirmed.
+pub fn enter_exact_window_key(pid: pid_t, target_wid: u32) -> Result<WindowKeyScope, String> {
+ let previous = crate::ax::bindings::focused_window_id_of_pid(pid);
+ // An app with no focused window has never been active: it has no key
+ // window to hand over, and keys posted to it are dropped silently while
+ // the call still reports success (LibreOffice, CUA-1767). Refuse.
+ if previous.is_none() {
+ return Err(format!(
+ "pid {pid} has no focused window (it has not been active yet), so window \
+ {target_wid} cannot be confirmed as its key window and no keys were sent. \
+ Activate the app once (bring_to_front) or use delivery_mode:\"foreground\""
+ ));
+ }
+ let settle = || std::thread::sleep(std::time::Duration::from_millis(120));
+ let primed = prime_exact_window_key(
+ target_wid,
+ || crate::ax::bindings::focused_window_id_of_pid(pid),
+ || {
+ let ok = activate_without_raise_with(pid, target_wid, false);
+ if ok {
+ settle();
+ }
+ ok
+ },
+ || {
+ let ok = set_app_focused_window(pid, target_wid);
+ if ok {
+ settle();
+ }
+ ok
+ },
+ )?;
+ Ok(WindowKeyScope {
+ pid,
+ previous,
+ primed,
+ })
+}
+
// ── NSMenu shortcut activation SPIs ──────────────────────────────────────────
/// `OSStatus SLPSSetFrontProcessWithOptions(const void *psn, uint32_t windowID, uint32_t options)`
@@ -1359,3 +1494,55 @@ mod tests {
));
}
}
+
+#[cfg(test)]
+mod key_prime_tests {
+ use super::{prime_exact_window_key, KeyPrime};
+ use std::cell::Cell;
+
+ #[test]
+ fn already_key_sends_nothing() {
+ let r = prime_exact_window_key(7, || Some(7), || panic!("record"), || panic!("ax"));
+ assert_eq!(r, Ok(KeyPrime::AlreadyKey));
+ }
+
+ #[test]
+ fn focus_record_is_accepted_only_after_the_reread_names_the_target() {
+ let focus = Cell::new(Some(9));
+ let r = prime_exact_window_key(
+ 7,
+ || focus.get(),
+ || {
+ focus.set(Some(7));
+ true
+ },
+ || panic!("ax"),
+ );
+ assert_eq!(r, Ok(KeyPrime::FocusRecord));
+ }
+
+ #[test]
+ fn ignored_focus_record_falls_back_to_ax_then_confirms() {
+ let focus = Cell::new(Some(9));
+ let r = prime_exact_window_key(
+ 7,
+ || focus.get(),
+ || true, // record posted but the app ignored it
+ || {
+ focus.set(Some(7));
+ true
+ },
+ );
+ assert_eq!(r, Ok(KeyPrime::AxFocusedWindow));
+ }
+
+ /// The Find & Replace leak: the dialog keeps key, nothing redirects. Must
+ /// refuse, never post.
+ #[test]
+ fn unconfirmed_key_window_refuses_instead_of_leaking() {
+ for focused in [Some(9), None] {
+ let r = prime_exact_window_key(7, || focused, || true, || true);
+ assert!(r.unwrap_err().contains("no keys were sent"));
+ }
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs
index d197f10666..4cbfe75468 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs
@@ -296,6 +296,7 @@ impl Tool for HotkeyTool {
// key equivalent in a native app that is not in front, run that menu
// item in the background instead (verified for Save), or refuse when
// macOS keeps it disabled. No key events are sent on this route.
+ let mut key_scope: Option<crate::input::skylight::WindowKeyScope> = None;
if !fg && px.is_none() {
if let (Some(wid), Some(combo)) = (
window_id,
@@ -315,6 +316,13 @@ impl Tool for HotkeyTool {
.ok()
.flatten();
if let Some(item) = menu_item {
+ // The menu command acts on the app's key window. Make the
+ // addressed window key first, confirmed, or refuse: never
+ // act on another window (CUA-1767).
+ match super::acquire_key_scope(pid, Some(wid), false).await {
+ Ok(scope) => key_scope = scope,
+ Err(refusal) => return refusal,
+ }
if let Some(ptr) = element_ptr {
// The menu command acts on the focused element.
let focused =
@@ -327,13 +335,21 @@ impl Tool for HotkeyTool {
);
}
}
- return super::invoke_menu::run_in_background(
+ let menu_result = super::invoke_menu::run_in_background(
pid,
wid,
item.path,
- super::invoke_menu::MenuRequest::KeyEquivalent(key_display),
+ super::invoke_menu::MenuRequest::KeyEquivalent(key_display.clone()),
)
.await;
+ // The menu route is only a better route when it works. A
+ // refusal (an item macOS keeps disabled while the app is in
+ // the background, as LibreOffice's Format > Bold does) must
+ // not end the call: the caller named a window, so the combo
+ // goes to that window as key events below (CUA-1767).
+ if menu_result.is_error != Some(true) {
+ return menu_result;
+ }
}
}
}
@@ -380,7 +396,9 @@ impl Tool for HotkeyTool {
pid,
wid,
element_ptr,
- cua_driver_core::background_input::BackgroundAction::GenericKey,
+ super::window_keyboard_action(
+ cua_driver_core::background_input::BackgroundAction::GenericKey,
+ ),
)
.await
{
@@ -394,6 +412,14 @@ impl Tool for HotkeyTool {
None
};
+ // Key post: the addressed window must be the confirmed key window.
+ if key_scope.is_none() {
+ match super::acquire_key_scope(pid, window_id, fg).await {
+ Ok(scope) => key_scope = scope,
+ Err(refusal) => return refusal,
+ }
+ }
+
// Web-content AX nodes can acknowledge AXFocused without moving the
// renderer's real first responder. That makes a direct focus write an
// unsafe oracle for Chromium/WebKit hotkeys: the following PID/HID
@@ -472,6 +498,8 @@ impl Tool for HotkeyTool {
}
};
+ let _hold_key_scope = &key_scope;
+
// A focus click already moved the cursor. Otherwise place a named
// session's cursor on the element, its remembered position, or the
// window centre, so a keyboard-first session stays visible.
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs
index 1d14c4e638..8fb4c66b76 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs
@@ -389,6 +389,69 @@ async fn decide_background_window_action(
}
}
+/// The keyboard action class for a window-addressed background key or text
+/// request. When the focus-record transport exists the exact window is made
+/// the app's key window for the duration of the post (see
+/// [`post_keys_to_window`]), so a second same-pid window is not ambiguous.
+/// Without it the caller's fail-closed `generic` class applies.
+pub(crate) fn window_keyboard_action(
+ generic: cua_driver_core::background_input::BackgroundAction,
+) -> cua_driver_core::background_input::BackgroundAction {
+ if crate::input::skylight::is_focus_without_raise_available() {
+ cua_driver_core::background_input::BackgroundAction::ExactWindowKeys
+ } else {
+ generic
+ }
+}
+
+/// Make `window_id` the app's key window, confirmed, for pid-routed keys.
+/// `Ok(None)` when no window is addressed or the SPIs are missing (keys go
+/// where they always did). `Err` carries the refusal text: no key may be
+/// posted, it could reach a sibling window.
+pub(crate) fn key_scope(
+ pid: i32,
+ window_id: Option<u32>,
+) -> Result<Option<crate::input::skylight::WindowKeyScope>, String> {
+ match window_id {
+ Some(wid) if crate::input::skylight::is_focus_without_raise_available() => {
+ crate::input::skylight::enter_exact_window_key(pid as libc::pid_t, wid).map(Some)
+ }
+ _ => Ok(None),
+ }
+}
+
+/// [`key_scope`] for an async tool: runs the blocking prime off the runtime and
+/// turns an unconfirmed key window into the structured `window_not_key`
+/// refusal. Foreground requests and pid-only calls hold no scope.
+pub(crate) async fn acquire_key_scope(
+ pid: i32,
+ window_id: Option<u32>,
+ foreground: bool,
+) -> Result<Option<crate::input::skylight::WindowKeyScope>, cua_driver_core::protocol::ToolResult> {
+ let Some(wid) = window_id.filter(|_| !foreground) else {
+ return Ok(None);
+ };
+ match tokio::task::spawn_blocking(move || key_scope(pid, Some(wid))).await {
+ Ok(Ok(scope)) => Ok(scope),
+ Ok(Err(reason)) => Err(background_refusal_result(
+ pid,
+ wid,
+ &cua_driver_core::background_input::BackgroundRefusal {
+ code: WINDOW_NOT_KEY,
+ reason,
+ advice: Some("foreground"),
+ },
+ )),
+ Err(error) => Err(cua_driver_core::protocol::ToolResult::error(format!(
+ "Could not prepare the key window for pid {pid}: {error}"
+ ))),
+ }
+}
+
+/// Refusal code when the addressed window cannot be confirmed as the app's
+/// key window: no key is sent, none could be proven to reach it.
+pub(crate) const WINDOW_NOT_KEY: &str = "window_not_key";
+
/// Acquire the per-PID mutation coordinator, gather fresh exact-target facts,
/// and ask the pure core for one background decision. `element_ptr` must stay
/// retained by the caller until the returned lease is dropped.
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs
index 9ea393dc2f..1069b452f9 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs
@@ -315,7 +315,9 @@ impl Tool for PressKeyTool {
pid,
wid,
pre_focus_ptr,
- cua_driver_core::background_input::BackgroundAction::GenericKey,
+ super::window_keyboard_action(
+ cua_driver_core::background_input::BackgroundAction::GenericKey,
+ ),
)
.await
{
@@ -329,6 +331,13 @@ impl Tool for PressKeyTool {
None
};
+ // The addressed window must be the app's confirmed key window before
+ // any key (or focus click) goes out; otherwise refuse (window_not_key).
+ let _key_scope = match super::acquire_key_scope(pid, window_id, fg).await {
+ Ok(scope) => scope,
+ Err(refusal) => return refusal,
+ };
+
// px form: pixel-click to focus, then the key goes to the focused element.
// Reuses click's translation + delivery_mode; after it, deliver via the
// plain background path (the focus-click already handled fronting if fg).
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs
index 0c5d3c8e20..f8fe1500e5 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs
@@ -378,6 +378,7 @@ impl Tool for TypeTextTool {
window_id,
keyboard_policy: blocking_policy,
explicit_target: element_ptr.is_some() || used_pixel_focus,
+ key_window_confirmed: false,
},
)
})
@@ -859,7 +860,10 @@ async fn background_keyboard_policy(
}
};
let target = ExactWindowTarget { pid, window_id };
- match decide_background_input(target, &facts, BackgroundAction::InsertText) {
+ // The exact window is made key for the key post (see
+ // `super::post_keys_to_window`), so a sibling window is not ambiguous.
+ let action = super::window_keyboard_action(BackgroundAction::InsertText);
+ match decide_background_input(target, &facts, action) {
BackgroundInputDecision::Execute { .. } => Ok((lease, BackgroundKeyboardPolicy::Allowed)),
BackgroundInputDecision::Refuse(refusal) => {
let semantic_available = element_ptr.is_some()
@@ -1296,6 +1300,10 @@ struct KeyboardRoute {
/// The caller chose where the text goes: an `element_token`, or `x,y`
/// that pixel-focused a field first.
explicit_target: bool,
+ /// The addressed window was confirmed as the app's key window before this
+ /// call. An app that is not active reports no focused element even then
+ /// (LibreOffice), so a missing focus is not a reason to refuse.
+ key_window_confirmed: bool,
}
/// Best-effort-background ladder for `type_text`.
@@ -1315,12 +1323,55 @@ fn type_text_blocking(
delay_ms: u64,
is_terminal_target: bool,
route: KeyboardRoute,
+) -> anyhow::Result<TypeTextDelivery> {
+ // A window-addressed background insert first makes that window the app's
+ // key window, proven by re-reading the app's focused window, BEFORE the
+ // focused element is read or any key is posted. Unconfirmed: refuse
+ // (CUA-1767: with Find & Replace open the keys went into its Find field).
+ let _scope = if !route.delivery_mode.is_foreground()
+ && matches!(route.keyboard_policy, BackgroundKeyboardPolicy::Allowed)
+ {
+ match super::key_scope(pid, route.window_id) {
+ Ok(scope) => scope,
+ Err(reason) => {
+ return Ok(TypeTextDelivery::Refused(BackgroundRefusal {
+ code: super::WINDOW_NOT_KEY,
+ reason,
+ advice: Some("foreground"),
+ }))
+ }
+ }
+ } else {
+ None
+ };
+ let route = KeyboardRoute {
+ key_window_confirmed: _scope.is_some(),
+ ..route
+ };
+ type_text_in_key_window(
+ pid,
+ text,
+ element_ptr_and_idx,
+ delay_ms,
+ is_terminal_target,
+ route,
+ )
+}
+
+fn type_text_in_key_window(
+ pid: i32,
+ text: &str,
+ element_ptr_and_idx: Option<(usize, Option<usize>)>,
+ delay_ms: u64,
+ is_terminal_target: bool,
+ route: KeyboardRoute,
) -> anyhow::Result<TypeTextDelivery> {
let KeyboardRoute {
delivery_mode,
window_id,
keyboard_policy,
explicit_target,
+ key_window_confirmed,
} = route;
// Original field value before any rung drives read-back verification only.
// An unreadable value is not evidence that the field is empty — and for a
@@ -1543,10 +1594,16 @@ fn type_text_blocking(
// selection to another file, and the next Return renamed the wrong
// item). Refuse before anything is sent.
if !explicit_target && window_id.is_some() {
- let accepts_text = ax_target.is_some_and(|(element, _, _)| unsafe {
- focused_accepts_text(element)
- || target_in_web_area(pid, Some((element as usize, None)), window_id)
- });
+ let accepts_text = match ax_target {
+ Some((element, _, _)) => unsafe {
+ focused_accepts_text(element)
+ || focused_is_editor_surface(pid, element)
+ || target_in_web_area(pid, Some((element as usize, None)), window_id)
+ },
+ // No focused element, but the window is the confirmed key window:
+ // post the keys to it, as press_key does.
+ None => unfocused_window_accepts_keys(key_window_confirmed, &bundle_id_of(pid)),
+ };
if !accepts_text {
if let Some((element, true, _)) = ax_target {
unsafe { CFRelease(element as _) };
@@ -1686,6 +1743,52 @@ fn is_text_input_role(role: &str) -> bool {
)
}
+/// Roles of a focused element that is an editing surface without being a text
+/// field: a spreadsheet grid or cell, a document canvas, the scroll view that
+/// hosts one (LibreOffice Calc and Writer, Numbers, Pages). Keys typed into
+/// them edit the document, so a pid-routed key post is the right route.
+/// Lists and outlines stay out: there typed keys move the selection.
+fn is_editor_surface_role(role: &str) -> bool {
+ matches!(
+ role,
+ "AXLayoutArea" | "AXLayoutItem" | "AXScrollArea" | "AXTable" | "AXCell" | "AXGroup"
+ )
+}
+
+/// Apps whose focused views of those roles are file or item browsers, where
+/// typed keys drive type-to-select rather than edit.
+/// A window with no focused element takes keys only when it is the confirmed
+/// key window and the app is not an item browser.
+fn unfocused_window_accepts_keys(key_window_confirmed: bool, bundle_id: &str) -> bool {
+ key_window_confirmed && !is_item_browser_app(bundle_id)
+}
+
+fn bundle_id_of(pid: i32) -> String {
+ crate::apps::bundle_id_for_pid(pid).unwrap_or_default()
+}
+
+fn is_item_browser_app(bundle_id: &str) -> bool {
+ matches!(bundle_id, "com.apple.finder" | "com.apple.dock")
+}
+
+fn editor_surface_accepts_keys(role: &str, bundle_id: &str) -> bool {
+ is_editor_surface_role(role) && !is_item_browser_app(bundle_id)
+}
+
+/// Whether the focused element is an editing surface that takes typed keys
+/// even though it is not a text field (CUA-1767).
+///
+/// # Safety
+/// `element` must be a live `AXUIElementRef`.
+unsafe fn focused_is_editor_surface(pid: i32, element: AXUIElementRef) -> bool {
+ let role = copy_string_attr(element, "AXRole").unwrap_or_default();
+ if !is_editor_surface_role(&role) {
+ return false;
+ }
+ let bundle_id = crate::apps::bundle_id_for_pid(pid).unwrap_or_default();
+ editor_surface_accepts_keys(&role, &bundle_id)
+}
+
const NO_FOCUSED_TEXT_INPUT: &str = "no_focused_text_input";
fn no_focused_text_input_refusal(
@@ -1710,6 +1813,46 @@ fn no_focused_text_input_refusal(
#[cfg(test)]
mod tests {
+ #[test]
+ fn unfocused_window_needs_a_confirmed_key_window() {
+ assert!(super::unfocused_window_accepts_keys(
+ true,
+ "org.libreoffice.script"
+ ));
+ assert!(!super::unfocused_window_accepts_keys(
+ false,
+ "org.libreoffice.script"
+ ));
+ assert!(!super::unfocused_window_accepts_keys(
+ true,
+ "com.apple.finder"
+ ));
+ }
+
+ #[test]
+ fn editor_surfaces_take_keys_but_item_browsers_do_not() {
+ for role in ["AXLayoutArea", "AXScrollArea", "AXTable", "AXCell"] {
+ assert!(super::editor_surface_accepts_keys(
+ role,
+ "org.libreoffice.script"
+ ));
+ assert!(super::editor_surface_accepts_keys(
+ role,
+ "com.apple.iWork.Numbers"
+ ));
+ assert!(!super::editor_surface_accepts_keys(
+ role,
+ "com.apple.finder"
+ ));
+ }
+ for role in ["AXOutline", "AXList", "AXRow", "AXButton", "AXMenuItem", ""] {
+ assert!(!super::editor_surface_accepts_keys(
+ role,
+ "org.libreoffice.script"
+ ));
+ }
+ }
+
#[test]
fn chromium_native_field_refuses_only_background_submits() {
let refusal = super::chromium_native_field_submit_refusal(true, "file:///tmp/report.log\n")
@@ -1768,6 +1911,7 @@ mod tests {
window_id: Some(7),
keyboard_policy: BackgroundKeyboardPolicy::SemanticOnly(refusal.clone()),
explicit_target: false,
+ key_window_confirmed: false,
},
);
match r {
@@ -1780,7 +1924,7 @@ mod tests {
/// window is refused before any key event (Finder type-to-select).
#[test]
fn untargeted_text_without_a_focused_text_input_is_refused() {
- let result = type_text_blocking(
+ let result = type_text_in_key_window(
-1,
"renamed.txt",
None,
@@ -1791,6 +1935,7 @@ mod tests {
window_id: Some(7),
keyboard_policy: BackgroundKeyboardPolicy::Allowed,
explicit_target: false,
+ key_window_confirmed: false,
},
)
.expect("the refusal is a delivery outcome");
@@ -1834,6 +1979,7 @@ mod tests {
window_id: None,
keyboard_policy: BackgroundKeyboardPolicy::Allowed,
explicit_target: false,
+ key_window_confirmed: false,
},
)
.expect("preflight refusal must not attempt the invalid pid");
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text_chars.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text_chars.rs
index 712a9ebfac..0a3bee810e 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text_chars.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text_chars.rs
@@ -91,7 +91,9 @@ impl Tool for TypeTextCharsTool {
pid,
wid,
element_ptr,
- cua_driver_core::background_input::BackgroundAction::InsertText,
+ super::window_keyboard_action(
+ cua_driver_core::background_input::BackgroundAction::InsertText,
+ ),
)
.await
{
@@ -102,6 +104,11 @@ impl Tool for TypeTextCharsTool {
None
};
+ let _key_scope = match super::acquire_key_scope(pid, window_id, false).await {
+ Ok(scope) => scope,
+ Err(refusal) => return refusal,
+ };
+
// Pre-focus element if requested.
if !type_chars_only {
if let Some(guard) = element_guard.as_ref().cloned() {