Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

test(cua-driver): harden background E2E oracles (#2193)

Francesco Bonaccie2596a4e554b
diff --git a/libs/cua-driver/docs/e2e-ci-reporting.md b/libs/cua-driver/docs/e2e-ci-reporting.md
index dc395962db..f84d12c9fa 100644
--- a/libs/cua-driver/docs/e2e-ci-reporting.md
+++ b/libs/cua-driver/docs/e2e-ci-reporting.md
@@ -67,8 +67,12 @@ artifacts/cua-driver/<platform>/
`cases.jsonl` is the executed catalog. `results.jsonl` contains one result for
every declared cell. In canonical mode, the reporter rejects duplicates,
-missing results, undeclared results, contradictory statuses, missing required
-videos, and missing or invalid expected turn evidence. An `evidence.json`
+missing results, undeclared results, contradictory statuses, forbidden skips,
+an under-sized unfiltered shared catalog, missing required videos, and missing
+or invalid expected turn evidence. Canonical runners set
+`CUA_E2E_FORBID_SKIPS=1` and `CUA_E2E_EXPECTED_MIN_CELLS` (80 on
+Windows/Linux, 120 on macOS). Explicit diagnostic filters omit only the
+minimum-count policy; a filter matching zero cells still fails in Rust. An `evidence.json`
classification normally makes capture failure auditable without making it
pass. The one narrow exception is a successful Windows `bring_to_front`
restoration:
diff --git a/libs/cua-driver/docs/test-harnesses-guide.md b/libs/cua-driver/docs/test-harnesses-guide.md
index 32f64e7d39..c75a8b742b 100644
--- a/libs/cua-driver/docs/test-harnesses-guide.md
+++ b/libs/cua-driver/docs/test-harnesses-guide.md
@@ -212,6 +212,10 @@ target activation. A portal/libei grant persists until the user revokes it, so
subsequent driver processes do not reopen the consent dialog.
KDE requires a future target-addressable KWin adapter; portal availability by
itself is not evidence that input can be sent safely to a named window.
+Standard Wayland does not expose the physical pointer position, so canonical
+Wayland rows do not claim the real-cursor preservation oracle. Focus, full
+occlusion, sentinel input isolation, liveness, and fixture-state oracles remain
+mandatory.
## AX, PX, and Delivery
@@ -252,8 +256,10 @@ question for any action, harness, or catalog area:
`cua-driver-testkit::DesktopObserver` owns the shared interface. Native Windows,
macOS, and Linux backends snapshot foreground-window, target z-order, cursor,
-and leaked-input state before and after an action. Background rows opt into the
-observer directly; there is no special guard suite.
+and focus state before and after an action. A separate full-desktop Electron
+sentinel journals keyboard, pointer, wheel, visibility, focus, and heartbeat
+events while it fully covers the target. Background rows opt into both pieces
+of instrumentation directly; there is no special guard suite.
| Invariant or scenario | What it checks |
| --- | --- |
@@ -264,6 +270,18 @@ observer directly; there is no special guard suite.
| Background screenshot | Reading the target does not change focus or z-order |
| Agent cursor visibility | The cursor appears in the captured pixels when enabled and moved |
+The sentinel contract fails closed when the target is only partly covered or
+the heartbeat stops. Before any behavioral cells run, the strict environment
+preflight deliberately sends input to the sentinel and deliberately raises the
+background target. The lane proceeds only if the leaked input and transient
+focus loss are observed, the sentinel is restored, and it once again fully
+occludes the target. Windows, macOS, and X11 require the sentinel's live focus
+journal to report the loss. Wayland uses the compositor-backed native focus
+observer because Electron/Ozone does not reliably emit a DOM `blur` event for
+an external surface focus transition. The sentinel heartbeat and leaked-input
+journal remain mandatory on Wayland. This positive control prevents a broken
+guard from making every background row look green.
+
A focus assertion can prove "no focus steal" while failing to prove that a
click changed the target application state. An action row must therefore check
both the target's external state and, when background delivery is under test,
@@ -325,7 +343,13 @@ need normal test output and logs; they do not need desktop video.
established. The sentinel remains a separate test fixture and is reasserted
after console cleanup.
- Strict lane preflights fail on missing fixtures, desktop access, permissions,
- accessibility, capture, or recording support instead of silently skipping.
+ accessibility, capture, recording support, or ineffective background guards
+ instead of silently skipping.
+- Canonical runners set `CUA_E2E_FORBID_SKIPS=1`. Unfiltered shared runs also
+ set `CUA_E2E_EXPECTED_MIN_CELLS` to 80 on Windows/Linux and 120 on macOS, so
+ a filtered, shortened, or accidentally emptied catalog cannot report green.
+ Explicit diagnostic cell or harness filters disable only the minimum-count
+ check; matching no cells still fails inside the Rust matrix.
- GitHub summaries link every evidence-bearing row to its lane archive and
display the exact recording path. The trajectory path remains in the typed
evidence and archive.
diff --git a/libs/cua-driver/docs/test-matrix.md b/libs/cua-driver/docs/test-matrix.md
index efcbdb3016..b380d5c0d0 100644
--- a/libs/cua-driver/docs/test-matrix.md
+++ b/libs/cua-driver/docs/test-matrix.md
@@ -74,22 +74,28 @@ The shared harness exposes deterministic external markers for these actions:
| Action family | Actions | Addressing | Delivery |
| --- | --- | --- | --- |
| Pointer | Left click, right click, double click | AX and PX | Background and foreground |
-| Keyboard and text | Type text, Return, hotkey | AX and PX where supported | Background and foreground |
+| Keyboard and text | Type text, Return, hotkey, type then Return | AX and PX where supported | Background and foreground |
| Scroll | Scroll | AX and PX where supported | Background and foreground |
| Child windows | Open child window | AX and PX | Background and foreground |
| Drag | Drag source to drop target | PX | Background and foreground |
| State controls | Checkbox, radio, combo, slider | AX or PX by control | Mode declared per action |
| Editor | Type, save, saved-state readback | AX and PX where supported | Mode declared per action |
-The Rust shared catalog declares 36 evidence-bearing cells per harness
-application. Windows and Linux run 72 shared cells across Electron and Tauri;
-macOS also runs the native WKWebView host for 108 shared cells. Each host covers
+The Rust shared catalog declares 40 evidence-bearing cells per harness
+application. Windows and Linux run 80 shared cells across Electron and Tauri;
+macOS also runs the native WKWebView host for 120 shared cells. Each host covers
the full AX/PX and foreground/background cross-product for click, text,
-keyboard, scroll, and child-window actions, plus both delivery modes for PX
-drag and AX editor-save. A
+keyboard, sequential type-then-Return, scroll, and child-window actions, plus
+both delivery modes for PX drag and AX editor-save. A
background capability refusal is a valid result only when the test verifies the
declared structured refusal and the no-focus/no-z-order/no-input-leak side
-effect. A refusal fails a cell whose contract requires delivery.
+effect. Background posture requires one foreground window to contain the target
+geometry, not merely overlap it. The lane preflight also injects deliberate
+focus and input violations and requires the sentinel to detect both before its
+results are trusted. A refusal fails a cell whose contract requires delivery.
+Canonical reporting rejects skips and enforces the complete shared catalog
+size for unfiltered runs. Diagnostic filters are allowed, but matching zero
+cells is always an error.
## Harness E2E: Native Windows
diff --git a/libs/cua-driver/rust/crates/cua-driver-testkit/src/e2e.rs b/libs/cua-driver/rust/crates/cua-driver-testkit/src/e2e.rs
index d47fdc136d..3785e2eac5 100644
--- a/libs/cua-driver/rust/crates/cua-driver-testkit/src/e2e.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-testkit/src/e2e.rs
@@ -166,6 +166,13 @@ fn shared_web_route_for_environment(
) -> Result<DriverRoute, String> {
use DriverRoute as Route;
+ if action == "type_submit" && matches!(targeting, Targeting::Ax | Targeting::Px) {
+ return match delivery {
+ Delivery::Background | Delivery::Foreground => Ok(Route::Composite),
+ Delivery::NotApplicable => Err(format!("{action}: delivery mode is required")),
+ };
+ }
+
let pointer_or_key_route = |background, foreground| match delivery {
Delivery::Background => Ok(background),
Delivery::Foreground => Ok(foreground),
@@ -1159,9 +1166,18 @@ pub fn validate_catalog_with_evidence(
require_turn_evidence: bool,
) -> Result<ValidationSummary, Vec<String>> {
let mut errors = Vec::new();
+ let policy = CatalogPolicy::from_environment(&mut errors);
if declarations.is_empty() {
errors.push("E2E catalog has no declarations".to_owned());
}
+ if let Some(minimum) = policy.minimum_cells {
+ if declarations.len() < minimum {
+ errors.push(format!(
+ "E2E catalog declared {} cells, below the required minimum of {minimum}",
+ declarations.len()
+ ));
+ }
+ }
let mut declared = BTreeMap::new();
for case in declarations {
if let Err(error) = case.validate() {
@@ -1213,7 +1229,14 @@ pub fn validate_catalog_with_evidence(
_ => errors.push(format!("passing cell has no valid behavior: {cell_id}")),
},
TestStatus::Fail | TestStatus::EnvironmentError => summary.failed += 1,
- TestStatus::Skip => summary.skipped += 1,
+ TestStatus::Skip => {
+ summary.skipped += 1;
+ if policy.forbid_skips {
+ errors.push(format!(
+ "canonical E2E catalog may not skip cell: {cell_id}"
+ ));
+ }
+ }
}
if require_video {
@@ -1267,9 +1290,53 @@ pub fn validate_catalog_with_evidence(
}
}
+#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
+struct CatalogPolicy {
+ minimum_cells: Option<usize>,
+ forbid_skips: bool,
+}
+
+impl CatalogPolicy {
+ fn from_environment(errors: &mut Vec<String>) -> Self {
+ Self::parse(
+ std::env::var("CUA_E2E_EXPECTED_MIN_CELLS").ok().as_deref(),
+ std::env::var("CUA_E2E_FORBID_SKIPS").ok().as_deref(),
+ errors,
+ )
+ }
+
+ fn parse(minimum: Option<&str>, forbid_skips: Option<&str>, errors: &mut Vec<String>) -> Self {
+ let minimum_cells = minimum.and_then(|value| match value.parse::<usize>() {
+ Ok(value) if value > 0 => Some(value),
+ _ => {
+ errors.push(format!(
+ "CUA_E2E_EXPECTED_MIN_CELLS must be a positive integer, got {value:?}"
+ ));
+ None
+ }
+ });
+ let forbid_skips = match forbid_skips {
+ None | Some("") | Some("0") | Some("false") => false,
+ Some("1") | Some("true") => true,
+ Some(value) => {
+ errors.push(format!(
+ "CUA_E2E_FORBID_SKIPS must be 0/1/false/true, got {value:?}"
+ ));
+ false
+ }
+ };
+ Self {
+ minimum_cells,
+ forbid_skips,
+ }
+ }
+}
+
fn case_requires_action_turn(case: &CaseSpec) -> bool {
- !matches!(case.driver_route, DriverRoute::AxRead | DriverRoute::WindowState)
- && case.action != "screenshot"
+ !matches!(
+ case.driver_route,
+ DriverRoute::AxRead | DriverRoute::WindowState
+ ) && case.action != "screenshot"
}
fn recording_directory(evidence: &Evidence) -> Option<&Path> {
@@ -1317,7 +1384,9 @@ fn validate_turn_evidence(
));
}
}
- Err(error) => errors.push(format!("invalid trajectory evidence for {cell_id}: {error}")),
+ Err(error) => errors.push(format!(
+ "invalid trajectory evidence for {cell_id}: {error}"
+ )),
}
let mut turns = match std::fs::read_dir(recording_dir) {
@@ -1393,9 +1462,9 @@ fn validate_one_turn(turn: &Path, cell_id: &str, errors: &mut Vec<String>) {
.as_str()
.is_some_and(|summary| summary.starts_with("✅ bring_to_front:"))
});
- let restored_state_captured = manifest.as_ref().is_some_and(|value| {
- value["after"]["state"]["status"].as_str() == Some("captured")
- });
+ let restored_state_captured = manifest
+ .as_ref()
+ .is_some_and(|value| value["after"]["state"]["status"].as_str() == Some("captured"));
let expected_unavailable_screenshot = |phase: &str| {
tool == Some("bring_to_front")
&& ((phase == "before"
@@ -1743,6 +1812,20 @@ mod tests {
assert!(errors.iter().any(|error| error.contains("no declarations")));
}
+ #[test]
+ fn strict_catalog_policy_is_explicit_and_rejects_invalid_values() {
+ let mut errors = Vec::new();
+ let policy = CatalogPolicy::parse(Some("80"), Some("true"), &mut errors);
+ assert!(errors.is_empty());
+ assert_eq!(policy.minimum_cells, Some(80));
+ assert!(policy.forbid_skips);
+
+ let mut errors = Vec::new();
+ let policy = CatalogPolicy::parse(Some("0"), Some("sometimes"), &mut errors);
+ assert_eq!(policy, CatalogPolicy::default());
+ assert_eq!(errors.len(), 2);
+ }
+
#[test]
fn validator_rejects_contradictory_status_and_missing_video() {
let case = delivered_case("contradiction");
@@ -1895,7 +1978,9 @@ mod tests {
let errors = validate_catalog(&[case], &[result], Some(root.path()), true)
.expect_err("mutable not-applicable cells still need a recorded turn");
- assert!(errors.iter().any(|error| error.contains("missing turn evidence")));
+ assert!(errors
+ .iter()
+ .any(|error| error.contains("missing turn evidence")));
}
#[test]
@@ -2284,6 +2369,7 @@ mod tests {
"right_click",
"double_click",
"type_text",
+ "type_submit",
"press_key",
"hotkey",
"scroll",
@@ -2299,7 +2385,7 @@ mod tests {
cells.push(("drag", Targeting::Px, delivery));
cells.push(("editor_save", Targeting::Ax, delivery));
}
- assert_eq!(cells.len(), 36);
+ assert_eq!(cells.len(), 40);
for (platform, display_server) in [
(Platform::Windows, DisplayServer::Win32),
(Platform::Macos, DisplayServer::Quartz),
diff --git a/libs/cua-driver/rust/crates/cua-driver-testkit/src/observer.rs b/libs/cua-driver/rust/crates/cua-driver-testkit/src/observer.rs
index 90f2ce9fc7..1a6235ef87 100644
--- a/libs/cua-driver/rust/crates/cua-driver-testkit/src/observer.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-testkit/src/observer.rs
@@ -440,14 +440,32 @@ pub mod windows {
},
];
let target_root = unsafe { root(hwnd) };
- let covered = points
- .into_iter()
- .filter(|point| {
- let owner = unsafe { WindowFromPoint(*point) };
- !owner.0.is_null() && unsafe { root(owner) } != target_root
- })
- .count();
- Ok(covered >= 2)
+ let mut cover_root = None;
+ for point in points {
+ let owner = unsafe { root(WindowFromPoint(point)) };
+ if owner.0.is_null() || owner == target_root {
+ return Ok(false);
+ }
+ match cover_root {
+ Some(expected) if expected != owner => return Ok(false),
+ None => cover_root = Some(owner),
+ _ => {}
+ }
+ }
+ let Some(cover_root) = cover_root else {
+ return Ok(false);
+ };
+ let mut cover = RECT::default();
+ unsafe { GetWindowRect(cover_root, &mut cover) }
+ .map_err(|error| ObserverError::new(format!("read covering window bounds: {error}")))?;
+ Ok(rect_fully_contains(cover, rect, 2))
+ }
+
+ fn rect_fully_contains(cover: RECT, target: RECT, tolerance: i32) -> bool {
+ cover.left <= target.left + tolerance
+ && cover.top <= target.top + tolerance
+ && cover.right >= target.right - tolerance
+ && cover.bottom >= target.bottom - tolerance
}
}
@@ -506,12 +524,11 @@ pub mod macos {
self.width.max(0.0) * self.height.max(0.0)
}
- fn intersection_area(self, other: Self) -> f64 {
- let left = self.x.max(other.x);
- let top = self.y.max(other.y);
- let right = (self.x + self.width).min(other.x + other.width);
- let bottom = (self.y + self.height).min(other.y + other.height);
- (right - left).max(0.0) * (bottom - top).max(0.0)
+ fn fully_contains(self, other: Self, tolerance: f64) -> bool {
+ self.x <= other.x + tolerance
+ && self.y <= other.y + tolerance
+ && self.x + self.width >= other.x + other.width - tolerance
+ && self.y + self.height >= other.y + other.height - tolerance
}
}
@@ -563,12 +580,11 @@ pub mod macos {
Some(_) if frontmost_pid() == Some(target.pid as u64) => TargetZ::Foreground,
Some(index) => {
let target_bounds = rows[index].bounds;
- let meaningful_cover = rows[..index].iter().any(|row| {
- row.pid != target.pid
- && target_bounds.intersection_area(row.bounds)
- >= target_bounds.area() * 0.10
- });
- if meaningful_cover {
+ let full_cover = target_bounds.area() > 0.0
+ && rows[..index].iter().any(|row| {
+ row.pid != target.pid && row.bounds.fully_contains(target_bounds, 2.0)
+ });
+ if full_cover {
TargetZ::BackgroundOccluded
} else {
TargetZ::BackgroundVisible
@@ -738,6 +754,30 @@ pub mod macos {
mod tests {
use super::*;
+ #[test]
+ fn partial_overlap_is_not_full_occlusion() {
+ let target = Bounds {
+ x: 100.0,
+ y: 100.0,
+ width: 400.0,
+ height: 300.0,
+ };
+ let partial = Bounds {
+ x: 0.0,
+ y: 0.0,
+ width: 350.0,
+ height: 900.0,
+ };
+ let full = Bounds {
+ x: 0.0,
+ y: 0.0,
+ width: 1280.0,
+ height: 900.0,
+ };
+ assert!(!partial.fully_contains(target, 2.0));
+ assert!(full.fully_contains(target, 2.0));
+ }
+
#[test]
fn native_snapshot_reads_desktop_without_a_target() {
let snapshot = MacosObserver::new()
@@ -1216,7 +1256,7 @@ pub mod linux {
target_pid: u32,
workspace: Option<u64>,
state: &mut SwayTreeState,
- ) {
+ ) -> bool {
let id = node["id"].as_u64();
let workspace = if node["type"].as_str() == Some("workspace") {
id
@@ -1227,21 +1267,26 @@ pub mod linux {
if focused {
state.focused = id;
state.focused_workspace = workspace;
- if node["fullscreen_mode"].as_i64().unwrap_or(0) != 0 {
- state.focused_fullscreen = id;
- }
}
if node["pid"].as_u64() == Some(u64::from(target_pid)) {
if let Some(id) = id {
state.target = Some((id, node["visible"].as_bool().unwrap_or(true), workspace));
}
}
+ let mut subtree_focused = focused;
for child in ["nodes", "floating_nodes"]
.into_iter()
.flat_map(|key| node[key].as_array().into_iter().flatten())
{
- walk_sway_tree(child, target_pid, workspace, state);
+ subtree_focused |= walk_sway_tree(child, target_pid, workspace, state);
}
+ // Sway commonly puts fullscreen_mode on a container while focus belongs
+ // to a descendant surface. Treat the focused subtree as fullscreen so a
+ // sibling hidden behind that container is classified as occluded.
+ if subtree_focused && node["fullscreen_mode"].as_i64().unwrap_or(0) != 0 {
+ state.focused_fullscreen = id;
+ }
+ subtree_focused
}
fn classify_sway_target(state: &SwayTreeState) -> TargetZ {
@@ -1264,7 +1309,7 @@ pub mod linux {
fn sway_snapshot(target: TargetWindow) -> Result<DesktopSnapshot, ObserverError> {
let tree = sway_tree()?;
let mut state = SwayTreeState::default();
- walk_sway_tree(&tree, target.pid, None, &mut state);
+ let _ = walk_sway_tree(&tree, target.pid, None, &mut state);
let target_z = classify_sway_target(&state);
Ok(DesktopSnapshot {
foreground: state.focused,
@@ -1277,7 +1322,7 @@ pub mod linux {
fn sway_focus_identity() -> Result<Option<u64>, ObserverError> {
let tree = sway_tree()?;
let mut state = SwayTreeState::default();
- walk_sway_tree(&tree, 0, None, &mut state);
+ let _ = walk_sway_tree(&tree, 0, None, &mut state);
Ok(state.focused)
}
@@ -1453,18 +1498,43 @@ pub mod linux {
if target_bounds.width <= 4 || target_bounds.height <= 4 {
return Ok(false);
}
- let covered = sample_points(target_bounds)
- .into_iter()
- .filter(|(x, y)| {
- connection
- .translate_coordinates(root, root, *x, *y)
- .ok()
- .and_then(|cookie| cookie.reply().ok())
- .map(|reply| reply.child != 0 && reply.child != target)
- .unwrap_or(true)
- })
- .count();
- Ok(covered >= 2)
+ let mut cover = None;
+ for (x, y) in sample_points(target_bounds) {
+ let child = connection
+ .translate_coordinates(root, root, x, y)
+ .map_err(x11_error("request X11 occlusion point"))?
+ .reply()
+ .map_err(x11_error("read X11 occlusion point"))?
+ .child;
+ if child == 0 || child == target {
+ return Ok(false);
+ }
+ match cover {
+ Some(expected) if expected != child => return Ok(false),
+ None => cover = Some(child),
+ _ => {}
+ }
+ }
+ let Some(cover) = cover else {
+ return Ok(false);
+ };
+ let cover_bounds = absolute_bounds(connection, cover, root)?;
+ Ok(rectangle_fully_contains(cover_bounds, target_bounds, 2))
+ }
+
+ fn rectangle_fully_contains(cover: Rectangle, target: Rectangle, tolerance: i32) -> bool {
+ let cover_left = i32::from(cover.x);
+ let cover_top = i32::from(cover.y);
+ let cover_right = cover_left + i32::from(cover.width);
+ let cover_bottom = cover_top + i32::from(cover.height);
+ let target_left = i32::from(target.x);
+ let target_top = i32::from(target.y);
+ let target_right = target_left + i32::from(target.width);
+ let target_bottom = target_top + i32::from(target.height);
+ cover_left <= target_left + tolerance
+ && cover_top <= target_top + tolerance
+ && cover_right >= target_right - tolerance
+ && cover_bottom >= target_bottom - tolerance
}
fn sample_points(bounds: Rectangle) -> [(i16, i16); 5] {
@@ -1531,6 +1601,30 @@ pub mod linux {
);
}
+ #[test]
+ fn x11_partial_overlap_is_not_full_occlusion() {
+ let target = Rectangle {
+ x: 100,
+ y: 100,
+ width: 400,
+ height: 300,
+ };
+ let partial = Rectangle {
+ x: 0,
+ y: 0,
+ width: 350,
+ height: 900,
+ };
+ let full = Rectangle {
+ x: 0,
+ y: 0,
+ width: 1280,
+ height: 900,
+ };
+ assert!(!rectangle_fully_contains(partial, target, 2));
+ assert!(rectangle_fully_contains(full, target, 2));
+ }
+
#[test]
fn sway_hidden_target_behind_fullscreen_sibling_is_occluded() {
let state = SwayTreeState {
@@ -1542,6 +1636,46 @@ pub mod linux {
assert_eq!(classify_sway_target(&state), TargetZ::BackgroundOccluded);
}
+ #[test]
+ fn sway_fullscreen_ancestor_of_focused_surface_occludes_target() {
+ let tree = serde_json::json!({
+ "id": 1,
+ "type": "root",
+ "nodes": [{
+ "id": 10,
+ "type": "workspace",
+ "nodes": [
+ {
+ "id": 20,
+ "pid": 100,
+ "visible": false,
+ "focused": false,
+ "fullscreen_mode": 0,
+ "nodes": []
+ },
+ {
+ "id": 30,
+ "focused": false,
+ "fullscreen_mode": 1,
+ "nodes": [{
+ "id": 31,
+ "pid": 200,
+ "visible": true,
+ "focused": true,
+ "fullscreen_mode": 0,
+ "nodes": []
+ }]
+ }
+ ]
+ }]
+ });
+ let mut state = SwayTreeState::default();
+ assert!(walk_sway_tree(&tree, 100, None, &mut state));
+ assert_eq!(state.focused, Some(31));
+ assert_eq!(state.focused_fullscreen, Some(30));
+ assert_eq!(classify_sway_target(&state), TargetZ::BackgroundOccluded);
+ }
+
#[test]
fn sway_hidden_target_on_another_workspace_is_not_occluded() {
let state = SwayTreeState {
diff --git a/libs/cua-driver/rust/crates/cua-driver-testkit/src/sentinel.rs b/libs/cua-driver/rust/crates/cua-driver-testkit/src/sentinel.rs
index 3dd32c7288..3d707f30df 100644
--- a/libs/cua-driver/rust/crates/cua-driver-testkit/src/sentinel.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-testkit/src/sentinel.rs
@@ -115,27 +115,50 @@ impl ForegroundSentinel {
pub fn observe(&self) -> (Vec<OracleKind>, Vec<String>) {
std::thread::sleep(Duration::from_millis(200));
- let journal = match fs::read_to_string(&self.journal_path) {
- Ok(journal) => journal,
+ let events = match read_journal_events(&self.journal_path) {
+ Ok(events) => events,
Err(error) => {
return (
Vec::new(),
- vec![format!(
- "foreground sentinel journal could not be read: {error}"
- )],
+ vec![format!("foreground sentinel journal is invalid: {error}")],
)
}
};
let mut passed = Vec::new();
let mut violations = Vec::new();
- if journal.contains(r#""kind":"blur""#) {
- violations.push("foreground sentinel lost focus".to_owned());
- } else {
- passed.push(OracleKind::Focus);
+ if !events
+ .iter()
+ .any(|event| event_kind(event) == Some("heartbeat"))
+ {
+ violations.push("foreground sentinel heartbeat stopped".to_owned());
+ }
+ if !is_wayland_session() {
+ if events.iter().any(|event| {
+ event_kind(event) == Some("blur")
+ || (event_kind(event) == Some("visibility")
+ && event["state"].as_str() == Some("hidden"))
+ }) {
+ violations.push("foreground sentinel lost focus".to_owned());
+ } else {
+ passed.push(OracleKind::Focus);
+ }
}
- let leaked = ["keydown", "pointerdown", "wheel", "contextmenu"]
- .into_iter()
- .filter(|kind| journal.contains(&format!(r#""kind":"{kind}""#)))
+ let leaked = events
+ .iter()
+ .filter_map(|event| event_kind(event))
+ .filter(|kind| {
+ matches!(
+ *kind,
+ "keydown"
+ | "keyup"
+ | "pointerdown"
+ | "pointerup"
+ | "click"
+ | "wheel"
+ | "contextmenu"
+ | "input"
+ )
+ })
.collect::<Vec<_>>();
if leaked.is_empty() {
passed.push(OracleKind::NoLeakedInput);
@@ -148,6 +171,83 @@ impl ForegroundSentinel {
(passed, violations)
}
+ /// Prove once per canonical lane that the foreground guard can detect both
+ /// leaked input and transient focus loss. A guard that cannot observe its
+ /// deliberate violations must never be trusted to certify background rows.
+ pub fn assert_guard_canaries(
+ &self,
+ driver: &mut impl Driver,
+ background_target: TargetWindow,
+ ) -> Result<(), String> {
+ wait_for_event(&self.journal_path, "heartbeat", Duration::from_secs(2))?;
+ reset_journal(&self.journal_path)?;
+
+ let leaked_key = driver.call(
+ "press_key",
+ serde_json::json!({
+ "pid": self.target.pid,
+ "window_id": self.target.native_id,
+ "key": "a",
+ "delivery_mode": "foreground",
+ }),
+ );
+ if leaked_key.is_error() {
+ return Err(format!(
+ "foreground input canary could not inject a key: {}",
+ leaked_key.text()
+ ));
+ }
+ wait_for_event(&self.journal_path, "keydown", Duration::from_secs(2))?;
+ let (_, leaked_input_violations) = self.observe();
+ if !leaked_input_violations
+ .iter()
+ .any(|violation| violation.contains("received input events"))
+ {
+ return Err(format!(
+ "foreground input canary was not detected: {leaked_input_violations:?}"
+ ));
+ }
+ reset_journal(&self.journal_path)?;
+
+ let raised = driver.call(
+ "bring_to_front",
+ serde_json::json!({
+ "pid": background_target.pid,
+ "window_id": background_target.native_id,
+ }),
+ );
+ if raised.is_error() {
+ return Err(format!(
+ "focus-loss canary could not raise the background target: {}",
+ raised.text()
+ ));
+ }
+ #[cfg(target_os = "linux")]
+ focus_sway_target(driver, background_target)?;
+ if is_wayland_session() {
+ wait_for_native_focus_lost(self.target)?;
+ } else {
+ wait_for_event(&self.journal_path, "blur", Duration::from_secs(3))?;
+ let (_, focus_violations) = self.observe();
+ if !focus_violations
+ .iter()
+ .any(|violation| violation.contains("lost focus"))
+ {
+ return Err(format!(
+ "focus-loss canary was not detected: {focus_violations:?}"
+ ));
+ }
+ }
+
+ activate_native_foreground(driver, self.target);
+ wait_for_native_focus_stable(self.target);
+ std::thread::sleep(Duration::from_millis(150));
+ reset_journal(&self.journal_path)?;
+ wait_for_event(&self.journal_path, "heartbeat", Duration::from_secs(2))?;
+ reset_journal(&self.journal_path)?;
+ self.assert_background_posture(background_target)
+ }
+
pub fn target(&self) -> TargetWindow {
self.target
}
@@ -178,8 +278,7 @@ impl ForegroundSentinel {
activate_native_foreground(driver, self.target);
wait_for_native_focus_stable(self.target);
std::thread::sleep(Duration::from_millis(100));
- fs::write(&self.journal_path, "")
- .map_err(|error| format!("reset foreground sentinel journal: {error}"))?;
+ reset_journal(&self.journal_path)?;
self.assert_background_posture(target)
}
@@ -271,6 +370,43 @@ fn wait_for_journal(path: &std::path::Path, deadline: Instant, marker: &str, sta
}
}
+fn reset_journal(path: &std::path::Path) -> Result<(), String> {
+ fs::write(path, "").map_err(|error| format!("reset foreground sentinel journal: {error}"))
+}
+
+fn read_journal_events(path: &std::path::Path) -> Result<Vec<serde_json::Value>, String> {
+ let journal =
+ fs::read_to_string(path).map_err(|error| format!("read {}: {error}", path.display()))?;
+ journal
+ .lines()
+ .filter(|line| !line.trim().is_empty())
+ .map(|line| {
+ serde_json::from_str(line)
+ .map_err(|error| format!("parse sentinel event {line:?}: {error}"))
+ })
+ .collect()
+}
+
+fn event_kind(event: &serde_json::Value) -> Option<&str> {
+ event["kind"].as_str()
+}
+
+fn wait_for_event(path: &std::path::Path, kind: &str, timeout: Duration) -> Result<(), String> {
+ let deadline = Instant::now() + timeout;
+ loop {
+ let events = read_journal_events(path)?;
+ if events.iter().any(|event| event_kind(event) == Some(kind)) {
+ return Ok(());
+ }
+ if Instant::now() >= deadline {
+ return Err(format!(
+ "foreground sentinel did not emit {kind:?} within {timeout:?}: {events:?}"
+ ));
+ }
+ std::thread::sleep(Duration::from_millis(25));
+ }
+}
+
fn is_wayland_session() -> bool {
cfg!(target_os = "linux")
&& std::env::var("XDG_SESSION_TYPE")
@@ -290,10 +426,192 @@ fn activate_native_foreground(driver: &mut impl Driver, target: TargetWindow) {
"could not activate foreground sentinel: {}",
response.text()
);
+ #[cfg(target_os = "linux")]
+ focus_sway_target(driver, target)
+ .expect("could not focus foreground sentinel through Sway IPC");
#[cfg(target_os = "windows")]
physically_focus_windows_sentinel(target);
}
+#[cfg(target_os = "linux")]
+fn focus_sway_target(driver: &mut impl Driver, target: TargetWindow) -> Result<(), String> {
+ let is_sway = is_wayland_session()
+ && std::env::var("CUA_E2E_WAYLAND_SESSION").is_ok_and(|session| session == "sway");
+ if !is_sway {
+ return Ok(());
+ }
+
+ let tree_output = Command::new("swaymsg")
+ .args(["-r", "-t", "get_tree"])
+ .output()
+ .map_err(|error| format!("read Sway tree for focus canary: {error}"))?;
+ if !tree_output.status.success() {
+ return Err(format!(
+ "read Sway tree for focus canary: {}",
+ String::from_utf8_lossy(&tree_output.stderr).trim()
+ ));
+ }
+ let tree: serde_json::Value = serde_json::from_slice(&tree_output.stdout)
+ .map_err(|error| format!("parse Sway tree for focus canary: {error}"))?;
+
+ let windows = driver.call("list_windows", serde_json::json!({}));
+ if windows.is_error() {
+ return Err(format!(
+ "list windows before Sway focus canary: {}",
+ windows.text()
+ ));
+ }
+ let title = sway_target_title(windows.structured(), target);
+ let con_id = title
+ .and_then(|title| sway_container_id(&tree, title))
+ .or_else(|| sway_container_id_by_unique_pid(&tree, target.pid))
+ .ok_or_else(|| {
+ format!(
+ "Sway focus canary could not resolve container for pid {} window {}",
+ target.pid, target.native_id
+ )
+ })?;
+
+ let criterion = format!("[con_id={con_id}]");
+ let output = Command::new("swaymsg")
+ .args(["-r", criterion.as_str(), "focus"])
+ .output()
+ .map_err(|error| format!("run Sway focus canary for con_id {con_id}: {error}"))?;
+ let result: serde_json::Value = serde_json::from_slice(&output.stdout)
+ .map_err(|error| format!("parse Sway focus result for con_id {con_id}: {error}"))?;
+ let focused = output.status.success()
+ && result.as_array().is_some_and(|results| {
+ !results.is_empty() && results.iter().all(|item| item["success"] == true)
+ });
+ if focused {
+ Ok(())
+ } else {
+ Err(format!(
+ "Sway focus canary for con_id {con_id} failed: stdout={} stderr={}",
+ String::from_utf8_lossy(&output.stdout).trim(),
+ String::from_utf8_lossy(&output.stderr).trim(),
+ ))
+ }
+}
+
+#[cfg(target_os = "linux")]
+fn sway_target_title(windows: &serde_json::Value, target: TargetWindow) -> Option<&str> {
+ let windows = windows["windows"].as_array()?;
+ windows
+ .iter()
+ .find(|window| {
+ window["pid"].as_u64() == Some(target.pid as u64)
+ && window["window_id"].as_u64() == Some(target.native_id)
+ })
+ .and_then(|window| window["title"].as_str())
+ .or_else(|| {
+ let mut titles = windows
+ .iter()
+ .filter(|window| window["pid"].as_u64() == Some(target.pid as u64))
+ .filter_map(|window| window["title"].as_str())
+ .filter(|title| !title.is_empty());
+ let title = titles.next()?;
+ titles.next().is_none().then_some(title)
+ })
+}
+
+#[cfg(target_os = "linux")]
+fn collect_sway_container_ids_by_pid(node: &serde_json::Value, pid: u32, ids: &mut Vec<i64>) {
+ if node["pid"].as_u64() == Some(pid as u64) {
+ if let Some(id) = node["id"].as_i64() {
+ ids.push(id);
+ }
+ }
+ for key in ["nodes", "floating_nodes"] {
+ if let Some(children) = node[key].as_array() {
+ for child in children {
+ collect_sway_container_ids_by_pid(child, pid, ids);
+ }
+ }
+ }
+}
+
+#[cfg(target_os = "linux")]
+fn sway_container_id_by_unique_pid(node: &serde_json::Value, pid: u32) -> Option<i64> {
+ let mut ids = Vec::new();
+ collect_sway_container_ids_by_pid(node, pid, &mut ids);
+ (ids.len() == 1).then(|| ids[0])
+}
+
+#[cfg(target_os = "linux")]
+fn sway_container_id(node: &serde_json::Value, title: &str) -> Option<i64> {
+ let raw_title = title
+ .rsplit_once(" [")
+ .map(|(candidate, _)| candidate)
+ .unwrap_or(title);
+ if node["name"]
+ .as_str()
+ .is_some_and(|name| name == title || name == raw_title)
+ {
+ return node["id"].as_i64();
+ }
+ ["nodes", "floating_nodes"].into_iter().find_map(|key| {
+ node[key].as_array().and_then(|children| {
+ children
+ .iter()
+ .find_map(|child| sway_container_id(child, title))
+ })
+ })
+}
+
+#[cfg(all(test, target_os = "linux"))]
+mod sway_tests {
+ use super::*;
+
+ #[test]
+ fn focus_target_uses_unique_sway_pid_when_title_is_unavailable() {
+ let tree = serde_json::json!({
+ "id": 1,
+ "nodes": [{
+ "id": 9,
+ "nodes": [{
+ "id": 42,
+ "pid": 1234,
+ "name": "CuaTestHarness Electron"
+ }]
+ }]
+ });
+ assert_eq!(sway_container_id_by_unique_pid(&tree, 1234), Some(42));
+ }
+
+ #[test]
+ fn focus_target_does_not_guess_when_pid_owns_multiple_windows() {
+ let tree = serde_json::json!({
+ "id": 1,
+ "nodes": [
+ {"id": 41, "pid": 1234, "name": "First"},
+ {"id": 42, "pid": 1234, "name": "Second"}
+ ]
+ });
+ assert_eq!(sway_container_id_by_unique_pid(&tree, 1234), None);
+ assert_eq!(sway_container_id(&tree, "Second"), Some(42));
+ }
+
+ #[test]
+ fn focus_target_uses_unique_same_process_title_when_window_id_changes() {
+ let windows = serde_json::json!({
+ "windows": [{
+ "pid": 1234,
+ "window_id": 88,
+ "title": "CuaTestHarness Electron [cdp=9223]"
+ }]
+ });
+ let target = TargetWindow {
+ pid: 1234,
+ native_id: 99,
+ };
+ assert_eq!(
+ sway_target_title(&windows, target),
+ Some("CuaTestHarness Electron [cdp=9223]")
+ );
+ }
+}
+
#[cfg(target_os = "windows")]
fn physically_focus_windows_sentinel(target: TargetWindow) {
use windows::Win32::Foundation::{HWND, POINT, RECT};
@@ -374,6 +692,32 @@ fn wait_for_native_focus_stable(target: TargetWindow) {
}
}
+#[cfg(target_os = "linux")]
+fn wait_for_native_focus_lost(target: TargetWindow) -> Result<(), String> {
+ use crate::observer::{ObserverBackend, TargetZ};
+
+ let backend = NativeObserver::new();
+ let deadline = Instant::now() + Duration::from_secs(3);
+ loop {
+ let lost = backend
+ .snapshot(target)
+ .map(|snapshot| snapshot.target_z != TargetZ::Foreground)
+ .unwrap_or(false);
+ if lost {
+ return Ok(());
+ }
+ if Instant::now() >= deadline {
+ return Err("native Wayland focus-loss canary was not detected".to_owned());
+ }
+ std::thread::sleep(Duration::from_millis(25));
+ }
+}
+
+#[cfg(not(target_os = "linux"))]
+fn wait_for_native_focus_lost(_target: TargetWindow) -> Result<(), String> {
+ Err("native Wayland focus observation is only available on Linux".to_owned())
+}
+
#[cfg(not(any(target_os = "windows", target_os = "linux")))]
fn wait_for_native_focus_stable(_target: TargetWindow) {}
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rs
index e1944140f6..ec7e1f539a 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rs
@@ -399,8 +399,7 @@ fn refresh_wayland_window_geometry(fixture: &mut Fixture) {
}
#[cfg(not(target_os = "linux"))]
-fn refresh_wayland_window_geometry(_fixture: &mut Fixture) {
-}
+fn refresh_wayland_window_geometry(_fixture: &mut Fixture) {}
fn window_origin(fixture: &Fixture, _state: &ToolResponse) -> (f64, f64) {
#[cfg(target_os = "macos")]
@@ -511,6 +510,78 @@ fn assert_fixture_contains(fixture: &Fixture, marker: &str) {
}
}
+fn assert_fixture_value(fixture: &Fixture, id: &str, expected: &str) {
+ let deadline = Instant::now() + Duration::from_secs(2);
+ loop {
+ let state = fixture.journal.snapshot();
+ if state[id]["value"].as_str() == Some(expected) {
+ return;
+ }
+ if Instant::now() >= deadline {
+ panic!(
+ "{}: fixture value {id:?} did not reach {expected:?}: {state}",
+ fixture.name
+ );
+ }
+ thread::sleep(Duration::from_millis(100));
+ }
+}
+
+fn assert_fixture_single_insertion(
+ fixture: &Fixture,
+ id: &str,
+ initial: &str,
+ inserted: &str,
+) -> String {
+ let deadline = Instant::now() + Duration::from_secs(2);
+ loop {
+ let state = fixture.journal.snapshot();
+ if let Some(value) = state[id]["value"].as_str() {
+ if value.len() == initial.len() + inserted.len() {
+ for (offset, _) in value.match_indices(inserted) {
+ let mut without_insertion = value.to_owned();
+ without_insertion.replace_range(offset..offset + inserted.len(), "");
+ if without_insertion == initial {
+ return value.to_owned();
+ }
+ }
+ }
+ }
+ if Instant::now() >= deadline {
+ panic!(
+ "{}: fixture value {id:?} did not insert {inserted:?} exactly once into {initial:?}: {state}",
+ fixture.name
+ );
+ }
+ thread::sleep(Duration::from_millis(100));
+ }
+}
+
+fn assert_fixture_text(fixture: &Fixture, id: &str, expected: &str) {
+ let deadline = Instant::now() + Duration::from_secs(2);
+ loop {
+ let state = fixture.journal.snapshot();
+ if state[id]["text"].as_str() == Some(expected) {
+ return;
+ }
+ if Instant::now() >= deadline {
+ panic!(
+ "{}: fixture text {id:?} did not reach {expected:?}: {state}",
+ fixture.name
+ );
+ }
+ thread::sleep(Duration::from_millis(100));
+ }
+}
+
+fn fixture_marker_number(fixture: &Fixture, id: &str, prefix: &str) -> Option<u64> {
+ fixture
+ .journal
+ .text(id)
+ .and_then(|text| text.strip_prefix(prefix).map(str::to_owned))
+ .and_then(|value| value.parse().ok())
+}
+
fn action_target_args(
fixture: &Fixture,
state: &ToolResponse,
@@ -661,6 +732,60 @@ fn run_text_action(fixture: &mut Fixture, addressing: &str, delivery: &str) -> O
Observation::delivered(passed, Evidence::default())
}
+fn run_type_submit_action(fixture: &mut Fixture, addressing: &str, delivery: &str) -> Observation {
+ let pre = snapshot(fixture);
+ let journal_before = fixture.journal.snapshot();
+ let text = format!("cua-submit-{addressing}-{delivery}");
+ let mut type_args = action_target_args(fixture, &pre, "keyboard-input", addressing, delivery);
+ type_args
+ .as_object_mut()
+ .expect("type-submit type_text arguments object")
+ .insert("text".to_owned(), serde_json::json!(text));
+ let type_response = fixture.driver.call("type_text", type_args);
+ if let Some(code) = background_refusal_code(&type_response, delivery) {
+ return refused_without_fixture_mutation(fixture, &journal_before, code, &type_response);
+ }
+ assert!(
+ !type_response.is_error(),
+ "{}: type-submit type_text {addressing}/{delivery} failed: {}",
+ fixture.name,
+ type_response.text()
+ );
+ assert_fixture_value(fixture, "keyboard-input", &text);
+
+ let post_type = snapshot(fixture);
+ let mut enter_args =
+ action_target_args(fixture, &post_type, "keyboard-input", addressing, delivery);
+ enter_args
+ .as_object_mut()
+ .expect("type-submit press_key arguments object")
+ .insert("key".to_owned(), serde_json::json!("return"));
+ let enter_response = fixture.driver.call("press_key", enter_args);
+ if let Some(code) = background_refusal_code(&enter_response, delivery) {
+ return refused_without_fixture_mutation(fixture, &journal_before, code, &enter_response);
+ }
+ assert!(
+ !enter_response.is_error(),
+ "{}: type-submit Enter {addressing}/{delivery} failed: {}",
+ fixture.name,
+ enter_response.text()
+ );
+ assert_fixture_contains(fixture, &format!("key_state=submitted:{text}"));
+
+ let mut passed = vec![OracleKind::FixtureState];
+ if addressing == "px" {
+ passed.extend(unverified_background_protocol_oracle(
+ &type_response,
+ delivery,
+ ));
+ }
+ passed.extend(unverified_background_protocol_oracle(
+ &enter_response,
+ delivery,
+ ));
+ Observation::delivered(passed, Evidence::default())
+}
+
fn run_press_key_action(fixture: &mut Fixture, addressing: &str, delivery: &str) -> Observation {
let pre = snapshot(fixture);
let journal_before = fixture.journal.snapshot();
@@ -728,26 +853,26 @@ fn run_scroll_action(fixture: &mut Fixture, addressing: &str, delivery: &str) ->
response.text(),
response.raw
);
- thread::sleep(Duration::from_millis(250));
- let offset = fixture
- .journal
- .text("lbl-scroll-offset")
- .and_then(|text| text.split("scroll_offset=").nth(1).map(str::to_owned))
- .and_then(|tail| {
- tail.split(|ch: char| !ch.is_ascii_digit())
- .next()
- .map(str::to_owned)
- })
- .and_then(|value| value.parse::<u64>().ok())
- .unwrap_or(0);
- assert!(
- offset > 0,
- "{}: scroll did not advance the external oracle: {}; response={}; raw={}",
- fixture.name,
- fixture.journal.snapshot(),
- response.text(),
- response.raw
- );
+ let deadline = Instant::now() + Duration::from_secs(2);
+ loop {
+ let offset =
+ fixture_marker_number(fixture, "lbl-scroll-offset", "scroll_offset=").unwrap_or(0);
+ let page =
+ fixture_marker_number(fixture, "lbl-scroll-client-height", "scroll_client_height=")
+ .unwrap_or(0);
+ if page > 0 && offset >= page {
+ break;
+ }
+ assert!(
+ Instant::now() < deadline,
+ "{}: two-page scroll did not advance by at least one viewport: {}; response={}; raw={}",
+ fixture.name,
+ fixture.journal.snapshot(),
+ response.text(),
+ response.raw
+ );
+ thread::sleep(Duration::from_millis(100));
+ }
delivered_observation()
}
@@ -814,6 +939,14 @@ fn run_editor_save_action(fixture: &mut Fixture, delivery: &str) -> Observation
let pre = snapshot(fixture);
let journal_before = fixture.journal.snapshot();
let text = format!("cua-editor-{delivery}");
+ let initial_value = journal_before["editor-document"]["value"]
+ .as_str()
+ .unwrap_or_else(|| {
+ panic!(
+ "{}: editor fixture did not publish its initial value: {journal_before}",
+ fixture.name
+ )
+ });
let mut text_args = action_target_args(fixture, &pre, "editor-document", "ax", delivery);
text_args
.as_object_mut()
@@ -829,6 +962,8 @@ fn run_editor_save_action(fixture: &mut Fixture, delivery: &str) -> Observation
fixture.name,
response.text()
);
+ let expected_value =
+ assert_fixture_single_insertion(fixture, "editor-document", initial_value, &text);
let post_text = snapshot(fixture);
let journal_before_save = fixture.journal.snapshot();
@@ -843,7 +978,11 @@ fn run_editor_save_action(fixture: &mut Fixture, delivery: &str) -> Observation
fixture.name,
response.text()
);
- assert_fixture_contains(fixture, "editor_status=saved:");
+ assert_fixture_text(
+ fixture,
+ "editor-status",
+ &format!("editor_status=saved:{expected_value}"),
+ );
delivered_observation()
}
@@ -876,7 +1015,11 @@ fn shared_case(spec: &HostSpec, action: &str, addressing: &str, delivery: &str)
("electron", "right_click" | "double_click" | "drag", _) => {
vec![RefusalCode::BackgroundOccluded]
}
- ("electron", "type_text" | "press_key" | "hotkey" | "scroll" | "editor_save", _) => {
+ (
+ "electron",
+ "type_text" | "type_submit" | "press_key" | "hotkey" | "scroll" | "editor_save",
+ _,
+ ) => {
vec![RefusalCode::BackgroundUnavailable]
}
("tauri", "hotkey", _) | ("tauri", "scroll", Targeting::Px) => {
@@ -917,7 +1060,9 @@ fn shared_case(spec: &HostSpec, action: &str, addressing: &str, delivery: &str)
// must refuse those background keyboard composites instead of reporting
// a successful write that the page never observes.
match (action, targeting) {
- ("type_text", _) | ("editor_save", Targeting::Ax) | ("press_key" | "hotkey", _) => {
+ ("type_text" | "type_submit", _)
+ | ("editor_save", Targeting::Ax)
+ | ("press_key" | "hotkey", _) => {
vec![RefusalCode::BackgroundUnavailable]
}
("right_click" | "double_click" | "scroll", _) | ("drag", Targeting::Px)
@@ -949,7 +1094,7 @@ fn shared_case(spec: &HostSpec, action: &str, addressing: &str, delivery: &str)
}
if !expected_background_refusal
&& cfg!(target_os = "windows")
- && (matches!(action, "press_key" | "hotkey")
+ && (matches!(action, "press_key" | "hotkey" | "type_submit")
|| (action == "type_text" && targeting == Targeting::Px))
{
oracles.push(OracleKind::Protocol);
@@ -1010,12 +1155,6 @@ fn cell_selected(case: &CaseSpec) -> bool {
parts.peek().is_none() || parts.any(|part| case.cell_id == part || case.cell_id.contains(part))
}
-fn cell_filter_active() -> bool {
- std::env::var("CUA_E2E_CELL_FILTER")
- .map(|filter| filter.split(',').any(|part| !part.trim().is_empty()))
- .unwrap_or(false)
-}
-
#[test]
#[ignore]
fn shared_web_action_matrix_is_state_verified() {
@@ -1048,6 +1187,10 @@ fn shared_web_action_matrix_is_state_verified() {
"type_text",
run_text_action as fn(&mut Fixture, &str, &str) -> Observation,
),
+ (
+ "type_submit",
+ run_type_submit_action as fn(&mut Fixture, &str, &str) -> Observation,
+ ),
(
"press_key",
run_press_key_action as fn(&mut Fixture, &str, &str) -> Observation,
@@ -1108,8 +1251,8 @@ fn shared_web_action_matrix_is_state_verified() {
}
}
assert!(
- !cell_filter_active() || selected > 0,
- "CUA_E2E_CELL_FILTER matched no shared E2E cells"
+ selected > 0,
+ "no shared E2E cells were selected; check CUA_E2E_HARNESS_FILTER and CUA_E2E_CELL_FILTER"
);
resume_first_failure(failure);
}
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/e2e_environment_preflight_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/e2e_environment_preflight_test.rs
index 4666962983..05274e8e0f 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/e2e_environment_preflight_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/e2e_environment_preflight_test.rs
@@ -5,12 +5,14 @@
use std::any::Any;
use std::collections::HashSet;
use std::panic::{self, AssertUnwindSafe};
-use std::process::{Command, Stdio};
+use std::process::{Child, Command, Stdio};
use std::time::{Duration, Instant};
#[cfg(target_os = "linux")]
use cua_driver_testkit::e2e::DisplayServer;
use cua_driver_testkit::e2e::{write_environment_from_env, EnvironmentRecord};
+use cua_driver_testkit::observer::TargetWindow;
+use cua_driver_testkit::sentinel::ForegroundSentinel;
use cua_driver_testkit::{driver_binary, harness_app, spawn_in_job, Driver, McpDriver};
struct PreflightFixture {
@@ -20,6 +22,33 @@ struct PreflightFixture {
ax_marker: &'static str,
}
+struct FixtureChildGuard {
+ child: Option<Child>,
+}
+
+impl FixtureChildGuard {
+ fn new(child: Child) -> Self {
+ Self { child: Some(child) }
+ }
+
+ fn child_mut(&mut self) -> &mut Child {
+ self.child.as_mut().expect("fixture child guard is armed")
+ }
+
+ fn into_child(mut self) -> Child {
+ self.child.take().expect("fixture child guard is armed")
+ }
+}
+
+impl Drop for FixtureChildGuard {
+ fn drop(&mut self) {
+ if let Some(child) = &mut self.child {
+ let _ = child.kill();
+ let _ = child.wait();
+ }
+ }
+}
+
fn preflight_fixture() -> PreflightFixture {
#[cfg(target_os = "windows")]
{
@@ -98,6 +127,29 @@ fn has_image(response: &cua_driver_testkit::ToolResponse) -> bool {
.unwrap_or(false)
}
+fn readiness_contract(
+ is_error: bool,
+ element_count: usize,
+ marker_present: bool,
+ image_present: bool,
+) -> bool {
+ !is_error && element_count > 0 && marker_present && image_present
+}
+
+fn preflight_state_ready(response: &cua_driver_testkit::ToolResponse, ax_marker: &str) -> bool {
+ let element_count = response.structured()["element_count"]
+ .as_u64()
+ .map(|count| count as usize)
+ .or_else(|| response.structured()["elements"].as_array().map(Vec::len))
+ .unwrap_or_default();
+ readiness_contract(
+ response.is_error(),
+ element_count,
+ response.tree_text().contains(ax_marker),
+ has_image(response),
+ )
+}
+
fn run_preflight() {
if let Ok(expected_sha) = std::env::var("CUA_E2E_SOURCE_SHA") {
assert!(
@@ -192,84 +244,92 @@ fn run_preflight() {
.args(&fixture.args)
.stdout(Stdio::null())
.stderr(Stdio::inherit());
- let mut child = spawn_in_job(&mut command).expect("preflight fixture failed to launch");
+ let child = spawn_in_job(&mut command).expect("preflight fixture failed to launch");
let launched_pid = child.id() as i64;
+ let mut child = FixtureChildGuard::new(child);
- let deadline = Instant::now() + Duration::from_secs(20);
+ let deadline = Instant::now() + Duration::from_secs(35);
+ let mut last_readiness = "fixture window has not appeared".to_owned();
+ #[cfg(target_os = "linux")]
+ let mut activated_window_id = None;
let (pid, window_id) = loop {
if let Some(status) = child
+ .child_mut()
.try_wait()
.expect("could not inspect preflight fixture process")
{
panic!("preflight fixture exited before mapping a window: {status}");
}
- let response = driver.call("list_windows", serde_json::json!({}));
- if let Some(window) = response.structured()["windows"]
+ let windows = driver.call("list_windows", serde_json::json!({}));
+ let candidate = windows.structured()["windows"]
.as_array()
.and_then(|windows| {
- windows.iter().find(|window| {
- window["window_id"]
- .as_u64()
- .map(|id| !before_ids.contains(&id))
- .unwrap_or(false)
- && window["title"]
- .as_str()
- .unwrap_or("")
- .contains(fixture.title)
+ windows.iter().find_map(|window| {
+ let window_id = window["window_id"].as_u64()?;
+ let is_new = !before_ids.contains(&window_id);
+ let title_matches = window["title"]
+ .as_str()
+ .unwrap_or("")
+ .contains(fixture.title);
+ (is_new && title_matches)
+ .then(|| (window["pid"].as_i64().unwrap_or(launched_pid), window_id))
})
- })
- {
- if let Some(window_id) = window["window_id"].as_u64() {
- break (window["pid"].as_i64().unwrap_or(launched_pid), window_id);
- }
+ });
+ if windows.is_error() {
+ last_readiness = format!("list_windows failed: {}", windows.text());
}
- if Instant::now() >= deadline {
- let _ = child.kill();
- let _ = child.wait();
- panic!("preflight fixture window did not appear");
- }
- std::thread::sleep(Duration::from_millis(200));
- };
- driver.reaper().push(child);
- #[cfg(target_os = "linux")]
- {
- if DisplayServer::current() == DisplayServer::X11 {
- let activated = driver.call(
- "bring_to_front",
- serde_json::json!({ "pid": pid, "window_id": window_id }),
- );
- assert!(
- !activated.is_error(),
- "preflight fixture could not be placed on the Linux desktop: {}",
- activated.text()
+ if let Some((pid, window_id)) = candidate {
+ #[cfg(target_os = "linux")]
+ if DisplayServer::current() == DisplayServer::X11
+ && activated_window_id != Some(window_id)
+ {
+ let activated = driver.call(
+ "bring_to_front",
+ serde_json::json!({ "pid": pid, "window_id": window_id }),
+ );
+ assert!(
+ !activated.is_error(),
+ "preflight fixture could not be placed on the Linux desktop: {}",
+ activated.text()
+ );
+ activated_window_id = Some(window_id);
+ std::thread::sleep(Duration::from_millis(300));
+ }
+
+ let state = driver.call(
+ "get_window_state",
+ serde_json::json!({
+ "pid": pid,
+ "window_id": window_id,
+ "capture_mode": "ax"
+ }),
);
- std::thread::sleep(Duration::from_millis(300));
+ if preflight_state_ready(&state, fixture.ax_marker) {
+ break (pid, window_id);
+ }
+ last_readiness = state.text().to_owned();
}
- }
- let deadline = Instant::now() + Duration::from_secs(15);
- loop {
- let state = driver.call(
- "get_window_state",
- serde_json::json!({
- "pid": pid,
- "window_id": window_id,
- "capture_mode": "ax"
- }),
- );
- if !state.is_error() && state.tree_text().contains(fixture.ax_marker) && has_image(&state) {
- break;
+ if Instant::now() >= deadline {
+ panic!(
+ "preflight could not map a ready fixture window with AX state and screenshot: {last_readiness}"
+ );
}
- assert!(
- Instant::now() < deadline,
- "preflight could not read both AX state and screenshot: {}",
- state.text()
- );
std::thread::sleep(Duration::from_millis(200));
- }
+ };
+ driver.reaper().push(child.into_child());
driver.start_behavior_recording();
+ let target = TargetWindow {
+ pid: pid as u32,
+ native_id: window_id,
+ };
+ let sentinel = ForegroundSentinel::launch(&mut driver);
+ sentinel
+ .assert_guard_canaries(&mut driver, target)
+ .expect("foreground sentinel guard canaries failed");
+ drop(sentinel);
drop(driver);
let video = recording_dir.join("recording.mp4");
@@ -338,6 +398,15 @@ fn panic_message(payload: &Box<dyn Any + Send>) -> String {
.unwrap_or_else(|| "preflight panicked without a string payload".to_owned())
}
+#[test]
+fn readiness_requires_nonempty_ax_marker_and_screenshot_together() {
+ assert!(readiness_contract(false, 1, true, true));
+ assert!(!readiness_contract(false, 0, true, true));
+ assert!(!readiness_contract(false, 1, false, true));
+ assert!(!readiness_contract(false, 1, true, false));
+ assert!(!readiness_contract(true, 1, true, true));
+}
+
#[test]
#[ignore]
fn canonical_e2e_environment_is_ready() {
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs
index 0c465a8004..aeb7a288f0 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs
@@ -136,9 +136,10 @@ fn post_key(pid: i32, key_code: u16, key_down: bool, flags: CGEventFlags) -> any
.map_err(|_| anyhow::anyhow!("CGEventSource::new failed"))?;
let event = CGEvent::new_keyboard_event(source, key_code, key_down)
.map_err(|_| anyhow::anyhow!("CGEvent::new_keyboard_event failed"))?;
- if flags != CGEventFlags::CGEventFlagNull {
- event.set_flags(flags);
- }
+ // HIDSystemState can inherit physically held modifiers. Always overwrite
+ // the event flags, including the empty case, so an unrelated Shift/Caps
+ // state cannot leak into a targeted key press.
+ event.set_flags(flags);
post_keyboard_event(pid, &event);
Ok(())
}
@@ -148,9 +149,7 @@ fn post_key_no_auth(pid: i32, key_code: u16, key_down: bool, flags: CGEventFlags
.map_err(|_| anyhow::anyhow!("CGEventSource::new failed"))?;
let event = CGEvent::new_keyboard_event(source, key_code, key_down)
.map_err(|_| anyhow::anyhow!("CGEvent::new_keyboard_event failed"))?;
- if flags != CGEventFlags::CGEventFlagNull {
- event.set_flags(flags);
- }
+ event.set_flags(flags);
let event_ptr = event.as_ptr() as *mut std::ffi::c_void;
// attach_auth_message = false → IOHIDPostEvent path → NSMenu fires
if !crate::input::skylight::post_to_pid(pid as libc::pid_t, event_ptr, false) {
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
index ad0b54dda6..12a07efd7d 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
@@ -58,12 +58,19 @@ fn resolve_onscreen_point_with_scroll(
cy: i32,
action_verb: &str,
) -> Result<(i32, i32), String> {
- if crate::input::point_in_window_bounds(hwnd, cx, cy) {
- return Ok((cx, cy));
- }
- // Off-screen: ask the control to scroll itself into view, then re-resolve.
+ let cached_center_in_window = crate::input::point_in_window_bounds(hwnd, cx, cy);
+ // A center can still be clipped by a ScrollViewer or a docked sibling while
+ // remaining inside the outer HWND rectangle. UIA's IsOffscreen property is
+ // authoritative for that case; without it a foreground tap can land on the
+ // visible control covering the stale point (for example a bottom toolbar).
if let Some(retained) = element_cache.get_element_retained(pid, hwnd, idx) {
if retained.is_uia() {
+ let is_offscreen = unsafe {
+ crate::uia::scroll::element_is_offscreen(retained.as_ptr())
+ };
+ if cached_center_in_window && is_offscreen != Some(true) {
+ return Ok((cx, cy));
+ }
if let Some((nx, ny)) = unsafe {
crate::uia::scroll::scroll_into_view_and_recenter(hwnd, retained.as_ptr())
} {
@@ -71,14 +78,18 @@ fn resolve_onscreen_point_with_scroll(
return Ok((nx, ny));
}
}
+ } else if cached_center_in_window {
+ return Ok((cx, cy));
}
// `retained` drops here → COM Release.
+ } else if cached_center_in_window {
+ return Ok((cx, cy));
}
Err(format!(
- "Element [{idx}] resolves to ({cx},{cy}), outside its window — tried UIA \
- ScrollIntoView but it is still off-screen, so {action_verb} would land on \
- whatever is there (e.g. the taskbar). Make the window taller or scroll \
- the region into view, then retry."
+ "Element [{idx}] resolves to ({cx},{cy}) but is not visibly actionable — \
+ tried UIA ScrollIntoView but it is still off-screen, so {action_verb} \
+ would land on whatever covers that point. Make the window taller or \
+ scroll the region into view, then retry."
))
}
@@ -3742,7 +3753,16 @@ impl Tool for TypeTextTool {
// top-level can't be foregrounded) while the a11y-channel
// SetValue still lands. Same fix as the UIA Invoke path.
crate::uia::fg_bypass::run_with_uwp_bypass(hwnd as isize, || unsafe {
- vp.SetValue(&BSTR::from(text_for_uia.as_str()))
+ // ValuePattern has no caret/selection insertion API. Preserve
+ // the current document and append as the deterministic
+ // background fallback; replacing the whole value violates
+ // type_text's insertion contract for editors.
+ let current = vp
+ .CurrentValue()
+ .map(|value| value.to_string())
+ .unwrap_or_default();
+ let inserted = format!("{current}{text_for_uia}");
+ vp.SetValue(&BSTR::from(inserted.as_str()))
})?;
Ok(())
})()
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/uia/scroll.rs b/libs/cua-driver/rust/crates/platform-windows/src/uia/scroll.rs
index a084bd590b..2e0cd6331e 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/uia/scroll.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/uia/scroll.rs
@@ -47,6 +47,21 @@ use windows::Win32::UI::Accessibility::{
UIA_ScrollItemPatternId, UIA_ScrollPatternId,
};
+/// Return UIA's live IsOffscreen state for a retained element pointer.
+///
+/// `None` means the property could not be read. Callers fail open in that case
+/// so a transient UIA query never blocks an otherwise valid coordinate action.
+pub unsafe fn element_is_offscreen(element_ptr: usize) -> Option<bool> {
+ if element_ptr == 0 {
+ return None;
+ }
+ let _ = CoInitializeEx(None, COINIT_MULTITHREADED);
+ let elem: IUIAutomationElement = IUIAutomationElement::from_raw(element_ptr as *mut _);
+ let result = elem.CurrentIsOffscreen().ok().map(|value| value.as_bool());
+ std::mem::forget(elem);
+ result
+}
+
/// Ask the UIA element behind `element_ptr` to scroll itself into view, then
/// return its fresh on-screen center (from the element's *live* bounding rect,
/// since the cached center is stale once the control has moved).
@@ -84,8 +99,12 @@ pub unsafe fn scroll_into_view_and_recenter(
// - otherwise the coordinate tap would still miss. This mirrors the caller's
// own bounds check so an ineffective ScrollIntoView falls through to the
// ancestor ScrollPattern fallback instead of returning a bogus center.
- let in_bounds = |c: (i32, i32)| -> Option<(i32, i32)> {
- if crate::input::point_in_window_bounds(host_hwnd, c.0, c.1) {
+ let actionable = |c: (i32, i32)| -> Option<(i32, i32)> {
+ let visible = elem
+ .CurrentIsOffscreen()
+ .map(|value| !value.as_bool())
+ .unwrap_or(true);
+ if visible && crate::input::point_in_window_bounds(host_hwnd, c.0, c.1) {
Some(c)
} else {
None
@@ -93,8 +112,8 @@ pub unsafe fn scroll_into_view_and_recenter(
};
let result = scroll_item_into_view(host_hwnd, &elem)
- .and_then(in_bounds)
- .or_else(|| scroll_ancestor_into_view(host_hwnd, &elem).and_then(in_bounds));
+ .and_then(actionable)
+ .or_else(|| scroll_ancestors_into_view(host_hwnd, &elem).and_then(actionable));
// Don't Release the cache's ref - the RetainedElement guard owns it.
std::mem::forget(elem);
@@ -155,9 +174,10 @@ unsafe fn scroll_item_into_view(host_hwnd: u64, elem: &IUIAutomationElement) ->
Some(((rect.left + rect.right) / 2, (rect.top + rect.bottom) / 2))
}
-/// Strategy 2: walk up to the first vertically-scrollable ancestor and
-/// `ScrollPattern::Scroll` until the target's live rect re-enters the viewport.
-unsafe fn scroll_ancestor_into_view(
+/// Strategy 2: walk every vertically-scrollable ancestor and use
+/// `ScrollPattern::Scroll` until the target's live rect re-enters a visible
+/// viewport.
+unsafe fn scroll_ancestors_into_view(
host_hwnd: u64,
elem: &IUIAutomationElement,
) -> Option<(i32, i32)> {
@@ -167,9 +187,11 @@ unsafe fn scroll_ancestor_into_view(
CoCreateInstance(&CUIAutomation, None, CLSCTX_INPROC_SERVER).ok()?;
let walker = automation.ControlViewWalker().ok()?;
- // Find the nearest ancestor that exposes a vertically-scrollable ScrollPattern.
- let mut scroll_pat: Option<IUIAutomationScrollPattern> = None;
- let mut container: Option<IUIAutomationElement> = None;
+ // Collect every vertically-scrollable ancestor. Nested controls are common:
+ // a ListItem may already be visible inside its ListBox while that entire
+ // ListBox is clipped by an outer page ScrollViewer. Trying only the nearest
+ // pattern leaves the element hidden even though another ancestor can reveal it.
+ let mut containers: Vec<(IUIAutomationElement, IUIAutomationScrollPattern)> = Vec::new();
let mut cur = walker.GetParentElement(elem).ok()?;
for _ in 0..16 {
if let Ok(p) = cur.GetCurrentPattern(UIA_ScrollPatternId) {
@@ -179,9 +201,7 @@ unsafe fn scroll_ancestor_into_view(
.map(|b| b.as_bool())
.unwrap_or(false);
if scrollable {
- container = Some(cur.clone());
- scroll_pat = Some(sp);
- break;
+ containers.push((cur.clone(), sp));
}
}
}
@@ -190,36 +210,45 @@ unsafe fn scroll_ancestor_into_view(
Err(_) => break,
}
}
- let sp = scroll_pat?;
- let container = container?;
- let crect = container.CurrentBoundingRectangle().ok()?;
- // Iteratively scroll the container until the target's center sits inside the
- // viewport. Large steps to close distance, small steps once we overshoot
- // (direction flip) so we don't oscillate. All wrapped in the fg bypass.
- let mut last_dir = 0i32;
- for _ in 0..60 {
- let tr = elem.CurrentBoundingRectangle().ok()?;
- let tcy = (tr.top + tr.bottom) / 2;
- if tcy >= crect.top && tcy <= crect.bottom {
- break; // center is within the viewport
+ for (container, sp) in containers {
+ let crect = container.CurrentBoundingRectangle().ok()?;
+ // Iteratively scroll this container until the target's center sits inside
+ // its viewport. If UIA still marks the target off-screen, continue with
+ // the next outer scrollable ancestor.
+ let mut last_dir = 0i32;
+ for _ in 0..60 {
+ let tr = elem.CurrentBoundingRectangle().ok()?;
+ let tcy = (tr.top + tr.bottom) / 2;
+ if tcy >= crect.top && tcy <= crect.bottom {
+ break;
+ }
+ let dir = if tcy < crect.top { -1 } else { 1 };
+ let overshot = last_dir != 0 && dir != last_dir;
+ let amount = match (dir, overshot) {
+ (-1, false) => ScrollAmount_LargeDecrement,
+ (-1, true) => ScrollAmount_SmallDecrement,
+ (_, false) => ScrollAmount_LargeIncrement,
+ (_, true) => ScrollAmount_SmallIncrement,
+ };
+ let scrolled = crate::uia::fg_bypass::run_with_uwp_bypass(host_hwnd as isize, || {
+ sp.Scroll(ScrollAmount_NoAmount, amount)
+ });
+ if scrolled.is_err() {
+ break;
+ }
+ last_dir = dir;
+ sleep(Duration::from_millis(15));
}
- let dir = if tcy < crect.top { -1 } else { 1 };
- let overshot = last_dir != 0 && dir != last_dir;
- let amount = match (dir, overshot) {
- (-1, false) => ScrollAmount_LargeDecrement,
- (-1, true) => ScrollAmount_SmallDecrement,
- (_, false) => ScrollAmount_LargeIncrement,
- (_, true) => ScrollAmount_SmallIncrement,
- };
- let scrolled = crate::uia::fg_bypass::run_with_uwp_bypass(host_hwnd as isize, || {
- sp.Scroll(ScrollAmount_NoAmount, amount)
- });
- if scrolled.is_err() {
- break;
+ let rect = elem.CurrentBoundingRectangle().ok()?;
+ let center = ((rect.left + rect.right) / 2, (rect.top + rect.bottom) / 2);
+ let visible = elem
+ .CurrentIsOffscreen()
+ .map(|value| !value.as_bool())
+ .unwrap_or(true);
+ if visible && crate::input::point_in_window_bounds(host_hwnd, center.0, center.1) {
+ return Some(center);
}
- last_dir = dir;
- sleep(Duration::from_millis(15)); // let the layout settle before re-reading
}
let rect = elem.CurrentBoundingRectangle().ok()?;
diff --git a/libs/cua-driver/tests/fixtures/apps/cross-platform/electron/main.js b/libs/cua-driver/tests/fixtures/apps/cross-platform/electron/main.js
index 4b00250f8e..8f027d1117 100644
--- a/libs/cua-driver/tests/fixtures/apps/cross-platform/electron/main.js
+++ b/libs/cua-driver/tests/fixtures/apps/cross-platform/electron/main.js
@@ -61,13 +61,17 @@ function createWindow() {
const fixedTitle = sentinelMode
? `CuaTestHarness Sentinel [cdp=${CDP_PORT}]`
: `CuaTestHarness Electron [cdp=${CDP_PORT}]`;
+ // GitHub's interactive Windows desktop can be 1024x768. Keep the normal
+ // fixture wholly inside its work area so a maximized sentinel can prove
+ // geometric full occlusion instead of accepting an off-screen target.
+ const compactWindowsFixture = !sentinelMode && process.platform === 'win32';
mainWindow = new BrowserWindow({
- width: sentinelMode ? 1280 : 940,
- height: sentinelMode ? 900 : 780,
+ width: sentinelMode ? 1280 : compactWindowsFixture ? 900 : 940,
+ height: sentinelMode ? 900 : compactWindowsFixture ? 640 : 780,
// Keep the normal fixture inside virtual desktops whose window manager
// has no persisted placement policy (notably Openbox under Xvfb).
- x: sentinelMode ? 0 : 120,
- y: sentinelMode ? 0 : 120,
+ x: sentinelMode ? 0 : compactWindowsFixture ? 40 : 120,
+ y: sentinelMode ? 0 : compactWindowsFixture ? 40 : 120,
title: fixedTitle,
// Map the normal harness immediately. Xvfb/Openbox can enumerate a
// deferred BrowserWindow while never painting it into the root desktop.
@@ -120,7 +124,11 @@ function createWindow() {
if (process.platform !== 'darwin') {
mainWindow.setAlwaysOnTop(true);
}
- mainWindow.maximize();
+ if (process.platform === 'linux' && process.env.WAYLAND_DISPLAY) {
+ mainWindow.setFullScreen(true);
+ } else {
+ mainWindow.maximize();
+ }
mainWindow.show();
mainWindow.focus();
} else {
diff --git a/libs/cua-driver/tests/fixtures/apps/cross-platform/electron/preload.js b/libs/cua-driver/tests/fixtures/apps/cross-platform/electron/preload.js
index c111490cdd..32a3a5f534 100644
--- a/libs/cua-driver/tests/fixtures/apps/cross-platform/electron/preload.js
+++ b/libs/cua-driver/tests/fixtures/apps/cross-platform/electron/preload.js
@@ -33,16 +33,29 @@ if (sentinelMode) {
});
window.addEventListener('focus', () => record('focus'));
window.addEventListener('blur', () => record('blur'));
+ window.addEventListener('visibilitychange', () =>
+ record('visibility', { state: document.visibilityState })
+ );
window.addEventListener('keydown', event =>
record('keydown', { key: event.key, code: event.code })
);
+ window.addEventListener('keyup', event =>
+ record('keyup', { key: event.key, code: event.code })
+ );
window.addEventListener('pointerdown', event =>
record('pointerdown', { button: event.button, x: event.clientX, y: event.clientY })
);
+ window.addEventListener('pointerup', event =>
+ record('pointerup', { button: event.button, x: event.clientX, y: event.clientY })
+ );
+ window.addEventListener('click', event =>
+ record('click', { button: event.button, x: event.clientX, y: event.clientY })
+ );
window.addEventListener('wheel', event =>
record('wheel', { delta_x: event.deltaX, delta_y: event.deltaY })
);
window.addEventListener('contextmenu', event =>
record('contextmenu', { x: event.clientX, y: event.clientY })
);
+ setInterval(() => record('heartbeat'), 100);
}
diff --git a/libs/cua-driver/tests/fixtures/apps/cross-platform/tauri/src-tauri/src/main.rs b/libs/cua-driver/tests/fixtures/apps/cross-platform/tauri/src-tauri/src/main.rs
index f689a4d095..f974558e81 100644
--- a/libs/cua-driver/tests/fixtures/apps/cross-platform/tauri/src-tauri/src/main.rs
+++ b/libs/cua-driver/tests/fixtures/apps/cross-platform/tauri/src-tauri/src/main.rs
@@ -9,6 +9,17 @@ fn main() {
.build();
tauri::Builder::default()
.plugin(journal_plugin)
+ .setup(|_app| {
+ #[cfg(target_os = "windows")]
+ {
+ use tauri::{LogicalSize, Manager, Size};
+
+ _app.get_webview_window("main")
+ .expect("Tauri main fixture window")
+ .set_size(Size::Logical(LogicalSize::new(900.0, 640.0)))?;
+ }
+ Ok(())
+ })
.run(tauri::generate_context!())
.expect("error while running CuaTestHarness.Tauri");
}
diff --git a/libs/cua-driver/tests/fixtures/apps/windows/webview2/MainWindow.xaml b/libs/cua-driver/tests/fixtures/apps/windows/webview2/MainWindow.xaml
index 12b4acb0c2..6c1dbb0893 100644
--- a/libs/cua-driver/tests/fixtures/apps/windows/webview2/MainWindow.xaml
+++ b/libs/cua-driver/tests/fixtures/apps/windows/webview2/MainWindow.xaml
@@ -4,7 +4,7 @@
xmlns:wv2="clr-namespace:Microsoft.Web.WebView2.Wpf;assembly=Microsoft.Web.WebView2.Wpf"
Title="Loading WebView2 fixture"
AutomationProperties.AutomationId="wnd-main"
- Width="940" Height="780"
+ Width="900" Height="650"
WindowStartupLocation="CenterScreen">
<DockPanel LastChildFill="True">
<Border DockPanel.Dock="Top" Background="#F0F0F0" Padding="10,6">
diff --git a/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml b/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml
index c500005cac..3f47176450 100644
--- a/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml
+++ b/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml
@@ -5,7 +5,7 @@
xmlns:auto="clr-namespace:System.Windows.Automation;assembly=PresentationCore"
Title="CuaTestHarness WPF"
AutomationProperties.AutomationId="wnd-main"
- Width="900" Height="900"
+ Width="900" Height="650"
WindowStartupLocation="CenterScreen">
<Window.InputBindings>
<!-- F5 (no modifiers) — chosen for automated testing. WPF's
diff --git a/libs/cua-driver/tests/fixtures/shared/web/index.html b/libs/cua-driver/tests/fixtures/shared/web/index.html
index f44e725425..4b58b8ad4d 100644
--- a/libs/cua-driver/tests/fixtures/shared/web/index.html
+++ b/libs/cua-driver/tests/fixtures/shared/web/index.html
@@ -69,7 +69,8 @@
<legend>text_input</legend>
<div class="row">
<input type="text" id="txt-input" data-cua-id="txt-input"
- aria-label="txt-input" placeholder="type here" />
+ aria-label="txt-input" placeholder="type here"
+ autocapitalize="off" autocorrect="off" spellcheck="false" />
<span class="mirror" id="lbl-input-mirror" data-cua-id="lbl-input-mirror">mirror=</span>
</div>
</fieldset>
@@ -78,7 +79,8 @@
<legend>keyboard_task</legend>
<div class="row">
<input type="text" id="keyboard-input" data-cua-id="keyboard-input" autofocus
- aria-label="keyboard-input" placeholder="keyboard input" />
+ aria-label="keyboard-input" placeholder="keyboard input"
+ autocapitalize="off" autocorrect="off" spellcheck="false" />
<span class="mirror" id="keyboard-status" data-cua-id="keyboard-status">key_state=none</span>
</div>
</fieldset>
@@ -86,7 +88,9 @@
<fieldset>
<legend>text_editor_task</legend>
<div class="row">
- <textarea id="editor-document" data-cua-id="editor-document" aria-label="editor-document" rows="3" cols="42">Initial conformance text.\nStatus: DRAFT\n</textarea>
+ <textarea id="editor-document" data-cua-id="editor-document" aria-label="editor-document"
+ autocapitalize="off" autocorrect="off" spellcheck="false"
+ rows="3" cols="42">Initial conformance text.\nStatus: DRAFT\n</textarea>
<button id="editor-save" data-cua-id="editor-save" aria-label="editor-save Save">Save</button>
<span class="mirror" id="editor-status" data-cua-id="editor-status">editor_status=draft</span>
</div>
@@ -121,6 +125,7 @@
<div id="scroll-bottom" data-cua-id="scroll-bottom">SCROLL_BOTTOM_MARKER_v1</div>
</div>
<span class="mirror" id="lbl-scroll-offset" data-cua-id="lbl-scroll-offset">scroll_offset=0</span>
+ <span class="mirror" id="lbl-scroll-client-height" data-cua-id="lbl-scroll-client-height">scroll_client_height=0</span>
</div>
</fieldset>
@@ -207,7 +212,7 @@
$('editor-save').addEventListener('click', () => {
const value = $('editor-document').value;
localStorage.setItem('cua-editor-document', value);
- $('editor-status').textContent = `editor_status=saved:${value.endsWith('\\n') ? 'newline' : 'exact'}`;
+ $('editor-status').textContent = `editor_status=saved:${value}`;
});
$('txt-input').addEventListener('input', e => {
@@ -218,7 +223,10 @@
if (e.ctrlKey && e.shiftKey && e.code === 'Digit7') {
$('keyboard-status').textContent = 'key_state=hotkey';
} else if (e.key === 'Enter') {
- $('keyboard-status').textContent = 'key_state=enter';
+ const value = $('keyboard-input').value;
+ $('keyboard-status').textContent = value
+ ? `key_state=submitted:${value}`
+ : 'key_state=enter';
}
});
@@ -234,6 +242,7 @@
d.textContent = 'line ' + String(i).padStart(2, '0');
lines.appendChild(d);
}
+ $('lbl-scroll-client-height').textContent = `scroll_client_height=${$('scroll-tall').clientHeight}`;
$('scroll-tall').addEventListener('scroll', e => {
$('lbl-scroll-offset').textContent = `scroll_offset=${Math.round(e.target.scrollTop)}`;
});
diff --git a/scripts/ci/linux/run-rust-e2e.sh b/scripts/ci/linux/run-rust-e2e.sh
index e5e5e55975..a01f36bbe0 100755
--- a/scripts/ci/linux/run-rust-e2e.sh
+++ b/scripts/ci/linux/run-rust-e2e.sh
@@ -56,6 +56,13 @@ export CUA_TEST_DRIVER_BIN="${RUST_ROOT}/target/release/cua-driver"
export CUA_TEST_APPS_ROOT="${RUST_ROOT}/test-apps"
export CUA_TEST_REQUIRE_FIXTURES=1
export CUA_TEST_DRIVER_STDERR=1
+export CUA_E2E_FORBID_SKIPS=1
+unset CUA_E2E_EXPECTED_MIN_CELLS
+if [[ ("${SUITE}" == shared || "${SUITE}" == all) \
+ && -z "${CUA_E2E_CELL_FILTER:-}" \
+ && -z "${CUA_E2E_HARNESS_FILTER:-}" ]]; then
+ export CUA_E2E_EXPECTED_MIN_CELLS=80
+fi
export RUST_BACKTRACE="${RUST_BACKTRACE:-1}"
if [[ -z "${CUA_E2E_SOURCE_SHA:-}" && -f "${REPO_ROOT}/.cua-e2e-source-sha" ]]; then
export CUA_E2E_SOURCE_SHA="$(tr -d '[:space:]' < "${REPO_ROOT}/.cua-e2e-source-sha")"
diff --git a/scripts/ci/macos/run-rust-e2e.sh b/scripts/ci/macos/run-rust-e2e.sh
index ddc647fbcc..b735cb6d55 100755
--- a/scripts/ci/macos/run-rust-e2e.sh
+++ b/scripts/ci/macos/run-rust-e2e.sh
@@ -74,6 +74,13 @@ export CUA_TEST_DRIVER_BIN="${RUST_ROOT}/target/release/cua-driver"
export CUA_TEST_APPS_ROOT="${RUST_ROOT}/test-apps"
export CUA_TEST_REQUIRE_FIXTURES=1
export CUA_TEST_DRIVER_STDERR=1
+export CUA_E2E_FORBID_SKIPS=1
+unset CUA_E2E_EXPECTED_MIN_CELLS
+if [[ ("${SUITE}" == shared || "${SUITE}" == all) \
+ && -z "${CUA_E2E_CELL_FILTER:-}" \
+ && -z "${CUA_E2E_HARNESS_FILTER:-}" ]]; then
+ export CUA_E2E_EXPECTED_MIN_CELLS=120
+fi
command -v ffmpeg >/dev/null || { echo "ffmpeg is required for E2E trajectory videos" >&2; exit 1; }
command -v ffprobe >/dev/null || { echo "ffprobe is required for E2E trajectory validation" >&2; exit 1; }
diff --git a/scripts/ci/windows/run-rust-e2e.ps1 b/scripts/ci/windows/run-rust-e2e.ps1
index e67533d3c8..1f7d4278d9 100644
--- a/scripts/ci/windows/run-rust-e2e.ps1
+++ b/scripts/ci/windows/run-rust-e2e.ps1
@@ -49,6 +49,13 @@ $env:CUA_TEST_DRIVER_BIN = Join-Path $rustRoot "target\release\cua-driver.exe"
$env:CUA_TEST_APPS_ROOT = Join-Path $rustRoot "test-apps"
$env:CUA_TEST_REQUIRE_FIXTURES = "1"
$env:CUA_TEST_DRIVER_STDERR = "1"
+$env:CUA_E2E_FORBID_SKIPS = "1"
+Remove-Item Env:CUA_E2E_EXPECTED_MIN_CELLS -ErrorAction SilentlyContinue
+if (($suite -in @("shared", "all")) -and
+ [string]::IsNullOrWhiteSpace($env:CUA_E2E_CELL_FILTER) -and
+ [string]::IsNullOrWhiteSpace($env:CUA_E2E_HARNESS_FILTER)) {
+ $env:CUA_E2E_EXPECTED_MIN_CELLS = "80"
+}
if (-not $NoBuild) {
& cargo build --release -p cua-driver --manifest-path (Join-Path $rustRoot "Cargo.toml")