Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
test(cua-driver): harden background E2E oracles (#2193)
Francesco Bonaccie2596a4e554b
diff --git a/libs/cua-driver/docs/e2e-ci-reporting.md b/libs/cua-driver/docs/e2e-ci-reporting.mdindex dc395962db..f84d12c9fa 100644--- a/libs/cua-driver/docs/e2e-ci-reporting.md+++ b/libs/cua-driver/docs/e2e-ci-reporting.md@@ -67,8 +67,12 @@ artifacts/cua-driver/<platform>/`cases.jsonl` is the executed catalog. `results.jsonl` contains one result forevery declared cell. In canonical mode, the reporter rejects duplicates,-missing results, undeclared results, contradictory statuses, missing required-videos, and missing or invalid expected turn evidence. An `evidence.json`+missing results, undeclared results, contradictory statuses, forbidden skips,+an under-sized unfiltered shared catalog, missing required videos, and missing+or invalid expected turn evidence. Canonical runners set+`CUA_E2E_FORBID_SKIPS=1` and `CUA_E2E_EXPECTED_MIN_CELLS` (80 on+Windows/Linux, 120 on macOS). Explicit diagnostic filters omit only the+minimum-count policy; a filter matching zero cells still fails in Rust. An `evidence.json`classification normally makes capture failure auditable without making itpass. The one narrow exception is a successful Windows `bring_to_front`restoration:diff --git a/libs/cua-driver/docs/test-harnesses-guide.md b/libs/cua-driver/docs/test-harnesses-guide.mdindex 32f64e7d39..c75a8b742b 100644--- a/libs/cua-driver/docs/test-harnesses-guide.md+++ b/libs/cua-driver/docs/test-harnesses-guide.md@@ -212,6 +212,10 @@ target activation. A portal/libei grant persists until the user revokes it, sosubsequent driver processes do not reopen the consent dialog.KDE requires a future target-addressable KWin adapter; portal availability byitself is not evidence that input can be sent safely to a named window.+Standard Wayland does not expose the physical pointer position, so canonical+Wayland rows do not claim the real-cursor preservation oracle. Focus, full+occlusion, sentinel input isolation, liveness, and fixture-state oracles remain+mandatory.## AX, PX, and Delivery@@ -252,8 +256,10 @@ question for any action, harness, or catalog area:`cua-driver-testkit::DesktopObserver` owns the shared interface. Native Windows,macOS, and Linux backends snapshot foreground-window, target z-order, cursor,-and leaked-input state before and after an action. Background rows opt into the-observer directly; there is no special guard suite.+and focus state before and after an action. A separate full-desktop Electron+sentinel journals keyboard, pointer, wheel, visibility, focus, and heartbeat+events while it fully covers the target. Background rows opt into both pieces+of instrumentation directly; there is no special guard suite.| Invariant or scenario | What it checks || --- | --- |@@ -264,6 +270,18 @@ observer directly; there is no special guard suite.| Background screenshot | Reading the target does not change focus or z-order || Agent cursor visibility | The cursor appears in the captured pixels when enabled and moved |+The sentinel contract fails closed when the target is only partly covered or+the heartbeat stops. Before any behavioral cells run, the strict environment+preflight deliberately sends input to the sentinel and deliberately raises the+background target. The lane proceeds only if the leaked input and transient+focus loss are observed, the sentinel is restored, and it once again fully+occludes the target. Windows, macOS, and X11 require the sentinel's live focus+journal to report the loss. Wayland uses the compositor-backed native focus+observer because Electron/Ozone does not reliably emit a DOM `blur` event for+an external surface focus transition. The sentinel heartbeat and leaked-input+journal remain mandatory on Wayland. This positive control prevents a broken+guard from making every background row look green.+A focus assertion can prove "no focus steal" while failing to prove that aclick changed the target application state. An action row must therefore checkboth the target's external state and, when background delivery is under test,@@ -325,7 +343,13 @@ need normal test output and logs; they do not need desktop video.established. The sentinel remains a separate test fixture and is reassertedafter console cleanup.- Strict lane preflights fail on missing fixtures, desktop access, permissions,- accessibility, capture, or recording support instead of silently skipping.+ accessibility, capture, recording support, or ineffective background guards+ instead of silently skipping.+- Canonical runners set `CUA_E2E_FORBID_SKIPS=1`. Unfiltered shared runs also+ set `CUA_E2E_EXPECTED_MIN_CELLS` to 80 on Windows/Linux and 120 on macOS, so+ a filtered, shortened, or accidentally emptied catalog cannot report green.+ Explicit diagnostic cell or harness filters disable only the minimum-count+ check; matching no cells still fails inside the Rust matrix.- GitHub summaries link every evidence-bearing row to its lane archive anddisplay the exact recording path. The trajectory path remains in the typedevidence and archive.diff --git a/libs/cua-driver/docs/test-matrix.md b/libs/cua-driver/docs/test-matrix.mdindex efcbdb3016..b380d5c0d0 100644--- a/libs/cua-driver/docs/test-matrix.md+++ b/libs/cua-driver/docs/test-matrix.md@@ -74,22 +74,28 @@ The shared harness exposes deterministic external markers for these actions:| Action family | Actions | Addressing | Delivery || --- | --- | --- | --- || Pointer | Left click, right click, double click | AX and PX | Background and foreground |-| Keyboard and text | Type text, Return, hotkey | AX and PX where supported | Background and foreground |+| Keyboard and text | Type text, Return, hotkey, type then Return | AX and PX where supported | Background and foreground || Scroll | Scroll | AX and PX where supported | Background and foreground || Child windows | Open child window | AX and PX | Background and foreground || Drag | Drag source to drop target | PX | Background and foreground || State controls | Checkbox, radio, combo, slider | AX or PX by control | Mode declared per action || Editor | Type, save, saved-state readback | AX and PX where supported | Mode declared per action |-The Rust shared catalog declares 36 evidence-bearing cells per harness-application. Windows and Linux run 72 shared cells across Electron and Tauri;-macOS also runs the native WKWebView host for 108 shared cells. Each host covers+The Rust shared catalog declares 40 evidence-bearing cells per harness+application. Windows and Linux run 80 shared cells across Electron and Tauri;+macOS also runs the native WKWebView host for 120 shared cells. Each host coversthe full AX/PX and foreground/background cross-product for click, text,-keyboard, scroll, and child-window actions, plus both delivery modes for PX-drag and AX editor-save. A+keyboard, sequential type-then-Return, scroll, and child-window actions, plus+both delivery modes for PX drag and AX editor-save. Abackground capability refusal is a valid result only when the test verifies thedeclared structured refusal and the no-focus/no-z-order/no-input-leak side-effect. A refusal fails a cell whose contract requires delivery.+effect. Background posture requires one foreground window to contain the target+geometry, not merely overlap it. The lane preflight also injects deliberate+focus and input violations and requires the sentinel to detect both before its+results are trusted. A refusal fails a cell whose contract requires delivery.+Canonical reporting rejects skips and enforces the complete shared catalog+size for unfiltered runs. Diagnostic filters are allowed, but matching zero+cells is always an error.## Harness E2E: Native Windowsdiff --git a/libs/cua-driver/rust/crates/cua-driver-testkit/src/e2e.rs b/libs/cua-driver/rust/crates/cua-driver-testkit/src/e2e.rsindex d47fdc136d..3785e2eac5 100644--- a/libs/cua-driver/rust/crates/cua-driver-testkit/src/e2e.rs+++ b/libs/cua-driver/rust/crates/cua-driver-testkit/src/e2e.rs@@ -166,6 +166,13 @@ fn shared_web_route_for_environment() -> Result<DriverRoute, String> {use DriverRoute as Route;+ if action == "type_submit" && matches!(targeting, Targeting::Ax | Targeting::Px) {+ return match delivery {+ Delivery::Background | Delivery::Foreground => Ok(Route::Composite),+ Delivery::NotApplicable => Err(format!("{action}: delivery mode is required")),+ };+ }+let pointer_or_key_route = |background, foreground| match delivery {Delivery::Background => Ok(background),Delivery::Foreground => Ok(foreground),@@ -1159,9 +1166,18 @@ pub fn validate_catalog_with_evidence(require_turn_evidence: bool,) -> Result<ValidationSummary, Vec<String>> {let mut errors = Vec::new();+ let policy = CatalogPolicy::from_environment(&mut errors);if declarations.is_empty() {errors.push("E2E catalog has no declarations".to_owned());}+ if let Some(minimum) = policy.minimum_cells {+ if declarations.len() < minimum {+ errors.push(format!(+ "E2E catalog declared {} cells, below the required minimum of {minimum}",+ declarations.len()+ ));+ }+ }let mut declared = BTreeMap::new();for case in declarations {if let Err(error) = case.validate() {@@ -1213,7 +1229,14 @@ pub fn validate_catalog_with_evidence(_ => errors.push(format!("passing cell has no valid behavior: {cell_id}")),},TestStatus::Fail | TestStatus::EnvironmentError => summary.failed += 1,- TestStatus::Skip => summary.skipped += 1,+ TestStatus::Skip => {+ summary.skipped += 1;+ if policy.forbid_skips {+ errors.push(format!(+ "canonical E2E catalog may not skip cell: {cell_id}"+ ));+ }+ }}if require_video {@@ -1267,9 +1290,53 @@ pub fn validate_catalog_with_evidence(}}+#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]+struct CatalogPolicy {+ minimum_cells: Option<usize>,+ forbid_skips: bool,+}++impl CatalogPolicy {+ fn from_environment(errors: &mut Vec<String>) -> Self {+ Self::parse(+ std::env::var("CUA_E2E_EXPECTED_MIN_CELLS").ok().as_deref(),+ std::env::var("CUA_E2E_FORBID_SKIPS").ok().as_deref(),+ errors,+ )+ }++ fn parse(minimum: Option<&str>, forbid_skips: Option<&str>, errors: &mut Vec<String>) -> Self {+ let minimum_cells = minimum.and_then(|value| match value.parse::<usize>() {+ Ok(value) if value > 0 => Some(value),+ _ => {+ errors.push(format!(+ "CUA_E2E_EXPECTED_MIN_CELLS must be a positive integer, got {value:?}"+ ));+ None+ }+ });+ let forbid_skips = match forbid_skips {+ None | Some("") | Some("0") | Some("false") => false,+ Some("1") | Some("true") => true,+ Some(value) => {+ errors.push(format!(+ "CUA_E2E_FORBID_SKIPS must be 0/1/false/true, got {value:?}"+ ));+ false+ }+ };+ Self {+ minimum_cells,+ forbid_skips,+ }+ }+}+fn case_requires_action_turn(case: &CaseSpec) -> bool {- !matches!(case.driver_route, DriverRoute::AxRead | DriverRoute::WindowState)- && case.action != "screenshot"+ !matches!(+ case.driver_route,+ DriverRoute::AxRead | DriverRoute::WindowState+ ) && case.action != "screenshot"}fn recording_directory(evidence: &Evidence) -> Option<&Path> {@@ -1317,7 +1384,9 @@ fn validate_turn_evidence());}}- Err(error) => errors.push(format!("invalid trajectory evidence for {cell_id}: {error}")),+ Err(error) => errors.push(format!(+ "invalid trajectory evidence for {cell_id}: {error}"+ )),}let mut turns = match std::fs::read_dir(recording_dir) {@@ -1393,9 +1462,9 @@ fn validate_one_turn(turn: &Path, cell_id: &str, errors: &mut Vec<String>) {.as_str().is_some_and(|summary| summary.starts_with("✅ bring_to_front:"))});- let restored_state_captured = manifest.as_ref().is_some_and(|value| {- value["after"]["state"]["status"].as_str() == Some("captured")- });+ let restored_state_captured = manifest+ .as_ref()+ .is_some_and(|value| value["after"]["state"]["status"].as_str() == Some("captured"));let expected_unavailable_screenshot = |phase: &str| {tool == Some("bring_to_front")&& ((phase == "before"@@ -1743,6 +1812,20 @@ mod tests {assert!(errors.iter().any(|error| error.contains("no declarations")));}+ #[test]+ fn strict_catalog_policy_is_explicit_and_rejects_invalid_values() {+ let mut errors = Vec::new();+ let policy = CatalogPolicy::parse(Some("80"), Some("true"), &mut errors);+ assert!(errors.is_empty());+ assert_eq!(policy.minimum_cells, Some(80));+ assert!(policy.forbid_skips);++ let mut errors = Vec::new();+ let policy = CatalogPolicy::parse(Some("0"), Some("sometimes"), &mut errors);+ assert_eq!(policy, CatalogPolicy::default());+ assert_eq!(errors.len(), 2);+ }+#[test]fn validator_rejects_contradictory_status_and_missing_video() {let case = delivered_case("contradiction");@@ -1895,7 +1978,9 @@ mod tests {let errors = validate_catalog(&[case], &[result], Some(root.path()), true).expect_err("mutable not-applicable cells still need a recorded turn");- assert!(errors.iter().any(|error| error.contains("missing turn evidence")));+ assert!(errors+ .iter()+ .any(|error| error.contains("missing turn evidence")));}#[test]@@ -2284,6 +2369,7 @@ mod tests {"right_click","double_click","type_text",+ "type_submit","press_key","hotkey","scroll",@@ -2299,7 +2385,7 @@ mod tests {cells.push(("drag", Targeting::Px, delivery));cells.push(("editor_save", Targeting::Ax, delivery));}- assert_eq!(cells.len(), 36);+ assert_eq!(cells.len(), 40);for (platform, display_server) in [(Platform::Windows, DisplayServer::Win32),(Platform::Macos, DisplayServer::Quartz),diff --git a/libs/cua-driver/rust/crates/cua-driver-testkit/src/observer.rs b/libs/cua-driver/rust/crates/cua-driver-testkit/src/observer.rsindex 90f2ce9fc7..1a6235ef87 100644--- a/libs/cua-driver/rust/crates/cua-driver-testkit/src/observer.rs+++ b/libs/cua-driver/rust/crates/cua-driver-testkit/src/observer.rs@@ -440,14 +440,32 @@ pub mod windows {},];let target_root = unsafe { root(hwnd) };- let covered = points- .into_iter()- .filter(|point| {- let owner = unsafe { WindowFromPoint(*point) };- !owner.0.is_null() && unsafe { root(owner) } != target_root- })- .count();- Ok(covered >= 2)+ let mut cover_root = None;+ for point in points {+ let owner = unsafe { root(WindowFromPoint(point)) };+ if owner.0.is_null() || owner == target_root {+ return Ok(false);+ }+ match cover_root {+ Some(expected) if expected != owner => return Ok(false),+ None => cover_root = Some(owner),+ _ => {}+ }+ }+ let Some(cover_root) = cover_root else {+ return Ok(false);+ };+ let mut cover = RECT::default();+ unsafe { GetWindowRect(cover_root, &mut cover) }+ .map_err(|error| ObserverError::new(format!("read covering window bounds: {error}")))?;+ Ok(rect_fully_contains(cover, rect, 2))+ }++ fn rect_fully_contains(cover: RECT, target: RECT, tolerance: i32) -> bool {+ cover.left <= target.left + tolerance+ && cover.top <= target.top + tolerance+ && cover.right >= target.right - tolerance+ && cover.bottom >= target.bottom - tolerance}}@@ -506,12 +524,11 @@ pub mod macos {self.width.max(0.0) * self.height.max(0.0)}- fn intersection_area(self, other: Self) -> f64 {- let left = self.x.max(other.x);- let top = self.y.max(other.y);- let right = (self.x + self.width).min(other.x + other.width);- let bottom = (self.y + self.height).min(other.y + other.height);- (right - left).max(0.0) * (bottom - top).max(0.0)+ fn fully_contains(self, other: Self, tolerance: f64) -> bool {+ self.x <= other.x + tolerance+ && self.y <= other.y + tolerance+ && self.x + self.width >= other.x + other.width - tolerance+ && self.y + self.height >= other.y + other.height - tolerance}}@@ -563,12 +580,11 @@ pub mod macos {Some(_) if frontmost_pid() == Some(target.pid as u64) => TargetZ::Foreground,Some(index) => {let target_bounds = rows[index].bounds;- let meaningful_cover = rows[..index].iter().any(|row| {- row.pid != target.pid- && target_bounds.intersection_area(row.bounds)- >= target_bounds.area() * 0.10- });- if meaningful_cover {+ let full_cover = target_bounds.area() > 0.0+ && rows[..index].iter().any(|row| {+ row.pid != target.pid && row.bounds.fully_contains(target_bounds, 2.0)+ });+ if full_cover {TargetZ::BackgroundOccluded} else {TargetZ::BackgroundVisible@@ -738,6 +754,30 @@ pub mod macos {mod tests {use super::*;+ #[test]+ fn partial_overlap_is_not_full_occlusion() {+ let target = Bounds {+ x: 100.0,+ y: 100.0,+ width: 400.0,+ height: 300.0,+ };+ let partial = Bounds {+ x: 0.0,+ y: 0.0,+ width: 350.0,+ height: 900.0,+ };+ let full = Bounds {+ x: 0.0,+ y: 0.0,+ width: 1280.0,+ height: 900.0,+ };+ assert!(!partial.fully_contains(target, 2.0));+ assert!(full.fully_contains(target, 2.0));+ }+#[test]fn native_snapshot_reads_desktop_without_a_target() {let snapshot = MacosObserver::new()@@ -1216,7 +1256,7 @@ pub mod linux {target_pid: u32,workspace: Option<u64>,state: &mut SwayTreeState,- ) {+ ) -> bool {let id = node["id"].as_u64();let workspace = if node["type"].as_str() == Some("workspace") {id@@ -1227,21 +1267,26 @@ pub mod linux {if focused {state.focused = id;state.focused_workspace = workspace;- if node["fullscreen_mode"].as_i64().unwrap_or(0) != 0 {- state.focused_fullscreen = id;- }}if node["pid"].as_u64() == Some(u64::from(target_pid)) {if let Some(id) = id {state.target = Some((id, node["visible"].as_bool().unwrap_or(true), workspace));}}+ let mut subtree_focused = focused;for child in ["nodes", "floating_nodes"].into_iter().flat_map(|key| node[key].as_array().into_iter().flatten()){- walk_sway_tree(child, target_pid, workspace, state);+ subtree_focused |= walk_sway_tree(child, target_pid, workspace, state);}+ // Sway commonly puts fullscreen_mode on a container while focus belongs+ // to a descendant surface. Treat the focused subtree as fullscreen so a+ // sibling hidden behind that container is classified as occluded.+ if subtree_focused && node["fullscreen_mode"].as_i64().unwrap_or(0) != 0 {+ state.focused_fullscreen = id;+ }+ subtree_focused}fn classify_sway_target(state: &SwayTreeState) -> TargetZ {@@ -1264,7 +1309,7 @@ pub mod linux {fn sway_snapshot(target: TargetWindow) -> Result<DesktopSnapshot, ObserverError> {let tree = sway_tree()?;let mut state = SwayTreeState::default();- walk_sway_tree(&tree, target.pid, None, &mut state);+ let _ = walk_sway_tree(&tree, target.pid, None, &mut state);let target_z = classify_sway_target(&state);Ok(DesktopSnapshot {foreground: state.focused,@@ -1277,7 +1322,7 @@ pub mod linux {fn sway_focus_identity() -> Result<Option<u64>, ObserverError> {let tree = sway_tree()?;let mut state = SwayTreeState::default();- walk_sway_tree(&tree, 0, None, &mut state);+ let _ = walk_sway_tree(&tree, 0, None, &mut state);Ok(state.focused)}@@ -1453,18 +1498,43 @@ pub mod linux {if target_bounds.width <= 4 || target_bounds.height <= 4 {return Ok(false);}- let covered = sample_points(target_bounds)- .into_iter()- .filter(|(x, y)| {- connection- .translate_coordinates(root, root, *x, *y)- .ok()- .and_then(|cookie| cookie.reply().ok())- .map(|reply| reply.child != 0 && reply.child != target)- .unwrap_or(true)- })- .count();- Ok(covered >= 2)+ let mut cover = None;+ for (x, y) in sample_points(target_bounds) {+ let child = connection+ .translate_coordinates(root, root, x, y)+ .map_err(x11_error("request X11 occlusion point"))?+ .reply()+ .map_err(x11_error("read X11 occlusion point"))?+ .child;+ if child == 0 || child == target {+ return Ok(false);+ }+ match cover {+ Some(expected) if expected != child => return Ok(false),+ None => cover = Some(child),+ _ => {}+ }+ }+ let Some(cover) = cover else {+ return Ok(false);+ };+ let cover_bounds = absolute_bounds(connection, cover, root)?;+ Ok(rectangle_fully_contains(cover_bounds, target_bounds, 2))+ }++ fn rectangle_fully_contains(cover: Rectangle, target: Rectangle, tolerance: i32) -> bool {+ let cover_left = i32::from(cover.x);+ let cover_top = i32::from(cover.y);+ let cover_right = cover_left + i32::from(cover.width);+ let cover_bottom = cover_top + i32::from(cover.height);+ let target_left = i32::from(target.x);+ let target_top = i32::from(target.y);+ let target_right = target_left + i32::from(target.width);+ let target_bottom = target_top + i32::from(target.height);+ cover_left <= target_left + tolerance+ && cover_top <= target_top + tolerance+ && cover_right >= target_right - tolerance+ && cover_bottom >= target_bottom - tolerance}fn sample_points(bounds: Rectangle) -> [(i16, i16); 5] {@@ -1531,6 +1601,30 @@ pub mod linux {);}+ #[test]+ fn x11_partial_overlap_is_not_full_occlusion() {+ let target = Rectangle {+ x: 100,+ y: 100,+ width: 400,+ height: 300,+ };+ let partial = Rectangle {+ x: 0,+ y: 0,+ width: 350,+ height: 900,+ };+ let full = Rectangle {+ x: 0,+ y: 0,+ width: 1280,+ height: 900,+ };+ assert!(!rectangle_fully_contains(partial, target, 2));+ assert!(rectangle_fully_contains(full, target, 2));+ }+#[test]fn sway_hidden_target_behind_fullscreen_sibling_is_occluded() {let state = SwayTreeState {@@ -1542,6 +1636,46 @@ pub mod linux {assert_eq!(classify_sway_target(&state), TargetZ::BackgroundOccluded);}+ #[test]+ fn sway_fullscreen_ancestor_of_focused_surface_occludes_target() {+ let tree = serde_json::json!({+ "id": 1,+ "type": "root",+ "nodes": [{+ "id": 10,+ "type": "workspace",+ "nodes": [+ {+ "id": 20,+ "pid": 100,+ "visible": false,+ "focused": false,+ "fullscreen_mode": 0,+ "nodes": []+ },+ {+ "id": 30,+ "focused": false,+ "fullscreen_mode": 1,+ "nodes": [{+ "id": 31,+ "pid": 200,+ "visible": true,+ "focused": true,+ "fullscreen_mode": 0,+ "nodes": []+ }]+ }+ ]+ }]+ });+ let mut state = SwayTreeState::default();+ assert!(walk_sway_tree(&tree, 100, None, &mut state));+ assert_eq!(state.focused, Some(31));+ assert_eq!(state.focused_fullscreen, Some(30));+ assert_eq!(classify_sway_target(&state), TargetZ::BackgroundOccluded);+ }+#[test]fn sway_hidden_target_on_another_workspace_is_not_occluded() {let state = SwayTreeState {diff --git a/libs/cua-driver/rust/crates/cua-driver-testkit/src/sentinel.rs b/libs/cua-driver/rust/crates/cua-driver-testkit/src/sentinel.rsindex 3dd32c7288..3d707f30df 100644--- a/libs/cua-driver/rust/crates/cua-driver-testkit/src/sentinel.rs+++ b/libs/cua-driver/rust/crates/cua-driver-testkit/src/sentinel.rs@@ -115,27 +115,50 @@ impl ForegroundSentinel {pub fn observe(&self) -> (Vec<OracleKind>, Vec<String>) {std::thread::sleep(Duration::from_millis(200));- let journal = match fs::read_to_string(&self.journal_path) {- Ok(journal) => journal,+ let events = match read_journal_events(&self.journal_path) {+ Ok(events) => events,Err(error) => {return (Vec::new(),- vec![format!(- "foreground sentinel journal could not be read: {error}"- )],+ vec![format!("foreground sentinel journal is invalid: {error}")],)}};let mut passed = Vec::new();let mut violations = Vec::new();- if journal.contains(r#""kind":"blur""#) {- violations.push("foreground sentinel lost focus".to_owned());- } else {- passed.push(OracleKind::Focus);+ if !events+ .iter()+ .any(|event| event_kind(event) == Some("heartbeat"))+ {+ violations.push("foreground sentinel heartbeat stopped".to_owned());+ }+ if !is_wayland_session() {+ if events.iter().any(|event| {+ event_kind(event) == Some("blur")+ || (event_kind(event) == Some("visibility")+ && event["state"].as_str() == Some("hidden"))+ }) {+ violations.push("foreground sentinel lost focus".to_owned());+ } else {+ passed.push(OracleKind::Focus);+ }}- let leaked = ["keydown", "pointerdown", "wheel", "contextmenu"]- .into_iter()- .filter(|kind| journal.contains(&format!(r#""kind":"{kind}""#)))+ let leaked = events+ .iter()+ .filter_map(|event| event_kind(event))+ .filter(|kind| {+ matches!(+ *kind,+ "keydown"+ | "keyup"+ | "pointerdown"+ | "pointerup"+ | "click"+ | "wheel"+ | "contextmenu"+ | "input"+ )+ }).collect::<Vec<_>>();if leaked.is_empty() {passed.push(OracleKind::NoLeakedInput);@@ -148,6 +171,83 @@ impl ForegroundSentinel {(passed, violations)}+ /// Prove once per canonical lane that the foreground guard can detect both+ /// leaked input and transient focus loss. A guard that cannot observe its+ /// deliberate violations must never be trusted to certify background rows.+ pub fn assert_guard_canaries(+ &self,+ driver: &mut impl Driver,+ background_target: TargetWindow,+ ) -> Result<(), String> {+ wait_for_event(&self.journal_path, "heartbeat", Duration::from_secs(2))?;+ reset_journal(&self.journal_path)?;++ let leaked_key = driver.call(+ "press_key",+ serde_json::json!({+ "pid": self.target.pid,+ "window_id": self.target.native_id,+ "key": "a",+ "delivery_mode": "foreground",+ }),+ );+ if leaked_key.is_error() {+ return Err(format!(+ "foreground input canary could not inject a key: {}",+ leaked_key.text()+ ));+ }+ wait_for_event(&self.journal_path, "keydown", Duration::from_secs(2))?;+ let (_, leaked_input_violations) = self.observe();+ if !leaked_input_violations+ .iter()+ .any(|violation| violation.contains("received input events"))+ {+ return Err(format!(+ "foreground input canary was not detected: {leaked_input_violations:?}"+ ));+ }+ reset_journal(&self.journal_path)?;++ let raised = driver.call(+ "bring_to_front",+ serde_json::json!({+ "pid": background_target.pid,+ "window_id": background_target.native_id,+ }),+ );+ if raised.is_error() {+ return Err(format!(+ "focus-loss canary could not raise the background target: {}",+ raised.text()+ ));+ }+ #[cfg(target_os = "linux")]+ focus_sway_target(driver, background_target)?;+ if is_wayland_session() {+ wait_for_native_focus_lost(self.target)?;+ } else {+ wait_for_event(&self.journal_path, "blur", Duration::from_secs(3))?;+ let (_, focus_violations) = self.observe();+ if !focus_violations+ .iter()+ .any(|violation| violation.contains("lost focus"))+ {+ return Err(format!(+ "focus-loss canary was not detected: {focus_violations:?}"+ ));+ }+ }++ activate_native_foreground(driver, self.target);+ wait_for_native_focus_stable(self.target);+ std::thread::sleep(Duration::from_millis(150));+ reset_journal(&self.journal_path)?;+ wait_for_event(&self.journal_path, "heartbeat", Duration::from_secs(2))?;+ reset_journal(&self.journal_path)?;+ self.assert_background_posture(background_target)+ }+pub fn target(&self) -> TargetWindow {self.target}@@ -178,8 +278,7 @@ impl ForegroundSentinel {activate_native_foreground(driver, self.target);wait_for_native_focus_stable(self.target);std::thread::sleep(Duration::from_millis(100));- fs::write(&self.journal_path, "")- .map_err(|error| format!("reset foreground sentinel journal: {error}"))?;+ reset_journal(&self.journal_path)?;self.assert_background_posture(target)}@@ -271,6 +370,43 @@ fn wait_for_journal(path: &std::path::Path, deadline: Instant, marker: &str, sta}}+fn reset_journal(path: &std::path::Path) -> Result<(), String> {+ fs::write(path, "").map_err(|error| format!("reset foreground sentinel journal: {error}"))+}++fn read_journal_events(path: &std::path::Path) -> Result<Vec<serde_json::Value>, String> {+ let journal =+ fs::read_to_string(path).map_err(|error| format!("read {}: {error}", path.display()))?;+ journal+ .lines()+ .filter(|line| !line.trim().is_empty())+ .map(|line| {+ serde_json::from_str(line)+ .map_err(|error| format!("parse sentinel event {line:?}: {error}"))+ })+ .collect()+}++fn event_kind(event: &serde_json::Value) -> Option<&str> {+ event["kind"].as_str()+}++fn wait_for_event(path: &std::path::Path, kind: &str, timeout: Duration) -> Result<(), String> {+ let deadline = Instant::now() + timeout;+ loop {+ let events = read_journal_events(path)?;+ if events.iter().any(|event| event_kind(event) == Some(kind)) {+ return Ok(());+ }+ if Instant::now() >= deadline {+ return Err(format!(+ "foreground sentinel did not emit {kind:?} within {timeout:?}: {events:?}"+ ));+ }+ std::thread::sleep(Duration::from_millis(25));+ }+}+fn is_wayland_session() -> bool {cfg!(target_os = "linux")&& std::env::var("XDG_SESSION_TYPE")@@ -290,10 +426,192 @@ fn activate_native_foreground(driver: &mut impl Driver, target: TargetWindow) {"could not activate foreground sentinel: {}",response.text());+ #[cfg(target_os = "linux")]+ focus_sway_target(driver, target)+ .expect("could not focus foreground sentinel through Sway IPC");#[cfg(target_os = "windows")]physically_focus_windows_sentinel(target);}+#[cfg(target_os = "linux")]+fn focus_sway_target(driver: &mut impl Driver, target: TargetWindow) -> Result<(), String> {+ let is_sway = is_wayland_session()+ && std::env::var("CUA_E2E_WAYLAND_SESSION").is_ok_and(|session| session == "sway");+ if !is_sway {+ return Ok(());+ }++ let tree_output = Command::new("swaymsg")+ .args(["-r", "-t", "get_tree"])+ .output()+ .map_err(|error| format!("read Sway tree for focus canary: {error}"))?;+ if !tree_output.status.success() {+ return Err(format!(+ "read Sway tree for focus canary: {}",+ String::from_utf8_lossy(&tree_output.stderr).trim()+ ));+ }+ let tree: serde_json::Value = serde_json::from_slice(&tree_output.stdout)+ .map_err(|error| format!("parse Sway tree for focus canary: {error}"))?;++ let windows = driver.call("list_windows", serde_json::json!({}));+ if windows.is_error() {+ return Err(format!(+ "list windows before Sway focus canary: {}",+ windows.text()+ ));+ }+ let title = sway_target_title(windows.structured(), target);+ let con_id = title+ .and_then(|title| sway_container_id(&tree, title))+ .or_else(|| sway_container_id_by_unique_pid(&tree, target.pid))+ .ok_or_else(|| {+ format!(+ "Sway focus canary could not resolve container for pid {} window {}",+ target.pid, target.native_id+ )+ })?;++ let criterion = format!("[con_id={con_id}]");+ let output = Command::new("swaymsg")+ .args(["-r", criterion.as_str(), "focus"])+ .output()+ .map_err(|error| format!("run Sway focus canary for con_id {con_id}: {error}"))?;+ let result: serde_json::Value = serde_json::from_slice(&output.stdout)+ .map_err(|error| format!("parse Sway focus result for con_id {con_id}: {error}"))?;+ let focused = output.status.success()+ && result.as_array().is_some_and(|results| {+ !results.is_empty() && results.iter().all(|item| item["success"] == true)+ });+ if focused {+ Ok(())+ } else {+ Err(format!(+ "Sway focus canary for con_id {con_id} failed: stdout={} stderr={}",+ String::from_utf8_lossy(&output.stdout).trim(),+ String::from_utf8_lossy(&output.stderr).trim(),+ ))+ }+}++#[cfg(target_os = "linux")]+fn sway_target_title(windows: &serde_json::Value, target: TargetWindow) -> Option<&str> {+ let windows = windows["windows"].as_array()?;+ windows+ .iter()+ .find(|window| {+ window["pid"].as_u64() == Some(target.pid as u64)+ && window["window_id"].as_u64() == Some(target.native_id)+ })+ .and_then(|window| window["title"].as_str())+ .or_else(|| {+ let mut titles = windows+ .iter()+ .filter(|window| window["pid"].as_u64() == Some(target.pid as u64))+ .filter_map(|window| window["title"].as_str())+ .filter(|title| !title.is_empty());+ let title = titles.next()?;+ titles.next().is_none().then_some(title)+ })+}++#[cfg(target_os = "linux")]+fn collect_sway_container_ids_by_pid(node: &serde_json::Value, pid: u32, ids: &mut Vec<i64>) {+ if node["pid"].as_u64() == Some(pid as u64) {+ if let Some(id) = node["id"].as_i64() {+ ids.push(id);+ }+ }+ for key in ["nodes", "floating_nodes"] {+ if let Some(children) = node[key].as_array() {+ for child in children {+ collect_sway_container_ids_by_pid(child, pid, ids);+ }+ }+ }+}++#[cfg(target_os = "linux")]+fn sway_container_id_by_unique_pid(node: &serde_json::Value, pid: u32) -> Option<i64> {+ let mut ids = Vec::new();+ collect_sway_container_ids_by_pid(node, pid, &mut ids);+ (ids.len() == 1).then(|| ids[0])+}++#[cfg(target_os = "linux")]+fn sway_container_id(node: &serde_json::Value, title: &str) -> Option<i64> {+ let raw_title = title+ .rsplit_once(" [")+ .map(|(candidate, _)| candidate)+ .unwrap_or(title);+ if node["name"]+ .as_str()+ .is_some_and(|name| name == title || name == raw_title)+ {+ return node["id"].as_i64();+ }+ ["nodes", "floating_nodes"].into_iter().find_map(|key| {+ node[key].as_array().and_then(|children| {+ children+ .iter()+ .find_map(|child| sway_container_id(child, title))+ })+ })+}++#[cfg(all(test, target_os = "linux"))]+mod sway_tests {+ use super::*;++ #[test]+ fn focus_target_uses_unique_sway_pid_when_title_is_unavailable() {+ let tree = serde_json::json!({+ "id": 1,+ "nodes": [{+ "id": 9,+ "nodes": [{+ "id": 42,+ "pid": 1234,+ "name": "CuaTestHarness Electron"+ }]+ }]+ });+ assert_eq!(sway_container_id_by_unique_pid(&tree, 1234), Some(42));+ }++ #[test]+ fn focus_target_does_not_guess_when_pid_owns_multiple_windows() {+ let tree = serde_json::json!({+ "id": 1,+ "nodes": [+ {"id": 41, "pid": 1234, "name": "First"},+ {"id": 42, "pid": 1234, "name": "Second"}+ ]+ });+ assert_eq!(sway_container_id_by_unique_pid(&tree, 1234), None);+ assert_eq!(sway_container_id(&tree, "Second"), Some(42));+ }++ #[test]+ fn focus_target_uses_unique_same_process_title_when_window_id_changes() {+ let windows = serde_json::json!({+ "windows": [{+ "pid": 1234,+ "window_id": 88,+ "title": "CuaTestHarness Electron [cdp=9223]"+ }]+ });+ let target = TargetWindow {+ pid: 1234,+ native_id: 99,+ };+ assert_eq!(+ sway_target_title(&windows, target),+ Some("CuaTestHarness Electron [cdp=9223]")+ );+ }+}+#[cfg(target_os = "windows")]fn physically_focus_windows_sentinel(target: TargetWindow) {use windows::Win32::Foundation::{HWND, POINT, RECT};@@ -374,6 +692,32 @@ fn wait_for_native_focus_stable(target: TargetWindow) {}}+#[cfg(target_os = "linux")]+fn wait_for_native_focus_lost(target: TargetWindow) -> Result<(), String> {+ use crate::observer::{ObserverBackend, TargetZ};++ let backend = NativeObserver::new();+ let deadline = Instant::now() + Duration::from_secs(3);+ loop {+ let lost = backend+ .snapshot(target)+ .map(|snapshot| snapshot.target_z != TargetZ::Foreground)+ .unwrap_or(false);+ if lost {+ return Ok(());+ }+ if Instant::now() >= deadline {+ return Err("native Wayland focus-loss canary was not detected".to_owned());+ }+ std::thread::sleep(Duration::from_millis(25));+ }+}++#[cfg(not(target_os = "linux"))]+fn wait_for_native_focus_lost(_target: TargetWindow) -> Result<(), String> {+ Err("native Wayland focus observation is only available on Linux".to_owned())+}+#[cfg(not(any(target_os = "windows", target_os = "linux")))]fn wait_for_native_focus_stable(_target: TargetWindow) {}diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rsindex e1944140f6..ec7e1f539a 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/cross_platform_behavior_test.rs@@ -399,8 +399,7 @@ fn refresh_wayland_window_geometry(fixture: &mut Fixture) {}#[cfg(not(target_os = "linux"))]-fn refresh_wayland_window_geometry(_fixture: &mut Fixture) {-}+fn refresh_wayland_window_geometry(_fixture: &mut Fixture) {}fn window_origin(fixture: &Fixture, _state: &ToolResponse) -> (f64, f64) {#[cfg(target_os = "macos")]@@ -511,6 +510,78 @@ fn assert_fixture_contains(fixture: &Fixture, marker: &str) {}}+fn assert_fixture_value(fixture: &Fixture, id: &str, expected: &str) {+ let deadline = Instant::now() + Duration::from_secs(2);+ loop {+ let state = fixture.journal.snapshot();+ if state[id]["value"].as_str() == Some(expected) {+ return;+ }+ if Instant::now() >= deadline {+ panic!(+ "{}: fixture value {id:?} did not reach {expected:?}: {state}",+ fixture.name+ );+ }+ thread::sleep(Duration::from_millis(100));+ }+}++fn assert_fixture_single_insertion(+ fixture: &Fixture,+ id: &str,+ initial: &str,+ inserted: &str,+) -> String {+ let deadline = Instant::now() + Duration::from_secs(2);+ loop {+ let state = fixture.journal.snapshot();+ if let Some(value) = state[id]["value"].as_str() {+ if value.len() == initial.len() + inserted.len() {+ for (offset, _) in value.match_indices(inserted) {+ let mut without_insertion = value.to_owned();+ without_insertion.replace_range(offset..offset + inserted.len(), "");+ if without_insertion == initial {+ return value.to_owned();+ }+ }+ }+ }+ if Instant::now() >= deadline {+ panic!(+ "{}: fixture value {id:?} did not insert {inserted:?} exactly once into {initial:?}: {state}",+ fixture.name+ );+ }+ thread::sleep(Duration::from_millis(100));+ }+}++fn assert_fixture_text(fixture: &Fixture, id: &str, expected: &str) {+ let deadline = Instant::now() + Duration::from_secs(2);+ loop {+ let state = fixture.journal.snapshot();+ if state[id]["text"].as_str() == Some(expected) {+ return;+ }+ if Instant::now() >= deadline {+ panic!(+ "{}: fixture text {id:?} did not reach {expected:?}: {state}",+ fixture.name+ );+ }+ thread::sleep(Duration::from_millis(100));+ }+}++fn fixture_marker_number(fixture: &Fixture, id: &str, prefix: &str) -> Option<u64> {+ fixture+ .journal+ .text(id)+ .and_then(|text| text.strip_prefix(prefix).map(str::to_owned))+ .and_then(|value| value.parse().ok())+}+fn action_target_args(fixture: &Fixture,state: &ToolResponse,@@ -661,6 +732,60 @@ fn run_text_action(fixture: &mut Fixture, addressing: &str, delivery: &str) -> OObservation::delivered(passed, Evidence::default())}+fn run_type_submit_action(fixture: &mut Fixture, addressing: &str, delivery: &str) -> Observation {+ let pre = snapshot(fixture);+ let journal_before = fixture.journal.snapshot();+ let text = format!("cua-submit-{addressing}-{delivery}");+ let mut type_args = action_target_args(fixture, &pre, "keyboard-input", addressing, delivery);+ type_args+ .as_object_mut()+ .expect("type-submit type_text arguments object")+ .insert("text".to_owned(), serde_json::json!(text));+ let type_response = fixture.driver.call("type_text", type_args);+ if let Some(code) = background_refusal_code(&type_response, delivery) {+ return refused_without_fixture_mutation(fixture, &journal_before, code, &type_response);+ }+ assert!(+ !type_response.is_error(),+ "{}: type-submit type_text {addressing}/{delivery} failed: {}",+ fixture.name,+ type_response.text()+ );+ assert_fixture_value(fixture, "keyboard-input", &text);++ let post_type = snapshot(fixture);+ let mut enter_args =+ action_target_args(fixture, &post_type, "keyboard-input", addressing, delivery);+ enter_args+ .as_object_mut()+ .expect("type-submit press_key arguments object")+ .insert("key".to_owned(), serde_json::json!("return"));+ let enter_response = fixture.driver.call("press_key", enter_args);+ if let Some(code) = background_refusal_code(&enter_response, delivery) {+ return refused_without_fixture_mutation(fixture, &journal_before, code, &enter_response);+ }+ assert!(+ !enter_response.is_error(),+ "{}: type-submit Enter {addressing}/{delivery} failed: {}",+ fixture.name,+ enter_response.text()+ );+ assert_fixture_contains(fixture, &format!("key_state=submitted:{text}"));++ let mut passed = vec![OracleKind::FixtureState];+ if addressing == "px" {+ passed.extend(unverified_background_protocol_oracle(+ &type_response,+ delivery,+ ));+ }+ passed.extend(unverified_background_protocol_oracle(+ &enter_response,+ delivery,+ ));+ Observation::delivered(passed, Evidence::default())+}+fn run_press_key_action(fixture: &mut Fixture, addressing: &str, delivery: &str) -> Observation {let pre = snapshot(fixture);let journal_before = fixture.journal.snapshot();@@ -728,26 +853,26 @@ fn run_scroll_action(fixture: &mut Fixture, addressing: &str, delivery: &str) ->response.text(),response.raw);- thread::sleep(Duration::from_millis(250));- let offset = fixture- .journal- .text("lbl-scroll-offset")- .and_then(|text| text.split("scroll_offset=").nth(1).map(str::to_owned))- .and_then(|tail| {- tail.split(|ch: char| !ch.is_ascii_digit())- .next()- .map(str::to_owned)- })- .and_then(|value| value.parse::<u64>().ok())- .unwrap_or(0);- assert!(- offset > 0,- "{}: scroll did not advance the external oracle: {}; response={}; raw={}",- fixture.name,- fixture.journal.snapshot(),- response.text(),- response.raw- );+ let deadline = Instant::now() + Duration::from_secs(2);+ loop {+ let offset =+ fixture_marker_number(fixture, "lbl-scroll-offset", "scroll_offset=").unwrap_or(0);+ let page =+ fixture_marker_number(fixture, "lbl-scroll-client-height", "scroll_client_height=")+ .unwrap_or(0);+ if page > 0 && offset >= page {+ break;+ }+ assert!(+ Instant::now() < deadline,+ "{}: two-page scroll did not advance by at least one viewport: {}; response={}; raw={}",+ fixture.name,+ fixture.journal.snapshot(),+ response.text(),+ response.raw+ );+ thread::sleep(Duration::from_millis(100));+ }delivered_observation()}@@ -814,6 +939,14 @@ fn run_editor_save_action(fixture: &mut Fixture, delivery: &str) -> Observationlet pre = snapshot(fixture);let journal_before = fixture.journal.snapshot();let text = format!("cua-editor-{delivery}");+ let initial_value = journal_before["editor-document"]["value"]+ .as_str()+ .unwrap_or_else(|| {+ panic!(+ "{}: editor fixture did not publish its initial value: {journal_before}",+ fixture.name+ )+ });let mut text_args = action_target_args(fixture, &pre, "editor-document", "ax", delivery);text_args.as_object_mut()@@ -829,6 +962,8 @@ fn run_editor_save_action(fixture: &mut Fixture, delivery: &str) -> Observationfixture.name,response.text());+ let expected_value =+ assert_fixture_single_insertion(fixture, "editor-document", initial_value, &text);let post_text = snapshot(fixture);let journal_before_save = fixture.journal.snapshot();@@ -843,7 +978,11 @@ fn run_editor_save_action(fixture: &mut Fixture, delivery: &str) -> Observationfixture.name,response.text());- assert_fixture_contains(fixture, "editor_status=saved:");+ assert_fixture_text(+ fixture,+ "editor-status",+ &format!("editor_status=saved:{expected_value}"),+ );delivered_observation()}@@ -876,7 +1015,11 @@ fn shared_case(spec: &HostSpec, action: &str, addressing: &str, delivery: &str)("electron", "right_click" | "double_click" | "drag", _) => {vec![RefusalCode::BackgroundOccluded]}- ("electron", "type_text" | "press_key" | "hotkey" | "scroll" | "editor_save", _) => {+ (+ "electron",+ "type_text" | "type_submit" | "press_key" | "hotkey" | "scroll" | "editor_save",+ _,+ ) => {vec![RefusalCode::BackgroundUnavailable]}("tauri", "hotkey", _) | ("tauri", "scroll", Targeting::Px) => {@@ -917,7 +1060,9 @@ fn shared_case(spec: &HostSpec, action: &str, addressing: &str, delivery: &str)// must refuse those background keyboard composites instead of reporting// a successful write that the page never observes.match (action, targeting) {- ("type_text", _) | ("editor_save", Targeting::Ax) | ("press_key" | "hotkey", _) => {+ ("type_text" | "type_submit", _)+ | ("editor_save", Targeting::Ax)+ | ("press_key" | "hotkey", _) => {vec![RefusalCode::BackgroundUnavailable]}("right_click" | "double_click" | "scroll", _) | ("drag", Targeting::Px)@@ -949,7 +1094,7 @@ fn shared_case(spec: &HostSpec, action: &str, addressing: &str, delivery: &str)}if !expected_background_refusal&& cfg!(target_os = "windows")- && (matches!(action, "press_key" | "hotkey")+ && (matches!(action, "press_key" | "hotkey" | "type_submit")|| (action == "type_text" && targeting == Targeting::Px)){oracles.push(OracleKind::Protocol);@@ -1010,12 +1155,6 @@ fn cell_selected(case: &CaseSpec) -> bool {parts.peek().is_none() || parts.any(|part| case.cell_id == part || case.cell_id.contains(part))}-fn cell_filter_active() -> bool {- std::env::var("CUA_E2E_CELL_FILTER")- .map(|filter| filter.split(',').any(|part| !part.trim().is_empty()))- .unwrap_or(false)-}-#[test]#[ignore]fn shared_web_action_matrix_is_state_verified() {@@ -1048,6 +1187,10 @@ fn shared_web_action_matrix_is_state_verified() {"type_text",run_text_action as fn(&mut Fixture, &str, &str) -> Observation,),+ (+ "type_submit",+ run_type_submit_action as fn(&mut Fixture, &str, &str) -> Observation,+ ),("press_key",run_press_key_action as fn(&mut Fixture, &str, &str) -> Observation,@@ -1108,8 +1251,8 @@ fn shared_web_action_matrix_is_state_verified() {}}assert!(- !cell_filter_active() || selected > 0,- "CUA_E2E_CELL_FILTER matched no shared E2E cells"+ selected > 0,+ "no shared E2E cells were selected; check CUA_E2E_HARNESS_FILTER and CUA_E2E_CELL_FILTER");resume_first_failure(failure);}diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/e2e_environment_preflight_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/e2e_environment_preflight_test.rsindex 4666962983..05274e8e0f 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/e2e_environment_preflight_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/e2e_environment_preflight_test.rs@@ -5,12 +5,14 @@use std::any::Any;use std::collections::HashSet;use std::panic::{self, AssertUnwindSafe};-use std::process::{Command, Stdio};+use std::process::{Child, Command, Stdio};use std::time::{Duration, Instant};#[cfg(target_os = "linux")]use cua_driver_testkit::e2e::DisplayServer;use cua_driver_testkit::e2e::{write_environment_from_env, EnvironmentRecord};+use cua_driver_testkit::observer::TargetWindow;+use cua_driver_testkit::sentinel::ForegroundSentinel;use cua_driver_testkit::{driver_binary, harness_app, spawn_in_job, Driver, McpDriver};struct PreflightFixture {@@ -20,6 +22,33 @@ struct PreflightFixture {ax_marker: &'static str,}+struct FixtureChildGuard {+ child: Option<Child>,+}++impl FixtureChildGuard {+ fn new(child: Child) -> Self {+ Self { child: Some(child) }+ }++ fn child_mut(&mut self) -> &mut Child {+ self.child.as_mut().expect("fixture child guard is armed")+ }++ fn into_child(mut self) -> Child {+ self.child.take().expect("fixture child guard is armed")+ }+}++impl Drop for FixtureChildGuard {+ fn drop(&mut self) {+ if let Some(child) = &mut self.child {+ let _ = child.kill();+ let _ = child.wait();+ }+ }+}+fn preflight_fixture() -> PreflightFixture {#[cfg(target_os = "windows")]{@@ -98,6 +127,29 @@ fn has_image(response: &cua_driver_testkit::ToolResponse) -> bool {.unwrap_or(false)}+fn readiness_contract(+ is_error: bool,+ element_count: usize,+ marker_present: bool,+ image_present: bool,+) -> bool {+ !is_error && element_count > 0 && marker_present && image_present+}++fn preflight_state_ready(response: &cua_driver_testkit::ToolResponse, ax_marker: &str) -> bool {+ let element_count = response.structured()["element_count"]+ .as_u64()+ .map(|count| count as usize)+ .or_else(|| response.structured()["elements"].as_array().map(Vec::len))+ .unwrap_or_default();+ readiness_contract(+ response.is_error(),+ element_count,+ response.tree_text().contains(ax_marker),+ has_image(response),+ )+}+fn run_preflight() {if let Ok(expected_sha) = std::env::var("CUA_E2E_SOURCE_SHA") {assert!(@@ -192,84 +244,92 @@ fn run_preflight() {.args(&fixture.args).stdout(Stdio::null()).stderr(Stdio::inherit());- let mut child = spawn_in_job(&mut command).expect("preflight fixture failed to launch");+ let child = spawn_in_job(&mut command).expect("preflight fixture failed to launch");let launched_pid = child.id() as i64;+ let mut child = FixtureChildGuard::new(child);- let deadline = Instant::now() + Duration::from_secs(20);+ let deadline = Instant::now() + Duration::from_secs(35);+ let mut last_readiness = "fixture window has not appeared".to_owned();+ #[cfg(target_os = "linux")]+ let mut activated_window_id = None;let (pid, window_id) = loop {if let Some(status) = child+ .child_mut().try_wait().expect("could not inspect preflight fixture process"){panic!("preflight fixture exited before mapping a window: {status}");}- let response = driver.call("list_windows", serde_json::json!({}));- if let Some(window) = response.structured()["windows"]+ let windows = driver.call("list_windows", serde_json::json!({}));+ let candidate = windows.structured()["windows"].as_array().and_then(|windows| {- windows.iter().find(|window| {- window["window_id"]- .as_u64()- .map(|id| !before_ids.contains(&id))- .unwrap_or(false)- && window["title"]- .as_str()- .unwrap_or("")- .contains(fixture.title)+ windows.iter().find_map(|window| {+ let window_id = window["window_id"].as_u64()?;+ let is_new = !before_ids.contains(&window_id);+ let title_matches = window["title"]+ .as_str()+ .unwrap_or("")+ .contains(fixture.title);+ (is_new && title_matches)+ .then(|| (window["pid"].as_i64().unwrap_or(launched_pid), window_id))})- })- {- if let Some(window_id) = window["window_id"].as_u64() {- break (window["pid"].as_i64().unwrap_or(launched_pid), window_id);- }+ });+ if windows.is_error() {+ last_readiness = format!("list_windows failed: {}", windows.text());}- if Instant::now() >= deadline {- let _ = child.kill();- let _ = child.wait();- panic!("preflight fixture window did not appear");- }- std::thread::sleep(Duration::from_millis(200));- };- driver.reaper().push(child);- #[cfg(target_os = "linux")]- {- if DisplayServer::current() == DisplayServer::X11 {- let activated = driver.call(- "bring_to_front",- serde_json::json!({ "pid": pid, "window_id": window_id }),- );- assert!(- !activated.is_error(),- "preflight fixture could not be placed on the Linux desktop: {}",- activated.text()+ if let Some((pid, window_id)) = candidate {+ #[cfg(target_os = "linux")]+ if DisplayServer::current() == DisplayServer::X11+ && activated_window_id != Some(window_id)+ {+ let activated = driver.call(+ "bring_to_front",+ serde_json::json!({ "pid": pid, "window_id": window_id }),+ );+ assert!(+ !activated.is_error(),+ "preflight fixture could not be placed on the Linux desktop: {}",+ activated.text()+ );+ activated_window_id = Some(window_id);+ std::thread::sleep(Duration::from_millis(300));+ }++ let state = driver.call(+ "get_window_state",+ serde_json::json!({+ "pid": pid,+ "window_id": window_id,+ "capture_mode": "ax"+ }),);- std::thread::sleep(Duration::from_millis(300));+ if preflight_state_ready(&state, fixture.ax_marker) {+ break (pid, window_id);+ }+ last_readiness = state.text().to_owned();}- }- let deadline = Instant::now() + Duration::from_secs(15);- loop {- let state = driver.call(- "get_window_state",- serde_json::json!({- "pid": pid,- "window_id": window_id,- "capture_mode": "ax"- }),- );- if !state.is_error() && state.tree_text().contains(fixture.ax_marker) && has_image(&state) {- break;+ if Instant::now() >= deadline {+ panic!(+ "preflight could not map a ready fixture window with AX state and screenshot: {last_readiness}"+ );}- assert!(- Instant::now() < deadline,- "preflight could not read both AX state and screenshot: {}",- state.text()- );std::thread::sleep(Duration::from_millis(200));- }+ };+ driver.reaper().push(child.into_child());driver.start_behavior_recording();+ let target = TargetWindow {+ pid: pid as u32,+ native_id: window_id,+ };+ let sentinel = ForegroundSentinel::launch(&mut driver);+ sentinel+ .assert_guard_canaries(&mut driver, target)+ .expect("foreground sentinel guard canaries failed");+ drop(sentinel);drop(driver);let video = recording_dir.join("recording.mp4");@@ -338,6 +398,15 @@ fn panic_message(payload: &Box<dyn Any + Send>) -> String {.unwrap_or_else(|| "preflight panicked without a string payload".to_owned())}+#[test]+fn readiness_requires_nonempty_ax_marker_and_screenshot_together() {+ assert!(readiness_contract(false, 1, true, true));+ assert!(!readiness_contract(false, 0, true, true));+ assert!(!readiness_contract(false, 1, false, true));+ assert!(!readiness_contract(false, 1, true, false));+ assert!(!readiness_contract(true, 1, true, true));+}+#[test]#[ignore]fn canonical_e2e_environment_is_ready() {diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rsindex 0c465a8004..aeb7a288f0 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs@@ -136,9 +136,10 @@ fn post_key(pid: i32, key_code: u16, key_down: bool, flags: CGEventFlags) -> any.map_err(|_| anyhow::anyhow!("CGEventSource::new failed"))?;let event = CGEvent::new_keyboard_event(source, key_code, key_down).map_err(|_| anyhow::anyhow!("CGEvent::new_keyboard_event failed"))?;- if flags != CGEventFlags::CGEventFlagNull {- event.set_flags(flags);- }+ // HIDSystemState can inherit physically held modifiers. Always overwrite+ // the event flags, including the empty case, so an unrelated Shift/Caps+ // state cannot leak into a targeted key press.+ event.set_flags(flags);post_keyboard_event(pid, &event);Ok(())}@@ -148,9 +149,7 @@ fn post_key_no_auth(pid: i32, key_code: u16, key_down: bool, flags: CGEventFlags.map_err(|_| anyhow::anyhow!("CGEventSource::new failed"))?;let event = CGEvent::new_keyboard_event(source, key_code, key_down).map_err(|_| anyhow::anyhow!("CGEvent::new_keyboard_event failed"))?;- if flags != CGEventFlags::CGEventFlagNull {- event.set_flags(flags);- }+ event.set_flags(flags);let event_ptr = event.as_ptr() as *mut std::ffi::c_void;// attach_auth_message = false → IOHIDPostEvent path → NSMenu firesif !crate::input::skylight::post_to_pid(pid as libc::pid_t, event_ptr, false) {diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rsindex ad0b54dda6..12a07efd7d 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs@@ -58,12 +58,19 @@ fn resolve_onscreen_point_with_scroll(cy: i32,action_verb: &str,) -> Result<(i32, i32), String> {- if crate::input::point_in_window_bounds(hwnd, cx, cy) {- return Ok((cx, cy));- }- // Off-screen: ask the control to scroll itself into view, then re-resolve.+ let cached_center_in_window = crate::input::point_in_window_bounds(hwnd, cx, cy);+ // A center can still be clipped by a ScrollViewer or a docked sibling while+ // remaining inside the outer HWND rectangle. UIA's IsOffscreen property is+ // authoritative for that case; without it a foreground tap can land on the+ // visible control covering the stale point (for example a bottom toolbar).if let Some(retained) = element_cache.get_element_retained(pid, hwnd, idx) {if retained.is_uia() {+ let is_offscreen = unsafe {+ crate::uia::scroll::element_is_offscreen(retained.as_ptr())+ };+ if cached_center_in_window && is_offscreen != Some(true) {+ return Ok((cx, cy));+ }if let Some((nx, ny)) = unsafe {crate::uia::scroll::scroll_into_view_and_recenter(hwnd, retained.as_ptr())} {@@ -71,14 +78,18 @@ fn resolve_onscreen_point_with_scroll(return Ok((nx, ny));}}+ } else if cached_center_in_window {+ return Ok((cx, cy));}// `retained` drops here → COM Release.+ } else if cached_center_in_window {+ return Ok((cx, cy));}Err(format!(- "Element [{idx}] resolves to ({cx},{cy}), outside its window — tried UIA \- ScrollIntoView but it is still off-screen, so {action_verb} would land on \- whatever is there (e.g. the taskbar). Make the window taller or scroll \- the region into view, then retry."+ "Element [{idx}] resolves to ({cx},{cy}) but is not visibly actionable — \+ tried UIA ScrollIntoView but it is still off-screen, so {action_verb} \+ would land on whatever covers that point. Make the window taller or \+ scroll the region into view, then retry."))}@@ -3742,7 +3753,16 @@ impl Tool for TypeTextTool {// top-level can't be foregrounded) while the a11y-channel// SetValue still lands. Same fix as the UIA Invoke path.crate::uia::fg_bypass::run_with_uwp_bypass(hwnd as isize, || unsafe {- vp.SetValue(&BSTR::from(text_for_uia.as_str()))+ // ValuePattern has no caret/selection insertion API. Preserve+ // the current document and append as the deterministic+ // background fallback; replacing the whole value violates+ // type_text's insertion contract for editors.+ let current = vp+ .CurrentValue()+ .map(|value| value.to_string())+ .unwrap_or_default();+ let inserted = format!("{current}{text_for_uia}");+ vp.SetValue(&BSTR::from(inserted.as_str()))})?;Ok(())})()diff --git a/libs/cua-driver/rust/crates/platform-windows/src/uia/scroll.rs b/libs/cua-driver/rust/crates/platform-windows/src/uia/scroll.rsindex a084bd590b..2e0cd6331e 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/uia/scroll.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/uia/scroll.rs@@ -47,6 +47,21 @@ use windows::Win32::UI::Accessibility::{UIA_ScrollItemPatternId, UIA_ScrollPatternId,};+/// Return UIA's live IsOffscreen state for a retained element pointer.+///+/// `None` means the property could not be read. Callers fail open in that case+/// so a transient UIA query never blocks an otherwise valid coordinate action.+pub unsafe fn element_is_offscreen(element_ptr: usize) -> Option<bool> {+ if element_ptr == 0 {+ return None;+ }+ let _ = CoInitializeEx(None, COINIT_MULTITHREADED);+ let elem: IUIAutomationElement = IUIAutomationElement::from_raw(element_ptr as *mut _);+ let result = elem.CurrentIsOffscreen().ok().map(|value| value.as_bool());+ std::mem::forget(elem);+ result+}+/// Ask the UIA element behind `element_ptr` to scroll itself into view, then/// return its fresh on-screen center (from the element's *live* bounding rect,/// since the cached center is stale once the control has moved).@@ -84,8 +99,12 @@ pub unsafe fn scroll_into_view_and_recenter(// - otherwise the coordinate tap would still miss. This mirrors the caller's// own bounds check so an ineffective ScrollIntoView falls through to the// ancestor ScrollPattern fallback instead of returning a bogus center.- let in_bounds = |c: (i32, i32)| -> Option<(i32, i32)> {- if crate::input::point_in_window_bounds(host_hwnd, c.0, c.1) {+ let actionable = |c: (i32, i32)| -> Option<(i32, i32)> {+ let visible = elem+ .CurrentIsOffscreen()+ .map(|value| !value.as_bool())+ .unwrap_or(true);+ if visible && crate::input::point_in_window_bounds(host_hwnd, c.0, c.1) {Some(c)} else {None@@ -93,8 +112,8 @@ pub unsafe fn scroll_into_view_and_recenter(};let result = scroll_item_into_view(host_hwnd, &elem)- .and_then(in_bounds)- .or_else(|| scroll_ancestor_into_view(host_hwnd, &elem).and_then(in_bounds));+ .and_then(actionable)+ .or_else(|| scroll_ancestors_into_view(host_hwnd, &elem).and_then(actionable));// Don't Release the cache's ref - the RetainedElement guard owns it.std::mem::forget(elem);@@ -155,9 +174,10 @@ unsafe fn scroll_item_into_view(host_hwnd: u64, elem: &IUIAutomationElement) ->Some(((rect.left + rect.right) / 2, (rect.top + rect.bottom) / 2))}-/// Strategy 2: walk up to the first vertically-scrollable ancestor and-/// `ScrollPattern::Scroll` until the target's live rect re-enters the viewport.-unsafe fn scroll_ancestor_into_view(+/// Strategy 2: walk every vertically-scrollable ancestor and use+/// `ScrollPattern::Scroll` until the target's live rect re-enters a visible+/// viewport.+unsafe fn scroll_ancestors_into_view(host_hwnd: u64,elem: &IUIAutomationElement,) -> Option<(i32, i32)> {@@ -167,9 +187,11 @@ unsafe fn scroll_ancestor_into_view(CoCreateInstance(&CUIAutomation, None, CLSCTX_INPROC_SERVER).ok()?;let walker = automation.ControlViewWalker().ok()?;- // Find the nearest ancestor that exposes a vertically-scrollable ScrollPattern.- let mut scroll_pat: Option<IUIAutomationScrollPattern> = None;- let mut container: Option<IUIAutomationElement> = None;+ // Collect every vertically-scrollable ancestor. Nested controls are common:+ // a ListItem may already be visible inside its ListBox while that entire+ // ListBox is clipped by an outer page ScrollViewer. Trying only the nearest+ // pattern leaves the element hidden even though another ancestor can reveal it.+ let mut containers: Vec<(IUIAutomationElement, IUIAutomationScrollPattern)> = Vec::new();let mut cur = walker.GetParentElement(elem).ok()?;for _ in 0..16 {if let Ok(p) = cur.GetCurrentPattern(UIA_ScrollPatternId) {@@ -179,9 +201,7 @@ unsafe fn scroll_ancestor_into_view(.map(|b| b.as_bool()).unwrap_or(false);if scrollable {- container = Some(cur.clone());- scroll_pat = Some(sp);- break;+ containers.push((cur.clone(), sp));}}}@@ -190,36 +210,45 @@ unsafe fn scroll_ancestor_into_view(Err(_) => break,}}- let sp = scroll_pat?;- let container = container?;- let crect = container.CurrentBoundingRectangle().ok()?;- // Iteratively scroll the container until the target's center sits inside the- // viewport. Large steps to close distance, small steps once we overshoot- // (direction flip) so we don't oscillate. All wrapped in the fg bypass.- let mut last_dir = 0i32;- for _ in 0..60 {- let tr = elem.CurrentBoundingRectangle().ok()?;- let tcy = (tr.top + tr.bottom) / 2;- if tcy >= crect.top && tcy <= crect.bottom {- break; // center is within the viewport+ for (container, sp) in containers {+ let crect = container.CurrentBoundingRectangle().ok()?;+ // Iteratively scroll this container until the target's center sits inside+ // its viewport. If UIA still marks the target off-screen, continue with+ // the next outer scrollable ancestor.+ let mut last_dir = 0i32;+ for _ in 0..60 {+ let tr = elem.CurrentBoundingRectangle().ok()?;+ let tcy = (tr.top + tr.bottom) / 2;+ if tcy >= crect.top && tcy <= crect.bottom {+ break;+ }+ let dir = if tcy < crect.top { -1 } else { 1 };+ let overshot = last_dir != 0 && dir != last_dir;+ let amount = match (dir, overshot) {+ (-1, false) => ScrollAmount_LargeDecrement,+ (-1, true) => ScrollAmount_SmallDecrement,+ (_, false) => ScrollAmount_LargeIncrement,+ (_, true) => ScrollAmount_SmallIncrement,+ };+ let scrolled = crate::uia::fg_bypass::run_with_uwp_bypass(host_hwnd as isize, || {+ sp.Scroll(ScrollAmount_NoAmount, amount)+ });+ if scrolled.is_err() {+ break;+ }+ last_dir = dir;+ sleep(Duration::from_millis(15));}- let dir = if tcy < crect.top { -1 } else { 1 };- let overshot = last_dir != 0 && dir != last_dir;- let amount = match (dir, overshot) {- (-1, false) => ScrollAmount_LargeDecrement,- (-1, true) => ScrollAmount_SmallDecrement,- (_, false) => ScrollAmount_LargeIncrement,- (_, true) => ScrollAmount_SmallIncrement,- };- let scrolled = crate::uia::fg_bypass::run_with_uwp_bypass(host_hwnd as isize, || {- sp.Scroll(ScrollAmount_NoAmount, amount)- });- if scrolled.is_err() {- break;+ let rect = elem.CurrentBoundingRectangle().ok()?;+ let center = ((rect.left + rect.right) / 2, (rect.top + rect.bottom) / 2);+ let visible = elem+ .CurrentIsOffscreen()+ .map(|value| !value.as_bool())+ .unwrap_or(true);+ if visible && crate::input::point_in_window_bounds(host_hwnd, center.0, center.1) {+ return Some(center);}- last_dir = dir;- sleep(Duration::from_millis(15)); // let the layout settle before re-reading}let rect = elem.CurrentBoundingRectangle().ok()?;diff --git a/libs/cua-driver/tests/fixtures/apps/cross-platform/electron/main.js b/libs/cua-driver/tests/fixtures/apps/cross-platform/electron/main.jsindex 4b00250f8e..8f027d1117 100644--- a/libs/cua-driver/tests/fixtures/apps/cross-platform/electron/main.js+++ b/libs/cua-driver/tests/fixtures/apps/cross-platform/electron/main.js@@ -61,13 +61,17 @@ function createWindow() {const fixedTitle = sentinelMode? `CuaTestHarness Sentinel [cdp=${CDP_PORT}]`: `CuaTestHarness Electron [cdp=${CDP_PORT}]`;+ // GitHub's interactive Windows desktop can be 1024x768. Keep the normal+ // fixture wholly inside its work area so a maximized sentinel can prove+ // geometric full occlusion instead of accepting an off-screen target.+ const compactWindowsFixture = !sentinelMode && process.platform === 'win32';mainWindow = new BrowserWindow({- width: sentinelMode ? 1280 : 940,- height: sentinelMode ? 900 : 780,+ width: sentinelMode ? 1280 : compactWindowsFixture ? 900 : 940,+ height: sentinelMode ? 900 : compactWindowsFixture ? 640 : 780,// Keep the normal fixture inside virtual desktops whose window manager// has no persisted placement policy (notably Openbox under Xvfb).- x: sentinelMode ? 0 : 120,- y: sentinelMode ? 0 : 120,+ x: sentinelMode ? 0 : compactWindowsFixture ? 40 : 120,+ y: sentinelMode ? 0 : compactWindowsFixture ? 40 : 120,title: fixedTitle,// Map the normal harness immediately. Xvfb/Openbox can enumerate a// deferred BrowserWindow while never painting it into the root desktop.@@ -120,7 +124,11 @@ function createWindow() {if (process.platform !== 'darwin') {mainWindow.setAlwaysOnTop(true);}- mainWindow.maximize();+ if (process.platform === 'linux' && process.env.WAYLAND_DISPLAY) {+ mainWindow.setFullScreen(true);+ } else {+ mainWindow.maximize();+ }mainWindow.show();mainWindow.focus();} else {diff --git a/libs/cua-driver/tests/fixtures/apps/cross-platform/electron/preload.js b/libs/cua-driver/tests/fixtures/apps/cross-platform/electron/preload.jsindex c111490cdd..32a3a5f534 100644--- a/libs/cua-driver/tests/fixtures/apps/cross-platform/electron/preload.js+++ b/libs/cua-driver/tests/fixtures/apps/cross-platform/electron/preload.js@@ -33,16 +33,29 @@ if (sentinelMode) {});window.addEventListener('focus', () => record('focus'));window.addEventListener('blur', () => record('blur'));+ window.addEventListener('visibilitychange', () =>+ record('visibility', { state: document.visibilityState })+ );window.addEventListener('keydown', event =>record('keydown', { key: event.key, code: event.code }));+ window.addEventListener('keyup', event =>+ record('keyup', { key: event.key, code: event.code })+ );window.addEventListener('pointerdown', event =>record('pointerdown', { button: event.button, x: event.clientX, y: event.clientY }));+ window.addEventListener('pointerup', event =>+ record('pointerup', { button: event.button, x: event.clientX, y: event.clientY })+ );+ window.addEventListener('click', event =>+ record('click', { button: event.button, x: event.clientX, y: event.clientY })+ );window.addEventListener('wheel', event =>record('wheel', { delta_x: event.deltaX, delta_y: event.deltaY }));window.addEventListener('contextmenu', event =>record('contextmenu', { x: event.clientX, y: event.clientY }));+ setInterval(() => record('heartbeat'), 100);}diff --git a/libs/cua-driver/tests/fixtures/apps/cross-platform/tauri/src-tauri/src/main.rs b/libs/cua-driver/tests/fixtures/apps/cross-platform/tauri/src-tauri/src/main.rsindex f689a4d095..f974558e81 100644--- a/libs/cua-driver/tests/fixtures/apps/cross-platform/tauri/src-tauri/src/main.rs+++ b/libs/cua-driver/tests/fixtures/apps/cross-platform/tauri/src-tauri/src/main.rs@@ -9,6 +9,17 @@ fn main() {.build();tauri::Builder::default().plugin(journal_plugin)+ .setup(|_app| {+ #[cfg(target_os = "windows")]+ {+ use tauri::{LogicalSize, Manager, Size};++ _app.get_webview_window("main")+ .expect("Tauri main fixture window")+ .set_size(Size::Logical(LogicalSize::new(900.0, 640.0)))?;+ }+ Ok(())+ }).run(tauri::generate_context!()).expect("error while running CuaTestHarness.Tauri");}diff --git a/libs/cua-driver/tests/fixtures/apps/windows/webview2/MainWindow.xaml b/libs/cua-driver/tests/fixtures/apps/windows/webview2/MainWindow.xamlindex 12b4acb0c2..6c1dbb0893 100644--- a/libs/cua-driver/tests/fixtures/apps/windows/webview2/MainWindow.xaml+++ b/libs/cua-driver/tests/fixtures/apps/windows/webview2/MainWindow.xaml@@ -4,7 +4,7 @@xmlns:wv2="clr-namespace:Microsoft.Web.WebView2.Wpf;assembly=Microsoft.Web.WebView2.Wpf"Title="Loading WebView2 fixture"AutomationProperties.AutomationId="wnd-main"- Width="940" Height="780"+ Width="900" Height="650"WindowStartupLocation="CenterScreen"><DockPanel LastChildFill="True"><Border DockPanel.Dock="Top" Background="#F0F0F0" Padding="10,6">diff --git a/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml b/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xamlindex c500005cac..3f47176450 100644--- a/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml+++ b/libs/cua-driver/tests/fixtures/apps/windows/wpf/MainWindow.xaml@@ -5,7 +5,7 @@xmlns:auto="clr-namespace:System.Windows.Automation;assembly=PresentationCore"Title="CuaTestHarness WPF"AutomationProperties.AutomationId="wnd-main"- Width="900" Height="900"+ Width="900" Height="650"WindowStartupLocation="CenterScreen"><Window.InputBindings><!-- F5 (no modifiers) — chosen for automated testing. WPF'sdiff --git a/libs/cua-driver/tests/fixtures/shared/web/index.html b/libs/cua-driver/tests/fixtures/shared/web/index.htmlindex f44e725425..4b58b8ad4d 100644--- a/libs/cua-driver/tests/fixtures/shared/web/index.html+++ b/libs/cua-driver/tests/fixtures/shared/web/index.html@@ -69,7 +69,8 @@<legend>text_input</legend><div class="row"><input type="text" id="txt-input" data-cua-id="txt-input"- aria-label="txt-input" placeholder="type here" />+ aria-label="txt-input" placeholder="type here"+ autocapitalize="off" autocorrect="off" spellcheck="false" /><span class="mirror" id="lbl-input-mirror" data-cua-id="lbl-input-mirror">mirror=</span></div></fieldset>@@ -78,7 +79,8 @@<legend>keyboard_task</legend><div class="row"><input type="text" id="keyboard-input" data-cua-id="keyboard-input" autofocus- aria-label="keyboard-input" placeholder="keyboard input" />+ aria-label="keyboard-input" placeholder="keyboard input"+ autocapitalize="off" autocorrect="off" spellcheck="false" /><span class="mirror" id="keyboard-status" data-cua-id="keyboard-status">key_state=none</span></div></fieldset>@@ -86,7 +88,9 @@<fieldset><legend>text_editor_task</legend><div class="row">- <textarea id="editor-document" data-cua-id="editor-document" aria-label="editor-document" rows="3" cols="42">Initial conformance text.\nStatus: DRAFT\n</textarea>+ <textarea id="editor-document" data-cua-id="editor-document" aria-label="editor-document"+ autocapitalize="off" autocorrect="off" spellcheck="false"+ rows="3" cols="42">Initial conformance text.\nStatus: DRAFT\n</textarea><button id="editor-save" data-cua-id="editor-save" aria-label="editor-save Save">Save</button><span class="mirror" id="editor-status" data-cua-id="editor-status">editor_status=draft</span></div>@@ -121,6 +125,7 @@<div id="scroll-bottom" data-cua-id="scroll-bottom">SCROLL_BOTTOM_MARKER_v1</div></div><span class="mirror" id="lbl-scroll-offset" data-cua-id="lbl-scroll-offset">scroll_offset=0</span>+ <span class="mirror" id="lbl-scroll-client-height" data-cua-id="lbl-scroll-client-height">scroll_client_height=0</span></div></fieldset>@@ -207,7 +212,7 @@$('editor-save').addEventListener('click', () => {const value = $('editor-document').value;localStorage.setItem('cua-editor-document', value);- $('editor-status').textContent = `editor_status=saved:${value.endsWith('\\n') ? 'newline' : 'exact'}`;+ $('editor-status').textContent = `editor_status=saved:${value}`;});$('txt-input').addEventListener('input', e => {@@ -218,7 +223,10 @@if (e.ctrlKey && e.shiftKey && e.code === 'Digit7') {$('keyboard-status').textContent = 'key_state=hotkey';} else if (e.key === 'Enter') {- $('keyboard-status').textContent = 'key_state=enter';+ const value = $('keyboard-input').value;+ $('keyboard-status').textContent = value+ ? `key_state=submitted:${value}`+ : 'key_state=enter';}});@@ -234,6 +242,7 @@d.textContent = 'line ' + String(i).padStart(2, '0');lines.appendChild(d);}+ $('lbl-scroll-client-height').textContent = `scroll_client_height=${$('scroll-tall').clientHeight}`;$('scroll-tall').addEventListener('scroll', e => {$('lbl-scroll-offset').textContent = `scroll_offset=${Math.round(e.target.scrollTop)}`;});diff --git a/scripts/ci/linux/run-rust-e2e.sh b/scripts/ci/linux/run-rust-e2e.shindex e5e5e55975..a01f36bbe0 100755--- a/scripts/ci/linux/run-rust-e2e.sh+++ b/scripts/ci/linux/run-rust-e2e.sh@@ -56,6 +56,13 @@ export CUA_TEST_DRIVER_BIN="${RUST_ROOT}/target/release/cua-driver"export CUA_TEST_APPS_ROOT="${RUST_ROOT}/test-apps"export CUA_TEST_REQUIRE_FIXTURES=1export CUA_TEST_DRIVER_STDERR=1+export CUA_E2E_FORBID_SKIPS=1+unset CUA_E2E_EXPECTED_MIN_CELLS+if [[ ("${SUITE}" == shared || "${SUITE}" == all) \+ && -z "${CUA_E2E_CELL_FILTER:-}" \+ && -z "${CUA_E2E_HARNESS_FILTER:-}" ]]; then+ export CUA_E2E_EXPECTED_MIN_CELLS=80+fiexport RUST_BACKTRACE="${RUST_BACKTRACE:-1}"if [[ -z "${CUA_E2E_SOURCE_SHA:-}" && -f "${REPO_ROOT}/.cua-e2e-source-sha" ]]; thenexport CUA_E2E_SOURCE_SHA="$(tr -d '[:space:]' < "${REPO_ROOT}/.cua-e2e-source-sha")"diff --git a/scripts/ci/macos/run-rust-e2e.sh b/scripts/ci/macos/run-rust-e2e.shindex ddc647fbcc..b735cb6d55 100755--- a/scripts/ci/macos/run-rust-e2e.sh+++ b/scripts/ci/macos/run-rust-e2e.sh@@ -74,6 +74,13 @@ export CUA_TEST_DRIVER_BIN="${RUST_ROOT}/target/release/cua-driver"export CUA_TEST_APPS_ROOT="${RUST_ROOT}/test-apps"export CUA_TEST_REQUIRE_FIXTURES=1export CUA_TEST_DRIVER_STDERR=1+export CUA_E2E_FORBID_SKIPS=1+unset CUA_E2E_EXPECTED_MIN_CELLS+if [[ ("${SUITE}" == shared || "${SUITE}" == all) \+ && -z "${CUA_E2E_CELL_FILTER:-}" \+ && -z "${CUA_E2E_HARNESS_FILTER:-}" ]]; then+ export CUA_E2E_EXPECTED_MIN_CELLS=120+ficommand -v ffmpeg >/dev/null || { echo "ffmpeg is required for E2E trajectory videos" >&2; exit 1; }command -v ffprobe >/dev/null || { echo "ffprobe is required for E2E trajectory validation" >&2; exit 1; }diff --git a/scripts/ci/windows/run-rust-e2e.ps1 b/scripts/ci/windows/run-rust-e2e.ps1index e67533d3c8..1f7d4278d9 100644--- a/scripts/ci/windows/run-rust-e2e.ps1+++ b/scripts/ci/windows/run-rust-e2e.ps1@@ -49,6 +49,13 @@ $env:CUA_TEST_DRIVER_BIN = Join-Path $rustRoot "target\release\cua-driver.exe"$env:CUA_TEST_APPS_ROOT = Join-Path $rustRoot "test-apps"$env:CUA_TEST_REQUIRE_FIXTURES = "1"$env:CUA_TEST_DRIVER_STDERR = "1"+$env:CUA_E2E_FORBID_SKIPS = "1"+Remove-Item Env:CUA_E2E_EXPECTED_MIN_CELLS -ErrorAction SilentlyContinue+if (($suite -in @("shared", "all")) -and+ [string]::IsNullOrWhiteSpace($env:CUA_E2E_CELL_FILTER) -and+ [string]::IsNullOrWhiteSpace($env:CUA_E2E_HARNESS_FILTER)) {+ $env:CUA_E2E_EXPECTED_MIN_CELLS = "80"+}if (-not $NoBuild) {& cargo build --release -p cua-driver --manifest-path (Join-Path $rustRoot "Cargo.toml")