Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
Remove legacy Python e2e suite
Francesco Bonacci0f4714fbf942
diff --git a/docs/content/docs/reference/cua-driver/development.mdx b/docs/content/docs/reference/cua-driver/development.mdxindex d90b09cba5..b2b17944d6 100644--- a/docs/content/docs/reference/cua-driver/development.mdx+++ b/docs/content/docs/reference/cua-driver/development.mdx@@ -14,7 +14,7 @@ they stay close to implementation changes.| `libs/cua-driver/rust` | Rust workspace for the daemon, platform crates, testkit, and helper crates || `libs/cua-driver/python` | Python package wrapper around the driver binary || `libs/cua-driver/tests/fixtures` | Source-built GUI harness apps and shared fixtures |-| `libs/cua-driver/tests/e2e` | Manual Python e2e tests against real apps and harness apps |+| `libs/cua-driver/rust/crates/cua-driver/tests` | Rust integration tests for the driver and GUI harnesses || `libs/cua-driver/scripts` | Install, uninstall, local build, and VM sync helpers |## Contributor Entry Points@@ -23,7 +23,7 @@ they stay close to implementation changes.- `libs/cua-driver/rust/README.md` maps the Cargo workspace and test classes.- `libs/cua-driver/rust/crates/cua-driver/tests/README.md` explains Rust test naming and ignored GUI lanes.- `libs/cua-driver/tests/fixtures/README.md` explains source-built harness apps and staged outputs.-- `libs/cua-driver/tests/e2e/README.md` explains the Python e2e suite.+- `libs/cua-driver/rust/crates/cua-driver/tests/README.md` explains Rust test naming and ignored GUI lanes.- `libs/cua-driver/scripts/README.md` explains install and VM sync helpers.Build artifacts, VM logs, and local verification journals should stay out ofdiff --git a/libs/cua-driver/README.md b/libs/cua-driver/README.mdindex d71449e9ff..0c8aef4558 100644--- a/libs/cua-driver/README.md+++ b/libs/cua-driver/README.md@@ -11,12 +11,12 @@ Background computer-use driver for any agents. Speaks MCP over stdio; drives nat| `rust/` | Cargo workspace for the driver daemon, platform crates, and Rust tests || `python/` | Python package wrapper and package tests || `tests/fixtures/` | Source-built GUI harness apps and shared fixtures |-| `tests/e2e/` | Manual Python e2e tests against real apps and harness apps |+| `rust/crates/cua-driver/tests/` | Rust integration tests for the driver and GUI harnesses || `scripts/` | Install, uninstall, local build, and VM sync helpers || `docs/` | Small repo-local specs that are not part of the hosted docs site |-Start with `rust/README.md`, `tests/fixtures/README.md`, and-`tests/e2e/README.md` when changing driver behavior or tests.+Start with `rust/README.md`, `rust/crates/cua-driver/tests/README.md`, and+`tests/fixtures/README.md` when changing driver behavior or tests.## Claude Code computer-use compatibilitydiff --git a/libs/cua-driver/rust/crates/cua-driver/tests/guard_ux_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/guard_ux_test.rsindex dbded6e185..cd8d793c93 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/guard_ux_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/guard_ux_test.rs@@ -1,10 +1,11 @@//! UX-guard integration tests for Windows.//!-//! These mirror the Python e2e tests in libs/cua-driver/tests/e2e/:-//! - test_background_focus.py-//! - test_click_opens_new_window.py-//! - test_launch_app_visible.py-//! - test_background_menu_shortcut.py+//! These cover the UX-guard scenarios previously exercised by the legacy+//! Python suite:+//! - background focus preservation+//! - new-window click delivery+//! - visible app launch+//! - background menu shortcuts//!//! Invariant under test (the "UX guard")://! The agent must be able to click, type, and launch apps in background@@ -14,43 +15,87 @@//! test-apps/harness-electron/CuaTestHarness.Electron.exe. Notepad is used//! only as a secondary fallback check.//!-//! All tests:-//! 1. Launch focus-monitor-win (the "user's foreground window").-//! 2. Perform background operations via cua-driver.-//! 3. Assert focus-monitor-win's act_losses counter stayed at 0.+//! Background-action tests launch the target first, then foreground+//! focus-monitor-win (the "user's foreground window") before measuring focus+//! loss. The launch_app test starts the monitor first because launch behavior is+//! the action under test.//!//! Run in sandbox via://! ..\tests\runners\windows-sandbox\run-tests-in-sandbox.ps1 guard_ux#![cfg(target_os = "windows")]+use std::collections::HashSet;use std::path::PathBuf;use std::process::{Child, Command, Stdio};-use std::time::Duration;+use std::time::{Duration, Instant};-use cua_driver_testkit::{driver_binary, spawn_in_job, workspace_root, Driver, McpDriver};+use cua_driver_testkit::{ax, driver_binary, spawn_in_job, workspace_root, Driver, McpDriver};// ── focus-monitor + test-app fixtures ────────────────────────────────────────-fn require_interactive_desktop(context: &str) -> bool {- if platform_windows::diagnostics::current_session_id() == Some(0) {- eprintln!(- "{context}: running in Windows Session 0; skipping GUI UX guard. \- Re-run from an interactive logon session (RDP/console/scheduled task in user session)."+fn gui_required() -> bool {+ std::env::var("CUA_REQUIRE_GUI")+ .ok()+ .map(|v| matches!(v.to_ascii_lowercase().as_str(), "1" | "true" | "yes" | "on"))+ .unwrap_or(false)+}++fn skip_desktop(context: &str, reason: String) -> bool {+ let msg = format!("{context}: {reason}; skipping GUI UX guard");+ if gui_required() {+ panic!("{msg}");+ }+ eprintln!("{msg}");+ false+}++fn require_seedable_desktop(context: &str) -> bool {+ let state = platform_windows::diagnostics::desktop_state();+ if state.session_id == Some(0) {+ return skip_desktop(+ context,+ format!(+ "running in Windows Session 0 ({}) - re-run from RDP/console/scheduled task in user session",+ state.summary()+ ),+ );+ }+ if !state.has_process_window_station {+ return skip_desktop(+ context,+ format!("no attached process window station ({})", state.summary()),+ );+ }+ if !state.input_desktop_is_default() {+ return skip_desktop(+ context,+ format!(+ "input desktop is not the user Default desktop ({})",+ state.summary()+ ),);- return false;}+ true+}- match platform_windows::diagnostics::interactive_desktop_check() {- Ok(true) => true,- Ok(false) => {- eprintln!("{context}: no foreground interactive desktop; skipping GUI UX guard");- false+fn require_focus_monitor_foreground(context: &str, expected_hwnd: u64) -> bool {+ let deadline = std::time::Instant::now() + Duration::from_secs(2);+ loop {+ let state = platform_windows::diagnostics::desktop_state();+ if state.foreground_hwnd == Some(expected_hwnd as usize) {+ return true;}- Err(e) => {- eprintln!("{context}: interactive desktop check failed ({e}); skipping GUI UX guard");- false+ if std::time::Instant::now() >= deadline {+ return skip_desktop(+ context,+ format!(+ "focus monitor did not become foreground (expected HWND 0x{expected_hwnd:x}; {})",+ state.summary()+ ),+ );}+ std::thread::sleep(Duration::from_millis(100));}}@@ -92,6 +137,40 @@ fn launch_test_app(driver: &mut McpDriver) -> Option<u32> {Some(pid)}+fn launch_driver_and_test_app() -> Option<(McpDriver, u32, u64)> {+ let Some(mut driver) = McpDriver::spawn() else {+ return None;+ };++ let Some(app_pid) = launch_test_app(&mut driver) else {+ eprintln!("test app not available — skipping");+ return None;+ };+ let Some(app_wid) = find_window_for_pid(&mut driver, app_pid as i64) else {+ eprintln!("test app window not found — skipping");+ return None;+ };++ Some((driver, app_pid, app_wid))+}++fn kill_process_tree_by_image(image: &str) {+ Command::new("taskkill")+ .args(["/F", "/T", "/IM", image])+ .stdout(Stdio::null())+ .stderr(Stdio::null())+ .spawn()+ .ok();+}++struct KillProcessTreeOnDrop(&'static str);++impl Drop for KillProcessTreeOnDrop {+ fn drop(&mut self) {+ kill_process_tree_by_image(self.0);+ }+}+fn loss_file() -> PathBuf {std::env::temp_dir().join("focus_monitor_losses.txt")}@@ -117,7 +196,7 @@ fn focus_hwnd_file() -> PathBuf {/// Reads FOCUS_PID and FOCUS_HWND from temp files written by the monitor/// (avoids blocking on the stdout pipe if the sandbox redirects I/O).fn launch_focus_monitor() -> Option<(Child, u64, u32)> {- if !require_interactive_desktop("focus-monitor-win") {+ if !require_seedable_desktop("focus-monitor-win") {return None;}@@ -132,7 +211,7 @@ fn launch_focus_monitor() -> Option<(Child, u64, u32)> {let _ = std::fs::remove_file(focus_pid_file());let _ = std::fs::remove_file(focus_hwnd_file());- let child = spawn_in_job(+ let mut child = spawn_in_job(Command::new(&exe).stdout(Stdio::null()).stderr(Stdio::null()),@@ -163,8 +242,10 @@ fn launch_focus_monitor() -> Option<(Child, u64, u32)> {}std::thread::sleep(Duration::from_millis(100));}- // Give the window time to become foreground.- std::thread::sleep(Duration::from_millis(400));+ if !require_focus_monitor_foreground("focus-monitor-win", hwnd_val) {+ child.kill().ok();+ return None;+ }Some((child, hwnd_val, pid_val))}@@ -180,6 +261,26 @@ fn find_window_for_pid(driver: &mut McpDriver, pid: i64) -> Option<u64> {.find_map(|w| w["window_id"].as_u64())}+fn window_ids(driver: &mut McpDriver) -> HashSet<u64> {+ let resp = driver.call("list_windows", serde_json::json!({}));+ resp.structured()["windows"]+ .as_array()+ .map(|a| a.iter().filter_map(|w| w["window_id"].as_u64()).collect())+ .unwrap_or_default()+}++fn wait_for_new_window(driver: &mut McpDriver, before: &HashSet<u64>) -> bool {+ let deadline = Instant::now() + Duration::from_secs(3);+ while Instant::now() < deadline {+ let after = window_ids(driver);+ if after.iter().any(|id| !before.contains(id)) {+ return true;+ }+ std::thread::sleep(Duration::from_millis(100));+ }+ false+}+// ── UX guard assertion ────────────────────────────────────────────────────────/// Assert act_losses stayed at `max_allowed` (usually 0) since `before`.@@ -199,10 +300,10 @@ fn assert_ux_guard(before: u32, max_allowed: u32, context: &str) {#[test]fn test_background_click_and_type_no_focus_steal() {- //! Equivalent of macOS test_background_focus.py.+ //! Background focus-preservation coverage.//!- //! 1. Launch FocusMonitorWin (simulates the user's active window).- //! 2. Launch the Electron harness in the background.+ //! 1. Launch the Electron harness.+ //! 2. Foreground FocusMonitorWin (simulates the user's active window).//! 3. Click inside the app and type text via cua-driver.//! 4. Assert act_losses on FocusMonitorWin stayed at 0.@@ -211,26 +312,14 @@ fn test_background_click_and_type_no_focus_steal() {return;}- let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {- return;- };- let losses_before = read_losses(&loss_file());-- let Some(mut driver) = McpDriver::spawn() else {- fm_proc.kill().ok();+ let Some((mut driver, app_pid, app_wid)) = launch_driver_and_test_app() else {return;};- let Some(app_pid) = launch_test_app(&mut driver) else {- eprintln!("test app not available — skipping");- fm_proc.kill().ok();- return;- };- let Some(app_wid) = find_window_for_pid(&mut driver, app_pid as i64) else {- eprintln!("test app window not found — skipping");- fm_proc.kill().ok();+ let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {return;};+ let losses_before = read_losses(&loss_file());// Click inside the app (background, via PostMessage).let r = driver.call(@@ -253,6 +342,36 @@ fn test_background_click_and_type_no_focus_steal() {"Protocol error from type_text: {:?}",r.raw);+ assert_eq!(+ r.verified(),+ Some(false),+ "background type_text without an element read-back must not report confirmed success: {}",+ r.text()+ );+ assert_ne!(+ r.structured()["verify"].as_str(),+ Some("confirmed"),+ "background type_text reported a confirmed read-back on an unreadable path: {}",+ r.text()+ );++ // Plain background key dispatch is likewise an unverified PostMessage send,+ // not a confirmed keypress.+ let r = driver.call(+ "press_key",+ serde_json::json!({"pid": app_pid, "window_id": app_wid, "key": "F24"}),+ );+ assert!(+ r.raw["error"].is_null(),+ "Protocol error from press_key: {:?}",+ r.raw+ );+ assert_eq!(+ r.verified(),+ Some(false),+ "background press_key must not report confirmed success: {}",+ r.text()+ );// ux_guard: FocusMonitorWin must not have lost activation.assert_ux_guard(@@ -265,14 +384,15 @@ fn test_background_click_and_type_no_focus_steal() {}// ─────────────────────────────────────────────────────────────────────────────-// Test 2: launch_app does not steal focus+// Test 2: launch_app minimized mode does not steal focus// ─────────────────────────────────────────────────────────────────────────────#[test]-fn test_launch_app_no_focus_steal() {- //! Equivalent of macOS test_launch_app_visible.py.+fn test_launch_app_minimized_no_focus_steal() {+ //! Visible app-launch coverage.//!- //! launch_app (path variant) must open a window without displacing FocusMonitorWin.+ //! launch_app with start_minimized=true is the strict Windows background+ //! launch mode: the app starts without displacing FocusMonitorWin.if !driver_binary().exists() {return;@@ -283,6 +403,7 @@ fn test_launch_app_no_focus_steal() {eprintln!("test app not available — skipping");return;}+ let _cleanup = KillProcessTreeOnDrop("CuaTestHarness.Electron.exe");let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {return;@@ -294,9 +415,12 @@ fn test_launch_app_no_focus_steal() {return;};- // Launch the test app via cua-driver launch_app (full path, SW_SHOWNOACTIVATE).+ // Launch the test app via cua-driver launch_app in strict background mode.let path_str = exe.to_string_lossy().into_owned();- let r = driver.call("launch_app", serde_json::json!({"path": path_str}));+ let r = driver.call(+ "launch_app",+ serde_json::json!({"path": path_str, "start_minimized": true}),+ );if r.is_error() {eprintln!("launch_app failed — skipping: {:?}", r.raw);fm_proc.kill().ok();@@ -330,15 +454,14 @@ fn test_launch_app_no_focus_steal() {}// ux_guard: FocusMonitorWin must not have lost activation.- assert_ux_guard(losses_before, 0, "launch_app CuaTestHarness.Electron");+ assert_ux_guard(+ losses_before,+ 0,+ "launch_app CuaTestHarness.Electron with start_minimized=true",+ );// Kill the launched app by exe name.- Command::new("taskkill")- .args(["/F", "/T", "/IM", "CuaTestHarness.Electron.exe"])- .stdout(Stdio::null())- .stderr(Stdio::null())- .spawn()- .ok();+ kill_process_tree_by_image("CuaTestHarness.Electron.exe");fm_proc.kill().ok();}@@ -348,7 +471,7 @@ fn test_launch_app_no_focus_steal() {#[test]fn test_background_hotkey_no_focus_steal() {- //! Equivalent of macOS test_background_menu_shortcut.py.+ //! Background menu-shortcut coverage.//!//! Send Ctrl+A to a background Electron harness window.//! FocusMonitorWin must never lose activation.@@ -357,26 +480,14 @@ fn test_background_hotkey_no_focus_steal() {return;}- let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {+ let Some((mut driver, app_pid, app_wid)) = launch_driver_and_test_app() else {return;};- let losses_before = read_losses(&loss_file());- let Some(mut driver) = McpDriver::spawn() else {- fm_proc.kill().ok();- return;- };-- let Some(app_pid) = launch_test_app(&mut driver) else {- eprintln!("test app not available — skipping");- fm_proc.kill().ok();- return;- };- let Some(app_wid) = find_window_for_pid(&mut driver, app_pid as i64) else {- eprintln!("test app window not found — skipping");- fm_proc.kill().ok();+ let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {return;};+ let losses_before = read_losses(&loss_file());// Send Ctrl+A hotkey to background app (PostMessage, no focus steal).let r = driver.call(@@ -405,44 +516,41 @@ fn test_background_hotkey_no_focus_steal() {#[test]fn test_background_click_opens_new_window_focus_preserved() {- //! Equivalent of macOS test_click_opens_new_window.py.+ //! New-window click-delivery coverage.//!//! 1. FocusMonitorWin is foreground.- //! 2. Click in CuaTestHarness.Electron - a new window/tab may appear.+ //! 2. Click the CuaTestHarness.Electron child-window button.//! 3. FocusMonitorWin must remain active throughout (UX guard).//!- //! Verifies PostMessage doesn't inadvertently activate any new window that- //! appears as a side-effect of the click.+ //! Verifies that a background click can cause a child window to appear+ //! without activating either the original target or the new window.if !driver_binary().exists() {return;}- let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {+ let Some((mut driver, app_pid, app_wid)) = launch_driver_and_test_app() else {return;};- let losses_before = read_losses(&loss_file());-- let Some(mut driver) = McpDriver::spawn() else {- fm_proc.kill().ok();+ let snap = driver.call(+ "get_window_state",+ serde_json::json!({"pid": app_pid, "window_id": app_wid, "capture_mode": "ax"}),+ );+ let Some(open_idx) = ax::element_index_containing(snap.text(), "Open child window") else {+ eprintln!("child-window button not found in test app — skipping");return;};- let Some(app_pid) = launch_test_app(&mut driver) else {- eprintln!("test app not available — skipping");- fm_proc.kill().ok();- return;- };- let Some(app_wid) = find_window_for_pid(&mut driver, app_pid as i64) else {- eprintln!("test app window not found — skipping");- fm_proc.kill().ok();+ let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {return;};+ let windows_before = window_ids(&mut driver);+ let losses_before = read_losses(&loss_file());- // Click somewhere in the app content area (may trigger navigation/new window).+ // Click the explicit child-window button.let r = driver.call("click",- serde_json::json!({"pid": app_pid, "window_id": app_wid, "x": 400.0, "y": 350.0}),+ serde_json::json!({"pid": app_pid, "window_id": app_wid, "element_index": open_idx}),);assert!(r.raw["error"].is_null(),@@ -450,8 +558,10 @@ fn test_background_click_opens_new_window_focus_preserved() {r.raw);- // Brief wait for any side-effect windows to appear.- std::thread::sleep(Duration::from_millis(500));+ assert!(+ wait_for_new_window(&mut driver, &windows_before),+ "background click did not open a new harness window"+ );// ux_guard: FocusMonitorWin must not have lost activation.assert_ux_guard(@@ -481,26 +591,14 @@ fn test_background_screenshot_no_focus_steal() {return;}- let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {+ let Some((mut driver, _app_pid, app_wid)) = launch_driver_and_test_app() else {return;};- let losses_before = read_losses(&loss_file());- let Some(mut driver) = McpDriver::spawn() else {- fm_proc.kill().ok();- return;- };-- let Some(app_pid) = launch_test_app(&mut driver) else {- eprintln!("test app not available — skipping");- fm_proc.kill().ok();- return;- };- let Some(app_wid) = find_window_for_pid(&mut driver, app_pid as i64) else {- eprintln!("test app window not found — skipping");- fm_proc.kill().ok();+ let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {return;};+ let losses_before = read_losses(&loss_file());// Screenshot via PrintWindow — must not activate the window.let r = driver.call("screenshot", serde_json::json!({"window_id": app_wid}));@@ -540,7 +638,7 @@ fn test_agent_cursor_visible_on_screen() {eprintln!("Binary not found — skipping");return;}- if !require_interactive_desktop("agent cursor visibility") {+ if !require_seedable_desktop("agent cursor visibility") {return;}@@ -551,11 +649,12 @@ fn test_agent_cursor_visible_on_screen() {// Safe centre-ish position on primary monitor.let cx = 640.0_f64;let cy = 400.0_f64;+ let cursor_id = "guard-ux-cursor";// Enable cursor overlay and glide to target.let r = driver.call("set_agent_cursor_enabled",- serde_json::json!({"enabled": true}),+ serde_json::json!({"enabled": true, "cursor_id": cursor_id}),);assert!(r.raw["error"].is_null(),@@ -563,11 +662,28 @@ fn test_agent_cursor_visible_on_screen() {r.raw);- let r = driver.call("move_cursor", serde_json::json!({"x": cx, "y": cy}));+ let r = driver.call(+ "set_agent_cursor_motion",+ serde_json::json!({+ "cursor_id": cursor_id,+ "glide_duration_ms": 100,+ "idle_hide_ms": 0+ }),+ );+ assert!(+ r.raw["error"].is_null(),+ "set_agent_cursor_motion failed: {:?}",+ r.raw+ );++ let r = driver.call(+ "move_cursor",+ serde_json::json!({"x": cx, "y": cy, "cursor_id": cursor_id}),+ );assert!(r.raw["error"].is_null(), "move_cursor failed: {:?}", r.raw);- // Wait for the glide animation (750ms default) + a few render frames.- std::thread::sleep(Duration::from_millis(900));+ // Wait for the fixed 100ms glide plus a few render frames.+ std::thread::sleep(Duration::from_millis(350));// Capture the screen directly (includes layered windows like the overlay).let png_bytes = platform_windows::capture::screenshot_display_bytes()diff --git a/libs/cua-driver/rust/test-apps/.gitignore b/libs/cua-driver/rust/test-apps/.gitignoreindex afcacfa198..fe5703f95e 100644--- a/libs/cua-driver/rust/test-apps/.gitignore+++ b/libs/cua-driver/rust/test-apps/.gitignore@@ -1,5 +1,5 @@-# Pre-built test app binaries are downloaded by run-tests-in-sandbox.ps1-# Run: .\sandbox\run-tests-in-sandbox.ps1 to auto-download before testing+# Pre-built test app binaries are staged by the source-build scripts under+# ../../tests/fixtures/build/ and are never committed.*.exe*.app*.dmgdiff --git a/libs/cua-driver/tests/e2e/PHILOSOPHY.md b/libs/cua-driver/tests/e2e/PHILOSOPHY.mddeleted file mode 100644index 5e0d62aeba..0000000000--- a/libs/cua-driver/tests/e2e/PHILOSOPHY.md+++ /dev/null@@ -1,136 +0,0 @@-# cua-driver Python E2E Test Philosophy--## Core Principle--**Tests exercise the public MCP tool interface only.**--The driver exposes a small, stable interface:--| Tool | Purpose |-|------|---------|-| `get_window_state` | Snapshot: AX/UIA tree + screenshot |-| `click` | Click at element or pixel |-| `type_text` | Type a string |-| `type_text_chars` | Type via keystroke synthesis |-| `press_key` | Send a named key |-| `hotkey` | Send a key combo |-| `scroll` | Scroll wheel |-| `drag` | Mouse drag |--Tests **never** reach for `page.execute_javascript`, CDP, Apple Events, or-any other internal mechanism. The driver is a black box that delivers the-action via whatever technique works best for the target app. Tests verify-the *observable outcome*, not the delivery path.-------## Canonical Test Pattern--Every test follows the same four-step loop:--```-before = get_window_state(pid, wid) # 1. snapshot-element = find(before.tree, ...) # 2. locate-act(pid, wid, element or coords) # 3. act (click / type / key)-after = get_window_state(pid, wid) # 4. assert-assert <state changed in after>-```--- **Snapshot first** — always read the AX tree before acting so element- indices are fresh.-- **Act once** — do not poll between action and assertion. One `get_window_state`- after the action is enough (add a short `sleep` if the app needs to settle).-- **Assert observable state** — AX tree values, window titles, screenshot diffs.- Never assert internal driver behaviour.-------## UX Invariants (checked throughout every test)--A background thread polls at 5 ms and records any violation of:--| Signal | Violation |-|--------|-----------|-| Real cursor position (`NSEvent.mouseLocation`) | Moved > 4 px from baseline |-| Frontmost app PID (`NSWorkspace.frontmostApplication`) | Changed from sentinel |-| Agent overlay z-order (`CGWindowListCopyWindowInfo`) | cua-driver layer ≥ frontmost non-agent window layer |--The `ux_guard` pytest fixture starts the monitor before each test and calls-`monitor.assert_clean()` in teardown. If any violation was recorded the test-fails with a list of every incident.-------## App-specific Oracle Strategy--| App | Primary oracle | Cross-check |-|-----|---------------|-------------|-| Safari | AX tree text values | - |-| Chrome | AX tree text values | - |-| Electron | AX tree text values from shared DOM | - |-| Tauri | AX tree text values from shared DOM | - |-| Blender | Screenshot + NCC template match | AX tree window title |-| Calculator | AX tree static text | - |--For browser apps the test page (`assets/test_page.html`) provides a-self-contained oracle: every interactive element reflects its state back into-the DOM (button click counter, text field value, checkbox label, select-label, textarea content). The AX tree exposes those DOM values directly.-------## Canvas Test Pattern--Canvas pixels are not in the AX tree. Verify canvas interaction by screenshot-diff:--```python-before = get_window_state(pid, wid)-before_crop = cv.crop(cv.decode(before.screenshot), canvas_bbox)-click(pid, wid, x=target_x, y=target_y)-after = get_window_state(pid, wid)-after_crop = cv.crop(cv.decode(after.screenshot), canvas_bbox)-assert cv.diff_ratio(before_crop, after_crop) > 0.001-```-------## Test File Structure--```-tests/e2e/- PHILOSOPHY.md ← this file- conftest.py ← pytest fixtures (driver, ux_guard, sentinel, html_server, ...)- driver_client.py ← raw MCP client wrapper- harness/- __init__.py- monitor.py ← UXMonitor (5 ms PyObjC thread)- driver.py ← thin typed wrapper around DriverClient- tree.py ← tree_markdown parser + find / value / bbox helpers- cv.py ← PIL decode / crop / diff / template-match utilities- assets/- test_page.html ← all-elements test page for browser tests- blender/- toolbar_ref.png ← stored template target- test_safari.py- test_chrome.py- test_electron.py- test_tauri.py- test_blender.py- test_*.py ← older e2e coverage kept here until audited/ported-```-------## Running--```bash-cd libs/cua-driver/tests/e2e-export CUA_DRIVER_BINARY=../../rust/target/debug/cua-driver--# Single file-python3 -m pytest test_safari.py -v--# Full suite-python3 -m pytest -v-```diff --git a/libs/cua-driver/tests/e2e/README.md b/libs/cua-driver/tests/e2e/README.mddeleted file mode 100644index 367dc722c0..0000000000--- a/libs/cua-driver/tests/e2e/README.md+++ /dev/null@@ -1,46 +0,0 @@-# cua-driver Python e2e tests--Python e2e tests drive the public MCP tool interface against real apps and-shared harness apps. They are manual/VM-backed tests, not default CI tests.--## Layout--```text-tests/e2e/-├── driver_client.py # raw MCP client-├── conftest.py # pytest fixtures-├── harness/ # typed wrappers, tree helpers, CV helpers, UX monitor-├── assets/ # symlinks/references for browser and Blender tests-├── PHILOSOPHY.md # black-box test pattern and UX invariants-├── run_tests.sh # pytest runner that builds cua-driver if needed-└── test_*.py # app and legacy e2e modules-```--## Running--```bash-cd libs/cua-driver/tests/e2e-./run_tests.sh test_electron -v-python3 -m pytest test_tauri.py -v-```--Set `CUA_DRIVER_BINARY` to test a specific binary. Without it, `run_tests.sh`-builds `libs/cua-driver/rust` and uses `rust/target/debug/cua-driver`.--## Harness Apps--`test_electron.py` and `test_tauri.py` use repo-local shared harnesses:--- `tests/fixtures/apps/cross-platform/electron`-- `tests/fixtures/apps/cross-platform/tauri`--The fixtures build missing staged apps on demand with each harness `build.sh`.-They no longer download prebuilt app releases.--## Rules--- Assert through observable state in the AX/UIA/AT-SPI tree or screenshots.-- Do not call internal browser APIs from tests.-- Keep app-specific setup in fixtures, not in driver code.-- Treat older root-level tests as legacy coverage until they are audited and- either ported into the shared harness pattern or removed.diff --git a/libs/cua-driver/tests/e2e/assets/blender/toolbar_ref.png b/libs/cua-driver/tests/e2e/assets/blender/toolbar_ref.pngdeleted file mode 100644index 6854ba2aab..0000000000Binary files a/libs/cua-driver/tests/e2e/assets/blender/toolbar_ref.png and /dev/null differdiff --git a/libs/cua-driver/tests/e2e/assets/test_page.html b/libs/cua-driver/tests/e2e/assets/test_page.htmldeleted file mode 120000index 2771775117..0000000000--- a/libs/cua-driver/tests/e2e/assets/test_page.html+++ /dev/null@@ -1 +0,0 @@-../../../../cua-driver-fixtures/test_page.html\ No newline at end of filediff --git a/libs/cua-driver/tests/e2e/conftest.py b/libs/cua-driver/tests/e2e/conftest.pydeleted file mode 100644index 125bd4583a..0000000000--- a/libs/cua-driver/tests/e2e/conftest.py+++ /dev/null@@ -1,292 +0,0 @@-"""pytest fixtures for cua-driver Python e2e tests.--Fixtures:- binary → str — path to cua-driver binary- driver → Driver (context manager, yields a started Driver)- focus_monitor → (proc, pid) — FocusMonitorApp running as frontmost sentinel- ux_guard → UXMonitor — started before the test, assert_clean() after- html_server → str — base URL of a local HTTP server serving assets/- tauri_app → (proc, pid, base_url) — repo-local Tauri harness- electron_app → (proc, pid, base_url) — repo-local Electron harness-"""--from __future__ import annotations--import os-import subprocess-import sys-import time-import threading-from http.server import HTTPServer, SimpleHTTPRequestHandler--import pytest--_HERE = os.path.dirname(os.path.abspath(__file__))-_TESTS_ROOT = os.path.dirname(_HERE)-_CUA_DRIVER_ROOT = os.path.dirname(_TESTS_ROOT)-_DRIVER_RS_ROOT = os.path.join(_CUA_DRIVER_ROOT, "rust")-_LIBS_ROOT = os.path.dirname(_CUA_DRIVER_ROOT)-_TEST_HARNESS_ROOT = os.path.join(_TESTS_ROOT, "fixtures")--sys.path.insert(0, _HERE)-from driver_client import default_binary_path # noqa: E402--_ASSETS_DIR = os.path.join(_HERE, "assets")-_LOSS_FILE = "/tmp/focus_monitor_losses.txt"-_FOCUS_APP_DIR = os.path.join(_LIBS_ROOT, "cua-driver", "Tests", "FocusMonitorApp")-_FOCUS_APP_BUNDLE = os.path.join(_FOCUS_APP_DIR, "FocusMonitorApp.app")-_FOCUS_APP_EXE = os.path.join(_FOCUS_APP_BUNDLE, "Contents", "MacOS", "FocusMonitorApp")--# Tauri is built from the repo-local shared harness rather than downloaded.-_TAURI_HARNESS_DIR = os.path.join(- _TEST_HARNESS_ROOT, "apps", "cross-platform", "tauri"-)-_TAURI_BUILD_SCRIPT = os.path.join(_TAURI_HARNESS_DIR, "build.sh")-_TAURI_APP_BUNDLE = os.path.join(- _DRIVER_RS_ROOT,- "test-apps",- "harness-tauri",- "CuaTestHarness.Tauri.app",-)-_TAURI_APP_EXE = os.path.join(- _TAURI_APP_BUNDLE, "Contents", "MacOS", "CuaTestHarness.Tauri"-)--# Electron is built from the repo-local shared harness rather than downloaded.-_ELECTRON_HARNESS_DIR = os.path.join(- _TEST_HARNESS_ROOT, "apps", "cross-platform", "electron"-)-_ELECTRON_BUILD_SCRIPT = os.path.join(_ELECTRON_HARNESS_DIR, "build.sh")-_ELECTRON_APP_BUNDLE = os.path.join(- _DRIVER_RS_ROOT,- "test-apps",- "harness-electron",- "CuaTestHarness.Electron.app",-)-_ELECTRON_APP_EXE = os.path.join(- _ELECTRON_APP_BUNDLE, "Contents", "MacOS", "Electron"-)---# ── binary ────────────────────────────────────────────────────────────────────-[email protected](scope="session")-def binary() -> str:- return default_binary_path()---# ── driver ────────────────────────────────────────────────────────────────────--def driver(binary):- """Yield a started Driver instance."""- sys.path.insert(0, _HERE)- from harness.driver import Driver- with Driver(binary) as d:- yield d---# ── focus monitor ─────────────────────────────────────────────────────────────--def _build_focus_app() -> None:- if not os.path.exists(_FOCUS_APP_EXE):- build_sh = os.path.join(_FOCUS_APP_DIR, "build.sh")- subprocess.run([build_sh], check=True)---def _launch_focus_app():- """Launch FocusMonitorApp, return (proc, pid)."""- _build_focus_app()- proc = subprocess.Popen(- [_FOCUS_APP_EXE],- stdout=subprocess.PIPE,- stderr=subprocess.DEVNULL,- text=True,- )- for _ in range(40):- line = proc.stdout.readline().strip()- if line.startswith("FOCUS_PID="):- pid = int(line.split("=", 1)[1])- return proc, pid- time.sleep(0.1)- proc.terminate()- raise RuntimeError("FocusMonitorApp did not print FOCUS_PID in time")---def _wait_for_no_focus_app(timeout_s: float = 3.0) -> None:- """Block until no FocusMonitorApp process is running."""- deadline = time.monotonic() + timeout_s- while time.monotonic() < deadline:- r = subprocess.run(- ["pgrep", "-x", "FocusMonitorApp"],- capture_output=True,- )- if r.returncode != 0:- return- time.sleep(0.1)--[email protected](scope="module")-def focus_monitor():- """Launch FocusMonitorApp as frontmost sentinel for the test module."""- # Kill any stale instances from previous modules and wait for full exit.- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)- _wait_for_no_focus_app()- try:- os.remove(_LOSS_FILE)- except FileNotFoundError:- pass-- proc, pid = _launch_focus_app()- time.sleep(0.8)- yield proc, pid-- proc.terminate()- try:- proc.wait(timeout=3)- except subprocess.TimeoutExpired:- proc.kill()- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)- # Wait for the process to fully exit before the next module starts.- _wait_for_no_focus_app()- try:- os.remove(_LOSS_FILE)- except FileNotFoundError:- pass--[email protected](autouse=False)-def activate_focus_monitor(focus_monitor):- """Re-activate FocusMonitorApp before each test (by pid)."""- _, pid = focus_monitor- subprocess.run(- ["osascript", "-e",- f'tell application "System Events" to set frontmost of (first process whose unix id is {pid}) to true'],- check=False,- )- time.sleep(0.4)---# ── UX guard ──────────────────────────────────────────────────────────────────--def ux_guard(focus_monitor):- """Start UXMonitor before the test, call assert_clean() after."""- _, sentinel_pid = focus_monitor- sys.path.insert(0, _HERE)- from harness.monitor import UXMonitor- mon = UXMonitor(sentinel_pid=sentinel_pid)- mon.start()- yield mon- mon.stop()- mon.assert_clean()---# ── local HTML server ─────────────────────────────────────────────────────────--class _SilentHandler(SimpleHTTPRequestHandler):- def log_message(self, *args):- pass--[email protected](scope="session")-def html_server():- """Serve the assets/ directory over HTTP; return base URL."""- handler = lambda *args, **kwargs: _SilentHandler(- *args, directory=_ASSETS_DIR, **kwargs- )- server = HTTPServer(("127.0.0.1", 0), handler)- port = server.server_address[1]- thread = threading.Thread(target=server.serve_forever, daemon=True)- thread.start()- yield f"http://127.0.0.1:{port}"- server.shutdown()---# ── Tauri app ─────────────────────────────────────────────────────────────────-[email protected](scope="session")-def tauri_app():- """Build, launch, and yield (proc, pid, base_url) for the Tauri harness.-- This uses tests/fixtures/apps/cross-platform/tauri, the same shared DOM- harness used by the Electron/WebView/WKWebView harnesses. It does not- download a prebuilt Tauri app release.- """- if not os.path.exists(_TAURI_APP_EXE):- if not os.path.exists(_TAURI_BUILD_SCRIPT):- pytest.skip("repo-local Tauri harness build script not found")- try:- subprocess.run([_TAURI_BUILD_SCRIPT], check=True)- except (OSError, subprocess.CalledProcessError) as e:- pytest.skip(f"Tauri harness build failed: {e}")-- if not os.path.exists(_TAURI_APP_EXE):- pytest.skip("Tauri harness app not found after build")-- subprocess.run(["pkill", "-f", "CuaTestHarness.Tauri"], check=False)- time.sleep(0.5)-- proc = subprocess.Popen(- [_TAURI_APP_EXE],- stdout=subprocess.DEVNULL,- stderr=subprocess.DEVNULL,- )- pid = proc.pid- time.sleep(3.0)-- base_url = "file://shared-web-harness"- yield proc, pid, base_url-- proc.terminate()- try:- proc.wait(timeout=5)- except subprocess.TimeoutExpired:- proc.kill()- subprocess.run(["pkill", "-f", "CuaTestHarness.Tauri"], check=False)---# ── Electron app ──────────────────────────────────────────────────────────────-[email protected](scope="session")-def electron_app():- """Build, launch, and yield (proc, pid, base_url) for the Electron harness.-- This uses tests/fixtures/apps/cross-platform/electron, the same shared DOM- harness used by the Rust harness tests. It does not download a prebuilt- Electron app release.- """- if not os.path.exists(_ELECTRON_APP_EXE):- if not os.path.exists(_ELECTRON_BUILD_SCRIPT):- pytest.skip("repo-local Electron harness build script not found")- try:- subprocess.run([_ELECTRON_BUILD_SCRIPT], check=True)- except (OSError, subprocess.CalledProcessError) as e:- pytest.skip(f"Electron harness build failed: {e}")-- if not os.path.exists(_ELECTRON_APP_EXE):- pytest.skip("Electron harness app not found after build")-- subprocess.run(["pkill", "-f", "CuaTestHarness.Electron"], check=False)- subprocess.run(["pkill", "-f", "Electron.app/Contents/MacOS/Electron"], check=False)- time.sleep(0.5)-- proc = subprocess.Popen(- [_ELECTRON_APP_EXE, "--force-renderer-accessibility"],- stdout=subprocess.DEVNULL,- stderr=subprocess.DEVNULL,- )- time.sleep(4.0) # Electron takes a bit longer with AX enabled-- pid = proc.pid-- base_url = "file://shared-web-harness"- yield proc, pid, base_url-- proc.terminate()- try:- proc.wait(timeout=5)- except subprocess.TimeoutExpired:- proc.kill()- subprocess.run(["pkill", "-f", "CuaTestHarness.Electron"], check=False)- subprocess.run(["pkill", "-f", "Electron.app/Contents/MacOS/Electron"], check=False)diff --git a/libs/cua-driver/tests/e2e/driver_client.py b/libs/cua-driver/tests/e2e/driver_client.pydeleted file mode 100644index e6aefd43d2..0000000000--- a/libs/cua-driver/tests/e2e/driver_client.py+++ /dev/null@@ -1,215 +0,0 @@-"""Minimal MCP stdio client used by cua-driver-rs integration tests.--Speaks JSON-RPC 2.0 line-framed over stdin/stdout. Keeps request IDs monotonic-and strips notifications from the response stream. Deliberately tiny — we want-the test failures to point at cua-driver-rs, not at a heavy client dependency.-"""--from __future__ import annotations--import json-import os-import subprocess-import threading-import time-from collections import deque-from typing import Any, Optional---class DriverClient:- def __init__(self, binary_path: str, subcommand: str = "mcp") -> None:- self.binary_path = binary_path- self.subcommand = subcommand- self.process: Optional[subprocess.Popen] = None- self._next_id = 0- self._lines: deque[str] = deque()- self._reader: Optional[threading.Thread] = None- self._stderr_reader: Optional[threading.Thread] = None-- def __enter__(self) -> "DriverClient":- self.process = subprocess.Popen(- [self.binary_path, self.subcommand],- stdin=subprocess.PIPE,- stdout=subprocess.PIPE,- stderr=subprocess.PIPE,- text=True,- bufsize=1,- )- self._reader = threading.Thread(target=self._pump_stdout, daemon=True)- self._reader.start()- # Drain stderr to prevent pipe-buffer deadlock: the driver writes tracing- # logs to stderr; if the 64KB OS pipe buffer fills the driver blocks and- # the test hangs indefinitely waiting for an MCP response that never comes.- self._stderr_reader = threading.Thread(target=self._drain_stderr, daemon=True)- self._stderr_reader.start()- self._handshake()- return self-- def __exit__(self, *exc: Any) -> None:- if not self.process:- return- for stream in (self.process.stdin, self.process.stdout, self.process.stderr):- try:- if stream is not None:- stream.close()- except Exception:- pass- try:- self.process.terminate()- self.process.wait(timeout=3)- except Exception:- self.process.kill()- self.process.wait(timeout=1)-- def _pump_stdout(self) -> None:- assert self.process and self.process.stdout- for line in self.process.stdout:- line = line.strip()- if line:- self._lines.append(line)-- def _drain_stderr(self) -> None:- """Discard stderr output to prevent pipe-buffer deadlock."""- assert self.process and self.process.stderr- try:- for _ in self.process.stderr:- pass- except Exception:- pass-- def _handshake(self) -> None:- self._call(- "initialize",- {- "protocolVersion": "2025-06-18",- "capabilities": {},- "clientInfo": {"name": "cua-driver-rs-integration", "version": "0.0.1"},- },- )- self._notify("notifications/initialized")-- def _notify(self, method: str, params: Optional[dict] = None) -> None:- payload = {"jsonrpc": "2.0", "method": method}- if params is not None:- payload["params"] = params- self._write(payload)-- def _call(- self, method: str, params: Optional[dict] = None, timeout: float = 20.0- ) -> dict:- self._next_id += 1- request_id = self._next_id- payload = {"jsonrpc": "2.0", "id": request_id, "method": method}- if params is not None:- payload["params"] = params- self._write(payload)-- deadline = time.monotonic() + timeout- while time.monotonic() < deadline:- while self._lines:- line = self._lines.popleft()- try:- msg = json.loads(line)- except json.JSONDecodeError:- continue- if msg.get("id") == request_id:- if "error" in msg:- raise MCPCallError(msg["error"])- return msg["result"]- time.sleep(0.02)- raise TimeoutError(f"no response for {method} within {timeout}s")-- def _write(self, payload: dict) -> None:- assert self.process and self.process.stdin- self.process.stdin.write(json.dumps(payload) + "\n")- self.process.stdin.flush()-- def list_tools(self) -> list[dict]:- return self._call("tools/list")["tools"]-- def call_tool(self, name: str, arguments: Optional[dict] = None) -> dict:- return self._call(- "tools/call",- {"name": name, "arguments": arguments or {}},- )---class MCPCallError(RuntimeError):- def __init__(self, error: dict) -> None:- super().__init__(f"MCP error {error.get('code')}: {error.get('message')}")- self.code = error.get("code")- self.message = error.get("message")---def default_binary_path() -> str:- """Return the path to the cua-driver-rs binary.-- Checks CUA_DRIVER_BINARY env var first, then falls back to the debug build- in the Rust workspace target directory.- """- return os.environ.get(- "CUA_DRIVER_BINARY",- os.path.join(- os.path.dirname(os.path.abspath(__file__)),- "..",- "..",- "rust",- "target",- "debug",- "cua-driver",- ),- )---def reset_calculator(settle_s: float = 0.3) -> None:- """Force a known clean state for Calculator.-- Several tests drive Calculator as a 'freshly launched, never frontmost'- target. `pkill -x Calculator` + short settle is the shared boilerplate.- """- subprocess.run(["pkill", "-x", "Calculator"], check=False)- time.sleep(settle_s)---def frontmost_bundle_id(client: "DriverClient") -> Optional[str]:- """Return the bundle id of the currently-frontmost app, or None."""- apps = client.call_tool("list_apps")["structuredContent"]["apps"]- for app in apps:- if app.get("active"):- return app.get("bundle_id")- return None---def resolve_window_id(- client: "DriverClient", pid: int, require_on_current_space: bool = True-) -> int:- """Pick a `window_id` for `pid` via `list_windows`.-- Prefers windows that are on-screen AND on the user's current Space,- then max z_index. Falls back to any layer-0 window for the pid when- the preferred filter finds nothing (hidden-launched / minimized apps- still have layer-0 windows we can snapshot).-- Raises `RuntimeError` when the pid has no layer-0 window at all.- """- result = client.call_tool("list_windows", {"pid": pid})- windows = result["structuredContent"]["windows"]- if not windows:- raise RuntimeError(f"pid {pid} has no windows")-- if require_on_current_space:- preferred = [- w for w in windows- if w.get("is_on_screen") and w.get("on_current_space") is not False- ]- if preferred:- preferred.sort(key=lambda w: w.get("z_index", 0), reverse=True)- return preferred[0]["window_id"]-- # Fallback: any layer-0 window. Pick the one with the biggest bounds.- def _area(w: dict) -> float:- b = w.get("bounds", {})- return float(b.get("width", 0)) * float(b.get("height", 0))-- windows.sort(key=_area, reverse=True)- return windows[0]["window_id"]diff --git a/libs/cua-driver/tests/e2e/fixtures/interactive.html b/libs/cua-driver/tests/e2e/fixtures/interactive.htmldeleted file mode 120000index d053b042a9..0000000000--- a/libs/cua-driver/tests/e2e/fixtures/interactive.html+++ /dev/null@@ -1 +0,0 @@-../../../../cua-driver-fixtures/interactive.html\ No newline at end of filediff --git a/libs/cua-driver/tests/e2e/fixtures/test_cursor.png b/libs/cua-driver/tests/e2e/fixtures/test_cursor.pngdeleted file mode 100644index 72129ae29a..0000000000Binary files a/libs/cua-driver/tests/e2e/fixtures/test_cursor.png and /dev/null differdiff --git a/libs/cua-driver/tests/e2e/harness/__init__.py b/libs/cua-driver/tests/e2e/harness/__init__.pydeleted file mode 100644index e69de29bb2..0000000000diff --git a/libs/cua-driver/tests/e2e/harness/cv.py b/libs/cua-driver/tests/e2e/harness/cv.pydeleted file mode 100644index d254ebfffe..0000000000--- a/libs/cua-driver/tests/e2e/harness/cv.py+++ /dev/null@@ -1,132 +0,0 @@-"""Computer-vision helpers for screenshot-based assertions.--All operations use PIL (Pillow) + numpy only — no OpenCV dependency.--Functions:- decode(b64_str) → PIL.Image- crop(img, bbox) → PIL.Image (bbox: (x, y, w, h))- diff_ratio(img_a, img_b) → float in [0, 1]- find_template(haystack, needle, threshold) → (x, y) or None (NCC)- save_reference(img, path) → None- load_reference(path) → PIL.Image-"""--from __future__ import annotations--import base64-import io-import os-from typing import Optional, Tuple--try:- from PIL import Image, ImageChops- import numpy as np- _PIL_AVAILABLE = True-except ImportError:- _PIL_AVAILABLE = False---BBox = Tuple[int, int, int, int] # x, y, w, h---def decode(b64_str: str) -> "Image.Image":- """Decode a base64 PNG/JPEG string into a PIL Image."""- if not _PIL_AVAILABLE:- raise ImportError("Pillow is required for cv helpers: pip install pillow")- data = base64.b64decode(b64_str)- return Image.open(io.BytesIO(data)).convert("RGB")---def crop(img: "Image.Image", bbox: BBox) -> "Image.Image":- """Crop a PIL image using (x, y, w, h) bounding box."""- x, y, w, h = bbox- return img.crop((x, y, x + w, y + h))---def diff_ratio(img_a: "Image.Image", img_b: "Image.Image") -> float:- """Return fraction of pixels that differ between two same-size images.-- Returns a value in [0, 1]. 0 = identical, 1 = completely different.- """- if not _PIL_AVAILABLE:- raise ImportError("Pillow is required for cv helpers")- import numpy as np # noqa: F811-- a = np.array(img_a.resize(img_b.size), dtype=np.float32)- b = np.array(img_b, dtype=np.float32)- diff = np.abs(a - b)- changed = np.any(diff > 8, axis=-1) # threshold per-channel- return float(changed.sum()) / float(changed.size)---def find_template(- haystack: "Image.Image",- needle: "Image.Image",- threshold: float = 0.85,-) -> Optional[Tuple[int, int]]:- """Find `needle` inside `haystack` using normalised cross-correlation (NCC).-- Returns (x, y) top-left of the best match if similarity >= threshold,- else None. No OpenCV required — uses numpy FFT-based NCC.- """- if not _PIL_AVAILABLE:- raise ImportError("Pillow + numpy required for find_template")- import numpy as np-- h_arr = np.array(haystack.convert("L"), dtype=np.float32)- n_arr = np.array(needle.convert("L"), dtype=np.float32)-- H, W = h_arr.shape- th, tw = n_arr.shape-- if th > H or tw > W:- return None-- # Normalise needle- n_mean = n_arr.mean()- n_std = n_arr.std()- if n_std < 1e-6:- return None- n_norm = (n_arr - n_mean) / n_std-- # Sliding-window NCC via FFT- from numpy.fft import fft2, ifft2, fftshift-- # Zero-pad haystack to same size for FFT correlation- pad_h = H + th - 1- pad_w = W + tw - 1-- H_fft = fft2(h_arr, s=(pad_h, pad_w))- N_fft = fft2(n_norm[::-1, ::-1], s=(pad_h, pad_w))- corr = np.real(ifft2(H_fft * N_fft))-- # Local haystack std for normalisation- from numpy.lib.stride_tricks import sliding_window_view- windows = sliding_window_view(h_arr, (th, tw))- local_std = windows.reshape(-1, th * tw).std(axis=1).reshape(H - th + 1, W - tw + 1)- local_std[local_std < 1e-6] = 1.0-- # Trim correlation to valid region- corr_valid = corr[th - 1:th - 1 + (H - th + 1), tw - 1:tw - 1 + (W - tw + 1)]- ncc = corr_valid / (local_std * th * tw)-- best_idx = np.argmax(ncc)- best_score = ncc.flat[best_idx]- if best_score < threshold:- return None-- ry, rx = divmod(int(best_idx), W - tw + 1)- return (rx, ry)---def save_reference(img: "Image.Image", path: str) -> None:- """Save a PIL Image as a PNG reference for later template matching."""- os.makedirs(os.path.dirname(path), exist_ok=True)- img.save(path, "PNG")---def load_reference(path: str) -> "Image.Image":- """Load a reference PNG for template matching."""- if not _PIL_AVAILABLE:- raise ImportError("Pillow required")- return Image.open(path).convert("RGB")diff --git a/libs/cua-driver/tests/e2e/harness/driver.py b/libs/cua-driver/tests/e2e/harness/driver.pydeleted file mode 100644index 1409768f5b..0000000000--- a/libs/cua-driver/tests/e2e/harness/driver.py+++ /dev/null@@ -1,256 +0,0 @@-"""Typed MCP driver wrapper for Python e2e tests.--Wraps DriverClient with:-- Typed WindowState return value-- Helper to find a window for a given pid-- Convenient call shortcuts--Usage::-- with Driver(binary_path) as d:- state = d.get_window_state(pid, wid)- idx = state.find_element(label="Click Me")- d.click(pid, wid, element_index=idx)- state2 = d.get_window_state(pid, wid)- assert "clicks: 1" in state2.tree-"""--from __future__ import annotations--import os-import sys-import time-from dataclasses import dataclass, field-from typing import Optional--# Allow importing from the e2e root.-_HERE = os.path.dirname(os.path.abspath(__file__))-_E2E_DIR = os.path.dirname(_HERE)-sys.path.insert(0, _E2E_DIR)--from driver_client import DriverClient, default_binary_path, MCPCallError # noqa: E402---@dataclass-class WindowState:- tree: str- screenshot_b64: str- screenshot_width: int- screenshot_height: int- raw: dict = field(default_factory=dict)-- def find_element(self, label: str) -> Optional[int]:- """Return the first element index whose line contains `label`."""- import re- for line in self.tree.split("\n"):- if label in line:- m = re.search(r'\[(\d+)\]', line)- if m:- return int(m.group(1))- return None-- def find_button(self, label: str) -> Optional[int]:- """Return the first AXButton element index matching `label`."""- import re- for line in self.tree.split("\n"):- if "AXButton" not in line:- continue- if label not in line:- continue- m = re.search(r'\[(\d+)\]', line)- if m:- return int(m.group(1))- return None-- def find_text_field(self, skip_url_bar: bool = True) -> Optional[int]:- """Return the first AXTextField index (skipping Safari URL bar)."""- import re- for line in self.tree.split("\n"):- if "AXTextField" not in line:- continue- if skip_url_bar and "smart search field" in line:- continue- m = re.search(r'\[(\d+)\]', line)- if m:- return int(m.group(1))- return None-- def ax_value(self, element_index: int) -> Optional[str]:- """Return the value= attribute of the element at `element_index`."""- import re- prefix = f"[{element_index}]"- for line in self.tree.split("\n"):- if prefix not in line:- continue- m = re.search(r'value="([^"]*)"', line)- if m:- return m.group(1)- return None-- def has_text(self, text: str) -> bool:- return text in self.tree---class Driver:- """Thin typed wrapper around DriverClient."""-- def __init__(self, binary_path: Optional[str] = None) -> None:- self.binary_path = binary_path or default_binary_path()- self._client: Optional[DriverClient] = None-- def __enter__(self) -> "Driver":- self._client = DriverClient(self.binary_path).__enter__()- return self-- def __exit__(self, *exc) -> None:- if self._client is not None:- self._client.__exit__(*exc)- self._client = None-- # ── window helpers ────────────────────────────────────────────────────────-- def list_apps(self) -> list[dict]:- return self._c().call_tool("list_apps")["structuredContent"]["apps"]-- def find_app(self, bundle_id: str) -> Optional[dict]:- for app in self.list_apps():- if app.get("bundle_id") == bundle_id:- return app- return None-- def find_window(self, pid: int, min_area: float = 400.0) -> int:- """Return a usable window_id for `pid`.-- Prefers on-screen windows on the current space. Among those, picks- the one with the LARGEST area so that Electron/Chromium overlay/helper- sub-windows (which are typically tiny or zero-size) are skipped in- favour of the main content window.- """- result = self._c().call_tool("list_windows", {"pid": pid})- windows = result["structuredContent"]["windows"]- if not windows:- raise RuntimeError(f"pid {pid} has no windows")-- def _area(w: dict) -> float:- b = w.get("bounds", {})- return float(b.get("width", 0)) * float(b.get("height", 0))-- # Filter to on-screen windows with meaningful size.- candidates = [- w for w in windows- if w.get("is_on_screen") and w.get("on_current_space") is not False- and _area(w) >= min_area- ]- if not candidates:- # Fallback: all on-screen windows, then all windows.- candidates = [w for w in windows if w.get("is_on_screen")] or windows-- # Among candidates, pick the one with the largest area.- candidates.sort(key=_area, reverse=True)- return candidates[0]["window_id"]-- # ── MCP tool wrappers ─────────────────────────────────────────────────────-- def get_window_state(- self,- pid: int,- window_id: int,- query: Optional[str] = None,- ) -> WindowState:- args: dict = {"pid": pid, "window_id": window_id}- if query is not None:- args["query"] = query- result = self._c().call_tool("get_window_state", args)- sc = result.get("structuredContent", result)- tree = sc.get("tree_markdown", "")- screenshot = ""- for item in result.get("content", []):- if item.get("type") == "image":- screenshot = item.get("data", "")- break- width = sc.get("screenshot_width", 0)- height = sc.get("screenshot_height", 0)- return WindowState(- tree=tree,- screenshot_b64=screenshot,- screenshot_width=width,- screenshot_height=height,- raw=result,- )-- def click(- self,- pid: int,- window_id: int,- element_index: Optional[int] = None,- x: Optional[float] = None,- y: Optional[float] = None,- ) -> dict:- args: dict = {"pid": pid, "window_id": window_id}- if element_index is not None:- args["element_index"] = element_index- if x is not None:- args["x"] = x- if y is not None:- args["y"] = y- return self._c().call_tool("click", args)-- def double_click(- self,- pid: int,- window_id: int,- element_index: Optional[int] = None,- x: Optional[float] = None,- y: Optional[float] = None,- ) -> dict:- args: dict = {"pid": pid, "window_id": window_id}- if element_index is not None:- args["element_index"] = element_index- if x is not None:- args["x"] = x- if y is not None:- args["y"] = y- return self._c().call_tool("double_click", args)-- def type_text(self, pid: int, text: str) -> dict:- return self._c().call_tool("type_text", {"pid": pid, "text": text})-- def type_text_chars(self, pid: int, text: str) -> dict:- return self._c().call_tool("type_text_chars", {"pid": pid, "text": text})-- def press_key(self, pid: int, key: str) -> dict:- return self._c().call_tool("press_key", {"pid": pid, "key": key})-- def hotkey(self, pid: int, keys: list[str]) -> dict:- return self._c().call_tool("hotkey", {"pid": pid, "keys": keys})-- def scroll(- self,- pid: int,- window_id: int,- x: float,- y: float,- delta_x: float = 0.0,- delta_y: float = -3.0,- ) -> dict:- return self._c().call_tool("scroll", {- "pid": pid,- "window_id": window_id,- "x": x,- "y": y,- "delta_x": delta_x,- "delta_y": delta_y,- })-- def launch_app(self, bundle_id: str) -> dict:- return self._c().call_tool("launch_app", {"bundle_id": bundle_id})-- def call_tool(self, name: str, args: Optional[dict] = None) -> dict:- return self._c().call_tool(name, args or {})-- # ── internals ────────────────────────────────────────────────────────────-- def _c(self) -> DriverClient:- if self._client is None:- raise RuntimeError("Driver not started — use as context manager")- return self._clientdiff --git a/libs/cua-driver/tests/e2e/harness/monitor.py b/libs/cua-driver/tests/e2e/harness/monitor.pydeleted file mode 100644index c3469a5451..0000000000--- a/libs/cua-driver/tests/e2e/harness/monitor.py+++ /dev/null@@ -1,224 +0,0 @@-"""UX invariant monitor — polls at 5 ms via a background thread.--Signals checked on every tick:- - Real cursor position: NSEvent.mouseLocation() — violation if moved > 4 px- - Frontmost app PID: NSWorkspace.sharedWorkspace().frontmostApplication() — violation if != sentinel_pid- - Agent overlay z-order: CGWindowListCopyWindowInfo — violation if a cua-driver- overlay window layer >= frontmost non-agent window layer--Usage::-- mon = UXMonitor(sentinel_pid=focus_monitor_pid)- mon.start()- # ... run test ...- mon.stop()- mon.assert_clean() # raises AssertionError if any violation was recorded-"""--from __future__ import annotations--import threading-import time-from dataclasses import dataclass, field-from typing import Optional---# Try to import PyObjC; fall back to no-op monitor on systems without it.-try:- from AppKit import NSWorkspace- from AppKit import NSEvent- from Quartz import (- CGWindowListCopyWindowInfo,- kCGWindowListOptionOnScreenOnly,- kCGNullWindowID,- kCGWindowLayer,- kCGWindowOwnerPID,- kCGWindowName,- )- _PYOBJC_AVAILABLE = True-except ImportError:- _PYOBJC_AVAILABLE = False---# Cursor movement tolerance in logical screen pixels.-_CURSOR_TOLERANCE_PX = 4.0--# Polling interval in seconds.-_POLL_INTERVAL_S = 0.005 # 5 ms---@dataclass-class Violation:- kind: str # "cursor" | "frontmost" | "overlay_z"- detail: str- timestamp: float = field(default_factory=time.monotonic)-- def __str__(self) -> str:- return f"[{self.kind}] {self.detail} (t={self.timestamp:.3f})"---class UXMonitor:- """5 ms background poller that records UX invariant violations."""-- def __init__(- self,- sentinel_pid: Optional[int] = None,- cursor_tolerance_px: float = _CURSOR_TOLERANCE_PX,- check_cursor: bool = True,- check_frontmost: bool = True,- check_overlay_z: bool = True,- ) -> None:- self.sentinel_pid = sentinel_pid- self.cursor_tolerance_px = cursor_tolerance_px- self.check_cursor = check_cursor and _PYOBJC_AVAILABLE- self.check_frontmost = check_frontmost and _PYOBJC_AVAILABLE- self.check_overlay_z = check_overlay_z and _PYOBJC_AVAILABLE-- self._violations: list[Violation] = []- self._lock = threading.Lock()- self._stop_event = threading.Event()- self._thread: Optional[threading.Thread] = None-- # Baseline cursor position captured at start().- self._baseline_x: float = 0.0- self._baseline_y: float = 0.0-- # Grace period: only record frontmost violations after the sentinel has- # been observed as frontmost at least once. This prevents false positives- # when the monitor starts before the activation fixture runs.- self._sentinel_seen: bool = False-- # ── public API ───────────────────────────────────────────────────────────-- def start(self) -> "UXMonitor":- if not _PYOBJC_AVAILABLE:- return self- self._baseline_x, self._baseline_y = self._cursor_pos()- self._stop_event.clear()- self._thread = threading.Thread(target=self._run, daemon=True, name="ux-monitor")- self._thread.start()- return self-- def stop(self) -> None:- if self._thread is None:- return- self._stop_event.set()- self._thread.join(timeout=1.0)- self._thread = None-- def violations(self) -> list[Violation]:- with self._lock:- return list(self._violations)-- def assert_clean(self) -> None:- """Raise AssertionError if any violations were recorded."""- with self._lock:- vs = list(self._violations)- if vs:- lines = "\n".join(f" {v}" for v in vs)- raise AssertionError(f"UX violations detected ({len(vs)}):\n{lines}")-- def reset_baseline(self) -> None:- """Reset cursor baseline to current position (e.g. after intentional move)."""- if _PYOBJC_AVAILABLE:- self._baseline_x, self._baseline_y = self._cursor_pos()-- # ── internals ────────────────────────────────────────────────────────────-- def _run(self) -> None:- while not self._stop_event.wait(timeout=_POLL_INTERVAL_S):- self._tick()-- def _tick(self) -> None:- try:- if self.check_cursor:- self._check_cursor()- if self.check_frontmost and self.sentinel_pid is not None:- self._check_frontmost()- if self.check_overlay_z:- self._check_overlay_z()- except Exception:- pass # never crash the monitor thread-- def _record(self, kind: str, detail: str) -> None:- with self._lock:- self._violations.append(Violation(kind=kind, detail=detail))-- # ── signal checkers ──────────────────────────────────────────────────────-- @staticmethod- def _cursor_pos() -> tuple[float, float]:- loc = NSEvent.mouseLocation()- return float(loc.x), float(loc.y)-- def _check_cursor(self) -> None:- x, y = self._cursor_pos()- dx = x - self._baseline_x- dy = y - self._baseline_y- dist = (dx * dx + dy * dy) ** 0.5- if dist > self.cursor_tolerance_px:- self._record(- "cursor",- f"moved {dist:.1f}px from baseline "- f"({self._baseline_x:.0f},{self._baseline_y:.0f}) "- f"→ ({x:.0f},{y:.0f})",- )- # Update baseline so we don't spam the same violation.- self._baseline_x, self._baseline_y = x, y-- def _check_frontmost(self) -> None:- app = NSWorkspace.sharedWorkspace().frontmostApplication()- if app is None:- return- pid = int(app.processIdentifier())- if pid == self.sentinel_pid:- # Sentinel is frontmost — record this so we exit the grace period.- self._sentinel_seen = True- return- if not self._sentinel_seen:- # Grace period: don't flag until sentinel has been frontmost at least once.- return- name = str(app.localizedName() or "?")- self._record(- "frontmost",- f"frontmost changed to pid={pid} ({name}), "- f"expected sentinel pid={self.sentinel_pid}",- )- # Don't spam — pause checking until next start- self.check_frontmost = False-- def _check_overlay_z(self) -> None:- """Check that no cua-driver overlay window is above foreground windows."""- wins = CGWindowListCopyWindowInfo(- kCGWindowListOptionOnScreenOnly, kCGNullWindowID- )- if not wins:- return-- # Separate overlay windows from non-overlay windows- overlay_layers: list[int] = []- non_overlay_min_layer: int = 10000-- for w in wins:- pid = int(w.get(kCGWindowOwnerPID, 0))- name = str(w.get(kCGWindowName, "") or "")- layer = int(w.get(kCGWindowLayer, 0))-- is_overlay = name in ("cua-agent-cursor", "cua-overlay") or \- "cua-driver" in str(name).lower()-- if is_overlay:- overlay_layers.append(layer)- elif layer >= 0: # normal on-screen window- non_overlay_min_layer = min(non_overlay_min_layer, layer)-- if not overlay_layers:- return-- max_overlay = max(overlay_layers)- if max_overlay >= non_overlay_min_layer and non_overlay_min_layer < 10000:- self._record(- "overlay_z",- f"cua-driver overlay layer={max_overlay} >= "- f"foreground window layer={non_overlay_min_layer}",- )diff --git a/libs/cua-driver/tests/e2e/harness/tree.py b/libs/cua-driver/tests/e2e/harness/tree.pydeleted file mode 100644index adc2c4fe34..0000000000--- a/libs/cua-driver/tests/e2e/harness/tree.py+++ /dev/null@@ -1,99 +0,0 @@-"""AX tree_markdown parser and query helpers.--tree_markdown is the line-based format emitted by get_window_state, e.g.::-- - [0] AXWindow "Safari" [id=main actions=[raise]]- - [1] AXButton "Click Me" [actions=[press]]- - [2] AXTextField [value="hello"] [actions=[set_value]]- - [3] AXStaticText = "clicks: 3"--Helpers:- find(tree, role=None, label=None) → first matching element index- find_all(tree, role=None, label=None) → all matching element indices- ax_value(tree, index) → value= attribute or static text value- ax_text(tree, index) → the label / title / description text on that line-"""--from __future__ import annotations--import re-from typing import Optional---def _lines(tree: str) -> list[tuple[int, str]]:- """Return (element_index, raw_line) pairs for every indexed line."""- result: list[tuple[int, str]] = []- for line in tree.split("\n"):- m = re.search(r'\[(\d+)\]', line)- if m:- result.append((int(m.group(1)), line))- return result---def find(- tree: str,- role: Optional[str] = None,- label: Optional[str] = None,- skip_url_bar: bool = True,-) -> Optional[int]:- """Return the element_index of the first matching element."""- for idx, line in _lines(tree):- if role is not None and role not in line:- continue- if label is not None and label not in line:- continue- if skip_url_bar and "smart search field" in line:- continue- return idx- return None---def find_all(- tree: str,- role: Optional[str] = None,- label: Optional[str] = None,-) -> list[int]:- """Return all matching element indices."""- result: list[int] = []- for idx, line in _lines(tree):- if role is not None and role not in line:- continue- if label is not None and label not in line:- continue- result.append(idx)- return result---def ax_value(tree: str, index: int) -> Optional[str]:- """Return value= attribute or static text (= "...") for `index`."""- prefix = f"[{index}]"- for line in tree.split("\n"):- if prefix not in line:- continue- # value="..." attribute- m = re.search(r'value="([^"]*)"', line)- if m:- return m.group(1)- # AXStaticText = "..."- m = re.search(r'= "([^"]*)"', line)- if m:- return m.group(1)- return None---def ax_text(tree: str, index: int) -> Optional[str]:- """Return the display text (title or quoted string) for `index`."""- prefix = f"[{index}]"- for line in tree.split("\n"):- if prefix not in line:- continue- # Quoted string after role: AXButton "label"- m = re.search(r'"([^"]+)"', line)- if m:- return m.group(1)- return line.strip()- return None---def contains_text(tree: str, text: str) -> bool:- return text in treediff --git a/libs/cua-driver/tests/e2e/run_tests.sh b/libs/cua-driver/tests/e2e/run_tests.shdeleted file mode 100755index 2f98162239..0000000000--- a/libs/cua-driver/tests/e2e/run_tests.sh+++ /dev/null@@ -1,97 +0,0 @@-#!/usr/bin/env bash-# Run cua-driver Python e2e tests.-#-# Usage:-# ./run_tests.sh # run all tests (Rust binary)-# ./run_tests.sh test_list_windows # run a specific test module-# ./run_tests.sh test_cli test_api_parity # run multiple modules-# ./run_tests.sh --parity # run parity suite against BOTH binaries-#-# The script auto-builds cua-driver (debug) before running unless-# CUA_DRIVER_BINARY is already set in the environment.-#-# Options:-# -v / --verbose Pass -v to pytest for verbose output.-# --release Build with --release (sets CUA_DRIVER_BINARY to release binary).-# --parity Run test_api_parity.py against both Swift and Rust binaries.-# Requires CUA_SWIFT_BINARY or ~/.local/bin/cua-driver.-#-set -euo pipefail--SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"-CUA_DRIVER_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"-RUST_ROOT="$CUA_DRIVER_ROOT/rust"--VERBOSE=""-PROFILE="debug"-PARITY_MODE=0-MODULES=()--for arg in "$@"; do- case "$arg" in- -v|--verbose) VERBOSE="-v" ;;- --release) PROFILE="release" ;;- --parity) PARITY_MODE=1 ;;- test_*) MODULES+=("$arg") ;;- *) echo "Unknown argument: $arg"; exit 1 ;;- esac-done--# Build the binary if CUA_DRIVER_BINARY is not already set.-if [ -z "${CUA_DRIVER_BINARY:-}" ]; then- echo "==> Building cua-driver ($PROFILE)..."- CARGO_FLAGS=""- [ "$PROFILE" = "release" ] && CARGO_FLAGS="--release"- (cd "$RUST_ROOT" && cargo build $CARGO_FLAGS -p cua-driver 2>&1)- export CUA_DRIVER_BINARY="$RUST_ROOT/target/$PROFILE/cua-driver"-fi--echo "==> Using Rust binary: $CUA_DRIVER_BINARY"-if [ ! -x "$CUA_DRIVER_BINARY" ]; then- echo "ERROR: binary not found or not executable: $CUA_DRIVER_BINARY"- exit 1-fi--# Auto-detect Swift binary for parity mode.-if [ "$PARITY_MODE" -eq 1 ] && [ -z "${CUA_SWIFT_BINARY:-}" ]; then- for candidate in "$HOME/.local/bin/cua-driver" "$(which cua-driver 2>/dev/null || true)"; do- if [ -n "$candidate" ] && [ -x "$candidate" ]; then- export CUA_SWIFT_BINARY="$candidate"- break- fi- done- if [ -z "${CUA_SWIFT_BINARY:-}" ]; then- echo "WARNING: --parity requested but no Swift binary found."- echo " Install cua-driver or set CUA_SWIFT_BINARY."- echo " SwiftParityTests will be skipped."- else- echo "==> Using Swift binary: $CUA_SWIFT_BINARY"- fi-fi--cd "$SCRIPT_DIR"--if [ "$PARITY_MODE" -eq 1 ]; then- echo "==> Running parity suite against both binaries..."- python3 -m pytest ${VERBOSE} test_api_parity.py- exit $?-fi--if [ "${#MODULES[@]}" -eq 0 ]; then- # Discover all test_*.py files.- mapfile -t MODULES < <(find . -maxdepth 1 -name 'test_*.py' -exec basename {} .py \; | sort)-fi--echo "==> Running ${#MODULES[@]} test module(s): ${MODULES[*]}"-echo ""--FILES=()-for module in "${MODULES[@]}"; do- if [[ "$module" == *.py ]]; then- FILES+=("$module")- else- FILES+=("$module.py")- fi-done--python3 -m pytest ${VERBOSE} "${FILES[@]}"diff --git a/libs/cua-driver/tests/e2e/test_api_parity.py b/libs/cua-driver/tests/e2e/test_api_parity.pydeleted file mode 100644index 5413319378..0000000000--- a/libs/cua-driver/tests/e2e/test_api_parity.py+++ /dev/null@@ -1,1182 +0,0 @@-"""Binary-level API parity tests: Swift cua-driver vs Rust cua-driver-rs.--Covers every public API surface of both binaries:- • CLI subcommands (list-tools, describe, call, mcp-config, serve/stop/status,- recording, dump-docs, update, doctor, diagnose, config, --version, --help)- • Stdio MCP protocol (tools/list + every registered tool)- • Embedded daemon mode (serve socket lifecycle)--Both binaries are tested via the *same* mixin class so any divergence shows-up as a test failure attributed to the specific binary.--Environment variables:- CUA_SWIFT_BINARY – path to the Swift binary- (default: ~/.local/bin/cua-driver or `which cua-driver`)- CUA_DRIVER_BINARY – path to the Rust binary- (default: ../../target/debug/cua-driver)--Run all parity tests against both binaries::-- python3 -m unittest test_api_parity -v--Run against Rust binary only::-- CUA_DRIVER_BINARY=../../target/release/cua-driver \\- python3 -m unittest test_api_parity.RustParityTests -v--Run against Swift binary only::-- python3 -m unittest test_api_parity.SwiftParityTests -v-"""--from __future__ import annotations--import json-import os-import shutil-import subprocess-import sys-import tempfile-import time-import unittest--_HERE = os.path.dirname(os.path.abspath(__file__))-sys.path.insert(0, _HERE)--from driver_client import DriverClient, MCPCallError, default_binary_path # noqa: E402--# ── helpers ────────────────────────────────────────────────────────────────────--def _run(- args: list[str],- timeout: int = 20,- stdin: str | None = None,-) -> subprocess.CompletedProcess:- return subprocess.run(- args,- capture_output=True,- text=True,- timeout=timeout,- input=stdin,- )---def _swift_binary() -> str | None:- """Return Swift cua-driver binary path, or None if not found."""- from_env = os.environ.get("CUA_SWIFT_BINARY", "")- if from_env and os.path.isfile(from_env):- return from_env- candidates = [- os.path.expanduser("~/.local/bin/cua-driver"),- ]- found = shutil.which("cua-driver")- if found:- candidates.insert(0, found)- for p in candidates:- if os.path.isfile(p) and os.access(p, os.X_OK):- return p- return None---# ── tool presence lists ────────────────────────────────────────────────────────-#-# REQUIRED_TOOLS – tools that BOTH binaries must expose (tests fail on either-# binary when a listed tool is absent).-# RUST_ONLY_TOOLS – tools in Rust but not (yet) in Swift. Tests annotated-# with these are expected to fail on Swift.-# SWIFT_ONLY_TOOLS – tools in Swift but not (yet) registered in Rust. Tests-# annotated with these are expected to fail on Rust.-#-# Update these lists as parity gaps close.--REQUIRED_TOOLS = [- # Core enumeration- "get_screen_size",- "screenshot",- "list_apps",- "list_windows",- "get_window_state",- # Interaction- "click",- "double_click",- "right_click",- "drag",- "type_text",- "press_key",- "hotkey",- "scroll",- "set_value",- "zoom",- # App lifecycle- "launch_app",- # Permissions- "check_permissions",- # Cursor overlay- "move_cursor",- "get_cursor_position",- "set_agent_cursor_style",- "get_agent_cursor_state",- "set_agent_cursor_enabled",- "set_agent_cursor_motion",- # Config- "get_config",- "set_config",- # Recording / replay- "set_recording",- "get_recording_state",- "replay_trajectory",-]--# In Rust, not in Swift yet. Tests for these will fail on the Swift binary.-RUST_ONLY_TOOLS = [- "type_text_chars", # per-character delay typing; Swift uses type_text- "get_accessibility_tree",# lightweight desktop AX snapshot (separate from get_window_state)-]--# In Swift, not registered in Rust yet. (`page` is now cross-platform in Rust.)-SWIFT_ONLY_TOOLS: list[str] = []---# ── parity mixin ───────────────────────────────────────────────────────────────--class _ParityMixin:- """All API surface tests. Concrete subclasses set `cls.binary`."""-- binary: str = "" # set by setUpClass-- # ── CLI: version / help ───────────────────────────────────────────────────-- def test_version_flag_exits_zero(self) -> None:- """--version must exit 0 (even if the binary enters MCP mode silently)."""- r = _run([self.binary, "--version"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")-- def test_version_flag_outputs_version_number(self) -> None:- """--version must print a semver string to stdout or stderr.-- Parity gap: Rust binary currently exits 0 with no output because- '--version' is treated as an unknown bare flag and silently enters- MCP mode. This test documents the requirement; it will fail until- '--version' is handled explicitly in cli::parse_command.- """- r = _run([self.binary, "--version"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- self.assertRegex(- r.stdout + r.stderr,- r"\d+\.\d+\.\d+",- "--version did not print a semver string (N.N.N)",- )-- def test_help_flag_exits_zero(self) -> None:- r = _run([self.binary, "--help"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- self.assertIn("cua-driver", r.stdout + r.stderr)-- # ── CLI: list-tools ───────────────────────────────────────────────────────-- def test_list_tools_exits_zero(self) -> None:- r = _run([self.binary, "list-tools"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")-- def test_list_tools_contains_required_tool_names(self) -> None:- r = _run([self.binary, "list-tools"])- self.assertEqual(r.returncode, 0)- for name in [- "click", "screenshot", "list_windows", "press_key",- "get_window_state", "type_text", "hotkey", "scroll",- ]:- self.assertIn(name, r.stdout, f"list-tools missing: {name!r}")-- def test_list_tools_every_required_tool_present(self) -> None:- r = _run([self.binary, "list-tools"])- self.assertEqual(r.returncode, 0)- for name in REQUIRED_TOOLS:- self.assertIn(name, r.stdout, f"list-tools missing: {name!r}")-- # ── CLI: describe ─────────────────────────────────────────────────────────-- def test_describe_click_has_pid_and_schema(self) -> None:- r = _run([self.binary, "describe", "click"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- self.assertIn("click", r.stdout)- self.assertIn("pid", r.stdout)- self.assertIn("input_schema", r.stdout)-- def test_describe_screenshot_has_schema(self) -> None:- r = _run([self.binary, "describe", "screenshot"])- self.assertEqual(r.returncode, 0)- self.assertIn("screenshot", r.stdout)- self.assertIn("input_schema", r.stdout)-- def test_describe_each_required_tool_exits_zero(self) -> None:- for name in REQUIRED_TOOLS:- with self.subTest(tool=name):- r = _run([self.binary, "describe", name])- self.assertEqual(- r.returncode, 0,- f"describe {name!r} exited {r.returncode}: {r.stderr}",- )- self.assertIn(name, r.stdout)- self.assertIn("input_schema", r.stdout)-- def test_describe_unknown_tool_exits_64(self) -> None:- r = _run([self.binary, "describe", "no_such_tool_xyzzy"])- self.assertEqual(r.returncode, 64)- self.assertIn("no_such_tool_xyzzy", r.stderr)-- # ── CLI: call (stateless tools) ───────────────────────────────────────────-- def test_call_get_screen_size_exits_zero(self) -> None:- r = _run([self.binary, "call", "get_screen_size"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")-- def test_call_get_screen_size_returns_json(self) -> None:- """call get_screen_size must emit JSON with width and height.-- Parity gap: Swift binary outputs human-readable text- (e.g. "✅ Main display: 1920x1080 points @ 1.0x") instead of JSON.- This test documents the requirement; it fails on Swift.- """- r = _run([self.binary, "call", "get_screen_size"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- try:- data = json.loads(r.stdout)- except json.JSONDecodeError:- self.fail(- f"call get_screen_size did not output JSON.\n"- f"stdout: {r.stdout!r}\n"- f"(Swift binary outputs human-readable text instead of JSON — parity gap)"- )- self.assertIn("width", data)- self.assertIn("height", data)- self.assertGreater(data["width"], 0)- self.assertGreater(data["height"], 0)-- def test_call_screenshot_returns_b64_png(self) -> None:- """call screenshot (no args) must return JSON with screenshot_png_b64.-- Parity gap: Swift binary requires `window_id` for screenshot and- returns a non-JSON error when called without args. The Rust binary- defaults to a full-display screenshot without requiring window_id.- This test fails on Swift on both counts; it documents the requirement.- """- r = _run([self.binary, "call", "screenshot"], timeout=30)- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}\nstdout: {r.stdout!r}")- try:- data = json.loads(r.stdout)- except json.JSONDecodeError:- self.fail(- f"call screenshot did not output JSON.\n"- f"stdout: {r.stdout!r}\n"- f"(Swift requires window_id; Rust defaults to full display)"- )- self.assertIn("screenshot_png_b64", data)- self.assertGreater(len(data["screenshot_png_b64"]), 500)-- def test_call_list_apps_returns_app_list(self) -> None:- r = _run([self.binary, "call", "list_apps"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- data = json.loads(r.stdout)- self.assertIn("apps", data)- self.assertIsInstance(data["apps"], list)- self.assertGreater(len(data["apps"]), 0)-- def test_call_list_apps_entries_have_required_fields(self) -> None:- r = _run([self.binary, "call", "list_apps"])- self.assertEqual(r.returncode, 0)- data = json.loads(r.stdout)- for app in data["apps"][:5]: # spot-check first 5- with self.subTest(app=app.get("name") or app.get("bundle_id")):- self.assertIn("pid", app)- self.assertIn("name", app)-- def test_call_list_windows_returns_window_list(self) -> None:- r = _run([self.binary, "call", "list_windows"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- data = json.loads(r.stdout)- self.assertIn("windows", data)- self.assertIsInstance(data["windows"], list)-- def test_call_list_windows_entries_have_required_fields(self) -> None:- r = _run([self.binary, "call", "list_windows"])- self.assertEqual(r.returncode, 0)- data = json.loads(r.stdout)- for w in data["windows"][:5]:- with self.subTest(window_id=w.get("window_id")):- self.assertIn("window_id", w)- self.assertIn("pid", w)- self.assertIn("bounds", w)- b = w["bounds"]- for field in ("x", "y", "width", "height"):- self.assertIn(field, b)-- def test_call_check_permissions_exits_zero(self) -> None:- r = _run([self.binary, "call", "check_permissions"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")-- def test_call_check_permissions_returns_accessibility_and_screen_recording(- self,- ) -> None:- """call check_permissions must emit JSON with accessibility/screen_recording.-- Parity gap: Swift binary outputs human-readable text with emoji- (✅/❌) instead of JSON. This test documents the JSON requirement- and will fail on Swift until its CLI output is standardised.- """- r = _run([self.binary, "call", "check_permissions"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- try:- data = json.loads(r.stdout)- except json.JSONDecodeError:- self.fail(- f"call check_permissions did not output JSON.\n"- f"stdout: {r.stdout!r}"- )- self.assertIn("accessibility", data)- self.assertIn("screen_recording", data)-- def test_call_get_cursor_position_returns_xy(self) -> None:- r = _run([self.binary, "call", "get_cursor_position"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- data = json.loads(r.stdout)- self.assertIn("x", data)- self.assertIn("y", data)-- def test_call_get_config_returns_config_object(self) -> None:- r = _run([self.binary, "call", "get_config"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- data = json.loads(r.stdout)- # Config must have at least one of the known top-level keys.- known_keys = {- "schema_version", "capture_mode", "max_image_dimension",- "agent_cursor", "telemetry_enabled",- }- self.assertTrue(- known_keys.intersection(data.keys()),- f"get_config returned no known keys: {list(data.keys())}",- )-- def test_call_get_agent_cursor_state_returns_state(self) -> None:- r = _run([self.binary, "call", "get_agent_cursor_state"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- data = json.loads(r.stdout)- self.assertIn("enabled", data)-- def test_call_get_recording_state_returns_state(self) -> None:- r = _run([self.binary, "call", "get_recording_state"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- data = json.loads(r.stdout)- self.assertIn("recording", data)-- # ── CLI: call with a pid-bearing tool (launch_app + get_window_state) ─────-- def test_call_launch_app_calculator_returns_pid(self) -> None:- subprocess.run(["pkill", "-x", "Calculator"], check=False)- time.sleep(0.3)- r = _run(- [self.binary, "call", "launch_app", '{"bundle_id": "com.apple.calculator"}'],- timeout=20,- )- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- data = json.loads(r.stdout)- self.assertIn("pid", data)- self.assertGreater(data["pid"], 0)-- def test_call_get_window_state_for_calculator(self) -> None:- """launch Calculator, get its window, then fetch get_window_state."""- # Launch- rl = _run(- [self.binary, "call", "launch_app", '{"bundle_id": "com.apple.calculator"}'],- timeout=20,- )- self.assertEqual(rl.returncode, 0, f"launch stderr: {rl.stderr}")- pid = json.loads(rl.stdout)["pid"]- time.sleep(1.0)-- # List windows to get a window_id.- rw = _run(- [self.binary, "call", "list_windows", f'{{"pid": {pid}}}'],- )- self.assertEqual(rw.returncode, 0, f"list_windows stderr: {rw.stderr}")- windows = json.loads(rw.stdout)["windows"]- self.assertTrue(windows, "Calculator has no windows")- window_id = windows[0]["window_id"]-- # get_window_state- rs = _run(- [self.binary, "call", "get_window_state",- f'{{"pid": {pid}, "window_id": {window_id}}}'],- timeout=30,- )- self.assertEqual(rs.returncode, 0, f"get_window_state stderr: {rs.stderr}")- data = json.loads(rs.stdout)- self.assertIn("tree_markdown", data)- self.assertIn("AX", data["tree_markdown"])-- # ── CLI: call — error / missing args paths ────────────────────────────────-- def test_call_unknown_tool_exits_64(self) -> None:- r = _run([self.binary, "call", "no_such_tool_xyzzy"])- self.assertEqual(r.returncode, 64)-- def test_call_press_key_missing_pid_exits_1(self) -> None:- r = _run([self.binary, "call", "press_key", '{"key": "a"}'])- self.assertEqual(r.returncode, 1, f"stdout: {r.stdout}")-- def test_call_click_missing_required_args_exits_1(self) -> None:- r = _run([self.binary, "call", "click", '{}'])- self.assertEqual(r.returncode, 1, f"stdout: {r.stdout}")-- def test_call_type_text_missing_pid_exits_1(self) -> None:- r = _run([self.binary, "call", "type_text", '{"text": "hello"}'])- self.assertEqual(r.returncode, 1, f"stdout: {r.stdout}")-- def test_call_hotkey_missing_pid_exits_1(self) -> None:- r = _run([self.binary, "call", "hotkey", '{"keys": ["cmd", "a"]}'])- self.assertEqual(r.returncode, 1, f"stdout: {r.stdout}")-- def test_call_scroll_missing_required_args_exits_1(self) -> None:- r = _run([self.binary, "call", "scroll", '{}'])- self.assertEqual(r.returncode, 1, f"stdout: {r.stdout}")-- def test_call_drag_missing_required_args_exits_1(self) -> None:- r = _run([self.binary, "call", "drag", '{}'])- self.assertEqual(r.returncode, 1, f"stdout: {r.stdout}")-- def test_call_get_window_state_missing_pid_exits_1(self) -> None:- r = _run([self.binary, "call", "get_window_state", '{"window_id": 1}'])- self.assertEqual(r.returncode, 1, f"stdout: {r.stdout}")-- def test_call_launch_app_no_args_exits_1(self) -> None:- r = _run([self.binary, "call", "launch_app", '{}'])- self.assertEqual(r.returncode, 1, f"stdout: {r.stdout}")-- # ── CLI: implicit call (tool name as first positional) ────────────────────-- def test_implicit_call_get_screen_size(self) -> None:- r = _run([self.binary, "get_screen_size"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- data = json.loads(r.stdout)- self.assertIn("width", data)- self.assertIn("height", data)-- def test_implicit_call_check_permissions(self) -> None:- r = _run([self.binary, "check_permissions"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- data = json.loads(r.stdout)- self.assertIn("accessibility", data)-- def test_implicit_call_list_apps(self) -> None:- r = _run([self.binary, "list_apps"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- data = json.loads(r.stdout)- self.assertIn("apps", data)-- def test_implicit_call_screenshot_exits_zero(self) -> None:- """Implicit-call screenshot must exit 0.-- Note: Swift binary requires window_id so exits non-zero with an- error message — this is a parity gap in Swift.- """- r = _run([self.binary, "screenshot"], timeout=30)- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}\nstdout: {r.stdout!r}")-- def test_implicit_unknown_exits_64(self) -> None:- r = _run([self.binary, "no_such_tool_xyzzy"])- self.assertEqual(r.returncode, 64)-- # ── CLI: mcp-config ───────────────────────────────────────────────────────-- def test_mcp_config_default_outputs_json(self) -> None:- r = _run([self.binary, "mcp-config"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- out = r.stdout.strip()- # Should either be a JSON object or a shell snippet containing "cua-driver"- self.assertIn("cua-driver", out)-- def test_mcp_config_client_claude_outputs_add_command(self) -> None:- r = _run([self.binary, "mcp-config", "--client", "claude"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- self.assertIn("cua-driver", r.stdout)-- def test_mcp_config_client_cursor_exits_zero(self) -> None:- r = _run([self.binary, "mcp-config", "--client", "cursor"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")-- # ── CLI: dump-docs ────────────────────────────────────────────────────────-- def test_dump_docs_exits_zero(self) -> None:- r = _run([self.binary, "dump-docs"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- self.assertTrue(len(r.stdout) > 200, "dump-docs output too short")-- def test_dump_docs_contains_tool_names(self) -> None:- r = _run([self.binary, "dump-docs"])- self.assertEqual(r.returncode, 0)- for name in ["click", "screenshot", "get_window_state"]:- self.assertIn(name, r.stdout)-- def test_dump_docs_pretty_is_valid_json(self) -> None:- r = _run([self.binary, "dump-docs", "--pretty"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- # --pretty outputs JSON- try:- data = json.loads(r.stdout)- except json.JSONDecodeError:- self.fail(f"dump-docs --pretty is not valid JSON:\n{r.stdout[:500]}")- self.assertIsInstance(data, (list, dict))-- # ── CLI: diagnose ─────────────────────────────────────────────────────────-- def test_diagnose_exits_zero(self) -> None:- r = _run([self.binary, "diagnose"], timeout=30)- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- self.assertTrue(len(r.stdout) > 50)-- def test_diagnose_mentions_accessibility(self) -> None:- r = _run([self.binary, "diagnose"], timeout=30)- self.assertEqual(r.returncode, 0)- out = (r.stdout + r.stderr).lower()- self.assertIn("access", out)-- # ── CLI: doctor ───────────────────────────────────────────────────────────-- def test_doctor_exits_zero_or_one(self) -> None:- """doctor returns 0 if all checks pass, 1 if any check fails."""- r = _run([self.binary, "doctor"], timeout=20)- self.assertIn(r.returncode, (0, 1), f"unexpected exit: {r.returncode}")-- def test_doctor_produces_output(self) -> None:- r = _run([self.binary, "doctor"], timeout=20)- self.assertIn(r.returncode, (0, 1))- out = r.stdout + r.stderr- self.assertGreater(len(out.strip()), 0)-- # ── CLI: config ───────────────────────────────────────────────────────────-- def test_config_show_exits_zero(self) -> None:- r = _run([self.binary, "config", "show"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- self.assertGreater(len(r.stdout.strip()), 0)-- def test_config_get_max_image_dimension(self) -> None:- r = _run([self.binary, "config", "get", "max_image_dimension"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- # Value should look like an integer- out = r.stdout.strip()- self.assertRegex(out, r"\d+", f"expected integer, got: {out!r}")-- def test_config_get_agent_cursor_enabled(self) -> None:- r = _run([self.binary, "config", "get", "agent_cursor.enabled"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- val = r.stdout.strip().lower()- self.assertIn(val, ("true", "false"), f"unexpected value: {val!r}")-- def test_config_set_and_restore_max_image_dimension(self) -> None:- # Read original value.- r0 = _run([self.binary, "config", "get", "max_image_dimension"])- self.assertEqual(r0.returncode, 0)- original = r0.stdout.strip()-- # Set a new value.- r1 = _run([self.binary, "config", "set", "max_image_dimension", "1024"])- self.assertEqual(r1.returncode, 0, f"set stderr: {r1.stderr}")-- # Confirm it was applied.- r2 = _run([self.binary, "config", "get", "max_image_dimension"])- self.assertEqual(r2.returncode, 0)- self.assertEqual(r2.stdout.strip(), "1024")-- # Restore original value.- _run([self.binary, "config", "set", "max_image_dimension", original])-- def test_config_reset_exits_zero(self) -> None:- r = _run([self.binary, "config", "reset"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")-- def test_config_get_unknown_key_exits_nonzero(self) -> None:- r = _run([self.binary, "config", "get", "no_such_key_xyzzy"])- self.assertNotEqual(r.returncode, 0)-- # ── CLI: update ───────────────────────────────────────────────────────────-- def test_update_dry_run_exits_zero(self) -> None:- """update without --apply should exit 0 even if already up-to-date."""- r = _run([self.binary, "update"], timeout=30)- self.assertIn(r.returncode, (0, 1), f"unexpected exit: {r.returncode}")- out = r.stdout + r.stderr- # Should mention version or "up to date" or similar- self.assertGreater(len(out.strip()), 0)-- # ── CLI: serve / stop / status daemon lifecycle ───────────────────────────-- def _tmp_socket(self) -> str:- return tempfile.mktemp(suffix=".sock", prefix="cua-parity-test-")-- def test_status_exits_1_when_no_daemon(self) -> None:- sock = self._tmp_socket()- r = _run([self.binary, "status", "--socket", sock])- self.assertEqual(r.returncode, 1)- out = (r.stdout + r.stderr).lower()- self.assertIn("not running", out)-- def test_stop_exits_1_when_no_daemon(self) -> None:- sock = self._tmp_socket()- r = _run([self.binary, "stop", "--socket", sock])- self.assertEqual(r.returncode, 1)-- def test_serve_status_stop_lifecycle(self) -> None:- sock = self._tmp_socket()- proc = subprocess.Popen(- [self.binary, "serve", "--socket", sock],- stdout=subprocess.PIPE,- stderr=subprocess.PIPE,- )- try:- # Wait for daemon to bind (up to 5 s).- deadline = time.monotonic() + 5.0- running = False- while time.monotonic() < deadline:- r = _run([self.binary, "status", "--socket", sock])- if r.returncode == 0:- running = True- break- time.sleep(0.1)- self.assertTrue(running, "daemon did not become ready in 5 s")-- # status reports "running" and mentions the socket path.- r = _run([self.binary, "status", "--socket", sock])- self.assertEqual(r.returncode, 0, f"status stderr: {r.stderr}")- out = (r.stdout + r.stderr).lower()- self.assertIn("running", out)-- # stop the daemon.- r = _run([self.binary, "stop", "--socket", sock])- self.assertEqual(r.returncode, 0, f"stop stderr: {r.stderr}")-- # status should report not running.- time.sleep(0.3)- r = _run([self.binary, "status", "--socket", sock])- self.assertEqual(r.returncode, 1)- finally:- proc.terminate()- proc.wait(timeout=3)-- def test_serve_second_instance_exits_1(self) -> None:- sock = self._tmp_socket()- proc = subprocess.Popen(- [self.binary, "serve", "--socket", sock],- stdout=subprocess.PIPE,- stderr=subprocess.PIPE,- )- try:- # Wait for first daemon.- deadline = time.monotonic() + 5.0- while time.monotonic() < deadline:- r = _run([self.binary, "status", "--socket", sock])- if r.returncode == 0:- break- time.sleep(0.1)-- # Second serve should fail.- r = _run([self.binary, "serve", "--socket", sock])- self.assertEqual(r.returncode, 1, f"stdout: {r.stdout}\nstderr: {r.stderr}")- out = (r.stdout + r.stderr).lower()- self.assertIn("already running", out)- finally:- _run([self.binary, "stop", "--socket", sock])- proc.wait(timeout=3)-- # ── stdio MCP: tools/list ─────────────────────────────────────────────────-- def _mcp(self) -> DriverClient:- return DriverClient(self.binary)-- def test_mcp_tools_list_returns_all_required_tools(self) -> None:- with self._mcp() as c:- tools = c.list_tools()- names = {t["name"] for t in tools}- missing = [n for n in REQUIRED_TOOLS if n not in names]- self.assertFalse(- missing,- f"tools/list missing: {missing}\n got: {sorted(names)}",- )-- def test_mcp_tools_list_each_has_name_description_schema(self) -> None:- with self._mcp() as c:- tools = c.list_tools()- for t in tools:- with self.subTest(tool=t.get("name")):- self.assertIn("name", t)- self.assertIn("description", t)- self.assertIn("inputSchema", t)- schema = t["inputSchema"]- self.assertIn("type", schema)- self.assertEqual(schema["type"], "object")-- # ── stdio MCP: stateless read tools ──────────────────────────────────────-- def test_mcp_get_screen_size(self) -> None:- with self._mcp() as c:- result = c.call_tool("get_screen_size")- sc = result.get("structuredContent", result)- self.assertIn("width", sc)- self.assertIn("height", sc)- self.assertGreater(sc["width"], 0)- self.assertGreater(sc["height"], 0)-- def test_mcp_screenshot_has_png_image(self) -> None:- with self._mcp() as c:- result = c.call_tool("screenshot")- # Should have image content.- images = [i for i in result.get("content", []) if i.get("type") == "image"]- sc = result.get("structuredContent", {})- # At least one of: image in content OR screenshot_png_b64 in structuredContent.- has_image = bool(images) or bool(sc.get("screenshot_png_b64"))- self.assertTrue(has_image, f"screenshot returned no image: {result}")-- def test_mcp_screenshot_with_quality(self) -> None:- with self._mcp() as c:- result = c.call_tool("screenshot", {"quality": 50})- images = [i for i in result.get("content", []) if i.get("type") == "image"]- sc = result.get("structuredContent", {})- has_image = bool(images) or bool(sc.get("screenshot_png_b64"))- self.assertTrue(has_image, "screenshot with quality returned no image")-- def test_mcp_list_apps(self) -> None:- with self._mcp() as c:- result = c.call_tool("list_apps")- sc = result["structuredContent"]- self.assertIn("apps", sc)- apps = sc["apps"]- self.assertIsInstance(apps, list)- self.assertGreater(len(apps), 0)- # Every app entry must have pid + name.- for app in apps[:5]:- self.assertIn("pid", app)- self.assertIn("name", app)-- def test_mcp_list_apps_has_active_flag(self) -> None:- with self._mcp() as c:- result = c.call_tool("list_apps")- apps = result["structuredContent"]["apps"]- # At least one app should have an active field.- has_active = any("active" in app for app in apps)- self.assertTrue(has_active, "no app has 'active' field")-- def test_mcp_list_windows_default(self) -> None:- with self._mcp() as c:- result = c.call_tool("list_windows")- sc = result["structuredContent"]- self.assertIn("windows", sc)- windows = sc["windows"]- self.assertIsInstance(windows, list)-- def test_mcp_list_windows_fields(self) -> None:- with self._mcp() as c:- result = c.call_tool("list_windows")- windows = result["structuredContent"]["windows"]- for w in windows[:5]:- with self.subTest(window_id=w.get("window_id")):- self.assertIn("window_id", w)- self.assertIn("pid", w)- self.assertIn("bounds", w)- for f in ("x", "y", "width", "height"):- self.assertIn(f, w["bounds"])-- def test_mcp_list_windows_pid_filter(self) -> None:- """list_windows with pid filter should only return windows for that pid."""- with self._mcp() as c:- # Get any pid that has windows.- all_wins = c.call_tool("list_windows")["structuredContent"]["windows"]- if not all_wins:- self.skipTest("no windows available")- target_pid = all_wins[0]["pid"]- filtered = c.call_tool("list_windows", {"pid": target_pid})- windows = filtered["structuredContent"]["windows"]- for w in windows:- self.assertEqual(w["pid"], target_pid)-- def test_mcp_list_windows_on_screen_only(self) -> None:- with self._mcp() as c:- result = c.call_tool("list_windows", {"on_screen_only": True})- windows = result["structuredContent"]["windows"]- for w in windows:- self.assertTrue(- w.get("is_on_screen", True),- f"on_screen_only=true returned off-screen window: {w}",- )-- def test_mcp_check_permissions(self) -> None:- with self._mcp() as c:- result = c.call_tool("check_permissions")- sc = result.get("structuredContent", result)- self.assertIn("accessibility", sc)- self.assertIn("screen_recording", sc)- self.assertIsInstance(sc["accessibility"], bool)- self.assertIsInstance(sc["screen_recording"], bool)-- def test_mcp_get_cursor_position(self) -> None:- with self._mcp() as c:- result = c.call_tool("get_cursor_position")- sc = result.get("structuredContent", result)- self.assertIn("x", sc)- self.assertIn("y", sc)-- def test_mcp_get_agent_cursor_state(self) -> None:- with self._mcp() as c:- result = c.call_tool("get_agent_cursor_state")- sc = result.get("structuredContent", result)- self.assertIn("enabled", sc)-- def test_mcp_set_agent_cursor_enabled_toggle(self) -> None:- with self._mcp() as c:- # Read current.- state = c.call_tool("get_agent_cursor_state")- enabled = state.get("structuredContent", state).get("enabled", True)- # Toggle off.- c.call_tool("set_agent_cursor_enabled", {"enabled": False})- state2 = c.call_tool("get_agent_cursor_state")- self.assertFalse(state2.get("structuredContent", state2).get("enabled", True))- # Restore.- c.call_tool("set_agent_cursor_enabled", {"enabled": enabled})-- def test_mcp_set_agent_cursor_style_default(self) -> None:- with self._mcp() as c:- # Calling with no args or default args should succeed.- c.call_tool("set_agent_cursor_style", {})-- def test_mcp_get_config_roundtrip(self) -> None:- with self._mcp() as c:- result = c.call_tool("get_config")- sc = result.get("structuredContent", result)- known = {"schema_version", "capture_mode", "max_image_dimension",- "agent_cursor", "telemetry_enabled"}- self.assertTrue(known.intersection(sc.keys()),- f"get_config has no known keys: {list(sc.keys())}")-- def test_mcp_set_config_max_image_dimension(self) -> None:- with self._mcp() as c:- # Read original.- orig = c.call_tool("get_config")- orig_dim = orig.get("structuredContent", orig).get("max_image_dimension", 1920)- # Set new value.- c.call_tool("set_config", {"max_image_dimension": 800})- # Confirm.- updated = c.call_tool("get_config")- new_dim = updated.get("structuredContent", updated).get("max_image_dimension")- self.assertEqual(new_dim, 800)- # Restore.- c.call_tool("set_config", {"max_image_dimension": orig_dim})-- def test_mcp_get_recording_state(self) -> None:- with self._mcp() as c:- result = c.call_tool("get_recording_state")- sc = result.get("structuredContent", result)- self.assertIn("recording", sc)- self.assertIsInstance(sc["recording"], bool)-- def test_mcp_set_recording_enable_disable(self) -> None:- import tempfile- with tempfile.TemporaryDirectory() as tmpdir:- with self._mcp() as c:- # Enable recording — output_dir is required.- c.call_tool("set_recording", {"enabled": True, "output_dir": tmpdir})- state = c.call_tool("get_recording_state")- sc = state.get("structuredContent", state)- # Accept either "recording" (Swift) or "enabled" (Rust) key.- is_recording = sc.get("recording", sc.get("enabled", False))- self.assertTrue(is_recording, f"recording should be enabled: {sc}")- # Disable.- c.call_tool("set_recording", {"enabled": False})- state2 = c.call_tool("get_recording_state")- sc2 = state2.get("structuredContent", state2)- is_recording2 = sc2.get("recording", sc2.get("enabled", True))- self.assertFalse(is_recording2, f"recording should be disabled: {sc2}")-- # ── stdio MCP: tool error contracts ──────────────────────────────────────-- def _assert_tool_raises_mcp_error(- self, tool_name: str, args: dict, timeout: float = 10.0- ) -> None:- """Assert that calling `tool_name` with `args` signals an error.-- Accepts either a JSON-RPC MCPCallError (Swift behaviour) or a result- with ``isError=true`` in the tool-content list (Rust behaviour).- """- with self._mcp() as c:- try:- result = c.call_tool(tool_name, args)- # Rust returns isError:true in the content payload- self.assertTrue(- result.get("isError"),- f"{tool_name} did not raise an error (isError missing or false)",- )- except MCPCallError:- pass # Swift-style JSON-RPC error — acceptable-- def test_mcp_click_missing_pid_raises_error(self) -> None:- self._assert_tool_raises_mcp_error("click", {"window_id": 1})-- def test_mcp_double_click_missing_pid_raises_error(self) -> None:- self._assert_tool_raises_mcp_error("double_click", {"window_id": 1})-- def test_mcp_right_click_missing_pid_raises_error(self) -> None:- self._assert_tool_raises_mcp_error("right_click", {"window_id": 1})-- def test_mcp_press_key_missing_pid_raises_error(self) -> None:- self._assert_tool_raises_mcp_error("press_key", {"key": "a"})-- def test_mcp_hotkey_missing_pid_raises_error(self) -> None:- self._assert_tool_raises_mcp_error("hotkey", {"keys": ["cmd", "a"]})-- def test_mcp_type_text_missing_pid_raises_error(self) -> None:- self._assert_tool_raises_mcp_error("type_text", {"text": "hello"})-- def test_mcp_type_text_chars_missing_pid_raises_error(self) -> None:- self._assert_tool_raises_mcp_error("type_text_chars", {"text": "hello"})-- def test_mcp_scroll_missing_pid_raises_error(self) -> None:- self._assert_tool_raises_mcp_error("scroll", {"x": 100, "y": 100})-- def test_mcp_drag_missing_pid_raises_error(self) -> None:- self._assert_tool_raises_mcp_error("drag", {})-- def test_mcp_get_window_state_missing_pid_raises_error(self) -> None:- self._assert_tool_raises_mcp_error("get_window_state", {"window_id": 1})-- def test_mcp_launch_app_missing_args_raises_error(self) -> None:- self._assert_tool_raises_mcp_error("launch_app", {})-- def test_mcp_set_value_missing_pid_raises_error(self) -> None:- self._assert_tool_raises_mcp_error(- "set_value", {"element_index": 0, "value": "x"}- )-- def test_mcp_replay_trajectory_bad_dir_raises_error(self) -> None:- self._assert_tool_raises_mcp_error(- "replay_trajectory", {"dir": "/tmp/no_such_trajectory_xyzzy"}- )-- # ── stdio MCP: get_window_state live ─────────────────────────────────────-- def test_mcp_get_window_state_calculator(self) -> None:- """Launch Calculator, capture a window state, verify tree_markdown."""- subprocess.run(["pkill", "-x", "Calculator"], check=False)- time.sleep(0.3)-- with self._mcp() as c:- # Launch- launch = c.call_tool("launch_app", {"bundle_id": "com.apple.calculator"})- pid = launch.get("structuredContent", launch).get("pid", 0)- self.assertGreater(pid, 0, f"launch_app returned bad pid: {launch}")- time.sleep(1.0)-- # Find window- wins = c.call_tool("list_windows", {"pid": pid})["structuredContent"]["windows"]- self.assertTrue(wins, "Calculator has no windows")- window_id = wins[0]["window_id"]-- # get_window_state- state = c.call_tool("get_window_state", {"pid": pid, "window_id": window_id})-- sc = state.get("structuredContent", state)- tree = sc.get("tree_markdown", "")- self.assertIn("AX", tree, f"tree_markdown has no AX nodes:\n{tree}")- # Screenshot should be present.- images = [i for i in state.get("content", []) if i.get("type") == "image"]- self.assertTrue(images, "get_window_state returned no screenshot")-- def test_mcp_get_window_state_has_required_sc_keys(self) -> None:- subprocess.run(["pkill", "-x", "Calculator"], check=False)- time.sleep(0.3)- with self._mcp() as c:- launch = c.call_tool("launch_app", {"bundle_id": "com.apple.calculator"})- pid = launch.get("structuredContent", launch).get("pid", 0)- time.sleep(1.0)- wins = c.call_tool("list_windows", {"pid": pid})["structuredContent"]["windows"]- if not wins:- self.skipTest("Calculator has no windows")- state = c.call_tool("get_window_state", {- "pid": pid, "window_id": wins[0]["window_id"]- })- sc = state.get("structuredContent", state)- for key in ("tree_markdown", "screenshot_width", "screenshot_height"):- self.assertIn(key, sc, f"get_window_state structuredContent missing: {key!r}")-- # ── stdio MCP: launch_app ─────────────────────────────────────────────────-- def test_mcp_launch_app_by_bundle_id(self) -> None:- subprocess.run(["pkill", "-x", "Calculator"], check=False)- time.sleep(0.3)- with self._mcp() as c:- result = c.call_tool("launch_app", {"bundle_id": "com.apple.calculator"})- sc = result.get("structuredContent", result)- self.assertIn("pid", sc)- self.assertGreater(sc["pid"], 0)-- def test_mcp_launch_app_by_name(self) -> None:- subprocess.run(["pkill", "-x", "Calculator"], check=False)- time.sleep(0.3)- with self._mcp() as c:- result = c.call_tool("launch_app", {"name": "Calculator"})- sc = result.get("structuredContent", result)- self.assertIn("pid", sc)- self.assertGreater(sc["pid"], 0)-- def test_mcp_launch_app_unknown_bundle_id_raises_error(self) -> None:- self._assert_tool_raises_mcp_error(- "launch_app", {"bundle_id": "com.example.no_such_app_xyzzy"}- )-- # ── stdio MCP: set_agent_cursor_motion ────────────────────────────────────-- def test_mcp_set_agent_cursor_motion_spring(self) -> None:- with self._mcp() as c:- c.call_tool("set_agent_cursor_motion", {"spring": True})-- def test_mcp_set_agent_cursor_motion_arc(self) -> None:- with self._mcp() as c:- c.call_tool("set_agent_cursor_motion", {"arc_size": 0.5})-- # ── stdio MCP: zoom ───────────────────────────────────────────────────────-- def test_mcp_zoom_missing_pid_raises_error(self) -> None:- self._assert_tool_raises_mcp_error(- "zoom", {"x1": 0, "y1": 0, "x2": 100, "y2": 100}- )-- # ── stdio MCP: page ───────────────────────────────────────────────────────-- def test_mcp_page_is_registered(self) -> None:- """'page' tool must appear in tools/list.-- Parity gap: Rust binary has page.rs implemented but the tool is not- yet registered in platform_macos::tools::register_all. This test will- fail on the Rust binary until page::PageTool is added to register_all.- """- with self._mcp() as c:- names = {t["name"] for t in c.list_tools()}- self.assertIn("page", names, "'page' tool is not registered")-- def test_mcp_page_missing_pid_raises_error(self) -> None:- self._assert_tool_raises_mcp_error(- "page", {"action": "get_text"}- )-- # ── stdio MCP: Rust-only tools (missing in Swift) ────────────────────────-- def test_mcp_rust_only_tools_present(self) -> None:- """Rust-only tools (type_text_chars, get_accessibility_tree).-- These are in the Rust binary but not yet in Swift. Both binaries- should eventually have them. This test fails on Swift until ported.- """- with self._mcp() as c:- names = {t["name"] for t in c.list_tools()}- missing = [n for n in RUST_ONLY_TOOLS if n not in names]- self.assertFalse(- missing,- f"Rust-only tools missing: {missing} (not yet in this binary)",- )-- def test_mcp_type_text_chars_missing_pid_raises_error(self) -> None:- """type_text_chars (Rust-only) must raise MCP error with missing pid.-- Fails on Swift binary — tool not yet ported.- """- self._assert_tool_raises_mcp_error("type_text_chars", {"text": "hello"})-- def test_mcp_get_accessibility_tree_exits_without_pid(self) -> None:- """get_accessibility_tree (Rust-only) returns desktop snapshot without pid.-- Fails on Swift binary — tool not yet ported.- """- with self._mcp() as c:- result = c.call_tool("get_accessibility_tree")- sc = result.get("structuredContent", result)- # Should return some content — desktop always has windows.- self.assertIsInstance(sc, dict)-- # ── stdio MCP: Swift-only tools (missing/unregistered in Rust) ───────────-- def test_mcp_swift_only_tools_present(self) -> None:- """Swift-only tools list — currently empty after the page-tool- unification in Rust. Kept as a no-op so the matrix stays symmetric- with `test_mcp_rust_only_tools_present`.- """- with self._mcp() as c:- names = {t["name"] for t in c.list_tools()}- missing = [n for n in SWIFT_ONLY_TOOLS if n not in names]- self.assertFalse(- missing,- f"Swift-only tools missing: {missing} (not yet registered in this binary)",- )-- # ── stdio MCP: protocol-level contracts ──────────────────────────────────-- def test_mcp_tools_call_unknown_tool_returns_error(self) -> None:- """Calling an unknown tool must signal an error.-- Accepts MCPCallError (Swift / JSON-RPC error) or isError=true (Rust).- """- with self._mcp() as c:- try:- result = c.call_tool("no_such_tool_xyzzy")- self.assertTrue(- result.get("isError"),- "unknown tool should set isError=true",- )- except MCPCallError:- pass # JSON-RPC level error — acceptable-- def test_mcp_handles_rapid_sequential_calls(self) -> None:- """Send 5 sequential get_screen_size calls — all must succeed."""- with self._mcp() as c:- for i in range(5):- result = c.call_tool("get_screen_size")- sc = result.get("structuredContent", result)- self.assertIn("width", sc, f"call {i} failed: {result}")-- def test_mcp_protocol_version_echo(self) -> None:- """The initialize response should echo an MCP protocol version."""- # The DriverClient handshake already calls initialize; if it doesn't- # raise an exception, the server replied correctly.- with self._mcp() as c:- tools = c.list_tools()- self.assertGreater(len(tools), 0)---# ── concrete test classes (one per binary) ────────────────────────────────────--class SwiftParityTests(_ParityMixin, unittest.TestCase):- """Run the full parity suite against the Swift cua-driver binary."""-- @classmethod- def setUpClass(cls) -> None:- path = _swift_binary()- if not path:- raise unittest.SkipTest(- "Swift cua-driver not found. "- "Set CUA_SWIFT_BINARY or install cua-driver to ~/.local/bin/."- )- cls.binary = path- print(f"\n[SwiftParityTests] binary: {cls.binary}", flush=True)---class RustParityTests(_ParityMixin, unittest.TestCase):- """Run the full parity suite against the Rust cua-driver-rs binary."""-- @classmethod- def setUpClass(cls) -> None:- cls.binary = default_binary_path()- if not os.path.isfile(cls.binary):- raise unittest.SkipTest(- f"Rust binary not found at {cls.binary}. "- "Run `cargo build -p cua-driver` or set CUA_DRIVER_BINARY."- )- print(f"\n[RustParityTests] binary: {cls.binary}", flush=True)---if __name__ == "__main__":- unittest.main(verbosity=2)diff --git a/libs/cua-driver/tests/e2e/test_background_focus.py b/libs/cua-driver/tests/e2e/test_background_focus.pydeleted file mode 100644index b86e1fbe9a..0000000000--- a/libs/cua-driver/tests/e2e/test_background_focus.py+++ /dev/null@@ -1,568 +0,0 @@-"""Integration test: interact with backgrounded apps without stealing focus.--Setup:- 1. Open Safari to a local HTML page with a button and text input.- 2. Launch FocusMonitorApp (counts NSApplication.didResignActiveNotification).- 3. FocusMonitorApp is activated last so it owns focus.--Tests send clicks and keystrokes to backgrounded Safari via the MCP driver,-then assert:- - Safari's page state changed (button counter incremented, text typed).- - FocusMonitorApp's focus-loss counter stayed at 0 throughout.--Also includes CalculatorBackgroundClickTests that use AX-click on Calculator-while FocusMonitorApp is frontmost.--Run:- CUA_DRIVER_BINARY=../../target/debug/cua-driver python3 -m unittest test_background_focus -v-"""--from __future__ import annotations--import os-import re-import subprocess-import sys-import time-import unittest--sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))-from driver_client import (- DriverClient,- default_binary_path,- frontmost_bundle_id,- resolve_window_id,-)--# ----------------------------------------------------------------------------# Paths-# ----------------------------------------------------------------------------_THIS_DIR = os.path.dirname(os.path.abspath(__file__))-_DRIVER_RS_ROOT = os.path.dirname(os.path.dirname(_THIS_DIR))-_LIBS_ROOT = os.path.dirname(_DRIVER_RS_ROOT)-_HTML_PAGE = os.path.join(_THIS_DIR, "fixtures", "interactive.html")--# FocusMonitorApp lives in the Swift cua-driver tree (shared fixture).-_FOCUS_APP_DIR = os.path.join(_LIBS_ROOT, "cua-driver", "Tests", "FocusMonitorApp")-_FOCUS_APP_BUNDLE = os.path.join(_FOCUS_APP_DIR, "FocusMonitorApp.app")-_FOCUS_APP_EXE = os.path.join(- _FOCUS_APP_BUNDLE, "Contents", "MacOS", "FocusMonitorApp"-)-_LOSS_FILE = "/tmp/focus_monitor_losses.txt"--SAFARI_BUNDLE = "com.apple.Safari"-FOCUS_MONITOR_BUNDLE = "com.trycua.FocusMonitorApp"---# ----------------------------------------------------------------------------# Helpers-# -----------------------------------------------------------------------------def _build_focus_app() -> None:- if not os.path.exists(_FOCUS_APP_EXE):- subprocess.run(- [os.path.join(_FOCUS_APP_DIR, "build.sh")], check=True- )---def _launch_focus_app() -> tuple[subprocess.Popen, int]:- """Launch FocusMonitorApp, wait for FOCUS_PID= line, return (proc, pid)."""- proc = subprocess.Popen(- [_FOCUS_APP_EXE],- stdout=subprocess.PIPE,- stderr=subprocess.DEVNULL,- text=True,- )- for _ in range(40):- line = proc.stdout.readline().strip()- if line.startswith("FOCUS_PID="):- pid = int(line.split("=", 1)[1])- return proc, pid- time.sleep(0.1)- proc.terminate()- raise RuntimeError("FocusMonitorApp did not print FOCUS_PID in time")---def _read_focus_losses() -> int:- try:- with open(_LOSS_FILE) as f:- return int(f.read().strip())- except (FileNotFoundError, ValueError):- return -1---def _open_safari_to_html(client: DriverClient) -> int:- """Open the test HTML page in a fresh Safari window and return its pid.-- Kills any existing Safari first so we get a clean single-window state- and `resolve_window_id` reliably picks the interactive.html window.- """- # Kill any existing Safari so we start clean with exactly one window.- subprocess.run(["pkill", "-x", "Safari"], check=False)- time.sleep(0.5)-- file_url = f"file://{_HTML_PAGE}"- # Open Safari in background (-g) with the specific URL.- subprocess.run(["open", "-g", "-a", "Safari", file_url], check=True)- time.sleep(3.0) # let Safari launch and page load-- # Get the pid from list_apps now that Safari is running.- result = client.call_tool("list_apps")- apps = result["structuredContent"]["apps"]- safari = next((a for a in apps if a.get("bundle_id") == SAFARI_BUNDLE), None)- if safari is None:- raise RuntimeError("Safari did not appear in list_apps after launch")- return safari["pid"]---def _get_page_text(client: DriverClient, pid: int) -> str:- """Return the AX tree markdown for Safari."""- window_id = resolve_window_id(client, pid)- result = client.call_tool(- "get_window_state", {"pid": pid, "window_id": window_id}- )- return result.get("structuredContent", result).get("tree_markdown", "")---def _find_element_index(tree_markdown: str, label: str) -> int | None:- """Extract the first [N] element index from a line containing `label`."""- for line in tree_markdown.split("\n"):- if label in line:- m = re.search(r'\[(\d+)\]', line)- if m:- return int(m.group(1))- return None---def _find_calc_button(tree_markdown: str, label: str) -> int | None:- """Find a Calculator button by description in parentheses or help text."""- for line in tree_markdown.split("\n"):- if "AXButton" not in line:- continue- m = re.search(r'\[(\d+)\]', line)- if not m:- continue- idx = int(m.group(1))- if f'({label})' in line:- return idx- if f'help="{label}' in line:- return idx- if f'id={label}' in line:- return idx- return None---def _activate_focus_monitor() -> None:- """Re-activate FocusMonitorApp via osascript."""- subprocess.run(- ["osascript", "-e", 'tell application "FocusMonitorApp" to activate'],- check=False,- )- time.sleep(0.5)---# ----------------------------------------------------------------------------# Safari background focus tests-# -----------------------------------------------------------------------------class BackgroundFocusTests(unittest.TestCase):- """Click & type into backgrounded Safari without stealing focus."""-- _safari_pid: int- _focus_proc: subprocess.Popen- _focus_pid: int-- @classmethod- def setUpClass(cls) -> None:- _build_focus_app()-- # Kill any stale FocusMonitorApp left over from a crashed prior run.- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)- time.sleep(0.3)-- try:- os.remove(_LOSS_FILE)- except FileNotFoundError:- pass-- cls.binary = default_binary_path()-- # 1. Open Safari to the test page.- with DriverClient(cls.binary) as c:- cls._safari_pid = _open_safari_to_html(c)- print(f"\n Safari pid: {cls._safari_pid}")-- # 2. Launch FocusMonitorApp (becomes frontmost).- cls._focus_proc, cls._focus_pid = _launch_focus_app()- print(f" FocusMonitor pid: {cls._focus_pid}")- time.sleep(1.0)-- # Confirm FocusMonitorApp is frontmost.- with DriverClient(cls.binary) as c:- active = frontmost_bundle_id(c)- assert active == FOCUS_MONITOR_BUNDLE, (- f"Expected FocusMonitorApp frontmost, got {active}"- )-- # Baseline: 0 focus losses.- losses = _read_focus_losses()- assert losses == 0, f"Expected 0 focus losses at start, got {losses}"-- @classmethod- def tearDownClass(cls) -> None:- if hasattr(cls, '_focus_proc'):- cls._focus_proc.terminate()- try:- cls._focus_proc.wait(timeout=3)- except subprocess.TimeoutExpired:- cls._focus_proc.kill()- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)-- subprocess.run(["pkill", "-x", "Safari"], check=False)-- try:- os.remove(_LOSS_FILE)- except FileNotFoundError:- pass-- def setUp(self) -> None:- _activate_focus_monitor()- self._losses_before = _read_focus_losses()-- with DriverClient(self.binary) as c:- active = frontmost_bundle_id(c)- self.assertEqual(- active, FOCUS_MONITOR_BUNDLE,- f"FocusMonitorApp not frontmost at test start, got {active}",- )-- def _assert_no_focus_loss(self, label: str) -> None:- """Assert FocusMonitorApp is still frontmost (or restored) after interaction."""- time.sleep(0.3)- losses = _read_focus_losses()- with DriverClient(self.binary) as c:- active = frontmost_bundle_id(c)- focus_restored = (active == FOCUS_MONITOR_BUNDLE)- loss_delta = losses - self._losses_before- print(f" [{label}] losses: {self._losses_before}->{losses} "- f"(delta={loss_delta}), frontmost: {active}, "- f"restored: {focus_restored}")- self.assertEqual(- active, FOCUS_MONITOR_BUNDLE,- f"[{label}] Focus not restored — "- f"frontmost is {active}, not FocusMonitorApp",- )-- def test_01_ax_click_button(self) -> None:- """AX-click the 'Click Me' button in backgrounded Safari."""- with DriverClient(self.binary) as c:- window_id = resolve_window_id(c, self._safari_pid)- snap = c.call_tool("get_window_state", {- "pid": self._safari_pid,- "window_id": window_id,- "query": "Click Me",- })- tree = snap.get("structuredContent", snap).get("tree_markdown", "")- print(f"\n filtered tree:\n{tree}")-- idx = _find_element_index(tree, "Click Me")- self.assertIsNotNone(idx, "Could not find 'Click Me' button in AX tree")-- result = c.call_tool("click", {- "pid": self._safari_pid,- "window_id": window_id,- "element_index": idx,- })- print(f" click result: {result}")-- time.sleep(0.5)-- with DriverClient(self.binary) as c:- tree = _get_page_text(c, self._safari_pid)- self.assertIn("clicks: 1", tree, "Button click did not register on page")-- self._assert_no_focus_loss("01_ax_click_button")-- def test_02_type_text_chars(self) -> None:- """Type into the text field in backgrounded Safari via keystroke synthesis."""- with DriverClient(self.binary) as c:- window_id = resolve_window_id(c, self._safari_pid)- snap = c.call_tool("get_window_state", {- "pid": self._safari_pid,- "window_id": window_id,- "query": "AXTextField",- })- tree = snap.get("structuredContent", snap).get("tree_markdown", "")- print(f"\n filtered tree:\n{tree}")-- # Find the page's text field (not the Safari URL bar).- idx = None- for line in tree.split("\n"):- if "AXTextField" in line and "smart search field" not in line:- m = re.search(r'\[(\d+)\]', line)- if m:- idx = int(m.group(1))- break-- if idx is None:- snap = c.call_tool("get_window_state", {- "pid": self._safari_pid, "window_id": window_id,- })- tree = snap.get("structuredContent", snap).get("tree_markdown", "")- for line in tree.split("\n"):- if "AXTextField" in line and "smart search field" not in line:- m = re.search(r'\[(\d+)\]', line)- if m:- idx = int(m.group(1))- break-- self.assertIsNotNone(idx, "Could not find page text field in AX tree")- print(f" text field element_index: {idx}")-- # Focus the text field via AX click first.- c.call_tool("click", {- "pid": self._safari_pid,- "window_id": window_id,- "element_index": idx,- })- time.sleep(0.3)-- # Type via keystroke synthesis.- result = c.call_tool("type_text_chars", {- "pid": self._safari_pid,- "text": "hello bg",- })- print(f" type_text_chars result: {result}")-- has_text = False- for attempt in range(4):- time.sleep(0.5)- with DriverClient(self.binary) as c:- tree = _get_page_text(c, self._safari_pid)- has_text = ("hello bg" in tree)- if has_text:- break- print(f" attempt {attempt+1}: text not yet visible in tree")-- print(f" text found in tree: {has_text}")- self.assertTrue(has_text, "Typed text not found in Safari AX tree")-- self._assert_no_focus_loss("02_type_text_chars")-- def test_03_pixel_click_no_focus_steal(self) -> None:- """Pixel-click backgrounded Safari — verify no focus steal."""- with DriverClient(self.binary) as c:- window_id = resolve_window_id(c, self._safari_pid)- snap = c.call_tool("get_window_state", {- "pid": self._safari_pid,- "window_id": window_id,- "query": "Click Me",- })- sc = snap.get("structuredContent", snap)- width = sc.get("screenshot_width", 0)- height = sc.get("screenshot_height", 0)- print(f"\n screenshot: {width}x{height}")-- x = width // 2- y = int(height * 0.35)- print(f" pixel click at ({x}, {y})")-- result = c.call_tool("click", {- "pid": self._safari_pid,- "window_id": window_id,- "x": x,- "y": y,- })- print(f" click result: {result}")-- time.sleep(0.5)- self._assert_no_focus_loss("03_pixel_click_no_focus_steal")-- def test_04_press_key_tab(self) -> None:- """Send Tab key to backgrounded Safari without stealing focus."""- with DriverClient(self.binary) as c:- result = c.call_tool("press_key", {- "pid": self._safari_pid, "key": "tab",- })- print(f"\n press_key result: {result}")-- time.sleep(0.3)- self._assert_no_focus_loss("04_press_key_tab")---# ----------------------------------------------------------------------------# Calculator AX-click background tests-# -----------------------------------------------------------------------------class CalculatorBackgroundClickTests(unittest.TestCase):- """AX-click backgrounded Calculator without stealing focus."""-- CALC_BUNDLE = "com.apple.calculator"-- _calc_pid: int- _focus_proc: subprocess.Popen- _focus_pid: int-- @classmethod- def setUpClass(cls) -> None:- _build_focus_app()- try:- os.remove(_LOSS_FILE)- except FileNotFoundError:- pass- subprocess.run(["pkill", "-x", "Calculator"], check=False)- # Kill any stale FocusMonitorApp left over from a crashed prior run.- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)- time.sleep(0.3)-- cls.binary = default_binary_path()-- with DriverClient(cls.binary) as c:- result = c.call_tool("launch_app", {"bundle_id": cls.CALC_BUNDLE})- cls._calc_pid = result["structuredContent"]["pid"]- print(f"\n Calculator pid: {cls._calc_pid}")- time.sleep(1.5)-- cls._focus_proc, cls._focus_pid = _launch_focus_app()- print(f" FocusMonitor pid: {cls._focus_pid}")- time.sleep(1.0)-- with DriverClient(cls.binary) as c:- active = frontmost_bundle_id(c)- assert active == FOCUS_MONITOR_BUNDLE, (- f"Expected FocusMonitorApp frontmost, got {active}"- )-- @classmethod- def tearDownClass(cls) -> None:- if hasattr(cls, '_focus_proc'):- cls._focus_proc.terminate()- try:- cls._focus_proc.wait(timeout=3)- except subprocess.TimeoutExpired:- cls._focus_proc.kill()- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)- subprocess.run(["pkill", "-x", "Calculator"], check=False)- try:- os.remove(_LOSS_FILE)- except FileNotFoundError:- pass-- def setUp(self) -> None:- _activate_focus_monitor()- self._losses_before = _read_focus_losses()- with DriverClient(self.binary) as c:- active = frontmost_bundle_id(c)- self.assertEqual(active, FOCUS_MONITOR_BUNDLE)-- def _assert_no_focus_loss(self, label: str) -> None:- time.sleep(0.3)- losses = _read_focus_losses()- with DriverClient(self.binary) as c:- active = frontmost_bundle_id(c)- loss_delta = losses - self._losses_before- print(f" [{label}] losses: {self._losses_before}->{losses} "- f"(delta={loss_delta}), frontmost: {active}")- self.assertEqual(- active, FOCUS_MONITOR_BUNDLE,- f"[{label}] Focus not restored — "- f"frontmost is {active}, not FocusMonitorApp",- )-- def test_01_ax_click_2_plus_2(self) -> None:- """AX-click 2 + 2 = on Calculator while backgrounded."""- with DriverClient(self.binary) as c:- window_id = resolve_window_id(c, self._calc_pid)- snap = c.call_tool(- "get_window_state",- {"pid": self._calc_pid, "window_id": window_id},- )- tree = snap.get("structuredContent", snap).get("tree_markdown", "")- print(f"\n Calculator tree:\n{tree[:1500]}")-- btn_2 = _find_calc_button(tree, "2")- btn_add = _find_calc_button(tree, "Add")- btn_eq = _find_calc_button(tree, "Equals")- print(f" buttons: 2={btn_2}, Add={btn_add}, Equals={btn_eq}")-- self.assertIsNotNone(btn_2, "'2' button not found")- self.assertIsNotNone(btn_add, "'Add' button not found")- self.assertIsNotNone(btn_eq, "'Equals' button not found")-- for idx in [btn_2, btn_add, btn_2, btn_eq]:- c.call_tool("click", {- "pid": self._calc_pid,- "window_id": window_id,- "element_index": idx,- })- time.sleep(0.3)-- snap = c.call_tool("get_window_state", {- "pid": self._calc_pid,- "window_id": window_id,- "query": "AXStaticText",- })- tree = snap.get("structuredContent", snap).get("tree_markdown", "")- print(f" result tree:\n{tree}")-- self.assertIn("4", tree, "Calculator did not show 4 after 2+2=")- self._assert_no_focus_loss("01_ax_click_2_plus_2")-- def test_02_ax_click_3_plus_4(self) -> None:- """AX-click 3 + 4 = on Calculator (second computation)."""- with DriverClient(self.binary) as c:- window_id = resolve_window_id(c, self._calc_pid)- # Clear first- snap = c.call_tool("get_window_state", {- "pid": self._calc_pid, "window_id": window_id,- })- tree = snap.get("structuredContent", snap).get("tree_markdown", "")- btn_c = _find_calc_button(tree, "All Clear")- if btn_c is None:- btn_c = _find_calc_button(tree, "Clear")- if btn_c is not None:- c.call_tool("click", {- "pid": self._calc_pid,- "window_id": window_id,- "element_index": btn_c,- })- time.sleep(0.3)-- # Find buttons for 3+4=- snap = c.call_tool("get_window_state", {- "pid": self._calc_pid, "window_id": window_id,- })- tree = snap.get("structuredContent", snap).get("tree_markdown", "")- btn_3 = _find_calc_button(tree, "3")- btn_add = _find_calc_button(tree, "Add")- btn_4 = _find_calc_button(tree, "4")- btn_eq = _find_calc_button(tree, "Equals")- print(f"\n buttons: 3={btn_3}, Add={btn_add}, 4={btn_4}, Equals={btn_eq}")-- self.assertIsNotNone(btn_3, "'3' button not found")- self.assertIsNotNone(btn_add, "'Add' button not found")- self.assertIsNotNone(btn_4, "'4' button not found")- self.assertIsNotNone(btn_eq, "'Equals' button not found")-- for idx in [btn_3, btn_add, btn_4, btn_eq]:- c.call_tool("click", {- "pid": self._calc_pid,- "window_id": window_id,- "element_index": idx,- })- time.sleep(0.3)-- snap = c.call_tool("get_window_state", {- "pid": self._calc_pid,- "window_id": window_id,- "query": "AXStaticText",- })- tree = snap.get("structuredContent", snap).get("tree_markdown", "")- print(f" result tree:\n{tree}")-- self.assertIn("7", tree, "Calculator did not show 7 after 3+4=")- self._assert_no_focus_loss("02_ax_click_3_plus_4")---if __name__ == "__main__":- unittest.main(verbosity=2)diff --git a/libs/cua-driver/tests/e2e/test_blender.py b/libs/cua-driver/tests/e2e/test_blender.pydeleted file mode 100644index b0455e6055..0000000000--- a/libs/cua-driver/tests/e2e/test_blender.py+++ /dev/null@@ -1,198 +0,0 @@-"""Blender background automation tests — v2 harness.--Blender has no DOM, so assertions use:- Primary: AX tree window title / menu bar labels- Secondary: Screenshot + NCC template matching against reference crops--Workflow:- 1. Launch Blender (open -g -a Blender)- 2. FocusMonitorApp takes focus- 3. Tests interact via click / press_key / hotkey- 4. Verify state via AX tree window title or screenshot diff--Reference images for template matching are generated on first run and-stored under assets/blender/. On subsequent runs the stored images are-used as the match target.--Run: python3 -m pytest test_blender.py -v-"""--from __future__ import annotations--import subprocess-import sys-import time-import os-import json--import pytest--_HERE = os.path.dirname(os.path.abspath(__file__))-sys.path.insert(0, _HERE)-from harness.driver import Driver-from harness import tree as Tree-from harness.cv import decode, crop, diff_ratio, save_reference, load_reference--BLENDER_BUNDLE = "org.blenderfoundation.blender"-_ASSETS_BLENDER = os.path.join(_HERE, "assets", "blender")-_BBOXES_FILE = os.path.join(_ASSETS_BLENDER, "bboxes.json")---def _load_bboxes() -> dict:- if os.path.exists(_BBOXES_FILE):- with open(_BBOXES_FILE) as f:- return json.load(f)- return {}---def _save_bboxes(bboxes: dict) -> None:- os.makedirs(_ASSETS_BLENDER, exist_ok=True)- with open(_BBOXES_FILE, "w") as f:- json.dump(bboxes, f, indent=2)---# ── module-level Blender setup ────────────────────────────────────────────────-[email protected](scope="module")-def blender_pid(binary, focus_monitor):- """Launch Blender in background; return its pid."""- subprocess.run(["pkill", "-x", "Blender"], check=False)- time.sleep(0.5)-- subprocess.run(["open", "-g", "-a", "Blender"], check=False)- time.sleep(5.0) # Blender takes a while to start-- with Driver(binary) as d:- app = d.find_app(BLENDER_BUNDLE)- if app is None:- # Try by name- for a in d.list_apps():- if "blender" in str(a.get("name", "")).lower():- app = a- break- if app is None:- pytest.skip("Blender not installed or not found in list_apps")- pid = app["pid"]-- print(f"\n Blender pid: {pid}")- yield pid-- subprocess.run(["pkill", "-x", "Blender"], check=False)--[email protected](autouse=True)-def _reactivate_focus(focus_monitor):- _, pid = focus_monitor- subprocess.run(- ["osascript", "-e",- f'tell application "System Events" to set frontmost of (first process whose unix id is {pid}) to true'],- check=False,- )- time.sleep(0.4)---# ── tests ─────────────────────────────────────────────────────────────────────--class TestBlenderWindow:- """Basic window state and AX tree tests."""-- def test_window_state_returns_tree(self, driver, blender_pid, ux_guard):- wid = driver.find_window(blender_pid)- state = driver.get_window_state(blender_pid, wid)- assert state.tree, "Empty AX tree from Blender"- print(f"\n Blender tree (first 300 chars):\n{state.tree[:300]}")-- def test_window_state_has_screenshot(self, driver, blender_pid, ux_guard):- wid = driver.find_window(blender_pid)- state = driver.get_window_state(blender_pid, wid)- assert state.screenshot_b64, "No screenshot from Blender"- assert state.screenshot_width > 0- assert state.screenshot_height > 0- print(f"\n Blender screenshot: {state.screenshot_width}x{state.screenshot_height}")-- def test_ax_tree_has_window(self, driver, blender_pid, ux_guard):- wid = driver.find_window(blender_pid)- state = driver.get_window_state(blender_pid, wid)- assert "AXWindow" in state.tree, f"No AXWindow in:\n{state.tree[:500]}"---class TestBlenderMenuInteraction:- """Open a menu via hotkey and verify AX tree changes."""-- def test_file_menu_via_hotkey(self, driver, blender_pid, ux_guard):- """Open Blender's File menu via hotkey and verify menu appears."""- wid = driver.find_window(blender_pid)-- before = driver.get_window_state(blender_pid, wid)-- # F4 opens File menu in Blender (or use hotkey cmd+shift+n)- # Using press_key for a simple key that has observable AX effect- driver.hotkey(blender_pid, ["f4"])- time.sleep(0.8)-- after = driver.get_window_state(blender_pid, wid)- tree_changed = after.tree != before.tree- print(f"\n AX tree changed after F4: {tree_changed}")- # Press Escape to close any menu- driver.press_key(blender_pid, "escape")- time.sleep(0.3)-- # Just verify the key reached Blender (driver didn't error)- # AX tree change is a bonus — Blender menus may not expose via AX- print(f" tree before: {len(before.tree)} chars, after: {len(after.tree)} chars")---class TestBlenderScreenshotDiff:- """Verify screenshot changes after interaction."""-- def test_click_viewport_changes_screenshot(self, driver, blender_pid, ux_guard):- """Click in the 3D viewport and verify screenshot changes."""- wid = driver.find_window(blender_pid)-- before = driver.get_window_state(blender_pid, wid)- assert before.screenshot_b64-- # Click in the center of the viewport (assumed to be center of screen)- x = before.screenshot_width // 2- y = before.screenshot_height // 2-- before_img = decode(before.screenshot_b64)-- driver.click(blender_pid, wid, x=x, y=y)- time.sleep(0.5)-- after = driver.get_window_state(blender_pid, wid)- after_img = decode(after.screenshot_b64)-- ratio = diff_ratio(before_img, after_img)- print(f"\n viewport click diff_ratio: {ratio:.4f}")- # A click in Blender's viewport should change the selection highlight- assert ratio >= 0, "diff_ratio calculation succeeded"- # Note: Blender in background may not render selection changes- # so we don't assert ratio > threshold — just that the call succeeded-- def test_screenshot_reference_workflow(self, driver, blender_pid, ux_guard):- """Generate/verify a reference screenshot crop for NCC template matching."""- wid = driver.find_window(blender_pid)- state = driver.get_window_state(blender_pid, wid)- assert state.screenshot_b64-- img = decode(state.screenshot_b64)- w, h = img.size-- # Capture a reference crop of the top-left quadrant (toolbar area)- bbox = (0, 0, w // 4, h // 4)- ref_crop = crop(img, bbox)-- ref_path = os.path.join(_ASSETS_BLENDER, "toolbar_ref.png")- os.makedirs(_ASSETS_BLENDER, exist_ok=True)- save_reference(ref_crop, ref_path)- assert os.path.exists(ref_path), "Reference image was not saved"-- # Load it back and verify it's identical to the crop- loaded = load_reference(ref_path)- ratio = diff_ratio(ref_crop, loaded)- assert ratio < 0.001, f"Reference image roundtrip diff too high: {ratio:.4f}"-- print(f"\n reference saved: {ref_path} ({ref_crop.size})")- print(f" roundtrip diff_ratio: {ratio:.6f}")diff --git a/libs/cua-driver/tests/e2e/test_chrome.py b/libs/cua-driver/tests/e2e/test_chrome.pydeleted file mode 100644index 8cef3b3afd..0000000000--- a/libs/cua-driver/tests/e2e/test_chrome.py+++ /dev/null@@ -1,189 +0,0 @@-"""Chrome background automation tests — v2 harness.--Mirrors test_safari.py but targets Google Chrome. The same test_page.html-is loaded via the local HTTP server. Chrome uses CGEvent/SkyLight for-background delivery which is a different code path from Safari's AX.--Run: python3 -m pytest test_chrome.py -v-"""--from __future__ import annotations--import re-import subprocess-import sys-import time-import os--import pytest--_HERE = os.path.dirname(os.path.abspath(__file__))-sys.path.insert(0, _HERE)-from harness.driver import Driver-from harness import tree as Tree---def _parse_clicks(tree_text: str) -> int:- m = re.search(r'clicks:\s*(\d+)', tree_text)- return int(m.group(1)) if m else 0--CHROME_BUNDLE = "com.google.Chrome"-FOCUS_MONITOR_BUNDLE = "com.trycua.FocusMonitorApp"---# ── module-level Chrome setup ─────────────────────────────────────────────────--CHROME_APP = "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"--[email protected](scope="module")-def chrome_pid(binary, html_server, focus_monitor):- """Launch Chrome with --force-renderer-accessibility to enable AX for web content.-- Chrome only exposes web content in the AX tree when launched with this flag,- or when an AT (Assistive Technology) has connected. Using the flag is the most- reliable approach for integration testing.- """- subprocess.run(["pkill", "-x", "Google Chrome"], check=False)- time.sleep(0.5)-- if not os.path.exists(CHROME_APP):- pytest.skip("Google Chrome not installed at default path")-- page_url = f"{html_server}/test_page.html"- proc = subprocess.Popen(- [CHROME_APP, "--force-renderer-accessibility", page_url],- stdout=subprocess.DEVNULL,- stderr=subprocess.DEVNULL,- )- time.sleep(5.0) # allow full page load + AX initialization-- with Driver(binary) as d:- app = d.find_app(CHROME_BUNDLE)- if app is None:- proc.terminate()- pytest.skip("Google Chrome did not appear in list_apps")- pid = app["pid"]-- print(f"\n Chrome pid: {pid} (with renderer accessibility)")-- # Hand focus to FocusMonitorApp by pid to avoid activating stale instances- _, fm_pid = focus_monitor- subprocess.run(- ["osascript", "-e",- f'tell application "System Events" to set frontmost of (first process whose unix id is {fm_pid}) to true'],- check=False,- )- time.sleep(0.5)-- yield pid-- proc.terminate()- subprocess.run(["pkill", "-x", "Google Chrome"], check=False)--[email protected](autouse=True)-def _reactivate_focus(focus_monitor):- _, pid = focus_monitor- subprocess.run(- ["osascript", "-e",- f'tell application "System Events" to set frontmost of (first process whose unix id is {pid}) to true'],- check=False,- )- time.sleep(0.4)---# ── tests ─────────────────────────────────────────────────────────────────────--class TestChromeButton:- def test_click_increments_counter(self, driver, chrome_pid, ux_guard):- wid = driver.find_window(chrome_pid)-- full = driver.get_window_state(chrome_pid, wid)- idx = full.find_element("Click Me")- assert idx is not None, f"'Click Me' not found in tree (first 800 chars):\n{full.tree[:800]}"- count_before = _parse_clicks(full.tree)-- driver.click(chrome_pid, wid, element_index=idx)- time.sleep(0.5)-- after = driver.get_window_state(chrome_pid, wid)- count_after = _parse_clicks(after.tree)- print(f"\n click counter: {count_before} → {count_after}")- assert count_after == count_before + 1, f"Counter did not increment: {count_before} → {count_after}"-- def test_click_increments_twice(self, driver, chrome_pid, ux_guard):- wid = driver.find_window(chrome_pid)- full = driver.get_window_state(chrome_pid, wid)- idx = full.find_element("Click Me")- assert idx is not None- count_before = _parse_clicks(full.tree)-- driver.click(chrome_pid, wid, element_index=idx)- time.sleep(0.3)- driver.click(chrome_pid, wid, element_index=idx)- time.sleep(0.5)-- after = driver.get_window_state(chrome_pid, wid)- count_after = _parse_clicks(after.tree)- print(f"\n click counter: {count_before} → {count_after}")- assert count_after == count_before + 2, f"Counter should have incremented by 2: {count_before} → {count_after}"---class TestChromeTextInput:- def test_type_text_appears(self, driver, chrome_pid, ux_guard):- wid = driver.find_window(chrome_pid)- before = driver.get_window_state(chrome_pid, wid)- idx = before.find_text_field(skip_url_bar=True)- assert idx is not None, f"Text field not found:\n{before.tree}"-- driver.click(chrome_pid, wid, element_index=idx)- time.sleep(0.3)- driver.hotkey(chrome_pid, ["cmd", "a"])- time.sleep(0.1)- driver.type_text_chars(chrome_pid, "hello chrome")- time.sleep(0.6)-- after = driver.get_window_state(chrome_pid, wid)- assert after.has_text("hello chrome"), f"Text not found:\n{after.tree}"---class TestChromeCanvas:- def test_canvas_pixel_click_no_focus_steal(self, driver, chrome_pid, ux_guard):- """Send a pixel click to the canvas area without stealing focus.-- Canvas DOM events may not fire in background; we verify the click is- delivered without error and without UX violations via ux_guard.- """- wid = driver.find_window(chrome_pid)- mid = driver.get_window_state(chrome_pid, wid)- assert mid.screenshot_b64, "No screenshot"-- from harness.cv import decode, diff_ratio- before_img = decode(mid.screenshot_b64)-- x = mid.screenshot_width // 2- y = int(mid.screenshot_height * 0.88)- print(f"\n chrome canvas click at ({x}, {y})")-- result = driver.click(chrome_pid, wid, x=x, y=y)- time.sleep(0.5)-- after = driver.get_window_state(chrome_pid, wid)- after_img = decode(after.screenshot_b64)- ratio = diff_ratio(before_img, after_img)- canvas_clicked = "canvas: clicked" in after.tree- print(f" chrome canvas diff_ratio: {ratio:.4f}, JS fired: {canvas_clicked}")-- assert result is not None, "click returned None"- # UX assertions via ux_guard teardown---class TestChromePressKey:- def test_tab_key(self, driver, chrome_pid, ux_guard):- driver.press_key(chrome_pid, "tab")- time.sleep(0.3)-- def test_escape_key(self, driver, chrome_pid, ux_guard):- driver.press_key(chrome_pid, "escape")- time.sleep(0.2)diff --git a/libs/cua-driver/tests/e2e/test_cli.py b/libs/cua-driver/tests/e2e/test_cli.pydeleted file mode 100644index c713315d35..0000000000--- a/libs/cua-driver/tests/e2e/test_cli.py+++ /dev/null@@ -1,242 +0,0 @@-"""Integration test: cua-driver-rs CLI subcommands.--Covers:- 1. `list-tools` prints tool names and summaries.- 2. `describe <tool>` prints the tool schema.- 3. `call <tool> [json]` invokes a tool and prints ✅ result.- 4. Implicit call (first positional as tool name) works.- 5. Unknown tool exits 64.- 6. Error tool result exits 1.- 7. `screenshot` call returns JSON with a screenshot_png_b64 key.--Run:- CUA_DRIVER_BINARY=../../target/debug/cua-driver python3 -m unittest test_cli -v-"""--from __future__ import annotations--import json-import os-import subprocess-import sys-import unittest--sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))-from driver_client import default_binary_path--BINARY = None # set in setUpClass---def _run(args: list[str], timeout: int = 15) -> subprocess.CompletedProcess:- return subprocess.run(- args,- capture_output=True,- text=True,- timeout=timeout,- )---class CLISubcommandTests(unittest.TestCase):- @classmethod- def setUpClass(cls) -> None:- cls.binary = default_binary_path()-- def test_list_tools_prints_tool_names(self) -> None:- r = _run([self.binary, "list-tools"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- self.assertIn("click", r.stdout)- self.assertIn("screenshot", r.stdout)- self.assertIn("list_windows", r.stdout)- self.assertIn("press_key", r.stdout)-- def test_list_tools_has_name_colon_summary_format(self) -> None:- r = _run([self.binary, "list-tools"])- self.assertEqual(r.returncode, 0)- for line in r.stdout.strip().splitlines():- self.assertRegex(- line,- r"^\w+",- f"list-tools line should start with tool name: {line!r}",- )-- def test_describe_prints_schema(self) -> None:- r = _run([self.binary, "describe", "click"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- self.assertIn("click", r.stdout)- self.assertIn("pid", r.stdout)- self.assertIn("input_schema", r.stdout)-- def test_describe_unknown_tool_exits_64(self) -> None:- r = _run([self.binary, "describe", "no_such_tool_xyz"])- self.assertEqual(r.returncode, 64)- self.assertIn("no_such_tool_xyz", r.stderr)-- def test_call_get_screen_size(self) -> None:- r = _run([self.binary, "call", "get_screen_size"])- self.assertEqual(r.returncode, 0, f"stdout: {r.stdout}\nstderr: {r.stderr}")- data = json.loads(r.stdout)- self.assertIn("width", data)- self.assertIn("height", data)- self.assertGreater(data["width"], 0)- self.assertGreater(data["height"], 0)-- def test_implicit_call_get_screen_size(self) -> None:- """First positional arg is treated as tool name (no 'call' prefix)."""- r = _run([self.binary, "get_screen_size"])- self.assertEqual(r.returncode, 0, f"stdout: {r.stdout}\nstderr: {r.stderr}")- data = json.loads(r.stdout)- self.assertIn("width", data)- self.assertIn("height", data)-- def test_call_check_permissions_exits_zero(self) -> None:- """check_permissions is read-only and always exits 0."""- r = _run([self.binary, "call", "check_permissions"])- self.assertEqual(r.returncode, 0, f"stdout: {r.stdout}\nstderr: {r.stderr}")-- def test_call_check_permissions_output_has_green_checkmark(self) -> None:- r = _run([self.binary, "call", "check_permissions"])- self.assertEqual(r.returncode, 0)- # The structured JSON output should have accessibility/screen_recording fields.- data = json.loads(r.stdout)- self.assertIn("accessibility", data)- self.assertIn("screen_recording", data)-- def test_call_unknown_tool_exits_64(self) -> None:- r = _run([self.binary, "call", "no_such_tool_xyz"])- self.assertEqual(r.returncode, 64)-- def test_call_press_key_with_missing_pid_exits_1(self) -> None:- """Calling a tool with missing required args should return an MCP error → exit 1."""- r = _run([self.binary, "call", "press_key", '{"key": "a"}'])- self.assertEqual(r.returncode, 1, f"stdout: {r.stdout}\nstderr: {r.stderr}")-- def test_call_list_apps_json_output(self) -> None:- r = _run([self.binary, "call", "list_apps"])- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- data = json.loads(r.stdout)- self.assertIn("apps", data)- self.assertIsInstance(data["apps"], list)- self.assertGreater(len(data["apps"]), 0)-- def test_call_screenshot_returns_b64_image(self) -> None:- r = _run([self.binary, "call", "screenshot"], timeout=30)- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- data = json.loads(r.stdout)- self.assertIn("screenshot_png_b64", data, "screenshot output missing image key")- b64 = data["screenshot_png_b64"]- self.assertGreater(len(b64), 100, "screenshot base64 data seems too short")---class ServeDaemonTests(unittest.TestCase):- """Tests for `cua-driver serve` / `stop` / `status`."""-- @classmethod- def setUpClass(cls) -> None:- cls.binary = default_binary_path()- import tempfile, os- cls._sock_file = tempfile.mktemp(suffix=".sock", prefix="cua-driver-test-")-- def _start_daemon(self):- import subprocess, time- proc = subprocess.Popen(- [self.binary, "serve", "--socket", self._sock_file],- stdout=subprocess.PIPE, stderr=subprocess.PIPE,- )- # Wait for daemon to bind.- for _ in range(30):- time.sleep(0.1)- r = _run([self.binary, "status", "--socket", self._sock_file])- if r.returncode == 0:- break- return proc-- def test_status_exits_1_when_no_daemon(self) -> None:- import tempfile- sock = tempfile.mktemp(suffix=".sock", prefix="cua-noexist-")- r = _run([self.binary, "status", "--socket", sock])- self.assertEqual(r.returncode, 1)- self.assertIn("not running", r.stderr)-- def test_stop_exits_1_when_no_daemon(self) -> None:- import tempfile- sock = tempfile.mktemp(suffix=".sock", prefix="cua-noexist2-")- r = _run([self.binary, "stop", "--socket", sock])- self.assertEqual(r.returncode, 1)-- def test_serve_status_stop_lifecycle(self) -> None:- """Start daemon → status is running → stop → status is not running."""- proc = self._start_daemon()- try:- # Status should show running.- r = _run([self.binary, "status", "--socket", self._sock_file])- self.assertEqual(r.returncode, 0, f"status stderr: {r.stderr}")- self.assertIn("running", r.stdout)- self.assertIn("socket:", r.stdout)-- # Stop the daemon.- r = _run([self.binary, "stop", "--socket", self._sock_file])- self.assertEqual(r.returncode, 0, f"stop stderr: {r.stderr}")-- # Status should now show not running.- import time; time.sleep(0.2)- r = _run([self.binary, "status", "--socket", self._sock_file])- self.assertEqual(r.returncode, 1)- finally:- proc.terminate()- proc.wait(timeout=3)-- def test_serve_double_start_exits_1(self) -> None:- """Second `serve` on same socket should fail with exit 1."""- proc = self._start_daemon()- try:- r = _run([self.binary, "serve", "--socket", self._sock_file])- self.assertEqual(r.returncode, 1)- self.assertIn("already running", r.stderr)- finally:- _run([self.binary, "stop", "--socket", self._sock_file])- proc.wait(timeout=3)-- @unittest.skipIf(- sys.platform == "win32",- "Windows: daemon spawned by subprocess.Popen runs in same session as "- "the test runner (typically Session 0 / non-interactive in CI); the "- "GDI BitBlt path fails with `The handle is invalid (0x80070006)` "- "because Session 0 has no graphics. The regression-guard behaviour "- "still verifies on macOS/Linux runners.",- )- def test_call_screenshot_via_daemon_emits_b64(self) -> None:- """`cua-driver call screenshot` over the daemon socket must emit- `screenshot_png_b64` — same shape as the in-process path. Regression- guard for the 2026-05-23 fix where the daemon-forwarding path in- `run_call` was silently dropping the image bytes (only printing- structuredContent metadata: format / width / height).- """- proc = self._start_daemon()- try:- r = _run(- [self.binary, "call", "--socket", self._sock_file, "screenshot"],- timeout=30,- )- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")- data = json.loads(r.stdout)- self.assertIn(- "screenshot_png_b64", data,- "daemon-forwarded screenshot dropped the image — "- "merge into structuredContent regressed",- )- b64 = data["screenshot_png_b64"]- self.assertGreater(- len(b64), 100,- "screenshot base64 data seems too short — likely empty payload",- )- self.assertIn(- "screenshot_mime_type", data,- "merge into structuredContent missed the mime-type key",- )- finally:- _run([self.binary, "stop", "--socket", self._sock_file])- proc.wait(timeout=3)---if __name__ == "__main__":- unittest.main(verbosity=2)diff --git a/libs/cua-driver/tests/e2e/test_concurrent_drivers.py b/libs/cua-driver/tests/e2e/test_concurrent_drivers.pydeleted file mode 100644index 83585af4ae..0000000000--- a/libs/cua-driver/tests/e2e/test_concurrent_drivers.py+++ /dev/null@@ -1,324 +0,0 @@-"""Integration test: two cua-driver processes running concurrently.--Verifies that two independent cua-driver instances can coexist without:- - crashing each other- - stealing focus from FocusMonitorApp- - interfering with each other's operations--Scenario:- Driver A → AX-clicks Calculator (2 + 2 = 4)- Driver B → Concurrently moves its agent cursor and inspects window list--Run:- CUA_DRIVER_BINARY=../../target/debug/cua-driver python3 -m unittest test_concurrent_drivers -v-"""--from __future__ import annotations--import os-import re-import subprocess-import sys-import threading-import time-import unittest--sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))-from driver_client import (- DriverClient,- default_binary_path,- frontmost_bundle_id,- resolve_window_id,-)--_THIS_DIR = os.path.dirname(os.path.abspath(__file__))-_DRIVER_RS_ROOT = os.path.dirname(os.path.dirname(_THIS_DIR))-_LIBS_ROOT = os.path.dirname(_DRIVER_RS_ROOT)-_FOCUS_APP_DIR = os.path.join(_LIBS_ROOT, "cua-driver", "Tests", "FocusMonitorApp")-_FOCUS_APP_EXE = os.path.join(- _FOCUS_APP_DIR, "FocusMonitorApp.app", "Contents", "MacOS", "FocusMonitorApp"-)-_LOSS_FILE = "/tmp/focus_monitor_losses.txt"--CALC_BUNDLE = "com.apple.calculator"-FOCUS_MONITOR_BUNDLE = "com.trycua.FocusMonitorApp"---def _build_focus_app() -> None:- if not os.path.exists(_FOCUS_APP_EXE):- subprocess.run(- [os.path.join(_FOCUS_APP_DIR, "build.sh")], check=True- )---def _launch_focus_app() -> tuple[subprocess.Popen, int]:- proc = subprocess.Popen(- [_FOCUS_APP_EXE],- stdout=subprocess.PIPE,- stderr=subprocess.DEVNULL,- text=True,- )- for _ in range(40):- line = proc.stdout.readline().strip()- if line.startswith("FOCUS_PID="):- return proc, int(line.split("=", 1)[1])- time.sleep(0.1)- proc.terminate()- raise RuntimeError("FocusMonitorApp did not print FOCUS_PID in time")---def _read_focus_losses() -> int:- try:- with open(_LOSS_FILE) as f:- return int(f.read().strip())- except (FileNotFoundError, ValueError):- return -1---def _find_calc_button(tree: str, label: str) -> int | None:- for line in tree.split("\n"):- if "AXButton" not in line:- continue- m = re.search(r'\[(\d+)\]', line)- if not m:- continue- if f'({label})' in line or f'id={label}' in line:- return int(m.group(1))- return None---class ConcurrentDriversTest(unittest.TestCase):- """Two driver processes running side-by-side without interference."""-- _calc_pid: int- _focus_proc: subprocess.Popen- _focus_pid: int- binary: str-- @classmethod- def setUpClass(cls) -> None:- _build_focus_app()-- # Kill stale processes from any prior crashed run.- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)- subprocess.run(["pkill", "-x", "Calculator"], check=False)- time.sleep(0.3)-- try:- os.remove(_LOSS_FILE)- except FileNotFoundError:- pass-- cls.binary = default_binary_path()-- # Launch Calculator via Driver A.- with DriverClient(cls.binary) as c:- result = c.call_tool("launch_app", {"bundle_id": CALC_BUNDLE})- cls._calc_pid = result["structuredContent"]["pid"]- print(f"\n Calculator pid: {cls._calc_pid}")- time.sleep(1.5)-- # Launch FocusMonitorApp (becomes frontmost).- cls._focus_proc, cls._focus_pid = _launch_focus_app()- print(f" FocusMonitor pid: {cls._focus_pid}")- time.sleep(1.0)-- with DriverClient(cls.binary) as c:- active = frontmost_bundle_id(c)- assert active == FOCUS_MONITOR_BUNDLE, (- f"Expected FocusMonitorApp frontmost, got {active}"- )-- losses = _read_focus_losses()- assert losses == 0, f"Expected 0 focus losses at start, got {losses}"-- @classmethod- def tearDownClass(cls) -> None:- if hasattr(cls, '_focus_proc'):- cls._focus_proc.terminate()- try:- cls._focus_proc.wait(timeout=3)- except subprocess.TimeoutExpired:- cls._focus_proc.kill()- if cls._focus_proc.stdout:- cls._focus_proc.stdout.close()- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)- subprocess.run(["pkill", "-x", "Calculator"], check=False)- try:- os.remove(_LOSS_FILE)- except FileNotFoundError:- pass-- def setUp(self) -> None:- # Clear Calculator before each test so previous display state doesn't interfere.- with DriverClient(self.binary) as c:- window_id = resolve_window_id(c, self._calc_pid)- snap = c.call_tool("get_window_state", {- "pid": self._calc_pid, "window_id": window_id,- })- tree = snap.get("structuredContent", snap).get("tree_markdown", "")- btn_ac = _find_calc_button(tree, "All Clear") or _find_calc_button(tree, "Clear")- if btn_ac is not None:- c.call_tool("click", {- "pid": self._calc_pid, "window_id": window_id, "element_index": btn_ac,- })- time.sleep(0.2)-- subprocess.run(- ["osascript", "-e", 'tell application "FocusMonitorApp" to activate'],- check=False,- )- time.sleep(0.5)- self._losses_before = _read_focus_losses()-- # ── tests ─────────────────────────────────────────────────────────────────-- def test_two_drivers_run_concurrently(self) -> None:- """Open Driver A and Driver B simultaneously; both must succeed without crashing."""-- driver_a_result: dict = {}- driver_b_result: dict = {}- driver_a_error: list[str] = []- driver_b_error: list[str] = []-- def run_driver_a() -> None:- try:- with DriverClient(self.binary) as ca:- # Driver A: AX-click 2 + 2 = on Calculator.- window_id = resolve_window_id(ca, self._calc_pid)- snap = ca.call_tool("get_window_state", {- "pid": self._calc_pid, "window_id": window_id,- })- tree = snap.get("structuredContent", snap).get("tree_markdown", "")- btn_2 = _find_calc_button(tree, "2")- btn_add = _find_calc_button(tree, "Add")- btn_eq = _find_calc_button(tree, "Equals")- if None in (btn_2, btn_add, btn_eq):- driver_a_error.append("Missing calculator buttons")- return- for idx in [btn_2, btn_add, btn_2, btn_eq]:- ca.call_tool("click", {- "pid": self._calc_pid,- "window_id": window_id,- "element_index": idx,- })- time.sleep(0.15)- snap2 = ca.call_tool("get_window_state", {- "pid": self._calc_pid, "window_id": window_id,- "query": "AXStaticText",- })- tree2 = snap2.get("structuredContent", snap2).get("tree_markdown", "")- driver_a_result["tree"] = tree2- except Exception as e:- driver_a_error.append(str(e))-- def run_driver_b() -> None:- try:- with DriverClient(self.binary) as cb:- # Driver B: move its cursor, list windows, move cursor again.- cb.call_tool("move_cursor", {"x": 200.0, "y": 200.0, "cursor_id": "agent-b"})- wins = cb.call_tool("list_windows", {"on_screen_only": True})- count = len(wins.get("structuredContent", {}).get("windows", []))- cb.call_tool("move_cursor", {"x": 300.0, "y": 300.0, "cursor_id": "agent-b"})- driver_b_result["window_count"] = count- except Exception as e:- driver_b_error.append(str(e))-- # Launch both threads simultaneously.- ta = threading.Thread(target=run_driver_a, daemon=True)- tb = threading.Thread(target=run_driver_b, daemon=True)- ta.start()- tb.start()- ta.join(timeout=30)- tb.join(timeout=30)-- # Both must have completed without error.- self.assertFalse(- driver_a_error,- f"Driver A errored: {driver_a_error}",- )- self.assertFalse(- driver_b_error,- f"Driver B errored: {driver_b_error}",- )-- print(f"\n Driver A result tree:\n{driver_a_result.get('tree', '(empty)')}")- print(f" Driver B window count: {driver_b_result.get('window_count', -1)}")-- # Driver A should have produced "4".- self.assertIn(- "4",- driver_a_result.get("tree", ""),- "Calculator did not show 4 after Driver A clicked 2+2=",- )-- # Driver B should have seen at least some windows.- self.assertGreater(- driver_b_result.get("window_count", 0),- 0,- "Driver B saw no windows in list_windows",- )-- # Focus must not have been stolen.- time.sleep(0.3)- losses = _read_focus_losses()- with DriverClient(self.binary) as c:- active = frontmost_bundle_id(c)- print(f" losses: {self._losses_before}->{losses}, frontmost: {active}")- self.assertEqual(- active, FOCUS_MONITOR_BUNDLE,- f"Focus stolen during concurrent operation — frontmost is {active}",- )-- def test_sequential_driver_reuse(self) -> None:- """Open/close/reopen a DriverClient — state from prior session does not bleed."""-- # First session — clear Calculator.- with DriverClient(self.binary) as c:- window_id = resolve_window_id(c, self._calc_pid)- snap = c.call_tool("get_window_state", {- "pid": self._calc_pid, "window_id": window_id,- })- tree = snap.get("structuredContent", snap).get("tree_markdown", "")- btn_ac = _find_calc_button(tree, "All Clear") or _find_calc_button(tree, "Clear")- if btn_ac is not None:- c.call_tool("click", {- "pid": self._calc_pid, "window_id": window_id, "element_index": btn_ac,- })- time.sleep(0.2)-- time.sleep(0.2)-- # Second session — press 5.- with DriverClient(self.binary) as c:- window_id = resolve_window_id(c, self._calc_pid)- snap = c.call_tool("get_window_state", {- "pid": self._calc_pid, "window_id": window_id,- })- tree = snap.get("structuredContent", snap).get("tree_markdown", "")- btn_5 = _find_calc_button(tree, "5")- self.assertIsNotNone(btn_5, "Could not find '5' button in fresh session")- c.call_tool("click", {- "pid": self._calc_pid, "window_id": window_id, "element_index": btn_5,- })- time.sleep(0.3)- snap2 = c.call_tool("get_window_state", {- "pid": self._calc_pid, "window_id": window_id, "query": "AXStaticText",- })- tree2 = snap2.get("structuredContent", snap2).get("tree_markdown", "")- print(f"\n After pressing 5: {tree2[:200]}")-- self.assertIn("5", tree2, "Calculator did not show 5 after clicking it in a new session")-- # Focus check.- time.sleep(0.3)- with DriverClient(self.binary) as c:- active = frontmost_bundle_id(c)- self.assertEqual(- active, FOCUS_MONITOR_BUNDLE,- f"Focus stolen — frontmost is {active}",- )---if __name__ == "__main__":- unittest.main(verbosity=2)diff --git a/libs/cua-driver/tests/e2e/test_cursor_visibility.py b/libs/cua-driver/tests/e2e/test_cursor_visibility.pydeleted file mode 100644index 1bfdccc316..0000000000--- a/libs/cua-driver/tests/e2e/test_cursor_visibility.py+++ /dev/null@@ -1,426 +0,0 @@-"""Integration test: agent-cursor custom-icon visibility on screen.--Verifies that after:- 1. Setting a custom cursor icon via set_agent_cursor_style- 2. Performing a click (which triggers ClickPulse moving the cursor to the- click point and PinAbove ordering the overlay above the target window)--... the custom cursor icon is visually present at the expected screen position-in a full-screen screenshot.--The test cursor is a bright-magenta circle with a yellow cross — chosen because-it's easy to detect via colour thresholding without needing a heavy CV library.-Only the stdlib + base64/struct/zlib are needed for the screenshot decode.--Run:- CUA_DRIVER_BINARY=../../target/debug/cua-driver python3 -m unittest test_cursor_visibility -v-"""--from __future__ import annotations--import base64-import os-import struct-import subprocess-import sys-import time-import unittest-import zlib--sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))-from driver_client import DriverClient, default_binary_path, resolve_window_id--_THIS_DIR = os.path.dirname(os.path.abspath(__file__))-_FIXTURE_CURSOR = os.path.join(_THIS_DIR, "fixtures", "test_cursor.png")--CALCULATOR_BUNDLE = "com.apple.calculator"--# Colour tolerances for detection-_MAGENTA_MIN = (200, 0, 200) # R≥200, G≤55, B≥200-_MAGENTA_MAX = (255, 55, 255)-_YELLOW_MIN = (200, 200, 0) # R≥200, G≥200, B≤55-_YELLOW_MAX = (255, 255, 55)--# Minimum fraction of pixels in the search region that must match the cursor colours.-_MATCH_THRESHOLD = 0.002 # 0.2% of the search window---def _decode_png_pixels(png_bytes: bytes) -> tuple[int, int, list[tuple[int, int, int]]]:- """Minimal PNG decoder: returns (width, height, [(r,g,b), ...]) for RGB/RGBA PNGs.-- Only supports filter types 0 (None) and 1 (Sub) — which covers the screenshots- produced by cua-driver's `screenshot` tool. Raises ValueError for unsupported formats.- """- if png_bytes[:8] != b'\x89PNG\r\n\x1a\n':- raise ValueError("Not a PNG file")-- pos = 8- width = height = 0- idat_chunks: list[bytes] = []- bit_depth = color_type = 0-- while pos < len(png_bytes):- if pos + 8 > len(png_bytes):- break- length = struct.unpack('>I', png_bytes[pos:pos+4])[0]- chunk_type = png_bytes[pos+4:pos+8]- data = png_bytes[pos+8:pos+8+length]- pos += 12 + length-- if chunk_type == b'IHDR':- width, height = struct.unpack('>II', data[:8])- bit_depth = data[8]- color_type = data[9]- elif chunk_type == b'IDAT':- idat_chunks.append(data)- elif chunk_type == b'IEND':- break-- if width == 0 or height == 0:- raise ValueError("No IHDR found")- if bit_depth != 8:- raise ValueError(f"Unsupported bit depth: {bit_depth}")-- raw = zlib.decompress(b''.join(idat_chunks))-- # Channels: 2=RGB(3), 6=RGBA(4)- channels = {2: 3, 6: 4}.get(color_type)- if channels is None:- raise ValueError(f"Unsupported color_type: {color_type}")-- stride = width * channels- pixels: list[tuple[int, int, int]] = []- prev_row = bytes(stride)-- offset = 0- for _y in range(height):- filt = raw[offset]- offset += 1- row = bytearray(raw[offset:offset+stride])- offset += stride-- if filt == 0:- pass # None- elif filt == 1:- for i in range(channels, stride):- row[i] = (row[i] + row[i - channels]) & 0xFF- elif filt == 2:- for i in range(stride):- row[i] = (row[i] + prev_row[i]) & 0xFF- elif filt == 3:- for i in range(stride):- a = row[i - channels] if i >= channels else 0- b = prev_row[i]- row[i] = (row[i] + (a + b) // 2) & 0xFF- elif filt == 4:- for i in range(stride):- a = row[i - channels] if i >= channels else 0- b = prev_row[i]- c = prev_row[i - channels] if i >= channels else 0- p = a + b - c- pa = abs(p - a); pb = abs(p - b); pc = abs(p - c)- pr = a if pa <= pb and pa <= pc else (b if pb <= pc else c)- row[i] = (row[i] + pr) & 0xFF- else:- raise ValueError(f"Unknown PNG filter type: {filt}")-- for i in range(0, stride, channels):- pixels.append((row[i], row[i+1], row[i+2]))- prev_row = bytes(row)-- return width, height, pixels---def _colour_in_region(- pixels: list[tuple[int, int, int]],- img_w: int,- img_h: int,- cx: float,- cy: float,- radius: int,- colour_min: tuple[int, int, int],- colour_max: tuple[int, int, int],-) -> float:- """Return fraction of pixels in circle around (cx, cy) that match the colour range."""- x0 = max(0, int(cx) - radius)- x1 = min(img_w - 1, int(cx) + radius)- y0 = max(0, int(cy) - radius)- y1 = min(img_h - 1, int(cy) + radius)-- total = 0- matches = 0- for y in range(y0, y1 + 1):- for x in range(x0, x1 + 1):- idx = y * img_w + x- if idx >= len(pixels):- continue- r, g, b = pixels[idx]- total += 1- if (colour_min[0] <= r <= colour_max[0] and- colour_min[1] <= g <= colour_max[1] and- colour_min[2] <= b <= colour_max[2]):- matches += 1-- return matches / total if total > 0 else 0.0---class TestCursorVisibility(unittest.TestCase):- """Custom cursor icon is visible on screen after a click."""-- @classmethod- def setUpClass(cls) -> None:- if not os.path.exists(_FIXTURE_CURSOR):- cls.skipTest(cls, f"test cursor fixture missing: {_FIXTURE_CURSOR}") # type: ignore[arg-type]- subprocess.run(["pkill", "-x", "Calculator"], check=False)- time.sleep(0.3)-- @classmethod- def tearDownClass(cls) -> None:- subprocess.run(["pkill", "-x", "Calculator"], check=False)-- def setUp(self) -> None:- self.client = DriverClient(default_binary_path()).__enter__()- # Verify set_agent_cursor_style is available.- tools = {t["name"] for t in self.client.list_tools()}- if "set_agent_cursor_style" not in tools:- self.client.__exit__(None, None, None)- self.skipTest("set_agent_cursor_style tool not available in this build")-- def tearDown(self) -> None:- subprocess.run(["pkill", "-x", "Calculator"], check=False)- self.client.__exit__(None, None, None)-- def test_custom_cursor_visible_after_click(self) -> None:- """After set_agent_cursor_style + click, the custom icon is on-screen."""- client = self.client-- # 1. Enable cursor overlay.- r = client.call_tool("set_agent_cursor_enabled", {"enabled": True})- self.assertFalse(r.get("isError"), f"set_agent_cursor_enabled failed: {r}")-- # 2. Upload the bright-magenta test cursor.- r = client.call_tool("set_agent_cursor_style", {"image_path": _FIXTURE_CURSOR})- self.assertFalse(r.get("isError"), f"set_agent_cursor_style failed: {r}")- text = r.get("content", [{}])[0].get("text", "")- self.assertIn("✅", text, f"set_agent_cursor_style didn't return ✅: {text}")- self.assertIn(_FIXTURE_CURSOR, text, f"image_path not echoed in response: {text}")-- # 3. Launch Calculator in the background.- r = client.call_tool("launch_app", {"bundle_id": CALCULATOR_BUNDLE})- self.calc_pid = r["structuredContent"]["pid"]- time.sleep(0.6)-- # 4. Resolve the Calculator window and its screen bounds.- win_id = resolve_window_id(client, self.calc_pid)- wins = client.call_tool("list_windows", {"pid": self.calc_pid})["structuredContent"]["windows"]- win = next((w for w in wins if w["window_id"] == win_id), None)- self.assertIsNotNone(win, "Could not find Calculator window")- bounds = win["bounds"]-- # 5. Click the center of the Calculator window (window-local coords).- click_x = bounds["width"] / 2.0- click_y = bounds["height"] / 2.0- r = client.call_tool("click", {- "pid": self.calc_pid,- "window_id": win_id,- "x": click_x,- "y": click_y,- })- self.assertFalse(r.get("isError"), f"click failed: {r}")-- # 6. Compute expected screen coordinates of the click point.- # The driver translates window-local coords to screen coords by adding- # the window origin and dividing by the Retina scale (detected at click time).- # We use the same heuristic: try a scale of 2.0 (Retina), fall back to 1.0.- # If the click lands within ±64 px of our estimate it's still detectable.- win_x = bounds["x"]- win_y = bounds["y"]- # Best-effort Retina scale detection via screenshot width vs logical width.- try:- scr = client.call_tool("screenshot", {"pid": self.calc_pid, "window_id": win_id})- b64 = scr.get("content", [{}])[0].get("data", "")- if b64:- png = base64.b64decode(b64)- # PNG width is at bytes 16..20 in big-endian.- png_w = struct.unpack('>I', png[16:20])[0]- scale = png_w / bounds["width"] if bounds["width"] > 0 else 1.0- else:- scale = 1.0- except Exception:- scale = 1.0-- screen_x = win_x + click_x / scale- screen_y = win_y + click_y / scale-- # 7. Wait for cursor animation + render loop to settle.- time.sleep(0.5)-- # 8. Take a full-screen screenshot. Force PNG so the per-pixel decode- # below works — the screenshot tool defaults to JPEG, which would be- # lossy and skew the cursor-colour search.- scr = client.call_tool("screenshot", {"format": "png"})- b64 = scr.get("content", [{}])[0].get("data", "")- self.assertTrue(b64, "screenshot returned no data")- png_bytes = base64.b64decode(b64)-- # 9. Decode the PNG and check for our cursor colours near (screen_x, screen_y).- try:- img_w, img_h, pixels = _decode_png_pixels(png_bytes)- except Exception as e:- self.skipTest(f"PNG decode failed (unsupported screenshot format): {e}")-- # The screenshot may be at 2x resolution; scale screen_x/y accordingly.- ss_scale = img_w / _get_screen_width(client)- sx = screen_x * ss_scale- sy = screen_y * ss_scale- search_r = int(80 * ss_scale)-- magenta_frac = _colour_in_region(- pixels, img_w, img_h, sx, sy, search_r,- _MAGENTA_MIN, _MAGENTA_MAX,- )- yellow_frac = _colour_in_region(- pixels, img_w, img_h, sx, sy, search_r,- _YELLOW_MIN, _YELLOW_MAX,- )-- self.assertGreater(- magenta_frac, _MATCH_THRESHOLD,- f"Magenta cursor colour NOT found near click point "- f"(screen={screen_x:.0f},{screen_y:.0f} img={sx:.0f},{sy:.0f} "- f"scale={ss_scale:.2f}). "- f"Magenta fraction={magenta_frac:.5f} (threshold={_MATCH_THRESHOLD}). "- f"Yellow fraction={yellow_frac:.5f}. "- f"The overlay may be hidden or behind another window.",- )- self.assertGreater(- yellow_frac, _MATCH_THRESHOLD,- f"Yellow cross NOT found in cursor region "- f"(magenta={magenta_frac:.5f}, yellow={yellow_frac:.5f}). "- f"Cursor shape may not have loaded correctly.",- )-- def test_default_cursor_visible_after_click(self) -> None:- """After a plain click (no custom icon), the default gradient arrow is on-screen.-- Detects the ice-blue bloom/arrow colour (cursor_mid ≈ R94 G192 B232) that the- built-in gradient arrow uses. Does NOT set a custom cursor icon.- """- client = self.client-- # 1. Enable overlay with default cursor (no custom icon).- r = client.call_tool("set_agent_cursor_enabled", {"enabled": True})- self.assertFalse(r.get("isError"), f"set_agent_cursor_enabled failed: {r}")-- # Ensure no custom icon is active (revert to default arrow).- r = client.call_tool("set_agent_cursor_style", {"image_path": ""})- self.assertFalse(r.get("isError"), f"revert to default cursor failed: {r}")-- # 2. Launch Calculator in the background.- r = client.call_tool("launch_app", {"bundle_id": CALCULATOR_BUNDLE})- self.calc_pid = r["structuredContent"]["pid"]- time.sleep(0.6)-- # 3. Resolve window and screen bounds.- win_id = resolve_window_id(client, self.calc_pid)- wins = client.call_tool("list_windows", {"pid": self.calc_pid})["structuredContent"]["windows"]- win = next((w for w in wins if w["window_id"] == win_id), None)- self.assertIsNotNone(win, "Could not find Calculator window")- bounds = win["bounds"]-- click_x = bounds["width"] / 2.0- click_y = bounds["height"] / 2.0-- # Detect Retina scale.- try:- scr = client.call_tool("screenshot", {"pid": self.calc_pid, "window_id": win_id})- b64 = scr.get("content", [{}])[0].get("data", "")- png = base64.b64decode(b64) if b64 else b""- png_w = struct.unpack('>I', png[16:20])[0] if len(png) >= 24 else 0- scale = png_w / bounds["width"] if bounds["width"] > 0 and png_w > 0 else 1.0- except Exception:- scale = 1.0-- # 4. Click the centre of the window.- r = client.call_tool("click", {- "pid": self.calc_pid,- "window_id": win_id,- "x": click_x,- "y": click_y,- })- self.assertFalse(r.get("isError"), f"click failed: {r}")-- # 5. Wait for 750ms glide + spring settle + a safety margin.- time.sleep(1.2)-- # 6. Full-screen screenshot.- scr = client.call_tool("screenshot", {})- b64 = scr.get("content", [{}])[0].get("data", "")- self.assertTrue(b64, "screenshot returned no data")- png_bytes = base64.b64decode(b64)-- try:- img_w, img_h, pixels = _decode_png_pixels(png_bytes)- except Exception as e:- self.skipTest(f"PNG decode failed: {e}")-- win_x, win_y = bounds["x"], bounds["y"]- screen_x = win_x + click_x / scale- screen_y = win_y + click_y / scale-- ss_scale = img_w / _get_screen_width(client)- sx = screen_x * ss_scale- sy = screen_y * ss_scale- search_r = int(80 * ss_scale)-- # Default arrow: ice-blue/cyan — cursor_mid ≈ (94, 192, 232), bloom ≈ (188, 232, 252).- # Accept any pixel that is clearly blue-cyan (G and B significantly exceed R).- _CYAN_MIN = (40, 150, 180)- _CYAN_MAX = (180, 255, 255)-- cyan_frac = _colour_in_region(- pixels, img_w, img_h, sx, sy, search_r,- _CYAN_MIN, _CYAN_MAX,- )-- self.assertGreater(- cyan_frac, _MATCH_THRESHOLD,- f"Default ice-blue cursor NOT found near click point "- f"(screen={screen_x:.0f},{screen_y:.0f} img_pt={sx:.0f},{sy:.0f} "- f"scale={ss_scale:.2f}). "- f"Cyan fraction={cyan_frac:.5f} (threshold={_MATCH_THRESHOLD}). "- f"The default gradient arrow overlay may be invisible.",- )-- def test_set_agent_cursor_style_gradient_only(self) -> None:- """set_agent_cursor_style with gradient_colors updates without error."""- r = self.client.call_tool("set_agent_cursor_style", {- "gradient_colors": ["#FF0000", "#FF00FF", "#0000FF"],- "bloom_color": "#00FFFF",- })- self.assertFalse(r.get("isError"), f"gradient-only style update failed: {r}")- text = r.get("content", [{}])[0].get("text", "")- self.assertIn("✅", text)- self.assertIn("#FF0000", text)- self.assertIn("#00FFFF", text)-- def test_set_agent_cursor_style_revert(self) -> None:- """Empty image_path reverts the cursor to the procedural arrow without error."""- # First set a custom icon.- self.client.call_tool("set_agent_cursor_style", {"image_path": _FIXTURE_CURSOR})- # Then revert.- r = self.client.call_tool("set_agent_cursor_style", {"image_path": ""})- self.assertFalse(r.get("isError"), f"revert failed: {r}")- text = r.get("content", [{}])[0].get("text", "")- self.assertIn("✅", text)- self.assertIn("reverted", text)---def _get_screen_width(client: DriverClient) -> float:- """Return the logical screen width in points."""- r = client.call_tool("get_screen_size")- return float(r["structuredContent"].get("width", 1920))---if __name__ == "__main__":- unittest.main(verbosity=2)diff --git a/libs/cua-driver/tests/e2e/test_double_click_delivery.py b/libs/cua-driver/tests/e2e/test_double_click_delivery.pydeleted file mode 100644index f272bd859d..0000000000--- a/libs/cua-driver/tests/e2e/test_double_click_delivery.py+++ /dev/null@@ -1,253 +0,0 @@-"""Integration test: `double_click` tool delivers two presses to a-backgrounded target.--Two addressing modes, one invariant: double-clicking Calculator's "2"-button from behind FocusMonitorApp should produce "22" in the display-and must not steal focus.--- Pixel path: `double_click({pid, x, y})` → synthesized CGEvent double-click.-- Element-index path on an AXButton → performs two AX presses.--Run:- CUA_DRIVER_BINARY=../../target/debug/cua-driver python3 -m unittest test_double_click_delivery -v-"""--from __future__ import annotations--import os-import re-import subprocess-import sys-import time-import unittest--sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))-from driver_client import (- DriverClient,- default_binary_path,- frontmost_bundle_id,- resolve_window_id,-)--_THIS_DIR = os.path.dirname(os.path.abspath(__file__))-_DRIVER_RS_ROOT = os.path.dirname(os.path.dirname(_THIS_DIR))-_LIBS_ROOT = os.path.dirname(_DRIVER_RS_ROOT)-_FOCUS_APP_DIR = os.path.join(_LIBS_ROOT, "cua-driver", "Tests", "FocusMonitorApp")-_FOCUS_APP_EXE = os.path.join(- _FOCUS_APP_DIR, "FocusMonitorApp.app", "Contents", "MacOS", "FocusMonitorApp"-)-_LOSS_FILE = "/tmp/focus_monitor_losses.txt"--CALC_BUNDLE = "com.apple.calculator"-FOCUS_MONITOR_BUNDLE = "com.trycua.FocusMonitorApp"--CALC_TWO_BUTTON_XY = (86, 313)-CALC_DEFAULT_W = 230-CALC_DEFAULT_H = 408---def _build_focus_app() -> None:- if not os.path.exists(_FOCUS_APP_EXE):- subprocess.run(- [os.path.join(_FOCUS_APP_DIR, "build.sh")], check=True- )---def _launch_focus_app() -> tuple[subprocess.Popen, int]:- proc = subprocess.Popen(- [_FOCUS_APP_EXE],- stdout=subprocess.PIPE,- stderr=subprocess.DEVNULL,- text=True,- )- for _ in range(40):- line = proc.stdout.readline().strip()- if line.startswith("FOCUS_PID="):- return proc, int(line.split("=", 1)[1])- time.sleep(0.1)- proc.terminate()- raise RuntimeError("FocusMonitorApp did not print FOCUS_PID in time")---def _read_focus_losses() -> int:- try:- with open(_LOSS_FILE) as f:- return int(f.read().strip())- except (FileNotFoundError, ValueError):- return -1---def _find_calc_button(tree: str, label: str) -> int | None:- for line in tree.split("\n"):- if "AXButton" not in line:- continue- m = re.search(r"\[(\d+)\]", line)- if not m:- continue- if f"({label})" in line or f"id={label}" in line:- return int(m.group(1))- return None---def _calc_display(client: DriverClient, pid: int) -> str:- window_id = resolve_window_id(client, pid)- result = client.call_tool(- "get_window_state",- {"pid": pid, "window_id": window_id, "query": "AXStaticText"},- )- tree = result.get("structuredContent", result).get("tree_markdown", "")- for line in tree.split("\n"):- if "AXStaticText" in line and "=" in line:- m = re.search(r'= "([^"]*)"', line)- if m:- return m.group(1).replace("\u200e", "").replace("\u200f", "")- return ""---class DoubleClickDeliveryTests(unittest.TestCase):- """double_click on backgrounded Calculator — focus must not be stolen."""-- _calc_pid: int- _focus_proc: subprocess.Popen- _focus_pid: int- binary: str-- @classmethod- def setUpClass(cls) -> None:- _build_focus_app()- try:- os.remove(_LOSS_FILE)- except FileNotFoundError:- pass- subprocess.run(["pkill", "-x", "Calculator"], check=False)- # Kill any stale FocusMonitorApp left over from a crashed prior run.- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)- time.sleep(0.3)-- cls.binary = default_binary_path()-- with DriverClient(cls.binary) as c:- result = c.call_tool("launch_app", {"bundle_id": CALC_BUNDLE})- cls._calc_pid = result["structuredContent"]["pid"]- print(f"\n Calculator pid: {cls._calc_pid}")- time.sleep(1.5)-- cls._focus_proc, cls._focus_pid = _launch_focus_app()- print(f" FocusMonitor pid: {cls._focus_pid}")- time.sleep(1.0)- subprocess.run(- ["osascript", "-e", 'tell application "FocusMonitorApp" to activate'],- check=False,- )- time.sleep(0.8)-- with DriverClient(cls.binary) as c:- active = frontmost_bundle_id(c)- assert active == FOCUS_MONITOR_BUNDLE, (- f"Expected FocusMonitorApp frontmost, got {active}"- )- losses = _read_focus_losses()- assert losses == 0, f"Expected 0 focus losses at start, got {losses}"-- @classmethod- def tearDownClass(cls) -> None:- if hasattr(cls, '_focus_proc'):- cls._focus_proc.terminate()- try:- cls._focus_proc.wait(timeout=3)- except subprocess.TimeoutExpired:- cls._focus_proc.kill()- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)- subprocess.run(["pkill", "-x", "Calculator"], check=False)- try:- os.remove(_LOSS_FILE)- except FileNotFoundError:- pass-- def setUp(self) -> None:- with DriverClient(self.binary) as c:- window_id = resolve_window_id(c, self._calc_pid)- snap = c.call_tool(- "get_window_state",- {"pid": self._calc_pid, "window_id": window_id},- )- tree = snap.get("structuredContent", snap).get("tree_markdown", "")- btn = _find_calc_button(tree, "All Clear") or _find_calc_button(tree, "Clear")- if btn is not None:- c.call_tool("click", {- "pid": self._calc_pid,- "window_id": window_id,- "element_index": btn,- })- time.sleep(0.3)- subprocess.run(- ["osascript", "-e", 'tell application "FocusMonitorApp" to activate'],- check=False,- )- time.sleep(0.5)- self._losses_before = _read_focus_losses()-- def _assert_no_focus_steal(self) -> None:- time.sleep(0.3)- losses = _read_focus_losses()- with DriverClient(self.binary) as c:- active = frontmost_bundle_id(c)- print(f" losses: {self._losses_before}->{losses}, frontmost: {active}")- self.assertEqual(- active, FOCUS_MONITOR_BUNDLE,- f"Focus stolen — frontmost is {active}",- )-- def test_element_index_double_click_no_focus_steal(self) -> None:- """double_click via element_index on Calculator '2' button — no focus steal."""- with DriverClient(self.binary) as c:- window_id = resolve_window_id(c, self._calc_pid)- snap = c.call_tool("get_window_state", {- "pid": self._calc_pid, "window_id": window_id,- })- tree = snap.get("structuredContent", snap).get("tree_markdown", "")- two_button = _find_calc_button(tree, "2")- self.assertIsNotNone(two_button, "Could not locate Calculator's '2' AXButton")-- result = c.call_tool("double_click", {- "pid": self._calc_pid,- "window_id": window_id,- "element_index": two_button,- })- self.assertIsNone(result.get("isError"), msg=result)- text = result.get("content", [{}])[0].get("text", "")- print(f"\n double_click result: {text}")- self.assertTrue(text.startswith("✅"), f"Expected ✅ prefix, got: {text}")-- self._assert_no_focus_steal()-- def test_pixel_double_click_no_focus_steal(self) -> None:- """double_click via pixel coords — no focus steal."""- with DriverClient(self.binary) as c:- window_id = resolve_window_id(c, self._calc_pid)- snap = c.call_tool("get_window_state", {- "pid": self._calc_pid, "window_id": window_id,- })- sc = snap.get("structuredContent", snap)- w = sc.get("screenshot_width", CALC_DEFAULT_W)- h = sc.get("screenshot_height", CALC_DEFAULT_H)-- bx, by = CALC_TWO_BUTTON_XY- x = int(bx * w / CALC_DEFAULT_W)- y = int(by * h / CALC_DEFAULT_H)- result = c.call_tool("double_click", {- "pid": self._calc_pid,- "window_id": window_id,- "x": x,- "y": y,- })- self.assertIsNone(result.get("isError"), msg=result)- text = result.get("content", [{}])[0].get("text", "")- print(f"\n double_click pixel result: {text}")- self.assertTrue(text.startswith("✅"), f"Expected ✅ prefix, got: {text}")-- self._assert_no_focus_steal()---if __name__ == "__main__":- unittest.main(verbosity=2)diff --git a/libs/cua-driver/tests/e2e/test_electron.py b/libs/cua-driver/tests/e2e/test_electron.pydeleted file mode 100644index dd9d55e9ea..0000000000--- a/libs/cua-driver/tests/e2e/test_electron.py+++ /dev/null@@ -1,143 +0,0 @@-"""Electron shared-harness background automation tests — v2 harness.--The Electron test app is built from tests/fixtures/apps/cross-platform/electron-and loads tests/fixtures/shared/web/index.html, the same DOM used by the WebView2-and Rust Electron harness tests.--Run: python3 -m pytest test_electron.py -v-"""--from __future__ import annotations--import subprocess-import sys-import time-import os--import pytest--_HERE = os.path.dirname(os.path.abspath(__file__))-sys.path.insert(0, _HERE)--[email protected](autouse=True)-def _reactivate_focus(focus_monitor):- _, pid = focus_monitor- subprocess.run(- ["osascript", "-e",- f'tell application "System Events" to set frontmost of (first process whose unix id is {pid}) to true'],- check=False,- )- time.sleep(0.4)---# ── tests ─────────────────────────────────────────────────────────────────────--class TestElectronButton:- def test_click_increments_counter(self, driver, electron_app, ux_guard):- _, pid, _ = electron_app- if pid == 0:- pytest.skip("Electron app pid not found")-- wid = driver.find_window(pid)-- before = driver.get_window_state(pid, wid, query="Increment")- idx = before.find_button("Increment") or before.find_element("Increment")- assert idx is not None, f"'Increment' not found:\n{before.tree}"-- driver.click(pid, wid, element_index=idx)- time.sleep(0.5)-- after = driver.get_window_state(pid, wid)- assert after.has_text("counter=1"), f"Counter not incremented:\n{after.tree}"---class TestElectronTextInput:- def test_type_text(self, driver, electron_app, ux_guard):- _, pid, _ = electron_app- if pid == 0:- pytest.skip("Electron app pid not found")-- wid = driver.find_window(pid)-- before = driver.get_window_state(pid, wid)- idx = before.find_text_field(skip_url_bar=True)- assert idx is not None, f"Text field not found:\n{before.tree}"-- # All keyboard interactions with Electron may crash the driver due to a- # SkyLight SPI issue with Electron processes. Wrap everything in a- # try/except so the test can skip gracefully instead of failing.- try:- driver.click(pid, wid, element_index=idx)- time.sleep(0.4)- driver.call_tool("type_text_chars", {- "pid": pid,- "text": "hello electron",- "delay_ms": 30,- })- except Exception as e:- print(f"\n driver raised {type(e).__name__}: {e}")- pytest.skip(f"Driver crashed interacting with Electron text field (known issue): {e}")-- time.sleep(1.5)-- # Electron/Chromium may temporarily return an empty AX tree while it- # rebuilds the accessibility tree after a DOM update. Retry a few times.- after = None- for attempt in range(5):- try:- wid2 = driver.find_window(pid)- after = driver.get_window_state(pid, wid2)- except Exception:- time.sleep(1.0)- continue- if after and after.tree:- break- print(f"\n AX tree empty (attempt {attempt+1}/5), retrying …")- time.sleep(1.0)-- if after is None or not after.tree:- pytest.skip("AX tree unavailable after typing (Electron AX rebuild timeout)")-- print(f"\n after tree length: {len(after.tree)}, has mirror: {after.has_text('mirror=hello electron')}")- assert after.has_text("mirror=hello electron"), f"Text not in AX tree:\n{after.tree[:500]}"---class TestElectronClickTarget:- def test_click_target_no_focus_steal(self, driver, electron_app, ux_guard):- """Click the shared harness click target without stealing focus."""- _, pid, _ = electron_app- if pid == 0:- pytest.skip("Electron app pid not found")-- wid = driver.find_window(pid)- before = driver.get_window_state(pid, wid, query="Click target")- idx = before.find_element("Click target")- assert idx is not None, f"'Click target' not found:\n{before.tree}"-- result = driver.click(pid, wid, element_index=idx)- time.sleep(0.5)-- after = driver.get_window_state(pid, wid)- print(f" electron click target tree contains last_action: {after.has_text('last_action=left_click')}")-- assert result is not None, "click returned None"- assert after.has_text("last_action=left_click"), f"Click target not updated:\n{after.tree}"- # UX assertions via ux_guard teardown---class TestElectronNoFocusSteal:- def test_interactions_no_focus_steal(self, driver, electron_app, ux_guard):- _, pid, _ = electron_app- if pid == 0:- pytest.skip("Electron app pid not found")-- wid = driver.find_window(pid)- before = driver.get_window_state(pid, wid, query="Increment")- idx = before.find_button("Increment") or before.find_element("Increment")- if idx is not None:- driver.click(pid, wid, element_index=idx)- time.sleep(0.3)- driver.press_key(pid, "tab")- time.sleep(0.2)- # ux_guard teardown will assert_clean()diff --git a/libs/cua-driver/tests/e2e/test_focus_steal_parity.py b/libs/cua-driver/tests/e2e/test_focus_steal_parity.pydeleted file mode 100644index 51ff506f6d..0000000000--- a/libs/cua-driver/tests/e2e/test_focus_steal_parity.py+++ /dev/null@@ -1,356 +0,0 @@-"""Integration parity tests for macOS focus-steal prevention.--`launch_app` must NOT change the user's frontmost application — that's-the contract Swift's `SystemFocusStealPreventer` + 3-phase wrap in-`LaunchAppTool` already enforce. This test suite encodes the same-contract for Rust `cua-driver-rs` so we can catch regressions and-verify the Swift↔Rust parity needed to flip macOS Rust BETA → GA.--Each test runs against the Rust binary (`CUA_DRIVER_BINARY`). When-`CUA_SWIFT_BINARY` is set, the same checks run against the Swift-binary too — both must pass for parity. Set `FOCUS_STEAL_RUST_ONLY=1`-to skip the Swift half (useful while iterating on Rust changes-without a Swift build on the path).--Test cases (mirror the plan's Verification section 1:1):-- 1. test_launch_passive_app_preserves_frontmost — Calculator (passive)- 2. test_launch_self_activating_app_preserves_frontmost — Safari- 3. test_launch_with_url_preserves_frontmost — Safari + about:blank- 4. test_cold_launch_creates_window — kill + relaunch Calculator,- assert window appears (verifies the `oapp` AppleEvent on the- no-URL path)- 5. test_concurrent_launches_independent_suppression — back-to-back- launches, both targets suppressed, frontmost unchanged.- 6. test_deadline_reaps_leaked_entry — exercised by the Rust unit- test `focus_steal::tests::deadline_reaps_leaked_entry`. This file- re-asserts the integration-level behavior: a launch followed by- a stale-state probe doesn't keep re-activating the prior- frontmost.--The frontmost-baseline app is Finder (`com.apple.finder`). It's the-only app guaranteed to be running on every logged-in macOS session-and it never self-activates on launch, which gives us a stable,-predictable "prior" pid between tests. (Earlier iterations of this-suite used a shared FocusMonitorApp fixture under-`libs/cua-driver/Tests/FocusMonitorApp/`, but the dependency on a-built helper made the integration tests flaky to bootstrap on a-fresh runner — Finder removes that dependency.)--Run:- cd libs/cua-driver/tests/e2e- python3 -m unittest test_focus_steal_parity -v-"""--from __future__ import annotations--import os-import shutil-import subprocess-import sys-import time-import unittest--sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))-from driver_client import DriverClient, default_binary_path # noqa: E402---# ── Paths / fixtures ─────────────────────────────────────────────────────────-def _swift_binary() -> str | None:- """Locate the Swift binary, or None if not available."""- from_env = os.environ.get("CUA_SWIFT_BINARY", "")- if from_env and os.path.isfile(from_env):- return from_env- for p in (- os.path.expanduser("~/.local/bin/cua-driver"),- shutil.which("cua-driver") or "",- ):- if p and os.path.isfile(p) and os.access(p, os.X_OK):- return p- return None---def _frontmost_bundle_id() -> str | None:- """Bundle id of the currently-frontmost app, queried via osascript.-- We deliberately don't use the driver's `list_apps` for this — the- driver process itself may not be frontmost yet when invoked via- `call`, and we want a host-truth answer that doesn't go through- the same driver path we're testing.- """- try:- out = subprocess.run(- [- "osascript",- "-e",- 'tell application "System Events" to bundle identifier of first '- "application process whose frontmost is true",- ],- capture_output=True,- text=True,- timeout=5,- )- bid = out.stdout.strip()- return bid or None- except Exception:- return None---def _activate_bundle(bundle_id: str) -> None:- """Bring `bundle_id` to the foreground via osascript."""- subprocess.run(- ["osascript", "-e", f'tell application id "{bundle_id}" to activate'],- timeout=5,- check=False,- )---def _kill_bundle(bundle_id: str) -> None:- """pkill the running app for `bundle_id` if any. Idempotent."""- name = bundle_id.split(".")[-1]- subprocess.run(["pkill", "-x", name], check=False)- subprocess.run(["pkill", "-x", name.capitalize()], check=False)- # Safari sometimes needs a moment for its child helper processes- # (Networking, WebContent) to also exit before LaunchServices will- # cleanly relaunch it from the Cryptex path. The 1s settle below- # is conservative but cheap relative to the launch time itself.- time.sleep(1.0)---def _wait_for_frontmost(bundle_id: str, timeout_s: float = 3.0) -> bool:- deadline = time.time() + timeout_s- while time.time() < deadline:- if _frontmost_bundle_id() == bundle_id:- return True- time.sleep(0.1)- return False---# ── Skipping logic ───────────────────────────────────────────────────────────-def _rust_binary() -> str | None:- p = default_binary_path()- return p if os.path.isfile(p) and os.access(p, os.X_OK) else None---_RUST = _rust_binary()-_SWIFT = _swift_binary()-_RUST_ONLY = os.environ.get("FOCUS_STEAL_RUST_ONLY", "") == "1"---# ── Mixin: the actual test cases ─────────────────────────────────────────────-class _FocusStealMixin:- """Parametrized test cases. Subclasses set `BINARY` and `LABEL`."""-- BINARY: str = ""- LABEL: str = ""-- # A stable "prior frontmost" app id we can reasonably expect every- # macOS host to have running. Finder is always running on a logged-in- # macOS session and never self-activates on launch.- PRIOR_BUNDLE = "com.apple.finder"-- def setUp(self) -> None:- # Bring the baseline app to the foreground, confirm. We retry a- # few times because dev hosts with overlay / focus-grabber apps- # (e.g. Gumbranch, screen recorders, status-bar apps that- # forcibly steal focus on click) can racily re-steal during the- # setup window.- for _ in range(3):- _activate_bundle(self.PRIOR_BUNDLE)- if _wait_for_frontmost(self.PRIOR_BUNDLE, timeout_s=2.0):- return- time.sleep(0.5)- self.skipTest(- f"could not bring {self.PRIOR_BUNDLE} to the foreground after "- "3 attempts — test host has an aggressive focus-grabber that "- "makes the test environment unstable"- )-- def _launch(self, args: dict) -> dict:- with DriverClient(self.BINARY) as client:- return client.call_tool("launch_app", args)-- # ── Case 1: passive app (Calculator) ────────────────────────────────- def test_launch_passive_app_preserves_frontmost(self) -> None:- _kill_bundle("com.apple.calculator")- time.sleep(0.5)- _activate_bundle(self.PRIOR_BUNDLE)- time.sleep(0.5)- baseline = _frontmost_bundle_id()- self.assertEqual(baseline, self.PRIOR_BUNDLE)-- result = self._launch({"bundle_id": "com.apple.calculator"})- self.assertIn("structuredContent", result)- self.assertIn("pid", result["structuredContent"])-- # Settle, then assert the prior frontmost is unchanged.- time.sleep(1.0)- self.assertEqual(- _frontmost_bundle_id(),- self.PRIOR_BUNDLE,- f"{self.LABEL}: Calculator launch stole focus from {self.PRIOR_BUNDLE}",- )-- # ── Case 2: self-activating app (Safari, no URLs) ───────────────────- def test_launch_self_activating_app_preserves_frontmost(self) -> None:- _kill_bundle("com.apple.Safari")- time.sleep(0.5)- _activate_bundle(self.PRIOR_BUNDLE)- time.sleep(0.5)- self.assertEqual(_frontmost_bundle_id(), self.PRIOR_BUNDLE)-- # No URLs — exercises the `openApplicationAtURL` path with the- # `oapp` AppleEvent.- self._launch({"bundle_id": "com.apple.Safari"})- time.sleep(1.5) # allow Safari's reflex NSApp.activate to fire- self.assertEqual(- _frontmost_bundle_id(),- self.PRIOR_BUNDLE,- f"{self.LABEL}: Safari (no URL) launch stole focus from {self.PRIOR_BUNDLE}",- )-- # ── Case 3: launch-with-URL preserves frontmost ─────────────────────- def test_launch_with_url_preserves_frontmost(self) -> None:- _kill_bundle("com.apple.Safari")- time.sleep(0.5)- _activate_bundle(self.PRIOR_BUNDLE)- time.sleep(0.5)- self.assertEqual(_frontmost_bundle_id(), self.PRIOR_BUNDLE)-- result = self._launch(- {"bundle_id": "com.apple.Safari", "urls": ["about:blank"]}- )- # NOTE: the Swift binary on macOS Sonoma+ fails this launch with- # "The application 'Safari' could not be launched because it was- # not found" — that's a Cryptex-app + URL-handoff regression on- # the Swift side (`AppLauncher.launch` attaches an `oapp`- # AppleEvent to the openURLs:withApplicationAtURL: path, which- # LaunchServices rejects for Safari). The Rust port fixes this- # by skipping `oapp` on the URL-handoff path — see commit body- # for `feat(launch_app): wire focus-steal preventer into- # LaunchAppTool`. The Swift binary will fail this assertion;- # treat that as a pre-existing Swift bug, not a Rust regression,- # and surface it via `expectedFailure` on the Swift subclass.- self.assertNotIn("isError", result, f"launch_app reported isError={result}")-- time.sleep(1.5)- self.assertEqual(- _frontmost_bundle_id(),- self.PRIOR_BUNDLE,- f"{self.LABEL}: Safari (about:blank) launch stole focus",- )-- # ── Case 4: cold launch creates a window ────────────────────────────- def test_cold_launch_creates_window(self) -> None:- _kill_bundle("com.apple.calculator")- time.sleep(0.5)- _activate_bundle(self.PRIOR_BUNDLE)- time.sleep(0.5)-- result = self._launch({"bundle_id": "com.apple.calculator"})- windows = result["structuredContent"].get("windows", [])- self.assertGreater(- len(windows),- 0,- f"{self.LABEL}: cold-launched Calculator had no windows — "- "the `oapp` AppleEvent may not be reaching the target",- )-- # ── Case 5: back-to-back launches each preserve frontmost ───────────- def test_concurrent_launches_independent_suppression(self) -> None:- _kill_bundle("com.apple.calculator")- _kill_bundle("com.apple.TextEdit")- time.sleep(0.5)- _activate_bundle(self.PRIOR_BUNDLE)- time.sleep(0.5)- self.assertEqual(_frontmost_bundle_id(), self.PRIOR_BUNDLE)-- # Sequential, not truly concurrent — but each launch reads the- # current frontmost (which should still be `PRIOR_BUNDLE` after- # the first one succeeded if focus-steal worked). If the first- # launch leaked focus, the second's prior would be Calculator- # and the assertion below would fail.- self._launch({"bundle_id": "com.apple.calculator"})- time.sleep(0.7)- self.assertEqual(_frontmost_bundle_id(), self.PRIOR_BUNDLE)-- self._launch({"bundle_id": "com.apple.TextEdit"})- time.sleep(1.0)- self.assertEqual(- _frontmost_bundle_id(),- self.PRIOR_BUNDLE,- f"{self.LABEL}: back-to-back Calculator + TextEdit launches "- "leaked focus from one suppression to the next",- )-- # ── Case 6: deadline reaper integration probe ───────────────────────- def test_deadline_reaps_leaked_entry(self) -> None:- # The fine-grained reap behavior is unit-tested in- # `focus_steal::tests::deadline_reaps_leaked_entry`. This- # integration probe verifies the user-visible consequence:- # launching, waiting 6s, then activating the launched app- # manually does NOT cause an unsolicited re-activation of the- # prior frontmost (which would happen if a leaked entry kept- # firing past the 5s deadline).- _kill_bundle("com.apple.calculator")- time.sleep(0.5)- _activate_bundle(self.PRIOR_BUNDLE)- time.sleep(0.5)-- result = self._launch({"bundle_id": "com.apple.calculator"})- calc_pid = result["structuredContent"]["pid"]- self.assertIsInstance(calc_pid, int)- # Wait past the 5s deadline.- time.sleep(6.0)- # Now manually activate Calculator. If the dispatcher leaked an- # entry, it would yank focus back to PRIOR_BUNDLE here — and- # the assertion below would fail.- _activate_bundle("com.apple.calculator")- time.sleep(1.0)- self.assertEqual(- _frontmost_bundle_id(),- "com.apple.calculator",- f"{self.LABEL}: stale entry yanked focus back after the "- "5s deadline — the reaper or the observer skipped the prune",- )---# ── Concrete subclasses ──────────────────────────────────────────────────────[email protected](_RUST is None, "Rust cua-driver-rs binary not built")-class RustFocusStealTests(_FocusStealMixin, unittest.TestCase):- BINARY = _RUST or ""- LABEL = "rust"--- _SWIFT is None or _RUST_ONLY,- "Swift cua-driver binary not available (set CUA_SWIFT_BINARY) "- "or FOCUS_STEAL_RUST_ONLY=1 set",-)-class SwiftFocusStealTests(_FocusStealMixin, unittest.TestCase):- BINARY = _SWIFT or ""- LABEL = "swift"-- # Pre-existing Swift bug on macOS Sonoma+: launching Safari with- # `urls=["about:blank"]` returns "The application 'Safari' could not- # be launched because it was not found." The Rust port fixes this- # (the `oapp` AppleEvent on the URL-handoff path is incompatible- # with Cryptex-installed apps). Re-define the method on this- # subclass only (NOT via expectedFailure on the mixin attribute —- # that would also taint the Rust subclass) and mark the override- # `@expectedFailure`.- @unittest.expectedFailure- def test_launch_with_url_preserves_frontmost(self) -> None: # type: ignore[override]- super().test_launch_with_url_preserves_frontmost()-- # NOTE: `test_deadline_reaps_leaked_entry` is intentionally NOT- # marked expectedFailure on Swift. Swift's `SystemFocusStealPreventer`- # has no monotonic deadline reaper (PR #1521 is unmerged at port- # time), but in practice the Swift `LaunchAppTool.invoke` calls- # `endSuppression` cleanly before the integration test's 6s wait- # finishes, so the test sometimes passes on Swift anyway. The Rust- # subclass is what locks down the reaper-driven contract.---if __name__ == "__main__":- unittest.main()diff --git a/libs/cua-driver/tests/e2e/test_health_report_mcp.py b/libs/cua-driver/tests/e2e/test_health_report_mcp.pydeleted file mode 100644index 26c1908ba2..0000000000--- a/libs/cua-driver/tests/e2e/test_health_report_mcp.py+++ /dev/null@@ -1,238 +0,0 @@-"""Integration test: `health_report` MCP tool over the stdio protocol.--Boots the real `cua-driver mcp` server, performs the JSON-RPC handshake,-asks `tools/list` for the tool catalog, and exercises the tool with-several argument shapes:-- 1. `tools/list` advertises `health_report` (the consumer-facing- contract is that the tool exists and ships `schema_version="1"` in- its description).-- 2. Calling with no arguments produces a well-formed report with the- documented top-level keys and per-check key shape.-- 3. The `skip` filter is honored end-to-end: a skipped check appears in- the output with `status: "skip"`.-- 4. The `include` filter narrows the report to a single check.-- 5. A documented fail mode is observable end-to-end: when called via- `cua-driver mcp` from a terminal (which is what an XCTest / CI- host looks like on macOS), the `bundle_identity` check fails- because the stdio process is NOT attributed to com.trycua.driver.- The schema guarantees `status="fail"` entries carry a `hint` and- surface the runtime bundle identifier under- `data.bundle_identifier` — both are asserted here. On Windows /- Linux the same check returns `status="skip"`, which we tolerate.-- 6. health_report must NEVER itself set isError, even when every- check fails — that's the whole point of the tool.--Run with the standard integration test harness:- ./run_tests.sh test_health_report_mcp-"""--from __future__ import annotations--import json-import os-import sys-import unittest--sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))-from driver_client import DriverClient, default_binary_path---REQUIRED_TOP_LEVEL_KEYS = {- "schema_version",- "platform",- "driver_version",- "overall",- "checks",-}--REQUIRED_CHECK_KEYS = {"name", "status", "message"}-ALLOWED_STATUS_VALUES = {"pass", "fail", "skip"}-ALLOWED_OVERALL_VALUES = {"ok", "degraded", "failed"}-ALLOWED_PLATFORM_VALUES = {"darwin", "win32", "linux"}---def _structured_or_parsed(result: dict) -> dict:- """Pull the JSON report out of an MCP tool result.-- Tools that emit `structuredContent` (the canonical path here)- return it as a top-level field. We also tolerate a text-content- fallback so if structured emission ever breaks, the test fails- loudly with a clear message rather than crashing on a KeyError.- """- if "structuredContent" in result:- return result["structuredContent"]- for content in result.get("content", []):- if content.get("type") == "text":- try:- return json.loads(content.get("text", ""))- except json.JSONDecodeError:- continue- raise AssertionError(- "health_report response carried neither structuredContent nor a JSON text block: "- + json.dumps(result)- )---class HealthReportMCPTests(unittest.TestCase):- @classmethod- def setUpClass(cls) -> None:- cls.binary = default_binary_path()-- # ── tools/list ────────────────────────────────────────────────────-- def test_tools_list_advertises_health_report(self) -> None:- with DriverClient(self.binary) as client:- tools = client.list_tools()- names = {t["name"] for t in tools}- self.assertIn(- "health_report",- names,- "tools/list must advertise health_report so consumers can discover it",- )- descriptor = next(t for t in tools if t["name"] == "health_report")- # The schema_version commitment lives in the description text —- # that's the consumer-facing stability contract.- description = descriptor.get("description", "")- self.assertIn(- "schema_version",- description,- "tool description must call out the stable schema_version contract",- )-- # ── tools/call with no args ───────────────────────────────────────-- def test_call_with_no_args_returns_schema_shape(self) -> None:- with DriverClient(self.binary) as client:- result = client.call_tool("health_report")- report = _structured_or_parsed(result)-- # Top-level keys exactly match the documented schema.- self.assertEqual(- REQUIRED_TOP_LEVEL_KEYS,- REQUIRED_TOP_LEVEL_KEYS & set(report.keys()),- f"missing top-level keys; got {set(report.keys())!r}",- )- self.assertEqual(report["schema_version"], "1")- self.assertIn(report["platform"], ALLOWED_PLATFORM_VALUES)- self.assertIsInstance(report["driver_version"], str)- self.assertIn(report["overall"], ALLOWED_OVERALL_VALUES)-- checks = report["checks"]- self.assertIsInstance(checks, list)- self.assertGreater(len(checks), 0)-- for check in checks:- self.assertTrue(- REQUIRED_CHECK_KEYS.issubset(check.keys()),- f"check missing required keys: {check!r}",- )- self.assertIn(check["status"], ALLOWED_STATUS_VALUES)- # hint is only required when status == fail.- if check["status"] == "fail":- self.assertIn(- "hint",- check,- f"fail entries must carry a remediation hint: {check!r}",- )- self.assertTrue(- isinstance(check["hint"], str) and check["hint"].strip(),- f"hint must be a non-empty string: {check!r}",- )-- # ── skip filter ───────────────────────────────────────────────────-- def test_skip_filter_marks_check_as_skip(self) -> None:- with DriverClient(self.binary) as client:- result = client.call_tool(- "health_report",- {"skip": ["tcc_accessibility"]},- )- report = _structured_or_parsed(result)- names_to_status = {c["name"]: c["status"] for c in report["checks"]}- self.assertEqual(names_to_status.get("tcc_accessibility"), "skip")-- # ── include filter ───────────────────────────────────────────────-- def test_include_filter_runs_only_named_check(self) -> None:- with DriverClient(self.binary) as client:- result = client.call_tool(- "health_report",- {"include": ["binary_version"]},- )- report = _structured_or_parsed(result)- run_names = {- c["name"] for c in report["checks"] if c["status"] != "skip"- }- # Only `binary_version` should have been actually run; the rest- # show up with status: skip.- self.assertEqual(run_names, {"binary_version"})-- # ── documented fail mode (macOS only) ────────────────────────────-- def test_bundle_identity_fail_mode_under_stdio_mcp(self) -> None:- """On macOS the stdio MCP process spawned by an IDE/CI shell- runs outside CuaDriver.app, so its CFBundleIdentifier is NOT- com.trycua.driver. That's the canonical "wrong attribution"- fail mode the tool is designed to surface — schema-wise:- status=fail + non-empty message + non-empty hint +- data.bundle_identifier present so consumers can detect the- drift without parsing the message text.-- On Windows / Linux the same check returns status=skip ("not- applicable on Windows/Linux"); the test tolerates that.- """- with DriverClient(self.binary) as client:- result = client.call_tool(- "health_report",- {"include": ["bundle_identity"]},- )- report = _structured_or_parsed(result)- bundle_check = next(- c for c in report["checks"] if c["name"] == "bundle_identity"- )- if bundle_check["status"] == "skip":- # Windows / Linux path — `not applicable on <platform>`.- self.assertIn("not applicable", bundle_check.get("message", ""))- return- if bundle_check["status"] == "pass":- # macOS launched inside CuaDriver.app — legitimately passes.- return- # macOS fail mode — schema contract on fail entries.- self.assertEqual(bundle_check["status"], "fail")- self.assertIn("hint", bundle_check)- self.assertTrue(bundle_check["hint"].strip())- data = bundle_check.get("data") or {}- # bundle_identifier surfaces the runtime CFBundleIdentifier so- # consumers can detect drift without parsing the message.- # Optional only when the process literally has no bundle id;- # Rust-built binaries always have an executable_path.- self.assertTrue(- "bundle_identifier" in data or "executable_path" in data,- "fail mode must surface either bundle_identifier or executable_path in `data`",- )-- # ── invocation surface itself doesn't error ──────────────────────-- def test_call_does_not_set_isError(self) -> None:- """`health_report` is the consumer's "what's broken?" probe —- it must never itself report `isError: true`, even when every- check fails. Otherwise consumers can't tell apart "the tool- itself is broken" from "the driver has degraded health" —- defeating the whole point of the tool.- """- with DriverClient(self.binary) as client:- result = client.call_tool("health_report")- self.assertFalse(- result.get("isError", False),- f"health_report must never set isError; got {result!r}",- )---if __name__ == "__main__":- unittest.main()diff --git a/libs/cua-driver/tests/e2e/test_list_windows.py b/libs/cua-driver/tests/e2e/test_list_windows.pydeleted file mode 100644index a9f5c8d59b..0000000000--- a/libs/cua-driver/tests/e2e/test_list_windows.py+++ /dev/null@@ -1,121 +0,0 @@-"""Integration test: `list_windows` tool.--Exercises three contracts against the real driver over stdio MCP:-- 1. Default call returns at least a handful of windows across multiple- pids, every record layer-0 with the correct field names.- 2. `pid` filter narrows the result to that pid only.- 3. `on_screen_only: true` drops hidden / minimized / off-Space windows.--Calculator is launched once (setUpClass) so all five tests share one-driver session — avoids the 13-second Calculator restart per-test.--Run:- CUA_DRIVER_BINARY=../../target/debug/cua-driver python3 -m unittest test_list_windows -v-"""--from __future__ import annotations--import os-import subprocess-import sys-import time-import unittest--sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))-from driver_client import DriverClient, default_binary_path---CALCULATOR_BUNDLE = "com.apple.calculator"---class ListWindowsTests(unittest.TestCase):- client: DriverClient- calc_pid: int-- @classmethod- def setUpClass(cls) -> None:- subprocess.run(["pkill", "-x", "Calculator"], check=False)- time.sleep(0.3)- cls.client = DriverClient(default_binary_path()).__enter__()- result = cls.client.call_tool("launch_app", {"bundle_id": CALCULATOR_BUNDLE})- cls.calc_pid = result["structuredContent"]["pid"]- time.sleep(0.5)-- @classmethod- def tearDownClass(cls) -> None:- cls.client.__exit__(None, None, None)- subprocess.run(["pkill", "-x", "Calculator"], check=False)-- def _call_list_windows(self, **args):- return self.client.call_tool("list_windows", args)["structuredContent"]-- def test_default_returns_layer_zero_windows_with_field_names(self):- body = self._call_list_windows()- windows = body["windows"]- self.assertGreater(- len(windows), 0, "expected at least Calculator's window"- )-- required = {- "window_id", "pid", "app_name", "title", "bounds",- "layer", "z_index", "is_on_screen",- }- for w in windows:- self.assertTrue(- required.issubset(w.keys()),- f"window record missing fields: {required - set(w.keys())}",- )- self.assertEqual(w["layer"], 0)-- def test_pid_filter_narrows_to_one_pid(self):- body = self._call_list_windows(pid=self.calc_pid)- windows = body["windows"]- self.assertGreater(- len(windows), 0, "Calculator should have at least one window"- )- self.assertTrue(- all(w["pid"] == self.calc_pid for w in windows),- "pid filter leaked other pids into the result",- )- self.assertTrue(- all(w["app_name"] == "Calculator" for w in windows),- "app_name disagrees with the pid filter",- )-- def test_on_screen_only_drops_off_screen_entries(self):- everything = self._call_list_windows()["windows"]- only_visible = self._call_list_windows(on_screen_only=True)["windows"]-- self.assertLessEqual(len(only_visible), len(everything))- for w in only_visible:- self.assertTrue(- w["is_on_screen"],- "on_screen_only=true returned a window with is_on_screen=false",- )-- def test_bounds_fields_present_and_non_negative(self):- body = self._call_list_windows(pid=self.calc_pid)- windows = body["windows"]- self.assertGreater(len(windows), 0)- for w in windows:- b = w["bounds"]- for field in ("x", "y", "width", "height"):- self.assertIn(field, b, f"bounds missing '{field}'")- self.assertGreaterEqual(- b[field], 0,- f"bounds.{field} should be non-negative, got {b[field]}",- )-- def test_z_index_positive(self):- body = self._call_list_windows()- windows = body["windows"]- for w in windows:- self.assertGreater(- w["z_index"], 0,- f"z_index should be > 0 (front-to-back ordering), got {w['z_index']}",- )---if __name__ == "__main__":- unittest.main(verbosity=2)diff --git a/libs/cua-driver/tests/e2e/test_overlay_z_order.py b/libs/cua-driver/tests/e2e/test_overlay_z_order.pydeleted file mode 100644index e7121daf0f..0000000000--- a/libs/cua-driver/tests/e2e/test_overlay_z_order.py+++ /dev/null@@ -1,195 +0,0 @@-"""Integration test: agent-cursor overlay z-ordering.--Verifies that after driving a backgrounded window the cua-driver-rs overlay-is z-ordered JUST ABOVE the target window (NSWindowLevel.normal + ordered-above target) rather than at NSWindowLevel.floating (above ALL normal windows).--Expected ordering after pinning above a background window:- [... , target-window , overlay , windows-that-were-above-target , ...]--Two assertions are verified:--1. **overlay appears at layer 0** — the overlay's CGWindowLayer is 0- (NSWindowLevel.normal), so it shows up in list_windows. At .floating- (level 3) it would be absent from the layer-0 filtered list.--2. **sandwich ordering** — overlay z_index > target z_index, and any- window that was above the target BEFORE the click remains above the- overlay AFTER the click.--Run:- CUA_DRIVER_BINARY=../../target/debug/cua-driver python3 -m unittest test_overlay_z_order -v-"""--from __future__ import annotations--import os-import subprocess-import sys-import time-import unittest--sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))-from driver_client import DriverClient, default_binary_path, resolve_window_id--_THIS_DIR = os.path.dirname(os.path.abspath(__file__))-_DRIVER_RS_ROOT = os.path.dirname(os.path.dirname(_THIS_DIR))-_LIBS_ROOT = os.path.dirname(_DRIVER_RS_ROOT)-_FOCUS_APP_DIR = os.path.join(_LIBS_ROOT, "cua-driver", "Tests", "FocusMonitorApp")-_FOCUS_APP_EXE = os.path.join(- _FOCUS_APP_DIR, "FocusMonitorApp.app", "Contents", "MacOS", "FocusMonitorApp"-)--CALCULATOR_BUNDLE = "com.apple.calculator"---def _ensure_focus_app_built() -> None:- src = os.path.join(_FOCUS_APP_DIR, "FocusMonitorApp.swift")- if not os.path.exists(_FOCUS_APP_EXE) or (- os.path.exists(src) and os.path.getmtime(src) > os.path.getmtime(_FOCUS_APP_EXE)- ):- subprocess.run([os.path.join(_FOCUS_APP_DIR, "build.sh")], check=True)---def _launch_focus_app() -> tuple[subprocess.Popen, int]:- proc = subprocess.Popen(- [_FOCUS_APP_EXE], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True- )- for _ in range(40):- line = proc.stdout.readline().strip()- if line.startswith("FOCUS_PID="):- return proc, int(line.split("=", 1)[1])- time.sleep(0.1)- proc.terminate()- raise RuntimeError("FocusMonitorApp did not print FOCUS_PID= in time")---class TestOverlayZOrder(unittest.TestCase):- """Overlay is sandwiched just above the target window, not floating above all."""-- _focus_proc: subprocess.Popen- _focus_pid: int-- @classmethod- def setUpClass(cls) -> None:- _ensure_focus_app_built()- subprocess.run(["pkill", "-x", "Calculator"], check=False)- time.sleep(0.3)-- @classmethod- def tearDownClass(cls) -> None:- subprocess.run(["pkill", "-x", "Calculator"], check=False)-- def setUp(self) -> None:- # Kill any stale FocusMonitorApp left over from a crashed prior run.- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)- time.sleep(0.2)-- self.client = DriverClient(default_binary_path()).__enter__()- self.client.call_tool("set_agent_cursor_enabled", {"enabled": True})-- r = self.client.call_tool("launch_app", {"bundle_id": CALCULATOR_BUNDLE})- self.calc_pid = r["structuredContent"]["pid"]- time.sleep(0.5)-- self._focus_proc, self._focus_pid = _launch_focus_app()- time.sleep(0.5)-- def tearDown(self) -> None:- if hasattr(self, '_focus_proc'):- self._focus_proc.terminate()- try:- self._focus_proc.wait(timeout=3)- except subprocess.TimeoutExpired:- self._focus_proc.kill()- self._focus_proc.wait(timeout=1)- # Close the stdout pipe opened during launch.- if self._focus_proc.stdout:- self._focus_proc.stdout.close()- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)- subprocess.run(["pkill", "-x", "Calculator"], check=False)- self.client.__exit__(None, None, None)-- def test_overlay_at_normal_level_above_target(self) -> None:- """Overlay appears at layer=0 (not .floating/layer=3) and z_index > target."""- all_before = self._all_on_screen_windows()- driver_pid = self.client.process.pid-- calc_wins_before = [w for w in all_before if w["pid"] == self.calc_pid]- self.assertTrue(calc_wins_before, "Calculator window not visible before click")- calc_z_before = max(w["z_index"] for w in calc_wins_before)- calc_win_before = max(calc_wins_before, key=lambda w: w["z_index"])- calc_win_id = calc_win_before["window_id"]-- b = calc_win_before["bounds"]- cx = b["width"] / 2.0- cy = b["height"] / 2.0-- fg_wins_before = [- w for w in all_before- if w["z_index"] > calc_z_before- and w["pid"] != driver_pid- and w.get("is_on_screen")- ]-- self.client.call_tool("click", {- "pid": self.calc_pid,- "window_id": calc_win_id,- "x": cx,- "y": cy,- })-- # Let defensive-repin ticks fully settle.- time.sleep(1.5)-- all_after = self._all_on_screen_windows()- overlay_wins = [w for w in all_after if w["pid"] == driver_pid]- self.assertTrue(- overlay_wins,- f"cua-driver-rs overlay NOT visible in list_windows at layer=0 "- f"(driver pid={driver_pid}). "- f"Expected NSWindowLevel.normal overlay to appear as a layer-0 window.",- )-- overlay_z = max(w["z_index"] for w in overlay_wins)-- calc_wins_after = [w for w in all_after if w["pid"] == self.calc_pid]- calc_z_after = (- max(w["z_index"] for w in calc_wins_after) if calc_wins_after else 0- )- self.assertGreater(- overlay_z,- calc_z_after,- f"overlay (z={overlay_z}) must be ABOVE Calculator (z={calc_z_after}).",- )-- missed_above = []- for w_before in fg_wins_before:- w_after = next(- (a for a in all_after if a["window_id"] == w_before["window_id"]),- None,- )- if w_after and w_after.get("is_on_screen"):- if w_after["z_index"] <= overlay_z:- missed_above.append(- f"{w_before['app_name']} "- f"(window_id={w_before['window_id']}, "- f"z_before={w_before['z_index']}, "- f"z_after={w_after['z_index']}, overlay_z={overlay_z})"- )-- if fg_wins_before and missed_above:- self.fail(- "Window(s) that were ABOVE Calculator before the click are now "- "BELOW OR EQUAL TO the overlay — overlay is NOT sandwiched:\n"- + "\n".join(f" {m}" for m in missed_above)- + "\nExpected ordering: [target, overlay, fg-windows]."- )-- def _all_on_screen_windows(self) -> list[dict]:- result = self.client.call_tool("list_windows", {"on_screen_only": True})- return result["structuredContent"]["windows"]---if __name__ == "__main__":- unittest.main(verbosity=2)diff --git a/libs/cua-driver/tests/e2e/test_pixel_click_delivery.py b/libs/cua-driver/tests/e2e/test_pixel_click_delivery.pydeleted file mode 100644index 985047e868..0000000000--- a/libs/cua-driver/tests/e2e/test_pixel_click_delivery.py+++ /dev/null@@ -1,276 +0,0 @@-"""Integration test: pixel-click delivery to backgrounded apps.--Sends pixel-addressed clicks (x, y — NOT element_index) to Calculator-while it's backgrounded behind FocusMonitorApp, then asserts:- - Calculator display changed (click delivery).- - FocusMonitorApp is still frontmost (no focus steal).--Note: Calculator is a Mac Catalyst app. CGEventPostToPid alone doesn't-reach Catalyst. This test documents the current behaviour and will pass-once SkyLight SPI event posting is implemented.--Run:- CUA_DRIVER_BINARY=../../target/debug/cua-driver python3 -m unittest test_pixel_click_delivery -v-"""--from __future__ import annotations--import os-import re-import subprocess-import sys-import time-import unittest--sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))-from driver_client import (- DriverClient,- default_binary_path,- frontmost_bundle_id,- resolve_window_id,-)--_THIS_DIR = os.path.dirname(os.path.abspath(__file__))-_DRIVER_RS_ROOT = os.path.dirname(os.path.dirname(_THIS_DIR))-_LIBS_ROOT = os.path.dirname(_DRIVER_RS_ROOT)-_FOCUS_APP_DIR = os.path.join(_LIBS_ROOT, "cua-driver", "Tests", "FocusMonitorApp")-_FOCUS_APP_EXE = os.path.join(- _FOCUS_APP_DIR, "FocusMonitorApp.app", "Contents", "MacOS", "FocusMonitorApp"-)-_LOSS_FILE = "/tmp/focus_monitor_losses.txt"--CALC_BUNDLE = "com.apple.calculator"-FOCUS_MONITOR_BUNDLE = "com.trycua.FocusMonitorApp"--# Calculator standard-mode (230x408) button grid.-# Column centers (x): 29, 86, 144, 201-# Row centers (y): 133, 193, 253, 313, 373-CALC_BUTTONS = {- "1": (29, 313), "2": (86, 313), "3": (144, 313),-}---def _build_focus_app() -> None:- if not os.path.exists(_FOCUS_APP_EXE):- subprocess.run(- [os.path.join(_FOCUS_APP_DIR, "build.sh")], check=True- )---def _launch_focus_app() -> tuple[subprocess.Popen, int]:- proc = subprocess.Popen(- [_FOCUS_APP_EXE],- stdout=subprocess.PIPE,- stderr=subprocess.DEVNULL,- text=True,- )- for _ in range(40):- line = proc.stdout.readline().strip()- if line.startswith("FOCUS_PID="):- return proc, int(line.split("=", 1)[1])- time.sleep(0.1)- proc.terminate()- raise RuntimeError("FocusMonitorApp did not print FOCUS_PID in time")---def _read_focus_losses() -> int:- try:- with open(_LOSS_FILE) as f:- return int(f.read().strip())- except (FileNotFoundError, ValueError):- return -1---def _find_calc_button(tree: str, label: str) -> int | None:- for line in tree.split("\n"):- if "AXButton" not in line:- continue- m = re.search(r'\[(\d+)\]', line)- if not m:- continue- if f'({label})' in line or f'id={label}' in line:- return int(m.group(1))- return None---def _calc_display(client: DriverClient, pid: int) -> str:- window_id = resolve_window_id(client, pid)- result = client.call_tool(- "get_window_state",- {"pid": pid, "window_id": window_id, "query": "AXStaticText"},- )- tree = result.get("structuredContent", result).get("tree_markdown", "")- for line in tree.split("\n"):- if "AXStaticText" in line and "=" in line:- m = re.search(r'= "([^"]*)"', line)- if m:- return m.group(1).replace("\u200e", "").replace("\u200f", "")- return ""---class CalculatorPixelClickDelivery(unittest.TestCase):- """Pixel-click digits on backgrounded Calculator."""-- _calc_pid: int- _focus_proc: subprocess.Popen- _focus_pid: int-- @classmethod- def setUpClass(cls) -> None:- _build_focus_app()- try:- os.remove(_LOSS_FILE)- except FileNotFoundError:- pass- subprocess.run(["pkill", "-x", "Calculator"], check=False)- # Kill any stale FocusMonitorApp left over from a crashed prior run.- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)- time.sleep(0.3)-- cls.binary = default_binary_path()-- with DriverClient(cls.binary) as c:- result = c.call_tool("launch_app", {"bundle_id": CALC_BUNDLE})- cls._calc_pid = result["structuredContent"]["pid"]- print(f"\n Calculator pid: {cls._calc_pid}")- time.sleep(1.5)-- cls._focus_proc, cls._focus_pid = _launch_focus_app()- print(f" FocusMonitor pid: {cls._focus_pid}")- time.sleep(1.0)-- with DriverClient(cls.binary) as c:- active = frontmost_bundle_id(c)- assert active == FOCUS_MONITOR_BUNDLE, (- f"Expected FocusMonitorApp frontmost, got {active}"- )- losses = _read_focus_losses()- assert losses == 0, f"Expected 0 focus losses at start, got {losses}"-- @classmethod- def tearDownClass(cls) -> None:- if hasattr(cls, '_focus_proc'):- cls._focus_proc.terminate()- try:- cls._focus_proc.wait(timeout=3)- except subprocess.TimeoutExpired:- cls._focus_proc.kill()- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)- subprocess.run(["pkill", "-x", "Calculator"], check=False)- try:- os.remove(_LOSS_FILE)- except FileNotFoundError:- pass-- def setUp(self) -> None:- # Clear via AX before the test.- with DriverClient(self.binary) as c:- window_id = resolve_window_id(c, self._calc_pid)- snap = c.call_tool(- "get_window_state",- {"pid": self._calc_pid, "window_id": window_id},- )- tree = snap.get("structuredContent", snap).get("tree_markdown", "")- btn = _find_calc_button(tree, "All Clear") or _find_calc_button(tree, "Clear")- if btn is not None:- c.call_tool("click", {- "pid": self._calc_pid,- "window_id": window_id,- "element_index": btn,- })- time.sleep(0.3)- subprocess.run(- ["osascript", "-e", 'tell application "FocusMonitorApp" to activate'],- check=False,- )- time.sleep(0.5)- self._losses_before = _read_focus_losses()-- def test_pixel_click_no_focus_steal(self) -> None:- """Pixel-click Calculator buttons — verify no focus steal regardless of delivery."""- with DriverClient(self.binary) as c:- window_id = resolve_window_id(c, self._calc_pid)- snap = c.call_tool(- "get_window_state",- {"pid": self._calc_pid, "window_id": window_id},- )- sc = snap.get("structuredContent", snap)- w = sc.get("screenshot_width", 230)- h = sc.get("screenshot_height", 408)- print(f"\n screenshot: {w}x{h}")-- for digit in ["1", "2", "3"]:- bx, by = CALC_BUTTONS[digit]- x = int(bx * w / 230)- y = int(by * h / 408)- result = c.call_tool("click", {- "pid": self._calc_pid,- "window_id": window_id,- "x": x,- "y": y,- })- print(f" {digit}: pixel ({x},{y})")- time.sleep(0.15)-- time.sleep(0.3)-- # Assert no focus steal — this is the main invariant for background automation.- losses = _read_focus_losses()- with DriverClient(self.binary) as c:- active = frontmost_bundle_id(c)- print(f" losses: {self._losses_before}->{losses}, frontmost: {active}")- self.assertEqual(- active, FOCUS_MONITOR_BUNDLE,- f"Focus stolen — frontmost is {active}",- )-- def test_ax_click_delivery(self) -> None:- """AX-click 1+2 on backgrounded Calculator — verifies AX event delivery."""- with DriverClient(self.binary) as c:- window_id = resolve_window_id(c, self._calc_pid)- snap = c.call_tool("get_window_state", {- "pid": self._calc_pid, "window_id": window_id,- })- tree = snap.get("structuredContent", snap).get("tree_markdown", "")-- btn_1 = _find_calc_button(tree, "1")- btn_add = _find_calc_button(tree, "Add")- btn_2 = _find_calc_button(tree, "2")- btn_eq = _find_calc_button(tree, "Equals")- print(f"\n buttons: 1={btn_1}, Add={btn_add}, 2={btn_2}, Equals={btn_eq}")-- self.assertIsNotNone(btn_1, "'1' button not found in Calculator AX tree")- self.assertIsNotNone(btn_add, "'Add' button not found")- self.assertIsNotNone(btn_2, "'2' button not found")- self.assertIsNotNone(btn_eq, "'Equals' button not found")-- for idx in [btn_1, btn_add, btn_2, btn_eq]:- c.call_tool("click", {- "pid": self._calc_pid,- "window_id": window_id,- "element_index": idx,- })- time.sleep(0.3)-- snap = c.call_tool("get_window_state", {- "pid": self._calc_pid,- "window_id": window_id,- "query": "AXStaticText",- })- tree = snap.get("structuredContent", snap).get("tree_markdown", "")- print(f" result tree:\n{tree}")-- self.assertIn("3", tree, "Calculator did not show 3 after 1+2=")-- losses = _read_focus_losses()- with DriverClient(self.binary) as c:- active = frontmost_bundle_id(c)- print(f" losses: {self._losses_before}->{losses}, frontmost: {active}")- self.assertEqual(- active, FOCUS_MONITOR_BUNDLE,- f"Focus stolen by AX click — frontmost is {active}",- )---if __name__ == "__main__":- unittest.main(verbosity=2)diff --git a/libs/cua-driver/tests/e2e/test_safari.py b/libs/cua-driver/tests/e2e/test_safari.pydeleted file mode 100644index 5a33846482..0000000000--- a/libs/cua-driver/tests/e2e/test_safari.py+++ /dev/null@@ -1,332 +0,0 @@-"""Safari background automation tests — v2 harness.--Tests interact with Safari via the public MCP interface only:- get_window_state → click / type_text / press_key → get_window_state → assert--The driver picks the best internal technique (AX, CGEvent, SkyLight, etc.).-Tests verify observable DOM outcomes via the AX tree.--Setup: Safari is opened to test_page.html (served locally), then-FocusMonitorApp is activated so it owns the screen focus. Every test-asserts that focus never returned to Safari during the interaction.-"""--from __future__ import annotations--import subprocess-import sys-import time-import os--import re-import pytest--_HERE = os.path.dirname(os.path.abspath(__file__))-sys.path.insert(0, _HERE)-from harness.driver import Driver-from harness import tree as Tree---def _parse_clicks(tree_text: str) -> int:- """Extract the 'clicks: N' counter value from the AX tree."""- m = re.search(r'clicks:\s*(\d+)', tree_text)- return int(m.group(1)) if m else 0--SAFARI_BUNDLE = "com.apple.Safari"-FOCUS_MONITOR_BUNDLE = "com.trycua.FocusMonitorApp"---# ── module-level Safari setup ─────────────────────────────────────────────────-[email protected](scope="module")-def safari_pid(binary, html_server, focus_monitor):- """Open Safari to test_page.html; return its pid.-- Activated after Safari so FocusMonitorApp is frontmost during the tests.- """- subprocess.run(["pkill", "-x", "Safari"], check=False)- time.sleep(0.5)-- page_url = f"{html_server}/test_page.html"- subprocess.run(["open", "-g", "-a", "Safari", page_url], check=True)- time.sleep(3.5) # Safari launch + page load-- with Driver(binary) as d:- app = d.find_app(SAFARI_BUNDLE)- if app is None:- pytest.fail("Safari did not appear in list_apps after launch")- pid = app["pid"]-- print(f"\n Safari pid: {pid}, page: {page_url}")- yield pid-- subprocess.run(["pkill", "-x", "Safari"], check=False)--[email protected](autouse=True)-def _reactivate_focus(focus_monitor):- """Restore FocusMonitorApp as frontmost before each test (by pid)."""- _, pid = focus_monitor- subprocess.run(- ["osascript", "-e",- f'tell application "System Events" to set frontmost of (first process whose unix id is {pid}) to true'],- check=False,- )- time.sleep(0.4)---# ── helpers ───────────────────────────────────────────────────────────────────--def _get_state(d: Driver, pid: int, safari_pid_fixture) -> "harness.driver.WindowState":- wid = d.find_window(pid)- return d.get_window_state(pid, wid)---# ── tests ─────────────────────────────────────────────────────────────────────--class TestSafariButton:- """AX-click the 'Click Me' button without stealing focus."""-- def test_click_increments_counter(self, driver, safari_pid, ux_guard):- wid = driver.find_window(safari_pid)-- # Read full tree first to get current counter value- full_before = driver.get_window_state(safari_pid, wid)- count_before = _parse_clicks(full_before.tree)- idx = full_before.find_element("Click Me")- assert idx is not None, f"'Click Me' button not found in tree:\n{full_before.tree}"-- driver.click(safari_pid, wid, element_index=idx)- time.sleep(0.5)-- after = driver.get_window_state(safari_pid, wid)- count_after = _parse_clicks(after.tree)- print(f"\n click counter: {count_before} → {count_after}")- assert count_after == count_before + 1, (- f"Counter did not increment: {count_before} → {count_after}"- )-- def test_click_increments_twice(self, driver, safari_pid, ux_guard):- wid = driver.find_window(safari_pid)-- # Read full tree to get current counter value- full_before = driver.get_window_state(safari_pid, wid)- count_before = _parse_clicks(full_before.tree)- idx = full_before.find_element("Click Me")- assert idx is not None-- driver.click(safari_pid, wid, element_index=idx)- time.sleep(0.3)- driver.click(safari_pid, wid, element_index=idx)- time.sleep(0.5)-- after = driver.get_window_state(safari_pid, wid)- count_after = _parse_clicks(after.tree)- print(f"\n click counter: {count_before} → {count_after}")- assert count_after == count_before + 2, (- f"Counter should have incremented by 2: {count_before} → {count_after}"- )---class TestSafariTextInput:- """Type into a text field without stealing focus."""-- def test_type_text_appears_in_ax_tree(self, driver, safari_pid, ux_guard):- wid = driver.find_window(safari_pid)-- before = driver.get_window_state(safari_pid, wid)- idx = before.find_text_field(skip_url_bar=True)- assert idx is not None, f"Text field not found in:\n{before.tree}"-- # Focus the field via AX click- driver.click(safari_pid, wid, element_index=idx)- time.sleep(0.3)-- # Clear any prior content- driver.hotkey(safari_pid, ["cmd", "a"])- time.sleep(0.1)-- driver.type_text_chars(safari_pid, "hello safari")- time.sleep(0.6)-- after = driver.get_window_state(safari_pid, wid)- assert after.has_text("hello safari"), (- f"Typed text not visible in AX tree:\n{after.tree}"- )-- def test_type_text_with_type_text_tool(self, driver, safari_pid, ux_guard):- """Use type_text (not type_text_chars) and verify result."""- wid = driver.find_window(safari_pid)-- before = driver.get_window_state(safari_pid, wid)- idx = before.find_text_field(skip_url_bar=True)- assert idx is not None-- driver.click(safari_pid, wid, element_index=idx)- time.sleep(0.3)-- driver.hotkey(safari_pid, ["cmd", "a"])- time.sleep(0.1)-- driver.type_text(safari_pid, "type_text test")- time.sleep(0.6)-- after = driver.get_window_state(safari_pid, wid)- assert after.has_text("type_text test"), (- f"type_text result not in AX tree:\n{after.tree}"- )---class TestSafariCheckbox:- """Toggle the checkbox."""-- def test_checkbox_toggle(self, driver, safari_pid, ux_guard):- wid = driver.find_window(safari_pid)-- before = driver.get_window_state(safari_pid, wid, query="checkbox")- idx = Tree.find(before.tree, role="AXCheckBox")- if idx is None:- # Fallback: find by label text- idx = before.find_element("unchecked")- assert idx is not None, f"Checkbox not found:\n{before.tree}"-- driver.click(safari_pid, wid, element_index=idx)- time.sleep(0.4)-- after = driver.get_window_state(safari_pid, wid)- assert after.has_text("checked"), (- f"Checkbox did not toggle to checked:\n{after.tree}"- )---class TestSafariSelect:- """Change the dropdown selection."""-- def test_select_option_b(self, driver, safari_pid, ux_guard):- wid = driver.find_window(safari_pid)-- before = driver.get_window_state(safari_pid, wid, query="Option")- idx = Tree.find(before.tree, role="AXPopUpButton")- if idx is None:- idx = Tree.find(before.tree, role="AXComboBox")- if idx is None:- idx = before.find_element("Option A")- assert idx is not None, f"Select/dropdown not found:\n{before.tree}"-- driver.click(safari_pid, wid, element_index=idx)- time.sleep(0.5)-- # Option B in the dropdown menu- state2 = driver.get_window_state(safari_pid, wid)- opt_b = Tree.find(state2.tree, label="Option B")- if opt_b is not None:- driver.click(safari_pid, wid, element_index=opt_b)- else:- # Fallback: set value directly- driver.call_tool("set_value", {- "pid": safari_pid, "window_id": wid,- "element_index": idx, "value": "b",- })- time.sleep(0.4)-- after = driver.get_window_state(safari_pid, wid)- assert after.has_text("Option B") or after.has_text("selected: Option B"), (- f"Dropdown did not select Option B:\n{after.tree}"- )---class TestSafariTextarea:- """Type into the textarea.-- Safari's WKWebView AXTextArea does not reliably set keyboard focus via- AXPress in background mode. We use two strategies in sequence:- 1. Click the text INPUT first to warm up Safari's web content focus engine.- 2. Then click the textarea, wait 0.8s, type.- """-- def test_textarea_input(self, driver, safari_pid, ux_guard):- wid = driver.find_window(safari_pid)-- state = driver.get_window_state(safari_pid, wid)-- # Step 1: click the text input first to put focus inside Safari's web content- tf_idx = state.find_text_field(skip_url_bar=True)- if tf_idx is not None:- driver.click(safari_pid, wid, element_index=tf_idx)- time.sleep(0.3)-- # Step 2: click the textarea- ta_idx = Tree.find(state.tree, role="AXTextArea")- assert ta_idx is not None, f"Textarea not found:\n{state.tree[:500]}"- driver.click(safari_pid, wid, element_index=ta_idx)- time.sleep(0.8) # longer settle for WKWebView textarea focus-- # Step 3: type — keystrokes go to the focused element- driver.call_tool("type_text_chars", {- "pid": safari_pid,- "text": "multi line",- "delay_ms": 50,- })- time.sleep(0.6)-- after = driver.get_window_state(safari_pid, wid)- val = Tree.ax_value(after.tree, ta_idx)- print(f"\n textarea AX value: {val!r}")- assert after.has_text("multi line"), (- f"Textarea text not in AX tree; val={val!r};\n{after.tree[:800]}"- )---class TestSafariCanvas:- """Send a pixel click to the canvas area without stealing focus.-- Background note: synthetic CGEvent pixel clicks to backgrounded browser- windows may not trigger DOM mousedown events (web content is only rendered- and interactive when the window is frontmost). This test therefore only- asserts the key UX invariant: the click is delivered without error and- without stealing focus. A DOM update (AX tree change or screenshot diff)- is logged as a bonus if it occurs.- """-- def test_canvas_pixel_click_no_focus_steal(self, driver, safari_pid, ux_guard):- wid = driver.find_window(safari_pid)-- before = driver.get_window_state(safari_pid, wid)- print(f"\n screenshot: {before.screenshot_width}x{before.screenshot_height}")-- from harness.cv import decode, diff_ratio- before_img = decode(before.screenshot_b64)-- # Canvas is the last section on the page. With ~80px browser chrome- # in a 906px window, the canvas element center is near y=88% of height.- x = before.screenshot_width // 2- y = int(before.screenshot_height * 0.88)- print(f" pixel click at ({x}, {y})")-- result = driver.click(safari_pid, wid, x=x, y=y)- time.sleep(0.5)-- after = driver.get_window_state(safari_pid, wid)-- # Bonus: check if DOM or screenshot updated (non-deterministic in background)- canvas_clicked = "canvas: clicked" in after.tree- after_img = decode(after.screenshot_b64)- ratio = diff_ratio(before_img, after_img)- print(f" canvas JS fired: {canvas_clicked}, diff_ratio: {ratio:.4f}")-- # Primary assertion: click succeeded (no exception was raised)- # UX assertion: via ux_guard fixture teardown (no focus steal)- assert result is not None, "click tool returned None"---class TestSafariPressKey:- """Press keys without stealing focus."""-- def test_tab_key(self, driver, safari_pid, ux_guard):- driver.press_key(safari_pid, "tab")- time.sleep(0.3)- # No assertion on state — just verifying no exception + no UX violation-- def test_escape_key(self, driver, safari_pid, ux_guard):- driver.press_key(safari_pid, "escape")- time.sleep(0.2)diff --git a/libs/cua-driver/tests/e2e/test_tauri.py b/libs/cua-driver/tests/e2e/test_tauri.pydeleted file mode 100644index e8b6aaee8a..0000000000--- a/libs/cua-driver/tests/e2e/test_tauri.py+++ /dev/null@@ -1,97 +0,0 @@-"""Tauri shared-harness background automation tests.--The Tauri test app is built from tests/fixtures/apps/cross-platform/tauri-and loads tests/fixtures/shared/web/index.html, the same DOM used by the-Electron, WebView2, and WKWebView harnesses.--Run: python3 -m pytest test_tauri.py -v-"""--from __future__ import annotations--import subprocess-import sys-import time-import os--import pytest--_HERE = os.path.dirname(os.path.abspath(__file__))-sys.path.insert(0, _HERE)--[email protected](autouse=True)-def _reactivate_focus(focus_monitor):- _, pid = focus_monitor- subprocess.run(- ["osascript", "-e",- f'tell application "System Events" to set frontmost of (first process whose unix id is {pid}) to true'],- check=False,- )- time.sleep(0.4)---# ── tests ─────────────────────────────────────────────────────────────────────--class TestTauriButton:- def test_button_click(self, driver, tauri_app, ux_guard):- _, pid, _ = tauri_app- if pid == 0:- pytest.skip("Tauri app pid not found")-- wid = driver.find_window(pid)-- before = driver.get_window_state(pid, wid, query="Increment")- idx = before.find_button("Increment") or before.find_element("Increment")- assert idx is not None, f"'Increment' not found:\n{before.tree}"-- driver.click(pid, wid, element_index=idx)- time.sleep(0.5)-- after = driver.get_window_state(pid, wid)- assert after.has_text("counter=1"), f"Counter not incremented:\n{after.tree}"---class TestTauriTextInput:- def test_type_text(self, driver, tauri_app, ux_guard):- _, pid, _ = tauri_app- if pid == 0:- pytest.skip("Tauri app pid not found")-- wid = driver.find_window(pid)-- before = driver.get_window_state(pid, wid)- idx = before.find_text_field(skip_url_bar=False)- assert idx is not None, f"Text input not found:\n{before.tree}"-- driver.click(pid, wid, element_index=idx)- time.sleep(0.3)- driver.hotkey(pid, ["cmd", "a"])- time.sleep(0.1)- driver.type_text_chars(pid, "tauri test")- time.sleep(0.6)-- after = driver.get_window_state(pid, wid)- assert after.has_text("mirror=tauri test"), f"Text not mirrored in AX tree:\n{after.tree}"---class TestTauriNoFocusSteal:- def test_multiple_actions_no_focus_steal(self, driver, tauri_app, ux_guard):- _, pid, _ = tauri_app- if pid == 0:- pytest.skip("Tauri app pid not found")-- wid = driver.find_window(pid)-- # Take 3 actions in sequence — UX guard checks no focus steal- before = driver.get_window_state(pid, wid, query="Increment")- idx = before.find_button("Increment") or before.find_element("Increment")- if idx is not None:- driver.click(pid, wid, element_index=idx)- time.sleep(0.3)-- driver.press_key(pid, "tab")- time.sleep(0.2)- driver.press_key(pid, "tab")- time.sleep(0.2)- # ux_guard teardown will assert_clean()diff --git a/libs/cua-driver/tests/fixtures/README.md b/libs/cua-driver/tests/fixtures/README.mdindex c72f28a5b1..74df5105a3 100644--- a/libs/cua-driver/tests/fixtures/README.md+++ b/libs/cua-driver/tests/fixtures/README.md@@ -74,8 +74,8 @@ Rust tests under `libs/cua-driver/rust/crates/cua-driver/tests/` consume the stamodality coverage, ignored/manual or VM-backed.- `protocol_*_test.rs` and schema tests: headless protocol coverage, default.-The Python e2e suite under `libs/cua-driver/tests/e2e/` drives real apps and the shared-Electron/Tauri harnesses through the public MCP interface.+Rust integration tests under `libs/cua-driver/rust/crates/cua-driver/tests/` drive+the shared Electron/Tauri harnesses through the public MCP interface.## App Coverage@@ -99,5 +99,5 @@ Electron/Tauri harnesses through the public MCP interface.tests.- Keep platform-specific quirks in the relevant Rust test or app source, not inthe shared fixture.-- If a recorder finding graduates into an invariant, move it into a Rust or- Python test and summarize it here.+- If a recorder finding graduates into an invariant, move it into a Rust test+ and summarize it here.diff --git a/libs/cua-driver/tests/fixtures/apps/README.md b/libs/cua-driver/tests/fixtures/apps/README.mdindex 7e3346165c..0263fdeceb 100644--- a/libs/cua-driver/tests/fixtures/apps/README.md+++ b/libs/cua-driver/tests/fixtures/apps/README.md@@ -26,4 +26,4 @@ Adding a new app:2. Reuse `shared/scenarios.json` and `shared/web/index.html` instead of copyingprivate ids.3. Stage into `rust/test-apps/harness-<toolkit>/`.-4. Add or update the matching Rust/Python tests and this table.+4. Add or update the matching Rust tests and this table.