Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

Remove legacy Python e2e suite

Francesco Bonacci0f4714fbf942
diff --git a/docs/content/docs/reference/cua-driver/development.mdx b/docs/content/docs/reference/cua-driver/development.mdx
index d90b09cba5..b2b17944d6 100644
--- a/docs/content/docs/reference/cua-driver/development.mdx
+++ b/docs/content/docs/reference/cua-driver/development.mdx
@@ -14,7 +14,7 @@ they stay close to implementation changes.
| `libs/cua-driver/rust` | Rust workspace for the daemon, platform crates, testkit, and helper crates |
| `libs/cua-driver/python` | Python package wrapper around the driver binary |
| `libs/cua-driver/tests/fixtures` | Source-built GUI harness apps and shared fixtures |
-| `libs/cua-driver/tests/e2e` | Manual Python e2e tests against real apps and harness apps |
+| `libs/cua-driver/rust/crates/cua-driver/tests` | Rust integration tests for the driver and GUI harnesses |
| `libs/cua-driver/scripts` | Install, uninstall, local build, and VM sync helpers |
## Contributor Entry Points
@@ -23,7 +23,7 @@ they stay close to implementation changes.
- `libs/cua-driver/rust/README.md` maps the Cargo workspace and test classes.
- `libs/cua-driver/rust/crates/cua-driver/tests/README.md` explains Rust test naming and ignored GUI lanes.
- `libs/cua-driver/tests/fixtures/README.md` explains source-built harness apps and staged outputs.
-- `libs/cua-driver/tests/e2e/README.md` explains the Python e2e suite.
+- `libs/cua-driver/rust/crates/cua-driver/tests/README.md` explains Rust test naming and ignored GUI lanes.
- `libs/cua-driver/scripts/README.md` explains install and VM sync helpers.
Build artifacts, VM logs, and local verification journals should stay out of
diff --git a/libs/cua-driver/README.md b/libs/cua-driver/README.md
index d71449e9ff..0c8aef4558 100644
--- a/libs/cua-driver/README.md
+++ b/libs/cua-driver/README.md
@@ -11,12 +11,12 @@ Background computer-use driver for any agents. Speaks MCP over stdio; drives nat
| `rust/` | Cargo workspace for the driver daemon, platform crates, and Rust tests |
| `python/` | Python package wrapper and package tests |
| `tests/fixtures/` | Source-built GUI harness apps and shared fixtures |
-| `tests/e2e/` | Manual Python e2e tests against real apps and harness apps |
+| `rust/crates/cua-driver/tests/` | Rust integration tests for the driver and GUI harnesses |
| `scripts/` | Install, uninstall, local build, and VM sync helpers |
| `docs/` | Small repo-local specs that are not part of the hosted docs site |
-Start with `rust/README.md`, `tests/fixtures/README.md`, and
-`tests/e2e/README.md` when changing driver behavior or tests.
+Start with `rust/README.md`, `rust/crates/cua-driver/tests/README.md`, and
+`tests/fixtures/README.md` when changing driver behavior or tests.
## Claude Code computer-use compatibility
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/guard_ux_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/guard_ux_test.rs
index dbded6e185..cd8d793c93 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/guard_ux_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/guard_ux_test.rs
@@ -1,10 +1,11 @@
//! UX-guard integration tests for Windows.
//!
-//! These mirror the Python e2e tests in libs/cua-driver/tests/e2e/:
-//! - test_background_focus.py
-//! - test_click_opens_new_window.py
-//! - test_launch_app_visible.py
-//! - test_background_menu_shortcut.py
+//! These cover the UX-guard scenarios previously exercised by the legacy
+//! Python suite:
+//! - background focus preservation
+//! - new-window click delivery
+//! - visible app launch
+//! - background menu shortcuts
//!
//! Invariant under test (the "UX guard"):
//! The agent must be able to click, type, and launch apps in background
@@ -14,43 +15,87 @@
//! test-apps/harness-electron/CuaTestHarness.Electron.exe. Notepad is used
//! only as a secondary fallback check.
//!
-//! All tests:
-//! 1. Launch focus-monitor-win (the "user's foreground window").
-//! 2. Perform background operations via cua-driver.
-//! 3. Assert focus-monitor-win's act_losses counter stayed at 0.
+//! Background-action tests launch the target first, then foreground
+//! focus-monitor-win (the "user's foreground window") before measuring focus
+//! loss. The launch_app test starts the monitor first because launch behavior is
+//! the action under test.
//!
//! Run in sandbox via:
//! ..\tests\runners\windows-sandbox\run-tests-in-sandbox.ps1 guard_ux
#![cfg(target_os = "windows")]
+use std::collections::HashSet;
use std::path::PathBuf;
use std::process::{Child, Command, Stdio};
-use std::time::Duration;
+use std::time::{Duration, Instant};
-use cua_driver_testkit::{driver_binary, spawn_in_job, workspace_root, Driver, McpDriver};
+use cua_driver_testkit::{ax, driver_binary, spawn_in_job, workspace_root, Driver, McpDriver};
// ── focus-monitor + test-app fixtures ────────────────────────────────────────
-fn require_interactive_desktop(context: &str) -> bool {
- if platform_windows::diagnostics::current_session_id() == Some(0) {
- eprintln!(
- "{context}: running in Windows Session 0; skipping GUI UX guard. \
- Re-run from an interactive logon session (RDP/console/scheduled task in user session)."
+fn gui_required() -> bool {
+ std::env::var("CUA_REQUIRE_GUI")
+ .ok()
+ .map(|v| matches!(v.to_ascii_lowercase().as_str(), "1" | "true" | "yes" | "on"))
+ .unwrap_or(false)
+}
+
+fn skip_desktop(context: &str, reason: String) -> bool {
+ let msg = format!("{context}: {reason}; skipping GUI UX guard");
+ if gui_required() {
+ panic!("{msg}");
+ }
+ eprintln!("{msg}");
+ false
+}
+
+fn require_seedable_desktop(context: &str) -> bool {
+ let state = platform_windows::diagnostics::desktop_state();
+ if state.session_id == Some(0) {
+ return skip_desktop(
+ context,
+ format!(
+ "running in Windows Session 0 ({}) - re-run from RDP/console/scheduled task in user session",
+ state.summary()
+ ),
+ );
+ }
+ if !state.has_process_window_station {
+ return skip_desktop(
+ context,
+ format!("no attached process window station ({})", state.summary()),
+ );
+ }
+ if !state.input_desktop_is_default() {
+ return skip_desktop(
+ context,
+ format!(
+ "input desktop is not the user Default desktop ({})",
+ state.summary()
+ ),
);
- return false;
}
+ true
+}
- match platform_windows::diagnostics::interactive_desktop_check() {
- Ok(true) => true,
- Ok(false) => {
- eprintln!("{context}: no foreground interactive desktop; skipping GUI UX guard");
- false
+fn require_focus_monitor_foreground(context: &str, expected_hwnd: u64) -> bool {
+ let deadline = std::time::Instant::now() + Duration::from_secs(2);
+ loop {
+ let state = platform_windows::diagnostics::desktop_state();
+ if state.foreground_hwnd == Some(expected_hwnd as usize) {
+ return true;
}
- Err(e) => {
- eprintln!("{context}: interactive desktop check failed ({e}); skipping GUI UX guard");
- false
+ if std::time::Instant::now() >= deadline {
+ return skip_desktop(
+ context,
+ format!(
+ "focus monitor did not become foreground (expected HWND 0x{expected_hwnd:x}; {})",
+ state.summary()
+ ),
+ );
}
+ std::thread::sleep(Duration::from_millis(100));
}
}
@@ -92,6 +137,40 @@ fn launch_test_app(driver: &mut McpDriver) -> Option<u32> {
Some(pid)
}
+fn launch_driver_and_test_app() -> Option<(McpDriver, u32, u64)> {
+ let Some(mut driver) = McpDriver::spawn() else {
+ return None;
+ };
+
+ let Some(app_pid) = launch_test_app(&mut driver) else {
+ eprintln!("test app not available — skipping");
+ return None;
+ };
+ let Some(app_wid) = find_window_for_pid(&mut driver, app_pid as i64) else {
+ eprintln!("test app window not found — skipping");
+ return None;
+ };
+
+ Some((driver, app_pid, app_wid))
+}
+
+fn kill_process_tree_by_image(image: &str) {
+ Command::new("taskkill")
+ .args(["/F", "/T", "/IM", image])
+ .stdout(Stdio::null())
+ .stderr(Stdio::null())
+ .spawn()
+ .ok();
+}
+
+struct KillProcessTreeOnDrop(&'static str);
+
+impl Drop for KillProcessTreeOnDrop {
+ fn drop(&mut self) {
+ kill_process_tree_by_image(self.0);
+ }
+}
+
fn loss_file() -> PathBuf {
std::env::temp_dir().join("focus_monitor_losses.txt")
}
@@ -117,7 +196,7 @@ fn focus_hwnd_file() -> PathBuf {
/// Reads FOCUS_PID and FOCUS_HWND from temp files written by the monitor
/// (avoids blocking on the stdout pipe if the sandbox redirects I/O).
fn launch_focus_monitor() -> Option<(Child, u64, u32)> {
- if !require_interactive_desktop("focus-monitor-win") {
+ if !require_seedable_desktop("focus-monitor-win") {
return None;
}
@@ -132,7 +211,7 @@ fn launch_focus_monitor() -> Option<(Child, u64, u32)> {
let _ = std::fs::remove_file(focus_pid_file());
let _ = std::fs::remove_file(focus_hwnd_file());
- let child = spawn_in_job(
+ let mut child = spawn_in_job(
Command::new(&exe)
.stdout(Stdio::null())
.stderr(Stdio::null()),
@@ -163,8 +242,10 @@ fn launch_focus_monitor() -> Option<(Child, u64, u32)> {
}
std::thread::sleep(Duration::from_millis(100));
}
- // Give the window time to become foreground.
- std::thread::sleep(Duration::from_millis(400));
+ if !require_focus_monitor_foreground("focus-monitor-win", hwnd_val) {
+ child.kill().ok();
+ return None;
+ }
Some((child, hwnd_val, pid_val))
}
@@ -180,6 +261,26 @@ fn find_window_for_pid(driver: &mut McpDriver, pid: i64) -> Option<u64> {
.find_map(|w| w["window_id"].as_u64())
}
+fn window_ids(driver: &mut McpDriver) -> HashSet<u64> {
+ let resp = driver.call("list_windows", serde_json::json!({}));
+ resp.structured()["windows"]
+ .as_array()
+ .map(|a| a.iter().filter_map(|w| w["window_id"].as_u64()).collect())
+ .unwrap_or_default()
+}
+
+fn wait_for_new_window(driver: &mut McpDriver, before: &HashSet<u64>) -> bool {
+ let deadline = Instant::now() + Duration::from_secs(3);
+ while Instant::now() < deadline {
+ let after = window_ids(driver);
+ if after.iter().any(|id| !before.contains(id)) {
+ return true;
+ }
+ std::thread::sleep(Duration::from_millis(100));
+ }
+ false
+}
+
// ── UX guard assertion ────────────────────────────────────────────────────────
/// Assert act_losses stayed at `max_allowed` (usually 0) since `before`.
@@ -199,10 +300,10 @@ fn assert_ux_guard(before: u32, max_allowed: u32, context: &str) {
#[test]
fn test_background_click_and_type_no_focus_steal() {
- //! Equivalent of macOS test_background_focus.py.
+ //! Background focus-preservation coverage.
//!
- //! 1. Launch FocusMonitorWin (simulates the user's active window).
- //! 2. Launch the Electron harness in the background.
+ //! 1. Launch the Electron harness.
+ //! 2. Foreground FocusMonitorWin (simulates the user's active window).
//! 3. Click inside the app and type text via cua-driver.
//! 4. Assert act_losses on FocusMonitorWin stayed at 0.
@@ -211,26 +312,14 @@ fn test_background_click_and_type_no_focus_steal() {
return;
}
- let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {
- return;
- };
- let losses_before = read_losses(&loss_file());
-
- let Some(mut driver) = McpDriver::spawn() else {
- fm_proc.kill().ok();
+ let Some((mut driver, app_pid, app_wid)) = launch_driver_and_test_app() else {
return;
};
- let Some(app_pid) = launch_test_app(&mut driver) else {
- eprintln!("test app not available — skipping");
- fm_proc.kill().ok();
- return;
- };
- let Some(app_wid) = find_window_for_pid(&mut driver, app_pid as i64) else {
- eprintln!("test app window not found — skipping");
- fm_proc.kill().ok();
+ let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {
return;
};
+ let losses_before = read_losses(&loss_file());
// Click inside the app (background, via PostMessage).
let r = driver.call(
@@ -253,6 +342,36 @@ fn test_background_click_and_type_no_focus_steal() {
"Protocol error from type_text: {:?}",
r.raw
);
+ assert_eq!(
+ r.verified(),
+ Some(false),
+ "background type_text without an element read-back must not report confirmed success: {}",
+ r.text()
+ );
+ assert_ne!(
+ r.structured()["verify"].as_str(),
+ Some("confirmed"),
+ "background type_text reported a confirmed read-back on an unreadable path: {}",
+ r.text()
+ );
+
+ // Plain background key dispatch is likewise an unverified PostMessage send,
+ // not a confirmed keypress.
+ let r = driver.call(
+ "press_key",
+ serde_json::json!({"pid": app_pid, "window_id": app_wid, "key": "F24"}),
+ );
+ assert!(
+ r.raw["error"].is_null(),
+ "Protocol error from press_key: {:?}",
+ r.raw
+ );
+ assert_eq!(
+ r.verified(),
+ Some(false),
+ "background press_key must not report confirmed success: {}",
+ r.text()
+ );
// ux_guard: FocusMonitorWin must not have lost activation.
assert_ux_guard(
@@ -265,14 +384,15 @@ fn test_background_click_and_type_no_focus_steal() {
}
// ─────────────────────────────────────────────────────────────────────────────
-// Test 2: launch_app does not steal focus
+// Test 2: launch_app minimized mode does not steal focus
// ─────────────────────────────────────────────────────────────────────────────
#[test]
-fn test_launch_app_no_focus_steal() {
- //! Equivalent of macOS test_launch_app_visible.py.
+fn test_launch_app_minimized_no_focus_steal() {
+ //! Visible app-launch coverage.
//!
- //! launch_app (path variant) must open a window without displacing FocusMonitorWin.
+ //! launch_app with start_minimized=true is the strict Windows background
+ //! launch mode: the app starts without displacing FocusMonitorWin.
if !driver_binary().exists() {
return;
@@ -283,6 +403,7 @@ fn test_launch_app_no_focus_steal() {
eprintln!("test app not available — skipping");
return;
}
+ let _cleanup = KillProcessTreeOnDrop("CuaTestHarness.Electron.exe");
let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {
return;
@@ -294,9 +415,12 @@ fn test_launch_app_no_focus_steal() {
return;
};
- // Launch the test app via cua-driver launch_app (full path, SW_SHOWNOACTIVATE).
+ // Launch the test app via cua-driver launch_app in strict background mode.
let path_str = exe.to_string_lossy().into_owned();
- let r = driver.call("launch_app", serde_json::json!({"path": path_str}));
+ let r = driver.call(
+ "launch_app",
+ serde_json::json!({"path": path_str, "start_minimized": true}),
+ );
if r.is_error() {
eprintln!("launch_app failed — skipping: {:?}", r.raw);
fm_proc.kill().ok();
@@ -330,15 +454,14 @@ fn test_launch_app_no_focus_steal() {
}
// ux_guard: FocusMonitorWin must not have lost activation.
- assert_ux_guard(losses_before, 0, "launch_app CuaTestHarness.Electron");
+ assert_ux_guard(
+ losses_before,
+ 0,
+ "launch_app CuaTestHarness.Electron with start_minimized=true",
+ );
// Kill the launched app by exe name.
- Command::new("taskkill")
- .args(["/F", "/T", "/IM", "CuaTestHarness.Electron.exe"])
- .stdout(Stdio::null())
- .stderr(Stdio::null())
- .spawn()
- .ok();
+ kill_process_tree_by_image("CuaTestHarness.Electron.exe");
fm_proc.kill().ok();
}
@@ -348,7 +471,7 @@ fn test_launch_app_no_focus_steal() {
#[test]
fn test_background_hotkey_no_focus_steal() {
- //! Equivalent of macOS test_background_menu_shortcut.py.
+ //! Background menu-shortcut coverage.
//!
//! Send Ctrl+A to a background Electron harness window.
//! FocusMonitorWin must never lose activation.
@@ -357,26 +480,14 @@ fn test_background_hotkey_no_focus_steal() {
return;
}
- let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {
+ let Some((mut driver, app_pid, app_wid)) = launch_driver_and_test_app() else {
return;
};
- let losses_before = read_losses(&loss_file());
- let Some(mut driver) = McpDriver::spawn() else {
- fm_proc.kill().ok();
- return;
- };
-
- let Some(app_pid) = launch_test_app(&mut driver) else {
- eprintln!("test app not available — skipping");
- fm_proc.kill().ok();
- return;
- };
- let Some(app_wid) = find_window_for_pid(&mut driver, app_pid as i64) else {
- eprintln!("test app window not found — skipping");
- fm_proc.kill().ok();
+ let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {
return;
};
+ let losses_before = read_losses(&loss_file());
// Send Ctrl+A hotkey to background app (PostMessage, no focus steal).
let r = driver.call(
@@ -405,44 +516,41 @@ fn test_background_hotkey_no_focus_steal() {
#[test]
fn test_background_click_opens_new_window_focus_preserved() {
- //! Equivalent of macOS test_click_opens_new_window.py.
+ //! New-window click-delivery coverage.
//!
//! 1. FocusMonitorWin is foreground.
- //! 2. Click in CuaTestHarness.Electron - a new window/tab may appear.
+ //! 2. Click the CuaTestHarness.Electron child-window button.
//! 3. FocusMonitorWin must remain active throughout (UX guard).
//!
- //! Verifies PostMessage doesn't inadvertently activate any new window that
- //! appears as a side-effect of the click.
+ //! Verifies that a background click can cause a child window to appear
+ //! without activating either the original target or the new window.
if !driver_binary().exists() {
return;
}
- let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {
+ let Some((mut driver, app_pid, app_wid)) = launch_driver_and_test_app() else {
return;
};
- let losses_before = read_losses(&loss_file());
-
- let Some(mut driver) = McpDriver::spawn() else {
- fm_proc.kill().ok();
+ let snap = driver.call(
+ "get_window_state",
+ serde_json::json!({"pid": app_pid, "window_id": app_wid, "capture_mode": "ax"}),
+ );
+ let Some(open_idx) = ax::element_index_containing(snap.text(), "Open child window") else {
+ eprintln!("child-window button not found in test app — skipping");
return;
};
- let Some(app_pid) = launch_test_app(&mut driver) else {
- eprintln!("test app not available — skipping");
- fm_proc.kill().ok();
- return;
- };
- let Some(app_wid) = find_window_for_pid(&mut driver, app_pid as i64) else {
- eprintln!("test app window not found — skipping");
- fm_proc.kill().ok();
+ let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {
return;
};
+ let windows_before = window_ids(&mut driver);
+ let losses_before = read_losses(&loss_file());
- // Click somewhere in the app content area (may trigger navigation/new window).
+ // Click the explicit child-window button.
let r = driver.call(
"click",
- serde_json::json!({"pid": app_pid, "window_id": app_wid, "x": 400.0, "y": 350.0}),
+ serde_json::json!({"pid": app_pid, "window_id": app_wid, "element_index": open_idx}),
);
assert!(
r.raw["error"].is_null(),
@@ -450,8 +558,10 @@ fn test_background_click_opens_new_window_focus_preserved() {
r.raw
);
- // Brief wait for any side-effect windows to appear.
- std::thread::sleep(Duration::from_millis(500));
+ assert!(
+ wait_for_new_window(&mut driver, &windows_before),
+ "background click did not open a new harness window"
+ );
// ux_guard: FocusMonitorWin must not have lost activation.
assert_ux_guard(
@@ -481,26 +591,14 @@ fn test_background_screenshot_no_focus_steal() {
return;
}
- let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {
+ let Some((mut driver, _app_pid, app_wid)) = launch_driver_and_test_app() else {
return;
};
- let losses_before = read_losses(&loss_file());
- let Some(mut driver) = McpDriver::spawn() else {
- fm_proc.kill().ok();
- return;
- };
-
- let Some(app_pid) = launch_test_app(&mut driver) else {
- eprintln!("test app not available — skipping");
- fm_proc.kill().ok();
- return;
- };
- let Some(app_wid) = find_window_for_pid(&mut driver, app_pid as i64) else {
- eprintln!("test app window not found — skipping");
- fm_proc.kill().ok();
+ let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {
return;
};
+ let losses_before = read_losses(&loss_file());
// Screenshot via PrintWindow — must not activate the window.
let r = driver.call("screenshot", serde_json::json!({"window_id": app_wid}));
@@ -540,7 +638,7 @@ fn test_agent_cursor_visible_on_screen() {
eprintln!("Binary not found — skipping");
return;
}
- if !require_interactive_desktop("agent cursor visibility") {
+ if !require_seedable_desktop("agent cursor visibility") {
return;
}
@@ -551,11 +649,12 @@ fn test_agent_cursor_visible_on_screen() {
// Safe centre-ish position on primary monitor.
let cx = 640.0_f64;
let cy = 400.0_f64;
+ let cursor_id = "guard-ux-cursor";
// Enable cursor overlay and glide to target.
let r = driver.call(
"set_agent_cursor_enabled",
- serde_json::json!({"enabled": true}),
+ serde_json::json!({"enabled": true, "cursor_id": cursor_id}),
);
assert!(
r.raw["error"].is_null(),
@@ -563,11 +662,28 @@ fn test_agent_cursor_visible_on_screen() {
r.raw
);
- let r = driver.call("move_cursor", serde_json::json!({"x": cx, "y": cy}));
+ let r = driver.call(
+ "set_agent_cursor_motion",
+ serde_json::json!({
+ "cursor_id": cursor_id,
+ "glide_duration_ms": 100,
+ "idle_hide_ms": 0
+ }),
+ );
+ assert!(
+ r.raw["error"].is_null(),
+ "set_agent_cursor_motion failed: {:?}",
+ r.raw
+ );
+
+ let r = driver.call(
+ "move_cursor",
+ serde_json::json!({"x": cx, "y": cy, "cursor_id": cursor_id}),
+ );
assert!(r.raw["error"].is_null(), "move_cursor failed: {:?}", r.raw);
- // Wait for the glide animation (750ms default) + a few render frames.
- std::thread::sleep(Duration::from_millis(900));
+ // Wait for the fixed 100ms glide plus a few render frames.
+ std::thread::sleep(Duration::from_millis(350));
// Capture the screen directly (includes layered windows like the overlay).
let png_bytes = platform_windows::capture::screenshot_display_bytes()
diff --git a/libs/cua-driver/rust/test-apps/.gitignore b/libs/cua-driver/rust/test-apps/.gitignore
index afcacfa198..fe5703f95e 100644
--- a/libs/cua-driver/rust/test-apps/.gitignore
+++ b/libs/cua-driver/rust/test-apps/.gitignore
@@ -1,5 +1,5 @@
-# Pre-built test app binaries are downloaded by run-tests-in-sandbox.ps1
-# Run: .\sandbox\run-tests-in-sandbox.ps1 to auto-download before testing
+# Pre-built test app binaries are staged by the source-build scripts under
+# ../../tests/fixtures/build/ and are never committed.
*.exe
*.app
*.dmg
diff --git a/libs/cua-driver/tests/e2e/PHILOSOPHY.md b/libs/cua-driver/tests/e2e/PHILOSOPHY.md
deleted file mode 100644
index 5e0d62aeba..0000000000
--- a/libs/cua-driver/tests/e2e/PHILOSOPHY.md
+++ /dev/null
@@ -1,136 +0,0 @@
-# cua-driver Python E2E Test Philosophy
-
-## Core Principle
-
-**Tests exercise the public MCP tool interface only.**
-
-The driver exposes a small, stable interface:
-
-| Tool | Purpose |
-|------|---------|
-| `get_window_state` | Snapshot: AX/UIA tree + screenshot |
-| `click` | Click at element or pixel |
-| `type_text` | Type a string |
-| `type_text_chars` | Type via keystroke synthesis |
-| `press_key` | Send a named key |
-| `hotkey` | Send a key combo |
-| `scroll` | Scroll wheel |
-| `drag` | Mouse drag |
-
-Tests **never** reach for `page.execute_javascript`, CDP, Apple Events, or
-any other internal mechanism. The driver is a black box that delivers the
-action via whatever technique works best for the target app. Tests verify
-the *observable outcome*, not the delivery path.
-
----
-
-## Canonical Test Pattern
-
-Every test follows the same four-step loop:
-
-```
-before = get_window_state(pid, wid) # 1. snapshot
-element = find(before.tree, ...) # 2. locate
-act(pid, wid, element or coords) # 3. act (click / type / key)
-after = get_window_state(pid, wid) # 4. assert
-assert <state changed in after>
-```
-
-- **Snapshot first** — always read the AX tree before acting so element
- indices are fresh.
-- **Act once** — do not poll between action and assertion. One `get_window_state`
- after the action is enough (add a short `sleep` if the app needs to settle).
-- **Assert observable state** — AX tree values, window titles, screenshot diffs.
- Never assert internal driver behaviour.
-
----
-
-## UX Invariants (checked throughout every test)
-
-A background thread polls at 5 ms and records any violation of:
-
-| Signal | Violation |
-|--------|-----------|
-| Real cursor position (`NSEvent.mouseLocation`) | Moved > 4 px from baseline |
-| Frontmost app PID (`NSWorkspace.frontmostApplication`) | Changed from sentinel |
-| Agent overlay z-order (`CGWindowListCopyWindowInfo`) | cua-driver layer ≥ frontmost non-agent window layer |
-
-The `ux_guard` pytest fixture starts the monitor before each test and calls
-`monitor.assert_clean()` in teardown. If any violation was recorded the test
-fails with a list of every incident.
-
----
-
-## App-specific Oracle Strategy
-
-| App | Primary oracle | Cross-check |
-|-----|---------------|-------------|
-| Safari | AX tree text values | - |
-| Chrome | AX tree text values | - |
-| Electron | AX tree text values from shared DOM | - |
-| Tauri | AX tree text values from shared DOM | - |
-| Blender | Screenshot + NCC template match | AX tree window title |
-| Calculator | AX tree static text | - |
-
-For browser apps the test page (`assets/test_page.html`) provides a
-self-contained oracle: every interactive element reflects its state back into
-the DOM (button click counter, text field value, checkbox label, select
-label, textarea content). The AX tree exposes those DOM values directly.
-
----
-
-## Canvas Test Pattern
-
-Canvas pixels are not in the AX tree. Verify canvas interaction by screenshot
-diff:
-
-```python
-before = get_window_state(pid, wid)
-before_crop = cv.crop(cv.decode(before.screenshot), canvas_bbox)
-click(pid, wid, x=target_x, y=target_y)
-after = get_window_state(pid, wid)
-after_crop = cv.crop(cv.decode(after.screenshot), canvas_bbox)
-assert cv.diff_ratio(before_crop, after_crop) > 0.001
-```
-
----
-
-## Test File Structure
-
-```
-tests/e2e/
- PHILOSOPHY.md ← this file
- conftest.py ← pytest fixtures (driver, ux_guard, sentinel, html_server, ...)
- driver_client.py ← raw MCP client wrapper
- harness/
- __init__.py
- monitor.py ← UXMonitor (5 ms PyObjC thread)
- driver.py ← thin typed wrapper around DriverClient
- tree.py ← tree_markdown parser + find / value / bbox helpers
- cv.py ← PIL decode / crop / diff / template-match utilities
- assets/
- test_page.html ← all-elements test page for browser tests
- blender/
- toolbar_ref.png ← stored template target
- test_safari.py
- test_chrome.py
- test_electron.py
- test_tauri.py
- test_blender.py
- test_*.py ← older e2e coverage kept here until audited/ported
-```
-
----
-
-## Running
-
-```bash
-cd libs/cua-driver/tests/e2e
-export CUA_DRIVER_BINARY=../../rust/target/debug/cua-driver
-
-# Single file
-python3 -m pytest test_safari.py -v
-
-# Full suite
-python3 -m pytest -v
-```
diff --git a/libs/cua-driver/tests/e2e/README.md b/libs/cua-driver/tests/e2e/README.md
deleted file mode 100644
index 367dc722c0..0000000000
--- a/libs/cua-driver/tests/e2e/README.md
+++ /dev/null
@@ -1,46 +0,0 @@
-# cua-driver Python e2e tests
-
-Python e2e tests drive the public MCP tool interface against real apps and
-shared harness apps. They are manual/VM-backed tests, not default CI tests.
-
-## Layout
-
-```text
-tests/e2e/
-├── driver_client.py # raw MCP client
-├── conftest.py # pytest fixtures
-├── harness/ # typed wrappers, tree helpers, CV helpers, UX monitor
-├── assets/ # symlinks/references for browser and Blender tests
-├── PHILOSOPHY.md # black-box test pattern and UX invariants
-├── run_tests.sh # pytest runner that builds cua-driver if needed
-└── test_*.py # app and legacy e2e modules
-```
-
-## Running
-
-```bash
-cd libs/cua-driver/tests/e2e
-./run_tests.sh test_electron -v
-python3 -m pytest test_tauri.py -v
-```
-
-Set `CUA_DRIVER_BINARY` to test a specific binary. Without it, `run_tests.sh`
-builds `libs/cua-driver/rust` and uses `rust/target/debug/cua-driver`.
-
-## Harness Apps
-
-`test_electron.py` and `test_tauri.py` use repo-local shared harnesses:
-
-- `tests/fixtures/apps/cross-platform/electron`
-- `tests/fixtures/apps/cross-platform/tauri`
-
-The fixtures build missing staged apps on demand with each harness `build.sh`.
-They no longer download prebuilt app releases.
-
-## Rules
-
-- Assert through observable state in the AX/UIA/AT-SPI tree or screenshots.
-- Do not call internal browser APIs from tests.
-- Keep app-specific setup in fixtures, not in driver code.
-- Treat older root-level tests as legacy coverage until they are audited and
- either ported into the shared harness pattern or removed.
diff --git a/libs/cua-driver/tests/e2e/assets/blender/toolbar_ref.png b/libs/cua-driver/tests/e2e/assets/blender/toolbar_ref.png
deleted file mode 100644
index 6854ba2aab..0000000000
Binary files a/libs/cua-driver/tests/e2e/assets/blender/toolbar_ref.png and /dev/null differ
diff --git a/libs/cua-driver/tests/e2e/assets/test_page.html b/libs/cua-driver/tests/e2e/assets/test_page.html
deleted file mode 120000
index 2771775117..0000000000
--- a/libs/cua-driver/tests/e2e/assets/test_page.html
+++ /dev/null
@@ -1 +0,0 @@
-../../../../cua-driver-fixtures/test_page.html
\ No newline at end of file
diff --git a/libs/cua-driver/tests/e2e/conftest.py b/libs/cua-driver/tests/e2e/conftest.py
deleted file mode 100644
index 125bd4583a..0000000000
--- a/libs/cua-driver/tests/e2e/conftest.py
+++ /dev/null
@@ -1,292 +0,0 @@
-"""pytest fixtures for cua-driver Python e2e tests.
-
-Fixtures:
- binary → str — path to cua-driver binary
- driver → Driver (context manager, yields a started Driver)
- focus_monitor → (proc, pid) — FocusMonitorApp running as frontmost sentinel
- ux_guard → UXMonitor — started before the test, assert_clean() after
- html_server → str — base URL of a local HTTP server serving assets/
- tauri_app → (proc, pid, base_url) — repo-local Tauri harness
- electron_app → (proc, pid, base_url) — repo-local Electron harness
-"""
-
-from __future__ import annotations
-
-import os
-import subprocess
-import sys
-import time
-import threading
-from http.server import HTTPServer, SimpleHTTPRequestHandler
-
-import pytest
-
-_HERE = os.path.dirname(os.path.abspath(__file__))
-_TESTS_ROOT = os.path.dirname(_HERE)
-_CUA_DRIVER_ROOT = os.path.dirname(_TESTS_ROOT)
-_DRIVER_RS_ROOT = os.path.join(_CUA_DRIVER_ROOT, "rust")
-_LIBS_ROOT = os.path.dirname(_CUA_DRIVER_ROOT)
-_TEST_HARNESS_ROOT = os.path.join(_TESTS_ROOT, "fixtures")
-
-sys.path.insert(0, _HERE)
-from driver_client import default_binary_path # noqa: E402
-
-_ASSETS_DIR = os.path.join(_HERE, "assets")
-_LOSS_FILE = "/tmp/focus_monitor_losses.txt"
-_FOCUS_APP_DIR = os.path.join(_LIBS_ROOT, "cua-driver", "Tests", "FocusMonitorApp")
-_FOCUS_APP_BUNDLE = os.path.join(_FOCUS_APP_DIR, "FocusMonitorApp.app")
-_FOCUS_APP_EXE = os.path.join(_FOCUS_APP_BUNDLE, "Contents", "MacOS", "FocusMonitorApp")
-
-# Tauri is built from the repo-local shared harness rather than downloaded.
-_TAURI_HARNESS_DIR = os.path.join(
- _TEST_HARNESS_ROOT, "apps", "cross-platform", "tauri"
-)
-_TAURI_BUILD_SCRIPT = os.path.join(_TAURI_HARNESS_DIR, "build.sh")
-_TAURI_APP_BUNDLE = os.path.join(
- _DRIVER_RS_ROOT,
- "test-apps",
- "harness-tauri",
- "CuaTestHarness.Tauri.app",
-)
-_TAURI_APP_EXE = os.path.join(
- _TAURI_APP_BUNDLE, "Contents", "MacOS", "CuaTestHarness.Tauri"
-)
-
-# Electron is built from the repo-local shared harness rather than downloaded.
-_ELECTRON_HARNESS_DIR = os.path.join(
- _TEST_HARNESS_ROOT, "apps", "cross-platform", "electron"
-)
-_ELECTRON_BUILD_SCRIPT = os.path.join(_ELECTRON_HARNESS_DIR, "build.sh")
-_ELECTRON_APP_BUNDLE = os.path.join(
- _DRIVER_RS_ROOT,
- "test-apps",
- "harness-electron",
- "CuaTestHarness.Electron.app",
-)
-_ELECTRON_APP_EXE = os.path.join(
- _ELECTRON_APP_BUNDLE, "Contents", "MacOS", "Electron"
-)
-
-
-# ── binary ────────────────────────────────────────────────────────────────────
-
[email protected](scope="session")
-def binary() -> str:
- return default_binary_path()
-
-
-# ── driver ────────────────────────────────────────────────────────────────────
-
-def driver(binary):
- """Yield a started Driver instance."""
- sys.path.insert(0, _HERE)
- from harness.driver import Driver
- with Driver(binary) as d:
- yield d
-
-
-# ── focus monitor ─────────────────────────────────────────────────────────────
-
-def _build_focus_app() -> None:
- if not os.path.exists(_FOCUS_APP_EXE):
- build_sh = os.path.join(_FOCUS_APP_DIR, "build.sh")
- subprocess.run([build_sh], check=True)
-
-
-def _launch_focus_app():
- """Launch FocusMonitorApp, return (proc, pid)."""
- _build_focus_app()
- proc = subprocess.Popen(
- [_FOCUS_APP_EXE],
- stdout=subprocess.PIPE,
- stderr=subprocess.DEVNULL,
- text=True,
- )
- for _ in range(40):
- line = proc.stdout.readline().strip()
- if line.startswith("FOCUS_PID="):
- pid = int(line.split("=", 1)[1])
- return proc, pid
- time.sleep(0.1)
- proc.terminate()
- raise RuntimeError("FocusMonitorApp did not print FOCUS_PID in time")
-
-
-def _wait_for_no_focus_app(timeout_s: float = 3.0) -> None:
- """Block until no FocusMonitorApp process is running."""
- deadline = time.monotonic() + timeout_s
- while time.monotonic() < deadline:
- r = subprocess.run(
- ["pgrep", "-x", "FocusMonitorApp"],
- capture_output=True,
- )
- if r.returncode != 0:
- return
- time.sleep(0.1)
-
-
[email protected](scope="module")
-def focus_monitor():
- """Launch FocusMonitorApp as frontmost sentinel for the test module."""
- # Kill any stale instances from previous modules and wait for full exit.
- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)
- _wait_for_no_focus_app()
- try:
- os.remove(_LOSS_FILE)
- except FileNotFoundError:
- pass
-
- proc, pid = _launch_focus_app()
- time.sleep(0.8)
- yield proc, pid
-
- proc.terminate()
- try:
- proc.wait(timeout=3)
- except subprocess.TimeoutExpired:
- proc.kill()
- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)
- # Wait for the process to fully exit before the next module starts.
- _wait_for_no_focus_app()
- try:
- os.remove(_LOSS_FILE)
- except FileNotFoundError:
- pass
-
-
[email protected](autouse=False)
-def activate_focus_monitor(focus_monitor):
- """Re-activate FocusMonitorApp before each test (by pid)."""
- _, pid = focus_monitor
- subprocess.run(
- ["osascript", "-e",
- f'tell application "System Events" to set frontmost of (first process whose unix id is {pid}) to true'],
- check=False,
- )
- time.sleep(0.4)
-
-
-# ── UX guard ──────────────────────────────────────────────────────────────────
-
-def ux_guard(focus_monitor):
- """Start UXMonitor before the test, call assert_clean() after."""
- _, sentinel_pid = focus_monitor
- sys.path.insert(0, _HERE)
- from harness.monitor import UXMonitor
- mon = UXMonitor(sentinel_pid=sentinel_pid)
- mon.start()
- yield mon
- mon.stop()
- mon.assert_clean()
-
-
-# ── local HTML server ─────────────────────────────────────────────────────────
-
-class _SilentHandler(SimpleHTTPRequestHandler):
- def log_message(self, *args):
- pass
-
-
[email protected](scope="session")
-def html_server():
- """Serve the assets/ directory over HTTP; return base URL."""
- handler = lambda *args, **kwargs: _SilentHandler(
- *args, directory=_ASSETS_DIR, **kwargs
- )
- server = HTTPServer(("127.0.0.1", 0), handler)
- port = server.server_address[1]
- thread = threading.Thread(target=server.serve_forever, daemon=True)
- thread.start()
- yield f"http://127.0.0.1:{port}"
- server.shutdown()
-
-
-# ── Tauri app ─────────────────────────────────────────────────────────────────
-
[email protected](scope="session")
-def tauri_app():
- """Build, launch, and yield (proc, pid, base_url) for the Tauri harness.
-
- This uses tests/fixtures/apps/cross-platform/tauri, the same shared DOM
- harness used by the Electron/WebView/WKWebView harnesses. It does not
- download a prebuilt Tauri app release.
- """
- if not os.path.exists(_TAURI_APP_EXE):
- if not os.path.exists(_TAURI_BUILD_SCRIPT):
- pytest.skip("repo-local Tauri harness build script not found")
- try:
- subprocess.run([_TAURI_BUILD_SCRIPT], check=True)
- except (OSError, subprocess.CalledProcessError) as e:
- pytest.skip(f"Tauri harness build failed: {e}")
-
- if not os.path.exists(_TAURI_APP_EXE):
- pytest.skip("Tauri harness app not found after build")
-
- subprocess.run(["pkill", "-f", "CuaTestHarness.Tauri"], check=False)
- time.sleep(0.5)
-
- proc = subprocess.Popen(
- [_TAURI_APP_EXE],
- stdout=subprocess.DEVNULL,
- stderr=subprocess.DEVNULL,
- )
- pid = proc.pid
- time.sleep(3.0)
-
- base_url = "file://shared-web-harness"
- yield proc, pid, base_url
-
- proc.terminate()
- try:
- proc.wait(timeout=5)
- except subprocess.TimeoutExpired:
- proc.kill()
- subprocess.run(["pkill", "-f", "CuaTestHarness.Tauri"], check=False)
-
-
-# ── Electron app ──────────────────────────────────────────────────────────────
-
[email protected](scope="session")
-def electron_app():
- """Build, launch, and yield (proc, pid, base_url) for the Electron harness.
-
- This uses tests/fixtures/apps/cross-platform/electron, the same shared DOM
- harness used by the Rust harness tests. It does not download a prebuilt
- Electron app release.
- """
- if not os.path.exists(_ELECTRON_APP_EXE):
- if not os.path.exists(_ELECTRON_BUILD_SCRIPT):
- pytest.skip("repo-local Electron harness build script not found")
- try:
- subprocess.run([_ELECTRON_BUILD_SCRIPT], check=True)
- except (OSError, subprocess.CalledProcessError) as e:
- pytest.skip(f"Electron harness build failed: {e}")
-
- if not os.path.exists(_ELECTRON_APP_EXE):
- pytest.skip("Electron harness app not found after build")
-
- subprocess.run(["pkill", "-f", "CuaTestHarness.Electron"], check=False)
- subprocess.run(["pkill", "-f", "Electron.app/Contents/MacOS/Electron"], check=False)
- time.sleep(0.5)
-
- proc = subprocess.Popen(
- [_ELECTRON_APP_EXE, "--force-renderer-accessibility"],
- stdout=subprocess.DEVNULL,
- stderr=subprocess.DEVNULL,
- )
- time.sleep(4.0) # Electron takes a bit longer with AX enabled
-
- pid = proc.pid
-
- base_url = "file://shared-web-harness"
- yield proc, pid, base_url
-
- proc.terminate()
- try:
- proc.wait(timeout=5)
- except subprocess.TimeoutExpired:
- proc.kill()
- subprocess.run(["pkill", "-f", "CuaTestHarness.Electron"], check=False)
- subprocess.run(["pkill", "-f", "Electron.app/Contents/MacOS/Electron"], check=False)
diff --git a/libs/cua-driver/tests/e2e/driver_client.py b/libs/cua-driver/tests/e2e/driver_client.py
deleted file mode 100644
index e6aefd43d2..0000000000
--- a/libs/cua-driver/tests/e2e/driver_client.py
+++ /dev/null
@@ -1,215 +0,0 @@
-"""Minimal MCP stdio client used by cua-driver-rs integration tests.
-
-Speaks JSON-RPC 2.0 line-framed over stdin/stdout. Keeps request IDs monotonic
-and strips notifications from the response stream. Deliberately tiny — we want
-the test failures to point at cua-driver-rs, not at a heavy client dependency.
-"""
-
-from __future__ import annotations
-
-import json
-import os
-import subprocess
-import threading
-import time
-from collections import deque
-from typing import Any, Optional
-
-
-class DriverClient:
- def __init__(self, binary_path: str, subcommand: str = "mcp") -> None:
- self.binary_path = binary_path
- self.subcommand = subcommand
- self.process: Optional[subprocess.Popen] = None
- self._next_id = 0
- self._lines: deque[str] = deque()
- self._reader: Optional[threading.Thread] = None
- self._stderr_reader: Optional[threading.Thread] = None
-
- def __enter__(self) -> "DriverClient":
- self.process = subprocess.Popen(
- [self.binary_path, self.subcommand],
- stdin=subprocess.PIPE,
- stdout=subprocess.PIPE,
- stderr=subprocess.PIPE,
- text=True,
- bufsize=1,
- )
- self._reader = threading.Thread(target=self._pump_stdout, daemon=True)
- self._reader.start()
- # Drain stderr to prevent pipe-buffer deadlock: the driver writes tracing
- # logs to stderr; if the 64KB OS pipe buffer fills the driver blocks and
- # the test hangs indefinitely waiting for an MCP response that never comes.
- self._stderr_reader = threading.Thread(target=self._drain_stderr, daemon=True)
- self._stderr_reader.start()
- self._handshake()
- return self
-
- def __exit__(self, *exc: Any) -> None:
- if not self.process:
- return
- for stream in (self.process.stdin, self.process.stdout, self.process.stderr):
- try:
- if stream is not None:
- stream.close()
- except Exception:
- pass
- try:
- self.process.terminate()
- self.process.wait(timeout=3)
- except Exception:
- self.process.kill()
- self.process.wait(timeout=1)
-
- def _pump_stdout(self) -> None:
- assert self.process and self.process.stdout
- for line in self.process.stdout:
- line = line.strip()
- if line:
- self._lines.append(line)
-
- def _drain_stderr(self) -> None:
- """Discard stderr output to prevent pipe-buffer deadlock."""
- assert self.process and self.process.stderr
- try:
- for _ in self.process.stderr:
- pass
- except Exception:
- pass
-
- def _handshake(self) -> None:
- self._call(
- "initialize",
- {
- "protocolVersion": "2025-06-18",
- "capabilities": {},
- "clientInfo": {"name": "cua-driver-rs-integration", "version": "0.0.1"},
- },
- )
- self._notify("notifications/initialized")
-
- def _notify(self, method: str, params: Optional[dict] = None) -> None:
- payload = {"jsonrpc": "2.0", "method": method}
- if params is not None:
- payload["params"] = params
- self._write(payload)
-
- def _call(
- self, method: str, params: Optional[dict] = None, timeout: float = 20.0
- ) -> dict:
- self._next_id += 1
- request_id = self._next_id
- payload = {"jsonrpc": "2.0", "id": request_id, "method": method}
- if params is not None:
- payload["params"] = params
- self._write(payload)
-
- deadline = time.monotonic() + timeout
- while time.monotonic() < deadline:
- while self._lines:
- line = self._lines.popleft()
- try:
- msg = json.loads(line)
- except json.JSONDecodeError:
- continue
- if msg.get("id") == request_id:
- if "error" in msg:
- raise MCPCallError(msg["error"])
- return msg["result"]
- time.sleep(0.02)
- raise TimeoutError(f"no response for {method} within {timeout}s")
-
- def _write(self, payload: dict) -> None:
- assert self.process and self.process.stdin
- self.process.stdin.write(json.dumps(payload) + "\n")
- self.process.stdin.flush()
-
- def list_tools(self) -> list[dict]:
- return self._call("tools/list")["tools"]
-
- def call_tool(self, name: str, arguments: Optional[dict] = None) -> dict:
- return self._call(
- "tools/call",
- {"name": name, "arguments": arguments or {}},
- )
-
-
-class MCPCallError(RuntimeError):
- def __init__(self, error: dict) -> None:
- super().__init__(f"MCP error {error.get('code')}: {error.get('message')}")
- self.code = error.get("code")
- self.message = error.get("message")
-
-
-def default_binary_path() -> str:
- """Return the path to the cua-driver-rs binary.
-
- Checks CUA_DRIVER_BINARY env var first, then falls back to the debug build
- in the Rust workspace target directory.
- """
- return os.environ.get(
- "CUA_DRIVER_BINARY",
- os.path.join(
- os.path.dirname(os.path.abspath(__file__)),
- "..",
- "..",
- "rust",
- "target",
- "debug",
- "cua-driver",
- ),
- )
-
-
-def reset_calculator(settle_s: float = 0.3) -> None:
- """Force a known clean state for Calculator.
-
- Several tests drive Calculator as a 'freshly launched, never frontmost'
- target. `pkill -x Calculator` + short settle is the shared boilerplate.
- """
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
- time.sleep(settle_s)
-
-
-def frontmost_bundle_id(client: "DriverClient") -> Optional[str]:
- """Return the bundle id of the currently-frontmost app, or None."""
- apps = client.call_tool("list_apps")["structuredContent"]["apps"]
- for app in apps:
- if app.get("active"):
- return app.get("bundle_id")
- return None
-
-
-def resolve_window_id(
- client: "DriverClient", pid: int, require_on_current_space: bool = True
-) -> int:
- """Pick a `window_id` for `pid` via `list_windows`.
-
- Prefers windows that are on-screen AND on the user's current Space,
- then max z_index. Falls back to any layer-0 window for the pid when
- the preferred filter finds nothing (hidden-launched / minimized apps
- still have layer-0 windows we can snapshot).
-
- Raises `RuntimeError` when the pid has no layer-0 window at all.
- """
- result = client.call_tool("list_windows", {"pid": pid})
- windows = result["structuredContent"]["windows"]
- if not windows:
- raise RuntimeError(f"pid {pid} has no windows")
-
- if require_on_current_space:
- preferred = [
- w for w in windows
- if w.get("is_on_screen") and w.get("on_current_space") is not False
- ]
- if preferred:
- preferred.sort(key=lambda w: w.get("z_index", 0), reverse=True)
- return preferred[0]["window_id"]
-
- # Fallback: any layer-0 window. Pick the one with the biggest bounds.
- def _area(w: dict) -> float:
- b = w.get("bounds", {})
- return float(b.get("width", 0)) * float(b.get("height", 0))
-
- windows.sort(key=_area, reverse=True)
- return windows[0]["window_id"]
diff --git a/libs/cua-driver/tests/e2e/fixtures/interactive.html b/libs/cua-driver/tests/e2e/fixtures/interactive.html
deleted file mode 120000
index d053b042a9..0000000000
--- a/libs/cua-driver/tests/e2e/fixtures/interactive.html
+++ /dev/null
@@ -1 +0,0 @@
-../../../../cua-driver-fixtures/interactive.html
\ No newline at end of file
diff --git a/libs/cua-driver/tests/e2e/fixtures/test_cursor.png b/libs/cua-driver/tests/e2e/fixtures/test_cursor.png
deleted file mode 100644
index 72129ae29a..0000000000
Binary files a/libs/cua-driver/tests/e2e/fixtures/test_cursor.png and /dev/null differ
diff --git a/libs/cua-driver/tests/e2e/harness/__init__.py b/libs/cua-driver/tests/e2e/harness/__init__.py
deleted file mode 100644
index e69de29bb2..0000000000
diff --git a/libs/cua-driver/tests/e2e/harness/cv.py b/libs/cua-driver/tests/e2e/harness/cv.py
deleted file mode 100644
index d254ebfffe..0000000000
--- a/libs/cua-driver/tests/e2e/harness/cv.py
+++ /dev/null
@@ -1,132 +0,0 @@
-"""Computer-vision helpers for screenshot-based assertions.
-
-All operations use PIL (Pillow) + numpy only — no OpenCV dependency.
-
-Functions:
- decode(b64_str) → PIL.Image
- crop(img, bbox) → PIL.Image (bbox: (x, y, w, h))
- diff_ratio(img_a, img_b) → float in [0, 1]
- find_template(haystack, needle, threshold) → (x, y) or None (NCC)
- save_reference(img, path) → None
- load_reference(path) → PIL.Image
-"""
-
-from __future__ import annotations
-
-import base64
-import io
-import os
-from typing import Optional, Tuple
-
-try:
- from PIL import Image, ImageChops
- import numpy as np
- _PIL_AVAILABLE = True
-except ImportError:
- _PIL_AVAILABLE = False
-
-
-BBox = Tuple[int, int, int, int] # x, y, w, h
-
-
-def decode(b64_str: str) -> "Image.Image":
- """Decode a base64 PNG/JPEG string into a PIL Image."""
- if not _PIL_AVAILABLE:
- raise ImportError("Pillow is required for cv helpers: pip install pillow")
- data = base64.b64decode(b64_str)
- return Image.open(io.BytesIO(data)).convert("RGB")
-
-
-def crop(img: "Image.Image", bbox: BBox) -> "Image.Image":
- """Crop a PIL image using (x, y, w, h) bounding box."""
- x, y, w, h = bbox
- return img.crop((x, y, x + w, y + h))
-
-
-def diff_ratio(img_a: "Image.Image", img_b: "Image.Image") -> float:
- """Return fraction of pixels that differ between two same-size images.
-
- Returns a value in [0, 1]. 0 = identical, 1 = completely different.
- """
- if not _PIL_AVAILABLE:
- raise ImportError("Pillow is required for cv helpers")
- import numpy as np # noqa: F811
-
- a = np.array(img_a.resize(img_b.size), dtype=np.float32)
- b = np.array(img_b, dtype=np.float32)
- diff = np.abs(a - b)
- changed = np.any(diff > 8, axis=-1) # threshold per-channel
- return float(changed.sum()) / float(changed.size)
-
-
-def find_template(
- haystack: "Image.Image",
- needle: "Image.Image",
- threshold: float = 0.85,
-) -> Optional[Tuple[int, int]]:
- """Find `needle` inside `haystack` using normalised cross-correlation (NCC).
-
- Returns (x, y) top-left of the best match if similarity >= threshold,
- else None. No OpenCV required — uses numpy FFT-based NCC.
- """
- if not _PIL_AVAILABLE:
- raise ImportError("Pillow + numpy required for find_template")
- import numpy as np
-
- h_arr = np.array(haystack.convert("L"), dtype=np.float32)
- n_arr = np.array(needle.convert("L"), dtype=np.float32)
-
- H, W = h_arr.shape
- th, tw = n_arr.shape
-
- if th > H or tw > W:
- return None
-
- # Normalise needle
- n_mean = n_arr.mean()
- n_std = n_arr.std()
- if n_std < 1e-6:
- return None
- n_norm = (n_arr - n_mean) / n_std
-
- # Sliding-window NCC via FFT
- from numpy.fft import fft2, ifft2, fftshift
-
- # Zero-pad haystack to same size for FFT correlation
- pad_h = H + th - 1
- pad_w = W + tw - 1
-
- H_fft = fft2(h_arr, s=(pad_h, pad_w))
- N_fft = fft2(n_norm[::-1, ::-1], s=(pad_h, pad_w))
- corr = np.real(ifft2(H_fft * N_fft))
-
- # Local haystack std for normalisation
- from numpy.lib.stride_tricks import sliding_window_view
- windows = sliding_window_view(h_arr, (th, tw))
- local_std = windows.reshape(-1, th * tw).std(axis=1).reshape(H - th + 1, W - tw + 1)
- local_std[local_std < 1e-6] = 1.0
-
- # Trim correlation to valid region
- corr_valid = corr[th - 1:th - 1 + (H - th + 1), tw - 1:tw - 1 + (W - tw + 1)]
- ncc = corr_valid / (local_std * th * tw)
-
- best_idx = np.argmax(ncc)
- best_score = ncc.flat[best_idx]
- if best_score < threshold:
- return None
-
- ry, rx = divmod(int(best_idx), W - tw + 1)
- return (rx, ry)
-
-
-def save_reference(img: "Image.Image", path: str) -> None:
- """Save a PIL Image as a PNG reference for later template matching."""
- os.makedirs(os.path.dirname(path), exist_ok=True)
- img.save(path, "PNG")
-
-
-def load_reference(path: str) -> "Image.Image":
- """Load a reference PNG for template matching."""
- if not _PIL_AVAILABLE:
- raise ImportError("Pillow required")
- return Image.open(path).convert("RGB")
diff --git a/libs/cua-driver/tests/e2e/harness/driver.py b/libs/cua-driver/tests/e2e/harness/driver.py
deleted file mode 100644
index 1409768f5b..0000000000
--- a/libs/cua-driver/tests/e2e/harness/driver.py
+++ /dev/null
@@ -1,256 +0,0 @@
-"""Typed MCP driver wrapper for Python e2e tests.
-
-Wraps DriverClient with:
-- Typed WindowState return value
-- Helper to find a window for a given pid
-- Convenient call shortcuts
-
-Usage::
-
- with Driver(binary_path) as d:
- state = d.get_window_state(pid, wid)
- idx = state.find_element(label="Click Me")
- d.click(pid, wid, element_index=idx)
- state2 = d.get_window_state(pid, wid)
- assert "clicks: 1" in state2.tree
-"""
-
-from __future__ import annotations
-
-import os
-import sys
-import time
-from dataclasses import dataclass, field
-from typing import Optional
-
-# Allow importing from the e2e root.
-_HERE = os.path.dirname(os.path.abspath(__file__))
-_E2E_DIR = os.path.dirname(_HERE)
-sys.path.insert(0, _E2E_DIR)
-
-from driver_client import DriverClient, default_binary_path, MCPCallError # noqa: E402
-
-
-@dataclass
-class WindowState:
- tree: str
- screenshot_b64: str
- screenshot_width: int
- screenshot_height: int
- raw: dict = field(default_factory=dict)
-
- def find_element(self, label: str) -> Optional[int]:
- """Return the first element index whose line contains `label`."""
- import re
- for line in self.tree.split("\n"):
- if label in line:
- m = re.search(r'\[(\d+)\]', line)
- if m:
- return int(m.group(1))
- return None
-
- def find_button(self, label: str) -> Optional[int]:
- """Return the first AXButton element index matching `label`."""
- import re
- for line in self.tree.split("\n"):
- if "AXButton" not in line:
- continue
- if label not in line:
- continue
- m = re.search(r'\[(\d+)\]', line)
- if m:
- return int(m.group(1))
- return None
-
- def find_text_field(self, skip_url_bar: bool = True) -> Optional[int]:
- """Return the first AXTextField index (skipping Safari URL bar)."""
- import re
- for line in self.tree.split("\n"):
- if "AXTextField" not in line:
- continue
- if skip_url_bar and "smart search field" in line:
- continue
- m = re.search(r'\[(\d+)\]', line)
- if m:
- return int(m.group(1))
- return None
-
- def ax_value(self, element_index: int) -> Optional[str]:
- """Return the value= attribute of the element at `element_index`."""
- import re
- prefix = f"[{element_index}]"
- for line in self.tree.split("\n"):
- if prefix not in line:
- continue
- m = re.search(r'value="([^"]*)"', line)
- if m:
- return m.group(1)
- return None
-
- def has_text(self, text: str) -> bool:
- return text in self.tree
-
-
-class Driver:
- """Thin typed wrapper around DriverClient."""
-
- def __init__(self, binary_path: Optional[str] = None) -> None:
- self.binary_path = binary_path or default_binary_path()
- self._client: Optional[DriverClient] = None
-
- def __enter__(self) -> "Driver":
- self._client = DriverClient(self.binary_path).__enter__()
- return self
-
- def __exit__(self, *exc) -> None:
- if self._client is not None:
- self._client.__exit__(*exc)
- self._client = None
-
- # ── window helpers ────────────────────────────────────────────────────────
-
- def list_apps(self) -> list[dict]:
- return self._c().call_tool("list_apps")["structuredContent"]["apps"]
-
- def find_app(self, bundle_id: str) -> Optional[dict]:
- for app in self.list_apps():
- if app.get("bundle_id") == bundle_id:
- return app
- return None
-
- def find_window(self, pid: int, min_area: float = 400.0) -> int:
- """Return a usable window_id for `pid`.
-
- Prefers on-screen windows on the current space. Among those, picks
- the one with the LARGEST area so that Electron/Chromium overlay/helper
- sub-windows (which are typically tiny or zero-size) are skipped in
- favour of the main content window.
- """
- result = self._c().call_tool("list_windows", {"pid": pid})
- windows = result["structuredContent"]["windows"]
- if not windows:
- raise RuntimeError(f"pid {pid} has no windows")
-
- def _area(w: dict) -> float:
- b = w.get("bounds", {})
- return float(b.get("width", 0)) * float(b.get("height", 0))
-
- # Filter to on-screen windows with meaningful size.
- candidates = [
- w for w in windows
- if w.get("is_on_screen") and w.get("on_current_space") is not False
- and _area(w) >= min_area
- ]
- if not candidates:
- # Fallback: all on-screen windows, then all windows.
- candidates = [w for w in windows if w.get("is_on_screen")] or windows
-
- # Among candidates, pick the one with the largest area.
- candidates.sort(key=_area, reverse=True)
- return candidates[0]["window_id"]
-
- # ── MCP tool wrappers ─────────────────────────────────────────────────────
-
- def get_window_state(
- self,
- pid: int,
- window_id: int,
- query: Optional[str] = None,
- ) -> WindowState:
- args: dict = {"pid": pid, "window_id": window_id}
- if query is not None:
- args["query"] = query
- result = self._c().call_tool("get_window_state", args)
- sc = result.get("structuredContent", result)
- tree = sc.get("tree_markdown", "")
- screenshot = ""
- for item in result.get("content", []):
- if item.get("type") == "image":
- screenshot = item.get("data", "")
- break
- width = sc.get("screenshot_width", 0)
- height = sc.get("screenshot_height", 0)
- return WindowState(
- tree=tree,
- screenshot_b64=screenshot,
- screenshot_width=width,
- screenshot_height=height,
- raw=result,
- )
-
- def click(
- self,
- pid: int,
- window_id: int,
- element_index: Optional[int] = None,
- x: Optional[float] = None,
- y: Optional[float] = None,
- ) -> dict:
- args: dict = {"pid": pid, "window_id": window_id}
- if element_index is not None:
- args["element_index"] = element_index
- if x is not None:
- args["x"] = x
- if y is not None:
- args["y"] = y
- return self._c().call_tool("click", args)
-
- def double_click(
- self,
- pid: int,
- window_id: int,
- element_index: Optional[int] = None,
- x: Optional[float] = None,
- y: Optional[float] = None,
- ) -> dict:
- args: dict = {"pid": pid, "window_id": window_id}
- if element_index is not None:
- args["element_index"] = element_index
- if x is not None:
- args["x"] = x
- if y is not None:
- args["y"] = y
- return self._c().call_tool("double_click", args)
-
- def type_text(self, pid: int, text: str) -> dict:
- return self._c().call_tool("type_text", {"pid": pid, "text": text})
-
- def type_text_chars(self, pid: int, text: str) -> dict:
- return self._c().call_tool("type_text_chars", {"pid": pid, "text": text})
-
- def press_key(self, pid: int, key: str) -> dict:
- return self._c().call_tool("press_key", {"pid": pid, "key": key})
-
- def hotkey(self, pid: int, keys: list[str]) -> dict:
- return self._c().call_tool("hotkey", {"pid": pid, "keys": keys})
-
- def scroll(
- self,
- pid: int,
- window_id: int,
- x: float,
- y: float,
- delta_x: float = 0.0,
- delta_y: float = -3.0,
- ) -> dict:
- return self._c().call_tool("scroll", {
- "pid": pid,
- "window_id": window_id,
- "x": x,
- "y": y,
- "delta_x": delta_x,
- "delta_y": delta_y,
- })
-
- def launch_app(self, bundle_id: str) -> dict:
- return self._c().call_tool("launch_app", {"bundle_id": bundle_id})
-
- def call_tool(self, name: str, args: Optional[dict] = None) -> dict:
- return self._c().call_tool(name, args or {})
-
- # ── internals ────────────────────────────────────────────────────────────
-
- def _c(self) -> DriverClient:
- if self._client is None:
- raise RuntimeError("Driver not started — use as context manager")
- return self._client
diff --git a/libs/cua-driver/tests/e2e/harness/monitor.py b/libs/cua-driver/tests/e2e/harness/monitor.py
deleted file mode 100644
index c3469a5451..0000000000
--- a/libs/cua-driver/tests/e2e/harness/monitor.py
+++ /dev/null
@@ -1,224 +0,0 @@
-"""UX invariant monitor — polls at 5 ms via a background thread.
-
-Signals checked on every tick:
- - Real cursor position: NSEvent.mouseLocation() — violation if moved > 4 px
- - Frontmost app PID: NSWorkspace.sharedWorkspace().frontmostApplication() — violation if != sentinel_pid
- - Agent overlay z-order: CGWindowListCopyWindowInfo — violation if a cua-driver
- overlay window layer >= frontmost non-agent window layer
-
-Usage::
-
- mon = UXMonitor(sentinel_pid=focus_monitor_pid)
- mon.start()
- # ... run test ...
- mon.stop()
- mon.assert_clean() # raises AssertionError if any violation was recorded
-"""
-
-from __future__ import annotations
-
-import threading
-import time
-from dataclasses import dataclass, field
-from typing import Optional
-
-
-# Try to import PyObjC; fall back to no-op monitor on systems without it.
-try:
- from AppKit import NSWorkspace
- from AppKit import NSEvent
- from Quartz import (
- CGWindowListCopyWindowInfo,
- kCGWindowListOptionOnScreenOnly,
- kCGNullWindowID,
- kCGWindowLayer,
- kCGWindowOwnerPID,
- kCGWindowName,
- )
- _PYOBJC_AVAILABLE = True
-except ImportError:
- _PYOBJC_AVAILABLE = False
-
-
-# Cursor movement tolerance in logical screen pixels.
-_CURSOR_TOLERANCE_PX = 4.0
-
-# Polling interval in seconds.
-_POLL_INTERVAL_S = 0.005 # 5 ms
-
-
-@dataclass
-class Violation:
- kind: str # "cursor" | "frontmost" | "overlay_z"
- detail: str
- timestamp: float = field(default_factory=time.monotonic)
-
- def __str__(self) -> str:
- return f"[{self.kind}] {self.detail} (t={self.timestamp:.3f})"
-
-
-class UXMonitor:
- """5 ms background poller that records UX invariant violations."""
-
- def __init__(
- self,
- sentinel_pid: Optional[int] = None,
- cursor_tolerance_px: float = _CURSOR_TOLERANCE_PX,
- check_cursor: bool = True,
- check_frontmost: bool = True,
- check_overlay_z: bool = True,
- ) -> None:
- self.sentinel_pid = sentinel_pid
- self.cursor_tolerance_px = cursor_tolerance_px
- self.check_cursor = check_cursor and _PYOBJC_AVAILABLE
- self.check_frontmost = check_frontmost and _PYOBJC_AVAILABLE
- self.check_overlay_z = check_overlay_z and _PYOBJC_AVAILABLE
-
- self._violations: list[Violation] = []
- self._lock = threading.Lock()
- self._stop_event = threading.Event()
- self._thread: Optional[threading.Thread] = None
-
- # Baseline cursor position captured at start().
- self._baseline_x: float = 0.0
- self._baseline_y: float = 0.0
-
- # Grace period: only record frontmost violations after the sentinel has
- # been observed as frontmost at least once. This prevents false positives
- # when the monitor starts before the activation fixture runs.
- self._sentinel_seen: bool = False
-
- # ── public API ───────────────────────────────────────────────────────────
-
- def start(self) -> "UXMonitor":
- if not _PYOBJC_AVAILABLE:
- return self
- self._baseline_x, self._baseline_y = self._cursor_pos()
- self._stop_event.clear()
- self._thread = threading.Thread(target=self._run, daemon=True, name="ux-monitor")
- self._thread.start()
- return self
-
- def stop(self) -> None:
- if self._thread is None:
- return
- self._stop_event.set()
- self._thread.join(timeout=1.0)
- self._thread = None
-
- def violations(self) -> list[Violation]:
- with self._lock:
- return list(self._violations)
-
- def assert_clean(self) -> None:
- """Raise AssertionError if any violations were recorded."""
- with self._lock:
- vs = list(self._violations)
- if vs:
- lines = "\n".join(f" {v}" for v in vs)
- raise AssertionError(f"UX violations detected ({len(vs)}):\n{lines}")
-
- def reset_baseline(self) -> None:
- """Reset cursor baseline to current position (e.g. after intentional move)."""
- if _PYOBJC_AVAILABLE:
- self._baseline_x, self._baseline_y = self._cursor_pos()
-
- # ── internals ────────────────────────────────────────────────────────────
-
- def _run(self) -> None:
- while not self._stop_event.wait(timeout=_POLL_INTERVAL_S):
- self._tick()
-
- def _tick(self) -> None:
- try:
- if self.check_cursor:
- self._check_cursor()
- if self.check_frontmost and self.sentinel_pid is not None:
- self._check_frontmost()
- if self.check_overlay_z:
- self._check_overlay_z()
- except Exception:
- pass # never crash the monitor thread
-
- def _record(self, kind: str, detail: str) -> None:
- with self._lock:
- self._violations.append(Violation(kind=kind, detail=detail))
-
- # ── signal checkers ──────────────────────────────────────────────────────
-
- @staticmethod
- def _cursor_pos() -> tuple[float, float]:
- loc = NSEvent.mouseLocation()
- return float(loc.x), float(loc.y)
-
- def _check_cursor(self) -> None:
- x, y = self._cursor_pos()
- dx = x - self._baseline_x
- dy = y - self._baseline_y
- dist = (dx * dx + dy * dy) ** 0.5
- if dist > self.cursor_tolerance_px:
- self._record(
- "cursor",
- f"moved {dist:.1f}px from baseline "
- f"({self._baseline_x:.0f},{self._baseline_y:.0f}) "
- f"→ ({x:.0f},{y:.0f})",
- )
- # Update baseline so we don't spam the same violation.
- self._baseline_x, self._baseline_y = x, y
-
- def _check_frontmost(self) -> None:
- app = NSWorkspace.sharedWorkspace().frontmostApplication()
- if app is None:
- return
- pid = int(app.processIdentifier())
- if pid == self.sentinel_pid:
- # Sentinel is frontmost — record this so we exit the grace period.
- self._sentinel_seen = True
- return
- if not self._sentinel_seen:
- # Grace period: don't flag until sentinel has been frontmost at least once.
- return
- name = str(app.localizedName() or "?")
- self._record(
- "frontmost",
- f"frontmost changed to pid={pid} ({name}), "
- f"expected sentinel pid={self.sentinel_pid}",
- )
- # Don't spam — pause checking until next start
- self.check_frontmost = False
-
- def _check_overlay_z(self) -> None:
- """Check that no cua-driver overlay window is above foreground windows."""
- wins = CGWindowListCopyWindowInfo(
- kCGWindowListOptionOnScreenOnly, kCGNullWindowID
- )
- if not wins:
- return
-
- # Separate overlay windows from non-overlay windows
- overlay_layers: list[int] = []
- non_overlay_min_layer: int = 10000
-
- for w in wins:
- pid = int(w.get(kCGWindowOwnerPID, 0))
- name = str(w.get(kCGWindowName, "") or "")
- layer = int(w.get(kCGWindowLayer, 0))
-
- is_overlay = name in ("cua-agent-cursor", "cua-overlay") or \
- "cua-driver" in str(name).lower()
-
- if is_overlay:
- overlay_layers.append(layer)
- elif layer >= 0: # normal on-screen window
- non_overlay_min_layer = min(non_overlay_min_layer, layer)
-
- if not overlay_layers:
- return
-
- max_overlay = max(overlay_layers)
- if max_overlay >= non_overlay_min_layer and non_overlay_min_layer < 10000:
- self._record(
- "overlay_z",
- f"cua-driver overlay layer={max_overlay} >= "
- f"foreground window layer={non_overlay_min_layer}",
- )
diff --git a/libs/cua-driver/tests/e2e/harness/tree.py b/libs/cua-driver/tests/e2e/harness/tree.py
deleted file mode 100644
index adc2c4fe34..0000000000
--- a/libs/cua-driver/tests/e2e/harness/tree.py
+++ /dev/null
@@ -1,99 +0,0 @@
-"""AX tree_markdown parser and query helpers.
-
-tree_markdown is the line-based format emitted by get_window_state, e.g.::
-
- - [0] AXWindow "Safari" [id=main actions=[raise]]
- - [1] AXButton "Click Me" [actions=[press]]
- - [2] AXTextField [value="hello"] [actions=[set_value]]
- - [3] AXStaticText = "clicks: 3"
-
-Helpers:
- find(tree, role=None, label=None) → first matching element index
- find_all(tree, role=None, label=None) → all matching element indices
- ax_value(tree, index) → value= attribute or static text value
- ax_text(tree, index) → the label / title / description text on that line
-"""
-
-from __future__ import annotations
-
-import re
-from typing import Optional
-
-
-def _lines(tree: str) -> list[tuple[int, str]]:
- """Return (element_index, raw_line) pairs for every indexed line."""
- result: list[tuple[int, str]] = []
- for line in tree.split("\n"):
- m = re.search(r'\[(\d+)\]', line)
- if m:
- result.append((int(m.group(1)), line))
- return result
-
-
-def find(
- tree: str,
- role: Optional[str] = None,
- label: Optional[str] = None,
- skip_url_bar: bool = True,
-) -> Optional[int]:
- """Return the element_index of the first matching element."""
- for idx, line in _lines(tree):
- if role is not None and role not in line:
- continue
- if label is not None and label not in line:
- continue
- if skip_url_bar and "smart search field" in line:
- continue
- return idx
- return None
-
-
-def find_all(
- tree: str,
- role: Optional[str] = None,
- label: Optional[str] = None,
-) -> list[int]:
- """Return all matching element indices."""
- result: list[int] = []
- for idx, line in _lines(tree):
- if role is not None and role not in line:
- continue
- if label is not None and label not in line:
- continue
- result.append(idx)
- return result
-
-
-def ax_value(tree: str, index: int) -> Optional[str]:
- """Return value= attribute or static text (= "...") for `index`."""
- prefix = f"[{index}]"
- for line in tree.split("\n"):
- if prefix not in line:
- continue
- # value="..." attribute
- m = re.search(r'value="([^"]*)"', line)
- if m:
- return m.group(1)
- # AXStaticText = "..."
- m = re.search(r'= "([^"]*)"', line)
- if m:
- return m.group(1)
- return None
-
-
-def ax_text(tree: str, index: int) -> Optional[str]:
- """Return the display text (title or quoted string) for `index`."""
- prefix = f"[{index}]"
- for line in tree.split("\n"):
- if prefix not in line:
- continue
- # Quoted string after role: AXButton "label"
- m = re.search(r'"([^"]+)"', line)
- if m:
- return m.group(1)
- return line.strip()
- return None
-
-
-def contains_text(tree: str, text: str) -> bool:
- return text in tree
diff --git a/libs/cua-driver/tests/e2e/run_tests.sh b/libs/cua-driver/tests/e2e/run_tests.sh
deleted file mode 100755
index 2f98162239..0000000000
--- a/libs/cua-driver/tests/e2e/run_tests.sh
+++ /dev/null
@@ -1,97 +0,0 @@
-#!/usr/bin/env bash
-# Run cua-driver Python e2e tests.
-#
-# Usage:
-# ./run_tests.sh # run all tests (Rust binary)
-# ./run_tests.sh test_list_windows # run a specific test module
-# ./run_tests.sh test_cli test_api_parity # run multiple modules
-# ./run_tests.sh --parity # run parity suite against BOTH binaries
-#
-# The script auto-builds cua-driver (debug) before running unless
-# CUA_DRIVER_BINARY is already set in the environment.
-#
-# Options:
-# -v / --verbose Pass -v to pytest for verbose output.
-# --release Build with --release (sets CUA_DRIVER_BINARY to release binary).
-# --parity Run test_api_parity.py against both Swift and Rust binaries.
-# Requires CUA_SWIFT_BINARY or ~/.local/bin/cua-driver.
-#
-set -euo pipefail
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-CUA_DRIVER_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
-RUST_ROOT="$CUA_DRIVER_ROOT/rust"
-
-VERBOSE=""
-PROFILE="debug"
-PARITY_MODE=0
-MODULES=()
-
-for arg in "$@"; do
- case "$arg" in
- -v|--verbose) VERBOSE="-v" ;;
- --release) PROFILE="release" ;;
- --parity) PARITY_MODE=1 ;;
- test_*) MODULES+=("$arg") ;;
- *) echo "Unknown argument: $arg"; exit 1 ;;
- esac
-done
-
-# Build the binary if CUA_DRIVER_BINARY is not already set.
-if [ -z "${CUA_DRIVER_BINARY:-}" ]; then
- echo "==> Building cua-driver ($PROFILE)..."
- CARGO_FLAGS=""
- [ "$PROFILE" = "release" ] && CARGO_FLAGS="--release"
- (cd "$RUST_ROOT" && cargo build $CARGO_FLAGS -p cua-driver 2>&1)
- export CUA_DRIVER_BINARY="$RUST_ROOT/target/$PROFILE/cua-driver"
-fi
-
-echo "==> Using Rust binary: $CUA_DRIVER_BINARY"
-if [ ! -x "$CUA_DRIVER_BINARY" ]; then
- echo "ERROR: binary not found or not executable: $CUA_DRIVER_BINARY"
- exit 1
-fi
-
-# Auto-detect Swift binary for parity mode.
-if [ "$PARITY_MODE" -eq 1 ] && [ -z "${CUA_SWIFT_BINARY:-}" ]; then
- for candidate in "$HOME/.local/bin/cua-driver" "$(which cua-driver 2>/dev/null || true)"; do
- if [ -n "$candidate" ] && [ -x "$candidate" ]; then
- export CUA_SWIFT_BINARY="$candidate"
- break
- fi
- done
- if [ -z "${CUA_SWIFT_BINARY:-}" ]; then
- echo "WARNING: --parity requested but no Swift binary found."
- echo " Install cua-driver or set CUA_SWIFT_BINARY."
- echo " SwiftParityTests will be skipped."
- else
- echo "==> Using Swift binary: $CUA_SWIFT_BINARY"
- fi
-fi
-
-cd "$SCRIPT_DIR"
-
-if [ "$PARITY_MODE" -eq 1 ]; then
- echo "==> Running parity suite against both binaries..."
- python3 -m pytest ${VERBOSE} test_api_parity.py
- exit $?
-fi
-
-if [ "${#MODULES[@]}" -eq 0 ]; then
- # Discover all test_*.py files.
- mapfile -t MODULES < <(find . -maxdepth 1 -name 'test_*.py' -exec basename {} .py \; | sort)
-fi
-
-echo "==> Running ${#MODULES[@]} test module(s): ${MODULES[*]}"
-echo ""
-
-FILES=()
-for module in "${MODULES[@]}"; do
- if [[ "$module" == *.py ]]; then
- FILES+=("$module")
- else
- FILES+=("$module.py")
- fi
-done
-
-python3 -m pytest ${VERBOSE} "${FILES[@]}"
diff --git a/libs/cua-driver/tests/e2e/test_api_parity.py b/libs/cua-driver/tests/e2e/test_api_parity.py
deleted file mode 100644
index 5413319378..0000000000
--- a/libs/cua-driver/tests/e2e/test_api_parity.py
+++ /dev/null
@@ -1,1182 +0,0 @@
-"""Binary-level API parity tests: Swift cua-driver vs Rust cua-driver-rs.
-
-Covers every public API surface of both binaries:
- • CLI subcommands (list-tools, describe, call, mcp-config, serve/stop/status,
- recording, dump-docs, update, doctor, diagnose, config, --version, --help)
- • Stdio MCP protocol (tools/list + every registered tool)
- • Embedded daemon mode (serve socket lifecycle)
-
-Both binaries are tested via the *same* mixin class so any divergence shows
-up as a test failure attributed to the specific binary.
-
-Environment variables:
- CUA_SWIFT_BINARY – path to the Swift binary
- (default: ~/.local/bin/cua-driver or `which cua-driver`)
- CUA_DRIVER_BINARY – path to the Rust binary
- (default: ../../target/debug/cua-driver)
-
-Run all parity tests against both binaries::
-
- python3 -m unittest test_api_parity -v
-
-Run against Rust binary only::
-
- CUA_DRIVER_BINARY=../../target/release/cua-driver \\
- python3 -m unittest test_api_parity.RustParityTests -v
-
-Run against Swift binary only::
-
- python3 -m unittest test_api_parity.SwiftParityTests -v
-"""
-
-from __future__ import annotations
-
-import json
-import os
-import shutil
-import subprocess
-import sys
-import tempfile
-import time
-import unittest
-
-_HERE = os.path.dirname(os.path.abspath(__file__))
-sys.path.insert(0, _HERE)
-
-from driver_client import DriverClient, MCPCallError, default_binary_path # noqa: E402
-
-# ── helpers ────────────────────────────────────────────────────────────────────
-
-def _run(
- args: list[str],
- timeout: int = 20,
- stdin: str | None = None,
-) -> subprocess.CompletedProcess:
- return subprocess.run(
- args,
- capture_output=True,
- text=True,
- timeout=timeout,
- input=stdin,
- )
-
-
-def _swift_binary() -> str | None:
- """Return Swift cua-driver binary path, or None if not found."""
- from_env = os.environ.get("CUA_SWIFT_BINARY", "")
- if from_env and os.path.isfile(from_env):
- return from_env
- candidates = [
- os.path.expanduser("~/.local/bin/cua-driver"),
- ]
- found = shutil.which("cua-driver")
- if found:
- candidates.insert(0, found)
- for p in candidates:
- if os.path.isfile(p) and os.access(p, os.X_OK):
- return p
- return None
-
-
-# ── tool presence lists ────────────────────────────────────────────────────────
-#
-# REQUIRED_TOOLS – tools that BOTH binaries must expose (tests fail on either
-# binary when a listed tool is absent).
-# RUST_ONLY_TOOLS – tools in Rust but not (yet) in Swift. Tests annotated
-# with these are expected to fail on Swift.
-# SWIFT_ONLY_TOOLS – tools in Swift but not (yet) registered in Rust. Tests
-# annotated with these are expected to fail on Rust.
-#
-# Update these lists as parity gaps close.
-
-REQUIRED_TOOLS = [
- # Core enumeration
- "get_screen_size",
- "screenshot",
- "list_apps",
- "list_windows",
- "get_window_state",
- # Interaction
- "click",
- "double_click",
- "right_click",
- "drag",
- "type_text",
- "press_key",
- "hotkey",
- "scroll",
- "set_value",
- "zoom",
- # App lifecycle
- "launch_app",
- # Permissions
- "check_permissions",
- # Cursor overlay
- "move_cursor",
- "get_cursor_position",
- "set_agent_cursor_style",
- "get_agent_cursor_state",
- "set_agent_cursor_enabled",
- "set_agent_cursor_motion",
- # Config
- "get_config",
- "set_config",
- # Recording / replay
- "set_recording",
- "get_recording_state",
- "replay_trajectory",
-]
-
-# In Rust, not in Swift yet. Tests for these will fail on the Swift binary.
-RUST_ONLY_TOOLS = [
- "type_text_chars", # per-character delay typing; Swift uses type_text
- "get_accessibility_tree",# lightweight desktop AX snapshot (separate from get_window_state)
-]
-
-# In Swift, not registered in Rust yet. (`page` is now cross-platform in Rust.)
-SWIFT_ONLY_TOOLS: list[str] = []
-
-
-# ── parity mixin ───────────────────────────────────────────────────────────────
-
-class _ParityMixin:
- """All API surface tests. Concrete subclasses set `cls.binary`."""
-
- binary: str = "" # set by setUpClass
-
- # ── CLI: version / help ───────────────────────────────────────────────────
-
- def test_version_flag_exits_zero(self) -> None:
- """--version must exit 0 (even if the binary enters MCP mode silently)."""
- r = _run([self.binary, "--version"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
-
- def test_version_flag_outputs_version_number(self) -> None:
- """--version must print a semver string to stdout or stderr.
-
- Parity gap: Rust binary currently exits 0 with no output because
- '--version' is treated as an unknown bare flag and silently enters
- MCP mode. This test documents the requirement; it will fail until
- '--version' is handled explicitly in cli::parse_command.
- """
- r = _run([self.binary, "--version"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- self.assertRegex(
- r.stdout + r.stderr,
- r"\d+\.\d+\.\d+",
- "--version did not print a semver string (N.N.N)",
- )
-
- def test_help_flag_exits_zero(self) -> None:
- r = _run([self.binary, "--help"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- self.assertIn("cua-driver", r.stdout + r.stderr)
-
- # ── CLI: list-tools ───────────────────────────────────────────────────────
-
- def test_list_tools_exits_zero(self) -> None:
- r = _run([self.binary, "list-tools"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
-
- def test_list_tools_contains_required_tool_names(self) -> None:
- r = _run([self.binary, "list-tools"])
- self.assertEqual(r.returncode, 0)
- for name in [
- "click", "screenshot", "list_windows", "press_key",
- "get_window_state", "type_text", "hotkey", "scroll",
- ]:
- self.assertIn(name, r.stdout, f"list-tools missing: {name!r}")
-
- def test_list_tools_every_required_tool_present(self) -> None:
- r = _run([self.binary, "list-tools"])
- self.assertEqual(r.returncode, 0)
- for name in REQUIRED_TOOLS:
- self.assertIn(name, r.stdout, f"list-tools missing: {name!r}")
-
- # ── CLI: describe ─────────────────────────────────────────────────────────
-
- def test_describe_click_has_pid_and_schema(self) -> None:
- r = _run([self.binary, "describe", "click"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- self.assertIn("click", r.stdout)
- self.assertIn("pid", r.stdout)
- self.assertIn("input_schema", r.stdout)
-
- def test_describe_screenshot_has_schema(self) -> None:
- r = _run([self.binary, "describe", "screenshot"])
- self.assertEqual(r.returncode, 0)
- self.assertIn("screenshot", r.stdout)
- self.assertIn("input_schema", r.stdout)
-
- def test_describe_each_required_tool_exits_zero(self) -> None:
- for name in REQUIRED_TOOLS:
- with self.subTest(tool=name):
- r = _run([self.binary, "describe", name])
- self.assertEqual(
- r.returncode, 0,
- f"describe {name!r} exited {r.returncode}: {r.stderr}",
- )
- self.assertIn(name, r.stdout)
- self.assertIn("input_schema", r.stdout)
-
- def test_describe_unknown_tool_exits_64(self) -> None:
- r = _run([self.binary, "describe", "no_such_tool_xyzzy"])
- self.assertEqual(r.returncode, 64)
- self.assertIn("no_such_tool_xyzzy", r.stderr)
-
- # ── CLI: call (stateless tools) ───────────────────────────────────────────
-
- def test_call_get_screen_size_exits_zero(self) -> None:
- r = _run([self.binary, "call", "get_screen_size"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
-
- def test_call_get_screen_size_returns_json(self) -> None:
- """call get_screen_size must emit JSON with width and height.
-
- Parity gap: Swift binary outputs human-readable text
- (e.g. "✅ Main display: 1920x1080 points @ 1.0x") instead of JSON.
- This test documents the requirement; it fails on Swift.
- """
- r = _run([self.binary, "call", "get_screen_size"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- try:
- data = json.loads(r.stdout)
- except json.JSONDecodeError:
- self.fail(
- f"call get_screen_size did not output JSON.\n"
- f"stdout: {r.stdout!r}\n"
- f"(Swift binary outputs human-readable text instead of JSON — parity gap)"
- )
- self.assertIn("width", data)
- self.assertIn("height", data)
- self.assertGreater(data["width"], 0)
- self.assertGreater(data["height"], 0)
-
- def test_call_screenshot_returns_b64_png(self) -> None:
- """call screenshot (no args) must return JSON with screenshot_png_b64.
-
- Parity gap: Swift binary requires `window_id` for screenshot and
- returns a non-JSON error when called without args. The Rust binary
- defaults to a full-display screenshot without requiring window_id.
- This test fails on Swift on both counts; it documents the requirement.
- """
- r = _run([self.binary, "call", "screenshot"], timeout=30)
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}\nstdout: {r.stdout!r}")
- try:
- data = json.loads(r.stdout)
- except json.JSONDecodeError:
- self.fail(
- f"call screenshot did not output JSON.\n"
- f"stdout: {r.stdout!r}\n"
- f"(Swift requires window_id; Rust defaults to full display)"
- )
- self.assertIn("screenshot_png_b64", data)
- self.assertGreater(len(data["screenshot_png_b64"]), 500)
-
- def test_call_list_apps_returns_app_list(self) -> None:
- r = _run([self.binary, "call", "list_apps"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- data = json.loads(r.stdout)
- self.assertIn("apps", data)
- self.assertIsInstance(data["apps"], list)
- self.assertGreater(len(data["apps"]), 0)
-
- def test_call_list_apps_entries_have_required_fields(self) -> None:
- r = _run([self.binary, "call", "list_apps"])
- self.assertEqual(r.returncode, 0)
- data = json.loads(r.stdout)
- for app in data["apps"][:5]: # spot-check first 5
- with self.subTest(app=app.get("name") or app.get("bundle_id")):
- self.assertIn("pid", app)
- self.assertIn("name", app)
-
- def test_call_list_windows_returns_window_list(self) -> None:
- r = _run([self.binary, "call", "list_windows"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- data = json.loads(r.stdout)
- self.assertIn("windows", data)
- self.assertIsInstance(data["windows"], list)
-
- def test_call_list_windows_entries_have_required_fields(self) -> None:
- r = _run([self.binary, "call", "list_windows"])
- self.assertEqual(r.returncode, 0)
- data = json.loads(r.stdout)
- for w in data["windows"][:5]:
- with self.subTest(window_id=w.get("window_id")):
- self.assertIn("window_id", w)
- self.assertIn("pid", w)
- self.assertIn("bounds", w)
- b = w["bounds"]
- for field in ("x", "y", "width", "height"):
- self.assertIn(field, b)
-
- def test_call_check_permissions_exits_zero(self) -> None:
- r = _run([self.binary, "call", "check_permissions"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
-
- def test_call_check_permissions_returns_accessibility_and_screen_recording(
- self,
- ) -> None:
- """call check_permissions must emit JSON with accessibility/screen_recording.
-
- Parity gap: Swift binary outputs human-readable text with emoji
- (✅/❌) instead of JSON. This test documents the JSON requirement
- and will fail on Swift until its CLI output is standardised.
- """
- r = _run([self.binary, "call", "check_permissions"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- try:
- data = json.loads(r.stdout)
- except json.JSONDecodeError:
- self.fail(
- f"call check_permissions did not output JSON.\n"
- f"stdout: {r.stdout!r}"
- )
- self.assertIn("accessibility", data)
- self.assertIn("screen_recording", data)
-
- def test_call_get_cursor_position_returns_xy(self) -> None:
- r = _run([self.binary, "call", "get_cursor_position"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- data = json.loads(r.stdout)
- self.assertIn("x", data)
- self.assertIn("y", data)
-
- def test_call_get_config_returns_config_object(self) -> None:
- r = _run([self.binary, "call", "get_config"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- data = json.loads(r.stdout)
- # Config must have at least one of the known top-level keys.
- known_keys = {
- "schema_version", "capture_mode", "max_image_dimension",
- "agent_cursor", "telemetry_enabled",
- }
- self.assertTrue(
- known_keys.intersection(data.keys()),
- f"get_config returned no known keys: {list(data.keys())}",
- )
-
- def test_call_get_agent_cursor_state_returns_state(self) -> None:
- r = _run([self.binary, "call", "get_agent_cursor_state"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- data = json.loads(r.stdout)
- self.assertIn("enabled", data)
-
- def test_call_get_recording_state_returns_state(self) -> None:
- r = _run([self.binary, "call", "get_recording_state"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- data = json.loads(r.stdout)
- self.assertIn("recording", data)
-
- # ── CLI: call with a pid-bearing tool (launch_app + get_window_state) ─────
-
- def test_call_launch_app_calculator_returns_pid(self) -> None:
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
- time.sleep(0.3)
- r = _run(
- [self.binary, "call", "launch_app", '{"bundle_id": "com.apple.calculator"}'],
- timeout=20,
- )
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- data = json.loads(r.stdout)
- self.assertIn("pid", data)
- self.assertGreater(data["pid"], 0)
-
- def test_call_get_window_state_for_calculator(self) -> None:
- """launch Calculator, get its window, then fetch get_window_state."""
- # Launch
- rl = _run(
- [self.binary, "call", "launch_app", '{"bundle_id": "com.apple.calculator"}'],
- timeout=20,
- )
- self.assertEqual(rl.returncode, 0, f"launch stderr: {rl.stderr}")
- pid = json.loads(rl.stdout)["pid"]
- time.sleep(1.0)
-
- # List windows to get a window_id.
- rw = _run(
- [self.binary, "call", "list_windows", f'{{"pid": {pid}}}'],
- )
- self.assertEqual(rw.returncode, 0, f"list_windows stderr: {rw.stderr}")
- windows = json.loads(rw.stdout)["windows"]
- self.assertTrue(windows, "Calculator has no windows")
- window_id = windows[0]["window_id"]
-
- # get_window_state
- rs = _run(
- [self.binary, "call", "get_window_state",
- f'{{"pid": {pid}, "window_id": {window_id}}}'],
- timeout=30,
- )
- self.assertEqual(rs.returncode, 0, f"get_window_state stderr: {rs.stderr}")
- data = json.loads(rs.stdout)
- self.assertIn("tree_markdown", data)
- self.assertIn("AX", data["tree_markdown"])
-
- # ── CLI: call — error / missing args paths ────────────────────────────────
-
- def test_call_unknown_tool_exits_64(self) -> None:
- r = _run([self.binary, "call", "no_such_tool_xyzzy"])
- self.assertEqual(r.returncode, 64)
-
- def test_call_press_key_missing_pid_exits_1(self) -> None:
- r = _run([self.binary, "call", "press_key", '{"key": "a"}'])
- self.assertEqual(r.returncode, 1, f"stdout: {r.stdout}")
-
- def test_call_click_missing_required_args_exits_1(self) -> None:
- r = _run([self.binary, "call", "click", '{}'])
- self.assertEqual(r.returncode, 1, f"stdout: {r.stdout}")
-
- def test_call_type_text_missing_pid_exits_1(self) -> None:
- r = _run([self.binary, "call", "type_text", '{"text": "hello"}'])
- self.assertEqual(r.returncode, 1, f"stdout: {r.stdout}")
-
- def test_call_hotkey_missing_pid_exits_1(self) -> None:
- r = _run([self.binary, "call", "hotkey", '{"keys": ["cmd", "a"]}'])
- self.assertEqual(r.returncode, 1, f"stdout: {r.stdout}")
-
- def test_call_scroll_missing_required_args_exits_1(self) -> None:
- r = _run([self.binary, "call", "scroll", '{}'])
- self.assertEqual(r.returncode, 1, f"stdout: {r.stdout}")
-
- def test_call_drag_missing_required_args_exits_1(self) -> None:
- r = _run([self.binary, "call", "drag", '{}'])
- self.assertEqual(r.returncode, 1, f"stdout: {r.stdout}")
-
- def test_call_get_window_state_missing_pid_exits_1(self) -> None:
- r = _run([self.binary, "call", "get_window_state", '{"window_id": 1}'])
- self.assertEqual(r.returncode, 1, f"stdout: {r.stdout}")
-
- def test_call_launch_app_no_args_exits_1(self) -> None:
- r = _run([self.binary, "call", "launch_app", '{}'])
- self.assertEqual(r.returncode, 1, f"stdout: {r.stdout}")
-
- # ── CLI: implicit call (tool name as first positional) ────────────────────
-
- def test_implicit_call_get_screen_size(self) -> None:
- r = _run([self.binary, "get_screen_size"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- data = json.loads(r.stdout)
- self.assertIn("width", data)
- self.assertIn("height", data)
-
- def test_implicit_call_check_permissions(self) -> None:
- r = _run([self.binary, "check_permissions"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- data = json.loads(r.stdout)
- self.assertIn("accessibility", data)
-
- def test_implicit_call_list_apps(self) -> None:
- r = _run([self.binary, "list_apps"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- data = json.loads(r.stdout)
- self.assertIn("apps", data)
-
- def test_implicit_call_screenshot_exits_zero(self) -> None:
- """Implicit-call screenshot must exit 0.
-
- Note: Swift binary requires window_id so exits non-zero with an
- error message — this is a parity gap in Swift.
- """
- r = _run([self.binary, "screenshot"], timeout=30)
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}\nstdout: {r.stdout!r}")
-
- def test_implicit_unknown_exits_64(self) -> None:
- r = _run([self.binary, "no_such_tool_xyzzy"])
- self.assertEqual(r.returncode, 64)
-
- # ── CLI: mcp-config ───────────────────────────────────────────────────────
-
- def test_mcp_config_default_outputs_json(self) -> None:
- r = _run([self.binary, "mcp-config"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- out = r.stdout.strip()
- # Should either be a JSON object or a shell snippet containing "cua-driver"
- self.assertIn("cua-driver", out)
-
- def test_mcp_config_client_claude_outputs_add_command(self) -> None:
- r = _run([self.binary, "mcp-config", "--client", "claude"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- self.assertIn("cua-driver", r.stdout)
-
- def test_mcp_config_client_cursor_exits_zero(self) -> None:
- r = _run([self.binary, "mcp-config", "--client", "cursor"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
-
- # ── CLI: dump-docs ────────────────────────────────────────────────────────
-
- def test_dump_docs_exits_zero(self) -> None:
- r = _run([self.binary, "dump-docs"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- self.assertTrue(len(r.stdout) > 200, "dump-docs output too short")
-
- def test_dump_docs_contains_tool_names(self) -> None:
- r = _run([self.binary, "dump-docs"])
- self.assertEqual(r.returncode, 0)
- for name in ["click", "screenshot", "get_window_state"]:
- self.assertIn(name, r.stdout)
-
- def test_dump_docs_pretty_is_valid_json(self) -> None:
- r = _run([self.binary, "dump-docs", "--pretty"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- # --pretty outputs JSON
- try:
- data = json.loads(r.stdout)
- except json.JSONDecodeError:
- self.fail(f"dump-docs --pretty is not valid JSON:\n{r.stdout[:500]}")
- self.assertIsInstance(data, (list, dict))
-
- # ── CLI: diagnose ─────────────────────────────────────────────────────────
-
- def test_diagnose_exits_zero(self) -> None:
- r = _run([self.binary, "diagnose"], timeout=30)
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- self.assertTrue(len(r.stdout) > 50)
-
- def test_diagnose_mentions_accessibility(self) -> None:
- r = _run([self.binary, "diagnose"], timeout=30)
- self.assertEqual(r.returncode, 0)
- out = (r.stdout + r.stderr).lower()
- self.assertIn("access", out)
-
- # ── CLI: doctor ───────────────────────────────────────────────────────────
-
- def test_doctor_exits_zero_or_one(self) -> None:
- """doctor returns 0 if all checks pass, 1 if any check fails."""
- r = _run([self.binary, "doctor"], timeout=20)
- self.assertIn(r.returncode, (0, 1), f"unexpected exit: {r.returncode}")
-
- def test_doctor_produces_output(self) -> None:
- r = _run([self.binary, "doctor"], timeout=20)
- self.assertIn(r.returncode, (0, 1))
- out = r.stdout + r.stderr
- self.assertGreater(len(out.strip()), 0)
-
- # ── CLI: config ───────────────────────────────────────────────────────────
-
- def test_config_show_exits_zero(self) -> None:
- r = _run([self.binary, "config", "show"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- self.assertGreater(len(r.stdout.strip()), 0)
-
- def test_config_get_max_image_dimension(self) -> None:
- r = _run([self.binary, "config", "get", "max_image_dimension"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- # Value should look like an integer
- out = r.stdout.strip()
- self.assertRegex(out, r"\d+", f"expected integer, got: {out!r}")
-
- def test_config_get_agent_cursor_enabled(self) -> None:
- r = _run([self.binary, "config", "get", "agent_cursor.enabled"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- val = r.stdout.strip().lower()
- self.assertIn(val, ("true", "false"), f"unexpected value: {val!r}")
-
- def test_config_set_and_restore_max_image_dimension(self) -> None:
- # Read original value.
- r0 = _run([self.binary, "config", "get", "max_image_dimension"])
- self.assertEqual(r0.returncode, 0)
- original = r0.stdout.strip()
-
- # Set a new value.
- r1 = _run([self.binary, "config", "set", "max_image_dimension", "1024"])
- self.assertEqual(r1.returncode, 0, f"set stderr: {r1.stderr}")
-
- # Confirm it was applied.
- r2 = _run([self.binary, "config", "get", "max_image_dimension"])
- self.assertEqual(r2.returncode, 0)
- self.assertEqual(r2.stdout.strip(), "1024")
-
- # Restore original value.
- _run([self.binary, "config", "set", "max_image_dimension", original])
-
- def test_config_reset_exits_zero(self) -> None:
- r = _run([self.binary, "config", "reset"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
-
- def test_config_get_unknown_key_exits_nonzero(self) -> None:
- r = _run([self.binary, "config", "get", "no_such_key_xyzzy"])
- self.assertNotEqual(r.returncode, 0)
-
- # ── CLI: update ───────────────────────────────────────────────────────────
-
- def test_update_dry_run_exits_zero(self) -> None:
- """update without --apply should exit 0 even if already up-to-date."""
- r = _run([self.binary, "update"], timeout=30)
- self.assertIn(r.returncode, (0, 1), f"unexpected exit: {r.returncode}")
- out = r.stdout + r.stderr
- # Should mention version or "up to date" or similar
- self.assertGreater(len(out.strip()), 0)
-
- # ── CLI: serve / stop / status daemon lifecycle ───────────────────────────
-
- def _tmp_socket(self) -> str:
- return tempfile.mktemp(suffix=".sock", prefix="cua-parity-test-")
-
- def test_status_exits_1_when_no_daemon(self) -> None:
- sock = self._tmp_socket()
- r = _run([self.binary, "status", "--socket", sock])
- self.assertEqual(r.returncode, 1)
- out = (r.stdout + r.stderr).lower()
- self.assertIn("not running", out)
-
- def test_stop_exits_1_when_no_daemon(self) -> None:
- sock = self._tmp_socket()
- r = _run([self.binary, "stop", "--socket", sock])
- self.assertEqual(r.returncode, 1)
-
- def test_serve_status_stop_lifecycle(self) -> None:
- sock = self._tmp_socket()
- proc = subprocess.Popen(
- [self.binary, "serve", "--socket", sock],
- stdout=subprocess.PIPE,
- stderr=subprocess.PIPE,
- )
- try:
- # Wait for daemon to bind (up to 5 s).
- deadline = time.monotonic() + 5.0
- running = False
- while time.monotonic() < deadline:
- r = _run([self.binary, "status", "--socket", sock])
- if r.returncode == 0:
- running = True
- break
- time.sleep(0.1)
- self.assertTrue(running, "daemon did not become ready in 5 s")
-
- # status reports "running" and mentions the socket path.
- r = _run([self.binary, "status", "--socket", sock])
- self.assertEqual(r.returncode, 0, f"status stderr: {r.stderr}")
- out = (r.stdout + r.stderr).lower()
- self.assertIn("running", out)
-
- # stop the daemon.
- r = _run([self.binary, "stop", "--socket", sock])
- self.assertEqual(r.returncode, 0, f"stop stderr: {r.stderr}")
-
- # status should report not running.
- time.sleep(0.3)
- r = _run([self.binary, "status", "--socket", sock])
- self.assertEqual(r.returncode, 1)
- finally:
- proc.terminate()
- proc.wait(timeout=3)
-
- def test_serve_second_instance_exits_1(self) -> None:
- sock = self._tmp_socket()
- proc = subprocess.Popen(
- [self.binary, "serve", "--socket", sock],
- stdout=subprocess.PIPE,
- stderr=subprocess.PIPE,
- )
- try:
- # Wait for first daemon.
- deadline = time.monotonic() + 5.0
- while time.monotonic() < deadline:
- r = _run([self.binary, "status", "--socket", sock])
- if r.returncode == 0:
- break
- time.sleep(0.1)
-
- # Second serve should fail.
- r = _run([self.binary, "serve", "--socket", sock])
- self.assertEqual(r.returncode, 1, f"stdout: {r.stdout}\nstderr: {r.stderr}")
- out = (r.stdout + r.stderr).lower()
- self.assertIn("already running", out)
- finally:
- _run([self.binary, "stop", "--socket", sock])
- proc.wait(timeout=3)
-
- # ── stdio MCP: tools/list ─────────────────────────────────────────────────
-
- def _mcp(self) -> DriverClient:
- return DriverClient(self.binary)
-
- def test_mcp_tools_list_returns_all_required_tools(self) -> None:
- with self._mcp() as c:
- tools = c.list_tools()
- names = {t["name"] for t in tools}
- missing = [n for n in REQUIRED_TOOLS if n not in names]
- self.assertFalse(
- missing,
- f"tools/list missing: {missing}\n got: {sorted(names)}",
- )
-
- def test_mcp_tools_list_each_has_name_description_schema(self) -> None:
- with self._mcp() as c:
- tools = c.list_tools()
- for t in tools:
- with self.subTest(tool=t.get("name")):
- self.assertIn("name", t)
- self.assertIn("description", t)
- self.assertIn("inputSchema", t)
- schema = t["inputSchema"]
- self.assertIn("type", schema)
- self.assertEqual(schema["type"], "object")
-
- # ── stdio MCP: stateless read tools ──────────────────────────────────────
-
- def test_mcp_get_screen_size(self) -> None:
- with self._mcp() as c:
- result = c.call_tool("get_screen_size")
- sc = result.get("structuredContent", result)
- self.assertIn("width", sc)
- self.assertIn("height", sc)
- self.assertGreater(sc["width"], 0)
- self.assertGreater(sc["height"], 0)
-
- def test_mcp_screenshot_has_png_image(self) -> None:
- with self._mcp() as c:
- result = c.call_tool("screenshot")
- # Should have image content.
- images = [i for i in result.get("content", []) if i.get("type") == "image"]
- sc = result.get("structuredContent", {})
- # At least one of: image in content OR screenshot_png_b64 in structuredContent.
- has_image = bool(images) or bool(sc.get("screenshot_png_b64"))
- self.assertTrue(has_image, f"screenshot returned no image: {result}")
-
- def test_mcp_screenshot_with_quality(self) -> None:
- with self._mcp() as c:
- result = c.call_tool("screenshot", {"quality": 50})
- images = [i for i in result.get("content", []) if i.get("type") == "image"]
- sc = result.get("structuredContent", {})
- has_image = bool(images) or bool(sc.get("screenshot_png_b64"))
- self.assertTrue(has_image, "screenshot with quality returned no image")
-
- def test_mcp_list_apps(self) -> None:
- with self._mcp() as c:
- result = c.call_tool("list_apps")
- sc = result["structuredContent"]
- self.assertIn("apps", sc)
- apps = sc["apps"]
- self.assertIsInstance(apps, list)
- self.assertGreater(len(apps), 0)
- # Every app entry must have pid + name.
- for app in apps[:5]:
- self.assertIn("pid", app)
- self.assertIn("name", app)
-
- def test_mcp_list_apps_has_active_flag(self) -> None:
- with self._mcp() as c:
- result = c.call_tool("list_apps")
- apps = result["structuredContent"]["apps"]
- # At least one app should have an active field.
- has_active = any("active" in app for app in apps)
- self.assertTrue(has_active, "no app has 'active' field")
-
- def test_mcp_list_windows_default(self) -> None:
- with self._mcp() as c:
- result = c.call_tool("list_windows")
- sc = result["structuredContent"]
- self.assertIn("windows", sc)
- windows = sc["windows"]
- self.assertIsInstance(windows, list)
-
- def test_mcp_list_windows_fields(self) -> None:
- with self._mcp() as c:
- result = c.call_tool("list_windows")
- windows = result["structuredContent"]["windows"]
- for w in windows[:5]:
- with self.subTest(window_id=w.get("window_id")):
- self.assertIn("window_id", w)
- self.assertIn("pid", w)
- self.assertIn("bounds", w)
- for f in ("x", "y", "width", "height"):
- self.assertIn(f, w["bounds"])
-
- def test_mcp_list_windows_pid_filter(self) -> None:
- """list_windows with pid filter should only return windows for that pid."""
- with self._mcp() as c:
- # Get any pid that has windows.
- all_wins = c.call_tool("list_windows")["structuredContent"]["windows"]
- if not all_wins:
- self.skipTest("no windows available")
- target_pid = all_wins[0]["pid"]
- filtered = c.call_tool("list_windows", {"pid": target_pid})
- windows = filtered["structuredContent"]["windows"]
- for w in windows:
- self.assertEqual(w["pid"], target_pid)
-
- def test_mcp_list_windows_on_screen_only(self) -> None:
- with self._mcp() as c:
- result = c.call_tool("list_windows", {"on_screen_only": True})
- windows = result["structuredContent"]["windows"]
- for w in windows:
- self.assertTrue(
- w.get("is_on_screen", True),
- f"on_screen_only=true returned off-screen window: {w}",
- )
-
- def test_mcp_check_permissions(self) -> None:
- with self._mcp() as c:
- result = c.call_tool("check_permissions")
- sc = result.get("structuredContent", result)
- self.assertIn("accessibility", sc)
- self.assertIn("screen_recording", sc)
- self.assertIsInstance(sc["accessibility"], bool)
- self.assertIsInstance(sc["screen_recording"], bool)
-
- def test_mcp_get_cursor_position(self) -> None:
- with self._mcp() as c:
- result = c.call_tool("get_cursor_position")
- sc = result.get("structuredContent", result)
- self.assertIn("x", sc)
- self.assertIn("y", sc)
-
- def test_mcp_get_agent_cursor_state(self) -> None:
- with self._mcp() as c:
- result = c.call_tool("get_agent_cursor_state")
- sc = result.get("structuredContent", result)
- self.assertIn("enabled", sc)
-
- def test_mcp_set_agent_cursor_enabled_toggle(self) -> None:
- with self._mcp() as c:
- # Read current.
- state = c.call_tool("get_agent_cursor_state")
- enabled = state.get("structuredContent", state).get("enabled", True)
- # Toggle off.
- c.call_tool("set_agent_cursor_enabled", {"enabled": False})
- state2 = c.call_tool("get_agent_cursor_state")
- self.assertFalse(state2.get("structuredContent", state2).get("enabled", True))
- # Restore.
- c.call_tool("set_agent_cursor_enabled", {"enabled": enabled})
-
- def test_mcp_set_agent_cursor_style_default(self) -> None:
- with self._mcp() as c:
- # Calling with no args or default args should succeed.
- c.call_tool("set_agent_cursor_style", {})
-
- def test_mcp_get_config_roundtrip(self) -> None:
- with self._mcp() as c:
- result = c.call_tool("get_config")
- sc = result.get("structuredContent", result)
- known = {"schema_version", "capture_mode", "max_image_dimension",
- "agent_cursor", "telemetry_enabled"}
- self.assertTrue(known.intersection(sc.keys()),
- f"get_config has no known keys: {list(sc.keys())}")
-
- def test_mcp_set_config_max_image_dimension(self) -> None:
- with self._mcp() as c:
- # Read original.
- orig = c.call_tool("get_config")
- orig_dim = orig.get("structuredContent", orig).get("max_image_dimension", 1920)
- # Set new value.
- c.call_tool("set_config", {"max_image_dimension": 800})
- # Confirm.
- updated = c.call_tool("get_config")
- new_dim = updated.get("structuredContent", updated).get("max_image_dimension")
- self.assertEqual(new_dim, 800)
- # Restore.
- c.call_tool("set_config", {"max_image_dimension": orig_dim})
-
- def test_mcp_get_recording_state(self) -> None:
- with self._mcp() as c:
- result = c.call_tool("get_recording_state")
- sc = result.get("structuredContent", result)
- self.assertIn("recording", sc)
- self.assertIsInstance(sc["recording"], bool)
-
- def test_mcp_set_recording_enable_disable(self) -> None:
- import tempfile
- with tempfile.TemporaryDirectory() as tmpdir:
- with self._mcp() as c:
- # Enable recording — output_dir is required.
- c.call_tool("set_recording", {"enabled": True, "output_dir": tmpdir})
- state = c.call_tool("get_recording_state")
- sc = state.get("structuredContent", state)
- # Accept either "recording" (Swift) or "enabled" (Rust) key.
- is_recording = sc.get("recording", sc.get("enabled", False))
- self.assertTrue(is_recording, f"recording should be enabled: {sc}")
- # Disable.
- c.call_tool("set_recording", {"enabled": False})
- state2 = c.call_tool("get_recording_state")
- sc2 = state2.get("structuredContent", state2)
- is_recording2 = sc2.get("recording", sc2.get("enabled", True))
- self.assertFalse(is_recording2, f"recording should be disabled: {sc2}")
-
- # ── stdio MCP: tool error contracts ──────────────────────────────────────
-
- def _assert_tool_raises_mcp_error(
- self, tool_name: str, args: dict, timeout: float = 10.0
- ) -> None:
- """Assert that calling `tool_name` with `args` signals an error.
-
- Accepts either a JSON-RPC MCPCallError (Swift behaviour) or a result
- with ``isError=true`` in the tool-content list (Rust behaviour).
- """
- with self._mcp() as c:
- try:
- result = c.call_tool(tool_name, args)
- # Rust returns isError:true in the content payload
- self.assertTrue(
- result.get("isError"),
- f"{tool_name} did not raise an error (isError missing or false)",
- )
- except MCPCallError:
- pass # Swift-style JSON-RPC error — acceptable
-
- def test_mcp_click_missing_pid_raises_error(self) -> None:
- self._assert_tool_raises_mcp_error("click", {"window_id": 1})
-
- def test_mcp_double_click_missing_pid_raises_error(self) -> None:
- self._assert_tool_raises_mcp_error("double_click", {"window_id": 1})
-
- def test_mcp_right_click_missing_pid_raises_error(self) -> None:
- self._assert_tool_raises_mcp_error("right_click", {"window_id": 1})
-
- def test_mcp_press_key_missing_pid_raises_error(self) -> None:
- self._assert_tool_raises_mcp_error("press_key", {"key": "a"})
-
- def test_mcp_hotkey_missing_pid_raises_error(self) -> None:
- self._assert_tool_raises_mcp_error("hotkey", {"keys": ["cmd", "a"]})
-
- def test_mcp_type_text_missing_pid_raises_error(self) -> None:
- self._assert_tool_raises_mcp_error("type_text", {"text": "hello"})
-
- def test_mcp_type_text_chars_missing_pid_raises_error(self) -> None:
- self._assert_tool_raises_mcp_error("type_text_chars", {"text": "hello"})
-
- def test_mcp_scroll_missing_pid_raises_error(self) -> None:
- self._assert_tool_raises_mcp_error("scroll", {"x": 100, "y": 100})
-
- def test_mcp_drag_missing_pid_raises_error(self) -> None:
- self._assert_tool_raises_mcp_error("drag", {})
-
- def test_mcp_get_window_state_missing_pid_raises_error(self) -> None:
- self._assert_tool_raises_mcp_error("get_window_state", {"window_id": 1})
-
- def test_mcp_launch_app_missing_args_raises_error(self) -> None:
- self._assert_tool_raises_mcp_error("launch_app", {})
-
- def test_mcp_set_value_missing_pid_raises_error(self) -> None:
- self._assert_tool_raises_mcp_error(
- "set_value", {"element_index": 0, "value": "x"}
- )
-
- def test_mcp_replay_trajectory_bad_dir_raises_error(self) -> None:
- self._assert_tool_raises_mcp_error(
- "replay_trajectory", {"dir": "/tmp/no_such_trajectory_xyzzy"}
- )
-
- # ── stdio MCP: get_window_state live ─────────────────────────────────────
-
- def test_mcp_get_window_state_calculator(self) -> None:
- """Launch Calculator, capture a window state, verify tree_markdown."""
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
- time.sleep(0.3)
-
- with self._mcp() as c:
- # Launch
- launch = c.call_tool("launch_app", {"bundle_id": "com.apple.calculator"})
- pid = launch.get("structuredContent", launch).get("pid", 0)
- self.assertGreater(pid, 0, f"launch_app returned bad pid: {launch}")
- time.sleep(1.0)
-
- # Find window
- wins = c.call_tool("list_windows", {"pid": pid})["structuredContent"]["windows"]
- self.assertTrue(wins, "Calculator has no windows")
- window_id = wins[0]["window_id"]
-
- # get_window_state
- state = c.call_tool("get_window_state", {"pid": pid, "window_id": window_id})
-
- sc = state.get("structuredContent", state)
- tree = sc.get("tree_markdown", "")
- self.assertIn("AX", tree, f"tree_markdown has no AX nodes:\n{tree}")
- # Screenshot should be present.
- images = [i for i in state.get("content", []) if i.get("type") == "image"]
- self.assertTrue(images, "get_window_state returned no screenshot")
-
- def test_mcp_get_window_state_has_required_sc_keys(self) -> None:
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
- time.sleep(0.3)
- with self._mcp() as c:
- launch = c.call_tool("launch_app", {"bundle_id": "com.apple.calculator"})
- pid = launch.get("structuredContent", launch).get("pid", 0)
- time.sleep(1.0)
- wins = c.call_tool("list_windows", {"pid": pid})["structuredContent"]["windows"]
- if not wins:
- self.skipTest("Calculator has no windows")
- state = c.call_tool("get_window_state", {
- "pid": pid, "window_id": wins[0]["window_id"]
- })
- sc = state.get("structuredContent", state)
- for key in ("tree_markdown", "screenshot_width", "screenshot_height"):
- self.assertIn(key, sc, f"get_window_state structuredContent missing: {key!r}")
-
- # ── stdio MCP: launch_app ─────────────────────────────────────────────────
-
- def test_mcp_launch_app_by_bundle_id(self) -> None:
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
- time.sleep(0.3)
- with self._mcp() as c:
- result = c.call_tool("launch_app", {"bundle_id": "com.apple.calculator"})
- sc = result.get("structuredContent", result)
- self.assertIn("pid", sc)
- self.assertGreater(sc["pid"], 0)
-
- def test_mcp_launch_app_by_name(self) -> None:
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
- time.sleep(0.3)
- with self._mcp() as c:
- result = c.call_tool("launch_app", {"name": "Calculator"})
- sc = result.get("structuredContent", result)
- self.assertIn("pid", sc)
- self.assertGreater(sc["pid"], 0)
-
- def test_mcp_launch_app_unknown_bundle_id_raises_error(self) -> None:
- self._assert_tool_raises_mcp_error(
- "launch_app", {"bundle_id": "com.example.no_such_app_xyzzy"}
- )
-
- # ── stdio MCP: set_agent_cursor_motion ────────────────────────────────────
-
- def test_mcp_set_agent_cursor_motion_spring(self) -> None:
- with self._mcp() as c:
- c.call_tool("set_agent_cursor_motion", {"spring": True})
-
- def test_mcp_set_agent_cursor_motion_arc(self) -> None:
- with self._mcp() as c:
- c.call_tool("set_agent_cursor_motion", {"arc_size": 0.5})
-
- # ── stdio MCP: zoom ───────────────────────────────────────────────────────
-
- def test_mcp_zoom_missing_pid_raises_error(self) -> None:
- self._assert_tool_raises_mcp_error(
- "zoom", {"x1": 0, "y1": 0, "x2": 100, "y2": 100}
- )
-
- # ── stdio MCP: page ───────────────────────────────────────────────────────
-
- def test_mcp_page_is_registered(self) -> None:
- """'page' tool must appear in tools/list.
-
- Parity gap: Rust binary has page.rs implemented but the tool is not
- yet registered in platform_macos::tools::register_all. This test will
- fail on the Rust binary until page::PageTool is added to register_all.
- """
- with self._mcp() as c:
- names = {t["name"] for t in c.list_tools()}
- self.assertIn("page", names, "'page' tool is not registered")
-
- def test_mcp_page_missing_pid_raises_error(self) -> None:
- self._assert_tool_raises_mcp_error(
- "page", {"action": "get_text"}
- )
-
- # ── stdio MCP: Rust-only tools (missing in Swift) ────────────────────────
-
- def test_mcp_rust_only_tools_present(self) -> None:
- """Rust-only tools (type_text_chars, get_accessibility_tree).
-
- These are in the Rust binary but not yet in Swift. Both binaries
- should eventually have them. This test fails on Swift until ported.
- """
- with self._mcp() as c:
- names = {t["name"] for t in c.list_tools()}
- missing = [n for n in RUST_ONLY_TOOLS if n not in names]
- self.assertFalse(
- missing,
- f"Rust-only tools missing: {missing} (not yet in this binary)",
- )
-
- def test_mcp_type_text_chars_missing_pid_raises_error(self) -> None:
- """type_text_chars (Rust-only) must raise MCP error with missing pid.
-
- Fails on Swift binary — tool not yet ported.
- """
- self._assert_tool_raises_mcp_error("type_text_chars", {"text": "hello"})
-
- def test_mcp_get_accessibility_tree_exits_without_pid(self) -> None:
- """get_accessibility_tree (Rust-only) returns desktop snapshot without pid.
-
- Fails on Swift binary — tool not yet ported.
- """
- with self._mcp() as c:
- result = c.call_tool("get_accessibility_tree")
- sc = result.get("structuredContent", result)
- # Should return some content — desktop always has windows.
- self.assertIsInstance(sc, dict)
-
- # ── stdio MCP: Swift-only tools (missing/unregistered in Rust) ───────────
-
- def test_mcp_swift_only_tools_present(self) -> None:
- """Swift-only tools list — currently empty after the page-tool
- unification in Rust. Kept as a no-op so the matrix stays symmetric
- with `test_mcp_rust_only_tools_present`.
- """
- with self._mcp() as c:
- names = {t["name"] for t in c.list_tools()}
- missing = [n for n in SWIFT_ONLY_TOOLS if n not in names]
- self.assertFalse(
- missing,
- f"Swift-only tools missing: {missing} (not yet registered in this binary)",
- )
-
- # ── stdio MCP: protocol-level contracts ──────────────────────────────────
-
- def test_mcp_tools_call_unknown_tool_returns_error(self) -> None:
- """Calling an unknown tool must signal an error.
-
- Accepts MCPCallError (Swift / JSON-RPC error) or isError=true (Rust).
- """
- with self._mcp() as c:
- try:
- result = c.call_tool("no_such_tool_xyzzy")
- self.assertTrue(
- result.get("isError"),
- "unknown tool should set isError=true",
- )
- except MCPCallError:
- pass # JSON-RPC level error — acceptable
-
- def test_mcp_handles_rapid_sequential_calls(self) -> None:
- """Send 5 sequential get_screen_size calls — all must succeed."""
- with self._mcp() as c:
- for i in range(5):
- result = c.call_tool("get_screen_size")
- sc = result.get("structuredContent", result)
- self.assertIn("width", sc, f"call {i} failed: {result}")
-
- def test_mcp_protocol_version_echo(self) -> None:
- """The initialize response should echo an MCP protocol version."""
- # The DriverClient handshake already calls initialize; if it doesn't
- # raise an exception, the server replied correctly.
- with self._mcp() as c:
- tools = c.list_tools()
- self.assertGreater(len(tools), 0)
-
-
-# ── concrete test classes (one per binary) ────────────────────────────────────
-
-class SwiftParityTests(_ParityMixin, unittest.TestCase):
- """Run the full parity suite against the Swift cua-driver binary."""
-
- @classmethod
- def setUpClass(cls) -> None:
- path = _swift_binary()
- if not path:
- raise unittest.SkipTest(
- "Swift cua-driver not found. "
- "Set CUA_SWIFT_BINARY or install cua-driver to ~/.local/bin/."
- )
- cls.binary = path
- print(f"\n[SwiftParityTests] binary: {cls.binary}", flush=True)
-
-
-class RustParityTests(_ParityMixin, unittest.TestCase):
- """Run the full parity suite against the Rust cua-driver-rs binary."""
-
- @classmethod
- def setUpClass(cls) -> None:
- cls.binary = default_binary_path()
- if not os.path.isfile(cls.binary):
- raise unittest.SkipTest(
- f"Rust binary not found at {cls.binary}. "
- "Run `cargo build -p cua-driver` or set CUA_DRIVER_BINARY."
- )
- print(f"\n[RustParityTests] binary: {cls.binary}", flush=True)
-
-
-if __name__ == "__main__":
- unittest.main(verbosity=2)
diff --git a/libs/cua-driver/tests/e2e/test_background_focus.py b/libs/cua-driver/tests/e2e/test_background_focus.py
deleted file mode 100644
index b86e1fbe9a..0000000000
--- a/libs/cua-driver/tests/e2e/test_background_focus.py
+++ /dev/null
@@ -1,568 +0,0 @@
-"""Integration test: interact with backgrounded apps without stealing focus.
-
-Setup:
- 1. Open Safari to a local HTML page with a button and text input.
- 2. Launch FocusMonitorApp (counts NSApplication.didResignActiveNotification).
- 3. FocusMonitorApp is activated last so it owns focus.
-
-Tests send clicks and keystrokes to backgrounded Safari via the MCP driver,
-then assert:
- - Safari's page state changed (button counter incremented, text typed).
- - FocusMonitorApp's focus-loss counter stayed at 0 throughout.
-
-Also includes CalculatorBackgroundClickTests that use AX-click on Calculator
-while FocusMonitorApp is frontmost.
-
-Run:
- CUA_DRIVER_BINARY=../../target/debug/cua-driver python3 -m unittest test_background_focus -v
-"""
-
-from __future__ import annotations
-
-import os
-import re
-import subprocess
-import sys
-import time
-import unittest
-
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
-from driver_client import (
- DriverClient,
- default_binary_path,
- frontmost_bundle_id,
- resolve_window_id,
-)
-
-# ---------------------------------------------------------------------------
-# Paths
-# ---------------------------------------------------------------------------
-_THIS_DIR = os.path.dirname(os.path.abspath(__file__))
-_DRIVER_RS_ROOT = os.path.dirname(os.path.dirname(_THIS_DIR))
-_LIBS_ROOT = os.path.dirname(_DRIVER_RS_ROOT)
-_HTML_PAGE = os.path.join(_THIS_DIR, "fixtures", "interactive.html")
-
-# FocusMonitorApp lives in the Swift cua-driver tree (shared fixture).
-_FOCUS_APP_DIR = os.path.join(_LIBS_ROOT, "cua-driver", "Tests", "FocusMonitorApp")
-_FOCUS_APP_BUNDLE = os.path.join(_FOCUS_APP_DIR, "FocusMonitorApp.app")
-_FOCUS_APP_EXE = os.path.join(
- _FOCUS_APP_BUNDLE, "Contents", "MacOS", "FocusMonitorApp"
-)
-_LOSS_FILE = "/tmp/focus_monitor_losses.txt"
-
-SAFARI_BUNDLE = "com.apple.Safari"
-FOCUS_MONITOR_BUNDLE = "com.trycua.FocusMonitorApp"
-
-
-# ---------------------------------------------------------------------------
-# Helpers
-# ---------------------------------------------------------------------------
-
-def _build_focus_app() -> None:
- if not os.path.exists(_FOCUS_APP_EXE):
- subprocess.run(
- [os.path.join(_FOCUS_APP_DIR, "build.sh")], check=True
- )
-
-
-def _launch_focus_app() -> tuple[subprocess.Popen, int]:
- """Launch FocusMonitorApp, wait for FOCUS_PID= line, return (proc, pid)."""
- proc = subprocess.Popen(
- [_FOCUS_APP_EXE],
- stdout=subprocess.PIPE,
- stderr=subprocess.DEVNULL,
- text=True,
- )
- for _ in range(40):
- line = proc.stdout.readline().strip()
- if line.startswith("FOCUS_PID="):
- pid = int(line.split("=", 1)[1])
- return proc, pid
- time.sleep(0.1)
- proc.terminate()
- raise RuntimeError("FocusMonitorApp did not print FOCUS_PID in time")
-
-
-def _read_focus_losses() -> int:
- try:
- with open(_LOSS_FILE) as f:
- return int(f.read().strip())
- except (FileNotFoundError, ValueError):
- return -1
-
-
-def _open_safari_to_html(client: DriverClient) -> int:
- """Open the test HTML page in a fresh Safari window and return its pid.
-
- Kills any existing Safari first so we get a clean single-window state
- and `resolve_window_id` reliably picks the interactive.html window.
- """
- # Kill any existing Safari so we start clean with exactly one window.
- subprocess.run(["pkill", "-x", "Safari"], check=False)
- time.sleep(0.5)
-
- file_url = f"file://{_HTML_PAGE}"
- # Open Safari in background (-g) with the specific URL.
- subprocess.run(["open", "-g", "-a", "Safari", file_url], check=True)
- time.sleep(3.0) # let Safari launch and page load
-
- # Get the pid from list_apps now that Safari is running.
- result = client.call_tool("list_apps")
- apps = result["structuredContent"]["apps"]
- safari = next((a for a in apps if a.get("bundle_id") == SAFARI_BUNDLE), None)
- if safari is None:
- raise RuntimeError("Safari did not appear in list_apps after launch")
- return safari["pid"]
-
-
-def _get_page_text(client: DriverClient, pid: int) -> str:
- """Return the AX tree markdown for Safari."""
- window_id = resolve_window_id(client, pid)
- result = client.call_tool(
- "get_window_state", {"pid": pid, "window_id": window_id}
- )
- return result.get("structuredContent", result).get("tree_markdown", "")
-
-
-def _find_element_index(tree_markdown: str, label: str) -> int | None:
- """Extract the first [N] element index from a line containing `label`."""
- for line in tree_markdown.split("\n"):
- if label in line:
- m = re.search(r'\[(\d+)\]', line)
- if m:
- return int(m.group(1))
- return None
-
-
-def _find_calc_button(tree_markdown: str, label: str) -> int | None:
- """Find a Calculator button by description in parentheses or help text."""
- for line in tree_markdown.split("\n"):
- if "AXButton" not in line:
- continue
- m = re.search(r'\[(\d+)\]', line)
- if not m:
- continue
- idx = int(m.group(1))
- if f'({label})' in line:
- return idx
- if f'help="{label}' in line:
- return idx
- if f'id={label}' in line:
- return idx
- return None
-
-
-def _activate_focus_monitor() -> None:
- """Re-activate FocusMonitorApp via osascript."""
- subprocess.run(
- ["osascript", "-e", 'tell application "FocusMonitorApp" to activate'],
- check=False,
- )
- time.sleep(0.5)
-
-
-# ---------------------------------------------------------------------------
-# Safari background focus tests
-# ---------------------------------------------------------------------------
-
-class BackgroundFocusTests(unittest.TestCase):
- """Click & type into backgrounded Safari without stealing focus."""
-
- _safari_pid: int
- _focus_proc: subprocess.Popen
- _focus_pid: int
-
- @classmethod
- def setUpClass(cls) -> None:
- _build_focus_app()
-
- # Kill any stale FocusMonitorApp left over from a crashed prior run.
- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)
- time.sleep(0.3)
-
- try:
- os.remove(_LOSS_FILE)
- except FileNotFoundError:
- pass
-
- cls.binary = default_binary_path()
-
- # 1. Open Safari to the test page.
- with DriverClient(cls.binary) as c:
- cls._safari_pid = _open_safari_to_html(c)
- print(f"\n Safari pid: {cls._safari_pid}")
-
- # 2. Launch FocusMonitorApp (becomes frontmost).
- cls._focus_proc, cls._focus_pid = _launch_focus_app()
- print(f" FocusMonitor pid: {cls._focus_pid}")
- time.sleep(1.0)
-
- # Confirm FocusMonitorApp is frontmost.
- with DriverClient(cls.binary) as c:
- active = frontmost_bundle_id(c)
- assert active == FOCUS_MONITOR_BUNDLE, (
- f"Expected FocusMonitorApp frontmost, got {active}"
- )
-
- # Baseline: 0 focus losses.
- losses = _read_focus_losses()
- assert losses == 0, f"Expected 0 focus losses at start, got {losses}"
-
- @classmethod
- def tearDownClass(cls) -> None:
- if hasattr(cls, '_focus_proc'):
- cls._focus_proc.terminate()
- try:
- cls._focus_proc.wait(timeout=3)
- except subprocess.TimeoutExpired:
- cls._focus_proc.kill()
- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)
-
- subprocess.run(["pkill", "-x", "Safari"], check=False)
-
- try:
- os.remove(_LOSS_FILE)
- except FileNotFoundError:
- pass
-
- def setUp(self) -> None:
- _activate_focus_monitor()
- self._losses_before = _read_focus_losses()
-
- with DriverClient(self.binary) as c:
- active = frontmost_bundle_id(c)
- self.assertEqual(
- active, FOCUS_MONITOR_BUNDLE,
- f"FocusMonitorApp not frontmost at test start, got {active}",
- )
-
- def _assert_no_focus_loss(self, label: str) -> None:
- """Assert FocusMonitorApp is still frontmost (or restored) after interaction."""
- time.sleep(0.3)
- losses = _read_focus_losses()
- with DriverClient(self.binary) as c:
- active = frontmost_bundle_id(c)
- focus_restored = (active == FOCUS_MONITOR_BUNDLE)
- loss_delta = losses - self._losses_before
- print(f" [{label}] losses: {self._losses_before}->{losses} "
- f"(delta={loss_delta}), frontmost: {active}, "
- f"restored: {focus_restored}")
- self.assertEqual(
- active, FOCUS_MONITOR_BUNDLE,
- f"[{label}] Focus not restored — "
- f"frontmost is {active}, not FocusMonitorApp",
- )
-
- def test_01_ax_click_button(self) -> None:
- """AX-click the 'Click Me' button in backgrounded Safari."""
- with DriverClient(self.binary) as c:
- window_id = resolve_window_id(c, self._safari_pid)
- snap = c.call_tool("get_window_state", {
- "pid": self._safari_pid,
- "window_id": window_id,
- "query": "Click Me",
- })
- tree = snap.get("structuredContent", snap).get("tree_markdown", "")
- print(f"\n filtered tree:\n{tree}")
-
- idx = _find_element_index(tree, "Click Me")
- self.assertIsNotNone(idx, "Could not find 'Click Me' button in AX tree")
-
- result = c.call_tool("click", {
- "pid": self._safari_pid,
- "window_id": window_id,
- "element_index": idx,
- })
- print(f" click result: {result}")
-
- time.sleep(0.5)
-
- with DriverClient(self.binary) as c:
- tree = _get_page_text(c, self._safari_pid)
- self.assertIn("clicks: 1", tree, "Button click did not register on page")
-
- self._assert_no_focus_loss("01_ax_click_button")
-
- def test_02_type_text_chars(self) -> None:
- """Type into the text field in backgrounded Safari via keystroke synthesis."""
- with DriverClient(self.binary) as c:
- window_id = resolve_window_id(c, self._safari_pid)
- snap = c.call_tool("get_window_state", {
- "pid": self._safari_pid,
- "window_id": window_id,
- "query": "AXTextField",
- })
- tree = snap.get("structuredContent", snap).get("tree_markdown", "")
- print(f"\n filtered tree:\n{tree}")
-
- # Find the page's text field (not the Safari URL bar).
- idx = None
- for line in tree.split("\n"):
- if "AXTextField" in line and "smart search field" not in line:
- m = re.search(r'\[(\d+)\]', line)
- if m:
- idx = int(m.group(1))
- break
-
- if idx is None:
- snap = c.call_tool("get_window_state", {
- "pid": self._safari_pid, "window_id": window_id,
- })
- tree = snap.get("structuredContent", snap).get("tree_markdown", "")
- for line in tree.split("\n"):
- if "AXTextField" in line and "smart search field" not in line:
- m = re.search(r'\[(\d+)\]', line)
- if m:
- idx = int(m.group(1))
- break
-
- self.assertIsNotNone(idx, "Could not find page text field in AX tree")
- print(f" text field element_index: {idx}")
-
- # Focus the text field via AX click first.
- c.call_tool("click", {
- "pid": self._safari_pid,
- "window_id": window_id,
- "element_index": idx,
- })
- time.sleep(0.3)
-
- # Type via keystroke synthesis.
- result = c.call_tool("type_text_chars", {
- "pid": self._safari_pid,
- "text": "hello bg",
- })
- print(f" type_text_chars result: {result}")
-
- has_text = False
- for attempt in range(4):
- time.sleep(0.5)
- with DriverClient(self.binary) as c:
- tree = _get_page_text(c, self._safari_pid)
- has_text = ("hello bg" in tree)
- if has_text:
- break
- print(f" attempt {attempt+1}: text not yet visible in tree")
-
- print(f" text found in tree: {has_text}")
- self.assertTrue(has_text, "Typed text not found in Safari AX tree")
-
- self._assert_no_focus_loss("02_type_text_chars")
-
- def test_03_pixel_click_no_focus_steal(self) -> None:
- """Pixel-click backgrounded Safari — verify no focus steal."""
- with DriverClient(self.binary) as c:
- window_id = resolve_window_id(c, self._safari_pid)
- snap = c.call_tool("get_window_state", {
- "pid": self._safari_pid,
- "window_id": window_id,
- "query": "Click Me",
- })
- sc = snap.get("structuredContent", snap)
- width = sc.get("screenshot_width", 0)
- height = sc.get("screenshot_height", 0)
- print(f"\n screenshot: {width}x{height}")
-
- x = width // 2
- y = int(height * 0.35)
- print(f" pixel click at ({x}, {y})")
-
- result = c.call_tool("click", {
- "pid": self._safari_pid,
- "window_id": window_id,
- "x": x,
- "y": y,
- })
- print(f" click result: {result}")
-
- time.sleep(0.5)
- self._assert_no_focus_loss("03_pixel_click_no_focus_steal")
-
- def test_04_press_key_tab(self) -> None:
- """Send Tab key to backgrounded Safari without stealing focus."""
- with DriverClient(self.binary) as c:
- result = c.call_tool("press_key", {
- "pid": self._safari_pid, "key": "tab",
- })
- print(f"\n press_key result: {result}")
-
- time.sleep(0.3)
- self._assert_no_focus_loss("04_press_key_tab")
-
-
-# ---------------------------------------------------------------------------
-# Calculator AX-click background tests
-# ---------------------------------------------------------------------------
-
-class CalculatorBackgroundClickTests(unittest.TestCase):
- """AX-click backgrounded Calculator without stealing focus."""
-
- CALC_BUNDLE = "com.apple.calculator"
-
- _calc_pid: int
- _focus_proc: subprocess.Popen
- _focus_pid: int
-
- @classmethod
- def setUpClass(cls) -> None:
- _build_focus_app()
- try:
- os.remove(_LOSS_FILE)
- except FileNotFoundError:
- pass
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
- # Kill any stale FocusMonitorApp left over from a crashed prior run.
- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)
- time.sleep(0.3)
-
- cls.binary = default_binary_path()
-
- with DriverClient(cls.binary) as c:
- result = c.call_tool("launch_app", {"bundle_id": cls.CALC_BUNDLE})
- cls._calc_pid = result["structuredContent"]["pid"]
- print(f"\n Calculator pid: {cls._calc_pid}")
- time.sleep(1.5)
-
- cls._focus_proc, cls._focus_pid = _launch_focus_app()
- print(f" FocusMonitor pid: {cls._focus_pid}")
- time.sleep(1.0)
-
- with DriverClient(cls.binary) as c:
- active = frontmost_bundle_id(c)
- assert active == FOCUS_MONITOR_BUNDLE, (
- f"Expected FocusMonitorApp frontmost, got {active}"
- )
-
- @classmethod
- def tearDownClass(cls) -> None:
- if hasattr(cls, '_focus_proc'):
- cls._focus_proc.terminate()
- try:
- cls._focus_proc.wait(timeout=3)
- except subprocess.TimeoutExpired:
- cls._focus_proc.kill()
- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
- try:
- os.remove(_LOSS_FILE)
- except FileNotFoundError:
- pass
-
- def setUp(self) -> None:
- _activate_focus_monitor()
- self._losses_before = _read_focus_losses()
- with DriverClient(self.binary) as c:
- active = frontmost_bundle_id(c)
- self.assertEqual(active, FOCUS_MONITOR_BUNDLE)
-
- def _assert_no_focus_loss(self, label: str) -> None:
- time.sleep(0.3)
- losses = _read_focus_losses()
- with DriverClient(self.binary) as c:
- active = frontmost_bundle_id(c)
- loss_delta = losses - self._losses_before
- print(f" [{label}] losses: {self._losses_before}->{losses} "
- f"(delta={loss_delta}), frontmost: {active}")
- self.assertEqual(
- active, FOCUS_MONITOR_BUNDLE,
- f"[{label}] Focus not restored — "
- f"frontmost is {active}, not FocusMonitorApp",
- )
-
- def test_01_ax_click_2_plus_2(self) -> None:
- """AX-click 2 + 2 = on Calculator while backgrounded."""
- with DriverClient(self.binary) as c:
- window_id = resolve_window_id(c, self._calc_pid)
- snap = c.call_tool(
- "get_window_state",
- {"pid": self._calc_pid, "window_id": window_id},
- )
- tree = snap.get("structuredContent", snap).get("tree_markdown", "")
- print(f"\n Calculator tree:\n{tree[:1500]}")
-
- btn_2 = _find_calc_button(tree, "2")
- btn_add = _find_calc_button(tree, "Add")
- btn_eq = _find_calc_button(tree, "Equals")
- print(f" buttons: 2={btn_2}, Add={btn_add}, Equals={btn_eq}")
-
- self.assertIsNotNone(btn_2, "'2' button not found")
- self.assertIsNotNone(btn_add, "'Add' button not found")
- self.assertIsNotNone(btn_eq, "'Equals' button not found")
-
- for idx in [btn_2, btn_add, btn_2, btn_eq]:
- c.call_tool("click", {
- "pid": self._calc_pid,
- "window_id": window_id,
- "element_index": idx,
- })
- time.sleep(0.3)
-
- snap = c.call_tool("get_window_state", {
- "pid": self._calc_pid,
- "window_id": window_id,
- "query": "AXStaticText",
- })
- tree = snap.get("structuredContent", snap).get("tree_markdown", "")
- print(f" result tree:\n{tree}")
-
- self.assertIn("4", tree, "Calculator did not show 4 after 2+2=")
- self._assert_no_focus_loss("01_ax_click_2_plus_2")
-
- def test_02_ax_click_3_plus_4(self) -> None:
- """AX-click 3 + 4 = on Calculator (second computation)."""
- with DriverClient(self.binary) as c:
- window_id = resolve_window_id(c, self._calc_pid)
- # Clear first
- snap = c.call_tool("get_window_state", {
- "pid": self._calc_pid, "window_id": window_id,
- })
- tree = snap.get("structuredContent", snap).get("tree_markdown", "")
- btn_c = _find_calc_button(tree, "All Clear")
- if btn_c is None:
- btn_c = _find_calc_button(tree, "Clear")
- if btn_c is not None:
- c.call_tool("click", {
- "pid": self._calc_pid,
- "window_id": window_id,
- "element_index": btn_c,
- })
- time.sleep(0.3)
-
- # Find buttons for 3+4=
- snap = c.call_tool("get_window_state", {
- "pid": self._calc_pid, "window_id": window_id,
- })
- tree = snap.get("structuredContent", snap).get("tree_markdown", "")
- btn_3 = _find_calc_button(tree, "3")
- btn_add = _find_calc_button(tree, "Add")
- btn_4 = _find_calc_button(tree, "4")
- btn_eq = _find_calc_button(tree, "Equals")
- print(f"\n buttons: 3={btn_3}, Add={btn_add}, 4={btn_4}, Equals={btn_eq}")
-
- self.assertIsNotNone(btn_3, "'3' button not found")
- self.assertIsNotNone(btn_add, "'Add' button not found")
- self.assertIsNotNone(btn_4, "'4' button not found")
- self.assertIsNotNone(btn_eq, "'Equals' button not found")
-
- for idx in [btn_3, btn_add, btn_4, btn_eq]:
- c.call_tool("click", {
- "pid": self._calc_pid,
- "window_id": window_id,
- "element_index": idx,
- })
- time.sleep(0.3)
-
- snap = c.call_tool("get_window_state", {
- "pid": self._calc_pid,
- "window_id": window_id,
- "query": "AXStaticText",
- })
- tree = snap.get("structuredContent", snap).get("tree_markdown", "")
- print(f" result tree:\n{tree}")
-
- self.assertIn("7", tree, "Calculator did not show 7 after 3+4=")
- self._assert_no_focus_loss("02_ax_click_3_plus_4")
-
-
-if __name__ == "__main__":
- unittest.main(verbosity=2)
diff --git a/libs/cua-driver/tests/e2e/test_blender.py b/libs/cua-driver/tests/e2e/test_blender.py
deleted file mode 100644
index b0455e6055..0000000000
--- a/libs/cua-driver/tests/e2e/test_blender.py
+++ /dev/null
@@ -1,198 +0,0 @@
-"""Blender background automation tests — v2 harness.
-
-Blender has no DOM, so assertions use:
- Primary: AX tree window title / menu bar labels
- Secondary: Screenshot + NCC template matching against reference crops
-
-Workflow:
- 1. Launch Blender (open -g -a Blender)
- 2. FocusMonitorApp takes focus
- 3. Tests interact via click / press_key / hotkey
- 4. Verify state via AX tree window title or screenshot diff
-
-Reference images for template matching are generated on first run and
-stored under assets/blender/. On subsequent runs the stored images are
-used as the match target.
-
-Run: python3 -m pytest test_blender.py -v
-"""
-
-from __future__ import annotations
-
-import subprocess
-import sys
-import time
-import os
-import json
-
-import pytest
-
-_HERE = os.path.dirname(os.path.abspath(__file__))
-sys.path.insert(0, _HERE)
-from harness.driver import Driver
-from harness import tree as Tree
-from harness.cv import decode, crop, diff_ratio, save_reference, load_reference
-
-BLENDER_BUNDLE = "org.blenderfoundation.blender"
-_ASSETS_BLENDER = os.path.join(_HERE, "assets", "blender")
-_BBOXES_FILE = os.path.join(_ASSETS_BLENDER, "bboxes.json")
-
-
-def _load_bboxes() -> dict:
- if os.path.exists(_BBOXES_FILE):
- with open(_BBOXES_FILE) as f:
- return json.load(f)
- return {}
-
-
-def _save_bboxes(bboxes: dict) -> None:
- os.makedirs(_ASSETS_BLENDER, exist_ok=True)
- with open(_BBOXES_FILE, "w") as f:
- json.dump(bboxes, f, indent=2)
-
-
-# ── module-level Blender setup ────────────────────────────────────────────────
-
[email protected](scope="module")
-def blender_pid(binary, focus_monitor):
- """Launch Blender in background; return its pid."""
- subprocess.run(["pkill", "-x", "Blender"], check=False)
- time.sleep(0.5)
-
- subprocess.run(["open", "-g", "-a", "Blender"], check=False)
- time.sleep(5.0) # Blender takes a while to start
-
- with Driver(binary) as d:
- app = d.find_app(BLENDER_BUNDLE)
- if app is None:
- # Try by name
- for a in d.list_apps():
- if "blender" in str(a.get("name", "")).lower():
- app = a
- break
- if app is None:
- pytest.skip("Blender not installed or not found in list_apps")
- pid = app["pid"]
-
- print(f"\n Blender pid: {pid}")
- yield pid
-
- subprocess.run(["pkill", "-x", "Blender"], check=False)
-
-
[email protected](autouse=True)
-def _reactivate_focus(focus_monitor):
- _, pid = focus_monitor
- subprocess.run(
- ["osascript", "-e",
- f'tell application "System Events" to set frontmost of (first process whose unix id is {pid}) to true'],
- check=False,
- )
- time.sleep(0.4)
-
-
-# ── tests ─────────────────────────────────────────────────────────────────────
-
-class TestBlenderWindow:
- """Basic window state and AX tree tests."""
-
- def test_window_state_returns_tree(self, driver, blender_pid, ux_guard):
- wid = driver.find_window(blender_pid)
- state = driver.get_window_state(blender_pid, wid)
- assert state.tree, "Empty AX tree from Blender"
- print(f"\n Blender tree (first 300 chars):\n{state.tree[:300]}")
-
- def test_window_state_has_screenshot(self, driver, blender_pid, ux_guard):
- wid = driver.find_window(blender_pid)
- state = driver.get_window_state(blender_pid, wid)
- assert state.screenshot_b64, "No screenshot from Blender"
- assert state.screenshot_width > 0
- assert state.screenshot_height > 0
- print(f"\n Blender screenshot: {state.screenshot_width}x{state.screenshot_height}")
-
- def test_ax_tree_has_window(self, driver, blender_pid, ux_guard):
- wid = driver.find_window(blender_pid)
- state = driver.get_window_state(blender_pid, wid)
- assert "AXWindow" in state.tree, f"No AXWindow in:\n{state.tree[:500]}"
-
-
-class TestBlenderMenuInteraction:
- """Open a menu via hotkey and verify AX tree changes."""
-
- def test_file_menu_via_hotkey(self, driver, blender_pid, ux_guard):
- """Open Blender's File menu via hotkey and verify menu appears."""
- wid = driver.find_window(blender_pid)
-
- before = driver.get_window_state(blender_pid, wid)
-
- # F4 opens File menu in Blender (or use hotkey cmd+shift+n)
- # Using press_key for a simple key that has observable AX effect
- driver.hotkey(blender_pid, ["f4"])
- time.sleep(0.8)
-
- after = driver.get_window_state(blender_pid, wid)
- tree_changed = after.tree != before.tree
- print(f"\n AX tree changed after F4: {tree_changed}")
- # Press Escape to close any menu
- driver.press_key(blender_pid, "escape")
- time.sleep(0.3)
-
- # Just verify the key reached Blender (driver didn't error)
- # AX tree change is a bonus — Blender menus may not expose via AX
- print(f" tree before: {len(before.tree)} chars, after: {len(after.tree)} chars")
-
-
-class TestBlenderScreenshotDiff:
- """Verify screenshot changes after interaction."""
-
- def test_click_viewport_changes_screenshot(self, driver, blender_pid, ux_guard):
- """Click in the 3D viewport and verify screenshot changes."""
- wid = driver.find_window(blender_pid)
-
- before = driver.get_window_state(blender_pid, wid)
- assert before.screenshot_b64
-
- # Click in the center of the viewport (assumed to be center of screen)
- x = before.screenshot_width // 2
- y = before.screenshot_height // 2
-
- before_img = decode(before.screenshot_b64)
-
- driver.click(blender_pid, wid, x=x, y=y)
- time.sleep(0.5)
-
- after = driver.get_window_state(blender_pid, wid)
- after_img = decode(after.screenshot_b64)
-
- ratio = diff_ratio(before_img, after_img)
- print(f"\n viewport click diff_ratio: {ratio:.4f}")
- # A click in Blender's viewport should change the selection highlight
- assert ratio >= 0, "diff_ratio calculation succeeded"
- # Note: Blender in background may not render selection changes
- # so we don't assert ratio > threshold — just that the call succeeded
-
- def test_screenshot_reference_workflow(self, driver, blender_pid, ux_guard):
- """Generate/verify a reference screenshot crop for NCC template matching."""
- wid = driver.find_window(blender_pid)
- state = driver.get_window_state(blender_pid, wid)
- assert state.screenshot_b64
-
- img = decode(state.screenshot_b64)
- w, h = img.size
-
- # Capture a reference crop of the top-left quadrant (toolbar area)
- bbox = (0, 0, w // 4, h // 4)
- ref_crop = crop(img, bbox)
-
- ref_path = os.path.join(_ASSETS_BLENDER, "toolbar_ref.png")
- os.makedirs(_ASSETS_BLENDER, exist_ok=True)
- save_reference(ref_crop, ref_path)
- assert os.path.exists(ref_path), "Reference image was not saved"
-
- # Load it back and verify it's identical to the crop
- loaded = load_reference(ref_path)
- ratio = diff_ratio(ref_crop, loaded)
- assert ratio < 0.001, f"Reference image roundtrip diff too high: {ratio:.4f}"
-
- print(f"\n reference saved: {ref_path} ({ref_crop.size})")
- print(f" roundtrip diff_ratio: {ratio:.6f}")
diff --git a/libs/cua-driver/tests/e2e/test_chrome.py b/libs/cua-driver/tests/e2e/test_chrome.py
deleted file mode 100644
index 8cef3b3afd..0000000000
--- a/libs/cua-driver/tests/e2e/test_chrome.py
+++ /dev/null
@@ -1,189 +0,0 @@
-"""Chrome background automation tests — v2 harness.
-
-Mirrors test_safari.py but targets Google Chrome. The same test_page.html
-is loaded via the local HTTP server. Chrome uses CGEvent/SkyLight for
-background delivery which is a different code path from Safari's AX.
-
-Run: python3 -m pytest test_chrome.py -v
-"""
-
-from __future__ import annotations
-
-import re
-import subprocess
-import sys
-import time
-import os
-
-import pytest
-
-_HERE = os.path.dirname(os.path.abspath(__file__))
-sys.path.insert(0, _HERE)
-from harness.driver import Driver
-from harness import tree as Tree
-
-
-def _parse_clicks(tree_text: str) -> int:
- m = re.search(r'clicks:\s*(\d+)', tree_text)
- return int(m.group(1)) if m else 0
-
-CHROME_BUNDLE = "com.google.Chrome"
-FOCUS_MONITOR_BUNDLE = "com.trycua.FocusMonitorApp"
-
-
-# ── module-level Chrome setup ─────────────────────────────────────────────────
-
-CHROME_APP = "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
-
-
[email protected](scope="module")
-def chrome_pid(binary, html_server, focus_monitor):
- """Launch Chrome with --force-renderer-accessibility to enable AX for web content.
-
- Chrome only exposes web content in the AX tree when launched with this flag,
- or when an AT (Assistive Technology) has connected. Using the flag is the most
- reliable approach for integration testing.
- """
- subprocess.run(["pkill", "-x", "Google Chrome"], check=False)
- time.sleep(0.5)
-
- if not os.path.exists(CHROME_APP):
- pytest.skip("Google Chrome not installed at default path")
-
- page_url = f"{html_server}/test_page.html"
- proc = subprocess.Popen(
- [CHROME_APP, "--force-renderer-accessibility", page_url],
- stdout=subprocess.DEVNULL,
- stderr=subprocess.DEVNULL,
- )
- time.sleep(5.0) # allow full page load + AX initialization
-
- with Driver(binary) as d:
- app = d.find_app(CHROME_BUNDLE)
- if app is None:
- proc.terminate()
- pytest.skip("Google Chrome did not appear in list_apps")
- pid = app["pid"]
-
- print(f"\n Chrome pid: {pid} (with renderer accessibility)")
-
- # Hand focus to FocusMonitorApp by pid to avoid activating stale instances
- _, fm_pid = focus_monitor
- subprocess.run(
- ["osascript", "-e",
- f'tell application "System Events" to set frontmost of (first process whose unix id is {fm_pid}) to true'],
- check=False,
- )
- time.sleep(0.5)
-
- yield pid
-
- proc.terminate()
- subprocess.run(["pkill", "-x", "Google Chrome"], check=False)
-
-
[email protected](autouse=True)
-def _reactivate_focus(focus_monitor):
- _, pid = focus_monitor
- subprocess.run(
- ["osascript", "-e",
- f'tell application "System Events" to set frontmost of (first process whose unix id is {pid}) to true'],
- check=False,
- )
- time.sleep(0.4)
-
-
-# ── tests ─────────────────────────────────────────────────────────────────────
-
-class TestChromeButton:
- def test_click_increments_counter(self, driver, chrome_pid, ux_guard):
- wid = driver.find_window(chrome_pid)
-
- full = driver.get_window_state(chrome_pid, wid)
- idx = full.find_element("Click Me")
- assert idx is not None, f"'Click Me' not found in tree (first 800 chars):\n{full.tree[:800]}"
- count_before = _parse_clicks(full.tree)
-
- driver.click(chrome_pid, wid, element_index=idx)
- time.sleep(0.5)
-
- after = driver.get_window_state(chrome_pid, wid)
- count_after = _parse_clicks(after.tree)
- print(f"\n click counter: {count_before} → {count_after}")
- assert count_after == count_before + 1, f"Counter did not increment: {count_before} → {count_after}"
-
- def test_click_increments_twice(self, driver, chrome_pid, ux_guard):
- wid = driver.find_window(chrome_pid)
- full = driver.get_window_state(chrome_pid, wid)
- idx = full.find_element("Click Me")
- assert idx is not None
- count_before = _parse_clicks(full.tree)
-
- driver.click(chrome_pid, wid, element_index=idx)
- time.sleep(0.3)
- driver.click(chrome_pid, wid, element_index=idx)
- time.sleep(0.5)
-
- after = driver.get_window_state(chrome_pid, wid)
- count_after = _parse_clicks(after.tree)
- print(f"\n click counter: {count_before} → {count_after}")
- assert count_after == count_before + 2, f"Counter should have incremented by 2: {count_before} → {count_after}"
-
-
-class TestChromeTextInput:
- def test_type_text_appears(self, driver, chrome_pid, ux_guard):
- wid = driver.find_window(chrome_pid)
- before = driver.get_window_state(chrome_pid, wid)
- idx = before.find_text_field(skip_url_bar=True)
- assert idx is not None, f"Text field not found:\n{before.tree}"
-
- driver.click(chrome_pid, wid, element_index=idx)
- time.sleep(0.3)
- driver.hotkey(chrome_pid, ["cmd", "a"])
- time.sleep(0.1)
- driver.type_text_chars(chrome_pid, "hello chrome")
- time.sleep(0.6)
-
- after = driver.get_window_state(chrome_pid, wid)
- assert after.has_text("hello chrome"), f"Text not found:\n{after.tree}"
-
-
-class TestChromeCanvas:
- def test_canvas_pixel_click_no_focus_steal(self, driver, chrome_pid, ux_guard):
- """Send a pixel click to the canvas area without stealing focus.
-
- Canvas DOM events may not fire in background; we verify the click is
- delivered without error and without UX violations via ux_guard.
- """
- wid = driver.find_window(chrome_pid)
- mid = driver.get_window_state(chrome_pid, wid)
- assert mid.screenshot_b64, "No screenshot"
-
- from harness.cv import decode, diff_ratio
- before_img = decode(mid.screenshot_b64)
-
- x = mid.screenshot_width // 2
- y = int(mid.screenshot_height * 0.88)
- print(f"\n chrome canvas click at ({x}, {y})")
-
- result = driver.click(chrome_pid, wid, x=x, y=y)
- time.sleep(0.5)
-
- after = driver.get_window_state(chrome_pid, wid)
- after_img = decode(after.screenshot_b64)
- ratio = diff_ratio(before_img, after_img)
- canvas_clicked = "canvas: clicked" in after.tree
- print(f" chrome canvas diff_ratio: {ratio:.4f}, JS fired: {canvas_clicked}")
-
- assert result is not None, "click returned None"
- # UX assertions via ux_guard teardown
-
-
-class TestChromePressKey:
- def test_tab_key(self, driver, chrome_pid, ux_guard):
- driver.press_key(chrome_pid, "tab")
- time.sleep(0.3)
-
- def test_escape_key(self, driver, chrome_pid, ux_guard):
- driver.press_key(chrome_pid, "escape")
- time.sleep(0.2)
diff --git a/libs/cua-driver/tests/e2e/test_cli.py b/libs/cua-driver/tests/e2e/test_cli.py
deleted file mode 100644
index c713315d35..0000000000
--- a/libs/cua-driver/tests/e2e/test_cli.py
+++ /dev/null
@@ -1,242 +0,0 @@
-"""Integration test: cua-driver-rs CLI subcommands.
-
-Covers:
- 1. `list-tools` prints tool names and summaries.
- 2. `describe <tool>` prints the tool schema.
- 3. `call <tool> [json]` invokes a tool and prints ✅ result.
- 4. Implicit call (first positional as tool name) works.
- 5. Unknown tool exits 64.
- 6. Error tool result exits 1.
- 7. `screenshot` call returns JSON with a screenshot_png_b64 key.
-
-Run:
- CUA_DRIVER_BINARY=../../target/debug/cua-driver python3 -m unittest test_cli -v
-"""
-
-from __future__ import annotations
-
-import json
-import os
-import subprocess
-import sys
-import unittest
-
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
-from driver_client import default_binary_path
-
-BINARY = None # set in setUpClass
-
-
-def _run(args: list[str], timeout: int = 15) -> subprocess.CompletedProcess:
- return subprocess.run(
- args,
- capture_output=True,
- text=True,
- timeout=timeout,
- )
-
-
-class CLISubcommandTests(unittest.TestCase):
- @classmethod
- def setUpClass(cls) -> None:
- cls.binary = default_binary_path()
-
- def test_list_tools_prints_tool_names(self) -> None:
- r = _run([self.binary, "list-tools"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- self.assertIn("click", r.stdout)
- self.assertIn("screenshot", r.stdout)
- self.assertIn("list_windows", r.stdout)
- self.assertIn("press_key", r.stdout)
-
- def test_list_tools_has_name_colon_summary_format(self) -> None:
- r = _run([self.binary, "list-tools"])
- self.assertEqual(r.returncode, 0)
- for line in r.stdout.strip().splitlines():
- self.assertRegex(
- line,
- r"^\w+",
- f"list-tools line should start with tool name: {line!r}",
- )
-
- def test_describe_prints_schema(self) -> None:
- r = _run([self.binary, "describe", "click"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- self.assertIn("click", r.stdout)
- self.assertIn("pid", r.stdout)
- self.assertIn("input_schema", r.stdout)
-
- def test_describe_unknown_tool_exits_64(self) -> None:
- r = _run([self.binary, "describe", "no_such_tool_xyz"])
- self.assertEqual(r.returncode, 64)
- self.assertIn("no_such_tool_xyz", r.stderr)
-
- def test_call_get_screen_size(self) -> None:
- r = _run([self.binary, "call", "get_screen_size"])
- self.assertEqual(r.returncode, 0, f"stdout: {r.stdout}\nstderr: {r.stderr}")
- data = json.loads(r.stdout)
- self.assertIn("width", data)
- self.assertIn("height", data)
- self.assertGreater(data["width"], 0)
- self.assertGreater(data["height"], 0)
-
- def test_implicit_call_get_screen_size(self) -> None:
- """First positional arg is treated as tool name (no 'call' prefix)."""
- r = _run([self.binary, "get_screen_size"])
- self.assertEqual(r.returncode, 0, f"stdout: {r.stdout}\nstderr: {r.stderr}")
- data = json.loads(r.stdout)
- self.assertIn("width", data)
- self.assertIn("height", data)
-
- def test_call_check_permissions_exits_zero(self) -> None:
- """check_permissions is read-only and always exits 0."""
- r = _run([self.binary, "call", "check_permissions"])
- self.assertEqual(r.returncode, 0, f"stdout: {r.stdout}\nstderr: {r.stderr}")
-
- def test_call_check_permissions_output_has_green_checkmark(self) -> None:
- r = _run([self.binary, "call", "check_permissions"])
- self.assertEqual(r.returncode, 0)
- # The structured JSON output should have accessibility/screen_recording fields.
- data = json.loads(r.stdout)
- self.assertIn("accessibility", data)
- self.assertIn("screen_recording", data)
-
- def test_call_unknown_tool_exits_64(self) -> None:
- r = _run([self.binary, "call", "no_such_tool_xyz"])
- self.assertEqual(r.returncode, 64)
-
- def test_call_press_key_with_missing_pid_exits_1(self) -> None:
- """Calling a tool with missing required args should return an MCP error → exit 1."""
- r = _run([self.binary, "call", "press_key", '{"key": "a"}'])
- self.assertEqual(r.returncode, 1, f"stdout: {r.stdout}\nstderr: {r.stderr}")
-
- def test_call_list_apps_json_output(self) -> None:
- r = _run([self.binary, "call", "list_apps"])
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- data = json.loads(r.stdout)
- self.assertIn("apps", data)
- self.assertIsInstance(data["apps"], list)
- self.assertGreater(len(data["apps"]), 0)
-
- def test_call_screenshot_returns_b64_image(self) -> None:
- r = _run([self.binary, "call", "screenshot"], timeout=30)
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- data = json.loads(r.stdout)
- self.assertIn("screenshot_png_b64", data, "screenshot output missing image key")
- b64 = data["screenshot_png_b64"]
- self.assertGreater(len(b64), 100, "screenshot base64 data seems too short")
-
-
-class ServeDaemonTests(unittest.TestCase):
- """Tests for `cua-driver serve` / `stop` / `status`."""
-
- @classmethod
- def setUpClass(cls) -> None:
- cls.binary = default_binary_path()
- import tempfile, os
- cls._sock_file = tempfile.mktemp(suffix=".sock", prefix="cua-driver-test-")
-
- def _start_daemon(self):
- import subprocess, time
- proc = subprocess.Popen(
- [self.binary, "serve", "--socket", self._sock_file],
- stdout=subprocess.PIPE, stderr=subprocess.PIPE,
- )
- # Wait for daemon to bind.
- for _ in range(30):
- time.sleep(0.1)
- r = _run([self.binary, "status", "--socket", self._sock_file])
- if r.returncode == 0:
- break
- return proc
-
- def test_status_exits_1_when_no_daemon(self) -> None:
- import tempfile
- sock = tempfile.mktemp(suffix=".sock", prefix="cua-noexist-")
- r = _run([self.binary, "status", "--socket", sock])
- self.assertEqual(r.returncode, 1)
- self.assertIn("not running", r.stderr)
-
- def test_stop_exits_1_when_no_daemon(self) -> None:
- import tempfile
- sock = tempfile.mktemp(suffix=".sock", prefix="cua-noexist2-")
- r = _run([self.binary, "stop", "--socket", sock])
- self.assertEqual(r.returncode, 1)
-
- def test_serve_status_stop_lifecycle(self) -> None:
- """Start daemon → status is running → stop → status is not running."""
- proc = self._start_daemon()
- try:
- # Status should show running.
- r = _run([self.binary, "status", "--socket", self._sock_file])
- self.assertEqual(r.returncode, 0, f"status stderr: {r.stderr}")
- self.assertIn("running", r.stdout)
- self.assertIn("socket:", r.stdout)
-
- # Stop the daemon.
- r = _run([self.binary, "stop", "--socket", self._sock_file])
- self.assertEqual(r.returncode, 0, f"stop stderr: {r.stderr}")
-
- # Status should now show not running.
- import time; time.sleep(0.2)
- r = _run([self.binary, "status", "--socket", self._sock_file])
- self.assertEqual(r.returncode, 1)
- finally:
- proc.terminate()
- proc.wait(timeout=3)
-
- def test_serve_double_start_exits_1(self) -> None:
- """Second `serve` on same socket should fail with exit 1."""
- proc = self._start_daemon()
- try:
- r = _run([self.binary, "serve", "--socket", self._sock_file])
- self.assertEqual(r.returncode, 1)
- self.assertIn("already running", r.stderr)
- finally:
- _run([self.binary, "stop", "--socket", self._sock_file])
- proc.wait(timeout=3)
-
- @unittest.skipIf(
- sys.platform == "win32",
- "Windows: daemon spawned by subprocess.Popen runs in same session as "
- "the test runner (typically Session 0 / non-interactive in CI); the "
- "GDI BitBlt path fails with `The handle is invalid (0x80070006)` "
- "because Session 0 has no graphics. The regression-guard behaviour "
- "still verifies on macOS/Linux runners.",
- )
- def test_call_screenshot_via_daemon_emits_b64(self) -> None:
- """`cua-driver call screenshot` over the daemon socket must emit
- `screenshot_png_b64` — same shape as the in-process path. Regression
- guard for the 2026-05-23 fix where the daemon-forwarding path in
- `run_call` was silently dropping the image bytes (only printing
- structuredContent metadata: format / width / height).
- """
- proc = self._start_daemon()
- try:
- r = _run(
- [self.binary, "call", "--socket", self._sock_file, "screenshot"],
- timeout=30,
- )
- self.assertEqual(r.returncode, 0, f"stderr: {r.stderr}")
- data = json.loads(r.stdout)
- self.assertIn(
- "screenshot_png_b64", data,
- "daemon-forwarded screenshot dropped the image — "
- "merge into structuredContent regressed",
- )
- b64 = data["screenshot_png_b64"]
- self.assertGreater(
- len(b64), 100,
- "screenshot base64 data seems too short — likely empty payload",
- )
- self.assertIn(
- "screenshot_mime_type", data,
- "merge into structuredContent missed the mime-type key",
- )
- finally:
- _run([self.binary, "stop", "--socket", self._sock_file])
- proc.wait(timeout=3)
-
-
-if __name__ == "__main__":
- unittest.main(verbosity=2)
diff --git a/libs/cua-driver/tests/e2e/test_concurrent_drivers.py b/libs/cua-driver/tests/e2e/test_concurrent_drivers.py
deleted file mode 100644
index 83585af4ae..0000000000
--- a/libs/cua-driver/tests/e2e/test_concurrent_drivers.py
+++ /dev/null
@@ -1,324 +0,0 @@
-"""Integration test: two cua-driver processes running concurrently.
-
-Verifies that two independent cua-driver instances can coexist without:
- - crashing each other
- - stealing focus from FocusMonitorApp
- - interfering with each other's operations
-
-Scenario:
- Driver A → AX-clicks Calculator (2 + 2 = 4)
- Driver B → Concurrently moves its agent cursor and inspects window list
-
-Run:
- CUA_DRIVER_BINARY=../../target/debug/cua-driver python3 -m unittest test_concurrent_drivers -v
-"""
-
-from __future__ import annotations
-
-import os
-import re
-import subprocess
-import sys
-import threading
-import time
-import unittest
-
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
-from driver_client import (
- DriverClient,
- default_binary_path,
- frontmost_bundle_id,
- resolve_window_id,
-)
-
-_THIS_DIR = os.path.dirname(os.path.abspath(__file__))
-_DRIVER_RS_ROOT = os.path.dirname(os.path.dirname(_THIS_DIR))
-_LIBS_ROOT = os.path.dirname(_DRIVER_RS_ROOT)
-_FOCUS_APP_DIR = os.path.join(_LIBS_ROOT, "cua-driver", "Tests", "FocusMonitorApp")
-_FOCUS_APP_EXE = os.path.join(
- _FOCUS_APP_DIR, "FocusMonitorApp.app", "Contents", "MacOS", "FocusMonitorApp"
-)
-_LOSS_FILE = "/tmp/focus_monitor_losses.txt"
-
-CALC_BUNDLE = "com.apple.calculator"
-FOCUS_MONITOR_BUNDLE = "com.trycua.FocusMonitorApp"
-
-
-def _build_focus_app() -> None:
- if not os.path.exists(_FOCUS_APP_EXE):
- subprocess.run(
- [os.path.join(_FOCUS_APP_DIR, "build.sh")], check=True
- )
-
-
-def _launch_focus_app() -> tuple[subprocess.Popen, int]:
- proc = subprocess.Popen(
- [_FOCUS_APP_EXE],
- stdout=subprocess.PIPE,
- stderr=subprocess.DEVNULL,
- text=True,
- )
- for _ in range(40):
- line = proc.stdout.readline().strip()
- if line.startswith("FOCUS_PID="):
- return proc, int(line.split("=", 1)[1])
- time.sleep(0.1)
- proc.terminate()
- raise RuntimeError("FocusMonitorApp did not print FOCUS_PID in time")
-
-
-def _read_focus_losses() -> int:
- try:
- with open(_LOSS_FILE) as f:
- return int(f.read().strip())
- except (FileNotFoundError, ValueError):
- return -1
-
-
-def _find_calc_button(tree: str, label: str) -> int | None:
- for line in tree.split("\n"):
- if "AXButton" not in line:
- continue
- m = re.search(r'\[(\d+)\]', line)
- if not m:
- continue
- if f'({label})' in line or f'id={label}' in line:
- return int(m.group(1))
- return None
-
-
-class ConcurrentDriversTest(unittest.TestCase):
- """Two driver processes running side-by-side without interference."""
-
- _calc_pid: int
- _focus_proc: subprocess.Popen
- _focus_pid: int
- binary: str
-
- @classmethod
- def setUpClass(cls) -> None:
- _build_focus_app()
-
- # Kill stale processes from any prior crashed run.
- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
- time.sleep(0.3)
-
- try:
- os.remove(_LOSS_FILE)
- except FileNotFoundError:
- pass
-
- cls.binary = default_binary_path()
-
- # Launch Calculator via Driver A.
- with DriverClient(cls.binary) as c:
- result = c.call_tool("launch_app", {"bundle_id": CALC_BUNDLE})
- cls._calc_pid = result["structuredContent"]["pid"]
- print(f"\n Calculator pid: {cls._calc_pid}")
- time.sleep(1.5)
-
- # Launch FocusMonitorApp (becomes frontmost).
- cls._focus_proc, cls._focus_pid = _launch_focus_app()
- print(f" FocusMonitor pid: {cls._focus_pid}")
- time.sleep(1.0)
-
- with DriverClient(cls.binary) as c:
- active = frontmost_bundle_id(c)
- assert active == FOCUS_MONITOR_BUNDLE, (
- f"Expected FocusMonitorApp frontmost, got {active}"
- )
-
- losses = _read_focus_losses()
- assert losses == 0, f"Expected 0 focus losses at start, got {losses}"
-
- @classmethod
- def tearDownClass(cls) -> None:
- if hasattr(cls, '_focus_proc'):
- cls._focus_proc.terminate()
- try:
- cls._focus_proc.wait(timeout=3)
- except subprocess.TimeoutExpired:
- cls._focus_proc.kill()
- if cls._focus_proc.stdout:
- cls._focus_proc.stdout.close()
- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
- try:
- os.remove(_LOSS_FILE)
- except FileNotFoundError:
- pass
-
- def setUp(self) -> None:
- # Clear Calculator before each test so previous display state doesn't interfere.
- with DriverClient(self.binary) as c:
- window_id = resolve_window_id(c, self._calc_pid)
- snap = c.call_tool("get_window_state", {
- "pid": self._calc_pid, "window_id": window_id,
- })
- tree = snap.get("structuredContent", snap).get("tree_markdown", "")
- btn_ac = _find_calc_button(tree, "All Clear") or _find_calc_button(tree, "Clear")
- if btn_ac is not None:
- c.call_tool("click", {
- "pid": self._calc_pid, "window_id": window_id, "element_index": btn_ac,
- })
- time.sleep(0.2)
-
- subprocess.run(
- ["osascript", "-e", 'tell application "FocusMonitorApp" to activate'],
- check=False,
- )
- time.sleep(0.5)
- self._losses_before = _read_focus_losses()
-
- # ── tests ─────────────────────────────────────────────────────────────────
-
- def test_two_drivers_run_concurrently(self) -> None:
- """Open Driver A and Driver B simultaneously; both must succeed without crashing."""
-
- driver_a_result: dict = {}
- driver_b_result: dict = {}
- driver_a_error: list[str] = []
- driver_b_error: list[str] = []
-
- def run_driver_a() -> None:
- try:
- with DriverClient(self.binary) as ca:
- # Driver A: AX-click 2 + 2 = on Calculator.
- window_id = resolve_window_id(ca, self._calc_pid)
- snap = ca.call_tool("get_window_state", {
- "pid": self._calc_pid, "window_id": window_id,
- })
- tree = snap.get("structuredContent", snap).get("tree_markdown", "")
- btn_2 = _find_calc_button(tree, "2")
- btn_add = _find_calc_button(tree, "Add")
- btn_eq = _find_calc_button(tree, "Equals")
- if None in (btn_2, btn_add, btn_eq):
- driver_a_error.append("Missing calculator buttons")
- return
- for idx in [btn_2, btn_add, btn_2, btn_eq]:
- ca.call_tool("click", {
- "pid": self._calc_pid,
- "window_id": window_id,
- "element_index": idx,
- })
- time.sleep(0.15)
- snap2 = ca.call_tool("get_window_state", {
- "pid": self._calc_pid, "window_id": window_id,
- "query": "AXStaticText",
- })
- tree2 = snap2.get("structuredContent", snap2).get("tree_markdown", "")
- driver_a_result["tree"] = tree2
- except Exception as e:
- driver_a_error.append(str(e))
-
- def run_driver_b() -> None:
- try:
- with DriverClient(self.binary) as cb:
- # Driver B: move its cursor, list windows, move cursor again.
- cb.call_tool("move_cursor", {"x": 200.0, "y": 200.0, "cursor_id": "agent-b"})
- wins = cb.call_tool("list_windows", {"on_screen_only": True})
- count = len(wins.get("structuredContent", {}).get("windows", []))
- cb.call_tool("move_cursor", {"x": 300.0, "y": 300.0, "cursor_id": "agent-b"})
- driver_b_result["window_count"] = count
- except Exception as e:
- driver_b_error.append(str(e))
-
- # Launch both threads simultaneously.
- ta = threading.Thread(target=run_driver_a, daemon=True)
- tb = threading.Thread(target=run_driver_b, daemon=True)
- ta.start()
- tb.start()
- ta.join(timeout=30)
- tb.join(timeout=30)
-
- # Both must have completed without error.
- self.assertFalse(
- driver_a_error,
- f"Driver A errored: {driver_a_error}",
- )
- self.assertFalse(
- driver_b_error,
- f"Driver B errored: {driver_b_error}",
- )
-
- print(f"\n Driver A result tree:\n{driver_a_result.get('tree', '(empty)')}")
- print(f" Driver B window count: {driver_b_result.get('window_count', -1)}")
-
- # Driver A should have produced "4".
- self.assertIn(
- "4",
- driver_a_result.get("tree", ""),
- "Calculator did not show 4 after Driver A clicked 2+2=",
- )
-
- # Driver B should have seen at least some windows.
- self.assertGreater(
- driver_b_result.get("window_count", 0),
- 0,
- "Driver B saw no windows in list_windows",
- )
-
- # Focus must not have been stolen.
- time.sleep(0.3)
- losses = _read_focus_losses()
- with DriverClient(self.binary) as c:
- active = frontmost_bundle_id(c)
- print(f" losses: {self._losses_before}->{losses}, frontmost: {active}")
- self.assertEqual(
- active, FOCUS_MONITOR_BUNDLE,
- f"Focus stolen during concurrent operation — frontmost is {active}",
- )
-
- def test_sequential_driver_reuse(self) -> None:
- """Open/close/reopen a DriverClient — state from prior session does not bleed."""
-
- # First session — clear Calculator.
- with DriverClient(self.binary) as c:
- window_id = resolve_window_id(c, self._calc_pid)
- snap = c.call_tool("get_window_state", {
- "pid": self._calc_pid, "window_id": window_id,
- })
- tree = snap.get("structuredContent", snap).get("tree_markdown", "")
- btn_ac = _find_calc_button(tree, "All Clear") or _find_calc_button(tree, "Clear")
- if btn_ac is not None:
- c.call_tool("click", {
- "pid": self._calc_pid, "window_id": window_id, "element_index": btn_ac,
- })
- time.sleep(0.2)
-
- time.sleep(0.2)
-
- # Second session — press 5.
- with DriverClient(self.binary) as c:
- window_id = resolve_window_id(c, self._calc_pid)
- snap = c.call_tool("get_window_state", {
- "pid": self._calc_pid, "window_id": window_id,
- })
- tree = snap.get("structuredContent", snap).get("tree_markdown", "")
- btn_5 = _find_calc_button(tree, "5")
- self.assertIsNotNone(btn_5, "Could not find '5' button in fresh session")
- c.call_tool("click", {
- "pid": self._calc_pid, "window_id": window_id, "element_index": btn_5,
- })
- time.sleep(0.3)
- snap2 = c.call_tool("get_window_state", {
- "pid": self._calc_pid, "window_id": window_id, "query": "AXStaticText",
- })
- tree2 = snap2.get("structuredContent", snap2).get("tree_markdown", "")
- print(f"\n After pressing 5: {tree2[:200]}")
-
- self.assertIn("5", tree2, "Calculator did not show 5 after clicking it in a new session")
-
- # Focus check.
- time.sleep(0.3)
- with DriverClient(self.binary) as c:
- active = frontmost_bundle_id(c)
- self.assertEqual(
- active, FOCUS_MONITOR_BUNDLE,
- f"Focus stolen — frontmost is {active}",
- )
-
-
-if __name__ == "__main__":
- unittest.main(verbosity=2)
diff --git a/libs/cua-driver/tests/e2e/test_cursor_visibility.py b/libs/cua-driver/tests/e2e/test_cursor_visibility.py
deleted file mode 100644
index 1bfdccc316..0000000000
--- a/libs/cua-driver/tests/e2e/test_cursor_visibility.py
+++ /dev/null
@@ -1,426 +0,0 @@
-"""Integration test: agent-cursor custom-icon visibility on screen.
-
-Verifies that after:
- 1. Setting a custom cursor icon via set_agent_cursor_style
- 2. Performing a click (which triggers ClickPulse moving the cursor to the
- click point and PinAbove ordering the overlay above the target window)
-
-... the custom cursor icon is visually present at the expected screen position
-in a full-screen screenshot.
-
-The test cursor is a bright-magenta circle with a yellow cross — chosen because
-it's easy to detect via colour thresholding without needing a heavy CV library.
-Only the stdlib + base64/struct/zlib are needed for the screenshot decode.
-
-Run:
- CUA_DRIVER_BINARY=../../target/debug/cua-driver python3 -m unittest test_cursor_visibility -v
-"""
-
-from __future__ import annotations
-
-import base64
-import os
-import struct
-import subprocess
-import sys
-import time
-import unittest
-import zlib
-
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
-from driver_client import DriverClient, default_binary_path, resolve_window_id
-
-_THIS_DIR = os.path.dirname(os.path.abspath(__file__))
-_FIXTURE_CURSOR = os.path.join(_THIS_DIR, "fixtures", "test_cursor.png")
-
-CALCULATOR_BUNDLE = "com.apple.calculator"
-
-# Colour tolerances for detection
-_MAGENTA_MIN = (200, 0, 200) # R≥200, G≤55, B≥200
-_MAGENTA_MAX = (255, 55, 255)
-_YELLOW_MIN = (200, 200, 0) # R≥200, G≥200, B≤55
-_YELLOW_MAX = (255, 255, 55)
-
-# Minimum fraction of pixels in the search region that must match the cursor colours.
-_MATCH_THRESHOLD = 0.002 # 0.2% of the search window
-
-
-def _decode_png_pixels(png_bytes: bytes) -> tuple[int, int, list[tuple[int, int, int]]]:
- """Minimal PNG decoder: returns (width, height, [(r,g,b), ...]) for RGB/RGBA PNGs.
-
- Only supports filter types 0 (None) and 1 (Sub) — which covers the screenshots
- produced by cua-driver's `screenshot` tool. Raises ValueError for unsupported formats.
- """
- if png_bytes[:8] != b'\x89PNG\r\n\x1a\n':
- raise ValueError("Not a PNG file")
-
- pos = 8
- width = height = 0
- idat_chunks: list[bytes] = []
- bit_depth = color_type = 0
-
- while pos < len(png_bytes):
- if pos + 8 > len(png_bytes):
- break
- length = struct.unpack('>I', png_bytes[pos:pos+4])[0]
- chunk_type = png_bytes[pos+4:pos+8]
- data = png_bytes[pos+8:pos+8+length]
- pos += 12 + length
-
- if chunk_type == b'IHDR':
- width, height = struct.unpack('>II', data[:8])
- bit_depth = data[8]
- color_type = data[9]
- elif chunk_type == b'IDAT':
- idat_chunks.append(data)
- elif chunk_type == b'IEND':
- break
-
- if width == 0 or height == 0:
- raise ValueError("No IHDR found")
- if bit_depth != 8:
- raise ValueError(f"Unsupported bit depth: {bit_depth}")
-
- raw = zlib.decompress(b''.join(idat_chunks))
-
- # Channels: 2=RGB(3), 6=RGBA(4)
- channels = {2: 3, 6: 4}.get(color_type)
- if channels is None:
- raise ValueError(f"Unsupported color_type: {color_type}")
-
- stride = width * channels
- pixels: list[tuple[int, int, int]] = []
- prev_row = bytes(stride)
-
- offset = 0
- for _y in range(height):
- filt = raw[offset]
- offset += 1
- row = bytearray(raw[offset:offset+stride])
- offset += stride
-
- if filt == 0:
- pass # None
- elif filt == 1:
- for i in range(channels, stride):
- row[i] = (row[i] + row[i - channels]) & 0xFF
- elif filt == 2:
- for i in range(stride):
- row[i] = (row[i] + prev_row[i]) & 0xFF
- elif filt == 3:
- for i in range(stride):
- a = row[i - channels] if i >= channels else 0
- b = prev_row[i]
- row[i] = (row[i] + (a + b) // 2) & 0xFF
- elif filt == 4:
- for i in range(stride):
- a = row[i - channels] if i >= channels else 0
- b = prev_row[i]
- c = prev_row[i - channels] if i >= channels else 0
- p = a + b - c
- pa = abs(p - a); pb = abs(p - b); pc = abs(p - c)
- pr = a if pa <= pb and pa <= pc else (b if pb <= pc else c)
- row[i] = (row[i] + pr) & 0xFF
- else:
- raise ValueError(f"Unknown PNG filter type: {filt}")
-
- for i in range(0, stride, channels):
- pixels.append((row[i], row[i+1], row[i+2]))
- prev_row = bytes(row)
-
- return width, height, pixels
-
-
-def _colour_in_region(
- pixels: list[tuple[int, int, int]],
- img_w: int,
- img_h: int,
- cx: float,
- cy: float,
- radius: int,
- colour_min: tuple[int, int, int],
- colour_max: tuple[int, int, int],
-) -> float:
- """Return fraction of pixels in circle around (cx, cy) that match the colour range."""
- x0 = max(0, int(cx) - radius)
- x1 = min(img_w - 1, int(cx) + radius)
- y0 = max(0, int(cy) - radius)
- y1 = min(img_h - 1, int(cy) + radius)
-
- total = 0
- matches = 0
- for y in range(y0, y1 + 1):
- for x in range(x0, x1 + 1):
- idx = y * img_w + x
- if idx >= len(pixels):
- continue
- r, g, b = pixels[idx]
- total += 1
- if (colour_min[0] <= r <= colour_max[0] and
- colour_min[1] <= g <= colour_max[1] and
- colour_min[2] <= b <= colour_max[2]):
- matches += 1
-
- return matches / total if total > 0 else 0.0
-
-
-class TestCursorVisibility(unittest.TestCase):
- """Custom cursor icon is visible on screen after a click."""
-
- @classmethod
- def setUpClass(cls) -> None:
- if not os.path.exists(_FIXTURE_CURSOR):
- cls.skipTest(cls, f"test cursor fixture missing: {_FIXTURE_CURSOR}") # type: ignore[arg-type]
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
- time.sleep(0.3)
-
- @classmethod
- def tearDownClass(cls) -> None:
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
-
- def setUp(self) -> None:
- self.client = DriverClient(default_binary_path()).__enter__()
- # Verify set_agent_cursor_style is available.
- tools = {t["name"] for t in self.client.list_tools()}
- if "set_agent_cursor_style" not in tools:
- self.client.__exit__(None, None, None)
- self.skipTest("set_agent_cursor_style tool not available in this build")
-
- def tearDown(self) -> None:
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
- self.client.__exit__(None, None, None)
-
- def test_custom_cursor_visible_after_click(self) -> None:
- """After set_agent_cursor_style + click, the custom icon is on-screen."""
- client = self.client
-
- # 1. Enable cursor overlay.
- r = client.call_tool("set_agent_cursor_enabled", {"enabled": True})
- self.assertFalse(r.get("isError"), f"set_agent_cursor_enabled failed: {r}")
-
- # 2. Upload the bright-magenta test cursor.
- r = client.call_tool("set_agent_cursor_style", {"image_path": _FIXTURE_CURSOR})
- self.assertFalse(r.get("isError"), f"set_agent_cursor_style failed: {r}")
- text = r.get("content", [{}])[0].get("text", "")
- self.assertIn("✅", text, f"set_agent_cursor_style didn't return ✅: {text}")
- self.assertIn(_FIXTURE_CURSOR, text, f"image_path not echoed in response: {text}")
-
- # 3. Launch Calculator in the background.
- r = client.call_tool("launch_app", {"bundle_id": CALCULATOR_BUNDLE})
- self.calc_pid = r["structuredContent"]["pid"]
- time.sleep(0.6)
-
- # 4. Resolve the Calculator window and its screen bounds.
- win_id = resolve_window_id(client, self.calc_pid)
- wins = client.call_tool("list_windows", {"pid": self.calc_pid})["structuredContent"]["windows"]
- win = next((w for w in wins if w["window_id"] == win_id), None)
- self.assertIsNotNone(win, "Could not find Calculator window")
- bounds = win["bounds"]
-
- # 5. Click the center of the Calculator window (window-local coords).
- click_x = bounds["width"] / 2.0
- click_y = bounds["height"] / 2.0
- r = client.call_tool("click", {
- "pid": self.calc_pid,
- "window_id": win_id,
- "x": click_x,
- "y": click_y,
- })
- self.assertFalse(r.get("isError"), f"click failed: {r}")
-
- # 6. Compute expected screen coordinates of the click point.
- # The driver translates window-local coords to screen coords by adding
- # the window origin and dividing by the Retina scale (detected at click time).
- # We use the same heuristic: try a scale of 2.0 (Retina), fall back to 1.0.
- # If the click lands within ±64 px of our estimate it's still detectable.
- win_x = bounds["x"]
- win_y = bounds["y"]
- # Best-effort Retina scale detection via screenshot width vs logical width.
- try:
- scr = client.call_tool("screenshot", {"pid": self.calc_pid, "window_id": win_id})
- b64 = scr.get("content", [{}])[0].get("data", "")
- if b64:
- png = base64.b64decode(b64)
- # PNG width is at bytes 16..20 in big-endian.
- png_w = struct.unpack('>I', png[16:20])[0]
- scale = png_w / bounds["width"] if bounds["width"] > 0 else 1.0
- else:
- scale = 1.0
- except Exception:
- scale = 1.0
-
- screen_x = win_x + click_x / scale
- screen_y = win_y + click_y / scale
-
- # 7. Wait for cursor animation + render loop to settle.
- time.sleep(0.5)
-
- # 8. Take a full-screen screenshot. Force PNG so the per-pixel decode
- # below works — the screenshot tool defaults to JPEG, which would be
- # lossy and skew the cursor-colour search.
- scr = client.call_tool("screenshot", {"format": "png"})
- b64 = scr.get("content", [{}])[0].get("data", "")
- self.assertTrue(b64, "screenshot returned no data")
- png_bytes = base64.b64decode(b64)
-
- # 9. Decode the PNG and check for our cursor colours near (screen_x, screen_y).
- try:
- img_w, img_h, pixels = _decode_png_pixels(png_bytes)
- except Exception as e:
- self.skipTest(f"PNG decode failed (unsupported screenshot format): {e}")
-
- # The screenshot may be at 2x resolution; scale screen_x/y accordingly.
- ss_scale = img_w / _get_screen_width(client)
- sx = screen_x * ss_scale
- sy = screen_y * ss_scale
- search_r = int(80 * ss_scale)
-
- magenta_frac = _colour_in_region(
- pixels, img_w, img_h, sx, sy, search_r,
- _MAGENTA_MIN, _MAGENTA_MAX,
- )
- yellow_frac = _colour_in_region(
- pixels, img_w, img_h, sx, sy, search_r,
- _YELLOW_MIN, _YELLOW_MAX,
- )
-
- self.assertGreater(
- magenta_frac, _MATCH_THRESHOLD,
- f"Magenta cursor colour NOT found near click point "
- f"(screen={screen_x:.0f},{screen_y:.0f} img={sx:.0f},{sy:.0f} "
- f"scale={ss_scale:.2f}). "
- f"Magenta fraction={magenta_frac:.5f} (threshold={_MATCH_THRESHOLD}). "
- f"Yellow fraction={yellow_frac:.5f}. "
- f"The overlay may be hidden or behind another window.",
- )
- self.assertGreater(
- yellow_frac, _MATCH_THRESHOLD,
- f"Yellow cross NOT found in cursor region "
- f"(magenta={magenta_frac:.5f}, yellow={yellow_frac:.5f}). "
- f"Cursor shape may not have loaded correctly.",
- )
-
- def test_default_cursor_visible_after_click(self) -> None:
- """After a plain click (no custom icon), the default gradient arrow is on-screen.
-
- Detects the ice-blue bloom/arrow colour (cursor_mid ≈ R94 G192 B232) that the
- built-in gradient arrow uses. Does NOT set a custom cursor icon.
- """
- client = self.client
-
- # 1. Enable overlay with default cursor (no custom icon).
- r = client.call_tool("set_agent_cursor_enabled", {"enabled": True})
- self.assertFalse(r.get("isError"), f"set_agent_cursor_enabled failed: {r}")
-
- # Ensure no custom icon is active (revert to default arrow).
- r = client.call_tool("set_agent_cursor_style", {"image_path": ""})
- self.assertFalse(r.get("isError"), f"revert to default cursor failed: {r}")
-
- # 2. Launch Calculator in the background.
- r = client.call_tool("launch_app", {"bundle_id": CALCULATOR_BUNDLE})
- self.calc_pid = r["structuredContent"]["pid"]
- time.sleep(0.6)
-
- # 3. Resolve window and screen bounds.
- win_id = resolve_window_id(client, self.calc_pid)
- wins = client.call_tool("list_windows", {"pid": self.calc_pid})["structuredContent"]["windows"]
- win = next((w for w in wins if w["window_id"] == win_id), None)
- self.assertIsNotNone(win, "Could not find Calculator window")
- bounds = win["bounds"]
-
- click_x = bounds["width"] / 2.0
- click_y = bounds["height"] / 2.0
-
- # Detect Retina scale.
- try:
- scr = client.call_tool("screenshot", {"pid": self.calc_pid, "window_id": win_id})
- b64 = scr.get("content", [{}])[0].get("data", "")
- png = base64.b64decode(b64) if b64 else b""
- png_w = struct.unpack('>I', png[16:20])[0] if len(png) >= 24 else 0
- scale = png_w / bounds["width"] if bounds["width"] > 0 and png_w > 0 else 1.0
- except Exception:
- scale = 1.0
-
- # 4. Click the centre of the window.
- r = client.call_tool("click", {
- "pid": self.calc_pid,
- "window_id": win_id,
- "x": click_x,
- "y": click_y,
- })
- self.assertFalse(r.get("isError"), f"click failed: {r}")
-
- # 5. Wait for 750ms glide + spring settle + a safety margin.
- time.sleep(1.2)
-
- # 6. Full-screen screenshot.
- scr = client.call_tool("screenshot", {})
- b64 = scr.get("content", [{}])[0].get("data", "")
- self.assertTrue(b64, "screenshot returned no data")
- png_bytes = base64.b64decode(b64)
-
- try:
- img_w, img_h, pixels = _decode_png_pixels(png_bytes)
- except Exception as e:
- self.skipTest(f"PNG decode failed: {e}")
-
- win_x, win_y = bounds["x"], bounds["y"]
- screen_x = win_x + click_x / scale
- screen_y = win_y + click_y / scale
-
- ss_scale = img_w / _get_screen_width(client)
- sx = screen_x * ss_scale
- sy = screen_y * ss_scale
- search_r = int(80 * ss_scale)
-
- # Default arrow: ice-blue/cyan — cursor_mid ≈ (94, 192, 232), bloom ≈ (188, 232, 252).
- # Accept any pixel that is clearly blue-cyan (G and B significantly exceed R).
- _CYAN_MIN = (40, 150, 180)
- _CYAN_MAX = (180, 255, 255)
-
- cyan_frac = _colour_in_region(
- pixels, img_w, img_h, sx, sy, search_r,
- _CYAN_MIN, _CYAN_MAX,
- )
-
- self.assertGreater(
- cyan_frac, _MATCH_THRESHOLD,
- f"Default ice-blue cursor NOT found near click point "
- f"(screen={screen_x:.0f},{screen_y:.0f} img_pt={sx:.0f},{sy:.0f} "
- f"scale={ss_scale:.2f}). "
- f"Cyan fraction={cyan_frac:.5f} (threshold={_MATCH_THRESHOLD}). "
- f"The default gradient arrow overlay may be invisible.",
- )
-
- def test_set_agent_cursor_style_gradient_only(self) -> None:
- """set_agent_cursor_style with gradient_colors updates without error."""
- r = self.client.call_tool("set_agent_cursor_style", {
- "gradient_colors": ["#FF0000", "#FF00FF", "#0000FF"],
- "bloom_color": "#00FFFF",
- })
- self.assertFalse(r.get("isError"), f"gradient-only style update failed: {r}")
- text = r.get("content", [{}])[0].get("text", "")
- self.assertIn("✅", text)
- self.assertIn("#FF0000", text)
- self.assertIn("#00FFFF", text)
-
- def test_set_agent_cursor_style_revert(self) -> None:
- """Empty image_path reverts the cursor to the procedural arrow without error."""
- # First set a custom icon.
- self.client.call_tool("set_agent_cursor_style", {"image_path": _FIXTURE_CURSOR})
- # Then revert.
- r = self.client.call_tool("set_agent_cursor_style", {"image_path": ""})
- self.assertFalse(r.get("isError"), f"revert failed: {r}")
- text = r.get("content", [{}])[0].get("text", "")
- self.assertIn("✅", text)
- self.assertIn("reverted", text)
-
-
-def _get_screen_width(client: DriverClient) -> float:
- """Return the logical screen width in points."""
- r = client.call_tool("get_screen_size")
- return float(r["structuredContent"].get("width", 1920))
-
-
-if __name__ == "__main__":
- unittest.main(verbosity=2)
diff --git a/libs/cua-driver/tests/e2e/test_double_click_delivery.py b/libs/cua-driver/tests/e2e/test_double_click_delivery.py
deleted file mode 100644
index f272bd859d..0000000000
--- a/libs/cua-driver/tests/e2e/test_double_click_delivery.py
+++ /dev/null
@@ -1,253 +0,0 @@
-"""Integration test: `double_click` tool delivers two presses to a
-backgrounded target.
-
-Two addressing modes, one invariant: double-clicking Calculator's "2"
-button from behind FocusMonitorApp should produce "22" in the display
-and must not steal focus.
-
-- Pixel path: `double_click({pid, x, y})` → synthesized CGEvent double-click.
-- Element-index path on an AXButton → performs two AX presses.
-
-Run:
- CUA_DRIVER_BINARY=../../target/debug/cua-driver python3 -m unittest test_double_click_delivery -v
-"""
-
-from __future__ import annotations
-
-import os
-import re
-import subprocess
-import sys
-import time
-import unittest
-
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
-from driver_client import (
- DriverClient,
- default_binary_path,
- frontmost_bundle_id,
- resolve_window_id,
-)
-
-_THIS_DIR = os.path.dirname(os.path.abspath(__file__))
-_DRIVER_RS_ROOT = os.path.dirname(os.path.dirname(_THIS_DIR))
-_LIBS_ROOT = os.path.dirname(_DRIVER_RS_ROOT)
-_FOCUS_APP_DIR = os.path.join(_LIBS_ROOT, "cua-driver", "Tests", "FocusMonitorApp")
-_FOCUS_APP_EXE = os.path.join(
- _FOCUS_APP_DIR, "FocusMonitorApp.app", "Contents", "MacOS", "FocusMonitorApp"
-)
-_LOSS_FILE = "/tmp/focus_monitor_losses.txt"
-
-CALC_BUNDLE = "com.apple.calculator"
-FOCUS_MONITOR_BUNDLE = "com.trycua.FocusMonitorApp"
-
-CALC_TWO_BUTTON_XY = (86, 313)
-CALC_DEFAULT_W = 230
-CALC_DEFAULT_H = 408
-
-
-def _build_focus_app() -> None:
- if not os.path.exists(_FOCUS_APP_EXE):
- subprocess.run(
- [os.path.join(_FOCUS_APP_DIR, "build.sh")], check=True
- )
-
-
-def _launch_focus_app() -> tuple[subprocess.Popen, int]:
- proc = subprocess.Popen(
- [_FOCUS_APP_EXE],
- stdout=subprocess.PIPE,
- stderr=subprocess.DEVNULL,
- text=True,
- )
- for _ in range(40):
- line = proc.stdout.readline().strip()
- if line.startswith("FOCUS_PID="):
- return proc, int(line.split("=", 1)[1])
- time.sleep(0.1)
- proc.terminate()
- raise RuntimeError("FocusMonitorApp did not print FOCUS_PID in time")
-
-
-def _read_focus_losses() -> int:
- try:
- with open(_LOSS_FILE) as f:
- return int(f.read().strip())
- except (FileNotFoundError, ValueError):
- return -1
-
-
-def _find_calc_button(tree: str, label: str) -> int | None:
- for line in tree.split("\n"):
- if "AXButton" not in line:
- continue
- m = re.search(r"\[(\d+)\]", line)
- if not m:
- continue
- if f"({label})" in line or f"id={label}" in line:
- return int(m.group(1))
- return None
-
-
-def _calc_display(client: DriverClient, pid: int) -> str:
- window_id = resolve_window_id(client, pid)
- result = client.call_tool(
- "get_window_state",
- {"pid": pid, "window_id": window_id, "query": "AXStaticText"},
- )
- tree = result.get("structuredContent", result).get("tree_markdown", "")
- for line in tree.split("\n"):
- if "AXStaticText" in line and "=" in line:
- m = re.search(r'= "([^"]*)"', line)
- if m:
- return m.group(1).replace("\u200e", "").replace("\u200f", "")
- return ""
-
-
-class DoubleClickDeliveryTests(unittest.TestCase):
- """double_click on backgrounded Calculator — focus must not be stolen."""
-
- _calc_pid: int
- _focus_proc: subprocess.Popen
- _focus_pid: int
- binary: str
-
- @classmethod
- def setUpClass(cls) -> None:
- _build_focus_app()
- try:
- os.remove(_LOSS_FILE)
- except FileNotFoundError:
- pass
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
- # Kill any stale FocusMonitorApp left over from a crashed prior run.
- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)
- time.sleep(0.3)
-
- cls.binary = default_binary_path()
-
- with DriverClient(cls.binary) as c:
- result = c.call_tool("launch_app", {"bundle_id": CALC_BUNDLE})
- cls._calc_pid = result["structuredContent"]["pid"]
- print(f"\n Calculator pid: {cls._calc_pid}")
- time.sleep(1.5)
-
- cls._focus_proc, cls._focus_pid = _launch_focus_app()
- print(f" FocusMonitor pid: {cls._focus_pid}")
- time.sleep(1.0)
- subprocess.run(
- ["osascript", "-e", 'tell application "FocusMonitorApp" to activate'],
- check=False,
- )
- time.sleep(0.8)
-
- with DriverClient(cls.binary) as c:
- active = frontmost_bundle_id(c)
- assert active == FOCUS_MONITOR_BUNDLE, (
- f"Expected FocusMonitorApp frontmost, got {active}"
- )
- losses = _read_focus_losses()
- assert losses == 0, f"Expected 0 focus losses at start, got {losses}"
-
- @classmethod
- def tearDownClass(cls) -> None:
- if hasattr(cls, '_focus_proc'):
- cls._focus_proc.terminate()
- try:
- cls._focus_proc.wait(timeout=3)
- except subprocess.TimeoutExpired:
- cls._focus_proc.kill()
- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
- try:
- os.remove(_LOSS_FILE)
- except FileNotFoundError:
- pass
-
- def setUp(self) -> None:
- with DriverClient(self.binary) as c:
- window_id = resolve_window_id(c, self._calc_pid)
- snap = c.call_tool(
- "get_window_state",
- {"pid": self._calc_pid, "window_id": window_id},
- )
- tree = snap.get("structuredContent", snap).get("tree_markdown", "")
- btn = _find_calc_button(tree, "All Clear") or _find_calc_button(tree, "Clear")
- if btn is not None:
- c.call_tool("click", {
- "pid": self._calc_pid,
- "window_id": window_id,
- "element_index": btn,
- })
- time.sleep(0.3)
- subprocess.run(
- ["osascript", "-e", 'tell application "FocusMonitorApp" to activate'],
- check=False,
- )
- time.sleep(0.5)
- self._losses_before = _read_focus_losses()
-
- def _assert_no_focus_steal(self) -> None:
- time.sleep(0.3)
- losses = _read_focus_losses()
- with DriverClient(self.binary) as c:
- active = frontmost_bundle_id(c)
- print(f" losses: {self._losses_before}->{losses}, frontmost: {active}")
- self.assertEqual(
- active, FOCUS_MONITOR_BUNDLE,
- f"Focus stolen — frontmost is {active}",
- )
-
- def test_element_index_double_click_no_focus_steal(self) -> None:
- """double_click via element_index on Calculator '2' button — no focus steal."""
- with DriverClient(self.binary) as c:
- window_id = resolve_window_id(c, self._calc_pid)
- snap = c.call_tool("get_window_state", {
- "pid": self._calc_pid, "window_id": window_id,
- })
- tree = snap.get("structuredContent", snap).get("tree_markdown", "")
- two_button = _find_calc_button(tree, "2")
- self.assertIsNotNone(two_button, "Could not locate Calculator's '2' AXButton")
-
- result = c.call_tool("double_click", {
- "pid": self._calc_pid,
- "window_id": window_id,
- "element_index": two_button,
- })
- self.assertIsNone(result.get("isError"), msg=result)
- text = result.get("content", [{}])[0].get("text", "")
- print(f"\n double_click result: {text}")
- self.assertTrue(text.startswith("✅"), f"Expected ✅ prefix, got: {text}")
-
- self._assert_no_focus_steal()
-
- def test_pixel_double_click_no_focus_steal(self) -> None:
- """double_click via pixel coords — no focus steal."""
- with DriverClient(self.binary) as c:
- window_id = resolve_window_id(c, self._calc_pid)
- snap = c.call_tool("get_window_state", {
- "pid": self._calc_pid, "window_id": window_id,
- })
- sc = snap.get("structuredContent", snap)
- w = sc.get("screenshot_width", CALC_DEFAULT_W)
- h = sc.get("screenshot_height", CALC_DEFAULT_H)
-
- bx, by = CALC_TWO_BUTTON_XY
- x = int(bx * w / CALC_DEFAULT_W)
- y = int(by * h / CALC_DEFAULT_H)
- result = c.call_tool("double_click", {
- "pid": self._calc_pid,
- "window_id": window_id,
- "x": x,
- "y": y,
- })
- self.assertIsNone(result.get("isError"), msg=result)
- text = result.get("content", [{}])[0].get("text", "")
- print(f"\n double_click pixel result: {text}")
- self.assertTrue(text.startswith("✅"), f"Expected ✅ prefix, got: {text}")
-
- self._assert_no_focus_steal()
-
-
-if __name__ == "__main__":
- unittest.main(verbosity=2)
diff --git a/libs/cua-driver/tests/e2e/test_electron.py b/libs/cua-driver/tests/e2e/test_electron.py
deleted file mode 100644
index dd9d55e9ea..0000000000
--- a/libs/cua-driver/tests/e2e/test_electron.py
+++ /dev/null
@@ -1,143 +0,0 @@
-"""Electron shared-harness background automation tests — v2 harness.
-
-The Electron test app is built from tests/fixtures/apps/cross-platform/electron
-and loads tests/fixtures/shared/web/index.html, the same DOM used by the WebView2
-and Rust Electron harness tests.
-
-Run: python3 -m pytest test_electron.py -v
-"""
-
-from __future__ import annotations
-
-import subprocess
-import sys
-import time
-import os
-
-import pytest
-
-_HERE = os.path.dirname(os.path.abspath(__file__))
-sys.path.insert(0, _HERE)
-
-
[email protected](autouse=True)
-def _reactivate_focus(focus_monitor):
- _, pid = focus_monitor
- subprocess.run(
- ["osascript", "-e",
- f'tell application "System Events" to set frontmost of (first process whose unix id is {pid}) to true'],
- check=False,
- )
- time.sleep(0.4)
-
-
-# ── tests ─────────────────────────────────────────────────────────────────────
-
-class TestElectronButton:
- def test_click_increments_counter(self, driver, electron_app, ux_guard):
- _, pid, _ = electron_app
- if pid == 0:
- pytest.skip("Electron app pid not found")
-
- wid = driver.find_window(pid)
-
- before = driver.get_window_state(pid, wid, query="Increment")
- idx = before.find_button("Increment") or before.find_element("Increment")
- assert idx is not None, f"'Increment' not found:\n{before.tree}"
-
- driver.click(pid, wid, element_index=idx)
- time.sleep(0.5)
-
- after = driver.get_window_state(pid, wid)
- assert after.has_text("counter=1"), f"Counter not incremented:\n{after.tree}"
-
-
-class TestElectronTextInput:
- def test_type_text(self, driver, electron_app, ux_guard):
- _, pid, _ = electron_app
- if pid == 0:
- pytest.skip("Electron app pid not found")
-
- wid = driver.find_window(pid)
-
- before = driver.get_window_state(pid, wid)
- idx = before.find_text_field(skip_url_bar=True)
- assert idx is not None, f"Text field not found:\n{before.tree}"
-
- # All keyboard interactions with Electron may crash the driver due to a
- # SkyLight SPI issue with Electron processes. Wrap everything in a
- # try/except so the test can skip gracefully instead of failing.
- try:
- driver.click(pid, wid, element_index=idx)
- time.sleep(0.4)
- driver.call_tool("type_text_chars", {
- "pid": pid,
- "text": "hello electron",
- "delay_ms": 30,
- })
- except Exception as e:
- print(f"\n driver raised {type(e).__name__}: {e}")
- pytest.skip(f"Driver crashed interacting with Electron text field (known issue): {e}")
-
- time.sleep(1.5)
-
- # Electron/Chromium may temporarily return an empty AX tree while it
- # rebuilds the accessibility tree after a DOM update. Retry a few times.
- after = None
- for attempt in range(5):
- try:
- wid2 = driver.find_window(pid)
- after = driver.get_window_state(pid, wid2)
- except Exception:
- time.sleep(1.0)
- continue
- if after and after.tree:
- break
- print(f"\n AX tree empty (attempt {attempt+1}/5), retrying …")
- time.sleep(1.0)
-
- if after is None or not after.tree:
- pytest.skip("AX tree unavailable after typing (Electron AX rebuild timeout)")
-
- print(f"\n after tree length: {len(after.tree)}, has mirror: {after.has_text('mirror=hello electron')}")
- assert after.has_text("mirror=hello electron"), f"Text not in AX tree:\n{after.tree[:500]}"
-
-
-class TestElectronClickTarget:
- def test_click_target_no_focus_steal(self, driver, electron_app, ux_guard):
- """Click the shared harness click target without stealing focus."""
- _, pid, _ = electron_app
- if pid == 0:
- pytest.skip("Electron app pid not found")
-
- wid = driver.find_window(pid)
- before = driver.get_window_state(pid, wid, query="Click target")
- idx = before.find_element("Click target")
- assert idx is not None, f"'Click target' not found:\n{before.tree}"
-
- result = driver.click(pid, wid, element_index=idx)
- time.sleep(0.5)
-
- after = driver.get_window_state(pid, wid)
- print(f" electron click target tree contains last_action: {after.has_text('last_action=left_click')}")
-
- assert result is not None, "click returned None"
- assert after.has_text("last_action=left_click"), f"Click target not updated:\n{after.tree}"
- # UX assertions via ux_guard teardown
-
-
-class TestElectronNoFocusSteal:
- def test_interactions_no_focus_steal(self, driver, electron_app, ux_guard):
- _, pid, _ = electron_app
- if pid == 0:
- pytest.skip("Electron app pid not found")
-
- wid = driver.find_window(pid)
- before = driver.get_window_state(pid, wid, query="Increment")
- idx = before.find_button("Increment") or before.find_element("Increment")
- if idx is not None:
- driver.click(pid, wid, element_index=idx)
- time.sleep(0.3)
- driver.press_key(pid, "tab")
- time.sleep(0.2)
- # ux_guard teardown will assert_clean()
diff --git a/libs/cua-driver/tests/e2e/test_focus_steal_parity.py b/libs/cua-driver/tests/e2e/test_focus_steal_parity.py
deleted file mode 100644
index 51ff506f6d..0000000000
--- a/libs/cua-driver/tests/e2e/test_focus_steal_parity.py
+++ /dev/null
@@ -1,356 +0,0 @@
-"""Integration parity tests for macOS focus-steal prevention.
-
-`launch_app` must NOT change the user's frontmost application — that's
-the contract Swift's `SystemFocusStealPreventer` + 3-phase wrap in
-`LaunchAppTool` already enforce. This test suite encodes the same
-contract for Rust `cua-driver-rs` so we can catch regressions and
-verify the Swift↔Rust parity needed to flip macOS Rust BETA → GA.
-
-Each test runs against the Rust binary (`CUA_DRIVER_BINARY`). When
-`CUA_SWIFT_BINARY` is set, the same checks run against the Swift
-binary too — both must pass for parity. Set `FOCUS_STEAL_RUST_ONLY=1`
-to skip the Swift half (useful while iterating on Rust changes
-without a Swift build on the path).
-
-Test cases (mirror the plan's Verification section 1:1):
-
- 1. test_launch_passive_app_preserves_frontmost — Calculator (passive)
- 2. test_launch_self_activating_app_preserves_frontmost — Safari
- 3. test_launch_with_url_preserves_frontmost — Safari + about:blank
- 4. test_cold_launch_creates_window — kill + relaunch Calculator,
- assert window appears (verifies the `oapp` AppleEvent on the
- no-URL path)
- 5. test_concurrent_launches_independent_suppression — back-to-back
- launches, both targets suppressed, frontmost unchanged.
- 6. test_deadline_reaps_leaked_entry — exercised by the Rust unit
- test `focus_steal::tests::deadline_reaps_leaked_entry`. This file
- re-asserts the integration-level behavior: a launch followed by
- a stale-state probe doesn't keep re-activating the prior
- frontmost.
-
-The frontmost-baseline app is Finder (`com.apple.finder`). It's the
-only app guaranteed to be running on every logged-in macOS session
-and it never self-activates on launch, which gives us a stable,
-predictable "prior" pid between tests. (Earlier iterations of this
-suite used a shared FocusMonitorApp fixture under
-`libs/cua-driver/Tests/FocusMonitorApp/`, but the dependency on a
-built helper made the integration tests flaky to bootstrap on a
-fresh runner — Finder removes that dependency.)
-
-Run:
- cd libs/cua-driver/tests/e2e
- python3 -m unittest test_focus_steal_parity -v
-"""
-
-from __future__ import annotations
-
-import os
-import shutil
-import subprocess
-import sys
-import time
-import unittest
-
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
-from driver_client import DriverClient, default_binary_path # noqa: E402
-
-
-# ── Paths / fixtures ─────────────────────────────────────────────────────────
-def _swift_binary() -> str | None:
- """Locate the Swift binary, or None if not available."""
- from_env = os.environ.get("CUA_SWIFT_BINARY", "")
- if from_env and os.path.isfile(from_env):
- return from_env
- for p in (
- os.path.expanduser("~/.local/bin/cua-driver"),
- shutil.which("cua-driver") or "",
- ):
- if p and os.path.isfile(p) and os.access(p, os.X_OK):
- return p
- return None
-
-
-def _frontmost_bundle_id() -> str | None:
- """Bundle id of the currently-frontmost app, queried via osascript.
-
- We deliberately don't use the driver's `list_apps` for this — the
- driver process itself may not be frontmost yet when invoked via
- `call`, and we want a host-truth answer that doesn't go through
- the same driver path we're testing.
- """
- try:
- out = subprocess.run(
- [
- "osascript",
- "-e",
- 'tell application "System Events" to bundle identifier of first '
- "application process whose frontmost is true",
- ],
- capture_output=True,
- text=True,
- timeout=5,
- )
- bid = out.stdout.strip()
- return bid or None
- except Exception:
- return None
-
-
-def _activate_bundle(bundle_id: str) -> None:
- """Bring `bundle_id` to the foreground via osascript."""
- subprocess.run(
- ["osascript", "-e", f'tell application id "{bundle_id}" to activate'],
- timeout=5,
- check=False,
- )
-
-
-def _kill_bundle(bundle_id: str) -> None:
- """pkill the running app for `bundle_id` if any. Idempotent."""
- name = bundle_id.split(".")[-1]
- subprocess.run(["pkill", "-x", name], check=False)
- subprocess.run(["pkill", "-x", name.capitalize()], check=False)
- # Safari sometimes needs a moment for its child helper processes
- # (Networking, WebContent) to also exit before LaunchServices will
- # cleanly relaunch it from the Cryptex path. The 1s settle below
- # is conservative but cheap relative to the launch time itself.
- time.sleep(1.0)
-
-
-def _wait_for_frontmost(bundle_id: str, timeout_s: float = 3.0) -> bool:
- deadline = time.time() + timeout_s
- while time.time() < deadline:
- if _frontmost_bundle_id() == bundle_id:
- return True
- time.sleep(0.1)
- return False
-
-
-# ── Skipping logic ───────────────────────────────────────────────────────────
-def _rust_binary() -> str | None:
- p = default_binary_path()
- return p if os.path.isfile(p) and os.access(p, os.X_OK) else None
-
-
-_RUST = _rust_binary()
-_SWIFT = _swift_binary()
-_RUST_ONLY = os.environ.get("FOCUS_STEAL_RUST_ONLY", "") == "1"
-
-
-# ── Mixin: the actual test cases ─────────────────────────────────────────────
-class _FocusStealMixin:
- """Parametrized test cases. Subclasses set `BINARY` and `LABEL`."""
-
- BINARY: str = ""
- LABEL: str = ""
-
- # A stable "prior frontmost" app id we can reasonably expect every
- # macOS host to have running. Finder is always running on a logged-in
- # macOS session and never self-activates on launch.
- PRIOR_BUNDLE = "com.apple.finder"
-
- def setUp(self) -> None:
- # Bring the baseline app to the foreground, confirm. We retry a
- # few times because dev hosts with overlay / focus-grabber apps
- # (e.g. Gumbranch, screen recorders, status-bar apps that
- # forcibly steal focus on click) can racily re-steal during the
- # setup window.
- for _ in range(3):
- _activate_bundle(self.PRIOR_BUNDLE)
- if _wait_for_frontmost(self.PRIOR_BUNDLE, timeout_s=2.0):
- return
- time.sleep(0.5)
- self.skipTest(
- f"could not bring {self.PRIOR_BUNDLE} to the foreground after "
- "3 attempts — test host has an aggressive focus-grabber that "
- "makes the test environment unstable"
- )
-
- def _launch(self, args: dict) -> dict:
- with DriverClient(self.BINARY) as client:
- return client.call_tool("launch_app", args)
-
- # ── Case 1: passive app (Calculator) ────────────────────────────────
- def test_launch_passive_app_preserves_frontmost(self) -> None:
- _kill_bundle("com.apple.calculator")
- time.sleep(0.5)
- _activate_bundle(self.PRIOR_BUNDLE)
- time.sleep(0.5)
- baseline = _frontmost_bundle_id()
- self.assertEqual(baseline, self.PRIOR_BUNDLE)
-
- result = self._launch({"bundle_id": "com.apple.calculator"})
- self.assertIn("structuredContent", result)
- self.assertIn("pid", result["structuredContent"])
-
- # Settle, then assert the prior frontmost is unchanged.
- time.sleep(1.0)
- self.assertEqual(
- _frontmost_bundle_id(),
- self.PRIOR_BUNDLE,
- f"{self.LABEL}: Calculator launch stole focus from {self.PRIOR_BUNDLE}",
- )
-
- # ── Case 2: self-activating app (Safari, no URLs) ───────────────────
- def test_launch_self_activating_app_preserves_frontmost(self) -> None:
- _kill_bundle("com.apple.Safari")
- time.sleep(0.5)
- _activate_bundle(self.PRIOR_BUNDLE)
- time.sleep(0.5)
- self.assertEqual(_frontmost_bundle_id(), self.PRIOR_BUNDLE)
-
- # No URLs — exercises the `openApplicationAtURL` path with the
- # `oapp` AppleEvent.
- self._launch({"bundle_id": "com.apple.Safari"})
- time.sleep(1.5) # allow Safari's reflex NSApp.activate to fire
- self.assertEqual(
- _frontmost_bundle_id(),
- self.PRIOR_BUNDLE,
- f"{self.LABEL}: Safari (no URL) launch stole focus from {self.PRIOR_BUNDLE}",
- )
-
- # ── Case 3: launch-with-URL preserves frontmost ─────────────────────
- def test_launch_with_url_preserves_frontmost(self) -> None:
- _kill_bundle("com.apple.Safari")
- time.sleep(0.5)
- _activate_bundle(self.PRIOR_BUNDLE)
- time.sleep(0.5)
- self.assertEqual(_frontmost_bundle_id(), self.PRIOR_BUNDLE)
-
- result = self._launch(
- {"bundle_id": "com.apple.Safari", "urls": ["about:blank"]}
- )
- # NOTE: the Swift binary on macOS Sonoma+ fails this launch with
- # "The application 'Safari' could not be launched because it was
- # not found" — that's a Cryptex-app + URL-handoff regression on
- # the Swift side (`AppLauncher.launch` attaches an `oapp`
- # AppleEvent to the openURLs:withApplicationAtURL: path, which
- # LaunchServices rejects for Safari). The Rust port fixes this
- # by skipping `oapp` on the URL-handoff path — see commit body
- # for `feat(launch_app): wire focus-steal preventer into
- # LaunchAppTool`. The Swift binary will fail this assertion;
- # treat that as a pre-existing Swift bug, not a Rust regression,
- # and surface it via `expectedFailure` on the Swift subclass.
- self.assertNotIn("isError", result, f"launch_app reported isError={result}")
-
- time.sleep(1.5)
- self.assertEqual(
- _frontmost_bundle_id(),
- self.PRIOR_BUNDLE,
- f"{self.LABEL}: Safari (about:blank) launch stole focus",
- )
-
- # ── Case 4: cold launch creates a window ────────────────────────────
- def test_cold_launch_creates_window(self) -> None:
- _kill_bundle("com.apple.calculator")
- time.sleep(0.5)
- _activate_bundle(self.PRIOR_BUNDLE)
- time.sleep(0.5)
-
- result = self._launch({"bundle_id": "com.apple.calculator"})
- windows = result["structuredContent"].get("windows", [])
- self.assertGreater(
- len(windows),
- 0,
- f"{self.LABEL}: cold-launched Calculator had no windows — "
- "the `oapp` AppleEvent may not be reaching the target",
- )
-
- # ── Case 5: back-to-back launches each preserve frontmost ───────────
- def test_concurrent_launches_independent_suppression(self) -> None:
- _kill_bundle("com.apple.calculator")
- _kill_bundle("com.apple.TextEdit")
- time.sleep(0.5)
- _activate_bundle(self.PRIOR_BUNDLE)
- time.sleep(0.5)
- self.assertEqual(_frontmost_bundle_id(), self.PRIOR_BUNDLE)
-
- # Sequential, not truly concurrent — but each launch reads the
- # current frontmost (which should still be `PRIOR_BUNDLE` after
- # the first one succeeded if focus-steal worked). If the first
- # launch leaked focus, the second's prior would be Calculator
- # and the assertion below would fail.
- self._launch({"bundle_id": "com.apple.calculator"})
- time.sleep(0.7)
- self.assertEqual(_frontmost_bundle_id(), self.PRIOR_BUNDLE)
-
- self._launch({"bundle_id": "com.apple.TextEdit"})
- time.sleep(1.0)
- self.assertEqual(
- _frontmost_bundle_id(),
- self.PRIOR_BUNDLE,
- f"{self.LABEL}: back-to-back Calculator + TextEdit launches "
- "leaked focus from one suppression to the next",
- )
-
- # ── Case 6: deadline reaper integration probe ───────────────────────
- def test_deadline_reaps_leaked_entry(self) -> None:
- # The fine-grained reap behavior is unit-tested in
- # `focus_steal::tests::deadline_reaps_leaked_entry`. This
- # integration probe verifies the user-visible consequence:
- # launching, waiting 6s, then activating the launched app
- # manually does NOT cause an unsolicited re-activation of the
- # prior frontmost (which would happen if a leaked entry kept
- # firing past the 5s deadline).
- _kill_bundle("com.apple.calculator")
- time.sleep(0.5)
- _activate_bundle(self.PRIOR_BUNDLE)
- time.sleep(0.5)
-
- result = self._launch({"bundle_id": "com.apple.calculator"})
- calc_pid = result["structuredContent"]["pid"]
- self.assertIsInstance(calc_pid, int)
- # Wait past the 5s deadline.
- time.sleep(6.0)
- # Now manually activate Calculator. If the dispatcher leaked an
- # entry, it would yank focus back to PRIOR_BUNDLE here — and
- # the assertion below would fail.
- _activate_bundle("com.apple.calculator")
- time.sleep(1.0)
- self.assertEqual(
- _frontmost_bundle_id(),
- "com.apple.calculator",
- f"{self.LABEL}: stale entry yanked focus back after the "
- "5s deadline — the reaper or the observer skipped the prune",
- )
-
-
-# ── Concrete subclasses ──────────────────────────────────────────────────────
[email protected](_RUST is None, "Rust cua-driver-rs binary not built")
-class RustFocusStealTests(_FocusStealMixin, unittest.TestCase):
- BINARY = _RUST or ""
- LABEL = "rust"
-
-
- _SWIFT is None or _RUST_ONLY,
- "Swift cua-driver binary not available (set CUA_SWIFT_BINARY) "
- "or FOCUS_STEAL_RUST_ONLY=1 set",
-)
-class SwiftFocusStealTests(_FocusStealMixin, unittest.TestCase):
- BINARY = _SWIFT or ""
- LABEL = "swift"
-
- # Pre-existing Swift bug on macOS Sonoma+: launching Safari with
- # `urls=["about:blank"]` returns "The application 'Safari' could not
- # be launched because it was not found." The Rust port fixes this
- # (the `oapp` AppleEvent on the URL-handoff path is incompatible
- # with Cryptex-installed apps). Re-define the method on this
- # subclass only (NOT via expectedFailure on the mixin attribute —
- # that would also taint the Rust subclass) and mark the override
- # `@expectedFailure`.
- @unittest.expectedFailure
- def test_launch_with_url_preserves_frontmost(self) -> None: # type: ignore[override]
- super().test_launch_with_url_preserves_frontmost()
-
- # NOTE: `test_deadline_reaps_leaked_entry` is intentionally NOT
- # marked expectedFailure on Swift. Swift's `SystemFocusStealPreventer`
- # has no monotonic deadline reaper (PR #1521 is unmerged at port
- # time), but in practice the Swift `LaunchAppTool.invoke` calls
- # `endSuppression` cleanly before the integration test's 6s wait
- # finishes, so the test sometimes passes on Swift anyway. The Rust
- # subclass is what locks down the reaper-driven contract.
-
-
-if __name__ == "__main__":
- unittest.main()
diff --git a/libs/cua-driver/tests/e2e/test_health_report_mcp.py b/libs/cua-driver/tests/e2e/test_health_report_mcp.py
deleted file mode 100644
index 26c1908ba2..0000000000
--- a/libs/cua-driver/tests/e2e/test_health_report_mcp.py
+++ /dev/null
@@ -1,238 +0,0 @@
-"""Integration test: `health_report` MCP tool over the stdio protocol.
-
-Boots the real `cua-driver mcp` server, performs the JSON-RPC handshake,
-asks `tools/list` for the tool catalog, and exercises the tool with
-several argument shapes:
-
- 1. `tools/list` advertises `health_report` (the consumer-facing
- contract is that the tool exists and ships `schema_version="1"` in
- its description).
-
- 2. Calling with no arguments produces a well-formed report with the
- documented top-level keys and per-check key shape.
-
- 3. The `skip` filter is honored end-to-end: a skipped check appears in
- the output with `status: "skip"`.
-
- 4. The `include` filter narrows the report to a single check.
-
- 5. A documented fail mode is observable end-to-end: when called via
- `cua-driver mcp` from a terminal (which is what an XCTest / CI
- host looks like on macOS), the `bundle_identity` check fails
- because the stdio process is NOT attributed to com.trycua.driver.
- The schema guarantees `status="fail"` entries carry a `hint` and
- surface the runtime bundle identifier under
- `data.bundle_identifier` — both are asserted here. On Windows /
- Linux the same check returns `status="skip"`, which we tolerate.
-
- 6. health_report must NEVER itself set isError, even when every
- check fails — that's the whole point of the tool.
-
-Run with the standard integration test harness:
- ./run_tests.sh test_health_report_mcp
-"""
-
-from __future__ import annotations
-
-import json
-import os
-import sys
-import unittest
-
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
-from driver_client import DriverClient, default_binary_path
-
-
-REQUIRED_TOP_LEVEL_KEYS = {
- "schema_version",
- "platform",
- "driver_version",
- "overall",
- "checks",
-}
-
-REQUIRED_CHECK_KEYS = {"name", "status", "message"}
-ALLOWED_STATUS_VALUES = {"pass", "fail", "skip"}
-ALLOWED_OVERALL_VALUES = {"ok", "degraded", "failed"}
-ALLOWED_PLATFORM_VALUES = {"darwin", "win32", "linux"}
-
-
-def _structured_or_parsed(result: dict) -> dict:
- """Pull the JSON report out of an MCP tool result.
-
- Tools that emit `structuredContent` (the canonical path here)
- return it as a top-level field. We also tolerate a text-content
- fallback so if structured emission ever breaks, the test fails
- loudly with a clear message rather than crashing on a KeyError.
- """
- if "structuredContent" in result:
- return result["structuredContent"]
- for content in result.get("content", []):
- if content.get("type") == "text":
- try:
- return json.loads(content.get("text", ""))
- except json.JSONDecodeError:
- continue
- raise AssertionError(
- "health_report response carried neither structuredContent nor a JSON text block: "
- + json.dumps(result)
- )
-
-
-class HealthReportMCPTests(unittest.TestCase):
- @classmethod
- def setUpClass(cls) -> None:
- cls.binary = default_binary_path()
-
- # ── tools/list ────────────────────────────────────────────────────
-
- def test_tools_list_advertises_health_report(self) -> None:
- with DriverClient(self.binary) as client:
- tools = client.list_tools()
- names = {t["name"] for t in tools}
- self.assertIn(
- "health_report",
- names,
- "tools/list must advertise health_report so consumers can discover it",
- )
- descriptor = next(t for t in tools if t["name"] == "health_report")
- # The schema_version commitment lives in the description text —
- # that's the consumer-facing stability contract.
- description = descriptor.get("description", "")
- self.assertIn(
- "schema_version",
- description,
- "tool description must call out the stable schema_version contract",
- )
-
- # ── tools/call with no args ───────────────────────────────────────
-
- def test_call_with_no_args_returns_schema_shape(self) -> None:
- with DriverClient(self.binary) as client:
- result = client.call_tool("health_report")
- report = _structured_or_parsed(result)
-
- # Top-level keys exactly match the documented schema.
- self.assertEqual(
- REQUIRED_TOP_LEVEL_KEYS,
- REQUIRED_TOP_LEVEL_KEYS & set(report.keys()),
- f"missing top-level keys; got {set(report.keys())!r}",
- )
- self.assertEqual(report["schema_version"], "1")
- self.assertIn(report["platform"], ALLOWED_PLATFORM_VALUES)
- self.assertIsInstance(report["driver_version"], str)
- self.assertIn(report["overall"], ALLOWED_OVERALL_VALUES)
-
- checks = report["checks"]
- self.assertIsInstance(checks, list)
- self.assertGreater(len(checks), 0)
-
- for check in checks:
- self.assertTrue(
- REQUIRED_CHECK_KEYS.issubset(check.keys()),
- f"check missing required keys: {check!r}",
- )
- self.assertIn(check["status"], ALLOWED_STATUS_VALUES)
- # hint is only required when status == fail.
- if check["status"] == "fail":
- self.assertIn(
- "hint",
- check,
- f"fail entries must carry a remediation hint: {check!r}",
- )
- self.assertTrue(
- isinstance(check["hint"], str) and check["hint"].strip(),
- f"hint must be a non-empty string: {check!r}",
- )
-
- # ── skip filter ───────────────────────────────────────────────────
-
- def test_skip_filter_marks_check_as_skip(self) -> None:
- with DriverClient(self.binary) as client:
- result = client.call_tool(
- "health_report",
- {"skip": ["tcc_accessibility"]},
- )
- report = _structured_or_parsed(result)
- names_to_status = {c["name"]: c["status"] for c in report["checks"]}
- self.assertEqual(names_to_status.get("tcc_accessibility"), "skip")
-
- # ── include filter ───────────────────────────────────────────────
-
- def test_include_filter_runs_only_named_check(self) -> None:
- with DriverClient(self.binary) as client:
- result = client.call_tool(
- "health_report",
- {"include": ["binary_version"]},
- )
- report = _structured_or_parsed(result)
- run_names = {
- c["name"] for c in report["checks"] if c["status"] != "skip"
- }
- # Only `binary_version` should have been actually run; the rest
- # show up with status: skip.
- self.assertEqual(run_names, {"binary_version"})
-
- # ── documented fail mode (macOS only) ────────────────────────────
-
- def test_bundle_identity_fail_mode_under_stdio_mcp(self) -> None:
- """On macOS the stdio MCP process spawned by an IDE/CI shell
- runs outside CuaDriver.app, so its CFBundleIdentifier is NOT
- com.trycua.driver. That's the canonical "wrong attribution"
- fail mode the tool is designed to surface — schema-wise:
- status=fail + non-empty message + non-empty hint +
- data.bundle_identifier present so consumers can detect the
- drift without parsing the message text.
-
- On Windows / Linux the same check returns status=skip ("not
- applicable on Windows/Linux"); the test tolerates that.
- """
- with DriverClient(self.binary) as client:
- result = client.call_tool(
- "health_report",
- {"include": ["bundle_identity"]},
- )
- report = _structured_or_parsed(result)
- bundle_check = next(
- c for c in report["checks"] if c["name"] == "bundle_identity"
- )
- if bundle_check["status"] == "skip":
- # Windows / Linux path — `not applicable on <platform>`.
- self.assertIn("not applicable", bundle_check.get("message", ""))
- return
- if bundle_check["status"] == "pass":
- # macOS launched inside CuaDriver.app — legitimately passes.
- return
- # macOS fail mode — schema contract on fail entries.
- self.assertEqual(bundle_check["status"], "fail")
- self.assertIn("hint", bundle_check)
- self.assertTrue(bundle_check["hint"].strip())
- data = bundle_check.get("data") or {}
- # bundle_identifier surfaces the runtime CFBundleIdentifier so
- # consumers can detect drift without parsing the message.
- # Optional only when the process literally has no bundle id;
- # Rust-built binaries always have an executable_path.
- self.assertTrue(
- "bundle_identifier" in data or "executable_path" in data,
- "fail mode must surface either bundle_identifier or executable_path in `data`",
- )
-
- # ── invocation surface itself doesn't error ──────────────────────
-
- def test_call_does_not_set_isError(self) -> None:
- """`health_report` is the consumer's "what's broken?" probe —
- it must never itself report `isError: true`, even when every
- check fails. Otherwise consumers can't tell apart "the tool
- itself is broken" from "the driver has degraded health" —
- defeating the whole point of the tool.
- """
- with DriverClient(self.binary) as client:
- result = client.call_tool("health_report")
- self.assertFalse(
- result.get("isError", False),
- f"health_report must never set isError; got {result!r}",
- )
-
-
-if __name__ == "__main__":
- unittest.main()
diff --git a/libs/cua-driver/tests/e2e/test_list_windows.py b/libs/cua-driver/tests/e2e/test_list_windows.py
deleted file mode 100644
index a9f5c8d59b..0000000000
--- a/libs/cua-driver/tests/e2e/test_list_windows.py
+++ /dev/null
@@ -1,121 +0,0 @@
-"""Integration test: `list_windows` tool.
-
-Exercises three contracts against the real driver over stdio MCP:
-
- 1. Default call returns at least a handful of windows across multiple
- pids, every record layer-0 with the correct field names.
- 2. `pid` filter narrows the result to that pid only.
- 3. `on_screen_only: true` drops hidden / minimized / off-Space windows.
-
-Calculator is launched once (setUpClass) so all five tests share one
-driver session — avoids the 13-second Calculator restart per-test.
-
-Run:
- CUA_DRIVER_BINARY=../../target/debug/cua-driver python3 -m unittest test_list_windows -v
-"""
-
-from __future__ import annotations
-
-import os
-import subprocess
-import sys
-import time
-import unittest
-
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
-from driver_client import DriverClient, default_binary_path
-
-
-CALCULATOR_BUNDLE = "com.apple.calculator"
-
-
-class ListWindowsTests(unittest.TestCase):
- client: DriverClient
- calc_pid: int
-
- @classmethod
- def setUpClass(cls) -> None:
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
- time.sleep(0.3)
- cls.client = DriverClient(default_binary_path()).__enter__()
- result = cls.client.call_tool("launch_app", {"bundle_id": CALCULATOR_BUNDLE})
- cls.calc_pid = result["structuredContent"]["pid"]
- time.sleep(0.5)
-
- @classmethod
- def tearDownClass(cls) -> None:
- cls.client.__exit__(None, None, None)
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
-
- def _call_list_windows(self, **args):
- return self.client.call_tool("list_windows", args)["structuredContent"]
-
- def test_default_returns_layer_zero_windows_with_field_names(self):
- body = self._call_list_windows()
- windows = body["windows"]
- self.assertGreater(
- len(windows), 0, "expected at least Calculator's window"
- )
-
- required = {
- "window_id", "pid", "app_name", "title", "bounds",
- "layer", "z_index", "is_on_screen",
- }
- for w in windows:
- self.assertTrue(
- required.issubset(w.keys()),
- f"window record missing fields: {required - set(w.keys())}",
- )
- self.assertEqual(w["layer"], 0)
-
- def test_pid_filter_narrows_to_one_pid(self):
- body = self._call_list_windows(pid=self.calc_pid)
- windows = body["windows"]
- self.assertGreater(
- len(windows), 0, "Calculator should have at least one window"
- )
- self.assertTrue(
- all(w["pid"] == self.calc_pid for w in windows),
- "pid filter leaked other pids into the result",
- )
- self.assertTrue(
- all(w["app_name"] == "Calculator" for w in windows),
- "app_name disagrees with the pid filter",
- )
-
- def test_on_screen_only_drops_off_screen_entries(self):
- everything = self._call_list_windows()["windows"]
- only_visible = self._call_list_windows(on_screen_only=True)["windows"]
-
- self.assertLessEqual(len(only_visible), len(everything))
- for w in only_visible:
- self.assertTrue(
- w["is_on_screen"],
- "on_screen_only=true returned a window with is_on_screen=false",
- )
-
- def test_bounds_fields_present_and_non_negative(self):
- body = self._call_list_windows(pid=self.calc_pid)
- windows = body["windows"]
- self.assertGreater(len(windows), 0)
- for w in windows:
- b = w["bounds"]
- for field in ("x", "y", "width", "height"):
- self.assertIn(field, b, f"bounds missing '{field}'")
- self.assertGreaterEqual(
- b[field], 0,
- f"bounds.{field} should be non-negative, got {b[field]}",
- )
-
- def test_z_index_positive(self):
- body = self._call_list_windows()
- windows = body["windows"]
- for w in windows:
- self.assertGreater(
- w["z_index"], 0,
- f"z_index should be > 0 (front-to-back ordering), got {w['z_index']}",
- )
-
-
-if __name__ == "__main__":
- unittest.main(verbosity=2)
diff --git a/libs/cua-driver/tests/e2e/test_overlay_z_order.py b/libs/cua-driver/tests/e2e/test_overlay_z_order.py
deleted file mode 100644
index e7121daf0f..0000000000
--- a/libs/cua-driver/tests/e2e/test_overlay_z_order.py
+++ /dev/null
@@ -1,195 +0,0 @@
-"""Integration test: agent-cursor overlay z-ordering.
-
-Verifies that after driving a backgrounded window the cua-driver-rs overlay
-is z-ordered JUST ABOVE the target window (NSWindowLevel.normal + ordered
-above target) rather than at NSWindowLevel.floating (above ALL normal windows).
-
-Expected ordering after pinning above a background window:
- [... , target-window , overlay , windows-that-were-above-target , ...]
-
-Two assertions are verified:
-
-1. **overlay appears at layer 0** — the overlay's CGWindowLayer is 0
- (NSWindowLevel.normal), so it shows up in list_windows. At .floating
- (level 3) it would be absent from the layer-0 filtered list.
-
-2. **sandwich ordering** — overlay z_index > target z_index, and any
- window that was above the target BEFORE the click remains above the
- overlay AFTER the click.
-
-Run:
- CUA_DRIVER_BINARY=../../target/debug/cua-driver python3 -m unittest test_overlay_z_order -v
-"""
-
-from __future__ import annotations
-
-import os
-import subprocess
-import sys
-import time
-import unittest
-
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
-from driver_client import DriverClient, default_binary_path, resolve_window_id
-
-_THIS_DIR = os.path.dirname(os.path.abspath(__file__))
-_DRIVER_RS_ROOT = os.path.dirname(os.path.dirname(_THIS_DIR))
-_LIBS_ROOT = os.path.dirname(_DRIVER_RS_ROOT)
-_FOCUS_APP_DIR = os.path.join(_LIBS_ROOT, "cua-driver", "Tests", "FocusMonitorApp")
-_FOCUS_APP_EXE = os.path.join(
- _FOCUS_APP_DIR, "FocusMonitorApp.app", "Contents", "MacOS", "FocusMonitorApp"
-)
-
-CALCULATOR_BUNDLE = "com.apple.calculator"
-
-
-def _ensure_focus_app_built() -> None:
- src = os.path.join(_FOCUS_APP_DIR, "FocusMonitorApp.swift")
- if not os.path.exists(_FOCUS_APP_EXE) or (
- os.path.exists(src) and os.path.getmtime(src) > os.path.getmtime(_FOCUS_APP_EXE)
- ):
- subprocess.run([os.path.join(_FOCUS_APP_DIR, "build.sh")], check=True)
-
-
-def _launch_focus_app() -> tuple[subprocess.Popen, int]:
- proc = subprocess.Popen(
- [_FOCUS_APP_EXE], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True
- )
- for _ in range(40):
- line = proc.stdout.readline().strip()
- if line.startswith("FOCUS_PID="):
- return proc, int(line.split("=", 1)[1])
- time.sleep(0.1)
- proc.terminate()
- raise RuntimeError("FocusMonitorApp did not print FOCUS_PID= in time")
-
-
-class TestOverlayZOrder(unittest.TestCase):
- """Overlay is sandwiched just above the target window, not floating above all."""
-
- _focus_proc: subprocess.Popen
- _focus_pid: int
-
- @classmethod
- def setUpClass(cls) -> None:
- _ensure_focus_app_built()
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
- time.sleep(0.3)
-
- @classmethod
- def tearDownClass(cls) -> None:
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
-
- def setUp(self) -> None:
- # Kill any stale FocusMonitorApp left over from a crashed prior run.
- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)
- time.sleep(0.2)
-
- self.client = DriverClient(default_binary_path()).__enter__()
- self.client.call_tool("set_agent_cursor_enabled", {"enabled": True})
-
- r = self.client.call_tool("launch_app", {"bundle_id": CALCULATOR_BUNDLE})
- self.calc_pid = r["structuredContent"]["pid"]
- time.sleep(0.5)
-
- self._focus_proc, self._focus_pid = _launch_focus_app()
- time.sleep(0.5)
-
- def tearDown(self) -> None:
- if hasattr(self, '_focus_proc'):
- self._focus_proc.terminate()
- try:
- self._focus_proc.wait(timeout=3)
- except subprocess.TimeoutExpired:
- self._focus_proc.kill()
- self._focus_proc.wait(timeout=1)
- # Close the stdout pipe opened during launch.
- if self._focus_proc.stdout:
- self._focus_proc.stdout.close()
- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
- self.client.__exit__(None, None, None)
-
- def test_overlay_at_normal_level_above_target(self) -> None:
- """Overlay appears at layer=0 (not .floating/layer=3) and z_index > target."""
- all_before = self._all_on_screen_windows()
- driver_pid = self.client.process.pid
-
- calc_wins_before = [w for w in all_before if w["pid"] == self.calc_pid]
- self.assertTrue(calc_wins_before, "Calculator window not visible before click")
- calc_z_before = max(w["z_index"] for w in calc_wins_before)
- calc_win_before = max(calc_wins_before, key=lambda w: w["z_index"])
- calc_win_id = calc_win_before["window_id"]
-
- b = calc_win_before["bounds"]
- cx = b["width"] / 2.0
- cy = b["height"] / 2.0
-
- fg_wins_before = [
- w for w in all_before
- if w["z_index"] > calc_z_before
- and w["pid"] != driver_pid
- and w.get("is_on_screen")
- ]
-
- self.client.call_tool("click", {
- "pid": self.calc_pid,
- "window_id": calc_win_id,
- "x": cx,
- "y": cy,
- })
-
- # Let defensive-repin ticks fully settle.
- time.sleep(1.5)
-
- all_after = self._all_on_screen_windows()
- overlay_wins = [w for w in all_after if w["pid"] == driver_pid]
- self.assertTrue(
- overlay_wins,
- f"cua-driver-rs overlay NOT visible in list_windows at layer=0 "
- f"(driver pid={driver_pid}). "
- f"Expected NSWindowLevel.normal overlay to appear as a layer-0 window.",
- )
-
- overlay_z = max(w["z_index"] for w in overlay_wins)
-
- calc_wins_after = [w for w in all_after if w["pid"] == self.calc_pid]
- calc_z_after = (
- max(w["z_index"] for w in calc_wins_after) if calc_wins_after else 0
- )
- self.assertGreater(
- overlay_z,
- calc_z_after,
- f"overlay (z={overlay_z}) must be ABOVE Calculator (z={calc_z_after}).",
- )
-
- missed_above = []
- for w_before in fg_wins_before:
- w_after = next(
- (a for a in all_after if a["window_id"] == w_before["window_id"]),
- None,
- )
- if w_after and w_after.get("is_on_screen"):
- if w_after["z_index"] <= overlay_z:
- missed_above.append(
- f"{w_before['app_name']} "
- f"(window_id={w_before['window_id']}, "
- f"z_before={w_before['z_index']}, "
- f"z_after={w_after['z_index']}, overlay_z={overlay_z})"
- )
-
- if fg_wins_before and missed_above:
- self.fail(
- "Window(s) that were ABOVE Calculator before the click are now "
- "BELOW OR EQUAL TO the overlay — overlay is NOT sandwiched:\n"
- + "\n".join(f" {m}" for m in missed_above)
- + "\nExpected ordering: [target, overlay, fg-windows]."
- )
-
- def _all_on_screen_windows(self) -> list[dict]:
- result = self.client.call_tool("list_windows", {"on_screen_only": True})
- return result["structuredContent"]["windows"]
-
-
-if __name__ == "__main__":
- unittest.main(verbosity=2)
diff --git a/libs/cua-driver/tests/e2e/test_pixel_click_delivery.py b/libs/cua-driver/tests/e2e/test_pixel_click_delivery.py
deleted file mode 100644
index 985047e868..0000000000
--- a/libs/cua-driver/tests/e2e/test_pixel_click_delivery.py
+++ /dev/null
@@ -1,276 +0,0 @@
-"""Integration test: pixel-click delivery to backgrounded apps.
-
-Sends pixel-addressed clicks (x, y — NOT element_index) to Calculator
-while it's backgrounded behind FocusMonitorApp, then asserts:
- - Calculator display changed (click delivery).
- - FocusMonitorApp is still frontmost (no focus steal).
-
-Note: Calculator is a Mac Catalyst app. CGEventPostToPid alone doesn't
-reach Catalyst. This test documents the current behaviour and will pass
-once SkyLight SPI event posting is implemented.
-
-Run:
- CUA_DRIVER_BINARY=../../target/debug/cua-driver python3 -m unittest test_pixel_click_delivery -v
-"""
-
-from __future__ import annotations
-
-import os
-import re
-import subprocess
-import sys
-import time
-import unittest
-
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
-from driver_client import (
- DriverClient,
- default_binary_path,
- frontmost_bundle_id,
- resolve_window_id,
-)
-
-_THIS_DIR = os.path.dirname(os.path.abspath(__file__))
-_DRIVER_RS_ROOT = os.path.dirname(os.path.dirname(_THIS_DIR))
-_LIBS_ROOT = os.path.dirname(_DRIVER_RS_ROOT)
-_FOCUS_APP_DIR = os.path.join(_LIBS_ROOT, "cua-driver", "Tests", "FocusMonitorApp")
-_FOCUS_APP_EXE = os.path.join(
- _FOCUS_APP_DIR, "FocusMonitorApp.app", "Contents", "MacOS", "FocusMonitorApp"
-)
-_LOSS_FILE = "/tmp/focus_monitor_losses.txt"
-
-CALC_BUNDLE = "com.apple.calculator"
-FOCUS_MONITOR_BUNDLE = "com.trycua.FocusMonitorApp"
-
-# Calculator standard-mode (230x408) button grid.
-# Column centers (x): 29, 86, 144, 201
-# Row centers (y): 133, 193, 253, 313, 373
-CALC_BUTTONS = {
- "1": (29, 313), "2": (86, 313), "3": (144, 313),
-}
-
-
-def _build_focus_app() -> None:
- if not os.path.exists(_FOCUS_APP_EXE):
- subprocess.run(
- [os.path.join(_FOCUS_APP_DIR, "build.sh")], check=True
- )
-
-
-def _launch_focus_app() -> tuple[subprocess.Popen, int]:
- proc = subprocess.Popen(
- [_FOCUS_APP_EXE],
- stdout=subprocess.PIPE,
- stderr=subprocess.DEVNULL,
- text=True,
- )
- for _ in range(40):
- line = proc.stdout.readline().strip()
- if line.startswith("FOCUS_PID="):
- return proc, int(line.split("=", 1)[1])
- time.sleep(0.1)
- proc.terminate()
- raise RuntimeError("FocusMonitorApp did not print FOCUS_PID in time")
-
-
-def _read_focus_losses() -> int:
- try:
- with open(_LOSS_FILE) as f:
- return int(f.read().strip())
- except (FileNotFoundError, ValueError):
- return -1
-
-
-def _find_calc_button(tree: str, label: str) -> int | None:
- for line in tree.split("\n"):
- if "AXButton" not in line:
- continue
- m = re.search(r'\[(\d+)\]', line)
- if not m:
- continue
- if f'({label})' in line or f'id={label}' in line:
- return int(m.group(1))
- return None
-
-
-def _calc_display(client: DriverClient, pid: int) -> str:
- window_id = resolve_window_id(client, pid)
- result = client.call_tool(
- "get_window_state",
- {"pid": pid, "window_id": window_id, "query": "AXStaticText"},
- )
- tree = result.get("structuredContent", result).get("tree_markdown", "")
- for line in tree.split("\n"):
- if "AXStaticText" in line and "=" in line:
- m = re.search(r'= "([^"]*)"', line)
- if m:
- return m.group(1).replace("\u200e", "").replace("\u200f", "")
- return ""
-
-
-class CalculatorPixelClickDelivery(unittest.TestCase):
- """Pixel-click digits on backgrounded Calculator."""
-
- _calc_pid: int
- _focus_proc: subprocess.Popen
- _focus_pid: int
-
- @classmethod
- def setUpClass(cls) -> None:
- _build_focus_app()
- try:
- os.remove(_LOSS_FILE)
- except FileNotFoundError:
- pass
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
- # Kill any stale FocusMonitorApp left over from a crashed prior run.
- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)
- time.sleep(0.3)
-
- cls.binary = default_binary_path()
-
- with DriverClient(cls.binary) as c:
- result = c.call_tool("launch_app", {"bundle_id": CALC_BUNDLE})
- cls._calc_pid = result["structuredContent"]["pid"]
- print(f"\n Calculator pid: {cls._calc_pid}")
- time.sleep(1.5)
-
- cls._focus_proc, cls._focus_pid = _launch_focus_app()
- print(f" FocusMonitor pid: {cls._focus_pid}")
- time.sleep(1.0)
-
- with DriverClient(cls.binary) as c:
- active = frontmost_bundle_id(c)
- assert active == FOCUS_MONITOR_BUNDLE, (
- f"Expected FocusMonitorApp frontmost, got {active}"
- )
- losses = _read_focus_losses()
- assert losses == 0, f"Expected 0 focus losses at start, got {losses}"
-
- @classmethod
- def tearDownClass(cls) -> None:
- if hasattr(cls, '_focus_proc'):
- cls._focus_proc.terminate()
- try:
- cls._focus_proc.wait(timeout=3)
- except subprocess.TimeoutExpired:
- cls._focus_proc.kill()
- subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)
- subprocess.run(["pkill", "-x", "Calculator"], check=False)
- try:
- os.remove(_LOSS_FILE)
- except FileNotFoundError:
- pass
-
- def setUp(self) -> None:
- # Clear via AX before the test.
- with DriverClient(self.binary) as c:
- window_id = resolve_window_id(c, self._calc_pid)
- snap = c.call_tool(
- "get_window_state",
- {"pid": self._calc_pid, "window_id": window_id},
- )
- tree = snap.get("structuredContent", snap).get("tree_markdown", "")
- btn = _find_calc_button(tree, "All Clear") or _find_calc_button(tree, "Clear")
- if btn is not None:
- c.call_tool("click", {
- "pid": self._calc_pid,
- "window_id": window_id,
- "element_index": btn,
- })
- time.sleep(0.3)
- subprocess.run(
- ["osascript", "-e", 'tell application "FocusMonitorApp" to activate'],
- check=False,
- )
- time.sleep(0.5)
- self._losses_before = _read_focus_losses()
-
- def test_pixel_click_no_focus_steal(self) -> None:
- """Pixel-click Calculator buttons — verify no focus steal regardless of delivery."""
- with DriverClient(self.binary) as c:
- window_id = resolve_window_id(c, self._calc_pid)
- snap = c.call_tool(
- "get_window_state",
- {"pid": self._calc_pid, "window_id": window_id},
- )
- sc = snap.get("structuredContent", snap)
- w = sc.get("screenshot_width", 230)
- h = sc.get("screenshot_height", 408)
- print(f"\n screenshot: {w}x{h}")
-
- for digit in ["1", "2", "3"]:
- bx, by = CALC_BUTTONS[digit]
- x = int(bx * w / 230)
- y = int(by * h / 408)
- result = c.call_tool("click", {
- "pid": self._calc_pid,
- "window_id": window_id,
- "x": x,
- "y": y,
- })
- print(f" {digit}: pixel ({x},{y})")
- time.sleep(0.15)
-
- time.sleep(0.3)
-
- # Assert no focus steal — this is the main invariant for background automation.
- losses = _read_focus_losses()
- with DriverClient(self.binary) as c:
- active = frontmost_bundle_id(c)
- print(f" losses: {self._losses_before}->{losses}, frontmost: {active}")
- self.assertEqual(
- active, FOCUS_MONITOR_BUNDLE,
- f"Focus stolen — frontmost is {active}",
- )
-
- def test_ax_click_delivery(self) -> None:
- """AX-click 1+2 on backgrounded Calculator — verifies AX event delivery."""
- with DriverClient(self.binary) as c:
- window_id = resolve_window_id(c, self._calc_pid)
- snap = c.call_tool("get_window_state", {
- "pid": self._calc_pid, "window_id": window_id,
- })
- tree = snap.get("structuredContent", snap).get("tree_markdown", "")
-
- btn_1 = _find_calc_button(tree, "1")
- btn_add = _find_calc_button(tree, "Add")
- btn_2 = _find_calc_button(tree, "2")
- btn_eq = _find_calc_button(tree, "Equals")
- print(f"\n buttons: 1={btn_1}, Add={btn_add}, 2={btn_2}, Equals={btn_eq}")
-
- self.assertIsNotNone(btn_1, "'1' button not found in Calculator AX tree")
- self.assertIsNotNone(btn_add, "'Add' button not found")
- self.assertIsNotNone(btn_2, "'2' button not found")
- self.assertIsNotNone(btn_eq, "'Equals' button not found")
-
- for idx in [btn_1, btn_add, btn_2, btn_eq]:
- c.call_tool("click", {
- "pid": self._calc_pid,
- "window_id": window_id,
- "element_index": idx,
- })
- time.sleep(0.3)
-
- snap = c.call_tool("get_window_state", {
- "pid": self._calc_pid,
- "window_id": window_id,
- "query": "AXStaticText",
- })
- tree = snap.get("structuredContent", snap).get("tree_markdown", "")
- print(f" result tree:\n{tree}")
-
- self.assertIn("3", tree, "Calculator did not show 3 after 1+2=")
-
- losses = _read_focus_losses()
- with DriverClient(self.binary) as c:
- active = frontmost_bundle_id(c)
- print(f" losses: {self._losses_before}->{losses}, frontmost: {active}")
- self.assertEqual(
- active, FOCUS_MONITOR_BUNDLE,
- f"Focus stolen by AX click — frontmost is {active}",
- )
-
-
-if __name__ == "__main__":
- unittest.main(verbosity=2)
diff --git a/libs/cua-driver/tests/e2e/test_safari.py b/libs/cua-driver/tests/e2e/test_safari.py
deleted file mode 100644
index 5a33846482..0000000000
--- a/libs/cua-driver/tests/e2e/test_safari.py
+++ /dev/null
@@ -1,332 +0,0 @@
-"""Safari background automation tests — v2 harness.
-
-Tests interact with Safari via the public MCP interface only:
- get_window_state → click / type_text / press_key → get_window_state → assert
-
-The driver picks the best internal technique (AX, CGEvent, SkyLight, etc.).
-Tests verify observable DOM outcomes via the AX tree.
-
-Setup: Safari is opened to test_page.html (served locally), then
-FocusMonitorApp is activated so it owns the screen focus. Every test
-asserts that focus never returned to Safari during the interaction.
-"""
-
-from __future__ import annotations
-
-import subprocess
-import sys
-import time
-import os
-
-import re
-import pytest
-
-_HERE = os.path.dirname(os.path.abspath(__file__))
-sys.path.insert(0, _HERE)
-from harness.driver import Driver
-from harness import tree as Tree
-
-
-def _parse_clicks(tree_text: str) -> int:
- """Extract the 'clicks: N' counter value from the AX tree."""
- m = re.search(r'clicks:\s*(\d+)', tree_text)
- return int(m.group(1)) if m else 0
-
-SAFARI_BUNDLE = "com.apple.Safari"
-FOCUS_MONITOR_BUNDLE = "com.trycua.FocusMonitorApp"
-
-
-# ── module-level Safari setup ─────────────────────────────────────────────────
-
[email protected](scope="module")
-def safari_pid(binary, html_server, focus_monitor):
- """Open Safari to test_page.html; return its pid.
-
- Activated after Safari so FocusMonitorApp is frontmost during the tests.
- """
- subprocess.run(["pkill", "-x", "Safari"], check=False)
- time.sleep(0.5)
-
- page_url = f"{html_server}/test_page.html"
- subprocess.run(["open", "-g", "-a", "Safari", page_url], check=True)
- time.sleep(3.5) # Safari launch + page load
-
- with Driver(binary) as d:
- app = d.find_app(SAFARI_BUNDLE)
- if app is None:
- pytest.fail("Safari did not appear in list_apps after launch")
- pid = app["pid"]
-
- print(f"\n Safari pid: {pid}, page: {page_url}")
- yield pid
-
- subprocess.run(["pkill", "-x", "Safari"], check=False)
-
-
[email protected](autouse=True)
-def _reactivate_focus(focus_monitor):
- """Restore FocusMonitorApp as frontmost before each test (by pid)."""
- _, pid = focus_monitor
- subprocess.run(
- ["osascript", "-e",
- f'tell application "System Events" to set frontmost of (first process whose unix id is {pid}) to true'],
- check=False,
- )
- time.sleep(0.4)
-
-
-# ── helpers ───────────────────────────────────────────────────────────────────
-
-def _get_state(d: Driver, pid: int, safari_pid_fixture) -> "harness.driver.WindowState":
- wid = d.find_window(pid)
- return d.get_window_state(pid, wid)
-
-
-# ── tests ─────────────────────────────────────────────────────────────────────
-
-class TestSafariButton:
- """AX-click the 'Click Me' button without stealing focus."""
-
- def test_click_increments_counter(self, driver, safari_pid, ux_guard):
- wid = driver.find_window(safari_pid)
-
- # Read full tree first to get current counter value
- full_before = driver.get_window_state(safari_pid, wid)
- count_before = _parse_clicks(full_before.tree)
- idx = full_before.find_element("Click Me")
- assert idx is not None, f"'Click Me' button not found in tree:\n{full_before.tree}"
-
- driver.click(safari_pid, wid, element_index=idx)
- time.sleep(0.5)
-
- after = driver.get_window_state(safari_pid, wid)
- count_after = _parse_clicks(after.tree)
- print(f"\n click counter: {count_before} → {count_after}")
- assert count_after == count_before + 1, (
- f"Counter did not increment: {count_before} → {count_after}"
- )
-
- def test_click_increments_twice(self, driver, safari_pid, ux_guard):
- wid = driver.find_window(safari_pid)
-
- # Read full tree to get current counter value
- full_before = driver.get_window_state(safari_pid, wid)
- count_before = _parse_clicks(full_before.tree)
- idx = full_before.find_element("Click Me")
- assert idx is not None
-
- driver.click(safari_pid, wid, element_index=idx)
- time.sleep(0.3)
- driver.click(safari_pid, wid, element_index=idx)
- time.sleep(0.5)
-
- after = driver.get_window_state(safari_pid, wid)
- count_after = _parse_clicks(after.tree)
- print(f"\n click counter: {count_before} → {count_after}")
- assert count_after == count_before + 2, (
- f"Counter should have incremented by 2: {count_before} → {count_after}"
- )
-
-
-class TestSafariTextInput:
- """Type into a text field without stealing focus."""
-
- def test_type_text_appears_in_ax_tree(self, driver, safari_pid, ux_guard):
- wid = driver.find_window(safari_pid)
-
- before = driver.get_window_state(safari_pid, wid)
- idx = before.find_text_field(skip_url_bar=True)
- assert idx is not None, f"Text field not found in:\n{before.tree}"
-
- # Focus the field via AX click
- driver.click(safari_pid, wid, element_index=idx)
- time.sleep(0.3)
-
- # Clear any prior content
- driver.hotkey(safari_pid, ["cmd", "a"])
- time.sleep(0.1)
-
- driver.type_text_chars(safari_pid, "hello safari")
- time.sleep(0.6)
-
- after = driver.get_window_state(safari_pid, wid)
- assert after.has_text("hello safari"), (
- f"Typed text not visible in AX tree:\n{after.tree}"
- )
-
- def test_type_text_with_type_text_tool(self, driver, safari_pid, ux_guard):
- """Use type_text (not type_text_chars) and verify result."""
- wid = driver.find_window(safari_pid)
-
- before = driver.get_window_state(safari_pid, wid)
- idx = before.find_text_field(skip_url_bar=True)
- assert idx is not None
-
- driver.click(safari_pid, wid, element_index=idx)
- time.sleep(0.3)
-
- driver.hotkey(safari_pid, ["cmd", "a"])
- time.sleep(0.1)
-
- driver.type_text(safari_pid, "type_text test")
- time.sleep(0.6)
-
- after = driver.get_window_state(safari_pid, wid)
- assert after.has_text("type_text test"), (
- f"type_text result not in AX tree:\n{after.tree}"
- )
-
-
-class TestSafariCheckbox:
- """Toggle the checkbox."""
-
- def test_checkbox_toggle(self, driver, safari_pid, ux_guard):
- wid = driver.find_window(safari_pid)
-
- before = driver.get_window_state(safari_pid, wid, query="checkbox")
- idx = Tree.find(before.tree, role="AXCheckBox")
- if idx is None:
- # Fallback: find by label text
- idx = before.find_element("unchecked")
- assert idx is not None, f"Checkbox not found:\n{before.tree}"
-
- driver.click(safari_pid, wid, element_index=idx)
- time.sleep(0.4)
-
- after = driver.get_window_state(safari_pid, wid)
- assert after.has_text("checked"), (
- f"Checkbox did not toggle to checked:\n{after.tree}"
- )
-
-
-class TestSafariSelect:
- """Change the dropdown selection."""
-
- def test_select_option_b(self, driver, safari_pid, ux_guard):
- wid = driver.find_window(safari_pid)
-
- before = driver.get_window_state(safari_pid, wid, query="Option")
- idx = Tree.find(before.tree, role="AXPopUpButton")
- if idx is None:
- idx = Tree.find(before.tree, role="AXComboBox")
- if idx is None:
- idx = before.find_element("Option A")
- assert idx is not None, f"Select/dropdown not found:\n{before.tree}"
-
- driver.click(safari_pid, wid, element_index=idx)
- time.sleep(0.5)
-
- # Option B in the dropdown menu
- state2 = driver.get_window_state(safari_pid, wid)
- opt_b = Tree.find(state2.tree, label="Option B")
- if opt_b is not None:
- driver.click(safari_pid, wid, element_index=opt_b)
- else:
- # Fallback: set value directly
- driver.call_tool("set_value", {
- "pid": safari_pid, "window_id": wid,
- "element_index": idx, "value": "b",
- })
- time.sleep(0.4)
-
- after = driver.get_window_state(safari_pid, wid)
- assert after.has_text("Option B") or after.has_text("selected: Option B"), (
- f"Dropdown did not select Option B:\n{after.tree}"
- )
-
-
-class TestSafariTextarea:
- """Type into the textarea.
-
- Safari's WKWebView AXTextArea does not reliably set keyboard focus via
- AXPress in background mode. We use two strategies in sequence:
- 1. Click the text INPUT first to warm up Safari's web content focus engine.
- 2. Then click the textarea, wait 0.8s, type.
- """
-
- def test_textarea_input(self, driver, safari_pid, ux_guard):
- wid = driver.find_window(safari_pid)
-
- state = driver.get_window_state(safari_pid, wid)
-
- # Step 1: click the text input first to put focus inside Safari's web content
- tf_idx = state.find_text_field(skip_url_bar=True)
- if tf_idx is not None:
- driver.click(safari_pid, wid, element_index=tf_idx)
- time.sleep(0.3)
-
- # Step 2: click the textarea
- ta_idx = Tree.find(state.tree, role="AXTextArea")
- assert ta_idx is not None, f"Textarea not found:\n{state.tree[:500]}"
- driver.click(safari_pid, wid, element_index=ta_idx)
- time.sleep(0.8) # longer settle for WKWebView textarea focus
-
- # Step 3: type — keystrokes go to the focused element
- driver.call_tool("type_text_chars", {
- "pid": safari_pid,
- "text": "multi line",
- "delay_ms": 50,
- })
- time.sleep(0.6)
-
- after = driver.get_window_state(safari_pid, wid)
- val = Tree.ax_value(after.tree, ta_idx)
- print(f"\n textarea AX value: {val!r}")
- assert after.has_text("multi line"), (
- f"Textarea text not in AX tree; val={val!r};\n{after.tree[:800]}"
- )
-
-
-class TestSafariCanvas:
- """Send a pixel click to the canvas area without stealing focus.
-
- Background note: synthetic CGEvent pixel clicks to backgrounded browser
- windows may not trigger DOM mousedown events (web content is only rendered
- and interactive when the window is frontmost). This test therefore only
- asserts the key UX invariant: the click is delivered without error and
- without stealing focus. A DOM update (AX tree change or screenshot diff)
- is logged as a bonus if it occurs.
- """
-
- def test_canvas_pixel_click_no_focus_steal(self, driver, safari_pid, ux_guard):
- wid = driver.find_window(safari_pid)
-
- before = driver.get_window_state(safari_pid, wid)
- print(f"\n screenshot: {before.screenshot_width}x{before.screenshot_height}")
-
- from harness.cv import decode, diff_ratio
- before_img = decode(before.screenshot_b64)
-
- # Canvas is the last section on the page. With ~80px browser chrome
- # in a 906px window, the canvas element center is near y=88% of height.
- x = before.screenshot_width // 2
- y = int(before.screenshot_height * 0.88)
- print(f" pixel click at ({x}, {y})")
-
- result = driver.click(safari_pid, wid, x=x, y=y)
- time.sleep(0.5)
-
- after = driver.get_window_state(safari_pid, wid)
-
- # Bonus: check if DOM or screenshot updated (non-deterministic in background)
- canvas_clicked = "canvas: clicked" in after.tree
- after_img = decode(after.screenshot_b64)
- ratio = diff_ratio(before_img, after_img)
- print(f" canvas JS fired: {canvas_clicked}, diff_ratio: {ratio:.4f}")
-
- # Primary assertion: click succeeded (no exception was raised)
- # UX assertion: via ux_guard fixture teardown (no focus steal)
- assert result is not None, "click tool returned None"
-
-
-class TestSafariPressKey:
- """Press keys without stealing focus."""
-
- def test_tab_key(self, driver, safari_pid, ux_guard):
- driver.press_key(safari_pid, "tab")
- time.sleep(0.3)
- # No assertion on state — just verifying no exception + no UX violation
-
- def test_escape_key(self, driver, safari_pid, ux_guard):
- driver.press_key(safari_pid, "escape")
- time.sleep(0.2)
diff --git a/libs/cua-driver/tests/e2e/test_tauri.py b/libs/cua-driver/tests/e2e/test_tauri.py
deleted file mode 100644
index e8b6aaee8a..0000000000
--- a/libs/cua-driver/tests/e2e/test_tauri.py
+++ /dev/null
@@ -1,97 +0,0 @@
-"""Tauri shared-harness background automation tests.
-
-The Tauri test app is built from tests/fixtures/apps/cross-platform/tauri
-and loads tests/fixtures/shared/web/index.html, the same DOM used by the
-Electron, WebView2, and WKWebView harnesses.
-
-Run: python3 -m pytest test_tauri.py -v
-"""
-
-from __future__ import annotations
-
-import subprocess
-import sys
-import time
-import os
-
-import pytest
-
-_HERE = os.path.dirname(os.path.abspath(__file__))
-sys.path.insert(0, _HERE)
-
-
[email protected](autouse=True)
-def _reactivate_focus(focus_monitor):
- _, pid = focus_monitor
- subprocess.run(
- ["osascript", "-e",
- f'tell application "System Events" to set frontmost of (first process whose unix id is {pid}) to true'],
- check=False,
- )
- time.sleep(0.4)
-
-
-# ── tests ─────────────────────────────────────────────────────────────────────
-
-class TestTauriButton:
- def test_button_click(self, driver, tauri_app, ux_guard):
- _, pid, _ = tauri_app
- if pid == 0:
- pytest.skip("Tauri app pid not found")
-
- wid = driver.find_window(pid)
-
- before = driver.get_window_state(pid, wid, query="Increment")
- idx = before.find_button("Increment") or before.find_element("Increment")
- assert idx is not None, f"'Increment' not found:\n{before.tree}"
-
- driver.click(pid, wid, element_index=idx)
- time.sleep(0.5)
-
- after = driver.get_window_state(pid, wid)
- assert after.has_text("counter=1"), f"Counter not incremented:\n{after.tree}"
-
-
-class TestTauriTextInput:
- def test_type_text(self, driver, tauri_app, ux_guard):
- _, pid, _ = tauri_app
- if pid == 0:
- pytest.skip("Tauri app pid not found")
-
- wid = driver.find_window(pid)
-
- before = driver.get_window_state(pid, wid)
- idx = before.find_text_field(skip_url_bar=False)
- assert idx is not None, f"Text input not found:\n{before.tree}"
-
- driver.click(pid, wid, element_index=idx)
- time.sleep(0.3)
- driver.hotkey(pid, ["cmd", "a"])
- time.sleep(0.1)
- driver.type_text_chars(pid, "tauri test")
- time.sleep(0.6)
-
- after = driver.get_window_state(pid, wid)
- assert after.has_text("mirror=tauri test"), f"Text not mirrored in AX tree:\n{after.tree}"
-
-
-class TestTauriNoFocusSteal:
- def test_multiple_actions_no_focus_steal(self, driver, tauri_app, ux_guard):
- _, pid, _ = tauri_app
- if pid == 0:
- pytest.skip("Tauri app pid not found")
-
- wid = driver.find_window(pid)
-
- # Take 3 actions in sequence — UX guard checks no focus steal
- before = driver.get_window_state(pid, wid, query="Increment")
- idx = before.find_button("Increment") or before.find_element("Increment")
- if idx is not None:
- driver.click(pid, wid, element_index=idx)
- time.sleep(0.3)
-
- driver.press_key(pid, "tab")
- time.sleep(0.2)
- driver.press_key(pid, "tab")
- time.sleep(0.2)
- # ux_guard teardown will assert_clean()
diff --git a/libs/cua-driver/tests/fixtures/README.md b/libs/cua-driver/tests/fixtures/README.md
index c72f28a5b1..74df5105a3 100644
--- a/libs/cua-driver/tests/fixtures/README.md
+++ b/libs/cua-driver/tests/fixtures/README.md
@@ -74,8 +74,8 @@ Rust tests under `libs/cua-driver/rust/crates/cua-driver/tests/` consume the sta
modality coverage, ignored/manual or VM-backed.
- `protocol_*_test.rs` and schema tests: headless protocol coverage, default.
-The Python e2e suite under `libs/cua-driver/tests/e2e/` drives real apps and the shared
-Electron/Tauri harnesses through the public MCP interface.
+Rust integration tests under `libs/cua-driver/rust/crates/cua-driver/tests/` drive
+the shared Electron/Tauri harnesses through the public MCP interface.
## App Coverage
@@ -99,5 +99,5 @@ Electron/Tauri harnesses through the public MCP interface.
tests.
- Keep platform-specific quirks in the relevant Rust test or app source, not in
the shared fixture.
-- If a recorder finding graduates into an invariant, move it into a Rust or
- Python test and summarize it here.
+- If a recorder finding graduates into an invariant, move it into a Rust test
+ and summarize it here.
diff --git a/libs/cua-driver/tests/fixtures/apps/README.md b/libs/cua-driver/tests/fixtures/apps/README.md
index 7e3346165c..0263fdeceb 100644
--- a/libs/cua-driver/tests/fixtures/apps/README.md
+++ b/libs/cua-driver/tests/fixtures/apps/README.md
@@ -26,4 +26,4 @@ Adding a new app:
2. Reuse `shared/scenarios.json` and `shared/web/index.html` instead of copying
private ids.
3. Stage into `rust/test-apps/harness-<toolkit>/`.
-4. Add or update the matching Rust/Python tests and this table.
+4. Add or update the matching Rust tests and this table.