Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
Organize cua-driver harness tests and docs
Francesco Bonacciadcd70a9c3c4
diff --git a/docs/content/docs/reference/cua-driver/development.mdx b/docs/content/docs/reference/cua-driver/development.mdxnew file mode 100644index 0000000000..5e16d62068--- /dev/null+++ b/docs/content/docs/reference/cua-driver/development.mdx@@ -0,0 +1,30 @@+---+title: Development+description: Source layout, harnesses, and test entry points for contributors working on cua-driver.+---++cua-driver lives under `libs/cua-driver` in the repository. The hosted docs+describe the public interface; contributor details live next to the code so+they stay close to implementation changes.++## Source Map++| Path | Purpose |+|---|---|+| `libs/cua-driver/rust` | Rust workspace for the daemon, platform crates, testkit, and helper crates |+| `libs/cua-driver/python` | Python package wrapper around the driver binary |+| `libs/cua-driver/test-harness` | Source-built GUI harness apps and shared fixtures |+| `libs/cua-driver/tests/e2e` | Manual Python e2e tests against real apps and harness apps |+| `libs/cua-driver/scripts` | Install, uninstall, local build, and VM sync helpers |++## Contributor Entry Points++- `libs/cua-driver/README.md` gives the repo-local overview.+- `libs/cua-driver/rust/README.md` maps the Cargo workspace and test classes.+- `libs/cua-driver/rust/crates/cua-driver/tests/README.md` explains Rust test naming and ignored GUI lanes.+- `libs/cua-driver/test-harness/README.md` explains source-built harness apps and staged outputs.+- `libs/cua-driver/tests/e2e/README.md` explains the Python e2e suite.+- `libs/cua-driver/scripts/README.md` explains install and VM sync helpers.++Build artifacts, VM logs, and local verification journals should stay out of+git unless they have been promoted into a stable fixture or contributor doc.diff --git a/docs/content/docs/reference/cua-driver/meta.json b/docs/content/docs/reference/cua-driver/meta.jsonindex d37f264b67..2773873c78 100644--- a/docs/content/docs/reference/cua-driver/meta.json+++ b/docs/content/docs/reference/cua-driver/meta.json@@ -1 +1 @@-{ "title": "Cua Driver", "pages": ["cli-reference", "macos-permissions", "embedding", "mcp-tools", "mcp-tool-notes", "action-selection-policy", "contracts", "process-model", "limits"] }+{ "title": "Cua Driver", "pages": ["cli-reference", "macos-permissions", "embedding", "mcp-tools", "mcp-tool-notes", "action-selection-policy", "contracts", "process-model", "limits", "development"] }diff --git a/libs/cua-driver/README.md b/libs/cua-driver/README.mdindex bd124760c9..eeab1ed0e1 100644--- a/libs/cua-driver/README.md+++ b/libs/cua-driver/README.md@@ -4,6 +4,20 @@ Background computer-use driver for any agents. Speaks MCP over stdio; drives nat**[Documentation](https://cua.ai/docs/cua-driver)** - Installation, guides, and API reference.+## Repository Layout++| Path | Purpose |+| --- | --- |+| `rust/` | Cargo workspace for the driver daemon, platform crates, and Rust tests |+| `python/` | Python package wrapper and package tests |+| `test-harness/` | Source-built GUI harness apps and shared fixtures |+| `tests/e2e/` | Manual Python e2e tests against real apps and harness apps |+| `scripts/` | Install, uninstall, local build, and VM sync helpers |+| `docs/` | Small repo-local specs that are not part of the hosted docs site |++Start with `rust/README.md`, `test-harness/README.md`, and+`tests/e2e/README.md` when changing driver behavior or tests.+## Claude Code computer-use compatibilityStandard Claude Code MCP registration:diff --git a/libs/cua-driver/rust/.gitignore b/libs/cua-driver/rust/.gitignoreindex d70eb15e28..e69de29bb2 100644--- a/libs/cua-driver/rust/.gitignore+++ b/libs/cua-driver/rust/.gitignore@@ -1,3 +0,0 @@-# Large test assets — Electron app bundle exceeds GitHub's 100 MB file limit.-# Download separately for local integration testing.-tests/integration/v2/assets/electron/diff --git a/libs/cua-driver/rust/README.md b/libs/cua-driver/rust/README.mdnew file mode 100644index 0000000000..01338b1766--- /dev/null+++ b/libs/cua-driver/rust/README.md@@ -0,0 +1,58 @@+# cua-driver Rust workspace++This directory is the standalone Cargo workspace for the driver daemon, platform+implementations, testkit, and helper crates.++## Crates++| Crate | Purpose |+| --- | --- |+| `cua-driver` | Main CLI/MCP daemon and integration tests |+| `cua-driver-core` | Shared protocol, tool models, config, and CDP helpers |+| `platform-macos` | macOS AX, capture, input, browser, and daemon support |+| `platform-windows` | Windows UIA, capture, input, overlay, and diagnostics |+| `platform-linux` | Linux AT-SPI, X11/Wayland, capture, and input support |+| `cua-driver-testkit` | Test-only helpers for spawning the daemon and parsing responses |+| `cua-driver-uia` | Windows UIAccess worker |+| `focus-monitor-win` | Windows focus sentinel used by UX guard tests |+| `cursor-overlay` | Cursor overlay support |+| `pip-preview` | Packaging preview helper |++Platform crates are selected with `cfg(target_os)` from the main driver crate.+Keep platform-specific behavior inside the matching `platform-<os>` crate.++## Build++```bash+cargo build -p cua-driver+cargo build -p cua-driver --release+```++The Python wrapper and install scripts expect the built binary at+`target/<profile>/cua-driver` (or `cua-driver.exe` on Windows), unless+`CUA_DRIVER_BINARY` is set.++## Tests++Default tests should be headless and safe for CI:++```bash+cargo test -p cua-driver-core+cargo test -p cua-driver --test protocol_mcp_test+```++GUI and VM-backed tests are marked `#[ignore]` and require harness apps from+`../test-harness/build/` plus a real interactive desktop session:++```bash+cargo test -p cua-driver --test harness_appkit_test -- --ignored --nocapture+cargo test -p cua-driver --test harness_wpf_test -- --ignored --nocapture+```++See `crates/cua-driver/tests/README.md` for the test matrix.++## Generated Outputs++- `target/`: Cargo output, ignored.+- `test-apps/harness-*/`: staged harness apps, ignored.+- `test-apps/README.md`: explains how staged harness outputs are produced.diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/README.md b/libs/cua-driver/rust/crates/cua-driver/tests/README.mdnew file mode 100644index 0000000000..13fc3e2ea7--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver/tests/README.md@@ -0,0 +1,45 @@+# cua-driver Rust integration tests++Tests in this directory exercise the public driver interface. Headless protocol+tests run by default; GUI and modality tests are marked `#[ignore]` because they+need staged harness apps and an interactive desktop.++## Naming++| Prefix | Runs by default | Purpose |+| --- | --- | --- |+| `protocol_*_test.rs` | yes | MCP/CLI protocol and schema behavior |+| `schema_*_test.rs` | yes | Generated schema consistency |+| `harness_<toolkit>_test.rs` | no, `#[ignore]` | Toolkit-specific harness apps |+| `modality_<area>[_<os>]_test.rs` | no, `#[ignore]` | Background input, capture, desktop scope |+| `guard_*_test.rs` | usually ignored or self-skipping | UX guard and interactive desktop checks |++## Harness Requirements++Build harness apps before running ignored tests:++```bash+# macOS+../../../../test-harness/build/macos.sh++# Linux+../../../../test-harness/build/linux.sh+```++```powershell+# Windows+..\..\..\..\test-harness\build\windows.ps1+```++Staged outputs are read from `../../test-apps/harness-<name>/`.++## Running++```bash+cargo test -p cua-driver --test protocol_mcp_test+cargo test -p cua-driver --test harness_appkit_test -- --ignored --nocapture+cargo test -p cua-driver --test modality_desktop_scope_macos_test -- --ignored --nocapture+```++Windows Sandbox runs use `../../sandbox/run-tests-in-sandbox.ps1`, which builds+the Windows harness apps and maps them into the sandbox.diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/guard_ux_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/guard_ux_test.rsindex d26e572c29..a03eb5ff98 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/guard_ux_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/guard_ux_test.rs@@ -1,6 +1,6 @@//! UX-guard integration tests for Windows.//!-//! These mirror the macOS Python tests in libs/cua-driver/Tests/integration/:+//! These mirror the Python e2e tests in libs/cua-driver/tests/e2e/://! - test_background_focus.py//! - test_click_opens_new_window.py//! - test_launch_app_visible.py@@ -10,9 +10,9 @@//! The agent must be able to click, type, and launch apps in background//! windows WITHOUT stealing focus from the user's foreground window.//!-//! Background target: desktop-test-app-electron (prebuilt binary at-//! test-apps/desktop-test-app-electron.0.1.0.exe). Notepad is used only-//! as a secondary fallback check.+//! Background target: the repo-local Electron harness staged at+//! test-apps/harness-electron/CuaTestHarness.Electron.exe. Notepad is used+//! only as a secondary fallback check.//!//! All tests://! 1. Launch focus-monitor-win (the "user's foreground window").@@ -32,6 +32,28 @@ use cua_driver_testkit::{driver_binary, spawn_in_job, workspace_root, Driver, Mc// ── focus-monitor + test-app fixtures ────────────────────────────────────────+fn require_interactive_desktop(context: &str) -> bool {+ if platform_windows::diagnostics::current_session_id() == Some(0) {+ eprintln!(+ "{context}: running in Windows Session 0; skipping GUI UX guard. \+ Re-run from an interactive logon session (RDP/console/scheduled task in user session)."+ );+ return false;+ }++ match platform_windows::diagnostics::interactive_desktop_check() {+ Ok(true) => true,+ Ok(false) => {+ eprintln!("{context}: no foreground interactive desktop; skipping GUI UX guard");+ false+ }+ Err(e) => {+ eprintln!("{context}: interactive desktop check failed ({e}); skipping GUI UX guard");+ false+ }+ }+}+fn focus_monitor_path() -> PathBuf {workspace_root().join("target/debug/focus-monitor-win.exe")}@@ -39,44 +61,34 @@ fn focus_monitor_path() -> PathBuf {fn test_app_path() -> PathBuf {// In sandbox, sandbox-runner.ps1 copies the exe to %TEMP% to avoid the// ShellExecuteW zone-security dialog that blocks on mapped-folder exes.- if let Ok(p) = std::env::var("TEST_APP_EXE") {+ if let Ok(p) = std::env::var("HARNESS_ELECTRON_EXE") {let pb = PathBuf::from(p);- if pb.exists() { return pb; }+ if pb.exists() {+ return pb;+ }}- workspace_root().join("test-apps/desktop-test-app-electron.0.1.0.exe")+ workspace_root().join("test-apps/harness-electron/CuaTestHarness.Electron.exe")}-/// Launch the electron test app in the background (tied to the driver's reaper)-/// and return its pid. Waits up to 10s for the app's HTTP health endpoint to-/// respond. Returns None if the binary doesn't exist or the app fails to start.+/// Launch the Electron harness in the background (tied to the driver's reaper)+/// and return its pid. Returns None if the binary doesn't exist or the app+/// fails to start.fn launch_test_app(driver: &mut McpDriver) -> Option<u32> {let exe = test_app_path();if !exe.exists() {- eprintln!("desktop-test-app-electron not found at {exe:?} — skipping");+ eprintln!("CuaTestHarness.Electron not found at {exe:?} - skipping");return None;}let child = spawn_in_job(- Command::new(&exe).stdout(Stdio::null()).stderr(Stdio::null()),+ Command::new(&exe)+ .stdout(Stdio::null())+ .stderr(Stdio::null()),).ok()?;let pid = child.id();driver.reaper().push(child);- // Poll the HTTP health endpoint until it responds or timeout.- // Allow extra time for cold-start in sandbox (no cached Electron DLLs).- let deadline = std::time::Instant::now() + Duration::from_secs(20);- loop {- if std::time::Instant::now() > deadline {- eprintln!("desktop-test-app-electron HTTP /health did not respond within 10s");- return Some(pid);- }- if let Ok(stream) = std::net::TcpStream::connect("127.0.0.1:6769") {- drop(stream);- break;- }- std::thread::sleep(Duration::from_millis(200));- }- // Extra settle time for the window to appear.- std::thread::sleep(Duration::from_millis(500));+ // Cold-start in sandbox can take a few seconds.+ std::thread::sleep(Duration::from_secs(3));Some(pid)}@@ -94,16 +106,25 @@ fn read_losses(path: &std::path::Path) -> u32 {.unwrap_or(0)}-fn focus_pid_file() -> PathBuf { std::env::temp_dir().join("focus_monitor_pid.txt") }-fn focus_hwnd_file() -> PathBuf { std::env::temp_dir().join("focus_monitor_hwnd.txt") }+fn focus_pid_file() -> PathBuf {+ std::env::temp_dir().join("focus_monitor_pid.txt")+}+fn focus_hwnd_file() -> PathBuf {+ std::env::temp_dir().join("focus_monitor_hwnd.txt")+}/// Launch focus-monitor-win and return (process, hwnd, pid)./// Reads FOCUS_PID and FOCUS_HWND from temp files written by the monitor/// (avoids blocking on the stdout pipe if the sandbox redirects I/O).-fn launch_focus_monitor() -> (Child, u64, u32) {+fn launch_focus_monitor() -> Option<(Child, u64, u32)> {+ if !require_interactive_desktop("focus-monitor-win") {+ return None;+ }+let exe = focus_monitor_path();if !exe.exists() {- panic!("focus-monitor-win.exe not found at {exe:?} — run `cargo build` first");+ eprintln!("focus-monitor-win.exe not built at {exe:?} — skipping");+ return None;}// Reset all sentinel files so stale values are not mistaken for new ones.let _ = std::fs::write(loss_file(), "0");@@ -112,7 +133,9 @@ fn launch_focus_monitor() -> (Child, u64, u32) {let _ = std::fs::remove_file(focus_hwnd_file());let child = spawn_in_job(- Command::new(&exe).stdout(Stdio::null()).stderr(Stdio::null()),+ Command::new(&exe)+ .stdout(Stdio::null())+ .stderr(Stdio::null()),).expect("spawn focus-monitor-win");@@ -125,24 +148,32 @@ fn launch_focus_monitor() -> (Child, u64, u32) {}if pid_val == 0 {pid_val = std::fs::read_to_string(focus_pid_file())- .ok().and_then(|s| s.trim().parse().ok()).unwrap_or(0);+ .ok()+ .and_then(|s| s.trim().parse().ok())+ .unwrap_or(0);}if hwnd_val == 0 {hwnd_val = std::fs::read_to_string(focus_hwnd_file())- .ok().and_then(|s| s.trim().parse().ok()).unwrap_or(0);+ .ok()+ .and_then(|s| s.trim().parse().ok())+ .unwrap_or(0);+ }+ if pid_val != 0 && hwnd_val != 0 {+ break;}- if pid_val != 0 && hwnd_val != 0 { break; }std::thread::sleep(Duration::from_millis(100));}// Give the window time to become foreground.std::thread::sleep(Duration::from_millis(400));- (child, hwnd_val, pid_val)+ Some((child, hwnd_val, pid_val))}/// Find the first on-screen window belonging to the given pid.fn find_window_for_pid(driver: &mut McpDriver, pid: i64) -> Option<u64> {- let resp = driver.call("list_windows",- serde_json::json!({"pid": pid, "on_screen_only": true}));+ let resp = driver.call(+ "list_windows",+ serde_json::json!({"pid": pid, "on_screen_only": true}),+ );resp.structured()["windows"].as_array()?.iter()@@ -171,39 +202,64 @@ fn test_background_click_and_type_no_focus_steal() {//! Equivalent of macOS test_background_focus.py.//!//! 1. Launch FocusMonitorWin (simulates the user's active window).- //! 2. Launch desktop-test-app-electron in the background.+ //! 2. Launch the Electron harness in the background.//! 3. Click inside the app and type text via cua-driver.//! 4. Assert act_losses on FocusMonitorWin stayed at 0.- if !driver_binary().exists() { eprintln!("Binary not found — skipping"); return; }+ if !driver_binary().exists() {+ eprintln!("Binary not found — skipping");+ return;+ }- let (mut fm_proc, _fm_hwnd, _fm_pid) = launch_focus_monitor();+ let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {+ return;+ };let losses_before = read_losses(&loss_file());- let Some(mut driver) = McpDriver::spawn() else { fm_proc.kill().ok(); return; };+ let Some(mut driver) = McpDriver::spawn() else {+ fm_proc.kill().ok();+ return;+ };let Some(app_pid) = launch_test_app(&mut driver) else {eprintln!("test app not available — skipping");- fm_proc.kill().ok(); return;+ fm_proc.kill().ok();+ return;};let Some(app_wid) = find_window_for_pid(&mut driver, app_pid as i64) else {eprintln!("test app window not found — skipping");- fm_proc.kill().ok(); return;+ fm_proc.kill().ok();+ return;};// Click inside the app (background, via PostMessage).- let r = driver.call("click",- serde_json::json!({"pid": app_pid, "window_id": app_wid, "x": 200.0, "y": 200.0}));- assert!(r.raw["error"].is_null(), "Protocol error from click: {:?}", r.raw);+ let r = driver.call(+ "click",+ serde_json::json!({"pid": app_pid, "window_id": app_wid, "x": 200.0, "y": 200.0}),+ );+ assert!(+ r.raw["error"].is_null(),+ "Protocol error from click: {:?}",+ r.raw+ );// Type text into the app (background, via PostMessage).- let r = driver.call("type_text",- serde_json::json!({"pid": app_pid, "window_id": app_wid, "text": "ux-guard-test"}));- assert!(r.raw["error"].is_null(), "Protocol error from type_text: {:?}", r.raw);+ let r = driver.call(+ "type_text",+ serde_json::json!({"pid": app_pid, "window_id": app_wid, "text": "ux-guard-test"}),+ );+ assert!(+ r.raw["error"].is_null(),+ "Protocol error from type_text: {:?}",+ r.raw+ );// ux_guard: FocusMonitorWin must not have lost activation.- assert_ux_guard(losses_before, 0,- "background click + type_text into desktop-test-app-electron");+ assert_ux_guard(+ losses_before,+ 0,+ "background click + type_text into CuaTestHarness.Electron",+ );fm_proc.kill().ok();}@@ -218,22 +274,33 @@ fn test_launch_app_no_focus_steal() {//!//! launch_app (path variant) must open a window without displacing FocusMonitorWin.- if !driver_binary().exists() { return; }+ if !driver_binary().exists() {+ return;+ }let exe = test_app_path();- if !exe.exists() { eprintln!("test app not available — skipping"); return; }+ if !exe.exists() {+ eprintln!("test app not available — skipping");+ return;+ }- let (mut fm_proc, _fm_hwnd, _fm_pid) = launch_focus_monitor();+ let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {+ return;+ };let losses_before = read_losses(&loss_file());- let Some(mut driver) = McpDriver::spawn() else { fm_proc.kill().ok(); return; };+ let Some(mut driver) = McpDriver::spawn() else {+ fm_proc.kill().ok();+ return;+ };// Launch the test app via cua-driver launch_app (full path, SW_SHOWNOACTIVATE).let path_str = exe.to_string_lossy().into_owned();let r = driver.call("launch_app", serde_json::json!({"path": path_str}));if r.is_error() {eprintln!("launch_app failed — skipping: {:?}", r.raw);- fm_proc.kill().ok(); return;+ fm_proc.kill().ok();+ return;}// Wait for the app window to appear (Electron startup ~2-3s).@@ -243,7 +310,13 @@ fn test_launch_app_no_focus_steal() {let r2 = driver.call("list_apps", serde_json::json!({}));if let Some(procs) = r2.structured()["processes"].as_array() {if let Some(p) = procs.iter().find(|p| {- p["name"].as_str().map(|n| n.to_lowercase().contains("desktop-test-app")).unwrap_or(false)+ p["name"]+ .as_str()+ .map(|n| {+ let n = n.to_lowercase();+ n.contains("cuatestharness.electron") || n.contains("electron")+ })+ .unwrap_or(false)}) {app_pid = p["pid"].as_i64();break;@@ -251,16 +324,21 @@ fn test_launch_app_no_focus_steal() {}}if app_pid.is_none() {- eprintln!("desktop-test-app not found in process list after launch_app — skipping");- fm_proc.kill().ok(); return;+ eprintln!("CuaTestHarness.Electron not found in process list after launch_app - skipping");+ fm_proc.kill().ok();+ return;}// ux_guard: FocusMonitorWin must not have lost activation.- assert_ux_guard(losses_before, 0, "launch_app desktop-test-app-electron");+ assert_ux_guard(losses_before, 0, "launch_app CuaTestHarness.Electron");// Kill the launched app by exe name.- Command::new("taskkill").args(["/F", "/T", "/IM", "desktop-test-app-electron.0.1.0.exe"])- .stdout(Stdio::null()).stderr(Stdio::null()).spawn().ok();+ Command::new("taskkill")+ .args(["/F", "/T", "/IM", "CuaTestHarness.Electron.exe"])+ .stdout(Stdio::null())+ .stderr(Stdio::null())+ .spawn()+ .ok();fm_proc.kill().ok();}@@ -272,32 +350,51 @@ fn test_launch_app_no_focus_steal() {fn test_background_hotkey_no_focus_steal() {//! Equivalent of macOS test_background_menu_shortcut.py.//!- //! Send Ctrl+A to a background desktop-test-app-electron window.+ //! Send Ctrl+A to a background Electron harness window.//! FocusMonitorWin must never lose activation.- if !driver_binary().exists() { return; }+ if !driver_binary().exists() {+ return;+ }- let (mut fm_proc, _fm_hwnd, _fm_pid) = launch_focus_monitor();+ let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {+ return;+ };let losses_before = read_losses(&loss_file());- let Some(mut driver) = McpDriver::spawn() else { fm_proc.kill().ok(); return; };+ let Some(mut driver) = McpDriver::spawn() else {+ fm_proc.kill().ok();+ return;+ };let Some(app_pid) = launch_test_app(&mut driver) else {eprintln!("test app not available — skipping");- fm_proc.kill().ok(); return;+ fm_proc.kill().ok();+ return;};let Some(app_wid) = find_window_for_pid(&mut driver, app_pid as i64) else {eprintln!("test app window not found — skipping");- fm_proc.kill().ok(); return;+ fm_proc.kill().ok();+ return;};// Send Ctrl+A hotkey to background app (PostMessage, no focus steal).- let r = driver.call("hotkey",- serde_json::json!({"pid": app_pid, "window_id": app_wid, "keys": ["ctrl", "a"]}));- assert!(r.raw["error"].is_null(), "Protocol error from hotkey: {:?}", r.raw);+ let r = driver.call(+ "hotkey",+ serde_json::json!({"pid": app_pid, "window_id": app_wid, "keys": ["ctrl", "a"]}),+ );+ assert!(+ r.raw["error"].is_null(),+ "Protocol error from hotkey: {:?}",+ r.raw+ );// ux_guard: FocusMonitorWin must not have lost activation.- assert_ux_guard(losses_before, 0, "background hotkey ctrl+a to desktop-test-app-electron");+ assert_ux_guard(+ losses_before,+ 0,+ "background hotkey ctrl+a to CuaTestHarness.Electron",+ );fm_proc.kill().ok();}@@ -311,39 +408,57 @@ fn test_background_click_opens_new_window_focus_preserved() {//! Equivalent of macOS test_click_opens_new_window.py.//!//! 1. FocusMonitorWin is foreground.- //! 2. Click a link in desktop-test-app-electron — a new window/tab may appear.+ //! 2. Click in CuaTestHarness.Electron - a new window/tab may appear.//! 3. FocusMonitorWin must remain active throughout (UX guard).//!//! Verifies PostMessage doesn't inadvertently activate any new window that//! appears as a side-effect of the click.- if !driver_binary().exists() { return; }+ if !driver_binary().exists() {+ return;+ }- let (mut fm_proc, _fm_hwnd, _fm_pid) = launch_focus_monitor();+ let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {+ return;+ };let losses_before = read_losses(&loss_file());- let Some(mut driver) = McpDriver::spawn() else { fm_proc.kill().ok(); return; };+ let Some(mut driver) = McpDriver::spawn() else {+ fm_proc.kill().ok();+ return;+ };let Some(app_pid) = launch_test_app(&mut driver) else {eprintln!("test app not available — skipping");- fm_proc.kill().ok(); return;+ fm_proc.kill().ok();+ return;};let Some(app_wid) = find_window_for_pid(&mut driver, app_pid as i64) else {eprintln!("test app window not found — skipping");- fm_proc.kill().ok(); return;+ fm_proc.kill().ok();+ return;};// Click somewhere in the app content area (may trigger navigation/new window).- let r = driver.call("click",- serde_json::json!({"pid": app_pid, "window_id": app_wid, "x": 400.0, "y": 350.0}));- assert!(r.raw["error"].is_null(), "Protocol error from click: {:?}", r.raw);+ let r = driver.call(+ "click",+ serde_json::json!({"pid": app_pid, "window_id": app_wid, "x": 400.0, "y": 350.0}),+ );+ assert!(+ r.raw["error"].is_null(),+ "Protocol error from click: {:?}",+ r.raw+ );// Brief wait for any side-effect windows to appear.std::thread::sleep(Duration::from_millis(500));// ux_guard: FocusMonitorWin must not have lost activation.- assert_ux_guard(losses_before, 0,- "background click in desktop-test-app-electron (may open new window)");+ assert_ux_guard(+ losses_before,+ 0,+ "background click in CuaTestHarness.Electron (may open new window)",+ );// Verify FocusMonitorWin is still alive.assert!(@@ -362,28 +477,45 @@ fn test_background_click_opens_new_window_focus_preserved() {fn test_background_screenshot_no_focus_steal() {//! PrintWindow captures a background window without activating it.- if !driver_binary().exists() { return; }+ if !driver_binary().exists() {+ return;+ }- let (mut fm_proc, _fm_hwnd, _fm_pid) = launch_focus_monitor();+ let Some((mut fm_proc, _fm_hwnd, _fm_pid)) = launch_focus_monitor() else {+ return;+ };let losses_before = read_losses(&loss_file());- let Some(mut driver) = McpDriver::spawn() else { fm_proc.kill().ok(); return; };+ let Some(mut driver) = McpDriver::spawn() else {+ fm_proc.kill().ok();+ return;+ };let Some(app_pid) = launch_test_app(&mut driver) else {eprintln!("test app not available — skipping");- fm_proc.kill().ok(); return;+ fm_proc.kill().ok();+ return;};let Some(app_wid) = find_window_for_pid(&mut driver, app_pid as i64) else {eprintln!("test app window not found — skipping");- fm_proc.kill().ok(); return;+ fm_proc.kill().ok();+ return;};// Screenshot via PrintWindow — must not activate the window.let r = driver.call("screenshot", serde_json::json!({"window_id": app_wid}));- assert!(r.raw["error"].is_null(), "Protocol error from screenshot: {:?}", r.raw);+ assert!(+ r.raw["error"].is_null(),+ "Protocol error from screenshot: {:?}",+ r.raw+ );// ux_guard- assert_ux_guard(losses_before, 0, "screenshot of background desktop-test-app-electron");+ assert_ux_guard(+ losses_before,+ 0,+ "screenshot of background CuaTestHarness.Electron",+ );fm_proc.kill().ok();}@@ -404,17 +536,32 @@ fn test_agent_cursor_visible_on_screen() {//! 4. Decode the PNG and sample a 40×40 px patch centred on the cursor.//! 5. Assert the patch contains cursor-like pixels (bright or saturated).- if !driver_binary().exists() { eprintln!("Binary not found — skipping"); return; }+ if !driver_binary().exists() {+ eprintln!("Binary not found — skipping");+ return;+ }+ if !require_interactive_desktop("agent cursor visibility") {+ return;+ }- let Some(mut driver) = McpDriver::spawn() else { return };+ let Some(mut driver) = McpDriver::spawn() else {+ return;+ };// Safe centre-ish position on primary monitor.let cx = 640.0_f64;let cy = 400.0_f64;// Enable cursor overlay and glide to target.- let r = driver.call("set_agent_cursor_enabled", serde_json::json!({"enabled": true}));- assert!(r.raw["error"].is_null(), "set_agent_cursor_enabled failed: {:?}", r.raw);+ let r = driver.call(+ "set_agent_cursor_enabled",+ serde_json::json!({"enabled": true}),+ );+ assert!(+ r.raw["error"].is_null(),+ "set_agent_cursor_enabled failed: {:?}",+ r.raw+ );let r = driver.call("move_cursor", serde_json::json!({"x": cx, "y": cy}));assert!(r.raw["error"].is_null(), "move_cursor failed: {:?}", r.raw);@@ -444,7 +591,9 @@ fn test_agent_cursor_visible_on_screen() {for py in y0..y1 {for px in x0..x1 {let [r, g, b, a] = rgba.get_pixel(px, py).0;- if a < 10 { continue; }+ if a < 10 {+ continue;+ }let brightness = r as u32 + g as u32 + b as u32;let saturation = r.max(g).max(b) as u32 - r.min(g).min(b) as u32;// Accept bright-white stroke pixels OR coloured gradient pixels.diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/modality_input_e2e_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/modality_input_e2e_test.rsindex 511522449d..9efc8e7ef9 100644--- a/libs/cua-driver/rust/crates/cua-driver/tests/modality_input_e2e_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver/tests/modality_input_e2e_test.rs@@ -18,15 +18,15 @@//! held ports. Broker-spawned window pids (packaged apps / Electron) are//! tree-killed via `reaper().track_pid`.//!-//! Targets (trycua test apps, auto-downloaded to %TEMP% if not provided):-//! - Electron (Chromium content) — `trycua/desktop-test-app-electron`.-//! - Tauri (WebView2 content) — `trycua/desktop-test-app`.+//! Targets (repo-local harness apps staged by `test-harness/build/windows.ps1`):+//! - Electron (Chromium content) — `CuaTestHarness.Electron.exe`.+//! - Tauri (WebView2 content) — `CuaTestHarness.Tauri.exe`.//! - Win32 baseline — `notepad.exe`.-//! Override with `CUA_ELECTRON_EXE` / `CUA_TAURI_EXE`, or drop the exe in-//! `test-apps/`. Auto-download uses `curl.exe`.+//! Override with `HARNESS_ELECTRON_EXE` / `HARNESS_TAURI_EXE`+//! (or legacy `CUA_ELECTRON_EXE` / `CUA_TAURI_EXE`).//!//! All tests are `#[ignore]` (GUI, real desktop session). Run explicitly,-//! serially (apps bind a fixed HTTP port, so never in parallel):+//! serially (GUI focus oracle, so never in parallel)://! cargo test -p cua-driver --test e2e_windows_bg_input_test -- --ignored --nocapture --test-threads=1#![cfg(target_os = "windows")]@@ -39,10 +39,7 @@ use std::time::{Duration, Instant};use cua_driver_testkit::{ax, workspace_root, Driver, McpDriver};-const ELECTRON_URL: &str = "https://github.com/trycua/desktop-test-app-electron/releases/download/v0.1.0/desktop-test-app-electron.0.1.0.exe";-const TAURI_URL: &str = "https://github.com/trycua/desktop-test-app/releases/download/v0.2.2/desktop-test-app-windows-x86_64.exe";--// ── paths / downloads ─────────────────────────────────────────────────────────+// ── paths ────────────────────────────────────────────────────────────────────fn focus_monitor_binary() -> PathBuf {workspace_root().join("target/debug/focus-monitor-win.exe")@@ -60,56 +57,30 @@ fn kill_prior_by_name(exe: &Path) {}}-fn curl_download(url: &str, dst: &Path) -> bool {- eprintln!("[e2e] downloading {url}\n -> {dst:?}");- let ok = Command::new("curl.exe")- .args(["-L", "--fail", "--silent", "--show-error", "-o"])- .arg(dst)- .arg(url)- .status()- .map(|s| s.success())- .unwrap_or(false);- ok && fs::metadata(dst).map(|m| m.len() > 4096).unwrap_or(false)-}-fn resolve_or_download(env_var: &str, prefix: &str, url: &str, tmp_name: &str) -> Option<PathBuf> {- if let Ok(p) = std::env::var(env_var) {- let pb = PathBuf::from(p);- if pb.exists() {- return Some(pb);- }- }- if let Ok(entries) = fs::read_dir(workspace_root().join("test-apps")) {- for e in entries.flatten() {- let name = e.file_name().to_string_lossy().to_lowercase();- if name.starts_with(prefix) && name.ends_with(".exe") && !name.contains("setup") {- return Some(e.path());+fn resolve_harness(primary_env: &str, legacy_env: &str, rel_path: &str) -> Option<PathBuf> {+ for env_var in [primary_env, legacy_env] {+ if let Ok(p) = std::env::var(env_var) {+ let pb = PathBuf::from(p);+ if pb.exists() {+ return Some(pb);}}}- let dst = std::env::temp_dir().join(tmp_name);- if dst.exists() && fs::metadata(&dst).map(|m| m.len() > 4096).unwrap_or(false) {- return Some(dst);- }- if curl_download(url, &dst) {- Some(dst)- } else {- None- }+ let pb = workspace_root().join(rel_path);+ pb.exists().then_some(pb)}fn electron_exe() -> Option<PathBuf> {- resolve_or_download(+ resolve_harness(+ "HARNESS_ELECTRON_EXE","CUA_ELECTRON_EXE",- "desktop-test-app-electron",- ELECTRON_URL,- "desktop-test-app-electron.0.1.0.exe",+ "test-apps/harness-electron/CuaTestHarness.Electron.exe",)}fn tauri_exe() -> Option<PathBuf> {- resolve_or_download(+ resolve_harness(+ "HARNESS_TAURI_EXE","CUA_TAURI_EXE",- "desktop-test-app-windows",- TAURI_URL,- "desktop-test-app-windows-x86_64.exe",+ "test-apps/harness-tauri/CuaTestHarness.Tauri.exe",)}@@ -126,7 +97,10 @@ fn focus_hwnd_file() -> PathBuf {std::env::temp_dir().join("focus_monitor_hwnd.txt")}fn read_count(p: &Path) -> u32 {- fs::read_to_string(p).ok().and_then(|s| s.trim().parse::<u32>().ok()).unwrap_or(0)+ fs::read_to_string(p)+ .ok()+ .and_then(|s| s.trim().parse::<u32>().ok())+ .unwrap_or(0)}fn window_ids(driver: &mut McpDriver) -> HashSet<u64> {@@ -174,7 +148,11 @@ fn setup(target_exe: &Path, _title_hint: &str) -> Option<E2eFixture> {let before = window_ids(&mut driver);if driver.reaper()- .spawn(Command::new(target_exe).stdout(Stdio::null()).stderr(Stdio::null()))+ .spawn(+ Command::new(target_exe)+ .stdout(Stdio::null())+ .stderr(Stdio::null()),+ ).is_err(){eprintln!("[e2e] target spawn failed"); // driver drops → kills itself@@ -189,7 +167,9 @@ fn setup(target_exe: &Path, _title_hint: &str) -> Option<E2eFixture> {let r = driver.call("list_windows", serde_json::json!({}));if let Some(arr) = r.structured()["windows"].as_array() {for w in arr {- let Some(wid) = w["window_id"].as_u64() else { continue };+ let Some(wid) = w["window_id"].as_u64() else {+ continue;+ };let pid = w["pid"].as_u64().unwrap_or(0) as u32;let title = w["title"].as_str().unwrap_or("");if !before.contains(&wid) && !title.is_empty() && pid != 0 {@@ -218,7 +198,11 @@ fn setup(target_exe: &Path, _title_hint: &str) -> Option<E2eFixture> {// 4. Sentinel grabs foreground; app drops to z+1 (the background target).if driver.reaper()- .spawn(Command::new(&fm_bin).stdout(Stdio::null()).stderr(Stdio::null()))+ .spawn(+ Command::new(&fm_bin)+ .stdout(Stdio::null())+ .stderr(Stdio::null()),+ ).is_err(){eprintln!("[e2e] sentinel spawn failed — skipping");@@ -340,7 +324,9 @@ fn webview_click_case(label: &str, exe: PathBuf) {let mut needs_foreground = false;assert_target_stays_background(label, pid, || {let args = match elem {- Some(idx) => serde_json::json!({"pid": pid as i64, "window_id": wid, "element_index": idx}),+ Some(idx) => {+ serde_json::json!({"pid": pid as i64, "window_id": wid, "element_index": idx})+ }None => serde_json::json!({"pid": pid as i64, "window_id": wid, "x": 200, "y": 200}),};let last = fx.driver.call("click", args);@@ -358,7 +344,9 @@ fn webview_click_case(label: &str, exe: PathBuf) {// only — it does NOT capture accessibility-driven UIA Invoke, so a `[]`// there is expected for the UIA path. The hard invariant is no-foreground-steal.std::thread::sleep(Duration::from_millis(300));- let reg = registered_since_reset().map(|b| b.to_string()).unwrap_or_else(|| "n/a".into());+ let reg = registered_since_reset()+ .map(|b| b.to_string())+ .unwrap_or_else(|| "n/a".into());println!("{label}: delivered={delivered:?} pointer-events-logged={reg}");}diff --git a/libs/cua-driver/rust/sandbox/run-tests-in-sandbox.ps1 b/libs/cua-driver/rust/sandbox/run-tests-in-sandbox.ps1index 2299ab2d4a..d532dfef9a 100644--- a/libs/cua-driver/rust/sandbox/run-tests-in-sandbox.ps1+++ b/libs/cua-driver/rust/sandbox/run-tests-in-sandbox.ps1@@ -53,18 +53,6 @@ if ($running) {Write-Host "[OK] Existing sandbox closed. Proceeding." -ForegroundColor Green}-# ── 0. Download test-app binaries if missing ─────────────────────────────────-$testAppExe = "$wsRoot\test-apps\desktop-test-app-electron.0.1.0.exe"-if (-not (Test-Path $testAppExe)) {- Write-Host "`n[FETCH] Downloading desktop-test-app-electron..." -ForegroundColor Yellow- New-Item -ItemType Directory -Force "$wsRoot\test-apps" | Out-Null- $url = "https://github.com/trycua/desktop-test-app-electron/releases/download/v0.1.0/desktop-test-app-electron.0.1.0.exe"- Invoke-WebRequest -Uri $url -OutFile $testAppExe -UseBasicParsing- Write-Host " Downloaded: $testAppExe" -ForegroundColor Green-} else {- Write-Host "`n[FETCH] test-app already present: $testAppExe" -ForegroundColor Green-}-# ── 1. Build ──────────────────────────────────────────────────────────────────Push-Location $wsRoottry {@@ -97,10 +85,10 @@ try {if ($LASTEXITCODE -ne 0) { throw "cargo test --no-run (harness_bg_modality_test) failed" }} finally { Pop-Location }-# ── 1.5. Build the .NET test-harness if dotnet is on PATH ────────────────────-# The harness produces test-apps/harness-{wpf,winui3}/ which the sandbox+# ── 1.5. Build the test-harness if dependencies are on PATH ──────────────────+# The harness produces test-apps/harness-*/ which the sandbox# runner picks up via the existing mapped-folder route. Skip silently if-# dotnet isn't available — the smoke tests degrade to "skipped" inside+# a dependency isn't available — the smoke tests degrade to "skipped" inside# the sandbox rather than failing the whole run.if (Get-Command dotnet -ErrorAction SilentlyContinue) {$harnessBuild = Join-Path $wsRoot "..\test-harness\build\windows.ps1"diff --git a/libs/cua-driver/rust/sandbox/sandbox-runner.ps1 b/libs/cua-driver/rust/sandbox/sandbox-runner.ps1index 08b15db307..805452a948 100644--- a/libs/cua-driver/rust/sandbox/sandbox-runner.ps1+++ b/libs/cua-driver/rust/sandbox/sandbox-runner.ps1@@ -18,19 +18,6 @@ Log "sandbox-runner.ps1 starting"$env:CARGO_MANIFEST_DIR = "C:\cua-driver-rs\crates\cua-driver"Log "CARGO_MANIFEST_DIR = $env:CARGO_MANIFEST_DIR"-# ── copy test-app exe to %TEMP% so ShellExecuteW doesn't hit a zone dialog ──-# Files on Sandbox-mapped folders are Zone 3 (Internet); ShellExecuteW blocks-# on the "Do you want to run this file?" dialog. A local %TEMP% copy is Zone 1.-$testAppSrc = "C:\cua-driver-rs\test-apps\desktop-test-app-electron.0.1.0.exe"-$testAppDst = "$env:TEMP\desktop-test-app-electron.0.1.0.exe"-if (Test-Path $testAppSrc) {- Copy-Item $testAppSrc $testAppDst -Force- $env:TEST_APP_EXE = $testAppDst- Log "test-app copied to $testAppDst (TEST_APP_EXE=$testAppDst)"-} else {- Log "WARNING: test-app not found at $testAppSrc"-}-$driverExe = "C:\cua-driver-rs\target\debug\cua-driver.exe"if (-not (Test-Path $driverExe)) {Log "ERROR: $driverExe not found"@@ -57,7 +44,8 @@ $harnessRoots = @(@{ Src = "C:\cua-driver-rs\test-apps\harness-wpf"; EnvVar = "HARNESS_WPF_EXE"; Exe = "CuaTestHarness.Wpf.exe" },@{ Src = "C:\cua-driver-rs\test-apps\harness-winui3"; EnvVar = "HARNESS_WINUI3_EXE"; Exe = "CuaTestHarness.WinUI3.exe" },@{ Src = "C:\cua-driver-rs\test-apps\harness-webview"; EnvVar = "HARNESS_WEBVIEW_EXE"; Exe = "CuaTestHarness.WebView.exe" },- @{ Src = "C:\cua-driver-rs\test-apps\harness-electron"; EnvVar = "HARNESS_ELECTRON_EXE"; Exe = "CuaTestHarness.Electron.exe" }+ @{ Src = "C:\cua-driver-rs\test-apps\harness-electron"; EnvVar = "HARNESS_ELECTRON_EXE"; Exe = "CuaTestHarness.Electron.exe" },+ @{ Src = "C:\cua-driver-rs\test-apps\harness-tauri"; EnvVar = "HARNESS_TAURI_EXE"; Exe = "CuaTestHarness.Tauri.exe" })foreach ($h in $harnessRoots) {if (-not (Test-Path $h.Src)) {diff --git a/libs/cua-driver/rust/test-apps/.gitignore b/libs/cua-driver/rust/test-apps/.gitignoreindex e3ba2f39e6..56b007e740 100644--- a/libs/cua-driver/rust/test-apps/.gitignore+++ b/libs/cua-driver/rust/test-apps/.gitignore@@ -15,3 +15,4 @@ harness-wpf/harness-winui3/harness-webview/harness-electron/+harness-tauri/diff --git a/libs/cua-driver/rust/test-apps/README.md b/libs/cua-driver/rust/test-apps/README.mdindex 6b2419a5ad..ff1e4f8d13 100644--- a/libs/cua-driver/rust/test-apps/README.md+++ b/libs/cua-driver/rust/test-apps/README.md@@ -1,15 +1,30 @@# test-apps-Pre-built test application binaries used by `ux_guard_test`.+Local staging directory for cua-driver test harness applications.-## Contents+Build scripts under `../test-harness/build/` write `harness-<name>/`+directories here. The staged outputs are intentionally git-ignored; rebuild+them from source instead of committing binaries.-- `desktop-test-app-electron.0.1.0.exe` — Electron test app from- [trycua/desktop-test-app-electron v0.1.0](https://github.com/trycua/desktop-test-app-electron/releases/tag/v0.1.0).- **Not committed to git** (76 MB). Downloaded automatically by- `sandbox\run-tests-in-sandbox.ps1` on first run.+Common outputs:-## Updating+- `harness-wpf/`+- `harness-winui3/`+- `harness-webview/`+- `harness-electron/`+- `harness-tauri/`+- `harness-appkit/`+- `harness-swiftui/`+- `harness-wkwebview/`+- `harness-gtk3/`-To update to a new version, edit the download URL in-`sandbox\run-tests-in-sandbox.ps1` and delete the old binary.+Run the host build script for your platform:++```bash+../../test-harness/build/macos.sh+../../test-harness/build/linux.sh+```++```powershell+..\..\test-harness\build\windows.ps1+```diff --git a/libs/cua-driver/rust/tests/integration/v2/assets/tauri/desktop-test-app-macos-arm64 b/libs/cua-driver/rust/tests/integration/v2/assets/tauri/desktop-test-app-macos-arm64deleted file mode 100755index 412f68af9a..0000000000Binary files a/libs/cua-driver/rust/tests/integration/v2/assets/tauri/desktop-test-app-macos-arm64 and /dev/null differdiff --git a/libs/cua-driver/rust/tests/integration/v2/assets/test_page.html b/libs/cua-driver/rust/tests/integration/v2/assets/test_page.htmldeleted file mode 120000index 57fd293961..0000000000--- a/libs/cua-driver/rust/tests/integration/v2/assets/test_page.html+++ /dev/null@@ -1 +0,0 @@-../../../../../cua-driver-fixtures/test_page.html\ No newline at end of filediff --git a/libs/cua-driver/rust/tests/integration/v2/test_tauri.py b/libs/cua-driver/rust/tests/integration/v2/test_tauri.pydeleted file mode 100644index e1bb465f56..0000000000--- a/libs/cua-driver/rust/tests/integration/v2/test_tauri.py+++ /dev/null@@ -1,155 +0,0 @@-"""Tauri desktop-test-app background automation tests — v2 harness.--The Tauri test app (v0.2.2) renders a 4-panel WKWebView UI and exposes-an HTTP API on port 6769 that logs all mouse/keyboard/clipboard events.--Primary assertion: AX tree state changes.-Cross-check: HTTP event log.--Run: python3 -m pytest test_tauri.py -v-"""--from __future__ import annotations--import subprocess-import sys-import time-import os--import pytest--try:- import requests as _requests- _REQUESTS_AVAILABLE = True-except ImportError:- _REQUESTS_AVAILABLE = False--_HERE = os.path.dirname(os.path.abspath(__file__))-sys.path.insert(0, _HERE)-from harness.driver import Driver-from harness import tree as Tree--_TAURI_API = "http://localhost:6769"---def _http_events(base_url: str = _TAURI_API) -> list[dict]:- if not _REQUESTS_AVAILABLE:- return []- try:- resp = _requests.get(f"{base_url}/events", timeout=2)- return resp.json() if resp.ok else []- except Exception:- return []---def _http_reset(base_url: str = _TAURI_API) -> None:- if not _REQUESTS_AVAILABLE:- return- try:- _requests.post(f"{base_url}/reset", timeout=2)- except Exception:- pass--[email protected](autouse=True)-def _reactivate_focus(focus_monitor):- _, pid = focus_monitor- subprocess.run(- ["osascript", "-e",- f'tell application "System Events" to set frontmost of (first process whose unix id is {pid}) to true'],- check=False,- )- time.sleep(0.4)---# ── tests ─────────────────────────────────────────────────────────────────────--class TestTauriButton:- def test_button_click(self, driver, tauri_app, ux_guard):- _, pid, base_url = tauri_app- if pid == 0:- pytest.skip("Tauri app pid not found")-- _http_reset(base_url)- wid = driver.find_window(pid)-- before = driver.get_window_state(pid, wid, query="button")- idx = before.find_element("click")- if idx is None:- idx = Tree.find(before.tree, role="AXButton")- assert idx is not None, f"Button not found:\n{before.tree}"-- driver.click(pid, wid, element_index=idx)- time.sleep(0.5)-- after = driver.get_window_state(pid, wid)- # Verify some state change happened (counter or AX value)- changed = after.tree != before.tree- print(f"\n tauri tree changed: {changed}")-- # Cross-check: HTTP events should contain a mouse event- events = _http_events(base_url)- print(f" tauri events: {len(events)}")- # Either AX tree changed or HTTP logged a mouse event- assert changed or any(- e.get("type") in ("mouse_down", "click", "mousedown") for e in events- ), f"No observable change after button click:\n{after.tree}"---class TestTauriTextInput:- def test_type_text(self, driver, tauri_app, ux_guard):- _, pid, base_url = tauri_app- if pid == 0:- pytest.skip("Tauri app pid not found")-- _http_reset(base_url)- wid = driver.find_window(pid)-- before = driver.get_window_state(pid, wid)- idx = before.find_text_field(skip_url_bar=False)- if idx is None:- idx = Tree.find(before.tree, role="AXTextField")- if idx is None:- idx = Tree.find(before.tree, role="AXTextArea")- assert idx is not None, f"Text input not found:\n{before.tree}"-- driver.click(pid, wid, element_index=idx)- time.sleep(0.3)- driver.hotkey(pid, ["cmd", "a"])- time.sleep(0.1)- driver.type_text_chars(pid, "tauri test")- time.sleep(0.6)-- after = driver.get_window_state(pid, wid)- text_in_tree = after.has_text("tauri test")- print(f"\n text in AX tree: {text_in_tree}")-- events = _http_events(base_url)- key_events = [e for e in events if e.get("type") in ("key_down", "keydown", "input")]- print(f" key events: {len(key_events)}")-- assert text_in_tree or len(key_events) >= len("tauri test"), (- f"Text not found in AX tree and insufficient key events:\n{after.tree}"- )---class TestTauriNoFocusSteal:- def test_multiple_actions_no_focus_steal(self, driver, tauri_app, ux_guard):- _, pid, base_url = tauri_app- if pid == 0:- pytest.skip("Tauri app pid not found")-- wid = driver.find_window(pid)-- # Take 3 actions in sequence — UX guard checks no focus steal- before = driver.get_window_state(pid, wid)- idx = Tree.find(before.tree, role="AXButton")- if idx is not None:- driver.click(pid, wid, element_index=idx)- time.sleep(0.3)-- driver.press_key(pid, "tab")- time.sleep(0.2)- driver.press_key(pid, "tab")- time.sleep(0.2)- # ux_guard teardown will assert_clean()diff --git a/libs/cua-driver/scripts/README.md b/libs/cua-driver/scripts/README.mdnew file mode 100644index 0000000000..f995a109b7--- /dev/null+++ b/libs/cua-driver/scripts/README.md@@ -0,0 +1,16 @@+# cua-driver scripts++Install, uninstall, local-build, and VM sync helpers for cua-driver.++| Script | Purpose |+| --- | --- |+| `install.sh` / `install.ps1` | Install released cua-driver binaries |+| `install-local.sh` / `install-local.ps1` | Build and install from this checkout |+| `uninstall.sh` / `uninstall.ps1` | Remove installed driver artifacts |+| `_install-common.sh` / `_install-common.psm1` | Shared install helper logic |+| `_install-rust.sh` / `_install-local-rust.sh` | Rust build/install internals |+| `sync-vm-worktree.sh` | Sync this checkout to verification VMs and pull artifacts back |+| `post-install-hints.txt` | User-facing hints printed by install scripts |++Keep source commits host-owned. Verification machines should sync from this+checkout and return artifacts, not push code.diff --git a/libs/cua-driver/scripts/sync-vm-worktree.sh b/libs/cua-driver/scripts/sync-vm-worktree.shnew file mode 100755index 0000000000..c110abab30--- /dev/null+++ b/libs/cua-driver/scripts/sync-vm-worktree.sh@@ -0,0 +1,164 @@+#!/usr/bin/env bash+# Sync this host checkout to verification VMs and pull logs/artifacts back.+#+# Source changes should flow from this host to VMs. The guarded pull-code mode+# exists only for recovering intentional VM-side edits.+set -euo pipefail++usage() {+ cat <<'USAGE'+Usage:+ sync-vm-worktree.sh push <ssh-target> [remote-dir]+ sync-vm-worktree.sh pull-artifacts <ssh-target> [remote-dir]+ ALLOW_PULL_CODE=1 sync-vm-worktree.sh pull-code <ssh-target> [remote-dir]++Modes:+ push+ Sync this host checkout to a VM. This is the normal way to update Linux+ and Windows verification machines.++ pull-artifacts+ Pull VM logs/artifacts from <remote-dir>/vm-out into this host checkout.++ pull-code+ Pull source changes made on a VM back to this host checkout. This is+ intentionally gated by ALLOW_PULL_CODE=1 because this host is the only+ checkout that should commit or push.++Environment:+ REMOTE_ARTIFACT_DIR Remote artifact directory, relative to remote-dir.+ Default: vm-out+ RSYNC_SSH SSH command for rsync. Default: ssh+ SYNC_TRANSPORT rsync or tar. Default: rsync+ REMOTE_OS posix or windows. Only needed for tar push mkdir.+ Default: posix+USAGE+}++if [[ $# -lt 2 || $# -gt 3 ]]; then+ usage >&2+ exit 2+fi++mode="$1"+target="$2"+remote_dir="${3:-~/cua}"+remote_artifact_dir="${REMOTE_ARTIFACT_DIR:-vm-out}"+rsync_ssh="${RSYNC_SSH:-ssh}"+transport="${SYNC_TRANSPORT:-rsync}"+remote_os="${REMOTE_OS:-posix}"++script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+repo_root="$(cd "$script_dir/../../.." && pwd)"+artifact_root="$repo_root/libs/cua-driver/docs/vm-artifacts"+target_slug="$(printf '%s' "$target" | tr -c 'A-Za-z0-9_.-' '_')"+timestamp="$(date -u +%Y%m%dT%H%M%SZ)"++exclude_args=(+ --exclude=.git/+ --exclude=.DS_Store+ --exclude='._*'+ --exclude=target/+ --exclude=node_modules/+ --exclude=.venv/+ --exclude=__pycache__/+ --exclude='*.pyc'+ --exclude=dist/+)++tar_exclude_args=(+ --exclude ./.git+ --exclude ./.DS_Store+ --exclude '._*'+ --exclude ./target+ --exclude ./node_modules+ --exclude ./.venv+ --exclude '*/__pycache__'+ --exclude '*.pyc'+ --exclude ./dist+)++remote_mkdir() {+ case "$remote_os" in+ windows)+ "$rsync_ssh" "$target" \+ "powershell -NoProfile -Command \"New-Item -ItemType Directory -Force '$remote_dir' | Out-Null\""+ ;;+ posix)+ "$rsync_ssh" "$target" "mkdir -p $remote_dir"+ ;;+ *)+ echo "REMOTE_OS must be posix or windows, got: $remote_os" >&2+ exit 2+ ;;+ esac+}++push_rsync() {+ ssh_cmd=${remote_dir/#\~/"\$HOME"}+ "$rsync_ssh" "$target" "mkdir -p $ssh_cmd"+ rsync -az --delete -e "$rsync_ssh" "${exclude_args[@]}" "$repo_root/" "$target:$remote_dir/"+}++push_tar() {+ remote_mkdir+ COPYFILE_DISABLE=1 tar "${tar_exclude_args[@]}" -czf - -C "$repo_root" . \+ | "$rsync_ssh" "$target" "tar -xzf - -C \"$remote_dir\""+}++pull_artifacts_rsync() {+ rsync -az -e "$rsync_ssh" "$target:$remote_dir/$remote_artifact_dir/" "$dest/"+}++pull_artifacts_tar() {+ "$rsync_ssh" "$target" "tar -czf - -C \"$remote_dir/$remote_artifact_dir\" ." \+ | tar -xzf - -C "$dest"+}++pull_code_rsync() {+ rsync -az --backup --suffix=".vm-backup-$timestamp" -e "$rsync_ssh" \+ "${exclude_args[@]}" "$target:$remote_dir/" "$repo_root/"+}++pull_code_tar() {+ "$rsync_ssh" "$target" "tar -czf - -C \"$remote_dir\" ." \+ | COPYFILE_DISABLE=1 tar "${tar_exclude_args[@]}" -xzf - -C "$repo_root"+}++case "$mode" in+ push)+ case "$transport" in+ rsync) push_rsync ;;+ tar) push_tar ;;+ *) echo "SYNC_TRANSPORT must be rsync or tar, got: $transport" >&2; exit 2 ;;+ esac+ ;;++ pull-artifacts)+ dest="$artifact_root/$target_slug/$timestamp"+ mkdir -p "$dest"+ case "$transport" in+ rsync) pull_artifacts_rsync ;;+ tar) pull_artifacts_tar ;;+ *) echo "SYNC_TRANSPORT must be rsync or tar, got: $transport" >&2; exit 2 ;;+ esac+ echo "Pulled artifacts into $dest"+ ;;++ pull-code)+ if [[ "${ALLOW_PULL_CODE:-}" != "1" ]]; then+ echo "Refusing to pull VM source without ALLOW_PULL_CODE=1." >&2+ exit 2+ fi+ case "$transport" in+ rsync) pull_code_rsync ;;+ tar) pull_code_tar ;;+ *) echo "SYNC_TRANSPORT must be rsync or tar, got: $transport" >&2; exit 2 ;;+ esac+ ;;++ *)+ usage >&2+ exit 2+ ;;+esacdiff --git a/libs/cua-driver/test-harness/MODALITY_TEST_SUITE.md b/libs/cua-driver/test-harness/MODALITY_TEST_SUITE.mddeleted file mode 100644index 8594983a7b..0000000000--- a/libs/cua-driver/test-harness/MODALITY_TEST_SUITE.md+++ /dev/null@@ -1,627 +0,0 @@-# Modality test suite and harnesses--This is an internal contributor runbook for the Cua Driver modality test suite.-It used to live in the public docs reference tree, but it describes the harness,-CI lanes, recorder scripts, and regression matrices rather than public product-behavior. User-facing limits harvested from this page belong in-`docs/content/docs/reference/cua-driver/limits.mdx`.--This document describes the cua-driver test-harness: a cross-OS, cross-toolkit-rig that exercises every driver action against a controlled application and-scores each action **twice** — did it land an effect, and did it keep the app in-the background. It is the authoritative map of *which driver mechanisms work-where*, and why each limitation manifests in the mode/scope it does.--## 1. Overview — what the suite validates--The suite validates two things, per driver action, per surface:--1. **The no-foreground / background contract** — cua-driver's differentiator.- The driver is supposed to drive an app **without bringing it to the- foreground** and **without moving the visible cursor** (it dispatches via the- accessibility tree or via background coordinate injection, and paints a- per-session overlay cursor instead of warping the real pointer). The suite- measures, for every action, whether the target app stayed in the background- (`held`) or jumped frontmost (`STOLE FOCUS`).--2. **Cross-toolkit dispatch coverage** — whether an action *actually lands* on- each UI toolkit. The same 8-action × 6-control matrix runs against WPF,- WinUI3, WebView2, Electron (Windows), GTK3 + Electron (Linux), and AppKit,- SwiftUI, WKWebView, Electron (macOS). Toolkits route input differently (UIA- Invoke vs AT-SPI `doAction` vs AX actions vs synthetic pixels vs composition- input-sites), so "click works" is a per-toolkit, per-dispatch-path fact, not- a global one.--**Scope caveat — this tests driver *mechanisms*, not agent task success.** A cell-that reads ✓ means the driver delivered the action and the target's own-instrumented state changed (a checkbox toggled, a slider moved, `last_action=`-updated). It does **not** mean an LLM agent located the control correctly or-chose the right action. Agent locator quality and agent self-judgment are-explicitly kept on separate axes (see §7) so a bad locator can never mask — or be-masked by — a driver regression.-------## 2. How the suite works — the modality recorder--The core instrument is the **modality recorder** (one per OS:-`windows/wpf-recorder.ps1`, `linux/lin-rec.py` + `lin-rec-electron.py`,-`macos/mac-rec.py`). Each run produces one annotated screen recording:--- **Test harness on the LEFT**, **`cua-driver-panel` dashboard on the RIGHT**- (always-on-top). The dashboard is a tiny HTML page served over loopback and- shown via a `chrome --app` (Windows/macOS) or WebKit (Linux) window; it polls a- `status.json` the recorder writes (~5 Hz) and renders the live verdict.-- The dashboard shows the run's **modality**, a live **foreground/background- indicator** (current frontmost window title), and a **per-action** dual verdict.-- The pink/cyan overlay is the driver's **per-session agent cursor** — proof the- driver is acting without a real pointer move.--### The 5 modality modes (per surface)--Action rung × foreground/background × scope = 5 single-modality runs:--| Mode | Action rung | Dispatch | Foreground? | Scope |-|------------------|-----------------------|-----------------|-------------|-----------|-| `ax-fg` | element `ax` action (accessibility) | `element_index` | app kept FRONT | window |-| `ax-bg` | element `ax` action | `element_index` | app stays BACKGROUND (contract measured) | window |-| `px-fg` | element `px` action (pixel) | pixel coords | app kept FRONT | window |-| `px-bg` | element `px` action | pixel coords | app stays BACKGROUND (contract measured) | window |-| `px-desktop` | element `px` action | screen pixels (window-less) | FRONT | desktop |--The run names use `ax`/`px` as labels for the-element-dispatch vs pixel-dispatch runs; **`capture_mode` itself is now-deprecated and ignored** — `get_window_state` returns both the tree and a-screenshot regardless of what you pass. The distinction the recorder exercises-is the **action rung** (the element `ax` action addressing by `element_index`-vs the element `px` action addressing by pixel `x,y`), not a capture toggle.--The fg/bg split only matters for the contract (measured in the `*-bg` runs).-`px-desktop` sets `capture_scope=desktop` and issues window-less,-screen-absolute actions.--### Dual scoring per action--Every action row carries **two independent measurements** (rendered as two-badges on the dashboard):--1. **✓ worked / ✗ no-op (effect)** — did the action change the harness's *own*- state? Read back after the action via the harness's instrumented status- labels (`agreed=`, `slider_value=`, `last_action=`, `mirror=`, `menu_action=`,- `scroll_offset=`).-2. **held / STOLE FOCUS (contract)** — sampled for ~1.5 s after the action: did- the target window's title become the frontmost window? In fg modes this badge- is replaced by `foreground` (steal is expected/intentional).--The dashboard tally at the bottom is the live count: `effects: N/M actions-changed the app` and, in bg modes, `focus contract: S/A stole focus`.--### How the verifier reads each toolkit's instrumented labels--The harnesses all expose the **same set of status labels** (single source of-truth: `shared/scenarios.json` + `shared/web/index.html`), but the verifier-reads them differently per platform:--| Platform / toolkit | How the verifier reads state |-|-------------------------------|------------------------------|-| Windows WPF / WinUI3 | **UIAutomation** — finds the harness window by title substring, scans `ControlType.Text` descendants, regex-extracts `agreed=`, `slider_value=`, `last_action=`, `mirror=`, `menu_action=`, `scroll_offset=` |-| Windows WebView2 / Electron | Same UIA scan, but the window is matched by **substring** because Chromium titles carry a `[cdp=NNNN]` suffix; the web-AX tree only surfaces with `--force-renderer-accessibility` |-| Linux GTK3 | The harness writes a **state file** (`/tmp/cua-lin-state.json`); the recorder's `hstate()` reads it (AT-SPI text isn't reliably enumerable for all labels) |-| Linux Electron | Chromium **web-AX over AT-SPI** (`--force-renderer-accessibility`), read from the `get_window_state` tree |-| macOS AppKit / SwiftUI | **AX** — status labels render as AX text nodes in `tree_markdown`; `do_native()` drives the action, then re-reads the labels |-| macOS WKWebView / Electron | Chromium/WebKit web-AX text (`WEBISH` path), same label contract |--The per-action `verify(action, before, after)` logic is consistent across-recorders, e.g.: `click` → `agreed` flipped; `double` → `last_action ==-double_click`; `right` → `last_action == right_click` **or** `menu_action`-changed; `drag` → `slider` increased; `scroll` → `scroll_offset` increased;-`setval` → `mirror == set-by-cua`; `type` → `mirror` contains `typed-by-cua`;-`press-key` (Tab) → no asserted effect (`na`).--> **Honesty principle (FINDINGS.md):** the harness/recorder stays honest — real-> driver gaps are documented as driver work, fixed in the driver first, then-> re-recorded. The recorder does not paper over a no-op.--### Session lifecycle & disposal--Each recorder declares a **session** (the agent-cursor identity, e.g. `d1` /-`macd-…`) via `start_session`, drives every action under it, and must-`end_session` it on teardown so the cursor + per-session state are disposed —-not leaked.--- **Disposal is double-guarded.** `end_session` and the **idle-TTL sweep**- (`evict_idle`, default **300 s**, overridable via- `CUA_DRIVER_RS_SESSION_IDLE_TTL_SECS`, swept every 30 s) both dispose through- the same `fire_session_end` hook fan-out: agent cursor removed, recording- stopped, per-session config cleared. A reused id starts **fresh, not poisoned**.-- **Test:** `cua-driver-core/tests/session_lifecycle.rs` asserts start → use →- `end_session` disposes (and a reused id starts fresh), and that idle-TTL- eviction disposes the **same** way — verified via the disposal-hook contract- both paths fan out to (the concrete cursor/config registries are a daemon-layer- assertion). It passes a short TTL straight to `evict_idle`; the **300 s- production default is never lowered** for tests.-- **The recorders dispose explicitly** — `end_session` in a `try/finally` before- they kill the daemon (`mac-rec.py` via an `atexit` net, since it leaves the- daemon running) — so a session can't leak even if a run errors.-------## 3. Test-app harnesses--Each harness is a self-contained host app the driver drives. They share fixtures-via `shared/scenarios.json` (AutomationIds / AX identifiers / DOM ids /-window titles) and, for the web hosts, `shared/web/index.html`, so identifiers-never drift between apps and the Rust integration tests under-`rust/crates/cua-driver/tests/harness_*.rs`. Built outputs stage into-`rust/test-apps/harness-{name}/` via `build/{macos.sh,windows.ps1,linux.sh}`.--### The 6 parity controls--The "control-parity pass" brought every surface up to the WPF baseline of **6-controls**, so the same 8-action matrix is exercisable everywhere:--1. **checkbox** (`chk-agree*`) → `agreed=`-2. **click-target** (left / right / double) → `last_action=`, `clicks=`-3. **slider** (`sld-value`) → `slider_value=`-4. **scroll-target** (tall `scroll-tall` region) → `scroll_offset=`-5. **text-input** (`txt-input`, mirrored) → `mirror=`-6. **context-menu** (`btn-context`) → `menu_action=`--(Plus a `counter=` increment button used as a generic AX-press probe.) WPF-carries an extended scenario set on top (modal MessageBox, owned/layered popups,-native Win32 child HWND, menu bar, combo/list boxes) — it is the "fullest-harness".--### Per OS / toolkit--| OS | Harness | What it is | Build | Source |-|---------|-----------|------------|-------|--------|-| Windows | **WPF** | WPF / UIA host — popups, layered windows, modal MessageBox, HwndHost native child | `build/windows.ps1` → `harness-wpf` | `apps/windows/wpf/` (C#/XAML) |-| Windows | **WinUI3** | WinUI3 unpackaged — DirectComposition popups, XAML Popup primitive, CommandBarFlyout; HWND subclassed to translate `WM_VSCROLL` → `ChangeView` | `build/windows.ps1` → `harness-winui3` | `apps/windows/winui3/` |-| Windows | **WebView2** | WPF host + `Microsoft.Edge.WebView2` loading the shared DOM; CDP on `:9222` | `build/windows.ps1` → `harness-webview` | `apps/windows/webview2/` |-| Windows | **Electron** | Electron host loading the same shared DOM; CDP on `:9223` | `apps/cross-platform/electron/build.ps1` | `apps/cross-platform/electron/` |-| Linux | **GTK3** | PyGObject GTK3 app; AT-SPI exposes accessible **NAME** (not an AutomationId); writes `/tmp/cua-lin-state.json` for the verifier | `build/linux.sh` → `harness-gtk3` | `apps/linux/gtk3/main.py` |-| Linux | **Electron** | Same Electron bundle run from `node_modules` under Xvfb + dbus with `--force-renderer-accessibility` | run via `lin-run-electron.sh` | `apps/cross-platform/electron/` |-| macOS | **AppKit** | Cocoa — NSButton, NSScrollView, NSMenu, NSSlider, menubar item | `build/macos.sh` → `harness-appkit` | `apps/macos/appkit/main.swift` |-| macOS | **SwiftUI** | SwiftUI — `.popover()`, `.contextMenu`, declarative `Slider`/`Toggle`/`ScrollView` | `build/macos.sh` → `harness-swiftui` | `apps/macos/swiftui/main.swift` |-| macOS | **WKWebView** | Apple WebKit host loading the shared DOM — the native analogue of WebView2 | `build/macos.sh` → `harness-wkwebview` | `apps/macos/wkwebview/main.swift` |-| macOS | **Electron** | Same shared DOM under Chromium-on-macOS | `apps/cross-platform/electron/build.sh` | `apps/cross-platform/electron/` |--### Instrumented labels (the verifier contract)--Every harness updates these labels in response to a successful action; the-verifier asserts on them:--| Label | Set by | Verifies |-|------------------|--------------------------------|----------|-| `last_action=` | click-target (left/right/double)| click / double / right landed |-| `clicks=` | click-target | click count |-| `slider_value=` | slider | drag / set_value on a range control |-| `scroll_offset=` | scroll-target | scroll landed |-| `agreed=` | checkbox | click toggled a toggle |-| `menu_action=` | context-menu item | right-click → context menu → item invoked |-| `mirror=` | text-input | type / set_value into a text field (`typed-by-cua` / `set-by-cua`) |-| `counter=` | increment button | generic AX-press probe |-------## 4. Action rungs & scopes--### Action rung (how the action addresses the target)--`get_window_state` is perception-mode-agnostic: it returns **both** the-accessibility tree and a screenshot by default (`include_screenshot:false` skips-the screenshot grab — a perf opt-out, not a modality choice). `capture_mode` is-**deprecated and ignored** — there is no `ax`/`vision` capture toggle anymore.-The modality is chosen at **action time** by how you address the target, and-that one choice selects the rung:--| Rung | How the agent addresses the target | How actions dispatch |-|------|------------------------------------|----------------------|-| element `ax` action | by `element_index` (read off the tree) | UIA Invoke (Windows) / AT-SPI `doAction` (Linux) / AX actions (macOS) — backgroundable, driver-verifiable |-| element `px` action | by pixel `x,y` (read off the screenshot in the same response) | synthetic/injected pointer events at (x,y) — best-effort, caller-confirmed |--> **Deprecated aliases.** `som` (the old set-of-marks mode) and `screenshot` are-> no longer advertised capture modes — like `ax`/`vision`/`capture_mode` itself,-> they still decode but are **ignored** (both the tree and the screenshot come-> back regardless). The modality recorder's runs exercise the element-dispatch-> path and the pixel-dispatch path, so the recorder runs and the results matrices-> below are unchanged — they are framed here as the element `ax` action and the-> element `px` action rungs.--### Capture scope (`capture_scope`)--| Scope | Meaning |-|-----------|--------------------------------|-| `window` | single target window (default) |-| `desktop` | full display |--### Why mode/scope determines where a limitation shows--This is the central organizing idea (from `CUA_DRIVER_LIMITATIONS_AND_TEST_MATRIX.md`):--- **The element `ax` action (`element_index`) path.** Limits in the element path- (a toolkit with no `doAction` for right-click, a WinUI3 composition input-site- the WPF path can't reach, an AXValue a SwiftUI slider rejects) surface in this- rung — the `ax-` runs.-- **The element `px` action (pixel) path.** Limits in the pixel path (GTK- dropping synthetic X events, NSButton ignoring a synthetic mouseDown, a 2×- Retina coordinate conversion) surface in this rung — the `px-` runs.-- **fg/bg** isolates the *contract* (only `*-bg` measures focus steal).-- **`desktop` scope** is where window-less / coordinate-space-conversion bugs- surface (e.g. the macOS 2× Retina desktop path, the macOS no-windowless-click- gap).-------## 5. Action × control matrix--**8 actions** × **6 controls**. Each modality run filters the plan:-`set_value` is dropped in the `px` modes (the pixel-dispatch runs — it's-AX-only); `px-desktop` keeps only `{click, scroll, type, press-key}`-(window-less screen actions).--| Action ↓ / Control → | checkbox | click-target | slider | scroll-target | text-input | context-menu |-|----------------------|:--------:|:------------:|:------:|:-------------:|:----------:|:------------:|-| **click** | ● (toggle) | ● (left) | | | | |-| **double-click** | | ● | | | | |-| **right-click** | | ● (web/WinUI3 records `last_action`) | | | | ● (opens menu) |-| **drag** | | | ● (thumb→X) | | | |-| **scroll** | | | | ● | | |-| **set_value** (AX) | | | ● (range value) | | ● (`set-by-cua`) | |-| **type** | | | | | ● (`typed-by-cua`) | |-| **press-key** (Tab) | | | | | ● (focus move; no asserted effect) | |--Notes:-- WPF has a **dedicated context-menu control**; WinUI3 / web harnesses have none,- so right-click is aimed at the click-target and verified via- `last_action=right_click`.-- SwiftUI's AX has no distinct right-press action, so right-click on its button- coerces to a normal AXPress — right-click semantics there are covered by the- dedicated `context_menu` scenario.-------## 6. RESULTS MATRIX (the core)--Per cell: **✓ landed · ✗ no-op · ⚠ guarded/fail-loud** for the *effect*, with the-reason. The **Contract** column is the focus verdict measured in the `*-bg` runs.-"land" = the harness's own instrumented label changed. Verification status is-drawn from FINDINGS.md, the limitations doc, and the fix-commit bodies; runtime-status is flagged where a commit deferred it.--> Legend: ✓ lands · ✗ no-op (honest) · ⚠ guarded clean-fail / fail-loud ·-> — not exercised · n/a not in plan / no asserted effect.--### macOS--Contract: **HOLDS — 0/8 stole** on AppKit in both `ax-bg` and `px-bg`-(pixel left-click/double/drag landed while Chrome stayed frontmost). WKWebView-also holds — confirming the Windows Chromium steal is Windows-specific, not-WebKit-on-macOS.--| Harness | mode | click | double | right | drag | scroll | set_value | type | press-key | Contract | Notes |-|---------|------|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|------|-| AppKit | `ax` | ✓ | ✓ (fixed bg, `click_at_xy_with_window_local`, `80b4e3d7`) | ✓ (AXShowMenu→delivering-pixel fallback, `80b4e3d7`) | ✓ | ⚠ element scroll no-ops on content-height containers (use x,y) | ✓ CFNumber NSSlider (`92d8aebf`, verified 0→50) | ✓ | n/a | **0/8 stole** | ax-bg 5/7 effects |-| AppKit | `px` | ⚠ **no-op on standard NSButton even frontmost** (modal mouseDown reads window-server queue, not `postToPid` → use `element_index`) | ✓ | ✓ now fires `rightMouseDown` (window-number + button + primer, `e7145c18`) | ✓ | ✓ via per-pid pixel-wheel `scroll_wheel_at_xy` (`e7145c18`) | n/a | ✓ | n/a | **0/8 stole** | needs target frontmost; px-bg 4/6 |-| SwiftUI | `ax` | ✓ | ✓ | ✓ (via dedicated context-menu) | ✗ slider drag is a composition no-op | ⚠ nested scroll no-op | ⚠ **AXValue unsettable** (`-25200`); needs AXIncrement/AXDecrement stepping (`d2b29230`) | ✓ | n/a | **0/8 stole** | full 6-control parity confirmed live |-| WKWebView | `ax`+`px` | ✓ | ✓ | ✓ | ✓ slider (pixel drag at live frame) | ⚠ **page scroll works, nested `overflow:auto` div no-op** (div has no `tabindex` → never keyboard-focusable; needs pixel-wheel) | ✗ **AXValue set, no DOM `input` event** (web mirror unchanged) | ✓ | n/a | **HOLDS** | native analogue of WebView2. Recorder fix (`mac-rec.py`): web dispatch now uses the **live element frame**, not a stale hardcoded coord map that was ~185px off → every WKWebView pixel action used to miss. ax-fg re-measured **5/6** (click/double/right/drag/type land; only `set_value` the honest web no-op). |-| Electron (mac) | `ax`/`px` | ✓ | ✓ | — | — | ✗ (web scroll, same as WKWebView) | — | ✓ | n/a | holds (Chromium-on-mac does **not** steal like Windows) | local-only recordings |--macOS gotchas filed as driver bugs: **`end_session` poisons a reused session id**-(subsequent actions silently no-op); **AppKit window height drifts between-launches** (store targets as window-local points, convert to live screenshot px).--### Linux--| Harness | mode | click | double | right | drag | scroll | set_value | type | press-key | Contract | Notes |-|---------|------|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|------|-| GTK3 | `ax` (AT-SPI) | ✓ left-click via hit-test→`doAction` (`7d358283`) | ✗ no `doAction` equiv | ✗ no `doAction` equiv | ⚠ value-only (no Action) — driven via `set_value` | ⚠ surfaced now (`is_indexable = actions \|\| has_value`); driven via `set_value` | ✓ slider/scroll value widgets now surface | ✓ | n/a | ax-bg **1/8 stole** (only `set_value`; corrected via genuine-anchor baseline `5c0a1d3c` — was 3/8) | left/set_value/type land |-| GTK3 | `px` (Xvfb) | ✗ | ✗ | ✗ | ✗ | ✗ | n/a | ✗ | n/a | **0/7 stole** | **GTK drops synthetic XSendEvent**; XTEST core events don't reach its XInput2 path |-| GTK3 | `px` (**real Xorg**) | ✓ | ✓ | ✓ | ✓ | ✓ | n/a | ✓ | n/a | holds focus | right/double/middle-click + scroll **land via uinput/XInput2-MPX + shield-grab** (`79e546ca`); capability auto-detected via `real_pointer_input_available()` — **runtime-verified on a real Xorg server (dummy-driver): the probe flips TRUE, all four actions LAND and HOLD focus (`_NET_ACTIVE_WINDOW` unchanged), confirmed by the harness oracle + a middle-click PRIMARY paste. Xvfb can't bind uinput as an X slave, so the path auto-skips there.** |-| Electron (Linux) | `ax` | ✓ click | ✗ | ✗ | ✓ drag | ✗ scroll resolves but no-op (AT-SPI synthetic) | ✗ | ✓ type | n/a | ax-bg **2/8 stole** (`set_value` + `drag`; was 3/8 — recorder baseline artifact, `5c0a1d3c`) | drag lands a value-change but its synthetic window-coord activates Chromium (never reaches the slider thumb); still holds far better than Windows Electron's reported 7/8 (also an artifact) |-| Electron (Linux) | `px` | ✗ | ✓ pixel double-click fires on click-target | ✗ | ✗ | ✗ | n/a | ✗ | n/a | px-bg **6/7 stole** (pixel dispatch foregrounds Chromium) | |-| GTK4 (gnome-calculator) | `ax` (AT-SPI) | ✓ via `doAction` (coordinate-free; lands without a valid `frame`) | ✗ no `doAction` equiv | ✗ no `doAction` equiv | ⚠ value-only (no Action) — driven via `set_value` | ⚠ driven via `set_value` | ✓ | ✓ | n/a | holds | **GTK4 coord fix required for `frame`/agent-cursor.** GTK4's AT-SPI bridge returns `Component.GetExtents(SCREEN)=(0,0)` for every widget (GNOME/gtk #1564/#1739 a11y rework). Fix: queries `CoordType::Window` (GTK4 reports correctly per-widget) and reconstructs screen as `x11_window_origin + _GTK_FRAME_EXTENTS.(left,top) + WINDOW_xy`, gated on presence of `_GTK_FRAME_EXTENTS` so non-GTK toolkits (Qt) keep their correct SCREEN path. Verified live: button "7" = window(55,27) + inset(61,55) + WINDOW(16,293) = screen(132,375). `doAction` element clicks land regardless (coordinate-free). GNOME VM lane only. |-| GTK4 (gnome-calculator) | `px` (real Xorg) | ✓ (after GTK4 coord fix) | ✓ | ✓ | ✓ | ✓ | n/a | ✓ | n/a | holds | Pixel coords reliable **only after** WINDOW+`_GTK_FRAME_EXTENTS` reconstruction; without the fix, cursor/frame collapses to the window corner. GNOME Shell requires real console Xorg (software GLX) — not Xvnc. |--### Windows--Contract: **HOLDS on WPF / WinUI3 / WebView2.** **Electron** — see the caveat-below (the prior "7/8 stole" is a recorder measurement artifact).--| Harness | mode | click | double | right | drag | scroll | set_value | type | press-key | Contract | Notes |-|---------|------|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|------|-| WPF | `ax` | ✓ (UIA Toggle) | ✓ | ⚠ off-screen at 556px reflow → clean no-op (guarded; UIA Invoke works off-screen) | ✓ slider 0→48 | ⚠ off-screen reflow → guarded no-op | ✓ (`mirror=set-by-cua`) | ✓ (recorder focuses the box first) | n/a | **2/8 stole** | "fullest harness"; ax-bg ~5/7 effects |-| WPF | `px` | ✓ | ✓ | ⚠ guarded (`point_in_window_bounds` refuses off-screen point; off-screen controls now ScrollIntoView + actuate via ancestor ScrollPattern, `c3efb587`, runtime-verified counter 0→1) | ✓ | ⚠ off-screen reflow | n/a | ✓ | n/a | 1–2/7 stole | off-screen guard prevents taskbar misfire |-| WinUI3 | `ax` | ✓ left-click (UIA Invoke) | ✓ **now LANDS** — double UIA-Invoke under `WS_EX_NOACTIVATE` guard (`531aa6de`, runtime-verified live) | ⚠ **fail-loud `background_unavailable`** — no contract-safe path (pen injection steals; WM_*BUTTON doesn't land) | ⚠ fail-loud (same composition input-site gap) | ✓ (HWND subclass: `WM_VSCROLL`→`ChangeView`) | ✓ | ✓ | n/a | **0/7 stole** | gap needs a WinUI3-specific composition InputSite path |-| WebView2 | `ax` | ✗ checkbox below the fold (web won't scroll in ax) | ✓ | ✓ records `last_action=right_click` | ✓ | ✗ host HWND doesn't route scroll to renderer | ✓ | ✓ | n/a | **0/7 stole** | needs `--force-renderer-accessibility`; 4/7 effects |-| Electron | `ax` | ✓ | ✓ | — | ✓ | ✗ web scroll | ✓ | ✓ | n/a | **see caveat** | 5/7 effects |--Other Windows: **UIA element-cache use-after-free** (concurrent `ax` sessions on-the same window) — fixed with a `RetainedElement` retain-under-lock guard-(`d95b89a1`); proven by a deterministic `cache_uaf_repro` test (pre-fix path-takes a real access violation, fixed path survives a 6-thread stress loop,-`531aa6de`). Recording the VM needs **Session 2 attached** (`tscon … /dest:console`)-and **ffmpeg on PATH before `serve`**.--> **Windows Electron contract caveat — RESOLVED.** FINDINGS.md records Electron as-> stealing **7/8** in `ax-bg`. That was a **recorder MEASUREMENT ARTIFACT**: the-> recorder re-asserted its dashboard panel with `SetWindowPos` (z-order only, no-> activation), so it never held a real foreground baseline — after the first inject-> action click-activated Chromium, the harness stayed frontmost and every later-> step false-positived as a steal. The recorder was fixed (`49bdb41b`) to genuinely-> `SetForegroundWindow` a real non-harness anchor (mspaint) before each step.-> **Re-measured: Electron `ax-bg` = 0/8 (held)** — corroborated by `baseline.log`-> (anchor held ×8), an independent `metric.log` `MEASURE=0/8`, and frame f03 (actions-> landed while the harness stayed non-foreground). So **Electron HOLDS the contract**,-> consistent with Linux/macOS Electron. (The committed `matrix-electron-ax-bg.mp4`-> still shows the old 7/8 until re-recorded with the fixed recorder.)-------## 7. Vision-agent coordinate test--`vision-agent-test/` (`vision_agent_test.py`, `010fdd78`) is a newer test that-hits the driver **the way a vision agent actually does** — and removes the-modality recorder's overfit (hand-tuned window-local points run through a private-ratio, which never exercises the driver's image→screen mapping).--**Invariant under test:** *the pixel an agent reads off the returned screenshot is-the pixel that gets clicked* — verified by the target's own instrumented state-changing.--The loop (no cheating in locate or click):-1. **capture** — `get_window_state` (window — reads the screenshot it returns by- default) / `get_desktop_state` (desktop, true pixels): the exact image an agent- receives.-2. **locate** — a deterministic pixel **in the returned-image coordinate space**- (`PixelRegistryLocator`: a pre-measured pixel read off the real PNG, with a- dims-guard that fails loud if the pinned geometry drifts). No `element_index`,- no hand-converted window-local points. `locate(image, target, dims) → (x,y)`- is pluggable (an LLM locator can drop in later).-3. **act** — `click`/`right_click`/`scroll` at that pixel (scope matches capture).-4. **verify** — the harness oracle (`last_action=`, `clicks=`, …) → objective- pass/fail. A coordinate mis-map leaves the oracle unchanged → FAIL.--Run: `python3 vision_agent_test.py {wkwebview-click-window|wkwebview-click-desktop|appkit-click-window|safari-learnmore-desktop|all}`.--**Three axes kept separate** so none can mask the others: (a) the **driver-coordinate invariant** (this test, deterministic → the regression guard); (b)-**agent locator quality** (future, LLM, same `locate()` signature, scored-separately, never gates the regression); (c) **agent self-judged success with no-oracle** (future, isolated track).--What the deterministic version caught that the modality suite structurally-couldn't:-- **2× Retina desktop path now correct + guarded** — pixel (340,1358) in the- 3024×1964 desktop PNG converts to screen-point (170,679) and lands (oracle +- a real Safari navigation confirm it). This turns the Retina escape into a- permanently-guarded one-liner.-- **Pixel-click is a no-op on AppKit `NSButton` even frontmost** (modal- mouseDown reads the window-server queue, not the per-pid `postToPid` queue) —- the modality suite drives this via AXPress, so it never saw it.-- **The pixel path requires the target app frontmost** (the AX path doesn't).-- **The AppKit harness window AX returns only the menu bar** — its `clicks=`- oracle is unreachable that way; WKWebView exposes it fine.-------## 8. Known overfitting caveats--The suite's own authors flag two ways the modality recorder could have lied, and-the de-risking work for each:--**(a) Recorder hand-coords masked the desktop 2× Retina bug.** The modality-recorder stores targets as hand-tuned window-local points and converts them-through a private ratio. That conversion is *self-consistent* with the driver's-own assumption, so it never exercised the driver's real image→screen mapping —-and the **desktop-scope 2× Retina off-by-backing-scale bug** (a center-pixel pick-warping to the corner) slipped through every `px-desktop` run. De-risk: the-**vision-agent coordinate test** (§7) reads a pixel straight off the returned PNG-with a dims-guard, caught the bug, and now guards it (`80b4e3d7` fixes the desktop-branch to divide x,y by the native/logical ratio; `010fdd78` adds the regression-test).--**(b) Recorder contract measurement had no real foreground baseline → the false-"Electron 7/8 stole".** The recorder judges the contract by checking whether the-target's title is the frontmost window, but it never established a genuine,-distinct foreground window to begin with — so Windows Electron read as stealing-**7/8** when the corrected understanding is that Chromium-on-Windows holds the-contract (matching macOS/Linux Electron). De-risk: the **recorder-contract fix**-(`49bdb41b`) now genuinely `SetForegroundWindow`s a real anchor before each step;-**re-measured Electron `ax-bg` = 0/8 (held)**, so the claim is now verified and the-legacy FINDINGS "7/8" is a confirmed measurement artifact. The Linux recorders had-the same flaw — also fixed (`5c0a1d3c`): GTK3 + Linux-Electron `ax-bg` corrected-3/8 → 1/8 (only `set_value` genuinely steals); macOS was already fine.--> Both caveats reflect the same lesson: a recorder that hand-feeds the driver its-> own assumptions can hide bugs in exactly the path a real agent uses. The-> deterministic, oracle-checked vision-agent test and the re-baselined contract-> measurement are the two structural fixes.-------## 9. Edge cases — real closed-source apps--The synthetic harnesses pin down the heuristics; real apps surface behaviours a-single-process test window never can. These were found driving live closed-source-apps (Finder, System Settings, Calculator, Safari on macOS; Calculator/Notepad UWP,-Edge, Explorer on Windows) and are why the suite is a floor, not the ceiling.--### macOS (Finder, System Settings, Calculator, Safari)--1. **Pixel click hits the right pixel but the wrong window.** With ~8 overlapping- same-pid Finder windows, a crosshair dead-centre on the target file still posts to- *screen* coordinates, so an occluding sibling intercepts it → no-op. A per-window- screenshot masks it. Strong argument for the `ax`/element path, which is z-order- independent.-2. **`set_value` and `type_text` both falsely succeed on a background search field**- (System Settings, SwiftUI). `set_value` writes `AXValue` (text appears, the search- action never fires); `type_text` posts keys that a non-first-responder window drops.- Neither drives the field; **both report success.**-3. **Finder column filenames don't advertise `AXPress`** → a default click is an honest- no-op; the driver surfaces `AXOpen`/`AXShowMenu`/`AXConfirm`, so an agent must pick- `action:"open"`/`"pick"`.-4. **The Calculator result is AX-invisible** (no AX node for the display) → an AX-only- agent can't read the answer; the keypad itself is labelled. A `vision` readout is- required.-5. **AppKit AX-tree duplication** (System Settings ~half duplicated, Safari a duplicated- toolbar) plus whole-menu-bar walking inflates real-app context versus synthetic.--### Windows (Calculator/Notepad UWP, Edge, Explorer)--A. **UWP window-identity split.** `launch_app("Calculator")` returns the package backing- PID and a `window_id` that's stale by the next call; the real top-level HWND belongs to- `ApplicationFrameHost.exe`. Driving by the returned pid+window_id errors `No window with- window_id … exists`. The driver should resolve UWP windows to their `ApplicationFrameHost`- host / relink the churned HWND.-B. **Real UWP is drivable via the element path without the uiAccess worker.** Calculator's- `num5Button` drove the display 0→5 via UIA Invoke from the Medium-IL daemon with no- `cua-driver-uia.exe` running. This refines the `#1602`/`serve.rs` assumption: only the- **pixel/SendInput** path needs the uiAccess worker for AppContainer apps — the- **`element_index` UIA Invoke/ValuePattern path works on real UWP as-is.**-C. **Real Edge (Chromium) holds the foreground contract.** With Notepad pinned top, 3×- click + double-click + right-click on background Edge left the z-order unchanged- (`(background, no foreground swap)`). The shield validated on synthetic Electron (0/8)- **generalises to a real closed-source Chromium** — and the double/right-click that needed- the WinUI3 composition fix synthetically just work on real Chromium in the background.-D. **`element_index` requires `pid` (+`window_id`).** Element actions with `element_index`- alone fail-fast with `Missing required integer field: pid`. Correct, but the MCP tool- descriptions under-emphasise it — an agent that omits `pid` and filters stderr perceives a- silent no-op. The `element_index` tool schemas should state `pid`'s necessity explicitly.-E. **`get_screen_size` under-reports desktop width** (1024 reported vs 1824 actual span,- likely an RDP dynamic-resolution artifact). The element path is unaffected (window-local- frames stay consistent); a pixel/`vision` agent computing against 1024 width misplaces- clicks in the right ~800 px band.--F. **⚠ Unhandled-protocol `launch_app` deadlocks the whole daemon (DoS-class).** Launching an- app whose protocol has no registered handler (e.g. `bingmaps:` with Maps uninstalled) spawns- Windows' "you'll need a new app" modal **on the daemon's session desktop** and blocks the- worker thread inside the shell-launch call. The client wedges indefinitely and a subsequent- `stop` reports "daemon is not running" because the wedged daemon can't service the pipe;- recovery needs the modal dismissed from inside that session (Session-0 SSH can't see it —- per-session window stations). **Any bad app name/protocol is a full-daemon DoS.** The driver- should launch via a non-blocking path with a timeout and/or validate handler registration- before `ShellExecute`. (Needs a fix + a tracking issue.)-G. **One `ApplicationFrameHost` pid multiplexes N unrelated UWP apps** (Settings, Calculator,- and Store were all hosted under the same pid). Anything keying state/cursor/element-cache by- `pid` alone is ambiguous across apps — only `window_id` is a real identity. (Generalises A.)-H. **`launch_app`'s return contract is effectively per-app-architecture** — three topologies- seen across five apps: brokered/`pid:0` (Settings, Store), real-pid-but-empty-`windows` race- (Photos exposes its pid a beat before its HWND), and clean pid+window (Snipping Tool, the- well-behaved baseline). Store is a three-way split: launch-pid `0` ≠ window-pid (AFH) ≠- content-pid (`WinStore.App`), yet UIA still walks fully into the content provider.-I. **The AFH UIA root is a caption-only ~188×32 strip** for Settings and Store while the content- frames are full and correct — the AppFrame chrome and the XAML content are different UIA- providers stitched at the window. A single-provider synthetic WinUI harness won't reproduce- this dual-provider root.--> Methodology note: on a disconnected-RDP console session, `GetForegroundWindow` returns `0`-> (no foreground window), so focus-steal can't be measured by that probe there — verify "landed"-> by content-state change instead. A single-process harness never hits this.-------## Linux dispatch ladder — container and VM lanes--Linux (X11/XFCE and real-desktop VMs) now validates the same `delivery_mode` background/foreground dispatch ladder that macOS and Windows exercise, closing the parity gap. The matrix below shows each modality's path and whether the driver can verify the action landed without a screenshot.--### Validated modality matrix--| Modality | `delivery_mode` | Path reported | Driver-verifiable? |-|----------|----------------|---------------|-------------------|-| Element click (`element_index`) | background | `x11_atspi` (AT-SPI `doAction`) | yes — a11y action confirmed |-| Element `px` action (pixel) | background | `x11_atspi` (AT-SPI `doAction`-at-point) for AX apps; else MPX `x11_pixel` | yes when AT-SPI-at-point lands; best-effort otherwise |-| Element `px` action (escalated) | foreground | `x11_pixel_fg` (EWMH activate → inject → restore) | no — confirm via screenshot |-| `type_text` into editable | background | `ax` (AT-SPI `insertText`) | yes (`verified: true`) |-| `type_text`, non-editable focus | background / foreground | `key_events` / `key_events_fg` | no — confirm via screenshot |-| `press_key` / `hotkey` | foreground | `x11_xtest_fg` (XSetInputFocus → XTEST key) | no — confirm via screenshot / effect (`keytest`) |--Background element `px` actions (pixel clicks) do land on X11: apps that expose AT-SPI take the focus-free `doAction`-at-point path (`x11_atspi`), matching the macOS/Windows background-click behavior. The fallback to MPX `x11_pixel` (which requires a real Xorg server with `/dev/uinput` — unavailable under Xvnc or minimal containers) fires only for non-AX surfaces; escalate to `delivery_mode: foreground` there.--**Foreground keyboard (`press_key` / `hotkey`, and `type_text` foreground) lands under Xvnc.** The XTEST keyboard path injects on a short-lived X connection; it now **round-trips after injection** so the server delivers the key before the connection closes — under Xvnc a connection that closed right after `flush()` had its queued KeyPress/KeyRelease dropped (pointer events survived the same close, which masked the bug as "foreground clicks work but keys don't"). Two companion fixes: shifted-level symbols (`*`, `+`, `(`) **auto-hold Shift** via slot-0/slot-1 keysym resolution (a bare keycode press emitted the unshifted glyph, e.g. `*`→`8`); and the foreground rung now **`XSetInputFocus`es the target** (KWin honours EWMH `_NET_ACTIVE_WINDOW` as raise-only — fine for clicks that land by stacking, but keys route to the X input focus). `derec.sh keytest` effect-confirms the whole path across toolkits (AC, `3*4=` → display reads `12`).--### The XFCE container lane--The `trycua/cua-xfce` Docker image is a reproducible, WM-equipped X11 target — xfwm4 + EWMH + full AT-SPI stack — that exercises both the foreground/EWMH and background AT-SPI modalities. It is richer than the existing Xvfb harness (no window manager, no EWMH) for those paths.--The harness script lives at `libs/cua-driver/test-harness/linux-container/calc.sh`. It drives galculator through four modalities: background element `ax` click, background element `px` (pixel) click, foreground EWMH type, and background type with focus fallback.--**Gotchas:**--- **AT-SPI session-bus auto-discovery.** AT-SPI lives on the desktop session's D-Bus (`DBUS_SESSION_BUS_ADDRESS`). When the daemon starts outside the session (container entrypoint, headless, `runuser`/`su`, systemd system unit, VNC ad-hoc bus), that variable is unset, yielding an empty AT-SPI tree. The driver now auto-discovers the bus at startup — preferring `/run/user/<uid>/bus`, else reading it from a running desktop-session process's `/proc/<pid>/environ` (`xfce4-session`, `gnome-session`, etc.). An a11y bus must be running with toolkit-accessibility enabled, and the daemon must run as the desktop user — running the daemon as root against a user session is the Linux analogue of the Windows Session 0 isolation problem.-- **`install-local` refuses root.** Use `runuser -u <desktop-user> -- cua-driver serve` to start the daemon as the session owner.-- **Single-instance apps + zombie children.** Unreaped zombie children can pollute `pgrep`; a full daemon restart reaps them.-- **Stale daemon, stale tool schema.** A daemon that was not restarted after an update can serve an outdated tool schema. Verify with `cua-driver describe <tool>` from a fresh shell.--### Toolkit coverage--| Toolkit | App under test | Lane | AT-SPI coord path |-|---------|---------------|------|-------------------|-| GTK3 | galculator | Container (XFCE / Kasm, `az exec`) | `SCREEN` — AT-SPI returns correct screen coords |-| GTK4 | gnome-calculator | VM (GNOME Shell, SSH) | `WINDOW` + `_GTK_FRAME_EXTENTS` reconstruction (see below) |-| Qt | kcalc | VM (KDE Plasma, SSH) | `SCREEN` — Qt AT-SPI reports correct screen coords |-| Electron / Chromium | shared harness | Container (XFCE, `az exec`) | web-AX + pixel (`--force-renderer-accessibility`) |--### Lane matrix--| Lane | Desktop | Access method | Test app | Harness driver |-|------|---------|---------------|----------|----------------|-| XFCE container | `trycua/cua-xfce` (xfwm4) | `az exec` | galculator (GTK3), Electron | `libs/cua-driver/test-harness/linux-container/calc.sh` |-| Kasm container | `trycua/cua-ubuntu` (Kasm/XFCE) | `az exec` | galculator (GTK3) | `libs/cua-driver/test-harness/linux-container/calc.sh` |-| KDE Plasma VM | Azure VM, KDE Plasma X11 | SSH → `derec.sh` | kcalc (Qt) | `libs/cua-driver/test-harness/linux-container/derec.sh` |-| GNOME Shell VM | Azure VM, GNOME Shell X11 (console Xorg) | SSH → `derec.sh` | gnome-calculator (GTK4) | `libs/cua-driver/test-harness/linux-container/derec.sh` |-| GNOME Wayland VM | Azure VM, GNOME Mutter **Wayland** | SSH → `derec.sh` | gnome-calculator (GTK4) | `libs/cua-driver/test-harness/linux-container/derec.sh` |--The VM lanes are driven over SSH by `derec.sh`, in contrast to the container lanes which use `az exec`. GNOME Shell requires the real console Xorg session (software GLX) and is not drivable over VNC (no GLX). KDE Plasma works over both console Xorg and VNC.--### GTK4 coordinate reconstruction — invariant--GTK4's AT-SPI bridge returns `Component.GetExtents(SCREEN) = (0, 0)` for every widget (GNOME/gtk #1564/#1739, a11y rework). The driver fix:--1. Detects `_GTK_FRAME_EXTENTS` on the X11 window (present for GTK4; absent for Qt and GTK3).-2. Queries `CoordType::Window` (which GTK4 reports correctly, per-widget).-3. Reconstructs: `screen_xy = x11_window_origin + _GTK_FRAME_EXTENTS.(left, top) + WINDOW_xy`.--Verified live: gnome-calculator button "7" = window-origin (55, 27) + inset (61, 55) + WINDOW (16, 293) = screen (132, 375).--Non-GTK toolkits (Qt) keep the existing SCREEN path — the gate is the presence of `_GTK_FRAME_EXTENTS`.--**Invariant worth guarding:** `frame_center ≈ x11_window_origin + _GTK_FRAME_EXTENTS + atspi_WINDOW_xy`. Any divergence means the reconstruction is broken. Before the fix, pixel clicks silently collapsed to the window corner; `doAction` element clicks landed regardless (coordinate-free).--### The Wayland lane (GNOME Mutter / KDE Plasma)--On a native Wayland session the **background AX rung is the same coordinate-free AT-SPI `doAction`** as on X11, and it is the one rung that works unchanged: `Action.DoAction` is dispatched toolkit-side (GTK/Qt invoke the widget's own action), so it needs no coordinates, no portal, no cursor move, and no focus steal. Element-index clicks therefore land on Mutter/KWin exactly as on X11.--**Element `px` (pixel) coordinate clicks now land too — without pointer injection.** There is no global coordinate space on Wayland and Mutter drops synthetic virtual-pointer events, so a "click at (x,y)" can't be delivered as input. Instead the driver resolves the screen pixel back to the accessible element under it and fires *that* element's action — the proven `element_index` rung, reached by coordinates. The screen frame of every element is reconstructed the same way `get_window_state` exposes it (the `org.cua.WinRects` GNOME Shell helper supplies the window origin; GTK4's correct `CoordType::Window` extents supply the per-widget offset). The covering element is chosen **role-aware**: GTK4 nests a `label` inside every `button` with a near-identical, slightly *smaller* frame, so an area-only hit-test lands on the inert label and `do_action` silently no-ops (a "false success"); the selector prefers the smallest covering *real actuator* and only falls back to a passive label if nothing else covers the point. Validated live on Mutter Wayland: an element `px` action targeting buttons by `{x,y}` (no `element_index`) types `789` into gnome-calculator (rung `wayland_atspi`). The same role-aware fix hardened the X11 `perform_action_at_point` rung, which had the identical GTK4 false-success.--Two Wayland-specific facts the lane exercises:--- **Window enumeration via AT-SPI.** Native Wayland apps have no X11 XID, and GNOME Mutter / KDE KWin do **not** implement `zwlr_foreign_toplevel_management` (that is wlroots-only), so `wayland::list_windows` is empty there. `list_windows_dispatch` now falls back to **`atspi::list_windows`**, which enumerates top-level frames from the AT-SPI registry and hands back a synthetic, stable `window_id`. Downstream `get_window_state` / `click` walk the AT-SPI tree by **pid** (the xid is unused for the walk), so the full `list_windows → get_window_state → click` flow works. Validated live on Mutter Wayland: `7+8=` via `element_index` background clicks computes `15`.-- **Engaging the Wayland backend.** `is_wayland()` requires the opt-in `CUA_DRIVER_RS_ENABLE_WAYLAND=1`, `WAYLAND_DISPLAY` set, **and** `DISPLAY` unset — otherwise the daemon stays on the X11 path. Start the daemon with all three for a pure-Wayland session.--Bringing up a headless GNOME **Wayland** VM (the lane's host) has two gotchas worth recording, neither cua-driver's concern but both block the lane:--- **GDM disables Wayland for the Azure virtual GPU.** The `hyperv_drm` DRM driver exposes no `parameters/modeset`, so GDM's `61-gdm.rules` udev gate (`ATTR{parameters/modeset}!="Y"`) force-selects Xorg. Override by shadowing the rule in `/etc/udev/rules.d/61-gdm.rules` (drop the modeset + bare `GOTO="gdm_disable_wayland"` lines) and set `Session=ubuntu-wayland` in `/var/lib/AccountsService/users/<user>`, then restart gdm.-- **GTK4 apps need software GL on a GPU-less box.** Launch with `LIBGL_ALWAYS_SOFTWARE=1 GSK_RENDERER=cairo`, else the GL renderer fails (EGL/ZINK) and the app exits.-------## Appendix — file map--| Path | Role |-|------|------|-| `shared/scenarios.json` | single source of truth: control ids, window titles, scenarios |-| `shared/web/index.html` | shared DOM for WebView2 / Electron / WKWebView harnesses |-| `apps/` | per-OS/toolkit harness sources (see §3 table) |-| `build/{macos.sh,windows.ps1,linux.sh}` | stage harnesses into `rust/test-apps/harness-*` |-| `modality-recordings/FINDINGS.md` | authoritative per-action findings |-| `modality-recordings/windows/wpf-recorder.ps1` | Windows recorder (`-Mode`, `-Toolkit`) |-| `modality-recordings/windows/run-one.ps1` | single-run launcher |-| `modality-recordings/linux/lin-rec.py`, `lin-rec-electron.py` | Linux recorders (+ `lin-run*.sh`, `lin-dash*`, `lin-harness.py`) |-| `modality-recordings/macos/mac-rec.py` | macOS recorder (`MODE SURFACE`) |-| `vision-agent-test/vision_agent_test.py` + `README.md` | coordinate-invariant regression test |-| `smoke/macos.sh` | macOS smoke check |-| `rust/crates/cua-driver/tests/harness_*.rs` | Rust integration tests consuming staged harnesses |-| `TEST_SUITE.md` (this dir) | sibling doc: the Rust integration-test taxonomy / `cua-driver-testkit` |-| `CUA_DRIVER_LIMITATIONS_AND_TEST_MATRIX.md` (repo root) | limitations + mode/scope mapping |diff --git a/libs/cua-driver/test-harness/README.md b/libs/cua-driver/test-harness/README.mdindex 340b275263..69dbca7d8e 100644--- a/libs/cua-driver/test-harness/README.md+++ b/libs/cua-driver/test-harness/README.md@@ -1,184 +1,103 @@# cua-driver test harness-Deterministic host apps that present a fixed scenario set for cua-driver-to drive. One layout, three buckets: cross-OS reusable assets in-`shared/`, OS-specific (or cross-platform) target apps in `apps/`,-host-OS build scripts in `build/`, host-OS CLI smoke runners in-`smoke/`.+Deterministic host apps and shared fixtures used to verify cua-driver across+Windows, macOS, and Linux. The harness is source-first: build scripts stage+local outputs into `../rust/test-apps/harness-<name>/`; binaries are not+committed.## Layout```texttest-harness/-├── shared/ # cross-OS reusable-│ ├── scenarios.json # single SoT for AutomationIds / window titles-│ └── web/index.html # WebView2 + Electron load this same file-├── apps/ # the test-target host apps+├── shared/+│ ├── scenarios.json # scenario ids, titles, and expected controls+│ └── web/index.html # shared DOM for webview-style harnesses+├── apps/│ ├── cross-platform/-│ │ └── electron/ # CuaTestHarness.Electron (CDP host, runs on any OS)+│ │ ├── electron/ # Chromium/Electron host, CDP port 9223+│ │ └── tauri/ # native webview/Tauri host+│ ├── linux/+│ │ └── gtk3/ # PyGObject GTK3 app│ ├── macos/-│ │ ├── appkit/ # CuaTestHarness.AppKit (single-file Swift)-│ │ └── swiftui/ # CuaTestHarness.SwiftUI (single-file Swift)+│ │ ├── appkit/ # single-file Swift AppKit app+│ │ ├── swiftui/ # single-file SwiftUI app+│ │ └── wkwebview/ # native WKWebView host for shared DOM│ └── windows/-│ ├── wpf/ # CuaTestHarness.Wpf (.NET 8 WPF)-│ ├── winui3/ # CuaTestHarness.WinUI3 (.NET 8 WinUI3 unpackaged)-│ └── webview2/ # CuaTestHarness.WebView (WPF + Microsoft.Edge.WebView2)-├── build/ # host-OS build scripts (publish into ../rust/test-apps/)-│ ├── macos.sh # builds appkit + swiftui + wkwebview + electron-│ └── windows.ps1 # builds wpf + winui3 + webview2 + electron-└── smoke/ # host-OS per-tool CLI smoke runners- ├── macos.sh # spawns appkit harness, iterates every tool- └── results/macos.txt # checked-in baseline for diff-based regression detection+│ ├── wpf/ # .NET WPF app+│ ├── winui3/ # unpackaged WinUI3 app+│ └── webview2/ # WPF + WebView2 host for shared DOM+├── build/ # host build scripts+├── smoke/ # lightweight local smoke runners+├── linux-container/ # Azure/XFCE Linux verification helpers+└── modality-recordings/ # recorder scripts and dashboards```-`shared/scenarios.json` is the single source of truth — both the host-apps and the Rust integration tests read it so AutomationIds, AX-identifiers, and window titles never drift between the two halves.--Published build outputs land in `../rust/test-apps/harness-{wpf,winui3,webview,electron,appkit,swiftui}/`-so the existing sandbox runner and Rust integration tests pick them up-without further wiring.--## What's exercised--WPF (`apps/windows/wpf`):-- Plain XAML controls + AutomationIds (`counter`, `text_body`)-- TextBox + mirrored label (`text_input`)-- Right/double-click target (`click_target`)-- Scroll body with VerticalOffset label + WM_VSCROLL hook (`scroll_target`)-- Modal `MessageBox` (`message_box`)-- Save/Cancel bottom strip — regression guard for PR #1696 (`bottom_strip`)-- Native Win32 child HWNDs via `HwndHost` (`child_hwnd`)-- Owned secondary `Window` (`owned_popup`)-- Layered transparent window (`layered_popup`)-- Keyboard accelerators — F5 + Ctrl+Shift+H (`accelerator`)--WinUI3 (`apps/windows/winui3`):-- `counter` / `text_body` / `exit` (parity with WPF)-- TextBox + mirror (`text_input` — UIA `ValuePattern`)-- `CommandBarFlyout` — popup in same HWND, DComp-rendered-- XAML `Popup` — primitive, not a separate HWND--WebView2 (`apps/windows/webview2`):-- Loads `shared/web/index.html` — same page as the Electron harness.-- Exposes CDP on `default_cdp_port=9222` so cua-driver's `page` tool can drive it.--Electron (`apps/cross-platform/electron`):-- Loads the same `shared/web/index.html`. CDP on `default_cdp_port=9223`.--AppKit (`apps/macos/appkit`):-- Plain NSButton + NSTextField counter (`counter`)-- NSTextField label with shared marker (`text_body`)-- Editable NSTextField + mirror label (`text_input`)-- Custom NSView click-target (`click_target`)-- NSScrollView with tall NSTextView body + offset label (`scroll_target`)-- Top NSMenu item with known title (`ns_menubar`)-- Exit button--SwiftUI (`apps/macos/swiftui`):-- Counter / text_body / text_input parity with AppKit-- SwiftUI `.popover()` — Mac analogue of WinUI3 CommandBarFlyout / XAML Popup-- Exit button--## Coverage matrices (Rust integration tests)--Windows:--| Capability | WPF | WinUI3 |-|-------------------------------------|-----------------|--------------|-| Smoke (UIA tree + AutomationIds) | yes | yes |-| UIA Invoke click | yes | (implicit) |-| `right_click` | yes | — |-| `double_click` | yes | — |-| `type_text` (PostMessage / UIA) | yes | yes |-| `set_value` (UIA ValuePattern) | yes | (covered) |-| `scroll` (WM_VSCROLL via hook) | yes | — |-| `press_key` accelerator (F5) | yes | — |-| Modal `MessageBox` open + dismiss | yes | — |-| Owned popup open + parse | yes | — |-| Layered popup capture | yes | — |-| XAML Popup open + parse | — | yes |--macOS:--| Capability | AppKit | SwiftUI |-|-------------------------------------|-----------------|--------------|-| Smoke (AX tree + identifiers) | yes | yes |-| Counter click via element_index | yes | (implicit) |-| `set_value` (AXSetAttribute) | yes | (covered) |-| Popover open + body assert | — | yes |-| Per-tool CLI smoke | `smoke/macos.sh`| — |+`shared/scenarios.json` is the source of truth for AutomationIds, AX+identifiers, expected window titles, and fixture names. Webview-style harnesses+load `shared/web/index.html`, either directly or through a build-staged copy.## Build-macOS (requires Xcode CLT for `xcrun swiftc`; Node.js + npm for Electron):+Build scripts stage outputs into `../rust/test-apps/`.```bash-libs/cua-driver/test-harness/build/macos.sh # all macOS-runnable harnesses-libs/cua-driver/test-harness/build/macos.sh --skip swiftui # skip one target-libs/cua-driver/test-harness/build/macos.sh --skip electron # skip Electron / npm-libs/cua-driver/test-harness/build/macos.sh --clean # nuke stage dirs first-```+# macOS: AppKit, SwiftUI, WKWebView, Electron, Tauri+libs/cua-driver/test-harness/build/macos.sh+libs/cua-driver/test-harness/build/macos.sh --skip electron-Windows (requires .NET 8 SDK on PATH; Node.js + npm for the Electron app):+# Linux: GTK3, Electron, Tauri+libs/cua-driver/test-harness/build/linux.sh+libs/cua-driver/test-harness/build/linux.sh --skip tauri+``````powershell+# Windows: WPF, WinUI3, WebView2, Electron, Tauricd libs\cua-driver\test-harness-.\build\windows.ps1 # all four apps-.\build\windows.ps1 -Skip winui3 # skip a specific one-```--Linux (requires Node.js + npm for the Electron app):--```bash-cd libs/cua-driver/test-harness/apps/cross-platform/electron-./build.sh-```--## Run the integration tests--```bash-# macOS-cd libs/cua-driver/rust-cargo test --release --test harness_appkit_test -- --ignored --nocapture-cargo test --release --test harness_swiftui_test -- --ignored --nocapture--# Windows (locally)-cd libs\cua-driver\rust-cargo test --test harness_wpf_test -- --ignored --nocapture-cargo test --test harness_winui3_test -- --ignored --nocapture--# Windows Sandbox (matches CI)-.\rust\sandbox\run-tests-in-sandbox.ps1 harness_wpf-.\rust\sandbox\run-tests-in-sandbox.ps1 harness_winui3-```--Tests are `#[ignore]` so they don't run in plain `cargo test` (they-require the harness apps to be built and, on macOS, TCC Accessibility-granted to the test runner).--## Per-tool CLI smoke--Companion to the MCP integration tests. Spawns the AppKit harness as a-victim, then iterates every cua-driver tool with sensible JSON args and-reports PASS / FAIL / SKIP in a sorted table. Runs in ~25 seconds.--```bash-libs/cua-driver/test-harness/smoke/macos.sh+.\build\windows.ps1+.\build\windows.ps1 -Skip winui3```-A baseline result file (`smoke/results/macos.txt`) is checked in for the-current driver version — diff against it to catch regressions in tool-coverage.--## AppKit / SwiftUI AX quirks captured in the tests--- `accessibilityIdentifier` on AXStaticText doesn't propagate.- NSTextField label-mode + SwiftUI `Text` views render as AXStaticText- leaves and the OS drops the identifier slot. Tests assert AX-id only- on actionable controls and assert on text-content for labels.-- SwiftUI popovers may live in a separate AXWindow. The popover test- walks `list_windows` for any new window owned by the harness pid if- the trigger window doesn't contain the marker.-- AX element budget caps tree depth + width. The scroll-body is capped- at 30 lines so later scenarios don't get truncated.+Host requirements:++- macOS: Xcode command line tools; Node.js/npm for Electron; Rust for Tauri.+- Linux: GTK3/PyGObject/AT-SPI; Node.js/npm for Electron; Rust plus WebKitGTK+ build/runtime deps for Tauri.+- Windows: .NET 8 SDK; Node.js/npm for Electron; Rust for Tauri.++## Test Mapping++Rust tests under `../rust/crates/cua-driver/tests/` consume the staged+`rust/test-apps/harness-<name>/` outputs:++- `harness_<toolkit>_test.rs`: toolkit-specific app coverage, usually ignored.+- `modality_<area>[_<os>]_test.rs`: background input, capture, and desktop+ modality coverage, ignored/manual or VM-backed.+- `protocol_*_test.rs` and schema tests: headless protocol coverage, default.++The Python e2e suite under `../tests/e2e/` drives real apps and the shared+Electron/Tauri harnesses through the public MCP interface.++## App Coverage++| Harness | Source | Staged output | Primary surface |+| --- | --- | --- | --- |+| AppKit | `apps/macos/appkit` | `harness-appkit` | AX/AppKit controls |+| SwiftUI | `apps/macos/swiftui` | `harness-swiftui` | AX/SwiftUI controls |+| WKWebView | `apps/macos/wkwebview` | `harness-wkwebview` | WebKit AX tree |+| WPF | `apps/windows/wpf` | `harness-wpf` | UIA/WPF controls |+| WinUI3 | `apps/windows/winui3` | `harness-winui3` | UIA/XAML controls |+| WebView2 | `apps/windows/webview2` | `harness-webview` | Chromium web UIA |+| GTK3 | `apps/linux/gtk3` | `harness-gtk3` | AT-SPI/GTK controls |+| Electron | `apps/cross-platform/electron` | `harness-electron` | Chromium web AX/UIA/AT-SPI |+| Tauri | `apps/cross-platform/tauri` | `harness-tauri` | native webview AX/UIA/AT-SPI |++## Maintenance Rules++- Do not commit staged binaries, `node_modules`, copied `web/` folders, or Tauri+ `target/` output.+- Add new shared ids to `shared/scenarios.json` before asserting on them in+ tests.+- Keep platform-specific quirks in the relevant Rust test or app source, not in+ the shared fixture.+- If a recorder finding graduates into an invariant, move it into a Rust or+ Python test and summarize it here.diff --git a/libs/cua-driver/test-harness/TEST_HARNESS_STRUCTURE.md b/libs/cua-driver/test-harness/TEST_HARNESS_STRUCTURE.mddeleted file mode 100644index 5ef85ef18c..0000000000--- a/libs/cua-driver/test-harness/TEST_HARNESS_STRUCTURE.md+++ /dev/null@@ -1,117 +0,0 @@-# cua-driver test harnesses — how they're structured--The harness is built around **one principle**: a single source of truth (`shared/scenarios.json`) that both the **test apps** and the **Rust integration tests** read, so they can never drift. The directory tree itself encodes *which app runs on which OS* and *what's shared* (this layout landed in #1727).--## Directory layout--```-libs/cua-driver/test-harness/-│-├── shared/ ◀── cross-OS single source of truth-│ ├── scenarios.json # defines every app + scenario (titles, AutomationIds,-│ │ # expected tree shape). Read by BOTH the apps and the tests.-│ └── web/index.html # the page WebView2 AND Electron both load-│-├── apps/ ◀── the test-target GUI apps, grouped by where they run-│ ├── cross-platform/-│ │ └── electron/ # main.js · package.json (Win / macOS / Linux)-│ ├── macos/-│ │ ├── appkit/ # main.swift (macOS)-│ │ └── swiftui/ # main.swift (macOS)-│ └── windows/-│ ├── wpf/ # WPF: MainWindow, Owned/Layered popups,-│ │ # NativeButtonHost, Scenarios.cs (Windows)-│ ├── winui3/ # WinUI3: App, MainWindow, Program.cs (Windows)-│ └── webview2/ # WebView2 host (WPF) → loads shared/web (Windows)-│-├── build/ ◀── host-OS build scripts-│ ├── macos.sh # swiftc the macOS apps + electron-│ └── windows.ps1 # dotnet publish the .NET apps + electron-│-├── smoke/ ◀── per-tool CLI smoke runner + baseline-│ ├── macos.sh-│ └── results/macos.txt-│-├── CuaTestHarness.sln # .NET solution tying the Windows apps together-└── README.md--libs/cua-driver/rust/crates/cua-driver/tests/ ◀── the CONSUMERS (Rust integration tests)-│ (all build on the cua-driver-testkit crate; named by 4-family taxonomy —-│ harness_ / modality_ / protocol_ / guard_. See TEST_SUITE.md.)-├── harness_wpf_test.rs (Windows)-├── harness_winui3_test.rs (Windows)-├── harness_web_test.rs (Windows — WebView2 + Electron via CDP)-├── harness_libreoffice_test.rs (Windows — LibreOffice VCL/SAL via MSAA)-├── harness_appkit_test.rs (macOS)-└── harness_swiftui_test.rs (macOS)- (the focus-steal sentinel + deprecated capture_mode compatibility tests now live under- modality_background_test.rs — see TEST_SUITE.md for the full inventory)-```--## How the pieces connect--```mermaid-flowchart TD- SoT["shared/scenarios.json<br/>(single source of truth:<br/>titles · AutomationIds · expected tree)"]- WEB["shared/web/index.html"]-- subgraph APPS["apps/ — test-target GUI apps"]- direction LR- WPF["windows/wpf"]- WINUI["windows/winui3"]- WV["windows/webview2"]- ELEC["cross-platform/electron"]- AK["macos/appkit"]- SW["macos/swiftui"]- end-- BUILD["build/ → test-apps/harness-*<br/>(macos.sh · windows.ps1)"]-- subgraph TESTS["Rust integration tests (crates/cua-driver/tests/)"]- direction LR- T1["harness_wpf · winui3 · web_windows<br/>bg_modality · lo_vcl (Windows)"]- T2["harness_appkit · swiftui (macOS)"]- end-- DRIVER(["cua-driver tools<br/>(click / type / get_window_state / page / …)"])-- SoT -->|apps read at startup:<br/>set window titles & AutomationIds| APPS- WEB --> WV- WEB --> ELEC- APPS --> BUILD- BUILD -->|each test launches its built app| TESTS- SoT -->|tests read: assert expected shape<br/>without hardcoding strings| TESTS- TESTS -->|drive| DRIVER- DRIVER -->|act on| APPS- TESTS -->|re-snapshot & assert| TESTS-- classDef sot fill:#1f883d,stroke:#116329,color:#fff;- class SoT,WEB sot;-```--The loop per test: **launch the built app → read `scenarios.json` for the expected shape → drive cua-driver against the app → re-snapshot and assert.** Because both sides read the same `scenarios.json`, a changed AutomationId or title updates the apps and the assertions together.--## Coverage matrix--| App (`apps/…`) | OS | `scenarios.json` key | What it exercises | Rust test(s) |-|---|---|---|---|---|-| `windows/wpf` | Windows | `wpf` | UIA Invoke, PostMessage type, right/double-click, scroll, modal MessageBox, owned + layered popups, native child HWNDs, accelerators | `harness_wpf_test`, `modality_background_test` |-| `windows/winui3` | Windows | `winui3` | UIA ValuePattern text, CommandBarFlyout, XAML Popup primitive | `harness_winui3_test` |-| `windows/webview2` | Windows | `webview` | Chromium DevTools Protocol (CDP) `page` tool over the shared web page | `harness_web_test` |-| `cross-platform/electron` | Win / macOS / Linux | `electron` | CDP `page` tool over the shared web page | `harness_web_test` (+ macOS) |-| `macos/appkit` | macOS | `appkit` | AX tree, NSButton AXPress, NSTextField, NSScrollView, NSMenu | `harness_appkit_test` |-| `macos/swiftui` | macOS | `swiftui` | AX tree, SwiftUI `.popover()` (separate AXWindow) | `harness_swiftui_test` |-| *(LibreOffice — not a harness app)* | Windows | — | VCL/SAL SplitButton, modal dialogs, color picker via MSAA fallback | `harness_libreoffice_test` |-| *(WPF + focus sentinel)* | Windows | `wpf` | background-modality / no-focus-steal sentinel + `capture_mode` (deprecated/ignored) | `modality_background_test` |--## Runtime requirements (why CI coverage is uneven)--- **Linux** (Electron via CDP, plus the Nix GUI scenarios) → runs **headless in CI** (Xvfb / NixOS VMs on `ubuntu-latest`).-- **Windows** (WPF / WinUI3 / WebView2 / bg-modality) and **macOS** (AppKit / SwiftUI) → need a **real interactive desktop** (UIA / AX / SendInput / z-order / TCC), so they need self-hosted runners; they're not in CI today.--## TL;DR--- **`shared/scenarios.json` is the contract.** Apps and tests both read it ⇒ no drift.-- **Tree = topology:** `apps/{cross-platform,macos,windows}/…` says exactly where each runs; `shared/` is reusable.-- **Tests live with the driver** (`crates/cua-driver/tests/harness_*`), launch the built app, and assert against the same scenarios.-- The layout itself is the standardization — landed in **#1727**; Linux parity is still being extended in **#1693**.diff --git a/libs/cua-driver/test-harness/TEST_SUITE.md b/libs/cua-driver/test-harness/TEST_SUITE.mddeleted file mode 100644index c128ceba28..0000000000--- a/libs/cua-driver/test-harness/TEST_SUITE.md+++ /dev/null@@ -1,128 +0,0 @@-# cua-driver test suite — how the tests look--The integration tests in `libs/cua-driver/rust/crates/cua-driver/tests/` all sit-on one shared foundation — the **`cua-driver-testkit`** crate — and are named by-a four-family taxonomy so `ls tests/` reads as a map. Companion:-`TEST_HARNESS_STRUCTURE.md` (the harness *apps* the `harness_*` tests drive).--## The shared foundation: `cua-driver-testkit`--A dev-only crate (`crates/cua-driver-testkit`) that every test builds on, so no-test re-implements process plumbing:--- **Two transports, one shape.** `McpDriver` (one long-lived `cua-driver` server- over stdio JSON-RPC, session-scoped state) and `CliDriver` (stateless- `cua-driver call` per action) both implement the `Driver` trait and normalize- their differing payloads — the CLI prints `structuredContent` directly, MCP- returns the JSON-RPC envelope — into one `ToolResponse` (`text()` /- `structured()` / `is_error()`). `RawDriver` adds raw send/recv with no- auto-initialize for the protocol tests that drive the handshake themselves.-- **`ChildReaper`** kills every spawned child on drop; on Windows it assigns them- to a kill-on-close Job Object (no orphaned windows / held ports).-- **`paths`** (`driver_binary()` resolves release-or-debug + `.exe`), **`ax`**- (snapshot-text parsers), `McpDriver::find_window`.--## The loop every behavior test runs--```mermaid-flowchart LR- T["test (tests/*.rs)"]- TK(["cua-driver-testkit<br/>McpDriver / CliDriver / RawDriver"])- D(["cua-driver"])- A["target app / harness app"]- T -->|"Driver::call(tool, args)"| TK- TK -->|"spawn + drive"| D- D -->|"acts on"| A- T -->|"re-snapshot & assert (ToolResponse)"| T-```--## Inventory by family (18 files)--| Family | File | Platform | What it covers |-|---|---|---|---|-| **protocol_** | `protocol_handshake_test` | mac+win | initialize handshake, tools/list registration, per-tool capability + version fields, unknown-method/tool errors |-| | `protocol_tools_call_test` | mac+win | per-tool `tools/call` shape: list_apps, get_config, permissions, AX tree, list_windows, screen size, scroll, type_text, hotkey, press_key, click, set_value |-| | `protocol_schema_test` | mac+win | inputSchema shape for the `type_text` chars variants |-| | `protocol_media_test` | mac+win | screenshot / zoom (jpeg) / recording / replay / screenshot-resize |-| | `protocol_session_test` | mac+win | concurrent clients, multi-driver isolation, cursor-overlay liveness |-| | `protocol_element_token_test` | mac/linux | Surface 6 opaque `element_token` in schema + capabilities |-| **transport_** | `transport_config_persistence_test` | any | `set_config` persists to disk across stateless **CLI** invocations (#2034) vs visible within an **MCP** session |-| **harness_** | `harness_wpf_test` | win | WPF: UIA Invoke, type, clicks, scroll, modal, owned/layered popups, native child HWNDs |-| | `harness_winui3_test` | win | WinUI3: ValuePattern, CommandBarFlyout, XAML Popup |-| | `harness_web_test` | win | WebView2 + Electron via CDP `page` |-| | `harness_libreoffice_test` | win | LibreOffice VCL/SAL via MSAA (some `dispatch:"foreground"`) |-| | `harness_appkit_test` | mac | AppKit: AX tree, AXPress, NSTextField, NSScrollView, NSMenu |-| | `harness_swiftui_test` | mac | SwiftUI: AX tree, `.popover()` |-| **modality_** | `modality_background_test` | win | background-modality / no-focus-steal sentinel + deprecated `capture_mode` values accepted but ignored |-| | `modality_capture_mode_test` | mac+win+linux | deprecated `capture_mode` parameter on each native harness: values are accepted but ignored |-| | `modality_input_e2e_test` | win | unified background input across Electron/Tauri/Win32, no z-raise |-| | `modality_desktop_scope_test` | win | desktop-scope (foreground): `capture_scope=desktop`, `get_desktop_state`, window-less screen-absolute actions |-| | `modality_focus_test` | mac | background automation does not steal focus |-| **guard_** | `guard_ux_test` | win | UX guards: background focus, click-opens-window, launch-visible, menu shortcut |--The `protocol_*` family is the renamed + split successor of the old-`mcp_protocol_test` (3,412 lines / 65 tests). The split deduped 24 macOS↔Windows-mirror pairs into single `cfg!`-branching tests, sharing one `RawDriver`.--## Coverage on three axes: transport × modality × platform--```- ┌────────────── MODALITY ──────────────┐- │ background foreground/desktop-scope│- ─────────────────┼───────────────────────────────────────- MCP transport │ ✅ thorough ✅ desktop_scope (win) │- │ (background, focus (mac) │- │ input_e2e, │- │ focus, guard) │- CLI transport │ ⬚ ✅ config-persistence │- │ (transport_ test) │- └───────────────────────────────────────┘- Per-app AX (MCP): harness_{wpf,winui3,web,libreoffice}=win · {appkit,swiftui}=mac- Protocol (RawDriver): mac+win, deduped; element_token = mac/linux-```--- **Background modality** is the most-covered: the sentinel oracle- (`modality_background`), unified input (`modality_input_e2e`), the macOS- no-focus-steal check (`modality_focus`), and the UX guards.-- **Foreground / desktop-scope** is covered by `modality_desktop_scope`.-- **Transport** is now a first-class axis: `transport_config_persistence`- exercises CLI (disk) vs MCP (session) directly; most other tests run over MCP.--### The modality matrix: action target × delivery mode × `capture_scope`--The user-facing matrix is documented in-`docs/content/docs/concepts/capture-and-delivery-modalities.mdx`. `get_window_state`-always returns both the accessibility tree and a screenshot; AX vs pixel modality is-chosen at action time by using `element_index` vs `x,y`. Coverage of its valid cells,-per platform:--| Cell (`scope`/action target/`delivery_mode`) | Windows | macOS | Linux |-|---|---|---|---|-| `window`/`element_index`/`background` (default) | `harness_*`, `modality_background` | `harness_{appkit,swiftui}`, `modality_focus`, `modality_capture_mode` | `harness_gtk3`, `modality_capture_mode` |-| `window`/`element_index`/`foreground` | `harness_wpf` (`delivery_mode:"foreground"`) | activation differs | activation differs |-| `window`/`x,y`/`background` | `modality_background`, `modality_capture_mode`, `modality_input_e2e` | `modality_capture_mode` | `modality_capture_mode` |-| `window`/`x,y`/`foreground` | gap | n/a (no `bring_to_front`) | n/a (stubbed) |-| `desktop`/`x,y`/`foreground` | `modality_desktop_scope` | rolling out | rolling out |-| **negative gate** (`desktop_scope_disabled`) | `window_scope_rejects_windowless_click` | — | — |--`capture_mode` is deprecated and ignored: `get_window_state` returns both the-accessibility tree and a screenshot regardless of the parameter. The-`modality_capture_mode_test` now exercises compatibility only: deprecated-`ax`/`vision`/`som` values are accepted but do not change the returned shape.--- **Remaining gaps** (honest): `window`/`x,y`/`foreground` has no explicit- test on any platform; the desktop-scope window-less actuator is Windows-only- (macOS click still requires `pid`), so the desktop cell and its negative gate- are Windows-only until that loop lands on macOS/Linux; CLI transport is- otherwise lightly exercised.--## Where they run--| Lane | Tests | Runner |-|---|---|---|-| **Runs without `#[ignore]`** | `protocol_*` (mac/win dev machines; linux excludes them via cfg) | local / VM |-| **Interactive Windows** (`#[ignore]`) | `harness_{wpf,winui3,web,libreoffice}`, `modality_*` (win), `guard_ux`, `transport_*` | real desktop / VM |-| **Interactive macOS** (`#[ignore]`) | `harness_{appkit,swiftui}`, `modality_focus` | logged-in session + TCC |--Run an `#[ignore]` suite explicitly:-`cargo test -p cua-driver --test <name> -- --ignored --nocapture --test-threads=1`diff --git a/libs/cua-driver/test-harness/apps/README.md b/libs/cua-driver/test-harness/apps/README.mdindex e0da6574b6..fabff8c5f9 100644--- a/libs/cua-driver/test-harness/apps/README.md+++ b/libs/cua-driver/test-harness/apps/README.md@@ -1,30 +1,29 @@# test-harness apps-Each subdirectory is one self-contained host app cua-driver drives. They-share fixtures via `../shared/scenarios.json` and (where relevant)-`../shared/web/index.html` so AutomationIds, AX identifiers, and DOM-ids never drift between the apps and the integration tests.+Each subdirectory is one deterministic host app cua-driver can drive. Apps+share fixtures through `../shared/scenarios.json` and, for webview hosts,+`../shared/web/index.html`.-| Path | Host OS | What it tests |-|-------------------------------|-----------------|-----------------------------------------------------------------|-| `cross-platform/electron` | any | Electron + CDP — exercises cua-driver's `page` tool |-| `macos/appkit` | macOS 13+ | AppKit / Cocoa hosting — NSButton, NSScrollView, NSMenu, etc. |-| `macos/swiftui` | macOS 13+ | SwiftUI hosting — `.popover()`, declarative state |-| `windows/wpf` | Windows | WPF / UIA — popups, layered windows, modal MessageBox, hwndhost |-| `windows/winui3` | Windows | WinUI3 unpackaged — DComp popups, XAML Popup primitive |-| `windows/webview2` | Windows | Microsoft.Edge.WebView2 hosting the same DOM as Electron |+| Path | Host OS | What it tests |+| --- | --- | --- |+| `cross-platform/electron` | any | Electron/Chromium accessibility and CDP |+| `cross-platform/tauri` | any | Tauri/native-webview accessibility |+| `linux/gtk3` | Linux | GTK3/AT-SPI widgets |+| `macos/appkit` | macOS | AppKit controls and menus |+| `macos/swiftui` | macOS | SwiftUI controls and popovers |+| `macos/wkwebview` | macOS | WKWebView loading the shared DOM |+| `windows/wpf` | Windows | WPF/UIA controls, popups, child HWNDs |+| `windows/winui3` | Windows | WinUI3/XAML controls and popups |+| `windows/webview2` | Windows | WebView2 loading the shared DOM |-Built outputs land in `../../rust/test-apps/harness-{appkit,swiftui,wpf,winui3,webview,electron}/`-via `../build/macos.sh` or `..\build\windows.ps1`. The Rust integration-tests under `../../rust/crates/cua-driver/tests/harness_*.rs` consume-those staged outputs.+Built outputs land in `../../rust/test-apps/harness-<name>/` via+`../build/<host>`.Adding a new app:-1. Pick the right OS bucket (or `cross-platform/` if Electron-equivalent).-2. Mirror an existing app's layout — Swift `main.swift` for macOS,- `.csproj` + XAML for Windows.-3. Add a section to `../shared/scenarios.json` with the AutomationIds /- AX identifiers / window title your tests will assert on.-4. Wire it into `../build/{macos.sh,windows.ps1}` so it stages into- `rust/test-apps/harness-<name>/`.+1. Put native source under `apps/<platform>/<toolkit>/`, or+ `apps/cross-platform/<toolkit>/` when the same source runs on multiple OSes.+2. Reuse `shared/scenarios.json` and `shared/web/index.html` instead of copying+ private ids.+3. Stage into `rust/test-apps/harness-<toolkit>/`.+4. Add or update the matching Rust/Python tests and this table.diff --git a/libs/cua-driver/test-harness/apps/cross-platform/tauri/.gitignore b/libs/cua-driver/test-harness/apps/cross-platform/tauri/.gitignorenew file mode 100644index 0000000000..3922fb1e49--- /dev/null+++ b/libs/cua-driver/test-harness/apps/cross-platform/tauri/.gitignore@@ -0,0 +1,4 @@+web/+src-tauri/gen/+src-tauri/target/+src-tauri/Cargo.lockdiff --git a/libs/cua-driver/test-harness/apps/cross-platform/tauri/build.ps1 b/libs/cua-driver/test-harness/apps/cross-platform/tauri/build.ps1new file mode 100644index 0000000000..d7d82a6b85--- /dev/null+++ b/libs/cua-driver/test-harness/apps/cross-platform/tauri/build.ps1@@ -0,0 +1,44 @@+# build.ps1 - stage the Tauri test harness for cua-driver tests on Windows.+#+# Output: ../../../../rust/test-apps/harness-tauri/CuaTestHarness.Tauri.exe++Set-StrictMode -Version Latest+$ErrorActionPreference = "Stop"++$tauriDir = Split-Path -Parent $MyInvocation.MyCommand.Definition+$crossDir = Split-Path -Parent $tauriDir+$appsDir = Split-Path -Parent $crossDir+$harnessDir = Split-Path -Parent $appsDir+$cuaDriverDir = Split-Path -Parent $harnessDir+$testAppsDir = Join-Path $cuaDriverDir "rust\test-apps"+$outDir = Join-Path $testAppsDir "harness-tauri"++if (-not (Get-Command cargo -ErrorAction SilentlyContinue)) {+ Write-Host "[ERROR] cargo not on PATH. Install Rust first." -ForegroundColor Red+ exit 1+}++$webDir = Join-Path $tauriDir "web"+if (-not (Test-Path $webDir)) { New-Item -ItemType Directory $webDir | Out-Null }+Copy-Item (Join-Path $harnessDir "shared\web\*") $webDir -Recurse -Force++$manifest = Join-Path $tauriDir "src-tauri\Cargo.toml"+Write-Host "[BUILD] cargo build --release --manifest-path $manifest" -ForegroundColor Cyan+$oldTargetDir = $env:CARGO_TARGET_DIR+$env:CARGO_TARGET_DIR = Join-Path $tauriDir "src-tauri\target"+try {+ & cargo build --release --manifest-path $manifest+} finally {+ $env:CARGO_TARGET_DIR = $oldTargetDir+}+if ($LASTEXITCODE -ne 0) { throw "cargo build failed for Tauri harness" }++$srcExe = Join-Path $tauriDir "src-tauri\target\release\cua-test-harness-tauri.exe"+if (-not (Test-Path $srcExe)) {+ throw "Tauri exe not found after build: $srcExe"+}++if (Test-Path $outDir) { Remove-Item $outDir -Recurse -Force }+New-Item -ItemType Directory -Force $outDir | Out-Null+Copy-Item $srcExe (Join-Path $outDir "CuaTestHarness.Tauri.exe") -Force+Write-Host "[OK] Staged: $outDir\CuaTestHarness.Tauri.exe" -ForegroundColor Greendiff --git a/libs/cua-driver/test-harness/apps/cross-platform/tauri/build.sh b/libs/cua-driver/test-harness/apps/cross-platform/tauri/build.shnew file mode 100755index 0000000000..4aa7f363ac--- /dev/null+++ b/libs/cua-driver/test-harness/apps/cross-platform/tauri/build.sh@@ -0,0 +1,69 @@+#!/usr/bin/env bash+# Stage the Tauri test harness for cua-driver tests on Linux and macOS.+#+# The app embeds the same shared/web/index.html DOM used by Electron,+# WebView2, and WKWebView. Build output is copied into+# rust/test-apps/harness-tauri/ with deterministic executable names.+set -euo pipefail++tauriDir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+crossDir="$(dirname "$tauriDir")"+appsDir="$(dirname "$crossDir")"+harnessDir="$(dirname "$appsDir")"+cuaDriverDir="$(dirname "$harnessDir")"+outDir="$cuaDriverDir/rust/test-apps/harness-tauri"+platform="$(uname -s)"++if ! command -v cargo >/dev/null 2>&1; then+ echo "[ERROR] cargo not on PATH. Install Rust first." >&2+ exit 1+fi++mkdir -p "$tauriDir/web"+cp -r "$harnessDir/shared/web/." "$tauriDir/web/"++echo "[BUILD] cargo build --release ..."+CARGO_TARGET_DIR="$tauriDir/src-tauri/target" \+ cargo build --release --manifest-path "$tauriDir/src-tauri/Cargo.toml"++rm -rf "$outDir"+mkdir -p "$outDir"++if [ "$platform" = "Darwin" ]; then+ srcBin="$tauriDir/src-tauri/target/release/cua-test-harness-tauri"+ [ -x "$srcBin" ] || { echo "[ERROR] Tauri binary missing after build" >&2; exit 1; }++ bundle="$outDir/CuaTestHarness.Tauri.app"+ appMacOS="$bundle/Contents/MacOS"+ mkdir -p "$appMacOS" "$bundle/Contents/Resources"+ cp "$srcBin" "$appMacOS/CuaTestHarness.Tauri"+ chmod +x "$appMacOS/CuaTestHarness.Tauri"++ cat > "$bundle/Contents/Info.plist" <<'PLIST'+<?xml version="1.0" encoding="UTF-8"?>+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"+ "http://www.apple.com/DTDs/PropertyList-1.0.dtd">+<plist version="1.0">+<dict>+ <key>CFBundleExecutable</key><string>CuaTestHarness.Tauri</string>+ <key>CFBundleIdentifier</key><string>com.trycua.harness.tauri</string>+ <key>CFBundleName</key><string>CuaTestHarness.Tauri</string>+ <key>CFBundleDisplayName</key><string>CuaTestHarness.Tauri</string>+ <key>CFBundlePackageType</key><string>APPL</string>+ <key>CFBundleShortVersionString</key><string>1.0.0</string>+ <key>CFBundleVersion</key><string>1</string>+ <key>LSMinimumSystemVersion</key><string>13.0</string>+ <key>NSHighResolutionCapable</key><true/>+</dict>+</plist>+PLIST+ xattr -cr "$bundle" 2>/dev/null || true+ echo "[OK] Staged: $bundle"+else+ srcBin="$tauriDir/src-tauri/target/release/cua-test-harness-tauri"+ [ -x "$srcBin" ] || { echo "[ERROR] Tauri binary missing after build" >&2; exit 1; }++ cp "$srcBin" "$outDir/CuaTestHarness.Tauri"+ chmod +x "$outDir/CuaTestHarness.Tauri"+ echo "[OK] Staged: $outDir/CuaTestHarness.Tauri"+fidiff --git a/libs/cua-driver/test-harness/apps/cross-platform/tauri/package.json b/libs/cua-driver/test-harness/apps/cross-platform/tauri/package.jsonnew file mode 100644index 0000000000..d487ea07e0--- /dev/null+++ b/libs/cua-driver/test-harness/apps/cross-platform/tauri/package.json@@ -0,0 +1,10 @@+{+ "name": "cua-test-harness-tauri",+ "version": "1.0.0",+ "private": true,+ "description": "cua-driver Tauri test harness loading the shared web DOM",+ "scripts": {+ "build": "cargo build --release --manifest-path src-tauri/Cargo.toml",+ "dev": "cargo run --manifest-path src-tauri/Cargo.toml"+ }+}diff --git a/libs/cua-driver/test-harness/apps/cross-platform/tauri/src-tauri/Cargo.toml b/libs/cua-driver/test-harness/apps/cross-platform/tauri/src-tauri/Cargo.tomlnew file mode 100644index 0000000000..2b6401a089--- /dev/null+++ b/libs/cua-driver/test-harness/apps/cross-platform/tauri/src-tauri/Cargo.toml@@ -0,0 +1,12 @@+[package]+name = "cua-test-harness-tauri"+version = "1.0.0"+description = "cua-driver Tauri test harness"+edition = "2021"+rust-version = "1.77"++[build-dependencies]+tauri-build = { version = "2" }++[dependencies]+tauri = { version = "2", features = [] }diff --git a/libs/cua-driver/test-harness/apps/cross-platform/tauri/src-tauri/build.rs b/libs/cua-driver/test-harness/apps/cross-platform/tauri/src-tauri/build.rsnew file mode 100644index 0000000000..d860e1e6a7--- /dev/null+++ b/libs/cua-driver/test-harness/apps/cross-platform/tauri/src-tauri/build.rs@@ -0,0 +1,3 @@+fn main() {+ tauri_build::build()+}diff --git a/libs/cua-driver/test-harness/apps/cross-platform/tauri/src-tauri/icons/icon.png b/libs/cua-driver/test-harness/apps/cross-platform/tauri/src-tauri/icons/icon.pngnew file mode 100644index 0000000000..4c53796631Binary files /dev/null and b/libs/cua-driver/test-harness/apps/cross-platform/tauri/src-tauri/icons/icon.png differdiff --git a/libs/cua-driver/test-harness/apps/cross-platform/tauri/src-tauri/src/main.rs b/libs/cua-driver/test-harness/apps/cross-platform/tauri/src-tauri/src/main.rsnew file mode 100644index 0000000000..2331526d10--- /dev/null+++ b/libs/cua-driver/test-harness/apps/cross-platform/tauri/src-tauri/src/main.rs@@ -0,0 +1,7 @@+#![cfg_attr(not(debug_assertions), windows_subsystem = "windows")]++fn main() {+ tauri::Builder::default()+ .run(tauri::generate_context!())+ .expect("error while running CuaTestHarness.Tauri");+}diff --git a/libs/cua-driver/test-harness/apps/cross-platform/tauri/src-tauri/tauri.conf.json b/libs/cua-driver/test-harness/apps/cross-platform/tauri/src-tauri/tauri.conf.jsonnew file mode 100644index 0000000000..c994485b39--- /dev/null+++ b/libs/cua-driver/test-harness/apps/cross-platform/tauri/src-tauri/tauri.conf.json@@ -0,0 +1,28 @@+{+ "$schema": "https://schema.tauri.app/config/2",+ "productName": "CuaTestHarness.Tauri",+ "version": "1.0.0",+ "identifier": "com.trycua.harness.tauri",+ "build": {+ "beforeBuildCommand": "",+ "beforeDevCommand": "",+ "frontendDist": "../web"+ },+ "app": {+ "windows": [+ {+ "label": "main",+ "title": "CuaTestHarness Tauri",+ "width": 900,+ "height": 760,+ "resizable": true+ }+ ],+ "security": {+ "csp": null+ }+ },+ "bundle": {+ "active": false+ }+}diff --git a/libs/cua-driver/test-harness/build/README.md b/libs/cua-driver/test-harness/build/README.mdindex d7aa5cd09a..156654ecc5 100644--- a/libs/cua-driver/test-harness/build/README.md+++ b/libs/cua-driver/test-harness/build/README.md@@ -1,22 +1,23 @@# test-harness build scripts-Run `macos.sh` or `windows.ps1` to build every harness app for that host-OS. Outputs are staged into `../../rust/test-apps/harness-<name>/` so-the Rust integration tests and the Windows Sandbox runner pick them up-unchanged.+These host-specific scripts stage source-built harness apps into+`../../rust/test-apps/harness-<name>/`.-```bash-# macOS — builds appkit + swiftui via xcrun swiftc into .app bundles-./macos.sh+| Script | Host | Builds |+| --- | --- | --- |+| `macos.sh` | macOS | AppKit, SwiftUI, WKWebView, Electron, Tauri |+| `linux.sh` | Linux | GTK3, Electron, Tauri |+| `windows.ps1` | Windows | WPF, WinUI3, WebView2, Electron, Tauri |++Each script supports skipping one target while iterating locally:-# Windows — dotnet publish for wpf/winui3/webview2 + electron-builder for electron-.\windows.ps1+```bash+./macos.sh --skip electron+./linux.sh --skip tauri```-Both scripts accept a `--skip` / `-Skip` argument to build one app at a-time during local iteration; see the file headers for the full surface.+```powershell+.\windows.ps1 -Skip webview+```-Cross-platform Electron is built by `windows.ps1` today (which runs-`apps/cross-platform/electron/build.ps1`). When a non-Windows test path-needs the Electron host, lift the staging step into `macos.sh` (or-`linux.sh`).+Build products are local artifacts. Rebuild them instead of committing them.diff --git a/libs/cua-driver/test-harness/build/linux.sh b/libs/cua-driver/test-harness/build/linux.shindex ac22a184e4..4f8fabf586 100755--- a/libs/cua-driver/test-harness/build/linux.sh+++ b/libs/cua-driver/test-harness/build/linux.sh@@ -1,27 +1,54 @@#!/usr/bin/env bash-# Stage the Linux GTK3 test-harness app into ../../rust/test-apps/harness-gtk3/ —-# the Linux peer of macos.sh (AppKit/SwiftUI) and windows.ps1 (WPF/WinUI3).+# Stage Linux-runnable test-harness apps into ../../rust/test-apps/harness-*/.#-# The app is PyGObject (GTK3): a real GTK3 widget tree (identical AT-SPI exposure-# to a C app), so there's no compile step — we stage main.py plus an executable-# launcher named CuaTestHarness.Gtk3 (matching scenarios.json's exe_relative_path).+# GTK3 is PyGObject: a real GTK3 widget tree (identical AT-SPI exposure to a C+# app), so there's no compile step. Electron and Tauri are shared cross-platform+# harnesses built from apps/cross-platform/.#-# Runtime deps (NOT installed here): python3-gi, gir1.2-gtk-3.0, at-spi2-core.-# apt-get install -y python3-gi gir1.2-gtk-3.0 at-spi2-core+# Runtime deps (NOT installed here): python3-gi, gir1.2-gtk-3.0, at-spi2-core,+# Node.js/npm for Electron, Rust + WebKitGTK build deps for Tauri.#-# Usage: ./linux.sh [--clean]+# Usage:+# ./linux.sh+# ./linux.sh --skip gtk3 # skip one target (gtk3|electron|tauri)+# ./linux.sh --cleanset -euo pipefailSCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"HARNESS_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"-STAGE="$(cd "$HARNESS_DIR/../rust/test-apps" && pwd)/harness-gtk3"+TEST_APPS_DIR="$(cd "$HARNESS_DIR/../rust/test-apps" && pwd)"+STAGE="$TEST_APPS_DIR/harness-gtk3"SRC="$HARNESS_DIR/apps/linux/gtk3"+SKIP="none"+CLEAN=0-rm -rf "$STAGE"-mkdir -p "$STAGE"-cp "$SRC/main.py" "$STAGE/main.py"+while [[ $# -gt 0 ]]; do+ case "$1" in+ --skip)+ SKIP="${2:-}"+ shift 2+ ;;+ --clean)+ CLEAN=1+ shift+ ;;+ *)+ echo "Usage: $0 [--skip gtk3|electron|tauri] [--clean]" >&2+ exit 2+ ;;+ esac+done-cat > "$STAGE/CuaTestHarness.Gtk3" <<'LAUNCHER'+if [[ "$CLEAN" == "1" ]]; then+ rm -rf "$TEST_APPS_DIR/harness-gtk3" "$TEST_APPS_DIR/harness-electron" "$TEST_APPS_DIR/harness-tauri"+fi++if [[ "$SKIP" != "gtk3" ]]; then+ rm -rf "$STAGE"+ mkdir -p "$STAGE"+ cp "$SRC/main.py" "$STAGE/main.py"++ cat > "$STAGE/CuaTestHarness.Gtk3" <<'LAUNCHER'#!/usr/bin/env bash# Force the X11 backend so the window is enumerable via cua-driver's X11# list_windows (_NET_CLIENT_LIST) — under a Wayland session this routes through@@ -29,11 +56,20 @@ cat > "$STAGE/CuaTestHarness.Gtk3" <<'LAUNCHER'export GDK_BACKEND=x11exec python3 "$(dirname "$(readlink -f "$0")")/main.py" "$@"LAUNCHER-chmod +x "$STAGE/CuaTestHarness.Gtk3"+ chmod +x "$STAGE/CuaTestHarness.Gtk3"++ echo "==> Staged GTK3 harness -> $STAGE/CuaTestHarness.Gtk3"+ if python3 -c "import gi; gi.require_version('Gtk','3.0'); from gi.repository import Gtk" 2>/dev/null; then+ echo "==> PyGObject/GTK3 present"+ else+ echo "WARNING: PyGObject/GTK3 not importable - install python3-gi gir1.2-gtk-3.0 at-spi2-core" >&2+ fi+fi++if [[ "$SKIP" != "electron" ]]; then+ "$HARNESS_DIR/apps/cross-platform/electron/build.sh"+fi-echo "==> Staged GTK3 harness → $STAGE/CuaTestHarness.Gtk3"-if python3 -c "import gi; gi.require_version('Gtk','3.0'); from gi.repository import Gtk" 2>/dev/null; then- echo "==> PyGObject/GTK3 present"-else- echo "WARNING: PyGObject/GTK3 not importable — install python3-gi gir1.2-gtk-3.0 at-spi2-core" >&2+if [[ "$SKIP" != "tauri" ]]; then+ "$HARNESS_DIR/apps/cross-platform/tauri/build.sh"fidiff --git a/libs/cua-driver/test-harness/build/macos.sh b/libs/cua-driver/test-harness/build/macos.shindex 1bd8e0a756..233e464b72 100755--- a/libs/cua-driver/test-harness/build/macos.sh+++ b/libs/cua-driver/test-harness/build/macos.sh@@ -1,13 +1,14 @@#!/usr/bin/env bash-# Build the macOS test-harness apps (AppKit + SwiftUI + WKWebView + Electron)+# Build the macOS test-harness apps (AppKit + SwiftUI + WKWebView + Electron + Tauri)# into app bundles under ../../rust/test-apps/harness-* —# matching the convention used by windows.ps1 for the Windows WPF + WinUI3 +# WebView2 apps. WKWebView is the Apple-WebKit host; Electron is the shared-# Chromium harness used across Windows, Linux, and macOS.+# Chromium harness used across Windows, Linux, and macOS. Tauri is the shared+# native-webview harness used across Windows, Linux, and macOS.## Usage:# ./macos.sh # build all macOS-runnable harnesses-# ./macos.sh --skip swiftui # skip one target (appkit|swiftui|wkwebview|electron)+# ./macos.sh --skip swiftui # skip one target (appkit|swiftui|wkwebview|electron|tauri)# ./macos.sh --clean # remove staged outputs firstset -euo pipefail@@ -28,8 +29,8 @@ while [[ $# -gt 0 ]]; dodoneif [[ "$CLEAN" == "1" ]]; then- rm -rf "$STAGE_DIR/harness-appkit" "$STAGE_DIR/harness-swiftui" "$STAGE_DIR/harness-wkwebview" "$STAGE_DIR/harness-electron"- mkdir -p "$STAGE_DIR/harness-appkit" "$STAGE_DIR/harness-swiftui" "$STAGE_DIR/harness-wkwebview" "$STAGE_DIR/harness-electron"+ rm -rf "$STAGE_DIR/harness-appkit" "$STAGE_DIR/harness-swiftui" "$STAGE_DIR/harness-wkwebview" "$STAGE_DIR/harness-electron" "$STAGE_DIR/harness-tauri"+ mkdir -p "$STAGE_DIR/harness-appkit" "$STAGE_DIR/harness-swiftui" "$STAGE_DIR/harness-wkwebview" "$STAGE_DIR/harness-electron" "$STAGE_DIR/harness-tauri"echo "==> Cleaned stage dirs"fi@@ -108,6 +109,10 @@ if [[ "$SKIP" != "electron" ]]; then"$HARNESS_DIR/apps/cross-platform/electron/build.sh"fi+if [[ "$SKIP" != "tauri" ]]; then+ "$HARNESS_DIR/apps/cross-platform/tauri/build.sh"+fi+echo ""echo "Built apps:"find "$STAGE_DIR" -maxdepth 3 -name "*.app" -type ddiff --git a/libs/cua-driver/test-harness/build/windows.ps1 b/libs/cua-driver/test-harness/build/windows.ps1index 23e0f8127e..aafe772a2b 100644--- a/libs/cua-driver/test-harness/build/windows.ps1+++ b/libs/cua-driver/test-harness/build/windows.ps1@@ -1,6 +1,6 @@# windows.ps1 - publish the cua-driver test harness binaries.#-# Outputs land in libs/cua-driver/rust/test-apps/harness-{wpf,winui3,webview,electron}/+# Outputs land in libs/cua-driver/rust/test-apps/harness-{wpf,winui3,webview,electron,tauri}/# so the existing sandbox runner picks them up via its mapped folder.## Usage:@@ -8,10 +8,10 @@# .\windows.ps1 -Skip wpf # skip WPF# .\windows.ps1 -Skip winui3#-# Requires: .NET 8 SDK on PATH (Node.js + npm for the Electron app).+# Requires: .NET 8 SDK on PATH (Node.js + npm for Electron, Rust for Tauri).param(- [ValidateSet("none","wpf","winui3","webview","electron")]+ [ValidateSet("none","wpf","winui3","webview","electron","tauri")][string]$Skip = "none")@@ -92,6 +92,23 @@ if ($Skip -ne "electron") {Write-Host "[SKIP] Electron project not present yet - skipping." -ForegroundColor Yellow}}+if ($Skip -ne "tauri") {+ $tauriBuild = Join-Path $harnessDir "apps\cross-platform\tauri\build.ps1"+ if (Test-Path $tauriBuild) {+ Write-Host ""+ Write-Host "[BUILD] tauri -> $testAppsDir\harness-tauri\" -ForegroundColor Cyan+ try {+ & $tauriBuild+ if ($LASTEXITCODE -ne 0) {+ Write-Host "[WARN] Tauri build failed (exit $LASTEXITCODE)" -ForegroundColor Yellow+ }+ } catch {+ Write-Host "[WARN] Tauri build errored: $($_.Exception.Message)" -ForegroundColor Yellow+ }+ } else {+ Write-Host "[SKIP] Tauri project not present yet - skipping." -ForegroundColor Yellow+ }+}Write-Host ""Write-Host "[DONE] Test harness build complete." -ForegroundColor Greendiff --git a/libs/cua-driver/test-harness/smoke/results/macos.txt b/libs/cua-driver/test-harness/smoke/results/macos.txtdeleted file mode 100644index d18911ccff..0000000000--- a/libs/cua-driver/test-harness/smoke/results/macos.txt+++ /dev/null@@ -1,72 +0,0 @@-============================================================-macOS cua-driver smoke test-Driver: ~/cua/libs/cua-driver/rust/target/release/cua-driver-Version: cua-driver 0.2.18-Harness: ~/cua/libs/cua-driver/rust/test-apps/harness-appkit/CuaTestHarness.AppKit.app-Date: Tue May 26 01:00:13 CEST 2026-============================================================--==> Spawning AppKit harness as victim- Harness pid=52884- Window id=18195--=== group 1: no-arg / config tools ===--=== group 2: setters ===--=== group 3: app lifecycle ===- TextEdit pid=55979--=== group 4: per-window state (harness pid=52884 wid=18195) ===--=== group 5: input synthesis (against harness) ===--=== group 6: aliases + special ===--==> cleanup-~/cua/scripts/mac-smoke.sh: line 184: 52884 Killed: 9 "$HARNESS_EXE" > /dev/null 2>&1--============================================================-SUMMARY-============================================================-TOOL VERDICT DETAIL------------------------------------------------------------------bring_to_front SKIP intentional cross-platform stub on macOS-check_permissions PASS {-click PASS ✅ Posted click to pid 52884.-double_click PASS ✅ Double-clicked at (670.0, 102.0).-drag PASS ✅ Posted drag to pid 52884 from window-pixel (120, 80) → (180, 120), screen (670-get_accessibility_tree PASS {-get_agent_cursor_state PASS {-get_config PASS {-get_cursor_position PASS {-get_recording_state PASS {-get_screen_size PASS {-get_window_state PASS {-hotkey PASS Pressed cmd+a on pid 52884.-kill_app PASS ✅ Sent SIGKILL to pid 55979.-launch_app PASS {-list_apps PASS {-list_windows PASS {-move_cursor PASS Agent cursor 'default' moved to (400.0, 400.0).-page SKIP needs Chromium with --remote-debugging-port-press_key PASS ✅ Pressed a on pid 52884.-replay_trajectory SKIP needs a recorded trajectory file-right_click PASS Right-clicked at (670.0, 102.0).-scroll PASS Scrolled down by line × 3.-set_agent_cursor_enabled PASS Agent cursor 'default' enabled.-set_agent_cursor_motion PASS {-set_agent_cursor_style PASS ✅ cursor style: gradient_colors=(unchanged) bloom_color=(unchanged) image_path=(-set_config PASS Config updated: capture_mode=som, max_image_dimension=1024-set_recording PASS {-set_value SKIP exercised by harness_appkit_text_input integration test instead-type_text PASS ✅ Inserted 2 char(s) into focused AXTextField "".-type_text_chars SKIP deprecated alias resolved by mcp-server, not registered-zoom PASS {--Tools probed: 32 PASS=27 FAIL=0 SKIP=5--Interpretation:- PASS = tool ran cleanly (exit 0, no ❌ in output)- FAIL = error or non-zero exit- SKIP = intentionally not probed (covered by integration tests, missing fixture, etc.)diff --git a/libs/cua-driver/rust/tests/integration/v2/PHILOSOPHY.md b/libs/cua-driver/tests/e2e/PHILOSOPHY.mdsimilarity index 82%rename from libs/cua-driver/rust/tests/integration/v2/PHILOSOPHY.mdrename to libs/cua-driver/tests/e2e/PHILOSOPHY.mdindex ec1b83144e..5e0d62aeba 100644--- a/libs/cua-driver/rust/tests/integration/v2/PHILOSOPHY.md+++ b/libs/cua-driver/tests/e2e/PHILOSOPHY.md@@ -1,4 +1,4 @@-# cua-driver v2 Integration Test Philosophy+# cua-driver Python E2E Test Philosophy## Core Principle@@ -65,12 +65,12 @@ fails with a list of every incident.| App | Primary oracle | Cross-check ||-----|---------------|-------------|-| Safari | AX tree text values | — |-| Chrome | AX tree text values | — |-| Electron | AX tree text values | HTTP API on port 6769 |-| Tauri | AX tree text values | HTTP API on port 6769 |+| Safari | AX tree text values | - |+| Chrome | AX tree text values | - |+| Electron | AX tree text values from shared DOM | - |+| Tauri | AX tree text values from shared DOM | - || Blender | Screenshot + NCC template match | AX tree window title |-| Calculator | AX tree static text | — |+| Calculator | AX tree static text | - |For browser apps the test page (`assets/test_page.html`) provides aself-contained oracle: every interactive element reflects its state back into@@ -95,28 +95,13 @@ assert cv.diff_ratio(before_crop, after_crop) > 0.001----## Tauri / Electron HTTP API--The HTTP API is a **cross-check**, not the primary assertion:--```python-# Primary: AX tree-after = get_window_state(tauri_pid, wid)-assert ax_value(after.tree, field_idx) == "tauri test"--# Cross-check: HTTP event log-events = requests.get("http://localhost:6769/events").json()-assert any(e["type"] == "key_down" for e in events)-```------## Test File Structure```-v2/+tests/e2e/PHILOSOPHY.md ← this fileconftest.py ← pytest fixtures (driver, ux_guard, sentinel, html_server, ...)+ driver_client.py ← raw MCP client wrapperharness/__init__.pymonitor.py ← UXMonitor (5 ms PyObjC thread)@@ -126,12 +111,13 @@ v2/assets/test_page.html ← all-elements test page for browser testsblender/- bboxes.json ← pre-measured bounding boxes for template targets+ toolbar_ref.png ← stored template targettest_safari.pytest_chrome.pytest_electron.pytest_tauri.pytest_blender.py+ test_*.py ← older e2e coverage kept here until audited/ported```---@@ -139,8 +125,8 @@ v2/## Running```bash-cd tests/integration/v2-export CUA_DRIVER_BINARY=../../../target/debug/cua-driver+cd libs/cua-driver/tests/e2e+export CUA_DRIVER_BINARY=../../rust/target/debug/cua-driver# Single filepython3 -m pytest test_safari.py -vdiff --git a/libs/cua-driver/tests/e2e/README.md b/libs/cua-driver/tests/e2e/README.mdnew file mode 100644index 0000000000..463f03bf71--- /dev/null+++ b/libs/cua-driver/tests/e2e/README.md@@ -0,0 +1,46 @@+# cua-driver Python e2e tests++Python e2e tests drive the public MCP tool interface against real apps and+shared harness apps. They are manual/VM-backed tests, not default CI tests.++## Layout++```text+tests/e2e/+├── driver_client.py # raw MCP client+├── conftest.py # pytest fixtures+├── harness/ # typed wrappers, tree helpers, CV helpers, UX monitor+├── assets/ # symlinks/references for browser and Blender tests+├── PHILOSOPHY.md # black-box test pattern and UX invariants+├── run_tests.sh # pytest runner that builds cua-driver if needed+└── test_*.py # app and legacy e2e modules+```++## Running++```bash+cd libs/cua-driver/tests/e2e+./run_tests.sh test_electron -v+python3 -m pytest test_tauri.py -v+```++Set `CUA_DRIVER_BINARY` to test a specific binary. Without it, `run_tests.sh`+builds `libs/cua-driver/rust` and uses `rust/target/debug/cua-driver`.++## Harness Apps++`test_electron.py` and `test_tauri.py` use repo-local shared harnesses:++- `test-harness/apps/cross-platform/electron`+- `test-harness/apps/cross-platform/tauri`++The fixtures build missing staged apps on demand with each harness `build.sh`.+They no longer download prebuilt app releases.++## Rules++- Assert through observable state in the AX/UIA/AT-SPI tree or screenshots.+- Do not call internal browser APIs from tests.+- Keep app-specific setup in fixtures, not in driver code.+- Treat older root-level tests as legacy coverage until they are audited and+ either ported into the shared harness pattern or removed.diff --git a/libs/cua-driver/rust/tests/integration/v2/assets/blender/toolbar_ref.png b/libs/cua-driver/tests/e2e/assets/blender/toolbar_ref.pngsimilarity index 100%rename from libs/cua-driver/rust/tests/integration/v2/assets/blender/toolbar_ref.pngrename to libs/cua-driver/tests/e2e/assets/blender/toolbar_ref.pngdiff --git a/libs/cua-driver/tests/e2e/assets/test_page.html b/libs/cua-driver/tests/e2e/assets/test_page.htmlnew file mode 120000index 0000000000..2771775117--- /dev/null+++ b/libs/cua-driver/tests/e2e/assets/test_page.html@@ -0,0 +1 @@+../../../../cua-driver-fixtures/test_page.html\ No newline at end of filediff --git a/libs/cua-driver/rust/tests/integration/v2/conftest.py b/libs/cua-driver/tests/e2e/conftest.pysimilarity index 82%rename from libs/cua-driver/rust/tests/integration/v2/conftest.pyrename to libs/cua-driver/tests/e2e/conftest.pyindex 8580c8f429..d6f818dfe6 100644--- a/libs/cua-driver/rust/tests/integration/v2/conftest.py+++ b/libs/cua-driver/tests/e2e/conftest.py@@ -1,4 +1,4 @@-"""pytest fixtures for cua-driver v2 integration tests.+"""pytest fixtures for cua-driver Python e2e tests.Fixtures:binary → str — path to cua-driver binary@@ -6,7 +6,7 @@ Fixtures:focus_monitor → (proc, pid) — FocusMonitorApp running as frontmost sentinelux_guard → UXMonitor — started before the test, assert_clean() afterhtml_server → str — base URL of a local HTTP server serving assets/- tauri_app → (proc, pid, base_url) — Tauri desktop-test-app+ tauri_app → (proc, pid, base_url) — repo-local Tauri harnesselectron_app → (proc, pid, base_url) — repo-local Electron harness"""@@ -17,20 +17,18 @@ import subprocessimport sysimport timeimport threading-import urllib.requestfrom http.server import HTTPServer, SimpleHTTPRequestHandlerimport pytest_HERE = os.path.dirname(os.path.abspath(__file__))-_INTEG_DIR = os.path.dirname(_HERE)-_TESTS_DIR = os.path.dirname(_INTEG_DIR)-_DRIVER_RS_ROOT = os.path.dirname(_TESTS_DIR)-_LIBS_ROOT = os.path.dirname(_DRIVER_RS_ROOT)-_CUA_DRIVER_ROOT = os.path.join(_LIBS_ROOT, "cua-driver")+_TESTS_ROOT = os.path.dirname(_HERE)+_CUA_DRIVER_ROOT = os.path.dirname(_TESTS_ROOT)+_DRIVER_RS_ROOT = os.path.join(_CUA_DRIVER_ROOT, "rust")+_LIBS_ROOT = os.path.dirname(_CUA_DRIVER_ROOT)_TEST_HARNESS_ROOT = os.path.join(_CUA_DRIVER_ROOT, "test-harness")-sys.path.insert(0, _INTEG_DIR)+sys.path.insert(0, _HERE)from driver_client import default_binary_path # noqa: E402_ASSETS_DIR = os.path.join(_HERE, "assets")@@ -39,16 +37,20 @@ _FOCUS_APP_DIR = os.path.join(_LIBS_ROOT, "cua-driver", "Tests", "FocusMonitorAp_FOCUS_APP_BUNDLE = os.path.join(_FOCUS_APP_DIR, "FocusMonitorApp.app")_FOCUS_APP_EXE = os.path.join(_FOCUS_APP_BUNDLE, "Contents", "MacOS", "FocusMonitorApp")-# Tauri app download config-# Tauri releases a raw Mach-O binary (no .app bundle, no zip).-_TAURI_VERSION = "v0.2.2"-_TAURI_BINARY_NAME = "desktop-test-app-macos-arm64"-_TAURI_DOWNLOAD_URL = (- f"https://github.com/trycua/desktop-test-app/releases/download/"- f"{_TAURI_VERSION}/{_TAURI_BINARY_NAME}"+# Tauri is built from the repo-local shared harness rather than downloaded.+_TAURI_HARNESS_DIR = os.path.join(+ _TEST_HARNESS_ROOT, "apps", "cross-platform", "tauri"+)+_TAURI_BUILD_SCRIPT = os.path.join(_TAURI_HARNESS_DIR, "build.sh")+_TAURI_APP_BUNDLE = os.path.join(+ _DRIVER_RS_ROOT,+ "test-apps",+ "harness-tauri",+ "CuaTestHarness.Tauri.app",+)+_TAURI_APP_EXE = os.path.join(+ _TAURI_APP_BUNDLE, "Contents", "MacOS", "CuaTestHarness.Tauri")-_TAURI_APP_DIR = os.path.join(_HERE, "assets", "tauri")-_TAURI_BINARY_PATH = os.path.join(_TAURI_APP_DIR, _TAURI_BINARY_NAME)# Electron is built from the repo-local shared harness rather than downloaded._ELECTRON_HARNESS_DIR = os.path.join(@@ -201,42 +203,39 @@ def html_server():server.shutdown()-# ── Tauri / Electron helpers ──────────────────────────────────────────────────---def _download_binary(url: str, dest_path: str) -> None:- """Download a raw binary; skip if it already exists."""- if os.path.exists(dest_path):- return- os.makedirs(os.path.dirname(dest_path), exist_ok=True)- print(f"\n Downloading {url} …")- urllib.request.urlretrieve(url, dest_path)- os.chmod(dest_path, 0o755)--# ── Tauri app ─────────────────────────────────────────────────────────────────@pytest.fixture(scope="session")def tauri_app():- """Download, launch, and yield (proc, pid, base_url) for the Tauri test app.+ """Build, launch, and yield (proc, pid, base_url) for the Tauri harness.- The Tauri release ships a raw Mach-O binary (no .app bundle, no zip).- We launch it directly via subprocess.Popen.+ This uses test-harness/apps/cross-platform/tauri, the same shared DOM+ harness used by the Electron/WebView/WKWebView harnesses. It does not+ download a prebuilt Tauri app release."""- _download_binary(_TAURI_DOWNLOAD_URL, _TAURI_BINARY_PATH)+ if not os.path.exists(_TAURI_APP_EXE):+ if not os.path.exists(_TAURI_BUILD_SCRIPT):+ pytest.skip("repo-local Tauri harness build script not found")+ try:+ subprocess.run([_TAURI_BUILD_SCRIPT], check=True)+ except (OSError, subprocess.CalledProcessError) as e:+ pytest.skip(f"Tauri harness build failed: {e}")++ if not os.path.exists(_TAURI_APP_EXE):+ pytest.skip("Tauri harness app not found after build")- subprocess.run(["pkill", "-f", _TAURI_BINARY_NAME], check=False)+ subprocess.run(["pkill", "-f", "CuaTestHarness.Tauri"], check=False)time.sleep(0.5)proc = subprocess.Popen(- [_TAURI_BINARY_PATH],+ [_TAURI_APP_EXE],stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL,)pid = proc.pidtime.sleep(3.0)- base_url = "http://localhost:6769"+ base_url = "file://shared-web-harness"yield proc, pid, base_urlproc.terminate()@@ -244,7 +243,7 @@ def tauri_app():proc.wait(timeout=5)except subprocess.TimeoutExpired:proc.kill()- subprocess.run(["pkill", "-f", _TAURI_BINARY_NAME], check=False)+ subprocess.run(["pkill", "-f", "CuaTestHarness.Tauri"], check=False)# ── Electron app ──────────────────────────────────────────────────────────────diff --git a/libs/cua-driver/rust/tests/integration/driver_client.py b/libs/cua-driver/tests/e2e/driver_client.pysimilarity index 99%rename from libs/cua-driver/rust/tests/integration/driver_client.pyrename to libs/cua-driver/tests/e2e/driver_client.pyindex 850b16abeb..e6aefd43d2 100644--- a/libs/cua-driver/rust/tests/integration/driver_client.py+++ b/libs/cua-driver/tests/e2e/driver_client.py@@ -153,6 +153,7 @@ def default_binary_path() -> str:os.path.dirname(os.path.abspath(__file__)),"..","..",+ "rust","target","debug","cua-driver",diff --git a/libs/cua-driver/rust/tests/integration/fixtures/interactive.html b/libs/cua-driver/tests/e2e/fixtures/interactive.htmlsimilarity index 100%rename from libs/cua-driver/rust/tests/integration/fixtures/interactive.htmlrename to libs/cua-driver/tests/e2e/fixtures/interactive.htmldiff --git a/libs/cua-driver/rust/tests/integration/fixtures/test_cursor.png b/libs/cua-driver/tests/e2e/fixtures/test_cursor.pngsimilarity index 100%rename from libs/cua-driver/rust/tests/integration/fixtures/test_cursor.pngrename to libs/cua-driver/tests/e2e/fixtures/test_cursor.pngdiff --git a/libs/cua-driver/rust/tests/integration/v2/harness/__init__.py b/libs/cua-driver/tests/e2e/harness/__init__.pysimilarity index 100%rename from libs/cua-driver/rust/tests/integration/v2/harness/__init__.pyrename to libs/cua-driver/tests/e2e/harness/__init__.pydiff --git a/libs/cua-driver/rust/tests/integration/v2/harness/cv.py b/libs/cua-driver/tests/e2e/harness/cv.pysimilarity index 100%rename from libs/cua-driver/rust/tests/integration/v2/harness/cv.pyrename to libs/cua-driver/tests/e2e/harness/cv.pydiff --git a/libs/cua-driver/rust/tests/integration/v2/harness/driver.py b/libs/cua-driver/tests/e2e/harness/driver.pysimilarity index 97%rename from libs/cua-driver/rust/tests/integration/v2/harness/driver.pyrename to libs/cua-driver/tests/e2e/harness/driver.pyindex cdec18f487..1409768f5b 100644--- a/libs/cua-driver/rust/tests/integration/v2/harness/driver.py+++ b/libs/cua-driver/tests/e2e/harness/driver.py@@ -1,4 +1,4 @@-"""Typed MCP driver wrapper for v2 integration tests.+"""Typed MCP driver wrapper for Python e2e tests.Wraps DriverClient with:- Typed WindowState return value@@ -23,11 +23,10 @@ import timefrom dataclasses import dataclass, fieldfrom typing import Optional-# Allow importing from parent directory (old driver_client.py).+# Allow importing from the e2e root._HERE = os.path.dirname(os.path.abspath(__file__))-_V2_DIR = os.path.dirname(_HERE)-_INTEG_DIR = os.path.dirname(_V2_DIR)-sys.path.insert(0, _INTEG_DIR)+_E2E_DIR = os.path.dirname(_HERE)+sys.path.insert(0, _E2E_DIR)from driver_client import DriverClient, default_binary_path, MCPCallError # noqa: E402diff --git a/libs/cua-driver/rust/tests/integration/v2/harness/monitor.py b/libs/cua-driver/tests/e2e/harness/monitor.pysimilarity index 100%rename from libs/cua-driver/rust/tests/integration/v2/harness/monitor.pyrename to libs/cua-driver/tests/e2e/harness/monitor.pydiff --git a/libs/cua-driver/rust/tests/integration/v2/harness/tree.py b/libs/cua-driver/tests/e2e/harness/tree.pysimilarity index 100%rename from libs/cua-driver/rust/tests/integration/v2/harness/tree.pyrename to libs/cua-driver/tests/e2e/harness/tree.pydiff --git a/libs/cua-driver/rust/tests/integration/run_tests.sh b/libs/cua-driver/tests/e2e/run_tests.shsimilarity index 81%rename from libs/cua-driver/rust/tests/integration/run_tests.shrename to libs/cua-driver/tests/e2e/run_tests.shindex 0a1d3c6052..2f98162239 100755--- a/libs/cua-driver/rust/tests/integration/run_tests.sh+++ b/libs/cua-driver/tests/e2e/run_tests.sh@@ -1,5 +1,5 @@#!/usr/bin/env bash-# Run cua-driver-rs integration tests.+# Run cua-driver Python e2e tests.## Usage:# ./run_tests.sh # run all tests (Rust binary)@@ -11,7 +11,7 @@# CUA_DRIVER_BINARY is already set in the environment.## Options:-# -v / --verbose Pass -v to unittest for verbose output.+# -v / --verbose Pass -v to pytest for verbose output.# --release Build with --release (sets CUA_DRIVER_BINARY to release binary).# --parity Run test_api_parity.py against both Swift and Rust binaries.# Requires CUA_SWIFT_BINARY or ~/.local/bin/cua-driver.@@ -19,7 +19,8 @@set -euo pipefailSCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"-WORKSPACE_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"+CUA_DRIVER_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"+RUST_ROOT="$CUA_DRIVER_ROOT/rust"VERBOSE=""PROFILE="debug"@@ -41,8 +42,8 @@ if [ -z "${CUA_DRIVER_BINARY:-}" ]; thenecho "==> Building cua-driver ($PROFILE)..."CARGO_FLAGS=""[ "$PROFILE" = "release" ] && CARGO_FLAGS="--release"- (cd "$WORKSPACE_ROOT" && cargo build $CARGO_FLAGS -p cua-driver 2>&1)- export CUA_DRIVER_BINARY="$WORKSPACE_ROOT/target/$PROFILE/cua-driver"+ (cd "$RUST_ROOT" && cargo build $CARGO_FLAGS -p cua-driver 2>&1)+ export CUA_DRIVER_BINARY="$RUST_ROOT/target/$PROFILE/cua-driver"fiecho "==> Using Rust binary: $CUA_DRIVER_BINARY"@@ -72,7 +73,7 @@ cd "$SCRIPT_DIR"if [ "$PARITY_MODE" -eq 1 ]; thenecho "==> Running parity suite against both binaries..."- python3 -m unittest ${VERBOSE} test_api_parity+ python3 -m pytest ${VERBOSE} test_api_parity.pyexit $?fi@@ -84,4 +85,13 @@ fiecho "==> Running ${#MODULES[@]} test module(s): ${MODULES[*]}"echo ""-python3 -m unittest ${VERBOSE} "${MODULES[@]}"+FILES=()+for module in "${MODULES[@]}"; do+ if [[ "$module" == *.py ]]; then+ FILES+=("$module")+ else+ FILES+=("$module.py")+ fi+done++python3 -m pytest ${VERBOSE} "${FILES[@]}"diff --git a/libs/cua-driver/rust/tests/integration/test_api_parity.py b/libs/cua-driver/tests/e2e/test_api_parity.pysimilarity index 100%rename from libs/cua-driver/rust/tests/integration/test_api_parity.pyrename to libs/cua-driver/tests/e2e/test_api_parity.pydiff --git a/libs/cua-driver/rust/tests/integration/test_background_focus.py b/libs/cua-driver/tests/e2e/test_background_focus.pysimilarity index 100%rename from libs/cua-driver/rust/tests/integration/test_background_focus.pyrename to libs/cua-driver/tests/e2e/test_background_focus.pydiff --git a/libs/cua-driver/rust/tests/integration/v2/test_blender.py b/libs/cua-driver/tests/e2e/test_blender.pysimilarity index 100%rename from libs/cua-driver/rust/tests/integration/v2/test_blender.pyrename to libs/cua-driver/tests/e2e/test_blender.pydiff --git a/libs/cua-driver/rust/tests/integration/v2/test_chrome.py b/libs/cua-driver/tests/e2e/test_chrome.pysimilarity index 100%rename from libs/cua-driver/rust/tests/integration/v2/test_chrome.pyrename to libs/cua-driver/tests/e2e/test_chrome.pydiff --git a/libs/cua-driver/rust/tests/integration/test_cli.py b/libs/cua-driver/tests/e2e/test_cli.pysimilarity index 100%rename from libs/cua-driver/rust/tests/integration/test_cli.pyrename to libs/cua-driver/tests/e2e/test_cli.pydiff --git a/libs/cua-driver/rust/tests/integration/test_concurrent_drivers.py b/libs/cua-driver/tests/e2e/test_concurrent_drivers.pysimilarity index 100%rename from libs/cua-driver/rust/tests/integration/test_concurrent_drivers.pyrename to libs/cua-driver/tests/e2e/test_concurrent_drivers.pydiff --git a/libs/cua-driver/rust/tests/integration/test_cursor_visibility.py b/libs/cua-driver/tests/e2e/test_cursor_visibility.pysimilarity index 100%rename from libs/cua-driver/rust/tests/integration/test_cursor_visibility.pyrename to libs/cua-driver/tests/e2e/test_cursor_visibility.pydiff --git a/libs/cua-driver/rust/tests/integration/test_double_click_delivery.py b/libs/cua-driver/tests/e2e/test_double_click_delivery.pysimilarity index 100%rename from libs/cua-driver/rust/tests/integration/test_double_click_delivery.pyrename to libs/cua-driver/tests/e2e/test_double_click_delivery.pydiff --git a/libs/cua-driver/rust/tests/integration/v2/test_electron.py b/libs/cua-driver/tests/e2e/test_electron.pysimilarity index 100%rename from libs/cua-driver/rust/tests/integration/v2/test_electron.pyrename to libs/cua-driver/tests/e2e/test_electron.pydiff --git a/libs/cua-driver/rust/tests/integration/test_focus_steal_parity.py b/libs/cua-driver/tests/e2e/test_focus_steal_parity.pysimilarity index 99%rename from libs/cua-driver/rust/tests/integration/test_focus_steal_parity.pyrename to libs/cua-driver/tests/e2e/test_focus_steal_parity.pyindex 47c4221f7c..51ff506f6d 100644--- a/libs/cua-driver/rust/tests/integration/test_focus_steal_parity.py+++ b/libs/cua-driver/tests/e2e/test_focus_steal_parity.py@@ -38,7 +38,7 @@ built helper made the integration tests flaky to bootstrap on afresh runner — Finder removes that dependency.)Run:- cd libs/cua-driver/rust/tests/integration+ cd libs/cua-driver/tests/e2epython3 -m unittest test_focus_steal_parity -v"""diff --git a/libs/cua-driver/rust/tests/integration/test_health_report_mcp.py b/libs/cua-driver/tests/e2e/test_health_report_mcp.pysimilarity index 100%rename from libs/cua-driver/rust/tests/integration/test_health_report_mcp.pyrename to libs/cua-driver/tests/e2e/test_health_report_mcp.pydiff --git a/libs/cua-driver/rust/tests/integration/test_list_windows.py b/libs/cua-driver/tests/e2e/test_list_windows.pysimilarity index 100%rename from libs/cua-driver/rust/tests/integration/test_list_windows.pyrename to libs/cua-driver/tests/e2e/test_list_windows.pydiff --git a/libs/cua-driver/rust/tests/integration/test_overlay_z_order.py b/libs/cua-driver/tests/e2e/test_overlay_z_order.pysimilarity index 100%rename from libs/cua-driver/rust/tests/integration/test_overlay_z_order.pyrename to libs/cua-driver/tests/e2e/test_overlay_z_order.pydiff --git a/libs/cua-driver/rust/tests/integration/test_pixel_click_delivery.py b/libs/cua-driver/tests/e2e/test_pixel_click_delivery.pysimilarity index 100%rename from libs/cua-driver/rust/tests/integration/test_pixel_click_delivery.pyrename to libs/cua-driver/tests/e2e/test_pixel_click_delivery.pydiff --git a/libs/cua-driver/rust/tests/integration/v2/test_safari.py b/libs/cua-driver/tests/e2e/test_safari.pysimilarity index 100%rename from libs/cua-driver/rust/tests/integration/v2/test_safari.pyrename to libs/cua-driver/tests/e2e/test_safari.pydiff --git a/libs/cua-driver/tests/e2e/test_tauri.py b/libs/cua-driver/tests/e2e/test_tauri.pynew file mode 100644index 0000000000..5845c83824--- /dev/null+++ b/libs/cua-driver/tests/e2e/test_tauri.py@@ -0,0 +1,97 @@+"""Tauri shared-harness background automation tests.++The Tauri test app is built from test-harness/apps/cross-platform/tauri+and loads test-harness/shared/web/index.html, the same DOM used by the+Electron, WebView2, and WKWebView harnesses.++Run: python3 -m pytest test_tauri.py -v+"""++from __future__ import annotations++import subprocess+import sys+import time+import os++import pytest++_HERE = os.path.dirname(os.path.abspath(__file__))+sys.path.insert(0, _HERE)++[email protected](autouse=True)+def _reactivate_focus(focus_monitor):+ _, pid = focus_monitor+ subprocess.run(+ ["osascript", "-e",+ f'tell application "System Events" to set frontmost of (first process whose unix id is {pid}) to true'],+ check=False,+ )+ time.sleep(0.4)+++# ── tests ─────────────────────────────────────────────────────────────────────++class TestTauriButton:+ def test_button_click(self, driver, tauri_app, ux_guard):+ _, pid, _ = tauri_app+ if pid == 0:+ pytest.skip("Tauri app pid not found")++ wid = driver.find_window(pid)++ before = driver.get_window_state(pid, wid, query="Increment")+ idx = before.find_button("Increment") or before.find_element("Increment")+ assert idx is not None, f"'Increment' not found:\n{before.tree}"++ driver.click(pid, wid, element_index=idx)+ time.sleep(0.5)++ after = driver.get_window_state(pid, wid)+ assert after.has_text("counter=1"), f"Counter not incremented:\n{after.tree}"+++class TestTauriTextInput:+ def test_type_text(self, driver, tauri_app, ux_guard):+ _, pid, _ = tauri_app+ if pid == 0:+ pytest.skip("Tauri app pid not found")++ wid = driver.find_window(pid)++ before = driver.get_window_state(pid, wid)+ idx = before.find_text_field(skip_url_bar=False)+ assert idx is not None, f"Text input not found:\n{before.tree}"++ driver.click(pid, wid, element_index=idx)+ time.sleep(0.3)+ driver.hotkey(pid, ["cmd", "a"])+ time.sleep(0.1)+ driver.type_text_chars(pid, "tauri test")+ time.sleep(0.6)++ after = driver.get_window_state(pid, wid)+ assert after.has_text("mirror=tauri test"), f"Text not mirrored in AX tree:\n{after.tree}"+++class TestTauriNoFocusSteal:+ def test_multiple_actions_no_focus_steal(self, driver, tauri_app, ux_guard):+ _, pid, _ = tauri_app+ if pid == 0:+ pytest.skip("Tauri app pid not found")++ wid = driver.find_window(pid)++ # Take 3 actions in sequence — UX guard checks no focus steal+ before = driver.get_window_state(pid, wid, query="Increment")+ idx = before.find_button("Increment") or before.find_element("Increment")+ if idx is not None:+ driver.click(pid, wid, element_index=idx)+ time.sleep(0.3)++ driver.press_key(pid, "tab")+ time.sleep(0.2)+ driver.press_key(pid, "tab")+ time.sleep(0.2)+ # ux_guard teardown will assert_clean()