Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
CUA Driver Test Harnesses Guide
This contributor guide explains where the CUA Driver tests live, how to run
them, and where to add a new case. For current coverage and platform gaps, use
test-matrix.md and action-support.md.
The Short Version
There are two main test layers:
- Unit and protocol tests exercise Rust code and the public MCP/CLI contract without launching a real application.
- Harness E2E tests launch a small application built from this repository, drive it through the Rust driver, and verify an external application or desktop state.
The Rust tests are the source of truth. Python tests, old shell runners, and historical recording scripts are not part of the canonical E2E path.
The canonical E2E command on every OS runs the complete matrix and takes no suite selector:
Linux: scripts/ci/linux/run-rust-e2e.sh
Windows: .\scripts\ci\windows\run-rust-e2e.ps1 -RequireGui
macOS: libs/cua-driver/tests/runners/macos-lume/run-all.sh
scripts/ci/README.md
lists every runner as canonical, scoped, or supporting. The Windows convenience
wrapper in tests/runners/windows/ and the legacy tests/runners/windows-sandbox/
runner are supporting runners, not canonical entrypoints.
The OS workflow may fan the complete matrix out into independent jobs for reporting and failure isolation. That is an execution detail; contributors should think of it as one canonical suite.
Evidence authority
A canonical result is the complete repository harness run at the exact source SHA. A one-off app smoke, manually assembled script, video, or environment replay can diagnose a failure or provide release presentation evidence, but it does not replace the complete matrix.
Windows and Linux use the repository's GitHub-hosted workflows when their
strict environment preflights pass. Windows Azure RDP runs are optional
environment-parity replays or a fallback when the hosted preflight cannot prove
a required capability. macOS uses the logged-in Lume maintainer wrapper.
The macOS wrapper always runs the standalone Chrome/Edge matrix after the
repo-local matrix. For browser-facing changes or browser-use release
certification on Windows and Linux, also run
.github/workflows/e2e-rust-standalone-browsers.yml.
Historical *-plan.md and release evidence documents record
what was run at that time. They are not current execution instructions and do
not override this guide or scripts/ci/README.md.
Hyprland validation decision (2026-09-07)
PR #3572 is the dated, exact-source result record for the input v3 candidate. It distinguishes ordinary CI, the complete hosted X11, Sway, and Windows harnesses, native Hyprland acceptance, and bounded app evidence. A native behavioral pass with a failed wrapper or source-provenance check is not an accepted run. Use that record for results; the coverage requirements below do not assert a passing result or add macOS certification.
The maintainer selects the same canonical Linux Rust harness for native
Hyprland: run scripts/ci/linux/run-rust-e2e.sh with its complete all suite
in the prepared native Hyprland desktop at the exact candidate SHA. Preserve
the runner's required cells, assertions, and evidence checks. Record compositor
and backend provenance; an X11 run does not establish native Hyprland coverage.
Environment failures and failed cells remain failures, not permission to skip
tests or change expected results. In the same desktop, also run
cargo test -p cua-driver-e2e --test hyprland_foreground_test -- --ignored --test-threads=1
and the same command for hyprland_native_observation_test. No automated lane
runs these foreground-safety and exact-identity rows, so
tests/manual-e2e-allowlist.txt records them as a manual native-Hyprland gate.
Before launching the native Hyprland harness, apply this map-time rule in the disposable desktop's Lua configuration and reload it:
hl.window_rule({
name = "cua-canonical-sentinel-animation",
match = {title = "^CuaTestHarness Sentinel \\[cdp=[0-9]+\\]$"},
no_anim = true,
})
The preflight independently verifies no_anim for the exact sentinel, then
waits for mapped fullscreen readiness before its first Driver activation.
hyprctl clients reports geometry goals, not animated surface bounds; repeated
equal goals alone cannot establish animation completion. This rule changes
only the test sentinel's animation, not its placement or the product's geometry
guard. Restore the original configuration after the run. It is a deterministic
fixture requirement, not an Omarchy user configuration requirement.
This is fixture regression coverage. For background TARGET input, the production compatibility check in
platform-linux/src/wayland/hyprland_compatibility.rs admits any native Wayland
window whose app bound the agent seats, so the GTK3 fixture's background pointer
and key rows are delivered through the agent seats on Hyprland. A client that
binds only its first wl_seat (Chromium and Electron through Ozone) refuses with
client_not_bound; Qt 6 keys refuse with client_drops_background_keys. XWayland
targets refuse. Do not add a test bypass to make a refusing client pass.
The accepted foreground extension
adds a separate production FOREGROUND_TARGET route for ordinary native
top-level surfaces, advertised by HELLO with foreground_target:true.
The existing
complete suite covers defined native GTK3, Electron, and Tauri foreground
cases. Acceptance requires those cases to pass. Preserve the runner and tests.
Foreground activation and primary-cursor movement are intentional, with no
restoration promise. Verify exact-target delivery and refusals for held
keys/buttons, grabs, constraints, and drag-and-drop before primary takeover.
Foreground drag cancellation on primary-input/focus transitions requires
review and native evidence. No background refusal may escalate to this route.
Retain a short real-app production smoke and instrumented isolation proof on
both qualified apps as supporting compatibility evidence. Verify actual app
effects, plugin transport attribution, primary-seat isolation, and cleanup;
the uninstrumented smoke alone cannot establish those trace-based claims.
The retained proof at source f180e8828b8f31cc153e3c44eaa89a9c13c5bc68
includes 20 instrumented actions, nine pointer effects, and six observation
intervals, plus an uninstrumented six-action smoke. Saved outputs confirm Calc
cell A1 contains a and the Inkscape object's x-coordinate changes from 40 to
42 while y remains 60. The plugin tree and uninstrumented module hash are
unchanged at 1133a06e4f205cf80188a7ac9e41102f37611fea. This is bounded
supporting evidence, not a complete app matrix or release-byte proof; it
predates the per-target admission above. Background input keeps the plain
compiled evdev/pc105/us keymap and two seats. Chromium, Electron, and
XWayland raw background input refuse; semantic AT-SPI actions are separate.
See production proof preparation
for the bounded plans and their limits.
The explicit Inkscape-only qualification profile
supports a bounded packaging candidate using exact Inkscape 1.4.4-6, with
independent native clients, two app lanes, separate SVG oracles, and third-owner
capacity refusal. It preserves the default Calc/Inkscape profile and the native
all-suite gate. Product, harness, kit, and mapped module identities remain
separate; adding the profile records no new native passing result and does not
let diagnostic trace evidence certify trace-disabled package bytes.
Three complete repetitions of the long Python Calc/Inkscape plan, including the 34 policy cases across both apps, are no longer a merge requirement. Extended Python stress runs remain diagnostics for specific unresolved failures. This decision changes test strategy, records no new passing result, and does not waive the affected CI, native evidence, or release gates in RFC 3550.
Repository Map
cua/
|-- libs/cua-driver/
| |-- rust/
| | |-- crates/
| | | |-- cua-driver/ Rust driver and hermetic integration tests
| | | |-- cua-driver-e2e/ Desktop E2E suites the OS runners select
| | | |-- cua-driver-core/ Shared driver logic and unit tests
| | | |-- cua-driver-testkit/ Shared Rust E2E helpers and evidence capture
| | | |-- platform-linux/ Linux backend
| | | |-- platform-macos/ macOS backend
| | | |-- platform-windows/ Windows backend
| | | `-- cursor-overlay/ Cursor evidence helper
| | |-- test-apps/ Ignored staged harness binaries
| | `-- Cargo.toml Rust workspace
| |-- tests/
| | |-- fixtures/
| | | |-- shared/web/ Shared web page and external markers
| | | |-- apps/ Repo-local fixture sources
| | | `-- build/ macOS/Linux/Windows fixture builders
| | `-- runners/
| | `-- macos-lume/ Maintainer Lume setup and guest entrypoint
| `-- docs/ Test matrix, reporting, and contributor docs
`-- scripts/ci/
|-- linux/run-rust-e2e.sh Linux canonical runner
|-- windows/run-rust-e2e.ps1 Windows canonical runner
`-- macos/run-rust-e2e.sh macOS canonical runner
The important separation is:
| Layer | Owns | Does not own |
|---|---|---|
| Rust integration test | Scenarios, driver calls, assertions, action metadata | OS setup and fixture compilation |
cua-driver-testkit |
Session helpers, fixture launching, screenshots, recordings, trajectories | The scenario list |
| Fixture app | Visible controls and externally observable state markers | Driver correctness assertions |
| OS runner | Build environment, user session, test selection, artifact collection | Test behavior definitions |
There is deliberately no second Python E2E implementation that the Rust suite has to mirror.
How One E2E Test Works
Every canonical E2E cell follows this shape:
OS runner
-> builds the Rust driver and repo-local fixture
-> starts or connects to a real desktop session
-> Rust testkit starts one harness application
-> Rust test discovers the target window
-> get_window_state provides accessibility tree and screenshot
-> action addresses a target by AX element index or PX coordinates
-> delivery is foreground or background where the action supports it
-> fixture state, focus, pixels, or protocol response is checked
-> testkit writes video, screenshots, trajectory, logs, and result data
A tool returning ok is not enough to pass an E2E cell. The fixture must show
that the action happened, or the test must verify a documented structured
refusal and the absence of focus or input side effects.
The Test Layers
Unit and Protocol Tests
These run without a repo-local GUI application and normally run without
--ignored:
| Location or prefix | What it proves |
|---|---|
rust/crates/*/src/** |
Core driver, platform-independent logic, schemas, and helpers |
protocol_*_test.rs |
MCP handshake, tool calls, sessions, media, and errors |
schema_*_test.rs |
Shared schema and backend consistency |
transport_config_persistence_test.rs |
CLI configuration persistence across processes |
protocol_element_token_test.rs |
Element-token protocol behavior |
tool_boundary_fuzz_smoke.rs |
Tool-call boundary fuzz targets over seeds and random bytes |
These tests should be fast, deterministic, and safe to run on ordinary CI workers. They do not prove that a real click, key, scroll, or background input reached an application.
The fuzz smoke test drives the target bodies in
cua_driver_testkit::boundary_fuzz (JSON-RPC parsing, reserved-argument
stripping, argument normalisation, authorization, typed contract inputs, and
ToolRegistry dispatch against stubs). The same bodies run under libFuzzer
from rust/fuzz/; see rust/fuzz/README.md and
docs/2026-09-04-tool-call-boundary-fuzzing-design.md.
Harness E2E Tests
These are Rust integration tests in their own crate:
libs/cua-driver/rust/crates/cua-driver-e2e/tests/
Hermetic protocol, CLI, and schema tests stay in
libs/cua-driver/rust/crates/cua-driver/tests/. A test that needs a desktop,
built fixtures, installed apps, or platform permissions belongs in
cua-driver-e2e. Most E2E tests are marked #[ignore]; the OS runner selects
them with cargo test -p cua-driver-e2e --test <suite> -- --ignored. The
crate does not depend on the driver, so build cua-driver first or set
CUA_TEST_DRIVER_BIN. Its few non-ignored tests are hermetic oracle and
parser checks that ordinary CI runs with the driver's tests.
The canonical E2E suite has two behavior owners:
| Owner | Purpose |
|---|---|
| Shared app | Same web behavior tested through Electron and Tauri |
| Native harness | Toolkit-specific controls and native window behavior |
WebView, CDP, and page-tool integration stays inside the shared or native owner that exercises it. It is not a third public test family or command.
Delivery is not a test family. It is a dimension on each action row: an action is tested in foreground and background modes whenever the driver and OS support both. Capture and desktop scope are separate environment checks, while focus preservation is a cross-cutting oracle that can be attached to any action row. Focus, z-order, cursor, and desktop-state checks are cross-cutting invariants, not a separate family. These names describe responsibilities, not separate sources of truth or commands to run instead of the selector-free canonical invocation.
What The Complete Run Includes
Windows
Runner: scripts/ci/windows/run-rust-e2e.ps1
| Runner area | Rust test | Real harness or app |
|---|---|---|
| Shared app matrix | cross_platform_behavior_test.rs |
Electron and Tauri |
| Native controls | harness_wpf_test.rs |
Repo-local WPF app |
| Native controls | harness_winui3_test.rs |
Repo-local WinUI3 app |
| Web integration | harness_web_test.rs |
WebView2 and Electron |
| Capture contract | capture_contract_test.rs |
WPF plus driver tree/image output |
| Launch contract | launch_windows_test.rs |
Repo-local Electron launch and focus behavior |
| Agent cursor | agent_cursor_windows_test.rs |
Source-built cursor overlay and pixel evidence |
| Desktop scope | desktop_scope_windows_test.rs |
Windowless desktop input and scope rejection |
Cross-cutting instrumentation used by these rows includes the testkit
DesktopObserver, capture validation, cursor evidence, and desktop-scope
checks. These invariants are attached to their owning action rows rather than
run as a separate test family.
Windows is currently the broadest native matrix. It covers UIA controls, web integration, background focus checks, and Windows-specific input routes. The desktop observer is attached to shared and native action rows wherever background delivery is tested.
macOS
Canonical runner: libs/cua-driver/tests/runners/macos-lume/run-all.sh
Canonical Actions wrapper: manually dispatch
.github/workflows/e2e-rust-macos.yml in lume mode at the exact source SHA. It runs the
logged-in Lume wrapper, including its standalone browser matrix, and emits the certification
artifact consumed by protected evidence workflows.
Supplemental hosted runner: manually dispatch
.github/workflows/e2e-rust-macos.yml with an exact 40-character source SHA.
Its probe must pass before independent shared, native, capture, and browser
jobs run. The browser job runs the standalone installed Chrome/Edge matrix that
the Lume gate runs after its repo-local matrix.
| Runner area | Rust test | Real harness or app |
|---|---|---|
| Shared app matrix | cross_platform_behavior_test.rs |
Electron and Tauri |
| Native web matrix | cross_platform_behavior_test.rs |
Repo-local WKWebView host |
| Native controls | harness_appkit_test.rs |
Repo-local AppKit app |
| Native controls | harness_swiftui_test.rs |
Repo-local SwiftUI app |
| Installed app launch | installed_app_launch_macos_test.rs |
Calculator and TextEdit |
| Installed app AX delivery | installed_app_textedit_macos_test.rs |
TextEdit |
| Exact window activation | bring_to_front_macos_test.rs |
Repo-local AppKit and SwiftUI apps |
| Pointer-reading toolkit | tk_pointer_click_macos_test.rs |
Tk visual-only canvas (skips sans Tk) |
| Capture contract | capture_contract_test.rs |
Installed driver and macOS capture APIs |
| Desktop scope | desktop_scope_macos_test.rs |
macOS window and desktop scope |
| Standalone browsers | standalone_browser_behavior_test.rs |
Installed Google Chrome and Edge |
The WKWebView host runs the same typed shared-web catalog as Electron and
Tauri. Calculator and TextEdit add typed supporting rows for built-in app
launch focus and native Cocoa background value delivery. They run in the
canonical logged-in macOS lane, but they do not replace repo-local fixtures.
The maintainer wrapper provisions the exact source build and verifies the
private Lume seed's TCC/signing contract before delegating the behavior matrix
to scripts/ci/macos/run-rust-e2e.sh.
The hosted wrapper uses the same behavior matrix on fresh macos-26 runners.
It requires the GitHub-hosted Aqua session and SIP-off VirtualMac environment,
creates a temporary code-signing Keychain, installs a certificate-signed local
app, seeds only Accessibility and Screen Capture for that app's exact csreq, and
verifies that permission status is attributed to the driver daemon before any
test begins. The browser lane additionally requires the image's vendor-signed
Google Chrome and Microsoft Edge and fails closed when either is missing. The
temporary certificate trust is removed during job cleanup.
Hosted results remain supplemental until the image and signing
identity provide the same release-parity guarantees as the maintained Lume
seed.
Workers cloned from a granted private seed verify and reuse its app-owned TCC
identity. A disposable SIP-disabled Lume worker without inherited grants must
use tests/runners/macos-lume/seed-tcc.sh after installing the exact candidate
as a certificate-signed CuaDriverLocal.app. The guarded helper is the only
supported automated path: it verifies VirtualMac*, disabled SIP, the expected
bundle identity, and the signed requirement before seeding only Accessibility
and Screen Recording. Never hand-edit TCC.db. Restart the app afterward and
prove live capture and input; database rows alone do not certify the desktop.
Linux
Runner: scripts/ci/linux/run-rust-e2e.sh
| Runner area | Rust test | Real harness or app |
|---|---|---|
| Shared app matrix | cross_platform_behavior_test.rs |
Electron and Tauri |
| Native controls | harness_gtk3_test.rs |
Repo-local GTK3 app |
| Capture contract | capture_contract_test.rs |
Linux capture backend |
| Desktop scope | desktop_scope_linux_test.rs |
X11/Wayland desktop scope |
Linux has separate X11 and Wayland concerns. Nix supplies the reproducible build and desktop environment, but the E2E test still needs an actual X11 or Wayland session. Linux does not need GIF output; MP4, screenshots, accessibility trees, trajectories, and logs are the useful evidence.
Wayland results are compositor-specific. The hosted lane uses Sway to prove
wlroots protocols. The niri lane (CUA_E2E_WAYLAND_SESSION=niri in
nix develop .#cua-driver-niri-e2e) runs niri's winit backend inside a
headless cage; its observer reads niri IPC, which reports no stacking order and
no tiled-window positions, and cannot tell a fullscreen tile from a maximized
one. It reports a floating harness window as occluded only behind the focused
sentinel while the sentinel itself reports being fullscreen, and one that
overlaps no other floating window as visible. It refuses tiled targets,
overlapping floating windows, an open overview, and any other output-sized
tile, including a maximized sentinel.
GNOME requires the optional WinRects Shell helper for
authoritative frame and buffer geometry, observation, capture, and verified
target activation. A portal/libei grant persists until the user revokes it, so
subsequent driver processes do not reopen the consent dialog.
KDE requires a future target-addressable KWin adapter; portal availability by
itself is not evidence that input can be sent safely to a named window.
Standard Wayland does not expose the physical pointer position, so canonical
Wayland rows do not claim the real-cursor preservation oracle. Focus, full
occlusion, sentinel input isolation, liveness, and fixture-state oracles remain
mandatory. Issue #2194 tracks
compositor, portal/libei, sentinel, and capture-based ways to add a proven
cursor observer where the environment supports one.
AX, PX, and Delivery
See action-support.md for the current Windows, macOS, and Linux
delivery, refusal, and unproven-action ledger.
These terms describe different dimensions:
| Term | Meaning |
|---|---|
| AX | Address a target through its accessibility/UI automation element |
| PX | Address a target by screen coordinates or pointer geometry |
| Foreground | The target may be brought to the foreground for delivery |
| Background | The target should receive the action without being raised or stealing focus |
| Window scope | Capture or action is limited to one target window |
| Desktop scope | Capture or action covers the full desktop |
The shared and native action matrices should test left click, right click,
double click, typing, keys, hotkeys, scroll, child windows, and drag across
AX/PX and foreground/background combinations where the driver supports them.
Unsupported background routes require an explicit refusal contract with an
allowed structured code and desktop-side-effect oracles. A desktop-scope
gesture that the platform's pointer route cannot carry uses the same contract
with the fixture under the gesture as its oracle; today that is only a modified
drag on native Wayland (modified_pointer_unavailable). A refusal fails a
cell that requires delivery. There should not be a separate "delivery" family
whose only purpose is to repeat those same actions in the background.
Native harness rows use the same typed case/result contract as the shared
matrix. Current native set_value rows declare background delivery because
their contract includes no-focus and no-raise observations; actions without a
delivery concept use not_applicable explicitly.
Cross-Cutting Invariants
The desktop observer is cross-cutting test instrumentation. It answers the same question for any action, harness, or catalog area:
Did the driver perform or reject the operation without disturbing the user's foreground application or desktop?
cua-driver-testkit::DesktopObserver owns the shared interface. Native Windows,
macOS, and Linux backends snapshot foreground-window, target z-order, cursor,
and focus state before and after an action. A separate full-desktop Electron
sentinel journals keyboard, pointer, wheel, visibility, focus, and heartbeat
events while it fully covers the target. Background rows opt into both pieces
of instrumentation directly; there is no special guard suite.
| Invariant or scenario | What it checks |
|---|---|
| Background click/type/key | The target action does not move focus away from the user's foreground window |
| Minimized app launch | launch_app(start_minimized=true) does not raise the new app |
| Background hotkey | A keyboard chord does not steal focus |
| Child-window click | A target-created window does not unexpectedly become foreground |
| Background screenshot | Reading the target does not change focus or z-order |
| Agent cursor visibility | The cursor appears in the captured pixels when enabled and moved |
The sentinel contract fails closed when the target is only partly covered or
the heartbeat stops. Before any behavioral cells run, the strict environment
preflight deliberately sends input to the sentinel and deliberately raises the
background target. The lane proceeds only if the leaked input and transient
focus loss are observed, the sentinel is restored, and it once again fully
occludes the target. Windows, macOS, and X11 require the sentinel's live focus
journal to report the loss. Wayland uses the compositor-backed native focus
observer because Electron/Ozone does not reliably emit a DOM blur event for
an external surface focus transition. The sentinel heartbeat and leaked-input
journal remain mandatory on Wayland. This positive control prevents a broken
guard from making every background row look green.
A focus assertion can prove "no focus steal" while failing to prove that a click changed the target application state. An action row must therefore check both the target's external state and, when background delivery is under test, the cross-cutting desktop observer.
These tests require a real interactive Windows user desktop. They reject
Session 0, locked desktops, and disconnected RDP sessions. Without
CUA_REQUIRE_GUI=1, an unusable desktop can self-skip for local development;
the canonical Windows runner enables the hard-failure behavior.
Evidence
Canonical GUI runs are expected to produce evidence per test cell:
artifacts/cua-driver/<os>/
|-- recordings/<cell-label>-pid<pid>-<sequence>/recording.mp4
|-- recordings/<cell-label>-pid<pid>-<sequence>/trajectory.json
|-- recordings/<cell-label>-pid<pid>-<sequence>/turn-*/before_state.json
|-- recordings/<cell-label>-pid<pid>-<sequence>/turn-*/before.png
|-- recordings/<cell-label>-pid<pid>-<sequence>/turn-*/after_state.json
|-- recordings/<cell-label>-pid<pid>-<sequence>/turn-*/after.png
|-- cases.jsonl
|-- environment.jsonl
|-- results.jsonl
|-- summary.md
`-- <rust-target>.log
The GitHub Actions summary contains one row per meaningful behavioral cell, including its OS, harness, action, AX/PX targeting, delivery mode, driver route, expected and observed behavior, oracles, and one evidence link. The link uses the exact video path as its label and opens the owning lane archive. Unit tests need normal test output and logs; they do not need desktop video.
What Is Implemented Today
- Rust owns the canonical scenario definitions and external-state assertions.
- Electron and Tauri use the same shared web fixture across supported OSs.
- Native Windows, macOS, and Linux harnesses are repo-local applications built from source.
- The three OS runners use a selector-free command for the complete matrix.
- Shared and native harness owners emit the same typed v2 result records.
- Canonical GUI rows collect a trajectory and MP4, validated before reporting.
- Canonical GUI rows require parseable pre/post state and non-empty pre/post
target-window images for each targeted turn. The one narrow exception is a
successful Windows
bring_to_frontrestore whose minimized target has no pre-action image or whose host capture remains unavailable afterward: it must retain the successful action response, captured post-action accessibility state, an explicit capture classification, trajectory, and MP4. Any other missing expected evidence fails the report. - Per-cell video starts after fixture readiness and foreground/background
posture. A 300 ms baseline precedes dispatch, and capture continues through
external oracle collection.
trajectory.jsonmust finish withbehavior_video.status = "finalized". - Use
agent_cursor_showcase_testfor cursor review media. Shared behavior matrix daemons deliberately use--no-overlayso synthetic cursor pixels do not contaminate action oracles; their videos prove tool behavior, not cursor rendering. - Windows hosted runs use
GetConsoleWindowto select the inherited HostedComputeAgent/runner console, verify its identity, and minimize it throughShowWindow(SW_MINIMIZE)before fixture or sentinel posture is established. The sentinel remains a separate test fixture and is reasserted after console cleanup. - Strict lane preflights fail on missing fixtures, desktop access, permissions, accessibility, capture, recording support, or ineffective background guards instead of silently skipping.
- Canonical runners set
CUA_E2E_FORBID_SKIPS=1. Unfiltered shared runs also setCUA_E2E_EXPECTED_MIN_CELLSto 80 on Windows/Linux and 120 on macOS, so a filtered, shortened, or accidentally emptied catalog cannot report green. Explicit diagnostic cell or harness filters disable only the minimum-count check; matching no cells still fails inside the Rust matrix. - GitHub summaries link every evidence-bearing row to its lane archive and display the exact recording path. The trajectory path remains in the typed evidence and archive.
- Unit/protocol tests remain separate from interactive E2E tests.
What Still Needs Implementation
The remaining work is platform coverage and validation, not another test hierarchy:
- Broaden native action rows. The shared web matrix covers every declared
AX/PX and foreground/background cell. AppKit, SwiftUI, WPF, WinUI3,
WebView2, and non-GTK3 Linux toolkits still have unproven native combinations listed in
action-support.md. - Preserve exact-source validation. Every accepted platform run must record one immutable source SHA and retain the typed evidence contract.
- Close representative-desktop gaps. Hosted Sway passes the complete
Electron, Tauri, GTK3, capture, and desktop-scope catalogs. A real GNOME 46
session passes GTK3, capture, and desktop scope, but still needs the shared
renderer catalog and portal-video parity. Plasma 6 still needs a verified
KWin activation adapter and its first accepted behavioral lane. Issue
#1922tracks the grouped backend work. - Add representative toolkit surfaces. GTK4, Qt5/Qt6, VTE, VCL, and GL canvases remain optional real-app gaps; shared Electron/Tauri coverage does not substitute for those native stacks.
- Flake cleanup. Replace remaining fixed native waits with external-state polling and add fixture reset tokens before reusing a harness process.
File Convergence Plan
The goal is not to put every assertion into one enormous test file. The goal is to give each behavior one clear owner and make cross-cutting evidence reusable.
Target Ownership
rust/crates/cua-driver-testkit/src/
`-- observer.rs Cross-OS desktop-side-effect interface
rust/crates/cua-driver-e2e/tests/
|-- cross_platform_behavior_test.rs Shared Electron/Tauri action matrix
|-- harness_wpf_test.rs Windows WPF action rows
|-- harness_winui3_test.rs Windows WinUI3 action rows
|-- harness_web_test.rs WebView2/Electron page and CDP rows
|-- harness_appkit_test.rs macOS AppKit action rows
|-- harness_swiftui_test.rs macOS SwiftUI action rows
|-- harness_gtk3_test.rs Linux GTK3 action rows
|-- capture_contract_test.rs Tree and screenshot read contract
`-- desktop_scope_<os>_test.rs Window/desktop scope invariants
rust/crates/cua-driver/tests/
`-- protocol_*_test.rs Protocol and schema tests
The desktop observer is a helper, not a test family. An action row invokes it when the row is testing background delivery. The row then records both outcomes:
- Did the target application state change, or did the driver return the documented structured refusal?
- Did focus, z-order, cursor, and desktop state remain within the contract?
Focus, z-order, cursor, and input-leak assertions belong to the typed action rows that exercise them; launch, capture, cursor, and desktop-scope contracts retain their narrowly owned scenarios. The canonical runner is the only user-facing command; lane selectors are internal diagnostics.
Contributor Workflow
When adding a new scenario:
- Add or update the repo-local fixture and its external state marker.
- Add the Rust scenario under
rust/crates/cua-driver-e2e/tests/. - Declare AX/PX addressing, foreground/background delivery, scope, and oracle.
- Add the scenario to
docs/test-matrix.mdand this guide when it changes the cross-OS structure. - Update only the OS runner selection when the test is platform-specific.
Every
#[ignore]test must be selected by a runner or workflow, or listed with a reason intests/manual-e2e-allowlist.txt;.github/scripts/tests/test_cua_driver_e2e_inventory.pyenforces this. - Run the smallest Rust test locally, then run the OS command before calling the matrix complete.
The goal is one understandable Rust E2E model across platforms, with platform-specific harnesses where the OS genuinely differs.
Test Layout Conventions
Rust tests in the driver workspace use one layout:
| Test kind | Where it lives |
|---|---|
| Small unit suite | Inline #[cfg(test)] mod tests { ... } at the end of the module |
| Large unit suite | src/<module>/tests.rs, declared as #[cfg(test)] mod tests; in src/<module>.rs |
| Several suites for one large module | src/<module>/<name>_tests.rs, one #[cfg(test)] mod <name>_tests; each, for example platform-{linux,windows}/src/tools/impl_/ |
| Crate-root suites | src/tests/, for example cua-driver-sdk/src/tests/ |
Integration test in cua-driver or its E2E suites |
tests/<subject>_test.rs; shared helpers go in tests/support/ |
Moving a suite into its own file keeps its module path, so test names and
CI filters stay the same. Prefer Rust's default module file lookup over
#[path]. Other crates' tests/ directories keep their existing names.
Scripts and their tests live next to their owner:
- Desktop runners and focused native gates live in
scripts/ci/<os>/, for examplescripts/ci/linux/run-mpx-recovery-e2e.py. - Example code keeps its tests in a
tests/directory beside it, for examplelibs/cua-driver/examples/agent-sdks/tests/. libs/cua-driver/rust/test-apps/is the staging directory for built harness apps, not a test directory. The fixture build scriptscdinto it, so its tracked README and.gitignoremust stay.
Perception tests have four owners, and each tests different code:
| Directory | Owns | Run by |
|---|---|---|
libs/cua-driver/rust/crates/cua-perception/{src,tests} |
Rust perception engine and extension protocol | ci-cua-perception-release.yml, ci-cua-driver-quick.yml |
libs/cua-driver/rust/crates/cua-perception/scripts/tests/ |
Model artifact tooling and quality measurement scripts | ci-cua-perception-release.yml |
libs/cua-driver/tests/perception-demo/ |
Visual demo evidence sanitizing, caching, and envelopes | ci-test-scripts.yml |
.github/scripts/tests/test_cua_perception_*.py, test_perception_release.py |
Perception release, review-trigger, and review-pipeline workflows and .github/scripts helpers |
ci-test-scripts.yml, ci-cua-perception-release.yml |
libs/cua-driver/rust/crates/cua-driver-e2e/tests/perception_capture_loop_test.rs |
Driver's desktop capture, parse, capture-bound click, and reobserve loop, with a deterministic developer-only worker and with the published extension installed from its signed release catalog | Capture lane of the canonical macOS, Windows, and Linux X11 desktop E2E |
libs/cua-driver/rust/crates/cua-driver-e2e/tests/perception_s1_decision_loop_test.rs |
The same loop with the published extension (OmniParser) and a resident Cua-S1-4B choosing the region, plus consumed and expired capture refusals | s1-perception lane of the Linux X11 runner, selected only by ci-cua-s1-weights.yml |
Add a perception test to the directory that owns the code it exercises.