Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
CUA Driver Test Matrix
This is the source map for the Rust test suites in libs/cua-driver. It uses
two top-level test classes:
- Unit and deterministic protocol tests. These do not depend on a repo-local application or an interactive desktop.
- Harness E2E tests. These build and launch a repo-local application, drive it through the Rust driver, and verify an external application or desktop oracle.
A test is E2E because it crosses the driver, OS, window system, and application boundary. The Rust harness catalog and its external oracles are the source of truth.
Matrix Dimensions
Every harness E2E result should identify these dimensions:
| Dimension | Values |
|---|---|
| OS | windows, macos, linux |
| Window system | Win32/UIA, AppKit/AX, X11/AT-SPI, Wayland/AT-SPI, WebView/CDP |
| Harness | Electron, Tauri, WPF, WinUI3, WebView2, AppKit, SwiftUI, WKWebView, GTK3 |
| Action targeting | ax, px, page, not_applicable |
| Delivery | background, foreground, N/A |
| Scope | window, desktop, N/A |
| Oracle | App state, accessibility state, focus state, pixel state, protocol state |
| Test status | pass, fail, skip, environment_error |
| Observed behavior | delivered, refused, no_effect, error, not_run |
Focus preservation is a cross-cutting oracle, not a separate matrix family. When a background action is tested, the row should attach the platform focus observer in addition to checking the target application's external state.
AX and PX describe how an action addresses a target. They are not capture
modes. get_window_state returns the tree and screenshot together; the action
uses an element index or coordinates.
Unit And Deterministic Tests
These run without the repo-local GUI applications:
| Area | Location | Coverage |
|---|---|---|
| Core driver logic | rust/crates/*/src/** |
Protocol values, sessions, schemas, image helpers, input helpers, configuration, telemetry, CLI behavior |
| MCP and CLI boundary | rust/crates/cua-driver/tests/protocol_* |
Handshake, tool registration, tool calls, media, sessions, and errors |
| Lifecycle sessions and target migration | Core session, session_tools, action_target, and compatibility capture_scope tests plus session_capture_scope_test.rs |
One implicit session per transport, five-minute idle policy, in-flight and exact-once cleanup, owner-scoped inspection and revival, per-call target validation, and legacy capture-scope compatibility |
| Tool-contract gate | schema_consistency_test.rs |
Shared tool schema parity and reviewed risk metadata across OS backends |
| Configuration transport | transport_config_persistence_test.rs |
CLI configuration persistence across daemon-backed processes |
| Token and protocol surfaces | protocol_element_token_test.rs, related tests |
JSON-RPC-visible contract behavior |
| Permission modes and policy startup | permission_policy_startup_test.rs, daemon_required_test.rs, core authorization, policy, and session_manifest tests |
Fail-before-bind policy loading, managed/user intersection, immutable standard/autonomous/unrestricted startup, danger acknowledgement, admin disable, deny-by-default manifests, and canonical daemon dispatch |
| Protected browser grants | Core consent, browser::grant, browser::engine, and browser::v2_tests |
Exact request digests, provider authentication seam and deadline, persistent-indicator activation, Stop/session teardown, forged legacy artifact refusal, exact PID/window manifest scope, and the live-origin decision path used before mutation |
Some protocol tests spawn the driver process. They remain deterministic because they do not launch a real target application or require a desktop. They should be reported with the unit gate, separately from Harness E2E.
The ordinary unit gate must not run #[ignore] GUI tests and must not turn
missing desktop fixtures into silent skips.
Harness E2E: Shared Web Applications
Electron and Tauri are separate repo-local applications that load the shared web harness. The fixture exposes action-specific controls and external state markers rather than an application task that duplicates those same actions.
Source:
tests/fixtures/shared/web/index.htmltests/fixtures/apps/cross-platform/electron/tests/fixtures/apps/cross-platform/tauri/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs
The shared harness exposes deterministic external markers for these actions:
| Action family | Actions | Addressing | Delivery |
|---|---|---|---|
| Pointer | Left click, right click, double click | AX and PX | Background and foreground |
| Keyboard and text | Type text, Return, hotkey, type then Return | AX and PX where supported | Background and foreground |
| Scroll | Scroll | AX and PX where supported | Background and foreground |
| Child windows | Open child window | AX and PX | Background and foreground |
| Drag | Drag source to drop target | PX | Background and foreground |
| State controls | Checkbox, radio, combo, slider | AX or PX by control | Mode declared per action |
| Editor | Type, save, saved-state readback | AX and PX where supported | Mode declared per action |
The Rust shared catalog declares 40 evidence-bearing cells per harness application. Windows and Linux run 80 shared cells across Electron and Tauri; macOS also runs the native WKWebView host for 120 shared cells. Each host covers the full AX/PX and foreground/background cross-product for click, text, keyboard, sequential type-then-Return, scroll, and child-window actions, plus both delivery modes for PX drag and AX editor-save. A background capability refusal is a valid result only when the test verifies the declared structured refusal and the no-focus/no-z-order/no-input-leak side effect. Background posture requires one foreground window to contain the target geometry, not merely overlap it. The lane preflight also injects deliberate focus and input violations and requires the sentinel to detect both before its results are trusted. A refusal fails a cell whose contract requires delivery. Canonical reporting rejects skips and enforces the complete shared catalog size for unfiltered runs. Diagnostic filters are allowed, but matching zero cells is always an error.
Harness E2E: Optional Perception Loop
perception_capture_loop_test.rs runs the documented capture-bound loop in
the capture lane of the macOS, Windows, and Linux X11 runners:
| Cell | Coverage |
|---|---|
<os>-electron-parse-visual-regions-not-installed-not-applicable-not-applicable |
A retained window capture parses to the typed not_installed refusal when no extension is installed |
<os>-electron-perception-capture-click-px-foreground |
get_window_state capture, parse_visual_regions on that capture, one foreground click with the same capture_id at the parsed region center, fixture-journal delivery oracle, capture_not_found refusal for the consumed capture, then a fresh capture and reparse |
<os>-tk-perception-published-capture-click-px-foreground |
The published cua-perception extension, installed from its signed release catalog as publisher-verified, parses a capture of the visual-only Tk canvas. One foreground capture-bound click on the OCR region that reads "Cancel" selects that card in the fixture journal. The consumed capture is refused with capture_not_found, a fresh capture reparses, and an unused capture older than 60 seconds is refused with capture_expired. Neither refusal changes the fixture |
The Electron rows use a developer-only unsigned worker compiled from
tests/support/perception_swatch_worker.rs. It speaks the real worker
protocol inside Driver's containment, echoes the retained PNG digest, and finds
the harness's solid #drag-source and #drop-target swatches in the captured
pixels, so those rows need no published artifact or model.
The published-catalog row pins the cua-perception-v0.2.1 catalog and archive
SHA-256 for each target and downloads them (about 425 MB) unless
CUA_E2E_PERCEPTION_CACHE_DIR already holds verified copies. The Driver also
checks the catalog signature. The canvas fixture needs a Python with Tk:
python3 on macOS and Linux, and py -3 or python on Windows.
CUA_E2E_TK_PYTHON overrides the search. The hosted macOS capture lane adds
the matching Homebrew python-tk formula, and the Linux capture lane installs
python3-tk and x11-utils. The row fails rather than skips when Tk is
missing.
macOS runs a second instance of the installed app with an isolated extension
home and socket, so the shared daemon never gains either extension. Wayland
lanes record a coverage limitation instead of running these rows. Broader
model quality belongs to the cua-perception crate and its artifact-gated
lanes.
Harness E2E: Model-Backed Perception Decision Loop
perception_s1_decision_loop_test.rs runs the same capture-bound loop with
real models, in the s1-perception lane of the Linux X11 runner
(scripts/ci/linux/run-rust-e2e.sh). Only
.github/workflows/ci-cua-s1-weights.yml selects that lane (nightly, manual
dispatch, and pull requests that touch Cua-S1, the jev-use chooser, or the
row), because it needs the pinned Cua-S1-4B weights:
| Cell | Coverage |
|---|---|
linux-electron-perception-s1-capture-click-px-foreground |
The published cua-perception release, installed from its signed catalog as publisher-verified, parses a retained Electron window capture with OmniParser; a resident Cua-S1-4B (text adapter, CPU) chooses one of the parsed text regions for the goal "Increment the counter once."; Driver clicks the region center once with the same capture_id within the 60-second capture lifetime; the fixture journal must reach counter=1; the consumed capture is refused with capture_not_found, an unused capture past its lifetime is refused with capture_expired, and the counter stays at 1 |
The model runs on CPU in bfloat16, which fits the capture lifetime only on
hosted runners that expose AVX512-BF16. With 16 offered regions, decisions
took 20 to 38 s and the click landed 23 to 42 s after the capture, so the row
offers at most 12. The same
ubuntu-latest label sometimes lands on a VM without AVX-512, where one text
decision takes over 50 s. On such a runner the job skips the row, writes a
cua-e2e-limitation-v1 record (perception-s1-decision-loop-limitation.json),
and warns; re-run the job to draw another runner.
The row writes the parse, the cua.jev_choice_request_v1 request, the S1
decision, and a timing record under artifacts/cua-driver/linux/s1-perception/
next to the trajectory video. macOS and Windows lanes do not run it: the
hosted macOS arm64 runner has 7 GB of memory, less than the 9.3 GB of 16-bit
base weights, and the Windows lane has no provisioned weights. Their
capture-bound loop is covered by the model-free row above.
Harness E2E: Native Windows
Windows native harnesses are repo-local applications built from source:
| Harness | Source test | Coverage |
|---|---|---|
| WPF | harness_wpf_test.rs |
UIA controls, text, keys, pointer actions, scroll, drag, popups, menus, modal windows; foreground single-line and multiline text integrity |
| WinUI3 | harness_winui3_test.rs |
XAML controls, text, checkbox/radio, slider, combo, popup; foreground single-line and multiline text integrity |
| WebView2 | harness_web_test.rs |
Window discovery, CDP page access, JavaScript, DOM click path |
| Desktop target | desktop_scope_windows_test.rs |
Per-call window/desktop modality, full-display capture, screen-absolute click/scroll, and strict rejection |
| Desktop invariants | Testkit DesktopObserver plus typed launch/capture/cursor owners |
Cross-cutting focus, z-order, minimized-launch, screenshot, cursor, and desktop checks |
Native controls use AX/UIA state as their oracle. The foreground typing rows
also compare each fixture's real TextBox.Text values from an isolated,
atomically replaced state file, including the target and leakage sentinels.
The single-line row covers CJK, digits, and emoji; the multiline sample covers
LF, CR, CRLF, repeated empty lines, and leading/trailing breaks. The oracle
normalizes line endings only and preserves all other characters and blank
lines. Foreground delivery keeps the target in front until its thread has
read every inserted event (a sentinel-key drain), so text after line breaks is
not handed to the previously active window. Pointer actions also use PX where
the tool contract requires coordinates. Current set_value rows declare
background delivery and attach desktop-side-effect oracles.
Harness E2E: macOS
| Harness | Source test | Coverage |
|---|---|---|
| AppKit | harness_appkit_test.rs |
AX tree/capture, AX value/text, AX scroll, PX clicks, and slider drag in both background and foreground |
| SwiftUI | harness_swiftui_test.rs |
AX tree/capture, background click/value, and foreground popover-trigger state |
| WKWebView | cross_platform_behavior_test.rs |
Dedicated native host running the full 40-cell shared web catalog |
| Desktop target | desktop_scope_macos_test.rs |
Per-call window/desktop modality, screen-absolute action delivery, and strict rejection |
| Installed-app launch/focus | installed_app_launch_macos_test.rs |
Real Calculator/TextEdit launch and focus behavior in the canonical logged-in lane |
| Installed-app text | installed_app_textedit_macos_test.rs |
Real TextEdit AX background write and verification in the canonical logged-in lane |
The AppKit snapshot_publication regression holds a fixture PNG write with FIFO
backpressure while replacing a button at the same AX index. It exercises the
built-in get_window_state and click tools and uses an independent fixture
activation journal to reject wrong-target activation and verify fresh-token
recovery. The macOS runner includes this regression.
macOS uses the installed ScreenCaptureKit/AX permissions for GUI runs. Its
maintainer acceptance gate runs in a disposable clone of the stopped Lume
SIP-off golden image described in the macOS Lume runner
guide. Maintainers build that private
seed from the sanitized public base macos-tahoe-cua:26.5.2. The repo-local
harnesses are canonical; Calculator and TextEdit are supporting real-app
checks. SwiftUI's popover trigger is proven independently from the remaining
transient-panel AX discovery gap.
Harness E2E: Linux
| Harness | Source test | Window system | Coverage |
|---|---|---|---|
| Electron | cross_platform_behavior_test.rs |
X11, hosted Sway, and nested niri | Shared web action matrix |
| Tauri | cross_platform_behavior_test.rs |
X11, hosted Sway, and nested niri | Shared web action matrix |
| GTK3 | harness_gtk3_test.rs |
X11/AT-SPI and Wayland/AT-SPI where configured | Native GTK controls and input |
| Desktop target | desktop_scope_linux_test.rs |
X11/Wayland | Per-call window/desktop modality, screen-absolute action delivery, and strict rejection |
Nix provides the Linux build and desktop environment. X11 and Wayland are separate matrix dimensions because their capture and input contracts differ. Linux runs do not produce GIF output. Every canonical GUI cell retains an MP4 and trajectory alongside screenshots, AX trees, structured results, and driver logs for E2E evidence.
Action Delivery Matrix
The current per-OS delivery/refusal ledger is maintained in
action-support.md. This section defines the coverage
policy; the ledger records empirical status.
For each OS and harness where the action is supported, the canonical E2E suite should cover both delivery modes:
| Action | Background | Foreground | Addressing |
|---|---|---|---|
| Left click | Required delivery or a declared refusal contract | Required | AX, PX |
| Right click | Required delivery or a declared refusal contract | Required | AX, PX |
| Double click | Required delivery or a declared refusal contract | Required | AX, PX |
| Drag | Required delivery or a declared refusal contract | Required | PX, with AX discovery |
| Scroll | Required delivery or a declared refusal contract | Required | AX, PX where supported |
| Type text | Required delivery or a declared refusal contract | Required | AX, PX where supported |
| Press key | Required delivery or a declared refusal contract | Required | AX, PX where supported |
| Hotkey | Required delivery or a declared refusal contract | Required | AX, PX where supported |
| Set value | Required in current native rows | Not separately declared | AX/UIA/AXValue |
| Screenshot | N/A | N/A | Window or desktop capture |
| Page/CDP | N/A | N/A | Page selector/JavaScript |
N/A means the API has no delivery mode for that operation. A refusal is a
passing observation only when the Rust case expects refusal, the exact code is
allowed, and the desktop-side-effect oracles pass.
CI And Local Entry Points
| Gate | Environment | Entry point |
|---|---|---|
| Linux unit/source | Nix Linux CI | nix build .#checks.x86_64-linux.cua-driver-build .#checks.x86_64-linux.cua-driver-linux-rust-unit |
| Windows unit/compile | windows-latest |
Package-scoped cargo test --all-targets --no-run --locked |
| Windows Harness E2E | Active Windows user session | scripts/ci/windows/run-rust-e2e.ps1 -RequireGui |
| Linux X11 Harness E2E | Nix X11 session | scripts/ci/linux/run-rust-e2e.sh |
| Linux Sway Harness E2E | Controlled wlroots session | scripts/ci/linux/run-rust-e2e-wayland.sh |
| Linux niri Harness E2E | Nested niri in a headless cage | CUA_E2E_WAYLAND_SESSION=niri scripts/ci/linux/run-rust-e2e-wayland.sh |
| Linux nested-compositor E2E | Controlled experimental session | scripts/ci/linux/run-rust-e2e-inject.sh |
| Linux representative desktop E2E | Existing GNOME, KDE, or Xorg login | scripts/ci/linux/run-rust-e2e-desktop.sh <desktop> |
| macOS Harness E2E | Maintainer Lume SIP-off worker with a logged-in session and inherited grants, or a disposable worker prepared with the checked-in macOS consent seed helper | libs/cua-driver/tests/runners/macos-lume/run-all.sh |
Workflows select private execution lanes. Rust source owns scenario definitions, fixture oracles, and result records. OS runners only build the driver, stage the local harnesses, establish the desktop session, collect evidence, and publish the shared report.
Evidence And Ownership
Every Harness E2E cell should produce:
recordings/<cell-label>-pid<pid>-<sequence>/recording.mp4
recordings/<cell-label>-pid<pid>-<sequence>/trajectory.json
results.jsonl
<rust-target>.log
The GitHub summary links the exact recording path from the corresponding row to its lane archive. Target logs remain lane-level diagnostics. Unit tests need logs and test-result output, but do not need desktop video.
When a new action or modality is added, update this document, the shared fixture oracle, the Rust test, and the OS-specific runner selection together. This is the cross-OS checklist that prevents a Windows-only test from being mistaken for cross-platform coverage.