Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

ci(cua-driver): run the verification proof in Linux CI (#5832)

injaneitye754cfec69c4
diff --git a/.cursor/skills/verify-cua-driver/SKILL.md b/.cursor/skills/verify-cua-driver/SKILL.md
new file mode 100644
index 0000000000..ab31b28509
--- /dev/null
+++ b/.cursor/skills/verify-cua-driver/SKILL.md
@@ -0,0 +1,108 @@
+---
+name: verify-cua-driver
+description: Verify the Cua Driver daemon and CLI/MCP desktop tools against real fixture apps after driver changes, using Linux diagnostics, AX-first actions, side-effect checks, and retained evidence.
+---
+
+# Verify Cua Driver
+
+Read [the feature map](features/README.md) first. This verifies the daemon's desktop-tool surface, not the bench, and does not certify every mapped entry point by exercising one convenient path. Use only a disposable desktop: the foreground cells intentionally move focus and input.
+
+## Launch
+
+Run from the repository root. Use the CI toolchain (Rust 1.97.1, Node 22) and install the Linux dependencies from `.github/workflows/e2e-rust-linux.yml`:
+
+```bash
+sudo apt-get update
+sudo apt-get install -y --no-install-recommends \
+ xvfb dbus-x11 at-spi2-core openbox picom python3-gi gir1.2-gtk-3.0 \
+ libgtk-3-dev clang pkg-config libdbus-1-dev libpipewire-0.3-dev \
+ libspa-0.2-dev libei-dev libx11-dev libxi-dev libxtst-dev libxext-dev \
+ libwebkit2gtk-4.1-dev libssl-dev libxdo-dev \
+ libayatana-appindicator3-dev librsvg2-dev ffmpeg jq x11-utils
+
+CUA_E2E_RETRY_ONCE=0 .cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11
+```
+
+The helper prints an absolute `Evidence:` directory under `artifacts/cua-driver/verify/run-XXXXXXXX`. Each invocation creates a new directory; never reuse an older canonical artifact directory. The canonical runner builds the release driver once and stages only Electron using `tests/fixtures/build/linux.sh --only electron`. Reuse that build with:
+
+```bash
+timeout 1260 .cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11 --no-build
+```
+
+The X11 recipe follows CI: `xvfb-run -a --server-args='-screen 0 1920x1080x24' dbus-run-session`, openbox, then picom with the xrender backend. `at-spi2-core` is D-Bus activated, not bootstrapped by another accessibility launcher. Bounded probes wait for the WM and `org.a11y.Bus.GetAddress`. Testkit creates private daemon sockets and empty isolated HOME/XDG state, then owns the daemon, MCP proxy, fixture, and sentinel processes. Do not attach to your installed daemon.
+
+The runner first executes `canonical_e2e_environment_is_ready`, then only `shared_web_action_matrix_is_state_verified`. The default substring filter runs **two** cells: `linux-electron-left-click-ax-background` and its `-typed-sdk` sibling. Both must deliver, not skip or refuse. `READY … daemon usable … AX and journal marker present` means a successful daemon tool path, a discovered real window, and `WEB_HARNESS_MARKER_v1` in both AX and the fixture's read-only event journal. Socket existence alone is not ready. The helper completes the action and tears down automatically; it is not an interactive server launcher.
+
+Nested Wayland uses the existing CI wrapper, with its default runner and the same focused lane:
+
+```bash
+timeout 1260 .cursor/skills/verify-cua-driver/scripts/verify-linux.sh wayland --no-build
+```
+
+Install the compositor dependencies from `.github/workflows/e2e-rust-linux-wayland.yml` before this lane. `scripts/ci/linux/run-rust-e2e-wayland.sh` provisions its runtime, Sway and AT-SPI; do not duplicate its bootstrap. Compositor and AT-SPI logs honor the unique artifact directory. Wayland is unverified here; X11 results do not certify its capture or input routes.
+
+## Doctor
+
+Read-only diagnostics run inside the same session before driving:
+
+- `gdbus call --session --dest org.a11y.Bus --object-path /org/a11y/bus --method org.a11y.Bus.GetAddress` probes the accessibility bus; `at-spi-address.txt` retains its address.
+- `libs/cua-driver/rust/target/release/cua-driver doctor --json` writes `doctor.json`; no `--fix`. Inspect warnings even if the exit status is zero. Linux `health_report` checks the display and AT-SPI availability.
+- `libs/cua-driver/rust/target/release/cua-driver --version` writes `version.txt`.
+- Preflight calls `get_config`, `health_report`, and `check_permissions` on its owned daemon. Its recording directory retains `health_report.json`, `check_permissions.json`, and `daemon.json` (PID, socket, owner UID, mode, version/build config). Preflight asserts source version and optional requested SHA, Unix socket type, same-user ownership and no other-user access. It also proves fixture AX, capture and video readiness.
+
+A permissions report is not authorization coverage: canonical testkit explicitly enables unrestricted GUI operations and bypasses interactive approvals. Health warnings must be interpreted alongside preflight, not silently called healthy.
+
+## Drive
+
+Reuse `libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs`. It launches the real Electron fixture from `libs/cua-driver/tests/fixtures/apps/cross-platform/electron`, discovers its new PID/window, and obtains fresh snapshot-bound handles. The actual MCP tool sequence is:
+
+```rust
+let windows = fixture.driver.call("list_windows", serde_json::json!({}));
+let pre = snapshot(fixture);
+let args = action_target_args(fixture, &pre, "border-click-target", "ax", "background");
+let response = fixture.driver.call("click", args);
+```
+
+`action_target_args` sends `pid`, `window_id`, `delivery_mode: "background"`, and the discovered `element_token` from `require_element`. Do not paste numeric handles or reuse a token after another snapshot. The fixture journal must independently report `last_action=left_click`; an accepted response alone is insufficient. The typed SDK sibling additionally exercises stale/mismatched handle refusals before a valid click.
+
+Use the commands in the feature map to select other user journeys. Prefer AX handles; use existing `action_target_args` pixel conversion only for pixel coverage, not guessed screen coordinates. Do not use DOM setters, fixture control endpoints, or direct journal writes to make target state pass. The fixture journal observes effects; the real driver performs the action.
+
+## Evidence
+
+Keep the printed unique directory. It contains `launch.log`, doctor/version output, preflight logs, `results.jsonl`, `cases.jsonl`, `environment.jsonl`, and `summary.md`. Each existing per-cell recording directory contains:
+
+- `tool-calls.jsonl`: actual MCP requests and full response envelopes, including transport errors/timeouts. Image/base64 payloads are replaced by encoded byte length and a hash; PNGs remain separate. Typed SDK traffic is not MCP transcript traffic.
+- `trajectory.json`, `recording.mp4`, and action `turn-*` directories with `action.json`, `before.png`, `after.png` from the product recording mechanism.
+- `fixture-state.json`: the independent fixture journal after the action, also saved during unwinding.
+- `sentinel-events.jsonl`: foreground heartbeat, focus and input journal.
+- `desktop-observation.txt`: native before/after focus, z-order, pointer, transient focus journal, unsupported oracles and violations. Result errors retain these diagnostics too.
+
+The runner uses `cua-e2e-report --require-video` and ffprobe validation. A passing summary must include delivered click rows and playable owned videos. Inspect action, response, before/after screenshots and fixture side effect together. For background input, require a live heartbeat, no leaked input/blur, stable native focus and pointer, no target raise, and no unsupported oracle. A refusal is a valid refusal test, **not** proof of successful delivery. Failed setup must remain a failure, not be replaced by a screenshot from another run.
+
+## Cleanup
+
+Testkit's child reaper owns daemon/fixture groups; the sentinel owns its own group. Scope exit reaps them. The X11 wrapper traps exit and kills/waits only the openbox and picom PIDs it captured with `$!`; xvfb-run and dbus-run-session own their sessions. The Wayland wrapper owns its compositor and bus PIDs. Never use process-name matching or kill another desktop.
+
+After session teardown, the helper invokes `scripts/check-evidence.sh` on the printed directory, even if the desktop command failed. The checker requires complete current-attempt evidence and checks `kill -0` against recorded daemon PIDs and discovered `CuaTestHarness` window PIDs, without killing them. Malformed extraction, zero recorded PIDs or any surviving PID fails verification, not permission to kill by name. Do not delete the evidence directory. A timeout or crash may leave incomplete evidence; inspect the launch log and recorded PIDs before retrying.
+
+## Helpers
+
+The shipped executable is:
+
+```bash
+.cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11
+.cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11 --no-build
+.cursor/skills/verify-cua-driver/scripts/verify-linux.sh wayland --no-build
+```
+
+It reuses `scripts/ci/linux/run-rust-e2e.sh` through `CUA_E2E_INTERNAL_LANE=verify` and `CUA_E2E_ARTIFACT_DIR`; the runner is responsible for strict fixtures, preflight, recordings and report validation. Each desktop run is bounded to 1200 seconds internally. No bench code or second harness is involved.
+
+## macOS and Windows: unverified
+
+macOS needs the installed TCC-authorized daemon (`libs/cua-driver/scripts/install-local.sh`, `cua-driver permissions grant`) and Accessibility/Screen Recording consent; its tests use the existing daemon proxy rather than Linux Xvfb. Windows needs an interactive desktop, native UIA and named-pipe transport; testkit uses Job Objects for ownership. Use their existing CI recipes and recording paths. Neither platform was run here; Linux's unrestricted testkit mode does not prove their production permission paths.
+
+## Execution status and maintenance
+
+Linux X11 CI proof is pending in `.github/workflows/e2e-rust-linux.yml`, job `verify-skill`. It runs on relevant skill/driver pull requests or `workflow_dispatch` with `lane=verify`, with retries disabled and a 25-minute job bound. The helper owns Electron staging and desktop setup; `check-evidence.sh` gates success after cleanup. The `verify-cua-driver-evidence` artifact retains the printed evidence directory even on failure. No successful CI run is claimed. See [execution instructions, historical attempts and remaining stand-ins](execution.md), including the same command for a networked local host. Claim proof only after both click cells deliver and all required evidence survives cleanup; setup logs alone do not prove delivery or owned-process teardown.
+
+Use `/maintain-verification-skill` when tools, fixture markers, CI lanes or supported input routes change.
diff --git a/.cursor/skills/verify-cua-driver/execution.md b/.cursor/skills/verify-cua-driver/execution.md
new file mode 100644
index 0000000000..f6ee108b16
--- /dev/null
+++ b/.cursor/skills/verify-cua-driver/execution.md
@@ -0,0 +1,39 @@
+# Linux execution outcome
+
+## CI proof pending
+
+The end-to-end proof runs in `.github/workflows/e2e-rust-linux.yml`, job `verify-skill`, on pull requests touching `.cursor/skills/verify-cua-driver/**`, `libs/cua-driver/**`, or that workflow. For a manual run, use `workflow_dispatch` with `lane=verify` and optionally a full source SHA in `ref`. Release-call certification lanes are unchanged.
+
+The job has a 25-minute bound, shares the Rust build cache, disables retries, and runs the offline checker regressions before invoking the helper. The canonical runner stages Electron from the registry; no runtime is vendored. Evidence is uploaded after cleanup as `verify-cua-driver-evidence`, including failed attempts. Missing artifacts or incomplete evidence fail the job.
+
+On a networked Linux host with the dependencies and toolchain listed in [the skill](SKILL.md), run from the repository root:
+
+```bash
+CUA_E2E_RETRY_ONCE=0 .cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11
+```
+
+The helper automatically checks its printed evidence directory after desktop teardown. To recheck retained evidence independently:
+
+```bash
+.cursor/skills/verify-cua-driver/scripts/check-evidence.sh /absolute/printed/evidence/directory
+```
+
+The checker requires both exact delivered/pass summary rows, current-attempt per-cell evidence, preflight diagnostics, and at least one recorded owned PID with none still alive. A helper timeout in CI is a defect to fix under requirement (3), for example by raising its internal bound if build/staging exhausts it; do not add a second pre-build harness.
+
+No successful CI run has been observed yet. Record the run URL and source SHA here only after that job succeeds. No end-to-end attempt was made in the offline sandbox for this change.
+
+## Historical offline attempts
+
+The documented `timeout 1260 .cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11` invocation was retried. X11 session setup and the release driver build succeeded, but Electron dependency staging failed. A fresh-lockfile installation retry also failed; the npm registry was unreachable and no cached Electron runtime was available.
+
+Setup logs survived cleanup. Doctor, preflight and both click cells did not run: there are no delivered summary rows, daemon diagnostics, transcripts, fixture-state snapshots, sentinel/native observations, screenshots or videos. No surviving-owned-PID warning was printed, but there were no recorded daemon/fixture PIDs to check. The transcript PID extraction, live-process cleanup check and fixture snapshot-on-drop paths remain unverified on real output. This is not an end-to-end proof; the CI job above is the pending proof location.
+
+## Stand-ins
+
+- Xvfb instead of a physical desktop session.
+- The real Electron fixture instead of a user's application; it did not launch in these attempts.
+- Electron sandbox/GPU disabled, forced accessibility and a fixture CDP port.
+- Disposable driver HOME/XDG state, disabled overlay/permissions gate and testkit unrestricted GUI authorization/approval bypass.
+- Fixture control only establishes sentinel posture; it does not set the target action result.
+
+These do not certify production admission, interactive consent, physical GPU capture or arbitrary app compatibility. Wayland, macOS and Windows remain unverified.
diff --git a/.cursor/skills/verify-cua-driver/features/README.md b/.cursor/skills/verify-cua-driver/features/README.md
new file mode 100644
index 0000000000..d791e16ebd
--- /dev/null
+++ b/.cursor/skills/verify-cua-driver/features/README.md
@@ -0,0 +1,19 @@
+# Cua Driver verification map
+
+Read [Launch, Doctor, Evidence and Cleanup](../SKILL.md) before every journey. These recipes use a disposable desktop, source-built driver and real Electron fixture; do not drive a user's existing session. Each invocation creates its own daemon socket, app instances and evidence directory. The GUI session is isolated, but simultaneous Cargo builds share the normal Cargo lock.
+
+## Features
+
+- [Click](click.md): AX/pixel targeting, left/right/double click, SDK handle validation.
+- [Type text](type-text.md): text entry, submit, isolation from unrelated fields.
+- [Scroll](scroll.md): target-local wheel movement and state read-back.
+- [Window discovery](window-discovery.md): list windows, select identity and obtain AX state.
+- [Background versus foreground input](background-input.md): explicit delivery, no stolen input, typed refusals.
+
+## Proof contract
+
+Each feature lists sub-features and user entry points; selecting one cell does not certify the rest. MCP tools and the CLI `call` surface address the same daemon, but only the default click also exercises the typed SDK. CLI transport parity is not proven by an MCP result.
+
+Run commands from the repository root. The shipped helper defaults to Electron and background AX left click; `CUA_E2E_CELL_FILTER` is a substring filter. Every journey performs the same doctor/preflight and automatic cleanup. The initial invocation builds; later commands below use `--no-build` only after that build and fixture staging succeed.
+
+Capture tool requests/responses, before/after images and independent state. A delivered mutation must change the expected fixture marker, not an unrelated field. A typed refusal must leave fixture state unchanged. Background actions additionally require native desktop and sentinel evidence. Missing prerequisites, skipped cells, unsupported observations and unreachable entry points are failures or explicit coverage gaps, never success through a substitute path. Evidence survives cleanup at the absolute path printed by the helper.
diff --git a/.cursor/skills/verify-cua-driver/features/background-input.md b/.cursor/skills/verify-cua-driver/features/background-input.md
new file mode 100644
index 0000000000..3c0dd95b4a
--- /dev/null
+++ b/.cursor/skills/verify-cua-driver/features/background-input.md
@@ -0,0 +1,27 @@
+# Background versus foreground input
+
+Delivery mode makes the intended effect on the user's desktop explicit: background operations must not steal focus, move the real pointer, raise the target or leak input into the foreground app.
+
+## Sub-features
+
+- `background-delivered`: AX click reaches a fully occluded target.
+- `foreground-delivered`: explicit foreground click activates the target path.
+- `no-stolen-input`: native observer and sentinel independently detect desktop changes.
+- `typed-refusal`: unsupported delivery fails without mutating the fixture.
+
+## How to get to it (user POV)
+
+Set `delivery_mode` explicitly to `background` or `foreground` on the desktop input tool. Address the intended window/element, not whichever app happens to own keyboard focus. MCP, CLI `call` and the typed SDK expose input operations.
+
+## Driving it with MCP/testkit
+
+Preconditions: [Launch and Doctor](../SKILL.md) succeed, and the sentinel completely occludes the target before capture and dispatch.
+
+- **Background:** run `timeout 1260 .cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11 --no-build`. Require both AX click cells delivered with `last_action=left_click`, a live sentinel heartbeat, no input/blur events, stable native focus and pointer, unchanged background z-order, and supported oracles.
+- **Foreground:** run `CUA_E2E_CELL_FILTER=linux-electron-left-click-ax-foreground timeout 1260 .cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11 --no-build`. Require the target click marker. Foreground input is allowed to activate the target; do not apply a background guarantee to this row.
+- **Unavailable route:** run `CUA_E2E_CELL_FILTER=linux-electron-type-text-ax-background timeout 1260 .cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11 --no-build`. Require the typed refusal, unchanged fixture journal and stable desktop, not delivery.
+- **Proof:** retain sentinel JSONL and native observation alongside the call, images and fixture state. Missing/unsupported observation is failure, not evidence of no change. CLI delivery parity is not covered by the MCP transcript.
+
+## Gotchas
+
+Linux Electron permits background AX left click and child-window activation in this matrix; other background actions are expected to refuse. Sentinel setup may intentionally manipulate focus before the recording boundary; only the drained action boundary certifies no stolen input. Fixture journal state is independent of AX text, which can lag. X11 results do not prove nested Wayland or other platforms.
diff --git a/.cursor/skills/verify-cua-driver/features/click.md b/.cursor/skills/verify-cua-driver/features/click.md
new file mode 100644
index 0000000000..2fc0bfbaaf
--- /dev/null
+++ b/.cursor/skills/verify-cua-driver/features/click.md
@@ -0,0 +1,28 @@
+# Click
+
+Click activates the intended app control without changing an unrelated app or stealing input during background delivery.
+
+## Sub-features
+
+- `left-click-ax`: activate the border target using a fresh AX handle.
+- `click-px`: address the same target through screenshot-local pixels.
+- `right-click` and `double-click`: preserve their distinct action semantics.
+- `sdk-identity`: reject stale tokens and mismatched windows before valid delivery.
+
+## How to get to it (user POV)
+
+Call MCP `click`, `right_click` or `double_click`; use the CLI `cua-driver call` surface or typed SDK `click` for the same native-window operation. Select the app window first and explicitly choose `background` or `foreground` delivery.
+
+## Driving it with MCP/testkit
+
+Preconditions: [Launch and Doctor](../SKILL.md) succeed, with the Electron fixture staged.
+
+- **AX background and SDK:** run `timeout 1260 .cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11 --no-build`. Both left-click AX background cells must deliver. The MCP request uses the fresh `border-click-target` element token; `fixture-state.json` must contain `last_action=left_click` and observer evidence must show no stolen focus/pointer/input.
+- **Pixel foreground:** run `CUA_E2E_CELL_FILTER=linux-electron-left-click-px-foreground timeout 1260 .cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11 --no-build`. Require the same marker and screenshot-local targeting in the transcript.
+- **Right click:** run `CUA_E2E_CELL_FILTER=linux-electron-right-click-ax-foreground timeout 1260 .cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11 --no-build`. Require `last_action=right_click`.
+- **Double click:** run `CUA_E2E_CELL_FILTER=linux-electron-double-click-ax-foreground timeout 1260 .cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11 --no-build`. Require `last_action=double_click`.
+- **Proof:** inspect both turn screenshots, the tool response and fixture journal. The SDK sibling must reject stale/mismatched targets without fixture mutation, then deliver a valid click. CLI transport remains separately unverified by these commands.
+
+## Gotchas
+
+A new snapshot invalidates old element tokens. Do not substitute tree indices for tokens. Pixel coordinates are local to the captured window, accounting for origin and screenshot scale; the existing harness converts them. Linux background right/double/pixel click cells expect typed refusals, not successful delivery. The default substring filter intentionally selects two cells, including `-typed-sdk`.
diff --git a/.cursor/skills/verify-cua-driver/features/scroll.md b/.cursor/skills/verify-cua-driver/features/scroll.md
new file mode 100644
index 0000000000..ca95c57f8e
--- /dev/null
+++ b/.cursor/skills/verify-cua-driver/features/scroll.md
@@ -0,0 +1,27 @@
+# Scroll
+
+Scroll advances the intended viewport and verifies movement independently of the tool response.
+
+## Sub-features
+
+- `scroll-ax`: address a scroll container by its fresh AX token.
+- `scroll-px`: address it by screenshot-local position.
+- `scroll-distance`: two requested pages advance at least one viewport.
+- `background-refusal`: unavailable background delivery leaves state unchanged.
+
+## How to get to it (user POV)
+
+Call MCP `scroll` or the CLI `call` surface with a window target, delivery mode, direction, unit and amount. The fixture's tall scroll region represents a real app viewport.
+
+## Driving it with MCP/testkit
+
+Preconditions: [Launch and Doctor](../SKILL.md) succeed; `scroll-tall` is present in the fixture AX tree.
+
+- **AX:** run `CUA_E2E_CELL_FILTER=linux-electron-scroll-ax-foreground timeout 1260 .cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11 --no-build`. The tool arguments include the `scroll-tall` token, `direction: "down"`, `by: "page"`, `amount: 2`. Require the independent fixture journal to show at least one viewport of movement within two seconds.
+- **Pixels:** run `CUA_E2E_CELL_FILTER=linux-electron-scroll-px-foreground timeout 1260 .cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11 --no-build`. Require the same distance contract and before/after images.
+- **Background:** run `CUA_E2E_CELL_FILTER=linux-electron-scroll-ax-background timeout 1260 .cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11 --no-build`. On Linux Electron expect `BackgroundUnavailable`, no fixture mutation and no sentinel/native side effects, not delivered scrolling.
+- **Proof:** preserve request/response, visible viewport change and `fixture-state.json`. CLI entry is not exercised by these MCP cells.
+
+## Gotchas
+
+A response acknowledgment is not movement. Do not loosen the viewport-distance assertion to make a small wheel twitch pass. Background delivery differs by platform; AX does not imply availability. Pixel coordinates require the existing window-origin and screenshot-scale conversion.
diff --git a/.cursor/skills/verify-cua-driver/features/type-text.md b/.cursor/skills/verify-cua-driver/features/type-text.md
new file mode 100644
index 0000000000..827b16e95b
--- /dev/null
+++ b/.cursor/skills/verify-cua-driver/features/type-text.md
@@ -0,0 +1,27 @@
+# Type text
+
+Text entry changes the addressed field and can submit it with a separate Enter action, without modifying another field.
+
+## Sub-features
+
+- `type-text`: replace the target field's text through AX or pixel addressing.
+- `type-submit`: type, obtain a fresh handle, then press Return.
+- `field-isolation`: keep the unrelated number input at `42`.
+- `background-refusal`: reject unavailable delivery without target mutation.
+
+## How to get to it (user POV)
+
+Use MCP `type_text` with text and a window/element target, then `press_key` with `key: "return"` to submit. The CLI `call` surface exposes the same tool names. Select delivery explicitly rather than assuming the currently focused app.
+
+## Driving it with MCP/testkit
+
+Preconditions: [Launch and Doctor](../SKILL.md) succeed and the fixture has `txt-input`, `keyboard-input` and `number-input`.
+
+- **Type:** run `CUA_E2E_CELL_FILTER=linux-electron-type-text-ax-foreground timeout 1260 .cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11 --no-build`. The harness uses a fresh `txt-input` token and `text: "cua-ax-foreground"`. Require `mirror=cua-ax-foreground` and `number-input` still `42` in fixture evidence.
+- **Submit:** run `CUA_E2E_CELL_FILTER=linux-electron-type-submit-ax-foreground timeout 1260 .cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11 --no-build`. Require `key_state=submitted:cua-submit-ax-foreground`, with a new snapshot/token between typing and Return.
+- **Refusal:** run `CUA_E2E_CELL_FILTER=linux-electron-type-text-ax-background timeout 1260 .cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11 --no-build`. Require `BackgroundUnavailable`, unchanged fixture state and stable sentinel/native observations. This is not a delivered-text proof.
+- **Proof:** retain the type/submit requests and responses, before/after captures and independent fixture values. CLI entry remains unverified by the MCP harness.
+
+## Gotchas
+
+Linux Electron background text input is expected to refuse, even with an AX token; never select it as the positive Linux proof. A successful tool response alone does not prove the text or submission. Do not use CDP or a DOM setter to populate the target. Pixel entry is a distinct matrix row, not implicitly covered by AX.
diff --git a/.cursor/skills/verify-cua-driver/features/window-discovery.md b/.cursor/skills/verify-cua-driver/features/window-discovery.md
new file mode 100644
index 0000000000..43a88f5b39
--- /dev/null
+++ b/.cursor/skills/verify-cua-driver/features/window-discovery.md
@@ -0,0 +1,27 @@
+# Window discovery and state
+
+Discover an app window, preserve its process/window identity, and obtain an accessibility snapshot before sending input.
+
+## Sub-features
+
+- `list-windows`: enumerate mapped windows and their identities.
+- `select-window`: distinguish the new fixture from existing windows and the sentinel.
+- `window-state`: obtain AX content and snapshot-bound element tokens.
+- `readiness`: require both AX and independent journal readiness.
+
+## How to get to it (user POV)
+
+Call MCP `list_windows` followed by `get_window_state`, or use those tool names through the CLI `call` surface. The typed SDK exposes native-window discovery and state as well.
+
+## Driving it with MCP/testkit
+
+Preconditions: [Launch and Doctor](../SKILL.md) have a usable display and accessibility bus.
+
+- **Discover and inspect:** run `timeout 1260 .cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11 --no-build`. Preflight and both click cells enumerate windows before launching, then select a newly appearing `CuaTestHarness Electron` title and retain its actual PID/window ID.
+- **AX:** inspect `tool-calls.jsonl` for `list_windows` with `{}` and `get_window_state` with the discovered `pid`, `window_id`, `capture_mode: "ax"` and testkit's `full_output`. Require `WEB_HARNESS_MARKER_v1` before the `READY` signal and the action.
+- **Identity:** the SDK sibling must preserve the fixture PID/window identity on post-action state and mint a new snapshot-bound token. Require typed stale/mismatched handle refusals, not fallback to another window.
+- **Proof:** retain discovered identities, AX tree, transcript and captured images. CLI discovery and broader desktop-state entry points remain unverified by this recipe.
+
+## Gotchas
+
+Do not identify a window solely by a guessed PID, title or socket presence. Fixtures can expose a window before their AX subtree is usable. A fresh snapshot invalidates old tokens. X11 window IDs, Wayland compositor handles and Windows handles are not interchangeable.
diff --git a/.cursor/skills/verify-cua-driver/scripts/check-evidence.sh b/.cursor/skills/verify-cua-driver/scripts/check-evidence.sh
new file mode 100755
index 0000000000..ca5d580ca0
--- /dev/null
+++ b/.cursor/skills/verify-cua-driver/scripts/check-evidence.sh
@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+set -euo pipefail
+shopt -s nullglob
+fail() { echo "Invalid evidence: $*" >&2; exit 1; }
+[[ $# == 1 && -d "$1" ]] || fail 'usage: check-evidence.sh EVIDENCE_DIRECTORY'
+root="$1"
+[[ -s "$root/summary.md" ]] || fail 'missing summary.md'
+trajectories=("$root"/recordings/*/trajectory.json)
+(( ${#trajectories[@]} > 0 )) || fail 'no current-attempt trajectories or owned PIDs'
+for trajectory in "${trajectories[@]}"; do
+ jq -e '.label | type == "string" and length > 0' "$trajectory" >/dev/null || fail "missing label: $trajectory"
+done
+for label in linux-electron-left-click-ax-background linux-electron-left-click-ax-background-typed-sdk environment-preflight; do
+ matches=()
+ for trajectory in "${trajectories[@]}"; do
+ if jq -e --arg label "$label" '.label == $label' "$trajectory" >/dev/null; then
+ matches+=("$(dirname "$trajectory")")
+ fi
+ done
+ (( ${#matches[@]} == 1 )) || fail "expected one current recording for $label"
+ directory="${matches[0]}"
+ if [[ "$label" == environment-preflight ]]; then
+ for name in daemon.json health_report.json check_permissions.json; do
+ [[ -s "$directory/$name" ]] || fail "missing preflight $name"
+ done
+ continue
+ fi
+ jq -Rse --arg label "$label" '
+ [split("\n")[] | gsub("\\\\\\|"; "\u0000") | split("|") |
+ map(gsub("^\\s+|\\s+$"; "")) | select(.[1] == $label)] |
+ length == 1 and (.[0][11] == "Delivered" and .[0][12] == "Pass")
+ ' "$root/summary.md" >/dev/null || fail "$label summary must observe Delivered / Pass"
+ for name in tool-calls.jsonl fixture-state.json sentinel-events.jsonl desktop-observation.txt; do
+ [[ -s "$directory/$name" ]] || fail "missing $label/$name"
+ done
+ jq -e '[.. | strings | test("(^|[^[:alnum:]_])last_action=left_click($|[^[:alnum:]_])")] | any' \
+ "$directory/fixture-state.json" >/dev/null || fail "$label missing left_click fixture marker"
+ jq -se 'any(.[]; .kind == "heartbeat")' "$directory/sentinel-events.jsonl" >/dev/null || fail "$label missing heartbeat"
+ paired=0
+ for turn in "$directory"/turn-*; do
+ [[ -s "$turn/before.png" && -s "$turn/after.png" ]] || continue
+ if [[ "$label" == *-typed-sdk ]] || jq -e '.tool == "click"' "$turn/action.json" >/dev/null; then
+ paired=1
+ fi
+ done
+ (( paired == 1 )) || fail "$label missing screenshot pair (click required for MCP)"
+done
+pids=$(jq -r '.daemon_pid | select(type == "number" and . > 0 and . == floor)' "${trajectories[@]}") || fail 'invalid trajectory PID extraction'
+transcripts=("$root"/recordings/*/tool-calls.jsonl)
+if (( ${#transcripts[@]} > 0 )); then
+ fixture_pids=$(jq -r '
+ select(.request.params.name == "list_windows") |
+ .response.result.structuredContent.windows[]? |
+ select(.title | if type == "string" then startswith("CuaTestHarness") else false end) |
+ .pid | select(type == "number" and . > 0 and . == floor)
+ ' "${transcripts[@]}") || fail 'invalid transcript PID extraction'
+ pids+=$'\n'"$fixture_pids"
+fi
+pids=$(printf '%s\n' "$pids" | sort -nu | grep -v '^$') || fail 'no owned PIDs recorded'
+[[ -n "$pids" ]] || fail 'no owned PIDs recorded'
+while IFS= read -r pid; do
+ kill -0 "$pid" 2>/dev/null && fail "owned PID still alive after cleanup: $pid"
+done <<< "$pids"
+echo "Evidence complete; owned PIDs checked after cleanup: $(tr '\n' ' ' <<< "$pids")"
diff --git a/.cursor/skills/verify-cua-driver/scripts/verify-linux.sh b/.cursor/skills/verify-cua-driver/scripts/verify-linux.sh
new file mode 100755
index 0000000000..05f833b12c
--- /dev/null
+++ b/.cursor/skills/verify-cua-driver/scripts/verify-linux.sh
@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+set -euo pipefail
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../.." && pwd)"
+cd "$ROOT"
+LANE="${1:-x11}"
+shift "$(( $# > 0 ? 1 : 0 ))"
+case "$LANE" in x11|wayland) ;; *) echo 'usage: verify-linux.sh x11|wayland [--no-build]' >&2; exit 2 ;; esac
+mkdir -p artifacts/cua-driver/verify
+export CUA_E2E_ARTIFACT_DIR
+CUA_E2E_ARTIFACT_DIR="$(mktemp -d "$ROOT/artifacts/cua-driver/verify/run-XXXXXXXX")"
+export CUA_E2E_INTERNAL_LANE=verify
+export CUA_E2E_HARNESS_FILTER="${CUA_E2E_HARNESS_FILTER:-electron}"
+export CUA_E2E_CELL_FILTER="${CUA_E2E_CELL_FILTER:-linux-electron-left-click-ax-background}"
+echo "Evidence: $CUA_E2E_ARTIFACT_DIR"
+set +e
+if [[ "$LANE" == wayland ]]; then
+ timeout 1200 scripts/ci/linux/run-rust-e2e-wayland.sh "$@"
+else
+ timeout 1200 xvfb-run -a --server-args='-screen 0 1920x1080x24' \
+ dbus-run-session -- bash -euo pipefail -c '
+ unset WAYLAND_DISPLAY
+ export XDG_SESSION_TYPE=x11
+ pids=()
+ cleanup() {
+ for pid in "${pids[@]}"; do kill "$pid" 2>/dev/null || true; done
+ for pid in "${pids[@]}"; do wait "$pid" 2>/dev/null || true; done
+ }
+ trap cleanup EXIT
+ trap "exit 143" TERM
+ trap "exit 130" INT
+ openbox >"$CUA_E2E_ARTIFACT_DIR/openbox.log" 2>&1 & pids+=("$!")
+ picom --backend xrender --config /dev/null >"$CUA_E2E_ARTIFACT_DIR/picom.log" 2>&1 & pids+=("$!")
+ timeout 15 bash -c '\''until xprop -root _NET_SUPPORTING_WM_CHECK | grep -q "window id"; do sleep .1; done'\''
+ timeout 15 gdbus call --session --dest org.a11y.Bus --object-path /org/a11y/bus \
+ --method org.a11y.Bus.GetAddress >"$CUA_E2E_ARTIFACT_DIR/at-spi-address.txt"
+ scripts/ci/linux/run-rust-e2e.sh "$@"
+ ' verify-x11 "$@"
+fi 2>&1 | tee "$CUA_E2E_ARTIFACT_DIR/launch.log"
+status=${PIPESTATUS[0]}
+set -e
+"$ROOT/.cursor/skills/verify-cua-driver/scripts/check-evidence.sh" "$CUA_E2E_ARTIFACT_DIR" || { [[ "$status" != 0 ]] || status=1; }
+test -r "$CUA_E2E_ARTIFACT_DIR/launch.log"
+echo "Evidence retained after cleanup: $CUA_E2E_ARTIFACT_DIR"
+exit "$status"
diff --git a/.github/scripts/tests/test_e2e_rust_linux_workflow.py b/.github/scripts/tests/test_e2e_rust_linux_workflow.py
index c23f094500..07faf0b142 100644
--- a/.github/scripts/tests/test_e2e_rust_linux_workflow.py
+++ b/.github/scripts/tests/test_e2e_rust_linux_workflow.py
@@ -1,6 +1,9 @@
+import json
+import subprocess
+import tempfile
+import unittest
from pathlib import Path
-
ROOT = Path(__file__).resolve().parents[3]
WORKFLOW = ROOT / ".github/workflows/e2e-rust-linux.yml"
@@ -28,3 +31,263 @@ def test_release_gate_call_matches_dispatch_and_ignores_caller_artifacts() -> No
assert name in dispatch
assert "github.event.inputs" not in workflow
assert "pattern: rust-linux-*" in workflow
+
+
+class VerifySkillTests(unittest.TestCase):
+ def test_workflow_proof_is_isolated_and_retained(self) -> None:
+ workflow = WORKFLOW.read_text(encoding="utf-8")
+ for path in (
+ ".cursor/skills/verify-cua-driver/**",
+ "libs/cua-driver/**",
+ ".github/workflows/e2e-rust-linux.yml",
+ ):
+ self.assertIn(f" - {path}", workflow.split(" workflow_call:")[0])
+ self.assertIn(" - verify", workflow)
+ job = workflow.split("\n verify-skill:\n")[1].split("\n shared:\n")[0]
+ for setting in (
+ "if: inputs.lane == '' || inputs.lane == 'verify'",
+ "timeout-minutes: 25",
+ 'CUA_E2E_RETRY_ONCE: "0"',
+ "key: e2e-shared",
+ "CUA_E2E_SOURCE_SHA: ${{ needs.source.outputs.sha }}",
+ "run: .cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11",
+ "name: verify-cua-driver-evidence",
+ "path: artifacts/cua-driver/verify/run-*",
+ "if-no-files-found: error",
+ "if: always()",
+ "test_e2e_rust_linux_workflow.py",
+ ):
+ self.assertIn(setting, job)
+ self.assertNotIn("name: rust-linux-", job)
+ for name in ("summary", "certify"):
+ section = workflow.split(f"\n {name}:\n")[1].split(" steps:")[0]
+ self.assertIn("needs: [source, shared, native, capture, installer]", section)
+ self.assertNotIn("verify-skill", section)
+ summary = workflow.split("\n summary:\n")[1].split(" needs:")[0]
+ self.assertIn("always() && inputs.lane != '' && inputs.lane != 'verify'", summary)
+
+ def test_evidence_boundary(self) -> None:
+ checker = ROOT / ".cursor/skills/verify-cua-driver/scripts/check-evidence.sh"
+ self.assertTrue(checker.is_file(), "standalone evidence checker is required")
+ cell = "linux-electron-left-click-ax-background"
+ sleeper = subprocess.Popen(["sleep", "60"])
+ dead = subprocess.Popen(["true"])
+ dead.wait(timeout=5)
+ try:
+ scenarios = [
+ "complete",
+ "zero-pids",
+ "survivor",
+ "fixture-survivor",
+ "malformed-transcript",
+ "null-window-title",
+ "observed-refused",
+ "status-failed",
+ "missing-sdk-row",
+ "retry-decoy",
+ "duplicate-label",
+ "missing-label",
+ "wrong-marker",
+ "no-heartbeat",
+ "no-click",
+ "incomplete-pair",
+ "tool-calls.jsonl",
+ "fixture-state.json",
+ "sentinel-events.jsonl",
+ "desktop-observation.txt",
+ "before.png",
+ "after.png",
+ "daemon.json",
+ "health_report.json",
+ "check_permissions.json",
+ "summary.md",
+ ]
+ for scenario in scenarios:
+ with self.subTest(scenario=scenario), tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ rows = []
+ for label in (cell, cell + "-typed-sdk", "environment-preflight"):
+ directory = root / "recordings" / label
+ directory.mkdir(parents=True)
+ (directory / "trajectory.json").write_text(
+ json.dumps(
+ {
+ "label": label,
+ "daemon_pid": dead.pid,
+ }
+ )
+ )
+ if label == "environment-preflight":
+ for name in (
+ "daemon.json",
+ "health_report.json",
+ "check_permissions.json",
+ ):
+ (directory / name).write_text("{}")
+ continue
+ rows.append(
+ "| "
+ + " | ".join(
+ [
+ label,
+ "Linux/X11",
+ "electron",
+ "left_click",
+ "Ax",
+ "Background",
+ "Window",
+ "Mcp",
+ "State",
+ "Delivered",
+ "Delivered",
+ "Pass",
+ "1 ms",
+ "-",
+ r"details \| escaped",
+ ]
+ )
+ + " |\n"
+ )
+ (directory / "tool-calls.jsonl").write_text(
+ json.dumps(
+ {
+ "request": {"params": {"name": "list_windows"}},
+ "response": {
+ "result": {
+ "structuredContent": {
+ "windows": [
+ {"title": "CuaTestHarness", "pid": dead.pid},
+ ]
+ }
+ }
+ },
+ }
+ )
+ + "\n"
+ )
+ (directory / "fixture-state.json").write_text(
+ json.dumps(
+ {
+ "text": "journal last_action=left_click",
+ }
+ )
+ )
+ (directory / "sentinel-events.jsonl").write_text('{"kind":"heartbeat"}\n')
+ (directory / "desktop-observation.txt").write_text("stable")
+ turn = directory / "turn-00001"
+ turn.mkdir()
+ for name in ("before.png", "after.png"):
+ (turn / name).write_bytes(b"png")
+ (turn / "action.json").write_text(
+ json.dumps(
+ {
+ "tool": (
+ "press_key" if label.endswith("-typed-sdk") else "click"
+ ),
+ }
+ )
+ )
+ (directory / "turn-00002").mkdir()
+ (root / "summary.md").write_text("".join(rows))
+ target = root / "recordings" / cell
+ if scenario == "zero-pids":
+ for path in root.glob("recordings/*/trajectory.json"):
+ data = json.loads(path.read_text())
+ data.pop("daemon_pid")
+ path.write_text(json.dumps(data))
+ for path in root.glob("recordings/*/tool-calls.jsonl"):
+ path.write_text("{}\n")
+ elif scenario == "survivor":
+ (target / "trajectory.json").write_text(
+ json.dumps(
+ {
+ "label": cell,
+ "daemon_pid": sleeper.pid,
+ }
+ )
+ )
+ elif scenario == "fixture-survivor":
+ path = target / "tool-calls.jsonl"
+ data = json.loads(path.read_text())
+ data["response"]["result"]["structuredContent"]["windows"][0][
+ "pid"
+ ] = sleeper.pid
+ path.write_text(json.dumps(data))
+ elif scenario == "malformed-transcript":
+ (target / "tool-calls.jsonl").write_text("{broken\n")
+ elif scenario == "null-window-title":
+ (target / "tool-calls.jsonl").write_text(
+ json.dumps(
+ {
+ "request": {"params": {"name": "list_windows"}},
+ "response": {
+ "result": {
+ "structuredContent": {
+ "windows": [
+ {"title": None},
+ {"title": 7},
+ ]
+ }
+ }
+ },
+ }
+ )
+ )
+ elif scenario in ("observed-refused", "status-failed"):
+ (root / "summary.md").write_text(
+ "".join(rows).replace(
+ "Delivered | Pass",
+ (
+ "Refused | Pass"
+ if scenario == "observed-refused"
+ else "Delivered | Fail"
+ ),
+ )
+ )
+ elif scenario == "missing-sdk-row":
+ (root / "summary.md").write_text(rows[0])
+ elif scenario == "retry-decoy":
+ destination = root / "retried-attempts/first/recordings"
+ destination.mkdir(parents=True)
+ (target).rename(destination / cell)
+ elif scenario == "duplicate-label":
+ duplicate = root / "recordings/duplicate"
+ duplicate.mkdir()
+ (duplicate / "trajectory.json").write_text(
+ (target / "trajectory.json").read_text()
+ )
+ elif scenario == "missing-label":
+ (target / "trajectory.json").write_text('{"daemon_pid":1}')
+ elif scenario == "wrong-marker":
+ (target / "fixture-state.json").write_text(
+ '{"text":"last_action=left_click_extra"}'
+ )
+ elif scenario == "no-heartbeat":
+ (target / "sentinel-events.jsonl").write_text('{"kind":"focus"}\n')
+ elif scenario == "no-click":
+ (target / "turn-00001/action.json").write_text('{"tool":"press_key"}')
+ elif scenario == "incomplete-pair":
+ (target / "turn-00001/after.png").rename(target / "turn-00002/after.png")
+ elif scenario in (
+ "daemon.json",
+ "health_report.json",
+ "check_permissions.json",
+ ):
+ (root / "recordings/environment-preflight" / scenario).unlink()
+ elif scenario in ("before.png", "after.png"):
+ (target / "turn-00001" / scenario).unlink()
+ elif scenario == "summary.md":
+ (root / scenario).unlink()
+ elif scenario.endswith((".json", ".jsonl", ".txt")):
+ (target / scenario).unlink()
+ result = subprocess.run(
+ [str(checker), str(root)], capture_output=True, text=True, timeout=10
+ )
+ if scenario in ("complete", "null-window-title"):
+ self.assertEqual(result.returncode, 0, result.stderr)
+ else:
+ self.assertNotEqual(result.returncode, 0, scenario)
+ self.assertTrue(result.stderr.strip(), scenario)
+ finally:
+ sleeper.terminate()
+ sleeper.wait(timeout=5)
diff --git a/.github/workflows/e2e-rust-linux.yml b/.github/workflows/e2e-rust-linux.yml
index 434824ae3d..0b4858efc6 100644
--- a/.github/workflows/e2e-rust-linux.yml
+++ b/.github/workflows/e2e-rust-linux.yml
@@ -1,6 +1,11 @@
name: "E2E: Rust Linux interactive"
on:
+ pull_request:
+ paths:
+ - .cursor/skills/verify-cua-driver/**
+ - libs/cua-driver/**
+ - .github/workflows/e2e-rust-linux.yml
# Stable Cua Driver tag runs call this matrix as a release gate. Keep the
# inputs in sync with workflow_dispatch; only `inputs.*` is read below.
workflow_call:
@@ -36,6 +41,7 @@ on:
- native
- capture
- installer
+ - verify
cell_filter:
description: "Optional shared-cell substring for targeted diagnostics"
required: false
@@ -84,6 +90,53 @@ jobs:
fi
echo "sha=${sha,,}" >> "$GITHUB_OUTPUT"
+ verify-skill:
+ if: inputs.lane == '' || inputs.lane == 'verify'
+ name: "Linux / verification skill proof"
+ needs: source
+ runs-on: ubuntu-latest
+ timeout-minutes: 25
+ env:
+ CUA_E2E_SOURCE_SHA: ${{ needs.source.outputs.sha }}
+ CUA_E2E_RETRY_ONCE: "0"
+ steps:
+ - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+ with:
+ ref: ${{ needs.source.outputs.sha }}
+ - name: Test evidence checker and workflow offline
+ run: timeout 60 python3 -m unittest discover -s .github/scripts/tests -p test_e2e_rust_linux_workflow.py
+ - uses: dtolnay/rust-toolchain@4716b85f2fac3e324e64fa2810f6b5c3905760a5 # 1.97.1
+ - uses: Swatinem/rust-cache@9d47c6ad4b02e050fd481d890b2ea34778fd09d6 # v2
+ with:
+ workspaces: "libs/cua-driver/rust -> target"
+ key: e2e-shared
+ save-if: ${{ github.ref_type == 'branch' && needs.source.outputs.sha == github.sha }}
+ - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
+ with:
+ node-version: "22"
+ cache: ${{ github.ref_type == 'branch' && needs.source.outputs.sha == github.sha && 'npm' || '' }}
+ cache-dependency-path: libs/cua-driver/tests/fixtures/apps/cross-platform/electron/package-lock.json
+ - name: Install GUI and Rust build dependencies
+ run: |
+ sudo apt-get update
+ sudo apt-get install -y --no-install-recommends \
+ xvfb dbus-x11 at-spi2-core openbox picom python3-gi gir1.2-gtk-3.0 \
+ libgtk-3-dev clang pkg-config libdbus-1-dev libpipewire-0.3-dev \
+ libspa-0.2-dev libei-dev libx11-dev libxi-dev libxtst-dev libxext-dev \
+ libwebkit2gtk-4.1-dev libssl-dev libxdo-dev \
+ libayatana-appindicator3-dev librsvg2-dev ffmpeg jq x11-utils
+ - name: Verify real Electron background clicks
+ run: .cursor/skills/verify-cua-driver/scripts/verify-linux.sh x11
+ - name: Upload verification evidence after cleanup
+ if: always()
+ uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08 # v4
+ with:
+ name: verify-cua-driver-evidence
+ path: artifacts/cua-driver/verify/run-*
+ if-no-files-found: error
+ compression-level: 0
+ retention-days: 14
+
shared:
if: inputs.lane == 'all' || inputs.lane == 'shared'
name: "Linux / shared Electron + Tauri"
@@ -301,7 +354,7 @@ jobs:
retention-days: 14
summary:
- if: always()
+ if: always() && inputs.lane != '' && inputs.lane != 'verify'
needs: [source, shared, native, capture, installer]
name: "Linux / matrix summary"
runs-on: ubuntu-latest
diff --git a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs
index d45ba7fe84..0d0ed3a324 100644
--- a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/cross_platform_behavior_test.rs
@@ -182,7 +182,11 @@ where
let outcome = panic::catch_unwind(AssertUnwindSafe(|| {
let mut fixture = launch_host_with_evidence(spec, &case.cell_id, &mut evidence);
let sentinel = if delivery == Delivery::Background {
- Some(ForegroundSentinel::launch(&mut fixture.driver))
+ let mut sentinel = ForegroundSentinel::launch(&mut fixture.driver);
+ if let Some(dir) = fixture.driver.recording_dir() {
+ sentinel.persist_evidence_to(dir);
+ }
+ Some(sentinel)
} else {
None
};
@@ -282,6 +286,19 @@ struct Fixture {
journal: FixtureJournal,
}
+impl Drop for Fixture {
+ fn drop(&mut self) {
+ if let Some(dir) = self.driver.recording_dir() {
+ std::fs::write(
+ dir.join("fixture-state.json"),
+ serde_json::to_vec_pretty(&self.journal.snapshot())
+ .expect("serialize fixture state"),
+ )
+ .expect("persist fixture state");
+ }
+ }
+}
+
fn evidence_for_driver(driver: &McpDriver) -> Evidence {
recording_evidence(driver.recording_dir())
}
@@ -392,6 +409,7 @@ fn launch_host_with_evidence(spec: &HostSpec, scenario: &str, evidence: &mut Evi
if last_tree.contains("WEB_HARNESS_MARKER_v1")
&& fixture.journal.contains("WEB_HARNESS_MARKER_v1")
{
+ eprintln!("READY {scenario}: daemon usable, fixture pid={actual_pid} window_id={wid}, AX and journal marker present");
return fixture;
}
thread::sleep(Duration::from_millis(250));
diff --git a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/e2e_environment_preflight_test.rs b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/e2e_environment_preflight_test.rs
index 08dd63a093..1dfcadba86 100644
--- a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/e2e_environment_preflight_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/e2e_environment_preflight_test.rs
@@ -290,6 +290,43 @@ fn run_preflight() {
"preflight recording escaped the artifact root"
);
+ for tool in ["health_report", "check_permissions"] {
+ let response = driver.call(tool, serde_json::json!({}));
+ std::fs::write(
+ recording_dir.join(format!("{tool}.json")),
+ serde_json::to_vec_pretty(&response.structured()).unwrap(),
+ )
+ .unwrap();
+ assert!(!response.is_error(), "{tool} failed: {}", response.text());
+ }
+ #[cfg(unix)]
+ {
+ use std::os::unix::fs::{FileTypeExt, MetadataExt};
+ let socket = std::fs::metadata(driver.daemon_socket()).expect("daemon socket metadata");
+ let uid = Command::new("id").arg("-u").output().unwrap();
+ let uid: u32 = String::from_utf8(uid.stdout)
+ .unwrap()
+ .trim()
+ .parse()
+ .unwrap();
+ assert!(socket.file_type().is_socket());
+ assert_eq!(socket.uid(), uid, "daemon socket is not owned by us");
+ assert_eq!(
+ socket.mode() & 0o077,
+ 0,
+ "daemon socket permits other users"
+ );
+ std::fs::write(
+ recording_dir.join("daemon.json"),
+ serde_json::to_vec_pretty(
+ &serde_json::json!({"pid": driver.daemon_pid(), "socket": driver.daemon_socket(),
+ "uid": socket.uid(), "mode": socket.mode() & 0o777, "config": config.structured()}),
+ )
+ .unwrap(),
+ )
+ .unwrap();
+ }
+
let before = driver.call("list_windows", serde_json::json!({}));
assert!(!before.is_error(), "list_windows failed: {}", before.text());
let before_ids = before.structured()["windows"]
diff --git a/libs/cua-driver/rust/crates/cua-driver-testkit/src/mcp.rs b/libs/cua-driver/rust/crates/cua-driver-testkit/src/mcp.rs
index 64d94f5dd5..c988e0554e 100644
--- a/libs/cua-driver/rust/crates/cua-driver-testkit/src/mcp.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-testkit/src/mcp.rs
@@ -301,6 +301,14 @@ impl McpDriver {
)
}
+ pub fn daemon_socket(&self) -> &str {
+ &self.socket
+ }
+
+ pub fn daemon_pid(&self) -> Option<u32> {
+ self._daemon.as_ref().map(|daemon| daemon.pid)
+ }
+
/// Isolated per-user state root (`HOME`, XDG/AppData directories) given to
/// the test-owned daemon, or `None` for external-daemon proxies and spawns
/// that passed [`crate::SHARE_HOST_STATE`].
@@ -342,6 +350,7 @@ impl McpDriver {
"label": label,
"rust_test_thread": thread_name,
"process_id": std::process::id(),
+ "daemon_pid": self.daemon_pid(),
"sequence": sequence,
"behavior_video": {
"status": "pending",
@@ -407,7 +416,7 @@ impl McpDriver {
}
fn stop_e2e_recording(&mut self) {
- let Some(output_dir) = self.recording_dir.take() else {
+ let Some(output_dir) = self.recording_dir.clone() else {
return;
};
if !self.recording_started {
@@ -488,17 +497,51 @@ impl McpDriver {
self.next_id += 1;
let timeout = call_timeout(self.recording_started, tool, cfg!(target_os = "macos"));
let args = crate::driver::with_full_window_state(tool, args);
- self.send(serde_json::json!({
+ let request = serde_json::json!({
"jsonrpc": "2.0", "id": id, "method": "tools/call",
"params": { "name": tool, "arguments": args }
- }));
- match self.rx.recv_timeout(timeout) {
+ });
+ self.send(request.clone());
+ let response = match self.rx.recv_timeout(timeout) {
Ok(line) => serde_json::from_str(&line)
.unwrap_or_else(|_| serde_json::json!({ "error": format!("bad json: {line}") })),
Err(_) => serde_json::json!({
"error": timeout_error_message(tool, timeout)
}),
+ };
+ if let Some(dir) = &self.recording_dir {
+ let mut row = serde_json::json!({"request": request, "response": response});
+ elide_images(&mut row);
+ let mut file = std::fs::OpenOptions::new()
+ .create(true)
+ .append(true)
+ .open(dir.join("tool-calls.jsonl"))
+ .expect("open tool transcript");
+ writeln!(file, "{row}").expect("append tool transcript");
}
+ response
+ }
+}
+
+fn elide_images(value: &mut Value) {
+ use std::hash::{Hash, Hasher};
+ match value {
+ Value::Object(object) => {
+ let image = object.get("type").and_then(Value::as_str) == Some("image");
+ for (key, value) in object {
+ if (image && key == "data") || key.contains("base64") {
+ if let Some(data) = value.as_str() {
+ let mut hash = std::collections::hash_map::DefaultHasher::new();
+ data.hash(&mut hash);
+ *value = serde_json::json!({"bytes": data.len(), "hash": format!("{:016x}", hash.finish())});
+ continue;
+ }
+ }
+ elide_images(value);
+ }
+ }
+ Value::Array(array) => array.iter_mut().for_each(elide_images),
+ _ => {}
}
}
@@ -621,6 +664,70 @@ mod tests {
use crate::CALL_TIMEOUT;
use std::time::Duration;
+ #[cfg(unix)]
+ #[test]
+ fn transcript_retains_timeout_request_and_elides_images() {
+ use super::*;
+ let dir = tempfile::tempdir().unwrap();
+ let mut command = Command::new("sh");
+ command
+ .args(["-c", "read line; sleep 1"])
+ .stdin(Stdio::piped());
+ let mut child = spawn_in_job(&mut command).unwrap();
+ let stdin = child.stdin.take().unwrap();
+ let mut reaper = ChildReaper::new();
+ reaper.push(child);
+ let (tx, rx) = channel();
+ let mut driver = McpDriver {
+ reaper,
+ _daemon: None,
+ socket: String::new(),
+ stdin,
+ rx,
+ next_id: 2,
+ recording_dir: Some(dir.path().to_owned()),
+ recording_started: false,
+ recording_started_at: None,
+ };
+ tx.send(
+ serde_json::json!({"result": {"content": [
+ {"type": "image", "data": "c2VjcmV0", "mimeType": "image/png"},
+ {"type": "text", "text": "unchanged"}],
+ "structuredContent": {"screenshot_base64": "c2VjcmV0", "count": 7}}})
+ .to_string(),
+ )
+ .unwrap();
+ driver.call_raw("get_window_state", serde_json::json!({"pid": 42}));
+ drop(tx);
+ let response = driver.call_raw("click", serde_json::json!({"element_token": "fresh"}));
+ let text = std::fs::read_to_string(dir.path().join("tool-calls.jsonl")).unwrap();
+ let rows: Vec<Value> = text
+ .lines()
+ .map(|line| serde_json::from_str(line).unwrap())
+ .collect();
+ assert_eq!(
+ rows[1]["request"]["params"]["arguments"]["element_token"],
+ "fresh"
+ );
+ assert_eq!(rows[1]["response"], response);
+ assert!(response["error"].as_str().unwrap().contains("TIMEOUT"));
+ assert!(!text.contains("c2VjcmV0"));
+ assert_eq!(
+ rows[0]["response"]["result"]["content"][0]["data"]["bytes"],
+ 8
+ );
+ assert!(rows[0]["response"]["result"]["content"][0]["data"]["hash"].is_string());
+ assert_eq!(
+ rows[0]["response"]["result"]["content"][1]["text"],
+ "unchanged"
+ );
+ assert_eq!(
+ rows[0]["response"]["result"]["structuredContent"]["count"],
+ 7
+ );
+ driver.recording_dir = None;
+ }
+
#[test]
fn recording_label_is_artifact_safe() {
assert_eq!(
diff --git a/libs/cua-driver/rust/crates/cua-driver-testkit/src/sentinel.rs b/libs/cua-driver/rust/crates/cua-driver-testkit/src/sentinel.rs
index d26e2e64c4..18ab8744da 100644
--- a/libs/cua-driver/rust/crates/cua-driver-testkit/src/sentinel.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-testkit/src/sentinel.rs
@@ -17,6 +17,7 @@ mod hyprland;
/// fully occludes the background target.
pub struct ForegroundSentinel {
cursor_calibrated: bool,
+ evidence_dir: Option<std::path::PathBuf>,
journal_path: std::path::PathBuf,
target: TargetWindow,
_reaper: ChildReaper,
@@ -158,6 +159,7 @@ impl ForegroundSentinel {
.map_err(|error| format!("reset focused sentinel journal: {error}"))?;
Ok(Self {
+ evidence_dir: None,
cursor_calibrated,
journal_path,
target,
@@ -303,6 +305,10 @@ impl ForegroundSentinel {
self.assert_background_posture(background_target)
}
+ pub fn persist_evidence_to(&mut self, dir: &std::path::Path) {
+ self.evidence_dir = Some(dir.to_owned());
+ }
+
pub fn target(&self) -> TargetWindow {
self.target
}
@@ -357,7 +363,7 @@ impl ForegroundSentinel {
target: TargetWindow,
action: impl FnOnce() -> R,
) -> Result<(R, Vec<OracleKind>), String> {
- self.observe_target(target, true, action)
+ self.observe_target(target, true, NativeObserver::new(), action)
}
/// Observe a desktop-wide action against the foreground sentinel itself.
@@ -367,55 +373,77 @@ impl ForegroundSentinel {
&self,
action: impl FnOnce() -> R,
) -> Result<(R, Vec<OracleKind>), String> {
- self.observe_target(self.target, false, action)
+ self.observe_target(self.target, false, NativeObserver::new(), action)
}
fn observe_target<R>(
&self,
target: TargetWindow,
require_occluded_target: bool,
+ backend: impl crate::observer::ObserverBackend,
action: impl FnOnce() -> R,
) -> Result<(R, Vec<OracleKind>), String> {
- let mut observer = DesktopObserver::new(NativeObserver::new(), target);
- let before = observer.snapshot().map_err(|error| error.to_string())?;
- if require_occluded_target
- && before.target_z != crate::observer::TargetZ::BackgroundOccluded
- {
- return Err(format!(
- "background target was not fully occluded before dispatch: {:?}",
- before.target_z
- ));
- }
- let mut native_oracles = vec![OracleKind::Focus, OracleKind::ZOrder];
- if self.cursor_calibrated {
- native_oracles.push(OracleKind::Cursor);
- }
- let (result, delta) = observer
- .observe(&native_oracles, action)
- .map_err(|error| error.to_string())?;
- if require_occluded_target
- && delta.before.target_z != crate::observer::TargetZ::BackgroundOccluded
- {
- return Err(format!(
- "background target was not fully occluded at dispatch: {:?}",
- delta.before.target_z
- ));
+ let mut native_evidence = String::new();
+ let outcome = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
+ let mut observer = DesktopObserver::new(backend, target);
+ let before = observer.snapshot().map_err(|error| error.to_string())?;
+ native_evidence = format!("{before:#?}");
+ if require_occluded_target
+ && before.target_z != crate::observer::TargetZ::BackgroundOccluded
+ {
+ return Err(format!(
+ "background target was not fully occluded before dispatch: {:?}",
+ before.target_z
+ ));
+ }
+ let mut native_oracles = vec![OracleKind::Focus, OracleKind::ZOrder];
+ if self.cursor_calibrated {
+ native_oracles.push(OracleKind::Cursor);
+ }
+ let (result, delta) = observer
+ .observe(&native_oracles, action)
+ .map_err(|error| error.to_string())?;
+ native_evidence = format!("{delta:#?}");
+ if require_occluded_target
+ && delta.before.target_z != crate::observer::TargetZ::BackgroundOccluded
+ {
+ return Err(format!(
+ "background target was not fully occluded at dispatch: {:?}",
+ delta.before.target_z
+ ));
+ }
+ delta
+ .ensure_supported()
+ .map_err(|error| error.to_string())?;
+
+ let mut passed = delta.passed().to_vec();
+ let mut violations = delta.violations().to_vec();
+ let (sentinel_passed, sentinel_violations) = self.observe();
+ passed.extend(sentinel_passed);
+ violations.extend(sentinel_violations);
+ passed.sort();
+ passed.dedup();
+ if violations.is_empty() {
+ Ok((result, passed))
+ } else {
+ Err(violations.join("; "))
+ }
+ }));
+ if let Some(dir) = &self.evidence_dir {
+ fs::create_dir_all(dir).map_err(|error| error.to_string())?;
+ fs::copy(&self.journal_path, dir.join("sentinel-events.jsonl"))
+ .map_err(|error| error.to_string())?;
+ match &outcome {
+ Ok(Err(error)) => native_evidence.push_str(&format!("\nERROR: {error}\n")),
+ Err(_) => native_evidence.push_str("\nERROR: action panicked\n"),
+ _ => {}
+ }
+ fs::write(dir.join("desktop-observation.txt"), native_evidence)
+ .map_err(|error| error.to_string())?;
}
- delta
- .ensure_supported()
- .map_err(|error| error.to_string())?;
-
- let mut passed = delta.passed().to_vec();
- let mut violations = delta.violations().to_vec();
- let (sentinel_passed, sentinel_violations) = self.observe();
- passed.extend(sentinel_passed);
- violations.extend(sentinel_violations);
- passed.sort();
- passed.dedup();
- if violations.is_empty() {
- Ok((result, passed))
- } else {
- Err(violations.join("; "))
+ match outcome {
+ Ok(result) => result,
+ Err(payload) => std::panic::resume_unwind(payload),
}
}
}
@@ -877,6 +905,79 @@ fn sway_container_id(node: &serde_json::Value, title: &str) -> Option<i64> {
mod sway_tests {
use super::*;
+ #[test]
+ fn observation_evidence_survives_violations_and_unsupported_oracles() {
+ use crate::observer::*;
+ struct Backend(bool);
+ impl ObserverBackend for Backend {
+ fn capabilities(&self) -> ObserverCapabilities {
+ ObserverCapabilities {
+ focus: self.0,
+ z_order: true,
+ cursor: true,
+ leaked_input: true,
+ }
+ }
+ fn snapshot(&self, _: TargetWindow) -> Result<DesktopSnapshot, ObserverError> {
+ Ok(DesktopSnapshot {
+ foreground: Some(1),
+ input_focus: Some(1),
+ target_z: TargetZ::BackgroundOccluded,
+ cursor_pos: Some((2.0, 3.0)),
+ })
+ }
+ fn start_journal(&mut self) -> Result<(), ObserverError> {
+ Ok(())
+ }
+ fn drain_journal(&mut self) -> Result<DesktopJournal, ObserverError> {
+ Ok(DesktopJournal {
+ focus_events: vec![FocusEvent {
+ from: Some(1),
+ to: Some(2),
+ }],
+ leaked_input_events: vec![],
+ })
+ }
+ }
+ for supported in [true, false] {
+ let scratch = tempfile::tempdir().unwrap();
+ let evidence = tempfile::tempdir().unwrap();
+ let journal = scratch.path().join("sentinel-events.jsonl");
+ fs::write(&journal, "{\"kind\":\"heartbeat\"}\n").unwrap();
+ let target = TargetWindow {
+ pid: 10,
+ native_id: 20,
+ };
+ let mut sentinel = ForegroundSentinel {
+ cursor_calibrated: true,
+ journal_path: journal,
+ target,
+ _reaper: ChildReaper::new(),
+ _user_data: scratch,
+ evidence_dir: None,
+ };
+ sentinel.persist_evidence_to(evidence.path());
+ let result = sentinel.observe_target(target, true, Backend(supported), || ());
+ let error = result.unwrap_err();
+ assert!(error.contains(if supported {
+ "transiently"
+ } else {
+ "does not support"
+ }));
+ drop(sentinel);
+ assert!(
+ fs::read_to_string(evidence.path().join("sentinel-events.jsonl"))
+ .unwrap()
+ .contains("heartbeat")
+ );
+ let native =
+ fs::read_to_string(evidence.path().join("desktop-observation.txt")).unwrap();
+ assert!(native.contains("cursor_pos"));
+ assert!(native.contains("focus_events"));
+ assert!(native.contains(&error));
+ }
+ }
+
#[test]
fn fullscreen_command_targets_the_holder_ancestor() {
let tree = serde_json::json!({
diff --git a/scripts/ci/linux/run-rust-e2e-wayland.sh b/scripts/ci/linux/run-rust-e2e-wayland.sh
index 4958d3a34a..58cc8b696b 100755
--- a/scripts/ci/linux/run-rust-e2e-wayland.sh
+++ b/scripts/ci/linux/run-rust-e2e-wayland.sh
@@ -14,8 +14,8 @@ if [[ "${SESSION_KIND}" == sway-xwayland ]]; then
# their established Sway paths while this wrapper additionally enables XWayland.
export CUA_E2E_WAYLAND_SESSION=sway
fi
-COMPOSITOR_LOG="${REPO_ROOT}/artifacts/cua-driver/linux/${SESSION_KIND}.log"
-ATSPI_LOG="${REPO_ROOT}/artifacts/cua-driver/linux/at-spi-bus.log"
+COMPOSITOR_LOG="${CUA_E2E_ARTIFACT_DIR:-${REPO_ROOT}/artifacts/cua-driver/linux}/${SESSION_KIND}.log"
+ATSPI_LOG="${CUA_E2E_ARTIFACT_DIR:-${REPO_ROOT}/artifacts/cua-driver/linux}/at-spi-bus.log"
COMPOSITOR_PID=""
DBUS_PID=""
ATSPI_PID=""
diff --git a/scripts/ci/linux/run-rust-e2e.sh b/scripts/ci/linux/run-rust-e2e.sh
index ed7f4f5160..764f190eb9 100755
--- a/scripts/ci/linux/run-rust-e2e.sh
+++ b/scripts/ci/linux/run-rust-e2e.sh
@@ -30,7 +30,7 @@ while (($#)); do
done
case "$SUITE" in
- shared|native|capture|all) ;;
+ shared|native|capture|all|verify) ;;
s1-perception)
# Model-backed perception decision row. It is not part of `all`: it needs
# the published cua-perception catalog and the pinned Cua-S1-4B weights,
@@ -50,7 +50,7 @@ case "$SUITE" in
*) echo "unsupported internal lane: $SUITE" >&2; exit 2 ;;
esac
-ARTIFACT_DIR="${REPO_ROOT}/artifacts/cua-driver/linux"
+ARTIFACT_DIR="${CUA_E2E_ARTIFACT_DIR:-${REPO_ROOT}/artifacts/cua-driver/linux}"
mkdir -p "${ARTIFACT_DIR}"
RECORDING_ROOT="${ARTIFACT_DIR}/recordings"
rm -rf "${RECORDING_ROOT}"
@@ -143,7 +143,7 @@ if [[ "${BUILD_FIXTURES}" == 1 ]]; then
fi
;;
capture) FIXTURE_TARGETS="electron,gtk3" ;;
- s1-perception) FIXTURE_TARGETS="electron" ;;
+ s1-perception|verify) FIXTURE_TARGETS="electron" ;;
*)
FIXTURE_TARGETS="${CUA_E2E_HARNESS_FILTER:-electron,tauri},gtk3"
if [[ -z "${WAYLAND_DISPLAY:-}" || -n "${DISPLAY:-}" ]]; then
@@ -193,6 +193,11 @@ run_report() {
--output "${SUMMARY_FILE}")
}
+if [[ "${SUITE}" == verify ]]; then
+ "${CUA_TEST_DRIVER_BIN}" doctor --json > "${ARTIFACT_DIR}/doctor.json"
+ "${CUA_TEST_DRIVER_BIN}" --version > "${ARTIFACT_DIR}/version.txt"
+fi
+
echo "[PREFLIGHT] Linux desktop, fixture, AX, capture, and video"
set +e
(cd "${RUST_ROOT}" && cargo test -p cua-driver-e2e \
@@ -298,11 +303,17 @@ if [[ "${SUITE}" == shared || "${SUITE}" == all ]]; then
run_test private-worker-lifecycle \
cargo test -p cua-driver "${CARGO_DRIVER_FEATURE_ARGS[@]}" \
--test private_worker_test -- --test-threads=1
+fi
+
+if [[ "${SUITE}" == shared || "${SUITE}" == all || "${SUITE}" == verify ]]; then
run_test shared-behavior-matrix \
cargo test -p cua-driver-e2e "${CARGO_DRIVER_FEATURE_ARGS[@]}" \
--test cross_platform_behavior_test -- \
--ignored --exact shared_web_action_matrix_is_state_verified \
--nocapture --test-threads=1
+fi
+
+if [[ "${SUITE}" == shared || "${SUITE}" == all ]]; then
run_test embedded-browser-routes \
cargo test -p cua-driver-e2e "${CARGO_DRIVER_FEATURE_ARGS[@]}" \
--test cross_platform_behavior_test -- \