Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

test(cua-driver): certify the published cua-perception catalog in canonical desktop E2E (#4322)

Francesco Bonacci4dc4cc0bd5db
diff --git a/.github/workflows/e2e-rust-linux.yml b/.github/workflows/e2e-rust-linux.yml
index a8ac30cd5a..153b85f2da 100644
--- a/.github/workflows/e2e-rust-linux.yml
+++ b/.github/workflows/e2e-rust-linux.yml
@@ -204,7 +204,8 @@ jobs:
libgtk-3-dev clang pkg-config libdbus-1-dev libpipewire-0.3-dev \
libspa-0.2-dev libei-dev libx11-dev libxi-dev libxtst-dev libxext-dev \
libwebkit2gtk-4.1-dev libssl-dev libxdo-dev \
- libayatana-appindicator3-dev librsvg2-dev ffmpeg
+ libayatana-appindicator3-dev librsvg2-dev ffmpeg \
+ python3-tk x11-utils
- name: Run capture and desktop-scope contracts
env:
CUA_E2E_INTERNAL_LANE: capture
diff --git a/.github/workflows/e2e-rust-macos.yml b/.github/workflows/e2e-rust-macos.yml
index ffc7bb4d3d..7593dd7578 100644
--- a/.github/workflows/e2e-rust-macos.yml
+++ b/.github/workflows/e2e-rust-macos.yml
@@ -182,9 +182,20 @@ jobs:
missing=()
command -v ffmpeg >/dev/null 2>&1 || missing+=(ffmpeg)
command -v jq >/dev/null 2>&1 || missing+=(jq)
+ # The capture lane's published-catalog perception row drives the
+ # Tk visual-only canvas fixture. Hosted Homebrew Python ships
+ # without _tkinter, so add the matching python-tk formula there.
+ if [[ "${CUA_E2E_INTERNAL_LANE}" == capture ]] \
+ && ! python3 -c 'import tkinter' >/dev/null 2>&1; then
+ python_minor="$(python3 -c 'import sys; print(f"{sys.version_info[0]}.{sys.version_info[1]}")')"
+ missing+=("python-tk@${python_minor}")
+ fi
if ((${#missing[@]})); then
brew install "${missing[@]}"
fi
+ if [[ "${CUA_E2E_INTERNAL_LANE}" == capture ]]; then
+ python3 -c 'import tkinter; print("Tk", tkinter.TkVersion)'
+ fi
- name: Run hosted macOS lane
shell: bash
diff --git a/libs/cua-driver/docs/perception-extension.md b/libs/cua-driver/docs/perception-extension.md
index c81c5eb176..d13db0384c 100644
--- a/libs/cua-driver/docs/perception-extension.md
+++ b/libs/cua-driver/docs/perception-extension.md
@@ -122,6 +122,11 @@ runs `.github/workflows/cd-cua-perception.yml`, which:
Pull requests that change this pipeline run steps 1 through 3 plus an unsigned
package and a developer-only install check. They never sign or publish.
+After publication, the capture lane of the canonical macOS, Windows, and Linux
+X11 desktop E2E installs a pinned published catalog as `publisher-verified`
+and runs the full capture-bound loop below against a visual-only fixture. The
+row is described in [the test matrix](test-matrix.md).
+
Inspect a replacement before running `extension update`. Remove only the
extension-owned installation with `extension remove cua-perception`. Neither
operation changes the default Driver or an external `cua-som` installation.
diff --git a/libs/cua-driver/docs/test-harnesses-guide.md b/libs/cua-driver/docs/test-harnesses-guide.md
index a839ff4f9b..99fd714661 100644
--- a/libs/cua-driver/docs/test-harnesses-guide.md
+++ b/libs/cua-driver/docs/test-harnesses-guide.md
@@ -646,7 +646,7 @@ Perception tests have four owners, and each tests different code:
| `libs/cua-driver/rust/crates/cua-perception/scripts/tests/` | Model artifact tooling and quality measurement scripts | `ci-cua-perception-release.yml` |
| `libs/cua-driver/tests/perception-demo/` | Visual demo evidence sanitizing, caching, and envelopes | `ci-test-scripts.yml` |
| `.github/scripts/tests/test_cua_perception_*.py`, `test_perception_release.py` | Perception release, review-trigger, and review-pipeline workflows and `.github/scripts` helpers | `ci-test-scripts.yml`, `ci-cua-perception-release.yml` |
-| `libs/cua-driver/rust/crates/cua-driver-e2e/tests/perception_capture_loop_test.rs` | Driver's desktop capture, parse, capture-bound click, and reobserve loop with a deterministic developer-only worker | Capture lane of the canonical macOS, Windows, and Linux X11 desktop E2E |
+| `libs/cua-driver/rust/crates/cua-driver-e2e/tests/perception_capture_loop_test.rs` | Driver's desktop capture, parse, capture-bound click, and reobserve loop, with a deterministic developer-only worker and with the published extension installed from its signed release catalog | Capture lane of the canonical macOS, Windows, and Linux X11 desktop E2E |
| `libs/cua-driver/rust/crates/cua-driver-e2e/tests/perception_s1_decision_loop_test.rs` | The same loop with the published extension (OmniParser) and a resident Cua-S1-4B choosing the region, plus consumed and expired capture refusals | `s1-perception` lane of the Linux X11 runner, selected only by `ci-cua-s1-weights.yml` |
Add a perception test to the directory that owns the code it exercises.
diff --git a/libs/cua-driver/docs/test-matrix.md b/libs/cua-driver/docs/test-matrix.md
index e8dd7cc538..17777dcb38 100644
--- a/libs/cua-driver/docs/test-matrix.md
+++ b/libs/cua-driver/docs/test-matrix.md
@@ -102,23 +102,36 @@ cells is always an error.
## Harness E2E: Optional Perception Loop
-`perception_capture_loop_test.rs` runs the documented capture-bound loop
-against the shared web harness in Electron in the capture lane of the macOS,
-Windows, and Linux X11 runners:
+`perception_capture_loop_test.rs` runs the documented capture-bound loop in
+the capture lane of the macOS, Windows, and Linux X11 runners:
| Cell | Coverage |
| --- | --- |
| `<os>-electron-parse-visual-regions-not-installed-not-applicable-not-applicable` | A retained window capture parses to the typed `not_installed` refusal when no extension is installed |
| `<os>-electron-perception-capture-click-px-foreground` | `get_window_state` capture, `parse_visual_regions` on that capture, one foreground `click` with the same `capture_id` at the parsed region center, fixture-journal delivery oracle, `capture_not_found` refusal for the consumed capture, then a fresh capture and reparse |
+| `<os>-tk-perception-published-capture-click-px-foreground` | The published `cua-perception` extension, installed from its signed release catalog as `publisher-verified`, parses a capture of the visual-only Tk canvas. One foreground capture-bound `click` on the OCR region that reads "Cancel" selects that card in the fixture journal. The consumed capture is refused with `capture_not_found`, a fresh capture reparses, and an unused capture older than 60 seconds is refused with `capture_expired`. Neither refusal changes the fixture |
-The installed extension is a developer-only unsigned worker compiled from
+The Electron rows use a developer-only unsigned worker compiled from
`tests/support/perception_swatch_worker.rs`. It speaks the real worker
protocol inside Driver's containment, echoes the retained PNG digest, and finds
the harness's solid `#drag-source` and `#drop-target` swatches in the captured
-pixels, so the row needs no published artifact or model. macOS runs a second
-instance of the installed app with an isolated extension home and socket.
-Wayland lanes record a coverage limitation instead of running the row. Model
-quality belongs to the `cua-perception` crate and its artifact-gated lanes.
+pixels, so those rows need no published artifact or model.
+
+The published-catalog row pins the `cua-perception-v0.2.1` catalog and archive
+SHA-256 for each target and downloads them (about 425 MB) unless
+`CUA_E2E_PERCEPTION_CACHE_DIR` already holds verified copies. The Driver also
+checks the catalog signature. The canvas fixture needs a Python with Tk:
+`python3` on macOS and Linux, and `py -3` or `python` on Windows.
+`CUA_E2E_TK_PYTHON` overrides the search. The hosted macOS capture lane adds
+the matching Homebrew `python-tk` formula, and the Linux capture lane installs
+`python3-tk` and `x11-utils`. The row fails rather than skips when Tk is
+missing.
+
+macOS runs a second instance of the installed app with an isolated extension
+home and socket, so the shared daemon never gains either extension. Wayland
+lanes record a coverage limitation instead of running these rows. Broader
+model quality belongs to the `cua-perception` crate and its artifact-gated
+lanes.
## Harness E2E: Model-Backed Perception Decision Loop
diff --git a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/README.md b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/README.md
index 4eb35942b7..18e422129d 100644
--- a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/README.md
+++ b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/README.md
@@ -29,7 +29,7 @@ action, addressing mode, delivery mode, or OS-specific window-system case.
| `cross_platform_behavior_test.rs` | Shared Electron/Tauri/WKWebView action matrix |
| `desktop_scope_<os>_test.rs` | Platform window/desktop scope contract |
| `installed_app_*_test.rs`, `standalone_browser_behavior_test.rs` | Installed apps and browsers |
-| `perception_capture_loop_test.rs` | Optional perception capture, parse, capture-bound click, and reobserve loop |
+| `perception_capture_loop_test.rs` | Optional perception capture, parse, capture-bound click, and reobserve loop, including the published signed extension |
| `*_<os>_test.rs`, `hyprland_*_test.rs`, `wayland_*_test.rs` | Platform- or compositor-specific desktop rows |
## Harness Requirements
diff --git a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/perception_capture_loop_test.rs b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/perception_capture_loop_test.rs
index 78b94b396c..776f2ef94b 100644
--- a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/perception_capture_loop_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/perception_capture_loop_test.rs
@@ -2,19 +2,27 @@
//!
//! The documented loop is `get_window_state` (retain `capture_id`) ->
//! `parse_visual_regions` -> one capture-bound `click` with the same
-//! `capture_id` -> reobserve. These rows drive it against the shared web
-//! harness in Electron, which every supported desktop runner builds.
+//! `capture_id` -> reobserve. Two kinds of row drive it.
//!
-//! No published artifact or model is required. The installed extension is a
-//! deterministic, developer-only unsigned worker compiled from
-//! `support/perception_swatch_worker.rs`. It speaks the real framed worker
-//! protocol, runs inside Driver's normal worker containment, receives the exact
-//! PNG Driver retained for the capture, and derives regions from those pixels by
-//! finding the harness's solid color swatches. Model quality is covered by the
-//! cua-perception crate; these rows certify Driver's capture, parse, action
-//! binding, and single-use refusal contract on each desktop.
+//! The contract rows run against the shared web harness in Electron, which
+//! every supported desktop runner builds, and need no published artifact or
+//! model. Their installed extension is a deterministic, developer-only unsigned
+//! worker compiled from `support/perception_swatch_worker.rs`. It speaks the
+//! real framed worker protocol, runs inside Driver's normal worker containment,
+//! receives the exact PNG Driver retained for the capture, and derives regions
+//! from those pixels by finding the harness's solid color swatches. These rows
+//! certify Driver's capture, parse, action binding, and single-use refusal
+//! contract on each desktop.
//!
-//! The fixture's loopback journal is the delivery oracle, independent of the
+//! The published-catalog row installs the released `cua-perception` extension
+//! from its signed `cua-perception-v<version>` release catalog, the documented
+//! user path, and requires `publisher-verified` trust. The released OmniParser
+//! model reads the "Cancel" label on the visual-only Tk canvas fixture, which
+//! exposes no accessibility tree, and one capture-bound click selects that
+//! card. The row downloads the pinned release assets (about 425 MB) and needs
+//! a Python with Tk.
+//!
+//! Each fixture's loopback journal is the delivery oracle, independent of the
//! Driver response. macOS runs a dedicated instance of the installed,
//! TCC-authorized app with an isolated extension home, so the shared daemon's
//! state is never changed.
@@ -814,6 +822,597 @@ fn capture_bound_click_from_parsed_region_is_state_verified_and_single_use() {
});
}
+// ---------------------------------------------------------------- published
+
+/// The signed release the published-catalog row installs. The Driver verifies
+/// the catalog signature and archive digest itself; these pins also stop the
+/// row from silently certifying a replaced release asset.
+const PUBLISHED_VERSION: &str = "0.2.1";
+/// `(target, catalog SHA-256, archive SHA-256)` from the release `SHA256SUMS`.
+const PUBLISHED_ASSETS: &[(&str, &str, &str)] = &[
+ (
+ "aarch64-apple-darwin",
+ "d20c1e1cbf5d90cfa85b956100846c2c7a43c387fdc5b344d75a15f91498aa42",
+ "5fbcf59c15fc5cd8beca6ac4aa5533c34359054810aceb12c9e26d7a45e750ab",
+ ),
+ (
+ "x86_64-pc-windows-msvc",
+ "bfb2300ff9da054d522e73351f7b337ba6157ebbe30f029a95a471fc3d13a042",
+ "6709579aa3f938e8200c0a51b1abe450d4fda51b09e97517e0cb4c272b5bff37",
+ ),
+ (
+ "x86_64-unknown-linux-gnu",
+ "853759659a7a9d77aa246bc6f083c6e939a9ce92899f1acd55c80e9345e1070b",
+ "b4d76b626df9ce1a48b8036f7313412525295d0cb818d82ecbe3d9cca5fd1d08",
+ ),
+];
+/// Driver's capture lifetime (`CaptureRegistryConfig::default().ttl`).
+const CAPTURE_TTL: Duration = Duration::from_secs(60);
+/// The canvas card the row clicks. "Save" is read unreliably by OCR on 1x
+/// macOS displays; "Cancel" reads on every certified desktop.
+const CANVAS_TARGET_LABEL: &str = "Cancel";
+const CANVAS_TARGET_ID: &str = "cancel";
+
+fn file_sha256(path: &Path) -> Option<String> {
+ use std::io::Read as _;
+ let mut file = fs::File::open(path).ok()?;
+ let mut hasher = Sha256::new();
+ let mut buffer = vec![0_u8; 1 << 20];
+ loop {
+ let read = file.read(&mut buffer).ok()?;
+ if read == 0 {
+ break;
+ }
+ hasher.update(&buffer[..read]);
+ }
+ Some(format!("{:x}", hasher.finalize()))
+}
+
+/// Download one release asset unless a copy with the pinned digest is cached.
+fn fetch_release_asset(directory: &Path, name: &str, expected_sha256: &str) {
+ let path = directory.join(name);
+ if file_sha256(&path).as_deref() == Some(expected_sha256) {
+ eprintln!("[perception-published] cached {name}");
+ return;
+ }
+ let url = format!(
+ "https://github.com/trycua/cua/releases/download/cua-perception-v{PUBLISHED_VERSION}/{name}"
+ );
+ let partial = directory.join(format!("{name}.partial"));
+ let started = Instant::now();
+ let status = Command::new("curl")
+ .args([
+ "--fail",
+ "--silent",
+ "--show-error",
+ "--location",
+ "--retry",
+ "4",
+ "--retry-all-errors",
+ "--output",
+ ])
+ .arg(&partial)
+ .arg(&url)
+ .stdin(Stdio::null())
+ .status()
+ .expect("run curl to download the published cua-perception release");
+ assert!(status.success(), "downloading {url} failed: {status}");
+ let actual = file_sha256(&partial).expect("hash downloaded release asset");
+ assert_eq!(
+ actual, expected_sha256,
+ "{name} from the cua-perception-v{PUBLISHED_VERSION} release does not match its pinned SHA-256"
+ );
+ fs::rename(&partial, &path).expect("move verified release asset into place");
+ eprintln!(
+ "[perception-published] downloaded {name} in {:.1}s",
+ started.elapsed().as_secs_f64()
+ );
+}
+
+/// Directory holding this target's published catalog next to its archive, the
+/// layout `extension install --catalog` expects. `CUA_E2E_PERCEPTION_CACHE_DIR`
+/// lets a runner keep the ~425 MB archive between rows.
+fn published_catalog() -> PathBuf {
+ let target = current_target();
+ let (_, catalog_sha256, archive_sha256) = PUBLISHED_ASSETS
+ .iter()
+ .find(|(candidate, _, _)| *candidate == target)
+ .unwrap_or_else(|| {
+ panic!("cua-perception-v{PUBLISHED_VERSION} publishes no archive for {target}")
+ });
+ let directory = std::env::var_os("CUA_E2E_PERCEPTION_CACHE_DIR")
+ .map(PathBuf::from)
+ .unwrap_or_else(|| {
+ std::env::temp_dir().join(format!("cua-perception-release-{PUBLISHED_VERSION}"))
+ });
+ fs::create_dir_all(&directory).expect("create published release cache");
+ let stem = format!("cua-perception-{PUBLISHED_VERSION}-{target}");
+ fetch_release_asset(&directory, &format!("{stem}.tar.gz"), archive_sha256);
+ let catalog = format!("{stem}.catalog.json");
+ fetch_release_asset(&directory, &catalog, catalog_sha256);
+ directory.join(catalog)
+}
+
+/// Install the published extension through the documented user path and
+/// require that Driver reports it as publisher-verified and healthy.
+fn install_published_extension(binary: &Path, home: &Path) -> Value {
+ let catalog = published_catalog();
+ let catalog = catalog.to_str().expect("UTF-8 catalog path");
+ let install = run_extension_command(
+ binary,
+ home,
+ &[
+ "extension",
+ "install",
+ "cua-perception",
+ "--catalog",
+ catalog,
+ ],
+ );
+ assert!(
+ install.status.success(),
+ "published extension install failed: stdout={} stderr={}",
+ String::from_utf8_lossy(&install.stdout),
+ String::from_utf8_lossy(&install.stderr)
+ );
+ let status = run_extension_command(
+ binary,
+ home,
+ &[
+ "extension",
+ "status",
+ "cua-perception",
+ "--self-test",
+ "--json",
+ ],
+ );
+ assert!(
+ status.status.success(),
+ "extension status failed: {}",
+ String::from_utf8_lossy(&status.stderr)
+ );
+ let status: Value = serde_json::from_slice(&status.stdout).expect("extension status JSON");
+ assert_eq!(status["installed"], true, "extension status: {status}");
+ assert_eq!(status["healthy"], true, "extension status: {status}");
+ assert_eq!(
+ status["trust"], "publisher-verified",
+ "extension status: {status}"
+ );
+ assert_eq!(
+ status["active_version"], PUBLISHED_VERSION,
+ "extension status: {status}"
+ );
+ status
+}
+
+fn canvas_fixture_path() -> PathBuf {
+ Path::new(env!("CARGO_MANIFEST_DIR"))
+ .join("../../../tests/fixtures/apps/cross-platform/visual-only-canvas/main.py")
+}
+
+/// The first Python that can import Tk. `CUA_E2E_TK_PYTHON` overrides the
+/// search; otherwise Windows tries the `py` launcher and then `python`.
+fn tk_python() -> Vec<String> {
+ let mut candidates: Vec<Vec<String>> = Vec::new();
+ if let Some(python) = std::env::var_os("CUA_E2E_TK_PYTHON") {
+ candidates.push(vec![python.to_string_lossy().into_owned()]);
+ } else if cfg!(windows) {
+ candidates.push(vec!["py".into(), "-3".into()]);
+ candidates.push(vec!["python".into()]);
+ } else {
+ candidates.push(vec!["python3".into()]);
+ }
+ let mut failures = Vec::new();
+ for candidate in &candidates {
+ let probe = Command::new(&candidate[0])
+ .args(&candidate[1..])
+ .args(["-c", "import tkinter; print(tkinter.TkVersion)"])
+ .stdin(Stdio::null())
+ .output();
+ match probe {
+ Ok(output) if output.status.success() => {
+ eprintln!(
+ "[perception-published] Tk {} from {candidate:?}",
+ String::from_utf8_lossy(&output.stdout).trim()
+ );
+ return candidate.clone();
+ }
+ Ok(output) => failures.push(format!(
+ "{candidate:?}: {}",
+ String::from_utf8_lossy(&output.stderr).trim()
+ )),
+ Err(error) => failures.push(format!("{candidate:?}: {error}")),
+ }
+ }
+ panic!(
+ "the published-catalog row needs Python with Tk for the visual-only canvas fixture \
+ (provision it or set CUA_E2E_TK_PYTHON): {failures:?}"
+ );
+}
+
+struct CanvasFixture {
+ pid: u32,
+ window_id: u64,
+ journal: FixtureJournal,
+}
+
+fn launch_canvas(driver: &mut McpDriver, label: &str) -> CanvasFixture {
+ let python = tk_python();
+ let journal = FixtureJournal::start();
+ let title = format!("Cua Visual-Only Canvas [{label}]");
+ let mut command = Command::new(&python[0]);
+ command
+ .args(&python[1..])
+ .arg(canvas_fixture_path())
+ .args(["--journal-url", journal.url(), "--title", &title])
+ .stdin(Stdio::null())
+ .stdout(Stdio::null())
+ .stderr(Stdio::inherit());
+ let child = spawn_in_job(&mut command).expect("launch the visual-only canvas fixture");
+ driver.reaper().push(child);
+ let deadline = Instant::now() + Duration::from_secs(30);
+ while journal.snapshot()["ready"].as_bool() != Some(true) {
+ assert!(
+ Instant::now() < deadline,
+ "the canvas fixture never published readiness"
+ );
+ thread::sleep(Duration::from_millis(100));
+ }
+ let state = journal.snapshot();
+ let pid = state["pid"]
+ .as_u64()
+ .filter(|pid| *pid > 0)
+ .unwrap_or_else(|| panic!("canvas fixture journal has no pid: {state}"))
+ as u32;
+ driver.reaper().track_pid(pid);
+ assert!(
+ state["selected"].is_null() && state["action_count"] == 0,
+ "canvas fixture did not start idle: {state}"
+ );
+ let (window_id, _) = driver
+ .find_window(pid as i64, &title)
+ .unwrap_or_else(|| panic!("the exact canvas fixture window for pid {pid} did not appear"));
+ CanvasFixture {
+ pid,
+ window_id,
+ journal,
+ }
+}
+
+fn capture_canvas(driver: &mut McpDriver, fixture: &CanvasFixture) -> (String, Instant) {
+ let state = driver.call(
+ "get_window_state",
+ json!({"pid": fixture.pid as i64, "window_id": fixture.window_id}),
+ );
+ let captured_at = Instant::now();
+ assert!(
+ !state.is_error(),
+ "get_window_state failed: {}",
+ state.text()
+ );
+ let capture_id = state.structured()["capture_id"]
+ .as_str()
+ .unwrap_or_else(|| panic!("window observation omitted capture_id: {}", state.text()))
+ .to_owned();
+ (capture_id, captured_at)
+}
+
+fn parse_text_regions(driver: &mut McpDriver, capture_id: &str) -> ToolResponse {
+ driver.call(
+ "parse_visual_regions",
+ json!({
+ "capture_id": capture_id,
+ "options": {"kinds": ["text"], "min_confidence": 0.3, "max_regions": 100}
+ }),
+ )
+}
+
+/// Validate a model-backed parse against its capture and the published parser.
+fn validated_published_parse(
+ parsed: &ToolResponse,
+ fixture: &CanvasFixture,
+ capture_id: &str,
+) -> Value {
+ assert!(
+ !parsed.is_error(),
+ "parse_visual_regions failed: {}; structured={}",
+ parsed.text(),
+ parsed.structured()
+ );
+ let result = parsed.structured().clone();
+ assert_eq!(result["schema"], "cua.visual_regions_v1");
+ assert_eq!(result["capture"]["capture_id"], capture_id);
+ assert_eq!(result["capture"]["source"]["kind"], "window");
+ assert_eq!(result["capture"]["source"]["pid"], fixture.pid);
+ assert_eq!(result["capture"]["source"]["window_id"], fixture.window_id);
+ assert_eq!(result["parser"]["extension_id"], "cua-perception");
+ assert_eq!(result["parser"]["extension_version"], PUBLISHED_VERSION);
+ assert_ne!(
+ result["parser"]["backend"], "deterministic_fixture",
+ "the published row must run the released model backend: {}",
+ result["parser"]
+ );
+ assert_eq!(
+ result["capture"]["action_coordinate_space"]["kind"], "screenshot_pixels",
+ "region centers are clicked in screenshot pixels: {}",
+ result["capture"]
+ );
+ result
+}
+
+fn normalized_label(text: &str) -> String {
+ text.chars()
+ .filter(|c| c.is_alphanumeric())
+ .flat_map(char::to_lowercase)
+ .collect()
+}
+
+/// Text regions whose OCR reads exactly the target label.
+fn label_regions<'a>(parse: &'a Value, label: &str) -> Vec<&'a Value> {
+ let wanted = normalized_label(label);
+ parse["regions"]
+ .as_array()
+ .into_iter()
+ .flatten()
+ .filter(|region| region["kind"] == "text")
+ .filter(|region| {
+ region["text"]
+ .as_str()
+ .is_some_and(|text| normalized_label(text) == wanted)
+ })
+ .collect()
+}
+
+fn region_texts(parse: &Value) -> Vec<String> {
+ parse["regions"]
+ .as_array()
+ .into_iter()
+ .flatten()
+ .filter_map(|region| region["text"].as_str().map(str::to_owned))
+ .collect()
+}
+
+/// Screenshot-pixel center of a region that must lie inside its capture.
+fn region_center(parse: &Value, region: &Value) -> (f64, f64) {
+ let width = parse["capture"]["screenshot"]["width"]
+ .as_f64()
+ .expect("screenshot width");
+ let height = parse["capture"]["screenshot"]["height"]
+ .as_f64()
+ .expect("screenshot height");
+ let bounds = &region["bounds"];
+ let value = |key: &str| bounds[key].as_f64().expect("numeric region bounds");
+ let (x, y, w, h) = (value("x"), value("y"), value("width"), value("height"));
+ assert!(
+ w > 0.0 && h > 0.0 && x >= 0.0 && y >= 0.0 && x + w <= width && y + h <= height,
+ "region lies outside its {width}x{height} capture: {region}"
+ );
+ (x + w / 2.0, y + h / 2.0)
+}
+
+fn canvas_click_args(fixture: &CanvasFixture, x: f64, y: f64, capture_id: &str) -> Value {
+ json!({
+ "pid": fixture.pid as i64,
+ "window_id": fixture.window_id,
+ "x": x,
+ "y": y,
+ "capture_id": capture_id,
+ "delivery_mode": "foreground"
+ })
+}
+
+fn assert_capture_refused(response: &ToolResponse, code: &str, what: &str) {
+ assert!(
+ response.is_error(),
+ "{what} authorized a click: {}",
+ response.text()
+ );
+ assert_eq!(
+ response.structured()["code"],
+ code,
+ "{what} used an unexpected refusal: {}",
+ response.structured()
+ );
+ assert_eq!(response.structured()["effect"], "refused", "{what}");
+}
+
+fn wait_for_canvas_selection(journal: &FixtureJournal) -> bool {
+ let deadline = Instant::now() + Duration::from_secs(10);
+ loop {
+ let state = journal.snapshot();
+ if state["selected"] == CANVAS_TARGET_ID && state["action_count"] == 1 {
+ return true;
+ }
+ if Instant::now() >= deadline {
+ return false;
+ }
+ thread::sleep(Duration::from_millis(50));
+ }
+}
+
+fn assert_canvas_selected_once(journal: &FixtureJournal, after: &str) {
+ thread::sleep(Duration::from_millis(750));
+ let state = journal.snapshot();
+ assert!(
+ state["selected"] == CANVAS_TARGET_ID && state["action_count"] == 1,
+ "{after} changed the fixture: {state}"
+ );
+}
+
+fn write_published_evidence(cell_id: &str, value: &Value) {
+ let Some(root) = std::env::var_os("CUA_E2E_RESULTS_FILE")
+ .map(PathBuf::from)
+ .and_then(|results| results.parent().map(Path::to_path_buf))
+ else {
+ return;
+ };
+ let directory = root.join("perception-published");
+ fs::create_dir_all(&directory).expect("create published-catalog evidence directory");
+ let path = directory.join(format!("{cell_id}.json"));
+ fs::write(&path, serde_json::to_vec_pretty(value).unwrap())
+ .unwrap_or_else(|error| panic!("write {path:?}: {error}"));
+}
+
+/// The released extension, installed from its signed catalog, parses a live
+/// capture of a custom-painted surface with no accessibility tree. One
+/// capture-bound click on the OCR'd "Cancel" label selects that card in the
+/// fixture's own journal. The consumed capture and an unused expired capture
+/// are refused without effect, and a fresh capture reparses.
+#[test]
+#[ignore]
+fn published_extension_parses_canvas_and_capture_bound_click_is_state_verified() {
+ let route = shared_web_route(
+ Platform::current(),
+ DisplayServer::current(),
+ "left_click",
+ Targeting::Px,
+ Delivery::Foreground,
+ )
+ .expect("foreground pixel click route");
+ let case = CaseSpec::delivered(
+ format!(
+ "{}-tk-perception-published-capture-click-px-foreground",
+ std::env::consts::OS
+ ),
+ "visual-only-canvas",
+ "tk",
+ "perception_published_capture_click",
+ Targeting::Px,
+ Delivery::Foreground,
+ Scope::Window,
+ route,
+ vec![OracleKind::FixtureState, OracleKind::Protocol],
+ );
+ let cell_id = case.cell_id.clone();
+ execute_case(case, |evidence| {
+ let home = extension_home();
+ let status = install_published_extension(&lifecycle_binary(), home.path());
+
+ let mut session = PerceptionDriver::start(&cell_id, home.path());
+ let driver = &mut session.driver;
+ *evidence = recording_evidence(driver.recording_dir());
+ let fixture = launch_canvas(driver, &cell_id);
+ let raised = driver.call(
+ "bring_to_front",
+ json!({"pid": fixture.pid as i64, "window_id": fixture.window_id}),
+ );
+ eprintln!("[perception-published] bring_to_front: {}", raised.text());
+ thread::sleep(Duration::from_millis(500));
+
+ // Start the worker on a capture that is never clicked, so it can
+ // prove expiry at the end of the row.
+ let (expiring_capture, expiring_at) = capture_canvas(driver, &fixture);
+ let warm_started = Instant::now();
+ let warm = parse_text_regions(driver, &expiring_capture);
+ validated_published_parse(&warm, &fixture, &expiring_capture);
+ let warm_parse_ms = warm_started.elapsed().as_millis() as u64;
+ driver.start_behavior_recording();
+
+ // 1. Observe and retain the native capture.
+ let (capture_id, captured_at) = capture_canvas(driver, &fixture);
+ // 2. The released model parses that exact capture.
+ let parse_started = Instant::now();
+ let parsed = parse_text_regions(driver, &capture_id);
+ let parse_ms = parse_started.elapsed().as_millis() as u64;
+ let parse = validated_published_parse(&parsed, &fixture, &capture_id);
+ let matches = label_regions(&parse, CANVAS_TARGET_LABEL);
+ assert_eq!(
+ matches.len(),
+ 1,
+ "expected exactly one OCR region reading {CANVAS_TARGET_LABEL:?}; read {:?}",
+ region_texts(&parse)
+ );
+ let region = matches[0].clone();
+ let (x, y) = region_center(&parse, &region);
+ let state = fixture.journal.snapshot();
+ assert!(
+ state["selected"].is_null() && state["action_count"] == 0,
+ "parsing must not change fixture state: {state}"
+ );
+
+ // 3. One capture-bound click derived from the parsed region.
+ let click_age = captured_at.elapsed();
+ assert!(
+ click_age < CAPTURE_TTL,
+ "the loop took {click_age:?}, beyond the {CAPTURE_TTL:?} capture lifetime"
+ );
+ let args = canvas_click_args(&fixture, x, y, &capture_id);
+ let click = driver.call("click", args.clone());
+ assert!(
+ !click.is_error(),
+ "capture-bound click failed: {}; structured={}",
+ click.text(),
+ click.structured()
+ );
+ // 4. The fixture's own journal, which Driver never reads.
+ let delivered = wait_for_canvas_selection(&fixture.journal);
+ let summary = json!({
+ "cell_id": cell_id,
+ "extension": {
+ "version": status["active_version"],
+ "trust": status["trust"],
+ "publisher_key_id": status["publisher_key_id"],
+ "catalog_version": status["catalog_version"],
+ },
+ "parser": parse["parser"],
+ "screenshot": parse["capture"]["screenshot"],
+ "regions_parsed": parse["regions"].as_array().map(Vec::len),
+ "texts": region_texts(&parse),
+ "target_region": region,
+ "click_point": {"x": x, "y": y},
+ "warm_parse_ms": warm_parse_ms,
+ "parse_ms": parse_ms,
+ "capture_age_at_click_ms": click_age.as_millis() as u64,
+ "click_route": click.action_route(),
+ "click_effect": click.action_effect(),
+ "fixture_state": fixture.journal.snapshot(),
+ });
+ write_published_evidence(&cell_id, &summary);
+ eprintln!("[perception-published] {summary}");
+ assert!(
+ delivered,
+ "the click on OCR region {CANVAS_TARGET_LABEL:?} at ({x:.0},{y:.0}) never selected \
+ the card: {}",
+ fixture.journal.snapshot()
+ );
+
+ // 5. The consumed capture is refused.
+ let reused = driver.call("click", args);
+ assert_capture_refused(&reused, "capture_not_found", "a consumed capture");
+ assert_canvas_selected_once(&fixture.journal, "a refused consumed capture");
+
+ // 6. Reobserve: a fresh capture parses under its own identity and
+ // still shows the target.
+ let (fresh_capture_id, _) = capture_canvas(driver, &fixture);
+ assert_ne!(fresh_capture_id, capture_id);
+ let reparsed = parse_text_regions(driver, &fresh_capture_id);
+ let reparse = validated_published_parse(&reparsed, &fixture, &fresh_capture_id);
+ assert!(
+ !label_regions(&reparse, CANVAS_TARGET_LABEL).is_empty(),
+ "the reobserved canvas no longer reads {CANVAS_TARGET_LABEL:?}: {:?}",
+ region_texts(&reparse)
+ );
+ eprintln!(
+ "[perception-published] reobserved {fresh_capture_id}: {:?}",
+ region_texts(&reparse)
+ );
+
+ // 7. An unused capture past its lifetime is refused.
+ let remaining = CAPTURE_TTL.saturating_sub(expiring_at.elapsed());
+ thread::sleep(remaining + Duration::from_secs(2));
+ let expired = driver.call(
+ "click",
+ canvas_click_args(&fixture, x, y, &expiring_capture),
+ );
+ assert_capture_refused(&expired, "capture_expired", "an expired capture");
+ assert_canvas_selected_once(&fixture.journal, "a refused expired capture");
+ Observation::delivered(
+ vec![OracleKind::FixtureState, OracleKind::Protocol],
+ Evidence::default(),
+ )
+ });
+}
+
// ---------------------------------------------------------------- hermetic
fn png_bytes(image: image::RgbaImage) -> Vec<u8> {
@@ -983,3 +1582,36 @@ fn swatch_extension_installs_and_parses_through_driver_containment() {
json!({"x": 40, "y": 60, "width": 110, "height": 48})
);
}
+
+/// The published row clicks only an OCR region that reads exactly the target
+/// label, so the reobserved "SELECTED: CANCEL" status line never matches.
+#[test]
+fn label_regions_match_only_the_exact_ocr_label() {
+ let region = |text: &str, kind: &str| json!({"kind": kind, "text": text, "bounds": {"x": 1, "y": 1, "width": 10, "height": 10}});
+ let parse = json!({"regions": [
+ region("Save", "text"),
+ region(" cancel ", "text"),
+ region("SELECTED: CANCEL", "text"),
+ region("Cancel", "icon"),
+ region("Canceled", "text"),
+ ]});
+ let matches = label_regions(&parse, CANVAS_TARGET_LABEL);
+ assert_eq!(matches.len(), 1);
+ assert_eq!(matches[0]["text"], " cancel ");
+ assert_eq!(normalized_label("Can-cel!"), "cancel");
+}
+
+/// Every pinned release asset is a well-formed SHA-256 for a distinct target.
+#[test]
+fn published_release_pins_are_well_formed() {
+ let mut targets = std::collections::HashSet::new();
+ for (target, catalog, archive) in PUBLISHED_ASSETS {
+ assert!(targets.insert(*target), "duplicate pin for {target}");
+ for digest in [catalog, archive] {
+ assert_eq!(digest.len(), 64, "{target}: {digest}");
+ assert!(digest
+ .bytes()
+ .all(|b| b.is_ascii_hexdigit() && !b.is_ascii_uppercase()));
+ }
+ }
+}
diff --git a/scripts/ci/linux/run-rust-e2e.sh b/scripts/ci/linux/run-rust-e2e.sh
index 504c749aa0..62baaacec3 100755
--- a/scripts/ci/linux/run-rust-e2e.sh
+++ b/scripts/ci/linux/run-rust-e2e.sh
@@ -406,10 +406,11 @@ if [[ "${SUITE}" == capture || "${SUITE}" == all ]]; then
--test perception_capture_loop_test -- \
--ignored --nocapture --test-threads=1
else
- # The perception loop is certified in the canonical X11 lane. Wayland
- # window capture and pointer routes differ per compositor and are not yet
- # part of this row, so record the coverage gap instead of a vacuous pass.
- limitation="The perception capture-loop row is certified on X11; Wayland compositor lanes do not run it yet."
+ # The perception loop, including the published-catalog row, is certified
+ # in the canonical X11 lane. Wayland window capture and pointer routes
+ # differ per compositor and are not yet part of these rows, so record the
+ # coverage gap instead of a vacuous pass.
+ limitation="The perception capture-loop rows (swatch contract and published cua-perception catalog) are certified on X11; Wayland compositor lanes do not run them yet."
jq -n \
--arg reason "${limitation}" \
'{
diff --git a/scripts/ci/macos/run-rust-e2e.sh b/scripts/ci/macos/run-rust-e2e.sh
index 2abab99699..07ade2ce77 100755
--- a/scripts/ci/macos/run-rust-e2e.sh
+++ b/scripts/ci/macos/run-rust-e2e.sh
@@ -420,7 +420,9 @@ if [[ "${SUITE}" == capture || "${SUITE}" == all ]]; then
run_test desktop-scope cargo test -p cua-driver-e2e --test desktop_scope_macos_test -- \
--ignored --nocapture --test-threads=1
# Runs a dedicated instance of the installed app with an isolated extension
- # home, so the shared daemon never gains the developer-only E2E extension.
+ # home, so the shared daemon never gains the developer-only E2E extension or
+ # the published cua-perception release. The published-catalog row downloads
+ # the pinned release assets and needs python3 with tkinter.
run_test perception-capture-loop cargo test -p cua-driver-e2e \
--test perception_capture_loop_test -- \
--ignored --nocapture --test-threads=1
diff --git a/scripts/ci/windows/run-rust-e2e.ps1 b/scripts/ci/windows/run-rust-e2e.ps1
index 36fcc403ba..05f567c6ff 100644
--- a/scripts/ci/windows/run-rust-e2e.ps1
+++ b/scripts/ci/windows/run-rust-e2e.ps1
@@ -436,6 +436,8 @@ if ($suite -in @("capture", "all")) {
"test", "-p", "cua-driver-e2e", "--test", "desktop_scope_windows_test", "--",
"--ignored", "--nocapture", "--test-threads=1"
)
+ # Includes the published-catalog row: it downloads the pinned
+ # cua-perception release and needs a Python with Tk (py -3 or python).
Invoke-CargoTest "perception capture loop" @(
"test", "-p", "cua-driver-e2e", "--test", "perception_capture_loop_test", "--",
"--ignored", "--nocapture", "--test-threads=1"