Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
test(cua-driver): certify the published cua-perception catalog in canonical desktop E2E (#4322)
Francesco Bonacci4dc4cc0bd5db
diff --git a/.github/workflows/e2e-rust-linux.yml b/.github/workflows/e2e-rust-linux.ymlindex a8ac30cd5a..153b85f2da 100644--- a/.github/workflows/e2e-rust-linux.yml+++ b/.github/workflows/e2e-rust-linux.yml@@ -204,7 +204,8 @@ jobs:libgtk-3-dev clang pkg-config libdbus-1-dev libpipewire-0.3-dev \libspa-0.2-dev libei-dev libx11-dev libxi-dev libxtst-dev libxext-dev \libwebkit2gtk-4.1-dev libssl-dev libxdo-dev \- libayatana-appindicator3-dev librsvg2-dev ffmpeg+ libayatana-appindicator3-dev librsvg2-dev ffmpeg \+ python3-tk x11-utils- name: Run capture and desktop-scope contractsenv:CUA_E2E_INTERNAL_LANE: capturediff --git a/.github/workflows/e2e-rust-macos.yml b/.github/workflows/e2e-rust-macos.ymlindex ffc7bb4d3d..7593dd7578 100644--- a/.github/workflows/e2e-rust-macos.yml+++ b/.github/workflows/e2e-rust-macos.yml@@ -182,9 +182,20 @@ jobs:missing=()command -v ffmpeg >/dev/null 2>&1 || missing+=(ffmpeg)command -v jq >/dev/null 2>&1 || missing+=(jq)+ # The capture lane's published-catalog perception row drives the+ # Tk visual-only canvas fixture. Hosted Homebrew Python ships+ # without _tkinter, so add the matching python-tk formula there.+ if [[ "${CUA_E2E_INTERNAL_LANE}" == capture ]] \+ && ! python3 -c 'import tkinter' >/dev/null 2>&1; then+ python_minor="$(python3 -c 'import sys; print(f"{sys.version_info[0]}.{sys.version_info[1]}")')"+ missing+=("python-tk@${python_minor}")+ fiif ((${#missing[@]})); thenbrew install "${missing[@]}"fi+ if [[ "${CUA_E2E_INTERNAL_LANE}" == capture ]]; then+ python3 -c 'import tkinter; print("Tk", tkinter.TkVersion)'+ fi- name: Run hosted macOS laneshell: bashdiff --git a/libs/cua-driver/docs/perception-extension.md b/libs/cua-driver/docs/perception-extension.mdindex c81c5eb176..d13db0384c 100644--- a/libs/cua-driver/docs/perception-extension.md+++ b/libs/cua-driver/docs/perception-extension.md@@ -122,6 +122,11 @@ runs `.github/workflows/cd-cua-perception.yml`, which:Pull requests that change this pipeline run steps 1 through 3 plus an unsignedpackage and a developer-only install check. They never sign or publish.+After publication, the capture lane of the canonical macOS, Windows, and Linux+X11 desktop E2E installs a pinned published catalog as `publisher-verified`+and runs the full capture-bound loop below against a visual-only fixture. The+row is described in [the test matrix](test-matrix.md).+Inspect a replacement before running `extension update`. Remove only theextension-owned installation with `extension remove cua-perception`. Neitheroperation changes the default Driver or an external `cua-som` installation.diff --git a/libs/cua-driver/docs/test-harnesses-guide.md b/libs/cua-driver/docs/test-harnesses-guide.mdindex a839ff4f9b..99fd714661 100644--- a/libs/cua-driver/docs/test-harnesses-guide.md+++ b/libs/cua-driver/docs/test-harnesses-guide.md@@ -646,7 +646,7 @@ Perception tests have four owners, and each tests different code:| `libs/cua-driver/rust/crates/cua-perception/scripts/tests/` | Model artifact tooling and quality measurement scripts | `ci-cua-perception-release.yml` || `libs/cua-driver/tests/perception-demo/` | Visual demo evidence sanitizing, caching, and envelopes | `ci-test-scripts.yml` || `.github/scripts/tests/test_cua_perception_*.py`, `test_perception_release.py` | Perception release, review-trigger, and review-pipeline workflows and `.github/scripts` helpers | `ci-test-scripts.yml`, `ci-cua-perception-release.yml` |-| `libs/cua-driver/rust/crates/cua-driver-e2e/tests/perception_capture_loop_test.rs` | Driver's desktop capture, parse, capture-bound click, and reobserve loop with a deterministic developer-only worker | Capture lane of the canonical macOS, Windows, and Linux X11 desktop E2E |+| `libs/cua-driver/rust/crates/cua-driver-e2e/tests/perception_capture_loop_test.rs` | Driver's desktop capture, parse, capture-bound click, and reobserve loop, with a deterministic developer-only worker and with the published extension installed from its signed release catalog | Capture lane of the canonical macOS, Windows, and Linux X11 desktop E2E || `libs/cua-driver/rust/crates/cua-driver-e2e/tests/perception_s1_decision_loop_test.rs` | The same loop with the published extension (OmniParser) and a resident Cua-S1-4B choosing the region, plus consumed and expired capture refusals | `s1-perception` lane of the Linux X11 runner, selected only by `ci-cua-s1-weights.yml` |Add a perception test to the directory that owns the code it exercises.diff --git a/libs/cua-driver/docs/test-matrix.md b/libs/cua-driver/docs/test-matrix.mdindex e8dd7cc538..17777dcb38 100644--- a/libs/cua-driver/docs/test-matrix.md+++ b/libs/cua-driver/docs/test-matrix.md@@ -102,23 +102,36 @@ cells is always an error.## Harness E2E: Optional Perception Loop-`perception_capture_loop_test.rs` runs the documented capture-bound loop-against the shared web harness in Electron in the capture lane of the macOS,-Windows, and Linux X11 runners:+`perception_capture_loop_test.rs` runs the documented capture-bound loop in+the capture lane of the macOS, Windows, and Linux X11 runners:| Cell | Coverage || --- | --- || `<os>-electron-parse-visual-regions-not-installed-not-applicable-not-applicable` | A retained window capture parses to the typed `not_installed` refusal when no extension is installed || `<os>-electron-perception-capture-click-px-foreground` | `get_window_state` capture, `parse_visual_regions` on that capture, one foreground `click` with the same `capture_id` at the parsed region center, fixture-journal delivery oracle, `capture_not_found` refusal for the consumed capture, then a fresh capture and reparse |+| `<os>-tk-perception-published-capture-click-px-foreground` | The published `cua-perception` extension, installed from its signed release catalog as `publisher-verified`, parses a capture of the visual-only Tk canvas. One foreground capture-bound `click` on the OCR region that reads "Cancel" selects that card in the fixture journal. The consumed capture is refused with `capture_not_found`, a fresh capture reparses, and an unused capture older than 60 seconds is refused with `capture_expired`. Neither refusal changes the fixture |-The installed extension is a developer-only unsigned worker compiled from+The Electron rows use a developer-only unsigned worker compiled from`tests/support/perception_swatch_worker.rs`. It speaks the real workerprotocol inside Driver's containment, echoes the retained PNG digest, and findsthe harness's solid `#drag-source` and `#drop-target` swatches in the captured-pixels, so the row needs no published artifact or model. macOS runs a second-instance of the installed app with an isolated extension home and socket.-Wayland lanes record a coverage limitation instead of running the row. Model-quality belongs to the `cua-perception` crate and its artifact-gated lanes.+pixels, so those rows need no published artifact or model.++The published-catalog row pins the `cua-perception-v0.2.1` catalog and archive+SHA-256 for each target and downloads them (about 425 MB) unless+`CUA_E2E_PERCEPTION_CACHE_DIR` already holds verified copies. The Driver also+checks the catalog signature. The canvas fixture needs a Python with Tk:+`python3` on macOS and Linux, and `py -3` or `python` on Windows.+`CUA_E2E_TK_PYTHON` overrides the search. The hosted macOS capture lane adds+the matching Homebrew `python-tk` formula, and the Linux capture lane installs+`python3-tk` and `x11-utils`. The row fails rather than skips when Tk is+missing.++macOS runs a second instance of the installed app with an isolated extension+home and socket, so the shared daemon never gains either extension. Wayland+lanes record a coverage limitation instead of running these rows. Broader+model quality belongs to the `cua-perception` crate and its artifact-gated+lanes.## Harness E2E: Model-Backed Perception Decision Loopdiff --git a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/README.md b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/README.mdindex 4eb35942b7..18e422129d 100644--- a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/README.md+++ b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/README.md@@ -29,7 +29,7 @@ action, addressing mode, delivery mode, or OS-specific window-system case.| `cross_platform_behavior_test.rs` | Shared Electron/Tauri/WKWebView action matrix || `desktop_scope_<os>_test.rs` | Platform window/desktop scope contract || `installed_app_*_test.rs`, `standalone_browser_behavior_test.rs` | Installed apps and browsers |-| `perception_capture_loop_test.rs` | Optional perception capture, parse, capture-bound click, and reobserve loop |+| `perception_capture_loop_test.rs` | Optional perception capture, parse, capture-bound click, and reobserve loop, including the published signed extension || `*_<os>_test.rs`, `hyprland_*_test.rs`, `wayland_*_test.rs` | Platform- or compositor-specific desktop rows |## Harness Requirementsdiff --git a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/perception_capture_loop_test.rs b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/perception_capture_loop_test.rsindex 78b94b396c..776f2ef94b 100644--- a/libs/cua-driver/rust/crates/cua-driver-e2e/tests/perception_capture_loop_test.rs+++ b/libs/cua-driver/rust/crates/cua-driver-e2e/tests/perception_capture_loop_test.rs@@ -2,19 +2,27 @@//!//! The documented loop is `get_window_state` (retain `capture_id`) ->//! `parse_visual_regions` -> one capture-bound `click` with the same-//! `capture_id` -> reobserve. These rows drive it against the shared web-//! harness in Electron, which every supported desktop runner builds.+//! `capture_id` -> reobserve. Two kinds of row drive it.//!-//! No published artifact or model is required. The installed extension is a-//! deterministic, developer-only unsigned worker compiled from-//! `support/perception_swatch_worker.rs`. It speaks the real framed worker-//! protocol, runs inside Driver's normal worker containment, receives the exact-//! PNG Driver retained for the capture, and derives regions from those pixels by-//! finding the harness's solid color swatches. Model quality is covered by the-//! cua-perception crate; these rows certify Driver's capture, parse, action-//! binding, and single-use refusal contract on each desktop.+//! The contract rows run against the shared web harness in Electron, which+//! every supported desktop runner builds, and need no published artifact or+//! model. Their installed extension is a deterministic, developer-only unsigned+//! worker compiled from `support/perception_swatch_worker.rs`. It speaks the+//! real framed worker protocol, runs inside Driver's normal worker containment,+//! receives the exact PNG Driver retained for the capture, and derives regions+//! from those pixels by finding the harness's solid color swatches. These rows+//! certify Driver's capture, parse, action binding, and single-use refusal+//! contract on each desktop.//!-//! The fixture's loopback journal is the delivery oracle, independent of the+//! The published-catalog row installs the released `cua-perception` extension+//! from its signed `cua-perception-v<version>` release catalog, the documented+//! user path, and requires `publisher-verified` trust. The released OmniParser+//! model reads the "Cancel" label on the visual-only Tk canvas fixture, which+//! exposes no accessibility tree, and one capture-bound click selects that+//! card. The row downloads the pinned release assets (about 425 MB) and needs+//! a Python with Tk.+//!+//! Each fixture's loopback journal is the delivery oracle, independent of the//! Driver response. macOS runs a dedicated instance of the installed,//! TCC-authorized app with an isolated extension home, so the shared daemon's//! state is never changed.@@ -814,6 +822,597 @@ fn capture_bound_click_from_parsed_region_is_state_verified_and_single_use() {});}+// ---------------------------------------------------------------- published++/// The signed release the published-catalog row installs. The Driver verifies+/// the catalog signature and archive digest itself; these pins also stop the+/// row from silently certifying a replaced release asset.+const PUBLISHED_VERSION: &str = "0.2.1";+/// `(target, catalog SHA-256, archive SHA-256)` from the release `SHA256SUMS`.+const PUBLISHED_ASSETS: &[(&str, &str, &str)] = &[+ (+ "aarch64-apple-darwin",+ "d20c1e1cbf5d90cfa85b956100846c2c7a43c387fdc5b344d75a15f91498aa42",+ "5fbcf59c15fc5cd8beca6ac4aa5533c34359054810aceb12c9e26d7a45e750ab",+ ),+ (+ "x86_64-pc-windows-msvc",+ "bfb2300ff9da054d522e73351f7b337ba6157ebbe30f029a95a471fc3d13a042",+ "6709579aa3f938e8200c0a51b1abe450d4fda51b09e97517e0cb4c272b5bff37",+ ),+ (+ "x86_64-unknown-linux-gnu",+ "853759659a7a9d77aa246bc6f083c6e939a9ce92899f1acd55c80e9345e1070b",+ "b4d76b626df9ce1a48b8036f7313412525295d0cb818d82ecbe3d9cca5fd1d08",+ ),+];+/// Driver's capture lifetime (`CaptureRegistryConfig::default().ttl`).+const CAPTURE_TTL: Duration = Duration::from_secs(60);+/// The canvas card the row clicks. "Save" is read unreliably by OCR on 1x+/// macOS displays; "Cancel" reads on every certified desktop.+const CANVAS_TARGET_LABEL: &str = "Cancel";+const CANVAS_TARGET_ID: &str = "cancel";++fn file_sha256(path: &Path) -> Option<String> {+ use std::io::Read as _;+ let mut file = fs::File::open(path).ok()?;+ let mut hasher = Sha256::new();+ let mut buffer = vec![0_u8; 1 << 20];+ loop {+ let read = file.read(&mut buffer).ok()?;+ if read == 0 {+ break;+ }+ hasher.update(&buffer[..read]);+ }+ Some(format!("{:x}", hasher.finalize()))+}++/// Download one release asset unless a copy with the pinned digest is cached.+fn fetch_release_asset(directory: &Path, name: &str, expected_sha256: &str) {+ let path = directory.join(name);+ if file_sha256(&path).as_deref() == Some(expected_sha256) {+ eprintln!("[perception-published] cached {name}");+ return;+ }+ let url = format!(+ "https://github.com/trycua/cua/releases/download/cua-perception-v{PUBLISHED_VERSION}/{name}"+ );+ let partial = directory.join(format!("{name}.partial"));+ let started = Instant::now();+ let status = Command::new("curl")+ .args([+ "--fail",+ "--silent",+ "--show-error",+ "--location",+ "--retry",+ "4",+ "--retry-all-errors",+ "--output",+ ])+ .arg(&partial)+ .arg(&url)+ .stdin(Stdio::null())+ .status()+ .expect("run curl to download the published cua-perception release");+ assert!(status.success(), "downloading {url} failed: {status}");+ let actual = file_sha256(&partial).expect("hash downloaded release asset");+ assert_eq!(+ actual, expected_sha256,+ "{name} from the cua-perception-v{PUBLISHED_VERSION} release does not match its pinned SHA-256"+ );+ fs::rename(&partial, &path).expect("move verified release asset into place");+ eprintln!(+ "[perception-published] downloaded {name} in {:.1}s",+ started.elapsed().as_secs_f64()+ );+}++/// Directory holding this target's published catalog next to its archive, the+/// layout `extension install --catalog` expects. `CUA_E2E_PERCEPTION_CACHE_DIR`+/// lets a runner keep the ~425 MB archive between rows.+fn published_catalog() -> PathBuf {+ let target = current_target();+ let (_, catalog_sha256, archive_sha256) = PUBLISHED_ASSETS+ .iter()+ .find(|(candidate, _, _)| *candidate == target)+ .unwrap_or_else(|| {+ panic!("cua-perception-v{PUBLISHED_VERSION} publishes no archive for {target}")+ });+ let directory = std::env::var_os("CUA_E2E_PERCEPTION_CACHE_DIR")+ .map(PathBuf::from)+ .unwrap_or_else(|| {+ std::env::temp_dir().join(format!("cua-perception-release-{PUBLISHED_VERSION}"))+ });+ fs::create_dir_all(&directory).expect("create published release cache");+ let stem = format!("cua-perception-{PUBLISHED_VERSION}-{target}");+ fetch_release_asset(&directory, &format!("{stem}.tar.gz"), archive_sha256);+ let catalog = format!("{stem}.catalog.json");+ fetch_release_asset(&directory, &catalog, catalog_sha256);+ directory.join(catalog)+}++/// Install the published extension through the documented user path and+/// require that Driver reports it as publisher-verified and healthy.+fn install_published_extension(binary: &Path, home: &Path) -> Value {+ let catalog = published_catalog();+ let catalog = catalog.to_str().expect("UTF-8 catalog path");+ let install = run_extension_command(+ binary,+ home,+ &[+ "extension",+ "install",+ "cua-perception",+ "--catalog",+ catalog,+ ],+ );+ assert!(+ install.status.success(),+ "published extension install failed: stdout={} stderr={}",+ String::from_utf8_lossy(&install.stdout),+ String::from_utf8_lossy(&install.stderr)+ );+ let status = run_extension_command(+ binary,+ home,+ &[+ "extension",+ "status",+ "cua-perception",+ "--self-test",+ "--json",+ ],+ );+ assert!(+ status.status.success(),+ "extension status failed: {}",+ String::from_utf8_lossy(&status.stderr)+ );+ let status: Value = serde_json::from_slice(&status.stdout).expect("extension status JSON");+ assert_eq!(status["installed"], true, "extension status: {status}");+ assert_eq!(status["healthy"], true, "extension status: {status}");+ assert_eq!(+ status["trust"], "publisher-verified",+ "extension status: {status}"+ );+ assert_eq!(+ status["active_version"], PUBLISHED_VERSION,+ "extension status: {status}"+ );+ status+}++fn canvas_fixture_path() -> PathBuf {+ Path::new(env!("CARGO_MANIFEST_DIR"))+ .join("../../../tests/fixtures/apps/cross-platform/visual-only-canvas/main.py")+}++/// The first Python that can import Tk. `CUA_E2E_TK_PYTHON` overrides the+/// search; otherwise Windows tries the `py` launcher and then `python`.+fn tk_python() -> Vec<String> {+ let mut candidates: Vec<Vec<String>> = Vec::new();+ if let Some(python) = std::env::var_os("CUA_E2E_TK_PYTHON") {+ candidates.push(vec![python.to_string_lossy().into_owned()]);+ } else if cfg!(windows) {+ candidates.push(vec!["py".into(), "-3".into()]);+ candidates.push(vec!["python".into()]);+ } else {+ candidates.push(vec!["python3".into()]);+ }+ let mut failures = Vec::new();+ for candidate in &candidates {+ let probe = Command::new(&candidate[0])+ .args(&candidate[1..])+ .args(["-c", "import tkinter; print(tkinter.TkVersion)"])+ .stdin(Stdio::null())+ .output();+ match probe {+ Ok(output) if output.status.success() => {+ eprintln!(+ "[perception-published] Tk {} from {candidate:?}",+ String::from_utf8_lossy(&output.stdout).trim()+ );+ return candidate.clone();+ }+ Ok(output) => failures.push(format!(+ "{candidate:?}: {}",+ String::from_utf8_lossy(&output.stderr).trim()+ )),+ Err(error) => failures.push(format!("{candidate:?}: {error}")),+ }+ }+ panic!(+ "the published-catalog row needs Python with Tk for the visual-only canvas fixture \+ (provision it or set CUA_E2E_TK_PYTHON): {failures:?}"+ );+}++struct CanvasFixture {+ pid: u32,+ window_id: u64,+ journal: FixtureJournal,+}++fn launch_canvas(driver: &mut McpDriver, label: &str) -> CanvasFixture {+ let python = tk_python();+ let journal = FixtureJournal::start();+ let title = format!("Cua Visual-Only Canvas [{label}]");+ let mut command = Command::new(&python[0]);+ command+ .args(&python[1..])+ .arg(canvas_fixture_path())+ .args(["--journal-url", journal.url(), "--title", &title])+ .stdin(Stdio::null())+ .stdout(Stdio::null())+ .stderr(Stdio::inherit());+ let child = spawn_in_job(&mut command).expect("launch the visual-only canvas fixture");+ driver.reaper().push(child);+ let deadline = Instant::now() + Duration::from_secs(30);+ while journal.snapshot()["ready"].as_bool() != Some(true) {+ assert!(+ Instant::now() < deadline,+ "the canvas fixture never published readiness"+ );+ thread::sleep(Duration::from_millis(100));+ }+ let state = journal.snapshot();+ let pid = state["pid"]+ .as_u64()+ .filter(|pid| *pid > 0)+ .unwrap_or_else(|| panic!("canvas fixture journal has no pid: {state}"))+ as u32;+ driver.reaper().track_pid(pid);+ assert!(+ state["selected"].is_null() && state["action_count"] == 0,+ "canvas fixture did not start idle: {state}"+ );+ let (window_id, _) = driver+ .find_window(pid as i64, &title)+ .unwrap_or_else(|| panic!("the exact canvas fixture window for pid {pid} did not appear"));+ CanvasFixture {+ pid,+ window_id,+ journal,+ }+}++fn capture_canvas(driver: &mut McpDriver, fixture: &CanvasFixture) -> (String, Instant) {+ let state = driver.call(+ "get_window_state",+ json!({"pid": fixture.pid as i64, "window_id": fixture.window_id}),+ );+ let captured_at = Instant::now();+ assert!(+ !state.is_error(),+ "get_window_state failed: {}",+ state.text()+ );+ let capture_id = state.structured()["capture_id"]+ .as_str()+ .unwrap_or_else(|| panic!("window observation omitted capture_id: {}", state.text()))+ .to_owned();+ (capture_id, captured_at)+}++fn parse_text_regions(driver: &mut McpDriver, capture_id: &str) -> ToolResponse {+ driver.call(+ "parse_visual_regions",+ json!({+ "capture_id": capture_id,+ "options": {"kinds": ["text"], "min_confidence": 0.3, "max_regions": 100}+ }),+ )+}++/// Validate a model-backed parse against its capture and the published parser.+fn validated_published_parse(+ parsed: &ToolResponse,+ fixture: &CanvasFixture,+ capture_id: &str,+) -> Value {+ assert!(+ !parsed.is_error(),+ "parse_visual_regions failed: {}; structured={}",+ parsed.text(),+ parsed.structured()+ );+ let result = parsed.structured().clone();+ assert_eq!(result["schema"], "cua.visual_regions_v1");+ assert_eq!(result["capture"]["capture_id"], capture_id);+ assert_eq!(result["capture"]["source"]["kind"], "window");+ assert_eq!(result["capture"]["source"]["pid"], fixture.pid);+ assert_eq!(result["capture"]["source"]["window_id"], fixture.window_id);+ assert_eq!(result["parser"]["extension_id"], "cua-perception");+ assert_eq!(result["parser"]["extension_version"], PUBLISHED_VERSION);+ assert_ne!(+ result["parser"]["backend"], "deterministic_fixture",+ "the published row must run the released model backend: {}",+ result["parser"]+ );+ assert_eq!(+ result["capture"]["action_coordinate_space"]["kind"], "screenshot_pixels",+ "region centers are clicked in screenshot pixels: {}",+ result["capture"]+ );+ result+}++fn normalized_label(text: &str) -> String {+ text.chars()+ .filter(|c| c.is_alphanumeric())+ .flat_map(char::to_lowercase)+ .collect()+}++/// Text regions whose OCR reads exactly the target label.+fn label_regions<'a>(parse: &'a Value, label: &str) -> Vec<&'a Value> {+ let wanted = normalized_label(label);+ parse["regions"]+ .as_array()+ .into_iter()+ .flatten()+ .filter(|region| region["kind"] == "text")+ .filter(|region| {+ region["text"]+ .as_str()+ .is_some_and(|text| normalized_label(text) == wanted)+ })+ .collect()+}++fn region_texts(parse: &Value) -> Vec<String> {+ parse["regions"]+ .as_array()+ .into_iter()+ .flatten()+ .filter_map(|region| region["text"].as_str().map(str::to_owned))+ .collect()+}++/// Screenshot-pixel center of a region that must lie inside its capture.+fn region_center(parse: &Value, region: &Value) -> (f64, f64) {+ let width = parse["capture"]["screenshot"]["width"]+ .as_f64()+ .expect("screenshot width");+ let height = parse["capture"]["screenshot"]["height"]+ .as_f64()+ .expect("screenshot height");+ let bounds = ®ion["bounds"];+ let value = |key: &str| bounds[key].as_f64().expect("numeric region bounds");+ let (x, y, w, h) = (value("x"), value("y"), value("width"), value("height"));+ assert!(+ w > 0.0 && h > 0.0 && x >= 0.0 && y >= 0.0 && x + w <= width && y + h <= height,+ "region lies outside its {width}x{height} capture: {region}"+ );+ (x + w / 2.0, y + h / 2.0)+}++fn canvas_click_args(fixture: &CanvasFixture, x: f64, y: f64, capture_id: &str) -> Value {+ json!({+ "pid": fixture.pid as i64,+ "window_id": fixture.window_id,+ "x": x,+ "y": y,+ "capture_id": capture_id,+ "delivery_mode": "foreground"+ })+}++fn assert_capture_refused(response: &ToolResponse, code: &str, what: &str) {+ assert!(+ response.is_error(),+ "{what} authorized a click: {}",+ response.text()+ );+ assert_eq!(+ response.structured()["code"],+ code,+ "{what} used an unexpected refusal: {}",+ response.structured()+ );+ assert_eq!(response.structured()["effect"], "refused", "{what}");+}++fn wait_for_canvas_selection(journal: &FixtureJournal) -> bool {+ let deadline = Instant::now() + Duration::from_secs(10);+ loop {+ let state = journal.snapshot();+ if state["selected"] == CANVAS_TARGET_ID && state["action_count"] == 1 {+ return true;+ }+ if Instant::now() >= deadline {+ return false;+ }+ thread::sleep(Duration::from_millis(50));+ }+}++fn assert_canvas_selected_once(journal: &FixtureJournal, after: &str) {+ thread::sleep(Duration::from_millis(750));+ let state = journal.snapshot();+ assert!(+ state["selected"] == CANVAS_TARGET_ID && state["action_count"] == 1,+ "{after} changed the fixture: {state}"+ );+}++fn write_published_evidence(cell_id: &str, value: &Value) {+ let Some(root) = std::env::var_os("CUA_E2E_RESULTS_FILE")+ .map(PathBuf::from)+ .and_then(|results| results.parent().map(Path::to_path_buf))+ else {+ return;+ };+ let directory = root.join("perception-published");+ fs::create_dir_all(&directory).expect("create published-catalog evidence directory");+ let path = directory.join(format!("{cell_id}.json"));+ fs::write(&path, serde_json::to_vec_pretty(value).unwrap())+ .unwrap_or_else(|error| panic!("write {path:?}: {error}"));+}++/// The released extension, installed from its signed catalog, parses a live+/// capture of a custom-painted surface with no accessibility tree. One+/// capture-bound click on the OCR'd "Cancel" label selects that card in the+/// fixture's own journal. The consumed capture and an unused expired capture+/// are refused without effect, and a fresh capture reparses.+#[test]+#[ignore]+fn published_extension_parses_canvas_and_capture_bound_click_is_state_verified() {+ let route = shared_web_route(+ Platform::current(),+ DisplayServer::current(),+ "left_click",+ Targeting::Px,+ Delivery::Foreground,+ )+ .expect("foreground pixel click route");+ let case = CaseSpec::delivered(+ format!(+ "{}-tk-perception-published-capture-click-px-foreground",+ std::env::consts::OS+ ),+ "visual-only-canvas",+ "tk",+ "perception_published_capture_click",+ Targeting::Px,+ Delivery::Foreground,+ Scope::Window,+ route,+ vec![OracleKind::FixtureState, OracleKind::Protocol],+ );+ let cell_id = case.cell_id.clone();+ execute_case(case, |evidence| {+ let home = extension_home();+ let status = install_published_extension(&lifecycle_binary(), home.path());++ let mut session = PerceptionDriver::start(&cell_id, home.path());+ let driver = &mut session.driver;+ *evidence = recording_evidence(driver.recording_dir());+ let fixture = launch_canvas(driver, &cell_id);+ let raised = driver.call(+ "bring_to_front",+ json!({"pid": fixture.pid as i64, "window_id": fixture.window_id}),+ );+ eprintln!("[perception-published] bring_to_front: {}", raised.text());+ thread::sleep(Duration::from_millis(500));++ // Start the worker on a capture that is never clicked, so it can+ // prove expiry at the end of the row.+ let (expiring_capture, expiring_at) = capture_canvas(driver, &fixture);+ let warm_started = Instant::now();+ let warm = parse_text_regions(driver, &expiring_capture);+ validated_published_parse(&warm, &fixture, &expiring_capture);+ let warm_parse_ms = warm_started.elapsed().as_millis() as u64;+ driver.start_behavior_recording();++ // 1. Observe and retain the native capture.+ let (capture_id, captured_at) = capture_canvas(driver, &fixture);+ // 2. The released model parses that exact capture.+ let parse_started = Instant::now();+ let parsed = parse_text_regions(driver, &capture_id);+ let parse_ms = parse_started.elapsed().as_millis() as u64;+ let parse = validated_published_parse(&parsed, &fixture, &capture_id);+ let matches = label_regions(&parse, CANVAS_TARGET_LABEL);+ assert_eq!(+ matches.len(),+ 1,+ "expected exactly one OCR region reading {CANVAS_TARGET_LABEL:?}; read {:?}",+ region_texts(&parse)+ );+ let region = matches[0].clone();+ let (x, y) = region_center(&parse, ®ion);+ let state = fixture.journal.snapshot();+ assert!(+ state["selected"].is_null() && state["action_count"] == 0,+ "parsing must not change fixture state: {state}"+ );++ // 3. One capture-bound click derived from the parsed region.+ let click_age = captured_at.elapsed();+ assert!(+ click_age < CAPTURE_TTL,+ "the loop took {click_age:?}, beyond the {CAPTURE_TTL:?} capture lifetime"+ );+ let args = canvas_click_args(&fixture, x, y, &capture_id);+ let click = driver.call("click", args.clone());+ assert!(+ !click.is_error(),+ "capture-bound click failed: {}; structured={}",+ click.text(),+ click.structured()+ );+ // 4. The fixture's own journal, which Driver never reads.+ let delivered = wait_for_canvas_selection(&fixture.journal);+ let summary = json!({+ "cell_id": cell_id,+ "extension": {+ "version": status["active_version"],+ "trust": status["trust"],+ "publisher_key_id": status["publisher_key_id"],+ "catalog_version": status["catalog_version"],+ },+ "parser": parse["parser"],+ "screenshot": parse["capture"]["screenshot"],+ "regions_parsed": parse["regions"].as_array().map(Vec::len),+ "texts": region_texts(&parse),+ "target_region": region,+ "click_point": {"x": x, "y": y},+ "warm_parse_ms": warm_parse_ms,+ "parse_ms": parse_ms,+ "capture_age_at_click_ms": click_age.as_millis() as u64,+ "click_route": click.action_route(),+ "click_effect": click.action_effect(),+ "fixture_state": fixture.journal.snapshot(),+ });+ write_published_evidence(&cell_id, &summary);+ eprintln!("[perception-published] {summary}");+ assert!(+ delivered,+ "the click on OCR region {CANVAS_TARGET_LABEL:?} at ({x:.0},{y:.0}) never selected \+ the card: {}",+ fixture.journal.snapshot()+ );++ // 5. The consumed capture is refused.+ let reused = driver.call("click", args);+ assert_capture_refused(&reused, "capture_not_found", "a consumed capture");+ assert_canvas_selected_once(&fixture.journal, "a refused consumed capture");++ // 6. Reobserve: a fresh capture parses under its own identity and+ // still shows the target.+ let (fresh_capture_id, _) = capture_canvas(driver, &fixture);+ assert_ne!(fresh_capture_id, capture_id);+ let reparsed = parse_text_regions(driver, &fresh_capture_id);+ let reparse = validated_published_parse(&reparsed, &fixture, &fresh_capture_id);+ assert!(+ !label_regions(&reparse, CANVAS_TARGET_LABEL).is_empty(),+ "the reobserved canvas no longer reads {CANVAS_TARGET_LABEL:?}: {:?}",+ region_texts(&reparse)+ );+ eprintln!(+ "[perception-published] reobserved {fresh_capture_id}: {:?}",+ region_texts(&reparse)+ );++ // 7. An unused capture past its lifetime is refused.+ let remaining = CAPTURE_TTL.saturating_sub(expiring_at.elapsed());+ thread::sleep(remaining + Duration::from_secs(2));+ let expired = driver.call(+ "click",+ canvas_click_args(&fixture, x, y, &expiring_capture),+ );+ assert_capture_refused(&expired, "capture_expired", "an expired capture");+ assert_canvas_selected_once(&fixture.journal, "a refused expired capture");+ Observation::delivered(+ vec![OracleKind::FixtureState, OracleKind::Protocol],+ Evidence::default(),+ )+ });+}+// ---------------------------------------------------------------- hermeticfn png_bytes(image: image::RgbaImage) -> Vec<u8> {@@ -983,3 +1582,36 @@ fn swatch_extension_installs_and_parses_through_driver_containment() {json!({"x": 40, "y": 60, "width": 110, "height": 48}));}++/// The published row clicks only an OCR region that reads exactly the target+/// label, so the reobserved "SELECTED: CANCEL" status line never matches.+#[test]+fn label_regions_match_only_the_exact_ocr_label() {+ let region = |text: &str, kind: &str| json!({"kind": kind, "text": text, "bounds": {"x": 1, "y": 1, "width": 10, "height": 10}});+ let parse = json!({"regions": [+ region("Save", "text"),+ region(" cancel ", "text"),+ region("SELECTED: CANCEL", "text"),+ region("Cancel", "icon"),+ region("Canceled", "text"),+ ]});+ let matches = label_regions(&parse, CANVAS_TARGET_LABEL);+ assert_eq!(matches.len(), 1);+ assert_eq!(matches[0]["text"], " cancel ");+ assert_eq!(normalized_label("Can-cel!"), "cancel");+}++/// Every pinned release asset is a well-formed SHA-256 for a distinct target.+#[test]+fn published_release_pins_are_well_formed() {+ let mut targets = std::collections::HashSet::new();+ for (target, catalog, archive) in PUBLISHED_ASSETS {+ assert!(targets.insert(*target), "duplicate pin for {target}");+ for digest in [catalog, archive] {+ assert_eq!(digest.len(), 64, "{target}: {digest}");+ assert!(digest+ .bytes()+ .all(|b| b.is_ascii_hexdigit() && !b.is_ascii_uppercase()));+ }+ }+}diff --git a/scripts/ci/linux/run-rust-e2e.sh b/scripts/ci/linux/run-rust-e2e.shindex 504c749aa0..62baaacec3 100755--- a/scripts/ci/linux/run-rust-e2e.sh+++ b/scripts/ci/linux/run-rust-e2e.sh@@ -406,10 +406,11 @@ if [[ "${SUITE}" == capture || "${SUITE}" == all ]]; then--test perception_capture_loop_test -- \--ignored --nocapture --test-threads=1else- # The perception loop is certified in the canonical X11 lane. Wayland- # window capture and pointer routes differ per compositor and are not yet- # part of this row, so record the coverage gap instead of a vacuous pass.- limitation="The perception capture-loop row is certified on X11; Wayland compositor lanes do not run it yet."+ # The perception loop, including the published-catalog row, is certified+ # in the canonical X11 lane. Wayland window capture and pointer routes+ # differ per compositor and are not yet part of these rows, so record the+ # coverage gap instead of a vacuous pass.+ limitation="The perception capture-loop rows (swatch contract and published cua-perception catalog) are certified on X11; Wayland compositor lanes do not run them yet."jq -n \--arg reason "${limitation}" \'{diff --git a/scripts/ci/macos/run-rust-e2e.sh b/scripts/ci/macos/run-rust-e2e.shindex 2abab99699..07ade2ce77 100755--- a/scripts/ci/macos/run-rust-e2e.sh+++ b/scripts/ci/macos/run-rust-e2e.sh@@ -420,7 +420,9 @@ if [[ "${SUITE}" == capture || "${SUITE}" == all ]]; thenrun_test desktop-scope cargo test -p cua-driver-e2e --test desktop_scope_macos_test -- \--ignored --nocapture --test-threads=1# Runs a dedicated instance of the installed app with an isolated extension- # home, so the shared daemon never gains the developer-only E2E extension.+ # home, so the shared daemon never gains the developer-only E2E extension or+ # the published cua-perception release. The published-catalog row downloads+ # the pinned release assets and needs python3 with tkinter.run_test perception-capture-loop cargo test -p cua-driver-e2e \--test perception_capture_loop_test -- \--ignored --nocapture --test-threads=1diff --git a/scripts/ci/windows/run-rust-e2e.ps1 b/scripts/ci/windows/run-rust-e2e.ps1index 36fcc403ba..05f567c6ff 100644--- a/scripts/ci/windows/run-rust-e2e.ps1+++ b/scripts/ci/windows/run-rust-e2e.ps1@@ -436,6 +436,8 @@ if ($suite -in @("capture", "all")) {"test", "-p", "cua-driver-e2e", "--test", "desktop_scope_windows_test", "--","--ignored", "--nocapture", "--test-threads=1")+ # Includes the published-catalog row: it downloads the pinned+ # cua-perception release and needs a Python with Tk (py -3 or python).Invoke-CargoTest "perception capture loop" @("test", "-p", "cua-driver-e2e", "--test", "perception_capture_loop_test", "--","--ignored", "--nocapture", "--test-threads=1"