Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
test(cua-driver): cross-platform capture_mode matrix coverage + modality docs (#2053)
Francesco Bonacci1397eb8a1885
diff --git a/docs/content/docs/explanation/capture-and-dispatch-modalities.mdx b/docs/content/docs/explanation/capture-and-dispatch-modalities.mdxnew file mode 100644index 0000000000..1aee5f55cc--- /dev/null+++ b/docs/content/docs/explanation/capture-and-dispatch-modalities.mdx@@ -0,0 +1,73 @@+---+title: "Capture and Dispatch Modalities"+description: "The three independent axes that define how Cua Driver observes and acts on an app — capture mode, dispatch, and capture scope — and which combinations are valid."+---++# Capture and Dispatch Modalities++Every Cua Driver action is defined by three independent choices: **what the agent observes**, **how input is delivered**, and **what coordinate space the action targets**. These are the *modalities*. Most callers never set them — the defaults give background, per-window, accessibility-first automation. This page documents the full set so you can reason about the combinations that are valid and the ones the driver rejects by design.++## The Three Axes++### 1. Capture mode — what the agent observes++Set with `set_config capture_mode=…`, or per call on `get_window_state`.++| `capture_mode` | What you get | Use when |+|---|---|---|+| `som` (default) | Set-of-marks: accessibility tree **plus** a screenshot, every actionable element tagged `[element_index N]`. | General driving — the model sees both structure and pixels. |+| `ax` | Accessibility tree only, no screenshot. | The target exposes good structure and you want the cheapest, most robust path. No Screen Recording grant needed. |+| `vision` | Screenshot only, no tree. | The target has no usable accessibility tree (canvas, custom-drawn UI, pure web content). |++`ax` and `som`/`vision` differ in how the agent then *acts*: an `ax` observation yields an `element_index`, which drives the semantic action path (UIA Invoke, `AXPerformAction`, AT-SPI). A `vision` observation yields pixels, so the agent grounds a target to `(x, y)` and acts by coordinate.++### 2. Dispatch — how input is delivered++Set per call on the input tools (`click`, `type_text`, `press_key`, …) with `dispatch`.++| `dispatch` | Behavior |+|---|---|+| `background` (default) | Input is routed to the target process/window/element directly. The user's frontmost app, real cursor, and window z-order are untouched. This is the [no-foreground contract](/explanation/the-no-foreground-contract). |+| `foreground` | The target is activated first (pair with `bring_to_front`), then input lands on the now-active window. Use for apps that only accept events when foregrounded — DirectInput games, raw-input canvases. |+| `auto` | Historical heuristic that picks per target. Prefer an explicit value. |++`element_index` (AX) actions are inherently background — they address an element, not the focused window — so they hold the contract without any `dispatch` flag. The `dispatch` axis matters most for coordinate (vision) input, where `background` routes the event to the target and `foreground` raises the window first.++### 3. Capture scope — what coordinate space the action targets++Set with `set_config capture_scope=…`.++| `capture_scope` | Coordinate space | Capture surface |+|---|---|---|+| `window` (default) | Per-window. Actions carry `pid` + `window_id`; coordinates are window-relative or addressed by `element_index`. | `get_window_state` |+| `desktop` | Screen-absolute. Window-less actions (no `pid`/`window_id`) land at absolute screen coordinates via hit-testing (`WindowFromPoint`). | `get_desktop_state` — full display, no downscale |++Desktop scope is the "Computer-Use 1.0" loop: the agent reads the whole screen and clicks absolute coordinates, the way a screenshot-only model expects. Window scope is the default because it is what makes background, concurrent automation possible.++## The Validity Matrix++Capture scope is the constraining axis. Window scope supports **any** combination of the other two. Desktop scope supports **only** vision capture and foreground dispatch.++| `capture_scope` | `capture_mode` | `dispatch` | Valid? | Why |+|---|---|---|---|---|+| `window` | `ax` | `background` | ✅ | The default. Semantic actions on a backgrounded window. |+| `window` | `ax` | `foreground` | ✅ | Activate, then act by element. |+| `window` | `vision` | `background` | ✅ | Screenshot a window, click a coordinate in it without raising it. |+| `window` | `vision` | `foreground` | ✅ | Activate, then click by coordinate. |+| `desktop` | `vision` | `foreground` | ✅ | The only desktop combination. Read the whole screen, click absolute coordinates on the active desktop. |+| `desktop` | `ax` | any | ❌ | A desktop-absolute action has no `window_id`, so there is no accessibility tree to resolve an `element_index` against. |+| `desktop` | any | `background` | ❌ | Screen-absolute input hits whatever owns those pixels on the active desktop; there is no per-process route, so it cannot be backgrounded. |+| `window` | any | — | ❌ | A window-less (no `pid`/`window_id`) action while scope is `window` is rejected with the structured `desktop_scope_disabled` error. |++The two rejections are enforced, not advisory. A window-less click while `capture_scope=window` returns `desktop_scope_disabled`, pointing the caller at `set_config capture_scope=desktop`. Desktop scope inherently foregrounds and works on pixels, so it cannot honor the background contract or the accessibility path — which is exactly why background, per-window automation is the default.++## Platform Support++| Axis / value | Windows | macOS | Linux |+|---|---|---|---|+| `capture_mode` `ax` / `vision` / `som` | ✅ | ✅ | ✅ |+| `dispatch: background` (the contract) | ✅ | ✅ | ✅ (X11/AT-SPI; native Wayland input is a [known gap](/explanation/the-no-foreground-contract#linux)) |+| `dispatch: foreground` + `bring_to_front` | ✅ explicit activation | activation differs (no `bring_to_front` tool) | activation differs (stubbed) |+| `capture_scope: desktop` (full screen-absolute loop) | ✅ | rolling out | rolling out |++`capture_mode` is recognized everywhere, so window-scope automation — the four window-scope rows of the matrix — works on all three platforms. The desktop-scope loop (`get_desktop_state` plus window-less screen-absolute input via hit-testing) is complete on Windows and rolling out to macOS and Linux; on those platforms a window-less action under window scope is still rejected. See the [MCP tool reference](/reference/cua-driver/mcp-tools) for per-tool parameters and the [no-foreground contract](/explanation/the-no-foreground-contract) for how background dispatch is implemented on each OS.diff --git a/docs/content/docs/explanation/index.mdx b/docs/content/docs/explanation/index.mdxindex b581e32acc..ad8c52de14 100644--- a/docs/content/docs/explanation/index.mdx+++ b/docs/content/docs/explanation/index.mdx@@ -5,4 +5,4 @@ description: "Understand the design ideas behind Cua and how its main pieces fitExplanation is for the parts of Cua that are easier to use once the model is clear. These pages are not recipes or API listings. They describe the **concepts, constraints, and tradeoffs** behind Cua's two main ways of giving an agent a computer: driving an existing machine with Cua Driver, or running an isolated cloud desktop with Cua Sandbox. Read this section when you want to understand why the system behaves the way it does, what guarantees it is trying to preserve, and where each component sits in the stack.-The section starts with [What is computer use?](/explanation/what-is-computer-use), which defines computer-use agents and where Cua fits. [The no foreground contract](/explanation/the-no-foreground-contract) explains the central invariant in Cua Driver: agent actions must not steal focus, move the real pointer, or interrupt the person using the machine. [Process model](/explanation/process-model) describes how Cua Driver runs as an MCP stdio server, daemon, and CLI. [How sandboxes work](/explanation/how-sandboxes-work) covers cloud desktop sandboxes, their lifecycle, and snapshots. [Architecture](/explanation/architecture) ties the pieces together: Cua Driver, Cua Sandbox, and the computer-server.+The section starts with [What is computer use?](/explanation/what-is-computer-use), which defines computer-use agents and where Cua fits. [The no foreground contract](/explanation/the-no-foreground-contract) explains the central invariant in Cua Driver: agent actions must not steal focus, move the real pointer, or interrupt the person using the machine. [Capture and dispatch modalities](/explanation/capture-and-dispatch-modalities) lays out the three axes of every action — what the agent observes, how input is delivered, and the coordinate space it targets — and which combinations are valid. [Process model](/explanation/process-model) describes how Cua Driver runs as an MCP stdio server, daemon, and CLI. [How sandboxes work](/explanation/how-sandboxes-work) covers cloud desktop sandboxes, their lifecycle, and snapshots. [Architecture](/explanation/architecture) ties the pieces together: Cua Driver, Cua Sandbox, and the computer-server.diff --git a/docs/content/docs/explanation/meta.json b/docs/content/docs/explanation/meta.jsonindex 257002ad0d..a4e02a18cc 100644--- a/docs/content/docs/explanation/meta.json+++ b/docs/content/docs/explanation/meta.json@@ -1 +1 @@-{ "title": "Explanation", "icon": "Lightbulb", "pages": ["index", "what-is-computer-use", "the-no-foreground-contract", "process-model", "how-sandboxes-work", "demonstrations-skills-and-trajectories", "architecture"] }+{ "title": "Explanation", "icon": "Lightbulb", "pages": ["index", "what-is-computer-use", "the-no-foreground-contract", "capture-and-dispatch-modalities", "process-model", "how-sandboxes-work", "demonstrations-skills-and-trajectories", "architecture"] }diff --git a/docs/content/docs/reference/cua-driver/contracts.mdx b/docs/content/docs/reference/cua-driver/contracts.mdxnew file mode 100644index 0000000000..969ca829d7--- /dev/null+++ b/docs/content/docs/reference/cua-driver/contracts.mdx@@ -0,0 +1,110 @@+---+title: Interface Contracts+description: The cross-cutting contracts behind the CLI and MCP surfaces — transport state, config persistence, capture scope, and how an action is routed.+---++import { Callout } from 'fumadocs-ui/components/callout';++The CLI (`cua-driver call …`) and the MCP server (`cua-driver serve` / `mcp`) run the same tool code, but they differ in **what state survives between calls**, **where configuration lands**, and **which parameters a call must carry**. This page is the contract map. For the *why* behind the process shapes, see [Process model](/explanation/process-model); for the modality axes, see [Capture and dispatch modalities](/explanation/capture-and-dispatch-modalities).++---++## CLI versus MCP at a glance++| Dimension | CLI (`call`) | MCP (`serve` / `mcp`) |+|---|---|---|+| Lifetime | one process per action | one process, many actions |+| `element_index` cache | only when proxied to a running daemon | per connection, lives across calls |+| Identity | anonymous, unless proxied to a daemon | a minted `_session_id` per connection, or an explicit `session` |+| Where `set_config` lands | the persisted global default on disk | an in-memory, session-scoped override |+| Agent cursor | none | shown when a `session` is declared |++<Callout type="warn">+When a daemon is already listening, `cua-driver call` **proxies to it**. A freshly built binary's behavior will not appear through the CLI until that daemon restarts, because the call runs in the daemon, not in the new process. Integration tests avoid this by spawning their own MCP server.+</Callout>++---++## Where settings live++`set_config` resolves *where* a setting is written from whether a session is declared. The daemon mirrors the public `session` argument into the reserved `_session_id`.++| Caller | `_session_id` | Effect |+|---|---|---|+| `cua-driver config set …`, one-shot `cua-driver call` | absent (anonymous) | writes the global `DriverConfig` and persists to `~/.cua-driver/config.json` |+| MCP call with a `session` | present | in-memory override for that session only — no disk write, no clobber of the default |++Every tool then reads the **effective** value with this precedence:++```+effective = call-argument > session override > global default (disk)+```++Keys that flow through this: `capture_mode`, `capture_scope`, `max_image_dimension`. See the [`set_config` reference](/reference/cua-driver/mcp-tools) for the per-session isolation details.++---++## The capture-scope contract++`capture_scope` decides **which parameters a call must carry** and **what coordinate space it speaks**.++```+ capture_scope = "window" (default) capture_scope = "desktop"+ ─────────────────────────────────── ──────────────────────────────────+ per window, background-capable whole screen, foreground, vision++ observe: get_window_state(pid, window_id) observe: get_desktop_state()+ act: click(pid, …) act: click(x, y)+ element_index + window_id (AX) true screen pixels+ or x, y + pid (window-local px) no pid, no window_id+ needs: pid (+ window_id for elements) skips: window_id, list_windows+```++| | `window` (default) | `desktop` |+|---|---|---|+| Coordinate space | window-local (the PNG `get_window_state` returns) | true screen pixels |+| Required params | `pid` (+ `window_id` for `element_index`) | only `x`, `y` |+| Capture surface | `get_window_state` | `get_desktop_state` |+| Dispatch | background (default) or foreground | foreground only |+| Capture mode | `ax` / `vision` / `som` | `vision` only |++Desktop scope is the screen-absolute "Computer-Use 1.0" loop: read the whole screen, click an absolute coordinate, the way a screenshot-only model expects. Window scope is the default because it is what makes background, concurrent automation possible.++---++## How an action is routed++A `click` (and `scroll`, `type_text`) picks its path from the arguments present:++| Arguments | Path | Behavior |+|---|---|---|+| `element_index` + `window_id` | accessibility action | UIA Invoke / `AXPerformAction` / AT-SPI — background, no cursor move, no focus steal |+| `x`, `y` + `pid` | window-local pixel | coordinates are relative to that window's screenshot |+| `x`, `y`, **no** `pid`/`window_id`, scope `desktop` | screen-absolute | true screen pixels, lands on whatever is frontmost there |+| `x`, `y`, **no** `pid`/`window_id`, scope `window` | rejected | structured `desktop_scope_disabled` error |++A window-less action is **never silently reinterpreted**. Under window scope it is rejected with a structured `desktop_scope_disabled` error that points the caller at `set_config capture_scope=desktop`, rather than treating screen pixels as window-local pixels.++---++## Platform support++| Capability | Windows | macOS | Linux |+|---|---|---|---|+| `capture_mode` `ax` / `vision` / `som` | ✅ | ✅ | ✅ |+| `dispatch: background` (the [no-foreground contract](/explanation/the-no-foreground-contract)) | ✅ | ✅ | ✅ (X11/AT-SPI; native Wayland input is a gap) |+| `dispatch: foreground` + `bring_to_front` | ✅ | not needed (post-to-pid is background-safe) | stubbed |+| `get_desktop_state` (desktop capture) | ✅ | ✅ | ✅ |+| Window-less desktop click (`click{x,y}`, no `pid`) | ✅ | rolling out | rolling out |++---++## Mental model++```+WINDOW scope = "talk to a window" → needs pid (+window_id for elements); background by default+DESKTOP scope = "talk to the screen" → needs only x,y; screen-absolute; foreground; vision++CLI = stateless; writes config to DISK (or proxies to a running daemon)+MCP = stateful session; config override in MEMORY; owns the agent cursor+```diff --git a/docs/content/docs/reference/cua-driver/meta.json b/docs/content/docs/reference/cua-driver/meta.jsonindex 22be9dae07..d02b4c3f48 100644--- a/docs/content/docs/reference/cua-driver/meta.json+++ b/docs/content/docs/reference/cua-driver/meta.json@@ -1 +1 @@-{ "title": "Cua Driver", "pages": ["cli-reference", "mcp-tools", "limits"] }+{ "title": "Cua Driver", "pages": ["cli-reference", "mcp-tools", "contracts", "limits"] }diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/modality_capture_mode_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/modality_capture_mode_test.rsnew file mode 100644index 0000000000..0a211b78a6--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver/tests/modality_capture_mode_test.rs@@ -0,0 +1,259 @@+//! modality_capture_mode_test — the **capture_mode** axis (`ax` / `vision` /+//! `som`) of the modality matrix, asserted on each platform's native+//! controlled-harness app.+//!+//! This generalizes the Windows-only oracle in `modality_background_test`+//! (`capture_mode_ax_returns_tree_only` / `capture_mode_vision_returns_image_only`)+//! to macOS (AppKit) and Linux (GTK3), closing the audited gap where the+//! `capture_mode` axis was only exercised on Windows. The matrix it covers is+//! documented in `docs/content/docs/explanation/capture-and-dispatch-modalities.mdx`.+//!+//! Oracle, against the harness window:+//! * `capture_mode=ax` → accessibility tree present (`btn-increment`),+//! NO `image` content entry.+//! * `capture_mode=vision` → `image` content present, NO tree markdown.+//! * `capture_mode=som` → both (the default).+//!+//! Caveats handled as graceful skips (these are `#[ignore]` interactive tests,+//! consistent with the rest of the harness suite):+//! * Native **vision** capture needs a screen-capture grant (Screen Recording+//! on macOS). Without it the driver returns a tree but no PNG, so the image+//! assertions skip-with-note rather than false-fail.+//! * The **ax** tree needs the platform accessibility grant (TCC on macOS,+//! AT-SPI bus on Linux). An empty tree skips the tree assertion.+//!+//! Run explicitly:+//! cargo test -p cua-driver --test modality_capture_mode_test -- --ignored --nocapture --test-threads=1++#![cfg(any(target_os = "windows", target_os = "macos", target_os = "linux"))]++use std::process::{Command, Stdio};+use std::time::{Duration, Instant};++use cua_driver_testkit::{harness_app, Driver, McpDriver, ToolResponse};++/// The aid every harness exposes on its increment button — the tree marker+/// (WPF AutomationId / AppKit AX identifier / GTK3 AT-SPI accessible name).+const TREE_MARKER: &str = "btn-increment";++/// Does the response carry a screenshot? Checks both the MCP `image` content+/// entry and the canonical `screenshot_png_b64` structured field (the CLI path+/// surfaces only the latter), so the oracle is transport-agnostic.+fn has_image(resp: &ToolResponse) -> bool {+ let mcp_image = resp.raw["result"]["content"]+ .as_array()+ .map(|a| a.iter().any(|c| c["type"].as_str() == Some("image")))+ .unwrap_or(false);+ let structured_png = resp.structured()["screenshot_png_b64"]+ .as_str()+ .map(|s| !s.is_empty())+ .unwrap_or(false);+ mcp_image || structured_png+}++/// Is the accessibility tree present (the increment-button marker rendered)?+/// Checks the canonical `tree_markdown` structured field — `som` puts a short+/// summary in the MCP text block but the full tree only in `tree_markdown`, so+/// asserting on `text()` would under-report the tree in `som` mode.+fn tree_has_marker(resp: &ToolResponse) -> bool {+ resp.structured()["tree_markdown"]+ .as_str()+ .map(|t| t.contains(TREE_MARKER))+ .unwrap_or(false)+ || resp.text().contains(TREE_MARKER)+}++fn snapshot(driver: &mut McpDriver, pid: u32, wid: u64, mode: &str) -> ToolResponse {+ driver.call(+ "get_window_state",+ serde_json::json!({ "pid": pid as i64, "window_id": wid, "capture_mode": mode }),+ )+}++/// Snapshot, retrying until the accessibility tree has populated (the marker+/// appears) or the attempts are exhausted. The AX/AT-SPI tree can lag a+/// cold-launched window by several seconds: macOS AX registration is a few+/// hundred ms, but a freshly launched GTK3 harness needs the AT-SPI atk-bridge+/// to register on the a11y bus, which on a cold bus (CI cold boot) can take a+/// few seconds. The budget is ~8s so the tree-bearing modes (`ax`, `som`)+/// reliably exercise their assertions instead of degrading to a cold-start skip.+fn snapshot_settled(driver: &mut McpDriver, pid: u32, wid: u64, mode: &str) -> ToolResponse {+ let mut resp = snapshot(driver, pid, wid, mode);+ for _ in 0..16 {+ if tree_has_marker(&resp) {+ break;+ }+ std::thread::sleep(Duration::from_millis(500));+ resp = snapshot(driver, pid, wid, mode);+ }+ resp+}++// ── per-platform harness launch → (pid, window_id) ──────────────────────────────+//+// Each `launch` records the harness window pids that already exist, spawns its+// own instance, then resolves the window whose pid is NEW. This keeps the three+// tests deterministic when run back-to-back (`--test-threads=1`): a prior test's+// window can linger past its driver's teardown, and resolving by title alone+// would race onto that stale instance.++#[cfg(target_os = "windows")]+fn launch(driver: &mut McpDriver) -> Option<(u32, u64)> {+ let exe = harness_app("harness-wpf", "CuaTestHarness.Wpf.exe");+ launch_new_instance(driver, &exe, "CuaTestHarness")+}++#[cfg(target_os = "linux")]+fn launch(driver: &mut McpDriver) -> Option<(u32, u64)> {+ let exe = std::env::var("HARNESS_GTK3_EXE")+ .map(std::path::PathBuf::from)+ .ok()+ .filter(|p| p.exists())+ .unwrap_or_else(|| harness_app("harness-gtk3", "CuaTestHarness.Gtk3"));+ launch_new_instance(driver, &exe, "CuaTestHarness GTK3")+}++#[cfg(target_os = "macos")]+fn launch(driver: &mut McpDriver) -> Option<(u32, u64)> {+ let app = std::env::var("HARNESS_APPKIT_APP")+ .map(std::path::PathBuf::from)+ .ok()+ .filter(|p| p.exists())+ .unwrap_or_else(|| harness_app("harness-appkit", "CuaTestHarness.AppKit.app"));+ // Launch the binary inside the `.app` directly so the reaper owns the pid.+ let exe = app.join("Contents/MacOS/CuaTestHarness.AppKit");+ launch_new_instance(driver, &exe, "CuaTestHarness AppKit")+}++/// Spawn `exe` under the driver's reaper and resolve the window of the *new*+/// instance (its pid was not among the harness windows before the spawn).+fn launch_new_instance(driver: &mut McpDriver, exe: &std::path::Path, title: &str) -> Option<(u32, u64)> {+ if !exe.exists() {+ eprintln!("[capture_mode] harness not built ({exe:?}) — skipping");+ return None;+ }+ let before = harness_pids(driver, title);+ driver+ .reaper()+ .spawn(Command::new(exe).stdout(Stdio::null()).stderr(Stdio::null()))+ .ok()?;+ resolve_new_window(driver, title, &before)+}++/// The pids of all currently-open windows whose title contains `title`.+fn harness_pids(driver: &mut McpDriver, title: &str) -> std::collections::HashSet<u32> {+ let mut pids = std::collections::HashSet::new();+ let r = driver.call("list_windows", serde_json::json!({}));+ if let Some(wins) = r.structured()["windows"].as_array() {+ for w in wins {+ if w["title"].as_str().unwrap_or("").contains(title) {+ if let Some(pid) = w["pid"].as_u64() {+ pids.insert(pid as u32);+ }+ }+ }+ }+ pids+}++/// Poll `list_windows` until a `title` window whose pid is NOT in `before`+/// appears; returns (pid, window_id) and tracks the pid with the reaper.+fn resolve_new_window(+ driver: &mut McpDriver,+ title: &str,+ before: &std::collections::HashSet<u32>,+) -> Option<(u32, u64)> {+ let deadline = Instant::now() + Duration::from_secs(15);+ while Instant::now() < deadline {+ let r = driver.call("list_windows", serde_json::json!({}));+ if let Some(wins) = r.structured()["windows"].as_array() {+ for w in wins {+ if !w["title"].as_str().unwrap_or("").contains(title) {+ continue;+ }+ let pid = w["pid"].as_u64().unwrap_or(0) as u32;+ let wid = w["window_id"].as_u64().unwrap_or(0);+ if pid != 0 && wid != 0 && !before.contains(&pid) {+ driver.reaper().track_pid(pid);+ return Some((pid, wid));+ }+ }+ }+ std::thread::sleep(Duration::from_millis(400));+ }+ eprintln!("[capture_mode] new harness window {title:?} never appeared — is a graphical session available? skipping");+ None+}++// ── tests ───────────────────────────────────────────────────────────────────────++/// `capture_mode=ax`: tree present, NO image content.+#[test]+#[ignore]+fn capture_mode_ax_returns_tree_only() {+ let Some(mut driver) = McpDriver::spawn() else { return };+ let Some((pid, wid)) = launch(&mut driver) else { return };++ let resp = snapshot_settled(&mut driver, pid, wid, "ax");+ assert!(!resp.is_error(), "get_window_state(ax) errored: {}", resp.text());++ if !tree_has_marker(&resp) {+ eprintln!("[capture_mode] ax tree has no {TREE_MARKER:?} — accessibility grant likely missing; skipping");+ return;+ }+ assert!(+ !has_image(&resp),+ "capture_mode=ax must NOT return an image content entry: {}",+ resp.text().chars().take(160).collect::<String>()+ );+ println!("✅ capture_mode=ax: tree present ({TREE_MARKER}), no image");+}++/// `capture_mode=vision`: image content present, NO tree markdown.+#[test]+#[ignore]+fn capture_mode_vision_returns_image_only() {+ let Some(mut driver) = McpDriver::spawn() else { return };+ let Some((pid, wid)) = launch(&mut driver) else { return };++ let resp = snapshot(&mut driver, pid, wid, "vision");+ assert!(!resp.is_error(), "get_window_state(vision) errored: {}", resp.text());++ if !has_image(&resp) {+ eprintln!("[capture_mode] vision returned no image — screen-capture grant likely missing; skipping");+ return;+ }+ assert!(+ !tree_has_marker(&resp),+ "capture_mode=vision must NOT return tree markdown (found {TREE_MARKER}): {}",+ resp.text().chars().take(160).collect::<String>()+ );+ println!("✅ capture_mode=vision: image present, no tree");+}++/// `capture_mode=som` (default): both tree and image present (each asserted only+/// when its grant is available, so the test is a true superset of ax + vision).+#[test]+#[ignore]+fn capture_mode_som_returns_tree_and_image() {+ let Some(mut driver) = McpDriver::spawn() else { return };+ let Some((pid, wid)) = launch(&mut driver) else { return };++ let resp = snapshot_settled(&mut driver, pid, wid, "som");+ assert!(!resp.is_error(), "get_window_state(som) errored: {}", resp.text());++ let img = has_image(&resp);+ let tree = tree_has_marker(&resp);+ assert!(+ img || tree,+ "capture_mode=som returned neither tree nor image — both grants missing? {}",+ resp.text().chars().take(160).collect::<String>()+ );+ if !img {+ eprintln!("[capture_mode] som: no image (screen-capture grant missing) — tree-only check");+ }+ if !tree {+ eprintln!("[capture_mode] som: no tree marker (accessibility grant missing) — image-only check");+ }+ println!("✅ capture_mode=som: tree={tree} image={img}");+}diff --git a/libs/cua-driver/test-harness/TEST_SUITE.md b/libs/cua-driver/test-harness/TEST_SUITE.mdindex 8e11f27ef0..d73440ee74 100644--- a/libs/cua-driver/test-harness/TEST_SUITE.md+++ b/libs/cua-driver/test-harness/TEST_SUITE.md@@ -54,6 +54,7 @@ flowchart LR| | `harness_appkit_test` | mac | AppKit: AX tree, AXPress, NSTextField, NSScrollView, NSMenu || | `harness_swiftui_test` | mac | SwiftUI: AX tree, `.popover()` || **modality_** | `modality_background_test` | win | background-modality / no-focus-steal sentinel + `capture_mode` ax/vision/som |+| | `modality_capture_mode_test` | mac+win+linux | `capture_mode` axis on each native harness: `ax`→tree-only, `vision`→image-only, `som`→both || | `modality_input_e2e_test` | win | unified background input across Electron/Tauri/Win32, no z-raise || | `modality_desktop_scope_test` | win | desktop-scope (foreground): `capture_scope=desktop`, `get_desktop_state`, window-less screen-absolute actions || | `modality_focus_test` | mac | background automation does not steal focus |@@ -86,8 +87,32 @@ mirror pairs into single `cfg!`-branching tests, sharing one `RawDriver`.- **Foreground / desktop-scope** is covered by `modality_desktop_scope`.- **Transport** is now a first-class axis: `transport_config_persistence`exercises CLI (disk) vs MCP (session) directly; most other tests run over MCP.-- **Known gaps** (honest): desktop-scope/foreground has no macOS or Linux test;- `capture_mode` only on Windows; CLI transport is otherwise lightly exercised.++### The modality matrix: `capture_mode` × `dispatch` × `capture_scope`++The user-facing matrix is documented in+`docs/content/docs/explanation/capture-and-dispatch-modalities.mdx`. Coverage of+its five valid cells, per platform:++| Cell (`scope`/`mode`/`dispatch`) | Windows | macOS | Linux |+|---|---|---|---|+| `window`/`ax`/`background` (default) | `harness_*`, `modality_background` | `harness_{appkit,swiftui}`, `modality_focus`, `modality_capture_mode` | `harness_gtk3`, `modality_capture_mode` |+| `window`/`ax`/`foreground` | `harness_wpf` (`dispatch:"foreground"`) | activation differs | activation differs |+| `window`/`vision`/`background` | `modality_background`, `modality_capture_mode`, `modality_input_e2e` | `modality_capture_mode` | `modality_capture_mode` |+| `window`/`vision`/`foreground` | gap | n/a (no `bring_to_front`) | n/a (stubbed) |+| `desktop`/`vision`/`foreground` | `modality_desktop_scope` | rolling out | rolling out |+| **negative gate** (`desktop_scope_disabled`) | `window_scope_rejects_windowless_click` | — | — |++`modality_capture_mode_test` is the cross-platform spine of the `capture_mode`+axis: it drives each platform's native harness and asserts `ax`→tree-only,+`vision`→image-only, `som`→both. It closes the prior "`capture_mode` only on+Windows" gap.++- **Remaining gaps** (honest): `window`/`vision`/`foreground` has no explicit+ test on any platform; the desktop-scope window-less actuator is Windows-only+ (macOS click still requires `pid`), so the desktop cell and its negative gate+ are Windows-only until that loop lands on macOS/Linux; CLI transport is+ otherwise lightly exercised.## Where they run