Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

docs(cua-driver): explain how the perception extension works (#4296)

Francesco Bonaccica0cd60bf6b3
diff --git a/docs/content/docs/concepts/capture-and-delivery-modalities.mdx b/docs/content/docs/concepts/capture-and-delivery-modalities.mdx
index 27614f6cf8..ce6d66d3f8 100644
--- a/docs/content/docs/concepts/capture-and-delivery-modalities.mdx
+++ b/docs/content/docs/concepts/capture-and-delivery-modalities.mdx
@@ -13,6 +13,8 @@ Every Cua Driver action is shaped by four things: **what the agent observes**, *
The accessibility tree is the ground truth for *what is clickable*: roles, labels, advertised actions, and an `element_index` handle on every actionable element. The screenshot tells you *which one*. It disambiguates repeated or empty labels and shows captions, colors, and layout the tree omits, which is common in Chromium and Electron. They come back together because each catches what the other misses.
+When neither the tree nor typed browser state exposes the target, the optional perception extension can parse that same screenshot into labelled regions; see [How the perception extension works](/concepts/how-the-perception-extension-works).
+
> **Performance opt-out: `include_screenshot`.** `include_screenshot` (boolean, default `true`) is the one performance knob. The default returns both. Pass `include_screenshot: false` to skip the screen grab and get the tree only when you are re-indexing before an element ax action and do not need fresh pixels. The `ax`-versus-`px` decision still lives at action time.
> **`capture_mode` is deprecated and ignored.** `get_window_state` still accepts it so old callers do not error, but both the tree and the screenshot come back regardless of what you pass. The old `ax` / `vision` / `som` / `screenshot` values all decode (`som` mapped to `ax`, `screenshot` to `vision`) but none changes what is captured. Perception is always both.
diff --git a/docs/content/docs/concepts/how-the-perception-extension-works.mdx b/docs/content/docs/concepts/how-the-perception-extension-works.mdx
new file mode 100644
index 0000000000..9fc646ef95
--- /dev/null
+++ b/docs/content/docs/concepts/how-the-perception-extension-works.mdx
@@ -0,0 +1,107 @@
+---
+title: 'How the perception extension works'
+description: 'How the optional cua-perception extension turns one screenshot into regions an agent can act on, and how Cua Driver binds every resulting action to that exact screenshot.'
+---
+
+import { Callout } from 'fumadocs-ui/components/callout';
+
+The perception extension lets an agent act on what it can see in a window when
+the application exposes no usable target. It turns one screenshot into
+labelled regions (text, icons, controls), and Cua Driver acts on exactly the
+region the agent chose, bound to the screenshot it came from.
+
+It is optional and runs entirely on the local machine. Cua Driver works without
+it: the accessibility tree and typed browser state remain the first choice for
+finding a target.
+
+## When it helps
+
+Many interfaces hide their targets. Canvas apps, custom-drawn views, games,
+remote desktops, and some web pages have no useful accessibility tree or page
+structure, so an agent has nothing to address by role or label. The extension
+gives those surfaces a structure without sending screenshots to a remote model.
+
+Every [`get_window_state`](/concepts/capture-and-delivery-modalities) call
+already returns the accessibility tree and a screenshot together. The extension
+adds one step on top of that observation: it parses the screenshot into
+regions. It does not replace the tree, and it never captures on its own.
+
+## The loop
+
+1. **Capture.** `get_window_state` captures the exact window and returns a
+ `capture_id` with the screenshot.
+2. **Parse.** `parse_visual_regions` turns that screenshot into typed regions
+ with bounds and, for text, the recognized words.
+3. **Choose.** The agent, or an external chooser such as a decision model,
+ selects one region.
+4. **Act.** Cua Driver clicks the region using the same `capture_id`, trying
+ background delivery first.
+5. **Reobserve.** The agent takes a fresh capture to confirm the result before
+ the next step.
+
+Without the extension installed, `parse_visual_regions` returns
+`not_installed`, and agents continue with the accessibility tree or page
+structure. Nothing is downloaded automatically.
+
+## Every action is bound to one screenshot
+
+A region only means something on the screen it was found on. Cua Driver
+therefore ties each perception-based action to its capture:
+
+| Situation | Result |
+| --- | --- |
+| The click carries the `capture_id` of the screenshot the region came from | Delivered |
+| The same capture is reused for another action | Refused with `capture_not_found` |
+| The capture is older than 60 seconds | Refused with `capture_expired` |
+
+Agents receive region bounds and text, never screenshot bytes, and Cua Driver
+keeps the authority to capture and to act. A stale or reused screen is refused
+instead of guessed at.
+
+## What the extension contains
+
+Each release packs local models and a CPU runtime, so parsing needs no GPU and
+no network access.
+
+| Component | Role | License |
+| --- | --- | --- |
+| OmniParser icon detector | Finds icons and controls | AGPL-3.0-only |
+| PP-OCR detector and recognizer | Finds and reads on-screen text | Apache-2.0 |
+| ONNX Runtime (CPU) | Runs the models locally | Shipped with its own notices |
+
+Releases are published as `cua-perception-v<version>` GitHub releases for macOS
+arm64, Linux x64, and Windows x64. Each target has a signed catalog, the
+archive, an SBOM, provenance, a runtime contract, and an install check, plus
+one `SHA256SUMS`. Cua Driver verifies the signed catalog before it installs
+anything, and a verified install reports `publisher-verified`.
+
+<Callout type="warning" title="License precautions">
+ The extension is not MIT licensed. Its OmniParser icon detector is AGPL-3.0-only, and Cua does
+ not relicense it. Installing the extension does not change Cua Driver's MIT license, but
+ redistributing the extension, or offering it to users over a network (for example, inside a
+ hosted product), can require you to provide the AGPL corresponding source. If your organization
+ does not accept AGPL components, skip the extension: Cua Driver keeps working, and
+ `parse_visual_regions` returns `not_installed`. This is not legal advice. See the [third-party
+ notices and
+ precautions](https://github.com/trycua/cua/blob/main/libs/cua-driver/docs/perception-third-party-notices.md#before-you-install-redistribute-or-host-the-extension).
+</Callout>
+
+## Install is always explicit
+
+Capture, parsing, Cua Driver startup, and update checks never install the
+extension. You inspect a catalog first and then install, check, update, or
+remove the extension with `cua-driver extension` commands. A running Driver
+picks up an install or removal without a restart.
+
+## Performance
+
+Parsing runs on the CPU. Measured per screenshot on the canonical harnesses:
+about 2–2.5 seconds on macOS, 3.5–4 seconds on Linux, and 8–9 seconds on
+Windows. Prefer accessibility or page targets when they exist, and reserve
+perception for surfaces that expose nothing addressable.
+
+## Related pages
+
+- [Parse visual regions with Cua Driver](/how-to-guides/driver/parse-visual-regions): install the extension and run the loop step by step.
+- [Use jev-use with visual regions](/how-to-guides/driver/use-jev-use-with-visual-regions): let a decision model choose among regions.
+- [Capture and delivery modalities](/concepts/capture-and-delivery-modalities): how observation, action rungs, and delivery fit together.
diff --git a/docs/content/docs/concepts/index.mdx b/docs/content/docs/concepts/index.mdx
index 245bbb82ba..4d9b2fad11 100644
--- a/docs/content/docs/concepts/index.mdx
+++ b/docs/content/docs/concepts/index.mdx
@@ -11,4 +11,4 @@ For Linux, read [Linux desktops and computer use](/concepts/linux-desktops-and-c
to understand X11, Wayland, XWayland, and the relationship between Omarchy and
Hyprland before choosing a support-matrix row.
-Start with [What is computer use?](/concepts/what-is-computer-use) for the basic model. If you are adding Cua Driver to an agent or product, use [Choose a Cua Driver integration](/concepts/choose-a-cua-driver-integration) before reading the deeper [SDK, MCP, and process hosting](/concepts/sdk-mcp-and-hosting) model. Read [Best-effort background](/concepts/the-no-foreground-contract) to understand Cua Driver's default behavior on a shared machine, then [Capture and delivery modalities](/concepts/capture-and-delivery-modalities) for the action axes. [How we continuously validate Cua Driver](/concepts/how-cua-driver-is-validated) explains why desktop support requires application-owned E2E evidence in addition to unit tests. Read [How sandboxes work](/concepts/how-sandboxes-work) when you need the model for disposable computers, then [Sandbox lifecycle](/concepts/sandbox-lifecycle) for the lifetime and connection patterns. Read [How Lume creates local macOS sandboxes](/concepts/how-lume-unattended-setup-works) for the local VM setup model, and [How SIP works in Lume VMs](/concepts/how-sip-works-in-lume-vms) for signed boot-policy changes. For evaluation, start with [What is Cua-Bench?](/concepts/what-is-cua-bench), then read [The Cua-Bench task lifecycle](/concepts/cua-bench-task-lifecycle).
+Start with [What is computer use?](/concepts/what-is-computer-use) for the basic model. If you are adding Cua Driver to an agent or product, use [Choose a Cua Driver integration](/concepts/choose-a-cua-driver-integration) before reading the deeper [SDK, MCP, and process hosting](/concepts/sdk-mcp-and-hosting) model. Read [Best-effort background](/concepts/the-no-foreground-contract) to understand Cua Driver's default behavior on a shared machine, then [Capture and delivery modalities](/concepts/capture-and-delivery-modalities) for the action axes. [How the perception extension works](/concepts/how-the-perception-extension-works) explains how the optional extension turns a screenshot into regions an agent can act on. [How we continuously validate Cua Driver](/concepts/how-cua-driver-is-validated) explains why desktop support requires application-owned E2E evidence in addition to unit tests. Read [How sandboxes work](/concepts/how-sandboxes-work) when you need the model for disposable computers, then [Sandbox lifecycle](/concepts/sandbox-lifecycle) for the lifetime and connection patterns. Read [How Lume creates local macOS sandboxes](/concepts/how-lume-unattended-setup-works) for the local VM setup model, and [How SIP works in Lume VMs](/concepts/how-sip-works-in-lume-vms) for signed boot-policy changes. For evaluation, start with [What is Cua-Bench?](/concepts/what-is-cua-bench), then read [The Cua-Bench task lifecycle](/concepts/cua-bench-task-lifecycle).
diff --git a/docs/content/docs/concepts/meta.json b/docs/content/docs/concepts/meta.json
index 705a17cfe5..27e14d3fc7 100644
--- a/docs/content/docs/concepts/meta.json
+++ b/docs/content/docs/concepts/meta.json
@@ -9,6 +9,7 @@
"sdk-mcp-and-hosting",
"the-no-foreground-contract",
"capture-and-delivery-modalities",
+ "how-the-perception-extension-works",
"linux-desktops-and-computer-use",
"browser-targeting-and-background-delivery",
"how-permission-policies-work",
diff --git a/docs/content/docs/how-to-guides/driver/parse-visual-regions.mdx b/docs/content/docs/how-to-guides/driver/parse-visual-regions.mdx
index dd32df0e57..bae3513d6b 100644
--- a/docs/content/docs/how-to-guides/driver/parse-visual-regions.mdx
+++ b/docs/content/docs/how-to-guides/driver/parse-visual-regions.mdx
@@ -10,6 +10,9 @@ accessibility tree or typed browser state does not identify the target. Cua
Driver keeps capture and action authority; the optional perception extension
turns one retained screenshot into model-neutral regions.
+For what the extension contains and how every action stays bound to one
+screenshot, read [How the perception extension works](/concepts/how-the-perception-extension-works).
+
<Callout type="info" title="Separate extension">
The `cua-perception` extension is separate from the default Cua Driver install. Each release is
published as a `cua-perception-v<version>` GitHub release with one signed catalog and archive per