Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Optional perception extension
parse_visual_regions is a model-neutral Cua Driver tool backed by the
separately installed cua-perception extension. It parses one screenshot into
text and icon regions. It does not capture the screen, choose an action, or
send input.
The extension is absent from the default MIT-licensed Driver installation.
Driver startup, capture, parsing, and update checks never install it or
download model weights. A missing extension returns not_installed, allowing
the caller to continue with accessibility, typed browser state, or its own
visual reasoning.
Local image CLI
The read-only CLI convenience mode parses an existing local PNG through the same installed worker and canonical visual-region validation:
cua-driver perception parse \
--image /tmp/window.png \
--capture /tmp/capture.json \
--json
The capture metadata file has this strict shape:
{
"source": { "kind": "window", "pid": 844, "window_id": 10725 },
"snapshot_id": "s0000002a",
"captured_at": "2026-09-18T12:00:00Z"
}
For the primary desktop, use
{"source":{"kind":"primary_desktop","display_id":"primary"}}.
snapshot_id and captured_at are optional. The command validates and hashes
the PNG header and declared image bounds, but it does not decode pixels in the
privileged CLI, create or retain a Driver capture, or include the local input
paths in its output. The JSON result has schema cua.visual_regions_v1 plus a
local_input provenance block that sets action_eligible to false and
action_authority to none. Its local_png_<digest> label uses a separate
namespace from Driver capture_<namespace>_<sequence> IDs and cannot authorize
a Driver action.
Because --json is required, every failure is written to stdout as one stable
envelope and the process exits nonzero:
{
"ok": false,
"error": {
"code": "invalid_capture_metadata",
"message": "captured_at must be an RFC3339 timestamp",
"retryable": false
}
}
CLI admission codes are invalid_arguments, input_open_failed,
input_not_regular_file, input_too_large, input_read_failed,
invalid_capture_metadata, invalid_png, and internal_error. Worker and
extension failures preserve the visual parsing codes, including
not_installed, artifact_invalid, incompatible_protocol, timeout, worker,
resource-limit, and inference failures. An optional detail string provides
diagnostic context and must not be parsed as a stable field.
Explicit lifecycle
Each Cua Perception release is published as the cua-perception-v<version>
GitHub release. It carries one signed catalog and one archive per target:
| Target | Catalog | Archive |
|---|---|---|
| macOS arm64 | cua-perception-<version>-aarch64-apple-darwin.catalog.json |
cua-perception-<version>-aarch64-apple-darwin.tar.gz |
| Linux x64 | cua-perception-<version>-x86_64-unknown-linux-gnu.catalog.json |
cua-perception-<version>-x86_64-unknown-linux-gnu.tar.gz |
| Windows x64 | cua-perception-<version>-x86_64-pc-windows-msvc.catalog.json |
cua-perception-<version>-x86_64-pc-windows-msvc.tar.gz |
The release also carries each target's SBOM, redacted provenance, runtime
contract, the release gate's install evidence, and a SHA256SUMS file. The
Driver resolves the archive named in the catalog relative to the catalog file,
so download both into the same directory. Select the release by its
cua-perception-v* tag; it is never marked as the repository's Latest release.
Inspect the exact target before changing local state:
cua-driver extension inspect cua-perception --catalog <catalog.json>
cua-driver extension install cua-perception --catalog <catalog.json>
cua-driver extension status cua-perception
Inspection must identify the extension version, platform target, protocol range, capabilities, artifact sizes and hashes, publisher, licenses, notices, and provenance. Do not substitute an unsigned archive or infer component availability from the repository-wide latest release.
Each catalog expires one year after its release commit. After that, install
from a newer cua-perception-v* release.
Release publication
Cua Perception follows Release Please with no manual publish step. Merging its
release pull request bumps VERSION, and the release workflow anchors the
merged commit with a lightweight cua-perception-v<version> tag. That tag push
runs .github/workflows/cd-cua-perception.yml, which:
- requires the tag to be a lightweight tag on
mainthat matchesVERSION; - verifies the reviewed OmniParser ONNX conversion against
scripts/artifacts.lock.json; - builds the worker on each target, assembles the bundle, and runs its health check, self-test, and real parse;
- re-verifies, packages, and signs each target in the reviewed candidate
workflow, whose signing job runs in the
cua-perception-candidate-signingenvironment; - installs the published Cua Driver with its canonical installer on each
target and runs
extension inspect,install,status --self-test,perception parse, andremoveagainst the signed catalog; and - publishes the verified assets only when every job passed for the exact tag commit.
Pull requests that change this pipeline run steps 1 through 3 plus an unsigned package and a developer-only install check. They never sign or publish.
After publication, the capture lane of the canonical macOS, Windows, and Linux
X11 desktop E2E installs a pinned published catalog as publisher-verified
and runs the full capture-bound loop below against a visual-only fixture. The
row is described in the test matrix.
Inspect a replacement before running extension update. Remove only the
extension-owned installation with extension remove cua-perception. Neither
operation changes the default Driver or an external cua-som installation.
Install, update, and remove take effect on a running Driver without a restart.
A running daemon or cua-driver mcp session checks the installed extension
before each parse_visual_regions call and fully reverifies it whenever the
installation changed. A parse that is already running finishes with the version
it started with.
Capture-bound parse and action
Keep capture, parse, action, and reobservation on one persistent MCP connection
or typed SDK runtime. Each one-shot CLI process owns a disposable capture
registry, so a later process cannot resolve its capture_id.
- Call
get_window_statefor the exact native(pid, window_id)target. Useget_desktop_stateonly when full-desktop observation is explicitly allowed and a window capture cannot represent the task. - Retain the returned native
capture_id, source, PNG dimensions, and action-coordinate mapping. An accessibilitysnapshot_idis a different identity. - Call
parse_visual_regionswith that capture ID and boundedkinds,min_confidence, andmax_regionsoptions. - Prefer a current accessibility element or typed browser ref. If pixels are
required, choose a point inside one current region and send one
clickwithx,y, the exact target,delivery_mode, and the samecapture_id. - Reobserve after every action attempt. Never remove
capture_idand retry an expired, stale, retired, or mismatched capture as an unbound click.
Capture-bound click refusals use one shared code table on every platform,
owned by CaptureActionError::wire_code in cua-driver-core:
capture_id_invalid, capture_not_found, capture_expired,
capture_generation_mismatch, capture_target_mismatch,
capture_coordinate_invalid, and capture_frame_mismatch. Live-target
inspection failures before admission remain adapter-owned
(capture_action_refused on Windows and Linux, capture_target_mismatch on
macOS).
A capture-bound click consumes the capture before native dispatch. At most one action may derive from a capture. Capture again after an action, timeout, unknown result, resize, move, scroll, navigation, display-layout change, or target-identity change.
Window parsing does not imply background input support. Driver first applies
the selected platform's normal background route. A structured
background_unavailable refusal does not authorize a foreground retry.
delivery_mode:"foreground" may change focus, workspace, or the system cursor
and requires the workflow's existing foreground authorization. Desktop input
is foreground/system input even when the parsed screenshot came from Driver.
Worker boundary and errors
The worker receives only the admitted PNG and bounded parse options. It has no desktop capture, accessibility, browser, input, credential-store, or provider credential authority. The local worker performs no network access and does not require Python.
Treat capture_not_found, capture_expired, capture_stale, and
capture_generation_mismatch as a request to capture again. Preserve
unsupported_target and unsupported_platform as explicit limitations. Stop
using the extension after incompatible_protocol or artifact_invalid until
the exact installed artifact is inspected. Never act from a partial result
after a worker failure, timeout, invalid frame, or resource-limit error.
Distribution boundary
The extension artifact has its own license and source obligations. The release ledgers identify the OmniParser detector artifact as AGPL-3.0-only and the PP-OCR detector and recognizer artifacts as Apache-2.0. Each release's SBOM records the exact version and hash of the ONNX Runtime it packages. Read the perception third-party notices, including its precautions for redistribution and hosted services. A release must include the reviewed notices, ledgers, SBOM, source/conversion materials, and any corresponding-source offer required for the way the artifact is distributed or offered over a network.