Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

6.6 KB
Raw Blame

Experimental cua-driver SDK contract

This directory contains the checked-in, generated portable cua-driver SDK contract. The Rust crate at rust/crates/cua-driver-contract is the source of truth. It generates this manifest and exports the request/result records consumed by the live daemon and the UniFFI SDK. Python and TypeScript bindings are generated from the compiled Rust library by scripts/generate-uniffi-bindings.mjs.

Execution, platform integration, policy, and permission handling remain in Rust. CuaDriver.create() owns that runtime in the importing process; connect() uses an existing daemon through the shared Rust socket client. Agents independently use the public cua-driver mcp surface through their runtime's existing MCP client.

Scope and compatibility

The typed slice covers the cross-platform session lifecycle tools:

  • start_session
  • get_session
  • list_sessions
  • end_session

Ordinary calls do not need start_session. The runtime creates one implicit session for the authenticated transport lease and reuses it until transport close, explicit end, or five minutes of inactivity. escalate_session and get_session_state remain deprecated compatibility tools for legacy capture-scope sessions. There is no deescalate_session tool.

It also covers the portable whole-desktop loop:

  • get_desktop_state
  • get_screen_size
  • get_cursor_position
  • move_cursor with an exact per-call window or desktop target
  • set_window_frame for exact, read-back-verified top-level window geometry
  • invoke_menu for an exact native application-menu path resolved live at each hop
  • click with an exact per-call window or desktop target
  • drag and scroll in native desktop coordinates
  • type_text, press_key, and hotkey against the foreground application
  • clipboard_read for available types and opt-in plain-text readback
  • clipboard_write for text, image, and file-URL clipboard content

The typed native-window flow includes list_apps, list_windows, and get_window_state, with discovery records, snapshot-bound element tokens, optional accessibility metadata, and screenshot images attached to the typed snapshot. click accepts one ClickPosition (coordinates or element token), an explicit ActionTarget, and an explicit InputDeliveryMode, and returns ActionResult directly. Native refusals become DriverError.Tool. See the 0.8 SDK contract migration for the intentional SDK break and unchanged CLI/MCP wire forms.

The canonical session-owned cursor slice is shared exactly by MCP and both generated SDKs:

  • set_agent_cursor_enabled
  • set_agent_cursor_motion
  • set_agent_cursor_theme
  • get_agent_cursor_state

Cursor artwork is selected by installed theme ID. Theme installation and dotLottie compilation are intentionally local CLI operations, not SDK or MCP tool calls.

The checked-observation slice is also shared by MCP and both generated SDKs:

  • verify_state evaluates one to eight ANDed predicates against one exact (pid, window_id).
  • Window existence/bounds and semantic accessibility element existence/value/enabled/selected predicates return satisfied, unsatisfied, or unknown.
  • A bounded poll can require consecutive stable samples. unknown never implies success.
  • include_screenshot=true adds final image content for a multimodal agent harness to interpret. Cua Driver does not OCR or assign task meaning to it.

Session contracts are marked canonical_runtime: the same typed Rust input, output, and metadata declaration builds the live MCP tool. The preferred action target is a tagged union: {kind:"window", pid, window_id} or {kind:"desktop", display_id:"primary"}. Legacy flat scope, pid, and window_id fields remain accepted during the compatibility window but cannot be mixed with target. Desktop contracts are marked portable_subset: their typed Rust inputs are a deliberately narrower projection of the richer macOS, Linux, and Windows runtime schemas. Platform handlers retain their existing wire parsers, including the legacy desktop-coordinate click projection. The SDK serializes its stricter click addressing to those same native fields. Successful SDK-path structured payloads are validated against the shared Rust output types in the live registry.

The platform schemas remain richer by design; they are not an independent SDK manifest. A cross-platform CI matrix proves every portable schema is accepted by each live registry, and the published tools resolve their capability tokens from the contract rather than a second runtime map.

Both SDKs retain a generic tool call so runtime-discovered and platform-specific tools remain usable. The generated manifest records tool platforms, capabilities, annotations, input schemas, and experimental success schemas. The live MCP tools/list response advertises these successful-result schemas as outputSchema; all action tools share the closed ActionResult schema even when their richer runtime input is not part of the portable SDK manifest.

See Action results and postcondition verification for the wire shape and 0.14 migration guidance.

Compatibility is tracked separately at each boundary:

Field Current Meaning
contract_version 0.8.0 Generated manifest and typed SDK shape
tools_list_schema_version 1 cua-driver tools/list extension shape
capability_version 1 Additive capability-token vocabulary
mcp_protocol_version 2025-06-18 Legacy initialize.params.protocolVersion and loopback HTTP compatibility version; modern stdio negotiation is endpoint-owned

This implementation does not use WASM. UniFFI distributes the shared Rust implementation to Python and Node. Permission identity follows the selected runtime host. The language packages do not generate or maintain separate MCP transports.

Generate and verify

From libs/cua-driver/rust:

cargo run -p cua-driver-contract --bin cua-contract-gen -- all
cargo run -p cua-driver-contract --bin cua-contract-gen -- all --check
cargo test -p cua-driver-contract
cargo test -p cua-driver-core --test contract_parity
cargo test -p cua-driver --test schema_consistency_test \
  portable_desktop_contracts_are_accepted_by_active_backend

SDK loader tests live in python/tests/test_uniffi_loader.py and typescript/test/native-loader.test.mjs. CI checks the manifest generator, deterministically regenerates both UniFFI binding sets, verifies parity against the live tool registry, and crosses the real Python and Node FFI loaders into a deterministic daemon-socket fixture.