Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
docs(cua-driver): add integration chooser (#3190)
Francesco Bonacci80b05014c22c
diff --git a/docs/content/docs/concepts/choose-a-cua-driver-integration.mdx b/docs/content/docs/concepts/choose-a-cua-driver-integration.mdxnew file mode 100644index 0000000000..aae59d5214--- /dev/null+++ b/docs/content/docs/concepts/choose-a-cua-driver-integration.mdx@@ -0,0 +1,152 @@+---+title: 'Choose a Cua Driver integration'+description: 'Choose between MCP, the SDK, a private worker, and an app-hosted service based on who owns the runtime.'+---++import { Callout } from 'fumadocs-ui/components/callout';++Most agent harnesses should connect to Cua Driver through MCP. Embed the SDK+when computer use is part of your product, and use an app-hosted service when a+desktop application must own the operating-system permissions.++## Start with who owns the runtime++The runtime is the process that owns Cua Driver's permissions, browser+connections, recordings, and lifecycle state. Choose its owner before choosing+the transport:++| You are building | Start with | Runtime owner |+| -------------------------------------------------- | ----------------------------------------------------------------------- | ------------------------------------------------- |+| An agent or harness | [MCP](/how-to-guides/driver/connect-your-agent) | The MCP process, or an explicitly selected daemon |+| An application with built-in computer use | [Same-process SDK](/how-to-guides/driver/use-sdk-in-process) | Your application process |+| A signed desktop app that serves an external agent | [App-hosted service](/how-to-guides/driver/expose-mcp-from-desktop-app) | The permission-owning desktop app |++These are starting points. Add a private worker or shared daemon only when you+need the process boundary it provides.++## Connect an agent through MCP++Use MCP when an existing agent or harness needs Cua Driver's tool catalog.++```text+Agent or harness -> cua-driver mcp -> Cua runtime -> desktop+```++The MCP client starts `cua-driver mcp` and keeps its standard input and output+open. That authenticated transport receives a private implicit lifecycle+session. Closing the transport releases its session state.++This is the normal route for Codex, Claude Code, Cursor, Hermes, and a standard+OpenClaw gateway. Generate the client-specific configuration with+`cua-driver mcp-config --client <client>`.++On Windows and Linux, bare `cua-driver mcp` owns its runtime. On macOS, it+normally proxies to the installed `CuaDriver.app` daemon so Accessibility and+Screen Recording remain attached to the app identity.++## Embed the SDK in an application++Use the Python or TypeScript SDK when your product owns computer use as a+feature and wants a typed API instead of an agent protocol.++```text+Your application -> CuaDriver.create() -> desktop+```++The application owns runtime startup, permissions, sessions, cancellation, and+shutdown. It can expose Cua operations directly or map them into a smaller+product-owned tool.++Use a private worker when the application needs the same SDK contract in a+supervised child process:++```text+Your application -> inherited pipes -> private Cua worker -> desktop+```++The worker has no listener or reconnect path. Closing its private channel ends+the runtime. This gives one host process isolation without creating a shared+desktop service.++## Host Cua Driver from a desktop app++Use an app-hosted service when a signed desktop app owns Accessibility and+Screen Recording permissions while a backend or external agent needs MCP.++```text+Desktop app -> private Cua service -> generated MCP connection -> agent+```++The desktop app starts `EmbeddedCuaDriverHost` and passes its returned MCP+connection to the backend. The backend launches that connection unchanged. It+must not start a second host.++This distinction matters on macOS. The permission-owning app must start the+private service directly so the child stays in the app's TCC responsibility+chain. Starting it from a gateway or backend gives the service that process's+identity instead.++## Less common variants++Use one-shot CLI calls when an agent can run shell commands but cannot keep an+MCP transport open:++```text+Agent -> cua-driver call <tool> -> Cua Driver daemon -> desktop+```++Use an explicit shared daemon when several clients must reconnect to one+long-lived desktop service or when the MCP process cannot reach the interactive+desktop:++```text+Trusted launcher -> cua-driver serve --socket <endpoint>+Agent A ---------> cua-driver mcp --socket <endpoint>+Agent B ---------> cua-driver mcp --socket <endpoint>+```++The daemon owns the runtime. Each MCP transport still receives its own private+lifecycle session.++## Permissions belong to the runtime++<Callout type="info">+ The trusted host fixes the permission mode, capability manifest, launch grants, and authorization+ callbacks when it starts the runtime. An agent can request actions, but it cannot widen the+ runtime's permissions.+</Callout>++The optional public session name is a label for display and explicit lifecycle+control. It is not a credential and does not carry permission. Two MCP+transports that use the same public label still have separate private lifecycle+sessions.++## Existing browser sessions++A driver-owned isolated browser profile is the default. Attaching to an+existing Chrome or Edge profile is more sensitive because it exposes signed-in+tabs, cookies, and storage through CDP.++Browser integrations that depend on launching Chromium with a remote debugging+flag cannot attach that way to a profile already running without the flag. Cua+Driver can instead use the browser's per-instance remote debugging control in+the exact approved window, complete the browser-owned consent flow, and bind+the endpoint to that browser process and lifecycle session. It does not copy,+edit, restart, or terminate the selected profile.++Standard mode requires the trusted launcher to pass `--grant+existing-profile`, or an embedding application to approve the exact request.+The old `browser-approve` token is not part of this model, and an MCP tool+argument cannot create the grant.++See [Browser Profile Attachment](/reference/cua-driver/browser-profile-attachment)+for the supported browsers, setup effects, grant lifetime, and refusal codes.++## Go deeper++- [Connect your agent](/how-to-guides/driver/connect-your-agent): configure an MCP client.+- [Use Cua Driver in process](/how-to-guides/driver/use-sdk-in-process): embed the typed SDK.+- [Expose MCP from a desktop app](/how-to-guides/driver/expose-mcp-from-desktop-app): host a private service from a signed app.+- [SDK, MCP, and process hosting](/concepts/sdk-mcp-and-hosting): understand the complete interface and topology grid.+- [Process model](/reference/cua-driver/process-model): check platform-specific runtime ownership.+- [Permission modes](/reference/cua-driver/permission-modes): check the authorization contract.diff --git a/docs/content/docs/concepts/index.mdx b/docs/content/docs/concepts/index.mdxindex 6e7b9dba98..608638f86f 100644--- a/docs/content/docs/concepts/index.mdx+++ b/docs/content/docs/concepts/index.mdx@@ -5,4 +5,4 @@ description: "Understand the design ideas behind Cua and how its main pieces fitUse these pages when you want the model behind Cua rather than a step-by-step guide or API table. They explain what computer use means in Cua, how Cua Driver keeps the desktop usable while it acts, and how Cua Sandbox gives an agent a disposable local computer.-Start with [What is computer use?](/concepts/what-is-computer-use) for the basic model. Read [Best-effort background](/concepts/the-no-foreground-contract) to understand Cua Driver's default behavior on a shared machine, then [Capture and delivery modalities](/concepts/capture-and-delivery-modalities) for the action axes. [How we continuously validate Cua Driver](/concepts/how-cua-driver-is-validated) explains why desktop support requires application-owned E2E evidence in addition to unit tests. Read [How sandboxes work](/concepts/how-sandboxes-work) when you need the model for disposable computers. Read [How Lume creates local macOS sandboxes](/concepts/how-lume-unattended-setup-works) for the local VM setup model, and [How SIP works in Lume VMs](/concepts/how-sip-works-in-lume-vms) for signed boot-policy changes. For evaluation, start with [What is Cua-Bench?](/concepts/what-is-cua-bench), then read [The Cua-Bench task lifecycle](/concepts/cua-bench-task-lifecycle).+Start with [What is computer use?](/concepts/what-is-computer-use) for the basic model. If you are adding Cua Driver to an agent or product, use [Choose a Cua Driver integration](/concepts/choose-a-cua-driver-integration) before reading the deeper [SDK, MCP, and process hosting](/concepts/sdk-mcp-and-hosting) model. Read [Best-effort background](/concepts/the-no-foreground-contract) to understand Cua Driver's default behavior on a shared machine, then [Capture and delivery modalities](/concepts/capture-and-delivery-modalities) for the action axes. [How we continuously validate Cua Driver](/concepts/how-cua-driver-is-validated) explains why desktop support requires application-owned E2E evidence in addition to unit tests. Read [How sandboxes work](/concepts/how-sandboxes-work) when you need the model for disposable computers. Read [How Lume creates local macOS sandboxes](/concepts/how-lume-unattended-setup-works) for the local VM setup model, and [How SIP works in Lume VMs](/concepts/how-sip-works-in-lume-vms) for signed boot-policy changes. For evaluation, start with [What is Cua-Bench?](/concepts/what-is-cua-bench), then read [The Cua-Bench task lifecycle](/concepts/cua-bench-task-lifecycle).diff --git a/docs/content/docs/concepts/meta.json b/docs/content/docs/concepts/meta.jsonindex 030f917219..22bfee672a 100644--- a/docs/content/docs/concepts/meta.json+++ b/docs/content/docs/concepts/meta.json@@ -5,6 +5,7 @@"index","what-is-computer-use","---Cua Driver---",+ "choose-a-cua-driver-integration","sdk-mcp-and-hosting","the-no-foreground-contract","capture-and-delivery-modalities",diff --git a/docs/content/docs/how-to-guides/driver/connect-your-agent.mdx b/docs/content/docs/how-to-guides/driver/connect-your-agent.mdxindex a9dfe8ee74..0d2d89e1e8 100644--- a/docs/content/docs/how-to-guides/driver/connect-your-agent.mdx+++ b/docs/content/docs/how-to-guides/driver/connect-your-agent.mdx@@ -7,6 +7,10 @@ import { Callout } from 'fumadocs-ui/components/callout';Cua Driver lets a [computer-use agent](/concepts/what-is-computer-use) drive the **host desktop**: installed apps, signed-in browser sessions, local files opened in apps, and the current OS user session. MCP-capable agents connect through `cua-driver mcp`; skill-based agents such as Prime Agent call the CLI directly. [Grok Bot](/use-cua-with/grok-bot) is a separate path; it is not Grok Build and does not use stdio MCP.+Building Cua Driver into an application instead? Start with [Choose a Cua+Driver integration](/concepts/choose-a-cua-driver-integration) to compare MCP,+the direct SDK, private workers, and app-hosted services.+<Callout type="info">This connects an agent to Cua Driver on the current machine. To create a disposable local desktop,use [Cua Sandbox](/tutorials/your-first-local-sandbox) instead.diff --git a/docs/content/docs/how-to-guides/driver/install.mdx b/docs/content/docs/how-to-guides/driver/install.mdxindex 9b9b74841e..ce38a52997 100644--- a/docs/content/docs/how-to-guides/driver/install.mdx+++ b/docs/content/docs/how-to-guides/driver/install.mdx@@ -253,6 +253,7 @@ cua-driver permissions status## Next steps+- [Choose a Cua Driver integration](/concepts/choose-a-cua-driver-integration): decide whether your agent or product should use MCP, the SDK, or an app-hosted service.- [Permission modes](/reference/cua-driver/permission-modes): what standard allows, and how bounded and unrestricted differ.- [Install the Cua Driver agent skill](/how-to-guides/driver/install-agent-skill): add the cross-platform instructions from ClawHub or install them directly for another agent.- [Keep Cua Driver running](/how-to-guides/driver/keep-running): configure autostart so the daemon comes back after reboots.diff --git a/docs/content/docs/index.mdx b/docs/content/docs/index.mdxindex 77d4ce96db..8ad8ae750c 100644--- a/docs/content/docs/index.mdx+++ b/docs/content/docs/index.mdx@@ -25,6 +25,10 @@ import { Card, Cards } from 'fumadocs-ui/components/card';</Card></Cards>+Building Cua Driver into your own product? Start with [Choose a Cua Driver+integration](/concepts/choose-a-cua-driver-integration) to decide whether MCP,+the SDK, or an app-hosted service should own the runtime.+## Cua Driver in actionThese recordings show agents operating desktop apps while the user's active window stays in place. Each demo was recorded by the Cua team on the named platform.diff --git a/docs/content/docs/tutorials/drive-your-first-app.mdx b/docs/content/docs/tutorials/drive-your-first-app.mdxindex 68feb48699..56c6affe5f 100644--- a/docs/content/docs/tutorials/drive-your-first-app.mdx+++ b/docs/content/docs/tutorials/drive-your-first-app.mdx@@ -13,6 +13,10 @@ Linux.New to agents that operate applications? Start with [What is computer use?](/concepts/what-is-computer-use) for the model behind the workflow, then return here to build one.+This tutorial connects an existing agent to Cua Driver. If you are embedding+Cua Driver in a product, start with [Choose a Cua Driver+integration](/concepts/choose-a-cua-driver-integration) instead.+<VideoDemosrc="https://github.com/user-attachments/assets/11b248e5-d505-4349-aaae-ad9f95a1162f"poster="https://github.com/user-attachments/assets/0fb62243-56ec-4d51-8771-35640a5e6031"