Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
docs: move sandbox lifecycle into concepts (#3415)
r33drichards4504254964c3
diff --git a/docs/content/docs/concepts/how-sandboxes-work.mdx b/docs/content/docs/concepts/how-sandboxes-work.mdxindex ac85f7cb7d..8c591c2645 100644--- a/docs/content/docs/concepts/how-sandboxes-work.mdx+++ b/docs/content/docs/concepts/how-sandboxes-work.mdx@@ -1,6 +1,6 @@----title: "How sandboxes work"-description: "How a Cua Sandbox gives an agent one isolated computer it can both run code in and drive through the GUI."+title: 'How sandboxes work'+description: 'How a Cua Sandbox gives an agent one isolated computer it can both run code in and drive through the GUI.'---A Cua Sandbox is a **full, isolated computer**, not a remote desktop session. In that one machine, an agent can run code through Python, shell commands, or a PTY, and it can also drive the graphical interface through screenshots, the accessibility tree, clicks, and typing.@@ -19,9 +19,9 @@ This isolation is what makes a sandbox useful for repeatable agent work. The age## The code half and the GUI half-The *code half* is how an agent runs programs inside the sandbox. It includes shell commands through `shell.run`, an interactive PTY or terminal through `computer.pty` and `cua do shell`, and sandboxed Python that runs a function inside the sandbox's own virtualenv through `venv_install`, `venv_exec`, and the `@sandboxed` decorator.+The _code half_ is how an agent runs programs inside the sandbox. It includes shell commands through `shell.run`, an interactive PTY or terminal through `computer.pty` and `cua do shell`, and sandboxed Python that runs a function inside the sandbox's own virtualenv through `venv_install`, `venv_exec`, and the `@sandboxed` decorator.-The *GUI half* is how an agent uses the sandbox like a desktop computer. It can observe the screen through screenshots and the accessibility tree, then act through clicks, typing, scrolling, keypresses, and other input events.+The _GUI half_ is how an agent uses the sandbox like a desktop computer. It can observe the screen through screenshots and the accessibility tree, then act through clicks, typing, scrolling, keypresses, and other input events.Because both halves are interfaces to the same machine, they can be mixed within one task. A shell command can create a file that is opened in the GUI. A GUI workflow can download data that is then inspected with Python. A PTY session can start a server, and the GUI half can open a browser against it. The important point is that all of these actions share the same filesystem and OS state.@@ -47,15 +47,15 @@ Sandbox lifetime is separate from agent connection lifetime. An ephemeral sandboA persistent or named sandbox is created once, identified by name, and can survive process exits. A later process can reconnect to the same sandbox and continue from the state it already has.-Connect mode attaches to an already-running sandbox. It does not create a sandbox and it does not delete one. Disconnecting drops the control connection. Deleting destroys the machine and its state. Those are different operations, and the distinction matters when the sandbox contains work that should survive the current process.+Connect mode attaches to an already-running sandbox. It does not create a sandbox and it does not delete one. Disconnecting drops the control connection. Deleting destroys the machine and its state. Those are different operations, and the distinction matters when the sandbox contains work that should survive the current process. [Sandbox lifecycle](/concepts/sandbox-lifecycle) shows the SDK patterns for each lifetime.## Snapshots and forks-A *snapshot* captures the disk state of a running sandbox and returns a new `Image`. That `Image` can start any number of fresh sandboxes from the captured point.+A _snapshot_ captures the disk state of a running sandbox and returns a new `Image`. That `Image` can start any number of fresh sandboxes from the captured point.Snapshots are useful when setup is expensive. Packages can be installed once, models can be downloaded once, and data can be prepared once. After the snapshot is created, new sandboxes can start from that prepared state instead of repeating the setup work.-Each *fork* starts from the same snapshot but gets its own writable disk. Changes in one fork do not affect the snapshot or sibling forks. On copy-on-write storage, forks can be near-instant because unchanged blocks are shared until a sandbox writes to them.+Each _fork_ starts from the same snapshot but gets its own writable disk. Changes in one fork do not affect the snapshot or sibling forks. On copy-on-write storage, forks can be near-instant because unchanged blocks are shared until a sandbox writes to them.## Local executiondiff --git a/docs/content/docs/concepts/index.mdx b/docs/content/docs/concepts/index.mdxindex 608638f86f..6e971b8fe2 100644--- a/docs/content/docs/concepts/index.mdx+++ b/docs/content/docs/concepts/index.mdx@@ -1,8 +1,8 @@----title: "Concepts"-description: "Understand the design ideas behind Cua and how its main pieces fit together."+title: 'Concepts'+description: 'Understand the design ideas behind Cua and how its main pieces fit together.'---Use these pages when you want the model behind Cua rather than a step-by-step guide or API table. They explain what computer use means in Cua, how Cua Driver keeps the desktop usable while it acts, and how Cua Sandbox gives an agent a disposable local computer.-Start with [What is computer use?](/concepts/what-is-computer-use) for the basic model. If you are adding Cua Driver to an agent or product, use [Choose a Cua Driver integration](/concepts/choose-a-cua-driver-integration) before reading the deeper [SDK, MCP, and process hosting](/concepts/sdk-mcp-and-hosting) model. Read [Best-effort background](/concepts/the-no-foreground-contract) to understand Cua Driver's default behavior on a shared machine, then [Capture and delivery modalities](/concepts/capture-and-delivery-modalities) for the action axes. [How we continuously validate Cua Driver](/concepts/how-cua-driver-is-validated) explains why desktop support requires application-owned E2E evidence in addition to unit tests. Read [How sandboxes work](/concepts/how-sandboxes-work) when you need the model for disposable computers. Read [How Lume creates local macOS sandboxes](/concepts/how-lume-unattended-setup-works) for the local VM setup model, and [How SIP works in Lume VMs](/concepts/how-sip-works-in-lume-vms) for signed boot-policy changes. For evaluation, start with [What is Cua-Bench?](/concepts/what-is-cua-bench), then read [The Cua-Bench task lifecycle](/concepts/cua-bench-task-lifecycle).+Start with [What is computer use?](/concepts/what-is-computer-use) for the basic model. If you are adding Cua Driver to an agent or product, use [Choose a Cua Driver integration](/concepts/choose-a-cua-driver-integration) before reading the deeper [SDK, MCP, and process hosting](/concepts/sdk-mcp-and-hosting) model. Read [Best-effort background](/concepts/the-no-foreground-contract) to understand Cua Driver's default behavior on a shared machine, then [Capture and delivery modalities](/concepts/capture-and-delivery-modalities) for the action axes. [How we continuously validate Cua Driver](/concepts/how-cua-driver-is-validated) explains why desktop support requires application-owned E2E evidence in addition to unit tests. Read [How sandboxes work](/concepts/how-sandboxes-work) when you need the model for disposable computers, then [Sandbox lifecycle](/concepts/sandbox-lifecycle) for the lifetime and connection patterns. Read [How Lume creates local macOS sandboxes](/concepts/how-lume-unattended-setup-works) for the local VM setup model, and [How SIP works in Lume VMs](/concepts/how-sip-works-in-lume-vms) for signed boot-policy changes. For evaluation, start with [What is Cua-Bench?](/concepts/what-is-cua-bench), then read [The Cua-Bench task lifecycle](/concepts/cua-bench-task-lifecycle).diff --git a/docs/content/docs/concepts/meta.json b/docs/content/docs/concepts/meta.jsonindex 22bfee672a..5e93b43aa5 100644--- a/docs/content/docs/concepts/meta.json+++ b/docs/content/docs/concepts/meta.json@@ -14,6 +14,7 @@"how-cua-driver-is-validated","---Cua Sandbox---","how-sandboxes-work",+ "sandbox-lifecycle","---Lume---","how-lume-unattended-setup-works","how-sip-works-in-lume-vms",diff --git a/docs/content/docs/how-to-guides/sandbox/lifecycle.mdx b/docs/content/docs/concepts/sandbox-lifecycle.mdxsimilarity index 84%rename from docs/content/docs/how-to-guides/sandbox/lifecycle.mdxrename to docs/content/docs/concepts/sandbox-lifecycle.mdxindex 20973a8cc3..6f62cf18b7 100644--- a/docs/content/docs/how-to-guides/sandbox/lifecycle.mdx+++ b/docs/content/docs/concepts/sandbox-lifecycle.mdx@@ -5,7 +5,7 @@ description: Choose the right Sandbox SDK lifecycle pattern for temporary, persiimport { Callout } from 'fumadocs-ui/components/callout';-Use the **lifecycle pattern** that matches whether you want the sandbox *deleted after the job*, *kept for later*, or *attached to while it is already running*.+Use the **lifecycle pattern** that matches whether you want the sandbox _deleted after the job_, _kept for later_, or _attached to while it is already running_.## Create and auto-destroy a sandbox@@ -43,11 +43,11 @@ async with Sandbox.connect('my-dev-sandbox', local=True) as sb:await sb.destroy()```-| Method | Effect | When |-|--------|--------|------|-| `await sb.disconnect()` | keeps running | reconnect later |-| `await sb.destroy()` | permanently destroyed | done with it |-| `await Sandbox.delete(name)` | permanently destroyed (by name, no connect) | classmethod |+| Method | Effect | When |+| ---------------------------- | ------------------------------------------- | --------------- |+| `await sb.disconnect()` | keeps running | reconnect later |+| `await sb.destroy()` | permanently destroyed | done with it |+| `await Sandbox.delete(name)` | permanently destroyed (by name, no connect) | classmethod |## Reconnect to a running sandboxdiff --git a/docs/content/docs/how-to-guides/recipes/automate-a-legacy-windows-app-behind-a-vpn.mdx b/docs/content/docs/how-to-guides/recipes/automate-a-legacy-windows-app-behind-a-vpn.mdxindex 2897cff71b..94497b7133 100644--- a/docs/content/docs/how-to-guides/recipes/automate-a-legacy-windows-app-behind-a-vpn.mdx+++ b/docs/content/docs/how-to-guides/recipes/automate-a-legacy-windows-app-behind-a-vpn.mdx@@ -79,6 +79,6 @@ When a desktop app has **no API** and only runs **behind the corporate VPN**, dr## Scale this out-When the same workflow must run concurrently, use local Cua Sandbox Windows VMs and cap concurrency to the resources available on the host. Start with [Run sandboxes in parallel](/how-to-guides/sandbox/scale-out) and [Sandbox lifecycle](/how-to-guides/sandbox/lifecycle).+When the same workflow must run concurrently, use local Cua Sandbox Windows VMs and cap concurrency to the resources available on the host. Start with [Run sandboxes in parallel](/how-to-guides/sandbox/scale-out) and [Sandbox lifecycle](/concepts/sandbox-lifecycle).The driving logic stays the same. Each local Windows VM needs the VPN configuration and application included in its starting image. See [Choose and build a sandbox image](/how-to-guides/sandbox/images).diff --git a/docs/content/docs/how-to-guides/recipes/export-contacts-overnight.mdx b/docs/content/docs/how-to-guides/recipes/export-contacts-overnight.mdxindex e0f5001033..a9802f381f 100644--- a/docs/content/docs/how-to-guides/recipes/export-contacts-overnight.mdx+++ b/docs/content/docs/how-to-guides/recipes/export-contacts-overnight.mdx@@ -9,7 +9,9 @@ import { Steps, Step } from 'fumadocs-ui/components/steps';After a networking event, use Cua Driver on the machine where you already have a logged-in browser. The agent inherits that **real authenticated session** from the browser profile, so there are no credentials in the script and no anti-bot fight. Let it run overnight, then wake up to a CSV.<Callout type="info">- Before you start: [install Cua Driver](/how-to-guides/driver/install), grant host permissions, [connect your agent](/how-to-guides/driver/connect-your-agent), and configure [Keep Cua Driver running](/how-to-guides/driver/keep-running).+ Before you start: [install Cua Driver](/how-to-guides/driver/install), grant host permissions,+ [connect your agent](/how-to-guides/driver/connect-your-agent), and configure [Keep Cua Driver+ running](/how-to-guides/driver/keep-running).</Callout><Steps>@@ -18,10 +20,11 @@ After a networking event, use Cua Driver on the machine where you already have a### Log in once-On your own machine, open the browser profile that Cua Driver will drive and log into LinkedIn, X, or both. Do the normal login yourself, including any two-factor prompts. The *session persistence* is the whole trick: the overnight job starts from a browser that is already authenticated.+On your own machine, open the browser profile that Cua Driver will drive and log into LinkedIn, X, or both. Do the normal login yourself, including any two-factor prompts. The _session persistence_ is the whole trick: the overnight job starts from a browser that is already authenticated.<Callout type="warn">- Keep the machine awake for the whole run. Disable sleep, leave the browser profile available, and keep the network connection up so the overnight run does not pause.+ Keep the machine awake for the whole run. Disable sleep, leave the browser profile available, and+ keep the network connection up so the overnight run does not pause.</Callout></Step>@@ -67,4 +70,4 @@ Kick off the task before you leave. In the morning, open `~/contacts.csv` and spOne logged-in browser profile handles one account at a time. To process several accounts, events, or platforms in parallel overnight, use one local Cua Sandbox per authenticated session and cap concurrency to the resources available on the host.-Use [Run sandboxes in parallel](/how-to-guides/sandbox/scale-out) to run several local desktops, [Images](/how-to-guides/sandbox/images) to prepare their starting state, and [Sandbox lifecycle](/how-to-guides/sandbox/lifecycle) for the basic local SDK lifecycle.+Use [Run sandboxes in parallel](/how-to-guides/sandbox/scale-out) to run several local desktops, [Images](/how-to-guides/sandbox/images) to prepare their starting state, and [Sandbox lifecycle](/concepts/sandbox-lifecycle) for the basic local SDK lifecycle.diff --git a/docs/content/docs/how-to-guides/recipes/fill-a-form-from-a-local-file.mdx b/docs/content/docs/how-to-guides/recipes/fill-a-form-from-a-local-file.mdxindex 76fcdb79ee..1713d69c03 100644--- a/docs/content/docs/how-to-guides/recipes/fill-a-form-from-a-local-file.mdx+++ b/docs/content/docs/how-to-guides/recipes/fill-a-form-from-a-local-file.mdx@@ -9,7 +9,8 @@ import { Steps, Step } from 'fumadocs-ui/components/steps';Use this recipe when the data already lives on your machine, for example in a PDF resume or a CSV file, and the form lives in a browser on the same desktop. **Cua Driver** drives both, so the **data never has to be uploaded anywhere** because the agent reads it locally and types it into the form.<Callout type="info">- Before you start: [install Cua Driver](/how-to-guides/driver/install), grant host permissions, and [connect your agent](/how-to-guides/driver/connect-your-agent).+ Before you start: [install Cua Driver](/how-to-guides/driver/install), grant host permissions, and+ [connect your agent](/how-to-guides/driver/connect-your-agent).</Callout><Steps>@@ -25,6 +26,7 @@ Use this recipe when the data already lives on your machine, for example in a PDCua Driver has **no filesystem read tool**. The agent reads local files by opening them in an app it can see, for example Preview, another PDF viewer, or a text editor, then reading the on-screen content with the accessibility tree and screenshots from `get_window_state`.For a CSV, open the file in a text editor, spreadsheet app, or browser tab. If the file contains many rows, start with one row and name the row or record the agent should use.+</Step><Step>@@ -40,6 +42,7 @@ Use this recipe when the data already lives on your machine, for example in a PD```The agent can use `launch_app` to open the PDF viewer, `page` to drive the browser, `get_window_state` or `get_accessibility_tree` to read visible fields, and `click`, `type_text`, or `set_value` to fill the form. `get_window_state` returns a screenshot alongside the tree by default, so the agent can ground on both.+</Step><Step>@@ -50,6 +53,7 @@ Use this recipe when the data already lives on your machine, for example in a PD<Callout type="info">Forms vary, so the agent works best when you name the exact fields to fill, such as name, email, and phone, instead of asking it to fill the whole form.</Callout>+</Step></Steps>@@ -57,4 +61,4 @@ Use this recipe when the data already lives on your machine, for example in a PDOne desktop fills one form at a time. To submit several forms in parallel, for example one per CSV row or one per applicant, run one local Cua Sandbox per job and cap concurrency to the resources available on the host.-Use [Run sandboxes in parallel](/how-to-guides/sandbox/scale-out) to run the jobs concurrently. Use [Choose and build a sandbox image](/how-to-guides/sandbox/images) to stage the source files in an image or mount them into each sandbox. Use [Sandbox lifecycle](/how-to-guides/sandbox/lifecycle) for the basic local SDK lifecycle.+Use [Run sandboxes in parallel](/how-to-guides/sandbox/scale-out) to run the jobs concurrently. Use [Choose and build a sandbox image](/how-to-guides/sandbox/images) to stage the source files in an image or mount them into each sandbox. Use [Sandbox lifecycle](/concepts/sandbox-lifecycle) for the basic local SDK lifecycle.diff --git a/docs/content/docs/how-to-guides/sandbox/meta.json b/docs/content/docs/how-to-guides/sandbox/meta.jsonindex 736da86c58..4dc9659afd 100644--- a/docs/content/docs/how-to-guides/sandbox/meta.json+++ b/docs/content/docs/how-to-guides/sandbox/meta.json@@ -1,7 +1,6 @@{"title": "Sandbox","pages": [- "lifecycle","configure-pool-with-terraform","create-pool-with-python","create-pool-with-typescript",diff --git a/docs/next.config.mjs b/docs/next.config.mjsindex fd2a256d15..281017ea07 100644--- a/docs/next.config.mjs+++ b/docs/next.config.mjs@@ -70,6 +70,11 @@ const config = {destination: '/how-to-guides/sandbox/images',permanent: true,},+ {+ source: '/how-to-guides/sandbox/lifecycle',+ destination: '/concepts/sandbox-lifecycle',+ permanent: true,+ },{source: '/how-to-guides/fleets/configure-run-cua-fleets',destination: '/how-to-guides/sandbox/configure-pool-with-terraform',