Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
docs: update architecture images and improve documentation (#827)
Francesco Bonacci0b9cf2ff62f2
diff --git a/README.md b/README.mdindex 282cacb12a..afc4ed772f 100644--- a/README.md+++ b/README.md@@ -7,7 +7,7 @@</picture></a>- <p align="center">Any OS. Any agent. Self-hostable.</p>+ <p align="center">Build, benchmark, and deploy agents that use computers</p><p align="center"><a href="https://cua.ai" target="_blank" rel="noopener noreferrer"><img src="https://img.shields.io/badge/cua.ai-0ea5e9" alt="cua.ai"></a>@@ -28,11 +28,11 @@## Choose Your Path-| If you want to... | Check out | For |-| ---------------------------------------------- | --------------------------------------------------------- | ---------------------------- |-| Build AI agents or run isolated code execution | [**Cua**](#cua---agentic-ui-automation--code-execution) | AI Engineers |-| Benchmark or train computer-use models with RL | [**Cua-Bench**](#cua-bench---benchmarks--rl-environments) | Researchers |-| Run macOS/Linux VMs on Apple Silicon | [**Lume**](#lume---macos-virtualization) | Developers, System Engineers |+| | If you want to... | Check out |+| - | ---------------------------------------------- | --------------------------------------------------------- |+| [<img src="img/cua-architecture.png" width="150">](#cua---agentic-ui-automation--code-execution) | Build AI agents or run isolated code execution | [**Cua**](#cua---agentic-ui-automation--code-execution) |+| [<img src="img/cua-bench-architecture.png" width="150">](#cua-bench---benchmarks--rl-environments) | Benchmark or train computer-use models with RL | [**Cua-Bench**](#cua-bench---benchmarks--rl-environments) |+| [<img src="img/lume-architecture.png" width="150">](#lume---macos-virtualization) | Run macOS/Linux VMs on Apple Silicon | [**Lume**](#lume---macos-virtualization) |---@@ -40,6 +40,8 @@Build agents that see screens, click buttons, and complete tasks autonomously. Run isolated code execution environments for AI coding assistants like Claude Code, Codex CLI, or OpenCode.+<img src="img/cua-architecture.png" alt="Cua Architecture" width="100%">+```python# Requires Python 3.12 or 3.13from computer import Computer@@ -60,6 +62,8 @@ async for result in agent.run([{"role": "user", "content": "Open Firefox and seaEvaluate computer-use agents on OSWorld, ScreenSpot, Windows Arena, and custom tasks. Export trajectories for training.+<img src="img/cua-bench-architecture.png" alt="Cua-Bench Architecture" width="100%">+```bash# Install and create base imagecd cua-bench@@ -77,6 +81,8 @@ cb run dataset datasets/cua-bench-basic --agent cua-agent --max-parallel 4Create and manage macOS/Linux VMs with near-native performance on Apple Silicon using Apple's Virtualization.Framework.+<img src="img/lume-architecture.png" alt="Lume Architecture" width="100%">+```bash# Install Lume/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/lume/scripts/install.sh)"diff --git a/docs/content/docs/cua/guide/get-started/what-is-cua.mdx b/docs/content/docs/cua/guide/get-started/what-is-cua.mdxindex 5f9b64eef8..b25568f097 100644--- a/docs/content/docs/cua/guide/get-started/what-is-cua.mdx+++ b/docs/content/docs/cua/guide/get-started/what-is-cua.mdx@@ -3,7 +3,7 @@ title: What is Cua?description: Learn about Cua, the open-source platform for building computer-use agents----Cua is an **open-source platform** for building, deploying, and managing computer-use agents. It provides everything you need to create AI agents that can interact with desktop environments just like a human would—clicking, typing, navigating, and completing tasks autonomously.+Cua is an open-source platform for building, benchmarking, and deploying agents that can use any computer, with isolated, self-hostable sandboxes (Docker, QEMU, Apple Vz).<div className="not-prose relative rounded-xl overflow-hidden my-8 w-full"><img src="/docs/img/hero.png" alt="Cua" className="w-full h-auto rounded-xl" />@@ -13,6 +13,10 @@ Cua is an **open-source platform** for building, deploying, and managing computeCua consists of three main components:+<div className="not-prose relative rounded-xl overflow-hidden my-8 w-full">+ <img src="/docs/img/cua-architecture.png" alt="Cua Architecture" className="w-full h-auto rounded-xl" />+</div>+### 1. Desktop SandboxesIsolated virtual environments where your agents can safely execute tasks. Cua supports:@@ -27,11 +31,12 @@ A unified SDK for controlling desktop environments programmatically:- Take screenshots and observe the screen- Simulate mouse clicks, movements, and scrolling- Type text and press keyboard shortcuts+- Run code and shell commands- Works identically across all sandbox types### 3. Agent Framework-Tools for building AI agents that understand and interact with GUIs:+Build agents that see screens, click buttons, and complete tasks autonomously. Run isolated code execution environments for AI coding assistants like Claude Code, Codex CLI, or OpenCode.- **100+ vision-language model options** through Cua VLM Router or direct provider access- **Pre-built agent loops** optimized for computer-use tasks@@ -40,24 +45,23 @@ Tools for building AI agents that understand and interact with GUIs:## Why Cua?-Traditional automation tools require you to write explicit scripts for every action. Cua agents can:+Cua provides isolated, reproducible environments for AI agents to operate safely:-- **Understand visual context** - See what's on screen and make decisions-- **Adapt to changes** - Handle UI variations without breaking-- **Complete complex tasks** - Chain multiple actions to achieve goals-- **Work across applications** - Navigate between different programs seamlessly+- **Secure execution** - Run AI coding assistants and computer-use agents in sandboxed environments+- **Self-hostable** - Deploy locally with Docker, QEMU, or Apple Virtualization+- **Cross-platform** - Same API across Linux, Windows, and macOS sandboxes+- **Visual understanding** - Agents see screens, adapt to UI changes, and complete complex tasks## Use CasesCua is ideal for:+- **AI coding assistants** - Isolated code execution environments for Claude Code, Codex CLI, OpenCode, and other AI coding tools+- **Computer-use agents** - Build agents that interact with any desktop application autonomously- **Workflow automation** - Automate repetitive tasks across any application- **Testing** - Run end-to-end tests that interact with real UIs-- **Data extraction** - Scrape information from applications that don't have APIs-- **RPA (Robotic Process Automation)** - Enterprise automation with AI understanding-- **Benchmarks** - Evaluate computer-use agents on standardized tasks (OSWorld, ScreenSpot, etc.)-- **RL/Gym Environments** - Train reinforcement learning agents with desktop environments as Gym-compatible envs-- **Research** - Build and evaluate computer-use AI agents+- **Benchmarks** - Evaluate agents on OSWorld, ScreenSpot, and other standardized tasks+- **Research** - Build, evaluate, and train computer-use AI agents## Getting Starteddiff --git a/docs/content/docs/cuabench/guide/getting-started/introduction.mdx b/docs/content/docs/cuabench/guide/getting-started/introduction.mdxindex 9c042a793a..f1f4618c6a 100644--- a/docs/content/docs/cuabench/guide/getting-started/introduction.mdx+++ b/docs/content/docs/cuabench/guide/getting-started/introduction.mdx@@ -19,10 +19,19 @@ Examples of Cua-Bench tasks are:- Installing design software and editing images to remove backgrounds- Using office applications to format and merge multiple documents-Cua-Bench consists of two parts:+Cua-Bench consists of three parts:-1. **A dataset of tasks** - Verifiable, dynamic computer-use environments across multiple operating systems-2. **An evaluation/training harness** - Tools for running benchmarks, generating data, and testing agents+<div className="not-prose relative rounded-xl overflow-hidden my-8 w-full">+ <img+ src="/docs/img/cua-bench-architecture.png"+ alt="Cua-Bench Architecture"+ className="w-full h-auto rounded-xl"+ />+</div>++1. **Base images** - Pre-packaged Windows, Linux, macOS, and Android environments with apps and dependencies needed for benchmark runs+2. **A dataset of tasks** - Verifiable, dynamic computer-use environments across multiple operating systems+3. **An evaluation/training harness** - Tools for running benchmarks, generating data, and testing agents# Dataset of tasksdiff --git a/docs/content/docs/lume/guide/getting-started/introduction.mdx b/docs/content/docs/lume/guide/getting-started/introduction.mdxindex 8a98f167bc..4ed66d3883 100644--- a/docs/content/docs/lume/guide/getting-started/introduction.mdx+++ b/docs/content/docs/lume/guide/getting-started/introduction.mdx@@ -7,6 +7,10 @@ import { Callout } from 'fumadocs-ui/components/callout';Lume is a VM runtime for building AI agents, running CI/CD pipelines, and automating macOS. It uses Apple's native Virtualization Framework to run macOS and Linux VMs at near-native speed on Apple Silicon.+<Callout type="info" title="Cloud macOS Sandboxes">+We're piloting a managed service for customers who want to run cloud macOS sandboxes for CI/CD and agent workloads. [Book a demo](https://cal.com/cua/cua-demo?overlayCalendar=true) if you're interested.+</Callout>+```bashlume create test-vm --os macos --ipsw latestlume run test-vm@@ -49,9 +53,7 @@ Apple's Virtualization Framework—the same technology Lume is built on—powers## What Lume doesn't do-Lume runs VMs locally on your Mac. It's not a cloud service, container runtime, or cluster orchestrator. If you need to run VMs in the cloud or across multiple machines, look elsewhere.--Lume also requires Apple Silicon—it won't work on Intel Macs or other platforms.+Lume requires Apple Silicon—it won't work on Intel Macs or other platforms.## Get starteddiff --git a/docs/public/img/cua-architecture.png b/docs/public/img/cua-architecture.pngnew file mode 100644index 0000000000..7276f43e15Binary files /dev/null and b/docs/public/img/cua-architecture.png differdiff --git a/docs/public/img/cua-bench-architecture.png b/docs/public/img/cua-bench-architecture.pngnew file mode 100644index 0000000000..b937a2314aBinary files /dev/null and b/docs/public/img/cua-bench-architecture.png differdiff --git a/docs/public/img/hero.png b/docs/public/img/hero.pngindex 17cb6e1e8e..f76a84d8d8 100644Binary files a/docs/public/img/hero.png and b/docs/public/img/hero.png differdiff --git a/docs/public/img/lume-architecture.png b/docs/public/img/lume-architecture.pngindex 79f1d90078..c1661aa712 100644Binary files a/docs/public/img/lume-architecture.png and b/docs/public/img/lume-architecture.png differdiff --git a/docs/src/app/global.css b/docs/src/app/global.cssindex 53c9ad28c2..124bb67330 100644--- a/docs/src/app/global.css+++ b/docs/src/app/global.css@@ -11,6 +11,12 @@--color-fd-accent: hsl(152, 76%, 92%); /* emerald-100 */--font-sans: var(--font-geist-sans);--font-mono: var(--font-geist-mono);+ --font-heading: var(--font-urbanist);+}++/* Urbanist for headings */+h1, h2, h3, h4 {+ font-family: var(--font-heading), var(--font-sans), sans-serif;}.dark {diff --git a/img/cua-architecture.png b/img/cua-architecture.pngnew file mode 100644index 0000000000..7276f43e15Binary files /dev/null and b/img/cua-architecture.png differdiff --git a/img/cua-bench-architecture.png b/img/cua-bench-architecture.pngnew file mode 100644index 0000000000..b937a2314aBinary files /dev/null and b/img/cua-bench-architecture.png differdiff --git a/img/lume-architecture.png b/img/lume-architecture.pngnew file mode 100644index 0000000000..c1661aa712Binary files /dev/null and b/img/lume-architecture.png differ