Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

updated CLI reference and SDK reference

Dillon DuPont9c8bc7af88bb
diff --git a/docs/content/docs/cuabench/reference/cli-reference.mdx b/docs/content/docs/cuabench/reference/cli-reference.mdx
index 8e75496d60..fee3f20485 100644
--- a/docs/content/docs/cuabench/reference/cli-reference.mdx
+++ b/docs/content/docs/cuabench/reference/cli-reference.mdx
@@ -112,7 +112,7 @@ cb run <subcommand> [options]
| `cb run status <id>` | Show detailed run status |
| `cb run watch <id>` | Watch a run in real-time |
| `cb run stop <id>` | Stop/cancel a run |
-| `cb run logs <id>` | View combined logs from a run |
+| `cb run logs <id>` | View combined logs from a run or session |
### Task Options
@@ -187,6 +187,7 @@ cb run list # List all runs
cb run watch <run_id> # Watch progress in real-time
cb run status <run_id> # Check status
cb run logs <run_id> # View logs
+cb run logs <run_id> --tail 100 # View last 100 lines
cb run stop <run_id> # Stop a run
```
@@ -200,10 +201,15 @@ cb interact <task> [options]
| Option | Description |
|--------|-------------|
-| `--variant-id <id>` | Run specific task variant |
-| `--oracle` | Run oracle solution |
+| `--task-id <id>` | Task ID to run (default: 0) |
+| `--dataset <name>` | Dataset name to resolve task from registry |
+| `--dataset-path <path>` | Path to dataset directory containing multiple tasks |
+| `--oracle` | Run the solution after setup |
+| `--max-steps <n>` | Maximum number of env.step() calls before stopping |
| `--screenshot <path>` | Save screenshot to file |
-| `--dataset <name>` | Use dataset from registry |
+| `--trace-out <path>` | If set, start tracing and save dataset to this path on exit |
+| `--view` | Open a trace viewer when done |
+| `--no-wait` | Skip the interactive prompt (useful for SSH/CI testing) |
## Task Commands
@@ -228,7 +234,7 @@ cb task generate "<description>" [options]
| Option | Description |
|--------|-------------|
-| `--output <path>` | Output directory for generated task |
+| `--output <path>`, `-o` | Output directory path. If not provided, auto-generates from prompt. |
| `--no-interaction` | Skip prompts and run Claude non-interactively |
## Trace Commands
@@ -271,3 +277,45 @@ cb dataset build <outputs> [options]
| `--push-to-hub` | Push to Hugging Face Hub |
| `--repo-id <id>` | Hugging Face repo ID |
| `--private` | Create private repo on Hub |
+
+## Prune Commands
+
+Clean up cua-bench data, images, and Docker resources.
+
+```bash
+cb prune [options]
+```
+
+When called without options, shows an interactive overview of storage usage. Use flags to clean specific resources.
+
+| Option | Description |
+|--------|-------------|
+| `--all`, `-a` | Remove everything (images, overlays, runs, docker) |
+| `--images` | Remove only stored images |
+| `--overlays` | Remove only task overlays |
+| `--runs` | Remove only run logs and registry |
+| `--docker` | Remove only docker containers/images |
+| `--dry-run` | Show what would be deleted without deleting |
+| `--force`, `-f` | Skip confirmation prompts |
+
+### Examples
+
+```bash
+# Show storage overview
+cb prune
+
+# Remove all data with confirmation
+cb prune --all
+
+# Remove all data without confirmation
+cb prune --all --force
+
+# Preview what would be deleted
+cb prune --all --dry-run
+
+# Remove only run logs
+cb prune --runs
+
+# Remove docker resources
+cb prune --docker
+```
diff --git a/docs/content/docs/cuabench/reference/sdk-reference.mdx b/docs/content/docs/cuabench/reference/sdk-reference.mdx
index 05c158a1ca..df637d1c1a 100644
--- a/docs/content/docs/cuabench/reference/sdk-reference.mdx
+++ b/docs/content/docs/cuabench/reference/sdk-reference.mdx
@@ -1,316 +1,270 @@
---
title: SDK Reference
-description: Programming API for writing agents and tasks
+description: Session interface for interacting with desktop environments
---
-# SDK Reference
-
-cua-bench provides a Modal-style Python SDK for writing agents and tasks.
+The cua-bench Session interface provides methods for interacting with desktop environments, whether they're simulated (Playwright-based) or native (Docker/QEMU-based).
## Quick Example
```python
-from cua_bench import Benchmark, Image, Secret, TaskDataset
+from cua_bench import make
-benchmark = Benchmark("my-eval")
-tasks = TaskDataset.from_registry("winarena")
+# Load an environment
+env = make("./my_task")
-image = (
- Image.from_registry("trycua/cua-bench:latest")
- .pip_install("anthropic")
-)
+# Reset to start a task
+screenshot, task_config = await env.reset(task_id=0)
- image=image,
- timeout=600,
- secrets=[Secret.from_name("anthropic-api-key")],
-)
-async def run_task(task_index: int, env) -> dict:
- session, task_config = await env.reset()
-
- for _ in range(50):
- screenshot = await session.screenshot()
- # ... agent logic ...
- await session.click(100, 200)
-
- result = await session.close()
- return {"reward": result["reward"]}
-
[email protected]_entrypoint()
-def main():
- results = run_task.map(
- [i for i, _ in enumerate(tasks)],
- env=[task for task in tasks]
- )
- print(f"Avg reward: {sum(r['reward'] for r in results) / len(results):.2%}")
+# Get the session from the environment
+session = env.session
+
+# Take actions
+await session.click(100, 200)
+await session.type("Hello, world!")
+await session.key("Enter")
+
+# Take a screenshot
+screenshot_bytes = await session.screenshot()
+
+# Close when done
+await env.close()
```
---
-## Core Classes
+## Session Interface
-### Benchmark
+The `DesktopSession` protocol defines the interface for interacting with desktop environments.
-Create a benchmark context for organizing agent functions and tasks.
+### Lifecycle Methods
```python
-from cua_bench import Benchmark
-
-benchmark = Benchmark("my-benchmark-name")
+# Async context manager (preferred)
+async with get_session("native")(os_type="linux") as session:
+ await session.screenshot()
+
+# Manual lifecycle
+session = get_session("native")(os_type="linux")
+await session.start()
+try:
+ await session.screenshot()
+finally:
+ await session.close()
```
-**Decorators:**
-
-| Decorator | Description |
-|-----------|-------------|
-| `@benchmark.function(...)` | Define an agent function |
-| `@benchmark.task(...)` | Define a task with setup/eval |
-| `@benchmark.local_entrypoint()` | Entry point for local runs |
+| Method | Description |
+|--------|-------------|
+| `start(config?, headless?)` | Start the session and connect to the environment |
+| `close()` | Close the session and cleanup resources |
---
-### Image
-
-Build agent runtime environments using a builder pattern:
+### Screenshot & State
```python
-from cua_bench import Image
-
-image = (
- Image.from_registry("trycua/cua-bench:latest")
- .pip_install("anthropic", "openai")
- .apt_install("ffmpeg", "imagemagick")
- .run_commands("python -c 'import anthropic'")
- .copy_local_dir("./src", "/app/src")
- .env("MY_VAR", "value")
-)
-```
+# Capture screenshot
+screenshot_bytes = await session.screenshot() # PNG bytes
-**Builder Methods:**
+# Get desktop state snapshot
+snapshot = await session.get_snapshot()
+# Returns: Snapshot(windows: List[WindowSnapshot])
+```
-| Method | Description |
-|--------|-------------|
-| `Image.from_registry(name)` | Start from a base Docker image |
-| `.pip_install(*packages)` | Install Python packages |
-| `.apt_install(*packages)` | Install system packages (Debian/Ubuntu) |
-| `.run_commands(*commands)` | Run shell commands during build |
-| `.copy_local_dir(src, dst)` | Copy local directory into image |
-| `.env(key, value)` | Set environment variable |
+| Method | Return Type | Description |
+|--------|-------------|-------------|
+| `screenshot()` | `bytes` | Capture screen as PNG bytes |
+| `get_snapshot()` | `Snapshot` | Get lightweight snapshot of desktop state (windows, geometry, metadata) |
---
-### Secret
-
-Reference API keys securely:
+### Mouse Actions
```python
-from cua_bench import Secret
+# Click actions
+await session.click(x, y) # Left click
+await session.right_click(x, y) # Right click
+await session.double_click(x, y) # Double click
+await session.middle_click(x, y) # Middle click
+
+# Movement and drag
+await session.move_to(x, y) # Move cursor
+await session.drag(from_x, from_y, to_x, to_y) # Drag gesture
+
+# Scroll
+await session.scroll(direction="down", amount=100) # Scroll up/down
+```
-# Reference by name (configured in .cua/secrets.yaml)
-anthropic_key = Secret.from_name("anthropic-api-key")
+| Method | Parameters | Description |
+|--------|------------|-------------|
+| `click(x, y)` | x: int, y: int | Left click at coordinates |
+| `right_click(x, y)` | x: int, y: int | Right click at coordinates |
+| `double_click(x, y)` | x: int, y: int | Double click at coordinates |
+| `middle_click(x, y)` | x: int, y: int | Middle click at coordinates |
+| `move_to(x, y)` | x: int, y: int | Move cursor to coordinates |
+| `drag(from_x, from_y, to_x, to_y)` | from_x: int, from_y: int, to_x: int, to_y: int | Drag from one position to another |
+| `scroll(direction, amount)` | direction: "up" \| "down", amount: int | Scroll in given direction |
-# Reference from environment variable
-openai_key = Secret.from_env("OPENAI_API_KEY")
+---
- image=image,
- secrets=[anthropic_key, openai_key],
-)
-async def run_task(task_index: int, env) -> dict:
- import os
- api_key = os.environ["ANTHROPIC_API_KEY"]
-```
+### Keyboard Actions
-**Methods:**
+```python
+# Type text
+await session.type("Hello, world!")
+
+# Press single key
+await session.key("Enter")
+await session.key("Escape")
+await session.key("Tab")
+
+# Key combinations (hotkeys)
+await session.hotkey(["ctrl", "c"]) # Copy
+await session.hotkey(["ctrl", "v"]) # Paste
+await session.hotkey(["ctrl", "shift", "t"]) # Multiple modifiers
+```
-| Method | Description |
-|--------|-------------|
-| `Secret.from_name(name)` | Reference a named secret |
-| `Secret.from_env(var)` | Reference an environment variable |
+| Method | Parameters | Description |
+|--------|------------|-------------|
+| `type(text)` | text: str | Type a string of text |
+| `key(key)` | key: str | Press a single key (e.g., "Enter", "Escape", "Tab") |
+| `hotkey(keys)` | keys: List[str] | Press a key combination (e.g., ["ctrl", "c"]) |
---
-### TaskDataset
-
-Load task definitions from the registry or local paths:
+### Window Management
```python
-from cua_bench import TaskDataset
-
-# Load from registry
-tasks = TaskDataset.from_registry("winarena")
-tasks = TaskDataset.from_registry("osworld")
+# Launch a window
+pid = await session.launch_window(
+ url="https://example.com",
+ title="My Window",
+ width=800,
+ height=600
+)
-# Load from local path
-tasks = TaskDataset.from_path("./my_tasks/")
+# Execute JavaScript in a window
+result = await session.execute_javascript(pid, "document.title")
-# Filter tasks
-tasks = tasks.filter(domain="chrome")
-tasks = tasks.filter(difficulty="easy")
-tasks = tasks[:10] # First 10 tasks
+# Get element position within a window
+rect = await session.get_element_rect(pid, ".button-class")
+# Returns: {"x": 10, "y": 20, "width": 100, "height": 30}
-# Iterate
-for task in tasks:
- print(task.id, task.description)
+# Close all windows
+await session.close_all_windows()
```
-**Methods:**
-
-| Method | Description |
-|--------|-------------|
-| `TaskDataset.from_registry(name)` | Load from Cua registry |
-| `TaskDataset.from_path(path)` | Load from local directory |
-| `.filter(**kwargs)` | Filter by attributes |
-| `[start:end]` | Slice tasks |
-| `len(tasks)` | Get task count |
+| Method | Parameters | Return Type | Description |
+|--------|------------|-------------|-------------|
+| `launch_window(url?, html?, folder?, ...)` | Various options for window creation | `int \| str` | Launch a window and return its process ID |
+| `execute_javascript(pid, js)` | pid: int \| str, js: str | `Any` | Execute JavaScript in a specific window |
+| `get_element_rect(pid, selector)` | pid: int \| str, selector: str | `dict \| None` | Get element rectangle in window or screen space |
+| `close_all_windows()` | - | `None` | Close or clear all open windows |
---
-### Session Interface
+### Action System
-Interact with the desktop environment:
+For more advanced use cases, you can use the action dataclasses directly:
```python
[email protected](image=image)
-async def run_task(task_index: int, env) -> dict:
- session, task_config = await env.reset()
-
- # Screenshot
- screenshot_bytes = await session.screenshot() # PNG bytes
- screenshot_b64 = await session.screenshot(format="base64")
-
- # Mouse actions
- await session.click(x, y)
- await session.right_click(x, y)
- await session.double_click(x, y)
- await session.drag(from_x, from_y, to_x, to_y)
- await session.scroll(x, y, direction="down", amount=3)
- await session.move_to(x, y)
-
- # Keyboard actions
- await session.type("Hello, world!")
- await session.key("Enter")
- await session.hotkey(["ctrl", "c"])
-
- # Advanced
- snapshot = await session.get_snapshot() # Window info
- pid = await session.launch_window("https://example.com")
- result = await session.execute_javascript(pid, "document.title")
-
- # Finalize
- result = await session.close() # Runs evaluation
- return {"reward": result["reward"]}
+from cua_bench.types import (
+ ClickAction,
+ RightClickAction,
+ DoubleClickAction,
+ MiddleClickAction,
+ DragAction,
+ MoveToAction,
+ ScrollAction,
+ TypeAction,
+ KeyAction,
+ HotkeyAction,
+ WaitAction,
+ DoneAction,
+)
+
+# Execute an action directly
+await session.execute_action(ClickAction(x=100, y=200))
+await session.execute_action(TypeAction(text="Hello"))
+await session.execute_action(HotkeyAction(keys=["ctrl", "c"]))
```
-**Session Methods:**
-
-| Method | Description |
-|--------|-------------|
-| `screenshot(format?)` | Capture screen (PNG bytes or base64) |
-| `click(x, y)` | Left click at coordinates |
-| `right_click(x, y)` | Right click |
-| `double_click(x, y)` | Double click |
-| `drag(x1, y1, x2, y2)` | Drag gesture |
-| `scroll(x, y, direction, amount)` | Scroll at position |
-| `move_to(x, y)` | Move cursor |
-| `type(text)` | Type text |
-| `key(key)` | Press single key |
-| `hotkey(keys)` | Key combination (e.g., `["ctrl", "c"]`) |
-| `get_snapshot()` | Get desktop state |
-| `launch_window(url)` | Launch browser window |
-| `execute_javascript(pid, js)` | Run JS in window |
-| `close()` | Finalize and get result |
+**Available Action Types:**
+
+| Action Class | Fields | Description |
+|--------------|--------|-------------|
+| `ClickAction` | x: int, y: int | Left click at position |
+| `RightClickAction` | x: int, y: int | Right click at position |
+| `DoubleClickAction` | x: int, y: int | Double click at position |
+| `MiddleClickAction` | x: int, y: int | Middle click at position |
+| `DragAction` | from_x: int, from_y: int, to_x: int, to_y: int, duration: float | Drag gesture |
+| `MoveToAction` | x: int, y: int, duration: float | Move cursor |
+| `ScrollAction` | direction: "up" \| "down", amount: int | Scroll action |
+| `TypeAction` | text: str | Type text |
+| `KeyAction` | key: str | Press single key |
+| `HotkeyAction` | keys: List[str] | Key combination |
+| `WaitAction` | seconds: float | Wait/pause |
+| `DoneAction` | - | Signal completion |
---
-## Execution Patterns
+## Provider Types
-Agent functions support multiple execution patterns:
+cua-bench supports two provider types:
-```python
[email protected](image=image, timeout=600)
-async def run_task(task_index: int, env) -> dict:
- ...
-
[email protected]_entrypoint()
-def main():
- # Single local execution
- result = run_task.local(task_index=0, env=tasks[0])
-
- # Single remote execution
- result = run_task.remote(task_index=0, env=tasks[0])
-
- # Parallel execution (recommended)
- results = run_task.map(
- [i for i, _ in enumerate(tasks)],
- env=[task for task in tasks]
- )
-
- # With concurrency limit
- results = run_task.map(
- task_indices,
- env=task_configs,
- concurrency=10,
- )
-```
+### Simulated Desktop (Playwright-based)
-**Execution Methods:**
+- **Name:** `"simulated"` or `"webtop"`
+- **Description:** HTML/CSS desktop simulation in headless browser
+- **Pros:** Fast, no Docker required
+- **Use cases:** Web app testing, UI benchmarks
-| Method | Description |
-|--------|-------------|
-| `.local(...)` | Run locally in current process |
-| `.remote(...)` | Run single task remotely |
-| `.map(...)` | Run multiple tasks in parallel |
+```python
+from cua_bench.computers.base import get_session
----
+SessionClass = get_session("simulated")
+session = SessionClass(env=env)
+```
-## Task Definition
+### Native Desktop (Docker/QEMU-based)
-Define tasks using a generator pattern with `yield`:
+- **Name:** `"native"` or `"computer"`
+- **Description:** Real OS in Docker/QEMU container
+- **Pros:** Actual desktop environment with real applications
+- **Requires:** Docker
+- **Use cases:** Real app testing, OS-level tasks
```python
-from cua_bench import benchmark, Demonstrations
+from cua_bench.computers.base import get_session
- environment="windows-qemu",
- oracle=Demonstrations.from_name("my-demo"),
- timeout=600,
-)
-async def my_task(session):
- # ─── SETUP (runs on env.reset()) ───
- import subprocess
- subprocess.run(["powershell", "-Command", "Start-Process notepad"])
-
- # ─── YIELD TASK DESCRIPTION ───
- yield {
- "description": "Type 'Hello World' in Notepad",
- "hints": ["Use Ctrl+S to save"],
- }
-
- # ─── EVALUATION (runs on session.close()) ───
- import os
- if os.path.exists("C:/Users/User/Desktop/hello.txt"):
- return {"success": True, "reward": 1.0}
-
- return {"success": False, "reward": 0.0}
+SessionClass = get_session("native")
+session = SessionClass(env=env)
```
-**Task Decorator Options:**
+---
-| Option | Type | Description |
-|--------|------|-------------|
-| `environment` | string | Platform type |
-| `oracle` | Demonstrations | Optional oracle demonstration |
-| `timeout` | int | Task timeout in seconds |
-| `tags` | list | Task tags for filtering |
+## Data Types
-**Task Return Value:**
+### Snapshot
```python
-{
- "success": bool, # Whether task succeeded
- "reward": float, # 0.0 to 1.0
- "reason": str, # Optional explanation
- "details": dict, # Optional extra data
-}
+@dataclass
+class WindowSnapshot:
+ window_type: Literal["webview", "process", "desktop"]
+ pid: Optional[str] = None
+ url: Optional[str] = None
+ html: Optional[str] = None
+ title: str = ""
+ x: int = 0
+ y: int = 0
+ width: int = 0
+ height: int = 0
+ active: bool = False
+ minimized: bool = False
+
+@dataclass
+class Snapshot:
+ windows: List[WindowSnapshot]
```
+
+The `get_snapshot()` method returns a `Snapshot` object containing information about all open windows, their positions, sizes, and states.