Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(skills): rewrite cua-do-cli as task-oriented gui-automation skill (#1121)
Sarina Lie2cddf51c2a5
diff --git a/skills/cua-do-cli/SKILL.md b/skills/cua-do-cli/SKILL.mddeleted file mode 100644index dde6189d10..0000000000--- a/skills/cua-do-cli/SKILL.md+++ /dev/null@@ -1,146 +0,0 @@-----name: cua-do-cli-description: Automates interactions with remote VMs, local Docker containers, cloud sandboxes, and the host PC via cua do. Use when navigating GUIs, taking snapshots/screenshots, clicking, typing, scrolling, running shell commands, or managing windows on a target machine.------# GUI Automation with cua do--One command = one action = one ✅/❌ line. Switch targets once, then operate.--## Start Here--1. Pick a flow:- - API key available: switch → snapshot → interact → snapshot- - No API key: switch → screenshot → read image → interact → screenshot- - Host control: consent → switch host → operate-2. Run the canonical flow below.-3. Check the command skeleton only if blocked.--## Decision Map--- No target set: `switch` → then operate-- **ANTHROPIC_API_KEY set** (normal): `switch` → `snapshot` → `click/type/key` → repeat snapshot after each UI change-- **No API key**: `switch` → `screenshot` → read image to determine coords → `click/type/key` → repeat screenshot-- Zoom to one window: `zoom "App Name"` → operate → `unzoom`-- Run shell: `shell "command"`-- Control host PC: `do-host-consent` (once) → `switch host` → operate--## Canonical Flows--### 1) Normal (ANTHROPIC_API_KEY set)--```bash-cua do switch docker my-container-cua do snapshot # AI summary + interactive elements with coords-cua do click 450 300-cua do type "hello"-cua do key enter-cua do snapshot # re-snapshot after UI change-```--### 2) No API Key--```bash-cua do switch docker my-container-cua do screenshot # save path printed; read the image to find coords-cua do click 450 300-cua do type "hello"-cua do key enter-cua do screenshot # re-screenshot after UI change-```--### 3) Focused Window (Zoom)--```bash-cua do zoom "Chrome"-cua do snapshot # cropped to Chrome; coords are window-relative-cua do click 120 80-cua do unzoom-```--### 4) Host PC Control--```bash-cua do-host-consent # one-time consent, no prompt-cua do switch host-cua do snapshot-cua do click 100 200-```--## Command Skeleton--### Target--```bash-cua do switch <provider> [name] # cloud, docker, lume, lumier, winsandbox, host-cua do status-cua do ls [provider]-```--### Snapshot / Screenshot--```bash-cua do snapshot ["extra instructions"] # screenshot + AI summary (needs ANTHROPIC_API_KEY)-cua do screenshot [--save path]-cua do zoom "Window Name"-cua do unzoom-```--### Input--```bash-cua do click <x> <y> [left|right|middle]-cua do dclick <x> <y>-cua do move <x> <y>-cua do type "text"-cua do key <key> # enter, escape, tab, space, f1–f12, …-cua do hotkey ctrl+c-cua do scroll <up|down|left|right> [n]-cua do drag <x1> <y1> <x2> <y2>-```--### Shell / Open--```bash-cua do shell "command"-cua do open <url|path>-```--### Window--```bash-cua do window ls [app]-cua do window focus <id>-cua do window unfocus-cua do window minimize/maximize/close <id>-cua do window resize <id> <w> <h>-cua do window move <id> <x> <y>-cua do window info <id>-```--## Guardrails--- Re-snapshot after navigation, modals, or list changes — coords go stale.-- `snapshot` needs `ANTHROPIC_API_KEY`; use `screenshot` otherwise.-- Coords are image-space: zoom + max-length scaling are applied automatically.-- `do-host-consent` is permanent until the consent file is deleted (`~/.cua/host_consented`).-- Set `PYTHONIOENCODING=utf-8` on Windows for correct emoji output.--## Trajectory Recording--Every `cua do` action is automatically recorded to a replayable trajectory.--- Stored at `~/.cua/trajectories/{machine_name}/{session_timestamp}/`-- New session starts on `cua do switch`-- Disable with `--no-record`: `cua do --no-record click 100 200`-- After a session, mention the trajectory to the user:- "Your session was recorded. Run `cua trajectory view` to replay."--```bash-cua trajectory ls # list all sessions-cua trajectory view # start local viewer for latest session-cua trajectory view my-container # latest for specific machine-cua trajectory view --port 9090 # use custom port-cua trajectory stop # stop the viewer server-cua trajectory clean --older-than 7 # delete old sessions-```diff --git a/skills/gui-automation/SKILL.md b/skills/gui-automation/SKILL.mdnew file mode 100644index 0000000000..e4908babec--- /dev/null+++ b/skills/gui-automation/SKILL.md@@ -0,0 +1,263 @@+---+name: gui-automation+description: >-+ Use when you need to visually interact with a GUI — test buttons, fill forms,+ verify visual layouts, fuzz web pages, automate user flows, take screenshots,+ or perform end-to-end QA on any application. Works on cloud VMs, Docker+ containers, local machines, and sandboxes. Install: pip install cua.+---++# GUI Automation++CUA gives you **eyes and hands on a real computer**. You can see the screen,+move the mouse, click buttons, type text, drag elements, and manage windows —+exactly like a human sitting at the keyboard.++Use this skill whenever the task requires **visual interaction** that cannot be+done through a shell command or API call: testing a UI, filling a form, verifying+a layout, fuzzing a web page, or driving an end-to-end user flow.++Every action you take is automatically recorded to a replayable trajectory.+When you finish, share the trajectory link so the user can review your work.++## Prerequisites++Before your first action, ensure the CLI is available:++```bash+# Check if cua is installed+cua --version++# If missing, install it+pip install cua+```++**Target setup** — connect to the machine you will automate:++```bash+# Cloud VM+cua do switch cloud my-vm++# Docker container+cua do switch docker my-container++# Local machine (requires one-time consent)+cua do-host-consent+cua do switch host+```++> **ANTHROPIC_API_KEY** is optional. If set, `cua do snapshot` returns an+> AI-annotated summary with element coordinates. Without it, use+> `cua do screenshot` and read the image yourself to determine coordinates.+> Both approaches work — snapshot is a convenience, not a requirement.++## Common Scenarios++### 1. Test a button++> "Click the Export PDF button and verify the download starts."++```bash+cua do switch cloud my-vm+cua do screenshot # see the current screen+# (read the image to locate the Export PDF button)+cua do click 450 280 # click it+cua do screenshot # confirm: did a download dialog appear?+```++### 2. Fill a form and submit++> "Fill out the signup form with test data and submit it."++```bash+cua do screenshot # see what fields are present+cua do click 400 200 # click the Name field+cua do type "Jane Doe"+cua do key tab+cua do type "[email protected]"+cua do key tab+cua do type "SecureP@ss123"+cua do click 400 500 # click Submit+cua do screenshot # verify success message+```++### 3. Handle a file upload dialog++> "Upload report.pdf through the file picker."++```bash+cua do screenshot # locate the upload button+cua do click 350 400 # click "Choose File"+cua do screenshot # file picker is now open+cua do type "/home/user/report.pdf" # type the file path+cua do key enter # confirm selection+cua do screenshot # verify the file name appears+```++### 4. Visual verification++> "Check that the dashboard chart renders correctly after filtering."++```bash+cua do screenshot # baseline state+cua do click 600 120 # click the date filter dropdown+cua do screenshot # see dropdown options+cua do click 600 180 # select "Last 7 days"+cua do screenshot # verify: chart updated, no blank areas, labels correct+```++### 5. End-to-end user flow++> "Walk through the checkout flow from cart to confirmation."++```bash+cua do screenshot # see the cart page+cua do click 700 500 # click "Proceed to Checkout"+cua do screenshot # shipping form+cua do click 400 200+cua do type "123 Main St"+cua do key tab+cua do type "Springfield"+cua do key tab+cua do type "62701"+cua do click 700 500 # click "Continue to Payment"+cua do screenshot # payment form+cua do click 400 200+cua do type "4111111111111111"+cua do key tab+cua do type "12/28"+cua do key tab+cua do type "123"+cua do click 700 500 # click "Place Order"+cua do screenshot # confirmation page — verify order number present+```++### 6. Cross-window drag and drop++> "Drag the file icon from the file manager into the upload zone in Chrome."++```bash+cua do window ls # list open windows with IDs+cua do screenshot # see both windows+cua do drag 150 300 650 400 # drag from file manager → upload zone+cua do screenshot # verify the file landed in the drop zone+```++### 7. Fuzz a web page++> "Try unexpected inputs on every text field in the form."++```bash+cua do screenshot # see the form+cua do click 400 200 # first field+cua do type "<script>alert(1)</script>"+cua do key tab+cua do type "'; DROP TABLE users; --"+cua do key tab+cua do type "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"+cua do click 400 500 # submit+cua do screenshot # check for errors, crashes, or unexpected behavior+```++## Workflow Pattern++Every GUI task follows the same loop:++1. **Connect** — `cua do switch <provider> <name>`+2. **Look** — `cua do screenshot` (or `snapshot` if API key is set)+3. **Act** — `click`, `type`, `key`, `drag`, `scroll`, etc.+4. **Verify** — `cua do screenshot` again to confirm the result+5. **Repeat** — continue until the task is done+6. **Share** — `cua do trajectory share` to give the user a replay link++> Always re-screenshot after any action that changes the UI. Coordinates go+> stale the moment the screen changes.++## Trajectory Recording and Sharing++Every `cua do` action is automatically recorded to a replayable trajectory at+`~/.cua/trajectories/{machine}/{session}/`. A new session starts each time you+run `cua do switch`.++### Sharing your work++When you finish a task, **always share the trajectory** so the user can replay+and review exactly what happened:++```bash+cua trajectory share # upload and get a shareable HTTPS link+```++Tell the user:++> "Here is the trajectory of my session: {url}"++### Other trajectory commands++```bash+cua trajectory ls # list all recorded sessions+cua trajectory view # open latest session in browser (local)+cua trajectory view my-container # view latest for a specific machine+cua trajectory export # generate a self-contained HTML report+cua trajectory export -o report.html # export to a specific path+cua trajectory clean --older-than 7 # delete sessions older than 7 days+cua trajectory stop # stop the local viewer server+```++### Disabling recording++If the user asks you not to record, pass `--no-record`:++```bash+cua do --no-record click 100 200+```++## Guardrails++- **Re-screenshot after every UI change.** Coordinates are image-space pixels+ from the last screenshot. If the screen changed, your coordinates are wrong.+- **snapshot vs screenshot.** `snapshot` calls an AI model and requires+ `ANTHROPIC_API_KEY`. If the key is missing, fall back to `screenshot` and+ read the image yourself. Both work.+- **Coordinates are image-space.** Zoom and max-length scaling are applied+ automatically — use the pixel coordinates from your most recent screenshot.+- **Host consent.** Controlling the local machine requires a one-time+ `cua do-host-consent` before `switch host`.+- **Window zoom.** Use `cua do zoom "App Name"` to crop screenshots to a+ single window. Coordinates become window-relative. `cua do unzoom` to reset.++## Quick Reference++| Action | Command |+| ------------------- | -------------------------------------------- |+| Connect to target | `cua do switch <provider> [name]` |+| See the screen | `cua do screenshot` |+| AI-annotated screen | `cua do snapshot ["instructions"]` |+| Click | `cua do click <x> <y> [left\|right\|middle]` |+| Double-click | `cua do dclick <x> <y>` |+| Type text | `cua do type "text"` |+| Press key | `cua do key <key>` |+| Hotkey | `cua do hotkey <combo>` (e.g. `cmd+c`) |+| Scroll | `cua do scroll <direction> [amount]` |+| Drag | `cua do drag <x1> <y1> <x2> <y2>` |+| Move cursor | `cua do move <x> <y>` |+| Run shell command | `cua do shell "command"` |+| Open URL or file | `cua do open <url\|path>` |+| List windows | `cua do window ls [app]` |+| Focus window | `cua do window focus <id>` |+| Share trajectory | `cua trajectory share` |++See [references/command-reference.md](references/command-reference.md) for full+argument syntax and all subcommands.++## Supported Providers++| Provider | Target | Example |+| ------------ | ----------------- | ----------------------------------- |+| `cloud` | CUA cloud VM | `cua do switch cloud my-vm` |+| `cloudv2` | CUA cloud VM (v2) | `cua do switch cloudv2 my-vm` |+| `docker` | Docker container | `cua do switch docker my-container` |+| `lume` | Lume VM | `cua do switch lume my-vm` |+| `lumier` | Lumier VM | `cua do switch lumier my-vm` |+| `winsandbox` | Windows Sandbox | `cua do switch winsandbox` |+| `host` | Local machine | `cua do switch host` |diff --git a/skills/gui-automation/references/command-reference.md b/skills/gui-automation/references/command-reference.mdnew file mode 100644index 0000000000..d580ec6352--- /dev/null+++ b/skills/gui-automation/references/command-reference.md@@ -0,0 +1,302 @@+# CUA Command Reference++Complete argument syntax for all `cua do` and `cua trajectory` commands.++## Global Flags++| Flag | Effect |+| ------------- | --------------------------------------------- |+| `--no-record` | Disable trajectory recording for this command |++Usage: `cua do --no-record <action> [args]`++## Target Management++```+cua do switch <provider> [name]+```++Providers: `cloud`, `cloudv2`, `docker`, `lume`, `lumier`, `winsandbox`, `host`++- `name` is required for all providers except `host` and `winsandbox`+- Target persists in `~/.cua/do_target.json` — set once, then operate++```+cua do status+```++Show the current target and zoom state.++```+cua do ls [provider]+```++List VMs for a provider. If `provider` is omitted, uses the current target's provider.++## Host Consent++```+cua do-host-consent+```++One-time consent to grant AI control of the local machine. Creates+`~/.cua/host_consented`. Required before `cua do switch host`.++## Screenshot and Snapshot++```+cua do screenshot [--save PATH]+```++Take a screenshot. Returns the image path. If `--save` / `-s` is provided,+saves to that path instead of the default temp directory.++```+cua do snapshot ["extra instructions"]+```++Screenshot + AI-powered screen summary. Returns a JSON object with:++- `summary`: human-readable description of the screen+- `elements`: list of interactive elements with `name`, `type`, `x`, `y`++Requires `ANTHROPIC_API_KEY`. Optional instructions refine the AI's focus+(e.g., `cua do snapshot "focus on the form fields"`).++## Window Zoom++```+cua do zoom "Window Name"+```++Crop all subsequent screenshots to the named window. Coordinates become+window-relative. Useful for focusing on a single app.++```+cua do unzoom+```++Return to full-screen screenshots.++## Input Actions++### Click++```+cua do click <x> <y> [left|right|middle]+```++Click at image-space coordinates. Default button: `left`.++### Double-Click++```+cua do dclick <x> <y>+```++Double-click at image-space coordinates.++### Move Cursor++```+cua do move <x> <y>+```++Move the cursor to image-space coordinates without clicking.++### Type Text++```+cua do type "text"+```++Type a string of text. Supports any Unicode characters.++### Press Key++```+cua do key <key>+```++Press a single key. Common keys: `enter`, `escape`, `tab`, `space`, `backspace`,+`delete`, `up`, `down`, `left`, `right`, `home`, `end`, `pageup`, `pagedown`,+`f1`–`f12`.++### Hotkey++```+cua do hotkey <combo>+```++Press a keyboard shortcut. Use `+` to combine modifiers.++Examples: `cmd+c`, `ctrl+shift+s`, `alt+f4`, `cmd+shift+p`++### Scroll++```+cua do scroll <direction> [amount]+```++Scroll in a direction. Default amount: `3`.++Directions: `up`, `down`, `left`, `right`++### Drag++```+cua do drag <x1> <y1> <x2> <y2>+```++Drag from `(x1, y1)` to `(x2, y2)` in image-space coordinates.++## Shell and Open++### Shell++```+cua do shell "command"+```++Run a shell command on the target machine. If no command is provided, opens an+interactive terminal session.++Optional flags:++- `--cols N` — terminal width (default: auto-detect)+- `--rows N` — terminal height (default: auto-detect)++### Open++```+cua do open <url|path>+```++Open a URL in the default browser or a file in its default application on the+target machine.++## Window Management++All window commands are under `cua do window`.++### List Windows++```+cua do window ls [app]+```++List all windows, optionally filtered by application name. Returns window IDs+needed for other window commands.++### Focus / Activate++```+cua do window focus <id>+cua do window activate <id>+```++Bring a window to the front.++### Unfocus++```+cua do window unfocus+```++Remove focus from the current window.++### Minimize / Maximize / Close++```+cua do window minimize <id>+cua do window maximize <id>+cua do window close <id>+```++### Resize++```+cua do window resize <id> <width> <height>+```++### Move++```+cua do window move <id> <x> <y>+```++### Info++```+cua do window info <id>+```++Get details (position, size, title) for a window.++## Trajectory Commands++All trajectory commands are under `cua trajectory` (alias: `cua traj`).++### List Sessions++```+cua trajectory ls [machine] [--json]+```++List all recorded sessions, optionally filtered by machine name.++### View (Local)++```+cua trajectory view [target] [--port PORT]+```++Zip the session and open it in the browser via `cua.ai/trajectory-viewer`.+Starts a local file server (default port: `8089`).++`target` can be a machine name, session timestamp, or directory path. Defaults+to the latest session.++### Share (Upload)++```+cua trajectory share [target] [--no-open] [--api-url URL]+```++Upload the trajectory and get a shareable HTTPS link. Opens the link in the+browser by default.++- `--no-open` — don't open browser after uploading+- `--api-url` — custom API URL (default: `CUA_API_URL` env or `https://cua.ai`)++### Export (HTML)++```+cua trajectory export [target] [--output PATH] [--quality N] [--no-open]+```++Generate a self-contained HTML report.++- `--output` / `-o` — output file path (default: `report.html` in session dir)+- `--quality` / `-q` — JPEG compression quality 1–100 (default: `75`)+- `--no-open` — don't open the report in browser++### Clean++```+cua trajectory clean [--older-than DAYS] [--machine NAME] [-y]+```++Delete old trajectory sessions.++- `--older-than DAYS` — only delete sessions older than N days+- `--machine NAME` — only delete sessions for a specific machine+- `-y` / `--yes` — skip confirmation prompt++### Stop++```+cua trajectory stop+```++Stop the local trajectory file server started by `cua trajectory view`.