Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(skills): rewrite cua-do-cli as task-oriented gui-automation skill (#1121)

Sarina Lie2cddf51c2a5
diff --git a/skills/cua-do-cli/SKILL.md b/skills/cua-do-cli/SKILL.md
deleted file mode 100644
index dde6189d10..0000000000
--- a/skills/cua-do-cli/SKILL.md
+++ /dev/null
@@ -1,146 +0,0 @@
----
-name: cua-do-cli
-description: Automates interactions with remote VMs, local Docker containers, cloud sandboxes, and the host PC via cua do. Use when navigating GUIs, taking snapshots/screenshots, clicking, typing, scrolling, running shell commands, or managing windows on a target machine.
----
-
-# GUI Automation with cua do
-
-One command = one action = one ✅/❌ line. Switch targets once, then operate.
-
-## Start Here
-
-1. Pick a flow:
- - API key available: switch → snapshot → interact → snapshot
- - No API key: switch → screenshot → read image → interact → screenshot
- - Host control: consent → switch host → operate
-2. Run the canonical flow below.
-3. Check the command skeleton only if blocked.
-
-## Decision Map
-
-- No target set: `switch` → then operate
-- **ANTHROPIC_API_KEY set** (normal): `switch` → `snapshot` → `click/type/key` → repeat snapshot after each UI change
-- **No API key**: `switch` → `screenshot` → read image to determine coords → `click/type/key` → repeat screenshot
-- Zoom to one window: `zoom "App Name"` → operate → `unzoom`
-- Run shell: `shell "command"`
-- Control host PC: `do-host-consent` (once) → `switch host` → operate
-
-## Canonical Flows
-
-### 1) Normal (ANTHROPIC_API_KEY set)
-
-```bash
-cua do switch docker my-container
-cua do snapshot # AI summary + interactive elements with coords
-cua do click 450 300
-cua do type "hello"
-cua do key enter
-cua do snapshot # re-snapshot after UI change
-```
-
-### 2) No API Key
-
-```bash
-cua do switch docker my-container
-cua do screenshot # save path printed; read the image to find coords
-cua do click 450 300
-cua do type "hello"
-cua do key enter
-cua do screenshot # re-screenshot after UI change
-```
-
-### 3) Focused Window (Zoom)
-
-```bash
-cua do zoom "Chrome"
-cua do snapshot # cropped to Chrome; coords are window-relative
-cua do click 120 80
-cua do unzoom
-```
-
-### 4) Host PC Control
-
-```bash
-cua do-host-consent # one-time consent, no prompt
-cua do switch host
-cua do snapshot
-cua do click 100 200
-```
-
-## Command Skeleton
-
-### Target
-
-```bash
-cua do switch <provider> [name] # cloud, docker, lume, lumier, winsandbox, host
-cua do status
-cua do ls [provider]
-```
-
-### Snapshot / Screenshot
-
-```bash
-cua do snapshot ["extra instructions"] # screenshot + AI summary (needs ANTHROPIC_API_KEY)
-cua do screenshot [--save path]
-cua do zoom "Window Name"
-cua do unzoom
-```
-
-### Input
-
-```bash
-cua do click <x> <y> [left|right|middle]
-cua do dclick <x> <y>
-cua do move <x> <y>
-cua do type "text"
-cua do key <key> # enter, escape, tab, space, f1–f12, …
-cua do hotkey ctrl+c
-cua do scroll <up|down|left|right> [n]
-cua do drag <x1> <y1> <x2> <y2>
-```
-
-### Shell / Open
-
-```bash
-cua do shell "command"
-cua do open <url|path>
-```
-
-### Window
-
-```bash
-cua do window ls [app]
-cua do window focus <id>
-cua do window unfocus
-cua do window minimize/maximize/close <id>
-cua do window resize <id> <w> <h>
-cua do window move <id> <x> <y>
-cua do window info <id>
-```
-
-## Guardrails
-
-- Re-snapshot after navigation, modals, or list changes — coords go stale.
-- `snapshot` needs `ANTHROPIC_API_KEY`; use `screenshot` otherwise.
-- Coords are image-space: zoom + max-length scaling are applied automatically.
-- `do-host-consent` is permanent until the consent file is deleted (`~/.cua/host_consented`).
-- Set `PYTHONIOENCODING=utf-8` on Windows for correct emoji output.
-
-## Trajectory Recording
-
-Every `cua do` action is automatically recorded to a replayable trajectory.
-
-- Stored at `~/.cua/trajectories/{machine_name}/{session_timestamp}/`
-- New session starts on `cua do switch`
-- Disable with `--no-record`: `cua do --no-record click 100 200`
-- After a session, mention the trajectory to the user:
- "Your session was recorded. Run `cua trajectory view` to replay."
-
-```bash
-cua trajectory ls # list all sessions
-cua trajectory view # start local viewer for latest session
-cua trajectory view my-container # latest for specific machine
-cua trajectory view --port 9090 # use custom port
-cua trajectory stop # stop the viewer server
-cua trajectory clean --older-than 7 # delete old sessions
-```
diff --git a/skills/gui-automation/SKILL.md b/skills/gui-automation/SKILL.md
new file mode 100644
index 0000000000..e4908babec
--- /dev/null
+++ b/skills/gui-automation/SKILL.md
@@ -0,0 +1,263 @@
+---
+name: gui-automation
+description: >-
+ Use when you need to visually interact with a GUI — test buttons, fill forms,
+ verify visual layouts, fuzz web pages, automate user flows, take screenshots,
+ or perform end-to-end QA on any application. Works on cloud VMs, Docker
+ containers, local machines, and sandboxes. Install: pip install cua.
+---
+
+# GUI Automation
+
+CUA gives you **eyes and hands on a real computer**. You can see the screen,
+move the mouse, click buttons, type text, drag elements, and manage windows —
+exactly like a human sitting at the keyboard.
+
+Use this skill whenever the task requires **visual interaction** that cannot be
+done through a shell command or API call: testing a UI, filling a form, verifying
+a layout, fuzzing a web page, or driving an end-to-end user flow.
+
+Every action you take is automatically recorded to a replayable trajectory.
+When you finish, share the trajectory link so the user can review your work.
+
+## Prerequisites
+
+Before your first action, ensure the CLI is available:
+
+```bash
+# Check if cua is installed
+cua --version
+
+# If missing, install it
+pip install cua
+```
+
+**Target setup** — connect to the machine you will automate:
+
+```bash
+# Cloud VM
+cua do switch cloud my-vm
+
+# Docker container
+cua do switch docker my-container
+
+# Local machine (requires one-time consent)
+cua do-host-consent
+cua do switch host
+```
+
+> **ANTHROPIC_API_KEY** is optional. If set, `cua do snapshot` returns an
+> AI-annotated summary with element coordinates. Without it, use
+> `cua do screenshot` and read the image yourself to determine coordinates.
+> Both approaches work — snapshot is a convenience, not a requirement.
+
+## Common Scenarios
+
+### 1. Test a button
+
+> "Click the Export PDF button and verify the download starts."
+
+```bash
+cua do switch cloud my-vm
+cua do screenshot # see the current screen
+# (read the image to locate the Export PDF button)
+cua do click 450 280 # click it
+cua do screenshot # confirm: did a download dialog appear?
+```
+
+### 2. Fill a form and submit
+
+> "Fill out the signup form with test data and submit it."
+
+```bash
+cua do screenshot # see what fields are present
+cua do click 400 200 # click the Name field
+cua do type "Jane Doe"
+cua do key tab
+cua do type "[email protected]"
+cua do key tab
+cua do type "SecureP@ss123"
+cua do click 400 500 # click Submit
+cua do screenshot # verify success message
+```
+
+### 3. Handle a file upload dialog
+
+> "Upload report.pdf through the file picker."
+
+```bash
+cua do screenshot # locate the upload button
+cua do click 350 400 # click "Choose File"
+cua do screenshot # file picker is now open
+cua do type "/home/user/report.pdf" # type the file path
+cua do key enter # confirm selection
+cua do screenshot # verify the file name appears
+```
+
+### 4. Visual verification
+
+> "Check that the dashboard chart renders correctly after filtering."
+
+```bash
+cua do screenshot # baseline state
+cua do click 600 120 # click the date filter dropdown
+cua do screenshot # see dropdown options
+cua do click 600 180 # select "Last 7 days"
+cua do screenshot # verify: chart updated, no blank areas, labels correct
+```
+
+### 5. End-to-end user flow
+
+> "Walk through the checkout flow from cart to confirmation."
+
+```bash
+cua do screenshot # see the cart page
+cua do click 700 500 # click "Proceed to Checkout"
+cua do screenshot # shipping form
+cua do click 400 200
+cua do type "123 Main St"
+cua do key tab
+cua do type "Springfield"
+cua do key tab
+cua do type "62701"
+cua do click 700 500 # click "Continue to Payment"
+cua do screenshot # payment form
+cua do click 400 200
+cua do type "4111111111111111"
+cua do key tab
+cua do type "12/28"
+cua do key tab
+cua do type "123"
+cua do click 700 500 # click "Place Order"
+cua do screenshot # confirmation page — verify order number present
+```
+
+### 6. Cross-window drag and drop
+
+> "Drag the file icon from the file manager into the upload zone in Chrome."
+
+```bash
+cua do window ls # list open windows with IDs
+cua do screenshot # see both windows
+cua do drag 150 300 650 400 # drag from file manager → upload zone
+cua do screenshot # verify the file landed in the drop zone
+```
+
+### 7. Fuzz a web page
+
+> "Try unexpected inputs on every text field in the form."
+
+```bash
+cua do screenshot # see the form
+cua do click 400 200 # first field
+cua do type "<script>alert(1)</script>"
+cua do key tab
+cua do type "'; DROP TABLE users; --"
+cua do key tab
+cua do type "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
+cua do click 400 500 # submit
+cua do screenshot # check for errors, crashes, or unexpected behavior
+```
+
+## Workflow Pattern
+
+Every GUI task follows the same loop:
+
+1. **Connect** — `cua do switch <provider> <name>`
+2. **Look** — `cua do screenshot` (or `snapshot` if API key is set)
+3. **Act** — `click`, `type`, `key`, `drag`, `scroll`, etc.
+4. **Verify** — `cua do screenshot` again to confirm the result
+5. **Repeat** — continue until the task is done
+6. **Share** — `cua do trajectory share` to give the user a replay link
+
+> Always re-screenshot after any action that changes the UI. Coordinates go
+> stale the moment the screen changes.
+
+## Trajectory Recording and Sharing
+
+Every `cua do` action is automatically recorded to a replayable trajectory at
+`~/.cua/trajectories/{machine}/{session}/`. A new session starts each time you
+run `cua do switch`.
+
+### Sharing your work
+
+When you finish a task, **always share the trajectory** so the user can replay
+and review exactly what happened:
+
+```bash
+cua trajectory share # upload and get a shareable HTTPS link
+```
+
+Tell the user:
+
+> "Here is the trajectory of my session: {url}"
+
+### Other trajectory commands
+
+```bash
+cua trajectory ls # list all recorded sessions
+cua trajectory view # open latest session in browser (local)
+cua trajectory view my-container # view latest for a specific machine
+cua trajectory export # generate a self-contained HTML report
+cua trajectory export -o report.html # export to a specific path
+cua trajectory clean --older-than 7 # delete sessions older than 7 days
+cua trajectory stop # stop the local viewer server
+```
+
+### Disabling recording
+
+If the user asks you not to record, pass `--no-record`:
+
+```bash
+cua do --no-record click 100 200
+```
+
+## Guardrails
+
+- **Re-screenshot after every UI change.** Coordinates are image-space pixels
+ from the last screenshot. If the screen changed, your coordinates are wrong.
+- **snapshot vs screenshot.** `snapshot` calls an AI model and requires
+ `ANTHROPIC_API_KEY`. If the key is missing, fall back to `screenshot` and
+ read the image yourself. Both work.
+- **Coordinates are image-space.** Zoom and max-length scaling are applied
+ automatically — use the pixel coordinates from your most recent screenshot.
+- **Host consent.** Controlling the local machine requires a one-time
+ `cua do-host-consent` before `switch host`.
+- **Window zoom.** Use `cua do zoom "App Name"` to crop screenshots to a
+ single window. Coordinates become window-relative. `cua do unzoom` to reset.
+
+## Quick Reference
+
+| Action | Command |
+| ------------------- | -------------------------------------------- |
+| Connect to target | `cua do switch <provider> [name]` |
+| See the screen | `cua do screenshot` |
+| AI-annotated screen | `cua do snapshot ["instructions"]` |
+| Click | `cua do click <x> <y> [left\|right\|middle]` |
+| Double-click | `cua do dclick <x> <y>` |
+| Type text | `cua do type "text"` |
+| Press key | `cua do key <key>` |
+| Hotkey | `cua do hotkey <combo>` (e.g. `cmd+c`) |
+| Scroll | `cua do scroll <direction> [amount]` |
+| Drag | `cua do drag <x1> <y1> <x2> <y2>` |
+| Move cursor | `cua do move <x> <y>` |
+| Run shell command | `cua do shell "command"` |
+| Open URL or file | `cua do open <url\|path>` |
+| List windows | `cua do window ls [app]` |
+| Focus window | `cua do window focus <id>` |
+| Share trajectory | `cua trajectory share` |
+
+See [references/command-reference.md](references/command-reference.md) for full
+argument syntax and all subcommands.
+
+## Supported Providers
+
+| Provider | Target | Example |
+| ------------ | ----------------- | ----------------------------------- |
+| `cloud` | CUA cloud VM | `cua do switch cloud my-vm` |
+| `cloudv2` | CUA cloud VM (v2) | `cua do switch cloudv2 my-vm` |
+| `docker` | Docker container | `cua do switch docker my-container` |
+| `lume` | Lume VM | `cua do switch lume my-vm` |
+| `lumier` | Lumier VM | `cua do switch lumier my-vm` |
+| `winsandbox` | Windows Sandbox | `cua do switch winsandbox` |
+| `host` | Local machine | `cua do switch host` |
diff --git a/skills/gui-automation/references/command-reference.md b/skills/gui-automation/references/command-reference.md
new file mode 100644
index 0000000000..d580ec6352
--- /dev/null
+++ b/skills/gui-automation/references/command-reference.md
@@ -0,0 +1,302 @@
+# CUA Command Reference
+
+Complete argument syntax for all `cua do` and `cua trajectory` commands.
+
+## Global Flags
+
+| Flag | Effect |
+| ------------- | --------------------------------------------- |
+| `--no-record` | Disable trajectory recording for this command |
+
+Usage: `cua do --no-record <action> [args]`
+
+## Target Management
+
+```
+cua do switch <provider> [name]
+```
+
+Providers: `cloud`, `cloudv2`, `docker`, `lume`, `lumier`, `winsandbox`, `host`
+
+- `name` is required for all providers except `host` and `winsandbox`
+- Target persists in `~/.cua/do_target.json` — set once, then operate
+
+```
+cua do status
+```
+
+Show the current target and zoom state.
+
+```
+cua do ls [provider]
+```
+
+List VMs for a provider. If `provider` is omitted, uses the current target's provider.
+
+## Host Consent
+
+```
+cua do-host-consent
+```
+
+One-time consent to grant AI control of the local machine. Creates
+`~/.cua/host_consented`. Required before `cua do switch host`.
+
+## Screenshot and Snapshot
+
+```
+cua do screenshot [--save PATH]
+```
+
+Take a screenshot. Returns the image path. If `--save` / `-s` is provided,
+saves to that path instead of the default temp directory.
+
+```
+cua do snapshot ["extra instructions"]
+```
+
+Screenshot + AI-powered screen summary. Returns a JSON object with:
+
+- `summary`: human-readable description of the screen
+- `elements`: list of interactive elements with `name`, `type`, `x`, `y`
+
+Requires `ANTHROPIC_API_KEY`. Optional instructions refine the AI's focus
+(e.g., `cua do snapshot "focus on the form fields"`).
+
+## Window Zoom
+
+```
+cua do zoom "Window Name"
+```
+
+Crop all subsequent screenshots to the named window. Coordinates become
+window-relative. Useful for focusing on a single app.
+
+```
+cua do unzoom
+```
+
+Return to full-screen screenshots.
+
+## Input Actions
+
+### Click
+
+```
+cua do click <x> <y> [left|right|middle]
+```
+
+Click at image-space coordinates. Default button: `left`.
+
+### Double-Click
+
+```
+cua do dclick <x> <y>
+```
+
+Double-click at image-space coordinates.
+
+### Move Cursor
+
+```
+cua do move <x> <y>
+```
+
+Move the cursor to image-space coordinates without clicking.
+
+### Type Text
+
+```
+cua do type "text"
+```
+
+Type a string of text. Supports any Unicode characters.
+
+### Press Key
+
+```
+cua do key <key>
+```
+
+Press a single key. Common keys: `enter`, `escape`, `tab`, `space`, `backspace`,
+`delete`, `up`, `down`, `left`, `right`, `home`, `end`, `pageup`, `pagedown`,
+`f1`–`f12`.
+
+### Hotkey
+
+```
+cua do hotkey <combo>
+```
+
+Press a keyboard shortcut. Use `+` to combine modifiers.
+
+Examples: `cmd+c`, `ctrl+shift+s`, `alt+f4`, `cmd+shift+p`
+
+### Scroll
+
+```
+cua do scroll <direction> [amount]
+```
+
+Scroll in a direction. Default amount: `3`.
+
+Directions: `up`, `down`, `left`, `right`
+
+### Drag
+
+```
+cua do drag <x1> <y1> <x2> <y2>
+```
+
+Drag from `(x1, y1)` to `(x2, y2)` in image-space coordinates.
+
+## Shell and Open
+
+### Shell
+
+```
+cua do shell "command"
+```
+
+Run a shell command on the target machine. If no command is provided, opens an
+interactive terminal session.
+
+Optional flags:
+
+- `--cols N` — terminal width (default: auto-detect)
+- `--rows N` — terminal height (default: auto-detect)
+
+### Open
+
+```
+cua do open <url|path>
+```
+
+Open a URL in the default browser or a file in its default application on the
+target machine.
+
+## Window Management
+
+All window commands are under `cua do window`.
+
+### List Windows
+
+```
+cua do window ls [app]
+```
+
+List all windows, optionally filtered by application name. Returns window IDs
+needed for other window commands.
+
+### Focus / Activate
+
+```
+cua do window focus <id>
+cua do window activate <id>
+```
+
+Bring a window to the front.
+
+### Unfocus
+
+```
+cua do window unfocus
+```
+
+Remove focus from the current window.
+
+### Minimize / Maximize / Close
+
+```
+cua do window minimize <id>
+cua do window maximize <id>
+cua do window close <id>
+```
+
+### Resize
+
+```
+cua do window resize <id> <width> <height>
+```
+
+### Move
+
+```
+cua do window move <id> <x> <y>
+```
+
+### Info
+
+```
+cua do window info <id>
+```
+
+Get details (position, size, title) for a window.
+
+## Trajectory Commands
+
+All trajectory commands are under `cua trajectory` (alias: `cua traj`).
+
+### List Sessions
+
+```
+cua trajectory ls [machine] [--json]
+```
+
+List all recorded sessions, optionally filtered by machine name.
+
+### View (Local)
+
+```
+cua trajectory view [target] [--port PORT]
+```
+
+Zip the session and open it in the browser via `cua.ai/trajectory-viewer`.
+Starts a local file server (default port: `8089`).
+
+`target` can be a machine name, session timestamp, or directory path. Defaults
+to the latest session.
+
+### Share (Upload)
+
+```
+cua trajectory share [target] [--no-open] [--api-url URL]
+```
+
+Upload the trajectory and get a shareable HTTPS link. Opens the link in the
+browser by default.
+
+- `--no-open` — don't open browser after uploading
+- `--api-url` — custom API URL (default: `CUA_API_URL` env or `https://cua.ai`)
+
+### Export (HTML)
+
+```
+cua trajectory export [target] [--output PATH] [--quality N] [--no-open]
+```
+
+Generate a self-contained HTML report.
+
+- `--output` / `-o` — output file path (default: `report.html` in session dir)
+- `--quality` / `-q` — JPEG compression quality 1–100 (default: `75`)
+- `--no-open` — don't open the report in browser
+
+### Clean
+
+```
+cua trajectory clean [--older-than DAYS] [--machine NAME] [-y]
+```
+
+Delete old trajectory sessions.
+
+- `--older-than DAYS` — only delete sessions older than N days
+- `--machine NAME` — only delete sessions for a specific machine
+- `-y` / `--yes` — skip confirmation prompt
+
+### Stop
+
+```
+cua trajectory stop
+```
+
+Stop the local trajectory file server started by `cua trajectory view`.