Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

make skill more token efficient (#1127)

dduponte2786233d68e
diff --git a/skills/gui-automation/SKILL.md b/skills/gui-automation/SKILL.md
index e4908babec..b6953e1d29 100644
--- a/skills/gui-automation/SKILL.md
+++ b/skills/gui-automation/SKILL.md
@@ -9,255 +9,153 @@ description: >-
# GUI Automation
-CUA gives you **eyes and hands on a real computer**. You can see the screen,
-move the mouse, click buttons, type text, drag elements, and manage windows —
-exactly like a human sitting at the keyboard.
+CUA gives you **eyes and hands on a real computer**: see the screen, move the
+mouse, click, type, drag, and manage windows — like a human at the keyboard.
-Use this skill whenever the task requires **visual interaction** that cannot be
-done through a shell command or API call: testing a UI, filling a form, verifying
-a layout, fuzzing a web page, or driving an end-to-end user flow.
+Use this skill for **visual interaction** that can't be done via shell or API.
-Every action you take is automatically recorded to a replayable trajectory.
-When you finish, share the trajectory link so the user can review your work.
-
-## Prerequisites
-
-Before your first action, ensure the CLI is available:
+## Setup
```bash
-# Check if cua is installed
-cua --version
-
-# If missing, install it
-pip install cua
-```
-
-**Target setup** — connect to the machine you will automate:
+cua --version # check install; if missing: pip install cua
-```bash
-# Cloud VM
+# Connect to target (pick one)
cua do switch cloud my-vm
-
-# Docker container
cua do switch docker my-container
-
-# Local machine (requires one-time consent)
-cua do-host-consent
-cua do switch host
+cua do-host-consent && cua do switch host # local machine (one-time consent)
```
-> **ANTHROPIC_API_KEY** is optional. If set, `cua do snapshot` returns an
-> AI-annotated summary with element coordinates. Without it, use
-> `cua do screenshot` and read the image yourself to determine coordinates.
-> Both approaches work — snapshot is a convenience, not a requirement.
-
-## Common Scenarios
+> `ANTHROPIC_API_KEY` is optional. With it, `cua do snapshot` returns an
+> AI-annotated screen with element coordinates. Without it, use `screenshot`
+> and read the image yourself.
-### 1. Test a button
+## Workflow
-> "Click the Export PDF button and verify the download starts."
+**Look → Act → Verify** — repeat until done, then share:
```bash
-cua do switch cloud my-vm
-cua do screenshot # see the current screen
-# (read the image to locate the Export PDF button)
-cua do click 450 280 # click it
-cua do screenshot # confirm: did a download dialog appear?
+cua do screenshot # look
+cua do click 450 280 # act
+cua do screenshot # verify
+cua trajectory share # share replay link with user
```
-### 2. Fill a form and submit
-
-> "Fill out the signup form with test data and submit it."
-
-```bash
-cua do screenshot # see what fields are present
-cua do click 400 200 # click the Name field
-cua do type "Jane Doe"
-cua do key tab
-cua do type "[email protected]"
-cua do key tab
-cua do type "SecureP@ss123"
-cua do click 400 500 # click Submit
-cua do screenshot # verify success message
-```
+> Re-screenshot after every UI change — coordinates go stale when the screen changes.
-### 3. Handle a file upload dialog
+## Scenarios
-> "Upload report.pdf through the file picker."
+### Click a button
```bash
-cua do screenshot # locate the upload button
-cua do click 350 400 # click "Choose File"
-cua do screenshot # file picker is now open
-cua do type "/home/user/report.pdf" # type the file path
-cua do key enter # confirm selection
-cua do screenshot # verify the file name appears
+cua do screenshot
+cua do click 450 280
+cua do screenshot
```
-### 4. Visual verification
-
-> "Check that the dashboard chart renders correctly after filtering."
+### Fill a form
```bash
-cua do screenshot # baseline state
-cua do click 600 120 # click the date filter dropdown
-cua do screenshot # see dropdown options
-cua do click 600 180 # select "Last 7 days"
-cua do screenshot # verify: chart updated, no blank areas, labels correct
+cua do screenshot
+cua do click 400 200 && cua do type "Jane Doe"
+cua do key tab && cua do type "[email protected]"
+cua do key tab && cua do type "SecureP@ss123"
+cua do click 400 500
+cua do screenshot
```
-### 5. End-to-end user flow
-
-> "Walk through the checkout flow from cart to confirmation."
+### File upload dialog
```bash
-cua do screenshot # see the cart page
-cua do click 700 500 # click "Proceed to Checkout"
-cua do screenshot # shipping form
-cua do click 400 200
-cua do type "123 Main St"
-cua do key tab
-cua do type "Springfield"
-cua do key tab
-cua do type "62701"
-cua do click 700 500 # click "Continue to Payment"
-cua do screenshot # payment form
-cua do click 400 200
-cua do type "4111111111111111"
-cua do key tab
-cua do type "12/28"
-cua do key tab
-cua do type "123"
-cua do click 700 500 # click "Place Order"
-cua do screenshot # confirmation page — verify order number present
+cua do click 350 400 # "Choose File"
+cua do type "/home/user/report.pdf"
+cua do key enter
+cua do screenshot
```
-### 6. Cross-window drag and drop
+### Zoom in for precision clicks (host or small targets)
-> "Drag the file icon from the file manager into the upload zone in Chrome."
+When clicking small or dense UI elements — especially on the host machine —
+zoom into the target window first. Coordinates become **window-relative** and
+screenshots show only that window, giving you higher effective resolution.
```bash
-cua do window ls # list open windows with IDs
-cua do screenshot # see both windows
-cua do drag 150 300 650 400 # drag from file manager → upload zone
-cua do screenshot # verify the file landed in the drop zone
+cua do zoom "Google Chrome" # crop to Chrome window; coords are now window-relative
+cua do screenshot # zoomed view — easier to locate small elements
+cua do click 112 44 # precise click on a small tab or button
+cua do screenshot # verify
+cua do unzoom # restore full-screen coords when done
+cua do screenshot # back to full desktop view
```
-### 7. Fuzz a web page
+> Use `zoom` any time click accuracy is uncertain. `unzoom` before switching
+> windows or when you need to see the full desktop again.
-> "Try unexpected inputs on every text field in the form."
+### Drag and drop
```bash
-cua do screenshot # see the form
-cua do click 400 200 # first field
-cua do type "<script>alert(1)</script>"
-cua do key tab
-cua do type "'; DROP TABLE users; --"
-cua do key tab
-cua do type "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
-cua do click 400 500 # submit
-cua do screenshot # check for errors, crashes, or unexpected behavior
+cua do window ls # list open windows
+cua do drag 150 300 650 400 # source → destination
+cua do screenshot
```
-## Workflow Pattern
-
-Every GUI task follows the same loop:
-
-1. **Connect** — `cua do switch <provider> <name>`
-2. **Look** — `cua do screenshot` (or `snapshot` if API key is set)
-3. **Act** — `click`, `type`, `key`, `drag`, `scroll`, etc.
-4. **Verify** — `cua do screenshot` again to confirm the result
-5. **Repeat** — continue until the task is done
-6. **Share** — `cua do trajectory share` to give the user a replay link
-
-> Always re-screenshot after any action that changes the UI. Coordinates go
-> stale the moment the screen changes.
-
-## Trajectory Recording and Sharing
-
-Every `cua do` action is automatically recorded to a replayable trajectory at
-`~/.cua/trajectories/{machine}/{session}/`. A new session starts each time you
-run `cua do switch`.
-
-### Sharing your work
-
-When you finish a task, **always share the trajectory** so the user can replay
-and review exactly what happened:
+### Fuzz a form
```bash
-cua trajectory share # upload and get a shareable HTTPS link
-```
-
-Tell the user:
-
-> "Here is the trajectory of my session: {url}"
-
-### Other trajectory commands
-
-```bash
-cua trajectory ls # list all recorded sessions
-cua trajectory view # open latest session in browser (local)
-cua trajectory view my-container # view latest for a specific machine
-cua trajectory export # generate a self-contained HTML report
-cua trajectory export -o report.html # export to a specific path
-cua trajectory clean --older-than 7 # delete sessions older than 7 days
-cua trajectory stop # stop the local viewer server
+cua do screenshot
+cua do click 400 200
+cua do type "<script>alert(1)</script>"
+cua do key tab && cua do type "'; DROP TABLE users; --"
+cua do key tab && cua do type "AAAAAAAAAAAAAAAAAAAAAAA"
+cua do click 400 500
+cua do screenshot # check for errors, crashes, unexpected behavior
```
-### Disabling recording
+## Trajectory
-If the user asks you not to record, pass `--no-record`:
+Every action is auto-recorded to `~/.cua/trajectories/{machine}/{session}/`.
```bash
-cua do --no-record click 100 200
+cua trajectory share # upload and get shareable HTTPS link (always do this at end)
+cua trajectory ls # list sessions
+cua trajectory export # generate HTML report
+cua do --no-record click 100 200 # disable recording for a single action
```
-## Guardrails
-
-- **Re-screenshot after every UI change.** Coordinates are image-space pixels
- from the last screenshot. If the screen changed, your coordinates are wrong.
-- **snapshot vs screenshot.** `snapshot` calls an AI model and requires
- `ANTHROPIC_API_KEY`. If the key is missing, fall back to `screenshot` and
- read the image yourself. Both work.
-- **Coordinates are image-space.** Zoom and max-length scaling are applied
- automatically — use the pixel coordinates from your most recent screenshot.
-- **Host consent.** Controlling the local machine requires a one-time
- `cua do-host-consent` before `switch host`.
-- **Window zoom.** Use `cua do zoom "App Name"` to crop screenshots to a
- single window. Coordinates become window-relative. `cua do unzoom` to reset.
+Tell the user: `"Here is the trajectory of my session: {url}"`
## Quick Reference
-| Action | Command |
-| ------------------- | -------------------------------------------- |
-| Connect to target | `cua do switch <provider> [name]` |
-| See the screen | `cua do screenshot` |
-| AI-annotated screen | `cua do snapshot ["instructions"]` |
-| Click | `cua do click <x> <y> [left\|right\|middle]` |
-| Double-click | `cua do dclick <x> <y>` |
-| Type text | `cua do type "text"` |
-| Press key | `cua do key <key>` |
-| Hotkey | `cua do hotkey <combo>` (e.g. `cmd+c`) |
-| Scroll | `cua do scroll <direction> [amount]` |
-| Drag | `cua do drag <x1> <y1> <x2> <y2>` |
-| Move cursor | `cua do move <x> <y>` |
-| Run shell command | `cua do shell "command"` |
-| Open URL or file | `cua do open <url\|path>` |
-| List windows | `cua do window ls [app]` |
-| Focus window | `cua do window focus <id>` |
-| Share trajectory | `cua trajectory share` |
-
-See [references/command-reference.md](references/command-reference.md) for full
-argument syntax and all subcommands.
-
-## Supported Providers
-
-| Provider | Target | Example |
-| ------------ | ----------------- | ----------------------------------- |
-| `cloud` | CUA cloud VM | `cua do switch cloud my-vm` |
-| `cloudv2` | CUA cloud VM (v2) | `cua do switch cloudv2 my-vm` |
-| `docker` | Docker container | `cua do switch docker my-container` |
-| `lume` | Lume VM | `cua do switch lume my-vm` |
-| `lumier` | Lumier VM | `cua do switch lumier my-vm` |
-| `winsandbox` | Windows Sandbox | `cua do switch winsandbox` |
-| `host` | Local machine | `cua do switch host` |
+| Action | Command |
+| ------------------- | ---------------------------------------------- |
+| Connect to target | `cua do switch <provider> [name]` |
+| Screenshot | `cua do screenshot` |
+| AI-annotated screen | `cua do snapshot ["instructions"]` |
+| Click | `cua do click <x> <y> [left\|right\|middle]` |
+| Double-click | `cua do dclick <x> <y>` |
+| Type text | `cua do type "text"` |
+| Press key | `cua do key <key>` |
+| Hotkey | `cua do hotkey <combo>` (e.g. `ctrl+c`) |
+| Scroll | `cua do scroll <direction> [amount]` |
+| Drag | `cua do drag <x1> <y1> <x2> <y2>` |
+| Move cursor | `cua do move <x> <y>` |
+| Shell command | `cua do shell "command"` |
+| Open URL/file | `cua do open <url\|path>` |
+| List windows | `cua do window ls [app]` |
+| Focus window | `cua do window focus <id>` |
+| Zoom to window | `cua do zoom "App Name"` |
+| Unzoom | `cua do unzoom` |
+| Share trajectory | `cua trajectory share` |
+
+## Providers
+
+| Provider | Example |
+| ------------ | ------------------------------------- |
+| `cloud` | `cua do switch cloud my-vm` |
+| `cloudv2` | `cua do switch cloudv2 my-vm` |
+| `docker` | `cua do switch docker my-container` |
+| `lume` | `cua do switch lume my-vm` |
+| `lumier` | `cua do switch lumier my-vm` |
+| `winsandbox` | `cua do switch winsandbox` |
+| `host` | `cua do switch host` |
+
+See [references/command-reference.md](references/command-reference.md) for full argument syntax.