Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
make skill more token efficient (#1127)
dduponte2786233d68e
diff --git a/skills/gui-automation/SKILL.md b/skills/gui-automation/SKILL.mdindex e4908babec..b6953e1d29 100644--- a/skills/gui-automation/SKILL.md+++ b/skills/gui-automation/SKILL.md@@ -9,255 +9,153 @@ description: >-# GUI Automation-CUA gives you **eyes and hands on a real computer**. You can see the screen,-move the mouse, click buttons, type text, drag elements, and manage windows —-exactly like a human sitting at the keyboard.+CUA gives you **eyes and hands on a real computer**: see the screen, move the+mouse, click, type, drag, and manage windows — like a human at the keyboard.-Use this skill whenever the task requires **visual interaction** that cannot be-done through a shell command or API call: testing a UI, filling a form, verifying-a layout, fuzzing a web page, or driving an end-to-end user flow.+Use this skill for **visual interaction** that can't be done via shell or API.-Every action you take is automatically recorded to a replayable trajectory.-When you finish, share the trajectory link so the user can review your work.--## Prerequisites--Before your first action, ensure the CLI is available:+## Setup```bash-# Check if cua is installed-cua --version--# If missing, install it-pip install cua-```--**Target setup** — connect to the machine you will automate:+cua --version # check install; if missing: pip install cua-```bash-# Cloud VM+# Connect to target (pick one)cua do switch cloud my-vm--# Docker containercua do switch docker my-container--# Local machine (requires one-time consent)-cua do-host-consent-cua do switch host+cua do-host-consent && cua do switch host # local machine (one-time consent)```-> **ANTHROPIC_API_KEY** is optional. If set, `cua do snapshot` returns an-> AI-annotated summary with element coordinates. Without it, use-> `cua do screenshot` and read the image yourself to determine coordinates.-> Both approaches work — snapshot is a convenience, not a requirement.--## Common Scenarios+> `ANTHROPIC_API_KEY` is optional. With it, `cua do snapshot` returns an+> AI-annotated screen with element coordinates. Without it, use `screenshot`+> and read the image yourself.-### 1. Test a button+## Workflow-> "Click the Export PDF button and verify the download starts."+**Look → Act → Verify** — repeat until done, then share:```bash-cua do switch cloud my-vm-cua do screenshot # see the current screen-# (read the image to locate the Export PDF button)-cua do click 450 280 # click it-cua do screenshot # confirm: did a download dialog appear?+cua do screenshot # look+cua do click 450 280 # act+cua do screenshot # verify+cua trajectory share # share replay link with user```-### 2. Fill a form and submit--> "Fill out the signup form with test data and submit it."--```bash-cua do screenshot # see what fields are present-cua do click 400 200 # click the Name field-cua do type "Jane Doe"-cua do key tab-cua do type "[email protected]"-cua do key tab-cua do type "SecureP@ss123"-cua do click 400 500 # click Submit-cua do screenshot # verify success message-```+> Re-screenshot after every UI change — coordinates go stale when the screen changes.-### 3. Handle a file upload dialog+## Scenarios-> "Upload report.pdf through the file picker."+### Click a button```bash-cua do screenshot # locate the upload button-cua do click 350 400 # click "Choose File"-cua do screenshot # file picker is now open-cua do type "/home/user/report.pdf" # type the file path-cua do key enter # confirm selection-cua do screenshot # verify the file name appears+cua do screenshot+cua do click 450 280+cua do screenshot```-### 4. Visual verification--> "Check that the dashboard chart renders correctly after filtering."+### Fill a form```bash-cua do screenshot # baseline state-cua do click 600 120 # click the date filter dropdown-cua do screenshot # see dropdown options-cua do click 600 180 # select "Last 7 days"-cua do screenshot # verify: chart updated, no blank areas, labels correct+cua do screenshot+cua do click 400 200 && cua do type "Jane Doe"+cua do key tab && cua do type "[email protected]"+cua do key tab && cua do type "SecureP@ss123"+cua do click 400 500+cua do screenshot```-### 5. End-to-end user flow--> "Walk through the checkout flow from cart to confirmation."+### File upload dialog```bash-cua do screenshot # see the cart page-cua do click 700 500 # click "Proceed to Checkout"-cua do screenshot # shipping form-cua do click 400 200-cua do type "123 Main St"-cua do key tab-cua do type "Springfield"-cua do key tab-cua do type "62701"-cua do click 700 500 # click "Continue to Payment"-cua do screenshot # payment form-cua do click 400 200-cua do type "4111111111111111"-cua do key tab-cua do type "12/28"-cua do key tab-cua do type "123"-cua do click 700 500 # click "Place Order"-cua do screenshot # confirmation page — verify order number present+cua do click 350 400 # "Choose File"+cua do type "/home/user/report.pdf"+cua do key enter+cua do screenshot```-### 6. Cross-window drag and drop+### Zoom in for precision clicks (host or small targets)-> "Drag the file icon from the file manager into the upload zone in Chrome."+When clicking small or dense UI elements — especially on the host machine —+zoom into the target window first. Coordinates become **window-relative** and+screenshots show only that window, giving you higher effective resolution.```bash-cua do window ls # list open windows with IDs-cua do screenshot # see both windows-cua do drag 150 300 650 400 # drag from file manager → upload zone-cua do screenshot # verify the file landed in the drop zone+cua do zoom "Google Chrome" # crop to Chrome window; coords are now window-relative+cua do screenshot # zoomed view — easier to locate small elements+cua do click 112 44 # precise click on a small tab or button+cua do screenshot # verify+cua do unzoom # restore full-screen coords when done+cua do screenshot # back to full desktop view```-### 7. Fuzz a web page+> Use `zoom` any time click accuracy is uncertain. `unzoom` before switching+> windows or when you need to see the full desktop again.-> "Try unexpected inputs on every text field in the form."+### Drag and drop```bash-cua do screenshot # see the form-cua do click 400 200 # first field-cua do type "<script>alert(1)</script>"-cua do key tab-cua do type "'; DROP TABLE users; --"-cua do key tab-cua do type "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"-cua do click 400 500 # submit-cua do screenshot # check for errors, crashes, or unexpected behavior+cua do window ls # list open windows+cua do drag 150 300 650 400 # source → destination+cua do screenshot```-## Workflow Pattern--Every GUI task follows the same loop:--1. **Connect** — `cua do switch <provider> <name>`-2. **Look** — `cua do screenshot` (or `snapshot` if API key is set)-3. **Act** — `click`, `type`, `key`, `drag`, `scroll`, etc.-4. **Verify** — `cua do screenshot` again to confirm the result-5. **Repeat** — continue until the task is done-6. **Share** — `cua do trajectory share` to give the user a replay link--> Always re-screenshot after any action that changes the UI. Coordinates go-> stale the moment the screen changes.--## Trajectory Recording and Sharing--Every `cua do` action is automatically recorded to a replayable trajectory at-`~/.cua/trajectories/{machine}/{session}/`. A new session starts each time you-run `cua do switch`.--### Sharing your work--When you finish a task, **always share the trajectory** so the user can replay-and review exactly what happened:+### Fuzz a form```bash-cua trajectory share # upload and get a shareable HTTPS link-```--Tell the user:--> "Here is the trajectory of my session: {url}"--### Other trajectory commands--```bash-cua trajectory ls # list all recorded sessions-cua trajectory view # open latest session in browser (local)-cua trajectory view my-container # view latest for a specific machine-cua trajectory export # generate a self-contained HTML report-cua trajectory export -o report.html # export to a specific path-cua trajectory clean --older-than 7 # delete sessions older than 7 days-cua trajectory stop # stop the local viewer server+cua do screenshot+cua do click 400 200+cua do type "<script>alert(1)</script>"+cua do key tab && cua do type "'; DROP TABLE users; --"+cua do key tab && cua do type "AAAAAAAAAAAAAAAAAAAAAAA"+cua do click 400 500+cua do screenshot # check for errors, crashes, unexpected behavior```-### Disabling recording+## Trajectory-If the user asks you not to record, pass `--no-record`:+Every action is auto-recorded to `~/.cua/trajectories/{machine}/{session}/`.```bash-cua do --no-record click 100 200+cua trajectory share # upload and get shareable HTTPS link (always do this at end)+cua trajectory ls # list sessions+cua trajectory export # generate HTML report+cua do --no-record click 100 200 # disable recording for a single action```-## Guardrails--- **Re-screenshot after every UI change.** Coordinates are image-space pixels- from the last screenshot. If the screen changed, your coordinates are wrong.-- **snapshot vs screenshot.** `snapshot` calls an AI model and requires- `ANTHROPIC_API_KEY`. If the key is missing, fall back to `screenshot` and- read the image yourself. Both work.-- **Coordinates are image-space.** Zoom and max-length scaling are applied- automatically — use the pixel coordinates from your most recent screenshot.-- **Host consent.** Controlling the local machine requires a one-time- `cua do-host-consent` before `switch host`.-- **Window zoom.** Use `cua do zoom "App Name"` to crop screenshots to a- single window. Coordinates become window-relative. `cua do unzoom` to reset.+Tell the user: `"Here is the trajectory of my session: {url}"`## Quick Reference-| Action | Command |-| ------------------- | -------------------------------------------- |-| Connect to target | `cua do switch <provider> [name]` |-| See the screen | `cua do screenshot` |-| AI-annotated screen | `cua do snapshot ["instructions"]` |-| Click | `cua do click <x> <y> [left\|right\|middle]` |-| Double-click | `cua do dclick <x> <y>` |-| Type text | `cua do type "text"` |-| Press key | `cua do key <key>` |-| Hotkey | `cua do hotkey <combo>` (e.g. `cmd+c`) |-| Scroll | `cua do scroll <direction> [amount]` |-| Drag | `cua do drag <x1> <y1> <x2> <y2>` |-| Move cursor | `cua do move <x> <y>` |-| Run shell command | `cua do shell "command"` |-| Open URL or file | `cua do open <url\|path>` |-| List windows | `cua do window ls [app]` |-| Focus window | `cua do window focus <id>` |-| Share trajectory | `cua trajectory share` |--See [references/command-reference.md](references/command-reference.md) for full-argument syntax and all subcommands.--## Supported Providers--| Provider | Target | Example |-| ------------ | ----------------- | ----------------------------------- |-| `cloud` | CUA cloud VM | `cua do switch cloud my-vm` |-| `cloudv2` | CUA cloud VM (v2) | `cua do switch cloudv2 my-vm` |-| `docker` | Docker container | `cua do switch docker my-container` |-| `lume` | Lume VM | `cua do switch lume my-vm` |-| `lumier` | Lumier VM | `cua do switch lumier my-vm` |-| `winsandbox` | Windows Sandbox | `cua do switch winsandbox` |-| `host` | Local machine | `cua do switch host` |+| Action | Command |+| ------------------- | ---------------------------------------------- |+| Connect to target | `cua do switch <provider> [name]` |+| Screenshot | `cua do screenshot` |+| AI-annotated screen | `cua do snapshot ["instructions"]` |+| Click | `cua do click <x> <y> [left\|right\|middle]` |+| Double-click | `cua do dclick <x> <y>` |+| Type text | `cua do type "text"` |+| Press key | `cua do key <key>` |+| Hotkey | `cua do hotkey <combo>` (e.g. `ctrl+c`) |+| Scroll | `cua do scroll <direction> [amount]` |+| Drag | `cua do drag <x1> <y1> <x2> <y2>` |+| Move cursor | `cua do move <x> <y>` |+| Shell command | `cua do shell "command"` |+| Open URL/file | `cua do open <url\|path>` |+| List windows | `cua do window ls [app]` |+| Focus window | `cua do window focus <id>` |+| Zoom to window | `cua do zoom "App Name"` |+| Unzoom | `cua do unzoom` |+| Share trajectory | `cua trajectory share` |++## Providers++| Provider | Example |+| ------------ | ------------------------------------- |+| `cloud` | `cua do switch cloud my-vm` |+| `cloudv2` | `cua do switch cloudv2 my-vm` |+| `docker` | `cua do switch docker my-container` |+| `lume` | `cua do switch lume my-vm` |+| `lumier` | `cua do switch lumier my-vm` |+| `winsandbox` | `cua do switch winsandbox` |+| `host` | `cua do switch host` |++See [references/command-reference.md](references/command-reference.md) for full argument syntax.