Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

Add agent-computer/peakaboo style CLI group to cua-cli, add `cua do` SKILL.md, and cua-auto package (#1107)

ddupont90278dd730c4
diff --git a/.github/workflows/cd-py-auto.yml b/.github/workflows/cd-py-auto.yml
new file mode 100644
index 0000000000..52be003fdd
--- /dev/null
+++ b/.github/workflows/cd-py-auto.yml
@@ -0,0 +1,74 @@
+name: "CD: cua-auto (PyPI)"
+
+on:
+ push:
+ tags:
+ - "auto-v*"
+ workflow_dispatch:
+ inputs:
+ version:
+ description: "Version to publish (without v prefix)"
+ required: true
+ default: "0.1.0"
+ workflow_call:
+ inputs:
+ version:
+ description: "Version to publish"
+ required: true
+ type: string
+
+# Adding permissions at workflow level
+permissions:
+ contents: write
+
+jobs:
+ prepare:
+ runs-on: macos-latest
+ outputs:
+ version: ${{ steps.get-version.outputs.version }}
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Determine version
+ id: get-version
+ run: |
+ # Check inputs.version first (set by workflow_call)
+ if [ -n "${{ inputs.version }}" ]; then
+ VERSION=${{ inputs.version }}
+ elif [ "${{ github.event_name }}" == "push" ]; then
+ # Extract version from tag (for package-specific tags)
+ if [[ "${{ github.ref }}" =~ ^refs/tags/auto-v([0-9]+\.[0-9]+\.[0-9]+) ]]; then
+ VERSION=${BASH_REMATCH[1]}
+ else
+ echo "Invalid tag format for auto"
+ exit 1
+ fi
+ elif [ "${{ github.event_name }}" == "workflow_dispatch" ]; then
+ # Use version from workflow dispatch
+ VERSION=${{ github.event.inputs.version }}
+ else
+ echo "No version provided"
+ exit 1
+ fi
+ echo "VERSION=$VERSION"
+ echo "version=$VERSION" >> $GITHUB_OUTPUT
+
+ publish:
+ needs: prepare
+ uses: ./.github/workflows/py-reusable-publish.yml
+ with:
+ package_name: "auto"
+ package_dir: "libs/python/cua-auto"
+ version: ${{ needs.prepare.outputs.version }}
+ base_package_name: "cua-auto"
+ secrets:
+ PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
+
+ create-release:
+ needs: [prepare, publish]
+ uses: ./.github/workflows/release-github-reusable.yml
+ with:
+ tag_name: "auto-v${{ needs.prepare.outputs.version }}"
+ release_name: "cua-auto v${{ needs.prepare.outputs.version }}"
+ module_path: "libs/python/cua-auto"
+ body: ""
diff --git a/.github/workflows/ci-release-reminder.yml b/.github/workflows/ci-release-reminder.yml
index d0d6c49821..262caf3b52 100644
--- a/.github/workflows/ci-release-reminder.yml
+++ b/.github/workflows/ci-release-reminder.yml
@@ -27,6 +27,7 @@ jobs:
# Map paths to publishable services
declare -A PATH_TO_SERVICE=(
["libs/python/cua-cli/"]="pypi/cli"
+ ["libs/python/cua-auto/"]="pypi/auto"
["libs/python/agent/"]="pypi/agent"
["libs/python/computer/"]="pypi/computer"
["libs/python/core/"]="pypi/core"
diff --git a/.github/workflows/release-bump-version.yml b/.github/workflows/release-bump-version.yml
index 68512cc1ab..5eab5ed8a2 100644
--- a/.github/workflows/release-bump-version.yml
+++ b/.github/workflows/release-bump-version.yml
@@ -9,6 +9,7 @@ on:
type: choice
options:
- pypi/agent
+ - pypi/auto
- pypi/bench
- pypi/bench-ui
- pypi/cli
@@ -54,6 +55,10 @@ jobs:
echo "directory=libs/python/agent" >> $GITHUB_OUTPUT
echo "type=python" >> $GITHUB_OUTPUT
;;
+ "pypi/auto")
+ echo "directory=libs/python/cua-auto" >> $GITHUB_OUTPUT
+ echo "type=python" >> $GITHUB_OUTPUT
+ ;;
"pypi/bench")
echo "directory=libs/cua-bench" >> $GITHUB_OUTPUT
echo "type=python" >> $GITHUB_OUTPUT
@@ -212,6 +217,15 @@ jobs:
echo "Agent version: $VERSION"
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
+ - name: Capture bumped auto version
+ if: ${{ inputs.service == 'pypi/auto' }}
+ id: auto_version
+ run: |
+ cd libs/python/cua-auto
+ VERSION=$(python -c "import tomllib; from pathlib import Path; data = tomllib.loads(Path('pyproject.toml').read_text()); print(data['project']['version'])")
+ echo "Auto version: $VERSION"
+ echo "version=$VERSION" >> "$GITHUB_OUTPUT"
+
- name: Capture bumped bench version
if: ${{ inputs.service == 'pypi/bench' }}
id: bench_version
diff --git a/.github/workflows/release-on-merge.yml b/.github/workflows/release-on-merge.yml
index 4b416aa92d..39e33fb511 100644
--- a/.github/workflows/release-on-merge.yml
+++ b/.github/workflows/release-on-merge.yml
@@ -47,6 +47,7 @@ jobs:
# Map paths to publishable services
declare -A PATH_TO_SERVICE=(
["libs/python/cua-cli/"]="pypi/cli"
+ ["libs/python/cua-auto/"]="pypi/auto"
["libs/python/agent/"]="pypi/agent"
["libs/python/computer/"]="pypi/computer"
["libs/python/core/"]="pypi/core"
diff --git a/.github/workflows/release-unreleased-digest.yml b/.github/workflows/release-unreleased-digest.yml
index c4582900d2..4ef890ee41 100644
--- a/.github/workflows/release-unreleased-digest.yml
+++ b/.github/workflows/release-unreleased-digest.yml
@@ -21,6 +21,7 @@ jobs:
# Service → tag prefix → directory
declare -A SERVICE_TAG_DIR
SERVICE_TAG_DIR["pypi/cli"]="cli-v|libs/python/cua-cli/"
+ SERVICE_TAG_DIR["pypi/auto"]="auto-v|libs/python/cua-auto/"
SERVICE_TAG_DIR["pypi/agent"]="agent-v|libs/python/agent/"
SERVICE_TAG_DIR["pypi/computer"]="computer-v|libs/python/computer/"
SERVICE_TAG_DIR["pypi/core"]="core-v|libs/python/core/"
diff --git a/Development.md b/Development.md
index 590f02a071..6d843eabb1 100644
--- a/Development.md
+++ b/Development.md
@@ -305,6 +305,7 @@ All packages are managed through a single consolidated workflow: [Bump Version &
**Python (PyPI):**
- `pypi/agent` - AI agent library
+- `pypi/auto` - Cross-platform automation library (mouse, keyboard, screen, window, clipboard, shell)
- `pypi/bench` - Benchmark toolkit for computer-use RL environments
- `pypi/computer` - Computer-use interface library
- `pypi/computer-server` - Server component for VM
@@ -365,6 +366,7 @@ make show-versions
Each package uses its own tag format defined in `.bumpversion.cfg`:
- **cua-agent**: `agent-v{version}` (e.g., `agent-v0.4.35`)
+- **cua-auto**: `auto-v{version}` (e.g., `auto-v0.1.0`)
- **cua-bench**: `bench-v{version}` (e.g., `bench-v0.1.0`)
- **cua-computer**: `computer-v{version}` (e.g., `computer-v0.4.7`)
- **cua-computer-server**: `computer-server-v{version}` (e.g., `computer-server-v0.1.27`)
diff --git a/libs/python/computer-server/computer_server/handlers/android.py b/libs/python/computer-server/computer_server/handlers/android.py
index 2e599cb6fe..583b758aea 100644
--- a/libs/python/computer-server/computer_server/handlers/android.py
+++ b/libs/python/computer-server/computer_server/handlers/android.py
@@ -234,6 +234,12 @@ class AndroidAutomationHandler(BaseAutomationHandler):
else:
raise RuntimeError(f"Long press failed: {output}")
+ async def middle_click(
+ self, x: Optional[int] = None, y: Optional[int] = None
+ ) -> Dict[str, Any]:
+ """Middle click is not supported on Android; returns not-supported error."""
+ return {"success": False, "error": "middle_click is not supported on Android"}
+
async def double_click(
self, x: Optional[int] = None, y: Optional[int] = None
) -> Dict[str, Any]:
diff --git a/libs/python/computer-server/computer_server/handlers/base.py b/libs/python/computer-server/computer_server/handlers/base.py
index ea977eb2f8..d38ef553a0 100644
--- a/libs/python/computer-server/computer_server/handlers/base.py
+++ b/libs/python/computer-server/computer_server/handlers/base.py
@@ -217,6 +217,13 @@ class BaseAutomationHandler(ABC):
"""Perform a right click at the current or specified position."""
pass
+ @abstractmethod
+ async def middle_click(
+ self, x: Optional[int] = None, y: Optional[int] = None
+ ) -> Dict[str, Any]:
+ """Perform a middle click at the current or specified position."""
+ pass
+
@abstractmethod
async def double_click(
self, x: Optional[int] = None, y: Optional[int] = None
diff --git a/libs/python/computer-server/computer_server/handlers/linux.py b/libs/python/computer-server/computer_server/handlers/linux.py
index 25ccd3f979..52b2586653 100644
--- a/libs/python/computer-server/computer_server/handlers/linux.py
+++ b/libs/python/computer-server/computer_server/handlers/linux.py
@@ -221,6 +221,20 @@ class LinuxAutomationHandler(BaseAutomationHandler):
except Exception as e:
return {"success": False, "error": str(e)}
+ async def middle_click(
+ self, x: Optional[int] = None, y: Optional[int] = None
+ ) -> Dict[str, Any]:
+ """Perform a middle mouse click at the specified coordinates."""
+ try:
+ from pynput.mouse import Button
+
+ if x is not None and y is not None:
+ self.mouse.position = (x, y)
+ self.mouse.click(Button.middle, 1)
+ return {"success": True}
+ except Exception as e:
+ return {"success": False, "error": str(e)}
+
async def double_click(
self, x: Optional[int] = None, y: Optional[int] = None
) -> Dict[str, Any]:
diff --git a/libs/python/computer-server/computer_server/handlers/macos.py b/libs/python/computer-server/computer_server/handlers/macos.py
index 11bff6c325..c6bf41e1ab 100644
--- a/libs/python/computer-server/computer_server/handlers/macos.py
+++ b/libs/python/computer-server/computer_server/handlers/macos.py
@@ -1058,6 +1058,18 @@ class MacOSAutomationHandler(BaseAutomationHandler):
except Exception as e:
return {"success": False, "error": str(e)}
+ async def middle_click(
+ self, x: Optional[int] = None, y: Optional[int] = None
+ ) -> Dict[str, Any]:
+ """Perform a middle mouse click at the specified coordinates."""
+ try:
+ if x is not None and y is not None:
+ self.mouse.position = (x, y)
+ self.mouse.click(Button.middle, 1)
+ return {"success": True}
+ except Exception as e:
+ return {"success": False, "error": str(e)}
+
async def double_click(
self, x: Optional[int] = None, y: Optional[int] = None
) -> Dict[str, Any]:
diff --git a/libs/python/computer-server/computer_server/handlers/windows.py b/libs/python/computer-server/computer_server/handlers/windows.py
index c1152207be..b8ff768ab3 100644
--- a/libs/python/computer-server/computer_server/handlers/windows.py
+++ b/libs/python/computer-server/computer_server/handlers/windows.py
@@ -374,6 +374,27 @@ class WindowsAutomationHandler(BaseAutomationHandler):
except Exception as e:
return {"success": False, "error": str(e)}
+ @require_unlocked_desktop
+ async def middle_click(
+ self, x: Optional[int] = None, y: Optional[int] = None
+ ) -> Dict[str, Any]:
+ """Perform a middle mouse click at the specified coordinates.
+
+ Args:
+ x (Optional[int]): The x-coordinate to click at. If None, clicks at current position.
+ y (Optional[int]): The y-coordinate to click at. If None, clicks at current position.
+
+ Returns:
+ Dict[str, Any]: A dictionary with success status and optional error message.
+ """
+ try:
+ if x is not None and y is not None:
+ self.mouse.position = (x, y)
+ self.mouse.click(MouseButton.middle, 1)
+ return {"success": True}
+ except Exception as e:
+ return {"success": False, "error": str(e)}
+
@require_unlocked_desktop
async def double_click(
self, x: Optional[int] = None, y: Optional[int] = None
diff --git a/libs/python/cua-auto/.bumpversion.cfg b/libs/python/cua-auto/.bumpversion.cfg
new file mode 100644
index 0000000000..e425745425
--- /dev/null
+++ b/libs/python/cua-auto/.bumpversion.cfg
@@ -0,0 +1,14 @@
+[bumpversion]
+current_version = 0.1.0
+commit = True
+tag = True
+tag_name = auto-v{new_version}
+message = Bump cua-auto to v{new_version}
+
+[bumpversion:file:pyproject.toml]
+search = version = "{current_version}"
+replace = version = "{new_version}"
+
+[bumpversion:file:cua_auto/__init__.py]
+search = __version__ = "{current_version}"
+replace = __version__ = "{new_version}"
diff --git a/libs/python/cua-auto/README.md b/libs/python/cua-auto/README.md
new file mode 100644
index 0000000000..b6f8e454b1
--- /dev/null
+++ b/libs/python/cua-auto/README.md
@@ -0,0 +1,62 @@
+# cua-auto
+
+`cua-auto` is a lightweight, cross-platform automation library providing a synchronous, MIT-licensed pyautogui-style API for mouse, keyboard, screen, window, clipboard, and shell operations. It runs on Windows, macOS, and Linux using [pynput](https://github.com/moses-palmer/pynput) for input control and [pywinctl](https://github.com/Kalmat/PyWinCtl) for window management.
+
+```python
+import cua_auto.mouse as mouse
+import cua_auto.keyboard as keyboard
+import cua_auto.screen as screen
+import cua_auto.window as window
+import cua_auto.clipboard as clipboard
+import cua_auto.shell as shell
+
+# Mouse
+mouse.click(100, 200) # left click
+mouse.right_click(100, 200)
+mouse.double_click(100, 200)
+mouse.move_to(500, 300)
+mouse.mouse_down(100, 200)
+mouse.mouse_up(100, 200)
+mouse.drag(100, 200, 400, 500) # start → end
+mouse.scroll_up(3)
+mouse.scroll_down(3)
+x, y = mouse.position()
+
+# Keyboard
+keyboard.press_key("enter")
+keyboard.type_text("hello world")
+keyboard.hotkey(["ctrl", "c"])
+keyboard.key_down("shift")
+keyboard.key_up("shift")
+
+# Screen
+img = screen.screenshot() # returns PIL.Image
+png = screen.screenshot_bytes() # raw PNG bytes
+b64 = screen.screenshot_b64() # base64 string
+w, h = screen.screen_size()
+x, y = screen.cursor_position()
+
+# Window
+title = window.get_active_window_title()
+handle = window.get_active_window_handle()
+handles = window.get_windows_with_title("Chrome")
+name = window.get_window_name(handle)
+x, y = window.get_window_position(handle)
+w, h = window.get_window_size(handle)
+window.activate_window(handle)
+window.minimize_window(handle)
+window.maximize_window(handle)
+window.close_window(handle)
+window.set_window_size(handle, 1280, 800)
+window.set_window_position(handle, 0, 0)
+window.open("https://example.com") # or file path
+pid = window.launch("notepad.exe")
+
+# Clipboard
+text = clipboard.get()
+clipboard.set("hello")
+
+# Shell
+result = shell.run("echo hi") # CommandResult
+print(result.stdout, result.returncode)
+```
diff --git a/libs/python/cua-auto/cua_auto/__init__.py b/libs/python/cua-auto/cua_auto/__init__.py
new file mode 100644
index 0000000000..1817577410
--- /dev/null
+++ b/libs/python/cua-auto/cua_auto/__init__.py
@@ -0,0 +1,25 @@
+"""cua-auto — cross-platform automation library.
+
+Provides a synchronous, pyautogui-style API for mouse, keyboard, screen,
+window, clipboard, and shell operations.
+
+Usage::
+
+ import cua_auto.mouse as mouse
+ import cua_auto.keyboard as keyboard
+ import cua_auto.screen as screen
+ import cua_auto.window as window
+ import cua_auto.clipboard as clipboard
+ import cua_auto.shell as shell
+
+ mouse.click(100, 200)
+ keyboard.hotkey(["ctrl", "c"])
+ img = screen.screenshot()
+ title = window.get_active_window_title()
+"""
+
+__version__ = "0.1.0"
+
+from cua_auto import clipboard, keyboard, mouse, screen, shell, window
+
+__all__ = ["mouse", "keyboard", "screen", "window", "clipboard", "shell"]
diff --git a/libs/python/cua-auto/cua_auto/_platform.py b/libs/python/cua-auto/cua_auto/_platform.py
new file mode 100644
index 0000000000..e9f5b4f65f
--- /dev/null
+++ b/libs/python/cua-auto/cua_auto/_platform.py
@@ -0,0 +1,11 @@
+"""Platform detection helpers."""
+
+import platform as _platform
+
+_sys = _platform.system().lower()
+
+IS_WINDOWS: bool = _sys == "windows"
+IS_MACOS: bool = _sys == "darwin"
+IS_LINUX: bool = _sys == "linux"
+
+PLATFORM: str = "windows" if IS_WINDOWS else ("macos" if IS_MACOS else "linux")
diff --git a/libs/python/cua-auto/cua_auto/clipboard.py b/libs/python/cua-auto/cua_auto/clipboard.py
new file mode 100644
index 0000000000..92a0f844b7
--- /dev/null
+++ b/libs/python/cua-auto/cua_auto/clipboard.py
@@ -0,0 +1,13 @@
+"""Cross-platform clipboard access via pyperclip."""
+
+import pyperclip as _pyperclip
+
+
+def get() -> str:
+ """Return the current clipboard text content."""
+ return _pyperclip.paste()
+
+
+def set(text: str) -> None:
+ """Set the clipboard text content."""
+ _pyperclip.copy(text)
diff --git a/libs/python/cua-auto/cua_auto/keyboard.py b/libs/python/cua-auto/cua_auto/keyboard.py
new file mode 100644
index 0000000000..25c15a8a92
--- /dev/null
+++ b/libs/python/cua-auto/cua_auto/keyboard.py
@@ -0,0 +1,153 @@
+"""Cross-platform keyboard control via pynput."""
+
+from typing import List, Optional, Union
+
+from pynput.keyboard import Controller as _KBController
+from pynput.keyboard import Key as _Key
+
+_kb = _KBController()
+
+# Unified key name → pynput Key map (sourced from computer-server/handlers/windows.py)
+_SPECIAL: dict = {
+ "enter": _Key.enter,
+ "return": _Key.enter,
+ "esc": _Key.esc,
+ "escape": _Key.esc,
+ "space": _Key.space,
+ "tab": _Key.tab,
+ "backspace": _Key.backspace,
+ "delete": _Key.delete,
+ "home": _Key.home,
+ "end": _Key.end,
+ "pageup": _Key.page_up,
+ "page_up": _Key.page_up,
+ "pagedown": _Key.page_down,
+ "page_down": _Key.page_down,
+ "up": _Key.up,
+ "down": _Key.down,
+ "left": _Key.left,
+ "right": _Key.right,
+ "shift": _Key.shift,
+ "shift_l": _Key.shift_l,
+ "shift_r": _Key.shift_r,
+ "ctrl": _Key.ctrl,
+ "ctrl_l": _Key.ctrl_l,
+ "ctrl_r": _Key.ctrl_r,
+ "control": _Key.ctrl,
+ "alt": _Key.alt,
+ "alt_l": _Key.alt_l,
+ "alt_r": _Key.alt_r,
+ "cmd": _Key.cmd,
+ "command": _Key.cmd,
+ "win": _Key.cmd,
+ "super": _Key.cmd,
+ "meta": _Key.cmd,
+ "option": _Key.alt,
+ "option_l": _Key.alt_l,
+ "option_r": _Key.alt_r,
+ "capslock": _Key.caps_lock,
+ "caps_lock": _Key.caps_lock,
+ "insert": _Key.insert,
+ "print_screen": _Key.print_screen,
+ "pause": _Key.pause,
+ "num_lock": _Key.num_lock,
+ "scroll_lock": _Key.scroll_lock,
+ "f1": _Key.f1,
+ "f2": _Key.f2,
+ "f3": _Key.f3,
+ "f4": _Key.f4,
+ "f5": _Key.f5,
+ "f6": _Key.f6,
+ "f7": _Key.f7,
+ "f8": _Key.f8,
+ "f9": _Key.f9,
+ "f10": _Key.f10,
+ "f11": _Key.f11,
+ "f12": _Key.f12,
+ "f13": _Key.f13,
+ "f14": _Key.f14,
+ "f15": _Key.f15,
+ "f16": _Key.f16,
+ "f17": _Key.f17,
+ "f18": _Key.f18,
+ "f19": _Key.f19,
+ "f20": _Key.f20,
+}
+
+
+def _resolve(key: str) -> Optional[Union[str, _Key]]:
+ """Resolve a key string to a pynput Key or single character."""
+ if not key:
+ return None
+ lk = key.lower()
+ if lk in _SPECIAL:
+ return _SPECIAL[lk]
+ if len(key) == 1:
+ return key
+ return None
+
+
+# ── Key press / release ───────────────────────────────────────────────────────
+
+
+def key_down(key: str) -> None:
+ """Press and hold a key."""
+ k = _resolve(key)
+ if k is None:
+ raise ValueError(f"Unknown key: {key!r}")
+ _kb.press(k)
+
+
+def key_up(key: str) -> None:
+ """Release a key."""
+ k = _resolve(key)
+ if k is None:
+ raise ValueError(f"Unknown key: {key!r}")
+ _kb.release(k)
+
+
+def press_key(key: str) -> None:
+ """Press and immediately release a key."""
+ k = _resolve(key)
+ if k is None:
+ raise ValueError(f"Unknown key: {key!r}")
+ _kb.press(k)
+ _kb.release(k)
+
+
+# ── Text typing ───────────────────────────────────────────────────────────────
+
+
+def type_text(text: str) -> None:
+ """Type a string of text (supports Unicode)."""
+ _kb.type(text)
+
+
+# ── Hotkeys ───────────────────────────────────────────────────────────────────
+
+
+def hotkey(keys: List[str]) -> None:
+ """Press a combination of keys, e.g. ['ctrl', 'c'].
+
+ Modifiers are held while the last key is tapped, then released in reverse
+ order (standard hotkey behaviour).
+ """
+ resolved = [_resolve(k) for k in keys]
+ if any(k is None for k in resolved):
+ bad = [k for k, r in zip(keys, resolved) if r is None]
+ raise ValueError(f"Unknown keys in hotkey: {bad}")
+
+ seq: List[Union[str, _Key]] = [k for k in resolved if k is not None]
+ if not seq:
+ return
+
+ # Hold all modifiers except the last key
+ for k in seq[:-1]:
+ _kb.press(k)
+ # Tap the action key
+ last = seq[-1]
+ _kb.press(last)
+ _kb.release(last)
+ # Release modifiers in reverse order
+ for k in reversed(seq[:-1]):
+ _kb.release(k)
diff --git a/libs/python/cua-auto/cua_auto/mouse.py b/libs/python/cua-auto/cua_auto/mouse.py
new file mode 100644
index 0000000000..4e0691de90
--- /dev/null
+++ b/libs/python/cua-auto/cua_auto/mouse.py
@@ -0,0 +1,138 @@
+"""Cross-platform mouse control via pynput."""
+
+from typing import List, Optional, Tuple
+
+from pynput.mouse import Button as _Button
+from pynput.mouse import Controller as _MouseController
+
+_mouse = _MouseController()
+
+
+def _map_button(button: str) -> _Button:
+ """Map a button name string to a pynput Button."""
+ b = (button or "left").lower()
+ if b == "right":
+ return _Button.right
+ if b == "middle":
+ return _Button.middle
+ return _Button.left
+
+
+# ── Basic clicks ──────────────────────────────────────────────────────────────
+
+
+def click(x: int, y: int, button: str = "left") -> None:
+ """Single click at (x, y)."""
+ _mouse.position = (x, y)
+ _mouse.click(_map_button(button), 1)
+
+
+def right_click(x: int, y: int) -> None:
+ """Right-click at (x, y)."""
+ click(x, y, "right")
+
+
+def double_click(x: int, y: int) -> None:
+ """Double left-click at (x, y)."""
+ _mouse.position = (x, y)
+ _mouse.click(_Button.left, 2)
+
+
+# ── Cursor movement ───────────────────────────────────────────────────────────
+
+
+def move_to(x: int, y: int) -> None:
+ """Move cursor to (x, y) without clicking."""
+ _mouse.position = (x, y)
+
+
+# ── Mouse button hold / release ───────────────────────────────────────────────
+
+
+def mouse_down(x: Optional[int] = None, y: Optional[int] = None, button: str = "left") -> None:
+ """Press and hold a mouse button, optionally moving to (x, y) first."""
+ if x is not None and y is not None:
+ _mouse.position = (x, y)
+ _mouse.press(_map_button(button))
+
+
+def mouse_up(x: Optional[int] = None, y: Optional[int] = None, button: str = "left") -> None:
+ """Release a mouse button, optionally moving to (x, y) first."""
+ if x is not None and y is not None:
+ _mouse.position = (x, y)
+ _mouse.release(_map_button(button))
+
+
+# ── Drag ──────────────────────────────────────────────────────────────────────
+
+
+def drag(
+ start_x: int,
+ start_y: int,
+ end_x: int,
+ end_y: int,
+ button: str = "left",
+) -> None:
+ """Drag from (start_x, start_y) to (end_x, end_y)."""
+ btn = _map_button(button)
+ _mouse.position = (start_x, start_y)
+ _mouse.press(btn)
+ _mouse.position = (end_x, end_y)
+ _mouse.release(btn)
+
+
+def drag_to(x: int, y: int, button: str = "left") -> None:
+ """Drag from the current cursor position to (x, y)."""
+ btn = _map_button(button)
+ _mouse.press(btn)
+ _mouse.position = (x, y)
+ _mouse.release(btn)
+
+
+def drag_path(path: List[Tuple[int, int]], button: str = "left") -> None:
+ """Drag through a sequence of (x, y) points."""
+ if not path:
+ return
+ btn = _map_button(button)
+ _mouse.position = path[0]
+ _mouse.press(btn)
+ for x, y in path[1:]:
+ _mouse.position = (x, y)
+ _mouse.release(btn)
+
+
+# ── Scroll ────────────────────────────────────────────────────────────────────
+
+
+def scroll(dx: int, dy: int) -> None:
+ """Scroll by (dx, dy). Positive dy = scroll up."""
+ _mouse.scroll(dx, dy)
+
+
+def scroll_up(clicks: int = 3) -> None:
+ """Scroll up by *clicks* notches."""
+ _mouse.scroll(0, abs(clicks))
+
+
+def scroll_down(clicks: int = 3) -> None:
+ """Scroll down by *clicks* notches."""
+ _mouse.scroll(0, -abs(clicks))
+
+
+def scroll_left(clicks: int = 3) -> None:
+ """Scroll left by *clicks* notches."""
+ _mouse.scroll(-abs(clicks), 0)
+
+
+def scroll_right(clicks: int = 3) -> None:
+ """Scroll right by *clicks* notches."""
+ _mouse.scroll(abs(clicks), 0)
+
+
+# ── Position ──────────────────────────────────────────────────────────────────
+
+
+def position() -> Tuple[int, int]:
+ """Return the current cursor position as (x, y)."""
+ x, y = _mouse.position
+ return int(x), int(y)
diff --git a/libs/python/cua-auto/cua_auto/screen.py b/libs/python/cua-auto/cua_auto/screen.py
new file mode 100644
index 0000000000..e1c1988af5
--- /dev/null
+++ b/libs/python/cua-auto/cua_auto/screen.py
@@ -0,0 +1,76 @@
+"""Cross-platform screen capture and info."""
+
+import base64
+import io
+from typing import Tuple
+
+from PIL import Image
+
+
+def screenshot() -> Image.Image:
+ """Capture a screenshot of all monitors and return a PIL Image.
+
+ Tries PIL.ImageGrab first (works on Windows and macOS).
+ Falls back to mss if ImageGrab fails (better Linux / multi-monitor support).
+ """
+ # Try PIL.ImageGrab (Windows + macOS)
+ try:
+ from PIL import ImageGrab
+
+ img = ImageGrab.grab(all_screens=True)
+ if isinstance(img, Image.Image):
+ return img
+ except Exception:
+ pass
+
+ # Fallback: mss (cross-platform, optional dependency)
+ try:
+ import mss # type: ignore[import-untyped]
+ import mss.tools # type: ignore[import-untyped]
+
+ with mss.mss() as sct:
+ monitor = sct.monitors[0] # combined virtual desktop
+ sct_img = sct.grab(monitor)
+ return Image.frombytes("RGB", sct_img.size, sct_img.bgra, "raw", "BGRX")
+ except Exception as e:
+ raise RuntimeError(f"screenshot failed: {e}") from e
+
+
+def screenshot_bytes(format: str = "PNG") -> bytes:
+ """Return screenshot as raw bytes in the specified image format."""
+ img = screenshot()
+ buf = io.BytesIO()
+ img.save(buf, format=format)
+ return buf.getvalue()
+
+
+def screenshot_b64(format: str = "PNG") -> str:
+ """Return screenshot as a base64-encoded string."""
+ return base64.b64encode(screenshot_bytes(format)).decode()
+
+
+def screen_size() -> Tuple[int, int]:
+ """Return the total virtual desktop size as (width, height)."""
+ img = screenshot()
+ return img.size
+
+
+def cursor_position() -> Tuple[int, int]:
+ """Return the current cursor position as (x, y).
+
+ Tries win32gui first on Windows (more reliable with DPI scaling).
+ Falls back to pynput if win32 is unavailable (cross-platform fallback).
+ """
+ try:
+ import win32gui # type: ignore[import-untyped]
+
+ pos = win32gui.GetCursorPos()
+ return int(pos[0]), int(pos[1])
+ except Exception:
+ pass
+
+ from pynput.mouse import Controller as _MouseController
+
+ c = _MouseController()
+ x, y = c.position
+ return int(x), int(y)
diff --git a/libs/python/cua-auto/cua_auto/shell.py b/libs/python/cua-auto/cua_auto/shell.py
new file mode 100644
index 0000000000..9ae019c509
--- /dev/null
+++ b/libs/python/cua-auto/cua_auto/shell.py
@@ -0,0 +1,53 @@
+"""Shell command execution."""
+
+import subprocess
+from dataclasses import dataclass
+
+
+@dataclass
+class CommandResult:
+ stdout: str
+ stderr: str
+ returncode: int
+
+ @property
+ def success(self) -> bool:
+ return self.returncode == 0
+
+
+def run(command: str, timeout: int = 30) -> CommandResult:
+ """Run a shell command and return stdout, stderr, and returncode.
+
+ The command is passed to the system shell (``shell=True``) so that
+ shell built-ins, pipes, and redirections work as expected.
+ """
+
+ def _decode(data: bytes) -> str:
+ if not data:
+ return ""
+ for enc in ("utf-8", "gbk", "gb2312", "cp936", "latin1"):
+ try:
+ return data.decode(enc)
+ except (UnicodeDecodeError, LookupError):
+ continue
+ return data.decode("utf-8", errors="replace")
+
+ try:
+ result = subprocess.run(
+ command,
+ shell=True,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ timeout=timeout,
+ )
+ return CommandResult(
+ stdout=_decode(result.stdout),
+ stderr=_decode(result.stderr),
+ returncode=result.returncode,
+ )
+ except subprocess.TimeoutExpired as e:
+ return CommandResult(
+ stdout=_decode(e.stdout or b""),
+ stderr=_decode(e.stderr or b""),
+ returncode=-1,
+ )
diff --git a/libs/python/cua-auto/cua_auto/window.py b/libs/python/cua-auto/cua_auto/window.py
new file mode 100644
index 0000000000..df0aa5bbea
--- /dev/null
+++ b/libs/python/cua-auto/cua_auto/window.py
@@ -0,0 +1,188 @@
+"""Cross-platform window management via pywinctl."""
+
+import os
+import platform
+import subprocess
+import webbrowser
+from typing import Any, List, Optional, Tuple
+
+try:
+ import pywinctl as _pwc # type: ignore[import-untyped]
+except Exception:
+ _pwc = None # type: ignore[assignment]
+
+
+def _require_pwc() -> Any:
+ if _pwc is None:
+ raise RuntimeError("pywinctl is not available. Install it with: pip install pywinctl")
+ return _pwc
+
+
+def _get_by_handle(handle: Any) -> Optional[Any]:
+ """Find a pywinctl window object by its native handle."""
+ pwc = _require_pwc()
+ try:
+ for w in pwc.getAllWindows():
+ if str(w.getHandle()) == str(handle):
+ return w
+ except Exception:
+ pass
+ return None
+
+
+# ── Active window ─────────────────────────────────────────────────────────────
+
+
+def get_active_window() -> Optional[Any]:
+ """Return the pywinctl window object for the currently focused window."""
+ pwc = _require_pwc()
+ return pwc.getActiveWindow()
+
+
+def get_active_window_title() -> str:
+ """Return the title of the currently focused window, or 'Desktop'."""
+ try:
+ win = get_active_window()
+ if win:
+ return win.title or "Desktop"
+ except Exception:
+ pass
+ return "Desktop"
+
+
+def get_active_window_handle() -> Optional[str]:
+ """Return the native handle of the active window as a string, or None."""
+ try:
+ win = get_active_window()
+ if win:
+ return str(win.getHandle())
+ except Exception:
+ pass
+ return None
+
+
+# ── Window lookup ─────────────────────────────────────────────────────────────
+
+
+def get_windows_with_title(title: str) -> List[str]:
+ """Return a list of native handle strings for windows whose title contains *title*."""
+ pwc = _require_pwc()
+ try:
+ wins = pwc.getWindowsWithTitle(title, condition=pwc.Re.CONTAINS, flags=pwc.Re.IGNORECASE)
+ return [str(w.getHandle()) for w in wins]
+ except Exception:
+ return []
+
+
+def get_window_name(handle: Any) -> Optional[str]:
+ """Return the title of the window with the given handle, or None."""
+ w = _get_by_handle(handle)
+ return w.title if w else None
+
+
+def get_window_size(handle: Any) -> Optional[Tuple[int, int]]:
+ """Return (width, height) of the window, or None if not found."""
+ w = _get_by_handle(handle)
+ if not w:
+ return None
+ width, height = w.size
+ return int(width), int(height)
+
+
+def get_window_position(handle: Any) -> Optional[Tuple[int, int]]:
+ """Return (x, y) position of the window, or None if not found."""
+ w = _get_by_handle(handle)
+ if not w:
+ return None
+ x, y = w.position
+ return int(x), int(y)
+
+
+# ── Window actions ────────────────────────────────────────────────────────────
+
+
+def activate_window(handle: Any) -> bool:
+ """Bring the window to the foreground and give it focus."""
+ w = _get_by_handle(handle)
+ if not w:
+ return False
+ return bool(w.activate())
+
+
+def minimize_window(handle: Any) -> bool:
+ """Minimize the window."""
+ w = _get_by_handle(handle)
+ if not w:
+ return False
+ return bool(w.minimize())
+
+
+def maximize_window(handle: Any) -> bool:
+ """Maximize the window."""
+ w = _get_by_handle(handle)
+ if not w:
+ return False
+ return bool(w.maximize())
+
+
+def close_window(handle: Any) -> bool:
+ """Close the window."""
+ w = _get_by_handle(handle)
+ if not w:
+ return False
+ return bool(w.close())
+
+
+def set_window_size(handle: Any, width: int, height: int) -> bool:
+ """Resize the window to (width, height)."""
+ w = _get_by_handle(handle)
+ if not w:
+ return False
+ return bool(w.resizeTo(int(width), int(height)))
+
+
+def set_window_position(handle: Any, x: int, y: int) -> bool:
+ """Move the window to (x, y)."""
+ w = _get_by_handle(handle)
+ if not w:
+ return False
+ return bool(w.moveTo(int(x), int(y)))
+
+
+# ── Open / launch ─────────────────────────────────────────────────────────────
+
+
+def open(target: str) -> bool: # noqa: A001
+ """Open a URL or file path with the default application.
+
+ URLs are opened in the default browser. Files are opened with the OS
+ default handler (``open`` on macOS, ``xdg-open`` on Linux,
+ ``os.startfile`` on Windows).
+ """
+ if target.startswith("http://") or target.startswith("https://"):
+ return bool(webbrowser.open(target))
+
+ sys = platform.system().lower()
+ if sys == "darwin":
+ subprocess.Popen(["open", target])
+ elif sys == "linux":
+ subprocess.Popen(["xdg-open", target])
+ elif sys == "windows":
+ os.startfile(target) # type: ignore[attr-defined]
+ else:
+ raise RuntimeError(f"Unsupported OS: {sys}")
+ return True
+
+
+def launch(app: str, args: Optional[List[str]] = None) -> int:
+ """Launch an application and return its PID.
+
+ If *args* is given, the app is launched with those arguments.
+ Otherwise the command string is passed to the system shell, allowing
+ strings like ``"libreoffice --writer"``.
+ """
+ if args:
+ proc = subprocess.Popen([app, *args])
+ else:
+ proc = subprocess.Popen(app, shell=True)
+ return proc.pid
diff --git a/libs/python/cua-auto/pyproject.toml b/libs/python/cua-auto/pyproject.toml
new file mode 100644
index 0000000000..bc16ef506d
--- /dev/null
+++ b/libs/python/cua-auto/pyproject.toml
@@ -0,0 +1,91 @@
+[project]
+name = "cua-auto"
+version = "0.1.0"
+description = "Cross-platform automation library — mouse, keyboard, screen, window, clipboard, shell"
+readme = "README.md"
+license = "MIT"
+authors = [
+ { name = "TryCua", email = "[email protected]" }
+]
+keywords = [
+ "automation",
+ "mouse",
+ "keyboard",
+ "screenshot",
+ "window",
+ "computer-use",
+ "pyautogui",
+]
+classifiers = [
+ "Development Status :: 3 - Alpha",
+ "Intended Audience :: Developers",
+ "License :: OSI Approved :: MIT License",
+ "Operating System :: OS Independent",
+ "Programming Language :: Python :: 3",
+ "Programming Language :: Python :: 3.11",
+ "Programming Language :: Python :: 3.12",
+ "Programming Language :: Python :: 3.13",
+ "Topic :: Software Development :: Libraries",
+]
+requires-python = ">=3.11,<3.14"
+
+dependencies = [
+ # Mouse + keyboard control (cross-platform)
+ "pynput>=1.7.0",
+ # Screenshot + image processing
+ "pillow>=10.0.0",
+ # Clipboard access (cross-platform)
+ "pyperclip>=1.9.0",
+ # Window management (cross-platform)
+ "pywinctl>=0.4",
+]
+
+[project.optional-dependencies]
+# Faster multi-monitor screenshot backend
+mss = [
+ "mss>=9.0.0",
+]
+# Windows-specific extras (cursor position via win32, etc.)
+windows = [
+ "pywin32>=306; sys_platform == 'win32'",
+]
+all = [
+ "cua-auto[mss,windows]",
+]
+dev = [
+ "pytest>=8.0.0",
+ "pytest-asyncio>=0.23.0",
+ "ruff>=0.1.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/trycua/cua"
+Documentation = "https://docs.trycua.com"
+Repository = "https://github.com/trycua/cua"
+Issues = "https://github.com/trycua/cua/issues"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build]
+include = [
+ "cua_auto/**",
+ "README.md",
+ "LICENSE",
+]
+exclude = [
+ "cua_auto/**/__pycache__",
+ "**/.DS_Store",
+]
+
+[tool.hatch.build.targets.wheel]
+packages = ["cua_auto"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["E", "F", "W"]
+ignore = ["E501"]
diff --git a/libs/python/cua-cli/cua_cli/commands/do.py b/libs/python/cua-cli/cua_cli/commands/do.py
new file mode 100644
index 0000000000..bc9ae738aa
--- /dev/null
+++ b/libs/python/cua-cli/cua_cli/commands/do.py
@@ -0,0 +1,1495 @@
+"""cua do — one-shot VM automation commands for agents.
+
+Output is always a single line starting with ✅ (success) or ❌ (failure),
+followed by a context line: 💻 vm-name\t🪟 Window Title (or Desktop).
+
+Target VM is persisted in ~/.cua/do_target.json.
+Zoom state (bbox + display scale) is also persisted there.
+"""
+
+from __future__ import annotations
+
+import argparse
+import base64
+import io
+import json
+import os
+import sys
+import tempfile
+from datetime import datetime
+from pathlib import Path
+from typing import Any
+
+# Ensure stdout/stderr can handle Unicode emojis on Windows
+if sys.platform == "win32":
+ import io as _io
+
+ if hasattr(sys.stdout, "buffer"):
+ sys.stdout = _io.TextIOWrapper(sys.stdout.buffer, encoding="utf-8", errors="replace")
+ if hasattr(sys.stderr, "buffer"):
+ sys.stderr = _io.TextIOWrapper(sys.stderr.buffer, encoding="utf-8", errors="replace")
+
+MAX_LENGTH = 1200 # max pixel dimension for screenshot output
+
+# ── state ─────────────────────────────────────────────────────────────────────
+
+_STATE_FILE = Path.home() / ".cua" / "do_target.json"
+_HOST_CONSENT_FILE = Path.home() / ".cua" / "host_consented"
+
+# Window titles that are internal OS/browser helper windows and should be hidden
+_SKIP_WINDOW_TITLES = {
+ "Chrome Legacy Window",
+}
+
+PROVIDERS = ("cloud", "cloudv2", "lume", "lumier", "docker", "winsandbox", "host")
+_REMOTE_PROVIDERS = ("cloud", "cloudv2", "lume", "lumier", "docker", "winsandbox")
+
+
+def _load_state() -> dict:
+ if _STATE_FILE.exists():
+ try:
+ return json.loads(_STATE_FILE.read_text())
+ except Exception:
+ pass
+ return {}
+
+
+def _save_state(state: dict) -> None:
+ _STATE_FILE.parent.mkdir(parents=True, exist_ok=True)
+ _STATE_FILE.write_text(json.dumps(state, indent=2))
+
+
+def _update_state(**kwargs) -> dict:
+ state = _load_state()
+ state.update(kwargs)
+ _save_state(state)
+ return state
+
+
+def _host_consented() -> bool:
+ return _HOST_CONSENT_FILE.exists()
+
+
+# ── output helpers ────────────────────────────────────────────────────────────
+
+
+def _ok(msg: str) -> int:
+ print(f"✅ {msg}")
+ return 0
+
+
+def _fail(msg: str) -> int:
+ print(f"❌ {msg}", file=sys.stderr)
+ return 1
+
+
+# ── provider / connection helpers ─────────────────────────────────────────────
+
+
+async def _get_api_url(provider_type: str, name: str) -> str:
+ """Resolve the computer-server API URL for the target VM."""
+ from computer.providers.base import VMProviderType
+ from computer.providers.factory import VMProviderFactory
+
+ if provider_type == "host":
+ return "http://localhost:8000"
+
+ if provider_type in ("cloud", "cloudv2"):
+ from cua_cli.auth.store import get_api_key
+
+ api_key = get_api_key()
+ if not api_key:
+ raise ValueError("Not authenticated. Run 'cua auth login' first")
+ ptype = VMProviderType.CLOUDV2 if provider_type == "cloudv2" else VMProviderType.CLOUD
+ provider = VMProviderFactory.create_provider(ptype, api_key=api_key)
+ async with provider:
+ vm = await provider.get_vm(name)
+ if not vm:
+ raise ValueError(f"VM not found: {name}")
+ url = vm.get("api_url") or vm.get("server_url")
+ if not url:
+ raise ValueError(f"VM '{name}' has no API URL (is it running?)")
+ return url
+
+ if provider_type == "docker":
+ provider = VMProviderFactory.create_provider(VMProviderType.DOCKER)
+ async with provider:
+ vm = await provider.get_vm(name)
+ if vm.get("status") == "not_found":
+ raise ValueError(f"Docker container not found: {name}")
+ ports = vm.get("ports", {})
+ for key in ("8000/tcp", "5000/tcp"):
+ if key in ports:
+ return f"http://localhost:{ports[key]}"
+ return "http://localhost:8000"
+
+ if provider_type in ("lume", "lumier"):
+ ptype = VMProviderType.LUME if provider_type == "lume" else VMProviderType.LUMIER
+ provider = VMProviderFactory.create_provider(ptype)
+ async with provider:
+ vm = await provider.get_vm(name)
+ ip = vm.get("ip_address") or "localhost"
+ return f"http://{ip}:8000"
+
+ if provider_type == "winsandbox":
+ return "http://localhost:8000"
+
+ raise ValueError(f"Unknown provider: {provider_type}")
+
+
+async def _host_dispatch(command: str, params: dict) -> dict:
+ """Dispatch a computer-server command to the local host via cua_auto."""
+ try:
+ import cua_auto.keyboard as _kb
+ import cua_auto.mouse as _mouse
+ import cua_auto.screen as _screen
+ import cua_auto.shell as _shell
+ import cua_auto.window as _win
+ except ImportError as e:
+ return {
+ "success": False,
+ "error": f"cua-auto not installed: {e}. Run: pip install cua-auto",
+ }
+
+ try:
+ if command == "screenshot":
+ b64 = _screen.screenshot_b64()
+ return {"success": True, "image_data": b64}
+
+ elif command == "get_screen_size":
+ w, h = _screen.screen_size()
+ return {"success": True, "size": {"width": w, "height": h}}
+
+ elif command == "get_cursor_position":
+ x, y = _screen.cursor_position()
+ return {"success": True, "position": {"x": x, "y": y}}
+
+ elif command == "get_current_window_id":
+ handle = _win.get_active_window_handle()
+ if not handle:
+ return {"success": False, "error": "No active window"}
+ return {"success": True, "window_id": handle}
+
+ elif command == "get_window_name":
+ title = _win.get_window_name(params.get("window_id", ""))
+ if title is None:
+ return {"success": False, "error": "Window not found"}
+ return {"success": True, "name": title}
+
+ elif command == "get_application_windows":
+ handles = _win.get_windows_with_title(params.get("app", ""))
+ return {"success": True, "windows": handles}
+
+ elif command == "get_window_size":
+ result = _win.get_window_size(params.get("window_id", ""))
+ if result is None:
+ return {"success": False, "error": "Window not found"}
+ return {"success": True, "size": [result[0], result[1]]}
+
+ elif command == "get_window_position":
+ result = _win.get_window_position(params.get("window_id", ""))
+ if result is None:
+ return {"success": False, "error": "Window not found"}
+ return {"success": True, "position": [result[0], result[1]]}
+
+ elif command == "left_click":
+ _mouse.click(int(params["x"]), int(params["y"]))
+ return {"success": True}
+
+ elif command == "right_click":
+ _mouse.right_click(int(params["x"]), int(params["y"]))
+ return {"success": True}
+
+ elif command == "middle_click":
+ _mouse.click(int(params["x"]), int(params["y"]), "middle")
+ return {"success": True}
+
+ elif command == "double_click":
+ _mouse.double_click(int(params["x"]), int(params["y"]))
+ return {"success": True}
+
+ elif command == "move_cursor":
+ _mouse.move_to(int(params["x"]), int(params["y"]))
+ return {"success": True}
+
+ elif command == "mouse_down":
+ x = params.get("x")
+ y = params.get("y")
+ _mouse.mouse_down(
+ int(x) if x is not None else None,
+ int(y) if y is not None else None,
+ params.get("button", "left"),
+ )
+ return {"success": True}
+
+ elif command == "mouse_up":
+ x = params.get("x")
+ y = params.get("y")
+ _mouse.mouse_up(
+ int(x) if x is not None else None,
+ int(y) if y is not None else None,
+ params.get("button", "left"),
+ )
+ return {"success": True}
+
+ elif command == "scroll_direction":
+ direction = params.get("direction", "down")
+ clicks = int(params.get("clicks", 3))
+ if direction == "up":
+ _mouse.scroll_up(clicks)
+ elif direction == "down":
+ _mouse.scroll_down(clicks)
+ elif direction == "left":
+ _mouse.scroll_left(clicks)
+ elif direction == "right":
+ _mouse.scroll_right(clicks)
+ return {"success": True}
+
+ elif command == "drag_to":
+ _mouse.drag(
+ int(params["start_x"]),
+ int(params["start_y"]),
+ int(params["end_x"]),
+ int(params["end_y"]),
+ )
+ return {"success": True}
+
+ elif command == "type_text":
+ _kb.type_text(params.get("text", ""))
+ return {"success": True}
+
+ elif command == "press_key":
+ _kb.press_key(params.get("key", ""))
+ return {"success": True}
+
+ elif command == "key_down":
+ _kb.key_down(params.get("key", ""))
+ return {"success": True}
+
+ elif command == "key_up":
+ _kb.key_up(params.get("key", ""))
+ return {"success": True}
+
+ elif command == "hotkey":
+ keys = params.get("keys", [])
+ if isinstance(keys, str):
+ keys = [k.strip() for k in keys.replace("-", "+").split("+") if k.strip()]
+ _kb.hotkey(keys)
+ return {"success": True}
+
+ elif command == "run_command":
+ result = _shell.run(params.get("command", ""))
+ return {
+ "success": True,
+ "stdout": result.stdout,
+ "stderr": result.stderr,
+ "returncode": result.returncode,
+ }
+
+ elif command == "open":
+ _win.open(params.get("path") or params.get("target", ""))
+ return {"success": True}
+
+ elif command == "launch":
+ pid = _win.launch(params.get("app", ""), params.get("args"))
+ return {"success": True, "pid": pid}
+
+ elif command == "activate_window":
+ ok = _win.activate_window(params.get("window_id", ""))
+ return {"success": bool(ok)}
+
+ elif command == "minimize_window":
+ ok = _win.minimize_window(params.get("window_id", ""))
+ return {"success": bool(ok)}
+
+ elif command == "maximize_window":
+ ok = _win.maximize_window(params.get("window_id", ""))
+ return {"success": bool(ok)}
+
+ elif command == "close_window":
+ ok = _win.close_window(params.get("window_id", ""))
+ return {"success": bool(ok)}
+
+ elif command == "set_window_size":
+ ok = _win.set_window_size(
+ params["window_id"], int(params["width"]), int(params["height"])
+ )
+ return {"success": bool(ok)}
+
+ elif command == "set_window_position":
+ ok = _win.set_window_position(params["window_id"], int(params["x"]), int(params["y"]))
+ return {"success": bool(ok)}
+
+ elif command == "deactivate_window":
+ return {"success": False, "error": "deactivate_window not supported on host"}
+
+ else:
+ return {"success": False, "error": f"Unknown host command: {command}"}
+
+ except Exception as e:
+ return {"success": False, "error": str(e)}
+
+
+async def _send(provider_type: str, name: str, command: str, params: dict) -> dict:
+ """Send a command to the computer-server and return the parsed response."""
+ if provider_type == "host":
+ return await _host_dispatch(command, params)
+
+ import aiohttp
+
+ api_url = await _get_api_url(provider_type, name)
+ headers = {"Content-Type": "application/json"}
+
+ if provider_type in ("cloud", "cloudv2"):
+ from cua_cli.auth.store import get_api_key
+
+ api_key = get_api_key()
+ if api_key:
+ headers["X-API-Key"] = api_key
+ if name:
+ headers["X-Container-Name"] = name
+
+ async with aiohttp.ClientSession() as session:
+ async with session.post(
+ f"{api_url}/cmd",
+ json={"command": command, "params": params},
+ headers=headers,
+ timeout=aiohttp.ClientTimeout(total=30),
+ ) as resp:
+ text = await resp.text()
+ for line in text.splitlines():
+ if line.startswith("data: "):
+ return json.loads(line[6:])
+ try:
+ return json.loads(text)
+ except Exception:
+ return {"success": False, "error": f"Unexpected response: {text[:200]}"}
+
+
+# ── zoom / screenshot helpers ─────────────────────────────────────────────────
+
+
+async def _resolve_zoom_bbox_by_id(provider_type: str, name: str, window_id: str) -> dict | None:
+ """Get the bounding box of a window by its native handle/id."""
+ pos_r = await _send(provider_type, name, "get_window_position", {"window_id": window_id})
+ size_r = await _send(provider_type, name, "get_window_size", {"window_id": window_id})
+
+ pos = pos_r.get("position") or pos_r.get("data")
+ size = size_r.get("size") or size_r.get("data")
+ if not pos or not size:
+ return None
+
+ if isinstance(pos, (list, tuple)):
+ x, y = int(pos[0]), int(pos[1])
+ else:
+ x, y = int(pos.get("x", 0)), int(pos.get("y", 0))
+
+ if isinstance(size, (list, tuple)):
+ w, h = int(size[0]), int(size[1])
+ else:
+ w, h = int(size.get("width", 0)), int(size.get("height", 0))
+
+ return {"x": x, "y": y, "width": w, "height": h}
+
+
+async def _resolve_zoom_bbox(provider_type: str, name: str, window_name: str) -> dict | None:
+ """Get the bounding box of a window by app/window name.
+
+ Filters out internal helper windows (e.g. 'Chrome Legacy Window') so the
+ correct top-level window is always selected.
+ """
+ wins_r = await _send(provider_type, name, "get_application_windows", {"app": window_name})
+ windows = wins_r.get("windows") or wins_r.get("data") or []
+ if not windows:
+ return None
+
+ # Pick the first window whose title is not an internal helper
+ window_id = None
+ for wid in windows:
+ try:
+ name_r = await _send(provider_type, name, "get_window_name", {"window_id": wid})
+ title = name_r.get("name") or ""
+ if title not in _SKIP_WINDOW_TITLES:
+ window_id = wid
+ break
+ except Exception:
+ pass
+ if window_id is None:
+ window_id = windows[0]
+
+ return await _resolve_zoom_bbox_by_id(provider_type, name, window_id)
+
+
+async def _take_screenshot_data(
+ provider_type: str, name: str, state: dict
+) -> tuple[bytes, float, dict | None]:
+ """Take screenshot, apply zoom crop and max-length scaling.
+
+ Returns (image_bytes, display_scale, zoom_bbox).
+ display_scale < 1.0 means the image was shrunk; bbox is None if not zoomed.
+ """
+ from PIL import Image
+
+ result = await _send(provider_type, name, "screenshot", {})
+ img_b64 = result.get("image_data") or result.get("data")
+ if not img_b64:
+ raise RuntimeError(result.get("error", "no image data returned"))
+ img = Image.open(io.BytesIO(base64.b64decode(img_b64)))
+
+ zoom_window = state.get("zoom_window")
+ zoom_window_id = state.get("zoom_window_id")
+ bbox: dict | None = None
+
+ if zoom_window:
+ try:
+ if zoom_window_id:
+ bbox = await _resolve_zoom_bbox_by_id(provider_type, name, str(zoom_window_id))
+ if not bbox:
+ bbox = await _resolve_zoom_bbox(provider_type, name, zoom_window)
+ if bbox:
+ img = img.crop(
+ (
+ bbox["x"],
+ bbox["y"],
+ bbox["x"] + bbox["width"],
+ bbox["y"] + bbox["height"],
+ )
+ )
+ except Exception:
+ bbox = None # fall back to full screen
+
+ w, h = img.size
+ scale = 1.0
+ if max(w, h) > MAX_LENGTH:
+ scale = MAX_LENGTH / max(w, h)
+ img = img.resize((int(w * scale), int(h * scale)), Image.LANCZOS)
+
+ buf = io.BytesIO()
+ img.save(buf, format="PNG")
+ return buf.getvalue(), scale, bbox
+
+
+def _coords(x: int, y: int, state: dict) -> tuple[int, int]:
+ """Translate image-space (x, y) to screen-space coordinates.
+
+ Accounts for:
+ - display_scale: image was scaled for output (divide to get original pixels)
+ - zoom_bbox: image was cropped, add offset to get screen coords
+ """
+ zoom_scale: float = state.get("zoom_scale", 1.0) or 1.0
+ zoom_bbox: dict | None = state.get("zoom_bbox")
+
+ sx = round(x / zoom_scale)
+ sy = round(y / zoom_scale)
+
+ if zoom_bbox:
+ sx += zoom_bbox["x"]
+ sy += zoom_bbox["y"]
+
+ return sx, sy
+
+
+async def _maybe_focus_zoom(provider_type: str, name: str, state: dict) -> None:
+ """If a zoom window is tracked, bring it to focus before performing an action."""
+ window_id = state.get("zoom_window_id")
+ if window_id:
+ try:
+ await _send(provider_type, name, "activate_window", {"window_id": window_id})
+ except Exception:
+ pass
+
+
+async def _print_context(provider_type: str, name: str, state: dict | None = None) -> None:
+ """Print the VM + zoom context line after a command."""
+ vm_label = name if name else provider_type
+ if state is None:
+ state = _load_state()
+ zoom_window = state.get("zoom_window")
+ zoom_window_id = state.get("zoom_window_id")
+ if zoom_window:
+ zoom_info = (
+ f"zoom: {zoom_window} ({zoom_window_id})" if zoom_window_id else f"zoom: {zoom_window}"
+ )
+ else:
+ zoom_info = "zoom: off"
+ print(f"💻 {vm_label}\t🔍 {zoom_info}")
+
+
+def _require_target() -> dict | None:
+ state = _load_state()
+ if not state.get("provider"):
+ _fail("No VM selected. Run: cua do switch <provider> [name]")
+ return None
+ return state
+
+
+# ── subcommand handlers ───────────────────────────────────────────────────────
+
+
+def _cmd_switch(args: argparse.Namespace) -> int:
+ provider = args.provider.lower()
+ old_state = _load_state()
+ had_zoom = bool(old_state.get("zoom_window"))
+
+ if provider == "host":
+ if not _host_consented():
+ print(
+ "❌ Warning: you are about to allow an AI to control your host PC directly.\n"
+ " This grants full keyboard, mouse, and screen access to your local desktop.\n"
+ " To continue, please run: cua do-host-consent",
+ file=sys.stderr,
+ )
+ return 1
+ _save_state(
+ {
+ "provider": "host",
+ "name": "",
+ "zoom_window": None,
+ "zoom_window_id": None,
+ "zoom_bbox": None,
+ "zoom_scale": 1.0,
+ }
+ )
+ msg = "Switched to host (local PC)"
+ if had_zoom:
+ msg += " — zoom reset"
+ return _ok(msg)
+
+ if provider not in PROVIDERS:
+ return _fail(f"Unknown provider '{provider}'. Choose from: {', '.join(PROVIDERS)}")
+
+ name = args.name or ""
+ _save_state(
+ {
+ "provider": provider,
+ "name": name,
+ "zoom_window": None,
+ "zoom_window_id": None,
+ "zoom_bbox": None,
+ "zoom_scale": 1.0,
+ }
+ )
+ label = f"{provider}/{name}" if name else provider
+ msg = f"Switched to {label}"
+ if had_zoom:
+ msg += " — zoom reset"
+ return _ok(msg)
+
+
+def _cmd_status(args: argparse.Namespace) -> int:
+ state = _load_state()
+ provider = state.get("provider")
+ if not provider:
+ return _fail("No VM selected. Run: cua do switch <provider> [name]")
+ name = state.get("name", "")
+ label = f"{provider}/{name}" if name else provider
+ zoom = state.get("zoom_window")
+ zoom_info = f" [zoomed to '{zoom}']" if zoom else ""
+ return _ok(f"Current target: {label}{zoom_info}")
+
+
+def _cmd_ls(args: argparse.Namespace) -> int:
+ from cua_cli.utils.async_utils import run_async
+
+ explicit_provider = getattr(args, "provider", None)
+
+ async def _list_one(ptype: str) -> int:
+ from computer.providers.base import VMProviderType
+ from computer.providers.factory import VMProviderFactory
+
+ if ptype == "host":
+ print(" host [local]")
+ return 0
+
+ kwargs: dict[str, Any] = {}
+ if ptype in ("cloud", "cloudv2"):
+ from cua_cli.auth.store import get_api_key
+
+ api_key = get_api_key()
+ if not api_key:
+ return _fail("Not authenticated. Run 'cua auth login' first")
+ kwargs["api_key"] = api_key
+
+ try:
+ enum_type = VMProviderType(ptype)
+ except ValueError:
+ return _fail(f"Unknown provider: {ptype}")
+
+ try:
+ provider = VMProviderFactory.create_provider(enum_type, **kwargs)
+ async with provider:
+ vms = await provider.list_vms()
+ except Exception as e:
+ return _fail(str(e))
+
+ if not vms:
+ print(f"No VMs found for provider '{ptype}'")
+ return 0
+
+ for vm in vms:
+ print(f" {vm.get('name', '?')} [{vm.get('status', '?')}]")
+ return 0
+
+ async def _list_all() -> int:
+ from computer.providers.base import VMProviderType
+ from computer.providers.factory import VMProviderFactory
+ from cua_cli.auth.store import get_api_key
+
+ print(" host [local]")
+
+ api_key = get_api_key()
+ if api_key:
+ try:
+ provider = VMProviderFactory.create_provider(
+ VMProviderType.CLOUDV2, api_key=api_key
+ )
+ async with provider:
+ vms = await provider.list_vms()
+ for vm in vms:
+ print(f" {vm.get('name', '?')} [{vm.get('status', '?')}] cloudv2")
+ except Exception:
+ pass
+
+ return 0
+
+ if explicit_provider:
+ return run_async(_list_one(explicit_provider.lower()))
+ else:
+ return run_async(_list_all())
+
+
+def _cmd_zoom(args: argparse.Namespace) -> int:
+ from cua_cli.utils.async_utils import run_async
+
+ t = _require_target()
+ if not t:
+ return 1
+ window_name = args.window_name
+
+ async def _run() -> int:
+ state = _load_state()
+ p, n = state["provider"], state.get("name", "")
+ try:
+ wins_r = await _send(p, n, "get_application_windows", {"app": window_name})
+ windows = wins_r.get("windows") or wins_r.get("data") or []
+ if not windows:
+ return _fail(f"No windows found matching '{window_name}'")
+ # Fetch titles and filter out internal helper windows
+ candidates = []
+ for wid in windows:
+ name_r = await _send(p, n, "get_window_name", {"window_id": wid})
+ title = name_r.get("name") or str(wid)
+ if title not in _SKIP_WINDOW_TITLES:
+ candidates.append((wid, title))
+ if not candidates:
+ return _fail(f"No windows found matching '{window_name}'")
+ if len(candidates) > 1:
+ labels = [f"{wid} ({title})" for wid, title in candidates]
+ return _fail(
+ f"Multiple windows matched '{window_name}' — be more specific. "
+ f"Found: {', '.join(labels)}"
+ )
+ wid, matched_title = candidates[0]
+ except Exception:
+ wid, matched_title = None, window_name # resolution failed, still set zoom
+
+ _update_state(zoom_window=window_name, zoom_window_id=wid, zoom_bbox=None, zoom_scale=1.0)
+ id_suffix = f" ({wid})" if wid else ""
+ return _ok(f"Zoomed to '{matched_title}'{id_suffix}")
+
+ return run_async(_run())
+
+
+def _cmd_unzoom(args: argparse.Namespace) -> int:
+ _update_state(zoom_window=None, zoom_window_id=None, zoom_bbox=None, zoom_scale=1.0)
+ return _ok("Unzoomed — full screen")
+
+
+def _cmd_screenshot(args: argparse.Namespace) -> int:
+ from cua_cli.utils.async_utils import run_async
+
+ t = _require_target()
+ if not t:
+ return 1
+
+ async def _run() -> int:
+ state = _load_state()
+ try:
+ img_bytes, scale, bbox = await _take_screenshot_data(
+ state["provider"], state.get("name", ""), state
+ )
+ except Exception as e:
+ await _print_context(state["provider"], state.get("name", ""), state)
+ return _fail(str(e))
+
+ # Persist zoom state for coordinate translation
+ _update_state(zoom_scale=scale, zoom_bbox=bbox)
+
+ save_path = getattr(args, "save", None)
+ if not save_path:
+ ts = datetime.now().strftime("%Y%m%d_%H%M%S")
+ save_path = f"{tempfile.gettempdir()}/cua_screenshot_{ts}.png"
+
+ with open(save_path, "wb") as f:
+ f.write(img_bytes)
+
+ rc = _ok(f"screenshot saved to {save_path}")
+ await _print_context(state["provider"], state.get("name", ""), state)
+ return rc
+
+ return run_async(_run())
+
+
+def _cmd_snapshot(args: argparse.Namespace) -> int:
+ from cua_cli.utils.async_utils import run_async
+
+ t = _require_target()
+ if not t:
+ return 1
+
+ extra = " ".join(args.instructions) if args.instructions else ""
+
+ async def _run() -> int:
+ state = _load_state()
+ try:
+ img_bytes, scale, bbox = await _take_screenshot_data(
+ state["provider"], state.get("name", ""), state
+ )
+ except Exception as e:
+ await _print_context(state["provider"], state.get("name", ""), state)
+ return _fail(str(e))
+
+ _update_state(zoom_scale=scale, zoom_bbox=bbox)
+
+ ts = datetime.now().strftime("%Y%m%d_%H%M%S")
+ save_path = f"{tempfile.gettempdir()}/cua_snapshot_{ts}.png"
+ with open(save_path, "wb") as f:
+ f.write(img_bytes)
+
+ api_key = os.environ.get("ANTHROPIC_API_KEY")
+ if not api_key:
+ await _print_context(state["provider"], state.get("name", ""), state)
+ return _fail("ANTHROPIC_API_KEY not set")
+
+ try:
+ import anthropic as _anthropic
+
+ client = _anthropic.Anthropic(api_key=api_key)
+ prompt = (
+ "You are analyzing a screenshot for an AI agent.\n"
+ "1. Write a 1-2 sentence summary of what is currently on screen.\n"
+ "2. List every interactive element visible (buttons, links, inputs, "
+ "menus, checkboxes, dropdowns, etc.) with its center coordinates "
+ "in image pixels (origin = top-left). Be precise.\n\n"
+ "Respond in this exact JSON format:\n"
+ '{"summary": "...", "elements": [{"name": "...", "type": "...", "x": N, "y": N}, ...]}\n'
+ )
+ if extra:
+ prompt += f"\nAdditional instructions: {extra}"
+
+ img_b64 = base64.b64encode(img_bytes).decode()
+ response = client.messages.create(
+ model="claude-haiku-4-5-20251001",
+ max_tokens=1024,
+ messages=[
+ {
+ "role": "user",
+ "content": [
+ {
+ "type": "image",
+ "source": {
+ "type": "base64",
+ "media_type": "image/png",
+ "data": img_b64,
+ },
+ },
+ {"type": "text", "text": prompt},
+ ],
+ }
+ ],
+ )
+
+ raw = response.content[0].text.strip()
+ # Try to parse JSON; fall back to raw text
+ try:
+ parsed = json.loads(raw)
+ summary = parsed.get("summary", "")
+ elements = parsed.get("elements", [])
+ print(f"✅ snapshot — {save_path}")
+ print()
+ print(summary)
+ if elements:
+ print()
+ print("Interactive elements:")
+ for el in elements:
+ print(
+ f" • {el.get('name','?')} [{el.get('type','?')}] ({el.get('x','?')}, {el.get('y','?')})"
+ )
+ except json.JSONDecodeError:
+ print(f"✅ snapshot — {save_path}")
+ print()
+ print(raw)
+
+ except ImportError:
+ await _print_context(state["provider"], state.get("name", ""), state)
+ return _fail("'anthropic' package not installed. Run: pip install anthropic")
+ except Exception as e:
+ await _print_context(state["provider"], state.get("name", ""), state)
+ return _fail(f"AI analysis failed: {e}")
+
+ await _print_context(state["provider"], state.get("name", ""), state)
+ return 0
+
+ return run_async(_run())
+
+
+def _cmd_click(args: argparse.Namespace) -> int:
+ from cua_cli.utils.async_utils import run_async
+
+ t = _require_target()
+ if not t:
+ return 1
+
+ button = getattr(args, "button", "left") or "left"
+ cmd_map = {"left": "left_click", "right": "right_click", "middle": "middle_click"}
+ cmd = cmd_map.get(button, "left_click")
+
+ async def _run() -> int:
+ state = _load_state()
+ p, n = state["provider"], state.get("name", "")
+ await _maybe_focus_zoom(p, n, state)
+ sx, sy = _coords(args.x, args.y, state)
+ try:
+ result = await _send(p, n, cmd, {"x": sx, "y": sy})
+ except Exception as e:
+ await _print_context(p, n, state)
+ return _fail(str(e))
+ if not result.get("success", True):
+ await _print_context(p, n, state)
+ return _fail(result.get("error", "click failed"))
+ rc = _ok(f"clicked ({args.x}, {args.y}) [{button}]")
+ await _print_context(p, n, state)
+ return rc
+
+ return run_async(_run())
+
+
+def _cmd_dclick(args: argparse.Namespace) -> int:
+ from cua_cli.utils.async_utils import run_async
+
+ t = _require_target()
+ if not t:
+ return 1
+
+ async def _run() -> int:
+ state = _load_state()
+ p, n = state["provider"], state.get("name", "")
+ await _maybe_focus_zoom(p, n, state)
+ sx, sy = _coords(args.x, args.y, state)
+ try:
+ result = await _send(p, n, "double_click", {"x": sx, "y": sy})
+ except Exception as e:
+ await _print_context(p, n, state)
+ return _fail(str(e))
+ if not result.get("success", True):
+ await _print_context(p, n, state)
+ return _fail(result.get("error", "double-click failed"))
+ rc = _ok(f"double-clicked ({args.x}, {args.y})")
+ await _print_context(p, n, state)
+ return rc
+
+ return run_async(_run())
+
+
+def _cmd_move(args: argparse.Namespace) -> int:
+ from cua_cli.utils.async_utils import run_async
+
+ t = _require_target()
+ if not t:
+ return 1
+
+ async def _run() -> int:
+ state = _load_state()
+ p, n = state["provider"], state.get("name", "")
+ await _maybe_focus_zoom(p, n, state)
+ sx, sy = _coords(args.x, args.y, state)
+ try:
+ result = await _send(p, n, "move_cursor", {"x": sx, "y": sy})
+ except Exception as e:
+ await _print_context(p, n, state)
+ return _fail(str(e))
+ if not result.get("success", True):
+ await _print_context(p, n, state)
+ return _fail(result.get("error", "move failed"))
+ rc = _ok(f"cursor moved to ({args.x}, {args.y})")
+ await _print_context(p, n, state)
+ return rc
+
+ return run_async(_run())
+
+
+def _cmd_type(args: argparse.Namespace) -> int:
+ from cua_cli.utils.async_utils import run_async
+
+ t = _require_target()
+ if not t:
+ return 1
+
+ async def _run() -> int:
+ state = _load_state()
+ p, n = state["provider"], state.get("name", "")
+ await _maybe_focus_zoom(p, n, state)
+ try:
+ result = await _send(p, n, "type_text", {"text": args.text})
+ except Exception as e:
+ await _print_context(p, n, state)
+ return _fail(str(e))
+ if not result.get("success", True):
+ await _print_context(p, n, state)
+ return _fail(result.get("error", "type failed"))
+ preview = args.text[:40] + ("…" if len(args.text) > 40 else "")
+ rc = _ok(f"typed: {preview!r}")
+ await _print_context(p, n, state)
+ return rc
+
+ return run_async(_run())
+
+
+def _cmd_key(args: argparse.Namespace) -> int:
+ from cua_cli.utils.async_utils import run_async
+
+ t = _require_target()
+ if not t:
+ return 1
+
+ async def _run() -> int:
+ state = _load_state()
+ p, n = state["provider"], state.get("name", "")
+ await _maybe_focus_zoom(p, n, state)
+ try:
+ result = await _send(p, n, "press_key", {"key": args.key})
+ except Exception as e:
+ await _print_context(p, n, state)
+ return _fail(str(e))
+ if not result.get("success", True):
+ await _print_context(p, n, state)
+ return _fail(result.get("error", "key press failed"))
+ rc = _ok(f"pressed key: {args.key}")
+ await _print_context(p, n, state)
+ return rc
+
+ return run_async(_run())
+
+
+def _cmd_hotkey(args: argparse.Namespace) -> int:
+ from cua_cli.utils.async_utils import run_async
+
+ t = _require_target()
+ if not t:
+ return 1
+
+ keys = [k.strip() for k in args.keys.replace("-", "+").split("+") if k.strip()]
+
+ async def _run() -> int:
+ state = _load_state()
+ p, n = state["provider"], state.get("name", "")
+ await _maybe_focus_zoom(p, n, state)
+ try:
+ result = await _send(p, n, "hotkey", {"keys": keys})
+ except Exception as e:
+ await _print_context(p, n, state)
+ return _fail(str(e))
+ if not result.get("success", True):
+ await _print_context(p, n, state)
+ return _fail(result.get("error", "hotkey failed"))
+ rc = _ok(f"hotkey: {'+'.join(keys)}")
+ await _print_context(p, n, state)
+ return rc
+
+ return run_async(_run())
+
+
+def _cmd_scroll(args: argparse.Namespace) -> int:
+ from cua_cli.utils.async_utils import run_async
+
+ t = _require_target()
+ if not t:
+ return 1
+
+ async def _run() -> int:
+ state = _load_state()
+ p, n = state["provider"], state.get("name", "")
+ await _maybe_focus_zoom(p, n, state)
+ try:
+ result = await _send(
+ p, n, "scroll_direction", {"direction": args.direction, "clicks": args.amount}
+ )
+ except Exception as e:
+ await _print_context(p, n, state)
+ return _fail(str(e))
+ if not result.get("success", True):
+ await _print_context(p, n, state)
+ return _fail(result.get("error", "scroll failed"))
+ rc = _ok(f"scrolled {args.direction} {args.amount}x")
+ await _print_context(p, n, state)
+ return rc
+
+ return run_async(_run())
+
+
+def _cmd_drag(args: argparse.Namespace) -> int:
+ from cua_cli.utils.async_utils import run_async
+
+ t = _require_target()
+ if not t:
+ return 1
+
+ async def _run() -> int:
+ state = _load_state()
+ p, n = state["provider"], state.get("name", "")
+ await _maybe_focus_zoom(p, n, state)
+ sx1, sy1 = _coords(args.x1, args.y1, state)
+ sx2, sy2 = _coords(args.x2, args.y2, state)
+ try:
+ result = await _send(
+ p,
+ n,
+ "drag_to",
+ {"start_x": sx1, "start_y": sy1, "end_x": sx2, "end_y": sy2},
+ )
+ except Exception as e:
+ await _print_context(p, n, state)
+ return _fail(str(e))
+ if not result.get("success", True):
+ await _print_context(p, n, state)
+ return _fail(result.get("error", "drag failed"))
+ rc = _ok(f"dragged ({args.x1},{args.y1}) → ({args.x2},{args.y2})")
+ await _print_context(p, n, state)
+ return rc
+
+ return run_async(_run())
+
+
+def _cmd_shell(args: argparse.Namespace) -> int:
+ from cua_cli.utils.async_utils import run_async
+
+ t = _require_target()
+ if not t:
+ return 1
+
+ command = " ".join(args.command)
+ if not command.strip():
+ return _fail("No command provided")
+
+ async def _run() -> int:
+ state = _load_state()
+ try:
+ result = await _send(
+ state["provider"], state.get("name", ""), "run_command", {"command": command}
+ )
+ except Exception as e:
+ await _print_context(state["provider"], state.get("name", ""), state)
+ return _fail(str(e))
+ if not result.get("success", True):
+ await _print_context(state["provider"], state.get("name", ""), state)
+ return _fail(result.get("error", "shell command failed"))
+ stdout = result.get("stdout", "").strip()
+ rc_code = result.get("returncode", 0)
+ if rc_code != 0:
+ stderr = result.get("stderr", "").strip()
+ await _print_context(state["provider"], state.get("name", ""), state)
+ return _fail(f"exit {rc_code}: {stderr or stdout}")
+ preview = (stdout[:80] + "…") if len(stdout) > 80 else stdout
+ rc = _ok(preview if preview else "done")
+ await _print_context(state["provider"], state.get("name", ""), state)
+ return rc
+
+ return run_async(_run())
+
+
+def _cmd_open(args: argparse.Namespace) -> int:
+ from cua_cli.utils.async_utils import run_async
+
+ t = _require_target()
+ if not t:
+ return 1
+
+ async def _run() -> int:
+ state = _load_state()
+ try:
+ result = await _send(
+ state["provider"], state.get("name", ""), "open", {"path": args.path}
+ )
+ except Exception as e:
+ await _print_context(state["provider"], state.get("name", ""), state)
+ return _fail(str(e))
+ if not result.get("success", True):
+ await _print_context(state["provider"], state.get("name", ""), state)
+ return _fail(result.get("error", "open failed"))
+ rc = _ok(f"opened: {args.path}")
+ await _print_context(state["provider"], state.get("name", ""), state)
+ return rc
+
+ return run_async(_run())
+
+
+# ── window subcommands ────────────────────────────────────────────────────────
+
+
+def _cmd_window(args: argparse.Namespace) -> int:
+ from cua_cli.utils.async_utils import run_async
+
+ t = _require_target()
+ if not t:
+ return 1
+
+ action = args.window_action
+
+ async def _run() -> int:
+ state = _load_state()
+ p, n = state["provider"], state.get("name", "")
+
+ if action == "ls":
+ app = getattr(args, "app", "") or ""
+ try:
+ result = await _send(p, n, "get_application_windows", {"app": app})
+ except Exception as e:
+ return _fail(str(e))
+ windows = result.get("windows") or result.get("data") or []
+ if not isinstance(windows, list):
+ print(result)
+ return 0
+ for wid in windows:
+ try:
+ name_r = await _send(p, n, "get_window_name", {"window_id": wid})
+ title = name_r.get("name") or "?"
+ if title in _SKIP_WINDOW_TITLES:
+ continue
+ pos_r = await _send(p, n, "get_window_position", {"window_id": wid})
+ size_r = await _send(p, n, "get_window_size", {"window_id": wid})
+ pos = pos_r.get("position") or pos_r.get("data")
+ size = size_r.get("size") or size_r.get("data")
+ if pos and size:
+ if isinstance(pos, (list, tuple)):
+ x, y = int(pos[0]), int(pos[1])
+ else:
+ x, y = int(pos.get("x", 0)), int(pos.get("y", 0))
+ if isinstance(size, (list, tuple)):
+ w, h = int(size[0]), int(size[1])
+ else:
+ w, h = int(size.get("width", 0)), int(size.get("height", 0))
+ bbox = f"{x},{y},{w},{h}"
+ else:
+ bbox = "?"
+ print(f" {wid} {title} [{bbox}]")
+ except Exception:
+ print(f" {wid}")
+ return 0
+
+ if action == "unfocus":
+ try:
+ result = await _send(p, n, "deactivate_window", {})
+ except Exception:
+ result = {"success": False}
+ if not result.get("success", True):
+ try:
+ result = await _send(p, n, "press_key", {"key": "escape"})
+ except Exception as e:
+ await _print_context(p, n)
+ return _fail(str(e))
+ rc = _ok("unfocused window")
+ await _print_context(p, n)
+ return rc
+
+ # Commands requiring window_id
+ wid = args.window_id
+
+ cmd_map = {
+ "focus": ("activate_window", {"window_id": wid}),
+ "activate": ("activate_window", {"window_id": wid}),
+ "minimize": ("minimize_window", {"window_id": wid}),
+ "maximize": ("maximize_window", {"window_id": wid}),
+ "close": ("close_window", {"window_id": wid}),
+ }
+
+ if action in cmd_map:
+ cmd, params = cmd_map[action]
+ try:
+ result = await _send(p, n, cmd, params)
+ except Exception as e:
+ await _print_context(p, n)
+ return _fail(str(e))
+ if not result.get("success", True):
+ await _print_context(p, n)
+ return _fail(result.get("error", f"{action} failed"))
+ rc = _ok(f"{action}d window {wid}")
+ await _print_context(p, n)
+ return rc
+
+ if action == "resize":
+ try:
+ result = await _send(
+ p,
+ n,
+ "set_window_size",
+ {"window_id": wid, "width": args.width, "height": args.height},
+ )
+ except Exception as e:
+ await _print_context(p, n)
+ return _fail(str(e))
+ if not result.get("success", True):
+ await _print_context(p, n)
+ return _fail(result.get("error", "resize failed"))
+ rc = _ok(f"resized window {wid} to {args.width}x{args.height}")
+ await _print_context(p, n)
+ return rc
+
+ if action == "move":
+ try:
+ result = await _send(
+ p,
+ n,
+ "set_window_position",
+ {"window_id": wid, "x": args.x, "y": args.y},
+ )
+ except Exception as e:
+ await _print_context(p, n)
+ return _fail(str(e))
+ if not result.get("success", True):
+ await _print_context(p, n)
+ return _fail(result.get("error", "move failed"))
+ rc = _ok(f"moved window {wid} to ({args.x},{args.y})")
+ await _print_context(p, n)
+ return rc
+
+ if action == "info":
+ try:
+ size_r = await _send(p, n, "get_window_size", {"window_id": wid})
+ pos_r = await _send(p, n, "get_window_position", {"window_id": wid})
+ name_r = await _send(p, n, "get_window_name", {"window_id": wid})
+ except Exception as e:
+ return _fail(str(e))
+ print(
+ json.dumps(
+ {
+ "id": wid,
+ "title": name_r.get("name") or name_r.get("data") or "",
+ "size": size_r.get("size") or size_r.get("data"),
+ "position": pos_r.get("position") or pos_r.get("data"),
+ },
+ indent=2,
+ )
+ )
+ return 0
+
+ return _fail(f"Unknown window action: {action}")
+
+ return run_async(_run())
+
+
+# ── host consent command (registered separately as cua do-host-consent) ───────
+
+
+def register_host_consent_parser(subparsers: argparse._SubParsersAction) -> None:
+ subparsers.add_parser(
+ "do-host-consent",
+ help="Grant consent to control the host PC with cua do",
+ description=(
+ "Grants permission for 'cua do switch host', allowing AI agents to control\n"
+ "your local keyboard, mouse, and screen via a local computer-server."
+ ),
+ )
+
+
+def execute_host_consent(args: argparse.Namespace) -> int:
+ _HOST_CONSENT_FILE.parent.mkdir(parents=True, exist_ok=True)
+ _HOST_CONSENT_FILE.write_text("consented")
+ _update_state(provider="host", name="")
+ return _ok("consented — target switched to host")
+
+
+# ── parser registration ───────────────────────────────────────────────────────
+
+
+def register_parser(subparsers: argparse._SubParsersAction) -> None:
+ p = subparsers.add_parser(
+ "do",
+ help="Send one-shot automation commands to a VM",
+ description=(
+ "Send automation commands to a target VM. "
+ "Output is a single line starting with ✅ or ❌.\n"
+ "A second line shows the current VM + focused window context.\n"
+ "Coordinates are in screenshot-image space; zoom/max-length translation is applied automatically."
+ ),
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ epilog="""
+Examples:
+ cua do switch docker my-container Select target VM
+ cua do switch host Control the local PC (requires consent)
+ cua do-host-consent Grant consent for host control
+ cua do ls List VMs for current provider
+ cua do ls docker
+ cua do status
+ cua do zoom "Chrome" Crop all screenshots to Chrome window
+ cua do unzoom
+ cua do screenshot
+ cua do screenshot --save /tmp/s.png
+ cua do snapshot Screenshot + AI summary (needs ANTHROPIC_API_KEY)
+ cua do snapshot "focus on the form"
+ cua do click 100 200
+ cua do click 100 200 right
+ cua do dclick 100 200
+ cua do move 500 300
+ cua do type "hello world"
+ cua do key enter
+ cua do hotkey cmd+c
+ cua do scroll down 5
+ cua do drag 100 200 400 500
+ cua do shell "ls -la"
+ cua do open https://example.com
+ cua do window ls
+ cua do window ls Terminal
+ cua do window focus <id>
+ cua do window unfocus
+ cua do window minimize/maximize/close <id>
+ cua do window resize <id> 1280 800
+ cua do window move <id> 0 0
+ cua do window info <id>
+""",
+ )
+
+ sub = p.add_subparsers(dest="do_action", metavar="action")
+ sub.required = True
+
+ # switch
+ sw = sub.add_parser("switch", help="Select target VM (use 'host' for local PC)")
+ sw.add_argument("provider", choices=PROVIDERS)
+ sw.add_argument("name", nargs="?", default="")
+
+ # status
+ sub.add_parser("status", help="Show current target and zoom state")
+
+ # ls
+ ls = sub.add_parser("ls", help="List VMs for a provider")
+ ls.add_argument("provider", nargs="?", choices=PROVIDERS)
+
+ # zoom / unzoom
+ zm = sub.add_parser("zoom", help="Crop screenshots + translate coords to a window")
+ zm.add_argument("window_name", help="App/window name to zoom to")
+ sub.add_parser("unzoom", help="Return to full-screen screenshots")
+
+ # screenshot
+ ss = sub.add_parser("screenshot", help="Take a screenshot (saved to temp dir)")
+ ss.add_argument("--save", "-s", metavar="PATH")
+
+ # snapshot
+ sn = sub.add_parser(
+ "snapshot",
+ help="Screenshot + AI screen summary and interactive elements (needs ANTHROPIC_API_KEY)",
+ )
+ sn.add_argument("instructions", nargs="*", help="Extra instructions for the AI")
+
+ # click
+ cl = sub.add_parser("click", help="Click at coordinates (image space)")
+ cl.add_argument("x", type=int)
+ cl.add_argument("y", type=int)
+ cl.add_argument("button", nargs="?", default="left", choices=["left", "right", "middle"])
+
+ # dclick
+ dc = sub.add_parser("dclick", help="Double-click at coordinates (image space)")
+ dc.add_argument("x", type=int)
+ dc.add_argument("y", type=int)
+
+ # move
+ mv = sub.add_parser("move", help="Move cursor (image space)")
+ mv.add_argument("x", type=int)
+ mv.add_argument("y", type=int)
+
+ # type
+ ty = sub.add_parser("type", help="Type text")
+ ty.add_argument("text")
+
+ # key
+ ky = sub.add_parser("key", help="Press a key (enter, escape, tab, …)")
+ ky.add_argument("key")
+
+ # hotkey
+ hk = sub.add_parser("hotkey", help="Keyboard shortcut (e.g. cmd+c, ctrl+shift+s)")
+ hk.add_argument("keys")
+
+ # scroll
+ sc = sub.add_parser("scroll", help="Scroll in a direction")
+ sc.add_argument("direction", choices=["up", "down", "left", "right"])
+ sc.add_argument("amount", nargs="?", type=int, default=3)
+
+ # drag
+ dr = sub.add_parser("drag", help="Drag (image space)")
+ dr.add_argument("x1", type=int)
+ dr.add_argument("y1", type=int)
+ dr.add_argument("x2", type=int)
+ dr.add_argument("y2", type=int)
+
+ # shell
+ sh = sub.add_parser("shell", help="Run a shell command in the VM")
+ sh.add_argument("command", nargs=argparse.REMAINDER)
+
+ # open
+ op = sub.add_parser("open", help="Open a file or URL")
+ op.add_argument("path")
+
+ # window
+ win = sub.add_parser("window", help="Window management")
+ win_sub = win.add_subparsers(dest="window_action", metavar="action")
+ win_sub.required = True
+
+ wls = win_sub.add_parser("ls", help="List windows")
+ wls.add_argument("app", nargs="?", default="")
+
+ win_sub.add_parser("unfocus", help="Remove focus from the current window")
+
+ for act in ("focus", "activate", "minimize", "maximize", "close"):
+ wp = win_sub.add_parser(act)
+ wp.add_argument("window_id")
+
+ wr = win_sub.add_parser("resize")
+ wr.add_argument("window_id")
+ wr.add_argument("width", type=int)
+ wr.add_argument("height", type=int)
+
+ wm = win_sub.add_parser("move")
+ wm.add_argument("window_id")
+ wm.add_argument("x", type=int)
+ wm.add_argument("y", type=int)
+
+ wi = win_sub.add_parser("info")
+ wi.add_argument("window_id")
+
+
+# ── dispatch ──────────────────────────────────────────────────────────────────
+
+
+def execute(args: argparse.Namespace) -> int:
+ dispatch = {
+ "switch": _cmd_switch,
+ "status": _cmd_status,
+ "ls": _cmd_ls,
+ "zoom": _cmd_zoom,
+ "unzoom": _cmd_unzoom,
+ "screenshot": _cmd_screenshot,
+ "snapshot": _cmd_snapshot,
+ "click": _cmd_click,
+ "dclick": _cmd_dclick,
+ "move": _cmd_move,
+ "type": _cmd_type,
+ "key": _cmd_key,
+ "hotkey": _cmd_hotkey,
+ "scroll": _cmd_scroll,
+ "drag": _cmd_drag,
+ "shell": _cmd_shell,
+ "open": _cmd_open,
+ "window": _cmd_window,
+ }
+ handler = dispatch.get(args.do_action)
+ if not handler:
+ return _fail(f"Unknown action: {args.do_action}")
+ return handler(args)
diff --git a/libs/python/cua-cli/cua_cli/commands/local_image.py b/libs/python/cua-cli/cua_cli/commands/local_image.py
index abb938b125..42997e459a 100644
--- a/libs/python/cua-cli/cua_cli/commands/local_image.py
+++ b/libs/python/cua-cli/cua_cli/commands/local_image.py
@@ -201,7 +201,9 @@ def cmd_local_list(args: argparse.Namespace) -> int:
style = "green" if "ready" in status else "red"
- console.print(f"{name:<20} {platform:<15} {size:<10} {created:<12} [{style}]{status}[/{style}]")
+ console.print(
+ f"{name:<20} {platform:<15} {size:<10} {created:<12} [{style}]{status}[/{style}]"
+ )
print("\n" + "=" * 85)
print("\nCommands:")
diff --git a/libs/python/cua-cli/cua_cli/commands/mcp.py b/libs/python/cua-cli/cua_cli/commands/mcp.py
index 3cf300a69d..8e3ca017da 100644
--- a/libs/python/cua-cli/cua_cli/commands/mcp.py
+++ b/libs/python/cua-cli/cua_cli/commands/mcp.py
@@ -706,6 +706,280 @@ async def _register_computer_tools(
result = await _send_command(sandbox, "open", {"path": path})
return json.dumps(result)
+ @server.tool()
+ async def computer_window_focus(ctx: Context, window_id: str, sandbox: str = "") -> str:
+ """Bring a window to the foreground and focus it.
+
+ Args:
+ window_id: Window identifier
+ sandbox: Sandbox name (optional)
+ """
+ result = await _send_command(sandbox, "activate_window", {"window_id": window_id})
+ return json.dumps(result)
+
+ @server.tool()
+ async def computer_window_unfocus(ctx: Context, sandbox: str = "") -> str:
+ """Remove focus from the currently focused window.
+
+ Args:
+ sandbox: Sandbox name (optional)
+ """
+ result = await _send_command(sandbox, "deactivate_window", {})
+ if not result.get("success"):
+ # Fallback: press Escape
+ result = await _send_command(sandbox, "press_key", {"key": "escape"})
+ return json.dumps(result)
+
+ @server.tool()
+ async def computer_window_minimize(ctx: Context, window_id: str, sandbox: str = "") -> str:
+ """Minimize a window.
+
+ Args:
+ window_id: Window identifier
+ sandbox: Sandbox name (optional)
+ """
+ result = await _send_command(sandbox, "minimize_window", {"window_id": window_id})
+ return json.dumps(result)
+
+ @server.tool()
+ async def computer_window_maximize(ctx: Context, window_id: str, sandbox: str = "") -> str:
+ """Maximize a window.
+
+ Args:
+ window_id: Window identifier
+ sandbox: Sandbox name (optional)
+ """
+ result = await _send_command(sandbox, "maximize_window", {"window_id": window_id})
+ return json.dumps(result)
+
+ @server.tool()
+ async def computer_window_close(ctx: Context, window_id: str, sandbox: str = "") -> str:
+ """Close a window.
+
+ Args:
+ window_id: Window identifier
+ sandbox: Sandbox name (optional)
+ """
+ result = await _send_command(sandbox, "close_window", {"window_id": window_id})
+ return json.dumps(result)
+
+ @server.tool()
+ async def computer_window_resize(
+ ctx: Context,
+ window_id: str,
+ width: int,
+ height: int,
+ sandbox: str = "",
+ ) -> str:
+ """Resize a window.
+
+ Args:
+ window_id: Window identifier
+ width: New width in pixels
+ height: New height in pixels
+ sandbox: Sandbox name (optional)
+ """
+ result = await _send_command(
+ sandbox,
+ "set_window_size",
+ {"window_id": window_id, "width": width, "height": height},
+ )
+ return json.dumps(result)
+
+ @server.tool()
+ async def computer_window_move(
+ ctx: Context,
+ window_id: str,
+ x: int,
+ y: int,
+ sandbox: str = "",
+ ) -> str:
+ """Move a window to a position on screen.
+
+ Args:
+ window_id: Window identifier
+ x: X coordinate for the window's top-left corner
+ y: Y coordinate for the window's top-left corner
+ sandbox: Sandbox name (optional)
+ """
+ result = await _send_command(
+ sandbox,
+ "set_window_position",
+ {"window_id": window_id, "x": x, "y": y},
+ )
+ return json.dumps(result)
+
+ @server.tool()
+ async def computer_window_get_info(
+ ctx: Context,
+ window_id: str,
+ sandbox: str = "",
+ ) -> str:
+ """Get a window's title, size, and position.
+
+ Args:
+ window_id: Window identifier
+ sandbox: Sandbox name (optional)
+ """
+ name_r = await _send_command(sandbox, "get_window_name", {"window_id": window_id})
+ size_r = await _send_command(sandbox, "get_window_size", {"window_id": window_id})
+ pos_r = await _send_command(sandbox, "get_window_position", {"window_id": window_id})
+ return json.dumps(
+ {
+ "window_id": window_id,
+ "title": name_r.get("name") or name_r.get("data"),
+ "size": size_r.get("size") or size_r.get("data"),
+ "position": pos_r.get("position") or pos_r.get("data"),
+ },
+ indent=2,
+ )
+
+ @server.tool()
+ async def computer_launch(
+ ctx: Context,
+ app: str,
+ args: list[str] | None = None,
+ sandbox: str = "",
+ ) -> str:
+ """Launch an application.
+
+ Args:
+ app: Application executable or bundle identifier
+ args: Optional list of arguments
+ sandbox: Sandbox name (optional)
+ """
+ result = await _send_command(sandbox, "launch", {"app": app, "args": args or []})
+ return json.dumps(result)
+
+ if Permission.COMPUTER_CLICK in permissions:
+
+ @server.tool()
+ async def computer_move_cursor(
+ ctx: Context,
+ x: int,
+ y: int,
+ sandbox: str = "",
+ ) -> str:
+ """Move the mouse cursor without clicking.
+
+ Args:
+ x: X coordinate
+ y: Y coordinate
+ sandbox: Sandbox name (optional)
+ """
+ result = await _send_command(sandbox, "move_cursor", {"x": x, "y": y})
+ return json.dumps(result)
+
+ @server.tool()
+ async def computer_mouse_down(
+ ctx: Context,
+ x: int,
+ y: int,
+ button: str = "left",
+ sandbox: str = "",
+ ) -> str:
+ """Press and hold a mouse button.
+
+ Args:
+ x: X coordinate
+ y: Y coordinate
+ button: Mouse button (left, right, middle)
+ sandbox: Sandbox name (optional)
+ """
+ result = await _send_command(sandbox, "mouse_down", {"x": x, "y": y, "button": button})
+ return json.dumps(result)
+
+ @server.tool()
+ async def computer_mouse_up(
+ ctx: Context,
+ x: int,
+ y: int,
+ button: str = "left",
+ sandbox: str = "",
+ ) -> str:
+ """Release a mouse button.
+
+ Args:
+ x: X coordinate
+ y: Y coordinate
+ button: Mouse button (left, right, middle)
+ sandbox: Sandbox name (optional)
+ """
+ result = await _send_command(sandbox, "mouse_up", {"x": x, "y": y, "button": button})
+ return json.dumps(result)
+
+ if Permission.COMPUTER_KEY in permissions:
+
+ @server.tool()
+ async def computer_key_down(ctx: Context, key: str, sandbox: str = "") -> str:
+ """Press and hold a key.
+
+ Args:
+ key: Key to hold (e.g. "shift", "ctrl", "a")
+ sandbox: Sandbox name (optional)
+ """
+ result = await _send_command(sandbox, "key_down", {"key": key})
+ return json.dumps(result)
+
+ @server.tool()
+ async def computer_key_up(ctx: Context, key: str, sandbox: str = "") -> str:
+ """Release a previously held key.
+
+ Args:
+ key: Key to release
+ sandbox: Sandbox name (optional)
+ """
+ result = await _send_command(sandbox, "key_up", {"key": key})
+ return json.dumps(result)
+
+ if Permission.COMPUTER_SCREENSHOT in permissions:
+
+ @server.tool()
+ async def computer_get_screen_size(ctx: Context, sandbox: str = "") -> str:
+ """Get the screen dimensions.
+
+ Args:
+ sandbox: Sandbox name (optional)
+ """
+ result = await _send_command(sandbox, "get_screen_size", {})
+ return json.dumps(result)
+
+ @server.tool()
+ async def computer_get_cursor_position(ctx: Context, sandbox: str = "") -> str:
+ """Get the current cursor position.
+
+ Args:
+ sandbox: Sandbox name (optional)
+ """
+ result = await _send_command(sandbox, "get_cursor_position", {})
+ return json.dumps(result)
+
+ @server.tool()
+ async def computer_get_accessibility_tree(ctx: Context, sandbox: str = "") -> str:
+ """Get the accessibility tree of the current screen.
+
+ Args:
+ sandbox: Sandbox name (optional)
+ """
+ result = await _send_command(sandbox, "get_accessibility_tree", {})
+ return json.dumps(result)
+
+ @server.tool()
+ async def computer_get_current_window(ctx: Context, sandbox: str = "") -> str:
+ """Get the currently focused window ID and title.
+
+ Args:
+ sandbox: Sandbox name (optional)
+ """
+ win_r = await _send_command(sandbox, "get_current_window_id", {})
+ window_id = win_r.get("window_id") or win_r.get("data")
+ if window_id:
+ name_r = await _send_command(sandbox, "get_window_name", {"window_id": window_id})
+ title = name_r.get("name") or name_r.get("data") or ""
+ else:
+ title = ""
+ return json.dumps({"window_id": window_id, "title": title or "Desktop"})
+
async def _register_skills_tools(server: "FastMCP", permissions: set[Permission]) -> None:
"""Register skills management tools."""
diff --git a/libs/python/cua-cli/cua_cli/main.py b/libs/python/cua-cli/cua_cli/main.py
index 2e290ca6e7..ebf8528044 100644
--- a/libs/python/cua-cli/cua_cli/main.py
+++ b/libs/python/cua-cli/cua_cli/main.py
@@ -5,7 +5,7 @@ import logging
import sys
from cua_cli import __version__
-from cua_cli.commands import auth, image, mcp, platform, sandbox, skills
+from cua_cli.commands import auth, do, image, mcp, platform, sandbox, skills
from cua_cli.utils.output import print_error
@@ -27,6 +27,10 @@ Examples:
cua image create linux-docker Create a local image
cua image shell <name> Interactive shell into image
cua platform list Show available platforms
+ cua do switch docker my-ct Select automation target VM
+ cua do screenshot Take a screenshot
+ cua do click 100 200 Click at coordinates
+ cua do type "hello" Type text
For more information, visit https://docs.trycua.com
""",
@@ -48,6 +52,8 @@ For more information, visit https://docs.trycua.com
platform.register_parser(subparsers)
skills.register_parser(subparsers)
mcp.register_parser(subparsers)
+ do.register_parser(subparsers)
+ do.register_host_consent_parser(subparsers)
return parser
@@ -81,6 +87,10 @@ def main() -> int:
return skills.execute(args)
elif args.command == "serve-mcp":
return mcp.execute(args)
+ elif args.command == "do":
+ return do.execute(args)
+ elif args.command == "do-host-consent":
+ return do.execute_host_consent(args)
else:
print_error(f"Unknown command: {args.command}")
return 1
diff --git a/libs/python/cua-cli/pyproject.toml b/libs/python/cua-cli/pyproject.toml
index 4dd9fda2a3..f4b141fc32 100644
--- a/libs/python/cua-cli/pyproject.toml
+++ b/libs/python/cua-cli/pyproject.toml
@@ -24,12 +24,14 @@ classifiers = [
"Programming Language :: Python :: 3.13",
"Environment :: Console",
]
-requires-python = ">=3.11,<3.14"
+requires-python = ">=3.12,<3.14"
dependencies = [
# Core CUA packages
- "cua-computer>=0.4.0",
+ "cua-computer>=0.5.0",
"cua-core>=0.1.0",
+ # Host automation (used by cua do switch host)
+ "cua-auto>=0.1.0",
# HTTP client
"aiohttp>=3.9.0",
# CLI output
diff --git a/skills/cua-do-cli/SKILL.md b/skills/cua-do-cli/SKILL.md
new file mode 100644
index 0000000000..a4232b0f8e
--- /dev/null
+++ b/skills/cua-do-cli/SKILL.md
@@ -0,0 +1,127 @@
+---
+name: cua-do-cli
+description: Automates interactions with remote VMs, local Docker containers, cloud sandboxes, and the host PC via cua do. Use when navigating GUIs, taking snapshots/screenshots, clicking, typing, scrolling, running shell commands, or managing windows on a target machine.
+---
+
+# VM Automation with cua do
+
+One command = one action = one ✅/❌ line. Switch targets once, then operate.
+
+## Start Here
+
+1. Pick a flow:
+ - API key available: switch → snapshot → interact → snapshot
+ - No API key: switch → screenshot → read image → interact → screenshot
+ - Host control: consent → switch host → operate
+2. Run the canonical flow below.
+3. Check the command skeleton only if blocked.
+
+## Decision Map
+
+- No target set: `switch` → then operate
+- **ANTHROPIC_API_KEY set** (normal): `switch` → `snapshot` → `click/type/key` → repeat snapshot after each UI change
+- **No API key**: `switch` → `screenshot` → read image to determine coords → `click/type/key` → repeat screenshot
+- Zoom to one window: `zoom "App Name"` → operate → `unzoom`
+- Run shell: `shell "command"`
+- Control host PC: `do-host-consent` (once) → `switch host` → operate
+
+## Canonical Flows
+
+### 1) Normal (ANTHROPIC_API_KEY set)
+
+```bash
+cua do switch docker my-container
+cua do snapshot # AI summary + interactive elements with coords
+cua do click 450 300
+cua do type "hello"
+cua do key enter
+cua do snapshot # re-snapshot after UI change
+```
+
+### 2) No API Key
+
+```bash
+cua do switch docker my-container
+cua do screenshot # save path printed; read the image to find coords
+cua do click 450 300
+cua do type "hello"
+cua do key enter
+cua do screenshot # re-screenshot after UI change
+```
+
+### 3) Focused Window (Zoom)
+
+```bash
+cua do zoom "Chrome"
+cua do snapshot # cropped to Chrome; coords are window-relative
+cua do click 120 80
+cua do unzoom
+```
+
+### 4) Host PC Control
+
+```bash
+cua do-host-consent # one-time consent, no prompt
+cua do switch host
+cua do snapshot
+cua do click 100 200
+```
+
+## Command Skeleton
+
+### Target
+
+```bash
+cua do switch <provider> [name] # cloud, docker, lume, lumier, winsandbox, host
+cua do status
+cua do ls [provider]
+```
+
+### Snapshot / Screenshot
+
+```bash
+cua do snapshot ["extra instructions"] # screenshot + AI summary (needs ANTHROPIC_API_KEY)
+cua do screenshot [--save path]
+cua do zoom "Window Name"
+cua do unzoom
+```
+
+### Input
+
+```bash
+cua do click <x> <y> [left|right|middle]
+cua do dclick <x> <y>
+cua do move <x> <y>
+cua do type "text"
+cua do key <key> # enter, escape, tab, space, f1–f12, …
+cua do hotkey ctrl+c
+cua do scroll <up|down|left|right> [n]
+cua do drag <x1> <y1> <x2> <y2>
+```
+
+### Shell / Open
+
+```bash
+cua do shell "command"
+cua do open <url|path>
+```
+
+### Window
+
+```bash
+cua do window ls [app]
+cua do window focus <id>
+cua do window unfocus
+cua do window minimize/maximize/close <id>
+cua do window resize <id> <w> <h>
+cua do window move <id> <x> <y>
+cua do window info <id>
+```
+
+## Guardrails
+
+- Re-snapshot after navigation, modals, or list changes — coords go stale.
+- `snapshot` needs `ANTHROPIC_API_KEY`; use `screenshot` otherwise.
+- Coords are image-space: zoom + max-length scaling are applied automatically.
+- `do-host-consent` is permanent until the consent file is deleted (`~/.cua/host_consented`).
+- Set `PYTHONIOENCODING=utf-8` on Windows for correct emoji output.