Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
Add agent-computer/peakaboo style CLI group to cua-cli, add `cua do` SKILL.md, and cua-auto package (#1107)
ddupont90278dd730c4
diff --git a/.github/workflows/cd-py-auto.yml b/.github/workflows/cd-py-auto.ymlnew file mode 100644index 0000000000..52be003fdd--- /dev/null+++ b/.github/workflows/cd-py-auto.yml@@ -0,0 +1,74 @@+name: "CD: cua-auto (PyPI)"++on:+ push:+ tags:+ - "auto-v*"+ workflow_dispatch:+ inputs:+ version:+ description: "Version to publish (without v prefix)"+ required: true+ default: "0.1.0"+ workflow_call:+ inputs:+ version:+ description: "Version to publish"+ required: true+ type: string++# Adding permissions at workflow level+permissions:+ contents: write++jobs:+ prepare:+ runs-on: macos-latest+ outputs:+ version: ${{ steps.get-version.outputs.version }}+ steps:+ - uses: actions/checkout@v4++ - name: Determine version+ id: get-version+ run: |+ # Check inputs.version first (set by workflow_call)+ if [ -n "${{ inputs.version }}" ]; then+ VERSION=${{ inputs.version }}+ elif [ "${{ github.event_name }}" == "push" ]; then+ # Extract version from tag (for package-specific tags)+ if [[ "${{ github.ref }}" =~ ^refs/tags/auto-v([0-9]+\.[0-9]+\.[0-9]+) ]]; then+ VERSION=${BASH_REMATCH[1]}+ else+ echo "Invalid tag format for auto"+ exit 1+ fi+ elif [ "${{ github.event_name }}" == "workflow_dispatch" ]; then+ # Use version from workflow dispatch+ VERSION=${{ github.event.inputs.version }}+ else+ echo "No version provided"+ exit 1+ fi+ echo "VERSION=$VERSION"+ echo "version=$VERSION" >> $GITHUB_OUTPUT++ publish:+ needs: prepare+ uses: ./.github/workflows/py-reusable-publish.yml+ with:+ package_name: "auto"+ package_dir: "libs/python/cua-auto"+ version: ${{ needs.prepare.outputs.version }}+ base_package_name: "cua-auto"+ secrets:+ PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}++ create-release:+ needs: [prepare, publish]+ uses: ./.github/workflows/release-github-reusable.yml+ with:+ tag_name: "auto-v${{ needs.prepare.outputs.version }}"+ release_name: "cua-auto v${{ needs.prepare.outputs.version }}"+ module_path: "libs/python/cua-auto"+ body: ""diff --git a/.github/workflows/ci-release-reminder.yml b/.github/workflows/ci-release-reminder.ymlindex d0d6c49821..262caf3b52 100644--- a/.github/workflows/ci-release-reminder.yml+++ b/.github/workflows/ci-release-reminder.yml@@ -27,6 +27,7 @@ jobs:# Map paths to publishable servicesdeclare -A PATH_TO_SERVICE=(["libs/python/cua-cli/"]="pypi/cli"+ ["libs/python/cua-auto/"]="pypi/auto"["libs/python/agent/"]="pypi/agent"["libs/python/computer/"]="pypi/computer"["libs/python/core/"]="pypi/core"diff --git a/.github/workflows/release-bump-version.yml b/.github/workflows/release-bump-version.ymlindex 68512cc1ab..5eab5ed8a2 100644--- a/.github/workflows/release-bump-version.yml+++ b/.github/workflows/release-bump-version.yml@@ -9,6 +9,7 @@ on:type: choiceoptions:- pypi/agent+ - pypi/auto- pypi/bench- pypi/bench-ui- pypi/cli@@ -54,6 +55,10 @@ jobs:echo "directory=libs/python/agent" >> $GITHUB_OUTPUTecho "type=python" >> $GITHUB_OUTPUT;;+ "pypi/auto")+ echo "directory=libs/python/cua-auto" >> $GITHUB_OUTPUT+ echo "type=python" >> $GITHUB_OUTPUT+ ;;"pypi/bench")echo "directory=libs/cua-bench" >> $GITHUB_OUTPUTecho "type=python" >> $GITHUB_OUTPUT@@ -212,6 +217,15 @@ jobs:echo "Agent version: $VERSION"echo "version=$VERSION" >> "$GITHUB_OUTPUT"+ - name: Capture bumped auto version+ if: ${{ inputs.service == 'pypi/auto' }}+ id: auto_version+ run: |+ cd libs/python/cua-auto+ VERSION=$(python -c "import tomllib; from pathlib import Path; data = tomllib.loads(Path('pyproject.toml').read_text()); print(data['project']['version'])")+ echo "Auto version: $VERSION"+ echo "version=$VERSION" >> "$GITHUB_OUTPUT"+- name: Capture bumped bench versionif: ${{ inputs.service == 'pypi/bench' }}id: bench_versiondiff --git a/.github/workflows/release-on-merge.yml b/.github/workflows/release-on-merge.ymlindex 4b416aa92d..39e33fb511 100644--- a/.github/workflows/release-on-merge.yml+++ b/.github/workflows/release-on-merge.yml@@ -47,6 +47,7 @@ jobs:# Map paths to publishable servicesdeclare -A PATH_TO_SERVICE=(["libs/python/cua-cli/"]="pypi/cli"+ ["libs/python/cua-auto/"]="pypi/auto"["libs/python/agent/"]="pypi/agent"["libs/python/computer/"]="pypi/computer"["libs/python/core/"]="pypi/core"diff --git a/.github/workflows/release-unreleased-digest.yml b/.github/workflows/release-unreleased-digest.ymlindex c4582900d2..4ef890ee41 100644--- a/.github/workflows/release-unreleased-digest.yml+++ b/.github/workflows/release-unreleased-digest.yml@@ -21,6 +21,7 @@ jobs:# Service → tag prefix → directorydeclare -A SERVICE_TAG_DIRSERVICE_TAG_DIR["pypi/cli"]="cli-v|libs/python/cua-cli/"+ SERVICE_TAG_DIR["pypi/auto"]="auto-v|libs/python/cua-auto/"SERVICE_TAG_DIR["pypi/agent"]="agent-v|libs/python/agent/"SERVICE_TAG_DIR["pypi/computer"]="computer-v|libs/python/computer/"SERVICE_TAG_DIR["pypi/core"]="core-v|libs/python/core/"diff --git a/Development.md b/Development.mdindex 590f02a071..6d843eabb1 100644--- a/Development.md+++ b/Development.md@@ -305,6 +305,7 @@ All packages are managed through a single consolidated workflow: [Bump Version &**Python (PyPI):**- `pypi/agent` - AI agent library+- `pypi/auto` - Cross-platform automation library (mouse, keyboard, screen, window, clipboard, shell)- `pypi/bench` - Benchmark toolkit for computer-use RL environments- `pypi/computer` - Computer-use interface library- `pypi/computer-server` - Server component for VM@@ -365,6 +366,7 @@ make show-versionsEach package uses its own tag format defined in `.bumpversion.cfg`:- **cua-agent**: `agent-v{version}` (e.g., `agent-v0.4.35`)+- **cua-auto**: `auto-v{version}` (e.g., `auto-v0.1.0`)- **cua-bench**: `bench-v{version}` (e.g., `bench-v0.1.0`)- **cua-computer**: `computer-v{version}` (e.g., `computer-v0.4.7`)- **cua-computer-server**: `computer-server-v{version}` (e.g., `computer-server-v0.1.27`)diff --git a/libs/python/computer-server/computer_server/handlers/android.py b/libs/python/computer-server/computer_server/handlers/android.pyindex 2e599cb6fe..583b758aea 100644--- a/libs/python/computer-server/computer_server/handlers/android.py+++ b/libs/python/computer-server/computer_server/handlers/android.py@@ -234,6 +234,12 @@ class AndroidAutomationHandler(BaseAutomationHandler):else:raise RuntimeError(f"Long press failed: {output}")+ async def middle_click(+ self, x: Optional[int] = None, y: Optional[int] = None+ ) -> Dict[str, Any]:+ """Middle click is not supported on Android; returns not-supported error."""+ return {"success": False, "error": "middle_click is not supported on Android"}+async def double_click(self, x: Optional[int] = None, y: Optional[int] = None) -> Dict[str, Any]:diff --git a/libs/python/computer-server/computer_server/handlers/base.py b/libs/python/computer-server/computer_server/handlers/base.pyindex ea977eb2f8..d38ef553a0 100644--- a/libs/python/computer-server/computer_server/handlers/base.py+++ b/libs/python/computer-server/computer_server/handlers/base.py@@ -217,6 +217,13 @@ class BaseAutomationHandler(ABC):"""Perform a right click at the current or specified position."""pass+ @abstractmethod+ async def middle_click(+ self, x: Optional[int] = None, y: Optional[int] = None+ ) -> Dict[str, Any]:+ """Perform a middle click at the current or specified position."""+ pass+@abstractmethodasync def double_click(self, x: Optional[int] = None, y: Optional[int] = Nonediff --git a/libs/python/computer-server/computer_server/handlers/linux.py b/libs/python/computer-server/computer_server/handlers/linux.pyindex 25ccd3f979..52b2586653 100644--- a/libs/python/computer-server/computer_server/handlers/linux.py+++ b/libs/python/computer-server/computer_server/handlers/linux.py@@ -221,6 +221,20 @@ class LinuxAutomationHandler(BaseAutomationHandler):except Exception as e:return {"success": False, "error": str(e)}+ async def middle_click(+ self, x: Optional[int] = None, y: Optional[int] = None+ ) -> Dict[str, Any]:+ """Perform a middle mouse click at the specified coordinates."""+ try:+ from pynput.mouse import Button++ if x is not None and y is not None:+ self.mouse.position = (x, y)+ self.mouse.click(Button.middle, 1)+ return {"success": True}+ except Exception as e:+ return {"success": False, "error": str(e)}+async def double_click(self, x: Optional[int] = None, y: Optional[int] = None) -> Dict[str, Any]:diff --git a/libs/python/computer-server/computer_server/handlers/macos.py b/libs/python/computer-server/computer_server/handlers/macos.pyindex 11bff6c325..c6bf41e1ab 100644--- a/libs/python/computer-server/computer_server/handlers/macos.py+++ b/libs/python/computer-server/computer_server/handlers/macos.py@@ -1058,6 +1058,18 @@ class MacOSAutomationHandler(BaseAutomationHandler):except Exception as e:return {"success": False, "error": str(e)}+ async def middle_click(+ self, x: Optional[int] = None, y: Optional[int] = None+ ) -> Dict[str, Any]:+ """Perform a middle mouse click at the specified coordinates."""+ try:+ if x is not None and y is not None:+ self.mouse.position = (x, y)+ self.mouse.click(Button.middle, 1)+ return {"success": True}+ except Exception as e:+ return {"success": False, "error": str(e)}+async def double_click(self, x: Optional[int] = None, y: Optional[int] = None) -> Dict[str, Any]:diff --git a/libs/python/computer-server/computer_server/handlers/windows.py b/libs/python/computer-server/computer_server/handlers/windows.pyindex c1152207be..b8ff768ab3 100644--- a/libs/python/computer-server/computer_server/handlers/windows.py+++ b/libs/python/computer-server/computer_server/handlers/windows.py@@ -374,6 +374,27 @@ class WindowsAutomationHandler(BaseAutomationHandler):except Exception as e:return {"success": False, "error": str(e)}+ @require_unlocked_desktop+ async def middle_click(+ self, x: Optional[int] = None, y: Optional[int] = None+ ) -> Dict[str, Any]:+ """Perform a middle mouse click at the specified coordinates.++ Args:+ x (Optional[int]): The x-coordinate to click at. If None, clicks at current position.+ y (Optional[int]): The y-coordinate to click at. If None, clicks at current position.++ Returns:+ Dict[str, Any]: A dictionary with success status and optional error message.+ """+ try:+ if x is not None and y is not None:+ self.mouse.position = (x, y)+ self.mouse.click(MouseButton.middle, 1)+ return {"success": True}+ except Exception as e:+ return {"success": False, "error": str(e)}+@require_unlocked_desktopasync def double_click(self, x: Optional[int] = None, y: Optional[int] = Nonediff --git a/libs/python/cua-auto/.bumpversion.cfg b/libs/python/cua-auto/.bumpversion.cfgnew file mode 100644index 0000000000..e425745425--- /dev/null+++ b/libs/python/cua-auto/.bumpversion.cfg@@ -0,0 +1,14 @@+[bumpversion]+current_version = 0.1.0+commit = True+tag = True+tag_name = auto-v{new_version}+message = Bump cua-auto to v{new_version}++[bumpversion:file:pyproject.toml]+search = version = "{current_version}"+replace = version = "{new_version}"++[bumpversion:file:cua_auto/__init__.py]+search = __version__ = "{current_version}"+replace = __version__ = "{new_version}"diff --git a/libs/python/cua-auto/README.md b/libs/python/cua-auto/README.mdnew file mode 100644index 0000000000..b6f8e454b1--- /dev/null+++ b/libs/python/cua-auto/README.md@@ -0,0 +1,62 @@+# cua-auto++`cua-auto` is a lightweight, cross-platform automation library providing a synchronous, MIT-licensed pyautogui-style API for mouse, keyboard, screen, window, clipboard, and shell operations. It runs on Windows, macOS, and Linux using [pynput](https://github.com/moses-palmer/pynput) for input control and [pywinctl](https://github.com/Kalmat/PyWinCtl) for window management.++```python+import cua_auto.mouse as mouse+import cua_auto.keyboard as keyboard+import cua_auto.screen as screen+import cua_auto.window as window+import cua_auto.clipboard as clipboard+import cua_auto.shell as shell++# Mouse+mouse.click(100, 200) # left click+mouse.right_click(100, 200)+mouse.double_click(100, 200)+mouse.move_to(500, 300)+mouse.mouse_down(100, 200)+mouse.mouse_up(100, 200)+mouse.drag(100, 200, 400, 500) # start → end+mouse.scroll_up(3)+mouse.scroll_down(3)+x, y = mouse.position()++# Keyboard+keyboard.press_key("enter")+keyboard.type_text("hello world")+keyboard.hotkey(["ctrl", "c"])+keyboard.key_down("shift")+keyboard.key_up("shift")++# Screen+img = screen.screenshot() # returns PIL.Image+png = screen.screenshot_bytes() # raw PNG bytes+b64 = screen.screenshot_b64() # base64 string+w, h = screen.screen_size()+x, y = screen.cursor_position()++# Window+title = window.get_active_window_title()+handle = window.get_active_window_handle()+handles = window.get_windows_with_title("Chrome")+name = window.get_window_name(handle)+x, y = window.get_window_position(handle)+w, h = window.get_window_size(handle)+window.activate_window(handle)+window.minimize_window(handle)+window.maximize_window(handle)+window.close_window(handle)+window.set_window_size(handle, 1280, 800)+window.set_window_position(handle, 0, 0)+window.open("https://example.com") # or file path+pid = window.launch("notepad.exe")++# Clipboard+text = clipboard.get()+clipboard.set("hello")++# Shell+result = shell.run("echo hi") # CommandResult+print(result.stdout, result.returncode)+```diff --git a/libs/python/cua-auto/cua_auto/__init__.py b/libs/python/cua-auto/cua_auto/__init__.pynew file mode 100644index 0000000000..1817577410--- /dev/null+++ b/libs/python/cua-auto/cua_auto/__init__.py@@ -0,0 +1,25 @@+"""cua-auto — cross-platform automation library.++Provides a synchronous, pyautogui-style API for mouse, keyboard, screen,+window, clipboard, and shell operations.++Usage::++ import cua_auto.mouse as mouse+ import cua_auto.keyboard as keyboard+ import cua_auto.screen as screen+ import cua_auto.window as window+ import cua_auto.clipboard as clipboard+ import cua_auto.shell as shell++ mouse.click(100, 200)+ keyboard.hotkey(["ctrl", "c"])+ img = screen.screenshot()+ title = window.get_active_window_title()+"""++__version__ = "0.1.0"++from cua_auto import clipboard, keyboard, mouse, screen, shell, window++__all__ = ["mouse", "keyboard", "screen", "window", "clipboard", "shell"]diff --git a/libs/python/cua-auto/cua_auto/_platform.py b/libs/python/cua-auto/cua_auto/_platform.pynew file mode 100644index 0000000000..e9f5b4f65f--- /dev/null+++ b/libs/python/cua-auto/cua_auto/_platform.py@@ -0,0 +1,11 @@+"""Platform detection helpers."""++import platform as _platform++_sys = _platform.system().lower()++IS_WINDOWS: bool = _sys == "windows"+IS_MACOS: bool = _sys == "darwin"+IS_LINUX: bool = _sys == "linux"++PLATFORM: str = "windows" if IS_WINDOWS else ("macos" if IS_MACOS else "linux")diff --git a/libs/python/cua-auto/cua_auto/clipboard.py b/libs/python/cua-auto/cua_auto/clipboard.pynew file mode 100644index 0000000000..92a0f844b7--- /dev/null+++ b/libs/python/cua-auto/cua_auto/clipboard.py@@ -0,0 +1,13 @@+"""Cross-platform clipboard access via pyperclip."""++import pyperclip as _pyperclip+++def get() -> str:+ """Return the current clipboard text content."""+ return _pyperclip.paste()+++def set(text: str) -> None:+ """Set the clipboard text content."""+ _pyperclip.copy(text)diff --git a/libs/python/cua-auto/cua_auto/keyboard.py b/libs/python/cua-auto/cua_auto/keyboard.pynew file mode 100644index 0000000000..25c15a8a92--- /dev/null+++ b/libs/python/cua-auto/cua_auto/keyboard.py@@ -0,0 +1,153 @@+"""Cross-platform keyboard control via pynput."""++from typing import List, Optional, Union++from pynput.keyboard import Controller as _KBController+from pynput.keyboard import Key as _Key++_kb = _KBController()++# Unified key name → pynput Key map (sourced from computer-server/handlers/windows.py)+_SPECIAL: dict = {+ "enter": _Key.enter,+ "return": _Key.enter,+ "esc": _Key.esc,+ "escape": _Key.esc,+ "space": _Key.space,+ "tab": _Key.tab,+ "backspace": _Key.backspace,+ "delete": _Key.delete,+ "home": _Key.home,+ "end": _Key.end,+ "pageup": _Key.page_up,+ "page_up": _Key.page_up,+ "pagedown": _Key.page_down,+ "page_down": _Key.page_down,+ "up": _Key.up,+ "down": _Key.down,+ "left": _Key.left,+ "right": _Key.right,+ "shift": _Key.shift,+ "shift_l": _Key.shift_l,+ "shift_r": _Key.shift_r,+ "ctrl": _Key.ctrl,+ "ctrl_l": _Key.ctrl_l,+ "ctrl_r": _Key.ctrl_r,+ "control": _Key.ctrl,+ "alt": _Key.alt,+ "alt_l": _Key.alt_l,+ "alt_r": _Key.alt_r,+ "cmd": _Key.cmd,+ "command": _Key.cmd,+ "win": _Key.cmd,+ "super": _Key.cmd,+ "meta": _Key.cmd,+ "option": _Key.alt,+ "option_l": _Key.alt_l,+ "option_r": _Key.alt_r,+ "capslock": _Key.caps_lock,+ "caps_lock": _Key.caps_lock,+ "insert": _Key.insert,+ "print_screen": _Key.print_screen,+ "pause": _Key.pause,+ "num_lock": _Key.num_lock,+ "scroll_lock": _Key.scroll_lock,+ "f1": _Key.f1,+ "f2": _Key.f2,+ "f3": _Key.f3,+ "f4": _Key.f4,+ "f5": _Key.f5,+ "f6": _Key.f6,+ "f7": _Key.f7,+ "f8": _Key.f8,+ "f9": _Key.f9,+ "f10": _Key.f10,+ "f11": _Key.f11,+ "f12": _Key.f12,+ "f13": _Key.f13,+ "f14": _Key.f14,+ "f15": _Key.f15,+ "f16": _Key.f16,+ "f17": _Key.f17,+ "f18": _Key.f18,+ "f19": _Key.f19,+ "f20": _Key.f20,+}+++def _resolve(key: str) -> Optional[Union[str, _Key]]:+ """Resolve a key string to a pynput Key or single character."""+ if not key:+ return None+ lk = key.lower()+ if lk in _SPECIAL:+ return _SPECIAL[lk]+ if len(key) == 1:+ return key+ return None+++# ── Key press / release ───────────────────────────────────────────────────────+++def key_down(key: str) -> None:+ """Press and hold a key."""+ k = _resolve(key)+ if k is None:+ raise ValueError(f"Unknown key: {key!r}")+ _kb.press(k)+++def key_up(key: str) -> None:+ """Release a key."""+ k = _resolve(key)+ if k is None:+ raise ValueError(f"Unknown key: {key!r}")+ _kb.release(k)+++def press_key(key: str) -> None:+ """Press and immediately release a key."""+ k = _resolve(key)+ if k is None:+ raise ValueError(f"Unknown key: {key!r}")+ _kb.press(k)+ _kb.release(k)+++# ── Text typing ───────────────────────────────────────────────────────────────+++def type_text(text: str) -> None:+ """Type a string of text (supports Unicode)."""+ _kb.type(text)+++# ── Hotkeys ───────────────────────────────────────────────────────────────────+++def hotkey(keys: List[str]) -> None:+ """Press a combination of keys, e.g. ['ctrl', 'c'].++ Modifiers are held while the last key is tapped, then released in reverse+ order (standard hotkey behaviour).+ """+ resolved = [_resolve(k) for k in keys]+ if any(k is None for k in resolved):+ bad = [k for k, r in zip(keys, resolved) if r is None]+ raise ValueError(f"Unknown keys in hotkey: {bad}")++ seq: List[Union[str, _Key]] = [k for k in resolved if k is not None]+ if not seq:+ return++ # Hold all modifiers except the last key+ for k in seq[:-1]:+ _kb.press(k)+ # Tap the action key+ last = seq[-1]+ _kb.press(last)+ _kb.release(last)+ # Release modifiers in reverse order+ for k in reversed(seq[:-1]):+ _kb.release(k)diff --git a/libs/python/cua-auto/cua_auto/mouse.py b/libs/python/cua-auto/cua_auto/mouse.pynew file mode 100644index 0000000000..4e0691de90--- /dev/null+++ b/libs/python/cua-auto/cua_auto/mouse.py@@ -0,0 +1,138 @@+"""Cross-platform mouse control via pynput."""++from typing import List, Optional, Tuple++from pynput.mouse import Button as _Button+from pynput.mouse import Controller as _MouseController++_mouse = _MouseController()+++def _map_button(button: str) -> _Button:+ """Map a button name string to a pynput Button."""+ b = (button or "left").lower()+ if b == "right":+ return _Button.right+ if b == "middle":+ return _Button.middle+ return _Button.left+++# ── Basic clicks ──────────────────────────────────────────────────────────────+++def click(x: int, y: int, button: str = "left") -> None:+ """Single click at (x, y)."""+ _mouse.position = (x, y)+ _mouse.click(_map_button(button), 1)+++def right_click(x: int, y: int) -> None:+ """Right-click at (x, y)."""+ click(x, y, "right")+++def double_click(x: int, y: int) -> None:+ """Double left-click at (x, y)."""+ _mouse.position = (x, y)+ _mouse.click(_Button.left, 2)+++# ── Cursor movement ───────────────────────────────────────────────────────────+++def move_to(x: int, y: int) -> None:+ """Move cursor to (x, y) without clicking."""+ _mouse.position = (x, y)+++# ── Mouse button hold / release ───────────────────────────────────────────────+++def mouse_down(x: Optional[int] = None, y: Optional[int] = None, button: str = "left") -> None:+ """Press and hold a mouse button, optionally moving to (x, y) first."""+ if x is not None and y is not None:+ _mouse.position = (x, y)+ _mouse.press(_map_button(button))+++def mouse_up(x: Optional[int] = None, y: Optional[int] = None, button: str = "left") -> None:+ """Release a mouse button, optionally moving to (x, y) first."""+ if x is not None and y is not None:+ _mouse.position = (x, y)+ _mouse.release(_map_button(button))+++# ── Drag ──────────────────────────────────────────────────────────────────────+++def drag(+ start_x: int,+ start_y: int,+ end_x: int,+ end_y: int,+ button: str = "left",+) -> None:+ """Drag from (start_x, start_y) to (end_x, end_y)."""+ btn = _map_button(button)+ _mouse.position = (start_x, start_y)+ _mouse.press(btn)+ _mouse.position = (end_x, end_y)+ _mouse.release(btn)+++def drag_to(x: int, y: int, button: str = "left") -> None:+ """Drag from the current cursor position to (x, y)."""+ btn = _map_button(button)+ _mouse.press(btn)+ _mouse.position = (x, y)+ _mouse.release(btn)+++def drag_path(path: List[Tuple[int, int]], button: str = "left") -> None:+ """Drag through a sequence of (x, y) points."""+ if not path:+ return+ btn = _map_button(button)+ _mouse.position = path[0]+ _mouse.press(btn)+ for x, y in path[1:]:+ _mouse.position = (x, y)+ _mouse.release(btn)+++# ── Scroll ────────────────────────────────────────────────────────────────────+++def scroll(dx: int, dy: int) -> None:+ """Scroll by (dx, dy). Positive dy = scroll up."""+ _mouse.scroll(dx, dy)+++def scroll_up(clicks: int = 3) -> None:+ """Scroll up by *clicks* notches."""+ _mouse.scroll(0, abs(clicks))+++def scroll_down(clicks: int = 3) -> None:+ """Scroll down by *clicks* notches."""+ _mouse.scroll(0, -abs(clicks))+++def scroll_left(clicks: int = 3) -> None:+ """Scroll left by *clicks* notches."""+ _mouse.scroll(-abs(clicks), 0)+++def scroll_right(clicks: int = 3) -> None:+ """Scroll right by *clicks* notches."""+ _mouse.scroll(abs(clicks), 0)+++# ── Position ──────────────────────────────────────────────────────────────────+++def position() -> Tuple[int, int]:+ """Return the current cursor position as (x, y)."""+ x, y = _mouse.position+ return int(x), int(y)diff --git a/libs/python/cua-auto/cua_auto/screen.py b/libs/python/cua-auto/cua_auto/screen.pynew file mode 100644index 0000000000..e1c1988af5--- /dev/null+++ b/libs/python/cua-auto/cua_auto/screen.py@@ -0,0 +1,76 @@+"""Cross-platform screen capture and info."""++import base64+import io+from typing import Tuple++from PIL import Image+++def screenshot() -> Image.Image:+ """Capture a screenshot of all monitors and return a PIL Image.++ Tries PIL.ImageGrab first (works on Windows and macOS).+ Falls back to mss if ImageGrab fails (better Linux / multi-monitor support).+ """+ # Try PIL.ImageGrab (Windows + macOS)+ try:+ from PIL import ImageGrab++ img = ImageGrab.grab(all_screens=True)+ if isinstance(img, Image.Image):+ return img+ except Exception:+ pass++ # Fallback: mss (cross-platform, optional dependency)+ try:+ import mss # type: ignore[import-untyped]+ import mss.tools # type: ignore[import-untyped]++ with mss.mss() as sct:+ monitor = sct.monitors[0] # combined virtual desktop+ sct_img = sct.grab(monitor)+ return Image.frombytes("RGB", sct_img.size, sct_img.bgra, "raw", "BGRX")+ except Exception as e:+ raise RuntimeError(f"screenshot failed: {e}") from e+++def screenshot_bytes(format: str = "PNG") -> bytes:+ """Return screenshot as raw bytes in the specified image format."""+ img = screenshot()+ buf = io.BytesIO()+ img.save(buf, format=format)+ return buf.getvalue()+++def screenshot_b64(format: str = "PNG") -> str:+ """Return screenshot as a base64-encoded string."""+ return base64.b64encode(screenshot_bytes(format)).decode()+++def screen_size() -> Tuple[int, int]:+ """Return the total virtual desktop size as (width, height)."""+ img = screenshot()+ return img.size+++def cursor_position() -> Tuple[int, int]:+ """Return the current cursor position as (x, y).++ Tries win32gui first on Windows (more reliable with DPI scaling).+ Falls back to pynput if win32 is unavailable (cross-platform fallback).+ """+ try:+ import win32gui # type: ignore[import-untyped]++ pos = win32gui.GetCursorPos()+ return int(pos[0]), int(pos[1])+ except Exception:+ pass++ from pynput.mouse import Controller as _MouseController++ c = _MouseController()+ x, y = c.position+ return int(x), int(y)diff --git a/libs/python/cua-auto/cua_auto/shell.py b/libs/python/cua-auto/cua_auto/shell.pynew file mode 100644index 0000000000..9ae019c509--- /dev/null+++ b/libs/python/cua-auto/cua_auto/shell.py@@ -0,0 +1,53 @@+"""Shell command execution."""++import subprocess+from dataclasses import dataclass+++@dataclass+class CommandResult:+ stdout: str+ stderr: str+ returncode: int++ @property+ def success(self) -> bool:+ return self.returncode == 0+++def run(command: str, timeout: int = 30) -> CommandResult:+ """Run a shell command and return stdout, stderr, and returncode.++ The command is passed to the system shell (``shell=True``) so that+ shell built-ins, pipes, and redirections work as expected.+ """++ def _decode(data: bytes) -> str:+ if not data:+ return ""+ for enc in ("utf-8", "gbk", "gb2312", "cp936", "latin1"):+ try:+ return data.decode(enc)+ except (UnicodeDecodeError, LookupError):+ continue+ return data.decode("utf-8", errors="replace")++ try:+ result = subprocess.run(+ command,+ shell=True,+ stdout=subprocess.PIPE,+ stderr=subprocess.PIPE,+ timeout=timeout,+ )+ return CommandResult(+ stdout=_decode(result.stdout),+ stderr=_decode(result.stderr),+ returncode=result.returncode,+ )+ except subprocess.TimeoutExpired as e:+ return CommandResult(+ stdout=_decode(e.stdout or b""),+ stderr=_decode(e.stderr or b""),+ returncode=-1,+ )diff --git a/libs/python/cua-auto/cua_auto/window.py b/libs/python/cua-auto/cua_auto/window.pynew file mode 100644index 0000000000..df0aa5bbea--- /dev/null+++ b/libs/python/cua-auto/cua_auto/window.py@@ -0,0 +1,188 @@+"""Cross-platform window management via pywinctl."""++import os+import platform+import subprocess+import webbrowser+from typing import Any, List, Optional, Tuple++try:+ import pywinctl as _pwc # type: ignore[import-untyped]+except Exception:+ _pwc = None # type: ignore[assignment]+++def _require_pwc() -> Any:+ if _pwc is None:+ raise RuntimeError("pywinctl is not available. Install it with: pip install pywinctl")+ return _pwc+++def _get_by_handle(handle: Any) -> Optional[Any]:+ """Find a pywinctl window object by its native handle."""+ pwc = _require_pwc()+ try:+ for w in pwc.getAllWindows():+ if str(w.getHandle()) == str(handle):+ return w+ except Exception:+ pass+ return None+++# ── Active window ─────────────────────────────────────────────────────────────+++def get_active_window() -> Optional[Any]:+ """Return the pywinctl window object for the currently focused window."""+ pwc = _require_pwc()+ return pwc.getActiveWindow()+++def get_active_window_title() -> str:+ """Return the title of the currently focused window, or 'Desktop'."""+ try:+ win = get_active_window()+ if win:+ return win.title or "Desktop"+ except Exception:+ pass+ return "Desktop"+++def get_active_window_handle() -> Optional[str]:+ """Return the native handle of the active window as a string, or None."""+ try:+ win = get_active_window()+ if win:+ return str(win.getHandle())+ except Exception:+ pass+ return None+++# ── Window lookup ─────────────────────────────────────────────────────────────+++def get_windows_with_title(title: str) -> List[str]:+ """Return a list of native handle strings for windows whose title contains *title*."""+ pwc = _require_pwc()+ try:+ wins = pwc.getWindowsWithTitle(title, condition=pwc.Re.CONTAINS, flags=pwc.Re.IGNORECASE)+ return [str(w.getHandle()) for w in wins]+ except Exception:+ return []+++def get_window_name(handle: Any) -> Optional[str]:+ """Return the title of the window with the given handle, or None."""+ w = _get_by_handle(handle)+ return w.title if w else None+++def get_window_size(handle: Any) -> Optional[Tuple[int, int]]:+ """Return (width, height) of the window, or None if not found."""+ w = _get_by_handle(handle)+ if not w:+ return None+ width, height = w.size+ return int(width), int(height)+++def get_window_position(handle: Any) -> Optional[Tuple[int, int]]:+ """Return (x, y) position of the window, or None if not found."""+ w = _get_by_handle(handle)+ if not w:+ return None+ x, y = w.position+ return int(x), int(y)+++# ── Window actions ────────────────────────────────────────────────────────────+++def activate_window(handle: Any) -> bool:+ """Bring the window to the foreground and give it focus."""+ w = _get_by_handle(handle)+ if not w:+ return False+ return bool(w.activate())+++def minimize_window(handle: Any) -> bool:+ """Minimize the window."""+ w = _get_by_handle(handle)+ if not w:+ return False+ return bool(w.minimize())+++def maximize_window(handle: Any) -> bool:+ """Maximize the window."""+ w = _get_by_handle(handle)+ if not w:+ return False+ return bool(w.maximize())+++def close_window(handle: Any) -> bool:+ """Close the window."""+ w = _get_by_handle(handle)+ if not w:+ return False+ return bool(w.close())+++def set_window_size(handle: Any, width: int, height: int) -> bool:+ """Resize the window to (width, height)."""+ w = _get_by_handle(handle)+ if not w:+ return False+ return bool(w.resizeTo(int(width), int(height)))+++def set_window_position(handle: Any, x: int, y: int) -> bool:+ """Move the window to (x, y)."""+ w = _get_by_handle(handle)+ if not w:+ return False+ return bool(w.moveTo(int(x), int(y)))+++# ── Open / launch ─────────────────────────────────────────────────────────────+++def open(target: str) -> bool: # noqa: A001+ """Open a URL or file path with the default application.++ URLs are opened in the default browser. Files are opened with the OS+ default handler (``open`` on macOS, ``xdg-open`` on Linux,+ ``os.startfile`` on Windows).+ """+ if target.startswith("http://") or target.startswith("https://"):+ return bool(webbrowser.open(target))++ sys = platform.system().lower()+ if sys == "darwin":+ subprocess.Popen(["open", target])+ elif sys == "linux":+ subprocess.Popen(["xdg-open", target])+ elif sys == "windows":+ os.startfile(target) # type: ignore[attr-defined]+ else:+ raise RuntimeError(f"Unsupported OS: {sys}")+ return True+++def launch(app: str, args: Optional[List[str]] = None) -> int:+ """Launch an application and return its PID.++ If *args* is given, the app is launched with those arguments.+ Otherwise the command string is passed to the system shell, allowing+ strings like ``"libreoffice --writer"``.+ """+ if args:+ proc = subprocess.Popen([app, *args])+ else:+ proc = subprocess.Popen(app, shell=True)+ return proc.piddiff --git a/libs/python/cua-auto/pyproject.toml b/libs/python/cua-auto/pyproject.tomlnew file mode 100644index 0000000000..bc16ef506d--- /dev/null+++ b/libs/python/cua-auto/pyproject.toml@@ -0,0 +1,91 @@+[project]+name = "cua-auto"+version = "0.1.0"+description = "Cross-platform automation library — mouse, keyboard, screen, window, clipboard, shell"+readme = "README.md"+license = "MIT"+authors = [+ { name = "TryCua", email = "[email protected]" }+]+keywords = [+ "automation",+ "mouse",+ "keyboard",+ "screenshot",+ "window",+ "computer-use",+ "pyautogui",+]+classifiers = [+ "Development Status :: 3 - Alpha",+ "Intended Audience :: Developers",+ "License :: OSI Approved :: MIT License",+ "Operating System :: OS Independent",+ "Programming Language :: Python :: 3",+ "Programming Language :: Python :: 3.11",+ "Programming Language :: Python :: 3.12",+ "Programming Language :: Python :: 3.13",+ "Topic :: Software Development :: Libraries",+]+requires-python = ">=3.11,<3.14"++dependencies = [+ # Mouse + keyboard control (cross-platform)+ "pynput>=1.7.0",+ # Screenshot + image processing+ "pillow>=10.0.0",+ # Clipboard access (cross-platform)+ "pyperclip>=1.9.0",+ # Window management (cross-platform)+ "pywinctl>=0.4",+]++[project.optional-dependencies]+# Faster multi-monitor screenshot backend+mss = [+ "mss>=9.0.0",+]+# Windows-specific extras (cursor position via win32, etc.)+windows = [+ "pywin32>=306; sys_platform == 'win32'",+]+all = [+ "cua-auto[mss,windows]",+]+dev = [+ "pytest>=8.0.0",+ "pytest-asyncio>=0.23.0",+ "ruff>=0.1.0",+]++[project.urls]+Homepage = "https://github.com/trycua/cua"+Documentation = "https://docs.trycua.com"+Repository = "https://github.com/trycua/cua"+Issues = "https://github.com/trycua/cua/issues"++[build-system]+requires = ["hatchling"]+build-backend = "hatchling.build"++[tool.hatch.build]+include = [+ "cua_auto/**",+ "README.md",+ "LICENSE",+]+exclude = [+ "cua_auto/**/__pycache__",+ "**/.DS_Store",+]++[tool.hatch.build.targets.wheel]+packages = ["cua_auto"]++[tool.ruff]+line-length = 100+target-version = "py311"++[tool.ruff.lint]+select = ["E", "F", "W"]+ignore = ["E501"]diff --git a/libs/python/cua-cli/cua_cli/commands/do.py b/libs/python/cua-cli/cua_cli/commands/do.pynew file mode 100644index 0000000000..bc9ae738aa--- /dev/null+++ b/libs/python/cua-cli/cua_cli/commands/do.py@@ -0,0 +1,1495 @@+"""cua do — one-shot VM automation commands for agents.++Output is always a single line starting with ✅ (success) or ❌ (failure),+followed by a context line: 💻 vm-name\t🪟 Window Title (or Desktop).++Target VM is persisted in ~/.cua/do_target.json.+Zoom state (bbox + display scale) is also persisted there.+"""++from __future__ import annotations++import argparse+import base64+import io+import json+import os+import sys+import tempfile+from datetime import datetime+from pathlib import Path+from typing import Any++# Ensure stdout/stderr can handle Unicode emojis on Windows+if sys.platform == "win32":+ import io as _io++ if hasattr(sys.stdout, "buffer"):+ sys.stdout = _io.TextIOWrapper(sys.stdout.buffer, encoding="utf-8", errors="replace")+ if hasattr(sys.stderr, "buffer"):+ sys.stderr = _io.TextIOWrapper(sys.stderr.buffer, encoding="utf-8", errors="replace")++MAX_LENGTH = 1200 # max pixel dimension for screenshot output++# ── state ─────────────────────────────────────────────────────────────────────++_STATE_FILE = Path.home() / ".cua" / "do_target.json"+_HOST_CONSENT_FILE = Path.home() / ".cua" / "host_consented"++# Window titles that are internal OS/browser helper windows and should be hidden+_SKIP_WINDOW_TITLES = {+ "Chrome Legacy Window",+}++PROVIDERS = ("cloud", "cloudv2", "lume", "lumier", "docker", "winsandbox", "host")+_REMOTE_PROVIDERS = ("cloud", "cloudv2", "lume", "lumier", "docker", "winsandbox")+++def _load_state() -> dict:+ if _STATE_FILE.exists():+ try:+ return json.loads(_STATE_FILE.read_text())+ except Exception:+ pass+ return {}+++def _save_state(state: dict) -> None:+ _STATE_FILE.parent.mkdir(parents=True, exist_ok=True)+ _STATE_FILE.write_text(json.dumps(state, indent=2))+++def _update_state(**kwargs) -> dict:+ state = _load_state()+ state.update(kwargs)+ _save_state(state)+ return state+++def _host_consented() -> bool:+ return _HOST_CONSENT_FILE.exists()+++# ── output helpers ────────────────────────────────────────────────────────────+++def _ok(msg: str) -> int:+ print(f"✅ {msg}")+ return 0+++def _fail(msg: str) -> int:+ print(f"❌ {msg}", file=sys.stderr)+ return 1+++# ── provider / connection helpers ─────────────────────────────────────────────+++async def _get_api_url(provider_type: str, name: str) -> str:+ """Resolve the computer-server API URL for the target VM."""+ from computer.providers.base import VMProviderType+ from computer.providers.factory import VMProviderFactory++ if provider_type == "host":+ return "http://localhost:8000"++ if provider_type in ("cloud", "cloudv2"):+ from cua_cli.auth.store import get_api_key++ api_key = get_api_key()+ if not api_key:+ raise ValueError("Not authenticated. Run 'cua auth login' first")+ ptype = VMProviderType.CLOUDV2 if provider_type == "cloudv2" else VMProviderType.CLOUD+ provider = VMProviderFactory.create_provider(ptype, api_key=api_key)+ async with provider:+ vm = await provider.get_vm(name)+ if not vm:+ raise ValueError(f"VM not found: {name}")+ url = vm.get("api_url") or vm.get("server_url")+ if not url:+ raise ValueError(f"VM '{name}' has no API URL (is it running?)")+ return url++ if provider_type == "docker":+ provider = VMProviderFactory.create_provider(VMProviderType.DOCKER)+ async with provider:+ vm = await provider.get_vm(name)+ if vm.get("status") == "not_found":+ raise ValueError(f"Docker container not found: {name}")+ ports = vm.get("ports", {})+ for key in ("8000/tcp", "5000/tcp"):+ if key in ports:+ return f"http://localhost:{ports[key]}"+ return "http://localhost:8000"++ if provider_type in ("lume", "lumier"):+ ptype = VMProviderType.LUME if provider_type == "lume" else VMProviderType.LUMIER+ provider = VMProviderFactory.create_provider(ptype)+ async with provider:+ vm = await provider.get_vm(name)+ ip = vm.get("ip_address") or "localhost"+ return f"http://{ip}:8000"++ if provider_type == "winsandbox":+ return "http://localhost:8000"++ raise ValueError(f"Unknown provider: {provider_type}")+++async def _host_dispatch(command: str, params: dict) -> dict:+ """Dispatch a computer-server command to the local host via cua_auto."""+ try:+ import cua_auto.keyboard as _kb+ import cua_auto.mouse as _mouse+ import cua_auto.screen as _screen+ import cua_auto.shell as _shell+ import cua_auto.window as _win+ except ImportError as e:+ return {+ "success": False,+ "error": f"cua-auto not installed: {e}. Run: pip install cua-auto",+ }++ try:+ if command == "screenshot":+ b64 = _screen.screenshot_b64()+ return {"success": True, "image_data": b64}++ elif command == "get_screen_size":+ w, h = _screen.screen_size()+ return {"success": True, "size": {"width": w, "height": h}}++ elif command == "get_cursor_position":+ x, y = _screen.cursor_position()+ return {"success": True, "position": {"x": x, "y": y}}++ elif command == "get_current_window_id":+ handle = _win.get_active_window_handle()+ if not handle:+ return {"success": False, "error": "No active window"}+ return {"success": True, "window_id": handle}++ elif command == "get_window_name":+ title = _win.get_window_name(params.get("window_id", ""))+ if title is None:+ return {"success": False, "error": "Window not found"}+ return {"success": True, "name": title}++ elif command == "get_application_windows":+ handles = _win.get_windows_with_title(params.get("app", ""))+ return {"success": True, "windows": handles}++ elif command == "get_window_size":+ result = _win.get_window_size(params.get("window_id", ""))+ if result is None:+ return {"success": False, "error": "Window not found"}+ return {"success": True, "size": [result[0], result[1]]}++ elif command == "get_window_position":+ result = _win.get_window_position(params.get("window_id", ""))+ if result is None:+ return {"success": False, "error": "Window not found"}+ return {"success": True, "position": [result[0], result[1]]}++ elif command == "left_click":+ _mouse.click(int(params["x"]), int(params["y"]))+ return {"success": True}++ elif command == "right_click":+ _mouse.right_click(int(params["x"]), int(params["y"]))+ return {"success": True}++ elif command == "middle_click":+ _mouse.click(int(params["x"]), int(params["y"]), "middle")+ return {"success": True}++ elif command == "double_click":+ _mouse.double_click(int(params["x"]), int(params["y"]))+ return {"success": True}++ elif command == "move_cursor":+ _mouse.move_to(int(params["x"]), int(params["y"]))+ return {"success": True}++ elif command == "mouse_down":+ x = params.get("x")+ y = params.get("y")+ _mouse.mouse_down(+ int(x) if x is not None else None,+ int(y) if y is not None else None,+ params.get("button", "left"),+ )+ return {"success": True}++ elif command == "mouse_up":+ x = params.get("x")+ y = params.get("y")+ _mouse.mouse_up(+ int(x) if x is not None else None,+ int(y) if y is not None else None,+ params.get("button", "left"),+ )+ return {"success": True}++ elif command == "scroll_direction":+ direction = params.get("direction", "down")+ clicks = int(params.get("clicks", 3))+ if direction == "up":+ _mouse.scroll_up(clicks)+ elif direction == "down":+ _mouse.scroll_down(clicks)+ elif direction == "left":+ _mouse.scroll_left(clicks)+ elif direction == "right":+ _mouse.scroll_right(clicks)+ return {"success": True}++ elif command == "drag_to":+ _mouse.drag(+ int(params["start_x"]),+ int(params["start_y"]),+ int(params["end_x"]),+ int(params["end_y"]),+ )+ return {"success": True}++ elif command == "type_text":+ _kb.type_text(params.get("text", ""))+ return {"success": True}++ elif command == "press_key":+ _kb.press_key(params.get("key", ""))+ return {"success": True}++ elif command == "key_down":+ _kb.key_down(params.get("key", ""))+ return {"success": True}++ elif command == "key_up":+ _kb.key_up(params.get("key", ""))+ return {"success": True}++ elif command == "hotkey":+ keys = params.get("keys", [])+ if isinstance(keys, str):+ keys = [k.strip() for k in keys.replace("-", "+").split("+") if k.strip()]+ _kb.hotkey(keys)+ return {"success": True}++ elif command == "run_command":+ result = _shell.run(params.get("command", ""))+ return {+ "success": True,+ "stdout": result.stdout,+ "stderr": result.stderr,+ "returncode": result.returncode,+ }++ elif command == "open":+ _win.open(params.get("path") or params.get("target", ""))+ return {"success": True}++ elif command == "launch":+ pid = _win.launch(params.get("app", ""), params.get("args"))+ return {"success": True, "pid": pid}++ elif command == "activate_window":+ ok = _win.activate_window(params.get("window_id", ""))+ return {"success": bool(ok)}++ elif command == "minimize_window":+ ok = _win.minimize_window(params.get("window_id", ""))+ return {"success": bool(ok)}++ elif command == "maximize_window":+ ok = _win.maximize_window(params.get("window_id", ""))+ return {"success": bool(ok)}++ elif command == "close_window":+ ok = _win.close_window(params.get("window_id", ""))+ return {"success": bool(ok)}++ elif command == "set_window_size":+ ok = _win.set_window_size(+ params["window_id"], int(params["width"]), int(params["height"])+ )+ return {"success": bool(ok)}++ elif command == "set_window_position":+ ok = _win.set_window_position(params["window_id"], int(params["x"]), int(params["y"]))+ return {"success": bool(ok)}++ elif command == "deactivate_window":+ return {"success": False, "error": "deactivate_window not supported on host"}++ else:+ return {"success": False, "error": f"Unknown host command: {command}"}++ except Exception as e:+ return {"success": False, "error": str(e)}+++async def _send(provider_type: str, name: str, command: str, params: dict) -> dict:+ """Send a command to the computer-server and return the parsed response."""+ if provider_type == "host":+ return await _host_dispatch(command, params)++ import aiohttp++ api_url = await _get_api_url(provider_type, name)+ headers = {"Content-Type": "application/json"}++ if provider_type in ("cloud", "cloudv2"):+ from cua_cli.auth.store import get_api_key++ api_key = get_api_key()+ if api_key:+ headers["X-API-Key"] = api_key+ if name:+ headers["X-Container-Name"] = name++ async with aiohttp.ClientSession() as session:+ async with session.post(+ f"{api_url}/cmd",+ json={"command": command, "params": params},+ headers=headers,+ timeout=aiohttp.ClientTimeout(total=30),+ ) as resp:+ text = await resp.text()+ for line in text.splitlines():+ if line.startswith("data: "):+ return json.loads(line[6:])+ try:+ return json.loads(text)+ except Exception:+ return {"success": False, "error": f"Unexpected response: {text[:200]}"}+++# ── zoom / screenshot helpers ─────────────────────────────────────────────────+++async def _resolve_zoom_bbox_by_id(provider_type: str, name: str, window_id: str) -> dict | None:+ """Get the bounding box of a window by its native handle/id."""+ pos_r = await _send(provider_type, name, "get_window_position", {"window_id": window_id})+ size_r = await _send(provider_type, name, "get_window_size", {"window_id": window_id})++ pos = pos_r.get("position") or pos_r.get("data")+ size = size_r.get("size") or size_r.get("data")+ if not pos or not size:+ return None++ if isinstance(pos, (list, tuple)):+ x, y = int(pos[0]), int(pos[1])+ else:+ x, y = int(pos.get("x", 0)), int(pos.get("y", 0))++ if isinstance(size, (list, tuple)):+ w, h = int(size[0]), int(size[1])+ else:+ w, h = int(size.get("width", 0)), int(size.get("height", 0))++ return {"x": x, "y": y, "width": w, "height": h}+++async def _resolve_zoom_bbox(provider_type: str, name: str, window_name: str) -> dict | None:+ """Get the bounding box of a window by app/window name.++ Filters out internal helper windows (e.g. 'Chrome Legacy Window') so the+ correct top-level window is always selected.+ """+ wins_r = await _send(provider_type, name, "get_application_windows", {"app": window_name})+ windows = wins_r.get("windows") or wins_r.get("data") or []+ if not windows:+ return None++ # Pick the first window whose title is not an internal helper+ window_id = None+ for wid in windows:+ try:+ name_r = await _send(provider_type, name, "get_window_name", {"window_id": wid})+ title = name_r.get("name") or ""+ if title not in _SKIP_WINDOW_TITLES:+ window_id = wid+ break+ except Exception:+ pass+ if window_id is None:+ window_id = windows[0]++ return await _resolve_zoom_bbox_by_id(provider_type, name, window_id)+++async def _take_screenshot_data(+ provider_type: str, name: str, state: dict+) -> tuple[bytes, float, dict | None]:+ """Take screenshot, apply zoom crop and max-length scaling.++ Returns (image_bytes, display_scale, zoom_bbox).+ display_scale < 1.0 means the image was shrunk; bbox is None if not zoomed.+ """+ from PIL import Image++ result = await _send(provider_type, name, "screenshot", {})+ img_b64 = result.get("image_data") or result.get("data")+ if not img_b64:+ raise RuntimeError(result.get("error", "no image data returned"))+ img = Image.open(io.BytesIO(base64.b64decode(img_b64)))++ zoom_window = state.get("zoom_window")+ zoom_window_id = state.get("zoom_window_id")+ bbox: dict | None = None++ if zoom_window:+ try:+ if zoom_window_id:+ bbox = await _resolve_zoom_bbox_by_id(provider_type, name, str(zoom_window_id))+ if not bbox:+ bbox = await _resolve_zoom_bbox(provider_type, name, zoom_window)+ if bbox:+ img = img.crop(+ (+ bbox["x"],+ bbox["y"],+ bbox["x"] + bbox["width"],+ bbox["y"] + bbox["height"],+ )+ )+ except Exception:+ bbox = None # fall back to full screen++ w, h = img.size+ scale = 1.0+ if max(w, h) > MAX_LENGTH:+ scale = MAX_LENGTH / max(w, h)+ img = img.resize((int(w * scale), int(h * scale)), Image.LANCZOS)++ buf = io.BytesIO()+ img.save(buf, format="PNG")+ return buf.getvalue(), scale, bbox+++def _coords(x: int, y: int, state: dict) -> tuple[int, int]:+ """Translate image-space (x, y) to screen-space coordinates.++ Accounts for:+ - display_scale: image was scaled for output (divide to get original pixels)+ - zoom_bbox: image was cropped, add offset to get screen coords+ """+ zoom_scale: float = state.get("zoom_scale", 1.0) or 1.0+ zoom_bbox: dict | None = state.get("zoom_bbox")++ sx = round(x / zoom_scale)+ sy = round(y / zoom_scale)++ if zoom_bbox:+ sx += zoom_bbox["x"]+ sy += zoom_bbox["y"]++ return sx, sy+++async def _maybe_focus_zoom(provider_type: str, name: str, state: dict) -> None:+ """If a zoom window is tracked, bring it to focus before performing an action."""+ window_id = state.get("zoom_window_id")+ if window_id:+ try:+ await _send(provider_type, name, "activate_window", {"window_id": window_id})+ except Exception:+ pass+++async def _print_context(provider_type: str, name: str, state: dict | None = None) -> None:+ """Print the VM + zoom context line after a command."""+ vm_label = name if name else provider_type+ if state is None:+ state = _load_state()+ zoom_window = state.get("zoom_window")+ zoom_window_id = state.get("zoom_window_id")+ if zoom_window:+ zoom_info = (+ f"zoom: {zoom_window} ({zoom_window_id})" if zoom_window_id else f"zoom: {zoom_window}"+ )+ else:+ zoom_info = "zoom: off"+ print(f"💻 {vm_label}\t🔍 {zoom_info}")+++def _require_target() -> dict | None:+ state = _load_state()+ if not state.get("provider"):+ _fail("No VM selected. Run: cua do switch <provider> [name]")+ return None+ return state+++# ── subcommand handlers ───────────────────────────────────────────────────────+++def _cmd_switch(args: argparse.Namespace) -> int:+ provider = args.provider.lower()+ old_state = _load_state()+ had_zoom = bool(old_state.get("zoom_window"))++ if provider == "host":+ if not _host_consented():+ print(+ "❌ Warning: you are about to allow an AI to control your host PC directly.\n"+ " This grants full keyboard, mouse, and screen access to your local desktop.\n"+ " To continue, please run: cua do-host-consent",+ file=sys.stderr,+ )+ return 1+ _save_state(+ {+ "provider": "host",+ "name": "",+ "zoom_window": None,+ "zoom_window_id": None,+ "zoom_bbox": None,+ "zoom_scale": 1.0,+ }+ )+ msg = "Switched to host (local PC)"+ if had_zoom:+ msg += " — zoom reset"+ return _ok(msg)++ if provider not in PROVIDERS:+ return _fail(f"Unknown provider '{provider}'. Choose from: {', '.join(PROVIDERS)}")++ name = args.name or ""+ _save_state(+ {+ "provider": provider,+ "name": name,+ "zoom_window": None,+ "zoom_window_id": None,+ "zoom_bbox": None,+ "zoom_scale": 1.0,+ }+ )+ label = f"{provider}/{name}" if name else provider+ msg = f"Switched to {label}"+ if had_zoom:+ msg += " — zoom reset"+ return _ok(msg)+++def _cmd_status(args: argparse.Namespace) -> int:+ state = _load_state()+ provider = state.get("provider")+ if not provider:+ return _fail("No VM selected. Run: cua do switch <provider> [name]")+ name = state.get("name", "")+ label = f"{provider}/{name}" if name else provider+ zoom = state.get("zoom_window")+ zoom_info = f" [zoomed to '{zoom}']" if zoom else ""+ return _ok(f"Current target: {label}{zoom_info}")+++def _cmd_ls(args: argparse.Namespace) -> int:+ from cua_cli.utils.async_utils import run_async++ explicit_provider = getattr(args, "provider", None)++ async def _list_one(ptype: str) -> int:+ from computer.providers.base import VMProviderType+ from computer.providers.factory import VMProviderFactory++ if ptype == "host":+ print(" host [local]")+ return 0++ kwargs: dict[str, Any] = {}+ if ptype in ("cloud", "cloudv2"):+ from cua_cli.auth.store import get_api_key++ api_key = get_api_key()+ if not api_key:+ return _fail("Not authenticated. Run 'cua auth login' first")+ kwargs["api_key"] = api_key++ try:+ enum_type = VMProviderType(ptype)+ except ValueError:+ return _fail(f"Unknown provider: {ptype}")++ try:+ provider = VMProviderFactory.create_provider(enum_type, **kwargs)+ async with provider:+ vms = await provider.list_vms()+ except Exception as e:+ return _fail(str(e))++ if not vms:+ print(f"No VMs found for provider '{ptype}'")+ return 0++ for vm in vms:+ print(f" {vm.get('name', '?')} [{vm.get('status', '?')}]")+ return 0++ async def _list_all() -> int:+ from computer.providers.base import VMProviderType+ from computer.providers.factory import VMProviderFactory+ from cua_cli.auth.store import get_api_key++ print(" host [local]")++ api_key = get_api_key()+ if api_key:+ try:+ provider = VMProviderFactory.create_provider(+ VMProviderType.CLOUDV2, api_key=api_key+ )+ async with provider:+ vms = await provider.list_vms()+ for vm in vms:+ print(f" {vm.get('name', '?')} [{vm.get('status', '?')}] cloudv2")+ except Exception:+ pass++ return 0++ if explicit_provider:+ return run_async(_list_one(explicit_provider.lower()))+ else:+ return run_async(_list_all())+++def _cmd_zoom(args: argparse.Namespace) -> int:+ from cua_cli.utils.async_utils import run_async++ t = _require_target()+ if not t:+ return 1+ window_name = args.window_name++ async def _run() -> int:+ state = _load_state()+ p, n = state["provider"], state.get("name", "")+ try:+ wins_r = await _send(p, n, "get_application_windows", {"app": window_name})+ windows = wins_r.get("windows") or wins_r.get("data") or []+ if not windows:+ return _fail(f"No windows found matching '{window_name}'")+ # Fetch titles and filter out internal helper windows+ candidates = []+ for wid in windows:+ name_r = await _send(p, n, "get_window_name", {"window_id": wid})+ title = name_r.get("name") or str(wid)+ if title not in _SKIP_WINDOW_TITLES:+ candidates.append((wid, title))+ if not candidates:+ return _fail(f"No windows found matching '{window_name}'")+ if len(candidates) > 1:+ labels = [f"{wid} ({title})" for wid, title in candidates]+ return _fail(+ f"Multiple windows matched '{window_name}' — be more specific. "+ f"Found: {', '.join(labels)}"+ )+ wid, matched_title = candidates[0]+ except Exception:+ wid, matched_title = None, window_name # resolution failed, still set zoom++ _update_state(zoom_window=window_name, zoom_window_id=wid, zoom_bbox=None, zoom_scale=1.0)+ id_suffix = f" ({wid})" if wid else ""+ return _ok(f"Zoomed to '{matched_title}'{id_suffix}")++ return run_async(_run())+++def _cmd_unzoom(args: argparse.Namespace) -> int:+ _update_state(zoom_window=None, zoom_window_id=None, zoom_bbox=None, zoom_scale=1.0)+ return _ok("Unzoomed — full screen")+++def _cmd_screenshot(args: argparse.Namespace) -> int:+ from cua_cli.utils.async_utils import run_async++ t = _require_target()+ if not t:+ return 1++ async def _run() -> int:+ state = _load_state()+ try:+ img_bytes, scale, bbox = await _take_screenshot_data(+ state["provider"], state.get("name", ""), state+ )+ except Exception as e:+ await _print_context(state["provider"], state.get("name", ""), state)+ return _fail(str(e))++ # Persist zoom state for coordinate translation+ _update_state(zoom_scale=scale, zoom_bbox=bbox)++ save_path = getattr(args, "save", None)+ if not save_path:+ ts = datetime.now().strftime("%Y%m%d_%H%M%S")+ save_path = f"{tempfile.gettempdir()}/cua_screenshot_{ts}.png"++ with open(save_path, "wb") as f:+ f.write(img_bytes)++ rc = _ok(f"screenshot saved to {save_path}")+ await _print_context(state["provider"], state.get("name", ""), state)+ return rc++ return run_async(_run())+++def _cmd_snapshot(args: argparse.Namespace) -> int:+ from cua_cli.utils.async_utils import run_async++ t = _require_target()+ if not t:+ return 1++ extra = " ".join(args.instructions) if args.instructions else ""++ async def _run() -> int:+ state = _load_state()+ try:+ img_bytes, scale, bbox = await _take_screenshot_data(+ state["provider"], state.get("name", ""), state+ )+ except Exception as e:+ await _print_context(state["provider"], state.get("name", ""), state)+ return _fail(str(e))++ _update_state(zoom_scale=scale, zoom_bbox=bbox)++ ts = datetime.now().strftime("%Y%m%d_%H%M%S")+ save_path = f"{tempfile.gettempdir()}/cua_snapshot_{ts}.png"+ with open(save_path, "wb") as f:+ f.write(img_bytes)++ api_key = os.environ.get("ANTHROPIC_API_KEY")+ if not api_key:+ await _print_context(state["provider"], state.get("name", ""), state)+ return _fail("ANTHROPIC_API_KEY not set")++ try:+ import anthropic as _anthropic++ client = _anthropic.Anthropic(api_key=api_key)+ prompt = (+ "You are analyzing a screenshot for an AI agent.\n"+ "1. Write a 1-2 sentence summary of what is currently on screen.\n"+ "2. List every interactive element visible (buttons, links, inputs, "+ "menus, checkboxes, dropdowns, etc.) with its center coordinates "+ "in image pixels (origin = top-left). Be precise.\n\n"+ "Respond in this exact JSON format:\n"+ '{"summary": "...", "elements": [{"name": "...", "type": "...", "x": N, "y": N}, ...]}\n'+ )+ if extra:+ prompt += f"\nAdditional instructions: {extra}"++ img_b64 = base64.b64encode(img_bytes).decode()+ response = client.messages.create(+ model="claude-haiku-4-5-20251001",+ max_tokens=1024,+ messages=[+ {+ "role": "user",+ "content": [+ {+ "type": "image",+ "source": {+ "type": "base64",+ "media_type": "image/png",+ "data": img_b64,+ },+ },+ {"type": "text", "text": prompt},+ ],+ }+ ],+ )++ raw = response.content[0].text.strip()+ # Try to parse JSON; fall back to raw text+ try:+ parsed = json.loads(raw)+ summary = parsed.get("summary", "")+ elements = parsed.get("elements", [])+ print(f"✅ snapshot — {save_path}")+ print()+ print(summary)+ if elements:+ print()+ print("Interactive elements:")+ for el in elements:+ print(+ f" • {el.get('name','?')} [{el.get('type','?')}] ({el.get('x','?')}, {el.get('y','?')})"+ )+ except json.JSONDecodeError:+ print(f"✅ snapshot — {save_path}")+ print()+ print(raw)++ except ImportError:+ await _print_context(state["provider"], state.get("name", ""), state)+ return _fail("'anthropic' package not installed. Run: pip install anthropic")+ except Exception as e:+ await _print_context(state["provider"], state.get("name", ""), state)+ return _fail(f"AI analysis failed: {e}")++ await _print_context(state["provider"], state.get("name", ""), state)+ return 0++ return run_async(_run())+++def _cmd_click(args: argparse.Namespace) -> int:+ from cua_cli.utils.async_utils import run_async++ t = _require_target()+ if not t:+ return 1++ button = getattr(args, "button", "left") or "left"+ cmd_map = {"left": "left_click", "right": "right_click", "middle": "middle_click"}+ cmd = cmd_map.get(button, "left_click")++ async def _run() -> int:+ state = _load_state()+ p, n = state["provider"], state.get("name", "")+ await _maybe_focus_zoom(p, n, state)+ sx, sy = _coords(args.x, args.y, state)+ try:+ result = await _send(p, n, cmd, {"x": sx, "y": sy})+ except Exception as e:+ await _print_context(p, n, state)+ return _fail(str(e))+ if not result.get("success", True):+ await _print_context(p, n, state)+ return _fail(result.get("error", "click failed"))+ rc = _ok(f"clicked ({args.x}, {args.y}) [{button}]")+ await _print_context(p, n, state)+ return rc++ return run_async(_run())+++def _cmd_dclick(args: argparse.Namespace) -> int:+ from cua_cli.utils.async_utils import run_async++ t = _require_target()+ if not t:+ return 1++ async def _run() -> int:+ state = _load_state()+ p, n = state["provider"], state.get("name", "")+ await _maybe_focus_zoom(p, n, state)+ sx, sy = _coords(args.x, args.y, state)+ try:+ result = await _send(p, n, "double_click", {"x": sx, "y": sy})+ except Exception as e:+ await _print_context(p, n, state)+ return _fail(str(e))+ if not result.get("success", True):+ await _print_context(p, n, state)+ return _fail(result.get("error", "double-click failed"))+ rc = _ok(f"double-clicked ({args.x}, {args.y})")+ await _print_context(p, n, state)+ return rc++ return run_async(_run())+++def _cmd_move(args: argparse.Namespace) -> int:+ from cua_cli.utils.async_utils import run_async++ t = _require_target()+ if not t:+ return 1++ async def _run() -> int:+ state = _load_state()+ p, n = state["provider"], state.get("name", "")+ await _maybe_focus_zoom(p, n, state)+ sx, sy = _coords(args.x, args.y, state)+ try:+ result = await _send(p, n, "move_cursor", {"x": sx, "y": sy})+ except Exception as e:+ await _print_context(p, n, state)+ return _fail(str(e))+ if not result.get("success", True):+ await _print_context(p, n, state)+ return _fail(result.get("error", "move failed"))+ rc = _ok(f"cursor moved to ({args.x}, {args.y})")+ await _print_context(p, n, state)+ return rc++ return run_async(_run())+++def _cmd_type(args: argparse.Namespace) -> int:+ from cua_cli.utils.async_utils import run_async++ t = _require_target()+ if not t:+ return 1++ async def _run() -> int:+ state = _load_state()+ p, n = state["provider"], state.get("name", "")+ await _maybe_focus_zoom(p, n, state)+ try:+ result = await _send(p, n, "type_text", {"text": args.text})+ except Exception as e:+ await _print_context(p, n, state)+ return _fail(str(e))+ if not result.get("success", True):+ await _print_context(p, n, state)+ return _fail(result.get("error", "type failed"))+ preview = args.text[:40] + ("…" if len(args.text) > 40 else "")+ rc = _ok(f"typed: {preview!r}")+ await _print_context(p, n, state)+ return rc++ return run_async(_run())+++def _cmd_key(args: argparse.Namespace) -> int:+ from cua_cli.utils.async_utils import run_async++ t = _require_target()+ if not t:+ return 1++ async def _run() -> int:+ state = _load_state()+ p, n = state["provider"], state.get("name", "")+ await _maybe_focus_zoom(p, n, state)+ try:+ result = await _send(p, n, "press_key", {"key": args.key})+ except Exception as e:+ await _print_context(p, n, state)+ return _fail(str(e))+ if not result.get("success", True):+ await _print_context(p, n, state)+ return _fail(result.get("error", "key press failed"))+ rc = _ok(f"pressed key: {args.key}")+ await _print_context(p, n, state)+ return rc++ return run_async(_run())+++def _cmd_hotkey(args: argparse.Namespace) -> int:+ from cua_cli.utils.async_utils import run_async++ t = _require_target()+ if not t:+ return 1++ keys = [k.strip() for k in args.keys.replace("-", "+").split("+") if k.strip()]++ async def _run() -> int:+ state = _load_state()+ p, n = state["provider"], state.get("name", "")+ await _maybe_focus_zoom(p, n, state)+ try:+ result = await _send(p, n, "hotkey", {"keys": keys})+ except Exception as e:+ await _print_context(p, n, state)+ return _fail(str(e))+ if not result.get("success", True):+ await _print_context(p, n, state)+ return _fail(result.get("error", "hotkey failed"))+ rc = _ok(f"hotkey: {'+'.join(keys)}")+ await _print_context(p, n, state)+ return rc++ return run_async(_run())+++def _cmd_scroll(args: argparse.Namespace) -> int:+ from cua_cli.utils.async_utils import run_async++ t = _require_target()+ if not t:+ return 1++ async def _run() -> int:+ state = _load_state()+ p, n = state["provider"], state.get("name", "")+ await _maybe_focus_zoom(p, n, state)+ try:+ result = await _send(+ p, n, "scroll_direction", {"direction": args.direction, "clicks": args.amount}+ )+ except Exception as e:+ await _print_context(p, n, state)+ return _fail(str(e))+ if not result.get("success", True):+ await _print_context(p, n, state)+ return _fail(result.get("error", "scroll failed"))+ rc = _ok(f"scrolled {args.direction} {args.amount}x")+ await _print_context(p, n, state)+ return rc++ return run_async(_run())+++def _cmd_drag(args: argparse.Namespace) -> int:+ from cua_cli.utils.async_utils import run_async++ t = _require_target()+ if not t:+ return 1++ async def _run() -> int:+ state = _load_state()+ p, n = state["provider"], state.get("name", "")+ await _maybe_focus_zoom(p, n, state)+ sx1, sy1 = _coords(args.x1, args.y1, state)+ sx2, sy2 = _coords(args.x2, args.y2, state)+ try:+ result = await _send(+ p,+ n,+ "drag_to",+ {"start_x": sx1, "start_y": sy1, "end_x": sx2, "end_y": sy2},+ )+ except Exception as e:+ await _print_context(p, n, state)+ return _fail(str(e))+ if not result.get("success", True):+ await _print_context(p, n, state)+ return _fail(result.get("error", "drag failed"))+ rc = _ok(f"dragged ({args.x1},{args.y1}) → ({args.x2},{args.y2})")+ await _print_context(p, n, state)+ return rc++ return run_async(_run())+++def _cmd_shell(args: argparse.Namespace) -> int:+ from cua_cli.utils.async_utils import run_async++ t = _require_target()+ if not t:+ return 1++ command = " ".join(args.command)+ if not command.strip():+ return _fail("No command provided")++ async def _run() -> int:+ state = _load_state()+ try:+ result = await _send(+ state["provider"], state.get("name", ""), "run_command", {"command": command}+ )+ except Exception as e:+ await _print_context(state["provider"], state.get("name", ""), state)+ return _fail(str(e))+ if not result.get("success", True):+ await _print_context(state["provider"], state.get("name", ""), state)+ return _fail(result.get("error", "shell command failed"))+ stdout = result.get("stdout", "").strip()+ rc_code = result.get("returncode", 0)+ if rc_code != 0:+ stderr = result.get("stderr", "").strip()+ await _print_context(state["provider"], state.get("name", ""), state)+ return _fail(f"exit {rc_code}: {stderr or stdout}")+ preview = (stdout[:80] + "…") if len(stdout) > 80 else stdout+ rc = _ok(preview if preview else "done")+ await _print_context(state["provider"], state.get("name", ""), state)+ return rc++ return run_async(_run())+++def _cmd_open(args: argparse.Namespace) -> int:+ from cua_cli.utils.async_utils import run_async++ t = _require_target()+ if not t:+ return 1++ async def _run() -> int:+ state = _load_state()+ try:+ result = await _send(+ state["provider"], state.get("name", ""), "open", {"path": args.path}+ )+ except Exception as e:+ await _print_context(state["provider"], state.get("name", ""), state)+ return _fail(str(e))+ if not result.get("success", True):+ await _print_context(state["provider"], state.get("name", ""), state)+ return _fail(result.get("error", "open failed"))+ rc = _ok(f"opened: {args.path}")+ await _print_context(state["provider"], state.get("name", ""), state)+ return rc++ return run_async(_run())+++# ── window subcommands ────────────────────────────────────────────────────────+++def _cmd_window(args: argparse.Namespace) -> int:+ from cua_cli.utils.async_utils import run_async++ t = _require_target()+ if not t:+ return 1++ action = args.window_action++ async def _run() -> int:+ state = _load_state()+ p, n = state["provider"], state.get("name", "")++ if action == "ls":+ app = getattr(args, "app", "") or ""+ try:+ result = await _send(p, n, "get_application_windows", {"app": app})+ except Exception as e:+ return _fail(str(e))+ windows = result.get("windows") or result.get("data") or []+ if not isinstance(windows, list):+ print(result)+ return 0+ for wid in windows:+ try:+ name_r = await _send(p, n, "get_window_name", {"window_id": wid})+ title = name_r.get("name") or "?"+ if title in _SKIP_WINDOW_TITLES:+ continue+ pos_r = await _send(p, n, "get_window_position", {"window_id": wid})+ size_r = await _send(p, n, "get_window_size", {"window_id": wid})+ pos = pos_r.get("position") or pos_r.get("data")+ size = size_r.get("size") or size_r.get("data")+ if pos and size:+ if isinstance(pos, (list, tuple)):+ x, y = int(pos[0]), int(pos[1])+ else:+ x, y = int(pos.get("x", 0)), int(pos.get("y", 0))+ if isinstance(size, (list, tuple)):+ w, h = int(size[0]), int(size[1])+ else:+ w, h = int(size.get("width", 0)), int(size.get("height", 0))+ bbox = f"{x},{y},{w},{h}"+ else:+ bbox = "?"+ print(f" {wid} {title} [{bbox}]")+ except Exception:+ print(f" {wid}")+ return 0++ if action == "unfocus":+ try:+ result = await _send(p, n, "deactivate_window", {})+ except Exception:+ result = {"success": False}+ if not result.get("success", True):+ try:+ result = await _send(p, n, "press_key", {"key": "escape"})+ except Exception as e:+ await _print_context(p, n)+ return _fail(str(e))+ rc = _ok("unfocused window")+ await _print_context(p, n)+ return rc++ # Commands requiring window_id+ wid = args.window_id++ cmd_map = {+ "focus": ("activate_window", {"window_id": wid}),+ "activate": ("activate_window", {"window_id": wid}),+ "minimize": ("minimize_window", {"window_id": wid}),+ "maximize": ("maximize_window", {"window_id": wid}),+ "close": ("close_window", {"window_id": wid}),+ }++ if action in cmd_map:+ cmd, params = cmd_map[action]+ try:+ result = await _send(p, n, cmd, params)+ except Exception as e:+ await _print_context(p, n)+ return _fail(str(e))+ if not result.get("success", True):+ await _print_context(p, n)+ return _fail(result.get("error", f"{action} failed"))+ rc = _ok(f"{action}d window {wid}")+ await _print_context(p, n)+ return rc++ if action == "resize":+ try:+ result = await _send(+ p,+ n,+ "set_window_size",+ {"window_id": wid, "width": args.width, "height": args.height},+ )+ except Exception as e:+ await _print_context(p, n)+ return _fail(str(e))+ if not result.get("success", True):+ await _print_context(p, n)+ return _fail(result.get("error", "resize failed"))+ rc = _ok(f"resized window {wid} to {args.width}x{args.height}")+ await _print_context(p, n)+ return rc++ if action == "move":+ try:+ result = await _send(+ p,+ n,+ "set_window_position",+ {"window_id": wid, "x": args.x, "y": args.y},+ )+ except Exception as e:+ await _print_context(p, n)+ return _fail(str(e))+ if not result.get("success", True):+ await _print_context(p, n)+ return _fail(result.get("error", "move failed"))+ rc = _ok(f"moved window {wid} to ({args.x},{args.y})")+ await _print_context(p, n)+ return rc++ if action == "info":+ try:+ size_r = await _send(p, n, "get_window_size", {"window_id": wid})+ pos_r = await _send(p, n, "get_window_position", {"window_id": wid})+ name_r = await _send(p, n, "get_window_name", {"window_id": wid})+ except Exception as e:+ return _fail(str(e))+ print(+ json.dumps(+ {+ "id": wid,+ "title": name_r.get("name") or name_r.get("data") or "",+ "size": size_r.get("size") or size_r.get("data"),+ "position": pos_r.get("position") or pos_r.get("data"),+ },+ indent=2,+ )+ )+ return 0++ return _fail(f"Unknown window action: {action}")++ return run_async(_run())+++# ── host consent command (registered separately as cua do-host-consent) ───────+++def register_host_consent_parser(subparsers: argparse._SubParsersAction) -> None:+ subparsers.add_parser(+ "do-host-consent",+ help="Grant consent to control the host PC with cua do",+ description=(+ "Grants permission for 'cua do switch host', allowing AI agents to control\n"+ "your local keyboard, mouse, and screen via a local computer-server."+ ),+ )+++def execute_host_consent(args: argparse.Namespace) -> int:+ _HOST_CONSENT_FILE.parent.mkdir(parents=True, exist_ok=True)+ _HOST_CONSENT_FILE.write_text("consented")+ _update_state(provider="host", name="")+ return _ok("consented — target switched to host")+++# ── parser registration ───────────────────────────────────────────────────────+++def register_parser(subparsers: argparse._SubParsersAction) -> None:+ p = subparsers.add_parser(+ "do",+ help="Send one-shot automation commands to a VM",+ description=(+ "Send automation commands to a target VM. "+ "Output is a single line starting with ✅ or ❌.\n"+ "A second line shows the current VM + focused window context.\n"+ "Coordinates are in screenshot-image space; zoom/max-length translation is applied automatically."+ ),+ formatter_class=argparse.RawDescriptionHelpFormatter,+ epilog="""+Examples:+ cua do switch docker my-container Select target VM+ cua do switch host Control the local PC (requires consent)+ cua do-host-consent Grant consent for host control+ cua do ls List VMs for current provider+ cua do ls docker+ cua do status+ cua do zoom "Chrome" Crop all screenshots to Chrome window+ cua do unzoom+ cua do screenshot+ cua do screenshot --save /tmp/s.png+ cua do snapshot Screenshot + AI summary (needs ANTHROPIC_API_KEY)+ cua do snapshot "focus on the form"+ cua do click 100 200+ cua do click 100 200 right+ cua do dclick 100 200+ cua do move 500 300+ cua do type "hello world"+ cua do key enter+ cua do hotkey cmd+c+ cua do scroll down 5+ cua do drag 100 200 400 500+ cua do shell "ls -la"+ cua do open https://example.com+ cua do window ls+ cua do window ls Terminal+ cua do window focus <id>+ cua do window unfocus+ cua do window minimize/maximize/close <id>+ cua do window resize <id> 1280 800+ cua do window move <id> 0 0+ cua do window info <id>+""",+ )++ sub = p.add_subparsers(dest="do_action", metavar="action")+ sub.required = True++ # switch+ sw = sub.add_parser("switch", help="Select target VM (use 'host' for local PC)")+ sw.add_argument("provider", choices=PROVIDERS)+ sw.add_argument("name", nargs="?", default="")++ # status+ sub.add_parser("status", help="Show current target and zoom state")++ # ls+ ls = sub.add_parser("ls", help="List VMs for a provider")+ ls.add_argument("provider", nargs="?", choices=PROVIDERS)++ # zoom / unzoom+ zm = sub.add_parser("zoom", help="Crop screenshots + translate coords to a window")+ zm.add_argument("window_name", help="App/window name to zoom to")+ sub.add_parser("unzoom", help="Return to full-screen screenshots")++ # screenshot+ ss = sub.add_parser("screenshot", help="Take a screenshot (saved to temp dir)")+ ss.add_argument("--save", "-s", metavar="PATH")++ # snapshot+ sn = sub.add_parser(+ "snapshot",+ help="Screenshot + AI screen summary and interactive elements (needs ANTHROPIC_API_KEY)",+ )+ sn.add_argument("instructions", nargs="*", help="Extra instructions for the AI")++ # click+ cl = sub.add_parser("click", help="Click at coordinates (image space)")+ cl.add_argument("x", type=int)+ cl.add_argument("y", type=int)+ cl.add_argument("button", nargs="?", default="left", choices=["left", "right", "middle"])++ # dclick+ dc = sub.add_parser("dclick", help="Double-click at coordinates (image space)")+ dc.add_argument("x", type=int)+ dc.add_argument("y", type=int)++ # move+ mv = sub.add_parser("move", help="Move cursor (image space)")+ mv.add_argument("x", type=int)+ mv.add_argument("y", type=int)++ # type+ ty = sub.add_parser("type", help="Type text")+ ty.add_argument("text")++ # key+ ky = sub.add_parser("key", help="Press a key (enter, escape, tab, …)")+ ky.add_argument("key")++ # hotkey+ hk = sub.add_parser("hotkey", help="Keyboard shortcut (e.g. cmd+c, ctrl+shift+s)")+ hk.add_argument("keys")++ # scroll+ sc = sub.add_parser("scroll", help="Scroll in a direction")+ sc.add_argument("direction", choices=["up", "down", "left", "right"])+ sc.add_argument("amount", nargs="?", type=int, default=3)++ # drag+ dr = sub.add_parser("drag", help="Drag (image space)")+ dr.add_argument("x1", type=int)+ dr.add_argument("y1", type=int)+ dr.add_argument("x2", type=int)+ dr.add_argument("y2", type=int)++ # shell+ sh = sub.add_parser("shell", help="Run a shell command in the VM")+ sh.add_argument("command", nargs=argparse.REMAINDER)++ # open+ op = sub.add_parser("open", help="Open a file or URL")+ op.add_argument("path")++ # window+ win = sub.add_parser("window", help="Window management")+ win_sub = win.add_subparsers(dest="window_action", metavar="action")+ win_sub.required = True++ wls = win_sub.add_parser("ls", help="List windows")+ wls.add_argument("app", nargs="?", default="")++ win_sub.add_parser("unfocus", help="Remove focus from the current window")++ for act in ("focus", "activate", "minimize", "maximize", "close"):+ wp = win_sub.add_parser(act)+ wp.add_argument("window_id")++ wr = win_sub.add_parser("resize")+ wr.add_argument("window_id")+ wr.add_argument("width", type=int)+ wr.add_argument("height", type=int)++ wm = win_sub.add_parser("move")+ wm.add_argument("window_id")+ wm.add_argument("x", type=int)+ wm.add_argument("y", type=int)++ wi = win_sub.add_parser("info")+ wi.add_argument("window_id")+++# ── dispatch ──────────────────────────────────────────────────────────────────+++def execute(args: argparse.Namespace) -> int:+ dispatch = {+ "switch": _cmd_switch,+ "status": _cmd_status,+ "ls": _cmd_ls,+ "zoom": _cmd_zoom,+ "unzoom": _cmd_unzoom,+ "screenshot": _cmd_screenshot,+ "snapshot": _cmd_snapshot,+ "click": _cmd_click,+ "dclick": _cmd_dclick,+ "move": _cmd_move,+ "type": _cmd_type,+ "key": _cmd_key,+ "hotkey": _cmd_hotkey,+ "scroll": _cmd_scroll,+ "drag": _cmd_drag,+ "shell": _cmd_shell,+ "open": _cmd_open,+ "window": _cmd_window,+ }+ handler = dispatch.get(args.do_action)+ if not handler:+ return _fail(f"Unknown action: {args.do_action}")+ return handler(args)diff --git a/libs/python/cua-cli/cua_cli/commands/local_image.py b/libs/python/cua-cli/cua_cli/commands/local_image.pyindex abb938b125..42997e459a 100644--- a/libs/python/cua-cli/cua_cli/commands/local_image.py+++ b/libs/python/cua-cli/cua_cli/commands/local_image.py@@ -201,7 +201,9 @@ def cmd_local_list(args: argparse.Namespace) -> int:style = "green" if "ready" in status else "red"- console.print(f"{name:<20} {platform:<15} {size:<10} {created:<12} [{style}]{status}[/{style}]")+ console.print(+ f"{name:<20} {platform:<15} {size:<10} {created:<12} [{style}]{status}[/{style}]"+ )print("\n" + "=" * 85)print("\nCommands:")diff --git a/libs/python/cua-cli/cua_cli/commands/mcp.py b/libs/python/cua-cli/cua_cli/commands/mcp.pyindex 3cf300a69d..8e3ca017da 100644--- a/libs/python/cua-cli/cua_cli/commands/mcp.py+++ b/libs/python/cua-cli/cua_cli/commands/mcp.py@@ -706,6 +706,280 @@ async def _register_computer_tools(result = await _send_command(sandbox, "open", {"path": path})return json.dumps(result)+ @server.tool()+ async def computer_window_focus(ctx: Context, window_id: str, sandbox: str = "") -> str:+ """Bring a window to the foreground and focus it.++ Args:+ window_id: Window identifier+ sandbox: Sandbox name (optional)+ """+ result = await _send_command(sandbox, "activate_window", {"window_id": window_id})+ return json.dumps(result)++ @server.tool()+ async def computer_window_unfocus(ctx: Context, sandbox: str = "") -> str:+ """Remove focus from the currently focused window.++ Args:+ sandbox: Sandbox name (optional)+ """+ result = await _send_command(sandbox, "deactivate_window", {})+ if not result.get("success"):+ # Fallback: press Escape+ result = await _send_command(sandbox, "press_key", {"key": "escape"})+ return json.dumps(result)++ @server.tool()+ async def computer_window_minimize(ctx: Context, window_id: str, sandbox: str = "") -> str:+ """Minimize a window.++ Args:+ window_id: Window identifier+ sandbox: Sandbox name (optional)+ """+ result = await _send_command(sandbox, "minimize_window", {"window_id": window_id})+ return json.dumps(result)++ @server.tool()+ async def computer_window_maximize(ctx: Context, window_id: str, sandbox: str = "") -> str:+ """Maximize a window.++ Args:+ window_id: Window identifier+ sandbox: Sandbox name (optional)+ """+ result = await _send_command(sandbox, "maximize_window", {"window_id": window_id})+ return json.dumps(result)++ @server.tool()+ async def computer_window_close(ctx: Context, window_id: str, sandbox: str = "") -> str:+ """Close a window.++ Args:+ window_id: Window identifier+ sandbox: Sandbox name (optional)+ """+ result = await _send_command(sandbox, "close_window", {"window_id": window_id})+ return json.dumps(result)++ @server.tool()+ async def computer_window_resize(+ ctx: Context,+ window_id: str,+ width: int,+ height: int,+ sandbox: str = "",+ ) -> str:+ """Resize a window.++ Args:+ window_id: Window identifier+ width: New width in pixels+ height: New height in pixels+ sandbox: Sandbox name (optional)+ """+ result = await _send_command(+ sandbox,+ "set_window_size",+ {"window_id": window_id, "width": width, "height": height},+ )+ return json.dumps(result)++ @server.tool()+ async def computer_window_move(+ ctx: Context,+ window_id: str,+ x: int,+ y: int,+ sandbox: str = "",+ ) -> str:+ """Move a window to a position on screen.++ Args:+ window_id: Window identifier+ x: X coordinate for the window's top-left corner+ y: Y coordinate for the window's top-left corner+ sandbox: Sandbox name (optional)+ """+ result = await _send_command(+ sandbox,+ "set_window_position",+ {"window_id": window_id, "x": x, "y": y},+ )+ return json.dumps(result)++ @server.tool()+ async def computer_window_get_info(+ ctx: Context,+ window_id: str,+ sandbox: str = "",+ ) -> str:+ """Get a window's title, size, and position.++ Args:+ window_id: Window identifier+ sandbox: Sandbox name (optional)+ """+ name_r = await _send_command(sandbox, "get_window_name", {"window_id": window_id})+ size_r = await _send_command(sandbox, "get_window_size", {"window_id": window_id})+ pos_r = await _send_command(sandbox, "get_window_position", {"window_id": window_id})+ return json.dumps(+ {+ "window_id": window_id,+ "title": name_r.get("name") or name_r.get("data"),+ "size": size_r.get("size") or size_r.get("data"),+ "position": pos_r.get("position") or pos_r.get("data"),+ },+ indent=2,+ )++ @server.tool()+ async def computer_launch(+ ctx: Context,+ app: str,+ args: list[str] | None = None,+ sandbox: str = "",+ ) -> str:+ """Launch an application.++ Args:+ app: Application executable or bundle identifier+ args: Optional list of arguments+ sandbox: Sandbox name (optional)+ """+ result = await _send_command(sandbox, "launch", {"app": app, "args": args or []})+ return json.dumps(result)++ if Permission.COMPUTER_CLICK in permissions:++ @server.tool()+ async def computer_move_cursor(+ ctx: Context,+ x: int,+ y: int,+ sandbox: str = "",+ ) -> str:+ """Move the mouse cursor without clicking.++ Args:+ x: X coordinate+ y: Y coordinate+ sandbox: Sandbox name (optional)+ """+ result = await _send_command(sandbox, "move_cursor", {"x": x, "y": y})+ return json.dumps(result)++ @server.tool()+ async def computer_mouse_down(+ ctx: Context,+ x: int,+ y: int,+ button: str = "left",+ sandbox: str = "",+ ) -> str:+ """Press and hold a mouse button.++ Args:+ x: X coordinate+ y: Y coordinate+ button: Mouse button (left, right, middle)+ sandbox: Sandbox name (optional)+ """+ result = await _send_command(sandbox, "mouse_down", {"x": x, "y": y, "button": button})+ return json.dumps(result)++ @server.tool()+ async def computer_mouse_up(+ ctx: Context,+ x: int,+ y: int,+ button: str = "left",+ sandbox: str = "",+ ) -> str:+ """Release a mouse button.++ Args:+ x: X coordinate+ y: Y coordinate+ button: Mouse button (left, right, middle)+ sandbox: Sandbox name (optional)+ """+ result = await _send_command(sandbox, "mouse_up", {"x": x, "y": y, "button": button})+ return json.dumps(result)++ if Permission.COMPUTER_KEY in permissions:++ @server.tool()+ async def computer_key_down(ctx: Context, key: str, sandbox: str = "") -> str:+ """Press and hold a key.++ Args:+ key: Key to hold (e.g. "shift", "ctrl", "a")+ sandbox: Sandbox name (optional)+ """+ result = await _send_command(sandbox, "key_down", {"key": key})+ return json.dumps(result)++ @server.tool()+ async def computer_key_up(ctx: Context, key: str, sandbox: str = "") -> str:+ """Release a previously held key.++ Args:+ key: Key to release+ sandbox: Sandbox name (optional)+ """+ result = await _send_command(sandbox, "key_up", {"key": key})+ return json.dumps(result)++ if Permission.COMPUTER_SCREENSHOT in permissions:++ @server.tool()+ async def computer_get_screen_size(ctx: Context, sandbox: str = "") -> str:+ """Get the screen dimensions.++ Args:+ sandbox: Sandbox name (optional)+ """+ result = await _send_command(sandbox, "get_screen_size", {})+ return json.dumps(result)++ @server.tool()+ async def computer_get_cursor_position(ctx: Context, sandbox: str = "") -> str:+ """Get the current cursor position.++ Args:+ sandbox: Sandbox name (optional)+ """+ result = await _send_command(sandbox, "get_cursor_position", {})+ return json.dumps(result)++ @server.tool()+ async def computer_get_accessibility_tree(ctx: Context, sandbox: str = "") -> str:+ """Get the accessibility tree of the current screen.++ Args:+ sandbox: Sandbox name (optional)+ """+ result = await _send_command(sandbox, "get_accessibility_tree", {})+ return json.dumps(result)++ @server.tool()+ async def computer_get_current_window(ctx: Context, sandbox: str = "") -> str:+ """Get the currently focused window ID and title.++ Args:+ sandbox: Sandbox name (optional)+ """+ win_r = await _send_command(sandbox, "get_current_window_id", {})+ window_id = win_r.get("window_id") or win_r.get("data")+ if window_id:+ name_r = await _send_command(sandbox, "get_window_name", {"window_id": window_id})+ title = name_r.get("name") or name_r.get("data") or ""+ else:+ title = ""+ return json.dumps({"window_id": window_id, "title": title or "Desktop"})+async def _register_skills_tools(server: "FastMCP", permissions: set[Permission]) -> None:"""Register skills management tools."""diff --git a/libs/python/cua-cli/cua_cli/main.py b/libs/python/cua-cli/cua_cli/main.pyindex 2e290ca6e7..ebf8528044 100644--- a/libs/python/cua-cli/cua_cli/main.py+++ b/libs/python/cua-cli/cua_cli/main.py@@ -5,7 +5,7 @@ import loggingimport sysfrom cua_cli import __version__-from cua_cli.commands import auth, image, mcp, platform, sandbox, skills+from cua_cli.commands import auth, do, image, mcp, platform, sandbox, skillsfrom cua_cli.utils.output import print_error@@ -27,6 +27,10 @@ Examples:cua image create linux-docker Create a local imagecua image shell <name> Interactive shell into imagecua platform list Show available platforms+ cua do switch docker my-ct Select automation target VM+ cua do screenshot Take a screenshot+ cua do click 100 200 Click at coordinates+ cua do type "hello" Type textFor more information, visit https://docs.trycua.com""",@@ -48,6 +52,8 @@ For more information, visit https://docs.trycua.complatform.register_parser(subparsers)skills.register_parser(subparsers)mcp.register_parser(subparsers)+ do.register_parser(subparsers)+ do.register_host_consent_parser(subparsers)return parser@@ -81,6 +87,10 @@ def main() -> int:return skills.execute(args)elif args.command == "serve-mcp":return mcp.execute(args)+ elif args.command == "do":+ return do.execute(args)+ elif args.command == "do-host-consent":+ return do.execute_host_consent(args)else:print_error(f"Unknown command: {args.command}")return 1diff --git a/libs/python/cua-cli/pyproject.toml b/libs/python/cua-cli/pyproject.tomlindex 4dd9fda2a3..f4b141fc32 100644--- a/libs/python/cua-cli/pyproject.toml+++ b/libs/python/cua-cli/pyproject.toml@@ -24,12 +24,14 @@ classifiers = ["Programming Language :: Python :: 3.13","Environment :: Console",]-requires-python = ">=3.11,<3.14"+requires-python = ">=3.12,<3.14"dependencies = [# Core CUA packages- "cua-computer>=0.4.0",+ "cua-computer>=0.5.0","cua-core>=0.1.0",+ # Host automation (used by cua do switch host)+ "cua-auto>=0.1.0",# HTTP client"aiohttp>=3.9.0",# CLI outputdiff --git a/skills/cua-do-cli/SKILL.md b/skills/cua-do-cli/SKILL.mdnew file mode 100644index 0000000000..a4232b0f8e--- /dev/null+++ b/skills/cua-do-cli/SKILL.md@@ -0,0 +1,127 @@+---+name: cua-do-cli+description: Automates interactions with remote VMs, local Docker containers, cloud sandboxes, and the host PC via cua do. Use when navigating GUIs, taking snapshots/screenshots, clicking, typing, scrolling, running shell commands, or managing windows on a target machine.+---++# VM Automation with cua do++One command = one action = one ✅/❌ line. Switch targets once, then operate.++## Start Here++1. Pick a flow:+ - API key available: switch → snapshot → interact → snapshot+ - No API key: switch → screenshot → read image → interact → screenshot+ - Host control: consent → switch host → operate+2. Run the canonical flow below.+3. Check the command skeleton only if blocked.++## Decision Map++- No target set: `switch` → then operate+- **ANTHROPIC_API_KEY set** (normal): `switch` → `snapshot` → `click/type/key` → repeat snapshot after each UI change+- **No API key**: `switch` → `screenshot` → read image to determine coords → `click/type/key` → repeat screenshot+- Zoom to one window: `zoom "App Name"` → operate → `unzoom`+- Run shell: `shell "command"`+- Control host PC: `do-host-consent` (once) → `switch host` → operate++## Canonical Flows++### 1) Normal (ANTHROPIC_API_KEY set)++```bash+cua do switch docker my-container+cua do snapshot # AI summary + interactive elements with coords+cua do click 450 300+cua do type "hello"+cua do key enter+cua do snapshot # re-snapshot after UI change+```++### 2) No API Key++```bash+cua do switch docker my-container+cua do screenshot # save path printed; read the image to find coords+cua do click 450 300+cua do type "hello"+cua do key enter+cua do screenshot # re-screenshot after UI change+```++### 3) Focused Window (Zoom)++```bash+cua do zoom "Chrome"+cua do snapshot # cropped to Chrome; coords are window-relative+cua do click 120 80+cua do unzoom+```++### 4) Host PC Control++```bash+cua do-host-consent # one-time consent, no prompt+cua do switch host+cua do snapshot+cua do click 100 200+```++## Command Skeleton++### Target++```bash+cua do switch <provider> [name] # cloud, docker, lume, lumier, winsandbox, host+cua do status+cua do ls [provider]+```++### Snapshot / Screenshot++```bash+cua do snapshot ["extra instructions"] # screenshot + AI summary (needs ANTHROPIC_API_KEY)+cua do screenshot [--save path]+cua do zoom "Window Name"+cua do unzoom+```++### Input++```bash+cua do click <x> <y> [left|right|middle]+cua do dclick <x> <y>+cua do move <x> <y>+cua do type "text"+cua do key <key> # enter, escape, tab, space, f1–f12, …+cua do hotkey ctrl+c+cua do scroll <up|down|left|right> [n]+cua do drag <x1> <y1> <x2> <y2>+```++### Shell / Open++```bash+cua do shell "command"+cua do open <url|path>+```++### Window++```bash+cua do window ls [app]+cua do window focus <id>+cua do window unfocus+cua do window minimize/maximize/close <id>+cua do window resize <id> <w> <h>+cua do window move <id> <x> <y>+cua do window info <id>+```++## Guardrails++- Re-snapshot after navigation, modals, or list changes — coords go stale.+- `snapshot` needs `ANTHROPIC_API_KEY`; use `screenshot` otherwise.+- Coords are image-space: zoom + max-length scaling are applied automatically.+- `do-host-consent` is permanent until the consent file is deleted (`~/.cua/host_consented`).+- Set `PYTHONIOENCODING=utf-8` on Windows for correct emoji output.