Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
Merge branch 'main' into feat/docs/init
Morgan Dean32a8e504d419
diff --cc README.mdindex e43e4c7465,9fe22246c2..92dd888f42--- a/README.md+++ b/README.md@@@ -47,20 -47,138 +47,20 @@@</details></details><br/>-# π Quick Start with a Computer-Use Agent UI+# π Quick Start-**Need to automate desktop tasks? Launch the Computer-Use Agent UI with a single command.**+Read our guide on getting started with a Computer-Use Agent:- [Computer-Use Agent Quickstart](https://docs.trycua.com/home/guides/usage-guide)++[Computer-Use Agent Quickstart](https://trycua.com/docs/guides/usage-guide)-### Option 1: Fully-managed install with Docker (recommended)+Get started using Cua services on your machine:+[Cua Usage Guide](https://docs.trycua.com/home/guides/cua-usage-guide)-*Docker-based guided install for quick use*+Set up a development environment with the Dev Container:+[Dev Container Setup](https://docs.trycua.com/home/guides/dev-container-setup)-**macOS/Linux/Windows (via WSL):**+## Lume-```bash-# Requires Docker-/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/scripts/playground-docker.sh)"-```--This script will guide you through setup using Docker containers and launch the Computer-Use Agent UI.-------### Option 2: [Dev Container](./.devcontainer/README.md)--*Best for contributors and development*--This repository includes a [Dev Container](./.devcontainer/README.md) configuration that simplifies setup to a few steps:--1. **Install the Dev Containers extension ([VS Code](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) or [WindSurf](https://docs.windsurf.com/windsurf/advanced#dev-containers-beta))**-2. **Open the repository in the Dev Container:**- - Press `Ctrl+Shift+P` (or `β+Shift+P` on macOS)- - Select `Dev Containers: Clone Repository in Container Volume...` and paste the repository URL: `https://github.com/trycua/cua.git` (if not cloned) or `Dev Containers: Open Folder in Container...` (if git cloned).- > **Note**: On WindSurf, the post install hook might not run automatically. If so, run `/bin/bash .devcontainer/post-install.sh` manually.-3. **Open the VS Code workspace:** Once the post-install.sh is done running, open the `.vscode/py.code-workspace` workspace and press -.-4. **Run the Agent UI example:** Click - to start the Gradio UI. If prompted to install **debugpy (Python Debugger)** to enable remote debugging, select 'Yes' to proceed.-5. **Access the Gradio UI:** The Gradio UI will be available at `http://localhost:7860` and will automatically forward to your host machine.-------### Option 3: PyPI--*Direct Python package installation*--```bash-# conda create -yn cua python==3.12--pip install -U "cua-computer[all]" "cua-agent[all]"-python -m agent.ui # Start the agent UI-```--Or check out the [Usage Guide](#-usage-guide) to learn how to use our Python SDK in your own code.-------## Supported [Agent Loops](https://github.com/trycua/cua/blob/main/libs/python/agent/README.md#agent-loops)--- [UITARS-1.5](https://github.com/trycua/cua/blob/main/libs/python/agent/README.md#agent-loops) - Run locally on Apple Silicon with MLX, or use cloud providers-- [OpenAI CUA](https://github.com/trycua/cua/blob/main/libs/python/agent/README.md#agent-loops) - Use OpenAI's Computer-Use Preview model-- [Anthropic CUA](https://github.com/trycua/cua/blob/main/libs/python/agent/README.md#agent-loops) - Use Anthropic's Computer-Use capabilities-- [OmniParser-v2.0](https://github.com/trycua/cua/blob/main/libs/python/agent/README.md#agent-loops) - Control UI with [Set-of-Marks prompting](https://som-gpt4v.github.io/) using any vision model--## π₯οΈ Compatibility--For detailed compatibility information including host OS support, VM emulation capabilities, and model provider compatibility, see the [Compatibility Matrix](./COMPATIBILITY.md).--<br/>-<br/>--# π Usage Guide--Follow these steps to use C/ua in your own Python code. See [Developer Guide](./docs/Developer-Guide.md) for building from source.--### Step 1: Install Lume CLI--```bash-/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/lume/scripts/install.sh)"-```--Lume CLI manages high-performance macOS/Linux VMs with near-native speed on Apple Silicon.--### Step 2: Pull the macOS CUA Image--```bash-lume pull macos-sequoia-cua:latest-```--The macOS CUA image contains the default Mac apps and the Computer Server for easy automation.--### Step 3: Install Python SDK--```bash-pip install "cua-computer[all]" "cua-agent[all]"-```--### Step 4: Use in Your Code--```python-from computer import Computer-from agent import ComputerAgent, LLM--async def main():- # Start a local macOS VM- computer = Computer(os_type="macos")- await computer.run()-- # Or with C/ua Cloud Container- computer = Computer(- os_type="linux",- api_key="your_cua_api_key_here",- name="your_container_name_here"- )-- # Example: Direct control of a macOS VM with Computer- computer.interface.delay = 0.1 # Wait 0.1 seconds between kb/m actions- await computer.interface.left_click(100, 200)- await computer.interface.type_text("Hello, world!")- screenshot_bytes = await computer.interface.screenshot()-- # Example: Create and run an agent locally using mlx-community/UI-TARS-1.5-7B-6bit- agent = ComputerAgent(- model="mlx/mlx-community/UI-TARS-1.5-7B-6bit",- tools=[computer],- )- async for result in agent.run("Find the trycua/cua repository on GitHub and follow the quick start guide"):- print(result)--if __name__ == "__main__":- asyncio.run(main())-```--For ready-to-use examples, check out our [Notebooks](./notebooks/) collection.--### Lume CLI Reference+For managing and creating virtual machines on macOS, check out [Lume](./libs/lume/README.md).```bash# Install Lume CLI and background service