Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
Updated notebooks for cloud flow
Dillon DuPontf6aa56b6915e
diff --git a/notebooks/agent_nb.ipynb b/notebooks/agent_nb.ipynbindex cdc466165c..84d6757402 100644--- a/notebooks/agent_nb.ipynb+++ b/notebooks/agent_nb.ipynb@@ -6,7 +6,7 @@"source": ["## Agent\n","\n",- "This notebook demonstrates how to use Cua's Agent to run a workflow in a virtual sandbox on Apple Silicon Macs."+ "This notebook demonstrates how to use Cua's Agent to run workflows in virtual sandboxes, either using C/ua Cloud Containers or local VMs on Apple Silicon Macs."]},{@@ -68,7 +68,7 @@"cell_type": "markdown","metadata": {},"source": [- "Agent allows you to run an agentic workflow in a virtual sandbox instances on Apple Silicon. Here's a basic example:"+ "Agent allows you to run an agentic workflow in virtual sandbox instances. You can choose between cloud containers or local VMs."]},{@@ -83,15 +83,17 @@},{"cell_type": "code",- "execution_count": 2,+ "execution_count": null,"metadata": {},"outputs": [],"source": ["import os\n","\n","# Get API keys from environment or prompt user\n",- "anthropic_key = os.getenv(\"ANTHROPIC_API_KEY\") or input(\"Enter your Anthropic API key: \")\n",- "openai_key = os.getenv(\"OPENAI_API_KEY\") or input(\"Enter your OpenAI API key: \")\n",+ "anthropic_key = os.getenv(\"ANTHROPIC_API_KEY\") or \\\n",+ " input(\"Enter your Anthropic API key: \")\n",+ "openai_key = os.getenv(\"OPENAI_API_KEY\") or \\\n",+ " input(\"Enter your OpenAI API key: \")\n","\n","os.environ[\"ANTHROPIC_API_KEY\"] = anthropic_key\n","os.environ[\"OPENAI_API_KEY\"] = openai_key"@@ -101,14 +103,65 @@"cell_type": "markdown","metadata": {},"source": [- "Similar to Computer, you can either use the async context manager pattern or initialize the ComputerAgent instance directly."+ "## Option 1: Agent with C/ua Cloud Containers\n",+ "\n",+ "Use cloud containers for running agents from any system without local setup."]},{"cell_type": "markdown","metadata": {},"source": [- "Let's start by creating an agent that relies on the OpenAI API computer-use-preview model."+ "### Prerequisites for Cloud Containers\n",+ "\n",+ "To use C/ua Cloud Containers, you need to:\n",+ "1. Sign up at https://trycua.com\n",+ "2. Create a Cloud Container\n",+ "3. Generate an API Key\n",+ "\n",+ "Once you have these, you can connect to your cloud container and run agents on it."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Get C/ua API credentials and container details"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "cua_api_key = os.getenv(\"CUA_API_KEY\") or \\\n",+ " input(\"Enter your C/ua API Key: \")\n",+ "container_name = os.getenv(\"CONTAINER_NAME\") or \\\n",+ " input(\"Enter your Cloud Container name: \")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Choose the OS type for your container (linux or macos)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "os_type = input(\"Enter the OS type of your container (linux/macos) [default: linux]: \").lower() or \"linux\""+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### Create an agent with cloud container"]},{@@ -120,19 +173,135 @@"import logging\n","from pathlib import Path\n","\n",- "computer = Computer(verbosity=logging.INFO, provider_type=VMProviderType.LUME)\n",+ "# Connect to your existing cloud container\n",+ "computer = Computer(\n",+ " os_type=os_type,\n",+ " api_key=cua_api_key,\n",+ " name=container_name,\n",+ " provider_type=VMProviderType.CLOUD,\n",+ " verbosity=logging.INFO\n",+ ")\n","\n",- "# Create agent with Anthropic loop and provider\n",+ "# Create agent\n","agent = ComputerAgent(\n",- " computer=computer,\n",- " loop=AgentLoop.OPENAI,\n",- " model=LLM(provider=LLMProvider.OPENAI),\n",- " save_trajectory=True,\n",- " trajectory_dir=str(Path(\"trajectories\")),\n",- " only_n_most_recent_images=3,\n",- " verbosity=logging.INFO\n",- " )\n",+ " computer=computer,\n",+ " loop=AgentLoop.OPENAI,\n",+ " model=LLM(provider=LLMProvider.OPENAI),\n",+ " save_trajectory=True,\n",+ " trajectory_dir=str(Path(\"trajectories\")),\n",+ " only_n_most_recent_images=3,\n",+ " verbosity=logging.INFO\n",+ ")\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Run tasks on cloud container"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "tasks = [\n",+ " \"Open a web browser and navigate to GitHub\",\n",+ " \"Search for the trycua/cua repository\",\n",+ " \"Take a screenshot of the repository page\"\n",+ "]\n",+ "\n",+ "for i, task in enumerate(tasks):\n",+ " print(f\"\\nExecuting task {i+1}/{len(tasks)}: {task}\")\n",+ " async for result in cloud_agent.run(task):\n",+ " # print(result)\n",+ " pass\n",+ " print(f\"✅ Task {i+1}/{len(tasks)} completed: {task}\")\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## Option 2: Agent with Local VMs (Lume daemon)\n",+ "\n",+ "For Apple Silicon Macs, run agents on local VMs with near-native performance."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Before we can create an agent, we need to initialize a local computer with Lume."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "import logging\n",+ "from pathlib import Path\n","\n",+ "\n",+ "computer = Computer(\n",+ " verbosity=logging.INFO, \n",+ " provider_type=VMProviderType.LUME,\n",+ " display=\"1024x768\",\n",+ " memory=\"8GB\",\n",+ " cpu=\"4\",\n",+ " os_type=\"macos\"\n",+ ")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### Create an agent with local VM"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Let's start by creating an agent that relies on the OpenAI API computer-use-preview model."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "# Create agent with Anthropic loop and provider\n",+ "agent = ComputerAgent(\n",+ " computer=computer,\n",+ " loop=AgentLoop.OPENAI,\n",+ " model=LLM(provider=LLMProvider.OPENAI),\n",+ " save_trajectory=True,\n",+ " trajectory_dir=str(Path(\"trajectories\")),\n",+ " only_n_most_recent_images=3,\n",+ " verbosity=logging.INFO\n",+ ")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Run tasks on a local Lume VM"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": ["tasks = [\n"," \"Look for a repository named trycua/cua on GitHub.\",\n"," \"Check the open issues, open the most recent one and read it.\",\n",@@ -212,18 +381,42 @@},{"cell_type": "code",- "execution_count": 4,+ "execution_count": null,"metadata": {},"outputs": [],"source": [- "import os\n",- "\n",- "# Get API keys from environment or prompt user\n",- "anthropic_key = os.getenv(\"ANTHROPIC_API_KEY\") or input(\"Enter your Anthropic API key: \")\n",- "openai_key = os.getenv(\"OPENAI_API_KEY\") or input(\"Enter your OpenAI API key: \")\n",+ "from agent.ui.gradio.app import create_gradio_ui\n","\n",- "os.environ[\"ANTHROPIC_API_KEY\"] = anthropic_key\n",- "os.environ[\"OPENAI_API_KEY\"] = openai_key"+ "app = create_gradio_ui()\n",+ "app.launch(share=False)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## Advanced Agent Configurations"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### Using different agent loops"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "You can use different agent loops depending on your needs:"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "1. OpenAI Agent Loop"]},{@@ -232,10 +425,110 @@"metadata": {},"outputs": [],"source": [- "from agent.ui.gradio.app import create_gradio_ui\n",- "\n",- "app = create_gradio_ui()\n",- "app.launch(share=False)"+ "openai_agent = ComputerAgent(\n",+ " computer=computer, # Can be cloud or local\n",+ " loop=AgentLoop.OPENAI,\n",+ " model=LLM(provider=LLMProvider.OPENAI),\n",+ " save_trajectory=True,\n",+ " trajectory_dir=str(Path(\"trajectories\")),\n",+ " verbosity=logging.INFO\n",+ ")\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "2. Anthropic Agent Loop"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "anthropic_agent = ComputerAgent(\n",+ " computer=computer,\n",+ " loop=AgentLoop.ANTHROPIC,\n",+ " model=LLM(provider=LLMProvider.ANTHROPIC),\n",+ " save_trajectory=True,\n",+ " trajectory_dir=str(Path(\"trajectories\")),\n",+ " verbosity=logging.INFO\n",+ ")\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "3. Omni Agent Loop (supports multiple providers)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "omni_agent = ComputerAgent(\n",+ " computer=computer,\n",+ " loop=AgentLoop.OMNI,\n",+ " model=LLM(provider=LLMProvider.ANTHROPIC, name=\"claude-3-7-sonnet-20250219\"),\n",+ " # model=LLM(provider=LLMProvider.OPENAI, name=\"gpt-4.5-preview\"),\n",+ " # model=LLM(provider=LLMProvider.OLLAMA, name=\"gemma3:12b-it-q4_K_M\"),\n",+ " save_trajectory=True,\n",+ " trajectory_dir=str(Path(\"trajectories\")),\n",+ " only_n_most_recent_images=3,\n",+ " verbosity=logging.INFO\n",+ ")\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "4. UITARS Agent Loop (for local inference on Apple Silicon)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "uitars_agent = ComputerAgent(\n",+ " computer=computer,\n",+ " loop=AgentLoop.UITARS,\n",+ " model=LLM(provider=LLMProvider.UITARS),\n",+ " save_trajectory=True,\n",+ " trajectory_dir=str(Path(\"trajectories\")),\n",+ " verbosity=logging.INFO\n",+ ")\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### Trajectory viewing"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "All agent runs save trajectories that can be viewed at https://trycua.com/trajectory-viewer"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "print(f\"Trajectories saved to: {Path('trajectories').absolute()}\")\n",+ "print(\"Upload trajectory files to https://trycua.com/trajectory-viewer to visualize agent actions\")\n"]}],diff --git a/notebooks/computer_nb.ipynb b/notebooks/computer_nb.ipynbindex ec460de5f7..eed9324009 100644--- a/notebooks/computer_nb.ipynb+++ b/notebooks/computer_nb.ipynb@@ -6,7 +6,7 @@"source": ["## Computer\n","\n",- "This notebook demonstrates how to use Computer to operate a Lume sandbox VMs programmatically on Apple Silicon macOS systems."+ "This notebook demonstrates how to use Computer to operate sandbox VMs programmatically, either using C/ua Cloud Containers or local Lume VMs on Apple Silicon macOS systems."]},{@@ -22,16 +22,15 @@"metadata": {},"outputs": [],"source": [- "!pip uninstall -y cua-computer"+ "!pip uninstall -y cua-computer\n",+ "!pip install \"cua-computer[all]\""]},{- "cell_type": "code",- "execution_count": null,+ "cell_type": "markdown","metadata": {},- "outputs": [],"source": [- "!pip install \"cua-computer[all]\""+ "If locally installed, use this instead:"]},{@@ -40,7 +39,6 @@"metadata": {},"outputs": [],"source": [- "# If locally installed, use this instead:\n","import os\n","\n","os.chdir('../libs/computer')\n",@@ -55,7 +53,126 @@"cell_type": "markdown","metadata": {},"source": [- "## Lume daemon\n",+ "## Option 1: C/ua Cloud Containers\n",+ "\n",+ "C/ua Cloud Containers provide remote VMs that can be accessed from any system without local setup."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### Prerequisites for Cloud Containers\n",+ "\n",+ "To use C/ua Cloud Containers, you need to:\n",+ "1. Sign up at https://trycua.com\n",+ "2. Create a Cloud Container\n",+ "3. Generate an API Key\n",+ "\n",+ "Once you have these, you can connect to your cloud container using its name."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "# Get API key and container name from environment or prompt user\n",+ "import os\n",+ "\n",+ "cua_api_key = os.getenv(\"CUA_API_KEY\") or \\\n",+ " input(\"Enter your C/ua API Key: \")\n",+ "container_name = os.getenv(\"CONTAINER_NAME\") or \\\n",+ " input(\"Enter your Cloud Container name: \")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Choose the OS type for your container (linux or macos)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "os_type = input(\"Enter the OS type of your container (linux/macos) [default: linux]: \").lower() or \"linux\""+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### Connect to your Cloud Container"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "from computer import Computer, VMProviderType"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Connect to your existing C/ua Cloud Container"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "computer = Computer(\n",+ " os_type=os_type, # Must match the OS type of your cloud container\n",+ " api_key=os.getenv(\"CUA_API_KEY\"),\n",+ " name=os.getenv(\"CONTAINER_NAME\"),\n",+ " provider_type=VMProviderType.CLOUD,\n",+ ")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Take a screenshot"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "screenshot = await computer.interface.screenshot()\n",+ "\n",+ "with open(\"screenshot.png\", \"wb\") as f:\n",+ " f.write(screenshot)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## Option 2: Local VMs (Lume daemon)\n",+ "\n",+ "For Apple Silicon Macs, you can run VMs locally using the Lume daemon."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### Lume daemon setup\n","\n","Refer to [../libs/lume/README.md](../libs/lume/README.md) for more details on the lume cli."]@@ -143,7 +260,7 @@"cell_type": "markdown","metadata": {},"source": [- "## Initialize a Computer instance"+ "### Initialize a Local Computer instance"]},{@@ -190,7 +307,7 @@" os_type=\"macos\",\n"," provider_type=VMProviderType.LUME,\n",") as computer:\n",- " await computer.run()\n",+ " pass\n"," # ... do something with the computer interface"]},@@ -217,6 +334,15 @@"await computer.run()"]},+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## Computer Interface\n",+ "\n",+ "Both cloud and local computers provide the same interface for interaction."+ ]+ },{"cell_type": "markdown","metadata": {},@@ -461,7 +587,7 @@],"metadata": {"kernelspec": {- "display_name": "cua312",+ "display_name": ".venv","language": "python","name": "python3"},@@ -475,7 +601,7 @@"name": "python","nbconvert_exporter": "python","pygments_lexer": "ipython3",- "version": "3.12.9"+ "version": "3.12.2"}},"nbformat": 4,