Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
Fix Agent demo
f-trycua14c81971249b
diff --git a/README.md b/README.mdindex 5006033d5d..5d9e980f82 100644--- a/README.md+++ b/README.md@@ -23,7 +23,7 @@ Create and run high-performance macOS and Linux VMs on Apple Silicon, with built|---------|-------------|--------------|---------|| [**Lume**](./libs/lume/README.md) | CLI for running macOS/Linux VMs with near-native performance using Apple's `Virtualization.Framework`. | [](https://github.com/trycua/cua/releases/latest/download/lume.pkg.tar.gz) | [](https://github.com/trycua/cua/releases) || [**Computer**](./libs/computer/README.md) | Computer-Use Interface (CUI) framework for interacting with macOS/Linux sandboxes | `pip install cua-computer` | [](https://pypi.org/project/cua-computer/) |-| [**Agent**](./libs/agent/README.md) | Computer-Use Agent (CUA) framework for running agentic workflows in macOS/Linux dedicated sandboxes | `pip install cua-agent` | [](https://pypi.org/project/cua-agent/) |+| [**Agent (Experimental)**](./libs/agent/README.md) | Computer-Use Agent (CUA) framework for running agentic workflows in macOS/Linux dedicated sandboxes | `pip install cua-agent` | [](https://pypi.org/project/cua-agent/) |## Lumediff --git a/libs/agent/README.md b/libs/agent/README.mdindex c008d99c81..1319ca184e 100644--- a/libs/agent/README.md+++ b/libs/agent/README.md@@ -17,43 +17,13 @@**Agent** is a Computer Use (CUA) framework for running multi-app agentic workflows targeting macOS and Linux sandbox, supporting local (Ollama) and cloud model providers (OpenAI, Anthropic, Groq, DeepSeek, Qwen). The framework integrates with Microsoft's OmniParser for enhanced UI understanding and interaction.-### Get started with Agent--```python-from agent import ComputerAgent, AgentLoop, LLMProvider-from computer import Computer--computer = Computer(verbosity=logging.INFO)--agent = ComputerAgent(- computer=computer,- loop=AgentLoop.ANTHROPIC,- # loop=AgentLoop.OMNI,- model=LLM(provider=LLMProvider.ANTHROPIC, name="claude-3-7-sonnet-20250219"),- # model=LLM(provider=LLMProvider.OPENAI, name="gpt-4.5-preview"),- save_trajectory=True,- trajectory_dir=str(Path("trajectories")),- only_n_most_recent_images=3,- verbosity=logging.INFO,-)+> While our north star is to create a 1-click experience, this preview of Agent might be still a bit rough around the edges. We appreciate your patience as we work to improve the experience.-tasks = [-"""-Please help me with the following task:-1. Open Safari browser-2. Go to Wikipedia.org-3. Search for "Claude AI"-4. Summarize the main points you find about Claude AI-"""-]+### Get started with Agent-async with agent:- for i, task in enumerate(tasks, 1):- print(f"\nExecuting task {i}/{len(tasks)}: {task}")- async for result in agent.run(task):- print(result)- print(f"Task {i} completed")-```+<div align="center">+ <img src="../../img/agent.png"/>+</div>## Installdiff --git a/libs/agent/agent/providers/omni/tools/__init__.py b/libs/agent/agent/providers/omni/tools/__init__.pyindex 7091d1503d..bcf561e1ba 100644--- a/libs/agent/agent/providers/omni/tools/__init__.py+++ b/libs/agent/agent/providers/omni/tools/__init__.py@@ -7,6 +7,5 @@ from .manager import OmniToolManager__all__ = ["OmniBashTool","OmniComputerTool",- "OmniEditTool","OmniToolManager",]diff --git a/notebooks/agent_nb.ipynb b/notebooks/agent_nb.ipynbindex 47f9118935..12e6e57778 100644--- a/notebooks/agent_nb.ipynb+++ b/notebooks/agent_nb.ipynb@@ -79,11 +79,11 @@},{"cell_type": "code",- "execution_count": 2,+ "execution_count": 4,"metadata": {},"outputs": [],"source": [- "from agent import ComputerAgent, AgenticLoop, LLMProvider\n",+ "from agent import AgentLoop, LLMProvider\n","from computer import Computer"]},@@ -96,12 +96,13 @@},{"cell_type": "code",- "execution_count": 12,+ "execution_count": 11,"metadata": {},"outputs": [],"source": [- "# Set your API key\n",- "!export ANTHROPIC_API_KEY=\"your-anthropic-api-key\""+ "import os\n",+ "os.environ[\"ANTHROPIC_API_KEY\"] = \"your-anthropic-api-key\"\n",+ "os.environ[\"OPENAI_API_KEY\"] = \"your-openai-api-key\""]},{@@ -120,21 +121,22 @@"from datetime import datetime\n","import logging\n","from pathlib import Path\n",+ "from agent.core.computer_agent import ComputerAgent, LLM\n","\n","computer = Computer(verbosity=logging.INFO)\n","\n","# Create agent with Anthropic loop and provider\n","agent = ComputerAgent(\n",- " computer=computer,\n",- " api_key=\"<your-anthropic-api-key>\",\n",- " loop_type=AgenticLoop.ANTHROPIC,\n",- " ai_provider=LLMProvider.ANTHROPIC,\n",- " model='claude-3-7-sonnet-20250219',\n",- " save_trajectory=True,\n",- " trajectory_dir=str(Path(\"trajectories\") / datetime.now().strftime(\"%Y%m%d_%H%M%S\")),\n",- " only_n_most_recent_images=3, # Slightly different from the omni example\n",- " verbosity=logging.INFO,\n",- ")\n",+ " computer=computer,\n",+ " loop=AgentLoop.ANTHROPIC,\n",+ " # loop=AgentLoop.OMNI,\n",+ " model=LLM(provider=LLMProvider.ANTHROPIC, name=\"claude-3-7-sonnet-20250219\"),\n",+ " # model=LLM(provider=LLMProvider.OPENAI, name=\"gpt-4.5-preview\"),\n",+ " save_trajectory=True,\n",+ " trajectory_dir=str(Path(\"trajectories\")),\n",+ " only_n_most_recent_images=3,\n",+ " verbosity=logging.INFO\n",+ " )\n","\n","tasks = [\n","\"\"\"\n",@@ -176,13 +178,13 @@"# Create agent with Anthropic loop and provider\n","agent = ComputerAgent(\n"," computer=computer,\n",- " api_key=\"<your-anthropic-api-key>\",\n",- " loop_type=AgenticLoop.OMNI,\n",- " ai_provider=LLMProvider.ANTHROPIC,\n",- " model='claude-3-7-sonnet-20250219',\n",+ " # loop=AgentLoop.ANTHROPIC,\n",+ " loop=AgentLoop.OMNI,\n",+ " model=LLM(provider=LLMProvider.OPENAI, name=\"gpt-4.5-preview\"),\n",+ " # model=LLM(provider=LLMProvider.ANTHROPIC, name=\"claude-3-7-sonnet-20250219\"),\n"," save_trajectory=True,\n",- " trajectory_dir=str(Path(\"trajectories\") / datetime.now().strftime(\"%Y%m%d_%H%M%S\")),\n",- " only_n_most_recent_images=3, # Slightly different from the omni example\n",+ " trajectory_dir=str(Path(\"trajectories\")),\n",+ " only_n_most_recent_images=3,\n"," verbosity=logging.INFO,\n",")\n","\n",diff --git a/notebooks/computer_nb.ipynb b/notebooks/computer_nb.ipynbindex 8a52f02c5e..552a08a62a 100644--- a/notebooks/computer_nb.ipynb+++ b/notebooks/computer_nb.ipynb@@ -18,9 +18,18 @@},{"cell_type": "code",- "execution_count": null,+ "execution_count": 2,"metadata": {},- "outputs": [],+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "\u001b[33mWARNING: Skipping cua-computer as it is not installed.\u001b[0m\u001b[33m\n",+ "\u001b[0m"+ ]+ }+ ],"source": ["!pip uninstall -y cua-computer"]@@ -116,9 +125,18 @@},{"cell_type": "code",- "execution_count": null,+ "execution_count": 4,"metadata": {},- "outputs": [],+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "name os cpu memory disk display status ip vnc \n",+ "macos-sequoia-cua_latest macOS 12 16.00G 64.5GB/80.0GB 1024x768 running 192.168.64.78 vnc://:[email protected]:56085 \n"+ ]+ }+ ],"source": ["!lume ls"]@@ -159,7 +177,7 @@},{"cell_type": "code",- "execution_count": 7,+ "execution_count": 2,"metadata": {},"outputs": [],"source": [@@ -182,41 +200,9 @@},{"cell_type": "code",- "execution_count": 8,+ "execution_count": null,"metadata": {},- "outputs": [- {- "name": "stdout",- "output_type": "stream",- "text": [- "\n",- "Equivalent curl command:\n",- "curl -X POST 'http://localhost:3000/lume/vms/macos-sequoia-cua_latest/run' -H 'Content-Type: application/json' -d '{\"noDisplay\": false, \"sharedDirectories\": []}'\n",- "\n",- "\n",- "Equivalent curl command:\n",- "curl -X POST \\\n",- " 'http://localhost:3000/lume/vms/macos-sequoia-cua_latest/run' \\\n",- " -H 'Content-Type: application/json' \\\n",- " -d '{\"noDisplay\": false, \"sharedDirectories\": []}'\n",- "\n"- ]- },- {- "name": "stderr",- "output_type": "stream",- "text": [- "Computer API Server connection lost. Will retry automatically.\n"- ]- },- {- "name": "stderr",- "output_type": "stream",- "text": [- "Computer API Server not ready yet. Will retry automatically.\n"- ]- }- ],+ "outputs": [],"source": ["async with Computer(\n"," # name=\"my_vm\", # optional, in case you want to use any other VM created using lume\n",