Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
Move text from README to Cua documentation
James Murdzae17f6106c806
diff --git a/docs/content/docs/agent-sdk/agent-loops.mdx b/docs/content/docs/agent-sdk/agent-loops.mdxindex d443c3cec9..6a18f064c8 100644--- a/docs/content/docs/agent-sdk/agent-loops.mdx+++ b/docs/content/docs/agent-sdk/agent-loops.mdx@@ -22,7 +22,7 @@ agent = ComputerAgent(tools=[computer])-prompt = "open github, navigate to trycua/cua"+prompt = "Take a screenshot and tell me what you see"async for result in agent.run(prompt):if result["output"][-1]["type"] == "message":@@ -31,6 +31,69 @@ async for result in agent.run(prompt):For a list of supported models and configurations, see the [Supported Agents](./supported-agents/computer-use-agents) page.+### Response Format++```python+{+ "output": [+ {+ "type": "message",+ "role": "assistant",+ "content": [{"type": "output_text", "text": "I can see..."}]+ },+ {+ "type": "computer_call",+ "action": {"type": "screenshot"},+ "call_id": "call_123"+ },+ {+ "type": "computer_call_output",+ "call_id": "call_123",+ "output": {"image_url": "data:image/png;base64,..."}+ }+ ],+ "usage": {+ "prompt_tokens": 150,+ "completion_tokens": 75,+ "total_tokens": 225,+ "response_cost": 0.01,+ }+}+```++### Environment Variables++Use the following environment variables to configure the agent and its access to cloud computers and LLM providers:++```bash+# Computer instance (cloud)+export CUA_CONTAINER_NAME="your-container-name"+export CUA_API_KEY="your-cua-api-key"++# LLM API keys+export ANTHROPIC_API_KEY="your-anthropic-key"+export OPENAI_API_KEY="your-openai-key"+```++### Input and output++The input prompt passed to `Agent.run` can either be a string or a list of message dictionaries:++```python+messages = [+ {+ "role": "user",+ "content": "Take a screenshot and describe what you see"+ },+ {+ "role": "assistant",+ "content": "I'll take a screenshot for you."+ }+]+```++The output is an AsyncGenerator that yields response chunks.+### ParametersThe `ComputerAgent` constructor provides a wide range of options for customizing agent behavior, tool integration, callbacks, resource management, and more.@@ -79,4 +142,30 @@ agent = ComputerAgent(use_prompt_caching=True,max_trajectory_budget={"max_budget": 5.0, "raise_error": True})+```++### Streaming Responses++```python+async for result in agent.run(messages, stream=True):+ # Process streaming chunks+ for item in result["output"]:+ if item["type"] == "message":+ print(item["content"][0]["text"], end="", flush=True)+ elif item["type"] == "computer_call":+ action = item["action"]+ print(f"\n[Action: {action['type']}]")+```++### Error Handling++```python+try:+ async for result in agent.run(messages):+ # Process results+ pass+except BudgetExceededException:+ print("Budget limit exceeded")+except Exception as e:+ print(f"Agent error: {e}")```\ No newline at end of filediff --git a/docs/content/docs/agent-sdk/callbacks/cost-saving.mdx b/docs/content/docs/agent-sdk/callbacks/cost-saving.mdxindex 9a543dc1ab..0787b1f6d6 100644--- a/docs/content/docs/agent-sdk/callbacks/cost-saving.mdx+++ b/docs/content/docs/agent-sdk/callbacks/cost-saving.mdx@@ -28,19 +28,23 @@ agent = ComputerAgent(## Budget Manager Shorthand```python+# Simple budget limitagent = ComputerAgent(model="anthropic/claude-3-5-sonnet-20241022",- tools=[computer],- max_trajectory_budget=5.0 # Auto-adds BudgetManagerCallback+ max_trajectory_budget=5.0 # $5 limit)```**Or with options:**```python+# Advanced budget configurationagent = ComputerAgent(model="anthropic/claude-3-5-sonnet-20241022",- tools=[computer],- max_trajectory_budget={"max_budget": 5.0, "raise_error": True}+ max_trajectory_budget={+ "max_budget": 10.0,+ "raise_error": True, # Raise error when exceeded+ "reset_after_each_run": False # Persistent across runs+ })```diff --git a/docs/content/docs/agent-sdk/callbacks/index.mdx b/docs/content/docs/agent-sdk/callbacks/index.mdxindex 0d0f7de4a6..590c236a6d 100644--- a/docs/content/docs/agent-sdk/callbacks/index.mdx+++ b/docs/content/docs/agent-sdk/callbacks/index.mdx@@ -4,26 +4,61 @@ title: CallbacksCallbacks in the Agent SDK provide hooks into the agent's lifecycle, allowing for custom functionality to be executed at various stages of an agent's run. They enable extensibility by allowing developers to integrate their own logic for tasks such as logging, cost management, and data anonymization.-The callback lifecycle is described in [Agent Lifecycle](callbacks/agent-lifecycle).-## Usage-You can add preprocessing and postprocessing hooks using callbacks, or write your own by subclassing `AsyncCallbackHandler`:+You can add preprocessing and postprocessing hooks using callbacks, or write your own by subclassing `AsyncCallbackHandler`.++### Built-in Callbacks++Built-in callbacks can be used as follows:```python-from agent.callbacks import ImageRetentionCallback, PIIAnonymizationCallback+from agent.callbacks import (+ ImageRetentionCallback,+ TrajectorySaverCallback,+ BudgetManagerCallback,+ LoggingCallback+)agent = ComputerAgent(model="anthropic/claude-3-5-sonnet-20241022",tools=[computer],- callbacks=[ImageRetentionCallback(only_n_most_recent_images=3)]+ callbacks=[+ ImageRetentionCallback(only_n_most_recent_images=3),+ TrajectorySaverCallback(trajectory_dir="trajectories"),+ BudgetManagerCallback(max_budget=10.0, raise_error=True),+ LoggingCallback(level=logging.INFO)+ ])```-## Built-in Callbacks+The following built-in callbacks are available:- [BudgetManagerCallback](callbacks/cost-saving): Stops execution when budget exceeded- [LoggingCallback](callbacks/trajectories): Logs agent activities- **ImageRetentionCallback**: Limits recent images in context- **TrajectorySaverCallback**: Saves conversation trajectories- [PII Anonymization](callbacks/pii-anonymization)++### Custom Callbacks++Create custom callbacks using knowlege of the callback lifecycle as described in [Agent Lifecycle](callbacks/agent-lifecycle).++```python+from agent.callbacks.base import AsyncCallbackHandler++class CustomCallback(AsyncCallbackHandler):+ async def on_llm_start(self, messages):+ """Preprocess messages before LLM call"""+ # Add custom preprocessing logic+ return messages++ async def on_llm_end(self, messages):+ """Postprocess messages after LLM call"""+ # Add custom postprocessing logic+ return messages++ async def on_usage(self, usage):+ """Track usage information"""+ print(f"Tokens used: {usage.total_tokens}")+```diff --git a/docs/content/docs/agent-sdk/callbacks/trajectories.mdx b/docs/content/docs/agent-sdk/callbacks/trajectories.mdxindex 8799c5ebb9..8118f217cf 100644--- a/docs/content/docs/agent-sdk/callbacks/trajectories.mdx+++ b/docs/content/docs/agent-sdk/callbacks/trajectories.mdx@@ -29,8 +29,8 @@ agent = ComputerAgent(```pythonagent = ComputerAgent(model="anthropic/claude-3-5-sonnet-20241022",- tools=[computer],- trajectory_dir="trajectories" # Auto-adds TrajectorySaverCallback+ trajectory_dir="trajectories", # Auto-save trajectories+ tools=[computer])```@@ -46,6 +46,12 @@ The viewer provides:## Trajectory Structure+Trajectories are saved with:+- Complete conversation history+- Usage statistics and costs+- Timestamps and metadata+- Screenshots and computer actions+Each trajectory contains:- **metadata.json**: Run info, timestamps, usage stats (`total_tokens`, `response_cost`)- **turn_000/**: Turn-by-turn conversation history (api calls, responses, computer calls, screenshots)diff --git a/docs/content/docs/agent-sdk/sandboxed-tools.mdx b/docs/content/docs/agent-sdk/custom-tools.mdxsimilarity index 65%rename from docs/content/docs/agent-sdk/sandboxed-tools.mdxrename to docs/content/docs/agent-sdk/custom-tools.mdxindex 8fafce7a70..00847cf6ee 100644--- a/docs/content/docs/agent-sdk/sandboxed-tools.mdx+++ b/docs/content/docs/agent-sdk/custom-tools.mdx@@ -1,11 +1,29 @@----title: Sandboxed Tools-slug: sandboxed-tools+title: Custom Tools+slug: custom-tools---The Agent SDK supports defining custom Python tools that run securely in sandboxed environments on remote Cua Computers. This enables safe execution of user-defined functions, isolation of dependencies, and robust automation workflows.-## Example: Defining a Sandboxed Tool+## Custom Tools++Define a custom tool for an agent:++```python+def calculate(a: int, b: int) -> int:+ """Calculate the sum of two integers"""+ return a + b++# Use with agent+agent = ComputerAgent(+ model="anthropic/claude-3-5-sonnet-20241022",+ tools=[computer, calculate]+)+```++## Sandboxed Tools++Define a sandboxed tool:```pythonfrom computer.helpers import sandboxeddiff --git a/docs/content/docs/agent-sdk/meta.json b/docs/content/docs/agent-sdk/meta.jsonindex 5db33148dc..07bf7199d6 100644--- a/docs/content/docs/agent-sdk/meta.json+++ b/docs/content/docs/agent-sdk/meta.json@@ -4,11 +4,11 @@"pages": ["agent-loops","supported-agents",+ "supported-model-providers","chat-history","callbacks",- "sandboxed-tools",+ "custom-tools","custom-computer-handlers",- "local-models","prompt-caching","usage-tracking","benchmarks",diff --git a/docs/content/docs/agent-sdk/supported-model-providers/index.mdx b/docs/content/docs/agent-sdk/supported-model-providers/index.mdxnew file mode 100644index 0000000000..68e372b1d4--- /dev/null+++ b/docs/content/docs/agent-sdk/supported-model-providers/index.mdx@@ -0,0 +1,32 @@+---+title: Supported Model Providers+---++## Supported Models++### Anthropic Claude (Computer Use API)+```python+model="anthropic/claude-3-5-sonnet-20241022"+model="anthropic/claude-3-7-sonnet-20250219"+model="anthropic/claude-opus-4-20250514"+model="anthropic/claude-sonnet-4-20250514"+```++### OpenAI Computer Use Preview+```python+model="openai/computer-use-preview"+```++### UI-TARS (Local or Huggingface Inference)+```python+model="huggingface-local/ByteDance-Seed/UI-TARS-1.5-7B"+model="ollama_chat/0000/ui-tars-1.5-7b"+```++### Omniparser + Any LLM+```python+model="omniparser+ollama_chat/mistral-small3.2"+model="omniparser+vertex_ai/gemini-pro"+model="omniparser+anthropic/claude-3-5-sonnet-20241022"+model="omniparser+openai/gpt-4o"+```\ No newline at end of filediff --git a/docs/content/docs/agent-sdk/local-models.mdx b/docs/content/docs/agent-sdk/supported-model-providers/local-models.mdxsimilarity index 100%rename from docs/content/docs/agent-sdk/local-models.mdxrename to docs/content/docs/agent-sdk/supported-model-providers/local-models.mdxdiff --git a/docs/content/docs/quickstart-devs.mdx b/docs/content/docs/quickstart-devs.mdxindex 9b2200816e..37367709bd 100644--- a/docs/content/docs/quickstart-devs.mdx+++ b/docs/content/docs/quickstart-devs.mdx@@ -87,6 +87,16 @@ Choose how you want to run your cua computer. **Cloud containers are recommended<Tab value="Python">```bashpip install "cua-agent[all]" cua-computer++ # or install specific providers+ pip install "cua-agent[openai]" # OpenAI computer-use-preview support+ pip install "cua-agent[anthropic]" # Anthropic Claude support+ pip install "cua-agent[omni]" # Omniparser + any LLM support+ pip install "cua-agent[uitars]" # UI-TARS+ pip install "cua-agent[uitars-mlx]" # UI-TARS + MLX support+ pip install "cua-agent[uitars-hf]" # UI-TARS + Huggingface support+ pip install "cua-agent[glm45v-hf]" # GLM-4.5V + Huggingface support+ pip install "cua-agent[ui]" # Gradio UI support```</Tab><Tab value="TypeScript">diff --git a/libs/python/agent/README.md b/libs/python/agent/README.mdindex cec3cd45a0..ed90076be0 100644--- a/libs/python/agent/README.md+++ b/libs/python/agent/README.md@@ -29,16 +29,6 @@```bashpip install "cua-agent[all]"--# or install specific providers-pip install "cua-agent[openai]" # OpenAI computer-use-preview support-pip install "cua-agent[anthropic]" # Anthropic Claude support-pip install "cua-agent[omni]" # Omniparser + any LLM support-pip install "cua-agent[uitars]" # UI-TARS-pip install "cua-agent[uitars-mlx]" # UI-TARS + MLX support-pip install "cua-agent[uitars-hf]" # UI-TARS + Huggingface support-pip install "cua-agent[glm45v-hf]" # GLM-4.5V + Huggingface support-pip install "cua-agent[ui]" # Gradio UI support```## Quick Start@@ -79,303 +69,18 @@ if __name__ == "__main__":asyncio.run(main())```-## Supported Models--### Anthropic Claude (Computer Use API)-```python-model="anthropic/claude-3-5-sonnet-20241022"-model="anthropic/claude-3-7-sonnet-20250219"-model="anthropic/claude-opus-4-20250514"-model="anthropic/claude-sonnet-4-20250514"-```--### OpenAI Computer Use Preview-```python-model="openai/computer-use-preview"-```--### UI-TARS (Local or Huggingface Inference)-```python-model="huggingface-local/ByteDance-Seed/UI-TARS-1.5-7B"-model="ollama_chat/0000/ui-tars-1.5-7b"-```--### Omniparser + Any LLM-```python-model="omniparser+ollama_chat/mistral-small3.2"-model="omniparser+vertex_ai/gemini-pro"-model="omniparser+anthropic/claude-3-5-sonnet-20241022"-model="omniparser+openai/gpt-4o"-```--## Custom Tools--Define custom tools using decorated functions:--```python-from computer.helpers import sandboxed--@sandboxed()-def read_file(location: str) -> str:- """Read contents of a file-- Parameters- ----------- location : str- Path to the file to read-- Returns- -------- str- Contents of the file or error message- """- try:- with open(location, 'r') as f:- return f.read()- except Exception as e:- return f"Error reading file: {str(e)}"--def calculate(a: int, b: int) -> int:- """Calculate the sum of two integers"""- return a + b--# Use with agent-agent = ComputerAgent(- model="anthropic/claude-3-5-sonnet-20241022",- tools=[computer, read_file, calculate]-)-```--## Callbacks System--agent provides a comprehensive callback system for extending functionality:--### Built-in Callbacks--```python-from agent.callbacks import (- ImageRetentionCallback,- TrajectorySaverCallback,- BudgetManagerCallback,- LoggingCallback-)--agent = ComputerAgent(- model="anthropic/claude-3-5-sonnet-20241022",- tools=[computer],- callbacks=[- ImageRetentionCallback(only_n_most_recent_images=3),- TrajectorySaverCallback(trajectory_dir="trajectories"),- BudgetManagerCallback(max_budget=10.0, raise_error=True),- LoggingCallback(level=logging.INFO)- ]-)-```--### Custom Callbacks--```python-from agent.callbacks.base import AsyncCallbackHandler--class CustomCallback(AsyncCallbackHandler):- async def on_llm_start(self, messages):- """Preprocess messages before LLM call"""- # Add custom preprocessing logic- return messages-- async def on_llm_end(self, messages):- """Postprocess messages after LLM call"""- # Add custom postprocessing logic- return messages-- async def on_usage(self, usage):- """Track usage information"""- print(f"Tokens used: {usage.total_tokens}")-```--## Budget Management--Control costs with built-in budget management:--```python-# Simple budget limit-agent = ComputerAgent(- model="anthropic/claude-3-5-sonnet-20241022",- max_trajectory_budget=5.0 # $5 limit-)--# Advanced budget configuration-agent = ComputerAgent(- model="anthropic/claude-3-5-sonnet-20241022",- max_trajectory_budget={- "max_budget": 10.0,- "raise_error": True, # Raise error when exceeded- "reset_after_each_run": False # Persistent across runs- }-)-```--## Trajectory Management+## Docs-Save and replay agent conversations:--```python-agent = ComputerAgent(- model="anthropic/claude-3-5-sonnet-20241022",- trajectory_dir="trajectories", # Auto-save trajectories- tools=[computer]-)--# Trajectories are saved with:-# - Complete conversation history-# - Usage statistics and costs-# - Timestamps and metadata-# - Screenshots and computer actions-```--## Configuration Options--### ComputerAgent Parameters--- `model`: Model identifier (required)-- `tools`: List of computer objects and decorated functions-- `callbacks`: List of callback handlers for extensibility-- `only_n_most_recent_images`: Limit recent images to prevent context overflow-- `verbosity`: Logging level (logging.INFO, logging.DEBUG, etc.)-- `trajectory_dir`: Directory to save conversation trajectories-- `max_retries`: Maximum API call retries (default: 3)-- `screenshot_delay`: Delay between actions and screenshots (default: 0.5s)-- `use_prompt_caching`: Enable prompt caching for supported models-- `max_trajectory_budget`: Budget limit configuration--### Environment Variables--```bash-# Computer instance (cloud)-export CUA_CONTAINER_NAME="your-container-name"-export CUA_API_KEY="your-cua-api-key"--# LLM API keys-export ANTHROPIC_API_KEY="your-anthropic-key"-export OPENAI_API_KEY="your-openai-key"-```--## Advanced Usage--### Streaming Responses--```python-async for result in agent.run(messages, stream=True):- # Process streaming chunks- for item in result["output"]:- if item["type"] == "message":- print(item["content"][0]["text"], end="", flush=True)- elif item["type"] == "computer_call":- action = item["action"]- print(f"\n[Action: {action['type']}]")-```--### Interactive Chat Loop--```python-history = []-while True:- user_input = input("> ")- if user_input.lower() in ['quit', 'exit']:- break-- history.append({"role": "user", "content": user_input})-- async for result in agent.run(history):- history += result["output"]-- # Display assistant responses- for item in result["output"]:- if item["type"] == "message":- print(item["content"][0]["text"])-```--### Error Handling--```python-try:- async for result in agent.run(messages):- # Process results- pass-except BudgetExceededException:- print("Budget limit exceeded")-except Exception as e:- print(f"Agent error: {e}")-```--## API Reference--### ComputerAgent.run()--```python-async def run(- self,- messages: Messages,- stream: bool = False,- **kwargs-) -> AsyncGenerator[Dict[str, Any], None]:- """- Run the agent with the given messages.-- Args:- messages: List of message dictionaries- stream: Whether to stream the response- **kwargs: Additional arguments-- Returns:- AsyncGenerator that yields response chunks- """-```--### Message Format--```python-messages = [- {- "role": "user",- "content": "Take a screenshot and describe what you see"- },- {- "role": "assistant",- "content": "I'll take a screenshot for you."- }-]-```--### Response Format--```python-{- "output": [- {- "type": "message",- "role": "assistant",- "content": [{"type": "output_text", "text": "I can see..."}]- },- {- "type": "computer_call",- "action": {"type": "screenshot"},- "call_id": "call_123"- },- {- "type": "computer_call_output",- "call_id": "call_123",- "output": {"image_url": "data:image/png;base64,..."}- }- ],- "usage": {- "prompt_tokens": 150,- "completion_tokens": 75,- "total_tokens": 225,- "response_cost": 0.01,- }-}-```+- [Agent Loops](https://trycua.com/docs/agent-sdk/agent-loops)+- [Supported Agents](https://trycua.com/docs/agent-sdk/supported-agents)+- [Supported Models](https://trycua.com/docs/agent-sdk/supported-models)+- [Chat History](https://trycua.com/docs/agent-sdk/chat-history)+- [Callbacks](https://trycua.com/docs/agent-sdk/callbacks)+- [Custom Tools](https://trycua.com/docs/agent-sdk/custom-tools)+- [Custom Computer Handlers](https://trycua.com/docs/agent-sdk/custom-computer-handlers)+- [Prompt Caching](https://trycua.com/docs/agent-sdk/prompt-caching)+- [Usage Tracking](https://trycua.com/docs/agent-sdk/usage-tracking)+- [Benchmarks](https://trycua.com/docs/agent-sdk/benchmarks)## License