Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
Undocumented old agent proxy
Dillon DuPontcbb79536248d
diff --git a/docs/content/docs/libraries/agent/PeerJS-API.mdx b/docs/content/docs/libraries/agent/PeerJS-API.mdxdeleted file mode 100644index f3edea6316..0000000000--- a/docs/content/docs/libraries/agent/PeerJS-API.mdx+++ /dev/null@@ -1,306 +0,0 @@-----title: PeerJS Proxy API Reference-description: Reference for the P2P WebRTC endpoint of the Agent Proxy Server.------The Agent Proxy Server supports peer-to-peer (P2P) connections using WebRTC through the PeerJS protocol. This allows direct browser-to-agent communication without intermediary servers.--## Connection Setup--### Prerequisites-- Install peerjs-python: `pip install peerjs-python aiortc`-- Running signaling server (default: localhost:9000)--### Starting P2P Server-```bash-# P2P only-python -m agent.proxy.cli --mode p2p --peer-id computer-agent-proxy--# Both HTTP and P2P-python -m agent.proxy.cli --mode both --peer-id computer-agent-proxy--# Custom signaling server-python -m agent.proxy.cli --mode p2p \- --peer-id my-agent \- --signaling-host signaling.example.com \- --signaling-port 443 \- --signaling-secure-```--## Client Connection (Python)--### Basic Connection-```python-import asyncio-import json-from peerjs import Peer, PeerOptions, ConnectionEventType-from aiortc import RTCConfiguration, RTCIceServer--async def connect_to_agent():- # Set up peer options- options = PeerOptions(- host="localhost",- port=9000,- secure=False,- config=RTCConfiguration(- iceServers=[RTCIceServer(urls="stun:stun.l.google.com:19302")]- )- )-- # Create client peer- client_peer = Peer(id="my-client", peer_options=options)- await client_peer.start()-- # Connect to agent proxy- connection = client_peer.connect("computer-agent-proxy")-- @connection.on(ConnectionEventType.Open)- async def connection_open():- print("Connected to agent proxy")-- # Send agent request- request = {- "model": "anthropic/claude-3-5-sonnet-20241022",- "input": "Hello from P2P client!"- }- await connection.send(json.dumps(request))-- @connection.on(ConnectionEventType.Data)- async def connection_data(data):- response = json.loads(data) if isinstance(data, str) else data- print(f"Agent response: {response}")- await client_peer.destroy()-- # Keep connection alive- await asyncio.sleep(10)--asyncio.run(connect_to_agent())-```--### HTTP-like Requests over P2P-```python-import asyncio-import json-from peerjs import Peer, PeerOptions, ConnectionEventType--async def http_over_p2p():- # Connect to peer- client_peer = Peer(id="http-client", peer_options=options)- await client_peer.start()- connection = client_peer.connect("computer-agent-proxy")-- @connection.on(ConnectionEventType.Open)- async def connection_open():- # Send HTTP-like request- http_request = {- "method": "POST",- "path": "/responses",- "body": {- "model": "anthropic/claude-3-5-sonnet-20241022",- "input": "Take a screenshot and describe what you see"- }- }- await connection.send(json.dumps(http_request))-- @connection.on(ConnectionEventType.Data)- async def connection_data(data):- response = json.loads(data)- # Response format: {"status": 200, "headers": {...}, "body": {...}}- print(f"Status: {response['status']}")- print(f"Body: {response['body']}")--asyncio.run(http_over_p2p())-```--## Client Connection (JavaScript/Browser)--### Basic Browser Client-```javascript-import Peer from 'peerjs';--// Create peer connection-const peer = new Peer('browser-client', {- host: 'localhost',- port: 9000,- path: '/peerjs'-});--// Connect to agent proxy-const conn = peer.connect('computer-agent-proxy');--conn.on('open', () => {- console.log('Connected to agent proxy');-- // Send agent request- const request = {- model: 'anthropic/claude-3-5-sonnet-20241022',- input: 'Hello from browser!'- };- conn.send(JSON.stringify(request));-});--conn.on('data', (data) => {- const response = JSON.parse(data);- console.log('Agent response:', response);-});--conn.on('error', (err) => {- console.error('Connection error:', err);-});-```--### Multi-modal Browser Request-```javascript-conn.on('open', () => {- const request = {- model: 'anthropic/claude-3-5-sonnet-20241022',- input: [- {- role: 'user',- content: [- { type: 'input_text', text: 'Analyze this image' },- {- type: 'input_image',- image_url: 'https://example.com/image.jpg'- }- ]- }- ]- };- conn.send(JSON.stringify(request));-});-```--## Message Formats--### Direct Agent Request-Send agent requests directly as JSON:-```json-{- "model": "anthropic/claude-3-5-sonnet-20241022",- "input": "Your instruction here",- "agent_kwargs": {- "save_trajectory": true- }-}-```--### HTTP-like Request-Send HTTP-style requests for REST API compatibility:-```json-{- "method": "POST",- "path": "/responses",- "body": {- "model": "anthropic/claude-3-5-sonnet-20241022",- "input": "Your instruction here"- }-}-```--## Response Formats--### Direct Response-```json-{- "success": true,- "result": {- // Agent response data- },- "model": "anthropic/claude-3-5-sonnet-20241022"-}-```--### HTTP-like Response-```json-{- "status": 200,- "headers": {- "Content-Type": "application/json"- },- "body": {- "success": true,- "result": {- // Agent response data- }- }-}-```--### Welcome Message-Upon connection, the server sends a welcome message:-```json-{- "type": "welcome",- "message": "Connected to ComputerAgent Proxy",- "endpoints": ["/responses"]-}-```--## Connection Events--### Client Events-- `open`: Connection established-- `data`: Data received from agent-- `close`: Connection closed-- `error`: Connection error--### Server Events-- `connection`: New peer connection-- `error`: Server error--## Configuration--### ICE Servers-Default STUN servers for NAT traversal:-```javascript-[- { urls: "stun:stun.l.google.com:19302" },- { urls: "stun:stun1.l.google.com:19302" }-]-```--### Custom Configuration-```python-options = PeerOptions(- host="your-signaling-server.com",- port=443,- secure=True,- config=RTCConfiguration(- iceServers=[- RTCIceServer(urls="stun:stun.l.google.com:19302"),- RTCIceServer(- urls="turn:your-turn-server.com:3478",- username="user",- credential="pass"- )- ]- )-)-```--## Error Handling--### Connection Errors-```python[email protected](ConnectionEventType.Error)-async def connection_error(error):- print(f"Connection error: {error}")- # Implement reconnection logic-```--### Request Errors-Invalid requests return error responses:-```json-{- "success": false,- "error": "Invalid JSON in request"-}-```--## Security Considerations--- WebRTC connections are encrypted by default-- Signaling server should use HTTPS/WSS in production-- Consider implementing peer authentication-- Use TURN servers for better connectivity in restrictive networksdiff --git a/docs/content/docs/libraries/agent/REST-API.mdx b/docs/content/docs/libraries/agent/REST-API.mdxdeleted file mode 100644index 9341c247d6..0000000000--- a/docs/content/docs/libraries/agent/REST-API.mdx+++ /dev/null@@ -1,166 +0,0 @@-----title: REST Proxy API Reference-description: Reference for the /responses REST endpoint of the Agent Proxy Server.------The Agent Proxy Server exposes a REST endpoint for ComputerAgent execution:--- `http://localhost:8000/responses`--## Starting HTTP Server--```bash-python -m agent.proxy.cli --mode http-```--## POST /responses--- Accepts agent requests as JSON in the request body-- Returns the first result from ComputerAgent execution-- Supports both simple text and multi-modal inputs--### Request Format-```json-{- "model": "anthropic/claude-3-5-sonnet-20241022",- "input": "Your instruction here",- "agent_kwargs": {- "save_trajectory": true,- "verbosity": 20- },- "computer_kwargs": {- "os_type": "linux",- "provider_type": "cloud"- }-}-```--### Multi-modal Request Format-```json-{- "model": "anthropic/claude-3-5-sonnet-20241022",- "input": [- {- "role": "user",- "content": [- {"type": "input_text", "text": "what is in this image?"},- {- "type": "input_image",- "image_url": "https://example.com/image.jpg"- }- ]- }- ]-}-```--### Required Parameters-- `model`: Model string (e.g., "anthropic/claude-3-5-sonnet-20241022")-- `input`: String or message array with the user's instruction--### Optional Parameters-- `agent_kwargs`: Dictionary of ComputerAgent configuration options-- `computer_kwargs`: Dictionary of Computer instance configuration options--### Example Request (Python)-```python-import requests--url = "http://localhost:8000/responses"-body = {- "model": "anthropic/claude-3-5-sonnet-20241022",- "input": "Take a screenshot and tell me what you see"-}-resp = requests.post(url, json=body)-print(resp.json())-```--### Example Request (cURL)-```bash-curl http://localhost:8000/responses \- -H "Content-Type: application/json" \- -d '{- "model": "anthropic/claude-3-5-sonnet-20241022",- "input": "Tell me a three sentence bedtime story about a unicorn."- }'-```--### Multi-modal Example (cURL)-```bash-curl http://localhost:8000/responses \- -H "Content-Type: application/json" \- -d '{- "model": "anthropic/claude-3-5-sonnet-20241022",- "input": [- {- "role": "user",- "content": [- {"type": "input_text", "text": "what is in this image?"},- {- "type": "input_image",- "image_url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"- }- ]- }- ]- }'-```--### Response Format-```json-{- "success": true,- "result": {- // Agent response data- },- "model": "anthropic/claude-3-5-sonnet-20241022"-}-```--### Error Response Format-```json-{- "success": false,- "error": "Error description",- "model": "anthropic/claude-3-5-sonnet-20241022"-}-```--## GET /health--Health check endpoint that returns server status.--### Response Format-```json-{- "status": "healthy"-}-```--## Configuration--### Environment Variables-- `CUA_CONTAINER_NAME`: Default container name for cloud provider-- `CUA_API_KEY`: Default API key for cloud provider--### Agent Configuration (`agent_kwargs`)-Common options:-- `save_trajectory`: Boolean - Save conversation trajectory-- `verbosity`: Integer - Logging level (10=DEBUG, 20=INFO, etc.)-- `max_trajectory_budget`: Float - Budget limit for trajectory-- `telemetry_enabled`: Boolean or Dict - Enable telemetry tracking--### Computer Configuration (`computer_kwargs`)-Common options:-- `os_type`: String - "linux", "windows", "macos"-- `provider_type`: String - "cloud", "local", "docker"-- `name`: String - Instance name-- `api_key`: String - Provider API key--## Error Handling--Common error scenarios:-- **400 Bad Request**: Invalid JSON or missing required parameters-- **500 Internal Server Error**: Agent execution errors or computer setup failures--All errors return a structured JSON response with `success: false` and an `error` field describing the issue.diff --git a/docs/content/docs/libraries/agent/index.mdx b/docs/content/docs/libraries/agent/index.mdxindex b45485a755..f0e1ab77b0 100644--- a/docs/content/docs/libraries/agent/index.mdx+++ b/docs/content/docs/libraries/agent/index.mdx@@ -114,37 +114,3 @@ agent = ComputerAgent(callbacks=[ImageRetentionCallback(only_n_most_recent_images=3)])```--## Proxy Server API--The Agent library includes a proxy server that exposes ComputerAgent functionality over HTTP and P2P (WebRTC) connections, allowing remote clients to interact with agents through REST-like APIs.--### Starting the Proxy Server--```bash-# HTTP server (default)-python -m agent.proxy.cli--# P2P server with WebRTC-python -m agent.proxy.cli --mode p2p --peer-id computer-agent-proxy--# Both HTTP and P2P-python -m agent.proxy.cli --mode both --peer-id computer-agent-proxy-```--### API References--- [REST API Reference](./REST-API) - HTTP endpoints for agent execution-- [PeerJS API Reference](./PeerJS-API) - P2P WebRTC connections for direct browser-to-agent communication--### Quick Example--```bash-curl http://localhost:8000/responses \- -H "Content-Type: application/json" \- -d '{- "model": "anthropic/claude-3-5-sonnet-20241022",- "input": "Take a screenshot and tell me what you see"- }'-```-```