Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

added proxy docs

Dillon DuPont61cd5e22cc88
diff --git a/docs/content/docs/libraries/agent/PeerJS-API.mdx b/docs/content/docs/libraries/agent/PeerJS-API.mdx
new file mode 100644
index 0000000000..f3edea6316
--- /dev/null
+++ b/docs/content/docs/libraries/agent/PeerJS-API.mdx
@@ -0,0 +1,306 @@
+---
+title: PeerJS Proxy API Reference
+description: Reference for the P2P WebRTC endpoint of the Agent Proxy Server.
+---
+
+The Agent Proxy Server supports peer-to-peer (P2P) connections using WebRTC through the PeerJS protocol. This allows direct browser-to-agent communication without intermediary servers.
+
+## Connection Setup
+
+### Prerequisites
+- Install peerjs-python: `pip install peerjs-python aiortc`
+- Running signaling server (default: localhost:9000)
+
+### Starting P2P Server
+```bash
+# P2P only
+python -m agent.proxy.cli --mode p2p --peer-id computer-agent-proxy
+
+# Both HTTP and P2P
+python -m agent.proxy.cli --mode both --peer-id computer-agent-proxy
+
+# Custom signaling server
+python -m agent.proxy.cli --mode p2p \
+ --peer-id my-agent \
+ --signaling-host signaling.example.com \
+ --signaling-port 443 \
+ --signaling-secure
+```
+
+## Client Connection (Python)
+
+### Basic Connection
+```python
+import asyncio
+import json
+from peerjs import Peer, PeerOptions, ConnectionEventType
+from aiortc import RTCConfiguration, RTCIceServer
+
+async def connect_to_agent():
+ # Set up peer options
+ options = PeerOptions(
+ host="localhost",
+ port=9000,
+ secure=False,
+ config=RTCConfiguration(
+ iceServers=[RTCIceServer(urls="stun:stun.l.google.com:19302")]
+ )
+ )
+
+ # Create client peer
+ client_peer = Peer(id="my-client", peer_options=options)
+ await client_peer.start()
+
+ # Connect to agent proxy
+ connection = client_peer.connect("computer-agent-proxy")
+
+ @connection.on(ConnectionEventType.Open)
+ async def connection_open():
+ print("Connected to agent proxy")
+
+ # Send agent request
+ request = {
+ "model": "anthropic/claude-3-5-sonnet-20241022",
+ "input": "Hello from P2P client!"
+ }
+ await connection.send(json.dumps(request))
+
+ @connection.on(ConnectionEventType.Data)
+ async def connection_data(data):
+ response = json.loads(data) if isinstance(data, str) else data
+ print(f"Agent response: {response}")
+ await client_peer.destroy()
+
+ # Keep connection alive
+ await asyncio.sleep(10)
+
+asyncio.run(connect_to_agent())
+```
+
+### HTTP-like Requests over P2P
+```python
+import asyncio
+import json
+from peerjs import Peer, PeerOptions, ConnectionEventType
+
+async def http_over_p2p():
+ # Connect to peer
+ client_peer = Peer(id="http-client", peer_options=options)
+ await client_peer.start()
+ connection = client_peer.connect("computer-agent-proxy")
+
+ @connection.on(ConnectionEventType.Open)
+ async def connection_open():
+ # Send HTTP-like request
+ http_request = {
+ "method": "POST",
+ "path": "/responses",
+ "body": {
+ "model": "anthropic/claude-3-5-sonnet-20241022",
+ "input": "Take a screenshot and describe what you see"
+ }
+ }
+ await connection.send(json.dumps(http_request))
+
+ @connection.on(ConnectionEventType.Data)
+ async def connection_data(data):
+ response = json.loads(data)
+ # Response format: {"status": 200, "headers": {...}, "body": {...}}
+ print(f"Status: {response['status']}")
+ print(f"Body: {response['body']}")
+
+asyncio.run(http_over_p2p())
+```
+
+## Client Connection (JavaScript/Browser)
+
+### Basic Browser Client
+```javascript
+import Peer from 'peerjs';
+
+// Create peer connection
+const peer = new Peer('browser-client', {
+ host: 'localhost',
+ port: 9000,
+ path: '/peerjs'
+});
+
+// Connect to agent proxy
+const conn = peer.connect('computer-agent-proxy');
+
+conn.on('open', () => {
+ console.log('Connected to agent proxy');
+
+ // Send agent request
+ const request = {
+ model: 'anthropic/claude-3-5-sonnet-20241022',
+ input: 'Hello from browser!'
+ };
+ conn.send(JSON.stringify(request));
+});
+
+conn.on('data', (data) => {
+ const response = JSON.parse(data);
+ console.log('Agent response:', response);
+});
+
+conn.on('error', (err) => {
+ console.error('Connection error:', err);
+});
+```
+
+### Multi-modal Browser Request
+```javascript
+conn.on('open', () => {
+ const request = {
+ model: 'anthropic/claude-3-5-sonnet-20241022',
+ input: [
+ {
+ role: 'user',
+ content: [
+ { type: 'input_text', text: 'Analyze this image' },
+ {
+ type: 'input_image',
+ image_url: 'https://example.com/image.jpg'
+ }
+ ]
+ }
+ ]
+ };
+ conn.send(JSON.stringify(request));
+});
+```
+
+## Message Formats
+
+### Direct Agent Request
+Send agent requests directly as JSON:
+```json
+{
+ "model": "anthropic/claude-3-5-sonnet-20241022",
+ "input": "Your instruction here",
+ "agent_kwargs": {
+ "save_trajectory": true
+ }
+}
+```
+
+### HTTP-like Request
+Send HTTP-style requests for REST API compatibility:
+```json
+{
+ "method": "POST",
+ "path": "/responses",
+ "body": {
+ "model": "anthropic/claude-3-5-sonnet-20241022",
+ "input": "Your instruction here"
+ }
+}
+```
+
+## Response Formats
+
+### Direct Response
+```json
+{
+ "success": true,
+ "result": {
+ // Agent response data
+ },
+ "model": "anthropic/claude-3-5-sonnet-20241022"
+}
+```
+
+### HTTP-like Response
+```json
+{
+ "status": 200,
+ "headers": {
+ "Content-Type": "application/json"
+ },
+ "body": {
+ "success": true,
+ "result": {
+ // Agent response data
+ }
+ }
+}
+```
+
+### Welcome Message
+Upon connection, the server sends a welcome message:
+```json
+{
+ "type": "welcome",
+ "message": "Connected to ComputerAgent Proxy",
+ "endpoints": ["/responses"]
+}
+```
+
+## Connection Events
+
+### Client Events
+- `open`: Connection established
+- `data`: Data received from agent
+- `close`: Connection closed
+- `error`: Connection error
+
+### Server Events
+- `connection`: New peer connection
+- `error`: Server error
+
+## Configuration
+
+### ICE Servers
+Default STUN servers for NAT traversal:
+```javascript
+[
+ { urls: "stun:stun.l.google.com:19302" },
+ { urls: "stun:stun1.l.google.com:19302" }
+]
+```
+
+### Custom Configuration
+```python
+options = PeerOptions(
+ host="your-signaling-server.com",
+ port=443,
+ secure=True,
+ config=RTCConfiguration(
+ iceServers=[
+ RTCIceServer(urls="stun:stun.l.google.com:19302"),
+ RTCIceServer(
+ urls="turn:your-turn-server.com:3478",
+ username="user",
+ credential="pass"
+ )
+ ]
+ )
+)
+```
+
+## Error Handling
+
+### Connection Errors
+```python
[email protected](ConnectionEventType.Error)
+async def connection_error(error):
+ print(f"Connection error: {error}")
+ # Implement reconnection logic
+```
+
+### Request Errors
+Invalid requests return error responses:
+```json
+{
+ "success": false,
+ "error": "Invalid JSON in request"
+}
+```
+
+## Security Considerations
+
+- WebRTC connections are encrypted by default
+- Signaling server should use HTTPS/WSS in production
+- Consider implementing peer authentication
+- Use TURN servers for better connectivity in restrictive networks
diff --git a/docs/content/docs/libraries/agent/REST-API.mdx b/docs/content/docs/libraries/agent/REST-API.mdx
new file mode 100644
index 0000000000..9341c247d6
--- /dev/null
+++ b/docs/content/docs/libraries/agent/REST-API.mdx
@@ -0,0 +1,166 @@
+---
+title: REST Proxy API Reference
+description: Reference for the /responses REST endpoint of the Agent Proxy Server.
+---
+
+The Agent Proxy Server exposes a REST endpoint for ComputerAgent execution:
+
+- `http://localhost:8000/responses`
+
+## Starting HTTP Server
+
+```bash
+python -m agent.proxy.cli --mode http
+```
+
+## POST /responses
+
+- Accepts agent requests as JSON in the request body
+- Returns the first result from ComputerAgent execution
+- Supports both simple text and multi-modal inputs
+
+### Request Format
+```json
+{
+ "model": "anthropic/claude-3-5-sonnet-20241022",
+ "input": "Your instruction here",
+ "agent_kwargs": {
+ "save_trajectory": true,
+ "verbosity": 20
+ },
+ "computer_kwargs": {
+ "os_type": "linux",
+ "provider_type": "cloud"
+ }
+}
+```
+
+### Multi-modal Request Format
+```json
+{
+ "model": "anthropic/claude-3-5-sonnet-20241022",
+ "input": [
+ {
+ "role": "user",
+ "content": [
+ {"type": "input_text", "text": "what is in this image?"},
+ {
+ "type": "input_image",
+ "image_url": "https://example.com/image.jpg"
+ }
+ ]
+ }
+ ]
+}
+```
+
+### Required Parameters
+- `model`: Model string (e.g., "anthropic/claude-3-5-sonnet-20241022")
+- `input`: String or message array with the user's instruction
+
+### Optional Parameters
+- `agent_kwargs`: Dictionary of ComputerAgent configuration options
+- `computer_kwargs`: Dictionary of Computer instance configuration options
+
+### Example Request (Python)
+```python
+import requests
+
+url = "http://localhost:8000/responses"
+body = {
+ "model": "anthropic/claude-3-5-sonnet-20241022",
+ "input": "Take a screenshot and tell me what you see"
+}
+resp = requests.post(url, json=body)
+print(resp.json())
+```
+
+### Example Request (cURL)
+```bash
+curl http://localhost:8000/responses \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "anthropic/claude-3-5-sonnet-20241022",
+ "input": "Tell me a three sentence bedtime story about a unicorn."
+ }'
+```
+
+### Multi-modal Example (cURL)
+```bash
+curl http://localhost:8000/responses \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "anthropic/claude-3-5-sonnet-20241022",
+ "input": [
+ {
+ "role": "user",
+ "content": [
+ {"type": "input_text", "text": "what is in this image?"},
+ {
+ "type": "input_image",
+ "image_url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
+ }
+ ]
+ }
+ ]
+ }'
+```
+
+### Response Format
+```json
+{
+ "success": true,
+ "result": {
+ // Agent response data
+ },
+ "model": "anthropic/claude-3-5-sonnet-20241022"
+}
+```
+
+### Error Response Format
+```json
+{
+ "success": false,
+ "error": "Error description",
+ "model": "anthropic/claude-3-5-sonnet-20241022"
+}
+```
+
+## GET /health
+
+Health check endpoint that returns server status.
+
+### Response Format
+```json
+{
+ "status": "healthy"
+}
+```
+
+## Configuration
+
+### Environment Variables
+- `CUA_CONTAINER_NAME`: Default container name for cloud provider
+- `CUA_API_KEY`: Default API key for cloud provider
+
+### Agent Configuration (`agent_kwargs`)
+Common options:
+- `save_trajectory`: Boolean - Save conversation trajectory
+- `verbosity`: Integer - Logging level (10=DEBUG, 20=INFO, etc.)
+- `max_trajectory_budget`: Float - Budget limit for trajectory
+- `telemetry_enabled`: Boolean or Dict - Enable telemetry tracking
+
+### Computer Configuration (`computer_kwargs`)
+Common options:
+- `os_type`: String - "linux", "windows", "macos"
+- `provider_type`: String - "cloud", "local", "docker"
+- `name`: String - Instance name
+- `api_key`: String - Provider API key
+
+## Error Handling
+
+Common error scenarios:
+- **400 Bad Request**: Invalid JSON or missing required parameters
+- **500 Internal Server Error**: Agent execution errors or computer setup failures
+
+All errors return a structured JSON response with `success: false` and an `error` field describing the issue.
diff --git a/docs/content/docs/libraries/agent/index.mdx b/docs/content/docs/libraries/agent/index.mdx
index f0e1ab77b0..b45485a755 100644
--- a/docs/content/docs/libraries/agent/index.mdx
+++ b/docs/content/docs/libraries/agent/index.mdx
@@ -114,3 +114,37 @@ agent = ComputerAgent(
callbacks=[ImageRetentionCallback(only_n_most_recent_images=3)]
)
```
+
+## Proxy Server API
+
+The Agent library includes a proxy server that exposes ComputerAgent functionality over HTTP and P2P (WebRTC) connections, allowing remote clients to interact with agents through REST-like APIs.
+
+### Starting the Proxy Server
+
+```bash
+# HTTP server (default)
+python -m agent.proxy.cli
+
+# P2P server with WebRTC
+python -m agent.proxy.cli --mode p2p --peer-id computer-agent-proxy
+
+# Both HTTP and P2P
+python -m agent.proxy.cli --mode both --peer-id computer-agent-proxy
+```
+
+### API References
+
+- [REST API Reference](./REST-API) - HTTP endpoints for agent execution
+- [PeerJS API Reference](./PeerJS-API) - P2P WebRTC connections for direct browser-to-agent communication
+
+### Quick Example
+
+```bash
+curl http://localhost:8000/responses \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "anthropic/claude-3-5-sonnet-20241022",
+ "input": "Take a screenshot and tell me what you see"
+ }'
+```
+```