Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(telemetry): add provider_type to computer initialization events (#921)
Sarina Lie008bcbfb905
diff --git a/README.md b/README.mdindex 2fbc871393..55da68996c 100644--- a/README.md+++ b/README.md@@ -164,15 +164,15 @@ lume run macos-sequoia-vanilla:latest## Packages-| Package | Description |-| --------------------------------------------------------------------- | ---------------------------------------------------------- |-| [cuabot](https://docs.trycua.com/cuabot/guide/getting-started/introduction) | Multi-agent computer-use sandbox CLI |-| [cua-agent](https://cua.ai/docs/cua/reference/agent-sdk) | AI agent framework for computer-use tasks |-| [cua-computer](https://cua.ai/docs/cua/reference/computer-sdk) | SDK for controlling desktop environments |-| [cua-computer-server](https://cua.ai/docs/cua/reference/computer-sdk) | Driver for UI interactions and code execution in sandboxes |-| [cua-bench](https://cua.ai/docs/cuabench) | Benchmarks and RL environments for computer-use |-| [lume](https://cua.ai/docs/lume) | macOS/Linux VM management on Apple Silicon |-| [lumier](https://cua.ai/docs/lume/guide/advanced/lumier) | Docker-compatible interface for Lume VMs |+| Package | Description |+| --------------------------------------------------------------------------- | ---------------------------------------------------------- |+| [cuabot](https://docs.trycua.com/cuabot/guide/getting-started/introduction) | Multi-agent computer-use sandbox CLI |+| [cua-agent](https://cua.ai/docs/cua/reference/agent-sdk) | AI agent framework for computer-use tasks |+| [cua-computer](https://cua.ai/docs/cua/reference/computer-sdk) | SDK for controlling desktop environments |+| [cua-computer-server](https://cua.ai/docs/cua/reference/computer-sdk) | Driver for UI interactions and code execution in sandboxes |+| [cua-bench](https://cua.ai/docs/cuabench) | Benchmarks and RL environments for computer-use |+| [lume](https://cua.ai/docs/lume) | macOS/Linux VM management on Apple Silicon |+| [lumier](https://cua.ai/docs/lume/guide/advanced/lumier) | Docker-compatible interface for Lume VMs |## Resourcesdiff --git a/docs/content/docs/cua/reference/agent-sdk/index.mdx b/docs/content/docs/cua/reference/agent-sdk/index.mdxindex 7ab38a8ff6..16ef2f1808 100644--- a/docs/content/docs/cua/reference/agent-sdk/index.mdx+++ b/docs/content/docs/cua/reference/agent-sdk/index.mdx@@ -7,7 +7,7 @@ description: Python API reference for building computer-use agentsAUTO-GENERATED FILE - DO NOT EDIT DIRECTLYGenerated by: npx tsx scripts/docs-generators/python-sdk.tsSource: libs/python/agent/agent- Version: 0.7.29+ Version: 0.7.30*/}import { Callout } from 'fumadocs-ui/components/callout';@@ -17,7 +17,7 @@ import { VersionHeader } from '@/components/version-selector';<VersionHeaderversions={[{"version":"0.7","href":"/cua/reference/agent-sdk","isCurrent":true},{"version":"0.6","href":"/cua/reference/agent-sdk/v0.6/api","isCurrent":false},{"version":"0.5","href":"/cua/reference/agent-sdk/v0.5/api","isCurrent":false},{"version":"0.4","href":"/cua/reference/agent-sdk/v0.4/api","isCurrent":false},{"version":"0.3","href":"/cua/reference/agent-sdk/v0.3/api","isCurrent":false}]}currentVersion="0.7"- fullVersion="0.7.29"+ fullVersion="0.7.30"packageName="cua-agent"/>diff --git a/docs/content/docs/cua/reference/cli/index.mdx b/docs/content/docs/cua/reference/cli/index.mdxindex e24add0bfe..165ac3ee38 100644--- a/docs/content/docs/cua/reference/cli/index.mdx+++ b/docs/content/docs/cua/reference/cli/index.mdx@@ -7,7 +7,7 @@ description: Python API reference for the Cua command-line interfaceAUTO-GENERATED FILE - DO NOT EDIT DIRECTLYGenerated by: npx tsx scripts/docs-generators/python-sdk.tsSource: libs/python/cua-cli/cua_cli- Version: 0.1.6+ Version: 0.1.7*/}import { Callout } from 'fumadocs-ui/components/callout';@@ -17,7 +17,7 @@ import { VersionHeader } from '@/components/version-selector';<VersionHeaderversions={[{"version":"0.1","href":"/cua/reference/cli","isCurrent":true}]}currentVersion="0.1"- fullVersion="0.1.6"+ fullVersion="0.1.7"packageName="cua-cli"/>diff --git a/libs/python/agent/tests/test_telemetry_events.py b/libs/python/agent/tests/test_telemetry_events.pynew file mode 100644index 0000000000..0ad8396b5b--- /dev/null+++ b/libs/python/agent/tests/test_telemetry_events.py@@ -0,0 +1,145 @@+"""+Test script to verify telemetry events are emitted correctly.+"""++from unittest.mock import AsyncMock, MagicMock, patch++import pytest+++class TestAgentTelemetryEvents:+ """Test telemetry events emitted by ComputerAgent."""++ @patch("agent.agent.record_event")+ @patch("agent.agent.is_telemetry_enabled", return_value=True)+ def test_agent_init_event(self, mock_telemetry_enabled, mock_record_event):+ """Test that agent_init event is emitted with correct args_provided."""+ from agent.agent import ComputerAgent++ # Create agent with various args+ agent = ComputerAgent(+ model="anthropic/claude-sonnet-4-5-20250929",+ instructions="Test instructions",+ max_retries=5, # non-default+ trajectory_dir="/tmp/test",+ )++ # Find the agent_init call+ agent_init_calls = [+ call for call in mock_record_event.call_args_list if call[0][0] == "agent_init"+ ]++ assert len(agent_init_calls) == 1, "agent_init should be called once"++ event_name, event_data = agent_init_calls[0][0]+ assert event_name == "agent_init"+ assert event_data["model"] == "anthropic/claude-sonnet-4-5-20250929"+ assert "instructions" in event_data["args_provided"]+ assert "max_retries" in event_data["args_provided"]+ assert "trajectory_dir" in event_data["args_provided"]++ @patch("agent.agent.record_event")+ @patch("agent.agent.is_telemetry_enabled", return_value=True)+ def test_agent_init_minimal_args(self, mock_telemetry_enabled, mock_record_event):+ """Test agent_init with minimal args (defaults)."""+ from agent.agent import ComputerAgent++ agent = ComputerAgent(model="anthropic/claude-sonnet-4-5-20250929")++ agent_init_calls = [+ call for call in mock_record_event.call_args_list if call[0][0] == "agent_init"+ ]++ assert len(agent_init_calls) == 1+ event_name, event_data = agent_init_calls[0][0]++ # With defaults, only model-related things should be tracked+ # instructions, trajectory_dir, etc. should NOT be in args_provided+ assert "instructions" not in event_data["args_provided"]+ assert "trajectory_dir" not in event_data["args_provided"]+ assert "max_retries" not in event_data["args_provided"] # default is 3++ @patch("agent.agent.record_event")+ @patch("agent.agent.is_telemetry_enabled", return_value=False)+ def test_no_events_when_telemetry_disabled(self, mock_telemetry_enabled, mock_record_event):+ """Test that no events are emitted when telemetry is disabled."""+ from agent.agent import ComputerAgent++ agent = ComputerAgent(+ model="anthropic/claude-sonnet-4-5-20250929",+ telemetry_enabled=False,+ )++ # No agent_init should be called (telemetry disabled)+ agent_init_calls = [+ call for call in mock_record_event.call_args_list if call[0][0] == "agent_init"+ ]++ assert len(agent_init_calls) == 0+++class TestActionTelemetryEvents:+ """Test telemetry events for computer actions."""++ @pytest.mark.asyncio+ @patch("agent.agent.record_event")+ @patch("agent.agent.is_telemetry_enabled", return_value=True)+ async def test_computer_action_executed_event(self, mock_telemetry_enabled, mock_record_event):+ """Test that computer_action_executed is emitted for computer calls."""+ from agent.agent import ComputerAgent++ agent = ComputerAgent(model="anthropic/claude-sonnet-4-5-20250929")+ agent.telemetry_enabled = True++ # Mock computer handler+ mock_computer = MagicMock()+ mock_computer.click = AsyncMock(return_value=None)+ mock_computer.screenshot = AsyncMock(return_value="base64screenshot")++ # Create a mock computer_call item+ item = {+ "type": "computer_call",+ "call_id": "test-call-id",+ "action": {+ "type": "click",+ "x": 100,+ "y": 200,+ },+ }++ # Process the item (this would normally happen in the agent loop)+ # Note: We can't easily test this without running the full agent loop+ # This is more of an integration test++ # For unit testing, we verify the event structure+ expected_event = {+ "action_type": "click",+ }++ # Verify event structure is correct+ assert "action_type" in expected_event+++class TestToolExecutedEvents:+ """Test telemetry events for tool execution."""++ def test_event_structure(self):+ """Test that agent_tool_executed event has correct structure."""+ expected_computer_tool_event = {+ "tool_type": "computer",+ "tool_name": "click",+ }++ expected_function_tool_event = {+ "tool_type": "function",+ "tool_name": "my_custom_function",+ }++ # Verify expected structure+ assert "tool_type" in expected_computer_tool_event+ assert "tool_name" in expected_computer_tool_event+ assert expected_computer_tool_event["tool_type"] in ["computer", "function"]+++if __name__ == "__main__":+ pytest.main([__file__, "-v"])diff --git a/libs/python/computer/computer/computer.py b/libs/python/computer/computer/computer.pyindex 1061fb52b5..599b8dbfd9 100644--- a/libs/python/computer/computer/computer.py+++ b/libs/python/computer/computer/computer.py@@ -294,7 +294,17 @@ class Computer:# Record initialization in telemetry (if enabled)if telemetry_enabled and is_telemetry_enabled():- record_event("computer_initialized", SYSTEM_INFO)+ record_event(+ "computer_initialized",+ {+ **SYSTEM_INFO,+ "provider_type": (+ self.provider_type.value+ if isinstance(self.provider_type, VMProviderType)+ else str(self.provider_type) if self.provider_type else "unknown"+ ),+ },+ )else:self.logger.debug("Telemetry disabled - skipping initialization tracking")diff --git a/libs/typescript/computer/src/computer/providers/base.ts b/libs/typescript/computer/src/computer/providers/base.tsindex a4e863522e..fb91082a66 100644--- a/libs/typescript/computer/src/computer/providers/base.ts+++ b/libs/typescript/computer/src/computer/providers/base.ts@@ -27,6 +27,7 @@ export abstract class BaseComputer {constructor(config: BaseComputerConfig) {this.name = config.name;this.osType = config.osType;+ this.vmProvider = config.vmProvider;this.telemetry = new Telemetry();this.telemetry.recordEvent('module_init', {module: 'computer',@@ -38,6 +39,7 @@ export abstract class BaseComputer {os: os.platform(),os_version: os.version(),node_version: process.version,+ provider_type: config.vmProvider ?? 'unknown',});// Track which config args were provided@@ -45,10 +47,12 @@ export abstract class BaseComputer {const apiKey = 'apiKey' in config ? (config as { apiKey?: string }).apiKey : undefined;if (apiKey) argsProvided.push('apiKey');if ('apiBase' in config) argsProvided.push('apiBase');+ if (config.vmProvider) argsProvided.push('vmProvider');const initEventData: Record<string, unknown> = {os_type: config.osType,args_provided: argsProvided,+ provider_type: config.vmProvider ?? 'unknown',};// Add hashed API keyif (apiKey) {diff --git a/libs/typescript/computer/src/computer/types.ts b/libs/typescript/computer/src/computer/types.tsindex 339d52ac8c..42b48c78cd 100644--- a/libs/typescript/computer/src/computer/types.ts+++ b/libs/typescript/computer/src/computer/types.ts@@ -22,6 +22,11 @@ export interface BaseComputerConfig {* @default "macos"*/osType: OSType;++ /**+ * The VM provider type+ */+ vmProvider?: VMProviderType;}export interface CloudComputerConfig extends BaseComputerConfig {diff --git a/scripts/docs-generators/python-sdk.ts b/scripts/docs-generators/python-sdk.tsindex a92e6dbee0..776bce5459 100644--- a/scripts/docs-generators/python-sdk.ts+++ b/scripts/docs-generators/python-sdk.ts@@ -199,10 +199,7 @@ async function main() {let docs: PythonPackage;try {// Prefer uv run --with griffe python (works cross-platform), fall back to python3- const pythonCmd =- process.platform === 'win32'- ? `uv run --with griffe python`- : `python3`;+ const pythonCmd = process.platform === 'win32' ? `uv run --with griffe python` : `python3`;const output = execSync(`${pythonCmd} "${PYTHON_SCRIPT}" "${packagePath}" "${config.packageName}"`,{diff --git a/skills/gui-automation/SKILL.md b/skills/gui-automation/SKILL.mdindex b6953e1d29..a90a152c1a 100644--- a/skills/gui-automation/SKILL.md+++ b/skills/gui-automation/SKILL.md@@ -125,37 +125,37 @@ Tell the user: `"Here is the trajectory of my session: {url}"`## Quick Reference-| Action | Command |-| ------------------- | ---------------------------------------------- |-| Connect to target | `cua do switch <provider> [name]` |-| Screenshot | `cua do screenshot` |-| AI-annotated screen | `cua do snapshot ["instructions"]` |-| Click | `cua do click <x> <y> [left\|right\|middle]` |-| Double-click | `cua do dclick <x> <y>` |-| Type text | `cua do type "text"` |-| Press key | `cua do key <key>` |-| Hotkey | `cua do hotkey <combo>` (e.g. `ctrl+c`) |-| Scroll | `cua do scroll <direction> [amount]` |-| Drag | `cua do drag <x1> <y1> <x2> <y2>` |-| Move cursor | `cua do move <x> <y>` |-| Shell command | `cua do shell "command"` |-| Open URL/file | `cua do open <url\|path>` |-| List windows | `cua do window ls [app]` |-| Focus window | `cua do window focus <id>` |-| Zoom to window | `cua do zoom "App Name"` |-| Unzoom | `cua do unzoom` |-| Share trajectory | `cua trajectory share` |+| Action | Command |+| ------------------- | -------------------------------------------- |+| Connect to target | `cua do switch <provider> [name]` |+| Screenshot | `cua do screenshot` |+| AI-annotated screen | `cua do snapshot ["instructions"]` |+| Click | `cua do click <x> <y> [left\|right\|middle]` |+| Double-click | `cua do dclick <x> <y>` |+| Type text | `cua do type "text"` |+| Press key | `cua do key <key>` |+| Hotkey | `cua do hotkey <combo>` (e.g. `ctrl+c`) |+| Scroll | `cua do scroll <direction> [amount]` |+| Drag | `cua do drag <x1> <y1> <x2> <y2>` |+| Move cursor | `cua do move <x> <y>` |+| Shell command | `cua do shell "command"` |+| Open URL/file | `cua do open <url\|path>` |+| List windows | `cua do window ls [app]` |+| Focus window | `cua do window focus <id>` |+| Zoom to window | `cua do zoom "App Name"` |+| Unzoom | `cua do unzoom` |+| Share trajectory | `cua trajectory share` |## Providers-| Provider | Example |-| ------------ | ------------------------------------- |-| `cloud` | `cua do switch cloud my-vm` |-| `cloudv2` | `cua do switch cloudv2 my-vm` |-| `docker` | `cua do switch docker my-container` |-| `lume` | `cua do switch lume my-vm` |-| `lumier` | `cua do switch lumier my-vm` |-| `winsandbox` | `cua do switch winsandbox` |-| `host` | `cua do switch host` |+| Provider | Example |+| ------------ | ----------------------------------- |+| `cloud` | `cua do switch cloud my-vm` |+| `cloudv2` | `cua do switch cloudv2 my-vm` |+| `docker` | `cua do switch docker my-container` |+| `lume` | `cua do switch lume my-vm` |+| `lumier` | `cua do switch lumier my-vm` |+| `winsandbox` | `cua do switch winsandbox` |+| `host` | `cua do switch host` |See [references/command-reference.md](references/command-reference.md) for full argument syntax.