Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(telemetry): add provider_type to computer initialization events (#921)

Sarina Lie008bcbfb905
diff --git a/README.md b/README.md
index 2fbc871393..55da68996c 100644
--- a/README.md
+++ b/README.md
@@ -164,15 +164,15 @@ lume run macos-sequoia-vanilla:latest
## Packages
-| Package | Description |
-| --------------------------------------------------------------------- | ---------------------------------------------------------- |
-| [cuabot](https://docs.trycua.com/cuabot/guide/getting-started/introduction) | Multi-agent computer-use sandbox CLI |
-| [cua-agent](https://cua.ai/docs/cua/reference/agent-sdk) | AI agent framework for computer-use tasks |
-| [cua-computer](https://cua.ai/docs/cua/reference/computer-sdk) | SDK for controlling desktop environments |
-| [cua-computer-server](https://cua.ai/docs/cua/reference/computer-sdk) | Driver for UI interactions and code execution in sandboxes |
-| [cua-bench](https://cua.ai/docs/cuabench) | Benchmarks and RL environments for computer-use |
-| [lume](https://cua.ai/docs/lume) | macOS/Linux VM management on Apple Silicon |
-| [lumier](https://cua.ai/docs/lume/guide/advanced/lumier) | Docker-compatible interface for Lume VMs |
+| Package | Description |
+| --------------------------------------------------------------------------- | ---------------------------------------------------------- |
+| [cuabot](https://docs.trycua.com/cuabot/guide/getting-started/introduction) | Multi-agent computer-use sandbox CLI |
+| [cua-agent](https://cua.ai/docs/cua/reference/agent-sdk) | AI agent framework for computer-use tasks |
+| [cua-computer](https://cua.ai/docs/cua/reference/computer-sdk) | SDK for controlling desktop environments |
+| [cua-computer-server](https://cua.ai/docs/cua/reference/computer-sdk) | Driver for UI interactions and code execution in sandboxes |
+| [cua-bench](https://cua.ai/docs/cuabench) | Benchmarks and RL environments for computer-use |
+| [lume](https://cua.ai/docs/lume) | macOS/Linux VM management on Apple Silicon |
+| [lumier](https://cua.ai/docs/lume/guide/advanced/lumier) | Docker-compatible interface for Lume VMs |
## Resources
diff --git a/docs/content/docs/cua/reference/agent-sdk/index.mdx b/docs/content/docs/cua/reference/agent-sdk/index.mdx
index 7ab38a8ff6..16ef2f1808 100644
--- a/docs/content/docs/cua/reference/agent-sdk/index.mdx
+++ b/docs/content/docs/cua/reference/agent-sdk/index.mdx
@@ -7,7 +7,7 @@ description: Python API reference for building computer-use agents
AUTO-GENERATED FILE - DO NOT EDIT DIRECTLY
Generated by: npx tsx scripts/docs-generators/python-sdk.ts
Source: libs/python/agent/agent
- Version: 0.7.29
+ Version: 0.7.30
*/}
import { Callout } from 'fumadocs-ui/components/callout';
@@ -17,7 +17,7 @@ import { VersionHeader } from '@/components/version-selector';
<VersionHeader
versions={[{"version":"0.7","href":"/cua/reference/agent-sdk","isCurrent":true},{"version":"0.6","href":"/cua/reference/agent-sdk/v0.6/api","isCurrent":false},{"version":"0.5","href":"/cua/reference/agent-sdk/v0.5/api","isCurrent":false},{"version":"0.4","href":"/cua/reference/agent-sdk/v0.4/api","isCurrent":false},{"version":"0.3","href":"/cua/reference/agent-sdk/v0.3/api","isCurrent":false}]}
currentVersion="0.7"
- fullVersion="0.7.29"
+ fullVersion="0.7.30"
packageName="cua-agent"
/>
diff --git a/docs/content/docs/cua/reference/cli/index.mdx b/docs/content/docs/cua/reference/cli/index.mdx
index e24add0bfe..165ac3ee38 100644
--- a/docs/content/docs/cua/reference/cli/index.mdx
+++ b/docs/content/docs/cua/reference/cli/index.mdx
@@ -7,7 +7,7 @@ description: Python API reference for the Cua command-line interface
AUTO-GENERATED FILE - DO NOT EDIT DIRECTLY
Generated by: npx tsx scripts/docs-generators/python-sdk.ts
Source: libs/python/cua-cli/cua_cli
- Version: 0.1.6
+ Version: 0.1.7
*/}
import { Callout } from 'fumadocs-ui/components/callout';
@@ -17,7 +17,7 @@ import { VersionHeader } from '@/components/version-selector';
<VersionHeader
versions={[{"version":"0.1","href":"/cua/reference/cli","isCurrent":true}]}
currentVersion="0.1"
- fullVersion="0.1.6"
+ fullVersion="0.1.7"
packageName="cua-cli"
/>
diff --git a/libs/python/agent/tests/test_telemetry_events.py b/libs/python/agent/tests/test_telemetry_events.py
new file mode 100644
index 0000000000..0ad8396b5b
--- /dev/null
+++ b/libs/python/agent/tests/test_telemetry_events.py
@@ -0,0 +1,145 @@
+"""
+Test script to verify telemetry events are emitted correctly.
+"""
+
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+
+
+class TestAgentTelemetryEvents:
+ """Test telemetry events emitted by ComputerAgent."""
+
+ @patch("agent.agent.record_event")
+ @patch("agent.agent.is_telemetry_enabled", return_value=True)
+ def test_agent_init_event(self, mock_telemetry_enabled, mock_record_event):
+ """Test that agent_init event is emitted with correct args_provided."""
+ from agent.agent import ComputerAgent
+
+ # Create agent with various args
+ agent = ComputerAgent(
+ model="anthropic/claude-sonnet-4-5-20250929",
+ instructions="Test instructions",
+ max_retries=5, # non-default
+ trajectory_dir="/tmp/test",
+ )
+
+ # Find the agent_init call
+ agent_init_calls = [
+ call for call in mock_record_event.call_args_list if call[0][0] == "agent_init"
+ ]
+
+ assert len(agent_init_calls) == 1, "agent_init should be called once"
+
+ event_name, event_data = agent_init_calls[0][0]
+ assert event_name == "agent_init"
+ assert event_data["model"] == "anthropic/claude-sonnet-4-5-20250929"
+ assert "instructions" in event_data["args_provided"]
+ assert "max_retries" in event_data["args_provided"]
+ assert "trajectory_dir" in event_data["args_provided"]
+
+ @patch("agent.agent.record_event")
+ @patch("agent.agent.is_telemetry_enabled", return_value=True)
+ def test_agent_init_minimal_args(self, mock_telemetry_enabled, mock_record_event):
+ """Test agent_init with minimal args (defaults)."""
+ from agent.agent import ComputerAgent
+
+ agent = ComputerAgent(model="anthropic/claude-sonnet-4-5-20250929")
+
+ agent_init_calls = [
+ call for call in mock_record_event.call_args_list if call[0][0] == "agent_init"
+ ]
+
+ assert len(agent_init_calls) == 1
+ event_name, event_data = agent_init_calls[0][0]
+
+ # With defaults, only model-related things should be tracked
+ # instructions, trajectory_dir, etc. should NOT be in args_provided
+ assert "instructions" not in event_data["args_provided"]
+ assert "trajectory_dir" not in event_data["args_provided"]
+ assert "max_retries" not in event_data["args_provided"] # default is 3
+
+ @patch("agent.agent.record_event")
+ @patch("agent.agent.is_telemetry_enabled", return_value=False)
+ def test_no_events_when_telemetry_disabled(self, mock_telemetry_enabled, mock_record_event):
+ """Test that no events are emitted when telemetry is disabled."""
+ from agent.agent import ComputerAgent
+
+ agent = ComputerAgent(
+ model="anthropic/claude-sonnet-4-5-20250929",
+ telemetry_enabled=False,
+ )
+
+ # No agent_init should be called (telemetry disabled)
+ agent_init_calls = [
+ call for call in mock_record_event.call_args_list if call[0][0] == "agent_init"
+ ]
+
+ assert len(agent_init_calls) == 0
+
+
+class TestActionTelemetryEvents:
+ """Test telemetry events for computer actions."""
+
+ @pytest.mark.asyncio
+ @patch("agent.agent.record_event")
+ @patch("agent.agent.is_telemetry_enabled", return_value=True)
+ async def test_computer_action_executed_event(self, mock_telemetry_enabled, mock_record_event):
+ """Test that computer_action_executed is emitted for computer calls."""
+ from agent.agent import ComputerAgent
+
+ agent = ComputerAgent(model="anthropic/claude-sonnet-4-5-20250929")
+ agent.telemetry_enabled = True
+
+ # Mock computer handler
+ mock_computer = MagicMock()
+ mock_computer.click = AsyncMock(return_value=None)
+ mock_computer.screenshot = AsyncMock(return_value="base64screenshot")
+
+ # Create a mock computer_call item
+ item = {
+ "type": "computer_call",
+ "call_id": "test-call-id",
+ "action": {
+ "type": "click",
+ "x": 100,
+ "y": 200,
+ },
+ }
+
+ # Process the item (this would normally happen in the agent loop)
+ # Note: We can't easily test this without running the full agent loop
+ # This is more of an integration test
+
+ # For unit testing, we verify the event structure
+ expected_event = {
+ "action_type": "click",
+ }
+
+ # Verify event structure is correct
+ assert "action_type" in expected_event
+
+
+class TestToolExecutedEvents:
+ """Test telemetry events for tool execution."""
+
+ def test_event_structure(self):
+ """Test that agent_tool_executed event has correct structure."""
+ expected_computer_tool_event = {
+ "tool_type": "computer",
+ "tool_name": "click",
+ }
+
+ expected_function_tool_event = {
+ "tool_type": "function",
+ "tool_name": "my_custom_function",
+ }
+
+ # Verify expected structure
+ assert "tool_type" in expected_computer_tool_event
+ assert "tool_name" in expected_computer_tool_event
+ assert expected_computer_tool_event["tool_type"] in ["computer", "function"]
+
+
+if __name__ == "__main__":
+ pytest.main([__file__, "-v"])
diff --git a/libs/python/computer/computer/computer.py b/libs/python/computer/computer/computer.py
index 1061fb52b5..599b8dbfd9 100644
--- a/libs/python/computer/computer/computer.py
+++ b/libs/python/computer/computer/computer.py
@@ -294,7 +294,17 @@ class Computer:
# Record initialization in telemetry (if enabled)
if telemetry_enabled and is_telemetry_enabled():
- record_event("computer_initialized", SYSTEM_INFO)
+ record_event(
+ "computer_initialized",
+ {
+ **SYSTEM_INFO,
+ "provider_type": (
+ self.provider_type.value
+ if isinstance(self.provider_type, VMProviderType)
+ else str(self.provider_type) if self.provider_type else "unknown"
+ ),
+ },
+ )
else:
self.logger.debug("Telemetry disabled - skipping initialization tracking")
diff --git a/libs/typescript/computer/src/computer/providers/base.ts b/libs/typescript/computer/src/computer/providers/base.ts
index a4e863522e..fb91082a66 100644
--- a/libs/typescript/computer/src/computer/providers/base.ts
+++ b/libs/typescript/computer/src/computer/providers/base.ts
@@ -27,6 +27,7 @@ export abstract class BaseComputer {
constructor(config: BaseComputerConfig) {
this.name = config.name;
this.osType = config.osType;
+ this.vmProvider = config.vmProvider;
this.telemetry = new Telemetry();
this.telemetry.recordEvent('module_init', {
module: 'computer',
@@ -38,6 +39,7 @@ export abstract class BaseComputer {
os: os.platform(),
os_version: os.version(),
node_version: process.version,
+ provider_type: config.vmProvider ?? 'unknown',
});
// Track which config args were provided
@@ -45,10 +47,12 @@ export abstract class BaseComputer {
const apiKey = 'apiKey' in config ? (config as { apiKey?: string }).apiKey : undefined;
if (apiKey) argsProvided.push('apiKey');
if ('apiBase' in config) argsProvided.push('apiBase');
+ if (config.vmProvider) argsProvided.push('vmProvider');
const initEventData: Record<string, unknown> = {
os_type: config.osType,
args_provided: argsProvided,
+ provider_type: config.vmProvider ?? 'unknown',
};
// Add hashed API key
if (apiKey) {
diff --git a/libs/typescript/computer/src/computer/types.ts b/libs/typescript/computer/src/computer/types.ts
index 339d52ac8c..42b48c78cd 100644
--- a/libs/typescript/computer/src/computer/types.ts
+++ b/libs/typescript/computer/src/computer/types.ts
@@ -22,6 +22,11 @@ export interface BaseComputerConfig {
* @default "macos"
*/
osType: OSType;
+
+ /**
+ * The VM provider type
+ */
+ vmProvider?: VMProviderType;
}
export interface CloudComputerConfig extends BaseComputerConfig {
diff --git a/scripts/docs-generators/python-sdk.ts b/scripts/docs-generators/python-sdk.ts
index a92e6dbee0..776bce5459 100644
--- a/scripts/docs-generators/python-sdk.ts
+++ b/scripts/docs-generators/python-sdk.ts
@@ -199,10 +199,7 @@ async function main() {
let docs: PythonPackage;
try {
// Prefer uv run --with griffe python (works cross-platform), fall back to python3
- const pythonCmd =
- process.platform === 'win32'
- ? `uv run --with griffe python`
- : `python3`;
+ const pythonCmd = process.platform === 'win32' ? `uv run --with griffe python` : `python3`;
const output = execSync(
`${pythonCmd} "${PYTHON_SCRIPT}" "${packagePath}" "${config.packageName}"`,
{
diff --git a/skills/gui-automation/SKILL.md b/skills/gui-automation/SKILL.md
index b6953e1d29..a90a152c1a 100644
--- a/skills/gui-automation/SKILL.md
+++ b/skills/gui-automation/SKILL.md
@@ -125,37 +125,37 @@ Tell the user: `"Here is the trajectory of my session: {url}"`
## Quick Reference
-| Action | Command |
-| ------------------- | ---------------------------------------------- |
-| Connect to target | `cua do switch <provider> [name]` |
-| Screenshot | `cua do screenshot` |
-| AI-annotated screen | `cua do snapshot ["instructions"]` |
-| Click | `cua do click <x> <y> [left\|right\|middle]` |
-| Double-click | `cua do dclick <x> <y>` |
-| Type text | `cua do type "text"` |
-| Press key | `cua do key <key>` |
-| Hotkey | `cua do hotkey <combo>` (e.g. `ctrl+c`) |
-| Scroll | `cua do scroll <direction> [amount]` |
-| Drag | `cua do drag <x1> <y1> <x2> <y2>` |
-| Move cursor | `cua do move <x> <y>` |
-| Shell command | `cua do shell "command"` |
-| Open URL/file | `cua do open <url\|path>` |
-| List windows | `cua do window ls [app]` |
-| Focus window | `cua do window focus <id>` |
-| Zoom to window | `cua do zoom "App Name"` |
-| Unzoom | `cua do unzoom` |
-| Share trajectory | `cua trajectory share` |
+| Action | Command |
+| ------------------- | -------------------------------------------- |
+| Connect to target | `cua do switch <provider> [name]` |
+| Screenshot | `cua do screenshot` |
+| AI-annotated screen | `cua do snapshot ["instructions"]` |
+| Click | `cua do click <x> <y> [left\|right\|middle]` |
+| Double-click | `cua do dclick <x> <y>` |
+| Type text | `cua do type "text"` |
+| Press key | `cua do key <key>` |
+| Hotkey | `cua do hotkey <combo>` (e.g. `ctrl+c`) |
+| Scroll | `cua do scroll <direction> [amount]` |
+| Drag | `cua do drag <x1> <y1> <x2> <y2>` |
+| Move cursor | `cua do move <x> <y>` |
+| Shell command | `cua do shell "command"` |
+| Open URL/file | `cua do open <url\|path>` |
+| List windows | `cua do window ls [app]` |
+| Focus window | `cua do window focus <id>` |
+| Zoom to window | `cua do zoom "App Name"` |
+| Unzoom | `cua do unzoom` |
+| Share trajectory | `cua trajectory share` |
## Providers
-| Provider | Example |
-| ------------ | ------------------------------------- |
-| `cloud` | `cua do switch cloud my-vm` |
-| `cloudv2` | `cua do switch cloudv2 my-vm` |
-| `docker` | `cua do switch docker my-container` |
-| `lume` | `cua do switch lume my-vm` |
-| `lumier` | `cua do switch lumier my-vm` |
-| `winsandbox` | `cua do switch winsandbox` |
-| `host` | `cua do switch host` |
+| Provider | Example |
+| ------------ | ----------------------------------- |
+| `cloud` | `cua do switch cloud my-vm` |
+| `cloudv2` | `cua do switch cloudv2 my-vm` |
+| `docker` | `cua do switch docker my-container` |
+| `lume` | `cua do switch lume my-vm` |
+| `lumier` | `cua do switch lumier my-vm` |
+| `winsandbox` | `cua do switch winsandbox` |
+| `host` | `cua do switch host` |
See [references/command-reference.md](references/command-reference.md) for full argument syntax.