Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
Move text from API Reference to Computer Playbook section
James Murdza1ff757c23bcb
diff --git a/docs/content/docs/computer-sdk/commands.mdx b/docs/content/docs/computer-sdk/commands.mdxindex 30657471b1..602de164ee 100644--- a/docs/content/docs/computer-sdk/commands.mdx+++ b/docs/content/docs/computer-sdk/commands.mdx@@ -243,3 +243,38 @@ Access system accessibility information:```</Tab></Tabs>++## Delay Configuration++Control timing between actions:++<Tabs items={['Python']}>+ <Tab value="Python">+ ```python+ # Set default delay between all actions (in seconds)+ computer.interface.delay = 0.5 # 500ms delay between actions++ # Or specify delay for individual actions+ await computer.interface.left_click(x, y, delay=1.0) # 1 second delay after click+ await computer.interface.type_text("Hello", delay=0.2) # 200ms delay after typing+ await computer.interface.press_key("enter", delay=0.5) # 500ms delay after key press+ ```++ </Tab>+</Tabs>++## Python Virtual Environment Operations++Manage Python environments:++<Tabs items={['Python']}>+ <Tab value="Python">+ ```python+ # Virtual environment management+ await computer.venv_install("demo_venv", ["requests", "macos-pyxa"]) # Install packages in a virtual environment+ await computer.venv_cmd("demo_venv", "python -c 'import requests; print(requests.get(`https://httpbin.org/ip`).json())'') # Run a shell command in a virtual environment+ await computer.venv_exec("demo_venv", python_function_or_code, *args, **kwargs) # Run a Python function in a virtual environment and return the result / raise an exception+ ```++ </Tab>+</Tabs>\ No newline at end of filediff --git a/docs/content/docs/computer-sdk/sandboxed-python.mdx b/docs/content/docs/computer-sdk/sandboxed-python.mdxindex 1e7f6b7854..5f1687bf83 100644--- a/docs/content/docs/computer-sdk/sandboxed-python.mdx+++ b/docs/content/docs/computer-sdk/sandboxed-python.mdx@@ -44,6 +44,32 @@ You can also install packages in the virtual environment using the `venv_installawait my_computer.venv_install("myenv", ["requests"])```+## Example: Interacting with macOS Applications++You can use sandboxed functions to interact with macOS applications on a local Cua Computer (requires `os_type="darwin"`). This is particularly useful for automation tasks that involve GUI applications.++```python+# Example: Use sandboxed functions to execute code in a Cua Container+from computer.helpers import sandboxed++await computer.venv_install("demo_venv", ["macos-pyxa"]) # Install packages in a virtual environment++@sandboxed("demo_venv")+def greet_and_print(name):+ """Get the HTML of the current Safari tab"""+ import PyXA+ safari = PyXA.Application("Safari")+ html = safari.current_document.source()+ print(f"Hello from inside the container, {name}!")+ return {"greeted": name, "safari_html": html}++# When a @sandboxed function is called, it will execute in the container+result = await greet_and_print("Cua")+# Result: {"greeted": "Cua", "safari_html": "<html>...</html>"}+# stdout and stderr are also captured and printed / raised+print("Result from sandboxed function:", result)+```+## Error HandlingIf the remote execution fails, the decorator will retry up to `max_retries` times. If all attempts fail, the last exception is raised locally.diff --git a/docs/content/docs/libraries/computer/index.mdx b/docs/content/docs/libraries/computer/index.mdxindex 4ac698d6c8..6638f8784d 100644--- a/docs/content/docs/libraries/computer/index.mdx+++ b/docs/content/docs/libraries/computer/index.mdx@@ -8,202 +8,16 @@ github:- https://github.com/trycua/cua/tree/main/libs/typescript/computer----The Computer library provides a Computer class that can be used to control and automate a container running the Computer Server.+The Computer library provides a Computer class for controlling and automating containers running the Computer Server.-## Reference+## Connecting to Computers-### Basic Usage+See the [Cua Computers](../computer-sdk/computers) documentation for how to connect to different computer types (cloud, local, or host desktop).-Connect to a cua cloud container:+## Computer Commands-<Tabs items={['Python', 'TypeScript']}>- <Tab value="Python">- ```python- from computer import Computer+See the [Commands](../computer-sdk/commands) documentation for all supported commands and interface methods (Shell, Mouse, Keyboard, File System, etc.).- computer = Computer(- os_type="linux",- provider_type="cloud",- name="your-container-name",- api_key="your-api-key"- )+## Sandboxed Python Functions- computer = await computer.run() # Connect to a cua cloud container- ```-- </Tab>- <Tab value="TypeScript">- ```typescript- import { Computer, OSType } from '@trycua/computer';-- const computer = new Computer({- osType: OSType.LINUX,- name: "your-container-name",- apiKey: "your-api-key"- });-- await computer.run(); // Connect to a cua cloud container- ```-- </Tab>-</Tabs>--Connect to a cua local container:--<Tabs items={['Python']}>- <Tab value="Python">- ```python- from computer import Computer-- computer = Computer(- os_type="macos"- )-- computer = await computer.run() # Connect to the container- ```-- </Tab>-</Tabs>--### Interface Actions--<Tabs items={['Python', 'TypeScript']}>- <Tab value="Python">- ```python- # Shell Actions- result = await computer.interface.run_command(cmd) # Run shell command- # result.stdout, result.stderr, result.returncode-- # Mouse Actions- await computer.interface.left_click(x, y) # Left click at coordinates- await computer.interface.right_click(x, y) # Right click at coordinates- await computer.interface.double_click(x, y) # Double click at coordinates- await computer.interface.move_cursor(x, y) # Move cursor to coordinates- await computer.interface.drag_to(x, y, duration) # Drag to coordinates- await computer.interface.get_cursor_position() # Get current cursor position- await computer.interface.mouse_down(x, y, button="left") # Press and hold a mouse button- await computer.interface.mouse_up(x, y, button="left") # Release a mouse button-- # Keyboard Actions- await computer.interface.type_text("Hello") # Type text- await computer.interface.press_key("enter") # Press a single key- await computer.interface.hotkey("command", "c") # Press key combination- await computer.interface.key_down("command") # Press and hold a key- await computer.interface.key_up("command") # Release a key-- # Scrolling Actions- await computer.interface.scroll(x, y) # Scroll the mouse wheel- await computer.interface.scroll_down(clicks) # Scroll down- await computer.interface.scroll_up(clicks) # Scroll up-- # Screen Actions- await computer.interface.screenshot() # Take a screenshot- await computer.interface.get_screen_size() # Get screen dimensions-- # Clipboard Actions- await computer.interface.set_clipboard(text) # Set clipboard content- await computer.interface.copy_to_clipboard() # Get clipboard content-- # File System Operations- await computer.interface.file_exists(path) # Check if file exists- await computer.interface.directory_exists(path) # Check if directory exists- await computer.interface.read_text(path, encoding="utf-8") # Read file content- await computer.interface.write_text(path, content, encoding="utf-8") # Write file content- await computer.interface.read_bytes(path) # Read file content as bytes- await computer.interface.write_bytes(path, content) # Write file content as bytes- await computer.interface.delete_file(path) # Delete file- await computer.interface.create_dir(path) # Create directory- await computer.interface.delete_dir(path) # Delete directory- await computer.interface.list_dir(path) # List directory contents-- # Accessibility- await computer.interface.get_accessibility_tree() # Get accessibility tree-- # Delay Configuration- # Set default delay between all actions (in seconds)- computer.interface.delay = 0.5 # 500ms delay between actions-- # Or specify delay for individual actions- await computer.interface.left_click(x, y, delay=1.0) # 1 second delay after click- await computer.interface.type_text("Hello", delay=0.2) # 200ms delay after typing- await computer.interface.press_key("enter", delay=0.5) # 500ms delay after key press-- # Python Virtual Environment Operations- await computer.venv_install("demo_venv", ["requests", "macos-pyxa"]) # Install packages in a virtual environment- await computer.venv_cmd("demo_venv", "python -c 'import requests; print(requests.get(`https://httpbin.org/ip`).json())'') # Run a shell command in a virtual environment- await computer.venv_exec("demo_venv", python_function_or_code, *args, **kwargs) # Run a Python function in a virtual environment and return the result / raise an exception-- # Example: Use sandboxed functions to execute code in a Cua Container- from computer.helpers import sandboxed-- @sandboxed("demo_venv")- def greet_and_print(name):- """Get the HTML of the current Safari tab"""- import PyXA- safari = PyXA.Application("Safari")- html = safari.current_document.source()- print(f"Hello from inside the container, {name}!")- return {"greeted": name, "safari_html": html}-- # When a @sandboxed function is called, it will execute in the container- result = await greet_and_print("Cua")- # Result: {"greeted": "Cua", "safari_html": "<html>...</html>"}- # stdout and stderr are also captured and printed / raised- print("Result from sandboxed function:", result)- ```-- </Tab>- <Tab value="TypeScript">- ```typescript- // Shell Actions- const result = await computer.interface.runCommand(cmd); // Run shell command- // result.stdout, result.stderr, result.returncode-- // Mouse Actions- await computer.interface.leftClick(x, y); // Left click at coordinates- await computer.interface.rightClick(x, y); // Right click at coordinates- await computer.interface.doubleClick(x, y); // Double click at coordinates- await computer.interface.moveCursor(x, y); // Move cursor to coordinates- await computer.interface.dragTo(x, y, duration); // Drag to coordinates- await computer.interface.getCursorPosition(); // Get current cursor position- await computer.interface.mouseDown(x, y, "left"); // Press and hold a mouse button- await computer.interface.mouseUp(x, y, "left"); // Release a mouse button-- // Keyboard Actions- await computer.interface.typeText("Hello"); // Type text- await computer.interface.pressKey("enter"); // Press a single key- await computer.interface.hotkey("command", "c"); // Press key combination- await computer.interface.keyDown("command"); // Press and hold a key- await computer.interface.keyUp("command"); // Release a key-- // Scrolling Actions- await computer.interface.scroll(x, y); // Scroll the mouse wheel- await computer.interface.scrollDown(clicks); // Scroll down- await computer.interface.scrollUp(clicks); // Scroll up-- // Screen Actions- await computer.interface.screenshot(); // Take a screenshot- await computer.interface.getScreenSize(); // Get screen dimensions-- // Clipboard Actions- await computer.interface.setClipboard(text); // Set clipboard content- await computer.interface.copyToClipboard(); // Get clipboard content-- // File System Operations- await computer.interface.fileExists(path); // Check if file exists- await computer.interface.directoryExists(path); // Check if directory exists- await computer.interface.readText(path, "utf-8"); // Read file content- await computer.interface.writeText(path, content, "utf-8"); // Write file content- await computer.interface.readBytes(path); // Read file content as bytes- await computer.interface.writeBytes(path, content); // Write file content as bytes- await computer.interface.deleteFile(path); // Delete file- await computer.interface.createDir(path); // Create directory- await computer.interface.deleteDir(path); // Delete directory- await computer.interface.listDir(path); // List directory contents-- // Accessibility- await computer.interface.getAccessibilityTree(); // Get accessibility tree- ```-- </Tab>-</Tabs>+See the [Sandboxed Python](../computer-sdk/sandboxed-python) documentation for running Python functions securely in isolated environments on a remote Cua Computer.\ No newline at end of file