Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

add native task example

Dillon DuPontde5c8b5b4dbb
diff --git a/docs/content/docs/cuabench/guide/examples/meta.json b/docs/content/docs/cuabench/guide/examples/meta.json
index b9743be76d..3c848c5342 100644
--- a/docs/content/docs/cuabench/guide/examples/meta.json
+++ b/docs/content/docs/cuabench/guide/examples/meta.json
@@ -4,6 +4,7 @@
"icon": "Blocks",
"pages": [
"universal-task",
+ "native-task",
"genai-task-generation",
"custom-agent"
]
diff --git a/docs/content/docs/cuabench/guide/examples/native-task.mdx b/docs/content/docs/cuabench/guide/examples/native-task.mdx
new file mode 100644
index 0000000000..421bf1e72e
--- /dev/null
+++ b/docs/content/docs/cuabench/guide/examples/native-task.mdx
@@ -0,0 +1,344 @@
+---
+title: Create a Native Task
+description: Build a file-based question-answering task using native OS commands and file manipulation
+---
+
+In this tutorial, you'll create a task that uses native OS capabilities to set up files and evaluate completion. This task uses **native provider** - it runs on real Docker/QEMU environments with full shell access.
+
+**Time:** ~10 minutes
+**Prerequisites:** cua-bench installed, Docker running
+
+## What You'll Build
+
+A question-answering task where agents must:
+1. Read a question from a text file on the Desktop
+2. Write their answer to another text file
+3. Get evaluated based on file contents
+
+You'll learn:
+- Using `run_command()` for OS-level operations
+- Setting up files and directories
+- Evaluating tasks by reading files
+- Working with native Linux/Windows environments
+
+## When to Use Native Tasks
+
+**Use native tasks when:**
+- You need to test real OS interactions (file system, registry, processes)
+- The task requires installing or using real applications
+- You need shell access for complex setup
+- You're building WinArena-style tasks
+
+**Use Universal GUI instead when:**
+- You're building custom interfaces or games
+- You want cross-platform compatibility
+- You don't need real OS features
+- You want faster, lightweight environments
+
+## Step 1: Create Task Directory
+
+```bash
+mkdir -p tasks/question_answer
+cd tasks/question_answer
+```
+
+## Step 2: Create the Task Configuration
+
+Create `main.py`:
+
+```python
+import cua_bench as cb
+from pathlib import Path
+
+
[email protected]_config(split="train")
+def load():
+ """Define task variants with different questions."""
+ qa_pairs = [
+ {
+ "question": "What is the capital of France?",
+ "answer": "Paris"
+ },
+ {
+ "question": "What is 2 + 2?",
+ "answer": "4"
+ },
+ {
+ "question": "What color is the sky on a clear day?",
+ "answer": "blue"
+ },
+ {
+ "question": "What is the largest planet in our solar system?",
+ "answer": "Jupiter"
+ }
+ ]
+
+ return [
+ cb.Task(
+ description=f'Open the question.txt file on the Desktop, read the question, and write your answer in answer.txt on the Desktop. Question: {qa["question"]}',
+ metadata={
+ "question": qa["question"],
+ "answer": qa["answer"]
+ },
+ computer={
+ "provider": "native", # Requires Docker/QEMU
+ "setup_config": {
+ "os_type": "linux", # or "windows"
+ "width": 1024,
+ "height": 768
+ }
+ }
+ )
+ for qa in qa_pairs
+ ]
+
+
[email protected]_task(split="train")
+async def start(task_cfg: cb.Task, session: cb.DesktopSession):
+ """Initialize the task environment with files."""
+ # Get the question from metadata
+ question = task_cfg.metadata["question"]
+
+ # Create Desktop directory if it doesn't exist
+ await session.run_command("mkdir -p ~/Desktop")
+
+ # Write the question to a text file on the Desktop
+ await session.run_command(f'echo "{question}" > ~/Desktop/question.txt')
+
+ # Ensure answer.txt doesn't exist yet
+ await session.run_command("rm -f ~/Desktop/answer.txt")
+
+
[email protected]_task(split="train")
+async def evaluate(task_cfg: cb.Task, session: cb.DesktopSession) -> list[float]:
+ """Return reward based on task completion."""
+ expected_answer = task_cfg.metadata["answer"].strip().lower()
+
+ # Check if answer.txt exists and read it
+ result = await session.run_command("cat ~/Desktop/answer.txt 2>/dev/null || echo ''")
+
+ # Extract the actual answer from the command result
+ user_answer = result["stdout"].strip().lower() if result else ""
+
+ # Check if the answer matches (case-insensitive, stripped)
+ if user_answer == expected_answer:
+ return [1.0]
+ else:
+ return [0.0]
+
+
[email protected]_task(split="train")
+async def solve(task_cfg: cb.Task, session: cb.DesktopSession):
+ """Demonstrate the solution."""
+ answer = task_cfg.metadata["answer"]
+
+ # Open the question file to read it (demonstration)
+ await session.run_command("cat ~/Desktop/question.txt")
+
+ # Write the answer to answer.txt
+ await session.run_command(f'echo "{answer}" > ~/Desktop/answer.txt')
+```
+
+## Step 3: Understanding run_command()
+
+The `run_command()` method executes shell commands and returns a dict:
+
+```python
+result = await session.run_command("cat ~/Desktop/answer.txt")
+
+# Result structure:
+{
+ "stdout": str, # Command output
+ "stderr": str, # Error output
+ "return_code": int, # Exit code (0 = success)
+ "success": bool # True if return_code == 0
+}
+
+# Access the output:
+output = result["stdout"]
+errors = result["stderr"]
+exit_code = result["return_code"]
+succeeded = result["success"]
+```
+
+### Common Patterns
+
+**Create directories:**
+```python
+await session.run_command("mkdir -p ~/Desktop/MyFolder")
+```
+
+**Write files:**
+```python
+await session.run_command('echo "content" > ~/Desktop/file.txt')
+```
+
+**Read files:**
+```python
+result = await session.run_command("cat ~/Desktop/file.txt")
+content = result["stdout"]
+```
+
+**Check if file exists:**
+```python
+result = await session.run_command("test -f ~/Desktop/file.txt && echo 'yes' || echo 'no'")
+exists = result["stdout"].strip() == "yes"
+```
+
+**Install applications (Linux):**
+```python
+await session.run_command("sudo apt update && sudo apt install -y firefox")
+```
+
+**Launch applications:**
+```python
+await session.run_command("firefox &") # Background process
+```
+
+## Step 4: Run the Task
+
+### Preview Interactively
+
+```bash
+cb interact tasks/question_answer --variant-id 0
+```
+
+A VNC viewer will open showing the native Linux desktop environment.
+
+### Run with Oracle
+
+```bash
+cb interact tasks/question_answer --oracle --variant-id 0
+```
+
+### Run with Agent
+
+```bash
+export ANTHROPIC_API_KEY=sk-...
+cb run task tasks/question_answer --agent cua-agent --model anthropic/claude-sonnet-4-20250514
+```
+
+## Advanced: Windows Tasks
+
+For Windows-specific tasks, use `os_type: "windows"`:
+
+```python
[email protected]_config(split="train")
+def load():
+ return [
+ cb.Task(
+ description='Open Notepad and write "Hello World" to a file',
+ computer={
+ "provider": "native",
+ "setup_config": {
+ "os_type": "windows", # Windows VM
+ "width": 1920,
+ "height": 1080
+ }
+ }
+ )
+ ]
+
[email protected]_task(split="train")
+async def start(task_cfg: cb.Task, session: cb.DesktopSession):
+ # Windows commands use PowerShell/CMD syntax
+ await session.run_command('powershell -Command "New-Item -Path C:\\Users\\Public\\Desktop -ItemType Directory -Force"')
+
+ # Launch Notepad
+ await session.run_command("notepad.exe")
+```
+
+### Windows Command Patterns
+
+**PowerShell commands:**
+```python
+result = await session.run_command('powershell -Command "Get-ChildItem C:\\Users"')
+```
+
+**Registry operations:**
+```python
+await session.run_command('reg add "HKCU\\Software\\MyApp" /v Setting /t REG_SZ /d "value"')
+```
+
+**File operations:**
+```python
+await session.run_command('powershell -Command "Get-Content C:\\path\\file.txt"')
+```
+
+## File API Alternative
+
+For simple file operations, use the dedicated file methods instead of `run_command()`:
+
+```python
+# Read file
+content = await session.read_file("/path/to/file.txt")
+
+# Write file
+await session.write_file("/path/to/file.txt", "content")
+
+# Check if file exists
+exists = await session.file_exists("/path/to/file.txt")
+
+# List directory
+files = await session.list_dir("/path/to/dir")
+```
+
+These methods are cleaner than shell commands for basic file operations.
+
+## Key Concepts
+
+**Native Provider**: Runs on real Docker (Linux) or QEMU (Windows) environments
+
+**Shell Commands**: Use `run_command()` for OS-level operations
+
+**Return Type**: Always returns a dict with `stdout`, `stderr`, `return_code`, `success`
+
+**File Setup**: Use setup to prepare the environment before agents interact
+
+**Evaluation**: Read files/check state to determine task completion
+
+## Performance Considerations
+
+Native tasks are **slower** than Universal GUI tasks because they:
+- Require Docker/QEMU startup time
+- Run full OS environments
+- Execute real shell commands
+
+**Optimization tips:**
+- Use Universal GUI when possible
+- Batch multiple commands with `&&`
+- Avoid unnecessary file operations
+- Use file API methods for simple reads/writes
+
+## Troubleshooting
+
+**Docker not running:**
+```
+Error: Cannot connect to Docker daemon
+```
+Solution: Start Docker Desktop
+
+**Permission denied:**
+```python
+# Add sudo for operations requiring permissions
+await session.run_command("sudo apt install -y package")
+```
+
+**Command not found:**
+```python
+# Install the package first
+await session.run_command("sudo apt update && sudo apt install -y package-name")
+```
+
+**Windows path escaping:**
+```python
+# Use raw strings or double backslashes
+await session.run_command('powershell -Command "Get-Item C:\\\\Users"')
+```
+
+## Next Steps
+
+- Learn about [Native Provider](/cuabench/guide/fundamentals/native-provider) for advanced OS features
+- See [Session API Reference](/cuabench/reference/session-api) for all available methods
+- Build a [Custom Agent](/cuabench/guide/examples/custom-agent) to solve tasks
+- Compare with [Universal Tasks](/cuabench/guide/examples/universal-task) to choose the right approach