Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
renamed HTML Desktops to Simulated Desktops
Dillon DuPont0e615556b03b
diff --git a/docs/content/docs/cuabench/guide/examples/genai-task-generation.mdx b/docs/content/docs/cuabench/guide/examples/genai-task-generation.mdxindex c1bc607312..a173aec9e8 100644--- a/docs/content/docs/cuabench/guide/examples/genai-task-generation.mdx+++ b/docs/content/docs/cuabench/guide/examples/genai-task-generation.mdx@@ -382,5 +382,5 @@ cb interact tasks/python-quiz-env --oracle --no-wait## Next Steps- Learn about [Task Structure](/cuabench/guide/fundamentals/tasks) in depth-- Explore [HTML Desktops](/cuabench/guide/fundamentals/html-desktops) for visual variants+- Explore [Simulated Desktop](/cuabench/guide/fundamentals/simulated-desktop) for visual variants- See [Creating Private Tasks](/cuabench/guide/advanced/creating-private-tasks) for advanced patternsdiff --git a/docs/content/docs/cuabench/guide/examples/universal-task.mdx b/docs/content/docs/cuabench/guide/examples/universal-task.mdxindex 8102e0d859..3839bb1dcd 100644--- a/docs/content/docs/cuabench/guide/examples/universal-task.mdx+++ b/docs/content/docs/cuabench/guide/examples/universal-task.mdx@@ -1,9 +1,9 @@---title: Create a Universal Task-description: Build a cross-platform Minesweeper game that works on QEMU, HTML desktops, and Docker+description: Build a cross-platform Minesweeper game that works on QEMU, simulated desktop, and Docker----In this tutorial, you'll create a Minesweeper game task using the Universal GUI API. This task is **universal** - it runs across QEMU, HTML desktops, and Docker environments without modification.+In this tutorial, you'll create a Minesweeper game task using the Universal GUI API. This task is **universal** - it runs across QEMU, simulated desktop, and Docker environments without modification.**Time:** ~15 minutes**Prerequisites:** cua-bench installed@@ -52,7 +52,7 @@ def load():"mines": mines,},computer={- "provider": "simulated", # Use HTML desktop for preview+ "provider": "simulated", # Use simulated desktop for preview"setup_config": {"os_type": "linux","width": 800,@@ -358,7 +358,7 @@ Create `gui/index.html` with the Minesweeper game:cb interact tasks/minesweeper --task-id 0```-A browser window will open showing the Minesweeper game in an HTML desktop environment.+A browser window will open showing the Minesweeper game in a simulated desktop environment.### Run with Oracle@@ -386,7 +386,7 @@ cb run task tasks/minesweeper --agent cua-agent --model anthropic/claude-sonnet-## Why Universal?This task works across all platforms:-- **HTML Desktops**: Set `provider: "simulated"` for lightweight browser preview+- **Simulated Desktop**: Set `provider: "simulated"` for lightweight browser preview- **Docker**: Use real Linux desktop environments- **QEMU**: Run on actual Windows/Linux VMs@@ -394,6 +394,6 @@ The same HTML GUI and task code runs everywhere.## Next Steps-- Learn about [HTML Desktops](/cuabench/guide/fundamentals/html-desktops) for themed previews+- Learn about [Simulated Desktop](/cuabench/guide/fundamentals/simulated-desktop) for themed previews- See [Universal GUI](/cuabench/guide/fundamentals/universal-gui) for the full API- Build a [Custom Agent](/cuabench/guide/examples/custom-agent) to solve tasksdiff --git a/docs/content/docs/cuabench/guide/fundamentals/meta.json b/docs/content/docs/cuabench/guide/fundamentals/meta.jsonindex ee7b09d754..c4abb69162 100644--- a/docs/content/docs/cuabench/guide/fundamentals/meta.json+++ b/docs/content/docs/cuabench/guide/fundamentals/meta.json@@ -2,5 +2,5 @@"title": "Fundamentals","description": "Core concepts of tasks and environments","icon": "Lightbulb",- "pages": ["tasks", "universal-gui", "html-desktops", "adapters", "registry"]+ "pages": ["tasks", "universal-gui", "simulated-desktop", "adapters", "registry"]}diff --git a/docs/content/docs/cuabench/guide/fundamentals/html-desktops.mdx b/docs/content/docs/cuabench/guide/fundamentals/simulated-desktop.mdxsimilarity index 65%rename from docs/content/docs/cuabench/guide/fundamentals/html-desktops.mdxrename to docs/content/docs/cuabench/guide/fundamentals/simulated-desktop.mdxindex 75c2239c4c..017af20e0d 100644--- a/docs/content/docs/cuabench/guide/fundamentals/html-desktops.mdx+++ b/docs/content/docs/cuabench/guide/fundamentals/simulated-desktop.mdx@@ -1,9 +1,11 @@----title: HTML Desktops-description: Preview tasks with themed desktop environments (no Docker required)+title: Simulated Desktop (Preview)+description: Preview feature for testing tasks with themed desktop environments (no Docker required)----# HTML Desktops+<Callout type="info">+This is a **preview feature** for rapid development and testing.+</Callout>When you set `provider="simulated"` in your task config, you can preview it in a lightweight HTML/CSS desktop environment without Docker. This renders OS-themed window chrome, taskbars, and desktop styling around your GUI.@@ -28,7 +30,7 @@ def load():cb.Task(description='Click the Submit button',computer={- "provider": "simulated", # Enables HTML desktop+ "provider": "simulated", # Enables simulated desktop"setup_config": {"os_type": "win11", # linux, win11, or macos"width": 1920,@@ -70,12 +72,22 @@ def load():]```-## Why Use HTML Desktops?+## Limitations++Simulated desktop runs in a browser, so it has some restrictions:++- **No shell actions**: Can't execute bash commands or scripts+- **No native apps**: Can't install or run real applications+- **HTML/CSS only**: Only works with custom GUIs created via `launch_window()` or `install_app()`++For tasks requiring shell access or native applications, use Docker or QEMU environments instead.++## Why Use Simulated Desktop?- **No Docker**: Runs directly in browser with Playwright- **Fast**: Instant startup, no VM boot time- **Visual Diversity**: Train agents on different OS appearances- **Deterministic**: Same theme always renders identically-- **Lightweight**: Run hundreds in parallel+- **Lightweight**: Run thousands in parallel-Use HTML desktops for quick iteration and agent development. Switch to Docker/QEMU when you need real OS environments.+Use simulated desktop for quick iteration and agent development. Switch to Docker/QEMU when you need real OS environments.diff --git a/docs/content/docs/cuabench/guide/fundamentals/tasks.mdx b/docs/content/docs/cuabench/guide/fundamentals/tasks.mdxindex 78e5611f5c..f0f64e4112 100644--- a/docs/content/docs/cuabench/guide/fundamentals/tasks.mdx+++ b/docs/content/docs/cuabench/guide/fundamentals/tasks.mdx@@ -34,14 +34,14 @@ def load():}})- for provider, os_type in [("computer", "linux"), ("computer", "windows"), ("webtop", "linux"), ("webtop", "win11"), ("webtop", "macos")]+ for provider, os_type in [("native", "linux"), ("native", "windows"), ("simulated", "linux"), ("simulated", "win11"), ("simulated", "macos")]# More task variants...]```Each task variant can have different metadata, enabling thousands of unique scenarios from a single template.-The `computer` provider **emulates** `windows` and `linux` operating systems, while the `simulated` provider renders HTML desktop themes including `linux`, `win11`, and `macos`. See [HTML Desktops](/cuabench/guide/fundamentals/html-desktops) for all available themes.+The `native` provider **emulates** `windows` and `linux` operating systems using QEMU/Docker, while the `simulated` provider lightweight Playwright-based desktop environments with themes such as `linux`, `win11`, and `macos`. See [Simulated Desktop](/cuabench/guide/fundamentals/simulated-desktop) for all available themes.### 2. Setup (`@cb.setup_task`)diff --git a/docs/content/docs/cuabench/guide/fundamentals/universal-gui.mdx b/docs/content/docs/cuabench/guide/fundamentals/universal-gui.mdxindex fea64d1264..86a8e9a91c 100644--- a/docs/content/docs/cuabench/guide/fundamentals/universal-gui.mdx+++ b/docs/content/docs/cuabench/guide/fundamentals/universal-gui.mdx@@ -3,8 +3,6 @@ title: Universal GUIdescription: Package custom GUIs with your tasks using the window management API----# Universal GUI-Cua-Bench provides an Electron-like API for packaging custom HTML/CSS/JS GUIs with your tasks. This allows you to create interactive GUI apps, GUI-based tools, or simulated clones of real applications when developing tasks.## Window Management API@@ -45,13 +43,13 @@ my_task/└── index.html # Your HTML/CSS/JS GUI```-## Previewing with HTML Desktops+## Previewing with Simulated DesktopFor lightweight previewing without Docker, set `provider: "simulated"` in your task config:```pythoncomputer={- "provider": "simulated", # Enables HTML desktop preview+ "provider": "simulated", # Enables simulated desktop preview"setup_config": {"os_type": "linux", # linux, win11, or macos theme"width": 800,@@ -66,13 +64,13 @@ Then preview with:cb interact tasks/my_game --task-id 0```-The [HTML Desktop](/cuabench/guide/fundamentals/html-desktops) renders your GUI in a themed desktop environment (Linux/Windows/macOS) without requiring Docker.+The [Simulated Desktop](/cuabench/guide/fundamentals/simulated-desktop) renders your GUI in a themed desktop environment (Linux/Windows/macOS) without requiring Docker.## Key Features- **JavaScript execution**: Communicate with your GUI via `execute_javascript()`- **CSS selectors**: Click elements with `click_element(pid, "#button")`- **State management**: Expose game/app state via `window.__variables`-- **Cross-platform**: Works in Docker containers and HTML desktops+- **Cross-platform**: Works in Docker containers and simulated desktopSee the [Create a Universal Task](/cuabench/guide/examples/universal-task) example for a complete walkthrough.