Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
docs(cua-driver-bench): Bench v2 release method notes and rerun steps (#6622)
Francesco Bonacci3eb7f3f8633c
diff --git a/libs/cua-driver-bench/METHOD.md b/libs/cua-driver-bench/METHOD.mdindex 1567a75378..427bf2fa3d 100644--- a/libs/cua-driver-bench/METHOD.md+++ b/libs/cua-driver-bench/METHOD.md@@ -121,6 +121,17 @@ and seed. They are reported side by side and never pooled.app or window to the front, and do not type or click into the User activitywindow. Do this task in the background."+### Foreground-only tasks++MB-10 (hover-only toolbar) and MB-11 (native tooltip) show their readout only+while the real pointer hovers, so no setup can do them without moving the user's+pointer. `taskpack/v2-taskset.json` lists them as `foreground` only: the runner+skips them in the background condition, and they are not part of the background+headline. They are scored in the foreground condition like every other task. If+a run does put them in the background (for example to compare with another+runner), report that apart from the headline, as a note on the driver gap, never+inside it.+## GUI-only guardNo setup gets a shell, file or web tool. During every trial a guard samples the@@ -169,6 +180,17 @@ frontmost status the whole time:A window owned by the computer-use tool itself (an agent cursor, a statusbanner) is logged but not counted as a raise.+**Full pass.** The term used in the release tables. In the foreground+condition a full pass is a trial whose checker passed and that had no GUI-only+violation. In the background condition it is the Amendment 22 background+completion: checker passed, no GUI-only violation, and all eight user-undisturbed+criteria held (key focus, window raised, typing, pointer, clipboard, Space,+dialog, activation; one that was not measured counts as disturbed). The+background headline covers only the tasks that run in the background (the ten+core tasks minus MB-10 and MB-11, so eight). Alongside it the tables split every+background trial into task failed, passed but the user was disturbed, and full+pass, and list the eight criteria per setup.+**Secondary:** success regardless of focus; the number of focus steals; keyloss; windows raised; simulated user keystrokes swallowed (ticks when the userwindow was not active and key); pointer movement over 10 px (recorded, not part@@ -232,5 +254,17 @@ differences are directional only.BenchLab, not by macOS or Chrome.* **Small samples.** 5 runs per cell; a 5/5 has a Wilson 95% lower bound of0.57.-* **One model, one OS build.** Not a claim about other models, apps, Windows or- Linux.+* **One model, one OS build per platform.** Not a claim about other models or+ apps. The Windows and Linux ports run the BenchLab tasks with the same seeds,+ checkers and sentinel measures as macOS, but a Windows or Linux result is its+ own condition: a different desktop, accessibility stack and window manager, so+ compare setups within a platform, not across platforms. The Windows release+ run covers the seven BenchLab tasks (MB-09, MB-10, MB-11, IR-01 to IR-04); it+ does not include CDB-G02 to CDB-G04.+* **Driver versions are named per platform.** The release tables name the+ version of each tool layer, and a table never mixes versions silently. macOS,+ foreground and background alike: Cua Driver 0.38.0, agent-desktop 1.2.0 and+ the Browser Use macOS harness 0.2.1 (Codex computer use as installed for the+ run; Claude Code 2.1.318, Claude Sonnet 5.5, 3 trials with the same seeds in+ the foreground run). Windows: the Cua Driver 0.38.0 Windows build,+ agent-desktop 1.2.0 and the Browser Use Windows harness 0.2.2.diff --git a/libs/cua-driver-bench/README.md b/libs/cua-driver-bench/README.mdindex b359c6103c..c1798d95d5 100644--- a/libs/cua-driver-bench/README.md+++ b/libs/cua-driver-bench/README.md@@ -87,6 +87,24 @@ only" skip the background condition: their readout appears only on a hover or atooltip, which follows the real pointer, so no setup can do them in thebackground.+### Reading and rerunning the release results++* **What is scored.** Success and the background headline follow+ [`METHOD.md`](METHOD.md): *full pass* is the checker passing with no GUI-only+ violation and, in the background condition, all eight user-undisturbed+ criteria held. The background headline covers the eight tasks that run in the+ background; MB-10 and MB-11 (foreground only) are scored in the foreground+ condition only.+* **Versions.** Each table names the driver versions it used: the same on+ both macOS conditions, different on Windows (see "Driver versions are named+ per platform" in `METHOD.md`). Rerun with the version you want+ to measure and record it.+* **Rerun on your platform.** macOS: steps 1 to 5 under "Reproduce", one+ condition per run. Linux: "Linux (the BenchLab tasks)". Windows: "Windows (the+ BenchLab tasks)". Then `tools/analyze_v2.py` on the run directories. The+ manifest decides which tasks run in which condition, so a rerun needs no task+ list.+## Fairness and health checksA trial is scored only when the harness gave the agent a working environment.@@ -314,6 +332,20 @@ completion with Wilson intervals, focus steals, key loss, windows raised,swallowed keystrokes, GUI-only violations with evidence, and the interruptionrates. Keep raw results, streams and recordings out of git.+### Linux (the BenchLab tasks)++The Linux port of BenchLab and the sentinel is in+[`automated-eval/macos_bench/linux`](automated-eval/macos_bench/linux) (GTK with AT-SPI). Run a trial inside+the single-owner headless X11 session that [`launch/linux_session.sh`](automated-eval/macos_bench/launch/linux_session.sh)+starts (Xvfb 1920 x 1080, a private D-Bus, openbox):++```bash+launch/linux_session.sh --evidence "$CDB_BENCH_WORK/evidence" -- <runner command>+```++Seeds, the state file, the event log, the checkers and the sentinel summary are the same as on macOS.+`tests/test_linux_session.py` and `tests/test_linux_tools.py` check the session and the tools without a model.+### Windows (the BenchLab tasks)The Windows port of BenchLab and the sentinel is in