Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver-bench): Windows guard treats a driver's own helper shell like macOS (disclosed rule) (#6655)
Francesco Bonaccid864a3bf515e
diff --git a/libs/cua-driver-bench/METHOD.md b/libs/cua-driver-bench/METHOD.mdindex cf0b564cb3..d61ad895ec 100644--- a/libs/cua-driver-bench/METHOD.md+++ b/libs/cua-driver-bench/METHOD.md@@ -152,6 +152,29 @@ setup. The evidence is kept in the row. A side-door app in front for one or twosamples, or a tool input that only names such an app, is recorded but is not aviolation.+**Windows.** The Windows guard has no list of side-door apps to look for, so it+counts any shell or script host (PowerShell, cmd, Windows Terminal, the script+hosts, mshta, WSL) that starts in the desktop session outside the runner's own+processes, plus the same tool-input scan. One rule keeps it equal to macOS for+every setup: a shell whose parent process is the setup's own binary (the+`--tool-exe` of `launch/windows_session.py`: `cua-driver.exe`,+`agent-desktop-win32-x64.exe`, `windows-harness.exe`, `codex-computer-use.exe`)+is part of the tool. It is logged in the row (`gui_only.tool_helpers`) and is+not a violation. On macOS the process rule names only side-door apps ("a process+named Script Editor, Automator, iTerm2 or Shortcuts exists"), and the input rule+flags only "a computer-use tool input [that] contains code that runs outside the+GUI or touches files", so a driver's own helper processes were never violations+there either. A shell started from code the agent supplied (Python passed to a+harness's `exec`, JavaScript passed to a REPL) has an interpreter as its parent+(`python.exe`, `node.exe`), which is never a `--tool-exe`, and still counts. The+rule was added on 6 November 2026, during the Windows multi-app run. Cua Driver+starts PowerShell to read browser processes in `get_browser_state` and+`browser_prepare`; before the rule, 7 of its 10 CDB-G04 trials were stopped for+that. Those trials were rerun with the same seeds. Applied to every earlier+Windows row, the rule changes no other verdict: every other process violation+on Windows followed a tool input in which the agent itself asked for PowerShell+or a subprocess.+## Metrics**Success:** every required check of the task's checker passed, and there was no@@ -259,8 +282,9 @@ differences are directional only.checkers and sentinel measures as macOS, but a Windows or Linux result is itsown condition: a different desktop, accessibility stack and window manager, socompare setups within a platform, not across platforms. The Windows release- run covers the seven BenchLab tasks (MB-09, MB-10, MB-11, IR-01 to IR-04); it- does not include CDB-G02 to CDB-G04.+ run covers all ten tasks: the seven BenchLab tasks and CDB-G02 to CDB-G04+ (LibreOffice 26.2.6.3, GnuCash 5.17, Google Chrome 157, the pack's Electron+ 43.8.1 apps), with the same seeds as macOS.* **Vendor setups are driven through Cua's adapters.** agent-desktop, the Browser Useharnesses and OpenAI's computer-use plugin are not built for Claude Code as theirclient, so each is run through a small adapter in `automated-eval/macos_bench/tools/`@@ -278,4 +302,5 @@ differences are directional only.the Browser Use macOS harness 0.2.1 (Codex computer use as installed for therun; Claude Code 2.1.318, Claude Sonnet 5.5, 3 trials with the same seeds inthe foreground run). Windows: the Cua Driver 0.38.0 Windows build,- agent-desktop 1.2.0 and the Browser Use Windows harness 0.2.2.+ agent-desktop 1.2.0, the Browser Use Windows harness 0.2.2 and Codex computer+ use from the ChatGPT app 26.1029.9216.0 (Claude Code 2.1.316, Claude Sonnet 5.5).diff --git a/libs/cua-driver-bench/README.md b/libs/cua-driver-bench/README.mdindex 0ab2575e9f..3bdf8985cf 100644--- a/libs/cua-driver-bench/README.md+++ b/libs/cua-driver-bench/README.md@@ -457,7 +457,10 @@ Where Windows differs:* **GUI-only guard.** A shell or script host (PowerShell, cmd, Windows Terminal, the script hosts,mshta, WSL) started in the desktop session outside the runner's own processes, or a side-door windowin front, fails and stops the trial. Tool inputs are also scanned for PowerShell, cmd and script-host- commands.+ commands. A shell whose parent is the setup's own binary (`--tool-exe cua-driver.exe`, for example: Cua+ Driver starts PowerShell to read browser processes) is part of the tool, as a driver's helper process is+ on macOS: it is logged in `gui_only.tool_helpers` and does not count. A shell started from code the agent+ supplied (a harness's Python `exec`, a JavaScript REPL) has an interpreter as its parent and still counts.* **Sentinel.** Windows has no signals, so the runner arms it through a control file. A window counts asraised when it is above the user window in the z-order and overlaps it. The desktop measure is`IVirtualDesktopManager`; a dialog on the user's app is a window owned by the user window.@@ -505,6 +508,10 @@ Where the multi-app tasks differ on Windows:it off for the bench user before the trials: `org.openoffice.Office.Update/Update/Enabled` and`org.openoffice.Office.Jobs/Jobs/UpdateCheck/Arguments/AutoCheckEnabled` set to `false` (as`Software\Policies\LibreOffice` registry policies, which an administrator writes).+* **Codex approvals.** Codex computer use names a Windows app by its process when it asks for approval:+ `soffice.bin` for LibreOffice, `GnuCash`, `Google Chrome`, and the Electron apps by their named executables+ (`copperfield mail`, ...). Put those names on the adapter's `--allow` list for CDB-G02 to CDB-G04; a name+ missing from it is declined and the setup cannot use that app.* **Leftovers.** Before and after a trial the launcher stops the task's own executables in the desktopsession by exact name and whatever listens on the task's loopback port (4184 for CDB-G04).* **Validation.** `windows\cdb_oracle.py` stands in for the agent: by default it applies the scripteddiff --git a/libs/cua-driver-bench/automated-eval/macos_bench/launch/windows_session.py b/libs/cua-driver-bench/automated-eval/macos_bench/launch/windows_session.pyindex 4cfca3103c..53e5c767c8 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/launch/windows_session.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/launch/windows_session.py@@ -194,6 +194,7 @@ def run(args: argparse.Namespace, command: list[str]) -> dict:)procs.append(agent)own_tree: set[int] = set()+ helpers: dict[int, dict] = {}session = W.session_of(os.getpid())front = G.FrontWatch()violations: list[dict] = []@@ -204,6 +205,14 @@ def run(args: argparse.Namespace, command: list[str]) -> dict:# Only the runner's own (interactive) session: remote administration (SSH in session 0) is not the GUI.here = [r for r in rows if W.session_of(r[0]) == session]for pid, label in G.new_side_doors(here, baseline, os.getpid(), own_tree):+ by = G.started_by_tool(rows, pid, args.tool_exe)+ if (+ by+ ): # the setup's own binary started it: part of the tool, logged, not a side door+ helpers.setdefault(+ pid, {"t_s": round(time.monotonic() - t0, 1), "detail": label, "parent": by}+ )+ continueviolations.append({"t_s": round(time.monotonic() - t0, 1),@@ -269,7 +278,8 @@ def run(args: argparse.Namespace, command: list[str]) -> dict:evaluation={k: evaluation.get(k) for k in ("passed", "score", "checks", "diagnostics")},checker_passed=bool(evaluation.get("passed")),gui_only={"violation": bool(violations), "violations": violations[:20],- "mentions": soft, "frontmost_seen": sorted(front.seen)},+ "mentions": soft, "frontmost_seen": sorted(front.seen),+ "tool_helpers": list(helpers.values())[:20]},passed=passed,disturbance=disturbance,background=bg,@@ -307,6 +317,10 @@ def main(argv: list[str]) -> int:parser.add_argument("--label", default="")parser.add_argument("--timeout", type=float, default=360.0)parser.add_argument("--no-enforce", dest="enforce", action="store_false")+ parser.add_argument(+ "--tool-exe", action="append", default=[],+ help="executable of the setup's own tool (repeatable): a shell it starts itself is not a side door",+ ) # fmt: skipargs = parser.parse_args(argv[:split])command = argv[split + 1 :]if sys.platform != "win32":diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_windows_tools.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_windows_tools.pyindex 03a7c99325..0ae57ae33d 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_windows_tools.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_windows_tools.py@@ -117,6 +117,19 @@ class GuardTest(unittest.TestCase):def test_descendants(self) -> None:self.assertEqual(G.descendants(self.ROWS, 200), {200, 210, 220})+ def test_shell_started_by_the_tool_binary_is_a_helper(self) -> None:+ rows = [+ (10, 1, "cua-driver.exe"),+ (11, 10, "powershell.exe"),+ (20, 1, "python.exe"),+ (21, 20, "powershell.exe"),+ ]+ self.assertEqual(G.started_by_tool(rows, 11, ["cua-driver.exe"]), "cua-driver")+ self.assertIsNone(+ G.started_by_tool(rows, 21, ["cua-driver.exe"])+ ) # agent code in an interpreter+ self.assertIsNone(G.started_by_tool(rows, 11, []))+def test_new_side_doors(self) -> None:hits = G.new_side_doors(self.ROWS, baseline={4, 100}, runner=200, own={250})self.assertEqual(sorted(hits), [(230, "Windows PowerShell"), (240, "Windows Terminal")])diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/windows/guard.py b/libs/cua-driver-bench/automated-eval/macos_bench/windows/guard.pyindex 2a6a1b9fbb..aa2ac4f56d 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/windows/guard.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/windows/guard.py@@ -92,6 +92,21 @@ def new_side_doors(]+def started_by_tool(+ rows: Iterable[tuple[int, int, str]], pid: int, tool_exes: Iterable[str]+) -> str | None:+ """The executable name of `pid`'s parent when that parent is one of the setup's own tool binaries+ (`--tool-exe`), else None. A shell a driver binary starts for itself (to query a process, say) is part of+ the tool, as a helper process is on macOS; a shell started by code the agent supplied (a harness's Python+ `exec`, a JavaScript REPL) has an interpreter as its parent and stays a side door."""+ names = {name_of(t) for t in tool_exes}+ if not names:+ return None+ parent = {p: pp for p, pp, _ in rows}.get(pid)+ exe = {p: e for p, _, e in rows}.get(parent, "") if parent is not None else ""+ return name_of(exe) if name_of(exe) in names else None++def scan_tool_use(name: str, tool_input: Any) -> tuple[list[str], list[str]]:hard, soft = v2.scan_tool_use(name, tool_input)if name.startswith("mcp__"):