Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver-bench): Windows guard treats a driver's own helper shell like macOS (disclosed rule) (#6655)

Francesco Bonaccid864a3bf515e
diff --git a/libs/cua-driver-bench/METHOD.md b/libs/cua-driver-bench/METHOD.md
index cf0b564cb3..d61ad895ec 100644
--- a/libs/cua-driver-bench/METHOD.md
+++ b/libs/cua-driver-bench/METHOD.md
@@ -152,6 +152,29 @@ setup. The evidence is kept in the row. A side-door app in front for one or two
samples, or a tool input that only names such an app, is recorded but is not a
violation.
+**Windows.** The Windows guard has no list of side-door apps to look for, so it
+counts any shell or script host (PowerShell, cmd, Windows Terminal, the script
+hosts, mshta, WSL) that starts in the desktop session outside the runner's own
+processes, plus the same tool-input scan. One rule keeps it equal to macOS for
+every setup: a shell whose parent process is the setup's own binary (the
+`--tool-exe` of `launch/windows_session.py`: `cua-driver.exe`,
+`agent-desktop-win32-x64.exe`, `windows-harness.exe`, `codex-computer-use.exe`)
+is part of the tool. It is logged in the row (`gui_only.tool_helpers`) and is
+not a violation. On macOS the process rule names only side-door apps ("a process
+named Script Editor, Automator, iTerm2 or Shortcuts exists"), and the input rule
+flags only "a computer-use tool input [that] contains code that runs outside the
+GUI or touches files", so a driver's own helper processes were never violations
+there either. A shell started from code the agent supplied (Python passed to a
+harness's `exec`, JavaScript passed to a REPL) has an interpreter as its parent
+(`python.exe`, `node.exe`), which is never a `--tool-exe`, and still counts. The
+rule was added on 6 November 2026, during the Windows multi-app run. Cua Driver
+starts PowerShell to read browser processes in `get_browser_state` and
+`browser_prepare`; before the rule, 7 of its 10 CDB-G04 trials were stopped for
+that. Those trials were rerun with the same seeds. Applied to every earlier
+Windows row, the rule changes no other verdict: every other process violation
+on Windows followed a tool input in which the agent itself asked for PowerShell
+or a subprocess.
+
## Metrics
**Success:** every required check of the task's checker passed, and there was no
@@ -259,8 +282,9 @@ differences are directional only.
checkers and sentinel measures as macOS, but a Windows or Linux result is its
own condition: a different desktop, accessibility stack and window manager, so
compare setups within a platform, not across platforms. The Windows release
- run covers the seven BenchLab tasks (MB-09, MB-10, MB-11, IR-01 to IR-04); it
- does not include CDB-G02 to CDB-G04.
+ run covers all ten tasks: the seven BenchLab tasks and CDB-G02 to CDB-G04
+ (LibreOffice 26.2.6.3, GnuCash 5.17, Google Chrome 157, the pack's Electron
+ 43.8.1 apps), with the same seeds as macOS.
* **Vendor setups are driven through Cua's adapters.** agent-desktop, the Browser Use
harnesses and OpenAI's computer-use plugin are not built for Claude Code as their
client, so each is run through a small adapter in `automated-eval/macos_bench/tools/`
@@ -278,4 +302,5 @@ differences are directional only.
the Browser Use macOS harness 0.2.1 (Codex computer use as installed for the
run; Claude Code 2.1.318, Claude Sonnet 5.5, 3 trials with the same seeds in
the foreground run). Windows: the Cua Driver 0.38.0 Windows build,
- agent-desktop 1.2.0 and the Browser Use Windows harness 0.2.2.
+ agent-desktop 1.2.0, the Browser Use Windows harness 0.2.2 and Codex computer
+ use from the ChatGPT app 26.1029.9216.0 (Claude Code 2.1.316, Claude Sonnet 5.5).
diff --git a/libs/cua-driver-bench/README.md b/libs/cua-driver-bench/README.md
index 0ab2575e9f..3bdf8985cf 100644
--- a/libs/cua-driver-bench/README.md
+++ b/libs/cua-driver-bench/README.md
@@ -457,7 +457,10 @@ Where Windows differs:
* **GUI-only guard.** A shell or script host (PowerShell, cmd, Windows Terminal, the script hosts,
mshta, WSL) started in the desktop session outside the runner's own processes, or a side-door window
in front, fails and stops the trial. Tool inputs are also scanned for PowerShell, cmd and script-host
- commands.
+ commands. A shell whose parent is the setup's own binary (`--tool-exe cua-driver.exe`, for example: Cua
+ Driver starts PowerShell to read browser processes) is part of the tool, as a driver's helper process is
+ on macOS: it is logged in `gui_only.tool_helpers` and does not count. A shell started from code the agent
+ supplied (a harness's Python `exec`, a JavaScript REPL) has an interpreter as its parent and still counts.
* **Sentinel.** Windows has no signals, so the runner arms it through a control file. A window counts as
raised when it is above the user window in the z-order and overlaps it. The desktop measure is
`IVirtualDesktopManager`; a dialog on the user's app is a window owned by the user window.
@@ -505,6 +508,10 @@ Where the multi-app tasks differ on Windows:
it off for the bench user before the trials: `org.openoffice.Office.Update/Update/Enabled` and
`org.openoffice.Office.Jobs/Jobs/UpdateCheck/Arguments/AutoCheckEnabled` set to `false` (as
`Software\Policies\LibreOffice` registry policies, which an administrator writes).
+* **Codex approvals.** Codex computer use names a Windows app by its process when it asks for approval:
+ `soffice.bin` for LibreOffice, `GnuCash`, `Google Chrome`, and the Electron apps by their named executables
+ (`copperfield mail`, ...). Put those names on the adapter's `--allow` list for CDB-G02 to CDB-G04; a name
+ missing from it is declined and the setup cannot use that app.
* **Leftovers.** Before and after a trial the launcher stops the task's own executables in the desktop
session by exact name and whatever listens on the task's loopback port (4184 for CDB-G04).
* **Validation.** `windows\cdb_oracle.py` stands in for the agent: by default it applies the scripted
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/launch/windows_session.py b/libs/cua-driver-bench/automated-eval/macos_bench/launch/windows_session.py
index 4cfca3103c..53e5c767c8 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/launch/windows_session.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/launch/windows_session.py
@@ -194,6 +194,7 @@ def run(args: argparse.Namespace, command: list[str]) -> dict:
)
procs.append(agent)
own_tree: set[int] = set()
+ helpers: dict[int, dict] = {}
session = W.session_of(os.getpid())
front = G.FrontWatch()
violations: list[dict] = []
@@ -204,6 +205,14 @@ def run(args: argparse.Namespace, command: list[str]) -> dict:
# Only the runner's own (interactive) session: remote administration (SSH in session 0) is not the GUI.
here = [r for r in rows if W.session_of(r[0]) == session]
for pid, label in G.new_side_doors(here, baseline, os.getpid(), own_tree):
+ by = G.started_by_tool(rows, pid, args.tool_exe)
+ if (
+ by
+ ): # the setup's own binary started it: part of the tool, logged, not a side door
+ helpers.setdefault(
+ pid, {"t_s": round(time.monotonic() - t0, 1), "detail": label, "parent": by}
+ )
+ continue
violations.append(
{
"t_s": round(time.monotonic() - t0, 1),
@@ -269,7 +278,8 @@ def run(args: argparse.Namespace, command: list[str]) -> dict:
evaluation={k: evaluation.get(k) for k in ("passed", "score", "checks", "diagnostics")},
checker_passed=bool(evaluation.get("passed")),
gui_only={"violation": bool(violations), "violations": violations[:20],
- "mentions": soft, "frontmost_seen": sorted(front.seen)},
+ "mentions": soft, "frontmost_seen": sorted(front.seen),
+ "tool_helpers": list(helpers.values())[:20]},
passed=passed,
disturbance=disturbance,
background=bg,
@@ -307,6 +317,10 @@ def main(argv: list[str]) -> int:
parser.add_argument("--label", default="")
parser.add_argument("--timeout", type=float, default=360.0)
parser.add_argument("--no-enforce", dest="enforce", action="store_false")
+ parser.add_argument(
+ "--tool-exe", action="append", default=[],
+ help="executable of the setup's own tool (repeatable): a shell it starts itself is not a side door",
+ ) # fmt: skip
args = parser.parse_args(argv[:split])
command = argv[split + 1 :]
if sys.platform != "win32":
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_windows_tools.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_windows_tools.py
index 03a7c99325..0ae57ae33d 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_windows_tools.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_windows_tools.py
@@ -117,6 +117,19 @@ class GuardTest(unittest.TestCase):
def test_descendants(self) -> None:
self.assertEqual(G.descendants(self.ROWS, 200), {200, 210, 220})
+ def test_shell_started_by_the_tool_binary_is_a_helper(self) -> None:
+ rows = [
+ (10, 1, "cua-driver.exe"),
+ (11, 10, "powershell.exe"),
+ (20, 1, "python.exe"),
+ (21, 20, "powershell.exe"),
+ ]
+ self.assertEqual(G.started_by_tool(rows, 11, ["cua-driver.exe"]), "cua-driver")
+ self.assertIsNone(
+ G.started_by_tool(rows, 21, ["cua-driver.exe"])
+ ) # agent code in an interpreter
+ self.assertIsNone(G.started_by_tool(rows, 11, []))
+
def test_new_side_doors(self) -> None:
hits = G.new_side_doors(self.ROWS, baseline={4, 100}, runner=200, own={250})
self.assertEqual(sorted(hits), [(230, "Windows PowerShell"), (240, "Windows Terminal")])
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/windows/guard.py b/libs/cua-driver-bench/automated-eval/macos_bench/windows/guard.py
index 2a6a1b9fbb..aa2ac4f56d 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/windows/guard.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/windows/guard.py
@@ -92,6 +92,21 @@ def new_side_doors(
]
+def started_by_tool(
+ rows: Iterable[tuple[int, int, str]], pid: int, tool_exes: Iterable[str]
+) -> str | None:
+ """The executable name of `pid`'s parent when that parent is one of the setup's own tool binaries
+ (`--tool-exe`), else None. A shell a driver binary starts for itself (to query a process, say) is part of
+ the tool, as a helper process is on macOS; a shell started by code the agent supplied (a harness's Python
+ `exec`, a JavaScript REPL) has an interpreter as its parent and stays a side door."""
+ names = {name_of(t) for t in tool_exes}
+ if not names:
+ return None
+ parent = {p: pp for p, pp, _ in rows}.get(pid)
+ exe = {p: e for p, _, e in rows}.get(parent, "") if parent is not None else ""
+ return name_of(exe) if name_of(exe) in names else None
+
+
def scan_tool_use(name: str, tool_input: Any) -> tuple[list[str], list[str]]:
hard, soft = v2.scan_tool_use(name, tool_input)
if name.startswith("mcp__"):