Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver-bench): accept both AT-SPI button role names in the GTK interaction test (#5775)

injaneity194348f44fa1
diff --git a/.github/workflows/ci-cua-driver-bench.yml b/.github/workflows/ci-cua-driver-bench.yml
index c2e6167341..c9df43c628 100644
--- a/.github/workflows/ci-cua-driver-bench.yml
+++ b/.github/workflows/ci-cua-driver-bench.yml
@@ -23,7 +23,6 @@ jobs:
harness:
if: ${{ !startsWith(github.head_ref, 'release-please--branches--') }}
name: Harness tests (no GUI, no model)
- # The harness is macOS-only (sw_vers, AppKit apps); BenchLab targets macOS 26.
runs-on: macos-26
timeout-minutes: 20
steps:
@@ -45,6 +44,28 @@ jobs:
uv run --no-project --python 3.12 --with pytest==8.4.2
python -m pytest -q tests probes/tests swift/tests
+ linux-desktop:
+ name: Linux headless desktop contracts (no model)
+ runs-on: ubuntu-latest
+ timeout-minutes: 10
+ steps:
+ - name: Check out Cua
+ uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
+ - name: Install desktop dependencies
+ run: |
+ sudo apt-get update
+ sudo apt-get install -y --no-install-recommends \
+ xvfb dbus-x11 at-spi2-core openbox picom python3-gi gir1.2-gtk-3.0 gir1.2-atspi-2.0 \
+ xdotool x11-utils libxss1 ffmpeg lsof python3-pytest
+ - name: Exercise GTK, doctor, sentinel and evidence contracts
+ working-directory: libs/cua-driver-bench/automated-eval/macos_bench
+ env:
+ CDB_BENCH_WORK: ${{ runner.temp }}/cdb-bench-work
+ run: |
+ timeout 180s xvfb-run -a --server-args="-screen 0 1920x1080x24" \
+ dbus-run-session -- bash -lc \
+ 'openbox >/tmp/cua-openbox.log 2>&1 & picom --backend xrender --config /dev/null >/tmp/cua-picom.log 2>&1 & sleep 2; /usr/bin/python3 -m pytest -q tests/test_pilot_runner.py tests/test_runner_flow.py tests/test_bench_v2.py tests/test_analyze.py tests/test_cdb_adapter.py'
+
taskpack:
if: ${{ !startsWith(github.head_ref, 'release-please--branches--') }}
name: Task pack tests
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/.gitignore b/libs/cua-driver-bench/automated-eval/macos_bench/.gitignore
new file mode 100644
index 0000000000..357b7dc03c
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/.gitignore
@@ -0,0 +1 @@
+/local-work/
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/README.md b/libs/cua-driver-bench/automated-eval/macos_bench/README.md
index dd86d19417..1f373f6ffe 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/README.md
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/README.md
@@ -1,4 +1,4 @@
-# Bench v2 harness (macOS)
+# Bench v2 harness (macOS and Linux/X11)
`cx-browser-use-cli` is staged and unavailable for scored execution: preflight always fails.
It requires `CDB_CODEX_AUTH=chatgpt` and browser-only tasks. Codex driving the CLI is not
@@ -35,3 +35,50 @@ Tool-reported setup errors become infrastructure failures with `infra_pattern`;
model prose alone never triggers this classification.
Keep results, streams and recordings out of git.
+
+## Linux desktop setup
+
+Install `xvfb dbus-x11 at-spi2-core openbox picom python3-gi gir1.2-gtk-3.0
+ gir1.2-atspi-2.0 xdotool x11-utils libxss1 ffmpeg lsof`, plus the applications
+required by each pack task's `platform/launch.linux.json` prerequisites. Use system
+Python for GTK/AT-SPI; no Swift build is required. Older non-v2 BenchLab modes are
+explicitly unsupported on Linux. Codex Sky and Claude Desktop arms are skipped
+with an `unsupported-platform` reason before any MCP or daemon setup.
+
+From this directory, using an explicitly selected Linux driver and matching skills:
+
+```sh
+export CDB_BENCH_WORK="$PWD/local-work"
+xvfb-run -a --server-args="-screen 0 1920x1080x24" \
+ dbus-run-session -- bash -lc \
+ 'openbox & picom --backend xrender --config /dev/null & sleep 2;
+ python3 run_bench.py preflight --bench-v2 --arms cc-cua-driver \
+ --cua-bin "$CUA_DRIVER_BIN" --cua-skills "$CUA_SKILLS" --pins "$LINUX_PINS"'
+```
+
+The pin file must declare `platform: "linux"`, `cua_driver_version`,
+`cua_driver_binary_sha256`, and `cua_skills_tree_sha256`, matching the selected
+executable and skill tree. Main/script arms additionally require the existing
+`cua_main` version, binary, git and skills identities. Never reuse the macOS pin
+file. Existing model, quota, disk-space and pack-integrity gates still apply.
+Keep `local-work/` outside git; the disposable-environment flag is still required
+for pack cleanup.
+
+`desktop_runtime.py` owns environment allowlisting, process groups, app command
+resolution, driver launch/readiness, health parsing, doctor and screenshots.
+Doctor connects to X11 and the AT-SPI registry and checks driver capabilities and
+ffmpeg's x11grab support. Linux uses the existing `CuaRecorder` on its dedicated
+control daemon, whose backend already uses x11grab; that daemon also handles
+window placement. No separate Linux recording implementation is used.
+
+The GTK helpers preserve BenchLab's state/event evaluator contracts. The sentinel
+observes EWMH focus/stacking/workspaces, AT-SPI focus, root pointer coordinates,
+clipboard ownership and transient windows. It never refocuses itself after arming.
+XScreenSaver idle values include synthetic input: Linux HID counts and human-input
+inference are **unmeasured**, not zero. Screenshots and app/sentinel/driver logs
+are retained in trial evidence directories. Raw evidence and results are internal.
+
+Headless contract tests use independent `xdotool` XTest actions. The Linux CI job
+runs doctor, GTK interactions, sentinel and screenshot tests without a driver;
+only driver health and recording smoke requires `CUA_DRIVER_BIN`. No model is
+called by these tests.
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/analyze.py b/libs/cua-driver-bench/automated-eval/macos_bench/analyze.py
index 6ec15cd9dc..5f7fde5769 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/analyze.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/analyze.py
@@ -444,7 +444,10 @@ def validate_row(row: Any, where: str) -> None:
f"{where}: field 'disturbance.available' must be true or false, got {_short(dist.get('available'))}"
)
if dist["available"]:
- if not isinstance(dist.get("human_input_suspected"), bool):
+ unmeasured_hid = row.get("platform") == "linux" and dist.get("hid_events") is None
+ if not isinstance(dist.get("human_input_suspected"), bool) and not (
+ unmeasured_hid and dist.get("human_input_suspected") is None
+ ):
raise TrialFormatError(
f"{where}: field 'disturbance.human_input_suspected' must be true or false when "
f"disturbance.available is true, got {_short(dist.get('human_input_suspected'))}"
@@ -460,12 +463,13 @@ def validate_row(row: Any, where: str) -> None:
_require_num(where, dist, key, label=f"disturbance.{key}")
_optional_num(where, dist, "scrolls_leaked", label="disturbance.scrolls_leaked")
hid = dist.get("hid_events")
- if not isinstance(hid, dict):
+ if not isinstance(hid, dict) and not unmeasured_hid:
raise TrialFormatError(
f"{where}: field 'disturbance.hid_events' must be an object when disturbance.available is true, got {_short(hid)}"
)
- for key in HID_KEYS:
- _require_num(where, hid, key, label=f"disturbance.hid_events.{key}")
+ if not unmeasured_hid:
+ for key in HID_KEYS:
+ _require_num(where, hid, key, label=f"disturbance.hid_events.{key}")
def _validate_rows(rows: Sequence[Any], labels: Sequence[str]) -> None:
@@ -864,16 +868,17 @@ def _latency(trials: list[Trial], arms: Sequence[str]) -> dict[str, Any]:
return out
-def _any_disturbance(d: dict[str, Any]) -> bool:
+def _any_disturbance(d: dict[str, Any]) -> bool | None:
hid = d["hid_events"]
- return bool(
+ observed = bool(
d["front_changes"] > 0
or d["key_loss"] > 0
or d["keystrokes_leaked"] > 0
or d["clicks_leaked"] > 0
- or any(hid[k] > 0 for k in HID_KEYS)
+ or (hid is not None and any(hid[k] > 0 for k in HID_KEYS))
or d["pointer_deviation_episodes"] > 0
)
+ return observed if observed or hid is not None else None
def _disturbance(trials: list[Trial], arms: Sequence[str]) -> dict[str, Any]:
@@ -885,24 +890,33 @@ def _disturbance(trials: list[Trial], arms: Sequence[str]) -> dict[str, Any]:
used = [t for t in available if not t.disturbance["human_input_suspected"]]
def series(fn) -> list[float]:
- return [float(fn(t.disturbance)) for t in used]
+ values = [fn(t.disturbance) for t in used]
+ return [float(value) for value in values if value is not None]
fields = {
"front_changes": series(lambda d: d["front_changes"]),
"key_loss": series(lambda d: d["key_loss"]),
"keystrokes_leaked": series(lambda d: d["keystrokes_leaked"]),
"clicks_leaked": series(lambda d: d["clicks_leaked"]),
- "hid_events_total": series(lambda d: sum(d["hid_events"][k] for k in HID_KEYS)),
+ "hid_events_total": series(
+ lambda d: sum(d["hid_events"][k] for k in HID_KEYS)
+ if d["hid_events"] is not None
+ else None
+ ),
"pointer_max_deviation_px": series(lambda d: d["pointer_max_deviation_px"]),
"pointer_deviation_episodes": series(lambda d: d["pointer_deviation_episodes"]),
}
for key in HID_KEYS:
- fields[f"hid_events_{key}"] = series(lambda d, key=key: d["hid_events"][key])
+ fields[f"hid_events_{key}"] = series(
+ lambda d, key=key: d["hid_events"][key] if d["hid_events"] is not None else None
+ )
if all(_is_num(t.disturbance.get("scrolls_leaked")) for t in used) and used:
fields["scrolls_leaked"] = series(lambda d: d["scrolls_leaked"])
- any_count = sum(1 for t in used if _any_disturbance(t.disturbance))
- interval = wilson_interval(any_count, len(used))
+ measured = all(t.disturbance["hid_events"] is not None for t in used)
+ any_count = sum(_any_disturbance(t.disturbance) for t in used) if measured else None
+ any_n = len(used) if measured else None
+ interval = wilson_interval(any_count, any_n) if measured else None
out[arm] = {
"n_non_excluded": len(kept),
"n_available": len(available),
@@ -918,9 +932,10 @@ def _disturbance(trials: list[Trial], arms: Sequence[str]) -> dict[str, Any]:
for name, vals in fields.items()
},
"any_disturbance": {
+ "measured": measured,
"count": any_count,
- "n": len(used),
- "fraction": any_count / len(used) if used else None,
+ "n": any_n,
+ "fraction": any_count / any_n if any_n else None,
"wilson_low": interval[0] if interval else None,
"wilson_high": interval[1] if interval else None,
},
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/bench_v2.py b/libs/cua-driver-bench/automated-eval/macos_bench/bench_v2.py
index 7f2def75db..94c746a59c 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/bench_v2.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/bench_v2.py
@@ -103,7 +103,20 @@ def validate_tasks(
# ------------------------------------------------------------------ GUI-only
+LINUX_TERMINALS = (
+ "xterm",
+ "gnome-terminal",
+ "gnome-terminal-server",
+ "gnome-terminal-",
+ "konsole",
+ "xfce4-terminal",
+ "kitty",
+ "alacritty",
+ "Alacritty",
+ "org.gnome.Terminal",
+)
SIDE_DOOR_BUNDLES = {
+ **{name: name for name in LINUX_TERMINALS},
"com.apple.Terminal": "Terminal",
"com.googlecode.iterm2": "iTerm",
"com.apple.ScriptEditor2": "Script Editor",
@@ -236,7 +249,9 @@ def terminal_session_ttys(rows: list[tuple[int, int, str, str]], exclude_pid: in
never reads, and any tty its descendants open (agents, MCP servers, apps it starts)."""
parent = {pid: ppid for pid, ppid, _, _ in rows}
tty_of = {pid: tty for pid, _, tty, _ in rows}
- terminals = {pid for pid, _, _, comm in rows if comm.rsplit("/", 1)[-1] == "Terminal"}
+ terminals = {
+ pid for pid, _, _, comm in rows if comm.rsplit("/", 1)[-1] in ("Terminal", *LINUX_TERMINALS)
+ }
def ancestors(pid: int) -> set[int]:
seen: set[int] = set()
@@ -246,9 +261,9 @@ def terminal_session_ttys(rows: list[tuple[int, int, str, str]], exclude_pid: in
return seen
def device(tty: str) -> str | None:
- if tty in ("??", "-", ""):
+ if tty in ("??", "?", "-", ""):
return None
- return "/dev/" + (tty if tty.startswith("tty") else "tty" + tty)
+ return "/dev/" + (tty if tty.startswith(("tty", "pts/")) else "tty" + tty)
excluded = {device(tty_of.get(p, "")) for p in ancestors(exclude_pid)} | {
device(tty_of.get(exclude_pid, ""))
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/cdb_adapter.py b/libs/cua-driver-bench/automated-eval/macos_bench/cdb_adapter.py
index 0ef695769c..18d731584d 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/cdb_adapter.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/cdb_adapter.py
@@ -28,7 +28,6 @@ import json
import os
import re
import shutil
-import signal
import subprocess
import sys
import time
@@ -37,6 +36,7 @@ from pathlib import Path
from typing import Any
import claude_arms as ca
+import desktop_runtime as desktop
SKIP_KINDS = {"terminal", "editor"}
EVAL_ENTRY = "/usr/local/libexec/cdb-eval-run"
@@ -87,7 +87,9 @@ def node_facts(node: str = "/usr/local/bin/node") -> dict[str, str]:
def _descriptor(spec: dict[str, Any]) -> dict[str, Any]:
- path = pack_tasks_root() / spec["pack_task"] / "platform/launch.macos.json"
+ path = (
+ pack_tasks_root() / spec["pack_task"] / f"platform/launch.{desktop.detect_platform()}.json"
+ )
return json.loads(path.read_text("utf-8"))
@@ -231,6 +233,13 @@ def with_force_accessibility(
if kind == "electron":
return [*argv, flag]
if kind == "browser":
+ if desktop.detect_platform() == "linux" and Path(argv[0]).name in (
+ "google-chrome",
+ "google-chrome-stable",
+ "chromium",
+ "chromium-browser",
+ ):
+ return [argv[0], flag, *argv[1:]]
for i, part in enumerate(argv):
if part.endswith(CHROME_EXE):
return [*argv[: i + 1], flag, *argv[i + 1 :]]
@@ -242,10 +251,14 @@ def kill_policy(spec: dict[str, Any]) -> tuple[list[str], list[int]]:
text = json.dumps(_descriptor(spec))
names: list[str] = []
if "electron" in text:
- names.append("Electron")
+ names.append("Electron" if desktop.detect_platform() == "macos" else "electron")
if "Google Chrome" in text:
names.append("Google Chrome")
- if "soffice" in text:
+ if desktop.detect_platform() == "linux" and any(
+ name in text for name in ("google-chrome", "chromium")
+ ):
+ names += ["chrome", "chromium"]
+ if "soffice" in text or "libreoffice" in text:
names += ["soffice", "soffice.bin"]
if "gnucash" in text.lower():
names += ["gnucash", "Gnucash"]
@@ -263,7 +276,9 @@ def clear_leftovers(spec: dict[str, Any]) -> None:
subprocess.run(["pkill", "-x", name], capture_output=True)
for port in ports:
pids = subprocess.run(
- ["/usr/sbin/lsof", "-ti", f"tcp:{port}", "-sTCP:LISTEN"], capture_output=True, text=True
+ [desktop.tool("lsof"), "-ti", f"tcp:{port}", "-sTCP:LISTEN"],
+ capture_output=True,
+ text=True,
).stdout.split()
for pid in pids:
subprocess.run(["kill", "-9", pid], capture_output=True)
@@ -277,9 +292,7 @@ class CdbTask:
self.spec = spec
self.seed = seed
self.bundle = pack_tasks_root() / spec["pack_task"]
- self.descriptor = json.loads(
- (self.bundle / "platform/launch.macos.json").read_text("utf-8")
- )
+ self.descriptor = _descriptor(spec)
self.artifacts = artifacts
self.workspace = Path(
self.descriptor["workspace_root"].replace("${HOME}", str(Path.home()))
@@ -309,7 +322,9 @@ class CdbTask:
self, command: list[str], timeout: int = 300, **extra: str
) -> subprocess.CompletedProcess[str]:
argv = [self.sub(part, **extra) for part in command]
- env = {"HOME": str(Path.home()), "PATH": APP_PATH, "LANG": "en_US.UTF-8"}
+ env = desktop.desktop_env(
+ Path.home(), {"PATH": APP_PATH} if desktop.detect_platform() == "macos" else None
+ )
return subprocess.run(
argv, cwd=str(self.bundle), env=env, capture_output=True, text=True, timeout=timeout
)
@@ -336,11 +351,19 @@ class CdbTask:
def start_apps(self, windows: Any = None, force_accessibility: bool = False) -> None:
for app in self.apps():
- env = {"HOME": str(Path.home()), "PATH": APP_PATH, "LANG": "en_US.UTF-8"}
+ env = desktop.desktop_env(
+ Path.home(), {"PATH": APP_PATH} if desktop.detect_platform() == "macos" else None
+ )
env.update({k: self.sub(v) for k, v in (app.get("env") or {}).items()})
cwd = self.sub(app["cwd"]) if app.get("cwd") else str(self.bundle)
argv = [self.sub(p) for p in app["command"]]
- if app.get("kind") == "electron" and os.environ.get("CDB_ELECTRON_SHARED") != "1":
+ for key in app.get("unset_env", []):
+ env.pop(key, None)
+ if (
+ desktop.detect_platform() == "macos"
+ and app.get("kind") == "electron"
+ and os.environ.get("CDB_ELECTRON_SHARED") != "1"
+ ):
argv = electron_argv(argv, cwd, app_display_name(app))
self.log.append(
f"electron {app['id']} as {electron_bundle_id(app_display_name(app))}"
@@ -350,16 +373,16 @@ class CdbTask:
if flagged != argv:
self.log.append(f"{app['id']}: {ca.FORCE_ACCESSIBILITY_FLAG}")
argv = flagged
- out = (self.artifacts / f"app-{app['id']}.log").open("ab")
- proc = subprocess.Popen(
- argv,
- cwd=cwd,
- env=env,
- stdin=subprocess.DEVNULL,
- stdout=out,
- stderr=out,
- start_new_session=True,
- )
+ with (self.artifacts / f"app-{app['id']}.log").open("ab") as out:
+ proc = subprocess.Popen(
+ argv,
+ cwd=cwd,
+ env=env,
+ stdin=subprocess.DEVNULL,
+ stdout=out,
+ stderr=out,
+ start_new_session=True,
+ )
self.procs.append(proc)
ready = (app.get("ready") or {}).get("http")
if ready:
@@ -456,12 +479,8 @@ class CdbTask:
def stop_apps(self) -> None:
for proc in reversed(self.procs):
- for sig in (signal.SIGTERM, signal.SIGKILL):
- try:
- os.killpg(proc.pid, sig)
- except (ProcessLookupError, PermissionError):
- break
- time.sleep(0.4)
+ desktop.stop_group(proc.pid)
+ proc.wait(timeout=5)
self.procs.clear()
clear_leftovers(self.spec)
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/claude_arms.py b/libs/cua-driver-bench/automated-eval/macos_bench/claude_arms.py
index f6d6e15d69..bdceb47b5d 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/claude_arms.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/claude_arms.py
@@ -32,6 +32,8 @@ from dataclasses import dataclass
from pathlib import Path
from typing import Any
+import desktop_runtime as desktop
+
HERE = Path(__file__).resolve().parent
# WORK holds everything the harness creates or downloads: the private Cua Driver install, the Codex MCP config,
# the spend ledger, the HOLD file and the run folders. CDB_BENCH_WORK overrides it.
@@ -119,6 +121,7 @@ class CuaBuild:
tuple[str, str], ...
] = () # Amendment 6: extra daemon + MCP env (the script flag)
prompt_addendum: str | None = None # Amendment 6: text appended to the system prompt
+ executable: Path | None = None
@property
def binary_pin(self) -> str:
@@ -128,7 +131,7 @@ class CuaBuild:
@property
def bin(self) -> Path:
- return self.app / "Contents/MacOS/cua-driver"
+ return self.executable or self.app / "Contents/MacOS/cua-driver"
@property
def home(self) -> Path:
@@ -376,12 +379,7 @@ def load_pins() -> dict[str, Any]:
def cua_env(home: Path) -> dict[str, str]:
"""Environment of the Cua Driver daemon and of its MCP client: own HOME, telemetry off."""
- return {
- "HOME": str(home),
- "PATH": "/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin",
- "CUA_DRIVER_RS_TELEMETRY_ENABLED": "false",
- "DO_NOT_TRACK": "1", # the main build's shared telemetry crate also honours this (Amendment 3)
- }
+ return desktop.desktop_env(home)
def resolve_codex_cu_config(explicit: Path | None = None) -> Path | None:
@@ -580,15 +578,7 @@ def cua_health(
socket: str = AGENT_SOCKET, home: Path | None = None, binary: Path | None = None
) -> dict[str, Any]:
"""Daemon-reported build (version, exe sha256, git sha) from the live daemon via health_report."""
- try:
- done = cua_cli(
- "call", "health_report", "{}", socket=socket, home=home, timeout=30, binary=binary
- )
- data = json.loads(done.stdout[done.stdout.index("{") :])
- checks = {c.get("name"): c.get("status") for c in data.get("checks", [])}
- return {"ok": done.returncode == 0, "build": data.get("build", {}), "checks": checks}
- except (OSError, ValueError, subprocess.TimeoutExpired) as error:
- return {"ok": False, "error": f"{type(error).__name__}: {error}"}
+ return desktop.cua_health(binary or CUA_BIN, socket, home or CUA_STATE / "home")
def stop_cua_daemon(socket: str, home: Path, binary: Path | None = None) -> None:
@@ -609,36 +599,14 @@ def start_cua_daemon(
env_extra: dict[str, str] | None = None,
) -> subprocess.Popen[bytes]:
"""Private daemon of the release under test: own socket, own HOME, telemetry off."""
- binary = binary or CUA_BIN
- home = state_dir / "home"
- home.mkdir(parents=True, exist_ok=True)
- try:
- os.unlink(socket)
- except OSError:
- pass
- args = [str(binary), "serve", "--socket", socket, "--dangerously-bypass-approvals"]
- if not overlay:
- args.append("--no-overlay")
- log = (state_dir / log_name).open("ab")
- proc = subprocess.Popen(
- args,
- env={**cua_env(home), **(env_extra or {})},
- stdin=subprocess.DEVNULL,
- stdout=log,
- stderr=log,
- start_new_session=True,
+ return desktop.start_driver(
+ binary or CUA_BIN,
+ socket,
+ state_dir,
+ overlay=overlay,
+ log_name=log_name,
+ env_extra=env_extra,
)
- deadline = time.monotonic() + 30
- while time.monotonic() < deadline:
- if proc.poll() is not None:
- raise RuntimeError(
- f"Cua Driver daemon exited early (rc={proc.returncode}); see {state_dir / log_name}"
- )
- done = _run([str(binary), "status", "--socket", socket], env=cua_env(home), timeout=10)
- if done.returncode == 0:
- return proc
- time.sleep(0.5)
- raise RuntimeError("Cua Driver daemon did not become ready")
# ---------------------------------------------------------------- pins (observed values)
@@ -682,8 +650,11 @@ def observed_pins(include_codex: bool = True, include_cua: bool = True) -> dict[
else None
)
out["claude_code_version"] = claude
- out["macos_version"] = _run(["sw_vers", "-productVersion"]).stdout.strip()
- out["macos_build"] = _run(["sw_vers", "-buildVersion"]).stdout.strip()
+ if desktop.detect_platform() == "macos":
+ out["macos_version"] = _run(["sw_vers", "-productVersion"]).stdout.strip()
+ out["macos_build"] = _run(["sw_vers", "-buildVersion"]).stdout.strip()
+ else:
+ out["platform"] = "linux"
if include_cua:
out["cua_driver_version_string"] = cua_version_string()
out["cua_driver_version"] = out["cua_driver_version_string"].replace("cua-driver ", "")
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/claude_driver.py b/libs/cua-driver-bench/automated-eval/macos_bench/claude_driver.py
index 1b6c5f1d01..8abeacbbbc 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/claude_driver.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/claude_driver.py
@@ -23,19 +23,11 @@ from pathlib import Path
from typing import Any
import claude_arms as ca
+from desktop_runtime import stop_group as kill_group
ELICIT_APP = re.compile(r'Allow Computer Use to use "(.+)"\?')
-def kill_group(pid: int, grace: float = 1.0) -> None:
- for sig in (signal.SIGTERM, signal.SIGKILL):
- try:
- os.killpg(pid, sig)
- except OSError:
- return
- time.sleep(grace)
-
-
def answer_elicitation(message: str, allowed_apps: set[str]) -> dict[str, Any]:
"""Decision for one MCP elicitation request. Only an exact `Allow Computer Use to use "<App>"?`
for an app on the task's allowlist is accepted."""
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/desktop_runtime.py b/libs/cua-driver-bench/automated-eval/macos_bench/desktop_runtime.py
new file mode 100644
index 0000000000..88741d09cd
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/desktop_runtime.py
@@ -0,0 +1,273 @@
+# SPDX-License-Identifier: FSL-1.1-MIT
+# Copyright (c) 2026 Cua AI, Inc.
+
+from __future__ import annotations
+
+import json
+import os
+import platform
+import re
+import shutil
+import signal
+import subprocess
+import time
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Literal, Mapping, Sequence
+
+Platform = Literal["macos", "linux"]
+HERE = Path(__file__).resolve().parent
+SESSION_KEYS = (
+ "DISPLAY",
+ "XAUTHORITY",
+ "DBUS_SESSION_BUS_ADDRESS",
+ "XDG_RUNTIME_DIR",
+ "XDG_DATA_DIRS",
+ "AT_SPI_BUS_ADDRESS",
+ "GTK_MODULES",
+ "QT_ACCESSIBILITY",
+ "QT_LINUX_ACCESSIBILITY_ALWAYS_ON",
+ "NO_AT_BRIDGE",
+)
+
+
+@dataclass(frozen=True)
+class Check:
+ name: str
+ status: Literal["pass", "fail", "skip"]
+ detail: str
+
+
+def detect_platform(system: str | None = None) -> Platform:
+ name = system if system is not None else platform.system()
+ if name == "Darwin":
+ return "macos"
+ if name == "Linux":
+ return "linux"
+ raise ValueError(f"unsupported-platform: {name}")
+
+
+def tool(name: str) -> str:
+ found = shutil.which(name)
+ if found:
+ return found
+ if detect_platform() == "macos":
+ fallback = Path("/opt/homebrew/bin") / name
+ if fallback.is_file():
+ return str(fallback)
+ raise FileNotFoundError(f"Install {name} and add it to PATH")
+
+
+def desktop_env(home: Path, extra: Mapping[str, str] | None = None) -> dict[str, str]:
+ linux = detect_platform() == "linux"
+ env = {
+ "HOME": str(home),
+ "TMPDIR": str(home / "tmp")
+ if (home / "tmp").is_dir()
+ else os.environ.get("TMPDIR", "/tmp"),
+ "PATH": os.environ.get("PATH", "/usr/local/bin:/usr/bin:/bin")
+ if linux
+ else "/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin",
+ "USER": os.environ.get("USER", ""),
+ "LANG": os.environ.get("LANG", "C.UTF-8") if linux else "en_US.UTF-8",
+ "CUA_DRIVER_RS_TELEMETRY_ENABLED": "false",
+ "DO_NOT_TRACK": "1",
+ }
+ if linux:
+ env.update({key: os.environ[key] for key in SESSION_KEYS if key in os.environ})
+ # HOME is private per trial; keep the session's default X authority reachable.
+ authority = Path(os.path.expanduser("~")) / ".Xauthority"
+ if "XAUTHORITY" not in env and authority.is_file():
+ env["XAUTHORITY"] = str(authority)
+ env.update(extra or {})
+ return env
+
+
+def bench_app_argv(name: str, build_dir: Path | None = None) -> list[str]:
+ name = name.removesuffix(".app")
+ if name not in ("BenchLab", "BenchSentinel"):
+ raise ValueError(f"unknown bench app: {name}")
+ if detect_platform() == "linux":
+ script = HERE / "linux" / ("benchlab.py" if name == "BenchLab" else "sentinel.py")
+ if not script.is_file():
+ raise FileNotFoundError(script)
+ return ["/usr/bin/python3", str(script)]
+ if build_dir is None:
+ raise ValueError("--build-dir is required on macOS")
+ exes = sorted((build_dir / f"{name}.app" / "Contents/MacOS").iterdir())
+ return [str(next(p for p in exes if p.is_file() and os.access(p, os.X_OK)))]
+
+
+def stop_group(pid: int) -> None:
+ for sig in (signal.SIGTERM, signal.SIGKILL):
+ try:
+ os.killpg(pid, sig)
+ except OSError:
+ return
+ if sig == signal.SIGTERM:
+ time.sleep(0.3)
+
+
+def start_app(
+ argv: Sequence[str], home: Path, log: Path, *, env: Mapping[str, str] | None = None
+) -> subprocess.Popen:
+ log.parent.mkdir(parents=True, exist_ok=True)
+ (home / "tmp").mkdir(parents=True, exist_ok=True)
+ with log.open("ab") as output:
+ proc = subprocess.Popen(
+ list(argv),
+ env=desktop_env(home, env),
+ stdin=subprocess.DEVNULL,
+ stdout=output,
+ stderr=output,
+ start_new_session=True,
+ )
+ time.sleep(0.15)
+ if proc.poll() is not None:
+ stop_group(proc.pid)
+ proc.wait()
+ raise RuntimeError(f"{argv[0]} exited ({proc.returncode}); see {log}")
+ return proc
+
+
+def start_driver(
+ binary: Path,
+ socket: str,
+ state: Path,
+ *,
+ overlay: bool = True,
+ log_name: str = "daemon.log",
+ env_extra: Mapping[str, str] | None = None,
+) -> subprocess.Popen:
+ Path(socket).unlink(missing_ok=True)
+ args = [str(binary), "serve", "--socket", socket, "--dangerously-bypass-approvals"]
+ if not overlay:
+ args.append("--no-overlay")
+ proc = start_app(args, state / "home", state / log_name, env=env_extra)
+ try:
+ deadline = time.monotonic() + 30
+ while time.monotonic() < deadline:
+ if proc.poll() is not None:
+ raise RuntimeError(f"driver exited ({proc.returncode}); see {state / log_name}")
+ done = subprocess.run(
+ [str(binary), "status", "--socket", socket],
+ env=desktop_env(state / "home", env_extra),
+ capture_output=True,
+ timeout=5,
+ )
+ if done.returncode == 0:
+ return proc
+ time.sleep(0.3)
+ raise TimeoutError(f"driver not ready; see {state / log_name}")
+ except BaseException:
+ stop_group(proc.pid)
+ proc.wait(timeout=5)
+ raise
+
+
+def cua_health(binary: Path, socket: str, home: Path) -> dict:
+ try:
+ done = subprocess.run(
+ [str(binary), "--socket", socket, "call", "health_report", "{}"],
+ env=desktop_env(home),
+ capture_output=True,
+ text=True,
+ timeout=30,
+ )
+ data = json.loads(done.stdout[done.stdout.index("{") :])
+ return {
+ "ok": done.returncode == 0,
+ "build": data.get("build", {}),
+ "checks": {c.get("name"): c.get("status") for c in data.get("checks", [])},
+ }
+ except (OSError, ValueError, subprocess.TimeoutExpired) as error:
+ return {"ok": False, "error": f"{type(error).__name__}: {error}"}
+
+
+def capabilities_ok(health: dict) -> bool:
+ keys = (
+ ("ax_capability", "screen_capture_capability")
+ if detect_platform() == "linux"
+ else ("tcc_accessibility", "tcc_screen_recording")
+ )
+ return bool(health.get("ok")) and all(
+ health.get("checks", {}).get(key) == "pass" for key in keys
+ )
+
+
+def doctor(binary: Path, socket: str, home: Path) -> list[Check]:
+ checks = []
+ if detect_platform() == "linux":
+ probes = {
+ "display": (
+ ["xdpyinfo", "-display", os.environ.get("DISPLAY", "")],
+ "Start Xvfb/X11 and set DISPLAY",
+ ),
+ "at-spi": (
+ [
+ "/usr/bin/python3",
+ "-c",
+ "import gi; gi.require_version('Atspi', '2.0'); from gi.repository import Atspi, Gio; bus=Gio.bus_get_sync(Gio.BusType.SESSION, None); address=bus.call_sync('org.a11y.Bus', '/org/a11y/bus', 'org.a11y.Bus', 'GetAddress', None, None, Gio.DBusCallFlags.NONE, 3000, None).unpack()[0]; ax=Gio.DBusConnection.new_for_address_sync(address, Gio.DBusConnectionFlags.AUTHENTICATION_CLIENT | Gio.DBusConnectionFlags.MESSAGE_BUS_CONNECTION, None, None); ax.call_sync('org.a11y.atspi.Registry', '/org/a11y/atspi/registry', 'org.freedesktop.DBus.Introspectable', 'Introspect', None, None, Gio.DBusCallFlags.NONE, 3000, None); d=Atspi.get_desktop(0); assert d is not None; d.get_child_count()",
+ ],
+ "Start dbus-run-session with at-spi2-core installed",
+ ),
+ "ffmpeg": (
+ ["ffmpeg", "-hide_banner", "-devices"],
+ "Install ffmpeg with x11grab support",
+ ),
+ }
+ for name, (argv, remedy) in probes.items():
+ try:
+ done = subprocess.run(
+ argv, env=desktop_env(home), capture_output=True, text=True, timeout=10
+ )
+ ok = done.returncode == 0 and (
+ name != "ffmpeg" or "x11grab" in done.stdout + done.stderr
+ )
+ checks.append(Check(name, "pass" if ok else "fail", "usable" if ok else remedy))
+ except (OSError, subprocess.TimeoutExpired) as error:
+ checks.append(Check(name, "fail", f"{remedy}: {error}"))
+ health = cua_health(binary, socket, home)
+ checks.append(
+ Check("driver", "pass" if capabilities_ok(health) else "fail", json.dumps(health))
+ )
+ return checks
+
+
+def capture_screenshot(display: str, destination: Path) -> Path:
+ destination.parent.mkdir(parents=True, exist_ok=True)
+ if detect_platform() == "macos":
+ subprocess.run(["/usr/sbin/screencapture", "-x", str(destination)], check=True, timeout=20)
+ return destination
+ info = subprocess.run(
+ [tool("xdpyinfo"), "-display", display],
+ capture_output=True,
+ text=True,
+ check=True,
+ timeout=5,
+ )
+ dimensions = re.search(r"dimensions:\s+(\d+x\d+)", info.stdout)
+ if dimensions is None:
+ raise RuntimeError(f"cannot determine X11 display dimensions: {display}")
+ with destination.with_suffix(".stderr").open("ab") as log:
+ subprocess.run(
+ [
+ tool("ffmpeg"),
+ "-nostdin",
+ "-y",
+ "-f",
+ "x11grab",
+ "-video_size",
+ dimensions[1],
+ "-i",
+ display,
+ "-frames:v",
+ "1",
+ str(destination),
+ ],
+ stdout=subprocess.DEVNULL,
+ stderr=log,
+ check=True,
+ timeout=20,
+ )
+ return destination
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/linux/benchlab.py b/libs/cua-driver-bench/automated-eval/macos_bench/linux/benchlab.py
new file mode 100644
index 0000000000..de50f715ef
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/linux/benchlab.py
@@ -0,0 +1,466 @@
+# SPDX-License-Identifier: FSL-1.1-MIT
+# Copyright (c) 2026 Cua AI, Inc.
+
+from __future__ import annotations
+
+import argparse
+import ctypes
+import json
+import os
+import sys
+import time
+from pathlib import Path
+
+import gi
+
+gi.require_version("Gtk", "3.0")
+gi.require_version("Gdk", "3.0")
+from gi.repository import Gdk, GLib, Gtk
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "probes" / "_common"))
+import benchlab_common as C
+
+MODES = ("listdrag", "hover", "tooltip", "irmodal", "irbanner", "irunsaved", "irconsent")
+
+
+class Lab:
+ def __init__(self, args: argparse.Namespace) -> None:
+ ctypes.CDLL(None).prctl(15, b"BenchLab", 0, 0, 0)
+ GLib.set_prgname("BenchLab")
+ GLib.set_application_name("BenchLab")
+ self.args = args
+ self.state = {"mode": args.mode, "seed": args.seed, "seq": 0}
+ self.params = getattr(C, "derive_" + args.mode)(args.seed)
+ self.fields = {}
+ self.loading = False
+ args.events.parent.mkdir(parents=True, exist_ok=True)
+ self.events = args.events.open("w", buffering=1)
+ self.window = Gtk.Window(title=f"BenchLab — {args.mode}")
+ self.window.set_wmclass("BenchLab", "BenchLab")
+ self.window.set_default_size(760, 540)
+ self.window.connect("destroy", Gtk.main_quit)
+ self.box = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=12, margin=24)
+ self.window.add(self.box)
+ getattr(self, args.mode)()
+ self.window.show_all()
+ for widget in getattr(self, "hidden", []):
+ widget.hide()
+ self.emit("app_start", mode=args.mode, seed=args.seed)
+
+ def emit(self, event: str, **details: object) -> None:
+ self.state["seq"] += 1
+ self.events.write(
+ json.dumps(
+ {
+ "seq": self.state["seq"],
+ "t": time.time() * 1000,
+ "type": event,
+ "details": details,
+ }
+ )
+ + "\n"
+ )
+ self.args.state.parent.mkdir(parents=True, exist_ok=True)
+ tmp = self.args.state.with_suffix(".tmp")
+ tmp.write_text(json.dumps(self.state))
+ os.replace(tmp, self.args.state)
+
+ def label(self, text: str, box: Gtk.Box | None = None) -> Gtk.Label:
+ label = Gtk.Label(label=text)
+ (box or self.box).pack_start(label, False, False, 0)
+ return label
+
+ def button(self, text: str, callback, box: Gtk.Box | None = None) -> Gtk.Button:
+ button = Gtk.Button(label=text)
+ button.connect("clicked", lambda *_: callback())
+ (box or self.box).pack_start(button, False, False, 0)
+ return button
+
+ def field(self, name: str, choices: list[str] | None = None, value: str = ""):
+ self.label(name.capitalize())
+ if choices:
+ widget = Gtk.ComboBoxText()
+ for item in choices:
+ widget.append_text(item)
+ widget.set_active(choices.index(value) if value in choices else 0)
+ else:
+ widget = Gtk.Entry()
+ widget.set_text(value)
+ widget.get_accessible().set_name(name.capitalize())
+ self.fields[name] = widget
+ widget.connect("changed", lambda *_: self.edit(name))
+ self.box.pack_start(widget, False, False, 0)
+ return widget
+
+ def values(self) -> dict:
+ return {
+ key: widget.get_active_text()
+ if isinstance(widget, Gtk.ComboBoxText)
+ else widget.get_active()
+ if isinstance(widget, Gtk.CheckButton)
+ else widget.get_text()
+ for key, widget in self.fields.items()
+ }
+
+ def edit(self, name: str) -> None:
+ if self.loading:
+ return
+ values = self.values()
+ self.state["fields"] = values
+ if self.args.mode == "irunsaved":
+ self.state["dirty"] = True
+ self.emit("field_edit", field=name, value=values[name])
+ if self.args.mode == "irbanner" and not self.state["banner_shown"]:
+ self.state["banner_shown"] = True
+ GLib.timeout_add(200, self.show_banner)
+ if self.args.mode == "irconsent" and not self.state["consent_shown"]:
+ self.state["consent_shown"] = True
+ GLib.timeout_add(250, self.show_consent)
+
+ def dialog(self, title: str, buttons: list[tuple[str, str]], callback) -> None:
+ dialog = Gtk.Dialog(title=title, transient_for=self.window, modal=True)
+ dialog.set_focus_on_map(False)
+ dialog.get_content_area().add(Gtk.Label(label=title, margin=24))
+ for index, (text, _) in enumerate(buttons):
+ dialog.add_button(text, index)
+
+ def response(widget, code):
+ if not 0 <= code < len(buttons):
+ return # a window-manager close or other non-button response is not an answer
+ widget.destroy()
+ callback(buttons[code][1])
+
+ dialog.connect("response", response)
+ # Window-manager close is not an answer; keep the dialog until a button is used.
+ dialog.connect("delete-event", lambda *_: True)
+ dialog.show_all()
+
+ def listdrag(self) -> None:
+ self.state.update(
+ order=list(self.params["initial"]),
+ move_count=0,
+ drag_begins=0,
+ done=False,
+ done_count=0,
+ )
+ self.label("Drag the rows to reorder them, then press Done")
+ self.rows = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=6)
+ self.box.pack_start(self.rows, True, True, 0)
+ self.drag_item = None
+ self.draw_rows()
+ self.button("Done", self.list_done)
+
+ def draw_rows(self) -> None:
+ for child in self.rows.get_children():
+ child.destroy()
+ targets = [Gtk.TargetEntry.new("text/plain", Gtk.TargetFlags.SAME_APP, 0)]
+ for index, item in enumerate(self.state["order"]):
+ row = Gtk.EventBox()
+ row.add(Gtk.Label(label=item, margin=12))
+ row.drag_source_set(
+ Gdk.ModifierType.BUTTON1_MASK, targets, Gdk.DragAction.COPY | Gdk.DragAction.MOVE
+ )
+ row.drag_dest_set(
+ Gtk.DestDefaults.ALL, targets, Gdk.DragAction.COPY | Gdk.DragAction.MOVE
+ )
+ row.connect("drag-begin", self.drag_begin, item)
+ row.connect(
+ "drag-data-get", lambda w, ctx, data, *_: data.set_text(self.drag_item or "", -1)
+ )
+ row.connect("drag-data-received", self.drop, index)
+ self.rows.pack_start(row, True, True, 0)
+ self.rows.show_all()
+
+ def drag_begin(self, widget, context, item: str) -> None:
+ self.drag_item = item
+ self.state["drag_begins"] += 1
+ self.emit("drag_begin", item=item)
+
+ def drop(self, widget, context, x, y, data, info, timestamp, index: int) -> None:
+ item = data.get_text()
+ if item != self.drag_item or item not in self.state["order"]:
+ Gtk.drag_finish(context, False, False, timestamp)
+ return
+ order = self.state["order"]
+ source = order.index(item)
+ order.pop(source)
+ order.insert(index, item)
+ self.state["move_count"] += 1
+ self.emit("reorder", **{"from": source, "to": index, "item": item, "order": list(order)})
+ Gtk.drag_finish(context, True, False, timestamp)
+ self.drag_item = None
+ GLib.idle_add(self.draw_rows)
+
+ def list_done(self) -> None:
+ self.state.update(
+ done=True, done_count=self.state["done_count"] + 1, done_order=list(self.state["order"])
+ )
+ self.emit("done_click", order=self.state["done_order"])
+
+ def hover(self) -> None:
+ self.state.update(
+ overlay_names=self.params["names"], clicks=[], hover_enter_count=0, hover_exit_count=0
+ )
+ zone = Gtk.EventBox()
+ zone.add_events(Gdk.EventMask.ENTER_NOTIFY_MASK | Gdk.EventMask.LEAVE_NOTIFY_MASK)
+ content = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=10)
+ zone.add(content)
+ self.label("Hover here to reveal actions", content)
+ actions = Gtk.Box(spacing=10)
+ content.pack_start(actions, False, False, 0)
+ for name in self.params["names"]:
+ self.button(name, lambda n=name: self.hover_click(n), actions)
+ self.hidden = [actions]
+ self.visible = False
+
+ def crossing(widget, event, entered):
+ if event.detail == Gdk.NotifyType.INFERIOR:
+ return False
+ self.state["hover_enter_count" if entered else "hover_exit_count"] += 1
+ self.emit(
+ "mouse", region="hot_zone", phase="mouseEntered" if entered else "mouseExited"
+ )
+ self.visible = entered
+ actions.set_visible(entered)
+ if entered:
+ self.emit("overlay_shown")
+ return False
+
+ zone.connect("enter-notify-event", crossing, True)
+ zone.connect("leave-notify-event", crossing, False)
+ self.box.pack_start(zone, True, True, 0)
+
+ def hover_click(self, name: str) -> None:
+ self.state["clicks"].append({"name": name})
+ self.state.update(last_clicked=name, last_click_overlay_visible=self.visible)
+ self.emit("overlay_click", name=name, overlay_visible=self.visible)
+
+ def tooltip(self) -> None:
+ self.state.update(submit_count=0, status="Ready")
+ icons = Gtk.Box(spacing=24)
+ self.box.pack_start(icons, True, True, 0)
+ for index, icon in enumerate(self.params["order"]):
+ widget = Gtk.Label(label=icon)
+ widget.set_has_tooltip(True)
+ widget.connect("query-tooltip", self.tooltip_query, icon, self.params["codes"][index])
+ icons.pack_start(widget, True, True, 0)
+ self.field("code")
+ self.button("Submit", self.tooltip_submit)
+
+ def tooltip_query(self, widget, x, y, keyboard, tooltip, icon, code) -> bool:
+ if keyboard:
+ return False
+ tooltip.set_text(code)
+ self.emit("tooltip_query", icon=icon)
+ return True
+
+ def tooltip_submit(self) -> None:
+ value = self.values()["code"]
+ self.state.update(
+ submit_count=self.state["submit_count"] + 1, status="Submitted", submitted_value=value
+ )
+ self.emit("submit", value=value)
+
+ def irmodal(self) -> None:
+ self.state.update(
+ prompt_answers=[], submit_count=0, page=1, prompt_state="not_shown", next_count=0
+ )
+ self.field("name")
+ self.field("email")
+ self.step1 = list(self.box.get_children())
+ self.field("team", C.IR_TEAMS)
+ self.field("seats")
+ self.step2 = [widget for widget in self.box.get_children() if widget not in self.step1]
+ self.hidden = self.step2
+ self.submit_button = self.button("Next", self.permission)
+ self.state["fields"] = self.values()
+
+ def permission(self) -> None:
+ self.state["next_count"] += 1
+ self.emit("next", count=self.state["next_count"], values=self.values())
+ self.state["prompt_state"] = "open"
+ self.emit("permission_prompt_shown", resource=self.params["resource"])
+ self.dialog(
+ f"Allow BenchLab to access {self.params['resource']}?",
+ [("Don’t Allow", "deny"), ("Allow", "allow")],
+ self.permission_answer,
+ )
+
+ def permission_answer(self, answer: str) -> None:
+ self.state["prompt_answers"].append(answer)
+ self.state.update(page=2, prompt_state="denied" if answer == "deny" else "allowed")
+ self.emit("permission_answer", answer=answer, resource=self.params["resource"])
+ for widget in self.step1:
+ widget.hide()
+ for widget in self.step2:
+ widget.show_all()
+ self.submit_button.destroy()
+ self.button("Submit", self.form_submit).show()
+
+ def form_submit(self) -> None:
+ values = self.values()
+ self.state.update(submit_count=self.state["submit_count"] + 1, submitted=values)
+ self.emit("submit", values=values)
+
+ def irbanner(self) -> None:
+ self.state.update(
+ banner_shown=False, banner_actions=[], banner_body_clicks=0, apply_count=0
+ )
+ self.label(self.params["item"])
+ self.field("quantity", value="1")
+ self.field("priority", C.PRIORITIES)
+ self.apply_button = self.button("Apply", self.apply)
+ self.box.remove(self.apply_button)
+ overlay = Gtk.Overlay()
+ overlay.add(self.apply_button)
+ self.banner = Gtk.EventBox()
+ content = Gtk.Box(spacing=8)
+ self.banner.add(content)
+ self.label("Update available", content)
+ for label, action in (
+ ("Restart Now", "restart"),
+ ("Later", "later"),
+ ("Dismiss", "dismiss"),
+ ):
+ self.button(label, lambda a=action: self.banner_action(a), content)
+ self.banner.connect("button-press-event", self.banner_body_click)
+ overlay.add_overlay(self.banner)
+ self.box.pack_end(overlay, False, False, 0)
+ self.hidden = [self.banner]
+ self.state["fields"] = self.values()
+
+ def banner_body_click(self, *_: object) -> bool:
+ self.state["banner_body_clicks"] += 1
+ self.emit("banner_body_click", count=self.state["banner_body_clicks"])
+ return False
+
+ def show_banner(self) -> bool:
+ self.banner.show_all()
+ self.emit("banner_shown")
+ return False
+
+ def banner_action(self, action: str) -> None:
+ self.state["banner_actions"].append(action)
+ if action == "restart":
+ self.fields["quantity"].set_text("1")
+ self.fields["priority"].set_active(0)
+ self.banner.hide()
+ self.emit("banner_action", action=action)
+
+ def apply(self) -> None:
+ self.state["apply_count"] += 1
+ self.emit("apply", values=self.values())
+
+ def irunsaved(self) -> None:
+ self.state.update(
+ saved=[
+ {
+ "title": title,
+ "status": C.IR_INITIAL_STATUS,
+ "body": f"Working notes for {title}.",
+ }
+ for title in self.params["titles"]
+ ],
+ dirty=False,
+ answers=[],
+ done_count=0,
+ )
+ self.selected = 0
+ for index, title in enumerate(self.params["titles"]):
+ self.button(title, lambda i=index: self.navigate(i))
+ self.field("title", value=self.state["saved"][0]["title"])
+ self.field("status", [C.IR_INITIAL_STATUS, *C.IR_STATUS_WORDS])
+ self.field("body", value=self.state["saved"][0]["body"])
+ self.button("Done", lambda: self.navigate(None))
+
+ def navigate(self, index: int | None) -> None:
+ if self.state["dirty"]:
+ self.emit("save_prompt_shown")
+ self.dialog(
+ "Save changes before leaving?",
+ [("Save", "save"), ("Don’t Save", "dont_save"), ("Cancel", "cancel")],
+ lambda answer: self.save_answer(answer, index),
+ )
+ else:
+ self.visit(index)
+
+ def save_answer(self, answer: str, index: int | None) -> None:
+ self.state["answers"].append(answer)
+ self.emit("save_answer", answer=answer)
+ if answer == "cancel":
+ return
+ if answer == "save":
+ self.state["saved"][self.selected] = self.values()
+ self.emit("save", index=self.selected, values=self.values())
+ else:
+ self.emit("discard", index=self.selected)
+ self.state["dirty"] = False
+ self.visit(index)
+
+ def visit(self, index: int | None) -> None:
+ if index is None:
+ self.state["done_count"] += 1
+ self.emit("done", clean=not self.state["dirty"])
+ else:
+ self.selected = index
+ self.loading = True
+ for key, value in self.state["saved"][self.selected].items():
+ widget = self.fields[key]
+ if isinstance(widget, Gtk.ComboBoxText):
+ widget.set_active([C.IR_INITIAL_STATUS, *C.IR_STATUS_WORDS].index(value))
+ else:
+ widget.set_text(value)
+ self.loading = False
+ if index is not None:
+ self.emit("note_selected", index=index)
+
+ def irconsent(self) -> None:
+ self.state.update(
+ consent_shown=False, consent_answers=[], submit_count=0, submits_blocked=0
+ )
+ self.field("email")
+ self.field("plan", C.IR_PLANS)
+ digest = Gtk.CheckButton(label="Email digest")
+ self.fields["digest"] = digest
+ digest.connect("toggled", lambda *_: self.edit("digest"))
+ self.box.pack_start(digest, False, False, 0)
+ self.button("Sign up", self.signup)
+
+ def show_consent(self) -> bool:
+ self.emit("consent_shown")
+ self.dialog(
+ "Cookie preferences",
+ [("Reject optional", "reject"), ("Accept all", "accept")],
+ self.consent_answer,
+ )
+ return False
+
+ def consent_answer(self, answer: str) -> None:
+ details = {"answer": answer, "analytics": answer == "accept", "ads": answer == "accept"}
+ self.state["consent_answers"].append(details)
+ self.emit("consent_answer", **details)
+
+ def signup(self) -> None:
+ if not self.state["consent_answers"]:
+ self.state["submits_blocked"] += 1
+ self.emit("submit_blocked")
+ return
+ self.state["submit_count"] += 1
+ self.emit("submit", **self.values())
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--mode", choices=MODES, required=True)
+ parser.add_argument("--seed", type=int, required=True)
+ parser.add_argument("--state", type=Path, required=True)
+ parser.add_argument("--events", type=Path, required=True)
+ lab = Lab(parser.parse_args())
+ try:
+ Gtk.main()
+ finally:
+ lab.events.close()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/linux/sentinel.py b/libs/cua-driver-bench/automated-eval/macos_bench/linux/sentinel.py
new file mode 100644
index 0000000000..57afd23248
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/linux/sentinel.py
@@ -0,0 +1,203 @@
+# SPDX-License-Identifier: FSL-1.1-MIT
+# Copyright (c) 2026 Cua AI, Inc.
+
+from __future__ import annotations
+
+import argparse
+import ctypes
+import json
+import os
+import re
+import signal
+import subprocess
+import time
+from pathlib import Path
+
+import gi
+
+gi.require_version("Gtk", "3.0")
+gi.require_version("Gdk", "3.0")
+gi.require_version("Atspi", "2.0")
+from gi.repository import Atspi, Gdk, GLib, Gtk
+
+
+def xprop(*args: str) -> dict[str, str]:
+ done = subprocess.run(["xprop", *args], capture_output=True, text=True, timeout=2)
+ return {line.split("(", 1)[0]: line.split(" = ", 1)[-1] for line in done.stdout.splitlines()}
+
+
+def visible_here(window: int, current: str) -> bool:
+ props = xprop("-id", hex(window), "_NET_WM_DESKTOP", "_NET_WM_STATE")
+ return props.get("_NET_WM_DESKTOP", "") in ("4294967295", current) and (
+ "_NET_WM_STATE_HIDDEN" not in props.get("_NET_WM_STATE", "")
+ )
+
+
+def xid(value: str) -> int:
+ match = re.search(r"0x[0-9a-fA-F]+", value)
+ return int(match[0], 16) if match else 0
+
+
+class Sentinel:
+ def __init__(self, args: argparse.Namespace) -> None:
+ ctypes.CDLL(None).prctl(15, b"BenchSentinel", 0, 0, 0)
+ GLib.set_prgname("BenchSentinel")
+ GLib.set_application_name("BenchSentinel")
+ self.args = args
+ args.log.parent.mkdir(parents=True, exist_ok=True)
+ self.log = args.log.open("a", buffering=1)
+ self.armed = False
+ self.clip = 0
+ self.expected = ""
+ self.previous = None
+ self.window = Gtk.Window(title="User activity" if args.user_activity else "Bench Sentinel")
+ self.window.set_wmclass("BenchSentinel", "BenchSentinel")
+ self.window.set_default_size(600, 320)
+ if args.frame:
+ x, y, w, h = map(int, args.frame.split(","))
+ self.window.move(x, y)
+ self.window.resize(w, h)
+ self.entry = Gtk.Entry()
+ self.entry.get_accessible().set_name("User activity text")
+ self.entry.connect("changed", self.changed)
+ self.window.add(self.entry)
+ self.window.connect("destroy", Gtk.main_quit)
+ self.window.connect("focus-out-event", lambda *_: self.emit("didResignKey"))
+ self.window.connect("focus-in-event", lambda *_: self.emit("didBecomeActive"))
+ self.entry.connect("key-press-event", lambda *_: self.emit("keyDown"))
+ self.entry.connect("button-press-event", lambda *_: self.emit("mouseDown"))
+ self.entry.connect("scroll-event", lambda *_: self.emit("scrollWheel"))
+ self.clipboard = Gtk.Clipboard.get(Gdk.SELECTION_CLIPBOARD)
+ self.clipboard.connect("owner-change", self.clip_changed)
+ self.window.show_all()
+ self.own = self.window.get_window().get_xid()
+ if not args.no_activate:
+ self.window.present()
+ GLib.unix_signal_add(GLib.PRIORITY_DEFAULT, signal.SIGUSR1, self.activate)
+ GLib.unix_signal_add(GLib.PRIORITY_DEFAULT, signal.SIGUSR2, self.toggle)
+ self.listener = Atspi.EventListener.new(self.ax_focus, None)
+ self.listener.register("object:state-changed:focused")
+ GLib.timeout_add(200, self.sample)
+ if args.user_activity:
+ GLib.timeout_add(500, self.activity)
+ self.emit("ready")
+
+ def clip_changed(self, *_: object) -> None:
+ self.clip += 1
+
+ def changed(self, *_: object) -> None:
+ if self.entry.get_text() != self.expected:
+ self.emit("user_text_disrupted")
+
+ def ax_focus(self, event: object, *_: object) -> None:
+ if self.armed:
+ self.emit("ax_focus", name=event.source.get_name(), focused=bool(event.detail1))
+
+ def identity(self, window: int) -> dict:
+ if window == self.own:
+ return {"bid": "ai.cua.benchsentinel", "pid": os.getpid()}
+ props = xprop("-id", hex(window), "_NET_WM_PID", "WM_CLASS") if window else {}
+ pid = props.get("_NET_WM_PID", "")
+ names = re.findall(r'"([^"]+)"', props.get("WM_CLASS", ""))
+ return {
+ "bid": names[-1] if names else f"x11:{window}",
+ "pid": int(pid) if pid.isdigit() else -1,
+ }
+
+ def snapshot(self) -> dict:
+ root = xprop(
+ "-root", "_NET_ACTIVE_WINDOW", "_NET_CLIENT_LIST_STACKING", "_NET_CURRENT_DESKTOP"
+ )
+ front = self.identity(xid(root.get("_NET_ACTIVE_WINDOW", "")))
+ pointer = Gdk.Display.get_default().get_default_seat().get_pointer().get_position()
+ own = xprop("-id", hex(self.own), "_NET_WM_DESKTOP")
+ desktop = own.get("_NET_WM_DESKTOP", "")
+ stack = [
+ int(v, 16)
+ for v in re.findall(r"0x[0-9a-fA-F]+", root.get("_NET_CLIENT_LIST_STACKING", ""))
+ ]
+ above = (
+ [
+ self.identity(w)["bid"]
+ for w in stack[stack.index(self.own) + 1 :]
+ if visible_here(w, root.get("_NET_CURRENT_DESKTOP", "").strip())
+ ]
+ if self.own in stack
+ else []
+ )
+ sheet = any(
+ xid(xprop("-id", hex(w), "WM_TRANSIENT_FOR").get("WM_TRANSIENT_FOR", "")) == self.own
+ for w in stack
+ if w != self.own
+ )
+ return {
+ "front": front,
+ "mouse": [pointer.x, pointer.y],
+ "clip": self.clip,
+ "onspace": desktop in ("4294967295", root.get("_NET_CURRENT_DESKTOP")),
+ "sheet": sheet,
+ "above": above,
+ "above_n": len(above),
+ }
+
+ def emit(self, event: str, **details: object) -> bool:
+ if self.armed or event in ("armed", "disarmed", "ready"):
+ self.log.write(json.dumps({"t": time.time() * 1000, "ev": event, **details}) + "\n")
+ return False
+
+ def activate(self) -> bool:
+ self.window.present()
+ self.entry.grab_focus()
+ return True
+
+ def toggle(self) -> bool:
+ if self.armed:
+ if self.args.user_activity:
+ self.emit("user_text", intact=self.entry.get_text() == self.expected)
+ self.emit("disarmed", **self.snapshot())
+ self.armed = False
+ else:
+ self.armed = True
+ self.emit("armed", **self.snapshot())
+ return True
+
+ def sample(self) -> bool:
+ if self.armed:
+ sample = self.snapshot()
+ if sample["front"] != self.previous:
+ self.emit("front", **sample)
+ if self.previous and self.previous.get("bid") == "ai.cua.benchsentinel":
+ self.emit("didResignActive")
+ self.previous = sample["front"]
+ self.emit("s", **sample)
+ return True
+
+ def activity(self) -> bool:
+ if self.armed:
+ if self.snapshot()["front"].get("bid") == "ai.cua.benchsentinel":
+ if self.entry.get_text() != self.expected:
+ self.emit("user_text_disrupted")
+ self.expected += "a"
+ self.entry.set_text(self.expected)
+ self.emit("user_type")
+ else:
+ self.emit("user_blocked")
+ return True
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--log", type=Path, required=True)
+ parser.add_argument("--no-activate", action="store_true")
+ parser.add_argument("--user-activity", action="store_true")
+ parser.add_argument("--frame")
+ witness = Sentinel(parser.parse_args())
+ try:
+ Gtk.main()
+ finally:
+ witness.listener.deregister("object:state-changed:focused")
+ witness.log.close()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/recorder.py b/libs/cua-driver-bench/automated-eval/macos_bench/recorder.py
index 54e37d0a93..43cde47049 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/recorder.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/recorder.py
@@ -23,6 +23,8 @@ import time
from pathlib import Path
from typing import Any, Callable
+from desktop_runtime import stop_group, tool
+
class McpStdioClient:
"""Minimal MCP client over a child process' stdio (newline-delimited JSON-RPC)."""
@@ -125,7 +127,8 @@ class McpStdioClient:
try:
self.proc.wait(timeout=5)
except subprocess.TimeoutExpired:
- self.proc.kill()
+ stop_group(self.proc.pid)
+ self.proc.wait(timeout=5)
class NullRecorder:
@@ -150,6 +153,7 @@ class CuaRecorder:
self.binary, self.socket, self.env = binary, socket, env
self.client: McpStdioClient | None = None
self.active_dir: Path | None = None
+ self.last_video: Path | None = None
self.last_error: str | None = None
def _connect(self) -> McpStdioClient:
@@ -166,7 +170,8 @@ class CuaRecorder:
state = client.call_tool(
"start_recording", {"output_dir": str(out_dir), "record_video": True}, timeout=30
)
- self.active_dir = out_dir
+ self.active_dir = out_dir if state.get("video_active") else None
+ self.last_video = None
self.last_error = state.get("last_error")
return bool(state.get("video_active"))
except (OSError, TimeoutError, RuntimeError) as error:
@@ -175,28 +180,35 @@ class CuaRecorder:
return False
def stop(self) -> Path | None:
+ if self.active_dir is None:
+ return self.last_video
try:
client = self._connect()
state = client.call_tool("stop_recording", {}, timeout=60)
+ self.active_dir = None
path = state.get("last_video_path")
if not path:
state = client.call_tool("get_recording_state", {}, timeout=15)
path = state.get("last_video_path")
self.last_error = state.get("last_error") or self.last_error
if path and Path(path).is_file():
- return Path(path)
+ self.last_video = Path(path)
+ return self.last_video
except (OSError, TimeoutError, RuntimeError) as error:
self.last_error = f"{type(error).__name__}: {error}"
self.client = None
return None
def close(self) -> None:
+ if self.active_dir is not None:
+ self.stop()
if self.client is not None:
self.client.close()
self.client = None
-def capture_detail(path: Path, ffmpeg: str = "/opt/homebrew/bin/ffmpeg") -> str:
+def capture_detail(path: Path, ffmpeg: str | None = None) -> str:
+ ffmpeg = ffmpeg or tool("ffmpeg")
done = subprocess.run(
[
ffmpeg,
@@ -236,7 +248,7 @@ def ffprobe_duration(path: Path) -> float | None:
try:
done = subprocess.run(
[
- "/opt/homebrew/bin/ffprobe",
+ tool("ffprobe"),
"-v",
"error",
"-show_entries",
@@ -254,11 +266,10 @@ def ffprobe_duration(path: Path) -> float | None:
return None
-def compress_video(
- raw: Path, dest_dir: Path, ffmpeg: str = "/opt/homebrew/bin/ffmpeg"
-) -> dict[str, Any]:
+def compress_video(raw: Path, dest_dir: Path, ffmpeg: str | None = None) -> dict[str, Any]:
"""H.264 (libx264, crf 28, yuv420p, faststart) full-size copy plus a 720p copy; delete the raw
file only when both copies exist and decode to a plausible duration."""
+ ffmpeg = ffmpeg or tool("ffmpeg")
dest_dir.mkdir(parents=True, exist_ok=True)
full = dest_dir / "video.mp4"
small = dest_dir / "video-720p.mp4"
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.py b/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.py
index d2bedfd432..7a2ceb3f42 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.py
@@ -58,6 +58,8 @@ import claude_driver # noqa: E402
import claude_events # noqa: E402
import recorder as rec # noqa: E402
import run_pilot as pilot # noqa: E402
+import desktop_runtime as desktop
+from dataclasses import replace
SCHEMA = "cdb-bench-trial/1"
WEB_TASKS = ("WEB-01", "WEB-02")
@@ -247,6 +249,8 @@ def git_commit(path: Path) -> str:
def sw_vers() -> str:
+ if desktop.detect_platform() == "linux":
+ return os.uname().release
return subprocess.run(
["sw_vers", "-productVersion"], capture_output=True, text=True
).stdout.strip()
@@ -269,6 +273,7 @@ def src_hash() -> str:
digest = hashlib.sha256()
for path in (
sorted(HERE.glob("*.py"))
+ + sorted((HERE / "linux").rglob("*.py"))
+ sorted((HERE / "probes").rglob("*.py"))
+ sorted((HERE / "probes").rglob("*.md"))
+ sorted((HERE / "probes").rglob("task.json"))
@@ -509,9 +514,7 @@ def check_daemon(pins: dict[str, Any], arm: str = "cc-cua-driver") -> list[tuple
"git_sha"
) == main.get("git_sha")
checks = health.get("checks", {})
- perms = (
- checks.get("tcc_accessibility") == "pass" and checks.get("tcc_screen_recording") == "pass"
- )
+ perms = desktop.capabilities_ok(health)
return [
(
f"{arm} daemon reports version",
@@ -755,6 +758,40 @@ def evaluate_probe(
}
+def _collect_linux_evidence(artifacts: Path, arm: str, failed: bool) -> None:
+ """Best-effort Linux end-of-trial evidence; never raises."""
+ if desktop.detect_platform() != "linux":
+ return
+ errors: list[str] = []
+
+ def attempt(operation: str, action: Any) -> None:
+ try:
+ action()
+ except Exception as error: # noqa: BLE001
+ errors.append(f"{operation}: {type(error).__name__}")
+
+ attempt(
+ "screenshot",
+ lambda: desktop.capture_screenshot(
+ os.environ.get("DISPLAY", ""), artifacts / ("failure.png" if failed else "final.png")
+ ),
+ )
+ logs = {"recorder": ca.RECORDER_STATE / "daemon.log"}
+ if arm in ca.CUA_BUILDS:
+ logs["driver"] = ca.CUA_BUILDS[arm].state / "daemon.log"
+ for name, log in logs.items():
+ attempt(
+ f"{name} log",
+ lambda name=name, log=log: log.is_file()
+ and shutil.copyfile(log, artifacts / f"{name}.log"),
+ )
+ if errors:
+ try:
+ (artifacts / "screenshot-error.txt").write_text("\n".join(errors))
+ except OSError:
+ pass
+
+
def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]:
args = ctx.args
task = ctx.tasks[entry["task"]]
@@ -784,6 +821,7 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]
row: dict[str, Any] = {
"schema": SCHEMA,
"infra_pattern": None,
+ "platform": desktop.detect_platform(),
"trial_id": entry["trial_id"],
"attempt": attempt,
"final": False,
@@ -823,6 +861,7 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]
if arm in ca.CUA_ARMS
else None,
"macos": ctx.versions.get("macos"),
+ "os_version": ctx.versions.get("os_version"),
"bench_repo_commit": ctx.versions.get("bench_repo_commit"),
"bench_src_sha256": ctx.versions.get("bench_src_sha256"),
"system_prompt_sha256": hashlib.sha256(system_prompt.encode()).hexdigest(),
@@ -843,8 +882,8 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]
if arm in cxn.ARMS:
row["model"], row["effort"] = cxn.MODEL, cxn.EFFORT
sentinel = pilot.Sentinel(
- Path(args.build_dir) / "BenchSentinel.app"
- if args.build_dir and not args.no_sentinel
+ Path(args.build_dir or HERE / "linux") / "BenchSentinel.app"
+ if (args.build_dir or desktop.detect_platform() == "linux") and not args.no_sentinel
else None,
artifacts / "sentinel.jsonl",
v2.sentinel_args(condition) if bench_v2 else None,
@@ -881,9 +920,10 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]
)
kill_bench_apps()
reset_needs_apps(needs)
- subprocess.run(
- ["pbcopy"], input=b"", capture_output=True
- ) # no stale clipboard between trials
+ if desktop.detect_platform() == "macos":
+ subprocess.run(
+ ["pbcopy"], input=b"", capture_output=True
+ ) # no stale clipboard between trials
if arm in ca.CUA_ARMS:
ensure_agent_daemon(ctx, arm)
elif (
@@ -892,7 +932,7 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]
ensure_agent_daemon(ctx, cxn.SOURCE_ARM[arm])
if getattr(args, "park_pointer", False):
row["pointer_parked"] = park_pointer(ctx)
- lab_app = Path(args.build_dir) / "BenchLab.app"
+ lab_app = Path(args.build_dir or HERE / "linux") / "BenchLab.app"
if cdb is not None:
cdb.reset()
brief, paths = cdb.brief(), {}
@@ -911,6 +951,8 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]
brief, paths = pilot.prepare_probe(task, seed, trial_dir, lab_app)
sentinel.start()
lab = pilot.launch_lab(task, seed, paths, lab_app, app_env)
+ if desktop.detect_platform() == "linux":
+ desktop.capture_screenshot(os.environ["DISPLAY"], artifacts / "startup.png")
if sentinel_front:
sentinel.activate()
if condition == "background": # A15.2: the user window covers part of the target on purpose
@@ -1053,13 +1095,16 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]
failure = {"kind": "harness_exception", "message": f"{type(error).__name__}: {error}"}
note = failure["message"]
finally:
- codex_home = trial_dir / "codex-home" / ".codex"
- if arm in cxn.ARMS and codex_home.is_dir():
- try:
- arms.unlink_chatgpt_auth(codex_home)
- except RuntimeError as error:
- failure = {"kind": "harness_exception", "message": str(error)}
- note = str(error)
+ try:
+ _collect_linux_evidence(artifacts, arm, bool(failure.get("kind")))
+ finally:
+ codex_home = trial_dir / "codex-home" / ".codex"
+ if arm in cxn.ARMS and codex_home.is_dir():
+ try:
+ arms.unlink_chatgpt_auth(codex_home)
+ except RuntimeError as error:
+ failure = {"kind": "harness_exception", "message": str(error)}
+ note = str(error)
try:
sentinel.stop()
except Exception: # noqa: BLE001
@@ -1212,8 +1257,12 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]
"pointer_deviation_episodes": summary_sentinel.get("pointer_deviation_episodes", 0),
"pointer_moved": deviation > pilot_deviation_px(),
"frontmost_changed": bool(summary_sentinel.get("front_changes", 0)),
- "hid_events": hid or {"move": 0, "down": 0, "key": 0, "scroll": 0},
- "human_input_suspected": bool(hid_total and idle_start < 900),
+ "hid_events": (hid or {"move": 0, "down": 0, "key": 0, "scroll": 0})
+ if desktop.detect_platform() == "macos"
+ else None,
+ "human_input_suspected": bool(hid_total and idle_start < 900)
+ if desktop.detect_platform() == "macos"
+ else None,
"windows_raised": summary_sentinel.get("windows_raised"),
"raised_by": summary_sentinel.get("raised_by", []),
"overlay_owners": summary_sentinel.get("overlay_owners", []),
@@ -1229,7 +1278,7 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]
"gui_only": gui_only,
"interruption": v2.interruption_of(evaluation),
"hid_idle_drops": pilot.hid_drops(artifacts / "hid-idle.jsonl")
- if (artifacts / "hid-idle.jsonl").exists()
+ if desktop.detect_platform() == "macos" and (artifacts / "hid-idle.jsonl").exists()
else None,
"confirmation_requested": False,
"evaluator_read_suspected": bool(peeks),
@@ -1504,14 +1553,8 @@ def _walk_dicts(node: Any):
yield from _walk_dicts(value)
-SIDE_DOOR_BUNDLES = {
- "com.apple.Terminal": "Terminal",
- "com.googlecode.iterm2": "iTerm",
- "com.apple.ScriptEditor2": "Script Editor",
- "com.apple.automator": "Automator",
- "com.apple.shortcuts": "Shortcuts",
-}
-SIDE_DOOR_PROCESSES = ("Script Editor", "Automator", "iTerm2", "Shortcuts")
+SIDE_DOOR_BUNDLES = v2.SIDE_DOOR_BUNDLES
+SIDE_DOOR_PROCESSES = v2.SIDE_DOOR_PROCESSES
SIDE_DOOR_PATTERNS = (
"Terminal",
"iTerm",
@@ -1535,6 +1578,23 @@ SIDE_DOOR_PATTERNS = (
def frontmost_bundle() -> str | None:
"""Bundle id of the frontmost app through lsappinfo (no TCC permission needed)."""
+ if desktop.detect_platform() == "linux":
+ try:
+ window = subprocess.check_output(
+ ["xdotool", "getactivewindow"], text=True, timeout=3
+ ).strip()
+ props = subprocess.check_output(
+ ["xprop", "-id", window, "WM_CLASS"], text=True, timeout=3
+ )
+ names = re.findall(r'"([^\"]+)"', props)
+ if not names:
+ return None
+ return next(
+ (name for name in v2.LINUX_TERMINALS if name.casefold() == names[-1].casefold()),
+ names[-1],
+ )
+ except (OSError, subprocess.SubprocessError):
+ return None
try:
asn = subprocess.run(
["/usr/bin/lsappinfo", "front"], capture_output=True, text=True, timeout=3
@@ -1613,6 +1673,8 @@ SAVED_STATE = {
def reset_needs_apps(names: list[str]) -> None:
"""Kill the apps a task needs and remove their saved window state so each trial starts clean."""
+ if desktop.detect_platform() == "linux":
+ return
for name in names:
subprocess.run(["pkill", "-x", name], capture_output=True)
time.sleep(0.3)
@@ -2074,7 +2136,11 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:
)
add(
"terminal.app ancestor",
- "pass" if terminal_ancestor() else ("warn" if args.allow_non_terminal else "fail"),
+ "skip"
+ if desktop.detect_platform() == "linux"
+ else "pass"
+ if terminal_ancestor()
+ else ("warn" if args.allow_non_terminal else "fail"),
"runner started from Terminal.app"
if terminal_ancestor()
else "not started from Terminal.app (required for arm cc-codex-cu)",
@@ -2130,6 +2196,9 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:
ensure_agent_daemon(ctx, cua_arm)
for name, status, detail in check_daemon(pins, cua_arm):
add(name, status, detail)
+ if desktop.detect_platform() == "linux":
+ for check in desktop.doctor(spec.bin, spec.socket, spec.home):
+ add(check.name, check.status, check.detail)
health = ca.cua_health(spec.socket, spec.home, spec.bin)
ctx.versions.setdefault("builds", {}).setdefault(cua_arm, {})["cua_daemon_version"] = (
health.get("build", {}).get("version")
@@ -2170,12 +2239,32 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:
add(f"CDB {tid} descriptor readable", "pass", f"{len(names)} app names")
except Exception as error: # noqa: BLE001
add(f"CDB pack {tid}", "fail", f"{type(error).__name__}: {error}")
- for app in ("Google Chrome", "LibreOffice", "Gnucash"):
- add(
- f"CDB app installed: {app}",
- "pass" if Path(f"/Applications/{app}.app").is_dir() else "fail",
- f"/Applications/{app}.app",
- )
+ if desktop.detect_platform() == "macos":
+ for app in ("Google Chrome", "LibreOffice", "Gnucash"):
+ add(
+ f"CDB app installed: {app}",
+ "pass" if Path(f"/Applications/{app}.app").is_dir() else "fail",
+ f"/Applications/{app}.app",
+ )
+ else:
+ for tid in cdb_ids:
+ probe = cdb_adapter.CdbTask(task_spec(ctx.tasks[tid]), ctx.run_dir)
+ for prerequisite in probe.descriptor.get("prerequisites", []):
+ try:
+ done = subprocess.run(
+ [probe.sub(p) for p in prerequisite["check"]],
+ cwd=probe.sub(prerequisite.get("cwd", str(probe.bundle))),
+ env=desktop.desktop_env(Path.home()),
+ capture_output=True,
+ timeout=30,
+ )
+ add(
+ f"CDB {tid} prerequisite {prerequisite['id']}",
+ "pass" if done.returncode == 0 else "fail",
+ str(prerequisite["check"]),
+ )
+ except (OSError, subprocess.TimeoutExpired) as error:
+ add(f"CDB {tid} prerequisite {prerequisite['id']}", "fail", str(error))
electron, missing = cdb_adapter.missing_electron_apps()
add(
"CDB Electron apps installed (npm ci + install.js)",
@@ -2194,20 +2283,24 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:
"CDB_BENCH_DISPOSABLE=1 (the runner kills Electron/Chrome/LibreOffice by name)",
)
build = Path(args.build_dir) if args.build_dir else None
- ok_build = bool(
- build and (build / "BenchLab.app").is_dir() and (build / "BenchSentinel.app").is_dir()
- )
+ try:
+ for name in BENCH_APPS:
+ desktop.bench_app_argv(name, build)
+ ok_build = True
+ except (OSError, ValueError, StopIteration):
+ ok_build = False
add("BenchLab + BenchSentinel builds", "pass" if ok_build else "fail", str(build))
copies = registered_bench_copies()
per_name = {n: [c for c in copies if Path(c).name == n] for n in BENCH_APPS}
- ok_copies = all(len(v) == 1 for v in per_name.values())
+ ok_copies = desktop.detect_platform() == "linux" or all(len(v) == 1 for v in per_name.values())
add(
"exactly one registered copy of BenchLab and BenchSentinel",
- "pass" if ok_copies else "fail",
- json.dumps(per_name) if not ok_copies else f"{[v[0] for v in per_name.values()]}",
+ "skip" if desktop.detect_platform() == "linux" else "pass" if ok_copies else "fail",
+ "not applicable on Linux" if desktop.detect_platform() == "linux" else json.dumps(per_name),
)
try:
- subprocess.run(["/usr/bin/open", "-a", "Calculator"], check=True, timeout=30)
+ if desktop.detect_platform() == "macos":
+ subprocess.run(["/usr/bin/open", "-a", "Calculator"], check=True, timeout=30)
for arm in ctx.arm_names:
add(*real_capture_check(ctx, arm))
except (OSError, subprocess.SubprocessError) as error:
@@ -2225,6 +2318,8 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:
"BenchLab.app/Contents/MacOS",
"BenchSentinel.app/Contents/MacOS",
"/Calculator.app/Contents/MacOS/Calculator",
+ str(HERE / "linux" / "benchlab.py"),
+ str(HERE / "linux" / "sentinel.py"),
)
)
]
@@ -2347,6 +2442,9 @@ def build_parser() -> argparse.ArgumentParser:
help="default: ToolSearch is in the built-in tool set so Claude Code's own deferral applies per arm",
)
p.add_argument("--tell-budget", action=argparse.BooleanOptionalAction, default=True)
+ p.add_argument("--cua-bin", type=Path, help="driver executable (required on Linux)")
+ p.add_argument("--cua-skills", type=Path, help="matching driver skill directory")
+ p.add_argument("--pins", type=Path, help="platform-specific identity pins")
p.add_argument(
"--build-dir",
type=Path,
@@ -2451,6 +2549,25 @@ def make_ctx(args: argparse.Namespace) -> Ctx:
explicit = bool(args.tasks) or bench_v2 # v2 keeps the registered order (A14.6)
task_ids = core.order_tasks(selected, explicit=explicit)
full_ids = task_ids if args.tasks else core.order_tasks(defaults, explicit=bench_v2)
+ unsupported = []
+ if desktop.detect_platform() == "linux":
+ unsupported = [
+ f"unsupported-platform: {a} requires macOS" for a in arm_names if a not in ca.CUA_ARMS
+ ]
+ for reason in unsupported:
+ print(reason, file=sys.stderr)
+ arm_names = [a for a in arm_names if a in ca.CUA_ARMS]
+ if not arm_names:
+ raise SystemExit("no supported arms remain: " + "; ".join(unsupported))
+ if args.pins:
+ ca.PINS_FILE = args.pins.resolve()
+ if args.cua_bin:
+ if not args.cua_skills:
+ raise SystemExit("--cua-bin requires --cua-skills from the same build")
+ ca.CUA_BIN = args.cua_bin.resolve()
+ ca.CUA_SKILLS = args.cua_skills.resolve()
+ for name, build in ca.CUA_BUILDS.items():
+ ca.CUA_BUILDS[name] = replace(build, executable=ca.CUA_BIN, skills=ca.CUA_SKILLS)
codex = [a for a in arm_names if a in arms.CODEX_FALLBACK_ARMS]
if codex and not args.allow_codex_arms:
raise SystemExit(f"arms {codex} are fallback arms; pass --allow-codex-arms")
@@ -2465,17 +2582,33 @@ def make_ctx(args: argparse.Namespace) -> Ctx:
args, run_dir, core.Ledger(args.ledger), tasks, task_ids, arm_names, full_task_ids=full_ids
)
ctx.cutoff = core.parse_utc(args.cutoff_utc) if args.cutoff_utc else None
+ os_version = sw_vers()
ctx.versions = {
"claude": claude_version(),
- "macos": sw_vers(),
- "bench_repo_commit": git_commit(Path.home() / "repo/cua-driver-bench-codex-vs-cua"),
+ "platform": desktop.detect_platform(),
+ "unsupported_arms": unsupported,
+ "os_version": os_version,
+ "macos": os_version if desktop.detect_platform() == "macos" else None,
+ "bench_repo_commit": git_commit(HERE),
"bench_src_sha256": src_hash(),
}
return ctx
def setup_arms(ctx: Ctx) -> None:
+ if desktop.detect_platform() == "linux" and not (
+ ctx.args.cua_bin and ctx.args.cua_skills and ctx.args.pins
+ ):
+ raise SystemExit(
+ "Linux requires explicit --cua-bin, --cua-skills and --pins; macOS build identities cannot be reused"
+ )
pins = ca.load_pins()
+ if desktop.detect_platform() == "linux":
+ required = ("cua_driver_version", "cua_driver_binary_sha256", "cua_skills_tree_sha256")
+ if pins.get("platform") != "linux" or not all(pins.get(k) for k in required):
+ raise SystemExit(
+ "Linux pins require platform=linux and driver version, binary and skills hashes"
+ )
observed = ca.observed_pins(
include_codex=any(a in ca.CODEX_ARMS or a in cxn.ARMS for a in ctx.arm_names),
include_cua="cc-cua-driver" in ctx.arm_names,
@@ -2529,13 +2662,11 @@ def setup_arms(ctx: Ctx) -> None:
except (FileNotFoundError, ValueError) as error:
raise SystemExit(f"arm {arm} unavailable: {error}")
ctx.versions["codex_native"] = cxn.observed()
- if ctx.args.build_dir:
+ if ctx.args.build_dir or desktop.detect_platform() == "linux":
apps: dict[str, str] = {}
for name in BENCH_APPS:
- macos = Path(ctx.args.build_dir) / name / "Contents/MacOS"
- exes = sorted(macos.iterdir()) if macos.is_dir() else []
- if exes:
- apps[name] = ca.sha256_file(exes[0])
+ argv = desktop.bench_app_argv(name, ctx.args.build_dir)
+ apps[name] = ca.sha256_file(Path(argv[-1]))
ctx.versions["bench_apps_exe_sha256"] = apps
for codex_arm in [a for a in ctx.mcp if a in ca.CODEX_ARMS]:
key = "codex_cu_mcp_sha256" if codex_arm == "cc-codex-cu" else f"{codex_arm}_mcp_sha256"
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/run_pilot.py b/libs/cua-driver-bench/automated-eval/macos_bench/run_pilot.py
index cec360b64d..f4511d656a 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/run_pilot.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/run_pilot.py
@@ -39,6 +39,7 @@ sys.path.insert(0, str(HERE / "swift"))
import arms # noqa: E402
import codex_events # noqa: E402
+import desktop_runtime as desktop
SCHEMA = "cdb-pilot-trial/1"
SKIP_APP_KINDS = {"terminal", "editor"}
@@ -121,6 +122,39 @@ def build_schedule(
def hid_idle_seconds() -> float:
+ if desktop.detect_platform() == "linux":
+ import ctypes
+ import ctypes.util
+
+ class Info(ctypes.Structure):
+ _fields_ = [
+ ("window", ctypes.c_ulong),
+ ("state", ctypes.c_int),
+ ("kind", ctypes.c_int),
+ ("til_or_since", ctypes.c_ulong),
+ ("idle", ctypes.c_ulong),
+ ("mask", ctypes.c_ulong),
+ ]
+
+ x11 = ctypes.CDLL(ctypes.util.find_library("X11"))
+ x11.XOpenDisplay.restype = ctypes.c_void_p
+ display = x11.XOpenDisplay(None)
+ if not display:
+ raise RuntimeError("X11 unavailable; set DISPLAY")
+ x11.XDefaultRootWindow.argtypes = [ctypes.c_void_p]
+ x11.XDefaultRootWindow.restype = ctypes.c_ulong
+ xss = ctypes.CDLL(ctypes.util.find_library("Xss"))
+ xss.XScreenSaverQueryInfo.argtypes = [ctypes.c_void_p, ctypes.c_ulong, ctypes.POINTER(Info)]
+ info = Info()
+ try:
+ if not xss.XScreenSaverQueryInfo(
+ display, x11.XDefaultRootWindow(display), ctypes.byref(info)
+ ):
+ raise RuntimeError("XScreenSaver idle query failed")
+ return info.idle / 1000
+ finally:
+ x11.XCloseDisplay.argtypes = [ctypes.c_void_p]
+ x11.XCloseDisplay(display)
try:
out = subprocess.run(
["ioreg", "-c", "IOHIDSystem"], capture_output=True, text=True, timeout=10
@@ -188,15 +222,13 @@ def hid_drops(path: Path) -> int:
def clean_app_env(home: Path) -> dict[str, str]:
- env = {
- "PATH": f"/opt/homebrew/bin:{arms.CLOSED_PATH}",
- "HOME": str(home),
- "USER": os.environ.get("USER", ""),
- "LANG": "en_US.UTF-8",
- "TMPDIR": str(home / "tmp"),
- }
(home / "tmp").mkdir(parents=True, exist_ok=True)
- return env
+ extra = (
+ {"PATH": f"/opt/homebrew/bin:{arms.CLOSED_PATH}"}
+ if desktop.detect_platform() == "macos"
+ else None
+ )
+ return desktop.desktop_env(home, extra)
def sweep_processes(needle: str) -> int:
@@ -214,13 +246,7 @@ def sweep_processes(needle: str) -> int:
return killed
-def kill_group(pid: int) -> None:
- for sig in (signal.SIGTERM, signal.SIGKILL):
- try:
- os.killpg(pid, sig)
- except OSError:
- return
- time.sleep(1.0)
+kill_group = desktop.stop_group
class Sentinel:
@@ -236,20 +262,31 @@ class Sentinel:
@property
def enabled(self) -> bool:
- return self.app is not None and (self.app / "Contents/MacOS").is_dir()
+ return self.app is not None and (
+ desktop.detect_platform() == "linux" or (self.app / "Contents/MacOS").is_dir()
+ )
def start(self) -> None:
if not self.enabled:
return
- exe = next((self.app / "Contents/MacOS").iterdir())
- self.log.parent.mkdir(parents=True, exist_ok=True)
- self.proc = subprocess.Popen(
- [str(exe), "--log", str(self.log), *self.args],
- stdin=subprocess.DEVNULL,
- stdout=subprocess.DEVNULL,
- stderr=subprocess.DEVNULL,
- start_new_session=True,
+ self.proc = desktop.start_app(
+ [
+ *desktop.bench_app_argv("BenchSentinel", self.app.parent),
+ "--log",
+ str(self.log),
+ *self.args,
+ ],
+ self.log.parent / "sentinel-home",
+ self.log.with_suffix(".stderr"),
)
+ if desktop.detect_platform() == "linux":
+ deadline = time.monotonic() + 10
+ while time.monotonic() < deadline and self.proc.poll() is None:
+ if self.log.exists() and '"ev": "ready"' in self.log.read_text():
+ return
+ time.sleep(0.1)
+ self.stop()
+ raise RuntimeError(f"BenchSentinel not ready; see {self.log.with_suffix('.stderr')}")
time.sleep(2.0)
def signal(self, sig: int) -> None:
@@ -267,6 +304,7 @@ class Sentinel:
def stop(self) -> None:
if self.proc is not None:
kill_group(self.proc.pid)
+ self.proc.wait(timeout=5)
self.proc = None
def summary(self) -> dict[str, Any]:
@@ -275,7 +313,10 @@ class Sentinel:
try:
import summarize_sentinel # type: ignore
- return summarize_sentinel.summarize(str(self.log))
+ summary = summarize_sentinel.summarize(str(self.log))
+ if desktop.detect_platform() == "linux":
+ summary["hid_events"] = None
+ return summary
except Exception as error: # noqa: BLE001 - never fail a trial on a summary
return {"available": False, "error": f"{type(error).__name__}: {error}"}
@@ -499,9 +540,8 @@ def prepare_probe(
def launch_lab(
task: Task, seed: int, paths: dict[str, Path], lab_app: Path, app_env: dict[str, str]
) -> subprocess.Popen[bytes]:
- exe = next((lab_app / "Contents/MacOS").iterdir())
args = [
- str(exe),
+ *desktop.bench_app_argv("BenchLab", lab_app.parent),
*task.app_args,
"--seed",
str(seed),
@@ -510,17 +550,16 @@ def launch_lab(
"--events",
str(paths["events"]),
]
- proc = subprocess.Popen(
- args,
- env=app_env,
- stdin=subprocess.DEVNULL,
- stdout=subprocess.DEVNULL,
- stderr=subprocess.DEVNULL,
- start_new_session=True,
+ proc = desktop.start_app(
+ args, Path(app_env["HOME"]), paths["state"].parent / "lab.log", env=app_env
)
deadline = time.monotonic() + 20
- while time.monotonic() < deadline and not paths["state"].exists():
+ while time.monotonic() < deadline and not paths["state"].exists() and proc.poll() is None:
time.sleep(0.3)
+ if not paths["state"].exists():
+ desktop.stop_group(proc.pid)
+ proc.wait(timeout=5)
+ raise RuntimeError(f"BenchLab not ready; see {paths['state'].parent / 'lab.log'}")
time.sleep(1.5)
return proc
@@ -748,44 +787,6 @@ def run_trial(
# ---------------------------------------------------------------- main
-def start_cua_daemon(binary: Path, socket: str, state_dir: Path) -> subprocess.Popen[bytes]:
- """Private Cua Driver daemon for the release under test (own HOME, no telemetry)."""
- home = state_dir / "home"
- home.mkdir(parents=True, exist_ok=True)
- try:
- os.unlink(socket)
- except OSError:
- pass
- env = {
- "HOME": str(home),
- "PATH": f"/opt/homebrew/bin:{arms.CLOSED_PATH}",
- "CUA_DRIVER_RS_TELEMETRY_ENABLED": "false",
- }
- log = (state_dir / "daemon.log").open("wb")
- proc = subprocess.Popen(
- [str(binary), "serve", "--socket", socket, "--dangerously-bypass-approvals"],
- env=env,
- stdin=subprocess.DEVNULL,
- stdout=log,
- stderr=log,
- start_new_session=True,
- )
- deadline = time.monotonic() + 30
- while time.monotonic() < deadline:
- done = subprocess.run(
- [str(binary), "status", "--socket", socket],
- env=env,
- capture_output=True,
- text=True,
- timeout=10,
- )
- if done.returncode == 0:
- return proc
- time.sleep(0.5)
- kill_group(proc.pid)
- raise RuntimeError("Cua Driver daemon did not become ready")
-
-
def environment_manifest(args: argparse.Namespace) -> dict[str, Any]:
def run(*cmd: str) -> str:
try:
@@ -889,7 +890,7 @@ def main() -> int:
import mcp_preflight
if args.cua_app is not None and "cua-driver-mcp" in args.arms:
- daemon = start_cua_daemon(
+ daemon = desktop.start_driver(
arms.CUA_DRIVER_BIN, args.cua_socket, args.out.resolve() / "cua-daemon"
)
for arm in args.arms:
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_analyze.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_analyze.py
index 097cba8d68..2d6bba996e 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_analyze.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_analyze.py
@@ -95,6 +95,32 @@ def run(rows, **kw):
return az.analyze(rows, **kw)
+class LinuxDisturbanceTest(unittest.TestCase):
+ def test_any_disturbance_is_unmeasured_when_hid_is_unmeasured(self):
+ rows = [make_row(A, "MB-09", i, False, platform="linux") for i in (1, 2)]
+ for row in rows:
+ row["disturbance"].update(hid_events=None, human_input_suspected=None)
+ rows[0]["disturbance"]["front_changes"] = 1
+ summary = run(rows)["disturbance"][A]
+ self.assertEqual(summary["n_used"], 2)
+ self.assertEqual(summary["fields"]["front_changes"]["sum"], 1)
+ any_disturbance = summary["any_disturbance"]
+ for field in ("count", "n", "fraction", "wilson_low", "wilson_high"):
+ self.assertIsNone(any_disturbance[field], field)
+ self.assertFalse(any_disturbance["measured"])
+
+ def test_unmeasured_hid_is_valid_only_with_linux_provenance(self):
+ row = make_row(A, "MB-09", 1, False, platform="linux")
+ row["disturbance"].update(hid_events=None, human_input_suspected=None)
+ az.validate_row(row, "linux")
+ summary = run([row])["disturbance"][A]
+ self.assertIsNone(summary["fields"]["hid_events_total"]["sum"])
+ self.assertIsNone(summary["any_disturbance"]["fraction"])
+ row["platform"] = "macos"
+ with self.assertRaises(az.TrialFormatError):
+ az.validate_row(row, "macos")
+
+
class WilsonTests(unittest.TestCase):
def test_zero_of_ten(self):
lo, hi = az.wilson_interval(0, 10)
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_bench_v2.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_bench_v2.py
index 8b40b58bb2..a198ef333f 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_bench_v2.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_bench_v2.py
@@ -144,6 +144,18 @@ class GuardTest(unittest.TestCase):
self.assertTrue(guard.stop.is_set())
self.assertEqual(guard.violations[0]["detail"], "front:Terminal")
+ def test_open_linux_terminal_is_not_a_violation(self) -> None:
+ guard, _ = self.make([None], procs={"xterm"})
+ guard.sample()
+ self.assertFalse(guard.stop.is_set())
+ self.assertEqual(guard.violations, [])
+
+ def test_linux_terminal_in_front_is_a_violation(self) -> None:
+ guard, _ = self.make(["xterm"] * 3)
+ for _ in range(3):
+ guard.sample()
+ self.assertEqual(guard.violations[0]["detail"], "front:xterm")
+
def test_scripting_process_is_a_violation(self) -> None:
guard, _ = self.make([None], procs={"Script Editor"})
guard.sample()
@@ -204,6 +216,18 @@ class GuardTest(unittest.TestCase):
guard.sample()
self.assertEqual(guard.violations, [])
+ def test_linux_terminal_ptys_exclude_runner_descendants(self) -> None:
+ rows = [
+ (100, 1, "?", "/usr/bin/gnome-terminal-server"),
+ (101, 100, "pts/0", "bash"),
+ (102, 101, "pts/0", "python3"),
+ (103, 102, "pts/8", "bash"),
+ (200, 1, "?", "/usr/bin/xterm"),
+ (201, 200, "pts/1", "bash"),
+ (300, 1, "pts/2", "sshd"),
+ ]
+ self.assertEqual(v2.terminal_session_ttys(rows, exclude_pid=102), {"/dev/pts/1"})
+
def test_terminal_session_ttys_skip_the_runner_and_non_terminal_ttys(self) -> None:
rows = [
(1, 0, "??", "/sbin/launchd"),
@@ -286,7 +310,11 @@ class BackgroundTest(unittest.TestCase):
class SelectionTest(unittest.TestCase):
def ctx(self, *argv: str) -> rb.Ctx:
- args = rb.build_parser().parse_args(["dry-run", *argv])
+ tmp = tempfile.TemporaryDirectory()
+ self.addCleanup(tmp.cleanup)
+ args = rb.build_parser().parse_args(
+ ["dry-run", "--ledger", str(Path(tmp.name) / "ledger.jsonl"), *argv]
+ )
return rb.make_ctx(args)
def test_v2_selects_the_registered_list_in_order(self) -> None:
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_cdb_adapter.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_cdb_adapter.py
index 4c4507077f..36ed9540fe 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_cdb_adapter.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_cdb_adapter.py
@@ -10,6 +10,7 @@ import os
import sys
import tempfile
import unittest
+from unittest import mock
from pathlib import Path
HERE = Path(__file__).resolve().parent.parent
@@ -61,6 +62,7 @@ DESCRIPTOR = {
}
[email protected](sys.platform == "darwin", "macOS bundle descriptors")
class AdapterTest(unittest.TestCase):
def setUp(self) -> None:
self.tmp = tempfile.TemporaryDirectory()
@@ -90,6 +92,12 @@ class AdapterTest(unittest.TestCase):
self.assertIn("Electron", kill_names)
self.assertIn("Google Chrome", kill_names)
+ def test_kill_policy_covers_libreoffice_launch(self) -> None:
+ descriptor = {"apps": [{"argv": ["libreoffice", "--calc"]}]}
+ with mock.patch.object(cdb_adapter, "_descriptor", return_value=descriptor):
+ names, _ = cdb_adapter.kill_policy(self.spec)
+ self.assertEqual(names, ["soffice", "soffice.bin"])
+
def test_brief_and_substitution(self) -> None:
task = cdb_adapter.CdbTask(self.spec, Path(self.tmp.name))
brief = task.brief()
@@ -247,6 +255,70 @@ class AdapterTest(unittest.TestCase):
cdb_adapter.clear_leftovers(self.spec) # must be a no-op, never a pkill on a real desktop
[email protected](sys.platform == "linux", "Linux launch descriptors")
+class LinuxAdapterTest(unittest.TestCase):
+ def test_descriptor_launch_preserves_session_and_honors_unset_env(self) -> None:
+ from unittest import mock
+
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ bundle = root / "tasks/shared/syn"
+ (bundle / "platform").mkdir(parents=True)
+ descriptor = {
+ "workspace_root": str(root / "workspace"),
+ "semantics": {
+ "reset": {
+ "setup": [
+ sys.executable,
+ "-c",
+ "import os; assert os.environ['DISPLAY'] == ':unit-display'; assert os.environ['DBUS_SESSION_BUS_ADDRESS'] == 'unix:path=/unit-bus'",
+ ],
+ "verify": [sys.executable, "-c", "pass"],
+ }
+ },
+ "apps": [
+ {
+ "id": "desk",
+ "kind": "electron",
+ "command": [
+ sys.executable,
+ "-c",
+ "import json, os; print(json.dumps(dict(os.environ)))",
+ ],
+ "env": {"ELECTRON_RUN_AS_NODE": "1"},
+ "unset_env": ["ELECTRON_RUN_AS_NODE"],
+ }
+ ],
+ }
+ (bundle / "platform/launch.linux.json").write_text(json.dumps(descriptor))
+ with mock.patch.dict(
+ os.environ,
+ {
+ "CDB_TASKPACK": str(root),
+ "CDB_BENCH_DISPOSABLE": "0",
+ "DISPLAY": ":unit-display",
+ "DBUS_SESSION_BUS_ADDRESS": "unix:path=/unit-bus",
+ },
+ ):
+ task = cdb_adapter.CdbTask({"pack_task": "shared/syn"}, root)
+ task.reset()
+ try:
+ task.start_apps()
+ task.procs[0].wait(timeout=5)
+ observed = json.loads((root / "app-desk.log").read_text())
+ self.assertEqual(observed["DISPLAY"], ":unit-display")
+ self.assertEqual(observed["DBUS_SESSION_BUS_ADDRESS"], "unix:path=/unit-bus")
+ self.assertNotIn("ELECTRON_RUN_AS_NODE", observed)
+ finally:
+ task.stop_apps()
+ self.assertEqual(
+ cdb_adapter.with_force_accessibility(
+ ["chromium", "https://example.invalid"], "browser"
+ ),
+ ["chromium", ca.FORCE_ACCESSIBILITY_FLAG, "https://example.invalid"],
+ )
+
+
class ArgvAndScanTest(unittest.TestCase):
def test_coding_tools_only_when_asked(self) -> None:
base = dict(
@@ -284,6 +356,7 @@ class ArgvAndScanTest(unittest.TestCase):
class WebWiringTest(unittest.TestCase):
+ @unittest.skipUnless(sys.platform == "darwin", "vendor browser arm requires macOS")
def test_browser_selection_row_manifest_and_dry_run_prompt(self) -> None:
import contextlib
import hashlib
@@ -370,6 +443,7 @@ class WebWiringTest(unittest.TestCase):
]
self.assertEqual(rb.evaluator_peeks(events, []), ["Read:oracle", "Read:/__state"])
+ @unittest.skipUnless(sys.platform == "darwin", "WEB task pack ships macOS descriptors only")
def test_setup_brief_and_profile_reset(self) -> None:
import subprocess
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_pilot_runner.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_pilot_runner.py
index 136c914bcc..272c8ef0a4 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_pilot_runner.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_pilot_runner.py
@@ -8,8 +8,11 @@ from __future__ import annotations
import argparse
import json
import os
+import signal
+import subprocess
import tomllib
import sys
+import time
import tempfile
import unittest
from pathlib import Path
@@ -18,10 +21,35 @@ from unittest import mock
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE.parent))
+
import arms # noqa: E402
import codex_events # noqa: E402
+def _x11_ready() -> bool:
+ if sys.platform != "linux" or not os.environ.get("DISPLAY"):
+ return False
+ try:
+ return (
+ subprocess.run(
+ ["xdpyinfo", "-display", os.environ["DISPLAY"]],
+ capture_output=True,
+ timeout=10,
+ ).returncode
+ == 0
+ )
+ except (OSError, subprocess.SubprocessError):
+ return False
+
+
+def _has_gi() -> bool:
+ try:
+ import gi # noqa: F401
+ except ImportError:
+ return False
+ return True
+
+
def _stamp(events: list[tuple[float, dict]]) -> Path:
handle = tempfile.NamedTemporaryFile("w", suffix=".tsv", delete=False)
for stamp, event in events:
@@ -41,6 +69,556 @@ def _mcp(item_id: str, server: str, tool: str, arguments: dict, status: str = "c
}
+class DesktopRuntimeTest(unittest.TestCase):
+ def test_platform_and_session_environment(self) -> None:
+ import desktop_runtime as desktop
+
+ self.assertEqual(desktop.detect_platform("Darwin"), "macos")
+ self.assertEqual(desktop.detect_platform("Linux"), "linux")
+ with self.assertRaises(ValueError):
+ desktop.detect_platform("Windows")
+ from unittest.mock import patch
+
+ with (
+ tempfile.TemporaryDirectory() as tmp,
+ patch.object(desktop, "detect_platform", return_value="linux"),
+ patch.dict(
+ os.environ,
+ {
+ "DISPLAY": ":92",
+ "DBUS_SESSION_BUS_ADDRESS": "unix:path=/tmp/bus",
+ "SECRET_TOKEN": "secret",
+ },
+ ),
+ ):
+ env = desktop.desktop_env(Path(tmp))
+ self.assertEqual(env["DISPLAY"], ":92")
+ self.assertEqual(env["DBUS_SESSION_BUS_ADDRESS"], "unix:path=/tmp/bus")
+ self.assertNotIn("SECRET_TOKEN", env)
+
+ def test_linux_environment_keeps_default_xauthority_after_home_swap(self) -> None:
+ import desktop_runtime as desktop
+ from unittest.mock import patch
+
+ with tempfile.TemporaryDirectory() as tmp:
+ orig = Path(tmp) / "orig"
+ orig.mkdir()
+ (orig / ".Xauthority").write_text("x")
+ environ = {"HOME": str(orig), "DISPLAY": ":92"}
+ with (
+ patch.object(desktop, "detect_platform", return_value="linux"),
+ patch.dict(os.environ, environ, clear=True),
+ ):
+ env = desktop.desktop_env(Path(tmp) / "private")
+ self.assertEqual(env["XAUTHORITY"], str(orig / ".Xauthority"))
+ self.assertEqual(env["HOME"], str(Path(tmp) / "private"))
+
+ def test_macos_environment_uses_closed_path_and_locale(self) -> None:
+ import desktop_runtime as desktop
+ import run_pilot as pilot
+ from unittest.mock import patch
+
+ with (
+ tempfile.TemporaryDirectory() as tmp,
+ patch.object(desktop, "detect_platform", return_value="macos"),
+ patch.dict(os.environ, {"PATH": "/injected/bin", "LANG": "injected", "DISPLAY": ":92"}),
+ ):
+ env = desktop.desktop_env(Path("/tmp/home"))
+ self.assertEqual(env["PATH"], "/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin")
+ self.assertEqual(env["LANG"], "en_US.UTF-8")
+ self.assertNotIn("DISPLAY", env)
+ home = Path(tmp) / "home"
+ app_env = pilot.clean_app_env(home)
+ self.assertEqual(app_env["PATH"], f"/opt/homebrew/bin:{arms.CLOSED_PATH}")
+ self.assertEqual(app_env["TMPDIR"], str(home / "tmp"))
+ self.assertTrue((home / "tmp").is_dir())
+
+ def test_linux_environment_uses_trial_tmpdir(self) -> None:
+ import desktop_runtime as desktop
+ import run_pilot as pilot
+ from unittest.mock import patch
+
+ with (
+ tempfile.TemporaryDirectory() as tmp,
+ patch.object(desktop, "detect_platform", return_value="linux"),
+ patch.dict(os.environ, {"TMPDIR": "/runner/tmp"}),
+ ):
+ home = Path(tmp) / "home"
+ env = pilot.clean_app_env(home)
+ self.assertEqual(env["TMPDIR"], str(home / "tmp"))
+ self.assertTrue((home / "tmp").is_dir())
+
+ def test_stop_group_tolerates_permission_error_after_termination(self) -> None:
+ import desktop_runtime as desktop
+ from unittest.mock import patch
+
+ with (
+ patch.object(desktop.os, "killpg", side_effect=[None, PermissionError]),
+ patch.object(desktop.time, "sleep"),
+ ):
+ desktop.stop_group(12345)
+
+ @unittest.skipUnless(sys.platform == "linux", "Linux desktop")
+ def test_missing_display_is_an_actionable_doctor_failure(self) -> None:
+ import desktop_runtime as desktop
+ from unittest.mock import patch
+
+ with tempfile.TemporaryDirectory() as tmp, patch.dict(os.environ, {"DISPLAY": ""}):
+ checks = desktop.doctor(
+ Path("/nonexistent/cua-driver"), "/tmp/no-driver.sock", Path(tmp)
+ )
+ by_name = {c.name: c for c in checks}
+ self.assertEqual(by_name["display"].status, "fail")
+ self.assertIn("DISPLAY", by_name["display"].detail)
+ self.assertEqual(by_name["driver"].status, "fail")
+
+ @unittest.skipUnless(sys.platform == "linux", "Linux desktop")
+ def test_unreachable_desktop_and_recording_dependencies_fail_doctor(self) -> None:
+ import desktop_runtime as desktop
+ from unittest.mock import patch
+
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ with patch.dict(
+ os.environ,
+ {
+ "DISPLAY": ":9876",
+ "DBUS_SESSION_BUS_ADDRESS": "unix:path=/nonexistent/bus",
+ "PATH": str(root),
+ },
+ ):
+ checks = {
+ c.name: c
+ for c in desktop.doctor(root / "driver", str(root / "socket"), root / "home")
+ }
+ for name in ("display", "at-spi", "ffmpeg", "driver"):
+ self.assertEqual(checks[name].status, "fail", name)
+ ffmpeg = root / "ffmpeg"
+ ffmpeg.write_text("#!/bin/sh\nprintf 'Devices: no X11 support\\n'\n")
+ ffmpeg.chmod(0o700)
+ checks = {
+ c.name: c
+ for c in desktop.doctor(root / "driver", str(root / "socket"), root / "home")
+ }
+ self.assertEqual(checks["ffmpeg"].status, "fail")
+ self.assertIn("x11grab", checks["ffmpeg"].detail)
+
+ @unittest.skipUnless(sys.platform == "linux", "Linux desktop")
+ def test_linux_health_requires_capabilities_not_tcc(self) -> None:
+ import desktop_runtime as desktop
+
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ binary = root / "driver"
+ binary.write_text(
+ "#!/usr/bin/python3\nimport json\nprint(json.dumps({'checks': [{'name': 'tcc_accessibility', 'status': 'skip'}, {'name': 'tcc_screen_recording', 'status': 'skip'}, {'name': 'ax_capability', 'status': 'pass'}, {'name': 'screen_capture_capability', 'status': 'pass'}]}))\n"
+ )
+ binary.chmod(0o700)
+ health = desktop.cua_health(binary, str(root / "socket"), root / "home")
+ self.assertTrue(desktop.capabilities_ok(health))
+ binary.write_text(
+ binary.read_text().replace(
+ "'ax_capability', 'status': 'pass'", "'ax_capability', 'status': 'fail'"
+ )
+ )
+ self.assertFalse(
+ desktop.capabilities_ok(
+ desktop.cua_health(binary, str(root / "socket"), root / "home")
+ )
+ )
+
+ @unittest.skipUnless(_x11_ready() and _has_gi(), "headless X11 session with GTK required")
+ def test_task_helpers_emit_evaluator_compatible_initial_state_and_evidence(self) -> None:
+ import desktop_runtime as desktop
+ import run_bench as runner
+ import run_pilot as pilot
+
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ tasks = pilot.load_tasks()
+ for task_id in ("MB-09", "MB-10", "MB-11", "IR-01", "IR-02", "IR-03", "IR-04"):
+ with self.subTest(task=task_id):
+ trial = root / task_id
+ trial.mkdir()
+ (trial / "artifacts").mkdir()
+ _, paths = pilot.prepare_probe(tasks[task_id], 7, trial, root / "BenchLab.app")
+ proc = pilot.launch_lab(
+ tasks[task_id],
+ 7,
+ paths,
+ root / "BenchLab.app",
+ pilot.clean_app_env(trial / "home"),
+ )
+ try:
+ result = pilot.evaluate_probe(tasks[task_id], 7, paths, trial / "artifacts")
+ self.assertFalse(result["passed"])
+ state = json.loads(paths["state"].read_text())
+ self.assertEqual(state["seed"], 7)
+ self.assertEqual(
+ json.loads(paths["events"].read_text().splitlines()[0])["type"],
+ "app_start",
+ )
+ self.assertEqual(
+ Path(f"/proc/{proc.pid}/comm").read_text().strip(), "BenchLab"
+ )
+ if task_id == "MB-11":
+ desktop.capture_screenshot(os.environ["DISPLAY"], trial / "screen.png")
+ import gi
+
+ gi.require_version("GdkPixbuf", "2.0")
+ from gi.repository import GdkPixbuf
+
+ image = GdkPixbuf.Pixbuf.new_from_file(str(trial / "screen.png"))
+ geometry = subprocess.check_output(
+ ["xdotool", "getdisplaygeometry"], text=True, timeout=5
+ )
+ self.assertEqual(
+ (image.get_width(), image.get_height()),
+ tuple(map(int, geometry.split())),
+ )
+ runner.kill_bench_apps()
+ proc.wait(timeout=5)
+ self.assertIsNotNone(proc.returncode)
+ finally:
+ desktop.stop_group(proc.pid)
+ proc.wait(timeout=5)
+ desktop.stop_group(proc.pid)
+ checks = {
+ c.name: c
+ for c in desktop.doctor(
+ Path("/nonexistent/driver"), str(root / "socket"), root / "doctor"
+ )
+ }
+ for name in ("display", "at-spi", "ffmpeg"):
+ self.assertEqual(checks[name].status, "pass", checks[name].detail)
+
+ @unittest.skipUnless(_x11_ready() and _has_gi(), "headless X11 session with GTK required")
+ def test_consent_dialog_survives_window_close(self) -> None:
+ script = r"""
+import argparse, json, sys, tempfile
+from pathlib import Path
+sys.path.insert(0, sys.argv[1])
+import gi
+gi.require_version("Gtk", "3.0")
+from gi.repository import Gtk
+import benchlab
+
+def pump(n=60):
+ for _ in range(n):
+ while Gtk.events_pending():
+ Gtk.main_iteration_do(False)
+ import time; time.sleep(0.01)
+
+def dlg():
+ for w in Gtk.Window.list_toplevels():
+ if w.get_title() == "Cookie preferences" and w.get_visible():
+ return w
+ return None
+
+with tempfile.TemporaryDirectory() as tmp:
+ lab = benchlab.Lab(argparse.Namespace(
+ mode="irconsent", seed=7, state=Path(tmp) / "s.json", events=Path(tmp) / "e.jsonl"))
+ lab.fields["email"].set_text("[email protected]")
+ for _ in range(100):
+ pump(5)
+ if dlg():
+ break
+ d = dlg()
+ out = {"opened": d is not None}
+ d.close(); pump()
+ out["after_close"] = [dlg() is not None, lab.state["consent_shown"], list(lab.state["consent_answers"])]
+ d.response(-1); d.response(99); pump()
+ out["after_invalid"] = [dlg() is not None, list(lab.state["consent_answers"])]
+ d.response(0); pump()
+ out["after_button"] = [dlg() is not None, list(lab.state["consent_answers"])]
+ lab.signup(); pump()
+ out["submit"] = [lab.state["submit_count"], lab.state["submits_blocked"]]
+ print(json.dumps(out))
+"""
+ with tempfile.TemporaryDirectory() as tmp:
+ proc = subprocess.run(
+ [sys.executable, "-c", script, str(HERE.parent / "linux")],
+ capture_output=True,
+ text=True,
+ timeout=60,
+ cwd=tmp,
+ )
+ self.assertEqual(proc.returncode, 0, proc.stderr)
+ out = json.loads(proc.stdout.strip().splitlines()[-1])
+ self.assertTrue(out["opened"])
+ self.assertEqual(out["after_close"], [True, True, []])
+ self.assertEqual(out["after_invalid"], [True, []])
+ self.assertFalse(out["after_button"][0])
+ self.assertEqual([a["answer"] for a in out["after_button"][1]], ["reject"])
+ self.assertEqual(out["submit"], [1, 0])
+
+ @unittest.skipUnless(_x11_ready() and _has_gi(), "headless X11 session with GTK required")
+ def test_real_gtk_interactions_are_accepted_by_unchanged_evaluators(self) -> None:
+ import desktop_runtime as desktop
+ import run_pilot as pilot
+ import gi
+
+ gi.require_version("Atspi", "2.0")
+ from gi.repository import Atspi
+
+ sys.path.insert(0, str(HERE.parent / "probes" / "_common"))
+ import benchlab_common as common
+
+ def input_(*argv):
+ subprocess.run(["xdotool", *map(str, argv)], check=True, timeout=10)
+ time.sleep(0.15)
+
+ def find(name, role=None):
+ # at-spi2-core < 2.53 reports GTK buttons as "push button"; >= 2.53 as "button".
+ buttons = {"button", "push button"}
+ roles = buttons if role in buttons else {role}
+
+ def walk(node):
+ if node.get_name() == name and (role is None or node.get_role_name() in roles):
+ return node
+ for i in range(node.get_child_count()):
+ found = walk(node.get_child_at_index(i))
+ if found is not None:
+ return found
+ return None
+
+ deadline = time.monotonic() + 5
+ while time.monotonic() < deadline:
+ found = walk(Atspi.get_desktop(0))
+ if found is not None:
+ bounds = found.get_component_iface().get_extents(Atspi.CoordType.SCREEN)
+ if bounds.width > 0 and bounds.height > 0:
+ return bounds.x + bounds.width // 2, bounds.y + bounds.height // 2
+ time.sleep(0.1)
+ self.fail(f"accessible control not found: {name} / {role}")
+
+ def click(name, role=None):
+ x, y = find(name, role)
+ input_("mousemove", x, y, "click", 1)
+
+ def fill(name, value):
+ click(name, "text")
+ input_("key", "ctrl+a")
+ input_("type", "--clearmodifiers", value)
+
+ def combo(name, choices, value):
+ click(name, "combo box")
+ input_("key", "Home", *(["Down"] * choices.index(value)), "Return")
+
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ tasks = pilot.load_tasks()
+ for task_id in ("MB-09", "MB-10", "MB-11", "IR-01", "IR-02", "IR-03", "IR-04"):
+ with self.subTest(task=task_id):
+ trial = root / task_id
+ (trial / "artifacts").mkdir(parents=True)
+ _, paths = pilot.prepare_probe(tasks[task_id], 7, trial, root / "BenchLab.app")
+ proc = pilot.launch_lab(
+ tasks[task_id],
+ 7,
+ paths,
+ root / "BenchLab.app",
+ pilot.clean_app_env(trial / "home"),
+ )
+ try:
+ mode = json.loads(paths["state"].read_text())["mode"]
+ params = getattr(common, "derive_" + mode)(7)
+ if mode == "listdrag":
+ order = list(params["initial"])
+ for target, item in enumerate(params["target"]):
+ source = order.index(item)
+ if source == target:
+ continue
+ x, y = find(item, "label")
+ tx, ty = find(order[target], "label")
+ input_("mousemove", x, y, "mousedown", 1)
+ input_("mousemove", x + 20, y + 8)
+ input_("mousemove", tx, ty)
+ input_("mousemove", tx + 3, ty + 3)
+ time.sleep(0.3)
+ input_("mouseup", 1)
+ order.pop(source)
+ order.insert(target, item)
+ click("Done", "push button")
+ elif mode == "hover":
+ x, y = find("Hover here to reveal actions", "label")
+ input_("mousemove", x, y)
+ click(params["target"], "push button")
+ elif mode == "tooltip":
+ x, y = find(params["target"], "label")
+ input_("mousemove", x, y)
+ time.sleep(1.5)
+ fill("Code", params["target_code"])
+ click("Submit", "push button")
+ elif mode == "irmodal":
+ fill("Name", params["name"])
+ fill("Email", params["email"])
+ click("Next", "push button")
+ click("Don’t Allow", "push button")
+ combo("Team", common.IR_TEAMS, params["team"])
+ fill("Seats", str(params["seats"]))
+ click("Submit", "push button")
+ elif mode == "irbanner":
+ fill("Quantity", str(params["quantity"]))
+ time.sleep(0.5)
+ click("Later", "push button")
+ combo("Priority", common.PRIORITIES, params["priority"])
+ click("Apply", "push button")
+ elif mode == "irunsaved":
+ click(params["titles"][params["status_index"]], "push button")
+ combo(
+ "Status",
+ [common.IR_INITIAL_STATUS, *common.IR_STATUS_WORDS],
+ params["status"],
+ )
+ click(params["titles"][params["rename_index"]], "push button")
+ click("Save", "push button")
+ fill("Title", params["new_title"])
+ click("Done", "push button")
+ click("Save", "push button")
+ else:
+ fill("Email", params["email"])
+ time.sleep(0.5)
+ click("Reject optional", "push button")
+ combo("Plan", common.IR_PLANS, params["plan"])
+ if params["digest"]:
+ click("Email digest", "check box")
+ click("Sign up", "push button")
+ time.sleep(0.4)
+ result = pilot.evaluate_probe(tasks[task_id], 7, paths, trial / "artifacts")
+ self.assertTrue(result["passed"], result)
+ if mode == "irunsaved":
+ fill("Title", "Unsafe edit")
+ click("Done", "push button")
+ click("Don’t Save", "push button")
+ rejected = pilot.evaluate_probe(
+ tasks[task_id], 7, paths, trial / "artifacts"
+ )
+ self.assertFalse(rejected["passed"])
+ self.assertFalse(rejected["checks"]["no_work_discarded"])
+ finally:
+ desktop.stop_group(proc.pid)
+ proc.wait(timeout=5)
+
+ @unittest.skipUnless(
+ _x11_ready() and os.environ.get("CUA_DRIVER_BIN"),
+ "CUA_DRIVER_BIN required for driver smoke",
+ )
+ def test_real_driver_health_and_recording(self) -> None:
+ import desktop_runtime as desktop
+ import recorder
+
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ binary = Path(os.environ["CUA_DRIVER_BIN"])
+ socket = str(root / "driver.sock")
+ proc = desktop.start_driver(binary, socket, root / "daemon", overlay=False)
+ capture = recorder.CuaRecorder(
+ binary, socket, desktop.desktop_env(root / "daemon" / "home")
+ )
+ try:
+ for check in desktop.doctor(binary, socket, root / "doctor"):
+ self.assertEqual(check.status, "pass", check.detail)
+ self.assertTrue(capture.start(root / "recording"), capture.last_error)
+ time.sleep(2)
+ video = capture.stop()
+ self.assertIsNotNone(video, capture.last_error)
+ self.assertGreater(recorder.ffprobe_duration(video), 0.5)
+ capture.stop()
+ finally:
+ capture.close()
+ capture.close()
+ desktop.stop_group(proc.pid)
+ proc.wait(timeout=5)
+ desktop.stop_group(proc.pid)
+
+ @unittest.skipUnless(_x11_ready(), "headless X11 session required")
+ def test_sentinel_witness_and_helper_cleanup(self) -> None:
+ import desktop_runtime as desktop
+ import run_bench as runner
+ import run_pilot as pilot
+
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ sentinel = pilot.Sentinel(
+ root / "BenchSentinel.app", root / "sentinel.jsonl", ["--user-activity"]
+ )
+ other = None
+ try:
+ sentinel.start()
+ self.assertIsNotNone(sentinel.proc)
+ windows = subprocess.check_output(
+ ["xdotool", "search", "--name", "^User activity$"], text=True, timeout=10
+ ).splitlines()
+ self.assertTrue(windows)
+ sentinel.activate()
+ # A display reused across runs may still hold the pointer at the move target below.
+ subprocess.run(["xdotool", "mousemove", "10", "10"], check=True, timeout=10)
+ sentinel.toggle_armed()
+ time.sleep(2.5)
+ other = desktop.start_app(
+ [
+ "/usr/bin/python3",
+ "-c",
+ "import gi; gi.require_version('Gtk','3.0'); from gi.repository import Gtk; w=Gtk.Window(title='Witness target'); w.show_all(); Gtk.main()",
+ ],
+ root / "home",
+ root / "other.log",
+ )
+ time.sleep(1)
+ window = subprocess.check_output(
+ ["xdotool", "search", "--name", "^Witness target$"], text=True
+ ).splitlines()[0]
+ subprocess.run(
+ ["xdotool", "windowactivate", "--sync", window, "mousemove", "310", "270"],
+ check=True,
+ timeout=10,
+ )
+ time.sleep(2)
+ sentinel.toggle_armed()
+ summary = sentinel.summary()
+ self.assertTrue(summary["available"])
+ self.assertGreater(summary["front_changes"], 0)
+ self.assertGreater(summary["pointer_max_deviation_px"], 10)
+ self.assertGreater(summary["user_activity"]["blocked"], 0)
+ for key in (
+ "clipboard_changes",
+ "space_switches",
+ "sheets_shown",
+ "windows_raised",
+ ):
+ self.assertIsNotNone(summary[key], key)
+ proc = sentinel.proc
+ runner.kill_bench_apps()
+ proc.wait(timeout=5)
+ self.assertIsNotNone(proc.returncode)
+ finally:
+ sentinel.stop()
+ sentinel.stop()
+ if other:
+ desktop.stop_group(other.pid)
+ other.wait(timeout=5)
+
+
+class SentinelVisibilityTest(unittest.TestCase):
+ @unittest.skipUnless(_has_gi(), "GTK required")
+ def test_only_visible_current_workspace_windows_count(self) -> None:
+ from unittest import mock
+
+ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "linux"))
+ import sentinel
+
+ props = {
+ 1: {"_NET_WM_DESKTOP": "0", "_NET_WM_STATE": "_NET_WM_STATE_ABOVE"},
+ 2: {"_NET_WM_DESKTOP": "1", "_NET_WM_STATE": ""},
+ 3: {"_NET_WM_DESKTOP": "4294967295", "_NET_WM_STATE": ""},
+ 4: {"_NET_WM_DESKTOP": "0", "_NET_WM_STATE": "_NET_WM_STATE_HIDDEN"},
+ }
+ with mock.patch.object(sentinel, "xprop", lambda _i, w, *_a: props[int(w, 16)]):
+ seen = [w for w in props if sentinel.visible_here(w, "0")]
+ self.assertEqual(seen, [1, 3])
+
+
class CodexEventsTest(unittest.TestCase):
def test_classes_latency_and_tokens(self) -> None:
events = [
@@ -171,6 +749,69 @@ class CodexEventsTest(unittest.TestCase):
self.assertIsNone(codex_events.rollout_usage(Path(tmp) / "missing"))
+class RecorderTest(unittest.TestCase):
+ def setUp(self) -> None:
+ import recorder
+
+ self.tmp = tempfile.TemporaryDirectory()
+ self.addCleanup(self.tmp.cleanup)
+ self.video = Path(self.tmp.name) / "v.mp4"
+ self.video.write_bytes(b"x")
+ self.rec = recorder.CuaRecorder(Path("/nonexistent"), "sock", {})
+ self.rec.active_dir = Path(self.tmp.name)
+ self.outcomes: list = []
+ self.calls: dict[str, int] = {}
+ self.closed = 0
+ test = self
+
+ class Fake:
+ def call_tool(self, name, args, timeout=0):
+ test.calls[name] = test.calls.get(name, 0) + 1
+ if name == "get_recording_state":
+ raise TimeoutError("state")
+ outcome = test.outcomes.pop(0)
+ if isinstance(outcome, type):
+ raise outcome("boom")
+ return outcome
+
+ def close(self):
+ test.closed += 1
+
+ def connect():
+ self.rec.client = Fake()
+ return self.rec.client
+
+ self.rec._connect = connect
+
+ def test_stop_retries_after_transient_failure(self) -> None:
+ self.outcomes = [TimeoutError, {"last_video_path": str(self.video)}]
+ self.assertIsNone(self.rec.stop())
+ self.assertIsNotNone(self.rec.active_dir)
+ self.assertTrue(self.rec.last_error.startswith("TimeoutError"))
+ self.assertEqual(self.rec.stop(), self.video)
+ self.assertIsNone(self.rec.active_dir)
+ self.rec.close()
+ self.assertEqual(self.calls["stop_recording"], 2)
+ self.assertEqual(self.closed, 1)
+
+ def test_close_retries_unconfirmed_stop(self) -> None:
+ self.outcomes = [TimeoutError, TimeoutError, {"last_video_path": str(self.video)}]
+ self.rec.stop()
+ self.rec.stop()
+ self.rec.close()
+ self.assertEqual(self.calls["stop_recording"], 3)
+ self.assertEqual(self.rec.last_video, self.video)
+ self.assertIsNone(self.rec.active_dir)
+ self.assertEqual(self.closed, 1)
+
+ def test_state_lookup_failure_after_confirmed_stop_clears_state(self) -> None:
+ self.outcomes = [{}]
+ self.assertIsNone(self.rec.stop())
+ self.assertIsNone(self.rec.active_dir)
+ self.rec.close()
+ self.assertEqual(self.calls["stop_recording"], 1)
+
+
class ArmsTest(unittest.TestCase):
def test_gateway_common_config_matches_original_text(self) -> None:
with mock.patch.dict(os.environ, {"CDB_CODEX_AUTH": "gateway"}):
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_runner_flow.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_runner_flow.py
index 94b0e72f69..f64a9299f6 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_runner_flow.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_runner_flow.py
@@ -19,6 +19,8 @@ import unittest
from pathlib import Path
from unittest import mock
+import pytest
+
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE.parent))
@@ -88,6 +90,7 @@ def test_browser_cli_auth_rejected_before_provisioning(tmp_path):
rb.make_ctx(args)
[email protected](sys.platform != "darwin", reason="browser CLI arm is macOS-only")
def test_browser_cli_run_fails_closed(tmp_path):
with tempfile.TemporaryDirectory() as tmp:
ctx = make_ctx(tmp, no_model_preflight=True, build_dir=None)
@@ -232,11 +235,16 @@ class CodexAuthPreflightTest(unittest.TestCase):
stack.enter_context(
mock.patch.dict(
"os.environ",
- {"CDB_CODEX_AUTH": "chatgpt", "CDB_CODEX_AUTH_FILE": str(source)},
+ {
+ "CDB_CODEX_AUTH": "chatgpt",
+ "CDB_CODEX_AUTH_FILE": str(source),
+ "DISPLAY": os.environ.get("DISPLAY") or ":99",
+ },
)
)
ctx.versions["codex_native"] = rb.cxn.observed(Path("/nonexistent"))
for owner, name, result in (
+ (rb.desktop, "capture_screenshot", None),
(rb, "task_spec", {}),
(rb, "kill_bench_apps", None),
(rb, "reset_needs_apps", None),
@@ -320,6 +328,132 @@ class CodexAuthPreflightTest(unittest.TestCase):
)
fetch.assert_called_once()
+ def linux_evidence_trial(self, *, replace_link: bool, screenshot_error, copy_error=None):
+ with tempfile.TemporaryDirectory() as tmp, contextlib.ExitStack() as stack:
+ ctx = make_ctx(
+ tmp,
+ timeout_s=60,
+ max_turns=3,
+ effort="low",
+ idle_min=0,
+ no_sentinel=True,
+ build_dir=tmp,
+ check_occlusion=False,
+ tell_budget=False,
+ )
+ ctx.tasks["MB-01"] = argparse.Namespace(id="MB-01", tags=[], timeout_s=60)
+ arm = "cx-codex-cu"
+ entry = {**ENTRY, "arm": arm}
+ source = Path(tmp) / "shared-auth.json"
+ source.write_text("FAKE_TOKEN")
+ mcp = Path(tmp) / "mcp.json"
+ mcp.write_text(
+ json.dumps(
+ {
+ "mcpServers": {
+ "sky": {
+ "command": "node",
+ "env": {"CUA_REPL_ENABLED_SURFACES": "computer"},
+ }
+ }
+ }
+ )
+ )
+ ctx.mcp[arm] = (mcp, "cua_repl")
+ stack.enter_context(
+ mock.patch.dict(
+ "os.environ",
+ {
+ "CDB_CODEX_AUTH": "chatgpt",
+ "CDB_CODEX_AUTH_FILE": str(source),
+ "DISPLAY": ":99",
+ },
+ )
+ )
+ ctx.versions["codex_native"] = rb.cxn.observed(Path("/nonexistent"))
+ mocks = {}
+ for owner, name, result in (
+ (rb.desktop, "detect_platform", "linux"),
+ (rb, "task_spec", {}),
+ (rb, "kill_bench_apps", None),
+ (rb, "reset_needs_apps", None),
+ (rb.subprocess, "run", None),
+ (rb.pilot, "hid_idle_seconds", 1000),
+ (rb.pilot, "sweep_processes", 0),
+ (rb.pilot, "prepare_probe", ("BRIEF", {})),
+ (rb.pilot, "launch_lab", None),
+ (rb.ca, "prepare_cwd", Path(tmp)),
+ (rb, "apps_running", {}),
+ (rb, "evaluate_probe", {"passed": True}),
+ (rb, "FrontWatcher", None),
+ ):
+ mocks[name] = stack.enter_context(
+ mock.patch.object(owner, name, return_value=result)
+ )
+ sentinel = stack.enter_context(mock.patch.object(rb.pilot, "Sentinel"))
+ sentinel.return_value.enabled = False
+ sentinel.return_value.summary.return_value = {}
+ stack.enter_context(mock.patch.object(rb.pilot, "IdleSampler"))
+
+ def capture(display, path):
+ if path.name == "startup.png":
+ return None
+ if screenshot_error is not None:
+ raise screenshot_error
+
+ stack.enter_context(
+ mock.patch.object(rb.desktop, "capture_screenshot", side_effect=capture)
+ )
+ log = Path(tmp) / "recorder.log"
+ log.write_text("log")
+ stack.enter_context(mock.patch.object(rb.ca, "RECORDER_STATE", Path(tmp)))
+ Path(tmp, "daemon.log").write_text("log")
+ if copy_error is not None:
+ stack.enter_context(
+ mock.patch.object(rb.shutil, "copyfile", side_effect=copy_error)
+ )
+
+ def run_codex(cmd, env, *args, **kwargs):
+ link = Path(env["CODEX_HOME"]) / "auth.json"
+ self.assertTrue(link.is_symlink())
+ if replace_link:
+ link.unlink()
+ link.write_text("FAKE_TOKEN")
+ return {"returncode": 0, "result_seen": True, "wall_s": 0}
+
+ stack.enter_context(mock.patch.object(rb.cxn, "run_codex", side_effect=run_codex))
+ row = rb.run_attempt(ctx, entry, 1)
+ trial = ctx.run_dir / "trials" / entry["trial_id"] / "a1"
+ link = trial / "codex-home/.codex/auth.json"
+ self.assertFalse(link.is_symlink())
+ self.assertEqual(source.read_text(), "FAKE_TOKEN")
+ mocks["sweep_processes"].assert_called()
+ mocks["kill_bench_apps"].assert_called()
+ errors = trial / "artifacts" / "screenshot-error.txt"
+ return row, (errors.read_text() if errors.is_file() else None)
+
+ def test_linux_screenshot_failure_still_unlinks_auth(self) -> None:
+ row, errors = self.linux_evidence_trial(
+ replace_link=False, screenshot_error=AttributeError("secret /path")
+ )
+ self.assertNotEqual(row["infra_failure"], "harness_exception")
+ self.assertEqual(errors, "screenshot: AttributeError")
+
+ def test_linux_log_copy_failure_still_unlinks_auth(self) -> None:
+ row, errors = self.linux_evidence_trial(
+ replace_link=False, screenshot_error=None, copy_error=OSError("secret /path")
+ )
+ self.assertNotEqual(row["infra_failure"], "harness_exception")
+ self.assertIn("recorder log: OSError", errors)
+ self.assertNotIn("secret", errors)
+
+ def test_linux_evidence_failure_keeps_credential_integrity_error(self) -> None:
+ row, _ = self.linux_evidence_trial(
+ replace_link=True, screenshot_error=AttributeError("boom")
+ )
+ self.assertEqual(row["infra_failure"], "harness_exception")
+ self.assertIn("shared ChatGPT credential symlink replaced", row["infra_message"])
+
def attempt_row(
attempt: int,
@@ -631,6 +765,11 @@ class PreflightTest(unittest.TestCase):
class ExecuteTrialTest(unittest.TestCase):
+ def setUp(self) -> None:
+ disk = mock.patch.object(rb.rec, "free_gb", return_value=100.0)
+ disk.start()
+ self.addCleanup(disk.stop)
+
def test_daemon_setup_failure_retries_with_restart_then_excludes(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
ctx = make_ctx(tmp)
@@ -757,6 +896,11 @@ class ExecuteTrialTest(unittest.TestCase):
class GateTest(unittest.TestCase):
+ def setUp(self) -> None:
+ disk = mock.patch.object(rb.rec, "free_gb", return_value=100.0)
+ disk.start()
+ self.addCleanup(disk.stop)
+
def test_quota_stop_rule_blocks_new_trials(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
ctx = make_ctx(tmp)
@@ -876,6 +1020,63 @@ class FinalizeTest(unittest.TestCase):
self.assertEqual(core.completed_trial_ids(rows), {"a"})
+class LinuxSetupTest(unittest.TestCase):
+ def test_macos_arms_are_skipped_before_setup(self) -> None:
+ import contextlib
+ import io
+ import desktop_runtime as desktop
+
+ tmp = tempfile.TemporaryDirectory()
+ self.addCleanup(tmp.cleanup)
+ args = rb.build_parser().parse_args(
+ [
+ "dry-run",
+ "--ledger",
+ str(Path(tmp.name) / "ledger.jsonl"),
+ "--bench-v2",
+ "--arms",
+ "cc-cua-driver",
+ "cc-codex-cu",
+ "cc-claude-cu-helper",
+ ]
+ )
+ output = io.StringIO()
+ with (
+ mock.patch.object(desktop, "detect_platform", return_value="linux"),
+ contextlib.redirect_stderr(output),
+ ):
+ ctx = rb.make_ctx(args)
+ self.assertEqual(ctx.arm_names, ["cc-cua-driver"])
+ self.assertIn("unsupported-platform: cc-codex-cu requires macOS", output.getvalue())
+ self.assertIn(
+ "unsupported-platform: cc-claude-cu-helper requires macOS",
+ ctx.versions["unsupported_arms"],
+ )
+
+ def test_no_supported_arms_fails_clearly(self) -> None:
+ import desktop_runtime as desktop
+
+ args = rb.build_parser().parse_args(["dry-run", "--arms", "cc-codex-cu"])
+ with (
+ mock.patch.object(desktop, "detect_platform", return_value="linux"),
+ self.assertRaisesRegex(SystemExit, "no supported arms remain"),
+ ):
+ rb.make_ctx(args)
+
+ def test_linux_pack_descriptor_selection(self) -> None:
+ import cdb_adapter
+ import desktop_runtime as desktop
+
+ with mock.patch.object(desktop, "detect_platform", return_value="linux"):
+ descriptor = cdb_adapter._descriptor({"pack_task": "shared/cdb-s02"})
+ self.assertEqual(descriptor["platform"], "linux")
+ self.assertTrue(any("libreoffice" in " ".join(a["command"]) for a in descriptor["apps"]))
+ with mock.patch.object(desktop, "detect_platform", return_value="macos"):
+ self.assertEqual(
+ cdb_adapter._descriptor({"pack_task": "shared/cdb-s02"})["platform"], "macos"
+ )
+
+
class MiscTest(unittest.TestCase):
def test_strip_bulk_removes_screenshots_but_keeps_text(self) -> None:
big = "A" * 50000
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tools/fixture.py b/libs/cua-driver-bench/automated-eval/macos_bench/tools/fixture.py
index ca74f5d6a6..0216d9556e 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/tools/fixture.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tools/fixture.py
@@ -28,6 +28,9 @@ from pathlib import Path
from typing import Any
ROOT = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(ROOT))
+import desktop_runtime as desktop
+
PROBES = ROOT / "probes"
DEFAULT_BUILD = ROOT.parent.parent / "build"
@@ -67,11 +70,11 @@ def setup(task: str, seed: int, d: Path, build: Path = DEFAULT_BUILD) -> dict[st
home.mkdir(exist_ok=True)
meta = task_meta(task)
(d / "brief.txt").write_text(render(task, seed), "utf-8")
- exe = next((build / "BenchLab.app" / "Contents" / "MacOS").iterdir())
- env = {"HOME": str(home), "PATH": "/usr/bin:/bin", "TMPDIR": str(d)}
+ argv = desktop.bench_app_argv("BenchLab", build)
+ env = desktop.desktop_env(home)
proc = subprocess.Popen(
[
- str(exe),
+ *argv,
*meta["app_args"],
"--seed",
str(seed),
@@ -130,7 +133,10 @@ def reset(task: str, d: Path) -> dict[str, Any]:
os.kill(int(p["pid"].read_text()), signal.SIGKILL)
except (OSError, ValueError):
pass
- sweep("BenchLab.app/Contents/MacOS/BenchLab")
+ if desktop.detect_platform() == "linux":
+ subprocess.run(["pkill", "-x", "BenchLab"], capture_output=True)
+ else:
+ sweep("BenchLab.app/Contents/MacOS/BenchLab")
meta = task_meta(task)
for app in meta.get("needs_apps", []):
sweep(f"/{app}.app/Contents/MacOS/{app}")
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tools/validate.py b/libs/cua-driver-bench/automated-eval/macos_bench/tools/validate.py
index 7443835623..ccdd0a10da 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/tools/validate.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tools/validate.py
@@ -393,7 +393,9 @@ def run_task(task: str, rnd: int, out: Path) -> dict[str, Any]:
clean.append(s2 == initial and len(ev) == 1)
F.reset(task, d)
left = subprocess.run(
- ["pgrep", "-f", "BenchLab.app/Contents/MacOS/BenchLab"], capture_output=True, text=True
+ ["pgrep", "-f", F.desktop.bench_app_argv("BenchLab", F.DEFAULT_BUILD)[-1]],
+ capture_output=True,
+ text=True,
).stdout.strip()
clean.append(
left == "" and not F.paths(d)["state"].exists() and not F.paths(d)["events"].exists()