Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver-bench): accept both AT-SPI button role names in the GTK interaction test (#5775)
injaneity194348f44fa1
diff --git a/.github/workflows/ci-cua-driver-bench.yml b/.github/workflows/ci-cua-driver-bench.ymlindex c2e6167341..c9df43c628 100644--- a/.github/workflows/ci-cua-driver-bench.yml+++ b/.github/workflows/ci-cua-driver-bench.yml@@ -23,7 +23,6 @@ jobs:harness:if: ${{ !startsWith(github.head_ref, 'release-please--branches--') }}name: Harness tests (no GUI, no model)- # The harness is macOS-only (sw_vers, AppKit apps); BenchLab targets macOS 26.runs-on: macos-26timeout-minutes: 20steps:@@ -45,6 +44,28 @@ jobs:uv run --no-project --python 3.12 --with pytest==8.4.2python -m pytest -q tests probes/tests swift/tests+ linux-desktop:+ name: Linux headless desktop contracts (no model)+ runs-on: ubuntu-latest+ timeout-minutes: 10+ steps:+ - name: Check out Cua+ uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5+ - name: Install desktop dependencies+ run: |+ sudo apt-get update+ sudo apt-get install -y --no-install-recommends \+ xvfb dbus-x11 at-spi2-core openbox picom python3-gi gir1.2-gtk-3.0 gir1.2-atspi-2.0 \+ xdotool x11-utils libxss1 ffmpeg lsof python3-pytest+ - name: Exercise GTK, doctor, sentinel and evidence contracts+ working-directory: libs/cua-driver-bench/automated-eval/macos_bench+ env:+ CDB_BENCH_WORK: ${{ runner.temp }}/cdb-bench-work+ run: |+ timeout 180s xvfb-run -a --server-args="-screen 0 1920x1080x24" \+ dbus-run-session -- bash -lc \+ 'openbox >/tmp/cua-openbox.log 2>&1 & picom --backend xrender --config /dev/null >/tmp/cua-picom.log 2>&1 & sleep 2; /usr/bin/python3 -m pytest -q tests/test_pilot_runner.py tests/test_runner_flow.py tests/test_bench_v2.py tests/test_analyze.py tests/test_cdb_adapter.py'+taskpack:if: ${{ !startsWith(github.head_ref, 'release-please--branches--') }}name: Task pack testsdiff --git a/libs/cua-driver-bench/automated-eval/macos_bench/.gitignore b/libs/cua-driver-bench/automated-eval/macos_bench/.gitignorenew file mode 100644index 0000000000..357b7dc03c--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/.gitignore@@ -0,0 +1 @@+/local-work/diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/README.md b/libs/cua-driver-bench/automated-eval/macos_bench/README.mdindex dd86d19417..1f373f6ffe 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/README.md+++ b/libs/cua-driver-bench/automated-eval/macos_bench/README.md@@ -1,4 +1,4 @@-# Bench v2 harness (macOS)+# Bench v2 harness (macOS and Linux/X11)`cx-browser-use-cli` is staged and unavailable for scored execution: preflight always fails.It requires `CDB_CODEX_AUTH=chatgpt` and browser-only tasks. Codex driving the CLI is not@@ -35,3 +35,50 @@ Tool-reported setup errors become infrastructure failures with `infra_pattern`;model prose alone never triggers this classification.Keep results, streams and recordings out of git.++## Linux desktop setup++Install `xvfb dbus-x11 at-spi2-core openbox picom python3-gi gir1.2-gtk-3.0+ gir1.2-atspi-2.0 xdotool x11-utils libxss1 ffmpeg lsof`, plus the applications+required by each pack task's `platform/launch.linux.json` prerequisites. Use system+Python for GTK/AT-SPI; no Swift build is required. Older non-v2 BenchLab modes are+explicitly unsupported on Linux. Codex Sky and Claude Desktop arms are skipped+with an `unsupported-platform` reason before any MCP or daemon setup.++From this directory, using an explicitly selected Linux driver and matching skills:++```sh+export CDB_BENCH_WORK="$PWD/local-work"+xvfb-run -a --server-args="-screen 0 1920x1080x24" \+ dbus-run-session -- bash -lc \+ 'openbox & picom --backend xrender --config /dev/null & sleep 2;+ python3 run_bench.py preflight --bench-v2 --arms cc-cua-driver \+ --cua-bin "$CUA_DRIVER_BIN" --cua-skills "$CUA_SKILLS" --pins "$LINUX_PINS"'+```++The pin file must declare `platform: "linux"`, `cua_driver_version`,+`cua_driver_binary_sha256`, and `cua_skills_tree_sha256`, matching the selected+executable and skill tree. Main/script arms additionally require the existing+`cua_main` version, binary, git and skills identities. Never reuse the macOS pin+file. Existing model, quota, disk-space and pack-integrity gates still apply.+Keep `local-work/` outside git; the disposable-environment flag is still required+for pack cleanup.++`desktop_runtime.py` owns environment allowlisting, process groups, app command+resolution, driver launch/readiness, health parsing, doctor and screenshots.+Doctor connects to X11 and the AT-SPI registry and checks driver capabilities and+ffmpeg's x11grab support. Linux uses the existing `CuaRecorder` on its dedicated+control daemon, whose backend already uses x11grab; that daemon also handles+window placement. No separate Linux recording implementation is used.++The GTK helpers preserve BenchLab's state/event evaluator contracts. The sentinel+observes EWMH focus/stacking/workspaces, AT-SPI focus, root pointer coordinates,+clipboard ownership and transient windows. It never refocuses itself after arming.+XScreenSaver idle values include synthetic input: Linux HID counts and human-input+inference are **unmeasured**, not zero. Screenshots and app/sentinel/driver logs+are retained in trial evidence directories. Raw evidence and results are internal.++Headless contract tests use independent `xdotool` XTest actions. The Linux CI job+runs doctor, GTK interactions, sentinel and screenshot tests without a driver;+only driver health and recording smoke requires `CUA_DRIVER_BIN`. No model is+called by these tests.diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/analyze.py b/libs/cua-driver-bench/automated-eval/macos_bench/analyze.pyindex 6ec15cd9dc..5f7fde5769 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/analyze.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/analyze.py@@ -444,7 +444,10 @@ def validate_row(row: Any, where: str) -> None:f"{where}: field 'disturbance.available' must be true or false, got {_short(dist.get('available'))}")if dist["available"]:- if not isinstance(dist.get("human_input_suspected"), bool):+ unmeasured_hid = row.get("platform") == "linux" and dist.get("hid_events") is None+ if not isinstance(dist.get("human_input_suspected"), bool) and not (+ unmeasured_hid and dist.get("human_input_suspected") is None+ ):raise TrialFormatError(f"{where}: field 'disturbance.human_input_suspected' must be true or false when "f"disturbance.available is true, got {_short(dist.get('human_input_suspected'))}"@@ -460,12 +463,13 @@ def validate_row(row: Any, where: str) -> None:_require_num(where, dist, key, label=f"disturbance.{key}")_optional_num(where, dist, "scrolls_leaked", label="disturbance.scrolls_leaked")hid = dist.get("hid_events")- if not isinstance(hid, dict):+ if not isinstance(hid, dict) and not unmeasured_hid:raise TrialFormatError(f"{where}: field 'disturbance.hid_events' must be an object when disturbance.available is true, got {_short(hid)}")- for key in HID_KEYS:- _require_num(where, hid, key, label=f"disturbance.hid_events.{key}")+ if not unmeasured_hid:+ for key in HID_KEYS:+ _require_num(where, hid, key, label=f"disturbance.hid_events.{key}")def _validate_rows(rows: Sequence[Any], labels: Sequence[str]) -> None:@@ -864,16 +868,17 @@ def _latency(trials: list[Trial], arms: Sequence[str]) -> dict[str, Any]:return out-def _any_disturbance(d: dict[str, Any]) -> bool:+def _any_disturbance(d: dict[str, Any]) -> bool | None:hid = d["hid_events"]- return bool(+ observed = bool(d["front_changes"] > 0or d["key_loss"] > 0or d["keystrokes_leaked"] > 0or d["clicks_leaked"] > 0- or any(hid[k] > 0 for k in HID_KEYS)+ or (hid is not None and any(hid[k] > 0 for k in HID_KEYS))or d["pointer_deviation_episodes"] > 0)+ return observed if observed or hid is not None else Nonedef _disturbance(trials: list[Trial], arms: Sequence[str]) -> dict[str, Any]:@@ -885,24 +890,33 @@ def _disturbance(trials: list[Trial], arms: Sequence[str]) -> dict[str, Any]:used = [t for t in available if not t.disturbance["human_input_suspected"]]def series(fn) -> list[float]:- return [float(fn(t.disturbance)) for t in used]+ values = [fn(t.disturbance) for t in used]+ return [float(value) for value in values if value is not None]fields = {"front_changes": series(lambda d: d["front_changes"]),"key_loss": series(lambda d: d["key_loss"]),"keystrokes_leaked": series(lambda d: d["keystrokes_leaked"]),"clicks_leaked": series(lambda d: d["clicks_leaked"]),- "hid_events_total": series(lambda d: sum(d["hid_events"][k] for k in HID_KEYS)),+ "hid_events_total": series(+ lambda d: sum(d["hid_events"][k] for k in HID_KEYS)+ if d["hid_events"] is not None+ else None+ ),"pointer_max_deviation_px": series(lambda d: d["pointer_max_deviation_px"]),"pointer_deviation_episodes": series(lambda d: d["pointer_deviation_episodes"]),}for key in HID_KEYS:- fields[f"hid_events_{key}"] = series(lambda d, key=key: d["hid_events"][key])+ fields[f"hid_events_{key}"] = series(+ lambda d, key=key: d["hid_events"][key] if d["hid_events"] is not None else None+ )if all(_is_num(t.disturbance.get("scrolls_leaked")) for t in used) and used:fields["scrolls_leaked"] = series(lambda d: d["scrolls_leaked"])- any_count = sum(1 for t in used if _any_disturbance(t.disturbance))- interval = wilson_interval(any_count, len(used))+ measured = all(t.disturbance["hid_events"] is not None for t in used)+ any_count = sum(_any_disturbance(t.disturbance) for t in used) if measured else None+ any_n = len(used) if measured else None+ interval = wilson_interval(any_count, any_n) if measured else Noneout[arm] = {"n_non_excluded": len(kept),"n_available": len(available),@@ -918,9 +932,10 @@ def _disturbance(trials: list[Trial], arms: Sequence[str]) -> dict[str, Any]:for name, vals in fields.items()},"any_disturbance": {+ "measured": measured,"count": any_count,- "n": len(used),- "fraction": any_count / len(used) if used else None,+ "n": any_n,+ "fraction": any_count / any_n if any_n else None,"wilson_low": interval[0] if interval else None,"wilson_high": interval[1] if interval else None,},diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/bench_v2.py b/libs/cua-driver-bench/automated-eval/macos_bench/bench_v2.pyindex 7f2def75db..94c746a59c 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/bench_v2.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/bench_v2.py@@ -103,7 +103,20 @@ def validate_tasks(# ------------------------------------------------------------------ GUI-only+LINUX_TERMINALS = (+ "xterm",+ "gnome-terminal",+ "gnome-terminal-server",+ "gnome-terminal-",+ "konsole",+ "xfce4-terminal",+ "kitty",+ "alacritty",+ "Alacritty",+ "org.gnome.Terminal",+)SIDE_DOOR_BUNDLES = {+ **{name: name for name in LINUX_TERMINALS},"com.apple.Terminal": "Terminal","com.googlecode.iterm2": "iTerm","com.apple.ScriptEditor2": "Script Editor",@@ -236,7 +249,9 @@ def terminal_session_ttys(rows: list[tuple[int, int, str, str]], exclude_pid: innever reads, and any tty its descendants open (agents, MCP servers, apps it starts)."""parent = {pid: ppid for pid, ppid, _, _ in rows}tty_of = {pid: tty for pid, _, tty, _ in rows}- terminals = {pid for pid, _, _, comm in rows if comm.rsplit("/", 1)[-1] == "Terminal"}+ terminals = {+ pid for pid, _, _, comm in rows if comm.rsplit("/", 1)[-1] in ("Terminal", *LINUX_TERMINALS)+ }def ancestors(pid: int) -> set[int]:seen: set[int] = set()@@ -246,9 +261,9 @@ def terminal_session_ttys(rows: list[tuple[int, int, str, str]], exclude_pid: inreturn seendef device(tty: str) -> str | None:- if tty in ("??", "-", ""):+ if tty in ("??", "?", "-", ""):return None- return "/dev/" + (tty if tty.startswith("tty") else "tty" + tty)+ return "/dev/" + (tty if tty.startswith(("tty", "pts/")) else "tty" + tty)excluded = {device(tty_of.get(p, "")) for p in ancestors(exclude_pid)} | {device(tty_of.get(exclude_pid, ""))diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/cdb_adapter.py b/libs/cua-driver-bench/automated-eval/macos_bench/cdb_adapter.pyindex 0ef695769c..18d731584d 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/cdb_adapter.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/cdb_adapter.py@@ -28,7 +28,6 @@ import jsonimport osimport reimport shutil-import signalimport subprocessimport sysimport time@@ -37,6 +36,7 @@ from pathlib import Pathfrom typing import Anyimport claude_arms as ca+import desktop_runtime as desktopSKIP_KINDS = {"terminal", "editor"}EVAL_ENTRY = "/usr/local/libexec/cdb-eval-run"@@ -87,7 +87,9 @@ def node_facts(node: str = "/usr/local/bin/node") -> dict[str, str]:def _descriptor(spec: dict[str, Any]) -> dict[str, Any]:- path = pack_tasks_root() / spec["pack_task"] / "platform/launch.macos.json"+ path = (+ pack_tasks_root() / spec["pack_task"] / f"platform/launch.{desktop.detect_platform()}.json"+ )return json.loads(path.read_text("utf-8"))@@ -231,6 +233,13 @@ def with_force_accessibility(if kind == "electron":return [*argv, flag]if kind == "browser":+ if desktop.detect_platform() == "linux" and Path(argv[0]).name in (+ "google-chrome",+ "google-chrome-stable",+ "chromium",+ "chromium-browser",+ ):+ return [argv[0], flag, *argv[1:]]for i, part in enumerate(argv):if part.endswith(CHROME_EXE):return [*argv[: i + 1], flag, *argv[i + 1 :]]@@ -242,10 +251,14 @@ def kill_policy(spec: dict[str, Any]) -> tuple[list[str], list[int]]:text = json.dumps(_descriptor(spec))names: list[str] = []if "electron" in text:- names.append("Electron")+ names.append("Electron" if desktop.detect_platform() == "macos" else "electron")if "Google Chrome" in text:names.append("Google Chrome")- if "soffice" in text:+ if desktop.detect_platform() == "linux" and any(+ name in text for name in ("google-chrome", "chromium")+ ):+ names += ["chrome", "chromium"]+ if "soffice" in text or "libreoffice" in text:names += ["soffice", "soffice.bin"]if "gnucash" in text.lower():names += ["gnucash", "Gnucash"]@@ -263,7 +276,9 @@ def clear_leftovers(spec: dict[str, Any]) -> None:subprocess.run(["pkill", "-x", name], capture_output=True)for port in ports:pids = subprocess.run(- ["/usr/sbin/lsof", "-ti", f"tcp:{port}", "-sTCP:LISTEN"], capture_output=True, text=True+ [desktop.tool("lsof"), "-ti", f"tcp:{port}", "-sTCP:LISTEN"],+ capture_output=True,+ text=True,).stdout.split()for pid in pids:subprocess.run(["kill", "-9", pid], capture_output=True)@@ -277,9 +292,7 @@ class CdbTask:self.spec = specself.seed = seedself.bundle = pack_tasks_root() / spec["pack_task"]- self.descriptor = json.loads(- (self.bundle / "platform/launch.macos.json").read_text("utf-8")- )+ self.descriptor = _descriptor(spec)self.artifacts = artifactsself.workspace = Path(self.descriptor["workspace_root"].replace("${HOME}", str(Path.home()))@@ -309,7 +322,9 @@ class CdbTask:self, command: list[str], timeout: int = 300, **extra: str) -> subprocess.CompletedProcess[str]:argv = [self.sub(part, **extra) for part in command]- env = {"HOME": str(Path.home()), "PATH": APP_PATH, "LANG": "en_US.UTF-8"}+ env = desktop.desktop_env(+ Path.home(), {"PATH": APP_PATH} if desktop.detect_platform() == "macos" else None+ )return subprocess.run(argv, cwd=str(self.bundle), env=env, capture_output=True, text=True, timeout=timeout)@@ -336,11 +351,19 @@ class CdbTask:def start_apps(self, windows: Any = None, force_accessibility: bool = False) -> None:for app in self.apps():- env = {"HOME": str(Path.home()), "PATH": APP_PATH, "LANG": "en_US.UTF-8"}+ env = desktop.desktop_env(+ Path.home(), {"PATH": APP_PATH} if desktop.detect_platform() == "macos" else None+ )env.update({k: self.sub(v) for k, v in (app.get("env") or {}).items()})cwd = self.sub(app["cwd"]) if app.get("cwd") else str(self.bundle)argv = [self.sub(p) for p in app["command"]]- if app.get("kind") == "electron" and os.environ.get("CDB_ELECTRON_SHARED") != "1":+ for key in app.get("unset_env", []):+ env.pop(key, None)+ if (+ desktop.detect_platform() == "macos"+ and app.get("kind") == "electron"+ and os.environ.get("CDB_ELECTRON_SHARED") != "1"+ ):argv = electron_argv(argv, cwd, app_display_name(app))self.log.append(f"electron {app['id']} as {electron_bundle_id(app_display_name(app))}"@@ -350,16 +373,16 @@ class CdbTask:if flagged != argv:self.log.append(f"{app['id']}: {ca.FORCE_ACCESSIBILITY_FLAG}")argv = flagged- out = (self.artifacts / f"app-{app['id']}.log").open("ab")- proc = subprocess.Popen(- argv,- cwd=cwd,- env=env,- stdin=subprocess.DEVNULL,- stdout=out,- stderr=out,- start_new_session=True,- )+ with (self.artifacts / f"app-{app['id']}.log").open("ab") as out:+ proc = subprocess.Popen(+ argv,+ cwd=cwd,+ env=env,+ stdin=subprocess.DEVNULL,+ stdout=out,+ stderr=out,+ start_new_session=True,+ )self.procs.append(proc)ready = (app.get("ready") or {}).get("http")if ready:@@ -456,12 +479,8 @@ class CdbTask:def stop_apps(self) -> None:for proc in reversed(self.procs):- for sig in (signal.SIGTERM, signal.SIGKILL):- try:- os.killpg(proc.pid, sig)- except (ProcessLookupError, PermissionError):- break- time.sleep(0.4)+ desktop.stop_group(proc.pid)+ proc.wait(timeout=5)self.procs.clear()clear_leftovers(self.spec)diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/claude_arms.py b/libs/cua-driver-bench/automated-eval/macos_bench/claude_arms.pyindex f6d6e15d69..bdceb47b5d 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/claude_arms.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/claude_arms.py@@ -32,6 +32,8 @@ from dataclasses import dataclassfrom pathlib import Pathfrom typing import Any+import desktop_runtime as desktop+HERE = Path(__file__).resolve().parent# WORK holds everything the harness creates or downloads: the private Cua Driver install, the Codex MCP config,# the spend ledger, the HOLD file and the run folders. CDB_BENCH_WORK overrides it.@@ -119,6 +121,7 @@ class CuaBuild:tuple[str, str], ...] = () # Amendment 6: extra daemon + MCP env (the script flag)prompt_addendum: str | None = None # Amendment 6: text appended to the system prompt+ executable: Path | None = None@propertydef binary_pin(self) -> str:@@ -128,7 +131,7 @@ class CuaBuild:@propertydef bin(self) -> Path:- return self.app / "Contents/MacOS/cua-driver"+ return self.executable or self.app / "Contents/MacOS/cua-driver"@propertydef home(self) -> Path:@@ -376,12 +379,7 @@ def load_pins() -> dict[str, Any]:def cua_env(home: Path) -> dict[str, str]:"""Environment of the Cua Driver daemon and of its MCP client: own HOME, telemetry off."""- return {- "HOME": str(home),- "PATH": "/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin",- "CUA_DRIVER_RS_TELEMETRY_ENABLED": "false",- "DO_NOT_TRACK": "1", # the main build's shared telemetry crate also honours this (Amendment 3)- }+ return desktop.desktop_env(home)def resolve_codex_cu_config(explicit: Path | None = None) -> Path | None:@@ -580,15 +578,7 @@ def cua_health(socket: str = AGENT_SOCKET, home: Path | None = None, binary: Path | None = None) -> dict[str, Any]:"""Daemon-reported build (version, exe sha256, git sha) from the live daemon via health_report."""- try:- done = cua_cli(- "call", "health_report", "{}", socket=socket, home=home, timeout=30, binary=binary- )- data = json.loads(done.stdout[done.stdout.index("{") :])- checks = {c.get("name"): c.get("status") for c in data.get("checks", [])}- return {"ok": done.returncode == 0, "build": data.get("build", {}), "checks": checks}- except (OSError, ValueError, subprocess.TimeoutExpired) as error:- return {"ok": False, "error": f"{type(error).__name__}: {error}"}+ return desktop.cua_health(binary or CUA_BIN, socket, home or CUA_STATE / "home")def stop_cua_daemon(socket: str, home: Path, binary: Path | None = None) -> None:@@ -609,36 +599,14 @@ def start_cua_daemon(env_extra: dict[str, str] | None = None,) -> subprocess.Popen[bytes]:"""Private daemon of the release under test: own socket, own HOME, telemetry off."""- binary = binary or CUA_BIN- home = state_dir / "home"- home.mkdir(parents=True, exist_ok=True)- try:- os.unlink(socket)- except OSError:- pass- args = [str(binary), "serve", "--socket", socket, "--dangerously-bypass-approvals"]- if not overlay:- args.append("--no-overlay")- log = (state_dir / log_name).open("ab")- proc = subprocess.Popen(- args,- env={**cua_env(home), **(env_extra or {})},- stdin=subprocess.DEVNULL,- stdout=log,- stderr=log,- start_new_session=True,+ return desktop.start_driver(+ binary or CUA_BIN,+ socket,+ state_dir,+ overlay=overlay,+ log_name=log_name,+ env_extra=env_extra,)- deadline = time.monotonic() + 30- while time.monotonic() < deadline:- if proc.poll() is not None:- raise RuntimeError(- f"Cua Driver daemon exited early (rc={proc.returncode}); see {state_dir / log_name}"- )- done = _run([str(binary), "status", "--socket", socket], env=cua_env(home), timeout=10)- if done.returncode == 0:- return proc- time.sleep(0.5)- raise RuntimeError("Cua Driver daemon did not become ready")# ---------------------------------------------------------------- pins (observed values)@@ -682,8 +650,11 @@ def observed_pins(include_codex: bool = True, include_cua: bool = True) -> dict[else None)out["claude_code_version"] = claude- out["macos_version"] = _run(["sw_vers", "-productVersion"]).stdout.strip()- out["macos_build"] = _run(["sw_vers", "-buildVersion"]).stdout.strip()+ if desktop.detect_platform() == "macos":+ out["macos_version"] = _run(["sw_vers", "-productVersion"]).stdout.strip()+ out["macos_build"] = _run(["sw_vers", "-buildVersion"]).stdout.strip()+ else:+ out["platform"] = "linux"if include_cua:out["cua_driver_version_string"] = cua_version_string()out["cua_driver_version"] = out["cua_driver_version_string"].replace("cua-driver ", "")diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/claude_driver.py b/libs/cua-driver-bench/automated-eval/macos_bench/claude_driver.pyindex 1b6c5f1d01..8abeacbbbc 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/claude_driver.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/claude_driver.py@@ -23,19 +23,11 @@ from pathlib import Pathfrom typing import Anyimport claude_arms as ca+from desktop_runtime import stop_group as kill_groupELICIT_APP = re.compile(r'Allow Computer Use to use "(.+)"\?')-def kill_group(pid: int, grace: float = 1.0) -> None:- for sig in (signal.SIGTERM, signal.SIGKILL):- try:- os.killpg(pid, sig)- except OSError:- return- time.sleep(grace)--def answer_elicitation(message: str, allowed_apps: set[str]) -> dict[str, Any]:"""Decision for one MCP elicitation request. Only an exact `Allow Computer Use to use "<App>"?`for an app on the task's allowlist is accepted."""diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/desktop_runtime.py b/libs/cua-driver-bench/automated-eval/macos_bench/desktop_runtime.pynew file mode 100644index 0000000000..88741d09cd--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/desktop_runtime.py@@ -0,0 +1,273 @@+# SPDX-License-Identifier: FSL-1.1-MIT+# Copyright (c) 2026 Cua AI, Inc.++from __future__ import annotations++import json+import os+import platform+import re+import shutil+import signal+import subprocess+import time+from dataclasses import dataclass+from pathlib import Path+from typing import Literal, Mapping, Sequence++Platform = Literal["macos", "linux"]+HERE = Path(__file__).resolve().parent+SESSION_KEYS = (+ "DISPLAY",+ "XAUTHORITY",+ "DBUS_SESSION_BUS_ADDRESS",+ "XDG_RUNTIME_DIR",+ "XDG_DATA_DIRS",+ "AT_SPI_BUS_ADDRESS",+ "GTK_MODULES",+ "QT_ACCESSIBILITY",+ "QT_LINUX_ACCESSIBILITY_ALWAYS_ON",+ "NO_AT_BRIDGE",+)+++@dataclass(frozen=True)+class Check:+ name: str+ status: Literal["pass", "fail", "skip"]+ detail: str+++def detect_platform(system: str | None = None) -> Platform:+ name = system if system is not None else platform.system()+ if name == "Darwin":+ return "macos"+ if name == "Linux":+ return "linux"+ raise ValueError(f"unsupported-platform: {name}")+++def tool(name: str) -> str:+ found = shutil.which(name)+ if found:+ return found+ if detect_platform() == "macos":+ fallback = Path("/opt/homebrew/bin") / name+ if fallback.is_file():+ return str(fallback)+ raise FileNotFoundError(f"Install {name} and add it to PATH")+++def desktop_env(home: Path, extra: Mapping[str, str] | None = None) -> dict[str, str]:+ linux = detect_platform() == "linux"+ env = {+ "HOME": str(home),+ "TMPDIR": str(home / "tmp")+ if (home / "tmp").is_dir()+ else os.environ.get("TMPDIR", "/tmp"),+ "PATH": os.environ.get("PATH", "/usr/local/bin:/usr/bin:/bin")+ if linux+ else "/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin",+ "USER": os.environ.get("USER", ""),+ "LANG": os.environ.get("LANG", "C.UTF-8") if linux else "en_US.UTF-8",+ "CUA_DRIVER_RS_TELEMETRY_ENABLED": "false",+ "DO_NOT_TRACK": "1",+ }+ if linux:+ env.update({key: os.environ[key] for key in SESSION_KEYS if key in os.environ})+ # HOME is private per trial; keep the session's default X authority reachable.+ authority = Path(os.path.expanduser("~")) / ".Xauthority"+ if "XAUTHORITY" not in env and authority.is_file():+ env["XAUTHORITY"] = str(authority)+ env.update(extra or {})+ return env+++def bench_app_argv(name: str, build_dir: Path | None = None) -> list[str]:+ name = name.removesuffix(".app")+ if name not in ("BenchLab", "BenchSentinel"):+ raise ValueError(f"unknown bench app: {name}")+ if detect_platform() == "linux":+ script = HERE / "linux" / ("benchlab.py" if name == "BenchLab" else "sentinel.py")+ if not script.is_file():+ raise FileNotFoundError(script)+ return ["/usr/bin/python3", str(script)]+ if build_dir is None:+ raise ValueError("--build-dir is required on macOS")+ exes = sorted((build_dir / f"{name}.app" / "Contents/MacOS").iterdir())+ return [str(next(p for p in exes if p.is_file() and os.access(p, os.X_OK)))]+++def stop_group(pid: int) -> None:+ for sig in (signal.SIGTERM, signal.SIGKILL):+ try:+ os.killpg(pid, sig)+ except OSError:+ return+ if sig == signal.SIGTERM:+ time.sleep(0.3)+++def start_app(+ argv: Sequence[str], home: Path, log: Path, *, env: Mapping[str, str] | None = None+) -> subprocess.Popen:+ log.parent.mkdir(parents=True, exist_ok=True)+ (home / "tmp").mkdir(parents=True, exist_ok=True)+ with log.open("ab") as output:+ proc = subprocess.Popen(+ list(argv),+ env=desktop_env(home, env),+ stdin=subprocess.DEVNULL,+ stdout=output,+ stderr=output,+ start_new_session=True,+ )+ time.sleep(0.15)+ if proc.poll() is not None:+ stop_group(proc.pid)+ proc.wait()+ raise RuntimeError(f"{argv[0]} exited ({proc.returncode}); see {log}")+ return proc+++def start_driver(+ binary: Path,+ socket: str,+ state: Path,+ *,+ overlay: bool = True,+ log_name: str = "daemon.log",+ env_extra: Mapping[str, str] | None = None,+) -> subprocess.Popen:+ Path(socket).unlink(missing_ok=True)+ args = [str(binary), "serve", "--socket", socket, "--dangerously-bypass-approvals"]+ if not overlay:+ args.append("--no-overlay")+ proc = start_app(args, state / "home", state / log_name, env=env_extra)+ try:+ deadline = time.monotonic() + 30+ while time.monotonic() < deadline:+ if proc.poll() is not None:+ raise RuntimeError(f"driver exited ({proc.returncode}); see {state / log_name}")+ done = subprocess.run(+ [str(binary), "status", "--socket", socket],+ env=desktop_env(state / "home", env_extra),+ capture_output=True,+ timeout=5,+ )+ if done.returncode == 0:+ return proc+ time.sleep(0.3)+ raise TimeoutError(f"driver not ready; see {state / log_name}")+ except BaseException:+ stop_group(proc.pid)+ proc.wait(timeout=5)+ raise+++def cua_health(binary: Path, socket: str, home: Path) -> dict:+ try:+ done = subprocess.run(+ [str(binary), "--socket", socket, "call", "health_report", "{}"],+ env=desktop_env(home),+ capture_output=True,+ text=True,+ timeout=30,+ )+ data = json.loads(done.stdout[done.stdout.index("{") :])+ return {+ "ok": done.returncode == 0,+ "build": data.get("build", {}),+ "checks": {c.get("name"): c.get("status") for c in data.get("checks", [])},+ }+ except (OSError, ValueError, subprocess.TimeoutExpired) as error:+ return {"ok": False, "error": f"{type(error).__name__}: {error}"}+++def capabilities_ok(health: dict) -> bool:+ keys = (+ ("ax_capability", "screen_capture_capability")+ if detect_platform() == "linux"+ else ("tcc_accessibility", "tcc_screen_recording")+ )+ return bool(health.get("ok")) and all(+ health.get("checks", {}).get(key) == "pass" for key in keys+ )+++def doctor(binary: Path, socket: str, home: Path) -> list[Check]:+ checks = []+ if detect_platform() == "linux":+ probes = {+ "display": (+ ["xdpyinfo", "-display", os.environ.get("DISPLAY", "")],+ "Start Xvfb/X11 and set DISPLAY",+ ),+ "at-spi": (+ [+ "/usr/bin/python3",+ "-c",+ "import gi; gi.require_version('Atspi', '2.0'); from gi.repository import Atspi, Gio; bus=Gio.bus_get_sync(Gio.BusType.SESSION, None); address=bus.call_sync('org.a11y.Bus', '/org/a11y/bus', 'org.a11y.Bus', 'GetAddress', None, None, Gio.DBusCallFlags.NONE, 3000, None).unpack()[0]; ax=Gio.DBusConnection.new_for_address_sync(address, Gio.DBusConnectionFlags.AUTHENTICATION_CLIENT | Gio.DBusConnectionFlags.MESSAGE_BUS_CONNECTION, None, None); ax.call_sync('org.a11y.atspi.Registry', '/org/a11y/atspi/registry', 'org.freedesktop.DBus.Introspectable', 'Introspect', None, None, Gio.DBusCallFlags.NONE, 3000, None); d=Atspi.get_desktop(0); assert d is not None; d.get_child_count()",+ ],+ "Start dbus-run-session with at-spi2-core installed",+ ),+ "ffmpeg": (+ ["ffmpeg", "-hide_banner", "-devices"],+ "Install ffmpeg with x11grab support",+ ),+ }+ for name, (argv, remedy) in probes.items():+ try:+ done = subprocess.run(+ argv, env=desktop_env(home), capture_output=True, text=True, timeout=10+ )+ ok = done.returncode == 0 and (+ name != "ffmpeg" or "x11grab" in done.stdout + done.stderr+ )+ checks.append(Check(name, "pass" if ok else "fail", "usable" if ok else remedy))+ except (OSError, subprocess.TimeoutExpired) as error:+ checks.append(Check(name, "fail", f"{remedy}: {error}"))+ health = cua_health(binary, socket, home)+ checks.append(+ Check("driver", "pass" if capabilities_ok(health) else "fail", json.dumps(health))+ )+ return checks+++def capture_screenshot(display: str, destination: Path) -> Path:+ destination.parent.mkdir(parents=True, exist_ok=True)+ if detect_platform() == "macos":+ subprocess.run(["/usr/sbin/screencapture", "-x", str(destination)], check=True, timeout=20)+ return destination+ info = subprocess.run(+ [tool("xdpyinfo"), "-display", display],+ capture_output=True,+ text=True,+ check=True,+ timeout=5,+ )+ dimensions = re.search(r"dimensions:\s+(\d+x\d+)", info.stdout)+ if dimensions is None:+ raise RuntimeError(f"cannot determine X11 display dimensions: {display}")+ with destination.with_suffix(".stderr").open("ab") as log:+ subprocess.run(+ [+ tool("ffmpeg"),+ "-nostdin",+ "-y",+ "-f",+ "x11grab",+ "-video_size",+ dimensions[1],+ "-i",+ display,+ "-frames:v",+ "1",+ str(destination),+ ],+ stdout=subprocess.DEVNULL,+ stderr=log,+ check=True,+ timeout=20,+ )+ return destinationdiff --git a/libs/cua-driver-bench/automated-eval/macos_bench/linux/benchlab.py b/libs/cua-driver-bench/automated-eval/macos_bench/linux/benchlab.pynew file mode 100644index 0000000000..de50f715ef--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/linux/benchlab.py@@ -0,0 +1,466 @@+# SPDX-License-Identifier: FSL-1.1-MIT+# Copyright (c) 2026 Cua AI, Inc.++from __future__ import annotations++import argparse+import ctypes+import json+import os+import sys+import time+from pathlib import Path++import gi++gi.require_version("Gtk", "3.0")+gi.require_version("Gdk", "3.0")+from gi.repository import Gdk, GLib, Gtk++sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "probes" / "_common"))+import benchlab_common as C++MODES = ("listdrag", "hover", "tooltip", "irmodal", "irbanner", "irunsaved", "irconsent")+++class Lab:+ def __init__(self, args: argparse.Namespace) -> None:+ ctypes.CDLL(None).prctl(15, b"BenchLab", 0, 0, 0)+ GLib.set_prgname("BenchLab")+ GLib.set_application_name("BenchLab")+ self.args = args+ self.state = {"mode": args.mode, "seed": args.seed, "seq": 0}+ self.params = getattr(C, "derive_" + args.mode)(args.seed)+ self.fields = {}+ self.loading = False+ args.events.parent.mkdir(parents=True, exist_ok=True)+ self.events = args.events.open("w", buffering=1)+ self.window = Gtk.Window(title=f"BenchLab — {args.mode}")+ self.window.set_wmclass("BenchLab", "BenchLab")+ self.window.set_default_size(760, 540)+ self.window.connect("destroy", Gtk.main_quit)+ self.box = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=12, margin=24)+ self.window.add(self.box)+ getattr(self, args.mode)()+ self.window.show_all()+ for widget in getattr(self, "hidden", []):+ widget.hide()+ self.emit("app_start", mode=args.mode, seed=args.seed)++ def emit(self, event: str, **details: object) -> None:+ self.state["seq"] += 1+ self.events.write(+ json.dumps(+ {+ "seq": self.state["seq"],+ "t": time.time() * 1000,+ "type": event,+ "details": details,+ }+ )+ + "\n"+ )+ self.args.state.parent.mkdir(parents=True, exist_ok=True)+ tmp = self.args.state.with_suffix(".tmp")+ tmp.write_text(json.dumps(self.state))+ os.replace(tmp, self.args.state)++ def label(self, text: str, box: Gtk.Box | None = None) -> Gtk.Label:+ label = Gtk.Label(label=text)+ (box or self.box).pack_start(label, False, False, 0)+ return label++ def button(self, text: str, callback, box: Gtk.Box | None = None) -> Gtk.Button:+ button = Gtk.Button(label=text)+ button.connect("clicked", lambda *_: callback())+ (box or self.box).pack_start(button, False, False, 0)+ return button++ def field(self, name: str, choices: list[str] | None = None, value: str = ""):+ self.label(name.capitalize())+ if choices:+ widget = Gtk.ComboBoxText()+ for item in choices:+ widget.append_text(item)+ widget.set_active(choices.index(value) if value in choices else 0)+ else:+ widget = Gtk.Entry()+ widget.set_text(value)+ widget.get_accessible().set_name(name.capitalize())+ self.fields[name] = widget+ widget.connect("changed", lambda *_: self.edit(name))+ self.box.pack_start(widget, False, False, 0)+ return widget++ def values(self) -> dict:+ return {+ key: widget.get_active_text()+ if isinstance(widget, Gtk.ComboBoxText)+ else widget.get_active()+ if isinstance(widget, Gtk.CheckButton)+ else widget.get_text()+ for key, widget in self.fields.items()+ }++ def edit(self, name: str) -> None:+ if self.loading:+ return+ values = self.values()+ self.state["fields"] = values+ if self.args.mode == "irunsaved":+ self.state["dirty"] = True+ self.emit("field_edit", field=name, value=values[name])+ if self.args.mode == "irbanner" and not self.state["banner_shown"]:+ self.state["banner_shown"] = True+ GLib.timeout_add(200, self.show_banner)+ if self.args.mode == "irconsent" and not self.state["consent_shown"]:+ self.state["consent_shown"] = True+ GLib.timeout_add(250, self.show_consent)++ def dialog(self, title: str, buttons: list[tuple[str, str]], callback) -> None:+ dialog = Gtk.Dialog(title=title, transient_for=self.window, modal=True)+ dialog.set_focus_on_map(False)+ dialog.get_content_area().add(Gtk.Label(label=title, margin=24))+ for index, (text, _) in enumerate(buttons):+ dialog.add_button(text, index)++ def response(widget, code):+ if not 0 <= code < len(buttons):+ return # a window-manager close or other non-button response is not an answer+ widget.destroy()+ callback(buttons[code][1])++ dialog.connect("response", response)+ # Window-manager close is not an answer; keep the dialog until a button is used.+ dialog.connect("delete-event", lambda *_: True)+ dialog.show_all()++ def listdrag(self) -> None:+ self.state.update(+ order=list(self.params["initial"]),+ move_count=0,+ drag_begins=0,+ done=False,+ done_count=0,+ )+ self.label("Drag the rows to reorder them, then press Done")+ self.rows = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=6)+ self.box.pack_start(self.rows, True, True, 0)+ self.drag_item = None+ self.draw_rows()+ self.button("Done", self.list_done)++ def draw_rows(self) -> None:+ for child in self.rows.get_children():+ child.destroy()+ targets = [Gtk.TargetEntry.new("text/plain", Gtk.TargetFlags.SAME_APP, 0)]+ for index, item in enumerate(self.state["order"]):+ row = Gtk.EventBox()+ row.add(Gtk.Label(label=item, margin=12))+ row.drag_source_set(+ Gdk.ModifierType.BUTTON1_MASK, targets, Gdk.DragAction.COPY | Gdk.DragAction.MOVE+ )+ row.drag_dest_set(+ Gtk.DestDefaults.ALL, targets, Gdk.DragAction.COPY | Gdk.DragAction.MOVE+ )+ row.connect("drag-begin", self.drag_begin, item)+ row.connect(+ "drag-data-get", lambda w, ctx, data, *_: data.set_text(self.drag_item or "", -1)+ )+ row.connect("drag-data-received", self.drop, index)+ self.rows.pack_start(row, True, True, 0)+ self.rows.show_all()++ def drag_begin(self, widget, context, item: str) -> None:+ self.drag_item = item+ self.state["drag_begins"] += 1+ self.emit("drag_begin", item=item)++ def drop(self, widget, context, x, y, data, info, timestamp, index: int) -> None:+ item = data.get_text()+ if item != self.drag_item or item not in self.state["order"]:+ Gtk.drag_finish(context, False, False, timestamp)+ return+ order = self.state["order"]+ source = order.index(item)+ order.pop(source)+ order.insert(index, item)+ self.state["move_count"] += 1+ self.emit("reorder", **{"from": source, "to": index, "item": item, "order": list(order)})+ Gtk.drag_finish(context, True, False, timestamp)+ self.drag_item = None+ GLib.idle_add(self.draw_rows)++ def list_done(self) -> None:+ self.state.update(+ done=True, done_count=self.state["done_count"] + 1, done_order=list(self.state["order"])+ )+ self.emit("done_click", order=self.state["done_order"])++ def hover(self) -> None:+ self.state.update(+ overlay_names=self.params["names"], clicks=[], hover_enter_count=0, hover_exit_count=0+ )+ zone = Gtk.EventBox()+ zone.add_events(Gdk.EventMask.ENTER_NOTIFY_MASK | Gdk.EventMask.LEAVE_NOTIFY_MASK)+ content = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=10)+ zone.add(content)+ self.label("Hover here to reveal actions", content)+ actions = Gtk.Box(spacing=10)+ content.pack_start(actions, False, False, 0)+ for name in self.params["names"]:+ self.button(name, lambda n=name: self.hover_click(n), actions)+ self.hidden = [actions]+ self.visible = False++ def crossing(widget, event, entered):+ if event.detail == Gdk.NotifyType.INFERIOR:+ return False+ self.state["hover_enter_count" if entered else "hover_exit_count"] += 1+ self.emit(+ "mouse", region="hot_zone", phase="mouseEntered" if entered else "mouseExited"+ )+ self.visible = entered+ actions.set_visible(entered)+ if entered:+ self.emit("overlay_shown")+ return False++ zone.connect("enter-notify-event", crossing, True)+ zone.connect("leave-notify-event", crossing, False)+ self.box.pack_start(zone, True, True, 0)++ def hover_click(self, name: str) -> None:+ self.state["clicks"].append({"name": name})+ self.state.update(last_clicked=name, last_click_overlay_visible=self.visible)+ self.emit("overlay_click", name=name, overlay_visible=self.visible)++ def tooltip(self) -> None:+ self.state.update(submit_count=0, status="Ready")+ icons = Gtk.Box(spacing=24)+ self.box.pack_start(icons, True, True, 0)+ for index, icon in enumerate(self.params["order"]):+ widget = Gtk.Label(label=icon)+ widget.set_has_tooltip(True)+ widget.connect("query-tooltip", self.tooltip_query, icon, self.params["codes"][index])+ icons.pack_start(widget, True, True, 0)+ self.field("code")+ self.button("Submit", self.tooltip_submit)++ def tooltip_query(self, widget, x, y, keyboard, tooltip, icon, code) -> bool:+ if keyboard:+ return False+ tooltip.set_text(code)+ self.emit("tooltip_query", icon=icon)+ return True++ def tooltip_submit(self) -> None:+ value = self.values()["code"]+ self.state.update(+ submit_count=self.state["submit_count"] + 1, status="Submitted", submitted_value=value+ )+ self.emit("submit", value=value)++ def irmodal(self) -> None:+ self.state.update(+ prompt_answers=[], submit_count=0, page=1, prompt_state="not_shown", next_count=0+ )+ self.field("name")+ self.field("email")+ self.step1 = list(self.box.get_children())+ self.field("team", C.IR_TEAMS)+ self.field("seats")+ self.step2 = [widget for widget in self.box.get_children() if widget not in self.step1]+ self.hidden = self.step2+ self.submit_button = self.button("Next", self.permission)+ self.state["fields"] = self.values()++ def permission(self) -> None:+ self.state["next_count"] += 1+ self.emit("next", count=self.state["next_count"], values=self.values())+ self.state["prompt_state"] = "open"+ self.emit("permission_prompt_shown", resource=self.params["resource"])+ self.dialog(+ f"Allow BenchLab to access {self.params['resource']}?",+ [("Don’t Allow", "deny"), ("Allow", "allow")],+ self.permission_answer,+ )++ def permission_answer(self, answer: str) -> None:+ self.state["prompt_answers"].append(answer)+ self.state.update(page=2, prompt_state="denied" if answer == "deny" else "allowed")+ self.emit("permission_answer", answer=answer, resource=self.params["resource"])+ for widget in self.step1:+ widget.hide()+ for widget in self.step2:+ widget.show_all()+ self.submit_button.destroy()+ self.button("Submit", self.form_submit).show()++ def form_submit(self) -> None:+ values = self.values()+ self.state.update(submit_count=self.state["submit_count"] + 1, submitted=values)+ self.emit("submit", values=values)++ def irbanner(self) -> None:+ self.state.update(+ banner_shown=False, banner_actions=[], banner_body_clicks=0, apply_count=0+ )+ self.label(self.params["item"])+ self.field("quantity", value="1")+ self.field("priority", C.PRIORITIES)+ self.apply_button = self.button("Apply", self.apply)+ self.box.remove(self.apply_button)+ overlay = Gtk.Overlay()+ overlay.add(self.apply_button)+ self.banner = Gtk.EventBox()+ content = Gtk.Box(spacing=8)+ self.banner.add(content)+ self.label("Update available", content)+ for label, action in (+ ("Restart Now", "restart"),+ ("Later", "later"),+ ("Dismiss", "dismiss"),+ ):+ self.button(label, lambda a=action: self.banner_action(a), content)+ self.banner.connect("button-press-event", self.banner_body_click)+ overlay.add_overlay(self.banner)+ self.box.pack_end(overlay, False, False, 0)+ self.hidden = [self.banner]+ self.state["fields"] = self.values()++ def banner_body_click(self, *_: object) -> bool:+ self.state["banner_body_clicks"] += 1+ self.emit("banner_body_click", count=self.state["banner_body_clicks"])+ return False++ def show_banner(self) -> bool:+ self.banner.show_all()+ self.emit("banner_shown")+ return False++ def banner_action(self, action: str) -> None:+ self.state["banner_actions"].append(action)+ if action == "restart":+ self.fields["quantity"].set_text("1")+ self.fields["priority"].set_active(0)+ self.banner.hide()+ self.emit("banner_action", action=action)++ def apply(self) -> None:+ self.state["apply_count"] += 1+ self.emit("apply", values=self.values())++ def irunsaved(self) -> None:+ self.state.update(+ saved=[+ {+ "title": title,+ "status": C.IR_INITIAL_STATUS,+ "body": f"Working notes for {title}.",+ }+ for title in self.params["titles"]+ ],+ dirty=False,+ answers=[],+ done_count=0,+ )+ self.selected = 0+ for index, title in enumerate(self.params["titles"]):+ self.button(title, lambda i=index: self.navigate(i))+ self.field("title", value=self.state["saved"][0]["title"])+ self.field("status", [C.IR_INITIAL_STATUS, *C.IR_STATUS_WORDS])+ self.field("body", value=self.state["saved"][0]["body"])+ self.button("Done", lambda: self.navigate(None))++ def navigate(self, index: int | None) -> None:+ if self.state["dirty"]:+ self.emit("save_prompt_shown")+ self.dialog(+ "Save changes before leaving?",+ [("Save", "save"), ("Don’t Save", "dont_save"), ("Cancel", "cancel")],+ lambda answer: self.save_answer(answer, index),+ )+ else:+ self.visit(index)++ def save_answer(self, answer: str, index: int | None) -> None:+ self.state["answers"].append(answer)+ self.emit("save_answer", answer=answer)+ if answer == "cancel":+ return+ if answer == "save":+ self.state["saved"][self.selected] = self.values()+ self.emit("save", index=self.selected, values=self.values())+ else:+ self.emit("discard", index=self.selected)+ self.state["dirty"] = False+ self.visit(index)++ def visit(self, index: int | None) -> None:+ if index is None:+ self.state["done_count"] += 1+ self.emit("done", clean=not self.state["dirty"])+ else:+ self.selected = index+ self.loading = True+ for key, value in self.state["saved"][self.selected].items():+ widget = self.fields[key]+ if isinstance(widget, Gtk.ComboBoxText):+ widget.set_active([C.IR_INITIAL_STATUS, *C.IR_STATUS_WORDS].index(value))+ else:+ widget.set_text(value)+ self.loading = False+ if index is not None:+ self.emit("note_selected", index=index)++ def irconsent(self) -> None:+ self.state.update(+ consent_shown=False, consent_answers=[], submit_count=0, submits_blocked=0+ )+ self.field("email")+ self.field("plan", C.IR_PLANS)+ digest = Gtk.CheckButton(label="Email digest")+ self.fields["digest"] = digest+ digest.connect("toggled", lambda *_: self.edit("digest"))+ self.box.pack_start(digest, False, False, 0)+ self.button("Sign up", self.signup)++ def show_consent(self) -> bool:+ self.emit("consent_shown")+ self.dialog(+ "Cookie preferences",+ [("Reject optional", "reject"), ("Accept all", "accept")],+ self.consent_answer,+ )+ return False++ def consent_answer(self, answer: str) -> None:+ details = {"answer": answer, "analytics": answer == "accept", "ads": answer == "accept"}+ self.state["consent_answers"].append(details)+ self.emit("consent_answer", **details)++ def signup(self) -> None:+ if not self.state["consent_answers"]:+ self.state["submits_blocked"] += 1+ self.emit("submit_blocked")+ return+ self.state["submit_count"] += 1+ self.emit("submit", **self.values())+++def main() -> None:+ parser = argparse.ArgumentParser()+ parser.add_argument("--mode", choices=MODES, required=True)+ parser.add_argument("--seed", type=int, required=True)+ parser.add_argument("--state", type=Path, required=True)+ parser.add_argument("--events", type=Path, required=True)+ lab = Lab(parser.parse_args())+ try:+ Gtk.main()+ finally:+ lab.events.close()+++if __name__ == "__main__":+ main()diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/linux/sentinel.py b/libs/cua-driver-bench/automated-eval/macos_bench/linux/sentinel.pynew file mode 100644index 0000000000..57afd23248--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/linux/sentinel.py@@ -0,0 +1,203 @@+# SPDX-License-Identifier: FSL-1.1-MIT+# Copyright (c) 2026 Cua AI, Inc.++from __future__ import annotations++import argparse+import ctypes+import json+import os+import re+import signal+import subprocess+import time+from pathlib import Path++import gi++gi.require_version("Gtk", "3.0")+gi.require_version("Gdk", "3.0")+gi.require_version("Atspi", "2.0")+from gi.repository import Atspi, Gdk, GLib, Gtk+++def xprop(*args: str) -> dict[str, str]:+ done = subprocess.run(["xprop", *args], capture_output=True, text=True, timeout=2)+ return {line.split("(", 1)[0]: line.split(" = ", 1)[-1] for line in done.stdout.splitlines()}+++def visible_here(window: int, current: str) -> bool:+ props = xprop("-id", hex(window), "_NET_WM_DESKTOP", "_NET_WM_STATE")+ return props.get("_NET_WM_DESKTOP", "") in ("4294967295", current) and (+ "_NET_WM_STATE_HIDDEN" not in props.get("_NET_WM_STATE", "")+ )+++def xid(value: str) -> int:+ match = re.search(r"0x[0-9a-fA-F]+", value)+ return int(match[0], 16) if match else 0+++class Sentinel:+ def __init__(self, args: argparse.Namespace) -> None:+ ctypes.CDLL(None).prctl(15, b"BenchSentinel", 0, 0, 0)+ GLib.set_prgname("BenchSentinel")+ GLib.set_application_name("BenchSentinel")+ self.args = args+ args.log.parent.mkdir(parents=True, exist_ok=True)+ self.log = args.log.open("a", buffering=1)+ self.armed = False+ self.clip = 0+ self.expected = ""+ self.previous = None+ self.window = Gtk.Window(title="User activity" if args.user_activity else "Bench Sentinel")+ self.window.set_wmclass("BenchSentinel", "BenchSentinel")+ self.window.set_default_size(600, 320)+ if args.frame:+ x, y, w, h = map(int, args.frame.split(","))+ self.window.move(x, y)+ self.window.resize(w, h)+ self.entry = Gtk.Entry()+ self.entry.get_accessible().set_name("User activity text")+ self.entry.connect("changed", self.changed)+ self.window.add(self.entry)+ self.window.connect("destroy", Gtk.main_quit)+ self.window.connect("focus-out-event", lambda *_: self.emit("didResignKey"))+ self.window.connect("focus-in-event", lambda *_: self.emit("didBecomeActive"))+ self.entry.connect("key-press-event", lambda *_: self.emit("keyDown"))+ self.entry.connect("button-press-event", lambda *_: self.emit("mouseDown"))+ self.entry.connect("scroll-event", lambda *_: self.emit("scrollWheel"))+ self.clipboard = Gtk.Clipboard.get(Gdk.SELECTION_CLIPBOARD)+ self.clipboard.connect("owner-change", self.clip_changed)+ self.window.show_all()+ self.own = self.window.get_window().get_xid()+ if not args.no_activate:+ self.window.present()+ GLib.unix_signal_add(GLib.PRIORITY_DEFAULT, signal.SIGUSR1, self.activate)+ GLib.unix_signal_add(GLib.PRIORITY_DEFAULT, signal.SIGUSR2, self.toggle)+ self.listener = Atspi.EventListener.new(self.ax_focus, None)+ self.listener.register("object:state-changed:focused")+ GLib.timeout_add(200, self.sample)+ if args.user_activity:+ GLib.timeout_add(500, self.activity)+ self.emit("ready")++ def clip_changed(self, *_: object) -> None:+ self.clip += 1++ def changed(self, *_: object) -> None:+ if self.entry.get_text() != self.expected:+ self.emit("user_text_disrupted")++ def ax_focus(self, event: object, *_: object) -> None:+ if self.armed:+ self.emit("ax_focus", name=event.source.get_name(), focused=bool(event.detail1))++ def identity(self, window: int) -> dict:+ if window == self.own:+ return {"bid": "ai.cua.benchsentinel", "pid": os.getpid()}+ props = xprop("-id", hex(window), "_NET_WM_PID", "WM_CLASS") if window else {}+ pid = props.get("_NET_WM_PID", "")+ names = re.findall(r'"([^"]+)"', props.get("WM_CLASS", ""))+ return {+ "bid": names[-1] if names else f"x11:{window}",+ "pid": int(pid) if pid.isdigit() else -1,+ }++ def snapshot(self) -> dict:+ root = xprop(+ "-root", "_NET_ACTIVE_WINDOW", "_NET_CLIENT_LIST_STACKING", "_NET_CURRENT_DESKTOP"+ )+ front = self.identity(xid(root.get("_NET_ACTIVE_WINDOW", "")))+ pointer = Gdk.Display.get_default().get_default_seat().get_pointer().get_position()+ own = xprop("-id", hex(self.own), "_NET_WM_DESKTOP")+ desktop = own.get("_NET_WM_DESKTOP", "")+ stack = [+ int(v, 16)+ for v in re.findall(r"0x[0-9a-fA-F]+", root.get("_NET_CLIENT_LIST_STACKING", ""))+ ]+ above = (+ [+ self.identity(w)["bid"]+ for w in stack[stack.index(self.own) + 1 :]+ if visible_here(w, root.get("_NET_CURRENT_DESKTOP", "").strip())+ ]+ if self.own in stack+ else []+ )+ sheet = any(+ xid(xprop("-id", hex(w), "WM_TRANSIENT_FOR").get("WM_TRANSIENT_FOR", "")) == self.own+ for w in stack+ if w != self.own+ )+ return {+ "front": front,+ "mouse": [pointer.x, pointer.y],+ "clip": self.clip,+ "onspace": desktop in ("4294967295", root.get("_NET_CURRENT_DESKTOP")),+ "sheet": sheet,+ "above": above,+ "above_n": len(above),+ }++ def emit(self, event: str, **details: object) -> bool:+ if self.armed or event in ("armed", "disarmed", "ready"):+ self.log.write(json.dumps({"t": time.time() * 1000, "ev": event, **details}) + "\n")+ return False++ def activate(self) -> bool:+ self.window.present()+ self.entry.grab_focus()+ return True++ def toggle(self) -> bool:+ if self.armed:+ if self.args.user_activity:+ self.emit("user_text", intact=self.entry.get_text() == self.expected)+ self.emit("disarmed", **self.snapshot())+ self.armed = False+ else:+ self.armed = True+ self.emit("armed", **self.snapshot())+ return True++ def sample(self) -> bool:+ if self.armed:+ sample = self.snapshot()+ if sample["front"] != self.previous:+ self.emit("front", **sample)+ if self.previous and self.previous.get("bid") == "ai.cua.benchsentinel":+ self.emit("didResignActive")+ self.previous = sample["front"]+ self.emit("s", **sample)+ return True++ def activity(self) -> bool:+ if self.armed:+ if self.snapshot()["front"].get("bid") == "ai.cua.benchsentinel":+ if self.entry.get_text() != self.expected:+ self.emit("user_text_disrupted")+ self.expected += "a"+ self.entry.set_text(self.expected)+ self.emit("user_type")+ else:+ self.emit("user_blocked")+ return True+++def main() -> None:+ parser = argparse.ArgumentParser()+ parser.add_argument("--log", type=Path, required=True)+ parser.add_argument("--no-activate", action="store_true")+ parser.add_argument("--user-activity", action="store_true")+ parser.add_argument("--frame")+ witness = Sentinel(parser.parse_args())+ try:+ Gtk.main()+ finally:+ witness.listener.deregister("object:state-changed:focused")+ witness.log.close()+++if __name__ == "__main__":+ main()diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/recorder.py b/libs/cua-driver-bench/automated-eval/macos_bench/recorder.pyindex 54e37d0a93..43cde47049 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/recorder.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/recorder.py@@ -23,6 +23,8 @@ import timefrom pathlib import Pathfrom typing import Any, Callable+from desktop_runtime import stop_group, tool+class McpStdioClient:"""Minimal MCP client over a child process' stdio (newline-delimited JSON-RPC)."""@@ -125,7 +127,8 @@ class McpStdioClient:try:self.proc.wait(timeout=5)except subprocess.TimeoutExpired:- self.proc.kill()+ stop_group(self.proc.pid)+ self.proc.wait(timeout=5)class NullRecorder:@@ -150,6 +153,7 @@ class CuaRecorder:self.binary, self.socket, self.env = binary, socket, envself.client: McpStdioClient | None = Noneself.active_dir: Path | None = None+ self.last_video: Path | None = Noneself.last_error: str | None = Nonedef _connect(self) -> McpStdioClient:@@ -166,7 +170,8 @@ class CuaRecorder:state = client.call_tool("start_recording", {"output_dir": str(out_dir), "record_video": True}, timeout=30)- self.active_dir = out_dir+ self.active_dir = out_dir if state.get("video_active") else None+ self.last_video = Noneself.last_error = state.get("last_error")return bool(state.get("video_active"))except (OSError, TimeoutError, RuntimeError) as error:@@ -175,28 +180,35 @@ class CuaRecorder:return Falsedef stop(self) -> Path | None:+ if self.active_dir is None:+ return self.last_videotry:client = self._connect()state = client.call_tool("stop_recording", {}, timeout=60)+ self.active_dir = Nonepath = state.get("last_video_path")if not path:state = client.call_tool("get_recording_state", {}, timeout=15)path = state.get("last_video_path")self.last_error = state.get("last_error") or self.last_errorif path and Path(path).is_file():- return Path(path)+ self.last_video = Path(path)+ return self.last_videoexcept (OSError, TimeoutError, RuntimeError) as error:self.last_error = f"{type(error).__name__}: {error}"self.client = Nonereturn Nonedef close(self) -> None:+ if self.active_dir is not None:+ self.stop()if self.client is not None:self.client.close()self.client = None-def capture_detail(path: Path, ffmpeg: str = "/opt/homebrew/bin/ffmpeg") -> str:+def capture_detail(path: Path, ffmpeg: str | None = None) -> str:+ ffmpeg = ffmpeg or tool("ffmpeg")done = subprocess.run([ffmpeg,@@ -236,7 +248,7 @@ def ffprobe_duration(path: Path) -> float | None:try:done = subprocess.run([- "/opt/homebrew/bin/ffprobe",+ tool("ffprobe"),"-v","error","-show_entries",@@ -254,11 +266,10 @@ def ffprobe_duration(path: Path) -> float | None:return None-def compress_video(- raw: Path, dest_dir: Path, ffmpeg: str = "/opt/homebrew/bin/ffmpeg"-) -> dict[str, Any]:+def compress_video(raw: Path, dest_dir: Path, ffmpeg: str | None = None) -> dict[str, Any]:"""H.264 (libx264, crf 28, yuv420p, faststart) full-size copy plus a 720p copy; delete the rawfile only when both copies exist and decode to a plausible duration."""+ ffmpeg = ffmpeg or tool("ffmpeg")dest_dir.mkdir(parents=True, exist_ok=True)full = dest_dir / "video.mp4"small = dest_dir / "video-720p.mp4"diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.py b/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.pyindex d2bedfd432..7a2ceb3f42 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.py@@ -58,6 +58,8 @@ import claude_driver # noqa: E402import claude_events # noqa: E402import recorder as rec # noqa: E402import run_pilot as pilot # noqa: E402+import desktop_runtime as desktop+from dataclasses import replaceSCHEMA = "cdb-bench-trial/1"WEB_TASKS = ("WEB-01", "WEB-02")@@ -247,6 +249,8 @@ def git_commit(path: Path) -> str:def sw_vers() -> str:+ if desktop.detect_platform() == "linux":+ return os.uname().releasereturn subprocess.run(["sw_vers", "-productVersion"], capture_output=True, text=True).stdout.strip()@@ -269,6 +273,7 @@ def src_hash() -> str:digest = hashlib.sha256()for path in (sorted(HERE.glob("*.py"))+ + sorted((HERE / "linux").rglob("*.py"))+ sorted((HERE / "probes").rglob("*.py"))+ sorted((HERE / "probes").rglob("*.md"))+ sorted((HERE / "probes").rglob("task.json"))@@ -509,9 +514,7 @@ def check_daemon(pins: dict[str, Any], arm: str = "cc-cua-driver") -> list[tuple"git_sha") == main.get("git_sha")checks = health.get("checks", {})- perms = (- checks.get("tcc_accessibility") == "pass" and checks.get("tcc_screen_recording") == "pass"- )+ perms = desktop.capabilities_ok(health)return [(f"{arm} daemon reports version",@@ -755,6 +758,40 @@ def evaluate_probe(}+def _collect_linux_evidence(artifacts: Path, arm: str, failed: bool) -> None:+ """Best-effort Linux end-of-trial evidence; never raises."""+ if desktop.detect_platform() != "linux":+ return+ errors: list[str] = []++ def attempt(operation: str, action: Any) -> None:+ try:+ action()+ except Exception as error: # noqa: BLE001+ errors.append(f"{operation}: {type(error).__name__}")++ attempt(+ "screenshot",+ lambda: desktop.capture_screenshot(+ os.environ.get("DISPLAY", ""), artifacts / ("failure.png" if failed else "final.png")+ ),+ )+ logs = {"recorder": ca.RECORDER_STATE / "daemon.log"}+ if arm in ca.CUA_BUILDS:+ logs["driver"] = ca.CUA_BUILDS[arm].state / "daemon.log"+ for name, log in logs.items():+ attempt(+ f"{name} log",+ lambda name=name, log=log: log.is_file()+ and shutil.copyfile(log, artifacts / f"{name}.log"),+ )+ if errors:+ try:+ (artifacts / "screenshot-error.txt").write_text("\n".join(errors))+ except OSError:+ pass++def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]:args = ctx.argstask = ctx.tasks[entry["task"]]@@ -784,6 +821,7 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]row: dict[str, Any] = {"schema": SCHEMA,"infra_pattern": None,+ "platform": desktop.detect_platform(),"trial_id": entry["trial_id"],"attempt": attempt,"final": False,@@ -823,6 +861,7 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]if arm in ca.CUA_ARMSelse None,"macos": ctx.versions.get("macos"),+ "os_version": ctx.versions.get("os_version"),"bench_repo_commit": ctx.versions.get("bench_repo_commit"),"bench_src_sha256": ctx.versions.get("bench_src_sha256"),"system_prompt_sha256": hashlib.sha256(system_prompt.encode()).hexdigest(),@@ -843,8 +882,8 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]if arm in cxn.ARMS:row["model"], row["effort"] = cxn.MODEL, cxn.EFFORTsentinel = pilot.Sentinel(- Path(args.build_dir) / "BenchSentinel.app"- if args.build_dir and not args.no_sentinel+ Path(args.build_dir or HERE / "linux") / "BenchSentinel.app"+ if (args.build_dir or desktop.detect_platform() == "linux") and not args.no_sentinelelse None,artifacts / "sentinel.jsonl",v2.sentinel_args(condition) if bench_v2 else None,@@ -881,9 +920,10 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any])kill_bench_apps()reset_needs_apps(needs)- subprocess.run(- ["pbcopy"], input=b"", capture_output=True- ) # no stale clipboard between trials+ if desktop.detect_platform() == "macos":+ subprocess.run(+ ["pbcopy"], input=b"", capture_output=True+ ) # no stale clipboard between trialsif arm in ca.CUA_ARMS:ensure_agent_daemon(ctx, arm)elif (@@ -892,7 +932,7 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]ensure_agent_daemon(ctx, cxn.SOURCE_ARM[arm])if getattr(args, "park_pointer", False):row["pointer_parked"] = park_pointer(ctx)- lab_app = Path(args.build_dir) / "BenchLab.app"+ lab_app = Path(args.build_dir or HERE / "linux") / "BenchLab.app"if cdb is not None:cdb.reset()brief, paths = cdb.brief(), {}@@ -911,6 +951,8 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]brief, paths = pilot.prepare_probe(task, seed, trial_dir, lab_app)sentinel.start()lab = pilot.launch_lab(task, seed, paths, lab_app, app_env)+ if desktop.detect_platform() == "linux":+ desktop.capture_screenshot(os.environ["DISPLAY"], artifacts / "startup.png")if sentinel_front:sentinel.activate()if condition == "background": # A15.2: the user window covers part of the target on purpose@@ -1053,13 +1095,16 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]failure = {"kind": "harness_exception", "message": f"{type(error).__name__}: {error}"}note = failure["message"]finally:- codex_home = trial_dir / "codex-home" / ".codex"- if arm in cxn.ARMS and codex_home.is_dir():- try:- arms.unlink_chatgpt_auth(codex_home)- except RuntimeError as error:- failure = {"kind": "harness_exception", "message": str(error)}- note = str(error)+ try:+ _collect_linux_evidence(artifacts, arm, bool(failure.get("kind")))+ finally:+ codex_home = trial_dir / "codex-home" / ".codex"+ if arm in cxn.ARMS and codex_home.is_dir():+ try:+ arms.unlink_chatgpt_auth(codex_home)+ except RuntimeError as error:+ failure = {"kind": "harness_exception", "message": str(error)}+ note = str(error)try:sentinel.stop()except Exception: # noqa: BLE001@@ -1212,8 +1257,12 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]"pointer_deviation_episodes": summary_sentinel.get("pointer_deviation_episodes", 0),"pointer_moved": deviation > pilot_deviation_px(),"frontmost_changed": bool(summary_sentinel.get("front_changes", 0)),- "hid_events": hid or {"move": 0, "down": 0, "key": 0, "scroll": 0},- "human_input_suspected": bool(hid_total and idle_start < 900),+ "hid_events": (hid or {"move": 0, "down": 0, "key": 0, "scroll": 0})+ if desktop.detect_platform() == "macos"+ else None,+ "human_input_suspected": bool(hid_total and idle_start < 900)+ if desktop.detect_platform() == "macos"+ else None,"windows_raised": summary_sentinel.get("windows_raised"),"raised_by": summary_sentinel.get("raised_by", []),"overlay_owners": summary_sentinel.get("overlay_owners", []),@@ -1229,7 +1278,7 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]"gui_only": gui_only,"interruption": v2.interruption_of(evaluation),"hid_idle_drops": pilot.hid_drops(artifacts / "hid-idle.jsonl")- if (artifacts / "hid-idle.jsonl").exists()+ if desktop.detect_platform() == "macos" and (artifacts / "hid-idle.jsonl").exists()else None,"confirmation_requested": False,"evaluator_read_suspected": bool(peeks),@@ -1504,14 +1553,8 @@ def _walk_dicts(node: Any):yield from _walk_dicts(value)-SIDE_DOOR_BUNDLES = {- "com.apple.Terminal": "Terminal",- "com.googlecode.iterm2": "iTerm",- "com.apple.ScriptEditor2": "Script Editor",- "com.apple.automator": "Automator",- "com.apple.shortcuts": "Shortcuts",-}-SIDE_DOOR_PROCESSES = ("Script Editor", "Automator", "iTerm2", "Shortcuts")+SIDE_DOOR_BUNDLES = v2.SIDE_DOOR_BUNDLES+SIDE_DOOR_PROCESSES = v2.SIDE_DOOR_PROCESSESSIDE_DOOR_PATTERNS = ("Terminal","iTerm",@@ -1535,6 +1578,23 @@ SIDE_DOOR_PATTERNS = (def frontmost_bundle() -> str | None:"""Bundle id of the frontmost app through lsappinfo (no TCC permission needed)."""+ if desktop.detect_platform() == "linux":+ try:+ window = subprocess.check_output(+ ["xdotool", "getactivewindow"], text=True, timeout=3+ ).strip()+ props = subprocess.check_output(+ ["xprop", "-id", window, "WM_CLASS"], text=True, timeout=3+ )+ names = re.findall(r'"([^\"]+)"', props)+ if not names:+ return None+ return next(+ (name for name in v2.LINUX_TERMINALS if name.casefold() == names[-1].casefold()),+ names[-1],+ )+ except (OSError, subprocess.SubprocessError):+ return Nonetry:asn = subprocess.run(["/usr/bin/lsappinfo", "front"], capture_output=True, text=True, timeout=3@@ -1613,6 +1673,8 @@ SAVED_STATE = {def reset_needs_apps(names: list[str]) -> None:"""Kill the apps a task needs and remove their saved window state so each trial starts clean."""+ if desktop.detect_platform() == "linux":+ returnfor name in names:subprocess.run(["pkill", "-x", name], capture_output=True)time.sleep(0.3)@@ -2074,7 +2136,11 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:)add("terminal.app ancestor",- "pass" if terminal_ancestor() else ("warn" if args.allow_non_terminal else "fail"),+ "skip"+ if desktop.detect_platform() == "linux"+ else "pass"+ if terminal_ancestor()+ else ("warn" if args.allow_non_terminal else "fail"),"runner started from Terminal.app"if terminal_ancestor()else "not started from Terminal.app (required for arm cc-codex-cu)",@@ -2130,6 +2196,9 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:ensure_agent_daemon(ctx, cua_arm)for name, status, detail in check_daemon(pins, cua_arm):add(name, status, detail)+ if desktop.detect_platform() == "linux":+ for check in desktop.doctor(spec.bin, spec.socket, spec.home):+ add(check.name, check.status, check.detail)health = ca.cua_health(spec.socket, spec.home, spec.bin)ctx.versions.setdefault("builds", {}).setdefault(cua_arm, {})["cua_daemon_version"] = (health.get("build", {}).get("version")@@ -2170,12 +2239,32 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:add(f"CDB {tid} descriptor readable", "pass", f"{len(names)} app names")except Exception as error: # noqa: BLE001add(f"CDB pack {tid}", "fail", f"{type(error).__name__}: {error}")- for app in ("Google Chrome", "LibreOffice", "Gnucash"):- add(- f"CDB app installed: {app}",- "pass" if Path(f"/Applications/{app}.app").is_dir() else "fail",- f"/Applications/{app}.app",- )+ if desktop.detect_platform() == "macos":+ for app in ("Google Chrome", "LibreOffice", "Gnucash"):+ add(+ f"CDB app installed: {app}",+ "pass" if Path(f"/Applications/{app}.app").is_dir() else "fail",+ f"/Applications/{app}.app",+ )+ else:+ for tid in cdb_ids:+ probe = cdb_adapter.CdbTask(task_spec(ctx.tasks[tid]), ctx.run_dir)+ for prerequisite in probe.descriptor.get("prerequisites", []):+ try:+ done = subprocess.run(+ [probe.sub(p) for p in prerequisite["check"]],+ cwd=probe.sub(prerequisite.get("cwd", str(probe.bundle))),+ env=desktop.desktop_env(Path.home()),+ capture_output=True,+ timeout=30,+ )+ add(+ f"CDB {tid} prerequisite {prerequisite['id']}",+ "pass" if done.returncode == 0 else "fail",+ str(prerequisite["check"]),+ )+ except (OSError, subprocess.TimeoutExpired) as error:+ add(f"CDB {tid} prerequisite {prerequisite['id']}", "fail", str(error))electron, missing = cdb_adapter.missing_electron_apps()add("CDB Electron apps installed (npm ci + install.js)",@@ -2194,20 +2283,24 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:"CDB_BENCH_DISPOSABLE=1 (the runner kills Electron/Chrome/LibreOffice by name)",)build = Path(args.build_dir) if args.build_dir else None- ok_build = bool(- build and (build / "BenchLab.app").is_dir() and (build / "BenchSentinel.app").is_dir()- )+ try:+ for name in BENCH_APPS:+ desktop.bench_app_argv(name, build)+ ok_build = True+ except (OSError, ValueError, StopIteration):+ ok_build = Falseadd("BenchLab + BenchSentinel builds", "pass" if ok_build else "fail", str(build))copies = registered_bench_copies()per_name = {n: [c for c in copies if Path(c).name == n] for n in BENCH_APPS}- ok_copies = all(len(v) == 1 for v in per_name.values())+ ok_copies = desktop.detect_platform() == "linux" or all(len(v) == 1 for v in per_name.values())add("exactly one registered copy of BenchLab and BenchSentinel",- "pass" if ok_copies else "fail",- json.dumps(per_name) if not ok_copies else f"{[v[0] for v in per_name.values()]}",+ "skip" if desktop.detect_platform() == "linux" else "pass" if ok_copies else "fail",+ "not applicable on Linux" if desktop.detect_platform() == "linux" else json.dumps(per_name),)try:- subprocess.run(["/usr/bin/open", "-a", "Calculator"], check=True, timeout=30)+ if desktop.detect_platform() == "macos":+ subprocess.run(["/usr/bin/open", "-a", "Calculator"], check=True, timeout=30)for arm in ctx.arm_names:add(*real_capture_check(ctx, arm))except (OSError, subprocess.SubprocessError) as error:@@ -2225,6 +2318,8 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:"BenchLab.app/Contents/MacOS","BenchSentinel.app/Contents/MacOS","/Calculator.app/Contents/MacOS/Calculator",+ str(HERE / "linux" / "benchlab.py"),+ str(HERE / "linux" / "sentinel.py"),))]@@ -2347,6 +2442,9 @@ def build_parser() -> argparse.ArgumentParser:help="default: ToolSearch is in the built-in tool set so Claude Code's own deferral applies per arm",)p.add_argument("--tell-budget", action=argparse.BooleanOptionalAction, default=True)+ p.add_argument("--cua-bin", type=Path, help="driver executable (required on Linux)")+ p.add_argument("--cua-skills", type=Path, help="matching driver skill directory")+ p.add_argument("--pins", type=Path, help="platform-specific identity pins")p.add_argument("--build-dir",type=Path,@@ -2451,6 +2549,25 @@ def make_ctx(args: argparse.Namespace) -> Ctx:explicit = bool(args.tasks) or bench_v2 # v2 keeps the registered order (A14.6)task_ids = core.order_tasks(selected, explicit=explicit)full_ids = task_ids if args.tasks else core.order_tasks(defaults, explicit=bench_v2)+ unsupported = []+ if desktop.detect_platform() == "linux":+ unsupported = [+ f"unsupported-platform: {a} requires macOS" for a in arm_names if a not in ca.CUA_ARMS+ ]+ for reason in unsupported:+ print(reason, file=sys.stderr)+ arm_names = [a for a in arm_names if a in ca.CUA_ARMS]+ if not arm_names:+ raise SystemExit("no supported arms remain: " + "; ".join(unsupported))+ if args.pins:+ ca.PINS_FILE = args.pins.resolve()+ if args.cua_bin:+ if not args.cua_skills:+ raise SystemExit("--cua-bin requires --cua-skills from the same build")+ ca.CUA_BIN = args.cua_bin.resolve()+ ca.CUA_SKILLS = args.cua_skills.resolve()+ for name, build in ca.CUA_BUILDS.items():+ ca.CUA_BUILDS[name] = replace(build, executable=ca.CUA_BIN, skills=ca.CUA_SKILLS)codex = [a for a in arm_names if a in arms.CODEX_FALLBACK_ARMS]if codex and not args.allow_codex_arms:raise SystemExit(f"arms {codex} are fallback arms; pass --allow-codex-arms")@@ -2465,17 +2582,33 @@ def make_ctx(args: argparse.Namespace) -> Ctx:args, run_dir, core.Ledger(args.ledger), tasks, task_ids, arm_names, full_task_ids=full_ids)ctx.cutoff = core.parse_utc(args.cutoff_utc) if args.cutoff_utc else None+ os_version = sw_vers()ctx.versions = {"claude": claude_version(),- "macos": sw_vers(),- "bench_repo_commit": git_commit(Path.home() / "repo/cua-driver-bench-codex-vs-cua"),+ "platform": desktop.detect_platform(),+ "unsupported_arms": unsupported,+ "os_version": os_version,+ "macos": os_version if desktop.detect_platform() == "macos" else None,+ "bench_repo_commit": git_commit(HERE),"bench_src_sha256": src_hash(),}return ctxdef setup_arms(ctx: Ctx) -> None:+ if desktop.detect_platform() == "linux" and not (+ ctx.args.cua_bin and ctx.args.cua_skills and ctx.args.pins+ ):+ raise SystemExit(+ "Linux requires explicit --cua-bin, --cua-skills and --pins; macOS build identities cannot be reused"+ )pins = ca.load_pins()+ if desktop.detect_platform() == "linux":+ required = ("cua_driver_version", "cua_driver_binary_sha256", "cua_skills_tree_sha256")+ if pins.get("platform") != "linux" or not all(pins.get(k) for k in required):+ raise SystemExit(+ "Linux pins require platform=linux and driver version, binary and skills hashes"+ )observed = ca.observed_pins(include_codex=any(a in ca.CODEX_ARMS or a in cxn.ARMS for a in ctx.arm_names),include_cua="cc-cua-driver" in ctx.arm_names,@@ -2529,13 +2662,11 @@ def setup_arms(ctx: Ctx) -> None:except (FileNotFoundError, ValueError) as error:raise SystemExit(f"arm {arm} unavailable: {error}")ctx.versions["codex_native"] = cxn.observed()- if ctx.args.build_dir:+ if ctx.args.build_dir or desktop.detect_platform() == "linux":apps: dict[str, str] = {}for name in BENCH_APPS:- macos = Path(ctx.args.build_dir) / name / "Contents/MacOS"- exes = sorted(macos.iterdir()) if macos.is_dir() else []- if exes:- apps[name] = ca.sha256_file(exes[0])+ argv = desktop.bench_app_argv(name, ctx.args.build_dir)+ apps[name] = ca.sha256_file(Path(argv[-1]))ctx.versions["bench_apps_exe_sha256"] = appsfor codex_arm in [a for a in ctx.mcp if a in ca.CODEX_ARMS]:key = "codex_cu_mcp_sha256" if codex_arm == "cc-codex-cu" else f"{codex_arm}_mcp_sha256"diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/run_pilot.py b/libs/cua-driver-bench/automated-eval/macos_bench/run_pilot.pyindex cec360b64d..f4511d656a 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/run_pilot.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/run_pilot.py@@ -39,6 +39,7 @@ sys.path.insert(0, str(HERE / "swift"))import arms # noqa: E402import codex_events # noqa: E402+import desktop_runtime as desktopSCHEMA = "cdb-pilot-trial/1"SKIP_APP_KINDS = {"terminal", "editor"}@@ -121,6 +122,39 @@ def build_schedule(def hid_idle_seconds() -> float:+ if desktop.detect_platform() == "linux":+ import ctypes+ import ctypes.util++ class Info(ctypes.Structure):+ _fields_ = [+ ("window", ctypes.c_ulong),+ ("state", ctypes.c_int),+ ("kind", ctypes.c_int),+ ("til_or_since", ctypes.c_ulong),+ ("idle", ctypes.c_ulong),+ ("mask", ctypes.c_ulong),+ ]++ x11 = ctypes.CDLL(ctypes.util.find_library("X11"))+ x11.XOpenDisplay.restype = ctypes.c_void_p+ display = x11.XOpenDisplay(None)+ if not display:+ raise RuntimeError("X11 unavailable; set DISPLAY")+ x11.XDefaultRootWindow.argtypes = [ctypes.c_void_p]+ x11.XDefaultRootWindow.restype = ctypes.c_ulong+ xss = ctypes.CDLL(ctypes.util.find_library("Xss"))+ xss.XScreenSaverQueryInfo.argtypes = [ctypes.c_void_p, ctypes.c_ulong, ctypes.POINTER(Info)]+ info = Info()+ try:+ if not xss.XScreenSaverQueryInfo(+ display, x11.XDefaultRootWindow(display), ctypes.byref(info)+ ):+ raise RuntimeError("XScreenSaver idle query failed")+ return info.idle / 1000+ finally:+ x11.XCloseDisplay.argtypes = [ctypes.c_void_p]+ x11.XCloseDisplay(display)try:out = subprocess.run(["ioreg", "-c", "IOHIDSystem"], capture_output=True, text=True, timeout=10@@ -188,15 +222,13 @@ def hid_drops(path: Path) -> int:def clean_app_env(home: Path) -> dict[str, str]:- env = {- "PATH": f"/opt/homebrew/bin:{arms.CLOSED_PATH}",- "HOME": str(home),- "USER": os.environ.get("USER", ""),- "LANG": "en_US.UTF-8",- "TMPDIR": str(home / "tmp"),- }(home / "tmp").mkdir(parents=True, exist_ok=True)- return env+ extra = (+ {"PATH": f"/opt/homebrew/bin:{arms.CLOSED_PATH}"}+ if desktop.detect_platform() == "macos"+ else None+ )+ return desktop.desktop_env(home, extra)def sweep_processes(needle: str) -> int:@@ -214,13 +246,7 @@ def sweep_processes(needle: str) -> int:return killed-def kill_group(pid: int) -> None:- for sig in (signal.SIGTERM, signal.SIGKILL):- try:- os.killpg(pid, sig)- except OSError:- return- time.sleep(1.0)+kill_group = desktop.stop_groupclass Sentinel:@@ -236,20 +262,31 @@ class Sentinel:@propertydef enabled(self) -> bool:- return self.app is not None and (self.app / "Contents/MacOS").is_dir()+ return self.app is not None and (+ desktop.detect_platform() == "linux" or (self.app / "Contents/MacOS").is_dir()+ )def start(self) -> None:if not self.enabled:return- exe = next((self.app / "Contents/MacOS").iterdir())- self.log.parent.mkdir(parents=True, exist_ok=True)- self.proc = subprocess.Popen(- [str(exe), "--log", str(self.log), *self.args],- stdin=subprocess.DEVNULL,- stdout=subprocess.DEVNULL,- stderr=subprocess.DEVNULL,- start_new_session=True,+ self.proc = desktop.start_app(+ [+ *desktop.bench_app_argv("BenchSentinel", self.app.parent),+ "--log",+ str(self.log),+ *self.args,+ ],+ self.log.parent / "sentinel-home",+ self.log.with_suffix(".stderr"),)+ if desktop.detect_platform() == "linux":+ deadline = time.monotonic() + 10+ while time.monotonic() < deadline and self.proc.poll() is None:+ if self.log.exists() and '"ev": "ready"' in self.log.read_text():+ return+ time.sleep(0.1)+ self.stop()+ raise RuntimeError(f"BenchSentinel not ready; see {self.log.with_suffix('.stderr')}")time.sleep(2.0)def signal(self, sig: int) -> None:@@ -267,6 +304,7 @@ class Sentinel:def stop(self) -> None:if self.proc is not None:kill_group(self.proc.pid)+ self.proc.wait(timeout=5)self.proc = Nonedef summary(self) -> dict[str, Any]:@@ -275,7 +313,10 @@ class Sentinel:try:import summarize_sentinel # type: ignore- return summarize_sentinel.summarize(str(self.log))+ summary = summarize_sentinel.summarize(str(self.log))+ if desktop.detect_platform() == "linux":+ summary["hid_events"] = None+ return summaryexcept Exception as error: # noqa: BLE001 - never fail a trial on a summaryreturn {"available": False, "error": f"{type(error).__name__}: {error}"}@@ -499,9 +540,8 @@ def prepare_probe(def launch_lab(task: Task, seed: int, paths: dict[str, Path], lab_app: Path, app_env: dict[str, str]) -> subprocess.Popen[bytes]:- exe = next((lab_app / "Contents/MacOS").iterdir())args = [- str(exe),+ *desktop.bench_app_argv("BenchLab", lab_app.parent),*task.app_args,"--seed",str(seed),@@ -510,17 +550,16 @@ def launch_lab("--events",str(paths["events"]),]- proc = subprocess.Popen(- args,- env=app_env,- stdin=subprocess.DEVNULL,- stdout=subprocess.DEVNULL,- stderr=subprocess.DEVNULL,- start_new_session=True,+ proc = desktop.start_app(+ args, Path(app_env["HOME"]), paths["state"].parent / "lab.log", env=app_env)deadline = time.monotonic() + 20- while time.monotonic() < deadline and not paths["state"].exists():+ while time.monotonic() < deadline and not paths["state"].exists() and proc.poll() is None:time.sleep(0.3)+ if not paths["state"].exists():+ desktop.stop_group(proc.pid)+ proc.wait(timeout=5)+ raise RuntimeError(f"BenchLab not ready; see {paths['state'].parent / 'lab.log'}")time.sleep(1.5)return proc@@ -748,44 +787,6 @@ def run_trial(# ---------------------------------------------------------------- main-def start_cua_daemon(binary: Path, socket: str, state_dir: Path) -> subprocess.Popen[bytes]:- """Private Cua Driver daemon for the release under test (own HOME, no telemetry)."""- home = state_dir / "home"- home.mkdir(parents=True, exist_ok=True)- try:- os.unlink(socket)- except OSError:- pass- env = {- "HOME": str(home),- "PATH": f"/opt/homebrew/bin:{arms.CLOSED_PATH}",- "CUA_DRIVER_RS_TELEMETRY_ENABLED": "false",- }- log = (state_dir / "daemon.log").open("wb")- proc = subprocess.Popen(- [str(binary), "serve", "--socket", socket, "--dangerously-bypass-approvals"],- env=env,- stdin=subprocess.DEVNULL,- stdout=log,- stderr=log,- start_new_session=True,- )- deadline = time.monotonic() + 30- while time.monotonic() < deadline:- done = subprocess.run(- [str(binary), "status", "--socket", socket],- env=env,- capture_output=True,- text=True,- timeout=10,- )- if done.returncode == 0:- return proc- time.sleep(0.5)- kill_group(proc.pid)- raise RuntimeError("Cua Driver daemon did not become ready")--def environment_manifest(args: argparse.Namespace) -> dict[str, Any]:def run(*cmd: str) -> str:try:@@ -889,7 +890,7 @@ def main() -> int:import mcp_preflightif args.cua_app is not None and "cua-driver-mcp" in args.arms:- daemon = start_cua_daemon(+ daemon = desktop.start_driver(arms.CUA_DRIVER_BIN, args.cua_socket, args.out.resolve() / "cua-daemon")for arm in args.arms:diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_analyze.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_analyze.pyindex 097cba8d68..2d6bba996e 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_analyze.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_analyze.py@@ -95,6 +95,32 @@ def run(rows, **kw):return az.analyze(rows, **kw)+class LinuxDisturbanceTest(unittest.TestCase):+ def test_any_disturbance_is_unmeasured_when_hid_is_unmeasured(self):+ rows = [make_row(A, "MB-09", i, False, platform="linux") for i in (1, 2)]+ for row in rows:+ row["disturbance"].update(hid_events=None, human_input_suspected=None)+ rows[0]["disturbance"]["front_changes"] = 1+ summary = run(rows)["disturbance"][A]+ self.assertEqual(summary["n_used"], 2)+ self.assertEqual(summary["fields"]["front_changes"]["sum"], 1)+ any_disturbance = summary["any_disturbance"]+ for field in ("count", "n", "fraction", "wilson_low", "wilson_high"):+ self.assertIsNone(any_disturbance[field], field)+ self.assertFalse(any_disturbance["measured"])++ def test_unmeasured_hid_is_valid_only_with_linux_provenance(self):+ row = make_row(A, "MB-09", 1, False, platform="linux")+ row["disturbance"].update(hid_events=None, human_input_suspected=None)+ az.validate_row(row, "linux")+ summary = run([row])["disturbance"][A]+ self.assertIsNone(summary["fields"]["hid_events_total"]["sum"])+ self.assertIsNone(summary["any_disturbance"]["fraction"])+ row["platform"] = "macos"+ with self.assertRaises(az.TrialFormatError):+ az.validate_row(row, "macos")++class WilsonTests(unittest.TestCase):def test_zero_of_ten(self):lo, hi = az.wilson_interval(0, 10)diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_bench_v2.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_bench_v2.pyindex 8b40b58bb2..a198ef333f 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_bench_v2.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_bench_v2.py@@ -144,6 +144,18 @@ class GuardTest(unittest.TestCase):self.assertTrue(guard.stop.is_set())self.assertEqual(guard.violations[0]["detail"], "front:Terminal")+ def test_open_linux_terminal_is_not_a_violation(self) -> None:+ guard, _ = self.make([None], procs={"xterm"})+ guard.sample()+ self.assertFalse(guard.stop.is_set())+ self.assertEqual(guard.violations, [])++ def test_linux_terminal_in_front_is_a_violation(self) -> None:+ guard, _ = self.make(["xterm"] * 3)+ for _ in range(3):+ guard.sample()+ self.assertEqual(guard.violations[0]["detail"], "front:xterm")+def test_scripting_process_is_a_violation(self) -> None:guard, _ = self.make([None], procs={"Script Editor"})guard.sample()@@ -204,6 +216,18 @@ class GuardTest(unittest.TestCase):guard.sample()self.assertEqual(guard.violations, [])+ def test_linux_terminal_ptys_exclude_runner_descendants(self) -> None:+ rows = [+ (100, 1, "?", "/usr/bin/gnome-terminal-server"),+ (101, 100, "pts/0", "bash"),+ (102, 101, "pts/0", "python3"),+ (103, 102, "pts/8", "bash"),+ (200, 1, "?", "/usr/bin/xterm"),+ (201, 200, "pts/1", "bash"),+ (300, 1, "pts/2", "sshd"),+ ]+ self.assertEqual(v2.terminal_session_ttys(rows, exclude_pid=102), {"/dev/pts/1"})+def test_terminal_session_ttys_skip_the_runner_and_non_terminal_ttys(self) -> None:rows = [(1, 0, "??", "/sbin/launchd"),@@ -286,7 +310,11 @@ class BackgroundTest(unittest.TestCase):class SelectionTest(unittest.TestCase):def ctx(self, *argv: str) -> rb.Ctx:- args = rb.build_parser().parse_args(["dry-run", *argv])+ tmp = tempfile.TemporaryDirectory()+ self.addCleanup(tmp.cleanup)+ args = rb.build_parser().parse_args(+ ["dry-run", "--ledger", str(Path(tmp.name) / "ledger.jsonl"), *argv]+ )return rb.make_ctx(args)def test_v2_selects_the_registered_list_in_order(self) -> None:diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_cdb_adapter.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_cdb_adapter.pyindex 4c4507077f..36ed9540fe 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_cdb_adapter.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_cdb_adapter.py@@ -10,6 +10,7 @@ import osimport sysimport tempfileimport unittest+from unittest import mockfrom pathlib import PathHERE = Path(__file__).resolve().parent.parent@@ -61,6 +62,7 @@ DESCRIPTOR = {}[email protected](sys.platform == "darwin", "macOS bundle descriptors")class AdapterTest(unittest.TestCase):def setUp(self) -> None:self.tmp = tempfile.TemporaryDirectory()@@ -90,6 +92,12 @@ class AdapterTest(unittest.TestCase):self.assertIn("Electron", kill_names)self.assertIn("Google Chrome", kill_names)+ def test_kill_policy_covers_libreoffice_launch(self) -> None:+ descriptor = {"apps": [{"argv": ["libreoffice", "--calc"]}]}+ with mock.patch.object(cdb_adapter, "_descriptor", return_value=descriptor):+ names, _ = cdb_adapter.kill_policy(self.spec)+ self.assertEqual(names, ["soffice", "soffice.bin"])+def test_brief_and_substitution(self) -> None:task = cdb_adapter.CdbTask(self.spec, Path(self.tmp.name))brief = task.brief()@@ -247,6 +255,70 @@ class AdapterTest(unittest.TestCase):cdb_adapter.clear_leftovers(self.spec) # must be a no-op, never a pkill on a real desktop[email protected](sys.platform == "linux", "Linux launch descriptors")+class LinuxAdapterTest(unittest.TestCase):+ def test_descriptor_launch_preserves_session_and_honors_unset_env(self) -> None:+ from unittest import mock++ with tempfile.TemporaryDirectory() as tmp:+ root = Path(tmp)+ bundle = root / "tasks/shared/syn"+ (bundle / "platform").mkdir(parents=True)+ descriptor = {+ "workspace_root": str(root / "workspace"),+ "semantics": {+ "reset": {+ "setup": [+ sys.executable,+ "-c",+ "import os; assert os.environ['DISPLAY'] == ':unit-display'; assert os.environ['DBUS_SESSION_BUS_ADDRESS'] == 'unix:path=/unit-bus'",+ ],+ "verify": [sys.executable, "-c", "pass"],+ }+ },+ "apps": [+ {+ "id": "desk",+ "kind": "electron",+ "command": [+ sys.executable,+ "-c",+ "import json, os; print(json.dumps(dict(os.environ)))",+ ],+ "env": {"ELECTRON_RUN_AS_NODE": "1"},+ "unset_env": ["ELECTRON_RUN_AS_NODE"],+ }+ ],+ }+ (bundle / "platform/launch.linux.json").write_text(json.dumps(descriptor))+ with mock.patch.dict(+ os.environ,+ {+ "CDB_TASKPACK": str(root),+ "CDB_BENCH_DISPOSABLE": "0",+ "DISPLAY": ":unit-display",+ "DBUS_SESSION_BUS_ADDRESS": "unix:path=/unit-bus",+ },+ ):+ task = cdb_adapter.CdbTask({"pack_task": "shared/syn"}, root)+ task.reset()+ try:+ task.start_apps()+ task.procs[0].wait(timeout=5)+ observed = json.loads((root / "app-desk.log").read_text())+ self.assertEqual(observed["DISPLAY"], ":unit-display")+ self.assertEqual(observed["DBUS_SESSION_BUS_ADDRESS"], "unix:path=/unit-bus")+ self.assertNotIn("ELECTRON_RUN_AS_NODE", observed)+ finally:+ task.stop_apps()+ self.assertEqual(+ cdb_adapter.with_force_accessibility(+ ["chromium", "https://example.invalid"], "browser"+ ),+ ["chromium", ca.FORCE_ACCESSIBILITY_FLAG, "https://example.invalid"],+ )++class ArgvAndScanTest(unittest.TestCase):def test_coding_tools_only_when_asked(self) -> None:base = dict(@@ -284,6 +356,7 @@ class ArgvAndScanTest(unittest.TestCase):class WebWiringTest(unittest.TestCase):+ @unittest.skipUnless(sys.platform == "darwin", "vendor browser arm requires macOS")def test_browser_selection_row_manifest_and_dry_run_prompt(self) -> None:import contextlibimport hashlib@@ -370,6 +443,7 @@ class WebWiringTest(unittest.TestCase):]self.assertEqual(rb.evaluator_peeks(events, []), ["Read:oracle", "Read:/__state"])+ @unittest.skipUnless(sys.platform == "darwin", "WEB task pack ships macOS descriptors only")def test_setup_brief_and_profile_reset(self) -> None:import subprocessdiff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_pilot_runner.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_pilot_runner.pyindex 136c914bcc..272c8ef0a4 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_pilot_runner.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_pilot_runner.py@@ -8,8 +8,11 @@ from __future__ import annotationsimport argparseimport jsonimport os+import signal+import subprocessimport tomllibimport sys+import timeimport tempfileimport unittestfrom pathlib import Path@@ -18,10 +21,35 @@ from unittest import mockHERE = Path(__file__).resolve().parentsys.path.insert(0, str(HERE.parent))+import arms # noqa: E402import codex_events # noqa: E402+def _x11_ready() -> bool:+ if sys.platform != "linux" or not os.environ.get("DISPLAY"):+ return False+ try:+ return (+ subprocess.run(+ ["xdpyinfo", "-display", os.environ["DISPLAY"]],+ capture_output=True,+ timeout=10,+ ).returncode+ == 0+ )+ except (OSError, subprocess.SubprocessError):+ return False+++def _has_gi() -> bool:+ try:+ import gi # noqa: F401+ except ImportError:+ return False+ return True++def _stamp(events: list[tuple[float, dict]]) -> Path:handle = tempfile.NamedTemporaryFile("w", suffix=".tsv", delete=False)for stamp, event in events:@@ -41,6 +69,556 @@ def _mcp(item_id: str, server: str, tool: str, arguments: dict, status: str = "c}+class DesktopRuntimeTest(unittest.TestCase):+ def test_platform_and_session_environment(self) -> None:+ import desktop_runtime as desktop++ self.assertEqual(desktop.detect_platform("Darwin"), "macos")+ self.assertEqual(desktop.detect_platform("Linux"), "linux")+ with self.assertRaises(ValueError):+ desktop.detect_platform("Windows")+ from unittest.mock import patch++ with (+ tempfile.TemporaryDirectory() as tmp,+ patch.object(desktop, "detect_platform", return_value="linux"),+ patch.dict(+ os.environ,+ {+ "DISPLAY": ":92",+ "DBUS_SESSION_BUS_ADDRESS": "unix:path=/tmp/bus",+ "SECRET_TOKEN": "secret",+ },+ ),+ ):+ env = desktop.desktop_env(Path(tmp))+ self.assertEqual(env["DISPLAY"], ":92")+ self.assertEqual(env["DBUS_SESSION_BUS_ADDRESS"], "unix:path=/tmp/bus")+ self.assertNotIn("SECRET_TOKEN", env)++ def test_linux_environment_keeps_default_xauthority_after_home_swap(self) -> None:+ import desktop_runtime as desktop+ from unittest.mock import patch++ with tempfile.TemporaryDirectory() as tmp:+ orig = Path(tmp) / "orig"+ orig.mkdir()+ (orig / ".Xauthority").write_text("x")+ environ = {"HOME": str(orig), "DISPLAY": ":92"}+ with (+ patch.object(desktop, "detect_platform", return_value="linux"),+ patch.dict(os.environ, environ, clear=True),+ ):+ env = desktop.desktop_env(Path(tmp) / "private")+ self.assertEqual(env["XAUTHORITY"], str(orig / ".Xauthority"))+ self.assertEqual(env["HOME"], str(Path(tmp) / "private"))++ def test_macos_environment_uses_closed_path_and_locale(self) -> None:+ import desktop_runtime as desktop+ import run_pilot as pilot+ from unittest.mock import patch++ with (+ tempfile.TemporaryDirectory() as tmp,+ patch.object(desktop, "detect_platform", return_value="macos"),+ patch.dict(os.environ, {"PATH": "/injected/bin", "LANG": "injected", "DISPLAY": ":92"}),+ ):+ env = desktop.desktop_env(Path("/tmp/home"))+ self.assertEqual(env["PATH"], "/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin")+ self.assertEqual(env["LANG"], "en_US.UTF-8")+ self.assertNotIn("DISPLAY", env)+ home = Path(tmp) / "home"+ app_env = pilot.clean_app_env(home)+ self.assertEqual(app_env["PATH"], f"/opt/homebrew/bin:{arms.CLOSED_PATH}")+ self.assertEqual(app_env["TMPDIR"], str(home / "tmp"))+ self.assertTrue((home / "tmp").is_dir())++ def test_linux_environment_uses_trial_tmpdir(self) -> None:+ import desktop_runtime as desktop+ import run_pilot as pilot+ from unittest.mock import patch++ with (+ tempfile.TemporaryDirectory() as tmp,+ patch.object(desktop, "detect_platform", return_value="linux"),+ patch.dict(os.environ, {"TMPDIR": "/runner/tmp"}),+ ):+ home = Path(tmp) / "home"+ env = pilot.clean_app_env(home)+ self.assertEqual(env["TMPDIR"], str(home / "tmp"))+ self.assertTrue((home / "tmp").is_dir())++ def test_stop_group_tolerates_permission_error_after_termination(self) -> None:+ import desktop_runtime as desktop+ from unittest.mock import patch++ with (+ patch.object(desktop.os, "killpg", side_effect=[None, PermissionError]),+ patch.object(desktop.time, "sleep"),+ ):+ desktop.stop_group(12345)++ @unittest.skipUnless(sys.platform == "linux", "Linux desktop")+ def test_missing_display_is_an_actionable_doctor_failure(self) -> None:+ import desktop_runtime as desktop+ from unittest.mock import patch++ with tempfile.TemporaryDirectory() as tmp, patch.dict(os.environ, {"DISPLAY": ""}):+ checks = desktop.doctor(+ Path("/nonexistent/cua-driver"), "/tmp/no-driver.sock", Path(tmp)+ )+ by_name = {c.name: c for c in checks}+ self.assertEqual(by_name["display"].status, "fail")+ self.assertIn("DISPLAY", by_name["display"].detail)+ self.assertEqual(by_name["driver"].status, "fail")++ @unittest.skipUnless(sys.platform == "linux", "Linux desktop")+ def test_unreachable_desktop_and_recording_dependencies_fail_doctor(self) -> None:+ import desktop_runtime as desktop+ from unittest.mock import patch++ with tempfile.TemporaryDirectory() as tmp:+ root = Path(tmp)+ with patch.dict(+ os.environ,+ {+ "DISPLAY": ":9876",+ "DBUS_SESSION_BUS_ADDRESS": "unix:path=/nonexistent/bus",+ "PATH": str(root),+ },+ ):+ checks = {+ c.name: c+ for c in desktop.doctor(root / "driver", str(root / "socket"), root / "home")+ }+ for name in ("display", "at-spi", "ffmpeg", "driver"):+ self.assertEqual(checks[name].status, "fail", name)+ ffmpeg = root / "ffmpeg"+ ffmpeg.write_text("#!/bin/sh\nprintf 'Devices: no X11 support\\n'\n")+ ffmpeg.chmod(0o700)+ checks = {+ c.name: c+ for c in desktop.doctor(root / "driver", str(root / "socket"), root / "home")+ }+ self.assertEqual(checks["ffmpeg"].status, "fail")+ self.assertIn("x11grab", checks["ffmpeg"].detail)++ @unittest.skipUnless(sys.platform == "linux", "Linux desktop")+ def test_linux_health_requires_capabilities_not_tcc(self) -> None:+ import desktop_runtime as desktop++ with tempfile.TemporaryDirectory() as tmp:+ root = Path(tmp)+ binary = root / "driver"+ binary.write_text(+ "#!/usr/bin/python3\nimport json\nprint(json.dumps({'checks': [{'name': 'tcc_accessibility', 'status': 'skip'}, {'name': 'tcc_screen_recording', 'status': 'skip'}, {'name': 'ax_capability', 'status': 'pass'}, {'name': 'screen_capture_capability', 'status': 'pass'}]}))\n"+ )+ binary.chmod(0o700)+ health = desktop.cua_health(binary, str(root / "socket"), root / "home")+ self.assertTrue(desktop.capabilities_ok(health))+ binary.write_text(+ binary.read_text().replace(+ "'ax_capability', 'status': 'pass'", "'ax_capability', 'status': 'fail'"+ )+ )+ self.assertFalse(+ desktop.capabilities_ok(+ desktop.cua_health(binary, str(root / "socket"), root / "home")+ )+ )++ @unittest.skipUnless(_x11_ready() and _has_gi(), "headless X11 session with GTK required")+ def test_task_helpers_emit_evaluator_compatible_initial_state_and_evidence(self) -> None:+ import desktop_runtime as desktop+ import run_bench as runner+ import run_pilot as pilot++ with tempfile.TemporaryDirectory() as tmp:+ root = Path(tmp)+ tasks = pilot.load_tasks()+ for task_id in ("MB-09", "MB-10", "MB-11", "IR-01", "IR-02", "IR-03", "IR-04"):+ with self.subTest(task=task_id):+ trial = root / task_id+ trial.mkdir()+ (trial / "artifacts").mkdir()+ _, paths = pilot.prepare_probe(tasks[task_id], 7, trial, root / "BenchLab.app")+ proc = pilot.launch_lab(+ tasks[task_id],+ 7,+ paths,+ root / "BenchLab.app",+ pilot.clean_app_env(trial / "home"),+ )+ try:+ result = pilot.evaluate_probe(tasks[task_id], 7, paths, trial / "artifacts")+ self.assertFalse(result["passed"])+ state = json.loads(paths["state"].read_text())+ self.assertEqual(state["seed"], 7)+ self.assertEqual(+ json.loads(paths["events"].read_text().splitlines()[0])["type"],+ "app_start",+ )+ self.assertEqual(+ Path(f"/proc/{proc.pid}/comm").read_text().strip(), "BenchLab"+ )+ if task_id == "MB-11":+ desktop.capture_screenshot(os.environ["DISPLAY"], trial / "screen.png")+ import gi++ gi.require_version("GdkPixbuf", "2.0")+ from gi.repository import GdkPixbuf++ image = GdkPixbuf.Pixbuf.new_from_file(str(trial / "screen.png"))+ geometry = subprocess.check_output(+ ["xdotool", "getdisplaygeometry"], text=True, timeout=5+ )+ self.assertEqual(+ (image.get_width(), image.get_height()),+ tuple(map(int, geometry.split())),+ )+ runner.kill_bench_apps()+ proc.wait(timeout=5)+ self.assertIsNotNone(proc.returncode)+ finally:+ desktop.stop_group(proc.pid)+ proc.wait(timeout=5)+ desktop.stop_group(proc.pid)+ checks = {+ c.name: c+ for c in desktop.doctor(+ Path("/nonexistent/driver"), str(root / "socket"), root / "doctor"+ )+ }+ for name in ("display", "at-spi", "ffmpeg"):+ self.assertEqual(checks[name].status, "pass", checks[name].detail)++ @unittest.skipUnless(_x11_ready() and _has_gi(), "headless X11 session with GTK required")+ def test_consent_dialog_survives_window_close(self) -> None:+ script = r"""+import argparse, json, sys, tempfile+from pathlib import Path+sys.path.insert(0, sys.argv[1])+import gi+gi.require_version("Gtk", "3.0")+from gi.repository import Gtk+import benchlab++def pump(n=60):+ for _ in range(n):+ while Gtk.events_pending():+ Gtk.main_iteration_do(False)+ import time; time.sleep(0.01)++def dlg():+ for w in Gtk.Window.list_toplevels():+ if w.get_title() == "Cookie preferences" and w.get_visible():+ return w+ return None++with tempfile.TemporaryDirectory() as tmp:+ lab = benchlab.Lab(argparse.Namespace(+ mode="irconsent", seed=7, state=Path(tmp) / "s.json", events=Path(tmp) / "e.jsonl"))+ lab.fields["email"].set_text("[email protected]")+ for _ in range(100):+ pump(5)+ if dlg():+ break+ d = dlg()+ out = {"opened": d is not None}+ d.close(); pump()+ out["after_close"] = [dlg() is not None, lab.state["consent_shown"], list(lab.state["consent_answers"])]+ d.response(-1); d.response(99); pump()+ out["after_invalid"] = [dlg() is not None, list(lab.state["consent_answers"])]+ d.response(0); pump()+ out["after_button"] = [dlg() is not None, list(lab.state["consent_answers"])]+ lab.signup(); pump()+ out["submit"] = [lab.state["submit_count"], lab.state["submits_blocked"]]+ print(json.dumps(out))+"""+ with tempfile.TemporaryDirectory() as tmp:+ proc = subprocess.run(+ [sys.executable, "-c", script, str(HERE.parent / "linux")],+ capture_output=True,+ text=True,+ timeout=60,+ cwd=tmp,+ )+ self.assertEqual(proc.returncode, 0, proc.stderr)+ out = json.loads(proc.stdout.strip().splitlines()[-1])+ self.assertTrue(out["opened"])+ self.assertEqual(out["after_close"], [True, True, []])+ self.assertEqual(out["after_invalid"], [True, []])+ self.assertFalse(out["after_button"][0])+ self.assertEqual([a["answer"] for a in out["after_button"][1]], ["reject"])+ self.assertEqual(out["submit"], [1, 0])++ @unittest.skipUnless(_x11_ready() and _has_gi(), "headless X11 session with GTK required")+ def test_real_gtk_interactions_are_accepted_by_unchanged_evaluators(self) -> None:+ import desktop_runtime as desktop+ import run_pilot as pilot+ import gi++ gi.require_version("Atspi", "2.0")+ from gi.repository import Atspi++ sys.path.insert(0, str(HERE.parent / "probes" / "_common"))+ import benchlab_common as common++ def input_(*argv):+ subprocess.run(["xdotool", *map(str, argv)], check=True, timeout=10)+ time.sleep(0.15)++ def find(name, role=None):+ # at-spi2-core < 2.53 reports GTK buttons as "push button"; >= 2.53 as "button".+ buttons = {"button", "push button"}+ roles = buttons if role in buttons else {role}++ def walk(node):+ if node.get_name() == name and (role is None or node.get_role_name() in roles):+ return node+ for i in range(node.get_child_count()):+ found = walk(node.get_child_at_index(i))+ if found is not None:+ return found+ return None++ deadline = time.monotonic() + 5+ while time.monotonic() < deadline:+ found = walk(Atspi.get_desktop(0))+ if found is not None:+ bounds = found.get_component_iface().get_extents(Atspi.CoordType.SCREEN)+ if bounds.width > 0 and bounds.height > 0:+ return bounds.x + bounds.width // 2, bounds.y + bounds.height // 2+ time.sleep(0.1)+ self.fail(f"accessible control not found: {name} / {role}")++ def click(name, role=None):+ x, y = find(name, role)+ input_("mousemove", x, y, "click", 1)++ def fill(name, value):+ click(name, "text")+ input_("key", "ctrl+a")+ input_("type", "--clearmodifiers", value)++ def combo(name, choices, value):+ click(name, "combo box")+ input_("key", "Home", *(["Down"] * choices.index(value)), "Return")++ with tempfile.TemporaryDirectory() as tmp:+ root = Path(tmp)+ tasks = pilot.load_tasks()+ for task_id in ("MB-09", "MB-10", "MB-11", "IR-01", "IR-02", "IR-03", "IR-04"):+ with self.subTest(task=task_id):+ trial = root / task_id+ (trial / "artifacts").mkdir(parents=True)+ _, paths = pilot.prepare_probe(tasks[task_id], 7, trial, root / "BenchLab.app")+ proc = pilot.launch_lab(+ tasks[task_id],+ 7,+ paths,+ root / "BenchLab.app",+ pilot.clean_app_env(trial / "home"),+ )+ try:+ mode = json.loads(paths["state"].read_text())["mode"]+ params = getattr(common, "derive_" + mode)(7)+ if mode == "listdrag":+ order = list(params["initial"])+ for target, item in enumerate(params["target"]):+ source = order.index(item)+ if source == target:+ continue+ x, y = find(item, "label")+ tx, ty = find(order[target], "label")+ input_("mousemove", x, y, "mousedown", 1)+ input_("mousemove", x + 20, y + 8)+ input_("mousemove", tx, ty)+ input_("mousemove", tx + 3, ty + 3)+ time.sleep(0.3)+ input_("mouseup", 1)+ order.pop(source)+ order.insert(target, item)+ click("Done", "push button")+ elif mode == "hover":+ x, y = find("Hover here to reveal actions", "label")+ input_("mousemove", x, y)+ click(params["target"], "push button")+ elif mode == "tooltip":+ x, y = find(params["target"], "label")+ input_("mousemove", x, y)+ time.sleep(1.5)+ fill("Code", params["target_code"])+ click("Submit", "push button")+ elif mode == "irmodal":+ fill("Name", params["name"])+ fill("Email", params["email"])+ click("Next", "push button")+ click("Don’t Allow", "push button")+ combo("Team", common.IR_TEAMS, params["team"])+ fill("Seats", str(params["seats"]))+ click("Submit", "push button")+ elif mode == "irbanner":+ fill("Quantity", str(params["quantity"]))+ time.sleep(0.5)+ click("Later", "push button")+ combo("Priority", common.PRIORITIES, params["priority"])+ click("Apply", "push button")+ elif mode == "irunsaved":+ click(params["titles"][params["status_index"]], "push button")+ combo(+ "Status",+ [common.IR_INITIAL_STATUS, *common.IR_STATUS_WORDS],+ params["status"],+ )+ click(params["titles"][params["rename_index"]], "push button")+ click("Save", "push button")+ fill("Title", params["new_title"])+ click("Done", "push button")+ click("Save", "push button")+ else:+ fill("Email", params["email"])+ time.sleep(0.5)+ click("Reject optional", "push button")+ combo("Plan", common.IR_PLANS, params["plan"])+ if params["digest"]:+ click("Email digest", "check box")+ click("Sign up", "push button")+ time.sleep(0.4)+ result = pilot.evaluate_probe(tasks[task_id], 7, paths, trial / "artifacts")+ self.assertTrue(result["passed"], result)+ if mode == "irunsaved":+ fill("Title", "Unsafe edit")+ click("Done", "push button")+ click("Don’t Save", "push button")+ rejected = pilot.evaluate_probe(+ tasks[task_id], 7, paths, trial / "artifacts"+ )+ self.assertFalse(rejected["passed"])+ self.assertFalse(rejected["checks"]["no_work_discarded"])+ finally:+ desktop.stop_group(proc.pid)+ proc.wait(timeout=5)++ @unittest.skipUnless(+ _x11_ready() and os.environ.get("CUA_DRIVER_BIN"),+ "CUA_DRIVER_BIN required for driver smoke",+ )+ def test_real_driver_health_and_recording(self) -> None:+ import desktop_runtime as desktop+ import recorder++ with tempfile.TemporaryDirectory() as tmp:+ root = Path(tmp)+ binary = Path(os.environ["CUA_DRIVER_BIN"])+ socket = str(root / "driver.sock")+ proc = desktop.start_driver(binary, socket, root / "daemon", overlay=False)+ capture = recorder.CuaRecorder(+ binary, socket, desktop.desktop_env(root / "daemon" / "home")+ )+ try:+ for check in desktop.doctor(binary, socket, root / "doctor"):+ self.assertEqual(check.status, "pass", check.detail)+ self.assertTrue(capture.start(root / "recording"), capture.last_error)+ time.sleep(2)+ video = capture.stop()+ self.assertIsNotNone(video, capture.last_error)+ self.assertGreater(recorder.ffprobe_duration(video), 0.5)+ capture.stop()+ finally:+ capture.close()+ capture.close()+ desktop.stop_group(proc.pid)+ proc.wait(timeout=5)+ desktop.stop_group(proc.pid)++ @unittest.skipUnless(_x11_ready(), "headless X11 session required")+ def test_sentinel_witness_and_helper_cleanup(self) -> None:+ import desktop_runtime as desktop+ import run_bench as runner+ import run_pilot as pilot++ with tempfile.TemporaryDirectory() as tmp:+ root = Path(tmp)+ sentinel = pilot.Sentinel(+ root / "BenchSentinel.app", root / "sentinel.jsonl", ["--user-activity"]+ )+ other = None+ try:+ sentinel.start()+ self.assertIsNotNone(sentinel.proc)+ windows = subprocess.check_output(+ ["xdotool", "search", "--name", "^User activity$"], text=True, timeout=10+ ).splitlines()+ self.assertTrue(windows)+ sentinel.activate()+ # A display reused across runs may still hold the pointer at the move target below.+ subprocess.run(["xdotool", "mousemove", "10", "10"], check=True, timeout=10)+ sentinel.toggle_armed()+ time.sleep(2.5)+ other = desktop.start_app(+ [+ "/usr/bin/python3",+ "-c",+ "import gi; gi.require_version('Gtk','3.0'); from gi.repository import Gtk; w=Gtk.Window(title='Witness target'); w.show_all(); Gtk.main()",+ ],+ root / "home",+ root / "other.log",+ )+ time.sleep(1)+ window = subprocess.check_output(+ ["xdotool", "search", "--name", "^Witness target$"], text=True+ ).splitlines()[0]+ subprocess.run(+ ["xdotool", "windowactivate", "--sync", window, "mousemove", "310", "270"],+ check=True,+ timeout=10,+ )+ time.sleep(2)+ sentinel.toggle_armed()+ summary = sentinel.summary()+ self.assertTrue(summary["available"])+ self.assertGreater(summary["front_changes"], 0)+ self.assertGreater(summary["pointer_max_deviation_px"], 10)+ self.assertGreater(summary["user_activity"]["blocked"], 0)+ for key in (+ "clipboard_changes",+ "space_switches",+ "sheets_shown",+ "windows_raised",+ ):+ self.assertIsNotNone(summary[key], key)+ proc = sentinel.proc+ runner.kill_bench_apps()+ proc.wait(timeout=5)+ self.assertIsNotNone(proc.returncode)+ finally:+ sentinel.stop()+ sentinel.stop()+ if other:+ desktop.stop_group(other.pid)+ other.wait(timeout=5)+++class SentinelVisibilityTest(unittest.TestCase):+ @unittest.skipUnless(_has_gi(), "GTK required")+ def test_only_visible_current_workspace_windows_count(self) -> None:+ from unittest import mock++ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "linux"))+ import sentinel++ props = {+ 1: {"_NET_WM_DESKTOP": "0", "_NET_WM_STATE": "_NET_WM_STATE_ABOVE"},+ 2: {"_NET_WM_DESKTOP": "1", "_NET_WM_STATE": ""},+ 3: {"_NET_WM_DESKTOP": "4294967295", "_NET_WM_STATE": ""},+ 4: {"_NET_WM_DESKTOP": "0", "_NET_WM_STATE": "_NET_WM_STATE_HIDDEN"},+ }+ with mock.patch.object(sentinel, "xprop", lambda _i, w, *_a: props[int(w, 16)]):+ seen = [w for w in props if sentinel.visible_here(w, "0")]+ self.assertEqual(seen, [1, 3])++class CodexEventsTest(unittest.TestCase):def test_classes_latency_and_tokens(self) -> None:events = [@@ -171,6 +749,69 @@ class CodexEventsTest(unittest.TestCase):self.assertIsNone(codex_events.rollout_usage(Path(tmp) / "missing"))+class RecorderTest(unittest.TestCase):+ def setUp(self) -> None:+ import recorder++ self.tmp = tempfile.TemporaryDirectory()+ self.addCleanup(self.tmp.cleanup)+ self.video = Path(self.tmp.name) / "v.mp4"+ self.video.write_bytes(b"x")+ self.rec = recorder.CuaRecorder(Path("/nonexistent"), "sock", {})+ self.rec.active_dir = Path(self.tmp.name)+ self.outcomes: list = []+ self.calls: dict[str, int] = {}+ self.closed = 0+ test = self++ class Fake:+ def call_tool(self, name, args, timeout=0):+ test.calls[name] = test.calls.get(name, 0) + 1+ if name == "get_recording_state":+ raise TimeoutError("state")+ outcome = test.outcomes.pop(0)+ if isinstance(outcome, type):+ raise outcome("boom")+ return outcome++ def close(self):+ test.closed += 1++ def connect():+ self.rec.client = Fake()+ return self.rec.client++ self.rec._connect = connect++ def test_stop_retries_after_transient_failure(self) -> None:+ self.outcomes = [TimeoutError, {"last_video_path": str(self.video)}]+ self.assertIsNone(self.rec.stop())+ self.assertIsNotNone(self.rec.active_dir)+ self.assertTrue(self.rec.last_error.startswith("TimeoutError"))+ self.assertEqual(self.rec.stop(), self.video)+ self.assertIsNone(self.rec.active_dir)+ self.rec.close()+ self.assertEqual(self.calls["stop_recording"], 2)+ self.assertEqual(self.closed, 1)++ def test_close_retries_unconfirmed_stop(self) -> None:+ self.outcomes = [TimeoutError, TimeoutError, {"last_video_path": str(self.video)}]+ self.rec.stop()+ self.rec.stop()+ self.rec.close()+ self.assertEqual(self.calls["stop_recording"], 3)+ self.assertEqual(self.rec.last_video, self.video)+ self.assertIsNone(self.rec.active_dir)+ self.assertEqual(self.closed, 1)++ def test_state_lookup_failure_after_confirmed_stop_clears_state(self) -> None:+ self.outcomes = [{}]+ self.assertIsNone(self.rec.stop())+ self.assertIsNone(self.rec.active_dir)+ self.rec.close()+ self.assertEqual(self.calls["stop_recording"], 1)++class ArmsTest(unittest.TestCase):def test_gateway_common_config_matches_original_text(self) -> None:with mock.patch.dict(os.environ, {"CDB_CODEX_AUTH": "gateway"}):diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_runner_flow.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_runner_flow.pyindex 94b0e72f69..f64a9299f6 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_runner_flow.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_runner_flow.py@@ -19,6 +19,8 @@ import unittestfrom pathlib import Pathfrom unittest import mock+import pytest+HERE = Path(__file__).resolve().parentsys.path.insert(0, str(HERE.parent))@@ -88,6 +90,7 @@ def test_browser_cli_auth_rejected_before_provisioning(tmp_path):rb.make_ctx(args)[email protected](sys.platform != "darwin", reason="browser CLI arm is macOS-only")def test_browser_cli_run_fails_closed(tmp_path):with tempfile.TemporaryDirectory() as tmp:ctx = make_ctx(tmp, no_model_preflight=True, build_dir=None)@@ -232,11 +235,16 @@ class CodexAuthPreflightTest(unittest.TestCase):stack.enter_context(mock.patch.dict("os.environ",- {"CDB_CODEX_AUTH": "chatgpt", "CDB_CODEX_AUTH_FILE": str(source)},+ {+ "CDB_CODEX_AUTH": "chatgpt",+ "CDB_CODEX_AUTH_FILE": str(source),+ "DISPLAY": os.environ.get("DISPLAY") or ":99",+ },))ctx.versions["codex_native"] = rb.cxn.observed(Path("/nonexistent"))for owner, name, result in (+ (rb.desktop, "capture_screenshot", None),(rb, "task_spec", {}),(rb, "kill_bench_apps", None),(rb, "reset_needs_apps", None),@@ -320,6 +328,132 @@ class CodexAuthPreflightTest(unittest.TestCase):)fetch.assert_called_once()+ def linux_evidence_trial(self, *, replace_link: bool, screenshot_error, copy_error=None):+ with tempfile.TemporaryDirectory() as tmp, contextlib.ExitStack() as stack:+ ctx = make_ctx(+ tmp,+ timeout_s=60,+ max_turns=3,+ effort="low",+ idle_min=0,+ no_sentinel=True,+ build_dir=tmp,+ check_occlusion=False,+ tell_budget=False,+ )+ ctx.tasks["MB-01"] = argparse.Namespace(id="MB-01", tags=[], timeout_s=60)+ arm = "cx-codex-cu"+ entry = {**ENTRY, "arm": arm}+ source = Path(tmp) / "shared-auth.json"+ source.write_text("FAKE_TOKEN")+ mcp = Path(tmp) / "mcp.json"+ mcp.write_text(+ json.dumps(+ {+ "mcpServers": {+ "sky": {+ "command": "node",+ "env": {"CUA_REPL_ENABLED_SURFACES": "computer"},+ }+ }+ }+ )+ )+ ctx.mcp[arm] = (mcp, "cua_repl")+ stack.enter_context(+ mock.patch.dict(+ "os.environ",+ {+ "CDB_CODEX_AUTH": "chatgpt",+ "CDB_CODEX_AUTH_FILE": str(source),+ "DISPLAY": ":99",+ },+ )+ )+ ctx.versions["codex_native"] = rb.cxn.observed(Path("/nonexistent"))+ mocks = {}+ for owner, name, result in (+ (rb.desktop, "detect_platform", "linux"),+ (rb, "task_spec", {}),+ (rb, "kill_bench_apps", None),+ (rb, "reset_needs_apps", None),+ (rb.subprocess, "run", None),+ (rb.pilot, "hid_idle_seconds", 1000),+ (rb.pilot, "sweep_processes", 0),+ (rb.pilot, "prepare_probe", ("BRIEF", {})),+ (rb.pilot, "launch_lab", None),+ (rb.ca, "prepare_cwd", Path(tmp)),+ (rb, "apps_running", {}),+ (rb, "evaluate_probe", {"passed": True}),+ (rb, "FrontWatcher", None),+ ):+ mocks[name] = stack.enter_context(+ mock.patch.object(owner, name, return_value=result)+ )+ sentinel = stack.enter_context(mock.patch.object(rb.pilot, "Sentinel"))+ sentinel.return_value.enabled = False+ sentinel.return_value.summary.return_value = {}+ stack.enter_context(mock.patch.object(rb.pilot, "IdleSampler"))++ def capture(display, path):+ if path.name == "startup.png":+ return None+ if screenshot_error is not None:+ raise screenshot_error++ stack.enter_context(+ mock.patch.object(rb.desktop, "capture_screenshot", side_effect=capture)+ )+ log = Path(tmp) / "recorder.log"+ log.write_text("log")+ stack.enter_context(mock.patch.object(rb.ca, "RECORDER_STATE", Path(tmp)))+ Path(tmp, "daemon.log").write_text("log")+ if copy_error is not None:+ stack.enter_context(+ mock.patch.object(rb.shutil, "copyfile", side_effect=copy_error)+ )++ def run_codex(cmd, env, *args, **kwargs):+ link = Path(env["CODEX_HOME"]) / "auth.json"+ self.assertTrue(link.is_symlink())+ if replace_link:+ link.unlink()+ link.write_text("FAKE_TOKEN")+ return {"returncode": 0, "result_seen": True, "wall_s": 0}++ stack.enter_context(mock.patch.object(rb.cxn, "run_codex", side_effect=run_codex))+ row = rb.run_attempt(ctx, entry, 1)+ trial = ctx.run_dir / "trials" / entry["trial_id"] / "a1"+ link = trial / "codex-home/.codex/auth.json"+ self.assertFalse(link.is_symlink())+ self.assertEqual(source.read_text(), "FAKE_TOKEN")+ mocks["sweep_processes"].assert_called()+ mocks["kill_bench_apps"].assert_called()+ errors = trial / "artifacts" / "screenshot-error.txt"+ return row, (errors.read_text() if errors.is_file() else None)++ def test_linux_screenshot_failure_still_unlinks_auth(self) -> None:+ row, errors = self.linux_evidence_trial(+ replace_link=False, screenshot_error=AttributeError("secret /path")+ )+ self.assertNotEqual(row["infra_failure"], "harness_exception")+ self.assertEqual(errors, "screenshot: AttributeError")++ def test_linux_log_copy_failure_still_unlinks_auth(self) -> None:+ row, errors = self.linux_evidence_trial(+ replace_link=False, screenshot_error=None, copy_error=OSError("secret /path")+ )+ self.assertNotEqual(row["infra_failure"], "harness_exception")+ self.assertIn("recorder log: OSError", errors)+ self.assertNotIn("secret", errors)++ def test_linux_evidence_failure_keeps_credential_integrity_error(self) -> None:+ row, _ = self.linux_evidence_trial(+ replace_link=True, screenshot_error=AttributeError("boom")+ )+ self.assertEqual(row["infra_failure"], "harness_exception")+ self.assertIn("shared ChatGPT credential symlink replaced", row["infra_message"])+def attempt_row(attempt: int,@@ -631,6 +765,11 @@ class PreflightTest(unittest.TestCase):class ExecuteTrialTest(unittest.TestCase):+ def setUp(self) -> None:+ disk = mock.patch.object(rb.rec, "free_gb", return_value=100.0)+ disk.start()+ self.addCleanup(disk.stop)+def test_daemon_setup_failure_retries_with_restart_then_excludes(self) -> None:with tempfile.TemporaryDirectory() as tmp:ctx = make_ctx(tmp)@@ -757,6 +896,11 @@ class ExecuteTrialTest(unittest.TestCase):class GateTest(unittest.TestCase):+ def setUp(self) -> None:+ disk = mock.patch.object(rb.rec, "free_gb", return_value=100.0)+ disk.start()+ self.addCleanup(disk.stop)+def test_quota_stop_rule_blocks_new_trials(self) -> None:with tempfile.TemporaryDirectory() as tmp:ctx = make_ctx(tmp)@@ -876,6 +1020,63 @@ class FinalizeTest(unittest.TestCase):self.assertEqual(core.completed_trial_ids(rows), {"a"})+class LinuxSetupTest(unittest.TestCase):+ def test_macos_arms_are_skipped_before_setup(self) -> None:+ import contextlib+ import io+ import desktop_runtime as desktop++ tmp = tempfile.TemporaryDirectory()+ self.addCleanup(tmp.cleanup)+ args = rb.build_parser().parse_args(+ [+ "dry-run",+ "--ledger",+ str(Path(tmp.name) / "ledger.jsonl"),+ "--bench-v2",+ "--arms",+ "cc-cua-driver",+ "cc-codex-cu",+ "cc-claude-cu-helper",+ ]+ )+ output = io.StringIO()+ with (+ mock.patch.object(desktop, "detect_platform", return_value="linux"),+ contextlib.redirect_stderr(output),+ ):+ ctx = rb.make_ctx(args)+ self.assertEqual(ctx.arm_names, ["cc-cua-driver"])+ self.assertIn("unsupported-platform: cc-codex-cu requires macOS", output.getvalue())+ self.assertIn(+ "unsupported-platform: cc-claude-cu-helper requires macOS",+ ctx.versions["unsupported_arms"],+ )++ def test_no_supported_arms_fails_clearly(self) -> None:+ import desktop_runtime as desktop++ args = rb.build_parser().parse_args(["dry-run", "--arms", "cc-codex-cu"])+ with (+ mock.patch.object(desktop, "detect_platform", return_value="linux"),+ self.assertRaisesRegex(SystemExit, "no supported arms remain"),+ ):+ rb.make_ctx(args)++ def test_linux_pack_descriptor_selection(self) -> None:+ import cdb_adapter+ import desktop_runtime as desktop++ with mock.patch.object(desktop, "detect_platform", return_value="linux"):+ descriptor = cdb_adapter._descriptor({"pack_task": "shared/cdb-s02"})+ self.assertEqual(descriptor["platform"], "linux")+ self.assertTrue(any("libreoffice" in " ".join(a["command"]) for a in descriptor["apps"]))+ with mock.patch.object(desktop, "detect_platform", return_value="macos"):+ self.assertEqual(+ cdb_adapter._descriptor({"pack_task": "shared/cdb-s02"})["platform"], "macos"+ )++class MiscTest(unittest.TestCase):def test_strip_bulk_removes_screenshots_but_keeps_text(self) -> None:big = "A" * 50000diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tools/fixture.py b/libs/cua-driver-bench/automated-eval/macos_bench/tools/fixture.pyindex ca74f5d6a6..0216d9556e 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/tools/fixture.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tools/fixture.py@@ -28,6 +28,9 @@ from pathlib import Pathfrom typing import AnyROOT = Path(__file__).resolve().parent.parent+sys.path.insert(0, str(ROOT))+import desktop_runtime as desktop+PROBES = ROOT / "probes"DEFAULT_BUILD = ROOT.parent.parent / "build"@@ -67,11 +70,11 @@ def setup(task: str, seed: int, d: Path, build: Path = DEFAULT_BUILD) -> dict[sthome.mkdir(exist_ok=True)meta = task_meta(task)(d / "brief.txt").write_text(render(task, seed), "utf-8")- exe = next((build / "BenchLab.app" / "Contents" / "MacOS").iterdir())- env = {"HOME": str(home), "PATH": "/usr/bin:/bin", "TMPDIR": str(d)}+ argv = desktop.bench_app_argv("BenchLab", build)+ env = desktop.desktop_env(home)proc = subprocess.Popen([- str(exe),+ *argv,*meta["app_args"],"--seed",str(seed),@@ -130,7 +133,10 @@ def reset(task: str, d: Path) -> dict[str, Any]:os.kill(int(p["pid"].read_text()), signal.SIGKILL)except (OSError, ValueError):pass- sweep("BenchLab.app/Contents/MacOS/BenchLab")+ if desktop.detect_platform() == "linux":+ subprocess.run(["pkill", "-x", "BenchLab"], capture_output=True)+ else:+ sweep("BenchLab.app/Contents/MacOS/BenchLab")meta = task_meta(task)for app in meta.get("needs_apps", []):sweep(f"/{app}.app/Contents/MacOS/{app}")diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tools/validate.py b/libs/cua-driver-bench/automated-eval/macos_bench/tools/validate.pyindex 7443835623..ccdd0a10da 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/tools/validate.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tools/validate.py@@ -393,7 +393,9 @@ def run_task(task: str, rnd: int, out: Path) -> dict[str, Any]:clean.append(s2 == initial and len(ev) == 1)F.reset(task, d)left = subprocess.run(- ["pgrep", "-f", "BenchLab.app/Contents/MacOS/BenchLab"], capture_output=True, text=True+ ["pgrep", "-f", F.desktop.bench_app_argv("BenchLab", F.DEFAULT_BUILD)[-1]],+ capture_output=True,+ text=True,).stdout.strip()clean.append(left == "" and not F.paths(d)["state"].exists() and not F.paths(d)["events"].exists()