Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
Validate per-arm captures and classify tool setup failures as infrastructure (#5767)
injaneityb63d82020017
diff --git a/libs/cua-driver-bench/README.md b/libs/cua-driver-bench/README.mdindex 2e67ffdde9..324b24f3d0 100644--- a/libs/cua-driver-bench/README.md+++ b/libs/cua-driver-bench/README.md@@ -179,7 +179,11 @@ donesha256 and the helper's sha256 and signature against `pins.json`(`claude_cu_helper`), unpacks it without moving it to `/Applications` orlaunching it, and builds `winlist` and `cu-disclaim` from source. Then give- `app-cu-helper` Accessibility in System Settings. No Claude Desktop binary,+ `app-cu-helper` Accessibility in System Settings. Separately grant Apple's+ `/usr/sbin/screencapture` Screen Recording: System Settings → Privacy & Security+ → Screen Recording → **+**, then Go to `/usr/sbin/screencapture`.+ The launcher wraps both helper and capture; Terminal, the adapter and launcher+ receive no new grants. No Claude Desktop binary,app code or asset is included in this repository; only Cua's adapter is.* **Codex CLI setups** (arms `cx-codex-cu`, `cx-cua-driver`, `cx-claude-cu-helper`). Nothing extra to install beyond the matching Claude Code setup: they use the Codex CLI bundled in the same ChatGPT app (`Contents/Resources/codex-cli/CodexCLI.app`) and the MCP config of the matching Claude Code setup (`cc-codex-cu`, `cc-cua-driver-script` or `cc-claude-cu-helper`). Point `CDB_CODEX_GATEWAY` at an OpenAI-compatible Responses endpoint serving `gpt-6.1-sol` (default `http://127.0.0.1:4000/v1`). The pinned CLI build is in `pins.json` (`codex_native`).diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/README.md b/libs/cua-driver-bench/automated-eval/macos_bench/README.mdindex e83637221c..dd86d19417 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/README.md+++ b/libs/cua-driver-bench/automated-eval/macos_bench/README.md@@ -23,4 +23,15 @@ how to reproduce it, and [`../../METHOD.md`](../../METHOD.md) for the method.| `pins.json` | Versions and digests the preflight enforces || `run_pilot.py`, `arms.py`, `codex_events.py`, `mcp_preflight.py` | The earlier Codex CLI pilot runner, which the main runner reuses for process and sentinel handling |+For the helper arm, grant `app-cu-helper` Accessibility separately from Apple's+Screen Recording grant: System Settings → Privacy & Security → Screen Recording+→ **+** → Go to `/usr/sbin/screencapture`. Terminal, adapter and launcher stay ungranted.++Preflight, including `--offline`, captures through every selected arm's own MCP+and runs the recorder's three-second test unless recording is disabled. It fails+on errors, missing images or uniform RGB frames; details contain size and path,+not image content. Offline skips model calls, not capture checks.+Tool-reported setup errors become infrastructure failures with `infra_pattern`;+model prose alone never triggers this classification.+Keep results, streams and recordings out of git.diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/RUNNER.md b/libs/cua-driver-bench/automated-eval/macos_bench/RUNNER.mdindex 8804173ecc..844761a6c0 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/RUNNER.md+++ b/libs/cua-driver-bench/automated-eval/macos_bench/RUNNER.md@@ -171,11 +171,29 @@ arm, task, category, condition, run, seed, versions and hashes, status, pass andchecks, turns, tool calls by class, tokens, cost, the background measures, theGUI-only evidence and the interruption diagnostic.+## Capture setup and preflight++Amendment 11: grant `app-cu-helper` Accessibility. Separately grant Apple's+`/usr/sbin/screencapture` Screen Recording in System Settings → Privacy & Security+→ Screen Recording → **+** → Go to `/usr/sbin/screencapture`. `cu-disclaim` wraps+both processes. Do not grant Terminal, the adapter or launcher new permissions.+Every selected arm gets a `real capture` row through its own MCP, even offline;+the recorder also decodes its three-second test unless disabled. ffmpeg must be+installed at `/opt/homebrew/bin/ffmpeg`. Empty, undecodable and uniform RGB+frames fail; this is a deterministic blank check, not semantic recognition.+Rows log dimensions, byte count and saved path, never image content.++Errored tool results reporting capture/TCC/Accessibility setup failures or a+stopped helper/daemon are infrastructure failures (`tool_setup`, `infra_pattern`).+Assistant prose and successful tool results do not trigger these patterns.+The existing policy retries once, then excludes; Cua daemon retries recheck and+restart a down daemon. Quota and authentication classification retains precedence.+## Commands```bashpython3 run_bench.py dry-run --bench-v2 --phase1-runs 3 --phase2-runs 2 # schedule and commands, no side effects-python3 run_bench.py preflight --offline --bench-v2 --build-dir WORK/build # static checks only+python3 run_bench.py preflight --offline --bench-v2 --build-dir WORK/build # captures and recording, no model callspython3 run_bench.py gate-check # HOLD file and quota, no model callpython3 run_bench.py preflight --bench-v2 --build-dir WORK/build # full: daemons, MCP, recording, 3 Haiku callspython3 run_bench.py run --bench-v2 --condition background --run-id RUN ... # see ../../README.mddiff --git a/libs/cua-driver-bench/automated-eval/macos_bench/claude_events.py b/libs/cua-driver-bench/automated-eval/macos_bench/claude_events.pyindex fb95239a23..7973fcc4ee 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/claude_events.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/claude_events.py@@ -183,6 +183,20 @@ def _text_of(content: Any) -> str:return "\n".join(parts)+TOOL_SETUP_PATTERNS = [+ (name, re.compile(r"\b" + words.replace(" ", "[ _-]") + r"\b", re.I))+ for name, words in [+ ("screencapture", "could not create image from window"),+ ("permissions_pending", "permissions pending"),+ ("screen_recording", "screen recording permission denied"),+ ("accessibility", "accessibility permission denied"),+ ("tcc", "tcc permission denied"),+ ("daemon_not_running", "Cua Driver daemon is not running"),+ ("helper_exited", "helper exited during"),+ ]+]++def summarize(events: Iterable[dict[str, Any]]) -> dict[str, Any]:"""Counts, usage, cost, errors and quota state from one trial's stream."""events = list(events)@@ -193,6 +207,7 @@ def summarize(events: Iterable[dict[str, Any]]) -> dict[str, Any]:per_message: dict[str, dict[str, int]] = {}message_order: list[str] = []quota_events: list[dict[str, Any]] = []+ tool_setup_failure = Nonecontrol_requests = 0api_retries: list[dict[str, Any]] = []stamps: list[float] = []@@ -236,6 +251,12 @@ def summarize(events: Iterable[dict[str, Any]]) -> dict[str, Any]:elif kind == "user":for block in (event.get("message") or {}).get("content") or []:if isinstance(block, dict) and block.get("type") == "tool_result":+ if block.get("is_error") and tool_setup_failure is None:+ text = _text_of(block.get("content"))+ for name, pattern in TOOL_SETUP_PATTERNS:+ if pattern.search(text):+ tool_setup_failure = {"pattern": name, "message": text[:300]}+ breakresults[block.get("tool_use_id", "")] = {"is_error": bool(block.get("is_error")),"ms": event.get("_ms"),@@ -321,6 +342,7 @@ def summarize(events: Iterable[dict[str, Any]]) -> dict[str, Any]:tools = list(init.get("tools") or [])mcp_tools = [t for t in tools if t.startswith("mcp__")]summary = {+ "tool_setup_failure": tool_setup_failure,"has_init": bool(init),"has_result": result is not None,"init": {@@ -414,6 +436,9 @@ def classify_failure(if status in (401, 403) or AUTH_TEXT.search(text):out.update(kind="auth", message=text.strip()[:300])return out+ if summary.get("tool_setup_failure"):+ out.update(kind="tool_setup", **summary["tool_setup_failure"])+ return outservers = (summary.get("init") or {}).get("mcp_servers") or []bad = [s for s in servers if isinstance(s, dict) and s.get("status") not in ("connected", "ready")diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/codex_native.py b/libs/cua-driver-bench/automated-eval/macos_bench/codex_native.pyindex c99b274e58..d56582a51b 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/codex_native.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/codex_native.py@@ -335,14 +335,23 @@ class Translator:self.started.add(item_id)out.append(self._use(item_id, name, tool_input))if method == "item/completed":- failed = item.get("status") != "completed" or item.get("error") is not None- failed = failed or bool(invocation and item.get("exitCode") != 0)+ from claude_events import _text_of++ result = item.get("result") or {}+ failed = (+ item.get("status") != "completed"+ or item.get("error") is not None+ or bool(result.get("isError"))+ or bool(invocation and item.get("exitCode") != 0)+ )text = (str(item.get("aggregatedOutput") or "")[:2000]if invocation- else json.dumps(item.get("error"))[:300]- if failed- else ""+ else (+ (_text_of(result.get("content")) or str(item.get("error") or ""))[:300]+ if failed+ else ""+ ))out.append({diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/pins.json b/libs/cua-driver-bench/automated-eval/macos_bench/pins.jsonindex 5bd1a6732b..5cd62cee40 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/pins.json+++ b/libs/cua-driver-bench/automated-eval/macos_bench/pins.json@@ -64,14 +64,14 @@"_signature": "Developer ID Application: OpenAI OpCo, LLC (2DC432GLL2), notarized"},"claude_cu_helper": {- "_note": "Amendment 11: arm cc-claude-cu-helper, \"Claude Desktop 2.31226.0 computer-use helper via a minimal adapter\". Claude Desktop 2.31226.0 downloaded and unpacked in a clone of the bench VM only (never launched, never moved to /Applications). The helper is used unmodified in place; tools/claude_cu_helper/install_cu_helper.sh checks the archive and helper hashes and the signature. The helper's Accessibility grant was given in System Settings (Privacy & Security > Accessibility > +), through the VM's UI. Public copy: adapter_sha256 is the published cu_helper_mcp.py with its licence header (the registered runs used the same adapter without it, d3f7797bee7b).",+ "_note": "Amendment 11: arm cc-claude-cu-helper, \"Claude Desktop 2.31226.0 computer-use helper via a minimal adapter\". Claude Desktop 2.31226.0 downloaded and unpacked in a clone of the bench VM only (never launched, never moved to /Applications). The helper is used unmodified in place; tools/claude_cu_helper/install_cu_helper.sh checks the archive and helper hashes and the signature. The helper's Accessibility grant was given in System Settings (Privacy & Security > Accessibility > +), through the VM's UI. Public copy: adapter_sha256 is the published cu_helper_mcp.py with its licence header (the registered runs used the same adapter without it, d3f7797bee7b). The previous published adapter_sha256 was ed0da1fc5d00da2d8a3f8a43e3b18df9a41150e97a41632107eb8f82bcd058eb; the new hash is for launcher-wrapped screencapture. Grant Apple's /usr/sbin/screencapture Screen Recording via System Settings > Privacy & Security > Screen Recording > + > Go to /usr/sbin/screencapture. The helper retains its separate Accessibility grant; Terminal, the adapter and launcher receive no new grants.","desktop_version": "2.31226.0","desktop_zip_url": "https://downloads.claude.ai/releases/darwin/universal/2.31226.0/Claude-eb794d1033f2a59c2cd89aa29ed1f8b5de12aa73.zip","desktop_zip_bytes": 383954641,"desktop_zip_sha256": "bd38f1051a14cabd893c7422e106ad6318e8785f19472c8064eafa64ce235820","helper_sha256": "7bd631e8b5ae36941ac551c474055e2eff32163db6684628eac5cfcabcb3dc14","helper_signature": "Developer ID Application: Anthropic PBC (Q6L2SF6YDW), identifier app-cu-helper",- "adapter_sha256": "ed0da1fc5d00da2d8a3f8a43e3b18df9a41150e97a41632107eb8f82bcd058eb",+ "adapter_sha256": "1a75682aea7d524455e3f1a159149090184cc536b21d277e8d0df916b3716b3f","winlist_sha256": "d150737bd4a3f7995a2abe7a615b951da8189236da7e3cb969e0de670d9614d4","launcher_sha256": "c9dd836899692149e0a7d2bf144aa5f1180b359661016fddd267140c77e0057f","actions": [diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/recorder.py b/libs/cua-driver-bench/automated-eval/macos_bench/recorder.pyindex 990bfdaacf..54e37d0a93 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/recorder.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/recorder.py@@ -21,13 +21,19 @@ import subprocessimport threadingimport timefrom pathlib import Path-from typing import Any+from typing import Any, Callableclass McpStdioClient:"""Minimal MCP client over a child process' stdio (newline-delimited JSON-RPC)."""- def __init__(self, argv: list[str], env: dict[str, str]) -> None:+ def __init__(+ self,+ argv: list[str],+ env: dict[str, str],+ elicitation: Callable[[str], dict[str, Any]] | None = None,+ ) -> None:+ self.elicitation = elicitationself.proc = subprocess.Popen(argv,env=env,@@ -43,16 +49,22 @@ class McpStdioClient:self._cond = threading.Condition()self._reader = threading.Thread(target=self._pump, daemon=True)self._reader.start()- self.request(- "initialize",- {- "protocolVersion": "2025-06-18",- "capabilities": {},- "clientInfo": {"name": "cdb-bench-recorder", "version": "1"},- },- timeout=30,- )- self._send({"jsonrpc": "2.0", "method": "notifications/initialized"})+ try:+ reply = self.request(+ "initialize",+ {+ "protocolVersion": "2025-06-18",+ "capabilities": {"elicitation": {}} if elicitation else {},+ "clientInfo": {"name": "cdb-bench-recorder", "version": "1"},+ },+ timeout=30,+ )+ if reply.get("error"):+ raise RuntimeError(str(reply["error"])[:300])+ self._send({"jsonrpc": "2.0", "method": "notifications/initialized"})+ except Exception:+ self.close()+ raisedef _pump(self) -> None:assert self.proc.stdout is not None@@ -61,7 +73,10 @@ class McpStdioClient:obj = json.loads(line)except json.JSONDecodeError:continue- if "id" in obj and ("result" in obj or "error" in obj):+ if "id" in obj and obj.get("method") == "elicitation/create" and self.elicitation:+ answer = self.elicitation((obj.get("params") or {}).get("message", ""))+ self._send({"jsonrpc": "2.0", "id": obj["id"], "result": answer})+ elif "id" in obj and ("result" in obj or "error" in obj):with self._cond:self._responses[int(obj["id"])] = objself._cond.notify_all()@@ -181,6 +196,42 @@ class CuaRecorder:self.client = None+def capture_detail(path: Path, ffmpeg: str = "/opt/homebrew/bin/ffmpeg") -> str:+ done = subprocess.run(+ [+ ffmpeg,+ "-nostdin",+ "-v",+ "error",+ "-i",+ str(path),+ "-frames:v",+ "1",+ "-f",+ "image2pipe",+ "-vcodec",+ "ppm",+ "-pix_fmt",+ "rgb24",+ "pipe:1",+ ],+ capture_output=True,+ timeout=30,+ )+ if done.returncode or not done.stdout:+ raise RuntimeError("capture decode failed or empty image")+ try:+ magic, dimensions, maximum, pixels = done.stdout.split(b"\n", 3)+ w, h = map(int, dimensions.split())+ if magic != b"P6" or maximum != b"255" or len(pixels) != w * h * 3 or not pixels:+ raise ValueError("invalid RGB frame")+ except ValueError as error:+ raise RuntimeError("capture decode failed") from error+ if pixels == pixels[:3] * (w * h):+ raise RuntimeError("blank image (uniform RGB frame)")+ return f"{w}x{h} bytes={path.stat().st_size} path={path}"++def ffprobe_duration(path: Path) -> float | None:try:done = subprocess.run(diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.py b/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.pyindex 14fa901808..855d1f050c 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.py@@ -25,6 +25,7 @@ clients whose responsible app cannot show the macOS Automation prompt.from __future__ import annotationsimport argparse+import base64import hashlibimport jsonimport os@@ -538,6 +539,93 @@ def mcp_list_tools(command: str, args: list[str], env: dict[str, str]) -> list[sclient.close()+def real_capture_check(ctx: "Ctx", arm: str) -> tuple[str, str, str]:+ client = None+ try:+ source = cxn.SOURCE_ARM.get(arm, arm)+ path, _ = ctx.mcp[arm]+ if source in ca.CUA_ARMS:+ ensure_agent_daemon(ctx, source)+ if arm in cxn.ARMS:+ entry = cxn.server_entry(path, arm)+ command, args, env = entry["command"], entry["args"], entry["env"]+ else:+ command, args, env = mcp_entry(path)+ client = rec.McpStdioClient(+ [command, *args],+ env,+ elicitation=lambda text: claude_driver.answer_elicitation(text, {"calculator"}),+ )++ def call(name: str, arguments: dict[str, Any]) -> dict[str, Any]:+ reply = client.request("tools/call", {"name": name, "arguments": arguments}, timeout=60)+ result = reply.get("result") or {}+ if reply.get("error") or result.get("isError"):+ raise RuntimeError(+ claude_events._text_of(result.get("content"))[:300]+ or str(reply.get("error"))[:300]+ )+ return result++ listed = client.request("tools/list", {}, timeout=60)+ if listed.get("error"):+ raise RuntimeError(str(listed["error"])[:300])+ tools = {t["name"]: t for t in (listed.get("result") or {}).get("tools", [])}+ dest = ctx.run_dir / "preflight-capture" / arm / "capture.png"+ dest.parent.mkdir(parents=True, exist_ok=True)+ dest.unlink(missing_ok=True)+ if source in ca.CUA_ARMS:+ if (+ "list_apps" not in tools+ or len(tools) < 50+ or ("run_script" in tools) != (ca.SCRIPT_FLAG in ca.CUA_BUILDS[source].daemon_env)+ ):+ raise RuntimeError("unexpected Cua Driver tools/list")+ call("check_permissions", {})+ tool = next((t for t in ("get_desktop_state", "screenshot") if t in tools), None)+ if tool is None:+ raise RuntimeError("no desktop capture tool")+ arguments = (+ {"screenshot_out_file": str(dest)}+ if "screenshot_out_file" in tools[tool].get("inputSchema", {}).get("properties", {})+ else {}+ )+ result = call(tool, arguments)+ elif source in ca.CU_HELPER_ARMS:+ if set(tools) != set(ca.load_pins()["claude_cu_helper"]["tools"]):+ raise RuntimeError("unexpected helper tools/list")+ windows = json.loads(+ claude_events._text_of(+ call("app_list_windows", {"app": "Calculator"}).get("content")+ )+ )+ window = next((w for w in windows if w.get("app") == "Calculator"), None)+ if window is None:+ raise RuntimeError("no layer-0 Calculator window")+ result = call("app_screenshot", {"app": "Calculator", "window_id": window["window_id"]})+ else:+ if not {"js", "js_reset"} <= tools.keys():+ raise RuntimeError("missing Codex tools")+ result = call(+ "js",+ {"code": 'const app = await cua.getApp("Calculator"); await app.getScreenshot();'},+ )+ image = next(+ (b for b in result.get("content") or [] if b.get("type") == "image" and b.get("data")),+ None,+ )+ if image is not None:+ dest.write_bytes(base64.b64decode(image["data"], validate=True))+ elif not dest.is_file():+ raise RuntimeError("no image block")+ return f"{arm} real capture", "pass", rec.capture_detail(dest)+ except Exception as error:+ return f"{arm} real capture", "fail", f"{type(error).__name__}: {error}"[:400]+ finally:+ if client is not None:+ client.close()++def cu_helper_probe(helper: Path, launcher: Path | None = None, timeout: float = 20.0) -> dict[str, Any]:@@ -581,9 +669,14 @@ def cu_helper_live_checks(ctx: "Ctx", arm: str) -> list[tuple[str, str, str]]:json.dumps(probe)[:400] + (f" missing {missing}" if missing else ""),))- path, _server = ctx.mcp[arm]- command, cargs, env = mcp_entry(path)try:+ selected = next(a for a in ctx.arm_names if cxn.SOURCE_ARM.get(a, a) == arm)+ path, _server = ctx.mcp[selected]+ if selected in cxn.ARMS:+ entry = cxn.server_entry(path, selected)+ command, cargs, env = entry["command"], entry["args"], entry["env"]+ else:+ command, cargs, env = mcp_entry(path)tools = sorted(mcp_list_tools(command, cargs, env))want = sorted(pins.get("tools") or [])out.append(@@ -686,6 +779,7 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]idle_start = pilot.hid_idle_seconds()row: dict[str, Any] = {"schema": SCHEMA,+ "infra_pattern": None,"trial_id": entry["trial_id"],"attempt": attempt,"final": False,@@ -1033,6 +1127,7 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]"status": status,"infra_failure": infra,"infra_message": failure.get("message", "")[:300] if infra else "",+ "infra_pattern": failure.get("pattern") if infra else None,"infra_reset_epoch": failure.get("reset_epoch") if infra else None,"excluded": bool(infra),"excluded_reason": f"infra_failure:{infra}" if infra else None,@@ -1647,9 +1742,12 @@ def execute_trial(ctx: Ctx, entry: dict[str, Any]) -> dict[str, Any]:trial_started_mono=None,)return row- if kind == "mcp_start" and entry["arm"] in ca.CUA_ARMS:+ if (+ kind in ("mcp_start", "tool_setup")+ and cxn.SOURCE_ARM.get(entry["arm"], entry["arm"]) in ca.CUA_ARMS+ ):try:- ensure_agent_daemon(ctx, entry["arm"])+ ensure_agent_daemon(ctx, cxn.SOURCE_ARM.get(entry["arm"], entry["arm"]))except Exception as error: # noqa: BLE001ctx.log(f"daemon restart failed: {error}")log_pause(ctx, f"retry_{kind}", wait_s, row.get("infra_message", ""), entry["trial_id"])@@ -2023,7 +2121,7 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:"pass" if got_skills and got_skills == main.get("skills_tree_sha256") else "fail",str(got_skills),)- for cua_arm in [a for a in ca.CUA_ARMS if a in ctx.arm_names and not offline]:+ for cua_arm in [a for a in ca.CUA_ARMS if a in ctx.arm_names]:spec = ca.CUA_BUILDS[cua_arm]try:ensure_agent_daemon(ctx, cua_arm)@@ -2035,63 +2133,12 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:)if cua_arm == "cc-cua-driver":ctx.versions["cua_daemon_version"] = health.get("build", {}).get("version")- tools = mcp_list_tools(- str(spec.bin), ["--socket", spec.socket, "mcp"], ca.build_env(cua_arm, spec.home)- )- add(- f"{cua_arm} MCP server starts and lists tools",- "pass" if "list_apps" in tools and len(tools) >= 50 else "fail",- f"{len(tools)} tools",- )- # Amendment 6: run_script is listed exactly in the arms whose daemon has the script flag- want_script = ca.SCRIPT_FLAG in spec.daemon_env- add(- f"{cua_arm} run_script {'listed' if want_script else 'not listed'}",- "pass" if ("run_script" in tools) == want_script else "fail",- f"run_script in tools/list: {'run_script' in tools}",- )except Exception as error: # noqa: BLE001add(f"{cua_arm} daemon/MCP", "fail", f"{type(error).__name__}: {error}")- for codex_arm in [a for a in ctx.arm_names if a in ca.CODEX_ARMS and not offline]:- try:- path, server = ctx.mcp[codex_arm]- command, cargs, env = mcp_entry(path)- tools = mcp_list_tools(command, cargs, env)- add(- "arm B MCP server starts and lists tools",- "pass" if {"js", "js_reset"} <= set(tools) else "fail",- f"server {server}: {tools}",- )- except Exception as error: # noqa: BLE001- add("arm B MCP server", "fail", f"{type(error).__name__}: {error}")for native_arm in [a for a in cxn.MCP_ARMS if a in ctx.arm_names]:if not cxn.CODEX_CLI.is_file():add("Codex CLI in the ChatGPT app", "fail", str(cxn.CODEX_CLI))continue- if not offline:- try:- path, server = ctx.mcp[native_arm]- entry = cxn.server_entry(path, native_arm)- source = cxn.SOURCE_ARM[native_arm]- if source in ca.CUA_ARMS: # A23: Cua Driver under Codex uses that build's daemon- ensure_agent_daemon(ctx, source)- tools = mcp_list_tools(entry["command"], entry["args"], entry["env"])- want = (- {"js", "js_reset"}- if server == cxn.SERVER- else set(- ca.load_pins().get("claude_cu_helper", {}).get("tools") or ["app_click"]- )- if server == ca.CU_HELPER_SERVER- else {"list_apps", "run_script"}- )- add(- f"{native_arm} {server} MCP server lists {sorted(want)}",- "pass" if want <= set(tools) else "fail",- f"{len(tools)} tools",- )- except Exception as error: # noqa: BLE001- add(f"{native_arm} MCP server", "fail", f"{type(error).__name__}: {error}")for name, status, detail in native_codex_checks(ctx, native_arm, with_models and not offline):@@ -2104,8 +2151,6 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:ctx.versions.setdefault("claude_cu_helper", {}).update(observed)for name, status, detail in ca.check_cu_helper_pins(pins, observed):add(name, status, detail)- if offline:- continuefor name, status, detail in cu_helper_live_checks(ctx, helper_arm):add(name, status, detail)cdb_ids = [t for t in ctx.task_ids if task_spec(ctx.tasks[t]).get("kind") == "cdb"]@@ -2158,6 +2203,15 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:"pass" if ok_copies else "fail",json.dumps(per_name) if not ok_copies else f"{[v[0] for v in per_name.values()]}",)+ try:+ subprocess.run(["/usr/bin/open", "-a", "Calculator"], check=True, timeout=30)+ for arm in ctx.arm_names:+ add(*real_capture_check(ctx, arm))+ except (OSError, subprocess.SubprocessError) as error:+ for arm in ctx.arm_names:+ add(f"{arm} real capture", "fail", str(error)[:300])+ finally:+ reset_needs_apps(["Calculator"])ps = subprocess.run(["ps", "-axo", "pid=,command="], capture_output=True, text=True).stdoutleftovers = [ln.split(None, 1)[1][:80]@@ -2176,20 +2230,21 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:"pass" if not leftovers else "warn",f"{leftovers}",)- if offline:- add("daemons, MCP servers, recording, model calls", "warn", "skipped (--offline)")- elif ctx.recorder is not None and ctx.recorder.name != "none":+ if ctx.recorder is not None and ctx.recorder.name != "none":test = ctx.run_dir / "preflight-rec"shutil.rmtree(test, ignore_errors=True)started = ctx.recorder.start(test)time.sleep(3.0)raw = ctx.recorder.stop() if started else Nonedur = rec.ffprobe_duration(raw) if raw else None- add(- "screen recording (3 s test)",- "pass" if raw and dur and dur > 0.5 else "fail",- f"{raw} duration={dur} err={getattr(ctx.recorder, 'last_error', None)}",- )+ try:+ if not raw or not dur or dur <= 0.5:+ raise RuntimeError(+ f"{raw} duration={dur} err={getattr(ctx.recorder, 'last_error', None)}"+ )+ add("recorder real capture (3 s test)", "pass", rec.capture_detail(raw))+ except (OSError, RuntimeError, subprocess.SubprocessError) as error:+ add("recorder real capture (3 s test)", "fail", str(error)[:300])else:add("screen recording", "warn", "disabled (--no-recorder)")gate = core.gate_decision(@@ -2339,7 +2394,7 @@ def build_parser() -> argparse.ArgumentParser:p.add_argument("--offline",action="store_true",- help="preflight only: static checks, no daemon, no recording, no GUI, no model call",+ help="preflight only: skip model calls; still start daemons, GUI captures and recording",)p.add_argument("--allow-unknown-quota",@@ -2794,8 +2849,7 @@ def cmd_preflight(args: argparse.Namespace) -> int:ctx = make_ctx(args)setup_arms(ctx)try:- if not args.offline:- start_recorder(ctx)+ start_recorder(ctx)checks = preflight(ctx, with_models=not args.no_model_preflight)(ctx.run_dir / "preflight.json").write_text(json.dumps(checks, indent=2) + "\n", "utf-8")write_manifest(diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_bench.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_bench.pyindex 4ff6ccac1c..753e28b192 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_bench.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_bench.py@@ -30,6 +30,64 @@ TASKS = [f"MB-{i:02d}" for i in range(1, 13)]ARMS = ["cc-cua-driver", "cc-codex-cu"]+class ToolSetupTest(unittest.TestCase):+ def test_only_errored_tools_report_setup_failures(self) -> None:+ errors = {+ "screencapture": "could not create image from window",+ "permissions_pending": "permissions_pending: macOS Accessibility or Screen Recording permission is still pending; no action started, retry after the permission gate completes",+ "screen_recording": "screen_recording_permission_denied",+ "accessibility": "Accessibility permission denied",+ "tcc": "tcc_permission_denied",+ "daemon_not_running": "Cua Driver daemon is not running",+ "helper_exited": "helper exited during probe",+ }+ for pattern, text in errors.items():+ for is_error in (True, False):+ with self.subTest(pattern=pattern, is_error=is_error):+ events = [+ {"type": "system", "subtype": "init"},+ {+ "type": "user",+ "message": {+ "content": [+ {+ "type": "tool_result",+ "is_error": is_error,+ "content": [+ {"type": "image", "data": "SECRET"},+ {"type": "text", "text": "x" * 310 + " " + text},+ ],+ }+ ]+ },+ },+ ]+ summary = ce.summarize(events)+ failure = ce.classify_failure(summary)+ self.assertEqual(failure["kind"], "tool_setup" if is_error else None)+ if is_error:+ self.assertEqual(failure["pattern"], pattern)+ self.assertLessEqual(len(failure["message"]), 300)+ self.assertNotIn("SECRET", json.dumps(summary))+ summary = ce.summarize(+ [{"type": "system", "subtype": "init"}, {"type": "result", "result": text}]+ )+ self.assertIsNone(ce.classify_failure(summary)["kind"])+ for text in ("background_unavailable", "permission_denied"):+ summary = ce.summarize(+ [+ {"type": "system", "subtype": "init"},+ {+ "type": "user",+ "message": {+ "content": [{"type": "tool_result", "is_error": True, "content": text}]+ },+ },+ ]+ )+ self.assertIsNone(ce.classify_failure(summary)["kind"])++class ScheduleTest(unittest.TestCase):def test_priority_order_mb_first_then_others(self) -> None:ids = ["CDB-S01", "MB-10", "MB-02", "MB-13", "MB-01"]diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_claude_cu_helper_arm.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_claude_cu_helper_arm.pyindex 882c7da00e..61d850178f 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_claude_cu_helper_arm.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_claude_cu_helper_arm.py@@ -163,6 +163,33 @@ class Fixture:return [r for r in rows if r["method"] == method]+class LauncherCaptureTest(unittest.TestCase):+ def test_capture_uses_launcher_without_wrapping_winlist(self) -> None:+ with tempfile.TemporaryDirectory() as tmp:+ fixture = Fixture(Path(tmp), shot=(800, 600))+ launcher = Path(tmp) / "launcher"+ log = Path(tmp) / "launcher.log"+ launcher.write_text('#!/bin/sh\nprintf "%s\\n" "$1" >> ' + str(log) + '\nexec "$@"\n')+ launcher.chmod(0o755)+ helper = ad.Helper([str(launcher), str(fixture.helper)])+ adapter = ad.Adapter(+ helper,+ str(fixture.winlist),+ list(ad.ALL_ACTIONS),+ screencapture=str(fixture.screencapture),+ sips=str(fixture.sips),+ launcher=str(launcher),+ )+ try:+ result = adapter.call("app_screenshot", {"app": "BenchLab", "window_id": 77})+ self.assertTrue(any(block.get("type") == "image" for block in result))+ self.assertEqual(+ log.read_text().splitlines(), [str(fixture.helper), str(fixture.screencapture)]+ )+ finally:+ helper.close()++class ArmRegistrationTest(unittest.TestCase):def test_arm_is_a_named_claude_arm_with_the_agreed_label(self) -> None:self.assertIn(ARM, arms.CLAUDE_ARMS)@@ -456,6 +483,59 @@ class AdapterTest(unittest.TestCase):class HelperClientTest(unittest.TestCase):+ def test_trusted_helper_and_tools_do_not_hide_capture_failure(self) -> None:+ from types import SimpleNamespace++ with tempfile.TemporaryDirectory() as tmp:+ fx = Fixture(Path(tmp))+ launcher = Path(tmp) / "launcher"+ launcher.write_text('#!/bin/sh\nexec "$@"\n')+ launcher.chmod(0o755)+ fx.winlist.write_text(+ "#!/bin/sh\nprintf '%s\\n' '"+ + json.dumps([{**WINDOWS[0], "app": "Calculator"}])+ + "'\n"+ )+ fx.screencapture.write_text(+ '#!/bin/sh\necho "could not create image from window" >&2\nexit 1\n'+ )+ config = Path(tmp) / "mcp.json"+ config.write_text(+ json.dumps(+ {+ "mcpServers": {+ "helper": {+ "command": sys.executable,+ "args": [+ str(ADAPTER),+ "--helper",+ str(fx.helper),+ "--launcher",+ str(launcher),+ "--winlist",+ str(fx.winlist),+ "--screencapture",+ str(fx.screencapture),+ "--sips",+ str(fx.sips),+ ],+ }+ }+ }+ )+ )+ ctx = SimpleNamespace(+ mcp={ARM: (config, "helper")}, arm_names=[ARM], versions={}, run_dir=Path(tmp)+ )+ with mock.patch.dict(+ ca.CU_HELPER_BUILDS, {ARM: SimpleNamespace(helper=fx.helper, launcher=launcher)}+ ):+ checks = rb.cu_helper_live_checks(ctx, ARM)+ self.assertTrue(all(status == "pass" for _, status, _ in checks), checks)+ name, status, detail = rb.real_capture_check(ctx, ARM)+ self.assertEqual((name, status), (f"{ARM} real capture", "fail"))+ self.assertIn("could not create image from window", detail)+def test_helper_errors_and_restart(self) -> None:with tempfile.TemporaryDirectory() as tmp:fx = Fixture(Path(tmp))diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_codex_native.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_codex_native.pyindex 9bfcac3605..1f57ad5f52 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_codex_native.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_codex_native.py@@ -16,6 +16,8 @@ import unittestfrom pathlib import Pathfrom unittest import mock+import pytest+HERE = Path(__file__).resolve().parent.parentsys.path.insert(0, str(HERE))@@ -25,7 +27,8 @@ import claude_events # noqa: E402import codex_native as cxn # noqa: E402-def test_browser_cli_translation_and_rejections():[email protected]("output", ["OUTPUT" * 1000, "short output"], ids=["truncated", "short"])+def test_browser_cli_translation_and_rejections(output):import browser_use_cli as buassert bu.ARM in arms.ALL_ARMS and bu.ARM in cxn.ARMS@@ -44,7 +47,7 @@ def test_browser_cli_translation_and_rejections():**item,"status": "completed","exitCode": 0,- "aggregatedOutput": "OUTPUT" * 1000,+ "aggregatedOutput": output,}},)@@ -57,7 +60,7 @@ def test_browser_cli_translation_and_rejections():]result = events[-1]["message"]["content"][0]assert result["tool_use_id"] == "cli1" and not result["is_error"]- assert result["content"].startswith("OUTPUT") and len(result["content"]) == 2000+ assert result["content"] == output[:2000]assert tr.non_gui == []for text in ("browser-use open x; curl x",@@ -545,6 +548,38 @@ for line in sys.stdin:class TranslatorTest(unittest.TestCase):+ def test_completed_mcp_is_error_becomes_setup_failure(self) -> None:+ tr = cxn.Translator()+ events = tr.feed("thread/started", {"thread": {"id": "t"}})+ events += tr.feed(+ "item/completed",+ {+ "item": {+ "id": "tool",+ "type": "mcpToolCall",+ "server": "cua",+ "tool": "screenshot",+ "status": "completed",+ "result": {+ "isError": True,+ "content": [+ {"type": "image", "data": "SECRET"},+ {+ "type": "text",+ "text": "screen_recording_permission_denied" + "!" * 500,+ },+ ],+ },+ }+ },+ )+ summary = claude_events.summarize(events)+ self.assertEqual(claude_events.classify_failure(summary)["kind"], "tool_setup")+ block = events[-1]["message"]["content"][0]+ self.assertTrue(block["is_error"])+ self.assertLessEqual(len(block["content"]), 300)+ self.assertNotIn("SECRET", json.dumps(events))+def run_events(self, events: list[tuple[str, dict]]) -> tuple[list[dict], cxn.Translator]:tr = cxn.Translator()out: list[dict] = []diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_runner_flow.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_runner_flow.pyindex 7df72314a7..94b0e72f69 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_runner_flow.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_runner_flow.py@@ -8,6 +8,8 @@ finalize. No GUI, no model."""from __future__ import annotationsimport argparse+import base64+import osimport contextlibimport jsonimport sys@@ -195,7 +197,7 @@ class CodexAuthPreflightTest(unittest.TestCase):self.assertNotIn(secret, json.dumps(checks))def test_native_trial_removes_auth_link_and_flags_replacement(self) -> None:- for replaced in (False, True):+ for replaced in (False, True, "setup"):with tempfile.TemporaryDirectory() as tmp, contextlib.ExitStack() as stack:ctx = make_ctx(tmp,@@ -257,15 +259,41 @@ class CodexAuthPreflightTest(unittest.TestCase):def run_codex(cmd, env, *args, **kwargs):link = Path(env["CODEX_HOME"]) / "auth.json"self.assertTrue(link.is_symlink())- if replaced:+ if replaced is True:link.unlink()link.write_text("FAKE_TOKEN")+ if replaced == "setup":+ events = [+ {"type": "system", "subtype": "init"},+ {+ "type": "user",+ "message": {+ "content": [+ {+ "type": "tool_result",+ "is_error": True,+ "content": "screen_recording_permission_denied",+ }+ ]+ },+ },+ {"type": "result", "result": "DONE"},+ ]+ args[5].write_text("\n".join(json.dumps(e) for e in events))return {"returncode": 0, "result_seen": True, "wall_s": 0}stack.enter_context(mock.patch.object(rb.cxn, "run_codex", side_effect=run_codex))row = rb.run_attempt(ctx, entry, 1)- self.assertEqual(row["infra_failure"] == "harness_exception", replaced)- if replaced:+ self.assertIn("infra_pattern", row)+ if replaced == "setup":+ self.assertEqual(row["infra_pattern"], "screen_recording")+ self.assertEqual(row["infra_failure"], "tool_setup")+ self.assertFalse(row["passed"])+ self.assertTrue(row["excluded"])+ else:+ self.assertIsNone(row["infra_pattern"])+ self.assertEqual(row["infra_failure"] == "harness_exception", replaced is True)+ if replaced is True:self.assertIn("symlink replaced", row["infra_message"])link = ctx.run_dir / "trials" / entry["trial_id"] / "a1/codex-home/.codex/auth.json"self.assertFalse(link.is_symlink())@@ -320,8 +348,198 @@ def attempt_row(return row+class RealCaptureTest(unittest.TestCase):+ def test_each_family_validates_capture_bytes_and_closes_client(self) -> None:+ with tempfile.TemporaryDirectory() as tmp:+ ctx = make_ctx(tmp)+ ffmpeg = Path(tmp) / "ffmpeg"+ ffmpeg.write_text(+ '#!/usr/bin/env python3\nimport sys\nfrom pathlib import Path\nsys.stdout.buffer.write(Path(sys.argv[sys.argv.index("-i")+1]).read_bytes())\n'+ )+ ffmpeg.chmod(0o755)+ image = b"P6\n2 1\n255\n" + b"\x00\x01\x02\xff\xfe\xfd"+ capture_detail = rb.rec.capture_detail+ families = [+ "cc-cua-driver",+ "cc-codex-cu",+ "cc-claude-cu-helper",+ "cx-cua-driver",+ "cx-codex-cu",+ "cx-claude-cu-helper",+ ]+ for arm in families:+ source = rb.cxn.SOURCE_ARM.get(arm, arm)+ tools = [+ {"name": n}+ for n in ["list_apps", "get_desktop_state", "js", "js_reset", "run_script"]+ + [f"tool{i}" for i in range(50)]+ ]+ if source in rb.ca.CU_HELPER_ARMS:+ tools = [{"name": n} for n in rb.ca.load_pins()["claude_cu_helper"]["tools"]]+ elif (+ source in rb.ca.CUA_ARMS+ and rb.ca.SCRIPT_FLAG not in rb.ca.CUA_BUILDS[source].daemon_env+ ):+ tools = [t for t in tools if t["name"] != "run_script"]+ ctx.mcp[arm] = (Path(tmp) / f"{arm}.json", "server")+ for payload, message in [+ (image, None),+ ("file", None),+ ("legacy", None),+ (b"", "no image block"),+ (b"garbage", "decode failed"),+ (b"P6\n2 1\n255\n" + b"\x00" * 6, "blank image"),+ (b"P6\n2 1\n255\n" + b"\xff" * 6, "blank image"),+ (None, "no image block"),+ ("error", "could not create image from window"),+ ("rpc", "permission denied"),+ ("windows", "no layer-0 Calculator window"),+ ]:+ if payload == "windows" and source not in rb.ca.CU_HELPER_ARMS:+ continue+ if payload in ("file", "legacy") and source not in rb.ca.CUA_ARMS:+ continue+ advertised = [+ {+ **t,+ "inputSchema": {+ "properties": {"screenshot_out_file": {"type": "string"}}+ },+ }+ if t["name"] == "get_desktop_state" and payload == "file"+ else {"name": "screenshot"}+ if t["name"] == "get_desktop_state" and payload == "legacy"+ else t+ for t in tools+ ]+ calls = []+ client = mock.Mock()++ def request(method, params, **kwargs):+ calls.append((method, params))+ if method == "tools/list":+ return {"result": {"tools": advertised}}+ if params["name"] == "check_permissions":+ return {"result": {"content": []}}+ if params["name"] == "app_list_windows":+ return {+ "result": {+ "content": [+ {+ "type": "text",+ "text": json.dumps(+ []+ if payload == "windows"+ else [{"app": "Calculator", "window_id": 7}]+ ),+ }+ ]+ }+ }+ if payload == "error":+ return {+ "result": {+ "isError": True,+ "content": [+ {+ "type": "text",+ "text": "could not create image from window",+ }+ ],+ }+ }+ if payload == "rpc":+ return {"error": {"message": "permission denied"}}+ if payload == "file":+ Path(params["arguments"]["screenshot_out_file"]).write_bytes(image)+ return {"result": {"content": None}}+ return {+ "result": {+ "content": None+ if payload is None+ else [+ {+ "type": "image",+ "data": base64.b64encode(+ image if payload == "legacy" else payload+ ).decode(),+ }+ ]+ }+ }++ client.request.side_effect = request+ with (+ self.subTest(arm=arm, message=message),+ mock.patch.object(rb.rec, "McpStdioClient", return_value=client),+ mock.patch.object(rb, "ensure_agent_daemon") as daemon,+ mock.patch.object(rb, "mcp_entry", return_value=("mcp", [], {})),+ mock.patch.object(+ rb.cxn,+ "server_entry",+ return_value={"command": "native", "args": [], "env": {}},+ ) as native,+ mock.patch.object(+ rb.rec,+ "capture_detail",+ side_effect=lambda path: capture_detail(path, str(ffmpeg)),+ ),+ ):+ name, status, detail = rb.real_capture_check(ctx, arm)+ self.assertEqual(name, f"{arm} real capture")+ self.assertEqual(status, "fail" if message else "pass")+ self.assertIn(message or "2x1 bytes=", detail)+ self.assertLess(len(detail), 500)+ client.close.assert_called_once()+ if arm in rb.cxn.ARMS:+ native.assert_called_once_with(ctx.mcp[arm][0], arm)+ if source in rb.ca.CUA_ARMS:+ daemon.assert_called_once_with(ctx, source)+ self.assertEqual(+ [p["name"] for m, p in calls if m == "tools/call"],+ [+ "check_permissions",+ "screenshot" if payload == "legacy" else "get_desktop_state",+ ],+ )+ if source == "cc-codex-cu":+ code = calls[-1][1]["arguments"]["code"]+ self.assertIn('cua.getApp("Calculator")', code)+ self.assertIn("await app.getScreenshot()", code)+ self.assertNotIn("emitImage", code)+ self.assertNotIn(".screenshot()", code)+ with self.assertRaises(FileNotFoundError):+ capture_detail(Path(tmp) / "missing.png", str(Path(tmp) / "missing-ffmpeg"))++ def test_stdio_client_answers_calculator_elicitation(self) -> None:+ with tempfile.TemporaryDirectory() as tmp:+ script = Path(tmp) / "server.py"+ script.write_text("""import json, sys+for line in sys.stdin:+ req = json.loads(line)+ if req.get("method") == "initialize":+ assert req["params"]["capabilities"] == {"elicitation": {}}+ print(json.dumps({"id": req["id"], "result": {}}), flush=True)+ elif req.get("method") == "tools/call":+ pending = req["id"]+ print(json.dumps({"id": "approval", "method": "elicitation/create", "params": {"message": 'Allow Computer Use to use "Calculator"?'}}), flush=True)+ elif req.get("id") == "approval":+ print(json.dumps({"id": pending, "result": req["result"]}), flush=True)+""")+ client = rb.rec.McpStdioClient(+ [sys.executable, str(script)],+ dict(os.environ),+ elicitation=lambda text: rb.claude_driver.answer_elicitation(text, {"calculator"}),+ )+ try:+ reply = client.request("tools/call", {"name": "js", "arguments": {}}, timeout=3)+ self.assertEqual(reply["result"]["action"], "accept")+ finally:+ client.close()++class PreflightTest(unittest.TestCase):- def test_native_mcp_is_live_only_online(self) -> None:+ def test_offline_captures_every_native_arm_without_models(self) -> None:with tempfile.TemporaryDirectory() as tmp:ctx = make_ctx(tmp, offline=True, allow_non_terminal=True)ctx.task_ids = []@@ -345,6 +563,16 @@ class PreflightTest(unittest.TestCase):mock.patch.object(rb.ca, "check_cu_helper_pins", return_value=[]),mock.patch.object(rb, "cu_helper_live_checks", return_value=[]),mock.patch.object(rb, "native_codex_checks", return_value=[]) as native,+ mock.patch.object(+ rb,+ "real_capture_check",+ side_effect=lambda ctx, arm: (+ f"{arm} real capture",+ "fail" if arm == rb.cxn.ARMS[0] else "pass",+ "capture",+ ),+ ) as capture,+ mock.patch.object(rb, "reset_needs_apps") as reset,mock.patch.object(rb, "mcp_list_tools") as probe,mock.patch.object(rb, "ensure_agent_daemon") as daemon,):@@ -355,20 +583,14 @@ class PreflightTest(unittest.TestCase):ctx.args.offline = Trueprobe.reset_mock()daemon.reset_mock()- probe.side_effect = AssertionError("offline MCP probe")- daemon.side_effect = AssertionError("offline daemon")+ capture.reset_mock()rows = rb.preflight(ctx)probe.assert_not_called()daemon.assert_not_called()native.assert_called_with(ctx, arm, False)- self.assertIn(- (- "daemons, MCP servers, recording, model calls",- "warn",- "skipped (--offline)",- ),- rows,- )+ capture.assert_called_once_with(ctx, arm)+ self.assertTrue(any(name == f"{arm} real capture" for name, _, _ in rows))+ reset.assert_called_with(["Calculator"])ctx.args.offline = Falseprobe.side_effect = Noneprobe.return_value = [@@ -379,23 +601,55 @@ class PreflightTest(unittest.TestCase):] + rb.ca.load_pins()["claude_cu_helper"]["tools"]daemon.side_effect = Nonerows = rb.preflight(ctx)- probe.assert_called_once_with("mcp", [], {})- source = rb.cxn.SOURCE_ARM[arm]- if source in rb.ca.CUA_ARMS:- daemon.assert_called_once_with(ctx, source)- else:- daemon.assert_not_called()- self.assertTrue(- any(- name.startswith(f"{arm} ")- and "MCP server lists" in name- and status == "pass"- for name, status, _ in rows- )- )+ self.assertEqual(capture.call_count, 2)+ ctx.arm_names = list(rb.cxn.MCP_ARMS)+ ctx.args.offline = True+ capture.reset_mock()+ rows = rb.preflight(ctx)+ self.assertEqual(capture.call_count, len(ctx.arm_names))+ self.assertEqual(+ len([r for r in rows if r[0].endswith(" real capture")]), len(ctx.arm_names)+ )+ ctx.recorder = mock.Mock(name="recording")+ raw = Path(tmp) / "video.mov"+ raw.write_bytes(b"video")+ ctx.recorder.stop.return_value = raw+ for blank in (True, False):+ with (+ mock.patch.object(rb.time, "sleep"),+ mock.patch.object(rb.rec, "ffprobe_duration", return_value=3),+ mock.patch.object(+ rb.rec,+ "capture_detail",+ side_effect=RuntimeError("blank image") if blank else None,+ return_value=f"2x1 bytes=5 path={raw}",+ ),+ ):+ rows = rb.preflight(ctx)+ recording = next(r for r in rows if r[0].startswith("recorder real capture"))+ self.assertEqual(recording[1], "fail" if blank else "pass")class ExecuteTrialTest(unittest.TestCase):+ def test_daemon_setup_failure_retries_with_restart_then_excludes(self) -> None:+ with tempfile.TemporaryDirectory() as tmp:+ ctx = make_ctx(tmp)+ rows = [+ attempt_row(i, "tool_setup", infra_pattern="daemon_not_running") for i in (1, 2)+ ]+ with (+ mock.patch.object(rb.rec, "free_gb", return_value=1000),+ mock.patch.object(rb, "ensure_agent_daemon") as restart,+ ):+ row, pauses, error = self.run_trial(ctx, rows)+ self.assertIsNone(error)+ self.assertEqual(row["infra_pattern"], "daemon_not_running")+ self.assertTrue(row["excluded"])+ self.assertFalse(row["passed"])+ self.assertEqual(row["attempt"], 2)+ self.assertEqual(len(pauses), 1)+ restart.assert_called_once_with(ctx, ENTRY["arm"])+def run_trial(self, ctx: rb.Ctx, rows: list[dict]) -> tuple[dict | None, list[tuple[str, float]], Exception | None]:diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tools/check_named_electron.py b/libs/cua-driver-bench/automated-eval/macos_bench/tools/check_named_electron.pyindex 4ef3d56ba6..2ae4100233 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/tools/check_named_electron.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tools/check_named_electron.py@@ -15,7 +15,6 @@ returns. App approvals asked by the Codex server are accepted only for the task'from __future__ import annotationsimport json-import selectimport subprocessimport sysimport tempfile@@ -28,6 +27,7 @@ sys.path.insert(0, str(HERE.parent))import cdb_adapter # noqa: E402import claude_arms as ca # noqa: E402import run_bench # noqa: E402+import recorder # noqa: E402def lsappinfo(pid: int) -> dict[str, str]:@@ -56,65 +56,15 @@ def codex_get_apps(names: list[str], allowed: set[str], timeout: float = 60.0) -if config is None:return {"error": "no Codex computer-use MCP config"}command, args, env = run_bench.mcp_entry(config)- proc = subprocess.Popen(- [command, *args],- env=env,- stdin=subprocess.PIPE,- stdout=subprocess.PIPE,- stderr=subprocess.DEVNULL,- text=True,- bufsize=1,- )approvals: list[dict] = []- def send(message: dict) -> None:- assert proc.stdin- proc.stdin.write(json.dumps(message) + "\n")- proc.stdin.flush()-- def wait_for(msg_id: int, limit: float) -> dict | None:- assert proc.stdout- deadline = time.monotonic() + limit- while time.monotonic() < deadline:- ready, _, _ = select.select([proc.stdout], [], [], deadline - time.monotonic())- if not ready:- return None- line = proc.stdout.readline()- if not line:- return None- msg = json.loads(line)- if msg.get("method") == "elicitation/create":- text = (msg.get("params") or {}).get("message", "")- answer = run_bench.claude_driver.answer_elicitation(text, allowed)- approvals.append({"message": text, "answer": answer["action"]})- send(- {- "jsonrpc": "2.0",- "id": msg["id"],- "result": {"action": answer["action"], "content": {}},- }- )- continue- if msg.get("id") == msg_id:- return msg- return None+ def approve(text: str) -> dict:+ answer = run_bench.claude_driver.answer_elicitation(text, allowed)+ approvals.append({"message": text, "answer": answer["action"]})+ return answer+ client = recorder.McpStdioClient([command, *args], env, elicitation=approve)try:- send(- {- "jsonrpc": "2.0",- "id": 1,- "method": "initialize",- "params": {- "protocolVersion": "2025-06-18",- "capabilities": {"elicitation": {}},- "clientInfo": {"name": "cdb-check-named-electron", "version": "1"},- },- }- )- if wait_for(1, 30) is None:- return {"error": "no initialize reply"}- send({"jsonrpc": "2.0", "method": "notifications/initialized"})code = (f"const names = {json.dumps(names)}; const out = {{}};""for (const n of names) { try { const a = await cua.getApp(n);"@@ -123,17 +73,9 @@ def codex_get_apps(names: list[str], allowed: set[str], timeout: float = 60.0) -"let listed = null; try { listed = await cua.listApps({emit:false}); } catch (e) { listed = String(e); }""nodeRepl.write(JSON.stringify({getApp: out, listApps: listed}));")- send(- {- "jsonrpc": "2.0",- "id": 2,- "method": "tools/call",- "params": {"name": "js", "arguments": {"code": code}},- }+ reply = client.request(+ "tools/call", {"name": "js", "arguments": {"code": code}}, timeout=timeout)- reply = wait_for(2, timeout)- if reply is None:- return {"error": "js call timed out", "approvals": approvals}texts = [c.get("text", "") for c in (reply.get("result") or {}).get("content", [])]data = {"raw": [t[:300] for t in texts]}for (@@ -149,7 +91,7 @@ def codex_get_apps(names: list[str], allowed: set[str], timeout: float = 60.0) -data["approvals"] = approvalsreturn datafinally:- proc.terminate()+ client.close()def main() -> int:diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tools/claude_cu_helper/cu_helper_mcp.py b/libs/cua-driver-bench/automated-eval/macos_bench/tools/claude_cu_helper/cu_helper_mcp.pyindex 6494c80753..86b835b9d6 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/tools/claude_cu_helper/cu_helper_mcp.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tools/claude_cu_helper/cu_helper_mcp.py@@ -13,8 +13,9 @@ does not provide:* **Which windows exist** (``app_list_windows``): ``winlist``, a small Swift tool built in the VM that prints``CGWindowListCopyWindowInfo`` (owner, pid, window id, bounds, title). Public macOS API only.-* **What a window shows** (``app_screenshot``): ``/usr/sbin/screencapture -l <window id>`` of that one window, the- stock macOS tool, downscaled with ``/usr/bin/sips`` to Claude Code's own screenshot budget. Desktop captures with+* **What a window shows** (``app_screenshot``): ``/usr/sbin/screencapture -l <window id>`` of that one window,+ wrapped by ``--launcher`` when supplied. Grant Apple's screencapture Screen Recording, never Terminal or our+ adapter/launcher. The stock macOS capture is downscaled with ``/usr/bin/sips`` to Claude Code's own screenshot budget. Desktop captures withits in-process native module (``@ant/claude-swift``, ``appScoped.captureWindow``), not with the helper, so that pathis not reachable here. Like Desktop, the adapter asks the helper to wake a Chromium window's compositor first.@@ -185,11 +186,11 @@ def target_size(width: int, height: int) -> tuple[int, int]:return max(1, int(width * scale)), max(1, int(height * scale))-def capture_window(window_id: int, screencapture: str = SCREENCAPTURE, sips: str = SIPS) -> tuple[bytes, int, int]:+def capture_window(window_id: int, screencapture: str = SCREENCAPTURE, sips: str = SIPS, launcher: str | None = None) -> tuple[bytes, int, int]:"""PNG of one window (no shadow), downscaled to the screenshot budget. Returns (png, width, height)."""with tempfile.TemporaryDirectory(prefix="cuhelper-") as tmp:path = os.path.join(tmp, "w.png")- done = subprocess.run([screencapture, "-x", "-o", "-t", "png", f"-l{int(window_id)}", path],+ done = subprocess.run(([launcher] if launcher else []) + [screencapture, "-x", "-o", "-t", "png", f"-l{int(window_id)}", path],capture_output=True, text=True, timeout=20)if done.returncode != 0 or not os.path.exists(path) or os.path.getsize(path) == 0:raise RuntimeError(f"screencapture failed (rc={done.returncode}): {done.stderr.strip()[:300]}")@@ -232,11 +233,12 @@ class Adapter:indexes, menus, AX text entry) is not part of the helper and is not offered."""def __init__(self, helper: Helper, winlist: str, actions: list[str], host_pid: int | None = None,- screencapture: str = SCREENCAPTURE, sips: str = SIPS, busy_wait_s: float = BUSY_WAIT_S) -> None:+ screencapture: str = SCREENCAPTURE, sips: str = SIPS, busy_wait_s: float = BUSY_WAIT_S, launcher: str | None = None) -> None:self.helper = helperself.winlist = winlistself.actions = [a for a in ALL_ACTIONS if a in actions]self.host_pid = host_pid if host_pid is not None else os.getpid()+ self.launcher = launcherself.screencapture = screencaptureself.sips = sipsself.busy_wait_s = busy_wait_s@@ -342,7 +344,7 @@ class Adapter:"hostPid": self.host_pid, "debug": True})except HelperError as error:log(f"wakeChromiumCompositor: {error}")- data, width, height = capture_window(wid, self.screencapture, self.sips)+ data, width, height = capture_window(wid, self.screencapture, self.sips, self.launcher)self.frames[wid] = (float(win["width"]), float(win["height"]), width, height)text = (f"{win.get('app')} window {wid} ({win.get('title') or 'untitled'}): screenshot {width}x{height} px; ""coordinates are pixels of this image.")@@ -493,14 +495,14 @@ def main(argv: list[str] | None = None) -> int:parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])parser.add_argument("--helper", required=True)parser.add_argument("--winlist", required=True)- parser.add_argument("--launcher", help="cu-disclaim: start the helper as its own responsible process (A11)")+ parser.add_argument("--launcher", help="cu-disclaim: start helper and screencapture as their own responsible processes (A11)")parser.add_argument("--actions", default=",".join(ALL_ACTIONS))parser.add_argument("--screencapture", default=SCREENCAPTURE)parser.add_argument("--sips", default=SIPS)args = parser.parse_args(argv)helper = Helper([args.launcher, args.helper] if args.launcher else [args.helper], env={"PATH": "/usr/bin:/bin", "HOME": os.environ.get("HOME", "/tmp")})adapter = Adapter(helper, args.winlist, [a for a in args.actions.split(",") if a],- screencapture=args.screencapture, sips=args.sips)+ screencapture=args.screencapture, sips=args.sips, launcher=args.launcher)log(f"adapter {ADAPTER_VERSION} up; helper {args.helper}; actions {adapter.actions}")serve(adapter)return 0