Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

Validate per-arm captures and classify tool setup failures as infrastructure (#5767)

injaneityb63d82020017
diff --git a/libs/cua-driver-bench/README.md b/libs/cua-driver-bench/README.md
index 2e67ffdde9..324b24f3d0 100644
--- a/libs/cua-driver-bench/README.md
+++ b/libs/cua-driver-bench/README.md
@@ -179,7 +179,11 @@ done
sha256 and the helper's sha256 and signature against `pins.json`
(`claude_cu_helper`), unpacks it without moving it to `/Applications` or
launching it, and builds `winlist` and `cu-disclaim` from source. Then give
- `app-cu-helper` Accessibility in System Settings. No Claude Desktop binary,
+ `app-cu-helper` Accessibility in System Settings. Separately grant Apple's
+ `/usr/sbin/screencapture` Screen Recording: System Settings → Privacy & Security
+ → Screen Recording → **+**, then Go to `/usr/sbin/screencapture`.
+ The launcher wraps both helper and capture; Terminal, the adapter and launcher
+ receive no new grants. No Claude Desktop binary,
app code or asset is included in this repository; only Cua's adapter is.
* **Codex CLI setups** (arms `cx-codex-cu`, `cx-cua-driver`, `cx-claude-cu-helper`). Nothing extra to install beyond the matching Claude Code setup: they use the Codex CLI bundled in the same ChatGPT app (`Contents/Resources/codex-cli/CodexCLI.app`) and the MCP config of the matching Claude Code setup (`cc-codex-cu`, `cc-cua-driver-script` or `cc-claude-cu-helper`). Point `CDB_CODEX_GATEWAY` at an OpenAI-compatible Responses endpoint serving `gpt-6.1-sol` (default `http://127.0.0.1:4000/v1`). The pinned CLI build is in `pins.json` (`codex_native`).
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/README.md b/libs/cua-driver-bench/automated-eval/macos_bench/README.md
index e83637221c..dd86d19417 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/README.md
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/README.md
@@ -23,4 +23,15 @@ how to reproduce it, and [`../../METHOD.md`](../../METHOD.md) for the method.
| `pins.json` | Versions and digests the preflight enforces |
| `run_pilot.py`, `arms.py`, `codex_events.py`, `mcp_preflight.py` | The earlier Codex CLI pilot runner, which the main runner reuses for process and sentinel handling |
+For the helper arm, grant `app-cu-helper` Accessibility separately from Apple's
+Screen Recording grant: System Settings → Privacy & Security → Screen Recording
+→ **+** → Go to `/usr/sbin/screencapture`. Terminal, adapter and launcher stay ungranted.
+
+Preflight, including `--offline`, captures through every selected arm's own MCP
+and runs the recorder's three-second test unless recording is disabled. It fails
+on errors, missing images or uniform RGB frames; details contain size and path,
+not image content. Offline skips model calls, not capture checks.
+Tool-reported setup errors become infrastructure failures with `infra_pattern`;
+model prose alone never triggers this classification.
+
Keep results, streams and recordings out of git.
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/RUNNER.md b/libs/cua-driver-bench/automated-eval/macos_bench/RUNNER.md
index 8804173ecc..844761a6c0 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/RUNNER.md
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/RUNNER.md
@@ -171,11 +171,29 @@ arm, task, category, condition, run, seed, versions and hashes, status, pass and
checks, turns, tool calls by class, tokens, cost, the background measures, the
GUI-only evidence and the interruption diagnostic.
+## Capture setup and preflight
+
+Amendment 11: grant `app-cu-helper` Accessibility. Separately grant Apple's
+`/usr/sbin/screencapture` Screen Recording in System Settings → Privacy & Security
+→ Screen Recording → **+** → Go to `/usr/sbin/screencapture`. `cu-disclaim` wraps
+both processes. Do not grant Terminal, the adapter or launcher new permissions.
+Every selected arm gets a `real capture` row through its own MCP, even offline;
+the recorder also decodes its three-second test unless disabled. ffmpeg must be
+installed at `/opt/homebrew/bin/ffmpeg`. Empty, undecodable and uniform RGB
+frames fail; this is a deterministic blank check, not semantic recognition.
+Rows log dimensions, byte count and saved path, never image content.
+
+Errored tool results reporting capture/TCC/Accessibility setup failures or a
+stopped helper/daemon are infrastructure failures (`tool_setup`, `infra_pattern`).
+Assistant prose and successful tool results do not trigger these patterns.
+The existing policy retries once, then excludes; Cua daemon retries recheck and
+restart a down daemon. Quota and authentication classification retains precedence.
+
## Commands
```bash
python3 run_bench.py dry-run --bench-v2 --phase1-runs 3 --phase2-runs 2 # schedule and commands, no side effects
-python3 run_bench.py preflight --offline --bench-v2 --build-dir WORK/build # static checks only
+python3 run_bench.py preflight --offline --bench-v2 --build-dir WORK/build # captures and recording, no model calls
python3 run_bench.py gate-check # HOLD file and quota, no model call
python3 run_bench.py preflight --bench-v2 --build-dir WORK/build # full: daemons, MCP, recording, 3 Haiku calls
python3 run_bench.py run --bench-v2 --condition background --run-id RUN ... # see ../../README.md
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/claude_events.py b/libs/cua-driver-bench/automated-eval/macos_bench/claude_events.py
index fb95239a23..7973fcc4ee 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/claude_events.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/claude_events.py
@@ -183,6 +183,20 @@ def _text_of(content: Any) -> str:
return "\n".join(parts)
+TOOL_SETUP_PATTERNS = [
+ (name, re.compile(r"\b" + words.replace(" ", "[ _-]") + r"\b", re.I))
+ for name, words in [
+ ("screencapture", "could not create image from window"),
+ ("permissions_pending", "permissions pending"),
+ ("screen_recording", "screen recording permission denied"),
+ ("accessibility", "accessibility permission denied"),
+ ("tcc", "tcc permission denied"),
+ ("daemon_not_running", "Cua Driver daemon is not running"),
+ ("helper_exited", "helper exited during"),
+ ]
+]
+
+
def summarize(events: Iterable[dict[str, Any]]) -> dict[str, Any]:
"""Counts, usage, cost, errors and quota state from one trial's stream."""
events = list(events)
@@ -193,6 +207,7 @@ def summarize(events: Iterable[dict[str, Any]]) -> dict[str, Any]:
per_message: dict[str, dict[str, int]] = {}
message_order: list[str] = []
quota_events: list[dict[str, Any]] = []
+ tool_setup_failure = None
control_requests = 0
api_retries: list[dict[str, Any]] = []
stamps: list[float] = []
@@ -236,6 +251,12 @@ def summarize(events: Iterable[dict[str, Any]]) -> dict[str, Any]:
elif kind == "user":
for block in (event.get("message") or {}).get("content") or []:
if isinstance(block, dict) and block.get("type") == "tool_result":
+ if block.get("is_error") and tool_setup_failure is None:
+ text = _text_of(block.get("content"))
+ for name, pattern in TOOL_SETUP_PATTERNS:
+ if pattern.search(text):
+ tool_setup_failure = {"pattern": name, "message": text[:300]}
+ break
results[block.get("tool_use_id", "")] = {
"is_error": bool(block.get("is_error")),
"ms": event.get("_ms"),
@@ -321,6 +342,7 @@ def summarize(events: Iterable[dict[str, Any]]) -> dict[str, Any]:
tools = list(init.get("tools") or [])
mcp_tools = [t for t in tools if t.startswith("mcp__")]
summary = {
+ "tool_setup_failure": tool_setup_failure,
"has_init": bool(init),
"has_result": result is not None,
"init": {
@@ -414,6 +436,9 @@ def classify_failure(
if status in (401, 403) or AUTH_TEXT.search(text):
out.update(kind="auth", message=text.strip()[:300])
return out
+ if summary.get("tool_setup_failure"):
+ out.update(kind="tool_setup", **summary["tool_setup_failure"])
+ return out
servers = (summary.get("init") or {}).get("mcp_servers") or []
bad = [
s for s in servers if isinstance(s, dict) and s.get("status") not in ("connected", "ready")
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/codex_native.py b/libs/cua-driver-bench/automated-eval/macos_bench/codex_native.py
index c99b274e58..d56582a51b 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/codex_native.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/codex_native.py
@@ -335,14 +335,23 @@ class Translator:
self.started.add(item_id)
out.append(self._use(item_id, name, tool_input))
if method == "item/completed":
- failed = item.get("status") != "completed" or item.get("error") is not None
- failed = failed or bool(invocation and item.get("exitCode") != 0)
+ from claude_events import _text_of
+
+ result = item.get("result") or {}
+ failed = (
+ item.get("status") != "completed"
+ or item.get("error") is not None
+ or bool(result.get("isError"))
+ or bool(invocation and item.get("exitCode") != 0)
+ )
text = (
str(item.get("aggregatedOutput") or "")[:2000]
if invocation
- else json.dumps(item.get("error"))[:300]
- if failed
- else ""
+ else (
+ (_text_of(result.get("content")) or str(item.get("error") or ""))[:300]
+ if failed
+ else ""
+ )
)
out.append(
{
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/pins.json b/libs/cua-driver-bench/automated-eval/macos_bench/pins.json
index 5bd1a6732b..5cd62cee40 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/pins.json
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/pins.json
@@ -64,14 +64,14 @@
"_signature": "Developer ID Application: OpenAI OpCo, LLC (2DC432GLL2), notarized"
},
"claude_cu_helper": {
- "_note": "Amendment 11: arm cc-claude-cu-helper, \"Claude Desktop 2.31226.0 computer-use helper via a minimal adapter\". Claude Desktop 2.31226.0 downloaded and unpacked in a clone of the bench VM only (never launched, never moved to /Applications). The helper is used unmodified in place; tools/claude_cu_helper/install_cu_helper.sh checks the archive and helper hashes and the signature. The helper's Accessibility grant was given in System Settings (Privacy & Security > Accessibility > +), through the VM's UI. Public copy: adapter_sha256 is the published cu_helper_mcp.py with its licence header (the registered runs used the same adapter without it, d3f7797bee7b).",
+ "_note": "Amendment 11: arm cc-claude-cu-helper, \"Claude Desktop 2.31226.0 computer-use helper via a minimal adapter\". Claude Desktop 2.31226.0 downloaded and unpacked in a clone of the bench VM only (never launched, never moved to /Applications). The helper is used unmodified in place; tools/claude_cu_helper/install_cu_helper.sh checks the archive and helper hashes and the signature. The helper's Accessibility grant was given in System Settings (Privacy & Security > Accessibility > +), through the VM's UI. Public copy: adapter_sha256 is the published cu_helper_mcp.py with its licence header (the registered runs used the same adapter without it, d3f7797bee7b). The previous published adapter_sha256 was ed0da1fc5d00da2d8a3f8a43e3b18df9a41150e97a41632107eb8f82bcd058eb; the new hash is for launcher-wrapped screencapture. Grant Apple's /usr/sbin/screencapture Screen Recording via System Settings > Privacy & Security > Screen Recording > + > Go to /usr/sbin/screencapture. The helper retains its separate Accessibility grant; Terminal, the adapter and launcher receive no new grants.",
"desktop_version": "2.31226.0",
"desktop_zip_url": "https://downloads.claude.ai/releases/darwin/universal/2.31226.0/Claude-eb794d1033f2a59c2cd89aa29ed1f8b5de12aa73.zip",
"desktop_zip_bytes": 383954641,
"desktop_zip_sha256": "bd38f1051a14cabd893c7422e106ad6318e8785f19472c8064eafa64ce235820",
"helper_sha256": "7bd631e8b5ae36941ac551c474055e2eff32163db6684628eac5cfcabcb3dc14",
"helper_signature": "Developer ID Application: Anthropic PBC (Q6L2SF6YDW), identifier app-cu-helper",
- "adapter_sha256": "ed0da1fc5d00da2d8a3f8a43e3b18df9a41150e97a41632107eb8f82bcd058eb",
+ "adapter_sha256": "1a75682aea7d524455e3f1a159149090184cc536b21d277e8d0df916b3716b3f",
"winlist_sha256": "d150737bd4a3f7995a2abe7a615b951da8189236da7e3cb969e0de670d9614d4",
"launcher_sha256": "c9dd836899692149e0a7d2bf144aa5f1180b359661016fddd267140c77e0057f",
"actions": [
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/recorder.py b/libs/cua-driver-bench/automated-eval/macos_bench/recorder.py
index 990bfdaacf..54e37d0a93 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/recorder.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/recorder.py
@@ -21,13 +21,19 @@ import subprocess
import threading
import time
from pathlib import Path
-from typing import Any
+from typing import Any, Callable
class McpStdioClient:
"""Minimal MCP client over a child process' stdio (newline-delimited JSON-RPC)."""
- def __init__(self, argv: list[str], env: dict[str, str]) -> None:
+ def __init__(
+ self,
+ argv: list[str],
+ env: dict[str, str],
+ elicitation: Callable[[str], dict[str, Any]] | None = None,
+ ) -> None:
+ self.elicitation = elicitation
self.proc = subprocess.Popen(
argv,
env=env,
@@ -43,16 +49,22 @@ class McpStdioClient:
self._cond = threading.Condition()
self._reader = threading.Thread(target=self._pump, daemon=True)
self._reader.start()
- self.request(
- "initialize",
- {
- "protocolVersion": "2025-06-18",
- "capabilities": {},
- "clientInfo": {"name": "cdb-bench-recorder", "version": "1"},
- },
- timeout=30,
- )
- self._send({"jsonrpc": "2.0", "method": "notifications/initialized"})
+ try:
+ reply = self.request(
+ "initialize",
+ {
+ "protocolVersion": "2025-06-18",
+ "capabilities": {"elicitation": {}} if elicitation else {},
+ "clientInfo": {"name": "cdb-bench-recorder", "version": "1"},
+ },
+ timeout=30,
+ )
+ if reply.get("error"):
+ raise RuntimeError(str(reply["error"])[:300])
+ self._send({"jsonrpc": "2.0", "method": "notifications/initialized"})
+ except Exception:
+ self.close()
+ raise
def _pump(self) -> None:
assert self.proc.stdout is not None
@@ -61,7 +73,10 @@ class McpStdioClient:
obj = json.loads(line)
except json.JSONDecodeError:
continue
- if "id" in obj and ("result" in obj or "error" in obj):
+ if "id" in obj and obj.get("method") == "elicitation/create" and self.elicitation:
+ answer = self.elicitation((obj.get("params") or {}).get("message", ""))
+ self._send({"jsonrpc": "2.0", "id": obj["id"], "result": answer})
+ elif "id" in obj and ("result" in obj or "error" in obj):
with self._cond:
self._responses[int(obj["id"])] = obj
self._cond.notify_all()
@@ -181,6 +196,42 @@ class CuaRecorder:
self.client = None
+def capture_detail(path: Path, ffmpeg: str = "/opt/homebrew/bin/ffmpeg") -> str:
+ done = subprocess.run(
+ [
+ ffmpeg,
+ "-nostdin",
+ "-v",
+ "error",
+ "-i",
+ str(path),
+ "-frames:v",
+ "1",
+ "-f",
+ "image2pipe",
+ "-vcodec",
+ "ppm",
+ "-pix_fmt",
+ "rgb24",
+ "pipe:1",
+ ],
+ capture_output=True,
+ timeout=30,
+ )
+ if done.returncode or not done.stdout:
+ raise RuntimeError("capture decode failed or empty image")
+ try:
+ magic, dimensions, maximum, pixels = done.stdout.split(b"\n", 3)
+ w, h = map(int, dimensions.split())
+ if magic != b"P6" or maximum != b"255" or len(pixels) != w * h * 3 or not pixels:
+ raise ValueError("invalid RGB frame")
+ except ValueError as error:
+ raise RuntimeError("capture decode failed") from error
+ if pixels == pixels[:3] * (w * h):
+ raise RuntimeError("blank image (uniform RGB frame)")
+ return f"{w}x{h} bytes={path.stat().st_size} path={path}"
+
+
def ffprobe_duration(path: Path) -> float | None:
try:
done = subprocess.run(
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.py b/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.py
index 14fa901808..855d1f050c 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.py
@@ -25,6 +25,7 @@ clients whose responsible app cannot show the macOS Automation prompt.
from __future__ import annotations
import argparse
+import base64
import hashlib
import json
import os
@@ -538,6 +539,93 @@ def mcp_list_tools(command: str, args: list[str], env: dict[str, str]) -> list[s
client.close()
+def real_capture_check(ctx: "Ctx", arm: str) -> tuple[str, str, str]:
+ client = None
+ try:
+ source = cxn.SOURCE_ARM.get(arm, arm)
+ path, _ = ctx.mcp[arm]
+ if source in ca.CUA_ARMS:
+ ensure_agent_daemon(ctx, source)
+ if arm in cxn.ARMS:
+ entry = cxn.server_entry(path, arm)
+ command, args, env = entry["command"], entry["args"], entry["env"]
+ else:
+ command, args, env = mcp_entry(path)
+ client = rec.McpStdioClient(
+ [command, *args],
+ env,
+ elicitation=lambda text: claude_driver.answer_elicitation(text, {"calculator"}),
+ )
+
+ def call(name: str, arguments: dict[str, Any]) -> dict[str, Any]:
+ reply = client.request("tools/call", {"name": name, "arguments": arguments}, timeout=60)
+ result = reply.get("result") or {}
+ if reply.get("error") or result.get("isError"):
+ raise RuntimeError(
+ claude_events._text_of(result.get("content"))[:300]
+ or str(reply.get("error"))[:300]
+ )
+ return result
+
+ listed = client.request("tools/list", {}, timeout=60)
+ if listed.get("error"):
+ raise RuntimeError(str(listed["error"])[:300])
+ tools = {t["name"]: t for t in (listed.get("result") or {}).get("tools", [])}
+ dest = ctx.run_dir / "preflight-capture" / arm / "capture.png"
+ dest.parent.mkdir(parents=True, exist_ok=True)
+ dest.unlink(missing_ok=True)
+ if source in ca.CUA_ARMS:
+ if (
+ "list_apps" not in tools
+ or len(tools) < 50
+ or ("run_script" in tools) != (ca.SCRIPT_FLAG in ca.CUA_BUILDS[source].daemon_env)
+ ):
+ raise RuntimeError("unexpected Cua Driver tools/list")
+ call("check_permissions", {})
+ tool = next((t for t in ("get_desktop_state", "screenshot") if t in tools), None)
+ if tool is None:
+ raise RuntimeError("no desktop capture tool")
+ arguments = (
+ {"screenshot_out_file": str(dest)}
+ if "screenshot_out_file" in tools[tool].get("inputSchema", {}).get("properties", {})
+ else {}
+ )
+ result = call(tool, arguments)
+ elif source in ca.CU_HELPER_ARMS:
+ if set(tools) != set(ca.load_pins()["claude_cu_helper"]["tools"]):
+ raise RuntimeError("unexpected helper tools/list")
+ windows = json.loads(
+ claude_events._text_of(
+ call("app_list_windows", {"app": "Calculator"}).get("content")
+ )
+ )
+ window = next((w for w in windows if w.get("app") == "Calculator"), None)
+ if window is None:
+ raise RuntimeError("no layer-0 Calculator window")
+ result = call("app_screenshot", {"app": "Calculator", "window_id": window["window_id"]})
+ else:
+ if not {"js", "js_reset"} <= tools.keys():
+ raise RuntimeError("missing Codex tools")
+ result = call(
+ "js",
+ {"code": 'const app = await cua.getApp("Calculator"); await app.getScreenshot();'},
+ )
+ image = next(
+ (b for b in result.get("content") or [] if b.get("type") == "image" and b.get("data")),
+ None,
+ )
+ if image is not None:
+ dest.write_bytes(base64.b64decode(image["data"], validate=True))
+ elif not dest.is_file():
+ raise RuntimeError("no image block")
+ return f"{arm} real capture", "pass", rec.capture_detail(dest)
+ except Exception as error:
+ return f"{arm} real capture", "fail", f"{type(error).__name__}: {error}"[:400]
+ finally:
+ if client is not None:
+ client.close()
+
+
def cu_helper_probe(
helper: Path, launcher: Path | None = None, timeout: float = 20.0
) -> dict[str, Any]:
@@ -581,9 +669,14 @@ def cu_helper_live_checks(ctx: "Ctx", arm: str) -> list[tuple[str, str, str]]:
json.dumps(probe)[:400] + (f" missing {missing}" if missing else ""),
)
)
- path, _server = ctx.mcp[arm]
- command, cargs, env = mcp_entry(path)
try:
+ selected = next(a for a in ctx.arm_names if cxn.SOURCE_ARM.get(a, a) == arm)
+ path, _server = ctx.mcp[selected]
+ if selected in cxn.ARMS:
+ entry = cxn.server_entry(path, selected)
+ command, cargs, env = entry["command"], entry["args"], entry["env"]
+ else:
+ command, cargs, env = mcp_entry(path)
tools = sorted(mcp_list_tools(command, cargs, env))
want = sorted(pins.get("tools") or [])
out.append(
@@ -686,6 +779,7 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]
idle_start = pilot.hid_idle_seconds()
row: dict[str, Any] = {
"schema": SCHEMA,
+ "infra_pattern": None,
"trial_id": entry["trial_id"],
"attempt": attempt,
"final": False,
@@ -1033,6 +1127,7 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]
"status": status,
"infra_failure": infra,
"infra_message": failure.get("message", "")[:300] if infra else "",
+ "infra_pattern": failure.get("pattern") if infra else None,
"infra_reset_epoch": failure.get("reset_epoch") if infra else None,
"excluded": bool(infra),
"excluded_reason": f"infra_failure:{infra}" if infra else None,
@@ -1647,9 +1742,12 @@ def execute_trial(ctx: Ctx, entry: dict[str, Any]) -> dict[str, Any]:
trial_started_mono=None,
)
return row
- if kind == "mcp_start" and entry["arm"] in ca.CUA_ARMS:
+ if (
+ kind in ("mcp_start", "tool_setup")
+ and cxn.SOURCE_ARM.get(entry["arm"], entry["arm"]) in ca.CUA_ARMS
+ ):
try:
- ensure_agent_daemon(ctx, entry["arm"])
+ ensure_agent_daemon(ctx, cxn.SOURCE_ARM.get(entry["arm"], entry["arm"]))
except Exception as error: # noqa: BLE001
ctx.log(f"daemon restart failed: {error}")
log_pause(ctx, f"retry_{kind}", wait_s, row.get("infra_message", ""), entry["trial_id"])
@@ -2023,7 +2121,7 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:
"pass" if got_skills and got_skills == main.get("skills_tree_sha256") else "fail",
str(got_skills),
)
- for cua_arm in [a for a in ca.CUA_ARMS if a in ctx.arm_names and not offline]:
+ for cua_arm in [a for a in ca.CUA_ARMS if a in ctx.arm_names]:
spec = ca.CUA_BUILDS[cua_arm]
try:
ensure_agent_daemon(ctx, cua_arm)
@@ -2035,63 +2133,12 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:
)
if cua_arm == "cc-cua-driver":
ctx.versions["cua_daemon_version"] = health.get("build", {}).get("version")
- tools = mcp_list_tools(
- str(spec.bin), ["--socket", spec.socket, "mcp"], ca.build_env(cua_arm, spec.home)
- )
- add(
- f"{cua_arm} MCP server starts and lists tools",
- "pass" if "list_apps" in tools and len(tools) >= 50 else "fail",
- f"{len(tools)} tools",
- )
- # Amendment 6: run_script is listed exactly in the arms whose daemon has the script flag
- want_script = ca.SCRIPT_FLAG in spec.daemon_env
- add(
- f"{cua_arm} run_script {'listed' if want_script else 'not listed'}",
- "pass" if ("run_script" in tools) == want_script else "fail",
- f"run_script in tools/list: {'run_script' in tools}",
- )
except Exception as error: # noqa: BLE001
add(f"{cua_arm} daemon/MCP", "fail", f"{type(error).__name__}: {error}")
- for codex_arm in [a for a in ctx.arm_names if a in ca.CODEX_ARMS and not offline]:
- try:
- path, server = ctx.mcp[codex_arm]
- command, cargs, env = mcp_entry(path)
- tools = mcp_list_tools(command, cargs, env)
- add(
- "arm B MCP server starts and lists tools",
- "pass" if {"js", "js_reset"} <= set(tools) else "fail",
- f"server {server}: {tools}",
- )
- except Exception as error: # noqa: BLE001
- add("arm B MCP server", "fail", f"{type(error).__name__}: {error}")
for native_arm in [a for a in cxn.MCP_ARMS if a in ctx.arm_names]:
if not cxn.CODEX_CLI.is_file():
add("Codex CLI in the ChatGPT app", "fail", str(cxn.CODEX_CLI))
continue
- if not offline:
- try:
- path, server = ctx.mcp[native_arm]
- entry = cxn.server_entry(path, native_arm)
- source = cxn.SOURCE_ARM[native_arm]
- if source in ca.CUA_ARMS: # A23: Cua Driver under Codex uses that build's daemon
- ensure_agent_daemon(ctx, source)
- tools = mcp_list_tools(entry["command"], entry["args"], entry["env"])
- want = (
- {"js", "js_reset"}
- if server == cxn.SERVER
- else set(
- ca.load_pins().get("claude_cu_helper", {}).get("tools") or ["app_click"]
- )
- if server == ca.CU_HELPER_SERVER
- else {"list_apps", "run_script"}
- )
- add(
- f"{native_arm} {server} MCP server lists {sorted(want)}",
- "pass" if want <= set(tools) else "fail",
- f"{len(tools)} tools",
- )
- except Exception as error: # noqa: BLE001
- add(f"{native_arm} MCP server", "fail", f"{type(error).__name__}: {error}")
for name, status, detail in native_codex_checks(
ctx, native_arm, with_models and not offline
):
@@ -2104,8 +2151,6 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:
ctx.versions.setdefault("claude_cu_helper", {}).update(observed)
for name, status, detail in ca.check_cu_helper_pins(pins, observed):
add(name, status, detail)
- if offline:
- continue
for name, status, detail in cu_helper_live_checks(ctx, helper_arm):
add(name, status, detail)
cdb_ids = [t for t in ctx.task_ids if task_spec(ctx.tasks[t]).get("kind") == "cdb"]
@@ -2158,6 +2203,15 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:
"pass" if ok_copies else "fail",
json.dumps(per_name) if not ok_copies else f"{[v[0] for v in per_name.values()]}",
)
+ try:
+ subprocess.run(["/usr/bin/open", "-a", "Calculator"], check=True, timeout=30)
+ for arm in ctx.arm_names:
+ add(*real_capture_check(ctx, arm))
+ except (OSError, subprocess.SubprocessError) as error:
+ for arm in ctx.arm_names:
+ add(f"{arm} real capture", "fail", str(error)[:300])
+ finally:
+ reset_needs_apps(["Calculator"])
ps = subprocess.run(["ps", "-axo", "pid=,command="], capture_output=True, text=True).stdout
leftovers = [
ln.split(None, 1)[1][:80]
@@ -2176,20 +2230,21 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:
"pass" if not leftovers else "warn",
f"{leftovers}",
)
- if offline:
- add("daemons, MCP servers, recording, model calls", "warn", "skipped (--offline)")
- elif ctx.recorder is not None and ctx.recorder.name != "none":
+ if ctx.recorder is not None and ctx.recorder.name != "none":
test = ctx.run_dir / "preflight-rec"
shutil.rmtree(test, ignore_errors=True)
started = ctx.recorder.start(test)
time.sleep(3.0)
raw = ctx.recorder.stop() if started else None
dur = rec.ffprobe_duration(raw) if raw else None
- add(
- "screen recording (3 s test)",
- "pass" if raw and dur and dur > 0.5 else "fail",
- f"{raw} duration={dur} err={getattr(ctx.recorder, 'last_error', None)}",
- )
+ try:
+ if not raw or not dur or dur <= 0.5:
+ raise RuntimeError(
+ f"{raw} duration={dur} err={getattr(ctx.recorder, 'last_error', None)}"
+ )
+ add("recorder real capture (3 s test)", "pass", rec.capture_detail(raw))
+ except (OSError, RuntimeError, subprocess.SubprocessError) as error:
+ add("recorder real capture (3 s test)", "fail", str(error)[:300])
else:
add("screen recording", "warn", "disabled (--no-recorder)")
gate = core.gate_decision(
@@ -2339,7 +2394,7 @@ def build_parser() -> argparse.ArgumentParser:
p.add_argument(
"--offline",
action="store_true",
- help="preflight only: static checks, no daemon, no recording, no GUI, no model call",
+ help="preflight only: skip model calls; still start daemons, GUI captures and recording",
)
p.add_argument(
"--allow-unknown-quota",
@@ -2794,8 +2849,7 @@ def cmd_preflight(args: argparse.Namespace) -> int:
ctx = make_ctx(args)
setup_arms(ctx)
try:
- if not args.offline:
- start_recorder(ctx)
+ start_recorder(ctx)
checks = preflight(ctx, with_models=not args.no_model_preflight)
(ctx.run_dir / "preflight.json").write_text(json.dumps(checks, indent=2) + "\n", "utf-8")
write_manifest(
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_bench.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_bench.py
index 4ff6ccac1c..753e28b192 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_bench.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_bench.py
@@ -30,6 +30,64 @@ TASKS = [f"MB-{i:02d}" for i in range(1, 13)]
ARMS = ["cc-cua-driver", "cc-codex-cu"]
+class ToolSetupTest(unittest.TestCase):
+ def test_only_errored_tools_report_setup_failures(self) -> None:
+ errors = {
+ "screencapture": "could not create image from window",
+ "permissions_pending": "permissions_pending: macOS Accessibility or Screen Recording permission is still pending; no action started, retry after the permission gate completes",
+ "screen_recording": "screen_recording_permission_denied",
+ "accessibility": "Accessibility permission denied",
+ "tcc": "tcc_permission_denied",
+ "daemon_not_running": "Cua Driver daemon is not running",
+ "helper_exited": "helper exited during probe",
+ }
+ for pattern, text in errors.items():
+ for is_error in (True, False):
+ with self.subTest(pattern=pattern, is_error=is_error):
+ events = [
+ {"type": "system", "subtype": "init"},
+ {
+ "type": "user",
+ "message": {
+ "content": [
+ {
+ "type": "tool_result",
+ "is_error": is_error,
+ "content": [
+ {"type": "image", "data": "SECRET"},
+ {"type": "text", "text": "x" * 310 + " " + text},
+ ],
+ }
+ ]
+ },
+ },
+ ]
+ summary = ce.summarize(events)
+ failure = ce.classify_failure(summary)
+ self.assertEqual(failure["kind"], "tool_setup" if is_error else None)
+ if is_error:
+ self.assertEqual(failure["pattern"], pattern)
+ self.assertLessEqual(len(failure["message"]), 300)
+ self.assertNotIn("SECRET", json.dumps(summary))
+ summary = ce.summarize(
+ [{"type": "system", "subtype": "init"}, {"type": "result", "result": text}]
+ )
+ self.assertIsNone(ce.classify_failure(summary)["kind"])
+ for text in ("background_unavailable", "permission_denied"):
+ summary = ce.summarize(
+ [
+ {"type": "system", "subtype": "init"},
+ {
+ "type": "user",
+ "message": {
+ "content": [{"type": "tool_result", "is_error": True, "content": text}]
+ },
+ },
+ ]
+ )
+ self.assertIsNone(ce.classify_failure(summary)["kind"])
+
+
class ScheduleTest(unittest.TestCase):
def test_priority_order_mb_first_then_others(self) -> None:
ids = ["CDB-S01", "MB-10", "MB-02", "MB-13", "MB-01"]
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_claude_cu_helper_arm.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_claude_cu_helper_arm.py
index 882c7da00e..61d850178f 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_claude_cu_helper_arm.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_claude_cu_helper_arm.py
@@ -163,6 +163,33 @@ class Fixture:
return [r for r in rows if r["method"] == method]
+class LauncherCaptureTest(unittest.TestCase):
+ def test_capture_uses_launcher_without_wrapping_winlist(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ fixture = Fixture(Path(tmp), shot=(800, 600))
+ launcher = Path(tmp) / "launcher"
+ log = Path(tmp) / "launcher.log"
+ launcher.write_text('#!/bin/sh\nprintf "%s\\n" "$1" >> ' + str(log) + '\nexec "$@"\n')
+ launcher.chmod(0o755)
+ helper = ad.Helper([str(launcher), str(fixture.helper)])
+ adapter = ad.Adapter(
+ helper,
+ str(fixture.winlist),
+ list(ad.ALL_ACTIONS),
+ screencapture=str(fixture.screencapture),
+ sips=str(fixture.sips),
+ launcher=str(launcher),
+ )
+ try:
+ result = adapter.call("app_screenshot", {"app": "BenchLab", "window_id": 77})
+ self.assertTrue(any(block.get("type") == "image" for block in result))
+ self.assertEqual(
+ log.read_text().splitlines(), [str(fixture.helper), str(fixture.screencapture)]
+ )
+ finally:
+ helper.close()
+
+
class ArmRegistrationTest(unittest.TestCase):
def test_arm_is_a_named_claude_arm_with_the_agreed_label(self) -> None:
self.assertIn(ARM, arms.CLAUDE_ARMS)
@@ -456,6 +483,59 @@ class AdapterTest(unittest.TestCase):
class HelperClientTest(unittest.TestCase):
+ def test_trusted_helper_and_tools_do_not_hide_capture_failure(self) -> None:
+ from types import SimpleNamespace
+
+ with tempfile.TemporaryDirectory() as tmp:
+ fx = Fixture(Path(tmp))
+ launcher = Path(tmp) / "launcher"
+ launcher.write_text('#!/bin/sh\nexec "$@"\n')
+ launcher.chmod(0o755)
+ fx.winlist.write_text(
+ "#!/bin/sh\nprintf '%s\\n' '"
+ + json.dumps([{**WINDOWS[0], "app": "Calculator"}])
+ + "'\n"
+ )
+ fx.screencapture.write_text(
+ '#!/bin/sh\necho "could not create image from window" >&2\nexit 1\n'
+ )
+ config = Path(tmp) / "mcp.json"
+ config.write_text(
+ json.dumps(
+ {
+ "mcpServers": {
+ "helper": {
+ "command": sys.executable,
+ "args": [
+ str(ADAPTER),
+ "--helper",
+ str(fx.helper),
+ "--launcher",
+ str(launcher),
+ "--winlist",
+ str(fx.winlist),
+ "--screencapture",
+ str(fx.screencapture),
+ "--sips",
+ str(fx.sips),
+ ],
+ }
+ }
+ }
+ )
+ )
+ ctx = SimpleNamespace(
+ mcp={ARM: (config, "helper")}, arm_names=[ARM], versions={}, run_dir=Path(tmp)
+ )
+ with mock.patch.dict(
+ ca.CU_HELPER_BUILDS, {ARM: SimpleNamespace(helper=fx.helper, launcher=launcher)}
+ ):
+ checks = rb.cu_helper_live_checks(ctx, ARM)
+ self.assertTrue(all(status == "pass" for _, status, _ in checks), checks)
+ name, status, detail = rb.real_capture_check(ctx, ARM)
+ self.assertEqual((name, status), (f"{ARM} real capture", "fail"))
+ self.assertIn("could not create image from window", detail)
+
def test_helper_errors_and_restart(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
fx = Fixture(Path(tmp))
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_codex_native.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_codex_native.py
index 9bfcac3605..1f57ad5f52 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_codex_native.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_codex_native.py
@@ -16,6 +16,8 @@ import unittest
from pathlib import Path
from unittest import mock
+import pytest
+
HERE = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(HERE))
@@ -25,7 +27,8 @@ import claude_events # noqa: E402
import codex_native as cxn # noqa: E402
-def test_browser_cli_translation_and_rejections():
[email protected]("output", ["OUTPUT" * 1000, "short output"], ids=["truncated", "short"])
+def test_browser_cli_translation_and_rejections(output):
import browser_use_cli as bu
assert bu.ARM in arms.ALL_ARMS and bu.ARM in cxn.ARMS
@@ -44,7 +47,7 @@ def test_browser_cli_translation_and_rejections():
**item,
"status": "completed",
"exitCode": 0,
- "aggregatedOutput": "OUTPUT" * 1000,
+ "aggregatedOutput": output,
}
},
)
@@ -57,7 +60,7 @@ def test_browser_cli_translation_and_rejections():
]
result = events[-1]["message"]["content"][0]
assert result["tool_use_id"] == "cli1" and not result["is_error"]
- assert result["content"].startswith("OUTPUT") and len(result["content"]) == 2000
+ assert result["content"] == output[:2000]
assert tr.non_gui == []
for text in (
"browser-use open x; curl x",
@@ -545,6 +548,38 @@ for line in sys.stdin:
class TranslatorTest(unittest.TestCase):
+ def test_completed_mcp_is_error_becomes_setup_failure(self) -> None:
+ tr = cxn.Translator()
+ events = tr.feed("thread/started", {"thread": {"id": "t"}})
+ events += tr.feed(
+ "item/completed",
+ {
+ "item": {
+ "id": "tool",
+ "type": "mcpToolCall",
+ "server": "cua",
+ "tool": "screenshot",
+ "status": "completed",
+ "result": {
+ "isError": True,
+ "content": [
+ {"type": "image", "data": "SECRET"},
+ {
+ "type": "text",
+ "text": "screen_recording_permission_denied" + "!" * 500,
+ },
+ ],
+ },
+ }
+ },
+ )
+ summary = claude_events.summarize(events)
+ self.assertEqual(claude_events.classify_failure(summary)["kind"], "tool_setup")
+ block = events[-1]["message"]["content"][0]
+ self.assertTrue(block["is_error"])
+ self.assertLessEqual(len(block["content"]), 300)
+ self.assertNotIn("SECRET", json.dumps(events))
+
def run_events(self, events: list[tuple[str, dict]]) -> tuple[list[dict], cxn.Translator]:
tr = cxn.Translator()
out: list[dict] = []
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_runner_flow.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_runner_flow.py
index 7df72314a7..94b0e72f69 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_runner_flow.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_runner_flow.py
@@ -8,6 +8,8 @@ finalize. No GUI, no model."""
from __future__ import annotations
import argparse
+import base64
+import os
import contextlib
import json
import sys
@@ -195,7 +197,7 @@ class CodexAuthPreflightTest(unittest.TestCase):
self.assertNotIn(secret, json.dumps(checks))
def test_native_trial_removes_auth_link_and_flags_replacement(self) -> None:
- for replaced in (False, True):
+ for replaced in (False, True, "setup"):
with tempfile.TemporaryDirectory() as tmp, contextlib.ExitStack() as stack:
ctx = make_ctx(
tmp,
@@ -257,15 +259,41 @@ class CodexAuthPreflightTest(unittest.TestCase):
def run_codex(cmd, env, *args, **kwargs):
link = Path(env["CODEX_HOME"]) / "auth.json"
self.assertTrue(link.is_symlink())
- if replaced:
+ if replaced is True:
link.unlink()
link.write_text("FAKE_TOKEN")
+ if replaced == "setup":
+ events = [
+ {"type": "system", "subtype": "init"},
+ {
+ "type": "user",
+ "message": {
+ "content": [
+ {
+ "type": "tool_result",
+ "is_error": True,
+ "content": "screen_recording_permission_denied",
+ }
+ ]
+ },
+ },
+ {"type": "result", "result": "DONE"},
+ ]
+ args[5].write_text("\n".join(json.dumps(e) for e in events))
return {"returncode": 0, "result_seen": True, "wall_s": 0}
stack.enter_context(mock.patch.object(rb.cxn, "run_codex", side_effect=run_codex))
row = rb.run_attempt(ctx, entry, 1)
- self.assertEqual(row["infra_failure"] == "harness_exception", replaced)
- if replaced:
+ self.assertIn("infra_pattern", row)
+ if replaced == "setup":
+ self.assertEqual(row["infra_pattern"], "screen_recording")
+ self.assertEqual(row["infra_failure"], "tool_setup")
+ self.assertFalse(row["passed"])
+ self.assertTrue(row["excluded"])
+ else:
+ self.assertIsNone(row["infra_pattern"])
+ self.assertEqual(row["infra_failure"] == "harness_exception", replaced is True)
+ if replaced is True:
self.assertIn("symlink replaced", row["infra_message"])
link = ctx.run_dir / "trials" / entry["trial_id"] / "a1/codex-home/.codex/auth.json"
self.assertFalse(link.is_symlink())
@@ -320,8 +348,198 @@ def attempt_row(
return row
+class RealCaptureTest(unittest.TestCase):
+ def test_each_family_validates_capture_bytes_and_closes_client(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ ctx = make_ctx(tmp)
+ ffmpeg = Path(tmp) / "ffmpeg"
+ ffmpeg.write_text(
+ '#!/usr/bin/env python3\nimport sys\nfrom pathlib import Path\nsys.stdout.buffer.write(Path(sys.argv[sys.argv.index("-i")+1]).read_bytes())\n'
+ )
+ ffmpeg.chmod(0o755)
+ image = b"P6\n2 1\n255\n" + b"\x00\x01\x02\xff\xfe\xfd"
+ capture_detail = rb.rec.capture_detail
+ families = [
+ "cc-cua-driver",
+ "cc-codex-cu",
+ "cc-claude-cu-helper",
+ "cx-cua-driver",
+ "cx-codex-cu",
+ "cx-claude-cu-helper",
+ ]
+ for arm in families:
+ source = rb.cxn.SOURCE_ARM.get(arm, arm)
+ tools = [
+ {"name": n}
+ for n in ["list_apps", "get_desktop_state", "js", "js_reset", "run_script"]
+ + [f"tool{i}" for i in range(50)]
+ ]
+ if source in rb.ca.CU_HELPER_ARMS:
+ tools = [{"name": n} for n in rb.ca.load_pins()["claude_cu_helper"]["tools"]]
+ elif (
+ source in rb.ca.CUA_ARMS
+ and rb.ca.SCRIPT_FLAG not in rb.ca.CUA_BUILDS[source].daemon_env
+ ):
+ tools = [t for t in tools if t["name"] != "run_script"]
+ ctx.mcp[arm] = (Path(tmp) / f"{arm}.json", "server")
+ for payload, message in [
+ (image, None),
+ ("file", None),
+ ("legacy", None),
+ (b"", "no image block"),
+ (b"garbage", "decode failed"),
+ (b"P6\n2 1\n255\n" + b"\x00" * 6, "blank image"),
+ (b"P6\n2 1\n255\n" + b"\xff" * 6, "blank image"),
+ (None, "no image block"),
+ ("error", "could not create image from window"),
+ ("rpc", "permission denied"),
+ ("windows", "no layer-0 Calculator window"),
+ ]:
+ if payload == "windows" and source not in rb.ca.CU_HELPER_ARMS:
+ continue
+ if payload in ("file", "legacy") and source not in rb.ca.CUA_ARMS:
+ continue
+ advertised = [
+ {
+ **t,
+ "inputSchema": {
+ "properties": {"screenshot_out_file": {"type": "string"}}
+ },
+ }
+ if t["name"] == "get_desktop_state" and payload == "file"
+ else {"name": "screenshot"}
+ if t["name"] == "get_desktop_state" and payload == "legacy"
+ else t
+ for t in tools
+ ]
+ calls = []
+ client = mock.Mock()
+
+ def request(method, params, **kwargs):
+ calls.append((method, params))
+ if method == "tools/list":
+ return {"result": {"tools": advertised}}
+ if params["name"] == "check_permissions":
+ return {"result": {"content": []}}
+ if params["name"] == "app_list_windows":
+ return {
+ "result": {
+ "content": [
+ {
+ "type": "text",
+ "text": json.dumps(
+ []
+ if payload == "windows"
+ else [{"app": "Calculator", "window_id": 7}]
+ ),
+ }
+ ]
+ }
+ }
+ if payload == "error":
+ return {
+ "result": {
+ "isError": True,
+ "content": [
+ {
+ "type": "text",
+ "text": "could not create image from window",
+ }
+ ],
+ }
+ }
+ if payload == "rpc":
+ return {"error": {"message": "permission denied"}}
+ if payload == "file":
+ Path(params["arguments"]["screenshot_out_file"]).write_bytes(image)
+ return {"result": {"content": None}}
+ return {
+ "result": {
+ "content": None
+ if payload is None
+ else [
+ {
+ "type": "image",
+ "data": base64.b64encode(
+ image if payload == "legacy" else payload
+ ).decode(),
+ }
+ ]
+ }
+ }
+
+ client.request.side_effect = request
+ with (
+ self.subTest(arm=arm, message=message),
+ mock.patch.object(rb.rec, "McpStdioClient", return_value=client),
+ mock.patch.object(rb, "ensure_agent_daemon") as daemon,
+ mock.patch.object(rb, "mcp_entry", return_value=("mcp", [], {})),
+ mock.patch.object(
+ rb.cxn,
+ "server_entry",
+ return_value={"command": "native", "args": [], "env": {}},
+ ) as native,
+ mock.patch.object(
+ rb.rec,
+ "capture_detail",
+ side_effect=lambda path: capture_detail(path, str(ffmpeg)),
+ ),
+ ):
+ name, status, detail = rb.real_capture_check(ctx, arm)
+ self.assertEqual(name, f"{arm} real capture")
+ self.assertEqual(status, "fail" if message else "pass")
+ self.assertIn(message or "2x1 bytes=", detail)
+ self.assertLess(len(detail), 500)
+ client.close.assert_called_once()
+ if arm in rb.cxn.ARMS:
+ native.assert_called_once_with(ctx.mcp[arm][0], arm)
+ if source in rb.ca.CUA_ARMS:
+ daemon.assert_called_once_with(ctx, source)
+ self.assertEqual(
+ [p["name"] for m, p in calls if m == "tools/call"],
+ [
+ "check_permissions",
+ "screenshot" if payload == "legacy" else "get_desktop_state",
+ ],
+ )
+ if source == "cc-codex-cu":
+ code = calls[-1][1]["arguments"]["code"]
+ self.assertIn('cua.getApp("Calculator")', code)
+ self.assertIn("await app.getScreenshot()", code)
+ self.assertNotIn("emitImage", code)
+ self.assertNotIn(".screenshot()", code)
+ with self.assertRaises(FileNotFoundError):
+ capture_detail(Path(tmp) / "missing.png", str(Path(tmp) / "missing-ffmpeg"))
+
+ def test_stdio_client_answers_calculator_elicitation(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ script = Path(tmp) / "server.py"
+ script.write_text("""import json, sys
+for line in sys.stdin:
+ req = json.loads(line)
+ if req.get("method") == "initialize":
+ assert req["params"]["capabilities"] == {"elicitation": {}}
+ print(json.dumps({"id": req["id"], "result": {}}), flush=True)
+ elif req.get("method") == "tools/call":
+ pending = req["id"]
+ print(json.dumps({"id": "approval", "method": "elicitation/create", "params": {"message": 'Allow Computer Use to use "Calculator"?'}}), flush=True)
+ elif req.get("id") == "approval":
+ print(json.dumps({"id": pending, "result": req["result"]}), flush=True)
+""")
+ client = rb.rec.McpStdioClient(
+ [sys.executable, str(script)],
+ dict(os.environ),
+ elicitation=lambda text: rb.claude_driver.answer_elicitation(text, {"calculator"}),
+ )
+ try:
+ reply = client.request("tools/call", {"name": "js", "arguments": {}}, timeout=3)
+ self.assertEqual(reply["result"]["action"], "accept")
+ finally:
+ client.close()
+
+
class PreflightTest(unittest.TestCase):
- def test_native_mcp_is_live_only_online(self) -> None:
+ def test_offline_captures_every_native_arm_without_models(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
ctx = make_ctx(tmp, offline=True, allow_non_terminal=True)
ctx.task_ids = []
@@ -345,6 +563,16 @@ class PreflightTest(unittest.TestCase):
mock.patch.object(rb.ca, "check_cu_helper_pins", return_value=[]),
mock.patch.object(rb, "cu_helper_live_checks", return_value=[]),
mock.patch.object(rb, "native_codex_checks", return_value=[]) as native,
+ mock.patch.object(
+ rb,
+ "real_capture_check",
+ side_effect=lambda ctx, arm: (
+ f"{arm} real capture",
+ "fail" if arm == rb.cxn.ARMS[0] else "pass",
+ "capture",
+ ),
+ ) as capture,
+ mock.patch.object(rb, "reset_needs_apps") as reset,
mock.patch.object(rb, "mcp_list_tools") as probe,
mock.patch.object(rb, "ensure_agent_daemon") as daemon,
):
@@ -355,20 +583,14 @@ class PreflightTest(unittest.TestCase):
ctx.args.offline = True
probe.reset_mock()
daemon.reset_mock()
- probe.side_effect = AssertionError("offline MCP probe")
- daemon.side_effect = AssertionError("offline daemon")
+ capture.reset_mock()
rows = rb.preflight(ctx)
probe.assert_not_called()
daemon.assert_not_called()
native.assert_called_with(ctx, arm, False)
- self.assertIn(
- (
- "daemons, MCP servers, recording, model calls",
- "warn",
- "skipped (--offline)",
- ),
- rows,
- )
+ capture.assert_called_once_with(ctx, arm)
+ self.assertTrue(any(name == f"{arm} real capture" for name, _, _ in rows))
+ reset.assert_called_with(["Calculator"])
ctx.args.offline = False
probe.side_effect = None
probe.return_value = [
@@ -379,23 +601,55 @@ class PreflightTest(unittest.TestCase):
] + rb.ca.load_pins()["claude_cu_helper"]["tools"]
daemon.side_effect = None
rows = rb.preflight(ctx)
- probe.assert_called_once_with("mcp", [], {})
- source = rb.cxn.SOURCE_ARM[arm]
- if source in rb.ca.CUA_ARMS:
- daemon.assert_called_once_with(ctx, source)
- else:
- daemon.assert_not_called()
- self.assertTrue(
- any(
- name.startswith(f"{arm} ")
- and "MCP server lists" in name
- and status == "pass"
- for name, status, _ in rows
- )
- )
+ self.assertEqual(capture.call_count, 2)
+ ctx.arm_names = list(rb.cxn.MCP_ARMS)
+ ctx.args.offline = True
+ capture.reset_mock()
+ rows = rb.preflight(ctx)
+ self.assertEqual(capture.call_count, len(ctx.arm_names))
+ self.assertEqual(
+ len([r for r in rows if r[0].endswith(" real capture")]), len(ctx.arm_names)
+ )
+ ctx.recorder = mock.Mock(name="recording")
+ raw = Path(tmp) / "video.mov"
+ raw.write_bytes(b"video")
+ ctx.recorder.stop.return_value = raw
+ for blank in (True, False):
+ with (
+ mock.patch.object(rb.time, "sleep"),
+ mock.patch.object(rb.rec, "ffprobe_duration", return_value=3),
+ mock.patch.object(
+ rb.rec,
+ "capture_detail",
+ side_effect=RuntimeError("blank image") if blank else None,
+ return_value=f"2x1 bytes=5 path={raw}",
+ ),
+ ):
+ rows = rb.preflight(ctx)
+ recording = next(r for r in rows if r[0].startswith("recorder real capture"))
+ self.assertEqual(recording[1], "fail" if blank else "pass")
class ExecuteTrialTest(unittest.TestCase):
+ def test_daemon_setup_failure_retries_with_restart_then_excludes(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ ctx = make_ctx(tmp)
+ rows = [
+ attempt_row(i, "tool_setup", infra_pattern="daemon_not_running") for i in (1, 2)
+ ]
+ with (
+ mock.patch.object(rb.rec, "free_gb", return_value=1000),
+ mock.patch.object(rb, "ensure_agent_daemon") as restart,
+ ):
+ row, pauses, error = self.run_trial(ctx, rows)
+ self.assertIsNone(error)
+ self.assertEqual(row["infra_pattern"], "daemon_not_running")
+ self.assertTrue(row["excluded"])
+ self.assertFalse(row["passed"])
+ self.assertEqual(row["attempt"], 2)
+ self.assertEqual(len(pauses), 1)
+ restart.assert_called_once_with(ctx, ENTRY["arm"])
+
def run_trial(
self, ctx: rb.Ctx, rows: list[dict]
) -> tuple[dict | None, list[tuple[str, float]], Exception | None]:
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tools/check_named_electron.py b/libs/cua-driver-bench/automated-eval/macos_bench/tools/check_named_electron.py
index 4ef3d56ba6..2ae4100233 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/tools/check_named_electron.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tools/check_named_electron.py
@@ -15,7 +15,6 @@ returns. App approvals asked by the Codex server are accepted only for the task'
from __future__ import annotations
import json
-import select
import subprocess
import sys
import tempfile
@@ -28,6 +27,7 @@ sys.path.insert(0, str(HERE.parent))
import cdb_adapter # noqa: E402
import claude_arms as ca # noqa: E402
import run_bench # noqa: E402
+import recorder # noqa: E402
def lsappinfo(pid: int) -> dict[str, str]:
@@ -56,65 +56,15 @@ def codex_get_apps(names: list[str], allowed: set[str], timeout: float = 60.0) -
if config is None:
return {"error": "no Codex computer-use MCP config"}
command, args, env = run_bench.mcp_entry(config)
- proc = subprocess.Popen(
- [command, *args],
- env=env,
- stdin=subprocess.PIPE,
- stdout=subprocess.PIPE,
- stderr=subprocess.DEVNULL,
- text=True,
- bufsize=1,
- )
approvals: list[dict] = []
- def send(message: dict) -> None:
- assert proc.stdin
- proc.stdin.write(json.dumps(message) + "\n")
- proc.stdin.flush()
-
- def wait_for(msg_id: int, limit: float) -> dict | None:
- assert proc.stdout
- deadline = time.monotonic() + limit
- while time.monotonic() < deadline:
- ready, _, _ = select.select([proc.stdout], [], [], deadline - time.monotonic())
- if not ready:
- return None
- line = proc.stdout.readline()
- if not line:
- return None
- msg = json.loads(line)
- if msg.get("method") == "elicitation/create":
- text = (msg.get("params") or {}).get("message", "")
- answer = run_bench.claude_driver.answer_elicitation(text, allowed)
- approvals.append({"message": text, "answer": answer["action"]})
- send(
- {
- "jsonrpc": "2.0",
- "id": msg["id"],
- "result": {"action": answer["action"], "content": {}},
- }
- )
- continue
- if msg.get("id") == msg_id:
- return msg
- return None
+ def approve(text: str) -> dict:
+ answer = run_bench.claude_driver.answer_elicitation(text, allowed)
+ approvals.append({"message": text, "answer": answer["action"]})
+ return answer
+ client = recorder.McpStdioClient([command, *args], env, elicitation=approve)
try:
- send(
- {
- "jsonrpc": "2.0",
- "id": 1,
- "method": "initialize",
- "params": {
- "protocolVersion": "2025-06-18",
- "capabilities": {"elicitation": {}},
- "clientInfo": {"name": "cdb-check-named-electron", "version": "1"},
- },
- }
- )
- if wait_for(1, 30) is None:
- return {"error": "no initialize reply"}
- send({"jsonrpc": "2.0", "method": "notifications/initialized"})
code = (
f"const names = {json.dumps(names)}; const out = {{}};"
"for (const n of names) { try { const a = await cua.getApp(n);"
@@ -123,17 +73,9 @@ def codex_get_apps(names: list[str], allowed: set[str], timeout: float = 60.0) -
"let listed = null; try { listed = await cua.listApps({emit:false}); } catch (e) { listed = String(e); }"
"nodeRepl.write(JSON.stringify({getApp: out, listApps: listed}));"
)
- send(
- {
- "jsonrpc": "2.0",
- "id": 2,
- "method": "tools/call",
- "params": {"name": "js", "arguments": {"code": code}},
- }
+ reply = client.request(
+ "tools/call", {"name": "js", "arguments": {"code": code}}, timeout=timeout
)
- reply = wait_for(2, timeout)
- if reply is None:
- return {"error": "js call timed out", "approvals": approvals}
texts = [c.get("text", "") for c in (reply.get("result") or {}).get("content", [])]
data = {"raw": [t[:300] for t in texts]}
for (
@@ -149,7 +91,7 @@ def codex_get_apps(names: list[str], allowed: set[str], timeout: float = 60.0) -
data["approvals"] = approvals
return data
finally:
- proc.terminate()
+ client.close()
def main() -> int:
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tools/claude_cu_helper/cu_helper_mcp.py b/libs/cua-driver-bench/automated-eval/macos_bench/tools/claude_cu_helper/cu_helper_mcp.py
index 6494c80753..86b835b9d6 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/tools/claude_cu_helper/cu_helper_mcp.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tools/claude_cu_helper/cu_helper_mcp.py
@@ -13,8 +13,9 @@ does not provide:
* **Which windows exist** (``app_list_windows``): ``winlist``, a small Swift tool built in the VM that prints
``CGWindowListCopyWindowInfo`` (owner, pid, window id, bounds, title). Public macOS API only.
-* **What a window shows** (``app_screenshot``): ``/usr/sbin/screencapture -l <window id>`` of that one window, the
- stock macOS tool, downscaled with ``/usr/bin/sips`` to Claude Code's own screenshot budget. Desktop captures with
+* **What a window shows** (``app_screenshot``): ``/usr/sbin/screencapture -l <window id>`` of that one window,
+ wrapped by ``--launcher`` when supplied. Grant Apple's screencapture Screen Recording, never Terminal or our
+ adapter/launcher. The stock macOS capture is downscaled with ``/usr/bin/sips`` to Claude Code's own screenshot budget. Desktop captures with
its in-process native module (``@ant/claude-swift``, ``appScoped.captureWindow``), not with the helper, so that path
is not reachable here. Like Desktop, the adapter asks the helper to wake a Chromium window's compositor first.
@@ -185,11 +186,11 @@ def target_size(width: int, height: int) -> tuple[int, int]:
return max(1, int(width * scale)), max(1, int(height * scale))
-def capture_window(window_id: int, screencapture: str = SCREENCAPTURE, sips: str = SIPS) -> tuple[bytes, int, int]:
+def capture_window(window_id: int, screencapture: str = SCREENCAPTURE, sips: str = SIPS, launcher: str | None = None) -> tuple[bytes, int, int]:
"""PNG of one window (no shadow), downscaled to the screenshot budget. Returns (png, width, height)."""
with tempfile.TemporaryDirectory(prefix="cuhelper-") as tmp:
path = os.path.join(tmp, "w.png")
- done = subprocess.run([screencapture, "-x", "-o", "-t", "png", f"-l{int(window_id)}", path],
+ done = subprocess.run(([launcher] if launcher else []) + [screencapture, "-x", "-o", "-t", "png", f"-l{int(window_id)}", path],
capture_output=True, text=True, timeout=20)
if done.returncode != 0 or not os.path.exists(path) or os.path.getsize(path) == 0:
raise RuntimeError(f"screencapture failed (rc={done.returncode}): {done.stderr.strip()[:300]}")
@@ -232,11 +233,12 @@ class Adapter:
indexes, menus, AX text entry) is not part of the helper and is not offered."""
def __init__(self, helper: Helper, winlist: str, actions: list[str], host_pid: int | None = None,
- screencapture: str = SCREENCAPTURE, sips: str = SIPS, busy_wait_s: float = BUSY_WAIT_S) -> None:
+ screencapture: str = SCREENCAPTURE, sips: str = SIPS, busy_wait_s: float = BUSY_WAIT_S, launcher: str | None = None) -> None:
self.helper = helper
self.winlist = winlist
self.actions = [a for a in ALL_ACTIONS if a in actions]
self.host_pid = host_pid if host_pid is not None else os.getpid()
+ self.launcher = launcher
self.screencapture = screencapture
self.sips = sips
self.busy_wait_s = busy_wait_s
@@ -342,7 +344,7 @@ class Adapter:
"hostPid": self.host_pid, "debug": True})
except HelperError as error:
log(f"wakeChromiumCompositor: {error}")
- data, width, height = capture_window(wid, self.screencapture, self.sips)
+ data, width, height = capture_window(wid, self.screencapture, self.sips, self.launcher)
self.frames[wid] = (float(win["width"]), float(win["height"]), width, height)
text = (f"{win.get('app')} window {wid} ({win.get('title') or 'untitled'}): screenshot {width}x{height} px; "
"coordinates are pixels of this image.")
@@ -493,14 +495,14 @@ def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("--helper", required=True)
parser.add_argument("--winlist", required=True)
- parser.add_argument("--launcher", help="cu-disclaim: start the helper as its own responsible process (A11)")
+ parser.add_argument("--launcher", help="cu-disclaim: start helper and screencapture as their own responsible processes (A11)")
parser.add_argument("--actions", default=",".join(ALL_ACTIONS))
parser.add_argument("--screencapture", default=SCREENCAPTURE)
parser.add_argument("--sips", default=SIPS)
args = parser.parse_args(argv)
helper = Helper([args.launcher, args.helper] if args.launcher else [args.helper], env={"PATH": "/usr/bin:/bin", "HOME": os.environ.get("HOME", "/tmp")})
adapter = Adapter(helper, args.winlist, [a for a in args.actions.split(",") if a],
- screencapture=args.screencapture, sips=args.sips)
+ screencapture=args.screencapture, sips=args.sips, launcher=args.launcher)
log(f"adapter {ADAPTER_VERSION} up; helper {args.helper}; actions {adapter.actions}")
serve(adapter)
return 0