Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cua-driver-bench): pi-hosted arms for the four Cua Driver setups (#6642)
injaneitybb42e3ec01e0
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/RUNNER.md b/libs/cua-driver-bench/automated-eval/macos_bench/RUNNER.mdindex 9310d8f864..b8b2ac970d 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/RUNNER.md+++ b/libs/cua-driver-bench/automated-eval/macos_bench/RUNNER.md@@ -26,6 +26,7 @@ observed-pin comparison and doctor/CDP/page-info/deny probes remain prerequisite| `cc-browser-use` | Browser Use harness | Browser Use macOS harness 0.2.1 behind `tools/browser_use_harness/bu_mcp.py` as MCP server `browseruse` (`macos_harness`, `view_screenshot`), the vendor skill as server instructions || `cx-cua-driver`, `cx-codex-cu`, `cx-claude-cu-helper` | Cua Driver, Codex Computer-Use (Sky) and Claude Computer-Use (Desktop helper, with the label above), driven by Codex CLI (Amendment 23) | Not Claude Code: the bundled Codex CLI as `codex app-server` (`codex_native.py`) with GPT-6.1 Sol (`gpt-6.1-sol`, default effort low), given the MCP entry and system prompt of `cc-cua-driver-script`, `cc-codex-cu` or `cc-claude-cu-helper` || `cc-cua-driver`, `cc-cua-driver-main`, `cc-cua-driver-main-skill` | Internal baselines | Cua Driver 0.34.0 release; main with `run_script` off; main with the skill in the system prompt |+| `pi-cua-driver`, `pi-cua-driver-main`, `pi-cua-driver-main-skill`, `pi-cua-driver-script` | The four Cua Driver setups above, driven by pi | Not Claude Code: `pi --mode json` (`pi_arms.py`) on `openai-codex` / `gpt-6.1-terra` by default (`CDB_PI_PROVIDER`, `CDB_PI_MODEL`, `CDB_PI_THINKING`), given the driver daemon, MCP entry, skill and system prompt of the matching `cc-*` arm. Tools: `read` (limited to the skill's `SKILL.md`, and the turn cap enforced, by `pi_gate.ts`), `tool_search` and the deferred `cua` server. Linux and macOS. |Arms `codex-native-cu` and `codex-cua-driver` (`codex exec --json`, from`run_pilot.py`) exist behind `--allow-codex-arms` and are not part of Bench v2.diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/arms.py b/libs/cua-driver-bench/automated-eval/macos_bench/arms.pyindex 3a947c35ca..832e5431c4 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/arms.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/arms.py@@ -53,7 +53,14 @@ NATIVE_CODEX_ARMS = ("cx-claude-cu-helper","cx-browser-use-cli",) # the last two: Amendment 23-ALL_ARMS = CLAUDE_ARMS + CODEX_FALLBACK_ARMS + NATIVE_CODEX_ARMS+# The Cua Driver arms with pi as the agent host (pi_arms.py)+PI_ARMS = (+ "pi-cua-driver",+ "pi-cua-driver-main",+ "pi-cua-driver-main-skill",+ "pi-cua-driver-script",+)+ALL_ARMS = CLAUDE_ARMS + CODEX_FALLBACK_ARMS + NATIVE_CODEX_ARMS + PI_ARMSCODEX_APP_RESOURCES = Path("/Applications/ChatGPT.app/Contents/Resources")CODEX_CU_DIR = Path.home() / ".codex" / "computer-use"diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/pi_arms.py b/libs/cua-driver-bench/automated-eval/macos_bench/pi_arms.pynew file mode 100644index 0000000000..7f06d7c5d4--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/pi_arms.py@@ -0,0 +1,495 @@+# SPDX-License-Identifier: FSL-1.1-MIT+# Copyright (c) 2026 Cua AI, Inc.++"""The Cua Driver setups with pi as the agent host instead of Claude Code.++Each ``pi-*`` arm mirrors one ``cc-*`` Cua Driver arm (``SOURCE_ARM``): the same driver build and daemon, the same MCP+server entry (that arm's ``--mcp-config``, copied), skill directory, system prompt, brief, turn cap and wall-time+limit. Only the host and the model differ: ``pi --mode json`` on ``openai-codex`` / ``gpt-6.1-terra`` by default+(``CDB_PI_PROVIDER``, ``CDB_PI_MODEL``, ``CDB_PI_THINKING``).++The tool surface mirrors Claude Code's ``Skill, Read, ToolSearch`` plus ``mcp__cua``: pi's ``read`` and ``tool_search``+built-ins, the ``cua`` server with ``deferred`` exposure (loaded through ``tool_search``, as Claude Code defers it+behind ToolSearch) and the build's skill through ``--skill``. Claude Code denies every ``Read`` in those arms+(``dontAsk`` with only ``mcp__cua`` allowed) and loads the skill through ``Skill``, so ``pi_gate.ts`` blocks+every ``read`` except the skill's ``SKILL.md``. Every other built-in, extension, context file, prompt+template and codemode is off. ``Translator`` turns pi's JSON events into the Claude-shaped stream that the GUI-only+guard and ``claude_events.summarize`` read; a pi tool outside that surface maps to a forbidden built-in, so it ends+the trial as a violation.++pi has no turn cap, so ``pi_gate.ts`` gives it Claude Code's ``--max-turns``: the ``max_turns``-th turn may run its+tools, the next one is aborted before its model request, and the host records ``error_max_turns``. A pi that exits+without settling (a crash, not a cap, timeout or abort) leaves no result event, so the row is a harness failure. Cost is pi's own figure for the reported tokens at the model's list price;+``openai-codex`` is a ChatGPT subscription, so it is not a billed amount.++Nothing here reads credentials: the trial's pi agent directory, under the private ``/tmp`` of the session, gets a+symlink to the operator's ``auth.json`` (``CDB_PI_AUTH_FILE``, default ``~/.pi/agent/auth.json``); only its+configuration is copied into the run directory.+"""++from __future__ import annotations++import hashlib+import json+import os+import re+import shutil+import signal+import subprocess+import tempfile+import threading+import time+from pathlib import Path+from typing import Any++from arms import PI_ARMS as ARMS+from claude_events import _text_of+from desktop_runtime import record_or_reap, stop_group++SOURCE_ARM = {arm: "cc-" + arm.removeprefix("pi-") for arm in ARMS}+PROVIDER = os.environ.get("CDB_PI_PROVIDER") or "openai-codex"+MODEL = os.environ.get("CDB_PI_MODEL") or "gpt-6.1-terra"+THINKING = (+ os.environ.get("CDB_PI_THINKING") or "medium"+) # pi's own default level, passed explicitly+PI_BIN = Path(os.environ.get("CDB_PI_BIN") or shutil.which("pi") or Path.home() / ".local/bin/pi")+AGENT_DIR = Path(os.environ.get("CDB_PI_AGENT_DIR") or Path.home() / ".pi" / "agent")+TOOLS = ("read", "tool_search", "mcp__cua__*")+GATE = Path(__file__).resolve().parent / "pi_gate.ts"+# pi built-in -> the Claude Code built-in the guard knows; any other non-MCP tool is "Task" (forbidden in v2)+AS_CLAUDE = {+ "read": "Read",+ "tool_search": "ToolSearch",+ "bash": "Bash",+ "powershell": "Bash",+ "edit": "Edit",+ "write": "Write",+}+COST_SOURCE = "pi usage cost at list price (openai-codex subscription, not billed)"+DESCRIPTIONS = {arm: f"pi + {PROVIDER}/{MODEL}, the {SOURCE_ARM[arm]} setup" for arm in ARMS}+++def claude_name(name: str) -> str:+ return name if name.startswith("mcp__cua__") else AS_CLAUDE.get(name, "Task")+++def skill_dir(cwd: Path) -> Path:+ return (+ cwd / ".claude" / "skills" / "cua-driver"+ ) # written by claude_arms.prepare_cwd(source arm)+++def render_agent_dir(root: Path, mcp_json: Path, skills: Path, max_turns: int) -> dict[str, str]:+ """A fresh pi agent directory and HOME under ``root`` for one trial; returns pi's closed environment."""+ agent_dir = root / "agent"+ agent_dir.mkdir(parents=True, exist_ok=True)+ entry = json.loads(mcp_json.read_text("utf-8"))["mcpServers"]["cua"]+ config = {"mcpServers": {"cua": {**entry, "exposure": "deferred"}}, "autoEnableCodemode": False}+ (agent_dir / "mcp.json").write_text(json.dumps(config, indent=2) + "\n", "utf-8")+ settings = {"defaultProvider": PROVIDER, "defaultModel": MODEL}+ (agent_dir / "settings.json").write_text(json.dumps(settings) + "\n", "utf-8")+ store = AGENT_DIR / "models-store.json" # the model catalog: context window and list prices+ if store.is_file():+ shutil.copyfile(store, agent_dir / "models-store.json")+ auth = (+ Path(os.environ.get("CDB_PI_AUTH_FILE") or AGENT_DIR / "auth.json").expanduser().absolute()+ )+ if not auth.is_file():+ raise FileNotFoundError("pi auth file is missing")+ (agent_dir / "auth.json").symlink_to(auth)+ home = root / "home"+ (home / "tmp").mkdir(parents=True, exist_ok=True)+ return {+ "HOME": str(home),+ "PATH": f"{PI_BIN.parent}:/usr/local/bin:/usr/bin:/bin",+ "TMPDIR": str(home / "tmp"),+ "LANG": "en_US.UTF-8",+ "PI_CODING_AGENT_DIR": str(agent_dir),+ "PI_OFFLINE": "1",+ "PI_SKIP_VERSION_CHECK": "1",+ "PI_TELEMETRY": "0",+ "CDB_PI_READ_ALLOW": str(skills / "SKILL.md"), # pi_gate.ts+ "CDB_PI_MAX_TURNS": str(max_turns),+ }+++def new_root() -> Path:+ """The trial's pi state, outside the run directory (the session's private /tmp on Linux)."""+ return Path(tempfile.mkdtemp(prefix="cdb-pi-"))+++def keep_config(root: Path, dest: Path) -> None:+ """Copy the trial's pi configuration (never the credential) to ``dest``, then delete ``root``."""+ dest.mkdir(parents=True, exist_ok=True)+ for name in ("mcp.json", "settings.json"):+ if (root / "agent" / name).is_file():+ shutil.copyfile(root / "agent" / name, dest / name)+ shutil.rmtree(root, ignore_errors=True)+++def argv(system_prompt: str, skills: Path | None, pi_bin: Path | None = None) -> list[str]:+ out = [+ str(pi_bin or PI_BIN),+ "--mode",+ "json",+ "--no-session",+ "--provider",+ PROVIDER,+ "--model",+ MODEL,+ "--thinking",+ THINKING,+ "--system-prompt",+ system_prompt,+ "--no-extensions",+ "-e",+ "builtin:mcp",+ "-e",+ "builtin:tool-search",+ "-e",+ str(GATE),+ "--no-skills",+ "--no-context-files",+ "--no-prompt-templates",+ "--no-themes",+ "--tools",+ ",".join(TOOLS),+ ]+ return out + (["--skill", str(skills)] if skills else [])+++class Translator:+ """pi JSON events -> Claude-shaped stream-json events (assistant tool_use / user tool_result / init / result)."""++ def __init__(self, max_turns: int) -> None:+ self.max_turns = max_turns+ self.turns = 0+ self.final_text = ""+ self.error = ""+ self.last_stop: str | None = None+ self.started = 0+ self.capped = (+ threading.Event()+ ) # a turn past the cap started; pi_gate.ts aborts it before its request+ self.usage = {"input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0}+ self.cost = 0.0+ self.models: set[str] = set()+ self.sections: dict[str, Any] = {}+ self.declared: list[str] = []+ self.mcp_loaded = False+ self.empty_search = False+ self.settled = False++ def feed(self, event: dict[str, Any]) -> list[dict[str, Any]]:+ if self.capped.is_set():+ return [] # the aborted turn past the cap is not a turn of the trial+ kind = event.get("type")+ if kind == "turn_start":+ self.started += 1+ if self.started > self.max_turns:+ self.capped.set()+ return []+ message = event.get("message") or {}+ role = message.get("role")+ if kind == "message_end" and role == "system":+ added = [str(t.get("name")) for t in message.get("toolsAdded") or []]+ self.declared += added+ self.mcp_loaded |= any(t.startswith("mcp__") for t in added)+ self.sections = self.sections or message.get("sections") or {}+ elif kind == "message_end" and role == "assistant":+ self.turns += 1+ usage = message.get("usage") or {}+ for key in self.usage:+ self.usage[key] += int(usage.get(key) or 0)+ self.cost += float((usage.get("cost") or {}).get("total") or 0.0)+ self.models.add(f"{message.get('provider')}/{message.get('model')}")+ content: list[dict[str, Any]] = []+ for block in message.get("content") or []:+ if block.get("type") == "text":+ content.append({"type": "text", "text": str(block.get("text") or "")})+ elif block.get("type") == "toolCall":+ content.append(+ {+ "type": "tool_use",+ "id": str(block.get("id")),+ "name": claude_name(str(block.get("name"))),+ "input": block.get("arguments") or {},+ }+ )+ text = "".join(b["text"] for b in content if b["type"] == "text")+ self.final_text = text if text.strip() else self.final_text+ self.last_stop = message.get("stopReason")+ if self.last_stop == "error":+ self.error = str(message.get("errorMessage") or "pi provider error")+ return [+ {+ "type": "assistant",+ "message": {+ "id": message.get("responseId") or f"pi-{self.turns}",+ "content": content,+ "usage": {+ "input_tokens": int(usage.get("input") or 0),+ "output_tokens": int(usage.get("output") or 0),+ "cache_read_input_tokens": int(usage.get("cacheRead") or 0),+ "cache_creation_input_tokens": int(usage.get("cacheWrite") or 0),+ },+ },+ }+ ]+ elif kind == "tool_execution_end":+ text = _text_of((event.get("result") or {}).get("content"))+ self.empty_search |= (+ event.get("toolName") == "tool_search" and "No matching tools" in text+ )+ failed = bool(event.get("isError"))+ block = {+ "type": "tool_result",+ "tool_use_id": str(event.get("toolCallId")),+ "is_error": failed,+ "content": text[:300] if failed else "",+ }+ return [{"type": "user", "message": {"content": [block]}}]+ elif kind == "agent_settled":+ self.settled = True+ elif kind == "auto_retry_start":+ retry = {"error": event.get("errorMessage"), "attempt": event.get("attempt")}+ return [+ {+ "type": "system",+ "subtype": "api_retry",+ **retry,+ "retry_delay_ms": event.get("delayMs"),+ }+ ]+ return []++ def finish(+ self, capped: bool, wall_ms: float, returncode: int | None = 0, stopped: bool = False+ ) -> list[dict[str, Any]]:+ """The init and result events, written when pi has stopped (``stopped``: the host ended it on purpose, at a+ timeout or an abort). None when pi produced no turn, exited without settling, or exited nonzero without a+ provider error, so the row is classified from stderr as a harness failure (retried, never scored) rather+ than scored as an agent result. A provider error keeps its message, so a 429 or 5xx is classified as such."""+ if not self.turns or not (capped or stopped or self.settled):+ return []+ is_error = self.last_stop == "error"+ if not (capped or stopped or is_error) and returncode not in (0, None):+ return []+ status = next(+ (c for c in (429, 401, 403, 500, 502, 503, 504, 529) if str(c) in self.error), None+ )+ # the server is "failed" only when tool_search found nothing and no cua tool ever loaded; unused is "ready"+ mcp = "connected" if self.mcp_loaded else ("failed" if self.empty_search else "ready")+ init = {+ "type": "system",+ "subtype": "init",+ "model": MODEL,+ "tools": [claude_name(t) for t in dict.fromkeys(self.declared)],+ "mcp_servers": [{"name": "cua", "status": mcp}],+ "skills": re.findall(r"<name>(.*?)</name>", str(self.sections.get("skills") or "")),+ "pi_prompt_sections": sorted(self.sections),+ }+ result = {+ "type": "result",+ "subtype": "error_max_turns"+ if capped+ else ("error_during_execution" if is_error else "success"),+ "is_error": is_error or capped,+ "api_error_status": status,+ "num_turns": self.turns,+ "duration_ms": round(wall_ms),+ "result": self.error[:2000] if is_error else self.final_text,+ "usage": {+ "input_tokens": self.usage["input"],+ "output_tokens": self.usage["output"],+ "cache_read_input_tokens": self.usage["cacheRead"],+ "cache_creation_input_tokens": self.usage["cacheWrite"],+ },+ "total_cost_usd": round(self.cost, 6),+ "pi_models": sorted(self.models),+ }+ return [init, result]+++def _strip(value: Any, limit: int = 4000) -> Any:+ if isinstance(value, str):+ return value if len(value) <= limit else f"{value[:120]}...<{len(value)} chars stripped>"+ if isinstance(value, list):+ return [_strip(v, limit) for v in value]+ if isinstance(value, dict):+ return {k: _strip(v, limit) for k, v in value.items()}+ return value+++def run_pi(+ cmd: list[str],+ env: dict[str, str],+ cwd: Path,+ prompt: str,+ timeout_s: float,+ max_turns: int,+ raw_path: Path,+ stream_path: Path,+ stderr_path: Path,+ abort: Any = None,+) -> dict[str, Any]:+ """One trial: pi in its own process group with the prompt on stdin, stopped at exit, the wall-time limit, the+ turn cap or ``abort``. pi's events (bulk stripped, streaming deltas dropped) go to ``raw_path``; the translated+ events go to ``stream_path`` as they arrive (both ``<epoch_ms>\\t<json>``), so the GUI-only guard reads them live."""+ started = time.monotonic()+ translator = Translator(max_turns)+ lock = threading.Lock()+ with (+ raw_path.open("w", encoding="utf-8") as raw,+ stream_path.open("w", encoding="utf-8") as stream,+ stderr_path.open("wb") as err,+ ):+ proc = subprocess.Popen(+ cmd,+ cwd=str(cwd),+ env=env,+ stdin=subprocess.PIPE,+ stdout=subprocess.PIPE,+ stderr=err,+ start_new_session=True,+ )+ record_or_reap(proc, "pi")++ def write(obj: dict[str, Any]) -> None:+ stream.write(f"{time.time() * 1000:.1f}\t{json.dumps(obj)}\n")+ stream.flush()++ def pump() -> None:+ assert proc.stdout is not None+ for line in proc.stdout: # binary: records split on LF only (pi's JSONL framing)+ try:+ event = json.loads(line)+ except ValueError:+ continue+ if not isinstance(event, dict):+ continue+ with lock:+ if event.get("type") != "message_update":+ raw.write(f"{time.time() * 1000:.1f}\t{json.dumps(_strip(event))}\n")+ raw.flush()+ for out in translator.feed(event):+ write(out)++ reader = threading.Thread(target=pump, daemon=True)+ reader.start()+ try:+ assert proc.stdin is not None+ proc.stdin.write(prompt.encode())+ proc.stdin.close()+ except OSError:+ pass+ timed_out = aborted = False+ while proc.poll() is None and not translator.capped.is_set():+ if time.monotonic() - started > timeout_s:+ timed_out = True+ break+ if abort is not None and abort.is_set():+ aborted = True+ break+ time.sleep(0.25)+ stop_group(proc.pid)+ reader.join(timeout=5)+ # read after the reader has drained pi's output: pi may print the over-cap turn_start and exit at once+ capped = translator.capped.is_set() and not timed_out and not aborted+ try:+ os.killpg(proc.pid, signal.SIGKILL) # the MCP server child must not outlive the trial+ except OSError:+ pass+ try:+ proc.wait(timeout=5)+ except subprocess.TimeoutExpired:+ pass+ with lock:+ wall_ms = (time.monotonic() - started) * 1000+ for out in translator.finish(capped, wall_ms, proc.returncode, timed_out or aborted):+ write(out)+ return {+ "returncode": proc.returncode,+ "timed_out": timed_out,+ "aborted": aborted,+ "capped": capped,+ "result_seen": bool(translator.turns),+ "wall_s": round(time.monotonic() - started, 2),+ "elicitations": [],+ "turns": translator.turns,+ "models": sorted(translator.models),+ }+++def observed(pi_bin: Path | None = None) -> dict[str, Any]:+ """The host's identity, recorded in the manifest and every row."""+ pi_bin = pi_bin or PI_BIN+ version = None+ sha = None+ if pi_bin.exists():+ env = {+ "PATH": f"{pi_bin.parent}:/usr/bin:/bin",+ "PI_OFFLINE": "1",+ "PI_SKIP_VERSION_CHECK": "1",+ }+ version = subprocess.run(+ [str(pi_bin), "--version"], capture_output=True, text=True, timeout=30, env=env+ ).stdout.strip()+ sha = hashlib.sha256(Path(os.path.realpath(pi_bin)).read_bytes()).hexdigest()+ store = AGENT_DIR / "models-store.json"+ return {+ "host": "pi --mode json",+ "pi_bin": str(pi_bin),+ "pi_version": version,+ "pi_cli_sha256": sha,+ "provider": PROVIDER,+ "model": MODEL,+ "thinking": THINKING,+ "tools": list(TOOLS),+ "mcp_exposure": "deferred",+ "gate_sha256": hashlib.sha256(GATE.read_bytes()).hexdigest(),+ "models_store_sha256": hashlib.sha256(store.read_bytes()).hexdigest()+ if store.is_file()+ else None,+ }+++def init_check(+ arm: str, mcp_json: Path, cwd: Path, system_prompt: str, work: Path+) -> tuple[str, str]:+ """One trivial turn through the real argv: the declared tools, the skill and the model that answered."""+ import claude_events++ shutil.rmtree(work, ignore_errors=True)+ work.mkdir(parents=True)+ stream = work / "stream.tsv"+ root = new_root()+ try:+ env = render_agent_dir(root, mcp_json, skill_dir(cwd), 3)+ result = run_pi(+ argv(system_prompt, skill_dir(cwd)),+ env,+ cwd,+ "Reply OK. Do not call any tool.",+ 180,+ 3,+ work / "pi-events.tsv",+ stream,+ work / "stderr.txt",+ )+ finally:+ keep_config(root, work / "pi-agent")+ summary = claude_events.summarize(claude_events.read_events(stream))+ text = (summary.get("final_text") or "").strip()+ init = summary["init"]+ problems = []+ if not text.upper().startswith(("OK", "DONE")) or summary["tool_calls"]["total"]:+ problems.append(f"reply={text[:40]!r} tools={summary['tool_calls']['total']}")+ if init["tools_builtin"] != ["Read", "ToolSearch"] or init["mcp_tool_count"]:+ problems.append(f"declared tools {init['tools_builtin']} + {init['mcp_tool_count']} MCP")+ if init["skills"] != ["cua-driver"]:+ problems.append(f"skills {init['skills']}")+ if result["models"] != [f"{PROVIDER}/{MODEL}"]:+ problems.append(f"answered by {result['models']}")+ detail = f"{arm}: models={result['models']} skills={init['skills']} tokens={summary['tokens']}"+ return ("fail", "; ".join(problems) + " | " + detail) if problems else ("pass", detail)diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/pi_gate.ts b/libs/cua-driver-bench/automated-eval/macos_bench/pi_gate.tsnew file mode 100644index 0000000000..b331293dbe--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/pi_gate.ts@@ -0,0 +1,37 @@+// SPDX-License-Identifier: FSL-1.1-MIT+// Copyright (c) 2026 Cua AI, Inc.++// pi extension for the pi-* bench arms (pi_arms.py), giving pi the limits Claude Code's own flags give the cc-* arms.+// Read: the Claude Code arms load the skill with `Skill`, and every `Read` is denied (permission mode dontAsk, only+// mcp__cua allowed), so `read` may open the skill's SKILL.md (CDB_PI_READ_ALLOW) and nothing else.+// Turns: like --max-turns, the CDB_PI_MAX_TURNS-th assistant turn may still run its tools; the next turn is aborted+// before its model request, and no tool call runs after the cap.+import { realpathSync } from "node:fs";+import { resolve } from "node:path";++function real(path: string): string {+ try {+ return realpathSync(path);+ } catch {+ return path;+ }+}++export default function (pi: any) {+ const allowed = process.env.CDB_PI_READ_ALLOW ? real(process.env.CDB_PI_READ_ALLOW) : undefined;+ const maxTurns = Number(process.env.CDB_PI_MAX_TURNS || "0");+ let turns = 0;+ pi.on("turn_start", async (_event: any, ctx: any) => {+ turns += 1;+ if (maxTurns > 0 && turns > maxTurns) ctx.abort();+ });+ pi.on("tool_call", async (event: any) => {+ if (maxTurns > 0 && turns > maxTurns) {+ return { block: true, reason: "The turn limit is reached.", terminate: true };+ }+ if (event.toolName !== "read") return undefined;+ const path = real(resolve(process.cwd(), String(event.input?.path ?? "")));+ if (allowed && path === allowed) return undefined;+ return { block: true, reason: "Permission to use read has been denied; only the skill's SKILL.md is readable." };+ });+}diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.py b/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.pyindex 958b31520a..0f608071cb 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.py@@ -55,6 +55,7 @@ import bench_core as core # noqa: E402import claude_arms as ca # noqa: E402import cdb_adapter # noqa: E402import codex_native as cxn # noqa: E402+import pi_arms as pia # noqa: E402import browser_use_cli as buimport claude_driver # noqa: E402import claude_events # noqa: E402@@ -64,6 +65,10 @@ import desktop_runtime as desktopfrom dataclasses import replaceSCHEMA = "cdb-bench-trial/1"+# The Claude Code arm whose driver build, MCP entry, skill and system prompt a Codex or pi arm uses+SOURCE_ARM = {**cxn.SOURCE_ARM, **pia.SOURCE_ARM}+# Arms that use no Claude seat: no quota gate, no Haiku probes+NO_CLAUDE_SEAT = cxn.ARMS + pia.ARMSWEB_TASKS = ("WEB-01", "WEB-02")WORK = ca.WORKDEFAULT_LEDGER = WORK / "ledger" / "spend.jsonl"@@ -487,7 +492,7 @@ def check_pins(pins: dict[str, Any], arm_names: list[str]) -> list[tuple[str, st"""Compare every pinned version and hash with what is installed now. Any difference fails."""observed = ca.observed_pins(include_codex=any(a in ca.CODEX_ARMS or a in cxn.ARMS for a in arm_names),- include_cua="cc-cua-driver" in arm_names,+ include_cua="cc-cua-driver" in {SOURCE_ARM.get(a, a) for a in arm_names},)if ("cc-codex-cu-1007-browser" in arm_names@@ -586,7 +591,7 @@ def capture_subject(work: Path):def real_capture_check(ctx: "Ctx", arm: str) -> tuple[str, str, str]:client = Nonetry:- source = cxn.SOURCE_ARM.get(arm, arm)+ source = SOURCE_ARM.get(arm, arm)path, _ = ctx.mcp[arm]if source in ca.CUA_ARMS:ensure_agent_daemon(ctx, source)@@ -726,7 +731,7 @@ def cu_helper_live_checks(ctx: "Ctx", arm: str) -> list[tuple[str, str, str]]:))try:- selected = next(a for a in ctx.arm_names if cxn.SOURCE_ARM.get(a, a) == arm)+ selected = next(a for a in ctx.arm_names if SOURCE_ARM.get(a, a) == arm)path, _server = ctx.mcp[selected]if selected in cxn.ARMS:entry = cxn.server_entry(path, selected)@@ -854,13 +859,13 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]run_index = entry["run_index"]seed = core.probe_seed(task.id, run_index)system_prompt = ca.system_prompt_for(- cxn.SOURCE_ARM.get(arm, arm), browser=bool(spec.get("browser"))+ SOURCE_ARM.get(arm, arm), browser=bool(spec.get("browser")))bench_v2 = bool(getattr(args, "bench_v2", False)) # Amendment 14sentinel_front = True if bench_v2 else bool(spec.get("sentinel_frontmost", True))condition = getattr(args, "condition", "foreground") if bench_v2 else None # Amendment 15# Per-build Cua Driver identity (Amendment 3); arm B rows keep the 0.34.0 values of the recorder build.- build_versions = (ctx.versions.get("builds") or {}).get(cxn.SOURCE_ARM.get(arm, arm)) or (+ build_versions = (ctx.versions.get("builds") or {}).get(SOURCE_ARM.get(arm, arm)) or (ctx.versions.get("builds") or {}).get("cc-cua-driver", {})trial_dir = ctx.run_dir / "trials" / entry["trial_id"] / f"a{attempt}"@@ -910,14 +915,16 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]"cua_driver_git_sha": build_versions.get("cua_driver_git_sha"),"cua_driver_daemon_version": build_versions.get("cua_daemon_version"),"skills_tree_sha256": build_versions.get("cua_skills_tree_sha256")- if arm in ca.CUA_ARMS+ if pia.SOURCE_ARM.get(arm, arm) in ca.CUA_ARMSelse None,"macos": ctx.versions.get("macos"),"os_version": ctx.versions.get("os_version"),"bench_repo_commit": ctx.versions.get("bench_repo_commit"),"bench_src_sha256": ctx.versions.get("bench_src_sha256"),"system_prompt_sha256": hashlib.sha256(system_prompt.encode()).hexdigest(),- "skill_in_prompt": bool(getattr(ca.CUA_BUILDS.get(arm), "skill_in_prompt", False)),+ "skill_in_prompt": bool(+ getattr(ca.CUA_BUILDS.get(pia.SOURCE_ARM.get(arm, arm)), "skill_in_prompt", False)+ ),"electron_named_bundles": spec.get("kind") == "cdb"and os.environ.get("CDB_ELECTRON_SHARED") != "1","quota_five_hour_before": (ctx.quota or {}).get("five_hour"),@@ -926,13 +933,16 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]"force_accessibility": ca.force_accessibility(arm),# Amendment 11: the helper arm's identity (Desktop version, helper and adapter hashes, offered actions)"claude_cu_helper": cu_helper_row_identity(ctx)- if cxn.SOURCE_ARM.get(arm, arm) in ca.CU_HELPER_ARMS+ if SOURCE_ARM.get(arm, arm) in ca.CU_HELPER_ARMSelse None,# Amendment 20: the native Codex arm's identity (Codex CLI build, model, effort)"codex_native": ctx.versions.get("codex_native") if arm in cxn.ARMS else None,+ "pi": ctx.versions.get("pi") if arm in pia.ARMS else None,}if arm in cxn.ARMS:row["model"], row["effort"] = cxn.MODEL, cxn.EFFORT+ if arm in pia.ARMS:+ row["model"], row["effort"] = f"{pia.PROVIDER}/{pia.MODEL}", pia.THINKINGsentinel = pilot.Sentinel(Path(args.build_dir or HERE / "linux") / "BenchSentinel.app"if (args.build_dir or desktop.detect_platform() == "linux") and not args.no_sentinel@@ -965,6 +975,7 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]side_door: set[str] = set()watcher: FrontWatcher | None = Noneguard: v2.GuiOnlyGuard | None = None+ pi_root: Path | None = Nonegui_only: dict[str, Any] | None = Nonetry:ctx.set_state(@@ -979,9 +990,9 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]if arm in ca.CUA_ARMS:ensure_agent_daemon(ctx, arm)elif (- cxn.SOURCE_ARM.get(arm) in ca.CUA_ARMS+ SOURCE_ARM.get(arm) in ca.CUA_ARMS): # A23: Cua Driver under Codex uses that build's daemon- ensure_agent_daemon(ctx, cxn.SOURCE_ARM[arm])+ ensure_agent_daemon(ctx, SOURCE_ARM[arm])if getattr(args, "park_pointer", False):row["pointer_parked"] = park_pointer(ctx)lab_app = Path(args.build_dir or HERE / "linux") / "BenchLab.app"@@ -1016,12 +1027,14 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]elif cdb is None and getattr(args, "check_occlusion", True):row["lab_occlusion"] = ensure_lab_unoccluded(ctx)row["lab_unoccluded"] = row["lab_occlusion"].get("unoccluded")- cwd = ca.prepare_cwd(arm)+ cwd = ca.prepare_cwd(pia.SOURCE_ARM.get(arm, arm)) # a pi arm gets its source arm's skillmcp_path, server = ctx.mcp[arm]native = arm in cxn.MCP_ARMSargv = (cxn.argv()if native+ else pia.argv(system_prompt, pia.skill_dir(cwd))+ if arm in pia.ARMSelse ca.claude_argv(mcp_config=mcp_path,server=server,@@ -1085,6 +1098,20 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]abort=v2.EitherEvent(ctx.abort, guard.stop) if guard else ctx.abort,server=cxn.SERVER_OF[arm],)+ elif arm in pia.ARMS:+ pi_root = pia.new_root()+ claude = pia.run_pi(+ argv,+ pia.render_agent_dir(pi_root, mcp_path, pia.skill_dir(cwd), max_turns),+ cwd,+ prompt,+ timeout_s,+ max_turns,+ trial_dir / "pi-events.tsv",+ stream_path,+ stderr_path,+ abort=v2.EitherEvent(ctx.abort, guard.stop) if guard else ctx.abort,+ )else:claude = claude_driver.run_claude(argv,@@ -1155,8 +1182,12 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]note = failure["message"]finally:try:- _collect_linux_evidence(artifacts, arm, bool(failure.get("kind")))+ _collect_linux_evidence(+ artifacts, pia.SOURCE_ARM.get(arm, arm), bool(failure.get("kind"))+ )finally:+ if pi_root is not None:+ pia.keep_config(pi_root, trial_dir / "pi-agent")codex_home = trial_dir / "codex-home" / ".codex"if arm in cxn.ARMS and codex_home.is_dir():try:@@ -1180,9 +1211,7 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]if cdb is not None:cdb.stop_apps()pilot.sweep_processes(str(trial_dir))- helper_arm = cxn.SOURCE_ARM.get(- arm, arm- ) # A23: the helper under Codex CLI uses the same build+ helper_arm = SOURCE_ARM.get(arm, arm) # A23: the helper under Codex CLI uses the same buildif helper_arm in ca.CU_HELPER_ARMS:# the adapter and its helper end with claude's process group; a survivor could post input laterrow["cu_helper_leftover_killed"] = pilot.sweep_processes(@@ -1203,6 +1232,8 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]None,"not priced (Codex subscription; tokens reported)",)+ elif entry["arm"] in pia.ARMS:+ cost_source = pia.COST_SOURCEelif cost is None:cost = estimate_cost(tokens)cost_source = "estimated_from_tokens (no result event)"@@ -1765,7 +1796,7 @@ def append_row(ctx: Ctx, row: dict[str, Any]) -> None:ctx.ledger.append(ctx.args.ledger_who,f"{'smoke ' if row.get('smoke') else ''}trial {row['trial_id']} a{row['attempt']}",- ctx.args.model,+ row.get("model") or ctx.args.model,cost,run_id=ctx.run_dir.name,status=row.get("status"),@@ -1869,10 +1900,10 @@ def execute_trial(ctx: Ctx, entry: dict[str, Any]) -> dict[str, Any]:return rowif (kind in ("mcp_start", "tool_setup")- and cxn.SOURCE_ARM.get(entry["arm"], entry["arm"]) in ca.CUA_ARMS+ and SOURCE_ARM.get(entry["arm"], entry["arm"]) in ca.CUA_ARMS):try:- ensure_agent_daemon(ctx, cxn.SOURCE_ARM.get(entry["arm"], entry["arm"]))+ ensure_agent_daemon(ctx, SOURCE_ARM.get(entry["arm"], entry["arm"]))except Exception as error: # noqa: BLE001ctx.log(f"daemon restart failed: {error}")log_pause(ctx, f"retry_{kind}", wait_s, row.get("infra_message", ""), entry["trial_id"])@@ -1990,7 +2021,7 @@ def native_codex_checks(ctx: Ctx, arm: str, with_models: bool) -> list[tuple[strchecks: list[tuple[str, str, str]] = []try:env = cxn.render_home(- home, mcp_path, cxn.MODEL, cxn.EFFORT, ca.system_prompt_for(cxn.SOURCE_ARM[arm]), arm+ home, mcp_path, cxn.MODEL, cxn.EFFORT, ca.system_prompt_for(SOURCE_ARM[arm]), arm)if chatgpt:checks.extend(cxn.chatgpt_checks(cxn.CODEX_CLI, env, cxn.MODEL))@@ -2151,6 +2182,17 @@ def v2_static_checks(ctx: Ctx) -> list[tuple[str, str, str]]:))for arm in ctx.arm_names:+ if arm in pia.ARMS:+ tools = [pia.claude_name(t) for t in pia.TOOLS if not t.startswith("mcp__")]+ bad = [t for t in tools if t in v2.FORBIDDEN_BUILTINS or t == "Task"]+ out.append(+ (+ f"v2 GUI-only surface {arm}",+ "fail" if bad else "pass",+ f"pi tools {list(pia.TOOLS)}" + (f"; forbidden {bad}" if bad else ""),+ )+ )+ continueargv = ca.claude_argv(mcp_config=Path("/dev/null"),server=(ctx.mcp.get(arm) or (None, "server"))[1],@@ -2189,11 +2231,15 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:if getattr(args, "bench_v2", False):for name, status, detail in v2_static_checks(ctx):add(name, status, detail)- add(- "claude binary",- "pass" if ca.CLAUDE_BIN.is_file() else "fail",- f"{ca.CLAUDE_BIN} {ctx.versions.get('claude')}",- )+ if any(a not in pia.ARMS for a in ctx.arm_names):+ add(+ "claude binary",+ "pass" if ca.CLAUDE_BIN.is_file() else "fail",+ f"{ca.CLAUDE_BIN} {ctx.versions.get('claude')}",+ )+ if any(a in pia.ARMS for a in ctx.arm_names):+ pi = ctx.versions.get("pi") or {}+ add("pi binary", "pass" if pi.get("pi_version") else "fail", f"{pia.PI_BIN} {pi}")add("terminal.app ancestor","skip"@@ -2224,12 +2270,15 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:for name, status, detail in check_pins(pins, ctx.arm_names):add(name, status, detail)offline = bool(getattr(args, "offline", False))- if any(ca.CUA_BUILDS[a].label == "main" for a in ctx.arm_names if a in ca.CUA_BUILDS):+ driver_arms = [pia.SOURCE_ARM.get(a, a) for a in ctx.arm_names]+ if any(ca.CUA_BUILDS[a].label == "main" for a in driver_arms if a in ca.CUA_BUILDS):main = pins.get("cua_main", {})spec = ca.CUA_BUILDS["cc-cua-driver-main"]got_bin = ca.sha256_file(spec.bin) if spec.bin.is_file() else Nonefor other in [- a for a in ctx.arm_names if a in ca.CUA_BUILDS and ca.CUA_BUILDS[a].label == "main"+ a+ for a in dict.fromkeys(driver_arms)+ if a in ca.CUA_BUILDS and ca.CUA_BUILDS[a].label == "main"]:ob = ca.CUA_BUILDS[other].bingot = ca.sha256_file(ob) if ob.is_file() else None@@ -2250,7 +2299,7 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:"pass" if got_skills and got_skills == main.get("skills_tree_sha256") else "fail",str(got_skills),)- for cua_arm in [a for a in ca.CUA_ARMS if a in ctx.arm_names]:+ for cua_arm in [a for a in ca.CUA_ARMS if a in driver_arms]:spec = ca.CUA_BUILDS[cua_arm]try:ensure_agent_daemon(ctx, cua_arm)@@ -2276,7 +2325,7 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:):add(name, status, detail)helper_sources = {- cxn.SOURCE_ARM.get(a, a) for a in ctx.arm_names+ SOURCE_ARM.get(a, a) for a in ctx.arm_names} # A23: also for cx-claude-cu-helperfor helper_arm in [a for a in ca.CU_HELPER_ARMS if a in helper_sources]:observed = ca.cu_helper_observed(ca.CU_HELPER_BUILDS[helper_arm])@@ -2426,7 +2475,7 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:"quota gate from the latest known reading","warn"if gate["action"] == "gated"- and all(a in cxn.ARMS for a in ctx.arm_names) # no Claude seat in use+ and all(a in NO_CLAUDE_SEAT for a in ctx.arm_names) # no Claude seat in useelse "fail"if gate["action"] == "gated"else ("warn" if gate["action"] == "unknown" else "pass"),@@ -2434,7 +2483,21 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:)if (WORK / "HOLD").exists():add("HOLD file", "fail", (WORK / "HOLD").read_text("utf-8").strip()[:200])- claude_arms = [a for a in ctx.arm_names if a not in cxn.ARMS]+ for arm in [a for a in ctx.arm_names if a in pia.ARMS] if with_models and not offline else []:+ try:+ source = pia.SOURCE_ARM[arm]+ ensure_agent_daemon(ctx, source)+ status, detail = pia.init_check(+ arm,+ ctx.mcp[arm][0],+ ca.prepare_cwd(source),+ ca.system_prompt_for(source),+ ctx.run_dir / f"preflight-{arm}",+ )+ add(f"init event check {arm}", status, detail)+ except Exception as error: # noqa: BLE001+ add(f"init event check {arm}", "fail", f"{type(error).__name__}: {error}")+ claude_arms = [a for a in ctx.arm_names if a not in NO_CLAUDE_SEAT]if with_models and not offline and claude_arms: # a native-Codex-only run uses no Claude seattry:ok, detail = quota_probe(ctx)@@ -2661,11 +2724,13 @@ def make_ctx(args: argparse.Namespace) -> Ctx:unsupported = []if desktop.detect_platform() == "linux":unsupported = [- f"unsupported-platform: {a} requires macOS" for a in arm_names if a not in ca.CUA_ARMS+ f"unsupported-platform: {a} requires macOS"+ for a in arm_names+ if a not in ca.CUA_ARMS + pia.ARMS]for reason in unsupported:print(reason, file=sys.stderr)- arm_names = [a for a in arm_names if a in ca.CUA_ARMS]+ arm_names = [a for a in arm_names if a in ca.CUA_ARMS + pia.ARMS]if not arm_names:raise SystemExit("no supported arms remain: " + "; ".join(unsupported))if args.pins:@@ -2725,7 +2790,7 @@ def setup_arms(ctx: Ctx) -> None:)observed = ca.observed_pins(include_codex=any(a in ca.CODEX_ARMS or a in cxn.ARMS for a in ctx.arm_names),- include_cua="cc-cua-driver" in ctx.arm_names,+ include_cua="cc-cua-driver" in {SOURCE_ARM.get(a, a) for a in ctx.arm_names},)ctx.versions.update(pins_expected=pins,@@ -2753,7 +2818,7 @@ def setup_arms(ctx: Ctx) -> None:for arm, build in ca.CUA_BUILDS.items()if arm in ctx.arm_namesor arm == "cc-cua-driver"- or arm in {cxn.SOURCE_ARM.get(a) for a in ctx.arm_names}+ or arm in {SOURCE_ARM.get(a) for a in ctx.arm_names}},pins=pins,system_prompt_sha256=hashlib.sha256(ca.SYSTEM_PROMPT.encode()).hexdigest(),@@ -2769,13 +2834,18 @@ def setup_arms(ctx: Ctx) -> None:): # A20/A23: the source Claude arm's MCP entry, copied unmodified, driven by Codex CLItry:path, _server = ca.mcp_config_for(- cxn.SOURCE_ARM[arm], ctx.run_dir, ctx.args.codex_cu_mcp+ SOURCE_ARM[arm], ctx.run_dir, ctx.args.codex_cu_mcp)cxn.server_entry(path, arm)ctx.mcp[arm] = (path, cxn.SERVER_OF[arm])except (FileNotFoundError, ValueError) as error:raise SystemExit(f"arm {arm} unavailable: {error}")ctx.versions["codex_native"] = cxn.observed()+ elif (+ arm in pia.ARMS+ ): # the source arm's MCP entry; pi_arms.render_agent_dir copies it per trial+ ctx.mcp[arm] = ca.mcp_config_for(SOURCE_ARM[arm], ctx.run_dir)+ ctx.versions["pi"] = pia.observed()if ctx.args.build_dir or desktop.detect_platform() == "linux":apps: dict[str, str] = {}for name in BENCH_APPS:@@ -2862,12 +2932,12 @@ def write_manifest(ctx: Ctx, first_schedule: list[dict[str, Any]]) -> None:"tool_search": args.tool_search,"system_prompt": ca.SYSTEM_PROMPT,"system_prompt_sha256_by_arm": {- a: hashlib.sha256(ca.system_prompt_for(cxn.SOURCE_ARM.get(a, a)).encode()).hexdigest()+ a: hashlib.sha256(ca.system_prompt_for(SOURCE_ARM.get(a, a)).encode()).hexdigest()for a in ctx.arm_names},"system_prompt_sha256_by_arm_browser": {a: hashlib.sha256(- ca.system_prompt_for(cxn.SOURCE_ARM.get(a, a), browser=True).encode()+ ca.system_prompt_for(SOURCE_ARM.get(a, a), browser=True).encode()).hexdigest()for a in ctx.arm_names},@@ -2877,7 +2947,9 @@ def write_manifest(ctx: Ctx, first_schedule: list[dict[str, Any]]) -> None:"versions": ctx.versions,"recorder": getattr(ctx.recorder, "name", "none"),"first_block_schedule": first_schedule,- "arm_descriptions": {a: ca.ARM_DESCRIPTIONS.get(a, a) for a in ctx.arm_names},+ "arm_descriptions": {+ a: ca.ARM_DESCRIPTIONS.get(a) or pia.DESCRIPTIONS.get(a, a) for a in ctx.arm_names+ },"bench_version": v2.BENCH_VERSION if getattr(args, "bench_v2", False) else 1,"bench_v2": {"categories": v2.CATEGORIES,@@ -2900,7 +2972,11 @@ def write_manifest(ctx: Ctx, first_schedule: list[dict[str, Any]]) -> None:"price_fallback_usd_per_mtok_ASSUMED": PRICE_FALLBACK,"example_argv_task": ctx.task_ids[0],"example_argv": {- a: ca.claude_argv(+ a: pia.argv(+ ca.system_prompt_for(SOURCE_ARM[a]), Path(ca.CWD_ROOT) / ".claude/skills/cua-driver"+ )+ if a in pia.ARMS+ else ca.claude_argv(mcp_config=ctx.mcp[a][0],server=ctx.mcp[a][1],model=args.model,@@ -2909,7 +2985,7 @@ def write_manifest(ctx: Ctx, first_schedule: list[dict[str, Any]]) -> None:tool_search=args.tool_search == "default",effort=args.effort,system_prompt=ca.system_prompt_for(- cxn.SOURCE_ARM.get(a, a),+ SOURCE_ARM.get(a, a),browser=bool(task_spec(ctx.tasks[ctx.task_ids[0]]).get("browser")),),)@@ -3031,7 +3107,7 @@ def cmd_run(args: argparse.Namespace) -> int:failed = [c for c in checks if c[1] == "fail"]known = ctx.quota is not None and ctx.quota.get("seven_day") is not Nonenative_only = all(- a in cxn.ARMS for a in ctx.arm_names+ a in NO_CLAUDE_SEAT for a in ctx.arm_names) # Amendment 20: no Claude seat in useif (not known@@ -3102,7 +3178,7 @@ def cmd_gate_check(args: argparse.Namespace) -> int:if decision["action"] == "gated":arm_names = [args.only_arm] if args.only_arm else list(args.arms)if arm_names and all(- a in cxn.ARMS for a in arm_names+ a in NO_CLAUDE_SEAT for a in arm_names): # Amendment 20: no Claude seat in useprint("quota gate not applied: only native Codex arms, which use no Claude seat")return 0@@ -3162,7 +3238,7 @@ def cmd_dry_run(args: argparse.Namespace) -> int:tool_search=args.tool_search == "default",effort=args.effort,system_prompt=ca.system_prompt_for(- cxn.SOURCE_ARM.get(arm, arm),+ SOURCE_ARM.get(arm, arm),browser=bool(task_spec(ctx.tasks[task_id]).get("browser")),),)diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_pi_arms.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_pi_arms.pynew file mode 100644index 0000000000..3153e5ff28--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_pi_arms.py@@ -0,0 +1,284 @@+# SPDX-License-Identifier: FSL-1.1-MIT+# Copyright (c) 2026 Cua AI, Inc.++"""The pi-hosted Cua Driver arms: the tool surface they offer and how pi's events reach the guard and the row."""++from __future__ import annotations++import json+import os+import stat+import sys+import tempfile+import textwrap+import unittest+from pathlib import Path+from unittest import mock++HERE = Path(__file__).resolve().parent.parent+sys.path.insert(0, str(HERE))++import arms # noqa: E402+import bench_v2 as v2 # noqa: E402+import claude_arms as ca # noqa: E402+import claude_events # noqa: E402+import pi_arms as pia # noqa: E402+++def assistant(content, stop="toolUse", usage=None):+ return {+ "type": "message_end",+ "message": {+ "role": "assistant",+ "content": content,+ "provider": "openai-codex",+ "model": "gpt-6.1-terra",+ "stopReason": stop,+ "usage": usage or {"input": 10, "output": 2, "cacheRead": 5, "cost": {"total": 0.5}},+ },+ }+++def call(call_id, name, arguments=None):+ return {"type": "toolCall", "id": call_id, "name": name, "arguments": arguments or {}}+++SYSTEM = {+ "type": "message_end",+ "message": {+ "role": "system",+ "toolsAdded": [{"name": "read"}, {"name": "tool_search"}],+ "sections": {"skills": "<skill>\n<name>cua-driver</name>\n</skill>", "cwd": "x"},+ },+}+++class PiArmsTest(unittest.TestCase):+ def test_each_arm_mirrors_one_driver_arm(self):+ self.assertEqual(set(pia.ARMS) & set(arms.ALL_ARMS), set(pia.ARMS))+ self.assertEqual(set(pia.SOURCE_ARM.values()), set(ca.CUA_ARMS))++ def test_argv_offers_read_tool_search_and_the_cua_server_only(self):+ argv = pia.argv("SYSTEM PROMPT", Path("/w/.claude/skills/cua-driver"), Path("/bin/pi"))+ self.assertEqual(argv[argv.index("--tools") + 1], "read,tool_search,mcp__cua__*")+ self.assertEqual(argv[argv.index("--system-prompt") + 1], "SYSTEM PROMPT")+ self.assertEqual(argv[argv.index("--skill") + 1], "/w/.claude/skills/cua-driver")+ for flag in ("--no-extensions", "--no-skills", "--no-context-files", "--no-session"):+ self.assertIn(flag, argv)+ loaded = [argv[i + 1] for i, a in enumerate(argv) if a == "-e"]+ self.assertEqual(loaded, ["builtin:mcp", "builtin:tool-search", str(pia.GATE)])+ surface = [pia.claude_name(t) for t in pia.TOOLS if not t.startswith("mcp__")]+ self.assertFalse(set(surface) & set(v2.FORBIDDEN_BUILTINS))++ def test_tools_outside_the_surface_are_forbidden_builtins(self):+ for name in ("bash", "edit", "write", "codemode", "grep", "mcp__other__run"):+ hard, _ = v2.scan_tool_use(pia.claude_name(name), {})+ self.assertTrue(hard, name)+ self.assertEqual(v2.scan_tool_use(pia.claude_name("read"), {"path": "x"}), ([], []))++ def test_agent_dir_copies_the_source_mcp_entry_deferred_and_links_auth(self):+ with tempfile.TemporaryDirectory() as tmp:+ root = Path(tmp)+ auth = root / "auth.json"+ auth.write_text("{}")+ source = root / "mcp.json"+ entry = {"type": "stdio", "command": "/d/cua-driver", "args": ["--socket", "/s", "mcp"]}+ source.write_text(json.dumps({"mcpServers": {"cua": entry}}))+ pi_root = root / "pi-root"+ with mock.patch.dict(os.environ, {"CDB_PI_AUTH_FILE": str(auth)}):+ env = pia.render_agent_dir(pi_root, source, root / "skill", 45)+ agent = Path(env["PI_CODING_AGENT_DIR"])+ config = json.loads((agent / "mcp.json").read_text())+ self.assertEqual(config["mcpServers"]["cua"], {**entry, "exposure": "deferred"})+ self.assertIs(config["autoEnableCodemode"], False)+ self.assertTrue(stat.S_ISLNK((agent / "auth.json").lstat().st_mode))+ self.assertNotEqual(env["HOME"], os.environ.get("HOME"))+ self.assertEqual(env["CDB_PI_READ_ALLOW"], str(root / "skill" / "SKILL.md"))+ self.assertEqual(env["CDB_PI_MAX_TURNS"], "45")+ pia.keep_config(pi_root, root / "trial" / "pi-agent")+ self.assertFalse(pi_root.exists())+ self.assertEqual(+ sorted(p.name for p in (root / "trial" / "pi-agent").iterdir()),+ ["mcp.json", "settings.json"],+ )++ def test_events_become_the_claude_shaped_stream(self):+ tr = pia.Translator(max_turns=45)+ events = tr.feed(SYSTEM)+ events += tr.feed(assistant([call("c1", "tool_search", {"query": "click"})]))+ events += tr.feed(+ {+ "type": "tool_execution_end",+ "toolCallId": "c1",+ "toolName": "tool_search",+ "result": {"content": [{"type": "text", "text": "Loaded 2 tools"}]},+ }+ )+ tr.feed(+ {+ "type": "message_end",+ "message": {"role": "system", "toolsAdded": [{"name": "mcp__cua__click"}]},+ }+ )+ events += tr.feed(assistant([call("c2", "mcp__cua__click", {"x": 1})]))+ events += tr.feed(+ {+ "type": "tool_execution_end",+ "toolCallId": "c2",+ "toolName": "mcp__cua__click",+ "isError": True,+ "result": {"content": [{"type": "text", "text": "no element"}]},+ }+ )+ events += tr.feed(assistant([{"type": "text", "text": "DONE"}], stop="stop"))+ events += tr.feed({"type": "agent_settled", "aborted": False})+ events += tr.finish(capped=False, wall_ms=1000)+ summary = claude_events.summarize(events)+ self.assertEqual(summary["init"]["tools_builtin"], ["Read", "ToolSearch"])+ self.assertEqual(summary["init"]["mcp_tools"], ["mcp__cua__click"])+ self.assertEqual(summary["init"]["skills"], ["cua-driver"])+ self.assertEqual(summary["init"]["mcp_servers"], [{"name": "cua", "status": "connected"}])+ self.assertEqual(summary["tool_calls"]["by_name"], {"ToolSearch": 1, "mcp__cua__click": 1})+ self.assertEqual(summary["tool_calls"]["failed_by_name"], {"mcp__cua__click": 1})+ self.assertEqual(summary["turns"], 3)+ self.assertEqual(summary["final_text"], "DONE")+ self.assertEqual(summary["result"]["subtype"], "success")+ self.assertEqual(+ summary["tokens"], {"input": 30, "output": 6, "cache_read": 15, "cache_write": 0}+ )+ self.assertAlmostEqual(summary["total_cost_usd"], 1.5)+ self.assertEqual(claude_events.classify_failure(summary, "", 0)["kind"], None)++ def test_turn_cap_is_reached_when_a_turn_past_the_cap_starts(self):+ tr = pia.Translator(max_turns=2)+ for n in (1, 2):+ tr.feed({"type": "turn_start"})+ tr.feed(assistant([call(f"c{n}", "mcp__cua__click")]))+ self.assertFalse(tr.capped.is_set())+ tr.feed({"type": "turn_start"})+ self.assertTrue(tr.capped.is_set())+ aborted = assistant([], stop="error")+ aborted["message"]["errorMessage"] = "This operation was aborted"+ self.assertEqual(tr.feed(aborted), [])+ result = tr.finish(capped=True, wall_ms=1)[-1]+ self.assertEqual(+ (result["subtype"], result["is_error"], result["num_turns"]),+ ("error_max_turns", True, 2),+ )++ def test_a_final_answer_on_the_last_allowed_turn_is_not_capped(self):+ tr = pia.Translator(max_turns=1)+ tr.feed({"type": "turn_start"})+ tr.feed(assistant([{"type": "text", "text": "DONE"}], stop="stop"))+ tr.feed({"type": "agent_settled"})+ self.assertFalse(tr.capped.is_set())+ self.assertEqual(tr.finish(capped=False, wall_ms=1)[-1]["subtype"], "success")++ def test_provider_error_is_classified_as_infrastructure(self):+ tr = pia.Translator(max_turns=45)+ message = assistant([], stop="error")+ message["message"]["errorMessage"] = "429 Too Many Requests: usage limit reached"+ events = tr.feed(message) + tr.feed({"type": "agent_settled"})+ summary = claude_events.summarize(events + tr.finish(capped=False, wall_ms=1, returncode=1))+ self.assertEqual(claude_events.classify_failure(summary, "", 0)["kind"], "rate_limit")++ def test_exit_without_settling_is_a_harness_failure_not_a_result(self):+ tr = pia.Translator(max_turns=45)+ events = tr.feed(SYSTEM) + tr.feed(assistant([call("c1", "mcp__cua__click")]))+ self.assertEqual(tr.finish(capped=False, wall_ms=1, returncode=1), [])+ summary = claude_events.summarize(events)+ self.assertEqual(+ claude_events.classify_failure(summary, "TypeError", 1)["kind"], "harness_crash"+ )++ def test_settled_with_nonzero_exit_is_a_harness_failure(self):+ tr = pia.Translator(max_turns=45)+ events = tr.feed(SYSTEM) + tr.feed(+ assistant([{"type": "text", "text": "DONE"}], stop="stop")+ )+ tr.feed({"type": "agent_settled"})+ self.assertEqual(tr.finish(capped=False, wall_ms=1, returncode=1), [])+ summary = claude_events.summarize(events)+ self.assertEqual(claude_events.classify_failure(summary, "", 1)["kind"], "harness_crash")++ def test_cap_is_read_after_pi_exits_right_after_the_over_cap_turn(self):+ lines = [+ assistant([call("c1", "mcp__cua__click")]),+ assistant([call("c2", "mcp__cua__click")]),+ ]+ script = textwrap.dedent(+ """\+ import json, sys+ sys.stdin.read()+ for event in json.load(open(sys.argv[1])):+ print(json.dumps({"type": "turn_start"}))+ print(json.dumps(event))+ print(json.dumps({"type": "turn_start"}), flush=True)+ """+ )+ with tempfile.TemporaryDirectory() as tmp:+ root = Path(tmp)+ (root / "fake.py").write_text(script)+ (root / "events.json").write_text(json.dumps(lines))+ result = pia.run_pi(+ [sys.executable, str(root / "fake.py"), str(root / "events.json")],+ {"PATH": os.environ["PATH"]},+ root,+ "brief",+ 20,+ 2,+ root / "raw.tsv",+ root / "stream.tsv",+ root / "stderr",+ )+ events = claude_events.read_events(root / "stream.tsv")+ self.assertTrue(result["capped"])+ self.assertEqual(events[-1]["subtype"], "error_max_turns")++ def test_no_turn_leaves_no_result_so_stderr_classifies_the_row(self):+ tr = pia.Translator(max_turns=45)+ self.assertEqual(tr.finish(capped=False, wall_ms=1), [])+ summary = claude_events.summarize([])+ failure = claude_events.classify_failure(summary, "Error: not logged in", 1)+ self.assertEqual(failure["kind"], "auth")++ def test_run_pi_streams_events_and_stops_at_the_turn_cap(self):+ lines = [SYSTEM] + [assistant([call(f"c{n}", "mcp__cua__click")]) for n in range(5)]+ script = textwrap.dedent(+ """\+ import json, sys, time+ sys.stdin.read()+ for event in json.load(open(sys.argv[1])):+ if event["message"]["role"] == "assistant":+ print(json.dumps({"type": "turn_start"}))+ print(json.dumps(event), flush=True)+ time.sleep(0.05)+ time.sleep(30)+ """+ )+ with tempfile.TemporaryDirectory() as tmp:+ root = Path(tmp)+ (root / "fake.py").write_text(script)+ (root / "events.json").write_text(json.dumps(lines))+ result = pia.run_pi(+ [sys.executable, str(root / "fake.py"), str(root / "events.json")],+ {"PATH": os.environ["PATH"]},+ root,+ "brief",+ 20,+ 3,+ root / "raw.tsv",+ root / "stream.tsv",+ root / "stderr",+ )+ events = claude_events.read_events(root / "stream.tsv")+ self.assertTrue(result["capped"])+ self.assertFalse(result["timed_out"])+ self.assertLess(result["wall_s"], 10)+ self.assertEqual(result["turns"], 3)+ self.assertEqual(events[-1]["subtype"], "error_max_turns")+ self.assertEqual([n for n, _ in v2.tool_uses(events[0])], ["mcp__cua__click"])+++if __name__ == "__main__":+ unittest.main()diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tools/check_run_evidence.py b/libs/cua-driver-bench/automated-eval/macos_bench/tools/check_run_evidence.pyindex 92cd3c27c6..d35cc108a9 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/tools/check_run_evidence.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tools/check_run_evidence.py@@ -21,6 +21,10 @@ CUA_ARMS = ("cc-cua-driver-main","cc-cua-driver-main-skill","cc-cua-driver-script",+ "pi-cua-driver",+ "pi-cua-driver-main",+ "pi-cua-driver-main-skill",+ "pi-cua-driver-script",)