Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cua-driver-bench): pi-hosted arms for the four Cua Driver setups (#6642)

injaneitybb42e3ec01e0
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/RUNNER.md b/libs/cua-driver-bench/automated-eval/macos_bench/RUNNER.md
index 9310d8f864..b8b2ac970d 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/RUNNER.md
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/RUNNER.md
@@ -26,6 +26,7 @@ observed-pin comparison and doctor/CDP/page-info/deny probes remain prerequisite
| `cc-browser-use` | Browser Use harness | Browser Use macOS harness 0.2.1 behind `tools/browser_use_harness/bu_mcp.py` as MCP server `browseruse` (`macos_harness`, `view_screenshot`), the vendor skill as server instructions |
| `cx-cua-driver`, `cx-codex-cu`, `cx-claude-cu-helper` | Cua Driver, Codex Computer-Use (Sky) and Claude Computer-Use (Desktop helper, with the label above), driven by Codex CLI (Amendment 23) | Not Claude Code: the bundled Codex CLI as `codex app-server` (`codex_native.py`) with GPT-6.1 Sol (`gpt-6.1-sol`, default effort low), given the MCP entry and system prompt of `cc-cua-driver-script`, `cc-codex-cu` or `cc-claude-cu-helper` |
| `cc-cua-driver`, `cc-cua-driver-main`, `cc-cua-driver-main-skill` | Internal baselines | Cua Driver 0.34.0 release; main with `run_script` off; main with the skill in the system prompt |
+| `pi-cua-driver`, `pi-cua-driver-main`, `pi-cua-driver-main-skill`, `pi-cua-driver-script` | The four Cua Driver setups above, driven by pi | Not Claude Code: `pi --mode json` (`pi_arms.py`) on `openai-codex` / `gpt-6.1-terra` by default (`CDB_PI_PROVIDER`, `CDB_PI_MODEL`, `CDB_PI_THINKING`), given the driver daemon, MCP entry, skill and system prompt of the matching `cc-*` arm. Tools: `read` (limited to the skill's `SKILL.md`, and the turn cap enforced, by `pi_gate.ts`), `tool_search` and the deferred `cua` server. Linux and macOS. |
Arms `codex-native-cu` and `codex-cua-driver` (`codex exec --json`, from
`run_pilot.py`) exist behind `--allow-codex-arms` and are not part of Bench v2.
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/arms.py b/libs/cua-driver-bench/automated-eval/macos_bench/arms.py
index 3a947c35ca..832e5431c4 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/arms.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/arms.py
@@ -53,7 +53,14 @@ NATIVE_CODEX_ARMS = (
"cx-claude-cu-helper",
"cx-browser-use-cli",
) # the last two: Amendment 23
-ALL_ARMS = CLAUDE_ARMS + CODEX_FALLBACK_ARMS + NATIVE_CODEX_ARMS
+# The Cua Driver arms with pi as the agent host (pi_arms.py)
+PI_ARMS = (
+ "pi-cua-driver",
+ "pi-cua-driver-main",
+ "pi-cua-driver-main-skill",
+ "pi-cua-driver-script",
+)
+ALL_ARMS = CLAUDE_ARMS + CODEX_FALLBACK_ARMS + NATIVE_CODEX_ARMS + PI_ARMS
CODEX_APP_RESOURCES = Path("/Applications/ChatGPT.app/Contents/Resources")
CODEX_CU_DIR = Path.home() / ".codex" / "computer-use"
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/pi_arms.py b/libs/cua-driver-bench/automated-eval/macos_bench/pi_arms.py
new file mode 100644
index 0000000000..7f06d7c5d4
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/pi_arms.py
@@ -0,0 +1,495 @@
+# SPDX-License-Identifier: FSL-1.1-MIT
+# Copyright (c) 2026 Cua AI, Inc.
+
+"""The Cua Driver setups with pi as the agent host instead of Claude Code.
+
+Each ``pi-*`` arm mirrors one ``cc-*`` Cua Driver arm (``SOURCE_ARM``): the same driver build and daemon, the same MCP
+server entry (that arm's ``--mcp-config``, copied), skill directory, system prompt, brief, turn cap and wall-time
+limit. Only the host and the model differ: ``pi --mode json`` on ``openai-codex`` / ``gpt-6.1-terra`` by default
+(``CDB_PI_PROVIDER``, ``CDB_PI_MODEL``, ``CDB_PI_THINKING``).
+
+The tool surface mirrors Claude Code's ``Skill, Read, ToolSearch`` plus ``mcp__cua``: pi's ``read`` and ``tool_search``
+built-ins, the ``cua`` server with ``deferred`` exposure (loaded through ``tool_search``, as Claude Code defers it
+behind ToolSearch) and the build's skill through ``--skill``. Claude Code denies every ``Read`` in those arms
+(``dontAsk`` with only ``mcp__cua`` allowed) and loads the skill through ``Skill``, so ``pi_gate.ts`` blocks
+every ``read`` except the skill's ``SKILL.md``. Every other built-in, extension, context file, prompt
+template and codemode is off. ``Translator`` turns pi's JSON events into the Claude-shaped stream that the GUI-only
+guard and ``claude_events.summarize`` read; a pi tool outside that surface maps to a forbidden built-in, so it ends
+the trial as a violation.
+
+pi has no turn cap, so ``pi_gate.ts`` gives it Claude Code's ``--max-turns``: the ``max_turns``-th turn may run its
+tools, the next one is aborted before its model request, and the host records ``error_max_turns``. A pi that exits
+without settling (a crash, not a cap, timeout or abort) leaves no result event, so the row is a harness failure. Cost is pi's own figure for the reported tokens at the model's list price;
+``openai-codex`` is a ChatGPT subscription, so it is not a billed amount.
+
+Nothing here reads credentials: the trial's pi agent directory, under the private ``/tmp`` of the session, gets a
+symlink to the operator's ``auth.json`` (``CDB_PI_AUTH_FILE``, default ``~/.pi/agent/auth.json``); only its
+configuration is copied into the run directory.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import re
+import shutil
+import signal
+import subprocess
+import tempfile
+import threading
+import time
+from pathlib import Path
+from typing import Any
+
+from arms import PI_ARMS as ARMS
+from claude_events import _text_of
+from desktop_runtime import record_or_reap, stop_group
+
+SOURCE_ARM = {arm: "cc-" + arm.removeprefix("pi-") for arm in ARMS}
+PROVIDER = os.environ.get("CDB_PI_PROVIDER") or "openai-codex"
+MODEL = os.environ.get("CDB_PI_MODEL") or "gpt-6.1-terra"
+THINKING = (
+ os.environ.get("CDB_PI_THINKING") or "medium"
+) # pi's own default level, passed explicitly
+PI_BIN = Path(os.environ.get("CDB_PI_BIN") or shutil.which("pi") or Path.home() / ".local/bin/pi")
+AGENT_DIR = Path(os.environ.get("CDB_PI_AGENT_DIR") or Path.home() / ".pi" / "agent")
+TOOLS = ("read", "tool_search", "mcp__cua__*")
+GATE = Path(__file__).resolve().parent / "pi_gate.ts"
+# pi built-in -> the Claude Code built-in the guard knows; any other non-MCP tool is "Task" (forbidden in v2)
+AS_CLAUDE = {
+ "read": "Read",
+ "tool_search": "ToolSearch",
+ "bash": "Bash",
+ "powershell": "Bash",
+ "edit": "Edit",
+ "write": "Write",
+}
+COST_SOURCE = "pi usage cost at list price (openai-codex subscription, not billed)"
+DESCRIPTIONS = {arm: f"pi + {PROVIDER}/{MODEL}, the {SOURCE_ARM[arm]} setup" for arm in ARMS}
+
+
+def claude_name(name: str) -> str:
+ return name if name.startswith("mcp__cua__") else AS_CLAUDE.get(name, "Task")
+
+
+def skill_dir(cwd: Path) -> Path:
+ return (
+ cwd / ".claude" / "skills" / "cua-driver"
+ ) # written by claude_arms.prepare_cwd(source arm)
+
+
+def render_agent_dir(root: Path, mcp_json: Path, skills: Path, max_turns: int) -> dict[str, str]:
+ """A fresh pi agent directory and HOME under ``root`` for one trial; returns pi's closed environment."""
+ agent_dir = root / "agent"
+ agent_dir.mkdir(parents=True, exist_ok=True)
+ entry = json.loads(mcp_json.read_text("utf-8"))["mcpServers"]["cua"]
+ config = {"mcpServers": {"cua": {**entry, "exposure": "deferred"}}, "autoEnableCodemode": False}
+ (agent_dir / "mcp.json").write_text(json.dumps(config, indent=2) + "\n", "utf-8")
+ settings = {"defaultProvider": PROVIDER, "defaultModel": MODEL}
+ (agent_dir / "settings.json").write_text(json.dumps(settings) + "\n", "utf-8")
+ store = AGENT_DIR / "models-store.json" # the model catalog: context window and list prices
+ if store.is_file():
+ shutil.copyfile(store, agent_dir / "models-store.json")
+ auth = (
+ Path(os.environ.get("CDB_PI_AUTH_FILE") or AGENT_DIR / "auth.json").expanduser().absolute()
+ )
+ if not auth.is_file():
+ raise FileNotFoundError("pi auth file is missing")
+ (agent_dir / "auth.json").symlink_to(auth)
+ home = root / "home"
+ (home / "tmp").mkdir(parents=True, exist_ok=True)
+ return {
+ "HOME": str(home),
+ "PATH": f"{PI_BIN.parent}:/usr/local/bin:/usr/bin:/bin",
+ "TMPDIR": str(home / "tmp"),
+ "LANG": "en_US.UTF-8",
+ "PI_CODING_AGENT_DIR": str(agent_dir),
+ "PI_OFFLINE": "1",
+ "PI_SKIP_VERSION_CHECK": "1",
+ "PI_TELEMETRY": "0",
+ "CDB_PI_READ_ALLOW": str(skills / "SKILL.md"), # pi_gate.ts
+ "CDB_PI_MAX_TURNS": str(max_turns),
+ }
+
+
+def new_root() -> Path:
+ """The trial's pi state, outside the run directory (the session's private /tmp on Linux)."""
+ return Path(tempfile.mkdtemp(prefix="cdb-pi-"))
+
+
+def keep_config(root: Path, dest: Path) -> None:
+ """Copy the trial's pi configuration (never the credential) to ``dest``, then delete ``root``."""
+ dest.mkdir(parents=True, exist_ok=True)
+ for name in ("mcp.json", "settings.json"):
+ if (root / "agent" / name).is_file():
+ shutil.copyfile(root / "agent" / name, dest / name)
+ shutil.rmtree(root, ignore_errors=True)
+
+
+def argv(system_prompt: str, skills: Path | None, pi_bin: Path | None = None) -> list[str]:
+ out = [
+ str(pi_bin or PI_BIN),
+ "--mode",
+ "json",
+ "--no-session",
+ "--provider",
+ PROVIDER,
+ "--model",
+ MODEL,
+ "--thinking",
+ THINKING,
+ "--system-prompt",
+ system_prompt,
+ "--no-extensions",
+ "-e",
+ "builtin:mcp",
+ "-e",
+ "builtin:tool-search",
+ "-e",
+ str(GATE),
+ "--no-skills",
+ "--no-context-files",
+ "--no-prompt-templates",
+ "--no-themes",
+ "--tools",
+ ",".join(TOOLS),
+ ]
+ return out + (["--skill", str(skills)] if skills else [])
+
+
+class Translator:
+ """pi JSON events -> Claude-shaped stream-json events (assistant tool_use / user tool_result / init / result)."""
+
+ def __init__(self, max_turns: int) -> None:
+ self.max_turns = max_turns
+ self.turns = 0
+ self.final_text = ""
+ self.error = ""
+ self.last_stop: str | None = None
+ self.started = 0
+ self.capped = (
+ threading.Event()
+ ) # a turn past the cap started; pi_gate.ts aborts it before its request
+ self.usage = {"input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0}
+ self.cost = 0.0
+ self.models: set[str] = set()
+ self.sections: dict[str, Any] = {}
+ self.declared: list[str] = []
+ self.mcp_loaded = False
+ self.empty_search = False
+ self.settled = False
+
+ def feed(self, event: dict[str, Any]) -> list[dict[str, Any]]:
+ if self.capped.is_set():
+ return [] # the aborted turn past the cap is not a turn of the trial
+ kind = event.get("type")
+ if kind == "turn_start":
+ self.started += 1
+ if self.started > self.max_turns:
+ self.capped.set()
+ return []
+ message = event.get("message") or {}
+ role = message.get("role")
+ if kind == "message_end" and role == "system":
+ added = [str(t.get("name")) for t in message.get("toolsAdded") or []]
+ self.declared += added
+ self.mcp_loaded |= any(t.startswith("mcp__") for t in added)
+ self.sections = self.sections or message.get("sections") or {}
+ elif kind == "message_end" and role == "assistant":
+ self.turns += 1
+ usage = message.get("usage") or {}
+ for key in self.usage:
+ self.usage[key] += int(usage.get(key) or 0)
+ self.cost += float((usage.get("cost") or {}).get("total") or 0.0)
+ self.models.add(f"{message.get('provider')}/{message.get('model')}")
+ content: list[dict[str, Any]] = []
+ for block in message.get("content") or []:
+ if block.get("type") == "text":
+ content.append({"type": "text", "text": str(block.get("text") or "")})
+ elif block.get("type") == "toolCall":
+ content.append(
+ {
+ "type": "tool_use",
+ "id": str(block.get("id")),
+ "name": claude_name(str(block.get("name"))),
+ "input": block.get("arguments") or {},
+ }
+ )
+ text = "".join(b["text"] for b in content if b["type"] == "text")
+ self.final_text = text if text.strip() else self.final_text
+ self.last_stop = message.get("stopReason")
+ if self.last_stop == "error":
+ self.error = str(message.get("errorMessage") or "pi provider error")
+ return [
+ {
+ "type": "assistant",
+ "message": {
+ "id": message.get("responseId") or f"pi-{self.turns}",
+ "content": content,
+ "usage": {
+ "input_tokens": int(usage.get("input") or 0),
+ "output_tokens": int(usage.get("output") or 0),
+ "cache_read_input_tokens": int(usage.get("cacheRead") or 0),
+ "cache_creation_input_tokens": int(usage.get("cacheWrite") or 0),
+ },
+ },
+ }
+ ]
+ elif kind == "tool_execution_end":
+ text = _text_of((event.get("result") or {}).get("content"))
+ self.empty_search |= (
+ event.get("toolName") == "tool_search" and "No matching tools" in text
+ )
+ failed = bool(event.get("isError"))
+ block = {
+ "type": "tool_result",
+ "tool_use_id": str(event.get("toolCallId")),
+ "is_error": failed,
+ "content": text[:300] if failed else "",
+ }
+ return [{"type": "user", "message": {"content": [block]}}]
+ elif kind == "agent_settled":
+ self.settled = True
+ elif kind == "auto_retry_start":
+ retry = {"error": event.get("errorMessage"), "attempt": event.get("attempt")}
+ return [
+ {
+ "type": "system",
+ "subtype": "api_retry",
+ **retry,
+ "retry_delay_ms": event.get("delayMs"),
+ }
+ ]
+ return []
+
+ def finish(
+ self, capped: bool, wall_ms: float, returncode: int | None = 0, stopped: bool = False
+ ) -> list[dict[str, Any]]:
+ """The init and result events, written when pi has stopped (``stopped``: the host ended it on purpose, at a
+ timeout or an abort). None when pi produced no turn, exited without settling, or exited nonzero without a
+ provider error, so the row is classified from stderr as a harness failure (retried, never scored) rather
+ than scored as an agent result. A provider error keeps its message, so a 429 or 5xx is classified as such."""
+ if not self.turns or not (capped or stopped or self.settled):
+ return []
+ is_error = self.last_stop == "error"
+ if not (capped or stopped or is_error) and returncode not in (0, None):
+ return []
+ status = next(
+ (c for c in (429, 401, 403, 500, 502, 503, 504, 529) if str(c) in self.error), None
+ )
+ # the server is "failed" only when tool_search found nothing and no cua tool ever loaded; unused is "ready"
+ mcp = "connected" if self.mcp_loaded else ("failed" if self.empty_search else "ready")
+ init = {
+ "type": "system",
+ "subtype": "init",
+ "model": MODEL,
+ "tools": [claude_name(t) for t in dict.fromkeys(self.declared)],
+ "mcp_servers": [{"name": "cua", "status": mcp}],
+ "skills": re.findall(r"<name>(.*?)</name>", str(self.sections.get("skills") or "")),
+ "pi_prompt_sections": sorted(self.sections),
+ }
+ result = {
+ "type": "result",
+ "subtype": "error_max_turns"
+ if capped
+ else ("error_during_execution" if is_error else "success"),
+ "is_error": is_error or capped,
+ "api_error_status": status,
+ "num_turns": self.turns,
+ "duration_ms": round(wall_ms),
+ "result": self.error[:2000] if is_error else self.final_text,
+ "usage": {
+ "input_tokens": self.usage["input"],
+ "output_tokens": self.usage["output"],
+ "cache_read_input_tokens": self.usage["cacheRead"],
+ "cache_creation_input_tokens": self.usage["cacheWrite"],
+ },
+ "total_cost_usd": round(self.cost, 6),
+ "pi_models": sorted(self.models),
+ }
+ return [init, result]
+
+
+def _strip(value: Any, limit: int = 4000) -> Any:
+ if isinstance(value, str):
+ return value if len(value) <= limit else f"{value[:120]}...<{len(value)} chars stripped>"
+ if isinstance(value, list):
+ return [_strip(v, limit) for v in value]
+ if isinstance(value, dict):
+ return {k: _strip(v, limit) for k, v in value.items()}
+ return value
+
+
+def run_pi(
+ cmd: list[str],
+ env: dict[str, str],
+ cwd: Path,
+ prompt: str,
+ timeout_s: float,
+ max_turns: int,
+ raw_path: Path,
+ stream_path: Path,
+ stderr_path: Path,
+ abort: Any = None,
+) -> dict[str, Any]:
+ """One trial: pi in its own process group with the prompt on stdin, stopped at exit, the wall-time limit, the
+ turn cap or ``abort``. pi's events (bulk stripped, streaming deltas dropped) go to ``raw_path``; the translated
+ events go to ``stream_path`` as they arrive (both ``<epoch_ms>\\t<json>``), so the GUI-only guard reads them live."""
+ started = time.monotonic()
+ translator = Translator(max_turns)
+ lock = threading.Lock()
+ with (
+ raw_path.open("w", encoding="utf-8") as raw,
+ stream_path.open("w", encoding="utf-8") as stream,
+ stderr_path.open("wb") as err,
+ ):
+ proc = subprocess.Popen(
+ cmd,
+ cwd=str(cwd),
+ env=env,
+ stdin=subprocess.PIPE,
+ stdout=subprocess.PIPE,
+ stderr=err,
+ start_new_session=True,
+ )
+ record_or_reap(proc, "pi")
+
+ def write(obj: dict[str, Any]) -> None:
+ stream.write(f"{time.time() * 1000:.1f}\t{json.dumps(obj)}\n")
+ stream.flush()
+
+ def pump() -> None:
+ assert proc.stdout is not None
+ for line in proc.stdout: # binary: records split on LF only (pi's JSONL framing)
+ try:
+ event = json.loads(line)
+ except ValueError:
+ continue
+ if not isinstance(event, dict):
+ continue
+ with lock:
+ if event.get("type") != "message_update":
+ raw.write(f"{time.time() * 1000:.1f}\t{json.dumps(_strip(event))}\n")
+ raw.flush()
+ for out in translator.feed(event):
+ write(out)
+
+ reader = threading.Thread(target=pump, daemon=True)
+ reader.start()
+ try:
+ assert proc.stdin is not None
+ proc.stdin.write(prompt.encode())
+ proc.stdin.close()
+ except OSError:
+ pass
+ timed_out = aborted = False
+ while proc.poll() is None and not translator.capped.is_set():
+ if time.monotonic() - started > timeout_s:
+ timed_out = True
+ break
+ if abort is not None and abort.is_set():
+ aborted = True
+ break
+ time.sleep(0.25)
+ stop_group(proc.pid)
+ reader.join(timeout=5)
+ # read after the reader has drained pi's output: pi may print the over-cap turn_start and exit at once
+ capped = translator.capped.is_set() and not timed_out and not aborted
+ try:
+ os.killpg(proc.pid, signal.SIGKILL) # the MCP server child must not outlive the trial
+ except OSError:
+ pass
+ try:
+ proc.wait(timeout=5)
+ except subprocess.TimeoutExpired:
+ pass
+ with lock:
+ wall_ms = (time.monotonic() - started) * 1000
+ for out in translator.finish(capped, wall_ms, proc.returncode, timed_out or aborted):
+ write(out)
+ return {
+ "returncode": proc.returncode,
+ "timed_out": timed_out,
+ "aborted": aborted,
+ "capped": capped,
+ "result_seen": bool(translator.turns),
+ "wall_s": round(time.monotonic() - started, 2),
+ "elicitations": [],
+ "turns": translator.turns,
+ "models": sorted(translator.models),
+ }
+
+
+def observed(pi_bin: Path | None = None) -> dict[str, Any]:
+ """The host's identity, recorded in the manifest and every row."""
+ pi_bin = pi_bin or PI_BIN
+ version = None
+ sha = None
+ if pi_bin.exists():
+ env = {
+ "PATH": f"{pi_bin.parent}:/usr/bin:/bin",
+ "PI_OFFLINE": "1",
+ "PI_SKIP_VERSION_CHECK": "1",
+ }
+ version = subprocess.run(
+ [str(pi_bin), "--version"], capture_output=True, text=True, timeout=30, env=env
+ ).stdout.strip()
+ sha = hashlib.sha256(Path(os.path.realpath(pi_bin)).read_bytes()).hexdigest()
+ store = AGENT_DIR / "models-store.json"
+ return {
+ "host": "pi --mode json",
+ "pi_bin": str(pi_bin),
+ "pi_version": version,
+ "pi_cli_sha256": sha,
+ "provider": PROVIDER,
+ "model": MODEL,
+ "thinking": THINKING,
+ "tools": list(TOOLS),
+ "mcp_exposure": "deferred",
+ "gate_sha256": hashlib.sha256(GATE.read_bytes()).hexdigest(),
+ "models_store_sha256": hashlib.sha256(store.read_bytes()).hexdigest()
+ if store.is_file()
+ else None,
+ }
+
+
+def init_check(
+ arm: str, mcp_json: Path, cwd: Path, system_prompt: str, work: Path
+) -> tuple[str, str]:
+ """One trivial turn through the real argv: the declared tools, the skill and the model that answered."""
+ import claude_events
+
+ shutil.rmtree(work, ignore_errors=True)
+ work.mkdir(parents=True)
+ stream = work / "stream.tsv"
+ root = new_root()
+ try:
+ env = render_agent_dir(root, mcp_json, skill_dir(cwd), 3)
+ result = run_pi(
+ argv(system_prompt, skill_dir(cwd)),
+ env,
+ cwd,
+ "Reply OK. Do not call any tool.",
+ 180,
+ 3,
+ work / "pi-events.tsv",
+ stream,
+ work / "stderr.txt",
+ )
+ finally:
+ keep_config(root, work / "pi-agent")
+ summary = claude_events.summarize(claude_events.read_events(stream))
+ text = (summary.get("final_text") or "").strip()
+ init = summary["init"]
+ problems = []
+ if not text.upper().startswith(("OK", "DONE")) or summary["tool_calls"]["total"]:
+ problems.append(f"reply={text[:40]!r} tools={summary['tool_calls']['total']}")
+ if init["tools_builtin"] != ["Read", "ToolSearch"] or init["mcp_tool_count"]:
+ problems.append(f"declared tools {init['tools_builtin']} + {init['mcp_tool_count']} MCP")
+ if init["skills"] != ["cua-driver"]:
+ problems.append(f"skills {init['skills']}")
+ if result["models"] != [f"{PROVIDER}/{MODEL}"]:
+ problems.append(f"answered by {result['models']}")
+ detail = f"{arm}: models={result['models']} skills={init['skills']} tokens={summary['tokens']}"
+ return ("fail", "; ".join(problems) + " | " + detail) if problems else ("pass", detail)
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/pi_gate.ts b/libs/cua-driver-bench/automated-eval/macos_bench/pi_gate.ts
new file mode 100644
index 0000000000..b331293dbe
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/pi_gate.ts
@@ -0,0 +1,37 @@
+// SPDX-License-Identifier: FSL-1.1-MIT
+// Copyright (c) 2026 Cua AI, Inc.
+
+// pi extension for the pi-* bench arms (pi_arms.py), giving pi the limits Claude Code's own flags give the cc-* arms.
+// Read: the Claude Code arms load the skill with `Skill`, and every `Read` is denied (permission mode dontAsk, only
+// mcp__cua allowed), so `read` may open the skill's SKILL.md (CDB_PI_READ_ALLOW) and nothing else.
+// Turns: like --max-turns, the CDB_PI_MAX_TURNS-th assistant turn may still run its tools; the next turn is aborted
+// before its model request, and no tool call runs after the cap.
+import { realpathSync } from "node:fs";
+import { resolve } from "node:path";
+
+function real(path: string): string {
+ try {
+ return realpathSync(path);
+ } catch {
+ return path;
+ }
+}
+
+export default function (pi: any) {
+ const allowed = process.env.CDB_PI_READ_ALLOW ? real(process.env.CDB_PI_READ_ALLOW) : undefined;
+ const maxTurns = Number(process.env.CDB_PI_MAX_TURNS || "0");
+ let turns = 0;
+ pi.on("turn_start", async (_event: any, ctx: any) => {
+ turns += 1;
+ if (maxTurns > 0 && turns > maxTurns) ctx.abort();
+ });
+ pi.on("tool_call", async (event: any) => {
+ if (maxTurns > 0 && turns > maxTurns) {
+ return { block: true, reason: "The turn limit is reached.", terminate: true };
+ }
+ if (event.toolName !== "read") return undefined;
+ const path = real(resolve(process.cwd(), String(event.input?.path ?? "")));
+ if (allowed && path === allowed) return undefined;
+ return { block: true, reason: "Permission to use read has been denied; only the skill's SKILL.md is readable." };
+ });
+}
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.py b/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.py
index 958b31520a..0f608071cb 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/run_bench.py
@@ -55,6 +55,7 @@ import bench_core as core # noqa: E402
import claude_arms as ca # noqa: E402
import cdb_adapter # noqa: E402
import codex_native as cxn # noqa: E402
+import pi_arms as pia # noqa: E402
import browser_use_cli as bu
import claude_driver # noqa: E402
import claude_events # noqa: E402
@@ -64,6 +65,10 @@ import desktop_runtime as desktop
from dataclasses import replace
SCHEMA = "cdb-bench-trial/1"
+# The Claude Code arm whose driver build, MCP entry, skill and system prompt a Codex or pi arm uses
+SOURCE_ARM = {**cxn.SOURCE_ARM, **pia.SOURCE_ARM}
+# Arms that use no Claude seat: no quota gate, no Haiku probes
+NO_CLAUDE_SEAT = cxn.ARMS + pia.ARMS
WEB_TASKS = ("WEB-01", "WEB-02")
WORK = ca.WORK
DEFAULT_LEDGER = WORK / "ledger" / "spend.jsonl"
@@ -487,7 +492,7 @@ def check_pins(pins: dict[str, Any], arm_names: list[str]) -> list[tuple[str, st
"""Compare every pinned version and hash with what is installed now. Any difference fails."""
observed = ca.observed_pins(
include_codex=any(a in ca.CODEX_ARMS or a in cxn.ARMS for a in arm_names),
- include_cua="cc-cua-driver" in arm_names,
+ include_cua="cc-cua-driver" in {SOURCE_ARM.get(a, a) for a in arm_names},
)
if (
"cc-codex-cu-1007-browser" in arm_names
@@ -586,7 +591,7 @@ def capture_subject(work: Path):
def real_capture_check(ctx: "Ctx", arm: str) -> tuple[str, str, str]:
client = None
try:
- source = cxn.SOURCE_ARM.get(arm, arm)
+ source = SOURCE_ARM.get(arm, arm)
path, _ = ctx.mcp[arm]
if source in ca.CUA_ARMS:
ensure_agent_daemon(ctx, source)
@@ -726,7 +731,7 @@ def cu_helper_live_checks(ctx: "Ctx", arm: str) -> list[tuple[str, str, str]]:
)
)
try:
- selected = next(a for a in ctx.arm_names if cxn.SOURCE_ARM.get(a, a) == arm)
+ selected = next(a for a in ctx.arm_names if SOURCE_ARM.get(a, a) == arm)
path, _server = ctx.mcp[selected]
if selected in cxn.ARMS:
entry = cxn.server_entry(path, selected)
@@ -854,13 +859,13 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]
run_index = entry["run_index"]
seed = core.probe_seed(task.id, run_index)
system_prompt = ca.system_prompt_for(
- cxn.SOURCE_ARM.get(arm, arm), browser=bool(spec.get("browser"))
+ SOURCE_ARM.get(arm, arm), browser=bool(spec.get("browser"))
)
bench_v2 = bool(getattr(args, "bench_v2", False)) # Amendment 14
sentinel_front = True if bench_v2 else bool(spec.get("sentinel_frontmost", True))
condition = getattr(args, "condition", "foreground") if bench_v2 else None # Amendment 15
# Per-build Cua Driver identity (Amendment 3); arm B rows keep the 0.34.0 values of the recorder build.
- build_versions = (ctx.versions.get("builds") or {}).get(cxn.SOURCE_ARM.get(arm, arm)) or (
+ build_versions = (ctx.versions.get("builds") or {}).get(SOURCE_ARM.get(arm, arm)) or (
ctx.versions.get("builds") or {}
).get("cc-cua-driver", {})
trial_dir = ctx.run_dir / "trials" / entry["trial_id"] / f"a{attempt}"
@@ -910,14 +915,16 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]
"cua_driver_git_sha": build_versions.get("cua_driver_git_sha"),
"cua_driver_daemon_version": build_versions.get("cua_daemon_version"),
"skills_tree_sha256": build_versions.get("cua_skills_tree_sha256")
- if arm in ca.CUA_ARMS
+ if pia.SOURCE_ARM.get(arm, arm) in ca.CUA_ARMS
else None,
"macos": ctx.versions.get("macos"),
"os_version": ctx.versions.get("os_version"),
"bench_repo_commit": ctx.versions.get("bench_repo_commit"),
"bench_src_sha256": ctx.versions.get("bench_src_sha256"),
"system_prompt_sha256": hashlib.sha256(system_prompt.encode()).hexdigest(),
- "skill_in_prompt": bool(getattr(ca.CUA_BUILDS.get(arm), "skill_in_prompt", False)),
+ "skill_in_prompt": bool(
+ getattr(ca.CUA_BUILDS.get(pia.SOURCE_ARM.get(arm, arm)), "skill_in_prompt", False)
+ ),
"electron_named_bundles": spec.get("kind") == "cdb"
and os.environ.get("CDB_ELECTRON_SHARED") != "1",
"quota_five_hour_before": (ctx.quota or {}).get("five_hour"),
@@ -926,13 +933,16 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]
"force_accessibility": ca.force_accessibility(arm),
# Amendment 11: the helper arm's identity (Desktop version, helper and adapter hashes, offered actions)
"claude_cu_helper": cu_helper_row_identity(ctx)
- if cxn.SOURCE_ARM.get(arm, arm) in ca.CU_HELPER_ARMS
+ if SOURCE_ARM.get(arm, arm) in ca.CU_HELPER_ARMS
else None,
# Amendment 20: the native Codex arm's identity (Codex CLI build, model, effort)
"codex_native": ctx.versions.get("codex_native") if arm in cxn.ARMS else None,
+ "pi": ctx.versions.get("pi") if arm in pia.ARMS else None,
}
if arm in cxn.ARMS:
row["model"], row["effort"] = cxn.MODEL, cxn.EFFORT
+ if arm in pia.ARMS:
+ row["model"], row["effort"] = f"{pia.PROVIDER}/{pia.MODEL}", pia.THINKING
sentinel = pilot.Sentinel(
Path(args.build_dir or HERE / "linux") / "BenchSentinel.app"
if (args.build_dir or desktop.detect_platform() == "linux") and not args.no_sentinel
@@ -965,6 +975,7 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]
side_door: set[str] = set()
watcher: FrontWatcher | None = None
guard: v2.GuiOnlyGuard | None = None
+ pi_root: Path | None = None
gui_only: dict[str, Any] | None = None
try:
ctx.set_state(
@@ -979,9 +990,9 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]
if arm in ca.CUA_ARMS:
ensure_agent_daemon(ctx, arm)
elif (
- cxn.SOURCE_ARM.get(arm) in ca.CUA_ARMS
+ SOURCE_ARM.get(arm) in ca.CUA_ARMS
): # A23: Cua Driver under Codex uses that build's daemon
- ensure_agent_daemon(ctx, cxn.SOURCE_ARM[arm])
+ ensure_agent_daemon(ctx, SOURCE_ARM[arm])
if getattr(args, "park_pointer", False):
row["pointer_parked"] = park_pointer(ctx)
lab_app = Path(args.build_dir or HERE / "linux") / "BenchLab.app"
@@ -1016,12 +1027,14 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]
elif cdb is None and getattr(args, "check_occlusion", True):
row["lab_occlusion"] = ensure_lab_unoccluded(ctx)
row["lab_unoccluded"] = row["lab_occlusion"].get("unoccluded")
- cwd = ca.prepare_cwd(arm)
+ cwd = ca.prepare_cwd(pia.SOURCE_ARM.get(arm, arm)) # a pi arm gets its source arm's skill
mcp_path, server = ctx.mcp[arm]
native = arm in cxn.MCP_ARMS
argv = (
cxn.argv()
if native
+ else pia.argv(system_prompt, pia.skill_dir(cwd))
+ if arm in pia.ARMS
else ca.claude_argv(
mcp_config=mcp_path,
server=server,
@@ -1085,6 +1098,20 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]
abort=v2.EitherEvent(ctx.abort, guard.stop) if guard else ctx.abort,
server=cxn.SERVER_OF[arm],
)
+ elif arm in pia.ARMS:
+ pi_root = pia.new_root()
+ claude = pia.run_pi(
+ argv,
+ pia.render_agent_dir(pi_root, mcp_path, pia.skill_dir(cwd), max_turns),
+ cwd,
+ prompt,
+ timeout_s,
+ max_turns,
+ trial_dir / "pi-events.tsv",
+ stream_path,
+ stderr_path,
+ abort=v2.EitherEvent(ctx.abort, guard.stop) if guard else ctx.abort,
+ )
else:
claude = claude_driver.run_claude(
argv,
@@ -1155,8 +1182,12 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]
note = failure["message"]
finally:
try:
- _collect_linux_evidence(artifacts, arm, bool(failure.get("kind")))
+ _collect_linux_evidence(
+ artifacts, pia.SOURCE_ARM.get(arm, arm), bool(failure.get("kind"))
+ )
finally:
+ if pi_root is not None:
+ pia.keep_config(pi_root, trial_dir / "pi-agent")
codex_home = trial_dir / "codex-home" / ".codex"
if arm in cxn.ARMS and codex_home.is_dir():
try:
@@ -1180,9 +1211,7 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]
if cdb is not None:
cdb.stop_apps()
pilot.sweep_processes(str(trial_dir))
- helper_arm = cxn.SOURCE_ARM.get(
- arm, arm
- ) # A23: the helper under Codex CLI uses the same build
+ helper_arm = SOURCE_ARM.get(arm, arm) # A23: the helper under Codex CLI uses the same build
if helper_arm in ca.CU_HELPER_ARMS:
# the adapter and its helper end with claude's process group; a survivor could post input later
row["cu_helper_leftover_killed"] = pilot.sweep_processes(
@@ -1203,6 +1232,8 @@ def run_attempt(ctx: Ctx, entry: dict[str, Any], attempt: int) -> dict[str, Any]
None,
"not priced (Codex subscription; tokens reported)",
)
+ elif entry["arm"] in pia.ARMS:
+ cost_source = pia.COST_SOURCE
elif cost is None:
cost = estimate_cost(tokens)
cost_source = "estimated_from_tokens (no result event)"
@@ -1765,7 +1796,7 @@ def append_row(ctx: Ctx, row: dict[str, Any]) -> None:
ctx.ledger.append(
ctx.args.ledger_who,
f"{'smoke ' if row.get('smoke') else ''}trial {row['trial_id']} a{row['attempt']}",
- ctx.args.model,
+ row.get("model") or ctx.args.model,
cost,
run_id=ctx.run_dir.name,
status=row.get("status"),
@@ -1869,10 +1900,10 @@ def execute_trial(ctx: Ctx, entry: dict[str, Any]) -> dict[str, Any]:
return row
if (
kind in ("mcp_start", "tool_setup")
- and cxn.SOURCE_ARM.get(entry["arm"], entry["arm"]) in ca.CUA_ARMS
+ and SOURCE_ARM.get(entry["arm"], entry["arm"]) in ca.CUA_ARMS
):
try:
- ensure_agent_daemon(ctx, cxn.SOURCE_ARM.get(entry["arm"], entry["arm"]))
+ ensure_agent_daemon(ctx, SOURCE_ARM.get(entry["arm"], entry["arm"]))
except Exception as error: # noqa: BLE001
ctx.log(f"daemon restart failed: {error}")
log_pause(ctx, f"retry_{kind}", wait_s, row.get("infra_message", ""), entry["trial_id"])
@@ -1990,7 +2021,7 @@ def native_codex_checks(ctx: Ctx, arm: str, with_models: bool) -> list[tuple[str
checks: list[tuple[str, str, str]] = []
try:
env = cxn.render_home(
- home, mcp_path, cxn.MODEL, cxn.EFFORT, ca.system_prompt_for(cxn.SOURCE_ARM[arm]), arm
+ home, mcp_path, cxn.MODEL, cxn.EFFORT, ca.system_prompt_for(SOURCE_ARM[arm]), arm
)
if chatgpt:
checks.extend(cxn.chatgpt_checks(cxn.CODEX_CLI, env, cxn.MODEL))
@@ -2151,6 +2182,17 @@ def v2_static_checks(ctx: Ctx) -> list[tuple[str, str, str]]:
)
)
for arm in ctx.arm_names:
+ if arm in pia.ARMS:
+ tools = [pia.claude_name(t) for t in pia.TOOLS if not t.startswith("mcp__")]
+ bad = [t for t in tools if t in v2.FORBIDDEN_BUILTINS or t == "Task"]
+ out.append(
+ (
+ f"v2 GUI-only surface {arm}",
+ "fail" if bad else "pass",
+ f"pi tools {list(pia.TOOLS)}" + (f"; forbidden {bad}" if bad else ""),
+ )
+ )
+ continue
argv = ca.claude_argv(
mcp_config=Path("/dev/null"),
server=(ctx.mcp.get(arm) or (None, "server"))[1],
@@ -2189,11 +2231,15 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:
if getattr(args, "bench_v2", False):
for name, status, detail in v2_static_checks(ctx):
add(name, status, detail)
- add(
- "claude binary",
- "pass" if ca.CLAUDE_BIN.is_file() else "fail",
- f"{ca.CLAUDE_BIN} {ctx.versions.get('claude')}",
- )
+ if any(a not in pia.ARMS for a in ctx.arm_names):
+ add(
+ "claude binary",
+ "pass" if ca.CLAUDE_BIN.is_file() else "fail",
+ f"{ca.CLAUDE_BIN} {ctx.versions.get('claude')}",
+ )
+ if any(a in pia.ARMS for a in ctx.arm_names):
+ pi = ctx.versions.get("pi") or {}
+ add("pi binary", "pass" if pi.get("pi_version") else "fail", f"{pia.PI_BIN} {pi}")
add(
"terminal.app ancestor",
"skip"
@@ -2224,12 +2270,15 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:
for name, status, detail in check_pins(pins, ctx.arm_names):
add(name, status, detail)
offline = bool(getattr(args, "offline", False))
- if any(ca.CUA_BUILDS[a].label == "main" for a in ctx.arm_names if a in ca.CUA_BUILDS):
+ driver_arms = [pia.SOURCE_ARM.get(a, a) for a in ctx.arm_names]
+ if any(ca.CUA_BUILDS[a].label == "main" for a in driver_arms if a in ca.CUA_BUILDS):
main = pins.get("cua_main", {})
spec = ca.CUA_BUILDS["cc-cua-driver-main"]
got_bin = ca.sha256_file(spec.bin) if spec.bin.is_file() else None
for other in [
- a for a in ctx.arm_names if a in ca.CUA_BUILDS and ca.CUA_BUILDS[a].label == "main"
+ a
+ for a in dict.fromkeys(driver_arms)
+ if a in ca.CUA_BUILDS and ca.CUA_BUILDS[a].label == "main"
]:
ob = ca.CUA_BUILDS[other].bin
got = ca.sha256_file(ob) if ob.is_file() else None
@@ -2250,7 +2299,7 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:
"pass" if got_skills and got_skills == main.get("skills_tree_sha256") else "fail",
str(got_skills),
)
- for cua_arm in [a for a in ca.CUA_ARMS if a in ctx.arm_names]:
+ for cua_arm in [a for a in ca.CUA_ARMS if a in driver_arms]:
spec = ca.CUA_BUILDS[cua_arm]
try:
ensure_agent_daemon(ctx, cua_arm)
@@ -2276,7 +2325,7 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:
):
add(name, status, detail)
helper_sources = {
- cxn.SOURCE_ARM.get(a, a) for a in ctx.arm_names
+ SOURCE_ARM.get(a, a) for a in ctx.arm_names
} # A23: also for cx-claude-cu-helper
for helper_arm in [a for a in ca.CU_HELPER_ARMS if a in helper_sources]:
observed = ca.cu_helper_observed(ca.CU_HELPER_BUILDS[helper_arm])
@@ -2426,7 +2475,7 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:
"quota gate from the latest known reading",
"warn"
if gate["action"] == "gated"
- and all(a in cxn.ARMS for a in ctx.arm_names) # no Claude seat in use
+ and all(a in NO_CLAUDE_SEAT for a in ctx.arm_names) # no Claude seat in use
else "fail"
if gate["action"] == "gated"
else ("warn" if gate["action"] == "unknown" else "pass"),
@@ -2434,7 +2483,21 @@ def preflight(ctx: Ctx, with_models: bool = True) -> list[tuple[str, str, str]]:
)
if (WORK / "HOLD").exists():
add("HOLD file", "fail", (WORK / "HOLD").read_text("utf-8").strip()[:200])
- claude_arms = [a for a in ctx.arm_names if a not in cxn.ARMS]
+ for arm in [a for a in ctx.arm_names if a in pia.ARMS] if with_models and not offline else []:
+ try:
+ source = pia.SOURCE_ARM[arm]
+ ensure_agent_daemon(ctx, source)
+ status, detail = pia.init_check(
+ arm,
+ ctx.mcp[arm][0],
+ ca.prepare_cwd(source),
+ ca.system_prompt_for(source),
+ ctx.run_dir / f"preflight-{arm}",
+ )
+ add(f"init event check {arm}", status, detail)
+ except Exception as error: # noqa: BLE001
+ add(f"init event check {arm}", "fail", f"{type(error).__name__}: {error}")
+ claude_arms = [a for a in ctx.arm_names if a not in NO_CLAUDE_SEAT]
if with_models and not offline and claude_arms: # a native-Codex-only run uses no Claude seat
try:
ok, detail = quota_probe(ctx)
@@ -2661,11 +2724,13 @@ def make_ctx(args: argparse.Namespace) -> Ctx:
unsupported = []
if desktop.detect_platform() == "linux":
unsupported = [
- f"unsupported-platform: {a} requires macOS" for a in arm_names if a not in ca.CUA_ARMS
+ f"unsupported-platform: {a} requires macOS"
+ for a in arm_names
+ if a not in ca.CUA_ARMS + pia.ARMS
]
for reason in unsupported:
print(reason, file=sys.stderr)
- arm_names = [a for a in arm_names if a in ca.CUA_ARMS]
+ arm_names = [a for a in arm_names if a in ca.CUA_ARMS + pia.ARMS]
if not arm_names:
raise SystemExit("no supported arms remain: " + "; ".join(unsupported))
if args.pins:
@@ -2725,7 +2790,7 @@ def setup_arms(ctx: Ctx) -> None:
)
observed = ca.observed_pins(
include_codex=any(a in ca.CODEX_ARMS or a in cxn.ARMS for a in ctx.arm_names),
- include_cua="cc-cua-driver" in ctx.arm_names,
+ include_cua="cc-cua-driver" in {SOURCE_ARM.get(a, a) for a in ctx.arm_names},
)
ctx.versions.update(
pins_expected=pins,
@@ -2753,7 +2818,7 @@ def setup_arms(ctx: Ctx) -> None:
for arm, build in ca.CUA_BUILDS.items()
if arm in ctx.arm_names
or arm == "cc-cua-driver"
- or arm in {cxn.SOURCE_ARM.get(a) for a in ctx.arm_names}
+ or arm in {SOURCE_ARM.get(a) for a in ctx.arm_names}
},
pins=pins,
system_prompt_sha256=hashlib.sha256(ca.SYSTEM_PROMPT.encode()).hexdigest(),
@@ -2769,13 +2834,18 @@ def setup_arms(ctx: Ctx) -> None:
): # A20/A23: the source Claude arm's MCP entry, copied unmodified, driven by Codex CLI
try:
path, _server = ca.mcp_config_for(
- cxn.SOURCE_ARM[arm], ctx.run_dir, ctx.args.codex_cu_mcp
+ SOURCE_ARM[arm], ctx.run_dir, ctx.args.codex_cu_mcp
)
cxn.server_entry(path, arm)
ctx.mcp[arm] = (path, cxn.SERVER_OF[arm])
except (FileNotFoundError, ValueError) as error:
raise SystemExit(f"arm {arm} unavailable: {error}")
ctx.versions["codex_native"] = cxn.observed()
+ elif (
+ arm in pia.ARMS
+ ): # the source arm's MCP entry; pi_arms.render_agent_dir copies it per trial
+ ctx.mcp[arm] = ca.mcp_config_for(SOURCE_ARM[arm], ctx.run_dir)
+ ctx.versions["pi"] = pia.observed()
if ctx.args.build_dir or desktop.detect_platform() == "linux":
apps: dict[str, str] = {}
for name in BENCH_APPS:
@@ -2862,12 +2932,12 @@ def write_manifest(ctx: Ctx, first_schedule: list[dict[str, Any]]) -> None:
"tool_search": args.tool_search,
"system_prompt": ca.SYSTEM_PROMPT,
"system_prompt_sha256_by_arm": {
- a: hashlib.sha256(ca.system_prompt_for(cxn.SOURCE_ARM.get(a, a)).encode()).hexdigest()
+ a: hashlib.sha256(ca.system_prompt_for(SOURCE_ARM.get(a, a)).encode()).hexdigest()
for a in ctx.arm_names
},
"system_prompt_sha256_by_arm_browser": {
a: hashlib.sha256(
- ca.system_prompt_for(cxn.SOURCE_ARM.get(a, a), browser=True).encode()
+ ca.system_prompt_for(SOURCE_ARM.get(a, a), browser=True).encode()
).hexdigest()
for a in ctx.arm_names
},
@@ -2877,7 +2947,9 @@ def write_manifest(ctx: Ctx, first_schedule: list[dict[str, Any]]) -> None:
"versions": ctx.versions,
"recorder": getattr(ctx.recorder, "name", "none"),
"first_block_schedule": first_schedule,
- "arm_descriptions": {a: ca.ARM_DESCRIPTIONS.get(a, a) for a in ctx.arm_names},
+ "arm_descriptions": {
+ a: ca.ARM_DESCRIPTIONS.get(a) or pia.DESCRIPTIONS.get(a, a) for a in ctx.arm_names
+ },
"bench_version": v2.BENCH_VERSION if getattr(args, "bench_v2", False) else 1,
"bench_v2": {
"categories": v2.CATEGORIES,
@@ -2900,7 +2972,11 @@ def write_manifest(ctx: Ctx, first_schedule: list[dict[str, Any]]) -> None:
"price_fallback_usd_per_mtok_ASSUMED": PRICE_FALLBACK,
"example_argv_task": ctx.task_ids[0],
"example_argv": {
- a: ca.claude_argv(
+ a: pia.argv(
+ ca.system_prompt_for(SOURCE_ARM[a]), Path(ca.CWD_ROOT) / ".claude/skills/cua-driver"
+ )
+ if a in pia.ARMS
+ else ca.claude_argv(
mcp_config=ctx.mcp[a][0],
server=ctx.mcp[a][1],
model=args.model,
@@ -2909,7 +2985,7 @@ def write_manifest(ctx: Ctx, first_schedule: list[dict[str, Any]]) -> None:
tool_search=args.tool_search == "default",
effort=args.effort,
system_prompt=ca.system_prompt_for(
- cxn.SOURCE_ARM.get(a, a),
+ SOURCE_ARM.get(a, a),
browser=bool(task_spec(ctx.tasks[ctx.task_ids[0]]).get("browser")),
),
)
@@ -3031,7 +3107,7 @@ def cmd_run(args: argparse.Namespace) -> int:
failed = [c for c in checks if c[1] == "fail"]
known = ctx.quota is not None and ctx.quota.get("seven_day") is not None
native_only = all(
- a in cxn.ARMS for a in ctx.arm_names
+ a in NO_CLAUDE_SEAT for a in ctx.arm_names
) # Amendment 20: no Claude seat in use
if (
not known
@@ -3102,7 +3178,7 @@ def cmd_gate_check(args: argparse.Namespace) -> int:
if decision["action"] == "gated":
arm_names = [args.only_arm] if args.only_arm else list(args.arms)
if arm_names and all(
- a in cxn.ARMS for a in arm_names
+ a in NO_CLAUDE_SEAT for a in arm_names
): # Amendment 20: no Claude seat in use
print("quota gate not applied: only native Codex arms, which use no Claude seat")
return 0
@@ -3162,7 +3238,7 @@ def cmd_dry_run(args: argparse.Namespace) -> int:
tool_search=args.tool_search == "default",
effort=args.effort,
system_prompt=ca.system_prompt_for(
- cxn.SOURCE_ARM.get(arm, arm),
+ SOURCE_ARM.get(arm, arm),
browser=bool(task_spec(ctx.tasks[task_id]).get("browser")),
),
)
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_pi_arms.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_pi_arms.py
new file mode 100644
index 0000000000..3153e5ff28
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_pi_arms.py
@@ -0,0 +1,284 @@
+# SPDX-License-Identifier: FSL-1.1-MIT
+# Copyright (c) 2026 Cua AI, Inc.
+
+"""The pi-hosted Cua Driver arms: the tool surface they offer and how pi's events reach the guard and the row."""
+
+from __future__ import annotations
+
+import json
+import os
+import stat
+import sys
+import tempfile
+import textwrap
+import unittest
+from pathlib import Path
+from unittest import mock
+
+HERE = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(HERE))
+
+import arms # noqa: E402
+import bench_v2 as v2 # noqa: E402
+import claude_arms as ca # noqa: E402
+import claude_events # noqa: E402
+import pi_arms as pia # noqa: E402
+
+
+def assistant(content, stop="toolUse", usage=None):
+ return {
+ "type": "message_end",
+ "message": {
+ "role": "assistant",
+ "content": content,
+ "provider": "openai-codex",
+ "model": "gpt-6.1-terra",
+ "stopReason": stop,
+ "usage": usage or {"input": 10, "output": 2, "cacheRead": 5, "cost": {"total": 0.5}},
+ },
+ }
+
+
+def call(call_id, name, arguments=None):
+ return {"type": "toolCall", "id": call_id, "name": name, "arguments": arguments or {}}
+
+
+SYSTEM = {
+ "type": "message_end",
+ "message": {
+ "role": "system",
+ "toolsAdded": [{"name": "read"}, {"name": "tool_search"}],
+ "sections": {"skills": "<skill>\n<name>cua-driver</name>\n</skill>", "cwd": "x"},
+ },
+}
+
+
+class PiArmsTest(unittest.TestCase):
+ def test_each_arm_mirrors_one_driver_arm(self):
+ self.assertEqual(set(pia.ARMS) & set(arms.ALL_ARMS), set(pia.ARMS))
+ self.assertEqual(set(pia.SOURCE_ARM.values()), set(ca.CUA_ARMS))
+
+ def test_argv_offers_read_tool_search_and_the_cua_server_only(self):
+ argv = pia.argv("SYSTEM PROMPT", Path("/w/.claude/skills/cua-driver"), Path("/bin/pi"))
+ self.assertEqual(argv[argv.index("--tools") + 1], "read,tool_search,mcp__cua__*")
+ self.assertEqual(argv[argv.index("--system-prompt") + 1], "SYSTEM PROMPT")
+ self.assertEqual(argv[argv.index("--skill") + 1], "/w/.claude/skills/cua-driver")
+ for flag in ("--no-extensions", "--no-skills", "--no-context-files", "--no-session"):
+ self.assertIn(flag, argv)
+ loaded = [argv[i + 1] for i, a in enumerate(argv) if a == "-e"]
+ self.assertEqual(loaded, ["builtin:mcp", "builtin:tool-search", str(pia.GATE)])
+ surface = [pia.claude_name(t) for t in pia.TOOLS if not t.startswith("mcp__")]
+ self.assertFalse(set(surface) & set(v2.FORBIDDEN_BUILTINS))
+
+ def test_tools_outside_the_surface_are_forbidden_builtins(self):
+ for name in ("bash", "edit", "write", "codemode", "grep", "mcp__other__run"):
+ hard, _ = v2.scan_tool_use(pia.claude_name(name), {})
+ self.assertTrue(hard, name)
+ self.assertEqual(v2.scan_tool_use(pia.claude_name("read"), {"path": "x"}), ([], []))
+
+ def test_agent_dir_copies_the_source_mcp_entry_deferred_and_links_auth(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ auth = root / "auth.json"
+ auth.write_text("{}")
+ source = root / "mcp.json"
+ entry = {"type": "stdio", "command": "/d/cua-driver", "args": ["--socket", "/s", "mcp"]}
+ source.write_text(json.dumps({"mcpServers": {"cua": entry}}))
+ pi_root = root / "pi-root"
+ with mock.patch.dict(os.environ, {"CDB_PI_AUTH_FILE": str(auth)}):
+ env = pia.render_agent_dir(pi_root, source, root / "skill", 45)
+ agent = Path(env["PI_CODING_AGENT_DIR"])
+ config = json.loads((agent / "mcp.json").read_text())
+ self.assertEqual(config["mcpServers"]["cua"], {**entry, "exposure": "deferred"})
+ self.assertIs(config["autoEnableCodemode"], False)
+ self.assertTrue(stat.S_ISLNK((agent / "auth.json").lstat().st_mode))
+ self.assertNotEqual(env["HOME"], os.environ.get("HOME"))
+ self.assertEqual(env["CDB_PI_READ_ALLOW"], str(root / "skill" / "SKILL.md"))
+ self.assertEqual(env["CDB_PI_MAX_TURNS"], "45")
+ pia.keep_config(pi_root, root / "trial" / "pi-agent")
+ self.assertFalse(pi_root.exists())
+ self.assertEqual(
+ sorted(p.name for p in (root / "trial" / "pi-agent").iterdir()),
+ ["mcp.json", "settings.json"],
+ )
+
+ def test_events_become_the_claude_shaped_stream(self):
+ tr = pia.Translator(max_turns=45)
+ events = tr.feed(SYSTEM)
+ events += tr.feed(assistant([call("c1", "tool_search", {"query": "click"})]))
+ events += tr.feed(
+ {
+ "type": "tool_execution_end",
+ "toolCallId": "c1",
+ "toolName": "tool_search",
+ "result": {"content": [{"type": "text", "text": "Loaded 2 tools"}]},
+ }
+ )
+ tr.feed(
+ {
+ "type": "message_end",
+ "message": {"role": "system", "toolsAdded": [{"name": "mcp__cua__click"}]},
+ }
+ )
+ events += tr.feed(assistant([call("c2", "mcp__cua__click", {"x": 1})]))
+ events += tr.feed(
+ {
+ "type": "tool_execution_end",
+ "toolCallId": "c2",
+ "toolName": "mcp__cua__click",
+ "isError": True,
+ "result": {"content": [{"type": "text", "text": "no element"}]},
+ }
+ )
+ events += tr.feed(assistant([{"type": "text", "text": "DONE"}], stop="stop"))
+ events += tr.feed({"type": "agent_settled", "aborted": False})
+ events += tr.finish(capped=False, wall_ms=1000)
+ summary = claude_events.summarize(events)
+ self.assertEqual(summary["init"]["tools_builtin"], ["Read", "ToolSearch"])
+ self.assertEqual(summary["init"]["mcp_tools"], ["mcp__cua__click"])
+ self.assertEqual(summary["init"]["skills"], ["cua-driver"])
+ self.assertEqual(summary["init"]["mcp_servers"], [{"name": "cua", "status": "connected"}])
+ self.assertEqual(summary["tool_calls"]["by_name"], {"ToolSearch": 1, "mcp__cua__click": 1})
+ self.assertEqual(summary["tool_calls"]["failed_by_name"], {"mcp__cua__click": 1})
+ self.assertEqual(summary["turns"], 3)
+ self.assertEqual(summary["final_text"], "DONE")
+ self.assertEqual(summary["result"]["subtype"], "success")
+ self.assertEqual(
+ summary["tokens"], {"input": 30, "output": 6, "cache_read": 15, "cache_write": 0}
+ )
+ self.assertAlmostEqual(summary["total_cost_usd"], 1.5)
+ self.assertEqual(claude_events.classify_failure(summary, "", 0)["kind"], None)
+
+ def test_turn_cap_is_reached_when_a_turn_past_the_cap_starts(self):
+ tr = pia.Translator(max_turns=2)
+ for n in (1, 2):
+ tr.feed({"type": "turn_start"})
+ tr.feed(assistant([call(f"c{n}", "mcp__cua__click")]))
+ self.assertFalse(tr.capped.is_set())
+ tr.feed({"type": "turn_start"})
+ self.assertTrue(tr.capped.is_set())
+ aborted = assistant([], stop="error")
+ aborted["message"]["errorMessage"] = "This operation was aborted"
+ self.assertEqual(tr.feed(aborted), [])
+ result = tr.finish(capped=True, wall_ms=1)[-1]
+ self.assertEqual(
+ (result["subtype"], result["is_error"], result["num_turns"]),
+ ("error_max_turns", True, 2),
+ )
+
+ def test_a_final_answer_on_the_last_allowed_turn_is_not_capped(self):
+ tr = pia.Translator(max_turns=1)
+ tr.feed({"type": "turn_start"})
+ tr.feed(assistant([{"type": "text", "text": "DONE"}], stop="stop"))
+ tr.feed({"type": "agent_settled"})
+ self.assertFalse(tr.capped.is_set())
+ self.assertEqual(tr.finish(capped=False, wall_ms=1)[-1]["subtype"], "success")
+
+ def test_provider_error_is_classified_as_infrastructure(self):
+ tr = pia.Translator(max_turns=45)
+ message = assistant([], stop="error")
+ message["message"]["errorMessage"] = "429 Too Many Requests: usage limit reached"
+ events = tr.feed(message) + tr.feed({"type": "agent_settled"})
+ summary = claude_events.summarize(events + tr.finish(capped=False, wall_ms=1, returncode=1))
+ self.assertEqual(claude_events.classify_failure(summary, "", 0)["kind"], "rate_limit")
+
+ def test_exit_without_settling_is_a_harness_failure_not_a_result(self):
+ tr = pia.Translator(max_turns=45)
+ events = tr.feed(SYSTEM) + tr.feed(assistant([call("c1", "mcp__cua__click")]))
+ self.assertEqual(tr.finish(capped=False, wall_ms=1, returncode=1), [])
+ summary = claude_events.summarize(events)
+ self.assertEqual(
+ claude_events.classify_failure(summary, "TypeError", 1)["kind"], "harness_crash"
+ )
+
+ def test_settled_with_nonzero_exit_is_a_harness_failure(self):
+ tr = pia.Translator(max_turns=45)
+ events = tr.feed(SYSTEM) + tr.feed(
+ assistant([{"type": "text", "text": "DONE"}], stop="stop")
+ )
+ tr.feed({"type": "agent_settled"})
+ self.assertEqual(tr.finish(capped=False, wall_ms=1, returncode=1), [])
+ summary = claude_events.summarize(events)
+ self.assertEqual(claude_events.classify_failure(summary, "", 1)["kind"], "harness_crash")
+
+ def test_cap_is_read_after_pi_exits_right_after_the_over_cap_turn(self):
+ lines = [
+ assistant([call("c1", "mcp__cua__click")]),
+ assistant([call("c2", "mcp__cua__click")]),
+ ]
+ script = textwrap.dedent(
+ """\
+ import json, sys
+ sys.stdin.read()
+ for event in json.load(open(sys.argv[1])):
+ print(json.dumps({"type": "turn_start"}))
+ print(json.dumps(event))
+ print(json.dumps({"type": "turn_start"}), flush=True)
+ """
+ )
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ (root / "fake.py").write_text(script)
+ (root / "events.json").write_text(json.dumps(lines))
+ result = pia.run_pi(
+ [sys.executable, str(root / "fake.py"), str(root / "events.json")],
+ {"PATH": os.environ["PATH"]},
+ root,
+ "brief",
+ 20,
+ 2,
+ root / "raw.tsv",
+ root / "stream.tsv",
+ root / "stderr",
+ )
+ events = claude_events.read_events(root / "stream.tsv")
+ self.assertTrue(result["capped"])
+ self.assertEqual(events[-1]["subtype"], "error_max_turns")
+
+ def test_no_turn_leaves_no_result_so_stderr_classifies_the_row(self):
+ tr = pia.Translator(max_turns=45)
+ self.assertEqual(tr.finish(capped=False, wall_ms=1), [])
+ summary = claude_events.summarize([])
+ failure = claude_events.classify_failure(summary, "Error: not logged in", 1)
+ self.assertEqual(failure["kind"], "auth")
+
+ def test_run_pi_streams_events_and_stops_at_the_turn_cap(self):
+ lines = [SYSTEM] + [assistant([call(f"c{n}", "mcp__cua__click")]) for n in range(5)]
+ script = textwrap.dedent(
+ """\
+ import json, sys, time
+ sys.stdin.read()
+ for event in json.load(open(sys.argv[1])):
+ if event["message"]["role"] == "assistant":
+ print(json.dumps({"type": "turn_start"}))
+ print(json.dumps(event), flush=True)
+ time.sleep(0.05)
+ time.sleep(30)
+ """
+ )
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ (root / "fake.py").write_text(script)
+ (root / "events.json").write_text(json.dumps(lines))
+ result = pia.run_pi(
+ [sys.executable, str(root / "fake.py"), str(root / "events.json")],
+ {"PATH": os.environ["PATH"]},
+ root,
+ "brief",
+ 20,
+ 3,
+ root / "raw.tsv",
+ root / "stream.tsv",
+ root / "stderr",
+ )
+ events = claude_events.read_events(root / "stream.tsv")
+ self.assertTrue(result["capped"])
+ self.assertFalse(result["timed_out"])
+ self.assertLess(result["wall_s"], 10)
+ self.assertEqual(result["turns"], 3)
+ self.assertEqual(events[-1]["subtype"], "error_max_turns")
+ self.assertEqual([n for n, _ in v2.tool_uses(events[0])], ["mcp__cua__click"])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tools/check_run_evidence.py b/libs/cua-driver-bench/automated-eval/macos_bench/tools/check_run_evidence.py
index 92cd3c27c6..d35cc108a9 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/tools/check_run_evidence.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tools/check_run_evidence.py
@@ -21,6 +21,10 @@ CUA_ARMS = (
"cc-cua-driver-main",
"cc-cua-driver-main-skill",
"cc-cua-driver-script",
+ "pi-cua-driver",
+ "pi-cua-driver-main",
+ "pi-cua-driver-main-skill",
+ "pi-cua-driver-script",
)