Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cua-driver-bench): Windows port of the BenchLab tasks (CUA-3563) (#6607)
Francesco Bonacci78951406bc05
diff --git a/.github/workflows/ci-cua-driver-bench.yml b/.github/workflows/ci-cua-driver-bench.ymlindex a21c67ac81..5796c35b31 100644--- a/.github/workflows/ci-cua-driver-bench.yml+++ b/.github/workflows/ci-cua-driver-bench.yml@@ -82,6 +82,25 @@ jobs:dbus-run-session -- bash -lc \'openbox >/tmp/cua-openbox.log 2>&1 & picom --backend xrender --config /dev/null >/tmp/cua-picom.log 2>&1 & sleep 2; /usr/bin/python3 -m pytest -q tests/test_pilot_runner.py tests/test_linux_session.py tests/test_runner_flow.py tests/test_bench_v2.py tests/test_analyze.py tests/test_cdb_adapter.py tests/test_linux_tools.py'+ windows-desktop:+ needs: quick+ if: ${{ !failure() && !cancelled() && (!startsWith(github.head_ref, 'release-please--branches--')) }}+ name: Windows desktop contracts (no model)+ runs-on: windows-latest+ timeout-minutes: 20+ steps:+ - name: Check out Cua+ uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4++ - name: Set up uv+ uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6++ - name: BenchLab, sentinel, guard and launcher on the Windows desktop+ working-directory: libs/cua-driver-bench/automated-eval/macos_bench+ run: >-+ uv run --no-project --python 3.12 --with pytest==8.4.2 --with wxPython==4.3.2 --with pywinauto==0.6.9+ python -m pytest -q tests/test_windows_tools.py tests/test_windows_desktop.py+taskpack:needs: quickif: ${{ !failure() && !cancelled() && (!startsWith(github.head_ref, 'release-please--branches--')) }}diff --git a/libs/cua-driver-bench/README.md b/libs/cua-driver-bench/README.mdindex cd63585ed7..b359c6103c 100644--- a/libs/cua-driver-bench/README.md+++ b/libs/cua-driver-bench/README.md@@ -7,9 +7,8 @@ harness, the agent, the model, the prompts and the limits are the same for everysetup; only the computer-use tool changes.This directory holds the harness, the public task set and the method of-**Bench v2**. The ten Bench v2 tasks run on macOS and Linux; the three multi-app-tasks (CDB-G02 to CDB-G04) also run on Windows. The measurement on macOS assumes-a macOS 26 desktop.+**Bench v2**. The ten Bench v2 tasks run on macOS, Linux and Windows. The+measurement on macOS assumes a macOS 26 desktop.| Path | What it is || --- | --- |@@ -44,7 +43,8 @@ Ten tasks in three categories. Each task has a `PROVENANCE.md` next to it.The CDB-G tasks are the GUI-only variants of pack tasks `cdb-s02` to `cdb-s04`:the same files, with no coding tools offered. The MB and IR tasks run inBenchLab, a small AppKit test app built from source in this directory (with a-Linux port in `automated-eval/macos_bench/linux`). A fourth category, Web apps &+Linux port in `automated-eval/macos_bench/linux` and a Windows port in+`automated-eval/macos_bench/windows`). A fourth category, Web apps &forms, has no Bench v2 task yet.### Platforms and conditions@@ -73,17 +73,16 @@ Public tasks, in schedule order:| CDB-G02 | multi_app | Electron mail, chat and calendar apps, LibreOffice Calc | macOS, Linux, Windows | both | Replan a schedule across mail, chat, a calendar and a Calc workbook || CDB-G03 | multi_app | Electron mail app, LibreOffice Calc, GnuCash | macOS, Linux, Windows | both | Review payables across mail, a Calc workbook and a GnuCash book || CDB-G04 | multi_app | Electron case tracker and manual, Google Chrome | macOS, Linux, Windows | both | Triage duplicate support cases with a tracker, a manual, an archive site and a log |-| MB-09 | precision | BenchLab | macOS, Linux | both | Reorder a native list by drag and drop |-| MB-10 | precision | BenchLab | macOS, Linux | foreground only | Reveal a hover-only toolbar and click one action |-| MB-11 | precision | BenchLab | macOS, Linux | foreground only | Read a native tooltip and type the code it shows |-| IR-01 | interruptions | BenchLab | macOS, Linux | both | Finish a two-step form through a permission-style prompt |-| IR-02 | interruptions | BenchLab | macOS, Linux | both | Apply settings while a notification-style banner covers the button |-| IR-03 | interruptions | BenchLab | macOS, Linux | both | Edit two notes and keep both changes through a save prompt |-| IR-04 | interruptions | BenchLab | macOS, Linux | both | Sign up on a web page while a cookie-consent overlay appears |+| MB-09 | precision | BenchLab | macOS, Linux, Windows | both | Reorder a native list by drag and drop |+| MB-10 | precision | BenchLab | macOS, Linux, Windows | foreground only | Reveal a hover-only toolbar and click one action |+| MB-11 | precision | BenchLab | macOS, Linux, Windows | foreground only | Read a native tooltip and type the code it shows |+| IR-01 | interruptions | BenchLab | macOS, Linux, Windows | both | Finish a two-step form through a permission-style prompt |+| IR-02 | interruptions | BenchLab | macOS, Linux, Windows | both | Apply settings while a notification-style banner covers the button |+| IR-03 | interruptions | BenchLab | macOS, Linux, Windows | both | Edit two notes and keep both changes through a save prompt |+| IR-04 | interruptions | BenchLab | macOS, Linux, Windows | both | Sign up on a web page while a cookie-consent overlay appears |<!-- taskset:end -->-The seven BenchLab tasks run on macOS and on Linux. The three multi-app tasks run-on macOS, Linux and Windows; CDB-G03 needs GnuCash. Tasks marked "foreground+All ten tasks run on macOS, Linux and Windows; CDB-G03 needs GnuCash. Tasks marked "foregroundonly" skip the background condition: their readout appears only on a hover or atooltip, which follows the real pointer, so no setup can do them in thebackground.@@ -315,11 +314,54 @@ completion with Wilson intervals, focus steals, key loss, windows raised,swallowed keystrokes, GUI-only violations with evidence, and the interruptionrates. Keep raw results, streams and recordings out of git.+### Windows (the BenchLab tasks)++The Windows port of BenchLab and the sentinel is in+[`automated-eval/macos_bench/windows`](automated-eval/macos_bench/windows): Python with wxPython, so+every control is a native Win32 control that UI Automation exposes. Seeds, the state file, the event log+and the checkers are the same as on macOS and Linux (`probes/_common/benchlab_common.py`), and the+sentinel log is read by the same `swift/summarize_sentinel.py`.++You need Windows 11, Python 3.12 with `wxPython` (and `pywinauto` for the oracle only), and an+interactive desktop session: run the trials as a standard user signed in at the console, not from an+SSH or service session. One trial:++```powershell+python launch\windows_session.py --task IR-02 --run-index 0 --condition background --out C:\bench\IR-02-r0 -- <agent command>+```++`{brief}`, `{out}`, `{seed}` and `{mode}` in the agent command are filled in. The launcher starts+BenchLab and the sentinel (as `BenchLab.exe` and `BenchSentinel.exe`, copies of `pythonw.exe`), brings+the "User activity" window to the front in the background condition, arms the sentinel, runs the agent+with the GUI-only guard watching, then runs the checker and writes `row.json`. `windows\oracle.py` solves+each task through the GUI with real input; `tests/test_windows_desktop.py` checks that the oracle passes+every task, that doing nothing fails and leaves the user undisturbed, and that a deliberate disturbance+is seen.++Where Windows differs:++* **GUI-only guard.** A shell or script host (PowerShell, cmd, Windows Terminal, the script hosts,+ mshta, WSL) started in the desktop session outside the runner's own processes, or a side-door window+ in front, fails and stops the trial. Tool inputs are also scanned for PowerShell, cmd and script-host+ commands.+* **Sentinel.** Windows has no signals, so the runner arms it through a control file. A window counts as+ raised when it is above the user window in the z-order and overlaps it. The desktop measure is+ `IVirtualDesktopManager`; a dialog on the user's app is a window owned by the user window.+* **Dialogs.** BenchLab's dialogs are owned by its window and disable it while open. When BenchLab is not+ in front, a dialog opens without activation just above BenchLab, so BenchLab never raises itself.+* **MB-09.** Rows reorder by a native list drag (the list view's begin-drag, then the button release+ over a row).+* **MB-10 and MB-11.** Hover and tooltips follow the real pointer on Windows too (mouse-leave tracking+ and the tooltip control read the cursor position), so both stay foreground-only. A tooltip counts as+ shown when BenchLab's own tooltip window is visible.+* **Brief.** "on your Mac" reads "on this Windows PC".+### Tests```bashcd automated-eval/macos_benchpython3 -m pytest tests probes/tests swift/tests # harness, checkers, sentinel (no GUI, no model)+python -m pytest tests/test_windows_desktop.py # on a Windows desktop: BenchLab, sentinel, guard, oraclecd ../../taskpack/tasks/shared/cdb-s02 && python3 -m unittest discover -s testscd ../../../../fairness && python3 -m unittest discover -s tests # fairness and health checkspython3 ../automated-eval/macos_bench/tools/taskset.py check # manifest, selectors, pins and docs agreediff --git a/libs/cua-driver-bench/automated-eval/macos_bench/TASKS.md b/libs/cua-driver-bench/automated-eval/macos_bench/TASKS.mdindex df7b8397d1..6aca52c46a 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/TASKS.md+++ b/libs/cua-driver-bench/automated-eval/macos_bench/TASKS.md@@ -259,7 +259,7 @@ Bench v2 and are not in this repository.## Platforms, splits and the task manifest[`../../taskpack/v2-taskset.json`](../../taskpack/v2-taskset.json) lists every task with its category, apps, OS-availability (the seven BenchLab tasks: macOS and Linux; CDB-G02 to CDB-G04: macOS, Linux and Windows), conditions,+availability (all ten tasks: macOS, Linux and Windows), conditions,health requirements and split; the bench README renders it as a table. Selectors are in `probes/<ID>/task.json`.A task whose selector says `"split": "heldout"` is read from a separate pack with the same layout, named by`CDB_TASKPACK_HELDOUT` (selectors in `probes/`, trees in `tasks/`, digests in `pins.heldout.json`); `--task-setdiff --git a/libs/cua-driver-bench/automated-eval/macos_bench/launch/windows_session.py b/libs/cua-driver-bench/automated-eval/macos_bench/launch/windows_session.pynew file mode 100644index 0000000000..2def1710cf--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/launch/windows_session.py@@ -0,0 +1,289 @@+#!/usr/bin/env python3+# SPDX-License-Identifier: FSL-1.1-MIT+# Copyright (c) 2026 Cua AI, Inc.+#+# Usage: windows_session.py --task ID --run-index N --condition foreground|background --out DIR+# [--timeout S] -- COMMAND [ARG...]+#+# One BenchLab trial on a Windows desktop, run in the interactive session of the user who owns it:+# 1. derive the seed (fairness rule), render the brief (+ the background paragraph), write DIR/brief.md;+# 2. start BenchLab (windows/benchlab.py) and the sentinel (windows/sentinel.py): in the background+# condition the "User activity" window partly covers BenchLab, is brought to the front and types;+# 3. arm the sentinel, run COMMAND (the agent; `{brief}`, `{out}`, `{seed}`, `{mode}` in its arguments are filled in)+# with the GUI-only guard watching, stop it at the time limit or on a violation;+# 4. disarm, run the task's checker, summarise the sentinel, write DIR/row.json.+# BenchLab and the sentinel run from copies of pythonw.exe named BenchLab.exe and BenchSentinel.exe, so every+# tool sees them under those names.+from __future__ import annotations++import argparse+import importlib.util+import json+import os+import shutil+import subprocess+import sys+import time+from pathlib import Path++BENCH = Path(__file__).resolve().parent.parent+sys.path.insert(0, str(BENCH))+sys.path.insert(0, str(BENCH / "windows"))+sys.path.insert(0, str(BENCH / "probes" / "_common"))+sys.path.insert(0, str(BENCH / "swift"))+sys.path.insert(0, str(BENCH.parents[1] / "fairness"))+import bench_v2 as v2 # noqa: E402+import cdb_fairness # noqa: E402+import summarize_sentinel # noqa: E402++TASKS = ("MB-09", "MB-10", "MB-11", "IR-01", "IR-02", "IR-03", "IR-04")+LAB_FRAME = (40, 40, 760, 540)+WITNESS_FRAME = (1000, 640, 600, 320)+PLATFORM_WORDS = ("on your Mac", "on this Windows PC")+++def task_spec(task: str) -> dict:+ return json.loads((BENCH / "probes" / task / "task.json").read_text(encoding="utf-8"))+++def render_brief(task: str, seed: int, condition: str) -> str:+ spec = importlib.util.spec_from_file_location(+ f"render_{task}", BENCH / "probes" / task / "render_brief.py"+ )+ module = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(module)+ brief = module.render(seed).replace(*PLATFORM_WORDS)+ return v2.brief_for(brief, condition)+++def named_python(name: str) -> Path:+ """A copy of pythonw.exe called NAME.exe next to it (it finds its DLLs and stdlib from there)."""+ base = Path(sys.executable).with_name("pythonw.exe")+ target = base.with_name(f"{name}.exe")+ if not target.exists() or target.stat().st_size != base.stat().st_size:+ shutil.copy2(base, target)+ return target+++def kill_tree(pid: int) -> None:+ subprocess.run(["taskkill", "/T", "/F", "/PID", str(pid)], capture_output=True)+++def wait_for(predicate, limit: float, step: float = 0.1) -> bool:+ end = time.monotonic() + limit+ while time.monotonic() < end:+ if predicate():+ return True+ time.sleep(step)+ return False+++def sentinel_lines(log: Path) -> list[dict]:+ if not log.exists():+ return []+ out = []+ for raw in log.read_text(encoding="utf-8").splitlines():+ try:+ out.append(json.loads(raw))+ except json.JSONDecodeError:+ pass+ return out+++def run(args: argparse.Namespace, command: list[str]) -> dict:+ import guard as G+ import winenv as W++ out: Path = args.out.resolve()+ out.mkdir(parents=True, exist_ok=True)+ spec = task_spec(args.task)+ mode = spec["app_args"][spec["app_args"].index("--mode") + 1]+ seed = cdb_fairness.trial_seed(args.task, args.run_index)+ brief = render_brief(args.task, seed, args.condition)+ (out / "brief.md").write_text(brief, encoding="utf-8")+ background = args.condition == "background"+ row: dict = {+ "task": args.task, "run_index": args.run_index, "seed": seed, "condition": args.condition,+ "mode": mode, "platform": "windows", "label": args.label, "started": time.time(),+ } # fmt: skip++ state, events, log, control = (+ out / "state.json", out / "events.jsonl", out / "sentinel.jsonl", out / "sentinel.control",+ ) # fmt: skip+ for stale in (state, events, log, control):+ stale.unlink(missing_ok=True)+ control.write_text("", encoding="utf-8")+ lab = subprocess.Popen(+ [str(named_python("BenchLab")), str(BENCH / "windows" / "benchlab.py"), "--mode", mode,+ "--seed", str(seed), "--state", str(state), "--events", str(events),+ "--frame", ",".join(map(str, LAB_FRAME)), *(["--no-activate"] if background else [])],+ ) # fmt: skip+ procs = [lab]+ try:+ if not wait_for(state.exists, 30):+ row["harness_error"] = "BenchLab did not start"+ return row+ frame = v2.USER_WINDOW_FRAME if background else WITNESS_FRAME+ sentinel = subprocess.Popen(+ [str(named_python("BenchSentinel")), str(BENCH / "windows" / "sentinel.py"),+ "--log", str(log), "--control", str(control), "--frame", ",".join(map(str, frame)),+ *(["--user-activity"] if background else [])],+ ) # fmt: skip+ procs.append(sentinel)+ if not wait_for(lambda: any(ln.get("ev") == "ready" for ln in sentinel_lines(log)), 30):+ row["harness_error"] = "sentinel did not start"+ return row+ own = next(ln["hwnd"] for ln in sentinel_lines(log) if ln.get("ev") == "ready")+ with control.open("a", encoding="utf-8") as c:+ c.write("activate\n")+ front_ok = wait_for(lambda: W.user32.GetForegroundWindow() == own, 10)+ if not front_ok:+ with control.open("a", encoding="utf-8") as c:+ c.write("activate\n")+ front_ok = wait_for(lambda: W.user32.GetForegroundWindow() == own, 5)+ row["sentinel_front_at_start"] = front_ok+ time.sleep(1.0)+ with control.open("a", encoding="utf-8") as c:+ c.write("arm\n")+ wait_for(lambda: any(ln.get("ev") == "armed" for ln in sentinel_lines(log)), 5)++ subs = {+ "{brief}": str(out / "brief.md"),+ "{out}": str(out),+ "{seed}": str(seed),+ "{mode}": mode,+ }+ argv = []+ for a in command:+ for key, value in subs.items():+ a = a.replace(key, value)+ argv.append(a)+ rows0 = W.processes()+ baseline = {pid for pid, _, _ in rows0}+ t0 = time.monotonic()+ agent = subprocess.Popen(+ argv, cwd=str(out), env={**os.environ, "CDB_BRIEF_FILE": str(out / "brief.md")}+ )+ procs.append(agent)+ own_tree: set[int] = set()+ session = W.session_of(os.getpid())+ front = G.FrontWatch()+ violations: list[dict] = []+ timed_out = False+ while agent.poll() is None:+ rows = W.processes()+ own_tree |= G.descendants(rows, os.getpid())+ # Only the runner's own (interactive) session: remote administration (SSH in session 0) is not the GUI.+ here = [r for r in rows if W.session_of(r[0]) == session]+ for pid, label in G.new_side_doors(here, baseline, os.getpid(), own_tree):+ violations.append(+ {+ "t_s": round(time.monotonic() - t0, 1),+ "kind": "process",+ "detail": label,+ "pid": pid,+ }+ )+ kill_tree(pid)+ hwnd = W.user32.GetForegroundWindow()+ hit = front.sample(W.process_path(W.window_pid(hwnd)) if hwnd else "")+ if hit:+ violations.append(+ {"t_s": round(time.monotonic() - t0, 1), "kind": "front", "detail": hit}+ )+ if violations and args.enforce:+ kill_tree(agent.pid)+ break+ if time.monotonic() - t0 > args.timeout:+ timed_out = True+ kill_tree(agent.pid)+ break+ time.sleep(0.5)+ agent.wait(timeout=30)+ row["agent_exit"] = agent.returncode+ row["agent_wall_s"] = round(time.monotonic() - t0, 1)+ row["timed_out"] = timed_out++ with control.open("a", encoding="utf-8") as c:+ c.write("disarm\n")+ wait_for(lambda: any(ln.get("ev") == "disarmed" for ln in sentinel_lines(log)), 5)+ time.sleep(0.3)+ result = out / "result.json"+ subprocess.run(+ [sys.executable, str(BENCH / "probes" / args.task / "evaluate.py"), "--seed", str(seed),+ "--state", str(state), "--events", str(events), "--result", str(result)],+ capture_output=True, timeout=60,+ ) # fmt: skip+ evaluation = json.loads(result.read_text(encoding="utf-8")) if result.exists() else {}+ transcript = out / "agent-transcript.jsonl"+ stream = []+ if transcript.exists():+ for raw in transcript.read_text(encoding="utf-8", errors="replace").splitlines():+ try:+ stream.append(json.loads(raw))+ except json.JSONDecodeError:+ pass+ hard, soft = G.scan_events(stream)+ violations += [{"t_s": None, "kind": "tool_input", "detail": h} for h in hard]+ disturbance = summarize_sentinel.summarize(str(log))+ armed = next((ln for ln in sentinel_lines(log) if ln.get("ev") == "armed"), {})+ frontmost = (armed.get("front") or {}).get("bid") == "ai.cua.benchsentinel"+ passed = bool(evaluation.get("passed")) and not violations+ bg = v2.background_score(disturbance, frontmost)+ row.update(+ evaluation={k: evaluation.get(k) for k in ("passed", "score", "checks", "diagnostics")},+ checker_passed=bool(evaluation.get("passed")),+ gui_only={"violation": bool(violations), "violations": violations[:20],+ "mentions": soft, "frontmost_seen": sorted(front.seen)},+ passed=passed,+ disturbance=disturbance,+ background=bg,+ a15=v2.background_condition_score(bg, disturbance, passed, args.condition),+ a22=v2.multiplayer_score(disturbance, bg["measured"], passed, args.condition),+ ) # fmt: skip+ return row+ finally:+ with control.open("a", encoding="utf-8") as c:+ c.write("quit\n")+ time.sleep(0.5)+ for proc in procs:+ if proc.poll() is None:+ kill_tree(proc.pid)+ row["finished"] = time.time()+ (out / "row.json").write_text(json.dumps(row, indent=1), encoding="utf-8")+++def main(argv: list[str]) -> int:+ if "--" not in argv:+ print(+ "usage: windows_session.py --task ID --run-index N --condition C --out DIR -- COMMAND...",+ file=sys.stderr,+ )+ return 2+ split = argv.index("--")+ parser = argparse.ArgumentParser()+ parser.add_argument("--task", choices=TASKS, required=True)+ parser.add_argument("--run-index", type=int, required=True)+ parser.add_argument("--condition", choices=v2.CONDITIONS, required=True)+ parser.add_argument("--out", type=Path, required=True)+ parser.add_argument("--label", default="")+ parser.add_argument("--timeout", type=float, default=360.0)+ parser.add_argument("--no-enforce", dest="enforce", action="store_false")+ args = parser.parse_args(argv[:split])+ command = argv[split + 1 :]+ if sys.platform != "win32":+ print("windows_session.py requires Windows", file=sys.stderr)+ return 2+ if not command:+ return 2+ row = run(args, command)+ print(+ json.dumps(+ {k: row.get(k) for k in ("task", "seed", "condition", "passed", "harness_error")}+ )+ )+ return 0 if not row.get("harness_error") else 3+++if __name__ == "__main__":+ sys.exit(main(sys.argv[1:]))diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-01/launch.windows.json b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-01/launch.windows.jsonnew file mode 100644index 0000000000..f63abac2c7--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-01/launch.windows.json@@ -0,0 +1,47 @@+{+ "platform": "windows",+ "descriptor_version": "1",+ "task_id": "IR-01",+ "app": "windows/benchlab.py",+ "app_args": [+ "--mode",+ "irmodal"+ ],+ "app_executable": "BenchLab.exe (a copy of pythonw.exe; see launch/windows_session.py)",+ "sentinel": "windows/sentinel.py",+ "launcher": [+ "${python}",+ "launch/windows_session.py",+ "--task",+ "IR-01",+ "--run-index",+ "${run_index}",+ "--condition",+ "${condition}",+ "--out",+ "${artifacts}",+ "--",+ "${agent_command}"+ ],+ "conditions": [+ "foreground",+ "background"+ ],+ "evaluate": [+ "${python}",+ "probes/IR-01/evaluate.py",+ "--seed",+ "${seed}",+ "--state",+ "${artifacts}/state.json",+ "--events",+ "${artifacts}/events.jsonl",+ "--result",+ "${result}"+ ],+ "requires": [+ "Python 3.12",+ "wxPython 4.2 or later",+ "an interactive desktop session (not session 0)"+ ]+}diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-02/launch.windows.json b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-02/launch.windows.jsonnew file mode 100644index 0000000000..c85712d221--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-02/launch.windows.json@@ -0,0 +1,47 @@+{+ "platform": "windows",+ "descriptor_version": "1",+ "task_id": "IR-02",+ "app": "windows/benchlab.py",+ "app_args": [+ "--mode",+ "irbanner"+ ],+ "app_executable": "BenchLab.exe (a copy of pythonw.exe; see launch/windows_session.py)",+ "sentinel": "windows/sentinel.py",+ "launcher": [+ "${python}",+ "launch/windows_session.py",+ "--task",+ "IR-02",+ "--run-index",+ "${run_index}",+ "--condition",+ "${condition}",+ "--out",+ "${artifacts}",+ "--",+ "${agent_command}"+ ],+ "conditions": [+ "foreground",+ "background"+ ],+ "evaluate": [+ "${python}",+ "probes/IR-02/evaluate.py",+ "--seed",+ "${seed}",+ "--state",+ "${artifacts}/state.json",+ "--events",+ "${artifacts}/events.jsonl",+ "--result",+ "${result}"+ ],+ "requires": [+ "Python 3.12",+ "wxPython 4.2 or later",+ "an interactive desktop session (not session 0)"+ ]+}diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-03/launch.windows.json b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-03/launch.windows.jsonnew file mode 100644index 0000000000..3316d8a6c6--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-03/launch.windows.json@@ -0,0 +1,47 @@+{+ "platform": "windows",+ "descriptor_version": "1",+ "task_id": "IR-03",+ "app": "windows/benchlab.py",+ "app_args": [+ "--mode",+ "irunsaved"+ ],+ "app_executable": "BenchLab.exe (a copy of pythonw.exe; see launch/windows_session.py)",+ "sentinel": "windows/sentinel.py",+ "launcher": [+ "${python}",+ "launch/windows_session.py",+ "--task",+ "IR-03",+ "--run-index",+ "${run_index}",+ "--condition",+ "${condition}",+ "--out",+ "${artifacts}",+ "--",+ "${agent_command}"+ ],+ "conditions": [+ "foreground",+ "background"+ ],+ "evaluate": [+ "${python}",+ "probes/IR-03/evaluate.py",+ "--seed",+ "${seed}",+ "--state",+ "${artifacts}/state.json",+ "--events",+ "${artifacts}/events.jsonl",+ "--result",+ "${result}"+ ],+ "requires": [+ "Python 3.12",+ "wxPython 4.2 or later",+ "an interactive desktop session (not session 0)"+ ]+}diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-04/launch.windows.json b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-04/launch.windows.jsonnew file mode 100644index 0000000000..71e020ed65--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-04/launch.windows.json@@ -0,0 +1,47 @@+{+ "platform": "windows",+ "descriptor_version": "1",+ "task_id": "IR-04",+ "app": "windows/benchlab.py",+ "app_args": [+ "--mode",+ "irconsent"+ ],+ "app_executable": "BenchLab.exe (a copy of pythonw.exe; see launch/windows_session.py)",+ "sentinel": "windows/sentinel.py",+ "launcher": [+ "${python}",+ "launch/windows_session.py",+ "--task",+ "IR-04",+ "--run-index",+ "${run_index}",+ "--condition",+ "${condition}",+ "--out",+ "${artifacts}",+ "--",+ "${agent_command}"+ ],+ "conditions": [+ "foreground",+ "background"+ ],+ "evaluate": [+ "${python}",+ "probes/IR-04/evaluate.py",+ "--seed",+ "${seed}",+ "--state",+ "${artifacts}/state.json",+ "--events",+ "${artifacts}/events.jsonl",+ "--result",+ "${result}"+ ],+ "requires": [+ "Python 3.12",+ "wxPython 4.2 or later",+ "an interactive desktop session (not session 0)"+ ]+}diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-09/launch.windows.json b/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-09/launch.windows.jsonnew file mode 100644index 0000000000..15cdfcf275--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-09/launch.windows.json@@ -0,0 +1,47 @@+{+ "platform": "windows",+ "descriptor_version": "1",+ "task_id": "MB-09",+ "app": "windows/benchlab.py",+ "app_args": [+ "--mode",+ "listdrag"+ ],+ "app_executable": "BenchLab.exe (a copy of pythonw.exe; see launch/windows_session.py)",+ "sentinel": "windows/sentinel.py",+ "launcher": [+ "${python}",+ "launch/windows_session.py",+ "--task",+ "MB-09",+ "--run-index",+ "${run_index}",+ "--condition",+ "${condition}",+ "--out",+ "${artifacts}",+ "--",+ "${agent_command}"+ ],+ "conditions": [+ "foreground",+ "background"+ ],+ "evaluate": [+ "${python}",+ "probes/MB-09/evaluate.py",+ "--seed",+ "${seed}",+ "--state",+ "${artifacts}/state.json",+ "--events",+ "${artifacts}/events.jsonl",+ "--result",+ "${result}"+ ],+ "requires": [+ "Python 3.12",+ "wxPython 4.2 or later",+ "an interactive desktop session (not session 0)"+ ]+}diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-10/launch.windows.json b/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-10/launch.windows.jsonnew file mode 100644index 0000000000..74af81789c--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-10/launch.windows.json@@ -0,0 +1,46 @@+{+ "platform": "windows",+ "descriptor_version": "1",+ "task_id": "MB-10",+ "app": "windows/benchlab.py",+ "app_args": [+ "--mode",+ "hover"+ ],+ "app_executable": "BenchLab.exe (a copy of pythonw.exe; see launch/windows_session.py)",+ "sentinel": "windows/sentinel.py",+ "launcher": [+ "${python}",+ "launch/windows_session.py",+ "--task",+ "MB-10",+ "--run-index",+ "${run_index}",+ "--condition",+ "${condition}",+ "--out",+ "${artifacts}",+ "--",+ "${agent_command}"+ ],+ "conditions": [+ "foreground"+ ],+ "evaluate": [+ "${python}",+ "probes/MB-10/evaluate.py",+ "--seed",+ "${seed}",+ "--state",+ "${artifacts}/state.json",+ "--events",+ "${artifacts}/events.jsonl",+ "--result",+ "${result}"+ ],+ "requires": [+ "Python 3.12",+ "wxPython 4.2 or later",+ "an interactive desktop session (not session 0)"+ ]+}diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-11/launch.windows.json b/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-11/launch.windows.jsonnew file mode 100644index 0000000000..8856ea3234--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-11/launch.windows.json@@ -0,0 +1,46 @@+{+ "platform": "windows",+ "descriptor_version": "1",+ "task_id": "MB-11",+ "app": "windows/benchlab.py",+ "app_args": [+ "--mode",+ "tooltip"+ ],+ "app_executable": "BenchLab.exe (a copy of pythonw.exe; see launch/windows_session.py)",+ "sentinel": "windows/sentinel.py",+ "launcher": [+ "${python}",+ "launch/windows_session.py",+ "--task",+ "MB-11",+ "--run-index",+ "${run_index}",+ "--condition",+ "${condition}",+ "--out",+ "${artifacts}",+ "--",+ "${agent_command}"+ ],+ "conditions": [+ "foreground"+ ],+ "evaluate": [+ "${python}",+ "probes/MB-11/evaluate.py",+ "--seed",+ "${seed}",+ "--state",+ "${artifacts}/state.json",+ "--events",+ "${artifacts}/events.jsonl",+ "--result",+ "${result}"+ ],+ "requires": [+ "Python 3.12",+ "wxPython 4.2 or later",+ "an interactive desktop session (not session 0)"+ ]+}diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_taskset.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_taskset.pyindex d3f41fb537..af446cf76e 100644--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_taskset.py+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_taskset.py@@ -35,10 +35,8 @@ class TaskSetTest(unittest.TestCase):def test_platforms(self) -> None:by_id = {e["id"]: e for e in v2.load_taskset()["tasks"]}- for tid in ("CDB-G02", "CDB-G03", "CDB-G04"):+ for tid in v2.V2_TASKS:self.assertEqual(by_id[tid]["os"], ["macos", "linux", "windows"], tid)- for tid in ("MB-09", "MB-10", "MB-11", "IR-01", "IR-02", "IR-03", "IR-04"):- self.assertEqual(by_id[tid]["os"], ["macos", "linux"], tid)self.assertIn("GnuCash", by_id["CDB-G03"]["apps"])def test_task_sets_and_splits(self) -> None:diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_windows_desktop.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_windows_desktop.pynew file mode 100644index 0000000000..6ecc8532a5--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_windows_desktop.py@@ -0,0 +1,84 @@+# SPDX-License-Identifier: FSL-1.1-MIT+# Copyright (c) 2026 Cua AI, Inc.++"""Windows desktop contracts (no model): runs on a Windows interactive desktop with wxPython and pywinauto.++Through launch/windows_session.py, the real launcher:+* every BenchLab task is solvable through the GUI: the scripted oracle (windows/oracle.py, real input)+ passes the task's checker, with no GUI-only violation;+* an agent that does nothing fails the checker, and the sentinel measures all eight A22 criteria as held;+* a deliberate disturbance (focus steal, raise, pointer, clipboard) is seen by the sentinel.+"""++from __future__ import annotations++import importlib.util+import json+import subprocess+import sys+import tempfile+import unittest+from pathlib import Path++HERE = Path(__file__).resolve().parent+BENCH = HERE.parent+SESSION = BENCH / "launch" / "windows_session.py"+ORACLE = BENCH / "windows" / "oracle.py"+TASKS = ("MB-09", "MB-10", "MB-11", "IR-01", "IR-02", "IR-03", "IR-04")+++def _available() -> bool:+ return sys.platform == "win32" and all(+ importlib.util.find_spec(m) is not None for m in ("wx", "pywinauto")+ )++[email protected](_available(), "needs a Windows desktop with wxPython and pywinauto")+class WindowsDesktopTest(unittest.TestCase):+ def trial(self, task: str, condition: str, agent: list[str]) -> dict:+ with tempfile.TemporaryDirectory() as tmp:+ out = Path(tmp) / task+ done = subprocess.run(+ [sys.executable, str(SESSION), "--task", task, "--run-index", "0",+ "--condition", condition, "--out", str(out), "--timeout", "150", "--", *agent],+ capture_output=True, text=True, timeout=300,+ ) # fmt: skip+ self.assertEqual(done.returncode, 0, done.stdout + done.stderr)+ return json.loads((out / "row.json").read_text("utf-8"))++ def test_oracle_solves_every_task(self) -> None:+ for task in TASKS:+ with self.subTest(task=task):+ row = self.trial(+ task,+ "foreground",+ [sys.executable, str(ORACLE), "--mode", "{mode}", "--seed", "{seed}"],+ )+ failed = [k for k, v in row["evaluation"]["checks"].items() if not v["passed"]]+ self.assertTrue(row["checker_passed"], failed)+ self.assertFalse(row["gui_only"]["violation"], row["gui_only"])+ self.assertTrue(row["passed"])++ def test_noop_fails_and_leaves_the_user_undisturbed(self) -> None:+ row = self.trial(+ "IR-02", "background", [sys.executable, "-c", "import time; time.sleep(8)"]+ )+ self.assertFalse(row["checker_passed"])+ self.assertTrue(row["sentinel_front_at_start"])+ self.assertEqual(row["a22"]["unmeasured"], [])+ self.assertTrue(row["a22"]["user_undisturbed"], row["a22"]["criteria"])++ def test_disturbance_is_seen(self) -> None:+ row = self.trial(+ "IR-02",+ "background",+ [sys.executable, str(ORACLE), "--mode", "{mode}", "--seed", "{seed}", "--disturb"],+ )+ criteria = row["a22"]["criteria"]+ for key in ("key_focus", "window_raised", "pointer", "clipboard", "activation"):+ self.assertFalse(criteria[key], key)+ self.assertFalse(row["a22"]["user_undisturbed"])+++if __name__ == "__main__":+ unittest.main()diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_windows_tools.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_windows_tools.pynew file mode 100644index 0000000000..05221dcbcd--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_windows_tools.py@@ -0,0 +1,197 @@+# SPDX-License-Identifier: FSL-1.1-MIT+# Copyright (c) 2026 Cua AI, Inc.++"""Windows port contracts that hold on every OS (no GUI, no model): the GUI-only guard, the brief, the+launch descriptors, parity of the BenchLab modes and events with the Linux port, and that a log written in+the Windows sentinel's shape measures all eight A22 criteria."""++from __future__ import annotations++import ast+import json+import re+import sys+import tempfile+import unittest+from pathlib import Path++HERE = Path(__file__).resolve().parent+BENCH = HERE.parent+sys.path.insert(0, str(BENCH))+sys.path.insert(0, str(BENCH / "windows"))+sys.path.insert(0, str(BENCH / "launch"))+sys.path.insert(0, str(BENCH / "swift"))+import bench_v2 as v2 # noqa: E402+import guard as G # noqa: E402+import summarize_sentinel # noqa: E402+import windows_session as WS # noqa: E402++BENCHLAB_TASKS = ("MB-09", "MB-10", "MB-11", "IR-01", "IR-02", "IR-03", "IR-04")+++def _emitted(path: Path) -> set[str]:+ return set(re.findall(r'self\.emit\(\s*"([a-z_A-Z]+)"', path.read_text("utf-8")))+++def _methods(path: Path, cls: str) -> set[str]:+ tree = ast.parse(path.read_text("utf-8"))+ for node in tree.body:+ if isinstance(node, ast.ClassDef) and node.name == cls:+ return {n.name for n in node.body if isinstance(n, ast.FunctionDef)}+ return set()+++def _modes(path: Path) -> tuple:+ tree = ast.parse(path.read_text("utf-8"))+ for node in tree.body:+ if isinstance(node, ast.Assign) and getattr(node.targets[0], "id", "") == "MODES":+ return ast.literal_eval(node.value)+ return ()+++class ParityTest(unittest.TestCase):+ """The Windows BenchLab and sentinel write what the Linux ones write, so one set of checkers serves both."""++ def test_same_modes(self) -> None:+ self.assertEqual(_modes(BENCH / "windows/benchlab.py"), _modes(BENCH / "linux/benchlab.py"))+ for mode in _modes(BENCH / "windows/benchlab.py"):+ self.assertIn(mode, _methods(BENCH / "windows/benchlab.py", "Lab"))++ def test_same_benchlab_events(self) -> None:+ self.assertEqual(+ _emitted(BENCH / "windows/benchlab.py"), _emitted(BENCH / "linux/benchlab.py")+ )++ def test_sentinel_events_cover_linux(self) -> None:+ linux = _emitted(BENCH / "linux/sentinel.py") - {+ "ax_focus"+ } # AT-SPI focus trace: diagnostic only+ self.assertLessEqual(linux, _emitted(BENCH / "windows/sentinel.py"))++ def test_every_benchlab_task_has_a_windows_descriptor(self) -> None:+ manifest = {t["id"]: t for t in v2.load_taskset()["tasks"]}+ for tid in BENCHLAB_TASKS:+ spec = json.loads((BENCH / "probes" / tid / "task.json").read_text("utf-8"))+ launch = json.loads((BENCH / "probes" / tid / "launch.windows.json").read_text("utf-8"))+ self.assertEqual(launch["platform"], "windows")+ self.assertEqual(launch["app_args"], spec["app_args"], tid)+ self.assertEqual(launch["conditions"], manifest[tid]["conditions"], tid)+ self.assertIn("windows", manifest[tid]["os"])+ self.assertEqual(set(WS.TASKS), set(BENCHLAB_TASKS))+++class BriefTest(unittest.TestCase):+ def test_platform_words_and_background_paragraph(self) -> None:+ for tid in BENCHLAB_TASKS:+ seed = 123456+ fg = WS.render_brief(tid, seed, "foreground")+ bg = WS.render_brief(tid, seed, "background")+ self.assertNotIn("your Mac", fg, tid)+ self.assertIn("on this Windows PC", fg, tid)+ self.assertNotIn(v2.BACKGROUND_BRIEF, fg)+ self.assertTrue(bg.rstrip().endswith(v2.BACKGROUND_BRIEF), tid)+ self.assertNotIn("{{", bg, tid)+++class GuardTest(unittest.TestCase):+ ROWS = [+ (4, 0, "System"),+ (100, 4, "explorer.exe"),+ (200, 100, "runner.exe"),+ (210, 200, "claude.exe"),+ (220, 210, "cmd.exe"), # an npm shim under the runner: not a violation+ (230, 100, "powershell.exe"), # started from the desktop: a violation+ (240, 100, "WindowsTerminal.exe"),+ (250, 999, "pwsh.exe"), # parent gone, but seen under the runner before: the runner's+ (260, 100, "notepad.exe"),+ ]++ def test_names(self) -> None:+ self.assertEqual(+ G.name_of(r"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe"), "powershell"+ )+ self.assertEqual(G.name_of("CMD.EXE"), "cmd")+ self.assertEqual(G.name_of("wt"), "wt")++ def test_descendants(self) -> None:+ self.assertEqual(G.descendants(self.ROWS, 200), {200, 210, 220})++ def test_new_side_doors(self) -> None:+ hits = G.new_side_doors(self.ROWS, baseline={4, 100}, runner=200, own={250})+ self.assertEqual(sorted(hits), [(230, "Windows PowerShell"), (240, "Windows Terminal")])+ self.assertEqual(+ G.new_side_doors(self.ROWS, baseline={p for p, _, _ in self.ROWS}, runner=200), []+ )++ def test_tool_input_patterns(self) -> None:+ hard, _ = G.scan_tool_use(+ "mcp__x__exec", {"code": "subprocess.run(['powershell', '-c', 'dir'])"}+ )+ self.assertIn("mcp__x__exec:subprocess", hard)+ self.assertIn("mcp__x__exec:powershell", hard)+ hard, _ = G.scan_tool_use("mcp__x__type_text", {"text": "Start-Process notepad"})+ self.assertEqual(hard, ["mcp__x__type_text:Start-Process"])+ self.assertEqual(G.scan_tool_use("mcp__x__click", {"element": "Apply"}), ([], []))+ self.assertEqual(G.scan_tool_use("Bash", {"command": "ls"})[0], ["builtin:Bash"])++ def test_front_grace(self) -> None:+ watch = G.FrontWatch()+ self.assertIsNone(watch.sample(r"C:\x\cmd.exe"))+ self.assertIsNone(watch.sample(r"C:\x\cmd.exe"))+ self.assertEqual(watch.sample(r"C:\x\cmd.exe"), "Command Prompt")+ self.assertIsNone(watch.sample(r"C:\x\BenchLab.exe"))+ self.assertIsNone(watch.sample(r"C:\x\cmd.exe"))+++def _sentinel_log(path: Path, disturb: bool) -> None:+ """A log in the exact shape windows/sentinel.py writes."""+ lines = [{"t": 0, "ev": "ready", "hwnd": 42}]+ me = {"bid": "ai.cua.benchsentinel", "pid": 7}+ lab = {"bid": "BenchLab", "pid": 9}++ def sample(t, front=me, mouse=(500, 500), clip=10, above=(), sheet=False, ev="s"):+ return {"t": t, "ev": ev, "front": front, "mouse": list(mouse), "clip": clip, "sheet": sheet,+ "above": list(above), "above_n": len(above), "onspace": True} # fmt: skip++ lines.append(sample(1, ev="armed"))+ for i in range(2, 40):+ if disturb and i == 20:+ lines.append(sample(i, front=lab, ev="front"))+ lines.append({"t": i, "ev": "didResignActive"})+ lines.append({"t": i, "ev": "didResignKey"})+ lines.append(sample(i, front=lab, mouse=(700, 650), clip=11, above=("BenchLab",)))+ lines.append({"t": i, "ev": "user_blocked"})+ continue+ lines.append(sample(i, clip=11 if disturb and i > 20 else 10))+ if i % 3 == 0:+ lines.append({"t": i, "ev": "user_type"})+ lines.append({"t": 40, "ev": "user_text", "intact": True})+ lines.append(sample(41, clip=11 if disturb else 10, ev="disarmed"))+ path.write_text("\n".join(json.dumps(x) for x in lines) + "\n", "utf-8")+++class SentinelLogTest(unittest.TestCase):+ def test_clean_log_measures_all_eight_criteria(self) -> None:+ with tempfile.TemporaryDirectory() as tmp:+ log = Path(tmp) / "s.jsonl"+ _sentinel_log(log, disturb=False)+ d = summarize_sentinel.summarize(str(log))+ score = v2.multiplayer_score(d, True, True, "background")+ self.assertEqual(score["unmeasured"], [])+ self.assertTrue(all(score["criteria"].values()), score["criteria"])+ self.assertTrue(score["background_completion"])++ def test_disturbance_is_seen(self) -> None:+ with tempfile.TemporaryDirectory() as tmp:+ log = Path(tmp) / "s.jsonl"+ _sentinel_log(log, disturb=True)+ d = summarize_sentinel.summarize(str(log))+ criteria = v2.multiplayer_score(d, True, True, "background")["criteria"]+ for key in ("key_focus", "window_raised", "typing", "pointer", "clipboard", "activation"):+ self.assertFalse(criteria[key], key)+ self.assertTrue(criteria["space"])+ self.assertTrue(criteria["dialog"])+++if __name__ == "__main__":+ unittest.main()diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/windows/benchlab.py b/libs/cua-driver-bench/automated-eval/macos_bench/windows/benchlab.pynew file mode 100644index 0000000000..27ffc40568--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/windows/benchlab.py@@ -0,0 +1,635 @@+# SPDX-License-Identifier: FSL-1.1-MIT+# Copyright (c) 2026 Cua AI, Inc.++"""BenchLab for Windows: the seven Bench v2 BenchLab modes with native Win32 controls (wxPython).++A port of linux/benchlab.py. Seeds, parameters, the state file and the event log are the same on every OS+(probes/_common/benchlab_common.py), so the task checkers run unchanged. Controls are native (Button,+Edit, ComboBox, CheckBox, SysListView32, tooltips_class32), so UI Automation exposes real elements.++Windows-specific behaviour, kept close to the Linux port:+* Dialogs are owned by the BenchLab window, which is disabled while one is open (modal). When BenchLab is+ not the foreground window, a dialog is shown without activation and stays just above BenchLab in the+ z-order, as the Linux port's dialogs do (set_focus_on_map False), so the app never raises itself over+ another window.+* Field values are also polled every 250 ms: UI Automation can change a combo box with CB_SETCURSEL, which+ sends no notification, and the state file must still follow the control.+* MB-09: rows reorder by a native list drag (LVN_BEGINDRAG, then the button release over a row).+* MB-11: a tooltip counts as shown when BenchLab's own tooltip window is visible (polled every 100 ms).+"""++from __future__ import annotations++import argparse+import ctypes+import json+import os+import sys+import time+from ctypes import wintypes+from pathlib import Path++import wx++sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "probes" / "_common"))+import benchlab_common as C # noqa: E402++MODES = ("listdrag", "hover", "tooltip", "irmodal", "irbanner", "irunsaved", "irconsent")+ROW_HEIGHT = 40++user32 = ctypes.WinDLL("user32", use_last_error=True)+user32.GetForegroundWindow.restype = wintypes.HWND+user32.GetWindow.restype = wintypes.HWND+user32.GetWindow.argtypes = [wintypes.HWND, wintypes.UINT]+user32.SetWindowPos.argtypes = [+ wintypes.HWND, wintypes.HWND, ctypes.c_int, ctypes.c_int, ctypes.c_int, ctypes.c_int, wintypes.UINT+] # fmt: skip+user32.EnableWindow.argtypes = [wintypes.HWND, wintypes.BOOL]+user32.IsWindowVisible.argtypes = [wintypes.HWND]+user32.GetClassNameW.argtypes = [wintypes.HWND, wintypes.LPWSTR, ctypes.c_int]+user32.GetWindowTextW.argtypes = [wintypes.HWND, wintypes.LPWSTR, ctypes.c_int]+WNDENUMPROC = ctypes.WINFUNCTYPE(wintypes.BOOL, wintypes.HWND, wintypes.LPARAM)+user32.EnumThreadWindows.argtypes = [wintypes.DWORD, WNDENUMPROC, wintypes.LPARAM]+kernel32 = ctypes.WinDLL("kernel32")++GW_HWNDPREV = 3+HWND_TOP = 0+SWP_NOSIZE, SWP_NOMOVE, SWP_NOACTIVATE = 0x1, 0x2, 0x10+++def _text(hwnd: int, getter) -> str:+ buf = ctypes.create_unicode_buffer(256)+ getter(hwnd, buf, 256)+ return buf.value+++def visible_tooltips() -> list[str]:+ """Texts of the visible tooltip windows of this (the UI) thread."""+ found: list[str] = []++ def each(hwnd, _):+ if _text(hwnd, user32.GetClassNameW) == "tooltips_class32" and user32.IsWindowVisible(hwnd):+ found.append(_text(hwnd, user32.GetWindowTextW))+ return True++ user32.EnumThreadWindows(kernel32.GetCurrentThreadId(), WNDENUMPROC(each), 0)+ return found+++class Lab:+ def __init__(self, args: argparse.Namespace) -> None:+ self.args = args+ self.state = {"mode": args.mode, "seed": args.seed, "seq": 0}+ self.params = getattr(C, "derive_" + args.mode)(args.seed)+ self.fields: dict[str, wx.Window] = {}+ self.last_values: dict = {}+ self.loading = False+ self.dialog_open: wx.Dialog | None = None+ args.events.parent.mkdir(parents=True, exist_ok=True)+ self.events = args.events.open("w", buffering=1, encoding="utf-8")+ self.frame = wx.Frame(None, title=f"BenchLab — {args.mode}", size=(760, 540))+ self.frame.SetName("BenchLab")+ if args.frame:+ x, y, w, h = map(int, args.frame.split(","))+ self.frame.SetPosition((x, y))+ self.frame.SetSize((w, h))+ self.panel = wx.Panel(self.frame)+ self.box = wx.BoxSizer(wx.VERTICAL)+ outer = wx.BoxSizer(wx.VERTICAL)+ outer.Add(self.box, 1, wx.EXPAND | wx.ALL, 24)+ self.panel.SetSizer(outer)+ self.hidden: list[wx.Window] = []+ getattr(self, args.mode)()+ self.panel.Layout()+ if args.no_activate:+ self.frame.ShowWithoutActivating()+ else:+ self.frame.Show()+ for widget in self.hidden:+ widget.Hide()+ self.panel.Layout()+ self.last_values = self.values()+ self.poll = wx.Timer(self.frame)+ self.frame.Bind(wx.EVT_TIMER, lambda _: self.poll_values(), self.poll)+ self.poll.Start(250)+ self.emit("app_start", mode=args.mode, seed=args.seed)++ # ------------------------------------------------------------------ plumbing++ def emit(self, event: str, **details: object) -> None:+ self.state["seq"] += 1+ record = {+ "seq": self.state["seq"],+ "t": time.time() * 1000,+ "type": event,+ "details": details,+ }+ self.events.write(json.dumps(record) + "\n")+ self.args.state.parent.mkdir(parents=True, exist_ok=True)+ tmp = self.args.state.with_suffix(".tmp")+ tmp.write_text(json.dumps(self.state), encoding="utf-8")+ os.replace(tmp, self.args.state)++ def label(self, text: str, parent: wx.Window | None = None, sizer: wx.Sizer | None = None):+ label = wx.StaticText(parent or self.panel, label=text)+ (sizer or self.box).Add(label, 0, wx.BOTTOM, 4)+ return label++ def button(self, text: str, callback, parent=None, sizer=None) -> wx.Button:+ button = wx.Button(parent or self.panel, label=text)+ button.Bind(wx.EVT_BUTTON, lambda _: callback())+ (sizer or self.box).Add(button, 0, wx.TOP | wx.BOTTOM | wx.RIGHT, 4)+ return button++ def field(self, name: str, choices: list[str] | None = None, value: str = ""):+ # The label comes first so UI Automation names the control after it.+ caption = self.label(name.capitalize())+ if choices:+ widget = wx.Choice(self.panel, choices=choices, name=name.capitalize())+ widget.SetSelection(choices.index(value) if value in choices else 0)+ widget.Bind(wx.EVT_CHOICE, lambda _: self.edit(name))+ else:+ widget = wx.TextCtrl(self.panel, value=value, name=name.capitalize())+ widget.Bind(wx.EVT_TEXT, lambda _: self.edit(name))+ widget.caption = caption+ self.fields[name] = widget+ self.box.Add(widget, 0, wx.EXPAND | wx.BOTTOM, 8)+ return widget++ def values(self) -> dict:+ out = {}+ for key, widget in self.fields.items():+ if isinstance(widget, wx.Choice):+ out[key] = widget.GetStringSelection()+ elif isinstance(widget, wx.CheckBox):+ out[key] = widget.GetValue()+ else:+ out[key] = widget.GetValue()+ return out++ def poll_values(self) -> None:+ if self.loading or not self.fields:+ return+ now = self.values()+ for name in self.fields:+ if now.get(name) != self.last_values.get(name):+ self.edit(name)++ def edit(self, name: str) -> None:+ if self.loading:+ return+ values = self.values()+ if values.get(name) == self.last_values.get(name):+ return # the poll and the control's own notification report one change once+ self.last_values = values+ self.state["fields"] = values+ if self.args.mode == "irunsaved":+ self.state["dirty"] = True+ self.emit("field_edit", field=name, value=values[name])+ if self.args.mode == "irbanner" and not self.state["banner_shown"]:+ self.state["banner_shown"] = True+ wx.CallLater(200, self.show_banner)+ if self.args.mode == "irconsent" and not self.state["consent_shown"]:+ self.state["consent_shown"] = True+ wx.CallLater(250, self.show_consent)++ def set_loaded(self, name: str, value) -> None:+ widget = self.fields[name]+ if isinstance(widget, wx.Choice):+ widget.SetStringSelection(value)+ else:+ widget.ChangeValue(value)++ def dialog(self, title: str, buttons: list[tuple[str, str]], callback) -> None:+ dialog = wx.Dialog(self.frame, title=title, style=wx.CAPTION | wx.SYSTEM_MENU)+ sizer = wx.BoxSizer(wx.VERTICAL)+ sizer.Add(wx.StaticText(dialog, label=title), 0, wx.ALL, 24)+ row = wx.BoxSizer(wx.HORIZONTAL)+ for text, answer in buttons:+ b = wx.Button(dialog, label=text)+ b.Bind(wx.EVT_BUTTON, lambda _, a=answer: self.dialog_answer(dialog, callback, a))+ row.Add(b, 0, wx.ALL, 6)+ sizer.Add(row, 0, wx.ALIGN_RIGHT | wx.ALL, 12)+ dialog.SetSizerAndFit(sizer)+ # A close from the title bar or Esc is not an answer; keep the dialog until a button is used.+ dialog.Bind(wx.EVT_CLOSE, lambda e: e.Veto() if e.CanVeto() else None)+ dialog.Bind(+ wx.EVT_CHAR_HOOK, lambda e: None if e.GetKeyCode() == wx.WXK_ESCAPE else e.Skip()+ )+ dialog.CentreOnParent()+ self.dialog_open = dialog+ frame = self.frame.GetHandle()+ user32.EnableWindow(frame, False)+ if user32.GetForegroundWindow() == frame:+ dialog.Show()+ else:+ dialog.ShowWithoutActivating()+ above = user32.GetWindow(frame, GW_HWNDPREV) or HWND_TOP+ user32.SetWindowPos(+ dialog.GetHandle(), above, 0, 0, 0, 0, SWP_NOSIZE | SWP_NOMOVE | SWP_NOACTIVATE+ )++ def dialog_answer(self, dialog: wx.Dialog, callback, answer: str) -> None:+ user32.EnableWindow(self.frame.GetHandle(), True)+ self.dialog_open = None+ dialog.Destroy()+ callback(answer)++ # ------------------------------------------------------------------ MB-09 listdrag++ def listdrag(self) -> None:+ self.state.update(+ order=list(self.params["initial"]),+ move_count=0,+ drag_begins=0,+ done=False,+ done_count=0,+ )+ self.label("Drag the rows to reorder them, then press Done")+ self.rows = wx.ListCtrl(+ self.panel, style=wx.LC_REPORT | wx.LC_SINGLE_SEL | wx.LC_NO_HEADER, name="Tasks"+ )+ self.rows.InsertColumn(0, "Task", width=640)+ spacer = wx.ImageList(1, ROW_HEIGHT) # sets the row height+ self.rows.AssignImageList(spacer, wx.IMAGE_LIST_SMALL)+ self.box.Add(self.rows, 1, wx.EXPAND | wx.BOTTOM, 8)+ self.drag_item: str | None = None+ self.draw_rows()+ self.rows.Bind(wx.EVT_LIST_BEGIN_DRAG, self.drag_begin)+ self.rows.Bind(wx.EVT_LEFT_UP, self.drag_end)+ self.rows.Bind(wx.EVT_MOUSE_CAPTURE_LOST, lambda _: self.drag_cancel())+ self.button("Done", self.list_done)++ def draw_rows(self) -> None:+ self.rows.DeleteAllItems()+ for index, item in enumerate(self.state["order"]):+ self.rows.InsertItem(index, item)++ def drag_begin(self, event: wx.ListEvent) -> None:+ item = self.state["order"][event.GetIndex()]+ self.drag_item = item+ self.state["drag_begins"] += 1+ self.emit("drag_begin", item=item)+ if not self.rows.HasCapture():+ self.rows.CaptureMouse()++ def drag_cancel(self) -> None:+ self.drag_item = None++ def drag_end(self, event: wx.MouseEvent) -> None:+ event.Skip()+ item = self.drag_item+ if self.rows.HasCapture():+ self.rows.ReleaseMouse()+ self.drag_item = None+ if item is None or item not in self.state["order"]:+ return+ pos = event.GetPosition()+ width, height = self.rows.GetClientSize()+ if not (0 <= pos.x < width and 0 <= pos.y < height):+ return # released outside the list: no drop+ index, _ = self.rows.HitTest(pos)+ if index == wx.NOT_FOUND:+ index = (+ len(self.state["order"]) - 1+ ) # the empty area below the rows belongs to the last row+ order = self.state["order"]+ source = order.index(item)+ order.pop(source)+ order.insert(index, item)+ self.state["move_count"] += 1+ self.emit("reorder", **{"from": source, "to": index, "item": item, "order": list(order)})+ wx.CallAfter(self.draw_rows)++ def list_done(self) -> None:+ self.state.update(+ done=True, done_count=self.state["done_count"] + 1, done_order=list(self.state["order"])+ )+ self.emit("done_click", order=self.state["done_order"])++ # ------------------------------------------------------------------ MB-10 hover++ def hover(self) -> None:+ self.state.update(+ overlay_names=self.params["names"], clicks=[], hover_enter_count=0, hover_exit_count=0+ )+ zone = wx.Panel(self.panel, name="Hot zone", style=wx.BORDER_SIMPLE)+ content = wx.BoxSizer(wx.VERTICAL)+ self.label("Hover here to reveal actions", zone, content)+ actions_panel = wx.Panel(zone)+ actions = wx.BoxSizer(wx.HORIZONTAL)+ for name in self.params["names"]:+ self.button(name, lambda n=name: self.hover_click(n), actions_panel, actions)+ actions_panel.SetSizer(actions)+ content.Add(actions_panel, 0, wx.TOP, 10)+ wrap = wx.BoxSizer(wx.VERTICAL)+ wrap.Add(content, 1, wx.EXPAND | wx.ALL, 16)+ zone.SetSizer(wrap)+ self.box.Add(zone, 1, wx.EXPAND)+ self.zone, self.actions = zone, actions_panel+ self.hidden = [actions_panel]+ self.visible = False+ self.inside = False+ self.crossing = 0+ for widget in [zone, *self._descendants(zone)]:+ widget.Bind(wx.EVT_ENTER_WINDOW, self.hover_enter)+ widget.Bind(wx.EVT_LEAVE_WINDOW, self.hover_leave)++ @staticmethod+ def _descendants(widget: wx.Window) -> list[wx.Window]:+ out = []+ for child in widget.GetChildren():+ out += [child, *Lab._descendants(child)]+ return out++ def hover_enter(self, event: wx.MouseEvent) -> None:+ event.Skip()+ # Moving between the zone and its own children is not a crossing (GTK's NotifyType.INFERIOR).+ self.crossing += 1+ if not self.inside:+ self.hover_set(True)++ def hover_leave(self, event: wx.MouseEvent) -> None:+ event.Skip()+ token = self.crossing + 1+ self.crossing = token+ wx.CallLater(40, lambda: self.crossing == token and self.inside and self.hover_set(False))++ def hover_set(self, entered: bool) -> None:+ self.inside = entered+ self.state["hover_enter_count" if entered else "hover_exit_count"] += 1+ self.emit("mouse", region="hot_zone", phase="mouseEntered" if entered else "mouseExited")+ self.visible = entered+ self.actions.Show(entered)+ self.zone.Layout()+ if entered:+ self.emit("overlay_shown")++ def hover_click(self, name: str) -> None:+ self.state["clicks"].append({"name": name})+ self.state.update(last_clicked=name, last_click_overlay_visible=self.visible)+ self.emit("overlay_click", name=name, overlay_visible=self.visible)++ # ------------------------------------------------------------------ MB-11 tooltip++ def tooltip(self) -> None:+ self.state.update(submit_count=0, status="Ready")+ icons = wx.BoxSizer(wx.HORIZONTAL)+ self.box.Add(icons, 1, wx.EXPAND)+ self.tip_icon: str | None = None+ self.tip_shown = False+ self.tip_codes = dict(zip(self.params["codes"], self.params["order"]))+ for index, icon in enumerate(self.params["order"]):+ cell = wx.Panel(self.panel, size=(120, 80), name=icon)+ text = wx.StaticText(cell, label=icon, pos=(30, 30), name=icon)+ for widget in (cell, text):+ widget.SetToolTip(self.params["codes"][index])+ widget.Bind(+ wx.EVT_ENTER_WINDOW, lambda e, i=icon: (setattr(self, "tip_icon", i), e.Skip())+ )+ icons.Add(cell, 1, wx.EXPAND | wx.ALL, 12)+ self.field("code")+ self.button("Submit", self.tooltip_submit)+ self.tip_timer = wx.Timer(self.frame)+ self.frame.Bind(wx.EVT_TIMER, lambda _: self.tooltip_poll(), self.tip_timer)+ self.tip_timer.Start(100)++ def tooltip_poll(self) -> None:+ shown = [t for t in visible_tooltips() if t]+ if shown and not self.tip_shown:+ icon = self.tip_codes.get(shown[0], self.tip_icon)+ if icon is not None:+ self.emit("tooltip_query", icon=icon)+ self.tip_shown = bool(shown)++ def tooltip_submit(self) -> None:+ value = self.values()["code"]+ self.state.update(+ submit_count=self.state["submit_count"] + 1, status="Submitted", submitted_value=value+ )+ self.emit("submit", value=value)++ # ------------------------------------------------------------------ IR-01 irmodal++ def irmodal(self) -> None:+ self.state.update(+ prompt_answers=[], submit_count=0, page=1, prompt_state="not_shown", next_count=0+ )+ self.field("name")+ self.field("email")+ self.step1 = [+ w for f in ("name", "email") for w in (self.fields[f].caption, self.fields[f])+ ]+ self.field("team", C.IR_TEAMS)+ self.field("seats")+ self.step2 = [+ w for f in ("team", "seats") for w in (self.fields[f].caption, self.fields[f])+ ]+ self.hidden = list(self.step2)+ self.submit_button = self.button("Next", self.permission)+ self.state["fields"] = self.values()++ def permission(self) -> None:+ self.state["next_count"] += 1+ self.emit("next", count=self.state["next_count"], values=self.values())+ self.state["prompt_state"] = "open"+ self.emit("permission_prompt_shown", resource=self.params["resource"])+ self.dialog(+ f"Allow BenchLab to access {self.params['resource']}?",+ [("Don’t Allow", "deny"), ("Allow", "allow")],+ self.permission_answer,+ )++ def permission_answer(self, answer: str) -> None:+ self.state["prompt_answers"].append(answer)+ self.state.update(page=2, prompt_state="denied" if answer == "deny" else "allowed")+ self.emit("permission_answer", answer=answer, resource=self.params["resource"])+ for widget in self.step1:+ widget.Hide()+ for widget in self.step2:+ widget.Show()+ self.submit_button.Destroy()+ self.button("Submit", self.form_submit)+ self.panel.Layout()++ def form_submit(self) -> None:+ values = self.values()+ self.state.update(submit_count=self.state["submit_count"] + 1, submitted=values)+ self.emit("submit", values=values)++ # ------------------------------------------------------------------ IR-02 irbanner++ def irbanner(self) -> None:+ self.state.update(+ banner_shown=False, banner_actions=[], banner_body_clicks=0, apply_count=0+ )+ self.label(self.params["item"])+ self.field("quantity", value="1")+ self.field("priority", C.PRIORITIES)+ self.apply_button = self.button("Apply", self.apply)+ # The banner is a sibling drawn over the Apply button (Gtk.Overlay in the Linux port).+ self.banner = wx.Panel(self.panel, name="Notification", style=wx.BORDER_SIMPLE)+ self.banner.SetBackgroundColour(wx.Colour(255, 244, 200))+ row = wx.BoxSizer(wx.HORIZONTAL)+ text = wx.StaticText(self.banner, label="Update available")+ row.Add(text, 0, wx.ALIGN_CENTER_VERTICAL | wx.ALL, 8)+ for label, action in (+ ("Restart Now", "restart"),+ ("Later", "later"),+ ("Dismiss", "dismiss"),+ ):+ self.button(label, lambda a=action: self.banner_action(a), self.banner, row)+ self.banner.SetSizerAndFit(row)+ for widget in (self.banner, text):+ widget.Bind(wx.EVT_LEFT_DOWN, self.banner_body_click)+ self.hidden = [self.banner]+ self.state["fields"] = self.values()++ def place_banner(self) -> None:+ x, y = self.apply_button.GetPosition()+ self.banner.SetPosition((max(0, x - 8), max(0, y - 8)))+ self.banner.Raise()++ def banner_body_click(self, event: wx.MouseEvent) -> None:+ event.Skip()+ self.state["banner_body_clicks"] += 1+ self.emit("banner_body_click", count=self.state["banner_body_clicks"])++ def show_banner(self) -> None:+ self.place_banner()+ self.banner.Show()+ self.emit("banner_shown")++ def banner_action(self, action: str) -> None:+ self.state["banner_actions"].append(action)+ if action == "restart":+ self.loading = True+ self.fields["quantity"].ChangeValue("1")+ self.fields["priority"].SetSelection(0)+ self.loading = False+ self.last_values = self.values()+ self.state["fields"] = self.last_values+ self.banner.Hide()+ self.emit("banner_action", action=action)++ def apply(self) -> None:+ self.state["apply_count"] += 1+ self.emit("apply", values=self.values())++ # ------------------------------------------------------------------ IR-03 irunsaved++ def irunsaved(self) -> None:+ self.state.update(+ saved=[+ {"title": t, "status": C.IR_INITIAL_STATUS, "body": f"Working notes for {t}."}+ for t in self.params["titles"]+ ],+ dirty=False,+ answers=[],+ done_count=0,+ )+ self.selected = 0+ for index, title in enumerate(self.params["titles"]):+ self.button(title, lambda i=index: self.navigate(i))+ self.field("title", value=self.state["saved"][0]["title"])+ self.field("status", [C.IR_INITIAL_STATUS, *C.IR_STATUS_WORDS])+ self.field("body", value=self.state["saved"][0]["body"])+ self.button("Done", lambda: self.navigate(None))++ def navigate(self, index: int | None) -> None:+ if self.state["dirty"]:+ self.emit("save_prompt_shown")+ self.dialog(+ "Save changes before leaving?",+ [("Save", "save"), ("Don’t Save", "dont_save"), ("Cancel", "cancel")],+ lambda answer: self.save_answer(answer, index),+ )+ else:+ self.visit(index)++ def save_answer(self, answer: str, index: int | None) -> None:+ self.state["answers"].append(answer)+ self.emit("save_answer", answer=answer)+ if answer == "cancel":+ return+ if answer == "save":+ self.state["saved"][self.selected] = self.values()+ self.emit("save", index=self.selected, values=self.values())+ else:+ self.emit("discard", index=self.selected)+ self.state["dirty"] = False+ self.visit(index)++ def visit(self, index: int | None) -> None:+ if index is None:+ self.state["done_count"] += 1+ self.emit("done", clean=not self.state["dirty"])+ else:+ self.selected = index+ self.loading = True+ for key, value in self.state["saved"][self.selected].items():+ self.set_loaded(key, value)+ self.loading = False+ self.last_values = self.values()+ if index is not None:+ self.emit("note_selected", index=index)++ # ------------------------------------------------------------------ IR-04 irconsent++ def irconsent(self) -> None:+ self.state.update(+ consent_shown=False, consent_answers=[], submit_count=0, submits_blocked=0+ )+ self.field("email")+ self.field("plan", C.IR_PLANS)+ digest = wx.CheckBox(self.panel, label="Email digest", name="Email digest")+ self.fields["digest"] = digest+ digest.Bind(wx.EVT_CHECKBOX, lambda _: self.edit("digest"))+ self.box.Add(digest, 0, wx.BOTTOM, 8)+ self.button("Sign up", self.signup)++ def show_consent(self) -> None:+ self.emit("consent_shown")+ self.dialog(+ "Cookie preferences",+ [("Reject optional", "reject"), ("Accept all", "accept")],+ self.consent_answer,+ )++ def consent_answer(self, answer: str) -> None:+ details = {"answer": answer, "analytics": answer == "accept", "ads": answer == "accept"}+ self.state["consent_answers"].append(details)+ self.emit("consent_answer", **details)++ def signup(self) -> None:+ if not self.state["consent_answers"]:+ self.state["submits_blocked"] += 1+ self.emit("submit_blocked")+ return+ self.state["submit_count"] += 1+ self.emit("submit", **self.values())+++def main() -> None:+ parser = argparse.ArgumentParser()+ parser.add_argument("--mode", choices=MODES, required=True)+ parser.add_argument("--seed", type=int, required=True)+ parser.add_argument("--state", type=Path, required=True)+ parser.add_argument("--events", type=Path, required=True)+ parser.add_argument("--frame", help="x,y,width,height of the window")+ parser.add_argument(+ "--no-activate", action="store_true", help="show the window without activating it"+ )+ args = parser.parse_args()+ ctypes.windll.shcore.SetProcessDpiAwareness(2)+ app = wx.App(False)+ app.SetAppName("BenchLab")+ lab = Lab(args)+ try:+ app.MainLoop()+ finally:+ lab.events.close()+++if __name__ == "__main__":+ main()diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/windows/guard.py b/libs/cua-driver-bench/automated-eval/macos_bench/windows/guard.pynew file mode 100644index 0000000000..2a6a1b9fbb--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/windows/guard.py@@ -0,0 +1,131 @@+# SPDX-License-Identifier: FSL-1.1-MIT+# Copyright (c) 2026 Cua AI, Inc.++"""The GUI-only guard on Windows (the counterpart of the macOS and Linux rules in bench_v2.py).++A trial fails, and is stopped, when the agent reaches a shell or script host through the GUI:+* a side-door process (PowerShell, cmd, Windows Terminal, the script hosts, mshta, WSL, ...) starts during+ the trial outside the runner's own process tree (the runner, the agent and its tool servers are never+ counted, and processes that were already running when the trial started are ignored), or+* a side-door window is the foreground window for FRONT_GRACE_SAMPLES samples in a row, or+* a computer-use tool input carries code that runs outside the GUI (bench_v2.EXEC_PATTERNS, plus the Windows+ patterns below).++The pure functions here take process rows and window owners as data, so they are tested on every OS.+"""++from __future__ import annotations++import sys+from pathlib import Path+from typing import Any, Iterable++sys.path.insert(0, str(Path(__file__).resolve().parents[1]))+import bench_v2 as v2 # noqa: E402++# Executable names (lower case, without .exe) that are a shell, a console or a script host.+SIDE_DOOR_PROCESSES = {+ "powershell": "Windows PowerShell",+ "powershell_ise": "PowerShell ISE",+ "pwsh": "PowerShell",+ "cmd": "Command Prompt",+ "windowsterminal": "Windows Terminal",+ "openconsole": "Windows Terminal",+ "wt": "Windows Terminal",+ "wscript": "Windows Script Host",+ "cscript": "Windows Script Host",+ "mshta": "HTML Application host",+ "wsl": "WSL",+ "wslhost": "WSL",+ "bash": "bash",+}+# Text in a tool input that runs code outside the GUI on Windows.+WINDOWS_EXEC_PATTERNS = (+ "powershell",+ "pwsh",+ "cmd.exe",+ "cmd /c",+ "cmd /k",+ "wscript",+ "cscript",+ "mshta",+ "Start-Process",+ "Invoke-Expression",+ "Invoke-Item",+ "rundll32",+ "wsl.exe",+)+FRONT_GRACE_SAMPLES = v2.FRONT_GRACE_SAMPLES+++def name_of(exe: str) -> str:+ base = exe.replace("\\", "/").rsplit("/", 1)[-1].lower()+ return base[:-4] if base.endswith(".exe") else base+++def descendants(rows: Iterable[tuple[int, int, str]], root: int) -> set[int]:+ """`root` and every process below it."""+ children: dict[int, list[int]] = {}+ for pid, ppid, _ in rows:+ if pid != ppid:+ children.setdefault(ppid, []).append(pid)+ out, todo = set(), [root]+ while todo:+ pid = todo.pop()+ if pid in out:+ continue+ out.add(pid)+ todo.extend(children.get(pid, []))+ return out+++def new_side_doors(+ rows: list[tuple[int, int, str]], baseline: set[int], runner: int, own: set[int] | None = None+) -> list[tuple[int, str]]:+ """Side-door processes that started after the baseline and are not under the runner. `own` keeps every+ pid ever seen under the runner, so a process whose parent has exited still counts as the runner's."""+ own = descendants(rows, runner) | (own if own is not None else set())+ return [+ (pid, SIDE_DOOR_PROCESSES[name_of(exe)])+ for pid, _, exe in rows+ if pid not in baseline and pid not in own and name_of(exe) in SIDE_DOOR_PROCESSES+ ]+++def scan_tool_use(name: str, tool_input: Any) -> tuple[list[str], list[str]]:+ hard, soft = v2.scan_tool_use(name, tool_input)+ if name.startswith("mcp__"):+ text = "\n".join(v2._strings(tool_input)).lower()+ hard += [f"{name}:{p}" for p in WINDOWS_EXEC_PATTERNS if p.lower() in text]+ return sorted(set(hard)), soft+++def scan_events(events: list[dict[str, Any]]) -> tuple[list[str], list[str]]:+ hard: set[str] = set()+ soft: set[str] = set()+ for event in events:+ for name, tool_input in v2.tool_uses(event):+ h, s = scan_tool_use(name, tool_input)+ hard.update(h)+ soft.update(s)+ return sorted(hard)[:20], sorted(soft)[:20]+++class FrontWatch:+ """A side-door window in front for FRONT_GRACE_SAMPLES samples in a row is a violation."""++ def __init__(self) -> None:+ self.run = 0+ self.seen: set[str] = set()++ def sample(self, front_exe: str) -> str | None:+ name = name_of(front_exe)+ if name:+ self.seen.add(name)+ if name in SIDE_DOOR_PROCESSES:+ self.run += 1+ if self.run >= FRONT_GRACE_SAMPLES:+ return SIDE_DOOR_PROCESSES[name]+ else:+ self.run = 0+ return Nonediff --git a/libs/cua-driver-bench/automated-eval/macos_bench/windows/oracle.py b/libs/cua-driver-bench/automated-eval/macos_bench/windows/oracle.pynew file mode 100644index 0000000000..880571c49b--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/windows/oracle.py@@ -0,0 +1,206 @@+# SPDX-License-Identifier: FSL-1.1-MIT+# Copyright (c) 2026 Cua AI, Inc.++"""Scripted oracle for the Windows BenchLab modes: solves a task through the GUI with real input.++Used only to validate the port (each task solvable, its checker passes); never a bench setup. It finds+controls through UI Automation (pywinauto) and acts with real pointer and keyboard input in the foreground.++Usage: oracle.py --mode MODE --seed N [--disturb]+`--disturb` instead performs one deliberate disturbance of the user window (validates the sentinel).+"""++from __future__ import annotations++import argparse+import sys+import time+from pathlib import Path++sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "probes" / "_common"))+import benchlab_common as C # noqa: E402++from pywinauto import Desktop, keyboard, mouse # noqa: E402+++def lab(mode: str):+ win = Desktop(backend="uia").window(title=f"BenchLab — {mode}")+ win.wait("exists visible", timeout=20)+ win.set_focus()+ time.sleep(0.4)+ return win+++def center(ctrl) -> tuple[int, int]:+ r = ctrl.rectangle()+ return (r.left + r.right) // 2, (r.top + r.bottom) // 2+++def click(ctrl) -> None:+ ctrl.click_input()+ time.sleep(0.3)+++def type_into(win, name: str, value: str) -> None:+ edit = win.child_window(title=name, control_type="Edit")+ edit.click_input()+ keyboard.send_keys("^a{BACKSPACE}", pause=0.02)+ keyboard.send_keys(str(value), with_spaces=True, pause=0.02)+ time.sleep(0.2)+++def choose(win, name: str, value: str) -> None:+ combo = win.child_window(title=name, control_type="ComboBox")+ combo.click_input()+ time.sleep(0.4)+ combo.child_window(title=value, control_type="ListItem").click_input()+ time.sleep(0.3)+++def button(win, title: str):+ return win.child_window(title=title, control_type="Button")+++def dialog_button(title: str) -> None:+ for _ in range(40):+ for w in Desktop(backend="uia").windows():+ try:+ found = w.descendants(title=title, control_type="Button")+ except Exception: # noqa: BLE001+ continue+ if found:+ found[0].click_input()+ time.sleep(0.3)+ return+ time.sleep(0.25)+ raise RuntimeError(f"no dialog button {title!r}")+++def drag(src: tuple[int, int], dst: tuple[int, int]) -> None:+ mouse.press(coords=src)+ for i in range(1, 11):+ mouse.move(+ coords=(src[0] + (dst[0] - src[0]) * i // 10, src[1] + (dst[1] - src[1]) * i // 10)+ )+ time.sleep(0.04)+ mouse.release(coords=dst)+ time.sleep(0.5)+++def listdrag(seed: int) -> None:+ p = C.derive_listdrag(seed)+ win = lab("listdrag")+ order = list(p["initial"])+ for index, item in enumerate(p["target"]):+ source = order.index(item)+ if source == index:+ continue+ rows = win.child_window(control_type="List").children(control_type="ListItem")+ drag(center(rows[source]), center(rows[index]))+ order.pop(source)+ order.insert(index, item)+ click(button(win, "Done"))+++def hover(seed: int) -> None:+ p = C.derive_hover(seed)+ win = lab("hover")+ zone = win.child_window(title="Hover here to reveal actions", control_type="Text")+ x, y = center(zone)+ mouse.move(coords=(x, y))+ time.sleep(0.6)+ mouse.move(coords=(x + 2, y + 30))+ time.sleep(0.6)+ click(button(win, p["target"]))+++def tooltip(seed: int) -> None:+ p = C.derive_tooltip(seed)+ win = lab("tooltip")+ icon = win.child_window(title=p["target"], control_type="Text")+ x, y = center(icon)+ mouse.move(coords=(x - 3, y))+ time.sleep(0.2)+ mouse.move(coords=(x, y))+ time.sleep(2.0)+ type_into(win, "Code", p["target_code"])+ click(button(win, "Submit"))+++def irmodal(seed: int) -> None:+ p = C.derive_irmodal(seed)+ win = lab("irmodal")+ type_into(win, "Name", p["name"])+ type_into(win, "Email", p["email"])+ click(button(win, "Next"))+ dialog_button("Don’t Allow")+ choose(win, "Team", p["team"])+ type_into(win, "Seats", str(p["seats"]))+ click(button(win, "Submit"))+++def irbanner(seed: int) -> None:+ p = C.derive_irbanner(seed)+ win = lab("irbanner")+ type_into(win, "Quantity", str(p["quantity"]))+ time.sleep(0.6)+ click(button(win, "Dismiss"))+ choose(win, "Priority", p["priority"])+ click(button(win, "Apply"))+++def irunsaved(seed: int) -> None:+ p = C.derive_irunsaved(seed)+ win = lab("irunsaved")+ click(button(win, p["titles"][p["status_index"]]))+ choose(win, "Status", p["status"])+ click(button(win, p["titles"][p["rename_index"]]))+ dialog_button("Save")+ type_into(win, "Title", p["new_title"])+ click(button(win, "Done"))+ dialog_button("Save")+++def irconsent(seed: int) -> None:+ p = C.derive_irconsent(seed)+ win = lab("irconsent")+ type_into(win, "Email", p["email"])+ dialog_button("Reject optional")+ type_into(win, "Email", p["email"]) # the consent prompt takes the keys typed after it opened+ choose(win, "Plan", p["plan"])+ box = win.child_window(title="Email digest", control_type="CheckBox")+ if bool(box.get_toggle_state()) != p["digest"]:+ click(box)+ click(button(win, "Sign up"))+++def disturb() -> None:+ """One deliberate disturbance of each kind the sentinel must see."""+ import ctypes++ time.sleep(2)+ win = Desktop(backend="uia").window(title_re="BenchLab — .*")+ win.set_focus() # focus steal, activation change, raise over the user window+ time.sleep(1)+ mouse.move(coords=(300, 300)) # real pointer movement+ ctypes.windll.user32.OpenClipboard(0)+ ctypes.windll.user32.EmptyClipboard() # clipboard change+ ctypes.windll.user32.CloseClipboard()+ time.sleep(2)+++def main() -> None:+ parser = argparse.ArgumentParser()+ parser.add_argument("--mode", required=True)+ parser.add_argument("--seed", type=int, required=True)+ parser.add_argument("--disturb", action="store_true")+ args = parser.parse_args()+ if args.disturb:+ disturb()+ else:+ globals()[args.mode](args.seed)+ time.sleep(1.0)+++if __name__ == "__main__":+ main()diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/windows/sentinel.py b/libs/cua-driver-bench/automated-eval/macos_bench/windows/sentinel.pynew file mode 100644index 0000000000..f5e2c7a8ec--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/windows/sentinel.py@@ -0,0 +1,192 @@+# SPDX-License-Identifier: FSL-1.1-MIT+# Copyright (c) 2026 Cua AI, Inc.++"""BenchSentinel for Windows: the witness / "User activity" window and its disturbance log.++A port of linux/sentinel.py with the same JSONL log, read by swift/summarize_sentinel.py unchanged.+Windows has no SIGUSR1/SIGUSR2, so the runner controls it through a control file (--control): each line+is a command, `activate`, `arm`, `disarm` or `quit`, applied once in order.++Measured while armed, every 200 ms:+ front GetForegroundWindow -> process stem (the sentinel itself is ai.cua.benchsentinel)+ mouse GetCursorPos (the real pointer)+ clip GetClipboardSequenceNumber+ onspace IVirtualDesktopManager::IsWindowOnCurrentVirtualDesktop on the sentinel window+ sheet a shown window owned by the sentinel window (a dialog on the user's app)+ above shown top-level windows above the sentinel in the z-order that overlap it+Window events: WM_ACTIVATE (didBecomeActive / didResignKey), key, click and wheel input landing in the+user window (keyDown, mouseDown, scrollWheel), and the simulated user's typing (user_type, user_blocked,+user_text_disrupted, user_text).+"""++from __future__ import annotations++import argparse+import ctypes+import json+import os+import sys+import time+from pathlib import Path++import wx++sys.path.insert(0, str(Path(__file__).resolve().parent))+import winenv as W # noqa: E402++BID = "ai.cua.benchsentinel"+++class Sentinel:+ def __init__(self, args: argparse.Namespace) -> None:+ self.args = args+ args.log.parent.mkdir(parents=True, exist_ok=True)+ self.log = args.log.open("a", buffering=1, encoding="utf-8")+ self.armed = False+ self.expected = ""+ self.previous = None+ self.done_commands = 0+ self.desktops = W.VirtualDesktops()+ title = "User activity" if args.user_activity else "Bench Sentinel"+ self.frame = wx.Frame(None, title=title, size=(600, 320))+ self.frame.SetName("BenchSentinel")+ if args.frame:+ x, y, w, h = map(int, args.frame.split(","))+ self.frame.SetPosition((x, y))+ self.frame.SetSize((w, h))+ panel = wx.Panel(self.frame)+ self.entry = wx.TextCtrl(panel, name="User activity text")+ sizer = wx.BoxSizer(wx.VERTICAL)+ sizer.Add(self.entry, 0, wx.EXPAND | wx.ALL, 16)+ panel.SetSizer(sizer)+ self.entry.Bind(wx.EVT_TEXT, self.changed)+ self.entry.Bind(wx.EVT_KEY_DOWN, lambda e: (self.emit("keyDown"), e.Skip()))+ self.entry.Bind(wx.EVT_LEFT_DOWN, lambda e: (self.emit("mouseDown"), e.Skip()))+ self.entry.Bind(wx.EVT_MOUSEWHEEL, lambda e: (self.emit("scrollWheel"), e.Skip()))+ panel.Bind(wx.EVT_LEFT_DOWN, lambda e: (self.emit("mouseDown"), e.Skip()))+ self.frame.Bind(wx.EVT_ACTIVATE, self.activated)+ if args.no_activate:+ self.frame.ShowWithoutActivating()+ else:+ self.frame.Show()+ self.activate()+ self.own = self.frame.GetHandle()+ self.timers = []+ self.every(100, self.control)+ self.every(200, self.sample)+ if args.user_activity:+ self.every(500, self.activity)+ self.emit("ready", hwnd=self.own)++ def every(self, ms: int, fn) -> None:+ timer = wx.Timer(self.frame)+ self.frame.Bind(wx.EVT_TIMER, lambda _: fn(), timer)+ timer.Start(ms)+ self.timers.append(timer)++ def activated(self, event: wx.ActivateEvent) -> None:+ event.Skip()+ if event.GetActive():+ self.emit("didBecomeActive")+ else:+ self.emit("didResignKey")++ def changed(self, _event: object) -> None:+ if self.entry.GetValue() != self.expected:+ self.emit("user_text_disrupted")++ def snapshot(self) -> dict:+ front = W.identity(W.user32.GetForegroundWindow(), self.own, BID)+ above = [W.identity(w, self.own, BID)["bid"] for w in W.windows_above(self.own)]+ on = self.desktops.on_current(self.own)+ sample = {+ "front": front,+ "mouse": W.cursor(),+ "clip": W.clipboard_seq(),+ "sheet": W.owned_dialog_shown(self.own),+ "above": above,+ "above_n": len(above),+ }+ if on is not None: # not measured is left out, which the summary reports as None+ sample["onspace"] = on+ return sample++ def emit(self, event: str, **details: object) -> None:+ if self.armed or event in ("armed", "disarmed", "ready"):+ self.log.write(json.dumps({"t": time.time() * 1000, "ev": event, **details}) + "\n")++ def activate(self) -> None:+ W.force_foreground(self.frame.GetHandle())+ self.entry.SetFocus()++ def control(self) -> None:+ path = self.args.control+ if not path or not path.exists():+ return+ try:+ lines = path.read_text(encoding="utf-8").splitlines()+ except OSError:+ return+ for command in lines[self.done_commands :]:+ self.done_commands += 1+ command = command.strip()+ if command == "activate":+ self.activate()+ elif command == "arm" and not self.armed:+ self.armed = True+ self.emit("armed", **self.snapshot())+ elif command == "disarm" and self.armed:+ if self.args.user_activity:+ self.emit("user_text", intact=self.entry.GetValue() == self.expected)+ self.emit("disarmed", **self.snapshot())+ self.armed = False+ elif command == "quit":+ self.frame.Close()+ return++ def sample(self) -> None:+ if not self.armed:+ return+ sample = self.snapshot()+ if sample["front"] != self.previous:+ self.emit("front", **sample)+ if self.previous and self.previous.get("bid") == BID:+ self.emit("didResignActive")+ self.previous = sample["front"]+ self.emit("s", **sample)++ def activity(self) -> None:+ if not self.armed:+ return+ if W.identity(W.user32.GetForegroundWindow(), self.own, BID)["bid"] == BID:+ if self.entry.GetValue() != self.expected:+ self.emit("user_text_disrupted")+ self.expected += "a"+ self.entry.ChangeValue(self.expected)+ self.entry.SetInsertionPointEnd()+ self.emit("user_type")+ else:+ self.emit("user_blocked")+++def main() -> None:+ parser = argparse.ArgumentParser()+ parser.add_argument("--log", type=Path, required=True)+ parser.add_argument("--control", type=Path, help="command file: activate, arm, disarm, quit")+ parser.add_argument("--no-activate", action="store_true")+ parser.add_argument("--user-activity", action="store_true")+ parser.add_argument("--frame", help="x,y,width,height")+ args = parser.parse_args()+ ctypes.windll.shcore.SetProcessDpiAwareness(2)+ app = wx.App(False)+ app.SetAppName("BenchSentinel")+ witness = Sentinel(args)+ try:+ app.MainLoop()+ finally:+ witness.log.close()+ os._exit(0)+++if __name__ == "__main__":+ main()diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/windows/winenv.py b/libs/cua-driver-bench/automated-eval/macos_bench/windows/winenv.pynew file mode 100644index 0000000000..1ce7a105ff--- /dev/null+++ b/libs/cua-driver-bench/automated-eval/macos_bench/windows/winenv.py@@ -0,0 +1,293 @@+# SPDX-License-Identifier: FSL-1.1-MIT+# Copyright (c) 2026 Cua AI, Inc.++"""Win32 reads shared by the Windows sentinel, the GUI-only guard and the session launcher (ctypes only).++Window identity is the executable's stem (`BenchLab`, `msedge`, `WindowsTerminal`), the Windows counterpart+of the bundle id (macOS) and WM_CLASS (Linux). Python processes are named after their window title when it+starts with BenchLab, so a BenchLab started from python.exe still reads as BenchLab.+"""++from __future__ import annotations++import ctypes+import os+from ctypes import wintypes++user32 = ctypes.WinDLL("user32", use_last_error=True)+kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)+dwmapi = ctypes.WinDLL("dwmapi")+ole32 = ctypes.WinDLL("ole32")++HWND = wintypes.HWND+user32.GetForegroundWindow.restype = HWND+user32.GetWindow.restype = HWND+user32.GetWindow.argtypes = [HWND, wintypes.UINT]+user32.GetWindowThreadProcessId.argtypes = [HWND, ctypes.POINTER(wintypes.DWORD)]+user32.GetWindowThreadProcessId.restype = wintypes.DWORD+user32.IsWindowVisible.argtypes = [HWND]+user32.IsIconic.argtypes = [HWND]+user32.GetWindowRect.argtypes = [HWND, ctypes.POINTER(wintypes.RECT)]+user32.GetWindowTextW.argtypes = [HWND, wintypes.LPWSTR, ctypes.c_int]+user32.GetClassNameW.argtypes = [HWND, wintypes.LPWSTR, ctypes.c_int]+user32.SetForegroundWindow.argtypes = [HWND]+user32.BringWindowToTop.argtypes = [HWND]+user32.ShowWindow.argtypes = [HWND, ctypes.c_int]+user32.AttachThreadInput.argtypes = [wintypes.DWORD, wintypes.DWORD, wintypes.BOOL]+user32.GetClipboardSequenceNumber.restype = wintypes.DWORD+user32.SetWindowPos.argtypes = [+ HWND, HWND, ctypes.c_int, ctypes.c_int, ctypes.c_int, ctypes.c_int, wintypes.UINT+] # fmt: skip+WNDENUMPROC = ctypes.WINFUNCTYPE(wintypes.BOOL, HWND, wintypes.LPARAM)+user32.EnumWindows.argtypes = [WNDENUMPROC, wintypes.LPARAM]+kernel32.OpenProcess.restype = wintypes.HANDLE+kernel32.OpenProcess.argtypes = [wintypes.DWORD, wintypes.BOOL, wintypes.DWORD]+kernel32.QueryFullProcessImageNameW.argtypes = [+ wintypes.HANDLE, wintypes.DWORD, wintypes.LPWSTR, ctypes.POINTER(wintypes.DWORD)+] # fmt: skip+kernel32.CloseHandle.argtypes = [wintypes.HANDLE]++GW_OWNER, GW_HWNDPREV = 4, 3+DWMWA_CLOAKED = 14+PROCESS_QUERY_LIMITED_INFORMATION = 0x1000+SW_RESTORE = 9+SWP_NOSIZE, SWP_NOMOVE, SWP_NOACTIVATE, SWP_NOZORDER = 0x1, 0x2, 0x10, 0x4+++def text(hwnd: int) -> str:+ buf = ctypes.create_unicode_buffer(512)+ user32.GetWindowTextW(hwnd, buf, 512)+ return buf.value+++def class_name(hwnd: int) -> str:+ buf = ctypes.create_unicode_buffer(256)+ user32.GetClassNameW(hwnd, buf, 256)+ return buf.value+++def window_pid(hwnd: int) -> int:+ pid = wintypes.DWORD()+ user32.GetWindowThreadProcessId(hwnd, ctypes.byref(pid))+ return pid.value+++def window_thread(hwnd: int) -> int:+ return user32.GetWindowThreadProcessId(hwnd, None)+++def process_path(pid: int) -> str:+ handle = kernel32.OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, False, pid)+ if not handle:+ return ""+ try:+ size = wintypes.DWORD(1024)+ buf = ctypes.create_unicode_buffer(1024)+ if kernel32.QueryFullProcessImageNameW(handle, 0, buf, ctypes.byref(size)):+ return buf.value+ return ""+ finally:+ kernel32.CloseHandle(handle)+++def stem(path: str) -> str:+ return os.path.splitext(os.path.basename(path))[0]+++def identity(hwnd: int, own: int = 0, own_bid: str = "ai.cua.benchsentinel") -> dict:+ if not hwnd:+ return {"bid": "win32:0", "pid": -1}+ if own and hwnd == own:+ return {"bid": own_bid, "pid": os.getpid()}+ pid = window_pid(hwnd)+ name = stem(process_path(pid)) if pid else ""+ if name.lower() in ("python", "pythonw") and text(hwnd).startswith("BenchLab"):+ name = "BenchLab"+ return {"bid": name or f"win32:{hwnd}", "pid": pid or -1}+++def rect(hwnd: int) -> tuple[int, int, int, int]:+ r = wintypes.RECT()+ user32.GetWindowRect(hwnd, ctypes.byref(r))+ return r.left, r.top, r.right, r.bottom+++def intersects(a: tuple[int, int, int, int], b: tuple[int, int, int, int]) -> bool:+ return a[0] < b[2] and b[0] < a[2] and a[1] < b[3] and b[1] < a[3]+++def cloaked(hwnd: int) -> bool:+ value = wintypes.DWORD()+ dwmapi.DwmGetWindowAttribute(+ HWND(hwnd), wintypes.DWORD(DWMWA_CLOAKED), ctypes.byref(value), ctypes.sizeof(value)+ )+ return bool(value.value)+++def shown(hwnd: int) -> bool:+ if not user32.IsWindowVisible(hwnd) or user32.IsIconic(hwnd) or cloaked(hwnd):+ return False+ left, top, right, bottom = rect(hwnd)+ return right > left and bottom > top+++def windows_above(own: int) -> list[int]:+ """Shown top-level windows above `own` in the z-order that overlap it (a raise over the user window)."""+ out = []+ mine = rect(own)+ hwnd = user32.GetWindow(own, GW_HWNDPREV)+ seen = 0+ while hwnd and seen < 2000:+ seen += 1+ if shown(hwnd) and intersects(rect(hwnd), mine):+ out.append(hwnd)+ hwnd = user32.GetWindow(hwnd, GW_HWNDPREV)+ return out+++def owned_dialog_shown(own: int) -> bool:+ """A shown top-level window owned by `own`: a dialog on the user's app."""+ found = []++ def each(hwnd, _):+ if hwnd != own and user32.GetWindow(hwnd, GW_OWNER) == own and shown(hwnd):+ found.append(hwnd)+ return True++ user32.EnumWindows(WNDENUMPROC(each), 0)+ return bool(found)+++def top_level_windows() -> list[int]:+ found: list[int] = []++ def each(hwnd, _):+ found.append(hwnd)+ return True++ user32.EnumWindows(WNDENUMPROC(each), 0)+ return found+++def cursor() -> list[int]:+ p = wintypes.POINT()+ user32.GetCursorPos(ctypes.byref(p))+ return [p.x, p.y]+++def clipboard_seq() -> int:+ return int(user32.GetClipboardSequenceNumber())+++class _GUID(ctypes.Structure):+ _fields_ = [+ ("d1", wintypes.DWORD), ("d2", wintypes.WORD), ("d3", wintypes.WORD), ("d4", ctypes.c_ubyte * 8)+ ] # fmt: skip++ @classmethod+ def parse(cls, s: str) -> _GUID:+ g = cls()+ ole32.CLSIDFromString(ctypes.c_wchar_p(s), ctypes.byref(g))+ return g+++class VirtualDesktops:+ """IVirtualDesktopManager (documented, Windows 10 and later) through a raw COM vtable."""++ CLSID = "{aa509086-5ca9-4c25-8f95-589d3c07b48a}"+ IID = "{a5cd92ff-29be-454c-8d04-d82879fb3f1b}"++ def __init__(self) -> None:+ ole32.CoInitializeEx(None, 2) # apartment threaded, like the GUI thread that calls it+ self.ptr = ctypes.c_void_p()+ hr = ole32.CoCreateInstance(+ ctypes.byref(_GUID.parse(self.CLSID)), None, 1 | 4,+ ctypes.byref(_GUID.parse(self.IID)), ctypes.byref(self.ptr),+ ) # fmt: skip+ if hr != 0 or not self.ptr.value:+ self.ptr = None+ return+ vtable = ctypes.cast(self.ptr, ctypes.POINTER(ctypes.POINTER(ctypes.c_void_p))).contents+ proto = ctypes.WINFUNCTYPE(+ ctypes.c_long, ctypes.c_void_p, HWND, ctypes.POINTER(wintypes.BOOL)+ )+ self._on_current = proto(vtable[3])++ def on_current(self, hwnd: int) -> bool | None:+ if not self.ptr:+ return None+ out = wintypes.BOOL()+ if self._on_current(self.ptr, hwnd, ctypes.byref(out)) != 0:+ return None+ return bool(out.value)+++def force_foreground(hwnd: int) -> bool:+ """Bring `hwnd` to the front from a process that may not own the foreground (setup only, never armed)."""+ if user32.IsIconic(hwnd):+ user32.ShowWindow(hwnd, SW_RESTORE)+ front = user32.GetForegroundWindow()+ if front == hwnd:+ return True+ mine = kernel32.GetCurrentThreadId()+ theirs = window_thread(front) if front else 0+ attached = bool(theirs and theirs != mine and user32.AttachThreadInput(mine, theirs, True))+ try:+ user32.BringWindowToTop(hwnd)+ user32.SetForegroundWindow(hwnd)+ finally:+ if attached:+ user32.AttachThreadInput(mine, theirs, False)+ if user32.GetForegroundWindow() != hwnd:+ # The documented unlock: a keyboard event lets this process set the foreground.+ user32.keybd_event(0x12, 0, 0, 0)+ user32.keybd_event(0x12, 0, 2, 0)+ user32.SetForegroundWindow(hwnd)+ return user32.GetForegroundWindow() == hwnd+++def move_window(hwnd: int, x: int, y: int, w: int, h: int) -> None:+ user32.SetWindowPos(hwnd, None, x, y, w, h, SWP_NOACTIVATE | SWP_NOZORDER)+++class _PROCESSENTRY32W(ctypes.Structure):+ _fields_ = [+ ("dwSize", wintypes.DWORD), ("cntUsage", wintypes.DWORD), ("th32ProcessID", wintypes.DWORD),+ ("th32DefaultHeapID", ctypes.c_void_p), ("th32ModuleID", wintypes.DWORD),+ ("cntThreads", wintypes.DWORD), ("th32ParentProcessID", wintypes.DWORD),+ ("pcPriClassBase", ctypes.c_long), ("dwFlags", wintypes.DWORD), ("szExeFile", ctypes.c_wchar * 260),+ ] # fmt: skip+++kernel32.CreateToolhelp32Snapshot.restype = wintypes.HANDLE+kernel32.Process32FirstW.argtypes = [wintypes.HANDLE, ctypes.POINTER(_PROCESSENTRY32W)]+kernel32.Process32NextW.argtypes = [wintypes.HANDLE, ctypes.POINTER(_PROCESSENTRY32W)]+++def processes() -> list[tuple[int, int, str]]:+ """(pid, parent pid, executable name) of every process."""+ snap = kernel32.CreateToolhelp32Snapshot(0x2, 0)+ rows: list[tuple[int, int, str]] = []+ if not snap or snap == wintypes.HANDLE(-1).value:+ return rows+ try:+ entry = _PROCESSENTRY32W()+ entry.dwSize = ctypes.sizeof(entry)+ ok = kernel32.Process32FirstW(snap, ctypes.byref(entry))+ while ok:+ rows.append((entry.th32ProcessID, entry.th32ParentProcessID, entry.szExeFile))+ ok = kernel32.Process32NextW(snap, ctypes.byref(entry))+ finally:+ kernel32.CloseHandle(snap)+ return rows+++kernel32.ProcessIdToSessionId.argtypes = [wintypes.DWORD, ctypes.POINTER(wintypes.DWORD)]+++def session_of(pid: int) -> int | None:+ """The Terminal Services session of a process (the interactive desktop is not session 0)."""+ out = wintypes.DWORD()+ if not kernel32.ProcessIdToSessionId(pid, ctypes.byref(out)):+ return None+ return int(out.value)diff --git a/libs/cua-driver-bench/taskpack/v2-taskset.json b/libs/cua-driver-bench/taskpack/v2-taskset.jsonindex 26bde59a68..67724c88ad 100644--- a/libs/cua-driver-bench/taskpack/v2-taskset.json+++ b/libs/cua-driver-bench/taskpack/v2-taskset.json@@ -1,7 +1,7 @@{"schema": "cdb-taskset/1","name": "v2",- "description": "Bench v2: the ten core tasks, all public. `os` lists where the task has a launch descriptor or runs today: the three multi-app tasks (CDB-G02 to CDB-G04, pack tasks with Electron apps, LibreOffice and, for CDB-G03, GnuCash) run on macOS, Linux and Windows; the seven BenchLab tasks (MB-09 to MB-11, IR-01 to IR-04) run on macOS and Linux. `split` is public (in this repository) or heldout (kept out of this repository; such a task would be listed by id and category only).",+ "description": "Bench v2: the ten core tasks, all public. `os` lists where the task has a launch descriptor or runs today: all ten run on macOS, Linux and Windows. The three multi-app tasks (CDB-G02 to CDB-G04) are pack tasks with Electron apps, LibreOffice and, for CDB-G03, GnuCash; the seven BenchLab tasks (MB-09 to MB-11, IR-01 to IR-04) run in BenchLab and its Linux and Windows ports. `split` is public (in this repository) or heldout (kept out of this repository; such a task would be listed by id and category only).","limits": {"timeout_s": 360,"max_turns": 45@@ -109,7 +109,8 @@],"os": ["macos",- "linux"+ "linux",+ "windows"],"cross_os_candidate": null,"conditions": [@@ -136,7 +137,8 @@],"os": ["macos",- "linux"+ "linux",+ "windows"],"cross_os_candidate": null,"conditions": [@@ -162,7 +164,8 @@],"os": ["macos",- "linux"+ "linux",+ "windows"],"cross_os_candidate": null,"conditions": [@@ -188,7 +191,8 @@],"os": ["macos",- "linux"+ "linux",+ "windows"],"cross_os_candidate": null,"conditions": [@@ -215,7 +219,8 @@],"os": ["macos",- "linux"+ "linux",+ "windows"],"cross_os_candidate": null,"conditions": [@@ -242,7 +247,8 @@],"os": ["macos",- "linux"+ "linux",+ "windows"],"cross_os_candidate": null,"conditions": [@@ -269,7 +275,8 @@],"os": ["macos",- "linux"+ "linux",+ "windows"],"cross_os_candidate": null,"conditions": [