Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cua-driver-bench): Windows port of the BenchLab tasks (CUA-3563) (#6607)

Francesco Bonacci78951406bc05
diff --git a/.github/workflows/ci-cua-driver-bench.yml b/.github/workflows/ci-cua-driver-bench.yml
index a21c67ac81..5796c35b31 100644
--- a/.github/workflows/ci-cua-driver-bench.yml
+++ b/.github/workflows/ci-cua-driver-bench.yml
@@ -82,6 +82,25 @@ jobs:
dbus-run-session -- bash -lc \
'openbox >/tmp/cua-openbox.log 2>&1 & picom --backend xrender --config /dev/null >/tmp/cua-picom.log 2>&1 & sleep 2; /usr/bin/python3 -m pytest -q tests/test_pilot_runner.py tests/test_linux_session.py tests/test_runner_flow.py tests/test_bench_v2.py tests/test_analyze.py tests/test_cdb_adapter.py tests/test_linux_tools.py'
+ windows-desktop:
+ needs: quick
+ if: ${{ !failure() && !cancelled() && (!startsWith(github.head_ref, 'release-please--branches--')) }}
+ name: Windows desktop contracts (no model)
+ runs-on: windows-latest
+ timeout-minutes: 20
+ steps:
+ - name: Check out Cua
+ uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+
+ - name: Set up uv
+ uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
+
+ - name: BenchLab, sentinel, guard and launcher on the Windows desktop
+ working-directory: libs/cua-driver-bench/automated-eval/macos_bench
+ run: >-
+ uv run --no-project --python 3.12 --with pytest==8.4.2 --with wxPython==4.3.2 --with pywinauto==0.6.9
+ python -m pytest -q tests/test_windows_tools.py tests/test_windows_desktop.py
+
taskpack:
needs: quick
if: ${{ !failure() && !cancelled() && (!startsWith(github.head_ref, 'release-please--branches--')) }}
diff --git a/libs/cua-driver-bench/README.md b/libs/cua-driver-bench/README.md
index cd63585ed7..b359c6103c 100644
--- a/libs/cua-driver-bench/README.md
+++ b/libs/cua-driver-bench/README.md
@@ -7,9 +7,8 @@ harness, the agent, the model, the prompts and the limits are the same for every
setup; only the computer-use tool changes.
This directory holds the harness, the public task set and the method of
-**Bench v2**. The ten Bench v2 tasks run on macOS and Linux; the three multi-app
-tasks (CDB-G02 to CDB-G04) also run on Windows. The measurement on macOS assumes
-a macOS 26 desktop.
+**Bench v2**. The ten Bench v2 tasks run on macOS, Linux and Windows. The
+measurement on macOS assumes a macOS 26 desktop.
| Path | What it is |
| --- | --- |
@@ -44,7 +43,8 @@ Ten tasks in three categories. Each task has a `PROVENANCE.md` next to it.
The CDB-G tasks are the GUI-only variants of pack tasks `cdb-s02` to `cdb-s04`:
the same files, with no coding tools offered. The MB and IR tasks run in
BenchLab, a small AppKit test app built from source in this directory (with a
-Linux port in `automated-eval/macos_bench/linux`). A fourth category, Web apps &
+Linux port in `automated-eval/macos_bench/linux` and a Windows port in
+`automated-eval/macos_bench/windows`). A fourth category, Web apps &
forms, has no Bench v2 task yet.
### Platforms and conditions
@@ -73,17 +73,16 @@ Public tasks, in schedule order:
| CDB-G02 | multi_app | Electron mail, chat and calendar apps, LibreOffice Calc | macOS, Linux, Windows | both | Replan a schedule across mail, chat, a calendar and a Calc workbook |
| CDB-G03 | multi_app | Electron mail app, LibreOffice Calc, GnuCash | macOS, Linux, Windows | both | Review payables across mail, a Calc workbook and a GnuCash book |
| CDB-G04 | multi_app | Electron case tracker and manual, Google Chrome | macOS, Linux, Windows | both | Triage duplicate support cases with a tracker, a manual, an archive site and a log |
-| MB-09 | precision | BenchLab | macOS, Linux | both | Reorder a native list by drag and drop |
-| MB-10 | precision | BenchLab | macOS, Linux | foreground only | Reveal a hover-only toolbar and click one action |
-| MB-11 | precision | BenchLab | macOS, Linux | foreground only | Read a native tooltip and type the code it shows |
-| IR-01 | interruptions | BenchLab | macOS, Linux | both | Finish a two-step form through a permission-style prompt |
-| IR-02 | interruptions | BenchLab | macOS, Linux | both | Apply settings while a notification-style banner covers the button |
-| IR-03 | interruptions | BenchLab | macOS, Linux | both | Edit two notes and keep both changes through a save prompt |
-| IR-04 | interruptions | BenchLab | macOS, Linux | both | Sign up on a web page while a cookie-consent overlay appears |
+| MB-09 | precision | BenchLab | macOS, Linux, Windows | both | Reorder a native list by drag and drop |
+| MB-10 | precision | BenchLab | macOS, Linux, Windows | foreground only | Reveal a hover-only toolbar and click one action |
+| MB-11 | precision | BenchLab | macOS, Linux, Windows | foreground only | Read a native tooltip and type the code it shows |
+| IR-01 | interruptions | BenchLab | macOS, Linux, Windows | both | Finish a two-step form through a permission-style prompt |
+| IR-02 | interruptions | BenchLab | macOS, Linux, Windows | both | Apply settings while a notification-style banner covers the button |
+| IR-03 | interruptions | BenchLab | macOS, Linux, Windows | both | Edit two notes and keep both changes through a save prompt |
+| IR-04 | interruptions | BenchLab | macOS, Linux, Windows | both | Sign up on a web page while a cookie-consent overlay appears |
<!-- taskset:end -->
-The seven BenchLab tasks run on macOS and on Linux. The three multi-app tasks run
-on macOS, Linux and Windows; CDB-G03 needs GnuCash. Tasks marked "foreground
+All ten tasks run on macOS, Linux and Windows; CDB-G03 needs GnuCash. Tasks marked "foreground
only" skip the background condition: their readout appears only on a hover or a
tooltip, which follows the real pointer, so no setup can do them in the
background.
@@ -315,11 +314,54 @@ completion with Wilson intervals, focus steals, key loss, windows raised,
swallowed keystrokes, GUI-only violations with evidence, and the interruption
rates. Keep raw results, streams and recordings out of git.
+### Windows (the BenchLab tasks)
+
+The Windows port of BenchLab and the sentinel is in
+[`automated-eval/macos_bench/windows`](automated-eval/macos_bench/windows): Python with wxPython, so
+every control is a native Win32 control that UI Automation exposes. Seeds, the state file, the event log
+and the checkers are the same as on macOS and Linux (`probes/_common/benchlab_common.py`), and the
+sentinel log is read by the same `swift/summarize_sentinel.py`.
+
+You need Windows 11, Python 3.12 with `wxPython` (and `pywinauto` for the oracle only), and an
+interactive desktop session: run the trials as a standard user signed in at the console, not from an
+SSH or service session. One trial:
+
+```powershell
+python launch\windows_session.py --task IR-02 --run-index 0 --condition background --out C:\bench\IR-02-r0 -- <agent command>
+```
+
+`{brief}`, `{out}`, `{seed}` and `{mode}` in the agent command are filled in. The launcher starts
+BenchLab and the sentinel (as `BenchLab.exe` and `BenchSentinel.exe`, copies of `pythonw.exe`), brings
+the "User activity" window to the front in the background condition, arms the sentinel, runs the agent
+with the GUI-only guard watching, then runs the checker and writes `row.json`. `windows\oracle.py` solves
+each task through the GUI with real input; `tests/test_windows_desktop.py` checks that the oracle passes
+every task, that doing nothing fails and leaves the user undisturbed, and that a deliberate disturbance
+is seen.
+
+Where Windows differs:
+
+* **GUI-only guard.** A shell or script host (PowerShell, cmd, Windows Terminal, the script hosts,
+ mshta, WSL) started in the desktop session outside the runner's own processes, or a side-door window
+ in front, fails and stops the trial. Tool inputs are also scanned for PowerShell, cmd and script-host
+ commands.
+* **Sentinel.** Windows has no signals, so the runner arms it through a control file. A window counts as
+ raised when it is above the user window in the z-order and overlaps it. The desktop measure is
+ `IVirtualDesktopManager`; a dialog on the user's app is a window owned by the user window.
+* **Dialogs.** BenchLab's dialogs are owned by its window and disable it while open. When BenchLab is not
+ in front, a dialog opens without activation just above BenchLab, so BenchLab never raises itself.
+* **MB-09.** Rows reorder by a native list drag (the list view's begin-drag, then the button release
+ over a row).
+* **MB-10 and MB-11.** Hover and tooltips follow the real pointer on Windows too (mouse-leave tracking
+ and the tooltip control read the cursor position), so both stay foreground-only. A tooltip counts as
+ shown when BenchLab's own tooltip window is visible.
+* **Brief.** "on your Mac" reads "on this Windows PC".
+
### Tests
```bash
cd automated-eval/macos_bench
python3 -m pytest tests probes/tests swift/tests # harness, checkers, sentinel (no GUI, no model)
+python -m pytest tests/test_windows_desktop.py # on a Windows desktop: BenchLab, sentinel, guard, oracle
cd ../../taskpack/tasks/shared/cdb-s02 && python3 -m unittest discover -s tests
cd ../../../../fairness && python3 -m unittest discover -s tests # fairness and health checks
python3 ../automated-eval/macos_bench/tools/taskset.py check # manifest, selectors, pins and docs agree
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/TASKS.md b/libs/cua-driver-bench/automated-eval/macos_bench/TASKS.md
index df7b8397d1..6aca52c46a 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/TASKS.md
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/TASKS.md
@@ -259,7 +259,7 @@ Bench v2 and are not in this repository.
## Platforms, splits and the task manifest
[`../../taskpack/v2-taskset.json`](../../taskpack/v2-taskset.json) lists every task with its category, apps, OS
-availability (the seven BenchLab tasks: macOS and Linux; CDB-G02 to CDB-G04: macOS, Linux and Windows), conditions,
+availability (all ten tasks: macOS, Linux and Windows), conditions,
health requirements and split; the bench README renders it as a table. Selectors are in `probes/<ID>/task.json`.
A task whose selector says `"split": "heldout"` is read from a separate pack with the same layout, named by
`CDB_TASKPACK_HELDOUT` (selectors in `probes/`, trees in `tasks/`, digests in `pins.heldout.json`); `--task-set
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/launch/windows_session.py b/libs/cua-driver-bench/automated-eval/macos_bench/launch/windows_session.py
new file mode 100644
index 0000000000..2def1710cf
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/launch/windows_session.py
@@ -0,0 +1,289 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: FSL-1.1-MIT
+# Copyright (c) 2026 Cua AI, Inc.
+#
+# Usage: windows_session.py --task ID --run-index N --condition foreground|background --out DIR
+# [--timeout S] -- COMMAND [ARG...]
+#
+# One BenchLab trial on a Windows desktop, run in the interactive session of the user who owns it:
+# 1. derive the seed (fairness rule), render the brief (+ the background paragraph), write DIR/brief.md;
+# 2. start BenchLab (windows/benchlab.py) and the sentinel (windows/sentinel.py): in the background
+# condition the "User activity" window partly covers BenchLab, is brought to the front and types;
+# 3. arm the sentinel, run COMMAND (the agent; `{brief}`, `{out}`, `{seed}`, `{mode}` in its arguments are filled in)
+# with the GUI-only guard watching, stop it at the time limit or on a violation;
+# 4. disarm, run the task's checker, summarise the sentinel, write DIR/row.json.
+# BenchLab and the sentinel run from copies of pythonw.exe named BenchLab.exe and BenchSentinel.exe, so every
+# tool sees them under those names.
+from __future__ import annotations
+
+import argparse
+import importlib.util
+import json
+import os
+import shutil
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+BENCH = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(BENCH))
+sys.path.insert(0, str(BENCH / "windows"))
+sys.path.insert(0, str(BENCH / "probes" / "_common"))
+sys.path.insert(0, str(BENCH / "swift"))
+sys.path.insert(0, str(BENCH.parents[1] / "fairness"))
+import bench_v2 as v2 # noqa: E402
+import cdb_fairness # noqa: E402
+import summarize_sentinel # noqa: E402
+
+TASKS = ("MB-09", "MB-10", "MB-11", "IR-01", "IR-02", "IR-03", "IR-04")
+LAB_FRAME = (40, 40, 760, 540)
+WITNESS_FRAME = (1000, 640, 600, 320)
+PLATFORM_WORDS = ("on your Mac", "on this Windows PC")
+
+
+def task_spec(task: str) -> dict:
+ return json.loads((BENCH / "probes" / task / "task.json").read_text(encoding="utf-8"))
+
+
+def render_brief(task: str, seed: int, condition: str) -> str:
+ spec = importlib.util.spec_from_file_location(
+ f"render_{task}", BENCH / "probes" / task / "render_brief.py"
+ )
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ brief = module.render(seed).replace(*PLATFORM_WORDS)
+ return v2.brief_for(brief, condition)
+
+
+def named_python(name: str) -> Path:
+ """A copy of pythonw.exe called NAME.exe next to it (it finds its DLLs and stdlib from there)."""
+ base = Path(sys.executable).with_name("pythonw.exe")
+ target = base.with_name(f"{name}.exe")
+ if not target.exists() or target.stat().st_size != base.stat().st_size:
+ shutil.copy2(base, target)
+ return target
+
+
+def kill_tree(pid: int) -> None:
+ subprocess.run(["taskkill", "/T", "/F", "/PID", str(pid)], capture_output=True)
+
+
+def wait_for(predicate, limit: float, step: float = 0.1) -> bool:
+ end = time.monotonic() + limit
+ while time.monotonic() < end:
+ if predicate():
+ return True
+ time.sleep(step)
+ return False
+
+
+def sentinel_lines(log: Path) -> list[dict]:
+ if not log.exists():
+ return []
+ out = []
+ for raw in log.read_text(encoding="utf-8").splitlines():
+ try:
+ out.append(json.loads(raw))
+ except json.JSONDecodeError:
+ pass
+ return out
+
+
+def run(args: argparse.Namespace, command: list[str]) -> dict:
+ import guard as G
+ import winenv as W
+
+ out: Path = args.out.resolve()
+ out.mkdir(parents=True, exist_ok=True)
+ spec = task_spec(args.task)
+ mode = spec["app_args"][spec["app_args"].index("--mode") + 1]
+ seed = cdb_fairness.trial_seed(args.task, args.run_index)
+ brief = render_brief(args.task, seed, args.condition)
+ (out / "brief.md").write_text(brief, encoding="utf-8")
+ background = args.condition == "background"
+ row: dict = {
+ "task": args.task, "run_index": args.run_index, "seed": seed, "condition": args.condition,
+ "mode": mode, "platform": "windows", "label": args.label, "started": time.time(),
+ } # fmt: skip
+
+ state, events, log, control = (
+ out / "state.json", out / "events.jsonl", out / "sentinel.jsonl", out / "sentinel.control",
+ ) # fmt: skip
+ for stale in (state, events, log, control):
+ stale.unlink(missing_ok=True)
+ control.write_text("", encoding="utf-8")
+ lab = subprocess.Popen(
+ [str(named_python("BenchLab")), str(BENCH / "windows" / "benchlab.py"), "--mode", mode,
+ "--seed", str(seed), "--state", str(state), "--events", str(events),
+ "--frame", ",".join(map(str, LAB_FRAME)), *(["--no-activate"] if background else [])],
+ ) # fmt: skip
+ procs = [lab]
+ try:
+ if not wait_for(state.exists, 30):
+ row["harness_error"] = "BenchLab did not start"
+ return row
+ frame = v2.USER_WINDOW_FRAME if background else WITNESS_FRAME
+ sentinel = subprocess.Popen(
+ [str(named_python("BenchSentinel")), str(BENCH / "windows" / "sentinel.py"),
+ "--log", str(log), "--control", str(control), "--frame", ",".join(map(str, frame)),
+ *(["--user-activity"] if background else [])],
+ ) # fmt: skip
+ procs.append(sentinel)
+ if not wait_for(lambda: any(ln.get("ev") == "ready" for ln in sentinel_lines(log)), 30):
+ row["harness_error"] = "sentinel did not start"
+ return row
+ own = next(ln["hwnd"] for ln in sentinel_lines(log) if ln.get("ev") == "ready")
+ with control.open("a", encoding="utf-8") as c:
+ c.write("activate\n")
+ front_ok = wait_for(lambda: W.user32.GetForegroundWindow() == own, 10)
+ if not front_ok:
+ with control.open("a", encoding="utf-8") as c:
+ c.write("activate\n")
+ front_ok = wait_for(lambda: W.user32.GetForegroundWindow() == own, 5)
+ row["sentinel_front_at_start"] = front_ok
+ time.sleep(1.0)
+ with control.open("a", encoding="utf-8") as c:
+ c.write("arm\n")
+ wait_for(lambda: any(ln.get("ev") == "armed" for ln in sentinel_lines(log)), 5)
+
+ subs = {
+ "{brief}": str(out / "brief.md"),
+ "{out}": str(out),
+ "{seed}": str(seed),
+ "{mode}": mode,
+ }
+ argv = []
+ for a in command:
+ for key, value in subs.items():
+ a = a.replace(key, value)
+ argv.append(a)
+ rows0 = W.processes()
+ baseline = {pid for pid, _, _ in rows0}
+ t0 = time.monotonic()
+ agent = subprocess.Popen(
+ argv, cwd=str(out), env={**os.environ, "CDB_BRIEF_FILE": str(out / "brief.md")}
+ )
+ procs.append(agent)
+ own_tree: set[int] = set()
+ session = W.session_of(os.getpid())
+ front = G.FrontWatch()
+ violations: list[dict] = []
+ timed_out = False
+ while agent.poll() is None:
+ rows = W.processes()
+ own_tree |= G.descendants(rows, os.getpid())
+ # Only the runner's own (interactive) session: remote administration (SSH in session 0) is not the GUI.
+ here = [r for r in rows if W.session_of(r[0]) == session]
+ for pid, label in G.new_side_doors(here, baseline, os.getpid(), own_tree):
+ violations.append(
+ {
+ "t_s": round(time.monotonic() - t0, 1),
+ "kind": "process",
+ "detail": label,
+ "pid": pid,
+ }
+ )
+ kill_tree(pid)
+ hwnd = W.user32.GetForegroundWindow()
+ hit = front.sample(W.process_path(W.window_pid(hwnd)) if hwnd else "")
+ if hit:
+ violations.append(
+ {"t_s": round(time.monotonic() - t0, 1), "kind": "front", "detail": hit}
+ )
+ if violations and args.enforce:
+ kill_tree(agent.pid)
+ break
+ if time.monotonic() - t0 > args.timeout:
+ timed_out = True
+ kill_tree(agent.pid)
+ break
+ time.sleep(0.5)
+ agent.wait(timeout=30)
+ row["agent_exit"] = agent.returncode
+ row["agent_wall_s"] = round(time.monotonic() - t0, 1)
+ row["timed_out"] = timed_out
+
+ with control.open("a", encoding="utf-8") as c:
+ c.write("disarm\n")
+ wait_for(lambda: any(ln.get("ev") == "disarmed" for ln in sentinel_lines(log)), 5)
+ time.sleep(0.3)
+ result = out / "result.json"
+ subprocess.run(
+ [sys.executable, str(BENCH / "probes" / args.task / "evaluate.py"), "--seed", str(seed),
+ "--state", str(state), "--events", str(events), "--result", str(result)],
+ capture_output=True, timeout=60,
+ ) # fmt: skip
+ evaluation = json.loads(result.read_text(encoding="utf-8")) if result.exists() else {}
+ transcript = out / "agent-transcript.jsonl"
+ stream = []
+ if transcript.exists():
+ for raw in transcript.read_text(encoding="utf-8", errors="replace").splitlines():
+ try:
+ stream.append(json.loads(raw))
+ except json.JSONDecodeError:
+ pass
+ hard, soft = G.scan_events(stream)
+ violations += [{"t_s": None, "kind": "tool_input", "detail": h} for h in hard]
+ disturbance = summarize_sentinel.summarize(str(log))
+ armed = next((ln for ln in sentinel_lines(log) if ln.get("ev") == "armed"), {})
+ frontmost = (armed.get("front") or {}).get("bid") == "ai.cua.benchsentinel"
+ passed = bool(evaluation.get("passed")) and not violations
+ bg = v2.background_score(disturbance, frontmost)
+ row.update(
+ evaluation={k: evaluation.get(k) for k in ("passed", "score", "checks", "diagnostics")},
+ checker_passed=bool(evaluation.get("passed")),
+ gui_only={"violation": bool(violations), "violations": violations[:20],
+ "mentions": soft, "frontmost_seen": sorted(front.seen)},
+ passed=passed,
+ disturbance=disturbance,
+ background=bg,
+ a15=v2.background_condition_score(bg, disturbance, passed, args.condition),
+ a22=v2.multiplayer_score(disturbance, bg["measured"], passed, args.condition),
+ ) # fmt: skip
+ return row
+ finally:
+ with control.open("a", encoding="utf-8") as c:
+ c.write("quit\n")
+ time.sleep(0.5)
+ for proc in procs:
+ if proc.poll() is None:
+ kill_tree(proc.pid)
+ row["finished"] = time.time()
+ (out / "row.json").write_text(json.dumps(row, indent=1), encoding="utf-8")
+
+
+def main(argv: list[str]) -> int:
+ if "--" not in argv:
+ print(
+ "usage: windows_session.py --task ID --run-index N --condition C --out DIR -- COMMAND...",
+ file=sys.stderr,
+ )
+ return 2
+ split = argv.index("--")
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--task", choices=TASKS, required=True)
+ parser.add_argument("--run-index", type=int, required=True)
+ parser.add_argument("--condition", choices=v2.CONDITIONS, required=True)
+ parser.add_argument("--out", type=Path, required=True)
+ parser.add_argument("--label", default="")
+ parser.add_argument("--timeout", type=float, default=360.0)
+ parser.add_argument("--no-enforce", dest="enforce", action="store_false")
+ args = parser.parse_args(argv[:split])
+ command = argv[split + 1 :]
+ if sys.platform != "win32":
+ print("windows_session.py requires Windows", file=sys.stderr)
+ return 2
+ if not command:
+ return 2
+ row = run(args, command)
+ print(
+ json.dumps(
+ {k: row.get(k) for k in ("task", "seed", "condition", "passed", "harness_error")}
+ )
+ )
+ return 0 if not row.get("harness_error") else 3
+
+
+if __name__ == "__main__":
+ sys.exit(main(sys.argv[1:]))
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-01/launch.windows.json b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-01/launch.windows.json
new file mode 100644
index 0000000000..f63abac2c7
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-01/launch.windows.json
@@ -0,0 +1,47 @@
+{
+ "platform": "windows",
+ "descriptor_version": "1",
+ "task_id": "IR-01",
+ "app": "windows/benchlab.py",
+ "app_args": [
+ "--mode",
+ "irmodal"
+ ],
+ "app_executable": "BenchLab.exe (a copy of pythonw.exe; see launch/windows_session.py)",
+ "sentinel": "windows/sentinel.py",
+ "launcher": [
+ "${python}",
+ "launch/windows_session.py",
+ "--task",
+ "IR-01",
+ "--run-index",
+ "${run_index}",
+ "--condition",
+ "${condition}",
+ "--out",
+ "${artifacts}",
+ "--",
+ "${agent_command}"
+ ],
+ "conditions": [
+ "foreground",
+ "background"
+ ],
+ "evaluate": [
+ "${python}",
+ "probes/IR-01/evaluate.py",
+ "--seed",
+ "${seed}",
+ "--state",
+ "${artifacts}/state.json",
+ "--events",
+ "${artifacts}/events.jsonl",
+ "--result",
+ "${result}"
+ ],
+ "requires": [
+ "Python 3.12",
+ "wxPython 4.2 or later",
+ "an interactive desktop session (not session 0)"
+ ]
+}
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-02/launch.windows.json b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-02/launch.windows.json
new file mode 100644
index 0000000000..c85712d221
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-02/launch.windows.json
@@ -0,0 +1,47 @@
+{
+ "platform": "windows",
+ "descriptor_version": "1",
+ "task_id": "IR-02",
+ "app": "windows/benchlab.py",
+ "app_args": [
+ "--mode",
+ "irbanner"
+ ],
+ "app_executable": "BenchLab.exe (a copy of pythonw.exe; see launch/windows_session.py)",
+ "sentinel": "windows/sentinel.py",
+ "launcher": [
+ "${python}",
+ "launch/windows_session.py",
+ "--task",
+ "IR-02",
+ "--run-index",
+ "${run_index}",
+ "--condition",
+ "${condition}",
+ "--out",
+ "${artifacts}",
+ "--",
+ "${agent_command}"
+ ],
+ "conditions": [
+ "foreground",
+ "background"
+ ],
+ "evaluate": [
+ "${python}",
+ "probes/IR-02/evaluate.py",
+ "--seed",
+ "${seed}",
+ "--state",
+ "${artifacts}/state.json",
+ "--events",
+ "${artifacts}/events.jsonl",
+ "--result",
+ "${result}"
+ ],
+ "requires": [
+ "Python 3.12",
+ "wxPython 4.2 or later",
+ "an interactive desktop session (not session 0)"
+ ]
+}
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-03/launch.windows.json b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-03/launch.windows.json
new file mode 100644
index 0000000000..3316d8a6c6
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-03/launch.windows.json
@@ -0,0 +1,47 @@
+{
+ "platform": "windows",
+ "descriptor_version": "1",
+ "task_id": "IR-03",
+ "app": "windows/benchlab.py",
+ "app_args": [
+ "--mode",
+ "irunsaved"
+ ],
+ "app_executable": "BenchLab.exe (a copy of pythonw.exe; see launch/windows_session.py)",
+ "sentinel": "windows/sentinel.py",
+ "launcher": [
+ "${python}",
+ "launch/windows_session.py",
+ "--task",
+ "IR-03",
+ "--run-index",
+ "${run_index}",
+ "--condition",
+ "${condition}",
+ "--out",
+ "${artifacts}",
+ "--",
+ "${agent_command}"
+ ],
+ "conditions": [
+ "foreground",
+ "background"
+ ],
+ "evaluate": [
+ "${python}",
+ "probes/IR-03/evaluate.py",
+ "--seed",
+ "${seed}",
+ "--state",
+ "${artifacts}/state.json",
+ "--events",
+ "${artifacts}/events.jsonl",
+ "--result",
+ "${result}"
+ ],
+ "requires": [
+ "Python 3.12",
+ "wxPython 4.2 or later",
+ "an interactive desktop session (not session 0)"
+ ]
+}
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-04/launch.windows.json b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-04/launch.windows.json
new file mode 100644
index 0000000000..71e020ed65
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-04/launch.windows.json
@@ -0,0 +1,47 @@
+{
+ "platform": "windows",
+ "descriptor_version": "1",
+ "task_id": "IR-04",
+ "app": "windows/benchlab.py",
+ "app_args": [
+ "--mode",
+ "irconsent"
+ ],
+ "app_executable": "BenchLab.exe (a copy of pythonw.exe; see launch/windows_session.py)",
+ "sentinel": "windows/sentinel.py",
+ "launcher": [
+ "${python}",
+ "launch/windows_session.py",
+ "--task",
+ "IR-04",
+ "--run-index",
+ "${run_index}",
+ "--condition",
+ "${condition}",
+ "--out",
+ "${artifacts}",
+ "--",
+ "${agent_command}"
+ ],
+ "conditions": [
+ "foreground",
+ "background"
+ ],
+ "evaluate": [
+ "${python}",
+ "probes/IR-04/evaluate.py",
+ "--seed",
+ "${seed}",
+ "--state",
+ "${artifacts}/state.json",
+ "--events",
+ "${artifacts}/events.jsonl",
+ "--result",
+ "${result}"
+ ],
+ "requires": [
+ "Python 3.12",
+ "wxPython 4.2 or later",
+ "an interactive desktop session (not session 0)"
+ ]
+}
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-09/launch.windows.json b/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-09/launch.windows.json
new file mode 100644
index 0000000000..15cdfcf275
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-09/launch.windows.json
@@ -0,0 +1,47 @@
+{
+ "platform": "windows",
+ "descriptor_version": "1",
+ "task_id": "MB-09",
+ "app": "windows/benchlab.py",
+ "app_args": [
+ "--mode",
+ "listdrag"
+ ],
+ "app_executable": "BenchLab.exe (a copy of pythonw.exe; see launch/windows_session.py)",
+ "sentinel": "windows/sentinel.py",
+ "launcher": [
+ "${python}",
+ "launch/windows_session.py",
+ "--task",
+ "MB-09",
+ "--run-index",
+ "${run_index}",
+ "--condition",
+ "${condition}",
+ "--out",
+ "${artifacts}",
+ "--",
+ "${agent_command}"
+ ],
+ "conditions": [
+ "foreground",
+ "background"
+ ],
+ "evaluate": [
+ "${python}",
+ "probes/MB-09/evaluate.py",
+ "--seed",
+ "${seed}",
+ "--state",
+ "${artifacts}/state.json",
+ "--events",
+ "${artifacts}/events.jsonl",
+ "--result",
+ "${result}"
+ ],
+ "requires": [
+ "Python 3.12",
+ "wxPython 4.2 or later",
+ "an interactive desktop session (not session 0)"
+ ]
+}
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-10/launch.windows.json b/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-10/launch.windows.json
new file mode 100644
index 0000000000..74af81789c
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-10/launch.windows.json
@@ -0,0 +1,46 @@
+{
+ "platform": "windows",
+ "descriptor_version": "1",
+ "task_id": "MB-10",
+ "app": "windows/benchlab.py",
+ "app_args": [
+ "--mode",
+ "hover"
+ ],
+ "app_executable": "BenchLab.exe (a copy of pythonw.exe; see launch/windows_session.py)",
+ "sentinel": "windows/sentinel.py",
+ "launcher": [
+ "${python}",
+ "launch/windows_session.py",
+ "--task",
+ "MB-10",
+ "--run-index",
+ "${run_index}",
+ "--condition",
+ "${condition}",
+ "--out",
+ "${artifacts}",
+ "--",
+ "${agent_command}"
+ ],
+ "conditions": [
+ "foreground"
+ ],
+ "evaluate": [
+ "${python}",
+ "probes/MB-10/evaluate.py",
+ "--seed",
+ "${seed}",
+ "--state",
+ "${artifacts}/state.json",
+ "--events",
+ "${artifacts}/events.jsonl",
+ "--result",
+ "${result}"
+ ],
+ "requires": [
+ "Python 3.12",
+ "wxPython 4.2 or later",
+ "an interactive desktop session (not session 0)"
+ ]
+}
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-11/launch.windows.json b/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-11/launch.windows.json
new file mode 100644
index 0000000000..8856ea3234
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-11/launch.windows.json
@@ -0,0 +1,46 @@
+{
+ "platform": "windows",
+ "descriptor_version": "1",
+ "task_id": "MB-11",
+ "app": "windows/benchlab.py",
+ "app_args": [
+ "--mode",
+ "tooltip"
+ ],
+ "app_executable": "BenchLab.exe (a copy of pythonw.exe; see launch/windows_session.py)",
+ "sentinel": "windows/sentinel.py",
+ "launcher": [
+ "${python}",
+ "launch/windows_session.py",
+ "--task",
+ "MB-11",
+ "--run-index",
+ "${run_index}",
+ "--condition",
+ "${condition}",
+ "--out",
+ "${artifacts}",
+ "--",
+ "${agent_command}"
+ ],
+ "conditions": [
+ "foreground"
+ ],
+ "evaluate": [
+ "${python}",
+ "probes/MB-11/evaluate.py",
+ "--seed",
+ "${seed}",
+ "--state",
+ "${artifacts}/state.json",
+ "--events",
+ "${artifacts}/events.jsonl",
+ "--result",
+ "${result}"
+ ],
+ "requires": [
+ "Python 3.12",
+ "wxPython 4.2 or later",
+ "an interactive desktop session (not session 0)"
+ ]
+}
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_taskset.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_taskset.py
index d3f41fb537..af446cf76e 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_taskset.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_taskset.py
@@ -35,10 +35,8 @@ class TaskSetTest(unittest.TestCase):
def test_platforms(self) -> None:
by_id = {e["id"]: e for e in v2.load_taskset()["tasks"]}
- for tid in ("CDB-G02", "CDB-G03", "CDB-G04"):
+ for tid in v2.V2_TASKS:
self.assertEqual(by_id[tid]["os"], ["macos", "linux", "windows"], tid)
- for tid in ("MB-09", "MB-10", "MB-11", "IR-01", "IR-02", "IR-03", "IR-04"):
- self.assertEqual(by_id[tid]["os"], ["macos", "linux"], tid)
self.assertIn("GnuCash", by_id["CDB-G03"]["apps"])
def test_task_sets_and_splits(self) -> None:
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_windows_desktop.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_windows_desktop.py
new file mode 100644
index 0000000000..6ecc8532a5
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_windows_desktop.py
@@ -0,0 +1,84 @@
+# SPDX-License-Identifier: FSL-1.1-MIT
+# Copyright (c) 2026 Cua AI, Inc.
+
+"""Windows desktop contracts (no model): runs on a Windows interactive desktop with wxPython and pywinauto.
+
+Through launch/windows_session.py, the real launcher:
+* every BenchLab task is solvable through the GUI: the scripted oracle (windows/oracle.py, real input)
+ passes the task's checker, with no GUI-only violation;
+* an agent that does nothing fails the checker, and the sentinel measures all eight A22 criteria as held;
+* a deliberate disturbance (focus steal, raise, pointer, clipboard) is seen by the sentinel.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+import json
+import subprocess
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+BENCH = HERE.parent
+SESSION = BENCH / "launch" / "windows_session.py"
+ORACLE = BENCH / "windows" / "oracle.py"
+TASKS = ("MB-09", "MB-10", "MB-11", "IR-01", "IR-02", "IR-03", "IR-04")
+
+
+def _available() -> bool:
+ return sys.platform == "win32" and all(
+ importlib.util.find_spec(m) is not None for m in ("wx", "pywinauto")
+ )
+
+
[email protected](_available(), "needs a Windows desktop with wxPython and pywinauto")
+class WindowsDesktopTest(unittest.TestCase):
+ def trial(self, task: str, condition: str, agent: list[str]) -> dict:
+ with tempfile.TemporaryDirectory() as tmp:
+ out = Path(tmp) / task
+ done = subprocess.run(
+ [sys.executable, str(SESSION), "--task", task, "--run-index", "0",
+ "--condition", condition, "--out", str(out), "--timeout", "150", "--", *agent],
+ capture_output=True, text=True, timeout=300,
+ ) # fmt: skip
+ self.assertEqual(done.returncode, 0, done.stdout + done.stderr)
+ return json.loads((out / "row.json").read_text("utf-8"))
+
+ def test_oracle_solves_every_task(self) -> None:
+ for task in TASKS:
+ with self.subTest(task=task):
+ row = self.trial(
+ task,
+ "foreground",
+ [sys.executable, str(ORACLE), "--mode", "{mode}", "--seed", "{seed}"],
+ )
+ failed = [k for k, v in row["evaluation"]["checks"].items() if not v["passed"]]
+ self.assertTrue(row["checker_passed"], failed)
+ self.assertFalse(row["gui_only"]["violation"], row["gui_only"])
+ self.assertTrue(row["passed"])
+
+ def test_noop_fails_and_leaves_the_user_undisturbed(self) -> None:
+ row = self.trial(
+ "IR-02", "background", [sys.executable, "-c", "import time; time.sleep(8)"]
+ )
+ self.assertFalse(row["checker_passed"])
+ self.assertTrue(row["sentinel_front_at_start"])
+ self.assertEqual(row["a22"]["unmeasured"], [])
+ self.assertTrue(row["a22"]["user_undisturbed"], row["a22"]["criteria"])
+
+ def test_disturbance_is_seen(self) -> None:
+ row = self.trial(
+ "IR-02",
+ "background",
+ [sys.executable, str(ORACLE), "--mode", "{mode}", "--seed", "{seed}", "--disturb"],
+ )
+ criteria = row["a22"]["criteria"]
+ for key in ("key_focus", "window_raised", "pointer", "clipboard", "activation"):
+ self.assertFalse(criteria[key], key)
+ self.assertFalse(row["a22"]["user_undisturbed"])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_windows_tools.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_windows_tools.py
new file mode 100644
index 0000000000..05221dcbcd
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_windows_tools.py
@@ -0,0 +1,197 @@
+# SPDX-License-Identifier: FSL-1.1-MIT
+# Copyright (c) 2026 Cua AI, Inc.
+
+"""Windows port contracts that hold on every OS (no GUI, no model): the GUI-only guard, the brief, the
+launch descriptors, parity of the BenchLab modes and events with the Linux port, and that a log written in
+the Windows sentinel's shape measures all eight A22 criteria."""
+
+from __future__ import annotations
+
+import ast
+import json
+import re
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+BENCH = HERE.parent
+sys.path.insert(0, str(BENCH))
+sys.path.insert(0, str(BENCH / "windows"))
+sys.path.insert(0, str(BENCH / "launch"))
+sys.path.insert(0, str(BENCH / "swift"))
+import bench_v2 as v2 # noqa: E402
+import guard as G # noqa: E402
+import summarize_sentinel # noqa: E402
+import windows_session as WS # noqa: E402
+
+BENCHLAB_TASKS = ("MB-09", "MB-10", "MB-11", "IR-01", "IR-02", "IR-03", "IR-04")
+
+
+def _emitted(path: Path) -> set[str]:
+ return set(re.findall(r'self\.emit\(\s*"([a-z_A-Z]+)"', path.read_text("utf-8")))
+
+
+def _methods(path: Path, cls: str) -> set[str]:
+ tree = ast.parse(path.read_text("utf-8"))
+ for node in tree.body:
+ if isinstance(node, ast.ClassDef) and node.name == cls:
+ return {n.name for n in node.body if isinstance(n, ast.FunctionDef)}
+ return set()
+
+
+def _modes(path: Path) -> tuple:
+ tree = ast.parse(path.read_text("utf-8"))
+ for node in tree.body:
+ if isinstance(node, ast.Assign) and getattr(node.targets[0], "id", "") == "MODES":
+ return ast.literal_eval(node.value)
+ return ()
+
+
+class ParityTest(unittest.TestCase):
+ """The Windows BenchLab and sentinel write what the Linux ones write, so one set of checkers serves both."""
+
+ def test_same_modes(self) -> None:
+ self.assertEqual(_modes(BENCH / "windows/benchlab.py"), _modes(BENCH / "linux/benchlab.py"))
+ for mode in _modes(BENCH / "windows/benchlab.py"):
+ self.assertIn(mode, _methods(BENCH / "windows/benchlab.py", "Lab"))
+
+ def test_same_benchlab_events(self) -> None:
+ self.assertEqual(
+ _emitted(BENCH / "windows/benchlab.py"), _emitted(BENCH / "linux/benchlab.py")
+ )
+
+ def test_sentinel_events_cover_linux(self) -> None:
+ linux = _emitted(BENCH / "linux/sentinel.py") - {
+ "ax_focus"
+ } # AT-SPI focus trace: diagnostic only
+ self.assertLessEqual(linux, _emitted(BENCH / "windows/sentinel.py"))
+
+ def test_every_benchlab_task_has_a_windows_descriptor(self) -> None:
+ manifest = {t["id"]: t for t in v2.load_taskset()["tasks"]}
+ for tid in BENCHLAB_TASKS:
+ spec = json.loads((BENCH / "probes" / tid / "task.json").read_text("utf-8"))
+ launch = json.loads((BENCH / "probes" / tid / "launch.windows.json").read_text("utf-8"))
+ self.assertEqual(launch["platform"], "windows")
+ self.assertEqual(launch["app_args"], spec["app_args"], tid)
+ self.assertEqual(launch["conditions"], manifest[tid]["conditions"], tid)
+ self.assertIn("windows", manifest[tid]["os"])
+ self.assertEqual(set(WS.TASKS), set(BENCHLAB_TASKS))
+
+
+class BriefTest(unittest.TestCase):
+ def test_platform_words_and_background_paragraph(self) -> None:
+ for tid in BENCHLAB_TASKS:
+ seed = 123456
+ fg = WS.render_brief(tid, seed, "foreground")
+ bg = WS.render_brief(tid, seed, "background")
+ self.assertNotIn("your Mac", fg, tid)
+ self.assertIn("on this Windows PC", fg, tid)
+ self.assertNotIn(v2.BACKGROUND_BRIEF, fg)
+ self.assertTrue(bg.rstrip().endswith(v2.BACKGROUND_BRIEF), tid)
+ self.assertNotIn("{{", bg, tid)
+
+
+class GuardTest(unittest.TestCase):
+ ROWS = [
+ (4, 0, "System"),
+ (100, 4, "explorer.exe"),
+ (200, 100, "runner.exe"),
+ (210, 200, "claude.exe"),
+ (220, 210, "cmd.exe"), # an npm shim under the runner: not a violation
+ (230, 100, "powershell.exe"), # started from the desktop: a violation
+ (240, 100, "WindowsTerminal.exe"),
+ (250, 999, "pwsh.exe"), # parent gone, but seen under the runner before: the runner's
+ (260, 100, "notepad.exe"),
+ ]
+
+ def test_names(self) -> None:
+ self.assertEqual(
+ G.name_of(r"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe"), "powershell"
+ )
+ self.assertEqual(G.name_of("CMD.EXE"), "cmd")
+ self.assertEqual(G.name_of("wt"), "wt")
+
+ def test_descendants(self) -> None:
+ self.assertEqual(G.descendants(self.ROWS, 200), {200, 210, 220})
+
+ def test_new_side_doors(self) -> None:
+ hits = G.new_side_doors(self.ROWS, baseline={4, 100}, runner=200, own={250})
+ self.assertEqual(sorted(hits), [(230, "Windows PowerShell"), (240, "Windows Terminal")])
+ self.assertEqual(
+ G.new_side_doors(self.ROWS, baseline={p for p, _, _ in self.ROWS}, runner=200), []
+ )
+
+ def test_tool_input_patterns(self) -> None:
+ hard, _ = G.scan_tool_use(
+ "mcp__x__exec", {"code": "subprocess.run(['powershell', '-c', 'dir'])"}
+ )
+ self.assertIn("mcp__x__exec:subprocess", hard)
+ self.assertIn("mcp__x__exec:powershell", hard)
+ hard, _ = G.scan_tool_use("mcp__x__type_text", {"text": "Start-Process notepad"})
+ self.assertEqual(hard, ["mcp__x__type_text:Start-Process"])
+ self.assertEqual(G.scan_tool_use("mcp__x__click", {"element": "Apply"}), ([], []))
+ self.assertEqual(G.scan_tool_use("Bash", {"command": "ls"})[0], ["builtin:Bash"])
+
+ def test_front_grace(self) -> None:
+ watch = G.FrontWatch()
+ self.assertIsNone(watch.sample(r"C:\x\cmd.exe"))
+ self.assertIsNone(watch.sample(r"C:\x\cmd.exe"))
+ self.assertEqual(watch.sample(r"C:\x\cmd.exe"), "Command Prompt")
+ self.assertIsNone(watch.sample(r"C:\x\BenchLab.exe"))
+ self.assertIsNone(watch.sample(r"C:\x\cmd.exe"))
+
+
+def _sentinel_log(path: Path, disturb: bool) -> None:
+ """A log in the exact shape windows/sentinel.py writes."""
+ lines = [{"t": 0, "ev": "ready", "hwnd": 42}]
+ me = {"bid": "ai.cua.benchsentinel", "pid": 7}
+ lab = {"bid": "BenchLab", "pid": 9}
+
+ def sample(t, front=me, mouse=(500, 500), clip=10, above=(), sheet=False, ev="s"):
+ return {"t": t, "ev": ev, "front": front, "mouse": list(mouse), "clip": clip, "sheet": sheet,
+ "above": list(above), "above_n": len(above), "onspace": True} # fmt: skip
+
+ lines.append(sample(1, ev="armed"))
+ for i in range(2, 40):
+ if disturb and i == 20:
+ lines.append(sample(i, front=lab, ev="front"))
+ lines.append({"t": i, "ev": "didResignActive"})
+ lines.append({"t": i, "ev": "didResignKey"})
+ lines.append(sample(i, front=lab, mouse=(700, 650), clip=11, above=("BenchLab",)))
+ lines.append({"t": i, "ev": "user_blocked"})
+ continue
+ lines.append(sample(i, clip=11 if disturb and i > 20 else 10))
+ if i % 3 == 0:
+ lines.append({"t": i, "ev": "user_type"})
+ lines.append({"t": 40, "ev": "user_text", "intact": True})
+ lines.append(sample(41, clip=11 if disturb else 10, ev="disarmed"))
+ path.write_text("\n".join(json.dumps(x) for x in lines) + "\n", "utf-8")
+
+
+class SentinelLogTest(unittest.TestCase):
+ def test_clean_log_measures_all_eight_criteria(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ log = Path(tmp) / "s.jsonl"
+ _sentinel_log(log, disturb=False)
+ d = summarize_sentinel.summarize(str(log))
+ score = v2.multiplayer_score(d, True, True, "background")
+ self.assertEqual(score["unmeasured"], [])
+ self.assertTrue(all(score["criteria"].values()), score["criteria"])
+ self.assertTrue(score["background_completion"])
+
+ def test_disturbance_is_seen(self) -> None:
+ with tempfile.TemporaryDirectory() as tmp:
+ log = Path(tmp) / "s.jsonl"
+ _sentinel_log(log, disturb=True)
+ d = summarize_sentinel.summarize(str(log))
+ criteria = v2.multiplayer_score(d, True, True, "background")["criteria"]
+ for key in ("key_focus", "window_raised", "typing", "pointer", "clipboard", "activation"):
+ self.assertFalse(criteria[key], key)
+ self.assertTrue(criteria["space"])
+ self.assertTrue(criteria["dialog"])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/windows/benchlab.py b/libs/cua-driver-bench/automated-eval/macos_bench/windows/benchlab.py
new file mode 100644
index 0000000000..27ffc40568
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/windows/benchlab.py
@@ -0,0 +1,635 @@
+# SPDX-License-Identifier: FSL-1.1-MIT
+# Copyright (c) 2026 Cua AI, Inc.
+
+"""BenchLab for Windows: the seven Bench v2 BenchLab modes with native Win32 controls (wxPython).
+
+A port of linux/benchlab.py. Seeds, parameters, the state file and the event log are the same on every OS
+(probes/_common/benchlab_common.py), so the task checkers run unchanged. Controls are native (Button,
+Edit, ComboBox, CheckBox, SysListView32, tooltips_class32), so UI Automation exposes real elements.
+
+Windows-specific behaviour, kept close to the Linux port:
+* Dialogs are owned by the BenchLab window, which is disabled while one is open (modal). When BenchLab is
+ not the foreground window, a dialog is shown without activation and stays just above BenchLab in the
+ z-order, as the Linux port's dialogs do (set_focus_on_map False), so the app never raises itself over
+ another window.
+* Field values are also polled every 250 ms: UI Automation can change a combo box with CB_SETCURSEL, which
+ sends no notification, and the state file must still follow the control.
+* MB-09: rows reorder by a native list drag (LVN_BEGINDRAG, then the button release over a row).
+* MB-11: a tooltip counts as shown when BenchLab's own tooltip window is visible (polled every 100 ms).
+"""
+
+from __future__ import annotations
+
+import argparse
+import ctypes
+import json
+import os
+import sys
+import time
+from ctypes import wintypes
+from pathlib import Path
+
+import wx
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "probes" / "_common"))
+import benchlab_common as C # noqa: E402
+
+MODES = ("listdrag", "hover", "tooltip", "irmodal", "irbanner", "irunsaved", "irconsent")
+ROW_HEIGHT = 40
+
+user32 = ctypes.WinDLL("user32", use_last_error=True)
+user32.GetForegroundWindow.restype = wintypes.HWND
+user32.GetWindow.restype = wintypes.HWND
+user32.GetWindow.argtypes = [wintypes.HWND, wintypes.UINT]
+user32.SetWindowPos.argtypes = [
+ wintypes.HWND, wintypes.HWND, ctypes.c_int, ctypes.c_int, ctypes.c_int, ctypes.c_int, wintypes.UINT
+] # fmt: skip
+user32.EnableWindow.argtypes = [wintypes.HWND, wintypes.BOOL]
+user32.IsWindowVisible.argtypes = [wintypes.HWND]
+user32.GetClassNameW.argtypes = [wintypes.HWND, wintypes.LPWSTR, ctypes.c_int]
+user32.GetWindowTextW.argtypes = [wintypes.HWND, wintypes.LPWSTR, ctypes.c_int]
+WNDENUMPROC = ctypes.WINFUNCTYPE(wintypes.BOOL, wintypes.HWND, wintypes.LPARAM)
+user32.EnumThreadWindows.argtypes = [wintypes.DWORD, WNDENUMPROC, wintypes.LPARAM]
+kernel32 = ctypes.WinDLL("kernel32")
+
+GW_HWNDPREV = 3
+HWND_TOP = 0
+SWP_NOSIZE, SWP_NOMOVE, SWP_NOACTIVATE = 0x1, 0x2, 0x10
+
+
+def _text(hwnd: int, getter) -> str:
+ buf = ctypes.create_unicode_buffer(256)
+ getter(hwnd, buf, 256)
+ return buf.value
+
+
+def visible_tooltips() -> list[str]:
+ """Texts of the visible tooltip windows of this (the UI) thread."""
+ found: list[str] = []
+
+ def each(hwnd, _):
+ if _text(hwnd, user32.GetClassNameW) == "tooltips_class32" and user32.IsWindowVisible(hwnd):
+ found.append(_text(hwnd, user32.GetWindowTextW))
+ return True
+
+ user32.EnumThreadWindows(kernel32.GetCurrentThreadId(), WNDENUMPROC(each), 0)
+ return found
+
+
+class Lab:
+ def __init__(self, args: argparse.Namespace) -> None:
+ self.args = args
+ self.state = {"mode": args.mode, "seed": args.seed, "seq": 0}
+ self.params = getattr(C, "derive_" + args.mode)(args.seed)
+ self.fields: dict[str, wx.Window] = {}
+ self.last_values: dict = {}
+ self.loading = False
+ self.dialog_open: wx.Dialog | None = None
+ args.events.parent.mkdir(parents=True, exist_ok=True)
+ self.events = args.events.open("w", buffering=1, encoding="utf-8")
+ self.frame = wx.Frame(None, title=f"BenchLab — {args.mode}", size=(760, 540))
+ self.frame.SetName("BenchLab")
+ if args.frame:
+ x, y, w, h = map(int, args.frame.split(","))
+ self.frame.SetPosition((x, y))
+ self.frame.SetSize((w, h))
+ self.panel = wx.Panel(self.frame)
+ self.box = wx.BoxSizer(wx.VERTICAL)
+ outer = wx.BoxSizer(wx.VERTICAL)
+ outer.Add(self.box, 1, wx.EXPAND | wx.ALL, 24)
+ self.panel.SetSizer(outer)
+ self.hidden: list[wx.Window] = []
+ getattr(self, args.mode)()
+ self.panel.Layout()
+ if args.no_activate:
+ self.frame.ShowWithoutActivating()
+ else:
+ self.frame.Show()
+ for widget in self.hidden:
+ widget.Hide()
+ self.panel.Layout()
+ self.last_values = self.values()
+ self.poll = wx.Timer(self.frame)
+ self.frame.Bind(wx.EVT_TIMER, lambda _: self.poll_values(), self.poll)
+ self.poll.Start(250)
+ self.emit("app_start", mode=args.mode, seed=args.seed)
+
+ # ------------------------------------------------------------------ plumbing
+
+ def emit(self, event: str, **details: object) -> None:
+ self.state["seq"] += 1
+ record = {
+ "seq": self.state["seq"],
+ "t": time.time() * 1000,
+ "type": event,
+ "details": details,
+ }
+ self.events.write(json.dumps(record) + "\n")
+ self.args.state.parent.mkdir(parents=True, exist_ok=True)
+ tmp = self.args.state.with_suffix(".tmp")
+ tmp.write_text(json.dumps(self.state), encoding="utf-8")
+ os.replace(tmp, self.args.state)
+
+ def label(self, text: str, parent: wx.Window | None = None, sizer: wx.Sizer | None = None):
+ label = wx.StaticText(parent or self.panel, label=text)
+ (sizer or self.box).Add(label, 0, wx.BOTTOM, 4)
+ return label
+
+ def button(self, text: str, callback, parent=None, sizer=None) -> wx.Button:
+ button = wx.Button(parent or self.panel, label=text)
+ button.Bind(wx.EVT_BUTTON, lambda _: callback())
+ (sizer or self.box).Add(button, 0, wx.TOP | wx.BOTTOM | wx.RIGHT, 4)
+ return button
+
+ def field(self, name: str, choices: list[str] | None = None, value: str = ""):
+ # The label comes first so UI Automation names the control after it.
+ caption = self.label(name.capitalize())
+ if choices:
+ widget = wx.Choice(self.panel, choices=choices, name=name.capitalize())
+ widget.SetSelection(choices.index(value) if value in choices else 0)
+ widget.Bind(wx.EVT_CHOICE, lambda _: self.edit(name))
+ else:
+ widget = wx.TextCtrl(self.panel, value=value, name=name.capitalize())
+ widget.Bind(wx.EVT_TEXT, lambda _: self.edit(name))
+ widget.caption = caption
+ self.fields[name] = widget
+ self.box.Add(widget, 0, wx.EXPAND | wx.BOTTOM, 8)
+ return widget
+
+ def values(self) -> dict:
+ out = {}
+ for key, widget in self.fields.items():
+ if isinstance(widget, wx.Choice):
+ out[key] = widget.GetStringSelection()
+ elif isinstance(widget, wx.CheckBox):
+ out[key] = widget.GetValue()
+ else:
+ out[key] = widget.GetValue()
+ return out
+
+ def poll_values(self) -> None:
+ if self.loading or not self.fields:
+ return
+ now = self.values()
+ for name in self.fields:
+ if now.get(name) != self.last_values.get(name):
+ self.edit(name)
+
+ def edit(self, name: str) -> None:
+ if self.loading:
+ return
+ values = self.values()
+ if values.get(name) == self.last_values.get(name):
+ return # the poll and the control's own notification report one change once
+ self.last_values = values
+ self.state["fields"] = values
+ if self.args.mode == "irunsaved":
+ self.state["dirty"] = True
+ self.emit("field_edit", field=name, value=values[name])
+ if self.args.mode == "irbanner" and not self.state["banner_shown"]:
+ self.state["banner_shown"] = True
+ wx.CallLater(200, self.show_banner)
+ if self.args.mode == "irconsent" and not self.state["consent_shown"]:
+ self.state["consent_shown"] = True
+ wx.CallLater(250, self.show_consent)
+
+ def set_loaded(self, name: str, value) -> None:
+ widget = self.fields[name]
+ if isinstance(widget, wx.Choice):
+ widget.SetStringSelection(value)
+ else:
+ widget.ChangeValue(value)
+
+ def dialog(self, title: str, buttons: list[tuple[str, str]], callback) -> None:
+ dialog = wx.Dialog(self.frame, title=title, style=wx.CAPTION | wx.SYSTEM_MENU)
+ sizer = wx.BoxSizer(wx.VERTICAL)
+ sizer.Add(wx.StaticText(dialog, label=title), 0, wx.ALL, 24)
+ row = wx.BoxSizer(wx.HORIZONTAL)
+ for text, answer in buttons:
+ b = wx.Button(dialog, label=text)
+ b.Bind(wx.EVT_BUTTON, lambda _, a=answer: self.dialog_answer(dialog, callback, a))
+ row.Add(b, 0, wx.ALL, 6)
+ sizer.Add(row, 0, wx.ALIGN_RIGHT | wx.ALL, 12)
+ dialog.SetSizerAndFit(sizer)
+ # A close from the title bar or Esc is not an answer; keep the dialog until a button is used.
+ dialog.Bind(wx.EVT_CLOSE, lambda e: e.Veto() if e.CanVeto() else None)
+ dialog.Bind(
+ wx.EVT_CHAR_HOOK, lambda e: None if e.GetKeyCode() == wx.WXK_ESCAPE else e.Skip()
+ )
+ dialog.CentreOnParent()
+ self.dialog_open = dialog
+ frame = self.frame.GetHandle()
+ user32.EnableWindow(frame, False)
+ if user32.GetForegroundWindow() == frame:
+ dialog.Show()
+ else:
+ dialog.ShowWithoutActivating()
+ above = user32.GetWindow(frame, GW_HWNDPREV) or HWND_TOP
+ user32.SetWindowPos(
+ dialog.GetHandle(), above, 0, 0, 0, 0, SWP_NOSIZE | SWP_NOMOVE | SWP_NOACTIVATE
+ )
+
+ def dialog_answer(self, dialog: wx.Dialog, callback, answer: str) -> None:
+ user32.EnableWindow(self.frame.GetHandle(), True)
+ self.dialog_open = None
+ dialog.Destroy()
+ callback(answer)
+
+ # ------------------------------------------------------------------ MB-09 listdrag
+
+ def listdrag(self) -> None:
+ self.state.update(
+ order=list(self.params["initial"]),
+ move_count=0,
+ drag_begins=0,
+ done=False,
+ done_count=0,
+ )
+ self.label("Drag the rows to reorder them, then press Done")
+ self.rows = wx.ListCtrl(
+ self.panel, style=wx.LC_REPORT | wx.LC_SINGLE_SEL | wx.LC_NO_HEADER, name="Tasks"
+ )
+ self.rows.InsertColumn(0, "Task", width=640)
+ spacer = wx.ImageList(1, ROW_HEIGHT) # sets the row height
+ self.rows.AssignImageList(spacer, wx.IMAGE_LIST_SMALL)
+ self.box.Add(self.rows, 1, wx.EXPAND | wx.BOTTOM, 8)
+ self.drag_item: str | None = None
+ self.draw_rows()
+ self.rows.Bind(wx.EVT_LIST_BEGIN_DRAG, self.drag_begin)
+ self.rows.Bind(wx.EVT_LEFT_UP, self.drag_end)
+ self.rows.Bind(wx.EVT_MOUSE_CAPTURE_LOST, lambda _: self.drag_cancel())
+ self.button("Done", self.list_done)
+
+ def draw_rows(self) -> None:
+ self.rows.DeleteAllItems()
+ for index, item in enumerate(self.state["order"]):
+ self.rows.InsertItem(index, item)
+
+ def drag_begin(self, event: wx.ListEvent) -> None:
+ item = self.state["order"][event.GetIndex()]
+ self.drag_item = item
+ self.state["drag_begins"] += 1
+ self.emit("drag_begin", item=item)
+ if not self.rows.HasCapture():
+ self.rows.CaptureMouse()
+
+ def drag_cancel(self) -> None:
+ self.drag_item = None
+
+ def drag_end(self, event: wx.MouseEvent) -> None:
+ event.Skip()
+ item = self.drag_item
+ if self.rows.HasCapture():
+ self.rows.ReleaseMouse()
+ self.drag_item = None
+ if item is None or item not in self.state["order"]:
+ return
+ pos = event.GetPosition()
+ width, height = self.rows.GetClientSize()
+ if not (0 <= pos.x < width and 0 <= pos.y < height):
+ return # released outside the list: no drop
+ index, _ = self.rows.HitTest(pos)
+ if index == wx.NOT_FOUND:
+ index = (
+ len(self.state["order"]) - 1
+ ) # the empty area below the rows belongs to the last row
+ order = self.state["order"]
+ source = order.index(item)
+ order.pop(source)
+ order.insert(index, item)
+ self.state["move_count"] += 1
+ self.emit("reorder", **{"from": source, "to": index, "item": item, "order": list(order)})
+ wx.CallAfter(self.draw_rows)
+
+ def list_done(self) -> None:
+ self.state.update(
+ done=True, done_count=self.state["done_count"] + 1, done_order=list(self.state["order"])
+ )
+ self.emit("done_click", order=self.state["done_order"])
+
+ # ------------------------------------------------------------------ MB-10 hover
+
+ def hover(self) -> None:
+ self.state.update(
+ overlay_names=self.params["names"], clicks=[], hover_enter_count=0, hover_exit_count=0
+ )
+ zone = wx.Panel(self.panel, name="Hot zone", style=wx.BORDER_SIMPLE)
+ content = wx.BoxSizer(wx.VERTICAL)
+ self.label("Hover here to reveal actions", zone, content)
+ actions_panel = wx.Panel(zone)
+ actions = wx.BoxSizer(wx.HORIZONTAL)
+ for name in self.params["names"]:
+ self.button(name, lambda n=name: self.hover_click(n), actions_panel, actions)
+ actions_panel.SetSizer(actions)
+ content.Add(actions_panel, 0, wx.TOP, 10)
+ wrap = wx.BoxSizer(wx.VERTICAL)
+ wrap.Add(content, 1, wx.EXPAND | wx.ALL, 16)
+ zone.SetSizer(wrap)
+ self.box.Add(zone, 1, wx.EXPAND)
+ self.zone, self.actions = zone, actions_panel
+ self.hidden = [actions_panel]
+ self.visible = False
+ self.inside = False
+ self.crossing = 0
+ for widget in [zone, *self._descendants(zone)]:
+ widget.Bind(wx.EVT_ENTER_WINDOW, self.hover_enter)
+ widget.Bind(wx.EVT_LEAVE_WINDOW, self.hover_leave)
+
+ @staticmethod
+ def _descendants(widget: wx.Window) -> list[wx.Window]:
+ out = []
+ for child in widget.GetChildren():
+ out += [child, *Lab._descendants(child)]
+ return out
+
+ def hover_enter(self, event: wx.MouseEvent) -> None:
+ event.Skip()
+ # Moving between the zone and its own children is not a crossing (GTK's NotifyType.INFERIOR).
+ self.crossing += 1
+ if not self.inside:
+ self.hover_set(True)
+
+ def hover_leave(self, event: wx.MouseEvent) -> None:
+ event.Skip()
+ token = self.crossing + 1
+ self.crossing = token
+ wx.CallLater(40, lambda: self.crossing == token and self.inside and self.hover_set(False))
+
+ def hover_set(self, entered: bool) -> None:
+ self.inside = entered
+ self.state["hover_enter_count" if entered else "hover_exit_count"] += 1
+ self.emit("mouse", region="hot_zone", phase="mouseEntered" if entered else "mouseExited")
+ self.visible = entered
+ self.actions.Show(entered)
+ self.zone.Layout()
+ if entered:
+ self.emit("overlay_shown")
+
+ def hover_click(self, name: str) -> None:
+ self.state["clicks"].append({"name": name})
+ self.state.update(last_clicked=name, last_click_overlay_visible=self.visible)
+ self.emit("overlay_click", name=name, overlay_visible=self.visible)
+
+ # ------------------------------------------------------------------ MB-11 tooltip
+
+ def tooltip(self) -> None:
+ self.state.update(submit_count=0, status="Ready")
+ icons = wx.BoxSizer(wx.HORIZONTAL)
+ self.box.Add(icons, 1, wx.EXPAND)
+ self.tip_icon: str | None = None
+ self.tip_shown = False
+ self.tip_codes = dict(zip(self.params["codes"], self.params["order"]))
+ for index, icon in enumerate(self.params["order"]):
+ cell = wx.Panel(self.panel, size=(120, 80), name=icon)
+ text = wx.StaticText(cell, label=icon, pos=(30, 30), name=icon)
+ for widget in (cell, text):
+ widget.SetToolTip(self.params["codes"][index])
+ widget.Bind(
+ wx.EVT_ENTER_WINDOW, lambda e, i=icon: (setattr(self, "tip_icon", i), e.Skip())
+ )
+ icons.Add(cell, 1, wx.EXPAND | wx.ALL, 12)
+ self.field("code")
+ self.button("Submit", self.tooltip_submit)
+ self.tip_timer = wx.Timer(self.frame)
+ self.frame.Bind(wx.EVT_TIMER, lambda _: self.tooltip_poll(), self.tip_timer)
+ self.tip_timer.Start(100)
+
+ def tooltip_poll(self) -> None:
+ shown = [t for t in visible_tooltips() if t]
+ if shown and not self.tip_shown:
+ icon = self.tip_codes.get(shown[0], self.tip_icon)
+ if icon is not None:
+ self.emit("tooltip_query", icon=icon)
+ self.tip_shown = bool(shown)
+
+ def tooltip_submit(self) -> None:
+ value = self.values()["code"]
+ self.state.update(
+ submit_count=self.state["submit_count"] + 1, status="Submitted", submitted_value=value
+ )
+ self.emit("submit", value=value)
+
+ # ------------------------------------------------------------------ IR-01 irmodal
+
+ def irmodal(self) -> None:
+ self.state.update(
+ prompt_answers=[], submit_count=0, page=1, prompt_state="not_shown", next_count=0
+ )
+ self.field("name")
+ self.field("email")
+ self.step1 = [
+ w for f in ("name", "email") for w in (self.fields[f].caption, self.fields[f])
+ ]
+ self.field("team", C.IR_TEAMS)
+ self.field("seats")
+ self.step2 = [
+ w for f in ("team", "seats") for w in (self.fields[f].caption, self.fields[f])
+ ]
+ self.hidden = list(self.step2)
+ self.submit_button = self.button("Next", self.permission)
+ self.state["fields"] = self.values()
+
+ def permission(self) -> None:
+ self.state["next_count"] += 1
+ self.emit("next", count=self.state["next_count"], values=self.values())
+ self.state["prompt_state"] = "open"
+ self.emit("permission_prompt_shown", resource=self.params["resource"])
+ self.dialog(
+ f"Allow BenchLab to access {self.params['resource']}?",
+ [("Don’t Allow", "deny"), ("Allow", "allow")],
+ self.permission_answer,
+ )
+
+ def permission_answer(self, answer: str) -> None:
+ self.state["prompt_answers"].append(answer)
+ self.state.update(page=2, prompt_state="denied" if answer == "deny" else "allowed")
+ self.emit("permission_answer", answer=answer, resource=self.params["resource"])
+ for widget in self.step1:
+ widget.Hide()
+ for widget in self.step2:
+ widget.Show()
+ self.submit_button.Destroy()
+ self.button("Submit", self.form_submit)
+ self.panel.Layout()
+
+ def form_submit(self) -> None:
+ values = self.values()
+ self.state.update(submit_count=self.state["submit_count"] + 1, submitted=values)
+ self.emit("submit", values=values)
+
+ # ------------------------------------------------------------------ IR-02 irbanner
+
+ def irbanner(self) -> None:
+ self.state.update(
+ banner_shown=False, banner_actions=[], banner_body_clicks=0, apply_count=0
+ )
+ self.label(self.params["item"])
+ self.field("quantity", value="1")
+ self.field("priority", C.PRIORITIES)
+ self.apply_button = self.button("Apply", self.apply)
+ # The banner is a sibling drawn over the Apply button (Gtk.Overlay in the Linux port).
+ self.banner = wx.Panel(self.panel, name="Notification", style=wx.BORDER_SIMPLE)
+ self.banner.SetBackgroundColour(wx.Colour(255, 244, 200))
+ row = wx.BoxSizer(wx.HORIZONTAL)
+ text = wx.StaticText(self.banner, label="Update available")
+ row.Add(text, 0, wx.ALIGN_CENTER_VERTICAL | wx.ALL, 8)
+ for label, action in (
+ ("Restart Now", "restart"),
+ ("Later", "later"),
+ ("Dismiss", "dismiss"),
+ ):
+ self.button(label, lambda a=action: self.banner_action(a), self.banner, row)
+ self.banner.SetSizerAndFit(row)
+ for widget in (self.banner, text):
+ widget.Bind(wx.EVT_LEFT_DOWN, self.banner_body_click)
+ self.hidden = [self.banner]
+ self.state["fields"] = self.values()
+
+ def place_banner(self) -> None:
+ x, y = self.apply_button.GetPosition()
+ self.banner.SetPosition((max(0, x - 8), max(0, y - 8)))
+ self.banner.Raise()
+
+ def banner_body_click(self, event: wx.MouseEvent) -> None:
+ event.Skip()
+ self.state["banner_body_clicks"] += 1
+ self.emit("banner_body_click", count=self.state["banner_body_clicks"])
+
+ def show_banner(self) -> None:
+ self.place_banner()
+ self.banner.Show()
+ self.emit("banner_shown")
+
+ def banner_action(self, action: str) -> None:
+ self.state["banner_actions"].append(action)
+ if action == "restart":
+ self.loading = True
+ self.fields["quantity"].ChangeValue("1")
+ self.fields["priority"].SetSelection(0)
+ self.loading = False
+ self.last_values = self.values()
+ self.state["fields"] = self.last_values
+ self.banner.Hide()
+ self.emit("banner_action", action=action)
+
+ def apply(self) -> None:
+ self.state["apply_count"] += 1
+ self.emit("apply", values=self.values())
+
+ # ------------------------------------------------------------------ IR-03 irunsaved
+
+ def irunsaved(self) -> None:
+ self.state.update(
+ saved=[
+ {"title": t, "status": C.IR_INITIAL_STATUS, "body": f"Working notes for {t}."}
+ for t in self.params["titles"]
+ ],
+ dirty=False,
+ answers=[],
+ done_count=0,
+ )
+ self.selected = 0
+ for index, title in enumerate(self.params["titles"]):
+ self.button(title, lambda i=index: self.navigate(i))
+ self.field("title", value=self.state["saved"][0]["title"])
+ self.field("status", [C.IR_INITIAL_STATUS, *C.IR_STATUS_WORDS])
+ self.field("body", value=self.state["saved"][0]["body"])
+ self.button("Done", lambda: self.navigate(None))
+
+ def navigate(self, index: int | None) -> None:
+ if self.state["dirty"]:
+ self.emit("save_prompt_shown")
+ self.dialog(
+ "Save changes before leaving?",
+ [("Save", "save"), ("Don’t Save", "dont_save"), ("Cancel", "cancel")],
+ lambda answer: self.save_answer(answer, index),
+ )
+ else:
+ self.visit(index)
+
+ def save_answer(self, answer: str, index: int | None) -> None:
+ self.state["answers"].append(answer)
+ self.emit("save_answer", answer=answer)
+ if answer == "cancel":
+ return
+ if answer == "save":
+ self.state["saved"][self.selected] = self.values()
+ self.emit("save", index=self.selected, values=self.values())
+ else:
+ self.emit("discard", index=self.selected)
+ self.state["dirty"] = False
+ self.visit(index)
+
+ def visit(self, index: int | None) -> None:
+ if index is None:
+ self.state["done_count"] += 1
+ self.emit("done", clean=not self.state["dirty"])
+ else:
+ self.selected = index
+ self.loading = True
+ for key, value in self.state["saved"][self.selected].items():
+ self.set_loaded(key, value)
+ self.loading = False
+ self.last_values = self.values()
+ if index is not None:
+ self.emit("note_selected", index=index)
+
+ # ------------------------------------------------------------------ IR-04 irconsent
+
+ def irconsent(self) -> None:
+ self.state.update(
+ consent_shown=False, consent_answers=[], submit_count=0, submits_blocked=0
+ )
+ self.field("email")
+ self.field("plan", C.IR_PLANS)
+ digest = wx.CheckBox(self.panel, label="Email digest", name="Email digest")
+ self.fields["digest"] = digest
+ digest.Bind(wx.EVT_CHECKBOX, lambda _: self.edit("digest"))
+ self.box.Add(digest, 0, wx.BOTTOM, 8)
+ self.button("Sign up", self.signup)
+
+ def show_consent(self) -> None:
+ self.emit("consent_shown")
+ self.dialog(
+ "Cookie preferences",
+ [("Reject optional", "reject"), ("Accept all", "accept")],
+ self.consent_answer,
+ )
+
+ def consent_answer(self, answer: str) -> None:
+ details = {"answer": answer, "analytics": answer == "accept", "ads": answer == "accept"}
+ self.state["consent_answers"].append(details)
+ self.emit("consent_answer", **details)
+
+ def signup(self) -> None:
+ if not self.state["consent_answers"]:
+ self.state["submits_blocked"] += 1
+ self.emit("submit_blocked")
+ return
+ self.state["submit_count"] += 1
+ self.emit("submit", **self.values())
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--mode", choices=MODES, required=True)
+ parser.add_argument("--seed", type=int, required=True)
+ parser.add_argument("--state", type=Path, required=True)
+ parser.add_argument("--events", type=Path, required=True)
+ parser.add_argument("--frame", help="x,y,width,height of the window")
+ parser.add_argument(
+ "--no-activate", action="store_true", help="show the window without activating it"
+ )
+ args = parser.parse_args()
+ ctypes.windll.shcore.SetProcessDpiAwareness(2)
+ app = wx.App(False)
+ app.SetAppName("BenchLab")
+ lab = Lab(args)
+ try:
+ app.MainLoop()
+ finally:
+ lab.events.close()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/windows/guard.py b/libs/cua-driver-bench/automated-eval/macos_bench/windows/guard.py
new file mode 100644
index 0000000000..2a6a1b9fbb
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/windows/guard.py
@@ -0,0 +1,131 @@
+# SPDX-License-Identifier: FSL-1.1-MIT
+# Copyright (c) 2026 Cua AI, Inc.
+
+"""The GUI-only guard on Windows (the counterpart of the macOS and Linux rules in bench_v2.py).
+
+A trial fails, and is stopped, when the agent reaches a shell or script host through the GUI:
+* a side-door process (PowerShell, cmd, Windows Terminal, the script hosts, mshta, WSL, ...) starts during
+ the trial outside the runner's own process tree (the runner, the agent and its tool servers are never
+ counted, and processes that were already running when the trial started are ignored), or
+* a side-door window is the foreground window for FRONT_GRACE_SAMPLES samples in a row, or
+* a computer-use tool input carries code that runs outside the GUI (bench_v2.EXEC_PATTERNS, plus the Windows
+ patterns below).
+
+The pure functions here take process rows and window owners as data, so they are tested on every OS.
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+from typing import Any, Iterable
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+import bench_v2 as v2 # noqa: E402
+
+# Executable names (lower case, without .exe) that are a shell, a console or a script host.
+SIDE_DOOR_PROCESSES = {
+ "powershell": "Windows PowerShell",
+ "powershell_ise": "PowerShell ISE",
+ "pwsh": "PowerShell",
+ "cmd": "Command Prompt",
+ "windowsterminal": "Windows Terminal",
+ "openconsole": "Windows Terminal",
+ "wt": "Windows Terminal",
+ "wscript": "Windows Script Host",
+ "cscript": "Windows Script Host",
+ "mshta": "HTML Application host",
+ "wsl": "WSL",
+ "wslhost": "WSL",
+ "bash": "bash",
+}
+# Text in a tool input that runs code outside the GUI on Windows.
+WINDOWS_EXEC_PATTERNS = (
+ "powershell",
+ "pwsh",
+ "cmd.exe",
+ "cmd /c",
+ "cmd /k",
+ "wscript",
+ "cscript",
+ "mshta",
+ "Start-Process",
+ "Invoke-Expression",
+ "Invoke-Item",
+ "rundll32",
+ "wsl.exe",
+)
+FRONT_GRACE_SAMPLES = v2.FRONT_GRACE_SAMPLES
+
+
+def name_of(exe: str) -> str:
+ base = exe.replace("\\", "/").rsplit("/", 1)[-1].lower()
+ return base[:-4] if base.endswith(".exe") else base
+
+
+def descendants(rows: Iterable[tuple[int, int, str]], root: int) -> set[int]:
+ """`root` and every process below it."""
+ children: dict[int, list[int]] = {}
+ for pid, ppid, _ in rows:
+ if pid != ppid:
+ children.setdefault(ppid, []).append(pid)
+ out, todo = set(), [root]
+ while todo:
+ pid = todo.pop()
+ if pid in out:
+ continue
+ out.add(pid)
+ todo.extend(children.get(pid, []))
+ return out
+
+
+def new_side_doors(
+ rows: list[tuple[int, int, str]], baseline: set[int], runner: int, own: set[int] | None = None
+) -> list[tuple[int, str]]:
+ """Side-door processes that started after the baseline and are not under the runner. `own` keeps every
+ pid ever seen under the runner, so a process whose parent has exited still counts as the runner's."""
+ own = descendants(rows, runner) | (own if own is not None else set())
+ return [
+ (pid, SIDE_DOOR_PROCESSES[name_of(exe)])
+ for pid, _, exe in rows
+ if pid not in baseline and pid not in own and name_of(exe) in SIDE_DOOR_PROCESSES
+ ]
+
+
+def scan_tool_use(name: str, tool_input: Any) -> tuple[list[str], list[str]]:
+ hard, soft = v2.scan_tool_use(name, tool_input)
+ if name.startswith("mcp__"):
+ text = "\n".join(v2._strings(tool_input)).lower()
+ hard += [f"{name}:{p}" for p in WINDOWS_EXEC_PATTERNS if p.lower() in text]
+ return sorted(set(hard)), soft
+
+
+def scan_events(events: list[dict[str, Any]]) -> tuple[list[str], list[str]]:
+ hard: set[str] = set()
+ soft: set[str] = set()
+ for event in events:
+ for name, tool_input in v2.tool_uses(event):
+ h, s = scan_tool_use(name, tool_input)
+ hard.update(h)
+ soft.update(s)
+ return sorted(hard)[:20], sorted(soft)[:20]
+
+
+class FrontWatch:
+ """A side-door window in front for FRONT_GRACE_SAMPLES samples in a row is a violation."""
+
+ def __init__(self) -> None:
+ self.run = 0
+ self.seen: set[str] = set()
+
+ def sample(self, front_exe: str) -> str | None:
+ name = name_of(front_exe)
+ if name:
+ self.seen.add(name)
+ if name in SIDE_DOOR_PROCESSES:
+ self.run += 1
+ if self.run >= FRONT_GRACE_SAMPLES:
+ return SIDE_DOOR_PROCESSES[name]
+ else:
+ self.run = 0
+ return None
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/windows/oracle.py b/libs/cua-driver-bench/automated-eval/macos_bench/windows/oracle.py
new file mode 100644
index 0000000000..880571c49b
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/windows/oracle.py
@@ -0,0 +1,206 @@
+# SPDX-License-Identifier: FSL-1.1-MIT
+# Copyright (c) 2026 Cua AI, Inc.
+
+"""Scripted oracle for the Windows BenchLab modes: solves a task through the GUI with real input.
+
+Used only to validate the port (each task solvable, its checker passes); never a bench setup. It finds
+controls through UI Automation (pywinauto) and acts with real pointer and keyboard input in the foreground.
+
+Usage: oracle.py --mode MODE --seed N [--disturb]
+`--disturb` instead performs one deliberate disturbance of the user window (validates the sentinel).
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+import time
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "probes" / "_common"))
+import benchlab_common as C # noqa: E402
+
+from pywinauto import Desktop, keyboard, mouse # noqa: E402
+
+
+def lab(mode: str):
+ win = Desktop(backend="uia").window(title=f"BenchLab — {mode}")
+ win.wait("exists visible", timeout=20)
+ win.set_focus()
+ time.sleep(0.4)
+ return win
+
+
+def center(ctrl) -> tuple[int, int]:
+ r = ctrl.rectangle()
+ return (r.left + r.right) // 2, (r.top + r.bottom) // 2
+
+
+def click(ctrl) -> None:
+ ctrl.click_input()
+ time.sleep(0.3)
+
+
+def type_into(win, name: str, value: str) -> None:
+ edit = win.child_window(title=name, control_type="Edit")
+ edit.click_input()
+ keyboard.send_keys("^a{BACKSPACE}", pause=0.02)
+ keyboard.send_keys(str(value), with_spaces=True, pause=0.02)
+ time.sleep(0.2)
+
+
+def choose(win, name: str, value: str) -> None:
+ combo = win.child_window(title=name, control_type="ComboBox")
+ combo.click_input()
+ time.sleep(0.4)
+ combo.child_window(title=value, control_type="ListItem").click_input()
+ time.sleep(0.3)
+
+
+def button(win, title: str):
+ return win.child_window(title=title, control_type="Button")
+
+
+def dialog_button(title: str) -> None:
+ for _ in range(40):
+ for w in Desktop(backend="uia").windows():
+ try:
+ found = w.descendants(title=title, control_type="Button")
+ except Exception: # noqa: BLE001
+ continue
+ if found:
+ found[0].click_input()
+ time.sleep(0.3)
+ return
+ time.sleep(0.25)
+ raise RuntimeError(f"no dialog button {title!r}")
+
+
+def drag(src: tuple[int, int], dst: tuple[int, int]) -> None:
+ mouse.press(coords=src)
+ for i in range(1, 11):
+ mouse.move(
+ coords=(src[0] + (dst[0] - src[0]) * i // 10, src[1] + (dst[1] - src[1]) * i // 10)
+ )
+ time.sleep(0.04)
+ mouse.release(coords=dst)
+ time.sleep(0.5)
+
+
+def listdrag(seed: int) -> None:
+ p = C.derive_listdrag(seed)
+ win = lab("listdrag")
+ order = list(p["initial"])
+ for index, item in enumerate(p["target"]):
+ source = order.index(item)
+ if source == index:
+ continue
+ rows = win.child_window(control_type="List").children(control_type="ListItem")
+ drag(center(rows[source]), center(rows[index]))
+ order.pop(source)
+ order.insert(index, item)
+ click(button(win, "Done"))
+
+
+def hover(seed: int) -> None:
+ p = C.derive_hover(seed)
+ win = lab("hover")
+ zone = win.child_window(title="Hover here to reveal actions", control_type="Text")
+ x, y = center(zone)
+ mouse.move(coords=(x, y))
+ time.sleep(0.6)
+ mouse.move(coords=(x + 2, y + 30))
+ time.sleep(0.6)
+ click(button(win, p["target"]))
+
+
+def tooltip(seed: int) -> None:
+ p = C.derive_tooltip(seed)
+ win = lab("tooltip")
+ icon = win.child_window(title=p["target"], control_type="Text")
+ x, y = center(icon)
+ mouse.move(coords=(x - 3, y))
+ time.sleep(0.2)
+ mouse.move(coords=(x, y))
+ time.sleep(2.0)
+ type_into(win, "Code", p["target_code"])
+ click(button(win, "Submit"))
+
+
+def irmodal(seed: int) -> None:
+ p = C.derive_irmodal(seed)
+ win = lab("irmodal")
+ type_into(win, "Name", p["name"])
+ type_into(win, "Email", p["email"])
+ click(button(win, "Next"))
+ dialog_button("Don’t Allow")
+ choose(win, "Team", p["team"])
+ type_into(win, "Seats", str(p["seats"]))
+ click(button(win, "Submit"))
+
+
+def irbanner(seed: int) -> None:
+ p = C.derive_irbanner(seed)
+ win = lab("irbanner")
+ type_into(win, "Quantity", str(p["quantity"]))
+ time.sleep(0.6)
+ click(button(win, "Dismiss"))
+ choose(win, "Priority", p["priority"])
+ click(button(win, "Apply"))
+
+
+def irunsaved(seed: int) -> None:
+ p = C.derive_irunsaved(seed)
+ win = lab("irunsaved")
+ click(button(win, p["titles"][p["status_index"]]))
+ choose(win, "Status", p["status"])
+ click(button(win, p["titles"][p["rename_index"]]))
+ dialog_button("Save")
+ type_into(win, "Title", p["new_title"])
+ click(button(win, "Done"))
+ dialog_button("Save")
+
+
+def irconsent(seed: int) -> None:
+ p = C.derive_irconsent(seed)
+ win = lab("irconsent")
+ type_into(win, "Email", p["email"])
+ dialog_button("Reject optional")
+ type_into(win, "Email", p["email"]) # the consent prompt takes the keys typed after it opened
+ choose(win, "Plan", p["plan"])
+ box = win.child_window(title="Email digest", control_type="CheckBox")
+ if bool(box.get_toggle_state()) != p["digest"]:
+ click(box)
+ click(button(win, "Sign up"))
+
+
+def disturb() -> None:
+ """One deliberate disturbance of each kind the sentinel must see."""
+ import ctypes
+
+ time.sleep(2)
+ win = Desktop(backend="uia").window(title_re="BenchLab — .*")
+ win.set_focus() # focus steal, activation change, raise over the user window
+ time.sleep(1)
+ mouse.move(coords=(300, 300)) # real pointer movement
+ ctypes.windll.user32.OpenClipboard(0)
+ ctypes.windll.user32.EmptyClipboard() # clipboard change
+ ctypes.windll.user32.CloseClipboard()
+ time.sleep(2)
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--mode", required=True)
+ parser.add_argument("--seed", type=int, required=True)
+ parser.add_argument("--disturb", action="store_true")
+ args = parser.parse_args()
+ if args.disturb:
+ disturb()
+ else:
+ globals()[args.mode](args.seed)
+ time.sleep(1.0)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/windows/sentinel.py b/libs/cua-driver-bench/automated-eval/macos_bench/windows/sentinel.py
new file mode 100644
index 0000000000..f5e2c7a8ec
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/windows/sentinel.py
@@ -0,0 +1,192 @@
+# SPDX-License-Identifier: FSL-1.1-MIT
+# Copyright (c) 2026 Cua AI, Inc.
+
+"""BenchSentinel for Windows: the witness / "User activity" window and its disturbance log.
+
+A port of linux/sentinel.py with the same JSONL log, read by swift/summarize_sentinel.py unchanged.
+Windows has no SIGUSR1/SIGUSR2, so the runner controls it through a control file (--control): each line
+is a command, `activate`, `arm`, `disarm` or `quit`, applied once in order.
+
+Measured while armed, every 200 ms:
+ front GetForegroundWindow -> process stem (the sentinel itself is ai.cua.benchsentinel)
+ mouse GetCursorPos (the real pointer)
+ clip GetClipboardSequenceNumber
+ onspace IVirtualDesktopManager::IsWindowOnCurrentVirtualDesktop on the sentinel window
+ sheet a shown window owned by the sentinel window (a dialog on the user's app)
+ above shown top-level windows above the sentinel in the z-order that overlap it
+Window events: WM_ACTIVATE (didBecomeActive / didResignKey), key, click and wheel input landing in the
+user window (keyDown, mouseDown, scrollWheel), and the simulated user's typing (user_type, user_blocked,
+user_text_disrupted, user_text).
+"""
+
+from __future__ import annotations
+
+import argparse
+import ctypes
+import json
+import os
+import sys
+import time
+from pathlib import Path
+
+import wx
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import winenv as W # noqa: E402
+
+BID = "ai.cua.benchsentinel"
+
+
+class Sentinel:
+ def __init__(self, args: argparse.Namespace) -> None:
+ self.args = args
+ args.log.parent.mkdir(parents=True, exist_ok=True)
+ self.log = args.log.open("a", buffering=1, encoding="utf-8")
+ self.armed = False
+ self.expected = ""
+ self.previous = None
+ self.done_commands = 0
+ self.desktops = W.VirtualDesktops()
+ title = "User activity" if args.user_activity else "Bench Sentinel"
+ self.frame = wx.Frame(None, title=title, size=(600, 320))
+ self.frame.SetName("BenchSentinel")
+ if args.frame:
+ x, y, w, h = map(int, args.frame.split(","))
+ self.frame.SetPosition((x, y))
+ self.frame.SetSize((w, h))
+ panel = wx.Panel(self.frame)
+ self.entry = wx.TextCtrl(panel, name="User activity text")
+ sizer = wx.BoxSizer(wx.VERTICAL)
+ sizer.Add(self.entry, 0, wx.EXPAND | wx.ALL, 16)
+ panel.SetSizer(sizer)
+ self.entry.Bind(wx.EVT_TEXT, self.changed)
+ self.entry.Bind(wx.EVT_KEY_DOWN, lambda e: (self.emit("keyDown"), e.Skip()))
+ self.entry.Bind(wx.EVT_LEFT_DOWN, lambda e: (self.emit("mouseDown"), e.Skip()))
+ self.entry.Bind(wx.EVT_MOUSEWHEEL, lambda e: (self.emit("scrollWheel"), e.Skip()))
+ panel.Bind(wx.EVT_LEFT_DOWN, lambda e: (self.emit("mouseDown"), e.Skip()))
+ self.frame.Bind(wx.EVT_ACTIVATE, self.activated)
+ if args.no_activate:
+ self.frame.ShowWithoutActivating()
+ else:
+ self.frame.Show()
+ self.activate()
+ self.own = self.frame.GetHandle()
+ self.timers = []
+ self.every(100, self.control)
+ self.every(200, self.sample)
+ if args.user_activity:
+ self.every(500, self.activity)
+ self.emit("ready", hwnd=self.own)
+
+ def every(self, ms: int, fn) -> None:
+ timer = wx.Timer(self.frame)
+ self.frame.Bind(wx.EVT_TIMER, lambda _: fn(), timer)
+ timer.Start(ms)
+ self.timers.append(timer)
+
+ def activated(self, event: wx.ActivateEvent) -> None:
+ event.Skip()
+ if event.GetActive():
+ self.emit("didBecomeActive")
+ else:
+ self.emit("didResignKey")
+
+ def changed(self, _event: object) -> None:
+ if self.entry.GetValue() != self.expected:
+ self.emit("user_text_disrupted")
+
+ def snapshot(self) -> dict:
+ front = W.identity(W.user32.GetForegroundWindow(), self.own, BID)
+ above = [W.identity(w, self.own, BID)["bid"] for w in W.windows_above(self.own)]
+ on = self.desktops.on_current(self.own)
+ sample = {
+ "front": front,
+ "mouse": W.cursor(),
+ "clip": W.clipboard_seq(),
+ "sheet": W.owned_dialog_shown(self.own),
+ "above": above,
+ "above_n": len(above),
+ }
+ if on is not None: # not measured is left out, which the summary reports as None
+ sample["onspace"] = on
+ return sample
+
+ def emit(self, event: str, **details: object) -> None:
+ if self.armed or event in ("armed", "disarmed", "ready"):
+ self.log.write(json.dumps({"t": time.time() * 1000, "ev": event, **details}) + "\n")
+
+ def activate(self) -> None:
+ W.force_foreground(self.frame.GetHandle())
+ self.entry.SetFocus()
+
+ def control(self) -> None:
+ path = self.args.control
+ if not path or not path.exists():
+ return
+ try:
+ lines = path.read_text(encoding="utf-8").splitlines()
+ except OSError:
+ return
+ for command in lines[self.done_commands :]:
+ self.done_commands += 1
+ command = command.strip()
+ if command == "activate":
+ self.activate()
+ elif command == "arm" and not self.armed:
+ self.armed = True
+ self.emit("armed", **self.snapshot())
+ elif command == "disarm" and self.armed:
+ if self.args.user_activity:
+ self.emit("user_text", intact=self.entry.GetValue() == self.expected)
+ self.emit("disarmed", **self.snapshot())
+ self.armed = False
+ elif command == "quit":
+ self.frame.Close()
+ return
+
+ def sample(self) -> None:
+ if not self.armed:
+ return
+ sample = self.snapshot()
+ if sample["front"] != self.previous:
+ self.emit("front", **sample)
+ if self.previous and self.previous.get("bid") == BID:
+ self.emit("didResignActive")
+ self.previous = sample["front"]
+ self.emit("s", **sample)
+
+ def activity(self) -> None:
+ if not self.armed:
+ return
+ if W.identity(W.user32.GetForegroundWindow(), self.own, BID)["bid"] == BID:
+ if self.entry.GetValue() != self.expected:
+ self.emit("user_text_disrupted")
+ self.expected += "a"
+ self.entry.ChangeValue(self.expected)
+ self.entry.SetInsertionPointEnd()
+ self.emit("user_type")
+ else:
+ self.emit("user_blocked")
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--log", type=Path, required=True)
+ parser.add_argument("--control", type=Path, help="command file: activate, arm, disarm, quit")
+ parser.add_argument("--no-activate", action="store_true")
+ parser.add_argument("--user-activity", action="store_true")
+ parser.add_argument("--frame", help="x,y,width,height")
+ args = parser.parse_args()
+ ctypes.windll.shcore.SetProcessDpiAwareness(2)
+ app = wx.App(False)
+ app.SetAppName("BenchSentinel")
+ witness = Sentinel(args)
+ try:
+ app.MainLoop()
+ finally:
+ witness.log.close()
+ os._exit(0)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/windows/winenv.py b/libs/cua-driver-bench/automated-eval/macos_bench/windows/winenv.py
new file mode 100644
index 0000000000..1ce7a105ff
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/windows/winenv.py
@@ -0,0 +1,293 @@
+# SPDX-License-Identifier: FSL-1.1-MIT
+# Copyright (c) 2026 Cua AI, Inc.
+
+"""Win32 reads shared by the Windows sentinel, the GUI-only guard and the session launcher (ctypes only).
+
+Window identity is the executable's stem (`BenchLab`, `msedge`, `WindowsTerminal`), the Windows counterpart
+of the bundle id (macOS) and WM_CLASS (Linux). Python processes are named after their window title when it
+starts with BenchLab, so a BenchLab started from python.exe still reads as BenchLab.
+"""
+
+from __future__ import annotations
+
+import ctypes
+import os
+from ctypes import wintypes
+
+user32 = ctypes.WinDLL("user32", use_last_error=True)
+kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
+dwmapi = ctypes.WinDLL("dwmapi")
+ole32 = ctypes.WinDLL("ole32")
+
+HWND = wintypes.HWND
+user32.GetForegroundWindow.restype = HWND
+user32.GetWindow.restype = HWND
+user32.GetWindow.argtypes = [HWND, wintypes.UINT]
+user32.GetWindowThreadProcessId.argtypes = [HWND, ctypes.POINTER(wintypes.DWORD)]
+user32.GetWindowThreadProcessId.restype = wintypes.DWORD
+user32.IsWindowVisible.argtypes = [HWND]
+user32.IsIconic.argtypes = [HWND]
+user32.GetWindowRect.argtypes = [HWND, ctypes.POINTER(wintypes.RECT)]
+user32.GetWindowTextW.argtypes = [HWND, wintypes.LPWSTR, ctypes.c_int]
+user32.GetClassNameW.argtypes = [HWND, wintypes.LPWSTR, ctypes.c_int]
+user32.SetForegroundWindow.argtypes = [HWND]
+user32.BringWindowToTop.argtypes = [HWND]
+user32.ShowWindow.argtypes = [HWND, ctypes.c_int]
+user32.AttachThreadInput.argtypes = [wintypes.DWORD, wintypes.DWORD, wintypes.BOOL]
+user32.GetClipboardSequenceNumber.restype = wintypes.DWORD
+user32.SetWindowPos.argtypes = [
+ HWND, HWND, ctypes.c_int, ctypes.c_int, ctypes.c_int, ctypes.c_int, wintypes.UINT
+] # fmt: skip
+WNDENUMPROC = ctypes.WINFUNCTYPE(wintypes.BOOL, HWND, wintypes.LPARAM)
+user32.EnumWindows.argtypes = [WNDENUMPROC, wintypes.LPARAM]
+kernel32.OpenProcess.restype = wintypes.HANDLE
+kernel32.OpenProcess.argtypes = [wintypes.DWORD, wintypes.BOOL, wintypes.DWORD]
+kernel32.QueryFullProcessImageNameW.argtypes = [
+ wintypes.HANDLE, wintypes.DWORD, wintypes.LPWSTR, ctypes.POINTER(wintypes.DWORD)
+] # fmt: skip
+kernel32.CloseHandle.argtypes = [wintypes.HANDLE]
+
+GW_OWNER, GW_HWNDPREV = 4, 3
+DWMWA_CLOAKED = 14
+PROCESS_QUERY_LIMITED_INFORMATION = 0x1000
+SW_RESTORE = 9
+SWP_NOSIZE, SWP_NOMOVE, SWP_NOACTIVATE, SWP_NOZORDER = 0x1, 0x2, 0x10, 0x4
+
+
+def text(hwnd: int) -> str:
+ buf = ctypes.create_unicode_buffer(512)
+ user32.GetWindowTextW(hwnd, buf, 512)
+ return buf.value
+
+
+def class_name(hwnd: int) -> str:
+ buf = ctypes.create_unicode_buffer(256)
+ user32.GetClassNameW(hwnd, buf, 256)
+ return buf.value
+
+
+def window_pid(hwnd: int) -> int:
+ pid = wintypes.DWORD()
+ user32.GetWindowThreadProcessId(hwnd, ctypes.byref(pid))
+ return pid.value
+
+
+def window_thread(hwnd: int) -> int:
+ return user32.GetWindowThreadProcessId(hwnd, None)
+
+
+def process_path(pid: int) -> str:
+ handle = kernel32.OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, False, pid)
+ if not handle:
+ return ""
+ try:
+ size = wintypes.DWORD(1024)
+ buf = ctypes.create_unicode_buffer(1024)
+ if kernel32.QueryFullProcessImageNameW(handle, 0, buf, ctypes.byref(size)):
+ return buf.value
+ return ""
+ finally:
+ kernel32.CloseHandle(handle)
+
+
+def stem(path: str) -> str:
+ return os.path.splitext(os.path.basename(path))[0]
+
+
+def identity(hwnd: int, own: int = 0, own_bid: str = "ai.cua.benchsentinel") -> dict:
+ if not hwnd:
+ return {"bid": "win32:0", "pid": -1}
+ if own and hwnd == own:
+ return {"bid": own_bid, "pid": os.getpid()}
+ pid = window_pid(hwnd)
+ name = stem(process_path(pid)) if pid else ""
+ if name.lower() in ("python", "pythonw") and text(hwnd).startswith("BenchLab"):
+ name = "BenchLab"
+ return {"bid": name or f"win32:{hwnd}", "pid": pid or -1}
+
+
+def rect(hwnd: int) -> tuple[int, int, int, int]:
+ r = wintypes.RECT()
+ user32.GetWindowRect(hwnd, ctypes.byref(r))
+ return r.left, r.top, r.right, r.bottom
+
+
+def intersects(a: tuple[int, int, int, int], b: tuple[int, int, int, int]) -> bool:
+ return a[0] < b[2] and b[0] < a[2] and a[1] < b[3] and b[1] < a[3]
+
+
+def cloaked(hwnd: int) -> bool:
+ value = wintypes.DWORD()
+ dwmapi.DwmGetWindowAttribute(
+ HWND(hwnd), wintypes.DWORD(DWMWA_CLOAKED), ctypes.byref(value), ctypes.sizeof(value)
+ )
+ return bool(value.value)
+
+
+def shown(hwnd: int) -> bool:
+ if not user32.IsWindowVisible(hwnd) or user32.IsIconic(hwnd) or cloaked(hwnd):
+ return False
+ left, top, right, bottom = rect(hwnd)
+ return right > left and bottom > top
+
+
+def windows_above(own: int) -> list[int]:
+ """Shown top-level windows above `own` in the z-order that overlap it (a raise over the user window)."""
+ out = []
+ mine = rect(own)
+ hwnd = user32.GetWindow(own, GW_HWNDPREV)
+ seen = 0
+ while hwnd and seen < 2000:
+ seen += 1
+ if shown(hwnd) and intersects(rect(hwnd), mine):
+ out.append(hwnd)
+ hwnd = user32.GetWindow(hwnd, GW_HWNDPREV)
+ return out
+
+
+def owned_dialog_shown(own: int) -> bool:
+ """A shown top-level window owned by `own`: a dialog on the user's app."""
+ found = []
+
+ def each(hwnd, _):
+ if hwnd != own and user32.GetWindow(hwnd, GW_OWNER) == own and shown(hwnd):
+ found.append(hwnd)
+ return True
+
+ user32.EnumWindows(WNDENUMPROC(each), 0)
+ return bool(found)
+
+
+def top_level_windows() -> list[int]:
+ found: list[int] = []
+
+ def each(hwnd, _):
+ found.append(hwnd)
+ return True
+
+ user32.EnumWindows(WNDENUMPROC(each), 0)
+ return found
+
+
+def cursor() -> list[int]:
+ p = wintypes.POINT()
+ user32.GetCursorPos(ctypes.byref(p))
+ return [p.x, p.y]
+
+
+def clipboard_seq() -> int:
+ return int(user32.GetClipboardSequenceNumber())
+
+
+class _GUID(ctypes.Structure):
+ _fields_ = [
+ ("d1", wintypes.DWORD), ("d2", wintypes.WORD), ("d3", wintypes.WORD), ("d4", ctypes.c_ubyte * 8)
+ ] # fmt: skip
+
+ @classmethod
+ def parse(cls, s: str) -> _GUID:
+ g = cls()
+ ole32.CLSIDFromString(ctypes.c_wchar_p(s), ctypes.byref(g))
+ return g
+
+
+class VirtualDesktops:
+ """IVirtualDesktopManager (documented, Windows 10 and later) through a raw COM vtable."""
+
+ CLSID = "{aa509086-5ca9-4c25-8f95-589d3c07b48a}"
+ IID = "{a5cd92ff-29be-454c-8d04-d82879fb3f1b}"
+
+ def __init__(self) -> None:
+ ole32.CoInitializeEx(None, 2) # apartment threaded, like the GUI thread that calls it
+ self.ptr = ctypes.c_void_p()
+ hr = ole32.CoCreateInstance(
+ ctypes.byref(_GUID.parse(self.CLSID)), None, 1 | 4,
+ ctypes.byref(_GUID.parse(self.IID)), ctypes.byref(self.ptr),
+ ) # fmt: skip
+ if hr != 0 or not self.ptr.value:
+ self.ptr = None
+ return
+ vtable = ctypes.cast(self.ptr, ctypes.POINTER(ctypes.POINTER(ctypes.c_void_p))).contents
+ proto = ctypes.WINFUNCTYPE(
+ ctypes.c_long, ctypes.c_void_p, HWND, ctypes.POINTER(wintypes.BOOL)
+ )
+ self._on_current = proto(vtable[3])
+
+ def on_current(self, hwnd: int) -> bool | None:
+ if not self.ptr:
+ return None
+ out = wintypes.BOOL()
+ if self._on_current(self.ptr, hwnd, ctypes.byref(out)) != 0:
+ return None
+ return bool(out.value)
+
+
+def force_foreground(hwnd: int) -> bool:
+ """Bring `hwnd` to the front from a process that may not own the foreground (setup only, never armed)."""
+ if user32.IsIconic(hwnd):
+ user32.ShowWindow(hwnd, SW_RESTORE)
+ front = user32.GetForegroundWindow()
+ if front == hwnd:
+ return True
+ mine = kernel32.GetCurrentThreadId()
+ theirs = window_thread(front) if front else 0
+ attached = bool(theirs and theirs != mine and user32.AttachThreadInput(mine, theirs, True))
+ try:
+ user32.BringWindowToTop(hwnd)
+ user32.SetForegroundWindow(hwnd)
+ finally:
+ if attached:
+ user32.AttachThreadInput(mine, theirs, False)
+ if user32.GetForegroundWindow() != hwnd:
+ # The documented unlock: a keyboard event lets this process set the foreground.
+ user32.keybd_event(0x12, 0, 0, 0)
+ user32.keybd_event(0x12, 0, 2, 0)
+ user32.SetForegroundWindow(hwnd)
+ return user32.GetForegroundWindow() == hwnd
+
+
+def move_window(hwnd: int, x: int, y: int, w: int, h: int) -> None:
+ user32.SetWindowPos(hwnd, None, x, y, w, h, SWP_NOACTIVATE | SWP_NOZORDER)
+
+
+class _PROCESSENTRY32W(ctypes.Structure):
+ _fields_ = [
+ ("dwSize", wintypes.DWORD), ("cntUsage", wintypes.DWORD), ("th32ProcessID", wintypes.DWORD),
+ ("th32DefaultHeapID", ctypes.c_void_p), ("th32ModuleID", wintypes.DWORD),
+ ("cntThreads", wintypes.DWORD), ("th32ParentProcessID", wintypes.DWORD),
+ ("pcPriClassBase", ctypes.c_long), ("dwFlags", wintypes.DWORD), ("szExeFile", ctypes.c_wchar * 260),
+ ] # fmt: skip
+
+
+kernel32.CreateToolhelp32Snapshot.restype = wintypes.HANDLE
+kernel32.Process32FirstW.argtypes = [wintypes.HANDLE, ctypes.POINTER(_PROCESSENTRY32W)]
+kernel32.Process32NextW.argtypes = [wintypes.HANDLE, ctypes.POINTER(_PROCESSENTRY32W)]
+
+
+def processes() -> list[tuple[int, int, str]]:
+ """(pid, parent pid, executable name) of every process."""
+ snap = kernel32.CreateToolhelp32Snapshot(0x2, 0)
+ rows: list[tuple[int, int, str]] = []
+ if not snap or snap == wintypes.HANDLE(-1).value:
+ return rows
+ try:
+ entry = _PROCESSENTRY32W()
+ entry.dwSize = ctypes.sizeof(entry)
+ ok = kernel32.Process32FirstW(snap, ctypes.byref(entry))
+ while ok:
+ rows.append((entry.th32ProcessID, entry.th32ParentProcessID, entry.szExeFile))
+ ok = kernel32.Process32NextW(snap, ctypes.byref(entry))
+ finally:
+ kernel32.CloseHandle(snap)
+ return rows
+
+
+kernel32.ProcessIdToSessionId.argtypes = [wintypes.DWORD, ctypes.POINTER(wintypes.DWORD)]
+
+
+def session_of(pid: int) -> int | None:
+ """The Terminal Services session of a process (the interactive desktop is not session 0)."""
+ out = wintypes.DWORD()
+ if not kernel32.ProcessIdToSessionId(pid, ctypes.byref(out)):
+ return None
+ return int(out.value)
diff --git a/libs/cua-driver-bench/taskpack/v2-taskset.json b/libs/cua-driver-bench/taskpack/v2-taskset.json
index 26bde59a68..67724c88ad 100644
--- a/libs/cua-driver-bench/taskpack/v2-taskset.json
+++ b/libs/cua-driver-bench/taskpack/v2-taskset.json
@@ -1,7 +1,7 @@
{
"schema": "cdb-taskset/1",
"name": "v2",
- "description": "Bench v2: the ten core tasks, all public. `os` lists where the task has a launch descriptor or runs today: the three multi-app tasks (CDB-G02 to CDB-G04, pack tasks with Electron apps, LibreOffice and, for CDB-G03, GnuCash) run on macOS, Linux and Windows; the seven BenchLab tasks (MB-09 to MB-11, IR-01 to IR-04) run on macOS and Linux. `split` is public (in this repository) or heldout (kept out of this repository; such a task would be listed by id and category only).",
+ "description": "Bench v2: the ten core tasks, all public. `os` lists where the task has a launch descriptor or runs today: all ten run on macOS, Linux and Windows. The three multi-app tasks (CDB-G02 to CDB-G04) are pack tasks with Electron apps, LibreOffice and, for CDB-G03, GnuCash; the seven BenchLab tasks (MB-09 to MB-11, IR-01 to IR-04) run in BenchLab and its Linux and Windows ports. `split` is public (in this repository) or heldout (kept out of this repository; such a task would be listed by id and category only).",
"limits": {
"timeout_s": 360,
"max_turns": 45
@@ -109,7 +109,8 @@
],
"os": [
"macos",
- "linux"
+ "linux",
+ "windows"
],
"cross_os_candidate": null,
"conditions": [
@@ -136,7 +137,8 @@
],
"os": [
"macos",
- "linux"
+ "linux",
+ "windows"
],
"cross_os_candidate": null,
"conditions": [
@@ -162,7 +164,8 @@
],
"os": [
"macos",
- "linux"
+ "linux",
+ "windows"
],
"cross_os_candidate": null,
"conditions": [
@@ -188,7 +191,8 @@
],
"os": [
"macos",
- "linux"
+ "linux",
+ "windows"
],
"cross_os_candidate": null,
"conditions": [
@@ -215,7 +219,8 @@
],
"os": [
"macos",
- "linux"
+ "linux",
+ "windows"
],
"cross_os_candidate": null,
"conditions": [
@@ -242,7 +247,8 @@
],
"os": [
"macos",
- "linux"
+ "linux",
+ "windows"
],
"cross_os_candidate": null,
"conditions": [
@@ -269,7 +275,8 @@
],
"os": [
"macos",
- "linux"
+ "linux",
+ "windows"
],
"cross_os_candidate": null,
"conditions": [