Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cua-driver-bench): run the BenchLab tasks on Hyprland (Wayland), e.g. Omarchy (#6630)

Francesco Bonaccic643cb396865
diff --git a/libs/cua-driver-bench/README.md b/libs/cua-driver-bench/README.md
index b956816545..0ab2575e9f 100644
--- a/libs/cua-driver-bench/README.md
+++ b/libs/cua-driver-bench/README.md
@@ -43,8 +43,8 @@ Ten tasks in three categories. Each task has a `PROVENANCE.md` next to it.
The CDB-G tasks are the GUI-only variants of pack tasks `cdb-s02` to `cdb-s04`:
the same files, with no coding tools offered. The MB and IR tasks run in
BenchLab, a small AppKit test app built from source in this directory (with a
-Linux port in `automated-eval/macos_bench/linux` and a Windows port in
-`automated-eval/macos_bench/windows`). A fourth category, Web apps &
+Linux port in `automated-eval/macos_bench/linux`, which `automated-eval/macos_bench/hyprland`
+runs on Hyprland (Wayland), and a Windows port in `automated-eval/macos_bench/windows`). A fourth category, Web apps &
forms, has no Bench v2 task yet.
### Platforms and conditions
@@ -356,6 +356,49 @@ Codex computer use is not available on Linux yet
([compatibility and limitations](https://learn.chatgpt.com/docs/linux/linux-app#compatibility-and-limitations)).
`tests/test_linux_session.py` and `tests/test_linux_tools.py` check the session and the tools without a model.
+### Linux on Hyprland (Wayland; the BenchLab tasks)
+
+The Hyprland port runs the same GTK3 BenchLab (`linux/benchlab.py`) as a native Wayland client, with a
+Wayland sentinel and GUI-only guard in [`automated-eval/macos_bench/hyprland`](automated-eval/macos_bench/hyprland).
+It is built for Omarchy (Arch Linux with Hyprland 0.55 or later and a Lua config), for example Cua's
+`ghcr.io/trycua/omarchy:edge` image. Seeds, the state file, the event log, the checkers and the sentinel
+summary are the same as on macOS, Linux/X11 and Windows.
+
+You need Python 3.11 or later with PyGObject, GTK 3 and AT-SPI, and `wl-clipboard`. Load
+`hyprland/bench.lua` last from `~/.config/hypr/hyprland.lua` (it sets 1920x1080 so the shared window frames
+fit), then run each trial as the desktop user with the session's environment (`WAYLAND_DISPLAY`,
+`HYPRLAND_INSTANCE_SIGNATURE`, `XDG_RUNTIME_DIR`, `DBUS_SESSION_BUS_ADDRESS`). One trial:
+
+```sh
+python3 launch/hyprland_session.py --task IR-02 --run-index 0 --condition background --out ~/bench/IR-02-r0 -- <agent command>
+```
+
+Where Hyprland differs:
+
+* **Placement.** A Wayland client cannot place its own windows, so the launcher floats BenchLab and the
+ sentinel and gives them their frames through Hyprland's IPC socket. Window rules (applied on every trial)
+ keep BenchLab's windows from taking focus when they open and open its dialogs over BenchLab's top-left
+ corner, clear of the "User activity" window, as a sheet hangs from BenchLab on macOS.
+* **Sentinel.** The compositor state comes from Hyprland's IPC: the active window, the stacking order of
+ `clients` (a window counts as raised when it comes after the user window, draws over it and overlaps it;
+ top and overlay layer-shell surfaces that overlap it count too, named after their owning process), the
+ real pointer (`cursorpos`), the active and special workspaces and full-screen windows (Space), and a
+ second window of the sentinel's own process (dialog). The clipboard is watched with `wl-paste --watch`
+ (data-control), so a change counts whether or not the user window has focus. Key focus, typed keys,
+ clicks and scrolls are GTK events on the user window, as on X11.
+* **GUI-only guard.** A terminal emulator (Alacritty, Ghostty, kitty, foot, ...) started outside the
+ runner's own processes, or a terminal or Omarchy TUI window active for three samples, fails and stops the
+ trial. Shells alone are not counted (tool servers reached over SSH run under one). Tool inputs are also
+ scanned for `hyprctl`, compositor `exec` dispatches, uwsm and Omarchy launchers, and terminal names.
+* **Brief.** "on your Mac" reads "on this Linux computer".
+
+`hyprland/oracle.py` solves each task through the GUI with real input (a uinput pointer and keyboard; the
+desktop user needs access to `/dev/uinput` while it runs). It validates the port: the oracle passes every
+checker, doing nothing fails every task and leaves the user undisturbed, and a deliberate focus change,
+raise, pointer move, clipboard change, keystroke or workspace switch is each seen.
+`tests/test_hyprland_tools.py` checks the guard, the brief, the descriptors and the window rules without a
+desktop or a model.
+
### Windows
The Windows port of BenchLab and the sentinel is in
@@ -476,6 +519,7 @@ Where the multi-app tasks differ on Windows:
cd automated-eval/macos_bench
python3 -m pytest tests probes/tests swift/tests # harness, checkers, sentinel (no GUI, no model)
python -m pytest tests/test_windows_desktop.py # on a Windows desktop: BenchLab, sentinel, guard, oracle
+python3 -m pytest tests/test_hyprland_tools.py # Hyprland port contracts (no desktop needed)
cd ../../taskpack/tasks/shared/cdb-s02 && python3 -m unittest discover -s tests
cd ../../../../fairness && python3 -m unittest discover -s tests # fairness and health checks
python3 ../automated-eval/macos_bench/tools/taskset.py check # manifest, selectors, pins and docs agree
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/hyprland/bench.lua b/libs/cua-driver-bench/automated-eval/macos_bench/hyprland/bench.lua
new file mode 100644
index 0000000000..15b74afd54
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/hyprland/bench.lua
@@ -0,0 +1,22 @@
+-- Cua Driver Bench v2, Hyprland port: the desktop settings the BenchLab trials need.
+-- Load it last from ~/.config/hypr/hyprland.lua: dofile(os.getenv("HOME") .. "/.config/hypr/bench.lua")
+-- (Hyprland 0.55+ with a Lua config, as on Omarchy edge). launch/hyprland_session.py applies the same window
+-- rules on every trial, so only the screen size needs this file.
+
+-- 1920x1080 so the shared frames fit: BenchLab at 40,40 760x540, the User activity window at 420,340 900x520.
+hl.monitor({ output = "", mode = "1920x1080@60", position = "0x0", scale = 1 })
+
+-- BenchLab and the sentinel float at the frames the session gives them (Hyprland would tile them).
+-- BenchLab's windows never take focus when they open, and its dialogs open over BenchLab's top-left corner,
+-- clear of the User activity window: an app working in the background does not cover the user's window with
+-- its own dialogs (on macOS a sheet hangs from BenchLab's title bar; on Windows the dialog opens just above
+-- BenchLab).
+hl.window_rule({ name = "cdb-benchlab", match = { class = "^BenchLab$" }, float = true, no_initial_focus = true })
+hl.window_rule({
+ name = "cdb-benchlab-dialogs",
+ match = { class = "^BenchLab$", title = "^(Allow BenchLab.*|Save changes.*|Cookie preferences)$" },
+ float = true,
+ no_initial_focus = true,
+ move = "80 120",
+})
+hl.window_rule({ name = "cdb-sentinel", match = { class = "^BenchSentinel$" }, float = true })
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/hyprland/guard.py b/libs/cua-driver-bench/automated-eval/macos_bench/hyprland/guard.py
new file mode 100644
index 0000000000..d4d4ba1f64
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/hyprland/guard.py
@@ -0,0 +1,178 @@
+# SPDX-License-Identifier: FSL-1.1-MIT
+# Copyright (c) 2026 Cua AI, Inc.
+
+"""The GUI-only guard on Hyprland (the counterpart of windows/guard.py and the X11 rules in bench_v2.py).
+
+A trial fails, and is stopped, when the agent reaches a shell through the GUI:
+* a terminal emulator starts during the trial outside the runner's own process tree (processes already running
+ when the trial started are ignored), or
+* a terminal window is the active window for FRONT_GRACE_SAMPLES samples in a row, or
+* a computer-use tool input carries code that runs outside the GUI (bench_v2.EXEC_PATTERNS, plus the Hyprland
+ patterns below: dispatching `exec` to the compositor, Omarchy's and uwsm's launchers, terminal names).
+
+Shells alone are not counted as processes: the tool servers reach the desktop through ssh, which runs them under a
+shell. A shell the agent could type into needs a terminal window, which is caught.
+
+The pure functions take process rows and window classes as data, so they are tested on every OS.
+"""
+
+from __future__ import annotations
+
+import os
+import re
+import sys
+from pathlib import Path
+from typing import Any, Iterable
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+import bench_v2 as v2 # noqa: E402
+
+# Process names (comm, lower case) of terminal emulators.
+TERMINALS = {
+ "alacritty": "Alacritty",
+ "kitty": "kitty",
+ "foot": "foot",
+ "footclient": "foot",
+ "ghostty": "Ghostty",
+ "wezterm": "WezTerm",
+ "wezterm-gui": "WezTerm",
+ "xterm": "xterm",
+ "uxterm": "xterm",
+ "urxvt": "urxvt",
+ "st": "st",
+ "konsole": "Konsole",
+ "gnome-terminal": "GNOME Terminal",
+ "gnome-terminal-": "GNOME Terminal",
+ "kgx": "GNOME Console",
+ "ptyxis": "Ptyxis",
+ "ptyxis-agent": "Ptyxis",
+ "xfce4-terminal": "Xfce Terminal",
+ "terminator": "Terminator",
+ "tilix": "Tilix",
+ "rio": "Rio",
+ "warp": "Warp",
+}
+# Window classes (lower case) of terminal windows, Omarchy's floating and tiled TUI classes included.
+TERMINAL_CLASSES = {
+ "alacritty": "Alacritty",
+ "kitty": "kitty",
+ "foot": "foot",
+ "com.mitchellh.ghostty": "Ghostty",
+ "ghostty": "Ghostty",
+ "org.wezfurlong.wezterm": "WezTerm",
+ "xterm": "xterm",
+ "urxvt": "urxvt",
+ "st-256color": "st",
+ "org.kde.konsole": "Konsole",
+ "org.gnome.terminal": "GNOME Terminal",
+ "org.gnome.console": "GNOME Console",
+ "org.gnome.ptyxis": "Ptyxis",
+ "xfce4-terminal": "Xfce Terminal",
+ "tui.float": "Omarchy TUI",
+ "tui.tile": "Omarchy TUI",
+}
+# Text in a tool input that runs code outside the GUI on Hyprland/Omarchy.
+HYPRLAND_EXEC_PATTERNS = (
+ "hyprctl",
+ "dispatch exec",
+ "uwsm app",
+ "uwsm-app",
+ "omarchy-launch",
+ "xdg-terminal-exec",
+ "systemd-run",
+ "sh -c",
+)
+# Terminal names in a tool input (launching one opens a shell): whole words only.
+TERMINAL_NAME_RE = re.compile(
+ r"\b(alacritty|ghostty|kitty|foot|footclient|wezterm|xterm|konsole|gnome-terminal|kgx|ptyxis)\b"
+)
+FRONT_GRACE_SAMPLES = v2.FRONT_GRACE_SAMPLES
+
+
+def terminal_of(name: str) -> str | None:
+ return TERMINALS.get(name.lower())
+
+
+def process_rows() -> list[tuple[int, int, str]]:
+ """(pid, ppid, comm) for every process."""
+ rows = []
+ for entry in os.listdir("/proc"):
+ if not entry.isdigit():
+ continue
+ try:
+ stat = Path("/proc", entry, "stat").read_text()
+ except OSError:
+ continue
+ name = stat[stat.index("(") + 1 : stat.rindex(")")]
+ ppid = int(stat[stat.rindex(")") + 2 :].split()[1])
+ rows.append((int(entry), ppid, name))
+ return rows
+
+
+def descendants(rows: Iterable[tuple[int, int, str]], root: int) -> set[int]:
+ children: dict[int, list[int]] = {}
+ for pid, ppid, _ in rows:
+ if pid != ppid:
+ children.setdefault(ppid, []).append(pid)
+ out, todo = set(), [root]
+ while todo:
+ pid = todo.pop()
+ if pid in out:
+ continue
+ out.add(pid)
+ todo.extend(children.get(pid, []))
+ return out
+
+
+def new_side_doors(
+ rows: list[tuple[int, int, str]], baseline: set[int], runner: int, own: set[int] | None = None
+) -> list[tuple[int, str]]:
+ own = descendants(rows, runner) | (own if own is not None else set())
+ return [
+ (pid, terminal_of(name))
+ for pid, _, name in rows
+ if pid not in baseline and pid not in own and terminal_of(name)
+ ]
+
+
+def scan_tool_use(name: str, tool_input: Any) -> tuple[list[str], list[str]]:
+ hard, soft = v2.scan_tool_use(name, tool_input)
+ if name.startswith("mcp__"):
+ text = "\n".join(v2._strings(tool_input)).lower()
+ hard += [f"{name}:{p}" for p in HYPRLAND_EXEC_PATTERNS if p in text]
+ hard += [f"{name}:{m}" for m in TERMINAL_NAME_RE.findall(text)]
+ return sorted(set(hard)), soft
+
+
+def scan_events(events: list[dict[str, Any]]) -> tuple[list[str], list[str]]:
+ hard: set[str] = set()
+ soft: set[str] = set()
+ for event in events:
+ for name, tool_input in v2.tool_uses(event):
+ h, s = scan_tool_use(name, tool_input)
+ hard.update(h)
+ soft.update(s)
+ return sorted(hard)[:20], sorted(soft)[:20]
+
+
+class FrontWatch:
+ """A terminal window active for FRONT_GRACE_SAMPLES samples in a row is a violation."""
+
+ def __init__(self) -> None:
+ self.run = 0
+ self.seen: set[str] = set()
+
+ def sample(self, window_class: str) -> str | None:
+ name = (window_class or "").lower()
+ if name:
+ self.seen.add(name)
+ label = TERMINAL_CLASSES.get(name) or (
+ "Omarchy TUI" if name.startswith("org.omarchy.") else None
+ )
+ if label:
+ self.run += 1
+ if self.run >= FRONT_GRACE_SAMPLES:
+ return label
+ else:
+ self.run = 0
+ return None
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/hyprland/hypr.py b/libs/cua-driver-bench/automated-eval/macos_bench/hyprland/hypr.py
new file mode 100644
index 0000000000..83e6e6a781
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/hyprland/hypr.py
@@ -0,0 +1,110 @@
+# SPDX-License-Identifier: FSL-1.1-MIT
+# Copyright (c) 2026 Cua AI, Inc.
+
+"""A small Hyprland IPC client (the request socket `hyprctl` uses), for the Hyprland port of BenchLab.
+
+The session, the sentinel and the guard read the compositor through it: clients (in stacking order), the active
+window and workspace, the cursor position, and dispatchers for window placement.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import socket
+from pathlib import Path
+from typing import Any
+
+
+def socket_path() -> Path:
+ sig = os.environ.get("HYPRLAND_INSTANCE_SIGNATURE", "")
+ runtime = os.environ.get("XDG_RUNTIME_DIR") or f"/run/user/{os.getuid()}"
+ if not sig:
+ found = sorted(
+ Path(runtime, "hypr").glob("*/.socket.sock"), key=lambda p: p.stat().st_mtime
+ )
+ if not found:
+ raise RuntimeError("no Hyprland instance (HYPRLAND_INSTANCE_SIGNATURE is unset)")
+ return found[-1]
+ return Path(runtime, "hypr", sig, ".socket.sock")
+
+
+def request(command: str, timeout: float = 8.0) -> str:
+ with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as conn:
+ conn.settimeout(timeout)
+ conn.connect(str(socket_path()))
+ conn.sendall(command.encode())
+ chunks = []
+ while True:
+ chunk = conn.recv(65536)
+ if not chunk:
+ break
+ chunks.append(chunk)
+ return b"".join(chunks).decode(errors="replace")
+
+
+def query(what: str) -> Any:
+ return json.loads(request(f"j/{what}") or "null")
+
+
+def clients() -> list[dict[str, Any]]:
+ return [c for c in query("clients") or [] if c.get("mapped", True)]
+
+
+def active_window() -> dict[str, Any]:
+ return query("activewindow") or {}
+
+
+def active_workspace() -> dict[str, Any]:
+ return query("activeworkspace") or {}
+
+
+def cursor() -> tuple[float, float]:
+ pos = query("cursorpos") or {}
+ return float(pos.get("x", 0)), float(pos.get("y", 0))
+
+
+def dispatch(expression: str) -> str:
+ """Run one dispatcher. Hyprland 0.55+ with a Lua config takes Lua (`hl.dsp.window.close()`)."""
+ return request("dispatch " + expression)
+
+
+def window_ref(address: str) -> str:
+ return f'window = "address:{address}"'
+
+
+def by_pid(pid: int) -> list[dict[str, Any]]:
+ return [c for c in clients() if c.get("pid") == pid]
+
+
+def place(address: str, frame: tuple[int, int, int, int]) -> None:
+ """Float the window and give it an exact frame (x, y, w, h) in layout coordinates."""
+ x, y, w, h = frame
+ ref = window_ref(address)
+ dispatch(f'hl.dsp.window.float({{ action = "on", {ref} }})')
+ dispatch(f"hl.dsp.window.resize({{ x = {w}, y = {h}, {ref} }})")
+ dispatch(f"hl.dsp.window.move({{ x = {x}, y = {y}, {ref} }})")
+
+
+def focus(address: str) -> None:
+ dispatch(f"hl.dsp.focus({{ {window_ref(address)} }})")
+
+
+def raise_to_top(address: str) -> None:
+ dispatch(f'hl.dsp.window.alter_zorder({{ mode = "top", {window_ref(address)} }})')
+
+
+def overlaps(a: dict[str, Any], b: dict[str, Any]) -> bool:
+ (ax, ay), (aw, ah) = a.get("at", (0, 0)), a.get("size", (0, 0))
+ (bx, by), (bw, bh) = b.get("at", (0, 0)), b.get("size", (0, 0))
+ return ax < bx + bw and bx < ax + aw and ay < by + bh and by < ay + ah
+
+
+def identity(client: dict[str, Any]) -> str:
+ """The name a window is reported under: its class (the app id), else its title, else its address."""
+ return (
+ client.get("class")
+ or client.get("initialClass")
+ or client.get("title")
+ or client.get("address", "")
+ )
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/hyprland/oracle.py b/libs/cua-driver-bench/automated-eval/macos_bench/hyprland/oracle.py
new file mode 100644
index 0000000000..0b5c5dc3c4
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/hyprland/oracle.py
@@ -0,0 +1,373 @@
+# SPDX-License-Identifier: FSL-1.1-MIT
+# Copyright (c) 2026 Cua AI, Inc.
+
+"""Scripted oracle for the BenchLab modes on Hyprland: solves a task through the GUI with real input.
+
+Used only to validate the port (each task solvable, its checker passes); never a bench setup. It finds controls
+through AT-SPI (window-relative extents plus the window's frame from Hyprland) and acts in the foreground with a
+uinput device: an absolute pointer with buttons and a keyboard, the same kernel path as a USB mouse and keyboard.
+The desktop user needs read-write access to /dev/uinput while it runs.
+
+Usage: oracle.py --mode MODE --seed N [--disturb KIND]
+`--disturb KIND` instead performs one deliberate disturbance of the user window (validates the sentinel):
+focus, raise, pointer, clipboard, typing, space or all.
+"""
+
+from __future__ import annotations
+
+import argparse
+import ctypes
+import fcntl
+import os
+import struct
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+import gi
+
+gi.require_version("Atspi", "2.0")
+from gi.repository import Atspi # noqa: E402
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "probes" / "_common"))
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import benchlab_common as C # noqa: E402
+import hypr # noqa: E402
+
+EV_SYN, EV_KEY, EV_ABS = 0, 1, 3
+ABS_X, ABS_Y = 0, 1
+BTN_LEFT = 0x110
+UI_SET_EVBIT, UI_SET_KEYBIT, UI_SET_ABSBIT = 0x40045564, 0x40045565, 0x40045567
+UI_DEV_CREATE, UI_DEV_DESTROY = 0x5501, 0x5502
+KEYS = {
+ "ctrl": 29,
+ "a": 30,
+ "backspace": 14,
+ "down": 108,
+ "up": 103,
+ "enter": 28,
+ "tab": 15,
+ "space": 57,
+}
+
+
+class Input:
+ """A uinput tablet-style pointer (absolute, mapped to the whole layout) and keyboard."""
+
+ def __init__(self) -> None:
+ width, height = self.layout()
+ self.size = (width, height)
+ self.fd = os.open("/dev/uinput", os.O_WRONLY | os.O_NONBLOCK)
+ fcntl.ioctl(self.fd, UI_SET_EVBIT, EV_KEY)
+ fcntl.ioctl(self.fd, UI_SET_EVBIT, EV_ABS)
+ for code in (BTN_LEFT, BTN_LEFT + 1, *KEYS.values()):
+ fcntl.ioctl(self.fd, UI_SET_KEYBIT, code)
+ fcntl.ioctl(self.fd, UI_SET_ABSBIT, ABS_X)
+ fcntl.ioctl(self.fd, UI_SET_ABSBIT, ABS_Y)
+ absmax = [0] * 64
+ absmax[ABS_X], absmax[ABS_Y] = width - 1, height - 1
+ dev = struct.pack("80sHHHHi", b"cdb-oracle", 0x03, 0x1234, 0x5678, 1, 0)
+ dev += struct.pack("64i", *absmax) + struct.pack("64i", *([0] * 64)) * 3
+ os.write(self.fd, dev)
+ fcntl.ioctl(self.fd, UI_DEV_CREATE)
+ time.sleep(1.0) # the compositor picks the device up
+
+ @staticmethod
+ def layout() -> tuple[int, int]:
+ monitors = hypr.query("monitors") or []
+ return (
+ max(m["x"] + m["width"] for m in monitors),
+ max(m["y"] + m["height"] for m in monitors),
+ )
+
+ def emit(self, kind: int, code: int, value: int) -> None:
+ os.write(self.fd, struct.pack("llHHi", 0, 0, kind, code, value))
+
+ def sync(self) -> None:
+ self.emit(EV_SYN, 0, 0)
+
+ def move(self, x: float, y: float) -> None:
+ self.emit(EV_ABS, ABS_X, int(x))
+ self.emit(EV_ABS, ABS_Y, int(y))
+ self.sync()
+ time.sleep(0.03)
+
+ def button(self, down: bool) -> None:
+ self.emit(EV_KEY, BTN_LEFT, 1 if down else 0)
+ self.sync()
+ time.sleep(0.05)
+
+ def click(self, x: float, y: float) -> None:
+ self.move(x, y)
+ time.sleep(0.15)
+ self.button(True)
+ self.button(False)
+ time.sleep(0.3)
+
+ def key(self, name: str, *mods: str) -> None:
+ for m in mods:
+ self.emit(EV_KEY, KEYS[m], 1)
+ self.emit(EV_KEY, KEYS[name], 1)
+ self.sync()
+ self.emit(EV_KEY, KEYS[name], 0)
+ for m in mods:
+ self.emit(EV_KEY, KEYS[m], 0)
+ self.sync()
+ time.sleep(0.05)
+
+ def close(self) -> None:
+ fcntl.ioctl(self.fd, UI_DEV_DESTROY)
+ os.close(self.fd)
+
+
+IN: Input | None = None
+
+
+def type_text(text: str) -> None:
+ # wtype: Wayland virtual-keyboard input to the focused surface (any character, no keymap table here).
+ subprocess.run(["wtype", "-d", "20", text], check=True, timeout=60)
+ time.sleep(0.2)
+
+
+def app() -> Atspi.Accessible:
+ desktop = Atspi.get_desktop(0)
+ for _ in range(50):
+ for i in range(desktop.get_child_count()):
+ child = desktop.get_child_at_index(i)
+ if child is not None and child.get_name() in ("BenchLab", "benchlab.py"):
+ return child
+ time.sleep(0.2)
+ raise RuntimeError("BenchLab is not on the accessibility bus")
+
+
+def walk(node: Atspi.Accessible):
+ yield node
+ for i in range(node.get_child_count()):
+ child = node.get_child_at_index(i)
+ if child is not None:
+ yield from walk(child)
+
+
+def toplevel(title_prefix: str) -> tuple[Atspi.Accessible, dict]:
+ for _ in range(80):
+ root = app()
+ for i in range(root.get_child_count()):
+ win = root.get_child_at_index(i)
+ if win is not None and win.get_name().startswith(title_prefix):
+ frames = [c for c in hypr.clients() if c.get("title", "").startswith(title_prefix)]
+ if frames:
+ return win, frames[-1]
+ time.sleep(0.25)
+ raise RuntimeError(f"no window {title_prefix!r}")
+
+
+ROLE_ALIASES = {"push button": ("push button", "button"), "check box": ("check box", "checkbox")}
+
+
+def find(win: Atspi.Accessible, role: str, name: str, nth: int = 0) -> Atspi.Accessible:
+ roles = ROLE_ALIASES.get(role, (role,))
+ hits = [
+ n for n in walk(win)
+ if n.get_role_name() in roles and n.get_name() == name and n.get_state_set().contains(Atspi.StateType.SHOWING)
+ ] # fmt: skip
+ if len(hits) <= nth:
+ raise RuntimeError(f"no {role} {name!r}")
+ return hits[nth]
+
+
+def center(node: Atspi.Accessible, frame: dict) -> tuple[float, float]:
+ box = node.get_extents(Atspi.CoordType.WINDOW)
+ x0, y0 = frame["at"]
+ return x0 + box.x + box.width / 2, y0 + box.y + box.height / 2
+
+
+def lab(mode: str) -> tuple[Atspi.Accessible, dict]:
+ win, frame = toplevel(f"BenchLab — {mode}")
+ hypr.focus(frame["address"])
+ hypr.raise_to_top(frame["address"])
+ time.sleep(0.4)
+ return win, frame
+
+
+def click(node: Atspi.Accessible, frame: dict) -> None:
+ IN.click(*center(node, frame))
+
+
+def button(mode: str, title: str) -> None:
+ win, frame = lab(mode)
+ click(find(win, "push button", title), frame)
+
+
+def type_into(mode: str, name: str, value: str) -> None:
+ win, frame = lab(mode)
+ click(find(win, "text", name), frame)
+ IN.key("a", "ctrl")
+ IN.key("backspace")
+ type_text(str(value))
+
+
+def choose(
+ mode: str, name: str, choices: list[str], value: str, current: str | None = None
+) -> None:
+ """Open the combo box (its popup opens on the current item) and step to VALUE with the arrow keys."""
+ win, frame = lab(mode)
+ click(find(win, "combo box", name), frame)
+ time.sleep(0.5)
+ steps = choices.index(value) - choices.index(current or choices[0])
+ for _ in range(abs(steps)):
+ IN.key("down" if steps > 0 else "up")
+ time.sleep(0.1)
+ IN.key("enter")
+ time.sleep(0.4)
+
+
+def dialog_button(title_prefix: str, label: str) -> None:
+ win, frame = toplevel(title_prefix)
+ hypr.focus(frame["address"])
+ time.sleep(0.3)
+ click(find(win, "push button", label), frame)
+
+
+def drag(src: tuple[float, float], dst: tuple[float, float]) -> None:
+ IN.move(*src)
+ time.sleep(0.2)
+ IN.button(True)
+ for i in range(1, 21):
+ IN.move(src[0] + (dst[0] - src[0]) * i / 20, src[1] + (dst[1] - src[1]) * i / 20)
+ time.sleep(0.03)
+ time.sleep(0.3)
+ IN.button(False)
+ time.sleep(0.6)
+
+
+def listdrag(seed: int) -> None:
+ p = C.derive_listdrag(seed)
+ order = list(p["initial"])
+ for index, item in enumerate(p["target"]):
+ source = order.index(item)
+ if source == index:
+ continue
+ win, frame = lab("listdrag")
+ rows = [find(win, "label", name) for name in order]
+ drag(center(rows[source], frame), center(rows[index], frame))
+ order.pop(source)
+ order.insert(index, item)
+ button("listdrag", "Done")
+
+
+def hover(seed: int) -> None:
+ p = C.derive_hover(seed)
+ win, frame = lab("hover")
+ x, y = center(find(win, "label", "Hover here to reveal actions"), frame)
+ IN.move(x, y)
+ time.sleep(0.6)
+ IN.move(x + 2, y + 20)
+ time.sleep(0.6)
+ win, frame = lab("hover")
+ click(find(win, "push button", p["target"]), frame)
+
+
+def tooltip(seed: int) -> None:
+ p = C.derive_tooltip(seed)
+ win, frame = lab("tooltip")
+ x, y = center(find(win, "label", p["target"]), frame)
+ IN.move(x - 3, y)
+ time.sleep(0.2)
+ IN.move(x, y)
+ time.sleep(2.0)
+ type_into("tooltip", "Code", p["target_code"])
+ button("tooltip", "Submit")
+
+
+def irmodal(seed: int) -> None:
+ p = C.derive_irmodal(seed)
+ type_into("irmodal", "Name", p["name"])
+ type_into("irmodal", "Email", p["email"])
+ button("irmodal", "Next")
+ dialog_button("Allow BenchLab", "Don’t Allow")
+ choose("irmodal", "Team", C.IR_TEAMS, p["team"])
+ type_into("irmodal", "Seats", str(p["seats"]))
+ button("irmodal", "Submit")
+
+
+def irbanner(seed: int) -> None:
+ p = C.derive_irbanner(seed)
+ type_into("irbanner", "Quantity", str(p["quantity"]))
+ time.sleep(0.6)
+ button("irbanner", "Dismiss")
+ choose("irbanner", "Priority", C.PRIORITIES, p["priority"])
+ button("irbanner", "Apply")
+
+
+def irunsaved(seed: int) -> None:
+ p = C.derive_irunsaved(seed)
+ statuses = [C.IR_INITIAL_STATUS, *C.IR_STATUS_WORDS]
+ button("irunsaved", p["titles"][p["status_index"]])
+ choose("irunsaved", "Status", statuses, p["status"])
+ button("irunsaved", p["titles"][p["rename_index"]])
+ dialog_button("Save changes", "Save")
+ type_into("irunsaved", "Title", p["new_title"])
+ button("irunsaved", "Done")
+ dialog_button("Save changes", "Save")
+
+
+def irconsent(seed: int) -> None:
+ p = C.derive_irconsent(seed)
+ type_into("irconsent", "Email", p["email"])
+ dialog_button("Cookie preferences", "Reject optional")
+ type_into(
+ "irconsent", "Email", p["email"]
+ ) # keys typed while the modal prompt was open went nowhere
+ choose("irconsent", "Plan", C.IR_PLANS, p["plan"])
+ win, frame = lab("irconsent")
+ box = find(win, "check box", "Email digest")
+ if box.get_state_set().contains(Atspi.StateType.CHECKED) != p["digest"]:
+ click(box, frame)
+ button("irconsent", "Sign up")
+
+
+def disturb(kind: str) -> None:
+ """One deliberate disturbance of the user window, of the given kind."""
+ time.sleep(2)
+ lab_frame = next(c for c in hypr.clients() if c.get("class") == "BenchLab")
+ if kind in ("focus", "all"):
+ hypr.focus(lab_frame["address"]) # key focus and activation move to BenchLab
+ if kind in ("raise", "all"):
+ hypr.raise_to_top(lab_frame["address"]) # BenchLab drawn over the user window
+ if kind in ("pointer", "all"):
+ IN.move(300, 300)
+ IN.move(320, 310)
+ if kind in ("clipboard", "all"):
+ subprocess.run(["wl-copy", "cdb-disturbance"], check=True, timeout=10)
+ if kind == "typing":
+ type_text("x") # a keystroke lands in the user window
+ if kind == "space":
+ hypr.dispatch('hl.dsp.focus({ workspace = "2" })')
+ time.sleep(1.5)
+ hypr.dispatch('hl.dsp.focus({ workspace = "1" })')
+ time.sleep(2)
+
+
+def main() -> None:
+ global IN
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--mode", required=True)
+ parser.add_argument("--seed", type=int, required=True)
+ parser.add_argument("--disturb")
+ args = parser.parse_args()
+ ctypes.CDLL(None).prctl(15, b"cdb-oracle", 0, 0, 0)
+ Atspi.init()
+ IN = Input()
+ try:
+ if args.disturb:
+ disturb(args.disturb)
+ else:
+ globals()[args.mode](args.seed)
+ time.sleep(1.0)
+ finally:
+ IN.close()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/hyprland/sentinel.py b/libs/cua-driver-bench/automated-eval/macos_bench/hyprland/sentinel.py
new file mode 100644
index 0000000000..7f05d51912
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/hyprland/sentinel.py
@@ -0,0 +1,317 @@
+# SPDX-License-Identifier: FSL-1.1-MIT
+# Copyright (c) 2026 Cua AI, Inc.
+
+"""BenchSentinel for Hyprland (Wayland): the witness and simulated user of the Linux/Omarchy BenchLab port.
+
+A GTK3 window ("User activity" in the background condition) that writes the same JSONL schema as the macOS
+BenchSentinel, read by swift/summarize_sentinel.py. A Wayland client cannot see other windows, so the compositor
+state comes from Hyprland's IPC socket (hypr.py):
+
+* front: the active window (`activewindow`); the sentinel's own window is reported as ai.cua.benchsentinel.
+* above: windows after the sentinel's in Hyprland's stacking order (`clients`) on its workspace (or pinned) that
+ overlap it and draw over it (floating, pinned or full screen), plus top and overlay layer-shell surfaces that
+ overlap it. A layer surface owned by a cua-driver process is named after it, so the overlay rule applies.
+* mouse: the real pointer (`cursorpos`); the agent seats of the cua Hyprland plugin have their own pointers.
+* onspace: the sentinel's workspace is the active one, no special workspace is open over it, and no other window
+ is full screen on it.
+* sheet: another window of the sentinel's own process (a dialog on the user's app).
+* clip: clipboard changes, from `wl-paste --watch` (data-control; GTK's owner-change without wl-clipboard).
+* key focus, typed keys, clicks and scrolls: GTK events on the sentinel's own window.
+
+Control: one command per line appended to --control (activate, arm, disarm, quit); SIGUSR1 activates and SIGUSR2
+toggles the armed state, as on X11.
+"""
+
+from __future__ import annotations
+
+import argparse
+import ctypes
+import json
+import os
+import shutil
+import signal
+import subprocess
+import sys
+import threading
+import time
+from pathlib import Path
+
+import gi
+
+gi.require_version("Gtk", "3.0")
+gi.require_version("Gdk", "3.0")
+from gi.repository import Gdk, GLib, Gtk # noqa: E402
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import hypr # noqa: E402
+
+OWN_BID = "ai.cua.benchsentinel"
+DRAWS_OVER = ("floating", "pinned")
+
+
+def comm(pid: int) -> str:
+ try:
+ return Path(f"/proc/{pid}/comm").read_text().strip()
+ except OSError:
+ return ""
+
+
+class Sentinel:
+ def __init__(self, args: argparse.Namespace) -> None:
+ ctypes.CDLL(None).prctl(15, b"BenchSentinel", 0, 0, 0)
+ GLib.set_prgname("BenchSentinel")
+ GLib.set_application_name("BenchSentinel")
+ self.args = args
+ args.log.parent.mkdir(parents=True, exist_ok=True)
+ self.log = args.log.open("a", buffering=1)
+ self.armed = False
+ self.clip = 0
+ self.expected = ""
+ self.previous = None
+ self.address = ""
+ self.control_at = 0
+ self.lock = threading.Lock()
+ self.latest_front: dict = {}
+ self.title = "User activity" if args.user_activity else "Bench Sentinel"
+ self.window = Gtk.Window(title=self.title)
+ self.window.set_wmclass("BenchSentinel", "BenchSentinel")
+ self.window.set_default_size(600, 320)
+ if args.frame:
+ _, _, w, h = map(int, args.frame.split(","))
+ self.window.set_default_size(w, h)
+ self.entry = Gtk.Entry()
+ self.entry.get_accessible().set_name("User activity text")
+ self.entry.connect("changed", self.changed)
+ self.window.add(self.entry)
+ self.window.connect("destroy", Gtk.main_quit)
+ self.window.connect("focus-out-event", lambda *_: self.emit("didResignKey"))
+ self.window.connect("focus-in-event", lambda *_: self.emit("didBecomeActive"))
+ self.entry.connect("key-press-event", lambda *_: self.emit("keyDown"))
+ self.entry.connect("button-press-event", lambda *_: self.emit("mouseDown"))
+ self.entry.connect("scroll-event", lambda *_: self.emit("scrollWheel"))
+ self.watcher = None
+ if not shutil.which("wl-paste"):
+ # GTK only hears of selection changes while the window has keyboard focus.
+ self.clipboard = Gtk.Clipboard.get(Gdk.SELECTION_CLIPBOARD)
+ self.clipboard.connect("owner-change", self.clip_changed)
+ else:
+ # data-control sees every selection change, focused or not (and the current one when it starts,
+ # long before the sentinel is armed; changes are counted from the armed sample).
+ self.watcher = subprocess.Popen(
+ ["wl-paste", "--watch", "echo", "change"], stdout=subprocess.PIPE,
+ stderr=subprocess.DEVNULL, text=True,
+ ) # fmt: skip
+ threading.Thread(target=self.watch_clipboard, daemon=True).start()
+ self.window.show_all()
+ GLib.unix_signal_add(GLib.PRIORITY_DEFAULT, signal.SIGUSR1, self.activate)
+ GLib.unix_signal_add(GLib.PRIORITY_DEFAULT, signal.SIGUSR2, self.toggle)
+ GLib.timeout_add(100, self.find_window)
+
+ def find_window(self) -> bool:
+ mine = [c for c in hypr.by_pid(os.getpid()) if c.get("title") == self.title]
+ if not mine:
+ return True
+ self.address = mine[0]["address"]
+ if self.args.frame:
+ hypr.place(self.address, tuple(map(int, self.args.frame.split(","))))
+ if not self.args.no_activate:
+ self.activate()
+ # Compositor reads can stall while the compositor is busy; they run off the GTK thread so the user's
+ # typing, focus events and control commands never wait on them.
+ threading.Thread(target=self.sampler, daemon=True).start()
+ GLib.timeout_add(100, self.poll_control)
+ if self.args.user_activity:
+ GLib.timeout_add(500, self.activity)
+ self.emit("ready", address=self.address, pid=os.getpid())
+ return False
+
+ def watch_clipboard(self) -> None:
+ for _ in self.watcher.stdout:
+ GLib.idle_add(self.clip_changed)
+
+ def clip_changed(self, *_: object) -> bool:
+ self.clip += 1
+ return False
+
+ def changed(self, *_: object) -> None:
+ if self.entry.get_text() != self.expected:
+ self.emit("user_text_disrupted")
+
+ def own(self, clients: list[dict]) -> dict:
+ return next((c for c in clients if c.get("address") == self.address), {})
+
+ def identity(self, client: dict) -> dict:
+ if not client:
+ return {"bid": "none", "pid": -1}
+ if client.get("address") == self.address:
+ return {"bid": OWN_BID, "pid": os.getpid()}
+ return {"bid": hypr.identity(client), "pid": int(client.get("pid", -1))}
+
+ def snapshot(self) -> dict:
+ clients = hypr.query("clients") or []
+ own = self.own(clients)
+ workspace = (own.get("workspace") or {}).get("id")
+ front = self.identity(hypr.active_window())
+ x, y = hypr.cursor()
+ above: list[str] = []
+ if own in clients:
+ for c in clients[clients.index(own) + 1 :]:
+ if not c.get("mapped", True) or c.get("hidden"):
+ continue
+ here = c.get("pinned") or (c.get("workspace") or {}).get("id") == workspace
+ over = c.get("floating") or c.get("pinned") or c.get("fullscreen")
+ if here and over and hypr.overlaps(c, own):
+ above.append(self.identity(c)["bid"])
+ for c in clients:
+ if (
+ c is not own
+ and c.get("fullscreen")
+ and (c.get("workspace") or {}).get("id") == workspace
+ ):
+ name = self.identity(c)["bid"]
+ if name not in above:
+ above.append(name)
+ layers = hypr.query("layers") or {}
+ for monitor in layers.values():
+ for level in ("2", "3"):
+ for layer in (monitor.get("levels") or {}).get(level, []):
+ box = {
+ "at": (layer.get("x", 0), layer.get("y", 0)),
+ "size": (layer.get("w", 0), layer.get("h", 0)),
+ }
+ if own and hypr.overlaps(box, own):
+ pid = int(layer.get("pid", -1))
+ owner = comm(pid) or f"pid{pid}"
+ above.append(f"{owner}:layer:{layer.get('namespace', '')}")
+ monitors = hypr.query("monitors") or []
+ monitor = next(
+ (m for m in monitors if m.get("id") == own.get("monitor")),
+ monitors[0] if monitors else {},
+ )
+ special = bool((monitor.get("specialWorkspace") or {}).get("id"))
+ fullscreen_other = any(
+ c is not own
+ and c.get("fullscreen")
+ and (c.get("workspace") or {}).get("id") == workspace
+ for c in clients
+ )
+ onspace = (
+ (monitor.get("activeWorkspace") or {}).get("id") == workspace
+ and not special
+ and not fullscreen_other
+ )
+ sheet = any(
+ c.get("pid") == os.getpid()
+ and c.get("address") != self.address
+ and c.get("mapped", True)
+ for c in clients
+ )
+ return {
+ "front": front,
+ "mouse": [x, y],
+ "clip": self.clip,
+ "onspace": bool(onspace),
+ "sheet": sheet,
+ "above": above,
+ "above_n": len(above),
+ }
+
+ def emit(self, event: str, **details: object) -> bool:
+ if self.armed or event in ("armed", "disarmed", "ready"):
+ with self.lock:
+ self.log.write(json.dumps({"t": time.time() * 1000, "ev": event, **details}) + "\n")
+ return False
+
+ def activate(self, *_: object) -> bool:
+ if self.address:
+ hypr.focus(self.address)
+ hypr.raise_to_top(self.address)
+ self.window.present()
+ self.entry.grab_focus()
+ return True
+
+ def toggle(self, *_: object) -> bool:
+ if self.armed:
+ if self.args.user_activity:
+ self.emit("user_text", intact=self.entry.get_text() == self.expected)
+ self.emit("disarmed", **self.snapshot())
+ self.armed = False
+ else:
+ self.armed = True
+ self.emit("armed", **self.snapshot())
+ return True
+
+ def poll_control(self) -> bool:
+ if not self.args.control or not self.args.control.exists():
+ return True
+ with self.args.control.open(encoding="utf-8") as handle:
+ handle.seek(self.control_at)
+ lines = handle.read().splitlines()
+ self.control_at = handle.tell()
+ for line in lines:
+ command = line.strip()
+ if command == "activate":
+ self.activate()
+ elif command == "arm" and not self.armed:
+ self.toggle()
+ elif command == "disarm" and self.armed:
+ self.toggle()
+ elif command == "quit":
+ Gtk.main_quit()
+ return False
+ return True
+
+ def sampler(self) -> None:
+ while True:
+ started = time.monotonic()
+ self.sample()
+ time.sleep(max(0.0, 0.2 - (time.monotonic() - started)))
+
+ def sample(self) -> None:
+ try:
+ sample = self.snapshot()
+ except (OSError, ValueError, RuntimeError) as error:
+ self.emit("sample_error", error=str(error))
+ return
+ self.latest_front = sample["front"]
+ if not self.armed:
+ return
+ if sample["front"] != self.previous:
+ self.emit("front", **sample)
+ if self.previous and self.previous.get("bid") == OWN_BID:
+ self.emit("didResignActive")
+ self.previous = sample["front"]
+ self.emit("s", **sample)
+
+ def activity(self) -> bool:
+ if self.armed:
+ if self.latest_front.get("bid") == OWN_BID and self.window.has_toplevel_focus():
+ if self.entry.get_text() != self.expected:
+ self.emit("user_text_disrupted")
+ self.expected += "a"
+ self.entry.set_text(self.expected)
+ self.entry.set_position(-1)
+ self.emit("user_type")
+ else:
+ self.emit("user_blocked")
+ return True
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--log", type=Path, required=True)
+ parser.add_argument("--control", type=Path)
+ parser.add_argument("--no-activate", action="store_true")
+ parser.add_argument("--user-activity", action="store_true")
+ parser.add_argument("--frame")
+ witness = Sentinel(parser.parse_args())
+ try:
+ Gtk.main()
+ finally:
+ if witness.watcher:
+ witness.watcher.terminate()
+ witness.log.close()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/launch/hyprland_session.py b/libs/cua-driver-bench/automated-eval/macos_bench/launch/hyprland_session.py
new file mode 100644
index 0000000000..937343f51e
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/launch/hyprland_session.py
@@ -0,0 +1,345 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: FSL-1.1-MIT
+# Copyright (c) 2026 Cua AI, Inc.
+#
+# Usage: hyprland_session.py --task ID --run-index N --condition foreground|background --out DIR
+# [--timeout S] -- COMMAND [ARG...]
+#
+# One BenchLab trial on a Hyprland (Wayland) desktop such as Omarchy, run as the desktop user with the session's
+# environment (WAYLAND_DISPLAY, HYPRLAND_INSTANCE_SIGNATURE, XDG_RUNTIME_DIR, DBUS_SESSION_BUS_ADDRESS):
+# 1. derive the seed (fairness rule), render the brief (+ the background paragraph), write DIR/brief.md;
+# 2. start BenchLab (linux/benchlab.py, GTK3) and the sentinel (hyprland/sentinel.py) and place both through
+# Hyprland's IPC: in the background condition the "User activity" window partly covers BenchLab, is brought
+# to the front and types;
+# 3. arm the sentinel, run COMMAND (the agent; `{brief}`, `{out}`, `{seed}`, `{mode}` in its arguments are filled
+# in) with the GUI-only guard watching, stop it at the time limit or on a violation;
+# 4. disarm, run the task's checker, summarise the sentinel, write DIR/row.json.
+# The desktop needs BenchLab and BenchSentinel to float, and BenchLab's windows not to take focus when they open
+# (hyprland/bench.lua).
+from __future__ import annotations
+
+import argparse
+import importlib.util
+import json
+import os
+import signal
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+BENCH = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(BENCH))
+sys.path.insert(0, str(BENCH / "probes" / "_common"))
+sys.path.insert(0, str(BENCH / "swift"))
+sys.path.insert(0, str(BENCH.parents[1] / "fairness"))
+import bench_v2 as v2 # noqa: E402
+import cdb_fairness # noqa: E402
+import summarize_sentinel # noqa: E402
+
+TASKS = ("MB-09", "MB-10", "MB-11", "IR-01", "IR-02", "IR-03", "IR-04")
+LAB_FRAME = (40, 40, 760, 540)
+WITNESS_FRAME = (1000, 640, 600, 320)
+PLATFORM_WORDS = ("on your Mac", "on this Linux computer")
+RULES = BENCH / "hyprland" / "bench.lua"
+
+
+def apply_rules(H) -> None:
+ """The window rules of hyprland/bench.lua (named, so applying them again replaces them)."""
+ text = RULES.read_text(encoding="utf-8")
+ for chunk in text.split("hl.window_rule(")[1:]:
+ depth, end = 1, 0
+ for i, ch in enumerate(chunk):
+ depth += ch == "("
+ depth -= ch == ")"
+ if depth == 0:
+ end = i
+ break
+ H.request("eval hl.window_rule(" + chunk[: end + 1])
+
+
+def hyprland_module(name: str):
+ """hyprland/NAME.py under its own module name (windows/ has a guard.py too)."""
+ spec = importlib.util.spec_from_file_location(
+ f"hyprland_{name}", BENCH / "hyprland" / f"{name}.py"
+ )
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ return module
+
+
+def task_spec(task: str) -> dict:
+ return json.loads((BENCH / "probes" / task / "task.json").read_text(encoding="utf-8"))
+
+
+def render_brief(task: str, seed: int, condition: str) -> str:
+ spec = importlib.util.spec_from_file_location(
+ f"render_{task}", BENCH / "probes" / task / "render_brief.py"
+ )
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ brief = module.render(seed).replace(*PLATFORM_WORDS)
+ return v2.brief_for(brief, condition)
+
+
+def kill_group(proc: subprocess.Popen) -> None:
+ for sig in (signal.SIGTERM, signal.SIGKILL):
+ try:
+ os.killpg(proc.pid, sig)
+ except OSError:
+ return
+ try:
+ proc.wait(timeout=3)
+ return
+ except subprocess.TimeoutExpired:
+ continue
+
+
+def kill_pid(pid: int) -> None:
+ try:
+ os.kill(pid, signal.SIGKILL)
+ except OSError:
+ pass
+
+
+def wait_for(predicate, limit: float, step: float = 0.1) -> bool:
+ end = time.monotonic() + limit
+ while time.monotonic() < end:
+ if predicate():
+ return True
+ time.sleep(step)
+ return False
+
+
+def sentinel_lines(log: Path) -> list[dict]:
+ if not log.exists():
+ return []
+ out = []
+ for raw in log.read_text(encoding="utf-8").splitlines():
+ try:
+ out.append(json.loads(raw))
+ except json.JSONDecodeError:
+ pass
+ return out
+
+
+def run(args: argparse.Namespace, command: list[str]) -> dict:
+ G = hyprland_module("guard")
+ H = hyprland_module("hypr")
+
+ out: Path = args.out.resolve()
+ out.mkdir(parents=True, exist_ok=True)
+ spec = task_spec(args.task)
+ mode = spec["app_args"][spec["app_args"].index("--mode") + 1]
+ seed = cdb_fairness.trial_seed(args.task, args.run_index)
+ brief = render_brief(args.task, seed, args.condition)
+ (out / "brief.md").write_text(brief, encoding="utf-8")
+ background = args.condition == "background"
+ row: dict = {
+ "task": args.task, "run_index": args.run_index, "seed": seed, "condition": args.condition,
+ "mode": mode, "platform": "linux-hyprland", "label": args.label, "started": time.time(),
+ } # fmt: skip
+
+ state, events, log, control = (
+ out / "state.json", out / "events.jsonl", out / "sentinel.jsonl", out / "sentinel.control",
+ ) # fmt: skip
+ for stale in (state, events, log, control):
+ stale.unlink(missing_ok=True)
+ control.write_text("", encoding="utf-8")
+ apply_rules(H)
+ monitors = H.query("monitors") or []
+ if (
+ not monitors
+ or max(m["width"] for m in monitors) < 1320
+ or max(m["height"] for m in monitors) < 860
+ ):
+ row["harness_error"] = "screen smaller than 1320x860 (load hyprland/bench.lua)"
+ (out / "row.json").write_text(json.dumps(row, indent=1), encoding="utf-8")
+ return row
+ env = {**os.environ, "GDK_BACKEND": "wayland", "NO_AT_BRIDGE": "0"}
+ lab = subprocess.Popen(
+ [sys.executable, str(BENCH / "linux" / "benchlab.py"), "--mode", mode, "--seed", str(seed),
+ "--state", str(state), "--events", str(events)],
+ env=env, start_new_session=True, stdout=(out / "benchlab.log").open("w"), stderr=subprocess.STDOUT,
+ ) # fmt: skip
+ procs = [lab]
+ try:
+ if not wait_for(lambda: state.exists() and H.by_pid(lab.pid), 30):
+ row["harness_error"] = "BenchLab did not start"
+ return row
+ lab_address = H.by_pid(lab.pid)[0]["address"]
+ H.place(lab_address, LAB_FRAME)
+ if not background:
+ H.focus(lab_address)
+ frame = v2.USER_WINDOW_FRAME if background else WITNESS_FRAME
+ sentinel = subprocess.Popen(
+ [sys.executable, str(BENCH / "hyprland" / "sentinel.py"), "--log", str(log),
+ "--control", str(control), "--frame", ",".join(map(str, frame)),
+ *(["--user-activity"] if background else [])],
+ env=env, start_new_session=True, stdout=(out / "sentinel.log").open("w"), stderr=subprocess.STDOUT,
+ ) # fmt: skip
+ procs.append(sentinel)
+ if not wait_for(lambda: any(ln.get("ev") == "ready" for ln in sentinel_lines(log)), 30):
+ row["harness_error"] = "sentinel did not start"
+ return row
+ own = next(ln["address"] for ln in sentinel_lines(log) if ln.get("ev") == "ready")
+
+ def in_front() -> bool:
+ return H.active_window().get("address") == own and H.clients()[-1].get("address") == own
+
+ with control.open("a", encoding="utf-8") as c:
+ c.write("activate\n")
+ front_ok = wait_for(in_front, 10)
+ if not front_ok:
+ with control.open("a", encoding="utf-8") as c:
+ c.write("activate\n")
+ front_ok = wait_for(in_front, 5)
+ row["sentinel_front_at_start"] = front_ok
+ row["frames"] = {
+ c.get("class"): [*c.get("at", []), *c.get("size", [])] for c in H.clients()
+ }
+ time.sleep(1.0)
+ with control.open("a", encoding="utf-8") as c:
+ c.write("arm\n")
+ wait_for(lambda: any(ln.get("ev") == "armed" for ln in sentinel_lines(log)), 5)
+
+ subs = {
+ "{brief}": str(out / "brief.md"),
+ "{out}": str(out),
+ "{seed}": str(seed),
+ "{mode}": mode,
+ }
+ argv = []
+ for a in command:
+ for key, value in subs.items():
+ a = a.replace(key, value)
+ argv.append(a)
+ baseline = {pid for pid, _, _ in G.process_rows()}
+ t0 = time.monotonic()
+ agent = subprocess.Popen(
+ argv, cwd=str(out), env={**os.environ, "CDB_BRIEF_FILE": str(out / "brief.md")},
+ start_new_session=True,
+ ) # fmt: skip
+ procs.append(agent)
+ own_tree: set[int] = set()
+ front = G.FrontWatch()
+ violations: list[dict] = []
+ timed_out = False
+ while agent.poll() is None:
+ rows = G.process_rows()
+ own_tree |= G.descendants(rows, os.getpid())
+ for pid, label in G.new_side_doors(rows, baseline, os.getpid(), own_tree):
+ violations.append(
+ {
+ "t_s": round(time.monotonic() - t0, 1),
+ "kind": "process",
+ "detail": label,
+ "pid": pid,
+ }
+ )
+ kill_pid(pid)
+ try:
+ hit = front.sample(H.active_window().get("class", ""))
+ except (OSError, ValueError, RuntimeError):
+ hit = None
+ if hit:
+ violations.append(
+ {"t_s": round(time.monotonic() - t0, 1), "kind": "front", "detail": hit}
+ )
+ if violations and args.enforce:
+ kill_group(agent)
+ break
+ if time.monotonic() - t0 > args.timeout:
+ timed_out = True
+ kill_group(agent)
+ break
+ time.sleep(0.5)
+ try:
+ agent.wait(timeout=30)
+ except subprocess.TimeoutExpired:
+ kill_group(agent)
+ row["agent_exit"] = agent.returncode
+ row["agent_wall_s"] = round(time.monotonic() - t0, 1)
+ row["timed_out"] = timed_out
+
+ with control.open("a", encoding="utf-8") as c:
+ c.write("disarm\n")
+ wait_for(lambda: any(ln.get("ev") == "disarmed" for ln in sentinel_lines(log)), 5)
+ time.sleep(0.3)
+ result = out / "result.json"
+ subprocess.run(
+ [sys.executable, str(BENCH / "probes" / args.task / "evaluate.py"), "--seed", str(seed),
+ "--state", str(state), "--events", str(events), "--result", str(result)],
+ capture_output=True, timeout=60,
+ ) # fmt: skip
+ evaluation = json.loads(result.read_text(encoding="utf-8")) if result.exists() else {}
+ transcript = out / "agent-transcript.jsonl"
+ stream = []
+ if transcript.exists():
+ for raw in transcript.read_text(encoding="utf-8", errors="replace").splitlines():
+ try:
+ stream.append(json.loads(raw))
+ except json.JSONDecodeError:
+ pass
+ hard, soft = G.scan_events(stream)
+ violations += [{"t_s": None, "kind": "tool_input", "detail": h} for h in hard]
+ disturbance = summarize_sentinel.summarize(str(log))
+ armed = next((ln for ln in sentinel_lines(log) if ln.get("ev") == "armed"), {})
+ frontmost = (armed.get("front") or {}).get("bid") == "ai.cua.benchsentinel"
+ passed = bool(evaluation.get("passed")) and not violations
+ bg = v2.background_score(disturbance, frontmost)
+ row.update(
+ evaluation={k: evaluation.get(k) for k in ("passed", "score", "checks", "diagnostics")},
+ checker_passed=bool(evaluation.get("passed")),
+ gui_only={"violation": bool(violations), "violations": violations[:20],
+ "mentions": soft, "frontmost_seen": sorted(front.seen)},
+ passed=passed,
+ disturbance=disturbance,
+ background=bg,
+ a15=v2.background_condition_score(bg, disturbance, passed, args.condition),
+ a22=v2.multiplayer_score(disturbance, bg["measured"], passed, args.condition),
+ ) # fmt: skip
+ return row
+ finally:
+ with control.open("a", encoding="utf-8") as c:
+ c.write("quit\n")
+ time.sleep(0.5)
+ for proc in procs:
+ if proc.poll() is None:
+ kill_group(proc)
+ row["finished"] = time.time()
+ (out / "row.json").write_text(json.dumps(row, indent=1), encoding="utf-8")
+
+
+def main(argv: list[str]) -> int:
+ if "--" not in argv:
+ print(
+ "usage: hyprland_session.py --task ID --run-index N --condition C --out DIR -- COMMAND...",
+ file=sys.stderr,
+ )
+ return 2
+ split = argv.index("--")
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--task", choices=TASKS, required=True)
+ parser.add_argument("--run-index", type=int, required=True)
+ parser.add_argument("--condition", choices=v2.CONDITIONS, required=True)
+ parser.add_argument("--out", type=Path, required=True)
+ parser.add_argument("--label", default="")
+ parser.add_argument("--timeout", type=float, default=360.0)
+ parser.add_argument("--no-enforce", dest="enforce", action="store_false")
+ args = parser.parse_args(argv[:split])
+ command = argv[split + 1 :]
+ if not os.environ.get("HYPRLAND_INSTANCE_SIGNATURE") or not command:
+ print("hyprland_session.py needs a Hyprland session and a command", file=sys.stderr)
+ return 2
+ row = run(args, command)
+ print(
+ json.dumps(
+ {k: row.get(k) for k in ("task", "seed", "condition", "passed", "harness_error")}
+ )
+ )
+ return 0 if not row.get("harness_error") else 3
+
+
+if __name__ == "__main__":
+ sys.exit(main(sys.argv[1:]))
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-01/launch.hyprland.json b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-01/launch.hyprland.json
new file mode 100644
index 0000000000..fbeae24d56
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-01/launch.hyprland.json
@@ -0,0 +1,48 @@
+{
+ "platform": "linux-hyprland",
+ "descriptor_version": "1",
+ "task_id": "IR-01",
+ "app": "linux/benchlab.py",
+ "app_args": [
+ "--mode",
+ "irmodal"
+ ],
+ "app_executable": "python3 linux/benchlab.py (GTK3, native Wayland; process name and app id BenchLab)",
+ "sentinel": "hyprland/sentinel.py",
+ "launcher": [
+ "${python}",
+ "launch/hyprland_session.py",
+ "--task",
+ "IR-01",
+ "--run-index",
+ "${run_index}",
+ "--condition",
+ "${condition}",
+ "--out",
+ "${artifacts}",
+ "--",
+ "${agent_command}"
+ ],
+ "conditions": [
+ "foreground",
+ "background"
+ ],
+ "evaluate": [
+ "${python}",
+ "probes/IR-01/evaluate.py",
+ "--seed",
+ "${seed}",
+ "--state",
+ "${artifacts}/state.json",
+ "--events",
+ "${artifacts}/events.jsonl",
+ "--result",
+ "${result}"
+ ],
+ "requires": [
+ "Python 3.11 or later with PyGObject, GTK 3 and AT-SPI",
+ "Hyprland 0.55 or later with a Lua config, hyprland/bench.lua loaded (1920x1080)",
+ "wl-clipboard",
+ "the desktop user's Hyprland session environment"
+ ]
+}
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-02/launch.hyprland.json b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-02/launch.hyprland.json
new file mode 100644
index 0000000000..74d934a222
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-02/launch.hyprland.json
@@ -0,0 +1,48 @@
+{
+ "platform": "linux-hyprland",
+ "descriptor_version": "1",
+ "task_id": "IR-02",
+ "app": "linux/benchlab.py",
+ "app_args": [
+ "--mode",
+ "irbanner"
+ ],
+ "app_executable": "python3 linux/benchlab.py (GTK3, native Wayland; process name and app id BenchLab)",
+ "sentinel": "hyprland/sentinel.py",
+ "launcher": [
+ "${python}",
+ "launch/hyprland_session.py",
+ "--task",
+ "IR-02",
+ "--run-index",
+ "${run_index}",
+ "--condition",
+ "${condition}",
+ "--out",
+ "${artifacts}",
+ "--",
+ "${agent_command}"
+ ],
+ "conditions": [
+ "foreground",
+ "background"
+ ],
+ "evaluate": [
+ "${python}",
+ "probes/IR-02/evaluate.py",
+ "--seed",
+ "${seed}",
+ "--state",
+ "${artifacts}/state.json",
+ "--events",
+ "${artifacts}/events.jsonl",
+ "--result",
+ "${result}"
+ ],
+ "requires": [
+ "Python 3.11 or later with PyGObject, GTK 3 and AT-SPI",
+ "Hyprland 0.55 or later with a Lua config, hyprland/bench.lua loaded (1920x1080)",
+ "wl-clipboard",
+ "the desktop user's Hyprland session environment"
+ ]
+}
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-03/launch.hyprland.json b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-03/launch.hyprland.json
new file mode 100644
index 0000000000..d00b3ad43c
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-03/launch.hyprland.json
@@ -0,0 +1,48 @@
+{
+ "platform": "linux-hyprland",
+ "descriptor_version": "1",
+ "task_id": "IR-03",
+ "app": "linux/benchlab.py",
+ "app_args": [
+ "--mode",
+ "irunsaved"
+ ],
+ "app_executable": "python3 linux/benchlab.py (GTK3, native Wayland; process name and app id BenchLab)",
+ "sentinel": "hyprland/sentinel.py",
+ "launcher": [
+ "${python}",
+ "launch/hyprland_session.py",
+ "--task",
+ "IR-03",
+ "--run-index",
+ "${run_index}",
+ "--condition",
+ "${condition}",
+ "--out",
+ "${artifacts}",
+ "--",
+ "${agent_command}"
+ ],
+ "conditions": [
+ "foreground",
+ "background"
+ ],
+ "evaluate": [
+ "${python}",
+ "probes/IR-03/evaluate.py",
+ "--seed",
+ "${seed}",
+ "--state",
+ "${artifacts}/state.json",
+ "--events",
+ "${artifacts}/events.jsonl",
+ "--result",
+ "${result}"
+ ],
+ "requires": [
+ "Python 3.11 or later with PyGObject, GTK 3 and AT-SPI",
+ "Hyprland 0.55 or later with a Lua config, hyprland/bench.lua loaded (1920x1080)",
+ "wl-clipboard",
+ "the desktop user's Hyprland session environment"
+ ]
+}
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-04/launch.hyprland.json b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-04/launch.hyprland.json
new file mode 100644
index 0000000000..ff398b7d01
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/probes/IR-04/launch.hyprland.json
@@ -0,0 +1,48 @@
+{
+ "platform": "linux-hyprland",
+ "descriptor_version": "1",
+ "task_id": "IR-04",
+ "app": "linux/benchlab.py",
+ "app_args": [
+ "--mode",
+ "irconsent"
+ ],
+ "app_executable": "python3 linux/benchlab.py (GTK3, native Wayland; process name and app id BenchLab)",
+ "sentinel": "hyprland/sentinel.py",
+ "launcher": [
+ "${python}",
+ "launch/hyprland_session.py",
+ "--task",
+ "IR-04",
+ "--run-index",
+ "${run_index}",
+ "--condition",
+ "${condition}",
+ "--out",
+ "${artifacts}",
+ "--",
+ "${agent_command}"
+ ],
+ "conditions": [
+ "foreground",
+ "background"
+ ],
+ "evaluate": [
+ "${python}",
+ "probes/IR-04/evaluate.py",
+ "--seed",
+ "${seed}",
+ "--state",
+ "${artifacts}/state.json",
+ "--events",
+ "${artifacts}/events.jsonl",
+ "--result",
+ "${result}"
+ ],
+ "requires": [
+ "Python 3.11 or later with PyGObject, GTK 3 and AT-SPI",
+ "Hyprland 0.55 or later with a Lua config, hyprland/bench.lua loaded (1920x1080)",
+ "wl-clipboard",
+ "the desktop user's Hyprland session environment"
+ ]
+}
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-09/launch.hyprland.json b/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-09/launch.hyprland.json
new file mode 100644
index 0000000000..e3d9b59738
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-09/launch.hyprland.json
@@ -0,0 +1,48 @@
+{
+ "platform": "linux-hyprland",
+ "descriptor_version": "1",
+ "task_id": "MB-09",
+ "app": "linux/benchlab.py",
+ "app_args": [
+ "--mode",
+ "listdrag"
+ ],
+ "app_executable": "python3 linux/benchlab.py (GTK3, native Wayland; process name and app id BenchLab)",
+ "sentinel": "hyprland/sentinel.py",
+ "launcher": [
+ "${python}",
+ "launch/hyprland_session.py",
+ "--task",
+ "MB-09",
+ "--run-index",
+ "${run_index}",
+ "--condition",
+ "${condition}",
+ "--out",
+ "${artifacts}",
+ "--",
+ "${agent_command}"
+ ],
+ "conditions": [
+ "foreground",
+ "background"
+ ],
+ "evaluate": [
+ "${python}",
+ "probes/MB-09/evaluate.py",
+ "--seed",
+ "${seed}",
+ "--state",
+ "${artifacts}/state.json",
+ "--events",
+ "${artifacts}/events.jsonl",
+ "--result",
+ "${result}"
+ ],
+ "requires": [
+ "Python 3.11 or later with PyGObject, GTK 3 and AT-SPI",
+ "Hyprland 0.55 or later with a Lua config, hyprland/bench.lua loaded (1920x1080)",
+ "wl-clipboard",
+ "the desktop user's Hyprland session environment"
+ ]
+}
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-10/launch.hyprland.json b/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-10/launch.hyprland.json
new file mode 100644
index 0000000000..dff53fac10
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-10/launch.hyprland.json
@@ -0,0 +1,47 @@
+{
+ "platform": "linux-hyprland",
+ "descriptor_version": "1",
+ "task_id": "MB-10",
+ "app": "linux/benchlab.py",
+ "app_args": [
+ "--mode",
+ "hover"
+ ],
+ "app_executable": "python3 linux/benchlab.py (GTK3, native Wayland; process name and app id BenchLab)",
+ "sentinel": "hyprland/sentinel.py",
+ "launcher": [
+ "${python}",
+ "launch/hyprland_session.py",
+ "--task",
+ "MB-10",
+ "--run-index",
+ "${run_index}",
+ "--condition",
+ "${condition}",
+ "--out",
+ "${artifacts}",
+ "--",
+ "${agent_command}"
+ ],
+ "conditions": [
+ "foreground"
+ ],
+ "evaluate": [
+ "${python}",
+ "probes/MB-10/evaluate.py",
+ "--seed",
+ "${seed}",
+ "--state",
+ "${artifacts}/state.json",
+ "--events",
+ "${artifacts}/events.jsonl",
+ "--result",
+ "${result}"
+ ],
+ "requires": [
+ "Python 3.11 or later with PyGObject, GTK 3 and AT-SPI",
+ "Hyprland 0.55 or later with a Lua config, hyprland/bench.lua loaded (1920x1080)",
+ "wl-clipboard",
+ "the desktop user's Hyprland session environment"
+ ]
+}
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-11/launch.hyprland.json b/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-11/launch.hyprland.json
new file mode 100644
index 0000000000..02ab4f0603
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/probes/MB-11/launch.hyprland.json
@@ -0,0 +1,47 @@
+{
+ "platform": "linux-hyprland",
+ "descriptor_version": "1",
+ "task_id": "MB-11",
+ "app": "linux/benchlab.py",
+ "app_args": [
+ "--mode",
+ "tooltip"
+ ],
+ "app_executable": "python3 linux/benchlab.py (GTK3, native Wayland; process name and app id BenchLab)",
+ "sentinel": "hyprland/sentinel.py",
+ "launcher": [
+ "${python}",
+ "launch/hyprland_session.py",
+ "--task",
+ "MB-11",
+ "--run-index",
+ "${run_index}",
+ "--condition",
+ "${condition}",
+ "--out",
+ "${artifacts}",
+ "--",
+ "${agent_command}"
+ ],
+ "conditions": [
+ "foreground"
+ ],
+ "evaluate": [
+ "${python}",
+ "probes/MB-11/evaluate.py",
+ "--seed",
+ "${seed}",
+ "--state",
+ "${artifacts}/state.json",
+ "--events",
+ "${artifacts}/events.jsonl",
+ "--result",
+ "${result}"
+ ],
+ "requires": [
+ "Python 3.11 or later with PyGObject, GTK 3 and AT-SPI",
+ "Hyprland 0.55 or later with a Lua config, hyprland/bench.lua loaded (1920x1080)",
+ "wl-clipboard",
+ "the desktop user's Hyprland session environment"
+ ]
+}
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/swift/summarize_sentinel.py b/libs/cua-driver-bench/automated-eval/macos_bench/swift/summarize_sentinel.py
index 7cdb32ba0b..c69bed17bc 100644
--- a/libs/cua-driver-bench/automated-eval/macos_bench/swift/summarize_sentinel.py
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/swift/summarize_sentinel.py
@@ -21,9 +21,10 @@ SENTINEL_BID = "ai.cua.benchsentinel"
DEVIATION_PX = 10.0
HID_KEYS = ("move", "down", "key", "scroll")
# Amendment 15: overlay and indicator windows of the computer-use tools themselves (the agent cursor, the
-# "ChatGPT Computer Use" banner, the `cua-driver` process on Windows) are not raises. They are logged as metadata (`overlay_owners`) only.
+# "ChatGPT Computer Use" banner, the `cua-driver` process on Windows, the cua-driver agent cursor's Wayland layer
+# `cua-agent-cursor`) are not raises. They are logged as metadata (`overlay_owners`) only.
OVERLAY_OWNER_RE = re.compile(
- r"(?i)cua[ _-]?driver|chatgpt computer use|computer use|codex|app-cu-helper|claude"
+ r"(?i)cua[ _-]?driver|cua-agent-cursor|chatgpt computer use|computer use|codex|app-cu-helper|claude"
)
FRONT_EVENTS = ("s", "front", "armed", "disarmed")
# Idle values only ever grow between samples unless a real event resets them.
diff --git a/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_hyprland_tools.py b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_hyprland_tools.py
new file mode 100644
index 0000000000..516baa25df
--- /dev/null
+++ b/libs/cua-driver-bench/automated-eval/macos_bench/tests/test_hyprland_tools.py
@@ -0,0 +1,153 @@
+# SPDX-License-Identifier: FSL-1.1-MIT
+# Copyright (c) 2026 Cua AI, Inc.
+
+"""Hyprland (Wayland) port contracts that hold on every OS (no GUI, no model): the GUI-only guard, the brief,
+the launch descriptors, the window rules, window geometry, and that the Hyprland sentinel writes every event
+the X11 sentinel writes."""
+
+from __future__ import annotations
+
+import importlib.util
+import json
+import re
+import sys
+import unittest
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+BENCH = HERE.parent
+sys.path.insert(0, str(BENCH))
+sys.path.insert(0, str(BENCH / "launch"))
+import bench_v2 as v2 # noqa: E402
+import hyprland_session as HS # noqa: E402
+
+G = HS.hyprland_module("guard")
+H = HS.hyprland_module("hypr")
+BENCHLAB_TASKS = ("MB-09", "MB-10", "MB-11", "IR-01", "IR-02", "IR-03", "IR-04")
+
+
+def _emitted(path: Path) -> set[str]:
+ return set(re.findall(r'self\.emit\(\s*"([a-z_A-Z]+)"', path.read_text("utf-8")))
+
+
+class PortTest(unittest.TestCase):
+ def test_sentinel_events_cover_linux(self) -> None:
+ linux = _emitted(BENCH / "linux/sentinel.py") - {
+ "ax_focus"
+ } # AT-SPI focus trace: diagnostic only
+ self.assertLessEqual(linux, _emitted(BENCH / "hyprland/sentinel.py"))
+
+ def test_every_benchlab_task_has_a_hyprland_descriptor(self) -> None:
+ manifest = {t["id"]: t for t in v2.load_taskset()["tasks"]}
+ for tid in BENCHLAB_TASKS:
+ spec = json.loads((BENCH / "probes" / tid / "task.json").read_text("utf-8"))
+ launch = json.loads(
+ (BENCH / "probes" / tid / "launch.hyprland.json").read_text("utf-8")
+ )
+ self.assertEqual(launch["platform"], "linux-hyprland")
+ self.assertEqual(launch["app_args"], spec["app_args"], tid)
+ self.assertEqual(launch["conditions"], manifest[tid]["conditions"], tid)
+ self.assertIn("linux", manifest[tid]["os"])
+ self.assertIn("launch/hyprland_session.py", launch["launcher"])
+ self.assertEqual(set(HS.TASKS), set(BENCHLAB_TASKS))
+
+ def test_window_rules(self) -> None:
+ sent = []
+
+ class Fake:
+ @staticmethod
+ def request(command: str) -> str:
+ sent.append(command)
+ return "ok"
+
+ HS.apply_rules(Fake)
+ self.assertEqual(len(sent), 3)
+ self.assertTrue(
+ all(c.startswith("eval hl.window_rule({") and c.endswith("})") for c in sent)
+ )
+ dialogs = next(c for c in sent if "cdb-benchlab-dialogs" in c)
+ x, y = map(int, re.search(r'move = "(\d+) (\d+)"', dialogs).groups())
+ ux, uy, uw, uh = v2.USER_WINDOW_FRAME
+ lx, ly, lw, lh = HS.LAB_FRAME
+ # Dialogs open inside BenchLab's frame, above the top edge of the user window.
+ self.assertTrue(lx <= x < lx + lw and ly <= y < uy, (x, y))
+
+
+class OverlayTest(unittest.TestCase):
+ def test_agent_cursor_layer_is_the_tools_own(self) -> None:
+ sys.path.insert(0, str(BENCH / "swift"))
+ import summarize_sentinel
+
+ for name in ("cua-driver:layer:cua-agent-cursor", "pid123:layer:cua-agent-cursor"):
+ self.assertTrue(summarize_sentinel.OVERLAY_OWNER_RE.search(name), name)
+ self.assertFalse(summarize_sentinel.OVERLAY_OWNER_RE.search("quickshell:layer:omarchy-bar"))
+
+
+class BriefTest(unittest.TestCase):
+ def test_platform_words_and_background_paragraph(self) -> None:
+ for tid in BENCHLAB_TASKS:
+ fg = HS.render_brief(tid, 123456, "foreground")
+ bg = HS.render_brief(tid, 123456, "background")
+ self.assertNotIn("your Mac", fg, tid)
+ self.assertIn("on this Linux computer", fg, tid)
+ self.assertNotIn(v2.BACKGROUND_BRIEF, fg)
+ self.assertTrue(bg.rstrip().endswith(v2.BACKGROUND_BRIEF), tid)
+ self.assertNotIn("{{", bg, tid)
+
+
+class GeometryTest(unittest.TestCase):
+ def test_overlaps(self) -> None:
+ user = {"at": [420, 340], "size": [900, 520]}
+ self.assertTrue(H.overlaps({"at": [40, 40], "size": [760, 540]}, user))
+ self.assertFalse(H.overlaps({"at": [80, 120], "size": [296, 105]}, user))
+ self.assertFalse(H.overlaps({"at": [1521, 0], "size": [399, 1080]}, user))
+
+ def test_identity(self) -> None:
+ self.assertEqual(H.identity({"class": "BenchLab", "title": "x"}), "BenchLab")
+ self.assertEqual(H.identity({"class": "", "initialClass": "", "title": "t"}), "t")
+
+
+class GuardTest(unittest.TestCase):
+ ROWS = [
+ (1, 0, "systemd"),
+ (100, 1, "Hyprland"),
+ (200, 1, "python3"), # the runner
+ (210, 200, "python3"),
+ (220, 210, "kitty"), # under the runner: not a violation
+ (230, 100, "alacritty"), # started from the desktop: a violation
+ (240, 100, "bash"), # a shell alone is not counted
+ (250, 999, "foot"), # parent gone, but seen under the runner before: the runner's
+ (260, 100, "ghostty"),
+ ]
+
+ def test_new_side_doors(self) -> None:
+ hits = G.new_side_doors(self.ROWS, baseline={1, 100}, runner=200, own={250})
+ self.assertEqual(sorted(hits), [(230, "Alacritty"), (260, "Ghostty")])
+ self.assertEqual(
+ G.new_side_doors(self.ROWS, baseline={p for p, _, _ in self.ROWS}, runner=200), []
+ )
+
+ def test_tool_input_patterns(self) -> None:
+ hard, _ = G.scan_tool_use("mcp__x__launch_app", {"name": "kitty"})
+ self.assertEqual(hard, ["mcp__x__launch_app:kitty"])
+ hard, _ = G.scan_tool_use("mcp__x__type_text", {"text": "hyprctl dispatch exec foot"})
+ self.assertIn("mcp__x__type_text:hyprctl", hard)
+ self.assertIn("mcp__x__type_text:foot", hard)
+ self.assertEqual(
+ G.scan_tool_use("mcp__x__type_text", {"text": "Football club footer"}), ([], [])
+ )
+ self.assertEqual(G.scan_tool_use("Bash", {"command": "ls"})[0], ["builtin:Bash"])
+
+ def test_front_grace(self) -> None:
+ watch = G.FrontWatch()
+ self.assertIsNone(watch.sample("com.mitchellh.ghostty"))
+ self.assertIsNone(watch.sample("com.mitchellh.ghostty"))
+ self.assertEqual(watch.sample("com.mitchellh.ghostty"), "Ghostty")
+ self.assertIsNone(watch.sample("BenchLab"))
+ for _ in range(2):
+ self.assertIsNone(watch.sample("org.omarchy.btop"))
+ self.assertEqual(watch.sample("org.omarchy.btop"), "Omarchy TUI")
+
+
+if __name__ == "__main__":
+ unittest.main()