Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): reach the jev-use OmniParser fallback for custom-painted native views (#4298)

Francesco Bonacci43328a558cb8
diff --git a/libs/cua-driver/examples/jev-use/README.md b/libs/cua-driver/examples/jev-use/README.md
index 226a358010..7ddc3a4b56 100644
--- a/libs/cua-driver/examples/jev-use/README.md
+++ b/libs/cua-driver/examples/jev-use/README.md
@@ -93,6 +93,21 @@ task parameters. Actions are element-bound `click`, `set_value`, or
observation. Native steps send `cua.jev_choice_request_v2`, which adds a
per-candidate `source` and compact value-free `elements`.
+Visual regions from OmniParser are parsed, from the same `capture_id`, only
+when the tree offers nothing to act through: Driver reports `ax_tree_empty`, a
+complete tree lacks a task's declared visual target, or a non-truncated tree,
+observed again with a larger budget, holds only window roots and window chrome
+(`no_application_elements`). The last case covers custom-painted surfaces,
+for which Linux X11 returns only window metadata, Windows UIA only the title
+bar, and macOS AX only the unlabeled window buttons and the application's
+global menu bar (which on macOS are not window content). A partial tree with
+any application element never falls back. Visual targets must reach an OCR
+confidence of 0.8 unless a task opts into a lower bar; an exact, unique text
+match is required either way.
+A visual click is background first; after a structured background refusal,
+the next step offers a separate `<id>:foreground` candidate when the task
+allows foreground delivery.
+
The same three tasks run on three repository harnesses: `<harness>-counter`
(set the counter to 3), `<harness>-save-note` (set the Note field and save),
and `<harness>-choose-size` (select Large and check I agree). The harness is
@@ -100,7 +115,16 @@ and `<harness>-choose-size` (select Large and check I agree). The harness is
mode, each harness shows the same labeled controls, so candidate IDs and the
mock provider's choices are identical on every platform. Task mode is selected
with `CUA_APPKIT_TASK_STATE`, `CUA_WPF_TASK_STATE`, or `CUA_GTK3_TASK_STATE`.
-WPF and GTK3 show a small dedicated task window. Each harness rewrites that
+WPF and GTK3 show a small dedicated task window.
+A fourth task, `canvas-save`, clicks the Save card on the cross-platform
+visual-only canvas fixture (a custom-painted Tk window with no accessibility
+tree), so it can only succeed through the visual fallback and needs the
+cua-perception extension. It observes only the window's top level
+(`max_depth: 1`), so macOS does not spend the walk budget in the menu bar, and
+accepts the Save label at OCR confidence 0.7, because a 1x macOS capture reads
+it at about 0.75. Run it with `verify_native.py --harness canvas`; the
+verifier serves the loopback journal the fixture publishes its state to and
+writes that state to the task's state file. Each harness rewrites that
state file on every change, and the file is the independent oracle. On
Windows, the title bar's System menu, Minimize, Maximize, and Close buttons
are window chrome and never become candidates. With Cua Driver 0.30.1 or
diff --git a/libs/cua-driver/examples/jev-use/fixtures/native/canvas-linux-window-state-initial-v1.json b/libs/cua-driver/examples/jev-use/fixtures/native/canvas-linux-window-state-initial-v1.json
new file mode 100644
index 0000000000..f2485eb93e
--- /dev/null
+++ b/libs/cua-driver/examples/jev-use/fixtures/native/canvas-linux-window-state-initial-v1.json
@@ -0,0 +1,47 @@
+{
+ "_fixture": {
+ "sanitized": "Recorded by verify_native.py --capture-dir. Screenshot bytes, tree_markdown, local file paths, and Driver guidance notes removed.",
+ "source": "Cua visual-only canvas fixture (tests/fixtures/apps/cross-platform/visual-only-canvas, Tk 8.6), observed by get_window_state: Linux X11 (Ubuntu 24.04, Xvfb + XFCE), released Cua Driver 0.30.2. Tk exposes no AT-SPI tree; Driver reports X11 property metadata only."
+ },
+ "app_name": "Tk",
+ "bounds_complete": true,
+ "capture_id": "capture_23db64ce474c4e579e06b77249bb18ca_0000000000000000",
+ "coordinate_frame": "window",
+ "degraded": true,
+ "degraded_reason": "x11_property_fallback_partial: AT-SPI was unavailable and Cua Driver only recovered window metadata. Treat it as discovery evidence; it cannot prove checked state.",
+ "element_count": 1,
+ "elements": [
+ {
+ "actions": [
+ "activate"
+ ],
+ "depth": 0,
+ "description": "tk.Tk",
+ "element_index": 0,
+ "element_token": "s00000001:0",
+ "label": "Cua Visual-Only Canvas Fixture",
+ "role": "window"
+ }
+ ],
+ "elements_complete": false,
+ "nodes_pending": 0,
+ "nodes_visited": 0,
+ "pid": 3549,
+ "returned_element_count": 1,
+ "screenshot_height": 460,
+ "screenshot_mime_type": "image/png",
+ "screenshot_width": 760,
+ "snapshot_id": "s00000001",
+ "timeout_ms": 1000,
+ "total_element_count": 1,
+ "truncated": false,
+ "walk_elapsed_ms": 465,
+ "window_bounds": {
+ "height": 460,
+ "width": 760,
+ "x": 580,
+ "y": 335
+ },
+ "window_id": 50331652,
+ "window_title": "Cua Visual-Only Canvas Fixture"
+}
diff --git a/libs/cua-driver/examples/jev-use/fixtures/native/canvas-macos-window-state-initial-v1.json b/libs/cua-driver/examples/jev-use/fixtures/native/canvas-macos-window-state-initial-v1.json
new file mode 100644
index 0000000000..7fb63d0855
--- /dev/null
+++ b/libs/cua-driver/examples/jev-use/fixtures/native/canvas-macos-window-state-initial-v1.json
@@ -0,0 +1,312 @@
+{
+ "_fixture": {
+ "sanitized": "Recorded with the runner's re-observation arguments (the canvas task's max_depth 1, timeout_ms 5000). Screenshot bytes, tree_markdown, local file paths, and Driver guidance notes removed.",
+ "source": "Cua visual-only canvas fixture (tests/fixtures/apps/cross-platform/visual-only-canvas, Tk 9.0), observed by get_window_state: macOS 26.5.2 Lume VM (1x display), released Cua Driver 0.30.2. AX exposes the window, its two unlabeled window buttons, and the application menu bar."
+ },
+ "app_name": "python3.12",
+ "background_input": {
+ "exact_window": {
+ "pid": 2272,
+ "status": "matched",
+ "window_id": 190
+ },
+ "observation": {
+ "frame_freshness": "unknown",
+ "one_shot_capture": "available"
+ },
+ "routes": [
+ {
+ "route": "accessibility",
+ "status": "available"
+ },
+ {
+ "route": "window_pointer",
+ "status": "available"
+ },
+ {
+ "route": "pid_keyboard",
+ "status": "available"
+ }
+ ]
+ },
+ "capture_id": "capture_951dfc0b72bd4f26b0615a0f6706ec8f_0000000000000023",
+ "element_count": 10,
+ "elements": [
+ {
+ "actions": [
+ "AXRaise"
+ ],
+ "depth": 0,
+ "element_index": 0,
+ "element_token": "s00000024:0",
+ "frame": {
+ "h": 488.0,
+ "w": 760.0,
+ "x": 5.0,
+ "y": 35.0
+ },
+ "label": "Cua Visual-Only Canvas Fixture",
+ "role": "AXWindow",
+ "screenshot_frame": {
+ "h": 488,
+ "w": 760,
+ "x": 0,
+ "y": 0
+ }
+ },
+ {
+ "actions": [
+ "AXPress"
+ ],
+ "depth": 1,
+ "element_index": 1,
+ "element_token": "s00000024:1",
+ "enabled": true,
+ "frame": {
+ "h": 14.0,
+ "w": 12.0,
+ "x": 13.0,
+ "y": 42.0
+ },
+ "parent_index": 0,
+ "role": "AXButton",
+ "screenshot_frame": {
+ "h": 14,
+ "w": 12,
+ "x": 8,
+ "y": 7
+ }
+ },
+ {
+ "actions": [
+ "AXPress"
+ ],
+ "depth": 1,
+ "element_index": 2,
+ "element_token": "s00000024:2",
+ "enabled": true,
+ "frame": {
+ "h": 14.0,
+ "w": 12.0,
+ "x": 33.0,
+ "y": 42.0
+ },
+ "parent_index": 0,
+ "role": "AXButton",
+ "screenshot_frame": {
+ "h": 14,
+ "w": 12,
+ "x": 28,
+ "y": 7
+ }
+ },
+ {
+ "actions": [
+ "AXCancel"
+ ],
+ "depth": 0,
+ "element_index": 3,
+ "element_token": "s00000024:3",
+ "enabled": true,
+ "frame": {
+ "h": 30.0,
+ "w": 1024.0,
+ "x": 0.0,
+ "y": 0.0
+ },
+ "role": "AXMenuBar",
+ "screenshot_frame": {
+ "h": 30,
+ "w": 1024,
+ "x": -5,
+ "y": -35
+ },
+ "selected": false
+ },
+ {
+ "actions": [
+ "AXCancel",
+ "AXPress",
+ "AXPick"
+ ],
+ "depth": 1,
+ "element_index": 4,
+ "element_token": "s00000024:4",
+ "enabled": true,
+ "frame": {
+ "h": 30.0,
+ "w": 34.0,
+ "x": 10.0,
+ "y": 0.0
+ },
+ "label": "Apple",
+ "parent_index": 3,
+ "role": "AXMenuBarItem",
+ "screenshot_frame": {
+ "h": 30,
+ "w": 34,
+ "x": 5,
+ "y": -35
+ },
+ "selected": false
+ },
+ {
+ "actions": [
+ "AXCancel",
+ "AXPress",
+ "AXPick"
+ ],
+ "depth": 1,
+ "element_index": 5,
+ "element_token": "s00000024:5",
+ "enabled": true,
+ "frame": {
+ "h": 30.0,
+ "w": 93.0,
+ "x": 44.0,
+ "y": 0.0
+ },
+ "label": "python3.12",
+ "parent_index": 3,
+ "role": "AXMenuBarItem",
+ "screenshot_frame": {
+ "h": 30,
+ "w": 93,
+ "x": 39,
+ "y": -35
+ },
+ "selected": false
+ },
+ {
+ "actions": [
+ "AXCancel",
+ "AXPress",
+ "AXPick"
+ ],
+ "depth": 1,
+ "element_index": 6,
+ "element_token": "s00000024:6",
+ "enabled": true,
+ "frame": {
+ "h": 30.0,
+ "w": 42.0,
+ "x": 137.0,
+ "y": 0.0
+ },
+ "label": "File",
+ "parent_index": 3,
+ "role": "AXMenuBarItem",
+ "screenshot_frame": {
+ "h": 30,
+ "w": 42,
+ "x": 132,
+ "y": -35
+ },
+ "selected": false
+ },
+ {
+ "actions": [
+ "AXCancel",
+ "AXPress",
+ "AXPick"
+ ],
+ "depth": 1,
+ "element_index": 7,
+ "element_token": "s00000024:7",
+ "enabled": true,
+ "frame": {
+ "h": 30.0,
+ "w": 44.0,
+ "x": 179.0,
+ "y": 0.0
+ },
+ "label": "Edit",
+ "parent_index": 3,
+ "role": "AXMenuBarItem",
+ "screenshot_frame": {
+ "h": 30,
+ "w": 44,
+ "x": 174,
+ "y": -35
+ },
+ "selected": false
+ },
+ {
+ "actions": [
+ "AXCancel",
+ "AXPress",
+ "AXPick"
+ ],
+ "depth": 1,
+ "element_index": 8,
+ "element_token": "s00000024:8",
+ "enabled": true,
+ "frame": {
+ "h": 30.0,
+ "w": 69.0,
+ "x": 223.0,
+ "y": 0.0
+ },
+ "label": "Window",
+ "parent_index": 3,
+ "role": "AXMenuBarItem",
+ "screenshot_frame": {
+ "h": 30,
+ "w": 69,
+ "x": 218,
+ "y": -35
+ },
+ "selected": false
+ },
+ {
+ "actions": [
+ "AXCancel",
+ "AXPress",
+ "AXPick"
+ ],
+ "depth": 1,
+ "element_index": 9,
+ "element_token": "s00000024:9",
+ "enabled": true,
+ "frame": {
+ "h": 30.0,
+ "w": 48.0,
+ "x": 292.0,
+ "y": 0.0
+ },
+ "label": "Help",
+ "parent_index": 3,
+ "role": "AXMenuBarItem",
+ "screenshot_frame": {
+ "h": 30,
+ "w": 48,
+ "x": 287,
+ "y": -35
+ },
+ "selected": false
+ }
+ ],
+ "elements_complete": false,
+ "nodes_pending": 0,
+ "nodes_visited": 12,
+ "pid": 2272,
+ "returned_element_count": 10,
+ "screenshot_frame_valid": true,
+ "screenshot_height": 488,
+ "screenshot_mime_type": "image/png",
+ "screenshot_scale": 1.0,
+ "screenshot_width": 760,
+ "snapshot_id": "s00000024",
+ "timeout_ms": 5000,
+ "total_element_count": 10,
+ "truncated": false,
+ "walk_elapsed_ms": 960,
+ "window_bounds": {
+ "height": 488.0,
+ "width": 760.0,
+ "x": 5.0,
+ "y": 35.0
+ },
+ "window_id": 190,
+ "window_title": "Cua Visual-Only Canvas Fixture"
+}
diff --git a/libs/cua-driver/examples/jev-use/fixtures/native/canvas-windows-window-state-initial-v1.json b/libs/cua-driver/examples/jev-use/fixtures/native/canvas-windows-window-state-initial-v1.json
new file mode 100644
index 0000000000..2cbe0210ed
--- /dev/null
+++ b/libs/cua-driver/examples/jev-use/fixtures/native/canvas-windows-window-state-initial-v1.json
@@ -0,0 +1,129 @@
+{
+ "_fixture": {
+ "sanitized": "Recorded by verify_native.py --capture-dir. Screenshot bytes, tree_markdown, local file paths, and Driver guidance notes removed.",
+ "source": "Cua visual-only canvas fixture (tests/fixtures/apps/cross-platform/visual-only-canvas, Tk 8.6), observed by get_window_state: Windows 11 25H2 interactive RDP session, released Cua Driver 0.30.2. UIA exposes only the title bar."
+ },
+ "app_name": "python.exe",
+ "capture_id": "capture_9fdc37ed0b5b489092199ca0013c7ba0_0000000000000000",
+ "element_count": 4,
+ "elements": [
+ {
+ "actions": [
+ "set_value"
+ ],
+ "depth": 1,
+ "element_index": 0,
+ "element_token": "s00000001:0",
+ "enabled": true,
+ "frame": {
+ "h": 28,
+ "w": 744,
+ "x": 232,
+ "y": 211
+ },
+ "label": "Cua Visual-Only Canvas Fixture",
+ "role": "TitleBar",
+ "screenshot_frame": {
+ "h": 28,
+ "w": 744,
+ "x": 16,
+ "y": 2
+ },
+ "value": "Cua Visual-Only Canvas Fixture"
+ },
+ {
+ "actions": [
+ "invoke",
+ "expand"
+ ],
+ "depth": 3,
+ "element_index": 1,
+ "element_token": "s00000001:1",
+ "enabled": true,
+ "frame": {
+ "h": 22,
+ "w": 22,
+ "x": 216,
+ "y": 216
+ },
+ "label": "System",
+ "parent_index": 0,
+ "role": "MenuItem",
+ "screenshot_frame": {
+ "h": 22,
+ "w": 22,
+ "x": 0,
+ "y": 7
+ }
+ },
+ {
+ "actions": [
+ "invoke"
+ ],
+ "depth": 2,
+ "element_index": 2,
+ "element_token": "s00000001:2",
+ "enabled": true,
+ "frame": {
+ "h": 30,
+ "w": 47,
+ "x": 837,
+ "y": 209
+ },
+ "label": "Minimize",
+ "parent_index": 0,
+ "role": "Button",
+ "screenshot_frame": {
+ "h": 30,
+ "w": 47,
+ "x": 621,
+ "y": 0
+ }
+ },
+ {
+ "actions": [
+ "invoke"
+ ],
+ "depth": 2,
+ "element_index": 3,
+ "element_token": "s00000001:3",
+ "enabled": true,
+ "frame": {
+ "h": 30,
+ "w": 47,
+ "x": 930,
+ "y": 209
+ },
+ "label": "Close",
+ "parent_index": 0,
+ "role": "Button",
+ "screenshot_frame": {
+ "h": 30,
+ "w": 47,
+ "x": 714,
+ "y": 0
+ }
+ }
+ ],
+ "elements_complete": false,
+ "nodes_pending": 0,
+ "nodes_visited": 9,
+ "pid": 10840,
+ "returned_element_count": 4,
+ "screenshot_height": 490,
+ "screenshot_mime_type": "image/png",
+ "screenshot_width": 760,
+ "snapshot_id": "s00000001",
+ "timeout_ms": 1000,
+ "total_element_count": 4,
+ "truncated": false,
+ "walk_elapsed_ms": 0,
+ "window_bounds": {
+ "height": 492,
+ "width": 762,
+ "x": 215,
+ "y": 208
+ },
+ "window_id": 196848,
+ "window_title": "Cua Visual-Only Canvas Fixture"
+}
diff --git a/libs/cua-driver/examples/jev-use/python/capture_window_state.py b/libs/cua-driver/examples/jev-use/python/capture_window_state.py
index ce4d51e1e1..e8ab82ee6e 100644
--- a/libs/cua-driver/examples/jev-use/python/capture_window_state.py
+++ b/libs/cua-driver/examples/jev-use/python/capture_window_state.py
@@ -12,6 +12,7 @@ from __future__ import annotations
import argparse
import asyncio
+import uuid
import json
import os
from pathlib import Path
@@ -42,14 +43,14 @@ def sanitize(payload: dict[str, Any], source: str) -> dict[str, Any]:
return result
-async def capture(pid: int, title: str, source: str) -> dict[str, Any]:
+async def capture(pid: int, title: str, source: str, max_depth: int | None = None) -> dict[str, Any]:
params = StdioServerParameters(
command=os.getenv("CUA_DRIVER_BIN", "cua-driver"), args=["mcp"], env=driver_environment()
)
async with stdio_client(params) as (read, write):
async with ClientSession(read, write) as session:
await session.initialize()
- driver = Driver(session, "jev-native-capture")
+ driver = Driver(session, f"jev-native-capture-{uuid.uuid4().hex[:8]}")
window = await find_window(driver, pid, title)
payload = await driver.call(
"get_window_state",
@@ -58,6 +59,7 @@ async def capture(pid: int, title: str, source: str) -> dict[str, Any]:
"window_id": int(window["window_id"]),
"include_accessibility_tree": True,
"include_screenshot": True,
+ **({"max_depth": max_depth} if max_depth is not None else {}),
},
)
return sanitize(payload, source)
@@ -69,8 +71,9 @@ def main() -> None:
parser.add_argument("--title", required=True, help="exact task window title")
parser.add_argument("--source", required=True, help="fixture provenance, e.g. harness and Driver version")
parser.add_argument("--output", type=Path, required=True)
+ parser.add_argument("--max-depth", type=int, help="the task scope's walk depth, if it sets one")
args = parser.parse_args()
- state = asyncio.run(capture(args.pid, args.title, args.source))
+ state = asyncio.run(capture(args.pid, args.title, args.source, args.max_depth))
args.output.write_text(json.dumps(state, indent=1, sort_keys=True) + "\n", encoding="utf-8")
diff --git a/libs/cua-driver/examples/jev-use/python/native.py b/libs/cua-driver/examples/jev-use/python/native.py
index be06daf588..4c1ef65782 100644
--- a/libs/cua-driver/examples/jev-use/python/native.py
+++ b/libs/cua-driver/examples/jev-use/python/native.py
@@ -20,6 +20,17 @@ Element rules (all must hold):
and the element is not marked ``unlabelled``;
5. ``in_web_content`` is not ``true`` (web content uses the browser source).
+An observation *has no application elements* when every element it reports is
+a window root (``window``, ``application``, ``frame``) or window chrome (the
+Windows title bar and its descendants). On macOS, the application's global menu
+bar and its descendants, and unlabeled, valueless direct children of the window
+root that are buttons (the standard window buttons), are not window content either. Driver
+reports such a tree for a custom-painted surface: X11 property metadata only on
+Linux, only the title bar on Windows, and only the window buttons and menu bar
+on macOS. That view has no accessibility tree to act through, so, like an empty
+tree, it may fall back to visual regions. This check never changes which
+elements become candidates.
+
Stable IDs never use ``element_index``: ``ax:<role_class>:<slug(label)>``,
plus a short hash of the actionable-ancestor path and ordinal when the base
repeats in one observation.
@@ -144,8 +155,9 @@ class NativeObservation:
``complete`` is Driver's ``elements_complete`` claim. ``truncated`` means
the walk ran out of budget (``truncated`` or a ``truncation_reason``).
``tree_empty`` means Driver reported the tree empty (``degraded_reason``
- ``ax_tree_empty``); only then, or for a complete tree, may a view without
- native candidates fall back to visual regions.
+ ``ax_tree_empty``). A view without native candidates may fall back to
+ visual regions only then, for a complete tree, or when the tree has no
+ application elements (see ``has_application_elements``).
"""
pid: int
@@ -203,6 +215,79 @@ class NativeObservation:
)
+WINDOW_ROOT_ROLES = frozenset({"window", "application", "frame"})
+MACOS_MENU_BAR_ROLE = "menubar"
+
+
+def _index_elements(observation: NativeObservation) -> dict[int, Mapping[str, Any]]:
+ return {
+ item["element_index"]: item
+ for item in observation.elements
+ if isinstance(item.get("element_index"), int)
+ }
+
+
+def _in_window_chrome(
+ element: Mapping[str, Any], by_index: Mapping[int, Mapping[str, Any]], platform: Platform
+) -> bool:
+ """Whether an ancestor of ``element`` is a window-chrome container."""
+ seen: set[int] = set()
+ parent = element.get("parent_index")
+ while isinstance(parent, int) and parent in by_index and parent not in seen:
+ seen.add(parent)
+ if is_window_chrome(by_index[parent].get("role"), platform):
+ return True
+ parent = by_index[parent].get("parent_index")
+ return False
+
+
+def _normalized(element: Mapping[str, Any]) -> str:
+ role = element.get("role")
+ return normalized_role(role) if isinstance(role, str) else ""
+
+
+def _in_macos_menu_bar(element: Mapping[str, Any], by_index: Mapping[int, Mapping[str, Any]]) -> bool:
+ seen: set[int] = set()
+ current: Mapping[str, Any] | None = element
+ while current is not None:
+ if _normalized(current) == MACOS_MENU_BAR_ROLE:
+ return True
+ parent = current.get("parent_index")
+ if not isinstance(parent, int) or parent in seen:
+ return False
+ seen.add(parent)
+ current = by_index.get(parent)
+ return False
+
+
+def _is_macos_window_button(element: Mapping[str, Any], by_index: Mapping[int, Mapping[str, Any]]) -> bool:
+ """An unlabeled, valueless button directly under the window root (close, minimize, zoom)."""
+ parent = by_index.get(element.get("parent_index")) if isinstance(element.get("parent_index"), int) else None
+ return (
+ _normalized(element) == "button"
+ and parent is not None
+ and _normalized(parent) in WINDOW_ROOT_ROLES
+ and not _string(element.get("label"))
+ and not _string(element.get("value"))
+ )
+
+
+def has_application_elements(observation: NativeObservation, platform: Platform) -> bool:
+ """Whether any element is window content (see the module docstring)."""
+ by_index = _index_elements(observation)
+ for element in observation.elements:
+ if _normalized(element) in WINDOW_ROOT_ROLES:
+ continue
+ if is_window_chrome(element.get("role"), platform) or _in_window_chrome(element, by_index, platform):
+ continue
+ if platform == "macos" and (
+ _in_macos_menu_bar(element, by_index) or _is_macos_window_button(element, by_index)
+ ):
+ continue
+ return True
+ return False
+
+
@dataclass(frozen=True)
class NativeControl:
"""One eligible native element in one observation.
@@ -240,11 +325,7 @@ def eligible_controls(
redact: Callable[[str], str] = lambda value: value,
) -> NativeControls:
"""Apply the element rules and assign stable IDs, in ``element_index`` order."""
- by_index = {
- item["element_index"]: item
- for item in observation.elements
- if isinstance(item.get("element_index"), int)
- }
+ by_index = _index_elements(observation)
excluded: dict[str, int] = {}
def exclude(reason: ExclusionReason) -> None:
@@ -267,23 +348,13 @@ def eligible_controls(
parent = ancestor.get("parent_index")
return tuple(reversed(path))
- def in_window_chrome(element: Mapping[str, Any]) -> bool:
- seen: set[int] = set()
- parent = element.get("parent_index")
- while isinstance(parent, int) and parent in by_index and parent not in seen:
- seen.add(parent)
- if is_window_chrome(by_index[parent].get("role"), platform):
- return True
- parent = by_index[parent].get("parent_index")
- return False
-
pending: list[tuple[Mapping[str, Any], RoleClass, str, tuple[tuple[str, str], ...]]] = []
for element in observation.elements:
klass = role_class(element.get("role"), platform)
if klass is None:
exclude("unknown_role")
continue
- if in_window_chrome(element):
+ if _in_window_chrome(element, by_index, platform):
exclude("window_chrome")
continue
if element.get("enabled") is False:
diff --git a/libs/cua-driver/examples/jev-use/python/native_tasks.py b/libs/cua-driver/examples/jev-use/python/native_tasks.py
index f26b5e0518..a5a0cbd8df 100644
--- a/libs/cua-driver/examples/jev-use/python/native_tasks.py
+++ b/libs/cua-driver/examples/jev-use/python/native_tasks.py
@@ -12,17 +12,32 @@ AppKit (``CUA_APPKIT_TASK_STATE``), WPF (``CUA_WPF_TASK_STATE``), and GTK3
(``CUA_GTK3_TASK_STATE``), under ``libs/cua-driver/tests/fixtures/apps``.
Their oracle is the harness's own JSON state file, which the app rewrites on
every change; it never depends on Driver output.
+
+``canvas-save`` drives the cross-platform visual-only canvas fixture
+(``tests/fixtures/apps/cross-platform/visual-only-canvas``), a custom-painted
+Tk surface with no accessibility tree. Its only executable candidate is a
+capture-bound visual click, so it proves the OmniParser fallback. The fixture
+publishes its state to a loopback journal; the caller (``verify_native.py``)
+writes each published state to the task's state file unchanged except for the
+schema name.
"""
from __future__ import annotations
import json
-from dataclasses import dataclass, field
+from dataclasses import dataclass, field, replace
from pathlib import Path
from typing import Any, Callable, Literal, Mapping
from choose_action import MAX_ELEMENTS, MAX_HISTORY, REQUEST_SCHEMA_V2
-from native import PARAMETER_NAME_PATTERN, NativeControl, element_state, field_state, slug
+from native import (
+ PARAMETER_NAME_PATTERN,
+ NativeControl,
+ element_state,
+ field_state,
+ has_application_elements,
+ slug,
+)
from native_roles import ACTION_KINDS
from sources import Candidate, NativeAccessibilitySource, TextMethod
from tasks import Outcome, TaskParameter, TaskSources, redact_token
@@ -195,6 +210,9 @@ class NativeTask:
max_steps: int = 6
text_method: TextMethod = "set_value"
visual_targets: tuple[str, ...] = ()
+ # OCR confidence a visual target must reach; the default matches the
+ # browser path. Exact-text uniqueness still applies at any bar.
+ visual_min_confidence: float = 0.8
mock_preferences: tuple[str, ...] = ()
# The oracle is polled after every action, so no candidate is special.
completion_candidate_ids: frozenset[str] = field(default=frozenset(), init=False)
@@ -309,13 +327,24 @@ class NativeTask:
if control is None:
continue
candidate_id = f"visual:{slug(target)}"
- candidate = visual.click(
+ description = (
+ f"Click the unique validated visual region reading {_quoted(target)}; "
+ "no accessibility element covers it."
+ )
+ source = visual
+ if candidate_id in sources.foreground_ids:
+ if not self.allow_foreground:
+ continue
+ candidate_id = f"{candidate_id}:foreground"
+ description += (
+ " Use foreground delivery, which activates the window, because Driver "
+ "refused background delivery for this region."
+ )
+ source = replace(visual, delivery="foreground")
+ candidate = source.click(
control,
candidate_id=candidate_id,
- description=(
- f"Click the unique validated visual region reading {_quoted(target)}; "
- "no accessibility element covers it."
- ),
+ description=description,
)
if candidate is not None:
candidates.append(candidate)
@@ -395,9 +424,12 @@ def visual_fallback_reason(sources: TaskSources, task: NativeTask, native_count:
"""Return why this step may parse visual regions, or ``None``.
Visual regions are consulted only when the task allows ``visual_click`` and
- declares visual targets, and either Driver reported the tree empty, or a
+ declares visual targets, and one of these holds: Driver reported the tree
+ empty; the tree is not truncated but has no application elements (only
+ window roots and window chrome, as for a custom-painted surface); or a
complete tree has no native candidate (or no native control labeled like a
- declared target). A truncated or merely partial tree never qualifies.
+ declared target). A truncated tree, or a partial tree that does contain
+ application elements, never qualifies.
"""
ax = sources.ax
if ax is None or "visual_click" not in task.allowed_actions or not task.visual_targets:
@@ -407,8 +439,14 @@ def visual_fallback_reason(sources: TaskSources, task: NativeTask, native_count:
return None
if observation.tree_empty:
return "tree_empty"
- if not observation.complete or observation.truncated:
+ if observation.truncated:
return None
+ if not observation.complete:
+ # A partial tree qualifies only when it holds nothing but window roots
+ # and window chrome, as for a custom-painted surface.
+ if has_application_elements(observation, ax.platform):
+ return None
+ return "no_application_elements"
if native_count == 0:
return "no_native_candidates"
labels = {control.label.lower() for control in ax.controls}
@@ -555,16 +593,46 @@ def _size_check(state: Mapping[str, Any]) -> Check:
return "pending"
+def _canvas_check(state: Mapping[str, Any]) -> Check:
+ selected = state.get("selected")
+ if selected == slug(CANVAS_TARGET) and state.get("action_count") == 1:
+ return "verified"
+ if selected is not None:
+ return "refuted" # another card, or Save clicked more than once
+ return "pending"
+
+
def harness_task_ids(harness: str) -> tuple[str, ...]:
return tuple(f"{harness}-{kind}" for kind in TASK_KINDS)
+# The cross-platform visual-only canvas has no accessibility tree, so it is not
+# a form harness: it has one task, reached only through the visual fallback.
+CANVAS = HarnessSpec(
+ "canvas", "any", "Cua Visual-Only Canvas Fixture", "cua.visual_canvas_task_state_v1",
+ "CUA_CANVAS_TASK_STATE",
+)
+CANVAS_TASK_ID = "canvas-save"
+CANVAS_TARGET = "Save"
+# A painted surface has no deeper tree. Walking only the window's top level
+# keeps macOS from spending the walk budget in the application menu bar (which
+# would truncate the tree and forbid the fallback). Any content container still
+# appears at depth 1 and blocks the fallback.
+CANVAS_MAX_DEPTH = 1
+# The macOS VM's 1x capture reads the canvas's "Save" label at about 0.75.
+CANVAS_MIN_CONFIDENCE = 0.7
+
APPKIT_TASK_IDS = harness_task_ids("appkit")
-NATIVE_TASK_IDS = tuple(task for name in HARNESSES for task in harness_task_ids(name))
+NATIVE_TASK_IDS = (
+ *(task for name in HARNESSES for task in harness_task_ids(name)),
+ CANVAS_TASK_ID,
+)
def split_task_id(task_id: str) -> tuple[HarnessSpec, str]:
"""Return the harness and task kind of a built-in task ID."""
+ if task_id == CANVAS_TASK_ID:
+ return CANVAS, "save"
harness, _, kind = task_id.partition("-")
if harness not in HARNESSES or kind not in TASK_KINDS:
raise ValueError(f"unknown native task: {task_id}")
@@ -585,6 +653,23 @@ def native_task(
scope = WindowScope(
harness.window_title, bundle_id=harness.bundle_id, process_name=harness.process_name
)
+ if kind == "save":
+ return NativeTask(
+ id=task_id,
+ goal=(
+ f"The window is a painted canvas with {CANVAS_TARGET}, Send, and Cancel cards. "
+ f"Click the {CANVAS_TARGET} card once, then stop."
+ ),
+ scope=replace(scope, max_depth=CANVAS_MAX_DEPTH),
+ allowed_actions=frozenset({"visual_click"}),
+ oracle=oracle,
+ check=_canvas_check,
+ allow_foreground=allow_foreground,
+ max_steps=4,
+ visual_targets=(CANVAS_TARGET,),
+ visual_min_confidence=CANVAS_MIN_CONFIDENCE,
+ mock_preferences=(f"visual:{slug(CANVAS_TARGET)}",),
+ )
if kind == "counter":
return NativeTask(
id=task_id,
diff --git a/libs/cua-driver/examples/jev-use/python/run_native.py b/libs/cua-driver/examples/jev-use/python/run_native.py
index df213676c7..215d0e8a59 100644
--- a/libs/cua-driver/examples/jev-use/python/run_native.py
+++ b/libs/cua-driver/examples/jev-use/python/run_native.py
@@ -170,12 +170,17 @@ async def maybe_visual(
if not capture_bound_click or "parse_visual_regions" not in available_tools:
return sources, {"status": "unavailable", "reason": reason}
observation = sources.ax.observation # type: ignore[union-attr]
+ started = time.perf_counter()
try:
result = await driver.call(
"parse_visual_regions",
{
"capture_id": observation.capture_id,
- "options": {"kinds": ["text", "icon"], "min_confidence": 0.8, "max_regions": 100},
+ "options": {
+ "kinds": ["text", "icon"],
+ "min_confidence": task.visual_min_confidence,
+ "max_regions": 100,
+ },
},
)
visual = parse_visual_regions(
@@ -191,11 +196,14 @@ async def maybe_visual(
return (
TaskSources(
ax=sources.ax,
- visual=VisualRegionSource(visual, "background", capture_bound_click),
+ visual=VisualRegionSource(
+ visual, "background", capture_bound_click, min_confidence=task.visual_min_confidence
+ ),
visual_path=True,
foreground_ids=sources.foreground_ids,
),
- {"status": "ok", "reason": reason, "region_count": len(visual.regions)},
+ {"status": "ok", "reason": reason, "region_count": len(visual.regions),
+ "parse_ms": round((time.perf_counter() - started) * 1000, 2)},
)
diff --git a/libs/cua-driver/examples/jev-use/python/sources.py b/libs/cua-driver/examples/jev-use/python/sources.py
index 426e382a3b..9960b76ea2 100644
--- a/libs/cua-driver/examples/jev-use/python/sources.py
+++ b/libs/cua-driver/examples/jev-use/python/sources.py
@@ -152,7 +152,8 @@ class VisualRegionSource:
"""Controls from validated OmniParser regions, clicked through their capture.
Regions carry no role, so ``find`` matches the visible text or icon label
- (ASCII case-insensitive) of regions at or above ``MIN_CONFIDENCE`` and
+ (ASCII case-insensitive) of regions at or above ``min_confidence`` (default
+ ``MIN_CONFIDENCE``; a task may opt into a lower bar) and
returns a control only when exactly one region matches. ``click`` is offered
only when Driver advertises a capture-bound ``click`` (``capture_bound``); it
targets the region's screenshot-pixel center with the exact ``capture_id``
@@ -162,6 +163,7 @@ class VisualRegionSource:
observation: VisualObservation
delivery: VisualDelivery = "background"
capture_bound: bool = False
+ min_confidence: float = 0.8
kind: ClassVar[SourceKind] = "visual"
MIN_CONFIDENCE: ClassVar[float] = 0.8
@@ -170,7 +172,7 @@ class VisualRegionSource:
matches = [
region
for region in self.observation.regions
- if region.confidence >= self.MIN_CONFIDENCE
+ if region.confidence >= self.min_confidence
and _ascii_lower(region.text or region.label or "") == wanted
]
if len(matches) != 1:
diff --git a/libs/cua-driver/examples/jev-use/python/tests/test_native_canvas.py b/libs/cua-driver/examples/jev-use/python/tests/test_native_canvas.py
new file mode 100644
index 0000000000..da7d4cfcb2
--- /dev/null
+++ b/libs/cua-driver/examples/jev-use/python/tests/test_native_canvas.py
@@ -0,0 +1,237 @@
+"""The visual fallback for views without application accessibility elements.
+
+The recorded fixtures are real ``get_window_state`` results for the
+cross-platform visual-only canvas (a custom-painted Tk surface): on Linux X11,
+Driver recovers only window metadata; on Windows, UIA exposes only the title
+bar; on macOS, AX exposes only the two unlabeled window buttons and the
+application menu bar. None holds an application element, so the canvas task may parse
+visual regions from the same capture, and its only executable candidate is a
+capture-bound visual click.
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from dataclasses import replace
+import tempfile
+import unittest
+import urllib.request
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+from choose_action import validate_request
+from core import VisualObservation, VisualRegion
+from jev_adapter import choose_mock_for_task
+from native import NativeObservation, has_application_elements
+from native_tasks import (
+ CANVAS,
+ CANVAS_TASK_ID,
+ HARNESSES,
+ NATIVE_TASK_IDS,
+ native_choice_request,
+ native_task,
+ split_task_id,
+ visual_fallback_reason,
+)
+from sources import NativeAccessibilitySource, VisualRegionSource
+from tasks import TaskSources
+
+FIXTURES = Path(__file__).resolve().parents[2] / "fixtures/native"
+CANVAS_FIXTURES = {"linux": "canvas-linux", "windows": "canvas-windows", "macos": "canvas-macos"}
+
+
+def observation(name: str) -> NativeObservation:
+ payload = json.loads((FIXTURES / f"{name}-window-state-initial-v1.json").read_text(encoding="utf-8"))
+ return NativeObservation.from_window_state(
+ payload, expected_pid=payload["pid"], expected_window_id=payload["window_id"]
+ )
+
+
+def canvas_sources(platform: str, *, visual: bool = True, foreground_ids=frozenset()) -> TaskSources:
+ observed = observation(CANVAS_FIXTURES[platform])
+ ax = NativeAccessibilitySource.from_observation(observed, platform)
+ if not visual:
+ return TaskSources(ax=ax, foreground_ids=foreground_ids)
+ regions = tuple(
+ VisualRegion(f"r{index}", "text", text, None, 0.95, False, x, 250, 80, 30)
+ for index, (text, x) in enumerate((("Save", 130), ("Send", 350), ("Cancel", 570)))
+ )
+ regions += (VisualRegion("r9", "text", "CHOOSE A SIGNAL", None, 0.95, False, 40, 30, 300, 30),)
+ parsed = VisualObservation(
+ observed.capture_id or "", "ref", 762, 492, observed.pid, observed.window_id,
+ 0.0, 0.0, 1.0, 1.0, regions,
+ )
+ return TaskSources(
+ ax=ax,
+ visual=VisualRegionSource(parsed, "background", True),
+ visual_path=True,
+ foreground_ids=foreground_ids,
+ )
+
+
+class ApplicationElementsTest(unittest.TestCase):
+ def test_recorded_canvas_trees_have_no_application_elements(self) -> None:
+ for platform, name in CANVAS_FIXTURES.items():
+ with self.subTest(platform=platform):
+ observed = observation(name)
+ self.assertFalse(observed.complete)
+ self.assertFalse(observed.truncated)
+ self.assertFalse(observed.tree_empty) # Driver does not report ax_tree_empty
+ self.assertFalse(has_application_elements(observed, platform))
+ ax = NativeAccessibilitySource.from_observation(observed, platform)
+ self.assertEqual(ax.controls, ())
+
+ def test_recorded_harness_trees_have_application_elements(self) -> None:
+ for harness in ("appkit", "wpf", "gtk3"):
+ with self.subTest(harness=harness):
+ self.assertTrue(has_application_elements(observation(harness), HARNESSES[harness].platform))
+
+ def test_windows_title_bar_counts_as_chrome_only_on_windows(self) -> None:
+ # Linux has no chrome rule, so the same buttons count as application elements.
+ self.assertTrue(has_application_elements(observation("canvas-windows"), "linux"))
+
+ def test_macos_menu_bar_and_window_buttons_are_not_content(self) -> None:
+ payload = json.loads((FIXTURES / "canvas-macos-window-state-initial-v1.json").read_text(encoding="utf-8"))
+ # The menu bar and unlabeled window buttons count as content on other platforms.
+ self.assertTrue(has_application_elements(observation("canvas-macos"), "linux"))
+ cases = {
+ "labeled window button": {"element_index": 90, "parent_index": 0, "role": "AXButton", "label": "Save"},
+ "unlabeled nested button": {"element_index": 91, "parent_index": 92, "role": "AXButton"},
+ "group": {"element_index": 92, "parent_index": 0, "role": "AXGroup"},
+ }
+ for name, extra in cases.items():
+ with self.subTest(name=name):
+ elements = payload["elements"] + [extra]
+ if name == "unlabeled nested button":
+ elements.append(cases["group"])
+ observed = NativeObservation.from_window_state(
+ {**payload, "elements": elements},
+ expected_pid=payload["pid"], expected_window_id=payload["window_id"],
+ )
+ self.assertTrue(has_application_elements(observed, "macos"))
+
+
+class FallbackRuleTest(unittest.TestCase):
+ def test_canvas_trees_fall_back_to_visual_regions(self) -> None:
+ task = native_task(CANVAS_TASK_ID, Path("/tmp/none.json"))
+ for platform in CANVAS_FIXTURES:
+ with self.subTest(platform=platform):
+ sources = canvas_sources(platform, visual=False)
+ self.assertEqual(visual_fallback_reason(sources, task, 0), "no_application_elements")
+
+ def test_partial_tree_with_application_elements_never_falls_back(self) -> None:
+ task = native_task(CANVAS_TASK_ID, Path("/tmp/none.json"))
+ observed = observation("appkit") # macOS never claims completeness
+ ax = NativeAccessibilitySource.from_observation(observed, "macos")
+ self.assertIsNone(visual_fallback_reason(TaskSources(ax=ax), task, 0))
+
+ def test_truncated_or_uncaptured_trees_never_fall_back(self) -> None:
+ task = native_task(CANVAS_TASK_ID, Path("/tmp/none.json"))
+ payload = json.loads((FIXTURES / "canvas-linux-window-state-initial-v1.json").read_text(encoding="utf-8"))
+ for change in ({"truncated": True}, {"capture_id": None}):
+ with self.subTest(change=change):
+ observed = NativeObservation.from_window_state(
+ {**payload, **change}, expected_pid=payload["pid"], expected_window_id=payload["window_id"]
+ )
+ ax = NativeAccessibilitySource.from_observation(observed, "linux")
+ self.assertIsNone(visual_fallback_reason(TaskSources(ax=ax), task, 0))
+
+ def test_form_tasks_never_parse_visual_regions(self) -> None:
+ task = native_task("gtk3-counter", Path("/tmp/none.json"))
+ self.assertIsNone(visual_fallback_reason(canvas_sources("linux", visual=False), task, 0))
+
+
+class CanvasTaskTest(unittest.TestCase):
+ def test_scope_and_confidence(self) -> None:
+ task = native_task(CANVAS_TASK_ID, Path("/tmp/none.json"))
+ self.assertEqual(task.scope.window_state_arguments(), {"max_depth": 1})
+ self.assertEqual(task.visual_min_confidence, 0.7)
+ form = native_task("appkit-counter", Path("/tmp/none.json"))
+ self.assertEqual(form.scope.window_state_arguments(), {})
+ self.assertEqual(form.visual_min_confidence, 0.8)
+ sources = canvas_sources("macos")
+ low = [replace(region, confidence=0.75) for region in sources.visual.observation.regions]
+ observed = replace(sources.visual.observation, regions=tuple(low))
+ self.assertIsNone(VisualRegionSource(observed, "background", True).find("button", "Save"))
+ self.assertIsNotNone(VisualRegionSource(observed, "background", True, 0.7).find("button", "Save"))
+
+ def test_registry(self) -> None:
+ self.assertIn(CANVAS_TASK_ID, NATIVE_TASK_IDS)
+ self.assertEqual(split_task_id(CANVAS_TASK_ID), (CANVAS, "save"))
+ task = native_task(CANVAS_TASK_ID, Path("/tmp/none.json"), pid=7)
+ self.assertEqual(task.scope.window_title, "Cua Visual-Only Canvas Fixture")
+ self.assertEqual(task.oracle.schema, "cua.visual_canvas_task_state_v1")
+ self.assertEqual(task.allowed_action_kinds, frozenset({"click"}))
+
+ def test_only_the_save_region_is_executable(self) -> None:
+ task = native_task(CANVAS_TASK_ID, Path("/tmp/none.json"))
+ for platform in CANVAS_FIXTURES:
+ with self.subTest(platform=platform):
+ sources = canvas_sources(platform)
+ step = task.plan(sources)
+ self.assertEqual([c.id for c in step.candidates], ["visual:save", "reobserve", "abstain"])
+ save = step.candidates[0]
+ self.assertEqual(save.source, "visual")
+ self.assertEqual(save.tool, "click")
+ self.assertEqual(save.arguments["delivery_mode"], "background")
+ self.assertEqual(save.arguments["capture_id"], sources.ax.observation.capture_id)
+ self.assertEqual((save.arguments["x"], save.arguments["y"]), (170.0, 265.0))
+ request = native_choice_request(task, sources, step, [])
+ validate_request(request)
+ self.assertEqual([c["source"] for c in request["candidates"][:1]], ["visual"])
+ self.assertEqual(request["elements"], [])
+ choice, _, _ = choose_mock_for_task(task, sources, step.candidates, [])
+ self.assertEqual(choice, "visual:save")
+
+ def test_background_refusal_offers_an_explicit_foreground_variant(self) -> None:
+ refused = frozenset({"visual:save"})
+ allowed = native_task(CANVAS_TASK_ID, Path("/tmp/none.json"), allow_foreground=True)
+ step = allowed.plan(canvas_sources("linux", foreground_ids=refused))
+ self.assertEqual([c.id for c in step.candidates], ["visual:save:foreground", "reobserve", "abstain"])
+ self.assertEqual(step.candidates[0].arguments["delivery_mode"], "foreground")
+ self.assertEqual(
+ choose_mock_for_task(allowed, canvas_sources("linux", foreground_ids=refused), step.candidates, [])[0],
+ "visual:save:foreground",
+ )
+ denied = native_task(CANVAS_TASK_ID, Path("/tmp/none.json"))
+ self.assertEqual(
+ [c.id for c in denied.plan(canvas_sources("linux", foreground_ids=refused)).candidates],
+ ["reobserve", "abstain"],
+ )
+
+ def test_oracle(self) -> None:
+ task = native_task(CANVAS_TASK_ID, Path("/tmp/none.json"))
+ self.assertEqual(task.check({"selected": None, "action_count": 0}), "pending")
+ self.assertEqual(task.check({"selected": "save", "action_count": 1}), "verified")
+ self.assertEqual(task.check({"selected": "save", "action_count": 2}), "refuted")
+ self.assertEqual(task.check({"selected": "send", "action_count": 1}), "refuted")
+
+
+class JournalTest(unittest.TestCase):
+ def test_journal_writes_each_published_state_with_the_task_schema(self) -> None:
+ import verify_native
+
+ with tempfile.TemporaryDirectory() as work:
+ state = Path(work) / "state.json"
+ url, close = verify_native.start_journal(verify_native.HARNESSES["canvas"], state)
+ try:
+ for published in ({"pid": 5, "selected": None, "action_count": 0},
+ {"pid": 5, "selected": "save", "action_count": 1}):
+ request = urllib.request.Request(
+ url, data=json.dumps(published).encode(), method="POST",
+ headers={"Content-Type": "application/json"},
+ )
+ with urllib.request.urlopen(request, timeout=5) as response:
+ self.assertEqual(response.status, 204)
+ written = json.loads(state.read_text(encoding="utf-8"))
+ self.assertEqual(written, {**published, "schema": "cua.visual_canvas_task_state_v1"})
+ finally:
+ close()
+ self.assertTrue(url.startswith("http://127.0.0.1:"))
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/libs/cua-driver/examples/jev-use/python/tests/test_native_platforms.py b/libs/cua-driver/examples/jev-use/python/tests/test_native_platforms.py
index bd232a0ee6..83e6a2f59b 100644
--- a/libs/cua-driver/examples/jev-use/python/tests/test_native_platforms.py
+++ b/libs/cua-driver/examples/jev-use/python/tests/test_native_platforms.py
@@ -50,9 +50,14 @@ class HarnessRegistryTest(unittest.TestCase):
def test_every_harness_has_the_same_three_tasks(self) -> None:
self.assertEqual(
NATIVE_TASK_IDS,
- tuple(f"{h}-{k}" for h in ("appkit", "wpf", "gtk3") for k in ("counter", "save-note", "choose-size")),
+ (
+ *(f"{h}-{k}" for h in ("appkit", "wpf", "gtk3") for k in ("counter", "save-note", "choose-size")),
+ "canvas-save",
+ ),
)
for task_id in NATIVE_TASK_IDS:
+ if task_id == "canvas-save":
+ continue # the visual-only canvas is not a form harness (see test_native_canvas.py)
harness, kind = split_task_id(task_id)
task = native_task(task_id, Path("/tmp/none.json"), pid=7)
self.assertEqual(task.scope.window_title, harness.window_title)
diff --git a/libs/cua-driver/examples/jev-use/typescript/native.ts b/libs/cua-driver/examples/jev-use/typescript/native.ts
index c98bbd1fa6..e19511d7f9 100644
--- a/libs/cua-driver/examples/jev-use/typescript/native.ts
+++ b/libs/cua-driver/examples/jev-use/typescript/native.ts
@@ -141,6 +141,68 @@ export function isTreeEmpty(observation: NativeObservation): boolean {
return Boolean(observation.degradedReason?.startsWith('ax_tree_empty'));
}
+export const WINDOW_ROOT_ROLES: ReadonlySet<string> = new Set(['window', 'application', 'frame']);
+const MACOS_MENU_BAR_ROLE = 'menubar';
+const normalizedOf = (item: WindowElement | undefined) =>
+ item && typeof item.role === 'string' ? normalizedRole(item.role) : '';
+
+function inMacosMenuBar(item: WindowElement, byIndex: Map<number, WindowElement>): boolean {
+ const seen = new Set<number>();
+ let current: WindowElement | undefined = item;
+ while (current) {
+ if (normalizedOf(current) === MACOS_MENU_BAR_ROLE) return true;
+ const parent = current.parent_index;
+ if (!Number.isInteger(parent) || seen.has(parent as number)) return false;
+ seen.add(parent as number);
+ current = byIndex.get(parent as number);
+ }
+ return false;
+}
+
+/** An unlabeled, valueless button directly under the window root (close, minimize, zoom). */
+function isMacosWindowButton(item: WindowElement, byIndex: Map<number, WindowElement>): boolean {
+ const parent = Number.isInteger(item.parent_index) ? byIndex.get(item.parent_index as number) : undefined;
+ return (
+ normalizedOf(item) === 'button' &&
+ Boolean(parent) &&
+ WINDOW_ROOT_ROLES.has(normalizedOf(parent)) &&
+ !str(item.label) &&
+ !str(item.value)
+ );
+}
+
+function indexElements(observation: NativeObservation): Map<number, WindowElement> {
+ const byIndex = new Map<number, WindowElement>();
+ for (const item of observation.elements) {
+ if (Number.isInteger(item.element_index)) byIndex.set(item.element_index as number, item);
+ }
+ return byIndex;
+}
+
+/** Whether an ancestor of `item` is a window-chrome container. */
+function inWindowChrome(item: WindowElement, byIndex: Map<number, WindowElement>, platform: Platform): boolean {
+ const seen = new Set<number>();
+ let parent = item.parent_index;
+ while (Number.isInteger(parent) && byIndex.has(parent as number) && !seen.has(parent as number)) {
+ seen.add(parent as number);
+ const ancestor = byIndex.get(parent as number)!;
+ if (isWindowChrome(ancestor.role, platform)) return true;
+ parent = ancestor.parent_index;
+ }
+ return false;
+}
+
+/** Whether any element is window content (mirrors native.py has_application_elements). */
+export function hasApplicationElements(observation: NativeObservation, platform: Platform): boolean {
+ const byIndex = indexElements(observation);
+ return observation.elements.some((item) => {
+ if (WINDOW_ROOT_ROLES.has(normalizedOf(item))) return false;
+ if (isWindowChrome(item.role, platform) || inWindowChrome(item, byIndex, platform)) return false;
+ if (platform === 'macos' && (inMacosMenuBar(item, byIndex) || isMacosWindowButton(item, byIndex))) return false;
+ return true;
+ });
+}
+
export function parseWindowState(
payload: Readonly<Record<string, unknown>>,
expectedPid: number,
@@ -195,10 +257,7 @@ export function eligibleControls(
platform: Platform,
redact: (value: string) => string = (value) => value
): NativeControls {
- const byIndex = new Map<number, WindowElement>();
- for (const item of observation.elements) {
- if (Number.isInteger(item.element_index)) byIndex.set(item.element_index as number, item);
- }
+ const byIndex = indexElements(observation);
const excluded: Record<string, number> = {};
const exclude = (reason: string) => {
excluded[reason] = (excluded[reason] ?? 0) + 1;
@@ -218,18 +277,6 @@ export function eligibleControls(
return path.reverse();
};
- const inWindowChrome = (item: WindowElement): boolean => {
- const seen = new Set<number>();
- let parent = item.parent_index;
- while (Number.isInteger(parent) && byIndex.has(parent as number) && !seen.has(parent as number)) {
- seen.add(parent as number);
- const ancestor = byIndex.get(parent as number)!;
- if (isWindowChrome(ancestor.role, platform)) return true;
- parent = ancestor.parent_index;
- }
- return false;
- };
-
const pending: { item: WindowElement; klass: RoleClass; label: string; path: [string, string][] }[] = [];
for (const item of observation.elements) {
const klass = roleClass(item.role, platform);
@@ -237,7 +284,7 @@ export function eligibleControls(
exclude('unknown_role');
continue;
}
- if (inWindowChrome(item)) {
+ if (inWindowChrome(item, byIndex, platform)) {
exclude('window_chrome');
continue;
}
diff --git a/libs/cua-driver/examples/jev-use/typescript/native_canvas.test.ts b/libs/cua-driver/examples/jev-use/typescript/native_canvas.test.ts
new file mode 100644
index 0000000000..11ea5597bf
--- /dev/null
+++ b/libs/cua-driver/examples/jev-use/typescript/native_canvas.test.ts
@@ -0,0 +1,151 @@
+// The visual fallback for views without application accessibility elements.
+// Mirrors python/tests/test_native_canvas.py; the fixtures are real
+// get_window_state results for the visual-only canvas on Linux and Windows.
+import assert from 'node:assert/strict';
+import { readFileSync } from 'node:fs';
+import test from 'node:test';
+
+import { validateRequest } from './choose_action.js';
+import type { VisualObservation, VisualRegion } from './core.js';
+import { chooseMockForTask } from './jev_adapter.js';
+import { hasApplicationElements, parseWindowState } from './native.js';
+import type { Platform } from './native_roles.js';
+import {
+ CANVAS,
+ CANVAS_TASK_ID,
+ HARNESSES,
+ NATIVE_TASK_IDS,
+ nativeChoiceRequest,
+ nativeTask,
+ splitTaskId,
+ visualFallbackReason,
+} from './native_tasks.js';
+import { NativeAccessibilitySource, VisualRegionSource } from './sources.js';
+import type { TaskSources } from './tasks.js';
+
+const CANVAS_FIXTURES = { linux: 'canvas-linux', windows: 'canvas-windows', macos: 'canvas-macos' } as const;
+
+function windowState(name: string): any {
+ const url = new URL(`../fixtures/native/${name}-window-state-initial-v1.json`, import.meta.url);
+ return JSON.parse(readFileSync(url, 'utf8'));
+}
+const observe = (payload: any) => parseWindowState(payload, payload.pid, payload.window_id);
+
+function canvasSources(
+ platform: keyof typeof CANVAS_FIXTURES,
+ { visual = true, foregroundIds = new Set<string>() }: { visual?: boolean; foregroundIds?: Set<string> } = {}
+): TaskSources {
+ const observed = observe(windowState(CANVAS_FIXTURES[platform]));
+ const ax = NativeAccessibilitySource.fromObservation(observed, platform);
+ if (!visual) return { ax, visualPath: false, foregroundIds };
+ const cards: [string, number][] = [['Save', 130], ['Send', 350], ['Cancel', 570]];
+ const regions: VisualRegion[] = [
+ ...cards.map(([text, x], index) => ({
+ id: `r${index}`, kind: 'text' as const, text, confidence: 0.95, interactive: false, x, y: 250, width: 80, height: 30,
+ })),
+ { id: 'r9', kind: 'text', text: 'CHOOSE A SIGNAL', confidence: 0.95, interactive: false, x: 40, y: 30, width: 300, height: 30 },
+ ];
+ const parsed: VisualObservation = {
+ captureId: observed.captureId ?? '', screenshotReference: 'ref', screenshotWidth: 762, screenshotHeight: 492,
+ pid: observed.pid, windowId: observed.windowId, actionOriginX: 0, actionOriginY: 0,
+ actionUnitsPerPixelX: 1, actionUnitsPerPixelY: 1, regions,
+ };
+ return { ax, visual: new VisualRegionSource(parsed, 'background', true), visualPath: true, foregroundIds };
+}
+
+test('recorded canvas trees have no application elements', () => {
+ for (const [platform, name] of Object.entries(CANVAS_FIXTURES) as [Platform, string][]) {
+ const observed = observe(windowState(name));
+ assert.equal(observed.complete, false, platform);
+ assert.equal(observed.truncated, false, platform);
+ assert.equal(hasApplicationElements(observed, platform), false, platform);
+ assert.deepEqual(NativeAccessibilitySource.fromObservation(observed, platform).controls, [], platform);
+ }
+ for (const harness of ['appkit', 'wpf', 'gtk3']) {
+ assert.equal(hasApplicationElements(observe(windowState(harness)), HARNESSES[harness].platform), true, harness);
+ }
+ // Linux has no chrome rule, so the same title-bar buttons count as application elements.
+ assert.equal(hasApplicationElements(observe(windowState('canvas-windows')), 'linux'), true);
+ // The macOS menu bar and unlabeled window buttons are content everywhere else.
+ assert.equal(hasApplicationElements(observe(windowState('canvas-macos')), 'linux'), true);
+ const mac = windowState('canvas-macos');
+ for (const extra of [
+ [{ element_index: 90, parent_index: 0, role: 'AXButton', label: 'Save' }],
+ [{ element_index: 92, parent_index: 0, role: 'AXGroup' }],
+ [{ element_index: 92, parent_index: 0, role: 'AXGroup' }, { element_index: 91, parent_index: 92, role: 'AXButton' }],
+ ]) {
+ assert.equal(hasApplicationElements(observe({ ...mac, elements: [...mac.elements, ...extra] }), 'macos'), true);
+ }
+});
+
+test('canvas trees fall back to visual regions; partial, truncated, and form cases do not', () => {
+ const task = nativeTask(CANVAS_TASK_ID, '/tmp/none.json');
+ for (const platform of ['linux', 'windows', 'macos'] as const) {
+ assert.equal(visualFallbackReason(canvasSources(platform, { visual: false }), task, 0), 'no_application_elements');
+ }
+ const appkit = NativeAccessibilitySource.fromObservation(observe(windowState('appkit')), 'macos');
+ assert.equal(visualFallbackReason({ ax: appkit, visualPath: false }, task, 0), undefined);
+ const payload = windowState('canvas-linux');
+ for (const change of [{ truncated: true }, { capture_id: null }]) {
+ const ax = NativeAccessibilitySource.fromObservation(observe({ ...payload, ...change }), 'linux');
+ assert.equal(visualFallbackReason({ ax, visualPath: false }, task, 0), undefined, JSON.stringify(change));
+ }
+ const form = nativeTask('gtk3-counter', '/tmp/none.json');
+ assert.equal(visualFallbackReason(canvasSources('linux', { visual: false }), form, 0), undefined);
+});
+
+test('the canvas task is registered and bound to its state file', () => {
+ assert.ok(NATIVE_TASK_IDS.includes(CANVAS_TASK_ID));
+ assert.deepEqual(splitTaskId(CANVAS_TASK_ID), [CANVAS, 'save']);
+ const task = nativeTask(CANVAS_TASK_ID, '/tmp/none.json', { pid: 7 });
+ assert.equal(task.scope.windowTitle, 'Cua Visual-Only Canvas Fixture');
+ assert.equal(task.oracle.schema, 'cua.visual_canvas_task_state_v1');
+ assert.equal(task.oracle.expectedPid, 7);
+ assert.deepEqual(task.scope.maxDepth, 1);
+ assert.equal(task.visualMinConfidence, 0.7);
+ assert.equal(nativeTask('appkit-counter', '/tmp/none.json').visualMinConfidence, 0.8);
+ const sources = canvasSources('macos');
+ const low = { ...sources.visual!.observation, regions: sources.visual!.observation.regions.map((r) => ({ ...r, confidence: 0.75 })) };
+ assert.equal(new VisualRegionSource(low, 'background', true).find('button', 'Save'), undefined);
+ assert.ok(new VisualRegionSource(low, 'background', true, 0.7).find('button', 'Save'));
+});
+
+test('only the Save region is executable, through a capture-bound click', () => {
+ const task = nativeTask(CANVAS_TASK_ID, '/tmp/none.json');
+ for (const platform of ['linux', 'windows', 'macos'] as const) {
+ const sources = canvasSources(platform);
+ const step = task.plan(sources);
+ assert.deepEqual(step.candidates.map((c) => c.id), ['visual:save', 'reobserve', 'abstain'], platform);
+ const save = step.candidates[0];
+ assert.equal(save.source, 'visual');
+ assert.equal(save.tool, 'click');
+ assert.equal(save.arguments.delivery_mode, 'background');
+ assert.equal(save.arguments.capture_id, sources.ax!.observation.captureId);
+ assert.deepEqual([save.arguments.x, save.arguments.y], [170, 265]);
+ const request = nativeChoiceRequest(task, sources, step, []);
+ validateRequest(request);
+ assert.deepEqual((request.elements as unknown[]) ?? [], []);
+ assert.equal(chooseMockForTask(task, sources, step.candidates, []).choice, 'visual:save');
+ }
+});
+
+test('a background refusal offers an explicit foreground variant only when allowed', () => {
+ const refused = new Set(['visual:save']);
+ const allowed = nativeTask(CANVAS_TASK_ID, '/tmp/none.json', { allowForeground: true });
+ const step = allowed.plan(canvasSources('linux', { foregroundIds: refused }));
+ assert.deepEqual(step.candidates.map((c) => c.id), ['visual:save:foreground', 'reobserve', 'abstain']);
+ assert.equal(step.candidates[0].arguments.delivery_mode, 'foreground');
+ const denied = nativeTask(CANVAS_TASK_ID, '/tmp/none.json');
+ assert.deepEqual(
+ denied.plan(canvasSources('linux', { foregroundIds: refused })).candidates.map((c) => c.id),
+ ['reobserve', 'abstain']
+ );
+});
+
+test('the canvas oracle', () => {
+ const task = nativeTask(CANVAS_TASK_ID, '/tmp/none.json');
+ assert.equal(task.check({ selected: null, action_count: 0 }), 'pending');
+ assert.equal(task.check({ selected: 'save', action_count: 1 }), 'verified');
+ assert.equal(task.check({ selected: 'save', action_count: 2 }), 'refuted');
+ assert.equal(task.check({ selected: 'send', action_count: 1 }), 'refuted');
+});
diff --git a/libs/cua-driver/examples/jev-use/typescript/native_platforms.test.ts b/libs/cua-driver/examples/jev-use/typescript/native_platforms.test.ts
index 9e3997d96b..a0efafb16b 100644
--- a/libs/cua-driver/examples/jev-use/typescript/native_platforms.test.ts
+++ b/libs/cua-driver/examples/jev-use/typescript/native_platforms.test.ts
@@ -42,9 +42,13 @@ function sources(task: NativeTask, harness: string, name: string): TaskSources {
test('every harness has the same three tasks', () => {
assert.deepEqual(
NATIVE_TASK_IDS,
- ['appkit', 'wpf', 'gtk3'].flatMap((h) => ['counter', 'save-note', 'choose-size'].map((k) => `${h}-${k}`))
+ [
+ ...['appkit', 'wpf', 'gtk3'].flatMap((h) => ['counter', 'save-note', 'choose-size'].map((k) => `${h}-${k}`)),
+ 'canvas-save',
+ ]
);
for (const taskId of NATIVE_TASK_IDS) {
+ if (taskId === 'canvas-save') continue; // not a form harness (see native_canvas.test.ts)
const [harness, kind] = splitTaskId(taskId);
const task = nativeTask(taskId, '/tmp/none.json', { pid: 7 });
assert.equal(task.scope.windowTitle, harness.windowTitle);
diff --git a/libs/cua-driver/examples/jev-use/typescript/native_tasks.ts b/libs/cua-driver/examples/jev-use/typescript/native_tasks.ts
index f3c47b24b3..c37be9c633 100644
--- a/libs/cua-driver/examples/jev-use/typescript/native_tasks.ts
+++ b/libs/cua-driver/examples/jev-use/typescript/native_tasks.ts
@@ -10,9 +10,22 @@
import { readFile } from 'node:fs/promises';
import { MAX_ELEMENTS, REQUEST_SCHEMA_V2 } from './choose_action.js';
-import { PARAMETER_NAME_PATTERN, elementState, fieldState, slug, type NativeControl } from './native.js';
+import {
+ PARAMETER_NAME_PATTERN,
+ elementState,
+ fieldState,
+ hasApplicationElements,
+ slug,
+ type NativeControl,
+} from './native.js';
import { ACTION_KINDS, type ActionKind } from './native_roles.js';
-import { immutableCandidate, type Candidate, type NativeAccessibilitySource, type TextMethod } from './sources.js';
+import {
+ immutableCandidate,
+ VisualRegionSource,
+ type Candidate,
+ type NativeAccessibilitySource,
+ type TextMethod,
+} from './sources.js';
import { redactToken, type HistoryEntry, type Outcome, type Task, type TaskParameter, type TaskSources } from './tasks.js';
export const MAX_EXECUTABLE_CANDIDATES = 24;
@@ -170,6 +183,7 @@ export type NativeTaskSpec = Readonly<{
maxSteps?: number;
textMethod?: TextMethod;
visualTargets?: readonly string[];
+ visualMinConfidence?: number;
mockPreferences?: readonly string[];
}>;
@@ -186,6 +200,8 @@ export class NativeTask implements Task {
readonly maxSteps: number;
readonly textMethod: TextMethod;
readonly visualTargets: readonly string[];
+ /** OCR confidence a visual target must reach; exact-text uniqueness still applies. */
+ readonly visualMinConfidence: number;
readonly mockPreferences: readonly string[];
/** The oracle is polled after every action, so no candidate is special. */
readonly completionCandidateIds: ReadonlySet<string> = new Set();
@@ -203,6 +219,7 @@ export class NativeTask implements Task {
this.maxSteps = spec.maxSteps ?? 6;
this.textMethod = spec.textMethod ?? 'set_value';
this.visualTargets = spec.visualTargets ?? [];
+ this.visualMinConfidence = spec.visualMinConfidence ?? 0.8;
this.mockPreferences = spec.mockPreferences ?? [];
const unknown = [...this.allowedActions].filter((action) => !ACTION_KINDS.has(action));
if (unknown.length) throw new Error(`unknown action kinds: ${unknown.sort().join(', ')}`);
@@ -296,12 +313,17 @@ export class NativeTask implements Task {
for (const target of this.visualTargets) {
const control = visual.find('button', target);
if (!control) continue;
- const id = `visual:${slug(target)}`;
- const candidate = visual.click(
- control,
- id,
- `Click the unique validated visual region reading ${quoted(target)}; no accessibility element covers it.`
- );
+ let id = `visual:${slug(target)}`;
+ let description = `Click the unique validated visual region reading ${quoted(target)}; no accessibility element covers it.`;
+ let source = visual;
+ if (sources.foregroundIds?.has(id)) {
+ if (!this.allowForeground) continue;
+ id = `${id}:foreground`;
+ description +=
+ ' Use foreground delivery, which activates the window, because Driver refused background delivery for this region.';
+ source = new VisualRegionSource(visual.observation, 'foreground', visual.captureBound, visual.minConfidence);
+ }
+ const candidate = source.click(control, id, description);
if (candidate) {
candidates.push(candidate);
outcomes[id] = `clicked the visual region ${quoted(target)}`;
@@ -385,7 +407,12 @@ export function visualFallbackReason(
const observation = ax.observation;
if (!observation.captureId) return undefined;
if (observation.degradedReason?.startsWith('ax_tree_empty')) return 'tree_empty';
- if (!observation.complete || observation.truncated) return undefined;
+ if (observation.truncated) return undefined;
+ // A partial tree qualifies only when it holds nothing but window roots and
+ // window chrome, as for a custom-painted surface.
+ if (!observation.complete) {
+ return hasApplicationElements(observation, ax.platform) ? undefined : 'no_application_elements';
+ }
if (nativeCount === 0) return 'no_native_candidates';
const labels = new Set(ax.controls.map((control) => control.label.toLowerCase()));
if (task.visualTargets.some((target) => !labels.has(target.toLowerCase()))) return 'target_without_element';
@@ -480,11 +507,31 @@ export const COUNTER_TARGET = 3;
export const DEFAULT_NOTE_TEXT = 'jev-use native note';
export const TASK_KINDS = ['counter', 'save-note', 'choose-size'] as const;
+// The cross-platform visual-only canvas has no accessibility tree, so it is not
+// a form harness: it has one task, reached only through the visual fallback.
+export type TaskHarness = Omit<HarnessSpec, 'platform'> & Readonly<{ platform: HarnessSpec['platform'] | 'any' }>;
+
+export const CANVAS: TaskHarness = {
+ name: 'canvas',
+ platform: 'any',
+ windowTitle: 'Cua Visual-Only Canvas Fixture',
+ stateSchema: 'cua.visual_canvas_task_state_v1',
+ stateEnv: 'CUA_CANVAS_TASK_STATE',
+};
+export const CANVAS_TASK_ID = 'canvas-save';
+export const CANVAS_TARGET = 'Save';
+// A painted surface has no deeper tree; see python/native_tasks.py CANVAS_MAX_DEPTH.
+export const CANVAS_MAX_DEPTH = 1;
+export const CANVAS_MIN_CONFIDENCE = 0.7;
+
export const harnessTaskIds = (harness: string): string[] => TASK_KINDS.map((kind) => `${harness}-${kind}`);
export const APPKIT_TASK_IDS = harnessTaskIds('appkit');
-export const NATIVE_TASK_IDS = Object.keys(HARNESSES).flatMap(harnessTaskIds);
+export const NATIVE_TASK_IDS = [...Object.keys(HARNESSES).flatMap(harnessTaskIds), CANVAS_TASK_ID];
-export function splitTaskId(taskId: string): [HarnessSpec, (typeof TASK_KINDS)[number]] {
+type TaskKind = (typeof TASK_KINDS)[number] | 'save';
+
+export function splitTaskId(taskId: string): [TaskHarness, TaskKind] {
+ if (taskId === CANVAS_TASK_ID) return [CANVAS, 'save'];
const index = taskId.indexOf('-');
const harness = index > 0 ? HARNESSES[taskId.slice(0, index)] : undefined;
const kind = taskId.slice(index + 1) as (typeof TASK_KINDS)[number];
@@ -503,6 +550,26 @@ export function nativeTask(taskId: string, statePath: string, options: HarnessTa
...(harness.processName ? { processName: harness.processName } : {}),
};
const allowForeground = options.allowForeground ?? false;
+ if (kind === 'save') {
+ return new NativeTask({
+ id: taskId,
+ goal:
+ `The window is a painted canvas with ${CANVAS_TARGET}, Send, and Cancel cards. ` +
+ `Click the ${CANVAS_TARGET} card once, then stop.`,
+ scope: { ...scope, maxDepth: CANVAS_MAX_DEPTH },
+ allowedActions: new Set<ActionKind>(['visual_click']),
+ oracle,
+ check: (state) => {
+ if (state.selected === slug(CANVAS_TARGET) && state.action_count === 1) return 'verified';
+ return state.selected != null ? 'refuted' : 'pending';
+ },
+ allowForeground,
+ maxSteps: 4,
+ visualTargets: [CANVAS_TARGET],
+ visualMinConfidence: CANVAS_MIN_CONFIDENCE,
+ mockPreferences: [`visual:${slug(CANVAS_TARGET)}`],
+ });
+ }
if (kind === 'counter') {
return new NativeTask({
id: taskId,
diff --git a/libs/cua-driver/examples/jev-use/typescript/run_native.ts b/libs/cua-driver/examples/jev-use/typescript/run_native.ts
index d88128522e..fabf718f87 100644
--- a/libs/cua-driver/examples/jev-use/typescript/run_native.ts
+++ b/libs/cua-driver/examples/jev-use/typescript/run_native.ts
@@ -184,15 +184,25 @@ async function maybeVisual(
return { sources, record: { status: 'unavailable', reason } };
}
const observation = sources.ax!.observation;
+ const started = performance.now();
try {
const result = await driver.call('parse_visual_regions', {
capture_id: observation.captureId,
- options: { kinds: ['text', 'icon'], min_confidence: 0.8, max_regions: 100 },
+ options: { kinds: ['text', 'icon'], min_confidence: task.visualMinConfidence, max_regions: 100 },
});
const visual = parseVisualRegions(result, observation.captureId!, observation.pid, observation.windowId);
return {
- sources: { ...sources, visual: new VisualRegionSource(visual, 'background', captureBoundClick), visualPath: true },
- record: { status: 'ok', reason, region_count: visual.regions.length },
+ sources: {
+ ...sources,
+ visual: new VisualRegionSource(visual, 'background', captureBoundClick, task.visualMinConfidence),
+ visualPath: true,
+ },
+ record: {
+ status: 'ok',
+ reason,
+ region_count: visual.regions.length,
+ parse_ms: Math.round((performance.now() - started) * 100) / 100,
+ },
};
} catch (error) {
if (error instanceof DriverToolError) {
diff --git a/libs/cua-driver/examples/jev-use/typescript/sources.ts b/libs/cua-driver/examples/jev-use/typescript/sources.ts
index 9083c98b61..00e6dda5ff 100644
--- a/libs/cua-driver/examples/jev-use/typescript/sources.ts
+++ b/libs/cua-driver/examples/jev-use/typescript/sources.ts
@@ -147,7 +147,8 @@ export class BrowserSemanticSource implements CandidateSource {
/**
* Controls from validated OmniParser regions, clicked through their capture.
* Regions carry no role, so find matches the visible text or icon label (ASCII
- * case-insensitive) of regions at or above MIN_CONFIDENCE and returns a
+ * case-insensitive) of regions at or above minConfidence (default
+ * MIN_CONFIDENCE; a task may opt into a lower bar) and returns a
* control only when exactly one region matches. click is offered only when
* Driver advertises a capture-bound click (captureBound); it targets the
* region's screenshot-pixel center with the exact capture_id and the source's
@@ -160,14 +161,15 @@ export class VisualRegionSource implements CandidateSource {
constructor(
readonly observation: VisualObservation,
readonly delivery: VisualDelivery = 'background',
- readonly captureBound = false
+ readonly captureBound = false,
+ readonly minConfidence: number = VisualRegionSource.MIN_CONFIDENCE
) {}
find(role: string, name: string): Control<VisualRegion> | undefined {
const wanted = asciiLower(name);
const matches = this.observation.regions.filter(
(region) =>
- region.confidence >= VisualRegionSource.MIN_CONFIDENCE &&
+ region.confidence >= this.minConfidence &&
asciiLower(region.text ?? region.label ?? '') === wanted
);
if (matches.length !== 1) return undefined;
diff --git a/libs/cua-driver/examples/jev-use/verify_native.py b/libs/cua-driver/examples/jev-use/verify_native.py
index 366541afb0..a467a68d3b 100644
--- a/libs/cua-driver/examples/jev-use/verify_native.py
+++ b/libs/cua-driver/examples/jev-use/verify_native.py
@@ -1,7 +1,11 @@
"""Verify the native jev-use tasks against a repository harness (RFC #4268).
``--harness`` selects the AppKit (macOS), WPF (Windows), or GTK3 (Linux)
-harness. For every task and language this script launches a fresh harness in
+harness, or the cross-platform visual-only ``canvas`` fixture, which has no
+accessibility tree and proves the OmniParser fallback (it needs the
+cua-perception extension). The canvas publishes its state to a loopback
+journal that this script serves; each published state is written to the task
+state file unchanged except for the schema name. For every task and language this script launches a fresh harness in
task mode with its own state file (``CUA_<HARNESS>_TASK_STATE``), runs the
native runner against that exact process, then reads the harness state file
itself and checks it against the task's expected end state. The runners' own
@@ -24,9 +28,11 @@ import os
import subprocess
import sys
import tempfile
+import threading
import time
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
-from typing import NamedTuple
+from typing import Callable, NamedTuple
BASE = Path(__file__).resolve().parent
REPO = BASE.parents[3]
@@ -40,6 +46,9 @@ class Harness(NamedTuple):
window_title: str
state_schema: str
state_env: str
+ journal: bool = False # the app posts its state to a loopback journal
+ max_depth: int | None = None # the task scope's walk depth, for --capture-dir
+
# Mirrors python/native_tasks.py HARNESSES; kept independent so the verifier
@@ -57,19 +66,28 @@ HARNESSES = {
TEST_APPS / "harness-gtk3/CuaTestHarness.Gtk3", "",
"CuaTestHarness GTK3 Tasks", "cua.gtk3_task_state_v1", "CUA_GTK3_TASK_STATE",
),
+ "canvas": Harness(
+ REPO / "libs/cua-driver/tests/fixtures/apps/cross-platform/visual-only-canvas/main.py", "",
+ "Cua Visual-Only Canvas Fixture", "cua.visual_canvas_task_state_v1", "CUA_CANVAS_TASK_STATE",
+ journal=True, max_depth=1,
+ ),
}
+FORM_KINDS = ("choose-size", "counter", "save-note")
+HARNESS_KINDS = {name: (("save",) if name == "canvas" else FORM_KINDS) for name in HARNESSES}
# The independent end-state check for each task kind, read from the app's own file.
EXPECTED = {
"counter": lambda state: state.get("counter") == 3,
"save-note": lambda state: state.get("note_saved") == NOTE_TEXT,
"choose-size": lambda state: state.get("size") == "large" and state.get("agreed") is True,
+ "save": lambda state: state.get("selected") == "save" and state.get("action_count") == 1,
}
# The executable candidates the deterministic mock provider must have acted on.
MOCK_ACTIONS = {
"counter": ["ax:button:increment"] * 3,
"save-note": ["ax:text_input:note:set:note", "ax:button:save-note"],
"choose-size": ["ax:radio:large", "ax:checkbox:i-agree"],
+ "save": ["visual:save"],
}
@@ -79,14 +97,72 @@ def runner_command(language: str) -> list[str]:
return ["node", "--import", "tsx", "typescript/run_native.ts"]
-def launch_harness(harness: Harness, app: Path, state: Path) -> subprocess.Popen:
+def canvas_python() -> str:
+ """A Tk-capable interpreter whose process ID is the fixture's own.
+
+ A virtual environment's launcher on Windows starts the real interpreter as
+ a child, so the fixture's published ``pid`` would not match the launched
+ process; the base interpreter avoids that. ``CUA_CANVAS_PYTHON`` overrides.
+ """
+ return os.environ.get("CUA_CANVAS_PYTHON") or getattr(sys, "_base_executable", "") or sys.executable
+
+
+def start_journal(harness: Harness, state: Path) -> tuple[str, Callable[[], None]]:
+ """Serve a loopback journal that writes each published app state to ``state``."""
+
+ class Journal(BaseHTTPRequestHandler):
+ def log_message(self, *_args: object) -> None:
+ pass
+
+ def do_POST(self) -> None: # noqa: N802 - http.server API
+ length = int(self.headers.get("Content-Length") or 0)
+ try:
+ published = json.loads(self.rfile.read(min(length, 65536)))
+ if not isinstance(published, dict):
+ raise ValueError("state must be an object")
+ except ValueError:
+ self.send_response(400)
+ self.end_headers()
+ return
+ temporary = state.with_suffix(".tmp")
+ temporary.write_text(json.dumps({**published, "schema": harness.state_schema}), encoding="utf-8")
+ os.replace(temporary, state)
+ self.send_response(204)
+ self.end_headers()
+
+ server = ThreadingHTTPServer(("127.0.0.1", 0), Journal)
+ threading.Thread(target=server.serve_forever, daemon=True).start()
+
+ def close() -> None:
+ server.shutdown()
+ server.server_close()
+
+ return f"http://127.0.0.1:{server.server_address[1]}/journal", close
+
+
+def launch_harness(
+ harness: Harness, app: Path, state: Path
+) -> tuple[subprocess.Popen, Callable[[], None]]:
executable = app / harness.executable if harness.executable else app
if not executable.is_file():
raise SystemExit(f"harness is missing at {executable}; run the fixture build")
+ close: Callable[[], None] = lambda: None
+ if harness.journal:
+ url, close = start_journal(harness, state)
+ command = [canvas_python(), str(executable), "--journal-url", url]
+ else:
+ command = [str(executable)]
env = {**os.environ, harness.state_env: str(state)}
- process = subprocess.Popen(
- [str(executable)], env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL
- )
+ process = subprocess.Popen(command, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+ try:
+ wait_for_initial_state(process, state)
+ except BaseException:
+ close()
+ raise
+ return process, close
+
+
+def wait_for_initial_state(process: subprocess.Popen, state: Path) -> None:
for _ in range(100):
if state.exists():
try:
@@ -94,7 +170,7 @@ def launch_harness(harness: Harness, app: Path, state: Path) -> subprocess.Popen
except json.JSONDecodeError:
initial = {}
if initial.get("pid") == process.pid:
- return process
+ return
if process.poll() is not None:
raise RuntimeError("harness exited during launch")
time.sleep(0.1)
@@ -134,7 +210,8 @@ def capture(harness_name: str, pid: int, output: Path, label: str) -> None:
completed = subprocess.run(
[sys.executable, "python/capture_window_state.py", "--pid", str(pid),
"--title", harness.window_title, "--output", str(output),
- "--source", f"CuaTestHarness {harness_name} task mode ({label}), observed by get_window_state"],
+ "--source", f"CuaTestHarness {harness_name} task mode ({label}), observed by get_window_state",
+ *(["--max-depth", str(harness.max_depth)] if harness.max_depth is not None else [])],
cwd=BASE, check=False, timeout=120,
)
if completed.returncode != 0:
@@ -149,7 +226,7 @@ def verify(
task = f"{harness_name}-{kind}"
state = work / f"{language}-{provider}-{task}-state.json"
log = output / f"{language}-{provider}-{task}.jsonl"
- process = launch_harness(harness, app, state)
+ process, close_journal = launch_harness(harness, app, state)
try:
initial = read_state(harness, state, process.pid)
if capture_dir is not None:
@@ -158,6 +235,10 @@ def verify(
"--task", task, "--provider", provider, "--pid", str(process.pid),
"--state-file", str(state), "--note-text", NOTE_TEXT, "--log", str(log),
]
+ if harness.journal:
+ # A visual click may be refused in the background on X11; the canvas
+ # task then offers an explicit foreground variant.
+ command.append("--allow-foreground")
completed = subprocess.run(command, cwd=BASE, check=False, timeout=300)
observed = read_state(harness, state, process.pid)
if capture_dir is not None:
@@ -165,6 +246,7 @@ def verify(
finally:
process.kill()
process.wait(timeout=10)
+ close_journal()
events = [json.loads(line) for line in log.read_text().splitlines()] if log.exists() else []
result = {
"language": language,
@@ -172,8 +254,8 @@ def verify(
"task": task,
"exit_code": completed.returncode,
"runner_outcome": events[-1].get("outcome") if events else None,
- "initial": {key: initial.get(key) for key in ("counter", "agreed", "size", "note_saved")},
- "observed": {key: observed.get(key) for key in ("counter", "agreed", "size", "note_saved")},
+ "initial": {key: initial.get(key) for key in ("counter", "agreed", "size", "note_saved", "selected", "action_count")},
+ "observed": {key: observed.get(key) for key in ("counter", "agreed", "size", "note_saved", "selected", "action_count")},
**summarize(events),
}
result["verified"] = bool(EXPECTED[kind](observed))
@@ -184,7 +266,8 @@ def verify(
raise RuntimeError(f"runner did not report verified: {json.dumps(result)}")
if not result["verified"]:
raise RuntimeError(f"independent harness state does not match: {json.dumps(result)}")
- if provider == "mock" and result["actions"] != MOCK_ACTIONS[kind]:
+ delivered = [action.removesuffix(":foreground") for action in result["actions"]]
+ if provider == "mock" and delivered != MOCK_ACTIONS[kind]:
raise RuntimeError(f"mock provider took unexpected actions: {json.dumps(result)}")
return result
@@ -207,6 +290,9 @@ def main() -> None:
if not sys.stdin.isatty():
raise SystemExit("Human prerequisite: provision TYPESAFE_API_KEY before a live run")
os.environ["TYPESAFE_API_KEY"] = getpass.getpass("TypeSafe API key: ").strip()
+ unsupported = sorted(set(args.task or ()) - set(HARNESS_KINDS[args.harness]))
+ if unsupported:
+ parser.error(f"--harness {args.harness} has no task kind {', '.join(unsupported)}")
if args.s1:
sys.path.insert(0, str(BASE / "python"))
from s1_service import S1ServiceError, s1_service_url
@@ -234,7 +320,7 @@ def main() -> None:
for language in ["python", "typescript"] if args.typescript else ["python"]:
providers = ["mock"] + (["live"] if args.live else []) + (["s1"] if args.s1 else [])
for provider in providers:
- for kind in args.task or sorted(EXPECTED):
+ for kind in args.task or HARNESS_KINDS[args.harness]:
result = verify(
language, provider, args.harness, kind, app, output, Path(work), capture_dir
)
diff --git a/rfcs/4268-jev-use-native-candidates.md b/rfcs/4268-jev-use-native-candidates.md
index 48466391fc..7242a88682 100644
--- a/rfcs/4268-jev-use-native-candidates.md
+++ b/rfcs/4268-jev-use-native-candidates.md
@@ -802,6 +802,17 @@ Implementation:
visual fallback is used only when Driver reports `ax_tree_empty` or a
complete tree lacks the target. On macOS this means only an empty tree
reaches OmniParser today.
+ - Phase 3 finding: no platform reports `ax_tree_empty` for a custom-painted
+ view. For the visual-only canvas, Linux X11 returns only window metadata
+ (`x11_property_fallback_partial`), and Windows UIA returns only the title
+ bar. Both are partial trees, so OmniParser was unreachable. The fallback
+ therefore also applies to a non-truncated tree, re-observed once with the
+ larger budget, whose elements are all window roots or window chrome
+ (`no_application_elements`). On macOS, the application's global menu bar
+ and the unlabeled window buttons do not count as window content. The
+ canvas task scopes its walk to `max_depth: 1`, because the menu bar
+ otherwise exhausts the walk budget and truncates the tree. A partial tree
+ with any application element still never falls back.
- A radio option that is already selected is not offered, and a `set_text`
candidate is offered only while the field differs from its parameter.
- `close_unsaved` matches its phrases whether or not the window reports