Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cua-driver): add native accessibility candidates to jev-use (#4288)
Francesco Bonacci6a70aeffe5e7
diff --git a/.github/scripts/tests/test_ci_jev_use_workflow.py b/.github/scripts/tests/test_ci_jev_use_workflow.pyindex d331484f53..c577389488 100644--- a/.github/scripts/tests/test_ci_jev_use_workflow.py+++ b/.github/scripts/tests/test_ci_jev_use_workflow.py@@ -110,3 +110,26 @@ def test_macos_seeds_tcc_only_for_the_released_app_identity() -> None:assert "seed-tcc-guest.sh" in macosassert "--app /Applications/CuaDriver.app --expected-client com.trycua.driver\n" in macosassert "install-local.sh" not in macos+++def test_native_appkit_job_uses_released_driver_and_source_harness() -> None:+ """RFC #4268 Phase 1: native tasks on the AppKit harness with the mock provider."""+ jobs = load()["jobs"]+ assert jobs["native-appkit-macos"]["runs-on"].startswith("macos-")+ native = job_text("native-appkit-macos")+ assert "https://cua.ai/driver/install.sh" in native+ assert "cargo build" not in native+ assert 'MIN_RELEASED_DRIVER_VERSION: "0.30.1"' in native+ assert "tests/fixtures/build/macos.sh --only appkit" in native+ assert "--app /Applications/CuaDriver.app --expected-client com.trycua.driver" in native+ assert "verify_native.py --typescript" in native+ assert "--live" not in native+ # The audit reads the harness-owned oracle result and the v2 contract.+ assert 'check["verified"] is True' in native+ assert "cua.jev_choice_request_v2" in native+++def test_native_harness_changes_trigger_the_workflow() -> None:+ triggers = load()[True]+ for event in ("pull_request", "push"):+ assert "libs/cua-driver/tests/fixtures/apps/macos/appkit/**" in triggers[event]["paths"]diff --git a/.github/workflows/ci-jev-use.yml b/.github/workflows/ci-jev-use.ymlindex 30bce99ab2..f779455d56 100644--- a/.github/workflows/ci-jev-use.yml+++ b/.github/workflows/ci-jev-use.yml@@ -4,6 +4,8 @@ on:pull_request:paths:- "libs/cua-driver/examples/jev-use/**"+ - "libs/cua-driver/tests/fixtures/apps/macos/appkit/**"+ - "libs/cua-driver/tests/fixtures/build/macos.sh"- "libs/cua-driver/docs/jev-use-validation.md"- "docs/content/docs/how-to-guides/driver/jev-use.mdx"- "docs/content/docs/how-to-guides/driver/meta.json"@@ -16,6 +18,8 @@ on:branches: [main]paths:- "libs/cua-driver/examples/jev-use/**"+ - "libs/cua-driver/tests/fixtures/apps/macos/appkit/**"+ - "libs/cua-driver/tests/fixtures/build/macos.sh"- "libs/cua-driver/docs/jev-use-validation.md"- "docs/content/docs/how-to-guides/driver/jev-use.mdx"- "docs/content/docs/how-to-guides/driver/meta.json"@@ -62,6 +66,10 @@ jobs:run: uv run --frozen python verify_choice_cli.py- name: Verify closed-candidate decision CLI in mock moderun: uv run --frozen python verify_decision_cli.py --model mock+ - name: Verify both CLIs with the native cua.jev_choice_request_v2 fixture+ run: |+ uv run --frozen python verify_choice_cli.py --request-version v2+ uv run --frozen python verify_decision_cli.py --model mock --fixture nativeunit-typescript:name: TypeScript unit and typecheck@@ -409,6 +417,133 @@ jobs:if-no-files-found: errorretention-days: 14+ # RFC #4268 Phase 1: native accessibility candidates on the repository's+ # AppKit harness, built from this checkout, driven by the released Driver+ # (canonical installer, same TCC seeding as above) with the deterministic+ # mock provider. verify_native.py launches a fresh harness per run and reads+ # the harness's own state file as the independent oracle.+ native-appkit-macos:+ name: macOS native AppKit mock E2E+ runs-on: macos-26+ timeout-minutes: 45+ env:+ CUA_DRIVER_BIN: /Applications/CuaDriver.app/Contents/MacOS/cua-driver+ MIN_RELEASED_DRIVER_VERSION: "0.30.1"+ steps:+ - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4+ with:+ ref: ${{ github.event.pull_request.head.sha || github.sha }}+ persist-credentials: false+ - uses: actions/setup-python@v5+ with:+ python-version: "3.12"+ - uses: astral-sh/setup-uv@v6+ with:+ version: "0.8.15"+ enable-cache: true+ cache-dependency-glob: libs/cua-driver/examples/jev-use/uv.lock+ - uses: actions/setup-node@v4+ with:+ node-version: "22"+ cache: npm+ cache-dependency-path: libs/cua-driver/examples/jev-use/package-lock.json+ - name: Install the released Driver with the canonical installer+ working-directory: ${{ github.workspace }}+ env:+ CUA_DRIVER_NO_MODIFY_PATH: "1"+ run: |+ /bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)"+ version_output="$("$CUA_DRIVER_BIN" --version)"+ echo "$version_output"+ resolved="$(grep -Eo '[0-9]+\.[0-9]+\.[0-9]+' <<< "$version_output" | head -n 1)"+ test -n "$resolved"+ lowest="$(printf '%s\n%s\n' "$MIN_RELEASED_DRIVER_VERSION" "$resolved" | sort -V | head -n 1)"+ if [ "$lowest" != "$MIN_RELEASED_DRIVER_VERSION" ]; then+ echo "::error::canonical installer resolved Cua Driver $resolved, older than $MIN_RELEASED_DRIVER_VERSION"+ exit 1+ fi+ echo "Released Cua Driver under test: $resolved"+ codesign --verify --strict /Applications/CuaDriver.app+ - name: Build the AppKit harness from source+ working-directory: ${{ github.workspace }}+ run: |+ bash libs/cua-driver/tests/fixtures/build/macos.sh --only appkit+ test -x libs/cua-driver/rust/test-apps/harness-appkit/CuaTestHarness.AppKit.app/Contents/MacOS/CuaTestHarness.AppKit+ - name: Install locked example dependencies+ run: |+ uv sync --frozen+ npm ci --ignore-scripts+ - name: Grant the released Driver its TCC permissions+ working-directory: ${{ github.workspace }}+ run: |+ # Same seeding as the released-Driver macOS job: only the released,+ # Developer ID-signed Driver receives Accessibility and Screen+ # Recording. The harness itself needs no grant.+ "$CUA_DRIVER_BIN" stop >/dev/null 2>&1 || true+ bash libs/cua-driver/tests/runners/macos-lume/seed-tcc-guest.sh \+ --app /Applications/CuaDriver.app --expected-client com.trycua.driver+ open -n -g -a CuaDriver --args serve+ for _ in $(seq 1 60); do+ if "$CUA_DRIVER_BIN" permissions status --json > "$RUNNER_TEMP/permissions.json" 2>/dev/null \+ && jq -e '.accessibility == true and .screen_recording == true' "$RUNNER_TEMP/permissions.json" >/dev/null; then+ break+ fi+ sleep 1+ done+ cat "$RUNNER_TEMP/permissions.json"+ jq -e '.accessibility == true and .screen_recording == true' "$RUNNER_TEMP/permissions.json" >/dev/null+ "$CUA_DRIVER_BIN" status+ - name: Run the native tasks with both mock agents+ run: uv run --frozen python verify_native.py --typescript --output-dir "$RUNNER_TEMP/jev-use-native-proof"+ - name: Audit redacted native evidence+ run: |+ uv run --frozen python - "$RUNNER_TEMP/jev-use-native-proof" <<'PY'+ import json, pathlib, sys+ root = pathlib.Path(sys.argv[1])+ summary = json.loads((root / "summary.json").read_text())+ assert summary["complete"] is True+ assert summary["live_requested"] is False+ assert summary["harness"] == "appkit"+ tasks = {"appkit-counter", "appkit-save-note", "appkit-choose-size"}+ assert {(c["language"], c["task"]) for c in summary["checks"]} == {+ (language, task) for language in ("python", "typescript") for task in tasks+ }+ for check in summary["checks"]:+ # verify_native.py read the harness's own state file; the+ # runners' outcome events are not the oracle.+ assert check["verified"] is True and check["runner_outcome"] == "verified", check+ assert check["provider"] == "mock"+ assert check["schemas"] == ["cua.jev_choice_request_v2"], check+ assert check["sources"] == ["ax"], check+ assert check["delivery_modes"] in (["background"], ["background", "none"]), check+ assert check["visual_statuses"] == ["skipped"], check+ counter = [c for c in summary["checks"] if c["task"] == "appkit-counter"]+ # Reset (destructive) and Exit (close_unsaved) never reach the chooser.+ assert all(c["risk_excluded"] == ["close_unsaved", "destructive"] for c in counter)+ assert all(c["observed"]["counter"] == 3 for c in counter)+ note = [c for c in summary["checks"] if c["task"] == "appkit-save-note"]+ assert all(c["tools"] == ["click", "set_value"] for c in note)+ forbidden = ("typesafe_api_key", "authorization", "screenshot_png_b64", "image_base64",+ "element_token", "jev-use native note")+ for path in root.iterdir():+ if path.name == "summary.json" or path.suffix == ".jsonl":+ text = path.read_text().lower()+ assert not any(value in text for value in forbidden), (path, [v for v in forbidden if v in text])+ PY+ - name: Stop the Driver daemon+ if: always()+ run: '"$CUA_DRIVER_BIN" stop || true'+ - name: Upload redacted native evidence+ if: always()+ uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08 # v4+ with:+ name: jev-use-macos-native-appkit-mock+ path: |+ ${{ runner.temp }}/jev-use-native-proof/summary.json+ ${{ runner.temp }}/jev-use-native-proof/*.jsonl+ if-no-files-found: error+ retention-days: 14+released-driver-windows:name: Windows released Driver mock E2Eruns-on: windows-latestdiff --git a/libs/cua-driver/examples/jev-use/README.md b/libs/cua-driver/examples/jev-use/README.mdindex f85f5cfb76..477a83dca0 100644--- a/libs/cua-driver/examples/jev-use/README.md+++ b/libs/cua-driver/examples/jev-use/README.md@@ -67,13 +67,44 @@ regions and acts through a capture-bound `click`. A task spec(`python/tasks.py`, `typescript/tasks.ts`) holds the goal, the parameters withthe secret token marked for redaction, the step budget, the allowed Driveraction kinds, the `form` state summary, the candidate IDs and descriptions, and-the success oracle. The fixture above is the one built-in task,-`FixtureFormTask`, and its oracle is the fixture's `/state` endpoint. A native-accessibility source for desktop applications is planned in-[RFC #4268](https://github.com/trycua/cua/issues/4268) and is not implemented.-The model request and response are unchanged: the golden files-`fixtures/jev-provider-request-golden-*-v1.json` pin the exact provider-requests and candidate tables for the page-structure and visual fixtures.+the success oracle. The browser fixture above is `FixtureFormTask`, and its+oracle is the fixture's `/state` endpoint. Browser requests are unchanged: the+golden files `fixtures/jev-provider-request-golden-*-v1.json` pin the exact+provider requests and candidate tables for the page-structure and visual+fixtures, and browser tasks keep sending `cua.jev_choice_request_v1`.++### Native desktop tasks++`NativeAccessibilitySource` ([RFC #4268](../../../../rfcs/4268-jev-use-native-candidates.md))+builds candidates from the native elements of one `get_window_state` call that+returns the tree and the screenshot together, so element tokens and the+`capture_id` describe the same moment. `native_roles.py` / `native_roles.ts`+map raw macOS AX, Windows UIA, and Linux AT-SPI roles to a closed set of role+classes (`button`, `toggle`, `checkbox`, `radio`, `popup`, `menu_item`, `link`,+`text_input`), keyed by the same normalization as Driver's `verify_state`.+Only enabled, on-screen, labeled, native (not `in_web_content`) elements+become candidates; a label equal to the element's value counts as unlabeled.+Candidate IDs come from the role class, label, and actionable-ancestor path,+never `element_index`. At most 24 action candidates are offered, plus+`reobserve` and `abstain`. Labels that suggest deleting, sending, purchasing,+or closing are excluded unless the task allows that risk. Text comes only from+task parameters. Actions are element-bound `click`, `set_value`, or+`type_text` with background delivery first; a stale token leads to a fresh+observation. Native steps send `cua.jev_choice_request_v2`, which adds a+per-candidate `source` and compact value-free `elements`.++Three tasks drive the repository's AppKit harness, whose opt-in+`CUA_APPKIT_TASK_STATE` file is the independent oracle: `appkit-counter` (set+the counter to 3), `appkit-save-note` (set the Note field and save), and+`appkit-choose-size` (select Large and check I agree). On macOS with Cua+Driver 0.30.1 or later:++```bash+bash ../../tests/fixtures/build/macos.sh --only appkit+uv run --frozen python verify_native.py --typescript --output-dir /tmp/jev-native-proof+```++Windows (UIA) and Linux (AT-SPI) tasks are the next RFC phase.The MCP connection stays open across the entire loop. This preserves theexplicit named Cua Driver session and avoids rebuilding tool state for everydiff --git a/libs/cua-driver/examples/jev-use/fixtures/jev-choice-request-v2.json b/libs/cua-driver/examples/jev-use/fixtures/jev-choice-request-v2.jsonnew file mode 100644index 0000000000..573b7a6882--- /dev/null+++ b/libs/cua-driver/examples/jev-use/fixtures/jev-choice-request-v2.json@@ -0,0 +1,28 @@+{+ "schema": "cua.jev_choice_request_v2",+ "goal": "Set the counter to 3 by pressing Increment.",+ "capture_id": "capture-native-1",+ "snapshot_id": "s0000002a",+ "regions": [],+ "elements": [+ { "role_class": "button", "label": "Increment", "state": "enabled" },+ { "role_class": "checkbox", "label": "I agree", "state": "unchecked" },+ { "role_class": "text_input", "label": "Note", "state": "empty" }+ ],+ "history": [{ "selected_id": "ax:button:increment", "outcome": "pressed the button \"Increment\"" }],+ "candidates": [+ {+ "id": "ax:button:increment",+ "source": "ax",+ "description": "Press the button labeled \"Increment\"."+ },+ {+ "id": "reobserve",+ "description": "Take no action and obtain a fresh observation."+ },+ {+ "id": "abstain",+ "description": "Stop without acting if no supplied action is safe."+ }+ ]+}diff --git a/libs/cua-driver/examples/jev-use/fixtures/native/appkit-window-state-after-actions-v1.json b/libs/cua-driver/examples/jev-use/fixtures/native/appkit-window-state-after-actions-v1.jsonnew file mode 100644index 0000000000..db183f43fd--- /dev/null+++ b/libs/cua-driver/examples/jev-use/fixtures/native/appkit-window-state-after-actions-v1.json@@ -0,0 +1,800 @@+{+ "_fixture": {+ "source": "CuaTestHarness.AppKit with CUA_APPKIT_TASK_STATE set, observed by cua-driver 0.28.2 get_window_state (macOS 26)",+ "sanitized": "Screenshot bytes and tree_markdown removed. The system Apple menu's machine-specific items (recent items, account names) were removed; two generic items remain. capture_id was added because Cua Driver 0.30 and later return it with the screenshot."+ },+ "pid": 20535,+ "window_id": 71364,+ "app_name": "CuaTestHarness.AppKit",+ "window_title": "CuaTestHarness AppKit",+ "snapshot_id": "s00000087",+ "element_count": 103,+ "total_element_count": 103,+ "returned_element_count": 103,+ "elements_complete": false,+ "screenshot_width": 700,+ "screenshot_height": 892,+ "screenshot_scale": 1.0,+ "window_bounds": {+ "height": 892.0,+ "width": 700.0,+ "x": 610.0,+ "y": 48.0+ },+ "capture_id": "cap-appkit-after",+ "elements": [+ {+ "actions": [+ "AXRaise"+ ],+ "depth": 0,+ "element_index": 0,+ "element_token": "s00000087:0",+ "frame": {+ "h": 892.0,+ "w": 700.0,+ "x": 610.0,+ "y": 48.0+ },+ "label": "CuaTestHarness AppKit",+ "role": "AXWindow"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 1,+ "element_token": "s00000087:1",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 87.0,+ "x": 629.0,+ "y": 115.0+ },+ "label": "Increment",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 2,+ "element_token": "s00000087:2",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 61.0,+ "x": 726.0,+ "y": 115.0+ },+ "label": "Reset",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXShowMenu",+ "AXConfirm"+ ],+ "depth": 1,+ "element_index": 3,+ "element_token": "s00000087:3",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 242.0,+ "x": 629.0,+ "y": 219.0+ },+ "label": "Type here…",+ "parent_index": 0,+ "role": "AXTextField",+ "value": "Type here…"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 4,+ "element_token": "s00000087:4",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 221.0,+ "x": 629.0,+ "y": 275.0+ },+ "label": "Click target (left / right / double)",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXIncrement",+ "AXDecrement"+ ],+ "depth": 1,+ "element_index": 5,+ "element_token": "s00000087:5",+ "enabled": true,+ "frame": {+ "h": 18.0,+ "w": 322.0,+ "x": 629.0,+ "y": 331.0+ },+ "label": "sld-value",+ "max": 100.0,+ "min": 0.0,+ "parent_index": 0,+ "role": "AXSlider",+ "value": "0"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 6,+ "element_token": "s00000087:6",+ "enabled": true,+ "frame": {+ "h": 18.0,+ "w": 66.0,+ "x": 629.0,+ "y": 406.0+ },+ "label": "I agree",+ "parent_index": 0,+ "role": "AXCheckBox",+ "selected": true,+ "value": "1"+ },+ {+ "actions": [+ "AXShowMenu"+ ],+ "depth": 4,+ "element_index": 7,+ "element_token": "s00000087:7",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 138.0,+ "x": 822.0,+ "y": 381.0+ },+ "label": "alpha",+ "parent_index": 0,+ "role": "AXStaticText",+ "value": "alpha"+ },+ {+ "actions": [+ "AXShowMenu"+ ],+ "depth": 4,+ "element_index": 8,+ "element_token": "s00000087:8",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 138.0,+ "x": 822.0,+ "y": 405.0+ },+ "label": "beta",+ "parent_index": 0,+ "role": "AXStaticText",+ "value": "beta"+ },+ {+ "actions": [+ "AXShowMenu"+ ],+ "depth": 4,+ "element_index": 9,+ "element_token": "s00000087:9",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 138.0,+ "x": 822.0,+ "y": 429.0+ },+ "label": "gamma",+ "parent_index": 0,+ "role": "AXStaticText",+ "value": "gamma"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 2,+ "element_index": 10,+ "element_token": "s00000087:10",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 2,+ "element_index": 11,+ "element_token": "s00000087:11",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 2,+ "element_index": 12,+ "element_token": "s00000087:12",+ "frame": {+ "h": 5.0,+ "w": 6.0,+ "x": 954.0,+ "y": 440.0+ },+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 2,+ "element_index": 13,+ "element_token": "s00000087:13",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXShowMenu",+ "AXPress"+ ],+ "depth": 1,+ "element_index": 14,+ "element_token": "s00000087:14",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 198.0,+ "x": 629.0,+ "y": 480.0+ },+ "label": "Right-click for context menu",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXShowMenu"+ ],+ "depth": 1,+ "element_index": 15,+ "element_token": "s00000087:15",+ "frame": {+ "h": 448.0,+ "w": 478.0,+ "x": 631.0,+ "y": 538.0+ },+ "label": "SCROLL_TOP_MARKER_v1\nLine 0\nLine 1\nLine 2\nLine 3\nLine 4\nLine 5\nLine 6\nLine 7\nLine 8\nLine 9\nLine 10\nLine 11\nLine 12\nLine 13\nLine 14\nLine 15\nLine 16\nLine 17\nLine 18\nLine 19\nLine 20\nLine 21\nLine 22\nLine 23\nLine 24\nLine 25\nLine 26\nLine 27\nLine 28\nLine 29\nSCROLL_BOTTOM_MARKER_v1",+ "parent_index": 0,+ "role": "AXTextArea",+ "value": "SCROLL_TOP_MARKER_v1\nLine 0\nLine 1\nLine 2\nLine 3\nLine 4\nLine 5\nLine 6\nLine 7\nLine 8\nLine 9\nLine 10\nLine 11\nLine 12\nLine 13\nLine 14\nLine 15\nLine 16\nLine 17\nLine 18\nLine 19\nLine 20\nLine 21\nLine 22\nLine 23\nLine 24\nLine 25\nLine 26\nLine 27\nLine 28\nLine 29\nSCROLL_BOTTOM_MARKER_v1"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 2,+ "element_index": 16,+ "element_token": "s00000087:16",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 2,+ "element_index": 17,+ "element_token": "s00000087:17",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 2,+ "element_index": 18,+ "element_token": "s00000087:18",+ "frame": {+ "h": 82.0,+ "w": 6.0,+ "x": 1100.0,+ "y": 571.0+ },+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 2,+ "element_index": 19,+ "element_token": "s00000087:19",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 20,+ "element_token": "s00000087:20",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 49.0,+ "x": 629.0,+ "y": 664.0+ },+ "label": "Exit",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXShowMenu",+ "AXConfirm"+ ],+ "depth": 1,+ "element_index": 21,+ "element_token": "s00000087:21",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 242.0,+ "x": 629.0,+ "y": 720.0+ },+ "label": "Note",+ "parent_index": 0,+ "role": "AXTextField",+ "value": "hello jev"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 22,+ "element_token": "s00000087:22",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 86.0,+ "x": 881.0,+ "y": 720.0+ },+ "label": "Save note",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 23,+ "element_token": "s00000087:23",+ "enabled": true,+ "frame": {+ "h": 18.0,+ "w": 57.0,+ "x": 629.0,+ "y": 752.0+ },+ "label": "Small",+ "parent_index": 0,+ "role": "AXRadioButton",+ "selected": false,+ "value": "0"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 24,+ "element_token": "s00000087:24",+ "enabled": true,+ "frame": {+ "h": 18.0,+ "w": 73.0,+ "x": 696.0,+ "y": 752.0+ },+ "label": "Medium",+ "parent_index": 0,+ "role": "AXRadioButton",+ "selected": false,+ "value": "0"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 25,+ "element_token": "s00000087:25",+ "enabled": true,+ "frame": {+ "h": 18.0,+ "w": 59.0,+ "x": 779.0,+ "y": 752.0+ },+ "label": "Large",+ "parent_index": 0,+ "role": "AXRadioButton",+ "selected": true,+ "value": "1"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 26,+ "element_token": "s00000087:26",+ "enabled": true,+ "frame": {+ "h": 16.0,+ "w": 16.0,+ "x": 618.0,+ "y": 56.0+ },+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 27,+ "element_token": "s00000087:27",+ "enabled": true,+ "frame": {+ "h": 16.0,+ "w": 16.0,+ "x": 641.0,+ "y": 56.0+ },+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXCancel"+ ],+ "depth": 0,+ "element_index": 28,+ "element_token": "s00000087:28",+ "enabled": true,+ "frame": {+ "h": 30.0,+ "w": 1920.0,+ "x": 0.0,+ "y": 0.0+ },+ "role": "AXMenuBar",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 1,+ "element_index": 29,+ "element_token": "s00000087:29",+ "enabled": true,+ "frame": {+ "h": 30.0,+ "w": 34.0,+ "x": 10.0,+ "y": 0.0+ },+ "label": "Apple",+ "parent_index": 28,+ "role": "AXMenuBarItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel"+ ],+ "depth": 2,+ "element_index": 30,+ "element_token": "s00000087:30",+ "enabled": true,+ "parent_index": 29,+ "role": "AXMenu",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 3,+ "element_index": 31,+ "element_token": "s00000087:31",+ "enabled": true,+ "label": "About This Mac",+ "parent_index": 30,+ "role": "AXMenuItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 3,+ "element_index": 32,+ "element_token": "s00000087:32",+ "enabled": true,+ "label": "System Information",+ "parent_index": 30,+ "role": "AXMenuItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 1,+ "element_index": 88,+ "element_token": "s00000087:88",+ "enabled": true,+ "frame": {+ "h": 30.0,+ "w": 175.0,+ "x": 44.0,+ "y": 0.0+ },+ "label": "CuaTestHarness.AppKit",+ "parent_index": 28,+ "role": "AXMenuBarItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel"+ ],+ "depth": 2,+ "element_index": 89,+ "element_token": "s00000087:89",+ "enabled": true,+ "parent_index": 88,+ "role": "AXMenu",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 3,+ "element_index": 90,+ "element_token": "s00000087:90",+ "enabled": true,+ "label": "Quit",+ "parent_index": 89,+ "role": "AXMenuItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 3,+ "element_index": 91,+ "element_token": "s00000087:91",+ "enabled": true,+ "label": "Quit and Keep Windows",+ "parent_index": 89,+ "role": "AXMenuItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 1,+ "element_index": 92,+ "element_token": "s00000087:92",+ "enabled": true,+ "frame": {+ "h": 30.0,+ "w": 69.0,+ "x": 219.0,+ "y": 0.0+ },+ "label": "Window",+ "parent_index": 28,+ "role": "AXMenuBarItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel"+ ],+ "depth": 2,+ "element_index": 93,+ "element_token": "s00000087:93",+ "enabled": true,+ "parent_index": 92,+ "role": "AXMenu",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 3,+ "element_index": 94,+ "element_token": "s00000087:94",+ "enabled": true,+ "label": "Move & Resize",+ "parent_index": 93,+ "role": "AXMenuItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel"+ ],+ "depth": 4,+ "element_index": 95,+ "element_token": "s00000087:95",+ "enabled": true,+ "parent_index": 94,+ "role": "AXMenu",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 5,+ "element_index": 96,+ "element_token": "s00000087:96",+ "enabled": true,+ "label": "Left",+ "parent_index": 95,+ "role": "AXMenuItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 5,+ "element_index": 97,+ "element_token": "s00000087:97",+ "enabled": true,+ "label": "Right",+ "parent_index": 95,+ "role": "AXMenuItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 5,+ "element_index": 98,+ "element_token": "s00000087:98",+ "enabled": true,+ "label": "Return to Previous Size",+ "parent_index": 95,+ "role": "AXMenuItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 3,+ "element_index": 99,+ "element_token": "s00000087:99",+ "enabled": true,+ "label": "Arrange",+ "parent_index": 93,+ "role": "AXMenuItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel"+ ],+ "depth": 4,+ "element_index": 100,+ "element_token": "s00000087:100",+ "enabled": true,+ "parent_index": 99,+ "role": "AXMenu",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 5,+ "element_index": 101,+ "element_token": "s00000087:101",+ "enabled": true,+ "label": "Left",+ "parent_index": 100,+ "role": "AXMenuItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 3,+ "element_index": 102,+ "element_token": "s00000087:102",+ "enabled": true,+ "label": "CuaTestHarness AppKit",+ "parent_index": 93,+ "role": "AXMenuItem",+ "selected": false+ }+ ]+}diff --git a/libs/cua-driver/examples/jev-use/fixtures/native/appkit-window-state-initial-v1.json b/libs/cua-driver/examples/jev-use/fixtures/native/appkit-window-state-initial-v1.jsonnew file mode 100644index 0000000000..0820e7df0a--- /dev/null+++ b/libs/cua-driver/examples/jev-use/fixtures/native/appkit-window-state-initial-v1.json@@ -0,0 +1,799 @@+{+ "_fixture": {+ "source": "CuaTestHarness.AppKit with CUA_APPKIT_TASK_STATE set, observed by cua-driver 0.28.2 get_window_state (macOS 26)",+ "sanitized": "Screenshot bytes and tree_markdown removed. The system Apple menu's machine-specific items (recent items, account names) were removed; two generic items remain. capture_id was added because Cua Driver 0.30 and later return it with the screenshot."+ },+ "pid": 20535,+ "window_id": 71364,+ "app_name": "CuaTestHarness.AppKit",+ "window_title": "CuaTestHarness AppKit",+ "snapshot_id": "s00000086",+ "element_count": 103,+ "total_element_count": 103,+ "returned_element_count": 103,+ "elements_complete": false,+ "screenshot_width": 700,+ "screenshot_height": 892,+ "screenshot_scale": 1.0,+ "window_bounds": {+ "height": 892.0,+ "width": 700.0,+ "x": 610.0,+ "y": 48.0+ },+ "capture_id": "cap-appkit-initial",+ "elements": [+ {+ "actions": [+ "AXRaise"+ ],+ "depth": 0,+ "element_index": 0,+ "element_token": "s00000086:0",+ "frame": {+ "h": 892.0,+ "w": 700.0,+ "x": 610.0,+ "y": 48.0+ },+ "label": "CuaTestHarness AppKit",+ "role": "AXWindow"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 1,+ "element_token": "s00000086:1",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 87.0,+ "x": 629.0,+ "y": 115.0+ },+ "label": "Increment",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 2,+ "element_token": "s00000086:2",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 61.0,+ "x": 726.0,+ "y": 115.0+ },+ "label": "Reset",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXShowMenu",+ "AXConfirm"+ ],+ "depth": 1,+ "element_index": 3,+ "element_token": "s00000086:3",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 242.0,+ "x": 629.0,+ "y": 219.0+ },+ "label": "Type here…",+ "parent_index": 0,+ "role": "AXTextField",+ "value": "Type here…"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 4,+ "element_token": "s00000086:4",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 221.0,+ "x": 629.0,+ "y": 275.0+ },+ "label": "Click target (left / right / double)",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXIncrement",+ "AXDecrement"+ ],+ "depth": 1,+ "element_index": 5,+ "element_token": "s00000086:5",+ "enabled": true,+ "frame": {+ "h": 18.0,+ "w": 322.0,+ "x": 629.0,+ "y": 331.0+ },+ "label": "sld-value",+ "max": 100.0,+ "min": 0.0,+ "parent_index": 0,+ "role": "AXSlider",+ "value": "0"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 6,+ "element_token": "s00000086:6",+ "enabled": true,+ "frame": {+ "h": 18.0,+ "w": 66.0,+ "x": 629.0,+ "y": 406.0+ },+ "label": "I agree",+ "parent_index": 0,+ "role": "AXCheckBox",+ "selected": false,+ "value": "0"+ },+ {+ "actions": [+ "AXShowMenu"+ ],+ "depth": 4,+ "element_index": 7,+ "element_token": "s00000086:7",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 138.0,+ "x": 830.0,+ "y": 381.0+ },+ "label": "alpha",+ "parent_index": 0,+ "role": "AXStaticText",+ "value": "alpha"+ },+ {+ "actions": [+ "AXShowMenu"+ ],+ "depth": 4,+ "element_index": 8,+ "element_token": "s00000086:8",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 138.0,+ "x": 830.0,+ "y": 405.0+ },+ "label": "beta",+ "parent_index": 0,+ "role": "AXStaticText",+ "value": "beta"+ },+ {+ "actions": [+ "AXShowMenu"+ ],+ "depth": 4,+ "element_index": 9,+ "element_token": "s00000086:9",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 138.0,+ "x": 830.0,+ "y": 429.0+ },+ "label": "gamma",+ "parent_index": 0,+ "role": "AXStaticText",+ "value": "gamma"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 2,+ "element_index": 10,+ "element_token": "s00000086:10",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 2,+ "element_index": 11,+ "element_token": "s00000086:11",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 2,+ "element_index": 12,+ "element_token": "s00000086:12",+ "frame": {+ "h": 5.0,+ "w": 6.0,+ "x": 962.0,+ "y": 440.0+ },+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 2,+ "element_index": 13,+ "element_token": "s00000086:13",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXShowMenu",+ "AXPress"+ ],+ "depth": 1,+ "element_index": 14,+ "element_token": "s00000086:14",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 198.0,+ "x": 629.0,+ "y": 480.0+ },+ "label": "Right-click for context menu",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXShowMenu"+ ],+ "depth": 1,+ "element_index": 15,+ "element_token": "s00000086:15",+ "frame": {+ "h": 448.0,+ "w": 478.0,+ "x": 631.0,+ "y": 538.0+ },+ "label": "SCROLL_TOP_MARKER_v1\nLine 0\nLine 1\nLine 2\nLine 3\nLine 4\nLine 5\nLine 6\nLine 7\nLine 8\nLine 9\nLine 10\nLine 11\nLine 12\nLine 13\nLine 14\nLine 15\nLine 16\nLine 17\nLine 18\nLine 19\nLine 20\nLine 21\nLine 22\nLine 23\nLine 24\nLine 25\nLine 26\nLine 27\nLine 28\nLine 29\nSCROLL_BOTTOM_MARKER_v1",+ "parent_index": 0,+ "role": "AXTextArea",+ "value": "SCROLL_TOP_MARKER_v1\nLine 0\nLine 1\nLine 2\nLine 3\nLine 4\nLine 5\nLine 6\nLine 7\nLine 8\nLine 9\nLine 10\nLine 11\nLine 12\nLine 13\nLine 14\nLine 15\nLine 16\nLine 17\nLine 18\nLine 19\nLine 20\nLine 21\nLine 22\nLine 23\nLine 24\nLine 25\nLine 26\nLine 27\nLine 28\nLine 29\nSCROLL_BOTTOM_MARKER_v1"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 2,+ "element_index": 16,+ "element_token": "s00000086:16",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 2,+ "element_index": 17,+ "element_token": "s00000086:17",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 2,+ "element_index": 18,+ "element_token": "s00000086:18",+ "frame": {+ "h": 82.0,+ "w": 6.0,+ "x": 1100.0,+ "y": 571.0+ },+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 2,+ "element_index": 19,+ "element_token": "s00000086:19",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 20,+ "element_token": "s00000086:20",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 49.0,+ "x": 629.0,+ "y": 664.0+ },+ "label": "Exit",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXShowMenu",+ "AXConfirm"+ ],+ "depth": 1,+ "element_index": 21,+ "element_token": "s00000086:21",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 242.0,+ "x": 629.0,+ "y": 720.0+ },+ "label": "Note",+ "parent_index": 0,+ "role": "AXTextField"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 22,+ "element_token": "s00000086:22",+ "enabled": true,+ "frame": {+ "h": 26.0,+ "w": 86.0,+ "x": 881.0,+ "y": 720.0+ },+ "label": "Save note",+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 23,+ "element_token": "s00000086:23",+ "enabled": true,+ "frame": {+ "h": 18.0,+ "w": 57.0,+ "x": 629.0,+ "y": 752.0+ },+ "label": "Small",+ "parent_index": 0,+ "role": "AXRadioButton",+ "selected": false,+ "value": "0"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 24,+ "element_token": "s00000086:24",+ "enabled": true,+ "frame": {+ "h": 18.0,+ "w": 73.0,+ "x": 696.0,+ "y": 752.0+ },+ "label": "Medium",+ "parent_index": 0,+ "role": "AXRadioButton",+ "selected": false,+ "value": "0"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 25,+ "element_token": "s00000086:25",+ "enabled": true,+ "frame": {+ "h": 18.0,+ "w": 59.0,+ "x": 779.0,+ "y": 752.0+ },+ "label": "Large",+ "parent_index": 0,+ "role": "AXRadioButton",+ "selected": false,+ "value": "0"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 26,+ "element_token": "s00000086:26",+ "enabled": true,+ "frame": {+ "h": 16.0,+ "w": 16.0,+ "x": 618.0,+ "y": 56.0+ },+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXPress"+ ],+ "depth": 1,+ "element_index": 27,+ "element_token": "s00000086:27",+ "enabled": true,+ "frame": {+ "h": 16.0,+ "w": 16.0,+ "x": 641.0,+ "y": 56.0+ },+ "parent_index": 0,+ "role": "AXButton"+ },+ {+ "actions": [+ "AXCancel"+ ],+ "depth": 0,+ "element_index": 28,+ "element_token": "s00000086:28",+ "enabled": true,+ "frame": {+ "h": 30.0,+ "w": 1920.0,+ "x": 0.0,+ "y": 0.0+ },+ "role": "AXMenuBar",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 1,+ "element_index": 29,+ "element_token": "s00000086:29",+ "enabled": true,+ "frame": {+ "h": 30.0,+ "w": 34.0,+ "x": 10.0,+ "y": 0.0+ },+ "label": "Apple",+ "parent_index": 28,+ "role": "AXMenuBarItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel"+ ],+ "depth": 2,+ "element_index": 30,+ "element_token": "s00000086:30",+ "enabled": true,+ "parent_index": 29,+ "role": "AXMenu",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 3,+ "element_index": 31,+ "element_token": "s00000086:31",+ "enabled": true,+ "label": "About This Mac",+ "parent_index": 30,+ "role": "AXMenuItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 3,+ "element_index": 32,+ "element_token": "s00000086:32",+ "enabled": true,+ "label": "System Information",+ "parent_index": 30,+ "role": "AXMenuItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 1,+ "element_index": 88,+ "element_token": "s00000086:88",+ "enabled": true,+ "frame": {+ "h": 30.0,+ "w": 175.0,+ "x": 44.0,+ "y": 0.0+ },+ "label": "CuaTestHarness.AppKit",+ "parent_index": 28,+ "role": "AXMenuBarItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel"+ ],+ "depth": 2,+ "element_index": 89,+ "element_token": "s00000086:89",+ "enabled": true,+ "parent_index": 88,+ "role": "AXMenu",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 3,+ "element_index": 90,+ "element_token": "s00000086:90",+ "enabled": true,+ "label": "Quit",+ "parent_index": 89,+ "role": "AXMenuItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 3,+ "element_index": 91,+ "element_token": "s00000086:91",+ "enabled": true,+ "label": "Quit and Keep Windows",+ "parent_index": 89,+ "role": "AXMenuItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 1,+ "element_index": 92,+ "element_token": "s00000086:92",+ "enabled": true,+ "frame": {+ "h": 30.0,+ "w": 69.0,+ "x": 219.0,+ "y": 0.0+ },+ "label": "Window",+ "parent_index": 28,+ "role": "AXMenuBarItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel"+ ],+ "depth": 2,+ "element_index": 93,+ "element_token": "s00000086:93",+ "enabled": true,+ "parent_index": 92,+ "role": "AXMenu",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 3,+ "element_index": 94,+ "element_token": "s00000086:94",+ "enabled": true,+ "label": "Move & Resize",+ "parent_index": 93,+ "role": "AXMenuItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel"+ ],+ "depth": 4,+ "element_index": 95,+ "element_token": "s00000086:95",+ "enabled": true,+ "parent_index": 94,+ "role": "AXMenu",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 5,+ "element_index": 96,+ "element_token": "s00000086:96",+ "enabled": true,+ "label": "Left",+ "parent_index": 95,+ "role": "AXMenuItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 5,+ "element_index": 97,+ "element_token": "s00000086:97",+ "enabled": true,+ "label": "Right",+ "parent_index": 95,+ "role": "AXMenuItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 5,+ "element_index": 98,+ "element_token": "s00000086:98",+ "enabled": true,+ "label": "Return to Previous Size",+ "parent_index": 95,+ "role": "AXMenuItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 3,+ "element_index": 99,+ "element_token": "s00000086:99",+ "enabled": true,+ "label": "Arrange",+ "parent_index": 93,+ "role": "AXMenuItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel"+ ],+ "depth": 4,+ "element_index": 100,+ "element_token": "s00000086:100",+ "enabled": true,+ "parent_index": 99,+ "role": "AXMenu",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 5,+ "element_index": 101,+ "element_token": "s00000086:101",+ "enabled": true,+ "label": "Left",+ "parent_index": 100,+ "role": "AXMenuItem",+ "selected": false+ },+ {+ "actions": [+ "AXCancel",+ "AXPress",+ "AXPick"+ ],+ "depth": 3,+ "element_index": 102,+ "element_token": "s00000086:102",+ "enabled": true,+ "label": "CuaTestHarness AppKit",+ "parent_index": 93,+ "role": "AXMenuItem",+ "selected": false+ }+ ]+}diff --git a/libs/cua-driver/examples/jev-use/fixtures/native/native-candidate-ids-v1.json b/libs/cua-driver/examples/jev-use/fixtures/native/native-candidate-ids-v1.jsonnew file mode 100644index 0000000000..9692331cad--- /dev/null+++ b/libs/cua-driver/examples/jev-use/fixtures/native/native-candidate-ids-v1.json@@ -0,0 +1,357 @@+{+ "_fixture": "Expected stable candidate IDs; generated from python/native.py and checked by both languages.",+ "cases": [+ {+ "name": "collisions",+ "platform": "macos",+ "window_state": {+ "pid": 7,+ "window_id": 9,+ "snapshot_id": "s0000000a",+ "capture_id": "cap-1",+ "elements_complete": false,+ "window_bounds": {+ "x": 0,+ "y": 0,+ "width": 500,+ "height": 400+ },+ "elements": [+ {+ "element_index": 0,+ "role": "AXWindow",+ "depth": 0,+ "label": "Main",+ "element_token": "s0000000a:0",+ "frame": {+ "x": 0,+ "y": 0,+ "w": 500,+ "h": 400+ }+ },+ {+ "element_index": 1,+ "role": "AXButton",+ "depth": 1,+ "parent_index": 0,+ "enabled": true,+ "element_token": "s0000000a:1",+ "frame": {+ "x": 11,+ "y": 10,+ "w": 40,+ "h": 20+ },+ "label": "Options"+ },+ {+ "element_index": 2,+ "role": "AXButton",+ "depth": 1,+ "parent_index": 1,+ "enabled": true,+ "element_token": "s0000000a:2",+ "frame": {+ "x": 12,+ "y": 10,+ "w": 40,+ "h": 20+ },+ "label": "OK"+ },+ {+ "element_index": 3,+ "role": "AXButton",+ "depth": 1,+ "parent_index": 0,+ "enabled": true,+ "element_token": "s0000000a:3",+ "frame": {+ "x": 13,+ "y": 10,+ "w": 40,+ "h": 20+ },+ "label": "OK"+ },+ {+ "element_index": 4,+ "role": "AXButton",+ "depth": 1,+ "parent_index": 0,+ "enabled": true,+ "element_token": "s0000000a:4",+ "frame": {+ "x": 14,+ "y": 10,+ "w": 40,+ "h": 20+ },+ "label": "OK"+ },+ {+ "element_index": 5,+ "role": "AXButton",+ "depth": 1,+ "parent_index": 0,+ "enabled": true,+ "element_token": "s0000000a:5",+ "frame": {+ "x": 15,+ "y": 10,+ "w": 40,+ "h": 20+ },+ "label": "你好"+ },+ {+ "element_index": 6,+ "role": "AXCheckBox",+ "depth": 1,+ "parent_index": 0,+ "enabled": true,+ "element_token": "s0000000a:6",+ "frame": {+ "x": 16,+ "y": 10,+ "w": 40,+ "h": 20+ },+ "label": "Café — réglages"+ }+ ]+ },+ "expected_ids": [+ "ax:button:options",+ "ax:button:ok:8f0f",+ "ax:button:ok:88b0",+ "ax:button:ok:33f5",+ "ax:button:670d9743",+ "ax:checkbox:caf-r-glages"+ ]+ },+ {+ "name": "linux",+ "platform": "linux",+ "window_state": {+ "pid": 7,+ "window_id": 9,+ "snapshot_id": "s0000000a",+ "capture_id": "cap-1",+ "elements_complete": false,+ "window_bounds": {+ "x": 0,+ "y": 0,+ "width": 500,+ "height": 400+ },+ "elements": [+ {+ "element_index": 0,+ "role": "AXWindow",+ "depth": 0,+ "label": "Main",+ "element_token": "s0000000a:0",+ "frame": {+ "x": 0,+ "y": 0,+ "w": 500,+ "h": 400+ }+ },+ {+ "element_index": 1,+ "role": "push button",+ "depth": 1,+ "parent_index": 0,+ "enabled": true,+ "element_token": "s0000000a:1",+ "frame": {+ "x": 11,+ "y": 10,+ "w": 40,+ "h": 20+ },+ "label": "Apply"+ },+ {+ "element_index": 2,+ "role": "toggle button",+ "depth": 1,+ "parent_index": 0,+ "enabled": true,+ "element_token": "s0000000a:2",+ "frame": {+ "x": 12,+ "y": 10,+ "w": 40,+ "h": 20+ },+ "label": "Bold"+ },+ {+ "element_index": 3,+ "role": "entry",+ "depth": 1,+ "parent_index": 0,+ "enabled": true,+ "element_token": "s0000000a:3",+ "frame": {+ "x": 13,+ "y": 10,+ "w": 40,+ "h": 20+ },+ "label": "Search"+ },+ {+ "element_index": 4,+ "role": "label",+ "depth": 1,+ "parent_index": 0,+ "enabled": true,+ "element_token": "s0000000a:4",+ "frame": {+ "x": 14,+ "y": 10,+ "w": 40,+ "h": 20+ },+ "label": "Static"+ }+ ]+ },+ "expected_ids": [+ "ax:button:apply",+ "ax:toggle:bold",+ "ax:text_input:search"+ ]+ },+ {+ "name": "windows",+ "platform": "windows",+ "window_state": {+ "pid": 7,+ "window_id": 9,+ "snapshot_id": "s0000000a",+ "capture_id": "cap-1",+ "elements_complete": false,+ "window_bounds": {+ "x": 0,+ "y": 0,+ "width": 500,+ "height": 400+ },+ "elements": [+ {+ "element_index": 0,+ "role": "AXWindow",+ "depth": 0,+ "label": "Main",+ "element_token": "s0000000a:0",+ "frame": {+ "x": 0,+ "y": 0,+ "w": 500,+ "h": 400+ }+ },+ {+ "element_index": 1,+ "role": "SplitButton",+ "depth": 1,+ "parent_index": 0,+ "enabled": true,+ "element_token": "s0000000a:1",+ "frame": {+ "x": 11,+ "y": 10,+ "w": 40,+ "h": 20+ },+ "label": "New"+ },+ {+ "element_index": 2,+ "role": "Edit",+ "depth": 1,+ "parent_index": 0,+ "enabled": true,+ "element_token": "s0000000a:2",+ "frame": {+ "x": 12,+ "y": 10,+ "w": 40,+ "h": 20+ },+ "label": "Name"+ },+ {+ "element_index": 3,+ "role": "Text",+ "depth": 1,+ "parent_index": 0,+ "enabled": true,+ "element_token": "s0000000a:3",+ "frame": {+ "x": 13,+ "y": 10,+ "w": 40,+ "h": 20+ },+ "label": "Static"+ },+ {+ "element_index": 4,+ "role": "Hyperlink",+ "depth": 1,+ "parent_index": 0,+ "enabled": true,+ "element_token": "s0000000a:4",+ "frame": {+ "x": 14,+ "y": 10,+ "w": 40,+ "h": 20+ },+ "label": "Help"+ }+ ]+ },+ "expected_ids": [+ "ax:button:new",+ "ax:text_input:name",+ "ax:link:help"+ ]+ }+ ],+ "slugs": {+ "Click target (left / right / double)": "click-target-left-right-double",+ "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",+ "你好": "670d9743",+ " --Save-- note ": "save-note"+ },+ "risks": {+ "Delete file": [+ "destructive"+ ],+ "Don’t Save": [+ "close_unsaved"+ ],+ "Buy now": [+ "purchase"+ ],+ "Submit": [+ "send"+ ],+ "Resetting": [],+ "Save note": [],+ "Clear All": [+ "destructive"+ ],+ "Quit": [+ "close_unsaved"+ ]+ }+}\ No newline at end of filediff --git a/libs/cua-driver/examples/jev-use/python/choose_action.py b/libs/cua-driver/examples/jev-use/python/choose_action.pyindex 9dee37acd1..e558cbdfcf 100644--- a/libs/cua-driver/examples/jev-use/python/choose_action.py+++ b/libs/cua-driver/examples/jev-use/python/choose_action.py@@ -7,14 +7,27 @@ import sysfrom typing import Any, Mappingfrom jev_adapter import ProviderChoice, choose_bounded_with_typesafe+from native_roles import ROLE_CLASSESREQUEST_SCHEMA = "cua.jev_choice_request_v1"+# Additive over v1 (RFC #4268): per-candidate ``source``, an optional root+# ``snapshot_id``, and an optional compact ``elements`` list of native controls.+REQUEST_SCHEMA_V2 = "cua.jev_choice_request_v2"+REQUEST_SCHEMAS = (REQUEST_SCHEMA, REQUEST_SCHEMA_V2)RESPONSE_SCHEMA = "cua.jev_choice_v1"MAX_INPUT_BYTES = 65_536MAX_CANDIDATES = 32MAX_REGIONS = 100MAX_HISTORY = 16+MAX_ELEMENTS = 64ID_PATTERN = re.compile(r"[A-Za-z0-9][A-Za-z0-9._:-]{0,63}\Z")+RESERVED_IDS = frozenset({"reobserve", "abstain"})+CANDIDATE_SOURCES = frozenset({"page", "ax", "visual"})+ELEMENT_STATES = frozenset(+ {"enabled", "checked", "unchecked", "selected", "not_selected", "empty", "has_text"}+)+V1_ROOT_KEYS = frozenset({"schema", "goal", "capture_id", "regions", "history", "candidates"})+V2_OPTIONAL_ROOT_KEYS = frozenset({"snapshot_id", "elements"})def _record(value: Any, message: str) -> dict[str, Any]:@@ -36,11 +49,49 @@ def _identifier(value: Any, name: str) -> str:return value+def _validate_elements(value: Any) -> list[dict[str, str]]:+ if not isinstance(value, list) or len(value) > MAX_ELEMENTS:+ raise ValueError(f"elements must be an array of at most {MAX_ELEMENTS} items")+ elements: list[dict[str, str]] = []+ for raw_value in value:+ raw = _record(raw_value, "element must be an object")+ if set(raw) != {"role_class", "label", "state"}:+ raise ValueError("element may contain only role_class, label, and state")+ if raw["role_class"] not in ROLE_CLASSES:+ raise ValueError("element role_class is not a supported role class")+ if raw["state"] not in ELEMENT_STATES:+ raise ValueError("element state is not a supported state")+ elements.append(+ {+ "role_class": raw["role_class"],+ "label": _bounded_string(raw["label"], "element label", 200),+ "state": raw["state"],+ }+ )+ return elements++def validate_request(value: Any) -> dict[str, Any]:+ """Validate a ``cua.jev_choice_request_v1`` or ``_v2`` request strictly.++ v1 is unchanged: exactly its six root keys and ``id``/``description``+ candidates, so any v2 field in a v1 request is rejected. v2 may add a root+ ``snapshot_id`` and ``elements`` and a per-candidate ``source``; reserved+ candidates never carry a source.+ """root = _record(value, "request must be a JSON object")- expected_keys = {"schema", "goal", "capture_id", "regions", "history", "candidates"}- if set(root) != expected_keys or root.get("schema") != REQUEST_SCHEMA:- raise ValueError(f"request must match {REQUEST_SCHEMA}")+ schema = root.get("schema")+ if schema == REQUEST_SCHEMA:+ if set(root) != V1_ROOT_KEYS:+ raise ValueError(f"request must match {REQUEST_SCHEMA}")+ elif schema == REQUEST_SCHEMA_V2:+ if not V1_ROOT_KEYS.issubset(root) or not set(root).issubset(+ V1_ROOT_KEYS | V2_OPTIONAL_ROOT_KEYS+ ):+ raise ValueError(f"request must match {REQUEST_SCHEMA_V2}")+ else:+ raise ValueError(f"request must match {REQUEST_SCHEMA} or {REQUEST_SCHEMA_V2}")+ v2 = schema == REQUEST_SCHEMA_V2goal = _bounded_string(root["goal"], "goal", 4_000)capture_id = _bounded_string(root["capture_id"], "capture_id", 256)@@ -126,28 +177,68 @@ def validate_request(value: Any) -> dict[str, Any]:candidate_ids: set[str] = set()for raw_value in raw_candidates:raw = _record(raw_value, "candidate must be an object")- if set(raw) != {"id", "description"}:+ keys = set(raw)+ if v2:+ if not {"id", "description"}.issubset(keys) or not keys.issubset(+ {"id", "description", "source"}+ ):+ raise ValueError("candidate may contain only id, description, and source")+ elif keys != {"id", "description"}:raise ValueError("candidate may contain only id and description")candidate_id = _identifier(raw["id"], "candidate id")if candidate_id in candidate_ids:raise ValueError("candidate IDs must be unique")candidate_ids.add(candidate_id)- candidates.append(- {- "id": candidate_id,- "description": _bounded_string(raw["description"], "description", 1_000),- }- )- if not {"reobserve", "abstain"}.issubset(candidate_ids):+ candidate = {+ "id": candidate_id,+ "description": _bounded_string(raw["description"], "description", 1_000),+ }+ if "source" in raw:+ if candidate_id in RESERVED_IDS:+ raise ValueError("reserved candidates must not carry a source")+ if raw["source"] not in CANDIDATE_SOURCES:+ raise ValueError("candidate source must be page, ax, or visual")+ candidate["source"] = raw["source"]+ candidates.append(candidate)+ if not RESERVED_IDS.issubset(candidate_ids):raise ValueError("candidates must include reobserve and abstain")- return {+ validated: dict[str, Any] = {+ "schema": schema,"goal": goal,"capture_id": capture_id,"regions": regions,"history": compact_history,"candidates": candidates,}+ if v2:+ snapshot_id = root.get("snapshot_id")+ validated["snapshot_id"] = (+ None if snapshot_id is None else _bounded_string(snapshot_id, "snapshot_id", 64)+ )+ validated["elements"] = _validate_elements(root.get("elements", []))+ return validated+++def provider_observation(validated: Mapping[str, Any]) -> dict[str, Any]:+ """Build the observation a provider sees for a validated request.++ A v1 request keeps its exact v1 observation. A v2 request adds the native+ ``snapshot_id``, the compact ``elements``, and each candidate's ``source``;+ none of these contain element tokens, values, or pixels.+ """+ observation: dict[str, Any] = {+ "capture_id": validated["capture_id"],+ "regions": validated["regions"],+ "history": validated["history"],+ }+ if validated.get("schema") == REQUEST_SCHEMA_V2:+ observation["snapshot_id"] = validated.get("snapshot_id")+ observation["elements"] = validated.get("elements", [])+ observation["candidate_sources"] = {+ item["id"]: item["source"] for item in validated["candidates"] if "source" in item+ }+ return observationdef choose_request(@@ -181,22 +272,14 @@ def choose_request(choice = choose_bounded_with_typesafe(live_client,goal=validated["goal"],- observation={- "capture_id": validated["capture_id"],- "regions": validated["regions"],- "history": validated["history"],- },+ observation=provider_observation(validated),criteria=criteria,)else:choice = choose_bounded_with_typesafe(client,goal=validated["goal"],- observation={- "capture_id": validated["capture_id"],- "regions": validated["regions"],- "history": validated["history"],- },+ observation=provider_observation(validated),criteria=criteria,)return {diff --git a/libs/cua-driver/examples/jev-use/python/decision_models.py b/libs/cua-driver/examples/jev-use/python/decision_models.pyindex 8d2a2bc63a..d59069dbaf 100644--- a/libs/cua-driver/examples/jev-use/python/decision_models.py+++ b/libs/cua-driver/examples/jev-use/python/decision_models.py@@ -14,6 +14,7 @@ from pathlib import Pathfrom types import MappingProxyTypefrom typing import Any, Literal, Mapping, Protocol+from choose_action import REQUEST_SCHEMA, REQUEST_SCHEMA_V2, provider_observationfrom jev_adapter import choose_bounded_with_typesafeDecisionKind = Literal["selected", "reobserve", "abstain", "error"]@@ -26,6 +27,10 @@ class DecisionRequest:regions: tuple[Mapping[str, Any], ...]history: tuple[Mapping[str, str], ...]candidates: tuple[Mapping[str, str], ...]+ # v2 only (RFC #4268): native snapshot and compact accessibility elements.+ schema: str = REQUEST_SCHEMA+ snapshot_id: str | None = None+ elements: tuple[Mapping[str, str], ...] = ()@classmethoddef from_validated(cls, request: Mapping[str, Any]) -> "DecisionRequest":@@ -35,12 +40,29 @@ class DecisionRequest:regions=tuple(request["regions"]),history=tuple(request["history"]),candidates=tuple(MappingProxyType(dict(item)) for item in request["candidates"]),+ schema=request.get("schema", REQUEST_SCHEMA),+ snapshot_id=request.get("snapshot_id"),+ elements=tuple(MappingProxyType(dict(item)) for item in request.get("elements", ())),)@propertydef criteria(self) -> dict[str, str]:return {item["id"]: item["description"] for item in self.candidates}+ def provider_observation(self) -> dict[str, Any]:+ """The provider observation; byte-identical to v1 for a v1 request."""+ validated: dict[str, Any] = {+ "schema": self.schema,+ "capture_id": self.capture_id,+ "regions": list(self.regions),+ "history": list(self.history),+ "candidates": [dict(item) for item in self.candidates],+ }+ if self.schema == REQUEST_SCHEMA_V2:+ validated["snapshot_id"] = self.snapshot_id+ validated["elements"] = [dict(item) for item in self.elements]+ return provider_observation(validated)+@dataclass(frozen=True)class ModelScores:@@ -151,11 +173,7 @@ class TypeSafeDecisionModel:choice = choose_bounded_with_typesafe(self.client,goal=request.goal,- observation={- "capture_id": request.capture_id,- "regions": list(request.regions),- "history": list(request.history),- },+ observation=request.provider_observation(),criteria=request.criteria,)if choice.selected_id not in request.criteria:@@ -197,6 +215,31 @@ def visual_regions_as_text(request: DecisionRequest) -> str:return "\n".join(lines)+def native_elements_as_text(request: DecisionRequest) -> str:+ """Render a v2 request's native elements as an accessibility tree for S1.++ Each line is one supplied native control: its role class, label, and state.+ Visual regions from the same capture, if any, follow as a separate section.+ """+ lines = [+ f"Accessibility tree for snapshot {json.dumps(request.snapshot_id)} "+ f"(capture {json.dumps(request.capture_id)}):"+ ]+ for element in request.elements:+ lines.append(+ f"- {element['role_class']} {json.dumps(element['label'], ensure_ascii=False)} "+ f"({element['state']})"+ )+ if request.regions:+ lines.append(visual_regions_as_text(DecisionRequest(+ request.goal, request.capture_id, request.regions, (), request.candidates+ )))+ if request.history:+ lines.append("Prior bounded decisions:")+ lines.extend(json.dumps(item, ensure_ascii=False) for item in request.history)+ return "\n".join(lines)++S1_MODALITIES = ("text", "multimodal")_HF_SNAPSHOT_PARENT = "snapshots"_REVISION_PATTERN = re.compile(r"[0-9a-f]{40}")@@ -320,7 +363,11 @@ class S1DecisionModel:"modality": self.modality,}if self.modality == "text":- kwargs["ax_tree"] = visual_regions_as_text(request)+ kwargs["ax_tree"] = (+ native_elements_as_text(request)+ if request.schema == REQUEST_SCHEMA_V2 and request.elements+ else visual_regions_as_text(request)+ )else:kwargs["screenshot"] = self.screenshot_pathscores = self.scorer.forward(options, **kwargs)diff --git a/libs/cua-driver/examples/jev-use/python/jev_adapter.py b/libs/cua-driver/examples/jev-use/python/jev_adapter.pyindex c263edf499..b376752f32 100644--- a/libs/cua-driver/examples/jev-use/python/jev_adapter.py+++ b/libs/cua-driver/examples/jev-use/python/jev_adapter.py@@ -109,7 +109,7 @@ def task_decision_state(outline. Every secret task parameter is replaced everywhere, including theoutline and visual text."""- snapshot = sources.page.snapshot+ snapshot = sources.require_page().snapshotvisual = sources.visual.observation if sources.visual is not None else Nonereturn {"goal": task.goal,@@ -197,12 +197,33 @@ def choose_live_for_task(def choose_mock_for_task(- _task: Task,+ task: Task,_sources: TaskSources,candidates: list[Candidate],_history: list[dict[str, Any]],) -> tuple[str | None, float, dict[str, float]]:- return choose_mock(candidates)+ """Deterministic mock provider.++ A task may declare ``mock_preferences``: the first preferred ID present in+ the candidate set wins (a refused control's ``<id>:foreground`` variant+ counts as its ID), otherwise ``reobserve``. Tasks without preferences keep+ the fixed browser-fixture choice order.+ """+ preferences = getattr(task, "mock_preferences", ())+ if not preferences:+ return choose_mock(candidates)+ ids = [candidate.id for candidate in candidates]+ selected = next(+ (+ candidate_id+ for preferred in preferences+ for candidate_id in (preferred, f"{preferred}:foreground")+ if candidate_id in ids+ ),+ "reobserve" if "reobserve" in ids else None,+ )+ probabilities = {candidate_id: float(candidate_id == selected) for candidate_id in ids}+ return selected, 1.0 if selected else 0.0, probabilitiesdef choose_live(diff --git a/libs/cua-driver/examples/jev-use/python/native.py b/libs/cua-driver/examples/jev-use/python/native.pynew file mode 100644index 0000000000..702afb74ed--- /dev/null+++ b/libs/cua-driver/examples/jev-use/python/native.py@@ -0,0 +1,343 @@+"""Native accessibility observations for jev-use (RFC #4268, Phase 1).++This module turns one ``get_window_state`` result into the eligible native+controls that ``NativeAccessibilitySource`` (``sources.py``) offers as+candidates. It owns the element rules, stable candidate IDs, and the+risky-action policy. It never builds Driver arguments; the source does.++One observation is exactly one ``get_window_state`` call with both the tree+and the screenshot, so its ``snapshot_id`` (element tokens) and ``capture_id``+(visual regions and capture-bound clicks) describe the same moment. A second+call would publish a new snapshot and invalidate every token in the set.++Element rules (all must hold):++1. the raw role maps to a role class (``native_roles.py``);+2. ``enabled`` is not ``false``;+3. ``frame`` exists, has positive size, and intersects ``window_bounds``;+4. the redacted label is non-empty, is not a copy of the element's own value,+ and the element is not marked ``unlabelled``;+5. ``in_web_content`` is not ``true`` (web content uses the browser source).++Stable IDs never use ``element_index``: ``ax:<role_class>:<slug(label)>``,+plus a short hash of the actionable-ancestor path and ordinal when the base+repeats in one observation.+"""++from __future__ import annotations++import hashlib+import json+import re+from dataclasses import dataclass+from typing import Any, Callable, Literal, Mapping++from native_roles import ROLE_CLASS_ACTION, ActionKind, Platform, RoleClass, normalized_role, role_class++ID_PATTERN = re.compile(r"[A-Za-z0-9][A-Za-z0-9._:-]{0,63}\Z")+PARAMETER_NAME_PATTERN = re.compile(r"[a-z][a-z0-9_]{0,7}\Z")+MAX_LABEL_CHARS = 120+SLUG_CHARS = 32++RiskCategory = Literal["destructive", "send", "purchase", "close_unsaved"]+RISK_CATEGORIES: tuple[RiskCategory, ...] = ("destructive", "send", "purchase", "close_unsaved")++# Whole-word, case-insensitive label phrases. This guards against accidents;+# it is not an authorization boundary. Localized lists extend these tuples.+# ``close_unsaved`` is matched whether or not the window reports unsaved+# state, because Driver does not expose that state; it is the safe default.+RISK_PHRASES: Mapping[RiskCategory, tuple[str, ...]] = {+ "destructive": (+ "delete", "remove", "erase", "trash", "discard", "clear all", "format", "reset",+ ),+ "send": ("send", "submit", "post", "publish", "share", "reply"),+ "purchase": ("buy", "purchase", "pay", "checkout", "order", "subscribe"),+ "close_unsaved": ("close", "quit", "exit", "don't save", "discard changes"),+}+++def _phrase_pattern(phrase: str) -> re.Pattern[str]:+ return re.compile(r"(?<![a-z0-9])" + re.escape(phrase) + r"(?![a-z0-9])")+++_RISK_PATTERNS = {+ category: tuple(_phrase_pattern(phrase) for phrase in phrases)+ for category, phrases in RISK_PHRASES.items()+}+++def risk_categories(label: str) -> frozenset[str]:+ """Return the risk categories whose phrases occur in ``label``."""+ text = label.lower().replace("’", "'")+ return frozenset(+ category+ for category, patterns in _RISK_PATTERNS.items()+ if any(pattern.search(text) for pattern in patterns)+ )+++def slug(label: str) -> str:+ """Lowercase ASCII, collapse other runs to ``-``, and trim to 32 characters.++ A label without ASCII alphanumerics becomes ``hex8(sha256(label))``.+ """+ lowered = "".join(+ char.lower() if char.isascii() and char.isalnum() else "-" for char in label+ )+ value = re.sub(r"-+", "-", lowered).strip("-")[:SLUG_CHARS].strip("-")+ return value or hashlib.sha256(label.encode("utf-8")).hexdigest()[:8]+++def _hex4(value: Any) -> str:+ encoded = json.dumps(value, ensure_ascii=False, separators=(",", ":"))+ return hashlib.sha256(encoded.encode("utf-8")).hexdigest()[:4]+++def _string(value: Any) -> str | None:+ return value if isinstance(value, str) else None+++def _number(value: Any) -> float | None:+ if isinstance(value, bool) or not isinstance(value, (int, float)):+ return None+ return float(value)+++def _rect(value: Any) -> tuple[float, float, float, float] | None:+ """Return ``(x, y, w, h)`` from a Driver frame or window bounds."""+ if not isinstance(value, Mapping):+ return None+ x = _number(value.get("x"))+ y = _number(value.get("y"))+ w = _number(value.get("w", value.get("width")))+ h = _number(value.get("h", value.get("height")))+ if None in (x, y, w, h):+ return None+ return (x, y, w, h) # type: ignore[return-value]+++def _intersects(a: tuple[float, float, float, float], b: tuple[float, float, float, float]) -> bool:+ ax, ay, aw, ah = a+ bx, by, bw, bh = b+ return min(ax + aw, bx + bw) > max(ax, bx) and min(ay + ah, by + bh) > max(ay, by)+++class NativeObservationError(ValueError):+ def __init__(self, message: str, code: str = "invalid_window_state") -> None:+ super().__init__(message)+ self.code = code+++@dataclass(frozen=True)+class NativeObservation:+ """A validated projection of one ``get_window_state`` result.++ ``complete`` is Driver's ``elements_complete`` claim. ``truncated`` means+ the walk ran out of budget (``truncated`` or a ``truncation_reason``).+ ``tree_empty`` means Driver reported the tree empty (``degraded_reason``+ ``ax_tree_empty``); only then, or for a complete tree, may a view without+ native candidates fall back to visual regions.+ """++ pid: int+ window_id: int+ snapshot_id: str | None+ capture_id: str | None+ window_bounds: tuple[float, float, float, float] | None+ elements: tuple[Mapping[str, Any], ...]+ complete: bool+ truncated: bool+ degraded_reason: str | None+ tree_markdown: str++ @property+ def partial(self) -> bool:+ return not self.complete++ @property+ def tree_empty(self) -> bool:+ return bool(self.degraded_reason and self.degraded_reason.startswith("ax_tree_empty"))++ @classmethod+ def from_window_state(+ cls, payload: Mapping[str, Any], *, expected_pid: int, expected_window_id: int+ ) -> "NativeObservation":+ if payload.get("pid") != expected_pid or payload.get("window_id") != expected_window_id:+ raise NativeObservationError(+ "window state belongs to a different window", code="window_mismatch"+ )+ raw_elements = payload.get("elements", [])+ if not isinstance(raw_elements, list) or not all(+ isinstance(item, Mapping) for item in raw_elements+ ):+ raise NativeObservationError("window state has malformed elements")+ elements = tuple(+ sorted(+ raw_elements,+ key=lambda item: item.get("element_index")+ if isinstance(item.get("element_index"), int)+ else -1,+ )+ )+ degraded = _string(payload.get("degraded_reason")) if payload.get("degraded") else None+ return cls(+ pid=expected_pid,+ window_id=expected_window_id,+ snapshot_id=_string(payload.get("snapshot_id")),+ capture_id=_string(payload.get("capture_id")),+ window_bounds=_rect(payload.get("window_bounds")),+ elements=elements,+ complete=payload.get("elements_complete") is True,+ truncated=payload.get("truncated") is True or bool(payload.get("truncation_reason")),+ degraded_reason=degraded,+ tree_markdown=_string(payload.get("tree_markdown")) or "",+ )+++@dataclass(frozen=True)+class NativeControl:+ """One eligible native element in one observation.++ ``value`` is runner-only (never sent to a model). ``element_token`` is bound+ to the observation's snapshot and is never sent to a model either.+ """++ id: str+ role_class: RoleClass+ action: ActionKind+ label: str+ value: str | None+ selected: bool | None+ element_index: int+ element_token: str+ risk: frozenset[str]+++ExclusionReason = Literal[+ "unknown_role", "disabled", "off_screen", "unlabeled", "web_content", "no_token"+]+++@dataclass(frozen=True)+class NativeControls:+ controls: tuple[NativeControl, ...]+ excluded: Mapping[str, int]+++def eligible_controls(+ observation: NativeObservation,+ platform: Platform,+ *,+ redact: Callable[[str], str] = lambda value: value,+) -> NativeControls:+ """Apply the element rules and assign stable IDs, in ``element_index`` order."""+ by_index = {+ item["element_index"]: item+ for item in observation.elements+ if isinstance(item.get("element_index"), int)+ }+ excluded: dict[str, int] = {}++ def exclude(reason: ExclusionReason) -> None:+ excluded[reason] = excluded.get(reason, 0) + 1++ def label_of(element: Mapping[str, Any]) -> str:+ label = _string(element.get("label")) or ""+ return " ".join(redact(label).split())++ def path_of(element: Mapping[str, Any]) -> tuple[tuple[str, str], ...]:+ path: list[tuple[str, str]] = []+ seen: set[int] = set()+ parent = element.get("parent_index")+ while isinstance(parent, int) and parent in by_index and parent not in seen:+ seen.add(parent)+ ancestor = by_index[parent]+ raw_role = _string(ancestor.get("role")) or ""+ klass = role_class(raw_role, platform) or normalized_role(raw_role)+ path.append((klass, label_of(ancestor)))+ parent = ancestor.get("parent_index")+ return tuple(reversed(path))++ pending: list[tuple[Mapping[str, Any], RoleClass, str, tuple[tuple[str, str], ...]]] = []+ for element in observation.elements:+ klass = role_class(element.get("role"), platform)+ if klass is None:+ exclude("unknown_role")+ continue+ if element.get("enabled") is False:+ exclude("disabled")+ continue+ frame = _rect(element.get("frame"))+ if (+ frame is None+ or frame[2] <= 0+ or frame[3] <= 0+ or observation.window_bounds is None+ or not _intersects(frame, observation.window_bounds)+ ):+ exclude("off_screen")+ continue+ label = label_of(element)+ value = _string(element.get("value"))+ redacted_value = " ".join(redact(value).split()) if value is not None else None+ if element.get("unlabelled") is True or not label or label == redacted_value:+ # macOS and Windows labels can fall back to the value (or an empty+ # field's placeholder); typed content never becomes an ID.+ exclude("unlabeled")+ continue+ if element.get("in_web_content") is True:+ exclude("web_content")+ continue+ if not isinstance(element.get("element_token"), str) or not element["element_token"]:+ exclude("no_token")+ continue+ pending.append((element, klass, label, path_of(element)))++ bases = [f"ax:{klass}:{slug(label)}" for _, klass, label, _ in pending]+ counts: dict[str, int] = {}+ for base in bases:+ counts[base] = counts.get(base, 0) + 1+ ordinals: dict[tuple[Any, ...], int] = {}+ controls: list[NativeControl] = []+ for (element, klass, label, path), base in zip(pending, bases):+ key = (klass, label, path)+ ordinal = ordinals.get(key, 0)+ ordinals[key] = ordinal + 1+ candidate_id = base if counts[base] == 1 else f"{base}:{_hex4([list(map(list, path)), ordinal])}"+ if not ID_PATTERN.fullmatch(candidate_id): # pragma: no cover - slug bounds this+ raise NativeObservationError("derived candidate ID is invalid")+ selected = element.get("selected")+ controls.append(+ NativeControl(+ id=candidate_id,+ role_class=klass,+ action=ROLE_CLASS_ACTION[klass],+ label=label[:MAX_LABEL_CHARS],+ value=_string(element.get("value")),+ selected=selected if isinstance(selected, bool) else None,+ element_index=element["element_index"],+ element_token=element["element_token"],+ risk=risk_categories(label),+ )+ )+ return NativeControls(tuple(controls), excluded)+++def element_state(control: NativeControl) -> str:+ """The compact, value-free state sent to the model for one control."""+ if control.role_class in {"checkbox", "toggle"}:+ return "checked" if control.selected else "unchecked"+ if control.role_class == "radio":+ return "selected" if control.selected else "not_selected"+ if control.role_class == "text_input":+ return "has_text" if control.value else "empty"+ return "enabled"+++def field_state(value: str | None, required: str) -> str:+ """The existing field vocabulary; never includes the value itself."""+ if not value:+ return "empty"+ if value == required:+ return "contains_required_value"+ return "contains_other_value"diff --git a/libs/cua-driver/examples/jev-use/python/native_roles.py b/libs/cua-driver/examples/jev-use/python/native_roles.pynew file mode 100644index 0000000000..c38223c006--- /dev/null+++ b/libs/cua-driver/examples/jev-use/python/native_roles.py@@ -0,0 +1,134 @@+"""Per-platform role table for native accessibility candidates (RFC #4268).++Cua Driver reports each element's raw platform role: macOS AX names+(``AXButton``), Windows UIA ``control_type_name`` values (``Button``), or Linux+AT-SPI role names (``push button``). It does not normalize them across+platforms, so jev-use owns this closed mapping from raw roles to a small set of+role classes. Unknown roles are excluded, never guessed.++Lookups go through ``normalized_role``, a port of Driver's private+``normalized_role`` in ``cua-driver-core/src/expectation.rs``: keep ASCII+alphanumerics, lowercase, strip an ``ax`` prefix, and map ``pushbutton`` to+``button`` and ``pagetab``/``tabitem`` to ``tab``. Because every table is keyed+by that normalized form, two raw roles that Driver's ``verify_state`` treats as+the same role (for example ``AXCheckBox``, ``CheckBox`` and ``check box``) can+never map to different role classes here. The tables stay per platform because+the same normalized name can mean different controls: Linux AT-SPI ``text`` is+an editable text control, while Windows UIA ``Text`` is static text.+"""++from __future__ import annotations++from typing import Literal, Mapping++Platform = Literal["macos", "windows", "linux"]+RoleClass = Literal[+ "button", "toggle", "checkbox", "radio", "popup", "menu_item", "link", "text_input"+]+ActionKind = Literal["press", "toggle", "select", "open_menu", "set_text", "visual_click"]++PLATFORMS: tuple[Platform, ...] = ("macos", "windows", "linux")++ROLE_CLASSES: tuple[RoleClass, ...] = (+ "button",+ "toggle",+ "checkbox",+ "radio",+ "popup",+ "menu_item",+ "link",+ "text_input",+)++# The one action kind each role class offers.+ROLE_CLASS_ACTION: Mapping[RoleClass, ActionKind] = {+ "button": "press",+ "toggle": "toggle",+ "checkbox": "toggle",+ "radio": "select",+ "popup": "open_menu",+ "menu_item": "press",+ "link": "press",+ "text_input": "set_text",+}++ACTION_KINDS: frozenset[ActionKind] = frozenset(+ {"press", "toggle", "select", "open_menu", "set_text", "visual_click"}+)++# Raw platform roles, as Driver reports them, for each role class. The+# normalized tables below are derived from these, so this is the reviewed data.+RAW_ROLES: Mapping[Platform, Mapping[RoleClass, tuple[str, ...]]] = {+ "macos": {+ "button": ("AXButton",),+ "toggle": ("AXSwitch",),+ "checkbox": ("AXCheckBox",),+ "radio": ("AXRadioButton",),+ "popup": ("AXPopUpButton", "AXComboBox", "AXMenuButton"),+ "menu_item": ("AXMenuItem", "AXMenuBarItem"),+ "link": ("AXLink",),+ "text_input": ("AXTextField", "AXTextArea", "AXSearchField", "AXSecureTextField"),+ },+ "windows": {+ "button": ("Button", "SplitButton"),+ "toggle": (),+ "checkbox": ("CheckBox",),+ "radio": ("RadioButton",),+ "popup": ("ComboBox",),+ "menu_item": ("MenuItem",),+ "link": ("Hyperlink",),+ "text_input": ("Edit",),+ },+ "linux": {+ "button": ("push button", "button"),+ "toggle": ("toggle button", "switch"),+ "checkbox": ("check box",),+ "radio": ("radio button",),+ "popup": ("combo box",),+ "menu_item": ("menu item", "check menu item", "radio menu item"),+ "link": ("link",),+ "text_input": ("entry", "text", "password text"),+ },+}+++def normalized_role(role: str) -> str:+ """Port of Driver's ``normalized_role`` (``cua-driver-core/src/expectation.rs``)."""+ normalized = "".join(+ character.lower()+ for character in role+ if character.isascii() and character.isalnum()+ )+ if normalized.startswith("ax"):+ normalized = normalized[2:]+ if normalized == "pushbutton":+ return "button"+ if normalized in {"pagetab", "tabitem"}:+ return "tab"+ return normalized+++def _normalized_table(platform: Platform) -> dict[str, RoleClass]:+ table: dict[str, RoleClass] = {}+ for role_class, raw_roles in RAW_ROLES[platform].items():+ for raw in raw_roles:+ key = normalized_role(raw)+ existing = table.get(key)+ if existing is not None and existing != role_class:+ raise ValueError(+ f"{platform} roles normalized to {key!r} map to both {existing} and {role_class}"+ )+ table[key] = role_class+ return table+++ROLE_TABLES: Mapping[Platform, Mapping[str, RoleClass]] = {+ platform: _normalized_table(platform) for platform in PLATFORMS+}+++def role_class(raw_role: object, platform: Platform) -> RoleClass | None:+ """Return the role class for a raw Driver role, or ``None`` when excluded."""+ if not isinstance(raw_role, str) or not raw_role:+ return None+ return ROLE_TABLES[platform].get(normalized_role(raw_role))diff --git a/libs/cua-driver/examples/jev-use/python/native_tasks.py b/libs/cua-driver/examples/jev-use/python/native_tasks.pynew file mode 100644index 0000000000..839b92bd36--- /dev/null+++ b/libs/cua-driver/examples/jev-use/python/native_tasks.py@@ -0,0 +1,571 @@+"""Native task specs for jev-use (RFC #4268, Phase 1).++A ``NativeTask`` declares one application task: its goal, parameters (text is+taken only from these), the window it may observe and act on, the allowed+action kinds, opt-in risk categories, whether foreground delivery is allowed,+a step budget, and an app-owned success oracle. It builds each step's closed+candidate set from the native accessibility source, plus visual regions only+under the fallback rule, and composes them in the fixed order page, ax, visual.++The built-in tasks drive the repository's AppKit harness+(``libs/cua-driver/tests/fixtures/apps/macos/appkit``) launched with+``CUA_APPKIT_TASK_STATE``. Their oracle is the harness's own JSON state file,+which the app rewrites on every change; it never depends on Driver output.+"""++from __future__ import annotations++import json+from dataclasses import dataclass, field+from pathlib import Path+from typing import Any, Callable, Literal, Mapping++from choose_action import MAX_ELEMENTS, MAX_HISTORY, REQUEST_SCHEMA_V2+from native import PARAMETER_NAME_PATTERN, NativeControl, element_state, field_state, slug+from native_roles import ACTION_KINDS+from sources import Candidate, NativeAccessibilitySource, TextMethod+from tasks import Outcome, TaskParameter, TaskSources, redact_token++MAX_EXECUTABLE_CANDIDATES = 24+SOURCE_ORDER = ("page", "ax", "visual")+Check = Literal["verified", "refuted", "pending"]++NATIVE_RESERVED = (+ Candidate(+ "reobserve",+ "Take no action and obtain a fresh observation of the window, because the current "+ "observation looks stale, incomplete, or contradicts the goal's progress.",+ None,+ {},+ ),+ Candidate(+ "abstain",+ "Stop without acting if none of the proposed actions is safe or moves toward the goal.",+ None,+ {},+ ),+)++_VERBS = {+ "press": "pressed",+ "toggle": "toggled",+ "select": "selected",+ "open_menu": "opened",+ "visual_click": "clicked the visual region",+}+++@dataclass(frozen=True)+class WindowScope:+ """The one window a task may observe and act on."""++ window_title: str+ bundle_id: str | None = None+ process_name: str | None = None+ query: str | None = None+ max_elements: int | None = None+ max_depth: int | None = None++ def window_state_arguments(self) -> dict[str, Any]:+ arguments: dict[str, Any] = {}+ if self.query is not None:+ arguments["query"] = self.query+ if self.max_elements is not None:+ arguments["max_elements"] = self.max_elements+ if self.max_depth is not None:+ arguments["max_depth"] = self.max_depth+ return arguments+++class OracleError(RuntimeError):+ pass+++@dataclass(frozen=True)+class AppStateOracle:+ """An ``app_check`` oracle over an app-owned JSON state file.++ The file must carry the expected ``schema`` and, when given, the ``pid`` of+ the exact app process under test, so a stale file from another run never+ counts as evidence.+ """++ path: Path+ schema: str+ expected_pid: int | None = None++ def read(self) -> dict[str, Any]:+ try:+ state = json.loads(Path(self.path).read_text(encoding="utf-8"))+ except (OSError, json.JSONDecodeError) as error:+ raise OracleError(f"app state is unreadable: {type(error).__name__}") from None+ if not isinstance(state, dict) or state.get("schema") != self.schema:+ raise OracleError("app state has an unexpected schema")+ if self.expected_pid is not None and state.get("pid") != self.expected_pid:+ raise OracleError("app state belongs to a different process")+ return state+++@dataclass(frozen=True)+class ComposeStats:+ """Content-free counts logged for every step."""++ sources: Mapping[str, int]+ duplicates: int+ risk_excluded: Mapping[str, int]+ dropped: int++ def to_log(self) -> dict[str, Any]:+ return {+ "sources": dict(self.sources),+ "duplicates": self.duplicates,+ "risk_excluded": dict(self.risk_excluded),+ "dropped": self.dropped,+ }+++def compose(+ groups: Mapping[str, list[Candidate]],+ *,+ allowed_risks: frozenset[str],+ reserved: tuple[Candidate, ...] = NATIVE_RESERVED,+ cap: int = MAX_EXECUTABLE_CANDIDATES,+) -> tuple[list[Candidate], ComposeStats]:+ """Merge source outputs in the order page, ax, visual.++ Duplicate IDs are dropped deterministically (first source wins). A+ candidate tagged with a risk category the task did not allow is removed.+ At most ``cap`` executable candidates remain, in source then element+ order; the count dropped is reported, never silently truncated. The+ reserved ``reobserve`` and ``abstain`` candidates are always appended.+ """+ seen: set[str] = {candidate.id for candidate in reserved}+ merged: list[Candidate] = []+ duplicates = 0+ risk_excluded: dict[str, int] = {}+ for source in SOURCE_ORDER:+ for candidate in groups.get(source, []):+ if candidate.id in seen:+ duplicates += 1+ continue+ seen.add(candidate.id)+ blocked = sorted(candidate.risk - allowed_risks)+ if blocked:+ for category in blocked:+ risk_excluded[category] = risk_excluded.get(category, 0) + 1+ continue+ merged.append(candidate)+ kept = merged[:cap]+ counts = {source: 0 for source in SOURCE_ORDER}+ for candidate in kept:+ if candidate.source in counts:+ counts[candidate.source] += 1+ stats = ComposeStats(counts, duplicates, risk_excluded, len(merged) - len(kept))+ return kept + list(reserved), stats+++@dataclass(frozen=True)+class NativeStep:+ """One step's closed candidate set and its model-safe context."""++ candidates: list[Candidate]+ stats: ComposeStats+ elements: list[dict[str, str]]+ outcomes: Mapping[str, str]+++def _quoted(label: str, limit: int = 60) -> str:+ return json.dumps(label if len(label) <= limit else label[: limit - 1] + "…", ensure_ascii=False)+++@dataclass(frozen=True)+class NativeTask:+ """A declarative native task. See the module docstring."""++ id: str+ goal: str+ scope: WindowScope+ allowed_actions: frozenset[str]+ oracle: AppStateOracle+ check: Callable[[Mapping[str, Any]], Check]+ parameters: tuple[TaskParameter, ...] = ()+ allowed_risks: frozenset[str] = frozenset()+ allow_foreground: bool = False+ max_steps: int = 6+ text_method: TextMethod = "set_value"+ visual_targets: tuple[str, ...] = ()+ mock_preferences: tuple[str, ...] = ()+ # The oracle is polled after every action, so no candidate is special.+ completion_candidate_ids: frozenset[str] = field(default=frozenset(), init=False)++ def __post_init__(self) -> None:+ unknown = set(self.allowed_actions) - ACTION_KINDS+ if unknown:+ raise ValueError(f"unknown action kinds: {sorted(unknown)}")+ for parameter in self.parameters:+ if not PARAMETER_NAME_PATTERN.fullmatch(parameter.name):+ raise ValueError("parameter names must match [a-z][a-z0-9_]{0,7}")+ if self.max_steps < 1:+ raise ValueError("max_steps must be positive")++ @property+ def allowed_action_kinds(self) -> frozenset[str]:+ """The Driver tools this task can dispatch."""+ tools = set()+ if self.allowed_actions & {"press", "toggle", "select", "open_menu", "visual_click"}:+ tools.add("click")+ if "set_text" in self.allowed_actions:+ tools.add(self.text_method)+ return frozenset(tools)++ def redact(self, value: Any) -> Any:+ for parameter in self.parameters:+ if parameter.secret:+ value = redact_token(value, parameter.value, parameter.redaction)+ return value++ def redact_text(self, value: str) -> str:+ return self.redact(value)++ # -- candidates -------------------------------------------------------++ def _native_candidates(+ self, ax: NativeAccessibilitySource, foreground_ids: frozenset[str]+ ) -> tuple[list[Candidate], dict[str, str]]:+ candidates: list[Candidate] = []+ outcomes: dict[str, str] = {}+ for native in ax.controls:+ if native.action not in self.allowed_actions:+ continue+ control = ax.control(native)+ label = _quoted(native.label)+ if native.action == "set_text":+ for parameter in self.parameters:+ if native.value == parameter.value:+ continue+ candidate_id = f"{native.id}:set:{parameter.name}"+ state = field_state(native.value, parameter.value)+ candidates.append(+ ax.type_text(+ control,+ parameter.value,+ candidate_id=candidate_id,+ description=(+ f"Set the text field {label} to the task parameter "+ f"{json.dumps(parameter.name)}, replacing its contents. The field "+ f"currently reports {state}."+ ),+ )+ )+ outcomes[candidate_id] = f"set {label} to parameter {parameter.name}"+ continue+ if native.action == "select" and native.selected:+ continue # selecting an already selected option is a no-op+ description = self._describe(native, label)+ candidate_id = native.id+ delivery: Literal["background", "foreground"] = "background"+ if native.id in foreground_ids:+ if not self.allow_foreground:+ continue+ candidate_id = f"{native.id}:foreground"+ delivery = "foreground"+ description += (+ " Use foreground delivery, which activates the window, because Driver "+ "refused background delivery for this control."+ )+ candidates.append(+ ax.click(control, candidate_id=candidate_id, description=description, delivery=delivery)+ )+ outcomes[candidate_id] = f"{_VERBS[native.action]} {label}"+ return candidates, outcomes++ @staticmethod+ def _describe(native: NativeControl, label: str) -> str:+ kind = native.role_class+ if native.action == "toggle":+ now = "checked" if native.selected else "unchecked"+ after = "unchecked" if native.selected else "checked"+ noun = "switch" if kind == "toggle" else "checkbox"+ return f"Toggle the {noun} {label}. It is currently {now}; afterward it will be {after}."+ if native.action == "select":+ return f"Select the radio option {label}. It is currently not selected."+ if native.action == "open_menu":+ return (+ f"Open the pop-up menu {label}. Its options become candidates on the next "+ "observation."+ )+ noun = {"menu_item": "menu item", "link": "link"}.get(kind, "button")+ return f"Press the {noun} labeled {label}."++ def _visual_candidates(self, sources: TaskSources) -> tuple[list[Candidate], dict[str, str]]:+ visual = sources.visual+ if visual is None or "visual_click" not in self.allowed_actions:+ return [], {}+ candidates: list[Candidate] = []+ outcomes: dict[str, str] = {}+ for target in self.visual_targets:+ control = visual.find("button", target)+ if control is None:+ continue+ candidate_id = f"visual:{slug(target)}"+ candidate = visual.click(+ control,+ candidate_id=candidate_id,+ description=(+ f"Click the unique validated visual region reading {_quoted(target)}; "+ "no accessibility element covers it."+ ),+ )+ if candidate is not None:+ candidates.append(candidate)+ outcomes[candidate_id] = f"clicked the visual region {_quoted(target)}"+ return candidates, outcomes++ def plan(self, sources: TaskSources) -> NativeStep:+ """Build the step's closed candidate set, stats, and compact elements."""+ groups: dict[str, list[Candidate]] = {}+ outcomes: dict[str, str] = {}+ elements: list[dict[str, str]] = []+ if sources.ax is not None:+ groups["ax"], native_outcomes = self._native_candidates(+ sources.ax, sources.foreground_ids+ )+ outcomes.update(native_outcomes)+ elements = [+ {+ "role_class": control.role_class,+ "label": control.label,+ "state": element_state(control),+ }+ for control in sources.ax.controls[:MAX_ELEMENTS]+ ]+ groups["visual"], visual_outcomes = self._visual_candidates(sources)+ outcomes.update(visual_outcomes)+ candidates, stats = compose(groups, allowed_risks=self.allowed_risks)+ for candidate in candidates:+ if candidate.tool is not None and candidate.tool not in self.allowed_action_kinds:+ raise ValueError(f"task {self.id} does not allow action kind {candidate.tool}")+ return NativeStep(candidates, stats, elements, outcomes)++ def candidates(self, sources: TaskSources) -> list[Candidate]:+ return self.plan(sources).candidates++ def state_summary(self, sources: TaskSources) -> dict[str, str]:+ """Value-free control states, keyed by stable candidate ID."""+ if sources.ax is None:+ return {}+ return {control.id: element_state(control) for control in sources.ax.controls}++ def history_entry(+ self,+ step: int,+ candidate_id: str,+ *,+ refusal: str | None = None,+ outcome: str | None = None,+ stale: bool = False,+ ) -> dict[str, Any]:+ if stale:+ text = "the observation was stale; nothing happened and the window is observed again"+ elif refusal is not None:+ text = f"Driver refused background delivery ({refusal}); nothing happened"+ elif candidate_id == "reobserve":+ text = "took no action and requested a fresh observation"+ else:+ text = outcome or "completed"+ return {"step": step, "selected_id": candidate_id, "outcome": self.redact(text)[:128]}++ def reset(self) -> None:+ """The harness starts fresh for every run; there is nothing to reset."""++ def read_oracle(self) -> Mapping[str, Any]:+ return self.oracle.read()++ def classify(self, oracle: Mapping[str, Any], *, steps: int) -> Outcome:+ result = self.check(oracle)+ if result in {"verified", "refuted"}:+ return result # type: ignore[return-value]+ if steps >= self.max_steps:+ return "budget_exhausted"+ return "unknown"+++def visual_fallback_reason(sources: TaskSources, task: NativeTask, native_count: int) -> str | None:+ """Return why this step may parse visual regions, or ``None``.++ Visual regions are consulted only when the task allows ``visual_click`` and+ declares visual targets, and either Driver reported the tree empty, or a+ complete tree has no native candidate (or no native control labeled like a+ declared target). A truncated or merely partial tree never qualifies.+ """+ ax = sources.ax+ if ax is None or "visual_click" not in task.allowed_actions or not task.visual_targets:+ return None+ observation = ax.observation+ if observation.capture_id is None:+ return None+ if observation.tree_empty:+ return "tree_empty"+ if not observation.complete or observation.truncated:+ return None+ if native_count == 0:+ return "no_native_candidates"+ labels = {control.label.lower() for control in ax.controls}+ if any(target.lower() not in labels for target in task.visual_targets):+ return "target_without_element"+ return None+++def visual_regions_wire(sources: TaskSources, task: NativeTask) -> list[dict[str, Any]]:+ if sources.visual is None:+ return []+ return [+ task.redact(+ {+ "id": region.id,+ "kind": region.kind,+ "bounds": {+ "x": region.x,+ "y": region.y,+ "width": region.width,+ "height": region.height,+ },+ "text": region.text,+ "label": region.label,+ "confidence": region.confidence,+ "interactive": region.interactive,+ }+ )+ for region in sources.visual.observation.regions+ ]+++def native_choice_request(+ task: NativeTask,+ sources: TaskSources,+ step: NativeStep,+ history: list[Mapping[str, Any]],+) -> dict[str, Any]:+ """Build the ``cua.jev_choice_request_v2`` the provider receives.++ It carries candidate IDs, descriptions and sources, compact value-free+ elements, and compact history. Element tokens, values, and pixels stay in+ the runner.+ """+ if sources.ax is None or sources.ax.observation.capture_id is None:+ raise ValueError("a native request needs an observation with a capture_id")+ observation = sources.ax.observation+ candidates = []+ for candidate in step.candidates:+ item: dict[str, Any] = {"id": candidate.id, "description": task.redact(candidate.description)}+ if candidate.source is not None:+ item["source"] = candidate.source+ candidates.append(item)+ return {+ "schema": REQUEST_SCHEMA_V2,+ "goal": task.redact(task.goal),+ "capture_id": observation.capture_id,+ "snapshot_id": observation.snapshot_id,+ "regions": visual_regions_wire(sources, task),+ "elements": [task.redact(item) for item in step.elements],+ "history": [+ {"selected_id": item["selected_id"], "outcome": item["outcome"]}+ for item in history[-MAX_HISTORY:]+ ],+ "candidates": candidates,+ }+++# -- AppKit harness tasks ---------------------------------------------------++APPKIT_WINDOW_TITLE = "CuaTestHarness AppKit"+APPKIT_BUNDLE_ID = "com.trycua.harness.appkit"+APPKIT_STATE_SCHEMA = "cua.appkit_task_state_v1"+APPKIT_STATE_ENV = "CUA_APPKIT_TASK_STATE"+COUNTER_TARGET = 3+DEFAULT_NOTE_TEXT = "jev-use native note"+TARGET_SIZE = "large"+++def _counter_check(state: Mapping[str, Any]) -> Check:+ counter = state.get("counter")+ if counter == COUNTER_TARGET:+ return "verified"+ if isinstance(counter, int) and counter > COUNTER_TARGET:+ return "refuted" # Reset is a destructive control this task never offers+ return "pending"+++def _note_check(note: str) -> Callable[[Mapping[str, Any]], Check]:+ def check(state: Mapping[str, Any]) -> Check:+ saved = state.get("note_saved")+ if saved == note:+ return "verified"+ if saved is not None:+ return "refuted"+ return "pending"++ return check+++def _size_check(state: Mapping[str, Any]) -> Check:+ if state.get("size") == TARGET_SIZE and state.get("agreed") is True:+ return "verified"+ return "pending"+++APPKIT_TASK_IDS = ("appkit-counter", "appkit-save-note", "appkit-choose-size")+++def appkit_task(+ task_id: str,+ state_path: Path,+ *,+ pid: int | None = None,+ note_text: str = DEFAULT_NOTE_TEXT,+ allow_foreground: bool = False,+) -> NativeTask:+ """Build one AppKit harness task bound to its app-owned state file."""+ oracle = AppStateOracle(Path(state_path), APPKIT_STATE_SCHEMA, pid)+ scope = WindowScope(APPKIT_WINDOW_TITLE, bundle_id=APPKIT_BUNDLE_ID)+ if task_id == "appkit-counter":+ return NativeTask(+ id=task_id,+ goal=(+ f"The counter starts at 0. Set it to exactly {COUNTER_TARGET} by pressing "+ "Increment once per step, then stop."+ ),+ scope=scope,+ allowed_actions=frozenset({"press"}),+ oracle=oracle,+ check=_counter_check,+ allow_foreground=allow_foreground,+ max_steps=COUNTER_TARGET + 3,+ mock_preferences=("ax:button:increment",),+ )+ if task_id == "appkit-save-note":+ return NativeTask(+ id=task_id,+ goal="Enter the note text into the Note field, then save the note.",+ scope=scope,+ allowed_actions=frozenset({"press", "set_text"}),+ oracle=oracle,+ check=_note_check(note_text),+ parameters=(TaskParameter("note", note_text, secret=True, redaction="[note text]"),),+ allow_foreground=allow_foreground,+ max_steps=5,+ mock_preferences=("ax:text_input:note:set:note", "ax:button:save-note"),+ )+ if task_id == "appkit-choose-size":+ return NativeTask(+ id=task_id,+ goal="Choose the Large size option and check the I agree checkbox.",+ scope=scope,+ allowed_actions=frozenset({"select", "toggle"}),+ oracle=oracle,+ check=_size_check,+ allow_foreground=allow_foreground,+ max_steps=5,+ mock_preferences=("ax:radio:large", "ax:checkbox:i-agree"),+ )+ raise ValueError(f"unknown AppKit task: {task_id}")diff --git a/libs/cua-driver/examples/jev-use/python/run.py b/libs/cua-driver/examples/jev-use/python/run.pyindex 66f4edd454..9aae4e65ff 100644--- a/libs/cua-driver/examples/jev-use/python/run.py+++ b/libs/cua-driver/examples/jev-use/python/run.py@@ -85,6 +85,9 @@ class Driver:structured = getattr(result, "structuredContent", None)structured = structured if isinstance(structured, dict) else {}code = structured.get("code")+ refusal = structured.get("refusal")+ if not code and isinstance(refusal, dict):+ code = refusal.get("code")escalation = structured.get("escalation")recommended = escalation.get("recommended") if isinstance(escalation, dict) else Noneraise DriverToolError(@@ -96,7 +99,13 @@ class Driver:if not isinstance(data, dict):raise RuntimeError(f"{name} returned no structured result")if data.get("status") == "refused" or data.get("refusal"):- raise RuntimeError(f"{name} refused: {data.get('refusal', data)}")+ refusal = data.get("refusal")+ code = refusal.get("code") if isinstance(refusal, dict) else None+ # DriverToolError is a RuntimeError, so existing handlers still match.+ raise DriverToolError(+ f"{name} refused: {data.get('refusal', data)}",+ code if isinstance(code, str) and code else None,+ )return datadiff --git a/libs/cua-driver/examples/jev-use/python/run_native.py b/libs/cua-driver/examples/jev-use/python/run_native.pynew file mode 100644index 0000000000..23bf42042b--- /dev/null+++ b/libs/cua-driver/examples/jev-use/python/run_native.py@@ -0,0 +1,383 @@+"""Run one native jev-use task against a desktop window (RFC #4268, Phase 1).++The runner attaches to an already running application (``--pid``), finds the+task's window by title, and loops: read the app-owned oracle, observe the+window once with ``get_window_state`` (tree and screenshot together), build+the closed candidate set, send a validated ``cua.jev_choice_request_v2`` to+the chooser, dispatch the one selected element-bound action, and read the+oracle again. The chooser only ever selects a supplied ID.++Built-in tasks drive the AppKit harness launched with+``CUA_APPKIT_TASK_STATE=<path>``; ``verify_native.py`` launches it, runs this+runner, and checks the state file independently.+"""++from __future__ import annotations++import argparse+import asyncio+import json+import os+import sys+import time+import uuid+from pathlib import Path+from typing import Any++from mcp import ClientSession, StdioServerParameters+from mcp.client.stdio import stdio_client++from choose_action import validate_request+from core import Candidate, VisualObservationError, parse_visual_regions, validate_choice+from decision_models import DecisionRequest, TypeSafeDecisionModel, choose+from driver_env import driver_environment+from jev_adapter import choose_mock_for_task+from native import NativeObservation, NativeObservationError+from native_roles import Platform+from native_tasks import (+ APPKIT_TASK_IDS,+ NativeTask,+ OracleError,+ appkit_task,+ native_choice_request,+ visual_fallback_reason,+)+from run import Driver, DriverToolError, background_refusal_code, supports_capture_bound_click+from sources import NativeAccessibilitySource, VisualRegionSource+from tasks import TaskSources++STALE_TOKEN_CODES = frozenset({"stale_element_token"})+# A partial or truncated tree is observed once more with this walk budget.+REOBSERVE_TIMEOUT_MS = 5_000+++def host_platform() -> Platform:+ if sys.platform == "darwin":+ return "macos"+ if sys.platform.startswith("win"):+ return "windows"+ return "linux"+++def is_stale_token_error(error: BaseException) -> bool:+ if isinstance(error, DriverToolError) and error.code in STALE_TOKEN_CODES:+ return True+ return "element_token is stale" in str(error)+++def write_event(log_path: Path | None, event: dict[str, Any]) -> None:+ line = json.dumps(event, sort_keys=True)+ print(line)+ if log_path:+ with log_path.open("a", encoding="utf-8") as stream:+ stream.write(line + "\n")+++async def find_window(driver: Driver, pid: int, title: str) -> dict[str, Any]:+ for _ in range(40):+ windows = (await driver.call("list_windows", {"pid": pid})).get("windows", [])+ matches = [+ window+ for window in windows+ if window.get("title") == title and window.get("is_on_screen") is not False+ ]+ if matches:+ return matches[0]+ await asyncio.sleep(0.25)+ raise RuntimeError(f"window {title!r} of pid {pid} did not appear")+++async def observe(+ driver: Driver, task: NativeTask, pid: int, window_id: int, timeout_ms: int | None = None+) -> NativeObservation:+ arguments: dict[str, Any] = {+ "pid": pid,+ "window_id": window_id,+ "include_accessibility_tree": True,+ "include_screenshot": True,+ **task.scope.window_state_arguments(),+ }+ if timeout_ms is not None:+ arguments["timeout_ms"] = timeout_ms+ payload = await driver.call("get_window_state", arguments)+ return NativeObservation.from_window_state(+ payload, expected_pid=pid, expected_window_id=window_id+ )+++async def observe_step(+ driver: Driver,+ task: NativeTask,+ pid: int,+ window_id: int,+ platform: Platform,+ foreground_ids: frozenset[str],+) -> tuple[TaskSources, dict[str, Any]]:+ """Take one observation, reobserving once with a larger walk budget if needed.++ A truncated tree, or a partial tree with no native candidate, is observed+ once more with ``REOBSERVE_TIMEOUT_MS``. The newer observation replaces the+ older one entirely, so every token in the step comes from one snapshot.+ """+ started = time.perf_counter()+ record: dict[str, Any] = {"reobserved": False}++ def build(observation: NativeObservation) -> TaskSources:+ ax = NativeAccessibilitySource.from_observation(+ observation, platform, redact=task.redact_text, text_method=task.text_method+ )+ return TaskSources(ax=ax, foreground_ids=foreground_ids)++ sources = build(await observe(driver, task, pid, window_id))+ observation = sources.ax.observation # type: ignore[union-attr]+ native_count = sum(+ 1 for candidate in task.plan(sources).candidates if candidate.source == "ax"+ )+ if observation.truncated or (observation.partial and native_count == 0):+ record = {"reobserved": True, "reason": "truncated" if observation.truncated else "partial_empty"}+ sources = build(+ await observe(driver, task, pid, window_id, timeout_ms=REOBSERVE_TIMEOUT_MS)+ )+ observation = sources.ax.observation # type: ignore[union-attr]+ record.update(+ {+ "snapshot_id": observation.snapshot_id,+ "capture_id": observation.capture_id,+ "elements_complete": observation.complete,+ "truncated": observation.truncated,+ "controls": len(sources.ax.controls), # type: ignore[union-attr]+ "excluded": dict(sources.ax.native.excluded), # type: ignore[union-attr]+ "observe_ms": round((time.perf_counter() - started) * 1000, 2),+ }+ )+ return sources, record+++async def maybe_visual(+ driver: Driver,+ task: NativeTask,+ sources: TaskSources,+ available_tools: set[str],+ capture_bound_click: bool,+) -> tuple[TaskSources, dict[str, Any]]:+ """Parse visual regions from the same capture, only under the fallback rule."""+ native_count = sum(1 for c in task.plan(sources).candidates if c.source == "ax")+ reason = visual_fallback_reason(sources, task, native_count)+ if reason is None:+ return sources, {"status": "skipped"}+ if not capture_bound_click or "parse_visual_regions" not in available_tools:+ return sources, {"status": "unavailable", "reason": reason}+ observation = sources.ax.observation # type: ignore[union-attr]+ try:+ result = await driver.call(+ "parse_visual_regions",+ {+ "capture_id": observation.capture_id,+ "options": {"kinds": ["text", "icon"], "min_confidence": 0.8, "max_regions": 100},+ },+ )+ visual = parse_visual_regions(+ result,+ expected_capture_id=observation.capture_id or "",+ expected_pid=observation.pid,+ expected_window_id=observation.window_id,+ )+ except DriverToolError as error:+ return sources, {"status": "error", "reason": reason, "error_code": error.code or "driver_error"}+ except VisualObservationError as error:+ return sources, {"status": "error", "reason": reason, "error_code": error.code}+ return (+ TaskSources(+ ax=sources.ax,+ visual=VisualRegionSource(visual, "background", capture_bound_click),+ visual_path=True,+ foreground_ids=sources.foreground_ids,+ ),+ {"status": "ok", "reason": reason, "region_count": len(visual.regions)},+ )+++def choose_live(request: dict[str, Any]) -> tuple[str | None, float, dict[str, float]]:+ from typesafe_sdk import TypeSafeClient++ with TypeSafeClient() as client:+ result = choose(+ TypeSafeDecisionModel(client), DecisionRequest.from_validated(validate_request(request))+ )+ if result.kind == "error":+ raise RuntimeError(f"provider decision failed: {result.reason}")+ return result.selected_id, result.confidence or 0.0, dict(result.probabilities)+++def assert_in_scope(candidate: Candidate, pid: int, window_id: int) -> None:+ """Refuse any action addressed outside the task's one window."""+ if candidate.tool is None:+ return+ if candidate.arguments.get("pid") != pid or candidate.arguments.get("window_id") != window_id:+ raise RuntimeError("candidate addresses a window outside the task scope")+++async def poll_oracle(task: NativeTask, steps: int) -> str:+ outcome = "unknown"+ for _ in range(20):+ outcome = task.classify(task.read_oracle(), steps=steps)+ if outcome in {"verified", "refuted"}:+ return outcome+ await asyncio.sleep(0.1)+ return outcome+++async def run_task(args: argparse.Namespace, task: NativeTask) -> str:+ platform: Platform = args.platform or host_platform()+ log_path = Path(args.log) if args.log else None+ if log_path:+ log_path.write_text("", encoding="utf-8")+ history: list[dict[str, Any]] = []+ foreground_ids: set[str] = set()+ label = f"jev-native-python-{uuid.uuid4().hex[:8]}"+ params = StdioServerParameters(+ command=os.getenv("CUA_DRIVER_BIN", "cua-driver"), args=["mcp"], env=driver_environment()+ )+ async with stdio_client(params) as (read, write):+ async with ClientSession(read, write) as session:+ await session.initialize()+ advertised = (await session.list_tools()).tools+ available_tools = {tool.name for tool in advertised}+ capture_bound_click = supports_capture_bound_click(advertised)+ driver = Driver(session, label)+ window = await find_window(driver, args.pid, task.scope.window_title)+ window_id = int(window["window_id"])+ write_event(+ log_path,+ {"event": "start", "task": task.id, "language": "python", "provider": args.provider,+ "platform": platform, "pid": args.pid, "window_id": window_id},+ )+ for step in range(1, task.max_steps + 1):+ current = task.classify(task.read_oracle(), steps=step - 1)+ if current in {"verified", "refuted"}:+ write_event(log_path, {"event": "outcome", "outcome": current, "step": step - 1})+ return current++ sources, observed = await observe_step(+ driver, task, args.pid, window_id, platform, frozenset(foreground_ids)+ )+ sources, visual_record = await maybe_visual(+ driver, task, sources, available_tools, capture_bound_click+ )+ plan = task.plan(sources)+ request = native_choice_request(task, sources, plan, history)+ validate_request(request)++ decide_started = time.perf_counter()+ if args.provider == "mock":+ choice, confidence, probabilities = choose_mock_for_task(+ task, sources, plan.candidates, history+ )+ else:+ choice, confidence, probabilities = await asyncio.to_thread(choose_live, request)+ decide_ms = round((time.perf_counter() - decide_started) * 1000, 2)+ base_event = {+ "event": "step",+ "step": step,+ "observation": observed,+ "visual": visual_record,+ "compose": plan.stats.to_log(),+ "candidate_count": len(plan.candidates),+ "schema": request["schema"],+ "confidence": confidence,+ "probabilities": probabilities,+ "decide_ms": decide_ms,+ }+ if choice is None:+ write_event(log_path, {**base_event, "event": "outcome", "outcome": "abstained"})+ return "abstained"+ visual = sources.visual.observation if sources.visual is not None else None+ candidate = validate_choice(+ choice, plan.candidates, current_capture_id=visual.capture_id if visual else None+ )+ assert_in_scope(candidate, args.pid, window_id)+ base_event["candidate"] = candidate.id+ base_event["source"] = candidate.source++ if candidate.id == "reobserve":+ history.append(task.history_entry(step, candidate.id))+ write_event(log_path, {**base_event, "tool": None, "act_ms": 0.0})+ continue+ if candidate.id == "abstain":+ write_event(log_path, {**base_event, "event": "outcome", "outcome": "abstained"})+ return "abstained"++ act_started = time.perf_counter()+ try:+ assert candidate.tool is not None+ await driver.call(candidate.tool, dict(candidate.arguments))+ except Exception as error:+ act_ms = round((time.perf_counter() - act_started) * 1000, 2)+ if is_stale_token_error(error):+ # Nothing else is dispatched; the next step observes again.+ history.append(task.history_entry(step, candidate.id, stale=True))+ write_event(log_path, {**base_event, "tool": candidate.tool,+ "act_ms": act_ms, "action_error": "stale_element_token"})+ continue+ refusal = background_refusal_code(candidate, error)+ if refusal is not None:+ foreground_ids.add(candidate.id)+ history.append(task.history_entry(step, candidate.id, refusal=refusal))+ write_event(log_path, {**base_event, "tool": candidate.tool, "act_ms": act_ms,+ "action_error": refusal,+ "escalation": {"from": "background", "to": "foreground",+ "allowed": task.allow_foreground}})+ continue+ write_event(log_path, {**base_event, "event": "outcome", "outcome": "unknown",+ "phase": "action", "error": type(error).__name__,+ "tool": candidate.tool})+ return "unknown"+ act_ms = round((time.perf_counter() - act_started) * 1000, 2)+ history.append(+ task.history_entry(step, candidate.id, outcome=plan.outcomes.get(candidate.id))+ )+ write_event(log_path, {**base_event, "tool": candidate.tool, "act_ms": act_ms,+ "delivery_mode": candidate.arguments.get("delivery_mode")})+ outcome = await poll_oracle(task, step)+ if outcome in {"verified", "refuted"}:+ write_event(log_path, {"event": "outcome", "outcome": outcome, "step": step})+ return outcome++ outcome = task.classify(task.read_oracle(), steps=task.max_steps)+ write_event(log_path, {"event": "outcome", "outcome": outcome, "step": task.max_steps})+ return outcome+++def parse_args(argv: list[str] | None = None) -> argparse.Namespace:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--task", choices=APPKIT_TASK_IDS, required=True)+ parser.add_argument("--provider", choices=("mock", "live"), default="mock")+ parser.add_argument("--pid", type=int, required=True, help="the running harness process")+ parser.add_argument("--state-file", required=True, help="the harness CUA_APPKIT_TASK_STATE path")+ parser.add_argument("--note-text", help="note text for appkit-save-note")+ parser.add_argument("--allow-foreground", action="store_true")+ parser.add_argument("--platform", choices=("macos", "windows", "linux"))+ parser.add_argument("--log", help="optional JSONL output path")+ return parser.parse_args(argv)+++def task_from_args(args: argparse.Namespace) -> NativeTask:+ options: dict[str, Any] = {"pid": args.pid, "allow_foreground": args.allow_foreground}+ if args.note_text:+ options["note_text"] = args.note_text+ return appkit_task(args.task, Path(args.state_file), **options)+++def main() -> None:+ args = parse_args()+ try:+ outcome = asyncio.run(run_task(args, task_from_args(args)))+ except (OracleError, NativeObservationError) as error:+ write_event(Path(args.log) if args.log else None,+ {"event": "outcome", "outcome": "unknown", "error": type(error).__name__})+ raise SystemExit(1) from None+ raise SystemExit(0 if outcome == "verified" else 1)+++if __name__ == "__main__":+ main()diff --git a/libs/cua-driver/examples/jev-use/python/sources.py b/libs/cua-driver/examples/jev-use/python/sources.pyindex 3fa2531bbb..426e382a3b 100644--- a/libs/cua-driver/examples/jev-use/python/sources.py+++ b/libs/cua-driver/examples/jev-use/python/sources.py@@ -13,12 +13,9 @@ Implemented sources:snapshot and acts through ``browser_type`` / ``browser_click`` on its refs.- ``VisualRegionSource`` reads a validated ``parse_visual_regions`` result andacts through a capture-bound ``click`` on a region's center.--Extension point (RFC #4268, Phase 1): a ``NativeAccessibilitySource`` will read-``get_window_state`` elements, find controls by normalized role and label, and-act through the snapshot's ``element_token``. It is deliberately not-implemented here; it must satisfy ``CandidateSource`` and report-``kind = "ax"``.+- ``NativeAccessibilitySource`` (RFC #4268) reads the eligible native elements+ of one ``get_window_state`` observation (see ``native.py``), finds controls+ by role class and label, and acts through that snapshot's ``element_token``."""from __future__ import annotations@@ -27,11 +24,14 @@ from dataclasses import dataclassfrom types import MappingProxyTypefrom typing import TYPE_CHECKING, Any, ClassVar, Literal, Mapping, Protocol+from native import NativeControl, NativeControls, NativeObservation, eligible_controls+from native_roles import Platform+if TYPE_CHECKING:from core import VisualDelivery, VisualObservation, VisualRegion-# "ax" is reserved for the future NativeAccessibilitySource.SourceKind = Literal["page", "visual", "ax"]+TextMethod = Literal["set_value", "type_text"]def _freeze(value: Any) -> Any:@@ -54,9 +54,15 @@ class Candidate:arguments: Mapping[str, Any]capture_id: str | None = Nonescreenshot_reference: str | None = None+ # RFC #4268: the source that built the candidate (None for reserved+ # candidates), the native snapshot it is bound to, and its risk categories.+ source: SourceKind | None = None+ snapshot_id: str | None = None+ risk: frozenset[str] = frozenset()def __post_init__(self) -> None:object.__setattr__(self, "arguments", _freeze(dict(self.arguments)))+ object.__setattr__(self, "risk", frozenset(self.risk))@dataclass(frozen=True)@@ -126,6 +132,7 @@ class BrowserSemanticSource:description,"browser_click",{**self.require_target(), "ref": control.handle["ref"], "input_route": "dom_event"},+ source="page",)def type_text(@@ -136,6 +143,7 @@ class BrowserSemanticSource:description,"browser_type",{**self.require_target(), "ref": control.handle["ref"], "text": text, "replace": True},+ source="page",)@@ -189,9 +197,98 @@ class VisualRegionSource:},capture_id=visual.capture_id,screenshot_reference=visual.screenshot_reference,+ source="visual",)def type_text(self, control: Control, text: str, *, candidate_id: str, description: str) -> None:return None+++@dataclass(frozen=True)+class NativeAccessibilitySource:+ """Controls from one ``get_window_state`` observation, acted on by token.++ ``controls`` holds the eligible elements in ``element_index`` order with+ stable IDs (see ``native.py``). ``click`` binds the control's+ ``element_token`` with ``delivery_mode`` (background unless Driver refused+ background delivery for that control and the task allows foreground).+ ``type_text`` sets text through ``set_value`` (default) or an element-bound+ ``type_text``, with the text supplied by the task, never by the model.+ """++ observation: NativeObservation+ platform: Platform+ native: NativeControls+ text_method: TextMethod = "set_value"+ kind: ClassVar[SourceKind] = "ax"++ @classmethod+ def from_observation(+ cls,+ observation: NativeObservation,+ platform: Platform,+ *,+ redact: Any = None,+ text_method: TextMethod = "set_value",+ ) -> "NativeAccessibilitySource":+ native = (+ eligible_controls(observation, platform)+ if redact is None+ else eligible_controls(observation, platform, redact=redact)+ )+ return cls(observation, platform, native, text_method)++ @property+ def controls(self) -> tuple[NativeControl, ...]:+ return self.native.controls++ def control(self, native: NativeControl) -> Control:+ return Control("ax", native.role_class, native.label, native.value, native)++ def find(self, role: str, name: str) -> Control | None:+ matches = [+ item for item in self.controls if item.role_class == role and item.label == name+ ]+ return self.control(matches[0]) if len(matches) == 1 else None++ def _target(self) -> dict[str, Any]:+ return {"pid": self.observation.pid, "window_id": self.observation.window_id}++ def click(+ self,+ control: Control,+ *,+ candidate_id: str,+ description: str,+ delivery: VisualDelivery = "background",+ ) -> Candidate:+ native: NativeControl = control.handle+ return Candidate(+ candidate_id,+ description,+ "click",+ {**self._target(), "element_token": native.element_token, "delivery_mode": delivery},+ source="ax",+ snapshot_id=self.observation.snapshot_id,+ risk=native.risk,+ )++ def type_text(+ self, control: Control, text: str, *, candidate_id: str, description: str+ ) -> Candidate:+ native: NativeControl = control.handle+ if self.text_method == "type_text":+ tool, arguments = "type_text", {"text": text}+ else:+ tool, arguments = "set_value", {"value": text}+ return Candidate(+ candidate_id,+ description,+ tool,+ {**self._target(), "element_token": native.element_token, **arguments},+ source="ax",+ snapshot_id=self.observation.snapshot_id,+ risk=native.risk,+ )diff --git a/libs/cua-driver/examples/jev-use/python/tasks.py b/libs/cua-driver/examples/jev-use/python/tasks.pyindex 5cb1bbe923..7878df5b23 100644--- a/libs/cua-driver/examples/jev-use/python/tasks.py+++ b/libs/cua-driver/examples/jev-use/python/tasks.py@@ -23,7 +23,7 @@ from dataclasses import dataclass, fieldfrom typing import TYPE_CHECKING, Any, Literal, Mapping, Protocolfrom urllib.request import Request, urlopen-from sources import BrowserSemanticSource, Candidate, VisualRegionSource+from sources import BrowserSemanticSource, Candidate, NativeAccessibilitySource, VisualRegionSourceif TYPE_CHECKING:from core import VisualDelivery, VisualObservation, VisualRegion@@ -64,11 +64,22 @@ class TaskSources:``visual`` is present only when this step parsed visual regions.``visual_path`` reports whether a capture-bound visual path exists at all,so a task can say a control is still pending a visual check.++ ``page`` is present for browser tasks. ``ax`` is present for native tasks+ (RFC #4268); ``foreground_ids`` names the native candidate IDs for which+ Driver refused background delivery on an earlier step."""- page: BrowserSemanticSource+ page: BrowserSemanticSource | None = Nonevisual: VisualRegionSource | None = Nonevisual_path: bool = False+ ax: NativeAccessibilitySource | None = None+ foreground_ids: frozenset[str] = frozenset()++ def require_page(self) -> BrowserSemanticSource:+ if self.page is None:+ raise ValueError("this task needs a browser page source")+ return self.pageclass Task(Protocol):@@ -215,7 +226,7 @@ class FixtureFormTask:delivery yields a distinct ``submit-form-foreground`` candidate afterDriver refused background delivery; the chooser must pick it explicitly."""- page = sources.page+ page = sources.require_page()# Every page candidate addresses the snapshot's target; reject a snapshot# without one before offering anything.page.require_target()@@ -283,8 +294,9 @@ class FixtureFormTask:action remains), and ``not_found_visually`` otherwise. Without a visual pathit is ``not_in_page_structure``."""- field_control = sources.page.find("textbox", FIELD_NAME)- button = sources.page.find("button", SUBMIT_NAME)+ page = sources.require_page()+ field_control = page.find("textbox", FIELD_NAME)+ button = page.find("button", SUBMIT_NAME)if field_control is None:field_state = "not_found"elif not field_control.value:diff --git a/libs/cua-driver/examples/jev-use/python/tests/test_native_contract.py b/libs/cua-driver/examples/jev-use/python/tests/test_native_contract.pynew file mode 100644index 0000000000..1ff6047edf--- /dev/null+++ b/libs/cua-driver/examples/jev-use/python/tests/test_native_contract.py@@ -0,0 +1,278 @@+"""Role table and cua.jev_choice_request_v2 contract tests (RFC #4268)."""++from __future__ import annotations++import copy+import json+import subprocess+import sys+import unittest+from pathlib import Path++sys.path.insert(0, str(Path(__file__).resolve().parents[1]))++from choose_action import provider_observation, validate_request+from choose_decision import MockDecisionModel+from decision_models import (+ DecisionRequest,+ S1DecisionModel,+ TypeSafeDecisionModel,+ choose,+ native_elements_as_text,+)+from native_roles import (+ PLATFORMS,+ RAW_ROLES,+ ROLE_CLASS_ACTION,+ ROLE_CLASSES,+ ROLE_TABLES,+ normalized_role,+ role_class,+)++ROOT = Path(__file__).resolve().parents[2]+++def v2_request() -> dict:+ return json.loads((ROOT / "fixtures/jev-choice-request-v2.json").read_text(encoding="utf-8"))+++def v1_request() -> dict:+ return json.loads((ROOT / "fixtures/jev-choice-request-v1.json").read_text(encoding="utf-8"))+++class RoleTableTest(unittest.TestCase):+ def test_normalized_role_matches_driver_cases(self) -> None:+ # The cases Driver's own expectation.rs normalizer documents.+ cases = {+ "AXCheckBox": "checkbox",+ "CheckBox": "checkbox",+ "check box": "checkbox",+ "push button": "button",+ "AXButton": "button",+ "page tab": "tab",+ "TabItem": "tab",+ "AXTextField": "textfield",+ "combo box": "combobox",+ }+ for raw, expected in cases.items():+ with self.subTest(raw=raw):+ self.assertEqual(normalized_role(raw), expected)++ def test_each_platform_maps_its_raw_roles(self) -> None:+ expectations = {+ "macos": {+ "AXButton": "button",+ "AXCheckBox": "checkbox",+ "AXRadioButton": "radio",+ "AXPopUpButton": "popup",+ "AXComboBox": "popup",+ "AXMenuItem": "menu_item",+ "AXMenuBarItem": "menu_item",+ "AXLink": "link",+ "AXTextField": "text_input",+ "AXTextArea": "text_input",+ "AXSecureTextField": "text_input",+ "AXSwitch": "toggle",+ },+ "windows": {+ "Button": "button",+ "SplitButton": "button",+ "CheckBox": "checkbox",+ "RadioButton": "radio",+ "ComboBox": "popup",+ "MenuItem": "menu_item",+ "Hyperlink": "link",+ "Edit": "text_input",+ },+ "linux": {+ "push button": "button",+ "toggle button": "toggle",+ "check box": "checkbox",+ "radio button": "radio",+ "combo box": "popup",+ "menu item": "menu_item",+ "check menu item": "menu_item",+ "link": "link",+ "entry": "text_input",+ "text": "text_input",+ "password text": "text_input",+ },+ }+ for platform, cases in expectations.items():+ for raw, expected in cases.items():+ with self.subTest(platform=platform, raw=raw):+ self.assertEqual(role_class(raw, platform), expected)++ def test_unknown_and_platform_specific_roles_are_excluded(self) -> None:+ excluded = {+ "macos": ["AXStaticText", "AXSlider", "AXWindow", "AXMenuBar", "AXMenu", "AXGroup", ""],+ # UIA Text is static text on Windows, unlike AT-SPI text on Linux.+ "windows": ["Text", "Slider", "TabItem", "Pane", "Window", "Unknown"],+ "linux": ["label", "slider", "page tab", "frame", "filler"],+ }+ for platform, roles in excluded.items():+ for raw in roles:+ with self.subTest(platform=platform, raw=raw):+ self.assertIsNone(role_class(raw, platform))+ self.assertIsNone(role_class(None, "macos")) # type: ignore[arg-type]++ def test_tables_are_keyed_by_driver_normalization(self) -> None:+ # Raw roles that Driver's verify_state treats as one role (equal+ # normalized_role) always share one role class on every platform.+ for platform in PLATFORMS:+ for klass, raw_roles in RAW_ROLES[platform].items():+ self.assertIn(klass, ROLE_CLASSES)+ for raw in raw_roles:+ self.assertEqual(ROLE_TABLES[platform][normalized_role(raw)], klass)+ # The same control spelled per platform lands in the same class.+ for names, klass in (+ (("AXCheckBox", "CheckBox", "check box"), "checkbox"),+ (("AXButton", "Button", "push button"), "button"),+ (("AXRadioButton", "RadioButton", "radio button"), "radio"),+ (("AXLink", "Hyperlink", "link"), "link"),+ (("AXTextField", "Edit", "entry"), "text_input"),+ (("AXPopUpButton", "ComboBox", "combo box"), "popup"),+ (("AXMenuItem", "MenuItem", "menu item"), "menu_item"),+ ):+ for platform, raw in zip(PLATFORMS, names):+ self.assertEqual(role_class(raw, platform), klass, (platform, raw))++ def test_every_role_class_has_one_action(self) -> None:+ self.assertEqual(set(ROLE_CLASS_ACTION), set(ROLE_CLASSES))+++class V2ContractTest(unittest.TestCase):+ def test_v2_request_validates_with_sources_and_elements(self) -> None:+ validated = validate_request(v2_request())+ self.assertEqual(validated["schema"], "cua.jev_choice_request_v2")+ self.assertEqual(validated["snapshot_id"], "s0000002a")+ self.assertEqual(len(validated["elements"]), 3)+ self.assertEqual(validated["candidates"][0]["source"], "ax")+ self.assertNotIn("source", validated["candidates"][1])++ def test_v2_without_optional_fields_is_accepted(self) -> None:+ request = v2_request()+ del request["snapshot_id"], request["elements"]+ for candidate in request["candidates"]:+ candidate.pop("source", None)+ validated = validate_request(request)+ self.assertIsNone(validated["snapshot_id"])+ self.assertEqual(validated["elements"], [])++ def test_v1_rejects_every_v2_field(self) -> None:+ for mutate in (+ lambda r: r.__setitem__("snapshot_id", "s1"),+ lambda r: r.__setitem__("elements", []),+ lambda r: r["candidates"][0].__setitem__("source", "visual"),+ ):+ request = v1_request()+ mutate(request)+ with self.assertRaises(ValueError):+ validate_request(request)++ def test_v2_rejects_malformed_fields(self) -> None:+ mutations = {+ "unknown root key": lambda r: r.__setitem__("state", {}),+ "unknown source": lambda r: r["candidates"][0].__setitem__("source", "dom"),+ "reserved with source": lambda r: r["candidates"][1].__setitem__("source", "ax"),+ "candidate token": lambda r: r["candidates"][0].__setitem__("element_token", "s1:3"),+ "element value": lambda r: r["elements"][0].__setitem__("value", "secret"),+ "unknown role class": lambda r: r["elements"][0].__setitem__("role_class", "slider"),+ "unknown state": lambda r: r["elements"][0].__setitem__("state", "hidden"),+ "too many elements": lambda r: r.__setitem__("elements", r["elements"] * 22),+ "missing required": lambda r: r.pop("regions"),+ "unknown schema": lambda r: r.__setitem__("schema", "cua.jev_choice_request_v3"),+ }+ for name, mutate in mutations.items():+ request = copy.deepcopy(v2_request())+ mutate(request)+ with self.subTest(name=name), self.assertRaises(ValueError):+ validate_request(request)++ def test_provider_observation_is_unchanged_for_v1(self) -> None:+ observation = provider_observation(validate_request(v1_request()))+ self.assertEqual(set(observation), {"capture_id", "regions", "history"})+ v2 = provider_observation(validate_request(v2_request()))+ self.assertEqual(v2["candidate_sources"], {"ax:button:increment": "ax"})+ self.assertEqual(v2["snapshot_id"], "s0000002a")+ self.assertNotIn("element_token", json.dumps(v2))++ def test_mock_cli_answers_v2_with_the_unchanged_response(self) -> None:+ script = ROOT / "python/choose_action.py"+ result = subprocess.run(+ [sys.executable, str(script), "--mock"],+ input=json.dumps(v2_request()),+ text=True,+ capture_output=True,+ check=True,+ )+ response = json.loads(result.stdout)+ self.assertEqual(response["schema"], "cua.jev_choice_v1")+ self.assertEqual(response["selected_id"], "ax:button:increment")++ def test_decision_cli_answers_v2_with_decision_choice_v1(self) -> None:+ script = ROOT / "python/choose_decision.py"+ result = subprocess.run(+ [sys.executable, str(script), "--model", "mock"],+ input=json.dumps(v2_request()),+ text=True,+ capture_output=True,+ check=True,+ )+ response = json.loads(result.stdout)+ self.assertEqual(response["schema"], "cua.decision_choice_v1")+ self.assertEqual(response["kind"], "selected")+ self.assertEqual(response["selected_id"], "ax:button:increment")++ def test_jev_adapter_receives_native_observation(self) -> None:+ sent: dict = {}++ class Answer:+ choice = "ax:button:increment"+ confidence = 0.9+ probabilities = {"ax:button:increment": 0.9, "reobserve": 0.05, "abstain": 0.05}++ class Client:+ def system_one(self, **request):+ sent.update(request)+ return type("Response", (), {"choices": {"candidate": Answer()}, "model": "jev"})()++ request = DecisionRequest.from_validated(validate_request(v2_request()))+ result = choose(TypeSafeDecisionModel(Client()), request)+ self.assertEqual(result.selected_id, "ax:button:increment")+ observation = sent["state"]["observation"]+ self.assertEqual(observation["elements"][0]["label"], "Increment")+ self.assertEqual(observation["candidate_sources"], {"ax:button:increment": "ax"})++ def test_s1_text_adapter_renders_an_accessibility_tree(self) -> None:+ request = DecisionRequest.from_validated(validate_request(v2_request()))+ text = native_elements_as_text(request)+ self.assertIn('- button "Increment" (enabled)', text)+ self.assertIn('- text_input "Note" (empty)', text)+ self.assertIn("Prior bounded decisions:", text)++ seen: dict = {}++ class Scorer:+ modality = "text"++ def forward(self, options, **kwargs):+ seen.update(kwargs)+ return [+ type("Score", (), {"element_id": option.element_id, "probability": p})()+ for option, p in zip(options, (0.8, 0.1, 0.1))+ ]++ result = choose(S1DecisionModel(Scorer()), request)+ self.assertEqual(result.selected_id, "ax:button:increment")+ self.assertIn("Accessibility tree for snapshot", seen["ax_tree"])++ def test_mock_decision_model_is_unchanged_for_v2(self) -> None:+ request = DecisionRequest.from_validated(validate_request(v2_request()))+ result = choose(MockDecisionModel(), request)+ self.assertEqual(result.selected_id, "ax:button:increment")+++if __name__ == "__main__":+ unittest.main()diff --git a/libs/cua-driver/examples/jev-use/python/tests/test_native_source.py b/libs/cua-driver/examples/jev-use/python/tests/test_native_source.pynew file mode 100644index 0000000000..d322493475--- /dev/null+++ b/libs/cua-driver/examples/jev-use/python/tests/test_native_source.py@@ -0,0 +1,355 @@+"""NativeAccessibilitySource, stable IDs, policy, and AppKit tasks (RFC #4268)."""++from __future__ import annotations++import copy+import json+import sys+import tempfile+import unittest+from pathlib import Path++sys.path.insert(0, str(Path(__file__).resolve().parents[1]))++from choose_action import validate_request+from jev_adapter import choose_mock_for_task+from native import NativeObservation, eligible_controls, risk_categories, slug+from native_tasks import (+ APPKIT_TASK_IDS,+ MAX_EXECUTABLE_CANDIDATES,+ AppStateOracle,+ OracleError,+ appkit_task,+ compose,+ native_choice_request,+ visual_fallback_reason,+)+from sources import Candidate, NativeAccessibilitySource+from tasks import TaskSources++ROOT = Path(__file__).resolve().parents[2]+FIXTURES = ROOT / "fixtures/native"+++def window_state(name: str = "initial") -> dict:+ return json.loads((FIXTURES / f"appkit-window-state-{name}-v1.json").read_text(encoding="utf-8"))+++def observe(payload: dict) -> NativeObservation:+ return NativeObservation.from_window_state(+ payload, expected_pid=payload["pid"], expected_window_id=payload["window_id"]+ )+++def element(index: int, role: str, label: str | None, **extra) -> dict:+ item = {+ "element_index": index,+ "role": role,+ "depth": 1,+ "parent_index": 0,+ "enabled": True,+ "element_token": f"s0000000a:{index}",+ "frame": {"x": 10 + index, "y": 10, "w": 40, "h": 20},+ }+ if label is not None:+ item["label"] = label+ item.update(extra)+ return item+++def synthetic(elements: list[dict], **extra) -> dict:+ payload = {+ "pid": 7,+ "window_id": 9,+ "snapshot_id": "s0000000a",+ "capture_id": "cap-1",+ "elements_complete": False,+ "window_bounds": {"x": 0, "y": 0, "width": 500, "height": 400},+ "elements": [+ {"element_index": 0, "role": "AXWindow", "depth": 0, "label": "Main",+ "element_token": "s0000000a:0", "frame": {"x": 0, "y": 0, "w": 500, "h": 400}},+ *elements,+ ],+ }+ payload.update(extra)+ return payload+++class CrossLanguageParityTest(unittest.TestCase):+ def test_shared_id_fixture(self) -> None:+ fixture = json.loads((FIXTURES / "native-candidate-ids-v1.json").read_text(encoding="utf-8"))+ for case in fixture["cases"]:+ ids = [c.id for c in eligible_controls(observe(case["window_state"]), case["platform"]).controls]+ self.assertEqual(ids, case["expected_ids"], case["name"])+ for label, expected in fixture["slugs"].items():+ self.assertEqual(slug(label), expected)+ for label, expected in fixture["risks"].items():+ self.assertEqual(sorted(risk_categories(label)), expected)+++class ElementRulesTest(unittest.TestCase):+ def test_fixture_controls_and_exclusions(self) -> None:+ controls = eligible_controls(observe(window_state()), "macos")+ ids = [control.id for control in controls.controls]+ self.assertEqual(+ ids,+ [+ "ax:button:increment",+ "ax:button:reset",+ "ax:button:click-target-left-right-double",+ "ax:checkbox:i-agree",+ "ax:button:right-click-for-context-menu",+ "ax:button:exit",+ "ax:text_input:note",+ "ax:button:save-note",+ "ax:radio:small",+ "ax:radio:medium",+ "ax:radio:large",+ ],+ )+ # The placeholder-only field reports its placeholder as label and value.+ self.assertNotIn("ax:text_input:type-here", ids)+ # Menu-bar items lie outside the window; static text and sliders have no class.+ self.assertGreater(controls.excluded["off_screen"], 0)+ self.assertGreater(controls.excluded["unknown_role"], 0)+ self.assertGreater(controls.excluded["unlabeled"], 0)++ def test_each_rule_excludes(self) -> None:+ cases = {+ "unknown_role": element(1, "AXStaticText", "Hello"),+ "disabled": element(1, "AXButton", "Go", enabled=False),+ "off_screen": element(1, "AXButton", "Go", frame={"x": 900, "y": 900, "w": 10, "h": 10}),+ "zero size": element(1, "AXButton", "Go", frame={"x": 1, "y": 1, "w": 0, "h": 10}),+ "no frame": {k: v for k, v in element(1, "AXButton", "Go").items() if k != "frame"},+ "unlabeled": element(1, "AXButton", None),+ "blank label": element(1, "AXButton", " "),+ "label equals value": element(1, "AXTextField", "typed", value="typed"),+ "marked unlabelled": element(1, "AXButton", "Go", unlabelled=True),+ "web content": element(1, "AXButton", "Go", in_web_content=True),+ "no token": {k: v for k, v in element(1, "AXButton", "Go").items() if k != "element_token"},+ }+ for name, item in cases.items():+ with self.subTest(name=name):+ self.assertEqual(eligible_controls(observe(synthetic([item])), "macos").controls, ())+ kept = eligible_controls(observe(synthetic([element(1, "AXButton", "Go")])), "macos")+ self.assertEqual([c.id for c in kept.controls], ["ax:button:go"])++ def test_platform_tables_apply(self) -> None:+ items = [element(1, "Edit", "Name"), element(2, "Text", "Static"), element(3, "push button", "OK")]+ windows = eligible_controls(observe(synthetic(items)), "windows")+ linux = eligible_controls(observe(synthetic(items)), "linux")+ # UIA Text is static text; "push button" normalizes to "button" as Driver does.+ self.assertEqual([c.id for c in windows.controls], ["ax:text_input:name", "ax:button:ok"])+ self.assertEqual(+ [c.id for c in linux.controls],+ ["ax:text_input:static", "ax:button:ok"], # "Edit" is not an AT-SPI role+ )++ def test_secret_labels_are_redacted(self) -> None:+ item = element(1, "AXButton", "Send hunter2 now")+ controls = eligible_controls(+ observe(synthetic([item])), "macos", redact=lambda value: value.replace("hunter2", "[secret]")+ )+ self.assertEqual(controls.controls[0].label, "Send [secret] now")+ self.assertNotIn("hunter2", controls.controls[0].id)+++class StableIdTest(unittest.TestCase):+ def test_ids_ignore_element_index(self) -> None:+ first = synthetic([element(1, "AXButton", "Apply"), element(2, "AXCheckBox", "Wrap")])+ shifted = copy.deepcopy(first)+ for item in shifted["elements"][1:]:+ item["element_index"] += 40+ item["element_token"] = f"s0000000b:{item['element_index']}"+ a = [c.id for c in eligible_controls(observe(first), "macos").controls]+ b = [c.id for c in eligible_controls(observe(shifted), "macos").controls]+ self.assertEqual(a, b)+ # Across real snapshots too.+ initial = [c.id for c in eligible_controls(observe(window_state()), "macos").controls]+ after = [c.id for c in eligible_controls(observe(window_state("after-actions")), "macos").controls]+ self.assertEqual(initial, after)++ def test_collisions_get_path_suffixes(self) -> None:+ payload = synthetic(+ [+ element(1, "AXButton", "Options"),+ element(2, "AXButton", "OK", parent_index=1),+ element(3, "AXButton", "OK"),+ ]+ )+ ids = [c.id for c in eligible_controls(observe(payload), "macos").controls]+ self.assertEqual(ids[0], "ax:button:options")+ self.assertTrue(ids[1].startswith("ax:button:ok:") and ids[2].startswith("ax:button:ok:"))+ self.assertNotEqual(ids[1], ids[2])+ twins = synthetic([element(1, "AXButton", "OK"), element(2, "AXButton", "OK")])+ twin_ids = [c.id for c in eligible_controls(observe(twins), "macos").controls]+ self.assertEqual(len(set(twin_ids)), 2)++ def test_slug_bounds(self) -> None:+ self.assertEqual(slug("Click target (left / right / double)"), "click-target-left-right-double")+ self.assertLessEqual(len(slug("x" * 200)), 32)+ self.assertRegex(slug("你好"), r"^[0-9a-f]{8}$")+++class PolicyTest(unittest.TestCase):+ def test_risk_categories(self) -> None:+ self.assertEqual(risk_categories("Delete file"), {"destructive"})+ self.assertEqual(risk_categories("Don’t Save"), {"close_unsaved"})+ self.assertEqual(risk_categories("Buy now"), {"purchase"})+ self.assertEqual(risk_categories("Submit"), {"send"})+ self.assertEqual(risk_categories("Resetting"), frozenset()) # whole words only+ self.assertEqual(risk_categories("Save note"), frozenset())++ def test_compose_order_dedup_risk_and_cap(self) -> None:+ def cand(i: str, source: str, risk=frozenset()) -> Candidate:+ return Candidate(i, "d", "click", {}, source=source, risk=risk)++ ax = [cand(f"ax:button:b{n}", "ax") for n in range(30)]+ page = [cand("ax:button:b0", "page"), cand("p1", "page")]+ risky = [cand("ax:button:delete", "ax", frozenset({"destructive"}))]+ candidates, stats = compose(+ {"visual": [cand("v1", "visual")], "ax": risky + ax, "page": page},+ allowed_risks=frozenset(),+ )+ ids = [c.id for c in candidates]+ self.assertEqual(ids[:3], ["ax:button:b0", "p1", "ax:button:b1"])+ self.assertEqual(candidates[0].source, "page") # first source wins+ self.assertEqual(ids[-2:], ["reobserve", "abstain"])+ self.assertEqual(len(ids), MAX_EXECUTABLE_CANDIDATES + 2)+ self.assertEqual(stats.duplicates, 1)+ self.assertEqual(stats.risk_excluded, {"destructive": 1})+ self.assertEqual(stats.dropped, 32 - MAX_EXECUTABLE_CANDIDATES) # 2 page + 29 ax + 1 visual+ allowed, _ = compose({"ax": risky}, allowed_risks=frozenset({"destructive"}))+ self.assertIn("ax:button:delete", [c.id for c in allowed])+++class AppKitTaskTest(unittest.TestCase):+ def sources(self, task, name: str = "initial", foreground=frozenset()) -> TaskSources:+ ax = NativeAccessibilitySource.from_observation(+ observe(window_state(name)), "macos", redact=task.redact_text, text_method=task.text_method+ )+ return TaskSources(ax=ax, foreground_ids=frozenset(foreground))++ def test_candidates_and_arguments(self) -> None:+ task = appkit_task("appkit-save-note", Path("/tmp/none.json"), note_text="secret note")+ step = task.plan(self.sources(task))+ by_id = {c.id: c for c in step.candidates}+ self.assertNotIn("ax:button:reset", by_id)+ self.assertNotIn("ax:button:exit", by_id)+ self.assertNotIn("ax:checkbox:i-agree", by_id) # toggle not allowed+ setter = by_id["ax:text_input:note:set:note"]+ self.assertEqual(setter.tool, "set_value")+ self.assertEqual(setter.arguments["value"], "secret note")+ self.assertEqual(setter.arguments["element_token"], "s00000086:21")+ self.assertEqual(setter.snapshot_id, "s00000086")+ save = by_id["ax:button:save-note"]+ self.assertEqual(dict(save.arguments), {+ "pid": 20535, "window_id": 71364, "element_token": "s00000086:22",+ "delivery_mode": "background",+ })+ request = native_choice_request(task, self.sources(task), step, [])+ validate_request(request)+ wire = json.dumps(request)+ self.assertNotIn("secret note", wire)+ self.assertNotIn("element_token", wire)+ self.assertNotIn("s00000086:", wire)+ self.assertEqual(request["schema"], "cua.jev_choice_request_v2")+ self.assertEqual({c.get("source") for c in request["candidates"][:-2]}, {"ax"})++ def test_type_text_method(self) -> None:+ ax = NativeAccessibilitySource.from_observation(observe(window_state()), "macos", text_method="type_text")+ control = ax.find("text_input", "Note")+ candidate = ax.type_text(control, "hi", candidate_id="x", description="d")+ self.assertEqual(candidate.tool, "type_text")+ self.assertEqual(candidate.arguments["text"], "hi")++ def test_satisfied_controls_leave_the_set(self) -> None:+ note = appkit_task("appkit-save-note", Path("/tmp/none.json"), note_text="hello jev")+ ids = [c.id for c in note.plan(self.sources(note, "after-actions")).candidates]+ self.assertNotIn("ax:text_input:note:set:note", ids)+ size = appkit_task("appkit-choose-size", Path("/tmp/none.json"))+ ids = [c.id for c in size.plan(self.sources(size, "after-actions")).candidates]+ self.assertNotIn("ax:radio:large", ids) # already selected+ self.assertIn("ax:radio:small", ids)++ def test_foreground_escalation_requires_permission(self) -> None:+ refused = {"ax:button:increment"}+ denied = appkit_task("appkit-counter", Path("/tmp/none.json"))+ ids = [c.id for c in denied.plan(self.sources(denied, foreground=refused)).candidates]+ self.assertNotIn("ax:button:increment", ids)+ self.assertNotIn("ax:button:increment:foreground", ids)+ allowed = appkit_task("appkit-counter", Path("/tmp/none.json"), allow_foreground=True)+ step = allowed.plan(self.sources(allowed, foreground=refused))+ fg = {c.id: c for c in step.candidates}["ax:button:increment:foreground"]+ self.assertEqual(fg.arguments["delivery_mode"], "foreground")++ def test_mock_follows_task_preferences(self) -> None:+ for task_id, expected in (+ ("appkit-counter", "ax:button:increment"),+ ("appkit-save-note", "ax:text_input:note:set:note"),+ ("appkit-choose-size", "ax:radio:large"),+ ):+ task = appkit_task(task_id, Path("/tmp/none.json"))+ sources = self.sources(task)+ choice, confidence, _ = choose_mock_for_task(task, sources, task.candidates(sources), [])+ self.assertEqual((choice, confidence), (expected, 1.0))+ size = appkit_task("appkit-choose-size", Path("/tmp/none.json"))+ sources = self.sources(size, "after-actions")+ choice, _, _ = choose_mock_for_task(size, sources, size.candidates(sources), [])+ self.assertEqual(choice, "ax:checkbox:i-agree")++ def test_oracle_and_classification(self) -> None:+ with tempfile.TemporaryDirectory() as directory:+ path = Path(directory) / "state.json"+ state = {"schema": "cua.appkit_task_state_v1", "pid": 42, "counter": 3,+ "agreed": True, "size": "large", "note_saved": "n"}+ path.write_text(json.dumps(state))+ counter = appkit_task("appkit-counter", path, pid=42)+ self.assertEqual(counter.classify(counter.read_oracle(), steps=3), "verified")+ self.assertEqual(appkit_task("appkit-save-note", path, note_text="n").classify(state, steps=2), "verified")+ self.assertEqual(appkit_task("appkit-save-note", path, note_text="m").classify(state, steps=2), "refuted")+ self.assertEqual(appkit_task("appkit-choose-size", path).classify(state, steps=2), "verified")+ self.assertEqual(counter.classify({**state, "counter": 4}, steps=4), "refuted")+ self.assertEqual(counter.classify({**state, "counter": 1}, steps=1), "unknown")+ self.assertEqual(counter.classify({**state, "counter": 1}, steps=6), "budget_exhausted")+ with self.assertRaises(OracleError):+ appkit_task("appkit-counter", path, pid=43).read_oracle()+ with self.assertRaises(OracleError):+ AppStateOracle(Path(directory) / "missing.json", "cua.appkit_task_state_v1").read()+ self.assertEqual(set(APPKIT_TASK_IDS), {"appkit-counter", "appkit-save-note", "appkit-choose-size"})++ def test_history_is_compact_and_redacted(self) -> None:+ task = appkit_task("appkit-save-note", Path("/tmp/none.json"), note_text="topsecret")+ entry = task.history_entry(2, "x", outcome="typed topsecret")+ self.assertEqual(entry["outcome"], "typed [note text]")+ self.assertIn("stale", task.history_entry(3, "x", stale=True)["outcome"])++ def test_visual_fallback_rule(self) -> None:+ task = appkit_task("appkit-counter", Path("/tmp/none.json"))+ visual_task = type(task)(**{+ **{k: getattr(task, k) for k in task.__dataclass_fields__ if k != "completion_candidate_ids"},+ "allowed_actions": frozenset({"press", "visual_click"}),+ "visual_targets": ("Increment",),+ })++ def reason(payload: dict, count: int) -> str | None:+ ax = NativeAccessibilitySource.from_observation(observe(payload), "macos")+ return visual_fallback_reason(TaskSources(ax=ax), visual_task, count)++ self.assertIsNone(reason(window_state(), 0)) # macOS never claims completeness+ self.assertIsNone(reason(synthetic([], truncated=True, elements_complete=True), 0))+ self.assertEqual(reason(synthetic([], degraded=True, degraded_reason="ax_tree_empty: none"), 0), "tree_empty")+ self.assertEqual(reason(synthetic([], elements_complete=True), 0), "no_native_candidates")+ complete = synthetic([element(1, "AXButton", "Other")], elements_complete=True)+ self.assertEqual(reason(complete, 1), "target_without_element")+ self.assertIsNone(visual_fallback_reason(+ TaskSources(ax=NativeAccessibilitySource.from_observation(observe(synthetic([], elements_complete=True)), "macos")),+ task, 0)) # visual_click not allowed++ def test_observation_rejects_other_windows(self) -> None:+ with self.assertRaises(ValueError):+ NativeObservation.from_window_state(window_state(), expected_pid=1, expected_window_id=2)+++if __name__ == "__main__":+ unittest.main()diff --git a/libs/cua-driver/examples/jev-use/typescript/choose_action.ts b/libs/cua-driver/examples/jev-use/typescript/choose_action.tsindex 3e65db1b04..e193c783b8 100644--- a/libs/cua-driver/examples/jev-use/typescript/choose_action.ts+++ b/libs/cua-driver/examples/jev-use/typescript/choose_action.ts@@ -4,22 +4,54 @@ import { pathToFileURL } from 'node:url';import { TypeSafeClient } from '@typesafe-ai/sdk';import { chooseBoundedWithTypeSafe, type ProviderChoice } from './jev_adapter.js';+import { ROLE_CLASSES, type RoleClass } from './native_roles.js';export const REQUEST_SCHEMA = 'cua.jev_choice_request_v1';+// Additive over v1 (RFC #4268): per-candidate `source`, an optional root+// `snapshot_id`, and an optional compact `elements` list of native controls.+export const REQUEST_SCHEMA_V2 = 'cua.jev_choice_request_v2';export const RESPONSE_SCHEMA = 'cua.jev_choice_v1';const MAX_INPUT_BYTES = 65_536;const MAX_CANDIDATES = 32;const MAX_REGIONS = 100;const MAX_HISTORY = 16;+export const MAX_ELEMENTS = 64;const ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,63}$/;+const RESERVED_IDS = new Set(['reobserve', 'abstain']);+export type CandidateSourceKind = 'page' | 'ax' | 'visual';+const CANDIDATE_SOURCES = new Set<string>(['page', 'ax', 'visual']);+export type ElementState =+ | 'enabled'+ | 'checked'+ | 'unchecked'+ | 'selected'+ | 'not_selected'+ | 'empty'+ | 'has_text';+const ELEMENT_STATES = new Set<string>([+ 'enabled',+ 'checked',+ 'unchecked',+ 'selected',+ 'not_selected',+ 'empty',+ 'has_text',+]);+const V1_ROOT_KEYS = ['candidates', 'capture_id', 'goal', 'history', 'regions', 'schema'];+const V2_OPTIONAL_ROOT_KEYS = ['elements', 'snapshot_id'];type JsonRecord = Record<string, unknown>;-type ValidatedRequest = {+export type CompactElement = { role_class: RoleClass; label: string; state: ElementState };+export type ValidatedCandidate = { id: string; description: string; source?: CandidateSourceKind };+export type ValidatedRequest = {+ schema: typeof REQUEST_SCHEMA | typeof REQUEST_SCHEMA_V2;goal: string;capture_id: string;regions: JsonRecord[];history: unknown[];- candidates: { id: string; description: string }[];+ candidates: ValidatedCandidate[];+ snapshot_id?: string | null;+ elements?: CompactElement[];};function record(value: unknown, message: string): JsonRecord {@@ -49,11 +81,51 @@ function exactKeys(value: JsonRecord, expected: readonly string[]): boolean {);}+function validateElements(value: unknown): CompactElement[] {+ if (!Array.isArray(value) || value.length > MAX_ELEMENTS) {+ throw new Error(`elements must be an array of at most ${MAX_ELEMENTS} items`);+ }+ return value.map((item) => {+ const raw = record(item, 'element must be an object');+ if (!exactKeys(raw, ['label', 'role_class', 'state'])) {+ throw new Error('element may contain only role_class, label, and state');+ }+ if (typeof raw.role_class !== 'string' || !(ROLE_CLASSES as readonly string[]).includes(raw.role_class)) {+ throw new Error('element role_class is not a supported role class');+ }+ if (typeof raw.state !== 'string' || !ELEMENT_STATES.has(raw.state)) {+ throw new Error('element state is not a supported state');+ }+ return {+ role_class: raw.role_class as RoleClass,+ label: boundedString(raw.label, 'element label', 200),+ state: raw.state as ElementState,+ };+ });+}++/**+ * Validate a cua.jev_choice_request_v1 or _v2 request strictly. v1 is+ * unchanged, so any v2 field in a v1 request is rejected. v2 may add a root+ * snapshot_id and elements and a per-candidate source; reserved candidates+ * never carry a source.+ */export function validateRequest(value: unknown): ValidatedRequest {const root = record(value, 'request must be a JSON object');- const rootKeys = ['candidates', 'capture_id', 'goal', 'history', 'regions', 'schema'];- if (!exactKeys(root, rootKeys) || root.schema !== REQUEST_SCHEMA) {- throw new Error(`request must match ${REQUEST_SCHEMA}`);+ const keys = Object.keys(root);+ let v2 = false;+ if (root.schema === REQUEST_SCHEMA) {+ if (!exactKeys(root, V1_ROOT_KEYS)) throw new Error(`request must match ${REQUEST_SCHEMA}`);+ } else if (root.schema === REQUEST_SCHEMA_V2) {+ v2 = true;+ if (+ !V1_ROOT_KEYS.every((key) => key in root) ||+ keys.some((key) => !V1_ROOT_KEYS.includes(key) && !V2_OPTIONAL_ROOT_KEYS.includes(key))+ ) {+ throw new Error(`request must match ${REQUEST_SCHEMA_V2}`);+ }+ } else {+ throw new Error(`request must match ${REQUEST_SCHEMA} or ${REQUEST_SCHEMA_V2}`);}const goal = boundedString(root.goal, 'goal', 4_000);const captureId = boundedString(root.capture_id, 'capture_id', 256);@@ -150,20 +222,75 @@ export function validateRequest(value: unknown): ValidatedRequest {throw new Error(`candidates must contain between 2 and ${MAX_CANDIDATES} items`);}const candidateIds = new Set<string>();- const candidates = root.candidates.map((item) => {+ const candidates = root.candidates.map((item): ValidatedCandidate => {const raw = record(item, 'candidate must be an object');- if (!exactKeys(raw, ['description', 'id'])) {+ if (v2) {+ const candidateKeys = Object.keys(raw);+ if (+ !('id' in raw) ||+ !('description' in raw) ||+ candidateKeys.some((key) => key !== 'id' && key !== 'description' && key !== 'source')+ ) {+ throw new Error('candidate may contain only id, description, and source');+ }+ } else if (!exactKeys(raw, ['description', 'id'])) {throw new Error('candidate may contain only id and description');}const id = identifier(raw.id, 'candidate id');if (candidateIds.has(id)) throw new Error('candidate IDs must be unique');candidateIds.add(id);- return { id, description: boundedString(raw.description, 'description', 1_000) };+ const candidate: ValidatedCandidate = {+ id,+ description: boundedString(raw.description, 'description', 1_000),+ };+ if ('source' in raw) {+ if (RESERVED_IDS.has(id)) throw new Error('reserved candidates must not carry a source');+ if (typeof raw.source !== 'string' || !CANDIDATE_SOURCES.has(raw.source)) {+ throw new Error('candidate source must be page, ax, or visual');+ }+ candidate.source = raw.source as CandidateSourceKind;+ }+ return candidate;});if (!candidateIds.has('reobserve') || !candidateIds.has('abstain')) {throw new Error('candidates must include reobserve and abstain');}- return { goal, capture_id: captureId, regions, history, candidates };+ const validated: ValidatedRequest = {+ schema: v2 ? REQUEST_SCHEMA_V2 : REQUEST_SCHEMA,+ goal,+ capture_id: captureId,+ regions,+ history,+ candidates,+ };+ if (v2) {+ validated.snapshot_id =+ root.snapshot_id === undefined || root.snapshot_id === null+ ? null+ : boundedString(root.snapshot_id, 'snapshot_id', 64);+ validated.elements = validateElements(root.elements ?? []);+ }+ return validated;+}++/**+ * The observation a provider sees. A v1 request keeps its exact v1+ * observation; v2 adds the snapshot, compact elements, and candidate sources.+ */+export function providerObservation(validated: ValidatedRequest): JsonRecord {+ const observation: JsonRecord = {+ capture_id: validated.capture_id,+ regions: validated.regions,+ history: validated.history,+ };+ if (validated.schema === REQUEST_SCHEMA_V2) {+ observation.snapshot_id = validated.snapshot_id ?? null;+ observation.elements = validated.elements ?? [];+ observation.candidate_sources = Object.fromEntries(+ validated.candidates.filter((item) => item.source).map((item) => [item.id, item.source])+ );+ }+ return observation;}export async function chooseRequest(@@ -191,11 +318,7 @@ export async function chooseRequest(result = await chooseBoundedWithTypeSafe(options.client ?? new TypeSafeClient(),validated.goal,- {- capture_id: validated.capture_id,- regions: validated.regions,- history: validated.history,- },+ providerObservation(validated),criteria);}diff --git a/libs/cua-driver/examples/jev-use/typescript/jev_adapter.ts b/libs/cua-driver/examples/jev-use/typescript/jev_adapter.tsindex 482e20bd28..530c4106c3 100644--- a/libs/cua-driver/examples/jev-use/typescript/jev_adapter.ts+++ b/libs/cua-driver/examples/jev-use/typescript/jev_adapter.ts@@ -7,7 +7,14 @@ import {type Candidate,type VisualObservation,} from './core.js';-import { FIXTURE_GOAL, FixtureFormTask, fixtureSources, type Task, type TaskSources } from './tasks.js';+import {+ FIXTURE_GOAL,+ FixtureFormTask,+ fixtureSources,+ requirePage,+ type Task,+ type TaskSources,+} from './tasks.js';type TypeSafeClientLike = Pick<TypeSafeClient, 'systemOne'>;@@ -112,7 +119,7 @@ export function taskDecisionState(sources: TaskSources,history: readonly HistoryEntry[]) {- const snapshot = sources.page.snapshot;+ const snapshot = requirePage(sources).snapshot;return {goal: task.goal,observation: {@@ -192,13 +199,30 @@ export function chooseLiveForTask(return chooseForTask(new TypeSafeClient(), task, sources, candidates, history);}+/**+ * Deterministic mock provider. A task may declare mockPreferences: the first+ * preferred ID present wins (a refused control's `<id>:foreground` variant+ * counts as its ID), otherwise reobserve. Tasks without preferences keep the+ * fixed browser-fixture order.+ */export function chooseMockForTask(- _task: Task,+ task: Task,_sources: TaskSources,candidates: Candidate[],_history: readonly HistoryEntry[]) {- return chooseMock(candidates);+ const preferences = (task as { mockPreferences?: readonly string[] }).mockPreferences ?? [];+ if (!preferences.length) return chooseMock(candidates);+ const ids = candidates.map((candidate) => candidate.id);+ const selected =+ preferences+ .flatMap((preferred) => [preferred, `${preferred}:foreground`])+ .find((id) => ids.includes(id)) ?? (ids.includes('reobserve') ? 'reobserve' : null);+ return {+ choice: selected,+ confidence: selected ? 1 : 0,+ probabilities: Object.fromEntries(ids.map((id) => [id, Number(id === selected)])),+ };}export function chooseLive(diff --git a/libs/cua-driver/examples/jev-use/typescript/native.ts b/libs/cua-driver/examples/jev-use/typescript/native.tsnew file mode 100644index 0000000000..ca69ab690c--- /dev/null+++ b/libs/cua-driver/examples/jev-use/typescript/native.ts@@ -0,0 +1,309 @@+/**+ * Native accessibility observations for jev-use (RFC #4268, Phase 1).+ * Mirrors python/native.py; see that module for the full element rules.+ *+ * One observation is exactly one get_window_state call with both the tree and+ * the screenshot, so its snapshot_id (element tokens) and capture_id (visual+ * regions) describe the same moment. Stable IDs never use element_index.+ */+import { createHash } from 'node:crypto';++import {+ ROLE_CLASS_ACTION,+ normalizedRole,+ roleClass,+ type ActionKind,+ type Platform,+ type RoleClass,+} from './native_roles.js';++export const ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,63}$/;+export const PARAMETER_NAME_PATTERN = /^[a-z][a-z0-9_]{0,7}$/;+const MAX_LABEL_CHARS = 120;+const SLUG_CHARS = 32;++export type RiskCategory = 'destructive' | 'send' | 'purchase' | 'close_unsaved';++/** Whole-word, case-insensitive phrases; a guard, not an authorization boundary. */+export const RISK_PHRASES: Readonly<Record<RiskCategory, readonly string[]>> = {+ destructive: ['delete', 'remove', 'erase', 'trash', 'discard', 'clear all', 'format', 'reset'],+ send: ['send', 'submit', 'post', 'publish', 'share', 'reply'],+ purchase: ['buy', 'purchase', 'pay', 'checkout', 'order', 'subscribe'],+ close_unsaved: ['close', 'quit', 'exit', "don't save", 'discard changes'],+};++function escapeRegExp(value: string): string {+ return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');+}++const RISK_PATTERNS = Object.entries(RISK_PHRASES).map(+ ([category, phrases]) =>+ [+ category,+ phrases.map((phrase) => new RegExp(`(?<![a-z0-9])${escapeRegExp(phrase)}(?![a-z0-9])`)),+ ] as const+);++export function riskCategories(label: string): Set<string> {+ const text = label.toLowerCase().replaceAll('’', "'");+ return new Set(+ RISK_PATTERNS.filter(([, patterns]) => patterns.some((pattern) => pattern.test(text))).map(+ ([category]) => category+ )+ );+}++function sha256Hex(value: string): string {+ return createHash('sha256').update(value, 'utf8').digest('hex');+}++/** Lowercase ASCII, collapse other runs to '-', trim to 32 characters. */+export function slug(label: string): string {+ const lowered = [...label]+ .map((char) => (/^[A-Za-z0-9]$/.test(char) ? char.toLowerCase() : '-'))+ .join('');+ const value = lowered+ .replace(/-+/g, '-')+ .replace(/^-+|-+$/g, '')+ .slice(0, SLUG_CHARS)+ .replace(/^-+|-+$/g, '');+ return value || sha256Hex(label).slice(0, 8);+}++/** Python json.dumps(value, ensure_ascii=False, separators=(",", ":")) for arrays of strings/numbers. */+function compactJson(value: unknown): string {+ return JSON.stringify(value);+}++function hex4(value: unknown): string {+ return sha256Hex(compactJson(value)).slice(0, 4);+}++type Rect = readonly [number, number, number, number];++function num(value: unknown): number | undefined {+ return typeof value === 'number' && Number.isFinite(value) ? value : undefined;+}++function rect(value: unknown): Rect | undefined {+ if (!value || typeof value !== 'object') return undefined;+ const item = value as Record<string, unknown>;+ const x = num(item.x);+ const y = num(item.y);+ const w = num(item.w ?? item.width);+ const h = num(item.h ?? item.height);+ if (x === undefined || y === undefined || w === undefined || h === undefined) return undefined;+ return [x, y, w, h];+}++function intersects(a: Rect, b: Rect): boolean {+ return (+ Math.min(a[0] + a[2], b[0] + b[2]) > Math.max(a[0], b[0]) &&+ Math.min(a[1] + a[3], b[1] + b[3]) > Math.max(a[1], b[1])+ );+}++function str(value: unknown): string | undefined {+ return typeof value === 'string' ? value : undefined;+}++export class NativeObservationError extends Error {+ constructor(+ message: string,+ readonly code = 'invalid_window_state'+ ) {+ super(message);+ this.name = 'NativeObservationError';+ }+}++export type WindowElement = Readonly<Record<string, unknown>>;++export type NativeObservation = Readonly<{+ pid: number;+ windowId: number;+ snapshotId?: string;+ captureId?: string;+ windowBounds?: Rect;+ elements: readonly WindowElement[];+ complete: boolean;+ truncated: boolean;+ degradedReason?: string;+ treeMarkdown: string;+}>;++export function isPartial(observation: NativeObservation): boolean {+ return !observation.complete;+}++export function isTreeEmpty(observation: NativeObservation): boolean {+ return Boolean(observation.degradedReason?.startsWith('ax_tree_empty'));+}++export function parseWindowState(+ payload: Readonly<Record<string, unknown>>,+ expectedPid: number,+ expectedWindowId: number+): NativeObservation {+ if (payload.pid !== expectedPid || payload.window_id !== expectedWindowId) {+ throw new NativeObservationError('window state belongs to a different window', 'window_mismatch');+ }+ const raw = payload.elements ?? [];+ if (!Array.isArray(raw) || !raw.every((item) => item && typeof item === 'object')) {+ throw new NativeObservationError('window state has malformed elements');+ }+ const index = (item: WindowElement) =>+ Number.isInteger(item.element_index) ? (item.element_index as number) : -1;+ const elements = [...(raw as WindowElement[])].sort((a, b) => index(a) - index(b));+ return {+ pid: expectedPid,+ windowId: expectedWindowId,+ snapshotId: str(payload.snapshot_id),+ captureId: str(payload.capture_id),+ windowBounds: rect(payload.window_bounds),+ elements,+ complete: payload.elements_complete === true,+ truncated: payload.truncated === true || Boolean(payload.truncation_reason),+ degradedReason: payload.degraded ? str(payload.degraded_reason) : undefined,+ treeMarkdown: str(payload.tree_markdown) ?? '',+ };+}++export type NativeControl = Readonly<{+ id: string;+ roleClass: RoleClass;+ action: ActionKind;+ label: string;+ value?: string;+ selected?: boolean;+ elementIndex: number;+ elementToken: string;+ risk: ReadonlySet<string>;+}>;++export type NativeControls = Readonly<{+ controls: readonly NativeControl[];+ excluded: Readonly<Record<string, number>>;+}>;++const collapse = (value: string) => value.split(/\s+/).filter(Boolean).join(' ');++/** Apply the element rules and assign stable IDs, in element_index order. */+export function eligibleControls(+ observation: NativeObservation,+ platform: Platform,+ redact: (value: string) => string = (value) => value+): NativeControls {+ const byIndex = new Map<number, WindowElement>();+ for (const item of observation.elements) {+ if (Number.isInteger(item.element_index)) byIndex.set(item.element_index as number, item);+ }+ const excluded: Record<string, number> = {};+ const exclude = (reason: string) => {+ excluded[reason] = (excluded[reason] ?? 0) + 1;+ };+ const labelOf = (item: WindowElement) => collapse(redact(str(item.label) ?? ''));+ const pathOf = (item: WindowElement): [string, string][] => {+ const path: [string, string][] = [];+ const seen = new Set<number>();+ let parent = item.parent_index;+ while (Number.isInteger(parent) && byIndex.has(parent as number) && !seen.has(parent as number)) {+ seen.add(parent as number);+ const ancestor = byIndex.get(parent as number)!;+ const rawRole = str(ancestor.role) ?? '';+ path.push([roleClass(rawRole, platform) ?? normalizedRole(rawRole), labelOf(ancestor)]);+ parent = ancestor.parent_index;+ }+ return path.reverse();+ };++ const pending: { item: WindowElement; klass: RoleClass; label: string; path: [string, string][] }[] = [];+ for (const item of observation.elements) {+ const klass = roleClass(item.role, platform);+ if (!klass) {+ exclude('unknown_role');+ continue;+ }+ if (item.enabled === false) {+ exclude('disabled');+ continue;+ }+ const frame = rect(item.frame);+ if (+ !frame ||+ frame[2] <= 0 ||+ frame[3] <= 0 ||+ !observation.windowBounds ||+ !intersects(frame, observation.windowBounds)+ ) {+ exclude('off_screen');+ continue;+ }+ const label = labelOf(item);+ const value = str(item.value);+ const redactedValue = value === undefined ? undefined : collapse(redact(value));+ if (item.unlabelled === true || !label || label === redactedValue) {+ exclude('unlabeled');+ continue;+ }+ if (item.in_web_content === true) {+ exclude('web_content');+ continue;+ }+ if (typeof item.element_token !== 'string' || !item.element_token) {+ exclude('no_token');+ continue;+ }+ pending.push({ item, klass, label, path: pathOf(item) });+ }++ const bases = pending.map(({ klass, label }) => `ax:${klass}:${slug(label)}`);+ const counts = new Map<string, number>();+ for (const base of bases) counts.set(base, (counts.get(base) ?? 0) + 1);+ const ordinals = new Map<string, number>();+ const controls = pending.map(({ item, klass, label, path }, position) => {+ const base = bases[position];+ const key = JSON.stringify([klass, label, path]);+ const ordinal = ordinals.get(key) ?? 0;+ ordinals.set(key, ordinal + 1);+ const id = counts.get(base) === 1 ? base : `${base}:${hex4([path, ordinal])}`;+ if (!ID_PATTERN.test(id)) throw new NativeObservationError('derived candidate ID is invalid');+ const control: NativeControl = {+ id,+ roleClass: klass,+ action: ROLE_CLASS_ACTION[klass],+ label: label.slice(0, MAX_LABEL_CHARS),+ value: str(item.value),+ selected: typeof item.selected === 'boolean' ? item.selected : undefined,+ elementIndex: item.element_index as number,+ elementToken: item.element_token as string,+ risk: riskCategories(label),+ };+ return control;+ });+ return { controls, excluded };+}++export type ElementState =+ | 'enabled'+ | 'checked'+ | 'unchecked'+ | 'selected'+ | 'not_selected'+ | 'empty'+ | 'has_text';++export function elementState(control: NativeControl): ElementState {+ if (control.roleClass === 'checkbox' || control.roleClass === 'toggle') {+ return control.selected ? 'checked' : 'unchecked';+ }+ if (control.roleClass === 'radio') return control.selected ? 'selected' : 'not_selected';+ if (control.roleClass === 'text_input') return control.value ? 'has_text' : 'empty';+ return 'enabled';+}++export function fieldState(value: string | undefined, required: string): string {+ if (!value) return 'empty';+ if (value === required) return 'contains_required_value';+ return 'contains_other_value';+}diff --git a/libs/cua-driver/examples/jev-use/typescript/native_contract.test.ts b/libs/cua-driver/examples/jev-use/typescript/native_contract.test.tsnew file mode 100644index 0000000000..8f9499bbca--- /dev/null+++ b/libs/cua-driver/examples/jev-use/typescript/native_contract.test.ts@@ -0,0 +1,151 @@+// Role table and cua.jev_choice_request_v2 contract tests (RFC #4268).+// Mirrors python/tests/test_native_contract.py.+import assert from 'node:assert/strict';+import { readFileSync } from 'node:fs';+import test from 'node:test';++import { chooseRequest, providerObservation, validateRequest } from './choose_action.js';+import {+ PLATFORMS,+ RAW_ROLES,+ ROLE_CLASSES,+ ROLE_CLASS_ACTION,+ ROLE_TABLES,+ normalizedRole,+ roleClass,+ type Platform,+} from './native_roles.js';++function fixture(name: string): any {+ return JSON.parse(readFileSync(new URL(`../fixtures/${name}`, import.meta.url), 'utf8'));+}++test('normalizedRole matches the Driver normalizer cases', () => {+ const cases: Record<string, string> = {+ AXCheckBox: 'checkbox',+ CheckBox: 'checkbox',+ 'check box': 'checkbox',+ 'push button': 'button',+ AXButton: 'button',+ 'page tab': 'tab',+ TabItem: 'tab',+ AXTextField: 'textfield',+ 'combo box': 'combobox',+ };+ for (const [raw, expected] of Object.entries(cases)) assert.equal(normalizedRole(raw), expected, raw);+});++test('each platform maps its raw roles and excludes the rest', () => {+ const mapped: Record<Platform, Record<string, string>> = {+ macos: {+ AXButton: 'button',+ AXCheckBox: 'checkbox',+ AXRadioButton: 'radio',+ AXPopUpButton: 'popup',+ AXMenuBarItem: 'menu_item',+ AXLink: 'link',+ AXTextField: 'text_input',+ AXSwitch: 'toggle',+ },+ windows: {+ Button: 'button',+ SplitButton: 'button',+ CheckBox: 'checkbox',+ RadioButton: 'radio',+ ComboBox: 'popup',+ MenuItem: 'menu_item',+ Hyperlink: 'link',+ Edit: 'text_input',+ },+ linux: {+ 'push button': 'button',+ 'toggle button': 'toggle',+ 'check box': 'checkbox',+ 'radio button': 'radio',+ 'combo box': 'popup',+ 'radio menu item': 'menu_item',+ link: 'link',+ entry: 'text_input',+ text: 'text_input',+ },+ };+ for (const platform of PLATFORMS) {+ for (const [raw, expected] of Object.entries(mapped[platform])) {+ assert.equal(roleClass(raw, platform), expected, `${platform} ${raw}`);+ }+ }+ const excluded: Record<Platform, string[]> = {+ macos: ['AXStaticText', 'AXSlider', 'AXWindow', 'AXMenuBar', ''],+ windows: ['Text', 'Slider', 'TabItem', 'Pane', 'Unknown'],+ linux: ['label', 'slider', 'page tab', 'frame'],+ };+ for (const platform of PLATFORMS) {+ for (const raw of excluded[platform]) assert.equal(roleClass(raw, platform), null, `${platform} ${raw}`);+ }+ assert.equal(roleClass(undefined, 'macos'), null);+});++test('role tables are keyed by the Driver normalization', () => {+ for (const platform of PLATFORMS) {+ for (const klass of ROLE_CLASSES) {+ for (const raw of RAW_ROLES[platform][klass]) {+ assert.equal(ROLE_TABLES[platform].get(normalizedRole(raw)), klass);+ }+ }+ }+ assert.deepEqual(new Set(Object.keys(ROLE_CLASS_ACTION)), new Set(ROLE_CLASSES));+});++test('v2 requests validate with sources and elements', () => {+ const validated = validateRequest(fixture('jev-choice-request-v2.json'));+ assert.equal(validated.schema, 'cua.jev_choice_request_v2');+ assert.equal(validated.snapshot_id, 's0000002a');+ assert.equal(validated.elements?.length, 3);+ assert.equal(validated.candidates[0].source, 'ax');+ assert.equal('source' in validated.candidates[1], false);+ const observation = providerObservation(validated);+ assert.deepEqual(observation.candidate_sources, { 'ax:button:increment': 'ax' });+ const v1 = providerObservation(validateRequest(fixture('jev-choice-request-v1.json')));+ assert.deepEqual(Object.keys(v1).sort(), ['capture_id', 'history', 'regions']);+});++test('v1 rejects v2 fields and v2 rejects malformed fields', () => {+ const v1Mutations: ((r: any) => void)[] = [+ (r) => (r.snapshot_id = 's1'),+ (r) => (r.elements = []),+ (r) => (r.candidates[0].source = 'visual'),+ ];+ for (const mutate of v1Mutations) {+ const request = fixture('jev-choice-request-v1.json');+ mutate(request);+ assert.throws(() => validateRequest(request));+ }+ const v2Mutations: Record<string, (r: any) => void> = {+ 'unknown root key': (r) => (r.state = {}),+ 'unknown source': (r) => (r.candidates[0].source = 'dom'),+ 'reserved with source': (r) => (r.candidates[1].source = 'ax'),+ 'candidate token': (r) => (r.candidates[0].element_token = 's1:3'),+ 'element value': (r) => (r.elements[0].value = 'secret'),+ 'unknown role class': (r) => (r.elements[0].role_class = 'slider'),+ 'unknown state': (r) => (r.elements[0].state = 'hidden'),+ 'too many elements': (r) => (r.elements = Array(22).fill(r.elements).flat()),+ 'missing required': (r) => delete r.regions,+ 'unknown schema': (r) => (r.schema = 'cua.jev_choice_request_v3'),+ };+ for (const [name, mutate] of Object.entries(v2Mutations)) {+ const request = fixture('jev-choice-request-v2.json');+ mutate(request);+ assert.throws(() => validateRequest(request), Error, name);+ }+ const minimal = fixture('jev-choice-request-v2.json');+ delete minimal.snapshot_id;+ delete minimal.elements;+ for (const candidate of minimal.candidates) delete candidate.source;+ assert.deepEqual(validateRequest(minimal).elements, []);+});++test('mock chooser answers v2 with the unchanged response schema', async () => {+ const response = await chooseRequest(fixture('jev-choice-request-v2.json'), { mock: true });+ assert.equal(response.schema, 'cua.jev_choice_v1');+ assert.equal(response.selected_id, 'ax:button:increment');+});diff --git a/libs/cua-driver/examples/jev-use/typescript/native_roles.ts b/libs/cua-driver/examples/jev-use/typescript/native_roles.tsnew file mode 100644index 0000000000..114d528baa--- /dev/null+++ b/libs/cua-driver/examples/jev-use/typescript/native_roles.ts@@ -0,0 +1,126 @@+// Per-platform role table for native accessibility candidates (RFC #4268).+// Mirrors python/native_roles.py; see that module for the full rationale.+//+// Cua Driver reports raw platform roles (macOS AX, Windows UIA+// control_type_name, Linux AT-SPI role names) and does not normalize them, so+// jev-use owns this closed mapping. Every table is keyed by the output of+// normalizedRole, a port of Driver's private normalized_role in+// cua-driver-core/src/expectation.rs, so raw roles that Driver's verify_state+// treats as the same role can never map to different role classes here.++export type Platform = 'macos' | 'windows' | 'linux';+export type RoleClass =+ | 'button'+ | 'toggle'+ | 'checkbox'+ | 'radio'+ | 'popup'+ | 'menu_item'+ | 'link'+ | 'text_input';+export type ActionKind = 'press' | 'toggle' | 'select' | 'open_menu' | 'set_text' | 'visual_click';++export const PLATFORMS: readonly Platform[] = ['macos', 'windows', 'linux'];++export const ROLE_CLASSES: readonly RoleClass[] = [+ 'button',+ 'toggle',+ 'checkbox',+ 'radio',+ 'popup',+ 'menu_item',+ 'link',+ 'text_input',+];++export const ROLE_CLASS_ACTION: Readonly<Record<RoleClass, ActionKind>> = {+ button: 'press',+ toggle: 'toggle',+ checkbox: 'toggle',+ radio: 'select',+ popup: 'open_menu',+ menu_item: 'press',+ link: 'press',+ text_input: 'set_text',+};++export const ACTION_KINDS: ReadonlySet<ActionKind> = new Set<ActionKind>([+ 'press',+ 'toggle',+ 'select',+ 'open_menu',+ 'set_text',+ 'visual_click',+]);++export const RAW_ROLES: Readonly<Record<Platform, Readonly<Record<RoleClass, readonly string[]>>>> = {+ macos: {+ button: ['AXButton'],+ toggle: ['AXSwitch'],+ checkbox: ['AXCheckBox'],+ radio: ['AXRadioButton'],+ popup: ['AXPopUpButton', 'AXComboBox', 'AXMenuButton'],+ menu_item: ['AXMenuItem', 'AXMenuBarItem'],+ link: ['AXLink'],+ text_input: ['AXTextField', 'AXTextArea', 'AXSearchField', 'AXSecureTextField'],+ },+ windows: {+ button: ['Button', 'SplitButton'],+ toggle: [],+ checkbox: ['CheckBox'],+ radio: ['RadioButton'],+ popup: ['ComboBox'],+ menu_item: ['MenuItem'],+ link: ['Hyperlink'],+ text_input: ['Edit'],+ },+ linux: {+ button: ['push button', 'button'],+ toggle: ['toggle button', 'switch'],+ checkbox: ['check box'],+ radio: ['radio button'],+ popup: ['combo box'],+ menu_item: ['menu item', 'check menu item', 'radio menu item'],+ link: ['link'],+ text_input: ['entry', 'text', 'password text'],+ },+};++/** Port of Driver's normalized_role (cua-driver-core/src/expectation.rs). */+export function normalizedRole(role: string): string {+ let normalized = [...role]+ .filter((character) => /^[A-Za-z0-9]$/.test(character))+ .map((character) => character.toLowerCase())+ .join('');+ if (normalized.startsWith('ax')) normalized = normalized.slice(2);+ if (normalized === 'pushbutton') return 'button';+ if (normalized === 'pagetab' || normalized === 'tabitem') return 'tab';+ return normalized;+}++function normalizedTable(platform: Platform): ReadonlyMap<string, RoleClass> {+ const table = new Map<string, RoleClass>();+ for (const roleClass of ROLE_CLASSES) {+ for (const raw of RAW_ROLES[platform][roleClass]) {+ const key = normalizedRole(raw);+ const existing = table.get(key);+ if (existing !== undefined && existing !== roleClass) {+ throw new Error(`${platform} roles normalized to ${key} map to both ${existing} and ${roleClass}`);+ }+ table.set(key, roleClass);+ }+ }+ return table;+}++export const ROLE_TABLES: Readonly<Record<Platform, ReadonlyMap<string, RoleClass>>> = {+ macos: normalizedTable('macos'),+ windows: normalizedTable('windows'),+ linux: normalizedTable('linux'),+};++/** Return the role class for a raw Driver role, or null when excluded. */+export function roleClass(rawRole: unknown, platform: Platform): RoleClass | null {+ if (typeof rawRole !== 'string' || !rawRole) return null;+ return ROLE_TABLES[platform].get(normalizedRole(rawRole)) ?? null;+}diff --git a/libs/cua-driver/examples/jev-use/typescript/native_source.test.ts b/libs/cua-driver/examples/jev-use/typescript/native_source.test.tsnew file mode 100644index 0000000000..1739f38c19--- /dev/null+++ b/libs/cua-driver/examples/jev-use/typescript/native_source.test.ts@@ -0,0 +1,191 @@+// NativeAccessibilitySource, stable IDs, policy, and AppKit tasks (RFC #4268).+// Mirrors python/tests/test_native_source.py.+import assert from 'node:assert/strict';+import { mkdtempSync, readFileSync, writeFileSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import test from 'node:test';++import { validateRequest } from './choose_action.js';+import { chooseMockForTask } from './jev_adapter.js';+import { eligibleControls, parseWindowState, riskCategories, slug } from './native.js';+import type { Platform } from './native_roles.js';+import {+ MAX_EXECUTABLE_CANDIDATES,+ OracleError,+ appkitTask,+ compose,+ nativeChoiceRequest,+ type NativeTask,+} from './native_tasks.js';+import { immutableCandidate, NativeAccessibilitySource, type Candidate } from './sources.js';+import type { TaskSources } from './tasks.js';++function fixture(name: string): any {+ return JSON.parse(readFileSync(new URL(`../fixtures/native/${name}`, import.meta.url), 'utf8'));+}+const windowState = (name = 'initial') => fixture(`appkit-window-state-${name}-v1.json`);+const observe = (payload: any) => parseWindowState(payload, payload.pid, payload.window_id);++function sources(task: NativeTask, name = 'initial', foreground: string[] = []): TaskSources {+ return {+ ax: NativeAccessibilitySource.fromObservation(observe(windowState(name)), 'macos', {+ redact: task.redactText,+ textMethod: task.textMethod,+ }),+ visualPath: false,+ foregroundIds: new Set(foreground),+ };+}++test('shared ID, slug, and risk fixture matches Python', () => {+ const shared = fixture('native-candidate-ids-v1.json');+ for (const item of shared.cases) {+ const ids = eligibleControls(observe(item.window_state), item.platform as Platform).controls.map((c) => c.id);+ assert.deepEqual(ids, item.expected_ids, item.name);+ }+ for (const [label, expected] of Object.entries(shared.slugs)) assert.equal(slug(label), expected);+ for (const [label, expected] of Object.entries(shared.risks)) {+ assert.deepEqual([...riskCategories(label)].sort(), expected);+ }+});++test('AppKit fixture controls, exclusions, and ID stability', () => {+ const initial = eligibleControls(observe(windowState()), 'macos');+ const ids = initial.controls.map((control) => control.id);+ assert.deepEqual(ids, [+ 'ax:button:increment',+ 'ax:button:reset',+ 'ax:button:click-target-left-right-double',+ 'ax:checkbox:i-agree',+ 'ax:button:right-click-for-context-menu',+ 'ax:button:exit',+ 'ax:text_input:note',+ 'ax:button:save-note',+ 'ax:radio:small',+ 'ax:radio:medium',+ 'ax:radio:large',+ ]);+ assert.ok(initial.excluded.off_screen > 0 && initial.excluded.unlabeled > 0 && initial.excluded.unknown_role > 0);+ const after = eligibleControls(observe(windowState('after-actions')), 'macos');+ assert.deepEqual(after.controls.map((control) => control.id), ids);+ assert.throws(() => parseWindowState(windowState(), 1, 2));+});++test('each element rule excludes', () => {+ const base = windowState();+ const window = base.elements[0];+ const make = (extra: Record<string, unknown>, drop?: string) => {+ const item: Record<string, unknown> = {+ element_index: 1, role: 'AXButton', depth: 1, parent_index: window.element_index, enabled: true,+ element_token: 's1:1', label: 'Go', frame: { x: window.frame.x + 5, y: window.frame.y + 5, w: 30, h: 20 }, ...extra,+ };+ if (drop) delete item[drop];+ return { ...base, elements: [window, item] };+ };+ const cases: [string, any][] = [+ ['unknown role', make({ role: 'AXStaticText' })],+ ['disabled', make({ enabled: false })],+ ['off screen', make({ frame: { x: 99999, y: 99999, w: 10, h: 10 } })],+ ['zero size', make({ frame: { x: window.frame.x, y: window.frame.y, w: 0, h: 10 } })],+ ['no frame', make({}, 'frame')],+ ['unlabeled', make({}, 'label')],+ ['label equals value', make({ role: 'AXTextField', label: 'typed', value: 'typed' })],+ ['marked unlabelled', make({ unlabelled: true })],+ ['web content', make({ in_web_content: true })],+ ['no token', make({}, 'element_token')],+ ];+ for (const [name, payload] of cases) {+ assert.deepEqual(eligibleControls(observe(payload), 'macos').controls, [], name);+ }+ assert.equal(eligibleControls(observe(make({})), 'macos').controls[0].id, 'ax:button:go');+});++test('compose orders by source, dedups, filters risk, and caps', () => {+ const cand = (id: string, source: 'page' | 'ax' | 'visual', risk: string[] = []): Candidate =>+ immutableCandidate({ id, description: 'd', tool: 'click', arguments: {}, source, risk: new Set(risk) });+ const ax = Array.from({ length: 30 }, (_, n) => cand(`ax:button:b${n}`, 'ax'));+ const { candidates, stats } = compose(+ {+ visual: [cand('v1', 'visual')],+ ax: [cand('ax:button:delete', 'ax', ['destructive']), ...ax],+ page: [cand('ax:button:b0', 'page'), cand('p1', 'page')],+ },+ new Set()+ );+ const ids = candidates.map((candidate) => candidate.id);+ assert.deepEqual(ids.slice(0, 3), ['ax:button:b0', 'p1', 'ax:button:b1']);+ assert.equal(candidates[0].source, 'page');+ assert.deepEqual(ids.slice(-2), ['reobserve', 'abstain']);+ assert.equal(ids.length, MAX_EXECUTABLE_CANDIDATES + 2);+ assert.equal(stats.duplicates, 1);+ assert.deepEqual(stats.risk_excluded, { destructive: 1 });+ assert.equal(stats.dropped, 32 - MAX_EXECUTABLE_CANDIDATES);+});++test('save-note task candidates, arguments, and redacted v2 request', () => {+ const task = appkitTask('appkit-save-note', '/tmp/none.json', { noteText: 'secret note' });+ const step = task.plan(sources(task));+ const byId = new Map(step.candidates.map((candidate) => [candidate.id, candidate]));+ assert.equal(byId.has('ax:button:reset'), false);+ assert.equal(byId.has('ax:button:exit'), false);+ const setter = byId.get('ax:text_input:note:set:note')!;+ assert.equal(setter.tool, 'set_value');+ assert.equal(setter.arguments.value, 'secret note');+ assert.equal(setter.arguments.element_token, 's00000086:21');+ assert.deepEqual({ ...byId.get('ax:button:save-note')!.arguments }, {+ pid: 20535, window_id: 71364, element_token: 's00000086:22', delivery_mode: 'background',+ });+ const request = nativeChoiceRequest(task, sources(task), step, []);+ validateRequest(request);+ const wire = JSON.stringify(request);+ assert.equal(wire.includes('secret note'), false);+ assert.equal(wire.includes('element_token'), false);+ assert.equal(wire.includes('s00000086:'), false);+});++test('satisfied controls leave the set and foreground needs permission', () => {+ const note = appkitTask('appkit-save-note', '/tmp/none.json', { noteText: 'hello jev' });+ assert.equal(note.candidates(sources(note, 'after-actions')).some((c) => c.id === 'ax:text_input:note:set:note'), false);+ const size = appkitTask('appkit-choose-size', '/tmp/none.json');+ const sizeIds = size.candidates(sources(size, 'after-actions')).map((c) => c.id);+ assert.equal(sizeIds.includes('ax:radio:large'), false);+ const denied = appkitTask('appkit-counter', '/tmp/none.json');+ const deniedIds = denied.candidates(sources(denied, 'initial', ['ax:button:increment'])).map((c) => c.id);+ assert.equal(deniedIds.some((id) => id.startsWith('ax:button:increment')), false);+ const allowed = appkitTask('appkit-counter', '/tmp/none.json', { allowForeground: true });+ const fg = allowed+ .candidates(sources(allowed, 'initial', ['ax:button:increment']))+ .find((c) => c.id === 'ax:button:increment:foreground')!;+ assert.equal(fg.arguments.delivery_mode, 'foreground');+});++test('mock provider follows task preferences', () => {+ for (const [id, expected] of [+ ['appkit-counter', 'ax:button:increment'],+ ['appkit-save-note', 'ax:text_input:note:set:note'],+ ['appkit-choose-size', 'ax:radio:large'],+ ]) {+ const task = appkitTask(id, '/tmp/none.json');+ const taskSources = sources(task);+ assert.equal(chooseMockForTask(task, taskSources, task.candidates(taskSources), []).choice, expected);+ }+});++test('app-state oracle and classification', async () => {+ const directory = mkdtempSync(join(tmpdir(), 'jev-native-'));+ const path = join(directory, 'state.json');+ const state = { schema: 'cua.appkit_task_state_v1', pid: 42, counter: 3, agreed: true, size: 'large', note_saved: 'n' };+ writeFileSync(path, JSON.stringify(state));+ const counter = appkitTask('appkit-counter', path, { pid: 42 });+ assert.equal(counter.classify(await counter.readOracle(), 3), 'verified');+ assert.equal(counter.classify({ ...state, counter: 4 }, 4), 'refuted');+ assert.equal(counter.classify({ ...state, counter: 1 }, 6), 'budget_exhausted');+ assert.equal(appkitTask('appkit-save-note', path, { noteText: 'm' }).classify(state, 2), 'refuted');+ assert.equal(appkitTask('appkit-choose-size', path).classify(state, 2), 'verified');+ await assert.rejects(appkitTask('appkit-counter', path, { pid: 43 }).readOracle(), OracleError);+ const entry = appkitTask('appkit-save-note', path, { noteText: 'topsecret' }).historyEntry(1, 'x', undefined, {+ outcome: 'typed topsecret',+ });+ assert.equal(entry.outcome, 'typed [note text]');+});diff --git a/libs/cua-driver/examples/jev-use/typescript/native_tasks.ts b/libs/cua-driver/examples/jev-use/typescript/native_tasks.tsnew file mode 100644index 0000000000..1239895940--- /dev/null+++ b/libs/cua-driver/examples/jev-use/typescript/native_tasks.ts@@ -0,0 +1,497 @@+/**+ * Native task specs for jev-use (RFC #4268, Phase 1). Mirrors+ * python/native_tasks.py: a NativeTask declares goal, parameters (the only+ * source of text), window scope, allowed action kinds, opt-in risks,+ * foreground permission, step budget, and an app-owned oracle. The built-in+ * tasks drive the AppKit harness launched with CUA_APPKIT_TASK_STATE, whose+ * JSON state file is the oracle.+ */+import { readFile } from 'node:fs/promises';++import { MAX_ELEMENTS, REQUEST_SCHEMA_V2 } from './choose_action.js';+import { PARAMETER_NAME_PATTERN, elementState, fieldState, slug, type NativeControl } from './native.js';+import { ACTION_KINDS, type ActionKind } from './native_roles.js';+import { immutableCandidate, type Candidate, type NativeAccessibilitySource, type TextMethod } from './sources.js';+import { redactToken, type HistoryEntry, type Outcome, type Task, type TaskParameter, type TaskSources } from './tasks.js';++export const MAX_EXECUTABLE_CANDIDATES = 24;+const MAX_HISTORY = 16;+const SOURCE_ORDER = ['page', 'ax', 'visual'] as const;+export type Check = 'verified' | 'refuted' | 'pending';++export const NATIVE_RESERVED: readonly Candidate[] = [+ immutableCandidate({+ id: 'reobserve',+ description:+ 'Take no action and obtain a fresh observation of the window, because the current ' ++ "observation looks stale, incomplete, or contradicts the goal's progress.",+ tool: null,+ arguments: {},+ }),+ immutableCandidate({+ id: 'abstain',+ description:+ 'Stop without acting if none of the proposed actions is safe or moves toward the goal.',+ tool: null,+ arguments: {},+ }),+];++const VERBS: Readonly<Record<string, string>> = {+ press: 'pressed',+ toggle: 'toggled',+ select: 'selected',+ open_menu: 'opened',+ visual_click: 'clicked the visual region',+};++export type WindowScope = Readonly<{+ windowTitle: string;+ bundleId?: string;+ processName?: string;+ query?: string;+ maxElements?: number;+ maxDepth?: number;+}>;++export function windowStateArguments(scope: WindowScope): Record<string, unknown> {+ return {+ ...(scope.query !== undefined ? { query: scope.query } : {}),+ ...(scope.maxElements !== undefined ? { max_elements: scope.maxElements } : {}),+ ...(scope.maxDepth !== undefined ? { max_depth: scope.maxDepth } : {}),+ };+}++export class OracleError extends Error {+ constructor(message: string) {+ super(message);+ this.name = 'OracleError';+ }+}++/** An app_check oracle over an app-owned JSON state file bound to one pid. */+export class AppStateOracle {+ constructor(+ readonly path: string,+ readonly schema: string,+ readonly expectedPid?: number+ ) {}++ async read(): Promise<Record<string, unknown>> {+ let state: unknown;+ try {+ state = JSON.parse(await readFile(this.path, 'utf8'));+ } catch (error) {+ throw new OracleError(`app state is unreadable: ${(error as Error).name}`);+ }+ if (!state || typeof state !== 'object' || (state as Record<string, unknown>).schema !== this.schema) {+ throw new OracleError('app state has an unexpected schema');+ }+ if (this.expectedPid !== undefined && (state as Record<string, unknown>).pid !== this.expectedPid) {+ throw new OracleError('app state belongs to a different process');+ }+ return state as Record<string, unknown>;+ }+}++export type ComposeStats = Readonly<{+ sources: Readonly<Record<string, number>>;+ duplicates: number;+ risk_excluded: Readonly<Record<string, number>>;+ dropped: number;+}>;++/**+ * Merge source outputs in the order page, ax, visual: first source wins on a+ * duplicate ID, unallowed risk categories are removed, at most `cap`+ * executable candidates remain (the drop count is reported), and the reserved+ * candidates are appended.+ */+export function compose(+ groups: Partial<Record<(typeof SOURCE_ORDER)[number], readonly Candidate[]>>,+ allowedRisks: ReadonlySet<string>,+ reserved: readonly Candidate[] = NATIVE_RESERVED,+ cap = MAX_EXECUTABLE_CANDIDATES+): { candidates: Candidate[]; stats: ComposeStats } {+ const seen = new Set(reserved.map((candidate) => candidate.id));+ const merged: Candidate[] = [];+ let duplicates = 0;+ const riskExcluded: Record<string, number> = {};+ for (const source of SOURCE_ORDER) {+ for (const candidate of groups[source] ?? []) {+ if (seen.has(candidate.id)) {+ duplicates += 1;+ continue;+ }+ seen.add(candidate.id);+ const blocked = [...(candidate.risk ?? [])].filter((risk) => !allowedRisks.has(risk)).sort();+ if (blocked.length) {+ for (const category of blocked) riskExcluded[category] = (riskExcluded[category] ?? 0) + 1;+ continue;+ }+ merged.push(candidate);+ }+ }+ const kept = merged.slice(0, cap);+ const counts: Record<string, number> = { page: 0, ax: 0, visual: 0 };+ for (const candidate of kept) {+ if (candidate.source && candidate.source in counts) counts[candidate.source] += 1;+ }+ return {+ candidates: [...kept, ...reserved],+ stats: { sources: counts, duplicates, risk_excluded: riskExcluded, dropped: merged.length - kept.length },+ };+}++export type CompactElement = { role_class: string; label: string; state: string };++export type NativeStep = Readonly<{+ candidates: Candidate[];+ stats: ComposeStats;+ elements: CompactElement[];+ outcomes: Readonly<Record<string, string>>;+}>;++function quoted(label: string, limit = 60): string {+ return JSON.stringify(label.length <= limit ? label : `${label.slice(0, limit - 1)}…`);+}++export type NativeTaskSpec = Readonly<{+ id: string;+ goal: string;+ scope: WindowScope;+ allowedActions: ReadonlySet<ActionKind>;+ oracle: AppStateOracle;+ check: (state: Readonly<Record<string, unknown>>) => Check;+ parameters?: readonly TaskParameter[];+ allowedRisks?: ReadonlySet<string>;+ allowForeground?: boolean;+ maxSteps?: number;+ textMethod?: TextMethod;+ visualTargets?: readonly string[];+ mockPreferences?: readonly string[];+}>;++export class NativeTask implements Task {+ readonly id: string;+ readonly goal: string;+ readonly scope: WindowScope;+ readonly allowedActions: ReadonlySet<ActionKind>;+ readonly oracle: AppStateOracle;+ readonly check: NativeTaskSpec['check'];+ readonly parameters: readonly TaskParameter[];+ readonly allowedRisks: ReadonlySet<string>;+ readonly allowForeground: boolean;+ readonly maxSteps: number;+ readonly textMethod: TextMethod;+ readonly visualTargets: readonly string[];+ readonly mockPreferences: readonly string[];+ /** The oracle is polled after every action, so no candidate is special. */+ readonly completionCandidateIds: ReadonlySet<string> = new Set();++ constructor(spec: NativeTaskSpec) {+ this.id = spec.id;+ this.goal = spec.goal;+ this.scope = spec.scope;+ this.allowedActions = spec.allowedActions;+ this.oracle = spec.oracle;+ this.check = spec.check;+ this.parameters = spec.parameters ?? [];+ this.allowedRisks = spec.allowedRisks ?? new Set();+ this.allowForeground = spec.allowForeground ?? false;+ this.maxSteps = spec.maxSteps ?? 6;+ this.textMethod = spec.textMethod ?? 'set_value';+ this.visualTargets = spec.visualTargets ?? [];+ this.mockPreferences = spec.mockPreferences ?? [];+ const unknown = [...this.allowedActions].filter((action) => !ACTION_KINDS.has(action));+ if (unknown.length) throw new Error(`unknown action kinds: ${unknown.sort().join(', ')}`);+ for (const parameter of this.parameters) {+ if (!PARAMETER_NAME_PATTERN.test(parameter.name)) {+ throw new Error('parameter names must match [a-z][a-z0-9_]{0,7}');+ }+ }+ if (!Number.isInteger(this.maxSteps) || this.maxSteps < 1) throw new Error('maxSteps must be positive');+ }++ get allowedActionKinds(): ReadonlySet<string> {+ const tools = new Set<string>();+ if ([...this.allowedActions].some((action) => action !== 'set_text')) tools.add('click');+ if (this.allowedActions.has('set_text')) tools.add(this.textMethod);+ return tools;+ }++ redact(value: unknown): unknown {+ let result = value;+ for (const parameter of this.parameters) {+ if (parameter.secret) result = redactToken(result, parameter.value, parameter.redaction);+ }+ return result;+ }++ redactText = (value: string): string => this.redact(value) as string;++ private nativeCandidates(ax: NativeAccessibilitySource, foregroundIds: ReadonlySet<string>) {+ const candidates: Candidate[] = [];+ const outcomes: Record<string, string> = {};+ for (const native of ax.controls) {+ if (!this.allowedActions.has(native.action)) continue;+ const control = ax.control(native);+ const label = quoted(native.label);+ if (native.action === 'set_text') {+ for (const parameter of this.parameters) {+ if (native.value === parameter.value) continue;+ const id = `${native.id}:set:${parameter.name}`;+ candidates.push(+ ax.typeText(+ control,+ parameter.value,+ id,+ `Set the text field ${label} to the task parameter ${JSON.stringify(parameter.name)}, ` ++ `replacing its contents. The field currently reports ${fieldState(native.value, parameter.value)}.`+ )+ );+ outcomes[id] = `set ${label} to parameter ${parameter.name}`;+ }+ continue;+ }+ if (native.action === 'select' && native.selected) continue;+ let description = NativeTask.describe(native, label);+ let id = native.id;+ let delivery: 'background' | 'foreground' = 'background';+ if (foregroundIds.has(native.id)) {+ if (!this.allowForeground) continue;+ id = `${native.id}:foreground`;+ delivery = 'foreground';+ description +=+ ' Use foreground delivery, which activates the window, because Driver refused ' ++ 'background delivery for this control.';+ }+ candidates.push(ax.click(control, id, description, delivery));+ outcomes[id] = `${VERBS[native.action]} ${label}`;+ }+ return { candidates, outcomes };+ }++ static describe(native: NativeControl, label: string): string {+ if (native.action === 'toggle') {+ const now = native.selected ? 'checked' : 'unchecked';+ const after = native.selected ? 'unchecked' : 'checked';+ const noun = native.roleClass === 'toggle' ? 'switch' : 'checkbox';+ return `Toggle the ${noun} ${label}. It is currently ${now}; afterward it will be ${after}.`;+ }+ if (native.action === 'select') return `Select the radio option ${label}. It is currently not selected.`;+ if (native.action === 'open_menu') {+ return `Open the pop-up menu ${label}. Its options become candidates on the next observation.`;+ }+ const noun = native.roleClass === 'menu_item' ? 'menu item' : native.roleClass === 'link' ? 'link' : 'button';+ return `Press the ${noun} labeled ${label}.`;+ }++ private visualCandidates(sources: TaskSources) {+ const candidates: Candidate[] = [];+ const outcomes: Record<string, string> = {};+ const visual = sources.visual;+ if (!visual || !this.allowedActions.has('visual_click')) return { candidates, outcomes };+ for (const target of this.visualTargets) {+ const control = visual.find('button', target);+ if (!control) continue;+ const id = `visual:${slug(target)}`;+ const candidate = visual.click(+ control,+ id,+ `Click the unique validated visual region reading ${quoted(target)}; no accessibility element covers it.`+ );+ if (candidate) {+ candidates.push(candidate);+ outcomes[id] = `clicked the visual region ${quoted(target)}`;+ }+ }+ return { candidates, outcomes };+ }++ plan(sources: TaskSources): NativeStep {+ const outcomes: Record<string, string> = {};+ let axCandidates: Candidate[] = [];+ let elements: CompactElement[] = [];+ if (sources.ax) {+ const native = this.nativeCandidates(sources.ax, sources.foregroundIds ?? new Set());+ axCandidates = native.candidates;+ Object.assign(outcomes, native.outcomes);+ elements = sources.ax.controls.slice(0, MAX_ELEMENTS).map((control) => ({+ role_class: control.roleClass,+ label: control.label,+ state: elementState(control),+ }));+ }+ const visual = this.visualCandidates(sources);+ Object.assign(outcomes, visual.outcomes);+ const { candidates, stats } = compose({ ax: axCandidates, visual: visual.candidates }, this.allowedRisks);+ for (const candidate of candidates) {+ if (candidate.tool !== null && !this.allowedActionKinds.has(candidate.tool)) {+ throw new Error(`task ${this.id} does not allow action kind ${candidate.tool}`);+ }+ }+ return { candidates, stats, elements, outcomes };+ }++ candidates(sources: TaskSources): Candidate[] {+ return this.plan(sources).candidates;+ }++ stateSummary(sources: TaskSources): Readonly<Record<string, string>> {+ return Object.fromEntries((sources.ax?.controls ?? []).map((control) => [control.id, elementState(control)]));+ }++ historyEntry(+ step: number,+ candidateId: string,+ refusal?: string,+ options: { outcome?: string; stale?: boolean } = {}+ ): HistoryEntry {+ const text = options.stale+ ? 'the observation was stale; nothing happened and the window is observed again'+ : refusal !== undefined+ ? `Driver refused background delivery (${refusal}); nothing happened`+ : candidateId === 'reobserve'+ ? 'took no action and requested a fresh observation'+ : (options.outcome ?? 'completed');+ return { step, selected_id: candidateId, outcome: (this.redact(text) as string).slice(0, 128) };+ }++ async reset(): Promise<void> {+ // The harness starts fresh for every run.+ }++ readOracle(): Promise<Record<string, unknown>> {+ return this.oracle.read();+ }++ classify(oracle: Readonly<Record<string, unknown>>, steps: number): Outcome {+ const result = this.check(oracle);+ if (result === 'verified' || result === 'refuted') return result;+ return steps >= this.maxSteps ? 'budget_exhausted' : 'unknown';+ }+}++/** Why this step may parse visual regions, or undefined (see python/native_tasks.py). */+export function visualFallbackReason(+ sources: TaskSources,+ task: NativeTask,+ nativeCount: number+): string | undefined {+ const ax = sources.ax;+ if (!ax || !task.allowedActions.has('visual_click') || !task.visualTargets.length) return undefined;+ const observation = ax.observation;+ if (!observation.captureId) return undefined;+ if (observation.degradedReason?.startsWith('ax_tree_empty')) return 'tree_empty';+ if (!observation.complete || observation.truncated) return undefined;+ if (nativeCount === 0) return 'no_native_candidates';+ const labels = new Set(ax.controls.map((control) => control.label.toLowerCase()));+ if (task.visualTargets.some((target) => !labels.has(target.toLowerCase()))) return 'target_without_element';+ return undefined;+}++/** Build the cua.jev_choice_request_v2 the provider receives; no tokens, values, or pixels. */+export function nativeChoiceRequest(+ task: NativeTask,+ sources: TaskSources,+ step: NativeStep,+ history: readonly HistoryEntry[]+): Record<string, unknown> {+ const observation = sources.ax?.observation;+ if (!observation?.captureId) throw new Error('a native request needs an observation with a capture_id');+ const regions = (sources.visual?.observation.regions ?? []).map((region) =>+ task.redact({+ id: region.id,+ kind: region.kind,+ bounds: { x: region.x, y: region.y, width: region.width, height: region.height },+ text: region.text ?? null,+ label: region.label ?? null,+ confidence: region.confidence,+ interactive: region.interactive,+ })+ );+ return {+ schema: REQUEST_SCHEMA_V2,+ goal: task.redact(task.goal),+ capture_id: observation.captureId,+ snapshot_id: observation.snapshotId ?? null,+ regions,+ elements: step.elements.map((item) => task.redact(item)),+ history: history.slice(-MAX_HISTORY).map((item) => ({ selected_id: item.selected_id, outcome: item.outcome })),+ candidates: step.candidates.map((candidate) => ({+ id: candidate.id,+ description: task.redact(candidate.description),+ ...(candidate.source ? { source: candidate.source } : {}),+ })),+ };+}++// AppKit harness tasks.++export const APPKIT_WINDOW_TITLE = 'CuaTestHarness AppKit';+export const APPKIT_BUNDLE_ID = 'com.trycua.harness.appkit';+export const APPKIT_STATE_SCHEMA = 'cua.appkit_task_state_v1';+export const COUNTER_TARGET = 3;+export const DEFAULT_NOTE_TEXT = 'jev-use native note';+export const APPKIT_TASK_IDS = ['appkit-counter', 'appkit-save-note', 'appkit-choose-size'] as const;+export type AppKitTaskId = (typeof APPKIT_TASK_IDS)[number];++export function appkitTask(+ taskId: string,+ statePath: string,+ options: { pid?: number; noteText?: string; allowForeground?: boolean } = {}+): NativeTask {+ const oracle = new AppStateOracle(statePath, APPKIT_STATE_SCHEMA, options.pid);+ const scope: WindowScope = { windowTitle: APPKIT_WINDOW_TITLE, bundleId: APPKIT_BUNDLE_ID };+ const allowForeground = options.allowForeground ?? false;+ if (taskId === 'appkit-counter') {+ return new NativeTask({+ id: taskId,+ goal:+ `The counter starts at 0. Set it to exactly ${COUNTER_TARGET} by pressing ` ++ 'Increment once per step, then stop.',+ scope,+ allowedActions: new Set<ActionKind>(['press']),+ oracle,+ check: (state) => {+ const counter = state.counter;+ if (counter === COUNTER_TARGET) return 'verified';+ return typeof counter === 'number' && counter > COUNTER_TARGET ? 'refuted' : 'pending';+ },+ allowForeground,+ maxSteps: COUNTER_TARGET + 3,+ mockPreferences: ['ax:button:increment'],+ });+ }+ if (taskId === 'appkit-save-note') {+ const note = options.noteText ?? DEFAULT_NOTE_TEXT;+ return new NativeTask({+ id: taskId,+ goal: 'Enter the note text into the Note field, then save the note.',+ scope,+ allowedActions: new Set<ActionKind>(['press', 'set_text']),+ oracle,+ check: (state) =>+ state.note_saved === note ? 'verified' : state.note_saved != null ? 'refuted' : 'pending',+ parameters: [{ name: 'note', value: note, secret: true, redaction: '[note text]' }],+ allowForeground,+ maxSteps: 5,+ mockPreferences: ['ax:text_input:note:set:note', 'ax:button:save-note'],+ });+ }+ if (taskId === 'appkit-choose-size') {+ return new NativeTask({+ id: taskId,+ goal: 'Choose the Large size option and check the I agree checkbox.',+ scope,+ allowedActions: new Set<ActionKind>(['select', 'toggle']),+ oracle,+ check: (state) => (state.size === 'large' && state.agreed === true ? 'verified' : 'pending'),+ allowForeground,+ maxSteps: 5,+ mockPreferences: ['ax:radio:large', 'ax:checkbox:i-agree'],+ });+ }+ throw new Error(`unknown AppKit task: ${taskId}`);+}diff --git a/libs/cua-driver/examples/jev-use/typescript/run.ts b/libs/cua-driver/examples/jev-use/typescript/run.tsindex 853153b5d6..52e7d868b8 100644--- a/libs/cua-driver/examples/jev-use/typescript/run.ts+++ b/libs/cua-driver/examples/jev-use/typescript/run.ts@@ -112,7 +112,13 @@ export class Driver {});if (result.isError) {const structured = result.structuredContent as Record<string, unknown> | undefined;- const code = typeof structured?.code === 'string' && structured.code ? structured.code : undefined;+ const refusalCode = (structured?.refusal as Record<string, unknown> | undefined)?.code;+ const code =+ typeof structured?.code === 'string' && structured.code+ ? structured.code+ : typeof refusalCode === 'string' && refusalCode+ ? refusalCode+ : undefined;const escalation = structured?.escalation as Record<string, unknown> | undefined;const recommended =typeof escalation?.recommended === 'string' && escalation.recommended@@ -127,7 +133,12 @@ export class Driver {const data = result.structuredContent as Record<string, any> | undefined;if (!data) throw new Error(`${name} returned no structured result`);if (data.status === 'refused' || data.refusal) {- throw new Error(`${name} refused: ${JSON.stringify(data.refusal ?? data)}`);+ const code = data.refusal?.code;+ // DriverToolError is an Error, so existing handlers still match.+ throw new DriverToolError(+ `${name} refused: ${JSON.stringify(data.refusal ?? data)}`,+ typeof code === 'string' && code ? code : undefined+ );}return data;}diff --git a/libs/cua-driver/examples/jev-use/typescript/run_native.ts b/libs/cua-driver/examples/jev-use/typescript/run_native.tsnew file mode 100644index 0000000000..6372c1ec76--- /dev/null+++ b/libs/cua-driver/examples/jev-use/typescript/run_native.ts@@ -0,0 +1,361 @@+/**+ * Run one native jev-use task against a desktop window (RFC #4268, Phase 1).+ * Mirrors python/run_native.py: attach to --pid, find the task window, and+ * loop oracle -> one get_window_state (tree and screenshot) -> closed+ * candidates -> validated cua.jev_choice_request_v2 -> one element-bound+ * action -> oracle.+ */+import { appendFile, writeFile } from 'node:fs/promises';+import process from 'node:process';+import { randomUUID } from 'node:crypto';+import { pathToFileURL } from 'node:url';++import { Client } from '@modelcontextprotocol/sdk/client/index.js';+import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';+import { TypeSafeClient } from '@typesafe-ai/sdk';++import { providerObservation, validateRequest } from './choose_action.js';+import { parseVisualRegions, validateChoice, VisualObservationError, type Candidate } from './core.js';+import { driverEnvironment } from './driver_env.js';+import { chooseBoundedWithTypeSafe, chooseMockForTask } from './jev_adapter.js';+import { parseWindowState, type NativeObservation } from './native.js';+import type { Platform } from './native_roles.js';+import {+ APPKIT_TASK_IDS,+ appkitTask,+ nativeChoiceRequest,+ visualFallbackReason,+ windowStateArguments,+ type NativeTask,+} from './native_tasks.js';+import { backgroundRefusalCode, Driver, DriverToolError, supportsCaptureBoundClick } from './run.js';+import { NativeAccessibilitySource, VisualRegionSource } from './sources.js';+import type { HistoryEntry, Outcome, TaskSources } from './tasks.js';++const STALE_TOKEN_CODES = new Set(['stale_element_token']);+const REOBSERVE_TIMEOUT_MS = 5_000;++type Arguments = {+ task: string;+ provider: 'mock' | 'live';+ pid: number;+ stateFile: string;+ noteText?: string;+ allowForeground: boolean;+ platform?: Platform;+ log?: string;+};++export function hostPlatform(): Platform {+ if (process.platform === 'darwin') return 'macos';+ if (process.platform === 'win32') return 'windows';+ return 'linux';+}++export function parseArgs(argv: string[]): Arguments {+ const result: Partial<Arguments> = { provider: 'mock', allowForeground: false };+ for (let index = 0; index < argv.length; index += 1) {+ const value = argv[index];+ if (value === '--task') result.task = argv[++index];+ else if (value === '--provider') result.provider = argv[++index] as Arguments['provider'];+ else if (value === '--pid') result.pid = Number(argv[++index]);+ else if (value === '--state-file') result.stateFile = argv[++index];+ else if (value === '--note-text') result.noteText = argv[++index];+ else if (value === '--allow-foreground') result.allowForeground = true;+ else if (value === '--platform') result.platform = argv[++index] as Platform;+ else if (value === '--log') result.log = argv[++index];+ else throw new Error(`unknown argument: ${value}`);+ }+ if (!result.task || !(APPKIT_TASK_IDS as readonly string[]).includes(result.task)) {+ throw new Error(`--task must be one of ${APPKIT_TASK_IDS.join(', ')}`);+ }+ if (result.provider !== 'mock' && result.provider !== 'live') throw new Error('--provider must be mock or live');+ if (!Number.isInteger(result.pid) || (result.pid as number) <= 0) throw new Error('--pid is required');+ if (!result.stateFile) throw new Error('--state-file is required');+ if (result.platform && !['macos', 'windows', 'linux'].includes(result.platform)) {+ throw new Error('--platform must be macos, windows, or linux');+ }+ return result as Arguments;+}++export function isStaleTokenError(error: unknown): boolean {+ if (error instanceof DriverToolError && error.code && STALE_TOKEN_CODES.has(error.code)) return true;+ return String(error instanceof Error ? error.message : error).includes('element_token is stale');+}++/** Refuse any action addressed outside the task's one window. */+export function assertInScope(candidate: Candidate, pid: number, windowId: number): void {+ if (candidate.tool === null) return;+ if (candidate.arguments.pid !== pid || candidate.arguments.window_id !== windowId) {+ throw new Error('candidate addresses a window outside the task scope');+ }+}++async function writeEvent(path: string | undefined, event: Record<string, unknown>) {+ const line = JSON.stringify(event);+ console.log(line);+ if (path) await appendFile(path, `${line}\n`, 'utf8');+}++const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));++async function findWindow(driver: Driver, pid: number, title: string) {+ for (let attempt = 0; attempt < 40; attempt += 1) {+ const windows = ((await driver.call('list_windows', { pid })).windows ?? []) as Record<string, any>[];+ const match = windows.find((window) => window.title === title && window.is_on_screen !== false);+ if (match) return match;+ await sleep(250);+ }+ throw new Error(`window ${JSON.stringify(title)} of pid ${pid} did not appear`);+}++async function observe(+ driver: Driver,+ task: NativeTask,+ pid: number,+ windowId: number,+ timeoutMs?: number+): Promise<NativeObservation> {+ const payload = await driver.call('get_window_state', {+ pid,+ window_id: windowId,+ include_accessibility_tree: true,+ include_screenshot: true,+ ...windowStateArguments(task.scope),+ ...(timeoutMs !== undefined ? { timeout_ms: timeoutMs } : {}),+ });+ return parseWindowState(payload, pid, windowId);+}++function nativeCount(task: NativeTask, sources: TaskSources): number {+ return task.plan(sources).candidates.filter((candidate) => candidate.source === 'ax').length;+}++async function observeStep(+ driver: Driver,+ task: NativeTask,+ pid: number,+ windowId: number,+ platform: Platform,+ foregroundIds: ReadonlySet<string>+): Promise<{ sources: TaskSources; record: Record<string, unknown> }> {+ const started = performance.now();+ const build = (observation: NativeObservation): TaskSources => ({+ ax: NativeAccessibilitySource.fromObservation(observation, platform, {+ redact: task.redactText,+ textMethod: task.textMethod,+ }),+ visualPath: false,+ foregroundIds,+ });+ let sources = build(await observe(driver, task, pid, windowId));+ let record: Record<string, unknown> = { reobserved: false };+ const first = sources.ax!.observation;+ if (first.truncated || (!first.complete && nativeCount(task, sources) === 0)) {+ record = { reobserved: true, reason: first.truncated ? 'truncated' : 'partial_empty' };+ sources = build(await observe(driver, task, pid, windowId, REOBSERVE_TIMEOUT_MS));+ }+ const observation = sources.ax!.observation;+ Object.assign(record, {+ snapshot_id: observation.snapshotId ?? null,+ capture_id: observation.captureId ?? null,+ elements_complete: observation.complete,+ truncated: observation.truncated,+ controls: sources.ax!.controls.length,+ excluded: sources.ax!.native.excluded,+ observe_ms: Math.round((performance.now() - started) * 100) / 100,+ });+ return { sources, record };+}++async function maybeVisual(+ driver: Driver,+ task: NativeTask,+ sources: TaskSources,+ availableTools: ReadonlySet<string>,+ captureBoundClick: boolean+): Promise<{ sources: TaskSources; record: Record<string, unknown> }> {+ const reason = visualFallbackReason(sources, task, nativeCount(task, sources));+ if (!reason) return { sources, record: { status: 'skipped' } };+ if (!captureBoundClick || !availableTools.has('parse_visual_regions')) {+ return { sources, record: { status: 'unavailable', reason } };+ }+ const observation = sources.ax!.observation;+ try {+ const result = await driver.call('parse_visual_regions', {+ capture_id: observation.captureId,+ options: { kinds: ['text', 'icon'], min_confidence: 0.8, max_regions: 100 },+ });+ const visual = parseVisualRegions(result, observation.captureId!, observation.pid, observation.windowId);+ return {+ sources: { ...sources, visual: new VisualRegionSource(visual, 'background', captureBoundClick), visualPath: true },+ record: { status: 'ok', reason, region_count: visual.regions.length },+ };+ } catch (error) {+ if (error instanceof DriverToolError) {+ return { sources, record: { status: 'error', reason, error_code: error.code ?? 'driver_error' } };+ }+ if (error instanceof VisualObservationError) {+ return { sources, record: { status: 'error', reason, error_code: error.code } };+ }+ throw error;+ }+}++async function chooseLive(request: Record<string, unknown>) {+ const validated = validateRequest(request);+ const criteria = Object.fromEntries(validated.candidates.map(({ id, description }) => [id, description]));+ const result = await chooseBoundedWithTypeSafe(+ new TypeSafeClient(),+ validated.goal,+ providerObservation(validated),+ criteria+ );+ return { choice: result.selectedId, confidence: result.confidence, probabilities: result.probabilities };+}++async function pollOracle(task: NativeTask, steps: number): Promise<Outcome> {+ let outcome: Outcome = 'unknown';+ for (let attempt = 0; attempt < 20; attempt += 1) {+ outcome = task.classify(await task.readOracle(), steps);+ if (outcome === 'verified' || outcome === 'refuted') return outcome;+ await sleep(100);+ }+ return outcome;+}++export async function runTask(args: Arguments, task: NativeTask): Promise<Outcome> {+ const platform = args.platform ?? hostPlatform();+ if (args.log) await writeFile(args.log, '', 'utf8');+ const history: HistoryEntry[] = [];+ const foregroundIds = new Set<string>();+ const transport = new StdioClientTransport({+ command: process.env.CUA_DRIVER_BIN ?? 'cua-driver',+ args: ['mcp'],+ env: driverEnvironment(),+ });+ const client = new Client({ name: 'cua-driver-jev-use-native', version: '0.1.0' });+ try {+ await client.connect(transport);+ const advertised = (await client.listTools()).tools;+ const availableTools = new Set(advertised.map((tool) => tool.name));+ const captureBoundClick = supportsCaptureBoundClick(advertised);+ const driver = new Driver(client, `jev-native-typescript-${randomUUID().slice(0, 8)}`);+ const window = await findWindow(driver, args.pid, task.scope.windowTitle);+ const windowId = Number(window.window_id);+ await writeEvent(args.log, {+ event: 'start', task: task.id, language: 'typescript', provider: args.provider,+ platform, pid: args.pid, window_id: windowId,+ });+ for (let step = 1; step <= task.maxSteps; step += 1) {+ const current = task.classify(await task.readOracle(), step - 1);+ if (current === 'verified' || current === 'refuted') {+ await writeEvent(args.log, { event: 'outcome', outcome: current, step: step - 1 });+ return current;+ }+ const observed = await observeStep(driver, task, args.pid, windowId, platform, new Set(foregroundIds));+ const visual = await maybeVisual(driver, task, observed.sources, availableTools, captureBoundClick);+ const sources = visual.sources;+ const plan = task.plan(sources);+ const request = nativeChoiceRequest(task, sources, plan, history);+ validateRequest(request);++ const decideStarted = performance.now();+ const decision =+ args.provider === 'mock'+ ? chooseMockForTask(task, sources, plan.candidates, history)+ : await chooseLive(request);+ const baseEvent: Record<string, unknown> = {+ event: 'step',+ step,+ observation: observed.record,+ visual: visual.record,+ compose: plan.stats,+ candidate_count: plan.candidates.length,+ schema: request.schema,+ confidence: decision.confidence,+ probabilities: decision.probabilities,+ decide_ms: Math.round((performance.now() - decideStarted) * 100) / 100,+ };+ if (!decision.choice) {+ await writeEvent(args.log, { ...baseEvent, event: 'outcome', outcome: 'abstained' });+ return 'abstained';+ }+ const candidate = validateChoice(decision.choice, plan.candidates, sources.visual?.observation.captureId);+ assertInScope(candidate, args.pid, windowId);+ baseEvent.candidate = candidate.id;+ baseEvent.source = candidate.source ?? null;++ if (candidate.id === 'reobserve') {+ history.push(task.historyEntry(step, candidate.id));+ await writeEvent(args.log, { ...baseEvent, tool: null, act_ms: 0 });+ continue;+ }+ if (candidate.id === 'abstain') {+ await writeEvent(args.log, { ...baseEvent, event: 'outcome', outcome: 'abstained' });+ return 'abstained';+ }+ const actStarted = performance.now();+ try {+ await driver.call(candidate.tool!, { ...candidate.arguments });+ } catch (error) {+ const actMs = Math.round((performance.now() - actStarted) * 100) / 100;+ if (isStaleTokenError(error)) {+ history.push(task.historyEntry(step, candidate.id, undefined, { stale: true }));+ await writeEvent(args.log, { ...baseEvent, tool: candidate.tool, act_ms: actMs, action_error: 'stale_element_token' });+ continue;+ }+ const refusal = backgroundRefusalCode(candidate, error);+ if (refusal) {+ foregroundIds.add(candidate.id);+ history.push(task.historyEntry(step, candidate.id, refusal));+ await writeEvent(args.log, {+ ...baseEvent, tool: candidate.tool, act_ms: actMs, action_error: refusal,+ escalation: { from: 'background', to: 'foreground', allowed: task.allowForeground },+ });+ continue;+ }+ await writeEvent(args.log, {+ ...baseEvent, event: 'outcome', outcome: 'unknown', phase: 'action',+ error: error instanceof Error ? error.name : 'Error', tool: candidate.tool,+ });+ return 'unknown';+ }+ const actMs = Math.round((performance.now() - actStarted) * 100) / 100;+ history.push(task.historyEntry(step, candidate.id, undefined, { outcome: plan.outcomes[candidate.id] }));+ await writeEvent(args.log, {+ ...baseEvent, tool: candidate.tool, act_ms: actMs,+ delivery_mode: candidate.arguments.delivery_mode ?? null,+ });+ const outcome = await pollOracle(task, step);+ if (outcome === 'verified' || outcome === 'refuted') {+ await writeEvent(args.log, { event: 'outcome', outcome, step });+ return outcome;+ }+ }+ const outcome = task.classify(await task.readOracle(), task.maxSteps);+ await writeEvent(args.log, { event: 'outcome', outcome, step: task.maxSteps });+ return outcome;+ } finally {+ await client.close();+ }+}++if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {+ const args = parseArgs(process.argv.slice(2));+ const task = appkitTask(args.task, args.stateFile, {+ pid: args.pid,+ noteText: args.noteText,+ allowForeground: args.allowForeground,+ });+ runTask(args, task)+ .then((outcome) => {+ process.exitCode = outcome === 'verified' ? 0 : 1;+ })+ .catch(async (error: unknown) => {+ await writeEvent(args.log, {+ event: 'outcome', outcome: 'unknown', error: error instanceof Error ? error.name : 'Error',+ });+ process.exitCode = 1;+ });+}diff --git a/libs/cua-driver/examples/jev-use/typescript/sources.ts b/libs/cua-driver/examples/jev-use/typescript/sources.tsindex 9e3c5b3e43..9083c98b61 100644--- a/libs/cua-driver/examples/jev-use/typescript/sources.ts+++ b/libs/cua-driver/examples/jev-use/typescript/sources.ts@@ -14,11 +14,9 @@* acts through browser_type / browser_click on its refs.* - VisualRegionSource reads a validated parse_visual_regions result and acts* through a capture-bound click on a region's center.- *- * Extension point (RFC #4268, Phase 1): a NativeAccessibilitySource will read- * get_window_state elements, find controls by normalized role and label, and- * act through the snapshot's element_token. It is deliberately not implemented- * here; it must implement CandidateSource with kind 'ax'.+ * - NativeAccessibilitySource (RFC #4268) reads the eligible native elements of+ * one get_window_state observation (see native.ts), finds controls by role+ * class and label, and acts through that snapshot's element_token.*/import type {BrowserSnapshot,@@ -27,9 +25,16 @@ import type {VisualObservation,VisualRegion,} from './core.js';+import {+ eligibleControls,+ type NativeControl,+ type NativeControls,+ type NativeObservation,+} from './native.js';+import type { Platform } from './native_roles.js';-/** 'ax' is reserved for the future NativeAccessibilitySource. */export type SourceKind = 'page' | 'visual' | 'ax';+export type TextMethod = 'set_value' | 'type_text';export type Candidate = Readonly<{id: string;@@ -38,6 +43,12 @@ export type Candidate = Readonly<{arguments: Readonly<Record<string, unknown>>;captureId?: string;screenshotReference?: string;+ /** RFC #4268: the building source (absent for reserved candidates). */+ source?: SourceKind;+ /** The native snapshot an 'ax' candidate's element_token is bound to. */+ snapshotId?: string;+ /** Risk categories (see native.ts); absent means none. */+ risk?: ReadonlySet<string>;}>;export function immutableCandidate(candidate: Candidate): Candidate {@@ -113,6 +124,7 @@ export class BrowserSemanticSource implements CandidateSource {ref: (control.handle as PageRef).ref,input_route: 'dom_event',},+ source: 'page',});}@@ -127,6 +139,7 @@ export class BrowserSemanticSource implements CandidateSource {text,replace: true,},+ source: 'page',});}}@@ -186,6 +199,7 @@ export class VisualRegionSource implements CandidateSource {},captureId: visual.captureId,screenshotReference: visual.screenshotReference,+ source: 'visual',});}@@ -193,3 +207,82 @@ export class VisualRegionSource implements CandidateSource {return undefined;}}++/**+ * Controls from one get_window_state observation, acted on by element token.+ * click binds the control's element_token with delivery_mode; typeText sets+ * task-supplied text through set_value (default) or element-bound type_text.+ */+export class NativeAccessibilitySource implements CandidateSource {+ readonly kind = 'ax' as const;++ constructor(+ readonly observation: NativeObservation,+ readonly platform: Platform,+ readonly native: NativeControls,+ readonly textMethod: TextMethod = 'set_value'+ ) {}++ static fromObservation(+ observation: NativeObservation,+ platform: Platform,+ options: { redact?: (value: string) => string; textMethod?: TextMethod } = {}+ ): NativeAccessibilitySource {+ return new NativeAccessibilitySource(+ observation,+ platform,+ eligibleControls(observation, platform, options.redact),+ options.textMethod ?? 'set_value'+ );+ }++ get controls(): readonly NativeControl[] {+ return this.native.controls;+ }++ control(native: NativeControl): Control<NativeControl> {+ return { source: 'ax', role: native.roleClass, name: native.label, value: native.value, handle: native };+ }++ find(role: string, name: string): Control<NativeControl> | undefined {+ const matches = this.controls.filter((item) => item.roleClass === role && item.label === name);+ return matches.length === 1 ? this.control(matches[0]) : undefined;+ }++ private target() {+ return { pid: this.observation.pid, window_id: this.observation.windowId };+ }++ click(+ control: Control,+ candidateId: string,+ description: string,+ delivery: VisualDelivery = 'background'+ ): Candidate {+ const native = control.handle as NativeControl;+ return immutableCandidate({+ id: candidateId,+ description,+ tool: 'click',+ arguments: { ...this.target(), element_token: native.elementToken, delivery_mode: delivery },+ source: 'ax',+ snapshotId: this.observation.snapshotId,+ risk: native.risk,+ });+ }++ typeText(control: Control, text: string, candidateId: string, description: string): Candidate {+ const native = control.handle as NativeControl;+ const [tool, extra] =+ this.textMethod === 'type_text' ? ['type_text', { text }] : ['set_value', { value: text }];+ return immutableCandidate({+ id: candidateId,+ description,+ tool,+ arguments: { ...this.target(), element_token: native.elementToken, ...extra },+ source: 'ax',+ snapshotId: this.observation.snapshotId,+ risk: native.risk,+ });+ }+}diff --git a/libs/cua-driver/examples/jev-use/typescript/tasks.ts b/libs/cua-driver/examples/jev-use/typescript/tasks.tsindex 29d7718e84..93fac5a78e 100644--- a/libs/cua-driver/examples/jev-use/typescript/tasks.ts+++ b/libs/cua-driver/examples/jev-use/typescript/tasks.ts@@ -23,6 +23,7 @@ import {immutableCandidate,VisualRegionSource,type Candidate,+ type NativeAccessibilitySource,} from './sources.js';export type Outcome = 'verified' | 'refuted' | 'unknown' | 'abstained' | 'budget_exhausted';@@ -60,11 +61,21 @@ export type TaskParameter = Readonly<{* pending a visual check.*/export type TaskSources = Readonly<{- page: BrowserSemanticSource;+ /** Present for browser tasks. */+ page?: BrowserSemanticSource;visual?: VisualRegionSource;visualPath: boolean;+ /** Present for native tasks (RFC #4268). */+ ax?: NativeAccessibilitySource;+ /** Native candidate IDs whose background delivery Driver refused earlier. */+ foregroundIds?: ReadonlySet<string>;}>;+export function requirePage(sources: TaskSources): BrowserSemanticSource {+ if (!sources.page) throw new Error('this task needs a browser page source');+ return sources.page;+}+export type HistoryEntry = Readonly<{ step: number; selected_id: string; outcome: string }>;/** The interface the runner uses for every task. */@@ -218,7 +229,8 @@ export class FixtureFormTask implements Task {* the chooser must pick it explicitly.*/candidates(sources: TaskSources): Candidate[] {- const { page, visual } = sources;+ const { visual } = sources;+ const page = requirePage(sources);const token = this.token;const field = page.find('textbox', FIELD_NAME);const button = page.find('button', SUBMIT_NAME);@@ -282,8 +294,9 @@ export class FixtureFormTask implements Task {* it is 'not_in_page_structure'.*/stateSummary(sources: TaskSources): FormState {- const field = sources.page.find('textbox', FIELD_NAME);- const button = sources.page.find('button', SUBMIT_NAME);+ const page = requirePage(sources);+ const field = page.find('textbox', FIELD_NAME);+ const button = page.find('button', SUBMIT_NAME);const verificationField = !field? 'not_found': !field.valuediff --git a/libs/cua-driver/examples/jev-use/verify_choice_cli.py b/libs/cua-driver/examples/jev-use/verify_choice_cli.pyindex 39d5a2daac..e6393c4e61 100644--- a/libs/cua-driver/examples/jev-use/verify_choice_cli.py+++ b/libs/cua-driver/examples/jev-use/verify_choice_cli.py@@ -10,11 +10,15 @@ from pathlib import Pathdef main() -> None:parser = argparse.ArgumentParser()parser.add_argument("--live", action="store_true")+ parser.add_argument(+ "--request-version", choices=("v1", "v2"), default="v1",+ help="v2 sends the native cua.jev_choice_request_v2 fixture",+ )args = parser.parse_args()root = Path(__file__).resolve().parentrequest = json.loads(- (root / "fixtures/jev-choice-request-v1.json").read_text(encoding="utf-8")+ (root / f"fixtures/jev-choice-request-{args.request_version}.json").read_text(encoding="utf-8"))command = [sys.executable, str((root / "python/choose_action.py").resolve())]if not args.live:diff --git a/libs/cua-driver/examples/jev-use/verify_decision_cli.py b/libs/cua-driver/examples/jev-use/verify_decision_cli.pyindex 27f4b85304..3054cf1dfa 100644--- a/libs/cua-driver/examples/jev-use/verify_decision_cli.py+++ b/libs/cua-driver/examples/jev-use/verify_decision_cli.py@@ -12,8 +12,16 @@ from pathlib import Pathdef main() -> None:parser = argparse.ArgumentParser(description=__doc__)parser.add_argument("--model", choices=("mock", "jev", "s1"), required=True)- parser.add_argument("--expected-id", default="submit-form")- parser.add_argument("--fixture", choices=("positive", "negative"), default="positive")+ parser.add_argument(+ "--expected-id",+ help="expected selection (default: submit-form, or ax:button:increment for --fixture native)",+ )+ parser.add_argument(+ "--fixture",+ choices=("positive", "negative", "native"),+ default="positive",+ help="native uses the cua.jev_choice_request_v2 fixture",+ )parser.add_argument("--screenshot",type=Path,@@ -24,11 +32,13 @@ def main() -> None:parser.error("--screenshot requires --model s1")root = Path(__file__).resolve().parent- fixture_name = (- "jev-choice-request-v1.json"- if args.fixture == "positive"- else "jev-choice-negative-v1.json"- )+ fixture_name = {+ "positive": "jev-choice-request-v1.json",+ "negative": "jev-choice-negative-v1.json",+ "native": "jev-choice-request-v2.json",+ }[args.fixture]+ if args.expected_id is None:+ args.expected_id = "ax:button:increment" if args.fixture == "native" else "submit-form"request = json.loads((root / "fixtures" / fixture_name).read_text(encoding="utf-8"))command = [sys.executable, str(root / "python/choose_decision.py"), "--model", args.model]if args.screenshot is not None:diff --git a/libs/cua-driver/examples/jev-use/verify_native.py b/libs/cua-driver/examples/jev-use/verify_native.pynew file mode 100644index 0000000000..43ac91fd9b--- /dev/null+++ b/libs/cua-driver/examples/jev-use/verify_native.py@@ -0,0 +1,176 @@+"""Verify the native jev-use tasks against the macOS AppKit harness (RFC #4268).++For every task and language this script launches a fresh AppKit harness with+its own ``CUA_APPKIT_TASK_STATE`` file, runs the native runner against that+exact process, then reads the harness state file itself and checks it against+the task's expected end state. The runners' own outcome events are not the+oracle. Evidence (redacted JSONL logs plus ``summary.json``) goes to a new+``--output-dir``.++Build the harness first:+``libs/cua-driver/tests/fixtures/build/macos.sh --only appkit``.+"""++from __future__ import annotations++import argparse+import getpass+import json+import os+import subprocess+import sys+import tempfile+import time+from pathlib import Path++BASE = Path(__file__).resolve().parent+REPO = BASE.parents[3]+DEFAULT_APP = REPO / "libs/cua-driver/rust/test-apps/harness-appkit/CuaTestHarness.AppKit.app"+STATE_SCHEMA = "cua.appkit_task_state_v1"+NOTE_TEXT = "jev-use native note"++# The independent end-state check for each task, read from the app's own file.+EXPECTED = {+ "appkit-counter": lambda state: state.get("counter") == 3,+ "appkit-save-note": lambda state: state.get("note_saved") == NOTE_TEXT,+ "appkit-choose-size": lambda state: state.get("size") == "large" and state.get("agreed") is True,+}+# The executable candidates the deterministic mock provider must have acted on.+MOCK_ACTIONS = {+ "appkit-counter": ["ax:button:increment"] * 3,+ "appkit-save-note": ["ax:text_input:note:set:note", "ax:button:save-note"],+ "appkit-choose-size": ["ax:radio:large", "ax:checkbox:i-agree"],+}+++def runner_command(language: str) -> list[str]:+ if language == "python":+ return [sys.executable, "python/run_native.py"]+ return ["node", "--import", "tsx", "typescript/run_native.ts"]+++def launch_harness(app: Path, state: Path) -> subprocess.Popen:+ executable = app / "Contents/MacOS/CuaTestHarness.AppKit"+ if not executable.is_file():+ raise SystemExit(f"AppKit harness is missing at {executable}; run the fixture build")+ env = {**os.environ, "CUA_APPKIT_TASK_STATE": str(state)}+ process = subprocess.Popen(+ [str(executable)], env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL+ )+ for _ in range(100):+ if state.exists():+ try:+ initial = json.loads(state.read_text(encoding="utf-8"))+ except json.JSONDecodeError:+ initial = {}+ if initial.get("pid") == process.pid:+ return process+ if process.poll() is not None:+ raise RuntimeError("AppKit harness exited during launch")+ time.sleep(0.1)+ process.kill()+ raise RuntimeError("AppKit harness did not publish its initial task state")+++def read_state(state: Path, pid: int) -> dict:+ observed = json.loads(state.read_text(encoding="utf-8"))+ if observed.get("schema") != STATE_SCHEMA or observed.get("pid") != pid:+ raise RuntimeError("harness state file does not belong to the launched harness")+ return observed+++def summarize(events: list[dict]) -> dict:+ steps = [event for event in events if event.get("event") == "step"]+ acted = [+ event for event in steps+ if event.get("tool") and not event.get("action_error")+ ]+ return {+ "actions": [event["candidate"] for event in acted],+ "tools": sorted({event["tool"] for event in acted}),+ "delivery_modes": sorted({event.get("delivery_mode") or "none" for event in acted}),+ "schemas": sorted({event.get("schema") for event in steps if event.get("schema")}),+ "sources": sorted({event.get("source") for event in acted if event.get("source")}),+ "risk_excluded": sorted({key for event in steps for key in event.get("compose", {}).get("risk_excluded", {})}),+ "max_candidates": max((event.get("candidate_count", 0) for event in steps), default=0),+ "visual_statuses": sorted({event.get("visual", {}).get("status") for event in steps}),+ "reobserved": any(event.get("observation", {}).get("reobserved") for event in steps),+ }+++def verify(language: str, provider: str, task: str, app: Path, output: Path, work: Path) -> dict:+ state = work / f"{language}-{provider}-{task}-state.json"+ log = output / f"{language}-{provider}-{task}.jsonl"+ harness = launch_harness(app, state)+ try:+ initial = read_state(state, harness.pid)+ command = runner_command(language) + [+ "--task", task, "--provider", provider, "--pid", str(harness.pid),+ "--state-file", str(state), "--note-text", NOTE_TEXT, "--log", str(log),+ ]+ completed = subprocess.run(command, cwd=BASE, check=False, timeout=300)+ observed = read_state(state, harness.pid)+ finally:+ harness.kill()+ harness.wait(timeout=10)+ events = [json.loads(line) for line in log.read_text().splitlines()] if log.exists() else []+ result = {+ "language": language,+ "provider": provider,+ "task": task,+ "exit_code": completed.returncode,+ "runner_outcome": events[-1].get("outcome") if events else None,+ "initial": {key: initial.get(key) for key in ("counter", "agreed", "size", "note_saved")},+ "observed": {key: observed.get(key) for key in ("counter", "agreed", "size", "note_saved")},+ **summarize(events),+ }+ result["verified"] = bool(EXPECTED[task](observed))+ if result["observed"].get("note_saved") == NOTE_TEXT:+ # The note text is a secret task parameter; never keep it in evidence.+ result["observed"]["note_saved"] = "[note text]"+ if completed.returncode != 0 or result["runner_outcome"] != "verified":+ raise RuntimeError(f"runner did not report verified: {json.dumps(result)}")+ if not result["verified"]:+ raise RuntimeError(f"independent harness state does not match: {json.dumps(result)}")+ if provider == "mock" and result["actions"] != MOCK_ACTIONS[task]:+ raise RuntimeError(f"mock provider took unexpected actions: {json.dumps(result)}")+ return result+++def main() -> None:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--app", type=Path, default=DEFAULT_APP, help="CuaTestHarness.AppKit.app")+ parser.add_argument("--typescript", action="store_true", help="also verify the TypeScript runner")+ parser.add_argument("--live", action="store_true", help="also verify live Jev; needs a TypeSafe key")+ parser.add_argument("--task", action="append", choices=sorted(EXPECTED), help="limit to tasks")+ parser.add_argument("--output-dir", type=Path, required=True, help="new evidence directory")+ args = parser.parse_args()+ if args.live and not os.environ.get("TYPESAFE_API_KEY", "").strip():+ if not sys.stdin.isatty():+ raise SystemExit("Human prerequisite: provision TYPESAFE_API_KEY before a live run")+ os.environ["TYPESAFE_API_KEY"] = getpass.getpass("TypeSafe API key: ").strip()+ output = args.output_dir.resolve()+ output.mkdir(mode=0o700, parents=True, exist_ok=False)+ summary: dict = {+ "complete": False,+ "harness": "appkit",+ "live_requested": args.live,+ "typescript_requested": args.typescript,+ "checks": [],+ }+ try:+ with tempfile.TemporaryDirectory(prefix="jev-native-") as work:+ for language in ["python", "typescript"] if args.typescript else ["python"]:+ for provider in ["mock", "live"] if args.live else ["mock"]:+ for task in args.task or sorted(EXPECTED):+ result = verify(language, provider, task, args.app, output, Path(work))+ summary["checks"].append(result)+ print(json.dumps({"event": "independently_verified", **result}), flush=True)+ summary["complete"] = True+ finally:+ (output / "summary.json").write_text(json.dumps(summary, indent=2) + "\n")+ print(json.dumps({"event": "native_verify_complete", "checks": len(summary["checks"])}), flush=True)+++if __name__ == "__main__":+ main()diff --git a/libs/cua-driver/tests/fixtures/apps/macos/appkit/main.swift b/libs/cua-driver/tests/fixtures/apps/macos/appkit/main.swiftindex 096b70e383..f2601dd107 100644--- a/libs/cua-driver/tests/fixtures/apps/macos/appkit/main.swift+++ b/libs/cua-driver/tests/fixtures/apps/macos/appkit/main.swift@@ -16,6 +16,11 @@// scroll_target — NSScrollView with a tall body and offset label// ns_menubar — main menu item with known title (Mac-specific)// exit — NSButton terminates the app+// jev_use_tasks — opt-in (CUA_APPKIT_TASK_STATE=<path>): a labeled Note+// field with a Save button and Small/Medium/Large radio+// buttons, plus an app-owned JSON state file rewritten on+// every counter, checkbox, size, and save change. Ordinary+// launches do not add these controls or write any file.//// AX identifiers (via `setAccessibilityIdentifier(_:)`) match the IDs in// scenarios.json. Window title is set to "CuaTestHarness AppKit" so the@@ -54,6 +59,11 @@ let kMenuItemTitle = "Harness Test Item"let kSecondaryWindowTitle = "CuaTestHarness AppKit Secondary"let kSheetWindowTitle = "CuaTestHarness AppKit Sheet"let kFloatingWindowTitle = "CuaTestHarness AppKit Floating"+let kTaskStateEnv = "CUA_APPKIT_TASK_STATE"+let kTaskStateSchema = "cua.appkit_task_state_v1"+let kNoteFieldAID = "txt-note"+let kSaveNoteButtonAID = "btn-save-note"+let kSizeOptions = ["Small", "Medium", "Large"]// MARK: - Controller@@ -77,6 +87,13 @@ final class HarnessWindowController: NSObject, NSTextFieldDelegate, NSTableViewDlet accelCountLabel = NSTextField(labelWithString: "accel_fired=0")var accelCount = 0var keyMonitor: Any?+ // Opt-in jev-use task controls and their app-owned state file.+ let taskStatePath = ProcessInfo.processInfo.environment[kTaskStateEnv]+ let noteField = NSTextField(string: "")+ var agreed = false+ var size = "none"+ var savedNote: String?+ var taskStateSequence = 0// Pinned content size — every launch MUST produce a byte-identical window// so screenshot dimensions (and the hardcoded pixel coords the harness tests@@ -105,6 +122,7 @@ final class HarnessWindowController: NSObject, NSTextFieldDelegate, NSTableViewDsuper.init()buildContent()installKeyboardMonitor()+ writeTaskState()}func show() {@@ -313,6 +331,10 @@ final class HarnessWindowController: NSObject, NSTextFieldDelegate, NSTableViewDexit.setAccessibilityIdentifier(kExitButtonAID)content.addArrangedSubview(exit)+ if taskStatePath != nil {+ addTaskControls(to: content)+ }+// No outer scroll-view wrap: the content is sized to fit the window// so the only scrollable surface is the inner scroll_target NSScrollView.// Otherwise scroll events delivered at window-local coords get@@ -331,6 +353,62 @@ final class HarnessWindowController: NSObject, NSTextFieldDelegate, NSTableViewDwindow.contentView = container}+ /// Opt-in controls for the jev-use native tasks. They are appended below+ /// Exit, inside the pinned window's spare height, so every control above+ /// keeps its ordinary position and AX identity.+ private func addTaskControls(to content: NSStackView) {+ content.addArrangedSubview(sectionLabel("jev_use_tasks"))+ noteField.setAccessibilityIdentifier(kNoteFieldAID)+ // An accessibility label and no placeholder: macOS reports an empty+ // field's placeholder as its value, which jev-use treats as unlabeled.+ noteField.setAccessibilityLabel("Note")+ noteField.translatesAutoresizingMaskIntoConstraints = false+ let save = NSButton(title: "Save note", target: self, action: #selector(onSaveNote))+ save.setAccessibilityIdentifier(kSaveNoteButtonAID)+ let noteRow = NSStackView()+ noteRow.orientation = .horizontal+ noteRow.spacing = 12+ noteRow.addArrangedSubview(noteField)+ noteRow.addArrangedSubview(save)+ NSLayoutConstraint.activate([noteField.widthAnchor.constraint(equalToConstant: 240)])+ content.addArrangedSubview(noteRow)++ let sizeRow = NSStackView()+ sizeRow.orientation = .horizontal+ sizeRow.spacing = 12+ for title in kSizeOptions {+ // Radio buttons that share a superview and an action form one group.+ let radio = NSButton(radioButtonWithTitle: title, target: self,+ action: #selector(onSize(_:)))+ radio.setAccessibilityIdentifier("rad-size-\(title.lowercased())")+ radio.state = .off+ sizeRow.addArrangedSubview(radio)+ }+ content.addArrangedSubview(sizeRow)+ }++ /// Atomically rewrite the app-owned task state. The jev-use task oracle+ /// reads this file; it never depends on Cua Driver's own observations.+ private func writeTaskState() {+ guard let path = taskStatePath else { return }+ taskStateSequence += 1+ let state: [String: Any] = [+ "schema": kTaskStateSchema,+ "pid": Int(ProcessInfo.processInfo.processIdentifier),+ "seq": taskStateSequence,+ "counter": counterValue,+ "agreed": agreed,+ "size": size,+ "note_saved": savedNote.map { $0 as Any } ?? NSNull(),+ ]+ do {+ let data = try JSONSerialization.data(withJSONObject: state, options: [.sortedKeys])+ try data.write(to: URL(fileURLWithPath: path), options: .atomic)+ } catch {+ fputs("failed to write task state: \(error)\n", stderr)+ }+ }+private func sectionLabel(_ id: String) -> NSTextField {let f = NSTextField(labelWithString: "▸ \(id)")f.font = NSFont.systemFont(ofSize: 13, weight: .bold)@@ -368,11 +446,13 @@ final class HarnessWindowController: NSObject, NSTextFieldDelegate, NSTableViewD@objc private func onIncrement() {counterValue += 1counterLabel.stringValue = "counter=\(counterValue)"+ writeTaskState()}@objc private func onReset() {counterValue = 0counterLabel.stringValue = "counter=0"+ writeTaskState()}@objc private func onExit() {@@ -390,6 +470,18 @@ final class HarnessWindowController: NSObject, NSTextFieldDelegate, NSTableViewD@objc private func onCheckbox(_ sender: NSButton) {checkStateLabel.stringValue = "agreed=\(sender.state == .on)"+ agreed = sender.state == .on+ writeTaskState()+ }++ @objc private func onSaveNote() {+ savedNote = noteField.stringValue+ writeTaskState()+ }++ @objc private func onSize(_ sender: NSButton) {+ size = sender.title.lowercased()+ writeTaskState()}func numberOfRows(in tableView: NSTableView) -> Int {diff --git a/rfcs/4268-jev-use-native-candidates.md b/rfcs/4268-jev-use-native-candidates.mdnew file mode 100644index 0000000000..48466391fc--- /dev/null+++ b/rfcs/4268-jev-use-native-candidates.md@@ -0,0 +1,814 @@+---+title: Native accessibility candidates for jev-use decision loops+authors:+ - f-trycua+created: 2026-09-27+last_updated: 2026-09-28+status: accepted+discussion: https://github.com/trycua/cua/issues/4268+rfc_pr: https://github.com/trycua/cua/pull/4269+implementation:+ - https://github.com/trycua/cua/issues/4268+ - https://github.com/trycua/cua/pull/4270+ - https://github.com/trycua/cua/pull/4288+supersedes:+superseded_by:+---++# RFC: Native accessibility candidates for jev-use decision loops++## Summary++jev-use will be able to drive native desktop applications on macOS, Windows,+and Linux, not only browser pages. The runner will build candidates from the+native accessibility elements that `get_window_state` already returns. Each+executable candidate is bound to an `element_token` from one exact snapshot.+OmniParser regions from `parse_visual_regions` remain a fallback for views+that have no usable accessibility tree. The bounded-choice contract does not+change in spirit. The application builds every complete action. TypeSafe Jev+or Cua-S1 returns one supplied ID. Cua Driver performs the action, and an+independent task oracle verifies the result.++To make this possible, the example's hard-coded "type a token, then submit"+loop becomes three candidate sources behind one interface and a declarative+task spec. The chooser request gains a backward-compatible version that tags+each candidate with its `source`. Cua Driver gains no model logic, task+semantics, or credentials. Code inspection shows that Driver does not+normalize accessibility roles across platforms. The native source therefore+owns an explicit, tested mapping from platform roles to a closed set of role+classes (see [Role normalization](#role-normalization-resolved-question-1)).++## Motivation++jev-use reached GA for browser pages in+[#4190](https://github.com/trycua/cua/pull/4190), with 40 of 40 live runs on+Linux and Windows across the page-structure and visual paths. Its decision+space comes from the browser page's accessibility tree (`get_browser_state`)+and from OmniParser regions (`parse_visual_regions`). The runner calls+`get_window_state` only to obtain a screenshot and `capture_id`, with+`include_accessibility_tree: false`. It never uses the native elements that+the same tool can return as candidates.++The loop is also bound to one task. `build_candidates`, `form_state`,+`_form_refs`, and `classify` assume a browser form with a textbox named+"verification value" and a button named "Submit". As a result, jev-use cannot+drive Calculator, Notepad, a settings pane, or any other native application,+even though Driver already exposes snapshot-bound, background-capable element+actions for them on every supported platform.++The decision now matters because the browser path is stable and has evidence.+The next extension of the chooser contract will be copied by downstream+integrations. It should be designed once, across all three platforms, before+someone special-cases a single native application.++## Goals++- Put candidate sources behind one interface: browser semantic (existing),+ native accessibility (new), and visual regions (existing).+- Replace the hard-coded fixture with a task spec that declares the goal,+ parameters (secrets redacted), allowed application and window, allowed action+ kinds, risky-action allowances, step budget, and a success oracle.+- Make native candidates bounded, deterministic, and stable across snapshots.+- Keep the same safety properties on every platform: snapshot-bound element+ actions, background-first delivery, and independent verification.+- Provide deterministic CI proof on macOS, Windows, and Linux, plus live Jev+ and S1 evidence for each platform.++## Non-goals++- No model logic, credentials, prompts, or task semantics inside Cua Driver.+- No free-form actions, coordinates, tool names, or generated text from the+ model.+- No change to the browser path's behavior or GA status.+- Not a general autonomous agent. Tasks remain application-defined.+- No new Driver tool and no change to the `get_window_state` output contract+ in this RFC. A Driver-side normalized role field is discussed as a possible+ follow-up only.++## Terminology++**Candidate source**+: A runner component that turns one observation into zero or more complete,+executable candidates. Each source tags its candidates with a `source` value.++**Task spec**+: An application-owned, declarative description of one task: its goal,+parameters, scope, allowed actions, budget, and success oracle. It replaces the+constants in `core.py`.++**Role class**+: One member of the closed, platform-neutral set of actionable roles that+jev-use recognizes, such as `button` or `text_entry`. The native source maps+each platform's raw `role` string to a role class or excludes the element.++**Stable candidate ID**+: A candidate ID derived from an element's role class, label, and actionable+ancestor path rather than from `element_index`. The same control keeps the same+ID across snapshots while the UI stays structurally the same.++**Observation**+: One `get_window_state` result. When it includes both the tree and the+screenshot, its `snapshot_id` (element tokens) and `capture_id` (visual regions+and capture-bound clicks) describe the same moment.++## Current state++### jev-use structure++The runnable reference lives in+[`libs/cua-driver/examples/jev-use/`](../libs/cua-driver/examples/jev-use/).+[`python/core.py`](../libs/cua-driver/examples/jev-use/python/core.py) and its+TypeScript mirror+[`typescript/core.ts`](../libs/cua-driver/examples/jev-use/typescript/core.ts)+define:++- `Candidate`: `id`, `description`, `tool` (`None` for reserved candidates),+ frozen `arguments`, and optional `capture_id` and `screenshot_reference` for+ capture-bound visual clicks.+- `VisualRegion` and `VisualObservation`: a validated projection of+ `cua.visual_regions_v1`. The observation carries the capture ID, target `pid`+ and `window_id`, screenshot size, and a scaled top-left action mapping.+- `_form_refs` (`formRefs`): finds the textbox named `verification value` and+ the button named `Submit` in the `get_browser_state` `refs`.+- `form_state` (`formState`): summarizes the form for the model without+ revealing the token (`empty`, `contains_required_token`,+ `contains_other_value`), plus the Submit availability states+ (`available`, `visual_only`, `visual_check_pending`, `not_found_visually`,+ `not_in_page_structure`).+- `build_candidates` (`buildCandidates`): emits at most one of+ `type-verification-value` (`browser_type`), `submit-form` (`browser_click`+ with `input_route: dom_event`, or a capture-bound `click`), or+ `submit-form-foreground`, followed by the reserved `reobserve` and `abstain`.+- `classify`: maps the fixture server's `submitted` value to `verified`,+ `refuted`, `budget_exhausted`, or `unknown`.++[`python/run.py`](../libs/cua-driver/examples/jev-use/python/run.py) and+[`typescript/run.ts`](../libs/cua-driver/examples/jev-use/typescript/run.ts)+call these Driver tools over MCP:++| Tool | Use today |+| ---------------------- | ------------------------------------------------------------------------------ |+| `browser_prepare` | Launch an isolated browser profile (`allow_launch`, `isolated_new`). |+| `list_windows` | Find the largest on-screen window for the browser `pid`. |+| `get_browser_state` | Bind `target_id` and tab, then take a `semantic_v2` snapshot every step. |+| `browser_navigate` | Load the loopback fixture. |+| `get_window_state` | Screenshot and `capture_id` only (`include_accessibility_tree: false`). |+| `parse_visual_regions` | OmniParser `text` and `icon` regions at `min_confidence` 0.8, 100 regions max. |+| `browser_type` | Type the token into the field ref, replacing its contents. |+| `browser_click` | Click the Submit ref through `dom_event`. |+| `click` | Capture-bound visual click (`capture_id`, `x`, `y`, `delivery_mode`). |++A background refusal (`background_*` code or `escalation.recommended ==+"foreground"`) is never retried. The next step takes a fresh capture and offers+a distinct `submit-form-foreground` candidate that the chooser must select+explicitly.++### Chooser contract++[`python/choose_action.py`](../libs/cua-driver/examples/jev-use/python/choose_action.py)+validates `cua.jev_choice_request_v1` strictly. The root must have exactly+`schema`, `goal`, `capture_id`, `regions`, `history`, and `candidates`. Each+candidate must have exactly `id` and `description`. IDs match+`[A-Za-z0-9][A-Za-z0-9._:-]{0,63}`, the set holds 2 to 32 candidates, and+`reobserve` and `abstain` are required. It answers with `cua.jev_choice_v1`.+[`python/decision_models.py`](../libs/cua-driver/examples/jev-use/python/decision_models.py)+answers the same request with `cua.decision_choice_v1` (`kind`, `capture_id`,+`selected_id`, `model`, `confidence`, `probabilities`, `reason`). The S1+adapter supports at most 26 options, including the two reserved IDs, and+returns `reason: "option_limit"` instead of truncating+([`decision-models.md`](../libs/cua-driver/examples/jev-use/decision-models.md)).++Because v1 validators reject unknown keys, adding `source` to a v1 candidate+would break every existing v1 validator. The contract change must therefore be+a new request version, not an in-place field addition.++### Driver native element output++`get_window_state` returns a structured `elements` array. The contract type is+`WindowElement` in+[`cua-driver-contract/src/windows.rs`](../libs/cua-driver/rust/crates/cua-driver-contract/src/windows.rs):++| Field | Type | Notes |+| ------------------- | -------------- | -------------------------------------------------------------------------------- |+| `element_index` | integer | Depth-first order. Only actionable rows appear. |+| `role` | string | **Raw platform role**. Not normalized (see below). |+| `depth` | integer | Tree depth. |+| `element_token` | string, opt. | `s<snapshot_id as 8 hex>:<element_index>`. Present when a snapshot is published. |+| `label` | string, opt. | Best-effort display string. Fallback order differs per platform (see below). |+| `value` | string, opt. | Current value, emitted separately from `label`. |+| `value_description` | string, opt. | macOS only in practice. |+| `enabled` | boolean, opt. | |+| `selected` | boolean, opt. | Toggle and selection state; macOS derives it for checkboxes and radios. |+| `in_web_content` | boolean, opt. | Element is inside a web area (browser, Electron, WebView). |+| `actions` | string[], opt. | **Raw platform action names**. Not normalized. |+| `parent_index` | integer, opt. | Nearest _actionable_ ancestor's `element_index`. |+| `frame` | object, opt. | `{x, y, w, h}` in screen coordinates of the platform's native unit. |+| `min`, `max` | number, opt. | Range controls only. |++The output also carries `snapshot_id`, `capture_id` (when both a snapshot and a+screenshot exist), `element_count`, `total_element_count`,+`returned_element_count`, `filtered_element_count`, `elements_complete`,+`truncated`, `truncation_reason`, `degraded`, `degraded_reason`, screenshot+dimensions and scale, and `window_bounds`. The Linux adapter also emits+`description` and `unlabelled` on elements. Those fields are allowed by the+output schema but are not part of `WindowElement`.++Element tokens are validated by Driver. A new snapshot for the same+`(pid, window_id)` invalidates every token from the previous snapshot, and a+stale token returns the explicit error `element_token is stale; call+get_window_state again to refresh`+([`cua-driver-core/src/element_token.rs`](../libs/cua-driver/rust/crates/cua-driver-core/src/element_token.rs),+[`snapshot_invariants.rs`](../libs/cua-driver/rust/crates/cua-driver-core/tests/snapshot_invariants.rs)).+`click`, `type_text`, and `set_value` accept `element_token` on all three+platforms, and `click` accepts `delivery_mode`.++### Role normalization (resolved question 1)++**Finding: `elements[].role` values are not normalized.** Each adapter writes+its native role string directly:++| Platform | Source | Example values |+| ------------ | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |+| macOS AX | `node.role` in `platform-macos/src/tools/get_window_state.rs` | `AXButton`, `AXCheckBox`, `AXRadioButton`, `AXPopUpButton`, `AXMenuItem`, `AXLink`, `AXTextField`, `AXTextArea` |+| Windows UIA | `control_type_name` in `platform-windows/src/uia/mod.rs` | `Button`, `CheckBox`, `RadioButton`, `ComboBox`, `MenuItem`, `Hyperlink`, `Edit`, `SplitButton` |+| Linux AT-SPI | `get_role_name()` in `platform-linux/src/atspi/native.rs` | `push button`, `toggle button`, `check box`, `radio button`, `combo box`, `menu item`, `link`, `entry`, `text`, `password text` |++`actions` are not normalized either. macOS reports AX action names, Windows+reports pattern-derived names (`invoke`, `toggle`, `select`, `expand`,+`set_value`, `text`, `scroll`), and Linux reports AT-SPI action names such as+`click`. `label` fallbacks also differ. macOS uses title, then description,+then **value**, then identifier. Windows uses name, then **value**, then+automation ID, then help text. Linux uses name, then description, and never+value; it marks a value control without a name as `unlabelled`.++Driver does have a private, partial normalizer, `normalized_role` in+[`cua-driver-core/src/expectation.rs`](../libs/cua-driver/rust/crates/cua-driver-core/src/expectation.rs).+It keeps ASCII alphanumerics, lowercases, strips an `ax` prefix, and maps+`pushbutton` to `button` and `pagetab`/`tabitem` to `tab`. `verify_state`+uses it only to match `ElementSelector.role`. It unifies `AXCheckBox`,+`CheckBox`, and `check box`, but not `AXTextField`, `Edit`, and `entry`, not+`AXPopUpButton`, `ComboBox`, and `combo box`, and not `AXLink` and `Hyperlink`.+It is not exposed in the output.++**Consequence for the design:** `NativeAccessibilitySource` must own an+explicit per-platform mapping from raw `role` strings to a closed set of role+classes. The mapping is data, versioned with the example, and covered by+fixtures captured from each platform's harness application. Where possible,+it agrees with `normalized_role`, so that a task oracle written as a+`verify_state` selector and a candidate built from the same control use the+same notion of role. Unknown roles are excluded, never guessed. Promoting a+normalized `role_class` into the Driver output contract would be the+cross-platform-correct long-term home under the repository's shared-semantics+guidance. That is left as a follow-up question, not a prerequisite, because+it changes a public Driver contract that this RFC does not need.++### Scoping large trees (input to question 2)++`get_window_state` already offers three scoping controls with the same names on+every platform:++| Parameter | Semantics | Default |+| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- |+| `query` | Case-insensitive substring. Returns matching actionable rows plus their actionable ancestors, keeping the original `element_index` values. Compare `total_element_count` with `returned_element_count`. | none |+| `max_elements` | Cap on nodes walked. Truncates depth-first; markdown and structured elements truncate together. | macOS 2,000; Windows 5,000; Linux 5,000 |+| `max_depth` | Cap on walk depth. | macOS 25; Windows 25; Linux uncapped |+| `timeout_ms` | Wall-clock walk budget, 100 to 120,000. An exhausted budget returns a partial tree with `truncated: true` and `elements_complete: false`. | 1,000 on every platform |++Two consequences follow. First, `max_elements` truncation is depth-first, so+a low cap can silently drop the part of the window a task needs. The runner+must treat `elements_complete: false` as "partial". Partial trees may still+produce candidates, but they must never be used to conclude that an area has+_no_ accessibility element, and so must never trigger the OmniParser fallback+by themselves. Second, `query` preserves indices and tokens, which makes it the+natural carrier for task-supplied label filters.++## Proposal++### Component ownership++Ownership is unchanged from+[RFC #3931](3931-cua-perception-and-jev-use.md). Driver owns observation,+element and capture binding, actions, and `verify_state`. The jev-use runner+owns candidate sources, the task spec, the role-class mapping, redaction,+policy, and the loop. The chooser owns only the choice among supplied IDs.++```mermaid+flowchart LR+ TASK["Task spec<br/>goal, params, scope, oracle"]+ RUNNER["jev-use runner<br/>sources, policy, loop"]+ DRIVER["Cua Driver<br/>get_window_state, parse_visual_regions,<br/>click, set_value, type_text, verify_state"]+ CHOOSER["Jev or S1<br/>one supplied ID"]++ TASK --> RUNNER+ RUNNER -->|observe and act| DRIVER+ RUNNER -->|IDs, descriptions, source| CHOOSER+ CHOOSER -->|selected_id| RUNNER+```++### Candidate sources++All three sources implement one interface. The Python sketch below is+normative for fields; TypeScript mirrors it with the same names in camelCase.++```python+CandidateSourceKind = Literal["page", "ax", "visual"]++@dataclass(frozen=True)+class Candidate:+ id: str # stable ID, ID_PATTERN, unique per step+ description: str # bounded, redacted, <= 1,000 chars+ source: CandidateSourceKind | None # None only for reserved candidates+ tool: str | None # None for reobserve / abstain+ arguments: Mapping[str, Any] # complete Driver arguments, frozen+ snapshot_id: str | None = None # set for "ax" candidates+ capture_id: str | None = None # set for "visual" candidates+ risk: frozenset[str] = frozenset() # e.g. {"destructive"}; see policy++@dataclass(frozen=True)+class Observation:+ pid: int+ window_id: int+ snapshot_id: str | None+ capture_id: str | None+ elements: tuple[Mapping[str, Any], ...]+ elements_complete: bool+ browser_snapshot: Mapping[str, Any] | None+ visual: VisualObservation | None++class CandidateSource(Protocol):+ kind: CandidateSourceKind+ def candidates(self, observation: Observation, task: TaskSpec,+ state: RunState) -> list[Candidate]: ...+```++- `BrowserSemanticSource` wraps today's `build_candidates` page-structure branch+ unchanged (`browser_type`, `browser_click`).+- `NativeAccessibilitySource` applies the native rules below to+ `observation.elements`.+- `VisualRegionSource` wraps today's capture-bound visual branch (`click` with+ `capture_id`) and is consulted only under the fallback rule.++A composer merges the source outputs in the fixed order page, ax, visual. It+drops duplicate IDs deterministically (first source wins), applies the+risky-action policy, caps the executable candidates at 24, and appends+`reobserve` and `abstain`. The limit of 24 plus 2 fits the S1 26-option limit+and the v1 schema's 32-candidate limit. A step that would exceed the cap+drops the lowest-priority candidates and records the count dropped in the log.+It never truncates silently.++### Task spec++```python+@dataclass(frozen=True)+class TaskParameter:+ name: str # e.g. "search_text"+ value: str+ secret: bool = False # redacted in every model input and log++@dataclass(frozen=True)+class WindowScope:+ bundle_id: str | None = None # macOS+ process_name: str | None = None # Windows / Linux+ window_title_contains: str | None = None+ query: str | None = None # forwarded to get_window_state.query+ max_elements: int | None = None # forwarded; defaults stay platform defaults+ max_depth: int | None = None++@dataclass(frozen=True)+class Oracle:+ kind: Literal["verify_state", "app_check"]+ predicates: tuple[Mapping[str, Any], ...] = () # verify_state `expect`+ app_check: str | None = None # name of an application-owned callable++@dataclass(frozen=True)+class TaskSpec:+ id: str+ goal: str # shown to the model; must not embed secrets+ parameters: tuple[TaskParameter, ...]+ scope: WindowScope+ allowed_actions: frozenset[Literal["press", "toggle", "select", "open_menu",+ "set_text", "visual_click"]]+ allowed_risks: frozenset[str] = frozenset() # opt-in only+ allow_foreground: bool = False+ max_steps: int = 6+ oracle: Oracle+```++The browser fixture becomes one `TaskSpec` whose oracle is an `app_check` that+calls the fixture server's `/state`, so `classify` becomes one oracle adapter+rather than loop logic. `verify_state` oracles use Driver's existing+`StatePredicate` (`window`, `element` with `selector.role`,+`selector.label_contains`, `value_equals`, `enabled`, `selected`), combined+with logical AND, 1 to 8 predicates, plus the optional `timeout_ms` and+`stable_samples`. The oracle runs after every action and before every+decision, as the fixture check does today. The model never sees oracle output+beyond the compact history outcome.++### Native candidate rules++`NativeAccessibilitySource` observes with a single call:++```json+{+ "pid": 4242,+ "window_id": 17,+ "include_accessibility_tree": true,+ "include_screenshot": true,+ "query": null,+ "max_elements": null+}+```++The single call matters. A second `get_window_state` for the same window would+publish a new snapshot and invalidate every token in the candidate set. The+visual fallback therefore parses the `capture_id` from the _same_ observation+instead of capturing again, as the browser path does today.++An element becomes a candidate only when all of the following hold:++1. **Role class.** Its raw `role` maps to one of these role classes:++ | Role class | macOS AX | Windows UIA | Linux AT-SPI | Candidate actions |+ | ------------ | ----------------------------------------------------------------- | ----------------------- | ------------------------------------------------- | ----------------- |+ | `button` | `AXButton` | `Button`, `SplitButton` | `push button`, `button` | press |+ | `checkbox` | `AXCheckBox`, `AXSwitch` | `CheckBox` | `check box`, `toggle button` | toggle |+ | `radio` | `AXRadioButton` | `RadioButton` | `radio button` | select |+ | `popup` | `AXPopUpButton`, `AXComboBox`, `AXMenuButton` | `ComboBox` | `combo box` | open_menu |+ | `menu_item` | `AXMenuItem`, `AXMenuBarItem` | `MenuItem` | `menu item`, `check menu item`, `radio menu item` | press |+ | `link` | `AXLink` | `Hyperlink` | `link` | press |+ | `text_entry` | `AXTextField`, `AXTextArea`, `AXSearchField`, `AXSecureTextField` | `Edit` | `entry`, `text`, `password text` | set_text |++ Text fields and text areas share one class because Windows reports both as+ `Edit` and Linux reports both as `text`. Tabs, sliders, trees, and table+ cells are out of scope for v1. The table is the initial proposal and is+ finalized in Phase 1 against captured harness fixtures.++2. **Enabled.** `enabled` is not `false`.+3. **On screen.** `frame` exists, has positive size, and intersects+ `window_bounds`. Frame units differ per platform, but the element and window+ rectangles use the same unit on each platform, so an intersection test is+ valid. Frames are never used to act.+4. **Labeled.** After redaction, the element has a non-empty label that is not+ a copy of its own value. On macOS and Windows, `label` can fall back to+ `value`. For `text_entry`, a label equal to `value` is treated as+ unlabeled, so that typed content never becomes a candidate ID or+ description. Elements with `unlabelled: true` are excluded.+5. **Native content.** `in_web_content` is not `true`. Web content goes through+ the browser source. This matches `verify_state`, which already treats+ web-content elements as an untrusted source.+6. **Allowed.** The action kind is in `task.allowed_actions`, and the element+ passes the scope filters and the risky-action policy.++Candidate actions:++| Action | Driver call |+| --------- | ------------------------------------------------------------------------------------------------------------ |+| press | `click` with `pid`, `window_id`, `element_token`, `delivery_mode: "background"` |+| toggle | same as press; the description states the current `selected` value and the target state |+| select | same as press |+| open_menu | same as press; the menu's items are candidates only after the next observation |+| set_text | `set_value` with `pid`, `window_id`, `element_token`, and `value` taken **only** from a named task parameter |++`set_text` candidates are emitted once for each allowed task parameter and+text entry pair, and only when the field's current value differs from the+parameter. The description reports the field state with the existing+vocabulary (`empty`, `contains_required_value`, `contains_other_value`) and+never includes the value. Ordering is depth-first `element_index` order within+the ax source. That order is deterministic for a fixed tree, and it is the+same order as `tree_markdown`.++**Fallback to visual regions.** `VisualRegionSource` runs only when the task+allows `visual_click`, the Driver advertises `parse_visual_regions` and a+`click` with `capture_id`, and one of these holds:++- the observation is complete (`elements_complete: true`) and has no native+ candidates after the rules above; or+- the task declares a named visual target, and no native candidate covers that+ target's region.++A truncated or degraded tree never triggers the fallback by itself. The runner+first reobserves once with a larger `timeout_ms`, then offers only `reobserve`+and `abstain` if the tree is still partial.++### Stable candidate IDs++`element_index` changes whenever anything above a control is added or removed,+so it is not used in IDs. IDs are derived as follows:++```text+base = "ax:" + role_class + ":" + slug(label)+path = [(role_class(a), label(a)) for a in actionable ancestors via parent_index]+ord = position among earlier elements with the same (role_class, label, path)+id = base if unique in this observation+ = base + ":" + hex4(sha256(path, ord)) otherwise+```++`slug` lowercases ASCII, replaces each run of other characters with `-`, and+trims the result to 32 characters. A label with no ASCII alphanumerics uses+`hex8(sha256(label))`. The result always matches the v1 ID pattern and stays+within 64 characters. A `set_text` candidate appends `:set:` and the parameter+name. `parent_index` points only to actionable ancestors because `elements`+omits non-actionable rows, so the path is the actionable-ancestor chain rather+than the full tree path. The chooser never sees the `element_token`. The runner+keeps the ID-to-token mapping for the current snapshot and discards it on the+next observation.++Stability is a convenience for history and evaluation, not a safety property.+Safety comes from binding each action to the current snapshot's token.++### Acting and staleness++- Element actions use only the `element_token` from the observation that built+ the candidate set. If Driver returns the stale-token error, the runner+ records the error, dispatches nothing else, and reobserves. It never remaps+ a stale ID to a new token without a new decision.+- Delivery is background first. If Driver refuses background delivery (a+ `background_*` code or `escalation.recommended == "foreground"`), the runner+ offers a distinct `<id>:foreground` candidate on the next step, and only when+ `task.allow_foreground` is true. This reuses today's refusal handling.+- Each observation authorizes at most one action, then the runner reobserves.+ This matches the capture rule in RFC #3931.++### Risky-action policy++Candidates are tagged with risk categories before they reach the chooser:++| Category | Matched when (case-insensitive label match, whole words, localized lists extensible) |+| --------------- | -------------------------------------------------------------------------------------- |+| `destructive` | delete, remove, erase, trash, discard, clear all, format, reset |+| `send` | send, submit, post, publish, share, reply |+| `purchase` | buy, purchase, pay, checkout, order, subscribe |+| `close_unsaved` | close, quit, exit, don't save, discard changes, while the window reports unsaved state |++A tagged candidate is removed before the provider request unless+`task.allowed_risks` contains its category. The browser fixture's Submit is+therefore allowed explicitly through `allowed_risks: {"send"}`. Label matching+is a guard against accidents, not an authorization boundary. A task that needs+stronger guarantees narrows `scope.query` and `allowed_actions`. The model can+never unlock a category.++### Contract change++The request adds version `cua.jev_choice_request_v2`. It is additive relative+to v1:++- each candidate may include `source` (`page`, `ax`, or `visual`); reserved+ candidates omit it;+- the root may include `snapshot_id` alongside the required `capture_id`; and+- the root may include `elements`, a compact, redacted list of at most 64+ native elements (`role_class`, `label`, `state`). S1's text adapter renders+ it as a genuine accessibility tree.++The v1 schema, validators, and fixtures remain supported unchanged. A v2+request that has no `source`, `snapshot_id`, or `elements` is semantically+identical to v1. The responses `cua.jev_choice_v1` and `cua.decision_choice_v1`+are unchanged. The runner sends v1 whenever every candidate comes from the+`page` or `visual` source, so the browser path's wire traffic does not change.++Example request for a native step:++```json+{+ "schema": "cua.jev_choice_request_v2",+ "goal": "Increment the counter once.",+ "capture_id": "cap-7f3a",+ "snapshot_id": "s0000002a",+ "regions": [],+ "elements": [+ { "role_class": "button", "label": "Increment", "state": "enabled" },+ { "role_class": "button", "label": "Reset", "state": "enabled" }+ ],+ "history": [],+ "candidates": [+ {+ "id": "ax:button:increment",+ "source": "ax",+ "description": "Press the button labeled \"Increment\"."+ },+ {+ "id": "ax:button:reset",+ "source": "ax",+ "description": "Press the button labeled \"Reset\"."+ },+ { "id": "reobserve", "description": "Take no action and obtain a fresh observation." },+ { "id": "abstain", "description": "Stop without acting if no supplied action is safe." }+ ]+}+```++Response (unchanged `cua.decision_choice_v1`):++```json+{+ "schema": "cua.decision_choice_v1",+ "kind": "selected",+ "capture_id": "cap-7f3a",+ "selected_id": "ax:button:increment",+ "model": "jev",+ "confidence": 0.93,+ "probabilities": {+ "ax:button:increment": 0.93,+ "ax:button:reset": 0.04,+ "reobserve": 0.02,+ "abstain": 0.01+ },+ "reason": null+}+```++The runner then validates the ID and dispatches+`click {"pid": 4242, "window_id": 17, "element_token": "s0000002a:3",+"delivery_mode": "background"}`, reobserves, and runs the oracle.++## Alternatives considered++### Visual-only (OmniParser everywhere)++This is simpler and uniform, but it discards the accessibility route, which is+verifiable, background-capable, and label-exact. It also adds CPU OCR and+detection latency of 2 to 9 seconds per turn. Visual regions stay as the+fallback.++### Let the model emit tool calls or coordinates++Rejected. It breaks the bounded-choice property that makes jev-use auditable,+and it reintroduces stale-frame and coordinate-space confusion.++### Put candidate construction in Cua Driver++Rejected. Candidate construction encodes task and model policy, which stays+out of the Driver's MIT core, as RFC #3931 decided.++### Use `element_index` as the candidate ID++This is trivial to implement, but the IDs shift between snapshots, which makes+history misleading and makes accuracy evaluation noisy. It also invites the+chooser to learn positional shortcuts.++### Normalize roles in Driver first++This is the cleanest cross-platform home for role classes, and Driver already+has a partial private normalizer. It is not a prerequisite: it changes a+public Driver output contract, needs its own cross-platform evidence, and+would block this work on a Driver release. The runner-side table can later be+replaced by a Driver field with no change to the chooser contract.++## Compatibility and migration++- Browser tasks keep today's behavior. Phase 0 must pass the existing+ deterministic Python and TypeScript tests, `verify_choice_cli.py`,+ `verify_decision_cli.py --model mock`, and replays of+ `fixtures/jev-page-structure-replay-v1.json` and+ `fixtures/jev-visual-replay-v1.json` without changes.+- The browser path keeps its candidate IDs (`type-verification-value`,+ `submit-form`, `submit-form-foreground`) and continues to send v1 requests.+- `cua.jev_choice_request_v2` is additive, and providers that ignore `source`+ keep working. v1 remains accepted indefinitely by the example CLIs.+- No Driver change is required. Rollback means reverting the example and+ skill changes. The browser path is unaffected at every phase.++## Security, privacy, and telemetry++Native applications can expose more sensitive UI than a test page. Mitigations:++- **Scope.** A task names an allowed application and window. The runner+ refuses to observe or act on any other `pid` or `window_id`, and it forwards+ `scope.query` to limit what is walked and returned.+- **Risky actions** are excluded by default, as described above.+- **Redaction.** Every secret parameter value is replaced in all model input,+ logs, and evidence, which extends `redact_token` to all parameters. Because+ macOS and Windows labels can fall back to `value`, the native source also+ removes `value` from descriptions and never uses a text entry's value as its+ label.+- **No pixels to providers.** Screenshot bytes and element tokens are never+ sent to providers. The multimodal S1 adapter keeps its current local-only+ screenshot binding.+- **Snapshot binding.** Every element action is bound to the exact snapshot.+ Stale tokens fail closed in Driver.+- **Permissions.** Accessibility and Screen Recording permissions are+ unchanged; Cua Driver already requires them. There is no new telemetry. Logs+ keep the existing content-free fields plus candidate counts, source+ counts, and dropped-candidate counts.++## Implementation plan++Each phase is one reviewable pull request or a small stack, with Python and+TypeScript parity in the same change.++0. **Refactor with no behavior change.** Introduce `CandidateSource`,+ `TaskSpec`, `Oracle`, and the composer. Express the browser fixture as a+ task spec, and move `classify` into an oracle adapter. Gate: all existing+ deterministic tests and replays pass unchanged. The requests sent are+ byte-identical v1.+1. **macOS AppKit with the mock provider.** Add `NativeAccessibilitySource`,+ the role-class table, stable IDs, the risky-action policy,+ `cua.jev_choice_request_v2` validation in `choose_action.py` and+ `decision_models.py`, and native task specs against the AppKit harness+ (`counter`, `text_input`, `checkable_controls`, `context_menu`). Capture+ `get_window_state` fixtures for unit tests.+2. **Windows and Linux.** Run the same task specs against WPF and WinUI3+ (UIA) and GTK3 (AT-SPI). Extend the role table from captured fixtures, and+ document any per-platform limitation explicitly.+3. **Live evidence.** Run live Jev and S1 on each platform, including the+ `visual-only-canvas` harness as the view without an accessibility tree+ (OmniParser fallback).+4. **Documentation and skill.** Update the jev-use README, decision-models+ guide, and `skills/jev-use`.++## Test and acceptance plan++- Deterministic unit tests, run in the existing `CI: jev-use` workflow: role+ mapping per platform from captured fixtures; each rule (disabled, off-screen,+ unlabeled, label-equals-value, `in_web_content`, unknown role); ID stability+ across two fixtures that differ only in `element_index`; collision suffixes;+ the 24-candidate cap and its logged drop count; risky-action exclusion and+ opt-in; secret redaction; v1 and v2 validation, including rejection of v2+ fields in a v1 request; and stale-token handling that leads to a reobserve.+- Deterministic desktop proof per platform with the mock provider, on the+ AppKit, WPF, WinUI3, and GTK3 harnesses, verified by `verify_state` oracles.+- Live Jev and S1: 5 of 5 runs per platform, provider, and language, verified+ by task oracles. Report an accuracy table by candidate-set size (about 4,+ about 12, and about 24), latency per turn split into observe, decide, and+ act, and one view without an accessibility tree that exercises the OmniParser+ fallback.+- Before the implementation pull request is made ready, run the canonical+ desktop E2E harnesses on the exact candidate SHA, as AGENTS.md requires.++## Unresolved questions++- **Resolved: role normalization.** Roles and actions are raw platform values.+ The native source owns a per-platform role-class table that stays consistent+ with Driver's `normalized_role`. Should a later RFC promote a normalized+ `role_class` into the `get_window_state` contract?+- **Scoping on large trees** (for example about 1,600 elements in Finder).+ Proposed default: the focused window only, the platform's default+ `max_elements`, and task-supplied `query` when the task names its controls.+ Is a task-supplied subtree anchor also needed, given that `query` matching is+ substring-only?+- **Accuracy at scale.** Jev and S1 accuracy with about 24 candidates, compared+ with about 4 today. Should the composer rank by task relevance before capping,+ or keep strict depth-first order for determinism?+- **Electron and Catalyst.** These report misleading accessibility values. The+ proposal excludes `in_web_content` elements from the native source. Should+ Electron applications with a reachable CDP endpoint route through the browser+ path by default?+- **S1 on native tasks.** S1 was trained on web and form tasks, so its native+ accuracy is unknown until Phase 3.+- **Text entry method.** Is `set_value` sufficient as the only `set_text`+ action, or do tasks need an opt-in `type_text` for applications that react+ only to key events?++## Decision record++**Accepted** on 2026-09-28. The maintainer decision summary is in+[#4268](https://github.com/trycua/cua/issues/4268). The review window was+shortened at the maintainer's direction because the change is additive and+confined to the jev-use example layer: the browser path and its+`cua.jev_choice_request_v1` traffic are unchanged, and every phase is gated by+deterministic CI.++Accepted changes from code review, reflected above: one `get_window_state`+observation per step feeds both native and visual candidates; a truncated tree+triggers a reobserve, never an OmniParser fallback; the native source owns a+tested per-platform role table; a label equal to the element's value counts as+unlabeled; `in_web_content` elements are excluded; per-candidate `source`+ships in `cua.jev_choice_request_v2`; about 24 action candidates fit S1's+26-option limit with `reobserve` and `abstain`.++Rejected alternatives: visual-only candidates everywhere, model-emitted tool+calls or coordinates, and candidate construction inside Cua Driver.++Remaining risks: Jev and S1 accuracy with larger candidate sets, S1 accuracy on+native applications, per-step latency on large trees, and misleading+accessibility values in Electron and Catalyst applications.++Implementation:++- Phase 0, [#4270](https://github.com/trycua/cua/pull/4270): candidate sources+ and the task spec, with no behavior change.+- Phase 1, [#4288](https://github.com/trycua/cua/pull/4288): the native source,+ role table, `cua.jev_choice_request_v2`, and AppKit tasks with the mock+ provider in the macOS CI job. Phase 1 finalized these details:+ - The role classes are `button`, `toggle`, `checkbox`, `radio`, `popup`,+ `menu_item`, `link`, and `text_input` (called `text_entry` in the draft+ table). Each platform table is keyed by the output of Driver's+ `normalized_role`, so raw roles that `verify_state` treats as equal always+ share a class. `AXSwitch` and AT-SPI `toggle button`/`switch` map to+ `toggle`; Windows has no `toggle` row yet.+ - The macOS adapter always reports `elements_complete: false`, because+ Driver cannot yet prove absence over the actionable projection. Partial+ trees still produce candidates. A truncated tree, or a partial tree with no+ native candidate, is observed once more with a larger `timeout_ms`. The+ visual fallback is used only when Driver reports `ax_tree_empty` or a+ complete tree lacks the target. On macOS this means only an empty tree+ reaches OmniParser today.+ - A radio option that is already selected is not offered, and a `set_text`+ candidate is offered only while the field differs from its parameter.+ - `close_unsaved` matches its phrases whether or not the window reports+ unsaved state, because Driver does not expose that state.+ - The AppKit harness gained an opt-in `CUA_APPKIT_TASK_STATE` mode with a+ labeled Note field, a Save button, Small/Medium/Large radio buttons, and an+ app-owned JSON state file that serves as the oracle. Ordinary launches are+ unchanged. The Phase 1 tasks use radio buttons in place of a pop-up+ button, because an open AppKit menu runs a modal tracking loop that is not+ deterministic for background CI.