Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
docs(cua-driver): converge validation guidance and cleanup (#2185)
Francesco Bonaccid39b377de51e
diff --git a/.github/scripts/tests/test_cua_driver_release_wiring.py b/.github/scripts/tests/test_cua_driver_release_wiring.pyindex bdeb74c38f..ad1bb9350c 100644--- a/.github/scripts/tests/test_cua_driver_release_wiring.py+++ b/.github/scripts/tests/test_cua_driver_release_wiring.py@@ -43,6 +43,11 @@ class TestCuaDriverReleaseWiring(unittest.TestCase):workflow = self.read(".github/workflows/ci-release-reminder.yml")self.assertIn('["libs/cua-driver/rust/"]="cua-driver-rs"', workflow)+ self.assertIn("cua-driver desktop release validation", workflow)+ self.assertIn("e2e-rust-windows.yml", workflow)+ self.assertIn("scripts/ci/macos/run-rust-e2e.sh", workflow)+ self.assertIn("e2e-rust-linux.yml", workflow)+ self.assertIn("e2e-rust-linux-wayland.yml", workflow)def test_unreleased_digest_tracks_rust_driver(self) -> None:workflow = self.read(".github/workflows/release-unreleased-digest.yml")diff --git a/.github/workflows/ci-release-reminder.yml b/.github/workflows/ci-release-reminder.ymlindex dea14f8774..c52735ba24 100644--- a/.github/workflows/ci-release-reminder.yml+++ b/.github/workflows/ci-release-reminder.yml@@ -111,6 +111,16 @@ jobs:BODY+="\nOr add \`no-release\` to skip."fi+ if printf '%s\n' "${AFFECTED[@]}" | grep -qx 'cua-driver-rs'; then+ BODY+="\n\n### cua-driver desktop release validation\n\n"+ BODY+="Before adding the release label, a maintainer records green runs on the exact release SHA:\n"+ BODY+="- [ ] [Windows canonical matrix](https://github.com/${REPO}/actions/workflows/e2e-rust-windows.yml)\n"+ BODY+="- [ ] macOS logged-in host: \`scripts/ci/macos/run-rust-e2e.sh\`, with the typed summary attached to the PR\n"+ BODY+="- [ ] [Linux X11 canonical matrix](https://github.com/${REPO}/actions/workflows/e2e-rust-linux.yml)\n"+ BODY+="- [ ] [Linux Sway canonical matrix](https://github.com/${REPO}/actions/workflows/e2e-rust-linux-wayland.yml) with environment \`sway\`\n"+ BODY+="- [ ] Result links and remaining experimental environment gaps are recorded in the PR\n"+ fi+# Delete previous reminder comment if exists, then post new oneEXISTING_COMMENT_ID=$(gh api "repos/${REPO}/issues/${PR_NUMBER}/comments" \--jq ".[] | select(.body | contains(\"${COMMENT_MARKER}\")) | .id" 2>/dev/null | head -1)diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex 75c349948b..29d7bdc0ce 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -1,59 +1,67 @@# Contributing to Cua-We deeply appreciate your interest in contributing to Cua! Whether you're reporting bugs, suggesting enhancements, improving docs, or submitting pull requests, your contributions help improve the project for everyone.--## Reporting Bugs--If you've encountered a bug in the project, we encourage you to report it. Please follow these steps:--1. **Check the Issue Tracker**: Before submitting a new bug report, please check our issue tracker to see if the bug has already been reported.-2. **Create a New Issue**: If the bug hasn't been reported, create a new issue with:- - A clear title and detailed description- - Steps to reproduce the issue- - Expected vs actual behavior- - Your environment (macOS version, cua version)- - Any relevant logs or error messages-3. **Label Your Issue**: Label your issue as a `bug` to help maintainers identify it quickly.--## Suggesting Enhancements--We're always looking for suggestions to make Cua better. If you have an idea:--1. **Check Existing Issues**: See if someone else has already suggested something similar.-2. **Create a New Issue**: If your enhancement is new, create an issue describing:- - The problem your enhancement solves- - How your enhancement would work- - Any potential implementation details- - Why this enhancement would benefit Cua users--## Code Formatting--We follow strict code formatting guidelines to ensure consistency across the codebase. Before submitting any code:--1. **Review Our Format Guide**: Please review our [Code Formatting Standards](Development.md#code-formatting-standards) section in the Getting Started guide.-2. **Configure Your IDE**: We recommend using the workspace settings provided in `.vscode/` for automatic formatting.-3. **Run Formatting Tools**: Always run the formatting tools before submitting a PR:- ```bash- # For Python code- uv run black .- uv run isort .- uv run ruff check --fix .- ```-4. **Validate Your Code**: Ensure your code passes all checks:- ```bash- uv run mypy .- ```-5. Every time you try to commit code, a pre-commit hook will automatically run the formatting and validation tools. If any issues are found, the commit will be blocked until they are resolved. Please make sure to address any issues reported by the pre-commit hook before attempting to commit again. Once all issues are resolved, you can proceed with your commit.+Thanks for contributing to Cua. The repository includes Python and TypeScript+SDKs, a Rust desktop driver, Swift virtualization tools, container images, and+public documentation. Start with the component that owns the behavior you want+to change.++## Report a Bug++Before opening an issue, search the existing issue tracker. Include:++- a concise description and reproducible steps;+- expected and actual behavior;+- Cua package or driver version;+- operating system, window system, and application when relevant;+- logs, structured errors, screenshots, or recordings that help reproduce it.++Do not include credentials or private application data.++## Propose a Change++For feature requests, describe the user problem and the expected behavior+before prescribing an implementation. Mention affected platforms and existing+workarounds when known.++## Submit Code++1. Read [`Development.md`](Development.md) and the guide next to the component.+2. Keep changes scoped to the component that owns the behavior.+3. Add or update tests that observe the public effect of the change.+4. Run the applicable commands in [`TESTING.md`](TESTING.md).+5. Run the formatters and linters owned by the changed component.+6. Open a focused pull request that explains behavior, validation, and known gaps.++Root pre-commit hooks are optional local helpers. Install them with:++```bash+uv sync --group dev+uv run pre-commit install+```++Mypy is configured but is not currently a pre-commit gate. Rust, TypeScript,+Swift, and documentation checks remain component-owned.++## Desktop Behavior Changes++cua-driver behavior must be verified through the canonical Rust harnesses. A+successful tool response alone is not evidence that an action reached the+application. Delivery tests should observe fixture state and attach focus,+z-order, cursor, leaked-input, capture, or refusal oracles as required.++Do not weaken a test to match the current driver. Add a capability, return an+exact structured refusal, or record the behavior as an explicit gap.## Documentation-Documentation improvements are always welcome. You can:+Public documentation lives under `docs/content/docs` and follows Diataxis. See+[`docs/README.md`](docs/README.md) before adding a page. Contributor-only plans,+journals, and implementation notes belong next to their component.-- Fix typos or unclear explanations-- Add examples and use cases-- Improve API documentation-- Add tutorials or guides+Documentation changes should pass generator drift, hygiene, internal links,+and the production Fumadocs build.-For detailed instructions on setting up your development environment and submitting code contributions, please see our [Developer-Guide](Development.md).+## Community-Feel free to join our [Discord community](https://discord.com/invite/mVnXXpdE85) to discuss ideas or get help with your contributions.+For design discussion and contributor help, join the+[Cua Discord community](https://discord.com/invite/mVnXXpdE85).diff --git a/Development.md b/Development.mdindex f652882895..7844fe0dd0 100644--- a/Development.md+++ b/Development.md@@ -1,346 +1,145 @@-# Development Guide+# Development-This guide covers setting up and developing the Cua monorepo.+This file is the contributor map for the Cua monorepo. Each component owns its+detailed setup, build, and test instructions. Start here, then follow the guide+next to the code you plan to change.-## Project Structure+## Choose a Component-The project is organized as a monorepo with these main packages:+| Area | Main paths | Toolchain | Start here |+| ---------------------------- | ----------------------------------------------------------- | ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |+| cua-driver | `libs/cua-driver/rust`, `libs/cua-driver/python` | Rust, platform SDKs, Nix on Linux | [`libs/cua-driver/README.md`](libs/cua-driver/README.md), [`libs/cua-driver/rust/README.md`](libs/cua-driver/rust/README.md) |+| Python SDKs and services | `libs/python`, `libs/cua-bench` | Python 3.12, uv | Package `pyproject.toml` and README |+| TypeScript SDKs | `libs/typescript` | Node.js, pnpm | [`libs/typescript/README.md`](libs/typescript/README.md) and its `package.json` scripts |+| CuaBot | `libs/cuabot` | Node.js, pnpm | [`libs/cuabot/README.md`](libs/cuabot/README.md) |+| Lume | `libs/lume` | Swift, Xcode command-line tools | [`libs/lume/Development.md`](libs/lume/Development.md) |+| Sandbox and container images | `libs/kasm`, `libs/lumier`, `libs/qemu-docker`, `libs/xfce` | Docker plus the component toolchain | README or Development file in the component |+| Public documentation | `docs/content/docs` | Node.js, pnpm, Fumadocs | [`docs/README.md`](docs/README.md) |+| Samples | `samples` | Depends on the sample | README next to the sample |-**Python Packages** (located in `libs/python/`):+The repository changes quickly. Directory listings in this root guide are an+orientation aid, not a package registry. Release targets come from+[`.github/workflows/release-bump-version.yml`](.github/workflows/release-bump-version.yml),+and test ownership comes from CI plus the component guides.-- `core/` - Base package with telemetry support-- `computer/` - Computer-use interface (CUI) library-- `agent/` - AI agent library with multi-provider support-- `som/` - Set-of-Mark parser-- `computer-server/` - Server component for VM-- `mcp-server/` - MCP server implementation-- `bench-ui/` - Benchmark UI utilities+## Common Setup-**Other Packages**:--- `libs/lume/` - Lume CLI (Swift)-- `libs/typescript/` - TypeScript packages including `cua-cli`-- `libs/cuabot/` - CuaBot multi-agent computer-use sandbox CLI--All Python packages are part of a [uv workspace](https://docs.astral.sh/uv/concepts/projects/workspaces/) which manages a shared virtual environment and dependencies.--## Quick Start--1. **Install Lume CLI** (required for local VM management):-- ```bash- /bin/bash -c "$(curl -fsSL https://cua.ai/lume/install.sh)"- ```--2. **Clone the repository**:-- ```bash- git clone https://github.com/trycua/cua.git- cd cua- ```--3. **Create `.env.local`** in the root directory with your API keys:-- ```bash- ANTHROPIC_API_KEY=your_anthropic_key_here- OPENAI_API_KEY=your_openai_key_here- ```--4. **Install Node.js dependencies**:-- ```bash- npm install -g pnpm # if not already installed- pnpm install- ```--5. **Install Python dependencies**:-- ```bash- pip install uv # if not already installed- uv sync- ```--6. **Open workspace in VS Code/Cursor**:-- ```bash- # For Python development- code .vscode/py.code-workspace-- # For Lume (Swift) development- code .vscode/lume.code-workspace- ```--7. **Install pre-commit hooks**:- ```bash- uv run pre-commit install- ```--<details>-<summary>Why use workspace files?</summary>--Using the workspace file is strongly recommended as it:--- Sets up correct Python environments for each package-- Configures proper import paths-- Enables debugging configurations-- Maintains consistent settings across packages--</details>--## Python Development--### Requirements--- **Python 3.12+** (see [`pyproject.toml`](./pyproject.toml) for exact requirements)-- **uv** - Python package manager-- **pnpm** - Node.js package manager--### Setup--Install all workspace dependencies with a single command:+Clone the repository and enter it:```bash-uv sync+git clone https://github.com/trycua/cua.git+cd cua```-This installs all dependencies in the virtual environment `.venv`. Each Cua package is installed in editable mode, so changes to source code are immediately reflected.--The `.venv` environment is configured as the default VS Code Python interpreter in [`.vscode/settings.json`](.vscode/settings.json).--## Code Formatting--The Cua project follows strict code formatting standards to ensure consistency across all packages.--### Python Formatting--#### Tools--- **[Black](https://black.readthedocs.io/)** - Code formatter-- **[isort](https://pycqa.github.io/isort/)** - Import sorter-- **[Ruff](https://beta.ruff.rs/docs/)** - Fast linter and formatter-- **[MyPy](https://mypy.readthedocs.io/)** - Static type checker (configured but not enforced in pre-commit)--All tools are automatically installed when you run `uv sync`.--#### Configuration--Formatting configuration is defined in [`pyproject.toml`](./pyproject.toml). See the `[tool.black]`, `[tool.ruff]`, `[tool.mypy]`, and `[tool.isort]` sections for all settings.--#### Key Rules--- **Line Length**: Maximum of 100 characters-- **Python Version**: Code must be compatible with Python 3.12+-- **Imports**: Automatically sorted (using Ruff's "I" rule)-- **Type Hints**: Required for all function definitions (strict mypy mode)--<details>-<summary>IDE configuration details</summary>--##### Python-specific settings--Python-specific IDE settings are configured in [`.vscode/settings.json`](.vscode/settings.json), including:--- Python interpreter path-- Format on save-- Code actions on save-- Black formatter configuration-- Ruff and MyPy integration--##### JS/TS-specific settings+Install only the toolchains required by your component:-JavaScript/TypeScript formatting settings are also in [`.vscode/settings.json`](.vscode/settings.json), ensuring Prettier is used for all JS/TS files.+| Work | Required tools |+| --------------------------- | --------------------------------------------------------------------------- |+| Python packages | Python 3.12 and [uv](https://docs.astral.sh/uv/) |+| TypeScript, CuaBot, or docs | The Node.js version required by the component and its declared pnpm version |+| cua-driver | Rust plus the target OS SDK; Nix for the reproducible Linux lanes |+| Lume | macOS, Swift, and Xcode command-line tools |+| Images | Docker or the image-specific builder documented by the component |-##### Recommended VS Code Extensions+The root uv workspace contains only the members declared in+[`pyproject.toml`](pyproject.toml). Other directories under `libs/python` are+independent packages; use their own `pyproject.toml` and CI workflow.-- **Black Formatter** – `ms-python.black-formatter`-- **Ruff** – `charliermarsh.ruff`-- **Pylance** – `ms-python.vscode-pylance`-- **isort** – `ms-python.isort`-- **Prettier** – `esbenp.prettier-vscode`-- **Mypy Type Checker** – `ms-python.mypy-type-checker`+The root Node package installs repository-wide Prettier only. Run `pnpm+install` inside `libs/typescript`, `libs/cuabot`, or `docs` for those+components' dependencies.-> VS Code will automatically suggest installing the recommended extensions when you open the workspace.+API keys are not required for ordinary builds and deterministic tests. Add+credentials only for a test or example that explicitly calls an external+provider, and never commit them.-</details>+## Root Formatting Hooks-<details>-<summary>Manual formatting commands</summary>--To manually format code:+Install the root Python development tools and optional Git hooks with:```bash-# Format all Python files using Black-uv run black .--# Sort imports using isort-uv run isort .--# Run Ruff linter with auto-fix-uv run ruff check .--# Run type checking with MyPy-uv run mypy .+uv sync --group dev+uv run pre-commit install```-</details>--#### Pre-commit Validation--Before submitting a pull request, ensure your code passes all formatting checks.--**Recommended: Run all hooks via pre-commit**+Run all configured hooks against the repository with:```bash-uv run pre-commit run+uv run pre-commit run --all-files```-This automatically runs Black, Ruff, isort, Prettier, TypeScript type checking, and other configured hooks. See [`.pre-commit-config.yaml`](.pre-commit-config.yaml) for the complete list.--> **Note:** MyPy is currently disabled in pre-commit hooks due to untyped codebase, but it's still configured and can be run manually.+The hooks currently run Prettier, the TypeScript workspace typecheck, Black,+isort, and Ruff. Install the TypeScript workspace dependencies before running+the typecheck hook. Mypy is configured in `pyproject.toml` but is not a+pre-commit gate.-<details>-<summary>Run individual tools manually</summary>+For a read-only repository-wide formatting check:```bash-# Python checks-uv run black --check .-uv run isort --check .-uv run ruff check .-uv run mypy .--# JavaScript/TypeScript checks+pnpm install --frozen-lockfilepnpm prettier:check```-</details>--### JavaScript / TypeScript Formatting--The project uses **Prettier** to ensure consistent formatting across all JS/TS/JSON/Markdown/YAML files.--#### Installation--All Node.js dependencies are managed via `pnpm`:--```bash-npm install -g pnpm # if not already installed-pnpm install-```--#### Usage--- **Check formatting** (without making changes):-- ```bash- pnpm prettier:check- ```--- **Automatically format files**:-- ```bash- pnpm prettier:format- ```+Component-specific Rust, TypeScript, Swift, and documentation checks remain in+their component guides. See [`TESTING.md`](TESTING.md) for the test map.-#### VS Code Integration+## cua-driver Development-The workspace config ensures Prettier is used automatically for JS/TS/JSON/Markdown/YAML files. Ensure `editor.formatOnSave` is enabled in VS Code.+cua-driver has three distinct validation layers:-### Swift Code (Lume)+1. Rust unit and protocol tests that do not require a target GUI application.+2. Source-built harness E2E tests that drive Electron, Tauri, and native toolkit+ fixtures in a real user desktop session.+3. Supporting installed-application checks whose runner declares whether they+ are canonical (the macOS Calculator/TextEdit rows) or optional (LibreOffice).-For Swift code in the `libs/lume` directory:+The Rust harnesses are the source of truth for desktop behavior. Python tests+do not duplicate that matrix. Start with:-- Follow the [Swift API Design Guidelines](https://www.swift.org/documentation/api-design-guidelines/)-- Use SwiftFormat for consistent formatting-- Code will be automatically formatted on save when using the lume workspace+- [`libs/cua-driver/rust/README.md`](libs/cua-driver/rust/README.md) for the Cargo workspace.+- [`libs/cua-driver/rust/crates/cua-driver/tests/README.md`](libs/cua-driver/rust/crates/cua-driver/tests/README.md) for test ownership.+- [`scripts/ci/README.md`](scripts/ci/README.md) for canonical OS runners.+- [Platform support](https://cua.ai/docs/reference/cua-driver/platform-support) for current capability boundaries.+- [How Cua Driver is validated](https://cua.ai/docs/concepts/how-cua-driver-is-validated) for the public evidence model.+- [Platform roadmap](https://cua.ai/docs/reference/cua-driver/platform-roadmap) for remaining work and platform boundaries.-Refer to [`libs/lume/Development.md`](./libs/lume/Development.md) for detailed Lume development instructions.+Windows and macOS desktop tests need a real user session. Windows requires an+active console or RDP session. macOS requires a logged-in session with+Accessibility and Screen Recording permissions. The hosted Linux Sway and+nested-compositor runners create controlled sessions; GNOME, KDE, and real+Xorg validation use an existing graphical login.-## Releasing Packages+## Documentation-Cua uses an automated GitHub Actions workflow to bump package versions and publish to PyPI/NPM.--> **Note:** The main branch is currently not protected. If branch protection is enabled in the future, the github-actions bot must be added to the bypass list for these workflows to commit directly.--### Version Bump & Publish Workflow--All packages are managed through a single consolidated workflow: [Bump Version & Publish](https://github.com/trycua/cua/actions/workflows/release-bump-version.yml)--**Supported packages:**--**Python (PyPI):**--- `pypi/agent` - AI agent library-- `pypi/auto` - Cross-platform automation library (mouse, keyboard, screen, window, clipboard, shell)-- `pypi/bench` - Benchmark toolkit for computer-use RL environments-- `pypi/computer` - Computer-use interface library-- `pypi/computer-server` - Server component for VM-- `pypi/core` - Base package with telemetry-- `pypi/mcp-server` - MCP server implementation-- `pypi/som` - Set-of-Mark parser--**JavaScript/TypeScript (NPM):**--- `npm/cli` - Cua command-line interface-- `npm/computer` - Computer client for TypeScript-- `npm/core` - Core TypeScript utilities-- `npm/cuabot` - Multi-agent computer-use sandbox--**Docker:**--- `docker/cuabot` - CuaBot container image--**How to use:**--1. Navigate to the [Bump Version & Publish workflow](https://github.com/trycua/cua/actions/workflows/release-bump-version.yml)-2. Click the "Run workflow" button in the GitHub UI-3. Select the **service/package** from the dropdown (e.g., `pypi/computer` or `npm/cli`)-4. Select the **bump type** (patch/minor/major) from the second dropdown-5. Click "Run workflow" to start the process--**What happens automatically:**--1. Version is bumped in the package configuration file-2. Changes are committed and pushed to main-3. Package is automatically published to PyPI or NPM-4. For `npm/cli`: Binaries are built and a GitHub release is created--> **Note:** For `pypi/computer`, the workflow also automatically bumps `pypi/agent` to maintain version compatibility.--<details>-<summary>Local Testing (Advanced)</summary>--The Makefile provides utility targets for local testing only:--```bash-# Test version bump locally (dry run)-make dry-run-patch-core--# View current versions-make show-versions-```+Public docs use Fumadocs and follow Diataxis:-**Note:** For production releases, always use the GitHub Actions workflows above instead of running Makefile commands directly.+- tutorials teach a first success;+- how-to guides solve a specific task;+- concepts explain constraints and design;+- reference pages state commands, contracts, support, and limits.-</details>+Run documentation commands from `docs`; see [`docs/README.md`](docs/README.md).+Contributor-only implementation notes should remain next to their component+instead of entering the public docs navigation.----+## Releases-<details>-<summary>Per-package tag patterns</summary>+Maintainers release packages through the+[CD: Bump Version](https://github.com/trycua/cua/actions/workflows/release-bump-version.yml)+workflow. Its `service` input is the current release-target registry. Each+package's `.bumpversion.cfg`, Cargo manifest, or package manifest owns its+version and tag format.-Each package uses its own tag format defined in `.bumpversion.cfg`:+Do not duplicate the complete release-target list or example versions in this+guide. They become stale as components are added. The workflow bumps one target+at a time and tag-triggered CD workflows perform publication.-- **cua-agent**: `agent-v{version}` (e.g., `agent-v0.4.35`)-- **cua-auto**: `auto-v{version}` (e.g., `auto-v0.1.0`)-- **cua-bench**: `bench-v{version}` (e.g., `bench-v0.1.0`)-- **cua-computer**: `computer-v{version}` (e.g., `computer-v0.4.7`)-- **cua-computer-server**: `computer-server-v{version}` (e.g., `computer-server-v0.1.27`)-- **cua-core**: `core-v{version}` (e.g., `core-v0.1.9`)-- **cua-mcp-server**: `mcp-server-v{version}` (e.g., `mcp-server-v0.1.14`)-- **cua-som**: `som-v{version}` (e.g., `som-v0.1.3`)-- **cuabot**: `cuabot-v{version}` (e.g., `cuabot-v1.0.0`)-- **cuabot (docker)**: `docker-cuabot-v{version}` (e.g., `docker-cuabot-v1.0.0`)+The root `Makefile` provides local version inspection and dry-run helpers. It+does not publish production releases.-</details>+## Generated and Local Files-</details>+Do not commit build products, staged harness binaries, local VM artifacts,+credentials, permission databases, or editor-specific state. Promote an+artifact into source control only when it becomes a stable fixture, sample, or+maintained document.diff --git a/README.md b/README.mdindex 04a52135fd..fd7d0edca5 100644--- a/README.md+++ b/README.md@@ -59,9 +59,9 @@----## Cua Drivers - Background computer-use on macOS and Windows, with Linux pre-release+## Cua Drivers - Background computer-use on macOS, Windows, and Linux-Drive native desktop apps **in the background**. Agents click, type, and verify without stealing the cursor or focus. Use the same CLI and MCP server on macOS and Windows from Claude Code, Cursor, Codex, OpenClaw, and custom clients. Linux support is available as a pre-release backend while platform testing is still in progress.+Drive native desktop apps **in the background**. Agents click, type, and verify without stealing the cursor or focus. Use the same CLI and MCP server on macOS, Windows, and Linux from Claude Code, Cursor, Codex, OpenClaw, and custom clients. Linux supports X11 and compositor-specific Wayland routes with explicit limits for raw background input.**macOS / Linux**@@ -93,7 +93,6 @@ Build agents that see screens, click buttons, and complete tasks autonomously. Opip install cua```-```python# Requires Python 3.11 or laterfrom cua import Sandbox, Image@@ -120,7 +119,6 @@ async with Sandbox.ephemeral(Image.linux()) as sb: # or .macos() .windows() .aEvaluate computer-use agents on OSWorld, ScreenSpot, Windows Arena, and custom tasks. Export trajectories for training.-```bash# Clone, install, and create base imagegit clone https://github.com/trycua/cua && cd cua/cua-bench@@ -138,7 +136,6 @@ cb run dataset datasets/cua-bench-basic --agent cua-agent --max-parallel 4Create and manage macOS/Linux VMs with near-native performance on Apple Silicon using Apple's Virtualization.Framework.-```bash# Install Lume/bin/bash -c "$(curl -fsSL https://cua.ai/lume/install.sh)"@@ -163,15 +160,15 @@ of Setup Assistant on its first display boot; see [issue #2155](https://github.c## Packages-| Package | Description |-| --------------------------------------------------------------------------- | ---------------------------------------------------------- |-| [cua-driver](libs/cua-driver/README.md) | Background computer-use agent for macOS, Windows, and Linux |-| [cua-agent](https://cua.ai/docs/cua/reference/agent-sdk) | AI agent framework for computer-use tasks |-| [cua-sandbox](https://cua.ai/docs/cua/reference/sandbox-sdk) | SDK for creating and controlling sandboxes |-| [cua-computer-server](https://cua.ai/docs/cua/reference/sandbox-sdk) | Driver for UI interactions and code execution in sandboxes |-| [cua-bench](https://cua.ai/docs/cuabench) | Benchmarks and RL environments for computer-use |-| [lume](https://cua.ai/docs/lume) | macOS/Linux VM management on Apple Silicon |-| [lumier](https://cua.ai/docs/lume/guide/advanced/lumier) | Docker-compatible interface for Lume VMs |+| Package | Description |+| -------------------------------------------------------------------- | ----------------------------------------------------------- |+| [cua-driver](libs/cua-driver/README.md) | Background computer-use agent for macOS, Windows, and Linux |+| [cua-agent](https://cua.ai/docs/cua/reference/agent-sdk) | AI agent framework for computer-use tasks |+| [cua-sandbox](https://cua.ai/docs/cua/reference/sandbox-sdk) | SDK for creating and controlling sandboxes |+| [cua-computer-server](https://cua.ai/docs/cua/reference/sandbox-sdk) | Driver for UI interactions and code execution in sandboxes |+| [cua-bench](https://cua.ai/docs/cuabench) | Benchmarks and RL environments for computer-use |+| [lume](https://cua.ai/docs/lume) | macOS/Linux VM management on Apple Silicon |+| [lumier](https://cua.ai/docs/lume/guide/advanced/lumier) | Docker-compatible interface for Lume VMs |## Resourcesdiff --git a/TESTING.md b/TESTING.mdindex 38600a95ad..374f8fe03c 100644--- a/TESTING.md+++ b/TESTING.md@@ -1,106 +1,135 @@-# Testing Guide for Cua+# Testing-Quick guide to running tests and understanding the test architecture.+Cua is a multi-language monorepo. There is no single root command that proves+every package, desktop, VM, and image. Run the tests owned by the components you+changed and use the corresponding CI workflow as the executable source of+truth.-## 🚀 Quick Start+## Test Map-```bash-# Install dependencies-pip install pytest pytest-asyncio pytest-mock pytest-cov+| Area | Deterministic tests | Integration or E2E owner |+| -------------------- | ----------------------------------------------------- | -------------------------------------------------------------------------------- |+| Python SDKs | Package `tests/` directories with pytest | Package-specific integration tests and `tests/integration` |+| TypeScript SDKs | Package Vitest/typecheck scripts | Package-owned integration tests |+| cua-driver | Rust unit, schema, protocol, and compile tests | Canonical Rust desktop harnesses on Windows, macOS, Linux X11, and Linux Wayland |+| Lume | Swift package tests | VM and unattended-setup checks documented by Lume |+| Public docs | Generator drift, hygiene, links, and production build | Rendered Fumadocs site |+| Images and sandboxes | Component build and schema tests | Image-specific smoke or VM tests |-# Install package-cd libs/python/core-pip install -e .+Path-filtered CI avoids running unrelated operating systems, so a green job for+one component does not validate another component.-# Run tests-export CUA_TELEMETRY_ENABLED=false # or $env:CUA_TELEMETRY_ENABLED="false" on Windows-pytest tests/ -v-```+## Python-## 🧪 Running Tests+For a member of the root uv workspace:```bash-# All packages-pytest libs/python/*/tests/ -v+uv sync --group test+CUA_TELEMETRY_ENABLED=false uv run pytest libs/python/<package>/tests -v+```++Packages outside the root uv workspace should be installed from their own+`pyproject.toml`. The current package matrix and installation sequence live in+[`.github/workflows/ci-test-python.yml`](.github/workflows/ci-test-python.yml).-# Specific package-cd libs/python/core && pytest tests/ -v+## TypeScript-# With coverage-pytest tests/ --cov --cov-report=html+Run from `libs/typescript`:-# Specific test-pytest tests/test_telemetry.py::TestTelemetryEnabled::test_telemetry_enabled_by_default -v+```bash+pnpm install --frozen-lockfile+pnpm test+pnpm typecheck+pnpm format:check```-## 🏗️ Test Architecture+Use a package's own `package.json` scripts when working outside that workspace,+including CuaBot and the documentation site.-**Principles**: SRP (Single Responsibility) + Vertical Slices + Testability+## cua-driver Unit and Protocol Tests-```-libs/python/-├── core/tests/ # Tests ONLY core-├── agent/tests/ # Tests ONLY agent-└── computer/tests/ # Tests ONLY computer+Run from `libs/cua-driver/rust`. Focused examples:++```bash+cargo test -p cua-driver-core --locked+cargo test -p cua-driver --test protocol_mcp_test --locked```-Each test file = ONE feature. Each test class = ONE concern.+Linux source and package checks run through Nix. Windows and Linux compile gates+are split into OS-specific workflows. See+[`libs/cua-driver/rust/README.md`](libs/cua-driver/rust/README.md) for workspace+commands.-## ➕ Adding New Tests+Unit and protocol tests do not prove that desktop input reached a real+application.-1. Create `test_*.py` in the appropriate package's `tests/` directory-2. Follow the pattern:+## cua-driver Harness E2E-```python-"""Unit tests for my_feature."""-import pytest-from unittest.mock import patch+The canonical desktop suites build repository-owned applications, drive them+through the Rust driver, and verify application or desktop state independently+from the tool response. Foreground/background delivery and AX/PX addressing are+dimensions of each action row.-class TestMyFeature:- """Test MyFeature class."""+Canonical entry points:- def test_initialization(self):- """Test that feature initializes."""- from my_package import MyFeature- feature = MyFeature()- assert feature is not None+```text+Linux X11/session: scripts/ci/linux/run-rust-e2e.sh+Linux Sway: scripts/ci/linux/run-rust-e2e-wayland.sh+Linux nested: scripts/ci/linux/run-rust-e2e-inject.sh+Linux GNOME/KDE: scripts/ci/linux/run-rust-e2e-desktop.sh <gnome|kde>+Linux real Xorg: scripts/ci/linux/run-rust-e2e-desktop.sh xorg+Windows: .\scripts\ci\windows\run-rust-e2e.ps1 -RequireGui+macOS: scripts/ci/macos/run-rust-e2e.sh```-3. Mock external dependencies:+The hosted Sway and nested-compositor runners create controlled sessions.+GNOME, KDE, real Xorg, Windows, and macOS use an existing graphical login. The+suites are often maintainer-triggered and retain typed case/results,+screenshots, accessibility state, trajectories, logs, and video where the lane+supports it. The reporter rejects missing rows, false-success responses,+undeclared outcomes, and incomplete required evidence.-```python-def mock_api():- with patch("my_package.api_client") as mock:- yield mock-```+See:-## 🔄 CI/CD+- [`libs/cua-driver/docs/test-harnesses-guide.md`](libs/cua-driver/docs/test-harnesses-guide.md)+- [`libs/cua-driver/docs/test-matrix.md`](libs/cua-driver/docs/test-matrix.md)+- [Platform support and validation](https://cua.ai/docs/reference/cua-driver/platform-support)-Tests run automatically on every PR via GitHub Actions (`.github/workflows/ci-python-tests.yml`):+## Lume-- Matrix strategy: each package tested separately-- Python 3.12-- ~2 minute runtime+Run from `libs/lume`:-## 🐛 Troubleshooting+```bash+swift test+```-**ModuleNotFoundError**: Run `pip install -e .` in package directory+VM-dependent and unattended-setup checks have additional prerequisites in+[`libs/lume/Development.md`](libs/lume/Development.md).-**Tests fail in CI but pass locally**: Set `CUA_TELEMETRY_ENABLED=false`+## Public Documentation-**Async tests error**: Install `pytest-asyncio` and use `@pytest.mark.asyncio`+Run from `docs`:-**Mock not working**: Patch at usage location, not definition:+```bash+pnpm install --frozen-lockfile+pnpm docs:check+pnpm docs:check-hygiene+pnpm docs:check-links+pnpm build+```-```python-# ✅ Right-@patch("my_package.module.external_function")+The production build validates MDX compilation and static route generation.+The generator check prevents generated CLI and API references from drifting+from source.-# ❌ Wrong-@patch("external_library.function")-```+## Before Opening a Pull Request----+1. Run focused tests while developing.+2. Run the complete deterministic test owner for every component changed.+3. Run the affected interactive E2E lane when desktop behavior or its contract changes.+4. Run formatting and documentation checks for modified files.+5. Record any test that could not run and why.-**Questions?** Check existing tests for examples or open an issue.+Do not turn missing dependencies, desktop sessions, fixtures, or permissions+into a reduced green run. Environment failures and unsupported capabilities+must remain visible.diff --git a/docs/content/docs/concepts/how-cua-driver-is-validated.mdx b/docs/content/docs/concepts/how-cua-driver-is-validated.mdxnew file mode 100644index 0000000000..310c8dd3e7--- /dev/null+++ b/docs/content/docs/concepts/how-cua-driver-is-validated.mdx@@ -0,0 +1,115 @@+---+title: "How Cua Driver Is Validated"+description: "Why Cua Driver uses source-built desktop harnesses, independent application oracles, and retained evidence to validate behavior between releases."+---++# How Cua Driver Is Validated++Desktop automation crosses boundaries that ordinary unit tests cannot observe.+A request can be valid, reach an operating-system API, and return success while+the target application receives nothing. Cua Driver therefore treats protocol+correctness and observed desktop behavior as two different kinds of evidence.++## Two layers answer different questions++Unit and protocol tests answer whether the driver made a deterministic decision+correctly. They cover schemas, transport, sessions, element identity, route+selection, capture helpers, coordinate conversion, and structured errors. They+run without a target GUI application and catch inexpensive regressions early.++Harness end-to-end tests answer whether an action actually reached a real+desktop surface. They build a small application from source, launch it in a+real graphical user session, drive it through the Rust driver, and inspect+state owned by the application or desktop rather than trusting the response.++Neither layer replaces the other. An E2E result does not exhaustively test+protocol edge cases, and a unit test cannot prove that a click changed an+application.++## The harnesses model representative surfaces++The shared harness presents the same deterministic web behavior through+representative renderer hosts on each operating system. Native harnesses then+exercise accessibility, controls, windowing, capture, and input APIs that a+shared renderer cannot represent. The current surface inventory belongs in+[Platform support](/reference/cua-driver/platform-support); this explanation+describes the validation model that remains stable as surfaces are added.++These applications are fixtures, not mocks. Each is compiled and launched as a+real process. Their purpose is to expose deterministic state for clicks, text,+keys, scrolling, dragging, child windows, controls, and editor behavior. This+makes a failure reproducible without depending on the changing state of an+installed third-party application.++## One catalog describes each behavior cell++The Rust catalog records the dimensions that affect delivery:++| Dimension | Examples |+| --------------- | ------------------------------------------------------------------------------------------------------ |+| Action | left click, right click, double click, type text, key, hotkey, scroll, drag, child window, editor save |+| Addressing | AX element or PX coordinate |+| Delivery | foreground or background |+| Scope | target window or full desktop |+| Surface | shared renderer, native toolkit, embedded web view, or compositor |+| Expected result | delivered or one exact structured refusal |++Foreground and background are dimensions of an action, not separate test+families. When a surface supports both delivery modes, both belong in the same+catalog. A platform-specific runner establishes the desktop session and+collects evidence, but it does not redefine the expected behavior.++## Independent oracles define success++A successful driver response is not a passing E2E result. A delivered action+must change state that the fixture or desktop independently owns.++| Oracle | What it establishes |+| -------------------- | --------------------------------------------------------------------------------------- |+| Fixture state | The application recorded the click, text, key, selection, scroll, drag, or saved state. |+| Accessibility state | UIA, AX, or AT-SPI reports the expected control value or structure. |+| Pixel state | Before and after images contain the required visible change. |+| Focus and z-order | A background action did not activate or raise the target. |+| Cursor state | The physical user cursor did not move when background delivery promised that property. |+| Leaked-input journal | The foreground sentinel did not receive input intended for the background target. |+| Protocol state | An unsupported route returned the exact declared refusal. |++Background checks combine target-state and side-effect oracles. This matters+because input delivered to the wrong foreground application is worse than an+honest refusal. A refusal passes only when its exact code is expected and the+desktop remains unchanged.++## Evidence makes a result auditable++Canonical hosted GUI runs retain the typed result for every declared cell, the+source commit, per-cell desktop video, before and after state, the tool+trajectory, fixture journals, driver logs, and environment preflight. The+GitHub Actions summary links each matrix row to its artifacts.++The reporter rejects missing or duplicate rows, undeclared outcomes,+contradictory results, and incomplete required evidence. A failed environment+cannot silently turn into a smaller green matrix, and an `ok` response without+an observed effect remains a failure.++## Release validation balances cost and fidelity++Deterministic unit, protocol, compile, and packaging checks run automatically+where configured. Interactive E2E suites require a real graphical session and+are maintainer-triggered because they are slower and more environment-sensitive.+Windows and Linux have dispatchable GitHub Actions workflows; macOS runs the+same Rust catalog in a logged-in session with Accessibility and Screen+Recording permissions.++Each accepted run builds the driver and fixtures from one exact source commit.+Support claims change only after an unchanged behavior cell produces new+application-owned evidence or an explicit structured refusal. Skips, weaker+oracles, and command-return-only checks do not establish support.++## Related reference++- [Platform support](/reference/cua-driver/platform-support) records currently+ proven environments and limitations.+- [Platform roadmap](/reference/cua-driver/platform-roadmap) records remaining+ engineering work and hard platform boundaries.+- [Interface contracts](/reference/cua-driver/contracts) defines public driver+ behavior and refusal semantics.diff --git a/docs/content/docs/concepts/index.mdx b/docs/content/docs/concepts/index.mdxindex 4f317ee608..37cd8a7fe9 100644--- a/docs/content/docs/concepts/index.mdx+++ b/docs/content/docs/concepts/index.mdx@@ -5,4 +5,4 @@ description: "Understand the design ideas behind Cua and how its main pieces fitUse these pages when you want the model behind Cua rather than a step-by-step guide or API table. They explain what computer use means in Cua, how Cua Driver keeps the desktop usable while it acts, and how Cua Sandbox gives an agent a disposable computer.-Start with [What is computer use?](/concepts/what-is-computer-use) for the basic model. Read [Best-effort background](/concepts/the-no-foreground-contract) to understand Cua Driver's default behavior on a shared machine, then [Capture and delivery modalities](/concepts/capture-and-delivery-modalities) for the action axes. Read [How sandboxes work](/concepts/how-sandboxes-work) when you need the model for disposable computers. Read [How Lume prepares vanilla macOS VMs](/concepts/how-lume-unattended-setup-works) for the local VM setup model, and [How SIP works in Lume VMs](/concepts/how-sip-works-in-lume-vms) for signed boot-policy changes.+Start with [What is computer use?](/concepts/what-is-computer-use) for the basic model. Read [Best-effort background](/concepts/the-no-foreground-contract) to understand Cua Driver's default behavior on a shared machine, then [Capture and delivery modalities](/concepts/capture-and-delivery-modalities) for the action axes. [How Cua Driver is validated](/concepts/how-cua-driver-is-validated) explains why desktop support requires application-owned E2E evidence in addition to unit tests. Read [How sandboxes work](/concepts/how-sandboxes-work) when you need the model for disposable computers. Read [How Lume prepares vanilla macOS VMs](/concepts/how-lume-unattended-setup-works) for the local VM setup model, and [How SIP works in Lume VMs](/concepts/how-sip-works-in-lume-vms) for signed boot-policy changes.diff --git a/docs/content/docs/concepts/meta.json b/docs/content/docs/concepts/meta.jsonindex f2be3f720f..165fdab216 100644--- a/docs/content/docs/concepts/meta.json+++ b/docs/content/docs/concepts/meta.json@@ -1 +1 @@-{ "title": "Concepts", "icon": "Lightbulb", "pages": ["index", "what-is-computer-use", "the-no-foreground-contract", "capture-and-delivery-modalities", "how-sandboxes-work", "how-lume-unattended-setup-works", "how-sip-works-in-lume-vms", "how-lume-expands-macos-disks"] }+{ "title": "Concepts", "icon": "Lightbulb", "pages": ["index", "what-is-computer-use", "the-no-foreground-contract", "capture-and-delivery-modalities", "how-cua-driver-is-validated", "how-sandboxes-work", "how-lume-unattended-setup-works", "how-sip-works-in-lume-vms", "how-lume-expands-macos-disks"] }diff --git a/docs/content/docs/reference/cua-driver/development.mdx b/docs/content/docs/reference/cua-driver/development.mdxindex b2b17944d6..70a939334e 100644--- a/docs/content/docs/reference/cua-driver/development.mdx+++ b/docs/content/docs/reference/cua-driver/development.mdx@@ -23,8 +23,34 @@ they stay close to implementation changes.- `libs/cua-driver/rust/README.md` maps the Cargo workspace and test classes.- `libs/cua-driver/rust/crates/cua-driver/tests/README.md` explains Rust test naming and ignored GUI lanes.- `libs/cua-driver/tests/fixtures/README.md` explains source-built harness apps and staged outputs.-- `libs/cua-driver/rust/crates/cua-driver/tests/README.md` explains Rust test naming and ignored GUI lanes.- `libs/cua-driver/scripts/README.md` explains install and VM sync helpers.+## Desktop Validation Entry Points++The canonical GUI runners execute the complete Rust harness catalog. Their+internal CI lanes may split shared, native, and capture owners for reporting,+but contributors do not select those partitions directly.++```text+Linux X11/session: scripts/ci/linux/run-rust-e2e.sh+Linux Sway: scripts/ci/linux/run-rust-e2e-wayland.sh+Linux nested: scripts/ci/linux/run-rust-e2e-inject.sh+Linux GNOME/KDE: scripts/ci/linux/run-rust-e2e-desktop.sh <gnome|kde>+Linux real Xorg: scripts/ci/linux/run-rust-e2e-desktop.sh xorg+Windows: .\scripts\ci\windows\run-rust-e2e.ps1 -RequireGui+macOS: scripts/ci/macos/run-rust-e2e.sh+```++The Sway runner creates a controlled stock-wlroots session. Nix owns the Linux+source/package gate and provides the optional nested `cua-compositor` session;+it is not the wrapper for every Linux E2E environment. GNOME, KDE, real Xorg,+Windows, and macOS use an existing graphical login. Windows needs an+interactive console or RDP session, and macOS needs a logged-in session with+the required Accessibility and Screen Recording permissions. See+[Platform support](/reference/cua-driver/platform-support) for current capability+boundaries, [How Cua Driver is validated](/concepts/how-cua-driver-is-validated)+for the evidence model, and [Platform roadmap](/reference/cua-driver/platform-roadmap)+for remaining work.+Build artifacts, VM logs, and local verification journals should stay out ofgit unless they have been promoted into a stable fixture or contributor doc.diff --git a/docs/content/docs/reference/cua-driver/limits.mdx b/docs/content/docs/reference/cua-driver/limits.mdxindex 30654104a2..75b7cdc1a0 100644--- a/docs/content/docs/reference/cua-driver/limits.mdx+++ b/docs/content/docs/reference/cua-driver/limits.mdx@@ -1,11 +1,16 @@---title: Known Limits-description: Documented behavioral limits of Cua Driver and available workarounds+description: Documented behavioral limits of Cua Driver and available alternatives---import { Callout } from 'fumadocs-ui/components/callout';-Cua Driver uses best-effort background delivery for every app it can reach via accessibility or routed pixel input. A handful of targets and platform quirks fall outside that envelope. The macOS cases come first, followed by the Linux session-stack limits.+Cua Driver attempts background delivery only when the operating system and+target expose a route that can be addressed safely. Unsupported shapes return a+structured refusal instead of reporting a silent success. This page lists+target-specific constraints and available alternatives; see+[Platform Support](/reference/cua-driver/platform-support) for the broader+operating-system matrix.---@@ -69,7 +74,7 @@ Cua Driver uses best-effort background delivery for every app it can reach via a1. Use `set_value({pid, element_index, value: "..."})` to write the field's value directly. No keyboard event involved; no focus handoff required.2. AX-click a commit-equivalent button (Go, Submit, Send, OK) rather than relying on Return.-3. Un-minimize the window (`hotkey({pid, keys: ["cmd", "m"]})` or click the Dock icon). This interrupts best-effort background behavior for that window.+3. Restore the window through the Dock or an application-specific Window menu command. This interrupts best-effort background behavior for that window. (`Cmd+M` minimizes a window; it does not restore one.)<Callout type="info">`set_value` is the correct approach 90% of the time. It sidesteps both the minimized-focus issue and the general "which event commits this field" ambiguity.@@ -77,24 +82,42 @@ Cua Driver uses best-effort background delivery for every app it can reach via a----## Native Wayland apps can't receive synthetic keystrokes+## Native Wayland background keyboard input is focus-bound-**Affected:** GTK/Qt apps running as native Wayland clients on GNOME Mutter or KDE KWin (no X11 surface).+**Affected:** Unfocused GTK/Qt apps running as native clients on standard+Wayland compositors, including Sway/wlroots, GNOME/Mutter, and KDE/KWin (no X11+surface).-**Symptom:** `press_key` / `hotkey`, and `type_text` into a field that is not AT-SPI-editable, return success but the keystroke never reaches the app. Clicks and element actions on the same app work fine.+**Symptom:** A background `press_key`, `hotkey`, or `type_text` request for a+field that is not AT-SPI-editable returns structured+`background_unavailable`. Foreground delivery may also refuse when the desktop+has no target-addressable activation or raw-input backend.-**Cause:** Wayland blocks one client from synthesizing input into another by design. The compositor-cooperative paths that would re-enable it are not free here: `libei` requires a one-time RemoteDesktop-portal grant, and `wtype` (virtual-keyboard) is wlroots-only, so it fails on Mutter/KWin.+**Cause:** Wayland blocks one ordinary client from targeting another client's+surface with synthetic input. Portal/libei input on GNOME and KDE follows the+compositor's active seat and requires a RemoteDesktop grant. The virtual+keyboard used on wlroots compositors is also focus-bound, even though it does+not require the same portal route.**Workarounds:**1. Type into accessible text fields with `type_text`. AT-SPI `insertText` writes the field directly, with no synthetic key event involved.2. Drive controls by `element_index` (`click`, `set_value`) instead of keyboard shortcuts where an equivalent control exists.-3. Run the app under XWayland (`GDK_BACKEND=x11` / `QT_QPA_PLATFORM=xcb`). It then exposes an X11 surface and the X11 `XTEST` keyboard path applies.+3. Retry with `delivery_mode:"foreground"` when the desktop exposes a verified+ activation and input adapter.+4. Run the app under XWayland (`GDK_BACKEND=x11` / `QT_QPA_PLATFORM=xcb`). It+ then exposes an X11 surface and the X11 keyboard paths apply.<Callout type="info">- This is keyboard-only. Background **clicks and element actions** land on native Wayland via AT-SPI; coordinate (vision) clicks land too, resolved to the element under the pixel.+ Background element actions can land through AT-SPI. A coordinate left click+ can also land when its point resolves to an actionable AT-SPI element. This+ does not establish arbitrary raw background PX delivery.</Callout>+The opt-in nested `cua-compositor` is a separate, compositor-owned environment.+Its private per-surface injection protocol is not constrained like an ordinary+client on Sway, GNOME, or KDE, and its capabilities are documented separately.+---## GTK4 reports (0,0) screen coordinates over AT-SPI (handled)diff --git a/docs/content/docs/reference/cua-driver/meta.json b/docs/content/docs/reference/cua-driver/meta.jsonindex 2773873c78..f90c04713f 100644--- a/docs/content/docs/reference/cua-driver/meta.json+++ b/docs/content/docs/reference/cua-driver/meta.json@@ -1 +1,17 @@-{ "title": "Cua Driver", "pages": ["cli-reference", "macos-permissions", "embedding", "mcp-tools", "mcp-tool-notes", "action-selection-policy", "contracts", "process-model", "limits", "development"] }+{+ "title": "Cua Driver",+ "pages": [+ "cli-reference",+ "macos-permissions",+ "embedding",+ "mcp-tools",+ "mcp-tool-notes",+ "action-selection-policy",+ "contracts",+ "platform-support",+ "platform-roadmap",+ "process-model",+ "limits",+ "development"+ ]+}diff --git a/docs/content/docs/reference/cua-driver/platform-roadmap.mdx b/docs/content/docs/reference/cua-driver/platform-roadmap.mdxnew file mode 100644index 0000000000..d6658a9610--- /dev/null+++ b/docs/content/docs/reference/cua-driver/platform-roadmap.mdx@@ -0,0 +1,108 @@+---+title: Platform Roadmap+description: Remaining Cua Driver platform work, evidence gaps, and operating-system boundaries without implied delivery dates.+---++This roadmap records capability and validation work for Cua Driver. It does not+promise dates. A capability moves to [Platform Support](/reference/cua-driver/platform-support)+only after a canonical Rust harness observes the required application and+desktop state.++## Status vocabulary++| Status | Meaning |+| ----------------- | ------------------------------------------------------------------------------------------------- |+| Proven | Canonical harness evidence supports the current public claim. |+| Evidence gap | Code or platform APIs may support the behavior, but representative harness evidence is missing. |+| Engineering work | A concrete driver, fixture, observer, or runner change is required. |+| Experimental | An opt-in backend or environment has evidence but is not part of the general support contract. |+| Platform boundary | The ordinary OS security or windowing model does not expose a safe general route for the behavior. |++An evidence gap does not mean a behavior is impossible. A platform boundary+does not prevent narrower semantic routes, foreground delivery, or operation+inside an environment that owns the compositor.++## Cross-platform priorities++| Area | Current state | Next acceptance condition |+| ---------------------------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |+| Shared application catalog | Electron and Tauri use one typed Rust behavior catalog across operating systems. | Keep the same action, AX/PX, foreground/background, scope, and oracle dimensions on every representative host. |+| Native application catalogs | Windows, macOS, and Linux have toolkit-specific source-built harnesses. | Expand native rows toward the shared action cross-product where the toolkit exposes an equivalent behavior. |+| Background safety | Background cells attach focus, z-order, cursor, and leaked-input oracles. | Add the same side-effect owners to every new background delivery or refusal row. |+| Release evidence | Hosted GUI lanes retain typed results, trajectories, screenshots, logs, and per-cell video. | Require accepted exact-source-SHA runs for affected platforms before changing release support claims. |+| Representative environments | Hosted lanes cover Windows, Linux X11, Sway, and the nested compositor; macOS uses a logged-in host. | Close named GNOME, KDE, real-Xorg, and renderer evidence gaps without weakening fixtures or oracles. |++## Windows++| Work item | Status | Acceptance condition |+| ------------------------------------------ | ---------------- | ------------------------------------------------------------------------------------------------------------------------------ |+| Broader WPF PX gestures and keyboard paths | Evidence gap | Typed foreground/background rows observe native control state and all required desktop side effects. |+| WinUI 3 pointer and background coverage | Evidence gap | Right click, double click, drag, scroll, and keyboard contracts are declared and empirically observed or refused. |+| WebView2 native-input coverage | Evidence gap | Native pointer and keyboard cells complement the existing page/CDP and background-left-click evidence. |+| Elevated-integrity boundary | Engineering work | A controlled fixture proves the `background_uipi_blocked` contract across process integrity levels. |++Windows integrity isolation is a platform boundary: a lower-integrity process+cannot generally inject input into a higher-integrity target. The roadmap item+is to detect and prove that refusal, not bypass the operating-system boundary.++## macOS++| Work item | Status | Acceptance condition |+| ---------------------------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------- |+| Native AppKit action cross-product | Evidence gap | Press key, hotkey, AX-addressed pointer gestures, and additional controls have application-owned evidence. |+| SwiftUI transient-window discovery | Engineering work | The opened popover or panel is independently visible through targeted window and AX enumeration. |+| Installed-app confidence checks | Proven | Keep the typed Calculator and TextEdit rows as supporting evidence; repo-local fixtures remain the primary behavioral contract. |++Accessibility and Screen Recording consent remain platform prerequisites.+Off-Space SwiftUI tree stripping, minimized keyboard commits, and applications+that accept only HID-tap input are platform or target boundaries; their safe+alternatives remain documented in [Known Limits](/reference/cua-driver/limits).++## Linux X11++| Work item | Status | Acceptance condition |+| --------------------------------- | ------------ | ----------------------------------------------------------------------------------------------------- |+| Hosted Openbox/Xvfb catalog | Proven | Preserve the complete shared and GTK catalog with exact delivery or refusal outcomes. |+| Real-Xorg MPX and uinput behavior | Evidence gap | A maintainer lane proves routes that Xvfb cannot represent, including multi-pointer desktop behavior. |+| Additional toolkit confidence | Evidence gap | Add a surface only when it represents a materially different input or accessibility contract. |++X11 permits more synthetic-input routes than Wayland, but an X server accepting+an event does not prove the target toolkit handled it. Application-owned state+remains required.++## Linux Wayland++| Environment | Work item | Status | Acceptance condition |+| ----------------------- | --------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------- |+| Sway/wlroots | Preserve the accepted shared and native catalog | Proven | Keep every declared Electron, Tauri, and GTK cell delivering or returning its exact refusal through unchanged oracles. |+| GNOME/Mutter | Shared renderer catalog and portal video | Evidence gap | Electron and representative WebKitGTK rows pass with reporter-owned per-cell recordings. |+| KDE/KWin | Target-addressable activation and full matrix | Engineering work | A supported KWin adapter activates the selected target and an independent observer verifies it before portal input. |+| Nested `cua-compositor` | Complete shared catalog and protocol coverage | Experimental | One accepted full catalog proves renderer behavior, Unicode text, and canonical parallel drag. |+| Other wlroots | Representative compatibility | Evidence gap | Add a lane only after a user report demonstrates behavior that the Sway lane does not represent. |++### Standard Wayland boundary++An ordinary client on a standard Wayland compositor cannot generally send raw+pointer or keyboard input to an arbitrary occluded, unfocused surface.+Reconstructed coordinates solve target geometry, but they do not change which+surface receives compositor-seat input. Semantic AT-SPI actions and PX-addressed+hit-testing can still operate in the background when a coordinate resolves to+an actionable accessible element.++The optional nested `cua-compositor` has a different capability because it owns+the compositor and can route input directly to a selected client surface. Work+in that lane can improve controlled nested sessions, but cannot establish a+general Sway, GNOME, KDE, or stock-Wayland claim.++## Promotion rules++A roadmap item becomes supported only when all of the following are true:++- the behavior is a typed Rust case rather than a second OS-specific matrix;+- delivery changes fixture-owned state, or refusal returns the exact declared code;+- every required focus, z-order, cursor, leaked-input, capture, and scope oracle passes;+- the result identifies the operating system, window system, surface, delivery route, and source commit;+- representative evidence is retained and the public support reference is updated.++For the reasoning behind these requirements, see+[How Cua Driver is validated](/concepts/how-cua-driver-is-validated).diff --git a/docs/content/docs/reference/cua-driver/platform-support.mdx b/docs/content/docs/reference/cua-driver/platform-support.mdxnew file mode 100644index 0000000000..c396814529--- /dev/null+++ b/docs/content/docs/reference/cua-driver/platform-support.mdx@@ -0,0 +1,91 @@+---+title: Platform Support+description: Current operating-system and window-system support, proven surfaces, and known capability boundaries.+---++import { Callout } from 'fumadocs-ui/components/callout';++cua-driver supports Windows, macOS, and Linux. Support is defined by observed+behavior in a real application, not by whether a tool call returned success.+The exact delivery route depends on the operating system, window system,+application toolkit, action, and whether the target may be brought forward.++For definitions of AX, PX, foreground, background, window scope, and desktop+scope, see [Capture and delivery modalities](/concepts/capture-and-delivery-modalities).++## Support levels++| Level | Meaning |+| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |+| Supported | A canonical Rust harness test proves the result against application-owned or desktop-owned state. |+| Supported with limits | Common paths are proven, but the platform or window system cannot safely provide every delivery shape. Unsupported paths return a structured refusal. |+| Experimental | The backend exists, but representative coverage is incomplete. Do not assume unlisted actions work. |++## Platform overview++| Platform | Window system and automation APIs | Current state |+| ------------- | -------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| Windows | Win32, UI Automation (UIA), native input, and targeted window messages | Supported. Canonical coverage includes Electron, Tauri, WPF, WinUI 3, and WebView2. Some background Chromium gestures and elevated-integrity boundaries remain unavailable or unproven. |+| macOS | AppKit, Accessibility (AX), Quartz/HID, and ScreenCaptureKit | Supported. Canonical coverage includes Electron, Tauri, AppKit, SwiftUI, and WKWebView. Accessibility and screen-recording permissions are required. Some background scroll and drag shapes return structured refusals. |+| Linux X11 | X11/EWMH, XTest, AT-SPI, and toolkit accessibility bridges | Supported with toolkit-specific limits. Foreground input and semantic background actions are broadly covered. Toolkits that reject synthetic background events receive an explicit refusal instead of a silent success. |+| Linux Wayland | AT-SPI plus compositor-specific discovery, capture, activation, and portal input | Supported with compositor-specific limits. Semantic background actions work where the application exposes them. Raw input cannot generally be sent to an arbitrary occluded surface. |++## Linux window systems++Linux support is recorded per window system and Wayland compositor. "Wayland"+is not one uniform automation API: compositors expose different discovery,+capture, activation, and input protocols.++| Environment | State | What is proven | Main limits |+| ------------------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| X11/Xorg | Supported | Window discovery and capture, AT-SPI trees and actions, foreground pointer and keyboard input, semantic background delivery, desktop scope, and structured refusals | Raw background delivery depends on the target toolkit. X11 accepting an event does not prove the application handled it. |+| Sway (wlroots reference lane) | Supported with limits | The complete typed Electron, Tauri, GTK, capture, and desktop-scope catalog, including foreground/background and AX/PX outcomes | Focus-bound and raw background input shapes without a target-addressed protocol return structured refusals. Other wlroots compositors are expected to share protocol support but are not yet proven. |+| GNOME/Mutter | Supported with limits | AT-SPI actions, GTK controls, compositor-backed window geometry and capture, verified foreground activation, and portal/libei foreground input | The bundled WinRects Shell helper and one Shell-session restart are prerequisites for authoritative geometry and activation. Portal video recording remains incomplete. |+| KDE/KWin | Experimental | Plasma 6 session startup, GTK AT-SPI discovery, generic discovery where exposed, and portal interface availability | cua-driver does not yet have a target-addressable KWin activation adapter, and no complete behavioral matrix is accepted. Focus-bound input refuses rather than risking delivery to the wrong application. |+| `cua-compositor` nested session | Experimental | Native GTK behavior, capture and scope, private route metadata, independent observation, and per-cell video | The complete shared renderer matrix is not accepted. Unicode text and a canonical parallel-drag row remain unproven; this route does not establish a stock-Wayland capability. |+| XWayland | Supported with limits | X11 routes are used when the application exposes a real X11 window; native Wayland and AT-SPI fallbacks cover mixed sessions | Capabilities depend on whether the application is actually using X11 or native Wayland. |++### Wayland background AX and PX++**Background AX works when the target exposes a semantic AT-SPI action.** For+example, the driver can invoke an accessible button without raising its window.+The passing test must also prove that focus and z-order did not change and that+input did not leak into the foreground application.++**Background PX is proven only for the action/surface cells declared by the+catalog.** A caller may address a target by pixel while the safe delivery route+hit-tests that point and invokes a semantic AT-SPI action. Renderer and gesture+cells without a safe target-addressed route return an exact refusal. Neither+outcome establishes arbitrary raw background injection.++**Arbitrary raw background PX is not available to an ordinary client on a+standard Wayland compositor.** Reconstructing a window's coordinate system+tells cua-driver where the target is, but portal/libei and virtual input still+deliver through the compositor's active seat. An occluding surface therefore+receives a raw event sent at that screen coordinate.++cua-driver also contains an opt-in exception: the nested `cua-compositor`+backend owns the compositor and can route `wl_pointer` and `wl_keyboard` events+directly to a selected client surface through `CUA_INJECT_SOCKET`, without+changing seat focus. This implements focus-free raw click, text, named-key, and+multi-pointer drag paths. The backend is covered by the converged typed matrix,+but remains experimental until the shared renderer matrix also passes. Unicode+text, multi-pointer behavior, canvas/game input, and any row without external+fixture evidence remain unproven.++<Callout type="warn">+ A structured `background_unavailable` or `background_occluded` result is part of the contract. It+ means cua-driver refused an unsafe or unsupported route before it could disturb the user's active+ desktop. It is not a silent success.+</Callout>++## Related documentation++- [How Cua Driver is validated](/concepts/how-cua-driver-is-validated) explains+ why unit tests and application-owned E2E evidence have different roles.+- [Platform roadmap](/reference/cua-driver/platform-roadmap) records the+ remaining engineering work, evidence gaps, and platform boundaries.+- [Known limits](/reference/cua-driver/limits) lists target-specific constraints+ and available alternatives.+- [Development](/reference/cua-driver/development) provides contributor entry+ points and canonical validation commands.diff --git a/docs/content/docs/reference/lume/cli-reference.mdx b/docs/content/docs/reference/lume/cli-reference.mdxindex 6e024248b2..917006d04a 100644--- a/docs/content/docs/reference/lume/cli-reference.mdx+++ b/docs/content/docs/reference/lume/cli-reference.mdx@@ -7,12 +7,12 @@ description: Command Line Interface reference for LumeAUTO-GENERATED FILE - DO NOT EDIT DIRECTLYGenerated by: npx tsx scripts/docs-generators/lume.tsSource: lume dump-docs --type cli- Version: 0.3.14+ Version: 0.3.15*/}A lightweight CLI and local API server to build, run and manage macOS VMs.-Documented against Lume **0.3.14**. Run `lume --version` for your installed version.+Documented against Lume **0.3.15**. Run `lume --version` for your installed version.For installation steps, see [Install Lume](/how-to-guides/lume/install-lume).diff --git a/docs/content/docs/reference/lume/http-api.mdx b/docs/content/docs/reference/lume/http-api.mdxindex e8e562a059..1f40540dbd 100644--- a/docs/content/docs/reference/lume/http-api.mdx+++ b/docs/content/docs/reference/lume/http-api.mdx@@ -7,14 +7,14 @@ description: HTTP API reference for Lume serverAUTO-GENERATED FILE - DO NOT EDIT DIRECTLYGenerated by: npx tsx scripts/docs-generators/lume.tsSource: lume dump-docs --type api- Version: 0.3.14+ Version: 0.3.15*/}import { Tabs, Tab } from 'fumadocs-ui/components/tabs';HTTP API for managing macOS and Linux virtual machines-Documented against Lume **0.3.14**. Run `lume --version` for your installed version.+Documented against Lume **0.3.15**. Run `lume --version` for your installed version.## Default URLdiff --git a/libs/cua-driver/README.md b/libs/cua-driver/README.mdindex 0c8aef4558..3fe6be4dfe 100644--- a/libs/cua-driver/README.md+++ b/libs/cua-driver/README.md@@ -18,6 +18,14 @@ Background computer-use driver for any agents. Speaks MCP over stdio; drives natStart with `rust/README.md`, `rust/crates/cua-driver/tests/README.md`, and`tests/fixtures/README.md` when changing driver behavior or tests.+Contributor documentation:++- `docs/test-matrix.md` maps unit and canonical harness E2E suites.+- `docs/action-support.md` is the empirical platform behavior ledger.+- `docs/test-harnesses-guide.md` explains fixture and runner ownership.+- `docs/linux-desktop-validation.md` covers representative Linux sessions.+- `docs/linux-support-completion-plan.md` preserves the historical Linux plan.+## Claude Code computer-use compatibilityStandard Claude Code MCP registration:diff --git a/libs/cua-driver/docs/action-support.md b/libs/cua-driver/docs/action-support.mdnew file mode 100644index 0000000000..394df7fb56--- /dev/null+++ b/libs/cua-driver/docs/action-support.md@@ -0,0 +1,89 @@+# Desktop action support++This ledger records the canonical Rust harness contracts for Windows, macOS,+and Linux. It is derived from typed `CaseSpec` rows and accepted E2E evidence,+not from a successful driver response alone.++- **Delivered** means a fixture-owned state change was observed.+- **Refused** means the exact structured refusal code and all required desktop+ side-effect oracles passed.+- **Gap** means unsupported or not yet proven. A missing row is never evidence+ that an action is impossible.++AX and PX describe how the target is selected. They do not require the same+delivery backend: a PX target may be hit-tested and delivered through AX/UIA+when that is the background-safe route.++## Accepted baselines++| Environment | Exact source | Accepted evidence |+| --- | --- | --- |+| Windows/Win32 | `6d7f02e4` | Run `29211506624`: 114/114 rows, 93 delivered and 21 exact refusals |+| macOS/Quartz | `448d052f` | Logged-in local matrix: 133/133 rows with owned per-cell MP4 evidence |+| Linux/X11 | `72083bb4` | Run `29213365684`: 108/108 rows, 71 delivered and 37 exact refusals |+| Linux/Sway | `72083bb4` | Run `29213349962`: 108/108 rows, 71 delivered and 37 exact refusals |++## Shared web harnesses++Foreground rows listed here are delivered for both AX and PX.++| OS | Harness | Background delivered | Background refused |+| --- | --- | --- | --- |+| Windows | Electron | left click and child window AX/PX | right click and double click AX/PX, plus drag PX: `background_occluded`; type text, press key, hotkey, and scroll AX/PX plus editor save AX: `background_unavailable`; `start_minimized` launch: `background_unavailable` when Windows denies the foreground lock, with no process spawned |+| Windows | Tauri | left/right/double click AX/PX, type text AX/PX, press key AX/PX, child window AX/PX, scroll AX, editor save AX | hotkey AX/PX and scroll PX: `background_unavailable`; drag PX: `background_occluded` |+| macOS | Electron | left/right/double click AX/PX, type text AX/PX, press key AX/PX, hotkey AX/PX, child window AX/PX, editor save AX | scroll AX/PX and drag PX: `background_unavailable` |+| macOS | Tauri | left/right/double click AX/PX, type text AX/PX, press key AX/PX, hotkey AX/PX, scroll AX/PX, child window AX/PX, editor save AX | drag PX: `background_unavailable` |+| macOS | WKWebView | left/right/double click AX/PX, type text AX/PX, press key AX/PX, hotkey AX/PX, scroll AX/PX, child window AX/PX, editor save AX | drag PX: `background_unavailable` |+| Linux X11 | Electron | Background AX left click and child-window actions deliver with strict focus, z-order, cursor, and input-leak oracles | Background PX left click, AX/PX right and double click, PX child window and drag, AX/PX keyboard and scroll, and AX editor save return exact `background_unavailable` |+| Linux X11 | Tauri | Background AX/PX left click and child-window actions deliver with strict focus, z-order, cursor, and input-leak oracles | Other background pointer, keyboard, scroll, drag, and editor-save shapes return exact `background_unavailable` |+| Linux Sway | Electron | The full 36-cell catalog has accepted fixture-owned evidence; safe semantic and compositor routes deliver | Focus-bound and raw background shapes without a target-addressed route return their declared exact refusal |+| Linux Sway | Tauri | The full 36-cell WebKitGTK catalog has accepted fixture-owned evidence, including live compositor geometry and roleless WebProcess targeting | Focus-bound and raw background shapes without a target-addressed route return their declared exact refusal |++All shared background rows require fixture state, focus, z-order, cursor, and+no-leaked-input evidence. Foreground rows require fixture state and retain a+video plus before/after turn evidence.++## Native Windows++| Harness | Proven contracts | Refusals and gaps |+| --- | --- | --- |+| WPF | Native AX controls; background combo selection, left click, and value changes; PX background left click through UIA hit-testing; foreground pointer gestures | Background F5 and PX drag are `background_unavailable`. Additional PX gesture rows remain unproven. |+| WinUI3 | Current UIA control, value, selection, popup, and slider rows | Background right/double-click refusal behavior and broader PX coverage remain unproven. |+| WebView2 | CDP page operations; native PX background left click through UIA hit-testing | Native keyboard and broader pointer cells remain unproven outside the shared Tauri/Electron hosts. |++`background_uipi_blocked` is a production refusal code with no canonical+elevated fixture today. It must not be counted as covered until a controlled+integrity-level harness can exercise it.++## Native macOS++| Harness | Proven contracts | Refusals and gaps |+| --- | --- | --- |+| AppKit | AX tree/capture; AX background left click, set value, and type text; PX background left/right/double click; PX foreground right/double click and slider drag; AX foreground/background scroll; desktop PX foreground left click | PX background slider drag returns exact `background_unavailable`. Native press key, hotkey, AX-addressed right/double click, and broader control combinations remain unproven. |+| SwiftUI | AX tree/capture; AX background left click and set value; foreground popover-trigger activation | The fixture proves `popover_open=true`, but the transient panel remains absent from targeted AX enumeration. Other native pointer and keyboard combinations remain unproven. |+| WKWebView | Full 36-cell shared-web catalog through the dedicated repo-local native host | PX background drag returns exact `background_unavailable`; the other 35 shared cells deliver. Native host-specific controls are outside this fixture. |++## Native Linux++Linux is recorded per display server and compositor because Wayland protocols+are capabilities, not one uniform API.++| Environment | Proven contracts | Refusals and gaps |+| --- | --- | --- |+| X11/Openbox | Exact run `29213365684` passed all 108 declared outcomes: 71 deliveries and 37 exact refusals. GTK3 AT-SPI actions and values, foreground XTest pointer/keyboard routes, capture, desktop scope, and the complete Electron/Tauri catalog all reached fixture-owned oracles. | Xvfb does not prove the real-Xorg MPX/uinput background pointer route. Toolkits that reject XSendEvent retain exact refusals. |+| Sway/wlroots | Exact run `29213349962` passed the same 108 outcomes as X11: 71 deliveries and 37 exact refusals. The complete Electron, Tauri, GTK3, capture, and desktop-scope catalogs use live Sway identity/geometry and external fixture oracles. | Stock Wayland cannot target raw focus-bound input at an occluded surface. Those rows retain exact refusals; another wlroots compositor is not assumed equivalent until a material divergence is reported and tested. |+| GNOME/Mutter | A real GNOME 46 Wayland run passed the full GTK3 matrix, 31/31. The WinRects helper supplies stable window ids, frame and buffer geometry, stacking, verified activation, stage capture, and the compositor cursor. AT-SPI handles semantic actions; persistent portal/libei sessions deliver foreground PX click, right/double click, drag, scroll, type, key, and hotkey. Background rows either deliver through AT-SPI with focus and leak guards or return the declared exact refusal. | The helper requires installation plus one Shell-session restart. Without it, target-bound foreground input refuses. Portal video and a shared Electron/Tauri GNOME run remain open. |+| KDE/KWin | A Plasma 6 session reached GTK AT-SPI discovery, generic toplevel discovery, and portal-interface preflight. Portal input is compiled into release binaries. | No behavioral matrix is accepted. A target-addressable KWin activation adapter is not implemented, so foreground portal/libei input refuses instead of injecting into the wrong focused app. |+| Nested `cua-compositor` | The optional backend owns its nested session and is covered by typed shared, native, and capture catalogs. Run `29197643541` passed native GTK3 31/31; run `29197887387` passed capture/scope 5/5. Full shared run `29199596935` passed 26/36 Electron rows before the occluded-subtree and wheel-frame repairs. | The lane remains experimental. The accepted full shared run still has 10 failures; focused replacement runs reduce that set but do not promote the environment. Unicode text and a canonical parallel-drag row remain unproven. This private route is not evidence of a stock-Wayland capability. |++PX background left-click rows may resolve the screen point to an actionable+AT-SPI node. Such a pass proves the public PX-addressed behavior and its desktop+side effects, but it does not prove raw pixel delivery to canvases or games.++## Maintenance rule++Update this document only when a typed row is added, removed, or changes+contract and an empirical run supports the change. Keep unsupported delivery+as an exact refusal where the driver can determine it before dispatch. When an+OS API reports success but offers no effect read-back, retain a visible gap+rather than inventing a fixture-specific refusal in production code.diff --git a/libs/cua-driver/docs/cross-platform-ci-test-plan.md b/libs/cua-driver/docs/cross-platform-ci-test-plan.mddeleted file mode 100644index 2dbc1420f8..0000000000--- a/libs/cua-driver/docs/cross-platform-ci-test-plan.md+++ /dev/null@@ -1,286 +0,0 @@-# Cross-Platform CI and E2E Test Plan--Status: implementation plan, with the source-built unit and manual runner-foundation now landed on the working branch--## Goal--Make the Rust harnesses the source of truth for user behavior on Linux and-Windows. Keep unit tests cheap and OS-scoped. Run desktop end-to-end tests as a-maintainer-controlled gate with the real session each platform requires.--The same Rust scenario should run on both platforms whenever the behavior is-supported. The runner may use different platform APIs, but the scenario ID,-application oracle, and outcome rules should stay the same.--## Test Tiers--### Tier 1: Rust unit and compile checks--Run automatically on pull requests when an affected platform or shared Rust-path changes.--- No desktop session, TCC, RDP, AT-SPI, or Azure credentials.-- Run on `ubuntu-latest` and `windows-latest` in separate jobs.-- Use `cargo test --workspace --all-targets --locked`.-- Run Clippy in the same OS-scoped workflow where its cost remains acceptable.--Shared code changes run both OS jobs. A platform-only change runs one job.--### Tier 2: Fast Linux integration checks--Keep the small Xvfb and AT-SPI checks that do not require a real user desktop.-These checks can remain automatic if their runtime stays short and stable.--They cover protocol setup, capture modes, accessibility reachability, and-desktop-scope contract errors. They do not replace the behavioral harness.--### Tier 3: Maintainer e2e gate--Run the same Rust behavioral matrix against real harness applications.--- Linux: the manual Rust runner for the canonical matrix; Nix X11 and Nix- Wayland remain supporting compositor checks until the source-built app pilot- is complete.-- Windows: Azure Windows with an active RDP user session.-- Trigger: `workflow_dispatch` with a commit ref, PR number, and suite.-- Optional nightly and post-merge runs can catch drift without slowing every PR.-- The VM or runner never pushes code.--### Tier 4: Supporting compatibility checks--Keep toolkit and distro checks that cover behavior outside the canonical harness-matrix. These checks should be labeled as supporting coverage and should not-claim that a user workflow passed when they only prove window discovery or a-non-error response.--## Canonical Behavioral Matrix--The Rust test suite owns the scenario list. CI workflows should select a suite,-not duplicate the scenario definitions in YAML or Python.--### Applications--- Electron-- Tauri--### Surfaces--- Accessibility action path-- Pixel-coordinate path--### Delivery modes--- Background-- Foreground--### Scenarios--- Calculator click sequence and result oracle-- Editor type, save, and saved-state oracle-- Enter key and hotkey state oracles-- Nested scroll and scroll-offset oracle-- Child-window creation oracle-- Drag and drop-state oracle-- Focus sentinel and window posture checks--Each result has one of these outcomes:--```text-pass-expected_refusal-known_gap-not_applicable-unexpected_failure-```--An expected refusal must include the structured driver error and capability-reason. A driver response without an external application-state change is not a-pass.--The runner should emit a common result record:--```json-{- "scenario": "editor_type_and_save",- "os": "windows",- "app": "electron",- "surface": "ax",- "delivery": "background",- "outcome": "pass",- "oracle": "editor_status=saved:exact"-}-```--Platform-specific applications such as WPF and WinUI 3 remain focused Windows-coverage. GTK and Qt remain focused Linux coverage. They supplement the shared-matrix and do not define cross-platform acceptance.--## Linux Structure--Nix remains the Linux desktop test environment. It should build the driver and-repo-local harness applications from source and run the Rust tests against those-outputs. The test should not download a prebuilt application and Linux e2e runs-do not need GIF recording.--Proposed layout:--```text-nix/cua-driver/- package.nix- module.nix- README.md- lib/- app-catalog.nix- artifacts.nix- atspi-session.nix- mcp-client.nix- wayland-session.nix- x11-session.nix- checks/- x11/- harness-electron.nix- harness-tauri.nix- input-cursor-click.nix- input-parallel-drag.nix- service-config-persistence.nix- service-integration.nix- wayland/- app-background-read.nix- input-background-terminal.nix- input-cursor-click.nix- input-parallel-drag.nix- service-integration.nix-```--The current generic `linux-background-gui.nix` should be split. Its read-only-app entries should become supporting checks or be removed after equivalent-canonical harness coverage exists. Its Chromium and Tk write paths should be-retained until the Rust harness covers the same contract.--Use explicit flake check names such as:--```text-cua-driver-linux-x11-harness-electron-cua-driver-linux-x11-harness-tauri-cua-driver-linux-x11-service-config-persistence-cua-driver-linux-wayland-gnome-input-background-terminal-```--Do not put `gif` in a test name. Screenshots, AX trees, structured results, and-driver logs are enough for routine artifacts. Capture a PNG on failure or when-diagnosing a pixel scenario.--The testkit needs test-only path overrides so Nix can use immutable build-outputs:--```text-CUA_TEST_DRIVER_BIN-CUA_TEST_APPS_ROOT-CUA_TEST_WORKSPACE_ROOT-```--Local runs keep their current defaults.--## Windows Structure--Windows uses the same Rust scenarios and repo-local harness source. The Azure VM-builds the applications on demand inside the active RDP session.--Workflow and helper names:--```text-.github/workflows/ci-rust-windows.yml-.github/workflows/e2e-rust-windows.yml-scripts/ci/windows/- verify-user-session.ps1- build-harnesses.ps1- run-rust-e2e.ps1- collect-artifacts.ps1-```--The Windows e2e workflow now:--1. Check out or sync the requested commit.-2. Confirm that the target process runs in the active RDP user session, not- Session 0.-3. Build Electron, Tauri, WPF, and WinUI 3 harnesses from source.-4. Build the Rust driver and integration tests.-5. Run the shared behavioral matrix, then Windows-native tests.-6. Collect structured results, UIA trees, screenshots, and driver logs.-7. Leave the VM unable to push or modify the source branch.--The Windows unit workflow runs on `windows-latest` and does not attempt these-desktop steps.--## Workflow Names--Use the distinction between `ci` and `e2e` in the filename and displayed job-name:--```text-CI: Rust Linux unit-CI: Rust Windows unit-CI: Linux fast integration-E2E: Linux Nix X11-E2E: Linux Nix Wayland-E2E: Linux Azure desktop-E2E: Windows Azure RDP-```--The Nix X11 and Wayland workflows should stop running automatically for every-PR update once the maintainer gate is available. Keep manual dispatch, nightly-runs, and post-merge regression runs.--## Legacy Test Migration--Keep an inventory before deleting tests. For each test record its scenario,-oracle, OS, window system, app, and current CI role.--Classify each test as:--- `canonical`: a trusted user-behavior acceptance test.-- `supporting`: useful backend or toolkit coverage.-- `diagnostic`: useful during investigation but not a product gate.-- `retire`: redundant or weaker than a canonical replacement.--Migration order:--1. Add the testkit path overrides and strict fixture preflight.-2. Land the OS-scoped unit checks and the manual desktop runners.-3. Pilot source-built harness packaging inside Nix before moving app builds- into the Nix sandbox.-4. Add missing scenario coverage before removing an old test.-5. Move weaker tests out of required CI.-6. Delete tests only after the replacement has passed repeatedly on the target- platform.-7. Update the Nix and workflow READMEs to describe the new ownership.--Do not delete specialized Wayland, MPX, WPF, or WinUI coverage merely because-the shared Electron/Tauri matrix exists. Retain those tests until the shared-matrix covers the behavior they uniquely exercise.--## Acceptance Criteria--- Shared scenario IDs and external oracles are identical on Linux and Windows.-- Shared Rust changes run both unit-test jobs.-- Platform-only changes run only the affected OS unit job.-- Linux e2e builds harness applications from source in the Linux desktop runner;- a fully sandboxed Nix app-build pilot remains a separate follow-up.-- Windows e2e builds harness applications from source in the Azure VM.-- No Linux e2e test requires GIF recording.-- No canonical test passes on driver success alone.-- Unsupported background behavior produces a structured refusal or remains a- visible known gap.-- Maintainer e2e runs publish a result table and failure artifacts linked to the- PR.-- Legacy tests are either classified, upgraded, moved out of required CI, or- removed with an equivalent canonical replacement.--## Non-Goals--- Do not make Windows use Nix.-- Do not create a second Python behavioral framework.-- Do not download opaque prebuilt desktop fixtures.-- Do not weaken external-state assertions to make a matrix green.-- Do not make Azure or RDP credentials available to untrusted PR code.diff --git a/libs/cua-driver/docs/e2e-ci-reporting.md b/libs/cua-driver/docs/e2e-ci-reporting.mdindex f460e4affd..dc395962db 100644--- a/libs/cua-driver/docs/e2e-ci-reporting.md+++ b/libs/cua-driver/docs/e2e-ci-reporting.md@@ -1,67 +1,196 @@# Rust E2E CI Reporting-The Rust desktop harness is the behavioral source of truth. GitHub Actions only-selects the lane and publishes its results.+Rust owns behavioral case declarations, observations, result classification,+and report validation. OS runners build the environment, execute Rust targets,+and upload the validated artifacts.-## Workflow UI+## Canonical Invocation-The Linux and Windows manual workflows split the expensive suites into-independent jobs. On Windows, `suite=all` covers the default Rust tests, UX-guards, shared Electron/Tauri behavior, WPF/WinUI3/WebView2 harnesses, and-modality-input E2E. The workflow summary job collects their artifacts and-writes one Markdown table to the GitHub Actions run summary.+The contributor-facing invocation runs the complete matrix on every OS and+takes no suite selector:-This means a failure in the shared Electron/Tauri lane does not prevent the-native or modality lane from running. The workflow still fails when a required-lane fails.+```text+Windows: scripts/ci/windows/run-rust-e2e.ps1 -RequireGui+Linux: scripts/ci/linux/run-rust-e2e.sh+macOS: scripts/ci/macos/run-rust-e2e.sh+```-## Result files+Workflows set a private lane value to fan that matrix into shared, native, and+capture jobs so one failure does not hide another lane. These are execution+partitions, not public suite choices or separate behavioral sources of truth.++## Execution Order++Each runner follows the same sequence:++1. Resolve one immutable source SHA and check out that SHA in every lane.+2. Build the Rust driver and required repo-local fixtures.+3. Run `e2e_environment_preflight_test` once; it verifies `git rev-parse HEAD`+ against the workflow's resolved SHA.+4. Abort before behavioral cells if the desktop, permissions, fixture, capture,+ or video lifecycle is unavailable.+5. Run the selected Rust integration targets.+6. Validate declarations, results, and evidence with `cua-e2e-report`.+7. Upload the report, logs, and recordings.++The preflight prevents a single TCC or desktop-session problem from appearing+as the same failure on every behavioral cell.++## Artifact Files++Each OS lane writes:++```text+artifacts/cua-driver/<platform>/+|-- environment.jsonl+|-- cases.jsonl+|-- results.jsonl+|-- summary.md+|-- environment-preflight.log+|-- <rust-target>.log+`-- recordings/<cell-label>-pid<process>-<sequence>/+ |-- recording.mp4+ |-- trajectory.json+ |-- session.json+ |-- cursor.jsonl+ `-- turn-*/+ |-- action.json+ |-- evidence.json+ |-- before_state.json+ |-- before.png+ |-- after_state.json+ |-- after.png+ |-- app_state.json+ |-- screenshot.png+ `-- click.png (click-family turns)+```-When `CUA_E2E_RESULTS_FILE` is set, Rust shared behavior tests append one JSON-object per host and scenario:+`cases.jsonl` is the executed catalog. `results.jsonl` contains one result for+every declared cell. In canonical mode, the reporter rejects duplicates,+missing results, undeclared results, contradictory statuses, missing required+videos, and missing or invalid expected turn evidence. An `evidence.json`+classification normally makes capture failure auditable without making it+pass. The one narrow exception is a successful Windows `bring_to_front`+restoration:+the before image may be `target_minimized`, and the after image may be+`capture_failed`, only when the action reports success, after-state+accessibility data was captured, and the required trajectory and finalized MP4+remain present. Any+other missing image still fails closed.++The testkit prepares each cell's artifact directory before fixture setup but+does not start `recording.mp4` yet. On Windows GitHub-hosted runners it obtains+the test process's inherited top-level console with `GetConsoleWindow`, verifies+the HostedComputeAgent/runner title and console class, then minimizes it with+`ShowWindow(SW_MINIMIZE)`. Fixture readiness+and the required foreground or sentinel-occluded posture are established after+that cleanup. The cell then calls `start_behavior_recording`, re-establishes+the sentinel posture after the capture backend attaches, and clears setup-only+focus events. It dispatches the action only after this observation boundary,+then keeps recording through oracle collection.++`trajectory.json` records the behavioral video phase as `pending`, `started`,+or `finalized`, its start/baseline/finalization timestamps, and the hosted-runner+console cleanup status. Strict reporting+requires `finalized`. A setup or posture failure before the boundary remains in+the test result and logs, and the testkit writes `recording-error.txt`; it does+not create a misleading behavioral clip.++## Environment Record++The preflight emits exactly one record:```json{- "schema": "cua-e2e-result/v1",+ "schema": "cua-e2e-environment/v2",+ "platform": "macos",+ "display_server": "quartz",+ "source_sha": "0123456789abcdef0123456789abcdef01234567",+ "status": "ready",+ "duration_ms": 912,+ "message": ""+}+```++An error record produces an environment section in `summary.md` and aborts the+lane before case declarations are executed. The same source SHA appears in the+successful typed summary.++## Case And Result Contract++A case declaration uses `cua-e2e-case/v2`. A result flattens the same contract+fields into `cua-e2e-result/v2` and adds the observation:++```json+{+ "schema": "cua-e2e-result/v2",+ "cell_id": "windows-tauri-left-click-ax-background","platform": "windows",- "host": "tauri",- "scenario": "keyboard",- "status": "FAIL",- "message": "application state did not reach key_state=enter",- "duration_ms": 24360+ "display_server": "win32",+ "harness": "tauri",+ "toolkit": "platform-webview",+ "action": "left_click",+ "targeting": "ax",+ "delivery": "background",+ "scope": "window",+ "driver_route": "uia_invoke",+ "expected_behavior": { "kind": "deliver" },+ "oracles": ["fixture_state", "focus", "z_order", "no_leaked_input", "cursor"],+ "test_status": "pass",+ "observed_behavior": "delivered",+ "passed_oracles": ["fixture_state", "focus", "z_order", "no_leaked_input", "cursor"],+ "duration_ms": 1482,+ "message": "",+ "evidence": {+ "video": "recordings/windows-tauri-left-click-ax-background-pid123-001/recording.mp4",+ "trajectory": "recordings/windows-tauri-left-click-ax-background-pid123-001/trajectory.json"+ }}```-When `CUA_E2E_SUMMARY_FILE` is set, the same test writes a Markdown row for the-GitHub summary. The CI scripts also parse each Cargo `test ... ok/FAILED` line,-so native, guard, default, and modality suites appear as individual test rows-as well as lane-level rows. Full logs are retained alongside them.+`targeting` describes how the action identifies its target: `ax`, `px`,+`page`, or `not_applicable`. It is separate from screenshot/tree capture.-Statuses are deliberately explicit:+## Classification Rules-- `PASS`: the external application-state oracle passed.-- `FAIL`: the driver response or external oracle failed.-- `SKIP`: the fixture was unavailable and was not required by the invocation.+`test_status` answers whether the cell met its declared contract.+`observed_behavior` records what the driver did.-The CI scripts set `CUA_TEST_REQUIRE_FIXTURES=1`, so missing required fixtures-are reported as failures rather than silently becoming skips.+| Expected | Observed | Required oracles | Status |+| --- | --- | --- | --- |+| Deliver | Delivered | Passed | Pass |+| Deliver | Refused | Any | Fail |+| Deliver | No effect or error | Any | Fail |+| Refuse | Allowed structured refusal | Passed | Pass |+| Refuse | Different refusal code | Any | Fail |+| Refuse | Delivered | Any | Fail pending contract review |-## Running locally+The structured refusal enum currently recognizes+`background_unavailable`, `background_occluded`, and+`background_uipi_blocked`. Each refusal case declares the exact subset allowed+by its controlled setup. String-prefix or message-substring matching is not+accepted.-Windows:+A refusal contract also requires focus, z-order, and leaked-input observations.+An honest refusal does not satisfy a case whose contract requires delivery.-```powershell-.\scripts\ci\windows\run-rust-e2e.ps1 -Suite shared -RequireGui-.\scripts\ci\windows\run-rust-e2e.ps1 -Suite native -RequireGui-```+## Summary Ownership-Linux:+`cua-e2e-report` is the only behavioral Markdown renderer. Shell and+PowerShell runners must not parse `test ... ok` output or create `host=cargo`+and `host=lane` rows. Cargo logs remain available for unit-test annotations and+lane diagnostics, outside the behavioral population.-```bash-xvfb-run -a --server-args="-screen 0 1920x1080x24" \- dbus-run-session -- bash -lc \- "scripts/ci/linux/run-rust-e2e.sh --suite shared"-```+The summary reports delivered passes, refused passes, failures, and skips+separately. Shared and native targets emit the same typed records; a compile or+runner failure that prevents declaration remains visible as a failed job and+cannot be mistaken for a green behavioral row.++## Evidence Links-The generated files are under `artifacts/cua-driver/<platform>/` and should-not be committed.+Every canonical behavioral cell records a validated MP4 and trajectory. The+recording directory also contains its session metadata and turn-level action+and screenshot files; Cargo target logs remain lane-level diagnostics. GitHub+cannot deep-link to a file inside a multi-cell artifact archive, so each summary+row displays the exact MP4 path as a link to the owning lane archive while the+trajectory rollup provides bulk-download links and video counts.diff --git a/libs/cua-driver/docs/e2e-convergence-journal.md b/libs/cua-driver/docs/e2e-convergence-journal.mdnew file mode 100644index 0000000000..b98ecbc06a--- /dev/null+++ b/libs/cua-driver/docs/e2e-convergence-journal.md@@ -0,0 +1,167 @@+# E2E Convergence Journal++**Historical record.** Use `action-support.md` and the hosted platform support+reference for current support claims.++This journal records changes and test evidence for the Rust desktop harness+convergence work. It omits machine names, credentials, and partner names.++## 2026-07-12++| Commit | Change | Verification or finding |+| --- | --- | --- |+| `8296b82b` to `16fb314a` | Reused stable Sway origins, normalized occluded zero-geometry fallbacks, and retained renderer identities across accessibility and compositor queries | Shared run `29199656600` passed 29/36 Electron rows, up from 23/36. Focused run `29200827296` passed all 9 Electron PX-background rows at the replacement source, leaving foreground AX right/double click and foreground AX/PX hotkey open. All 9 focused Tauri rows remained explicitly environment-limited by the missing DRM/EGL renderer tree. |+| `88d700f4` to `a78f525a` | Made nested pointer hit-testing target-subtree-local, reused the entered child pointer resource, fell back to the owning root resource, and completed wheel frames | Full shared run `29199596935` passed 26/36 Electron rows before these focused repairs. Run `29200462634` then passed 7/9 occluded PX background rows, including raw right click, text, key, hotkey, child-window click, and drag. Dedicated run `29200854535` failed both scroll rows, so renderer scroll, two click rows, and one AX right-click timeout keep the shared environment experimental. |+| `8937209c` to `edb64a25` | Executed the Linux support-completion plan: reconciled stale fixture docs, added compositor/input-route metadata, packaged the nested compositor, restored typed shared/native/capture lanes, pinned VM source identity, and formalized Linux release validation | Automatic Linux unit, distribution, script, docs, and source-package gates remained green. X11 run `29194456173` retained all 108 accepted outcomes on the exact source. |+| `d11c026e` to `676b9ef4` | Replaced fire-and-forget nested injection with a versioned acknowledged protocol, PID targeting, exact resource errors, pointer/keyboard/scroll/hotkey/drag verbs, logical focus routing, and stale-surface cleanup | Early typed runs deliberately failed on ambiguous targets, missing logical seat focus, and stale surface resources. Each failure became a driver/compositor fix; unsupported keys and missing resources now refuse instead of returning false success. |+| `6ab4e930` to `0b0b0e2c` | Stabilized nested client identity and coordinate spaces across AT-SPI, screencopy, scene hit-testing, desktop scope, and private surface input | Run `29197643541` passed the native GTK3 matrix 31/31. Run `29197887387` passed capture/scope 5/5, including the previously failing screen-absolute GTK click and exact `LinuxCuaCompositorInject` route metadata. The shared renderer result remains recorded separately. |+| `1280021b` to `1fe2ae35` | Reconstructed Sway renderer geometry through stable compositor identity, cropped window capture by that identity, and connected renderer AT-SPI frames to the same window origin | Run `29195551765` retained native GTK3 31/31 and capture/scope 5/5. Shared run `29196452551` passed 23 Electron rows, exposed 13 renderer-coordinate/hotkey gaps without losing the 36 declared Tauri environment failures, and supplied the focused replacement target. |+| Representative GNOME/KDE audit | Reused the unchanged desktop entrypoint against real compositor sessions | GNOME's accepted 31 native GTK, three capture, and two desktop-scope contracts remain valid. Plasma 6 reached session startup, GTK AT-SPI discovery, and portal-interface preflight, but no KDE behavioral row was accepted because target-addressable activation and interactive portal consent remain absent. Real-Xorg MPX and DRM/EGL WebKitGTK also remain explicit environment gaps. |+| `9736e19b` to `ff1071af` | Removed nested AT-SPI runtimes, repaired Wayland trajectory target resolution, preserved 64-bit action identities, and routed foreground renderer typing through real keyboard events | Focused Sway run `29180903640` proved foreground/background Electron AX clicks delivered after the runtime repair. Run `29181552459` proved complete before/after AX state, screenshots, and click markers after the recorder fix. Run `29182130646` then passed Electron foreground AX type text with fixture-state evidence. AX hotkey now reaches the target but has no DOM effect, and PX hotkey still exposes a hosted-Sway renderer-origin gap; both remain explicit failures. |+| `f8e4b37b` and `74a29a20` | Completed the Linux X11, Sway, and GNOME Wayland convergence pass and published WinRects helper metadata v3 | Exact X11 run `29179817332` passed all 108 rows: 71 delivered and 37 exact refusals. The split Sway run proved all 31 native GTK3 rows and all five capture/scope rows with playable per-cell videos. A real GNOME 46 session passed 31 GTK3 rows, three capture contracts, and two desktop-scope contracts. |+| `346f6f73` | Added the direct xkbcommon link dependency to the Nix release package | The final source gate exposed `-lxkbcommon` after the portal/libei feature became part of the release build. `libxkbcommon` now belongs to the package's build inputs rather than only the interactive Wayland shell. |+| `fabf4a20` | Stabilized GNOME Wayland capture and foreground input through the WinRects helper and portal/libei | A real GNOME 46 session passed all 31 GTK3 rows. The fix separates Mutter frame and surface-buffer origins, waits for the complete EIS device announcement, selects the newest resumed pointer, uses monotonic frame timestamps, restores target activation after consent, and persists the grant until the user revokes it. AX background rows passed with the target fully occluded; unsupported background PX gestures returned their declared exact refusals. |+| `79e5ac2a` to `8161cf56` | Repaired Linux fixture scrolling, Wayland event flushing, accessibility enablement, metadata, portal-input feature packaging, and Linux CI compilation | Exact X11 run `29173767528` passed the expanded GTK3 matrix, while the prior complete X11 run `29148643166` passed all 80 declared outcomes. The portable release build now includes GNOME/KDE RemoteDesktop/libei input without requiring the newer PipeWire capture dependency. |+| `8719e292` | Made native Wayland pointer coordinates, scroll-at-point, keyboard priming, libei key sequences, and cropped window capture target-aware | Sway GTK3 run `29175153270` passed all 23 rows: 17 delivered outcomes and 6 exact background refusals, with no failures or skips. |+| `9c65c153` and `7b985276` | Added native GNOME observation, targeted Chromium accessibility, generic `ext_foreign_toplevel_list_v1`, stable identity enrichment, and Chromium frame-origin reconstruction | Linux unit and distro compatibility checks passed. A controlled GNOME probe showed driver-launched Electron in both the Shell window list and AT-SPI without globally enabling the screen reader. |+| `c9c82908` | Split pure-Wayland shared, native, and capture owners onto independent clean runners | Run `29175548524` no longer exhausted one runner disk. It isolated one Sway hotkey failure, one obsolete standalone-activation failure, and the hosted WebKitGTK renderer limitation instead of hiding later lanes. |+| `a07ba769` | Routed hybrid GNOME/KDE sessions through native Wayland, enumerated GNOME windows from authoritative Shell metadata, translated every pointer gesture from window-local pixels, and required verified target activation before portal/libei input | Clean Linux unit and six-distro compatibility runs passed at the exact SHA. GNOME now has a verified Shell activation adapter; unsupported KDE foreground injection fails closed rather than acting on the user's currently focused application. Exact X11 and split Wayland E2E reruns are active. |++Native GNOME validation also exposed a verification-host issue: Nix `path:`+inputs copy untracked `target/` directories when a synced VM checkout has no+Git metadata. Stale source snapshots consumed the VM disk and were removed;+this is not a driver failure. Clean GitHub runners are unaffected.++## 2026-07-11++| Commit | Change | Verification or finding |+| --- | --- | --- |+| `7623f7be` to `55307b38` | Added canonical native WebView2 and WPF PX background single-left-click rows with fixture-owned state and foreground geometry probes | An active Windows RDP replay passed both rows through UIA Invoke. Each result proved FixtureState, Focus, ZOrder, NoLeakedInput, and Cursor and produced complete before/after evidence plus video. WebView2 derives its real DOM screen point through CDP, proves it with a foreground PX click, then dispatches the occluded PX action. WPF reads an atomic JSON state file written by the fixture. All 31 testkit tests, strict testkit Clippy, and host compile checks passed. |+| `6aaa914b` | Restored direct Chromium/Electron single-left background clicks through `PostMessage` for both AX-resolved and PX-resolved targets | In an active Azure RDP session, raw UIA delivered the click but raised Electron and displaced the occlusion sentinel. A VM-only `PostMessage` diagnostic then delivered both AX and PX cells in 3/3 seeded runs while FixtureState, Focus, ZOrder, NoLeakedInput, and Cursor all passed. The production build passed all 12 Electron left/right/double-click rows; only the two single-left background rows changed from refusal to delivery. |+| `768f0650` to `6e05424a` | Completed the macOS convergence pass and bound local evidence to a clean exact source SHA | The full local matrix passed 83/83 contractual outcomes: 79 deliveries and 4 exact refusals. Fixes cover PX background hit-testing, foreground typing's first character, structured scroll/drag refusals, desktop HID scope, AppKit fixture stability, and per-connection daemon sessions. |+| `d39c71c9` | Made the Windows sentinel physically acquire foreground before each background action | Hosted runs no longer fail during sentinel posture. WinUI3, WebView2, WPF background actions, and the shared catalog all reach their behavior boundary with the original focus, z-order, cursor, and leaked-input checks intact. |+| `0c2331b4` to `3f6fbd53` | Completed foreground PX activation, removed volatile UIA metadata from the WPF drag-refusal oracle, and aligned WPF combo selection with the background UIA recipe | The replay passed 19/20 WPF rows, all WinUI3 and WebView2 rows, and all capture/scope rows; its only native failure was the separately owned minimized-launch contract. Electron child-window delivery and keyboard refusal codes were updated from empirical side-effect-clean results. |+| `00a29c49` and `e43c2946` | Tested `CreateProcessW` startup show-state control, then made minimized launch fail closed when Windows cannot guarantee no activation | Electron can override `STARTF_USESHOWWINDOW`, so startup hints alone are insufficient. The driver now returns exact `background_unavailable` before spawning when the foreground lock is denied; the canonical refusal proves no new window and no desktop side effect. |+| `8d28beec` | Added the native WKWebView shared catalog, fixed AppKit AX scroll, separated SwiftUI popover activation from transient AX discovery, and expanded native AppKit/SwiftUI action rows | The exact local run passed all 130 rows: 124 deliveries and 6 structured refusals. WKWebView passed 35 deliveries plus its background-drag refusal. AppKit foreground/background AX scroll, foreground/background PX right and double click, foreground slider drag, SwiftUI background click/value, and SwiftUI foreground popover activation passed fixture-state checks. AppKit background drag was a silent no-op, so the driver now returns exact `background_unavailable`; the refusal row proves unchanged fixture state, focus, z-order, cursor, and input journal. |++## 2026-07-10++| Commit | Change | Verification or finding |+| --- | --- | --- |+| `0fda31bf` | Added typed case declarations, results, and lane preflight | Testkit unit tests passed locally. The first Windows and Linux runs exposed fixture and recording failures before behavioral classification. |+| `f2e5804c` | Added full result dimensions to the reporter | Reporter tests reject changed contracts and missing rows. |+| `6fa9311e` | Removed the synthetic calculator fixture | The canonical shared target now uses only controls in the repo-local web harness. |+| `891d4c94` | Added the platform-independent desktop observer core and Windows backend | Unit tests prove that transient focus changes, z-order changes, cursor movement, and input leaks fail independently. |+| `3a48ba0d` | Made the Linux desktop preflight reject blank recordings | Run `29131453240` stopped with `94/2073600` non-dark pixels. This was an environment failure, so it produced no behavioral verdicts. |+| `28ce18b6` | Added fail-closed cell filtering | An unmatched filter now fails instead of reporting a green zero-cell run. |+| `303c7bf3` to `056728ec` | Added the occluding Electron sentinel and staged its preload | A local sentinel smoke test wrote direct ready and focus events to its JSONL journal. |+| `640fcc9a` | Replaced the default route label with explicit backend routes | Unit coverage keeps Windows Chromium PX background delivery on `WindowsTargetedInjection`; it remains a required capability. |+| `580708f7` | Added the direct macOS observer | Fifteen testkit tests and strict testkit Clippy passed locally. A native snapshot smoke test read the foreground app and real cursor. |+| `00a1d31f` | Added the direct X11 observer and attached desktop checks to background shared cells | Independent review found and corrected XWayland, reparenting, raw input-focus, and occlusion traps. Sixteen testkit tests passed locally. |+| `bcc53fe7` | Expanded the shared catalog from a 19-cell diagonal to 32 cells per host | Route tests cover every declared Win32, Quartz, X11, and Wayland combination. |+| `21f3f6e0` | Fixed workflow dispatch checkout semantics | Earlier branch dispatches had selected the branch workflow but checked out the default `inputs.ref=main`. Their behavioral results were discarded. |+| `69c7c2c1` | Made each OS runner select only the typed shared catalog | Legacy tests remain in source until their assertions pass the deletion ledger, but they no longer define the canonical shared run. |+| `6733efe9` to `b52d239f` | Removed duplicate Windows modality-input probes, normalized desktop-scope ownership, and corrected Electron's X11 show path | Local testkit checks stayed green. Linux run `29133849724` still rejected a blank root-desktop recording, so it produced no behavioral verdict. |+| `c3454d1b` to `e54088f3` | Shared the Electron foreground sentinel and attached its Focus, ZOrder, Cursor, and NoLeakedInput contract to WPF background actions | Windows run `29134124533` passed all 18 WPF tests. Its lane failed later on an independent WebView2 CDP-readiness race. |+| `48afc293` to `842f3c86` | Moved background capture, minimized launch, and agent-cursor behavior into their action-owned tests | The replacements assert application or pixel state plus the desktop observer instead of a focus-only guard. |+| `c49dd59d` | Rejected Chromium UIA Invoke's false-positive and tested coordinate injection as a replacement route | Focused run `29134320325` proved the route still refused a fully occluded target. That route was not accepted as a fix. |+| `521508aa` to `99897fed` | Added the reusable typed executor and typed WPF, launch, capture, and cursor records | Sixteen testkit tests and host compile checks passed. Fresh native and capture runs are validating the Windows-only bodies. |+| `ad891b48` | Made CDP page discovery wait for the first usable target | The earlier native run's only failure was WebView2 exposing a listener before `/json` contained a page. The DOM-state assertion remains unchanged. |+| `10a1a749` | Preserved the old guard's anti-overclaim check in canonical background text/key rows and aligned action-owned video labels | An independent deletion audit identified protocol honesty as the only unique old-guard assertion. Chromium AX background click now probes a target-bound LegacyIAccessible action rather than a coordinate route. |+| `5bca25be` | Mapped the normal Electron harness at construction | Linux's deferred BrowserWindow was enumerable and capturable but absent from the X11 root recording. A focused preflight run is validating this lifecycle correction. |+| `c83c99ac` | Limited each CI lane to the fixtures it consumes | Windows and Linux focused runs built only their internal shared, native, or capture partition while the contributor command remained selector-free. |+| `8e41859f` | Published WebView2 readiness after DOM navigation, gave typed WPF rows exact recording labels, removed the duplicate modality-background target, and classified strict Chromium AX background click as a refusal | Run `29135420638` passed all 18 WPF, 7 WinUI3, and 3 WebView2 tests plus all four typed WPF background rows. Run `29135420139` returned the exact `background_occluded` refusal with all desktop-side-effect oracles passing. |+| `0a5352c5` | Reduced workflow dispatch to one canonical matrix and added row-level evidence links | Shared, native, and capture remain internal parallel jobs. The link renderer converts each exact MP4 path into a link to its owning lane artifact. |+| `ac1f19cc` | Added scoped Windows foreground locking and X11 alpha-shaped cursor-overlay bounds | A placement-only Linux rerun (`29135421864`) retained the exact blank-root signature, confirming placement was not the cause. The X11 fix addresses the full-screen ARGB overlay on non-composited sessions; Windows now prevents activation before launch instead of restoring it afterward. |+| `c190228e` | Tightened the controlled Chromium refusal declaration | The fully occluded test setup allows only `background_occluded`; generic or message-matched refusals remain rejected. |+| `a9953533` to `2c960eaf` | Corrected Windows foreground dispatch and made unsupported background routes report structured refusals instead of silent success | Run `29137293235` passed every shared foreground row. Its remaining background failures were used to define exact Chromium and embedded-WebView capability boundaries rather than weakening fixture-state assertions. |+| `f50b7170` to `64377935` | Required real fixture launch, split keyboard actions into independent cells, and targeted visible native controls | The shared catalog is now 36 cells per host. GTK3 and Linux desktop-scope lanes passed in run `29137481789`; Windows fixture geometry failures from run `29137143798` were removed without hard-coded replacement coordinates. |+| `6d973d2c` and `309740a6` | Awaited the first minimized-launch sweep and routed explicit Windows expand requests through UIA ExpandCollapse | Run `29137992025` caught a Windows-only compile error in the new pattern route before any behavioral verdict; `465f4a2f` unwraps the pattern result and the replacement run compiles. |+| `b284b521` | Exposed complete Linux renderer targets and removed the last Python accessibility write path | Shared lookup now searches the raw accessible name, fixture-state assertions poll fresh trees, AT-SPI text writes stay native Rust, and the silent 150-node bounds cutoff is gone while the 20-second budget remains. |+| `c473e045` | Reduced each OS to one contributor-facing E2E command | CI partitions remain internal through `CUA_E2E_INTERNAL_LANE`; `shared`, `native`, and `capture` are no longer public runner selectors. |+| `a86f8208` to `94a0509e` | Allocated CDP ports per process and replaced Windows page-test sleeps with DOM polling | WebView2 and Electron page owners no longer share ports `9222/9223` or assume fixed startup/action delays. |+| `cd922be9` | Preserved the macOS signing identity during install-local | The installer now signs and verifies a staged bundle, refuses to downgrade a certificate-signed live app to ad-hoc, and keeps a rollback copy until installation succeeds. |+| `d835c15e` and `e820aaf5` | Rejected known Windows Tk keyboard drops and activated desktop-scope click targets before SendInput | Run `29138176800` passed all 29 native rows and 71 of 72 shared rows. Its capture video proved the WPF desktop click coordinates were correct but the target was non-activating; the branch now activates the resolved root window before dispatch. |+| `541b06ba` and `7a369cfa` | Kept one process-lifetime AT-SPI registry listener and initialized it outside the daemon Tokio runtime | Run `29138635957` first exposed the nested-runtime startup panic before any cells. The replacement run `29138842890` passed GTK3 native and capture/desktop-scope preflight and behavior; the shared renderer lane remains the active validation. |+| `2ef2d9bd` | Looked for keyboard focus across every descendant UI thread of a Windows target | The previous top-level-thread-only lookup posted Tauri/WebView2 keys to the native frame while the renderer lived on another input thread. The existing external fixture-state cell remains unchanged and is validating the route. |+| `e8b8aee5` and `c3ef25ab` | Exposed the Windows foreground helper and made strict desktop fixture launch/readiness fail closed | Run `29138842358` caught the missing helper export at compile time. Canonical desktop-scope helpers now reject launch errors and readiness timeouts instead of returning a reduced green run. |+| `676fce44` | Cleaned interrupted macOS signing state before fallback signing | A bounded certificate signing attempt left `cua-driver.cstemp`; signing over it produced a resource seal for a transient file. The installer now removes partial seals and temp files, and a local reinstall produced a bundle that passes `codesign --verify --deep --strict`. |+| `f7a010c9` to `3c7d03a3` | Corrected foreground pointer ordering, renderer focus lookup, fixture isolation, targeted input, and Linux CI ownership | Windows replays reduced the matrix to one fixture-side WPF false negative; Linux native GTK3 and capture/scope owners stayed green while the shared renderer failures became individually classified. |+| `16275cb0`, `c18a8142`, and `85ed9d25` | Made the fixture journal retry, observed WPF's handled routed event correctly, and removed speculative driver delays | Windows run `29143137292` reported all 110 typed cells passing. The later physically foreground sentinel invalidated its background focus verdicts, but the WPF routed-event finding remains valid and no timing workaround remains in the driver. |+| `793def12` to `cf900891` | Built a pure-Wayland Nix session with an explicit D-Bus/AT-SPI host, compositor-aware preflight, honest oracle availability, and startup diagnostics | The original hard-coded `/run/current-system/sw/bin/dbus-daemon` failure is gone. Run `29143078987` reached a real Sway session but mapped no fixture surface; later runs now preserve Electron stderr, detect early process exit, and save the Sway tree without weakening the AX, capture, or non-blank-video gates. |+| `689f732a` and `628239fd` | Made source tests select the profile-local driver and aligned Linux tool discovery | The Nix schema test now executes the binary built in the same derivation. Linux no longer advertises the deprecated `type_text_chars` alias, matching macOS and Windows while retaining invoke-time compatibility. |+| `71c07881` to `26be3644` | Made Linux text, keyboard, pointer, and scroll routes target-aware and refusal-safe | Exact AT-SPI editable writes no longer choose another process field; editable-only nodes enter the index space; focus-dependent fallbacks refuse background dispatch; foreground keyboard and wheel input execute inside the foreground transaction. The latest Nix and X11 runs are validating these source changes. |+| `ab12e6a7` | Kept Linux accessibility geometry in the same captured tree snapshot | Chromium AT-SPI traversal order can change between walks. The driver now returns element metadata and bounds from one traversal, so an element index cannot be paired with another node's rectangle. Existing PX cells remain the regression test and contain no fixture-layout coordinates. |+| `ecffaf34` | Added a compact declared-coverage grid to the typed summary | The grid groups cells by harness and action across AX/PX and foreground/background delivery while preserving the detailed result and evidence tables. Undeclared combinations stay visibly distinct from pass, refusal, failure, and skip. |+| `3a5eec8b` | Isolated the prebuilt Linux Electron fixture from Nix's glibc library path | Wayland run `29143812672` proved that the fixture exited before mapping because `LD_LIBRARY_PATH` mixed Nix glibc 2.42 libraries with the host Electron binary. The staged launcher removes those variables for Electron only; the native Wayland, accessibility, capture, and video gates remain strict. |+| `afb16821` to `8eed7730` | Tightened source identity, fixture ownership, readiness, and incomplete-result checks | The Nix source gate now tests the platform schema it actually advertises, required shared fixtures fail closed, and cold WebView2 startup has a bounded 30-second readiness budget instead of disappearing from the matrix. |+| `349e308c` | Stabilized Windows background key targeting | Known silent-drop classes are rejected before UIA focus, while embedded renderer keys wait for an observable focused descendant rather than posting to the native frame. |+| `8b1c7357` and `19e2eaab` | Corrected X11 window discovery, double-click timing, and Linux cursor motion | Empty EWMH client lists no longer fall back to stale root children, no-window-manager discovery keeps only viewable windows, XTest double clicks flush between pairs, and Linux accepts the shared Bezier/timing cursor schema. |+| `c4214672` | Made the Windows sentinel physically own foreground before every background cell | The next run invalidated the earlier all-green Windows result: a visible but unfocused sentinel had not exercised the no-activation contract. With real foreground ownership, Electron remained clean while WPF F5 and six Tauri routes exposed honest product failures. |+| `cb91c556` and `f075f3c9` | Required a live pure-Wayland AT-SPI registry and fixed the Linux source build | The exact Nix AT-SPI launcher starts and activates `org.a11y.atspi.Registry`; the subsequent preflight now fails later because no application registers in the accessibility tree. The Nix source check passes at `f075f3c9`. |+| `4fe1047f` | Classified background WPF key drops without removing foreground delivery | WPF pointer messages remain class-level refusals. Keystrokes and chords refuse only while another native window owns foreground; the canonical F5 cell asserts exact `background_unavailable`, no fixture mutation, and all desktop-side-effect oracles. |+| `470d4827` | Hardened background WebView input mechanics | Posted clicks hold a non-activation guard, preserve unrelated window styles, restore the user's foreground if a renderer self-activates, and emit `WM_*BUTTONDBLCLK` only for classes that advertise `CS_DBLCLKS`. Embedded WebView AX keys focus through the proven posted-click route and fail closed on focus timeout; UIA scroll is guarded through handler settlement. |+| `0cef4992` to `557a59c4` | Completed the supported X11 matrix without weakening background contracts | Linux WebKit foreground typing now uses XTest, exact AT-SPI writes retry transient registry races, and unsupported background WebKit text and save actions return `background_unavailable`. Exact run `29147126435` passed all 80 X11 rows. |+| `e65f49f4` and `6f8d4368` | Matched addressed WebView key delivery to the proven pixel route | Run `29147294209` proved that releasing the outer no-activation guard did not fix the remaining Tauri AX key cell. Artifact comparison showed that `GUITHREADINFO` focus metadata was absent even when the equivalent PX route delivered. The AX route now posts its focus click, settles, and returns an explicitly unverifiable result while fixture state remains the delivery oracle. |+| `eaa59957` | Made backend-reported video failure visible to callers and manifests | A recorder backend `Err` now fails session shutdown, persists `video.error`, and has a focused core regression test. Commit `022dd537` closes the separate invalid-success metadata path. |+| `18005272` | Changed pure-Wayland sentinel discovery and screenshot scaling | The sentinel uses its unique CDP title and launched PID instead of comparing connection-local Wayland IDs. Run `29148643790` validated discovery and native screenshot scaling; remaining backend gaps stay required failures under issue `#1922`. |+| `022dd537` | Rejected invalid successful video metadata | Core now treats `finalized:false`, a missing file, or an empty file as a recording-stop error even when a backend returned `Ok`. Unit tests cover unfinalized, empty, and valid metadata. |+| `0a28495a` | Closed desktop-oracle false passes | Window-scope gates require the exact structured `desktop_scope_disabled` error, background actions require the target to be fully occluded before dispatch, and sentinel journal read errors fail the cell. |+| `8abf0950` | Pinned every interactive lane to one exact source SHA | Dispatch accepts only a full commit SHA or the immutable dispatch SHA. Every lane verifies `git rev-parse HEAD`, records the SHA in `environment.jsonl`, and prints it in the typed summary. |+| `9c76eb5f` | Routed addressed WebView key focus through the complete pixel actuator | Run `29147987757` proved that a direct posted focus click still did not deliver the Tauri AX key. The AX center now converts back to screenshot coordinates and calls the same layered ClickTool route as the passing PX cell, including targeted-injection fallback. |+| `69d3538c` | Checked occlusion at the authoritative dispatch snapshot | The sentinel rejects a cell when the target rises between the initial precondition check and the desktop observer's before-action snapshot. |+| `900bfe8d` | Separated desktop-wide sentinel observations from background-target observations | Run `29148642577` passed all 72 shared rows and all 8 capture rows, proving the WebView key and strict scope fixes. Its two native failures showed that launch and agent-cursor cells intentionally observe the foreground sentinel because no background target exists. They now use an explicit desktop-observation API; real background actions keep both occlusion checks. |+| `7fafe2c0` | Ordered pure-Wayland native and renderer focus checks | The Sway-only sentinel path waits for native readiness, explicitly focuses the launched PID, verifies the compositor foreground state, and only then waits for the renderer focus journal. Windows and X11 ordering is unchanged. |+| `516cddb2` | Replaced WinUI3's cold-start sleep with external readiness polling | Run `29149160683` passed all 72 shared rows and all 8 capture rows. Its sole native failure was the first WinUI3 UIA lookup; six later WinUI3 rows passed. Native setup now polls the external UIA tree for the app marker and canonical checkbox before executing any WinUI3 cell. |+| `3b67ee30` | Made compositor focus authoritative for pure-Wayland sentinel startup | Run `29149710714` proved the sentinel was natively foreground and stable, but Electron emitted no renderer focus event because the window could be focused before preload listeners attached. The startup gate now uses the compositor snapshot; action-time native focus, z-order, and journal input-leak checks remain mandatory. |++The earlier conclusion that fully occluded Chromium clicks required a z-order+change was incomplete. A VM-only diagnostic confirmed that UIA Invoke delivers+but activates Electron, so UIA still violates the background contract. The+same diagnostic also proved that `post_click_screen` reaches the renderer while+it remains fully occluded: both AX-resolved and PX-resolved left clicks+delivered in three consecutive runs without changing focus, z-order, or the+real cursor. Commit `6aaa914b` restores that route only for the proven+single-left-click shape; right-click, double-click, drag, and other unproven+gestures retain their existing refusal behavior.++## Validation Snapshot (2026-07-12)++The first rows are the accepted landing evidence. Older rows remain below as+superseded historical checkpoints; they are not current support baselines.++| Platform | Run | Purpose | State |+| --- | --- | --- | --- |+| Windows | `29211506624` | Final conv2 matrix at `6d7f02e4` | Passed all 114 rows: 93 delivered and 21 exact refusals, with 0 failures or skips. Shared, native, and capture/scope artifacts contain per-row trajectories and MP4 evidence. |+| macOS | Logged-in local matrix at `448d052f` | Final conv2 matrix with the installed app's stable TCC identity | Passed all 133 rows with 0 failures or skips and an owned MP4 for every row. |+| Linux X11 | `29213365684` | Final Openbox/Xvfb matrix at `72083bb4` | Passed all 108 rows: 71 delivered and 37 exact refusals, with 0 failures or skips. |+| Linux Sway | `29213349962` | Final stock-wlroots matrix at `72083bb4` | Passed the same 108-row contract as X11: 71 delivered and 37 exact refusals, with 0 failures or skips. Roleless WebKitGTK and live compositor geometry are covered by the accepted Tauri rows. |+| Nix Linux source | `29213921494` | Final package, compositor, and Rust source checks at `49e8a4cd` | Passed all three checks on the exact Nix landing head after its rebase onto merged Linux. |+| Linux X11 | `29179817332` | Final expanded shared, native, capture, and desktop-scope matrix at `f8e4b37b` | Passed all 108 rows: 71 delivered and 37 exact refusals, with 0 failures or skips. The three lane artifacts contain 108 cell videos plus three preflight videos, and the combined summary links every row to its owning artifact. |+| Linux Wayland | `29179817787`, `29182130646` | Split Sway shared, native, capture, and focused Electron validation | Native passed all 31 rows: 24 delivered and 7 exact refusals. Capture and desktop scope passed all 5 rows. The shared lane executed all 72 Electron/Tauri rows: Electron exposed a small keyboard cluster and Tauri was environment-limited by WebKitGTK's missing DRM/EGL renderer tree. Focused repairs now prove Electron foreground/background AX left click and foreground AX type text; foreground hotkey and hosted renderer PX-origin reconstruction remain explicit failures. |+| Linux GNOME | Real GNOME 46 session | Validate native Mutter discovery, capture, AX/PX dispatch, occlusion, and persistent portal/libei input | Passed 31 GTK3 rows, 3 capture contracts, and 2 desktop-scope contracts. A second driver process reused the explicitly revocable portal grant without another consent dialog. |+| Windows | Azure RDP focused replay | Recheck occluded Electron AX/PX background left-click delivery and neighboring pointer contracts at `6aaa914b` | The background-safe `PostMessage` diagnostic delivered AX and PX in 3/3 runs with all five side-effect oracles. The production build then passed 12/12 Electron left/right/double-click rows and emitted a video for every row. |+| Windows | `29149710089` | Final exact full matrix at `516cddb2` with strengthened desktop oracles and external WinUI3 readiness | Passed all 110 rows: 87 delivered and 23 exact refusals, with 0 failures or skips. All three lane summaries identify the full source SHA; 110 cell videos plus 3 preflight videos were uploaded and parsed successfully. |+| Windows | `29166454913` | Exact converged matrix at `e43c2946` after macOS parity, native PX, sentinel, and launch-contract work | Passed all 113 rows: 92 delivered and 21 exact refusals, with 0 failures or skips. Shared passed 54/18, native passed 30/3, and capture/scope passed 8/0. The three lane artifacts contain 113 cell videos plus 3 preflight videos; every row links its exact trajectory path. |+| Nix Linux source | `29150690892` | Package build, Linux unit/protocol gate, and source identity at `3b67ee30` | Passed. The package and source-owned Rust tests completed in the pinned Nix environment at the final code SHA. |+| Linux X11 | `29148643166` | Exact supported Linux matrix at `69d3538c` with strengthened occlusion and scope gates | Passed all 80 rows: 48 delivered and 32 exact refusals, with 0 failures or skips. All three summaries and environment records identify the full source SHA; 80 cell videos plus 3 preflight videos were uploaded. |+| macOS | Local complete matrix at `6e05424a` | Validate the converged shared, native, capture, and desktop-scope catalog with stable TCC identity | Passed all 83 declared rows: 79 delivered and 4 exact refusals, with 0 failures or skips. Every row produced a typed result and video, and both the summary and environment record contain the full source SHA. |+| macOS | Local complete matrix at `8d28beec` | Validate native WKWebView and the expanded AppKit/SwiftUI catalog at one exact source SHA | Passed all 130 declared rows: 124 delivered and 6 exact refusals, with 0 failures or skips. The 108 shared rows cover Electron, Tauri, and WKWebView; all native, capture, and desktop-scope rows passed. Every row produced a typed result and validated video. |++The earlier macOS replay fixed PX background click focus steals, foreground PX typing's+first-character loss, silent web-content background scroll/drag no-effects,+desktop-scope HID delivery, AppKit native PX background click coverage, and+daemon transport loss of structured refusal codes. Electron background scroll+and Electron/Tauri background drag now return exact `background_unavailable`+refusals. The current pass proves SwiftUI popover activation through fixture+state; only transient-panel AX enumeration remains unproven. A targeted CDP+wheel experiment could mutate Electron while visible or ordinarily+backgrounded, but had no effect under the canonical full-occlusion sentinel.+The experiment was removed, and Electron background scroll keeps its exact+`background_unavailable` contract.++Repeated local runs also exposed two evidence-harness defects. The macOS+runner now refuses a dirty tracked tree and derives a full source SHA when CI+does not supply one. Desktop-scope tests no longer reuse a fixed daemon session+ID after that session has ended; each proxy connection keeps its own minted+session identity.diff --git a/libs/cua-driver/docs/e2e-results-summary-design.md b/libs/cua-driver/docs/e2e-results-summary-design.mdnew file mode 100644index 0000000000..c823c8fb07--- /dev/null+++ b/libs/cua-driver/docs/e2e-results-summary-design.md@@ -0,0 +1,123 @@+# E2E Results Summary Design++**Historical design record.** The implemented reporting contract is documented+in `e2e-ci-reporting.md`.++This document specifies the GitHub Actions presentation built from the Rust v2+case and result contract. Field semantics live in `e2e-ci-reporting.md`; the+convergence sequence lives in `test-harness-convergence-plan.md`.++## Goals++The typed artifacts and run summary together must answer:++- Was the desktop environment ready?+- Which declared behavioral cells delivered, refused, failed, or did not run?+- Which harness, action, targeting mode, delivery mode, and driver route did+ each cell cover?+- Which external and desktop oracles passed?+- Which lane archive owns that cell's recording evidence?+- Are any declared cells missing or duplicated?++Cargo test names and lane exit codes do not answer these questions and are not+behavioral rows.++## Summary Layout++The generated Markdown starts with aggregate behavioral totals, followed by a+declared-coverage grid and the detailed rows:++```markdown+# CUA Driver E2E++**Result:** 42 delivered, 3 refused, 2 failed, 0 skipped++**Source SHA:** `0123456789abcdef0123456789abcdef01234567`++## Declared Coverage++## Detailed Results+```++Environment readiness and source identity live in `environment.jsonl`, the+generated summary, and the workflow metadata. An environment failure prevents+behavioral cells and makes the reporter fail; it is not converted into a+partial green summary.++## Behavioral Table++One row represents one declared `cell_id`:++```markdown+| Cell | Harness | Action | Targeting | Delivery | Route | Expected | Observed | Oracles | Status | Time | Evidence |+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---: | --- |+| linux-electron-left-click-ax-background | Electron | Left click | AX | Background | AT-SPI action | Refuse | Refused | Focus, z-order, no leak | PASS | 1.5s | [recordings/.../recording.mp4] |+| linux-electron-left-click-ax-foreground | Electron | Left click | AX | Foreground | AT-SPI action | Deliver | Delivered | Fixture | PASS | 2.1s | [recordings/.../recording.mp4] |+```++The row does not label a refusal as pass unless the case declaration expects+refusal and every required no-side-effect oracle passed.++## Coverage Table++The reporter also renders declared coverage. A dash means that the Rust catalog+does not declare that combination; it is not a pass or an inferred result:++```markdown+| Harness | Action | AX/BG | AX/FG | PX/BG | PX/FG | Page | NotApplicable |+| --- | --- | --- | --- | --- | --- | --- | --- |+| electron | left_click | REFUSED | PASS | REFUSED | PASS | - | - |+| electron | drag | - | - | REFUSED | PASS | - | - |+```++The detailed table remains the authority for route, contract, oracle, and+failure information for every declared cell.++## Validation++The reporter fails before publishing a green summary when it finds:++- duplicate declarations or results;+- a declared cell with no result;+- a result with no declaration;+- a serialized status that contradicts expected and observed behavior;+- an unknown refusal code;+- a passing cell missing a required oracle;+- unfinalized, missing, or empty required video evidence;+- a behavioral video phase that never reached `finalized` after setup/posture;+- an invalid hosted-runner console cleanup status;+- more or fewer than one environment record.++The reporter validates the exact video it renders. It records the trajectory+path but does not currently validate that file independently. Workflow summary+jobs may concatenate validated platform summaries, but they do not recalculate+results.++## Evidence Packaging++Each internal workflow lane uploads one archive. Every cell owns a stable+subdirectory derived from `cell_id` inside that archive:++```text+recordings/<cell-label>-pid<pid>-<sequence>/+|-- recording.mp4+|-- trajectory.json+|-- session.json+|-- cursor.jsonl+`-- turn-*/+ |-- action.json+ `-- screenshot.png+```++GitHub artifact archives do not provide stable URLs to individual files. The+row therefore links its exact MP4 path text to the owning lane archive; the+trajectory rollup links the same archive and reports its video count. A future+static report may provide inline playback; it is not required for the GitHub+summary. Cargo target logs sit at the lane root and diagnose runner or test+failures; they are not invented as per-cell evidence.++## Unit And Protocol Results++Unit, schema, transport, and CLI tests remain separate workflow output. They do+not share the behavioral case schema, appear as invented behavioral rows, or+require desktop video.diff --git a/libs/cua-driver/docs/linux-desktop-validation.md b/libs/cua-driver/docs/linux-desktop-validation.mdnew file mode 100644index 0000000000..23fb3adaf2--- /dev/null+++ b/libs/cua-driver/docs/linux-desktop-validation.md@@ -0,0 +1,54 @@+# Linux representative desktop validation++Hosted CI owns the canonical Xvfb/Openbox and headless Sway environments. Some+contracts require a real user desktop and therefore run only when a maintainer+explicitly provisions the corresponding environment.++## Environments++| Environment | Required evidence | Current status | Preflight |+| --- | --- | --- | --- |+| GNOME/Mutter | WinRects geometry and activation, portal/libei input, portal recording, and shared renderer apps | Native GTK behavior, capture, and desktop scope are accepted; shared renderers and portal video remain open | Wayland user session, enabled WinRects helper, and portal grant |+| KDE/KWin | KWin-specific activation and portal behavior | Plasma 6 session startup, GTK AT-SPI discovery, and portal interfaces were observed; no behavioral matrix is accepted | Plasma/KWin 6 Wayland session; Plasma 5.27 is rejected |+| Real Xorg | MPX/uinput behavior that Xvfb cannot provide | Not yet validated | Non-Wayland Xorg session with `/dev/uinput` access |+| DRM/EGL renderer | Representative WebKitGTK/Tauri accessibility tree | Not available on the hosted Sway or representative desktop hosts used so far | A real `/dev/dri/renderD128`; software-only headless rendering is rejected |++## Source ownership++The host checkout is the only checkout that commits or pushes. Sync it with:++```bash+libs/cua-driver/scripts/sync-vm-worktree.sh push user@host '~/cua'+```++The sync writes `.cua-e2e-source-sha` after transferring the worktree. The+canonical preflight validates that marker when the VM intentionally has no+`.git` directory, so reports still identify the exact host commit.++## Run++Start the command from the graphical user's systemd user manager or an+equivalent terminal inside that user's session. The wrapper rejects the wrong+desktop generation before building fixtures.++```bash+scripts/ci/linux/run-rust-e2e-desktop.sh gnome+scripts/ci/linux/run-rust-e2e-desktop.sh kde+scripts/ci/linux/run-rust-e2e-desktop.sh xorg+```++The default is the complete canonical matrix. `CUA_E2E_INTERNAL_LANE` and+`CUA_E2E_HARNESS_FILTER` remain diagnostic/maintainer controls; they do not+define a second catalog.++## Evidence acceptance++Behavioral acceptance requires typed rows, the exact source SHA, independent+fixture-state or refusal evidence, and every action-specific desktop oracle. A+representative result without reporter-owned per-cell video may establish that+behavior, but it is recorded as lacking full evidence parity.++Full hosted parity also requires the Markdown summary, screenshots,+trajectories, and per-cell videos enforced by the reporter. Record accepted+behavior and any evidence-parity gap in `action-support.md`. A setup failure is+an environment error, never a smaller green matrix.diff --git a/libs/cua-driver/docs/linux-support-completion-plan.md b/libs/cua-driver/docs/linux-support-completion-plan.mdnew file mode 100644index 0000000000..b767e687bd--- /dev/null+++ b/libs/cua-driver/docs/linux-support-completion-plan.md@@ -0,0 +1,305 @@+# Linux support completion plan++**Status:** Historical implementation plan. Current outcomes live in+`action-support.md`; remaining work lives in the+[public platform roadmap](https://cua.ai/docs/reference/cua-driver/platform-roadmap).++**Scope:** Linux driver behavior, canonical Rust E2E evidence, and release validation++**Principle:** Standard Wayland and the optional compositor-owned injection environment are separate products with separate claims.++## Goal++Close the remaining Linux gaps without restoring the deleted Python, shell, GIF,+or real-app fixture suites. Reuse the repo-local Rust harness catalogs and their+external state, focus, z-order, cursor, input-leak, screenshot, trajectory, and+video oracles.++The work is complete when each supported action either:++- delivers and changes fixture-owned state under every required side-effect oracle;+- returns the declared structured refusal before dispatch; or+- is recorded as an environment gap with a named owner and representative test lane.++A successful driver response is never delivery evidence.++## Execution status++| Phase | Result |+| --- | --- |+| Truth and route metadata | Complete. Current docs distinguish X11, stock compositor routes, portal/libei, and the nested private protocol. Typed results record compositor and input backend. |+| Nested protocol hardening | Complete. The protocol has a version handshake, acknowledgements, stable PID targeting, focus/z-order queries, geometry queries, strict key/text validation, and explicit errors for missing resources. |+| Nested typed environment | Complete as an experimental lane. Nix packages the compositor, the canonical Rust catalogs run inside it, and every lane retains typed evidence and video. Promotion still depends on a complete accepted run. |+| Protocol action coverage | Implemented for left/right/double click, printable ASCII text, named keys, hotkeys, scroll, and single/parallel drag. Unicode text and a canonical parallel-drag behavior row remain unproven. |+| Stock Sway repairs | Complete for the declared hosted catalog. Exact run `29213349962` passed all 108 Electron, Tauri, GTK3, capture, and desktop-scope outcomes: 71 deliveries and 37 exact refusals. |+| Representative desktops | Partial. GNOME native GTK behavior is accepted; Plasma 6 reached session, AT-SPI, and portal preflight only. Shared GNOME renderers, portal video, real Xorg MPX, and DRM/EGL WebKitGTK remain named gaps. |+| Release policy | Complete. Linux unit/source/distribution gates run automatically; X11, Sway, and nested GUI matrices are exact-SHA maintainer dispatches with row-level artifacts. |++## Current baseline++| Environment | Current evidence | Main gaps |+| --- | --- | --- |+| X11/Openbox on Xvfb | 108/108 declared outcomes: 71 deliveries and 37 exact refusals | Real-Xorg MPX/uinput and parallel pointer behavior are not proven by Xvfb |+| Sway/wlroots, native Wayland | 108/108 declared outcomes: 71 deliveries and 37 exact refusals across Electron, Tauri, GTK3, capture, and desktop scope | Other wlroots compositors remain unproven; stock Wayland still cannot provide arbitrary focus-free raw input outside declared target-addressable routes |+| GNOME 46/Mutter | GTK3 31/31 using WinRects, AT-SPI, and portal/libei | Shared Electron/Tauri matrix and portal MP4 evidence |+| KDE/KWin | Plasma 6 session startup, GTK AT-SPI discovery, and portal interfaces | Target-addressable activation adapter and an accepted behavioral lane |+| Nested `cua-compositor` | Packaged experimental environment, route metadata, observer, native GTK3 31/31, capture/scope 5/5, and Electron shared 26/36 before focused repairs | Stable renderer click/scroll coverage, a complete accepted shared matrix, Unicode text, and a canonical parallel-drag row |++Standard Wayland cannot generally route raw pointer or keyboard events to an+arbitrary occluded, unfocused surface. The nested `cua-compositor` is different:+it owns the compositor and can route directly to a client surface through its+private control socket. Evidence from that environment must never be presented+as a stock Sway, GNOME, KDE, or general Wayland capability.++## Target environment matrix++Keep the matrix small. An environment earns a lane only when it proves a+contract that another lane cannot.++| ID | Environment | Purpose | Ownership |+| --- | --- | --- | --- |+| E1 | X11/Openbox on Xvfb | XSendEvent, XTest, AT-SPI, capture, scope, complete shared/native catalogs | Hosted GitHub runner; canonical |+| E2 | Sway on native Wayland | wlroots virtual pointer, screencopy, foreign toplevel, and standard-Wayland refusals | Hosted GitHub runner; canonical |+| E3 | Nested `cua-compositor` | Raw focus-free per-surface input and multi-pointer behavior | Hosted GitHub runner; experimental until proven |+| E4 | GNOME/Mutter real session | WinRects, portal/libei, shared renderer apps, portal recording | Maintainer VM; representative |+| E5 | Real Xorg session | MPX/uinput behavior unavailable under Xvfb | Maintainer VM; optional |+| E6 | KDE Plasma 6/KWin | KWin activation and portal behavior | Maintainer VM; experimental |+| E7 | Real DRM/EGL renderer session | WebKitGTK/Tauri accessibility tree and native renderer geometry | Prefer E4; use a separate VM only if needed |++Labwc, Hyprland, and other wlroots compositors remain expected-compatible but+unproven unless a user report demonstrates a meaningful divergence. XWayland is+not a separate lane.++## Phase 0: Reconcile the truth++The phase sections below preserve the implementation and acceptance criteria+used for this work. Current outcomes are summarized above; unchecked evidence+gaps remain follow-up work rather than implied support.++Update stale internal references before changing behavior.++**Work**++- Mark stale sections of `libs/cua-driver/docs/test-harness-convergence-plan.md`+ as superseded by current evidence.+- Rewrite `nix/cua-driver/tests/README.md`; it currently names deleted fixtures.+- Remove the empty `nix/cua-driver/tests/wayland/` directory if it is still empty.+- Add the experimental nested-compositor row to+ `libs/cua-driver/docs/action-support.md` and link it from the public platform+ support page.+- Replace misleading "EIS compositor" wording. The current route is a private+ Unix socket protocol, not libei/EIS.++**Acceptance**++- No documentation points to a deleted fixture or workflow.+- Every public capability has current evidence or an explicit experimental gap.++## Phase 1: Make results identify the real route++The result schema currently says only `Wayland` and can make nested injection+look like a standard virtual-pointer or AT-SPI route.++**Work**++- Add `LinuxCuaCompositorInject` to `DriverRoute` in+ `cua-driver-testkit/src/e2e.rs`.+- Add environment fields for `compositor` and available `input_backends`.+- Detect values such as `openbox-x11`, `sway`, `gnome-mutter`, `kwin`, and+ `cua-compositor-nested` in the environment preflight.+- Update `cua-e2e-report` and the Markdown summary to display the compositor and+ input route without changing test-family or action naming.+- Keep backward compatibility with existing result artifacts.++**Acceptance**++- E1 and E2 retain identical cell outcomes.+- Every delivered cell identifies both its display server and actual route.+- E2 and E3 artifacts cannot be mistaken for each other.++## Phase 2: Audit and harden nested injection++This is a proof gate, not an assumption. The existing path has two correctness+risks:++1. `app_id_for_window` appears to compare a toplevel object ID obtained from one+ Wayland connection with objects from a new connection. Protocol IDs are+ connection-scoped, so target resolution must be tested and likely replaced+ with stable identity resolution.+2. The compositor silently ignores unsupported Unicode and keys and has no+ scroll or hotkey verb. The driver must never return success for those cases.++**Work**++- Add protocol handshake/version and acknowledgements.+- Resolve targets through stable app/window identity, including duplicate app IDs.+- Pre-validate all text and key shapes before sending.+- Return exact `background_unavailable` refusals for unsupported text, key,+ hotkey, scroll, or drag shapes until their protocol verb exists.+- Add focused unit tests with a mock Unix socket for encoding, acknowledgements,+ validation, target ambiguity, and error mapping.+- Verify or correct the stale `LIBEI_SOCKET` comment in `wayland/libei.rs`.++**Acceptance**++- No unsupported command can be silently dropped.+- No route depends on connection-local Wayland object IDs as stable identities.+- A compositor protocol mismatch fails preflight once, before the matrix starts.++## Phase 3: Restore the compositor as a typed environment++Do not restore the old Nix/Python fixture suite. Package only the environment,+then run the existing Rust catalogs inside it.++**Work**++- Export `cua-compositor` as a flake package and build check.+- Add a `cua-driver-inject-e2e` development shell containing the compositor and+ existing Rust harness dependencies.+- Add `scripts/ci/linux/run-rust-e2e-inject.sh` to start the nested session,+ perform capability preflight, and delegate to the canonical+ `scripts/ci/linux/run-rust-e2e.sh` command.+- Add an independent `CuaCompositor` observer in the testkit. Read-only control+ queries must report focus and z-order directly; the driver response cannot be+ the observer.+- Reuse the GTK3 and shared `CaseSpec` catalogs. Key expectations by detected+ environment capability, not by weakening an oracle.+- Add a repo-local parallel-drag journal action because the old test asserted+ only command completion and therefore supplied insufficient evidence.+- Record every cell with before/after screenshots, trajectory data, and video.++**Minimum Step A matrix**++| Background PX action | Initial contract |+| --- | --- |+| Left, right, and double click | Deliver |+| ASCII type text | Deliver |+| Supported named key | Deliver |+| Single drag | Deliver after socket routing exists; otherwise exact refusal |+| Two simultaneous drags | Deliver with fixture state for both paths |+| Scroll | Exact refusal until an axis verb exists |+| Hotkey | Exact refusal until a chord verb exists |+| Unsupported Unicode or key | Exact refusal before socket dispatch |++All delivery rows require fixture state, unchanged sentinel focus and z-order,+and no leaked input. Cursor preservation is not an oracle when the compositor+route intentionally has no physical cursor.++**Acceptance**++- `nix build .#cua-compositor` is a CI check.+- The hosted experimental lane completes the Step A matrix three consecutive+ times on one source SHA with owned videos and no orphan evidence.+- The two useful historical claims, focus-free typing and simultaneous drags,+ are superseded by stronger typed fixture evidence.++**Deletion gate**++If the compositor cannot build, target stable identities, or support truthful+focus/z-order observation after two focused implementation cycles, remove the+entire dormant subsystem and its public capability claim in one PR. Do not leave+advertised but untested production branches indefinitely.++## Phase 4: Complete the injection protocol++After Step A is green, add protocol verbs for axis/scroll, modifier chords,+Unicode text, broader named keys, and single drag. Flip a test from refusal to+delivery only when the unchanged fixture and side-effect oracles pass.++**Acceptance**++- Background left/right/double click, Unicode typing, named keys, hotkeys,+ scroll, single drag, and parallel drag all have typed delivery evidence.+- Remaining refusals describe genuinely unsupported public shapes.++## Phase 5: Close stock Sway gaps++Work independently of the custom-compositor lane.++**Work**++- Repair Electron foreground AX/PX hotkey delivery in the driver.+- Repair renderer-to-compositor PX origin reconstruction.+- Keep Tauri/WebKitGTK environment-limited on hosted pixman Sway; do not use+ fixture changes or renderer-disabling flags as representative proof.++**Acceptance**++- Existing failing cells become delivered through external fixture state, or+ remain named gaps with exact diagnostics and a linked issue.++## Phase 6: Representative desktop coverage++### GNOME and WebKitGTK++- Run the unchanged shared Electron catalog on the existing GNOME VM.+- Add portal MP4 capture with the same ownership and nonblank checks as Sway.+- Check `/dev/dri` first and run the Tauri/WebKitGTK catalog there when a real+ render node is available.+- If it is unavailable, provision a dedicated DRM/EGL VM rather than adapting+ the fixture to software-renderer behavior.++### KDE++- Implement a target-addressable KWin activation adapter using supported KWin+ scripting or D-Bus interfaces.+- Validate activation through an independent observer before enabling portal+ input.+- Use Plasma 6; do not accept the broken Plasma 5.27 cloud image as evidence.++### Real Xorg++- Add an optional maintainer lane for MPX/uinput and parallel pointer behavior.+- Do not describe Xvfb refusals as proof that real-Xorg delivery is impossible.++## Phase 7: Release validation policy++- Keep E1 X11 and E2 Sway dispatch-only or maintainer-triggered, but require+ green run links on the exact release SHA in the release checklist.+- Keep E3-E7 optional until each environment has an owner and three stable runs.+- Store results, summary, screenshots, trajectories, logs, and videos per lane.+- Fail preflight once for a missing session capability; do not turn all cells+ into skips.++## Suggested PR sequence++1. Documentation reconciliation.+2. Result route and environment metadata.+3. Nested-injection refusal hardening and mock-socket tests.+4. Sway hotkey repair and PX geometry repair as independent changes.+5. Compositor package, build check, protocol query support, and stable identity.+6. Testkit observer, typed Step A matrix, runner, and experimental workflow.+7. Protocol completion and expectation flips.+8. GNOME shared/video and representative Tauri renderer evidence.+9. KWin adapter and Plasma 6 evidence.+10. Release checklist gating.++Each PR must leave existing canonical lanes green. A new environment may fail+only in a clearly labeled experimental job and may not change the public+contract of E1 or E2.++## Anti-overfitting rules++- No Python behavioral runner, second matrix file, terminal/GIF fixture, or old+ Nix test restoration.+- No command-return-only oracle.+- No AT-SPI delivery described as raw PX injection.+- No raw-background claim without physical occlusion and focus/z-order guards.+- No fixture expectation change made solely to accommodate driver behavior.+- No compositor proliferation without a unique contract and named owner.+- No software-renderer workaround accepted as representative WebKitGTK evidence.++## Definition of done++- E1 remains 108/108 on the final source SHA.+- E2 has no silent no-effect outcome; hotkey and PX geometry are delivered or+ recorded as precise gaps.+- E3 has the complete typed raw-injection matrix and truthful route metadata, or+ the dormant subsystem and its claim are removed.+- GNOME has shared Electron, portal video, and representative WebKitGTK evidence.+- KDE has verified target-addressable activation on Plasma 6 or remains clearly+ experimental with that adapter as the named blocker.+- Every supported or refused action is represented by a typed Rust row and+ external evidence across the environments where its contract differs.+- Public and contributor documentation link current evidence and never conflate+ standard Wayland with compositor-owned raw injection.diff --git a/libs/cua-driver/docs/pr-2161-landing-plan.md b/libs/cua-driver/docs/pr-2161-landing-plan.mdnew file mode 100644index 0000000000..507135badc--- /dev/null+++ b/libs/cua-driver/docs/pr-2161-landing-plan.md@@ -0,0 +1,666 @@+# PR #2161 Landing Plan - Second-Wave E2E Convergence++**Historical execution plan.** This file records how the original draft was+split and landed. It is not the current test or platform support reference.++**Status:** executed as a replacement PR stack; outcome recorded below - **Author:** convergence audit follow-up - **Date:** 2026-07-12+**Source PR:** #2161 (`codex/link-e2e-videos`, draft) - 229 files, +24,821 / -17,033, 310 commits.+**Merge state vs `main`:** conflicts only in `docs/content/docs/concepts/meta.json`.++## Landing outcome++| Planned slice | Landed PR | Result |+| --- | --- | --- |+| Docker cleanup sidecar | #2177 | Merged as `8cfdd416` |+| TypeScript tooling sidecar | #2178 | Merged as `dd0102eb` |+| 1/7 typed E2E foundation | #2179 | Merged as `107915af` |+| Shared fixture journal correction | #2181 | Merged as `2df3fbd` |+| 2/7 Windows convergence | #2180 | Merged as `88f88e1b` after 114/114 accepted rows |+| 3/7 macOS convergence | #2183 | Merged as `1d0ff7cc` after 133/133 accepted rows |+| 4-5/7 Linux X11 and Wayland convergence | #2182 | Merged as `51a8e259` after 108/108 accepted rows on both X11 and Sway |+| 6/7 Nix consolidation | #2184 | Replacement PR; final exact run is recorded in the convergence journal |+| 7/7 docs and replacement cleanup | This replacement PR | Current platform truth, contributor guidance, release wiring, obsolete-fixture removal, and source-generated Lume 0.3.15 references |++The original seven-slice design landed Linux X11 and Wayland together because+their final fixes shared the Linux backend and exact validation source. The+small shared-fixture correction landed separately to keep the foundation PR+reviewable. Those adaptations preserve the ownership and dependency intent of+the plan without replaying intermediate branch history.++---++## 1. Decision and rationale++**Decision: do not merge #2161. Re-land its delta as a stack of 7 fresh PRs cut from+current `main`, each built from a scoped final-state patch for its owned path set, not+by cherry-picking commits or overwriting current `main` with whole files blindly.**++Rationale:++- **The first-wave split already landed.** The #2135 split plan was executed in merged+ PRs #2138-#2154. #2161 is the _second-wave convergence delta_ that accumulated on the+ long-lived branch afterwards. Only that delta needs to land; the branch history is+ unreviewable as-is (310 commits, many of which rework each other).+- **Cherry-picks are the wrong tool here.** The 310 commits repeatedly rewrite the same+ files (e.g. `recording.rs`, the e2e workflows). Cherry-pick sequences would conflict+ with each other and reproduce dead intermediate states. The branch's _final tree_ is+ the reviewed artifact; scoped binary patches preserve its final add, modify, delete,+ and rename intent while still forcing conflicts against newer `main` to be resolved.+- **The branch also contains intentional work outside the Cua Driver convergence+ scope.** The TypeScript tooling migration and Docker cleanup need their own review+ and rollback boundaries. The snapshot's Lume 0.3.14 output must not be replayed;+ the final docs PR regenerates the references from current Lume 0.3.15 source. A+ preservation queue keeps this work visible without mixing it into the platform+ PRs.+- **7 PRs matches the natural seams**: one shared foundation (typed evidence testkit)+ that everything compiles against, three platform silos with disjoint crates, two+ Linux-display follow-ons with a real dependency chain, and one docs/cleanup capstone.+ Fewer PRs would recreate the unreviewable blob; more would multiply shared-file+ splits (Cargo manifests, `platform-linux/src/lib.rs`) for no review benefit.++### Historical evidence vs fresh gates - the governing rule++- **Historical evidence** = CI runs on #2161 / the convergence branch, and the run+ records in `libs/cua-driver/docs/e2e-convergence-journal.md`. It proves the _final+ tree_ once worked as a whole. Cite it in PR descriptions for reviewer context.+- **Fresh required gates** = commands/workflow runs executed **on each new stack+ branch at its head SHA**. Only these gate merging. Historical evidence never+ substitutes for a fresh gate, because each stack PR is a tree state that never+ existed on the convergence branch (foundation without platforms, X11 without+ Wayland, etc.).++---++## 2. Global allowlist and preservation queue++### 2.1 Strict global allowlist (in scope for the 7-PR stack)++Everything in the #2161 delta under:++- `libs/cua-driver/rust/**` (crates, tests, Skills, `Cargo.toml`, `Cargo.lock`, READMEs)+- `libs/cua-driver/tests/**` (fixtures, runners - including the deletions)+- `libs/cua-driver/docs/**`, `libs/cua-driver/README.md`, `libs/cua-driver/scripts/**`+- `libs/cua-driver/wayland-helper/**`+- `scripts/ci/**` (`README.md`, `link-e2e-evidence.sh`, `linux/`, `macos/`, `windows/`)+- `nix/cua-driver/**`, `flake.nix`+- `.github/workflows/`: `e2e-rust-windows.yml`, `e2e-rust-linux.yml`,+ `e2e-rust-linux-wayland.yml`, `ci-rust-linux.yml`, `ci-rust-windows.yml`,+ `ci-nix-linux.yml`, `cd-rust-cua-driver.yml`, `ci-release-reminder.yml`,+ and the deletions `ci-cua-driver-interactive-linux.yml`, `nix-build.yml`,+ `nix-screenshot.yml`, `nix-wayland.yml`+- `.github/scripts/tests/test_cua_driver_release_wiring.py`+- `docs/content/docs/concepts/{how-cua-driver-is-validated.mdx,index.mdx,meta.json}`+- `docs/content/docs/reference/cua-driver/**`+- Root docs: `README.md`, `CONTRIBUTING.md`, `Development.md`, `TESTING.md`++### 2.2 Preservation queue (excluded from the seven Cua Driver PRs)++| Owner | Paths | Intent | Disposition |+| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |+| Sidecar A: Docker cleanup | `Dockerfile`, `.dockerignore` | Remove the obsolete root development image and its ignore file | Fresh two-file PR from current `main` |+| Sidecar B: TypeScript tooling | `package-lock.json`, `libs/typescript/package.json`, `libs/typescript/.prettierignore`, `.github/workflows/ci-lint-typescript.yml`, `.pre-commit-config.yaml`, `.github/workflows/claude-auto-fix.yml` | Make the TypeScript workspace own pnpm setup, build core before typecheck, and keep generated output out of formatting | Fresh six-file PR from current `main` |+| Generator adaptation | `docs/content/docs/reference/lume/cli-reference.mdx`, `docs/content/docs/reference/lume/http-api.mdx` | Keep generated Lume references current | Regenerate from current 0.3.15 source in the final docs PR; do not replay the snapshot's 0.3.14 bytes |++Separation enforcement: every Cua Driver stack branch must satisfy+`git diff --name-only origin/main...HEAD | grep -E -x '(Dockerfile|\.dockerignore|package-lock\.json|\.pre-commit-config\.yaml|libs/typescript/.*|\.github/workflows/(ci-lint-typescript|claude-auto-fix)\.yml|docs/content/docs/reference/lume/.*)'`+-> **empty output** (add as a pre-push checklist item on each PR).++#### Sidecar A - `codex/conv2-sidecar-docker-cleanup`++Suggested title: `chore(repo): remove obsolete development Docker image`.++Fresh gates:++- [ ] Repo-wide reference search proves no maintained command, workflow, or guide uses+ the root `Dockerfile` or `.dockerignore`+- [ ] Existing container workflows still point to their owned Dockerfiles+- [ ] The PR contains exactly the two deletions++#### Sidecar B - `codex/conv2-sidecar-typescript-tooling`++Suggested title: `ci(typescript): use workspace-local pnpm tooling`.++Fresh gates:++- [ ] `pnpm -C libs/typescript install --frozen-lockfile`+- [ ] `pnpm -C libs/typescript typecheck`+- [ ] `pnpm -C libs/typescript format:check`+- [ ] The TypeScript lint workflow passes on the sidecar head SHA+- [ ] The pre-commit TypeScript hook invokes the same workspace command++The sidecars are independent of the seven-PR stack and may land while PR 1 is under+review. They still belong to the #2161 preservation record.++### 2.3 Coverage invariant++Every file in `git diff --name-status origin/main...codex/link-e2e-videos` is owned by+**exactly one** stack PR, sidecar PR, or superseded-by-main row. The audit in Section 7+verifies this mechanically;+if a file turns out unowned or double-owned during execution, stop and amend this plan+first.++---++## 3. The 7-PR stack++Branch prefix: `codex/conv2-0N-<slug>`. PR titles: conventional-commit style with a+`(conv2 N/7)` suffix. All PRs squash-merge. All PR bodies link #2161 and this plan.++Common transfer recipe (per PR, from a clean worktree). Replace `<owned paths>` with+the explicit files for that PR; do not use a broad directory when the directory also+contains sidecar-owned or shared files:++```bash+git fetch origin+git switch -c codex/conv2-0N-<slug> origin/main+# Preserve the final #2161 add/modify/delete/rename intent and three-way it+# against the current main tree.+SNAPSHOT_BASE=$(git merge-base origin/main codex/link-e2e-videos)+git diff --binary --find-renames "$SNAPSHOT_BASE" codex/link-e2e-videos -- \+ <owned paths> | git apply --3way --index+# shared-file hunks (see Section 4): hand-edit, do NOT checkout the whole file+cd libs/cua-driver/rust && cargo check --workspace # regenerates Cargo.lock honestly+```++Common fresh gates (every PR that touches `libs/cua-driver/rust`):++```bash+cargo fmt --all --check+cargo clippy --workspace --all-targets -- -D warnings+cargo test --workspace # host-runnable subset+cargo test --workspace --no-run # proves e2e/platform tests still compile+```++---++### PR 1 - `codex/conv2-01-typed-evidence` - "test(cua-driver): typed e2e evidence foundation (conv2 1/7)"++**Purpose:** land the typed testkit (journal/observer/sentinel/e2e result types), the+`cua-e2e-report` binary, core recording rework, the shared scenario catalog and+cross-platform fixture updates, and the evidence-linking script. Everything later PRs+compile against.++**Owns (final state):**++- `libs/cua-driver/rust/crates/cua-driver-testkit/**` - whole crate: `Cargo.toml`,+ `src/lib.rs`, `src/bin/cua-e2e-report.rs` (A), `driver.rs`, `e2e.rs` (A),+ `journal.rs` (A), `mcp.rs`, `observer.rs` (A), `paths.rs`, `reaper.rs`,+ `response.rs`, `sentinel.rs` (A), `windows_setup.rs` (A - cfg-gated, ships here so+ `lib.rs` can be final-state)+- `crates/cua-driver-core/src/`: `cdp.rs`, `recording.rs`, `recording_tools.rs`, `tool.rs`+- `crates/cua-driver/src/`: `main.rs`, `serve.rs`; `crates/cua-driver/tests/README.md`+- Cross-platform tests: `capture_contract_test.rs` (rename of+ `modality_capture_mode_test.rs`), `cross_platform_behavior_test.rs`,+ `e2e_environment_preflight_test.rs` (A), `protocol_schema_test.rs`+- **Deletes** superseded cross-platform modality tests (they consume the pre-typed+ testkit API and won't compile after this PR): `modality_background_test.rs`,+ `modality_desktop_scope_test.rs`, `modality_dispatch_test.rs`,+ `modality_focus_test.rs`, `modality_input_e2e_test.rs`+- Shared fixtures: `tests/fixtures/shared/scenarios.json`,+ `tests/fixtures/shared/web/index.html`, electron `build.sh`, `main.js`,+ `preload.js` (A), `tauri/src-tauri/src/main.rs`+- `scripts/ci/link-e2e-evidence.sh` (A)+- Dev tooling: `libs/cua-driver/scripts/_install-local-rust.sh`, `sync-vm-worktree.sh`+- `rust/Skills/cua-driver/RECORDING.md` (documents recording behavior changed here)+- `Cargo.lock` (regenerated via `cargo check`, not copied)++**Excludes:** all platform crate changes, all workflows, all `libs/cua-driver/docs/**`,+electron `build.ps1` / `launcher-linux.sh` (platform PRs).++**Dependencies:** none (base = `main`).++**Fresh gates:**++- [ ] Common cargo gates (above), on Linux CI at minimum; `cargo test --no-run` on a+ Windows and macOS host or cross-check to prove cfg-gated testkit compiles+- [ ] `shellcheck scripts/ci/link-e2e-evidence.sh libs/cua-driver/scripts/*.sh`+- [ ] `cargo run -p cua-driver-testkit --bin cua-e2e-report -- --help` (or its+ self-test invocation) exits 0+- [ ] Existing `ci-rust-*` workflows green on the PR (unchanged workflow files must+ still pass against the new testkit)++**Historical evidence:** #2161 CI runs; `e2e-convergence-journal.md` (lands in PR 7 -+link to the branch copy).++**Rollback boundary:** revert restores the pre-typed testkit and the five deleted+modality tests; nothing on `main` yet depends on the new API. Must be reverted only+if PRs 2-4 haven't merged (they consume `e2e::*`/`journal::*`).++---++### PR 2 - `codex/conv2-02-windows` - "fix(cua-driver): Windows e2e convergence (conv2 2/7)"++**Purpose:** Windows platform fixes (input injection, WGC capture, UIA cache,+recording hooks), typed Windows e2e tests, sandbox/runner + CI wiring; retire+`focus-monitor-win`.++**Owns:**++- `crates/platform-windows/src/`: `capture.rs`, `input/{delivery,inject,keyboard,mod,mouse}.rs`,+ `recording_hooks.rs`, `tools/impl_.rs`, `uia/{cache,windows_enum}.rs`, `wgc.rs`,+ `win32/windows.rs`+- **Deletes** `crates/focus-monitor-win/{Cargo.toml,src/main.rs}`; owns the one-line+ member removal in workspace `rust/Cargo.toml` (hand-edit, see Section 4)+- Tests: `agent_cursor_windows_test.rs` (A), `desktop_scope_windows_test.rs` (A),+ `launch_windows_test.rs` (A), `harness_winui3_test.rs`, `harness_wpf_test.rs`;+ `harness_web_test.rs` (Windows-only WPF + WebView2/Electron CDP coverage);+ **deletes** `guard_ux_test.rs` (superseded by the typed agent-cursor/launch suites)+- Fixtures: `windows/webview2/MainWindow.xaml{,.cs}`, `windows/wpf/MainWindow.xaml{,.cs}`,+ `fixtures/build/windows.ps1`, electron `build.ps1`+- Runners: `tests/runners/windows-sandbox/{run-tests-in-sandbox,sandbox-runner}.ps1`,+ `tests/runners/windows/{README.md,run-all.ps1}`+- CI: `scripts/ci/windows/{build-harnesses,run-rust-e2e}.ps1`,+ `.github/workflows/{e2e-rust-windows.yml,ci-rust-windows.yml}`+- `Cargo.lock` regen++**Excludes:** testkit sources (PR 1), all docs.++**Dependencies:** PR 1 (typed testkit API, `windows_setup` module).++**Fresh gates:**++- [ ] Common cargo gates on a Windows runner (`cargo test -p platform-windows`,+ `cargo test --no-run` for the e2e suites)+- [ ] `workflow_dispatch` of `e2e-rust-windows.yml` **on this branch head** -> green,+ with `cua-e2e-report` summary artifact and evidence links present+- [ ] `ci-rust-windows.yml` green on the PR+- [ ] Grep proves no residual `focus-monitor-win` references:+ `git grep -n focus-monitor-win` -> empty++**Historical evidence:** #2161 `e2e-rust-windows` runs; `windows-hosted-e2e` /+`windows-px-background-clicks` branch history. Context only.++**Rollback boundary:** revert restores old Windows behavior + `guard_ux_test.rs` ++`focus-monitor-win`; independent of PRs 3-6. Safe to revert alone before PR 7 (whose+docs describe the new Windows surface).++---++### PR 3 - `codex/conv2-03-macos` - "fix(cua-driver): macOS e2e convergence (conv2 3/7)"++**Purpose:** macOS AX/input/tool fixes, typed macOS suites (desktop scope, installed+app launch, TextEdit), fixture and smoke updates, hosted-runner e2e script.++**Owns:**++- `crates/platform-macos/src/`: `ax/bindings.rs`, `input/mouse.rs`,+ `tools/{click,drag,mod,scroll,type_text}.rs`, `window_change_detector.rs`+- Tests: renames `modality_desktop_scope_macos_test.rs -> desktop_scope_macos_test.rs`+ and `modality_launch_focus_macos_test.rs -> installed_app_launch_macos_test.rs`;+ `installed_app_textedit_macos_test.rs` (A), `harness_appkit_test.rs`,+ `harness_swiftui_test.rs`+- Fixtures: `macos/appkit/main.swift`, `macos/swiftui/main.swift`,+ `macos/wkwebview/main.swift`, `fixtures/build/macos.sh`, `fixtures/smoke/macos.sh`+- `scripts/ci/macos/run-rust-e2e.sh` (A)+- `Cargo.lock` regen (if any)++**Excludes:** testkit, docs, everything Windows/Linux. Note: there is no macOS e2e+GitHub workflow in the delta - the script is invoked on self-hosted/local runners.++**Dependencies:** PR 1 only. Parallel with PR 2 and PR 4.++**Fresh gates:**++- [ ] Common cargo gates on a macOS host+- [ ] `bash libs/cua-driver/tests/fixtures/build/macos.sh` ++ `bash libs/cua-driver/tests/fixtures/smoke/macos.sh` pass locally+- [ ] `bash scripts/ci/macos/run-rust-e2e.sh` full run on a macOS host with+ Accessibility/Screen Recording grants -> summary posted to the PR (paste the+ `cua-e2e-report` output; this is the fresh gate standing in for hosted CI)++**Historical evidence:** convergence-journal macOS entries; #2161 local run logs.++**Rollback boundary:** revert restores old macOS tool behavior and the `modality_*`+test names. Independent of PRs 2/4-6.++---++### PR 4 - `codex/conv2-04-linux-x11` - "fix(cua-driver): Linux core and X11 e2e convergence (conv2 4/7)"++**Purpose:** Linux platform core (AT-SPI, input delivery, overlay, health report,+recording hooks), X11 path, typed Linux desktop-scope suite, X11 e2e scripts and+workflows; retire the interactive-Linux workflow.++**Owns:**++- `crates/platform-linux/src/`: `a11y.rs`, `atspi/{mod,native}.rs`, `health_report.rs`,+ `input/{delivery,mod}.rs`, `overlay.rs`, `recording_hooks.rs` (A),+ `tools/{impl_,stubs}.rs`, `x11/mod.rs`+- `platform-linux/src/lib.rs` - **hunk only**: `pub mod recording_hooks;` (see Section 4)+- Tests: `desktop_scope_linux_test.rs` (A), `harness_gtk3_test.rs`; **deletes**+ `modality_desktop_scope_linux_test.rs`, `modality_dispatch_linux_test.rs`+- Fixtures: `linux/gtk3/main.py`, `fixtures/build/linux.sh`,+ electron `launcher-linux.sh` (A)+- CI: `scripts/ci/linux/run-rust-e2e.sh`, `run-rust-e2e-desktop.sh` (A),+ `.github/workflows/{e2e-rust-linux.yml,ci-rust-linux.yml}`;+ **deletes** `.github/workflows/ci-cua-driver-interactive-linux.yml`+- `rust/Skills/cua-driver/LINUX.md`+- `Cargo.lock` regen++**Excludes:** everything under `platform-linux/src/wayland/`, `video_wayland.rs`,+`platform-linux/Cargo.toml` (all PR 5).++**Compile caveat:** if `recording_hooks.rs` in final state imports `video_wayland` or+`wayland::sway_ipc`, `cargo check` fails here - in that case move the offending file(s)+to PR 5 and note the move in both PR bodies. The `--no-run` gate catches this.++**Dependencies:** PR 1. Parallel with PRs 2-3.++**Fresh gates:**++- [ ] Common cargo gates on Linux+- [ ] `workflow_dispatch` of `e2e-rust-linux.yml` on this branch head -> green X11+ canonical matrix, evidence artifacts present+- [ ] `ci-rust-linux.yml` green on the PR+- [ ] `shellcheck scripts/ci/linux/run-rust-e2e*.sh`+- [ ] Confirm nothing still references the deleted interactive workflow:+ `git grep -n ci-cua-driver-interactive-linux` -> empty++**Historical evidence:** "record Linux completion evidence" commits; journal entries.++**Rollback boundary:** revert restores old Linux core + interactive workflow. Must+revert PR 5 first if PR 5 has merged (Wayland modules reference core changes).++---++### PR 5 - `codex/conv2-05-wayland` - "fix(cua-driver): Wayland backend convergence (conv2 5/7)"++**Purpose:** Wayland backend work (ext-toplevel, sway IPC, portal screenshot,+persistent virtual pointer, libei, and the shared nested-injection protocol code),+Wayland video, the `portal-libei -> portal-input + portal-capture` feature split,+Wayland e2e workflow/script, GNOME helper extension, and shipping `portal-input` in+release binaries. The nested route remains dormant/experimental until PR 6 packages+and validates its owned compositor environment.++**Owns:**++- `crates/platform-linux/src/wayland/` - everything: `ext_toplevel.rs` (A),+ `libei.rs`, `mod.rs`, `persistent_vptr.rs`, `portal_screenshot.rs`,+ `shell_helper.rs`, `sway_ipc.rs` (A)+- `platform-linux/src/video_wayland.rs` (A); `platform-linux/src/lib.rs` -+ **hunk only**: `pub mod video_wayland;`+- Manifests (whole-file final state - the delta in each is entirely this PR's):+ `crates/platform-linux/Cargo.toml` (feature split), `crates/cua-driver/Cargo.toml`+ (feature forwarding)+- CI: `scripts/ci/linux/run-rust-e2e-wayland.sh` (A),+ `.github/workflows/e2e-rust-linux-wayland.yml`,+ `.github/workflows/cd-rust-cua-driver.yml` (release builds gain+ `--features portal-input` + `libxkbcommon-dev`)+- `libs/cua-driver/wayland-helper/{README.md,winrects@cua/extension.js,winrects@cua/metadata.json}`+- `Cargo.lock` regen++**Excludes:** X11/core files (PR 4), Nix (PR 6).++**Dependencies:** PR 4 (shares `platform-linux`; core/input changes land first).++**Fresh gates:**++- [ ] Common cargo gates, plus feature-matrix compiles:+ `cargo check -p platform-linux`, `... --features portal-input`,+ `... --features portal-capture`, `... --features portal-libei`+- [ ] `workflow_dispatch` of `e2e-rust-linux-wayland.yml` on this branch head with+ environment `sway` -> green canonical matrix+- [ ] `cd-rust-cua-driver.yml` build job dry-run (workflow_dispatch or `act`/container+ rehearsal) proves the Debian 11 container builds with `--features portal-input`+- [ ] `.github/scripts/tests/` release-wiring tests still pass **unmodified**+ (`python -m unittest` - the reminder-text update itself lands in PR 7)++**Historical evidence:** `wayland-e2e-dbus`/`wayland-e2e-dispatch` branch runs;+journal Wayland entries.++**Rollback boundary:** revert restores `portal-libei`-only gating and the old release+build. Must revert PR 6 first if merged (Nix consumes the new feature names).++---++### PR 6 - `codex/conv2-06-nix` - "ci(cua-driver): nested compositor Nix consolidation (conv2 6/7)"++**Purpose:** consolidate the Nix story - rewritten `flake.nix` and+`nix/cua-driver/package.nix` (new feature names), nested-compositor patch, delete the+21 superseded `.nix` test files and the three standalone nix workflows in favor of+`ci-nix-linux.yml`.++**Owns:**++- `flake.nix`, `nix/cua-driver/{README.md,package.nix,compositor/cua_compositor_patch.py,tests/README.md}`+- `scripts/ci/linux/run-rust-e2e-inject.sh` (A), which activates the nested+ environment packaged by this PR+- **Deletes** `nix/cua-driver/tests/*.nix` (integration, linux-background-gui,+ linux-background-terminal-gif, linux-cursor-click-gif, linux-parallel-drag-gif,+ linux-parallel-drag-xserver, openbox-rc, record-x11-gif, screenshot, set-config)+ and `nix/cua-driver/tests/wayland/**` (README.md + 10 `.nix` files)+- CI: `.github/workflows/ci-nix-linux.yml`; **deletes**+ `.github/workflows/{nix-build.yml,nix-screenshot.yml,nix-wayland.yml}`++**Excludes:** all Rust sources (PRs 1-5), docs.++**Dependencies:** PR 5 (`package.nix` builds with the split features).++**Fresh gates:**++- [ ] `nix flake check` and `nix build .#cua-driver` (or the flake's package attr)+ succeed locally/CI+- [ ] `ci-nix-linux.yml` green on this branch head (workflow_dispatch), covering what+ the three deleted workflows covered+- [ ] Dispatch `e2e-rust-linux-wayland.yml` with the nested-compositor environment on+ this branch head. Native and capture owners must remain green; any experimental+ shared-row failures must match the named PR description exactly, with no new+ setup, protocol, or evidence regression. This run is evidence, not a promotion+ of the experimental lane.+- [ ] `git grep -nE 'nix-(build|screenshot|wayland)\.yml'` -> empty (no dangling refs)++**Historical evidence:** nested-compositor GIF/screenshot runs recorded in the journal.++**Rollback boundary:** revert restores the old nix tests/workflows; independent of+Rust code paths (Nix is packaging/CI only). Safe to revert alone.++---++### PR 7 - `codex/conv2-07-docs-cleanup` - "docs(cua-driver): convergence docs and replacement cleanup (conv2 7/7)"++**Purpose:** land all documentation describing the converged state, delete the+superseded recording/fixture corpora that the typed evidence pipeline replaces, and+wire the release-reminder validation checklist. **Resolves the sole `main` conflict.**++**Owns:**++- `docs/content/docs/concepts/`: `how-cua-driver-is-validated.mdx` (A), `index.mdx`;+ `meta.json` - **hand-merge, never final-state** (see Section 4): insert the+ `how-cua-driver-is-validated` entry into _current main's_ `meta.json`+- `docs/content/docs/reference/cua-driver/`: `development.mdx`, `limits.mdx`,+ `mcp-tools.mdx`, `meta.json`, `platform-roadmap.mdx` (A), `platform-support.mdx` (A)+- `libs/cua-driver/docs/`: adds `action-support.md`, `e2e-convergence-journal.md`,+ `e2e-results-summary-design.md`, `linux-desktop-validation.md`,+ `linux-support-completion-plan.md`, `test-harness-convergence-plan.md`,+ `test-harnesses-guide.md`, `test-matrix.md`; modifies `e2e-ci-reporting.md`;+ **deletes** `cross-platform-ci-test-plan.md`, `pr-split-plan.md`; includes this+ file (`pr-2161-landing-plan.md`) for the record+- Root docs: `README.md`, `CONTRIBUTING.md`, `Development.md`, `TESTING.md`,+ `libs/cua-driver/README.md`, `libs/cua-driver/rust/README.md`+- Replacement cleanup - **deletes whole directories**:+ `libs/cua-driver/tests/fixtures/linux-container/**` (5 files),+ `libs/cua-driver/tests/fixtures/modality-recordings/**` (16 files),+ `libs/cua-driver/tests/fixtures/vision-agent-test/**` (2 files);+ modifies `tests/fixtures/README.md` to match+- `scripts/ci/README.md` (final state - documents scripts landed in PRs 1-5)+- Release wiring: `.github/workflows/ci-release-reminder.yml` ++ `.github/scripts/tests/test_cua_driver_release_wiring.py` (checklist links the+ workflows finalized in PRs 4-5)++**Initial transfer excludes:** the snapshot's Lume generated docs (Section 2.2),+even though they sit next to owned docs paths. The final post-rebase generator gate+owns any current-source regeneration as an explicit adaptation.++**Dependencies:** PRs 2-6 all merged (docs describe every platform; cleanup deletes+corpora only once every replacement suite is live; reminder links final workflows).++**Fresh gates:**++- [ ] Docs site builds: `pnpm install && pnpm build` in `docs/` (or the repo's docs+ build command) with zero broken-link errors for the new pages+- [ ] `python -m unittest .github/scripts/tests/test_cua_driver_release_wiring.py` passes+- [ ] `git grep -n 'modality-recordings\|linux-container\|vision-agent-test' -- ':!libs/cua-driver/docs'`+ -> empty (nothing references deleted fixtures)+- [ ] Rendered-page spot check of `platform-support.mdx` / `platform-roadmap.mdx` /+ `how-cua-driver-is-validated.mdx`++**Historical evidence:** n/a - docs are the evidence record itself; verify links resolve.++**Rollback boundary:** pure docs + inert fixture deletions + reminder text; safe to+revert alone at any time.++---++## 4. Shared-file overlap and conflict handling++Disjoint path ownership eliminates most conflicts. The exceptions, with their rule:++| Shared file | Touched by | Rule |+| ------------------------------------------------------------------ | ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| `rust/Cargo.lock` | PRs 1, 2, 4, 5 (+3 if deps shift) | **Never final-state-copied.** Each PR hand-edits its manifests and regenerates via `cargo check --workspace`. Parity audit compares it semantically (`cargo tree`), not byte-wise. |+| `rust/Cargo.toml` (workspace) | PR 2 only | Whole delta is the `focus-monitor-win` member removal -> PR 2 hand-edits that one line. |+| `crates/cua-driver-testkit/**` | PR 1 only | Entire crate (incl. cfg-gated `windows_setup.rs`) lands in PR 1 so `lib.rs` is final-state and compiles everywhere. |+| `crates/platform-linux/src/lib.rs` | PRs 4, 5 | Hunk-split: PR 4 adds `pub mod recording_hooks;`, PR 5 adds `pub mod video_wayland;`. Neither checks out the whole file. |+| `crates/platform-linux/Cargo.toml`, `crates/cua-driver/Cargo.toml` | PR 5 only | Verified: the entire delta in both is the portal feature split -> PR 5 takes final state. |+| `platform-linux/src/wayland/mod.rs` | PR 5 only | All Wayland module wiring (incl. `sway_ipc`, `ext_toplevel`, feature-gate renames) is PR 5's. |+| Shared scenario catalog `tests/fixtures/shared/scenarios.json` | PR 1 only | Final-state superset lands in PR 1; entries for platform suites not yet landed are inert data. Platform PRs must not touch it - if a platform needs a catalog fix, it amends PR 1 before landing, not its own PR. |+| `tests/fixtures/README.md` | PR 7 only | Final state; documents both adds (PRs 1-5) and deletions (PR 7). Acceptable to be momentarily ahead/behind between merges. |+| `scripts/ci/README.md` | PR 7 only | Same rule as above. |+| Workflows | one owner each (table in Section 2.1 / per-PR lists) | A workflow file never spans two PRs. `cd-rust-cua-driver.yml` -> PR 5; `ci-release-reminder.yml` -> PR 7 (its wiring test rides with it). |+| `docs/content/docs/concepts/meta.json` | PR 7 only | **The only `main` conflict.** Hand-merge: apply the #2161 _intent_ (add the new page entry) onto current main's file; do not `git checkout` the branch version. |++Conflict-with-main policy for the whole stack: because every branch is cut fresh from+current `origin/main` and transfers final state per path, the only possible conflicts+are files that changed on `main` after #2161 diverged. Before each transfer, run+`git diff origin/main...codex/link-e2e-videos --name-only -- <owned paths>` vs+`git log origin/main --since=<branch-point> -- <owned paths>`; any path with fresh+`main` history gets the `meta.json` treatment (hand-merge, note in PR body).++## 5. Dependency graph++```mermaid+graph TD+ PR1["PR 1: typed evidence foundation"]+ PR2["PR 2: Windows"]+ PR3["PR 3: macOS"]+ PR4["PR 4: Linux core / X11"]+ PR5["PR 5: Wayland backends"]+ PR6["PR 6: nested compositor / Nix"]+ PR7["PR 7: docs + replacement cleanup"]+ SA["Sidecar A: Docker cleanup"]+ SB["Sidecar B: TypeScript tooling"]+ PR1 --> PR2+ PR1 --> PR3+ PR1 --> PR4+ PR4 --> PR5+ PR5 --> PR6+ PR2 --> PR7+ PR3 --> PR7+ PR6 --> PR7+```++PRs 2, 3, 4 may be prepared and reviewed in parallel once PR 1 merges. PRs 5-7 are+strictly sequential behind their parents. Sidecars A and B are independent and may+land at any point before #2161 closes.++## 6. Sequential landing procedure++For each PR, in dependency order:++- [ ] Cut `codex/conv2-0N-<slug>` from **current** `origin/main` (never from a sibling+ stack branch) and run the transfer recipe (Section 3)+- [ ] Run the sidecar-separation grep (Section 2.2) -> empty+- [ ] Run the PR's fresh gates; attach workflow-run links / command output to the PR body+- [ ] PR body: purpose, owned-path list, link to #2161 + this plan, historical-evidence+ links clearly labeled "historical - superseded by fresh runs above"+- [ ] Review -> squash-merge+- [ ] After merge: the next PR in line re-cuts from the new `main` (re-running the+ final-state transfer is cheap and conflict-free by construction - prefer re-cut+ over rebase)+- [ ] Post a progress comment on #2161: "conv2 N/7 landed as #NNNN"++If a fresh gate fails on a stack PR: fix **on that stack branch** (these are real+review findings - the point of the split); if the fix belongs to an earlier layer,+land a follow-up to that layer first. Never "fix" a Cua Driver PR by pulling in a+sidecar-owned file.++## 7. Final allowlisted tree-parity audit++After PR 7 merges, from an up-to-date checkout:++```bash+git fetch origin+# 1) Allowlisted parity: for every Section 2.1 path, main must equal the #2161 tree.+git diff origin/main codex/link-e2e-videos -- \+ libs/cua-driver nix/cua-driver flake.nix scripts/ci \+ .github/workflows/e2e-rust-windows.yml .github/workflows/e2e-rust-linux.yml \+ .github/workflows/e2e-rust-linux-wayland.yml .github/workflows/ci-rust-linux.yml \+ .github/workflows/ci-rust-windows.yml .github/workflows/ci-nix-linux.yml \+ .github/workflows/cd-rust-cua-driver.yml .github/workflows/ci-release-reminder.yml \+ .github/scripts/tests/test_cua_driver_release_wiring.py \+ docs/content/docs/concepts docs/content/docs/reference/cua-driver \+ README.md CONTRIBUTING.md Development.md TESTING.md+```++Expected residual diff - exactly three explainable classes, anything else is a missed+file and blocks closing #2161. Maintain an **adaptation ledger** in the final #2161+comment: one row per intentionally divergent path, with the stack PR/commit and exact+reason. A generic "main advanced" exception is not acceptable.++1. `libs/cua-driver/rust/Cargo.lock` - regenerated, semantically compared via+ `cargo tree --workspace` on both trees+2. `docs/content/docs/concepts/meta.json` - hand-merged against newer `main`+3. Files intentionally adapted while landing against newer `main`, each listed in the+ adaptation ledger with its owning stack PR and commit++```bash+# 2) Preservation parity: after both sidecars merge, their paths should match+# the intended #2161 state unless the adaptation ledger says otherwise.+git diff --name-only origin/main codex/link-e2e-videos -- \+ Dockerfile .dockerignore package-lock.json .pre-commit-config.yaml \+ libs/typescript .github/workflows/ci-lint-typescript.yml \+ .github/workflows/claude-auto-fix.yml+# expected: empty after Sidecars A and B merge, except ledgered adaptations.++# The Lume references are intentionally different: main must retain the+# current-source 0.3.15 output rather than the snapshot's 0.3.14 bytes.+rg -n 'Version: 0\.3\.15|Documented against Lume \*\*0\.3\.15\*\*' \+ docs/content/docs/reference/lume/{cli-reference,http-api}.mdx++# 3) Coverage invariant: every delta file is stack-owned, sidecar-owned, or+# superseded by the two source-generated Lume files.+git diff --name-only origin/main...codex/link-e2e-videos | sort > /tmp/delta.txt+# compare against the union of Section 3 ownership lists + Section 2.2 - must match 1:1.+```++Record the audit output as a final comment on #2161.++## 8. Disposition of PR #2161 and its branch++**Link:** each stack PR body carries `Part of the #2161 second-wave landing+(conv2 N/7, plan: libs/cua-driver/docs/pr-2161-landing-plan.md)`. #2161's description+gets a checklist of the seven PRs, updated as they merge.++**Close #2161 when all of:**++- [ ] All replacement convergence PRs in the landing-outcome table merged to+ `main`+- [ ] Sidecars #2177 and #2178 merged to `main`+- [ ] Section 7 parity audit run and posted, residual diff limited to the three explained classes+- [ ] The two Lume paths are recorded as source-generated 0.3.15 adaptations+- [ ] Closing comment maps every #2161 theme -> landing PR, sidecar PR, or+ superseded-by-main record++Close as **closed, not merged**.++**Delete `codex/link-e2e-videos` when all of:**++- [ ] #2161 closed for at least 30 days (not before 2026-08-11) with no parity regressions reported+- [ ] The first post-landing cua-driver release (per the `ci-release-reminder`+ validation checklist) has shipped green on the converged workflows+- [ ] A final `git diff` against `main` over the allowlist is re-run and still clean -+ confirming the branch holds nothing unlanded outside the two explicitly+ generator-adapted Lume files++Until then the branch stays as the forensic reference for the parity audit and the+historical-evidence links in the stack PRs.diff --git a/libs/cua-driver/docs/pr-split-plan.md b/libs/cua-driver/docs/pr-split-plan.mddeleted file mode 100644index 9293a3ad3e..0000000000--- a/libs/cua-driver/docs/pr-split-plan.md+++ /dev/null@@ -1,297 +0,0 @@-# Cua-driver PR Split Plan--Status: draft for review--## Decision--Do not merge PR #2135 as one change. It currently contains 204 files and-roughly 11.5k additions/deletions across fixture migration, Rust test-ownership, Windows behavior, Linux/macOS behavior, and CI/Nix architecture.--Keep the current branch as an untouched snapshot while smaller PRs are built-from `main`. The historical commits are useful provenance, but several of them-mix unrelated file families. Cherry-picking those commits wholesale would-reintroduce the same review problem.--## Target PRs--### PR 0: Documentation lockfile housekeeping--Suggested title:--```text-chore(docs): sync pnpm lockfile with pnpm 9-```--Scope:--- `docs/pnpm-lock.yaml`-- Current provenance: `8fee84dd`--This is independent and can merge first. It should not be coupled to desktop-behavior or CI changes.--Acceptance:--- Docs dependency install/check passes.-- No `libs/cua-driver` files change.--### PR 1: Repo-local Rust harness foundation--Suggested title:--```text-test(cua-driver): make repo-local Rust harnesses canonical-```--Purpose: establish where fixtures, Rust integration tests, and runner scripts-live. This PR should make the source tree coherent without changing the-platform input algorithms.--Scope by file family:--- Move `libs/cua-driver/test-harness` content to- `libs/cua-driver/tests/fixtures`.-- Add the repo-local Electron and Tauri source/build scripts for each host.-- Move the Windows sandbox runner to `tests/runners/windows-sandbox`.-- Add the Rust-only Windows `tests/runners/windows/run-all.ps1` entrypoint.-- Remove the old Python e2e harness and downloaded desktop binaries.-- Port the optional LibreOffice and macOS launch-focus tests to Rust where- needed, keeping them outside the default run-all path.-- Update path references, `.gitignore` files, fixture READMEs, Rust test README,- and the small contributor-facing Fumadocs page.-- Include `libs/cua-driver/scripts/sync-vm-worktree.sh` because it belongs to- the host-owned fixture workflow.--Historical provenance:--- `811ffa6d`, `d44971c`, `98e59d75`, `d9e4311f`, `9ab85981`,- `5e8759e1`, and `4e1636b9`.--Do not take the platform implementation portions of `a759840b` or-`d01c7a14` into this PR. If a test path needs a behavior change to compile,-make the smallest path-only adjustment and leave the behavioral assertion-change for the relevant behavior PR.--Acceptance:--- No old downloaded Electron/Tauri application is tracked.-- No Python behavioral suite remains under the cua-driver test tree.-- Fixture build scripts pass shell/PowerShell parsing on their native hosts.-- `cargo test -p cua-driver --tests --no-run` passes on macOS, Linux, and- Windows.-- Staged outputs remain under ignored `rust/test-apps`.-- Optional external-app tests are visibly excluded from the default runner.--### PR 2: Shared Rust behavioral matrix--Suggested title:--```text-test(cua-driver): add shared external-oracle behavior matrix-```--Purpose: make the trusted user-behavior scenarios explicit before wiring them-into expensive CI. The same scenario IDs, fixture DOM, delivery modes, and-external-state oracles should be used on every supported host.--Scope:--- Add `cross_platform_behavior_test.rs`.-- Add/update the shared web fixture and scenario catalog.-- Add only the generic testkit response/tree helpers required by the matrix.-- Keep the matrix ignored by default and runnable through one Rust command.-- Keep assertions based on fresh application state, not driver success alone.--Historical provenance: primarily `ca60e0ba`, but extract only the matrix,-fixture, and generic testkit pieces. The platform algorithm changes from that-commit belong in PRs 3 and 4.--Acceptance:--- Matrix compiles on all three hosts.-- Electron and Tauri are selected from repo-local source-built fixtures.-- Every scenario has an external oracle and a documented outcome rule.-- Missing fixtures are visible as a runner/setup failure on dedicated e2e- lanes, not a green silent skip.--### PR 3: Windows delivery and interactive UX fixes--Suggested title:--```text-fix(cua-driver): harden Windows delivery and GUI validation-```--Scope:--- Windows Session 0 and interactive-desktop diagnostics.-- Focus sentinel and guard UX behavior.-- Background keyboard/click refusal reporting and preserved actuator causes.-- UIA focus, scroll, WebView2/Tauri, WPF, and WinUI3 behavior changes.-- Windows-specific test assertions and fixture details that validate those- behaviors.--Historical provenance: Windows portions of `a759840b`, `98e59d75`,-`d01c7a14`, and `ca60e0ba`.--Acceptance:--- Windows unit and compile checks pass on `windows-latest`.-- SSH/Session 0 runs skip GUI-only assertions with an explicit reason.-- An active user-session run passes the guard and native harness suites.-- W1-style background input never reports verified delivery without an- external state change.-- W8-style unsupported background delivery returns a structured refusal with- the concrete cause.--### PR 4: Linux delivery fixes--Suggested title:--```text-fix(cua-driver): harden Linux AT-SPI and input delivery-```--Scope:--- Linux AT-SPI reference handling, action delivery, input capability errors,- and related external-oracle test updates.-- Linux focused tests and strict known-gap assertions.--Historical provenance: Linux portions of `a759840b`, `ca60e0ba`, and-`d848e365`. The VM sync helper from `d848e365` belongs in PR 1, not here.--Acceptance:--- Linux unit and interactive Xvfb/AT-SPI checks pass.-- Known gaps remain strict and visible; no assertion is weakened to fit the- current driver behavior.-- Linux behavior changes are covered by the shared matrix where the platform- supports the scenario.--### PR 5: macOS input and capture delivery--Suggested title:--```text-fix(cua-driver): harden macOS input and capture delivery-```--Scope:--- macOS background click, drag, targeted wheel, and accessibility scrolling.-- macOS Swift runtime/build fixes required for Rust test binaries.-- macOS focused tests and strict nested-scroll known-gap assertions.--Historical provenance: macOS portions of `a759840b`, `ca60e0ba`, and-`9c3c719a`, plus the macOS portions of `0447bd52`.--Acceptance:--- macOS default tests and installed-daemon harness checks pass with TCC- already authorized.-- Electron and Tauri click/drag rows use external application-state oracles.-- The nested web scrolling gap remains strict and visible.--### PR 6: OS-scoped CI and Nix e2e architecture--Suggested title:--```text-ci(cua-driver): add OS-scoped Rust and manual e2e lanes-```--Scope:--- Testkit path overrides and their unit test:- `CUA_TEST_DRIVER_BIN`, `CUA_TEST_APPS_ROOT`, and- `CUA_TEST_WORKSPACE_ROOT`.-- Strict fixture preflight for canonical e2e runs.-- Linux and Windows OS-scoped unit workflows.-- Maintainer-dispatched Linux and Windows interactive e2e workflows.-- Source-built Linux Rust check in Nix.-- Nix and workflow READMEs describing canonical versus supporting coverage.-- Move the legacy GUI/toolkit and compositor matrices to maintainer dispatch;- preserve them as supporting diagnostics.-- Keep new canonical Linux tests free of GIF requirements.--Historical provenance: `eef79f2a` and `0447bd52`, after extracting the Linux-and macOS driver edits from `0447bd52` into PRs 4 and 5.--Acceptance:--- Shared Rust changes trigger both OS unit workflows.-- Platform-specific changes trigger only the relevant OS unit workflow.-- Linux Rust source check passes in Nix.-- Manual Linux e2e builds repo-local fixtures from source.-- Windows e2e verifies an interactive user session and supports an Azure- active-RDP runner label.-- No VM or GitHub runner can push source changes.-- The PR description says clearly that fully sandboxed Nix Electron/Tauri- packaging remains a follow-up requiring committed dependency lockfiles.--## Dependency Graph--```mermaid-graph LR- P0[Docs lockfile] --> Merge- P1[Repo-local harness foundation] --> P2[Shared Rust behavior matrix]- P2 --> P3[Windows behavior fixes]- P2 --> P4[Linux behavior fixes]- P2 --> P5[macOS behavior fixes]- P3 --> P6[OS-scoped CI and Nix]-```--PR 0 is independent. PR 6 is based on PR 3 because its Windows unit workflow-compiles the Windows guard tests and therefore needs the Windows desktop-state-diagnostics. It can merge after PR 3; PRs 4 and 5 remain independent platform-slices and should merge before enabling the full cross-platform e2e gate.--## Branching Procedure--1. Freeze PR #2135. Do not add more mixed commits to it.-2. Preserve the current branch as the snapshot containing all work. The- private `cross-platform-fix-journal.md` stays untracked and must not be- included in any PR.-3. Create each split branch from `main`, then apply the final file families- from the snapshot. Use `git diff --find-renames main...HEAD -- <paths>` to- preserve moves and deletions.-4. Do not cherry-pick `a759840b`, `ca60e0ba`, `d848e365`, or `0447bd52`- wholesale. Each mixes concerns that belong in different PRs.-5. Cherry-picking `8fee84dd` is safe for PR 0. `eef79f2a` is safe only after- confirming the old interactive workflow remains compatible with the new CI- split.-6. Run the acceptance checks for each branch before opening its PR. Keep PR- descriptions narrowly scoped and list known failures instead of hiding- them behind broad “cross-platform fixes” language.-7. Open the PRs in dependency order. Stack PR 2 on PR 1 if needed, then- retarget it to `main` after PR 1 merges. Repeat for later PRs.-8. Close PR #2135 only after the smaller PRs are open and its body links to- them. Keep the snapshot branch available until all split PRs merge.--## Review and Rollback Rules--- A PR must have one primary reason to change and one obvious owner.-- Fixture moves must be reviewed with rename detection enabled.-- Deleting the Python suite and downloaded binaries is only part of PR 1 when- the Rust replacement and build scripts are present in the same diff.-- Platform behavior PRs must not change CI trigger policy.-- CI/Nix PRs must not contain opportunistic platform algorithm changes.-- If a split branch cannot compile independently, either add the missing- foundation to its parent PR or make the dependency explicit; do not silently- retain unrelated files from PR #2135.-- Revert the smallest PR that caused a regression. The dependency order keeps- the harness foundation and behavior fixes independently revertible from CI.--## Final State--After all PRs merge, the repository should have:--- Rust harnesses as the canonical behavioral source of truth.-- One shared scenario matrix with platform-specific implementations behind it.-- Cheap OS-scoped unit checks on normal PRs.-- Manual, real-session e2e gates for expensive desktop behavior.-- Nix retained as the Linux environment, with legacy visual checks clearly- labeled as supporting coverage.-- No opaque downloaded desktop fixture in the canonical test path.diff --git a/libs/cua-driver/docs/test-harness-convergence-plan.md b/libs/cua-driver/docs/test-harness-convergence-plan.mdnew file mode 100644index 0000000000..61206a69d9--- /dev/null+++ b/libs/cua-driver/docs/test-harness-convergence-plan.md@@ -0,0 +1,448 @@+# Rust E2E Harness Convergence Plan++**Historical plan.** Use `test-harnesses-guide.md`, `test-matrix.md`, and+`e2e-ci-reporting.md` for the implemented contributor workflow.++## Re-review Verdict++The original direction remains sound:++- Rust owns scenarios, assertions, and result records.+- Repo-local harness applications are the canonical E2E targets.+- AX/PX targeting and foreground/background delivery are dimensions on an+ action. They are not test families.+- Focus, z-order, cursor, and desktop checks are cross-cutting observations.+- The selector-free contributor invocation runs the complete matrix on every+ OS. CI may fan it into internal jobs for failure isolation.++This review changes five parts of the earlier plan:++1. Do not make a historical cell count a target. The shared matrix must cover+ supported route combinations, while every omission needs a route+ equivalence or unsupported-capability reason.+2. Separate test status from driver behavior. A test may pass because a+ required action was delivered or because a declared unsupported route was+ refused correctly. Those outcomes must remain distinct in reports.+3. Treat environment readiness as a lane preflight. A missing desktop, TCC+ grant, fixture, AT-SPI bus, recorder, or interactive Windows session must+ fail once before behavioral cells run.+4. Use a typed Rust case catalog as the matrix source. Do not add a second+ `matrix.yaml`, Python collector, or shell-owned scenario list.+5. Preserve one evidence bundle per cell, but reuse the driver and harness+ process where explicit state reset proves isolation.++The target is a smaller set of tests with named coverage reasons and strong+external oracles. Test count is not a success metric.++## Non-negotiable Rules++1. A successful tool response never proves delivery. A delivered action must+ change fixture or desktop state that the test reads independently.+2. A background action must also prove that it did not steal focus, raise the+ target, move the real cursor, or leak partial input when those invariants+ apply.+3. A refusal is valid only when the cell contract expects refusal, the driver+ returns an allowed structured refusal code, and the desktop observer sees no+ side effect.+4. A cell that requires delivery fails when the driver refuses, even if the+ refusal is honest.+5. A required canonical fixture or desktop capability cannot become a passing+ early return. Optional tests must be declared optional before execution.+6. Known gaps do not use `#[should_panic]` and do not count as green coverage.+ They run in a named optional lane with a linked issue until fixed.+7. Shell and PowerShell runners build the environment and collect artifacts.+ They do not infer behavioral results from Cargo output.+8. The Rust source build under test must be recorded in every run. macOS may+ proxy through the installed app bundle for TCC, but that bundle must come+ from the same source revision.++## Current Branch State++The branch implements the target ownership and reporting model. Platform+validation below replaces the older 60-cell proposal and records the remaining+environment or backend gaps separately.++### Implemented++- `cross_platform_behavior_test.rs` has one typed 36-cell catalog per shared+ harness application: 72 shared cells across Electron and Tauri per platform.+ It covers AX and PX with foreground and background delivery for click,+ text, keyboard, scroll, and child-window actions; PX drag and AX editor-save+ each cover both delivery modes.+- Declaration and result schema v2 record action, `targeting`, delivery, scope,+ backend route, expected and observed behavior, independent test status,+ required oracles, duration, and evidence.+- Background refusal matching uses the explicit structured set:+ `background_unavailable`, `background_occluded`, and+ `background_uipi_blocked`.+- Every catalog route is explicit for Win32, Quartz, X11, and Wayland. Windows+ Chromium PX background keeps its targeted-injection route metadata, while a+ fully occluded target expects the exact `background_occluded` refusal because+ temporarily raising the window would violate the desktop-side-effect contract.+- Every background shared cell attaches independent focus, z-order, real+ cursor, leaked-input, and fixture-state observations. Windows, macOS, and X11+ use direct testkit observers; an occluding Electron sentinel supplies the+ focus and leaked-input journal. Unsupported Wayland observations fail closed.+- Electron and Tauri use repo-local builds of the same shared web fixture.+- Windows, Linux, and macOS runners have strict source, fixture, AX, capture,+ and video preflights. Shell runners no longer synthesize behavioral rows.+- Rust validates duplicate, missing, contradictory, and evidence-less results+ before rendering the GitHub summary.+- Per-cell source-driver recording support exists in `cua-driver-testkit`.+- The typed catalog is the sole shared behavior owner; the three older shared+ tests and duplicate guard/modality targets were removed after their unique+ assertions moved to typed shared, native, launch, capture, cursor, and+ desktop-scope rows.+- WPF, WinUI3, WebView2, AppKit, SwiftUI, GTK3, capture, launch, cursor, and+ desktop-scope owners emit the same typed result records as the shared matrix.++### Execution record++The plan reached its ownership and reporting target on 2026-07-12. Current+support details remain in `action-support.md`; these landing checkpoints record+completion without turning this historical plan into a live status page.++- Windows run `29211506624` passed 114/114 rows at `6d7f02e4`.+- The logged-in macOS matrix passed 133/133 rows at `448d052f` with the stable+ installed-app TCC identity.+- Linux X11 run `29213365684` and Sway run `29213349962` each passed 108/108+ rows at `72083bb4`, with the same 71 deliveries and 37 exact refusals.+- The optional nested `cua-compositor` lane remains experimental and reports+ its typed failures rather than contributing stock-Wayland support claims.+- Some native targets still use fixed waits; shared and Windows web targets+ poll external state and allocate CDP ports per process.++## Target Test Model++### Typed case catalog++Every behavioral cell is declared in Rust with one structure shared by web and+native harness tests:++```rust+struct CaseSpec {+ id: &'static str,+ platform: Platform,+ display_server: DisplayServer,+ harness: Harness,+ action: Action,+ targeting: Targeting,+ delivery: Delivery,+ scope: Scope,+ expectation: ContractExpectation,+ oracles: &'static [OracleKind],+ route: DriverRoute,+}++enum ContractExpectation {+ Deliver,+ Refuse { allowed_codes: &'static [RefusalCode] },+}+```++`RefusalCode` is an enum, not a string-prefix check. Its initial Windows set is+`BackgroundUnavailable`, `BackgroundOccluded`, and `BackgroundUipiBlocked`;+Linux currently declares only `BackgroundUnavailable`. A cell lists the exact+codes allowed by its controlled setup.++`Targeting` uses `Ax`, `Px`, `Page`, or `NotApplicable`. Use `targeting` in the+schema instead of `capture_mode`; capture is a separate read contract.++`DriverRoute` names the implementation path that justifies coverage, such as+UIA Invoke, PostMessage, coordinate injection, CGEvent, AT-SPI action, libei,+or CDP. It is test metadata, not a request parameter.++The catalog is the machine-readable inventory. Contributor documentation and+the coverage table are generated from it or checked against it. There is no+second matrix file to keep in sync.++### Result record++Use one schema version and these independent fields:++| Field | Meaning |+| --- | --- |+| `cell_id` | Stable case id |+| `platform`, `display_server` | OS and Win32/Quartz/X11/Wayland environment |+| `harness`, `toolkit` | Electron, Tauri, WPF, WinUI3, WebView2, AppKit, SwiftUI, GTK3 |+| `action`, `targeting`, `delivery`, `scope` | Contract dimensions |+| `driver_route` | Backend path covered by the cell |+| `expected_behavior` | `DELIVER` or `REFUSE` |+| `test_status` | `PASS`, `FAIL`, `SKIP`, or `ENVIRONMENT_ERROR` |+| `observed_behavior` | `DELIVERED`, `REFUSED`, `NO_EFFECT`, `ERROR`, or `NOT_RUN` |+| `refusal_code` | Structured code when observed behavior is `REFUSED` |+| `oracles` | App state and attached desktop observations |+| `known_issue` | Optional issue id; it never changes a failure to a pass |+| `evidence` | Video, trajectory, screenshots, structured state, and log paths |++This removes the ambiguous `EXPECTED_REFUSAL` status. A refusal contract passes+only when `expected_behavior=REFUSE`, `observed_behavior=REFUSED`, the code is+allowed, and all no-side-effect oracles pass. Reports count delivered and+refused passes separately.++### Coverage selection++Do not restore the full Cartesian product automatically. Select cells by+driver route:++1. Every supported action has foreground and background coverage when the tool+ exposes both delivery modes.+2. Every distinct targeting path used by that action has at least one cell.+3. Every distinct OS backend route has at least one cell.+4. Renderer or toolkit duplication is kept only when it changes the route or+ has produced a real compatibility defect.+5. Every omitted combination names an `equivalent_to` cell or an unsupported+ contract reason.++The shared catalog currently declares 36 cells per harness application, or 72+cells across Electron and Tauri per platform. Add a missing combination when it+reaches a different driver route. Remove a combination only when another cell+proves the same route with an equal or stronger oracle.++## Cross-cutting Desktop Observer++Add one testkit interface that snapshots desktop state before and after an+operation:++```text+DesktopObservation+ foreground window+ target z-order and minimized state+ focus-change journal+ real cursor position+ optional leaked-input journal+```++The testkit `DesktopObserver` provides this interface with native Windows,+macOS, and Linux backends.++Attach the observer to:++- every background delivery cell;+- refusal cells;+- launch/minimize cells;+- screenshot and capture cells that promise no focus change;+- cursor evidence cells.++For successful background delivery, the cell requires both target-state change+and unchanged desktop invariants. For refusal, the cell requires no target+change and unchanged desktop invariants. A focus-only pass never proves input+delivery.++## Process And Evidence Lifecycle++Use these boundaries:++- one Rust source build per lane;+- one driver daemon or MCP process per lane;+- one harness process per harness group when the fixture has a verified reset+ operation;+- one recording session and result record per cell;+- a harness restart after crash, reset failure, or window identity change.++Each fixture reset must return a generation token. The next cell verifies the+new token and clean marker state before acting. Until a harness has this reset+contract, keep process-per-cell isolation.++Video remains required for every canonical E2E cell. Test video capture once in+the lane preflight. A recorder failure aborts the lane before the case catalog+runs, rather than generating the same permission failure for every cell.++## Environment Preflight++Each OS runner performs one preflight and emits one environment record.++Common checks:++- source revision and driver version match;+- required fixture binaries exist;+- the display/user session is interactive;+- the driver can list and inspect a preflight fixture;+- accessibility and capture permissions work;+- a short video starts, stops, and passes `ffprobe`;+- artifact directories are writable.++Platform checks:++| Platform | Required preflight |+| --- | --- |+| Windows | Non-Session-0 interactive desktop, input desktop, foreground sentinel, FFmpeg, UIA visibility |+| macOS | App-bundle daemon identity, live socket, Accessibility, Screen Recording, fixture window visibility |+| Linux X11 | X server, DBus, AT-SPI, window manager, capture, input backend |+| Linux Wayland | Compositor, DBus, AT-SPI, portal/capture path, libei or declared refusal path |++Canonical invocations set strict mode. Missing required capabilities produce+`ENVIRONMENT_ERROR`; they never return from a test as a pass.++## File Ownership And Disposition++| Current file | Final owner or action |+| --- | --- |+| `cross_platform_behavior_test.rs` | Shared Electron/Tauri case catalog and external fixture-state and desktop-side-effect oracles |+| `harness_wpf_test.rs` | WPF-specific rows using the common case/result runner |+| `harness_winui3_test.rs` | WinUI3-specific rows; keep only toolkit-distinct behavior |+| `harness_web_test.rs` | WebView2 and Page/CDP behavior; do not mix Page targeting with AX/PX labels |+| `harness_appkit_test.rs` | Canonical AppKit rows, including foreground/background AX scroll and the exact PX background-drag refusal |+| `harness_swiftui_test.rs` | Canonical SwiftUI tree/capture, background action, and popover-trigger rows; transient-panel enumeration remains a separate gap |+| `harness_gtk3_test.rs` | Minimal GTK3/AT-SPI rows for X11 and Wayland |+| Legacy Windows UX guard target | Deleted after typed launch, cursor, shared, capture, and desktop-scope owners passed the replacement audit |+| `modality_input_e2e_test.rs` | Deleted; shared cells own web actions and the Notepad row had no delivery oracle |+| `modality_background_test.rs` | Deleted; typed WPF background action rows and capture ownership passed the replacement audit |+| `capture_contract_test.rs` | Sole owner for tree/image inclusion behavior; canonical prerequisites fail instead of skipping |+| `desktop_scope_<os>_test.rs` | Platform-specific window/desktop scope contracts |+| `modality_focus_test.rs` | Deleted; shared click/type cells own focus preservation and launch focus has its typed action owner |+| `installed_app_launch_macos_test.rs` | Supporting typed Calculator/TextEdit launch and focus rows in the canonical logged-in macOS lane |+| `installed_app_textedit_macos_test.rs` | Supporting typed TextEdit AX integration in the canonical logged-in macOS lane; schema assertions remain in protocol tests |+| `harness_libreoffice_test.rs` | Optional installed-app lane; exclude from the canonical run and counts |+| `protocol_*`, schema, transport tests | Unit/protocol gate; no desktop video and no behavioral matrix rows |+| `tests/fixtures/shared/scenarios.json` | Prune only after selector and marker-reference audit |++## CI Shape++The contributor invocation stays selector-free. Lane selection remains private+CI plumbing and diagnostic state.++### Pull requests++- Run unit/protocol jobs by affected OS paths.+- Shared core or schema changes trigger Linux and Windows unit jobs.+- Platform-only changes trigger that platform's unit job.+- E2E is maintainer-dispatched and may become an optional pre-merge gate.++### Maintainer E2E++- Windows GitHub-hosted runners are canonical when the preflight proves an+ interactive desktop. An Azure RDP runner is an optional environment-parity+ replay, not a second source of behavioral truth.+- Linux GitHub-hosted runners run X11 under Xvfb with hosted packages. The Nix+ source gate is separate; the pure-Wayland maintainer lane uses the Nix dev+ shell. Linux produces no GIF requirement.+- macOS runs on a logged-in, TCC-authorized host through the canonical macOS+ runner. A future self-hosted runner must use the same preflight.++CI may fan the complete matrix into shared, native, and capture/scope jobs.+Those are execution partitions, not alternate public test suites.++## Reporting And Evidence++Rust emits one record for every declared cell. A shared Rust reporter then:++1. rejects duplicate or missing cell ids;+2. verifies the result against the case contract;+3. verifies required video evidence exists and is non-empty;+4. renders the behavioral table and the declared coverage table;+5. fails when a declared cell produced no result.++Do not parse `test ... ok` lines to create behavioral rows. Cargo/JUnit output+may still provide failure annotations for unit tests.++Upload one artifact archive per internal lane, with one stable evidence+subdirectory per behavioral cell. Each GitHub summary row links its exact+video path text to the owning lane archive. The adjacent `trajectory.json`+path remains in `results.jsonl` and the archive. GitHub cannot deep-link to a+file inside an archive; the exact path keeps the row unambiguous without+multiplying artifact uploads by the cell count.++## Implementation Slices++### Slice 1: Contract and preflight++- Finalize `CaseSpec`, result enums, refusal-code enums, and one schema version.+- Add reporter validation for duplicate, missing, and contradictory records.+- Add strict environment preflight to Windows, Linux, and macOS runners.+- Stop shell runners from inventing behavioral rows.++Exit: a missing session, fixture, permission, or recorder fails once as an+environment error, and a synthetic CaseSpec set renders a valid report.++### Slice 2: Shared matrix integrity++- Map every shared cell to an explicit driver route.+- Add any missing route cells; document every omitted equivalent.+- Add the desktop observer to background and refusal cells.+- Preserve editor-save and all existing external markers.+- Remove the three legacy shared tests after mapping every assertion.++Exit: no fake drag pass, no arbitrary error accepted as refusal, no shared+legacy test, and one result/evidence bundle per shared cell. This source+convergence is complete; platform delivery defects remain visible as cell+failures.++### Slice 3: Windows convergence++- Move guard, modality-input, and modality-background assertions into shared,+ WPF, WinUI3, capture, launch, or desktop-scope owners.+- Preserve every current failing action as a failing required-delivery cell or+ an issue-linked optional cell. Do not convert it to a green refusal.+- Add Windows desktop-scope to the canonical run.+- Delete the three transitional files only after cell-by-cell parity.++Exit: Windows has no guard/modality family, and every background cell has both+target-state and desktop-side-effect evidence.++### Slice 4: macOS and Linux convergence++- Adopt the common case/result runner in AppKit, SwiftUI, and GTK3 tests.+- Replace canonical early-return skips with preflight failures.+- Split Linux schema checks from real desktop behavior.+- Rename capture and desktop-scope ownership files.+- Move AppKit scroll and real-app checks to explicit optional issue lanes.++Exit: native cells emit the same records as shared cells, X11 and Wayland are+separate dimensions, and macOS failures distinguish TCC from driver behavior.++### Slice 5: Flake and fixture cleanup++- Replace fixed coordinates with discovered geometry.+- Allocate CDP ports per process.+- Replace sleeps with deadline polling on external markers.+- Add fixture generation-token reset and reuse harness processes only after it+ proves clean state.+- Prune fixture controls and markers with no live selector or oracle reference.++Exit: no canonical cell depends on VM-specific coordinates, fixed shared ports,+or an unexplained sleep.++### Slice 6: CI validation and deletion++- Keep the complete macOS matrix green after `install-local` and TCC preflight.+- Keep the accepted GitHub-hosted Windows matrix as the canonical result; use+ an optional RDP runner only for environment-parity investigations.+- Keep the accepted Linux X11 and Nix source runs as the supported Linux gates.+- Run pure Wayland as an experimental issue-linked lane until `#1922` closes.+- Compare old/new cells before each transitional file deletion.+- Update contributor docs and the PR description from the generated catalog.++Exit: every declared cell is delivered, refused according to contract, or+fails with a linked unresolved bug. Environment failures are separate.++## Deletion Gates++A test or fixture path may be deleted only when:++1. every assertion maps to a CaseSpec or unit/optional owner;+2. the replacement oracle is equal or stronger;+3. old and replacement outcomes were compared on each affected OS;+4. a required-delivery failure remains visible;+5. refusal cells use an explicit allowed code and desktop-side-effect proof;+6. runners, docs, and artifact labels stop referencing the old target in the+ same change;+7. the reporter finds no missing declared cells.++## Definition Of Done++- Rust has one typed behavioral catalog and one result schema.+- Required GUI prerequisites cannot pass through early returns.+- Test status and observed driver behavior are separate fields.+- Every supported action has foreground/background coverage by distinct driver+ route, with reasons for omitted combinations.+- Every delivered action has an external target-state oracle.+- Every background or refusal cell has desktop-side-effect evidence.+- No permanent guard, modality, or delivery test family remains.+- No `#[should_panic]` known-gap E2E test remains.+- No orphaned target is counted as coverage.+- Shared and native harnesses emit the same result/evidence shape.+- Unit/protocol tests stay desktop-independent and video-free.+- Windows, macOS, and supported Linux X11 complete runs produce classified+ outcomes and per-cell evidence links. Experimental Wayland and environment+ failures stop or fail explicitly without producing a false behavioral pass.diff --git a/libs/cua-driver/docs/test-harnesses-guide.md b/libs/cua-driver/docs/test-harnesses-guide.mdnew file mode 100644index 0000000000..32f64e7d39--- /dev/null+++ b/libs/cua-driver/docs/test-harnesses-guide.md@@ -0,0 +1,407 @@+# CUA Driver Test Harnesses Guide++This contributor guide explains where the CUA Driver tests live, how to run+them, and where to add a new case. For current coverage and platform gaps, use+`test-matrix.md` and `action-support.md`.++## The Short Version++There are two main test layers:++1. **Unit and protocol tests** exercise Rust code and the public MCP/CLI+ contract without launching a real application.+2. **Harness E2E tests** launch a small application built from this repository,+ drive it through the Rust driver, and verify an external application or+ desktop state.++The Rust tests are the source of truth. Python tests, old shell runners, and+historical recording scripts are not part of the canonical E2E path.++The canonical E2E command on every OS runs the complete matrix and takes no+suite selector:++```text+Linux: scripts/ci/linux/run-rust-e2e.sh+Windows: .\scripts\ci\windows\run-rust-e2e.ps1 -RequireGui+macOS: scripts/ci/macos/run-rust-e2e.sh+```++The OS workflow may fan the complete matrix out into independent jobs for+reporting and failure isolation. That is an execution detail; contributors+should think of it as one canonical suite.++## Repository Map++```text+cua/+|-- libs/cua-driver/+| |-- rust/+| | |-- crates/+| | | |-- cua-driver/ Rust driver and integration tests+| | | |-- cua-driver-core/ Shared driver logic and unit tests+| | | |-- cua-driver-testkit/ Shared Rust E2E helpers and evidence capture+| | | |-- platform-linux/ Linux backend+| | | |-- platform-macos/ macOS backend+| | | |-- platform-windows/ Windows backend+| | | `-- cursor-overlay/ Cursor evidence helper+| | |-- test-apps/ Ignored staged harness binaries+| | `-- Cargo.toml Rust workspace+| |-- tests/+| | |-- fixtures/+| | | |-- shared/web/ Shared web page and external markers+| | | |-- apps/ Repo-local fixture sources+| | | `-- build/ macOS/Linux/Windows fixture builders+| | `-- runners/ Auxiliary VM and sandbox entrypoints+| `-- docs/ Test matrix, reporting, and contributor docs+`-- scripts/ci/+ |-- linux/run-rust-e2e.sh Linux canonical runner+ |-- windows/run-rust-e2e.ps1 Windows canonical runner+ `-- macos/run-rust-e2e.sh macOS canonical runner+```++The important separation is:++| Layer | Owns | Does not own |+| --- | --- | --- |+| Rust integration test | Scenarios, driver calls, assertions, action metadata | OS setup and fixture compilation |+| `cua-driver-testkit` | Session helpers, fixture launching, screenshots, recordings, trajectories | The scenario list |+| Fixture app | Visible controls and externally observable state markers | Driver correctness assertions |+| OS runner | Build environment, user session, test selection, artifact collection | Test behavior definitions |++There is deliberately no second Python E2E implementation that the Rust suite+has to mirror.++## How One E2E Test Works++Every canonical E2E cell follows this shape:++```text+OS runner+ -> builds the Rust driver and repo-local fixture+ -> starts or connects to a real desktop session+ -> Rust testkit starts one harness application+ -> Rust test discovers the target window+ -> get_window_state provides accessibility tree and screenshot+ -> action addresses a target by AX element index or PX coordinates+ -> delivery is foreground or background where the action supports it+ -> fixture state, focus, pixels, or protocol response is checked+ -> testkit writes video, screenshots, trajectory, logs, and result data+```++A tool returning `ok` is not enough to pass an E2E cell. The fixture must show+that the action happened, or the test must verify a documented structured+refusal and the absence of focus or input side effects.++## The Test Layers++### Unit and Protocol Tests++These run without a repo-local GUI application and normally run without+`--ignored`:++| Location or prefix | What it proves |+| --- | --- |+| `rust/crates/*/src/**` | Core driver, platform-independent logic, schemas, and helpers |+| `protocol_*_test.rs` | MCP handshake, tool calls, sessions, media, and errors |+| `schema_*_test.rs` | Shared schema and backend consistency |+| `transport_config_persistence_test.rs` | CLI/MCP configuration persistence |+| `protocol_element_token_test.rs` | Element-token protocol behavior |++These tests should be fast, deterministic, and safe to run on ordinary CI+workers. They do not prove that a real click, key, scroll, or background input+reached an application.++### Harness E2E Tests++These are Rust integration tests under:++```text+libs/cua-driver/rust/crates/cua-driver/tests/+```++Most are marked `#[ignore]` because they require a desktop, built fixtures, and+platform permissions. They are selected by the OS runner rather than the+ordinary unit command.++The canonical E2E suite has two behavior owners:++| Owner | Purpose |+| --- | --- |+| Shared app | Same web behavior tested through Electron and Tauri |+| Native harness | Toolkit-specific controls and native window behavior |++WebView, CDP, and page-tool integration stays inside the shared or native+owner that exercises it. It is not a third public test family or command.++Delivery is not a test family. It is a dimension on each action row: an action+is tested in foreground and background modes whenever the driver and OS support+both. Capture and desktop scope are separate environment checks, while focus+preservation is a cross-cutting oracle that can be attached to any action row.+Focus, z-order, cursor, and desktop-state checks are cross-cutting invariants,+not a separate family. These names describe responsibilities, not separate+sources of truth or commands to run instead of the selector-free canonical+invocation.++## What The Complete Run Includes++### Windows++Runner: `scripts/ci/windows/run-rust-e2e.ps1`++| Runner area | Rust test | Real harness or app |+| --- | --- | --- |+| Shared app matrix | `cross_platform_behavior_test.rs` | Electron and Tauri |+| Native controls | `harness_wpf_test.rs` | Repo-local WPF app |+| Native controls | `harness_winui3_test.rs` | Repo-local WinUI3 app |+| Web integration | `harness_web_test.rs` | WebView2 and Electron |+| Capture contract | `capture_contract_test.rs` | WPF plus driver tree/image output |+| Launch contract | `launch_windows_test.rs` | Repo-local Electron launch and focus behavior |+| Agent cursor | `agent_cursor_windows_test.rs` | Source-built cursor overlay and pixel evidence |+| Desktop scope | `desktop_scope_windows_test.rs` | Windowless desktop input and scope rejection |++Cross-cutting instrumentation used by these rows includes the testkit+`DesktopObserver`, capture validation, cursor evidence, and desktop-scope+checks. These invariants are attached to their owning action rows rather than+run as a separate test family.++Windows is currently the broadest native matrix. It covers UIA controls, web+integration, background focus checks, and Windows-specific input routes. The+desktop observer is attached to shared and native action rows wherever+background delivery is tested.++### macOS++Runner: `scripts/ci/macos/run-rust-e2e.sh`++| Runner area | Rust test | Real harness or app |+| --- | --- | --- |+| Shared app matrix | `cross_platform_behavior_test.rs` | Electron and Tauri |+| Native web matrix | `cross_platform_behavior_test.rs` | Repo-local WKWebView host |+| Native controls | `harness_appkit_test.rs` | Repo-local AppKit app |+| Native controls | `harness_swiftui_test.rs` | Repo-local SwiftUI app |+| Installed app launch | `installed_app_launch_macos_test.rs` | Calculator and TextEdit |+| Installed app AX delivery | `installed_app_textedit_macos_test.rs` | TextEdit |+| Capture contract | `capture_contract_test.rs` | Installed driver and macOS capture APIs |+| Desktop scope | `desktop_scope_macos_test.rs` | macOS window and desktop scope |++The WKWebView host runs the same typed shared-web catalog as Electron and+Tauri. Calculator and TextEdit add typed supporting rows for built-in app+launch focus and native Cocoa background value delivery. They run in the+canonical logged-in macOS lane, but they do not replace repo-local fixtures.++### Linux++Runner: `scripts/ci/linux/run-rust-e2e.sh`++| Runner area | Rust test | Real harness or app |+| --- | --- | --- |+| Shared app matrix | `cross_platform_behavior_test.rs` | Electron and Tauri |+| Native controls | `harness_gtk3_test.rs` | Repo-local GTK3 app |+| Capture contract | `capture_contract_test.rs` | Linux capture backend |+| Desktop scope | `desktop_scope_linux_test.rs` | X11/Wayland desktop scope |++Linux has separate X11 and Wayland concerns. Nix supplies the reproducible+build and desktop environment, but the E2E test still needs an actual X11 or+Wayland session. Linux does not need GIF output; MP4, screenshots, accessibility+trees, trajectories, and logs are the useful evidence.++Wayland results are compositor-specific. The hosted lane uses Sway to prove+wlroots protocols. GNOME requires the optional WinRects Shell helper for+authoritative frame and buffer geometry, observation, capture, and verified+target activation. A portal/libei grant persists until the user revokes it, so+subsequent driver processes do not reopen the consent dialog.+KDE requires a future target-addressable KWin adapter; portal availability by+itself is not evidence that input can be sent safely to a named window.++## AX, PX, and Delivery++See [`action-support.md`](action-support.md) for the current Windows, macOS, and Linux+delivery, refusal, and unproven-action ledger.++These terms describe different dimensions:++| Term | Meaning |+| --- | --- |+| AX | Address a target through its accessibility/UI automation element |+| PX | Address a target by screen coordinates or pointer geometry |+| Foreground | The target may be brought to the foreground for delivery |+| Background | The target should receive the action without being raised or stealing focus |+| Window scope | Capture or action is limited to one target window |+| Desktop scope | Capture or action covers the full desktop |++The shared and native action matrices should test left click, right click,+double click, typing, keys, hotkeys, scroll, child windows, and drag across+AX/PX and foreground/background combinations where the driver supports them.+Unsupported background routes require an explicit refusal contract with an+allowed structured code and desktop-side-effect oracles. A refusal fails a+cell that requires delivery. There should not be a separate "delivery" family+whose only purpose is to repeat those same actions in the background.++Native harness rows use the same typed case/result contract as the shared+matrix. Current native `set_value` rows declare background delivery because+their contract includes no-focus and no-raise observations; actions without a+delivery concept use `not_applicable` explicitly.++## Cross-Cutting Invariants++The desktop observer is cross-cutting test instrumentation. It answers the same+question for any action, harness, or catalog area:++> Did the driver perform or reject the operation without disturbing the user's+> foreground application or desktop?++`cua-driver-testkit::DesktopObserver` owns the shared interface. Native Windows,+macOS, and Linux backends snapshot foreground-window, target z-order, cursor,+and leaked-input state before and after an action. Background rows opt into the+observer directly; there is no special guard suite.++| Invariant or scenario | What it checks |+| --- | --- |+| Background click/type/key | The target action does not move focus away from the user's foreground window |+| Minimized app launch | `launch_app(start_minimized=true)` does not raise the new app |+| Background hotkey | A keyboard chord does not steal focus |+| Child-window click | A target-created window does not unexpectedly become foreground |+| Background screenshot | Reading the target does not change focus or z-order |+| Agent cursor visibility | The cursor appears in the captured pixels when enabled and moved |++A focus assertion can prove "no focus steal" while failing to prove that a+click changed the target application state. An action row must therefore check+both the target's external state and, when background delivery is under test,+the cross-cutting desktop observer.++These tests require a real interactive Windows user desktop. They reject+Session 0, locked desktops, and disconnected RDP sessions. Without+`CUA_REQUIRE_GUI=1`, an unusable desktop can self-skip for local development;+the canonical Windows runner enables the hard-failure behavior.++## Evidence++Canonical GUI runs are expected to produce evidence per test cell:++```text+artifacts/cua-driver/<os>/+|-- recordings/<cell-label>-pid<pid>-<sequence>/recording.mp4+|-- recordings/<cell-label>-pid<pid>-<sequence>/trajectory.json+|-- recordings/<cell-label>-pid<pid>-<sequence>/turn-*/before_state.json+|-- recordings/<cell-label>-pid<pid>-<sequence>/turn-*/before.png+|-- recordings/<cell-label>-pid<pid>-<sequence>/turn-*/after_state.json+|-- recordings/<cell-label>-pid<pid>-<sequence>/turn-*/after.png+|-- cases.jsonl+|-- environment.jsonl+|-- results.jsonl+|-- summary.md+`-- <rust-target>.log+```++The GitHub Actions summary contains one row per meaningful behavioral cell,+including its OS, harness, action, AX/PX targeting, delivery mode, driver route,+expected and observed behavior, oracles, and one evidence link. The link uses+the exact video path as its label and opens the owning lane archive. Unit tests+need normal test output and logs; they do not need desktop video.++## What Is Implemented Today++- Rust owns the canonical scenario definitions and external-state assertions.+- Electron and Tauri use the same shared web fixture across supported OSs.+- Native Windows, macOS, and Linux harnesses are repo-local applications built+ from source.+- The three OS runners use a selector-free command for the complete matrix.+- Shared and native harness owners emit the same typed v2 result records.+- Canonical GUI rows collect a trajectory and MP4, validated before reporting.+- Canonical GUI rows require parseable pre/post state and non-empty pre/post+ target-window images for each targeted turn. The one narrow exception is a+ successful Windows `bring_to_front` restore whose minimized target has no+ pre-action image or whose host capture remains unavailable afterward: it must+ retain the successful action response, captured post-action accessibility+ state, an explicit capture classification, trajectory, and MP4. Any other+ missing expected evidence fails the report.+- Per-cell video starts after fixture readiness and foreground/background+ posture. A 300 ms baseline precedes dispatch, and capture continues through+ external oracle collection. `trajectory.json` must finish with+ `behavior_video.status = "finalized"`.+- Windows hosted runs use `GetConsoleWindow` to select the inherited+ HostedComputeAgent/runner console, verify its identity, and minimize it+ through `ShowWindow(SW_MINIMIZE)` before fixture or sentinel posture is+ established. The sentinel remains a separate test fixture and is reasserted+ after console cleanup.+- Strict lane preflights fail on missing fixtures, desktop access, permissions,+ accessibility, capture, or recording support instead of silently skipping.+- GitHub summaries link every evidence-bearing row to its lane archive and+ display the exact recording path. The trajectory path remains in the typed+ evidence and archive.+- Unit/protocol tests remain separate from interactive E2E tests.++## What Still Needs Implementation++The remaining work is platform coverage and validation, not another test+hierarchy:++1. **Broaden native action rows.** The shared web matrix covers every declared+ AX/PX and foreground/background cell. AppKit, SwiftUI, WPF, WinUI3,+ WebView2, and non-GTK3 Linux toolkits still have unproven native combinations listed in+ [`action-support.md`](action-support.md).+2. **Preserve exact-source validation.** Every accepted platform run must record+ one immutable source SHA and retain the typed evidence contract.+3. **Close representative-desktop gaps.** Hosted Sway passes the complete+ Electron, Tauri, GTK3, capture, and desktop-scope catalogs. A real GNOME 46+ session passes GTK3, capture, and desktop scope, but still needs the shared+ renderer catalog and portal-video parity. Plasma 6 still needs a verified+ KWin activation adapter and its first accepted behavioral lane. Issue `#1922`+ tracks the grouped backend work.+4. **Add representative toolkit surfaces.** GTK4, Qt5/Qt6, VTE, VCL, and GL+ canvases remain optional real-app gaps; shared Electron/Tauri coverage does+ not substitute for those native stacks.+5. **Flake cleanup.** Replace remaining fixed native waits with external-state+ polling and add fixture reset tokens before reusing a harness process.++## File Convergence Plan++The goal is not to put every assertion into one enormous test file. The goal is+to give each behavior one clear owner and make cross-cutting evidence reusable.++### Target Ownership++```text+rust/crates/cua-driver-testkit/src/+`-- observer.rs Cross-OS desktop-side-effect interface++rust/crates/cua-driver/tests/+|-- cross_platform_behavior_test.rs Shared Electron/Tauri action matrix+|-- harness_wpf_test.rs Windows WPF action rows+|-- harness_winui3_test.rs Windows WinUI3 action rows+|-- harness_web_test.rs WebView2/Electron page and CDP rows+|-- harness_appkit_test.rs macOS AppKit action rows+|-- harness_swiftui_test.rs macOS SwiftUI action rows+|-- harness_gtk3_test.rs Linux GTK3 action rows+|-- capture_contract_test.rs Tree and screenshot read contract+|-- desktop_scope_<os>_test.rs Window/desktop scope invariants+`-- protocol_*_test.rs Protocol and schema tests+```++The desktop observer is a helper, not a test family. An action row invokes it when+the row is testing background delivery. The row then records both outcomes:++1. Did the target application state change, or did the driver return the+ documented structured refusal?+2. Did focus, z-order, cursor, and desktop state remain within the contract?++Focus, z-order, cursor, and input-leak assertions belong to the typed action+rows that exercise them; launch, capture, cursor, and desktop-scope contracts+retain their narrowly owned scenarios. The canonical runner is the only+user-facing command; lane selectors are internal diagnostics.++## Contributor Workflow++When adding a new scenario:++1. Add or update the repo-local fixture and its external state marker.+2. Add the Rust scenario under `rust/crates/cua-driver/tests/`.+3. Declare AX/PX addressing, foreground/background delivery, scope, and oracle.+4. Add the scenario to `docs/test-matrix.md` and this guide when it changes the+ cross-OS structure.+5. Update only the OS runner selection when the test is platform-specific.+6. Run the smallest Rust test locally, then run the OS command before+ calling the matrix complete.++The goal is one understandable Rust E2E model across platforms, with+platform-specific harnesses where the OS genuinely differs.diff --git a/libs/cua-driver/docs/test-matrix.md b/libs/cua-driver/docs/test-matrix.mdnew file mode 100644index 0000000000..efcbdb3016--- /dev/null+++ b/libs/cua-driver/docs/test-matrix.md@@ -0,0 +1,202 @@+# CUA Driver Test Matrix++This is the source map for the Rust test suites in `libs/cua-driver`. It uses+two top-level test classes:++1. **Unit and deterministic protocol tests.** These do not depend on a repo-local+ application or an interactive desktop.+2. **Harness E2E tests.** These build and launch a repo-local application, drive+ it through the Rust driver, and verify an external application or desktop+ oracle.++A test is E2E because it crosses the driver, OS, window system, and application+boundary. The Rust harness catalog and its external oracles are the source of+truth.++## Matrix Dimensions++Every harness E2E result should identify these dimensions:++| Dimension | Values |+| --- | --- |+| OS | `windows`, `macos`, `linux` |+| Window system | Win32/UIA, AppKit/AX, X11/AT-SPI, Wayland/AT-SPI, WebView/CDP |+| Harness | Electron, Tauri, WPF, WinUI3, WebView2, AppKit, SwiftUI, WKWebView, GTK3 |+| Action targeting | `ax`, `px`, `page`, `not_applicable` |+| Delivery | `background`, `foreground`, `N/A` |+| Scope | `window`, `desktop`, `N/A` |+| Oracle | App state, accessibility state, focus state, pixel state, protocol state |+| Test status | `pass`, `fail`, `skip`, `environment_error` |+| Observed behavior | `delivered`, `refused`, `no_effect`, `error`, `not_run` |++Focus preservation is a cross-cutting oracle, not a separate matrix family.+When a background action is tested, the row should attach the platform focus+observer in addition to checking the target application's external state.++`AX` and `PX` describe how an action addresses a target. They are not capture+modes. `get_window_state` returns the tree and screenshot together; the action+uses an element index or coordinates.++## Unit And Deterministic Tests++These run without the repo-local GUI applications:++| Area | Location | Coverage |+| --- | --- | --- |+| Core driver logic | `rust/crates/*/src/**` | Protocol values, sessions, schemas, image helpers, input helpers, configuration, telemetry, CLI behavior |+| MCP and CLI boundary | `rust/crates/cua-driver/tests/protocol_*` | Handshake, tool registration, tool calls, media, sessions, and errors |+| Schema gate | `schema_consistency_test.rs` | Shared tool schema parity across OS backends |+| Configuration transport | `transport_config_persistence_test.rs` | CLI and MCP configuration persistence |+| Token and protocol surfaces | `protocol_element_token_test.rs`, related tests | JSON-RPC-visible contract behavior |++Some protocol tests spawn the driver process. They remain deterministic because+they do not launch a real target application or require a desktop. They should+be reported with the unit gate, separately from Harness E2E.++The ordinary unit gate must not run `#[ignore]` GUI tests and must not turn+missing desktop fixtures into silent skips.++## Harness E2E: Shared Web Applications++Electron and Tauri are separate repo-local applications that load the shared+web harness. The fixture exposes action-specific controls and external state+markers rather than an application task that duplicates those same actions.++Source:++- `tests/fixtures/shared/web/index.html`+- `tests/fixtures/apps/cross-platform/electron/`+- `tests/fixtures/apps/cross-platform/tauri/`+- `rust/crates/cua-driver/tests/cross_platform_behavior_test.rs`++The shared harness exposes deterministic external markers for these actions:++| Action family | Actions | Addressing | Delivery |+| --- | --- | --- | --- |+| Pointer | Left click, right click, double click | AX and PX | Background and foreground |+| Keyboard and text | Type text, Return, hotkey | AX and PX where supported | Background and foreground |+| Scroll | Scroll | AX and PX where supported | Background and foreground |+| Child windows | Open child window | AX and PX | Background and foreground |+| Drag | Drag source to drop target | PX | Background and foreground |+| State controls | Checkbox, radio, combo, slider | AX or PX by control | Mode declared per action |+| Editor | Type, save, saved-state readback | AX and PX where supported | Mode declared per action |++The Rust shared catalog declares 36 evidence-bearing cells per harness+application. Windows and Linux run 72 shared cells across Electron and Tauri;+macOS also runs the native WKWebView host for 108 shared cells. Each host covers+the full AX/PX and foreground/background cross-product for click, text,+keyboard, scroll, and child-window actions, plus both delivery modes for PX+drag and AX editor-save. A+background capability refusal is a valid result only when the test verifies the+declared structured refusal and the no-focus/no-z-order/no-input-leak side+effect. A refusal fails a cell whose contract requires delivery.++## Harness E2E: Native Windows++Windows native harnesses are repo-local applications built from source:++| Harness | Source test | Coverage |+| --- | --- | --- |+| WPF | `harness_wpf_test.rs` | UIA controls, text, keys, pointer actions, scroll, drag, popups, menus, modal windows |+| WinUI3 | `harness_winui3_test.rs` | XAML controls, text, checkbox/radio, slider, combo, popup |+| WebView2 | `harness_web_test.rs` | Window discovery, CDP page access, JavaScript, DOM click path |+| Desktop invariants | Testkit `DesktopObserver` plus typed launch/capture/cursor owners | Cross-cutting focus, z-order, minimized-launch, screenshot, cursor, and desktop checks |++Native controls use AX/UIA state as their oracle. Pointer actions also use PX+where the tool contract requires coordinates. Current `set_value` rows declare+background delivery and attach desktop-side-effect oracles.++## Harness E2E: macOS++| Harness | Source test | Coverage |+| --- | --- | --- |+| AppKit | `harness_appkit_test.rs` | AX tree/capture, AX value/text, AX scroll, PX clicks, and foreground slider drag across the proven delivery modes; background drag is an exact refusal |+| SwiftUI | `harness_swiftui_test.rs` | AX tree/capture, background click/value, and foreground popover-trigger state |+| WKWebView | `cross_platform_behavior_test.rs` | Dedicated native host running the full 36-cell shared web catalog |+| Installed-app launch/focus | `installed_app_launch_macos_test.rs` | Real Calculator/TextEdit launch and focus behavior in the canonical logged-in lane |+| Installed-app text | `installed_app_textedit_macos_test.rs` | Real TextEdit AX background write and verification in the canonical logged-in lane |++macOS uses the installed ScreenCaptureKit/AX permissions for GUI runs. The+repo-local harnesses are canonical; Calculator and TextEdit are supporting+real-app checks. SwiftUI's popover trigger is proven independently from the+remaining transient-panel AX discovery gap.++## Harness E2E: Linux++| Harness | Source test | Window system | Coverage |+| --- | --- | --- | --- |+| Electron | `cross_platform_behavior_test.rs` | X11 and hosted Sway | Shared web action matrix |+| Tauri | `cross_platform_behavior_test.rs` | X11 and hosted Sway | Shared web action matrix |+| GTK3 | `harness_gtk3_test.rs` | X11/AT-SPI and Wayland/AT-SPI where configured | Native GTK controls and input |+| Desktop scope | `desktop_scope_linux_test.rs` | X11/Wayland | Desktop versus window scope |++Nix provides the Linux build and desktop environment. X11 and Wayland are+separate matrix dimensions because their capture and input contracts differ.+Linux runs do not produce GIF output. Every canonical GUI cell retains an MP4+and trajectory alongside screenshots, AX trees, structured results, and driver+logs for E2E evidence.++## Action Delivery Matrix++The current per-OS delivery/refusal ledger is maintained in+[`action-support.md`](action-support.md). This section defines the coverage+policy; the ledger records empirical status.++For each OS and harness where the action is supported, the canonical E2E suite+should cover both delivery modes:++| Action | Background | Foreground | Addressing |+| --- | --- | --- | --- |+| Left click | Required delivery or a declared refusal contract | Required | AX, PX |+| Right click | Required delivery or a declared refusal contract | Required | AX, PX |+| Double click | Required delivery or a declared refusal contract | Required | AX, PX |+| Drag | Required delivery or a declared refusal contract | Required | PX, with AX discovery |+| Scroll | Required delivery or a declared refusal contract | Required | AX, PX where supported |+| Type text | Required delivery or a declared refusal contract | Required | AX, PX where supported |+| Press key | Required delivery or a declared refusal contract | Required | AX, PX where supported |+| Hotkey | Required delivery or a declared refusal contract | Required | AX, PX where supported |+| Set value | Required in current native rows | Not separately declared | AX/UIA/AXValue |+| Screenshot | N/A | N/A | Window or desktop capture |+| Page/CDP | N/A | N/A | Page selector/JavaScript |++`N/A` means the API has no delivery mode for that operation. A refusal is a+passing observation only when the Rust case expects refusal, the exact code is+allowed, and the desktop-side-effect oracles pass.++## CI And Local Entry Points++| Gate | Environment | Entry point |+| --- | --- | --- |+| Linux unit/source | Nix Linux CI | `nix build .#checks.x86_64-linux.cua-driver-build .#checks.x86_64-linux.cua-driver-linux-rust-unit` |+| Windows unit/compile | `windows-latest` | Package-scoped `cargo test --all-targets --no-run --locked` |+| Windows Harness E2E | Active Windows user session | `scripts/ci/windows/run-rust-e2e.ps1 -RequireGui` |+| Linux X11 Harness E2E | Nix X11 session | `scripts/ci/linux/run-rust-e2e.sh` |+| Linux Sway Harness E2E | Controlled wlroots session | `scripts/ci/linux/run-rust-e2e-wayland.sh` |+| Linux nested-compositor E2E | Controlled experimental session | `scripts/ci/linux/run-rust-e2e-inject.sh` |+| Linux representative desktop E2E | Existing GNOME, KDE, or Xorg login | `scripts/ci/linux/run-rust-e2e-desktop.sh <desktop>` |+| macOS Harness E2E | Logged-in macOS session with permissions | `scripts/ci/macos/run-rust-e2e.sh` |++Workflows select private execution lanes. Rust source owns scenario definitions,+fixture oracles, and result records. OS runners only build the driver, stage the+local harnesses, establish the desktop session, collect evidence, and publish+the shared report.++## Evidence And Ownership++Every Harness E2E cell should produce:++```text+recordings/<cell-label>-pid<pid>-<sequence>/recording.mp4+recordings/<cell-label>-pid<pid>-<sequence>/trajectory.json+results.jsonl+<rust-target>.log+```++The GitHub summary links the exact recording path from the corresponding row to+its lane archive. Target logs remain lane-level diagnostics. Unit tests need+logs and test-result output, but do not need desktop video.++When a new action or modality is added, update this document, the shared fixture+oracle, the Rust test, and the OS-specific runner selection together. This is+the cross-OS checklist that prevents a Windows-only test from being mistaken+for cross-platform coverage.diff --git a/libs/cua-driver/tests/fixtures/README.md b/libs/cua-driver/tests/fixtures/README.mdindex 74df5105a3..30d3f494f5 100644--- a/libs/cua-driver/tests/fixtures/README.md+++ b/libs/cua-driver/tests/fixtures/README.md@@ -27,9 +27,7 @@ tests/fixtures/│ ├── winui3/ # unpackaged WinUI3 app│ └── webview2/ # WPF + WebView2 host for shared DOM├── build/ # host build scripts-├── smoke/ # lightweight local smoke runners-├── linux-container/ # Azure/XFCE Linux verification helpers-└── modality-recordings/ # recorder scripts and dashboards+└── smoke/ # lightweight local smoke runners````shared/scenarios.json` is the source of truth for AutomationIds, AX@@ -44,6 +42,7 @@ Build scripts stage outputs into `libs/cua-driver/rust/test-apps/`.# macOS: AppKit, SwiftUI, WKWebView, Electron, Taurilibs/cua-driver/tests/fixtures/build/macos.shlibs/cua-driver/tests/fixtures/build/macos.sh --skip electron+libs/cua-driver/tests/fixtures/build/macos.sh --only wkwebview# Linux: GTK3, Electron, Taurilibs/cua-driver/tests/fixtures/build/linux.sh@@ -69,9 +68,10 @@ Host requirements:Rust tests under `libs/cua-driver/rust/crates/cua-driver/tests/` consume the staged`rust/test-apps/harness-<name>/` outputs:-- `harness_<toolkit>_test.rs`: toolkit-specific app coverage, usually ignored.-- `modality_<area>[_<os>]_test.rs`: background input, capture, and desktop- modality coverage, ignored/manual or VM-backed.+- `cross_platform_behavior_test.rs`: typed Electron/Tauri action matrix.+- `harness_<toolkit>_test.rs`: toolkit-specific app coverage.+- `capture_contract_test.rs` and `desktop_scope_<os>_test.rs`: capture and+ desktop-scope contracts.- `protocol_*_test.rs` and schema tests: headless protocol coverage, default.Rust integration tests under `libs/cua-driver/rust/crates/cua-driver/tests/` drive@@ -99,5 +99,5 @@ the shared Electron/Tauri harnesses through the public MCP interface.tests.- Keep platform-specific quirks in the relevant Rust test or app source, not inthe shared fixture.-- If a recorder finding graduates into an invariant, move it into a Rust test- and summarize it here.+- Record behavioral evidence through the Rust testkit and canonical OS runner;+ do not add a second Python or shell assertion layer.diff --git a/libs/cua-driver/tests/fixtures/linux-container/README.md b/libs/cua-driver/tests/fixtures/linux-container/README.mddeleted file mode 100644index 3439e87e55..0000000000--- a/libs/cua-driver/tests/fixtures/linux-container/README.md+++ /dev/null@@ -1,148 +0,0 @@-# Linux dispatch-ladder lane — XFCE in a container--A reproducible rig for validating the cua-driver Linux **`delivery_mode`**-background/foreground ladder and the AT-SPI modalities against a **real window-manager** (xfwm4 + EWMH) and a **live AT-SPI tree** — the two things the-existing Xvfb harness (no WM) can't exercise.--The target is the `trycua/cua-xfce` image: a full XFCE desktop on X11 with an-accessibility bus, reachable over VNC and the computer-server API. It's a-richer target than Xvfb for exactly the rungs that need a WM (foreground / EWMH-activation) or a11y (AT-SPI element + at-point clicks).--## What it validates--`calc.sh` drives `galculator` through the four parity modalities; map each to a-`delivery_mode` rung and confirm the reported `path`:--| Modality | `delivery_mode` | Expected `path` | Driver-verifiable? |-|---|---|---|---|-| Element click (`element_index`) | `background` | `x11_atspi` (AT-SPI `do_action`) | yes (a11y action) |-| Pixel / vision click | `background` | `x11_atspi` (`do_action`-at-point) for AX apps; else MPX `x11_pixel` | yes when AT-SPI-at-point lands |-| Pixel click (escalated) | `foreground` | `x11_pixel_fg` (EWMH activate → inject → restore) | no — confirm via screenshot |-| `type_text` into editable | `background` | `ax` (AT-SPI `insertText`) | yes (`verified:true`) |-| `type_text`, non-editable focus | `foreground` | `key_events_fg` | no — confirm via screenshot |--Background pixel/vision click **lands** on X11: an AX app takes the focus-free-AT-SPI `do_action`-at-point path (`x11_atspi`) — the same background-pixel-behavior macOS/Windows have. It falls to the MPX virtual-pointer path-(`x11_pixel`, which needs a **real Xorg + `/dev/uinput`**, absent under-Xvnc/minimal containers) only for non-AX surfaces; escalate to `foreground`-there.--## Setup recipe--1. **Run the image** (Azure Container Instances is the recommended single-- container host; the image is public on Docker Hub):-- ```- az container create -g <rg> -n cua-xfce --image trycua/cua-xfce \- --ports 8000 5901 --ip-address Public --os-type Linux --cpu 2 --memory 4- ```--2. **Install cua-driver as the desktop user** (`install-local` refuses to run- as root; the container exec lands as root, so use `runuser`). Sync the repo- in or build from a checkout, then:-- ```- chown -R cua /opt/cua /home/cua/.cargo /home/cua/.rustup- runuser -u cua -- bash /opt/cua/libs/cua-driver/install-local.sh- ```--3. **Drive the scenario** (the daemon now self-discovers the session bus, so no- manual `DBUS_SESSION_BUS_ADDRESS` export is needed for `serve` — `calc.sh`- still exports it for the short-lived CLI calls):-- ```- runuser -u cua -- bash calc.sh fullreset # daemon + a11y + launch galculator- runuser -u cua -- bash calc.sh doctor # AT-SPI/org.a11y.Bus probe + dbus addr- runuser -u cua -- bash calc.sh state # AT-SPI tree (shows degraded flag if empty)- runuser -u cua -- bash calc.sh prep # resolve window id (needed before pxclick)- runuser -u cua -- bash calc.sh click 7 background # element click, bg- runuser -u cua -- bash calc.sh pxclick 3 background # pixel/vision click, bg- runuser -u cua -- bash calc.sh type 789 foreground # foreground EWMH type- runuser -u cua -- bash calc.sh btf # bring_to_front (EWMH)- ```--## Gotchas (learned the hard way)--- **AT-SPI needs the session bus.** A daemon started outside the desktop- session has no `DBUS_SESSION_BUS_ADDRESS` → the AT-SPI tree comes back empty- and `get_window_state` reports `degraded:true`. The daemon now auto-discovers- it (`platform-linux/src/session_bus.rs`); if it still can't, the session has- no a11y bus or the daemon isn't running as the desktop user.-- **Run as the desktop user.** Root-against-a-user-session can't read that- user's session-process environ or `/run/user/<uid>/bus`. This is the Linux- analogue of the Windows Session 0 isolation problem.-- **Zombie children pollute `pgrep`.** A single-instance app (galculator)- launched by the daemon can leave an unreaped zombie child after a kill; it- shows in `pgrep` and makes `list_windows` look empty. `fullreset` restarts- the daemon, which reaps it.-- **Stale daemon → stale schema.** A long-lived daemon proxies a cached tool- schema. To assert the *freshly built* binary's schema, shell- `cua-driver describe <tool>` (computes the ToolDef locally) instead of a- daemon round-trip — this is what `modality_dispatch_linux_test.rs` does.-- **VNC screenshots over the computer-server `/cmd` SSE endpoint** are `data:`-- prefixed JSON; strip the `data: ` prefix before `json.loads`.--## Second lane: trycua/cua-ubuntu (Kasm desktop)--The same harness validates the Kasm-based `trycua/cua-ubuntu` image (XFCE under-Kasm/VNC, user `kasm-user`). Two image-specific gotchas:--- **Startup hangs without a Kasm orchestrator.** `vnc_startup.sh` blocks in- `wait_for_network_devices`, which loops until `ip link show type veth` returns- an `eth*` interface — an interface type the Kasm orchestrator's networking- provides but plain ACI does not, so it waits forever (no `NET_ADMIN` to create- one, and the script's filesystem is read-only). Bypass it at container start- with an override command-line that patches the wait to a no-op (runs as the- main process, so no restart loop):-- ```- az container create -g <rg> -n cua-ubuntu --image trycua/cua-ubuntu:latest \- --cpu 2 --memory 4 --ports 6901 8000 --ip-address Public \- --os-type Linux --restart-policy OnFailure \- --command-line '/bin/bash -c "sed s/^wait_for_network_devices/true/ /dockerstartup/vnc_startup.sh>/tmp/vs.sh && exec /bin/bash /tmp/vs.sh /dockerstartup/kasm_startup.sh --wait"'- ```--- **Drive it as `kasm-user`** with `$HOME=/home/kasm-user`. The harness derives- its paths from `$HOME`, so no edits are needed:- `bash modality_matrix.sh setup` / `matrix` just work once cua-driver is built- to `~/.local/bin`. noVNC is on `:6901` (password `vncpassword`).--Both lanes produce the same modality matrix and both prove the session-bus-auto-discovery fix (daemon started with `DBUS_SESSION_BUS_ADDRESS` unset → 58-AT-SPI elements, `degraded` unset).--## Third lane: KDE / GNOME VMs (`derec.sh`)--The container lanes above are both **XFCE** (GTK3 galculator) over `az exec`.-The genuinely different desktops/toolkits run on full **Azure VMs** over SSH,-driven by **`derec.sh`** (also in this directory):--| VM | Desktop | App | Toolkit | Why a VM, not a container |-|---|---|---|---|---|-| `cua-kde` | KDE Plasma (X11) | `kcalc` | **Qt** | Plasma X11 renders fine over VNC |-| `cua-gnome` | GNOME Shell (X11) | `gnome-calculator` | **GTK4** | gnome-shell needs **real console Xorg** (software GLX); it won't run over Xvnc (no GLX) |--`derec.sh setup` / `record` / `env` mirror the container harness; **`derec.sh-verify`** asserts the **GTK4 coordinate invariant** — per-button `frame`s are-distinct (not collapsed to the window corner, the GTK4 `(0,0)` regression) and-every button center lies inside the window's X11 rect. That's the runtime guard-the schema-only `modality_dispatch_linux_test` can't provide.--## CI coverage--Pure logic is unit-tested **display-free** in `platform-linux`:-- session-bus discovery — the comm-truncation matcher *and* the- `/proc/<pid>/environ` parse (`session_bus.rs`);-- the GTK4 `_GTK_FRAME_EXTENTS` parse + the `origin+inset+WINDOW` screen- reconstruction, anchored to the live `(132,375)` capture (`atspi/native.rs`,- `coord_tests`).--The dispatch *contract* (every input tool advertises `delivery_mode`;-`bring_to_front` is a real EWMH activation) is asserted via-`rust/crates/cua-driver/tests/modality_dispatch_linux_test.rs`. Everything that-genuinely needs a live desktop + a11y bus — the dispatch ladder's *behaviour*,-the GTK4 coordinate invariant, recordings — is covered by the container lanes-(manual / `trycua/cua-xfce` rig) and the VM lanes (`derec.sh verify`), not CI.diff --git a/libs/cua-driver/tests/fixtures/linux-container/calc.sh b/libs/cua-driver/tests/fixtures/linux-container/calc.shdeleted file mode 100644index 6285a14ebb..0000000000--- a/libs/cua-driver/tests/fixtures/linux-container/calc.sh+++ /dev/null@@ -1,142 +0,0 @@-#!/usr/bin/env bash-# cua-driver modality test harness for the XFCE-in-a-container lane.-#-# Drives galculator through the four parity modalities (background AX click,-# background pixel/vision click, foreground EWMH type, background type-# focus-limit) so each `delivery_mode` rung can be validated against a real WM-# (xfwm4 + EWMH) and a live AT-SPI tree. Invoke ONE action per call; state-# (pid/window) persists in /tmp/cstate.* between calls.-#-# Usage: bash calc.sh <action> [args...]-# See README.md in this directory for the full lane (image, install, gotchas).--export DISPLAY=:1-export HOME=/home/cua-export XDG_RUNTIME_DIR=/run/user/1000--# AT-SPI needs the desktop session bus. The cua-driver daemon now AUTO-DISCOVERS-# DBUS_SESSION_BUS_ADDRESS at startup (platform-linux/src/session_bus.rs), so the-# `serve` process self-recovers it even when started outside the session. We-# still export it here for the short-lived `cua-driver call` CLI invocations and-# for `gsettings`, discovering it from a running session process's environ the-# same way the daemon does (the VNC/XFCE session runs an ad-hoc bus).-for _p in xfce4-session xfsettingsd xfwm4 xfdesktop Thunar; do- _pid=$(pgrep -x "$_p" | head -1)- if [ -n "$_pid" ] && [ -r /proc/$_pid/environ ]; then- _a=$(tr '\0' '\n' < /proc/$_pid/environ | sed -n 's/^DBUS_SESSION_BUS_ADDRESS=//p' | head -1)- [ -n "$_a" ] && export DBUS_SESSION_BUS_ADDRESS="$_a" && break- fi-done-export XAUTHORITY=/home/cua/.Xauthority-CUA=/home/cua/.local/bin/cua-driver-S=/tmp/cstate-act="$1"; shift--case "$act" in- daemon)- pgrep -f 'cua-driver serve' >/dev/null || (setsid "$CUA" serve >/tmp/cuad.log 2>&1 &)- sleep 2- "$CUA" status 2>&1 | head -4- ;;- doctor)- # Exercises the hardened AT-SPI probe (org.a11y.Bus name_has_owner +- # discovered DBUS_SESSION_BUS_ADDRESS).- "$CUA" call check_permissions '{}' 2>&1 | head -20- ;;- a11y)- gsettings set org.gnome.desktop.interface toolkit-accessibility true 2>/dev/null- pgrep -f at-spi-bus-launcher >/dev/null || (setsid /usr/libexec/at-spi-bus-launcher --launch-immediately >/tmp/atspi.log 2>&1 &)- sleep 1- pkill -x galculator 2>/dev/null; sleep 1- GTK_MODULES=gail:atk-bridge NO_AT_BRIDGE=0 setsid galculator >/dev/null 2>&1 &- sleep 2- pgrep -x galculator | head -1 > "$S.pid"- echo "a11y on; galculator pid=$(cat $S.pid) atspi=$(pgrep -f at-spi-bus-launcher | head -1)"- ;;- state) # snapshot the AT-SPI tree; prints element map (and degraded flag)- PID=$(cat "$S.pid")- [ -z "$PID" ] && PID=$(pgrep -x galculator | head -1) && echo "$PID" > "$S.pid"- WID=$("$CUA" call list_windows "{\"pid\":$PID}" 2>/dev/null | python3 -c "import sys,json;ws=json.load(sys.stdin).get('windows',[]);print(ws[0]['window_id'] if ws else '')" 2>/dev/null)- echo "$WID" > "$S.wid"- "$CUA" call get_window_state "{\"pid\":$PID,\"window_id\":$WID,\"capture_mode\":\"ax\"}" > /tmp/state.json 2>&1- echo "pid=$PID wid=$WID"- python3 - <<'PY'-import json-try:- d=json.load(open('/tmp/state.json'))-except Exception as e:- print("STATE PARSE FAIL:", str(e)[:200]); print(open('/tmp/state.json').read()[:400]); raise SystemExit-if d.get('degraded'):- print("DEGRADED:", d.get('degraded_reason','')[:160])-els=d.get('elements',[])-print("total elements:", len(els))-for e in els:- r=e.get('role',''); l=e.get('label')- if r and ('button' in r.lower() or 'push' in r.lower() or l):- print(e.get('element_index'), r, repr(l)[:22])-PY- ;;- click) # click <element_index> <background|foreground>- PID=$(cat "$S.pid"); WID=$(cat "$S.wid")- "$CUA" call click "{\"pid\":$PID,\"window_id\":$WID,\"element_index\":$1,\"delivery_mode\":\"$2\"}"- ;;- pxclick) # pxclick <label> <background|foreground> — click a button by PIXEL coords (window-local)- PID=$(cat "$S.pid"); WID=$(cat "$S.wid")- read LX LY SX SY < <(python3 - "$1" <<'PY'-import json, sys-label = sys.argv[1]-st = json.load(open('/tmp/state.json'))-win = json.load(open('/tmp/win.json'))['windows'][0]-wx, wy = win['x'], win['y']-for e in st.get('elements', []):- if str(e.get('label')) == label and e.get('frame'):- f = e['frame']; cx = f['x'] + f['w']/2; cy = f['y'] + f['h']/2- print(int(cx - wx), int(cy - wy), int(cx), int(cy)); break-PY-)- echo "label=$1 screen=($SX,$SY) win-local=($LX,$LY)"- "$CUA" call click "{\"pid\":$PID,\"window_id\":$WID,\"x\":$LX,\"y\":$LY,\"delivery_mode\":\"$2\"}"- ;;- type) # type <text> <background|foreground>- PID=$(cat "$S.pid")- "$CUA" call type_text "{\"pid\":$PID,\"text\":\"$1\",\"delivery_mode\":\"$2\"}"- ;;- btf)- PID=$(cat "$S.pid"); WID=$(cat "$S.wid")- "$CUA" call bring_to_front "{\"pid\":$PID,\"window_id\":$WID}"- ;;- prep) # resolve window id into /tmp/win.json + cstate.wid (needed before pxclick)- PID=$(cat "$S.pid")- [ -z "$PID" ] && PID=$(pgrep -x galculator | head -1) && echo "$PID" > "$S.pid"- echo "pid=$PID"- "$CUA" call list_windows "{\"pid\":$PID}" | tee /tmp/win.json- WID=$(python3 -c "import json;ws=json.load(open('/tmp/win.json')).get('windows',[]);print(ws[0]['window_id'] if ws else '')" 2>/dev/null)- echo "$WID" > "$S.wid"; echo "wid=$WID"- ;;- fullreset)- # kill daemon (reaps zombie galculator children) + all galculator, enable- # a11y, restart, launch. Use when list_windows comes back empty (a zombie- # unreaped child of the daemon pollutes pgrep).- pkill -9 -f 'cua-driver serve' 2>/dev/null; pkill -9 -f galculator 2>/dev/null; sleep 3- rm -f /home/cua/.cache/cua-driver/cua-driver.sock- gsettings set org.gnome.desktop.interface toolkit-accessibility true 2>/dev/null- pgrep -f at-spi-bus-launcher >/dev/null || (setsid /usr/libexec/at-spi-bus-launcher --launch-immediately >/tmp/atspi.log 2>&1 &)- sleep 1- (setsid "$CUA" serve >/tmp/cuad.log 2>&1 &); sleep 2- "$CUA" call launch_app '{"name":"galculator"}' > /tmp/launch.json 2>&1- sleep 3- python3 - <<'PY'-import json,re-t=open('/tmp/launch.json').read(); p=None-try: p=json.loads(t).get('pid')-except Exception: pass-if not p:- m=re.search(r'pid (\d+)',t); p=m.group(1) if m else ''-open('/tmp/cstate.pid','w').write(str(p or ''))-PY- echo "fullreset: galculator pid=$(cat $S.pid) remaining_galc=$(pgrep -x galculator|tr '\n' ' ')"- ;;- *)- echo "unknown action: $act"; echo "actions: daemon doctor a11y state click pxclick type btf prep fullreset"- ;;-esacdiff --git a/libs/cua-driver/tests/fixtures/linux-container/derec.sh b/libs/cua-driver/tests/fixtures/linux-container/derec.shdeleted file mode 100755index 6df8381098..0000000000--- a/libs/cua-driver/tests/fixtures/linux-container/derec.sh+++ /dev/null@@ -1,190 +0,0 @@-#!/bin/bash-# cua-driver desktop modality + recording harness — the SSH/VM lane.-#-# Companion to the container lane in this directory: where calc.sh /-# modality_matrix.sh drive ACI containers over the computer-server / `az exec`-# (XFCE, GTK3 galculator), derec.sh drives full Azure VMs over SSH and is the-# only lane that exercises the genuinely different desktops/toolkits:-# - KDE Plasma (X11) with kcalc (Qt)-# - GNOME Shell (X11, console Xorg) with gnome-calculator (GTK4)-# It auto-discovers the live session env (DISPLAY/XAUTHORITY/WAYLAND_DISPLAY/-# dbus) from a running DE process, so the same script works on any of them —-# including the GNOME Mutter / KDE KWin *Wayland* lane (WAYLAND=1), where it-# drives the native Wayland backend off the release build.-#-# derec.sh setup # fresh daemon + launch $APP + snapshot AT-SPI tree-# derec.sh verify # assert the coordinate invariant (see below)-# derec.sh keytest # foreground-keyboard EFFECT check (3*4=12)-# derec.sh vclick 789 C 7 8 9 # vision pixel-click EFFECT check (no element_index)-# derec.sh record i.. i.. # record start_recording -> element clicks -> stop-# derec.sh env # print the discovered session env-#-# APP defaults to gnome-calculator; set APP=kcalc / galculator for the others.-# WAYLAND=1 selects the Wayland lane (GTK4 gnome-calculator on Mutter).-#-# COORDINATE INVARIANT (the GTK4 regression guard):-# GTK4's AT-SPI returns GetExtents(SCREEN) as (0,0) for every widget, which-# used to collapse every element `frame` to the window corner. The fix queries-# CoordType::Window + reconstructs screen = x11_origin + _GTK_FRAME_EXTENTS +-# WINDOW. `verify` asserts the observable invariant: per-button frames are-# DISTINCT (not collapsed) and every button center lies inside the window's-# X11 rect, with "7" left of "8" on the same row when both exist.-# Use the caller's $HOME; the fallback is the GNOME/KDE VM lane's user. Override-# by exporting HOME (or running as that user) on a differently-laid-out box.-export HOME="${HOME:-/home/fbonacci}"-W=$HOME/derec; mkdir -p "$W"-DE_PID=""-for p in gnome-shell plasmashell kwin_x11 metacity xfce4-session; do- DE_PID=$(pgrep -x "$p" | head -1); [ -n "$DE_PID" ] && break-done-if [ -n "$DE_PID" ]; then- while IFS= read -r kv; do export "$kv"; done < <(tr '\0' '\n' </proc/$DE_PID/environ 2>/dev/null | grep -E '^(DISPLAY|XAUTHORITY|DBUS_SESSION_BUS_ADDRESS|XDG_RUNTIME_DIR)=')-fi-# Lane selector (mirrors APP=): WAYLAND=1 engages the native Wayland backend.-# `is_wayland()` needs WAYLAND_DISPLAY set, DISPLAY UNSET, and the opt-in env;-# GTK apps need the wayland GDK backend; and the daemon must be the release-# build (the .local/bin copy can lag the Wayland fixes). gnome-shell's environ-# (discovered above) already supplied WAYLAND_DISPLAY/XDG_RUNTIME_DIR/DBUS.-if [ -n "$WAYLAND" ]; then- export CUA_DRIVER_RS_ENABLE_WAYLAND=1- export WAYLAND_DISPLAY="${WAYLAND_DISPLAY:-wayland-0}"- unset DISPLAY- # Software-GL + wayland backend so GTK4 apps render on the GPU-less VM and- # are spawned as Wayland clients (children of the daemon inherit this env).- export GDK_BACKEND=wayland LIBGL_ALWAYS_SOFTWARE=1 GSK_RENDERER=cairo- CUA="${CUA_BIN:-$HOME/cua-rust/target/release/cua-driver}"-else- [ -n "$DISP" ] && export DISPLAY="$DISP"- export DISPLAY="${DISPLAY:-:0}"- CUA="${CUA_BIN:-$HOME/.local/bin/cua-driver}"-fi-APP="${APP:-gnome-calculator}"; SESS=demo--resolve_pid(){ python3 -c "import json,re-t=open('$W/launch.json').read(); p=None-try: p=json.loads(t).get('pid')-except: pass-if not p:- m=re.search(r'pid (\d+)',t); p=m.group(1) if m else ''-print(p or '')"; }--case "$1" in- env) echo "lane=$([ -n "$WAYLAND" ] && echo wayland || echo x11) DISPLAY=${DISPLAY:-<unset>} WAYLAND_DISPLAY=${WAYLAND_DISPLAY:-<unset>} XAUTHORITY=$XAUTHORITY dbus=${DBUS_SESSION_BUS_ADDRESS:+set} CUA=$CUA APP=$APP DE_PID=$DE_PID";;- setup)- pkill -f 'cua-driver serve' 2>/dev/null; pkill -f "$APP" 2>/dev/null; sleep 2- rm -f ~/.cache/cua-driver/cua-driver.sock- (setsid "$CUA" serve >$W/cuad.log 2>&1 &); sleep 2- "$CUA" call launch_app "{\"name\":\"$APP\"}" >$W/launch.json 2>&1; sleep 4- PID=$(resolve_pid); echo "$PID" >$W/pid- "$CUA" call list_windows "{\"pid\":$PID}" >$W/win.json 2>&1- WID=$(python3 -c "import json;ws=json.load(open('$W/win.json')).get('windows',[]);print(ws[0]['window_id'] if ws else '')" 2>/dev/null)- echo "$WID" >$W/wid- echo "app=$APP pid=$PID wid=$WID DISPLAY=$DISPLAY"- "$CUA" call get_window_state "{\"pid\":$PID,\"window_id\":$WID,\"capture_mode\":\"ax\"}" >$W/state.json 2>&1- python3 -c "import json-d=json.load(open('$W/state.json')); els=d.get('elements',[])-print('elements',len(els),'degraded',d.get('degraded'))-for e in els:- l=e.get('label')- if l and len(str(l))<=4: print(e.get('element_index'),repr(l))" 2>&1 | head -40- ;;- verify) # assert the coordinate invariant against the X11 window geometry- WID=$(cat $W/wid)- read WX WY WW WH < <(xwininfo -id "$WID" 2>/dev/null | awk '/Absolute upper-left X/{x=$NF}/Absolute upper-left Y/{y=$NF}/Width:/{w=$NF}/Height:/{h=$NF}END{print x,y,w,h}')- FE=$(xprop -id "$WID" _GTK_FRAME_EXTENTS 2>/dev/null | grep -oE '= .*' | tr -d ' =')- echo "app=$APP window=($WX,$WY) ${WW}x${WH} _GTK_FRAME_EXTENTS=${FE:-absent}"- python3 - "$WX" "$WY" "$WW" "$WH" "$W/state.json" <<'PY'-import json,sys-wx,wy,ww,wh=map(int,sys.argv[1:5]); d=json.load(open(sys.argv[5]))-btns=[]-for e in d.get('elements',[]):- l=str(e.get('label')); r=str(e.get('role','')).lower(); f=e.get('frame')- if f and 'button' in r and len(l)<=3:- btns.append((l, f['x']+f['w']//2, f['y']+f['h']//2))-fails=[]-xs={cx for _,cx,_ in btns}-if len(btns) >= 3 and len(xs) < 2:- fails.append(f'COLLAPSED: {len(btns)} buttons all at x={xs} (GTK4 (0,0) regression)')-for l,cx,cy in btns:- if not (wx-2 <= cx <= wx+ww+2 and wy-2 <= cy <= wy+wh+2):- fails.append(f'OUT-OF-WINDOW: {l!r} center=({cx},{cy}) outside [{wx},{wx+ww}]x[{wy},{wy+wh}]')-g={l:(cx,cy) for l,cx,cy in btns}-if '7' in g and '8' in g:- (x7,y7),(x8,y8)=g['7'],g['8']- if not (x8 > x7 and abs(y8-y7) <= 6):- fails.append(f"ROW: 7={g['7']} 8={g['8']} not left-to-right on one row")-print(f'buttons checked: {len(btns)}')-print('COORD INVARIANT:', 'PASS' if not fails else 'FAIL')-for f in fails[:8]: print(' -', f)-sys.exit(0 if not fails else 1)-PY- ;;- keytest) # foreground-keyboard EFFECT check: AC, 3 * 4 = -> display reads 12- # Guards the Xvnc XTEST keyboard fix (round-trip-before-close + shift-level- # auto-Shift so '*' multiplies instead of typing '8'). Effect-confirmed, not- # status-confirmed: scans the AT-SPI tree for the result "12" so it generalises- # across kcalc (Qt), gnome-calculator (GTK4) and galculator (GTK3), each of- # which exposes its LCD under a different role.- PID=$(cat $W/pid); WID=$(cat $W/wid)- "$CUA" call press_key "{\"pid\":$PID,\"window_id\":$WID,\"key\":\"Escape\",\"delivery_mode\":\"foreground\"}" >/dev/null 2>&1; sleep 0.4- for k in 3 asterisk 4 equal; do- "$CUA" call press_key "{\"pid\":$PID,\"window_id\":$WID,\"key\":\"$k\",\"delivery_mode\":\"foreground\"}" >/dev/null 2>&1- sleep 0.5- done- sleep 0.4- "$CUA" call get_window_state "{\"pid\":$PID,\"window_id\":$WID,\"capture_mode\":\"ax\"}" 2>/dev/null | python3 -c "import json,sys-d=json.load(sys.stdin)-hit=any(str(e.get('label')).strip()=='12' or str(e.get('value')).strip()=='12' for e in d.get('elements',[]))-print('FOREGROUND KEY EFFECT:', 'PASS (3*4=12 landed)' if hit else 'FAIL (display != 12)')-sys.exit(0 if hit else 1)"- ;;- vclick) # vclick <expected> <label>... VISION pixel-click EFFECT check.- # Clicks each labelled button by its SCREEN-frame CENTER via {x,y} (NO- # element_index), then confirms the calculator result — proving pixel/vision- # coordinates ACTUATE, not GTK4-false-succeed on the inner label. Works on- # X11 and Wayland; on Wayland the rung is `wayland_atspi` (screen pixel ->- # covering element -> element_index doAction), since Mutter drops synthetic- # pointer events. e.g. WAYLAND=1 derec.sh vclick 789 C 7 8 9- PID=$(cat $W/pid); WID=$(cat $W/wid); shift- EXPECT="$1"; shift- "$CUA" call start_session "{\"session\":\"$SESS\"}" >/dev/null 2>&1- for lbl in "$@"; do- read CX CY < <(python3 -c "import json-d=json.load(open('$W/state.json'))-for e in d['elements']:- if str(e.get('label'))=='$lbl' and 'button' in str(e.get('role','')).lower():- f=e.get('frame')- if f: print(int(f['x']+f['w']/2), int(f['y']+f['h']/2))- break")- if [ -z "$CX" ]; then echo " '$lbl' -> no frame (skipped)"; continue; fi- R=$("$CUA" call click "{\"pid\":$PID,\"window_id\":$WID,\"x\":$CX,\"y\":$CY,\"delivery_mode\":\"background\",\"session\":\"$SESS\"}" 2>&1)- PV=$(echo "$R" | python3 -c "import json,sys-try: print(json.load(sys.stdin).get('path','?'))-except: print('?')" 2>/dev/null)- echo " vision-click '$lbl' @($CX,$CY) -> path=$PV"- sleep 0.7- done- sleep 0.5- "$CUA" call get_window_state "{\"pid\":$PID,\"window_id\":$WID,\"capture_mode\":\"ax\"}" 2>/dev/null | python3 -c "import json,sys-d=json.load(sys.stdin); exp='$EXPECT'-hit=any(str(e.get('label')).strip()==exp or str(e.get('value')).strip()==exp for e in d.get('elements',[]))-print('VISION CLICK EFFECT:', f'PASS ({exp} via pixel coords)' if hit else f'FAIL (display != {exp})')-sys.exit(0 if hit else 1)"- ;;- record) # record <idx>... start_recording -> element clicks (in order) -> stop- PID=$(cat $W/pid); WID=$(cat $W/wid); shift- rm -rf $W/rec; mkdir -p $W/rec- "$CUA" call start_session "{\"session\":\"$SESS\"}" >/dev/null 2>&1- "$CUA" call start_recording "{\"output_dir\":\"$W/rec\",\"record_video\":true}" 2>&1 | head -1- sleep 2- for idx in "$@"; do- "$CUA" call click "{\"pid\":$PID,\"window_id\":$WID,\"element_index\":$idx,\"delivery_mode\":\"background\",\"session\":\"$SESS\"}" >/dev/null 2>&1- sleep 1.3- done- sleep 1.5- "$CUA" call stop_recording '{}' 2>&1 | head -1- ls -la $W/rec/recording.mp4 2>&1 | tail -1- ;;- *) echo "usage: derec.sh setup|verify|keytest|vclick <expected> <label>...|record <idx>...|env- APP=gnome-calculator|kcalc|galculator WAYLAND=1 for the Mutter/KWin Wayland lane";;-esacdiff --git a/libs/cua-driver/tests/fixtures/linux-container/modality_matrix.sh b/libs/cua-driver/tests/fixtures/linux-container/modality_matrix.shdeleted file mode 100755index a9eed91696..0000000000--- a/libs/cua-driver/tests/fixtures/linux-container/modality_matrix.sh+++ /dev/null@@ -1,151 +0,0 @@-#!/usr/bin/env bash-# Full modality matrix for the XFCE container, driven via `az container exec`-# (positional args only — the JSON is built inside the script). Proves the-# session-bus auto-discovery fix (the daemon is started with-# DBUS_SESSION_BUS_ADDRESS UNSET) and exercises every input tool across-# background/foreground × ax/vision. Screenshots land in /tmp/mm_*.png.-#-# modality_matrix.sh setup # fresh daemon (NO dbus env) + a11y + galculator + snapshot-# modality_matrix.sh matrix # run every tool × bg/fg, print path+result-# modality_matrix.sh shot <name> # screenshot -> /tmp/mm_<name>.png-# modality_matrix.sh b64 <name> # print a saved screenshot as base64 (host decodes)-set -uo pipefail-# Paths derive from $HOME so the same harness drives both container lanes:-# trycua/cua-xfce (user `cua`) and trycua/cua-ubuntu / Kasm (user `kasm-user`).-# Override CUA / XAUTHORITY / DISPLAY via env if your layout differs.-export DISPLAY=${DISPLAY:-:1}-export XAUTHORITY=${XAUTHORITY:-$HOME/.Xauthority}-CUA=${CUA:-$HOME/.local/bin/cua-driver}-S=/tmp/cstate--pidf() { cat "$S.pid" 2>/dev/null; }-widf() { cat "$S.wid" 2>/dev/null; }--snap() { # snapshot AT-SPI tree -> /tmp/state.json, print element count + degraded- local PID WID; PID=$(pidf); WID=$(widf)- "$CUA" call get_window_state "{\"pid\":$PID,\"window_id\":$WID,\"capture_mode\":\"ax\"}" >/tmp/state.json 2>&1- python3 - <<'PY'-import json-try: d=json.load(open('/tmp/state.json'))-except Exception as e: print(" STATE PARSE FAIL:",str(e)[:120]); raise SystemExit-els=d.get('elements',[])-print(f" AT-SPI elements={len(els)} degraded={d.get('degraded')}")-if d.get('degraded'): print(" degraded_reason:",str(d.get('degraded_reason'))[:120])-PY-}--call() { # call <label> <tool> <json> ; prints "<label> | path=.. ok=.."- local label="$1" tool="$2" body="$3"- "$CUA" call "$tool" "$body" >/tmp/last.json 2>&1- python3 - "$label" <<'PY'-import json,sys-lbl=sys.argv[1]-try: d=json.load(open('/tmp/last.json'))-except Exception:- print(f" {lbl:30s} | RAW: "+open('/tmp/last.json').read()[:90].replace(chr(10),' ')); raise SystemExit-path=d.get('path') or d.get('delivery_path') or '-'-ok=d.get('ok', d.get('success', d.get('clicked', d.get('typed','?'))))-ver=d.get('verified')-extra=f" verified={ver}" if ver is not None else ""-print(f" {lbl:30s} | path={path:14s} ok={ok}{extra}")-PY-}--case "${1:-help}" in- setup)- pkill -9 -f 'cua-driver serve' 2>/dev/null; pkill -9 -f galculator 2>/dev/null; sleep 2- rm -f /home/cua/.cache/cua-driver/cua-driver.sock- gsettings set org.gnome.desktop.interface toolkit-accessibility true 2>/dev/null- pgrep -f at-spi-bus-launcher >/dev/null || (setsid /usr/libexec/at-spi-bus-launcher --launch-immediately >/tmp/atspi.log 2>&1 &)- sleep 1- # THE TEST: start the daemon with DBUS_SESSION_BUS_ADDRESS removed from its- # environment. If AT-SPI still populates below, auto-discovery worked.- env -u DBUS_SESSION_BUS_ADDRESS setsid "$CUA" serve >/tmp/cuad.log 2>&1 &- sleep 3- echo "=== daemon started with DBUS_SESSION_BUS_ADDRESS UNSET ==="- echo "daemon env DBUS set? $(env -u DBUS_SESSION_BUS_ADDRESS bash -c 'echo ${DBUS_SESSION_BUS_ADDRESS:-<unset>}')"- grep -iE "session bus|adopted the desktop|DBUS_SESSION" /tmp/cuad.log | head -3 || echo "(no discovery log line)"- "$CUA" call launch_app '{"name":"galculator"}' >/tmp/launch.json 2>&1- sleep 3- PID=$(python3 -c "import json,re;t=open('/tmp/launch.json').read()-try: p=json.loads(t).get('pid')-except Exception: p=None-if not p:- import re; m=re.search(r'pid (\d+)',t); p=m.group(1) if m else ''-print(p or '')")- echo "$PID" > "$S.pid"- "$CUA" call list_windows "{\"pid\":$PID}" >/tmp/win.json 2>&1- WID=$(python3 -c "import json;ws=json.load(open('/tmp/win.json')).get('windows',[]);print(ws[0]['window_id'] if ws else '')" 2>/dev/null)- echo "$WID" > "$S.wid"- echo "galculator pid=$PID wid=$WID"- echo "=== AT-SPI tree (proves auto-discovery if elements>0) ==="- snap- ;;-- matrix)- PID=$(pidf); WID=$(widf)- echo "=== pid=$PID wid=$WID ==="- # resolve a couple of element indices + their pixel centers from the tree- read -r E7 PX7 PY7 E_PLUS < <(python3 - <<'PY'-import json-d=json.load(open('/tmp/state.json')); win=json.load(open('/tmp/win.json'))['windows'][0]-def find(lbl):- for e in d.get('elements',[]):- if str(e.get('label'))==lbl: return e- return None-e7=find('7'); ep=find('+')-def ctr(e):- f=e.get('frame');- return (int(f['x']+f['w']/2), int(f['y']+f['h']/2)) if f else (-1,-1)-x7,y7=ctr(e7) if e7 else (-1,-1)-print(e7.get('element_index') if e7 else -1, x7, y7, ep.get('element_index') if ep else -1)-PY-)- echo "indices: 7->$E7 (px $PX7,$PY7) +->$E_PLUS"- echo "--- AX modality (element_index) ---"- call "click 7 [ax/bg]" click "{\"pid\":$PID,\"window_id\":$WID,\"element_index\":$E7,\"delivery_mode\":\"background\"}"- call "click + [ax/fg]" click "{\"pid\":$PID,\"window_id\":$WID,\"element_index\":$E_PLUS,\"delivery_mode\":\"foreground\"}"- call "double_click 7 [ax/bg]" double_click "{\"pid\":$PID,\"window_id\":$WID,\"element_index\":$E7,\"delivery_mode\":\"background\"}"- call "right_click 7 [ax/bg]" right_click "{\"pid\":$PID,\"window_id\":$WID,\"element_index\":$E7,\"delivery_mode\":\"background\"}"- echo "--- VISION modality (pixel coords) ---"- call "click px7 [vision/bg]" click "{\"pid\":$PID,\"window_id\":$WID,\"x\":$PX7,\"y\":$PY7,\"delivery_mode\":\"background\"}"- call "click px7 [vision/fg]" click "{\"pid\":$PID,\"window_id\":$WID,\"x\":$PX7,\"y\":$PY7,\"delivery_mode\":\"foreground\"}"- call "double_click px7 [v/bg]" double_click "{\"pid\":$PID,\"window_id\":$WID,\"x\":$PX7,\"y\":$PY7,\"delivery_mode\":\"background\"}"- echo "--- keyboard / scroll ---"- call "type 789 [bg]" type_text "{\"pid\":$PID,\"text\":\"789\",\"delivery_mode\":\"background\"}"- call "type 789 [fg]" type_text "{\"pid\":$PID,\"text\":\"789\",\"delivery_mode\":\"foreground\"}"- call "press_key Return [bg]" press_key "{\"pid\":$PID,\"window_id\":$WID,\"key\":\"Return\",\"delivery_mode\":\"background\"}"- call "press_key Return [fg]" press_key "{\"pid\":$PID,\"window_id\":$WID,\"key\":\"Return\",\"delivery_mode\":\"foreground\"}"- call "hotkey ctrl+c [bg]" hotkey "{\"pid\":$PID,\"window_id\":$WID,\"keys\":[\"ctrl\",\"c\"],\"delivery_mode\":\"background\"}"- call "scroll down [bg]" scroll "{\"pid\":$PID,\"window_id\":$WID,\"direction\":\"down\",\"amount\":3,\"delivery_mode\":\"background\"}"- call "bring_to_front (EWMH)" bring_to_front "{\"pid\":$PID,\"window_id\":$WID}"- ;;-- shot)- # Grab the X display directly (no `screenshot` tool; ffmpeg x11grab is present).- ffmpeg -y -loglevel error -f x11grab -i :1 -frames:v 1 "/tmp/mm_${2:-shot}.png" 2>/tmp/shot.err- ls -la "/tmp/mm_${2:-shot}.png" 2>&1 | tail -1- [ -s /tmp/shot.err ] && head -c 200 /tmp/shot.err- ;;-- seq) # seq <label>... background element-click each labeled button (clean demo)- shift; PID=$(pidf); WID=$(widf)- for lbl in "$@"; do- IDX=$(python3 - "$lbl" <<'PY'-import json,sys-d=json.load(open('/tmp/state.json'))-for e in d.get('elements',[]):- if str(e.get('label'))==sys.argv[1]: print(e['element_index']); break-PY-)- [ -z "$IDX" ] && { echo " '$lbl' -> no element"; continue; }- "$CUA" call click "{\"pid\":$PID,\"window_id\":$WID,\"element_index\":$IDX,\"delivery_mode\":\"background\"}" >/dev/null 2>&1- echo " '$lbl' -> element $IDX clicked (bg)"- sleep 0.4- done- ;;-- b64) base64 -w0 "/tmp/mm_${2:-shot}.png" 2>/dev/null ;;-- *) echo "usage: modality_matrix.sh setup|matrix|shot <name>|b64 <name>" ;;-esacdiff --git a/libs/cua-driver/tests/fixtures/linux-container/validate.sh b/libs/cua-driver/tests/fixtures/linux-container/validate.shdeleted file mode 100755index 5c73c2793b..0000000000--- a/libs/cua-driver/tests/fixtures/linux-container/validate.sh+++ /dev/null@@ -1,32 +0,0 @@-#!/usr/bin/env bash-# Build + test cua-driver on the XFCE container, detached (survives `az-# container exec`). Output goes to /tmp/validate.log.-# validate.sh run # launch detached-# validate.sh log # print the log-set -uo pipefail-RUST=/opt/cua/libs/cua-driver/rust-LOG=/tmp/validate.log-export PATH=/home/cua/.cargo/bin:$PATH-export CARGO_TERM_COLOR=never--_work() {- echo "=== validate start $(date -u +%H:%M:%S) ==="- cd "$RUST" || { echo "no rust dir"; exit 9; }- echo "HEAD: $(git -C /opt/cua rev-parse --short HEAD)"- echo "--- cargo build -p platform-linux ---"- cargo build -p platform-linux 2>&1 | tail -25; echo "build_rc=${PIPESTATUS[0]}"- echo "--- cargo build -p cua-driver (bin) ---"- cargo build -p cua-driver 2>&1 | tail -15; echo "bin_rc=${PIPESTATUS[0]}"- echo "--- cargo test -p platform-linux session_bus ---"- cargo test -p platform-linux session_bus 2>&1 | tail -18; echo "sbtest_rc=${PIPESTATUS[0]}"- echo "--- cargo test --test modality_dispatch_linux_test ---"- cargo test -p cua-driver --test modality_dispatch_linux_test 2>&1 | tail -18; echo "dispatch_rc=${PIPESTATUS[0]}"- echo "VALIDATE_DONE $(date -u +%H:%M:%S)"-}--case "${1:-run}" in- run) : > "$LOG"; setsid bash "$0" _work >>"$LOG" 2>&1 </dev/null & echo "launched pid $! -> $LOG" ;;- _work) _work ;;- log) cat "$LOG" 2>/dev/null ;;- *) echo "usage: validate.sh run|log" ;;-esacdiff --git a/libs/cua-driver/tests/fixtures/modality-recordings/FINDINGS.md b/libs/cua-driver/tests/fixtures/modality-recordings/FINDINGS.mddeleted file mode 100644index d1dbcd996b..0000000000--- a/libs/cua-driver/tests/fixtures/modality-recordings/FINDINGS.md+++ /dev/null@@ -1,245 +0,0 @@-# Modality recordings — findings (cua-driver behavior surfaced by the per-action verifier)--These recordings run the 8-action matrix (click, double-click, right-click, drag, scroll,-set_value, type, press-key) against a controlled harness on Windows (WPF) and Linux (GTK3), in-5 modalities each (ax-fg, ax-bg, px-fg, px-bg, px-desktop). Each action now carries-**two independent measurements**:--- **✓ worked / ✗ no-op** — did the action change the harness's own state (verified by reading the- harness status: `agreed=`, `slider_value=`, `last_action=`, `mirror=`, `menu_action=`,- `scroll_offset=`). Read via UIAutomation on Windows, via a harness state file on Linux.-- **held / STOLE FOCUS** — the no-foreground contract.--Principle followed: the harness/recorder stays honest (no overfitting); real gaps are documented-here as cua-driver work, to be fixed in the driver first and then re-tested.--**Historical note:** the Electron **7/8** and Linux **3/8** violation figures below-are recorder-baseline artifacts from an earlier pass and have since been addressed.-See the newer fix note later in this file and the canonical-`reference/cua-driver/modality-test-suite.mdx`.--## What the verifier surfaced (to triage as driver work)--### Windows / WPF (ax-bg, representative) — after fixes: 5/7 land-| action | effect | note |-|--------|--------|------|-| left-click checkbox | ✓ worked | UIA toggle |-| double-click button | ✓ worked | last_action=double_click |-| right-click | ✗ no-op (clean) | **Root cause: off-screen target, not a driver coordinate bug.** At 1024×768 the panel forces the harness to 556px wide, so the form reflows taller than the screen and the right-click button lands at y≈786 (below the screen). The synthetic tap previously clamped onto the **taskbar** (opened its menu). **DRIVER FIX:** `point_in_window_bounds()` now refuses a click whose resolved point is outside its window and returns a clear error — no more taskbar misfire (frame-verified). To actually *exercise* right-click needs a screen tall enough to show the control (this RDP session won't take a programmatic resize). |-| drag slider | ✓ worked | slider_value 0→48 |-| scroll | ✗ no-op (clean) | Same off-screen root cause (scroll-tall pane is lower still); now guarded, not misfiring. |-| set_value | ✓ worked | mirror=set-by-cua |-| type | ✓ worked | **Fixed:** the recorder now clicks the text box (focus) before `type_text`; type targets the focused control, and `set_value` doesn't focus, so without this it was a no-op. |-| press-key (Tab) | n/a | no asserted effect |--**Driver fix shipped (this branch):** `platform-windows` — `point_in_window_bounds()` + guards in the-click / double_click / right_click element paths. Investigated and ruled out: UIA `GetClickablePoint`-for cursor-centering (returns the same rect center or fails), and SetFocus-to-scroll-into-view (would-steal foreground, breaking the very contract these bg runs measure).--### Linux / GTK3-- AT-SPI/element actions (single-click, set_value, type) **✓ work**.-- All **pixel-based** actions (double-click, right-click, drag, scroll, every coordinate click)- **✗ no-op**: cua-driver's Linux input is **XSendEvent** (synthetic, no focus steal — required for- the contract), but **GTK ignores synthetic XSendEvent** (`send_event` flag). Documented in- `platform-linux/src/input/mod.rs`. Real limitation; not a recorder bug.--### Linux / Electron (Chromium, via AT-SPI)--The cross-platform Electron harness runs on Linux straight from `node_modules` (no packaging),-launched `electron . --no-sandbox --disable-gpu --force-renderer-accessibility` under-Xvfb + a dbus session. `--force-renderer-accessibility` makes Chromium publish its full-**web-AX tree on AT-SPI**, which `get_window_state` returns cleanly alongside the-screenshot (check box "I agree", slider, entry "type here", the click-target section, and every-`agreed=`/`slider_value=`/`last_action=`/`mirror=` status label — 72 elements). Same verifier as-Windows, but reading AT-SPI text instead of UIA. Recorder: `linux/lin-rec-electron.py`-(+ `lin-run-electron.sh`, `lin-dashboard-electron.html`, `lin-all-electron.sh`).--| mode | effects landed | landed actions | focus contract (stole) |-|------|----------------|----------------|------------------------|-| ax-fg | 3/6 | click, drag, type | foreground (8 actions) |-| ax-bg | 2/6 | click, type | **3/8 stole** (set_value, type, press-key) |-| px-fg | 1/5 | double-click | foreground (7 actions) |-| px-bg | 1/5 | double-click | **6/7 stole** |-| px-desktop | 1/2 | click | foreground (4 actions) |--**Headline — Linux Electron breaks the no-foreground contract, but far less than Windows-Electron.** Windows Electron stole **7/8** in ax-bg (Chromium self-foregrounds on nearly every-action). Linux Electron steals **3/8 in ax-bg** — only the keyboard/focus actions (`set_value`,-`type`, `press-key`) pull the window frontmost via the X focus path; the AX pointer actions-(click/double/right/drag/scroll) all **held**. In **px-bg** it steals **6/7** (pixel dispatch-is coordinate-injection that foregrounds Chromium). Effect coverage mirrors the documented Linux-synthetic-input limit: AX `click`/`type` land; AX `double_click`/`right_click`/`drag`/`scroll` and-`set_value` are no-ops on the Chromium controls via AT-SPI, while **pixel** `double_click` *does*-fire on the click-target (`last_action=double_click`, frame-verified in px-fg/bg). All 5 modes-frame-verified; saved as `linux-electron-{ax-fg,ax-bg,px-fg,px-bg,px-desktop}.mp4`.--## Agent-cursor centering (user-reported)-The agent cursor lands off-center on the checkbox. Root cause: the WPF checkbox's AX-BoundingRectangle is the **full-width row** (≈803px), so its center is mid-row. Verified that-UIA `GetClickablePoint` does **not** help — it either fails (slider) or returns the same rect-center (text box). So this is **not a driver defect**: the bounds are reported faithfully and the-element action works regardless of cursor position. The visual is a function of the control's-geometry (a stretched CheckBox). Most-honest fix would be the harness laying the checkbox out at-content width (how real apps render it); a driver left-bias heuristic would be unreliable.--## Cross-toolkit coverage matrix (Windows, ax-bg)--Same recorder (parameterized by `-Toolkit`), same 8-action plan, one ax-bg pass each. This is the-high-signal output: which cua-driver actions actually LAND per toolkit, and whether the no-foreground-contract holds. (right-click + scroll are ✗ across the board here: the harnesses either lack those-controls or they're off-screen at the 556px layout — see the off-screen guard above.)--| Toolkit | effects landed | landed actions | focus contract (stole) | note |-|----------|---------------|------------------------------------------|------------------------|------|-| WPF | **5/7** | click, double, drag, set_value, type | 2/8 | fullest harness |-| WinUI3 | **3/7** | left-click(checkbox), scroll, set_value | 0/7 | re-recorded with parity controls; double/right/drag don't land (see WinUI3 dispatch below) |-| WebView2 | **4/7** | double, right, drag, set_value | 0/7 | re-recorded; left-click(checkbox) + scroll are honest no-ops (checkbox below the fold, web won't scroll in ax mode) |-| Electron | **5/7** | click, double, drag, set_value, type | **7/8** | **Electron/Chromium self-foregrounds → no-foreground contract VIOLATED** |--**Headline:** the no-foreground contract holds on WPF / WinUI3 / WebView2, but **breaks on Electron**-(7/8 actions stole focus — Chromium foregrounds itself; consistent with the #1984 dispatch note in the-code). The verifier reads each toolkit's own status labels (WPF/WinUI3: UIAutomation; WebView2/Electron:-Chromium web-AX text via a substring window-title match because their titles carry a `[cdp=NNNN]` suffix).--WinUI3 + WebView2 were **re-recorded** after the parity controls landed (frame-verified). Recorder fixes-that made WebView2 work at all: the web DOM only surfaces when `WEBVIEW2_ADDITIONAL_BROWSER_ARGUMENTS=---force-renderer-accessibility` is set (otherwise `get_window_state` returns only the chrome frame →-every web action resolves to nothing → the prior SIZE=0/empty-MP4 run); and the daemon's ffmpeg probe-missed the binary because the recorder runs as a different user than the one WinGet installed it under, so-ffmpeg must be on PATH before `serve`. Recording on the VM also requires Session 2 attached/active-(`tscon … /dest:console`) — a disconnected session blanks GPU content and fails `gdigrab`.--**WinUI3 double/right-click — confirmed not driver-fixable via the WPF path.** AX `double_click`/-`right_click` on the WinUI3 click-target produce `last_action=none, clicks=0` (no Click/DoubleTapped/-RightTapped), while the *identical* actions land on WebView2 — so the harness wiring is correct; the gap-is WinUI3-specific dispatch. Matches the `platform-windows/src/input/dispatch.rs` note: routing WinUI3-through the WPF synthetic-pen/coordinate path neither lands double/right NOR holds the contract (measured:-regressed ax-bg 0/8→8/8 stolen). Single left-click works via UIA Invoke. A real fix needs a WinUI3-specific-input path targeting the composition input-site (DirectComposition/ContentIsland InputSite), not-PostMessage-to-top-HWND; the same gap explains WinUI3 drag-slider being a no-op.--Evidence: `matrix-{winui3,webview2,electron}-ax-bg.mp4` (+ the WPF set) on the Desktop.--## Driver fixes shipped this round (cross-platform)--- **Windows — cached UIA element use-after-free** (`platform-windows`): the element cache handed out a bare- COM pointer; under concurrent sessions a `get_window_state` snapshot-replace could `Release` it mid-action- (click/type/set_value) → daemon crash. Now a `RetainedElement` guard `AddRef`s under the cache lock and- `Release`s on drop — the Windows port of the macOS #1796 retain-under-lock fix. Compile-verified.-- **Linux — GTK left-click + value-only widgets** (`platform-linux`): pixel left-clicks now land via AT-SPI- hit-test + `doAction` (GTK drops synthetic X11 events; XTEST core events don't reach its XInput2 path) —- without stealing focus. Sliders/scroll bars (Value interface, no Action) now surface in `get_window_state`- (`is_indexable = actions || has_value`), so `set_value` drives them. Verified on the Linux VM.-- **macOS — numeric `set_value`** (`platform-macos`): CFNumber write for NSSlider, AXIncrement/AXDecrement- stepping fallback for SwiftUI sliders. Verified live (AppKit 0→50, SwiftUI 0→50).--## Legacy modal popups (WPF) — driver CAN open + list them--Confirmed: a **background `click` via UIA Invoke** fires the harness's popup buttons *even when they-are off-screen* (y≈876–1026 on the 768px display), and `list_windows` then enumerates the dialogs:-`Open MessageBox` → **"Harness MessageBox"**, `Open Owned Window` → **"Harness Owned Popup"**,-`Open Layered Popup` → **"Harness Layered Popup"**.--This briefly regressed: the first cut of the off-screen guard (above) sat *before* the dispatch-branch and so blocked the UIA-Invoke path too (which needs no coordinates). **Fix:** the guard now-applies only to the coordinate-delivery paths (foreground SendInput tap + background coordinate-injection); UIA Invoke runs unguarded, so opening a modal from an off-screen button works again.--## macOS (AppKit) — 5 modes, contract HOLDS--Recorded on the macOS host (AppKit harness, ScreenCaptureKit). TCC already granted. Saved locally-only (`macos-appkit-*.mp4`) — these record a personal screen, not uploaded.--| mode | effects | focus contract |-|------|---------|----------------|-| ax-bg | 5/7 | **0/8 stole** |-| px-bg | 4/6 | **0/7 stole** |-| ax-fg / px-fg / px-desktop | 5/7 / 4/6 / 2/3 | foreground |--Headline: in **px-bg**, pixel left-click/double/drag landed on the harness while Chrome stayed-frontmost — **0 focus steals**; the no-foreground contract holds on macOS for both AX and pixel-dispatch. Honest macOS gaps surfaced: the NSView click-target isn't in the AX tree (needs pixels);-pixel right-click never fires `rightMouseDown`; pixel scroll doesn't move `NSScrollView` (element-scroll does); NSButton ignores synthetic pixel clicks (AX `element_index` targets it); and there's-no true window-less screen click on macOS (`click` requires `pid`; pixel dispatch is window-anchored).-Two driver gotchas to flag: **`end_session` poisons a reused session id** (subsequent actions-silently no-op), and AppKit **window height drifts between launches** (store targets as window-local-points, convert to live screenshot px).--## Control-parity pass (all surfaces → WPF baseline)--Brought every harness up to the WPF 6-control set (checkbox, click-target L/R/double,-slider, scroll-target, text-input, context-menu) so the same 8-action matrix is-exercisable everywhere; `shared/scenarios.json` + `shared/web/index.html` are the-single source of truth. Harness DOM/UI parity landed for GTK3, WinUI3, the shared web-(WebView2/Electron/WKWebView), and macOS AppKit/SwiftUI.--**scroll_target — Linux Electron (re-recorded, frame-verified).** The shared web-`scroll-tall` clipped region now resolves on Linux Electron in all 5 modes via the-Chromium web-AX tree; the recorder aims the scroll at that section and checks a real-`scroll_offset=` delta. **Result: scroll is a no-op in every mode** (`scroll_offset`-stays 0) — the same AT-SPI synthetic-input limit as the other Linux pointer actions-(double/right/drag), for both the element_index and the pixel/vision path. Headline-intact: Linux Electron ax-bg still **stole 3/8** (scroll is now a counted failure, not-`na`). 5 mp4s re-encoded + overwritten (`linux-electron-*.mp4`).--**WebView2 (Windows) — harness restored, full-parity recording deferred.** The shared-web DOM (with scroll-target) is deployed; a clean self-contained re-publish of the-harness was needed after an in-place publish corrupted the bundle (window stopped-resolving). With the clean bundle the window resolves again and `sld`/`txt` map, but the-new click-target/checkbox/scroll/context controls need the WebView2 recorder's resolver-extended (as was done for Linux Electron) before a full-parity recording is meaningful —-deferred as low marginal value: it would only re-confirm the captured headline (Windows-Chromium **steals 7/8**, the matrix-webview2-ax-bg.mp4 evidence).--## macOS — 4 surfaces + numeric-set_value driver fix--Surfaces recorded on the macOS host (local-only, personal screen — never uploaded):-**AppKit, SwiftUI, Electron, WKWebView**, 5 modes each. WKWebView (Apple WebKit, the-native analogue of WebView2) **holds the contract** — confirming the Windows Electron/-WebView2 steal is specific to Windows-Chromium, not WebKit-on-macOS.--**Driver fix shipped (this branch, `platform-macos`):** `set_value` now writes a-**CFNumber** for numeric `AXValue` controls (NSSlider/NSStepper reject a CFString);-falls back to CFString for text fields.--### Live host verification (AppKit + SwiftUI + WKWebView, against the running driver)--The CFNumber fix was verified live after a `install-local` reinstall (the new daemon is-the **Jun-27 post-fix** binary; the installer re-signed the `CuaDriver.app` bundle-ad-hoc and re-granted TCC — `permissions status` reports Accessibility + Screen-Recording both ✅). The daemon was driven directly via the `cua-driver call` CLI.--- **set_value CFNumber fix — VERIFIED on AppKit NSSlider.** `set_value(AXSlider, "50")`- on the AppKit harness now **succeeds** (`✅ Set AXValue on [5] AXSlider`) and- `slider_value` goes **0 → 50** (frame-verified — thumb at midpoint). This is the control- the fix targets; the CFNumber write lands where a CFString write was rejected.-- **SwiftUI slider — NOT a value-type problem; `AXValue` is unsettable there.** The- SwiftUI `AXSlider` exposes only `actions=[increment,decrement]` and rejects *any*- `AXValue` write (CFNumber and the CFString fallback both return `-25200`). So the- CFNumber fix cannot help SwiftUI sliders — driving them needs repeated AXIncrement/- AXDecrement (future driver work), not a value write. (Comment in `set_number_attr`- notes `-25200` as well as `-25201`.)-- **SwiftUI harness — full control parity confirmed.** `get_window_state` exposes all- six WPF-parity controls as actionable AX elements (text field, click-target,- `AXSlider sld-value`, `AXCheckBox "I agree"`, context-menu button, scroll region) and- the status labels (`counter=`, `mirror=`, `last_action=`, `clicks=`, `slider_value=`,- `agreed=`, `menu_action=`, `scroll_offset=`) render as text nodes in `tree_markdown`- for the verifier. An AX checkbox press flipped `agreed=false → true` (frame-verified).-- **WKWebView web-surface scroll — page works, nested overflow div is a no-op.** The- driver's keystroke `scroll` (PageDown on the `AXWebArea`) scrolls the whole page a full- page down (frame-verified). But the nested `scroll-tall` overflow div stays at- `scroll_offset=0`: keystroke-scroll only drives the focused/page scroller, and a CSS- `overflow:auto` div without `tabindex` never takes keyboard focus. Same root cause as the- Linux Electron inner-div no-op; a pixel-wheel scroll (not in the current `scroll` tool)- would be required to drive an arbitrary overflow container.--The two macOS gotchas above (`end_session` reuse poisoning, AppKit window-height drift)-were also filed for the driver.--## Deliverables-10 verified recordings (5 WPF + 5 GTK3) + index, published to the team demo blob-(URL tracked internally — not linked here to keep it out of the public repo).-Additional local-only sets (not uploaded): macOS AppKit/SwiftUI/Electron/WKWebView,-Linux Electron (with scroll-target), and the Windows cross-toolkit matrix.diff --git a/libs/cua-driver/tests/fixtures/modality-recordings/README.md b/libs/cua-driver/tests/fixtures/modality-recordings/README.mddeleted file mode 100644index acb9d04023..0000000000--- a/libs/cua-driver/tests/fixtures/modality-recordings/README.md+++ /dev/null@@ -1,73 +0,0 @@-# cua-driver — modality recordings (Windows WPF + Linux GTK3)--Per-toolkit, per-modality recordings of cua-driver's no-foreground contract.-Each video: the **test harness on the LEFT**, the **`cua-driver-panel` dashboard on the RIGHT**-(always-on-top) showing the run's modality, a live **foreground/background indicator**, and a-**per-action measurement** of the contract (`held` vs `STOLE FOCUS`). The pink/cyan overlay is-cua-driver's per-session agent cursor (no real pointer move).--Five single-modality runs per toolkit:-- **ax-fg** — accessibility-tree actions, app intentionally kept FOREGROUND-- **ax-bg** — accessibility-tree actions, app should stay BACKGROUND (contract measured)-- **px-fg** — pixel-only (screenshot + coordinates), kept FOREGROUND-- **px-bg** — pixel-only, should stay BACKGROUND (contract measured)-- **px-desktop** — whole-screen, window-less screen-pixel actions (no window targeted)--Action set per run: left-click · double-click · right-click · drag · scroll ·-set_value (AX-only) · type · press-key.--Each action carries **two** independent measurements on the dashboard:-- **✓ worked / ✗ no-op** — did the action actually change the app's state (checkbox toggled,- slider moved, text changed)? Verified by reading the harness's own state after each action.-- **held / STOLE FOCUS** — did the action keep the app in the background (the no-foreground contract)?--## Windows — WPF (`wpf-*.mp4`)-Recorded on the Windows VM (Session 2, 1024×768) via cua-driver `start_recording` (ffmpeg gdigrab).--| File | Measured |-|------|----------|-| `wpf-ax-fg.mp4` | foreground-mode, 8 actions |-| `wpf-ax-bg.mp4` | **1/8 actions stole focus** (double-click) |-| `wpf-px-fg.mp4` | foreground-mode, 7 actions |-| `wpf-px-bg.mp4` | **2/7 actions stole focus** (left-click, double-click) |-| `wpf-px-desktop.mp4` | foreground-mode, 4 actions |--## Linux — GTK3 (`gtk3-*.mp4`)-Recorded on the Ubuntu 24.04 VM under a headless **Xvfb + openbox + picom + AT-SPI** stack-(X11; picom composites the agent-cursor overlay), via cua-driver `start_recording` (x11grab).--| File | Effects landed | Focus contract |-|------|----------------|----------------|-| `gtk3-ax-fg.mp4` | 3/7 actions changed the app | foreground-mode, 8 actions |-| `gtk3-ax-bg.mp4` | 3/7 actions changed the app | **3/8 stole focus** |-| `gtk3-px-fg.mp4` | 1/6 actions changed the app | foreground-mode, 7 actions |-| `gtk3-px-bg.mp4` | 1/6 actions changed the app | **0/7 stole focus** |-| `gtk3-px-desktop.mp4` | 1/3 actions changed the app | foreground-mode, 4 actions |--> **Superseded:** GTK background pixel clicks now land via AT-SPI `doAction` at-> point. See `FINDINGS.md` and the canonical-> `reference/cua-driver/modality-test-suite.mdx` (§ Linux) for the current result.--**Key Linux finding (visible in the ✓/✗ column):** cua-driver injects Linux input via **XSendEvent**-(synthetic events delivered to a window without stealing focus — required for the no-foreground-contract). **GTK ignores synthetic XSendEvent events** (the `send_event` flag, a security feature),-so cua-driver's *pixel-based* actions (double-click, right-click, drag, scroll, and all coordinate-clicks) **do not land** on the GTK harness — they're honestly marked ✗ no-op. The *AT-SPI / element*-actions (single-click, set_value, type) use accessibility actions, which GTK honors, so they ✓ work.-This is a real, documented driver behavior (see `platform-linux/src/input/mod.rs`), surfaced here by-the per-action verifier rather than hidden. On Windows/WPF the equivalent synthetic input *is* honored,-which is why those runs land all actions (e.g. the slider visibly moves).--## Reading the result-In the background runs, a `STOLE FOCUS` row means that action brought the app to the foreground-(contract violated for that action); `held` means it stayed in the background. The dashboard-tally at the bottom is the live count. Results differ by toolkit and modality — that variance-is the point: the recordings *measure* the contract honestly rather than asserting it holds.--### Notes / known limitations-- **Linux AT-SPI** exposes only buttons/text/checkbox roles (no sliders/scrollers) and no element- frames, so drag/scroll and all coordinate pixel actions are driven from harness-exported widget- geometry. The drag/scroll actions fire and are measured but may not visibly move those widgets.-- **px-desktop** uses screen-absolute (window-less) clicks; under Xvfb these are not always- delivered faithfully, so some clicks register as no-ops while still being measured.-- macOS (AppKit/SwiftUI) was out of scope for this pass (no recordable session on the host).diff --git a/libs/cua-driver/tests/fixtures/modality-recordings/index.html b/libs/cua-driver/tests/fixtures/modality-recordings/index.htmldeleted file mode 100644index fc1f71d049..0000000000--- a/libs/cua-driver/tests/fixtures/modality-recordings/index.html+++ /dev/null@@ -1,34 +0,0 @@-<!doctype html><html lang="en"><head>-<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">-<title>cua-driver — modality recordings</title><style>- :root{color-scheme:dark}- body{margin:0;background:#0d1117;color:#e6edf3;font-family:-apple-system,Segoe UI,Roboto,sans-serif;padding:16px;max-width:760px;margin:0 auto}- h1{font-size:20px;margin:6px 0 2px}.sub{color:#8b949e;font-size:13px;margin:0 0 18px;line-height:1.5}- h2{font-size:15px;color:#58a6ff;margin:22px 0 8px;border-bottom:1px solid #21262d;padding-bottom:6px}- .card{display:block;background:#161b22;border:1px solid #21262d;border-radius:12px;padding:14px 16px;margin:10px 0;text-decoration:none;color:inherit}- .card:active{background:#1c2330}.row{display:flex;align-items:center;justify-content:space-between;gap:10px}- .name{font-weight:700;font-size:15px}.dl{font-size:12px;color:#58a6ff;white-space:nowrap}- .meas{font-size:12px;color:#8b949e;margin-top:5px;line-height:1.4}- .tag{display:inline-block;font-size:10px;font-weight:700;padding:2px 7px;border-radius:6px;background:#10301c;color:#56d364;margin-left:6px}- .foot{color:#6e7681;font-size:11px;margin-top:24px;line-height:1.5}-</style></head><body>-<h1>cua-driver — modality recordings</h1>-<p class="sub">Per-modality recordings of the no-foreground contract. Each: test harness (left) + live dashboard (right) measuring, per action, whether it kept the app in the background. Tap a card to play or download.</p>--<h2>Windows — WPF</h2>-<a class="card" href="wpf-ax-fg.mp4"><div class="row"><span class="name">AX · foreground</span><span class="dl">play / download ⤓</span></div><div class="meas">effects 5/7 landed · foreground, 8 actions</div></a>-<a class="card" href="wpf-ax-bg.mp4"><div class="row"><span class="name">AX · background</span><span class="dl">play / download ⤓</span></div><div class="meas">effects 5/7 landed · 2/8 stole focus</div></a>-<a class="card" href="wpf-px-fg.mp4"><div class="row"><span class="name">Vision · foreground</span><span class="dl">play / download ⤓</span></div><div class="meas">effects 3/6 landed · foreground, 7 actions</div></a>-<a class="card" href="wpf-px-bg.mp4"><div class="row"><span class="name">Vision · background</span><span class="dl">play / download ⤓</span></div><div class="meas">effects 4/6 landed · 4/7 stole focus</div></a>-<a class="card" href="wpf-px-desktop.mp4"><div class="row"><span class="name">Vision · full desktop</span><span class="dl">play / download ⤓</span></div><div class="meas">effects 1/3 landed · 4 actions</div></a>--<h2>Linux — GTK3</h2>-<a class="card" href="gtk3-ax-fg.mp4"><div class="row"><span class="name">AX · foreground</span><span class="dl">play / download ⤓</span></div><div class="meas">effects 3/7 landed · foreground, 8 actions</div></a>-<a class="card" href="gtk3-ax-bg.mp4"><div class="row"><span class="name">AX · background</span><span class="dl">play / download ⤓</span></div><div class="meas">effects 3/7 landed · 3/8 stole focus</div></a>-<a class="card" href="gtk3-px-fg.mp4"><div class="row"><span class="name">Vision · foreground</span><span class="dl">play / download ⤓</span></div><div class="meas">effects 1/6 landed · foreground, 7 actions</div></a>-<a class="card" href="gtk3-px-bg.mp4"><div class="row"><span class="name">Vision · background</span><span class="dl">play / download ⤓</span></div><div class="meas">effects 1/6 landed · 0/7 stole focus</div></a>-<a class="card" href="gtk3-px-desktop.mp4"><div class="row"><span class="name">Vision · full desktop</span><span class="dl">play / download ⤓</span></div><div class="meas">effects 1/3 landed · 4 actions</div></a>--<a class="card" href="README.md"><div class="row"><span class="name">README</span><span class="dl">open ⤓</span></div><div class="meas">full notes: action set, the two measurements, Linux XSendEvent finding</div></a>-<p class="foot">Both Windows and Linux runs show a per-action ✓ worked / ✗ no-op verifier alongside the focus contract.</p>-</body></html>diff --git a/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-all-electron.sh b/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-all-electron.shdeleted file mode 100755index f98edbe921..0000000000--- a/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-all-electron.sh+++ /dev/null@@ -1,11 +0,0 @@-#!/bin/bash-# Run all 5 modality recordings against the Electron harness, save re-encoded-# h264 mp4s to ~/Desktop/cua-driver-modality-videos/linux-electron-<mode>.mp4.-rm -f /tmp/lin-electron-allmodes.log-OUT="$HOME/Desktop/cua-driver-modality-videos"; mkdir -p "$OUT"-for MODE in ax-fg ax-bg px-fg px-bg px-desktop; do- bash /tmp/lin-run-electron.sh "$MODE" >/dev/null 2>&1- echo "DONE $MODE $(cat /tmp/cua-lin-electron-$MODE/metric.log 2>/dev/null)" | tee -a /tmp/lin-electron-allmodes.log- SRC="/tmp/cua-lin-electron-$MODE/rec/recording.mp4"- [ -f "$SRC" ] && ffmpeg -y -loglevel error -i "$SRC" -c:v libx264 -crf 26 -pix_fmt yuv420p "$OUT/linux-electron-$MODE.mp4"-donediff --git a/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-all.sh b/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-all.shdeleted file mode 100644index 09d94e9ff2..0000000000--- a/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-all.sh+++ /dev/null@@ -1,6 +0,0 @@-#!/bin/bash-rm -f /tmp/lin-allmodes.log-for MODE in ax-fg ax-bg px-fg px-bg px-desktop; do- bash /tmp/lin-run.sh "$MODE" >/dev/null 2>&1- echo "DONE $MODE $(cat /tmp/cua-lin-$MODE/metric.log 2>/dev/null)" | tee -a /tmp/lin-allmodes.log-donediff --git a/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-dash.py b/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-dash.pydeleted file mode 100644index 381d8ebe5e..0000000000--- a/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-dash.py+++ /dev/null@@ -1,26 +0,0 @@-#!/usr/bin/env python3-# Native dashboard panel: a GTK window hosting a WebKit2GTK webview that loads-# the SAME dashboard.html served over loopback (exact parity with the Windows-# chrome --app panel, but no browser process — works headless under Xvfb).-import sys-import gi-gi.require_version("Gtk", "3.0")-gi.require_version("WebKit2", "4.1")-from gi.repository import Gtk, WebKit2--URL = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:8146/"-X = int(sys.argv[2]) if len(sys.argv) > 2 else 544-Y = int(sys.argv[3]) if len(sys.argv) > 3 else 0-W = int(sys.argv[4]) if len(sys.argv) > 4 else 480-H = int(sys.argv[5]) if len(sys.argv) > 5 else 740--win = Gtk.Window(title="cua-driver-panel")-win.set_default_size(W, H)-win.set_resizable(False)-view = WebKit2.WebView()-win.add(view)-view.load_uri(URL)-win.connect("destroy", Gtk.main_quit)-win.show_all()-win.move(X, Y)-Gtk.main()diff --git a/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-dashboard-electron.html b/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-dashboard-electron.htmldeleted file mode 100644index 4a71e896a0..0000000000--- a/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-dashboard-electron.html+++ /dev/null@@ -1,31 +0,0 @@-<!doctype html><html><head><meta charset="utf-8"><title>cua-driver-panel</title><style>- html,body{margin:0;height:100%;font-family:Segoe UI,Ubuntu,-apple-system,sans-serif;background:#0d1117;color:#e6edf3}.wrap{padding:12px}- h1{font-size:11px;margin:0 0 1px;color:#8b949e}.run{font-size:17px;font-weight:800;color:#58a6ff;margin:0 0 5px}.exp{font-size:10.5px;color:#8b949e;margin:0 0 9px;line-height:1.35}- .fg{padding:9px 11px;border-radius:9px;font-weight:800;font-size:12.5px;margin-bottom:6px;line-height:1.25}.fg.ok{background:#0f2e1a;color:#3fb950;border:1px solid #2ea043}.fg.bad{background:#3d1418;color:#ff6a5f;border:1px solid #da3633}.fg.fgmode{background:#16202e;color:#58a6ff;border:1px solid #2f4664}- .sub{font-weight:500;font-size:9.5px;color:#c9d1d9;opacity:.85;margin-top:2px}.now{font-size:9.5px;font-family:Consolas,monospace;color:#6e7681;margin:5px 0 9px;word-break:break-all}- .row{padding:6px 8px;border-radius:7px;margin:4px 0;font-size:11.5px;display:flex;align-items:center;gap:7px}.row.done{background:#11161d}.row.active{background:#161b22;border:1px solid #d29922}.row.pending{opacity:.5}- .ic{width:14px;text-align:center}.lbl{flex:1}.res{font-size:9px;font-weight:700;padding:2px 5px;border-radius:5px;white-space:nowrap;margin-left:4px}.res.held{background:#0f2e1a;color:#3fb950}.res.stole{background:#3d1418;color:#ff6a5f}.res.front{background:#16202e;color:#58a6ff}- .res.did{background:#10301c;color:#56d364;border:1px solid #238636}.res.nope{background:#3d1418;color:#ff7b72;border:1px solid #da3633}- .legend{font-size:9px;color:#6e7681;margin:2px 0 8px;line-height:1.4}- .tally{margin-top:10px;font-size:11px;color:#8b949e}.tally b{color:#c9d1d9}-</style></head><body><div class="wrap">- <h1>cua-driver - single-modality run - Electron</h1><div class="run" id="run">...</div><div class="exp" id="exp"></div>- <div id="fg" class="fg fgmode">...</div><div class="now" id="now"></div>- <div class="legend"><b style="color:#56d364">✓ worked</b> / <b style="color:#ff7b72">✗ no-op</b> = did the action change the app · held / STOLE = focus contract</div>- <div id="rows"></div><div class="tally" id="tally"></div>-</div><script>-async function tick(){try{const r=await fetch('status.json',{cache:'no-store'});const s=await r.json();- run.textContent=s.run||'';exp.textContent=s.expect||'';now.textContent='foreground now: '+(s.foreground||'?');- if(s.fgmode){fg.className='fg fgmode';fg.innerHTML='FOREGROUND MODE - app intentionally in front<div class="sub">'+(s.appFront?'app is foreground':'...')+'</div>';}- else if(s.appFront){fg.className='fg bad';fg.innerHTML='APP IS FOREGROUND<div class="sub">'+(s.foreground||'')+' - current action stole focus</div>';}- else{fg.className='fg ok';fg.innerHTML='APP IN BACKGROUND<div class="sub">no-foreground contract holding - '+(s.foreground||'')+' stays frontmost</div>';}- rows.innerHTML='';(s.steps||[]).forEach(st=>{const d=document.createElement('div');d.className='row '+st.state;- const ic=st.state==='done'?'✓':st.state==='active'?'▶':'·';let res='';if(st.result==='held')res='<span class="res held">held</span>';else if(st.result==='stole')res='<span class="res stole">STOLE FOCUS</span>';else if(st.result==='front')res='<span class="res front">foreground</span>';- let ver='';if(st.verified==='ok')ver='<span class="res did">✓ worked</span>';else if(st.verified==='fail')ver='<span class="res nope">✗ no-op</span>';- d.innerHTML='<span class="ic">'+ic+'</span><span class="lbl">'+st.label+'</span>'+ver+res;rows.appendChild(d);});- const worked=(s.steps||[]).filter(x=>x.verified==='ok').length;const ver=(s.steps||[]).filter(x=>x.verified==='ok'||x.verified==='fail').length;- const eff='effects: <b>'+worked+'/'+ver+'</b> actions changed the app';- if(!s.fgmode){tally.innerHTML=eff+'<br>focus contract: <b>'+(s.steals||0)+'/'+(s.actions||0)+'</b> stole focus';}else{tally.innerHTML=eff+'<br><b>'+(s.actions||0)+'</b> actions performed (foreground mode)';}-}catch(e){}}-setInterval(tick,200);tick();-</script></body></html>diff --git a/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-dashboard.html b/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-dashboard.htmldeleted file mode 100644index d918f5e5d1..0000000000--- a/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-dashboard.html+++ /dev/null@@ -1,31 +0,0 @@-<!doctype html><html><head><meta charset="utf-8"><title>cua-driver-panel</title><style>- html,body{margin:0;height:100%;font-family:Segoe UI,Ubuntu,-apple-system,sans-serif;background:#0d1117;color:#e6edf3}.wrap{padding:12px}- h1{font-size:11px;margin:0 0 1px;color:#8b949e}.run{font-size:17px;font-weight:800;color:#58a6ff;margin:0 0 5px}.exp{font-size:10.5px;color:#8b949e;margin:0 0 9px;line-height:1.35}- .fg{padding:9px 11px;border-radius:9px;font-weight:800;font-size:12.5px;margin-bottom:6px;line-height:1.25}.fg.ok{background:#0f2e1a;color:#3fb950;border:1px solid #2ea043}.fg.bad{background:#3d1418;color:#ff6a5f;border:1px solid #da3633}.fg.fgmode{background:#16202e;color:#58a6ff;border:1px solid #2f4664}- .sub{font-weight:500;font-size:9.5px;color:#c9d1d9;opacity:.85;margin-top:2px}.now{font-size:9.5px;font-family:Consolas,monospace;color:#6e7681;margin:5px 0 9px;word-break:break-all}- .row{padding:6px 8px;border-radius:7px;margin:4px 0;font-size:11.5px;display:flex;align-items:center;gap:7px}.row.done{background:#11161d}.row.active{background:#161b22;border:1px solid #d29922}.row.pending{opacity:.5}- .ic{width:14px;text-align:center}.lbl{flex:1}.res{font-size:9px;font-weight:700;padding:2px 5px;border-radius:5px;white-space:nowrap;margin-left:4px}.res.held{background:#0f2e1a;color:#3fb950}.res.stole{background:#3d1418;color:#ff6a5f}.res.front{background:#16202e;color:#58a6ff}- .res.did{background:#10301c;color:#56d364;border:1px solid #238636}.res.nope{background:#3d1418;color:#ff7b72;border:1px solid #da3633}- .legend{font-size:9px;color:#6e7681;margin:2px 0 8px;line-height:1.4}- .tally{margin-top:10px;font-size:11px;color:#8b949e}.tally b{color:#c9d1d9}-</style></head><body><div class="wrap">- <h1>cua-driver - single-modality run - GTK3</h1><div class="run" id="run">...</div><div class="exp" id="exp"></div>- <div id="fg" class="fg fgmode">...</div><div class="now" id="now"></div>- <div class="legend"><b style="color:#56d364">✓ worked</b> / <b style="color:#ff7b72">✗ no-op</b> = did the action change the app · held / STOLE = focus contract</div>- <div id="rows"></div><div class="tally" id="tally"></div>-</div><script>-async function tick(){try{const r=await fetch('status.json',{cache:'no-store'});const s=await r.json();- run.textContent=s.run||'';exp.textContent=s.expect||'';now.textContent='foreground now: '+(s.foreground||'?');- if(s.fgmode){fg.className='fg fgmode';fg.innerHTML='FOREGROUND MODE - app intentionally in front<div class="sub">'+(s.appFront?'app is foreground':'...')+'</div>';}- else if(s.appFront){fg.className='fg bad';fg.innerHTML='APP IS FOREGROUND<div class="sub">'+(s.foreground||'')+' - current action stole focus</div>';}- else{fg.className='fg ok';fg.innerHTML='APP IN BACKGROUND<div class="sub">no-foreground contract holding - '+(s.foreground||'')+' stays frontmost</div>';}- rows.innerHTML='';(s.steps||[]).forEach(st=>{const d=document.createElement('div');d.className='row '+st.state;- const ic=st.state==='done'?'✓':st.state==='active'?'▶':'·';let res='';if(st.result==='held')res='<span class="res held">held</span>';else if(st.result==='stole')res='<span class="res stole">STOLE FOCUS</span>';else if(st.result==='front')res='<span class="res front">foreground</span>';- let ver='';if(st.verified==='ok')ver='<span class="res did">✓ worked</span>';else if(st.verified==='fail')ver='<span class="res nope">✗ no-op</span>';- d.innerHTML='<span class="ic">'+ic+'</span><span class="lbl">'+st.label+'</span>'+ver+res;rows.appendChild(d);});- const worked=(s.steps||[]).filter(x=>x.verified==='ok').length;const ver=(s.steps||[]).filter(x=>x.verified==='ok'||x.verified==='fail').length;- const eff='effects: <b>'+worked+'/'+ver+'</b> actions changed the app';- if(!s.fgmode){tally.innerHTML=eff+'<br>focus contract: <b>'+(s.steals||0)+'/'+(s.actions||0)+'</b> stole focus';}else{tally.innerHTML=eff+'<br><b>'+(s.actions||0)+'</b> actions performed (foreground mode)';}-}catch(e){}}-setInterval(tick,200);tick();-</script></body></html>diff --git a/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-harness.py b/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-harness.pydeleted file mode 100644index 30b9c08e30..0000000000--- a/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-harness.py+++ /dev/null@@ -1,167 +0,0 @@-#!/usr/bin/env python3-# CuaTestHarness GTK3 (recording edition) — rich controls so the Windows/WPF-# 8-action matrix maps onto Linux. Each actionable control sets its AT-SPI-# accessible name. The harness also (a) exports each named widget's screen rect-# (cua-driver's Linux AT-SPI snapshot has no sliders/scrollers or frames) and-# (b) writes its live state to a file so the recorder can VERIFY each action's-# effect (slider moved? checkbox toggled? text changed?), not just focus-steal.-import json-import gi-gi.require_version("Gtk", "3.0")-from gi.repository import Gtk, Gdk, GLib--STATE_FILE = "/tmp/cua-lin-state.json"-GEOM_FILE = "/tmp/cua-lin-geom.json"---def aid(widget, name):- widget.get_accessible().set_name(name)- return widget---class Harness(Gtk.Window):- def __init__(self):- super().__init__(title="CuaTestHarness GTK3")- self.set_default_size(536, 740)- self.set_resizable(False)- self.clicks = 0- self._last_action = "none"- self._ctx = "none"-- scroller = Gtk.ScrolledWindow()- scroller.set_policy(Gtk.PolicyType.NEVER, Gtk.PolicyType.AUTOMATIC)- self.add(scroller)- root = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=12)- root.set_border_width(14)- scroller.add(root)-- def section(title):- root.pack_start(Gtk.Label(label=title, xalign=0), False, False, 0)-- section("text input")- self.entry = aid(Gtk.Entry(), "txt-input")- self.entry.set_placeholder_text("type here")- self.entry.connect("changed", self._on_entry)- root.pack_start(self.entry, False, False, 0)- self.mirror = Gtk.Label(label="mirror=", xalign=0)- root.pack_start(self.mirror, False, False, 0)-- section("click target")- self.btn = aid(Gtk.Button(label="Click target (left / right / double)"), "btn-clicktarget")- self.btn.connect("clicked", self._on_click)- self.btn.connect("button-press-event", self._on_btn_press)- root.pack_start(self.btn, False, False, 0)- self.btn_status = Gtk.Label(label="last_action=none clicks=0", xalign=0)- root.pack_start(self.btn_status, False, False, 0)-- section("slider")- adj = Gtk.Adjustment(value=0, lower=0, upper=100, step_increment=1, page_increment=10)- self.scale = aid(Gtk.Scale(orientation=Gtk.Orientation.HORIZONTAL, adjustment=adj), "sld-value")- self.scale.set_draw_value(False)- self.scale.connect("value-changed", self._on_scale)- root.pack_start(self.scale, False, False, 0)- self.scale_status = Gtk.Label(label="slider_value=0", xalign=0)- root.pack_start(self.scale_status, False, False, 0)-- section("checkable")- self.chk = aid(Gtk.CheckButton(label="I agree"), "chk-agree")- self.chk.connect("toggled", self._on_chk)- root.pack_start(self.chk, False, False, 0)- self.chk_status = Gtk.Label(label="agreed=False", xalign=0)- root.pack_start(self.chk_status, False, False, 0)-- section("context menu")- self.ctx = aid(Gtk.Button(label="Right-click for context menu"), "btn-context")- self.ctx.connect("button-press-event", self._on_ctx_press)- root.pack_start(self.ctx, False, False, 0)- self.ctx_status = Gtk.Label(label="ctx=none", xalign=0)- root.pack_start(self.ctx_status, False, False, 0)- self.menu = Gtk.Menu()- for lbl in ("Cut", "Copy", "Paste"):- mi = Gtk.MenuItem(label=lbl)- mi.connect("activate", self._on_ctx_item, lbl)- self.menu.append(mi)- self.menu.show_all()-- section("scroll-tall")- inner = aid(Gtk.ScrolledWindow(), "scroll-tall")- inner.set_policy(Gtk.PolicyType.NEVER, Gtk.PolicyType.AUTOMATIC)- inner.set_size_request(-1, 150)- tall = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=2)- for i in range(1, 61):- tall.pack_start(Gtk.Label(label=f"row {i}", xalign=0), False, False, 0)- inner.add(tall)- root.pack_start(inner, False, False, 0)- self.scroll_inner = inner- inner.get_vadjustment().connect("value-changed", lambda *_: self._save_state())-- self.named = {"chk":self.chk,"btn":self.btn,"ctx":self.ctx,- "sld":self.scale,"scr":self.scroll_inner,"txt":self.entry}- self.connect("destroy", Gtk.main_quit)- self._save_state()-- # ---- state file (for the recorder's per-action effect verifier) ----- def _save_state(self):- try: sc = int(self.scroll_inner.get_vadjustment().get_value())- except Exception: sc = 0- st = {"slider": int(self.scale.get_value()), "agreed": self.chk.get_active(),- "mirror": self.entry.get_text(), "last_action": self._last_action,- "ctx": self._ctx, "clicks": self.clicks, "scroll": sc}- try: json.dump(st, open(STATE_FILE, "w"))- except Exception: pass-- def export_geom(self):- out = {}- for k, wdg in self.named.items():- try:- res = wdg.translate_coordinates(self, 0, 0)- if not res: continue- lx, ly = res- ok, gx, gy = self.get_window().get_origin()- a = wdg.get_allocation()- out[k] = {"x":gx+lx, "y":gy+ly, "w":a.width, "h":a.height}- except Exception:- pass- json.dump(out, open(GEOM_FILE, "w"))- self._save_state()- return False-- def _on_entry(self, e):- self.mirror.set_text(f"mirror={e.get_text()}"); self._save_state()-- def _on_click(self, *_):- self.clicks += 1; self._last_action = "click"- self.btn_status.set_text(f"last_action=click clicks={self.clicks}"); self._save_state()-- def _on_btn_press(self, _w, ev):- if ev.type == Gdk.EventType.DOUBLE_BUTTON_PRESS:- self._last_action = "double_click"- self.btn_status.set_text(f"last_action=double_click clicks={self.clicks}"); self._save_state()-- def _on_scale(self, s):- self.scale_status.set_text(f"slider_value={int(s.get_value())}"); self._save_state()-- def _on_chk(self, c):- self.chk_status.set_text(f"agreed={c.get_active()}"); self._save_state()-- def _on_ctx_press(self, _w, ev):- if ev.button == 3:- self._ctx = "opened"; self.ctx_status.set_text("ctx=opened")- self.menu.popup_at_pointer(ev); self._save_state()-- def _on_ctx_item(self, _w, lbl):- self._ctx = lbl; self.ctx_status.set_text(f"ctx={lbl}"); self._save_state()---if __name__ == "__main__":- import sys, traceback- try:- w = Harness()- print("INIT_DONE", flush=True)- w.move(0, 0)- w.show_all()- print("SHOWN", flush=True)- GLib.timeout_add(1200, w.export_geom)- except Exception:- traceback.print_exc(); sys.stdout.flush(); raise- Gtk.main()diff --git a/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-rec-electron.py b/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-rec-electron.pydeleted file mode 100644index 7721e779e1..0000000000--- a/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-rec-electron.py+++ /dev/null@@ -1,353 +0,0 @@-#!/usr/bin/env python3-# Linux modality recorder for the ELECTRON harness — parity with the Windows-# Electron lane in wpf-recorder.ps1 and the Linux GTK3 lin-rec.py.-# Runs under: xvfb-run -a --server-args="-screen 0 1024x768x24" dbus-run-session -- python3 lin-rec-electron.py MODE-# Electron LEFT, WebKit dashboard RIGHT (per-action held/STOLE + ✓worked/✗no-op verifier).-#-# Differences vs lin-rec.py (GTK):-# - launches the Electron app (Chromium) instead of lin-harness.py-# - resolves the harness window by substring "CuaTestHarness Electron" (cdp-suffixed title)-# - reads BOTH control geometry AND harness state from the web-AX tree-# (get_window_state capture_mode=ax), the way the Windows recorder reads UIA Text.-import json, os, subprocess, sys, time, glob, re--MODE = sys.argv[1] if len(sys.argv) > 1 else "ax-bg"-HOME = os.path.expanduser("~")-DRV = f"{HOME}/cua/libs/cua-driver/rust/target/release/cua-driver"-ELECTRON_DIR = f"{HOME}/cua/libs/cua-driver/tests/fixtures/apps/cross-platform/electron"-ELECTRON_BIN = f"{ELECTRON_DIR}/node_modules/electron/dist/electron"-DASH = "/tmp/lin-dash.py"-DASH_HTML = "/tmp/lin-dashboard-electron.html"-PORT = 8146-WORK = f"/tmp/cua-lin-electron-{MODE}"-REC = f"{WORK}/rec"-HARW, HARH = 536, 740-PANX, PANW, PANH = 544, 480, 740-WIN_TITLE = "CuaTestHarness Electron"--META = {- "ax-fg": {"title":"AX - FOREGROUND","scope":"window","see":"accessibility tree (element-level)","fg":True, "expect":"App kept in FRONT on purpose. Each action runs via the accessibility tree; we measure the foreground."},- "ax-bg": {"title":"AX - BACKGROUND","scope":"window","see":"accessibility tree (element-level)","fg":False,"expect":"App should stay in the BACKGROUND. Each action runs via the accessibility tree; we measure which actions steal focus."},- "px-fg": {"title":"VISION - FOREGROUND","scope":"window","see":"screenshot only (pixels)","fg":True, "expect":"Pure pixel-driven, app kept in FRONT. We measure the foreground."},- "px-bg": {"title":"VISION - BACKGROUND","scope":"window","see":"screenshot only (pixels)","fg":False,"expect":"Pure pixel-driven, app should stay in the BACKGROUND. We measure which pixel actions steal focus."},- "px-desktop":{"title":"VISION - FULL DESKTOP","scope":"desktop","see":"full-screen screenshot","fg":True,"expect":"Whole-screen, window-less screen-pixel actions (no window targeted)."},-}-m = META[MODE]-VISION = m["see"].startswith("screenshot") or m["scope"] == "desktop"-DESKTOP = m["scope"] == "desktop"--def sh(args, t=10):- try: return subprocess.run(args, capture_output=True, text=True, timeout=t).stdout- except Exception: return ""--def D(tool, payload):- try:- p = subprocess.run([DRV, "call", tool], input=json.dumps(payload), capture_output=True, text=True, timeout=25)- return p.stdout- except Exception:- return ""--def DJ(tool, payload):- try: return json.loads(D(tool, payload) or "{}")- except Exception: return {}--def active_name():- return sh(["xdotool", "getactivewindow", "getwindowname"]).strip()--def active_id():- return sh(["xdotool", "getactivewindow"]).strip()--# ---------- plan -----------PLAN = [- ("click","chk","left-click a checkbox"),- ("double","btn","double-click a button"),- ("right","ctx","right-click (context menu)"),- ("drag","sld","drag the slider"),- ("scroll","scr","scroll the panel"),- ("setval","txt","set_value on the text box"),- ("type","txt","type into the text box"),- ("key","txt","press a key (Tab)"),-]-if VISION: PLAN = [p for p in PLAN if p[0] != "setval"]-if DESKTOP: PLAN = [p for p in PLAN if p[0] in ("click","scroll","type","key")]--steps = [{"label":l,"state":"pending","result":"","verified":""} for (_,_,l) in PLAN]-state = {"steals":0,"actions":0}--# ---------- per-action EFFECT verifier: read the web harness's own status labels via web-AX -----------def hstate():- E = els()- joined = " || ".join(str(e.get("label", e.get("name",""))) for e in E)- h = {}- mm = re.search(r'agreed=(\w+)', joined); h["agreed"] = mm.group(1) if mm else None- mm = re.search(r'slider_value=(\d+)', joined); h["slider"] = int(mm.group(1)) if mm else 0- mm = re.search(r'last_action=(\w+)', joined); h["last_action"] = mm.group(1) if mm else None- mm = re.search(r'mirror=([^|]*)', joined); h["mirror"] = (mm.group(1).strip() if mm else "")- mm = re.search(r'counter=(\d+)', joined); h["counter"] = int(mm.group(1)) if mm else 0- mm = re.search(r'scroll_offset=(\d+)', joined); h["scroll"] = int(mm.group(1)) if mm else 0- h["_joined"] = joined- return h--def verify(t, before, after):- if t == "click": return "ok" if after.get("agreed") != before.get("agreed") else "fail"- if t == "double": return "ok" if after.get("last_action") == "double_click" else "fail"- if t == "right": return "ok" if after.get("last_action") == "right_click" else "fail"- if t == "drag": return "ok" if after.get("slider", 0) > before.get("slider", 0) else "fail"- if t == "scroll": return "ok" if after.get("scroll", 0) > before.get("scroll", 0) else "fail" # scroll_offset= now exposed by web-AX- if t == "setval": return "ok" if "set-by-cua" in str(after.get("mirror", "")) else "fail"- if t == "type": return "ok" if "typed-by-cua" in str(after.get("mirror", "")) else "fail"- return "na" # press-key (Tab): no observable harness-state effect--def flush():- fg = active_name()- af = WIN_TITLE in fg- st = {"run":m["title"],"expect":m["expect"],"fgmode":m["fg"],"foreground":fg,"appFront":af,- "steals":state["steals"],"actions":state["actions"],"steps":steps}- with open(f"{WORK}/status.json","w") as f: json.dump(st, f)--def pulse(sec):- end = time.time()+sec- while time.time() < end: flush(); time.sleep(0.15)--# ---------- setup -----------os.makedirs(REC, exist_ok=True)-subprocess.run("pkill -x electron; pkill -f lin-dash.py", shell=True)-time.sleep(1)-import shutil; shutil.copy(DASH_HTML, f"{WORK}/dashboard.html")-with open(f"{WORK}/status.json","w") as f: json.dump({"run":"","steps":[]}, f)--def dbgcnt(tag):- try:- w = json.loads(D("list_windows", {}) or "{}").get("windows", [])- line = "%s: %d %s\n" % (tag, len(w), [x.get("title") for x in w])- except Exception as e:- line = "%s: ERR %s\n" % (tag, e)- open(f"{WORK}/setup.log","a").write(line)--open(f"{WORK}/setup.log","a").write("DISPLAY=%s XAUTH=%s\n" % (os.environ.get("DISPLAY"), os.environ.get("XAUTHORITY")))-dbgcnt("at start")-D("set_agent_cursor_enabled", {"enabled":True,"session":"d1"})-D("set_agent_cursor_motion", {"session":"d1","cursor_color":"#FF2D2D","cursor_label":"cua-driver","glide_duration_ms":600,"dwell_after_click_ms":700,"idle_hide_ms":120000})-dbgcnt("after cursor")--subprocess.Popen(["python3","-m","http.server",str(PORT),"--directory",WORK], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)--# launch Electron — Chromium needs --no-sandbox under Xvfb, --disable-gpu for the-# software path, --force-renderer-accessibility so the web-AX tree populates via AT-SPI.-eenv = dict(os.environ)-eenv["ELECTRON_DISABLE_SECURITY_WARNINGS"] = "1"-elog = open(f"{WORK}/electron.log","w")-subprocess.Popen([ELECTRON_BIN, ELECTRON_DIR, "--no-sandbox", "--disable-gpu",- "--force-renderer-accessibility"],- cwd=ELECTRON_DIR, env=eenv, stdout=elog, stderr=elog)-time.sleep(10)-dbgcnt("after electron")-subprocess.Popen(["python3", DASH, f"http://127.0.0.1:{PORT}/dashboard.html", str(PANX),"0",str(PANW),str(PANH)], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL); time.sleep(4)-dbgcnt("after dashboard")-# position windows: harness LEFT, dashboard RIGHT + above-sh(["wmctrl","-r",WIN_TITLE,"-e","0,0,0,%d,%d"%(HARW,HARH)])-sh(["wmctrl","-r","cua-driver-panel","-e","0,%d,0,%d,%d"%(PANX,PANW,PANH)])-sh(["wmctrl","-r","cua-driver-panel","-b","add,above"])-time.sleep(1)--# ---------- foreground-baseline ANCHOR (background modes only) -----------# The no-foreground contract is "did this action steal foreground". Measuring that needs a-# GENUINE foreground baseline before each action: a real, ACTIVATED, non-harness window.-# Re-asserting the dashboard panel with `wmctrl -b add,above` is z-order / _NET_WM_STATE_ABOVE-# ONLY (no activation), so it never holds a true active-window baseline — once the first inject-# action click-activates the target, the harness silently stays the active window and every later-# step false-positives as a "steal". Anchor on a real xterm: park it under the (above) panel rect-# so it stays invisible in the recording but remains a valid activatable non-harness foreground-# window. (macOS mac-rec.py already does a real `activate` — no flaw; Windows fix was 49bdb41b.)-ANCHOR_ID = None-ANCHOR_PROC = None-if not m["fg"]:- ANCHOR_PROC = subprocess.Popen(- ["xterm","-class","cua-anchor","-T","cua-anchor","-geometry","18x3",- "-e","bash","-c","while true; do sleep 3600; done"],- stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)- for _ in range(20):- ids = sh(["xdotool","search","--class","cua-anchor"]).split()- if ids: ANCHOR_ID = ids[0]; break- time.sleep(0.4)- if ANCHOR_ID:- sh(["wmctrl","-i","-r",ANCHOR_ID,"-e","0,%d,0,%d,%d"%(PANX,PANW,PANH)])- sh(["wmctrl","-r","cua-driver-panel","-b","add,above"])- open(f"{WORK}/baseline.log","w").write("ANCHOR_ID=%s launched=%s\n" % (ANCHOR_ID, ANCHOR_PROC is not None))--def anchor_front():- # GENUINELY activate (not z-order) the anchor and confirm it actually became the active window,- # so each action is measured against a true non-harness foreground baseline. Then re-assert the- # dashboard panel ABOVE for the recording (z-order only — does not change the active window).- if not ANCHOR_ID: return False- held = False- for _ in range(8):- sh(["xdotool","windowactivate","--sync",ANCHOR_ID])- time.sleep(0.18)- if active_id() == ANCHOR_ID: held = True; break- sh(["wmctrl","-r","cua-driver-panel","-b","add,above"])- return held--# resolve harness window-w = None-for _ in range(30):- wins = DJ("list_windows", {}).get("windows", [])- w = next((x for x in wins if WIN_TITLE in (x.get("title","") or "")), None)- if w: break- time.sleep(0.5)-if not w:- dbg = "DISPLAY=%s\n" % os.environ.get("DISPLAY")- dbg += "raw_list_windows=%s\n" % (D("list_windows", {})[:2000])- dbg += "xwininfo=%s\n" % sh(["bash","-c","xwininfo -root -tree 2>/dev/null | grep -iE 'Cua|panel|Electron|harness'"])- dbg += "electron_log=%s\n" % (open(f"{WORK}/electron.log").read()[-2000:] if os.path.exists(f"{WORK}/electron.log") else "none")- open(f"{WORK}/debug.log","w").write(dbg)- open(f"{WORK}/metric.log","w").write("FATAL: no harness window"); sys.exit(1)-WP, WD = w["pid"], w["window_id"]-WB = w.get("bounds") or w.get("frame") or {"x":0,"y":0,"w":HARW,"h":HARH}--def els():- return DJ("get_window_state", {"pid":WP,"window_id":WD,"capture_mode":"ax"}).get("elements", [])--def rect(el):- r = el.get("frame") or el.get("bounds") or {}- return (int(r.get("x",0)), int(r.get("y",0)), int(r.get("w",0)), int(r.get("h",0)))--def find(E, name=None, role=None, anyname=None):- for e in E:- nm = str(e.get("label", e.get("name","")))- rl = str(e.get("role",""))- if name and name.lower() not in nm.lower(): continue- if role and role.lower() not in rl.lower(): continue- if anyname and not any(a.lower() in nm.lower() for a in anyname): continue- return e- return None--def find_scroll(E):- # scroll-tall is the CLIPPED viewport: a 'section' ~260w x ~120h (border/pad → ~270x131- # in web-AX), distinct from the inner lines container (h~880) and the 23px line rows.- for e in E:- if "section" not in str(e.get("role","")).lower(): continue- _, _, w, h = rect(e)- if 250 <= w <= 290 and 110 <= h <= 155:- return e- return None--E = []-resolve = {}-for _ in range(16):- E = els()- resolve = {- "chk": find(E, role="check") or find(E, name="agree"),- "btn": find(E, name="Click target") or find(E, name="click target"),- "ctx": find(E, name="Click target") or find(E, name="click target"),- "sld": find(E, role="slider"),- "scr": find_scroll(E),- "txt": find(E, role="text") or find(E, role="entry") or find(E, name="type here"),- }- if resolve["chk"] or resolve["txt"]: break- time.sleep(0.8)-dump = "\n".join("[%s] role=%r name=%r frame=%r" % (e.get("element_index"), e.get("role"), e.get("label", e.get("name","")), e.get("frame") or e.get("bounds")) for e in E)-open(f"{WORK}/resolve.log","w").write("WIN bounds=%r\nRESOLVE "%WB+" ".join("%s=%s"%(k,bool(v)) for k,v in resolve.items())+" count=%d\n--- elements ---\n%s"%(len(E),dump))--def gcenter(sel):- el = resolve.get(sel)- if not el: return None- x,y,ww,hh = rect(el); return (x+ww//2, y+hh//2)-def gwinlocal(sel):- el = resolve.get(sel)- if not el: return None- x,y,ww,hh = rect(el); return (x-int(WB["x"])+ww//2, y-int(WB["y"])+hh//2)--# seed agent cursor overlay BEFORE recording-D("move_cursor", {"x":HARW-30,"y":HARH-30,"session":"d1"})-time.sleep(0.4)-if DESKTOP: D("set_config", {"key":"capture_scope","value":"desktop"})-D("start_recording", {"output_dir":REC,"record_video":True})-pulse(2)--def do(t, sel):- el = resolve.get(sel)- eidx = el.get("element_index") if el else None- use_ax = (eidx is not None) and (not VISION) and (not DESKTOP)- c = gcenter(sel); wl = gwinlocal(sel) or (HARW//2, HARH//2)- if c: D("move_cursor", {"x":c[0],"y":c[1],"session":"d1"}); time.sleep(0.55)- if t == "click":- if DESKTOP and c: D("click", {"x":c[0],"y":c[1],"session":"d1"})- elif use_ax: D("click", {"pid":WP,"window_id":WD,"element_index":eidx,"session":"d1"})- elif c: D("click", {"pid":WP,"window_id":WD,"x":wl[0],"y":wl[1],"delivery_mode":("foreground" if m["fg"] else "background"),"session":"d1"})- elif t == "double":- if use_ax: D("double_click", {"pid":WP,"window_id":WD,"element_index":eidx,"session":"d1"})- elif c: D("double_click", {"pid":WP,"window_id":WD,"x":wl[0],"y":wl[1],"session":"d1"})- elif t == "right":- if use_ax: D("right_click", {"pid":WP,"window_id":WD,"element_index":eidx,"session":"d1"})- elif c: D("right_click", {"pid":WP,"window_id":WD,"x":wl[0],"y":wl[1],"session":"d1"})- time.sleep(0.5); D("press_key", {"pid":WP,"key":"escape","session":"d1"})- elif t == "drag":- if el:- x,y,ww,hh = rect(el); fx=x-int(WB["x"])+8; fy=y-int(WB["y"])+hh//2- D("drag", {"pid":WP,"window_id":WD,"from_x":fx,"from_y":fy,"to_x":fx+150,"to_y":fy,"session":"d1"})- elif t == "scroll":- # aim at the resolved scroll-tall viewport; fall back to window center if unresolved- sa = c or (HARW//2, HARH//2) # screen/abs center of scroll-tall- sl = wl # window-local center of scroll-tall- if DESKTOP: D("scroll", {"x":sa[0],"y":sa[1],"direction":"down","session":"d1"})- elif VISION: D("scroll", {"pid":WP,"window_id":WD,"x":sl[0],"y":sl[1],"direction":"down","session":"d1"})- elif use_ax: D("scroll", {"pid":WP,"window_id":WD,"element_index":eidx,"direction":"down","session":"d1"})- else: D("scroll", {"pid":WP,"window_id":WD,"x":sl[0],"y":sl[1],"direction":"down","session":"d1"})- elif t == "setval":- if eidx is not None: D("set_value", {"pid":WP,"window_id":WD,"element_index":eidx,"value":"set-by-cua","session":"d1"})- elif t == "type":- if eidx is not None and not VISION:- D("click", {"pid":WP,"window_id":WD,"element_index":eidx,"session":"d1"}); time.sleep(0.35)- elif c:- D("click", {"pid":WP,"window_id":WD,"x":wl[0],"y":wl[1],"delivery_mode":("foreground" if m["fg"] else "background"),"session":"d1"}); time.sleep(0.35)- D("type_text", {"pid":WP,"text":"typed-by-cua","session":"d1"})- elif t == "key":- D("press_key", {"pid":WP,"key":"tab","session":"d1"})--# ---------- run -----------try:- for i,(t,sel,label) in enumerate(PLAN):- steps[i]["state"]="active"; flush()- # establish the genuine foreground baseline: ACTIVATE the anchor and confirm it actually took- # the active window BEFORE the action runs. A "steal" is then the active window moving OFF the- # anchor ONTO the harness; "held" is the anchor staying active. (fg modes keep the harness in- # front by design, so no baseline anchor there.)- if not m["fg"] and ANCHOR_ID:- held = anchor_front()- open(f"{WORK}/baseline.log","a").write(- "step %d '%s': baseline active='%s' anchor_held=%s\n" % (i, t, active_name(), held))- before = hstate()- do(t, sel)- # measure: steal = active window moved OFF the anchor baseline onto the harness after the action.- stole=False; end=time.time()+1.5- while time.time()<end:- flush()- if WIN_TITLE in active_name(): stole=True- time.sleep(0.11)- steps[i]["verified"] = verify(t, before, hstate())- state["actions"]+=1- if m["fg"]: steps[i]["result"]="front"- elif stole: state["steals"]+=1; steps[i]["result"]="stole"- else: steps[i]["result"]="held"- steps[i]["state"]="done"; flush()- sh(["wmctrl","-r","cua-driver-panel","-b","add,above"])-- pulse(2.5)- if DESKTOP: D("set_config", {"key":"capture_scope","value":"window"})- D("stop_recording", {}); time.sleep(3)-finally:- # Dispose the agent-cursor session (remove its overlay cursor) BEFORE the daemon- # is pkill'd below. Runs even if the action loop raised, so the session is never- # leaked to the idle-TTL reaper. end_session is idempotent.- D("end_session", {"session":"d1"})-if ANCHOR_PROC is not None:- try: ANCHOR_PROC.terminate()- except Exception: pass-subprocess.run("pkill -f cua-anchor; pkill -x electron; pkill -f lin-dash.py; pkill -f 'http.server 8146'; pkill cua-driver; pkill openbox", shell=True)-mp4 = (glob.glob(f"{REC}/**/*.mp4", recursive=True) or [None])[0]-size = os.path.getsize(mp4) if mp4 else 0-worked = sum(1 for s in steps if s.get("verified")=="ok"); ver = sum(1 for s in steps if s.get("verified") in ("ok","fail"))-verdict = ("foreground-mode, %d actions"%state["actions"]) if m["fg"] else ("%d/%d stole focus"%(state["steals"],state["actions"]))-line = "MODE=%s MEASURE=%s EFFECTS=%d/%d_landed MP4=%s SIZE=%d" % (MODE, verdict, worked, ver, mp4, size)-open(f"{WORK}/metric.log","w").write(line); print(line)diff --git a/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-rec.py b/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-rec.pydeleted file mode 100644index 0d829b05c6..0000000000--- a/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-rec.py+++ /dev/null@@ -1,307 +0,0 @@-#!/usr/bin/env python3-# Linux modality recorder — parity with the Windows wpf-v7 golden recorder.-# Runs under: xvfb-run -a --server-args="-screen 0 1024x768x24" dbus-run-session -- python3 lin-rec.py MODE-# Produces one single-modality recording: GTK3 harness LEFT, WebKit dashboard RIGHT-# (per-action held/STOLE measurement + live fg/bg indicator).-import json, os, subprocess, sys, time, signal, glob--MODE = sys.argv[1] if len(sys.argv) > 1 else "ax-bg"-HOME = os.path.expanduser("~")-DRV = f"{HOME}/cua/libs/cua-driver/rust/target/release/cua-driver"-HARNESS = "/tmp/lin-harness.py"-DASH = "/tmp/lin-dash.py"-DASH_HTML = "/tmp/lin-dashboard.html"-PORT = 8146-WORK = f"/tmp/cua-lin-{MODE}"-REC = f"{WORK}/rec"-HARW, HARH = 536, 740-PANX, PANW, PANH = 544, 480, 740--META = {- "ax-fg": {"title":"AX - FOREGROUND","scope":"window","see":"accessibility tree (element-level)","fg":True, "expect":"App kept in FRONT on purpose. Each action runs via the accessibility tree; we measure the foreground."},- "ax-bg": {"title":"AX - BACKGROUND","scope":"window","see":"accessibility tree (element-level)","fg":False,"expect":"App should stay in the BACKGROUND. Each action runs via the accessibility tree; we measure which actions steal focus."},- "px-fg": {"title":"VISION - FOREGROUND","scope":"window","see":"screenshot only (pixels)","fg":True, "expect":"Pure pixel-driven, app kept in FRONT. We measure the foreground."},- "px-bg": {"title":"VISION - BACKGROUND","scope":"window","see":"screenshot only (pixels)","fg":False,"expect":"Pure pixel-driven, app should stay in the BACKGROUND. We measure which pixel actions steal focus."},- "px-desktop":{"title":"VISION - FULL DESKTOP","scope":"desktop","see":"full-screen screenshot","fg":True,"expect":"Whole-screen, window-less screen-pixel actions (no window targeted)."},-}-m = META[MODE]-VISION = m["see"].startswith("screenshot") or m["scope"] == "desktop"-DESKTOP = m["scope"] == "desktop"--def sh(args, t=10):- try: return subprocess.run(args, capture_output=True, text=True, timeout=t).stdout- except Exception: return ""--def D(tool, payload):- try:- p = subprocess.run([DRV, "call", tool], input=json.dumps(payload), capture_output=True, text=True, timeout=25)- return p.stdout- except Exception as e:- return ""--def DJ(tool, payload):- try: return json.loads(D(tool, payload) or "{}")- except Exception: return {}--def active_name():- return sh(["xdotool", "getactivewindow", "getwindowname"]).strip()--def active_id():- return sh(["xdotool", "getactivewindow"]).strip()--# ---------- plan -----------PLAN = [- ("click","chk","left-click a checkbox"),- ("double","btn","double-click a button"),- ("right","ctx","right-click (context menu)"),- ("drag","sld","drag the slider"),- ("scroll","scr","scroll the panel"),- ("setval","txt","set_value on the text box"),- ("type","txt","type into the text box"),- ("key","txt","press a key (Tab)"),-]-if VISION: PLAN = [p for p in PLAN if p[0] != "setval"]-if DESKTOP: PLAN = [p for p in PLAN if p[0] in ("click","scroll","type","key")]--steps = [{"label":l,"state":"pending","result":"","verified":""} for (_,_,l) in PLAN]-state = {"steals":0,"actions":0}--def hstate():- try: return json.load(open("/tmp/cua-lin-state.json"))- except Exception: return {}--def verify(t, before, after):- # did the action actually change the harness's state?- if t == "click": return "ok" if after.get("agreed") != before.get("agreed") else "fail"- if t == "double": return "ok" if after.get("last_action") == "double_click" else "fail"- if t == "right": return "ok" if after.get("ctx", "none") != "none" else "fail"- if t == "drag": return "ok" if after.get("slider", 0) > before.get("slider", 0) else "fail"- if t == "scroll": return "ok" if after.get("scroll", 0) > before.get("scroll", 0) else "fail"- if t == "setval": return "ok" if after.get("mirror", "") == "set-by-cua" else "fail"- if t == "type": return "ok" if "typed-by-cua" in str(after.get("mirror", "")) else "fail"- return "na" # press-key (Tab): no observable harness-state effect to assert--def flush():- fg = active_name()- af = "CuaTestHarness" in fg- st = {"run":m["title"],"expect":m["expect"],"fgmode":m["fg"],"foreground":fg,"appFront":af,- "steals":state["steals"],"actions":state["actions"],"steps":steps}- with open(f"{WORK}/status.json","w") as f: json.dump(st, f)--def pulse(sec):- end = time.time()+sec- while time.time() < end: flush(); time.sleep(0.15)--# ---------- setup -----------os.makedirs(REC, exist_ok=True)-# NB: do NOT pkill cua-driver here — the wrapper owns the daemon. Port is freed by the wrapper.-subprocess.run("pkill -f lin-harness.py; pkill -f lin-dash.py", shell=True)-time.sleep(1)-import shutil; shutil.copy(DASH_HTML, f"{WORK}/dashboard.html")-with open(f"{WORK}/status.json","w") as f: json.dump({"run":"","steps":[]}, f)--def dbgcnt(tag):- try:- w = json.loads(D("list_windows", {}) or "{}").get("windows", [])- line = "%s: %d %s\n" % (tag, len(w), [x.get("title") for x in w])- except Exception as e:- line = "%s: ERR %s\n" % (tag, e)- open(f"{WORK}/setup.log","a").write(line)--# openbox + `cua-driver serve` are launched by the bash wrapper (so the daemon-# reliably inherits DISPLAY/XAUTHORITY — a python-launched daemon saw 0 windows).-open(f"{WORK}/setup.log","a").write("DISPLAY=%s XAUTH=%s\n" % (os.environ.get("DISPLAY"), os.environ.get("XAUTHORITY")))-dbgcnt("at start")-D("set_agent_cursor_enabled", {"enabled":True,"session":"d1"})-D("set_agent_cursor_motion", {"session":"d1","cursor_color":"#FF2D2D","cursor_label":"cua-driver","glide_duration_ms":600,"dwell_after_click_ms":700,"idle_hide_ms":120000})-dbgcnt("after cursor")-subprocess.Popen(["python3","-m","http.server",str(PORT),"--directory",WORK], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)-subprocess.Popen(["python3", HARNESS], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL); time.sleep(3)-dbgcnt("after harness")-subprocess.Popen(["python3", DASH, f"http://127.0.0.1:{PORT}/dashboard.html", str(PANX),"0",str(PANW),str(PANH)], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL); time.sleep(4)-dbgcnt("after dashboard")-# position windows: harness LEFT, dashboard RIGHT + above-sh(["wmctrl","-r","CuaTestHarness GTK3","-e","0,0,0,%d,%d"%(HARW,HARH)])-sh(["wmctrl","-r","cua-driver-panel","-e","0,%d,0,%d,%d"%(PANX,PANW,PANH)])-sh(["wmctrl","-r","cua-driver-panel","-b","add,above"])-time.sleep(1)--# ---------- foreground-baseline ANCHOR (background modes only) -----------# The no-foreground contract is "did this action steal foreground". Measuring that needs a-# GENUINE foreground baseline before each action: a real, ACTIVATED, non-harness window.-# Re-asserting the dashboard panel with `wmctrl -b add,above` is z-order / _NET_WM_STATE_ABOVE-# ONLY (no activation), so it never holds a true active-window baseline — once the first inject-# action click-activates the target, the harness silently stays the active window and every later-# step false-positives as a "steal". Anchor on a real xterm: park it under the (above) panel rect-# so it stays invisible in the recording but remains a valid activatable non-harness foreground-# window. (macOS mac-rec.py already does a real `activate` — no flaw; Windows fix was 49bdb41b.)-ANCHOR_ID = None-ANCHOR_PROC = None-if not m["fg"]:- ANCHOR_PROC = subprocess.Popen(- ["xterm","-class","cua-anchor","-T","cua-anchor","-geometry","18x3",- "-e","bash","-c","while true; do sleep 3600; done"],- stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)- for _ in range(20):- ids = sh(["xdotool","search","--class","cua-anchor"]).split()- if ids: ANCHOR_ID = ids[0]; break- time.sleep(0.4)- if ANCHOR_ID:- sh(["wmctrl","-i","-r",ANCHOR_ID,"-e","0,%d,0,%d,%d"%(PANX,PANW,PANH)])- sh(["wmctrl","-r","cua-driver-panel","-b","add,above"])- open(f"{WORK}/baseline.log","w").write("ANCHOR_ID=%s launched=%s\n" % (ANCHOR_ID, ANCHOR_PROC is not None))--def anchor_front():- # GENUINELY activate (not z-order) the anchor and confirm it actually became the active window,- # so each action is measured against a true non-harness foreground baseline. Then re-assert the- # dashboard panel ABOVE for the recording (z-order only — does not change the active window).- if not ANCHOR_ID: return False- held = False- for _ in range(8):- sh(["xdotool","windowactivate","--sync",ANCHOR_ID])- time.sleep(0.18)- if active_id() == ANCHOR_ID: held = True; break- sh(["wmctrl","-r","cua-driver-panel","-b","add,above"])- return held--# resolve harness window-w = None-for _ in range(20):- wins = DJ("list_windows", {}).get("windows", [])- w = next((x for x in wins if "CuaTestHarness" in (x.get("title","") or "")), None)- if w: break- time.sleep(0.5)-if not w:- dbg = "DISPLAY=%s\n" % os.environ.get("DISPLAY")- try: xalive = subprocess.run(["xdpyinfo"], capture_output=True, timeout=8).returncode == 0- except Exception: xalive = False- dbg += "X_alive=%s\n" % xalive- dbg += "raw_list_windows=%s\n" % (D("list_windows", {})[:1800])- dbg += "xwininfo=%s\n" % sh(["bash","-c","xwininfo -root -tree 2>/dev/null | grep -iE 'Cua|panel|GTK'"])- open(f"{WORK}/debug.log","w").write(dbg)- open(f"{WORK}/metric.log","w").write("FATAL: no harness window"); sys.exit(1)-WP, WD = w["pid"], w["window_id"]-WB = w.get("bounds") or w.get("frame") or {"x":0,"y":0,"w":HARW,"h":HARH}--def els():- return DJ("get_window_state", {"pid":WP,"window_id":WD,"capture_mode":"ax"}).get("elements", [])--def find(E, **kw):- for e in E:- nm = str(e.get("label", e.get("name","")))- role = str(e.get("role",""))- if "name" in kw and kw["name"] not in nm: continue- if "role" in kw and kw["role"].lower() not in role.lower(): continue- return e- return None--E = []-resolve = {}-for _ in range(14):- E = els()- resolve = {- "chk": find(E, name="chk-agree") or find(E, role="check"),- "btn": find(E, name="btn-clicktarget"),- "ctx": find(E, name="btn-context"),- "sld": find(E, name="sld-value") or find(E, role="slider"),- "scr": find(E, name="scroll-tall"),- "txt": find(E, name="txt-input") or find(E, role="text"),- }- if resolve["chk"] or resolve["txt"]: break- time.sleep(0.7)-dump = "\n".join("[%s] role=%r name=%r" % (e.get("element_index"), e.get("role"), e.get("label", e.get("name",""))) for e in E)-open(f"{WORK}/resolve.log","w").write("RESOLVE "+" ".join("%s=%s"%(k,bool(v)) for k,v in resolve.items())+" count=%d\n--- elements ---\n%s"%(len(E),dump))--try: GEOM = json.load(open("/tmp/cua-lin-geom.json"))-except Exception: GEOM = {}-open(f"{WORK}/resolve.log","a").write("\n--- geom ---\n"+json.dumps(GEOM))-def gcenter(sel):- g = GEOM.get(sel)- return (g["x"]+g["w"]//2, g["y"]+g["h"]//2) if g else None-def gwinlocal(sel):- g = GEOM.get(sel)- return (g["x"]-int(WB["x"])+g["w"]//2, g["y"]-int(WB["y"])+g["h"]//2) if g else None--# seed agent cursor overlay BEFORE recording-D("move_cursor", {"x":HARW-30,"y":HARH-30,"session":"d1"})-time.sleep(0.4)-if DESKTOP: D("set_config", {"key":"capture_scope","value":"desktop"})-D("start_recording", {"output_dir":REC,"record_video":True})-pulse(2)--def do(t, sel):- el = resolve.get(sel)- g = GEOM.get(sel)- eidx = el.get("element_index") if el else None- use_ax = (eidx is not None) and (not VISION) and (not DESKTOP)- c = gcenter(sel); wl = gwinlocal(sel) or (0,0)- if c: D("move_cursor", {"x":c[0],"y":c[1],"session":"d1"}); time.sleep(0.55)- if t == "click":- if DESKTOP and c: D("click", {"x":c[0],"y":c[1],"session":"d1"})- elif use_ax: D("click", {"pid":WP,"window_id":WD,"element_index":eidx,"session":"d1"})- elif c: D("click", {"pid":WP,"window_id":WD,"x":wl[0],"y":wl[1],"delivery_mode":("foreground" if m["fg"] else "background"),"session":"d1"})- elif t == "double":- if use_ax: D("double_click", {"pid":WP,"window_id":WD,"element_index":eidx,"session":"d1"})- elif c: D("double_click", {"pid":WP,"window_id":WD,"x":wl[0],"y":wl[1],"session":"d1"})- elif t == "right":- if use_ax: D("right_click", {"pid":WP,"window_id":WD,"element_index":eidx,"session":"d1"})- elif c: D("right_click", {"pid":WP,"window_id":WD,"x":wl[0],"y":wl[1],"session":"d1"})- time.sleep(0.5); D("press_key", {"pid":WP,"key":"escape","session":"d1"})- elif t == "drag":- if g: fx=g["x"]-int(WB["x"])+8; fy=g["y"]-int(WB["y"])+g["h"]//2; D("drag", {"pid":WP,"from_x":fx,"from_y":fy,"to_x":fx+150,"to_y":fy,"session":"d1"})- elif t == "scroll":- if DESKTOP and c: D("scroll", {"x":c[0],"y":c[1],"direction":"down","session":"d1"})- elif use_ax: D("scroll", {"pid":WP,"window_id":WD,"element_index":eidx,"direction":"down","session":"d1"})- elif c: D("scroll", {"pid":WP,"window_id":WD,"x":wl[0],"y":wl[1],"direction":"down","session":"d1"})- elif t == "setval":- if eidx is not None: D("set_value", {"pid":WP,"window_id":WD,"element_index":eidx,"value":"set-by-cua","session":"d1"})- elif t == "type":- D("type_text", {"pid":WP,"text":"typed-by-cua","session":"d1"})- elif t == "key":- D("press_key", {"pid":WP,"key":"tab","session":"d1"})--# ---------- run -----------try:- for i,(t,sel,label) in enumerate(PLAN):- steps[i]["state"]="active"; flush()- # establish the genuine foreground baseline: ACTIVATE the anchor and confirm it actually took- # the active window BEFORE the action runs. A "steal" is then the active window moving OFF the- # anchor ONTO the harness; "held" is the anchor staying active. (fg modes keep the harness in- # front by design, so no baseline anchor there.)- if not m["fg"] and ANCHOR_ID:- held = anchor_front()- open(f"{WORK}/baseline.log","a").write(- "step %d '%s': baseline active='%s' anchor_held=%s\n" % (i, t, active_name(), held))- before = hstate()- do(t, sel)- # measure: steal = active window moved OFF the anchor baseline onto the harness after the action.- stole=False; end=time.time()+1.5- while time.time()<end:- flush()- if "CuaTestHarness" in active_name(): stole=True- time.sleep(0.11)- steps[i]["verified"] = verify(t, before, hstate())- state["actions"]+=1- if m["fg"]: steps[i]["result"]="front"- elif stole: state["steals"]+=1; steps[i]["result"]="stole"- else: steps[i]["result"]="held"- steps[i]["state"]="done"; flush()- sh(["wmctrl","-r","cua-driver-panel","-b","add,above"])-- pulse(2.5)- if DESKTOP: D("set_config", {"key":"capture_scope","value":"window"})- D("stop_recording", {}); time.sleep(3)-finally:- # Dispose the agent-cursor session (remove its overlay cursor) BEFORE the daemon- # is pkill'd below. Runs even if the action loop raised, so the session is never- # leaked to the idle-TTL reaper. end_session is idempotent.- D("end_session", {"session":"d1"})-if ANCHOR_PROC is not None:- try: ANCHOR_PROC.terminate()- except Exception: pass-subprocess.run("pkill -f cua-anchor; pkill -f lin-harness.py; pkill -f lin-dash.py; pkill -f 'http.server 8146'; pkill cua-driver; pkill openbox", shell=True)-mp4 = (glob.glob(f"{REC}/**/*.mp4", recursive=True) or [None])[0]-size = os.path.getsize(mp4) if mp4 else 0-worked = sum(1 for s in steps if s.get("verified")=="ok"); ver = sum(1 for s in steps if s.get("verified") in ("ok","fail")); verdict = ("foreground-mode, %d actions"%state["actions"]) if m["fg"] else ("%d/%d stole focus"%(state["steals"],state["actions"]))-line = "MODE=%s MEASURE=%s EFFECTS=%d/%d_landed MP4=%s SIZE=%d" % (MODE, verdict, worked, ver, mp4, size)-open(f"{WORK}/metric.log","w").write(line); print(line)diff --git a/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-run-electron.sh b/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-run-electron.shdeleted file mode 100755index 12339dbc8a..0000000000--- a/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-run-electron.sh+++ /dev/null@@ -1,27 +0,0 @@-#!/bin/bash-# Env wrapper for the Electron modality recorder — mirrors lin-run.sh but-# launches lin-rec-electron.py (which spawns the Electron harness itself).-MODE="${1:-ax-bg}"-DRV="$HOME/cua/libs/cua-driver/rust/target/release/cua-driver"-export NO_AT_BRIDGE=0 GTK_A11Y=1 GDK_BACKEND=x11 CUA_DRIVER_RS_DRAW_SYSTEM_CURSOR=0-pkill Xvfb 2>/dev/null; pkill cua-driver 2>/dev/null; pkill picom 2>/dev/null-pkill -f 'http.server' 2>/dev/null; fuser -k 8146/tcp 2>/dev/null-pkill -x electron 2>/dev/null; pkill -f lin-dash 2>/dev/null-sleep 1.5-rm -f "/tmp/cua-lin-electron-$MODE/setup.log"-xvfb-run -a --server-args="-screen 0 1024x768x24" dbus-run-session -- bash -c '- export GDK_BACKEND=x11 CUA_DRIVER_RS_DRAW_SYSTEM_CURSOR=0- DRV="'"$DRV"'"- # bring up the AT-SPI a11y bus so Chromium can register its web-AX tree- for p in /usr/libexec/at-spi-bus-launcher /usr/lib/at-spi2-core/at-spi-bus-launcher; do- [ -x "$p" ] && "$p" --launch-immediately & break- done- sleep 1- openbox & sleep 1- picom --backend xrender --config /dev/null >/tmp/picom.log 2>&1 & sleep 1- xdotool mousemove 1010 758- "$DRV" serve >"/tmp/cua-lin-electron-'"$MODE"'-drv.log" 2>&1 & sleep 3- python3 /tmp/lin-rec-electron.py "'"$MODE"'"- pkill -x electron; pkill cua-driver; pkill picom; pkill openbox-' > "/tmp/cua-lin-electron-$MODE-run.log" 2>&1-echo "exit=$? mode=$MODE"diff --git a/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-run.sh b/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-run.shdeleted file mode 100644index 8139c51c45..0000000000--- a/libs/cua-driver/tests/fixtures/modality-recordings/linux/lin-run.sh+++ /dev/null@@ -1,20 +0,0 @@-#!/bin/bash-MODE="${1:-ax-bg}"-DRV="$HOME/cua/libs/cua-driver/rust/target/release/cua-driver"-export NO_AT_BRIDGE=0 GTK_A11Y=1 GDK_BACKEND=x11 CUA_DRIVER_RS_DRAW_SYSTEM_CURSOR=0-pkill Xvfb 2>/dev/null; pkill cua-driver 2>/dev/null; pkill picom 2>/dev/null-pkill -f 'http.server' 2>/dev/null; fuser -k 8146/tcp 2>/dev/null-pkill -f lin-harness 2>/dev/null; pkill -f lin-dash 2>/dev/null-sleep 1.5-rm -f "/tmp/cua-lin-$MODE/setup.log" /tmp/cua-lin-state.json-xvfb-run -a --server-args="-screen 0 1024x768x24" dbus-run-session -- bash -c '- export GDK_BACKEND=x11 CUA_DRIVER_RS_DRAW_SYSTEM_CURSOR=0- DRV="'"$DRV"'"- openbox & sleep 1- picom --backend xrender --config /dev/null >/tmp/picom.log 2>&1 & sleep 1- xdotool mousemove 1010 758 # park the real X pointer in the corner (only the agent cursor should show)- "$DRV" serve >"/tmp/cua-lin-'"$MODE"'-drv.log" 2>&1 & sleep 3- python3 /tmp/lin-rec.py "'"$MODE"'"- pkill cua-driver; pkill picom; pkill openbox-' > "/tmp/cua-lin-$MODE-run.log" 2>&1-echo "exit=$? mode=$MODE"diff --git a/libs/cua-driver/tests/fixtures/modality-recordings/macos/mac-rec.py b/libs/cua-driver/tests/fixtures/modality-recordings/macos/mac-rec.pydeleted file mode 100644index f3446b3acc..0000000000--- a/libs/cua-driver/tests/fixtures/modality-recordings/macos/mac-rec.py+++ /dev/null@@ -1,546 +0,0 @@-#!/usr/bin/env python3-# macOS AppKit modality recorder — parity with the Linux GTK3 / Windows WPF golden recorders.-# Harness LEFT (CuaTestHarness.AppKit), Chrome --app dashboard RIGHT.-# Per-action: no-foreground contract (held / STOLE) + effect verifier (worked / no-op).-import json, os, subprocess, sys, time, re, glob, signal, atexit--MODE = sys.argv[1] if len(sys.argv) > 1 else "ax-bg"-SURFACE = sys.argv[2] if len(sys.argv) > 2 else "appkit"-HOME = os.path.expanduser("~")-DRV = f"{HOME}/cua/libs/cua-driver/rust/target/release/cua-driver"-DASH_HTML = "/tmp/mac-dashboard.html"-CHROME = "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"-PORT = 8147--# ---------- per-surface configuration -----------SURF = {- "appkit": {- "bundle":"com.trycua.harness.appkit", "title":"CuaTestHarness AppKit",- "dash":"AppKit", "appmatch":"CuaTestHarness",- # window-local POINTS (measured live against the rebuilt parity harness, window 700x1112).- "coords":{"click_target":(129.0,304.0),"slider":(180.0,372.0),"scroll":(279.0,791.0),- "txt":(140.0,232.0),"increment":(62.0,96.0)},- },- "swiftui": {- "bundle":"com.trycua.harness.swiftui", "title":"CuaTestHarness SwiftUI",- "dash":"SwiftUI", "appmatch":"CuaTestHarness",- # window-local POINTS (measured live against the rebuilt parity harness, window 700x820).- "coords":{"click_target":(179.0,348.0),"slider":(230.0,424.0),"scroll":(210.0,690.0),- "txt":(190.0,268.0),"increment":(112.0,100.0),"popover":(125.0,904.0)},- },- "electron": {- "bundle":None, "launch_match":"CuaTestHarness Electron", "title":"CuaTestHarness Electron",- "dash":"Electron", "appmatch":"Electron",- "coords":{"click_target":(167.0,459.0),"slider":(45.0,367.0),"txt":(106.0,282.0),- "scroll":(340.0,400.0),"increment":(70.0,197.0)},- },- "wkwebview": { # Apple WebKit (NOT Chromium); same shared web DOM as Electron- "bundle":"com.trycua.harness.wkwebview", "title":"CuaTestHarness WKWebView",- "dash":"WKWebView", "appmatch":"CuaTestHarness",- "coords":{"click_target":(167.0,459.0),"slider":(45.0,367.0),"txt":(106.0,282.0),- "scroll":(340.0,400.0),"increment":(70.0,197.0)},- },-}-WEBISH = ("electron", "wkwebview") # web-AX harnesses sharing the same DOM/verifier-S = SURF[SURFACE]-BUNDLE = S["bundle"]; TITLE = S["title"]; APPMATCH = S["appmatch"]-WORK = f"/tmp/cua-mac-{SURFACE}-{MODE}"-REC = f"{WORK}/rec"--# window layout (points). screen ~1512x982, menu bar ~33.-HX, HY = 0, 40 # harness top-left-DASH_X, DASH_Y = 712, 40 # dashboard top-left-DASH_W, DASH_H = 792, 900--META = {- "ax-fg": {"title":"AX - FOREGROUND","see":"accessibility tree (element-level)","fg":True, "expect":"App kept in FRONT on purpose. Each action runs via the accessibility tree / element_index; we measure the foreground."},- "ax-bg": {"title":"AX - BACKGROUND","see":"accessibility tree (element-level)","fg":False,"expect":"App should stay in the BACKGROUND. Each action runs via the accessibility tree / element_index; we measure which actions steal focus."},- "px-fg": {"title":"VISION - FOREGROUND","see":"screenshot only (pixels)","fg":True, "expect":"Pure pixel-driven, app kept in FRONT. We measure the foreground."},- "px-bg": {"title":"VISION - BACKGROUND","see":"screenshot only (pixels)","fg":False,"expect":"Pure pixel-driven, app should stay in the BACKGROUND. We measure which pixel actions steal focus."},- "px-desktop":{"title":"VISION - FULL DESKTOP","see":"full-screen screenshot","fg":True,"expect":"capture_scope=desktop (full-display capture). Pure pixel-driven, app in FRONT. On macOS pixel dispatch is window-anchored, so actions route to the on-screen window."},-}-m = META[MODE]-VISION = m["see"].startswith("screenshot")-DESKTOP = MODE == "px-desktop"-SESS = f"macd-{MODE}-{int(time.time())}"--def D(tool, payload, t=25):- try:- p = subprocess.run([DRV, "call", tool], input=json.dumps(payload),- capture_output=True, text=True, timeout=t)- return p.stdout- except Exception:- return ""--def DJ(tool, payload, t=25):- try: return json.loads(D(tool, payload, t) or "{}")- except Exception: return {}--# Safety net: dispose the agent-cursor session (remove its overlay cursor) even if-# the run raises before the explicit end_session below. This script does NOT kill-# the daemon on exit, so the daemon is still alive at interpreter shutdown and the-# atexit call reaches it. end_session is idempotent, so the happy-path call is a-# no-op here.-atexit.register(lambda: D("end_session", {"session": SESS}))--def osa(script, t=6):- try: return subprocess.run(["osascript","-e",script], capture_output=True, text=True, timeout=t).stdout.strip()- except Exception: return ""--def frontmost():- return osa('tell application "System Events" to get name of first application process whose frontmost is true')--def app_is_front():- return APPMATCH in frontmost()--def activate_harness():- osa(f'tell application "System Events" to set frontmost of (first process whose unix id is {PID}) to true')--def activate_dash():- osa('tell application "Google Chrome" to activate')--def position_harness():- # pin position AND size: window height otherwise varies (832/858) between launches,- # which shifts the screenshot dims and breaks pixel-coord calibration.- osa(f'''tell application "System Events"- set p to first process whose unix id is {PID}- set w to first window of p whose title contains "{TITLE}"- set position of w to {{{HX}, {HY}}}- set size of w to {{700, 820}}- end tell''')--# ---------- coords (window-local screenshot-px; window 700x858 -> shot 1279x1568) -----------# Targets stored as WINDOW-LOCAL POINTS (stable across window-height drift, top-anchored-# layout). Converted to live screenshot-pixels at runtime: px = pt * shot_dim / win_dim.-# Calibrated against a pinned 700x820 window (shot 1339x1568).-WIN_W = 700.0-COORDS_PT = S["coords"]-# live dims, measured right before recording-SHOT_W, SHOT_H, WIN_H = 1339.0, 1568.0, 820.0-def coord(sel):- px, py = COORDS_PT[sel]- return (px * SHOT_W / WIN_W, py * SHOT_H / WIN_H)--# ---------- plan (per surface) -----------PLANS = {- "appkit": [- ("click","click_target","left-click the click-target"),- ("double","click_target","double-click the click-target"),- ("right","click_target","right-click the click-target"),- ("drag","click_target","drag across the click-target"),- ("scroll","scroll","scroll the list"),- ("setval","txt","set_value on the text box"),- ("type","txt","type into the text box"),- ("key","txt","press a key (Tab)"),- ],- "swiftui": [- ("click","click_target","left-click the click-target"),- ("double","click_target","double-click the click-target"),- ("right","click_target","right-click the click-target"),- ("drag","slider","drag the slider"),- ("scroll","scroll","scroll the list"),- ("setval","txt","set_value on the text box"),- ("type","txt","type into the text box"),- ("key","txt","press a key (Tab)"),- ],- "electron": [- ("click","click_target","left-click the click-target"),- ("double","click_target","double-click the click-target"),- ("right","click_target","right-click the click-target"),- ("drag","slider","drag the slider"),- ("scroll","scroll","scroll the page"),- ("setval","txt","set_value on the text box"),- ("type","txt","type into the text box"),- ("key","txt","press a key (Tab)"),- ],-}-PLANS["wkwebview"] = PLANS["electron"]-PLAN = list(PLANS[SURFACE])-if VISION: PLAN = [p for p in PLAN if p[0] != "setval"]-if DESKTOP: PLAN = [p for p in PLAN if p[0] in ("click","scroll","type","key")]--steps = [{"label":l,"state":"pending","result":"","verified":""} for (_,_,l) in PLAN]-state = {"steals":0,"actions":0}--# ---------- harness state reader (parse AX tree_markdown) -----------def hstate():- d = DJ("get_window_state", {"pid":PID,"window_id":WID,"capture_mode":"ax"})- tm = d.get("tree_markdown","") or ""- if SURFACE in WEBISH:- # web harness exposes plain-text status labels (counter=N, mirror=..., etc.)- st = {"last":"none","clicks":0,"slider":0,"counter":0,"txt":""}- m1 = re.search(r"last_action=(\w+)", tm); st["last"] = m1.group(1) if m1 else "none"- m2 = re.search(r"clicks=(\d+)", tm); st["clicks"] = int(m2.group(1)) if m2 else 0- m3 = re.search(r"slider_value=(\d+)", tm); st["slider"] = int(m3.group(1)) if m3 else 0- m4 = re.search(r"counter=(\d+)", tm); st["counter"] = int(m4.group(1)) if m4 else 0- m5 = re.search(r"mirror=([^\"]*)\"", tm); st["txt"] = m5.group(1) if m5 else ""- return st- # NATIVE (appkit/swiftui): rebuilt parity harness exposes the SAME web-style status- # labels in the AX tree_markdown — last_action=, clicks=, slider_value=, scroll_offset=,- # agreed=, menu_action=. Text mirror is the AXTextField value (id=txt-input).- st = {"last":"none","clicks":0,"slider":0,"scroll":0,"menu":"none","agreed":"false","txt":""}- m1 = re.search(r"last_action=(\w+)", tm); st["last"] = m1.group(1) if m1 else "none"- m2 = re.search(r"clicks=(\d+)", tm); st["clicks"] = int(m2.group(1)) if m2 else 0- m3 = re.search(r"slider_value=(\d+)", tm); st["slider"] = int(m3.group(1)) if m3 else 0- m4 = re.search(r"scroll_offset=(\d+)", tm); st["scroll"] = int(m4.group(1)) if m4 else 0- m5 = re.search(r"menu_action=([^\"]+)\"", tm); st["menu"] = m5.group(1) if m5 else "none"- m6 = re.search(r"agreed=(\w+)", tm); st["agreed"] = m6.group(1) if m6 else "false"- # txt-input value (placeholder "Type here…" when empty — substring checks below won't match it)- tv = re.search(r'AXTextField = "([^"]*)" \[id=txt-input', tm) or re.search(r'AXTextField = "([^"]*)"', tm)- if tv: st["txt"] = tv.group(1)- return st--AXMODE = (not VISION) and (not DESKTOP)-def verify(t, b, a):- if SURFACE in WEBISH:- if t == "click": return "ok" if ("left_click" in a["last"] and a["clicks"]>b["clicks"]) else "fail"- if t == "double": return "ok" if "double_click" in a["last"] else "fail"- if t == "right": return "ok" if "right_click" in a["last"] else "fail"- if t == "drag": return "ok" if a["slider"]>b["slider"] else "fail"- if t == "setval": return "ok" if "set-by-cua" in a["txt"] else "fail"- if t == "type": return "ok" if "typed-by-cua" in a["txt"] else "fail"- return "na" # scroll/key: no verifiable readout- # NATIVE appkit/swiftui — real parity signals from the click-target / slider /- # scroll-target / text box (matches the WEBISH verifier; only the readouts differ).- if t == "click": return "ok" if (a["clicks"]>b["clicks"] or a["last"] in ("click","left_click")) else "fail"- # AppKit's click-target reports last_action=double_click for a pixel double; SwiftUI's- # does not distinguish it, so a landed double shows up as clicks incrementing (+2).- if t == "double": return "ok" if ("double" in a["last"] or a["clicks"]>b["clicks"]) else "fail"- # right-click: last_action=right_click (AppKit) or a context menu_action change.- if t == "right": return "ok" if ("right" in a["last"] or a["menu"]!=b["menu"]) else "fail"- if t == "drag": return "ok" if a["slider"]>b["slider"] else "fail"- if t == "scroll": return "ok" if a["scroll"]>b["scroll"] else "fail"- if t == "setval": return "ok" if "set-by-cua" in a["txt"] else "fail"- if t == "type": return "ok" if "typed-by-cua" in a["txt"] else "fail"- return "na" # key: no verifiable readout--# ---------- status.json writer for dashboard -----------_front_cache = {"v":"", "t":0}-def flush(refresh_front=True):- if refresh_front:- _front_cache["v"] = frontmost()- fg = _front_cache["v"]- af = APPMATCH in fg- st = {"run":m["title"],"expect":m["expect"],"fgmode":m["fg"],"foreground":fg,"appFront":af,- "steals":state["steals"],"actions":state["actions"],"steps":steps}- with open(f"{WORK}/status.json","w") as f: json.dump(st, f)--def pulse(sec):- end = time.time()+sec- while time.time() < end:- flush(); time.sleep(0.18)--# ---------- setup -----------os.makedirs(REC, exist_ok=True)-subprocess.run("pkill -f mac-dashboard; pkill -f 'http.server 8147'", shell=True)-subprocess.run(f"pkill -f CuaTestHarness", shell=True)-if SURFACE == "electron":- subprocess.run("pkill -f 'electron/dist/Electron'; pkill -f 'Electron.app/Contents/MacOS/Electron'", shell=True)-time.sleep(2.0)-# dashboard.html with per-surface title suffix (h1 "... - AppKit" -> "... - <surface>")-_html = open(DASH_HTML).read().replace("single-modality run - AppKit", f"single-modality run - {S['dash']}")-open(f"{WORK}/dashboard.html","w").write(_html)-with open(f"{WORK}/status.json","w") as f: json.dump({"run":"","steps":[]}, f)-log = open(f"{WORK}/setup.log","w")-def L(s): log.write(s+"\n"); log.flush()--# http server for dashboard-subprocess.Popen(["python3","-m","http.server",str(PORT),"--directory",WORK],- stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)-time.sleep(1)--# launch harness-if BUNDLE:- la = DJ("launch_app", {"bundle_id":BUNDLE})- PID = la.get("pid")- wins = la.get("windows", [])- main = next((w for w in wins if TITLE in (w.get("title") or "")), None)- WID = main["window_id"] if main else (wins[0]["window_id"] if wins else None)-else:- # electron: launched out-of-band (see ELECTRON_CMD); poll list_windows by title substring- if os.environ.get("ELECTRON_CMD"):- subprocess.Popen(os.environ["ELECTRON_CMD"], shell=True,- stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)- PID = WID = None- for _ in range(40):- for w in DJ("list_windows", {}).get("windows", []):- if S["launch_match"] in (w.get("title") or ""):- PID, WID = w["pid"], w["window_id"]; break- if PID: break- time.sleep(0.5)-L(f"launched pid={PID} wid={WID}")-if not PID or not WID:- L("FATAL no harness"); open(f"{WORK}/metric.log","w").write("FATAL no harness"); sys.exit(1)--# make harness visible (recording carve-out) + position LEFT-activate_harness(); time.sleep(0.6)-position_harness(); time.sleep(0.4)-# refresh window id (re-list, pick on-screen main window)-wl = DJ("list_windows", {"pid":PID}).get("windows", [])-mm = next((w for w in wl if TITLE in (w.get("title") or "") and w.get("is_on_screen")), None)-if mm: WID = mm["window_id"]; WIN_ORIGIN = (mm["bounds"]["x"], mm["bounds"]["y"])-else: WIN_ORIGIN = (HX, HY)-L(f"after position wid={WID} origin={WIN_ORIGIN} onscreen_main={bool(mm)}")--# launch Chrome --app dashboard, positioned RIGHT-prof = f"/tmp/cua-dash-profile-{SURFACE}-{MODE}"-subprocess.Popen([CHROME, f"--app=http://127.0.0.1:{PORT}/dashboard.html",- f"--user-data-dir={prof}", "--no-first-run", "--no-default-browser-check",- f"--window-position={DASH_X},{DASH_Y}", f"--window-size={DASH_W},{DASH_H}"],- stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)-time.sleep(4)-# re-position dashboard via System Events as a backstop-osa(f'''tell application "System Events"- set cp to first process whose name is "Google Chrome"- try- set position of front window of cp to {{{DASH_X}, {DASH_Y}}}- set size of front window of cp to {{{DASH_W}, {DASH_H}}}- end try-end tell''')-time.sleep(1)-L(f"frontmost after chrome: {frontmost()}")--# resolve element indices FRESH from a snapshot (first occurrence = primary window subtree).-# Re-resolved right before each AX action to honour the snapshot-before-action invariant.-def axidx():- d = DJ("get_window_state", {"pid":PID,"window_id":WID,"capture_mode":"ax"})- tm = d.get("tree_markdown","") or ""- idx = {}- for mt in re.finditer(r'\[(\d+)\] (AX\w+)[^\n]*?\[id=([\w-]+)', tm):- if mt.group(3) not in idx: idx[mt.group(3)] = int(mt.group(1))- ta = re.search(r'\[(\d+)\] AXTextArea', tm)- if ta: idx["scroll-area"] = int(ta.group(1))- # role/label fallbacks (Electron web-AX exposes no [id=...])- if "txt-input" not in idx:- mt = re.search(r'\[(\d+)\] AXTextField', tm)- if mt: idx["txt-input"] = int(mt.group(1))- if "slider" not in idx:- mt = re.search(r'\[(\d+)\] AXSlider', tm)- if mt: idx["slider"] = int(mt.group(1))- if "btn-increment" not in idx:- mt = re.search(r'\[(\d+)\] AXButton "Increment"', tm)- if mt: idx["btn-increment"] = int(mt.group(1))- return idx--IDX0 = axidx()-open(f"{WORK}/resolve.log","w").write("IDX="+json.dumps(IDX0))-L(f"resolved idx={IDX0}")--# agent cursor-D("start_session", {"session":SESS})-D("set_agent_cursor_enabled", {"enabled":True,"session":SESS})-D("set_agent_cursor_motion", {"session":SESS,"cursor_color":"#FF2D2D","cursor_label":"cua-driver",- "glide_duration_ms":600,"dwell_after_click_ms":650,"idle_hide_ms":120000})--# re-pin size and measure LIVE screenshot dims + window height (drift-proof coords)-position_harness(); time.sleep(0.6)-ws = DJ("get_window_state", {"pid":PID,"window_id":WID,"capture_mode":"som"})-if ws.get("screenshot_width"): SHOT_W = float(ws["screenshot_width"])-if ws.get("screenshot_height"): SHOT_H = float(ws["screenshot_height"])-wl2 = DJ("list_windows", {"pid":PID}).get("windows", [])-mm2 = next((w for w in wl2 if w.get("window_id")==WID), None)-if mm2:- WIN_H = float(mm2["bounds"]["height"]); WIN_ORIGIN = (mm2["bounds"]["x"], mm2["bounds"]["y"])-L(f"live dims shot={SHOT_W}x{SHOT_H} win_h={WIN_H} origin={WIN_ORIGIN}")--# foreground/background precondition-if m["fg"]: activate_harness()-else: activate_dash()-time.sleep(0.8)--# seed cursor on-screen before recording (so AX actions glide visibly)-D("move_cursor", {"pid":PID,"window_id":WID,"x":480,"y":760,"session":SESS})-time.sleep(0.4)---if DESKTOP:- D("set_config", {"key":"capture_scope","value":"desktop"})-D("start_recording", {"output_dir":REC,"record_video":True})-time.sleep(0.5)-pulse(2.0)--# ---------- action dispatch -----------def to_screen(px, py):- # screenshot-px -> screen points- return (WIN_ORIGIN[0] + px*WIN_W/SHOT_W, WIN_ORIGIN[1] + py*WIN_H/SHOT_H)--def glide(sel):- px, py = coord(sel)- if DESKTOP:- sx, sy = to_screen(px, py)- D("move_cursor", {"x":sx,"y":sy,"session":SESS})- else:- D("move_cursor", {"pid":PID,"window_id":WID,"x":px,"y":py,"session":SESS})- time.sleep(0.5)--def web_snapshot():- d = DJ("get_window_state", {"pid":PID,"window_id":WID,"capture_mode":"ax"})- return d.get("elements", []) or []--def web_el(els, sel):- # Resolve a web control to (element_index, screenshot_px_x, screenshot_px_y) from its- # ACTUAL live frame — never the per-surface COORDS_PT, which drifted ~185px off WKWebView's- # real web layout so every pixel action missed. WebKit duplicates the web subtree; the- # lower-index copy is the live/hittable one, so pick the smallest matching element_index.- def first(pred):- cand = [e for e in els if pred(e)]- return min(cand, key=lambda e: e.get("element_index", 9999)) if cand else None- role = lambda e: str(e.get("role", ""))- lbl = lambda e: str(e.get("label") or e.get("name") or "")- if sel == "click_target": e = first(lambda e: "Click target" in lbl(e))- elif sel == "slider": e = first(lambda e: "AXSlider" in role(e))- elif sel == "txt": e = first(lambda e: "AXTextField" in role(e))- elif sel == "scroll": e = first(lambda e: "scroll_target" in lbl(e)) or first(lambda e: "AXTextArea" in role(e))- else: e = None- if not e: return (None, None, None)- f = e["frame"]; cx = f["x"] + f["w"]/2.0; cy = f["y"] + f["h"]/2.0- px = (cx - WIN_ORIGIN[0]) * SHOT_W / WIN_W- py = (cy - WIN_ORIGIN[1]) * SHOT_H / WIN_H- return (e.get("element_index"), px, py)--def glide_to(px, py):- if px is None: return- if DESKTOP:- sx, sy = to_screen(px, py); D("move_cursor", {"x":sx,"y":sy,"session":SESS})- else:- D("move_cursor", {"pid":PID,"window_id":WID,"x":px,"y":py,"session":SESS})- time.sleep(0.5)--def do_electron(t, sel):- # Web harness (WKWebView / Electron). AX mode dispatches by element_index — AXPress works on- # the web spans (verified live: click/double/right/type all land). Vision mode uses pixel- # coords derived from the live element frame. Both replace the stale hardcoded COORDS_PT that- # made every WKWebView pixel action miss. set_value on a web input sets AXValue but doesn't- # fire the DOM input event (mirror stays empty) — an honest web limitation; type_text works.- els = web_snapshot()- ct_idx, cpx, cpy = web_el(els, "click_target")- sl_idx, spx, spy = web_el(els, "slider")- tx_idx, tpx, tpy = web_el(els, "txt")- sc_idx, rpx, rpy = web_el(els, "scroll")- if t == "click":- glide_to(cpx, cpy)- if AXMODE and ct_idx is not None: D("click", {"pid":PID,"window_id":WID,"element_index":ct_idx,"session":SESS})- else: D("click", {"pid":PID,"window_id":WID,"x":cpx,"y":cpy,"session":SESS})- elif t == "double":- glide_to(cpx, cpy)- if AXMODE and ct_idx is not None: D("double_click", {"pid":PID,"window_id":WID,"element_index":ct_idx,"session":SESS})- else: D("double_click", {"pid":PID,"window_id":WID,"x":cpx,"y":cpy,"count":2,"session":SESS})- elif t == "right":- glide_to(cpx, cpy)- if AXMODE and ct_idx is not None: D("right_click", {"pid":PID,"window_id":WID,"element_index":ct_idx,"session":SESS})- else: D("right_click", {"pid":PID,"window_id":WID,"x":cpx,"y":cpy,"session":SESS})- elif t == "drag":- # slider thumb: a real synthetic pixel drag fires the range input's event (AXValue-set- # does not). Correct coords come from the live slider frame.- glide_to(spx, spy)- if spx is not None: D("drag", {"pid":PID,"window_id":WID,"from_x":spx,"from_y":spy,"to_x":spx+220,"to_y":spy,"session":SESS})- elif t == "scroll":- glide_to(rpx, rpy)- if rpx is not None: D("scroll", {"pid":PID,"window_id":WID,"x":rpx,"y":rpy,"direction":"down","amount":5,"session":SESS})- elif t == "setval":- if tx_idx is not None: D("set_value", {"pid":PID,"window_id":WID,"element_index":tx_idx,"value":"set-by-cua","session":SESS})- elif t == "type":- if AXMODE and tx_idx is not None:- glide_to(tpx, tpy)- D("set_value", {"pid":PID,"window_id":WID,"element_index":tx_idx,"value":"","session":SESS}); time.sleep(0.4)- D("click", {"pid":PID,"window_id":WID,"element_index":tx_idx,"session":SESS}); time.sleep(0.3)- D("type_text", {"pid":PID,"window_id":WID,"element_index":tx_idx,"text":"typed-by-cua","session":SESS})- else:- glide_to(tpx, tpy); D("click", {"pid":PID,"window_id":WID,"x":tpx,"y":tpy,"session":SESS}); time.sleep(0.3)- D("type_text", {"pid":PID,"text":"typed-by-cua","session":SESS})- elif t == "key":- D("press_key", {"pid":PID,"key":"tab","session":SESS})--def do_native(t, sel):- # appkit / swiftui — rebuilt parity controls (click-target / slider / scroll-target /- # text box), mirroring the electron plan. AX press lands the click-target's primary- # action (last_action=click, works backgrounded). double/right/drag are real- # mouse-button gestures the view records via pixel events. scroll: AppKit exposes an- # AXTextArea (id=scroll-tall) addressable by element_index; SwiftUI's nested ScrollView- # is pixel-only. type: a PIXEL click grabs first responder (AX-press on a textfield does- # not), then keyboard type_text — set_value clears first in AX modes.- IDX = axidx() if AXMODE else {}- ct = IDX.get("btn-clicktarget"); txt = IDX.get("txt-input")- sa = IDX.get("scroll-tall") or IDX.get("scroll-area") # AppKit AXTextArea (multiline value hides its [id=])- cpx, cpy = coord("click_target"); spx, spy = coord("slider"); tpx, tpy = coord("txt")- if t == "click":- glide("click_target")- if AXMODE and ct is not None: D("click", {"pid":PID,"window_id":WID,"element_index":ct,"session":SESS})- else: D("click", {"pid":PID,"window_id":WID,"x":cpx,"y":cpy,"session":SESS})- elif t == "double":- glide("click_target"); D("double_click", {"pid":PID,"window_id":WID,"x":cpx,"y":cpy,"count":2,"session":SESS})- elif t == "right":- glide("click_target"); D("right_click", {"pid":PID,"window_id":WID,"x":cpx,"y":cpy,"session":SESS})- D("press_key", {"pid":PID,"key":"escape","session":SESS}) # dismiss any context menu- elif t == "drag":- glide("slider"); D("drag", {"pid":PID,"window_id":WID,"from_x":spx,"from_y":spy,"to_x":spx+220,"to_y":spy,"session":SESS})- elif t == "scroll":- glide("scroll"); rpx, rpy = coord("scroll")- if AXMODE and sa is not None:- D("scroll", {"pid":PID,"window_id":WID,"element_index":sa,"direction":"down","amount":5,"session":SESS})- else:- D("scroll", {"pid":PID,"window_id":WID,"x":rpx,"y":rpy,"direction":"down","amount":8,"session":SESS})- elif t == "setval":- glide("txt")- if AXMODE and txt is not None:- D("set_value", {"pid":PID,"window_id":WID,"element_index":txt,"value":"set-by-cua","session":SESS})- elif t == "type":- glide("txt")- if AXMODE and txt is not None:- D("set_value", {"pid":PID,"window_id":WID,"element_index":txt,"value":"","session":SESS}); time.sleep(0.4)- D("click", {"pid":PID,"window_id":WID,"x":tpx,"y":tpy,"session":SESS}); time.sleep(0.4)- D("type_text", {"pid":PID,"text":"typed-by-cua","session":SESS})- elif t == "key":- D("press_key", {"pid":PID,"key":"tab","session":SESS})--def do(t, sel):- if SURFACE in WEBISH:- return do_electron(t, sel)- return do_native(t, sel)--# ---------- run -----------for i,(t,sel,label) in enumerate(PLAN):- steps[i]["state"]="active"; flush()- if not m["fg"]:- activate_dash(); time.sleep(0.4) # re-assert background precondition- before = hstate()- do(t, sel)- stole=False; end=time.time()+1.6- while time.time()<end:- flush()- if app_is_front(): stole=True- time.sleep(0.12)- after = hstate()- if SURFACE in WEBISH and verify(t, before, after) == "fail":- # Chromium web-AX status text can lag the DOM by >1.5s, esp. backgrounded;- # re-poll so a landed-but-late effect isn't mis-scored as a no-op.- for _ in range(7):- time.sleep(0.4); after = hstate()- if verify(t, before, after) != "fail": break- steps[i]["verified"] = verify(t, before, after)- if t == "popover": # dismiss so it doesn't poison later snapshots- D("press_key", {"pid":PID,"key":"escape","session":SESS}); time.sleep(0.3)- state["actions"]+=1- if m["fg"]: steps[i]["result"]="front"- elif stole: state["steals"]+=1; steps[i]["result"]="stole"- else: steps[i]["result"]="held"- steps[i]["state"]="done"; flush()- L(f"step {t}: result={steps[i]['result']} verified={steps[i]['verified']} before={before} after={after}")--pulse(2.5)-if DESKTOP:- D("set_config", {"key":"capture_scope","value":"window"})-D("stop_recording", {}); time.sleep(3)-D("end_session", {"session":SESS})-subprocess.run("pkill -f mac-dashboard; pkill -f 'http.server 8147'", shell=True)-subprocess.run(f"pkill -f 'user-data-dir=/tmp/cua-dash-profile-{SURFACE}-{MODE}'", shell=True)-if SURFACE == "electron":- subprocess.run("pkill -f 'Electron.app/Contents/MacOS/Electron'", shell=True)--mp4 = (glob.glob(f"{REC}/**/*.mp4", recursive=True) or [None])[0]-size = os.path.getsize(mp4) if mp4 else 0-worked = sum(1 for s in steps if s.get("verified")=="ok")-ver = sum(1 for s in steps if s.get("verified") in ("ok","fail"))-verdict = ("foreground-mode, %d actions"%state["actions"]) if m["fg"] else ("%d/%d stole focus"%(state["steals"],state["actions"]))-line = "MODE=%s MEASURE=%s EFFECTS=%d/%d_landed MP4=%s SIZE=%d" % (MODE, verdict, worked, ver, mp4, size)-open(f"{WORK}/metric.log","w").write(line); print(line)diff --git a/libs/cua-driver/tests/fixtures/modality-recordings/windows/run-one.ps1 b/libs/cua-driver/tests/fixtures/modality-recordings/windows/run-one.ps1deleted file mode 100644index 09a84543c8..0000000000--- a/libs/cua-driver/tests/fixtures/modality-recordings/windows/run-one.ps1+++ /dev/null@@ -1,4 +0,0 @@-param([string]$Mode="ax-bg")-Get-Process cua-driver,CuaTestHarness.Wpf,chrome -EA SilentlyContinue | Stop-Process -Force -EA SilentlyContinue-Start-Sleep 2-& powershell -NoProfile -ExecutionPolicy Bypass -File "$PSScriptRoot\wpf-recorder.ps1" -Mode $Mode *> "C:\Users\Public\cua-$Mode-run.log"diff --git a/libs/cua-driver/tests/fixtures/modality-recordings/windows/wpf-recorder.ps1 b/libs/cua-driver/tests/fixtures/modality-recordings/windows/wpf-recorder.ps1deleted file mode 100644index 5d847e19f2..0000000000--- a/libs/cua-driver/tests/fixtures/modality-recordings/windows/wpf-recorder.ps1+++ /dev/null@@ -1,365 +0,0 @@-param([string]$Mode="ax-bg",[string]$Toolkit="wpf") # Mode: ax-fg|ax-bg|px-fg|px-bg|px-desktop ; Toolkit: wpf|winui3|webview2|electron-$TK=@{- wpf =@{exe="C:\Users\cuademo\cua\libs\cua-driver\rust\test-apps\harness-wpf\CuaTestHarness.Wpf.exe"; title="CuaTestHarness WPF"; label="WPF"}- winui3 =@{exe="C:\Users\cuademo\cua\libs\cua-driver\rust\test-apps\harness-winui3\CuaTestHarness.WinUI3.exe"; title="CuaTestHarness WinUI3"; label="WinUI3"}- webview2=@{exe="C:\Users\cuademo\cua\libs\cua-driver\rust\test-apps\harness-webview\CuaTestHarness.WebView.exe"; title="CuaTestHarness WebView"; label="WebView2"}- electron=@{exe="C:\Users\cuademo\cua\libs\cua-driver\rust\test-apps\harness-electron\CuaTestHarness.Electron.exe";title="CuaTestHarness Electron"; label="Electron"}-}-$tk=$TK[$Toolkit]; if(-not $tk){ Write-Output "unknown toolkit $Toolkit"; exit 3 }-$ErrorActionPreference="Continue"-# screen is 1024x768 -> harness LEFT, dashboard panel RIGHT, both fully on-screen.-# Harness LEFT, dashboard panel RIGHT (1024x768). At this width the WPF form reflows-# taller than the screen, so the right-click button + scroll area sit off-screen; the-# cua-driver off-screen guard now reports those as a clean no-op (no taskbar misfire)-# instead of clicking the wrong target. Full-width avoids that but makes the harness-# the foreground window (breaks the no-foreground measurement) — so we keep this layout.-$HARX=0;$HARY=0;$HARW=556;$HARH=742-$PANX=560;$PANY=0;$PANW=462;$PANH=742-$META=@{- "ax-fg" =@{title="AX - FOREGROUND"; scope="window"; see="accessibility tree (element-level)"; fg=$true; expect="App kept in FRONT on purpose. Each action runs via the accessibility tree; we measure the foreground."}- "ax-bg" =@{title="AX - BACKGROUND"; scope="window"; see="accessibility tree (element-level)"; fg=$false; expect="App should stay in the BACKGROUND. Each action runs via the accessibility tree; we measure which actions steal focus."}- "px-fg" =@{title="VISION - FOREGROUND"; scope="window"; see="screenshot only (pixels)"; fg=$true; expect="Pure pixel-driven, app kept in FRONT. We measure the foreground."}- "px-bg" =@{title="VISION - BACKGROUND"; scope="window"; see="screenshot only (pixels)"; fg=$false; expect="Pure pixel-driven, app should stay in the BACKGROUND. We measure which pixel actions steal focus."}- "px-desktop"=@{title="VISION - FULL DESKTOP"; scope="desktop"; see="full-screen screenshot"; fg=$true; expect="Whole-screen, window-less screen-pixel actions (no window targeted)."}-}-$m=$META[$Mode]; if(-not $m){ Write-Output "unknown mode $Mode"; exit 2 }-$vision=($m.see -like 'screenshot*'); $desktop=($m.scope -eq 'desktop')-$dir="C:\Users\Public\cua-$Toolkit-$Mode"; $rec="$dir\rec"-Remove-Item $dir -Recurse -Force -EA SilentlyContinue; New-Item -ItemType Directory -Force $rec | Out-Null-# ---------- dashboard (compact for 462px) -----------$html=@'-<!doctype html><html><head><meta charset="utf-8"><title>cua-driver-panel</title><style>- html,body{margin:0;height:100%;font-family:Segoe UI,-apple-system,sans-serif;background:#0d1117;color:#e6edf3}.wrap{padding:12px}- h1{font-size:11px;margin:0 0 1px;color:#8b949e}.run{font-size:17px;font-weight:800;color:#58a6ff;margin:0 0 5px}.exp{font-size:10.5px;color:#8b949e;margin:0 0 9px;line-height:1.35}- .fg{padding:9px 11px;border-radius:9px;font-weight:800;font-size:12.5px;margin-bottom:6px;line-height:1.25}.fg.ok{background:#0f2e1a;color:#3fb950;border:1px solid #2ea043}.fg.bad{background:#3d1418;color:#ff6a5f;border:1px solid #da3633}.fg.fgmode{background:#16202e;color:#58a6ff;border:1px solid #2f4664}- .sub{font-weight:500;font-size:9.5px;color:#c9d1d9;opacity:.85;margin-top:2px}.now{font-size:9.5px;font-family:Consolas,monospace;color:#6e7681;margin:5px 0 9px;word-break:break-all}- .row{padding:6px 8px;border-radius:7px;margin:4px 0;font-size:11.5px;display:flex;align-items:center;gap:7px}.row.done{background:#11161d}.row.active{background:#161b22;border:1px solid #d29922}.row.pending{opacity:.5}- .ic{width:14px;text-align:center}.lbl{flex:1}.res{font-size:9px;font-weight:700;padding:2px 5px;border-radius:5px;white-space:nowrap;margin-left:4px}.res.held{background:#0f2e1a;color:#3fb950}.res.stole{background:#3d1418;color:#ff6a5f}.res.front{background:#16202e;color:#58a6ff}- .res.did{background:#10301c;color:#56d364;border:1px solid #238636}.res.nope{background:#3d1418;color:#ff7b72;border:1px solid #da3633}- .legend{font-size:9px;color:#6e7681;margin:2px 0 8px;line-height:1.4}- .tally{margin-top:10px;font-size:11px;color:#8b949e}.tally b{color:#c9d1d9}-</style></head><body><div class="wrap">- <h1>cua-driver - single-modality run - TKLABEL</h1><div class="run" id="run">...</div><div class="exp" id="exp"></div>- <div id="fg" class="fg fgmode">...</div><div class="now" id="now"></div>- <div class="legend"><b style="color:#56d364">✓ worked</b> / <b style="color:#ff7b72">✗ no-op</b> = did the action change the app · held / STOLE = focus contract</div>- <div id="rows"></div><div class="tally" id="tally"></div>-</div><script>-async function tick(){try{const r=await fetch('status.json',{cache:'no-store'});const s=await r.json();- run.textContent=s.run||'';exp.textContent=s.expect||'';now.textContent='foreground now: '+(s.foreground||'?');- if(s.fgmode){fg.className='fg fgmode';fg.innerHTML='FOREGROUND MODE - app intentionally in front<div class="sub">'+(s.appFront?'app is foreground':'...')+'</div>';}- else if(s.appFront){fg.className='fg bad';fg.innerHTML='APP IS FOREGROUND<div class="sub">'+(s.foreground||'')+' - current action stole focus</div>';}- else{fg.className='fg ok';fg.innerHTML='APP IN BACKGROUND<div class="sub">no-foreground contract holding - '+(s.foreground||'')+' stays frontmost</div>';}- rows.innerHTML='';(s.steps||[]).forEach(st=>{const d=document.createElement('div');d.className='row '+st.state;- const ic=st.state==='done'?'\u2713':st.state==='active'?'\u25B6':'\u00B7';let res='';if(st.result==='held')res='<span class="res held">held</span>';else if(st.result==='stole')res='<span class="res stole">STOLE FOCUS</span>';else if(st.result==='front')res='<span class="res front">foreground</span>';- let ver='';if(st.verified==='ok')ver='<span class="res did">\u2713 worked</span>';else if(st.verified==='fail')ver='<span class="res nope">\u2717 no-op</span>';- d.innerHTML='<span class="ic">'+ic+'</span><span class="lbl">'+st.label+'</span>'+ver+res;rows.appendChild(d);});- const worked=(s.steps||[]).filter(x=>x.verified==='ok').length;const ve=(s.steps||[]).filter(x=>x.verified==='ok'||x.verified==='fail').length;- const eff='effects: <b>'+worked+'/'+ve+'</b> actions changed the app';- if(!s.fgmode){tally.innerHTML=eff+'<br>focus contract: <b>'+(s.steals||0)+'/'+(s.actions||0)+'</b> stole focus';}else{tally.innerHTML=eff+'<br><b>'+(s.actions||0)+'</b> actions performed (foreground mode)';}-}catch(e){}}-setInterval(tick,200);tick();-</script></body></html>-'@-$html=$html -replace 'TKLABEL', $tk.label-Set-Content "$dir\dashboard.html" $html -Encoding UTF8-# ---------- loopback server -----------$srv=Start-Job -ScriptBlock { param($d,$port)- $l=[System.Net.Sockets.TcpListener]::new([System.Net.IPAddress]::Loopback,$port);$l.Start()- while($true){try{$c=$l.AcceptTcpClient();$st=$c.GetStream();$rd=[IO.StreamReader]::new($st);$ln=$rd.ReadLine()- if($ln -match 'GET\s+(\S+)'){$pp=($matches[1].TrimStart('/') -split '\?')[0];if($pp -eq ''){$pp='dashboard.html'}- $fp=Join-Path $d $pp; if(Test-Path $fp){$b=[IO.File]::ReadAllBytes($fp);$ct=if($fp -like '*.html'){'text/html; charset=utf-8'}else{'application/json; charset=utf-8'}- $h="HTTP/1.1 200 OK`r`nContent-Type: $ct`r`nContent-Length: $($b.Length)`r`nCache-Control: no-store`r`nConnection: close`r`n`r`n";$hb=[Text.Encoding]::ASCII.GetBytes($h);$st.Write($hb,0,$hb.Length);$st.Write($b,0,$b.Length)}}- $st.Flush();$c.Close()}catch{}}-} -ArgumentList $dir,8146-Add-Type @"-using System;using System.Runtime.InteropServices;using System.Text;-public class W{[DllImport("user32.dll")]public static extern bool MoveWindow(IntPtr h,int x,int y,int w,int ht,bool r);- [DllImport("user32.dll")]public static extern bool SetForegroundWindow(IntPtr h);- [DllImport("user32.dll")]public static extern IntPtr GetForegroundWindow();- [DllImport("user32.dll")]public static extern int GetWindowText(IntPtr h,StringBuilder s,int n);- [DllImport("user32.dll")]public static extern bool SetWindowPos(IntPtr h,IntPtr after,int x,int y,int cx,int cy,uint flags);- [DllImport("user32.dll")]public static extern bool ShowWindow(IntPtr h,int cmd);- [DllImport("user32.dll")]static extern void keybd_event(byte vk,byte scan,uint flags,UIntPtr extra);- public static void Restore(IntPtr h){ ShowWindow(h,9); }- public static void Topmost(IntPtr h,int x,int y,int cx,int cy){ SetWindowPos(h,(IntPtr)(-1),x,y,cx,cy,0x40); }- // Genuinely ACTIVATE (not just z-order) a real anchor window so each action is measured- // against a true foreground baseline. The synthetic ALT tap clears the foreground lock so- // SetForegroundWindow is honoured even when our thread is not already the foreground one.- public static bool Front(IntPtr h){ keybd_event(0x12,0,0,UIntPtr.Zero); keybd_event(0x12,0,2,UIntPtr.Zero); ShowWindow(h,9); return SetForegroundWindow(h); }- public static string Title(IntPtr h){var sb=new StringBuilder(256);GetWindowText(h,sb,256);return sb.ToString();}}-"@-$drv="C:\Users\cuademo\cua\libs\cua-driver\rust\target\release\cua-driver.exe"; if(-not(Test-Path $drv)){$drv=$drv -replace 'release','debug'}-$wpf=$tk.exe-# WebView2 renderer builds its UIA tree only when an AT requests it. Force it so the-# web DOM (click-target/checkbox/scroll-tall/etc.) surfaces in get_window_state — the-# Windows analog of the Electron recorder's --force-renderer-accessibility. Without this-# get_window_state returns only the chrome frame (TitleBar/Min/Max/Close) and every-# web action resolves to nothing (the prior empty-MP4 / SIZE=0 run).-if($Toolkit -eq 'webview2'){ $env:WEBVIEW2_ADDITIONAL_BROWSER_ARGUMENTS="--force-renderer-accessibility" }-$chrome="C:\Program Files\Google\Chrome\Application\chrome.exe"-$recfwd=$rec -replace '\\','/'-function D($t,$j){ ($j | & $drv call $t 2>&1 | Out-String) }-function Els(){ ((D "get_window_state" ('{{"pid":{0},"window_id":{1},"capture_mode":"ax"}}' -f $script:wp,$script:wd))|ConvertFrom-Json).elements }-# ---------- per-action EFFECT verifier: read the WPF harness's own status labels via UIAutomation -----------Add-Type -AssemblyName UIAutomationClient,UIAutomationTypes-$script:AE=[System.Windows.Automation.AutomationElement]; $script:UTS=[System.Windows.Automation.TreeScope]; $script:UCT=[System.Windows.Automation.ControlType]-function ReadState(){- try{- $win=$null # match by substring so cdp-suffixed web-harness titles ("... [cdp=9222]") resolve- foreach($c in $script:AE::RootElement.FindAll($script:UTS::Children,[System.Windows.Automation.Condition]::TrueCondition)){ if("$($c.Current.Name)" -like "*$($tk.title)*"){ $win=$c; break } }- if(-not $win){return @{}}- $tc=New-Object System.Windows.Automation.PropertyCondition($script:AE::ControlTypeProperty,$script:UCT::Text)- $all=(($win.FindAll($script:UTS::Descendants,$tc)|%{ $_.Current.Name }) -join " || ")- $h=@{}- if($all -match 'agreed=(\w+)'){$h.agreed=$matches[1]}- if($all -match 'slider_value=(\d+)'){$h.slider=[int]$matches[1]}- if($all -match 'last_action=(\w+)'){$h.last_action=$matches[1]}- if($all -match 'mirror=([^|]*)'){$h.mirror=$matches[1].Trim()}- if($all -match 'menu_action=(\w+)'){$h.menu=$matches[1]}- if($all -match 'scroll_offset=(\d+)'){$h.scroll=[int]$matches[1]}- if($all -match 'counter=(\d+)'){$h.counter=[int]$matches[1]}- return $h- }catch{ return @{} }-}-function Verify($t,$b,$a){- switch($t){- 'click' { if("$($a.agreed)" -ne "$($b.agreed)"){'ok'}else{'fail'} }- 'double' { if("$($a.last_action)" -eq 'double_click'){'ok'}else{'fail'} }- 'right' { if("$($a.last_action)" -eq 'right_click'){'ok'} # WinUI3/web: click-target records last_action=right_click- elseif("$($a.menu)" -ne "$($b.menu)" -and "$($a.menu)" -ne 'none' -and "$($a.menu)" -ne ''){'ok'} # WPF: dedicated context-menu sets menu_action=- else{'fail'} }- 'drag' { if([int]$a.slider -gt [int]$b.slider){'ok'}else{'fail'} }- 'scroll' { if([int]$a.scroll -gt [int]$b.scroll){'ok'}else{'fail'} }- 'setval' { if("$($a.mirror)" -match 'set-by-cua'){'ok'}else{'fail'} }- 'type' { if("$($a.mirror)" -match 'typed-by-cua'){'ok'}else{'fail'} }- 'key' { if("$($a.mirror)" -ne "$($b.mirror)"){'ok'}else{'fail'} } # Backspace edits the focused textbox -> mirror changes- default { 'na' }- }-}-# bring an off-screen control into the viewport (UIA ScrollItemPattern) so coordinate actions-# land and the control is actually visible in the recording. $ctype is a UIA ControlType.-function ScrollIntoView($ctype){- try{- $win=$null- foreach($c in $script:AE::RootElement.FindAll($script:UTS::Children,[System.Windows.Automation.Condition]::TrueCondition)){ if("$($c.Current.Name)" -like "*$($tk.title)*"){ $win=$c; break } }- if(-not $win){return}- $cond=New-Object System.Windows.Automation.PropertyCondition($script:AE::ControlTypeProperty,$ctype)- $el=$win.FindFirst($script:UTS::Descendants,$cond)- if($el){ $pat=$null; if($el.TryGetCurrentPattern([System.Windows.Automation.ScrollItemPattern]::Pattern,[ref]$pat)){ $pat.ScrollIntoView() } }- }catch{}-}-# re-resolve a single plan selector against a FRESH ax snapshot (frames change after a scroll).-function Pick($sel){- $EE=Els- switch($sel){- 'chk'{ $EE|?{ "$($_.role)" -match 'Check' -or "$($_.label)" -match 'agree' }|Select -First 1 }- 'btn'{ $EE|?{ "$($_.label)" -match 'lick target' -or "$($_.name)" -match 'lick target' }|Select -First 1 }- 'sld'{ $EE|?{ "$($_.role)" -match 'Slider' }|Select -First 1 }- 'txt'{ $EE|?{ "$($_.role)" -match 'Edit' }|Select -First 1 }- 'ctx'{ $cc=$EE|?{ "$($_.label)" -match 'context menu' }|Select -First 1; if(-not $cc){ $cc=$EE|?{ "$($_.label)" -match 'lick target' -or "$($_.name)" -match 'lick target' }|Select -First 1 }; $cc }- 'scr'{ $ss=$EE|?{ "$($_.role)" -match 'Pane|Group' -and "$($_.label)" -match 'scroll' }|Select -First 1; if(-not $ss){ $ss=$EE|?{ "$($_.role)" -match 'Pane|Group' }|Select -Last 1 }; $ss }- default{ $null }- }-}-# ---------- action plan (filtered per mode) -----------$plan=@(- @{t='click'; sel='chk'; label='left-click a checkbox'}- @{t='double'; sel='btn'; label='double-click a button'}- @{t='right'; sel='ctx'; label='right-click (context menu)'}- @{t='drag'; sel='sld'; label='drag the slider'}- @{t='scroll'; sel='scr'; label='scroll the panel'}- @{t='setval'; sel='txt'; label='set_value on the text box'}- @{t='type'; sel='txt'; label='type into the text box'}- @{t='key'; sel='txt'; label='press a key (Backspace)'}-)-if($vision){ $plan=@($plan | ? { $_.t -ne 'setval' }) } # set_value is AX-only-if($desktop){ $plan=@($plan | ? { $_.t -in @('click','scroll','type','key') }) } # window-less supports click+scroll+global-$script:steals=0;$script:actions=0;$script:cur=""-$script:steps=@(); foreach($p in $plan){ $script:steps+=@{label=$p.label;state='pending';result='';verified=''} }-function Flush(){ $fgt=[W]::Title([W]::GetForegroundWindow());$af=($fgt -like '*CuaTestHarness*')- (@{run=$m.title;expect=$m.expect;fgmode=$m.fg;foreground=$fgt;appFront=$af;steals=$script:steals;actions=$script:actions;steps=$script:steps}|ConvertTo-Json -Depth 6 -Compress)|Set-Content "$dir\status.json" -Encoding UTF8 }-function Pulse($sec){ $e=(Get-Date).AddSeconds($sec); while((Get-Date)-lt $e){ Flush; Start-Sleep -Milliseconds 150 } }-function Ctr($el){ ,@([int]($el.frame.x+$el.frame.w/2),[int]($el.frame.y+$el.frame.h/2)) } # screen center-function Win0($el){ ,@([int]($el.frame.x-$w.bounds.x+$el.frame.w/2),[int]($el.frame.y-$w.bounds.y+$el.frame.h/2)) } # window-local center-'{"run":"","expect":"","fgmode":true,"foreground":"","appFront":false,"steals":0,"actions":0,"steps":[]}' | Set-Content "$dir\status.json" -Encoding UTF8-# ---------- launch -----------# Put ffmpeg on PATH for the daemon BEFORE serve. The video backend's find_ffmpeg()-# checks `ffmpeg` on PATH first, then %LOCALAPPDATA%\Microsoft\WinGet\Packages — but this-# task runs as fbonacci (schtasks /ru fbonacci) while ffmpeg is installed under cuademo's-# WinGet, so the daemon's LOCALAPPDATA probe misses it and video silently degrades to-# present:false (the empty-MP4 / SIZE=0 failures). Find the Gyan.FFmpeg bin under any user-# profile and prepend it so the PATH check succeeds for whichever account serves.-$ffbin=Get-ChildItem "C:\Users\*\AppData\Local\Microsoft\WinGet\Packages\Gyan.FFmpeg*\*\bin\ffmpeg.exe" -EA SilentlyContinue | Select -First 1-if($ffbin){ $env:PATH=$ffbin.DirectoryName+";"+$env:PATH; "FFMPEG on PATH: $($ffbin.FullName)"|Set-Content "$dir\ffmpeg.log" } else { "FFMPEG NOT FOUND"|Set-Content "$dir\ffmpeg.log" }-Start-Process $drv -ArgumentList "serve" -WindowStyle Hidden; Start-Sleep 4-D "set_agent_cursor_enabled" '{"enabled":true,"session":"d1"}'|Out-Null-D "set_agent_cursor_motion" '{"session":"d1","cursor_color":"#FF2D2D","cursor_label":"cua-driver","glide_duration_ms":600,"dwell_after_click_ms":700,"idle_hide_ms":120000}'|Out-Null-$startWait = if($Toolkit -in @('electron','webview2')){18}else{5} # web harnesses are slow to start; WebView2 a11y tree needs extra settle-Start-Process $wpf; Start-Sleep $startWait-Start-Process $chrome -ArgumentList "--app=http://localhost:8146/","--user-data-dir=C:\Users\Public\cdp-$Mode","--no-first-run","--window-position=$PANX,$PANY","--window-size=$PANW,$PANH","--new-window"; Start-Sleep 4-# resolve harness window-$w=$null-for($i=0;$i -lt 20;$i++){ $w=(D "list_windows" "{}"|ConvertFrom-Json).windows | ? { $_.title -like "*$($tk.title)*" } | Select -First 1; if($w){break}; Start-Sleep -Milliseconds 500 }-if(-not $w){ "FATAL: harness window never appeared" | Set-Content "$dir\metric.log"; exit 1 }-$script:wp=$w.pid;$script:wd=$w.window_id;$hHar=[IntPtr][int64]$w.window_id-# layout: harness LEFT (un-maximize first), panel RIGHT + topmost. retry panel handle.-[W]::Restore($hHar)|Out-Null; Start-Sleep -Milliseconds 400; [W]::MoveWindow($hHar,$HARX,$HARY,$HARW,$HARH,$true)|Out-Null; Start-Sleep -Milliseconds 300; [W]::MoveWindow($hHar,$HARX,$HARY,$HARW,$HARH,$true)|Out-Null-$hPanel=[IntPtr]::Zero-for($i=0;$i -lt 16;$i++){ $hPanel=(Get-Process chrome -EA SilentlyContinue|?{$_.MainWindowTitle -like "*cua-driver-panel*"}|Select -First 1).MainWindowHandle; if($hPanel -and $hPanel -ne [IntPtr]::Zero){break}; Start-Sleep -Milliseconds 400 }-if($hPanel -and $hPanel -ne [IntPtr]::Zero){ [W]::MoveWindow($hPanel,$PANX,$PANY,$PANW,$PANH,$true)|Out-Null; [W]::Topmost($hPanel,$PANX,$PANY,$PANW,$PANH) }-# ---------- foreground-baseline ANCHOR (background modes only) -----------# The no-foreground contract is "did this action steal foreground". To measure that we must-# hold a GENUINE foreground baseline before each action: a real, ACTIVATED, non-harness window.-# Re-asserting the dashboard panel with SetWindowPos is z-order only (no activation), so it never-# defines a true foreground holder — after the first inject-based action click-activates the-# target, the harness silently stays frontmost and every later step false-positives as a "steal".-# Anchor on mspaint (Win11 Notepad exposes no MainWindowHandle, so it can never take foreground-# and would fabricate an all-steal). Park it under the topmost panel rect so it is invisible in-# the recording but still a valid activatable foreground window off to the side of the harness.-$hAnchor=[IntPtr]::Zero;$anchorPid=0-if(-not $m.fg){- Start-Process mspaint | Out-Null- for($i=0;$i -lt 20;$i++){ $ap=(Get-Process -EA SilentlyContinue|?{ $_.MainWindowTitle -like '*Paint*' -and $_.MainWindowHandle -ne [IntPtr]::Zero }|Select -First 1); if($ap){ $hAnchor=$ap.MainWindowHandle;$anchorPid=$ap.Id; break }; Start-Sleep -Milliseconds 500 }- if($hAnchor -ne [IntPtr]::Zero){ [W]::Restore($hAnchor)|Out-Null; [W]::MoveWindow($hAnchor,$PANX,$PANY,$PANW,$PANH,$true)|Out-Null; [W]::Topmost($hPanel,$PANX,$PANY,$PANW,$PANH); Start-Sleep -Milliseconds 300 }-}-"HANDLES hHar=$hHar hPanel=$hPanel hAnchor=$hAnchor anchorPid=$anchorPid screen=1024x768"|Set-Content "$dir\handles.log"-Start-Sleep 1-# re-read window bounds after the move (window-local coords need post-move origin)-$w2=(D "list_windows" "{}"|ConvertFrom-Json).windows | ? { $_.window_id -eq $script:wd } | Select -First 1; if($w2){ $w=$w2 }-# resolve control targets (post-move snapshot) with settle-retry.-# Resolver is shared across WPF / WinUI3 / WebView2: roles differ per toolkit-# click-target : label "Click target (left / right / double)" (button on WPF/WinUI3, span on web)-# scroll-tall : role Pane (WinUI3 ScrollViewer) | Group (WebView2 div) , label "scroll-tall"-# checkbox : role Check* | label "I agree"-# context-menu : WPF has a dedicated control (label "context menu"); WinUI3/web have none,-# so right-click targets the click-target and records last_action=right_click.-$E=$null;$resolve=$null-$web=($Toolkit -in @('webview2','electron'))-for($i=0;$i -lt 16;$i++){- $E=Els- $resolve=@{- chk=($E|?{ "$($_.role)" -match 'Check' -or "$($_.label)" -match 'agree' }|Select -First 1)- btn=($E|?{ "$($_.label)" -match 'lick target' -or "$($_.name)" -match 'lick target' }|Select -First 1)- sld=($E|?{ "$($_.role)" -match 'Slider' }|Select -First 1)- scr=($E|?{ "$($_.role)" -match 'Pane|Group' -and "$($_.label)" -match 'scroll' }|Select -First 1)- txt=($E|?{ "$($_.role)" -match 'Edit' }|Select -First 1)- }- if(-not $resolve.scr){ $resolve.scr=($E|?{ "$($_.role)" -match 'Pane|Group' }|Select -Last 1) }- $resolve.ctx=($E|?{ "$($_.label)" -match 'context menu' }|Select -First 1) # WPF dedicated control- if(-not $resolve.ctx){ $resolve.ctx=$resolve.btn } # WinUI3/web: right-click the click-target- # WinUI3 realizes the checkbox once the window is sized (break on chk). On the WebView2 web- # surface the click-target IS in the viewport but the checkbox sits below the fold, and the- # web content cannot be scrolled by the driver in ax mode (verified: AX-scroll on the- # Document/scroll-tall Group, coordinate WM_MOUSEWHEEL, and keyboard PageDown are all no-ops- # — the WebView2 host HWND does not route scroll to the Chromium renderer). So we can't bring- # the checkbox into the tree; break as soon as the click-target resolves and leave chk unset- # (its click step then honestly reports a no-op).- if($resolve.chk -or ($web -and $resolve.btn)){break}- Start-Sleep -Milliseconds 700-}-"RESOLVE "+(($resolve.GetEnumerator()|%{"$($_.Key)=$([bool]$_.Value)"}) -join ' ')+" count=$(@($E).Count)"|Set-Content "$dir\resolve.log"-("picked: "+(($resolve.GetEnumerator()|%{"$($_.Key)=[idx $($_.Value.element_index) role '$($_.Value.role)' lbl '$($_.Value.label)']"}) -join ' '))|Add-Content "$dir\resolve.log"-"--- elements ---"|Add-Content "$dir\resolve.log"-$E|%{ "[$($_.element_index)] role='$($_.role)' name='$($_.name)' label='$($_.label)' frame=$($_.frame.x),$($_.frame.y),$($_.frame.w),$($_.frame.h)" }|Add-Content "$dir\resolve.log"-# seed the agent-cursor overlay BEFORE recording (off the controls)-D "move_cursor" ('{{"x":{0},"y":{1},"session":"d1"}}' -f ($HARW-30),($HARH-30))|Out-Null; Start-Sleep -Milliseconds 400-if($desktop){ D "set_config" '{"key":"capture_scope","value":"desktop"}'|Out-Null }-D "start_recording" ('{{"output_dir":"{0}","record_video":true}}' -f $recfwd)|Out-Null-Pulse 2-# one action, dispatched per mode (mirrors v1's proven Click: glide cursor, then act)-function DoAct($t,$el){- if((-not $el) -and $t -notin @('type','key')){return}- if($el){ $c=Ctr $el; D "move_cursor" ('{{"x":{0},"y":{1},"session":"d1"}}' -f $c[0],$c[1])|Out-Null; Start-Sleep -Milliseconds 550 }- $wl= if($el){Win0 $el}else{$null}- switch($t){- 'click' { if($desktop){D "click" ('{{"x":{0},"y":{1},"session":"d1"}}' -f $c[0],$c[1])|Out-Null}- elseif($vision){$disp=if($m.fg){'foreground'}else{'background'};D "click" ('{{"pid":{0},"window_id":{1},"x":{2},"y":{3},"delivery_mode":"{4}","session":"d1"}}' -f $wp,$wd,$wl[0],$wl[1],$disp)|Out-Null}- else{D "click" ('{{"pid":{0},"window_id":{1},"element_index":{2},"session":"d1"}}' -f $wp,$wd,$el.element_index)|Out-Null} }- 'double' { if($vision){D "double_click" ('{{"pid":{0},"window_id":{1},"x":{2},"y":{3},"session":"d1"}}' -f $wp,$wd,$wl[0],$wl[1])|Out-Null}- else{D "double_click" ('{{"pid":{0},"window_id":{1},"element_index":{2},"session":"d1"}}' -f $wp,$wd,$el.element_index)|Out-Null} }- 'right' { if($vision){D "right_click" ('{{"pid":{0},"window_id":{1},"x":{2},"y":{3},"session":"d1"}}' -f $wp,$wd,$wl[0],$wl[1])|Out-Null}- else{D "right_click" ('{{"pid":{0},"window_id":{1},"element_index":{2},"session":"d1"}}' -f $wp,$wd,$el.element_index)|Out-Null}- if($Toolkit -eq 'wpf' -and $m.fg){ Start-Sleep -Milliseconds 800 # let the context menu render (visible in the recording), then pick the first item so menu_action=ctx_* -> the right-click is scored as landed instead of a false no-op- D "press_key" ('{{"pid":{0},"key":"down","session":"d1"}}' -f $wp)|Out-Null; Start-Sleep -Milliseconds 350- D "press_key" ('{{"pid":{0},"key":"enter","session":"d1"}}' -f $wp)|Out-Null }- else{ Start-Sleep -Milliseconds 500; D "press_key" ('{{"pid":{0},"key":"escape","session":"d1"}}' -f $wp)|Out-Null } }- 'drag' { if(-not $vision -and -not $desktop){- # AX mode: drive the slider through its RangeValue pattern (set_value) — the ax-path- # way to move a slider, reliable and FOREGROUND-INDEPENDENT. A coordinate thumb-drag- # only tracks under forced harness-foreground, which re-activates the window and eats- # the element-path double/right-click. Vision mode below does the real pixel drag.- D "set_value" ('{{"pid":{0},"window_id":{1},"element_index":{2},"value":"48","session":"d1"}}' -f $wp,$wd,$el.element_index)|Out-Null- } else {- $fx=[int]($el.frame.x-$w.bounds.x+8);$fy=[int]($el.frame.y-$w.bounds.y+$el.frame.h/2);$tx=$fx+150- D "drag" ('{{"pid":{0},"from_x":{1},"from_y":{2},"to_x":{3},"to_y":{4},"delivery_mode":"foreground","session":"d1"}}' -f $wp,$fx,$fy,$tx,$fy)|Out-Null- } }- 'scroll' { if($desktop){D "scroll" ('{{"x":{0},"y":{1},"direction":"down","session":"d1"}}' -f $c[0],$c[1])|Out-Null}- elseif($vision){D "scroll" ('{{"pid":{0},"window_id":{1},"x":{2},"y":{3},"direction":"down","session":"d1"}}' -f $wp,$wd,$wl[0],$wl[1])|Out-Null}- else{D "scroll" ('{{"pid":{0},"window_id":{1},"element_index":{2},"direction":"down","session":"d1"}}' -f $wp,$wd,$el.element_index)|Out-Null} }- 'setval' { D "set_value" ('{{"pid":{0},"window_id":{1},"element_index":{2},"value":"set-by-cua","session":"d1"}}' -f $wp,$wd,$el.element_index)|Out-Null }- 'type' { if($el){ D "click" ('{{"pid":{0},"window_id":{1},"element_index":{2},"session":"d1"}}' -f $wp,$wd,$el.element_index)|Out-Null; Start-Sleep -Milliseconds 350 }; D "type_text" ('{{"pid":{0},"text":"typed-by-cua","session":"d1"}}' -f $wp)|Out-Null }- 'key' { if($el){ D "click" ('{{"pid":{0},"window_id":{1},"element_index":{2},"session":"d1"}}' -f $wp,$wd,$el.element_index)|Out-Null; Start-Sleep -Milliseconds 300 }- D "press_key" ('{{"pid":{0},"key":"backspace","session":"d1"}}' -f $wp)|Out-Null }- }-}-# ---------- run -----------try {-for($i=0;$i -lt $plan.Count;$i++){- $p=$plan[$i]; $el=$resolve[$p.sel]- # the 556px reflow pushes the slider / textbox / context-menu button below the fold; bring the- # target into view (so coordinate dispatch lands and the step is visible) then re-read its frame.- if($p.sel -in @('sld','txt','ctx')){- $ct=switch($p.sel){ 'sld'{$script:UCT::Slider} 'txt'{$script:UCT::Edit} 'ctx'{$script:UCT::Button} }- ScrollIntoView $ct; Start-Sleep -Milliseconds 450- $re=Pick $p.sel; if($re){ $el=$re }- }- $script:steps[$i].state='active'; $script:cur=$p.label; Flush- # establish the genuine foreground baseline: ACTIVATE the anchor and confirm it actually took- # foreground BEFORE the action runs. A "steal" is then foreground moving OFF the anchor TO the- # harness; "held" is the anchor staying frontmost. (fg modes keep the harness in front by design.)- if((-not $m.fg) -and $hAnchor -ne [IntPtr]::Zero){- $anchorHeld=$false- for($a=0;$a -lt 8;$a++){ [W]::Front($hAnchor)|Out-Null; Start-Sleep -Milliseconds 180; if([W]::Title([W]::GetForegroundWindow()) -like '*Paint*'){ $anchorHeld=$true; break } }- if($hPanel -and $hPanel -ne [IntPtr]::Zero){ [W]::Topmost($hPanel,$PANX,$PANY,$PANW,$PANH) }- "step $i '$($p.t)': anchor-baseline foreground='$([W]::Title([W]::GetForegroundWindow()))' held=$anchorHeld"|Add-Content "$dir\baseline.log"- }- elseif(($vision -or $desktop) -and $hHar -ne [IntPtr]::Zero){- # VISION/DESKTOP fg modes only: every step dispatches by pixel coordinate, which needs the- # harness to be the real foreground window. Assert it before each step. (Not done in AX-fg:- # ax steps use element_index — foreground-independent — and forcing foreground there re-- # activates the window and eats the element double/right-click. The ax slider 'drag' is- # element-driven via set_value, so no ax step needs a forced foreground.)- $harFront=$false- for($a=0;$a -lt 8;$a++){ [W]::Front($hHar)|Out-Null; Start-Sleep -Milliseconds 150; if([W]::Title([W]::GetForegroundWindow()) -like '*CuaTestHarness*'){ $harFront=$true; break } }- if($hPanel -and $hPanel -ne [IntPtr]::Zero){ [W]::Topmost($hPanel,$PANX,$PANY,$PANW,$PANH) }- "step $i '$($p.t)': fg-mode harness foreground='$([W]::Title([W]::GetForegroundWindow()))' held=$harFront"|Add-Content "$dir\baseline.log"- }- $before=ReadState- DoAct $p.t $el- # measure: sample foreground ~1.5s while the dashboard updates live. steal = foreground moved- # off the anchor onto the harness after the action (with a real activated baseline now held).- $stole=$false; $e=(Get-Date).AddSeconds(1.5); while((Get-Date)-lt $e){ Flush; if([W]::Title([W]::GetForegroundWindow()) -like '*CuaTestHarness*'){$stole=$true}; Start-Sleep -Milliseconds 110 }- $script:steps[$i].verified=(Verify $p.t $before (ReadState))- $script:actions++- if($m.fg){ $script:steps[$i].result='front' } elseif($stole){ $script:steals++;$script:steps[$i].result='stole' } else { $script:steps[$i].result='held' }- $script:steps[$i].state='done'; Flush- if($hPanel -and $hPanel -ne [IntPtr]::Zero){ [W]::Topmost($hPanel,$PANX,$PANY,$PANW,$PANH) }-}-$script:cur="done"; Pulse 2.5-if($desktop){ D "set_config" '{"key":"capture_scope","value":"window"}'|Out-Null }-D "stop_recording" "{}"|Out-Null; Start-Sleep 3-}-finally {- # Dispose the agent-cursor session (remove its overlay cursor) BEFORE cua-driver is- # killed below. Runs even if the action loop threw, so the session is never leaked to- # the idle-TTL reaper. end_session is idempotent.- D "end_session" '{"session":"d1"}' | Out-Null-}-Stop-Job $srv -EA SilentlyContinue; Remove-Job $srv -Force -EA SilentlyContinue-if($anchorPid){ Stop-Process -Id $anchorPid -Force -EA SilentlyContinue }-Get-Process cua-driver,CuaTestHarness.Wpf,chrome,mspaint -EA SilentlyContinue | Stop-Process -Force -EA SilentlyContinue-Get-Process -EA SilentlyContinue | ?{ $_.MainWindowTitle -like '*Paint*' } | Stop-Process -Force -EA SilentlyContinue-$mp4=Get-ChildItem $rec -Recurse -Filter *.mp4 -EA SilentlyContinue | Select -First 1-$verdict= if($m.fg){"foreground-mode, $($script:actions) actions"}else{"$($script:steals)/$($script:actions) actions stole focus"}-$worked=@($script:steps|?{$_.verified -eq 'ok'}).Count; $ver=@($script:steps|?{$_.verified -in @('ok','fail')}).Count-"MODE=$Mode MEASURE=$verdict EFFECTS=$worked/$ver`_landed MP4=$($mp4.FullName) SIZE=$($mp4.Length)" | Tee-Object "$dir\metric.log"diff --git a/libs/cua-driver/tests/fixtures/vision-agent-test/README.md b/libs/cua-driver/tests/fixtures/vision-agent-test/README.mddeleted file mode 100644index b146155e9a..0000000000--- a/libs/cua-driver/tests/fixtures/vision-agent-test/README.md+++ /dev/null@@ -1,52 +0,0 @@-# Vision-agent coordinate-invariant test--Tests cua-driver the way a **vision agent** actually hits it — and without the-overfit the modality recorder has (hand-tuned window-local points run through a-private ratio, which never exercises the driver's image→screen mapping).--## The invariant under test-**The pixel an agent reads off the returned screenshot is the pixel that gets-clicked** — verified by the target's own instrumented state changing.--## The loop (no cheating in locate/click)-1. **capture** — `get_window_state` (window; returns the screenshot alongside the- tree by default) / `get_desktop_state` (desktop, true pixels): the exact image- an agent receives.-2. **locate** — a deterministic pixel **in the returned-image coordinate space**- (`PixelRegistryLocator`: a pre-measured pixel read off the real PNG, with a- dims-guard that fails loud if the pinned geometry drifts). No AX `element_index`,- no hand-converted window-local points. Pluggable `locate(image, target, dims)->(x,y)`.-3. **act** — `click`/`right_click`/`scroll` at that pixel (scope set to match capture).-4. **verify** — the harness oracle (`last_action=`, `clicks=`, …) — reliable pass/fail.- A coordinate mis-map leaves the oracle unchanged → FAIL.--Run: `python3 vision_agent_test.py {wkwebview-click-window|wkwebview-click-desktop|appkit-click-window|safari-learnmore-desktop|all}`--## Two separate axes (don't conflate)-- **Driver coordinate invariant** (this test): deterministic locate → objective- pass/fail → the regression guard. This is what turns the 2× Retina escape into- a permanently-guarded one-liner.-- **Agent locator quality** (future, LLM): same `locate()` signature, send the PNG- to a model, score localization hit-rate separately — never gates the coordinate- regression.-- **Agent self-judged success with no oracle** (future, separate track): re-capture,- ask the model "did it work", score its self-judgment vs the oracle. Kept isolated- so a bad self-judge can't mask a driver regression.--## What the deterministic version caught (that the modality suite couldn't)-- **2× Retina desktop path now correct + guarded** — a read pixel (340,1358) in the- 3024×1964 desktop PNG converts to screen-point (170,679) and lands (oracle + a real- Safari navigation confirm it).-- **Vision pixel-click is a no-op on AppKit `NSButton`** even frontmost — lands- pixel-perfect (crosshair) but `NSButton`'s modal mouseDown loop reads the- window-server queue, not the per-pid `CGEvent.postToPid` queue. The suite drives- this via AXPress, so it never saw a vision agent's pixel click do nothing here.-- **Pixel path requires the target app frontmost** (the AX path doesn't).-- **AppKit harness window AX returns only the menu bar** — its `clicks=` oracle is- unreachable that way; WKWebView exposes it fine.--## Full harness (scope)-Cross-product, each a one-line registry entry:-`{appkit, swiftui, wkwebview, electron, real-app} × {window, desktop(, secondary-display)} × {left, right, double, scroll, drag, type}`.-A color-fiducial locator (harness renders a unique-color dot per control) would make-the registry robust to window moves without OCR.diff --git a/libs/cua-driver/tests/fixtures/vision-agent-test/vision_agent_test.py b/libs/cua-driver/tests/fixtures/vision-agent-test/vision_agent_test.pydeleted file mode 100644index 1f6812e770..0000000000--- a/libs/cua-driver/tests/fixtures/vision-agent-test/vision_agent_test.py+++ /dev/null@@ -1,381 +0,0 @@-#!/usr/bin/env python3-"""-vision_agent_test.py — driver coordinate-invariant test for cua-driver's-vision/pixel action path on macOS.--WHY THIS EXISTS-----------------The modality suite overfits. It drives actions with hand-tuned window-local-POINTS run through a SHOT_W/ratio conversion, and it confirms success by-reading harness-injected AX labels via element_index. A real agent in vision-mode has NEITHER: it receives a screenshot, picks a PIXEL off THAT image, and-calls click(x,y). The AX/element_index path (what the modality suite uses) maps-coordinates through a totally different code path than the pixel path, so the-suite never exercised the driver's image->screen mapping. A real desktop-coordinate bug (clicks off by the 2x Retina factor) hid behind it.--WHAT THIS TESTS (and what it explicitly does NOT)---------------------------------------------------Under test -> the DRIVER COORDINATE INVARIANT: a pixel read off the returned- screenshot is the pixel that gets acted on. Deterministic.- Pass/fail is objective. THIS is the regression guard.-NOT under test here -> agent LOCATOR QUALITY (can a model find the button in- the image?). That is fuzzy, model-dependent, future work. We- keep the locate step behind a pluggable interface so an- LLM/OCR locator can be swapped in, but we do NOT couple the- coordinate-mapping regression to a flaky locator: a flaky- locator would conflate "locator missed" with "driver mis-mapped",- which defeats the test.--THE LOOP (per target)- 1. CAPTURE the exact image an agent receives:- window scope -> get_window_state(capture_mode=vision) (window-local PNG)- desktop scope -> get_desktop_state (full display, TRUE pixels)- 2. LOCATE the target's pixel IN THE RETURNED-IMAGE SPACE via a- DETERMINISTIC locator (a pinned, pre-measured pixel — see- PixelRegistryLocator). NOT a window-local POINT + ratio- conversion (that is the overfit we are replacing); the pixel- lives in the same coordinate space the agent reads off the PNG.- 3. ACT click(x,y) that pixel. Window scope passes window_id; desktop- scope omits it. (Pixel path needs the app frontmost — see- activate_pid — because CGEvent.postToPid mouse events are only- consumed by a control when its app/window is key. The AX path- works backgrounded; the pixel path does not.)- 4. VERIFY did the action LAND ON THE TARGET? Re-read the target control's- OWN state (AX ground truth for the harness; visible page change- for a real app) and assert it moved. A coordinate mis-map (e.g.- the 2x desktop bug) lands the click on empty space / the wrong- control, the target's state does NOT change, and the test FAILS.- We verify by RESULT, never by "we clicked where we intended".--The deterministic pixel is the FLOOR of agent capability, not a stand-in for an-agent: if the driver cannot honour a pixel a human/agent can plainly read off-the image, no locator on earth fixes it.-"""--import argparse-import json-import subprocess-import sys-import time--# ---------------------------------------------------------------------------# driver plumbing-# ---------------------------------------------------------------------------def driver(tool: str, payload: dict) -> dict:- """Invoke `cua-driver call <tool> <json>`; return parsed structuredContent."""- proc = subprocess.run(- ["cua-driver", "call", tool, json.dumps(payload)],- capture_output=True, text=True,- )- if proc.returncode != 0:- raise RuntimeError(f"{tool} failed: {proc.stderr.strip() or proc.stdout.strip()}")- txt = proc.stdout.strip()- try:- doc = json.loads(txt)- return doc.get("structuredContent", doc)- except json.JSONDecodeError:- return {"_raw": txt}---def resolve_window(app_substr: str, title_substr: str = "") -> tuple[int, int]:- """Find (pid, window_id) of the largest matching window. pids/window_ids- change every launch, so the test resolves them at runtime — the only thing- we hardcode is the image-space pixel (the deterministic locator)."""- wins = driver("list_windows", {}).get("windows", [])- cands = [- w for w in wins- if app_substr.lower() in w.get("app_name", "").lower()- and title_substr.lower() in (w.get("title") or "").lower()- and w["bounds"]["width"] > 200 and w["bounds"]["height"] > 200- ]- if not cands:- raise RuntimeError(f"no window matching app~{app_substr!r} title~{title_substr!r}")- best = max(cands, key=lambda w: w["bounds"]["width"] * w["bounds"]["height"])- return best["pid"], best["window_id"]---def activate_pid(pid: int):- """Bring the target app frontmost. The driver intentionally refuses to do- this (no-foreground contract). But the PIXEL path needs the target key:- CGEvent.postToPid mouse events are only consumed by a control when its app- is frontmost (the AX/element_index path has no such requirement). So the- TEST harness — not the driver — does the activation. This is a real- constraint a vision agent must satisfy, surfaced here on purpose."""- subprocess.run(- ["osascript", "-e",- f'tell application "System Events" to set frontmost of '- f'(first process whose unix id is {pid}) to true'],- capture_output=True,- )- time.sleep(1.2)---# ---------------------------------------------------------------------------# Step 1: CAPTURE-# ---------------------------------------------------------------------------def set_scope(scope: str):- """The driver gates desktop-scope clicks (no pid/window_id, screen-absolute- pixels) behind capture_scope=desktop, and window-scope clicks behind- capture_scope=window. Set it to match the target so the click path matches- the capture path the agent used."""- driver("set_config", {"capture_scope": scope})---def window_origin(pid: int, window_id: int) -> tuple[int, int]:- for w in driver("list_windows", {}).get("windows", []):- if w["pid"] == pid and w["window_id"] == window_id:- return int(w["bounds"]["x"]), int(w["bounds"]["y"])- return (-1, -1)---def capture(scope: str, pid: int, window_id: int, out: str) -> tuple[int, int]:- """Write the agent-visible PNG; return its (width, height) in pixels."""- if scope == "window":- r = driver("get_window_state", {- "pid": pid, "window_id": window_id,- "capture_mode": "vision", "screenshot_out_file": out,- })- return r["screenshot_width"], r["screenshot_height"]- elif scope == "desktop":- r = driver("get_desktop_state", {"screenshot_out_file": out})- return r["screenshot_width"], r["screenshot_height"]- raise ValueError(scope)---# ---------------------------------------------------------------------------# Step 2: LOCATE — pluggable. Contract: (image, expected_dims) -> pixel.-# ---------------------------------------------------------------------------class PixelRegistryLocator:- """DETERMINISTIC locator. Returns a pre-measured pixel in the returned-image- coordinate space. The pixel was read ONCE off a real capture of the actual- get_window_state / get_desktop_state PNG (see scripts/derive_pixel.md notes- in the report) — it is NOT a window-local point fed through a ratio.-- Guard: it asserts the live capture's dimensions match the dimensions the- pixel was measured at. The harness window is pinned (non-resizable, fixed- content size) so window-scope captures are byte-stable; if the dims drift,- the pixel is stale and we FAIL LOUD rather than click blind.-- Drop-in seam for the future fuzzy locator: an LLMLocator/OCRLocator would- implement the same locate(image_path, key, dims) -> (x, y) signature, taking- the PNG + an intent string instead of a registry key. The loop below is- identical. That locator's *quality* is a separate axis from this coordinate- invariant and must be scored separately (see report)."""- name = "deterministic-pixel-registry"-- def locate(self, image_path: str, target, live_dims: tuple[int, int]):- exp = target["image_dims"]- if tuple(live_dims) != tuple(exp):- raise RuntimeError(- f"image dims {live_dims} != measured {exp} for {target['key']!r}: "- f"pinned geometry drifted, registry pixel is stale")- return target["pixel"]---# ---------------------------------------------------------------------------# Step 3: ACT-# ---------------------------------------------------------------------------def act(action: str, scope: str, pid: int, window_id: int, x: int, y: int, crosshair=None):- tool = {"left": "click", "right": "right_click", "double": "double_click"}[action]- payload = {"x": x, "y": y}- if action == "left":- payload["button"] = "left"- if scope == "window":- payload["pid"] = pid- payload["window_id"] = window_id- if crosshair and tool == "click":- payload["debug_image_out"] = crosshair # crosshair only on pixel `click` + window_id- # desktop scope: no pid/window_id — driver hit-tests the screen point itself- return driver(tool, payload)---# ---------------------------------------------------------------------------# Step 4: VERIFY — pluggable oracle. Reads the TARGET's own resulting state.-# ---------------------------------------------------------------------------def ax_label(pid: int, window_id: int, prefix: str):- """Reliable ground-truth read of a harness mirror label via AX (e.g.- 'clicks=' -> 'clicks=12'). Used to answer 'did the click LAND', not to- drive the click. A coordinate miss leaves this unchanged -> FAIL."""- r = driver("get_window_state", {- "pid": pid, "window_id": window_id, "capture_mode": "ax", "query": prefix,- })- md = r.get("tree_markdown", "")- vals = []- for line in md.splitlines():- if prefix in line and '"' in line:- seg = line.split('"')[1]- if seg.startswith(prefix):- vals.append(seg)- # dedupe, keep order- return list(dict.fromkeys(vals))---# ---------------------------------------------------------------------------# Target registry — the demonstrated cases. Pixels measured off real captures.-# pid/window_id are resolved at runtime; ONLY the image-space pixel is fixed.-# ---------------------------------------------------------------------------TARGETS = {- # WKWebView harness click-target, WINDOW scope. Web <span> consumes- # synthetic clicks when frontmost. Pixel measured on the 1288x1568- # window-local vision PNG.- "wkwebview-click-window": {- "key": "wkwebview-click-window",- "scope": "window",- "app": "WKWebView", "title": "CuaTestHarness",- "image_dims": (1288, 1568),- "pixel": (305, 1186),- "action": "left",- "verify_prefix": "clicks=", # AX ground truth must increment- },- # AppKit harness click-target, WINDOW scope. NSButton. Coordinate maps- # pixel-perfect (crosshair) but the control does NOT consume synthetic- # CGEvent clicks -> verify is EXPECTED to show no change. Documents the- # pixel-vs-AX gap the modality suite (AXPress only) masks.- "appkit-click-window": {- "key": "appkit-click-window",- "scope": "window",- "app": "CuaTestHarness.AppKit", "title": "CuaTestHarness AppKit",- "image_dims": (987, 1568),- "pixel": (181, 431),- "action": "left",- "verify_prefix": "clicks=",- "expect_no_change": True, # known NSButton synthetic-click swallow- },- # WKWebView harness click-target, DESKTOP scope — THE 2x RETINA PATH, the- # one the modality suite never exercised and where the real bug hid. Pixel- # measured on the 3024x1964 TRUE-pixel desktop PNG. Verified by the harness- # oracle (clicks=/last_action via AX) — reliable. Requires the window pinned- # to a known origin (the desktop pixel is position-dependent); the test- # asserts the origin before clicking and FAILS LOUD if it drifted.- "wkwebview-click-desktop": {- "key": "wkwebview-click-desktop",- "scope": "desktop",- "app": "WKWebView", "title": "CuaTestHarness",- "image_dims": (3024, 1964),- "pixel": (340, 1358),- "require_window_origin": (0, 33),- "action": "left",- "verify_prefix": "clicks=",- },- # Safari "Learn more" on example.com, DESKTOP scope — a GENUINELY REAL app- # (no harness instrumentation). Pixel measured on the 3024x1964 desktop PNG.- # Verified by the visible navigation example.com -> iana.org (no oracle).- # Position-dependent: Safari must show example.com maximized-ish; re-measure- # the pixel if Safari's window moved. Kept as a real-app cross-check.- "safari-learnmore-desktop": {- "key": "safari-learnmore-desktop",- "scope": "desktop",- "app": "Safari", "title": "",- "image_dims": (3024, 1964),- "pixel": (797, 592),- "action": "left",- "verify_prefix": None, # real app: verify visibly (see report)- },-}---def run(target_key: str, scratch: str):- t = TARGETS[target_key]- scope = t["scope"]- loc = PixelRegistryLocator()- print(f"=== {target_key} scope={scope} locator={loc.name} ===")-- pid, window_id = resolve_window(t["app"], t["title"])- print(f"[resolve] pid={pid} window_id={window_id}")-- # pixel path requires the target frontmost- activate_pid(pid)-- # desktop-scope pixels are position-dependent: assert the pinned origin so a- # moved window FAILS LOUD instead of clicking a stale pixel.- if t.get("require_window_origin"):- org = window_origin(pid, window_id)- if tuple(org) != tuple(t["require_window_origin"]):- raise RuntimeError(- f"window origin {org} != required {t['require_window_origin']}: "- f"pin the window there (the desktop pixel was measured at that origin)")- print(f"[resolve] window origin {org} OK")-- # match the click path to the capture path- set_scope(scope)-- # 1. CAPTURE (before)- before_png = f"{scratch}/{target_key}_before.png"- dims = capture(scope, pid, window_id, before_png)- print(f"[capture] {before_png} dims={dims}")-- # ground-truth BEFORE (window-scope harness only)- before_state = None- if t.get("verify_prefix"):- before_state = ax_label(pid, window_id, t["verify_prefix"])- print(f"[verify ] before: {before_state}")-- # 2. LOCATE — deterministic pixel in returned-image space- x, y = loc.locate(before_png, t, dims)- print(f"[locate ] pixel ({x},{y}) [measured @ {t['image_dims']}]")-- # 3. ACT — click that exact pixel- crosshair = f"{scratch}/{target_key}_crosshair.png"- res = act(t["action"], scope, pid, window_id, x, y, crosshair)- msg = res.get("_raw") or json.dumps(res)- print(f"[act ] {t['action']} @ ({x},{y}): {msg[:120]}")- if scope == "window":- print(f"[act ] crosshair: {crosshair}")- time.sleep(0.8)-- # 4. VERIFY — did it LAND? (re-read target's own state)- landed = None- if t.get("verify_prefix"):- after_state = ax_label(pid, window_id, t["verify_prefix"])- print(f"[verify ] after : {after_state}")- if not before_state and not after_state:- # AX exposes no such label for this toolkit (AppKit mirror labels- # are not in the AX tree). The AX oracle is UNAVAILABLE — do NOT- # report a fake PASS from empty==empty. Fall back to the visible- # screenshot: the crosshair proves the pixel landed on the control;- # read the rendered label region to confirm the count.- print(f"[verify ] AX oracle UNAVAILABLE for this toolkit "- f"(no '{t['verify_prefix']}' AXStaticText). "- f"Confirm visibly: crosshair {scratch}/{target_key}_crosshair.png "- f"lands on the control; rendered label in "- f"{scratch}/{target_key}_before.png unchanged after the click.")- landed = None if t.get("expect_no_change") else None- else:- changed = after_state != before_state- if t.get("expect_no_change"):- landed = not changed- note = "(EXPECTED no change — known NSButton synthetic-click swallow)"- else:- landed = changed- note = ""- print(f"[verify ] state changed: {changed} -> coordinate-invariant "- f"{'PASS' if landed else 'FAIL'} {note}")- else:- after_png = f"{scratch}/{target_key}_after.png"- capture(scope, pid, window_id, after_png)- print(f"[verify ] desktop re-capture: {after_png}")- print(f"[verify ] real-app target: confirm visible change in {after_png} "- f"(e.g. address bar / page navigated). Driver echoes the screen-point "- f"it converted to in [act]; a 2x mis-map lands off-target.")- landed = None # human/visible confirm for real-app desktop demo-- return 0 if landed in (True, None) else 2---def main():- p = argparse.ArgumentParser(description=__doc__)- p.add_argument("target", choices=list(TARGETS) + ["all"])- p.add_argument("--scratch", default="/tmp/vision-agent-test")- a = p.parse_args()- subprocess.run(["mkdir", "-p", a.scratch])- keys = list(TARGETS) if a.target == "all" else [a.target]- rc = 0- for k in keys:- rc |= run(k, a.scratch)- print()- sys.exit(rc)---if __name__ == "__main__":- main()diff --git a/scripts/ci/README.md b/scripts/ci/README.mdindex 2e2bfa2a2a..8e9adb9f74 100644--- a/scripts/ci/README.md+++ b/scripts/ci/README.md@@ -1,15 +1,40 @@# Cua-driver CI runnersThese scripts are thin entrypoints around the Rust integration tests. They-build the repo-local fixture applications on demand, set the testkit path-overrides, and collect logs. They do not push code or alter branches.+build the repo-local fixture applications, run one strict Rust environment+preflight, set testkit paths, execute Rust targets, invoke the Rust report+validator, and collect artifacts. They do not define behavioral rows, push+code, or alter branches.++For the test layout and the distinction between unit tests, shared harnesses,+and native harnesses, see+`libs/cua-driver/docs/test-harnesses-guide.md`.| Runner | Session | Canonical command || --- | --- | --- |-| `linux/run-rust-e2e.sh` | Linux X11/Wayland desktop | `--suite shared` |-| `windows/run-rust-e2e.ps1` | Windows console/RDP user session | `-Suite shared -RequireGui` |+| `linux/run-rust-e2e.sh` | Existing Linux X11 or Wayland desktop | no selector |+| `linux/run-rust-e2e-wayland.sh` | Headless native Sway session | no selector |+| `linux/run-rust-e2e-inject.sh` | Nested `cua-compositor` session | no selector |+| `linux/run-rust-e2e-desktop.sh` | Existing representative Linux desktop | no selector |+| `windows/run-rust-e2e.ps1` | Windows console/RDP user session | `-RequireGui` |+| `macos/run-rust-e2e.sh` | Logged-in macOS session with TCC | no selector |++Use the command without a selector for the canonical complete run. CI sets the+private `CUA_E2E_INTERNAL_LANE` partition to `shared`, `native`, or `capture`+when it fans the same matrix into independent jobs. Those values are not public+alternate suites.++Run the Wayland wrapper through `nix develop .#cua-driver-wayland-e2e`. It+creates a pure Wayland session with Xwayland disabled and delegates every+scenario to `run-rust-e2e.sh`.++Run the nested compositor wrapper through+`nix develop .#cua-driver-inject-e2e`. This environment is experimental and+proves only the private compositor-owned route. Use `run-rust-e2e-desktop.sh`+for maintainer checks on representative GNOME, KDE, or real-Xorg sessions.-The Windows workflow accepts a runner label so the same command can execute on-the Azure VM's active-RDP runner when that self-hosted label is configured. A-GitHub-hosted Windows runner is useful for smoke validation, but it is not a-substitute for the Azure user-session run.+The GitHub-hosted Windows workflow is canonical when its strict preflight proves+an interactive desktop. The workflow also accepts a runner label so maintainers+can replay the same command on an Azure VM with an active RDP session for+environment parity; that replay is not a separate test definition or source of+behavioral truth.