Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.
TypeScript accessibilityagent-toolsai-agentbrowser-automationcomputer-usecuadesktop-automationelectronpipi-coding-agentpi-extensionpi-packageplaywrighttypescriptui-automationvisionwebview2windowswindows-automationwsl
fix: support embedded browser UIA controls
alexshpunt-clanker[bot]4e3b5ccb0297
diff --git a/docs/bundled-runtime.md b/docs/bundled-runtime.mdindex ab3d586..0ec035a 100644--- a/docs/bundled-runtime.md+++ b/docs/bundled-runtime.md@@ -1,16 +1,16 @@# Bundled CUA runtime-The npm package carries one native runtime: Cua Driver 0.38.0 for Windows x64, built from source `3e69266e` of the [alexshpunt/cua](https://github.com/alexshpunt/cua) fork. [FORK.md](https://github.com/alexshpunt/cua/blob/pi-computer-use/FORK.md) lists the fork's changes. The `windows-desktop` profile needs no separate driver configuration:+This source selects one native runtime: Cua Driver 0.38.0 for Windows x64, built from source `db235b24` of the [alexshpunt/cua](https://github.com/alexshpunt/cua) fork. A package built from this source carries that runtime. [FORK.md](https://github.com/alexshpunt/cua/blob/pi-computer-use/FORK.md) lists the fork's changes. The `windows-desktop` profile needs no separate driver configuration:```sh-pi install npm:pi-agent-computer-use+pi install git:github.com/alexshpunt/pi-agent-computer-use```An extension update selects its matching runtime. Computer use prepares that runtime automatically, including on configured remote machines; it does not select a moving upstream version.## Where the files come from-The files are published as the fork release [`pi-runtime-0.38.0-3e69266e`](https://github.com/alexshpunt/cua/releases/tag/pi-runtime-0.38.0-3e69266e): `cua-driver.exe`, `cua-driver-uia.exe`, `cua-cursor-theme.exe`, `LICENSE.md` and `THIRD_PARTY_NOTICES.md`. They were cross-built with Rust 1.97.1, cargo-xwin 0.23.1 and LLVM 18, with a static CRT and canonical Windows manifest resources. They are not copies of a GitHub Actions artifact.+The files are published as the fork release [`pi-runtime-0.38.0-db235b24`](https://github.com/alexshpunt/cua/releases/tag/pi-runtime-0.38.0-db235b24): `cua-driver.exe`, `cua-driver-uia.exe`, `cua-cursor-theme.exe`, `LICENSE.md` and `THIRD_PARTY_NOTICES.md`. They were built on GitHub's Windows runner with Rust 1.97.1 and a static CRT.`src/runtime-pins.json` pins the source, the version, the release URL and the SHA-256 of every file. Before packing, `npm pack` runs `scripts/fetch-runtime.mjs`, which downloads the files into `native-runtime/win32-x64/runtime/` and checks each hash. The directory is not committed to Git.diff --git a/docs/semantic-actions.md b/docs/semantic-actions.mdindex e3a5839..e012817 100644--- a/docs/semantic-actions.md+++ b/docs/semantic-actions.md@@ -16,4 +16,6 @@ The native tool admits one exact snapshot target and one UIA pattern. It has no**Focus and desktops.** A target on the current virtual desktop gets the WS_EX_NOACTIVATE guard for the call, so a provider that activates its window on Invoke (Chromium did) cannot take the foreground. A target on another desktop gets only the provider EnableWindow shield, because toggling that style there moved the window to the current desktop. In `test/live/semantic-focus.live.ts` (fork e6d26cb6, a disposable Windows Sandbox) a native Win32 target's invoke, select and set_value all landed while a 20 ms foreground sampler never saw the target; the previous runtime let the same target hold the foreground for about 0.8 s. Neither target left its desktop.-Limits from the same checks: many apps expose only their frame through UIA while on another desktop (Notepad, WinForms), so their content cannot be acted on there; Edge reports its page elements under another process, which the driver refuses before any call (`semantic_element_process_changed`). Browser pages belong to Playwright anyway. A passing app is not a guarantee for every UIA provider; read back the application state.+An element can belong to a different process than its window, as with embedded WebView2 pages. The driver accepts it only when its provider process still matches the snapshot and its current UIA ancestry reaches the exact target window. If either check cannot establish the target, the action refuses without sending input. Use Playwright for browser pages that can be attached; embedded pages without a debugging endpoint can use UIA where their providers support the requested pattern.++Many apps expose only their frame through UIA while on another desktop (Notepad, WinForms), so their content cannot be acted on there. A passing app is not a guarantee for every UIA provider; read back the application state.diff --git a/src/runtime-pins.json b/src/runtime-pins.jsonindex eafc584..6036e83 100644--- a/src/runtime-pins.json+++ b/src/runtime-pins.json@@ -1,13 +1,13 @@{"format": 2,- "source": "3e69266e1dd832dd57f0ddda25091a01537d84e6",+ "source": "db235b242bd42dc63b8e350fb6b4dcfe097e5170","version": "0.38.0",- "url": "https://github.com/alexshpunt/cua/releases/download/pi-runtime-0.38.0-3e69266e/",+ "url": "https://github.com/alexshpunt/cua/releases/download/pi-runtime-0.38.0-db235b24/","files": {- "cua-driver.exe": "d210ca0ce302c8acd6c024b57966e4369291d15ae5d83a3ab2fcdd7f9f1c36cc",- "cua-driver-uia.exe": "cd119e33f6726d718e900841127bf47cf2f5c6558e14cfbc48f7347f2b254f2e",- "cua-cursor-theme.exe": "19ebe9b201c70c0c921ee727f86bac612fbcbac04a4c1ac6928b93fdb872e0ec",- "LICENSE.md": "c0779290c1d4783169aa3dbfb55feb505e563ef8a004bbf55298ceffcfbda8d9",- "THIRD_PARTY_NOTICES.md": "1fb1119fed1ad8e3397558aeda43b3e323fd80a1cbbe58934546dc6cee9e8bf7"+ "cua-driver.exe": "7f59724096e6936d9c73a88e672a7c1fc89269bcb524b35319585fb416288650",+ "cua-driver-uia.exe": "e439f7dd08db7009badd22d6468f75fb3f959c0eec7373fb504cbffe22953962",+ "cua-cursor-theme.exe": "bfdf7c47de7f40d8fbc6c083b2a9bb6e8d1ff7c6bad5dde7a006c152640304b7",+ "LICENSE.md": "d0a32419a44fa38d5023d1431dfcbedd1457eb5e7b2e6c7c87ca064facb41b41",+ "THIRD_PARTY_NOTICES.md": "5ff37d532597071048d02015deeec5d4ee6c598fed0e296b036eb771e44dd3d1"}}