Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.
TypeScript accessibilityagent-toolsai-agentbrowser-automationcomputer-usecuadesktop-automationelectronpipi-coding-agentpi-extensionpi-packageplaywrighttypescriptui-automationvisionwebview2windowswindows-automationwsl
Docs: a viewport watch gets only the viewport and overlaps slow captures; runtime 2437a85e (LPT-950)
alexshpunt-clanker[bot]2acc59b9cdee
diff --git a/docs/bundled-runtime.md b/docs/bundled-runtime.mdindex c843f34..0380e35 100644--- a/docs/bundled-runtime.md+++ b/docs/bundled-runtime.md@@ -1,6 +1,6 @@# Bundled CUA runtime-The npm package carries one native runtime: Cua Driver 0.37.0 for Windows x64, built from the `pi-computer-use` branch of the [alexshpunt/cua](https://github.com/alexshpunt/cua) fork at source `d9f380f6`. [FORK.md](https://github.com/alexshpunt/cua/blob/pi-computer-use/FORK.md) lists what the fork adds to upstream Cua Driver. It is the qualified `windows-desktop` profile, so a fresh installation gets every native capability without any configuration:+The npm package carries one native runtime: Cua Driver 0.37.0 for Windows x64, built from the `pi-computer-use` branch of the [alexshpunt/cua](https://github.com/alexshpunt/cua) fork at source `2437a85e`. [FORK.md](https://github.com/alexshpunt/cua/blob/pi-computer-use/FORK.md) lists what the fork adds to upstream Cua Driver. It is the qualified `windows-desktop` profile, so a fresh installation gets every native capability without any configuration:```shpi install npm:pi-agent-computer-use@@ -10,7 +10,7 @@ An update installs a new extension version with its exact runtime. Computer use## Where the files come from-The files are published as the fork release [`pi-runtime-0.37.0-d9f380f6`](https://github.com/alexshpunt/cua/releases/tag/pi-runtime-0.37.0-d9f380f6): `cua-driver.exe`, `cua-driver-uia.exe`, `cua-cursor-theme.exe`, `LICENSE.md` and `THIRD_PARTY_NOTICES.md`. They are byte-identical to the fork's CI build of that commit.+The files are published as the fork release [`pi-runtime-0.37.0-2437a85e`](https://github.com/alexshpunt/cua/releases/tag/pi-runtime-0.37.0-2437a85e): `cua-driver.exe`, `cua-driver-uia.exe`, `cua-cursor-theme.exe`, `LICENSE.md` and `THIRD_PARTY_NOTICES.md`. They are byte-identical to the fork's CI build of that commit.`src/runtime-pins.json` pins the source, the version, the release URL and the SHA-256 of every file. Before packing, `npm pack` runs `scripts/fetch-runtime.mjs`, which downloads the files into `native-runtime/win32-x64/runtime/` and checks each hash. The directory is not committed to Git.diff --git a/docs/computer-resources.md b/docs/computer-resources.mdindex 2e08787..600977a 100644--- a/docs/computer-resources.md+++ b/docs/computer-resources.md@@ -214,7 +214,7 @@ To set one up:"lab": {"ssh": "lab-wsl","runtime": "windows-desktop",- "command": "/mnt/c/Users/user/AppData/Local/Programs/PiComputerUse/cua-driver-0.37.0-d9f380f6/runtime/cua-driver.exe",+ "command": "/mnt/c/Users/user/AppData/Local/Programs/PiComputerUse/cua-driver-0.37.0-2437a85e/runtime/cua-driver.exe","node": "/mnt/c/Users/user/AppData/Local/Programs/PiComputerUse/node-v22.22.2-win-x64/node.exe","package": "/home/user/.local/share/pi-computer-use/package"}diff --git a/docs/temporal-capture.md b/docs/temporal-capture.mdindex 42bbe36..174ff56 100644--- a/docs/temporal-capture.md+++ b/docs/temporal-capture.md@@ -68,6 +68,8 @@ Add `as` to a JSON selection to get one image instead of one per frame:| `timeout_ms` | 1..300000, default 60000; the watch stops by itself then || `maxLongEdge` | 64..4096, default 800; a viewport is captured so its own rectangle gets at most this many pixels on its long edge, never more than its full resolution |+A viewport watch asks the driver for the viewport's rectangle only: the driver copies and encodes just that part instead of the whole window. When a frame takes longer than the interval, a second capture starts on time while the first finishes, at most two at once; frames keep the order their captures started in. The rate reached is in the sequence metadata, while the watch runs and in the `watch_stop` result.+A watch only looks: it does not take the window's observation, so refs and points of the latest Read stay valid for input. A frame with the same bytes as the one before shares its image and costs no memory; 3000 frames or 48 MiB of distinct frames stop the watch by itself. It checks process and window id on every frame, and stops when the window closes or access ends.Each input or window action on the watched window notes its span, from the call to its result, in every running watch of that window; the receipt lists it with the frames captured during it. Read the sequence address with a `frames:` view while the watch runs, for example `frames:last`.