Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.
TypeScript accessibilityagent-toolsai-agentbrowser-automationcomputer-usecuadesktop-automationelectronpipi-coding-agentpi-extensionpi-packageplaywrighttypescriptui-automationvisionwebview2windowswindows-automationwsl
Document automatic computer runtime preparation
alexshpunt-clanker[bot]187da1ef2529
diff --git a/.pi/skills/computer-use/SKILL.md b/.pi/skills/computer-use/SKILL.mdindex 13b6444..4792aab 100644--- a/.pi/skills/computer-use/SKILL.md+++ b/.pi/skills/computer-use/SKILL.md@@ -47,9 +47,7 @@ In Codemode, every `computer_*` tool returns a receipt object; branch on `outcom## Missing parts-Some parts are optional installs: Windows Node for browsers (`browser`), Playwright's Firefox (`firefox`), Cua perception for `regions` (`perception`, about 420 MB, AGPL-3.0 icon detector), MXC for isolated sessions (`mxc`), the VS Code bridge (`vscode-bridge`) and preparing another machine (`machine <ssh alias> [name]`). A refusal that names `/computer-setup` means one is missing.--Never install on your own. Tell the user what is missing, what it adds, its size and where it goes, and ask them to run `/computer-setup <part> [machine]`. Ask before you start when the task clearly needs such a part, for example another machine or VS Code's browser. Retry the refused step once afterwards.+The driver, Windows Node and remote browser worker are prepared automatically. Optional parts such as Firefox, perception, MXC and the VS Code bridge still require the user to run `/computer-setup`; ask before using them.## Access and cleanupdiff --git a/README.md b/README.mdindex 307d589..a5546c8 100644--- a/README.md+++ b/README.md@@ -148,18 +148,16 @@ Pi packages run with your full system permissions, and this one drives your desk### Optional parts-Window control works right after the install. A few parts are extra downloads, and you install them with `/computer-setup`:+Window control works right after the install. The driver, Windows Node and remote browser worker are prepared automatically when needed. Optional parts are extra downloads installed with `/computer-setup`:| Part | What it's for || --------------------- | ---------------------------------------------------------------------------------------------- |-| `browser` | Node.js for Windows, which Playwright browsers need. Skip it if Node is already on Windows. || `firefox` | Playwright's own Firefox build. Chrome and Edge work without it. || `perception` | The `regions` view. About 420 MB, and its icon detector is AGPL-3.0. || `mxc` | Isolated agent sessions. Needs Windows 11 24H2 or newer. || `vscode-bridge` | VS Code's integrated browser. |-| `machine <ssh alias>` | Another Windows machine with WSL, reached over SSH. |-Add a machine name to install a part there, for example `/computer-setup perception lab`. Everything goes into your Windows user's own folders, and every download is checked against a pinned SHA-256. If a task needs a part you don't have, the agent tells you what's missing and asks before it installs anything.+Add a machine name to install a part there, for example `/computer-setup perception lab`. Everything goes into your Windows user's own folders, and every download is checked against a pinned SHA-256. If a task needs an optional part you do not have, the agent asks before installing it. A configured remote machine is prepared automatically on first use and when the package changes.### Unsigned binariesdiff --git a/docs/bundled-runtime.md b/docs/bundled-runtime.mdindex 6ca5bd0..c1f0294 100644--- a/docs/bundled-runtime.md+++ b/docs/bundled-runtime.md@@ -6,7 +6,7 @@ The npm package carries one native runtime: Cua Driver 0.38.0 for Windows x64, bpi install npm:pi-agent-computer-use```-An update installs a new extension version with its exact runtime. Computer use never updates the driver on its own.+An extension update selects its matching runtime. Computer use prepares that runtime automatically, including on configured remote machines; it does not select a moving upstream version.## Where the files come from@@ -50,4 +50,4 @@ Computer use owns one MCP child. It disables telemetry and update checks, requir## Overrides-A trusted `.pi/computer-use.json` (`command`, `runtime`) or `CUA_DRIVER_PATH` overrides the packaged runtime, for example to try another qualified fork build. Each named profile keeps its exact byte and version check. Agent tool arguments can never select an executable or a profile.+A trusted `.pi/computer-use.json` (`command`, `runtime`) or `CUA_DRIVER_PATH` overrides the packaged runtime. Named profiles normally keep their exact byte and version check. To test a different build, add `"driverOverride": true` beside `command`; this trusts the chosen executable instead of checking its packaged hash, while still checking its identity, protocol and advertised tools. Agent tool arguments can never select an executable or a profile.diff --git a/docs/computer-resources.md b/docs/computer-resources.mdindex bc8cead..e82e307 100644--- a/docs/computer-resources.md+++ b/docs/computer-resources.md@@ -189,7 +189,7 @@ Ways of working found in real sessions, per app.## Browser pages-Web pages run through [Playwright](https://playwright.dev/). The browser window itself, its toolbar, system dialogs and every other application stay native windows. Playwright runs in a small worker on Windows, so browsers need Node.js there: an installed one, or `/computer-setup browser`. When Pi runs on Windows, the worker uses Pi's own Node.+Web pages run through [Playwright](https://playwright.dev/). The browser window itself, its toolbar, system dialogs and every other application stay native windows. Playwright runs in a small worker on Windows. Computer use uses a compatible Windows Node when available and otherwise prepares one automatically. When Pi runs on Windows, the worker uses Pi's own Node.`computer_open` with `browser` or `url` launches a visible browser with a disposable profile: Chrome by default, installed Edge, or Playwright's own Firefox (`/computer-setup firefox` once per machine and after updating this package; it cannot be attached to). `profile: "agent"` uses the agent's separate persistent profile on that machine, not the user's default profile; logins and site data in it remain across sessions. Only one browser at a time can use each agent profile. Closing the computer session closes these browsers and deletes disposable profiles, not agent profiles.@@ -239,22 +239,17 @@ Another Windows machine with WSL can be operated over SSH: its windows, displaysTo set one up:1. Make it reachable through an SSH alias that logs in to its WSL by key, with a known host key.-2. Run `/computer-setup machine <alias> [name]`. It copies the CUA runtime, a portable Windows Node and the browser worker into the remote user's folders and prints the `machines` entry. Run it again after updating this package.-3. Add the entry to `.pi/computer-use.json`:+2. Add the alias to `.pi/computer-use.json`:```json"machines": {- "lab": {- "ssh": "lab-wsl",- "runtime": "windows-desktop",- "command": "/mnt/c/Users/user/AppData/Local/Programs/PiComputerUse/cua-driver-0.37.0-33c4b82b/runtime/cua-driver.exe",- "node": "/mnt/c/Users/user/AppData/Local/Programs/PiComputerUse/node-v22.22.2-win-x64/node.exe",- "package": "/home/user/.local/share/pi-computer-use/package"- }+ "lab": { "ssh": "lab-wsl" }}```-All paths are on the remote WSL. Without `node` and `package`, the machine works without a browser.+On first use, computer use prepares the package-compatible driver on the remote machine. It prepares Windows Node and the browser worker when a browser is opened, and uses new verified caches after package updates. Old caches are not replaced while they may be in use. `/computer-setup machine <alias> [name]` can prepare these ahead of time.++Trusted configuration can still specify remote WSL `command`, `runtime`, `node` or `package` overrides. Set `driverOverride: true` to try an experimental driver without the packaged hash check, and `browserOverride: true` to keep explicit browser paths even when they resemble old generated paths. Older generated paths otherwise migrate to automatic defaults; unrelated custom paths remain overrides.## Files, clipboard and recordingdiff --git a/docs/setup.md b/docs/setup.mdindex 096eaff..b70b7d0 100644--- a/docs/setup.md+++ b/docs/setup.md@@ -12,7 +12,9 @@ pi install npm:pi-agent-computer-useThe package carries the Windows runtime, so no driver download or executable config is needed. From WSL it controls the Windows host, not a Linux desktop. See [the bundled runtime](bundled-runtime.md) for platforms and checks.-Optional parts install with `/computer-setup PART [MACHINE]`: `browser`, `firefox`, `perception`, `mxc`, `vscode-bridge` and `machine`. It works with Pi on Windows and in WSL: each part runs as a PowerShell script on the target Windows, locally or over SSH. Everything goes into the Windows user's own folders, and downloads are checked against pinned SHA-256 sums on that Windows before use. An agent installs a part with Insert `{"action":"setup","part":...}` into a machine address, and only after the user explicitly allows that part.+The driver, Windows Node and remote browser worker are prepared automatically when needed. A configured remote machine also receives the package-compatible driver and worker automatically on first use and after an update. Downloads are checked before use.++Optional parts such as Firefox, perception, MXC and the VS Code bridge install with `/computer-setup PART [MACHINE]`. The user runs the command or explicitly approves that part before the agent installs it. `/computer-setup browser` and `/computer-setup machine <alias> [name]` remain available for manual preparation.## Choose a mode@@ -45,7 +47,7 @@ No `.pi/computer-use.json` is needed. An ignored local file in the project can o}```-`CUA_DRIVER_PATH` does the same as `command`. `runtime` names a qualified profile from `src/driver.ts`; each profile checks the exact version and SHA-256 of its executable and refuses different bytes before startup. Optional `args` replace `['mcp', '--direct', '--embedded']`, `interop` selects a live WSL interop socket, and `machines` names other machines (see [Work on another machine](computer-resources.md#work-on-another-machine)). These are trusted user settings, never agent tool arguments. Computer use never scans for other installs or updates the driver at startup; close the computer runtime and reload after changing them.+`CUA_DRIVER_PATH` does the same as `command`. `runtime` names a qualified profile from `src/driver.ts`; normally the executable must match its version and SHA-256. For an experimental driver, set `"driverOverride": true` beside `command`: this trusts those executable bytes and checks the driver identity, protocol and advertised tools instead. Only use an executable you trust. Optional `args` replace `['mcp', '--direct', '--embedded']`, `interop` selects a live WSL interop socket, and `machines` names other machines (see [Work on another machine](computer-resources.md#work-on-another-machine)). These are trusted user settings, never agent tool arguments. Close the computer runtime and reload after changing them.## WSL interop