Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.
TypeScript accessibilityagent-toolsai-agentbrowser-automationcomputer-usecuadesktop-automationelectronpipi-coding-agentpi-extensionpi-packageplaywrighttypescriptui-automationvisionwebview2windowswindows-automationwsl
Move human material out of docs:computer (LPT-995)
alexshpunt-clanker[bot]c3bafd7fad23
diff --git a/README.md b/README.mdindex a5546c8..c8062e6 100644--- a/README.md+++ b/README.md@@ -180,7 +180,7 @@ I use this every day on real tasks, but I can only test the machines, apps and w| Document | Contents || --------------------------------------------------------- | --------------------------------------------------------------------- |-| [Computer resources](docs/computer-resources.md) | Addresses, views, commands, browsers, sandboxes, sessions, machines |+| [Computer resources](docs/computer-resources.md) | The agent guide `docs:computer`: rules the tool fields do not give || [Operating skill](.pi/skills/computer-use/SKILL.md) | How the agent observes, plans, acts and verifies || [Complete input](docs/complete-input.md) | Keys, pointer, drags, sequences and their limits || [Semantic actions](docs/semantic-actions.md) | UI Automation invoke, select and set value |diff --git a/docs/complete-input.md b/docs/complete-input.mdindex eb3c89c..c2635f7 100644--- a/docs/complete-input.md+++ b/docs/complete-input.md@@ -23,7 +23,7 @@ Physical letter keys follow the active OS keyboard layout. Use `type_text` for lA relative drag endpoint is computed from the observed start: `to_x=from_x+dx`, `to_y=from_y+dy`. No global cursor or desktop offset is needed.-The typed tools choose delivery from the target (see [Act on windows](computer-resources.md#act-on-windows)): a click or scroll on an element goes through UI Automation with `uia_only`, and the driver refuses before sending anything when no pattern carries it; the tool then sends the same action as real input. Points, drags, keys and timelines are real input (`delivery_mode: "foreground"`). No route posts window messages.+The typed tools choose delivery from the target (see [Input](computer-resources.md#input)): a click or scroll on an element goes through UI Automation with `uia_only`, and the driver refuses before sending anything when no pattern carries it; the tool then sends the same action as real input. Points, drags, keys and timelines are real input (`delivery_mode: "foreground"`). No route posts window messages.## Bounded sequences@@ -72,4 +72,21 @@ Refusal, unknown/partial outcome, cancellation, access revocation or target lifeThe native driver owns each gesture's normal key/button release. No separate key-down/up, cross-call button hold, application hover or global release-all is exposed. Held paths use the bundled fork's EOF-aware receiver; adapter cancellation closes its owned transport and attempts native release. A crash or forced process kill cannot guarantee release. Never interpret a timeout as a completed stroke.+## What an action changed++Every action reports what it changed, the way Playwright MCP's incremental snapshots do: `~` changed, `+` added and `-` removed elements, plus URL, title or tab changes, such as `~ 3:a Button "Count 0" → "Count 1"`. Changed and added rows carry the new refs. A locator action compares with the walk that resolved its target; a page action compares Playwright's snapshots before and after.++Every action also compares pixels: on a native window the window's overview pixels, in a viewport only the viewport's pixels, on a browser page Playwright screenshots of the page in CSS pixels. Image 1 is a small copy of the window or page with each changed area outlined and numbered as listed. Before and after crops of the first three areas follow it, with small areas enlarged up to 4x:++```text+Pixels changed in 2 areas (overview pixels; 0.07% of the window); image 1 outlines and numbers them on the window:+ 1. x=465 y=645 w=24 h=24 (at the input point); images 2 and 3 show x=453 y=633 w=48 h=48 before and after, enlarged 4x+ 2. x=120 y=40 w=300 h=60; images 4 and 5 show x=108 y=28 w=324 h=84 before and after+Browser chrome outside the page also changed (x=0 y=0 w=1280 h=80, 2 areas): tab strip, address bar or loading state; not listed.+```++The area at or nearest to the input point comes first, otherwise the largest. In a browser window whose page elements reach UI Automation (Edge does; Chrome often does not), changes above the page, such as a loading tab or the address bar, are summarised in one line. When most of the window or page changed, or a page action navigated to another URL, image 1 is the whole window or page after the action instead of an overview and crops.++This shows a stroke on a canvas or a moved sprite when the UI tree says nothing. Changes under 12 pixels and caret-shaped spots do not count. When the first capture shows no change, it looks again for up to 300 ms, then says `Nothing changed in the UI or the window's pixels within N ms`; check the target before repeating the action. When something changed, it captures until the window settles, for up to 1 second.+See the [operating skill](../.pi/skills/computer-use/SKILL.md).diff --git a/docs/native-observability.md b/docs/native-observability.mdindex 87815b3..df450d9 100644--- a/docs/native-observability.md+++ b/docs/native-observability.md@@ -42,6 +42,12 @@ The adapter admits one native request at a time and retains one replaceable pend`highlight.status: queued` records adapter admission, not proof that a person saw pixels. An unavailable renderer does not repeat capture or input. Failed or cancelled startup releases the connection while keeping presentation preferences subscribed for a later attempt. Final session shutdown disposes those subscriptions. Backend refusal and capture exclusion still need qualification on the exact Windows runtime; the adapter alone cannot prove focus, transparency, screenshot exclusion or visual quality.+## Browser screenshots without a flash++A visible Chromium window shrinks for a moment while Chrome takes a screenshot with a clip or scale of its own, so the human would see a flash on every element crop and every page action's pixel comparison. The browser worker therefore takes one plain viewport screenshot, which does not move the window, and the host derives the image from it: an element crop is cut from it at the element's pixels, and the page image used by the change list is scaled to one pixel per CSS pixel (exactly `innerWidth` x `innerHeight`). The image the agent gets and its coordinates stay the same; an element crop is the same size as Chrome's own, and its pixels match except for slight resampling at fractional device pixels.++The worker falls back to Chrome's own screenshot, which can flash, when the derived image would not be exact: an element that is not wholly inside the viewport (for example taller than the window), a pinch-zoomed page, or a page that scrolls or resizes during the capture. In testing Firefox showed no flash with this path.+## Runtime qualificationUse the exact source and file hashes in [the runtime pins](bundled-runtime.md). Before shipping a native visual runtime, check a real Windows desktop for focus and hit testing, capture exclusion, source changes, multi-monitor geometry, concurrent sessions, reduced motion, cancellation and cleanup. Measure input latency independently of animation and bound idle/active resource use. Keep private desktop recordings in ignored local artifacts.diff --git a/docs/setup.md b/docs/setup.mdindex b70b7d0..758227b 100644--- a/docs/setup.md+++ b/docs/setup.md@@ -47,7 +47,26 @@ No `.pi/computer-use.json` is needed. An ignored local file in the project can o}```-`CUA_DRIVER_PATH` does the same as `command`. `runtime` names a qualified profile from `src/driver.ts`; normally the executable must match its version and SHA-256. For an experimental driver, set `"driverOverride": true` beside `command`: this trusts those executable bytes and checks the driver identity, protocol and advertised tools instead. Only use an executable you trust. Optional `args` replace `['mcp', '--direct', '--embedded']`, `interop` selects a live WSL interop socket, and `machines` names other machines (see [Work on another machine](computer-resources.md#work-on-another-machine)). These are trusted user settings, never agent tool arguments. Close the computer runtime and reload after changing them.+`CUA_DRIVER_PATH` does the same as `command`. `runtime` names a qualified profile from `src/driver.ts`; normally the executable must match its version and SHA-256. For an experimental driver, set `"driverOverride": true` beside `command`: this trusts those executable bytes and checks the driver identity, protocol and advertised tools instead. Only use an executable you trust. Optional `args` replace `['mcp', '--direct', '--embedded']`, `interop` selects a live WSL interop socket, and `machines` names other machines (see [Other machines](#other-machines)). These are trusted user settings, never agent tool arguments. Close the computer runtime and reload after changing them.++## Other machines++Another Windows machine with WSL can be operated over SSH: its windows, displays, browsers and sandboxes. Only machines named under `machines` in `.pi/computer-use.json` can be reached, by name, never by host or path. Its addresses work like local ones and never act on this machine; a browser `url` such as `127.0.0.1` means that machine. The remote user's Windows session must be active; a disconnected one returns white screenshots. Delete the machine address to stop its driver; its next use starts it again.++To set one up:++1. Make it reachable through an SSH alias that logs in to its WSL by key, with a known host key.+2. Add the alias to `.pi/computer-use.json`:++```json+"machines": {+ "lab": { "ssh": "lab-wsl" }+}+```++On first use, computer use prepares the package-compatible driver on the remote machine. It prepares Windows Node and the browser worker when a browser is opened, and uses new verified caches after package updates. Old caches are not replaced while they may be in use. `/computer-setup machine <alias> [name]` can prepare these ahead of time.++Trusted configuration can still specify remote WSL `command`, `runtime`, `node` or `package` overrides. Set `driverOverride: true` to try an experimental driver without the packaged hash check, and `browserOverride: true` to keep explicit browser paths even when they resemble old generated paths. Older generated paths otherwise migrate to automatic defaults; unrelated custom paths remain overrides.## WSL interop