Skip to content

Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.

Bring the docs in line with the shipped runtime (LPT-867)

alexshpunt-clanker[bot]18ac21bc1b1d
diff --git a/.pi/skills/computer-use/SKILL.md b/.pi/skills/computer-use/SKILL.md
index 1223a98..ee9dc1d 100644
--- a/.pi/skills/computer-use/SKILL.md
+++ b/.pi/skills/computer-use/SKILL.md
@@ -33,7 +33,7 @@ Write sends literal application text, including spaces, quotes, newlines and JSO
Insert sends one JSON action to that observation address. Omit `window_ref` and `observation_id`; the address binds them. Choose the address hint or a JSON target, not both. Do not send text anchors, ranges or insertion flags. Keyboard, pointer, drag, hold, semantic and sequence actions keep their existing native limits and qualified-runtime gates.
-If foreground input refuses because the human is using the computer, wait the stated time or use background delivery; never race the human. Start with background. After its observed refusal, Read a fresh overview before explicitly requesting foreground. Foreground can activate the window and disturb the real pointer. Native move/hover, held paths/keys and strict semantic input are available only on their qualified selected runtimes; never replace a refusal with global input, another backend or a browser script.
+If foreground input refuses because the human is using the computer, wait the stated time or use background delivery; never race the human. Start with background. After its observed refusal, Read a fresh overview before explicitly requesting foreground. Foreground can activate the window and disturb the real pointer. Native move/hover, held paths/keys and strict semantic input come with the bundled runtime; an older override refuses them. Never replace a refusal with global input, another backend or a browser script.
For canvases and other pixel work, keep using the same observation after pixel-only input: it stays valid until the window is resized, and a refusal says why. Put long known programs (hundreds of strokes and clicks) into one `sequence`. Use one `timeline` when input needs exact timing, a button or key held across moves, wheel during a drag, raw relative motion for pointer-locked apps, or a pen with pressure and tilt. Batch only known keyboard/stroke steps. Stop at navigation, a new dialog, changed layout or an effect requiring a new decision. A bounded sequence consumes one overview, validates the whole list and runs serially without intermediate captures. Every drag releases its button; no keys/buttons remain held across calls.
diff --git a/README.md b/README.md
index a5f53d4..cc58af8 100644
--- a/README.md
+++ b/README.md
@@ -112,13 +112,13 @@ Each session has its own folder on the controlled machine for files an app opens
## Platforms
-Computer use is tested on Windows 11 x64, controlled from Pi running in WSL on the same machine or on another one over SSH. Linux and macOS aren't supported: window control there isn't verified, so no runtime ships for them and computer use refuses with a clear message. Some features need more: Windows Sandbox needs a Windows edition that has it, and isolated sessions need Windows 11 24H2 or newer.
+Computer use works on Windows 11 x64, with Pi running on Windows itself or in WSL, on the same machine or on another one over SSH. Linux and macOS aren't supported: window control there isn't verified, so no runtime ships for them and computer use refuses with a clear message. Some features need more: Windows Sandbox needs a Windows edition that has it, and isolated sessions need Windows 11 24H2 or newer.
This is experimental and changes often, so interfaces and behavior may still move around.
## Installation
-Install [Pi](https://pi.dev/) in WSL first, then install the package:
+Install [Pi](https://pi.dev/) on Windows or in WSL, then install the package:
```bash
pi install npm:pi-agent-computer-use
@@ -157,7 +157,7 @@ Window control works right after the install. A few parts are extra downloads, a
| `vscode-bridge` | VS Code's integrated browser. |
| `machine <ssh alias>` | Another Windows machine with WSL, reached over SSH. |
-Add a machine name to install a part there, for example `/computer-setup perception lab`. Everything goes into your Windows user's own folders, and every download is checked against a pinned SHA-256. If a task needs a part you don't have, the agent tells you what's missing and asks before it installs anything.
+Add a machine name to install a part there, for example `/computer-setup perception lab`. `/computer-setup` is a Bash script, so run it from Pi in WSL. Everything goes into your Windows user's own folders, and every download is checked against a pinned SHA-256. If a task needs a part you don't have, the agent tells you what's missing and asks before it installs anything.
### Unsigned binaries
@@ -190,6 +190,7 @@ I use this every day on real tasks, but I can only test the machines, apps and w
| [Image guides](docs/image-guides.md) | Coordinate grid and last-action marks |
| [Vision debug viewer](docs/vision-debug.md) | What the agent saw and the capture controls |
| [Bundled runtime](docs/bundled-runtime.md) | Native runtime packages and platform boundaries |
+| [Setup and modes](docs/setup.md) | Install, Safe and YOLO, runtime overrides and window lifetime |
## License
diff --git a/docs/bundled-runtime.md b/docs/bundled-runtime.md
index 9a33753..2212925 100644
--- a/docs/bundled-runtime.md
+++ b/docs/bundled-runtime.md
@@ -20,7 +20,7 @@ An installation from Git has no packaged files. On first use, computer use downl
| Pi runs on | Runtime |
| --- | --- |
-| Windows x64 | Runs the packaged files directly. Not live-verified yet. |
+| Windows x64 | Runs the packaged files directly. |
| WSL x64 on Windows | Copies the packaged files to a Windows-host cache and controls the Windows desktop. |
| Linux outside WSL, macOS, ARM64 | No runtime is shipped; computer use refuses with `unsupported_native_target`. |
diff --git a/docs/complete-input.md b/docs/complete-input.md
index 205f383..8badc3a 100644
--- a/docs/complete-input.md
+++ b/docs/complete-input.md
@@ -10,12 +10,12 @@ For an explicit per-action short cursor glide, use `animation:"reactive"`. Ordin
- `key`: complete press/release of a letter, digit, return/tab/escape/space/backspace/delete, arrow, home/end/pageup/pagedown or f1..f12. Optional `modifiers` are ctrl/shift/alt/win.
- `hotkey`: `keys:["ctrl","a"]`, unique modifiers followed by one key.
-- `type_text`: literal text up to2000 characters; `delay_ms` is CUA's per-character delay, 0..200, default30.
-- `invoke`, `select`, `set_value`: [strict UIA intents](semantic-actions.md) on a current hint/token, with a separately qualified runtime. No physical or foreground fallback; `set_value` takes replacement `text`.
+- `type_text`: literal text up to 2000 characters; `delay_ms` is CUA's per-character delay, 0..200, default 30.
+- `invoke`, `select`, `set_value`: [strict UIA intents](semantic-actions.md) on a current hint/token. No physical or foreground fallback; `set_value` takes replacement `text`.
- `select_all`: complete Ctrl+A.
- `click`, `right_click`, `double_click`: observed token or window-image x/y. Optional modifiers; click also accepts a left/right/middle button.
-- `scroll`: observed element token, direction up/down/left/right, amount1..50, by line/page. Pixel scroll is refused before input: CUA0.33.2 does not correctly map points from resized screenshots.
-- `drag`: from_x/from_y/to_x/to_y in the window PNG. Optional duration_ms0..10000 (default500), steps1..200 (default20), left/right/middle button and modifiers.
+- `scroll`: observed element token, direction up/down/left/right, amount 1..50, by line/page; or a window-image x/y with signed `delta_x`/`delta_y` in wheel detents (`unit:"wheel_detents"`).
+- `drag`: from_x/from_y/to_x/to_y in the window PNG. Optional duration_ms 0..10000 (default 500), steps 1..200 (default 20), left/right/middle button and modifiers.
Physical letter keys follow the active OS keyboard layout. Use `type_text` for literal text; the adapter does not change the user's layout.
@@ -25,14 +25,14 @@ Background delivery is the default. Explicit foreground is allowed only after an
## Bounded sequences
-One overview may admit one sequence of known-target `key`, `hotkey`, `type_text`, `select_all` or `drag` steps. The entire list is validated before any input. Every drag is a complete stroke with release at the end. Without `via`, it follows the existing straight route. There are no nested sequences, pointer clicks or scroll steps inside this batch.
+One overview may admit one sequence of known-target `key`, `hotkey`, `type_text`, `select_all`, `drag`, `click`, `double_click` or `right_click` steps. The entire list is validated before any input. Every drag is a complete stroke with release at the end. Without `via`, it follows the existing straight route. There are no nested sequences or scroll steps inside this batch.
Bounds:
-- 1..32 steps;
-- wait_ms0..5000 before each step, measured after the previous completes;
-- at most30s requested waits, typing delays and drag durations together;
-- no further dispatch after45s elapsed time, including admission work.
+- 1..2000 steps;
+- wait_ms 0..5000 before each step, measured after the previous completes;
+- at most 15 minutes of requested waits, typing delays and drag durations together;
+- no further dispatch after 30 minutes elapsed time, including admission work.
These bounds do not guarantee real-time timing. Native admission, transport, focus handling and OS scheduling add latency. A native call already in progress may finish after cancellation or exceed the remaining elapsed budget.
@@ -40,17 +40,17 @@ For computed drawing, choose points inside an actual observed canvas region. Map
## Continuous brush paths
-With the qualified Windows fork explicitly selected, `drag` accepts `via`: 1..254 intermediate `{x,y}` points. The endpoints and every intermediate point use the same current window PNG. One call presses the left button, follows all points and releases it. Do not pass `steps`, modifiers, or right/middle button with `via`.
+With the bundled runtime, `drag` accepts `via`: 1..254 intermediate `{x,y}` points. The endpoints and every intermediate point use the same current window PNG. One call presses the left button, follows all points and releases it. Do not pass `steps`, modifiers, or right/middle button with `via`.
-Background refuses before path input. Observe again after that refusal, then explicitly request foreground. It moves the real pointer; do not fight it during the stroke. The default installed driver refuses an entire batch containing `via` before its first step.
+Background refuses before path input. Observe again after that refusal, then explicitly request foreground. It moves the real pointer; do not fight it during the stroke.
A partial path returns `unknown`, preserving native `delivery.delivered_count` and the press/release summary. These count point moves, not completed batch steps. Later steps stop. Never replay the whole stroke or its uncertain remainder. Check the app's actual pixels before choosing another action.
-See [trusted runtime selection](standalone-poc.md#qualified-continuous-paths).
+See [runtime overrides](setup.md#override-the-runtime).
## Bounded key holds
-With a qualified Windows key-hold runtime selected, `hold` keeps 1..8 unique keys down for `duration_ms`1..10000. Ordinary keys use the same names as `key`; ctrl/shift/alt/win are also allowed on their own or with ordinary keys. Modifiers go down first. The action releases its keys in reverse order before returning success. It does not expose separate key-down/up tools or carry a hold between calls.
+With the bundled runtime, `hold` keeps 1..8 unique keys down for `duration_ms` 1..10000. Ordinary keys use the same names as `key`; ctrl/shift/alt/win are also allowed on their own or with ordinary keys. Modifiers go down first. The action releases its keys in reverse order before returning success. It does not expose separate key-down/up tools or carry a hold between calls.
Optional `actions` contains 1..32 complete left pixel clicks or drags from the same overview. Each drag may be straight or use `via`. Do not pass element tokens, per-action modifiers, button choices, key steps or nested holds. All requested pointer durations, including 50ms per click, must fit the hold interval. All keys and points are checked before input. Hold cannot nest in `sequence`.
@@ -68,6 +68,6 @@ These are adapter/native receipt times, not invented application-event timestamp
Refusal, unknown/partial outcome, cancellation, access revocation or target lifetime loss stops later steps. Completed steps are not rolled back. Never replay a completed prefix or uncertain in-flight step. Invalid preflight sends nothing and does not consume the overview. Once the first native call is dispatched, the overview is retired for external input, even if the sequence fails.
-The native driver owns each gesture's normal key/button release. No separate key-down/up, cross-call button hold, application hover or global release-all is exposed. Held paths use the qualified fork's EOF-aware receiver; adapter cancellation closes its owned transport and attempts native release. A crash or forced process kill cannot guarantee release. Never interpret a timeout as a completed stroke.
+The native driver owns each gesture's normal key/button release. No separate key-down/up, cross-call button hold, application hover or global release-all is exposed. Held paths use the bundled fork's EOF-aware receiver; adapter cancellation closes its owned transport and attempts native release. A crash or forced process kill cannot guarantee release. Never interpret a timeout as a completed stroke.
See the [operating skill](../.pi/skills/computer-use/SKILL.md).
diff --git a/docs/image-context.md b/docs/image-context.md
index add2392..0b9c5c5 100644
--- a/docs/image-context.md
+++ b/docs/image-context.md
@@ -1,6 +1,6 @@
# Images stay in normal Pi context
-The user superseded LPT-380's last-three request limit. This extension no longer removes or replaces older image blocks, rewrites text dumps, or creates a separate image archive. User attachments, tool results, Codemode images, overview and detail follow the same Pi history lifecycle as other messages.
+This extension never removes or replaces older image blocks, rewrites text dumps, or creates a separate image archive. User attachments, tool results, Codemode images, overview and detail follow the same Pi history lifecycle as other messages.
Pi's normal compaction may summarize older context and keep recent messages. It does not erase the original session entries. Reload alone does not remove images. This extension does not change compaction settings or provider cache controls. Other extensions and provider processing can still affect final delivery.
diff --git a/docs/native-detail.md b/docs/native-detail.md
index 4f21b55..07ce5fe 100644
--- a/docs/native-detail.md
+++ b/docs/native-detail.md
@@ -2,7 +2,7 @@
See [the resource guide](computer-resources.md) for the current addresses, views and commands.
-CUA 0.33.2 is the qualified Windows runtime for window, display and browser tools. It remains unmodified. Native window zoom is reused directly; the driver still has no display-region capture, so our adapter crops a fresh full-resolution display PNG.
+The bundled Cua Driver runtime serves window, display and browser tools. Native window zoom is reused directly; the driver still has no display-region capture, so our adapter crops a fresh full-resolution display PNG.
## Select from an overview
@@ -16,7 +16,7 @@ Window detail calls native `zoom` with the original overview rectangle. Moving,
Display detail maps the overview rectangle to native pixels once, rounding outwards. It requests `get_desktop_state(max_image_dimension:0)`, checks display/raster geometry and extracts only that region. A small Sharp processor optionally downsizes it with Lanczos3. No image is artificially enlarged.
-Display detail defaults to long-edge cap 1200 and scale 1. Bounds are 64..4096 and 0.05..1; effective cap is `floor(maxLongEdge * scale)`. These per-request settings do not change the shared 500px overview settings. Full-resolution intermediate display pixels stay local to that call and are not forwarded to the model or Vision debug.
+Display detail defaults to long-edge cap 1200 and scale 1. Bounds are 64..4096 and 0.05..1; effective cap is `floor(maxLongEdge * scale)`. These per-request settings do not change the shared overview settings (1280 px long edge by default). Full-resolution intermediate display pixels stay local to that call and are not forwarded to the model or Vision debug.
Only geometry metadata is retained between calls. A new overview replaces its source's previous context. A dispatched detail consumes that context, so another detail needs a new overview. Missing/retired IDs, rectangles outside the original overview and nested details refuse. Window geometry changes do not retire an overview. Display detail still checks its distinct display/raster mapping; neither route guesses DPI or captures a wider fallback.
diff --git a/docs/observation-sources.md b/docs/observation-sources.md
index 4b6371a..1004553 100644
--- a/docs/observation-sources.md
+++ b/docs/observation-sources.md
@@ -26,7 +26,7 @@ The source envelope and exact returned bytes appear in existing Vision debug. Th
## Native settings and limits
-Desktop windows and displays use session-local settings: Lanczos3, default long-edge cap 500 and scale 1. Settings are frozen before each capture. Overview pixels stay native. Only fresh display detail uses adapter crop/resize; no originals are retained and CUA may round below the requested edge. A qualified 0.32.0 primary-display capture was 499×333 for a 500 cap, from a native 3840×2560 display with reported scale factor 1.5.
+Desktop windows and displays use session-local settings: Lanczos3, default long-edge cap 1280 and scale 1. Settings are frozen before each capture. Overview pixels stay native. Only fresh display detail uses adapter crop/resize; no originals are retained and CUA may round below the requested edge.
Window and display routes use the selected qualified CUA runtime. Arbitrary window reads do not raise or focus that window. Neither route guarantees fresh paint.
diff --git a/docs/reactive-animation.md b/docs/reactive-animation.md
index 38c5cd9..2634455 100644
--- a/docs/reactive-animation.md
+++ b/docs/reactive-animation.md
@@ -18,7 +18,7 @@ The default is 1ms. The value must be an integer in 1..5000. Zero is rejected be
This controls decorative cursor glide only. Native click press/release and post-click waits stay unchanged (35ms and 50ms on the inspected Windows Chromium path). A 1ms glide does **not** mean a 1ms click. Identity checks, native transport, profile switching, delivery and scheduling still take time.
-The same option works on a configured machine and in a sandbox; their drivers use the configuration of the workspace that connects to them. It matters: by default CUA waits for the visible cursor to arrive at each new point before clicking. On a 4K test machine that made a click on a new spot take 650–930 ms, against about 160 ms with the 1 ms glide (LPT-702).
+The same option works on a configured machine and in a sandbox; their drivers use the configuration of the workspace that connects to them. It matters: by default CUA waits for the visible cursor to arrive at each new point before clicking. On a 4K test machine that made a click on a new spot take 650–930 ms, against about 160 ms with the 1 ms glide.
## Receipts and failures
diff --git a/docs/semantic-actions.md b/docs/semantic-actions.md
index c407cd2..740df82 100644
--- a/docs/semantic-actions.md
+++ b/docs/semantic-actions.md
@@ -2,7 +2,7 @@
See [the resource guide](computer-resources.md) for the current addresses, views and commands.
-An Insert action accepts explicit `invoke`, `select` and `set_value` intents with a current hint or element token. They need the separately qualified Windows semantic runtime. Older runtimes refuse before dispatch. Ordinary `click` keeps its existing native action ladder.
+An Insert action accepts explicit `invoke`, `select` and `set_value` intents with a current hint or element token. The bundled runtime supports them; an older runtime selected by override refuses before dispatch. Ordinary `click` keeps its existing native action ladder.
- `invoke` calls the control's UIA Invoke pattern.
- `select` calls UIA SelectionItem.Select, not Ctrl+A or a pointer click.
diff --git a/docs/setup.md b/docs/setup.md
new file mode 100644
index 0000000..0c1b0a5
--- /dev/null
+++ b/docs/setup.md
@@ -0,0 +1,75 @@
+# Setup, modes and window lifetime
+
+Pi talks to the bundled Windows Cua Driver over owned stdio MCP: directly when Pi runs on Windows, through interop when it runs in WSL. There is no SSH server, permanent service or autostart. This guide covers installation, modes, runtime overrides and window lifetime; [the resource guide](computer-resources.md) covers the tools.
+
+**YOLO is the default.** It reaches new and already open desktop windows within normal user permissions. The user can select Safe, which allows only windows individually opened by this extension session. Neither mode is an OS sandbox. Use only the apps and documents authorized by the actual task.
+
+## Install
+
+```sh
+pi install npm:pi-agent-computer-use
+```
+
+The package carries the Windows runtime, so no driver download or executable config is needed. From WSL it controls the Windows host, not a Linux desktop. See [the bundled runtime](bundled-runtime.md) for platforms and checks.
+
+Optional parts install with `/computer-setup PART [MACHINE]`: `browser`, `firefox`, `perception`, `mxc`, `vscode-bridge` and `machine`. The command runs `scripts/setup.sh`, which is a Bash script for Pi running in WSL. Everything goes into the Windows user's own folders, and downloads are checked against pinned SHA-256 sums. An agent runs it only after the user explicitly allows that part.
+
+## Choose a mode
+
+Use `/computer-mode safe` or `/computer-mode yolo` in Pi. `/computer-mode` shows the current choice. The footer and tool results label the mode. Switching needs no reload or per-window config. This is a user command, not a model tool or tool argument.
+
+The selected mode is saved in the Pi session and restored on reload or branch navigation. New sessions start in YOLO. Window ownership and references are not saved: after reload, session replacement, or closing `computer:session`, Safe cannot adopt surviving windows. Open a new window to establish ownership.
+
+Safe hides outside-session windows from discovery and denies their old references before native UI reads or input. Opening a new window does not grant access to another window in the same process. Recycled native handles do not inherit ownership. Launch reuse or ambiguous candidates grant no ownership. The native backend still enumerates window metadata internally to correlate launches and find owned windows; Safe does not hide that metadata from the extension itself.
+
+Switching cannot undo input already dispatched under YOLO. Pending work checks the current policy before dispatch; if the mode changes during a read or input, foreign returned UI state is withheld. Previously received YOLO observations remain in conversation history. Safe does not constrain shell tools, files, network, clipboard, shared app state, or what an authorized app can do.
+
+## Windows and input
+
+Computer use works through `computer:` addresses: the ordinary Pi Agent IDE Read, Search, Write and Insert tools, or `computer_read`, `computer_search`, `computer_write` and `computer_insert` without IDE. Without Pi Agent IDE, or with an IDE whose plugin API version does not match, those tools are deferred and `tool_search` is turned on.
+
+Opening means a **new window**, not an existing window or new tab. Apps differ: some are single-instance or create several startup windows. If the result is `unknown`, discover what actually appeared before deciding. Never blindly relaunch. A new window is not necessarily an empty document: applications can restore prior state. Read tree and pixels before editing.
+
+Background is the default. Foreground is explicit and only follows an observed background refusal or no-op. `delivered` does not prove the requested task is complete. Check fresh UI value and PNG. `unknown` may include a partial change; never replay without checking.
+
+Clear or close only task-owned or explicitly authorized documents. Closing `computer:session` releases the runtime; it does not close application windows or save documents.
+
+## Override the runtime
+
+No `.pi/computer-use.json` is needed. An ignored local file in the project can override the driver for an unusual setup:
+
+```json
+{
+ "command": "/mnt/c/path/to/cua-driver.exe",
+ "runtime": "windows-desktop"
+}
+```
+
+`CUA_DRIVER_PATH` does the same as `command`. `runtime` names a qualified profile from `src/driver.ts`; each profile checks the exact version and SHA-256 of its executable and refuses different bytes before startup. Optional `args` replace `['mcp', '--direct', '--embedded']`, `interop` selects a live WSL interop socket, and `machines` names other machines (see [Work on another machine](computer-resources.md#work-on-another-machine)). These are trusted user settings, never agent tool arguments. Computer use never scans for other installs or updates the driver at startup; close the computer runtime and reload after changing them.
+
+## WSL interop
+
+Windows processes must run in the interactive desktop session, not service Session 0. A long-lived Linux process may inherit a stale `WSL_INTEROP`. If Windows commands hang, check that its socket exists **and** a simple `cmd.exe /d /c ver` succeeds. A live WSL terminal launched from Windows can provide a current endpoint. Keep its owning session alive; do not persist this machine's ephemeral socket as a default.
+
+One-shot PowerShell commands need closed stdin (`</dev/null`). Owned MCP and the window-identity peer intentionally keep stdin open for their protocol and exit on EOF. The wrapper forwards telemetry-off and update-check-off settings through `WSLENV`; it does not edit saved driver preferences.
+
+## Window lifetime and limits
+
+A session-owned PowerShell peer attaches a private marker to each accessible window. Windows destroys that property with the window; a recycled HWND cannot inherit the old reference. Before observation or input, the extension checks PID and marker. On EOF the peer removes its surviving markers. It does not capture UI, send input, launch apps, or elevate permissions.
+
+An inaccessible or elevated window can appear in discovery metadata without a usable reference. The extension does not bypass Windows permissions. References are not recovered across Pi reloads or session replacement. In YOLO, discover again. In Safe, open a new window; discovery cannot adopt surviving windows.
+
+The marker check and driver dispatch are separate native operations, not an atomic OS transaction. New-window matching excludes prelaunch PID/HWND addresses, even if one is recycled during launch, and compares new candidates with the resulting app's windows, including packaged apps that use an existing process. It cannot establish causality against an unrelated concurrent launch of the same app.
+
+## Development
+
+Load exactly one copy of the extension: point its package entry in your Pi settings at the checkout you are working on, and point it back when you are done. Loading the package and a project wrapper together registers every tool twice. In a fresh checkout run `npm ci --ignore-scripts --legacy-peer-deps`, then `npm run dev:ide -- /absolute/path/to/pi-agent-ide` to link the IDE SDK.
+
+```bash
+npm run typecheck
+npm test
+npm rebuild node-pty
+npm run test:integration
+```
+
+The real-Pi MCP fixtures substitute Windows lifetime identity only; real registration, session logic, MCP, result forwarding and EOF run. Fixtures are not native Windows input evidence.
diff --git a/docs/source-capabilities.md b/docs/source-capabilities.md
index ad7e5e2..fe5daeb 100644
--- a/docs/source-capabilities.md
+++ b/docs/source-capabilities.md
@@ -27,8 +27,8 @@ Pre-dispatch validation failures have no native action capability receipt. Neith
## Native basis and limits
-The original LPT-617 foundation used Windows fork `3b3305044b7b327310615aa99d3307948a63061c`. Its public window state carries PID/HWND, screenshot/window bounds, snapshot tokens and UIA pattern names. It does not expose monitor, Windows session or Task View desktop membership. Its primary desktop capture alias does not name a physical monitor. Internal virtual-screen coordinate checks are not Windows virtual-desktop membership.
+The public window state carries PID/HWND, screenshot/window bounds, snapshot tokens and UIA pattern names. Monitors come from [selected displays](selected-displays.md) and Task View desktops from [shell desktop membership](shell-desktop-membership.md); internal virtual-screen coordinate checks are not Windows virtual-desktop membership.
CUA's validated action result publishes route, effect and actual delivery. Background refusal publishes a code and optional foreground escalation. The adapter projects these fields without predicting capabilities from app names or UIA pattern labels.
-This foundation does not add other-monitor capture, virtual-desktop management or concurrent input arbitration. Those remain separate parts of LPT-615.
+This contract does not cover virtual-desktop management or concurrent input arbitration.
diff --git a/docs/standalone-poc.md b/docs/standalone-poc.md
deleted file mode 100644
index ebd5ccc..0000000
--- a/docs/standalone-poc.md
+++ /dev/null
@@ -1,124 +0,0 @@
-# Standalone Windows computer use
-
-Pi runs in WSL and talks to Windows Cua Driver over owned stdio MCP. There is no SSH server, permanent service, or autostart. This guide covers setup, runtime selection and window lifetime; [the resource guide](computer-resources.md) covers the tools.
-
-**YOLO is the default.** It reaches new and already open desktop windows within normal user permissions. The user can select Safe, which allows only windows individually opened by this extension session. Neither mode is an OS sandbox. Use only the apps and documents authorized by the actual task.
-
-## Choose a mode
-
-Use `/computer-mode safe` or `/computer-mode yolo` in Pi. `/computer-mode` shows the current choice. The footer and tool results label the mode. Switching needs no reload or per-window config. This is a user command, not a model tool or tool argument.
-
-The selected mode is saved in the Pi session and restored on reload or branch navigation. New sessions start in YOLO. Window ownership and references are not saved: after reload, session replacement, or closing `computer:session`, Safe cannot adopt surviving windows. Open a new window to establish ownership.
-
-Safe hides outside-session windows from discovery and denies their old references before native UI reads or input. Opening a new window does not grant access to another window in the same process. Recycled native handles do not inherit ownership. Launch reuse or ambiguous candidates grant no ownership. The native backend still enumerates window metadata internally to correlate launches and find owned windows; Safe does not hide that metadata from the extension itself.
-
-Switching cannot undo input already dispatched under YOLO. Pending work checks the current policy before dispatch; if the mode changes during a read or input, foreign returned UI state is withheld. Previously received YOLO observations remain in conversation history. Safe does not constrain shell tools, files, network, clipboard, shared app state, or what an authorized app can do.
-
-## Tools
-
-Computer use works through `computer:` addresses: the ordinary Pi Agent IDE Read, Search, Write and Insert tools, or `computer_read`, `computer_search`, `computer_write` and `computer_insert` without IDE. See [the resource guide](computer-resources.md) for addresses, views and commands.
-
-Opening means a **new window**, not an existing window or new tab. Apps differ: some are single-instance or create several startup windows. If the result is `unknown`, discover what actually appeared before deciding. Never blindly relaunch. A new window is not necessarily an empty document: applications can restore prior state. Read tree and pixels before editing.
-
-Background is the default. Foreground is explicit and only follows an observed background refusal/no-op. `delivered` does not prove the requested task is complete. Check fresh UI value and PNG. `unknown` may include a partial change; never replay without checking.
-
-Clear or close only task-owned or explicitly authorized documents. Closing `computer:session` releases the runtime; it does not close application windows or save documents.
-
-## Install the extension
-
-```sh
-pi install npm:pi-agent-computer-use
-```
-
-The package carries the Windows runtime. No manual CUA download or executable config is needed. WSL controls the Windows host, not a Linux desktop. See [the bundled runtime, system requirements and platform boundary](bundled-runtime.md).
-
-The following separate-driver setup is retained only for trusted legacy overrides. It is not the default installation.
-
-## Optional legacy Driver setup
-
-The pin is **Cua Driver 0.33.2**, official release `cua-driver-rs-v0.33.2`. Window, display and browser routes share this qualified pin. GitHub marks the monorepo release as prerelease; the archive uses a plain stable version. Later binaries need fresh qualification, not automatic startup upgrades.
-
-For Windows x86_64 use the official [binary archive](https://github.com/trycua/cua/releases/download/cua-driver-rs-v0.33.2/cua-driver-rs-0.33.2-windows-x86_64-binary.zip). Published SHA-256:
-
-```text
-82fd647e6f4b69ed811e52e9586a4e56291e477ad699fb02441193a45364337c
-```
-
-Check the hash before extraction. Extract the **whole archive** into a new user-local directory, for example `%LOCALAPPDATA%\Programs\PiComputerUsePoC\cua-driver-0.33.2`. Do not overwrite another install. Keep supplied native libraries and UIA helper beside the executable.
-
-Check signatures before running executables:
-
-```powershell
-Get-ChildItem "$env:LOCALAPPDATA\Programs\PiComputerUsePoC\cua-driver-0.33.2\*.exe" |
- ForEach-Object { Get-AuthenticodeSignature $_.FullName }
-```
-
-Expect `Valid`, publisher `Cua AI, Inc.`. No admin, PATH changes, scheduled tasks, firewall rules, or SSH setup are needed. Do not use an unqualified installer one-liner: the inspected installer enabled autostart by default.
-
-## No per-window setup
-
-The extension selects the installed native package automatically. To use a separate legacy executable instead, set `CUA_DRIVER_PATH` or a trusted local `command`. It does not scan user profiles for legacy installs or update CUA at startup.
-
-No `.pi/computer-use.json` is required. For an unusual environment, an ignored local file can override the trusted executable or transport:
-
-```json
-{
- "command": "/mnt/c/Users/<user>/AppData/Local/Programs/PiComputerUsePoC/cua-driver-0.33.2/cua-driver.exe"
-}
-```
-
-Optional `args` replace `['mcp', '--direct', '--embedded']`; optional `interop` selects a live WSL interop socket. These are trusted user settings, not agent-callable executable arguments. The old `target` field is rejected rather than silently changing its permission boundary. Remove old single-window PoC configuration before using YOLO. Environment overrides are read when the session is first used; ordinary opening and window selection never need a reload.
-
-### Qualified continuous paths
-
-The bundle includes the pinned Windows fork routes. To use the older separately qualified held-path artifact instead, select it in the ignored local config:
-
-```json
-{
- "command": "/absolute/path/to/qualified/artifact/runtime/cua-driver.exe",
- "runtime": "windows-held-paths"
-}
-```
-
-This is trusted user setup, not an agent tool parameter. Selection does not install or replace a runtime. The executable must match the exact source/version/SHA-256 in `HELD_PATH_RUNTIME` in `src/driver.ts`; different bytes refuse before startup. Fork arguments are fixed to owned direct embedded MCP. The adapter sets child-local `CUA_DRIVER_MCP_ENVELOPES=1` and requires the upstream EOF-aware capability. Default serial MCP cannot cancel a held path safely on EOF.
-
-Keep the supplied runtime companions beside the executable. Do not change signing trust, elevation or OS policy to run it.
-
-The key-hold selection below can replace this local choice. Close the current computer runtime and reload after a selection change. Remove the local selection to return to the bundled runtime; no startup update runs.
-
-### Qualified key holds
-
-For bounded keys and modifier-held pointer work, explicitly select `runtime:"windows-key-holds"` with the permanent qualified executable. The exact version/source/hash is `KEY_HOLD_RUNTIME` in `src/driver.ts`; both forks use fixed owned MCP arguments and the acknowledged EOF receiver above. The older mouse-path fork cannot hold keys.
-
-### WSL interop
-
-Windows processes must run in the interactive desktop session, not service Session 0. A long-lived Linux process may inherit a stale `WSL_INTEROP`. If Windows commands hang, check that its socket exists **and** a simple `cmd.exe /d /c ver` succeeds. A live WSL terminal launched from Windows can provide a current endpoint. Keep its owning session alive; do not persist this machine's ephemeral socket as a default. A Linux Driver does not control the Windows host desktop.
-
-One-shot PowerShell commands need closed stdin (`</dev/null`). Owned MCP and the window-identity peer intentionally keep stdin open for their protocol and exit on EOF. The wrapper forwards telemetry-off and update-check-off settings through `WSLENV`; it does not edit saved Driver preferences.
-
-## Window lifetime and limits
-
-A session-owned PowerShell peer attaches a private marker to each accessible window. Windows destroys that property with the window; a recycled HWND cannot inherit the old reference. Before observation or input, the extension checks PID and marker. On EOF the peer removes its surviving markers. It does not capture UI, send input, launch apps, or elevate permissions.
-
-An inaccessible/elevated window can appear in discovery metadata without a usable reference. The extension does not bypass Windows permissions. References are not recovered across Pi reloads or session replacement. In YOLO, discover again. In Safe, open a new window; discovery cannot adopt surviving windows.
-
-The marker check and Driver dispatch are separate native operations, not an atomic OS transaction. Concurrent external destruction or creation during that gap is not independently proven safe. New-window matching excludes prelaunch PID/HWND addresses, even if one is recycled during launch, and compares new candidates with the resulting app's windows, including packaged apps that use an existing process. It cannot establish causality against an unrelated concurrent launch of the same app. Do not claim OS isolation or proven focus/pointer non-interference from these checks.
-
-## Run without Pi Agent IDE
-
-Without Pi Agent IDE, or with an IDE whose plugin API version does not match, computer use registers deferred `computer_read`, `computer_search`, `computer_write` and `computer_insert` tools and turns `tool_search` on. See [the resource guide](computer-resources.md).
-
-Normal sessions load the globally configured package. There is no tracked project extension loader: loading both the package and a wrapper registers the same tools twice. For development, point the package entry in your Pi settings at the checkout you are working on, install its dependencies, and point the entry back when you are done.
-
-## Contract checks
-
-```bash
-npm test
-npm rebuild node-pty
-npm run test:integration
-npm run typecheck
-```
-
-The real-Pi MCP fixture substitutes Windows lifetime identity only; real registration, session logic, MCP, result forwarding, and EOF run. Unit checks cover new/existing windows, activation PID mismatch, ambiguous/missing apps, stale/cross-window references, lifetime reuse, cancellation, input uncertainty, and scoped pixel bounds. Safe checks also cover old refs, shared-PID siblings, runtime ownership loss, user-command restoration, invalid policy arguments, and mode changes during queued/native work. Fixtures are not native Windows input evidence.
-
-[The resource guide](computer-resources.md) describes the current tools.
diff --git a/docs/vision-debug.md b/docs/vision-debug.md
index dfbc9a3..cd4f052 100644
--- a/docs/vision-debug.md
+++ b/docs/vision-debug.md
@@ -1,6 +1,6 @@
# Vision debug viewer
-[LPT-373](https://linear.app/alexshpunt/issue/LPT-373) added the local observation viewer for the current Pi session. [The native CUA route](computer-resources.md) now supplies exact PNG/JPEG captures and cap/scale controls. The old Sharp kernel trial and saved-original route are retired. It uses existing extension hooks, not a provider override, a second JavaScript executor or a new model-facing tool.
+`/computer-debug` opens a local observation viewer for the current Pi session. [The native CUA route](computer-resources.md) supplies exact PNG/JPEG captures and cap/scale controls. The viewer uses existing extension hooks, not a provider override, a second JavaScript executor or a new model-facing tool.
## Open and stop
@@ -12,11 +12,11 @@ Closing/reloading the Pi session closes the server. A new session has a new view
## What the panel proves
-The user chose to use only Pi's existing API. `before_provider_request` provides a mutable application-boundary payload. The viewer copies image bytes from that event and hashes the decoded bytes with SHA-256. It also keeps bounded text excerpts and the payload paths where they appeared. It never mutates the payload, filters model history or fetches remote images.
+The viewer uses only Pi's existing API. `before_provider_request` provides a mutable application-boundary payload. The viewer copies image bytes from that event and hashes the decoded bytes with SHA-256. It also keeps bounded text excerpts and the payload paths where they appeared. It never mutates the payload, filters model history or fetches remote images.
**Final delivery is unconfirmed.** Later extension handlers can replace the payload. Codex WebSocket can derive a delta request and use server-side context. Provider-internal resize and embeddings are not available. A provider response is recorded, but does not prove the original hook payload was sent unchanged. Frames remain `prepared`, not `sent`.
-The three displayed images are a viewer limit, not the effective image budget tracked in LPT-380. Existing context can contain more images.
+The three displayed images are a viewer limit, not an image budget. Existing context can contain more images.
## Images, structure and actions
@@ -26,7 +26,7 @@ The three displayed images are a viewer limit, not the effective image budget tr
- Adapter structure is labelled separately from observed provider text. Metadata in tool `details` is not automatically model input. Text excerpts are labelled when truncated.
- Native window captures can carry snapshot hints added above the returned PNG. When annotation changes pixels, the private comparison retains the exact raw returned PNG with its hash. This is not a full-resolution original or a second capture. Missing or evicted comparison bytes remain unavailable.
- The UI hints switch affects later captures only; per-capture overrides are recorded. Stored sent images are not rewritten. A boundary card with available raw bytes can switch between exact marked and raw pixels.
-- Requested settings and known adapter timings are recorded. Effective settings not reported by the adapter remain unknown. The viewer cannot change Safe/YOLO. LPT-375 adds only resizing controls; the broader options in LPT-374 remain separate.
+- Requested settings and known adapter timings are recorded. Effective settings not reported by the adapter remain unknown. The viewer cannot change Safe/YOLO; its only controls are capture size settings.
## Bounds and cleanup
diff --git a/docs/vision-scale.md b/docs/vision-scale.md
deleted file mode 100644
index 94c96a1..0000000
--- a/docs/vision-scale.md
+++ /dev/null
@@ -1,49 +0,0 @@
-# Agent image size
-
-Historical pre-CUA route. Its code, runners and dependencies were retired by LPT-396; revision-linked results remain evidence, not current API guarantees. Use [the resource guide](computer-resources.md) for current actions, scaling and fresh zoom.
-
-[LPT-375](https://linear.app/alexshpunt/issue/LPT-375) adds physical PNG downsampling to the owned-browser tool path. It does not resize general desktop images or change cache, provider detail, model context or Safe/YOLO.
-
-## Controls
-
-Open the session's Vision debug URL (`/computer-debug status`). The panel has:
-
-- **Downsampling:** Lanczos2, Lanczos3, Mitchell or MKS2021.
-- **Max long edge:** 64–4096 pixels; initial value 1280.
-- **Scale:** 0.05–1; initial value 1. A slider and synchronized numeric box accept precise values such as 0.625.
-- **Apply to next capture:** validate and apply the whole tuple together.
-
-The cap is `floor(maxLongEdge × scale)`. For 800 and 0.5 it is 400 pixels. The whole image fits within that cap, keeping its aspect ratio to pixel rounding; small images are not enlarged. PNG encoding is fixed. MKS2021 is the chosen default for the 500px overview. A small 400px experiment scored 7/12 twice with MKS2021 and 6/12 with each other kernel; this is not a universal readability ranking or a guarantee for small text.
-
-Settings apply to the next operation's capture, including image-only observations and action screenshots. An operation snapshots the tuple before dispatch; a panel edit during input or capture affects the following operation, never half of the current frame. Historical frames retain their original settings version and bytes. Panel polling does not overwrite an unsubmitted edit or collapse unchanged expanded action rows.
-
-Reload and a new Pi session restore defaults. Stopping the debug viewer does not disable the browser's image processing or change its settings. No persistent machine configuration is added.
-
-## Coordinates and originals
-
-`click:{x,y}` and `move:{x,y}` use the **last returned agent PNG's pixels**. The parent bridge applies that frame's CSS-rectangle/output ratios and crop offset exactly once before passing source CSS coordinates to the Windows worker. PNG raster dimensions can differ by a rounding pixel; reported `sourceCss` geometry, not that extra raster pixel, controls input mapping. Existing freshness, viewport/scroll/zoom and target checks still run. Compact input retires pixel freshness; capture again before another point action.
-
-Region requests continue to use **source viewport CSS rectangles**. A crop is also physically bounded, and its returned image coordinates are translated back to the full viewport. This task does not replace the crop backend or claim to finish LPT-377.
-
-Each result reports `agent_image`: frozen configuration/version, effective cap, source/output hashes, actual dimensions, bytes, mapping and stage timings. The `screenshot` field still describes the source CSS viewport, not the resized PNG. Normal full observations include this metadata in their text; image-only results keep pixels only, with metadata available in structured results.
-
-The original is the exact **pre-resize captured PNG**, which can already include hint overlays. It is not a fabricated second capture or a promised clean pre-overlay image. Six recent distinct originals are retained by hash, within 48 MiB, with a 12 MiB per-source limit. The viewer links retained originals separately; old links can expire. Debug frame bytes have their own existing bounded store. Final provider delivery and internal model preprocessing remain unconfirmed.
-
-Processing runs in the parent Pi process, not in a new Windows service. Timings separate source decode/header work, spatial resampling and fixed PNG encoding. Browser bridge timings and model thinking are different costs.
-
-## Reproduce checks and experiments
-
-- `npm test` and `npm run typecheck` protect dimensions, no enlargement, four actual kernels, tuple validation, frozen settings, point/crop mapping, polling and server security.
-- `npm run test:integration` includes a native loopback provider adapter that receives the real Pi tool/Codemode PNGs at the default 500 and an edited 400 pixels and checks source-mapped input. It uses no paid model calls.
-
-## Interactive reading trial
-
-Use **Open the random text reading trial** in Vision debug. The same private server hosts two separate columns of six independent random strings each: white on the left, dark on the right. There are twelve answer fields, no confidence selectors, and resizing controls including the precise scale box. Each half of the 980×480 source PNG has font sizes 28, 20, 16, 12, 10 and 8 px and fine lines. CSS fitting and the viewport size also affect the final pixels; record actual image metadata, not just nominal font size.
-
-Read the actual agent image only. The expected phrases and digits stay in server memory, never in DOM text or a JSON answer field. Enter each string without its row number. Matching is exact, including case and spaces; an empty or different answer fails. **Apply: check and new trial** grades all twelve fields, keeps the previous result, applies the settings and immediately creates new unpredictable text. Inputs clear. Correct answers are never revealed, and a trial cannot be graded twice. No manual answer JSON files are needed.
-
-In Codemode, submit with `observe:false`, then capture again. Forward the PNG only after its metadata matches the selected settings and the page shows a new ready trial. Apply can finish during an observation that already froze the previous settings: discard that image and retry a bounded observation without repeating Apply. A fixed pause is not proof of readiness. Do not forward the automatic Apply screenshot as a smaller trial. Generic control discovery is safe because the sample has no DOM answer text. Do not read source PNGs, OCR, hidden state or expected answers to complete the trial.
-
-The server keeps six active samples and sixteen grades, in memory only. Reload expires them. Private loopback/session URL, Host/Origin, JSON shape and body-size checks also cover grading. A grade lists statuses, not expected or submitted text. Saved tool images and their model/settings metadata provide separate durable evidence.
-
-The first comparison is interactive, not a paid model matrix. A measured boundary belongs to the tested page, font, source size, settings and model; repeat observed mismatch boundaries before claiming a threshold. Four-kernel pixel/timing comparisons are not a model-readability ranking.
diff --git a/docs/window-overlay.md b/docs/window-overlay.md
index 93506a8..9fea668 100644
--- a/docs/window-overlay.md
+++ b/docs/window-overlay.md
@@ -14,7 +14,7 @@ Window coordinates remain in the CUA physical-pixel domain. Qt positions its own
Qt may round its own transparent canvas outwards at fractional DPI. The helper allows only that bounded canvas overscan, maps drawing against the actual HWND extent and clips it to the exact source rectangle. Source identity and geometry checks, and CUA input coordinates, remain unchanged.
-Window zoom outlines the requested rectangle, not CUA's extra padding. CUA 0.33.2 does not publish the padded rectangle. The adapter recovers the window zoom transform only when native AX frames and their rounded screenshot frames prove one unique native width. Missing or ambiguous geometry clears the old mark and reports unavailable; it never guesses a ratio. Display crop uses the exact region of the fresh full-resolution capture.
+Window zoom outlines the requested rectangle, not CUA's extra padding. Cua Driver does not publish the padded rectangle. The adapter recovers the window zoom transform only when native AX frames and their rounded screenshot frames prove one unique native width. Missing or ambiguous geometry clears the old mark and reports unavailable; it never guesses a ratio. Display crop uses the exact region of the fresh full-resolution capture.
AX action targets use the observed element's native bounds, clipped to the window. Pixel clicks use the resolved physical point from a recognized successful CUA receipt, after the one dispatch. Receipts without coordinates produce no point. Other ungrounded keyboard targets are not invented. Drawing failure does not change or repeat input. The helper watches primary-display geometry and hides a display mark if it changes.
diff --git a/src/config.ts b/src/config.ts
index 10473a0..6c61894 100644
--- a/src/config.ts
+++ b/src/config.ts
@@ -47,7 +47,7 @@ export async function loadComputerConfig(cwd: string): Promise<ComputerConfig> {
try {
raw = JSON.parse(await readFile(path.join(cwd, ".pi", "computer-use.json"), "utf8"));
} catch (error) {
- if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw new Error("Invalid .pi/computer-use.json; use the optional environment-only config in docs/standalone-poc.md.");
+ if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw new Error("Invalid .pi/computer-use.json; see docs/setup.md#override-the-runtime.");
}
if (!raw || typeof raw !== "object" || Array.isArray(raw)
|| Object.keys(raw).some(key => !["command", "args", "interop", "runtime", "machines"].includes(key))