Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.
TypeScript accessibilityagent-toolsai-agentbrowser-automationcomputer-usecuadesktop-automationelectronpipi-coding-agentpi-extensionpi-packageplaywrighttypescriptui-automationvisionwebview2windowswindows-automationwsl
chore(delivery): separate independent tests and documentation
Aleksandr Shpuntenkofa718e3e7768
diff --git a/.pi/skills/develop-computer-use/SKILL.md b/.pi/skills/develop-computer-use/SKILL.mdindex af7c968..c919df3 100644--- a/.pi/skills/develop-computer-use/SKILL.md+++ b/.pi/skills/develop-computer-use/SKILL.md@@ -15,7 +15,7 @@ Do not use a project `.pi/settings.json` override with `extensions: []` plus a p2. Back up `~/.pi/agent/settings.json` in the worktree's ignored `.tmp/` directory and record its SHA-256. Other agents may change the file while you work, so restore only your own line later, never the whole backup.3. Replace only that package path with the absolute worktree path. Keep every other entry unchanged. Remove any leftover project loader (`.pi/extensions/computer-use.ts`) or project package override from older setups.4. Put the runtime selection the check needs in the worktree's ignored `.pi/computer-use.json` (`command`, `runtime`, `interop`). The global checkout's own `.pi/computer-use.json` may name a profile the worktree code no longer has, so do not start Pi from the global checkout while the path is swapped.-5. Check dependencies in this worktree. A fresh Git worktree has no `node_modules`; the global checkout does not provide them. Install the locked public SDK with `npm ci --ignore-scripts --legacy-peer-deps`, then require a clean `npm ls --depth=0`. If the check needs SDK source, use `npm run dev:ide -- /absolute/path/to/pi-agent-ide` with a source version satisfying this package's dependency range and check the dependency tree again. A newer checkout outside the range is a setup failure, not a reason to change dependency versions or skip the check. Keep global installs and machine settings unchanged.+5. Check dependencies in this worktree. A fresh Git worktree has no `node_modules`; the global checkout does not provide them. Install with `npm ci --ignore-scripts --legacy-peer-deps`, link the public IDE SDK with `npm run dev:ide -- /absolute/path/to/pi-agent-ide`, then confirm `npm ls --depth=0` is clean. Do not change dependency versions, global installs or machine settings for this setup.## Verify before reloading@@ -26,11 +26,11 @@ Do not use a project `.pi/settings.json` override with `extensions: []` plus a ppi --mode rpc --no-session --approve --no-mcp --offline```- Require exit code zero and no extension load errors. The `computer-mode`, `computer-images` and `computer-visuals` commands must each appear once and name the worktree's `src/index.ts`. Missing or duplicate commands are a failure. This checks loading, not native Windows behavior.+ Require exit code zero and no extension load errors. The `computer-mode` and `computer-debug` commands must each appear once and name the worktree's `src/index.ts`. Missing or duplicate commands are a failure. This checks loading, not native Windows behavior.7. Only after startup passes, reload with `pi_extension_dev_reload_self` and explicit acceptance of state loss. Put the worktree, the swapped settings line and the exact live verification steps in the continuation. Follow the `pi-feature-verification` skill: exercise the changed behavior, inspect agent-facing results, and capture the real viewport with `inspect_tui` when the result is visible. Do not rely on old runtime state or window references after reload.## Restore global loading8. Before review, commit, push, or ending a paused/canceled task, put the global checkout path back in that one settings line. Compare the file with the backup: only changes made by others since the backup may remain.-9. Repeat the startup probe. Require the commands registered by the restored global checkout once each, from its `src/index.ts`, with no extension errors. Its command set can differ if the worktree is newer. Reload Pi before Git work or worktree removal so the session itself returns to global loading. If the user requests more changes, repeat setup and cleanup.+9. Repeat the startup probe. The commands must come from the global checkout's `src/index.ts`, once each, with no extension errors. Reload Pi before Git work or worktree removal so the session itself returns to global loading. If the user requests more changes, repeat setup and cleanup.10. Delete the settings backup after restoration is confirmed. Report any cleanup or startup blocker rather than leaving the personal settings pointing at a worktree.diff --git a/README.md b/README.mdindex 307d589..677b668 100644--- a/README.md+++ b/README.md@@ -108,9 +108,9 @@ The browser's toolbar, tabs and system dialogs are still ordinary windows, so thEach session has its own folder on the controlled machine for files an app opens or saves. In YOLO mode the agent can also read and write the clipboard, record the screen to video, and record what the machine plays, optionally with a transcript.-### Computer visuals+### Debug viewer and overlay-`/computer-visuals` chooses `off`, `subtle` or `full`, with optional image previews when the native driver supports them. A preview shows the final operation image; it does not prove delivery to a model. `/computer-images` changes screenshot size, hints and image guides for future captures.+`/computer-debug` opens a private viewer with the images the agent actually received, its action trace and the capture settings. On Windows an optional overlay briefly outlines the window and target the agent is working on, without taking focus.## Platforms@@ -170,8 +170,7 @@ The Windows binaries aren't code-signed. Signing certificates are hard to get fo| Command | Effect || -------------------------------- | -------------------------------------------------------- || `/computer-mode safe\|yolo` | Switch between Safe and YOLO for this session |-| `/computer-visuals off\|subtle\|full` | Choose native activity cues and image previews |-| `/computer-images` | Show or change capture size, hints and image guides |+| `/computer-debug on\|off` | Start or stop the private vision debug viewer || `/computer-setup PART [MACHINE]` | Install an optional part; without a part it lists them |## Feedback and contributions@@ -191,7 +190,7 @@ I use this every day on real tasks, but I can only test the machines, apps and w| [Temporal capture](docs/temporal-capture.md) | Frame sequences and their timing || [Selected displays](docs/selected-displays.md) | Monitor discovery and display reads || [Image guides](docs/image-guides.md) | Coordinate grid and last-action marks |-| [Native visuals](docs/native-observability.md) | Human controls, previews and presentation safety |+| [Vision debug viewer](docs/vision-debug.md) | What the agent saw and the capture controls || [Bundled runtime](docs/bundled-runtime.md) | Native runtime packages and platform boundaries || [Setup and modes](docs/setup.md) | Install, Safe and YOLO, runtime overrides and window lifetime |diff --git a/docs/bundled-runtime.md b/docs/bundled-runtime.mdindex 20d47ee..b3b207b 100644--- a/docs/bundled-runtime.md+++ b/docs/bundled-runtime.md@@ -1,6 +1,6 @@# Bundled CUA runtime-The npm package carries one native runtime: Cua Driver 0.38.0 for Windows x64, built from source `505d090e` of the [alexshpunt/cua](https://github.com/alexshpunt/cua) fork. [FORK.md](https://github.com/alexshpunt/cua/blob/pi-computer-use/FORK.md) lists the fork's changes. The `windows-desktop` profile needs no separate driver configuration:+The npm package carries one native runtime: Cua Driver 0.37.0 for Windows x64, built from the `pi-computer-use` branch of the [alexshpunt/cua](https://github.com/alexshpunt/cua) fork at source `33c4b82b`. [FORK.md](https://github.com/alexshpunt/cua/blob/pi-computer-use/FORK.md) lists what the fork adds to upstream Cua Driver. It is the qualified `windows-desktop` profile, so a fresh installation gets every native capability without any configuration:```shpi install npm:pi-agent-computer-use@@ -10,7 +10,7 @@ An update installs a new extension version with its exact runtime. Computer use## Where the files come from-The files are published as the fork release [`pi-runtime-0.38.0-505d090e`](https://github.com/alexshpunt/cua/releases/tag/pi-runtime-0.38.0-505d090e): `cua-driver.exe`, `cua-driver-uia.exe`, `cua-cursor-theme.exe`, `LICENSE.md` and `THIRD_PARTY_NOTICES.md`. They were cross-built with Rust 1.97.1, cargo-xwin 0.23.1 and LLVM 18, with a static CRT and canonical Windows manifest resources. They are not copies of a GitHub Actions artifact.+The files are published as the fork release [`pi-runtime-0.37.0-33c4b82b`](https://github.com/alexshpunt/cua/releases/tag/pi-runtime-0.37.0-33c4b82b): `cua-driver.exe`, `cua-driver-uia.exe`, `cua-cursor-theme.exe`, `LICENSE.md` and `THIRD_PARTY_NOTICES.md`. They are byte-identical to the fork's CI build of that commit.`src/runtime-pins.json` pins the source, the version, the release URL and the SHA-256 of every file. Before packing, `npm pack` runs `scripts/fetch-runtime.mjs`, which downloads the files into `native-runtime/win32-x64/runtime/` and checks each hash. The directory is not committed to Git.@@ -43,10 +43,8 @@ Computer use owns one MCP child. It disables telemetry and update checks, requir## System requirements- The Windows binaries are unsigned. Shipping a UIAccess helper does not grant UIAccess rights or bypass Windows trust policy.-- UIAccess/trusted-path deployment and UWP or elevated-window targets are not qualified by this runtime profile.-- Physical multi-monitor, mixed-DPI and negative-origin layouts, and concurrent desktop ownership, need separate qualification.- WSL needs a working Windows interop endpoint.-- [Native visuals](native-observability.md) require advertised driver capabilities. The adapter alone does not qualify a renderer or establish screenshot exclusion.+- The optional Windows Qt overlay is separate and not part of the runtime.## Overridesdiff --git a/docs/complete-input.md b/docs/complete-input.mdindex eb3c89c..92a210c 100644--- a/docs/complete-input.md+++ b/docs/complete-input.md@@ -4,9 +4,7 @@ See [the resource guide](computer-resources.md) for the current addresses, viewsThe input tools send each native action against the window's current overview. CUA remains the only input backend. A display capture or detail cannot ground input.-[Native visual drivers](native-observability.md) own asynchronous cursor playback. Presentation must not add an arrival wait to input execution. Intentional typing delays, held input and drag timing still belong to the input operation.--For a driver without native visual capabilities, the adapter sets the legacy cursor glide to 1 ms and verifies it before input. A refused or unapplied legacy setting prevents startup. Drivers running without an overlay draw no agent cursor.+CUA draws an agent cursor over the desktop and, by default, waits for it to glide to each new point before a pointer action: about 600 ms per 300 px. When the driver starts, the adapter sets that glide to 1 ms, so the agent cursor jumps to each point and still shows the click. A background click on a WinForms button then takes about 22 ms instead of 640 ms. The driver's own pauses between press and release stay: they are what makes double and right clicks take about 200 and 90 ms. If the driver refuses the setting or does not apply it, startup fails before any input. An isolated session's driver runs without the overlay: it draws no cursor and so never waits for one.## Single actionsdiff --git a/docs/computer-resources.md b/docs/computer-resources.mdindex cd13bd9..9b6f54d 100644--- a/docs/computer-resources.md+++ b/docs/computer-resources.md@@ -9,7 +9,7 @@ Everything else is a `computer_*` tool with typed fields, the same in both modesEvery `computer_*` tool is deferred: a session that never uses the computer pays nothing for them. `tool_search` loads at most 8 best matches per query, so load one group at a time: `computer open window`, `computer wait browser`, `computer files clipboard record`, `computer click hover drag scroll`, `computer type press select timeline`, `computer view viewport`. Computer use turns `tool_search` on.-The `/computer-mode`, `/computer-images`, `/computer-visuals` and `/computer-setup` commands belong to the user. [Native visuals](native-observability.md) describes presentation and capture preferences. When a part is missing, the refusal names the `/computer-setup` command that installs it; ask the user to run it.+The `/computer-mode`, `/computer-debug` and `/computer-setup` commands belong to the user. When a part is missing, the refusal names the `/computer-setup` command that installs it; ask the user to run it.The desktop scope is Windows, with Pi running on it directly or in its WSL, on this machine or on another one over SSH. Safe permits windows the session opened and forbids display reads. YOLO also permits existing windows the task needs; it is not OS isolation.@@ -60,7 +60,7 @@ Results show addresses in a short form, `computer:<machine>/<letter><number><tagSearch `computer:machine/<name>/windows` or `/apps` with a title substring or `regex:<pattern>`; sandbox addresses take the same queries. Discovery launches nothing. Search an observation to filter its UI or page lines without another walk.-Read a window for its UI tree and a small overview image. Overviews have a 1280 px long edge by default; add `capture:{"maxLongEdge":...}` to one Read for another size, and `/computer-images` changes the session default. A fresh Read issues an observation address. Read a display for readonly display pixels (YOLO).+Read a window for its UI tree and a small overview image. Overviews have a 1280 px long edge by default; add `capture:{"maxLongEdge":...}` to one Read for another size, and `/computer-debug` changes the session default. A fresh Read issues an observation address. Read a display for readonly display pixels (YOLO).| View | Meaning || --- | --- |@@ -287,4 +287,4 @@ else if (r.outcome === "refused") { /* nothing was sent */ }Close only task-owned windows. Delete a machine address to release that machine, and `computer:session` to release every machine, browser, sandbox and isolated session this computer use started; Pi's shutdown does the same. This does not close or save the user's documents. The result lists each released part; an uncertain cleanup is `unknown`.-For a source checkout, run `npm ci --ignore-scripts --legacy-peer-deps`. When SDK-source testing is needed, link a compatible IDE source version with `npm run dev:ide -- /absolute/path/to/pi-agent-ide`; keep it within this package's dependency range. To try a worktree, point the package entry in your Pi settings at it, and point it back afterwards.+For a source checkout, run `npm ci --ignore-scripts --legacy-peer-deps`, then `npm run dev:ide -- /absolute/path/to/pi-agent-ide`. To try a worktree, point the package entry in your Pi settings at it, and point it back afterwards.diff --git a/docs/image-context.md b/docs/image-context.mdindex 5c35fce..0b9c5c5 100644--- a/docs/image-context.md+++ b/docs/image-context.md@@ -4,9 +4,9 @@ This extension never removes or replaces older image blocks, rewrites text dumpsPi's normal compaction may summarize older context and keep recent messages. It does not erase the original session entries. Reload alone does not remove images. This extension does not change compaction settings or provider cache controls. Other extensions and provider processing can still affect final delivery.-## Previews are separate+## Debug cards are separate-[Native previews](native-observability.md) show a bounded final operation image. Their lifetime does not limit model context, and they do not prove downstream provider delivery. Computer use keeps no debugger-only image mirrors or separate viewer archive. Private session history and operational captures must not be shared.+Vision debug keeps three recent application-boundary cards and bounded action/request metadata. That is a viewer memory limit, not a model-context limit. Its request summary counts all observed image blocks. There is no image-budget footer or active/omitted projection. Private session history and debug pixels must not be shared.## Verificationdiff --git a/docs/native-observability.md b/docs/native-observability.mddeleted file mode 100644index 87815b3..0000000--- a/docs/native-observability.md+++ /dev/null@@ -1,47 +0,0 @@-# Native computer visuals--`/computer-visuals` controls human-only presentation on the machine that executes computer use. Native visuals require a driver advertising both `set_agent_visuals` and `present_agent_activity`. The adapter uses that driver's renderer and theme. It starts no viewer, HTTP listener or separate overlay process.--A driver without these capabilities keeps the legacy cursor behavior described in [complete input](complete-input.md). A version number alone does not enable native presentation.--## Human controls--Run `/computer-visuals` to show the current settings, or choose a preset:--- `off` disables presentation and discards pending previews.-- `subtle` keeps activity cues and disables previews by default.-- `full` enables previews by default.--A JSON patch controls individual categories and rendering preferences:--```text-/computer-visuals {"observations":true,"previews":false,"pointer":true,"scroll_drag":true,"keyboard":true,"opacity":0.8,"quality":"balanced","reduced_motion":false}-```--`opacity` is 0..1. `quality` is `performance`, `balanced` or `high`. Preferences are saved on the Pi session branch; navigating to another branch restores its preferences. These commands belong to the human. They do not change Safe/YOLO, capture sizing, input delivery or permission checks.--`/computer-images` controls future captures in this session: `cap 64..4096`, `scale 0.05..1`, `hints on|off`, `guides off|grid|action|all`, or `reset`. Calling it without arguments shows the settings. An individual Read can still override its capture options. Existing images keep their bytes.--## What a cue means--An overview identifies a captured window or display. A detail cue marks a requested physical region only when the adapter has a proven transform. Inspection and browser cues associate the operation with its native window; a Playwright CSS box does not establish a desktop rectangle. Element inspection uses native element bounds when available.--Action cues can report `delivered`, `refused` or `unknown`. `delivered` means input was sent, not that the application changed. Only a completed condition check can report `verified`. Quiet verification samples and background watch frames do not each create another cue.--Optional previews use a final PNG/JPEG returned by the operation, after cropping, resizing and annotation. A `changes` Read previews a returned change crop. A selected-frame batch admits at most its last returned image; a contact sheet or composite admits its final image. Capturing a sequence without returning images produces no preview. No extra capture or inference runs for presentation.--A preview does not establish that arbitrary Codemode forwarded that image or that a remote model received it. [Normal Pi image context](image-context.md) owns image retention and provider boundaries. Preview images are not archived or written to disk by presentation.--## Safety contract--Native presentation must be input-transparent, non-activating and protected from capture. It must not move the operating system cursor merely to animate. Input execution and visual playback have separate clocks: rendering and visual travel must not delay dispatch, while intentional typing, drag and system input timing stays intact.--Window events carry the existing HWND/PID lifetime identity and physical source bounds. The native renderer must revalidate them and hide stale drawings after movement, resize, minimize, closure or handle reuse. Selected displays use their exact native monitor bounds, including negative desktop origins. Ambiguous geometry never becomes a guessed rectangle or point.--The adapter admits one native request at a time and retains one replaceable pending activity. Pending activity expires after 250 ms; preview transport accepts only bounded PNG/JPEG payloads. Cancellation and mode changes revoke older generations. A cancellation of an old operation cannot clear a newer one. Capture companions disable their own presentation.--`highlight.status: queued` records adapter admission, not proof that a person saw pixels. An unavailable renderer does not repeat capture or input. Failed or cancelled startup releases the connection while keeping presentation preferences subscribed for a later attempt. Final session shutdown disposes those subscriptions. Backend refusal and capture exclusion still need qualification on the exact Windows runtime; the adapter alone cannot prove focus, transparency, screenshot exclusion or visual quality.--## Runtime qualification--Use the exact source and file hashes in [the runtime pins](bundled-runtime.md). Before shipping a native visual runtime, check a real Windows desktop for focus and hit testing, capture exclusion, source changes, multi-monitor geometry, concurrent sessions, reduced motion, cancellation and cleanup. Measure input latency independently of animation and bound idle/active resource use. Keep private desktop recordings in ignored local artifacts.diff --git a/docs/selected-displays.md b/docs/selected-displays.mdindex 0f9411b..1514a01 100644--- a/docs/selected-displays.md+++ b/docs/selected-displays.md@@ -26,4 +26,4 @@ Selected source metadata reports `selected_display` or `selected_display_region`Safe denies discovery, overview and detail. Revoked or cancelled reads withhold metadata/pixels. Paint freshness remains unverified. No selected-display input, window relocation, virtual-desktop management or concurrent-agent arbitration is added.-[Native visual events](native-observability.md) carry the selected monitor's exact host-global bounds and display identity. A renderer without those capabilities shows no selected-display cue; the returned image is unchanged.+The installed optional human-only Qt overlay supports primary and windows, not selected monitors. Selected reads clear its old mark instead of drawing a misleading primary outline. This does not change the returned image or debug viewer.diff --git a/docs/vision-debug.md b/docs/vision-debug.mdnew file mode 100644index 0000000..cd4f052--- /dev/null+++ b/docs/vision-debug.md@@ -0,0 +1,45 @@+# Vision debug viewer++`/computer-debug` opens a local observation viewer for the current Pi session. [The native CUA route](computer-resources.md) supplies exact PNG/JPEG captures and cap/scale controls. The viewer uses existing extension hooks, not a provider override, a second JavaScript executor or a new model-facing tool.++## Open and stop++The extension starts the viewer on `session_start` and prints its URL. `/computer-debug status` shows it again. `/computer-debug off` closes the server and clears retained pixels; `/computer-debug on` starts a fresh viewer.++The URL has a random per-server path. The server listens on `127.0.0.1` only, rejects foreign Host/Origin headers. Reads expose the trace and random reading samples. Same-origin, bounded JSON writes apply an image-settings tuple or grade one reading trial and start the next. Expected text stays out of the HTML/JSON. Do not publish or forward it. A process with access to this session's URL can read its retained UI data and change image resizing settings; this is not an OS sandbox. There is no persistent service or machine configuration.++Closing/reloading the Pi session closes the server. A new session has a new viewer URL and no previous-session frames. Reopening the same URL while its server is running reconnects to current state. After shutdown, the old URL is unavailable.++## What the panel proves++The viewer uses only Pi's existing API. `before_provider_request` provides a mutable application-boundary payload. The viewer copies image bytes from that event and hashes the decoded bytes with SHA-256. It also keeps bounded text excerpts and the payload paths where they appeared. It never mutates the payload, filters model history or fetches remote images.++**Final delivery is unconfirmed.** Later extension handlers can replace the payload. Codex WebSocket can derive a delta request and use server-side context. Provider-internal resize and embeddings are not available. A provider response is recorded, but does not prove the original hook payload was sent unchanged. Frames remain `prepared`, not `sent`.++The three displayed images are a viewer limit, not an image budget. Existing context can contain more images.++## Images, structure and actions++- The boundary-image cards show time, decoded byte count, PNG dimensions when known, hash, detail when provided, request ID and payload path.+- Browser/Windows tool hooks record adapter images and metadata separately. A capture not forwarded by Codemode stays in the adapter trace and does not become a new boundary-image card.+- Image hashes link pixels to matching computer-use actions. Identical captures are ambiguous: the panel lists candidate calls rather than assigning a unique source.+- Adapter structure is labelled separately from observed provider text. Metadata in tool `details` is not automatically model input. Text excerpts are labelled when truncated.+- Native window captures can carry snapshot hints added above the returned PNG. When annotation changes pixels, the private comparison retains the exact raw returned PNG with its hash. This is not a full-resolution original or a second capture. Missing or evicted comparison bytes remain unavailable.+- The UI hints switch affects later captures only; per-capture overrides are recorded. Stored sent images are not rewritten. A boundary card with available raw bytes can switch between exact marked and raw pixels.+- Requested settings and known adapter timings are recorded. Effective settings not reported by the adapter remain unknown. The viewer cannot change Safe/YOLO; its only controls are capture size settings.++## Bounds and cleanup++The in-memory trace retains 100 computer-use action records, 20 request summaries, three boundary frames and three recent adapter captures. Each retained image is limited to 12 MiB; the total asset budget is 48 MiB. Over-budget assets are evicted explicitly. Text and metadata are bounded, and the JSON state endpoint contains no image base64. Old source bytes are not promised after eviction.++The panel polls current state once per second. That refresh interval is not an input or provider-latency measurement. Action durations come from the hook timestamps, while capture/input stage timings come from the existing adapter when available. Concurrent nested calls remain distinct by their call IDs and parent IDs.++## Verification++`npm test -- --run test/vision-debug.test.ts` checks byte identity, captured/prepared separation, ambiguous provenance, bounded viewer history, session reset, host/origin rejection, reconnect and cleanup.++`npm run test:integration -- --run test/vision-extension.integration.test.ts` runs real Pi with its native OpenAI-compatible provider adapter against a loopback SSE fixture. It compares a boundary image hash with the bytes actually received by that fixture, exercises nested Codemode calls, deliberately does not forward one capture, records an input refusal and verifies server shutdown. It does not use real credentials or an external API.++The integration writes `vision-trace.json` and `provider-bodies.json` beside its normal run artifacts. A scripted test provider that skips `onPayload` cannot qualify this boundary; use the native-adapter fixture instead.++Action rows keep their expanded state during polling.diff --git a/docs/window-overlay.md b/docs/window-overlay.mdnew file mode 100644index 0000000..9fea668--- /dev/null+++ b/docs/window-overlay.md@@ -0,0 +1,55 @@+# Human observation and target marks++The Windows Qt helper flashes a cyan outline for a successful whole-window or primary-display overview, an amber rectangle for the requested zoom/crop area, and a pink rectangle or point for a grounded action target. Marks last two seconds; a newer mark replaces the old one. These are human-only drawings, not annotations in the model's image. Region support requires helper protocol 2; an older helper is refused instead of silently drawing the wrong whole-window frame.++The frame is a human aid, not permission, model annotation, or evidence that the screenshot has current paint. CUA still owns capture and input. The helper cannot click, type, capture, activate a source window, or choose a target from a model argument. Browser capture still has its separately approved CUA foreground behavior.++## Identity and geometry++The adapter supplies its existing window lifetime marker and the successful capture's native bounds through a private pipe. The helper checks the HWND, PID and property marker. It matches the visible DWM bounds, not `GetWindowRect`'s invisible resize borders. It watches both rectangles and hides after a move, resize, minimize, close or lifetime change. It does not follow a stale window.++The helper refuses to show if foreground or keyboard focus changes during its show operation, or if Windows capture exclusion is unavailable. `WindowTransparentForInput`, no-activate styles and native hit testing make the frame input-transparent. These flags are not a substitute for the live input check.++Window coordinates remain in the CUA physical-pixel domain. Qt positions its own per-monitor-aware HWND with native physical coordinates; it does not turn image pixels into desktop input coordinates. Other DPI setups and Windows builds still need qualification.++Qt may round its own transparent canvas outwards at fractional DPI. The helper allows only that bounded canvas overscan, maps drawing against the actual HWND extent and clips it to the exact source rectangle. Source identity and geometry checks, and CUA input coordinates, remain unchanged.++Window zoom outlines the requested rectangle, not CUA's extra padding. Cua Driver does not publish the padded rectangle. The adapter recovers the window zoom transform only when native AX frames and their rounded screenshot frames prove one unique native width. Missing or ambiguous geometry clears the old mark and reports unavailable; it never guesses a ratio. Display crop uses the exact region of the fresh full-resolution capture.++AX action targets use the observed element's native bounds, clipped to the window. Pixel clicks use the resolved physical point from a recognized successful CUA receipt, after the one dispatch. Receipts without coordinates produce no point. Other ungrounded keyboard targets are not invented. Drawing failure does not change or repeat input. The helper watches primary-display geometry and hides a display mark if it changes.++## Lifecycle and failures++No helper starts at extension registration, a policy restore, an unused close, or a refused read. The first authorized observation with valid drawing geometry starts one owned process. Permission changes clear the frame, and cancellation cannot hide a newer observation's frame. Both runtime close tools and Pi shutdown close this helper through pipe EOF. Confirmed close permits a fresh helper later; unknown shutdown remains blocked until reload.++A missing or failed helper reports `highlight.status: unavailable` without changing image bytes or repeating capture/input. `shown` is an acknowledgement, not a promise that a person saw the short flash. The adapter rechecks cancellation and desktop permissions after the callback before publishing pixels. Native frame refusal hides the previous frame.++## Build and local deployment++Only Windows x64 is implemented and qualified here. Qt's cross-platform API does not establish macOS/Linux support.++Use the installed Qt 6.11.3 MSVC2022 x64 qtbase kit and existing Visual Studio 2022 Build Tools:++```powershell+powershell -NoProfile -File scripts/build-window-overlay.ps1 -QtPrefix <Qt-kit-directory>+```++The build stages sources into the user-local `PiComputerUsePoC/.tmp` directory to avoid MSBuild case-folding problems on WSL's case-sensitive share. Deployment contains the helper, `qt.conf`, Qt6Core, Qt6Gui, Qt6Widgets and `platforms/qwindows.dll`. There is no WebEngine, QML or Python runtime. This build relies on the host's existing MSVC redistributable, Windows ICU and system DLLs; it is not a clean-machine installer.++The package generator takes the built bundle, kit, exact Qt source checkout and a new output directory:++```sh+node scripts/package-window-overlay.mjs <bundle> <Qt-kit> <Qt-source> .tmp/qt-overlay/release+```++It verifies the exact SBOM's LGPL options, keeps license texts/attributions, supplies the complete matching qtbase source archive and helper source, and records hashes and separate runtime/source costs. See [the Qt notice](../native/window-overlay/NOTICE.md). Public binary redistribution requires its own licensing review; dynamic linking alone is not enough.++Install the runtime files in `%LOCALAPPDATA%/Programs/PiComputerUsePoC/window-overlay-0.1.0`, preserving the `platforms` and `licenses` directories. Keep the source sidecar available with the deployment. Discovery refuses ambiguous installations. `PI_WINDOW_OVERLAY_PATH` is an optional trusted executable override, not a model argument. Installation never runs at Pi startup.++## Verification++Read-only detail keeps its original local region and scale, clips smaller bounds and positions a new mark against current bounds. Hiding an old drawing after geometry changes does not invalidate the window or its next detail read.++Unit contracts cover lazy startup, generations, cancellation, mode restoration, ownership, failure isolation and confirmed/unknown cleanup. The real-Pi browser contract checks unchanged PNG/JPEG delivery and frame acknowledgements without exposing private lifetime markers. Pipe tests verify EOF and malformed replies without request replay.++Check overlay bounds, focus, expiry, capture exclusion and cleanup live in a Pi session on newly opened disposable windows. Private full-display pixels stay in ignored local artifacts, never Git or Linear uploads.\ No newline at end of filediff --git a/test/fixtures/visual-commands-extension.ts b/test/fixtures/visual-commands-extension.tsdeleted file mode 100644index 5fcc654..0000000--- a/test/fixtures/visual-commands-extension.ts+++ /dev/null@@ -1,24 +0,0 @@-import { writeFile } from "node:fs/promises";-import path from "node:path";-import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";-import { CuaImageSettings } from "../../src/cua-image-settings.ts";-import { registerImageSettingsCommand } from "../../src/image-settings-command.ts";-import { VisualSettings } from "../../src/native-visuals.ts";-import { registerVisualSettingsCommand } from "../../src/visual-settings-command.ts";--/** Real Pi owns command dispatch, session entries and tree navigation; this fixture only reads the outcome. */-export default function(pi: ExtensionAPI): void {- const visuals = new VisualSettings(); const images = new CuaImageSettings();- registerVisualSettingsCommand(pi, visuals); registerImageSettingsCommand(pi, images);- pi.registerCommand("visual-probe", {- description: "Save the current test preferences.",- handler: async (_args, ctx) => {- await writeFile(path.join(ctx.cwd, "preferences.json"), JSON.stringify({ visuals: visuals.current(), images: images.settings(),- branch: ctx.sessionManager.getBranch() }));- },- });- pi.registerCommand("visual-navigate", {- description: "Navigate the test session without inference.",- handler: async (target, ctx) => { await ctx.navigateTree(target.trim(), { summarize: false }); },- });-}diff --git a/test/image-settings-command.test.ts b/test/image-settings-command.test.tsdeleted file mode 100644index f74c7e5..0000000--- a/test/image-settings-command.test.ts+++ /dev/null@@ -1,43 +0,0 @@-import type { ExtensionAPI, ExtensionCommandContext } from "@earendil-works/pi-coding-agent";-import { expect, test, vi } from "vitest";-import { CuaImageSettings } from "../src/cua-image-settings.ts";-import { registerImageSettingsCommand } from "../src/image-settings-command.ts";--type Command = { handler(args: string, ctx: ExtensionCommandContext): Promise<void> };-function fixture() {- const commands = new Map<string, Command>();- const images = new CuaImageSettings(() => "bilinear");- const notify = vi.fn();- registerImageSettingsCommand({ registerCommand: (name: string, command: Command) => commands.set(name, command) } as unknown as ExtensionAPI, images);- const ctx = { hasUI: true, ui: { notify } } as unknown as ExtensionCommandContext;- return { images, notify, run: (input: string) => commands.get("computer-images")!.handler(input, ctx), commands };-}--test("the human image command changes future native captures without a filter switch", async () => {- const f = fixture(); const old = f.images.settings();- await f.run("cap 800"); await f.run("scale 0.5"); await f.run("hints off"); await f.run("guides all");- expect(f.images.settings()).toMatchObject({ kernel: "bilinear", maxLongEdge: 800, scale: 0.5, hints: false,- guides: { grid: true, action: true }, version: 4 });- expect(old).toMatchObject({ maxLongEdge: 1280, scale: 1, version: 0 });- expect(old.hints).toBeUndefined(); expect(old.guides).toBeUndefined();- await f.run("reset"); expect(f.images.settings()).toEqual(old);-});--test("bad image command arguments preserve the full settings tuple", async () => {- const f = fixture(); await f.run("cap 800"); const saved = f.images.settings();- for (const input of ["cap 63", "cap 800.5", "cap Infinity", "scale 0", "scale NaN", "hints false", "hints off extra",- "guides on", "kernel lanczos3", "mode yolo", "reset extra"]) {- await f.run(input); expect(f.images.settings()).toEqual(saved);- expect(f.notify.mock.calls.at(-1)?.[1]).toBe("warning");- }-});--test("showing image settings does not change captures and instances stay independent", async () => {- const first = fixture(); const second = fixture();- await first.run("cap 600"); await first.run("guides grid");- const saved = first.images.settings(); await first.run(""); await first.run("show");- expect(first.images.settings()).toEqual(saved);- expect(second.images.settings()).toMatchObject({ maxLongEdge: 1280, scale: 1, version: 0 });- expect(first.commands.size).toBe(1);- expect(first.notify.mock.calls.at(-1)?.[0]).toContain("600");-});diff --git a/test/native-visuals.test.ts b/test/native-visuals.test.tsdeleted file mode 100644index 82cae1a..0000000--- a/test/native-visuals.test.ts+++ /dev/null@@ -1,196 +0,0 @@-import { expect, test, vi } from "vitest";-import { NativeVisuals, VisualSettings, notifyHighlight } from "../src/native-visuals.ts";-import sharp from "sharp";-import { ComputerSession, type Driver, type DriverResult } from "../src/session.ts";-import type { InputCommand } from "../src/complete-input.ts";-import { WebError, type WebAction, type WebBrowser, type WebLauncher } from "../src/web-browser.ts";--const frame = { target: { window_id: 101, pid: 42 }, identity: `PiComputerUse.${"a".repeat(32)}:42:1`,- bounds: { x: -100, y: 20, width: 800, height: 600 } };-const tick = async () => { for (let n = 0; n < 8; n++) await Promise.resolve(); };--test("presentation returns immediately while native admission is pending, with only the newest pending image retained", async () => {- const sent: { name: string; args: Record<string, unknown> }[] = [];- let release!: () => void;- const visuals = new NativeVisuals(async (name, args) => { sent.push({ name, args }); await new Promise<void>(r => { release = r; }); return true; });- expect(await visuals.flash(frame)).toMatchObject({ status: "queued" });- await tick();- for (let n = 1; n <= 100; n++) await visuals.flash({ ...frame, mark: { kind: "action", point: { x: n, y: 50 } } });- expect(sent).toHaveLength(1);- release(); await tick();- expect(sent).toHaveLength(2);- expect(sent[1].args).toMatchObject({ generation: 101, point: { x: 100, y: 50 } });- release(); await tick(); visuals.close();-});--test("previews contain the supplied output image and validated physical source, never a new capture", async () => {- const send = vi.fn(async (_name: string, _args: Record<string, unknown>) => true); const visuals = new NativeVisuals(send);- const image = { type: "image" as const, mimeType: "image/png", data: "aGVsbG8=" };- await notifyHighlight(visuals.flash, { ...frame, image }); await tick();- expect(send).toHaveBeenCalledExactlyOnceWith("present_agent_activity", {- generation: 1, kind: "overview", source: { window_id: 101, pid: 42, identity: frame.identity, bounds: frame.bounds },- image: { data: image.data, mime_type: image.mimeType },- });- visuals.close();-});--test("invalid or out-of-source geometry is cleared without sending image content", async () => {- const send = vi.fn(async (_name: string, _args: Record<string, unknown>) => true); const visuals = new NativeVisuals(send);- const result = await notifyHighlight(visuals.flash, { ...frame, identity: frame.identity.replace(":42:", ":99:") });- await tick();- expect(result?.status).toBe("unavailable");- expect(send.mock.calls).toEqual([["present_agent_activity", { kind: "clear", generation: 1 }]]);- visuals.close();-});--test("off discards queued images and old cancellation cannot clear a newer event", async () => {- const send = vi.fn(async (_name: string, _args: Record<string, unknown>) => true); const settings = new VisualSettings(); const visuals = new NativeVisuals(send, settings);- const abort = new AbortController();- await visuals.flash(frame, abort.signal); await tick();- await visuals.flash(frame); await tick();- abort.abort(); await tick();- expect(send).toHaveBeenCalledTimes(2);- settings.update({ preset: "off" }); await tick();- expect(send.mock.calls.at(-1)).toEqual(["set_agent_visuals", expect.objectContaining({ preset: "off" })]);- const count = send.mock.calls.length;- expect(await visuals.flash(frame)).toMatchObject({ status: "skipped" });- await tick(); expect(send).toHaveBeenCalledTimes(count); visuals.close();-});--test("selected-display regions retain their physical monitor origin", async () => {- const send = vi.fn(async (_name: string, _args: Record<string, unknown>) => true); const visuals = new NativeVisuals(send);- await visuals.flash({ display: "DISPLAY2", bounds: { x: -1920, y: 0, width: 1920, height: 1080 }, mark: {- kind: "detail", region: { x: -1900, y: 40, width: 200, height: 100 },- } }); await tick();- expect(send.mock.calls[0][1]).toMatchObject({ source: { display: "DISPLAY2", bounds: { x: -1920, y: 0 } }, region: { x: -1900, y: 40 } });- visuals.close();-});--test("restoring a branch resets unrelated preferences and rejects prototype keys", () => {- const settings = new VisualSettings();- settings.update({ keyboard: false, quality: "high" });- expect(settings.restore({ preset: "subtle" })).toMatchObject({ keyboard: true, quality: "balanced", previews: false });- const before = settings.current();- expect(() => settings.update(JSON.parse('{"constructor":false}'))).toThrow();- expect(settings.current()).toEqual(before);-});-test("settings validate before mutation and unavailable native presentation does not throw into operations", async () => {- const settings = new VisualSettings();- expect(() => settings.update({ opacity: NaN })).toThrow();- expect(settings.current().opacity).toBe(0.8);- const visuals = new NativeVisuals(async () => { throw new Error("not available"); }, settings);- expect(await visuals.flash(frame)).toMatchObject({ status: "queued" }); await tick();- expect(await visuals.flash(frame)).toMatchObject({ status: "unavailable" }); visuals.close();-});---async function actionFixture(outcome: "delivered" | "refused" | "unknown" = "delivered") {- const png = { type: "image" as const, mimeType: "image/png", data: (await sharp({ create: {- width: frame.bounds.width, height: frame.bounds.height, channels: 3, background: "white",- } }).png().toBuffer()).toString("base64") };- const send = vi.fn(async (_name: string, _args: Record<string, unknown>) => true);- const visuals = new NativeVisuals(send);- const driver: Driver = {- visuals: visuals.flash, heldKeys: true, heldPaths: true, captureBoundDrag: true, semanticActions: true,- identity: vi.fn(async () => frame.identity), close: vi.fn(async () => {}),- call: vi.fn<Driver["call"]>(async (name, args): Promise<DriverResult> => {- if (name === "list_windows") return { content: [], structuredContent: { windows: [{- ...frame.target, bounds: frame.bounds, app_name: "Fixture.exe", title: "Action fixture",- }] } };- if (name === "get_window_state") return { content: [png], structuredContent: {- ...frame.target, window_bounds: frame.bounds, capture_id: "11111111-1111-4111-8111-111111111111",- screenshot_width: frame.bounds.width, screenshot_height: frame.bounds.height,- elements: [{ element_token: "field", role: "TextBox", frame: { x: 0, y: 60, w: 80, h: 40 } }],- } };- if (outcome !== "delivered") return { content: [], structuredContent: { effect: outcome } };- return { content: [], structuredContent: name === "semantic_action"- ? { operation: args.operation, pattern: "value", route: "accessibility", effect: "unverifiable", fallback: false, delivery: { mode: "background" } }- : { effect: "unverifiable" } };- }),- };- const browser = { close: vi.fn(async () => {}), page: vi.fn(async () => ({- url: "https://fixture.invalid", title: "Action fixture", snapshot: '- textbox "Field"', tabs: [], events: [],- })) };- const launch: WebLauncher = async () => ({ browser: browser as unknown as WebBrowser,- opened: { pid: frame.target.pid, version: "155", url: "https://fixture.invalid", title: "Action fixture", via: "cdp" } });- const session = new ComputerSession(driver, () => "yolo", undefined, undefined, launch);- return { session, driver, send, browser, close: async () => { visuals.close(); await session.close(); } };-}--const nativeFamilies: { label: string; command: InputCommand; family: string; native: string }[] = [- { label: "typing", command: { action: "type_text", text: "synthetic-private-text" }, family: "type_text", native: "type_text" },- { label: "key press", command: { action: "key", key: "f12" }, family: "press_key", native: "press_key" },- { label: "select all", command: { action: "select_all" }, family: "hotkey", native: "hotkey" },- { label: "semantic value", command: { action: "set_value", element_token: "field", text: "synthetic-private-value" }, family: "fill", native: "semantic_action" },- { label: "scroll", command: { action: "scroll", element_token: "field", direction: "down" }, family: "scroll", native: "scroll" },- { label: "drag", command: { action: "drag", from_x: 10, from_y: 10, to_x: 100, to_y: 100, duration_ms: 10 }, family: "drag", native: "drag" },- { label: "keyboard hold", command: { action: "hold", keys: ["ctrl"], duration_ms: 10 }, family: "hotkey", native: "hold_keys" },- { label: "held pointer", command: { action: "hold", keys: ["ctrl"], duration_ms: 10,- actions: [{ action: "drag", from_x: 10, from_y: 10, to_x: 100, to_y: 100, duration_ms: 10 }] }, family: "input_timeline", native: "hold_keys" },- { label: "keyboard timeline", command: { action: "timeline", events: [- { at_ms: 0, type: "key_down", key: "ctrl" }, { at_ms: 10, type: "key_up", key: "ctrl" },- ] }, family: "hotkey", native: "input_timeline" },- { label: "pointer timeline", command: { action: "timeline", events: [- { at_ms: 0, type: "move", x: 10, y: 10 }, { at_ms: 10, type: "wheel", dy: 1 },- ] }, family: "input_timeline", native: "input_timeline" },-];--test.each(nativeFamilies)("native $label retains its presentation family before and after one dispatch", async ({ command, family, native }) => {- const f = await actionFixture();- try {- const listed = await f.session.windows();- const ref = (listed.details.state!.windows as { window_ref: string }[])[0].window_ref;- const read = await f.session.observe(ref); await tick(); f.send.mockClear(); vi.mocked(f.driver.call).mockClear();- const result = await f.session.act({ window_ref: ref, observation_id: read.details.observation_id!, ...command });- await tick();- expect(result.details.status).toBe("delivered");- expect(vi.mocked(f.driver.call).mock.calls.map(([name]) => name)).toEqual([native]);- const activities = f.send.mock.calls.filter(([name]) => name === "present_agent_activity").map(([, args]) => args);- expect(activities.map(args => ({ kind: args.kind, action: args.action, outcome: args.outcome }))).toEqual([- { kind: "action", action: family, outcome: undefined }, { kind: "action", action: family, outcome: "delivered" },- ]);- for (const args of activities) {- expect(args.source).toEqual({ ...frame.target, identity: frame.identity, bounds: frame.bounds });- expect(Object.keys(args).every(key => ["generation", "kind", "source", "region", "action", "outcome"].includes(key))).toBe(true);- if (!command.element_token) expect(args).not.toHaveProperty("region");- }- } finally { await f.close(); }-});--test.each(["refused", "unknown"] as const)("native keyboard %s keeps its family without replaying input", async outcome => {- const f = await actionFixture(outcome);- try {- const listed = await f.session.windows();- const ref = (listed.details.state!.windows as { window_ref: string }[])[0].window_ref;- const read = await f.session.observe(ref); await tick(); f.send.mockClear(); vi.mocked(f.driver.call).mockClear();- const result = await f.session.act({ window_ref: ref, observation_id: read.details.observation_id!, action: "key", key: "f12" });- await tick();- expect(result.details.status).toBe(outcome);- expect(vi.mocked(f.driver.call).mock.calls.map(([name]) => name)).toEqual(["press_key"]);- expect(f.send.mock.calls.map(([, args]) => ({ action: args.action, outcome: args.outcome }))).toEqual([- { action: "press_key", outcome: undefined }, { action: "press_key", outcome },- ]);- } finally { await f.close(); }-});--test.each(["dom", "page"] as const)("browser %s key outcomes use keyboard presentation without forwarding key or target", async route => {- for (const outcome of ["delivered", "refused", "unknown"] as const) {- const f = await actionFixture();- try {- const opened = await f.session.openBrowser(); const ref = opened.details.window_ref!;- const read = await f.session.observeDom(ref); await tick(); f.send.mockClear(); f.browser.page.mockClear();- if (outcome !== "delivered") f.browser.page.mockRejectedValueOnce(new WebError("Synthetic page failure", outcome === "unknown"));- const action: WebAction = { action: "press", target: { ref: "e1" }, key: "Control+Shift+F12" };- const result = route === "dom"- ? await f.session.actDom(ref, read.details.observation_id!, action)- : await f.session.actPage(ref, action);- await tick();- expect(result.details.status).toBe(outcome);- expect(f.browser.page).toHaveBeenCalledExactlyOnceWith("act", action, undefined);- expect(f.send.mock.calls).toEqual([["present_agent_activity", {- generation: expect.any(Number), kind: "action", source: { ...frame.target, identity: frame.identity, bounds: frame.bounds },- action: "press_key", outcome,- }]]);- } finally { await f.close(); }- }-});diff --git a/test/visual-commands.integration.test.ts b/test/visual-commands.integration.test.tsdeleted file mode 100644index 447c471..0000000--- a/test/visual-commands.integration.test.ts+++ /dev/null@@ -1,45 +0,0 @@-import { RpcClient } from "@earendil-works/pi-coding-agent";-import { mkdtemp, readFile, rm } from "node:fs/promises";-import { tmpdir } from "node:os";-import path from "node:path";-import { fileURLToPath } from "node:url";-import { expect, test } from "vitest";--test("human visual commands persist preferences and restore the selected Pi branch without model work", async () => {- const cwd = await mkdtemp(path.join(tmpdir(), "computer-visual-commands-"));- const client = new RpcClient({ cwd,- cliPath: path.join(path.dirname(fileURLToPath(import.meta.resolve("@earendil-works/pi-coding-agent"))), "cli.js"),- env: { PI_CODING_AGENT_DIR: path.join(cwd, "agent"), PI_OFFLINE: "1" },- args: ["--no-session", "--no-extensions", "--no-mcp", "--no-skills", "--no-context-files", "--approve",- "--extension", path.resolve("test/fixtures/visual-commands-extension.ts")],- });- const probe = async () => {- expect(await client.prompt("/visual-probe")).toBe("handled");- return JSON.parse(await readFile(path.join(cwd, "preferences.json"), "utf8")) as {- visuals: { preset: string; previews: boolean; keyboard: boolean; quality: string };- images: { maxLongEdge: number; hints: boolean };- };- };- try {- await client.start();- const commands = await client.getCommands();- expect(commands.filter(command => command.name === "computer-visuals")).toHaveLength(1);- expect(commands.filter(command => command.name === "computer-images")).toHaveLength(1);- expect(await client.prompt("/computer-visuals subtle")).toBe("handled");- const saved = (await client.getEntries()).entries.filter(entry => entry.type === "custom" && entry.customType === "computer-use-visuals");- expect(saved).toHaveLength(1);- expect((await probe()).visuals).toMatchObject({ preset: "subtle", previews: false });- expect(await client.prompt('/computer-visuals {"preset":"full","keyboard":false,"quality":"high"}')).toBe("handled");- expect((await probe()).visuals).toMatchObject({ preset: "full", keyboard: false, quality: "high" });- expect(await client.prompt(`/visual-navigate ${saved[0].id}`)).toBe("handled");- expect((await probe()).visuals).toMatchObject({ preset: "subtle", previews: false, keyboard: true, quality: "balanced" });- expect(await client.prompt("/computer-images cap 960")).toBe("handled");- expect(await client.prompt("/computer-images hints off")).toBe("handled");- expect((await probe()).images).toMatchObject({ maxLongEdge: 960, hints: false });- expect(await client.prompt('/computer-visuals {"constructor":false}')).toBe("handled");- expect((await probe()).visuals).toMatchObject({ preset: "subtle", previews: false });- expect((await client.getEntries()).entries.filter(entry => entry.type === "custom" && entry.customType === "computer-use-visuals")).toHaveLength(2);- expect((await client.getMessages()).filter(message => message.role === "assistant")).toHaveLength(0);- expect(client.getStderr()).toBe("");- } finally { await client.stop(); await rm(cwd, { recursive: true, force: true }); }-});