Skip to content

Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.

docs: describe native visuals and remove debugger guides

alexshpunt-clanker[bot]e6fef7052e73
diff --git a/.pi/skills/develop-computer-use/SKILL.md b/.pi/skills/develop-computer-use/SKILL.md
index c919df3..829f29d 100644
--- a/.pi/skills/develop-computer-use/SKILL.md
+++ b/.pi/skills/develop-computer-use/SKILL.md
@@ -26,11 +26,11 @@ Do not use a project `.pi/settings.json` override with `extensions: []` plus a p
pi --mode rpc --no-session --approve --no-mcp --offline
```
- Require exit code zero and no extension load errors. The `computer-mode` and `computer-debug` commands must each appear once and name the worktree's `src/index.ts`. Missing or duplicate commands are a failure. This checks loading, not native Windows behavior.
+ Require exit code zero and no extension load errors. The `computer-mode`, `computer-images` and `computer-visuals` commands must each appear once and name the worktree's `src/index.ts`. Missing or duplicate commands are a failure. This checks loading, not native Windows behavior.
7. Only after startup passes, reload with `pi_extension_dev_reload_self` and explicit acceptance of state loss. Put the worktree, the swapped settings line and the exact live verification steps in the continuation. Follow the `pi-feature-verification` skill: exercise the changed behavior, inspect agent-facing results, and capture the real viewport with `inspect_tui` when the result is visible. Do not rely on old runtime state or window references after reload.
## Restore global loading
8. Before review, commit, push, or ending a paused/canceled task, put the global checkout path back in that one settings line. Compare the file with the backup: only changes made by others since the backup may remain.
-9. Repeat the startup probe. The commands must come from the global checkout's `src/index.ts`, once each, with no extension errors. Reload Pi before Git work or worktree removal so the session itself returns to global loading. If the user requests more changes, repeat setup and cleanup.
+9. Repeat the startup probe. Require the commands registered by the restored global checkout once each, from its `src/index.ts`, with no extension errors. Its command set can differ if the worktree is newer. Reload Pi before Git work or worktree removal so the session itself returns to global loading. If the user requests more changes, repeat setup and cleanup.
10. Delete the settings backup after restoration is confirmed. Report any cleanup or startup blocker rather than leaving the personal settings pointing at a worktree.
diff --git a/README.md b/README.md
index 677b668..307d589 100644
--- a/README.md
+++ b/README.md
@@ -108,9 +108,9 @@ The browser's toolbar, tabs and system dialogs are still ordinary windows, so th
Each session has its own folder on the controlled machine for files an app opens or saves. In YOLO mode the agent can also read and write the clipboard, record the screen to video, and record what the machine plays, optionally with a transcript.
-### Debug viewer and overlay
+### Computer visuals
-`/computer-debug` opens a private viewer with the images the agent actually received, its action trace and the capture settings. On Windows an optional overlay briefly outlines the window and target the agent is working on, without taking focus.
+`/computer-visuals` chooses `off`, `subtle` or `full`, with optional image previews when the native driver supports them. A preview shows the final operation image; it does not prove delivery to a model. `/computer-images` changes screenshot size, hints and image guides for future captures.
## Platforms
@@ -170,7 +170,8 @@ The Windows binaries aren't code-signed. Signing certificates are hard to get fo
| Command | Effect |
| -------------------------------- | -------------------------------------------------------- |
| `/computer-mode safe\|yolo` | Switch between Safe and YOLO for this session |
-| `/computer-debug on\|off` | Start or stop the private vision debug viewer |
+| `/computer-visuals off\|subtle\|full` | Choose native activity cues and image previews |
+| `/computer-images` | Show or change capture size, hints and image guides |
| `/computer-setup PART [MACHINE]` | Install an optional part; without a part it lists them |
## Feedback and contributions
@@ -190,7 +191,7 @@ I use this every day on real tasks, but I can only test the machines, apps and w
| [Temporal capture](docs/temporal-capture.md) | Frame sequences and their timing |
| [Selected displays](docs/selected-displays.md) | Monitor discovery and display reads |
| [Image guides](docs/image-guides.md) | Coordinate grid and last-action marks |
-| [Vision debug viewer](docs/vision-debug.md) | What the agent saw and the capture controls |
+| [Native visuals](docs/native-observability.md) | Human controls, previews and presentation safety |
| [Bundled runtime](docs/bundled-runtime.md) | Native runtime packages and platform boundaries |
| [Setup and modes](docs/setup.md) | Install, Safe and YOLO, runtime overrides and window lifetime |
diff --git a/docs/bundled-runtime.md b/docs/bundled-runtime.md
index 43736d9..2b850f4 100644
--- a/docs/bundled-runtime.md
+++ b/docs/bundled-runtime.md
@@ -44,7 +44,7 @@ Computer use owns one MCP child. It disables telemetry and update checks, requir
- The Windows binaries are unsigned. Shipping a UIAccess helper does not grant UIAccess rights or bypass Windows trust policy.
- WSL needs a working Windows interop endpoint.
-- The optional Windows Qt overlay is separate and not part of the runtime.
+- [Native visuals](native-observability.md) require advertised driver capabilities. The adapter alone does not qualify a renderer or establish screenshot exclusion.
## Overrides
diff --git a/docs/complete-input.md b/docs/complete-input.md
index 92a210c..eb3c89c 100644
--- a/docs/complete-input.md
+++ b/docs/complete-input.md
@@ -4,7 +4,9 @@ See [the resource guide](computer-resources.md) for the current addresses, views
The input tools send each native action against the window's current overview. CUA remains the only input backend. A display capture or detail cannot ground input.
-CUA draws an agent cursor over the desktop and, by default, waits for it to glide to each new point before a pointer action: about 600 ms per 300 px. When the driver starts, the adapter sets that glide to 1 ms, so the agent cursor jumps to each point and still shows the click. A background click on a WinForms button then takes about 22 ms instead of 640 ms. The driver's own pauses between press and release stay: they are what makes double and right clicks take about 200 and 90 ms. If the driver refuses the setting or does not apply it, startup fails before any input. An isolated session's driver runs without the overlay: it draws no cursor and so never waits for one.
+[Native visual drivers](native-observability.md) own asynchronous cursor playback. Presentation must not add an arrival wait to input execution. Intentional typing delays, held input and drag timing still belong to the input operation.
+
+For a driver without native visual capabilities, the adapter sets the legacy cursor glide to 1 ms and verifies it before input. A refused or unapplied legacy setting prevents startup. Drivers running without an overlay draw no agent cursor.
## Single actions
diff --git a/docs/computer-resources.md b/docs/computer-resources.md
index 1fa2b9b..9e39ccb 100644
--- a/docs/computer-resources.md
+++ b/docs/computer-resources.md
@@ -9,7 +9,7 @@ Everything else is a `computer_*` tool with typed fields, the same in both modes
Every `computer_*` tool is deferred: a session that never uses the computer pays nothing for them. `tool_search` loads at most 8 best matches per query, so load one group at a time: `computer open window`, `computer wait browser`, `computer files clipboard record`, `computer click hover drag scroll`, `computer type press select timeline`, `computer view viewport`. Computer use turns `tool_search` on.
-The `/computer-mode`, `/computer-debug` and `/computer-setup` commands belong to the user. When a part is missing, the refusal names the `/computer-setup` command that installs it; ask the user to run it.
+The `/computer-mode`, `/computer-images`, `/computer-visuals` and `/computer-setup` commands belong to the user. [Native visuals](native-observability.md) describes presentation and capture preferences. When a part is missing, the refusal names the `/computer-setup` command that installs it; ask the user to run it.
The desktop scope is Windows, with Pi running on it directly or in its WSL, on this machine or on another one over SSH. Safe permits windows the session opened and forbids display reads. YOLO also permits existing windows the task needs; it is not OS isolation.
@@ -60,7 +60,7 @@ Results show addresses in a short form, `computer:<machine>/<letter><number><tag
Search `computer:machine/<name>/windows` or `/apps` with a title substring or `regex:<pattern>`; sandbox addresses take the same queries. Discovery launches nothing. Search an observation to filter its UI or page lines without another walk.
-Read a window for its UI tree and a small overview image. Overviews have a 1280 px long edge by default; add `capture:{"maxLongEdge":...}` to one Read for another size, and `/computer-debug` changes the session default. A fresh Read issues an observation address. Read a display for readonly display pixels (YOLO).
+Read a window for its UI tree and a small overview image. Overviews have a 1280 px long edge by default; add `capture:{"maxLongEdge":...}` to one Read for another size, and `/computer-images` changes the session default. A fresh Read issues an observation address. Read a display for readonly display pixels (YOLO).
| View | Meaning |
| --- | --- |
diff --git a/docs/image-context.md b/docs/image-context.md
index 0b9c5c5..5c35fce 100644
--- a/docs/image-context.md
+++ b/docs/image-context.md
@@ -4,9 +4,9 @@ This extension never removes or replaces older image blocks, rewrites text dumps
Pi's normal compaction may summarize older context and keep recent messages. It does not erase the original session entries. Reload alone does not remove images. This extension does not change compaction settings or provider cache controls. Other extensions and provider processing can still affect final delivery.
-## Debug cards are separate
+## Previews are separate
-Vision debug keeps three recent application-boundary cards and bounded action/request metadata. That is a viewer memory limit, not a model-context limit. Its request summary counts all observed image blocks. There is no image-budget footer or active/omitted projection. Private session history and debug pixels must not be shared.
+[Native previews](native-observability.md) show a bounded final operation image. Their lifetime does not limit model context, and they do not prove downstream provider delivery. Computer use keeps no debugger-only image mirrors or separate viewer archive. Private session history and operational captures must not be shared.
## Verification
diff --git a/docs/native-observability.md b/docs/native-observability.md
new file mode 100644
index 0000000..99fec8c
--- /dev/null
+++ b/docs/native-observability.md
@@ -0,0 +1,47 @@
+# Native computer visuals
+
+`/computer-visuals` controls human-only presentation on the machine that executes computer use. Native visuals require a driver advertising both `set_agent_visuals` and `present_agent_activity`. The adapter uses that driver's renderer and theme. It starts no viewer, HTTP listener or separate overlay process.
+
+A driver without these capabilities keeps the legacy cursor behavior described in [complete input](complete-input.md). A version number alone does not enable native presentation.
+
+## Human controls
+
+Run `/computer-visuals` to show the current settings, or choose a preset:
+
+- `off` disables presentation and discards pending previews.
+- `subtle` keeps activity cues and disables previews by default.
+- `full` enables previews by default.
+
+A JSON patch controls individual categories and rendering preferences:
+
+```text
+/computer-visuals {"observations":true,"previews":false,"pointer":true,"scroll_drag":true,"keyboard":true,"opacity":0.8,"quality":"balanced","reduced_motion":false}
+```
+
+`opacity` is 0..1. `quality` is `performance`, `balanced` or `high`. Preferences are saved on the Pi session branch; navigating to another branch restores its preferences. These commands belong to the human. They do not change Safe/YOLO, capture sizing, input delivery or permission checks.
+
+`/computer-images` controls future captures in this session: `cap 64..4096`, `scale 0.05..1`, `hints on|off`, `guides off|grid|action|all`, or `reset`. Calling it without arguments shows the settings. An individual Read can still override its capture options. Existing images keep their bytes.
+
+## What a cue means
+
+An overview identifies a captured window or display. A detail cue marks a requested physical region only when the adapter has a proven transform. Inspection and browser cues associate the operation with its native window; a Playwright CSS box does not establish a desktop rectangle. Element inspection uses native element bounds when available.
+
+Action cues can report `delivered`, `refused` or `unknown`. `delivered` means input was sent, not that the application changed. Only a completed condition check can report `verified`. Quiet verification samples and background watch frames do not each create another cue.
+
+Optional previews use a final PNG/JPEG returned by the operation, after cropping, resizing and annotation. A `changes` Read previews a returned change crop. A selected-frame batch admits at most its last returned image; a contact sheet or composite admits its final image. Capturing a sequence without returning images produces no preview. No extra capture or inference runs for presentation.
+
+A preview does not establish that arbitrary Codemode forwarded that image or that a remote model received it. [Normal Pi image context](image-context.md) owns image retention and provider boundaries. Preview images are not archived or written to disk by presentation.
+
+## Safety contract
+
+Native presentation must be input-transparent, non-activating and protected from capture. It must not move the operating system cursor merely to animate. Input execution and visual playback have separate clocks: rendering and visual travel must not delay dispatch, while intentional typing, drag and system input timing stays intact.
+
+Window events carry the existing HWND/PID lifetime identity and physical source bounds. The native renderer must revalidate them and hide stale drawings after movement, resize, minimize, closure or handle reuse. Selected displays use their exact native monitor bounds, including negative desktop origins. Ambiguous geometry never becomes a guessed rectangle or point.
+
+The adapter admits one native request at a time and retains one replaceable pending activity. Pending activity expires after 250 ms; preview transport accepts only bounded PNG/JPEG payloads. Cancellation and mode changes revoke older generations. A cancellation of an old operation cannot clear a newer one. Capture companions disable their own presentation.
+
+`highlight.status: queued` records adapter admission, not proof that a person saw pixels. An unavailable renderer does not repeat capture or input. Backend refusal and capture exclusion still need qualification on the exact Windows runtime; the adapter alone cannot prove focus, transparency, screenshot exclusion or visual quality.
+
+## Runtime qualification
+
+Use the exact source and file hashes in [the runtime pins](bundled-runtime.md). Before shipping a native visual runtime, check a real Windows desktop for focus and hit testing, capture exclusion, source changes, multi-monitor geometry, concurrent sessions, reduced motion, cancellation and cleanup. Measure input latency independently of animation and bound idle/active resource use. Keep private desktop recordings in ignored local artifacts.
diff --git a/docs/selected-displays.md b/docs/selected-displays.md
index 1514a01..0f9411b 100644
--- a/docs/selected-displays.md
+++ b/docs/selected-displays.md
@@ -26,4 +26,4 @@ Selected source metadata reports `selected_display` or `selected_display_region`
Safe denies discovery, overview and detail. Revoked or cancelled reads withhold metadata/pixels. Paint freshness remains unverified. No selected-display input, window relocation, virtual-desktop management or concurrent-agent arbitration is added.
-The installed optional human-only Qt overlay supports primary and windows, not selected monitors. Selected reads clear its old mark instead of drawing a misleading primary outline. This does not change the returned image or debug viewer.
+[Native visual events](native-observability.md) carry the selected monitor's exact host-global bounds and display identity. A renderer without those capabilities shows no selected-display cue; the returned image is unchanged.
diff --git a/docs/vision-debug.md b/docs/vision-debug.md
deleted file mode 100644
index cd4f052..0000000
--- a/docs/vision-debug.md
+++ /dev/null
@@ -1,45 +0,0 @@
-# Vision debug viewer
-
-`/computer-debug` opens a local observation viewer for the current Pi session. [The native CUA route](computer-resources.md) supplies exact PNG/JPEG captures and cap/scale controls. The viewer uses existing extension hooks, not a provider override, a second JavaScript executor or a new model-facing tool.
-
-## Open and stop
-
-The extension starts the viewer on `session_start` and prints its URL. `/computer-debug status` shows it again. `/computer-debug off` closes the server and clears retained pixels; `/computer-debug on` starts a fresh viewer.
-
-The URL has a random per-server path. The server listens on `127.0.0.1` only, rejects foreign Host/Origin headers. Reads expose the trace and random reading samples. Same-origin, bounded JSON writes apply an image-settings tuple or grade one reading trial and start the next. Expected text stays out of the HTML/JSON. Do not publish or forward it. A process with access to this session's URL can read its retained UI data and change image resizing settings; this is not an OS sandbox. There is no persistent service or machine configuration.
-
-Closing/reloading the Pi session closes the server. A new session has a new viewer URL and no previous-session frames. Reopening the same URL while its server is running reconnects to current state. After shutdown, the old URL is unavailable.
-
-## What the panel proves
-
-The viewer uses only Pi's existing API. `before_provider_request` provides a mutable application-boundary payload. The viewer copies image bytes from that event and hashes the decoded bytes with SHA-256. It also keeps bounded text excerpts and the payload paths where they appeared. It never mutates the payload, filters model history or fetches remote images.
-
-**Final delivery is unconfirmed.** Later extension handlers can replace the payload. Codex WebSocket can derive a delta request and use server-side context. Provider-internal resize and embeddings are not available. A provider response is recorded, but does not prove the original hook payload was sent unchanged. Frames remain `prepared`, not `sent`.
-
-The three displayed images are a viewer limit, not an image budget. Existing context can contain more images.
-
-## Images, structure and actions
-
-- The boundary-image cards show time, decoded byte count, PNG dimensions when known, hash, detail when provided, request ID and payload path.
-- Browser/Windows tool hooks record adapter images and metadata separately. A capture not forwarded by Codemode stays in the adapter trace and does not become a new boundary-image card.
-- Image hashes link pixels to matching computer-use actions. Identical captures are ambiguous: the panel lists candidate calls rather than assigning a unique source.
-- Adapter structure is labelled separately from observed provider text. Metadata in tool `details` is not automatically model input. Text excerpts are labelled when truncated.
-- Native window captures can carry snapshot hints added above the returned PNG. When annotation changes pixels, the private comparison retains the exact raw returned PNG with its hash. This is not a full-resolution original or a second capture. Missing or evicted comparison bytes remain unavailable.
-- The UI hints switch affects later captures only; per-capture overrides are recorded. Stored sent images are not rewritten. A boundary card with available raw bytes can switch between exact marked and raw pixels.
-- Requested settings and known adapter timings are recorded. Effective settings not reported by the adapter remain unknown. The viewer cannot change Safe/YOLO; its only controls are capture size settings.
-
-## Bounds and cleanup
-
-The in-memory trace retains 100 computer-use action records, 20 request summaries, three boundary frames and three recent adapter captures. Each retained image is limited to 12 MiB; the total asset budget is 48 MiB. Over-budget assets are evicted explicitly. Text and metadata are bounded, and the JSON state endpoint contains no image base64. Old source bytes are not promised after eviction.
-
-The panel polls current state once per second. That refresh interval is not an input or provider-latency measurement. Action durations come from the hook timestamps, while capture/input stage timings come from the existing adapter when available. Concurrent nested calls remain distinct by their call IDs and parent IDs.
-
-## Verification
-
-`npm test -- --run test/vision-debug.test.ts` checks byte identity, captured/prepared separation, ambiguous provenance, bounded viewer history, session reset, host/origin rejection, reconnect and cleanup.
-
-`npm run test:integration -- --run test/vision-extension.integration.test.ts` runs real Pi with its native OpenAI-compatible provider adapter against a loopback SSE fixture. It compares a boundary image hash with the bytes actually received by that fixture, exercises nested Codemode calls, deliberately does not forward one capture, records an input refusal and verifies server shutdown. It does not use real credentials or an external API.
-
-The integration writes `vision-trace.json` and `provider-bodies.json` beside its normal run artifacts. A scripted test provider that skips `onPayload` cannot qualify this boundary; use the native-adapter fixture instead.
-
-Action rows keep their expanded state during polling.
diff --git a/docs/window-overlay.md b/docs/window-overlay.md
deleted file mode 100644
index 9fea668..0000000
--- a/docs/window-overlay.md
+++ /dev/null
@@ -1,55 +0,0 @@
-# Human observation and target marks
-
-The Windows Qt helper flashes a cyan outline for a successful whole-window or primary-display overview, an amber rectangle for the requested zoom/crop area, and a pink rectangle or point for a grounded action target. Marks last two seconds; a newer mark replaces the old one. These are human-only drawings, not annotations in the model's image. Region support requires helper protocol 2; an older helper is refused instead of silently drawing the wrong whole-window frame.
-
-The frame is a human aid, not permission, model annotation, or evidence that the screenshot has current paint. CUA still owns capture and input. The helper cannot click, type, capture, activate a source window, or choose a target from a model argument. Browser capture still has its separately approved CUA foreground behavior.
-
-## Identity and geometry
-
-The adapter supplies its existing window lifetime marker and the successful capture's native bounds through a private pipe. The helper checks the HWND, PID and property marker. It matches the visible DWM bounds, not `GetWindowRect`'s invisible resize borders. It watches both rectangles and hides after a move, resize, minimize, close or lifetime change. It does not follow a stale window.
-
-The helper refuses to show if foreground or keyboard focus changes during its show operation, or if Windows capture exclusion is unavailable. `WindowTransparentForInput`, no-activate styles and native hit testing make the frame input-transparent. These flags are not a substitute for the live input check.
-
-Window coordinates remain in the CUA physical-pixel domain. Qt positions its own per-monitor-aware HWND with native physical coordinates; it does not turn image pixels into desktop input coordinates. Other DPI setups and Windows builds still need qualification.
-
-Qt may round its own transparent canvas outwards at fractional DPI. The helper allows only that bounded canvas overscan, maps drawing against the actual HWND extent and clips it to the exact source rectangle. Source identity and geometry checks, and CUA input coordinates, remain unchanged.
-
-Window zoom outlines the requested rectangle, not CUA's extra padding. Cua Driver does not publish the padded rectangle. The adapter recovers the window zoom transform only when native AX frames and their rounded screenshot frames prove one unique native width. Missing or ambiguous geometry clears the old mark and reports unavailable; it never guesses a ratio. Display crop uses the exact region of the fresh full-resolution capture.
-
-AX action targets use the observed element's native bounds, clipped to the window. Pixel clicks use the resolved physical point from a recognized successful CUA receipt, after the one dispatch. Receipts without coordinates produce no point. Other ungrounded keyboard targets are not invented. Drawing failure does not change or repeat input. The helper watches primary-display geometry and hides a display mark if it changes.
-
-## Lifecycle and failures
-
-No helper starts at extension registration, a policy restore, an unused close, or a refused read. The first authorized observation with valid drawing geometry starts one owned process. Permission changes clear the frame, and cancellation cannot hide a newer observation's frame. Both runtime close tools and Pi shutdown close this helper through pipe EOF. Confirmed close permits a fresh helper later; unknown shutdown remains blocked until reload.
-
-A missing or failed helper reports `highlight.status: unavailable` without changing image bytes or repeating capture/input. `shown` is an acknowledgement, not a promise that a person saw the short flash. The adapter rechecks cancellation and desktop permissions after the callback before publishing pixels. Native frame refusal hides the previous frame.
-
-## Build and local deployment
-
-Only Windows x64 is implemented and qualified here. Qt's cross-platform API does not establish macOS/Linux support.
-
-Use the installed Qt 6.11.3 MSVC2022 x64 qtbase kit and existing Visual Studio 2022 Build Tools:
-
-```powershell
-powershell -NoProfile -File scripts/build-window-overlay.ps1 -QtPrefix <Qt-kit-directory>
-```
-
-The build stages sources into the user-local `PiComputerUsePoC/.tmp` directory to avoid MSBuild case-folding problems on WSL's case-sensitive share. Deployment contains the helper, `qt.conf`, Qt6Core, Qt6Gui, Qt6Widgets and `platforms/qwindows.dll`. There is no WebEngine, QML or Python runtime. This build relies on the host's existing MSVC redistributable, Windows ICU and system DLLs; it is not a clean-machine installer.
-
-The package generator takes the built bundle, kit, exact Qt source checkout and a new output directory:
-
-```sh
-node scripts/package-window-overlay.mjs <bundle> <Qt-kit> <Qt-source> .tmp/qt-overlay/release
-```
-
-It verifies the exact SBOM's LGPL options, keeps license texts/attributions, supplies the complete matching qtbase source archive and helper source, and records hashes and separate runtime/source costs. See [the Qt notice](../native/window-overlay/NOTICE.md). Public binary redistribution requires its own licensing review; dynamic linking alone is not enough.
-
-Install the runtime files in `%LOCALAPPDATA%/Programs/PiComputerUsePoC/window-overlay-0.1.0`, preserving the `platforms` and `licenses` directories. Keep the source sidecar available with the deployment. Discovery refuses ambiguous installations. `PI_WINDOW_OVERLAY_PATH` is an optional trusted executable override, not a model argument. Installation never runs at Pi startup.
-
-## Verification
-
-Read-only detail keeps its original local region and scale, clips smaller bounds and positions a new mark against current bounds. Hiding an old drawing after geometry changes does not invalidate the window or its next detail read.
-
-Unit contracts cover lazy startup, generations, cancellation, mode restoration, ownership, failure isolation and confirmed/unknown cleanup. The real-Pi browser contract checks unchanged PNG/JPEG delivery and frame acknowledgements without exposing private lifetime markers. Pipe tests verify EOF and malformed replies without request replay.
-
-Check overlay bounds, focus, expiry, capture exclusion and cleanup live in a Pi session on newly opened disposable windows. Private full-display pixels stay in ignored local artifacts, never Git or Linear uploads.
\ No newline at end of file