Skip to content

Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.

Document agent browser profile modes (#134)

alexshpunt-clanker[bot]90a659206c82
diff --git a/.pi/skills/computer-use/SKILL.md b/.pi/skills/computer-use/SKILL.md
index a6fffa7..13b6444 100644
--- a/.pi/skills/computer-use/SKILL.md
+++ b/.pi/skills/computer-use/SKILL.md
@@ -13,7 +13,7 @@ Every machine is `computer:machine/<name>`: `local` is this one, and a name conf
Read the whole window first. Establish the app, dialogs, controls, layout and stopping point. `full` gives native UI and an overview image; `image` gives pixels only, `ui` current controls only, and `aria` the UI as a Playwright aria snapshot with refs. Refs such as `1:a` belong to that Read; duplicate labels stay separate, so inspect parent, role, enabled state and bounds. Partial-tree absence is unknown. Search the observation to refine collected UI without another walk; Read `element:inspect|scope|highlight,<ref>` for one element.
-For web pages, `computer_open` a Playwright browser (`browser`, `url`, and in YOLO `profile: "agent"` for the agent's persistent logged-in profile), or in YOLO attach to a running Chrome, Edge or embedded browser by giving `computer_open` its window address; see [Attach to a running browser](../../../docs/computer-resources.md#attach-to-a-running-browser). Read its window with `dom` for an aria snapshot with page refs such as `e12`. Act with the input tools on the window, and check the returned change list for the application's response. Use `computer_browser` for navigation, tabs, JavaScript dialogs, uploads, downloads and `evaluate`. A strict-mode refusal sent nothing; a timeout is unknown, so read again rather than replaying. Check the snapshot's `Log` lines to tell an application error from a refused or unknown action. The browser's toolbar, menus and system dialogs are native: give `space: "window"` for them.
+For web pages, `computer_open` a Playwright browser (`browser`, `url`, and `profile: "agent"` for the agent's persistent profile; in Safe the user enables and approves its debugging connection), or in YOLO attach to a running Chrome, Edge or embedded browser by giving `computer_open` its window address; see [Attach to a running browser](../../../docs/computer-resources.md#attach-to-a-running-browser). Read its window with `dom` for an aria snapshot with page refs such as `e12`. Act with the input tools on the window, and check the returned change list for the application's response. Use `computer_browser` for navigation, tabs, JavaScript dialogs, uploads, downloads and `evaluate`. A strict-mode refusal sent nothing; a timeout is unknown, so read again rather than replaying. Check the snapshot's `Log` lines to tell an application error from a refused or unknown action. The browser's toolbar, menus and system dialogs are native: give `space: "window"` for them.
When several agents must work at once, or the work needs foreground input without touching the human, `computer_open` an isolated session (640×480; start apps missing by name with `path`). When the work should not touch the human's desktop and no host window is needed, `computer_open` a Windows Sandbox and pass host folders only through `import` and `export`. Use the returned `computer:sandbox/...` address to Search guest windows and `computer_open` guest apps; Delete it when the work is done.
diff --git a/docs/computer-resources.md b/docs/computer-resources.md
index cd13bd9..bc8cead 100644
--- a/docs/computer-resources.md
+++ b/docs/computer-resources.md
@@ -191,7 +191,13 @@ Ways of working found in real sessions, per app.
Web pages run through [Playwright](https://playwright.dev/). The browser window itself, its toolbar, system dialogs and every other application stay native windows. Playwright runs in a small worker on Windows, so browsers need Node.js there: an installed one, or `/computer-setup browser`. When Pi runs on Windows, the worker uses Pi's own Node.
-`computer_open` with `browser` or `url` launches a visible browser with a disposable profile: Chrome by default, installed Edge, or Playwright's own Firefox (`/computer-setup firefox` once per machine and after updating this package; it cannot be attached to). `profile: "agent"` (YOLO) uses the agent's own persistent profile on that machine, so logins made there stay; it is never the user's profile, and one browser at a time uses it. Closing the computer session closes these browsers and deletes their disposable profiles.
+`computer_open` with `browser` or `url` launches a visible browser with a disposable profile: Chrome by default, installed Edge, or Playwright's own Firefox (`/computer-setup firefox` once per machine and after updating this package; it cannot be attached to). `profile: "agent"` uses the agent's separate persistent profile on that machine, not the user's default profile; logins and site data in it remain across sessions. Only one browser at a time can use each agent profile. Closing the computer session closes these browsers and deletes disposable profiles, not agent profiles.
+
+In **Safe**, an agent Chrome or Edge profile starts normally, without a command-line debugging endpoint. The person using the computer enables remote debugging at `chrome://inspect/#remote-debugging` or `edge://inspect/#remote-debugging`, attaches the opened window with `computer_open`, and approves the browser's connection request. The agent cannot send native input to this window before approval. A requested `url` is opened only after approval. The browser may remember the debugging setting between launches, but asks for approval for each connection. Safe agent profiles require a browser that supports this approval flow; Firefox agent profiles are YOLO-only.
+
+In **YOLO**, an agent Chrome or Edge profile starts with a nonzero local debugging port. Other local processes can use that port to control the browser and access signed-in sessions. Selecting Safe is refused while a YOLO agent browser is open or starting, or while native input is in flight. If cleanup cannot be confirmed, verify that the browser and any scheduled input have stopped before restarting Pi. The profile lock coordinates Pi launches; a separate launcher that does not use it can still race the check for an existing browser.
+
+A separate profile and a nonzero port do not guarantee sign-in: providers such as Google can still reject a browser they consider automated.
Each page snapshot lists the tabs (`Tab t1 (active): title | url | load state`), `Event:` lines for opened or closed tabs and dialogs, `Download` lines and `Log` lines for console errors, page errors, failed requests and HTTP errors since the previous snapshot. Input, snapshots and screenshots use only the active tab; a popup never becomes active by itself, so use `computer_browser` `switch_tab`. A JavaScript dialog stays open until `computer_browser` `dialog` answers it, and other page actions refuse until then. Downloads land in the browser's temporary directory, which closing the session deletes; copy what you need. `evaluate` returns its result as JSON, cut at 20,000 characters; reaching the time limit is `unknown`. A Playwright timeout or lost connection is `unknown`; read a fresh snapshot.
@@ -212,7 +218,7 @@ In YOLO, `computer_open` at a window address attaches Playwright to a Chromium-b
WebView2 apps take Chromium switches from their environment: `computer_open` with `path` and `debugging: true` starts one with debugging, and `webview2_debugging` turns it on for an app the user starts. That writes a per-exe WebView2 policy, so Windows shows a UAC prompt the person at that machine must approve; it applies from the app's next start until turned off.
-The result names the route and the active tab. When the window title does not identify one tab, no tab is active until `switch_tab`. Closing the session detaches and leaves the browser open. Safe refuses attaching. Firefox cannot be attached.
+The result names the route and the active tab. When the window title does not identify one tab, no tab is active until `switch_tab`. Closing the session detaches and leaves the browser open. Safe refuses attaching to an arbitrary existing browser; its own agent-profile window can be attached after browser approval as described above. Firefox cannot be attached.
## Work apart from the human's desktop