Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.
TypeScript accessibilityagent-toolsai-agentbrowser-automationcomputer-usecuadesktop-automationelectronpipi-coding-agentpi-extensionpi-packageplaywrighttypescriptui-automationvisionwebview2windowswindows-automationwsl
Computer
- Load the
computer_*tools withtool_search, one group per query:computer open window,computer wait browser,computer files clipboard record,computer click hover drag scroll,computer type press select timeline,computer view viewport. - The
/computer-*commands belong to the user: when a refusal names one, ask the user to run it. - In IDE mode, native discovery of a
computer_*action oride_capability_computerloads computer-address help into the shared tools. Use the loader for a resource-only workflow. The first associated Codemode call loads that help too;searchTools,describeToolanddescribeNamespaceleave it unloaded. - Ordinary file calls and a direct computer-address {read} do not activate schema help. A direct computer-address {read} still works and loads this guide.
Addresses
- A machine starts on its first use. An address belongs to the machine or sandbox that issued it, and every call on it runs there; nothing falls back to this machine.
- Short forms are
computer:<machine>/<letter><number><tag>:wwindow,oobservation,ssequence,aapp,bsandbox,ddisplay. The tag changes every Pi run, so an address from an earlier run refuses.
Look
- A sandbox address takes the same
/windowsand/appssearches as a machine. Discovery launches nothing. - {search} an observation to filter its lines without a new capture.
- An overview is 1280 px on the long edge by default.
offsetandlimitpage an observation's stored text without capturing again.- A UI walk cut by limits or time leaves missing elements unknown, not absent.
- OCR can be wrong. For small text, use a viewport or
capture:{"granularity":"image","maxLongEdge":3840}.
Input
- Input spends the window's native refs; its change list brings the new ones.
- On native windows, a ref or locator goes through UI Automation in the background, without the human's pointer or focus. When the app does not react to it, use a point instead.
- Drags,
computer_hoverandcomputer_timelineare real input: the window comes to the front.computer_selectand menus run through UI Automation. - Real input runs at once, even while the human uses the computer. Another agent's action on the same desktop finishes first.
human_stop: the human pressed Escape three times. Send no more input, tell the user what you were doing, and wait until they turn computer use on again.
Native locators run over the UI Automation tree:
getByRole— UIA control types as ARIA roles:Button→button,Edit→textbox,ListItem→listitemandoption.getByText,getByLabel,getByTitle— visible text, a control's label, UIA help text.getByTestId— UIA AutomationId..filter(),.nth(),.first(),.last(),.and(),.or()and chaining work as in Playwright. CSS and XPath refuse.
Waits and viewports
- After
computer_wait, the last sample's refs are current. - The change list after input on a viewport covers only the viewport.
- {read} of a viewport only looks: the window's latest observation and refs stay current.
- {delete} removes a viewport; it also goes away with its window.
- Maximize an app that resizes its own window before making viewports.
Displays
- Pick a resolution with the display's aspect ratio: a 3840×2560 display has no 1920×1280, and 2160×1440 is its closest smaller mode.
- The human sees the new resolution too: tell the user it stays until the computer session closes. Earlier display and window observations stop working; {read} again.
Browser pages
- Keys go to the page, so browser shortcuts such as
alt+ArrowLeftdo not run there: usecomputer_browser, orspace: "window". - A popup does not become the active tab by itself:
computer_browserswitch_tab. - Downloads land in a temporary folder that closing the session deletes; copy what you need.
- Providers such as Google can reject an automated browser even with
profile: "agent". - Closing the session detaches an attached browser and leaves it open.
computer_open at a window address attaches by one of these routes:
- debugging endpoint — a browser, Electron or CEF app started with
--remote-debugging-port=0inargs, or a WebView2 app with debugging on. Chrome and Edge need their own--user-data-dirfor this; Edge also--disable-sync. - remote debugging turned on at
chrome://inspect/#remote-debuggingoredge://inspect/#remote-debugging— the browser asks "Allow remote debugging?" and waits up to 2 minutes; the user, or the agent in another call, clicks Allow. - Playwright Extension — installed by the user; attaching waits up to 2 minutes for "Allow & select".
- VS Code's integrated browser — through
/computer-setup vscode-bridgeand a full VS Code restart.
Work apart from the human's desktop
- Windows Sandbox — its own focus, pointer, windows and clipboard; starts in 15–20 s; one at a time per machine, and the human's Windows session must be active. Networking is enabled. The guest has no Notepad.
- Sandbox runtime and control mappings are automatic and read-only; its private output mapping is writable.
sandbox.importis read-only unlesswritableis true, andsandbox.exportis writable. - Isolated session — invisible to the human; starts in 15–25 s; several can run at once. Its desktop is 640×480, some apps are missing by name (start them by
path), and no user folders are shared automatically. Its network policy allows ingress and egress. - Both use temporary control channels. Sandbox requires a driver bearer token and mutual identity proof; MXC reverse channels mutually prove their roles before forwarding application bytes. Their foreground UI stayed off the human desktop in verified Windows checks. Neither is qualified as hostile-code containment or protection from a local administrator.
- Another machine acts only there: a
urlsuch as127.0.0.1means that machine. Its Windows session must be active, or screenshots come back white. Remote control uses the existing SSH connection and temporary loopback forwards.
Files and recording
- Save app documents in the session folder of
computer_files, not in the user's folders. - An app with its own scripting can build content:
computer_filesputthe script,computer_openthe app with its path inargs, then {read} the window. - The clipboard is shared with the user's apps.
- A window
sequencespends the window's refs: {read} it again before input. Awatchkeeps them. - While a
watchruns, each action's receipt names its frames, andframes:lastshows the newest one.
Receipts in Codemode
Every computer_* tool resolves to a receipt object, also when it refused or its outcome is unknown, so a script branches without try/catch:
outcome—delivered(not proof the app responded),refused(nothing was sent),unknown(may have reached the target: {read} again, never replay),observedorclosed.effect—applied,not-appliedorunknown.stage—validationwhen the request was rejected before reaching the target, otherwisetarget.window,viewport,observation,sandbox— the addresses the call used or issued.reason,code— why a refused or unknown action ended.page,evaluation,verification,changes,steps,watch_spans— when the action produced them.text— the receipt the model sees.
Check outcome before the next input: after a refused click, keys go to whatever has focus. A script that clicks many points uses a viewport: an action's change-list image can have another size than the last {read}. store() of a script that later fails is lost.
Cleanup
Close only windows the task opened.