Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.
TypeScript accessibilityagent-toolsai-agentbrowser-automationcomputer-usecuadesktop-automationelectronpipi-coding-agentpi-extensionpi-packageplaywrighttypescriptui-automationvisionwebview2windowswindows-automationwsl
docs: the target decides the input route; no foreground parameter (LPT-942)
alexshpunt-clanker[bot]102db46d853a
diff --git a/.pi/skills/computer-use/SKILL.md b/.pi/skills/computer-use/SKILL.mdindex e131b11..a3b2aa4 100644--- a/.pi/skills/computer-use/SKILL.md+++ b/.pi/skills/computer-use/SKILL.md@@ -31,7 +31,7 @@ Know the target, expected effect and stopping point before input. The input tool- a ref from the window's latest Read;- a point in pixels of the latest image of the window, for visible controls without UI, canvases and games.-Reported disabled controls cannot act. Native input starts in the background and moves to the foreground by itself only after a refusal that sent nothing; the receipt says so. If foreground input refuses because the human is using the computer, wait the stated time; never race the human. Never replace a refusal with global input, another backend or a browser script.+Reported disabled controls cannot act. A ref or locator is pressed through UI Automation in the background; when the element has no pattern for the action the call uses the real pointer and says so. Points, drags, keys and typing are real input: the window comes to the front. If a UI Automation press is delivered but the app does not react, click the point instead. If real input refuses because the human is using the computer, wait the stated time; never race the human. Never replace a refusal with global input, another backend or a browser script.For timed input, a button or key held across moves, wheel during a drag, raw relative motion for pointer-locked apps or a pen with pressure and tilt, use one `computer_timeline`. Stop at navigation, a new dialog, changed layout or an effect requiring a new decision. Several known steps go into one Codemode script that checks each receipt. Stop on an `unknown` outcome or cancellation; never replay completed or uncertain input.diff --git a/README.md b/README.mdindex b4d37ce..75edf8f 100644--- a/README.md+++ b/README.md@@ -78,7 +78,7 @@ Input goes to a window by its address and targets an element by a ref from the l- run a timeline of timed key, button, mouse and pen events, with raw relative mouse motion for pointer-locked apps and pen pressure and tilt;- invoke, select or set values through UI Automation, and open menu items by their labels.-Input starts in the background where Windows allows it, so your own pointer and focus stay put. Foreground input is a separate step the agent has to ask for, and it pauses whenever you touch the keyboard or mouse.+What the agent sees as an element it presses through UI Automation in the background, so your pointer and focus stay put. What it sees only as pixels it clicks with the real mouse, and keys are real key presses: that input takes the front window, and it pauses whenever you touch the keyboard or mouse. Nothing is posted to windows as messages that an app could silently drop. To keep your desktop untouched, give the agent a sandbox or an isolated session.### Results and checksdiff --git a/docs/complete-input.md b/docs/complete-input.mdindex 762cf18..577e340 100644--- a/docs/complete-input.md+++ b/docs/complete-input.md@@ -21,7 +21,7 @@ Physical letter keys follow the active OS keyboard layout. Use `type_text` for lA relative drag endpoint is computed from the observed start: `to_x=from_x+dx`, `to_y=from_y+dy`. No global cursor or desktop offset is needed.-Background delivery is the default. Explicit foreground is allowed only after an observed failure/refusal on the authorized window. The adapter never retries or escalates automatically.+The typed tools choose delivery from the target (see [Act on windows](computer-resources.md#act-on-windows)): a click or scroll on an element goes through UI Automation with `uia_only`, and the driver refuses before sending anything when no pattern carries it; the tool then sends the same action as real input. Points, drags, keys and timelines are real input (`delivery_mode: "foreground"`). No route posts window messages.## Bounded sequences@@ -42,7 +42,7 @@ For computed drawing, choose points inside an actual observed canvas region. MapWith the bundled runtime, `drag` accepts `via`: 1..254 intermediate `{x,y}` points. The endpoints and every intermediate point use the same current window PNG. One call presses the left button, follows all points and releases it. Do not pass `steps`, modifiers, or right/middle button with `via`.-Background refuses before path input. Observe again after that refusal, then explicitly request foreground. It moves the real pointer; do not fight it during the stroke.+A path is real input: it moves the real pointer; do not fight it during the stroke.A partial path returns `unknown`, preserving native `delivery.delivered_count` and the press/release summary. These count point moves, not completed batch steps. Later steps stop. Never replay the whole stroke or its uncertain remainder. Check the app's actual pixels before choosing another action.@@ -54,7 +54,7 @@ With the bundled runtime, `hold` keeps 1..8 unique keys down for `duration_ms` 1Optional `actions` contains 1..32 complete left pixel clicks or drags from the same overview. Each drag may be straight or use `via`. Do not pass element tokens, per-action modifiers, button choices, key steps or nested holds. All requested pointer durations, including 50ms per click, must fit the hold interval. All keys and points are checked before input. Hold cannot nest in `sequence`.-Start with background. After its native refusal, take a new overview and explicitly use foreground. The real keyboard and pointer are in use during the action; do not fight them. Existing held keys/modifiers or mouse buttons cause a preflight refusal, not a release-all.+A hold is real input: the real keyboard and pointer are in use during the action; do not fight them. Existing held keys/modifiers or mouse buttons cause a preflight refusal, not a release-all.For Shift + brush, use `keys:["shift"]` and `actions:[{action:"drag",from_x:p.x,from_y:p.y,via:points,to_x:q.x,to_y:q.y,duration_ms:800}]` inside a hold of at least800ms. Points must come from the current overview. The native summary reports requested/measured time, inserted keys, completed pointer actions and press/release prefixes. Scheduling and input delivery add latency; timing is not real-time and OS auto-repeat is not promised.diff --git a/docs/computer-resources.md b/docs/computer-resources.mdindex 29766c3..d07ec1f 100644--- a/docs/computer-resources.md+++ b/docs/computer-resources.md@@ -84,9 +84,15 @@ The input tools take a window address, never an observation. The window decides- **A Playwright locator** such as `getByRole('button', { name: 'Save' })`, resolved with auto-waiting (`timeout_ms`, default 5000).- **A point** (`x`, `y`). On a native window it is in pixels of the latest image of that window, from a Read with an image or from an action's change list; without one the tool captures the window first, about 300 ms once. On a browser page a point is in CSS pixels; `space: "window"` uses pixels of a window Read, for example for the browser's own toolbar.-Native input starts in the background, without moving the human's pointer or focus. When the window refuses background input before anything was sent, the same call sends it in the foreground and says so, and later input to that window goes in the foreground at once. `foreground: true` goes there first; `foreground: false` never does. A foreground action activates the window and moves the real pointer. `computer_hover` and `computer_timeline` move the real pointer on native windows, so they always run in the foreground.+On native windows the target decides how input goes. What the agent sees structurally is pressed structurally, and what it sees only as pixels gets the real pointer:-While the human uses the keyboard or mouse, foreground input pauses: it refuses with the time since their last input and when to retry, until 10 s pass without it. Never try to outrun the human. When Windows refuses to activate the target (`foreground_unavailable`), the result names the process that holds the foreground; if it is a system service, for example after a UAC prompt, Windows blocks injected input until a person clicks or presses a key on that machine, so ask the user.+- A click or scroll on a **ref or locator** goes through the element's UI Automation pattern (invoke, toggle, select, expand, scroll) in the background, without moving the human's pointer or focus. When the element has no pattern for it, nothing is sent that way: the same call clicks or scrolls it with the real pointer at its centre and says so.+- **Points**, drags, `computer_hover` and `computer_timeline` get the real pointer, and `computer_press` and `computer_type` real key presses. The window comes to the front.+- `computer_type` with `replace`, `computer_select` and menus are UI Automation calls in the background.++No input is posted to a window as messages, so a `delivered` receipt never stands for input the app may have dropped. When a UI Automation call succeeds but the app does not react, give a point instead of the ref. Work that must not take the human's pointer or focus goes to a [Windows Sandbox or an isolated session](#work-apart-from-the-humans-desktop).++While the human uses the keyboard or mouse, real input pauses: it refuses with the time since their last input and when to retry, until 10 s pass without it. Never try to outrun the human. When Windows refuses to activate the target (`foreground_unavailable`), the result names the process that holds the foreground; if it is a system service, for example after a UAC prompt, Windows blocks injected input until a person clicks or presses a key on that machine, so ask the user.Keys on a browser page go to the page, so browser shortcuts such as `alt+ArrowLeft` do not run there; use `computer_browser` to go back, or `space: "window"` to press them natively. Pen eraser and barrel, the `x1`/`x2` buttons and `move_relative` exist only natively. Pen pressure and tilt reach apps as real pen input; on a page they arrive as a `pen` pointer in Chrome and Edge. Paint 11 ignores pen pressure; GIMP uses it.diff --git a/docs/setup.md b/docs/setup.mdindex d1a84c7..e24c58b 100644--- a/docs/setup.md+++ b/docs/setup.md@@ -30,7 +30,7 @@ Computer use works through `computer:` addresses: the ordinary Pi Agent IDE ReadOpening means a **new window**, not an existing window or new tab. Apps differ: some are single-instance or create several startup windows. If the result is `unknown`, discover what actually appeared before deciding. Never blindly relaunch. A new window is not necessarily an empty document: applications can restore prior state. Read tree and pixels before editing.-Background is the default. Foreground is explicit and only follows an observed background refusal or no-op. `delivered` does not prove the requested task is complete. Check fresh UI value and PNG. `unknown` may include a partial change; never replay without checking.+A click or scroll on an element goes through UI Automation in the background; points, drags and keys are real input in the foreground. `delivered` does not prove the requested task is complete. Check fresh UI value and PNG. `unknown` may include a partial change; never replay without checking.Clear or close only task-owned or explicitly authorized documents. Closing `computer:session` releases the runtime; it does not close application windows or save documents.