Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.
TypeScript accessibilityagent-toolsai-agentbrowser-automationcomputer-usecuadesktop-automationelectronpipi-coding-agentpi-extensionpi-packageplaywrighttypescriptui-automationvisionwebview2windowswindows-automationwsl
docs, tests: typed input tools in the guide, cluster query and refusals (LPT-927)
alexshpunt-clanker[bot]c492bf8cf3c2
diff --git a/docs/computer-resources.md b/docs/computer-resources.mdindex 8fbc92c..678d99c 100644--- a/docs/computer-resources.md+++ b/docs/computer-resources.md@@ -11,7 +11,7 @@ The current desktop scope is Windows, with Pi running on it directly or in its W## Typed tools-Some operations have their own tool with typed fields, in both modes. They are deferred like the tools above. `tool_search` loads at most 8 best matches per query, so search one group at a time: `computer open window`, `computer wait browser`, `computer files clipboard record`. Their receipts match Insert receipts.+Some operations have their own tool with typed fields, in both modes. They are deferred like the tools above. `tool_search` loads at most 8 best matches per query, so search one group at a time: `computer open window`, `computer wait browser`, `computer files clipboard record`, `computer click type press drag`. Their receipts match Insert receipts.| Tool | Does | Replaces || --- | --- | --- |@@ -23,10 +23,30 @@ Some operations have their own tool with typed fields, in both modes. They are d| `computer_files` | Copies a workspace file into the session's folder on a machine (`put`), copies one out (`get`), or lists the folder (`list`) | Insert `put`, `get`, `files` into a machine || `computer_clipboard` | Reads (`include_image` adds the image) or replaces a machine's clipboard with text or an image, in YOLO | Insert `clipboard_read`, `clipboard_write` || `computer_record` | Records the main display to video (`video_start`, `video_stop`) or playback audio (`audio`) on a machine in YOLO; captures frames of a window or display (`sequence`) and returns a sequence address for Read with a `frames:` view | Insert `record_start`, `record_stop`, `audio_record`; Read `sequence:` |+| `computer_click` | Clicks an element or a point with any button, double or triple, with modifier keys | Insert `click`, `double_click`, `right_click` |+| `computer_hover` | Rests the pointer on an element or a point | Insert `hover`, `move` |+| `computer_drag` | Presses at one element or point, moves through optional `via` points and releases at another | Insert `drag` |+| `computer_scroll` | Turns the wheel over an element, a point or the middle of the window | Insert `scroll` |+| `computer_type` | Types text into a field or the focused element; `replace` sets the whole value, `submit` presses Enter after it | Write, Insert `type_text`, `set_value`, `fill`, `type` |+| `computer_press` | Presses a key or a chord such as `ctrl+s`, or holds keys for `hold_ms` | Insert `key`, `hotkey`, `hold`, `press` |+| `computer_select` | Chooses options of a list, sets a checkbox, radio or toggle (`checked`), or invokes a menu item by its path | Insert `select`, `check`, `uncheck`, `menu` |+| `computer_timeline` | Plays timed key, button, pointer, wheel and pen events | Insert `timeline` || Delete (`computer_delete` without IDE) | Closes a window like its close button, or releases a sandbox, isolated session, machine or `computer:session` | Insert `close` and `kill` |Delete of a window asks it to close: a browser the session opened closes with its disposable profile, an attached browser is detached first, and any other window gets the standard close request. The receipt says whether the window closed; an app that asks about unsaved work keeps it open, so Read it next. Add `?force=true` to the window address to kill the hung app of a window this session opened. Safe mode closes only windows the session opened.+### Input tools++The input tools take a window address, never an observation. The window decides how the input goes: a browser opened or attached by computer use runs it through Playwright on its active page, any other window through the native driver and UI Automation. Give a `target` or a point (`x`, `y`):++- A ref from the window's latest Read: a native ref such as `1:a` from a window Read (`aria`, `ui` or `full`), or a page ref such as `e12` from a DOM snapshot. Input spends native refs, because the result's successor observation brings new ones.+- A Playwright locator such as `getByRole('button', { name: 'Save' })`, resolved with auto-waiting like the [native locators](#native-locators) or by Playwright on a page; `timeout_ms` bounds the wait.+- A point. On a native window it is in pixels of the latest image of that window: a Read with an image, or the successor of an action. Without one, the tool captures the window at the default overview size first, about 300 ms once. On a browser window a point is in CSS pixels of the page; add `space: "window"` for pixels of a window Read, for example the browser's own toolbar.++Native input starts in the background. When the window refuses it before anything was sent, the same call sends it in the foreground and says so; later input to that window goes in the foreground at once. `foreground: true` goes there first, `foreground: false` never does. A foreground action activates the window and moves the real pointer. `computer_hover` and `computer_timeline` move the real pointer on native windows, so they always run in the foreground. Pen eraser and barrel, the `x1`/`x2` buttons and `move_relative` exist only natively; on a browser page give `space: "window"`. Pen pressure and tilt on a page reach it as a `pen` pointer through CDP in Chrome and Edge.++A native Read and a DOM Read of the same browser window keep each other's observations; input retires both.+The JSON commands below keep working while the remaining operations move to their own tools.## Discover and readdiff --git a/test/extension-modes.integration.test.ts b/test/extension-modes.integration.test.tsindex 1218112..5f05075 100644--- a/test/extension-modes.integration.test.ts+++ b/test/extension-modes.integration.test.ts@@ -37,17 +37,23 @@ test("without Pi Agent IDE, tool_search finds the deferred computer tools and th{ id: "files", name: "tool_search", arguments: { query: "computer files clipboard record" } },{ id: "list", name: "computer_files", arguments: { machine: "computer:session", action: "list" } },{ id: "sequence", name: "computer_record", arguments: { at: "computer:machine/local/display/primary", action: "sequence" } },+ { id: "input", name: "tool_search", arguments: { query: "computer click type press drag" } },+ { id: "click", name: "computer_click", arguments: { window: "computer:machine/local", target: "1:a" } },{ id: "delete", name: "computer_delete", arguments: { path: "computer:session" } },]);- expect(getToolCallNames(result)).toEqual(["tool_search", "computer_read", "computer_read", "computer_insert", "tool_search", "tool_search", "computer_display", "tool_search", "computer_files", "computer_record", "computer_delete"]);+ expect(getToolCallNames(result)).toEqual(["tool_search", "computer_read", "computer_read", "computer_insert", "tool_search", "tool_search", "computer_display", "tool_search", "computer_files", "computer_record", "tool_search", "computer_click", "computer_delete"]);for (const id of ["find", "guide", "session", "close", "delete"])expect(getToolExecution(result, id).isError, getToolResultText(result, id)).toBe(false);expect(getToolResultText(result, "find")).toContain("computer_read");// tool_search loads the best matches of each query, at most 8 by default, so the guide names one query per cluster.// A query lists only tools it newly loads; together the guide's cluster queries load every computer tool.- const loaded = ["find", "open", "more", "files"].map(id => getToolResultText(result, id)).join("\n");+ const loaded = ["find", "open", "more", "files", "input"].map(id => getToolResultText(result, id)).join("\n");for (const tool of ["computer_search", "computer_insert", "computer_write", "computer_delete", "computer_open", "computer_window", "computer_display",- "computer_wait", "computer_browser", "computer_files", "computer_clipboard", "computer_record"]) expect(loaded).toContain(`- ${tool}:`);+ "computer_wait", "computer_browser", "computer_files", "computer_clipboard", "computer_record", "computer_click", "computer_hover", "computer_drag",+ "computer_scroll", "computer_type", "computer_press", "computer_select", "computer_timeline"]) expect(loaded).toContain(`- ${tool}:`);+ // Input goes to a window address and refuses anything else before native work.+ expect(getToolExecution(result, "click").isError).toBe(true);+ expect(getToolResultText(result, "click")).toContain("Give the window address");// Files, clipboard and recording run on a machine address, and a sequence needs its length; both refuse before native work.expect(getToolResultText(result, "list")).toContain("Give a machine address");expect(getToolResultText(result, "sequence")).toContain("sequence needs duration_ms.");@@ -76,6 +82,10 @@ test("with Pi Agent IDE, computer use extends Read, Search, Insert and Delete anfor (const extensions of [[ide, computerUse, "builtin:tool-search"], [computerUse, ide, "builtin:tool-search"]]) {const result = await run("with-ide", extensions, ["read", "insert", "delete", "tool_search"], [{ id: "find", name: "tool_search", arguments: { query: "computer read windows displays" } },+ { id: "open", name: "tool_search", arguments: { query: "computer open window" } },+ { id: "more", name: "tool_search", arguments: { query: "computer wait browser" } },+ { id: "files", name: "tool_search", arguments: { query: "computer files clipboard record" } },+ { id: "input", name: "tool_search", arguments: { query: "computer click type press drag" } },{ id: "session", name: "read", arguments: { path: "computer:session" } },{ id: "close", name: "insert", arguments: { path: "computer:session", text: '{"action":"close"}' } },{ id: "force", name: "delete", arguments: { path: "computer:machine/local?force=true" } },@@ -83,9 +93,12 @@ test("with Pi Agent IDE, computer use extends Read, Search, Insert and Delete an]);// The force option reaches computer use through the IDE's resolver, which then refuses it for a machine.expect(getToolResultText(result, "force")).toContain("?force=true applies to window addresses only.");- expect(getToolResultText(result, "find")).not.toContain("computer_read");- expect(getToolResultText(result, "find")).not.toContain("computer_delete");- for (const tool of ["computer_open", "computer_window", "computer_display", "computer_wait", "computer_browser"]) expect(getToolResultText(result, "find")).toContain(tool);+ // The IDE's own tools read and delete computer resources; the cluster queries load every typed tool.+ const loaded = ["find", "open", "more", "files", "input"].map(id => getToolResultText(result, id)).join("\n");+ expect(loaded).not.toContain("computer_read");+ expect(loaded).not.toContain("computer_delete");+ for (const tool of ["computer_open", "computer_window", "computer_display", "computer_wait", "computer_browser", "computer_files", "computer_clipboard", "computer_record",+ "computer_click", "computer_hover", "computer_drag", "computer_scroll", "computer_type", "computer_press", "computer_select", "computer_timeline"]) expect(loaded).toContain(`- ${tool}:`);for (const id of ["session", "close", "delete"])expect(getToolExecution(result, id).isError, getToolResultText(result, id)).toBe(false);expect(getToolResultText(result, "session")).toContain("computer:session");diff --git a/test/fixtures/window-management-extension.ts b/test/fixtures/window-management-extension.tsindex 6d39fed..96ac01c 100644--- a/test/fixtures/window-management-extension.ts+++ b/test/fixtures/window-management-extension.ts@@ -201,6 +201,14 @@ export default async function fixture(pi: ExtensionAPI): Promise<void> {await denied("computer_clipboard", { machine: "computer:machine/local/windows", action: "read" });await denied("computer_record", { at: first, action: "video_start", seconds: 5 });await denied("computer_record", { at: "computer:machine/local", action: "sequence", duration_ms: 100 });+ // Input tools check the address, the target and the fields before anything reaches the window.+ await denied("computer_click", { window: "computer:machine/local", target: "1:a" });+ await denied("computer_click", { window: first, target: "e3" });+ await denied("computer_click", { window: first, target: "9:z" });+ await denied("computer_click", { window: first, x: 5 });+ await denied("computer_type", { window: first, text: "x", replace: true });+ await denied("computer_press", { window: first, key: "hyper+a" });+ await denied("computer_select", { window: first, target: "1:a", checked: true, values: ["a"] });const external = reference(await ok("search", { path: "computer:machine/local/windows", query: "Unowned management fixture" }), "window");mode = "safe";const refused = await window({ action: "focus" }, external);diff --git a/test/unified-tools.integration.test.ts b/test/unified-tools.integration.test.tsindex 6b4d26d..451e9be 100644--- a/test/unified-tools.integration.test.ts+++ b/test/unified-tools.integration.test.ts@@ -30,7 +30,8 @@ test("production Pi exposes only computer tools, with no direct or Codemode brow],}).run("Check the production loadout. Only discover apps; never open a window or send input.");// Without Pi Agent IDE, production computer use offers deferred tools that tool_search loads.- const standalone = ["computer_browser", "computer_clipboard", "computer_delete", "computer_display", "computer_files", "computer_insert", "computer_open", "computer_read", "computer_record", "computer_search", "computer_wait", "computer_window", "computer_write"];+ const standalone = ["computer_browser", "computer_click", "computer_clipboard", "computer_delete", "computer_display", "computer_drag", "computer_files", "computer_hover", "computer_insert", "computer_open",+ "computer_press", "computer_read", "computer_record", "computer_scroll", "computer_search", "computer_select", "computer_timeline", "computer_type", "computer_wait", "computer_window", "computer_write"];expect(getToolExecutionDetails(getToolExecution(result, "registry"))).toEqual({names: standalone,exposure: Object.fromEntries(standalone.map(name => [name, "deferred"])),