Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.
TypeScript accessibilityagent-toolsai-agentbrowser-automationcomputer-usecuadesktop-automationelectronpipi-coding-agentpi-extensionpi-packageplaywrighttypescriptui-automationvisionwebview2windowswindows-automationwsl
docs: tool_search loads computer_view with its own group, computer view viewport (LPT-931)
alexshpunt-clanker[bot]57575c3bc9a2
diff --git a/.pi/skills/computer-use/SKILL.md b/.pi/skills/computer-use/SKILL.mdindex bbd09eb..9716914 100644--- a/.pi/skills/computer-use/SKILL.md+++ b/.pi/skills/computer-use/SKILL.md@@ -5,7 +5,7 @@ description: Read and operate authorized Windows app windows, displays and brows# Computer use-Use **Search → Read → inspect → plan → act → check**. Browsers are ordinary app windows. Read [the resource guide](../../../docs/computer-resources.md) before using computer addresses; the same guide is available as `docs:computer`. Read only looks, Search only discovers, Delete closes and releases, and each action is a `computer_*` tool. Load the tools with `tool_search`, one group per query: `computer open window`, `computer wait browser`, `computer files clipboard record`, `computer click hover drag scroll`, `computer type press select timeline`. Without Pi Agent IDE, use `computer_read`, `computer_search` and `computer_delete` wherever this skill says Read, Search or Delete.+Use **Search → Read → inspect → plan → act → check**. Browsers are ordinary app windows. Read [the resource guide](../../../docs/computer-resources.md) before using computer addresses; the same guide is available as `docs:computer`. Read only looks, Search only discovers, Delete closes and releases, and each action is a `computer_*` tool. Load the tools with `tool_search`, one group per query: `computer open window`, `computer wait browser`, `computer files clipboard record`, `computer click hover drag scroll`, `computer type press select timeline`, `computer view viewport`. Without Pi Agent IDE, use `computer_read`, `computer_search` and `computer_delete` wherever this skill says Read, Search or Delete.## Observe and inspectdiff --git a/docs/computer-resources.md b/docs/computer-resources.mdindex e92a894..07a241e 100644--- a/docs/computer-resources.md+++ b/docs/computer-resources.md@@ -7,7 +7,7 @@ Computer use reads and operates Windows app windows, displays and browser pagesEverything else is a `computer_*` tool with typed fields, the same in both modes. Read only looks, Search only discovers, and the `computer_*` tools act. Write and Insert do not act on computer resources: they refuse and name the tool to use.-Every `computer_*` tool is deferred: a session that never uses the computer pays nothing for them. `tool_search` loads at most 8 best matches per query, so load one group at a time: `computer open window`, `computer wait browser`, `computer files clipboard record`, `computer click hover drag scroll`, `computer type press select timeline`. Computer use turns `tool_search` on.+Every `computer_*` tool is deferred: a session that never uses the computer pays nothing for them. `tool_search` loads at most 8 best matches per query, so load one group at a time: `computer open window`, `computer wait browser`, `computer files clipboard record`, `computer click hover drag scroll`, `computer type press select timeline`, `computer view viewport`. Computer use turns `tool_search` on.The `/computer-mode`, `/computer-debug` and `/computer-setup` commands belong to the user. When a part is missing, the refusal names the `/computer-setup` command that installs it; ask the user to run it.diff --git a/src/provider.ts b/src/provider.tsindex cb2bcfe..7358fef 100644--- a/src/provider.ts+++ b/src/provider.ts@@ -39,7 +39,7 @@ export function computerEditRefusal(tool: "write" | "insert", input: Record<strilet action: unknown = tool === "write" ? "type_text" : undefined;if (tool === "insert" && typeof input.text === "string") try { action = (JSON.parse(input.text) as { action?: unknown })?.action; } catch { /* no action to map */ }const replacement = typeof action === "string" ? REPLACED.find(([pattern]) => pattern.test(action as string))?.[1] : undefined;- return `${tool === "write" ? "Write" : "Insert"} does not act on computer resources; nothing was sent. ${replacement ? `Use ${replacement}. ` : ""}Each operation has its own computer_* tool: tool_search loads them with "computer open window", "computer click hover drag scroll", "computer type press select timeline", "computer wait browser" or "computer files clipboard record".`;+ return `${tool === "write" ? "Write" : "Insert"} does not act on computer resources; nothing was sent. ${replacement ? `Use ${replacement}. ` : ""}Each operation has its own computer_* tool: tool_search loads them with "computer open window", "computer click hover drag scroll", "computer type press select timeline", "computer wait browser", "computer files clipboard record" or "computer view viewport".`;}/** Present native resources without dumping provenance; full receipts remain in structured data. */diff --git a/test/extension-modes.integration.test.ts b/test/extension-modes.integration.test.tsindex 5fb5218..004e1b2 100644--- a/test/extension-modes.integration.test.ts+++ b/test/extension-modes.integration.test.ts@@ -38,12 +38,13 @@ test("without Pi Agent IDE, tool_search finds the deferred computer tools and th{ id: "more", name: "tool_search", arguments: { query: "computer wait browser" } },{ id: "display", name: "computer_display", arguments: { display: "computer:display/primary", width: 1920, height: 1080 } },{ id: "files", name: "tool_search", arguments: { query: "computer files clipboard record" } },+ { id: "view", name: "tool_search", arguments: { query: "computer view viewport" } },{ id: "list", name: "computer_files", arguments: { machine: "computer:session", action: "list" } },{ id: "sequence", name: "computer_record", arguments: { at: "computer:machine/local/display/primary", action: "sequence" } },{ id: "click", name: "computer_click", arguments: { window: "computer:machine/local", target: "1:a" } },{ id: "delete", name: "computer_delete", arguments: { path: "computer:session" } },]);- expect(getToolCallNames(result)).toEqual(["tool_search", "tool_search", "tool_search", "computer_read", "computer_read", "computer_delete", "tool_search", "tool_search", "computer_display", "tool_search", "computer_files", "computer_record", "computer_click", "computer_delete"]);+ expect(getToolCallNames(result)).toEqual(["tool_search", "tool_search", "tool_search", "computer_read", "computer_read", "computer_delete", "tool_search", "tool_search", "computer_display", "tool_search", "tool_search", "computer_files", "computer_record", "computer_click", "computer_delete"]);for (const tool of ["computer_click", "computer_hover", "computer_drag", "computer_scroll", "computer_type", "computer_press", "computer_select", "computer_timeline"])expect(["find", "input", "keys"].map(id => getToolResultText(result, id)).join("\n")).toContain(`- ${tool}:`);for (const id of ["find", "guide", "session", "close", "delete"])@@ -51,7 +52,7 @@ test("without Pi Agent IDE, tool_search finds the deferred computer tools and thexpect(getToolResultText(result, "find")).toContain("computer_read");// tool_search loads the best matches of each query, at most 8 by default, so the guide names one query per cluster.// A query lists only tools it newly loads; together the guide's cluster queries load every computer tool.- const loaded = ["find", "open", "more", "files", "input", "keys"].map(id => getToolResultText(result, id)).join("\n");+ const loaded = ["find", "open", "more", "files", "view", "input", "keys"].map(id => getToolResultText(result, id)).join("\n");for (const tool of ["computer_search", "computer_delete", "computer_open", "computer_window", "computer_view", "computer_display","computer_wait", "computer_browser", "computer_files", "computer_clipboard", "computer_record", "computer_click", "computer_hover", "computer_drag","computer_scroll", "computer_type", "computer_press", "computer_select", "computer_timeline"]) expect(loaded).toContain(`- ${tool}:`);@@ -89,6 +90,7 @@ test("with Pi Agent IDE, computer use extends Read, Search and Delete, refuses W{ id: "open", name: "tool_search", arguments: { query: "computer open window" } },{ id: "more", name: "tool_search", arguments: { query: "computer wait browser" } },{ id: "files", name: "tool_search", arguments: { query: "computer files clipboard record" } },+ { id: "view", name: "tool_search", arguments: { query: "computer view viewport" } },{ id: "input", name: "tool_search", arguments: { query: "computer click hover drag scroll" } },{ id: "keys", name: "tool_search", arguments: { query: "computer type press select timeline" } },{ id: "session", name: "read", arguments: { path: "computer:session" } },@@ -101,7 +103,7 @@ test("with Pi Agent IDE, computer use extends Read, Search and Delete, refuses W// The force option reaches computer use through the IDE's resolver, which then refuses it for a machine.expect(getToolResultText(result, "force")).toContain("?force=true applies to window addresses only.");// The IDE's own tools read and delete computer resources; the cluster queries load every typed tool.- const loaded = ["find", "open", "more", "files", "input", "keys"].map(id => getToolResultText(result, id)).join("\n");+ const loaded = ["find", "open", "more", "files", "view", "input", "keys"].map(id => getToolResultText(result, id)).join("\n");expect(loaded).not.toContain("computer_read");expect(loaded).not.toContain("computer_delete");for (const tool of ["computer_open", "computer_window", "computer_view", "computer_display", "computer_wait", "computer_browser", "computer_files", "computer_clipboard", "computer_record",