Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.
TypeScript accessibilityagent-toolsai-agentbrowser-automationcomputer-usecuadesktop-automationelectronpipi-coding-agentpi-extensionpi-packageplaywrighttypescriptui-automationvisionwebview2windowswindows-automationwsl
fix: chords go as hotkeys, the namespace points to the guide so tool_search ranks tools by their own text (LPT-927)
alexshpunt-clanker[bot]b41eef4a2fb9
diff --git a/docs/computer-resources.md b/docs/computer-resources.mdindex 678d99c..a167cb7 100644--- a/docs/computer-resources.md+++ b/docs/computer-resources.md@@ -11,7 +11,7 @@ The current desktop scope is Windows, with Pi running on it directly or in its W## Typed tools-Some operations have their own tool with typed fields, in both modes. They are deferred like the tools above. `tool_search` loads at most 8 best matches per query, so search one group at a time: `computer open window`, `computer wait browser`, `computer files clipboard record`, `computer click type press drag`. Their receipts match Insert receipts.+Some operations have their own tool with typed fields, in both modes. They are deferred like the tools above. `tool_search` loads at most 8 best matches per query, so search one group at a time: `computer open window`, `computer wait browser`, `computer files clipboard record`, `computer click hover drag scroll`, `computer type press select timeline`. Their receipts match Insert receipts.| Tool | Does | Replaces || --- | --- | --- |@@ -43,7 +43,7 @@ The input tools take a window address, never an observation. The window decides- A Playwright locator such as `getByRole('button', { name: 'Save' })`, resolved with auto-waiting like the [native locators](#native-locators) or by Playwright on a page; `timeout_ms` bounds the wait.- A point. On a native window it is in pixels of the latest image of that window: a Read with an image, or the successor of an action. Without one, the tool captures the window at the default overview size first, about 300 ms once. On a browser window a point is in CSS pixels of the page; add `space: "window"` for pixels of a window Read, for example the browser's own toolbar.-Native input starts in the background. When the window refuses it before anything was sent, the same call sends it in the foreground and says so; later input to that window goes in the foreground at once. `foreground: true` goes there first, `foreground: false` never does. A foreground action activates the window and moves the real pointer. `computer_hover` and `computer_timeline` move the real pointer on native windows, so they always run in the foreground. Pen eraser and barrel, the `x1`/`x2` buttons and `move_relative` exist only natively; on a browser page give `space: "window"`. Pen pressure and tilt on a page reach it as a `pen` pointer through CDP in Chrome and Edge.+Native input starts in the background. When the window refuses it before anything was sent, the same call sends it in the foreground and says so; later input to that window goes in the foreground at once. `foreground: true` goes there first, `foreground: false` never does. A foreground action activates the window and moves the real pointer. `computer_hover` and `computer_timeline` move the real pointer on native windows, so they always run in the foreground. Pen eraser and barrel, the `x1`/`x2` buttons and `move_relative` exist only natively; on a browser page give `space: "window"`. Pen pressure and tilt on a page reach it as a `pen` pointer through CDP in Chrome and Edge. Keys on a page go to the page, so browser shortcuts such as `alt+ArrowLeft` do not run there; use `computer_browser` to go back, or `space: "window"` to press them natively.A native Read and a DOM Read of the same browser window keep each other's observations; input retires both.diff --git a/src/computer-resources.ts b/src/computer-resources.tsindex f340006..1846770 100644--- a/src/computer-resources.ts+++ b/src/computer-resources.ts@@ -49,7 +49,7 @@ const BATCH_STEPS = 200;/** Native actions a batch may run; each later step re-resolves its target in a fresh UI walk. */const NATIVE_BATCH_ACTIONS = ["click", "double_click", "right_click", "type_text", "key", "hotkey", "scroll", "invoke", "select", "set_value"];/** Native actions that take a Playwright locator or JSON target. */-const NATIVE_LOCATOR_ACTIONS = ["click", "double_click", "right_click", "hover", "type_text", "key", "scroll", "invoke", "select", "set_value", "check", "uncheck"];+const NATIVE_LOCATOR_ACTIONS = ["click", "double_click", "right_click", "hover", "type_text", "key", "hotkey", "scroll", "invoke", "select", "set_value", "check", "uncheck"];/** A point in pixels of a window's latest image, or in CSS pixels of a browser page. */export interface InputPoint { x: number; y: number }/** One typed input on a window, built by a computer_* input tool; the window decides which backend runs it. */diff --git a/src/computer-tools.ts b/src/computer-tools.tsindex fa9cea8..b88c8f8 100644--- a/src/computer-tools.ts+++ b/src/computer-tools.ts@@ -10,9 +10,14 @@ import { COMPUTER_DELETE_HELP, computerAction, deleteComputer } from "./providerconst plain = (value: unknown) => String(value).replace(/[\x00-\x1f\x7f-\x9f]/g, " ");type Action = Awaited<ReturnType<typeof computerAction>>;-/** The namespace every computer_* tool shares; tool_search finds them by it, and its instructions are the guide. */-export function computerNamespace(guide: string) {- return { name: "computer", description: "Read and operate Windows app windows, displays and browser pages through computer: addresses", instructions: guide };+/**+ * The namespace every computer_* tool shares. tool_search indexes its instructions into every tool's search text,+ * so they point to the guide instead of holding it: the guide names every operation and would make all tools match+ * every query equally.+ */+export function computerNamespace() {+ return { name: "computer", description: "Read and operate Windows app windows, displays and browser pages through computer: addresses",+ instructions: "Read docs:computer for the guide: with Read in Pi Agent IDE, with computer_read without it." };}/** Codemode scripts receive the receipt object for every outcome, so they branch on outcome without try/catch. */export const receiptSchema = Type.Object({ outcome: Type.String(), effect: Type.String() }, { additionalProperties: true });@@ -399,7 +404,8 @@ export function pressPlan(params: Static<typeof pressParameters>): InputPlan {const key = parts.at(-1)!, held = parts.slice(0, -1).map(part => part.toLowerCase().replace(/^(control)$/, "ctrl").replace(/^(meta|windows|cmd)$/, "win"));if (held.some(part => !PAGE_MODIFIERS[part])) throw new ComputerRequestError("Modifiers are ctrl, shift, alt and win.");const native = params.hold_ms !== undefined ? { action: "hold", keys: [...held, key.toLowerCase()], duration_ms: params.hold_ms }- : { action: "key", key: key.toLowerCase(), ...(held.length ? { modifiers: held } : {}) };+ // A chord goes as a hotkey: posted key messages do not carry modifier state, so the app would see the bare key.+ : held.length ? { action: "hotkey", keys: [...held, key.toLowerCase()] } : { action: "key", key: key.toLowerCase() };const keys = [...held.map(part => PAGE_MODIFIERS[part]), pageKey(key)];const aimed = aim(params, "press", false);if (params.hold_ms !== undefined && aimed.target !== undefined) throw new ComputerRequestError("hold_ms goes to the focused element; focus it with computer_click first.");@@ -462,8 +468,8 @@ export function timelinePlan(params: Static<typeof timelineParameters>): InputPl* Register the typed computer_* tools that replace Insert commands one cluster at a time. They are deferred: a* tool_search for computer loads them, so sessions that never use the computer pay nothing for them.*/-export function registerComputerTools(pi: ExtensionAPI, resources: ComputerResources, guide: string): void {- const namespace = computerNamespace(guide);+export function registerComputerTools(pi: ExtensionAPI, resources: ComputerResources): void {+ const namespace = computerNamespace();const common = { exposure: "deferred" as const, namespace, outputSchema: receiptSchema };const rendered = (name: string, call: (args: Record<string, unknown>) => string) => ({renderCall: (args: unknown) => new Text(`${name} ${plain(call(args as Record<string, unknown>))}`, 0, 0),@@ -653,9 +659,9 @@ export function registerComputerTools(pi: ExtensionAPI, resources: ComputerResou}/** Delete for computer resources when Pi Agent IDE is absent: the IDE's Delete handles them otherwise. */-export function registerComputerDelete(pi: ExtensionAPI, resources: ComputerResources, guide: string): void {+export function registerComputerDelete(pi: ExtensionAPI, resources: ComputerResources): void {pi.registerTool({- name: "computer_delete", label: "Computer delete", exposure: "deferred", namespace: computerNamespace(guide), outputSchema: receiptSchema,+ name: "computer_delete", label: "Computer delete", exposure: "deferred", namespace: computerNamespace(), outputSchema: receiptSchema,description: "Use computer_delete to close an app window, or to release a sandbox, isolated session, machine or all of computer use.",parameters: Type.Object({ path: Type.String({ description: COMPUTER_DELETE_HELP }) }, { additionalProperties: false }),annotations: { readOnlyHint: false, destructiveHint: true, openWorldHint: false },diff --git a/src/index.ts b/src/index.tsindex 8fb578a..d17b498 100644--- a/src/index.ts+++ b/src/index.ts@@ -59,7 +59,7 @@ export default async function computerUse(pi: ExtensionAPI): Promise<void> {pi.on("session_start", () => {if (registered) return;registered = true;- registerComputerTools(pi, provider.resources, provider.guide);+ registerComputerTools(pi, provider.resources);if (!provider.attached()) registerStandaloneTools(pi, provider.resources, provider.guide);// Deferred tools are reachable only through tool_search, which Pi registers inactive.const active = pi.getActiveTools();diff --git a/src/standalone-tools.ts b/src/standalone-tools.tsindex b14ad8b..a095778 100644--- a/src/standalone-tools.ts+++ b/src/standalone-tools.ts@@ -19,7 +19,7 @@ const plain = (value: unknown) => String(value).replace(/[\x00-\x1f\x7f-\x9f]/g,* computer_write, computer_insert and computer_delete tools that tool_search finds. Tools stay off the model's tool list until a search loads them.*/export function registerStandaloneTools(pi: ExtensionAPI, resources: ComputerResources, guide: string): void {- const namespace = computerNamespace(guide);+ const namespace = computerNamespace();pi.registerTool({name: "computer_read", label: "Computer read", exposure: "deferred", namespace,description: `Use computer_read to discover machines, see app windows, displays and browser pages, and get the observations that ground input. Read ${GUIDE} first for addresses, views and commands.`,@@ -93,5 +93,5 @@ export function registerStandaloneTools(pi: ExtensionAPI, resources: ComputerRes},});}- registerComputerDelete(pi, resources, guide);+ registerComputerDelete(pi, resources);}diff --git a/test/computer-tools.test.ts b/test/computer-tools.test.tsindex f169777..817b04f 100644--- a/test/computer-tools.test.ts+++ b/test/computer-tools.test.ts@@ -105,7 +105,8 @@ test("input tools build one plan for native windows and one for browser pages",expect(typed).toMatchObject({ native: { action: "set_value", text: "Ada" }, web: { action: "fill", text: "Ada" },then: { target: "1:a", native: { action: "key", key: "return" }, web: { action: "press", key: "Enter" } } });expect(() => typePlan({ window: w, text: "x", replace: true })).toThrow("replace needs the field as target.");- expect(pressPlan({ window: w, key: "ctrl+shift+Tab" })).toMatchObject({ native: { action: "key", key: "tab", modifiers: ["ctrl", "shift"] }, web: { action: "press", key: "Control+Shift+Tab" } });+ expect(pressPlan({ window: w, key: "ctrl+shift+Tab" })).toMatchObject({ native: { action: "hotkey", keys: ["ctrl", "shift", "tab"] }, web: { action: "press", key: "Control+Shift+Tab" } });+ expect(pressPlan({ window: w, key: "Enter" }).native).toEqual({ action: "key", key: "enter" });expect(pressPlan({ window: w, key: "w", hold_ms: 500 }).native).toEqual({ action: "hold", keys: ["w"], duration_ms: 500 });expect(pressPlan({ window: w, key: "f5" }).web).toEqual({ action: "press", key: "F5" });expect(() => pressPlan({ window: w, key: "hyper+a" })).toThrow("Modifiers are ctrl, shift, alt and win.");diff --git a/test/extension-modes.integration.test.ts b/test/extension-modes.integration.test.tsindex 5f05075..0f83783 100644--- a/test/extension-modes.integration.test.ts+++ b/test/extension-modes.integration.test.ts@@ -28,6 +28,9 @@ async function run(name: string, extensions: string[], tools: string[], calls: {test("without Pi Agent IDE, tool_search finds the deferred computer tools and they work", async () => {const result = await run("standalone", ["builtin:tool-search", computerUse], ["read"], [{ id: "find", name: "tool_search", arguments: { query: "computer read windows displays" } },+ // The input queries run while most typed tools are still unloaded, so they must win against all of them.+ { id: "input", name: "tool_search", arguments: { query: "computer click hover drag scroll" } },+ { id: "keys", name: "tool_search", arguments: { query: "computer type press select timeline" } },{ id: "guide", name: "computer_read", arguments: { path: "docs:computer" } },{ id: "session", name: "computer_read", arguments: { path: "computer:session" } },{ id: "close", name: "computer_insert", arguments: { path: "computer:session", text: '{"action":"close"}' } },@@ -37,17 +40,18 @@ test("without Pi Agent IDE, tool_search finds the deferred computer tools and th{ id: "files", name: "tool_search", arguments: { query: "computer files clipboard record" } },{ id: "list", name: "computer_files", arguments: { machine: "computer:session", action: "list" } },{ id: "sequence", name: "computer_record", arguments: { at: "computer:machine/local/display/primary", action: "sequence" } },- { id: "input", name: "tool_search", arguments: { query: "computer click type press drag" } },{ id: "click", name: "computer_click", arguments: { window: "computer:machine/local", target: "1:a" } },{ id: "delete", name: "computer_delete", arguments: { path: "computer:session" } },]);- expect(getToolCallNames(result)).toEqual(["tool_search", "computer_read", "computer_read", "computer_insert", "tool_search", "tool_search", "computer_display", "tool_search", "computer_files", "computer_record", "tool_search", "computer_click", "computer_delete"]);+ expect(getToolCallNames(result)).toEqual(["tool_search", "tool_search", "tool_search", "computer_read", "computer_read", "computer_insert", "tool_search", "tool_search", "computer_display", "tool_search", "computer_files", "computer_record", "computer_click", "computer_delete"]);+ for (const tool of ["computer_click", "computer_hover", "computer_drag", "computer_scroll", "computer_type", "computer_press", "computer_select", "computer_timeline"])+ expect(["find", "input", "keys"].map(id => getToolResultText(result, id)).join("\n")).toContain(`- ${tool}:`);for (const id of ["find", "guide", "session", "close", "delete"])expect(getToolExecution(result, id).isError, getToolResultText(result, id)).toBe(false);expect(getToolResultText(result, "find")).toContain("computer_read");// tool_search loads the best matches of each query, at most 8 by default, so the guide names one query per cluster.// A query lists only tools it newly loads; together the guide's cluster queries load every computer tool.- const loaded = ["find", "open", "more", "files", "input"].map(id => getToolResultText(result, id)).join("\n");+ const loaded = ["find", "open", "more", "files", "input", "keys"].map(id => getToolResultText(result, id)).join("\n");for (const tool of ["computer_search", "computer_insert", "computer_write", "computer_delete", "computer_open", "computer_window", "computer_display","computer_wait", "computer_browser", "computer_files", "computer_clipboard", "computer_record", "computer_click", "computer_hover", "computer_drag","computer_scroll", "computer_type", "computer_press", "computer_select", "computer_timeline"]) expect(loaded).toContain(`- ${tool}:`);@@ -85,7 +89,8 @@ test("with Pi Agent IDE, computer use extends Read, Search, Insert and Delete an{ id: "open", name: "tool_search", arguments: { query: "computer open window" } },{ id: "more", name: "tool_search", arguments: { query: "computer wait browser" } },{ id: "files", name: "tool_search", arguments: { query: "computer files clipboard record" } },- { id: "input", name: "tool_search", arguments: { query: "computer click type press drag" } },+ { id: "input", name: "tool_search", arguments: { query: "computer click hover drag scroll" } },+ { id: "keys", name: "tool_search", arguments: { query: "computer type press select timeline" } },{ id: "session", name: "read", arguments: { path: "computer:session" } },{ id: "close", name: "insert", arguments: { path: "computer:session", text: '{"action":"close"}' } },{ id: "force", name: "delete", arguments: { path: "computer:machine/local?force=true" } },@@ -94,7 +99,7 @@ test("with Pi Agent IDE, computer use extends Read, Search, Insert and Delete an// The force option reaches computer use through the IDE's resolver, which then refuses it for a machine.expect(getToolResultText(result, "force")).toContain("?force=true applies to window addresses only.");// The IDE's own tools read and delete computer resources; the cluster queries load every typed tool.- const loaded = ["find", "open", "more", "files", "input"].map(id => getToolResultText(result, id)).join("\n");+ const loaded = ["find", "open", "more", "files", "input", "keys"].map(id => getToolResultText(result, id)).join("\n");expect(loaded).not.toContain("computer_read");expect(loaded).not.toContain("computer_delete");for (const tool of ["computer_open", "computer_window", "computer_display", "computer_wait", "computer_browser", "computer_files", "computer_clipboard", "computer_record",diff --git a/test/fixtures/window-management-extension.ts b/test/fixtures/window-management-extension.tsindex 96ac01c..06fef7e 100644--- a/test/fixtures/window-management-extension.ts+++ b/test/fixtures/window-management-extension.ts@@ -59,7 +59,7 @@ export default async function fixture(pi: ExtensionAPI): Promise<void> {content: [{ type: "text", text: "No synthetic runtime is running." }], details: { status: "closed", mode },};const provider = await registerComputerProvider(pi, getSession, close);- registerComputerTools(pi, provider.resources, provider.guide);+ registerComputerTools(pi, provider.resources);pi.registerTool({name: "verify_window_management", label: "Verify one-use window management",description: "Exercise real IDE state and management with synthetic MCP receipts only.",diff --git a/test/live/typed-input.live.ts b/test/live/typed-input.live.tsindex 96e6466..dc8f1f2 100644--- a/test/live/typed-input.live.ts+++ b/test/live/typed-input.live.ts@@ -53,11 +53,14 @@ test("one action vocabulary drives a native window by locator, ref and point", aexpect((await act(window, pressPlan({ window, key: "ctrl+a" }))).result.details.status).toBe("delivered");expect((await act(window, pressPlan({ window, key: "Backspace" }))).result.details.status).toBe("delivered");// An empty field's value falls back to its name in conditions, so its aria line shows the emptiness: no ": value".- expect(presentComputerResult(await live.read(window, ["aria"]))).toMatch(/textbox "Field" \[ref=/);+ // A chord must reach the app as a chord: a bare "a" would leave "worlda" behind instead of an empty field.+ const field_line = /textbox "Field".*$/m.exec(presentComputerResult(await live.read(window, ["aria"])))![0];+ expect(field_line).toMatch(/^textbox "Field" \[ref=[0-9a-z]+:[a-z]+\]$/);expect((await act(window, selectPlan({ window, target: "getByRole('list', { name: 'Pick' })", values: ["beta"] }))).result.details.status).toBe("delivered");expect(await holds(window, [{ target: "getByRole('option', { name: 'beta' })", is: "selected" }])).toBe("satisfied");expect((await act(window, scrollPlan({ window, target: "getByRole('list', { name: 'Pick' })", delta_y: 1 }))).result.details.status).not.toBe("unknown");- expect((await act(window, scrollPlan({ window, x: centre.x, y: centre.y + 60, delta_y: 1 }))).result.details.status).toBe("delivered");+ const wheel = await act(window, scrollPlan({ window, x: centre.x, y: centre.y + 60, delta_y: 1 }));+ expect(wheel.result.details.status, wheel.body).toBe("delivered");expect((await act(window, hoverPlan({ window, target: button }))).result.details.status).toBe("delivered");expect((await act(window, dragPlan({ window, from: { x: centre.x, y: centre.y }, to: { x: centre.x + 40, y: centre.y } }))).result.details.status).toBe("delivered");const timeline = await act(window, timelinePlan({ window, events: [{ at_ms: 0, type: "move", x: centre.x, y: centre.y }, { at_ms: 50, type: "button_down" }, { at_ms: 100, type: "button_up" }] }));