Skip to content

Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.

fix: chords go as hotkeys, the namespace points to the guide so tool_search ranks tools by their own text (LPT-927)

alexshpunt-clanker[bot]b41eef4a2fb9
diff --git a/docs/computer-resources.md b/docs/computer-resources.md
index 678d99c..a167cb7 100644
--- a/docs/computer-resources.md
+++ b/docs/computer-resources.md
@@ -11,7 +11,7 @@ The current desktop scope is Windows, with Pi running on it directly or in its W
## Typed tools
-Some operations have their own tool with typed fields, in both modes. They are deferred like the tools above. `tool_search` loads at most 8 best matches per query, so search one group at a time: `computer open window`, `computer wait browser`, `computer files clipboard record`, `computer click type press drag`. Their receipts match Insert receipts.
+Some operations have their own tool with typed fields, in both modes. They are deferred like the tools above. `tool_search` loads at most 8 best matches per query, so search one group at a time: `computer open window`, `computer wait browser`, `computer files clipboard record`, `computer click hover drag scroll`, `computer type press select timeline`. Their receipts match Insert receipts.
| Tool | Does | Replaces |
| --- | --- | --- |
@@ -43,7 +43,7 @@ The input tools take a window address, never an observation. The window decides
- A Playwright locator such as `getByRole('button', { name: 'Save' })`, resolved with auto-waiting like the [native locators](#native-locators) or by Playwright on a page; `timeout_ms` bounds the wait.
- A point. On a native window it is in pixels of the latest image of that window: a Read with an image, or the successor of an action. Without one, the tool captures the window at the default overview size first, about 300 ms once. On a browser window a point is in CSS pixels of the page; add `space: "window"` for pixels of a window Read, for example the browser's own toolbar.
-Native input starts in the background. When the window refuses it before anything was sent, the same call sends it in the foreground and says so; later input to that window goes in the foreground at once. `foreground: true` goes there first, `foreground: false` never does. A foreground action activates the window and moves the real pointer. `computer_hover` and `computer_timeline` move the real pointer on native windows, so they always run in the foreground. Pen eraser and barrel, the `x1`/`x2` buttons and `move_relative` exist only natively; on a browser page give `space: "window"`. Pen pressure and tilt on a page reach it as a `pen` pointer through CDP in Chrome and Edge.
+Native input starts in the background. When the window refuses it before anything was sent, the same call sends it in the foreground and says so; later input to that window goes in the foreground at once. `foreground: true` goes there first, `foreground: false` never does. A foreground action activates the window and moves the real pointer. `computer_hover` and `computer_timeline` move the real pointer on native windows, so they always run in the foreground. Pen eraser and barrel, the `x1`/`x2` buttons and `move_relative` exist only natively; on a browser page give `space: "window"`. Pen pressure and tilt on a page reach it as a `pen` pointer through CDP in Chrome and Edge. Keys on a page go to the page, so browser shortcuts such as `alt+ArrowLeft` do not run there; use `computer_browser` to go back, or `space: "window"` to press them natively.
A native Read and a DOM Read of the same browser window keep each other's observations; input retires both.
diff --git a/src/computer-resources.ts b/src/computer-resources.ts
index f340006..1846770 100644
--- a/src/computer-resources.ts
+++ b/src/computer-resources.ts
@@ -49,7 +49,7 @@ const BATCH_STEPS = 200;
/** Native actions a batch may run; each later step re-resolves its target in a fresh UI walk. */
const NATIVE_BATCH_ACTIONS = ["click", "double_click", "right_click", "type_text", "key", "hotkey", "scroll", "invoke", "select", "set_value"];
/** Native actions that take a Playwright locator or JSON target. */
-const NATIVE_LOCATOR_ACTIONS = ["click", "double_click", "right_click", "hover", "type_text", "key", "scroll", "invoke", "select", "set_value", "check", "uncheck"];
+const NATIVE_LOCATOR_ACTIONS = ["click", "double_click", "right_click", "hover", "type_text", "key", "hotkey", "scroll", "invoke", "select", "set_value", "check", "uncheck"];
/** A point in pixels of a window's latest image, or in CSS pixels of a browser page. */
export interface InputPoint { x: number; y: number }
/** One typed input on a window, built by a computer_* input tool; the window decides which backend runs it. */
diff --git a/src/computer-tools.ts b/src/computer-tools.ts
index fa9cea8..b88c8f8 100644
--- a/src/computer-tools.ts
+++ b/src/computer-tools.ts
@@ -10,9 +10,14 @@ import { COMPUTER_DELETE_HELP, computerAction, deleteComputer } from "./provider
const plain = (value: unknown) => String(value).replace(/[\x00-\x1f\x7f-\x9f]/g, " ");
type Action = Awaited<ReturnType<typeof computerAction>>;
-/** The namespace every computer_* tool shares; tool_search finds them by it, and its instructions are the guide. */
-export function computerNamespace(guide: string) {
- return { name: "computer", description: "Read and operate Windows app windows, displays and browser pages through computer: addresses", instructions: guide };
+/**
+ * The namespace every computer_* tool shares. tool_search indexes its instructions into every tool's search text,
+ * so they point to the guide instead of holding it: the guide names every operation and would make all tools match
+ * every query equally.
+ */
+export function computerNamespace() {
+ return { name: "computer", description: "Read and operate Windows app windows, displays and browser pages through computer: addresses",
+ instructions: "Read docs:computer for the guide: with Read in Pi Agent IDE, with computer_read without it." };
}
/** Codemode scripts receive the receipt object for every outcome, so they branch on outcome without try/catch. */
export const receiptSchema = Type.Object({ outcome: Type.String(), effect: Type.String() }, { additionalProperties: true });
@@ -399,7 +404,8 @@ export function pressPlan(params: Static<typeof pressParameters>): InputPlan {
const key = parts.at(-1)!, held = parts.slice(0, -1).map(part => part.toLowerCase().replace(/^(control)$/, "ctrl").replace(/^(meta|windows|cmd)$/, "win"));
if (held.some(part => !PAGE_MODIFIERS[part])) throw new ComputerRequestError("Modifiers are ctrl, shift, alt and win.");
const native = params.hold_ms !== undefined ? { action: "hold", keys: [...held, key.toLowerCase()], duration_ms: params.hold_ms }
- : { action: "key", key: key.toLowerCase(), ...(held.length ? { modifiers: held } : {}) };
+ // A chord goes as a hotkey: posted key messages do not carry modifier state, so the app would see the bare key.
+ : held.length ? { action: "hotkey", keys: [...held, key.toLowerCase()] } : { action: "key", key: key.toLowerCase() };
const keys = [...held.map(part => PAGE_MODIFIERS[part]), pageKey(key)];
const aimed = aim(params, "press", false);
if (params.hold_ms !== undefined && aimed.target !== undefined) throw new ComputerRequestError("hold_ms goes to the focused element; focus it with computer_click first.");
@@ -462,8 +468,8 @@ export function timelinePlan(params: Static<typeof timelineParameters>): InputPl
* Register the typed computer_* tools that replace Insert commands one cluster at a time. They are deferred: a
* tool_search for computer loads them, so sessions that never use the computer pay nothing for them.
*/
-export function registerComputerTools(pi: ExtensionAPI, resources: ComputerResources, guide: string): void {
- const namespace = computerNamespace(guide);
+export function registerComputerTools(pi: ExtensionAPI, resources: ComputerResources): void {
+ const namespace = computerNamespace();
const common = { exposure: "deferred" as const, namespace, outputSchema: receiptSchema };
const rendered = (name: string, call: (args: Record<string, unknown>) => string) => ({
renderCall: (args: unknown) => new Text(`${name} ${plain(call(args as Record<string, unknown>))}`, 0, 0),
@@ -653,9 +659,9 @@ export function registerComputerTools(pi: ExtensionAPI, resources: ComputerResou
}
/** Delete for computer resources when Pi Agent IDE is absent: the IDE's Delete handles them otherwise. */
-export function registerComputerDelete(pi: ExtensionAPI, resources: ComputerResources, guide: string): void {
+export function registerComputerDelete(pi: ExtensionAPI, resources: ComputerResources): void {
pi.registerTool({
- name: "computer_delete", label: "Computer delete", exposure: "deferred", namespace: computerNamespace(guide), outputSchema: receiptSchema,
+ name: "computer_delete", label: "Computer delete", exposure: "deferred", namespace: computerNamespace(), outputSchema: receiptSchema,
description: "Use computer_delete to close an app window, or to release a sandbox, isolated session, machine or all of computer use.",
parameters: Type.Object({ path: Type.String({ description: COMPUTER_DELETE_HELP }) }, { additionalProperties: false }),
annotations: { readOnlyHint: false, destructiveHint: true, openWorldHint: false },
diff --git a/src/index.ts b/src/index.ts
index 8fb578a..d17b498 100644
--- a/src/index.ts
+++ b/src/index.ts
@@ -59,7 +59,7 @@ export default async function computerUse(pi: ExtensionAPI): Promise<void> {
pi.on("session_start", () => {
if (registered) return;
registered = true;
- registerComputerTools(pi, provider.resources, provider.guide);
+ registerComputerTools(pi, provider.resources);
if (!provider.attached()) registerStandaloneTools(pi, provider.resources, provider.guide);
// Deferred tools are reachable only through tool_search, which Pi registers inactive.
const active = pi.getActiveTools();
diff --git a/src/standalone-tools.ts b/src/standalone-tools.ts
index b14ad8b..a095778 100644
--- a/src/standalone-tools.ts
+++ b/src/standalone-tools.ts
@@ -19,7 +19,7 @@ const plain = (value: unknown) => String(value).replace(/[\x00-\x1f\x7f-\x9f]/g,
* computer_write, computer_insert and computer_delete tools that tool_search finds. Tools stay off the model's tool list until a search loads them.
*/
export function registerStandaloneTools(pi: ExtensionAPI, resources: ComputerResources, guide: string): void {
- const namespace = computerNamespace(guide);
+ const namespace = computerNamespace();
pi.registerTool({
name: "computer_read", label: "Computer read", exposure: "deferred", namespace,
description: `Use computer_read to discover machines, see app windows, displays and browser pages, and get the observations that ground input. Read ${GUIDE} first for addresses, views and commands.`,
@@ -93,5 +93,5 @@ export function registerStandaloneTools(pi: ExtensionAPI, resources: ComputerRes
},
});
}
- registerComputerDelete(pi, resources, guide);
+ registerComputerDelete(pi, resources);
}
diff --git a/test/computer-tools.test.ts b/test/computer-tools.test.ts
index f169777..817b04f 100644
--- a/test/computer-tools.test.ts
+++ b/test/computer-tools.test.ts
@@ -105,7 +105,8 @@ test("input tools build one plan for native windows and one for browser pages",
expect(typed).toMatchObject({ native: { action: "set_value", text: "Ada" }, web: { action: "fill", text: "Ada" },
then: { target: "1:a", native: { action: "key", key: "return" }, web: { action: "press", key: "Enter" } } });
expect(() => typePlan({ window: w, text: "x", replace: true })).toThrow("replace needs the field as target.");
- expect(pressPlan({ window: w, key: "ctrl+shift+Tab" })).toMatchObject({ native: { action: "key", key: "tab", modifiers: ["ctrl", "shift"] }, web: { action: "press", key: "Control+Shift+Tab" } });
+ expect(pressPlan({ window: w, key: "ctrl+shift+Tab" })).toMatchObject({ native: { action: "hotkey", keys: ["ctrl", "shift", "tab"] }, web: { action: "press", key: "Control+Shift+Tab" } });
+ expect(pressPlan({ window: w, key: "Enter" }).native).toEqual({ action: "key", key: "enter" });
expect(pressPlan({ window: w, key: "w", hold_ms: 500 }).native).toEqual({ action: "hold", keys: ["w"], duration_ms: 500 });
expect(pressPlan({ window: w, key: "f5" }).web).toEqual({ action: "press", key: "F5" });
expect(() => pressPlan({ window: w, key: "hyper+a" })).toThrow("Modifiers are ctrl, shift, alt and win.");
diff --git a/test/extension-modes.integration.test.ts b/test/extension-modes.integration.test.ts
index 5f05075..0f83783 100644
--- a/test/extension-modes.integration.test.ts
+++ b/test/extension-modes.integration.test.ts
@@ -28,6 +28,9 @@ async function run(name: string, extensions: string[], tools: string[], calls: {
test("without Pi Agent IDE, tool_search finds the deferred computer tools and they work", async () => {
const result = await run("standalone", ["builtin:tool-search", computerUse], ["read"], [
{ id: "find", name: "tool_search", arguments: { query: "computer read windows displays" } },
+ // The input queries run while most typed tools are still unloaded, so they must win against all of them.
+ { id: "input", name: "tool_search", arguments: { query: "computer click hover drag scroll" } },
+ { id: "keys", name: "tool_search", arguments: { query: "computer type press select timeline" } },
{ id: "guide", name: "computer_read", arguments: { path: "docs:computer" } },
{ id: "session", name: "computer_read", arguments: { path: "computer:session" } },
{ id: "close", name: "computer_insert", arguments: { path: "computer:session", text: '{"action":"close"}' } },
@@ -37,17 +40,18 @@ test("without Pi Agent IDE, tool_search finds the deferred computer tools and th
{ id: "files", name: "tool_search", arguments: { query: "computer files clipboard record" } },
{ id: "list", name: "computer_files", arguments: { machine: "computer:session", action: "list" } },
{ id: "sequence", name: "computer_record", arguments: { at: "computer:machine/local/display/primary", action: "sequence" } },
- { id: "input", name: "tool_search", arguments: { query: "computer click type press drag" } },
{ id: "click", name: "computer_click", arguments: { window: "computer:machine/local", target: "1:a" } },
{ id: "delete", name: "computer_delete", arguments: { path: "computer:session" } },
]);
- expect(getToolCallNames(result)).toEqual(["tool_search", "computer_read", "computer_read", "computer_insert", "tool_search", "tool_search", "computer_display", "tool_search", "computer_files", "computer_record", "tool_search", "computer_click", "computer_delete"]);
+ expect(getToolCallNames(result)).toEqual(["tool_search", "tool_search", "tool_search", "computer_read", "computer_read", "computer_insert", "tool_search", "tool_search", "computer_display", "tool_search", "computer_files", "computer_record", "computer_click", "computer_delete"]);
+ for (const tool of ["computer_click", "computer_hover", "computer_drag", "computer_scroll", "computer_type", "computer_press", "computer_select", "computer_timeline"])
+ expect(["find", "input", "keys"].map(id => getToolResultText(result, id)).join("\n")).toContain(`- ${tool}:`);
for (const id of ["find", "guide", "session", "close", "delete"])
expect(getToolExecution(result, id).isError, getToolResultText(result, id)).toBe(false);
expect(getToolResultText(result, "find")).toContain("computer_read");
// tool_search loads the best matches of each query, at most 8 by default, so the guide names one query per cluster.
// A query lists only tools it newly loads; together the guide's cluster queries load every computer tool.
- const loaded = ["find", "open", "more", "files", "input"].map(id => getToolResultText(result, id)).join("\n");
+ const loaded = ["find", "open", "more", "files", "input", "keys"].map(id => getToolResultText(result, id)).join("\n");
for (const tool of ["computer_search", "computer_insert", "computer_write", "computer_delete", "computer_open", "computer_window", "computer_display",
"computer_wait", "computer_browser", "computer_files", "computer_clipboard", "computer_record", "computer_click", "computer_hover", "computer_drag",
"computer_scroll", "computer_type", "computer_press", "computer_select", "computer_timeline"]) expect(loaded).toContain(`- ${tool}:`);
@@ -85,7 +89,8 @@ test("with Pi Agent IDE, computer use extends Read, Search, Insert and Delete an
{ id: "open", name: "tool_search", arguments: { query: "computer open window" } },
{ id: "more", name: "tool_search", arguments: { query: "computer wait browser" } },
{ id: "files", name: "tool_search", arguments: { query: "computer files clipboard record" } },
- { id: "input", name: "tool_search", arguments: { query: "computer click type press drag" } },
+ { id: "input", name: "tool_search", arguments: { query: "computer click hover drag scroll" } },
+ { id: "keys", name: "tool_search", arguments: { query: "computer type press select timeline" } },
{ id: "session", name: "read", arguments: { path: "computer:session" } },
{ id: "close", name: "insert", arguments: { path: "computer:session", text: '{"action":"close"}' } },
{ id: "force", name: "delete", arguments: { path: "computer:machine/local?force=true" } },
@@ -94,7 +99,7 @@ test("with Pi Agent IDE, computer use extends Read, Search, Insert and Delete an
// The force option reaches computer use through the IDE's resolver, which then refuses it for a machine.
expect(getToolResultText(result, "force")).toContain("?force=true applies to window addresses only.");
// The IDE's own tools read and delete computer resources; the cluster queries load every typed tool.
- const loaded = ["find", "open", "more", "files", "input"].map(id => getToolResultText(result, id)).join("\n");
+ const loaded = ["find", "open", "more", "files", "input", "keys"].map(id => getToolResultText(result, id)).join("\n");
expect(loaded).not.toContain("computer_read");
expect(loaded).not.toContain("computer_delete");
for (const tool of ["computer_open", "computer_window", "computer_display", "computer_wait", "computer_browser", "computer_files", "computer_clipboard", "computer_record",
diff --git a/test/fixtures/window-management-extension.ts b/test/fixtures/window-management-extension.ts
index 96ac01c..06fef7e 100644
--- a/test/fixtures/window-management-extension.ts
+++ b/test/fixtures/window-management-extension.ts
@@ -59,7 +59,7 @@ export default async function fixture(pi: ExtensionAPI): Promise<void> {
content: [{ type: "text", text: "No synthetic runtime is running." }], details: { status: "closed", mode },
};
const provider = await registerComputerProvider(pi, getSession, close);
- registerComputerTools(pi, provider.resources, provider.guide);
+ registerComputerTools(pi, provider.resources);
pi.registerTool({
name: "verify_window_management", label: "Verify one-use window management",
description: "Exercise real IDE state and management with synthetic MCP receipts only.",
diff --git a/test/live/typed-input.live.ts b/test/live/typed-input.live.ts
index 96e6466..dc8f1f2 100644
--- a/test/live/typed-input.live.ts
+++ b/test/live/typed-input.live.ts
@@ -53,11 +53,14 @@ test("one action vocabulary drives a native window by locator, ref and point", a
expect((await act(window, pressPlan({ window, key: "ctrl+a" }))).result.details.status).toBe("delivered");
expect((await act(window, pressPlan({ window, key: "Backspace" }))).result.details.status).toBe("delivered");
// An empty field's value falls back to its name in conditions, so its aria line shows the emptiness: no ": value".
- expect(presentComputerResult(await live.read(window, ["aria"]))).toMatch(/textbox "Field" \[ref=/);
+ // A chord must reach the app as a chord: a bare "a" would leave "worlda" behind instead of an empty field.
+ const field_line = /textbox "Field".*$/m.exec(presentComputerResult(await live.read(window, ["aria"])))![0];
+ expect(field_line).toMatch(/^textbox "Field" \[ref=[0-9a-z]+:[a-z]+\]$/);
expect((await act(window, selectPlan({ window, target: "getByRole('list', { name: 'Pick' })", values: ["beta"] }))).result.details.status).toBe("delivered");
expect(await holds(window, [{ target: "getByRole('option', { name: 'beta' })", is: "selected" }])).toBe("satisfied");
expect((await act(window, scrollPlan({ window, target: "getByRole('list', { name: 'Pick' })", delta_y: 1 }))).result.details.status).not.toBe("unknown");
- expect((await act(window, scrollPlan({ window, x: centre.x, y: centre.y + 60, delta_y: 1 }))).result.details.status).toBe("delivered");
+ const wheel = await act(window, scrollPlan({ window, x: centre.x, y: centre.y + 60, delta_y: 1 }));
+ expect(wheel.result.details.status, wheel.body).toBe("delivered");
expect((await act(window, hoverPlan({ window, target: button }))).result.details.status).toBe("delivered");
expect((await act(window, dragPlan({ window, from: { x: centre.x, y: centre.y }, to: { x: centre.x + 40, y: centre.y } }))).result.details.status).toBe("delivered");
const timeline = await act(window, timelinePlan({ window, events: [{ at_ms: 0, type: "move", x: centre.x, y: centre.y }, { at_ms: 50, type: "button_down" }, { at_ms: 100, type: "button_up" }] }));