Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.
TypeScript accessibilityagent-toolsai-agentbrowser-automationcomputer-usecuadesktop-automationelectronpipi-coding-agentpi-extensionpi-packageplaywrighttypescriptui-automationvisionwebview2windowswindows-automationwsl
feat: typed computer_open, computer_window, computer_display and Delete for computer resources (LPT-934)
alexshpunt-clanker[bot]73e7b1fce60c
diff --git a/docs/computer-resources.md b/docs/computer-resources.mdindex b9e39fa..f7bc23b 100644--- a/docs/computer-resources.md+++ b/docs/computer-resources.md@@ -9,6 +9,21 @@ Both modes take the same addresses, views, JSON commands and Codemode values. InThe current desktop scope is Windows, with Pi running on it directly or in its WSL, on this machine or on another one over SSH. A different interface does not qualify a new runtime or platform. The production `windows-desktop` profile selects the same checked bytes as `windows-window-management`, so it qualifies wherever this guide names that profile or one it inherits. Safe permits session-opened windows and forbids display reads. YOLO permits task-authorized existing windows; it is not OS isolation.+## Typed tools++Some operations have their own tool with typed fields, in both modes. They are deferred like the tools above: one `tool_search` for computer loads them. Their receipts match Insert receipts.++| Tool | Does | Replaces |+| --- | --- | --- |+| `computer_open` | Starts an app by name or path, a Playwright browser, a Windows Sandbox or an isolated session at a machine address; opens an app address; attaches Playwright to a browser window address | Insert `open`, `open_browser`, `open_sandbox`, `open_session`, `attach_browser` |+| `computer_window` | Moves, resizes, maximizes, minimizes, focuses or restores a window by its address | Read `state`, then Insert into the window-state address |+| `computer_display` | Changes a display's resolution | Insert `resize` into a display |+| Delete (`computer_delete` without IDE) | Closes a window like its close button, or releases a sandbox, isolated session, machine or `computer:session` | Insert `close` and `kill` |++Delete of a window asks it to close: a browser the session opened closes with its disposable profile, an attached browser is detached first, and any other window gets the standard close request. The receipt says whether the window closed; an app that asks about unsaved work keeps it open, so Read it next. Add `?force=true` to the window address to kill the hung app of a window this session opened. Safe mode closes only windows the session opened.++The JSON commands below keep working while the remaining operations move to their own tools.+## Discover and readEvery machine is addressed the same way: `computer:machine/local` is this one, and `computer:machine/<name>` is another Windows machine configured under `machines` (see [Work on another machine](#work-on-another-machine)). A machine starts on its first use; nothing needs connecting. Read `computer:session` for the machines, sandboxes and sessions started so far.@@ -258,7 +273,7 @@ All paths are on the remote WSL. Without `node` and `package`, the machine works## Change a display's resolution-A high-resolution display at a large Windows scale gives small, dense overviews. In YOLO, Insert `{"action":"resize","width":2160,"height":1440}` into a display address returned by a machine's `displays` address. It works on every machine; Safe refuses.+A high-resolution display at a large Windows scale gives small, dense overviews. In YOLO, call `computer_display` with the display address and the new size, or Insert `{"action":"resize","width":2160,"height":1440}` into a display address returned by a machine's `displays` address. It works on every machine; Safe refuses.Only modes the graphics driver lists are used, keeping the colour depth and preferring the current refresh rate. An unsupported size refuses and lists the supported ones; keep the display's aspect ratio to avoid stretching. The change also affects what the human sees on that display.@@ -276,7 +291,7 @@ The receipt separates the restore request, observed restoration, activation atteUse the explicitly selected, byte-checked `windows-window-management` runtime. Read the authorized window with `views:["state"]` to obtain its bounds, bounds source, minimized/maximized state, foreground, separately measured keyboard focus and independent virtual-desktop query results. This metadata issues one-use control state, not application-input or pixel authority. Cached Read does not resample or renew it.-Insert one command into the returned `computer:window-state/...` address:+`computer_window` samples that state and sends one command in a single call, so it needs no window-state address. Otherwise, insert one command into the returned `computer:window-state/...` address:| Operation | JSON command || --- | --- |@@ -418,6 +433,6 @@ On a browser observation, steps are the DOM actions above plus `wait_for` and `v## Cleanup and development-Close only task-owned disposable windows through grounded input first. Insert `{"action":"close"}` into a machine address to release that machine, or into `computer:session` to release every machine, sandbox and isolated session this computer use started. This does not close unrelated application documents or save them. It does close the browsers the session opened and delete their profiles, detaches from attached browsers, destroys sandboxes and isolated sessions and stops remote drivers. The result lists each released part with its own outcome; a failed or uncertain cleanup is reported as `unknown` rather than called complete.+Close only task-owned disposable windows, with Delete or through grounded input. Delete a machine address (or Insert `{"action":"close"}` into it) to release that machine, and `computer:session` to release every machine, sandbox and isolated session this computer use started. This does not close unrelated application documents or save them. It does close the browsers the session opened and delete their profiles, detaches from attached browsers, destroys sandboxes and isolated sessions and stops remote drivers. The result lists each released part with its own outcome; a failed or uncertain cleanup is reported as `unknown` rather than called complete.For a source checkout, run `npm ci --ignore-scripts --legacy-peer-deps`, then `npm run dev:ide -- /absolute/path/to/pi-agent-ide`. This links the IDE SDK to that checkout without modifying IDE or its dependencies. To try a worktree, point the package entry in your Pi settings at it, and point it back afterwards. Installation/interface switching and new native runtime adoption are separate decisions.diff --git a/src/computer-action-renderer.ts b/src/computer-action-renderer.tsindex 1c106b4..313e274 100644--- a/src/computer-action-renderer.ts+++ b/src/computer-action-renderer.ts@@ -7,7 +7,7 @@ const record = (value: unknown): Record<string, unknown> => value !== null && tyconst plain = (value: string) => value.replace(/[\x00-\x08\x0b-\x1f\x7f-\x9f]/g, "");/** Preserve computer receipts without replacing IDE's renderer for unrelated edits. */-export function computerActionRenderer(tool: "write" | "insert"): Registration {+export function computerActionRenderer(tool: string): Registration {return {tool,matches(value, phase) {diff --git a/src/computer-resources.ts b/src/computer-resources.tsindex 32a6b4b..ae0c702 100644--- a/src/computer-resources.ts+++ b/src/computer-resources.ts@@ -127,6 +127,8 @@ function openTarget(command: Record<string, unknown>, where: string): string | {if (!Array.isArray(args) || args.length > 32 || args.some(arg => typeof arg !== "string" || arg.length > 8192)) throw new ComputerRequestError("args must be up to 32 strings.");return { path: command.path, args: args as string[], ...(command.debugging ? { debugging: true } : {}) };}+/** The Delete option that kills the process of a hung window the session opened instead of asking it to close. */+const FORCE = "?force=true";const FILE_ACTIONS = ["put", "get", "files", "clipboard_read", "clipboard_write", "record_start", "record_stop", "audio_record"];/** Speech in a WAV file as text, with faster-whisper run through uv on the agent's own machine. */async function transcribe(wav: string): Promise<{ text: string; language: string; segments: { start: number; end: number; text: string }[] }> {@@ -806,6 +808,49 @@ export class ComputerResources {{ operation: "search", observation_id: address.ref, query, ...(limit !== undefined ? { limit } : {}) });}+ /**+ * Close what an address names: a window (like its close button; `?force=true` kills the hung process of a window+ * this session opened), a sandbox or isolated session, a machine, or computer:session. The receipt names its machine.+ */+ async delete(source: string, cwd: string, signal?: AbortSignal): Promise<ComputerResult> {+ const force = source.endsWith(FORCE);+ const address = this.address(force ? source.slice(0, -FORCE.length) : source);+ if (force && address.kind !== "window") throw new ComputerRequestError("?force=true applies to window addresses only.");+ return this.stamp(this.place(address), await this.deleteAt(address, cwd, force, signal));+ }+ private async deleteAt(address: ComputerAddress, cwd: string, force: boolean, signal?: AbortSignal): Promise<ComputerResult> {+ if (address.kind === "window") {+ const session = await this.sessionFor(cwd, { window_ref: address.ref }, signal);+ this.retireWindow(address.ref);+ return force ? session.killOwned(address.ref, signal) : session.closeWindow(address.ref, signal);+ }+ if (address.kind === "sandbox") return this.closeSandbox(this.sandbox(address.ref));+ if (address.kind === "machine" && !address.part) return this.closeMachine(address.ref);+ if (address.kind === "session") return this.closeAll();+ throw new ComputerRequestError("Delete closes a window, a sandbox or isolated session, a machine or computer:session. Observations, sequences, apps and displays need no closing.");+ }+ /**+ * Move, resize, minimize, maximize, focus or restore a window by its address: a fresh native state sample bounds the+ * one command, so no window-state address is handed out.+ */+ async manageWindow(address: ComputerAddress, cwd: string, command: Record<string, unknown>, signal?: AbortSignal): Promise<ComputerResult> {+ if (address.kind !== "window") throw new ComputerRequestError("computer_window needs a window address from Search, Read or a name.");+ let management: WindowManagementCommand | undefined;+ if (command.action !== "restore") {+ try { management = parseWindowManagement(command); }+ catch (error) { throw new ComputerRequestError(error instanceof Error ? error.message : String(error)); }+ }+ const place = this.place(address);+ const session = await this.sessionFor(cwd, { window_ref: address.ref }, signal);+ this.retireWindow(address.ref);+ const state = await session.observeWindowState(address.ref, signal);+ const id = state.details.window_state_id;+ if (state.details.status !== "observed" || !id) return this.stamp(place, state);+ if (management && state.details.window_state_kind !== "management")+ return this.stamp(place, { ...state, content: [{ type: "text", text: "This native runtime can only restore and activate windows, not move, resize, minimize, maximize or focus them." }],+ details: { ...state.details, status: "refused" } });+ return this.stamp(place, management ? await session.manageWindow(address.ref, id, management, signal) : await session.restoreWindow(address.ref, id, signal));+ }/** Dispatch exactly one literal Write or JSON Insert against its address-bound target; the receipt names its machine. */async perform(address: ComputerAddress, cwd: string, value: string, tool: "write" | "insert", signal?: AbortSignal): Promise<ComputerResult> {const place = this.place(address);diff --git a/src/computer-tools.ts b/src/computer-tools.tsnew file mode 100644index 0000000..837bb22--- /dev/null+++ b/src/computer-tools.ts@@ -0,0 +1,159 @@+import { Type, type JsonValue, type Static } from "@earendil-works/pi-ai";+import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";+import { Text } from "@earendil-works/pi-tui";+import { ComputerRequestError, type ComputerResources } from "./computer-resources.ts";+import type { ComputerAddress } from "./computer-address.ts";+import { computerActionRenderer } from "./computer-action-renderer.ts";+import { COMPUTER_DELETE_HELP, computerAction, deleteComputer } from "./provider.ts";++const plain = (value: unknown) => String(value).replace(/[\x00-\x1f\x7f-\x9f]/g, " ");+type Action = Awaited<ReturnType<typeof computerAction>>;++/** The namespace every computer_* tool shares; tool_search finds them by it, and its instructions are the guide. */+export function computerNamespace(guide: string) {+ return { name: "computer", description: "Read and operate Windows app windows, displays and browser pages through computer: addresses", instructions: guide };+}+/** Codemode scripts receive the receipt object for every outcome, so they branch on outcome without try/catch. */+export const receiptSchema = Type.Object({ outcome: Type.String(), effect: Type.String() }, { additionalProperties: true });++/** A tool result of one computer action: its summary and images for the model, its receipt for scripts. */+export function actionResult(action: Action) {+ // The receipt is plain data with a few untyped native fields; a JSON copy gives scripts exactly that.+ const receipt: JsonValue = JSON.parse(JSON.stringify(action.script));+ return { content: [{ type: "text" as const, text: action.summary }, ...("images" in action ? action.images ?? [] : [])],+ details: { results: [], metadata: { semanticAction: action.data } }, structuredContent: receipt, ...(action.data.ok ? {} : { isError: true }) };+}++const openParameters = Type.Object({+ at: Type.String({ description: "Where to open: computer:machine/<name> (local is this machine), a computer:sandbox/... address, an app address from Search computer:machine/<name>/apps, or the window address of a running Chromium-based browser or app to attach Playwright to." }),+ app: Type.Optional(Type.String({ description: "Installed app name as Search computer:machine/<name>/apps lists it, such as Notepad." })),+ path: Type.Optional(Type.String({ description: "YOLO: absolute Windows path of an .exe to start, such as C:\\Tools\\app.exe." })),+ args: Type.Optional(Type.Array(Type.String(), { maxItems: 32, description: "Command-line arguments for path." })),+ debugging: Type.Optional(Type.Boolean({ description: "Start the WebView2 app at path with its debugging port open, so its window can be attached." })),+ browser: Type.Optional(Type.Union([Type.Literal("chrome"), Type.Literal("msedge"), Type.Literal("firefox")], { description: "Open a new visible browser driven by Playwright; its window takes DOM reads and page actions." })),+ url: Type.Optional(Type.String({ description: "http(s) page the new browser opens." })),+ profile: Type.Optional(Type.Literal("agent", { description: "YOLO: use the agent's own persistent browser profile, which keeps its logins between sessions. Omit for a disposable profile." })),+ sandbox: Type.Optional(Type.Object({+ import: Type.Optional(Type.Array(Type.Union([Type.String(), Type.Object({ host: Type.String(), guest: Type.Optional(Type.String()), writable: Type.Optional(Type.Boolean()) }, { additionalProperties: false })]),+ { maxItems: 8, description: "Host folders shared into the sandbox, read-only unless writable." })),+ export: Type.Optional(Type.String({ description: "One host folder the sandbox can write to." })),+ }, { additionalProperties: false, description: "Open a disposable Windows Sandbox on that machine, apart from the user's desktop; pass {} for no shared folders." })),+ isolated_session: Type.Optional(Type.Boolean({ description: "Open an isolated agent session on that machine, with its own desktop and foreground input." })),+}, { additionalProperties: false });++/** The Insert command that opens what computer_open asks for at this address. */+export function openCommand(address: ComputerAddress, params: Static<typeof openParameters>): Record<string, unknown> {+ const { app, path, args, debugging, browser, url, profile, sandbox, isolated_session } = params;+ const chosen = [app !== undefined && "app", path !== undefined && "path", (browser ?? url ?? profile) !== undefined && "browser",+ sandbox !== undefined && "sandbox", isolated_session === true && "isolated_session"].filter(Boolean);+ if (chosen.length > 1) throw new ComputerRequestError(`Open one thing at a time: ${chosen.join(" and ")} were both given.`);+ if ((args !== undefined || debugging !== undefined) && path === undefined) throw new ComputerRequestError("args and debugging go with path.");+ const what = chosen[0];+ if (address.kind === "app" || address.kind === "window") {+ if (what) throw new ComputerRequestError(`${address.kind === "app" ? "An app address opens that app" : "A window address attaches Playwright to that browser"}; give a machine address to open ${what}.`);+ return { action: address.kind === "app" ? "open" : "attach_browser" };+ }+ if (address.kind !== "machine" && address.kind !== "sandbox" || address.kind === "machine" && address.part)+ throw new ComputerRequestError("Open at computer:machine/<name>, a sandbox address, an app address or a browser window address.");+ if (what === "app") return { action: "open", app };+ if (what === "path") return { action: "open", path, ...(args ? { args } : {}), ...(debugging !== undefined ? { debugging } : {}) };+ if (address.kind === "sandbox" && what) throw new ComputerRequestError("A sandbox opens apps by app or path.");+ if (what === "browser") return { action: "open_browser", ...(browser ? { browser } : {}), ...(url ? { url } : {}), ...(profile ? { profile } : {}) };+ if (what === "sandbox") return { action: "open_sandbox", ...sandbox };+ if (what === "isolated_session") return { action: "open_session" };+ throw new ComputerRequestError("Name what to open: app, path, browser or url, sandbox, or isolated_session.");+}++const windowParameters = Type.Object({+ window: Type.String({ description: "Window address from Search, Read or a name, such as computer:zinpc/w3qk or computer:game." }),+ action: Type.Union([Type.Literal("move"), Type.Literal("resize"), Type.Literal("maximize"), Type.Literal("minimize"), Type.Literal("focus"), Type.Literal("restore")],+ { description: "focus activates a visible window; restore brings a minimized window back and activates it." }),+ x: Type.Optional(Type.Integer({ description: "move: new left edge in Windows desktop pixels." })),+ y: Type.Optional(Type.Integer({ description: "move: new top edge in Windows desktop pixels." })),+ width: Type.Optional(Type.Integer({ minimum: 1, description: "resize: new width in Windows desktop pixels." })),+ height: Type.Optional(Type.Integer({ minimum: 1, description: "resize: new height in Windows desktop pixels." })),+}, { additionalProperties: false });++/** The window management command of computer_window. */+export function windowCommand(params: Static<typeof windowParameters>): Record<string, unknown> {+ const { window: _window, action, ...fields } = params;+ const wanted = action === "move" ? ["x", "y"] : action === "resize" ? ["width", "height"] : [];+ const given = Object.keys(fields);+ if (given.some(key => !wanted.includes(key)) || wanted.some(key => !given.includes(key)))+ throw new ComputerRequestError(wanted.length ? `${action} needs ${wanted.join(" and ")} and nothing else.` : `${action} takes no coordinates or size.`);+ return { action, ...fields, ...(wanted.length ? { unit: "windows_desktop_pixels" } : {}) };+}++const displayParameters = Type.Object({+ display: Type.String({ description: "computer:machine/<name>/display/primary, or a display address listed by Read computer:machine/<name>/displays." }),+ width: Type.Integer({ minimum: 640, maximum: 16384, description: "New width in physical pixels." }),+ height: Type.Integer({ minimum: 480, maximum: 16384, description: "New height in physical pixels." }),+}, { additionalProperties: false });++/**+ * Register the typed computer_* tools that replace Insert commands one cluster at a time. They are deferred: a+ * tool_search for computer loads them, so sessions that never use the computer pay nothing for them.+ */+export function registerComputerTools(pi: ExtensionAPI, resources: ComputerResources, guide: string): void {+ const namespace = computerNamespace(guide);+ const common = { exposure: "deferred" as const, namespace, outputSchema: receiptSchema };+ const rendered = (name: string, call: (args: Record<string, unknown>) => string) => ({+ renderCall: (args: unknown) => new Text(`${name} ${plain(call(args as Record<string, unknown>))}`, 0, 0),+ renderResult: computerActionRenderer(name).renderResult,+ });+ pi.registerTool({+ ...common, name: "computer_open", label: "Computer open",+ description: "Use computer_open to start an app, a Playwright browser, a Windows Sandbox or an isolated session on a machine, or to attach Playwright to a running browser window. The result gives the new window or sandbox address.",+ parameters: openParameters,+ annotations: { readOnlyHint: false, destructiveHint: false, openWorldHint: false },+ ...rendered("computer_open", args => [args.at, args.app ?? args.path ?? args.url ?? args.browser ?? (args.sandbox ? "sandbox" : args.isolated_session ? "isolated session" : "")].filter(Boolean).join(" ")),+ async execute(_id, params, signal, _update, ctx) {+ let command: Record<string, unknown> = {};+ return actionResult(await computerAction(resources, params.at, "open", () => {+ const address = resources.address(params.at);+ command = openCommand(address, params);+ return address;+ }, address => resources.perform(address, ctx.cwd, JSON.stringify(command), "insert", signal)));+ },+ });+ pi.registerTool({+ ...common, name: "computer_window", label: "Computer window",+ description: "Use computer_window to move, resize, maximize, minimize, focus or restore an app window by its address, without keys or clicks.",+ parameters: windowParameters,+ annotations: { readOnlyHint: false, destructiveHint: false, openWorldHint: false },+ ...rendered("computer_window", args => `${args.window} ${args.action}${args.action === "move" ? ` ${args.x},${args.y}` : args.action === "resize" ? ` ${args.width}x${args.height}` : ""}`),+ async execute(_id, params, signal, _update, ctx) {+ let command: Record<string, unknown> = {};+ return actionResult(await computerAction(resources, params.window, params.action, () => {+ command = windowCommand(params);+ return resources.address(params.window);+ }, address => resources.manageWindow(address, ctx.cwd, command, signal)));+ },+ });+ pi.registerTool({+ ...common, name: "computer_display", label: "Computer display",+ description: "Use computer_display to change a display's resolution in YOLO mode, for example to test an app at another size; closing computer use restores it.",+ parameters: displayParameters,+ annotations: { readOnlyHint: false, destructiveHint: false, openWorldHint: false },+ ...rendered("computer_display", args => `${args.display} ${args.width}x${args.height}`),+ async execute(_id, params, signal, _update, ctx) {+ return actionResult(await computerAction(resources, params.display, "resize", () => resources.address(params.display),+ address => resources.perform(address, ctx.cwd, JSON.stringify({ action: "resize", width: params.width, height: params.height }), "insert", signal)));+ },+ });+}++/** Delete for computer resources when Pi Agent IDE is absent: the IDE's Delete handles them otherwise. */+export function registerComputerDelete(pi: ExtensionAPI, resources: ComputerResources, guide: string): void {+ pi.registerTool({+ name: "computer_delete", label: "Computer delete", exposure: "deferred", namespace: computerNamespace(guide), outputSchema: receiptSchema,+ description: "Use computer_delete to close an app window, or to release a sandbox, isolated session, machine or all of computer use.",+ parameters: Type.Object({ path: Type.String({ description: COMPUTER_DELETE_HELP }) }, { additionalProperties: false }),+ annotations: { readOnlyHint: false, destructiveHint: true, openWorldHint: false },+ renderCall: args => new Text(`computer_delete ${plain(args.path)}`, 0, 0),+ renderResult: computerActionRenderer("computer_delete").renderResult,+ async execute(_id, params, signal, _update, ctx) {+ return actionResult(await deleteComputer(resources, params.path, params, ctx.cwd, signal));+ },+ });+}diff --git a/src/index.ts b/src/index.tsindex c597ee0..8fb578a 100644--- a/src/index.ts+++ b/src/index.ts@@ -9,6 +9,7 @@ import { WindowHighlights } from "./window-highlights.ts";import { ComputerSession, type ComputerResult } from "./session.ts";import { registerComputerProvider } from "./provider.ts";import { registerStandaloneTools } from "./standalone-tools.ts";+import { registerComputerTools } from "./computer-tools.ts";import { WebBrowser } from "./web-browser.ts";import { localRunner } from "./runner.ts";import { MachineHost } from "./machine-host.ts";@@ -54,11 +55,16 @@ export default async function computerUse(pi: ExtensionAPI): Promise<void> {return result;}, openSandbox, connectMachine, getMode);// Every extension has loaded by session_start, so an IDE that accepts the provider has done so by now.- let standalone = false;+ let registered = false;pi.on("session_start", () => {- if (standalone || provider.attached()) return;- standalone = true;- registerStandaloneTools(pi, provider.resources, provider.guide);+ if (registered) return;+ registered = true;+ registerComputerTools(pi, provider.resources, provider.guide);+ if (!provider.attached()) registerStandaloneTools(pi, provider.resources, provider.guide);+ // Deferred tools are reachable only through tool_search, which Pi registers inactive.+ const active = pi.getActiveTools();+ if (!active.includes("tool_search") && pi.getAllTools().some(tool => tool.name === "tool_search"))+ pi.setActiveTools([...active, "tool_search"]);});pi.on("session_shutdown", async () => {if (loading) await loading.catch(() => undefined);diff --git a/src/machine-host.ts b/src/machine-host.tsindex 9733719..c1203e8 100644--- a/src/machine-host.ts+++ b/src/machine-host.ts@@ -79,6 +79,24 @@ export class MachineHost {if (folder) await this.fs.remove(folder.wsl);}+ /**+ * Post WM_CLOSE to a window of one process and wait up to `timeoutMs` for it to go: `closed` when it is gone (or was+ * already), `open` when the app kept it, `refused` when Windows would not deliver the message.+ */+ async closeWindow(pid: number, window: number, timeoutMs: number): Promise<"closed" | "open" | "refused"> {+ if (![pid, window, timeoutMs].every(value => Number.isSafeInteger(value) && value > 0)) throw new Error("Invalid window to close.");+ const out = await this.powershell(`Add-Type -Name Close -Namespace PiComputerUse -MemberDefinition '[DllImport("user32.dll")] public static extern bool IsWindow(IntPtr h); [DllImport("user32.dll")] public static extern bool PostMessage(IntPtr h, uint m, IntPtr w, IntPtr l); [DllImport("user32.dll")] public static extern uint GetWindowThreadProcessId(IntPtr h, out uint p);'+$h = [IntPtr]${window}+function Present { $p = [uint32]0; [void][PiComputerUse.Close]::GetWindowThreadProcessId($h, [ref]$p); [PiComputerUse.Close]::IsWindow($h) -and $p -eq ${pid} }+if (-not (Present)) { 'closed'; exit }+if (-not [PiComputerUse.Close]::PostMessage($h, 0x10, [IntPtr]::Zero, [IntPtr]::Zero)) { 'refused'; exit }+$deadline = (Get-Date).AddMilliseconds(${timeoutMs})+while ((Get-Date) -lt $deadline) { if (-not (Present)) { 'closed'; exit }; Start-Sleep -Milliseconds 100 }+'open'`, undefined, timeoutMs + 20_000);+ const outcome = out.trim();+ if (outcome !== "closed" && outcome !== "open" && outcome !== "refused") throw new Error(`Unexpected close result: ${outcome.slice(0, 200)}`);+ return outcome;+ }/*** Whether a window's process still handles messages, and how much memory and CPU it has used.* With a window handle, Windows' own hung-window test decides; the process's main window may be another one.diff --git a/src/provider.ts b/src/provider.tsindex c79f5ba..716bc14 100644--- a/src/provider.ts+++ b/src/provider.ts@@ -5,8 +5,8 @@ import { connectTextEditorPlugin, TEXT_EDITOR_PROTOCOL, TEXT_EDITOR_API_VERSIONimport type { ResourceResolver } from "pi-agent-ide/api/resource";import { connectAgentDocumentation } from "pi-agent-ide/api/documentation";import { readFile } from "node:fs/promises";-import { ComputerResources, requireComputerAddress, computerResultText, ComputerRequestError, LOCAL_MACHINE, type SandboxFactory, type MachineFactory } from "./computer-resources.ts";-import { computerSource, displaySource } from "./computer-address.ts";+import { ComputerResources, computerResultText, ComputerRequestError, LOCAL_MACHINE, type SandboxFactory, type MachineFactory } from "./computer-resources.ts";+import { computerSource, displaySource, type ComputerAddress } from "./computer-address.ts";import type { WebLogEntry } from "./web-browser.ts";import { diffIsEmpty, type PixelReport, type StateDiff } from "./successor.ts";import type { PixelBox } from "./pixel-diff.ts";@@ -22,6 +22,7 @@ const label = (value: unknown) => typeof value === "string" ? value.replace(/[\rexport const COMPUTER_PATH_HELP = "Read docs:computer for native resources. computer:session, computer:machine/<name> (local, or a configured machine) with its /windows, /apps and /displays, its /display/primary or listed /display/ref, and returned computer:window/ref, observation/id, window-state/id or sequence/id, each running on the machine that issued it; results show short forms such as computer:<machine>/w3qk (o observation, s sequence), and computer:<name> after Insert {\"action\":\"name\",\"as\":\"<name>\"} into a window. Read window overviews before input; display/detail/frames are readonly.";export const COMPUTER_VIEWS_HELP = "Computer: image/ui/full; dom alone or dom:{target,boxes,depth,log} on a browser window opened by open_browser or attached with attach_browser returns a fresh Playwright aria snapshot with [ref=...] values of the active tab; log:true adds the retained console/page-error/network history; target is {ref} or locator fields role,name,label,text,placeholder,test_id,css,exact,frames. On a DOM observation: image is a page screenshot, element:crop|highlight,<ref> an element image; Search filters its lines. state alone on a window returns fresh restore or management metadata without pixels/UI, plus whether the window is hung and its process memory/CPU. ocr (alone or with capture options on a window, alone on an observation, or with region:x,y,width,height on an observation) adds Windows OCR lines with boxes in overview pixels; use maxLongEdge 3840 for small text. aria (alone or with ui capture options, on a window) returns its UI as a Playwright aria snapshot with hints as [ref=...]. changes (alone, or with ui, image or full and capture options, on a window) takes a fresh capture and lists only rows changed, added or removed since that window's previous observation, with the new observation's hints, plus changed pixel areas with before/after crops instead of the full image. regions (alone or with image/full options on a window, alone on its observation) lists icons and text areas found by Cua perception with boxes in overview pixels; click a box center with pixel input. capture:{maxLongEdge,scale,max_elements,max_depth,hints,guides}. region:x,y,width,height or element:inspect|scope|highlight|crop|text,hint on an observation; element:text returns one element's full text. sequence:{duration_ms<=60000,fps<=30,max_frames<=300,timeout_ms,maxLongEdge,scale,region} on window/display reports actual_fps and last_frame_age_ms; frames:first|last|all|endpoints|changed or frames:{indexes:[0]} / {from:0,to:2} / {select:'changed'} on sequence; add as:'sheet' (one tiled image with index/time labels, optional columns) or as:'max'|'min' (one per-pixel composite), with optional region:{x,y,width,height} in frame pixels and maxLongEdge. Detail/frames are readonly; text offset/limit select UI lines, never image cells.";export const COMPUTER_SEARCH_HELP = "Search computer:machine/<name>/apps or computer:machine/<name>/windows (local is this machine) for installed-app/permitted-window metadata, or an observation for its collected UI/readonly DOM. Read computer:machine/<name>/displays for monitor metadata. Discovery launches nothing; observation Search performs no new native walk or DOM query. Treat missing matches in a partial projection as unknown.";+export const COMPUTER_DELETE_HELP = "A computer window address closes that window like its close button (add ?force=true to kill the hung app of a window this session opened); a sandbox, isolated session or machine address releases it; computer:session releases everything computer use started.";export const COMPUTER_INPUT_HELP = "Computer Write sends content literally through type_text to a fresh UI/image observation address. On a DOM observation of a browser opened with Insert {\"action\":\"open_browser\"} (YOLO: \"profile\":\"agent\" keeps the agent's own logins between sessions), or of a running Chromium-based browser attached in YOLO with Insert {\"action\":\"attach_browser\"} into its window address (Chrome or Edge through a debugging port, the chrome://inspect/#remote-debugging switch or the Playwright Extension; an Electron/CEF app started with --remote-debugging-port=0 in open's args; a WebView2 app started with open \"debugging\":true or after {\"action\":\"webview2_debugging\",\"app\":\"x.exe\",\"enabled\":true} on its machine; VS Code's integrated browser through the Pi Browser Bridge; closing the session detaches), Insert one Playwright action such as {\"action\":\"fill\",\"target\":{\"label\":\"Name\"},\"text\":\"value\"}; actions target the active tab, and tab/navigation/dialog actions are listed in docs:computer. Insert {\"action\":\"open_sandbox\"} into a machine address to work in a separate disposable Windows Sandbox that does not touch the human's desktop, or {\"action\":\"open_session\"} for an isolated agent session on the same machine (several can run at once, each with its own desktop and foreground input); use the returned computer:sandbox/... address for Search, open and close (docs:computer). Machines are computer:machine/local (this one) and computer:machine/<configured name> (another Windows machine over SSH, started on first use); each takes the same commands. Insert {\"action\":\"put\",\"from\":\"<workspace file>\"}, {\"action\":\"get\",\"name\":\"<file>\",\"to\":\"<workspace path>\"} or {\"action\":\"files\"} into a machine address to move files through the session's own folder on that machine (removed on close); point app Open/Save dialogs there. The same addresses take {\"action\":\"clipboard_write\",\"text\"|\"image\":...}, {\"action\":\"clipboard_read\",\"image\":true}, {\"action\":\"record_start\",\"seconds\":30} and {\"action\":\"record_stop\",\"to\":\"<workspace path>\"} (main-display video) and {\"action\":\"audio_record\",\"seconds\":10,\"transcribe\":true} (playback audio with a transcript) in YOLO. Insert {\"action\":\"timeline\",\"events\":[{at_ms,type,...}]} with delivery=foreground into an observation for timed key/button/move/move_relative/wheel/pen events; everything held is released at the end (docs:computer). Insert {\"action\":\"resize\",\"width\":1920,\"height\":1080} into a display address to change that display's resolution in YOLO; closing the session restores it. Only after the user explicitly allows it, Insert {\"action\":\"setup\",\"part\":\"browser|firefox|perception|mxc|vscode-bridge\"} into a machine address to install a missing optional part there (docs:computer). Insert {\"action\":\"menu\",\"path\":[\"File\",\"Save As...\"]} into an observation to invoke a menu item by its labels, and {\"action\":\"kill\"} into the window address of an app this session opened to force-close it. Insert an action with a Playwright locator target, such as {\"action\":\"click\",\"target\":\"getByRole('button', { name: 'Save' })\"} or fill/check/select, into a native window address to act without a prior Read; it waits until exactly one element is ready. Insert {\"action\":\"wait_for\"|\"verify\",\"conditions\":[...]} into a window address to check UI state with the same locators, count or an aria snapshot template; every action reports what it changed in the window: UI rows, and changed pixel areas with before/after crops after an action on an image or full observation (\"successor\":false skips it), and \"expect\" adds a postcondition; or Insert {\"action\":\"batch\",\"steps\":[...]} into an observation for a fail-stop chain (docs:computer). Use no Write or native input on a DOM observation. Use optional ?hint=...&delivery=foreground&delay_ms=0 only on input observations. No implicit select-all, submit or program execution. Computer Insert sends {\"action\":\"restore\"} to a fresh window-state address for explicit restoration and activation without keys/clicks, or one JSON action to an observation, or {\"action\":\"open\",\"app\":\"Exact installed name\"} (YOLO also {\"action\":\"open\",\"path\":\"C:\\\\...\\\\app.exe\",\"args\":[...]}) to a machine address, and {\"action\":\"close\"} to a machine address or to computer:session for every machine (the result lists each released part). Close releases runtime, not documents. Omit window_ref/observation_id in JSON. Input consumes observations even on refusal; the action's successor observation is current for the next input. With \"successor\":false, delivered pixel-only input keeps its observation until the window is resized; otherwise refresh with Read. One sequence takes up to 2000 key, click and stroke steps. Never replay unknown prefixes. Foreground input requires an observed background refusal and pauses for 10 s after the human's own keyboard or mouse input; restore is a separate explicit activation, not input delivery. Safe/YOLO is user-controlled.";/** Present native resources without dumping provenance; full receipts remain in structured data. */@@ -355,13 +356,23 @@ export async function performComputer(resources: ComputerResources, tool: "writeconst literal = input[tool === "write" ? "content" : "text"];let action: string | null = tool === "write" ? "type_text" : null;if (tool === "insert" && typeof literal === "string") try { const parsed = JSON.parse(literal); if (typeof parsed?.action === "string") action = parsed.action; } catch { /* reported by perform */ }- let address: ReturnType<typeof requireComputerAddress> | undefined;- try {+ return computerAction(resources, source, action, () => {const fields = tool === "write" ? ["path", "content"] : ["path", "text"];if (Object.keys(input).some(key => !fields.includes(key))) throw new ComputerRequestError("Computer input accepts no text anchors, ranges or insertion flags; its observation address binds the target.");if (typeof literal !== "string") throw new ComputerRequestError("Provide literal content for Write or one JSON command in Insert text.");- address = resources.address(source);- const result = await resources.perform(address, cwd, literal, tool, signal);+ return resources.address(source);+ }, address => resources.perform(address, cwd, literal as string, tool, signal));+}+/**+ * Run one computer operation and present its outcome as an action result: the shortened summary, any images, and the+ * receipt that scripts branch on. `resolve` validates the request; its refusal is a not-applied receipt.+ */+export async function computerAction(resources: ComputerResources, source: string, action: string | null,+ resolve: () => ComputerAddress, run: (address: ComputerAddress) => Promise<ComputerResult>) {+ let address: ComputerAddress | undefined;+ try {+ address = resolve();+ const result = await run(address);const effect = computerEffect(result);// Completion of the native request is separate from application-effect certainty.const ok = !result.isError && ["delivered", "observed", "closed"].includes(result.details.status);@@ -381,6 +392,13 @@ export async function performComputer(resources: ComputerResources, tool: "writescript: { ...shortReceipt(resources, computerReceipt(source, action, { error, inputConsumed: address ? !resources.grantsAuthority(address) : false })) } };}}+/** Close what a computer address names through Delete. */+export function deleteComputer(resources: ComputerResources, source: string, input: Record<string, unknown>, cwd: string, signal?: AbortSignal) {+ return computerAction(resources, source, "close", () => {+ if (Object.keys(input).some(key => key !== "path")) throw new ComputerRequestError("Delete of a computer address takes only its path; it closes the whole resource.");+ return resources.address(source.replace(/\?force=true$/, ""));+ }, () => resources.delete(source, cwd, signal));+}/** A receipt whose window, sandbox and observation addresses are this run's short aliases. */function shortReceipt(resources: ComputerResources, receipt: ComputerReceipt): ComputerReceipt {const short = (value: string | null | undefined) => value ? resources.shorten(value) : value;@@ -463,13 +481,18 @@ export async function registerComputerProvider(pi: ExtensionAPI, getSession: (cwlet renderersRegistered = false;pi.on("session_start", () => {if (renderersRegistered) return;- for (const tool of ["write", "insert"] as const) api.addToolRenderer(computerActionRenderer(tool));+ for (const tool of ["write", "insert", "delete"] as const) api.addToolRenderer(computerActionRenderer(tool));renderersRegistered = true;});for (const tool of ["write", "insert"] as const) api.tool(tool).addSemanticHandler({matches: input => owns(record(input).path),execute: (context, input) => performComputer(resources, tool, context.sourceFor("path"), record(input), context.cwd, context.signal),});+ api.tool("delete").addSemanticHandler({+ matches: input => owns(record(input).path),+ execute: (context, input) => deleteComputer(resources, context.sourceFor("path"), record(input), context.cwd, context.signal),+ });+ api.tool("delete").describeField("path", COMPUTER_DELETE_HELP);api.describe(COMPUTER_INPUT_HELP);} })),]);diff --git a/src/session.ts b/src/session.tsindex ca415a6..69603ba 100644--- a/src/session.ts+++ b/src/session.ts@@ -1329,6 +1329,38 @@ export class ComputerSession {});}+ /**+ * Ask a window to close, like its close button: a browser this session opened closes with its disposable profile,+ * an attached one is detached first, other windows get WM_CLOSE. The app may keep the window open, for example to+ * ask about unsaved work; the result says whether it closed.+ */+ closeWindow(ref: string, signal?: AbortSignal): Promise<ComputerResult> {+ return this.queue(async () => {+ try {+ const known = this.entries.get(ref);+ if (known && !this.allowed(known)) return this.denied(ref);+ const entry = await this.current(ref, signal);+ if (!entry) return this.result("refused", "Unknown or closed window. Discover again.");+ if (!this.allowed(entry)) return this.denied(ref);+ if (signal?.aborted) return this.result("refused", "Cancelled before closing the window.");+ const key = `${entry.target.pid}:${entry.target.window_id}`, browser = this.browsers.get(key);+ if (browser) {+ this.browsers.delete(key);+ entry.web = undefined; entry.dom = undefined;+ await browser.close().catch(() => undefined);+ }+ if (!this.host) {+ if (browser) { this.entries.delete(ref); return this.result("delivered", "The browser this session opened was closed.", undefined, this.addressed(entry)); }+ return this.result("refused", "Closing windows needs the machine's own Windows host, which this environment lacks. Click the window's close button, or Delete it with ?force=true when this session opened it.", undefined, this.addressed(entry));+ }+ const outcome = await this.host.closeWindow(entry.target.pid, entry.target.window_id, 5000);+ if (outcome === "refused") return this.result("refused", "Windows refused the close request; the window may belong to an elevated app. Click its close button instead.", undefined, this.addressed(entry));+ if (outcome === "open") return this.result("delivered", "Close was requested, but the window is still open after 5 s; it may be asking about unsaved work. Read it to see what it shows.", { closed: false }, this.addressed(entry));+ this.entries.delete(ref);+ return this.result("closed", "The window closed; its address expired.", { closed: true }, this.addressed(entry));+ } catch (error) { return this.failed(error, true); }+ });+ }/*** Force-close the process of a window this session opened, when every window of that process is session-owned.* Unsaved work in it is lost; a process that also shows other windows is never killed.diff --git a/src/standalone-tools.ts b/src/standalone-tools.tsindex e460367..5691782 100644--- a/src/standalone-tools.ts+++ b/src/standalone-tools.ts@@ -4,6 +4,7 @@ import { Text } from "@earendil-works/pi-tui";import type { ComputerResources } from "./computer-resources.ts";import { computerActionRenderer } from "./computer-action-renderer.ts";import { computerReadRenderer } from "./computer-read-renderer.ts";+import { actionResult, computerNamespace, receiptSchema, registerComputerDelete } from "./computer-tools.ts";import {COMPUTER_INPUT_HELP, COMPUTER_PATH_HELP, COMPUTER_SEARCH_HELP, COMPUTER_VIEWS_HELP,performComputer, presentComputerResult, readComputer,@@ -18,11 +19,7 @@ const plain = (value: unknown) => String(value).replace(/[\x00-\x1f\x7f-\x9f]/g,* tool_search finds. Tools stay off the model's tool list until a search loads them.*/export function registerStandaloneTools(pi: ExtensionAPI, resources: ComputerResources, guide: string): void {- const namespace = {- name: "computer",- description: "Read and operate Windows app windows, displays and browser pages through computer: addresses",- instructions: guide,- };+ const namespace = computerNamespace(guide);pi.registerTool({name: "computer_read", label: "Computer read", exposure: "deferred", namespace,description: `Use computer_read to discover machines, see app windows, displays and browser pages, and get the observations that ground input. Read ${GUIDE} first for addresses, views and commands.`,@@ -72,8 +69,6 @@ export function registerStandaloneTools(pi: ExtensionAPI, resources: ComputerResreturn { content: [{ type: "text", text: resources.shorten(presentComputerResult(result)) }], details: result.details, ...(result.isError ? { isError: true } : {}) };},});- // Codemode scripts receive the receipt object for every outcome, so they branch on outcome without try/catch.- const receiptSchema = Type.Object({ outcome: Type.String(), effect: Type.String() }, { additionalProperties: true });for (const tool of ["write", "insert"] as const) {const field = tool === "write" ? "content" : "text";const renderer = computerActionRenderer(tool);@@ -94,16 +89,9 @@ export function registerStandaloneTools(pi: ExtensionAPI, resources: ComputerResrenderResult: renderer.renderResult,async execute(_id, params, signal, _update, ctx) {const input = params as Record<string, unknown>;- const action = await performComputer(resources, tool, String(input.path), input, ctx.cwd, signal);- // The receipt is plain data with a few untyped native fields; a JSON copy gives scripts exactly that.- const receipt: JsonValue = JSON.parse(JSON.stringify(action.script));- return { content: [{ type: "text", text: action.summary }, ...("images" in action ? action.images ?? [] : [])], details: { results: [], metadata: { semanticAction: action.data } },- structuredContent: receipt, ...(action.data.ok ? {} : { isError: true }) };+ return actionResult(await performComputer(resources, tool, String(input.path), input, ctx.cwd, signal));},});}- // Deferred tools are reachable only through tool_search, which Pi registers inactive.- const active = pi.getActiveTools();- if (!active.includes("tool_search") && pi.getAllTools().some(tool => tool.name === "tool_search"))- pi.setActiveTools([...active, "tool_search"]);+ registerComputerDelete(pi, resources, guide);}diff --git a/test/computer-tools.test.ts b/test/computer-tools.test.tsnew file mode 100644index 0000000..9999445--- /dev/null+++ b/test/computer-tools.test.ts@@ -0,0 +1,89 @@+import { expect, test, vi } from "vitest";+import { requireComputerAddress, ComputerResources } from "../src/computer-resources.ts";+import { openCommand, windowCommand } from "../src/computer-tools.ts";+import type { MachineHost } from "../src/machine-host.ts";+import { deleteComputer } from "../src/provider.ts";+import { ComputerSession, type ComputerMode, type Driver } from "../src/session.ts";++const cwd = "/synthetic-computer-tools";+const at = (source: string) => requireComputerAddress(source);++// LPT-934: each typed tool turns its fields into exactly one existing command.+test("computer_open opens one thing, chosen by its fields and the address", () => {+ const machine = at("computer:machine/lab");+ expect(openCommand(machine, { at: "", app: "Notepad" })).toEqual({ action: "open", app: "Notepad" });+ expect(openCommand(machine, { at: "", path: "C:\\Tools\\app.exe", args: ["--x"], debugging: true })).toEqual({ action: "open", path: "C:\\Tools\\app.exe", args: ["--x"], debugging: true });+ expect(openCommand(machine, { at: "", url: "https://example.com" })).toEqual({ action: "open_browser", url: "https://example.com" });+ expect(openCommand(machine, { at: "", browser: "msedge", profile: "agent" })).toEqual({ action: "open_browser", browser: "msedge", profile: "agent" });+ expect(openCommand(machine, { at: "", sandbox: { export: "C:\\out" } })).toEqual({ action: "open_sandbox", export: "C:\\out" });+ expect(openCommand(machine, { at: "", isolated_session: true })).toEqual({ action: "open_session" });+ expect(openCommand(at("computer:app/a1"), { at: "" })).toEqual({ action: "open" });+ expect(openCommand(at("computer:window/w1"), { at: "" })).toEqual({ action: "attach_browser" });+ expect(openCommand(at("computer:sandbox/s1"), { at: "", app: "Notepad" })).toEqual({ action: "open", app: "Notepad" });+ expect(() => openCommand(machine, { at: "", app: "Notepad", url: "https://example.com" })).toThrow("Open one thing at a time: app and browser");+ expect(() => openCommand(machine, { at: "", app: "Notepad", args: ["x"] })).toThrow("args and debugging go with path");+ expect(() => openCommand(machine, { at: "" })).toThrow("Name what to open");+ expect(() => openCommand(at("computer:window/w1"), { at: "", app: "Notepad" })).toThrow("A window address attaches Playwright");+ expect(() => openCommand(at("computer:sandbox/s1"), { at: "", url: "https://example.com" })).toThrow("A sandbox opens apps by app or path");+ expect(() => openCommand(at("computer:machine/lab/windows"), { at: "", app: "Notepad" })).toThrow("Open at computer:machine/<name>");+});++test("computer_window takes exactly the fields of its action", () => {+ expect(windowCommand({ window: "w", action: "move", x: -5, y: 10 })).toEqual({ action: "move", x: -5, y: 10, unit: "windows_desktop_pixels" });+ expect(windowCommand({ window: "w", action: "resize", width: 800, height: 600 })).toEqual({ action: "resize", width: 800, height: 600, unit: "windows_desktop_pixels" });+ expect(windowCommand({ window: "w", action: "restore" })).toEqual({ action: "restore" });+ expect(() => windowCommand({ window: "w", action: "move", x: 1 })).toThrow("move needs x and y and nothing else.");+ expect(() => windowCommand({ window: "w", action: "focus", width: 1 })).toThrow("focus takes no coordinates or size.");+});++/** One window (pid 10, id 7) a session can list and observe; the host decides how a close request ends. */+function fixture(outcome: "closed" | "open" | "refused") {+ let open = true, mode: ComputerMode = "yolo";+ const driver: Driver = {+ identity: vi.fn(async () => open ? "life-7" : undefined) as Driver["identity"],+ close: vi.fn(async () => {}),+ call: vi.fn<Driver["call"]>(async name => {+ if (name === "list_windows") return { content: [], structuredContent: { windows: open ? [{ pid: 10, window_id: 7, title: "Editor" }] : [] } };+ throw new Error(`Unexpected native call ${name}`);+ }),+ };+ const closeWindow = vi.fn(async () => { if (outcome === "closed") open = false; return outcome; });+ const session = new ComputerSession(driver, () => mode, undefined, undefined, undefined, undefined, { closeWindow, removeFiles: async () => {} } as unknown as MachineHost);+ const resources = new ComputerResources(async () => session, async () => session.close());+ const window = async () => ((await resources.search(at("computer:machine/local/windows"), cwd, "Editor")).details.state!.windows as { window_ref: string }[])[0].window_ref;+ const remove = (source: string) => deleteComputer(resources, source, { path: source }, cwd);+ return { resources, closeWindow, window, remove, safe: () => { mode = "safe"; } };+}++test("Delete closes a window like its close button, and its address expires", async () => {+ const f = fixture("closed");+ const ref = await f.window();+ const closed = await f.remove(`computer:window/${ref}`);+ expect(closed.data).toMatchObject({ ok: true, status: "closed", machine: "local" });+ expect(closed.script).toMatchObject({ action: "close", outcome: "closed" });+ expect(f.closeWindow).toHaveBeenCalledWith(10, 7, 5000);+ expect((await f.remove(`computer:window/${ref}`)).summary).toContain("Unknown or closed window reference");+});++test("Delete reports a window that stays open and refuses what it cannot close", async () => {+ const kept = fixture("open");+ const ref = await kept.window();+ const result = await kept.remove(`computer:window/${ref}`);+ expect(result.data).toMatchObject({ ok: true, status: "delivered" });+ expect(result.summary).toContain("still open after 5 s");+ // Force kills only processes this session started.+ expect((await kept.remove(`computer:window/${ref}?force=true`)).summary).toContain("Only windows this session opened can be force-closed");+ expect((await kept.remove("computer:machine/local?force=true")).summary).toContain("?force=true applies to window addresses only.");+ expect((await kept.remove("computer:machine/local/windows")).summary).toContain("Delete closes a window, a sandbox or isolated session, a machine or computer:session.");++ // Safe mode closes only windows this session opened, like any other input.+ const safe = fixture("closed");+ const foreign = await safe.window();+ safe.safe();+ const refused = await safe.remove(`computer:window/${foreign}`);+ expect(refused.data).toMatchObject({ ok: false, status: "refused" });+ expect(refused.summary).toContain("Safe mode allows only windows opened by this session.");+ expect(safe.closeWindow).not.toHaveBeenCalled();+ const session = await kept.remove("computer:session");+ expect(session.data).toMatchObject({ ok: true });+});diff --git a/test/extension-modes.integration.test.ts b/test/extension-modes.integration.test.tsindex 16b3c35..bcbd89e 100644--- a/test/extension-modes.integration.test.ts+++ b/test/extension-modes.integration.test.ts@@ -31,11 +31,19 @@ test("without Pi Agent IDE, tool_search finds the deferred computer tools and th{ id: "guide", name: "computer_read", arguments: { path: "docs:computer" } },{ id: "session", name: "computer_read", arguments: { path: "computer:session" } },{ id: "close", name: "computer_insert", arguments: { path: "computer:session", text: '{"action":"close"}' } },+ { id: "display", name: "computer_display", arguments: { display: "computer:display/primary", width: 1920, height: 1080 } },+ { id: "delete", name: "computer_delete", arguments: { path: "computer:session" } },]);- expect(getToolCallNames(result)).toEqual(["tool_search", "computer_read", "computer_read", "computer_insert"]);- for (const id of ["find", "guide", "session", "close"])+ expect(getToolCallNames(result)).toEqual(["tool_search", "computer_read", "computer_read", "computer_insert", "computer_display", "computer_delete"]);+ for (const id of ["find", "guide", "session", "close", "delete"])expect(getToolExecution(result, id).isError, getToolResultText(result, id)).toBe(false);expect(getToolResultText(result, "find")).toContain("computer_read");+ // One search for computer loads the whole namespace.+ for (const tool of ["computer_open", "computer_window", "computer_display", "computer_delete"]) expect(getToolResultText(result, "find")).toContain(tool);+ // A typed tool refuses a bad address with a receipt before any native work.+ expect(getToolExecution(result, "display").isError).toBe(true);+ expect(getToolResultText(result, "display")).toContain("Display addresses name their machine");+ expect(getToolResultText(result, "delete")).toContain("Computer closed");expect(getToolResultText(result, "guide")).toMatch(/^# Computer resources/);expect(getToolResultText(result, "session")).toContain("computer:machine/local");expect(getToolResultText(result, "close")).toContain("Computer closed");@@ -51,17 +59,21 @@ test("an IDE that refuses this plugin API version leaves computer use standaloneexpect(getToolResultText(result, "session")).toContain("computer:machine/local");}, 120_000);-test("with Pi Agent IDE, computer use stays an IDE plugin and adds no computer tools", async () => {+test("with Pi Agent IDE, computer use extends Read, Search, Insert and Delete and adds only the typed computer tools", async () => {for (const extensions of [[ide, computerUse, "builtin:tool-search"], [computerUse, ide, "builtin:tool-search"]]) {- const result = await run("with-ide", extensions, ["read", "insert", "tool_search"], [+ const result = await run("with-ide", extensions, ["read", "insert", "delete", "tool_search"], [{ id: "find", name: "tool_search", arguments: { query: "computer read windows displays" } },{ id: "session", name: "read", arguments: { path: "computer:session" } },{ id: "close", name: "insert", arguments: { path: "computer:session", text: '{"action":"close"}' } },+ { id: "delete", name: "delete", arguments: { path: "computer:session" } },]);expect(getToolResultText(result, "find")).not.toContain("computer_read");- for (const id of ["session", "close"])+ expect(getToolResultText(result, "find")).not.toContain("computer_delete");+ for (const tool of ["computer_open", "computer_window", "computer_display"]) expect(getToolResultText(result, "find")).toContain(tool);+ for (const id of ["session", "close", "delete"])expect(getToolExecution(result, id).isError, getToolResultText(result, id)).toBe(false);expect(getToolResultText(result, "session")).toContain("computer:session");expect(getToolResultText(result, "close")).toContain("Computer closed");+ expect(getToolResultText(result, "delete")).toContain("Computer closed");}}, 240_000);diff --git a/test/fixtures/window-management-extension.ts b/test/fixtures/window-management-extension.tsindex 3af2062..f407c58 100644--- a/test/fixtures/window-management-extension.ts+++ b/test/fixtures/window-management-extension.ts@@ -29,8 +29,8 @@ function semantic(result: Result): Record<string, unknown> {export default async function fixture(pi: ExtensionAPI): Promise<void> {const source = process.env.PI_WINDOW_MANAGEMENT_PROVIDER_SOURCE ?? path.resolve(import.meta.dirname, "../..");const load = (name: string) => import(pathToFileURL(path.join(source, "src", name)).href);- const [{ registerComputerProvider }, { loadComputerConfig }, { McpDriver }, { ComputerSession }] =- await Promise.all([load("provider.ts"), load("config.ts"), load("driver.ts"), load("session.ts")]);+ const [{ registerComputerProvider }, { loadComputerConfig }, { McpDriver }, { ComputerSession }, { registerComputerTools }] =+ await Promise.all([load("provider.ts"), load("config.ts"), load("driver.ts"), load("session.ts"), load("computer-tools.ts")]);const Base = McpDriver as typeof import("../../src/driver.ts").McpDriver;let management = true, restoration = true, mode: ComputerMode = "yolo";let session: ComputerSession | undefined, driver: FixtureDriver | undefined, log: string | undefined;@@ -58,11 +58,12 @@ export default async function fixture(pi: ExtensionAPI): Promise<void> {const close = async (): Promise<ComputerResult> => session ? session.close() : {content: [{ type: "text", text: "No synthetic runtime is running." }], details: { status: "closed", mode },};- await registerComputerProvider(pi, getSession, close);+ const provider = await registerComputerProvider(pi, getSession, close);+ registerComputerTools(pi, provider.resources, provider.guide);pi.registerTool({name: "verify_window_management", label: "Verify one-use window management",description: "Exercise real IDE state and management with synthetic MCP receipts only.",- parameters: Type.Object({ scenario: Type.Union([Type.Literal("workflow"), Type.Literal("admission"), Type.Literal("receipts"), Type.Literal("restore")]) }),+ parameters: Type.Object({ scenario: Type.Union([Type.Literal("workflow"), Type.Literal("admission"), Type.Literal("receipts"), Type.Literal("restore"), Type.Literal("typed")]) }),async execute(_id, { scenario }, signal, _update, ctx) {const call = async (name: string, args: Record<string, unknown>, callSignal = signal): Promise<Result> => {try { const result = await ctx.executeTool(name, args, { signal: callSignal }); return { ...result.result, isError: result.isError } as Result; }@@ -90,7 +91,7 @@ export default async function fixture(pi: ExtensionAPI): Promise<void> {const insert = (state: string, command: Record<string, unknown>) => call("insert", { path: state, text: JSON.stringify(command) });const move = { action: "move", x: -20, y: 120, unit: "windows_desktop_pixels" };const open = async () => reference(await ok("insert", { path: "computer:machine/local", text: '{"action":"open","app":"Notepad"}' }), "window");- assert(!pi.getAllTools().some(tool => /^computer_/.test(tool.name)), "Separate computer tools leaked into the IDE provider");+ assert(!pi.getAllTools().some(tool => /^computer_(read|search|write|insert)$/.test(tool.name)), "Standalone computer tools leaked into the IDE provider");const first = await open(), second = await open();const listed = await ok("search", { path: "computer:machine/local/windows", query: "Owned management fixture" });assert(body(listed).includes(first) && body(listed).includes(second), "Search lost exact-source references");@@ -167,6 +168,31 @@ export default async function fixture(pi: ExtensionAPI): Promise<void> {"Post-dispatch reuse became success or discarded completed prefix");await denied("insert", { path: state, text: JSON.stringify(move) });assert(await count("manage_window") === 1, "Post-dispatch reuse was replayed");+ } else if (scenario === "typed") {+ // computer_window samples its own one-use state: no window-state address, and invalid fields never reach native code.+ const window = (args: Record<string, unknown>, target = first) => call("computer_window", { window: target, ...args });+ for (const bad of [{ action: "move", x: 1 }, { action: "maximize", x: 1 }, { action: "resize", width: 10 }, { action: "restore", y: 2 }]) {+ const result = await window(bad);+ assert(result.isError, `computer_window accepted ${JSON.stringify(bad)}: ${body(result)}`);+ }+ await denied("computer_window", { window: state, action: "focus" });+ assert(await count("manage_window") === 0, "Invalid computer_window calls reached native management");+ const moved = await window({ action: "move", x: -20, y: 120 });+ assert(!moved.isError && semantic(moved).effect === "applied" && body(moved).includes("on local"), `computer_window move failed: ${body(moved)}`);+ await control("reset");+ const resized = await window({ action: "resize", width: 180, height: 120 });+ assert(!resized.isError && semantic(resized).effect === "applied", `computer_window resize failed: ${body(resized)}`);+ await control("reset");+ assert(!(await window({ action: "maximize" })).isError, "computer_window maximize failed");+ await control("reset"); await control("minimized");+ const restored = await window({ action: "restore" });+ assert(!restored.isError && find(restored, "restore_requested") === true, `computer_window restore failed: ${body(restored)}`);+ await control("reset");+ const external = reference(await ok("search", { path: "computer:machine/local/windows", query: "Unowned management fixture" }), "window");+ mode = "safe";+ const refused = await window({ action: "focus" }, external);+ assert(refused.isError && semantic(refused).effect === "not-applied", `Safe mode let computer_window manage a foreign window: ${body(refused)}`);+ mode = "yolo";} else if (scenario === "receipts") {for (const operation of ["refused", "partial", "final_cancel", "wrong_state", "wrong_window", "wrong_operation", "wrong_route","missing_focus", "missing_prefix", "missing_receipt", "wrong_request", "wrong_before", "constrained", "changed_source", "contradictory_foreground"]) {diff --git a/test/mode.test.ts b/test/mode.test.tsindex 345ec8f..d09df8e 100644--- a/test/mode.test.ts+++ b/test/mode.test.ts@@ -22,6 +22,7 @@ async function fixture(branch: unknown[] = []) {const previous = events.get(name);events.set(name, previous ? async (event, context) => { await previous(event, context); return handler(event, context); } : handler);},+ getActiveTools: () => [], getAllTools: () => [], setActiveTools: vi.fn(),appendEntry } as unknown as ExtensionAPI);return { commands, tools, events, appendEntry, ctx,close: vi.mocked(registerComputerProvider).mock.calls.at(-1)![2] };diff --git a/test/window-management.integration.test.ts b/test/window-management.integration.test.tsindex fab68fa..68dcd1e 100644--- a/test/window-management.integration.test.ts+++ b/test/window-management.integration.test.ts@@ -10,7 +10,7 @@ const driver = path.join(root, "test/fixtures/window-management-driver.mjs");// Pattern: executable real-Pi extension example and the ordinary-IDE restore suite.// Contract: native desktop geometry is one-use management authority, never pixel/input authority.-for (const scenario of ["workflow", "admission", "receipts", "restore"] as const) test.each([false, true])(+for (const scenario of ["workflow", "admission", "receipts", "restore", "typed"] as const) test.each([false, true])(`ordinary IDE window management protects ${scenario} (providerFirst=%s)`,async providerFirst => {const extension = IDE_EXTENSION;@@ -33,7 +33,7 @@ for (const scenario of ["workflow", "admission", "receipts", "restore"] as constexpect(getToolResultText(result, "management-proof")).toContain(`IDE window management ${scenario} contract passed`);const rows = (await readFile(path.join(cwd, "calls.jsonl"), "utf8")).trim().split("\n").filter(Boolean).map(line => JSON.parse(line));const mutations = rows.filter(row => row.name === "manage_window");- expect(mutations).toHaveLength({ workflow: 6, admission: 1, receipts: 16, restore: 3 }[scenario]);+ expect(mutations).toHaveLength({ workflow: 6, admission: 1, receipts: 16, restore: 3, typed: 3 }[scenario]);for (const { arguments: args } of mutations) {const fields = args.operation === "move" ? ["x", "y", "unit"] : args.operation === "resize" ? ["width", "height", "unit"] : [];expect(Object.keys(args).sort()).toEqual(["pid", "window_id", "state_id", "operation", ...fields].sort());@@ -46,7 +46,8 @@ for (const scenario of ["workflow", "admission", "receipts", "restore"] as constexpect(mutations[1].arguments).toMatchObject({ width: 180, height: 120 });}const restores = rows.filter(row => row.name === "restore_window");- expect(restores).toHaveLength(scenario === "restore" ? 2 : 0);+ if (scenario === "typed") expect(mutations.map(row => row.arguments.operation)).toEqual(["move", "resize", "maximize"]);+ expect(restores).toHaveLength(scenario === "restore" ? 2 : scenario === "typed" ? 1 : 0);for (const row of restores) expect(Object.keys(row.arguments).sort()).toEqual(["pid", "state_id", "window_id"]);expect(rows.filter(row => ["click", "type_text", "hotkey", "press_key", "bring_to_front", "set_window_frame", "move_pointer", "scroll_at", "get_desktop_state", "zoom"].includes(row.name))).toEqual([]);expect(result.tuiRenderedOutput).not.toContain("already handles");