Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.
TypeScript accessibilityagent-toolsai-agentbrowser-automationcomputer-usecuadesktop-automationelectronpipi-coding-agentpi-extensionpi-packageplaywrighttypescriptui-automationvisionwebview2windowswindows-automationwsl
Make the machine host work when Pi runs directly on Windows (LPT-867)
alexshpunt-clanker[bot]11aee1b633c9
diff --git a/.pi/skills/computer-use/SKILL.md b/.pi/skills/computer-use/SKILL.mdindex 48e5074..5311c1d 100644--- a/.pi/skills/computer-use/SKILL.md+++ b/.pi/skills/computer-use/SKILL.md@@ -59,6 +59,6 @@ Never install on your own. Tell the user what is missing, what it adds, its size## Access and cleanup-Only the user changes Safe/YOLO with `/computer-mode`. Safe limits UI access to session-opened windows and forbids displays; YOLO permits authorized existing windows. Neither mode is OS isolation or permission for unrelated tasks. Current live support is Windows controlled from WSL, on this machine or on a configured machine over SSH.+Only the user changes Safe/YOLO with `/computer-mode`. Safe limits UI access to session-opened windows and forbids displays; YOLO permits authorized existing windows. Neither mode is OS isolation or permission for unrelated tasks. Current live support is Windows, with Pi on it directly or in WSL, on this machine or on a configured machine over SSH.Close only task-owned disposable windows through grounded input first. Insert `{"action":"close"}` into a machine address to release that machine, or into `computer:session` to release everything this computer use started; the result lists each released part and its outcome. Closing ends the browsers the session opened, deletes their profiles, detaches from attached browsers and destroys sandboxes and sessions; it does not save or close unrelated documents. Discard stored references after close/reload. Preserve paused user tasks, personal windows and documents.diff --git a/docs/computer-resources.md b/docs/computer-resources.mdindex f2be58e..2172333 100644--- a/docs/computer-resources.md+++ b/docs/computer-resources.md@@ -7,7 +7,7 @@ Computer use works with or without Pi Agent IDE. Each Pi session picks the modeBoth modes take the same addresses, views, JSON commands and Codemode values. In this guide, Read, Search, Write and Insert mean either tool family. Without IDE, Read `docs:computer` with `computer_read` for this guide. The `/computer-mode`, `/computer-debug` and `/computer-setup` commands remain user controls. An agent installs an optional part with Insert `{"action":"setup","part":"browser|firefox|perception|mxc|vscode-bridge"}` into a machine address, or prepares another machine with `{"action":"setup","part":"machine","ssh":"<alias>","name":"<name>"}` into `computer:machine/local`, only after the user explicitly allows it. Setup does not start that machine's driver.-The current desktop scope is Windows controlled from WSL, on this machine or on another one over SSH. A different interface does not qualify a new runtime or platform. The production `windows-desktop` profile selects the same checked bytes as `windows-window-management`, so it qualifies wherever this guide names that profile or one it inherits. Safe permits session-opened windows and forbids display reads. YOLO permits task-authorized existing windows; it is not OS isolation.+The current desktop scope is Windows, with Pi running on it directly or in its WSL, on this machine or on another one over SSH. A different interface does not qualify a new runtime or platform. The production `windows-desktop` profile selects the same checked bytes as `windows-window-management`, so it qualifies wherever this guide names that profile or one it inherits. Safe permits session-opened windows and forbids display reads. YOLO permits task-authorized existing windows; it is not OS isolation.## Discover and readdiff --git a/src/display-mode.ts b/src/display-mode.tsindex fbdfa37..be4a2f2 100644--- a/src/display-mode.ts+++ b/src/display-mode.ts@@ -1,4 +1,4 @@-import { WINDOWS_POWERSHELL, type CommandRunner } from "./runner.ts";+import { powershellOf, type CommandRunner } from "./runner.ts";/** A display resolution in physical pixels. */export interface DisplayMode { width: number; height: number }@@ -70,7 +70,7 @@ if ($status -ne 0) { throw "Windows refused the mode change (code $status)." }/** Display modes changed through Windows PowerShell started by the runner. */export function displayModes(runner: CommandRunner): DisplayModes {const change = async (request: Record<string, unknown>, signal?: AbortSignal): Promise<DisplayModeChange> => {- const output = await runner.run(WINDOWS_POWERSHELL, ["-NoLogo", "-NoProfile", "-NonInteractive", "-ExecutionPolicy", "Bypass", "-EncodedCommand",+ const output = await runner.run(powershellOf(runner), ["-NoLogo", "-NoProfile", "-NonInteractive", "-ExecutionPolicy", "Bypass", "-EncodedCommand",Buffer.from(`$ProgressPreference = 'SilentlyContinue'; $ErrorActionPreference = 'Stop'\ntry {\n${SCRIPT}\n} catch { [Console]::Error.WriteLine($_.Exception.Message); exit 1 }`, "utf16le").toString("base64")],{ input: JSON.stringify(request), signal, timeoutMs: 30_000 });return JSON.parse(output.trim()) as DisplayModeChange;diff --git a/src/machine-host.ts b/src/machine-host.tsindex fa8d35d..9733719 100644--- a/src/machine-host.ts+++ b/src/machine-host.ts@@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto";import { readFile } from "node:fs/promises";import sharp from "sharp";import { displayModes as nativeDisplayModes, type DisplayModes } from "./display-mode.ts";-import { WINDOWS_POWERSHELL, runPowerShell, type CommandRunner } from "./runner.ts";+import { powershellOf, runPowerShell, runnerFiles, type CommandRunner, type RunnerFiles } from "./runner.ts";/** The Chromium switch a WebView2 app gets for debugging; port 0 lets each app choose a free port. */export const WEBVIEW2_DEBUGGING = "--remote-debugging-port=0";/** Quote one argument for a Windows command line as CommandLineToArgvW and the C runtime read it back. */@@ -23,18 +23,26 @@ export interface TextLine { text: string; x: number; y: number; width: number; hexport class MachineHost {private folder?: Promise<{ windows: string; wsl: string }>;private readonly folderId = randomUUID();+ private readonly fs: RunnerFiles;- constructor(private readonly runner: CommandRunner, readonly displayModes: DisplayModes = nativeDisplayModes(runner)) {}+ constructor(private readonly runner: CommandRunner, readonly displayModes: DisplayModes = nativeDisplayModes(runner)) {+ this.fs = runnerFiles(runner);+ }private powershell(script: string, input?: string | Buffer, timeoutMs = 60_000): Promise<string> {- return runPowerShell(this.runner, WINDOWS_POWERSHELL, script, { ...(input !== undefined ? { input } : {}), timeoutMs });+ return runPowerShell(this.runner, powershellOf(this.runner), script, { ...(input !== undefined ? { input } : {}), timeoutMs });+ }++ /** A file in the owned folder as the runner addresses it. */+ private async inFolder(name: string): Promise<string> {+ return this.fs.join((await this.files()).wsl, checkedName(name));}- /** This session's folder in the machine's Windows temp directory, created on first use. */+ /** This session's folder in the machine's Windows temp directory, created on first use; `wsl` is the runner's own path. */files(): Promise<{ windows: string; wsl: string }> {this.folder ??= (async () => {const windows = await this.powershell(`$d = Join-Path ([IO.Path]::GetTempPath()) 'pi-computer-use\\${this.folderId}'; New-Item -ItemType Directory -Force -Path $d | Out-Null; $d`);- const wsl = (await this.runner.run("wslpath", ["-u", windows])).trim();+ const wsl = await this.fs.native(windows);return { windows, wsl };})();this.folder.catch(() => { this.folder = undefined; });@@ -44,27 +52,23 @@ export class MachineHost {/** Write bytes as a file in the owned folder and return its Windows path. */async put(name: string, bytes: Buffer): Promise<string> {const folder = await this.files();- await this.runner.run("dd", ["status=none", `of=${folder.wsl}/${checkedName(name)}`], { input: bytes });+ await this.fs.write(await this.inFolder(name), bytes);return `${folder.windows}\\${name}`;}/** Read a file from the owned folder. */async get(name: string): Promise<Buffer> {- const folder = await this.files();- return Buffer.from((await this.runner.run("base64", ["-w0", `${folder.wsl}/${checkedName(name)}`], { timeoutMs: 120_000 })).trim(), "base64");+ return this.fs.read(await this.inFolder(name));}/** Rename a file inside the owned folder. */async rename(from: string, to: string): Promise<void> {- const folder = await this.files();- await this.runner.run("mv", ["-f", "--", `${folder.wsl}/${checkedName(from)}`, `${folder.wsl}/${checkedName(to)}`]);+ await this.fs.rename(await this.inFolder(from), await this.inFolder(to));}/** Files in the owned folder with their sizes. */async list(): Promise<{ name: string; bytes: number }[]> {- const folder = await this.files();- const out = await this.runner.run("find", [folder.wsl, "-maxdepth", "1", "-type", "f", "-printf", "%f\\t%s\\n"]);- return out.split("\n").filter(Boolean).map(line => { const [name, bytes] = line.split("\t"); return { name, bytes: Number(bytes) }; });+ return this.fs.list((await this.files()).wsl);}/** Remove the owned folder; called when the computer session closes. */@@ -72,7 +76,7 @@ export class MachineHost {if (!this.folder) return;const folder = await this.folder.catch(() => undefined);this.folder = undefined;- if (folder) await this.runner.run("rm", ["-rf", "--", folder.wsl]);+ if (folder) await this.fs.remove(folder.wsl);}/**@@ -95,7 +99,7 @@ export class MachineHost {const script = await this.put("audio-loopback.ps1", await readFile(new URL("../scripts/audio-loopback.ps1", import.meta.url)));const folder = await this.files();try {- await this.runner.run(WINDOWS_POWERSHELL, ["-NoLogo", "-NoProfile", "-NonInteractive", "-ExecutionPolicy", "Bypass", "-File", script, String(seconds), `${folder.windows}\\${checkedName(name)}`],+ await this.runner.run(powershellOf(this.runner), ["-NoLogo", "-NoProfile", "-NonInteractive", "-ExecutionPolicy", "Bypass", "-File", script, String(seconds), `${folder.windows}\\${checkedName(name)}`],{ timeoutMs: seconds * 1000 + 30_000 });} catch (error) {const message = error instanceof Error ? error.message : String(error);@@ -159,7 +163,7 @@ if ($proc) { @{ pid = [int]$p; name = $proc.ProcessName } | ConvertTo-Json -Compconst saved = await this.powershell(`Add-Type -AssemblyName System.Windows.Forms, System.Drawing; $i = [Windows.Forms.Clipboard]::GetImage(); if ($i) { $i.Save('${folder.windows.replace(/'/g, "''")}\\${name}', [Drawing.Imaging.ImageFormat]::Png); 'saved' }`, undefined, 20_000);if (saved !== "saved") return undefined;try { return await this.get(name); }- finally { await this.runner.run("rm", ["-f", "--", `${folder.wsl}/${name}`]).catch(() => undefined); }+ finally { await this.fs.remove(await this.inFolder(name)).catch(() => undefined); }}/**@@ -184,8 +188,7 @@ if ($proc) { @{ pid = [int]$p; name = $proc.ProcessName } | ConvertTo-Json -Compconst lines = out ? JSON.parse(out) as TextLine[] : [];return lines.map(line => ({ ...box(line), words: line.words.map(word => box(word) as TextLine["words"][number]) }));} finally {- const folder = await this.files();- await this.runner.run("rm", ["-f", "--", `${folder.wsl}/${name}`]).catch(() => undefined);+ await this.fs.remove(await this.inFolder(name)).catch(() => undefined);}}}diff --git a/src/runner.ts b/src/runner.tsindex 5383e37..299f6e3 100644--- a/src/runner.ts+++ b/src/runner.ts@@ -1,12 +1,16 @@import { execFile, spawn, type ChildProcessWithoutNullStreams } from "node:child_process";+import { mkdir, readFile, readdir, rename, rm, stat, writeFile } from "node:fs/promises";+import path from "node:path";/**- * Where native helpers run: this WSL (with Windows reached through interop) or another WSL reached over SSH.- * Paths are in the runner's own WSL; Windows programs are started by their /mnt/c path.+ * Where native helpers run: Windows itself when Pi runs there, this WSL (Windows reached through interop) or another+ * WSL reached over SSH. In WSL, paths are the runner's own WSL paths and Windows programs start by their /mnt/c path.*/export interface CommandRunner {/** SSH alias for a remote machine; undefined locally. */readonly ssh?: string;+ /** Pi itself runs on this Windows: no WSL, paths are Windows paths and programs start by their Windows names. */+ readonly windows?: boolean;/** Start a long-lived process with piped stdio. Ending its stdin is the shutdown signal. */spawn(file: string, args: readonly string[], env?: Record<string, string>): ChildProcessWithoutNullStreams;/** Run a short command, optionally with stdin, and return its stdout. */@@ -16,6 +20,54 @@ export interface CommandRunner {/** Windows PowerShell as seen from WSL. */export const WINDOWS_POWERSHELL = "/mnt/c/Windows/System32/WindowsPowerShell/v1.0/powershell.exe";+/** Windows PowerShell as the runner starts it. */+export const powershellOf = (runner: CommandRunner): string => runner.windows ? "powershell.exe" : WINDOWS_POWERSHELL;++/**+ * Files on the runner's machine, addressed by the runner's own paths (Windows paths when Pi runs on Windows).+ * Directly through Node there; through small WSL commands locally from WSL or over SSH.+ */+export interface RunnerFiles {+ /** A Windows path as this runner addresses it. */+ native(windows: string): Promise<string>;+ /** A runner path as Windows sees it. */+ windows(native: string): Promise<string>;+ join(...parts: string[]): string;+ mkdir(dir: string): Promise<void>;+ write(file: string, bytes: string | Buffer): Promise<void>;+ read(file: string): Promise<Buffer>;+ rename(from: string, to: string): Promise<void>;+ list(dir: string): Promise<{ name: string; bytes: number }[]>;+ remove(target: string): Promise<void>;+}++export function runnerFiles(runner: CommandRunner): RunnerFiles {+ if (runner.windows) return {+ native: async value => value,+ windows: async value => value,+ join: (...parts) => path.win32.join(...parts),+ mkdir: async dir => { await mkdir(dir, { recursive: true }); },+ write: (file, bytes) => writeFile(file, bytes),+ read: file => readFile(file),+ rename: (from, to) => rename(from, to),+ list: async dir => Promise.all((await readdir(dir, { withFileTypes: true })).filter(entry => entry.isFile())+ .map(async entry => ({ name: entry.name, bytes: (await stat(path.win32.join(dir, entry.name))).size }))),+ remove: target => rm(target, { recursive: true, force: true }),+ };+ return {+ native: async value => (await runner.run("wslpath", ["-u", value])).trim(),+ windows: async value => (await runner.run("wslpath", ["-w", value])).trim(),+ join: (...parts) => path.posix.join(...parts),+ mkdir: async dir => { await runner.run("mkdir", ["-p", "--", dir]); },+ write: async (file, bytes) => { await runner.run("dd", ["status=none", `of=${file}`], { input: bytes, timeoutMs: 600_000 }); },+ read: async file => Buffer.from((await runner.run("base64", ["-w0", "--", file], { timeoutMs: 120_000 })).trim(), "base64"),+ rename: async (from, to) => { await runner.run("mv", ["-f", "--", from, to]); },+ list: async dir => (await runner.run("find", [dir, "-maxdepth", "1", "-type", "f", "-printf", "%f\\t%s\\n"]))+ .split("\n").filter(Boolean).map(line => { const [name, bytes] = line.split("\t"); return { name, bytes: Number(bytes) }; }),+ remove: async target => { await runner.run("rm", ["-rf", "--", target]); },+ };+}+/*** Run a PowerShell script on the runner's Windows with UTF-8 output and stop-on-error; a thrown error exits 1 with* its message on stderr. `powershell` is the executable as the runner sees it: WINDOWS_POWERSHELL from WSL,@@ -40,6 +92,7 @@ export function runPowerShell(runner: CommandRunner, powershell: string, script:export function localRunner(interop?: string): CommandRunner {const environment = (env?: Record<string, string>) => ({ ...process.env, ...(interop ? { WSL_INTEROP: interop } : {}), ...env });return {+ ...(process.platform === "win32" ? { windows: true } : {}),spawn: (file, args, env) => spawn(file, [...args], { stdio: "pipe", windowsHide: true, env: environment(env) }),run: (file, args, options = {}) => execute(file, [...args], environment(options.env), options),};diff --git a/src/sandbox.ts b/src/sandbox.tsindex 678cf13..1424fd0 100644--- a/src/sandbox.ts+++ b/src/sandbox.ts@@ -5,7 +5,7 @@ import { connect, createServer, type AddressInfo } from "node:net";import path from "node:path";import type { ComputerConfig } from "./config.ts";import { ComputerRequestError } from "./computer-resources.ts";-import { SSH_OPTIONS, WINDOWS_POWERSHELL, type CommandRunner } from "./runner.ts";+import { SSH_OPTIONS, powershellOf, runnerFiles, type CommandRunner } from "./runner.ts";/** Host-side commands for the sandbox lifecycle; replaced by a fake in tests. */export interface SandboxHost {@@ -85,19 +85,20 @@ export function sandboxConfig(runtime: string, config: string, out: string, sharexport function runnerHost(runner: CommandRunner): SandboxHost {const run = async (file: string, args: string[], signal?: AbortSignal, timeoutMs = 60_000, input?: string) =>(await runner.run(file, args, { signal, timeoutMs, input })).replace(/\r/g, "");+ const files = runnerFiles(runner);return {// Plain-text errors: an encoded command otherwise reports progress and errors as CLIXML on stderr.- powershell: (script, signal, timeout) => run(WINDOWS_POWERSHELL,+ powershell: (script, signal, timeout) => run(powershellOf(runner),["-NoLogo", "-NoProfile", "-NonInteractive", "-ExecutionPolicy", "Bypass", "-EncodedCommand", Buffer.from(// Continue: under Stop, Windows PowerShell turns redirected native stderr into exceptions. Scripts throw real failures.// Explicit exit 0: -Command otherwise exits 1 when the last command only set $? to false.`$ProgressPreference = 'SilentlyContinue'; $ErrorActionPreference = 'Continue'\ntry {\n${script}\nexit 0\n} catch { [Console]::Error.WriteLine($_.Exception.Message); exit 1 }`,"utf16le").toString("base64")], signal, timeout),- path: async (value, to) => (await run("wslpath", [to === "windows" ? "-w" : "-u", value])).trim(),- mkdir: async dir => { await run("mkdir", ["-p", dir]); },- writeFile: async (file, content) => { await run("dd", ["status=none", `of=${file}`], undefined, 60_000, content); },- readFile: file => run("cat", [file]).catch(() => ""),- remove: async dir => { await run("rm", ["-rf", dir]); },+ path: (value, to) => to === "windows" ? files.windows(value) : files.native(value),+ mkdir: dir => files.mkdir(dir),+ writeFile: (file, content) => files.write(file, content),+ readFile: file => files.read(file).then(bytes => bytes.toString("utf8").replace(/\r/g, ""), () => ""),+ remove: dir => files.remove(dir),forward: async (port, signal) => runner.ssh ? tunnel(runner.ssh, port, signal) : { port, close: () => {} },};}diff --git a/src/setup.ts b/src/setup.tsindex 8d635ca..807fbdb 100644--- a/src/setup.ts+++ b/src/setup.ts@@ -126,6 +126,7 @@ async function resolveTarget(cwd: string, machine: string | undefined, deps: Setreturn { label: `machine ${machine}`, runner: deps.ssh(config.ssh), powershell: WINDOWS_POWERSHELL, native: false, machine: config };}if (deps.platform === "win32") return { label: "this machine", runner: deps.local(), powershell: "powershell.exe", native: true };+ // Pi in WSL: Windows reached through interop.if (deps.platform === "linux" && await deps.wsl())return { label: "this machine", runner: deps.local(settings.interop), powershell: WINDOWS_POWERSHELL, native: false, ...(settings.interop ? { interop: settings.interop } : {}) };throw new SetupError("Setup needs Pi on Windows or in WSL; nothing was installed.");