Skip to content

Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.

Run optional installs as TypeScript and PowerShell so they work on Windows and WSL (LPT-867)

alexshpunt-clanker[bot]2c69893507e4
diff --git a/.pi/skills/computer-use/SKILL.md b/.pi/skills/computer-use/SKILL.md
index ee9dc1d..48e5074 100644
--- a/.pi/skills/computer-use/SKILL.md
+++ b/.pi/skills/computer-use/SKILL.md
@@ -53,9 +53,9 @@ A classifier answer is a judgement about pixels: it does not renew observation/i
## Missing parts
-Some parts are optional installs: Windows Node for browsers (`browser`), Playwright's Firefox (`firefox`), Cua perception for `regions` (`perception`, about 420 MB, AGPL-3.0 icon detector), MXC for `open_session` (`mxc`), the VS Code bridge (`vscode-bridge`) and preparing another machine (`machine <ssh alias> [name]`). A refusal that names `/computer-setup` means one is missing.
+Some parts are optional installs: Windows Node for browsers (`browser`), Playwright's Firefox (`firefox`), Cua perception for `regions` (`perception`, about 420 MB, AGPL-3.0 icon detector), MXC for `open_session` (`mxc`), the VS Code bridge (`vscode-bridge`) and preparing another machine (`machine <ssh alias> [name]`). A refusal that names a setup part means one is missing.
-Never install on your own. Tell the user what is missing, what it adds, its size and where it goes, and ask with ask_user. Ask before you start when the task clearly needs such a part, for example another machine or VS Code's browser. Only after explicit permission for that part on that machine, run [scripts/setup.sh](../../../scripts/setup.sh) by its absolute path with Bash from the project folder: `bash <absolute path>/setup.sh <part> [machine]`; without arguments it lists the parts. Retry the refused step once afterwards. `machine` prints a `machines` entry: show it, and add it to `.pi/computer-use.json` only with permission. A failed install is reported with its output, not retried in a loop.
+Never install on your own. Tell the user what is missing, what it adds, its size and where it goes, and ask with ask_user. Ask before you start when the task clearly needs such a part, for example another machine or VS Code's browser. Only after explicit permission for that part on that machine, Insert `{"action":"setup","part":"<part>"}` into that machine's address (`computer:machine/local` or `computer:machine/<name>`); it works with Pi on Windows and in WSL and does not start the machine's driver. For a portable VS Code add `code` and `argv` paths. To prepare another machine, Insert `{"action":"setup","part":"machine","ssh":"<alias>","name":"<name>"}` into `computer:machine/local`; it prints a `machines` entry: show it, and add it to `.pi/computer-use.json` only with permission. Retry the refused step once afterwards. A failed install is reported with its output, not retried in a loop.
## Access and cleanup
diff --git a/README.md b/README.md
index cc58af8..4b5bbd9 100644
--- a/README.md
+++ b/README.md
@@ -157,7 +157,7 @@ Window control works right after the install. A few parts are extra downloads, a
| `vscode-bridge` | VS Code's integrated browser. |
| `machine <ssh alias>` | Another Windows machine with WSL, reached over SSH. |
-Add a machine name to install a part there, for example `/computer-setup perception lab`. `/computer-setup` is a Bash script, so run it from Pi in WSL. Everything goes into your Windows user's own folders, and every download is checked against a pinned SHA-256. If a task needs a part you don't have, the agent tells you what's missing and asks before it installs anything.
+Add a machine name to install a part there, for example `/computer-setup perception lab`. Everything goes into your Windows user's own folders, and every download is checked against a pinned SHA-256. If a task needs a part you don't have, the agent tells you what's missing and asks before it installs anything.
### Unsigned binaries
diff --git a/docs/computer-resources.md b/docs/computer-resources.md
index dbd0c73..f2be58e 100644
--- a/docs/computer-resources.md
+++ b/docs/computer-resources.md
@@ -5,7 +5,7 @@ Computer use works with or without Pi Agent IDE. Each Pi session picks the mode
- With Pi Agent IDE loaded, in either order, use the ordinary IDE Read, Search, Write and Insert tools on `computer:` addresses.
- Without IDE, or with an IDE whose plugin API version does not match, use the `computer_read`, `computer_search`, `computer_write` and `computer_insert` tools. They are deferred: find them with `tool_search`, and computer use turns `tool_search` on. Codemode scripts can call them directly.
-Both modes take the same addresses, views, JSON commands and Codemode values. In this guide, Read, Search, Write and Insert mean either tool family. Without IDE, Read `docs:computer` with `computer_read` for this guide. The `/computer-mode`, `/computer-debug` and `/computer-setup` commands remain user controls; an agent installs an optional part with `scripts/setup.sh` only after the user explicitly allows it.
+Both modes take the same addresses, views, JSON commands and Codemode values. In this guide, Read, Search, Write and Insert mean either tool family. Without IDE, Read `docs:computer` with `computer_read` for this guide. The `/computer-mode`, `/computer-debug` and `/computer-setup` commands remain user controls. An agent installs an optional part with Insert `{"action":"setup","part":"browser|firefox|perception|mxc|vscode-bridge"}` into a machine address, or prepares another machine with `{"action":"setup","part":"machine","ssh":"<alias>","name":"<name>"}` into `computer:machine/local`, only after the user explicitly allows it. Setup does not start that machine's driver.
The current desktop scope is Windows controlled from WSL, on this machine or on another one over SSH. A different interface does not qualify a new runtime or platform. The production `windows-desktop` profile selects the same checked bytes as `windows-window-management`, so it qualifies wherever this guide names that profile or one it inherits. Safe permits session-opened windows and forbids display reads. YOLO permits task-authorized existing windows; it is not OS isolation.
@@ -84,7 +84,7 @@ Detail retains the original local rectangle and native scale through movement/re
Web pages run through [Playwright](https://playwright.dev/). The browser window itself, its tabs and toolbar, system dialogs and every other application stay native Windows windows read and operated as above.
-Playwright runs in a small worker on Windows, so `open_browser` and `attach_browser` need Node.js there: an installed Node, or a portable one from `/computer-setup browser` in the Windows user's `%LOCALAPPDATA%\Programs\PiComputerUse`. Without either, both refuse and name that command.
+Playwright runs in a small worker on Windows, so `open_browser` and `attach_browser` need Node.js there: an installed Node, or a portable one from setup part `browser` in the Windows user's `%LOCALAPPDATA%\Programs\PiComputerUse`. Without either, both refuse and name that part. When Pi itself runs on Windows, the worker uses Pi's own Node.
Insert `{"action":"open_browser"}` into a machine address to launch a new visible browser with a disposable profile. Optional `"browser":"msedge"` selects installed Edge instead of Chrome, `"browser":"firefox"` selects Playwright's own Firefox build, and `"url"` opens an HTTP(S) page. Firefox is not the user's installed Firefox: install the build once per machine with `/computer-setup firefox` (add a machine name for another machine), and again after updating this package. It uses the same DOM views and actions; it cannot be attached to. The result returns the browser's window address, owned by this session under Safe. DOM reads refuse in windows that are neither opened this way nor [attached](#attach-to-a-running-browser). Never replay an uncertain launch. Closing the computer session closes these browsers and deletes their profiles.
diff --git a/docs/setup.md b/docs/setup.md
index 0c1b0a5..0b50506 100644
--- a/docs/setup.md
+++ b/docs/setup.md
@@ -12,7 +12,7 @@ pi install npm:pi-agent-computer-use
The package carries the Windows runtime, so no driver download or executable config is needed. From WSL it controls the Windows host, not a Linux desktop. See [the bundled runtime](bundled-runtime.md) for platforms and checks.
-Optional parts install with `/computer-setup PART [MACHINE]`: `browser`, `firefox`, `perception`, `mxc`, `vscode-bridge` and `machine`. The command runs `scripts/setup.sh`, which is a Bash script for Pi running in WSL. Everything goes into the Windows user's own folders, and downloads are checked against pinned SHA-256 sums. An agent runs it only after the user explicitly allows that part.
+Optional parts install with `/computer-setup PART [MACHINE]`: `browser`, `firefox`, `perception`, `mxc`, `vscode-bridge` and `machine`. It works with Pi on Windows and in WSL: each part runs as a PowerShell script on the target Windows, locally or over SSH. Everything goes into the Windows user's own folders, and downloads are checked against pinned SHA-256 sums on that Windows before use. An agent installs a part with Insert `{"action":"setup","part":...}` into a machine address, and only after the user explicitly allows that part.
## Choose a mode
diff --git a/scripts/install-firefox.sh b/scripts/install-firefox.sh
deleted file mode 100755
index 8d15957..0000000
--- a/scripts/install-firefox.sh
+++ /dev/null
@@ -1,17 +0,0 @@
-#!/usr/bin/env bash
-# Install Playwright's own Firefox build for open_browser {"browser":"firefox"}. It is the build this package's
-# playwright-core version expects, downloaded by playwright-core itself into the Windows user's
-# %LOCALAPPDATA%\ms-playwright; nothing system-wide changes. Run it again after updating playwright-core.
-# scripts/setup.sh firefox resolves the arguments.
-#
-# Usage: scripts/install-firefox.sh NODE_EXE PLAYWRIGHT_CLI [SSH_ALIAS]
-# NODE_EXE and PLAYWRIGHT_CLI are WSL paths on the target machine: Windows node.exe and playwright-core's cli.js.
-set -euo pipefail
-
-node_exe="${1:?usage: install-firefox.sh NODE_EXE PLAYWRIGHT_CLI [SSH_ALIAS]}"
-cli="${2:?usage: install-firefox.sh NODE_EXE PLAYWRIGHT_CLI [SSH_ALIAS]}"
-alias="${3:-}"
-
-install='cd /mnt/c && "$1" "$(wslpath -w "$2")" install firefox'
-if [[ -z "$alias" ]]; then bash -c "$install" _ "$node_exe" "$cli"
-else ssh -o BatchMode=yes "$alias" bash -s -- "$node_exe" "$cli" <<<"$install"; fi
diff --git a/scripts/install-mxc.sh b/scripts/install-mxc.sh
deleted file mode 100755
index f44dcca..0000000
--- a/scripts/install-mxc.sh
+++ /dev/null
@@ -1,29 +0,0 @@
-#!/usr/bin/env bash
-# Install wxc-exec.exe from Microsoft Execution Containers (MXC, MIT) for open_session: isolated Windows agent
-# sessions. It goes into the Windows user's %LOCALAPPDATA%\PiComputerUse\mxc; nothing system-wide changes. Session
-# isolation needs Windows 11 24H2/25H2 build 9550 or later; provisioning reports it otherwise.
-#
-# Usage: scripts/install-mxc.sh local|SSH_ALIAS
-set -euo pipefail
-
-target="${1:?usage: install-mxc.sh local|SSH_ALIAS}"
-release="v1.0.1"
-sha256="0e832239454ba7fcd338ff56218a287eef4c0a01c1484084b869f55feff8abe9"
-
-# Run a bash script from stdin on the target machine's WSL; the remote login shell may be fish.
-on_target() { if [[ "$target" == local ]]; then bash -s -- "$@"; else ssh -o BatchMode=yes "$target" bash -s -- "$@"; fi; }
-
-download="$(mktemp -d)"
-trap 'rm -rf "$download"' EXIT
-curl -fsSL -o "$download/mxc-release-binaries.zip" "https://github.com/microsoft/mxc/releases/download/$release/mxc-release-binaries.zip"
-echo "$sha256 $download/mxc-release-binaries.zip" | sha256sum -c --quiet
-python3 -c "import sys, zipfile; zipfile.ZipFile(sys.argv[1]).extract('x64/wxc-exec.exe', sys.argv[2])" "$download/mxc-release-binaries.zip" "$download"
-
-folder="$(on_target <<'EOF'
-cd /mnt/c && wslpath "$(/mnt/c/Windows/System32/cmd.exe /c 'echo %LOCALAPPDATA%' | tr -d '\r')"
-EOF
-)/PiComputerUse/mxc"
-on_target "$folder" <<<'mkdir -p "$1"'
-if [[ "$target" == local ]]; then cp "$download/x64/wxc-exec.exe" "$folder/"
-else rsync -t "$download/x64/wxc-exec.exe" "$target:$folder/"; fi
-echo "Installed MXC $release wxc-exec.exe in $folder"
diff --git a/scripts/install-node.sh b/scripts/install-node.sh
deleted file mode 100755
index 0f1be7f..0000000
--- a/scripts/install-node.sh
+++ /dev/null
@@ -1,32 +0,0 @@
-#!/usr/bin/env bash
-# Install a portable Windows Node for the Playwright browser worker (open_browser and attach_browser) into the
-# Windows user's %LOCALAPPDATA%\Programs\PiComputerUse. The version and its SHA-256 are pinned in windows-node.json;
-# nothing system-wide changes. Prints the node.exe path as a WSL path.
-#
-# Usage: scripts/install-node.sh local|SSH_ALIAS
-set -euo pipefail
-
-target="${1:?usage: install-node.sh local|SSH_ALIAS}"
-here="$(cd "$(dirname "$0")" && pwd)"
-read -r version sha256 < <(node -e 'const p = require(process.argv[1]); console.log(p.version, p.sha256)' "$here/windows-node.json")
-zip="node-$version-win-x64.zip"
-
-# Run a bash script from stdin on the target machine's WSL; the remote login shell may be fish.
-on_target() { if [[ "$target" == local ]]; then bash -s -- "$@"; else ssh -o BatchMode=yes "$target" bash -s -- "$@"; fi; }
-
-programs="$(on_target <<'EOF'
-cd /mnt/c && wslpath "$(/mnt/c/Windows/System32/cmd.exe /c 'echo %LOCALAPPDATA%' | tr -d '\r')"
-EOF
-)/Programs/PiComputerUse"
-node_exe="$programs/node-$version-win-x64/node.exe"
-if on_target "$node_exe" <<<'[[ -f "$1" ]]'; then echo "$node_exe"; exit; fi
-
-download="$(mktemp -d)"
-trap 'rm -rf "$download"' EXIT
-curl -fsSL -o "$download/$zip" "https://nodejs.org/dist/$version/$zip"
-echo "$sha256 $download/$zip" | sha256sum -c --quiet
-python3 -c "import sys, zipfile; zipfile.ZipFile(sys.argv[1]).extractall(sys.argv[2])" "$download/$zip" "$download/out"
-on_target "$programs" <<<'mkdir -p "$1"'
-if [[ "$target" == local ]]; then cp -r "$download/out/node-$version-win-x64" "$programs/"
-else rsync -rt "$download/out/node-$version-win-x64" "$target:$programs/"; fi
-echo "$node_exe"
diff --git a/scripts/install-perception.sh b/scripts/install-perception.sh
deleted file mode 100755
index 2ecc7f7..0000000
--- a/scripts/install-perception.sh
+++ /dev/null
@@ -1,49 +0,0 @@
-#!/usr/bin/env bash
-# Install Cua's signed perception extension (OmniParser v2 icons, PP-OCRv5 text) for the `regions` Read view.
-# The release files are checked against pinned SHA-256 sums; the driver then verifies Cua's catalog signature and
-# every hash, and installs into the Windows user's %USERPROFILE%\.cua-driver\extensions. About 420 MB; the icon
-# detector is AGPL-3.0. Nothing is bundled in this package. scripts/setup.sh perception resolves the arguments.
-#
-# Usage: scripts/install-perception.sh local|SSH_ALIAS DRIVER_EXE
-# DRIVER_EXE is that machine's WSL path to cua-driver.exe.
-set -euo pipefail
-
-target="${1:?usage: install-perception.sh local|SSH_ALIAS DRIVER_EXE}"
-driver="${2:?usage: install-perception.sh local|SSH_ALIAS DRIVER_EXE}"
-release="https://github.com/trycua/cua/releases/download/cua-perception-v0.2.1"
-name="cua-perception-0.2.1-x86_64-pc-windows-msvc"
-
-# Run a bash script from stdin on the target machine's WSL; the remote login shell may be fish.
-on_target() { if [[ "$target" == local ]]; then bash -s -- "$@"; else ssh -o BatchMode=yes "$target" bash -s -- "$@"; fi; }
-
-# An installed extension only re-runs its self-test; nothing is downloaded again.
-if on_target "$driver" <<<'cd /mnt/c && "$1" extension status perception | grep -q "Active version: 0.2.1"'; then
- on_target "$driver" <<<'cd /mnt/c && "$1" extension status perception --self-test'
- exit
-fi
-
-download="$(mktemp -d)"
-trap 'rm -rf "$download"' EXIT
-curl -fsSL -o "$download/$name.catalog.json" "$release/$name.catalog.json"
-curl -fSL --progress-bar -o "$download/$name.tar.gz" "$release/$name.tar.gz"
-(cd "$download" && sha256sum -c --quiet) <<EOF
-bfb2300ff9da054d522e73351f7b337ba6157ebbe30f029a95a471fc3d13a042 $name.catalog.json
-6709579aa3f938e8200c0a51b1abe450d4fda51b09e97517e0cb4c272b5bff37 $name.tar.gz
-EOF
-
-# The catalog names its archive beside it, in the target's Windows temp folder.
-temp="$(on_target <<'EOF'
-cd /mnt/c && wslpath "$(/mnt/c/Windows/System32/cmd.exe /c 'echo %TEMP%' | tr -d '\r')"
-EOF
-)/pi-perception"
-on_target "$temp" <<<'mkdir -p "$1"'
-if [[ "$target" == local ]]; then cp "$download/$name".{catalog.json,tar.gz} "$temp/"
-else rsync -t --partial "$download/$name".{catalog.json,tar.gz} "$target:$temp/"; fi
-
-on_target "$driver" "$temp" "$name" <<'EOF'
-set -euo pipefail
-trap 'cd / && rm -rf "$2"' EXIT
-cd "$2"
-"$1" extension install perception --catalog "$(wslpath -w "$2/$3.catalog.json")"
-"$1" extension status perception --self-test
-EOF
diff --git a/scripts/install-vscode-bridge.sh b/scripts/install-vscode-bridge.sh
deleted file mode 100755
index 8c337e3..0000000
--- a/scripts/install-vscode-bridge.sh
+++ /dev/null
@@ -1,70 +0,0 @@
-#!/usr/bin/env bash
-# Build and install the Pi Browser Bridge into a Windows VS Code, and allow its proposed browser API in that VS Code's
-# argv.json. VS Code must then be quit fully and started again; Reload Window does not reread argv.json.
-#
-# Usage: scripts/install-vscode-bridge.sh local|SSH_ALIAS [CODE_CMD ARGV_JSON]
-# Without paths, an installed VS Code is found on the target: its bin/code.cmd and %USERPROFILE%\.vscode\argv.json.
-# For a portable VS Code, pass both as WSL paths on the target: ".../VSCode/bin/code.cmd" ".../VSCode/data/argv.json".
-set -euo pipefail
-
-target="${1:?usage: install-vscode-bridge.sh local|SSH_ALIAS [CODE_CMD ARGV_JSON]}"
-code="${2:-}"
-argv="${3:-}"
-root="$(cd "$(dirname "$0")/.." && pwd)"
-build="$(mktemp -d)"
-trap 'rm -rf "$build"' EXIT
-bash "$root/scripts/package-vscode-bridge.sh" "$build/pi-browser-bridge.vsix" >/dev/null
-
-# Run a bash script from stdin on the target machine's WSL; the remote login shell may be fish.
-on_target() { if [[ "$target" == local ]]; then bash -s -- "$@"; else ssh -o BatchMode=yes "$target" bash -s -- "$@"; fi; }
-
-if [[ -z "$code" || -z "$argv" ]]; then
- read -r -d '' found < <(on_target <<'EOF'
-cd /mnt/c
-win() { wslpath "$(/mnt/c/Windows/System32/cmd.exe /c "echo %$1%" | tr -d '\r')"; }
-for code in "$(win LOCALAPPDATA)/Programs/Microsoft VS Code/bin/code.cmd" "/mnt/c/Program Files/Microsoft VS Code/bin/code.cmd"; do
- [[ -f "$code" ]] && { printf '%s\n%s\n' "$code" "$(win USERPROFILE)/.vscode/argv.json"; exit; }
-done
-EOF
- ) || true
- code="$(sed -n 1p <<<"$found")"; argv="$(sed -n 2p <<<"$found")"
- [[ -n "$code" ]] || { echo "No installed VS Code found on $target. For a portable one, pass its bin/code.cmd and data/argv.json." >&2; exit 1; }
-fi
-
-vsix="$(on_target <<'EOF'
-cd /mnt/c && wslpath "$(/mnt/c/Windows/System32/cmd.exe /c 'echo %TEMP%' | tr -d '\r')"
-EOF
-)/pi-browser-bridge.vsix"
-if [[ "$target" == local ]]; then cp "$build/pi-browser-bridge.vsix" "$vsix"; else scp -q "$build/pi-browser-bridge.vsix" "$target:$vsix"; fi
-
-on_target "$code" "$vsix" "$argv" <<'EOF'
-set -euo pipefail
-cd /mnt/c
-# PowerShell quotes paths with spaces; its stdin would otherwise read the rest of this script.
-/mnt/c/Windows/System32/WindowsPowerShell/v1.0/powershell.exe -NoProfile -Command "& '$(wslpath -w "$1")' --install-extension '$(wslpath -w "$2")' --force" < /dev/null \
- | tr -d '\r' | grep -v -e DeprecationWarning -e trace-deprecation || true
-rm -f "$2"
-# argv.json is JSONC with comments: add the id to an existing enable-proposed-api list, or insert the key after the
-# opening brace, and never rewrite the rest of the file.
-python3 - "$3" <<'P'
-import json, re, sys
-path, ext = sys.argv[1], "alexshpunt.pi-browser-bridge"
-try: text = open(path, encoding="utf-8").read()
-except FileNotFoundError: text = "{\n}\n"
-if ext in text: print(f"{path} already allows {ext}"); sys.exit()
-match = re.search(r'"enable-proposed-api"\s*:\s*\[', text)
-def empty(rest, close):
- # Whether only whitespace and comments come before the closing bracket, so no separating comma is needed.
- return re.sub(r'//[^\n]*|/\*.*?\*/|\s', "", rest, flags=re.S).startswith(close)
-if match:
- rest = text[match.end():]
- text = text[:match.end()] + json.dumps(ext) + ("" if empty(rest, "]") else ", ") + rest
-else:
- brace = text.index("{") + 1
- rest = text[brace:]
- text = text[:brace] + f'\n\t"enable-proposed-api": [{json.dumps(ext)}]' + ("" if empty(rest, "}") else ",") + rest
-open(path, "w", encoding="utf-8").write(text)
-print(f"{path} now allows {ext}")
-P
-EOF
-echo "Installed. Quit VS Code fully and start it again."
diff --git a/scripts/package-vscode-bridge.sh b/scripts/package-vscode-bridge.sh
deleted file mode 100755
index a935d26..0000000
--- a/scripts/package-vscode-bridge.sh
+++ /dev/null
@@ -1,15 +0,0 @@
-#!/usr/bin/env bash
-# Build the Pi Browser Bridge VS Code extension (vscode-browser-bridge/) into a .vsix with its ws dependency.
-# Install it with `code --install-extension <vsix>` and add "enable-proposed-api": ["alexshpunt.pi-browser-bridge"]
-# to VS Code's argv.json (Preferences: Configure Runtime Arguments), then restart VS Code fully.
-#
-# Usage: scripts/package-vscode-bridge.sh [output.vsix]
-set -euo pipefail
-
-root="$(cd "$(dirname "$0")/.." && pwd)"
-out="$(realpath -m "${1:-$root/.tmp/pi-browser-bridge.vsix}")"
-work="$(mktemp -d)"
-trap 'rm -rf "$work"' EXIT
-cp "$root/vscode-browser-bridge/package.json" "$root/vscode-browser-bridge/extension.js" "$root/vscode-browser-bridge/README.md" "$work/"
-(cd "$work" && npm install --omit=dev --no-audit --no-fund --silent && npx --yes @vscode/[email protected] package --skip-license -o "$out")
-echo "Built $out"
diff --git a/scripts/setup-remote-machine.sh b/scripts/setup-remote-machine.sh
deleted file mode 100755
index 734d036..0000000
--- a/scripts/setup-remote-machine.sh
+++ /dev/null
@@ -1,53 +0,0 @@
-#!/usr/bin/env bash
-# Prepare another machine (Windows with WSL, reached over SSH) for computer use from this machine:
-# copy the CUA runtime, install a portable Windows Node for the browser worker, copy the browser worker,
-# and print the `machines` entry for .pi/computer-use.json. Installs only into the remote user's folders.
-# scripts/setup.sh machine resolves the arguments.
-#
-# Usage: scripts/setup-remote-machine.sh SSH_ALIAS RUNTIME_DIR PLAYWRIGHT_DIR [NAME]
-# RUNTIME_DIR is a local CUA runtime folder containing runtime/cua-driver.exe; PLAYWRIGHT_DIR is playwright-core.
-set -euo pipefail
-
-usage="usage: setup-remote-machine.sh SSH_ALIAS RUNTIME_DIR PLAYWRIGHT_DIR [NAME]"
-alias="${1:?$usage}"
-runtime_dir="${2:?$usage}"
-playwright="${3:?$usage}"
-name="${4:-$alias}"
-here="$(cd "$(dirname "$0")" && pwd)"
-package="$(dirname "$here")"
-[[ -f "$runtime_dir/runtime/cua-driver.exe" ]] || { echo "No runtime/cua-driver.exe in $runtime_dir" >&2; exit 1; }
-[[ -f "$playwright/cli.js" ]] || { echo "No playwright-core in $playwright" >&2; exit 1; }
-ssh=(ssh -o BatchMode=yes -o ConnectTimeout=10 "$alias")
-
-# The remote login shell may be fish: run every remote script through bash.
-remote_home="$("${ssh[@]}" bash -s <<<'printf %s "$HOME"')"
-programs="$("${ssh[@]}" bash -s <<'REMOTE'
-cd /mnt/c && wslpath "$(/mnt/c/Windows/System32/cmd.exe /c 'echo %LOCALAPPDATA%' | tr -d '\r')"
-REMOTE
-)/Programs/PiComputerUse"
-runtime_name="$(basename "$runtime_dir")"
-remote_package="$remote_home/.local/share/pi-computer-use/package"
-
-"${ssh[@]}" mkdir -p "$programs/$runtime_name" "$remote_package/node_modules"
-rsync -rt -e "ssh -o BatchMode=yes" "$runtime_dir/" "$alias:$programs/$runtime_name/"
-local_hash="$(sha256sum "$runtime_dir/runtime/cua-driver.exe" | cut -d' ' -f1)"
-remote_hash="$("${ssh[@]}" sha256sum "$programs/$runtime_name/runtime/cua-driver.exe" | cut -d' ' -f1)"
-[[ "$local_hash" == "$remote_hash" ]] || { echo "Runtime copy does not match ($remote_hash)." >&2; exit 1; }
-
-node_exe="$(bash "$here/install-node.sh" "$alias")"
-
-# The browser worker needs only its own source and playwright-core.
-rsync -rt --delete -e "ssh -o BatchMode=yes" "$package/src/" "$alias:$remote_package/src/"
-rsync -rt --delete -e "ssh -o BatchMode=yes" "$playwright/" "$alias:$remote_package/node_modules/playwright-core/"
-rsync -t -e "ssh -o BatchMode=yes" "$package/package.json" "$alias:$remote_package/package.json"
-
-cat <<JSON
-Add to "machines" in .pi/computer-use.json:
-"$name": {
- "ssh": "$alias",
- "runtime": "windows-desktop",
- "command": "$programs/$runtime_name/runtime/cua-driver.exe",
- "node": "$node_exe",
- "package": "$remote_package"
-}
-JSON
diff --git a/scripts/setup.sh b/scripts/setup.sh
deleted file mode 100755
index a098a2b..0000000
--- a/scripts/setup.sh
+++ /dev/null
@@ -1,102 +0,0 @@
-#!/usr/bin/env bash
-# One entry point for the optional parts of computer use. /computer-setup runs it; an agent runs it only after the
-# user explicitly allowed that install. Run it from the Pi project folder: MACHINE is a name under "machines" in that
-# folder's .pi/computer-use.json, and "local" (the default) is this machine.
-set -euo pipefail
-
-here="$(cd "$(dirname "$0")" && pwd)"
-package="$(dirname "$here")"
-
-usage() {
- cat <<'EOF'
-Usage: setup.sh PART [local|MACHINE]
-
-browser
- Windows Node for open_browser and attach_browser (about 30 MB)
-firefox
- Playwright's Firefox for "browser":"firefox" (needs browser)
-perception
- Cua perception for the regions view (about 420 MB, AGPL-3.0)
-mxc
- Microsoft MXC for open_session (Windows 11 24H2 or later)
-vscode-bridge [local|MACHINE] [CODE_CMD ARGV_JSON]
- Pi Browser Bridge for VS Code's integrated browser
-machine SSH_ALIAS [NAME]
- Prepare another Windows machine with WSL over SSH
-
-Everything goes into the Windows user's own folders.
-EOF
-}
-
-# One field of a configured machine; refuse unknown names before any SSH work.
-machine_field() {
- node -e '
- const [file, name, field] = process.argv.slice(1);
- let config = {};
- try { config = JSON.parse(require("fs").readFileSync(file, "utf8")); } catch {}
- const machine = config.machines?.[name];
- if (!machine) { console.error(`No machine named ${name} in ${file}.`); process.exit(1); }
- if (!machine[field]) { console.error(`Machine ${name} has no "${field}"; prepare it with: setup.sh machine ${machine.ssh ?? "SSH_ALIAS"} ${name}`); process.exit(1); }
- console.log(machine[field]);' "$PWD/.pi/computer-use.json" "$1" "$2"
-}
-
-local_appdata() { (cd /mnt/c && wslpath "$(/mnt/c/Windows/System32/cmd.exe /c 'echo %LOCALAPPDATA%' | tr -d '\r')"); }
-
-# The Windows node.exe the browser worker uses here: the portable one from `browser`, else an installed Node.
-local_node() {
- local version portable
- version="$(node -p 'require(process.argv[1]).version' "$here/windows-node.json")"
- portable="$(local_appdata)/Programs/PiComputerUse/node-$version-win-x64/node.exe"
- for node_exe in "$portable" "/mnt/c/Program Files/nodejs/node.exe"; do
- [[ -f "$node_exe" ]] && { echo "$node_exe"; return; }
- done
- echo "No Windows Node on this machine; run: setup.sh browser" >&2
- return 1
-}
-
-# The driver this machine runs: CUA_DRIVER_PATH, the project's configured command, or the packaged runtime's cache.
-local_driver() {
- local configured
- configured="${CUA_DRIVER_PATH:-$(node -e '
- try { const c = JSON.parse(require("fs").readFileSync(process.argv[1], "utf8")).command; if (c) console.log(c); } catch {}' "$PWD/.pi/computer-use.json")}"
- if [[ -n "$configured" ]]; then (cd "$PWD" && realpath -m "$configured"); return; fi
- local source driver
- source="$(node -p 'require(process.argv[1]).source' "$package/src/runtime-pins.json")"
- driver="$(local_appdata)/PiComputerUse/native-runtimes/$source/win32-x64/runtime/cua-driver.exe"
- [[ -f "$driver" ]] || { echo "Computer use has not prepared its runtime yet. Use it once (for example, search for windows), then run this again." >&2; return 1; }
- echo "$driver"
-}
-
-playwright_dir() { (cd "$package" && dirname "$(node -p 'require.resolve("playwright-core/package.json")')"); }
-
-component="${1:-}"
-shift || true
-case "$component" in
- ""|help|-h|--help) usage; exit 0 ;;
- machine)
- alias="${1:?usage: setup.sh machine SSH_ALIAS [NAME]}"
- runtime="$package/native-runtime/win32-x64"
- [[ -f "$runtime/runtime/cua-driver.exe" ]] || { echo "The packaged runtime is not downloaded yet. Use computer use once on this machine, then run this again." >&2; exit 1; }
- playwright="$(playwright_dir)"
- exec bash "$here/setup-remote-machine.sh" "$alias" "$runtime" "$playwright" "${2:-$alias}" ;;
- browser|firefox|perception|mxc|vscode-bridge) ;;
- *) echo "Unknown component: $component" >&2; usage >&2; exit 2 ;;
-esac
-
-target="${1:-local}"
-shift || true
-ssh=""
-[[ "$target" == local ]] || ssh="$(machine_field "$target" ssh)"
-
-case "$component" in
- browser) node_exe="$(bash "$here/install-node.sh" "${ssh:-local}")"; echo "Windows Node ready: $node_exe" ;;
- firefox)
- if [[ -z "$ssh" ]]; then node_exe="$(local_node)"; cli="$(playwright_dir)/cli.js"
- else node_exe="$(machine_field "$target" node)"; cli="$(machine_field "$target" package)/node_modules/playwright-core/cli.js"; fi
- exec bash "$here/install-firefox.sh" "$node_exe" "$cli" $ssh ;;
- perception)
- if [[ -z "$ssh" ]]; then driver="$(local_driver)"; else driver="$(machine_field "$target" command)"; fi
- exec bash "$here/install-perception.sh" "${ssh:-local}" "$driver" ;;
- mxc) exec bash "$here/install-mxc.sh" "${ssh:-local}" ;;
- vscode-bridge) exec bash "$here/install-vscode-bridge.sh" "${ssh:-local}" "$@" ;;
-esac
diff --git a/scripts/windows-node.json b/scripts/windows-node.json
deleted file mode 100644
index 4a31a14..0000000
--- a/scripts/windows-node.json
+++ /dev/null
@@ -1,4 +0,0 @@
-{
- "version": "v22.22.2",
- "sha256": "7c93e9d92bf68c07182b471aa187e35ee6cd08ef0f24ab060dfff605fcc1c57c"
-}
diff --git a/src/computer-resources.ts b/src/computer-resources.ts
index 1fb08db..7875858 100644
--- a/src/computer-resources.ts
+++ b/src/computer-resources.ts
@@ -15,6 +15,7 @@ import { diffDom, diffUi } from "./successor.ts";
import type { UiResource } from "./ui-resources.ts";
import { displayMode } from "./display-mode.ts";
import { isSemanticAction } from "./semantic-actions.ts";
+import { SETUP_PARTS, SetupError, runSetup, type SetupPart } from "./setup.ts";
/** The machine this extension runs on, by its address name. */
export const LOCAL_MACHINE = "local";
@@ -183,6 +184,30 @@ async function fileCommand(session: ComputerSession, command: Record<string, unk
throw new ComputerRequestError(`File ${command.action} failed: ${error instanceof Error ? error.message : error}`);
}
}
+/** Install an optional part on a machine, or prepare another one from this machine; the user must have allowed it. */
+async function setupCommand(command: Record<string, unknown>, machine: string | undefined, cwd: string, mode: ComputerMode, signal?: AbortSignal): Promise<ComputerResult> {
+ only(command, ["action", "part", "ssh", "name", "code", "argv"]);
+ const part = command.part;
+ if (typeof part !== "string" || !SETUP_PARTS.some(item => item.name === part))
+ throw new ComputerRequestError(`setup needs part: one of ${SETUP_PARTS.map(item => item.name).join(", ")}. Install only after the user explicitly allowed that part on that machine.`);
+ const strings: Record<string, string> = {};
+ for (const key of ["ssh", "name", "code", "argv"]) {
+ if (command[key] === undefined) continue;
+ if (typeof command[key] !== "string" || !command[key]) throw new ComputerRequestError(`setup ${key} must be a non-empty string.`);
+ strings[key] = command[key] as string;
+ }
+ if (part === "machine" && machine) throw new ComputerRequestError("Prepare another machine from computer:machine/local.");
+ if (part !== "machine" && (strings.ssh || strings.name)) throw new ComputerRequestError("ssh and name belong to setup part machine.");
+ if (part !== "vscode-bridge" && (strings.code || strings.argv)) throw new ComputerRequestError("code and argv belong to setup part vscode-bridge.");
+ try {
+ const text = await runSetup({ part: part as SetupPart, ...(machine ? { machine } : {}), ...strings }, cwd, { ...(signal ? { signal } : {}) });
+ return { content: [{ type: "text", text }], details: { status: "delivered", mode, state: { operation: "setup", part, machine: machine ?? LOCAL_MACHINE } } };
+ } catch (error) {
+ if (error instanceof SetupError) throw new ComputerRequestError(error.message);
+ return { content: [{ type: "text", text: `Setup of ${part} failed: ${error instanceof Error ? error.message : error}` }],
+ details: { status: "refused", mode, state: { operation: "setup", part, machine: machine ?? LOCAL_MACHINE } }, isError: true };
+ }
+}
const unresponsive = (health: ProcessHealth) => `The app ${health.name} (pid ${health.pid}) is not responding; it uses ${health.memory_mb} MB memory. Capture and input cannot recover a hung app: wait and read again, or close it.`;
function object(value: unknown): Record<string, unknown> {
if (!value || typeof value !== "object" || Array.isArray(value)) throw new ComputerRequestError("Use a JSON command object.");
@@ -610,6 +635,8 @@ export class ComputerResources {
const command = commandObject(value);
if (command.action === "close") { only(command, ["action"]); return this.closeMachine(address.ref); }
const remote = address.ref === LOCAL_MACHINE ? undefined : address.ref;
+ // Setup prepares a machine, so it must not start that machine's driver first.
+ if (command.action === "setup") return setupCommand(command, remote, cwd, this.getMode(), signal);
if (command.action === "open_sandbox" || command.action === "open_session") {
// A remote machine starts before its sandbox, so an unknown name refuses first.
if (remote) await this.machine(cwd, remote, signal);
@@ -631,7 +658,7 @@ export class ComputerResources {
return result.details.window_ref ? this.remember({ window_ref: result.details.window_ref }, result) : result;
}
if (command.action !== "open")
- throw new ComputerRequestError(`A machine address accepts open, open_browser, open_sandbox, open_session, webview2_debugging, ${FILE_ACTIONS.join(", ")} or close.`);
+ throw new ComputerRequestError(`A machine address accepts open, open_browser, open_sandbox, open_session, webview2_debugging, setup, ${FILE_ACTIONS.join(", ")} or close.`);
const result = await session.open(openTarget(command, "on that machine"), signal);
return result.details.window_ref ? this.remember({ window_ref: result.details.window_ref }, result) : result;
}
diff --git a/src/config.ts b/src/config.ts
index 6c61894..f708ab5 100644
--- a/src/config.ts
+++ b/src/config.ts
@@ -70,6 +70,18 @@ export async function loadComputerConfig(cwd: string): Promise<ComputerConfig> {
...(raw.machines !== undefined ? { machines: parseMachines(raw.machines) } : {}) };
}
+/** The trusted file's interop socket and machines, without selecting or downloading a runtime. */
+export async function readMachineSettings(cwd: string): Promise<{ interop?: string; machines: Record<string, MachineConfig> }> {
+ let raw: unknown = {};
+ try { raw = JSON.parse(await readFile(path.join(cwd, ".pi", "computer-use.json"), "utf8")); }
+ catch (error) {
+ if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw new Error("Invalid .pi/computer-use.json; see docs/setup.md#override-the-runtime.");
+ }
+ if (!raw || typeof raw !== "object" || Array.isArray(raw)) throw new Error("Invalid .pi/computer-use.json; see docs/setup.md#override-the-runtime.");
+ const { interop, machines } = raw as Record<string, unknown>;
+ return { ...(typeof interop === "string" ? { interop } : {}), machines: machines === undefined ? {} : parseMachines(machines) };
+}
+
function parseMachines(value: unknown): Record<string, MachineConfig> {
const invalid = () => { throw new Error("Invalid machines in .pi/computer-use.json: use {\"<name>\": {\"ssh\", \"command\", \"runtime\"?, \"node\"?, \"package\"?}}."); };
if (!value || typeof value !== "object" || Array.isArray(value)) return invalid();
diff --git a/src/isolated-session.ts b/src/isolated-session.ts
index 27ce7f5..64737e2 100644
--- a/src/isolated-session.ts
+++ b/src/isolated-session.ts
@@ -23,7 +23,7 @@ const sleep = (ms: number, signal?: AbortSignal) => new Promise<void>((resolve,
* An isolated Windows agent session from Microsoft Execution Containers (MXC `isolation_session`): a fresh agent user
* with its own session, desktop, clipboard and input on the same machine, removed again on close. Several can run at
* once, unlike Windows Sandbox. A CUA driver runs inside; the host reaches it through a reverse relay because the
- * session can dial the host's loopback but not the other way round. Needs `wxc-exec.exe` from /computer-setup mxc.
+ * session can dial the host's loopback but not the other way round. Needs `wxc-exec.exe` from setup part mxc.
*/
export class IsolatedSession {
readonly ref = "is-" + randomUUID().slice(0, 8);
@@ -66,7 +66,7 @@ export class IsolatedSession {
const local = (await this.ps("$env:LOCALAPPDATA", signal)).trim();
this.wxc = `${local}\\PiComputerUse\\mxc\\wxc-exec.exe`;
if ((await this.ps(`Test-Path -LiteralPath ${psString(this.wxc)}`, signal)).trim() !== "True")
- throw new ComputerRequestError(`MXC is not installed on this machine (${this.wxc}). Ask the user before installing it; with their permission run /computer-setup mxc (add the machine name for another machine), then open the session again.`);
+ throw new ComputerRequestError(`MXC is not installed on this machine (${this.wxc}). Ask the user before installing it; with their permission Insert {"action":"setup","part":"mxc"} into this machine's address, then open the session again.`);
const windows = `${local}\\PiComputerUse\\sessions\\${this.ref}`;
this.workdir = { windows, unix: await this.host.path(`${local}\\PiComputerUse\\sessions`, "unix").then(parent => `${parent}/${this.ref}`) };
await this.host.mkdir(this.workdir.unix);
diff --git a/src/machine-host.ts b/src/machine-host.ts
index 3295cde..fa8d35d 100644
--- a/src/machine-host.ts
+++ b/src/machine-host.ts
@@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto";
import { readFile } from "node:fs/promises";
import sharp from "sharp";
import { displayModes as nativeDisplayModes, type DisplayModes } from "./display-mode.ts";
-import { WINDOWS_POWERSHELL, type CommandRunner } from "./runner.ts";
+import { WINDOWS_POWERSHELL, runPowerShell, type CommandRunner } from "./runner.ts";
/** The Chromium switch a WebView2 app gets for debugging; port 0 lets each app choose a free port. */
export const WEBVIEW2_DEBUGGING = "--remote-debugging-port=0";
/** Quote one argument for a Windows command line as CommandLineToArgvW and the C runtime read it back. */
@@ -27,10 +27,7 @@ export class MachineHost {
constructor(private readonly runner: CommandRunner, readonly displayModes: DisplayModes = nativeDisplayModes(runner)) {}
private powershell(script: string, input?: string | Buffer, timeoutMs = 60_000): Promise<string> {
- return this.runner.run(WINDOWS_POWERSHELL, ["-NoLogo", "-NoProfile", "-NonInteractive", "-ExecutionPolicy", "Bypass", "-EncodedCommand",
- // UTF-8 output: the console code page would turn characters such as • into control bytes.
- Buffer.from(`$ProgressPreference = 'SilentlyContinue'; $ErrorActionPreference = 'Stop'; [Console]::OutputEncoding = [Text.Encoding]::UTF8\ntry {\n${script}\n} catch { [Console]::Error.WriteLine($_.Exception.Message); exit 1 }`, "utf16le").toString("base64")],
- { ...(input !== undefined ? { input } : {}), timeoutMs }).then(output => output.replace(/\r/g, "").trim());
+ return runPowerShell(this.runner, WINDOWS_POWERSHELL, script, { ...(input !== undefined ? { input } : {}), timeoutMs });
}
/** This session's folder in the machine's Windows temp directory, created on first use. */
diff --git a/src/machine.ts b/src/machine.ts
index b408a34..2c4302f 100644
--- a/src/machine.ts
+++ b/src/machine.ts
@@ -77,7 +77,7 @@ export function sandboxFactory(options: { settings: () => CuaVisionSettings; ret
export function machineFactory(options: { getMode: () => ComputerMode; settings: () => CuaVisionSettings; retainRaw?: (image: Content) => void }): MachineFactory {
return async (cwd, name, signal) => {
const machine = await machineConfig(cwd, name);
- const refresh = `/computer-setup machine ${machine.ssh} ${name}`;
+ const refresh = `setup part machine (user: /computer-setup machine ${machine.ssh} ${name}; agent, after asking: Insert {"action":"setup","part":"machine","ssh":"${machine.ssh}","name":"${name}"} into computer:machine/local)`;
const runner = sshRunner(machine.ssh);
const driver = new ReactiveAnimationDriver(new McpDriver({ command: machine.command, ...(machine.runtime ? { runtime: machine.runtime } : {}), ssh: machine.ssh }),
await loadReactiveAnimationConfig(cwd));
diff --git a/src/provider.ts b/src/provider.ts
index 7b2a5fe..dd95d89 100644
--- a/src/provider.ts
+++ b/src/provider.ts
@@ -21,7 +21,7 @@ const label = (value: unknown) => typeof value === "string" ? value.replace(/[\r
export const COMPUTER_PATH_HELP = "Read docs:computer for native resources. computer:session, computer:machine/<name> (local, or a configured machine) with its /windows, /apps and /displays, and returned computer:window/ref, display/ref, observation/id, window-state/id or sequence/id. Read window overviews before input; display/detail/frames are readonly.";
export const COMPUTER_VIEWS_HELP = "Computer: image/ui/full; dom alone or dom:{target,boxes,depth,log} on a browser window opened by open_browser or attached with attach_browser returns a fresh Playwright aria snapshot with [ref=...] values of the active tab; log:true adds the retained console/page-error/network history; target is {ref} or locator fields role,name,label,text,placeholder,test_id,css,exact,frames. On a DOM observation: image is a page screenshot, element:crop|highlight,<ref> an element image; Search filters its lines. state alone on a window returns fresh restore or management metadata without pixels/UI, plus whether the window is hung and its process memory/CPU. ocr (alone or with capture options on a window, alone on an observation, or with region:x,y,width,height on an observation) adds Windows OCR lines with boxes in overview pixels; use maxLongEdge 3840 for small text. changes (alone or with ui capture options, on a window) takes a fresh UI walk and lists only rows changed, added or removed since that window's previous observation, with the new observation's hints. regions (alone or with image/full options on a window, alone on its observation) lists icons and text areas found by Cua perception with boxes in overview pixels; click a box center with pixel input. capture:{maxLongEdge,scale,max_elements,max_depth,hints,guides}. region:x,y,width,height or element:inspect|scope|highlight|crop|text,hint on an observation; element:text returns one element's full text. sequence:{duration_ms<=60000,fps<=30,max_frames<=300,timeout_ms,maxLongEdge,scale,region} on window/display reports actual_fps and last_frame_age_ms; frames:first|last|all|endpoints|changed or frames:{indexes:[0]} / {from:0,to:2} on sequence. Detail/frames are readonly; text offset/limit select UI lines, never image cells.";
export const COMPUTER_SEARCH_HELP = "Search computer:machine/<name>/apps or computer:machine/<name>/windows (local is this machine) for installed-app/permitted-window metadata, or an observation for its collected UI/readonly DOM. Read computer:machine/<name>/displays for monitor metadata. Discovery launches nothing; observation Search performs no new native walk or DOM query. Treat missing matches in a partial projection as unknown.";
-export const COMPUTER_INPUT_HELP = "Computer Write sends content literally through type_text to a fresh UI/image observation address. On a DOM observation of a browser opened with Insert {\"action\":\"open_browser\"} (YOLO: \"profile\":\"agent\" keeps the agent's own logins between sessions), or of a running Chromium-based browser attached in YOLO with Insert {\"action\":\"attach_browser\"} into its window address (Chrome or Edge through a debugging port, the chrome://inspect/#remote-debugging switch or the Playwright Extension; an Electron/CEF app started with --remote-debugging-port=0 in open's args; a WebView2 app started with open \"debugging\":true or after {\"action\":\"webview2_debugging\",\"app\":\"x.exe\",\"enabled\":true} on its machine; VS Code's integrated browser through the Pi Browser Bridge; closing the session detaches), Insert one Playwright action such as {\"action\":\"fill\",\"target\":{\"label\":\"Name\"},\"text\":\"value\"}; actions target the active tab, and tab/navigation/dialog actions are listed in docs:computer. Insert {\"action\":\"open_sandbox\"} into a machine address to work in a separate disposable Windows Sandbox that does not touch the human's desktop, or {\"action\":\"open_session\"} for an isolated agent session on the same machine (several can run at once, each with its own desktop and foreground input); use the returned computer:sandbox/... address for Search, open and close (docs:computer). Machines are computer:machine/local (this one) and computer:machine/<configured name> (another Windows machine over SSH, started on first use); each takes the same commands. Insert {\"action\":\"put\",\"from\":\"<workspace file>\"}, {\"action\":\"get\",\"name\":\"<file>\",\"to\":\"<workspace path>\"} or {\"action\":\"files\"} into a machine address to move files through the session's own folder on that machine (removed on close); point app Open/Save dialogs there. The same addresses take {\"action\":\"clipboard_write\",\"text\"|\"image\":...}, {\"action\":\"clipboard_read\",\"image\":true}, {\"action\":\"record_start\",\"seconds\":30} and {\"action\":\"record_stop\",\"to\":\"<workspace path>\"} (main-display video) and {\"action\":\"audio_record\",\"seconds\":10,\"transcribe\":true} (playback audio with a transcript) in YOLO. Insert {\"action\":\"timeline\",\"events\":[{at_ms,type,...}]} with delivery=foreground into an observation for timed key/button/move/move_relative/wheel/pen events; everything held is released at the end (docs:computer). Insert {\"action\":\"resize\",\"width\":1920,\"height\":1080} into a display address to change that display's resolution in YOLO; closing the session restores it. Insert {\"action\":\"menu\",\"path\":[\"File\",\"Save As...\"]} into an observation to invoke a menu item by its labels, and {\"action\":\"kill\"} into the window address of an app this session opened to force-close it. Insert {\"action\":\"wait_for\"|\"verify\",\"conditions\":[...]} into a window address to check UI state; add \"expect\" or \"successor\":true to an action for a postcondition and change list, or Insert {\"action\":\"batch\",\"steps\":[...]} into an observation for a fail-stop chain (docs:computer). Use no Write or native input on a DOM observation. Use optional ?hint=...&delivery=foreground&delay_ms=0 only on input observations. No implicit select-all, submit or program execution. Computer Insert sends {\"action\":\"restore\"} to a fresh window-state address for explicit restoration and activation without keys/clicks, or one JSON action to an observation, or {\"action\":\"open\",\"app\":\"Exact installed name\"} (YOLO also {\"action\":\"open\",\"path\":\"C:\\\\...\\\\app.exe\",\"args\":[...]}) to a machine address, and {\"action\":\"close\"} to a machine address or to computer:session for every machine (the result lists each released part). Close releases runtime, not documents. Omit window_ref/observation_id in JSON. Input consumes observations even on refusal, except that delivered pixel-only input keeps its observation until the window is resized; refresh with Read. One sequence takes up to 2000 key, click and stroke steps. Never replay unknown prefixes. Foreground input requires an observed background refusal and pauses for 10 s after the human's own keyboard or mouse input; restore is a separate explicit activation, not input delivery. Safe/YOLO is user-controlled.";
+export const COMPUTER_INPUT_HELP = "Computer Write sends content literally through type_text to a fresh UI/image observation address. On a DOM observation of a browser opened with Insert {\"action\":\"open_browser\"} (YOLO: \"profile\":\"agent\" keeps the agent's own logins between sessions), or of a running Chromium-based browser attached in YOLO with Insert {\"action\":\"attach_browser\"} into its window address (Chrome or Edge through a debugging port, the chrome://inspect/#remote-debugging switch or the Playwright Extension; an Electron/CEF app started with --remote-debugging-port=0 in open's args; a WebView2 app started with open \"debugging\":true or after {\"action\":\"webview2_debugging\",\"app\":\"x.exe\",\"enabled\":true} on its machine; VS Code's integrated browser through the Pi Browser Bridge; closing the session detaches), Insert one Playwright action such as {\"action\":\"fill\",\"target\":{\"label\":\"Name\"},\"text\":\"value\"}; actions target the active tab, and tab/navigation/dialog actions are listed in docs:computer. Insert {\"action\":\"open_sandbox\"} into a machine address to work in a separate disposable Windows Sandbox that does not touch the human's desktop, or {\"action\":\"open_session\"} for an isolated agent session on the same machine (several can run at once, each with its own desktop and foreground input); use the returned computer:sandbox/... address for Search, open and close (docs:computer). Machines are computer:machine/local (this one) and computer:machine/<configured name> (another Windows machine over SSH, started on first use); each takes the same commands. Insert {\"action\":\"put\",\"from\":\"<workspace file>\"}, {\"action\":\"get\",\"name\":\"<file>\",\"to\":\"<workspace path>\"} or {\"action\":\"files\"} into a machine address to move files through the session's own folder on that machine (removed on close); point app Open/Save dialogs there. The same addresses take {\"action\":\"clipboard_write\",\"text\"|\"image\":...}, {\"action\":\"clipboard_read\",\"image\":true}, {\"action\":\"record_start\",\"seconds\":30} and {\"action\":\"record_stop\",\"to\":\"<workspace path>\"} (main-display video) and {\"action\":\"audio_record\",\"seconds\":10,\"transcribe\":true} (playback audio with a transcript) in YOLO. Insert {\"action\":\"timeline\",\"events\":[{at_ms,type,...}]} with delivery=foreground into an observation for timed key/button/move/move_relative/wheel/pen events; everything held is released at the end (docs:computer). Insert {\"action\":\"resize\",\"width\":1920,\"height\":1080} into a display address to change that display's resolution in YOLO; closing the session restores it. Only after the user explicitly allows it, Insert {\"action\":\"setup\",\"part\":\"browser|firefox|perception|mxc|vscode-bridge\"} into a machine address to install a missing optional part there (docs:computer). Insert {\"action\":\"menu\",\"path\":[\"File\",\"Save As...\"]} into an observation to invoke a menu item by its labels, and {\"action\":\"kill\"} into the window address of an app this session opened to force-close it. Insert {\"action\":\"wait_for\"|\"verify\",\"conditions\":[...]} into a window address to check UI state; add \"expect\" or \"successor\":true to an action for a postcondition and change list, or Insert {\"action\":\"batch\",\"steps\":[...]} into an observation for a fail-stop chain (docs:computer). Use no Write or native input on a DOM observation. Use optional ?hint=...&delivery=foreground&delay_ms=0 only on input observations. No implicit select-all, submit or program execution. Computer Insert sends {\"action\":\"restore\"} to a fresh window-state address for explicit restoration and activation without keys/clicks, or one JSON action to an observation, or {\"action\":\"open\",\"app\":\"Exact installed name\"} (YOLO also {\"action\":\"open\",\"path\":\"C:\\\\...\\\\app.exe\",\"args\":[...]}) to a machine address, and {\"action\":\"close\"} to a machine address or to computer:session for every machine (the result lists each released part). Close releases runtime, not documents. Omit window_ref/observation_id in JSON. Input consumes observations even on refusal, except that delivered pixel-only input keeps its observation until the window is resized; refresh with Read. One sequence takes up to 2000 key, click and stroke steps. Never replay unknown prefixes. Foreground input requires an observed background refusal and pauses for 10 s after the human's own keyboard or mouse input; restore is a separate explicit activation, not input delivery. Safe/YOLO is user-controlled.";
/** Present native resources without dumping provenance; full receipts remain in structured data. */
export function presentComputerResult(result: ComputerResult, full = false): string {
diff --git a/src/runner.ts b/src/runner.ts
index 08a1f2e..5383e37 100644
--- a/src/runner.ts
+++ b/src/runner.ts
@@ -16,6 +16,26 @@ export interface CommandRunner {
/** Windows PowerShell as seen from WSL. */
export const WINDOWS_POWERSHELL = "/mnt/c/Windows/System32/WindowsPowerShell/v1.0/powershell.exe";
+/**
+ * Run a PowerShell script on the runner's Windows with UTF-8 output and stop-on-error; a thrown error exits 1 with
+ * its message on stderr. `powershell` is the executable as the runner sees it: WINDOWS_POWERSHELL from WSL,
+ * "powershell.exe" when Pi itself runs on Windows.
+ */
+export function runPowerShell(runner: CommandRunner, powershell: string, script: string,
+ options: { input?: string | Buffer; timeoutMs?: number; signal?: AbortSignal; scriptOnStdin?: boolean } = {}): Promise<string> {
+ // UTF-8 output: the console code page would turn characters such as • into control bytes.
+ const wrapped = `$ProgressPreference = 'SilentlyContinue'; $ErrorActionPreference = 'Stop'; [Console]::OutputEncoding = [Text.Encoding]::UTF8\ntry {\n${script}\n} catch { [Console]::Error.WriteLine($_.Exception.Message); exit 1 }`;
+ const run = (args: string[], input?: string | Buffer) => runner.run(powershell, ["-NoLogo", "-NoProfile", "-NonInteractive", "-ExecutionPolicy", "Bypass", ...args],
+ { ...(input !== undefined ? { input } : {}), timeoutMs: options.timeoutMs ?? 60_000, ...(options.signal ? { signal: options.signal } : {}) })
+ .then(output => output.replace(/\r/g, "").trim());
+ // Some long -EncodedCommand values fail to start through WSL interop ("Invalid argument"); a long script can
+ // instead arrive on stdin as base64 of UTF-8, behind a short fixed command line.
+ if (options.scriptOnStdin) return run(["-Command",
+ "& ([ScriptBlock]::Create([Text.Encoding]::UTF8.GetString([Convert]::FromBase64String([Console]::In.ReadToEnd().Trim()))))"],
+ Buffer.from(wrapped, "utf8").toString("base64"));
+ return run(["-EncodedCommand", Buffer.from(wrapped, "utf16le").toString("base64")], options.input);
+}
+
/** Run helpers on this machine. Windows programs need the WSL interop socket. */
export function localRunner(interop?: string): CommandRunner {
const environment = (env?: Record<string, string>) => ({ ...process.env, ...(interop ? { WSL_INTEROP: interop } : {}), ...env });
diff --git a/src/session.ts b/src/session.ts
index c2eb66a..e958b7c 100644
--- a/src/session.ts
+++ b/src/session.ts
@@ -1335,7 +1335,7 @@ export class ComputerSession {
const native = await this.driver.call("parse_visual_regions", { capture_id: observed.captureId, options: { max_regions: 300 } }, signal);
if (native.isError) throw new Error((native.content.filter(c => c.type === "text").map(c => c.text).join(" ") || "parse_visual_regions failed")
+ " The regions view needs the Cua perception extension on this machine; if it is missing, ask the user before installing it "
- + "(about 420 MB) and with their permission run /computer-setup perception (add the machine name for another machine).");
+ + "(about 420 MB) and with their permission Insert {\"action\":\"setup\",\"part\":\"perception\"} into this machine's address.");
const parsed = native.structuredContent as { capture: { screenshot: { width: number } }; regions: {
kind: "icon" | "text"; bounds: { x: number; y: number; width: number; height: number }; text?: string | null; label?: string | null; confidence: number }[] };
const scale = (observed.width ?? parsed.capture.screenshot.width) / parsed.capture.screenshot.width;
diff --git a/src/setup-command.ts b/src/setup-command.ts
index 3285b79..5894294 100644
--- a/src/setup-command.ts
+++ b/src/setup-command.ts
@@ -1,22 +1,5 @@
-import { spawn } from "node:child_process";
-import { createWriteStream } from "node:fs";
-import { mkdtemp } from "node:fs/promises";
-import { tmpdir } from "node:os";
-import path from "node:path";
-import { fileURLToPath } from "node:url";
import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
-
-export const SETUP_SCRIPT = fileURLToPath(new URL("../scripts/setup.sh", import.meta.url));
-
-/** Optional parts that scripts/setup.sh installs, for argument completion. */
-export const SETUP_COMPONENTS: { name: string; description: string }[] = [
- { name: "browser", description: "Portable Windows Node for open_browser and attach_browser" },
- { name: "firefox", description: "Playwright's Firefox for open_browser" },
- { name: "perception", description: "Cua perception for the regions view (about 420 MB)" },
- { name: "mxc", description: "Microsoft MXC for isolated agent sessions" },
- { name: "vscode-bridge", description: "Pi Browser Bridge for VS Code's integrated browser" },
- { name: "machine", description: "Prepare another Windows machine over SSH" },
-];
+import { SETUP_PARTS, runSetup, type SetupPart, type SetupRequest } from "./setup.ts";
/** Split command arguments on spaces, keeping single- or double-quoted paths whole. */
export function splitArgs(text: string): string[] {
@@ -25,47 +8,44 @@ export function splitArgs(text: string): string[] {
return args;
}
-/** Run the setup script; the whole output goes to a log file, the tail comes back for the notification. */
-export async function runSetup(args: string[], cwd: string, script = SETUP_SCRIPT): Promise<{ code: number; tail: string; log: string }> {
- const log = path.join(await mkdtemp(path.join(tmpdir(), "pi-computer-setup-")), "setup.log");
- const file = createWriteStream(log);
- let tail = "";
- const code = await new Promise<number>((resolve, reject) => {
- const child = spawn("bash", [script, ...args], { cwd, stdio: ["ignore", "pipe", "pipe"] });
- const take = (chunk: Buffer) => { file.write(chunk); tail = (tail + chunk.toString("utf8")).slice(-4000); };
- child.stdout.on("data", take);
- child.stderr.on("data", take);
- child.on("error", reject);
- child.on("close", status => resolve(status ?? 1));
- });
- await new Promise(resolve => file.end(resolve));
- // Progress bars redraw with carriage returns; keep only the last state of each line.
- const lines = tail.split("\n").map(line => line.split("\r").at(-1)!.trimEnd()).filter(Boolean);
- return { code, tail: lines.slice(-15).join("\n"), log };
+export const SETUP_USAGE = ["Usage: /computer-setup PART [local|MACHINE]", "",
+ ...SETUP_PARTS.flatMap(part => [part.name === "machine" ? "machine SSH_ALIAS [NAME]" : part.name === "vscode-bridge" ? "vscode-bridge [MACHINE] [CODE_CMD ARGV_JSON]" : part.name, ` ${part.description}`]),
+ "", "Everything goes into the Windows user's own folders; downloads are checked against pinned SHA-256 sums."].join("\n");
+
+/** Turn command words into a request; undefined shows the usage. */
+export function parseSetupArgs(args: string[]): SetupRequest | undefined {
+ const [part, ...rest] = args;
+ if (!part || !SETUP_PARTS.some(item => item.name === part)) return undefined;
+ if (part === "machine") return rest.length >= 1 && rest.length <= 2 ? { part, ssh: rest[0], ...(rest[1] ? { name: rest[1] } : {}) } : undefined;
+ if (part === "vscode-bridge") {
+ if (rest.length > 3) return undefined;
+ const machine = rest.length % 2 === 1 ? rest[0] : undefined;
+ const paths = machine === undefined ? rest : rest.slice(1);
+ return { part, ...(machine ? { machine } : {}), ...(paths.length ? { code: paths[0], argv: paths[1] } : {}) };
+ }
+ return rest.length <= 1 ? { part: part as SetupPart, ...(rest[0] ? { machine: rest[0] } : {}) } : undefined;
}
-/** /computer-setup COMPONENT [local|MACHINE]: the user's own entry point to the optional installs. */
-export function registerSetupCommand(pi: ExtensionAPI, script = SETUP_SCRIPT): void {
+/** /computer-setup PART [local|MACHINE]: the user's own entry point to the optional installs. */
+export function registerSetupCommand(pi: ExtensionAPI, run: typeof runSetup = runSetup): void {
pi.registerCommand("computer-setup", {
description: "Install an optional computer-use part: browser, firefox, perception, mxc, vscode-bridge or machine",
getArgumentCompletions: prefix => {
if (/\s/.test(prefix.trimStart())) return null;
- const items = SETUP_COMPONENTS.filter(item => item.name.startsWith(prefix.trim()))
+ const items = SETUP_PARTS.filter(item => item.name.startsWith(prefix.trim()))
.map(item => ({ value: item.name, label: item.name, description: item.description }));
return items.length ? items : null;
},
handler: async (text, ctx) => {
- const args = splitArgs(text);
- if (!args.length) {
- const { tail } = await runSetup([], ctx.cwd, script);
- ctx.ui.notify(tail, "info");
- return;
- }
- ctx.ui.setStatus("computer-setup", `Computer setup: ${args[0]}…`);
+ const request = parseSetupArgs(splitArgs(text));
+ if (!request) { ctx.ui.notify(SETUP_USAGE, text.trim() ? "error" : "info"); return; }
+ const label = `Computer setup: ${request.part}`;
+ ctx.ui.setStatus("computer-setup", `${label}…`);
try {
- const { code, tail, log } = await runSetup(args, ctx.cwd, script);
- ctx.ui.notify(`${code === 0 ? "Computer setup finished" : `Computer setup failed (exit ${code})`}: ${args.join(" ")}\n${tail}\nFull log: ${log}`,
- code === 0 ? "info" : "error");
+ const result = await run(request, ctx.cwd, { progress: step => ctx.ui.setStatus("computer-setup", `${label}: ${step}…`) });
+ ctx.ui.notify(result, "info");
+ } catch (error) {
+ ctx.ui.notify(`${label} failed: ${error instanceof Error ? error.message : error}`, "error");
} finally { ctx.ui.setStatus("computer-setup", undefined); }
},
});
diff --git a/src/setup-pins.ts b/src/setup-pins.ts
new file mode 100644
index 0000000..65a946d
--- /dev/null
+++ b/src/setup-pins.ts
@@ -0,0 +1,24 @@
+/** Exact downloads for the optional parts; every file is checked against its SHA-256 on the target before use. */
+
+/** Portable Windows Node for the Playwright worker when Pi runs in WSL or on another machine. */
+export const WINDOWS_NODE = {
+ version: "v22.22.2",
+ url: "https://nodejs.org/dist/v22.22.2/node-v22.22.2-win-x64.zip",
+ sha256: "7c93e9d92bf68c07182b471aa187e35ee6cd08ef0f24ab060dfff605fcc1c57c",
+};
+
+/** Microsoft Execution Containers; only x64/wxc-exec.exe is used, for isolated agent sessions. */
+export const MXC = {
+ version: "v1.0.1",
+ url: "https://github.com/microsoft/mxc/releases/download/v1.0.1/mxc-release-binaries.zip",
+ sha256: "0e832239454ba7fcd338ff56218a287eef4c0a01c1484084b869f55feff8abe9",
+};
+
+/** Cua's signed perception extension; the driver also verifies the catalog signature and every hash. */
+export const PERCEPTION = {
+ version: "0.2.1",
+ name: "cua-perception-0.2.1-x86_64-pc-windows-msvc",
+ release: "https://github.com/trycua/cua/releases/download/cua-perception-v0.2.1/",
+ catalog: "bfb2300ff9da054d522e73351f7b337ba6157ebbe30f029a95a471fc3d13a042",
+ archive: "6709579aa3f938e8200c0a51b1abe450d4fda51b09e97517e0cb4c272b5bff37",
+};
diff --git a/src/setup.ts b/src/setup.ts
new file mode 100644
index 0000000..8d635ca
--- /dev/null
+++ b/src/setup.ts
@@ -0,0 +1,351 @@
+import { execFile, spawn } from "node:child_process";
+import { copyFile, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
+import { createRequire } from "node:module";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+import { loadComputerConfig, readMachineSettings, type MachineConfig } from "./config.ts";
+import { PACKAGED_RUNTIME, fetchRuntime, isWsl, loadBundledRuntime, readRuntimePins } from "./native-runtime.ts";
+import { WINDOWS_POWERSHELL, localRunner, runPowerShell, sshRunner, type CommandRunner } from "./runner.ts";
+import { MXC, PERCEPTION, WINDOWS_NODE } from "./setup-pins.ts";
+import { findWindowsNode } from "./windows-node.ts";
+
+/** Optional parts and what each adds; `machine` prepares another machine instead of installing on one. */
+export const SETUP_PARTS = [
+ { name: "browser", description: "Windows Node for open_browser and attach_browser when Pi runs in WSL (about 30 MB)" },
+ { name: "firefox", description: "Playwright's Firefox for \"browser\":\"firefox\"" },
+ { name: "perception", description: "Cua perception for the regions view (about 420 MB, AGPL-3.0 icon detector)" },
+ { name: "mxc", description: "Microsoft MXC for open_session (Windows 11 24H2 or later)" },
+ { name: "vscode-bridge", description: "Pi Browser Bridge for VS Code's integrated browser" },
+ { name: "machine", description: "Prepare another Windows machine with WSL, reached over SSH" },
+] as const;
+export type SetupPart = (typeof SETUP_PARTS)[number]["name"];
+
+export interface SetupRequest {
+ part: SetupPart;
+ /** A configured machine name; this machine when omitted or "local". Not used by `machine`. */
+ machine?: string;
+ /** For `machine`: the SSH alias and the name to configure it under. */
+ ssh?: string;
+ name?: string;
+ /** For `vscode-bridge` with a portable VS Code: its bin\code.cmd and data\argv.json on the target. */
+ code?: string;
+ argv?: string;
+}
+
+/** Thrown for a request that cannot start; nothing was installed. */
+export class SetupError extends Error {}
+
+/** The Windows machine a part installs on, and how its commands run. */
+interface Target {
+ label: string;
+ runner: CommandRunner;
+ /** PowerShell as the runner sees it. */
+ powershell: string;
+ /** Pi itself runs on this Windows: paths are Windows paths and no WSL is involved. */
+ native: boolean;
+ machine?: MachineConfig;
+ /** WSL interop socket from the trusted settings. */
+ interop?: string;
+}
+
+export interface SetupDeps {
+ platform: NodeJS.Platform;
+ wsl: () => Promise<boolean>;
+ local: (interop?: string) => CommandRunner;
+ ssh: (alias: string) => CommandRunner;
+}
+const defaultDeps: SetupDeps = { platform: process.platform, wsl: isWsl, local: localRunner, ssh: sshRunner };
+
+type Progress = (message: string) => void;
+const BRIDGE_ID = "alexshpunt.pi-browser-bridge";
+const PACKAGE = fileURLToPath(new URL("..", import.meta.url));
+const psString = (value: string) => `'${value.replace(/'/g, "''")}'`;
+
+/**
+ * Downloads are checked on the target before use; native tools print their own output, and their failure stops
+ * the script with that output. tar.exe ships with Windows 10 and later and unpacks zip much faster than .NET ZipFile.
+ */
+const PRELUDE = `
+# A WSL working directory is a UNC path that cmd-based tools such as code.cmd cannot use.
+Set-Location -LiteralPath $env:TEMP
+[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12
+# .NET directly: Get-FileHash and Expand-Archive come from script modules that some PowerShell setups fail to load,
+# and WebClient downloads far faster than Invoke-WebRequest in Windows PowerShell.
+function Get-Sha256([string]$File) {
+ $stream = [IO.File]::OpenRead($File)
+ try { -join ([Security.Cryptography.SHA256]::Create().ComputeHash($stream) | ForEach-Object { $_.ToString('x2') }) } finally { $stream.Dispose() }
+}
+function Get-Pinned([string]$Url, [string]$File, [string]$Sha256) {
+ if ((Test-Path -LiteralPath $File) -and (Get-Sha256 $File) -eq $Sha256) { return }
+ (New-Object Net.WebClient).DownloadFile($Url, $File)
+ if ((Get-Sha256 $File) -ne $Sha256) {
+ Remove-Item -LiteralPath $File -Force
+ throw "The download from $Url does not match its pinned SHA-256; nothing was installed."
+ }
+}
+function Invoke-Native([string]$Exe, [string[]]$Arguments, [switch]$AllowFailure) {
+ $ErrorActionPreference = 'Continue'
+ $out = (& $Exe @Arguments 2>&1 | ForEach-Object { "$_" }) -join [Environment]::NewLine
+ if ($LASTEXITCODE -ne 0 -and !$AllowFailure) { throw ([IO.Path]::GetFileName($Exe) + " exited with $($LASTEXITCODE): " + $out) }
+ $out
+}
+function Expand-Zip([string]$Zip, [string]$Destination) {
+ New-Item -ItemType Directory -Force -Path $Destination | Out-Null
+ $tar = Join-Path $env:SystemRoot 'System32\\tar.exe'
+ if (Test-Path -LiteralPath $tar) { Invoke-Native $tar @('-xf', $Zip, '-C', $Destination) | Out-Null }
+ else { Add-Type -AssemblyName System.IO.Compression.FileSystem; [IO.Compression.ZipFile]::ExtractToDirectory($Zip, $Destination) }
+}
+`;
+
+const ps = (target: Target, script: string, timeoutMs: number, signal?: AbortSignal) =>
+ runPowerShell(target.runner, target.powershell, PRELUDE + script, { timeoutMs, scriptOnStdin: true, ...(signal ? { signal } : {}) });
+
+/** A Windows path on the target as its runner addresses files. */
+async function targetPath(target: Target, windows: string): Promise<string> {
+ return target.native ? windows : (await target.runner.run("wslpath", ["-u", windows])).trim();
+}
+
+/** A target path (Windows, or the target's WSL path) as Windows sees it. */
+async function windowsPath(target: Target, value: string): Promise<string> {
+ return target.native || /^[A-Za-z]:\\/.test(value) ? value : (await target.runner.run("wslpath", ["-w", value])).trim();
+}
+
+/** Write bytes to a Windows path on the target. */
+async function put(target: Target, windows: string, bytes: Buffer): Promise<void> {
+ const file = await targetPath(target, windows);
+ if (!target.machine) await writeFile(file, bytes);
+ else await target.runner.run("dd", ["status=none", `of=${file}`], { input: bytes, timeoutMs: 600_000 });
+}
+
+async function resolveTarget(cwd: string, machine: string | undefined, deps: SetupDeps): Promise<Target> {
+ const settings = await readMachineSettings(cwd);
+ if (machine && machine !== "local") {
+ const config = settings.machines[machine];
+ if (!config) throw new SetupError(`No machine named ${machine} is configured in .pi/computer-use.json; nothing was installed.`);
+ return { label: `machine ${machine}`, runner: deps.ssh(config.ssh), powershell: WINDOWS_POWERSHELL, native: false, machine: config };
+ }
+ if (deps.platform === "win32") return { label: "this machine", runner: deps.local(), powershell: "powershell.exe", native: true };
+ if (deps.platform === "linux" && await deps.wsl())
+ return { label: "this machine", runner: deps.local(settings.interop), powershell: WINDOWS_POWERSHELL, native: false, ...(settings.interop ? { interop: settings.interop } : {}) };
+ throw new SetupError("Setup needs Pi on Windows or in WSL; nothing was installed.");
+}
+
+/** Install the pinned portable Node on the target and return its node.exe as a Windows path. */
+async function installNode(target: Target, signal?: AbortSignal): Promise<{ exe: string; installed: boolean }> {
+ const name = `node-${WINDOWS_NODE.version}-win-x64`;
+ const out = await ps(target, `
+$root = Join-Path $env:LOCALAPPDATA 'Programs\\PiComputerUse'
+$exe = Join-Path $root ${psString(`${name}\\node.exe`)}
+if (Test-Path -LiteralPath $exe) { 'present'; $exe; return }
+New-Item -ItemType Directory -Force -Path $root | Out-Null
+$zip = Join-Path $env:TEMP ${psString(`pi-${name}.zip`)}
+Get-Pinned ${psString(WINDOWS_NODE.url)} $zip ${psString(WINDOWS_NODE.sha256)}
+$stage = Join-Path $root ('.staging-' + [guid]::NewGuid())
+Expand-Zip $zip $stage
+Move-Item -LiteralPath (Join-Path $stage ${psString(name)}) -Destination $root
+Remove-Item -LiteralPath $stage, $zip -Recurse -Force
+'installed'; $exe`, 600_000, signal);
+ const [state, exe] = out.split("\n").slice(-2);
+ return { exe, installed: state === "installed" };
+}
+
+async function browser(target: Target, signal?: AbortSignal): Promise<string> {
+ if (target.native) return "Pi runs on Windows here, so browsers use Pi's own Node. Nothing to install.";
+ const node = await installNode(target, signal);
+ return node.installed ? `Installed Windows Node ${WINDOWS_NODE.version} on ${target.label}: ${node.exe}` : `Windows Node is already installed on ${target.label}: ${node.exe}`;
+}
+
+function needsBrowserCopy(target: Target): never {
+ throw new SetupError(`${target.label} has no browser worker yet; prepare it with setup part machine (ssh ${target.machine!.ssh}) first. Nothing was installed.`);
+}
+
+async function firefox(target: Target, signal?: AbortSignal): Promise<string> {
+ const cli = path.join(path.dirname(createRequire(import.meta.url).resolve("playwright-core/package.json")), "cli.js");
+ let node: string, script: string;
+ if (target.machine) {
+ if (!target.machine.node || !target.machine.package) needsBrowserCopy(target);
+ node = target.machine.node;
+ script = await windowsPath(target, `${target.machine.package}/node_modules/playwright-core/cli.js`);
+ } else if (target.native) { node = process.execPath; script = cli; }
+ else {
+ node = await findWindowsNode(target.interop)
+ .catch(error => { throw new SetupError(`${error instanceof Error ? error.message : error} Firefox needs it too; install browser first.`); });
+ script = await windowsPath(target, cli);
+ }
+ const out = await target.runner.run(node, [script, "install", "firefox"], { timeoutMs: 900_000, ...(signal ? { signal } : {}) });
+ return `Playwright's Firefox is ready on ${target.label}.${tail(out)}`;
+}
+
+async function perception(target: Target, cwd: string, signal?: AbortSignal): Promise<string> {
+ const driver = await windowsPath(target, target.machine ? target.machine.command : (await loadComputerConfig(cwd)).command);
+ const { name, release, catalog, archive, version } = PERCEPTION;
+ const out = await ps(target, `
+$driver = ${psString(driver)}
+$status = Invoke-Native $driver @('extension', 'status', 'perception') -AllowFailure
+if ($status -notmatch ${psString(`Active version: ${version.replace(/\./g, "\\.")}`)}) {
+ $dir = Join-Path $env:TEMP 'pi-perception'
+ New-Item -ItemType Directory -Force -Path $dir | Out-Null
+ $catalog = Join-Path $dir ${psString(`${name}.catalog.json`)}
+ Get-Pinned ${psString(`${release}${name}.catalog.json`)} $catalog ${psString(catalog)}
+ Get-Pinned ${psString(`${release}${name}.tar.gz`)} (Join-Path $dir ${psString(`${name}.tar.gz`)}) ${psString(archive)}
+ Invoke-Native $driver @('extension', 'install', 'perception', '--catalog', $catalog)
+ Remove-Item -LiteralPath $dir -Recurse -Force
+}
+Invoke-Native $driver @('extension', 'status', 'perception', '--self-test')`, 2_400_000, signal);
+ return `Cua perception ${version} is ready on ${target.label}.${tail(out)}`;
+}
+
+async function mxc(target: Target, signal?: AbortSignal): Promise<string> {
+ const out = await ps(target, `
+$dir = Join-Path $env:LOCALAPPDATA 'PiComputerUse\\mxc'
+$zip = Join-Path $env:TEMP ${psString(`pi-mxc-${MXC.version}.zip`)}
+Get-Pinned ${psString(MXC.url)} $zip ${psString(MXC.sha256)}
+$stage = Join-Path $env:TEMP ('pi-mxc-' + [guid]::NewGuid())
+Expand-Zip $zip $stage
+New-Item -ItemType Directory -Force -Path $dir | Out-Null
+Copy-Item -LiteralPath (Join-Path $stage 'x64\\wxc-exec.exe') -Destination $dir -Force
+Remove-Item -LiteralPath $stage, $zip -Recurse -Force
+Join-Path $dir 'wxc-exec.exe'`, 600_000, signal);
+ return `Installed MXC ${MXC.version} on ${target.label}: ${out.split("\n").at(-1)}`;
+}
+
+/** Add the bridge to VS Code's argv.json (JSONC) without rewriting the rest; undefined when it is already allowed. */
+export function allowProposedApi(text: string, id = BRIDGE_ID): string | undefined {
+ if (text.includes(id)) return undefined;
+ if (!text.trim()) text = "{\n}\n";
+ // Whether only whitespace and comments come before the closing bracket, so no separating comma is needed.
+ const empty = (rest: string, close: string) => rest.replace(/\/\/[^\n]*|\/\*[\s\S]*?\*\/|\s/g, "").startsWith(close);
+ const list = /"enable-proposed-api"\s*:\s*\[/.exec(text);
+ if (list) {
+ const at = list.index + list[0].length;
+ return text.slice(0, at) + JSON.stringify(id) + (empty(text.slice(at), "]") ? "" : ", ") + text.slice(at);
+ }
+ const brace = text.indexOf("{") + 1;
+ if (!brace) throw new SetupError("VS Code's argv.json has no JSON object; fix or remove it, then run setup again.");
+ return `${text.slice(0, brace)}\n\t"enable-proposed-api": [${JSON.stringify(id)}]${empty(text.slice(brace), "}") ? "" : ","}${text.slice(brace)}`;
+}
+
+/** Run a command on the machine where Pi runs; npm and npx are .cmd files on Windows. */
+function local(command: string, args: string[], cwd: string, signal?: AbortSignal): Promise<string> {
+ return new Promise((resolve, reject) => execFile(command, args, { cwd, shell: process.platform === "win32", timeout: 600_000, maxBuffer: 16 * 1024 * 1024, ...(signal ? { signal } : {}) },
+ (error, stdout, stderr) => error ? reject(new Error(`${command} ${args[0]} failed: ${(stderr || error.message).trim().slice(-800)}`)) : resolve(stdout)));
+}
+
+/** Package the bridge with its ws dependency into a .vsix. */
+async function buildBridge(signal?: AbortSignal): Promise<Buffer> {
+ const work = await mkdtemp(path.join(tmpdir(), "pi-browser-bridge-"));
+ try {
+ for (const file of ["package.json", "extension.js", "README.md"]) await copyFile(path.join(PACKAGE, "vscode-browser-bridge", file), path.join(work, file));
+ await local("npm", ["install", "--omit=dev", "--no-audit", "--no-fund", "--silent"], work, signal);
+ await local("npx", ["--yes", "@vscode/[email protected]", "package", "--skip-license", "-o", "pi-browser-bridge.vsix"], work, signal);
+ return await readFile(path.join(work, "pi-browser-bridge.vsix"));
+ } finally { await rm(work, { recursive: true, force: true }); }
+}
+
+async function vscodeBridge(target: Target, request: SetupRequest, signal?: AbortSignal): Promise<string> {
+ if (!request.code !== !request.argv) throw new SetupError("For a portable VS Code pass both its bin\\code.cmd and its data\\argv.json; nothing was installed.");
+ const vsix = await buildBridge(signal);
+ const temp = await ps(target, "$env:TEMP", 30_000, signal);
+ const file = `${temp}\\pi-browser-bridge.vsix`;
+ await put(target, file, vsix);
+ const code = request.code ? await windowsPath(target, request.code) : "";
+ const argv = request.argv ? await windowsPath(target, request.argv) : "";
+ const found = await ps(target, `
+$code = ${psString(code)}
+if (!$code) {
+ foreach ($candidate in @((Join-Path $env:LOCALAPPDATA 'Programs\\Microsoft VS Code\\bin\\code.cmd'), (Join-Path $env:ProgramFiles 'Microsoft VS Code\\bin\\code.cmd'))) {
+ if (Test-Path -LiteralPath $candidate) { $code = $candidate; break }
+ }
+}
+if (!$code) { throw 'No installed VS Code found. For a portable VS Code pass its bin\\code.cmd and data\\argv.json.' }
+$argv = ${psString(argv)}
+if (!$argv) { $argv = Join-Path $env:USERPROFILE '.vscode\\argv.json' }
+Invoke-Native $code @('--install-extension', ${psString(file)}, '--force') | Out-Null
+Remove-Item -LiteralPath ${psString(file)} -Force
+$text = if (Test-Path -LiteralPath $argv) { [IO.File]::ReadAllText($argv) } else { '' }
+$code
+$argv
+[Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($text))`, 300_000, signal);
+ const [codeCmd, argvJson, encoded] = found.split("\n").slice(-3);
+ const next = allowProposedApi(Buffer.from(encoded, "base64").toString("utf8"));
+ if (next !== undefined) await ps(target, `
+$argv = ${psString(argvJson)}
+New-Item -ItemType Directory -Force -Path (Split-Path -Parent $argv) | Out-Null
+[IO.File]::WriteAllText($argv, [Text.Encoding]::UTF8.GetString([Convert]::FromBase64String(${psString(Buffer.from(next).toString("base64"))})), (New-Object Text.UTF8Encoding $false))`, 30_000, signal);
+ return `Installed the Pi Browser Bridge into ${codeCmd} on ${target.label}. ${argvJson} ${next === undefined ? "already allows" : "now allows"} its proposed API. Quit VS Code fully and start it again; Reload Window does not reread argv.json.`;
+}
+
+/** Stream a local folder into a fresh folder on the remote WSL through tar. */
+async function copyTree(runner: CommandRunner, from: string, to: string): Promise<void> {
+ await runner.run("rm", ["-rf", to]);
+ await runner.run("mkdir", ["-p", to]);
+ await new Promise<void>((resolve, reject) => {
+ const remote = runner.spawn("tar", ["-x", "-f", "-", "-C", to]);
+ const pack = spawn("tar", ["-c", "-f", "-", "-C", from, "."], { stdio: ["ignore", "pipe", "pipe"] });
+ let failed = "";
+ pack.stderr.on("data", chunk => { failed += chunk; });
+ remote.stderr.on("data", chunk => { failed += chunk; });
+ pack.stdout.pipe(remote.stdin);
+ pack.on("error", reject);
+ remote.on("error", reject);
+ remote.on("close", code => code === 0 ? resolve() : reject(new Error(`Copying ${path.basename(from)} failed: ${failed.trim().slice(-500)}`)));
+ });
+}
+
+async function prepareMachine(request: SetupRequest, deps: SetupDeps, progress: Progress, signal?: AbortSignal): Promise<string> {
+ const alias = request.ssh;
+ if (!alias || !/^[A-Za-z0-9][A-Za-z0-9._@-]{0,127}$/.test(alias)) throw new SetupError("machine needs the SSH alias of a Windows machine with WSL; nothing was installed.");
+ const name = request.name ?? alias;
+ if (!/^[A-Za-z0-9][A-Za-z0-9_-]{0,63}$/.test(name) || name === "local") throw new SetupError("The machine name must be letters, digits, - or _, and not local.");
+ const target: Target = { label: `machine ${name}`, runner: deps.ssh(alias), powershell: WINDOWS_POWERSHELL, native: false, machine: { ssh: alias, command: "" } };
+ const { runner } = target;
+
+ progress("checking the local runtime");
+ const pins = await readRuntimePins();
+ await fetchRuntime(PACKAGED_RUNTIME, pins, signal);
+ const runtime = await loadBundledRuntime(PACKAGED_RUNTIME, pins);
+ progress("copying the driver");
+ const appData = await targetPath(target, await ps(target, "$env:LOCALAPPDATA", 30_000, signal));
+ const folder = `${appData}/Programs/PiComputerUse/cua-driver-${pins.version}-${pins.source.slice(0, 8)}/runtime`;
+ await runner.run("mkdir", ["-p", folder]);
+ const names = Object.keys(pins.files);
+ for (const file of names) await runner.run("dd", ["status=none", `of=${folder}/${file}`], { input: await readFile(path.join(runtime.root, "runtime", file)), timeoutMs: 600_000 });
+ const sums = (await runner.run("sha256sum", names.map(file => `${folder}/${file}`))).trim().split("\n").map(line => line.split(" ")[0]);
+ const bad = names.filter((file, index) => sums[index] !== pins.files[file]);
+ if (bad.length) throw new SetupError(`The copied driver files do not match their pins (${bad.join(", ")}).`);
+
+ progress("installing Windows Node");
+ const node = await targetPath(target, (await installNode(target, signal)).exe);
+ progress("copying the browser worker");
+ const home = (await runner.run("bash", ["-c", 'printf %s "$HOME"'])).trim();
+ const remotePackage = `${home}/.local/share/pi-computer-use/package`;
+ await copyTree(runner, path.join(PACKAGE, "src"), `${remotePackage}/src`);
+ await copyTree(runner, path.dirname(createRequire(import.meta.url).resolve("playwright-core/package.json")), `${remotePackage}/node_modules/playwright-core`);
+ await runner.run("dd", ["status=none", `of=${remotePackage}/package.json`], { input: await readFile(path.join(PACKAGE, "package.json")) });
+
+ const entry = { ssh: alias, runtime: "windows-desktop", command: `${folder}/cua-driver.exe`, node, package: remotePackage };
+ return `Prepared ${alias} for computer use. Add this under "machines" in .pi/computer-use.json:\n${JSON.stringify({ [name]: entry }, null, 2)}`;
+}
+
+const tail = (out: string) => {
+ const lines = out.replace(/\r/g, "").split("\n").map(line => line.trimEnd()).filter(Boolean);
+ return lines.length ? `\n${lines.slice(-8).join("\n")}` : "";
+};
+
+/** Install one optional part on this machine or a configured one, or prepare another machine. */
+export async function runSetup(request: SetupRequest, cwd: string, options: { progress?: Progress; signal?: AbortSignal; deps?: Partial<SetupDeps> } = {}): Promise<string> {
+ const deps = { ...defaultDeps, ...options.deps };
+ const progress = options.progress ?? (() => undefined);
+ if (!SETUP_PARTS.some(part => part.name === request.part)) throw new SetupError(`Unknown part ${request.part}. Parts: ${SETUP_PARTS.map(part => part.name).join(", ")}.`);
+ if (request.part === "machine") return prepareMachine(request, deps, progress, options.signal);
+ const target = await resolveTarget(cwd, request.machine, deps);
+ progress(`working on ${target.label}`);
+ switch (request.part) {
+ case "browser": return browser(target, options.signal);
+ case "firefox": return firefox(target, options.signal);
+ case "perception": return perception(target, cwd, options.signal);
+ case "mxc": return mxc(target, options.signal);
+ case "vscode-bridge": return vscodeBridge(target, request, options.signal);
+ }
+}
diff --git a/src/web-browser.ts b/src/web-browser.ts
index 9335d34..2ebc9fe 100644
--- a/src/web-browser.ts
+++ b/src/web-browser.ts
@@ -61,7 +61,7 @@ export interface WebDownload {
state: "in_progress" | "completed" | "canceled" | "failed";
path?: string; bytes?: number; error?: string;
}
-/** Installed Chrome or Edge launched by Playwright, or Playwright's own Firefox build (/computer-setup firefox). */
+/** Installed Chrome or Edge launched by Playwright, or Playwright's own Firefox build (setup part firefox). */
export type WebChannel = "chrome" | "msedge" | "firefox";
/**
* Browser launch request. `profile: "agent"` uses the agent's own persistent profile on that machine (YOLO only):
diff --git a/src/web-worker.ts b/src/web-worker.ts
index 9776c68..76bb87f 100644
--- a/src/web-worker.ts
+++ b/src/web-worker.ts
@@ -163,8 +163,8 @@ async function open({ browser, url, profile: kind, attach: target }: WebLaunchOp
context = browser === "firefox"
? await firefox.launchPersistentContext(directory, { headless: false, viewport: null }).catch((error: Error) => {
if (!/Executable doesn't exist/.test(error.message)) throw error;
- throw new Error("Playwright's Firefox is not installed on this machine. Ask the user before installing it; with their permission run "
- + "/computer-setup firefox (add the machine name for another machine), then open the browser again.");
+ throw new Error("Playwright's Firefox is not installed on this machine. Ask the user before installing it; with their permission "
+ + "Insert {\"action\":\"setup\",\"part\":\"firefox\"} into this machine's address, then open the browser again.");
})
: await chromium.launchPersistentContext(directory, {
channel: browser, headless: false, chromiumSandbox: true, viewport: null, ignoreDefaultArgs: true,
diff --git a/src/windows-node.ts b/src/windows-node.ts
index d64c8ca..d21452d 100644
--- a/src/windows-node.ts
+++ b/src/windows-node.ts
@@ -1,14 +1,13 @@
-import { access, readFile } from "node:fs/promises";
+import { access } from "node:fs/promises";
import path from "node:path";
import { windowsLocalAppData } from "./native-runtime.ts";
+import { WINDOWS_NODE } from "./setup-pins.ts";
-const PIN = new URL("../scripts/windows-node.json", import.meta.url);
const INSTALLED_NODE = "/mnt/c/Program Files/nodejs/node.exe";
-/** Where `/computer-setup browser` puts the pinned portable Node under a Local Application Data folder (WSL path). */
-export async function portableNodePath(localAppData: string): Promise<string> {
- const { version } = JSON.parse(await readFile(PIN, "utf8")) as { version: string };
- return path.join(localAppData, "Programs", "PiComputerUse", `node-${version}-win-x64`, "node.exe");
+/** Where setup part browser puts the pinned portable Node under a Local Application Data folder (WSL path). */
+export function portableNodePath(localAppData: string): string {
+ return path.join(localAppData, "Programs", "PiComputerUse", `node-${WINDOWS_NODE.version}-win-x64`, "node.exe");
}
const exists = (file: string) => access(file).then(() => true, () => false);
@@ -18,8 +17,8 @@ const exists = (file: string) => access(file).then(() => true, () => false);
* Without either, refuse with the setup step instead of failing on a missing executable.
*/
export async function findWindowsNode(interop?: string, deps = { localAppData: windowsLocalAppData, exists }): Promise<string> {
- const candidates = [await portableNodePath(await deps.localAppData(interop)), INSTALLED_NODE];
+ const candidates = [portableNodePath(await deps.localAppData(interop)), INSTALLED_NODE];
for (const candidate of candidates) if (await deps.exists(candidate)) return candidate;
throw new Error("Browsers need Node.js on Windows, and none is installed here. Ask the user before installing it; "
- + "with their permission run /computer-setup browser (about 30 MB into their own folders), then open the browser again.");
+ + "with their permission Insert {\"action\":\"setup\",\"part\":\"browser\"} into this machine's address (about 30 MB into their own folders), then open the browser again.");
}
diff --git a/test/isolated-session.test.ts b/test/isolated-session.test.ts
index 747c2e1..51cfdeb 100644
--- a/test/isolated-session.test.ts
+++ b/test/isolated-session.test.ts
@@ -61,7 +61,7 @@ test("a session provisions, starts its driver through the reverse relay, and clo
test("a machine without MXC refuses before provisioning anything", async () => {
const f = await fakeHost({ installed: false });
- await expect(IsolatedSession.open(f.host, "/mnt/c/runtime/cua-driver.exe")).rejects.toThrow(/MXC is not installed.*Ask the user.*\/computer-setup mxc/);
+ await expect(IsolatedSession.open(f.host, "/mnt/c/runtime/cua-driver.exe")).rejects.toThrow(/MXC is not installed.*Ask the user.*"part":"mxc"/);
expect(f.phases).toEqual([]);
});
diff --git a/test/live/embedded-browsers.live.ts b/test/live/embedded-browsers.live.ts
index 4d4e4a9..84cc3e6 100644
--- a/test/live/embedded-browsers.live.ts
+++ b/test/live/embedded-browsers.live.ts
@@ -1,7 +1,6 @@
-import { execFileSync } from "node:child_process";
import { afterAll, expect, test } from "vitest";
import { connectLive, openWindow, serveFixture, text } from "./remote.ts";
-import { machineConfig } from "../../src/machine.ts";
+import { runSetup } from "../../src/setup.ts";
// LPT-857..859: Playwright attaches to browsers embedded in other apps. Everything here is started by the check in
// temporary folders on the machine: a portable VS Code (Electron) with its own data folder, the Pi Browser Bridge
@@ -29,7 +28,7 @@ async function portableVsCode() {
[ -f app/Code.exe ] || python3 -c "import zipfile;zipfile.ZipFile('code.zip').extractall('app')"
mkdir -p app/data; rm -f app/data/argv.json`], { timeoutMs: 600_000 });
// The same installer a user runs: the extension, and its proposed API allowed in that VS Code's argv.json.
- execFileSync("scripts/install-vscode-bridge.sh", [`${root}/app/bin/code.cmd`, `${root}/app/data/argv.json`, (await machineConfig(process.cwd(), "lab")).ssh], { stdio: "ignore" });
+ await runSetup({ part: "vscode-bridge", machine: "lab", code: `${root}/app/bin/code.cmd`, argv: `${root}/app/data/argv.json` }, process.cwd());
}
/** Read the attached page, then navigate it to the fixture and fill its field through Playwright. */
diff --git a/test/live/firefox.live.ts b/test/live/firefox.live.ts
index 279c74e..e07ed36 100644
--- a/test/live/firefox.live.ts
+++ b/test/live/firefox.live.ts
@@ -1,7 +1,7 @@
import { afterAll, expect, test } from "vitest";
import { connectLive, serveFixture, text } from "./remote.ts";
-// LPT-853: open_browser launches Playwright's own Firefox build (scripts/install-firefox.sh) with a disposable profile,
+// LPT-853: open_browser launches Playwright's own Firefox build (setup part firefox) with a disposable profile,
// binds its native window and serves the same DOM views and actions as Chrome and Edge; closing removes it.
const live = await connectLive();
const fixture = await serveFixture(live.runner);
diff --git a/test/remote-machine.test.ts b/test/remote-machine.test.ts
index 8b23c5a..69041c2 100644
--- a/test/remote-machine.test.ts
+++ b/test/remote-machine.test.ts
@@ -140,11 +140,11 @@ test("a remote browser refuses before launch when that machine's worker copy is
return output;
}) });
const current = runner(local);
- await checkRemoteWorker(current, "/pkg", "scripts/setup-remote-machine.sh lab /rt lab");
+ await checkRemoteWorker(current, "/pkg", "setup part machine");
expect(current.run).toHaveBeenCalledWith("sha256sum", ["/pkg/src/web-worker.ts", "/pkg/node_modules/playwright-core/package.json"], expect.anything());
const stale = local.replace(/^[0-9a-f]{64}/, "0".repeat(64));
- await expect(checkRemoteWorker(runner(stale), "/pkg", "scripts/setup-remote-machine.sh lab /rt lab"))
- .rejects.toThrow(/out of date \(src\/web-worker\.ts\); nothing was launched\. Refresh it from this package with scripts\/setup-remote-machine\.sh lab \/rt lab/);
+ await expect(checkRemoteWorker(runner(stale), "/pkg", "setup part machine"))
+ .rejects.toThrow(/out of date \(src\/web-worker\.ts\); nothing was launched\. Refresh it from this package with setup part machine/);
await expect(checkRemoteWorker(runner(new Error("No such file")), "/pkg", "setup"))
.rejects.toBeInstanceOf(ComputerRequestError);
});
\ No newline at end of file
diff --git a/test/setup-command.test.ts b/test/setup-command.test.ts
index a6a0038..1edc1f9 100644
--- a/test/setup-command.test.ts
+++ b/test/setup-command.test.ts
@@ -1,74 +1,141 @@
-import { execFile } from "node:child_process";
-import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
+import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
import path from "node:path";
-import { promisify } from "node:util";
import { afterEach, expect, test, vi } from "vitest";
import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
-import { SETUP_SCRIPT, registerSetupCommand, runSetup, splitArgs } from "../src/setup-command.ts";
+import { computerSource, parseComputerAddress } from "../src/computer-address.ts";
+import { ComputerRequestError, ComputerResources } from "../src/computer-resources.ts";
+import { WINDOWS_POWERSHELL, type CommandRunner } from "../src/runner.ts";
+import { parseSetupArgs, registerSetupCommand, splitArgs } from "../src/setup-command.ts";
+import { MXC } from "../src/setup-pins.ts";
+import { SetupError, allowProposedApi, runSetup, type SetupDeps } from "../src/setup.ts";
import { findWindowsNode } from "../src/windows-node.ts";
-const run = promisify(execFile);
const dirs: string[] = [];
afterEach(async () => { await Promise.all(dirs.splice(0).map(dir => rm(dir, { recursive: true, force: true }))); });
-async function temp(): Promise<string> {
+async function project(config?: unknown): Promise<string> {
await mkdir(".tmp", { recursive: true });
const dir = await mkdtemp(path.resolve(".tmp", "setup-"));
dirs.push(dir);
+ if (config) { await mkdir(path.join(dir, ".pi")); await writeFile(path.join(dir, ".pi", "computer-use.json"), JSON.stringify(config)); }
return dir;
}
-/** Run scripts/setup.sh from a project folder; these cases stop before any download, SSH or Windows call. */
-async function setup(cwd: string, ...args: string[]): Promise<{ code: number; out: string }> {
- try {
- const { stdout, stderr } = await run("bash", [SETUP_SCRIPT, ...args], { cwd, timeout: 20_000 });
- return { code: 0, out: stdout + stderr };
- } catch (error) {
- const failed = error as { code: number; stdout: string; stderr: string };
- return { code: failed.code, out: failed.stdout + failed.stderr };
- }
+/** A runner that records commands and answers PowerShell with the decoded script's last expected line. */
+function fakeRunner(answer: (script: string) => string = () => "") {
+ const calls: { file: string; args: readonly string[]; script?: string }[] = [];
+ const runner: CommandRunner = {
+ spawn: () => { throw new Error("no spawn in this test"); },
+ run: async (file, args, options) => {
+ // Setup sends its PowerShell script on stdin as base64 of UTF-8 behind a short fixed command.
+ const script = args.includes("-Command") && options?.input ? Buffer.from(String(options.input), "base64").toString("utf8") : undefined;
+ calls.push({ file, args, ...(script ? { script } : {}) });
+ return script ? answer(script) : "";
+ },
+ };
+ return { runner, calls };
}
-test("setup.sh lists its components and refuses unknown components and machines before doing anything", async () => {
- const cwd = await temp();
- await mkdir(path.join(cwd, ".pi"));
- await writeFile(path.join(cwd, ".pi", "computer-use.json"), JSON.stringify({ machines: { lab: { ssh: "lab-host", command: "/mnt/c/x/cua-driver.exe" } } }));
- const help = await setup(cwd);
- expect(help.code).toBe(0);
- for (const name of ["browser", "firefox", "perception", "mxc", "vscode-bridge", "machine"]) expect(help.out).toContain(name);
- expect(await setup(cwd, "flux")).toMatchObject({ code: 2, out: expect.stringContaining("Unknown component: flux") });
- expect(await setup(cwd, "mxc", "nowhere")).toMatchObject({ code: 1, out: expect.stringContaining("No machine named nowhere") });
- // A machine prepared without the browser worker names the step that adds it.
- expect(await setup(cwd, "firefox", "lab")).toMatchObject({ code: 1, out: expect.stringContaining('Machine lab has no "node"; prepare it with: setup.sh machine lab-host lab') });
- expect((await setup(cwd, "machine")).code).not.toBe(0);
+function deps(platform: NodeJS.Platform, runner: CommandRunner, wsl = true): Partial<SetupDeps> & { ssh: ReturnType<typeof vi.fn> } {
+ return { platform, wsl: async () => wsl, local: () => runner, ssh: vi.fn(() => runner) };
+}
+
+test("command words become setup requests; quoted paths stay whole and bad shapes show the usage", () => {
+ expect(splitArgs(` vscode-bridge "C:\\Tools\\VS Code\\bin\\code.cmd" 'C:\\a b\\argv.json'`))
+ .toEqual(["vscode-bridge", "C:\\Tools\\VS Code\\bin\\code.cmd", "C:\\a b\\argv.json"]);
+ expect(parseSetupArgs(["perception"])).toEqual({ part: "perception" });
+ expect(parseSetupArgs(["mxc", "lab"])).toEqual({ part: "mxc", machine: "lab" });
+ expect(parseSetupArgs(["machine", "lab-wsl", "lab"])).toEqual({ part: "machine", ssh: "lab-wsl", name: "lab" });
+ expect(parseSetupArgs(["vscode-bridge", "lab"])).toEqual({ part: "vscode-bridge", machine: "lab" });
+ expect(parseSetupArgs(["vscode-bridge", "c.cmd", "argv.json"])).toEqual({ part: "vscode-bridge", code: "c.cmd", argv: "argv.json" });
+ expect(parseSetupArgs(["vscode-bridge", "lab", "c.cmd", "argv.json"])).toEqual({ part: "vscode-bridge", machine: "lab", code: "c.cmd", argv: "argv.json" });
+ for (const bad of [[], ["flux"], ["machine"], ["mxc", "a", "b"]]) expect(parseSetupArgs(bad)).toBeUndefined();
+});
+
+test("argv.json gains the bridge in its proposed-API list without rewriting comments or other keys", () => {
+ const id = "alexshpunt.pi-browser-bridge";
+ expect(allowProposedApi("")).toBe(`{\n\t"enable-proposed-api": ["${id}"]\n}\n`);
+ expect(allowProposedApi(`{\n // keep\n "locale": "en"\n}`)).toBe(`{\n\t"enable-proposed-api": ["${id}"],\n // keep\n "locale": "en"\n}`);
+ expect(allowProposedApi(`{ "enable-proposed-api": [ /* none */ ] }`)).toBe(`{ "enable-proposed-api": ["${id}" /* none */ ] }`);
+ expect(allowProposedApi(`{ "enable-proposed-api": ["a.b"] }`)).toBe(`{ "enable-proposed-api": ["${id}", "a.b"] }`);
+ expect(allowProposedApi(`{ "enable-proposed-api": ["${id}"] }`)).toBeUndefined();
+});
+
+test("a part installs through PowerShell on the right Windows: interop from WSL, directly on Windows, SSH for a machine", async () => {
+ const cwd = await project({ interop: "/run/WSL/1_interop", machines: { lab: { ssh: "lab-wsl", command: "/mnt/c/rt/cua-driver.exe" } } });
+ const installed = (script: string) => script.includes("wxc-exec.exe") ? "C:\\Users\\u\\AppData\\Local\\PiComputerUse\\mxc\\wxc-exec.exe" : "";
+
+ const wsl = fakeRunner(installed);
+ const local = vi.fn(() => wsl.runner);
+ expect(await runSetup({ part: "mxc" }, cwd, { deps: { ...deps("linux", wsl.runner), local } }))
+ .toBe(`Installed MXC ${MXC.version} on this machine: C:\\Users\\u\\AppData\\Local\\PiComputerUse\\mxc\\wxc-exec.exe`);
+ expect(local).toHaveBeenCalledWith("/run/WSL/1_interop");
+ expect(wsl.calls[0].file).toBe(WINDOWS_POWERSHELL);
+ // The download is checked against its pin on that Windows before anything is unpacked.
+ expect(wsl.calls[0].script).toContain(`Get-Pinned '${MXC.url}' $zip '${MXC.sha256}'`);
+ expect(wsl.calls[0].script!.indexOf("Get-Pinned '")).toBeLessThan(wsl.calls[0].script!.indexOf("Expand-Zip $zip"));
+
+ const windows = fakeRunner(installed);
+ await runSetup({ part: "mxc" }, cwd, { deps: deps("win32", windows.runner) });
+ expect(windows.calls[0].file).toBe("powershell.exe");
+
+ const remote = fakeRunner(installed);
+ const viaSsh = deps("linux", remote.runner);
+ expect(await runSetup({ part: "mxc", machine: "lab" }, cwd, { deps: viaSsh })).toContain("on machine lab");
+ expect(viaSsh.ssh).toHaveBeenCalledWith("lab-wsl");
});
-test("the command passes quoted paths whole and reports the exit code with the cleaned output tail and full log", async () => {
- expect(splitArgs(` vscode-bridge local "/mnt/c/Tools/VS Code/bin/code.cmd" '/mnt/c/a b/argv.json'`))
- .toEqual(["vscode-bridge", "local", "/mnt/c/Tools/VS Code/bin/code.cmd", "/mnt/c/a b/argv.json"]);
- const dir = await temp();
- const script = path.join(dir, "fake-setup.sh");
- await writeFile(script, 'printf "args:%s|" "$@"; echo; printf "10%%\\r50%%\\r100%%\\n"; echo oops >&2; exit 3\n');
- const result = await runSetup(["perception", "lab"], dir, script);
- expect(result.code).toBe(3);
- expect(result.tail).toBe("args:perception|args:lab|\n100%\noops");
- expect(await readFile(result.log, "utf8")).toContain("10%\r50%\r100%");
+test("setup refuses before any command for an unknown machine, another OS or a machine without its browser worker", async () => {
+ const cwd = await project({ machines: { lab: { ssh: "lab-wsl", command: "/mnt/c/rt/cua-driver.exe" } } });
+ const { runner, calls } = fakeRunner();
+ const viaSsh = deps("linux", runner);
+ await expect(runSetup({ part: "mxc", machine: "nowhere" }, cwd, { deps: viaSsh })).rejects.toThrow(SetupError);
+ expect(viaSsh.ssh).not.toHaveBeenCalled();
+ await expect(runSetup({ part: "mxc" }, cwd, { deps: deps("linux", runner, false) })).rejects.toThrow(/Pi on Windows or in WSL/);
+ await expect(runSetup({ part: "firefox", machine: "lab" }, cwd, { deps: deps("linux", runner) })).rejects.toThrow(/no browser worker yet/);
+ await expect(runSetup({ part: "machine", ssh: "lab-wsl", name: "local" }, cwd, { deps: deps("linux", runner) })).rejects.toThrow(/not local/);
+ // Pi on Windows runs the Playwright worker with its own Node.
+ expect(await runSetup({ part: "browser" }, cwd, { deps: deps("win32", runner) })).toMatch(/Pi's own Node\. Nothing to install/);
+ expect(calls).toEqual([]);
+});
+
+test("the agent's setup Insert installs without starting the machine's driver and refuses malformed requests first", async () => {
+ const cwd = await project();
+ const getSession = vi.fn(), start = vi.fn();
+ const resources = new ComputerResources(getSession, async () => { throw new Error("unused"); }, undefined, start);
+ const insert = (ref: string, command: unknown) => resources.perform(parseComputerAddress(computerSource("machine", ref))!, cwd, JSON.stringify(command), "insert");
+ await expect(insert("local", { action: "setup", part: "flux" })).rejects.toThrow(/setup needs part.*explicitly allowed/);
+ await expect(insert("local", { action: "setup", part: "mxc", ssh: "x" })).rejects.toThrow(/belong to setup part machine/);
+ await expect(insert("lab", { action: "setup", part: "machine", ssh: "lab-wsl" })).rejects.toThrow(/from computer:machine\/local/);
+ await expect(insert("nowhere", { action: "setup", part: "mxc" })).rejects.toThrow(ComputerRequestError);
+ expect(getSession).not.toHaveBeenCalled();
+ expect(start).not.toHaveBeenCalled();
+});
+test("/computer-setup shows the usage, reports progress in the status line and the result or failure as a notification", async () => {
const commands = new Map<string, { getArgumentCompletions: (prefix: string) => unknown; handler: (args: string, ctx: unknown) => Promise<void> }>();
- registerSetupCommand({ registerCommand: (name: string, command: never) => commands.set(name, command) } as unknown as ExtensionAPI, script);
+ const run = vi.fn(async (_request: unknown, _cwd: string, options?: { progress?: (step: string) => void }) => { options?.progress?.("downloading"); return "Installed."; });
+ registerSetupCommand({ registerCommand: (name: string, command: never) => commands.set(name, command) } as unknown as ExtensionAPI, run as never);
const command = commands.get("computer-setup")!;
expect(command.getArgumentCompletions("pe")).toEqual([expect.objectContaining({ value: "perception" })]);
expect(command.getArgumentCompletions("perception l")).toBeNull();
const ui = { notify: vi.fn(), setStatus: vi.fn() };
- await command.handler("perception lab", { cwd: dir, ui });
- expect(ui.notify).toHaveBeenCalledWith(expect.stringMatching(/^Computer setup failed \(exit 3\): perception lab\n.*oops\nFull log: /s), "error");
+ await command.handler("", { cwd: "/p", ui });
+ expect(ui.notify).toHaveBeenLastCalledWith(expect.stringContaining("machine SSH_ALIAS [NAME]"), "info");
+ await command.handler("perception lab", { cwd: "/p", ui });
+ expect(run).toHaveBeenCalledWith({ part: "perception", machine: "lab" }, "/p", expect.anything());
+ expect(ui.setStatus).toHaveBeenCalledWith("computer-setup", "Computer setup: perception: downloading…");
+ expect(ui.notify).toHaveBeenLastCalledWith("Installed.", "info");
+ run.mockRejectedValueOnce(new Error("offline"));
+ await command.handler("mxc", { cwd: "/p", ui });
+ expect(ui.notify).toHaveBeenLastCalledWith("Computer setup: mxc failed: offline", "error");
expect(ui.setStatus).toHaveBeenLastCalledWith("computer-setup", undefined);
});
test("browsers use the portable Windows Node from setup first, an installed Node next, and otherwise name the setup step", async () => {
- const deps = (present: string[]) => ({ localAppData: async () => "/mnt/c/Users/u/AppData/Local", exists: async (file: string) => present.includes(file) });
+ const fake = (present: string[]) => ({ localAppData: async () => "/mnt/c/Users/u/AppData/Local", exists: async (file: string) => present.includes(file) });
const portable = "/mnt/c/Users/u/AppData/Local/Programs/PiComputerUse/node-v22.22.2-win-x64/node.exe";
const installed = "/mnt/c/Program Files/nodejs/node.exe";
- expect(await findWindowsNode(undefined, deps([portable, installed]))).toBe(portable);
- expect(await findWindowsNode(undefined, deps([installed]))).toBe(installed);
- await expect(findWindowsNode(undefined, deps([]))).rejects.toThrow(/Ask the user.*\/computer-setup browser/);
+ expect(await findWindowsNode(undefined, fake([portable, installed]))).toBe(portable);
+ expect(await findWindowsNode(undefined, fake([installed]))).toBe(installed);
+ await expect(findWindowsNode(undefined, fake([]))).rejects.toThrow(/Ask the user.*"part":"browser"/);
});
diff --git a/vscode-browser-bridge/README.md b/vscode-browser-bridge/README.md
index 6c514f5..b57ad3a 100644
--- a/vscode-browser-bridge/README.md
+++ b/vscode-browser-bridge/README.md
@@ -6,7 +6,7 @@ VS Code's proposed `browser` API gives each integrated browser tab a Chrome DevT
Setup:
-1. Install it with `/computer-setup vscode-bridge`, or build it with `scripts/package-vscode-bridge.sh` and install the `.vsix` with `code --install-extension`.
+1. Install it with `/computer-setup vscode-bridge`, or package this folder with `npx @vscode/vsce package` after `npm install --omit=dev` and install the `.vsix` with `code --install-extension`.
2. Run **Preferences: Configure Runtime Arguments** and add `"enable-proposed-api": ["alexshpunt.pi-browser-bridge"]`.
3. Quit VS Code fully and start it again; Reload Window does not reread that file.