Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.
TypeScript accessibilityagent-toolsai-agentbrowser-automationcomputer-usecuadesktop-automationelectronpipi-coding-agentpi-extensionpi-packageplaywrighttypescriptui-automationvisionwebview2windowswindows-automationwsl
Replace the local classifier with optional installs through /computer-setup (LPT-867)
alexshpunt-clanker[bot]2bbd376a1932
diff --git a/.gitignore b/.gitignoreindex 1adfac6..3cd0ec3 100644--- a/.gitignore+++ b/.gitignore@@ -2,6 +2,5 @@node_modules/native-runtime/.pi/computer-use.json-.pi/visual-classifier.json.pi/settings.json.pi/extensions/computer-use.ts\ No newline at end of filediff --git a/.pi/skills/computer-use/SKILL.md b/.pi/skills/computer-use/SKILL.mdindex 7e6595c..1223a98 100644--- a/.pi/skills/computer-use/SKILL.md+++ b/.pi/skills/computer-use/SKILL.md@@ -45,11 +45,17 @@ Read fresh pixels and UI after input and compare them with the expected effect.Prefer a deterministic check over comparing screenshots: add `"expect":{"conditions":[...]}` to an action, or Insert `wait_for`/`verify` into the window address, and branch on the verdict. Treat `unknown` as undecided, not as false. Use `"successor":true` for a compact change list instead of rereading everything. In Codemode, branch on the receipt object's `outcome` without try/catch. For a short known chain, use one `batch`; give later native steps a `target` instead of a hint so each is resolved in a fresh UI walk. Page long text with `offset`/`limit` until the footer says it ended, and use `element:text` for one long native element.-To react to something moving (a game, an animation), loop in Codemode: Read a small `image` overview (`maxLongEdge` 640), decide with `computer_classify` or pixel checks, and Insert one pixel action with `"animation":"reactive"` so the cursor does not glide first; each turn costs one capture and one input. Before the loop, set a deadline, frame, classification and action limits and a no-progress exit. Keep frames and classifier results inside the program and return only a small summary. Mark a clicked target pending and skip it until the app confirms it; a target can stay visible after it is done. To record a run for review, Insert `record_start`/`record_stop` into the session or machine address. For motion/progress, use a bounded sequence Read, inspect actual timestamps, skipped slots and stop reason, then explicitly Read selected frames. Sampling FPS is a goal. Partial/cancelled capture is not success. Frames are readonly; fresh window Read is required before input or detail.+To react to something moving (a game, an animation), loop in Codemode: Read a small `image` overview (`maxLongEdge` 640), decide with pixel checks, or with a classifier model through `models.classify` in Codemode when pixel checks cannot decide (each call is a paid model call), and Insert one pixel action with `"animation":"reactive"` so the cursor does not glide first; each turn costs one capture and one input. Before the loop, set a deadline, frame, classification and action limits and a no-progress exit. Keep frames and classifier results inside the program and return only a small summary. Mark a clicked target pending and skip it until the app confirms it; a target can stay visible after it is done. To record a run for review, Insert `record_start`/`record_stop` into the session or machine address. For motion/progress, use a bounded sequence Read, inspect actual timestamps, skipped slots and stop reason, then explicitly Read selected frames. Sampling FPS is a goal. Partial/cancelled capture is not success. Frames are readonly; fresh window Read is required before input or detail.Codemode may compose ordinary tools but is not required. Keep compact issued addresses/hints under short store aliases, not screenshots or whole results. Explicitly forward needed image blocks when composing; storage does not extend reference lifetime.-`computer_classify` stays separate and evaluates supplied pixels only. A classifier result does not renew observation/input authority. Bound reaction work and stop on unknown delivery; do not substitute a visual score for native identity or an application postcondition.+A classifier answer is a judgement about pixels: it does not renew observation/input authority. Bound reaction work and stop on unknown delivery; do not substitute it for native identity or an application postcondition.++## Missing parts++Some parts are optional installs: Windows Node for browsers (`browser`), Playwright's Firefox (`firefox`), Cua perception for `regions` (`perception`, about 420 MB, AGPL-3.0 icon detector), MXC for `open_session` (`mxc`), the VS Code bridge (`vscode-bridge`) and preparing another machine (`machine <ssh alias> [name]`). A refusal that names `/computer-setup` means one is missing.++Never install on your own. Tell the user what is missing, what it adds, its size and where it goes, and ask with ask_user. Ask before you start when the task clearly needs such a part, for example another machine or VS Code's browser. Only after explicit permission for that part on that machine, run [scripts/setup.sh](../../../scripts/setup.sh) by its absolute path with Bash from the project folder: `bash <absolute path>/setup.sh <part> [machine]`; without arguments it lists the parts. Retry the refused step once afterwards. `machine` prints a `machines` entry: show it, and add it to `.pi/computer-use.json` only with permission. A failed install is reported with its output, not retried in a loop.## Access and cleanupdiff --git a/README.md b/README.mdindex ceebc2f..a5f53d4 100644--- a/README.md+++ b/README.md@@ -106,10 +106,6 @@ The browser's toolbar, tabs and system dialogs are still ordinary windows, so thEach session has its own folder on the controlled machine for files an app opens or saves. In YOLO mode the agent can also read and write the clipboard, record the screen to video, and record what the machine plays, optionally with a transcript.-### Local image classifier--`computer_classify` checks one visual condition on an image with a small local model (TinyCLIP). It's for fast loops such as games, where asking the main model about every frame takes too long.-### Debug viewer and overlay`/computer-debug` opens a private viewer with the images the agent actually received, its action trace and the capture settings. On Windows an optional overlay briefly outlines the window and target the agent is working on, without taking focus.@@ -148,14 +144,32 @@ pi install npm:pi-agent-idePi packages run with your full system permissions, and this one drives your desktop. Read the code before you install it.-A few features need a one-time setup on the Windows side: Firefox for Playwright, Cua perception for `regions`, MXC for isolated sessions and the VS Code bridge. [The resource guide](docs/computer-resources.md) explains each one.+### Optional parts++Window control works right after the install. A few parts are extra downloads, and you install them with `/computer-setup`:++| Part | What it's for |+| --------------------- | ---------------------------------------------------------------------------------------------- |+| `browser` | Node.js for Windows, which Playwright browsers need. Skip it if Node is already on Windows. |+| `firefox` | Playwright's own Firefox build. Chrome and Edge work without it. |+| `perception` | The `regions` view. About 420 MB, and its icon detector is AGPL-3.0. |+| `mxc` | Isolated agent sessions. Needs Windows 11 24H2 or newer. |+| `vscode-bridge` | VS Code's integrated browser. |+| `machine <ssh alias>` | Another Windows machine with WSL, reached over SSH. |++Add a machine name to install a part there, for example `/computer-setup perception lab`. Everything goes into your Windows user's own folders, and every download is checked against a pinned SHA-256. If a task needs a part you don't have, the agent tells you what's missing and asks before it installs anything.++### Unsigned binaries++The Windows binaries aren't code-signed. Signing certificates are hard to get for a solo developer, and a fork of someone else's code doesn't qualify for the free open source programs. Started from WSL, they don't get a SmartScreen prompt, but antivirus, Smart App Control or company policy can still block them. Every file is checked against its pinned SHA-256 before it runs.## Commands-| Command | Effect |-| ----------------------------- | ----------------------------------------------------- |-| `/computer-mode safe\|yolo` | Switch between Safe and YOLO for this session |-| `/computer-debug on\|off` | Start or stop the private vision debug viewer |+| Command | Effect |+| -------------------------------- | -------------------------------------------------------- |+| `/computer-mode safe\|yolo` | Switch between Safe and YOLO for this session |+| `/computer-debug on\|off` | Start or stop the private vision debug viewer |+| `/computer-setup PART [MACHINE]` | Install an optional part; without a part it lists them |## Feedback and contributions@@ -174,7 +188,6 @@ I use this every day on real tasks, but I can only test the machines, apps and w| [Temporal capture](docs/temporal-capture.md) | Frame sequences and their timing || [Selected displays](docs/selected-displays.md) | Monitor discovery and display reads || [Image guides](docs/image-guides.md) | Coordinate grid and last-action marks |-| [Visual classifier](docs/visual-classifier.md) | `computer_classify` setup and use || [Vision debug viewer](docs/vision-debug.md) | What the agent saw and the capture controls || [Bundled runtime](docs/bundled-runtime.md) | Native runtime packages and platform boundaries |diff --git a/docs/bundled-runtime.md b/docs/bundled-runtime.mdindex 2c4c3b2..9a33753 100644--- a/docs/bundled-runtime.md+++ b/docs/bundled-runtime.md@@ -44,7 +44,7 @@ Computer use owns one MCP child. It disables telemetry and update checks, requir- The Windows binaries are unsigned. Shipping a UIAccess helper does not grant UIAccess rights or bypass Windows trust policy.- WSL needs a working Windows interop endpoint.-- The optional Windows Qt overlay and the classifier model environment are separate and not part of the runtime.+- The optional Windows Qt overlay is separate and not part of the runtime.## Overridesdiff --git a/docs/computer-resources.md b/docs/computer-resources.mdindex 17e2b84..dbd0c73 100644--- a/docs/computer-resources.md+++ b/docs/computer-resources.md@@ -5,7 +5,7 @@ Computer use works with or without Pi Agent IDE. Each Pi session picks the mode- With Pi Agent IDE loaded, in either order, use the ordinary IDE Read, Search, Write and Insert tools on `computer:` addresses.- Without IDE, or with an IDE whose plugin API version does not match, use the `computer_read`, `computer_search`, `computer_write` and `computer_insert` tools. They are deferred: find them with `tool_search`, and computer use turns `tool_search` on. Codemode scripts can call them directly.-Both modes take the same addresses, views, JSON commands and Codemode values. In this guide, Read, Search, Write and Insert mean either tool family. Without IDE, Read `docs:computer` with `computer_read` for this guide. `computer_classify` is available in both modes. The `/computer-mode` and `/computer-debug` commands remain user controls.+Both modes take the same addresses, views, JSON commands and Codemode values. In this guide, Read, Search, Write and Insert mean either tool family. Without IDE, Read `docs:computer` with `computer_read` for this guide. The `/computer-mode`, `/computer-debug` and `/computer-setup` commands remain user controls; an agent installs an optional part with `scripts/setup.sh` only after the user explicitly allows it.The current desktop scope is Windows controlled from WSL, on this machine or on another one over SSH. A different interface does not qualify a new runtime or platform. The production `windows-desktop` profile selects the same checked bytes as `windows-window-management`, so it qualifies wherever this guide names that profile or one it inherits. Safe permits session-opened windows and forbids display reads. YOLO permits task-authorized existing windows; it is not OS isolation.@@ -84,7 +84,9 @@ Detail retains the original local rectangle and native scale through movement/reWeb pages run through [Playwright](https://playwright.dev/). The browser window itself, its tabs and toolbar, system dialogs and every other application stay native Windows windows read and operated as above.-Insert `{"action":"open_browser"}` into a machine address to launch a new visible browser with a disposable profile. Optional `"browser":"msedge"` selects installed Edge instead of Chrome, `"browser":"firefox"` selects Playwright's own Firefox build, and `"url"` opens an HTTP(S) page. Firefox is not the user's installed Firefox: install the build once per machine with `scripts/install-firefox.sh local` or `scripts/install-firefox.sh <machine name>`, and again after updating this package. It uses the same DOM views and actions; it cannot be attached to. The result returns the browser's window address, owned by this session under Safe. DOM reads refuse in windows that are neither opened this way nor [attached](#attach-to-a-running-browser). Never replay an uncertain launch. Closing the computer session closes these browsers and deletes their profiles.+Playwright runs in a small worker on Windows, so `open_browser` and `attach_browser` need Node.js there: an installed Node, or a portable one from `/computer-setup browser` in the Windows user's `%LOCALAPPDATA%\Programs\PiComputerUse`. Without either, both refuse and name that command.++Insert `{"action":"open_browser"}` into a machine address to launch a new visible browser with a disposable profile. Optional `"browser":"msedge"` selects installed Edge instead of Chrome, `"browser":"firefox"` selects Playwright's own Firefox build, and `"url"` opens an HTTP(S) page. Firefox is not the user's installed Firefox: install the build once per machine with `/computer-setup firefox` (add a machine name for another machine), and again after updating this package. It uses the same DOM views and actions; it cannot be attached to. The result returns the browser's window address, owned by this session under Safe. DOM reads refuse in windows that are neither opened this way nor [attached](#attach-to-a-running-browser). Never replay an uncertain launch. Closing the computer session closes these browsers and deletes their profiles.For sites that need a login, add `"profile":"agent"` (YOLO only): the browser then uses the agent's own persistent profile on that machine (`%LOCALAPPDATA%\PiComputerUse\browser-profiles\<browser>`), so logins made there stay for later sessions. It is a separate profile, never the user's own; the user signs in to the needed sites once inside it. Only one browser can use it at a time, and closing the session keeps it.@@ -132,7 +134,7 @@ In YOLO, Playwright can also work in a Chromium-based browser that is already ru- **Debugging endpoint.** Any process of the window that listens for the Chrome DevTools Protocol is attached through it at once: a browser started with `--remote-debugging-port`, an Electron or CEF app started with it (for example `"args":["--remote-debugging-port=0"]` in `open`), or a WebView2 app with debugging turned on (below). Its pages, the app's own interface included, are tabs; pick one with `switch_tab`. Chrome and Edge ignore that switch for their default profile, so such a browser runs with its own `--user-data-dir`. In YOLO the agent can start one itself with `open` by `path` and `args`, for example `"args":["--user-data-dir=C:\\Temp\\agent-chrome","--remote-debugging-port=0","--disable-sync","https://example.com"]`; Edge needs `--disable-sync` so a fresh profile does not sign in to the Windows account.- **Remote debugging turned on in the running browser.** Chrome 144 and later can allow remote debugging in a running browser, even with its default profile: turn on "Allow remote debugging for this browser instance" at `chrome://inspect/#remote-debugging`. Chrome then writes `DevToolsActivePort` into its user data directory, and attaching connects to that endpoint. For every connection Chrome opens its own "Allow remote debugging?" window and waits up to 2 minutes; the user can click Allow, or the agent can find that window by its title and click Allow in another call while the attach waits. The setting stays on until it is turned off. Edge has the same switch at `edge://inspect/#remote-debugging` and the same "Allow remote debugging?" window. The first time a browser opens a debugging server, Windows may show a "Windows Security" firewall prompt for it; attaching uses `127.0.0.1` and works whichever way it is answered, but while the prompt is open Windows refuses foreground input to every other window.- **Playwright Extension.** Otherwise the browser needs Microsoft's [Playwright Extension](https://chromewebstore.google.com/detail/playwright-extension/mmlmfjhmonkocbjadbfplnigmagldckm), installed once by the user. Attaching opens the extension's connect page in that browser and waits up to 2 minutes until a tab is approved there with "Allow & select". The page is ordinary UI: the user can approve it, or the agent can read and click it in another call while the attach waits (for example two calls in one Codemode script). Approval exposes the whole browser, including signed-in sessions, to this session.-- **VS Code's integrated browser.** A VS Code window attaches to its integrated browser (Browser: Open Integrated Browser) through the Pi Browser Bridge extension in `vscode-browser-bridge/`, which relays the proposed `browser` API's per-tab DevTools channel on `127.0.0.1` behind a token. It uses the active integrated browser tab, or opens a blank one. Install it with `scripts/install-vscode-bridge.sh <code.cmd> <argv.json> [ssh alias]`, which also allows the proposed API in that VS Code's `argv.json`; quit VS Code fully and start it again. It works in local and remote (WSL) windows. The proposed API can change between VS Code releases.+- **VS Code's integrated browser.** A VS Code window attaches to its integrated browser (Browser: Open Integrated Browser) through the Pi Browser Bridge extension in `vscode-browser-bridge/`, which relays the proposed `browser` API's per-tab DevTools channel on `127.0.0.1` behind a token. It uses the active integrated browser tab, or opens a blank one. Install it with `/computer-setup vscode-bridge` (add a machine name for another machine), which finds an installed VS Code, installs the extension and allows the proposed API in that VS Code's `argv.json`; for a portable VS Code add its `bin/code.cmd` and `data/argv.json` paths. Quit VS Code fully and start it again. It works in local and remote (WSL) windows. The proposed API can change between VS Code releases.WebView2 apps take Chromium switches from their environment. Insert `"debugging":true` with `path` into `open` (YOLO) to start an app with WebView2 debugging. For an app the user starts, Insert `{"action":"webview2_debugging","app":"ms-teams.exe","enabled":true}` into its machine: this sets that exe's WebView2 policy under `HKCU\Software\Policies\Microsoft\Edge\WebView2\AdditionalBrowserArguments`, which needs administrator rights, so Windows shows a UAC prompt that the person at that machine must approve. It applies from the app's next start, whoever starts it, and stays until `"enabled":false`.@@ -191,7 +193,7 @@ Limits:- The session's desktop is 640×480, without a desktop compositor and with the classic window style.- The app list is the agent user's own. Some apps are missing by name, such as Notepad; start them by path, for example `{"action":"open","path":"C:\\Windows\\System32\\notepad.exe"}`.- No files are shared with the session yet.-- The machine needs Windows 11 24H2 or 25H2 build 9550 or later, and MXC's `wxc-exec.exe` in the Windows user's `%LOCALAPPDATA%\PiComputerUse\mxc`. Install it with `scripts/install-mxc.sh local` or `scripts/install-mxc.sh <ssh alias>`. Without them, `open_session` refuses with the reason.+- The machine needs Windows 11 24H2 or 25H2 build 9550 or later, and MXC's `wxc-exec.exe` in the Windows user's `%LOCALAPPDATA%\PiComputerUse\mxc`. Install it with `/computer-setup mxc` (add a machine name for another machine). Without them, `open_session` refuses with the reason.## Work on another machine@@ -215,12 +217,12 @@ Closing the computer session or Pi closes every machine, and a cancelled first u### Install perception for `regions`-The `regions` view needs Cua's signed perception extension on the machine whose windows it reads. Run `scripts/install-perception.sh local <driver exe>` for this machine or `scripts/install-perception.sh <ssh alias> <driver exe>` for another one, with that machine's WSL path to `cua-driver.exe`. It downloads the release with `gh`, checks its sums, and the driver verifies Cua's catalog signature and every hash before installing into the Windows user's `%USERPROFILE%\.cua-driver\extensions`. The bundle is about 420 MB; its icon detector (OmniParser v2) is AGPL-3.0. Running it again only re-runs the self-test.+The `regions` view needs Cua's signed perception extension on the machine whose windows it reads. Install it with `/computer-setup perception` (add a machine name for another machine). It downloads the release, checks its pinned SHA-256 sums, and the driver verifies Cua's catalog signature and every hash before installing into the Windows user's `%USERPROFILE%\.cua-driver\extensions`. The bundle is about 420 MB; its icon detector (OmniParser v2) is AGPL-3.0. Running it again only re-runs the self-test.### Set up a machine1. Make the machine reachable with an SSH alias from `~/.ssh/config` that logs in to its WSL by key, without a password prompt. Its host key must already be known.-2. Run `scripts/setup-remote-machine.sh <alias> <local runtime folder> [name]` from this repository after `npm ci`. It copies the CUA runtime and checks its hash, installs a checksum-verified portable Windows Node in the remote user's folder, copies the browser worker, and prints the `machines` entry. It changes nothing system-wide. Run it again after updating this package: a browser launch on that machine refuses, naming this command, while its worker copy differs from this package.+2. Run `/computer-setup machine <alias> [name]`. It copies the CUA runtime and checks its hash, installs a checksum-verified portable Windows Node in the remote user's folder, copies the browser worker, and prints the `machines` entry. It changes nothing system-wide. Run it again after updating this package: a browser launch on that machine refuses, naming this command, while its worker copy differs from this package.3. Add that entry to `machines` in `.pi/computer-use.json`:```json@@ -235,7 +237,7 @@ The `regions` view needs Cua's signed perception extension on the machine whose}```-All paths are on the remote WSL. Without `node` and `package`, the machine works without a browser. Run the script again after updating this package so the remote browser worker matches.+All paths are on the remote WSL. Without `node` and `package`, the machine works without a browser. Run `/computer-setup machine` again after updating this package so the remote browser worker matches.## Change a display's resolutiondiff --git a/docs/visual-classifier.md b/docs/visual-classifier.mddeleted file mode 100644index 8e29f24..0000000--- a/docs/visual-classifier.md+++ /dev/null@@ -1,78 +0,0 @@-# One visual condition--`computer_classify` evaluates supplied pixels against **one** text condition with local TinyCLIP-39M. It does not capture a screen, click, call another reasoning model, or ask for competing states.--The agent chooses the threshold for the wording and image domain. A raw cosine score is **not a probability**. This is a fast thresholded similarity check, not a general visual question-answering model.--## API--```js-// An image Read in Codemode returns an image block; without IDE use tools.computer_read.-const view = await tools.read({path: load("window"), views: ["image"]});-const result = await tools.computer_classify({- image: view,- condition: "a cartoon mole animal popping out of green grass",- threshold: 0.309,- uncertainty: 0.005-});-if (result.value === true) {- text("Condition met");-} else if (result.value === false) {- text("Condition not met");-} else {- text("Unknown: inspect or wait");-}-```--That threshold was checked on 86px saved mole tiles, **not on the whole window in the example and not on other wording or apps**. Choose and verify a cutoff for the actual input region; the example only shows image-block transfer and branching. Classification gives no input authority: a supplied image may be old, cropped or annotated. Follow the normal observation/action contract before acting. If precise classification matters, use suitable crops without distracting annotation.--- `image`: a saved local path (relative to the call's workspace) or an unchanged `{type:"image", data, mimeType}` block. PNG, JPEG or WebP; one frame, at most 10 MiB and 16 million pixels. No URL download or implicit capture. Selected sequence frame images also work.-- `region`: optional `{x,y,width,height}` rectangle in the supplied image’s encoded pixels. All four values are integers; x/y are nonnegative and width/height positive. It must lie fully inside the frame; there is no clipping, padding, resize or implicit capture. EXIF-rotated region input is refused. The warm worker crops in memory before encoding. The returned `image` hash/dimensions still identify the whole supplied frame; returned `region` identifies the evaluated part. Classification does not consume or renew an observation’s input authority.-- `regions`: alternative to `region`, an ordered array of 1..16 rectangles. One supplied image is decoded once and all crops share one image-encoder batch, one condition and one cutoff. Total crop area must not exceed 16 million pixels. Any invalid region rejects the entire request. Choose `region` or `regions`, never both.-- `condition`: one nonempty text description or question, up to 2048 characters and the model's token limit. Long text is rejected, never silently truncated. There is no opposite prompt.-- `threshold`: required finite number in [-1,1], chosen by the agent.-- `uncertainty`: optional nonnegative band half-width; defaults to 0. Both band edges must lie within [-1,1].--The rule is inspectable:--```text-score > threshold + uncertainty -> true-score < threshold - uncertainty -> false-inside the closed band -> null (unknown)-```--Boundary comparisons allow 1e-12 for floating-point rounding. Even with a zero-width band, an exact tie is unknown. Use `=== true`, `=== false` and `=== null`: do not collapse unknown into false.--The structured result contains `status:"classified"`, `value`, raw `score`, the supplied rule and condition, `model`, `cache_hit`, image SHA256/dimensions and `timing_ms` (full call, inference/preprocessing, model load). Errors throw and do not return false/null. Nothing in this result grants calibrated confidence.--With `regions`, the structured result has `results:[{region,value,score}, ...]` in input order, plus the common source image, rule, cache and timing metadata. It has no aggregate `value` or `score`. Each crop can independently be true, false or unknown. An empty batch is an error; skip the call when no eligible targets remain.--Use one batch request, not concurrent single-crop calls. The worker serializes requests; concurrency does not combine separate requests into a model batch. Exclude selected and pending targets before building the regions, and keep all intermediate results inside the reaction program.--## Local setup--The extension starts no model process until its first classification. Configure these trusted local paths in ignored `.pi/visual-classifier.json`:--```json-{- "python": "/path/to/your/qualified/python",- "model": "/path/to/the/qualified/TinyCLIP/snapshot"-}-```--Relative setup paths resolve from the workspace at first use. The worker is fixed repository code; requests cannot choose executables or process arguments. This setup is not an OS sandbox. Use a trusted Python environment.--The selected checkpoint is `wkcn/TinyCLIP-ViT-39M-16-Text-19M-YFCC15M`, revision `07a4b0bc751cb64fecd2b661c048c1dd98d69444`. Weight SHA256 is `112bdfb07ebc8994e9afba7a2a314e278493a675033595cf48e54cf8a17ae587`. The worker also verifies the exact config, image processor and tokenizer hashes pinned in its source. Extra processor/tokenizer configs are refused.--This installation reuses the already reviewed local Python environment: Python 3.14.4, Torch 2.11.0+cu128, Transformers 5.9.0 and Pillow 12.3.0, on the RTX 3060 Laptop GPU. CUDA is required for this qualified path. No CPU fallback, runtime installation, model download, remote code or paid service is enabled. Missing setup fails with guidance.--The publisher's pinned model card declares MIT and links Microsoft's MIT license. Keep attribution when redistributing; downloaded weights are not bundled with this extension.--## Warm runtime and limits--One owned offline Python worker keeps the model warm. It caches up to 128 exact condition strings using LRU eviction; thresholds are applied separately and do not change embeddings. Image preprocessing and encoding run per call; there is no image-result cache or score softmax.--Up to eight calls are serialized. A request has a 60-second execution timeout. Abort, timeout, closing `computer:session` and session shutdown stop the owned worker; no request is replayed. Queued requests cancelled by close do not restart it. Later calls may start a fresh worker. This does not close application windows or change the CUA runtime.--Wording, negation, scale and distractors can change similarity. Our tests found overlapping positive/negative scores for “Is a mole visible?” and for a wooden log. Raising the model size did not make arbitrary natural-language predicates reliable. Choose thresholds deliberately; test positives and hard negatives and retain unknown for borderline scores. Unknown is a **numerical band**, not a promise that all semantic uncertainty has been detected.-diff --git a/scripts/install-firefox.sh b/scripts/install-firefox.shindex 4140ba1..8d15957 100755--- a/scripts/install-firefox.sh+++ b/scripts/install-firefox.sh@@ -2,23 +2,16 @@# Install Playwright's own Firefox build for open_browser {"browser":"firefox"}. It is the build this package's# playwright-core version expects, downloaded by playwright-core itself into the Windows user's# %LOCALAPPDATA%\ms-playwright; nothing system-wide changes. Run it again after updating playwright-core.+# scripts/setup.sh firefox resolves the arguments.#-# Usage: scripts/install-firefox.sh local|MACHINE (MACHINE is a name under "machines" in .pi/computer-use.json)+# Usage: scripts/install-firefox.sh NODE_EXE PLAYWRIGHT_CLI [SSH_ALIAS]+# NODE_EXE and PLAYWRIGHT_CLI are WSL paths on the target machine: Windows node.exe and playwright-core's cli.js.set -euo pipefail-target="${1:?usage: install-firefox.sh local|MACHINE}"-package="$(cd "$(dirname "$0")/.." && pwd)"+node_exe="${1:?usage: install-firefox.sh NODE_EXE PLAYWRIGHT_CLI [SSH_ALIAS]}"+cli="${2:?usage: install-firefox.sh NODE_EXE PLAYWRIGHT_CLI [SSH_ALIAS]}"+alias="${3:-}"-if [[ "$target" == local ]]; then- cd /mnt/c && "/mnt/c/Program Files/nodejs/node.exe" "$(wslpath -w "$package/node_modules/playwright-core/cli.js")" install firefox- exit-fi--# The machine's Windows Node and the package copy that setup-remote-machine.sh installed there.-read -r ssh node remote < <(node -e '- const machine = JSON.parse(require("fs").readFileSync(process.argv[1], "utf8")).machines?.[process.argv[2]];- if (!machine?.node || !machine?.package) { console.error("No machine with node and package: " + process.argv[2]); process.exit(1); }- console.log(machine.ssh, machine.node, machine.package);' "$package/.pi/computer-use.json" "$target")-ssh -o BatchMode=yes "$ssh" bash -s -- "$node" "$remote" <<'EOF'-cd /mnt/c && "$1" "$(wslpath -w "$2/node_modules/playwright-core/cli.js")" install firefox-EOF+install='cd /mnt/c && "$1" "$(wslpath -w "$2")" install firefox'+if [[ -z "$alias" ]]; then bash -c "$install" _ "$node_exe" "$cli"+else ssh -o BatchMode=yes "$alias" bash -s -- "$node_exe" "$cli" <<<"$install"; fidiff --git a/scripts/install-mxc.sh b/scripts/install-mxc.shindex 5aedc18..f44dcca 100755--- a/scripts/install-mxc.sh+++ b/scripts/install-mxc.sh@@ -15,7 +15,7 @@ on_target() { if [[ "$target" == local ]]; then bash -s -- "$@"; else ssh -o Batdownload="$(mktemp -d)"trap 'rm -rf "$download"' EXIT-gh release download "$release" -R microsoft/mxc -D "$download" -p mxc-release-binaries.zip+curl -fsSL -o "$download/mxc-release-binaries.zip" "https://github.com/microsoft/mxc/releases/download/$release/mxc-release-binaries.zip"echo "$sha256 $download/mxc-release-binaries.zip" | sha256sum -c --quietpython3 -c "import sys, zipfile; zipfile.ZipFile(sys.argv[1]).extract('x64/wxc-exec.exe', sys.argv[2])" "$download/mxc-release-binaries.zip" "$download"diff --git a/scripts/install-node.sh b/scripts/install-node.shnew file mode 100755index 0000000..0f1be7f--- /dev/null+++ b/scripts/install-node.sh@@ -0,0 +1,32 @@+#!/usr/bin/env bash+# Install a portable Windows Node for the Playwright browser worker (open_browser and attach_browser) into the+# Windows user's %LOCALAPPDATA%\Programs\PiComputerUse. The version and its SHA-256 are pinned in windows-node.json;+# nothing system-wide changes. Prints the node.exe path as a WSL path.+#+# Usage: scripts/install-node.sh local|SSH_ALIAS+set -euo pipefail++target="${1:?usage: install-node.sh local|SSH_ALIAS}"+here="$(cd "$(dirname "$0")" && pwd)"+read -r version sha256 < <(node -e 'const p = require(process.argv[1]); console.log(p.version, p.sha256)' "$here/windows-node.json")+zip="node-$version-win-x64.zip"++# Run a bash script from stdin on the target machine's WSL; the remote login shell may be fish.+on_target() { if [[ "$target" == local ]]; then bash -s -- "$@"; else ssh -o BatchMode=yes "$target" bash -s -- "$@"; fi; }++programs="$(on_target <<'EOF'+cd /mnt/c && wslpath "$(/mnt/c/Windows/System32/cmd.exe /c 'echo %LOCALAPPDATA%' | tr -d '\r')"+EOF+)/Programs/PiComputerUse"+node_exe="$programs/node-$version-win-x64/node.exe"+if on_target "$node_exe" <<<'[[ -f "$1" ]]'; then echo "$node_exe"; exit; fi++download="$(mktemp -d)"+trap 'rm -rf "$download"' EXIT+curl -fsSL -o "$download/$zip" "https://nodejs.org/dist/$version/$zip"+echo "$sha256 $download/$zip" | sha256sum -c --quiet+python3 -c "import sys, zipfile; zipfile.ZipFile(sys.argv[1]).extractall(sys.argv[2])" "$download/$zip" "$download/out"+on_target "$programs" <<<'mkdir -p "$1"'+if [[ "$target" == local ]]; then cp -r "$download/out/node-$version-win-x64" "$programs/"+else rsync -rt "$download/out/node-$version-win-x64" "$target:$programs/"; fi+echo "$node_exe"diff --git a/scripts/install-perception.sh b/scripts/install-perception.shindex 0c144d9..2ecc7f7 100755--- a/scripts/install-perception.sh+++ b/scripts/install-perception.sh@@ -1,23 +1,35 @@#!/usr/bin/env bash# Install Cua's signed perception extension (OmniParser v2 icons, PP-OCRv5 text) for the `regions` Read view.-# The driver verifies Cua's catalog signature and every hash, and installs into the Windows user's-# %USERPROFILE%\.cua-driver\extensions. Nothing is bundled in this package.+# The release files are checked against pinned SHA-256 sums; the driver then verifies Cua's catalog signature and+# every hash, and installs into the Windows user's %USERPROFILE%\.cua-driver\extensions. About 420 MB; the icon+# detector is AGPL-3.0. Nothing is bundled in this package. scripts/setup.sh perception resolves the arguments.## Usage: scripts/install-perception.sh local|SSH_ALIAS DRIVER_EXE-# DRIVER_EXE is that machine's WSL path to cua-driver.exe (the configured `command`).+# DRIVER_EXE is that machine's WSL path to cua-driver.exe.set -euo pipefailtarget="${1:?usage: install-perception.sh local|SSH_ALIAS DRIVER_EXE}"driver="${2:?usage: install-perception.sh local|SSH_ALIAS DRIVER_EXE}"+release="https://github.com/trycua/cua/releases/download/cua-perception-v0.2.1"name="cua-perception-0.2.1-x86_64-pc-windows-msvc"# Run a bash script from stdin on the target machine's WSL; the remote login shell may be fish.on_target() { if [[ "$target" == local ]]; then bash -s -- "$@"; else ssh -o BatchMode=yes "$target" bash -s -- "$@"; fi; }+# An installed extension only re-runs its self-test; nothing is downloaded again.+if on_target "$driver" <<<'cd /mnt/c && "$1" extension status perception | grep -q "Active version: 0.2.1"'; then+ on_target "$driver" <<<'cd /mnt/c && "$1" extension status perception --self-test'+ exit+fi+download="$(mktemp -d)"trap 'rm -rf "$download"' EXIT-gh release download cua-perception-v0.2.1 -R trycua/cua -D "$download" -p "$name.catalog.json" -p "$name.tar.gz" -p SHA256SUMS-(cd "$download" && grep -E " $name\.(catalog\.json|tar\.gz)\$" SHA256SUMS | sha256sum -c --quiet)+curl -fsSL -o "$download/$name.catalog.json" "$release/$name.catalog.json"+curl -fSL --progress-bar -o "$download/$name.tar.gz" "$release/$name.tar.gz"+(cd "$download" && sha256sum -c --quiet) <<EOF+bfb2300ff9da054d522e73351f7b337ba6157ebbe30f029a95a471fc3d13a042 $name.catalog.json+6709579aa3f938e8200c0a51b1abe450d4fda51b09e97517e0cb4c272b5bff37 $name.tar.gz+EOF# The catalog names its archive beside it, in the target's Windows temp folder.temp="$(on_target <<'EOF'@@ -32,7 +44,6 @@ on_target "$driver" "$temp" "$name" <<'EOF'set -euo pipefailtrap 'cd / && rm -rf "$2"' EXITcd "$2"-"$1" extension status perception | grep -q "Active version: 0.2.1" ||- "$1" extension install perception --catalog "$(wslpath -w "$2/$3.catalog.json")"+"$1" extension install perception --catalog "$(wslpath -w "$2/$3.catalog.json")""$1" extension status perception --self-testEOFdiff --git a/scripts/install-vscode-bridge.sh b/scripts/install-vscode-bridge.shindex 09ab423..8c337e3 100755--- a/scripts/install-vscode-bridge.sh+++ b/scripts/install-vscode-bridge.sh@@ -2,24 +2,40 @@# Build and install the Pi Browser Bridge into a Windows VS Code, and allow its proposed browser API in that VS Code's# argv.json. VS Code must then be quit fully and started again; Reload Window does not reread argv.json.#-# Usage: scripts/install-vscode-bridge.sh <code.cmd as a WSL path> <argv.json as a WSL path> [ssh alias]-# Installed VS Code: ".../Microsoft VS Code/bin/code.cmd" "/mnt/c/Users/<user>/.vscode/argv.json"-# Portable VS Code: ".../VSCode/bin/code.cmd" ".../VSCode/data/argv.json"+# Usage: scripts/install-vscode-bridge.sh local|SSH_ALIAS [CODE_CMD ARGV_JSON]+# Without paths, an installed VS Code is found on the target: its bin/code.cmd and %USERPROFILE%\.vscode\argv.json.+# For a portable VS Code, pass both as WSL paths on the target: ".../VSCode/bin/code.cmd" ".../VSCode/data/argv.json".set -euo pipefail-code="${1:?usage: install-vscode-bridge.sh <code.cmd> <argv.json> [ssh alias]}"-argv="${2:?usage: install-vscode-bridge.sh <code.cmd> <argv.json> [ssh alias]}"-alias="${3:-}"+target="${1:?usage: install-vscode-bridge.sh local|SSH_ALIAS [CODE_CMD ARGV_JSON]}"+code="${2:-}"+argv="${3:-}"root="$(cd "$(dirname "$0")/.." && pwd)"-"$root/scripts/package-vscode-bridge.sh" "$root/.tmp/pi-browser-bridge.vsix" >/dev/null+build="$(mktemp -d)"+trap 'rm -rf "$build"' EXIT+bash "$root/scripts/package-vscode-bridge.sh" "$build/pi-browser-bridge.vsix" >/dev/null# Run a bash script from stdin on the target machine's WSL; the remote login shell may be fish.-on_target() { if [[ -z "$alias" ]]; then bash -s -- "$@"; else ssh -o BatchMode=yes "$alias" bash -s -- "$@"; fi; }+on_target() { if [[ "$target" == local ]]; then bash -s -- "$@"; else ssh -o BatchMode=yes "$target" bash -s -- "$@"; fi; }++if [[ -z "$code" || -z "$argv" ]]; then+ read -r -d '' found < <(on_target <<'EOF'+cd /mnt/c+win() { wslpath "$(/mnt/c/Windows/System32/cmd.exe /c "echo %$1%" | tr -d '\r')"; }+for code in "$(win LOCALAPPDATA)/Programs/Microsoft VS Code/bin/code.cmd" "/mnt/c/Program Files/Microsoft VS Code/bin/code.cmd"; do+ [[ -f "$code" ]] && { printf '%s\n%s\n' "$code" "$(win USERPROFILE)/.vscode/argv.json"; exit; }+done+EOF+ ) || true+ code="$(sed -n 1p <<<"$found")"; argv="$(sed -n 2p <<<"$found")"+ [[ -n "$code" ]] || { echo "No installed VS Code found on $target. For a portable one, pass its bin/code.cmd and data/argv.json." >&2; exit 1; }+fi+vsix="$(on_target <<'EOF'cd /mnt/c && wslpath "$(/mnt/c/Windows/System32/cmd.exe /c 'echo %TEMP%' | tr -d '\r')"EOF)/pi-browser-bridge.vsix"-if [[ -z "$alias" ]]; then cp "$root/.tmp/pi-browser-bridge.vsix" "$vsix"; else scp -q "$root/.tmp/pi-browser-bridge.vsix" "$alias:$vsix"; fi+if [[ "$target" == local ]]; then cp "$build/pi-browser-bridge.vsix" "$vsix"; else scp -q "$build/pi-browser-bridge.vsix" "$target:$vsix"; fion_target "$code" "$vsix" "$argv" <<'EOF'set -euo pipefaildiff --git a/scripts/setup-remote-machine.sh b/scripts/setup-remote-machine.shindex a6a86b2..734d036 100755--- a/scripts/setup-remote-machine.sh+++ b/scripts/setup-remote-machine.sh@@ -2,27 +2,29 @@# Prepare another machine (Windows with WSL, reached over SSH) for computer use from this machine:# copy the CUA runtime, install a portable Windows Node for the browser worker, copy the browser worker,# and print the `machines` entry for .pi/computer-use.json. Installs only into the remote user's folders.+# scripts/setup.sh machine resolves the arguments.#-# Usage: scripts/setup-remote-machine.sh SSH_ALIAS RUNTIME_DIR [NAME]-# RUNTIME_DIR is a local CUA runtime folder containing runtime/cua-driver.exe (for example the configured one).+# Usage: scripts/setup-remote-machine.sh SSH_ALIAS RUNTIME_DIR PLAYWRIGHT_DIR [NAME]+# RUNTIME_DIR is a local CUA runtime folder containing runtime/cua-driver.exe; PLAYWRIGHT_DIR is playwright-core.set -euo pipefail-alias="${1:?usage: setup-remote-machine.sh SSH_ALIAS RUNTIME_DIR [NAME]}"-runtime_dir="${2:?usage: setup-remote-machine.sh SSH_ALIAS RUNTIME_DIR [NAME]}"-name="${3:-$alias}"-node_version="v22.22.2"-package="$(cd "$(dirname "$0")/.." && pwd)"+usage="usage: setup-remote-machine.sh SSH_ALIAS RUNTIME_DIR PLAYWRIGHT_DIR [NAME]"+alias="${1:?$usage}"+runtime_dir="${2:?$usage}"+playwright="${3:?$usage}"+name="${4:-$alias}"+here="$(cd "$(dirname "$0")" && pwd)"+package="$(dirname "$here")"[[ -f "$runtime_dir/runtime/cua-driver.exe" ]] || { echo "No runtime/cua-driver.exe in $runtime_dir" >&2; exit 1; }-[[ -d "$package/node_modules/playwright-core" ]] || { echo "Install dependencies first (npm ci)." >&2; exit 1; }+[[ -f "$playwright/cli.js" ]] || { echo "No playwright-core in $playwright" >&2; exit 1; }ssh=(ssh -o BatchMode=yes -o ConnectTimeout=10 "$alias")# The remote login shell may be fish: run every remote script through bash.remote_home="$("${ssh[@]}" bash -s <<<'printf %s "$HOME"')"-windows_user="$("${ssh[@]}" bash -s <<'REMOTE'-/mnt/c/Windows/System32/WindowsPowerShell/v1.0/powershell.exe -NoProfile -Command '$env:USERNAME' </dev/null | tr -d '\r'+programs="$("${ssh[@]}" bash -s <<'REMOTE'+cd /mnt/c && wslpath "$(/mnt/c/Windows/System32/cmd.exe /c 'echo %LOCALAPPDATA%' | tr -d '\r')"REMOTE-)"-programs="/mnt/c/Users/$windows_user/AppData/Local/Programs/PiComputerUse"+)/Programs/PiComputerUse"runtime_name="$(basename "$runtime_dir")"remote_package="$remote_home/.local/share/pi-computer-use/package"@@ -32,19 +34,11 @@ local_hash="$(sha256sum "$runtime_dir/runtime/cua-driver.exe" | cut -d' ' -f1)"remote_hash="$("${ssh[@]}" sha256sum "$programs/$runtime_name/runtime/cua-driver.exe" | cut -d' ' -f1)"[[ "$local_hash" == "$remote_hash" ]] || { echo "Runtime copy does not match ($remote_hash)." >&2; exit 1; }-"${ssh[@]}" bash -s -- "$programs" "$node_version" <<'REMOTE'-set -euo pipefail-programs="$1"; version="$2"; zip="node-$version-win-x64.zip"-[[ -x "$programs/node-$version-win-x64/node.exe" ]] && exit 0-cd "$(mktemp -d)"-curl -fsSL -o "$zip" "https://nodejs.org/dist/$version/$zip"-curl -fsSL "https://nodejs.org/dist/$version/SHASUMS256.txt" | grep " $zip\$" | sha256sum -c --quiet-python3 -c "import sys, zipfile; zipfile.ZipFile(sys.argv[1]).extractall(sys.argv[2])" "$zip" "$programs"-REMOTE+node_exe="$(bash "$here/install-node.sh" "$alias")"# The browser worker needs only its own source and playwright-core.rsync -rt --delete -e "ssh -o BatchMode=yes" "$package/src/" "$alias:$remote_package/src/"-rsync -rt --delete -e "ssh -o BatchMode=yes" "$package/node_modules/playwright-core/" "$alias:$remote_package/node_modules/playwright-core/"+rsync -rt --delete -e "ssh -o BatchMode=yes" "$playwright/" "$alias:$remote_package/node_modules/playwright-core/"rsync -t -e "ssh -o BatchMode=yes" "$package/package.json" "$alias:$remote_package/package.json"cat <<JSON@@ -53,7 +47,7 @@ Add to "machines" in .pi/computer-use.json:"ssh": "$alias","runtime": "windows-desktop","command": "$programs/$runtime_name/runtime/cua-driver.exe",- "node": "$programs/node-$node_version-win-x64/node.exe",+ "node": "$node_exe","package": "$remote_package"}JSONdiff --git a/scripts/setup.sh b/scripts/setup.shnew file mode 100755index 0000000..a098a2b--- /dev/null+++ b/scripts/setup.sh@@ -0,0 +1,102 @@+#!/usr/bin/env bash+# One entry point for the optional parts of computer use. /computer-setup runs it; an agent runs it only after the+# user explicitly allowed that install. Run it from the Pi project folder: MACHINE is a name under "machines" in that+# folder's .pi/computer-use.json, and "local" (the default) is this machine.+set -euo pipefail++here="$(cd "$(dirname "$0")" && pwd)"+package="$(dirname "$here")"++usage() {+ cat <<'EOF'+Usage: setup.sh PART [local|MACHINE]++browser+ Windows Node for open_browser and attach_browser (about 30 MB)+firefox+ Playwright's Firefox for "browser":"firefox" (needs browser)+perception+ Cua perception for the regions view (about 420 MB, AGPL-3.0)+mxc+ Microsoft MXC for open_session (Windows 11 24H2 or later)+vscode-bridge [local|MACHINE] [CODE_CMD ARGV_JSON]+ Pi Browser Bridge for VS Code's integrated browser+machine SSH_ALIAS [NAME]+ Prepare another Windows machine with WSL over SSH++Everything goes into the Windows user's own folders.+EOF+}++# One field of a configured machine; refuse unknown names before any SSH work.+machine_field() {+ node -e '+ const [file, name, field] = process.argv.slice(1);+ let config = {};+ try { config = JSON.parse(require("fs").readFileSync(file, "utf8")); } catch {}+ const machine = config.machines?.[name];+ if (!machine) { console.error(`No machine named ${name} in ${file}.`); process.exit(1); }+ if (!machine[field]) { console.error(`Machine ${name} has no "${field}"; prepare it with: setup.sh machine ${machine.ssh ?? "SSH_ALIAS"} ${name}`); process.exit(1); }+ console.log(machine[field]);' "$PWD/.pi/computer-use.json" "$1" "$2"+}++local_appdata() { (cd /mnt/c && wslpath "$(/mnt/c/Windows/System32/cmd.exe /c 'echo %LOCALAPPDATA%' | tr -d '\r')"); }++# The Windows node.exe the browser worker uses here: the portable one from `browser`, else an installed Node.+local_node() {+ local version portable+ version="$(node -p 'require(process.argv[1]).version' "$here/windows-node.json")"+ portable="$(local_appdata)/Programs/PiComputerUse/node-$version-win-x64/node.exe"+ for node_exe in "$portable" "/mnt/c/Program Files/nodejs/node.exe"; do+ [[ -f "$node_exe" ]] && { echo "$node_exe"; return; }+ done+ echo "No Windows Node on this machine; run: setup.sh browser" >&2+ return 1+}++# The driver this machine runs: CUA_DRIVER_PATH, the project's configured command, or the packaged runtime's cache.+local_driver() {+ local configured+ configured="${CUA_DRIVER_PATH:-$(node -e '+ try { const c = JSON.parse(require("fs").readFileSync(process.argv[1], "utf8")).command; if (c) console.log(c); } catch {}' "$PWD/.pi/computer-use.json")}"+ if [[ -n "$configured" ]]; then (cd "$PWD" && realpath -m "$configured"); return; fi+ local source driver+ source="$(node -p 'require(process.argv[1]).source' "$package/src/runtime-pins.json")"+ driver="$(local_appdata)/PiComputerUse/native-runtimes/$source/win32-x64/runtime/cua-driver.exe"+ [[ -f "$driver" ]] || { echo "Computer use has not prepared its runtime yet. Use it once (for example, search for windows), then run this again." >&2; return 1; }+ echo "$driver"+}++playwright_dir() { (cd "$package" && dirname "$(node -p 'require.resolve("playwright-core/package.json")')"); }++component="${1:-}"+shift || true+case "$component" in+ ""|help|-h|--help) usage; exit 0 ;;+ machine)+ alias="${1:?usage: setup.sh machine SSH_ALIAS [NAME]}"+ runtime="$package/native-runtime/win32-x64"+ [[ -f "$runtime/runtime/cua-driver.exe" ]] || { echo "The packaged runtime is not downloaded yet. Use computer use once on this machine, then run this again." >&2; exit 1; }+ playwright="$(playwright_dir)"+ exec bash "$here/setup-remote-machine.sh" "$alias" "$runtime" "$playwright" "${2:-$alias}" ;;+ browser|firefox|perception|mxc|vscode-bridge) ;;+ *) echo "Unknown component: $component" >&2; usage >&2; exit 2 ;;+esac++target="${1:-local}"+shift || true+ssh=""+[[ "$target" == local ]] || ssh="$(machine_field "$target" ssh)"++case "$component" in+ browser) node_exe="$(bash "$here/install-node.sh" "${ssh:-local}")"; echo "Windows Node ready: $node_exe" ;;+ firefox)+ if [[ -z "$ssh" ]]; then node_exe="$(local_node)"; cli="$(playwright_dir)/cli.js"+ else node_exe="$(machine_field "$target" node)"; cli="$(machine_field "$target" package)/node_modules/playwright-core/cli.js"; fi+ exec bash "$here/install-firefox.sh" "$node_exe" "$cli" $ssh ;;+ perception)+ if [[ -z "$ssh" ]]; then driver="$(local_driver)"; else driver="$(machine_field "$target" command)"; fi+ exec bash "$here/install-perception.sh" "${ssh:-local}" "$driver" ;;+ mxc) exec bash "$here/install-mxc.sh" "${ssh:-local}" ;;+ vscode-bridge) exec bash "$here/install-vscode-bridge.sh" "${ssh:-local}" "$@" ;;+esacdiff --git a/scripts/visual-classifier-worker.py b/scripts/visual-classifier-worker.pydeleted file mode 100644index acc3e9b..0000000--- a/scripts/visual-classifier-worker.py+++ /dev/null@@ -1,132 +0,0 @@-"""Owned offline TinyCLIP-39M worker. One text condition; no competing labels or capture."""-import base64-from collections import OrderedDict-import hashlib-import io-import json-import os-from pathlib import Path-import sys-import time--# stdout is a bounded JSON-lines protocol, not a library progress stream.-protocol = sys.stdout-sys.stdout = sys.stderr-os.environ["HF_HUB_OFFLINE"] = "1"-os.environ["TRANSFORMERS_OFFLINE"] = "1"-PINNED = {- "model.safetensors": "112bdfb07ebc8994e9afba7a2a314e278493a675033595cf48e54cf8a17ae587",- "config.json": "de47e9e12875c7abb6e0ddd13b5b0f54851b96cefa5c2ed3aab881e3efd1647b",- "preprocessor_config.json": "910e70b3956ac9879ebc90b22fb3bc8a75b6a0677814500101a4c072bd7857bd",- "tokenizer.json": "b556ac8c99757ffb677208af34bc8c6721572114111a6e0aaf5fa69ff0b8d842",-}-model = processor = torch = None-cache = OrderedDict()---def load():- """Load only the reviewed local artifact. Never fetch weights or execute model code."""- global model, processor, torch- root = Path(sys.argv[1])- for name, expected in PINNED.items():- with (root / name).open("rb") as source:- if hashlib.file_digest(source, "sha256").hexdigest() != expected:- raise ValueError("Local TinyCLIP artifact does not match the qualified pin.")- # Avoid an extra processor/tokenizer config silently changing the pinned tokenizer.- extra = {"tokenizer_config.json", "special_tokens_map.json", "processor_config.json"}- if any((root / name).exists() for name in extra):- raise ValueError("Use the qualified TinyCLIP snapshot without extra processor configs.")- import torch as torch_module- from transformers import AutoModel, AutoProcessor- torch = torch_module- torch.set_num_threads(4)- if not torch.cuda.is_available():- raise ValueError("The qualified classifier requires the local CUDA runtime.")- loaded = AutoModel.from_pretrained(root, local_files_only=True, trust_remote_code=False,- use_safetensors=True, attn_implementation="sdpa").eval().to("cuda")- processed = AutoProcessor.from_pretrained(root, local_files_only=True, trust_remote_code=False)- model, processor = loaded, processed---def classify(request):- """Return raw normalized cosine and cache/timing evidence, not a probability."""- start = time.perf_counter()- load_ms = 0- if model is None:- load()- load_ms = (time.perf_counter() - start) * 1000- condition = request["condition"]- if not isinstance(condition, str) or not condition.strip() or len(condition) > 2048:- raise ValueError("Provide one nonempty visual condition.")- from PIL import Image, ImageOps- Image.MAX_IMAGE_PIXELS = 16_000_000- data = base64.b64decode(request["data"], validate=True)- if not 0 < len(data) <= 10 * 1024 * 1024:- raise ValueError("Image byte bound exceeded.")- with Image.open(io.BytesIO(data)) as source:- if source.width * source.height > 16_000_000 or getattr(source, "n_frames", 1) != 1:- raise ValueError("Use one bounded image frame.")- batch = "regions" in request- if batch and "region" in request:- raise ValueError("Choose region or regions, not both.")- regions = request.get("regions") if batch else [request["region"]] if "region" in request else []- if not isinstance(regions, list) or (batch and not 1 <= len(regions) <= 16):- raise ValueError("Use 1..16 image regions.")- images = []- area = 0- for region in regions:- if not isinstance(region, dict) or set(region) != {"x", "y", "width", "height"}:- raise ValueError("Invalid image region.")- x, y, width, height = (region[key] for key in ("x", "y", "width", "height"))- if any(type(value) is not int for value in (x, y, width, height)) or \- x < 0 or y < 0 or width < 1 or height < 1 or \- x + width > source.width or y + height > source.height:- raise ValueError("Region must lie fully inside the supplied image.")- if source.getexif().get(274, 1) != 1:- raise ValueError("Region input must have no EXIF rotation.")- area += width * height- if area > 16_000_000:- raise ValueError("Total crop area exceeds 16 million pixels.")- images.append(source.crop((x, y, x + width, y + height)).convert("RGB"))- if not regions:- images = [ImageOps.exif_transpose(source).convert("RGB")]- start = time.perf_counter()- hit = condition in cache- with torch.inference_mode():- if hit:- text = cache[condition]- cache.move_to_end(condition)- else:- tokens = processor(text=[condition], padding=True, truncation=False, return_tensors="pt")- if tokens["input_ids"].shape[-1] > model.config.text_config.max_position_embeddings:- raise ValueError("Condition exceeds the model token limit; it was not truncated.")- text = model.get_text_features(**tokens.to("cuda"), return_dict=True).pooler_output.float()- text /= text.norm(dim=-1, keepdim=True)- cache[condition] = text- if len(cache) > 128:- cache.popitem(last=False)- pixels = processor(images=images, return_tensors="pt")["pixel_values"].to("cuda")- with torch.autocast("cuda", dtype=torch.float16):- features = model.get_image_features(pixel_values=pixels, return_dict=True).pooler_output- features = features.float()- features /= features.norm(dim=-1, keepdim=True)- scores = (features @ text.T).squeeze(-1).tolist()- return {**({"scores": scores} if batch else {"score": scores[0]}),- "cache_hit": hit, "load_ms": load_ms,- "inference_ms": (time.perf_counter() - start) * 1000}---while True:- line = sys.stdin.readline(14 * 1024 * 1024 + 1)- if not line:- break- request = {}- try:- if len(line) > 14 * 1024 * 1024 or not line.endswith("\n"):- raise ValueError("Worker input bound exceeded.")- request = json.loads(line)- result = classify(request)- except Exception as error:- result = {"error": str(error)[:512]}- protocol.write(json.dumps({"id": request.get("id"), **result}, allow_nan=False) + "\n")- protocol.flush()diff --git a/scripts/windows-node.json b/scripts/windows-node.jsonnew file mode 100644index 0000000..4a31a14--- /dev/null+++ b/scripts/windows-node.json@@ -0,0 +1,4 @@+{+ "version": "v22.22.2",+ "sha256": "7c93e9d92bf68c07182b471aa187e35ee6cd08ef0f24ab060dfff605fcc1c57c"+}diff --git a/src/classifier-tool.ts b/src/classifier-tool.tsdeleted file mode 100644index f87bfde..0000000--- a/src/classifier-tool.ts+++ /dev/null@@ -1,58 +0,0 @@-import { Type } from "@earendil-works/pi-ai";-import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";-import { VisualClassifier, loadClassifierConfig } from "./visual-classifier.ts";--/** Register a read-only predicate tool. The returned close function owns only its model worker. */-export function registerClassifierTool(pi: ExtensionAPI,- factory: (cwd: string) => Promise<VisualClassifier> = async cwd => new VisualClassifier(await loadClassifierConfig(cwd))): () => Promise<void> {- let loading: Promise<VisualClassifier> | undefined;- let closing: Promise<void> | undefined;- const region = Type.Object({- x: Type.Integer({ minimum: 0 }), y: Type.Integer({ minimum: 0 }),- width: Type.Integer({ minimum: 1 }), height: Type.Integer({ minimum: 1 }),- }, { additionalProperties: false });- const common = {- status: Type.Literal("classified"),- threshold: Type.Number(), uncertainty: Type.Number(), condition: Type.String(),- model: Type.Literal("TinyCLIP-39M"), cache_hit: Type.Boolean(),- image: Type.Object({ sha256: Type.String(), width: Type.Number(), height: Type.Number() }),- timing_ms: Type.Object({ total: Type.Number(), inference: Type.Number(), load: Type.Number() }),- };- const decision = { value: Type.Union([Type.Boolean(), Type.Null()]), score: Type.Number() };- const schema = Type.Union([- Type.Object({ ...common, ...decision, region: Type.Optional(region) }),- Type.Object({ ...common, results: Type.Array(Type.Object({ ...decision, region }), { minItems: 1, maxItems: 16 }) }),- ]);- pi.registerTool({- name: "computer_classify", label: "Visual condition",- description: "Evaluate ONE visual condition on supplied pixels with local TinyCLIP-39M; no capture, input, paid call, opposite description or state list. image is a saved path or an unchanged image block from a computer Read. Optional region is a positive integer rectangle in that supplied image, cropped in memory without another capture; outside bounds and EXIF-rotated region input reject. Alternatively, regions contains 1..16 rectangles evaluated as ONE image-encoder batch with the same condition and cutoff; results are ordered independent value/score/region records, not an aggregate boolean. Choose region or regions, never both. Total crop area is at most 16 million pixels. Classification does not consume or renew observation authority. condition is one text description or question. The agent MUST choose threshold for that wording/domain; cosine is NOT a probability and free-form questions/negation may be unreliable. value is true above threshold+uncertainty, false below threshold-uncertainty, null (unknown) inside the closed band; uncertainty defaults to 0 and an exact tie stays unknown. In Codemode branch with value === true/false; do not treat null as false. Errors throw, never become detections. Model stays warm and caches up to 128 condition embeddings. Closing computer:session or session shutdown releases it. Setup: docs/visual-classifier.md.",- parameters: Type.Object({- image: Type.Union([Type.String({ minLength: 1 }), Type.Object({ type: Type.Literal("image"), data: Type.String({ minLength: 1, maxLength: 13981016 }),- mimeType: Type.Union([Type.Literal("image/png"), Type.Literal("image/jpeg"), Type.Literal("image/webp")]) }, { additionalProperties: false })]),- region: Type.Optional(region),- regions: Type.Optional(Type.Array(region, { minItems: 1, maxItems: 16 })),- condition: Type.String({ minLength: 1, maxLength: 2048 }),- threshold: Type.Number({ minimum: -1, maximum: 1 }),- uncertainty: Type.Optional(Type.Number({ minimum: 0, maximum: 1 })),- }, { additionalProperties: false }),- outputSchema: schema,- annotations: { readOnlyHint: true, destructiveHint: false, openWorldHint: false },- async execute(_id, params, signal, _update, ctx) {- if (closing) throw Error("Classifier is closing.");- if (!loading) loading = factory(ctx.cwd).catch(error => { loading = undefined; throw error; });- const classifier = await loading;- const result = params.regions !== undefined- ? await classifier.classify({ ...params, regions: params.regions }, signal, ctx.cwd)- : await classifier.classify(params, signal, ctx.cwd);- const summary = "results" in result- ? `Visual batch: ${result.results.length} regions`- : `Visual condition: ${result.value === null ? "unknown" : result.value}`;- return { content: [{ type: "text", text: `${summary}\n${JSON.stringify(result)}` }],- details: result, structuredContent: JSON.parse(JSON.stringify(result)) };- },- });- return async () => {- closing ??= (async () => { await (await loading)?.close(); loading = undefined; })();- try { await closing; } finally { closing = undefined; }- };-}diff --git a/src/index.ts b/src/index.tsindex 723be88..300b7ea 100644--- a/src/index.ts+++ b/src/index.ts@@ -1,4 +1,3 @@-import { registerClassifierTool } from "./classifier-tool.ts";import { registerVisionDebug } from "./vision-debug.ts";import { CuaImageSettings } from "./cua-image-settings.ts";import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";@@ -6,6 +5,7 @@ import { loadComputerConfig } from "./config.ts";import { ReactiveAnimationDriver, loadReactiveAnimationConfig } from "./reactive-animation.ts";import { McpDriver } from "./driver.ts";import { registerComputerMode } from "./mode.ts";+import { registerSetupCommand } from "./setup-command.ts";import { WindowHighlights } from "./window-highlights.ts";import { ComputerSession, type ComputerResult } from "./session.ts";import { registerComputerProvider } from "./provider.ts";@@ -17,15 +17,15 @@ import { machineFactory, sandboxFactory } from "./machine.ts";/*** Register native computer resources with ordinary IDE tools when Pi Agent IDE accepts them, otherwise as- * deferred computer_* tools, and retain the separate visual classifier.+ * deferred computer_* tools.*/export default async function computerUse(pi: ExtensionAPI): Promise<void> {- const closeClassifier = registerClassifierTool(pi);let overviewKernel: "lanczos3" | "bilinear" = "lanczos3";const images = new CuaImageSettings(() => overviewKernel);const debug = registerVisionDebug(pi, images);const highlights = new WindowHighlights();const getMode = registerComputerMode(pi, () => highlights.clear());+ registerSetupCommand(pi);let session: ComputerSession | undefined;let loading: Promise<ComputerSession> | undefined;async function getSession(cwd: string): Promise<ComputerSession> {@@ -44,7 +44,6 @@ export default async function computerUse(pi: ExtensionAPI): Promise<void> {const openSandbox = sandboxFactory({ settings: () => images.settings(), retainRaw: image => debug.trace.retainOriginal(image) });const connectMachine = machineFactory({ getMode, settings: () => images.settings(), retainRaw: image => debug.trace.retainOriginal(image) });const provider = await registerComputerProvider(pi, getSession, async () => {- await closeClassifier();const result: ComputerResult = session ? await session.close() : {content: [{ type: "text", text: "Not started." }],details: { status: "closed", mode: getMode() },@@ -63,7 +62,6 @@ export default async function computerUse(pi: ExtensionAPI): Promise<void> {registerStandaloneTools(pi, provider.resources, provider.guide);});pi.on("session_shutdown", async () => {- await closeClassifier();if (loading) await loading.catch(() => undefined);try { await session?.close(); } finally { await highlights.close(); }session = undefined;diff --git a/src/isolated-session.ts b/src/isolated-session.tsindex 411f0fc..27ce7f5 100644--- a/src/isolated-session.ts+++ b/src/isolated-session.ts@@ -23,7 +23,7 @@ const sleep = (ms: number, signal?: AbortSignal) => new Promise<void>((resolve,* An isolated Windows agent session from Microsoft Execution Containers (MXC `isolation_session`): a fresh agent user* with its own session, desktop, clipboard and input on the same machine, removed again on close. Several can run at* once, unlike Windows Sandbox. A CUA driver runs inside; the host reaches it through a reverse relay because the- * session can dial the host's loopback but not the other way round. Needs `wxc-exec.exe` from scripts/install-mxc.sh.+ * session can dial the host's loopback but not the other way round. Needs `wxc-exec.exe` from /computer-setup mxc.*/export class IsolatedSession {readonly ref = "is-" + randomUUID().slice(0, 8);@@ -66,7 +66,7 @@ export class IsolatedSession {const local = (await this.ps("$env:LOCALAPPDATA", signal)).trim();this.wxc = `${local}\\PiComputerUse\\mxc\\wxc-exec.exe`;if ((await this.ps(`Test-Path -LiteralPath ${psString(this.wxc)}`, signal)).trim() !== "True")- throw new ComputerRequestError(`MXC is not installed on this machine (${this.wxc}). Run scripts/install-mxc.sh for it, then open the session again.`);+ throw new ComputerRequestError(`MXC is not installed on this machine (${this.wxc}). Ask the user before installing it; with their permission run /computer-setup mxc (add the machine name for another machine), then open the session again.`);const windows = `${local}\\PiComputerUse\\sessions\\${this.ref}`;this.workdir = { windows, unix: await this.host.path(`${local}\\PiComputerUse\\sessions`, "unix").then(parent => `${parent}/${this.ref}`) };await this.host.mkdir(this.workdir.unix);diff --git a/src/machine.ts b/src/machine.tsindex 249b887..b408a34 100644--- a/src/machine.ts+++ b/src/machine.ts@@ -1,7 +1,6 @@import { createHash } from "node:crypto";import { readFile } from "node:fs/promises";import { createRequire } from "node:module";-import path from "node:path";import { fileURLToPath } from "node:url";import { loadComputerConfig, type ComputerConfig } from "./config.ts";import { ComputerRequestError, machineGuide, type MachineFactory, type SandboxFactory } from "./computer-resources.ts";@@ -78,9 +77,7 @@ export function sandboxFactory(options: { settings: () => CuaVisionSettings; retexport function machineFactory(options: { getMode: () => ComputerMode; settings: () => CuaVisionSettings; retainRaw?: (image: Content) => void }): MachineFactory {return async (cwd, name, signal) => {const machine = await machineConfig(cwd, name);- const local = (await loadComputerConfig(cwd)).command;- const runtimeFolder = local?.endsWith("/runtime/cua-driver.exe") ? path.dirname(path.dirname(local)) : "<local runtime folder>";- const refresh = `scripts/setup-remote-machine.sh ${machine.ssh} ${runtimeFolder} ${name}`;+ const refresh = `/computer-setup machine ${machine.ssh} ${name}`;const runner = sshRunner(machine.ssh);const driver = new ReactiveAnimationDriver(new McpDriver({ command: machine.command, ...(machine.runtime ? { runtime: machine.runtime } : {}), ssh: machine.ssh }),await loadReactiveAnimationConfig(cwd));diff --git a/src/native-runtime.ts b/src/native-runtime.tsindex bb18d79..1e6a2a4 100644--- a/src/native-runtime.ts+++ b/src/native-runtime.ts@@ -140,8 +140,8 @@ export async function cacheWindowsRuntime(runtime: BundledRuntime, base: string,} finally { await rm(staging, { recursive: true, force: true }); }}-/** Read the Windows user's cache location through a bounded system query; do not guess a user or install location. */-async function windowsCacheDirectory(interop?: string): Promise<string> {+/** The Windows user's Local Application Data folder as a WSL path, through a bounded system query; never guessed. */+export async function windowsLocalAppData(interop?: string): Promise<string> {const run = promisify(execFile);const { stdout } = await run("/mnt/c/Windows/System32/WindowsPowerShell/v1.0/powershell.exe", ["-NoLogo", "-NoProfile", "-NonInteractive", "-Command",@@ -151,7 +151,11 @@ async function windowsCacheDirectory(interop?: string): Promise<string> {if (!/^[A-Za-z]:\\/.test(local) || /[\r\n]/.test(local)) throw new Error("native_runtime_windows_cache_unavailable");const mapped = (await run("wslpath", ["-u", local], { timeout: 5_000, maxBuffer: 16_384 })).stdout.trim();if (!path.isAbsolute(mapped)) throw new Error("native_runtime_windows_cache_unavailable");- return path.join(mapped, "PiComputerUse", "native-runtimes");+ return mapped;+}++async function windowsCacheDirectory(interop?: string): Promise<string> {+ return path.join(await windowsLocalAppData(interop), "PiComputerUse", "native-runtimes");}/**diff --git a/src/session.ts b/src/session.tsindex 93927a1..c2eb66a 100644--- a/src/session.ts+++ b/src/session.ts@@ -1333,7 +1333,9 @@ export class ComputerSession {const observed = this.entries.get(ref)?.observation;if (observed?.id !== id || !observed.captureId) throw new Error("This observation's native capture is no longer current. Read the window with regions again.");const native = await this.driver.call("parse_visual_regions", { capture_id: observed.captureId, options: { max_regions: 300 } }, signal);- if (native.isError) throw new Error(native.content.filter(c => c.type === "text").map(c => c.text).join(" ") || "parse_visual_regions failed");+ if (native.isError) throw new Error((native.content.filter(c => c.type === "text").map(c => c.text).join(" ") || "parse_visual_regions failed")+ + " The regions view needs the Cua perception extension on this machine; if it is missing, ask the user before installing it "+ + "(about 420 MB) and with their permission run /computer-setup perception (add the machine name for another machine).");const parsed = native.structuredContent as { capture: { screenshot: { width: number } }; regions: {kind: "icon" | "text"; bounds: { x: number; y: number; width: number; height: number }; text?: string | null; label?: string | null; confidence: number }[] };const scale = (observed.width ?? parsed.capture.screenshot.width) / parsed.capture.screenshot.width;diff --git a/src/setup-command.ts b/src/setup-command.tsnew file mode 100644index 0000000..3285b79--- /dev/null+++ b/src/setup-command.ts@@ -0,0 +1,72 @@+import { spawn } from "node:child_process";+import { createWriteStream } from "node:fs";+import { mkdtemp } from "node:fs/promises";+import { tmpdir } from "node:os";+import path from "node:path";+import { fileURLToPath } from "node:url";+import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";++export const SETUP_SCRIPT = fileURLToPath(new URL("../scripts/setup.sh", import.meta.url));++/** Optional parts that scripts/setup.sh installs, for argument completion. */+export const SETUP_COMPONENTS: { name: string; description: string }[] = [+ { name: "browser", description: "Portable Windows Node for open_browser and attach_browser" },+ { name: "firefox", description: "Playwright's Firefox for open_browser" },+ { name: "perception", description: "Cua perception for the regions view (about 420 MB)" },+ { name: "mxc", description: "Microsoft MXC for isolated agent sessions" },+ { name: "vscode-bridge", description: "Pi Browser Bridge for VS Code's integrated browser" },+ { name: "machine", description: "Prepare another Windows machine over SSH" },+];++/** Split command arguments on spaces, keeping single- or double-quoted paths whole. */+export function splitArgs(text: string): string[] {+ const args: string[] = [];+ for (const match of text.matchAll(/"([^"]*)"|'([^']*)'|(\S+)/g)) args.push(match[1] ?? match[2] ?? match[3]);+ return args;+}++/** Run the setup script; the whole output goes to a log file, the tail comes back for the notification. */+export async function runSetup(args: string[], cwd: string, script = SETUP_SCRIPT): Promise<{ code: number; tail: string; log: string }> {+ const log = path.join(await mkdtemp(path.join(tmpdir(), "pi-computer-setup-")), "setup.log");+ const file = createWriteStream(log);+ let tail = "";+ const code = await new Promise<number>((resolve, reject) => {+ const child = spawn("bash", [script, ...args], { cwd, stdio: ["ignore", "pipe", "pipe"] });+ const take = (chunk: Buffer) => { file.write(chunk); tail = (tail + chunk.toString("utf8")).slice(-4000); };+ child.stdout.on("data", take);+ child.stderr.on("data", take);+ child.on("error", reject);+ child.on("close", status => resolve(status ?? 1));+ });+ await new Promise(resolve => file.end(resolve));+ // Progress bars redraw with carriage returns; keep only the last state of each line.+ const lines = tail.split("\n").map(line => line.split("\r").at(-1)!.trimEnd()).filter(Boolean);+ return { code, tail: lines.slice(-15).join("\n"), log };+}++/** /computer-setup COMPONENT [local|MACHINE]: the user's own entry point to the optional installs. */+export function registerSetupCommand(pi: ExtensionAPI, script = SETUP_SCRIPT): void {+ pi.registerCommand("computer-setup", {+ description: "Install an optional computer-use part: browser, firefox, perception, mxc, vscode-bridge or machine",+ getArgumentCompletions: prefix => {+ if (/\s/.test(prefix.trimStart())) return null;+ const items = SETUP_COMPONENTS.filter(item => item.name.startsWith(prefix.trim()))+ .map(item => ({ value: item.name, label: item.name, description: item.description }));+ return items.length ? items : null;+ },+ handler: async (text, ctx) => {+ const args = splitArgs(text);+ if (!args.length) {+ const { tail } = await runSetup([], ctx.cwd, script);+ ctx.ui.notify(tail, "info");+ return;+ }+ ctx.ui.setStatus("computer-setup", `Computer setup: ${args[0]}…`);+ try {+ const { code, tail, log } = await runSetup(args, ctx.cwd, script);+ ctx.ui.notify(`${code === 0 ? "Computer setup finished" : `Computer setup failed (exit ${code})`}: ${args.join(" ")}\n${tail}\nFull log: ${log}`,+ code === 0 ? "info" : "error");+ } finally { ctx.ui.setStatus("computer-setup", undefined); }+ },+ });+}diff --git a/src/visual-classifier.ts b/src/visual-classifier.tsdeleted file mode 100644index 3f41773..0000000--- a/src/visual-classifier.ts+++ /dev/null@@ -1,226 +0,0 @@-import { spawn, type ChildProcessWithoutNullStreams } from "node:child_process";-import { createHash } from "node:crypto";-import { open, readFile } from "node:fs/promises";-import path from "node:path";-import { fileURLToPath } from "node:url";-import sharp from "sharp";--const MAX_BYTES = 10 * 1024 * 1024;-const MAX_PIXELS = 16_000_000;-const workerPath = fileURLToPath(new URL("../scripts/visual-classifier-worker.py", import.meta.url));--/** Trusted local installation paths; never supplied by a classifier tool request. */-export interface ClassifierConfig { python: string; model: string }-/** Integer rectangle in the supplied image's encoded pixel coordinates; never clipped or rescaled. */-export interface ClassifierRegion { x: number; y: number; width: number; height: number }-/** A saved image or an unchanged image block from a computer-use result. No URL fetch or capture. */-export interface PredicateRequest {- image: string | { type: "image"; data: string; mimeType: string };- region?: ClassifierRegion;- condition: string;- threshold: number;- uncertainty?: number;-}-/** One image and one condition evaluated over 1..16 ordered regions in one model batch. */-export interface BatchPredicateRequest extends PredicateRequest { regions: ClassifierRegion[] }-/** null means a score in the caller's uncertainty band, not a runtime failure. */-export interface PredicateResult {- status: "classified";- value: boolean | null;- score: number;- threshold: number;- uncertainty: number;- condition: string;- model: "TinyCLIP-39M";- cache_hit: boolean;- /** Identity of the supplied frame, not a separately encoded crop. */- image: { sha256: string; width: number; height: number };- region?: ClassifierRegion;- timing_ms: { total: number; inference: number; load: number };-}--/** Each result corresponds to the region at the same input index; there is no aggregate boolean. */-export interface BatchPredicateResult extends Omit<PredicateResult, "value" | "score" | "region"> {- results: { region: ClassifierRegion; value: boolean | null; score: number }[];-}-/** Compare a raw cosine score with the agent's cutoff. The closed band is unknown. */-export function decide(score: number, threshold: number, uncertainty = 0): boolean | null {- if (![score, threshold, uncertainty].every(Number.isFinite) || Math.abs(score) > 1 || Math.abs(threshold) > 1 ||- uncertainty < 0 || threshold - uncertainty < -1 || threshold + uncertainty > 1)- throw Error("Use finite cosine scores/cutoffs in [-1,1] and an uncertainty band inside that range.");- const delta = score - threshold;- return Math.abs(delta) <= uncertainty + 1e-12 ? null : delta > 0;-}--/** Load explicit user-controlled setup. Missing setup never downloads or silently selects a model. */-export async function loadClassifierConfig(cwd: string): Promise<ClassifierConfig> {- let value: unknown;- try { value = JSON.parse(await readFile(path.join(cwd, ".pi/visual-classifier.json"), "utf8")); }- catch { throw Error("Configure local python and model paths in .pi/visual-classifier.json; see docs/visual-classifier.md."); }- const config = value as Partial<ClassifierConfig>;- if (!config || typeof config !== "object" || Array.isArray(config) || Object.keys(config).some(key => !["python", "model"].includes(key)) ||- typeof config.python !== "string" || !config.python.trim() || typeof config.model !== "string" || !config.model.trim())- throw Error("Classifier setup must contain only nonempty python and model paths.");- return { python: path.resolve(cwd, config.python), model: path.resolve(cwd, config.model) };-}--async function pixels(input: PredicateRequest["image"], cwd: string, regions: ClassifierRegion[] = []) {- let bytes: Buffer;- if (typeof input === "string") {- if (!input.trim()) throw Error("Choose a saved image path.");- const file = await open(path.resolve(cwd, input), "r");- try {- const stat = await file.stat();- if (!stat.isFile() || stat.size > MAX_BYTES) throw Error("Image must be a regular file of at most 10 MiB.");- bytes = Buffer.alloc(stat.size);- const { bytesRead } = await file.read(bytes, 0, bytes.length, 0);- if (bytesRead !== bytes.length) throw Error("Image changed while being read.");- } finally { await file.close(); }- } else {- if (!input || input.type !== "image" || !["image/png", "image/jpeg", "image/webp"].includes(input.mimeType) ||- typeof input.data !== "string" || input.data.length > Math.ceil(MAX_BYTES / 3) * 4 ||- !/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/.test(input.data))- throw Error("Use a PNG/JPEG/WebP base64 image block of at most 10 MiB.");- bytes = Buffer.from(input.data, "base64");- if (bytes.toString("base64") !== input.data) throw Error("Image base64 is not canonical.");- }- if (!bytes.length || bytes.length > MAX_BYTES) throw Error("Image must contain at most 10 MiB.");- const metadata = await sharp(bytes, { limitInputPixels: MAX_PIXELS }).metadata();- if (!metadata.width || !metadata.height || !["png", "jpeg", "webp"].includes(metadata.format ?? "") || (metadata.pages ?? 1) !== 1)- throw Error("Use one PNG, JPEG or WebP frame.");- let area = 0;- for (const region of regions) {- if (!region || typeof region !== "object" || Array.isArray(region) ||- Object.keys(region).some(key => !["x", "y", "width", "height"].includes(key)) ||- ![region.x, region.y, region.width, region.height].every(Number.isSafeInteger) ||- region.x < 0 || region.y < 0 || region.width < 1 || region.height < 1 ||- region.x + region.width > metadata.width || region.y + region.height > metadata.height)- throw Error("Use a positive integer region fully inside the supplied image; regions are not clipped.");- if (metadata.orientation && metadata.orientation !== 1)- throw Error("Region input must have no EXIF rotation; use an already oriented image.");- area += region.width * region.height;- if (area > MAX_PIXELS) throw Error("Total crop area must not exceed 16 million pixels.");- }- return { bytes, image: { sha256: createHash("sha256").update(bytes).digest("hex"), width: metadata.width, height: metadata.height } };-}--/** One lazy owned worker, bounded serialized requests, and no capture or desktop input. */-export class VisualClassifier {- private child?: ChildProcessWithoutNullStreams;- private tail: Promise<unknown> = Promise.resolve();- private queued = 0;- private nextId = 0;- private generation = 0;- private closing = false;- private pending?: { id: number; resolve: (value: Record<string, unknown>) => void; reject: (error: Error) => void };- constructor(private readonly config: ClassifierConfig, private readonly worker = workerPath) {}-- /** Evaluate one condition. Aborts/errors reject rather than becoming negative detections. */- async classify(request: BatchPredicateRequest, signal?: AbortSignal, cwd?: string): Promise<BatchPredicateResult>;- async classify(request: PredicateRequest, signal?: AbortSignal, cwd?: string): Promise<PredicateResult>;- async classify(request: PredicateRequest & { regions?: ClassifierRegion[] }, signal?: AbortSignal, cwd = process.cwd()): Promise<PredicateResult | BatchPredicateResult> {- if (this.closing) throw Error("Classifier is closing.");- if (this.queued >= 8) throw Error("Classifier queue is full (8 requests).");- const generation = this.generation;- const start = performance.now();- this.queued++;- const run = this.tail.catch(() => undefined).then(async () => {- signal?.throwIfAborted();- if (generation !== this.generation) throw Error("Classifier request was closed before execution.");- if (typeof request.condition !== "string" || !request.condition.trim() || request.condition.length > 2048)- throw Error("Provide one nonempty condition of at most 2048 characters.");- const uncertainty = request.uncertainty ?? 0;- decide(0, request.threshold, uncertainty);- if (request.region !== undefined && request.regions !== undefined)- throw Error("Choose region or regions, not both.");- if (request.regions !== undefined && (!Array.isArray(request.regions) || !request.regions.length || request.regions.length > 16))- throw Error("Use 1..16 ordered image regions.");- const requested = request.regions ?? (request.region !== undefined ? [request.region] : []);- const source = await pixels(request.image, cwd, requested); signal?.throwIfAborted();- const crops = requested.map(region => ({ x: region.x, y: region.y, width: region.width, height: region.height }));- const batch = request.regions !== undefined;- if (generation !== this.generation) throw Error("Classifier request was closed before execution.");- const response = await this.exchange({ condition: request.condition, data: source.bytes.toString("base64"),- ...(batch ? { regions: crops } : crops.length ? { region: crops[0] } : {}) }, signal);- const scores = batch ? response.scores : [response.score];- if (!Array.isArray(scores) || scores.length !== (batch ? crops.length : 1) || scores.some(score => typeof score !== "number") ||- typeof response.cache_hit !== "boolean" ||- typeof response.inference_ms !== "number" || !Number.isFinite(response.inference_ms) || response.inference_ms < 0 ||- typeof response.load_ms !== "number" || !Number.isFinite(response.load_ms) || response.load_ms < 0)- throw Error("Invalid classifier worker receipt.");- const decisions = scores.map(score => decide(score, request.threshold, uncertainty));- const common = { status: "classified" as const,- threshold: request.threshold, uncertainty, condition: request.condition, model: "TinyCLIP-39M" as const,- cache_hit: response.cache_hit, image: source.image,- timing_ms: { total: performance.now() - start, inference: response.inference_ms, load: response.load_ms } };- return batch- ? { ...common, results: crops.map((region, i) => ({ region, score: scores[i], value: decisions[i] })) }- : { ...common, value: decisions[0], score: scores[0], ...(crops.length ? { region: crops[0] } : {}) };- });- this.tail = run;- try { return await run; } finally { this.queued--; }- }-- private exchange(body: object, signal?: AbortSignal): Promise<Record<string, unknown>> {- signal?.throwIfAborted();- if (!this.child) {- const child = spawn(this.config.python, [this.worker, this.config.model], {- env: { ...process.env, HF_HUB_OFFLINE: "1", TRANSFORMERS_OFFLINE: "1", PYTHONUNBUFFERED: "1", TOKENIZERS_PARALLELISM: "false" },- stdio: "pipe",- });- this.child = child; let buffer = "";- const fail = (error: Error) => { if (this.child === child) { this.pending?.reject(error); this.pending = undefined; child.kill(); } };- child.stdout.setEncoding("utf8");- child.stdout.on("data", (chunk: string) => {- buffer += chunk;- if (buffer.length > 8192) return fail(Error("Classifier worker output exceeded its bound."));- let newline: number;- while ((newline = buffer.indexOf("\n")) >= 0) {- const line = buffer.slice(0, newline); buffer = buffer.slice(newline + 1);- try {- const response = JSON.parse(line);- if (!this.pending || response.id !== this.pending.id) throw Error("Classifier worker response identity mismatch.");- const pending = this.pending; this.pending = undefined;- if (typeof response.error === "string") pending.reject(Error(response.error)); else pending.resolve(response);- } catch { fail(Error("Invalid classifier worker protocol.")); }- }- });- // Library startup messages are not image results or unbounded retained logs.- child.stderr.resume();- child.stdin.on("error", () => fail(Error("Classifier worker input failed.")));- child.on("error", () => fail(Error("Cannot start the configured classifier Python runtime.")));- child.on("close", () => { if (this.child === child) { this.child = undefined; this.pending?.reject(Error("Classifier worker exited before its receipt.")); this.pending = undefined; } });- }- const child = this.child;- if (child.killed) return Promise.reject(Error("Previous classifier worker shutdown is unconfirmed."));- return new Promise((resolve, reject) => {- const id = ++this.nextId;- const stop = (error: Error) => { cleanup(); this.pending = undefined; void this.stopWorker(child).then(() => reject(error), reject); };- const abort = () => stop(Error("Classifier request aborted; no result was produced."));- const timer = setTimeout(() => stop(Error("Classifier request timed out after 60 seconds.")), 60_000);- const cleanup = () => { clearTimeout(timer); signal?.removeEventListener("abort", abort); };- this.pending = { id, resolve: value => { cleanup(); resolve(value); }, reject: error => { cleanup(); reject(error); } };- signal?.addEventListener("abort", abort, { once: true });- if (signal?.aborted) { cleanup(); abort(); return; }- child.stdin.write(JSON.stringify({ id, ...body }) + "\n", error => { if (error) this.pending?.reject(Error("Classifier worker input failed.")); });- });- }-- private async stopWorker(child = this.child): Promise<void> {- if (!child) return;- if (this.child === child) { this.pending?.reject(Error("Classifier worker closed.")); this.pending = undefined; }- if (child.exitCode !== null || child.signalCode !== null) return;- await new Promise<void>((resolve, reject) => {- const timer = setTimeout(() => reject(Error("Classifier worker shutdown is unconfirmed.")), 2000);- child.once("close", () => { clearTimeout(timer); resolve(); });- child.kill("SIGKILL");- });- }-- /** Release only the owned model worker. Future calls may start a fresh one. */- async close(): Promise<void> {- this.closing = true; this.generation++;- try { await this.stopWorker(); await this.tail.catch(() => undefined); }- finally { this.closing = false; }- }-}diff --git a/src/web-browser.ts b/src/web-browser.tsindex c37082c..9335d34 100644--- a/src/web-browser.ts+++ b/src/web-browser.ts@@ -3,9 +3,9 @@ import { fileURLToPath } from "node:url";import { promisify } from "node:util";import type { CommandRunner } from "./runner.ts";import { ATTACH_APPROVAL_MS } from "./browser-command-line.ts";+import { findWindowsNode } from "./windows-node.ts";const run = promisify(execFile);-const WINDOWS_NODE = "/mnt/c/Program Files/nodejs/node.exe";/** An element chosen by Playwright: an aria ref from the latest DOM snapshot, or locator predicates that must all match. */export interface WebTarget {@@ -61,7 +61,7 @@ export interface WebDownload {state: "in_progress" | "completed" | "canceled" | "failed";path?: string; bytes?: number; error?: string;}-/** Installed Chrome or Edge launched by Playwright, or Playwright's own Firefox build (scripts/install-firefox.sh). */+/** Installed Chrome or Edge launched by Playwright, or Playwright's own Firefox build (/computer-setup firefox). */export type WebChannel = "chrome" | "msedge" | "firefox";/*** Browser launch request. `profile: "agent"` uses the agent's own persistent profile on that machine (YOLO only):@@ -341,7 +341,7 @@ export class WebBrowser {child = runner.spawn(options.remote.node, ["--disable-warning=ExperimentalWarning", worker]);} else {if (process.platform === "linux") {- command = WINDOWS_NODE;+ command = await findWindowsNode(options.interop).catch((error: Error) => { throw new WebError(error.message); });worker = (await run("wslpath", ["-w", worker], { timeout: 5000 })).stdout.trim();} else if (process.platform !== "win32") throw new WebError("The Playwright browser backend runs only on Windows or WSL.");signal?.throwIfAborted();diff --git a/src/web-worker.ts b/src/web-worker.tsindex 0e21028..9776c68 100644--- a/src/web-worker.ts+++ b/src/web-worker.ts@@ -161,7 +161,11 @@ async function open({ browser, url, profile: kind, attach: target }: WebLaunchOp// Playwright's default flags trade browser security for test determinism; keep only what an owned pipe/profile needs.// Firefox is Playwright's own build, which needs Playwright's own flags.context = browser === "firefox"- ? await firefox.launchPersistentContext(directory, { headless: false, viewport: null })+ ? await firefox.launchPersistentContext(directory, { headless: false, viewport: null }).catch((error: Error) => {+ if (!/Executable doesn't exist/.test(error.message)) throw error;+ throw new Error("Playwright's Firefox is not installed on this machine. Ask the user before installing it; with their permission run "+ + "/computer-setup firefox (add the machine name for another machine), then open the browser again.");+ }): await chromium.launchPersistentContext(directory, {channel: browser, headless: false, chromiumSandbox: true, viewport: null, ignoreDefaultArgs: true,args: ["--user-data-dir=" + directory, "--remote-debugging-pipe", "--no-first-run", "--no-default-browser-check",diff --git a/src/windows-node.ts b/src/windows-node.tsnew file mode 100644index 0000000..d64c8ca--- /dev/null+++ b/src/windows-node.ts@@ -0,0 +1,25 @@+import { access, readFile } from "node:fs/promises";+import path from "node:path";+import { windowsLocalAppData } from "./native-runtime.ts";++const PIN = new URL("../scripts/windows-node.json", import.meta.url);+const INSTALLED_NODE = "/mnt/c/Program Files/nodejs/node.exe";++/** Where `/computer-setup browser` puts the pinned portable Node under a Local Application Data folder (WSL path). */+export async function portableNodePath(localAppData: string): Promise<string> {+ const { version } = JSON.parse(await readFile(PIN, "utf8")) as { version: string };+ return path.join(localAppData, "Programs", "PiComputerUse", `node-${version}-win-x64`, "node.exe");+}++const exists = (file: string) => access(file).then(() => true, () => false);++/**+ * The Windows node.exe that runs the Playwright worker from WSL: the portable one from setup, else an installed Node.+ * Without either, refuse with the setup step instead of failing on a missing executable.+ */+export async function findWindowsNode(interop?: string, deps = { localAppData: windowsLocalAppData, exists }): Promise<string> {+ const candidates = [await portableNodePath(await deps.localAppData(interop)), INSTALLED_NODE];+ for (const candidate of candidates) if (await deps.exists(candidate)) return candidate;+ throw new Error("Browsers need Node.js on Windows, and none is installed here. Ask the user before installing it; "+ + "with their permission run /computer-setup browser (about 30 MB into their own folders), then open the browser again.");+}diff --git a/test/fixtures/classifier-extension.ts b/test/fixtures/classifier-extension.tsdeleted file mode 100644index db983a4..0000000--- a/test/fixtures/classifier-extension.ts+++ /dev/null@@ -1,16 +0,0 @@-import { createCodemodeExtension, type ExtensionAPI } from "@earendil-works/pi-coding-agent";-import { Type } from "@earendil-works/pi-ai";-import { registerClassifierTool } from "../../src/classifier-tool.ts";-import { VisualClassifier } from "../../src/visual-classifier.ts";-import { fileURLToPath } from "node:url";--/** Deterministic transport/branching peer; this does not establish model recognition. */-export default function classifierFixture(pi: ExtensionAPI): void {- createCodemodeExtension({ mode: "on" })(pi);- const close = registerClassifierTool(pi, async () => new VisualClassifier(- { python: process.execPath, model: "fixture" },- fileURLToPath(new URL("./visual-classifier-worker.mjs", import.meta.url))));- pi.registerTool({ name: "close_fixture_classifier", label: "Close fixture classifier", description: "Release the owned fixture worker.",- parameters: Type.Object({}), async execute() { await close(); return { content: [{type:"text",text:"Closed"}],details:{} }; } });- pi.on("session_shutdown", close);-}diff --git a/test/isolated-session.test.ts b/test/isolated-session.test.tsindex ea3aa99..747c2e1 100644--- a/test/isolated-session.test.ts+++ b/test/isolated-session.test.ts@@ -61,7 +61,7 @@ test("a session provisions, starts its driver through the reverse relay, and clotest("a machine without MXC refuses before provisioning anything", async () => {const f = await fakeHost({ installed: false });- await expect(IsolatedSession.open(f.host, "/mnt/c/runtime/cua-driver.exe")).rejects.toThrow(/MXC is not installed.*install-mxc\.sh/);+ await expect(IsolatedSession.open(f.host, "/mnt/c/runtime/cua-driver.exe")).rejects.toThrow(/MXC is not installed.*Ask the user.*\/computer-setup mxc/);expect(f.phases).toEqual([]);});diff --git a/test/live/reaction-schedules.live.ts b/test/live/reaction-schedules.live.tsdeleted file mode 100644index 7f2e443..0000000--- a/test/live/reaction-schedules.live.ts+++ /dev/null@@ -1,197 +0,0 @@-import { afterAll, beforeAll, expect, test } from "vitest";-import { performance } from "node:perf_hooks";-import sharp from "sharp";-import { connectLive, serveFixture, text } from "./remote.ts";-import { VisualClassifier, loadClassifierConfig } from "../../src/visual-classifier.ts";--// LPT-702: which capture/classification schedule catches a short-lived target best. The page shows a red square for-// FLASH_MS at a random spot every 1.5 s and journals appearances, hits (with reaction time) and misses. Detection is-// TinyCLIP over a 4x4 grid of the 640 px overview (threshold calibrated on this page); the click goes to the red-// pixels inside the chosen cell. Every schedule clicks through the latest observation, which a newer Read replaces.-const FLASH_MS = Number(process.env.FLASH_MS ?? 600);-const RUN_MS = Number(process.env.RUN_MS ?? 30_000);-const CHROME_PX = 60;-// The visible agent cursor glides to each new point before a click, which takes most of a second; reactive animation-// shortens that glide to 1 ms. REACTIVE=0 measures the ordinary glide.-const REACTIVE = process.env.REACTIVE !== "0";-const live = await connectLive();-const fixture = await serveFixture(live.runner);-const classifier = new VisualClassifier(await loadClassifierConfig(process.cwd()));-let window = "";-const results: Record<string, unknown>[] = [];--interface Frame { at: number; observation: string; data: string; mimeType: string }-const capture = async (): Promise<Frame> => {- const view = await live.read(window, ['capture:{"granularity":"image","maxLongEdge":640}']);- const image = view.content.find(block => block.type === "image") as { data: string; mimeType: string };- return { at: performance.now(), observation: view.details.observation_id!, data: image.data, mimeType: image.mimeType };-};-/** Where to click in a frame, or undefined when TinyCLIP finds no red square. */-const detect = async (frame: Frame) => {- const raw = Buffer.from(frame.data, "base64");- const { width, height } = await sharp(raw).metadata();- const cw = Math.floor(width! / 4), ch = Math.floor(height! / 4);- const regions = Array.from({ length: 16 }, (_, k) => ({ x: (k % 4) * cw, y: Math.floor(k / 4) * ch, width: cw, height: ch }));- const scores = (await classifier.classify({ image: { type: "image", data: frame.data, mimeType: frame.mimeType }, regions, condition: "a red square", threshold: 0.24 })).results.map(r => r.score);- const best = scores.indexOf(Math.max(...scores));- if (scores[best] < 0.24) return undefined;- const cell = regions[best];- const { data, info } = await sharp(raw).extract({ left: cell.x, top: cell.y, width: cell.width, height: cell.height }).removeAlpha().raw().toBuffer({ resolveWithObject: true });- let n = 0, sx = 0, sy = 0;- for (let p = 0; p < data.length; p += 3) {- const q = p / 3, y = cell.y + Math.floor(q / info.width);- // The top of the overview is Edge's own toolbar, which has red icons of its own.- if (y >= CHROME_PX && data[p] > 150 && data[p] - data[p + 1] > 100 && data[p] - data[p + 2] > 100) { sx += q % info.width; sy += Math.floor(q / info.width); n++; }- }- return n < 20 ? undefined : { x: cell.x + Math.round(sx / n), y: cell.y + Math.round(sy / n) };-};--/** Run one schedule for RUN_MS and report what the page saw. */-async function measure(name: string, run: (deadline: number, stats: Stats) => Promise<void>) {- await new Promise(resolve => setTimeout(resolve, 2000));- const before = (await fixture.journal()).length;- const stats: Stats = { captured: 0, classified: 0, skipped: 0, clicks: 0, staleness: [], clickMs: [], lastClick: 0 };- await run(performance.now() + RUN_MS, stats);- await new Promise(resolve => setTimeout(resolve, 800));- const events = (await fixture.journal()).slice(before).filter(e => e.page === "/flash");- const reactions = events.filter(e => e.type === "hit").map(e => Number(e.reaction_ms)).sort((a, b) => a - b);- const median = (values: number[]) => values.length ? Math.round([...values].sort((a, b) => a - b)[Math.floor(values.length / 2)]) : null;- const row = { schedule: name, shown: events.filter(e => e.type === "shown").length, hits: reactions.length, misses: events.filter(e => e.type === "miss").length,- reaction_ms_p50: median(reactions), captured: stats.captured, classified: stats.classified, skipped: stats.skipped, clicks: stats.clicks, frame_age_at_click_ms_p50: median(stats.staleness), click_ms_p50: median(stats.clickMs),- miss_detail: events.filter(e => e.type === "miss").slice(0, 4).map(e => ({ n: e.n, click: [Math.round(Number(e.x)), Math.round(Number(e.y))], square: e.square })) };- console.log("SCHEDULE", JSON.stringify(row));- results.push(row);- return row;-}-interface Stats { captured: number; classified: number; skipped: number; clicks: number; staleness: number[]; clickMs: number[]; lastClick: number }--/** Click a detection unless a click was sent after this frame was captured (that appearance is already handled). */-async function act(frame: Frame, at: { x: number; y: number }, observation: string, stats: Stats) {- if (frame.at <= stats.lastClick) return;- stats.staleness.push(performance.now() - frame.at);- const sent = performance.now();- const click = await live.insert(`computer:observation/${observation}?delivery=foreground`, { action: "click", ...at, ...(REACTIVE ? { animation: "reactive" } : {}) });- expect(click.details.status, text(click)).toBe("delivered");- stats.clickMs.push(performance.now() - sent);- stats.clicks++; stats.lastClick = performance.now();-}--beforeAll(async () => {- const opened = await live.insert(live.machine, { action: "open_browser", browser: "msedge", url: `${fixture.base}/flash?ms=${FLASH_MS}` });- expect(opened.details.status, text(opened)).toBe("observed");- window = `computer:window/${opened.details.window_ref}`;- await detect(await capture()); // Warm the classifier model before timing anything.-});-afterAll(async () => {- console.log(`RESULTS flash ${FLASH_MS} ms, reactive ${REACTIVE}\n` + results.map(row => JSON.stringify(row)).join("\n"));- await classifier.close(); await live.close(); fixture.close();-});--test("serial: capture, classify, act", async () => {- const row = await measure("serial", async (deadline, stats) => {- while (performance.now() < deadline) {- const frame = await capture(); stats.captured++;- const at = await detect(frame); stats.classified++;- if (at) await act(frame, at, frame.observation, stats);- }- });- // With the 1 ms glide a 600 ms target is caught most of the time (about 70% on the test machine).- if (REACTIVE && FLASH_MS >= 600) expect(row.hits).toBeGreaterThanOrEqual(row.shown / 3);-});--test("burst: five captures, then classify all and act on the newest hit", async () => {- await measure("burst5", async (deadline, stats) => {- while (performance.now() < deadline) {- const frames: Frame[] = [];- for (let i = 0; i < 5; i++) { frames.push(await capture()); stats.captured++; }- let hit: { frame: Frame; at: { x: number; y: number } } | undefined;- for (const frame of frames) { const at = await detect(frame); stats.classified++; if (at) hit = { frame, at }; }- if (hit) await act(hit.frame, hit.at, frames.at(-1)!.observation, stats);- }- });-});--/** Captures run continuously; the classifier takes the newest waiting frame or every frame in order. */-async function pipelined(deadline: number, stats: Stats, newestOnly: boolean) {- const waiting: Frame[] = [];- let latest = "", done = false, clicking: Promise<void> = Promise.resolve(), inFlight: Promise<unknown> = Promise.resolve();- const capturing = (async () => {- while (performance.now() < deadline) {- // A Read replaces the window's observation, so no capture may run while a click uses the latest one.- await clicking;- const next = capture(); inFlight = next;- const frame = await next; stats.captured++; latest = frame.observation;- waiting.push(frame);- // A bounded queue: the oldest waiting frame is dropped when eight are queued.- if (waiting.length > 8) { waiting.shift(); stats.skipped++; }- }- done = true;- })();- while (!done || waiting.length) {- if (!waiting.length) { await new Promise(resolve => setTimeout(resolve, 5)); continue; }- const taken = newestOnly ? waiting.splice(0) : [waiting.shift()!];- stats.skipped += taken.length - 1;- const frame = taken.at(-1)!;- const at = await detect(frame); stats.classified++;- if (at && performance.now() < deadline) {- let release!: () => void;- clicking = new Promise(resolve => { release = resolve; });- await inFlight.catch(() => undefined);- try { await act(frame, at, latest, stats); } finally { release(); }- }- }- await capturing;-}--test("pipelined, newest frame only", async () => {- await measure("newest-only", (deadline, stats) => pipelined(deadline, stats, true));-});--test("pipelined, every frame in order", async () => {- await measure("every-frame", (deadline, stats) => pipelined(deadline, stats, false));-});--/** Whether the page's bottom-left block, which flips on every click, is dark in a frame. */-async function blockDark(frame: Frame) {- const raw = Buffer.from(frame.data, "base64");- const { height } = await sharp(raw).metadata();- const { data } = await sharp(raw).extract({ left: 8, top: height! - 16, width: 4, height: 4 }).removeAlpha().raw().toBuffer({ resolveWithObject: true });- return data[0] < 128;-}--/** Time from a delivered click until a returned frame shows its effect, capturing back to back. */-async function clickToFrame(rounds: number) {- const latencies: number[] = [], frames: number[] = [];- for (let round = 0; round < rounds; round++) {- const view = await capture();- const before = await blockDark(view);- // An empty spot below the toolbar; the square may be elsewhere, a click on it would also flip the block.- await live.insert(`computer:observation/${view.observation}?delivery=foreground`, { action: "click", x: 20, y: 100 });- const clicked = performance.now();- let n = 0;- while (performance.now() - clicked < 5000) {- const frame = await capture(); n++;- if (await blockDark(frame) !== before) { latencies.push(Math.round(frame.at - clicked)); frames.push(n); break; }- }- }- const sorted = [...latencies].sort((a, b) => a - b);- return { rounds, seen: latencies.length, p50_ms: sorted[Math.floor(sorted.length / 2)], max_ms: sorted.at(-1), frames_until_seen: frames };-}--test("time from a click until a frame shows it", async () => {- const result = await clickToFrame(15);- console.log("CLICK_TO_FRAME", JSON.stringify(result));- results.push({ schedule: "click to frame", ...result });-});-test("parallel captures of one window", async () => {- const time = async (parallel: boolean) => {- const started = performance.now();- if (parallel) await Promise.all(Array.from({ length: 4 }, () => capture()));- else for (let i = 0; i < 4; i++) await capture();- return Math.round(performance.now() - started);- };- const serial = [await time(false), await time(false), await time(false)];- const parallel = [await time(true), await time(true), await time(true)];- console.log("CAPTURE4", JSON.stringify({ serial_ms: serial, parallel_ms: parallel }));- results.push({ schedule: "4 captures", serial_ms: serial, parallel_ms: parallel });-});diff --git a/test/mode.test.ts b/test/mode.test.tsindex de8b106..345ec8f 100644--- a/test/mode.test.ts+++ b/test/mode.test.ts@@ -30,7 +30,7 @@ async function fixture(branch: unknown[] = []) {test("only a user command changes mode; the computer tool surface cannot promote it", async () => {const f = await fixture();expect(f.commands.has("computer-mode")).toBe(true);- expect([...f.tools.keys()]).toEqual(["computer_classify"]);+ expect([...f.tools.keys()]).toEqual([]);expect(await f.close()).toMatchObject({ details: { mode: "yolo" } });await f.commands.get("computer-mode")!.handler("safe", f.ctx);expect(await f.close()).toMatchObject({ details: { mode: "safe" } });diff --git a/test/setup-command.test.ts b/test/setup-command.test.tsnew file mode 100644index 0000000..a6a0038--- /dev/null+++ b/test/setup-command.test.ts@@ -0,0 +1,74 @@+import { execFile } from "node:child_process";+import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";+import path from "node:path";+import { promisify } from "node:util";+import { afterEach, expect, test, vi } from "vitest";+import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";+import { SETUP_SCRIPT, registerSetupCommand, runSetup, splitArgs } from "../src/setup-command.ts";+import { findWindowsNode } from "../src/windows-node.ts";++const run = promisify(execFile);+const dirs: string[] = [];+afterEach(async () => { await Promise.all(dirs.splice(0).map(dir => rm(dir, { recursive: true, force: true }))); });+async function temp(): Promise<string> {+ await mkdir(".tmp", { recursive: true });+ const dir = await mkdtemp(path.resolve(".tmp", "setup-"));+ dirs.push(dir);+ return dir;+}++/** Run scripts/setup.sh from a project folder; these cases stop before any download, SSH or Windows call. */+async function setup(cwd: string, ...args: string[]): Promise<{ code: number; out: string }> {+ try {+ const { stdout, stderr } = await run("bash", [SETUP_SCRIPT, ...args], { cwd, timeout: 20_000 });+ return { code: 0, out: stdout + stderr };+ } catch (error) {+ const failed = error as { code: number; stdout: string; stderr: string };+ return { code: failed.code, out: failed.stdout + failed.stderr };+ }+}++test("setup.sh lists its components and refuses unknown components and machines before doing anything", async () => {+ const cwd = await temp();+ await mkdir(path.join(cwd, ".pi"));+ await writeFile(path.join(cwd, ".pi", "computer-use.json"), JSON.stringify({ machines: { lab: { ssh: "lab-host", command: "/mnt/c/x/cua-driver.exe" } } }));+ const help = await setup(cwd);+ expect(help.code).toBe(0);+ for (const name of ["browser", "firefox", "perception", "mxc", "vscode-bridge", "machine"]) expect(help.out).toContain(name);+ expect(await setup(cwd, "flux")).toMatchObject({ code: 2, out: expect.stringContaining("Unknown component: flux") });+ expect(await setup(cwd, "mxc", "nowhere")).toMatchObject({ code: 1, out: expect.stringContaining("No machine named nowhere") });+ // A machine prepared without the browser worker names the step that adds it.+ expect(await setup(cwd, "firefox", "lab")).toMatchObject({ code: 1, out: expect.stringContaining('Machine lab has no "node"; prepare it with: setup.sh machine lab-host lab') });+ expect((await setup(cwd, "machine")).code).not.toBe(0);+});++test("the command passes quoted paths whole and reports the exit code with the cleaned output tail and full log", async () => {+ expect(splitArgs(` vscode-bridge local "/mnt/c/Tools/VS Code/bin/code.cmd" '/mnt/c/a b/argv.json'`))+ .toEqual(["vscode-bridge", "local", "/mnt/c/Tools/VS Code/bin/code.cmd", "/mnt/c/a b/argv.json"]);+ const dir = await temp();+ const script = path.join(dir, "fake-setup.sh");+ await writeFile(script, 'printf "args:%s|" "$@"; echo; printf "10%%\\r50%%\\r100%%\\n"; echo oops >&2; exit 3\n');+ const result = await runSetup(["perception", "lab"], dir, script);+ expect(result.code).toBe(3);+ expect(result.tail).toBe("args:perception|args:lab|\n100%\noops");+ expect(await readFile(result.log, "utf8")).toContain("10%\r50%\r100%");++ const commands = new Map<string, { getArgumentCompletions: (prefix: string) => unknown; handler: (args: string, ctx: unknown) => Promise<void> }>();+ registerSetupCommand({ registerCommand: (name: string, command: never) => commands.set(name, command) } as unknown as ExtensionAPI, script);+ const command = commands.get("computer-setup")!;+ expect(command.getArgumentCompletions("pe")).toEqual([expect.objectContaining({ value: "perception" })]);+ expect(command.getArgumentCompletions("perception l")).toBeNull();+ const ui = { notify: vi.fn(), setStatus: vi.fn() };+ await command.handler("perception lab", { cwd: dir, ui });+ expect(ui.notify).toHaveBeenCalledWith(expect.stringMatching(/^Computer setup failed \(exit 3\): perception lab\n.*oops\nFull log: /s), "error");+ expect(ui.setStatus).toHaveBeenLastCalledWith("computer-setup", undefined);+});++test("browsers use the portable Windows Node from setup first, an installed Node next, and otherwise name the setup step", async () => {+ const deps = (present: string[]) => ({ localAppData: async () => "/mnt/c/Users/u/AppData/Local", exists: async (file: string) => present.includes(file) });+ const portable = "/mnt/c/Users/u/AppData/Local/Programs/PiComputerUse/node-v22.22.2-win-x64/node.exe";+ const installed = "/mnt/c/Program Files/nodejs/node.exe";+ expect(await findWindowsNode(undefined, deps([portable, installed]))).toBe(portable);+ expect(await findWindowsNode(undefined, deps([installed]))).toBe(installed);+ await expect(findWindowsNode(undefined, deps([]))).rejects.toThrow(/Ask the user.*\/computer-setup browser/);+});diff --git a/test/unified-tools.integration.test.ts b/test/unified-tools.integration.test.tsindex e0fa926..b3f7bdd 100644--- a/test/unified-tools.integration.test.ts+++ b/test/unified-tools.integration.test.ts@@ -29,11 +29,11 @@ test("production Pi exposes only computer tools, with no direct or Codemode browassistantMessage([text("Unified tool loadout checked.")]),],}).run("Check the production loadout. Only discover apps; never open a window or send input.");- // Without Pi Agent IDE, production computer use offers deferred tools that tool_search loads, plus the classifier.+ // Without Pi Agent IDE, production computer use offers deferred tools that tool_search loads.const standalone = ["computer_insert", "computer_read", "computer_search", "computer_write"];expect(getToolExecutionDetails(getToolExecution(result, "registry"))).toEqual({- names: [...standalone, "computer_classify"].sort(),- exposure: { ...Object.fromEntries(standalone.map(name => [name, "deferred"])), computer_classify: "direct" },+ names: standalone,+ exposure: Object.fromEntries(standalone.map(name => [name, "deferred"])),callable: standalone, active: [],});for (const id of ["direct-browser", "nested-browser"]) {diff --git a/test/visual-classifier.integration.test.ts b/test/visual-classifier.integration.test.tsdeleted file mode 100644index ca038fb..0000000--- a/test/visual-classifier.integration.test.ts+++ /dev/null@@ -1,57 +0,0 @@-import { mkdir, mkdtemp, rm } from "node:fs/promises";-import path from "node:path";-import sharp from "sharp";-import { assistantMessage, getToolExecution, getToolResultText, PiIntegrationTest, testArtifactsDir, text, toolCall } from "pi-coding-agent-test";-import { expect, test } from "vitest";--test("real Pi Codemode independently branches on true, false, unknown and error", async () => {- await mkdir(".tmp/classifier-predicate", {recursive:true});- const cwd = await mkdtemp(path.resolve(".tmp/classifier-predicate/pi-"));- const red = await sharp({create:{width:2,height:2,channels:3,background:"red"}}).png().toBuffer();- await sharp({create:{width:8,height:8,channels:3,background:"green"}})- .composite([{input:red,left:5,top:3}]).png().toFile(path.join(cwd,"sample.png"));- const code = `const classify = extra => tools.computer_classify({image:"sample.png",condition:"one condition",threshold:.3,...extra});-const yes = await classify({});-const no = await classify({threshold:.5});-const unknown = await classify({threshold:.4,uncertainty:.02});-let error = false;-try { await classify({condition:"worker error"}); } catch { error = true; }-const region = {x:5,y:3,width:2,height:2};-const cropped = await classify({condition:"red pixel",threshold:.5,region});-const whole = await classify({condition:"red pixel",threshold:.5});-let rejectedRegion = false;-try { await classify({region:{x:7,y:7,width:2,height:2}}); } catch { rejectedRegion = true; }-if(cropped.value!==true || whole.value!==false || !rejectedRegion || JSON.stringify(cropped.region)!==JSON.stringify(region)) throw Error("Broken region transfer");-const regions = [region,{x:0,y:0,width:2,height:2}];-const batch = await classify({condition:"red pixel",threshold:.5,regions});-if(batch.results.length!==2 || batch.results[0].value!==true || batch.results[1].value!==false ||- JSON.stringify(batch.results.map(r=>r.region))!==JSON.stringify(regions)) throw Error("Broken batch transfer");-let rejectedBoth = false;-try { await classify({region,regions}); } catch { rejectedBoth = true; }-if(!rejectedBoth) throw Error("Conflicting regions were accepted");-const branches = [];-for (const result of [yes,no,unknown]) {- if(result.value === true) branches.push("react");- else if(result.value === false) branches.push("continue");- else if(result.value === null) branches.push("inspect");- else throw Error("Not a typed predicate");-}-if(branches.join(",")!=="react,continue,inspect" || !error || !no.cache_hit || yes.cache_hit) throw Error("Broken classifier contract");-await tools.close_fixture_classifier({});-const fresh = await classify({});-if(fresh.cache_hit) throw Error("Close kept the fixture worker warm");-text({branches,error,cache_reused:no.cache_hit,fresh_after_close:!fresh.cache_hit});`;- try {- const result = await new PiIntegrationTest({- testName:"visual-classifier",artifactsDir:testArtifactsDir(import.meta.filename),cwd,- extensions:[path.resolve("test/fixtures/classifier-extension.ts")],isolateUserResources:true,- tools:["codemode","computer_classify","close_fixture_classifier"],- conversation:[assistantMessage([toolCall({id:"predicate",name:"codemode",arguments:{code}})],{stopReason:"toolUse"}),- assistantMessage([text("Done")])],- }).run("Check independent visual predicate branching without capture or input.");- expect(getToolExecution(result,"predicate").isError,getToolResultText(result,"predicate")).toBe(false);- expect(getToolResultText(result,"predicate")).toContain("inspect");- const body = getToolResultText(result,"predicate");- expect(JSON.parse(body.slice(body.indexOf("{")))).toMatchObject({ branches:["react","continue","inspect"],error:true,cache_reused:true,fresh_after_close:true });- } finally { await rm(cwd,{recursive:true,force:true}); }-});diff --git a/test/visual-classifier.test.ts b/test/visual-classifier.test.tsdeleted file mode 100644index 17f9249..0000000--- a/test/visual-classifier.test.ts+++ /dev/null@@ -1,150 +0,0 @@-import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";-import path from "node:path";-import sharp from "sharp";-import { expect, test } from "vitest";-import { VisualClassifier, decide } from "../src/visual-classifier.ts";--const image = async () => ({ type: "image" as const, data: (await sharp({ create: { width: 8, height: 8, channels: 3, background: "green" } }).png().toBuffer()).toString("base64"), mimeType: "image/png" });--test("one raw score takes exactly one branch; uncertainty includes its boundaries", () => {- expect(decide(.4, .3, .02)).toBe(true);- expect(decide(.2, .3, .02)).toBe(false);- for (const score of [.28, .3, .32]) expect(decide(score, .3, .02)).toBe(null);- expect(decide(.3, .3, 0)).toBe(null);- for (const args of [[NaN,.3,0],[.2,Infinity,0],[.2,.99,.1],[.2,.3,-.1]])- expect(() => decide(...args as [number,number,number])).toThrow();-});--test("the owned worker stays warm, caches one condition, and errors never become false", async () => {- const classifier = new VisualClassifier({ python: process.execPath, model: "fixture" }, path.resolve("test/fixtures/visual-classifier-worker.mjs"));- try {- const pixels = await image();- const first = await classifier.classify({ image: pixels, condition: "visible", threshold: .3 });- const second = await classifier.classify({ image: pixels, condition: "visible", threshold: .5, uncertainty: .2 });- expect(first).toMatchObject({ status: "classified", value: true, score: .4, cache_hit: false });- expect(second).toMatchObject({ status: "classified", value: null, cache_hit: true });- await expect(classifier.classify({ image: pixels, condition: "worker error", threshold: .3 })).rejects.toThrow("Fixture error");- await expect(classifier.classify({ image: pixels, condition: "", threshold: .3 })).rejects.toThrow();- await expect(classifier.classify({ image: { ...pixels, data: "not base64" }, condition: "visible", threshold: .3 })).rejects.toThrow();- expect(await classifier.classify({ image: pixels, condition: "another", threshold: .5 })).toMatchObject({ value: false, cache_hit: false });- } finally { await classifier.close(); }-});--test("an aborted owned request is not replayed and close permits a fresh worker", async () => {- const classifier = new VisualClassifier({ python: process.execPath, model: "fixture" }, path.resolve("test/fixtures/visual-classifier-worker.mjs"));- try {- const pixels = await image(); const abort = new AbortController();- const pending = classifier.classify({ image: pixels, condition: "wait", threshold: .3 }, abort.signal);- setTimeout(() => abort.abort(), 100);- await expect(pending).rejects.toThrow(/abort/i);- expect(await classifier.classify({ image: pixels, condition: "visible", threshold: .3 })).toMatchObject({ cache_hit: false });- await classifier.close();- expect(await classifier.classify({ image: pixels, condition: "visible", threshold: .3 })).toMatchObject({ cache_hit: false });- } finally { await classifier.close(); }-});--test("close rejects queued work without starting a replacement model", async () => {- const classifier = new VisualClassifier({ python: process.execPath, model: "fixture" }, path.resolve("test/fixtures/visual-classifier-worker.mjs"));- const pixels = await image();- const active = classifier.classify({image:pixels,condition:"wait",threshold:.3});- const queued = classifier.classify({image:pixels,condition:"visible",threshold:.3});- const results = Promise.allSettled([active, queued]);- await new Promise(resolve => setTimeout(resolve,50));- await classifier.close();- expect((await results).map(result=>result.status)).toEqual(["rejected","rejected"]);-});--test("overflow and already-aborted work never bypass the bounded queue", async () => {- const classifier = new VisualClassifier({python:process.execPath,model:"fixture"},path.resolve("test/fixtures/visual-classifier-worker.mjs"));- const pixels = await image();- const requests = Array.from({length:8},()=>classifier.classify({image:pixels,condition:"wait",threshold:.3}));- const joined = Promise.allSettled(requests);- await expect(classifier.classify({image:pixels,condition:"visible",threshold:.3})).rejects.toThrow(/queue is full/);- await classifier.close(); expect((await joined).every(result=>result.status==="rejected")).toBe(true);- const abort = new AbortController(); abort.abort();- await expect(classifier.classify({image:pixels,condition:"visible",threshold:.3},abort.signal)).rejects.toThrow();- await classifier.close();-});-test("a missing Python runtime rejects and still closes without a detection", async () => {- const classifier = new VisualClassifier({python:"/no-such-classifier-python",model:"fixture"});- await expect(classifier.classify({image:await image(),condition:"visible",threshold:.3})).rejects.toThrow(/Cannot start/);- await classifier.close();-});-test("a region classifies only its pixels and keeps the source image identity", async () => {- const classifier = new VisualClassifier({ python: process.execPath, model: "fixture" }, path.resolve("test/fixtures/visual-classifier-worker.mjs"));- try {- const red = await sharp({ create: { width: 2, height: 2, channels: 3, background: "red" } }).png().toBuffer();- const frame = await sharp({ create: { width: 8, height: 8, channels: 3, background: "green" } })- .composite([{ input: red, left: 5, top: 3 }]).png().toBuffer();- const input = { type: "image" as const, data: frame.toString("base64"), mimeType: "image/png" };- const whole = await classifier.classify({ image: input, condition: "red pixel", threshold: .5 });- const region = { x: 5, y: 3, width: 2, height: 2 };- const selected = await classifier.classify({ image: input, region, condition: "red pixel", threshold: .5 });- expect(whole.value).toBe(false);- expect(selected).toMatchObject({ value: true, region, image: whole.image, cache_hit: true });- const green = await classifier.classify({ image: input, region: { x: 0, y: 0, width: 2, height: 2 }, condition: "red pixel", threshold: .5 });- expect(green.value).toBe(false);- const edge = await classifier.classify({ image: input, region: { x: 7, y: 7, width: 1, height: 1 }, condition: "red pixel", threshold: .5 });- expect(edge.value).toBe(false);- const rotated = await sharp(frame).withMetadata({ orientation: 6 }).jpeg().toBuffer();- await expect(classifier.classify({ image: { ...input, data: rotated.toString("base64"), mimeType: "image/jpeg" }, region, condition: "red pixel", threshold: .5 })).rejects.toThrow(/EXIF rotation/);- } finally { await classifier.close(); }-});--test("one batch returns ordered independent decisions for one shared image and condition", async () => {- const classifier = new VisualClassifier({ python: process.execPath, model: "fixture" }, path.resolve("test/fixtures/visual-classifier-worker.mjs"));- try {- const red = await sharp({ create: { width: 2, height: 2, channels: 3, background: "red" } }).png().toBuffer();- const bytes = await sharp({ create: { width: 8, height: 8, channels: 3, background: "green" } })- .composite([{ input: red, left: 5, top: 3 }]).png().toBuffer();- const input = { type: "image" as const, data: bytes.toString("base64"), mimeType: "image/png" };- const regions = [{ x: 5, y: 3, width: 2, height: 2 }, { x: 0, y: 0, width: 2, height: 2 }];- const batch = await classifier.classify({ image: input, regions, condition: "red pixel", threshold: .5 });- expect(batch.results).toEqual([{ region: regions[0], value: true, score: 1 }, { region: regions[1], value: false, score: 0 }]);- const single = await classifier.classify({ image: input, region: regions[0], condition: "red pixel", threshold: .5 });- expect(batch.image).toEqual(single.image);- expect(single.cache_hit).toBe(true);- const boundary = await classifier.classify({ image: input, regions, condition: "red pixel", threshold: 1 });- expect(boundary.results.map(result => result.value)).toEqual([null, false]);- await expect(classifier.classify({ image: input, regions, condition: "bad batch receipt", threshold: .5 })).rejects.toThrow(/receipt/);- } finally { await classifier.close(); }-});--test("a batch refuses invalid collections and excessive crop area before starting a worker", async () => {- const classifier = new VisualClassifier({ python: "/no-such-batch-worker", model: "fixture" });- try {- const input = await image(), region = { x: 0, y: 0, width: 8, height: 8 };- for (const regions of [[], Array.from({ length: 17 }, () => region), [region, { ...region, x: 1 }]]) {- await expect(classifier.classify({ image: input, regions, condition: "red pixel", threshold: .5 })).rejects.toThrow(/region/i);- }- await expect(classifier.classify({ image: input, region, regions: [region], condition: "red pixel", threshold: .5 })).rejects.toThrow(/region/i);- const bytes = await sharp({ create: { width: 4000, height: 2000, channels: 3, background: "green" } }).png().toBuffer();- const large = { ...input, data: bytes.toString("base64") };- const regions = Array.from({ length: 3 }, () => ({ x: 0, y: 0, width: 4000, height: 2000 }));- await expect(classifier.classify({ image: large, regions, condition: "red pixel", threshold: .5 })).rejects.toThrow(/area/);- } finally { await classifier.close(); }-});-test("invalid regions reject before starting the worker, without clipping", async () => {- const classifier = new VisualClassifier({ python: "/no-such-region-worker", model: "fixture" });- try {- const input = await image();- for (const region of [- { x: -1, y: 0, width: 2, height: 2 }, { x: 0, y: 0, width: 0, height: 2 },- { x: 7, y: 7, width: 2, height: 2 }, { x: .5, y: 0, width: 2, height: 2 },- { x: 0, y: NaN, width: 2, height: 2 }, { x: 0, y: 0, width: 2, height: Infinity },- ]) {- await expect(classifier.classify({ image: input, region, condition: "red pixel", threshold: .5 })).rejects.toThrow(/region/i);- }- } finally { await classifier.close(); }-});-test("saved image input is bounded and config is not part of the request", async () => {- await mkdir(".tmp/classifier-predicate", { recursive: true });- const dir = await mkdtemp(path.resolve(".tmp/classifier-predicate/contract-"));- const classifier = new VisualClassifier({ python: process.execPath, model: "fixture" }, path.resolve("test/fixtures/visual-classifier-worker.mjs"));- try {- const pixels = await image(); await writeFile(path.join(dir, "image.png"), Buffer.from(pixels.data, "base64"));- expect(await classifier.classify({ image: "image.png", condition: "visible", threshold: .3 }, undefined, dir)).toMatchObject({ value: true, image: { width: 8, height: 8 } });- await writeFile(path.join(dir, "bad.png"), "not pixels");- await expect(classifier.classify({ image: "bad.png", condition: "visible", threshold: .3 }, undefined, dir)).rejects.toThrow();- } finally { await classifier.close(); await rm(dir, { recursive: true, force: true }); }-});diff --git a/vscode-browser-bridge/README.md b/vscode-browser-bridge/README.mdindex 2aeaf89..6c514f5 100644--- a/vscode-browser-bridge/README.md+++ b/vscode-browser-bridge/README.md@@ -6,7 +6,7 @@ VS Code's proposed `browser` API gives each integrated browser tab a Chrome DevTSetup:-1. Build it with `scripts/package-vscode-bridge.sh` and install the `.vsix` with `code --install-extension`.+1. Install it with `/computer-setup vscode-bridge`, or build it with `scripts/package-vscode-bridge.sh` and install the `.vsix` with `code --install-extension`.2. Run **Preferences: Configure Runtime Arguments** and add `"enable-proposed-api": ["alexshpunt.pi-browser-bridge"]`.3. Quit VS Code fully and start it again; Reload Window does not reread that file.