Skip to content

Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.

Promote Computer Use settings and GIMP demo for 0.1.3 (#180)

alexshpunt-clanker[bot]6668cb4f9006
diff --git a/README.md b/README.md
index 83aa638..b07788f 100644
--- a/README.md
+++ b/README.md
@@ -19,12 +19,40 @@
[![Pi package](https://img.shields.io/badge/pi-package-6b4fbb)](https://pi.dev/packages/pi-agent-computer-use)
[![Works with Pi Agent IDE](https://img.shields.io/badge/works%20with-Pi%20Agent%20IDE-2f7d4a)](https://github.com/alexshpunt/pi-agent-ide)
-[![Platform: Windows from WSL](https://img.shields.io/badge/platform-Windows%20from%20WSL-0078d4)](#platforms)
+[![Platform: Windows and WSL](https://img.shields.io/badge/platform-Windows%20and%20WSL-0078d4)](#platforms)
</div>
Ask your agent to do something on your computer, for example "open Paint and draw a house", "fill in this form in my browser" or "check what the settings window says". It finds the window, reads its controls and screenshot, acts, and reads it again to see whether it worked.
+## Demo: an agent paints a picture in GIMP
+
+<p align="center">
+ <img src="images/gimp-demo/timelapse-preview.webp" alt="Time-lapse of a Pi agent painting a cartoon robot at a computer in GIMP, with the reference picture in the corner" width="100%">
+</p>
+
+We give a Pi agent [this reference picture](assets/draw_task.png) and an empty canvas in GIMP on another Windows machine, and ask it to paint a copy. It paints every mark itself with GIMP's tools: filled selections, soft shading inside them, tapered pen strokes for the line art, hatching and hand-drawn lettering. Along the way it compares its canvas with the reference and fixes the biggest differences. The animation above is a 61-minute run of `gpt-6.1-sol`, sped up about 120 times.
+
+We run it with different models. Click a painting to see it next to the reference and watch the full time-lapse:
+
+<!-- gimp-gallery:start -->
+
+<table>
+<tr>
+<td align="center" width="33%"><a href="demo/gimp-painting/gallery/README.md#deepseek-flash-2026-11-01"><img src="demo/gimp-painting/gallery/deepseek-flash-2026-11-01/painting.jpg" width="280" alt="Painting by deepseek/deepseek-flash"></a><br><b>deepseek-flash</b><br>82 min · $0.34</td>
+<td align="center" width="33%"><a href="demo/gimp-painting/gallery/README.md#claude-haiku-5-5-2026-11-02"><img src="demo/gimp-painting/gallery/claude-haiku-5-5-2026-11-02/painting.jpg" width="280" alt="Painting by anthropic/claude-haiku-5-5"></a><br><b>claude-haiku-5-5</b><br>27 min · $0.33</td>
+<td align="center" width="33%"><a href="demo/gimp-painting/gallery/README.md#claude-opus-5-5-2026-11-02"><img src="demo/gimp-painting/gallery/claude-opus-5-5-2026-11-02/painting.jpg" width="280" alt="Painting by anthropic/claude-opus-5-5"></a><br><b>claude-opus-5-5</b><br>78 min · $5.04</td>
+</tr>
+<tr>
+<td align="center" width="33%"><a href="demo/gimp-painting/gallery/README.md#gpt-61-sol-2026-11-02"><img src="demo/gimp-painting/gallery/gpt-6.1-sol-2026-11-02/painting.jpg" width="280" alt="Painting by openai-codex/gpt-6.1-sol"></a><br><b>gpt-6.1-sol</b><br>61 min · $1.05</td>
+<td align="center" width="33%"><a href="demo/gimp-painting/gallery/README.md#gpt-61-terra-2026-11-02"><img src="demo/gimp-painting/gallery/gpt-6.1-terra-2026-11-02/painting.jpg" width="280" alt="Painting by openai-codex/gpt-6.1-terra"></a><br><b>gpt-6.1-terra</b><br>14 min · $0.25</td>
+</tr>
+</table>
+
+<!-- gimp-gallery:end -->
+
+The agent follows a [painting skill](demo/gimp-painting/skill/SKILL.md) that breaks the work into stages and gives it a tested GIMP toolkit. [demo/gimp-painting](demo/gimp-painting/README.md) has the prompt and the script that sets up the machine, records the screen and makes the time-lapse, so you can repeat the run with another model or picture.
+
## Summary
Pi Agent Computer Use gives the [Pi coding agent](https://pi.dev/) one interface for every Windows app, browsers included. The agent finds a window, reads its accessibility tree and screenshot, and then types, clicks, drags or runs a browser action. Input always targets the latest thing the agent saw, and every result says what was delivered and what is still unknown.
@@ -82,7 +110,9 @@ Input goes to a window by its address and targets an element by a ref from the l
What the agent sees as an element it presses through UI Automation in the background, so your pointer and focus stay put. What it sees only as pixels it clicks with the real mouse, and keys are real key presses: that input brings its window to the front and runs at once, even while you use the computer. Nothing is posted to windows as messages that an app could silently drop. To keep foreground UI work off the human desktop, use a sandbox or an isolated session.
-Press Escape three times within a second to stop every agent on your desktop. Their input stops at once, and they refuse to act until you run `/computer-mode safe` or `/computer-mode yolo` in their sessions. An agent's own Escape doesn't count. Sandboxes and isolated sessions keep running.
+Press Escape three times within a second to stop agents on your desktop that have the shortcut enabled. Their input stops at once, and they refuse to act until you run `/computer-mode safe` or `/computer-mode yolo` in their sessions. An agent's own Escape doesn't count. Sandboxes and isolated sessions keep running.
+
+The shortcut is enabled by default. Open `/computer-settings` to change it globally or for a trusted project. Restart Pi and reconnect remote machines for the change to take effect. The setting applies to local and connected human desktops in this Pi session; disabling it removes their triple-Escape emergency stop.
Several Pi sessions can use the same desktop. Their actions take turns: a string one agent types or a drag it makes finishes before another agent's next action starts.
@@ -118,12 +148,14 @@ Each session has its own folder on the controlled machine for files an app opens
The agent's cursor carries the Pi session's name, or the project folder's name until the session has one, so you can tell agents apart. A card beside each action names it and shows the text being typed, except into password fields.
-`/computer-visuals` chooses `off`, `subtle` or `full`, with optional image previews when the native driver supports them. A preview shows the final operation image; it does not prove delivery to a model. `/computer-images` changes screenshot size, hints and image guides for future captures.
+Open `/computer-settings` to change the access mode, Escape stop, screenshot settings and visual preferences. Choose Project or Global first. Global settings apply across projects, while settings in a trusted project override individual global values. Select `inherit` to clear an override; for numeric values, enter `inherit`. The selected setting shows its effective value and source. Global preferences are stored in the Pi agent directory as `computer-use-settings.json`; project preferences are stored in `.pi/computer-use-settings.json`. A mode selected with `/computer-mode` remains in effect on its session branch, even when a scoped default differs. Existing session-branch visual preferences remain the fallback where no scoped value is set. An existing `.pi/computer-use.json` `escapeStop` remains a project override until changed in the panel. Screenshot changes affect future captures. A preview shows the final operation image; it does not prove delivery to a model.
## Platforms
Computer use works on Windows 11 x64, with Pi running on Windows itself or in WSL, on the same machine or on another one over SSH. Linux and macOS aren't supported: window control there isn't verified, so no runtime ships for them and computer use refuses with a clear message. Some features need more: Windows Sandbox needs a Windows edition that has it, and isolated sessions need Windows 11 24H2 or newer.
+The extension uses our [Cua Driver fork](https://github.com/alexshpunt/cua), whose driver also targets Linux and macOS. Support for those desktops may be possible, but it needs integration and validation, including for features currently specific to Windows. Contributions and testing on other platforms are very welcome. Please [open an issue](https://github.com/alexshpunt/pi-agent-computer-use/issues), share your findings, or send a pull request.
+
This is experimental and changes often, so interfaces and behavior may still move around.
## Installation
@@ -175,9 +207,8 @@ The Windows binaries aren't code-signed. Signing certificates are hard to get fo
| Command | Effect |
| -------------------------------- | -------------------------------------------------------- |
+| `/computer-settings` | Open the unified Computer Use settings panel |
| `/computer-mode safe\|yolo` | Switch between Safe and YOLO; also turns computer use on again after Escape stopped it |
-| `/computer-visuals off\|subtle\|full` | Choose native activity cues and image previews |
-| `/computer-images` | Show or change capture size, hints and image guides |
| `/computer-setup PART [MACHINE]` | Install an optional part; without a part it lists them |
## Feedback and contributions
diff --git a/assets/banner.png b/assets/banner.png
index accddaa..dec6a55 100644
Binary files a/assets/banner.png and b/assets/banner.png differ
diff --git a/assets/draw_task.png b/assets/draw_task.png
new file mode 100644
index 0000000..0420100
Binary files /dev/null and b/assets/draw_task.png differ
diff --git a/assets/logo.png b/assets/logo.png
index 0731a97..dec6a55 100644
Binary files a/assets/logo.png and b/assets/logo.png differ
diff --git a/assets/social-preview.png b/assets/social-preview.png
index a4ba2c3..90d29fb 100644
Binary files a/assets/social-preview.png and b/assets/social-preview.png differ
diff --git a/demo/gimp-painting/README.md b/demo/gimp-painting/README.md
new file mode 100644
index 0000000..d02943c
--- /dev/null
+++ b/demo/gimp-painting/README.md
@@ -0,0 +1,78 @@
+# GIMP painting demo
+
+A Pi agent paints a copy of [`assets/draw_task.png`](../../assets/draw_task.png) in GIMP on another Windows machine while the screen is recorded. The result is a sped-up video, a short animated preview for the main README and a side-by-side comparison of the reference and the painting.
+
+| File | Purpose |
+| --- | --- |
+| [`prompt.md`](prompt.md) | The task the agent gets |
+| [`skill/SKILL.md`](skill/SKILL.md) | `paint-from-reference`: how to paint a reference in stages, with verified GIMP techniques |
+| [`run.ts`](run.ts) | Prepares the machine, records the screen, runs the agent and cleans up |
+| [`postprocess.ts`](postprocess.ts) | Makes the time-lapse, the preview and the comparison from a recording |
+| [`gallery.ts`](gallery.ts) | Adds a finished run to the [gallery](gallery/README.md) and the thumbnails in the main README |
+
+## What a run does
+
+1. Moves the user's Windows session from Remote Desktop to the physical console, because a Remote Desktop session takes its resolution from the client. The Remote Desktop connection drops; reconnect after the run.
+2. Sets the primary display to 1920×1080.
+3. Starts GIMP with an empty 1200×800 canvas, maximizes it and brings it to the front.
+4. Records the display without the taskbar with ffmpeg's Windows Graphics Capture at 10 fps.
+5. Runs Pi with the prompt, the `computer-use` and `paint-from-reference` skills and no other skills, context files or MCP servers. Tools that ask the user or generate images are excluded.
+6. Stops the recording, closes GIMP once the painting is exported and restores the display resolution.
+7. Encodes `timelapse.mp4` (120 s) and `preview.webp` (30 s) on the Windows machine, both with the reference in a corner, and builds `comparison.png`.
+
+Outputs go to `.tmp/gimp-demo/<run>/` in the repository: the videos, `painting.png`, the full `recording.mkv`, the agent's `events.jsonl` and its Pi session.
+
+To publish a run, add it to the [gallery](#gallery).
+
+## Requirements
+
+- Pi in WSL on the controlling machine with this extension installed and a login for the chosen model.
+- A Windows machine with WSL reachable over SSH, set up as described in [Other machines](../../docs/setup.md#other-machines), with GIMP 3 (`winget install GIMP.GIMP.3`) and ffmpeg 8 or newer (`winget install Gyan.FFmpeg`). GIMP must not be running.
+- On that machine, a task that returns a disconnected Remote Desktop session to the console, such as one running `tscon <session> /dest:console` as SYSTEM on disconnect. Without it, sign in on the console before the run.
+
+## Run
+
+```bash
+node demo/gimp-painting/run.ts --ssh zinpc-benchmark --machine zinpc
+```
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `--ssh` | `zinpc-benchmark` | SSH alias of the Windows machine's WSL |
+| `--machine` | `zinpc` | Machine name the agent uses in `computer:` addresses |
+| `--model` | `openai-codex/gpt-6.1-terra` | Model for the agent |
+| `--thinking` | `high` | Thinking level |
+| `--reference` | `assets/draw_task.png` | Picture to paint |
+| `--canvas` | `1200x800` | Canvas size; keep the reference's aspect ratio and fit it at 100% zoom |
+| `--display` | `1920x1080` | Display resolution during the run |
+| `--max-minutes` | `240` | Time limit for the agent |
+| `--prepare-only` | off | Set the display and open GIMP, then stop: no recording, no agent, nothing restored. For trying the skill by hand |
+| `--video-seconds`, `--preview-seconds` | `120`, `30` | Lengths of the time-lapse and the preview |
+
+To redo the videos of an earlier run, pass its Windows folder, which `run.ts` prints at the end:
+
+```bash
+node demo/gimp-painting/postprocess.ts --ssh zinpc-benchmark --remote-dir 'C:\Users\user\pi-gimp-demo\<run>' --out .tmp/gimp-demo/<run>
+```
+
+## Gallery
+
+[gallery/](gallery/README.md) keeps one folder per run: a thumbnail of the painting, the painting next to the reference and `run.json` with the model, thinking level, date, time, cost, counts and the video URL. `gallery.ts` reads the model, time and cost from the run's Pi event log and writes the gallery page and the thumbnails between the `gimp-gallery` markers in the main README. Don't edit those generated parts by hand.
+
+To add a run:
+
+1. Add it from its output folder. `--feature` also makes its preview the animation at the top of the main README.
+
+ ```bash
+ node demo/gimp-painting/gallery.ts add .tmp/gimp-demo/<run> [--feature]
+ ```
+
+2. Upload the run's `timelapse.mp4` to GitHub: open any issue or pull request form on github.com, drag the file into the comment box or click "Paste, drop, or click to add files", copy the `https://github.com/user-attachments/assets/...` URL GitHub inserts and leave the page without posting. GitHub plays only these URLs as videos in a README, and it has no API for the upload.
+
+3. Store the URL:
+
+ ```bash
+ node demo/gimp-painting/gallery.ts video <run id> https://github.com/user-attachments/assets/<id>
+ ```
+
+`gallery.ts render` rebuilds the page and the README block from the `run.json` files.
diff --git a/demo/gimp-painting/gallery.ts b/demo/gimp-painting/gallery.ts
new file mode 100644
index 0000000..c8cf693
--- /dev/null
+++ b/demo/gimp-painting/gallery.ts
@@ -0,0 +1,145 @@
+// The demo gallery: one folder per run with its painting, a comparison with the reference and run.json, a gallery
+// page with a video player per run, and a block of thumbnails in the repository README.
+// node demo/gimp-painting/gallery.ts add .tmp/gimp-demo/<run> [--video <url>] [--feature]
+// node demo/gimp-painting/gallery.ts video <id> <url>
+// node demo/gimp-painting/gallery.ts render
+// See demo/gimp-painting/README.md#gallery.
+import { createReadStream } from "node:fs";
+import { copyFile, mkdir, readdir, readFile, stat, writeFile } from "node:fs/promises";
+import path from "node:path";
+import { createInterface } from "node:readline";
+import { fileURLToPath } from "node:url";
+import { parseArgs } from "node:util";
+import sharp from "sharp";
+
+const here = path.dirname(fileURLToPath(import.meta.url));
+const repo = path.resolve(here, "../..");
+const galleryDir = path.join(here, "gallery");
+const readmeFile = path.join(repo, "README.md");
+const featuredPreview = path.join(repo, "images/gimp-demo/timelapse-preview.webp");
+const START = "<!-- gimp-gallery:start -->", END = "<!-- gimp-gallery:end -->";
+
+export interface Run {
+ id: string;
+ /** provider/model as Pi names it. */
+ model: string;
+ thinking: string;
+ /** Day the run started, YYYY-MM-DD. */
+ date: string;
+ minutes: number;
+ /** Model cost as Pi reported it, in US dollars. */
+ cost: number;
+ turns: number;
+ toolCalls: number;
+ canvas: string;
+ /** A GitHub user-attachments URL of the full time-lapse, or null until it is uploaded. */
+ video: string | null;
+}
+
+/** Model, thinking level, duration, cost and counts from a run's Pi event log. */
+async function stats(eventsFile: string): Promise<Omit<Run, "id" | "canvas" | "video">> {
+ let start: number | undefined, end = 0, cost = 0, turns = 0, toolCalls = 0, model = "", thinking = "";
+ const lines = createInterface({ input: createReadStream(eventsFile), crlfDelay: Infinity });
+ for await (const line of lines) {
+ // Tool results carry screenshots; only the small event kinds below matter.
+ if (!line.startsWith('{"type":"session"') && !line.startsWith('{"type":"message_end"') && !line.startsWith('{"type":"tool_execution_start"')) continue;
+ const event = JSON.parse(line);
+ if (event.type === "session") start = Date.parse(event.timestamp);
+ else if (event.type === "tool_execution_start") toolCalls++;
+ else if (event.message?.role === "assistant") {
+ turns++;
+ cost += event.message.usage?.cost?.total ?? 0;
+ end = Math.max(end, event.message.timestamp ?? 0);
+ model = `${event.message.provider}/${event.message.model}`;
+ thinking = event.message.thinkingLevel ?? thinking;
+ }
+ }
+ if (start === undefined || !turns) throw new Error(`${eventsFile} has no session or assistant turns.`);
+ return { model, thinking: thinking || "off", date: new Date(start).toISOString().slice(0, 10), minutes: Math.round((end - start) / 60_000),
+ cost: Math.round(cost * 100) / 100, turns, toolCalls };
+}
+
+async function exists(file: string) { return stat(file).then(() => true, () => false); }
+
+async function runs(): Promise<Run[]> {
+ const ids = await readdir(galleryDir).catch(() => [] as string[]);
+ const found: Run[] = [];
+ for (const id of ids) {
+ const file = path.join(galleryDir, id, "run.json");
+ if (await exists(file)) found.push(JSON.parse(await readFile(file, "utf8")));
+ }
+ return found.sort((a, b) => a.date.localeCompare(b.date) || a.id.localeCompare(b.id));
+}
+
+const shortModel = (run: Run) => run.model.split("/").pop()!;
+const summary = (run: Run) => `${run.minutes} min · ${run.turns} turns · $${run.cost.toFixed(2)} · thinking ${run.thinking}`;
+
+/** Write the gallery page and the README block from the runs' run.json files. */
+async function render(): Promise<void> {
+ const all = await runs();
+ const page = [
+ "# GIMP painting gallery",
+ "",
+ "Each run gave a Pi agent the same [reference picture](../../../assets/draw_task.png) and an empty canvas in GIMP, with the same [prompt](../prompt.md) and [painting skill](../skill/SKILL.md). [The demo guide](../README.md) explains how a run is made and how to add one.",
+ "",
+ "Cost is the model cost Pi reported. Time is from the agent's start to its last answer.",
+ "",
+ ...all.flatMap(run => [
+ `## ${run.id}`,
+ "",
+ `**${run.model}** · ${run.date} · ${summary(run)} · ${run.toolCalls} tool calls`,
+ "",
+ `![The reference and the painting by ${run.model}](${run.id}/comparison.jpg)`,
+ "",
+ // GitHub plays a bare user-attachments URL on its own line as a video.
+ ...(run.video ? [run.video, ""] : ["The time-lapse of this run isn't uploaded yet.", ""]),
+ ]),
+ ].join("\n");
+ await mkdir(galleryDir, { recursive: true });
+ await writeFile(path.join(galleryDir, "README.md"), page);
+
+ const perRow = 3;
+ const cells = all.map(run => `<td align="center" width="${Math.floor(100 / perRow)}%"><a href="demo/gimp-painting/gallery/README.md#${run.id.replace(/\./g, "")}"><img src="demo/gimp-painting/gallery/${run.id}/painting.jpg" width="280" alt="Painting by ${run.model}"></a><br><b>${shortModel(run)}</b><br>${run.minutes} min · $${run.cost.toFixed(2)}</td>`);
+ const rows = [];
+ for (let i = 0; i < cells.length; i += perRow) rows.push(`<tr>\n${cells.slice(i, i + perRow).join("\n")}\n</tr>`);
+ const block = [START, "", `<table>\n${rows.join("\n")}\n</table>`, "", END].join("\n");
+ const readme = await readFile(readmeFile, "utf8");
+ if (!readme.includes(START) || !readme.includes(END)) throw new Error(`README.md needs ${START} and ${END} where the gallery goes.`);
+ await writeFile(readmeFile, readme.slice(0, readme.indexOf(START)) + block + readme.slice(readme.indexOf(END) + END.length));
+ console.log(`Gallery: ${all.length} run(s); wrote ${path.relative(repo, path.join(galleryDir, "README.md"))} and the README block.`);
+}
+
+/** Add a finished run from its output folder. */
+async function add(runDir: string, video: string | undefined, feature: boolean): Promise<void> {
+ const painting = path.join(runDir, "painting.png");
+ if (!(await exists(painting))) throw new Error(`${painting} is missing: the run exported no painting.`);
+ const info = await stats(path.join(runDir, "events.jsonl"));
+ const base = `${shortModel({ model: info.model } as Run)}-${info.date}`.toLowerCase().replace(/[^a-z0-9.-]+/g, "-");
+ let id = base;
+ for (let n = 2; await exists(path.join(galleryDir, id)); n++) id = `${base}-${n}`;
+ const dir = path.join(galleryDir, id);
+ await mkdir(dir, { recursive: true });
+ const { width = 0, height = 0 } = await sharp(painting).metadata();
+ await sharp(painting).resize(600).jpeg({ quality: 85, mozjpeg: true }).toFile(path.join(dir, "painting.jpg"));
+ await sharp(path.join(runDir, "comparison.png")).resize(1600).jpeg({ quality: 85, mozjpeg: true }).toFile(path.join(dir, "comparison.jpg"));
+ const run: Run = { id, ...info, canvas: `${width}x${height}`, video: video ?? null };
+ await writeFile(path.join(dir, "run.json"), JSON.stringify(run, null, 2) + "\n");
+ if (feature) await copyFile(path.join(runDir, "preview.webp"), featuredPreview);
+ console.log(`Added ${id}: ${run.model}, ${summary(run)}.${video ? "" : " Upload timelapse.mp4 and run: gallery.ts video " + id + " <url>"}`);
+ await render();
+}
+
+async function setVideo(id: string, url: string): Promise<void> {
+ if (!/^https:\/\/github\.com\/user-attachments\/assets\/[0-9a-f-]+$/.test(url)) throw new Error("Expected a https://github.com/user-attachments/assets/<id> URL.");
+ const file = path.join(galleryDir, id, "run.json");
+ const run: Run = JSON.parse(await readFile(file, "utf8"));
+ await writeFile(file, JSON.stringify({ ...run, video: url }, null, 2) + "\n");
+ await render();
+}
+
+const { positionals, values } = parseArgs({ allowPositionals: true, options: { video: { type: "string" }, feature: { type: "boolean", default: false } } });
+const [command, ...rest] = positionals;
+if (command === "add" && rest[0]) await add(path.resolve(rest[0]), values.video, values.feature);
+else if (command === "video" && rest[0] && rest[1]) await setVideo(rest[0], rest[1]);
+else if (command === "render") await render();
+else throw new Error("Usage: gallery.ts add <run folder> [--video <url>] [--feature] | video <id> <url> | render");
diff --git a/demo/gimp-painting/gallery/README.md b/demo/gimp-painting/gallery/README.md
new file mode 100644
index 0000000..7be9b9d
--- /dev/null
+++ b/demo/gimp-painting/gallery/README.md
@@ -0,0 +1,45 @@
+# GIMP painting gallery
+
+Each run gave a Pi agent the same [reference picture](../../../assets/draw_task.png) and an empty canvas in GIMP, with the same [prompt](../prompt.md) and [painting skill](../skill/SKILL.md). [The demo guide](../README.md) explains how a run is made and how to add one.
+
+Cost is the model cost Pi reported. Time is from the agent's start to its last answer.
+
+## deepseek-flash-2026-11-01
+
+**deepseek/deepseek-flash** · 2026-11-01 · 82 min · 105 turns · $0.34 · thinking high · 2790 tool calls
+
+![The reference and the painting by deepseek/deepseek-flash](deepseek-flash-2026-11-01/comparison.jpg)
+
+https://github.com/user-attachments/assets/e44e002f-c4a5-4030-ab76-fdc73d2765b5
+
+## claude-haiku-5-5-2026-11-02
+
+**anthropic/claude-haiku-5-5** · 2026-11-02 · 27 min · 40 turns · $0.33 · thinking high · 895 tool calls
+
+![The reference and the painting by anthropic/claude-haiku-5-5](claude-haiku-5-5-2026-11-02/comparison.jpg)
+
+https://github.com/user-attachments/assets/63a2b230-e861-4361-bfec-fe500a58fb5f
+
+## claude-opus-5-5-2026-11-02
+
+**anthropic/claude-opus-5-5** · 2026-11-02 · 78 min · 99 turns · $5.04 · thinking high · 2232 tool calls
+
+![The reference and the painting by anthropic/claude-opus-5-5](claude-opus-5-5-2026-11-02/comparison.jpg)
+
+https://github.com/user-attachments/assets/e272a1aa-bb8b-44af-964c-752c42a0bfa7
+
+## gpt-6.1-sol-2026-11-02
+
+**openai-codex/gpt-6.1-sol** · 2026-11-02 · 61 min · 69 turns · $1.05 · thinking high · 1928 tool calls
+
+![The reference and the painting by openai-codex/gpt-6.1-sol](gpt-6.1-sol-2026-11-02/comparison.jpg)
+
+https://github.com/user-attachments/assets/efce8341-6564-4a00-8a3a-ff8a6b349a15
+
+## gpt-6.1-terra-2026-11-02
+
+**openai-codex/gpt-6.1-terra** · 2026-11-02 · 14 min · 44 turns · $0.25 · thinking high · 552 tool calls
+
+![The reference and the painting by openai-codex/gpt-6.1-terra](gpt-6.1-terra-2026-11-02/comparison.jpg)
+
+https://github.com/user-attachments/assets/564263b1-ed17-42c3-919e-c76fa06d5fb8
diff --git a/demo/gimp-painting/gallery/claude-haiku-5-5-2026-11-02/comparison.jpg b/demo/gimp-painting/gallery/claude-haiku-5-5-2026-11-02/comparison.jpg
new file mode 100644
index 0000000..5cff510
Binary files /dev/null and b/demo/gimp-painting/gallery/claude-haiku-5-5-2026-11-02/comparison.jpg differ
diff --git a/demo/gimp-painting/gallery/claude-haiku-5-5-2026-11-02/painting.jpg b/demo/gimp-painting/gallery/claude-haiku-5-5-2026-11-02/painting.jpg
new file mode 100644
index 0000000..00f46f8
Binary files /dev/null and b/demo/gimp-painting/gallery/claude-haiku-5-5-2026-11-02/painting.jpg differ
diff --git a/demo/gimp-painting/gallery/claude-haiku-5-5-2026-11-02/run.json b/demo/gimp-painting/gallery/claude-haiku-5-5-2026-11-02/run.json
new file mode 100644
index 0000000..ccbe9f5
--- /dev/null
+++ b/demo/gimp-painting/gallery/claude-haiku-5-5-2026-11-02/run.json
@@ -0,0 +1,12 @@
+{
+ "id": "claude-haiku-5-5-2026-11-02",
+ "model": "anthropic/claude-haiku-5-5",
+ "thinking": "high",
+ "date": "2026-11-02",
+ "minutes": 27,
+ "cost": 0.33,
+ "turns": 40,
+ "toolCalls": 895,
+ "canvas": "1200x800",
+ "video": "https://github.com/user-attachments/assets/63a2b230-e861-4361-bfec-fe500a58fb5f"
+}
diff --git a/demo/gimp-painting/gallery/claude-opus-5-5-2026-11-02/comparison.jpg b/demo/gimp-painting/gallery/claude-opus-5-5-2026-11-02/comparison.jpg
new file mode 100644
index 0000000..0ffc4ca
Binary files /dev/null and b/demo/gimp-painting/gallery/claude-opus-5-5-2026-11-02/comparison.jpg differ
diff --git a/demo/gimp-painting/gallery/claude-opus-5-5-2026-11-02/painting.jpg b/demo/gimp-painting/gallery/claude-opus-5-5-2026-11-02/painting.jpg
new file mode 100644
index 0000000..210a9a9
Binary files /dev/null and b/demo/gimp-painting/gallery/claude-opus-5-5-2026-11-02/painting.jpg differ
diff --git a/demo/gimp-painting/gallery/claude-opus-5-5-2026-11-02/run.json b/demo/gimp-painting/gallery/claude-opus-5-5-2026-11-02/run.json
new file mode 100644
index 0000000..1acdd1b
--- /dev/null
+++ b/demo/gimp-painting/gallery/claude-opus-5-5-2026-11-02/run.json
@@ -0,0 +1,12 @@
+{
+ "id": "claude-opus-5-5-2026-11-02",
+ "model": "anthropic/claude-opus-5-5",
+ "thinking": "high",
+ "date": "2026-11-02",
+ "minutes": 78,
+ "cost": 5.04,
+ "turns": 99,
+ "toolCalls": 2232,
+ "canvas": "1200x800",
+ "video": "https://github.com/user-attachments/assets/e272a1aa-bb8b-44af-964c-752c42a0bfa7"
+}
diff --git a/demo/gimp-painting/gallery/deepseek-flash-2026-11-01/comparison.jpg b/demo/gimp-painting/gallery/deepseek-flash-2026-11-01/comparison.jpg
new file mode 100644
index 0000000..aae566a
Binary files /dev/null and b/demo/gimp-painting/gallery/deepseek-flash-2026-11-01/comparison.jpg differ
diff --git a/demo/gimp-painting/gallery/deepseek-flash-2026-11-01/painting.jpg b/demo/gimp-painting/gallery/deepseek-flash-2026-11-01/painting.jpg
new file mode 100644
index 0000000..859a5f5
Binary files /dev/null and b/demo/gimp-painting/gallery/deepseek-flash-2026-11-01/painting.jpg differ
diff --git a/demo/gimp-painting/gallery/deepseek-flash-2026-11-01/run.json b/demo/gimp-painting/gallery/deepseek-flash-2026-11-01/run.json
new file mode 100644
index 0000000..785f0d7
--- /dev/null
+++ b/demo/gimp-painting/gallery/deepseek-flash-2026-11-01/run.json
@@ -0,0 +1,12 @@
+{
+ "id": "deepseek-flash-2026-11-01",
+ "model": "deepseek/deepseek-flash",
+ "thinking": "high",
+ "date": "2026-11-01",
+ "minutes": 82,
+ "cost": 0.34,
+ "turns": 105,
+ "toolCalls": 2790,
+ "canvas": "1200x800",
+ "video": "https://github.com/user-attachments/assets/e44e002f-c4a5-4030-ab76-fdc73d2765b5"
+}
diff --git a/demo/gimp-painting/gallery/gpt-6.1-sol-2026-11-02/comparison.jpg b/demo/gimp-painting/gallery/gpt-6.1-sol-2026-11-02/comparison.jpg
new file mode 100644
index 0000000..c88624e
Binary files /dev/null and b/demo/gimp-painting/gallery/gpt-6.1-sol-2026-11-02/comparison.jpg differ
diff --git a/demo/gimp-painting/gallery/gpt-6.1-sol-2026-11-02/painting.jpg b/demo/gimp-painting/gallery/gpt-6.1-sol-2026-11-02/painting.jpg
new file mode 100644
index 0000000..7381620
Binary files /dev/null and b/demo/gimp-painting/gallery/gpt-6.1-sol-2026-11-02/painting.jpg differ
diff --git a/demo/gimp-painting/gallery/gpt-6.1-sol-2026-11-02/run.json b/demo/gimp-painting/gallery/gpt-6.1-sol-2026-11-02/run.json
new file mode 100644
index 0000000..43344d3
--- /dev/null
+++ b/demo/gimp-painting/gallery/gpt-6.1-sol-2026-11-02/run.json
@@ -0,0 +1,12 @@
+{
+ "id": "gpt-6.1-sol-2026-11-02",
+ "model": "openai-codex/gpt-6.1-sol",
+ "thinking": "high",
+ "date": "2026-11-02",
+ "minutes": 61,
+ "cost": 1.05,
+ "turns": 69,
+ "toolCalls": 1928,
+ "canvas": "1200x800",
+ "video": "https://github.com/user-attachments/assets/efce8341-6564-4a00-8a3a-ff8a6b349a15"
+}
diff --git a/demo/gimp-painting/gallery/gpt-6.1-terra-2026-11-02/comparison.jpg b/demo/gimp-painting/gallery/gpt-6.1-terra-2026-11-02/comparison.jpg
new file mode 100644
index 0000000..536b8e7
Binary files /dev/null and b/demo/gimp-painting/gallery/gpt-6.1-terra-2026-11-02/comparison.jpg differ
diff --git a/demo/gimp-painting/gallery/gpt-6.1-terra-2026-11-02/painting.jpg b/demo/gimp-painting/gallery/gpt-6.1-terra-2026-11-02/painting.jpg
new file mode 100644
index 0000000..8804596
Binary files /dev/null and b/demo/gimp-painting/gallery/gpt-6.1-terra-2026-11-02/painting.jpg differ
diff --git a/demo/gimp-painting/gallery/gpt-6.1-terra-2026-11-02/run.json b/demo/gimp-painting/gallery/gpt-6.1-terra-2026-11-02/run.json
new file mode 100644
index 0000000..72e5f85
--- /dev/null
+++ b/demo/gimp-painting/gallery/gpt-6.1-terra-2026-11-02/run.json
@@ -0,0 +1,12 @@
+{
+ "id": "gpt-6.1-terra-2026-11-02",
+ "model": "openai-codex/gpt-6.1-terra",
+ "thinking": "high",
+ "date": "2026-11-02",
+ "minutes": 14,
+ "cost": 0.25,
+ "turns": 44,
+ "toolCalls": 552,
+ "canvas": "1200x800",
+ "video": "https://github.com/user-attachments/assets/564263b1-ed17-42c3-919e-c76fa06d5fb8"
+}
diff --git a/demo/gimp-painting/postprocess.ts b/demo/gimp-painting/postprocess.ts
new file mode 100644
index 0000000..0ed981d
--- /dev/null
+++ b/demo/gimp-painting/postprocess.ts
@@ -0,0 +1,90 @@
+// Turn a demo recording into a sped-up video, a short animated preview and a reference/painting comparison.
+// Run it on its own to redo the outputs of an earlier run:
+// node demo/gimp-painting/postprocess.ts --ssh zinpc-benchmark --remote-dir 'C:\Users\user\pi-gimp-demo\<run>' --out .tmp/gimp-demo/<run>
+import { parseArgs } from "node:util";
+import path from "node:path";
+import { mkdir } from "node:fs/promises";
+import sharp from "sharp";
+import { findFfmpeg, remote, shQuote, wslPath, type Remote } from "./remote.ts";
+
+export interface PostprocessOptions {
+ /** Length of the full sped-up video in seconds. */
+ videoSeconds: number;
+ /** Length of the animated preview in seconds. */
+ previewSeconds: number;
+ /** Width of the reference picture-in-picture in the videos, in recording pixels; 0 leaves it out. */
+ pipWidth: number;
+}
+
+export const DEFAULT_POSTPROCESS: PostprocessOptions = { videoSeconds: 120, previewSeconds: 30, pipWidth: 360 };
+
+/**
+ * Encode `timelapse.mp4` and `preview.webp` from `recording.mkv` in the run's Windows folder with the machine's ffmpeg,
+ * copy them to `out`, and build `comparison.png` there from the reference and the exported painting.
+ */
+export async function postprocess(machine: Remote, remoteDir: string, out: string, options: PostprocessOptions = DEFAULT_POSTPROCESS): Promise<void> {
+ await mkdir(out, { recursive: true });
+ const ffmpeg = await findFfmpeg(machine);
+ const ffmpegWsl = await wslPath(machine, ffmpeg);
+ const ffprobeWsl = ffmpegWsl.replace(/ffmpeg\.exe$/i, "ffprobe.exe");
+ const dir = await wslPath(machine, remoteDir);
+ const run = (args: string, timeoutMs = 3_600_000) => machine.bash(`cd ${shQuote(dir)} && ${shQuote(ffmpegWsl)} -hide_banner -loglevel error -y ${args} </dev/null`, timeoutMs);
+
+ const duration = Number((await machine.bash(`cd ${shQuote(dir)} && ${shQuote(ffprobeWsl)} -v error -show_entries format=duration -of csv=p=0 recording.mkv </dev/null`)).trim());
+ if (!Number.isFinite(duration) || duration <= 0) throw new Error(`recording.mkv in ${remoteDir} has no readable duration.`);
+ const speed = (seconds: number) => Math.max(1, duration / seconds);
+ // The reference in a white frame in the bottom-right corner, so viewers see the target next to the painting.
+ const pip = (input: string, scale: number) => options.pipWidth > 0
+ ? `[1:v]scale=${Math.round(options.pipWidth * scale)}:-1,pad=iw+${Math.max(2, Math.round(6 * scale))}*2:ih+${Math.max(2, Math.round(6 * scale))}*2:(ow-iw)/2:(oh-ih)/2:white[pip];[${input}][pip]overlay=W-w-${Math.round(24 * scale)}:H-h-${Math.round(24 * scale)}`
+ : `[${input}]null`;
+ const inputs = `-i recording.mkv${options.pipWidth > 0 ? " -i reference.png" : ""}`;
+
+ const fullSpeed = speed(options.videoSeconds);
+ await run(`${inputs} -filter_complex ${shQuote(`[0:v]setpts=PTS/${fullSpeed},fps=30[base];${pip("base", 1)},tpad=stop_mode=clone:stop_duration=3,format=yuv420p[v]`)} -map [v] -c:v libx264 -preset slow -crf 20 -movflags +faststart timelapse.mp4`);
+
+ const previewSpeed = speed(options.previewSeconds);
+ // Small enough for a README: about 6 MB for 30 s.
+ const previewWidth = 800, previewScale = previewWidth / 1920;
+ await run(`${inputs} -filter_complex ${shQuote(`[0:v]setpts=PTS/${previewSpeed},fps=8,scale=${previewWidth}:-2:flags=lanczos[base];${pip("base", previewScale)},tpad=stop_mode=clone:stop_duration=2,format=bgra[v]`)} -map [v] -c:v libwebp -quality 50 -compression_level 6 -loop 0 preview.webp`);
+
+ for (const file of ["timelapse.mp4", "preview.webp", "painting.png", "reference.png"])
+ await machine.pull(`${dir}/${file}`, path.join(out, file)).catch(error => { if (file !== "painting.png") throw error; });
+ await comparison(path.join(out, "reference.png"), path.join(out, "painting.png"), path.join(out, "comparison.png"))
+ .catch(error => console.warn(`No comparison image: ${(error as Error).message}`));
+ console.log(`Recording ${Math.round(duration)} s; video ${options.videoSeconds} s at ${fullSpeed.toFixed(1)}x; preview ${options.previewSeconds} s at ${previewSpeed.toFixed(1)}x.`);
+}
+
+/** The reference and the painting side by side at the painting's size, each with a caption. */
+export async function comparison(referenceFile: string, paintingFile: string, outFile: string): Promise<void> {
+ const painting = sharp(paintingFile);
+ const { width = 0, height = 0 } = await painting.metadata();
+ const reference = await sharp(referenceFile).resize(width, height, { fit: "fill" }).png().toBuffer();
+ const gap = 32, caption = 56, margin = 32;
+ const total = { width: margin * 2 + width * 2 + gap, height: margin * 2 + height + caption };
+ const label = (text: string, x: number) =>
+ `<text x="${x + width / 2}" y="${margin + height + 40}" font-family="sans-serif" font-size="26" fill="#333" text-anchor="middle">${text}</text>`;
+ const svg = Buffer.from(`<svg xmlns="http://www.w3.org/2000/svg" width="${total.width}" height="${total.height}">${label("Reference", margin)}${label("Painted by the agent in GIMP", margin + width + gap)}</svg>`);
+ await sharp({ create: { ...total, channels: 3, background: "#ffffff" } })
+ .composite([
+ { input: reference, left: margin, top: margin },
+ { input: await painting.png().toBuffer(), left: margin + width + gap, top: margin },
+ { input: svg, left: 0, top: 0 },
+ ])
+ .png()
+ .toFile(outFile);
+}
+
+if (import.meta.url === `file://${process.argv[1]}`) {
+ const { values } = parseArgs({ options: {
+ ssh: { type: "string", default: "zinpc-benchmark" },
+ "remote-dir": { type: "string" },
+ out: { type: "string" },
+ "video-seconds": { type: "string", default: String(DEFAULT_POSTPROCESS.videoSeconds) },
+ "preview-seconds": { type: "string", default: String(DEFAULT_POSTPROCESS.previewSeconds) },
+ "pip-width": { type: "string", default: String(DEFAULT_POSTPROCESS.pipWidth) },
+ } });
+ if (!values["remote-dir"] || !values.out) throw new Error("Pass --remote-dir (the run's Windows folder) and --out (a local folder).");
+ await postprocess(remote(values.ssh), values["remote-dir"], values.out, {
+ videoSeconds: Number(values["video-seconds"]), previewSeconds: Number(values["preview-seconds"]), pipWidth: Number(values["pip-width"]),
+ });
+}
diff --git a/demo/gimp-painting/prompt.md b/demo/gimp-painting/prompt.md
new file mode 100644
index 0000000..302b443
--- /dev/null
+++ b/demo/gimp-painting/prompt.md
@@ -0,0 +1,7 @@
+Paint a copy of `reference.png` from your workspace in GIMP on the Windows machine `{{machine}}`. It is a copy of the repository's `assets/draw_task.png`: a cartoon robot at a computer, with the lettering "Oh! I can use a computer!". Follow the paint-from-reference skill: read it completely before you start, and use its setup, toolkit and stages.
+
+GIMP is already open and maximized there, with an empty {{canvasWidth}}×{{canvasHeight}} canvas named `canvas.png`. Use that window; don't open another GIMP or another image. The screen is being recorded for a time-lapse, so keep GIMP maximized and in front, and don't open other apps on that machine.
+
+This is a demonstration of what an agent can do with computer use, and you are expected to paint the whole picture yourself with GIMP's tools. A recognizable, stylized copy is a success: the same subject, composition, colours, sketchy style and lettering. Don't stop at a rough block-in, and don't end the task without a finished painting. Take the time you need. Nobody answers questions during this run, so make your own decisions.
+
+When the painting is finished, export it as PNG to `{{exportPath}}`: press `shift+ctrl+e`; in the "Export Image" window, triple-click the Name field, press `ctrl+a`, type that full path and press Enter; then find the "Export Image as PNG" window and click its Export button. Check that `{{exportPath}}` exists, for example with `computer_files` or by the title bar now naming it, then stop.
diff --git a/demo/gimp-painting/remote.ts b/demo/gimp-painting/remote.ts
new file mode 100644
index 0000000..6364062
--- /dev/null
+++ b/demo/gimp-painting/remote.ts
@@ -0,0 +1,56 @@
+// Helpers for the demo runner: commands on the painting machine's Windows, reached over SSH into its WSL.
+import { spawn } from "node:child_process";
+import { powershellOf, runPowerShell, sshRunner, SSH_OPTIONS, type CommandRunner } from "../../src/runner.ts";
+
+export interface Remote {
+ alias: string;
+ runner: CommandRunner;
+ /** Run a Bash script in the machine's WSL and return its stdout. */
+ bash(script: string, timeoutMs?: number): Promise<string>;
+ /** Run a PowerShell script on the machine's Windows and return its trimmed stdout. */
+ powershell(script: string, timeoutMs?: number): Promise<string>;
+ /** Copy a file from the machine's WSL to this machine. */
+ pull(remotePath: string, localPath: string): Promise<void>;
+ /** Copy a file from this machine to the machine's WSL. */
+ push(localPath: string, remotePath: string): Promise<void>;
+}
+
+export function remote(alias: string): Remote {
+ const runner = sshRunner(alias);
+ const scp = (from: string, to: string) => new Promise<void>((resolve, reject) => {
+ const child = spawn("scp", ["-q", ...SSH_OPTIONS.filter(option => option !== "-T"), from, to], { stdio: ["ignore", "ignore", "pipe"] });
+ let error = "";
+ child.stderr.on("data", chunk => { error += chunk; });
+ child.on("error", reject);
+ child.on("close", code => code === 0 ? resolve() : reject(new Error(`scp ${from} ${to} failed: ${error.trim()}`)));
+ });
+ return {
+ alias,
+ runner,
+ bash: (script, timeoutMs = 60_000) => runner.run("bash", ["-c", script], { timeoutMs }),
+ powershell: (script, timeoutMs = 60_000) => runPowerShell(runner, powershellOf(runner), script, { timeoutMs, scriptOnStdin: true }),
+ pull: (remotePath, localPath) => scp(`${alias}:${remotePath}`, localPath),
+ push: (localPath, remotePath) => scp(localPath, `${alias}:${remotePath}`),
+ };
+}
+
+/** Quote a value for a PowerShell single-quoted string. */
+export const psQuote = (value: string) => `'${value.replace(/'/g, "''")}'`;
+/** Quote a value for Bash. */
+export const shQuote = (value: string) => `'${value.replace(/'/g, `'\\''`)}'`;
+
+/** Find the newest ffmpeg.exe on the machine: on PATH or installed by winget (Gyan.FFmpeg). Returns its Windows path. */
+export async function findFfmpeg(machine: Remote): Promise<string> {
+ const found = await machine.powershell(String.raw`
+$command = Get-Command ffmpeg.exe -ErrorAction SilentlyContinue | Select-Object -First 1
+if ($command) { $command.Source; return }
+$candidate = Get-ChildItem "$env:LOCALAPPDATA\Microsoft\WinGet\Packages\Gyan.FFmpeg*\*\bin\ffmpeg.exe" -ErrorAction SilentlyContinue | Sort-Object LastWriteTime -Descending | Select-Object -First 1
+if ($candidate) { $candidate.FullName }`);
+ if (!found) throw new Error(`ffmpeg is missing on ${machine.alias}. Install it there with: winget install Gyan.FFmpeg`);
+ return found;
+}
+
+/** A Windows path as the machine's WSL sees it. */
+export async function wslPath(machine: Remote, windowsPath: string): Promise<string> {
+ return (await machine.bash(`wslpath -u ${shQuote(windowsPath)}`)).trim();
+}
diff --git a/demo/gimp-painting/run.ts b/demo/gimp-painting/run.ts
new file mode 100644
index 0000000..1550845
--- /dev/null
+++ b/demo/gimp-painting/run.ts
@@ -0,0 +1,240 @@
+// Record a Pi agent painting a reference picture in GIMP on another Windows machine, then make a time-lapse of it.
+// node demo/gimp-painting/run.ts [--ssh zinpc-benchmark] [--machine zinpc] [--model openai-codex/gpt-6.1-terra]
+// See demo/gimp-painting/README.md.
+import { spawn, type ChildProcess } from "node:child_process";
+import { createWriteStream } from "node:fs";
+import { copyFile, mkdir, readdir, readFile, writeFile } from "node:fs/promises";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+import { parseArgs } from "node:util";
+import sharp from "sharp";
+import { displayModes, type DisplayModeChange } from "../../src/display-mode.ts";
+import { DEFAULT_POSTPROCESS, postprocess } from "./postprocess.ts";
+import { findFfmpeg, psQuote, remote, wslPath } from "./remote.ts";
+
+const here = path.dirname(fileURLToPath(import.meta.url));
+const repo = path.resolve(here, "../..");
+const { values } = parseArgs({ options: {
+ ssh: { type: "string", default: "zinpc-benchmark" },
+ machine: { type: "string", default: "zinpc" },
+ model: { type: "string", default: "openai-codex/gpt-6.1-terra" },
+ thinking: { type: "string", default: "high" },
+ reference: { type: "string", default: path.join(repo, "assets/draw_task.png") },
+ canvas: { type: "string", default: "1200x800" },
+ display: { type: "string", default: "1920x1080" },
+ "max-minutes": { type: "string", default: "240" },
+ // Prepare the machine and open GIMP, then stop: no recording, no agent, nothing restored. For trying the skill by hand.
+ "prepare-only": { type: "boolean", default: false },
+ out: { type: "string" },
+ "video-seconds": { type: "string", default: String(DEFAULT_POSTPROCESS.videoSeconds) },
+ "preview-seconds": { type: "string", default: String(DEFAULT_POSTPROCESS.previewSeconds) },
+} });
+
+const size = (value: string) => {
+ const match = /^(\d+)x(\d+)$/.exec(value);
+ if (!match) throw new Error(`Expected WIDTHxHEIGHT, got ${value}.`);
+ return { width: Number(match[1]), height: Number(match[2]) };
+};
+const canvas = size(values.canvas);
+const display = size(values.display);
+const runId = new Date().toISOString().replace(/[:.]/g, "-");
+const out = path.resolve(values.out ?? path.join(repo, ".tmp/gimp-demo", runId));
+const workspace = path.join(out, "workspace");
+const machine = remote(values.ssh);
+const log = (message: string) => console.log(`[${new Date().toLocaleTimeString()}] ${message}`);
+const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms));
+
+/** The user's Windows session as `query session` shows it from WSL: its name (console, rdp-tcp#N or empty) and state. */
+async function windowsSession(): Promise<{ name: string; state: string }> {
+ const line = (await machine.bash("/mnt/c/Windows/System32/query.exe session </dev/null || true")).split("\n").find(row => row.startsWith(">"));
+ if (!line) throw new Error("query session shows no current session.");
+ const fields = line.slice(1).trim().split(/\s+/);
+ // A disconnected session has no session name: "> user 1 Disc".
+ return fields.length >= 4 ? { name: fields[0]!, state: fields[3]! } : { name: "", state: fields[2] ?? "" };
+}
+
+/** Make the user's session the physical console: Remote Desktop sessions take their resolution from the client. */
+async function moveToConsole(): Promise<void> {
+ let session = await windowsSession();
+ if (session.name === "console" && session.state === "Active") return;
+ if (!session.name.startsWith("rdp-tcp")) throw new Error(`The Windows session on ${machine.alias} is ${session.state || "unknown"}; sign in on its console or connect over Remote Desktop first.`);
+ log("Disconnecting Remote Desktop so the session moves to the physical console…");
+ await machine.bash("/mnt/c/Windows/System32/tsdiscon.exe </dev/null");
+ for (let attempt = 0; attempt < 30; attempt++) {
+ await sleep(2_000);
+ session = await windowsSession();
+ if (session.name === "console" && session.state === "Active") return;
+ }
+ throw new Error("The session did not return to the console within a minute. Is the PiAgents\\RdpToConsole task installed? Reconnect over Remote Desktop and try again.");
+}
+
+const DPI_AWARE = `Add-Type -Name D -Namespace PiDemo -MemberDefinition '[DllImport("user32.dll")] public static extern bool SetProcessDPIAware(); [DllImport("user32.dll")] public static extern IntPtr MonitorFromPoint(long point, int flags);'; [void][PiDemo.D]::SetProcessDPIAware(); Add-Type -AssemblyName System.Windows.Forms`;
+
+async function primaryMonitor(): Promise<string> {
+ // MONITOR_DEFAULTTOPRIMARY for the point 0,0.
+ return machine.powershell(`${DPI_AWARE}; [PiDemo.D]::MonitorFromPoint(0, 1).ToInt64()`);
+}
+
+/** The primary display's working area (the screen without the taskbar) in physical pixels. */
+async function workingArea(): Promise<{ x: number; y: number; width: number; height: number }> {
+ const area = JSON.parse(await machine.powershell(`${DPI_AWARE}; $a = [System.Windows.Forms.Screen]::PrimaryScreen.WorkingArea; @{ x = $a.X; y = $a.Y; width = $a.Width; height = $a.Height } | ConvertTo-Json -Compress`));
+ // yuv420p needs even sizes.
+ return { x: area.x, y: area.y, width: area.width & ~1, height: area.height & ~1 };
+}
+
+/** Start GIMP with the canvas, wait for its window and maximize it. Returns the process IDs it runs as. */
+async function openGimp(canvasPath: string): Promise<number[]> {
+ const running = await machine.powershell(`@(Get-Process gimp* -ErrorAction SilentlyContinue).Count`);
+ if (running !== "0") throw new Error(`GIMP is already running on ${machine.alias}. Close it first: it would open the canvas in its existing window.`);
+ const result = await machine.powershell(String.raw`
+${DPI_AWARE}
+Add-Type -Name W -Namespace PiDemo -MemberDefinition '[DllImport("user32.dll")] public static extern bool ShowWindow(IntPtr hwnd, int command); [DllImport("user32.dll")] public static extern IntPtr GetForegroundWindow(); [DllImport("user32.dll", CharSet = CharSet.Unicode)] public static extern IntPtr LoadKeyboardLayout(string id, uint flags); [DllImport("user32.dll")] public static extern bool PostMessage(IntPtr hwnd, uint message, IntPtr wParam, IntPtr lParam);'
+$exe = @("$env:LOCALAPPDATA\Programs\GIMP 3\bin\gimp-3.exe", "$env:ProgramFiles\GIMP 3\bin\gimp-3.exe") | Where-Object { Test-Path $_ } | Select-Object -First 1
+if (-not $exe) { throw "GIMP 3 is not installed. Install it with: winget install GIMP.GIMP.3" }
+# GIMP keeps separate tool options for every input device, so a pen stroke would ignore the brush size and hardness
+# the agent set with the mouse. Share them for this start only: GIMP reads gimprc at startup, and the user's file is
+# put back once the window is up.
+$rc = Get-ChildItem "$env:APPDATA\GIMP" -Directory -ErrorAction SilentlyContinue | Where-Object Name -like '3.*' | Sort-Object Name -Descending | Select-Object -First 1 | ForEach-Object { Join-Path $_.FullName 'gimprc' }
+$original = if ($rc -and (Test-Path $rc)) { [IO.File]::ReadAllText($rc) } else { $null }
+if ($rc) { [IO.File]::WriteAllText($rc, "$original" + [Environment]::NewLine + "(devices-share-tool yes)" + [Environment]::NewLine) }
+$started = Get-Date
+try {
+Start-Process -FilePath $exe -ArgumentList ('"' + ${psQuote(canvasPath)} + '"') | Out-Null
+$deadline = (Get-Date).AddSeconds(180)
+do {
+ Start-Sleep -Milliseconds 500
+ $window = Get-Process gimp* -ErrorAction SilentlyContinue | Where-Object { $_.MainWindowTitle -like '*canvas*GIMP*' } | Select-Object -First 1
+} while (-not $window -and (Get-Date) -lt $deadline)
+if (-not $window) { throw "GIMP did not show the canvas within three minutes." }
+Start-Sleep -Seconds 3
+} finally {
+ if ($rc) { if ($null -eq $original) { Remove-Item $rc -ErrorAction SilentlyContinue } else { [IO.File]::WriteAllText($rc, $original) } }
+}
+# GTK matches shortcuts such as ctrl+comma by the characters of the window's keyboard layout, so they do nothing under
+# a Cyrillic layout. Switch GIMP's window to US English (WM_INPUTLANGCHANGEREQUEST).
+[void][PiDemo.W]::PostMessage($window.MainWindowHandle, 0x50, [IntPtr]::Zero, [PiDemo.W]::LoadKeyboardLayout("00000409", 0))
+# Windows refuses SetForegroundWindow from a background process, and after a switch to the console a hidden service
+# window can keep the foreground, so the agent's own activation would be refused too. Restoring a minimized window
+# activates it.
+foreach ($command in 6, 9, 3) { [void][PiDemo.W]::ShowWindow($window.MainWindowHandle, $command); Start-Sleep -Milliseconds 500 }
+if ([PiDemo.W]::GetForegroundWindow() -ne $window.MainWindowHandle) { throw "GIMP did not become the foreground window." }
+(@(Get-Process gimp* | Where-Object { $_.StartTime -ge $started.AddSeconds(-1) }).Id) -join ','`, 240_000).catch(async error => {
+ // No GIMP ran before, so every GIMP process now is this run's.
+ await machine.powershell("Get-Process gimp* -ErrorAction SilentlyContinue | Stop-Process -Force").catch(() => {});
+ throw error;
+ });
+ return result.split(",").map(Number).filter(Number.isSafeInteger);
+}
+
+/** Record the working area of the primary display with ffmpeg until `stop`. */
+async function startRecording(ffmpeg: string, remoteDir: string, crop: { x: number; y: number; width: number; height: number }) {
+ const ffmpegWsl = await wslPath(machine, ffmpeg);
+ const encoders = await machine.bash(`'${ffmpegWsl}' -hide_banner -encoders </dev/null`);
+ const encoder = encoders.includes("h264_nvenc") ? ["-c:v", "h264_nvenc", "-preset", "p5", "-cq", "23"] : ["-c:v", "libx264", "-preset", "veryfast", "-crf", "23"];
+ // Windows Graphics Capture: Desktop Duplication (ddagrab) returns washed-out colours when the monitor runs in HDR.
+ const filter = `gfxcapture=monitor_idx=0:max_framerate=10:capture_cursor=1,hwdownload,format=bgra,fps=10,crop=${crop.width}:${crop.height}:${crop.x}:${crop.y},format=yuv420p`;
+ // Matroska stays playable even if the recording is cut off.
+ const child = machine.runner.spawn(ffmpegWsl, ["-hide_banner", "-loglevel", "warning", "-y", "-filter_complex", filter, ...encoder, `${remoteDir}\\recording.mkv`]);
+ const recordingLog = createWriteStream(path.join(out, "recording.log"));
+ child.stderr.pipe(recordingLog);
+ const exited = new Promise<number | null>(resolve => child.on("close", resolve));
+ await sleep(3_000);
+ if (child.exitCode !== null) throw new Error(`ffmpeg stopped right away; see ${path.join(out, "recording.log")}.`);
+ return {
+ async stop() {
+ if (child.exitCode !== null) return;
+ // ffmpeg finishes the file when it reads q.
+ child.stdin.end("q");
+ const code = await Promise.race([exited, sleep(30_000).then(() => "timeout" as const)]);
+ if (code === "timeout") { child.kill(); log("ffmpeg did not stop on q; the recording may lack its last seconds."); }
+ },
+ };
+}
+
+/** Run Pi until it exits or the time limit passes. */
+function runAgent(prompt: string): Promise<number | null> {
+ const args = ["-p", "--mode", "json", "--model", values.model, "--thinking", values.thinking,
+ "--no-skills", "--skill", path.join(repo, "skills/computer-use"), "--skill", path.join(here, "skill"),
+ "--no-context-files", "--no-mcp", "--session-dir", path.join(out, "session"),
+ // Nobody answers during the run, and the picture must come from the agent's own strokes.
+ "--exclude-tools", "*ask_user*,*wait_for_answers*,*generate_image*,*replicate*,*teleport*,*plannotator*,*show_to_user*,*show_diff_to_user*,compact",
+ prompt];
+ const child: ChildProcess = spawn("pi", args, { cwd: workspace, stdio: ["ignore", "pipe", "pipe"] });
+ child.stdout!.pipe(createWriteStream(path.join(out, "events.jsonl")));
+ child.stderr!.pipe(createWriteStream(path.join(out, "pi.log")));
+ const limit = setTimeout(() => { log("Time limit reached; stopping the agent."); child.kill("SIGTERM"); setTimeout(() => child.kill("SIGKILL"), 15_000).unref(); },
+ Number(values["max-minutes"]) * 60_000);
+ return new Promise(resolve => child.on("close", code => { clearTimeout(limit); resolve(code); }));
+}
+
+const cleanups: (() => Promise<void>)[] = [];
+async function cleanup() {
+ for (const step of cleanups.reverse().splice(0)) await step().catch(error => log(`Cleanup step failed: ${(error as Error).message}`));
+}
+for (const signal of ["SIGINT", "SIGTERM"] as const) process.once(signal, () => { log(`${signal}: cleaning up…`); void cleanup().then(() => process.exit(130)); });
+
+await mkdir(path.join(workspace, ".pi"), { recursive: true });
+log(`Run folder: ${out}`);
+const profile = await machine.powershell("$env:USERPROFILE");
+const remoteDir = `${profile}\\pi-gimp-demo\\${runId}`;
+const remoteDirWsl = await wslPath(machine, profile).then(dir => `${dir}/pi-gimp-demo/${runId}`);
+await machine.bash(`mkdir -p '${remoteDirWsl}'`);
+
+// The agent's workspace: the reference and the machine it may use.
+await copyFile(values.reference, path.join(workspace, "reference.png"));
+await writeFile(path.join(workspace, ".pi/computer-use.json"), JSON.stringify({ machines: { [values.machine]: { ssh: values.ssh } } }, null, 2));
+const exportPath = `${remoteDir}\\painting.png`;
+const prompt = (await readFile(path.join(here, "prompt.md"), "utf8"))
+ .replaceAll("{{machine}}", values.machine).replaceAll("{{exportPath}}", exportPath)
+ .replaceAll("{{canvasWidth}}", String(canvas.width)).replaceAll("{{canvasHeight}}", String(canvas.height));
+await writeFile(path.join(out, "prompt.txt"), prompt);
+
+// The empty canvas and the reference for the video's picture-in-picture.
+await sharp({ create: { ...canvas, channels: 3, background: "#ffffff" } }).png().toFile(path.join(out, "canvas.png"));
+await machine.push(path.join(out, "canvas.png"), `${remoteDirWsl}/canvas.png`);
+await machine.push(values.reference, `${remoteDirWsl}/reference.png`);
+const ffmpeg = await findFfmpeg(machine);
+
+let agentExit: number | null = null;
+try {
+ await moveToConsole();
+ const monitor = await primaryMonitor();
+ const change: DisplayModeChange = await displayModes(machine.runner).set(monitor, display);
+ cleanups.push(async () => { log(`Restoring ${change.previous.width}x${change.previous.height}…`); await displayModes(machine.runner).restore(change.device, change.previous); });
+ log(`Display ${change.device} set to ${change.current.width}x${change.current.height}.`);
+ await sleep(3_000);
+
+ log("Opening GIMP…");
+ const gimp = await openGimp(`${remoteDir}\\canvas.png`);
+ if (values["prepare-only"]) {
+ log(`GIMP is open on a ${change.current.width}x${change.current.height} display; the reference is ${path.join(workspace, "reference.png")}. Restore the display yourself (it was ${change.previous.width}x${change.previous.height}) and close GIMP when done.`);
+ process.exit(0);
+ }
+ cleanups.push(async () => {
+ const exported = await machine.bash(`test -s '${remoteDirWsl}/painting.png' && echo yes || echo no`);
+ if (exported.trim() !== "yes") { log("No exported painting: GIMP stays open so you can save it by hand."); return; }
+ await machine.powershell(`Stop-Process -Id ${gimp.join(",")} -Force -ErrorAction SilentlyContinue`);
+ });
+
+ const area = await workingArea();
+ log(`Recording ${area.width}x${area.height} at ${area.x},${area.y} (the screen without the taskbar)…`);
+ const recording = await startRecording(ffmpeg, remoteDir, area);
+ cleanups.push(async () => { log("Stopping the recording…"); await recording.stop(); });
+
+ log(`Starting the agent (${values.model}, thinking ${values.thinking}, limit ${values["max-minutes"]} min)…`);
+ const started = Date.now();
+ agentExit = await runAgent(prompt);
+ log(`Agent finished with exit code ${agentExit} after ${Math.round((Date.now() - started) / 60_000)} min.`);
+} finally {
+ await cleanup();
+}
+
+log("Encoding the time-lapse…");
+await postprocess(machine, remoteDir, out, {
+ ...DEFAULT_POSTPROCESS, videoSeconds: Number(values["video-seconds"]), previewSeconds: Number(values["preview-seconds"]),
+});
+await machine.pull(`${remoteDirWsl}/recording.mkv`, path.join(out, "recording.mkv"));
+const outputs = (await readdir(out)).filter(name => /\.(mp4|webp|png|mkv|jsonl)$/.test(name) && name !== "canvas.png");
+log(`Done. Outputs in ${out}: ${outputs.join(", ")}.`);
+log(`The Windows session stays on the console; reconnect over Remote Desktop when you need it. Remote files: ${remoteDir}`);
+process.exitCode = agentExit === 0 ? 0 : 1;
diff --git a/demo/gimp-painting/skill/SKILL.md b/demo/gimp-painting/skill/SKILL.md
new file mode 100644
index 0000000..8ae037f
--- /dev/null
+++ b/demo/gimp-painting/skill/SKILL.md
@@ -0,0 +1,314 @@
+---
+name: paint-from-reference
+description: Use when asked to paint a copy of a reference picture in GIMP through computer use.
+---
+
+# Paint a reference picture in GIMP
+
+You are going to paint a copy of a picture in GIMP with the mouse and a pen, through computer use. You can do this. Nobody expects a pixel-perfect or photorealistic copy: the goal is a painting that anyone recognizes as the reference at a glance, made the way an illustrator copies a picture: measure, block in big flat shapes, refine them, shade, ink the lines, add details and lettering, and compare with the reference after every stage.
+
+Every technique below was tried on this exact setup, and the code in [Your toolkit](#your-toolkit) works as written. A painting built from filled shapes, soft shading and tapered pen lines is exactly what is wanted here. Saying you can't paint, describing the picture instead of painting it, or stopping after a rough block-in all count as failing the task. Plan for many scripts and a long session.
+
+## The reference
+
+The picture to paint is `reference.png` in your workspace, a copy of the repository's `assets/draw_task.png` (1536×1024). It is a loose, sketchy marker-and-gouache cartoon on warm off-white paper:
+
+- top left, hand-lettered black text: a big bold **"Oh!"** with an orange underline, and below it, tilted upward to the right, "I can use" and "a computer!" with another orange underline;
+- centre left, a cute round-headed white robot with big shiny dark eyes, an open smiling mouth with a pink tongue, pink cheeks and round grey ear pieces, leaning forward happily; one hand on a black mouse, the other on a keyboard;
+- behind the robot, a dark grey office chair; a soft grey swoosh in the background;
+- right, a large dark grey monitor seen at an angle from behind its left side, on a stand;
+- bottom, a warm brown wooden desk drawn with loose orange-brown strokes;
+- orange "excitement" rays above the robot's head and small orange marks beside its face.
+
+On the 1200×800 canvas, roughly: the lettering fills x 30–360, y 20–390; the head is a circle around (520, 330) with a radius of about 170; the monitor screen spans x 780–1170, y 100–560 and its stand reaches y 700; the desk covers the bottom from y 560. Measure everything yourself on the grid copy; these numbers only help you start.
+
+## Rules
+
+- Make every mark with GIMP's own painting and selection tools driven by computer use input: filled selections, brush and pencil strokes, pen strokes with pressure, the eraser.
+- Never open, paste or place the reference or any other image in GIMP, and never use GIMP scripting, plug-ins, filters that generate content, or the Text tool. The lettering is painted by hand.
+- Code may measure the reference: resize it, overlay a grid, crop and zoom into parts, sample colours, find where a colour is, compare it with your canvas. Code may also generate the points of a stroke or shape you planned, such as an ellipse, a smooth curve through your control points or a hatch pattern inside a region you chose. Code must not trace the reference into outlines for you: you decide every shape's points by looking.
+- Use only the GIMP window you were given. Don't open other apps on that machine.
+
+## Set up
+
+### 1. Find GIMP and make the canvas 1:1
+
+Search the machine's windows for `GIMP` and read the main window. Click an empty grey area of the image window outside the canvas, then press `1`: the zoom shown at the bottom becomes 100%, so one screen pixel is one canvas pixel.
+
+### 2. Make two viewports
+
+All input goes through two viewports, so points never depend on the size of your last screenshot:
+
+- `canvas`: exactly over the canvas. Points on it are canvas pixels.
+- `panel`: the toolbox and tool options on the left, x 0–190 and the window's full height.
+
+Find the canvas rectangle with code instead of guessing it from a picture. In Codemode, with `W` the GIMP window's address:
+
+```js
+const shot = await tools.read({ path: W, views: ['capture:{"granularity":"image","maxLongEdge":1920}'] });
+await tools.write({ path: "gimp-window.png", content: shot });
+```
+
+Then in Bash:
+
+```bash
+python3 - <<'PY'
+from collections import Counter
+from PIL import Image
+im = Image.open("gimp-window.png").convert("RGB"); px = im.load(); W, H = im.size
+x0, x1, y0, y1 = int(W * .12), int(W * .84), int(H * .10), int(H * .95)
+pad = Counter(px[x, y] for x in range(x0, x1, 7) for y in (y0 + 5, y1 - 5)).most_common(1)[0][0]
+pts = [(x, y) for y in range(y0, y1) for x in range(x0, x1) if max(abs(a - b) for a, b in zip(px[x, y], pad)) > 24]
+xs, ys = [p[0] for p in pts], [p[1] for p in pts]
+print(min(xs), min(ys), max(xs) - min(xs) + 1, max(ys) - min(ys) + 1)
+PY
+```
+
+It prints `x y width height` of the canvas in the pixels of that 1920-px read, which is now the window's latest image. Make the viewports right away, before any other read of the window:
+
+```js
+await tools.computer_view({ viewport: W + "/canvas", x, y, width, height });
+await tools.computer_view({ viewport: W + "/panel", x: 0, y: 0, width: 190, height: 1030 });
+```
+
+The canvas viewport's image must be the canvas size, such as 1200×800. If it is a pixel off, fix it. Read `W + "/panel"` once and check where the swatches and the Opacity, Size and Hardness fields are; [the toolkit](#your-toolkit) assumes the positions measured on this setup.
+
+If the zoom ever changes, press `1` and measure the canvas again: GIMP may have scrolled it, and the old viewport would be off.
+
+### 3. Prepare the reference
+
+```bash
+python3 <this skill's folder>/scripts/prepare_reference.py reference.png 1200x800 ref
+```
+
+Use the canvas size from the title bar. It writes, in `ref/`:
+
+- `ref.png`: the reference at canvas size. Its pixel coordinates are canvas coordinates.
+- `ref-grid.png`: the same with a line every 50 px and labels every 100 px. Measure points here.
+- `ref-flat.png`: the reference reduced to 14 flat colours. It shows the big colour areas to block in.
+- `palette.txt`: those colours as hex values, with their share and where they are.
+
+Look at all three. For a small part, crop and enlarge it with code and look again. Sample exact colours with code, for example the orange of the rays or the pink of the tongue, which may be missing from the palette.
+
+## Plan
+
+Write `plan.md` before the first stroke:
+
+- **Palette**: 10–16 hex colours, each with what it paints: paper, shadow grey, desk browns, robot cream and its shadows, monitor and chair greys, near-black for lines, pink, orange.
+- **Objects back to front**, each with its main shape as 8–40 points in canvas coordinates, its colour and its shadow side. Measure the points on `ref-grid.png`.
+- **Lettering**: each line's baseline start and end, letter height and slant.
+- **Stages** as below, ticked off as you go.
+
+Check the shapes before painting: draw your polygons over `ref.png` with PIL, look at the result and fix points that miss.
+
+## Paint in stages
+
+Paint back to front and big to small, so later shapes cover the rough edges of earlier ones. Group work by colour: changing the colour costs four actions.
+
+1. **Paper**: fill the whole canvas with the paper colour.
+2. **Background**: the grey swoosh and other soft areas behind the subject, as filled shapes, then softened with a big soft brush at low opacity.
+3. **Large masses in flat mid-tones**: monitor, stand, chair, desk, robot body, head. One filled shape each; use the shape's dominant colour from `ref-flat.png`.
+4. **Medium shapes**: arms, hands, ear pieces, keyboard, mouse, monitor back panel, the eyes and mouth as dark shapes.
+5. **Shading and light**: select a shape again and paint inside the selection with a soft brush, so the paint stays inside: shadow sides, the darker underside of the head, highlights. Then the sketchy texture: hatching and dry strokes in the darker tone, especially on the monitor, chair and desk.
+6. **Line art**: near-black tapered pen lines around every shape and along inner edges, as in the reference. Break a long contour into several overlapping strokes the way a hand would, and let them wobble a little.
+7. **Details**: eye highlights, tongue, cheeks, keyboard keys, fingers.
+8. **Lettering**: "Oh!", "I can use", "a computer!" and the orange underlines.
+9. **Accents**: the orange rays and marks, the orange-brown desk strokes, white highlights.
+
+After each stage, compare with the reference (see [Review](#review)) and fix the three largest differences before you go on.
+
+## Your toolkit
+
+Paste this at the top of each painting script in Codemode and set `W` to the GIMP window's address. Every helper checks its receipt and throws at the first refusal or unknown outcome, so a script stops instead of painting blind. Then read the canvas viewport and decide what to do.
+
+```js
+const W = "computer:<machine>/<gimp window>";
+const V = W + "/canvas", P = W + "/panel";
+const MACHINE = W.split("/")[0].replace("computer:", "computer:machine/");
+// Measured on this setup (GIMP 3.2 at 1920x1080, 100% scale). Check them once by reading the panel and the dialog.
+const SWATCH = { x: 84, y: 233 }; // foreground swatch, in the panel viewport
+const FIELD = { opacity: 366, size: 444, hardness: 564 }; // field rows in the panel viewport, at x 80
+const DIALOG = { hex: { x: 440, y: 333 }, ok: { x: 432, y: 464 } }; // in the "Change Foreground Color" window
+
+async function ok(call, what) {
+ const r = await call;
+ if (r.outcome !== "delivered" && r.outcome !== "observed") throw new Error(`${what}: ${r.outcome} ${r.reason ?? ""}`);
+ return r;
+}
+const key = k => ok(tools.computer_press({ window: W, key: k }), "key " + k);
+async function findWindow(title, tries = 15) {
+ for (let i = 0; i < tries; i++) {
+ const found = await tools.search({ path: MACHINE + "/windows", query: title });
+ const address = /computer:[^\s|]+\/w\w+/.exec(String(found.result));
+ if (address) return address[0];
+ }
+ return undefined;
+}
+// Foreground colour, such as "e8a33c".
+async function color(hex) {
+ // A dialog left open by a failed script covers the swatch: use it.
+ let D = await findWindow("Change Foreground Color", 1);
+ if (!D) {
+ await ok(tools.computer_click({ window: P, ...SWATCH }), "swatch");
+ D = await findWindow("Change Foreground Color");
+ if (!D) throw new Error("The colour dialog did not open.");
+ }
+ await ok(tools.computer_click({ window: D, ...DIALOG.hex, count: 3 }), "hex field");
+ await ok(tools.computer_type({ window: D, text: hex.replace("#", ""), submit: true }), "hex");
+ await ok(tools.computer_click({ window: D, ...DIALOG.ok }), "OK");
+}
+// Paintbrush option: field("size", 12), field("opacity", 40), field("hardness", 100).
+async function field(name, value) {
+ await key("p"); // the fields belong to the painting tools; with a selection tool active they aren't there
+ await ok(tools.computer_click({ window: P, x: 80, y: FIELD[name], count: 3 }), name);
+ await ok(tools.computer_type({ window: P, text: String(value), submit: true }), name + " value");
+ await key("Escape"); // gives the keyboard back to the canvas; otherwise the next keys go into the field
+}
+const CANVAS = { width: 1200, height: 800 }; // the canvas size from the title bar
+// Whole pixels inside the canvas: a point on x = width or y = height is outside the viewport and refused.
+const round = p => ({ x: Math.min(CANVAS.width - 1, Math.max(0, Math.round(p.x))), y: Math.min(CANVAS.height - 1, Math.max(0, Math.round(p.y))) });
+// Points every `step` px along a polyline, so strokes are smooth and the taper spreads over the whole stroke.
+function densify(points, step = 4) {
+ const out = [points[0]];
+ for (let i = 1; i < points.length; i++) {
+ const a = points[i - 1], b = points[i], n = Math.max(1, Math.ceil(Math.hypot(b.x - a.x, b.y - a.y) / step));
+ for (let k = 1; k <= n; k++) out.push({ x: a.x + (b.x - a.x) * k / n, y: a.y + (b.y - a.y) * k / n });
+ }
+ return out;
+}
+// Polygon selection through canvas points; it replaces the current selection.
+async function select(points) {
+ await key("f");
+ const events = []; let t = 0;
+ for (const p of [...points, points[0]].map(round)) {
+ events.push({ at_ms: t, type: "move", x: p.x, y: p.y }, { at_ms: t + 40, type: "button_down" }, { at_ms: t + 80, type: "button_up" });
+ t += 150;
+ }
+ await ok(tools.computer_timeline({ window: V, events }), "select");
+ await key("Enter");
+}
+// Filled shape in the foreground colour.
+async function fill(points) { await select(points); await key("ctrl+comma"); await key("ctrl+shift+a"); }
+// Pen strokes with the paintbrush: a list of point lists. Pressure tapers each end unless taper is false.
+async function strokes(list, { taper = true, pressure = 1, msPerPoint = 12 } = {}) {
+ await key("p");
+ const events = []; let t = 0;
+ for (const raw of list) {
+ const pts = densify(raw).map(round);
+ pts.forEach((p, i) => {
+ const f = pts.length > 1 ? i / (pts.length - 1) : 0.5;
+ const pr = taper ? Math.max(0.15, Math.min(1, Math.sin(Math.PI * f) * 1.3)) * pressure : pressure;
+ events.push({ at_ms: t, type: i ? "pen_move" : "pen_down", x: p.x, y: p.y, pressure: pr });
+ t += msPerPoint;
+ });
+ const last = pts[pts.length - 1];
+ events.push({ at_ms: t, type: "pen_up", x: last.x, y: last.y });
+ t += 40;
+ }
+ await ok(tools.computer_timeline({ window: V, events }), "strokes");
+}
+// Shapes and curves as point lists. They keep fractions, so a letter can be designed in a unit box and scaled.
+const ellipse = (cx, cy, rx, ry, n = 48, rotation = 0) => Array.from({ length: n }, (_, i) => {
+ const a = 2 * Math.PI * i / n, x = rx * Math.cos(a), y = ry * Math.sin(a);
+ return { x: cx + x * Math.cos(rotation) - y * Math.sin(rotation), y: cy + x * Math.sin(rotation) + y * Math.cos(rotation) };
+});
+// A smooth curve through control points (Catmull-Rom), `step` points per segment.
+function curve(control, step = 12) {
+ const out = [], c = [control[0], ...control, control[control.length - 1]];
+ for (let i = 1; i < c.length - 2; i++) for (let s = 0; s < step; s++) {
+ const t = s / step, [p0, p1, p2, p3] = [c[i - 1], c[i], c[i + 1], c[i + 2]];
+ const f = (a, b, d, e) => 0.5 * (2 * b + (-a + d) * t + (2 * a - 5 * b + 4 * d - e) * t * t + (-a + 3 * b - 3 * d + e) * t * t * t);
+ out.push({ x: f(p0.x, p1.x, p2.x, p3.x), y: f(p0.y, p1.y, p2.y, p3.y) });
+ }
+ out.push(control[control.length - 1]);
+ return out;
+}
+// Slight hand wobble for a sketchy line.
+const wobble = (pts, amount = 1.5) => pts.map(p => ({ x: p.x + (Math.random() - 0.5) * 2 * amount, y: p.y + (Math.random() - 0.5) * 2 * amount }));
+// Place a shape designed in a unit box: scale it, rotate it by `angle` radians and move it to x, y.
+const place = (pts, x, y, scale, angle = 0) => pts.map(p => ({
+ x: x + scale * (p.x * Math.cos(angle) - p.y * Math.sin(angle)), y: y + scale * (p.x * Math.sin(angle) + p.y * Math.cos(angle)) }));
+// Save what the canvas looks like now, for compare.py.
+async function snapshot(file = "snapshot.png") { await tools.write({ path: file, content: await tools.read({ path: V }) }); }
+```
+
+A typical script then paints one object or one colour:
+
+```js
+await color("3f3a38");
+await fill(curve([{ x: 830, y: 105 }, { x: 1160, y: 230 }, { x: 1150, y: 520 }, { x: 790, y: 555 }, { x: 830, y: 105 }], 10));
+await snapshot();
+```
+
+### What each technique is for
+
+- **Flat shapes** (`fill`): every block-in. Use 8–40 points; for curved edges, put a few control points through `curve` and fill the result. The shape must not cross itself. Shapes may overlap: the later one covers the earlier one.
+- **Painting inside a shape**: `select(points)` with the shape's points, then `strokes` with a soft brush. Paint stays inside the selection, so shading can't spill out. `ctrl+shift+a` ends it.
+- **Soft shading and glow**: `field("hardness", 50)`, `field("opacity", 30–60)`, a big size such as 40–90, and a few broad strokes. Several light layers of strokes look better than one heavy one.
+- **Crisp line art**: `field("hardness", 100)`, `field("opacity", 100)`, size 4–8 for inner lines and 9–14 for outer contours, near-black. Use `strokes` with taper, each stroke 30–150 px long, overlapping its neighbours a little. Use `wobble` for the sketchy look, and draw some contours twice slightly apart, as the reference does.
+- **Hatching and dry texture**: size 5–7, opacity 60–80, many short parallel strokes, each just a start and an end point, with a little random spread. Generate them inside a region in one call: a whole hatching pass is one `strokes` call.
+- **Uniform strokes** without taper: `strokes(list, { taper: false })`, or `computer_drag` with `via` points on the canvas viewport.
+- **Hand lettering**: design each letter as one to three polylines in a unit box (x 0–0.6, y 0–1, baseline at y 1), with curves through `curve`. `place` scales, rotates and moves them along the line's slant, and `strokes` draws them tapered. "Oh!" is bold: size about 22 at a height of 120. The smaller lines need size 7–9 at a height of 50–60. Draw one word, look at it, then go on. This was checked on this canvas and gives legible marker lettering:
+
+ ```js
+ const G = {
+ O: [curve([{x:.35,y:0},{x:.05,y:.25},{x:.05,y:.75},{x:.35,y:1},{x:.62,y:.75},{x:.62,y:.25},{x:.35,y:0},{x:.15,y:.12}], 8)],
+ h: [[{x:.05,y:0},{x:.05,y:1}], curve([{x:.05,y:.6},{x:.25,y:.42},{x:.45,y:.52},{x:.47,y:1}], 8)],
+ "!": [[{x:.1,y:0},{x:.07,y:.72}], [{x:.06,y:.92},{x:.065,y:.97}]],
+ c: [curve([{x:.45,y:.5},{x:.25,y:.42},{x:.05,y:.7},{x:.25,y:1},{x:.47,y:.92}], 8)],
+ a: [curve([{x:.45,y:.55},{x:.25,y:.42},{x:.05,y:.7},{x:.22,y:1},{x:.45,y:.8}], 8), [{x:.45,y:.45},{x:.47,y:1}]],
+ n: [[{x:.05,y:.45},{x:.05,y:1}], curve([{x:.05,y:.62},{x:.25,y:.43},{x:.45,y:.55},{x:.46,y:1}], 8)],
+ // Design I, u, s, e, o, m, p, t, r the same way.
+ };
+ const advance = { O: .75, h: .6, "!": .25, c: .55, a: .58, n: .6, " ": .35 };
+ // Top-left corner x, y; letter height; slant in radians (negative rises to the right); brush size.
+ async function word(text, x, y, height, angle, brush) {
+ await field("size", brush);
+ const list = []; let pen = 0;
+ for (const ch of text) {
+ for (const s of G[ch] ?? []) list.push(wobble(place(s, x + Math.cos(angle) * pen * height, y + Math.sin(angle) * pen * height, height, angle), 0.8));
+ pen += advance[ch] ?? 0.5;
+ }
+ await strokes(list);
+ }
+ await word("Oh!", 60, 30, 120, -0.05, 22);
+ await word("can", 100, 230, 60, -0.35, 8);
+ ```
+- **Undo**: `ctrl+z` undoes the last stroke or fill. A wrong stroke is cheap; paint over it or undo it.
+
+### GIMP and computer use on this setup
+
+- Tool keys, with the canvas focused: `p` paintbrush, `n` pencil, `shift+e` eraser, `f` free select, `e` ellipse select, `r` rectangle select, `d` default black and white, `x` swap colours. The paint helpers press `p` themselves.
+- Digits change the zoom: `1` is 100%, `2` is 200%. A digit typed while no field has focus zooms instead of setting a value; check the zoom at the bottom of the window if the canvas looks wrong.
+- The tool option fields exist only for painting tools, so `field` selects the paintbrush first.
+- While the colour dialog is open it covers the toolbox, and a click on the swatch is refused. `color` uses a dialog that is already open and closes it with OK.
+- Dialogs appear a moment after the key or click that opens them; `findWindow` searches again until they do.
+- A helper that throws stops the script where it was. Read its reason and the canvas, then run only what is left. A refused input sent nothing and can be sent again after you checked why; an `unknown` one may have happened, so look before repeating it.
+- Use one script per object or colour, with up to a few hundred strokes. One `computer_timeline` call holds up to 5000 events and 120 seconds.
+- A Read of the canvas viewport returns 1200×800 pixels. Read it to look; return no images from scripts, only short text.
+
+## Review
+
+After each stage, and more often while refining:
+
+```js
+await snapshot("snapshot.png");
+```
+
+```bash
+python3 <this skill's folder>/scripts/compare.py ref/ref.png snapshot.png 50 compare.png
+```
+
+It prints the overall difference and the 50-px cells that differ most, with the reference's colour and yours. Look at `compare.png`: the reference, your canvas and a heat map side by side. Write down the three largest differences in position, shape, colour or a missing object, fix them, and compare again. Colour differences in big areas matter more than small details. On this reference, bare paper scores about 73; the number should drop with every stage.
+
+## Finish
+
+The painting is finished when:
+
+- every object from [the reference](#the-reference) is there, in the right place and colour;
+- no bare canvas shows where the reference has paint;
+- the shapes have line art, shading and the sketchy texture;
+- all three lines of lettering are legible;
+- the last comparison found only small differences.
+
+Then export as the prompt says. When you have time left, keep refining instead of stopping early.
diff --git a/demo/gimp-painting/skill/scripts/compare.py b/demo/gimp-painting/skill/scripts/compare.py
new file mode 100644
index 0000000..843c9c5
--- /dev/null
+++ b/demo/gimp-painting/skill/scripts/compare.py
@@ -0,0 +1,59 @@
+#!/usr/bin/env python3
+"""Compare a snapshot of the canvas with the reference and list where they differ most.
+
+Usage: python3 compare.py ref/ref.png snapshot.png [cell] [out.png]
+
+Both images are blurred a little and compared in cells (default 50 px). Prints the overall difference, then the
+cells that differ most with the reference's and the canvas's mean colour, merged into regions where neighbours
+differ too. Writes out.png (default compare.png): the reference, the canvas and a heat map side by side at half size.
+"""
+import sys
+
+from PIL import Image, ImageChops, ImageDraw, ImageFilter, ImageStat
+
+
+def mean(image: Image.Image, box: tuple[int, int, int, int]) -> tuple[int, int, int]:
+ return tuple(int(v) for v in ImageStat.Stat(image.crop(box)).mean[:3])
+
+
+def main() -> None:
+ ref = Image.open(sys.argv[1]).convert("RGB")
+ snap = Image.open(sys.argv[2]).convert("RGB")
+ cell = int(sys.argv[3]) if len(sys.argv) > 3 else 50
+ out = sys.argv[4] if len(sys.argv) > 4 else "compare.png"
+ if snap.size != ref.size:
+ print(f"Note: snapshot is {snap.size[0]}x{snap.size[1]}, reference {ref.size[0]}x{ref.size[1]}; the snapshot is resized.")
+ snap = snap.resize(ref.size, Image.LANCZOS)
+ width, height = ref.size
+ soft_ref = ref.filter(ImageFilter.GaussianBlur(4))
+ soft_snap = snap.filter(ImageFilter.GaussianBlur(4))
+ diff = ImageChops.difference(soft_ref, soft_snap).convert("L")
+
+ cells = []
+ for y in range(0, height, cell):
+ for x in range(0, width, cell):
+ box = (x, y, min(x + cell, width), min(y + cell, height))
+ cells.append((ImageStat.Stat(diff.crop(box)).mean[0], box))
+ overall = sum(score for score, _ in cells) / len(cells)
+ print(f"Mean difference {overall:.1f} (0 = identical, above 40 = very different). Cell {cell} px.")
+ print("Worst cells: box x0,y0-x1,y1 difference reference colour -> canvas colour")
+ for score, box in sorted(cells, key=lambda c: -c[0])[:12]:
+ r = mean(soft_ref, box)
+ s = mean(soft_snap, box)
+ print(f" {box[0]},{box[1]}-{box[2]},{box[3]} {score:5.1f} #{r[0]:02x}{r[1]:02x}{r[2]:02x} -> #{s[0]:02x}{s[1]:02x}{s[2]:02x}")
+
+ heat = Image.new("RGB", ref.size)
+ draw = ImageDraw.Draw(heat)
+ for score, box in cells:
+ level = min(255, int(score * 4))
+ draw.rectangle(box, fill=(level, 0, 255 - level))
+ heat = Image.blend(snap, heat, 0.55)
+ sheet = Image.new("RGB", (width * 3 // 2, height // 2), "white")
+ for index, image in enumerate((ref, snap, heat)):
+ sheet.paste(image.resize((width // 2, height // 2)), (index * width // 2, 0))
+ sheet.save(out)
+ print(f"Wrote {out}: reference | canvas | heat map (red = most different).")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/demo/gimp-painting/skill/scripts/prepare_reference.py b/demo/gimp-painting/skill/scripts/prepare_reference.py
new file mode 100644
index 0000000..4dc4546
--- /dev/null
+++ b/demo/gimp-painting/skill/scripts/prepare_reference.py
@@ -0,0 +1,83 @@
+#!/usr/bin/env python3
+"""Measuring aids for painting a reference: a copy at canvas size, a grid copy, a flat-colour copy and a palette.
+
+Usage: python3 prepare_reference.py reference.png 1200x800 [out_dir]
+
+Writes to out_dir (default: ref/):
+ ref.png the reference resized to the canvas: its pixel coordinates are canvas coordinates
+ ref-grid.png ref.png with a line every 50 px and labels every 100 px
+ ref-flat.png ref.png reduced to the palette colours: shows the big colour areas to block in
+ palette.txt the palette: hex colour, share of the picture and where it is mostly found
+"""
+import sys
+from pathlib import Path
+
+from PIL import Image, ImageDraw, ImageFilter
+
+COLOURS = 14
+
+
+def main() -> None:
+ source = Path(sys.argv[1])
+ width, height = (int(v) for v in sys.argv[2].lower().split("x"))
+ out = Path(sys.argv[3] if len(sys.argv) > 3 else "ref")
+ out.mkdir(parents=True, exist_ok=True)
+
+ ref = Image.open(source).convert("RGB").resize((width, height), Image.LANCZOS)
+ ref.save(out / "ref.png")
+
+ grid = ref.copy()
+ draw = ImageDraw.Draw(grid)
+ for x in range(0, width, 50):
+ draw.line([(x, 0), (x, height)], fill=(255, 0, 255) if x % 100 == 0 else (255, 160, 255), width=1)
+ for y in range(0, height, 50):
+ draw.line([(0, y), (width, y)], fill=(255, 0, 255) if y % 100 == 0 else (255, 160, 255), width=1)
+ for x in range(0, width, 100):
+ for y in range(0, height, 100):
+ draw.text((x + 3, y + 2), f"{x},{y}", fill=(160, 0, 160))
+ grid.save(out / "ref-grid.png")
+
+ # A light blur first, so hatching and paper texture merge into the areas they shade. Many candidate colours,
+ # then near-duplicates merged, so small accents such as an orange highlight keep their own entry.
+ soft = ref.filter(ImageFilter.GaussianBlur(2))
+ candidates = soft.quantize(colors=48, method=Image.Quantize.MEDIANCUT)
+ raw = candidates.getpalette()[: 48 * 3]
+ counts = sorted(((n, i) for n, i in candidates.getcolors(48 * 48)), reverse=True)
+ chosen: list[list[int]] = [] # [r, g, b, count]
+ for count, index in counts:
+ colour = raw[index * 3: index * 3 + 3]
+ near = next((c for c in chosen if sum((a - b) ** 2 for a, b in zip(c[:3], colour)) < 28 ** 2), None)
+ if near:
+ near[3] += count
+ elif len(chosen) < COLOURS:
+ chosen.append([*colour, count])
+ palette_image = Image.new("P", (1, 1))
+ palette_image.putpalette([v for c in chosen for v in c[:3]] + [0] * (768 - 3 * len(chosen)))
+ flat = soft.quantize(palette=palette_image, dither=Image.Dither.NONE)
+ flat.convert("RGB").save(out / "ref-flat.png")
+
+ pixels = flat.load()
+ count = [0] * len(chosen)
+ sums = [[0, 0] for _ in chosen]
+ for y in range(0, height, 2):
+ for x in range(0, width, 2):
+ index = pixels[x, y]
+ if index < len(chosen):
+ count[index] += 1
+ sums[index][0] += x
+ sums[index][1] += y
+ total = sum(count)
+ lines = []
+ for index in sorted(range(len(chosen)), key=lambda i: -count[i]):
+ if not count[index]:
+ continue
+ r, g, b = chosen[index][:3]
+ cx, cy = sums[index][0] // count[index], sums[index][1] // count[index]
+ lines.append(f"#{r:02x}{g:02x}{b:02x} {100 * count[index] / total:5.1f}% centre of mass {cx},{cy}")
+ (out / "palette.txt").write_text("\n".join(lines) + "\n")
+ print("\n".join(lines))
+ print(f"Wrote {out}/ref.png, ref-grid.png, ref-flat.png and palette.txt at {width}x{height}.")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/docs/native-observability.md b/docs/native-observability.md
index 7b80fde..d2b6810 100644
--- a/docs/native-observability.md
+++ b/docs/native-observability.md
@@ -1,28 +1,24 @@
# Native computer visuals
-`/computer-visuals` controls human-only presentation on the machine that executes computer use. Native visuals require a driver advertising both `set_agent_visuals` and `present_agent_activity`. The adapter uses that driver's renderer and theme. It starts no viewer, HTTP listener or separate overlay process.
+`/computer-settings` controls human-only presentation on the machine that executes computer use. Native visuals require a driver advertising both `set_agent_visuals` and `present_agent_activity`. The adapter uses that driver's renderer and theme. It starts no viewer, HTTP listener or separate overlay process.
A driver without these capabilities keeps the legacy cursor behavior described in [complete input](complete-input.md). A version number alone does not enable native presentation.
## Human controls
-Run `/computer-visuals` to show the current settings, or choose a preset:
+Open `/computer-settings` to view the current settings and choose a visual preset:
- `off` disables presentation and discards pending previews.
- `subtle` keeps activity cues and disables previews by default.
- `full` enables previews by default.
-A JSON patch controls individual categories and rendering preferences:
-
-```text
-/computer-visuals {"observations":true,"previews":false,"pointer":true,"scroll_drag":true,"keyboard":true,"typed_text":true,"opacity":0.8,"quality":"balanced","reduced_motion":false}
-```
+The panel also controls individual categories, opacity, quality and reduced motion.
Action cards show available key combinations and pointer details. Printable keys are shown only when a native observation identifies a non-password focused element; otherwise they are omitted unless part of a non-text shortcut. `keyboard` and `typed_text` must both be on for keyboard details and typed text. Text going into a field that UI Automation reports as a password is never sent to the renderer. Pointer and scroll details follow their respective visual settings. Control characters show as spaces, and only the last 200 characters are sent.
-`opacity` is 0..1. `quality` is `performance`, `balanced` or `high`. Preferences are saved on the Pi session branch; navigating to another branch restores its preferences. These commands belong to the human. They do not change Safe/YOLO, capture sizing, input delivery or permission checks.
+`opacity` is 0..1. `quality` is `performance`, `balanced` or `high`. Scoped visual preferences override older session-branch preferences. Where no scoped value exists, navigating to another branch restores that branch's preference. Visual settings do not change Safe/YOLO, capture sizing, input delivery or permission checks.
-`/computer-images` controls future captures in this session: `cap 64..4096`, `scale 0.05..1`, `hints on|off`, `guides off|grid|action|all`, or `reset`. Calling it without arguments shows the settings. An individual Read can still override its capture options. Existing images keep their bytes.
+The panel controls screenshot size, scale, hints and guides for future captures. Resetting them clears overrides in the selected scope. An individual Read can still override its capture options. Existing images keep their bytes.
## What a cue means
diff --git a/docs/observation-sources.md b/docs/observation-sources.md
index 1004553..a354813 100644
--- a/docs/observation-sources.md
+++ b/docs/observation-sources.md
@@ -26,7 +26,7 @@ The source envelope and exact returned bytes appear in existing Vision debug. Th
## Native settings and limits
-Desktop windows and displays use session-local settings: Lanczos3, default long-edge cap 1280 and scale 1. Settings are frozen before each capture. Overview pixels stay native. Only fresh display detail uses adapter crop/resize; no originals are retained and CUA may round below the requested edge.
+Desktop windows and displays use the configured screenshot settings: Lanczos3, default long-edge cap 1280 and scale 1. Settings are frozen before each capture. Overview pixels stay native. Only fresh display detail uses adapter crop/resize; no originals are retained and CUA may round below the requested edge.
Window and display routes use the selected qualified CUA runtime. Arbitrary window reads do not raise or focus that window. Neither route guarantees fresh paint.
diff --git a/images/gimp-demo/timelapse-preview.webp b/images/gimp-demo/timelapse-preview.webp
new file mode 100755
index 0000000..b13b3e8
Binary files /dev/null and b/images/gimp-demo/timelapse-preview.webp differ
diff --git a/package-lock.json b/package-lock.json
index 229c344..2cf6062 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -12,6 +12,7 @@
"@modelcontextprotocol/sdk": "1.32.2",
"pi-agent-ide": "^0.9.2",
"playwright-core": "1.64.1",
+ "proper-lockfile": "4.1.2",
"sharp": "0.35.5"
},
"devDependencies": {
@@ -19,6 +20,7 @@
"@types/cross-spawn": "6.0.6",
"@types/fs-extra": "^11.0.4",
"@types/node": "24.0.4",
+ "@types/proper-lockfile": "^4.1.4",
"pi-coding-agent-eval": "0.1.1",
"pi-coding-agent-test": "0.1.3",
"typescript": "5.9.3",
@@ -1078,35 +1080,6 @@
"node": "20 || >=22"
}
},
- "node_modules/@earendil-works/pi-coding-agent/node_modules/proper-lockfile": {
- "version": "4.1.2",
- "resolved": "https://registry.npmjs.org/proper-lockfile/-/proper-lockfile-4.1.2.tgz",
- "integrity": "sha512-TjNPblN4BwAWMXU8s9AEz4JmQxnD1NNL7bNOY/AKUzyamc379FWASUhc/K1pL2noVb+XmZKLL68cjzLsiOAMaA==",
- "dev": true,
- "license": "MIT",
- "dependencies": {
- "graceful-fs": "^4.2.4",
- "retry": "^0.12.0",
- "signal-exit": "^3.0.2"
- }
- },
- "node_modules/@earendil-works/pi-coding-agent/node_modules/proper-lockfile/node_modules/retry": {
- "version": "0.12.0",
- "resolved": "https://registry.npmjs.org/retry/-/retry-0.12.0.tgz",
- "integrity": "sha512-9LkiTwjUh6rT555DtE9rTX+BKByPfrMzEAtnlEtdEwr3Nkffwiihqe2bWADg+OQRjt9gl6ICdmB/ZFDCGAtSow==",
- "dev": true,
- "license": "MIT",
- "engines": {
- "node": ">= 4"
- }
- },
- "node_modules/@earendil-works/pi-coding-agent/node_modules/signal-exit": {
- "version": "3.0.7",
- "resolved": "https://registry.npmjs.org/signal-exit/-/signal-exit-3.0.7.tgz",
- "integrity": "sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ==",
- "dev": true,
- "license": "ISC"
- },
"node_modules/@earendil-works/pi-mcp": {
"version": "1.4.0",
"resolved": "https://registry.npmjs.org/@earendil-works/pi-mcp/-/pi-mcp-1.4.0.tgz",
@@ -3003,6 +2976,16 @@
"undici-types": "~7.8.0"
}
},
+ "node_modules/@types/proper-lockfile": {
+ "version": "4.1.4",
+ "resolved": "https://registry.npmjs.org/@types/proper-lockfile/-/proper-lockfile-4.1.4.tgz",
+ "integrity": "sha512-uo2ABllncSqg9F1D4nugVl9v93RmjxF6LJzQLMLDdPaXCUIDPeOJ21Gbqi43xNKzBi/WQ0Q0dICqufzQbMjipQ==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "@types/retry": "*"
+ }
+ },
"node_modules/@types/retry": {
"version": "0.12.0",
"resolved": "https://registry.npmjs.org/@types/retry/-/retry-0.12.0.tgz",
@@ -6056,6 +6039,26 @@
"node": "^10 || ^12 || >=14"
}
},
+ "node_modules/proper-lockfile": {
+ "version": "4.1.2",
+ "resolved": "https://registry.npmjs.org/proper-lockfile/-/proper-lockfile-4.1.2.tgz",
+ "integrity": "sha512-TjNPblN4BwAWMXU8s9AEz4JmQxnD1NNL7bNOY/AKUzyamc379FWASUhc/K1pL2noVb+XmZKLL68cjzLsiOAMaA==",
+ "license": "MIT",
+ "dependencies": {
+ "graceful-fs": "^4.2.4",
+ "retry": "^0.12.0",
+ "signal-exit": "^3.0.2"
+ }
+ },
+ "node_modules/proper-lockfile/node_modules/retry": {
+ "version": "0.12.0",
+ "resolved": "https://registry.npmjs.org/retry/-/retry-0.12.0.tgz",
+ "integrity": "sha512-9LkiTwjUh6rT555DtE9rTX+BKByPfrMzEAtnlEtdEwr3Nkffwiihqe2bWADg+OQRjt9gl6ICdmB/ZFDCGAtSow==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 4"
+ }
+ },
"node_modules/protobufjs": {
"version": "7.6.6",
"resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-7.6.6.tgz",
@@ -6474,6 +6477,12 @@
"dev": true,
"license": "ISC"
},
+ "node_modules/signal-exit": {
+ "version": "3.0.7",
+ "resolved": "https://registry.npmjs.org/signal-exit/-/signal-exit-3.0.7.tgz",
+ "integrity": "sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ==",
+ "license": "ISC"
+ },
"node_modules/source-map-js": {
"version": "1.2.2",
"resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.2.tgz",
diff --git a/package.json b/package.json
index 4e8e9d4..c6e2f3b 100644
--- a/package.json
+++ b/package.json
@@ -55,7 +55,8 @@
"skills": [
"./skills"
],
- "image": "https://raw.githubusercontent.com/alexshpunt/pi-agent-computer-use/develop/assets/banner.png"
+ "image": "https://raw.githubusercontent.com/alexshpunt/pi-agent-computer-use/develop/assets/banner.png",
+ "video": "https://github.com/alexshpunt/pi-agent-computer-use/releases/download/v0.1.2/gpt-6.1-sol-painting.mp4"
},
"engines": {
"node": ">=22.19.0"
@@ -72,6 +73,7 @@
"@modelcontextprotocol/sdk": "1.32.2",
"pi-agent-ide": "^0.9.2",
"playwright-core": "1.64.1",
+ "proper-lockfile": "4.1.2",
"sharp": "0.35.5"
},
"peerDependencies": {
@@ -85,6 +87,7 @@
"@types/cross-spawn": "6.0.6",
"@types/fs-extra": "^11.0.4",
"@types/node": "24.0.4",
+ "@types/proper-lockfile": "^4.1.4",
"pi-coding-agent-eval": "0.1.1",
"pi-coding-agent-test": "0.1.3",
"typescript": "5.9.3",
diff --git a/src/computer-settings.ts b/src/computer-settings.ts
new file mode 100644
index 0000000..7fe5cfe
--- /dev/null
+++ b/src/computer-settings.ts
@@ -0,0 +1,182 @@
+import { mkdir, readFile, rename, stat, unlink, writeFile } from "node:fs/promises";
+import path from "node:path";
+import { randomUUID } from "node:crypto";
+import type { ExtensionAPI, ExtensionCommandContext } from "@earendil-works/pi-coding-agent";
+import { getSettingsListTheme } from "@earendil-works/pi-coding-agent";
+import { Input, SettingsList, truncateToWidth, type SettingItem, type TuiMouseEvent } from "@earendil-works/pi-tui";
+import { CuaImageSettings } from "./cua-image-settings.ts";
+import { VisualSettings, type VisualOptions } from "./native-visuals.ts";
+import type { ComputerMode } from "./session.ts";
+import type { HumanStop } from "./human-stop.ts";
+import type { PreferencesController } from "./preferences-controller.ts";
+
+async function readConfig(cwd: string): Promise<Record<string, unknown>> {
+ let raw: unknown;
+ try { raw = JSON.parse(await readFile(path.join(cwd, ".pi", "computer-use.json"), "utf8")); }
+ catch (error) {
+ if ((error as NodeJS.ErrnoException).code === "ENOENT") return {};
+ throw new Error("Cannot read .pi/computer-use.json; settings were not changed.");
+ }
+ if (!raw || typeof raw !== "object" || Array.isArray(raw)) throw new Error("Invalid .pi/computer-use.json; settings were not changed.");
+ return raw as Record<string, unknown>;
+}
+
+export async function readLegacyEscapeStop(cwd: string): Promise<boolean | undefined> {
+ const value = (await readConfig(cwd)).escapeStop;
+ if (value !== undefined && typeof value !== "boolean") throw new Error("Invalid escapeStop in .pi/computer-use.json.");
+ return value as boolean | undefined;
+}
+
+export async function readEscapeStop(cwd: string): Promise<boolean> {
+ return (await readLegacyEscapeStop(cwd)) !== false;
+}
+
+export async function writeEscapeStop(cwd: string, value: boolean | undefined): Promise<void> {
+ const config = await readConfig(cwd);
+ if (config.escapeStop !== undefined && typeof config.escapeStop !== "boolean") throw new Error("Invalid escapeStop in .pi/computer-use.json.");
+ if (value === undefined && config.escapeStop === undefined) return;
+ if (value === undefined) delete config.escapeStop; else config.escapeStop = value;
+ const dir = path.join(cwd, ".pi");
+ const file = path.join(dir, "computer-use.json");
+ await mkdir(dir, { recursive: true });
+ const mode = await stat(file).then(info => info.mode & 0o777).catch((error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return 0o600; throw error; });
+ const temp = path.join(dir, `.computer-use-${randomUUID()}.tmp`);
+ try {
+ await writeFile(temp, `${JSON.stringify(config, null, 2)}\n`, { mode, flag: "wx" });
+ await rename(temp, file);
+ } finally { await unlink(temp).catch((error: NodeJS.ErrnoException) => { if (error.code !== "ENOENT") throw error; }); }
+}
+
+type ModeControl = (() => ComputerMode) & { select: (value: string, ctx: ExtensionCommandContext) => Promise<void> };
+type Options = { mode: ModeControl; images: CuaImageSettings; visuals: VisualSettings; stop: HumanStop; preferences?: PreferencesController };
+const onOff = (value: boolean) => value ? "on" : "off";
+const booleanValues = ["on", "off"];
+
+/** One human-facing view over the same stores used by the existing commands. */
+export function registerComputerSettingsCommand(pi: ExtensionAPI, options: Options): void {
+ pi.registerCommand("computer-settings", {
+ description: "Open Computer Use settings.",
+ handler: async (_args, ctx) => {
+ if (ctx.mode !== "tui") { ctx.ui.notify("Computer settings require an interactive terminal.", "error"); return; }
+ let escapeStop: boolean;
+ let scoped: Awaited<ReturnType<PreferencesController["read"]>> | undefined;
+ let scope: "project" | "global" = "project";
+ if (options.preferences) {
+ const choice = await ctx.ui.select("Computer Use settings", ctx.isProjectTrusted() ? ["Project", "Global"] : ["Global"]);
+ if (!choice) return;
+ scope = choice === "Project" ? "project" : "global";
+ }
+ try {
+ scoped = await options.preferences?.read(ctx);
+ escapeStop = scoped ? scoped.effective.escape ?? true : await readEscapeStop(ctx.cwd);
+ }
+ catch (error) { ctx.ui.notify(error instanceof Error ? error.message : String(error), "error"); return; }
+ await ctx.ui.custom<void>((tui, theme, _keys, done) => {
+ const image = options.images.settings();
+ const visual = options.visuals.current();
+ const defaultImage = new CuaImageSettings().settings();
+ const defaultValues = { mode: "yolo", escape: true, cap: defaultImage.maxLongEdge, scale: defaultImage.scale,
+ hints: true, grid: false, action: false, ...new VisualSettings().current() };
+ const numeric = (label: string): SettingItem["submenu"] => (current, finish) => {
+ const input = new Input({ prompt: `${label} (or inherit): ` });
+ input.setValue(current); input.handleInput("\u0005"); input.focused = true;
+ input.onSubmit = value => finish(value.trim());
+ input.onEscape = () => finish();
+ return input;
+ };
+ const items: SettingItem[] = [
+ { id: "mode", label: "Access mode", currentValue: options.mode(), values: ["yolo", "safe"], description: "yolo: access the desktop. safe: only session-opened windows. Safe is not an OS sandbox." },
+ { id: "resume", label: "Computer use", currentValue: options.stop.stopped ? "stopped" : "on", values: ["on"], description: "on: allow input. stopped: the physical Escape shortcut stopped input. Select on to resume." },
+ { id: "escape", label: "Triple Escape stop", currentValue: onOff(escapeStop), values: booleanValues, description: "on: three physical Escape presses stop input. off: disables this emergency stop. Restart Pi and reconnect machines after changing this setting." },
+ { id: "cap", label: "Screenshot long edge", currentValue: String(image.maxLongEdge), submenu: numeric("Long edge (64–4096)"), description: "64–4096 pixels: maximum long edge before scale is applied. Affects future captures." },
+ { id: "scale", label: "Screenshot scale", currentValue: String(image.scale), submenu: numeric("Scale (0.05–1)"), description: "0.05–1: multiplies the long-edge limit. 1 keeps the full limit; 0.5 halves it." },
+ { id: "hints", label: "Screenshot hints", currentValue: onOff(image.hints !== false), values: booleanValues, description: "on: add element labels to screenshots. off: omit them." },
+ { id: "grid", label: "Grid guide", currentValue: onOff(!!image.guides?.grid), values: booleanValues, description: "on: add a coordinate grid to screenshots. off: omit it." },
+ { id: "action", label: "Action guide", currentValue: onOff(!!image.guides?.action), values: booleanValues, description: "on: mark the action location on screenshots. off: omit the marker." },
+ { id: "preset", label: "Visual preset", currentValue: visual.preset, values: ["off", "subtle", "full"], description: "off: disable visuals. subtle: activity cues without previews by default. full: activity cues with previews by default." },
+ ...([ ["observations", "Observation frames", "on: show frames for observations. off: hide them."], ["previews", "Image previews", "on: show operation image previews. off: hide them."], ["pointer", "Pointer actions", "on: show pointer action details. off: hide them."], ["scroll_drag", "Scroll and drag", "on: show scroll and drag details. off: hide them."], ["keyboard", "Keyboard actions", "on: show keyboard action details. off: hide them."], ["typed_text", "Show typed text", "on: show typed text when keyboard details are enabled. off: hide it. Password fields stay private."], ["reduced_motion", "Reduced motion", "on: reduce visual animation. off: use normal animation."] ] as const).map(([id, label, description]) => ({ id, label, currentValue: onOff(visual[id]), values: booleanValues, description })),
+ { id: "quality", label: "Visual quality", currentValue: visual.quality, values: ["performance", "balanced", "high"], description: "performance: favor speed. balanced: balance speed and quality. high: favor visual quality." },
+ { id: "opacity", label: "Visual opacity", currentValue: String(visual.opacity), submenu: numeric("Opacity (0–1)"), description: "0–1: visual opacity. 0 is transparent; 1 is fully opaque." },
+ { id: "reset-images", label: "Reset screenshot settings", currentValue: "reset", values: ["reset"], description: "reset: clear screenshot overrides in the selected scope." },
+ ];
+ if (scoped) for (const item of items) {
+ if (item.id === "resume" || item.id === "reset-images") continue;
+ if (item.values) item.values = [...item.values, "inherit"];
+ }
+ const descriptions = new Map(items.map(item => [item.id, item.description ?? ""]));
+ let busy = false;
+ const sync = () => {
+ const current = options.images.settings(); const v = options.visuals.current();
+ list.updateValue("mode", options.mode()); list.updateValue("resume", options.stop.stopped ? "stopped" : "on");
+ list.updateValue("escape", onOff(escapeStop)); list.updateValue("cap", String(current.maxLongEdge)); list.updateValue("scale", String(current.scale));
+ list.updateValue("hints", onOff(current.hints !== false)); list.updateValue("grid", onOff(!!current.guides?.grid)); list.updateValue("action", onOff(!!current.guides?.action));
+ for (const key of ["preset", "quality", "opacity"] as const) list.updateValue(key, String(v[key]));
+ for (const key of ["observations", "previews", "pointer", "scroll_drag", "keyboard", "typed_text", "reduced_motion"] as const) list.updateValue(key, onOff(v[key]));
+ if (scoped) for (const item of items) {
+ if (["resume", "reset-images"].includes(item.id)) continue;
+ const key = item.id as keyof typeof scoped.sources;
+ const source = item.id === "mode" && scoped.effective.mode !== undefined && scoped.effective.mode !== options.mode() ? "session" : scoped.sources[key] ?? "default or session";
+ item.description = `${descriptions.get(item.id)} Effective: ${item.currentValue} (${source}). Select inherit to clear the ${scope} override.`;
+ const stored = scoped[scope][key];
+ const fallback = scope === "global" ? defaultValues[key] : undefined;
+ const value = stored ?? (scope === "global" && key === "previews" && scoped.global.preset === "subtle" ? false : fallback);
+ if (value !== undefined) list.updateValue(item.id, typeof value === "boolean" ? onOff(value) : String(value));
+ }
+ };
+ let unsubscribe = () => {};
+ const settingsTheme = getSettingsListTheme();
+ const listTheme = { ...settingsTheme, hint: (text: string) => settingsTheme.hint(text.replace("Esc to cancel", "Esc to close")) };
+ const visibleRows = Math.max(3, Math.min(12, tui.terminal.rows - 15));
+ const list = new SettingsList(items, visibleRows, listTheme, (id, value) => {
+ busy = true;
+ void (async () => {
+ if (options.preferences && id !== "resume") {
+ if (id === "reset-images") {
+ for (const key of ["cap", "scale", "hints", "grid", "action"]) scoped = await options.preferences.set(ctx, scope, key, undefined);
+ } else {
+ let parsed: unknown = value === "inherit" ? undefined : value;
+ if (parsed !== undefined && ["cap", "scale", "opacity"].includes(id)) {
+ if (!/^(?:\d+(?:\.\d+)?|\.\d+)$/.test(value)) throw new Error("Enter a number or inherit.");
+ parsed = Number(value);
+ } else if (parsed !== undefined && !["mode", "preset", "quality"].includes(id)) parsed = value === "on";
+ scoped = await options.preferences.set(ctx, scope, id, parsed);
+ if (id === "mode" && value !== "inherit" && scoped.effective.mode === value && options.mode() === value)
+ await options.mode.select(value, ctx);
+ }
+ escapeStop = scoped!.effective.escape ?? true;
+ }
+ else if (id === "mode" || id === "resume") await options.mode.select(id === "resume" ? options.mode() : value, ctx);
+ else if (id === "escape") { await writeEscapeStop(ctx.cwd, value === "on"); escapeStop = value === "on"; }
+ else if (id === "cap" || id === "scale") {
+ const current = options.images.settings();
+ if (!/^(?:\d+(?:\.\d+)?|\.\d+)$/.test(value)) throw new Error("Enter a number.");
+ options.images.update({ kernel: current.kernel, maxLongEdge: id === "cap" ? Number(value) : current.maxLongEdge, scale: id === "scale" ? Number(value) : current.scale });
+ } else if (id === "reset-images") options.images.reset();
+ else if (id === "hints") options.images.setHints(value === "on");
+ else if (id === "grid" || id === "action") {
+ const guides = options.images.settings().guides;
+ options.images.setGuides({ grid: id === "grid" ? value === "on" : !!guides?.grid, action: id === "action" ? value === "on" : !!guides?.action });
+ } else {
+ const patch = { [id]: id === "opacity" ? Number(value) : id === "preset" || id === "quality" ? value : value === "on" } as Partial<VisualOptions>;
+ if (id === "opacity" && !/^(?:\d+(?:\.\d+)?|\.\d+)$/.test(value)) throw new Error("Enter a number.");
+ pi.appendEntry("computer-use-visuals", options.visuals.update(patch));
+ }
+ })().catch(error => ctx.ui.notify(error instanceof Error ? error.message : String(error), "error"))
+ .finally(() => { busy = false; sync(); tui.requestRender(); });
+ }, () => { unsubscribe(); done(); });
+ sync();
+ const refresh = () => { sync(); tui.requestRender(); };
+ const offStop = options.stop.onStop(refresh);
+ const offResume = options.stop.onResume(refresh);
+ unsubscribe = () => { offStop(); offResume(); };
+ return {
+ dispose() { unsubscribe(); },
+ render(width: number) { return [truncateToWidth(theme.fg("accent", theme.bold(options.preferences ? `Computer Use settings · ${scope === "project" ? "Project" : "Global"}` : "Computer Use settings")), width), "", ...list.render(width)]; },
+ invalidate() { list.invalidate(); },
+ handleInput(data: string) { if (!busy) list.handleInput(data); tui.requestRender(); },
+ handleMouse(event: TuiMouseEvent) { if (busy || event.y < 2) return undefined; return list.handleMouse({ ...event, y: event.y - 2 }); },
+ };
+ });
+ },
+ });
+}
diff --git a/src/config.ts b/src/config.ts
index c2768d9..3c7ab0e 100644
--- a/src/config.ts
+++ b/src/config.ts
@@ -12,6 +12,8 @@ export interface ComputerConfig {
interop?: string;
runtime?: "windows-held-paths" | "windows-key-holds" | "windows-selected-displays" | "windows-capture-drag" | "windows-shell-desktops" | "windows-single-png" | "windows-semantic-actions" | "windows-fast-bilinear" | "windows-direct-process" | "windows-persistent-wgc" | "windows-native-pointer" | "windows-coordinate-scroll" | "windows-window-restore" | "windows-window-management" | "windows-desktop";
bundled?: BundledRuntime;
+ /** Install the physical triple-Escape stop hook; enabled by default. */
+ escapeStop?: boolean;
/** Explicitly trusted experimental driver, with protocol checks instead of packaged-byte checks. */
driverOverride?: true;
/** A sandbox guest's driver behind a host loopback relay. Set by the sandbox lifecycle, never read from config files. */
@@ -63,7 +65,8 @@ export async function loadComputerConfig(cwd: string): Promise<ComputerConfig> {
if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw new Error("Invalid .pi/computer-use.json; see docs/setup.md#override-the-runtime.");
}
if (!raw || typeof raw !== "object" || Array.isArray(raw)
- || Object.keys(raw).some(key => !["command", "args", "interop", "runtime", "machines", "driverOverride"].includes(key))
+ || Object.keys(raw).some(key => !["command", "args", "interop", "runtime", "machines", "driverOverride", "escapeStop"].includes(key))
+ || (raw.escapeStop !== undefined && typeof raw.escapeStop !== "boolean")
|| (raw.driverOverride !== undefined && raw.driverOverride !== true)
|| (raw.driverOverride === true && typeof raw.command !== "string")
|| (raw.command !== undefined && (typeof raw.command !== "string" || !raw.command))
@@ -87,21 +90,23 @@ export async function loadComputerConfig(cwd: string): Promise<ComputerConfig> {
return { command: command || bundled!.command,
...(bundled ? { bundled } : {}),
...(raw.args ? { args: raw.args as string[] } : {}), ...(interop ? { interop } : {}),
+ ...(raw.escapeStop !== undefined ? { escapeStop: raw.escapeStop as boolean } : {}),
...(runtime ? { runtime } : {}), ...(raw.driverOverride === true ? { driverOverride: true as const } : {}),
...(raw.machines !== undefined ? { machines: parseMachines(raw.machines) } : {}) };
}
/** The trusted file's interop socket and machines, without selecting or downloading a runtime. */
-export async function readMachineSettings(cwd: string): Promise<{ interop?: string; machines: Record<string, MachineConfig> }> {
+export async function readMachineSettings(cwd: string): Promise<{ interop?: string; machines: Record<string, MachineConfig>; escapeStop?: boolean }> {
let raw: unknown = {};
try { raw = JSON.parse(await readFile(path.join(cwd, ".pi", "computer-use.json"), "utf8")); }
catch (error) {
if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw new Error("Invalid .pi/computer-use.json; see docs/setup.md#override-the-runtime.");
}
if (!raw || typeof raw !== "object" || Array.isArray(raw)) throw new Error("Invalid .pi/computer-use.json; see docs/setup.md#override-the-runtime.");
- const { interop: configured, machines } = raw as Record<string, unknown>;
+ const { interop: configured, machines, escapeStop } = raw as Record<string, unknown>;
+ if (escapeStop !== undefined && typeof escapeStop !== "boolean") throw new Error("Invalid escapeStop in .pi/computer-use.json.");
const interop = typeof configured === "string" ? configured : await liveInterop();
- return { ...(interop ? { interop } : {}), machines: machines === undefined ? {} : parseMachines(machines) };
+ return { ...(interop ? { interop } : {}), machines: machines === undefined ? {} : parseMachines(machines), ...(escapeStop !== undefined ? { escapeStop: escapeStop as boolean } : {}) };
}
/** Paths emitted by the old setup command are package-managed defaults, not permanent user overrides. */
diff --git a/src/driver.ts b/src/driver.ts
index 78c36d0..6f223d8 100644
--- a/src/driver.ts
+++ b/src/driver.ts
@@ -191,7 +191,6 @@ export class McpDriver implements Driver {
}, visualSettings);
this.visuals = this.presentation.flash;
if (this.humanStop) this.stopSubscriptions.push(this.humanStop.onResume(() => {
- if (!this.stoppedNatively) return;
this.stoppedNatively = false;
// Window identities stay: only the stopped driver process is replaced.
void this.closeClient().catch(() => undefined);
@@ -215,6 +214,7 @@ export class McpDriver implements Driver {
try {
const client = await pending;
signal?.throwIfAborted();
+ if (client !== this.client) throw new Error("Driver connection was replaced after computer use resumed.");
return client;
} finally { if (this.connecting === pending) this.connecting = undefined; }
}
@@ -237,7 +237,7 @@ export class McpDriver implements Driver {
const client = new Client({ name: "pi-agent-computer-use-poc", version: "0.0.0" });
// The driver says at once when the human stops input, without waiting for a call.
client.fallbackNotificationHandler = async notification => {
- if (notification.method === "notifications/cua/human_stop") this.reportStop();
+ if (this.client === client && notification.method === "notifications/cua/human_stop") this.reportStop();
};
this.client = client;
try {
@@ -308,7 +308,7 @@ export class McpDriver implements Driver {
const label = this.sessionLabel?.()?.replace(/[^\p{L}\p{N} ._-]/gu, "").trim().slice(0, 28);
return { CUA_DRIVER_RS_TELEMETRY_ENABLED: "false", CUA_DRIVER_RS_UPDATE_CHECK: "false",
...(this.envelopes ? { CUA_DRIVER_MCP_ENVELOPES: "1" } : {}), DO_NOT_TRACK: "1",
- CUA_ESCAPE_STOP: "1", CUA_SHARED_INPUT_LOCK: "1", ...(label ? { CUA_SESSION_LABEL: label } : {}),
+ CUA_ESCAPE_STOP: this.config.escapeStop === false ? "0" : "1", CUA_SHARED_INPUT_LOCK: "1", ...(label ? { CUA_SESSION_LABEL: label } : {}),
CUA_DRIVER_ALLOW_DEBUGGING_LAUNCH: "1" };
}
@@ -360,7 +360,7 @@ export class McpDriver implements Driver {
companion(): McpDriver | undefined {
if (this.config.endpoint) return undefined;
const visuals = new VisualSettings(); visuals.update({ preset: "off" });
- return new McpDriver(this.config, this.version, visuals);
+ return new McpDriver(this.config, this.version, visuals, this.humanStop, this.sessionLabel);
}
/** One cancellable native call. This method never retries input. */
async call(name: string, args: Record<string, unknown>, signal?: AbortSignal): Promise<DriverResult> {
@@ -395,6 +395,8 @@ export class McpDriver implements Driver {
if (this.humanStop?.stopped)
return { content: [{ type: "text", text: this.humanStop.message() }], structuredContent: { status: "refused", effect: "refused", code: "human_stop", attempted: false }, isError: true };
const client = await this.connect(signal);
+ if (this.humanStop?.stopped)
+ return { content: [{ type: "text", text: this.humanStop.message() }], structuredContent: { status: "refused", effect: "refused", code: "human_stop", attempted: false }, isError: true };
if (this.nativeTools && !this.nativeTools.has(name))
return { content: [], structuredContent: { status: "refused", code: "unsupported_native_capability", platform: this.platform, tool: name, attempted: false, fallback: false }, isError: true };
// Since 0.35 the driver shapes window state for models; the adapter needs the full payload with `elements`.
@@ -403,7 +405,7 @@ export class McpDriver implements Driver {
// every other native call answers well within 15 s.
const timeout = name === "input_timeline" ? 15_000 + timelineLength(args) : name === "launch_app" ? 90_000 : 15_000;
const result = CallToolResultSchema.parse(await client.callTool({ name, arguments: args }, undefined, { signal, timeout }));
- if (reportsHumanStop(result as Parameters<typeof reportsHumanStop>[0])) this.reportStop();
+ if (this.client === client && reportsHumanStop(result as Parameters<typeof reportsHumanStop>[0])) this.reportStop();
const content: Content[] = [];
for (const block of result.content) {
if (block.type === "text") content.push({ type: "text", text: block.text });
diff --git a/src/image-settings-command.ts b/src/image-settings-command.ts
deleted file mode 100644
index 586836d..0000000
--- a/src/image-settings-command.ts
+++ /dev/null
@@ -1,39 +0,0 @@
-import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
-import { CuaImageSettings } from "./cua-image-settings.ts";
-
-const usage = "Usage: /computer-images [show | cap 64..4096 | scale 0.05..1 | hints on|off | guides off|grid|action|all | reset].";
-
-/** Human-only controls for future captures; no viewer, listener, native process or image retention. */
-export function registerImageSettingsCommand(pi: ExtensionAPI, images: CuaImageSettings): void {
- pi.registerCommand("computer-images", {
- description: "Show or change session-local screenshot sizing, hint labels and image guides for future captures.",
- getArgumentCompletions: prefix => ["show", "cap", "scale", "hints on", "hints off", "guides off", "guides grid", "guides action", "guides all", "reset"]
- .filter(value => value.startsWith(prefix)).map(value => ({ value, label: value })),
- handler: async (args, ctx) => {
- const parts = args.trim().toLowerCase().split(/\s+/).filter(Boolean);
- const [command = "show", value] = parts;
- try {
- const current = images.settings();
- if (command === "show" || command === "reset") {
- if (parts.length > 1) throw Error(usage);
- if (command === "reset") images.reset();
- } else if (command === "cap" || command === "scale") {
- if (parts.length !== 2 || !/^(?:\d+(?:\.\d+)?|\.\d+)$/.test(value)) throw Error(usage);
- images.update({ kernel: current.kernel, maxLongEdge: command === "cap" ? Number(value) : current.maxLongEdge,
- scale: command === "scale" ? Number(value) : current.scale });
- } else if (command === "hints") {
- if (parts.length !== 2 || (value !== "on" && value !== "off")) throw Error(usage);
- images.setHints(value === "on");
- } else if (command === "guides") {
- if (parts.length !== 2 || !["off", "grid", "action", "all"].includes(value)) throw Error(usage);
- images.setGuides({ grid: value === "grid" || value === "all", action: value === "action" || value === "all" });
- } else throw Error(usage);
- const settings = images.settings();
- ctx.ui.notify(`Computer images: ${settings.maxLongEdge} px maximum, scale ${settings.scale}, ${settings.kernel}; hints ${settings.hints === false ? "off" : "on"}, `
- + `grid ${settings.guides?.grid ? "on" : "off"}, action guide ${settings.guides?.action ? "on" : "off"}. Applies to future captures in this session.`, "info");
- } catch (error) {
- ctx.ui.notify(error instanceof Error ? error.message : usage, "warning");
- }
- },
- });
-}
diff --git a/src/index.ts b/src/index.ts
index e7221c6..e300855 100644
--- a/src/index.ts
+++ b/src/index.ts
@@ -1,6 +1,5 @@
import { basename } from "node:path";
-import { registerImageSettingsCommand } from "./image-settings-command.ts";
-import { registerVisualSettingsCommand } from "./visual-settings-command.ts";
+import { registerVisualPreferences } from "./visual-preferences.ts";
import { VisualSettings } from "./native-visuals.ts";
import { CuaImageSettings } from "./cua-image-settings.ts";
import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
@@ -19,6 +18,9 @@ import { localRunner } from "./runner.ts";
import { MachineHost } from "./machine-host.ts";
import { machineFactory, sandboxFactory } from "./machine.ts";
import { HumanStop } from "./human-stop.ts";
+import { registerComputerSettingsCommand } from "./computer-settings.ts";
+import { PreferencesController } from "./preferences-controller.ts";
+import type { ExtensionContext } from "@earendil-works/pi-coding-agent";
/**
* Register native computer resources with ordinary IDE tools when Pi Agent IDE accepts them, otherwise as
@@ -28,11 +30,23 @@ export default async function computerUse(pi: ExtensionAPI): Promise<void> {
let overviewKernel: "lanczos3" | "bilinear" = "lanczos3";
const images = new CuaImageSettings(() => overviewKernel);
const visuals = new VisualSettings();
- registerImageSettingsCommand(pi, images);
- registerVisualSettingsCommand(pi, visuals);
+ registerVisualPreferences(pi, visuals);
let canEnterSafe = () => true;
const humanStop = new HumanStop();
const getMode = registerComputerMode(pi, async () => { visuals.clear(); }, () => canEnterSafe(), humanStop);
+ const preferences = new PreferencesController(images, visuals, getMode);
+ let currentContext: ExtensionContext | undefined;
+ const restorePreferences = async (ctx: ExtensionContext) => {
+ currentContext = ctx;
+ if (!ctx.cwd || !ctx.isProjectTrusted) return;
+ preferences.captureBaseline(ctx);
+ await preferences.apply(ctx, true, true);
+ };
+ pi.on("session_start", (_event, ctx) => restorePreferences(ctx));
+ pi.on("session_tree", (_event, ctx) => restorePreferences(ctx));
+ const scopedEscapeStop = async () => currentContext?.cwd
+ ? (await preferences.read(currentContext)).effective.escape ?? true : undefined;
+ registerComputerSettingsCommand(pi, { mode: getMode, images, visuals, stop: humanStop, preferences });
let cwdLabel: string | undefined;
// The agent cursor names this session, or its project folder until the session has a name.
const sessionLabel = () => pi.getSessionName()?.trim() || cwdLabel;
@@ -43,6 +57,8 @@ export default async function computerUse(pi: ExtensionAPI): Promise<void> {
if (session) return session;
if (!loading) loading = loadComputerConfig(cwd).then(async config => {
cwdLabel = basename(cwd);
+ const escapeStop = await scopedEscapeStop();
+ if (escapeStop !== undefined) config.escapeStop = escapeStop;
const driver = new McpDriver(config, undefined, visuals, humanStop, sessionLabel);
overviewKernel = driver.overviewKernel ?? "lanczos3";
session = new ComputerSession(driver, getMode, () => images.settings(), undefined,
@@ -54,7 +70,7 @@ export default async function computerUse(pi: ExtensionAPI): Promise<void> {
finally { loading = undefined; }
}
const openSandbox = sandboxFactory({ settings: () => images.settings(), visuals });
- const connectMachine = machineFactory({ getMode, settings: () => images.settings(), visuals, humanStop, sessionLabel });
+ const connectMachine = machineFactory({ getMode, settings: () => images.settings(), visuals, humanStop, sessionLabel, escapeStop: scopedEscapeStop });
const provider = await registerComputerProvider(pi, getSession, async () => {
const result: ComputerResult = session ? await session.close() : {
content: [{ type: "text", text: "Not started." }],
diff --git a/src/machine.ts b/src/machine.ts
index 8af5d43..985be02 100644
--- a/src/machine.ts
+++ b/src/machine.ts
@@ -84,13 +84,14 @@ export function sandboxFactory(options: { settings: () => CuaVisionSettings; vis
* Another machine's desktop gets its own driver and session over SSH. Safe/YOLO is the user's choice there too;
* the host overlay is not shown, and browsers start on that machine.
*/
-export function machineFactory(options: { getMode: () => ComputerMode; settings: () => CuaVisionSettings; visuals?: VisualSettings; humanStop?: HumanStop; sessionLabel?: () => string | undefined }): MachineFactory {
+export function machineFactory(options: { getMode: () => ComputerMode; settings: () => CuaVisionSettings; visuals?: VisualSettings; humanStop?: HumanStop; sessionLabel?: () => string | undefined; escapeStop?: (cwd: string) => Promise<boolean | undefined> }): MachineFactory {
return async (cwd, name, signal) => {
const machine = await machineConfig(cwd, name);
+ const escapeStop = options.escapeStop ? await options.escapeStop(cwd) : (await readMachineSettings(cwd)).escapeStop;
const runner = sshRunner(machine.ssh);
const command = machine.command ?? await prepareRemoteRuntime(runner, signal);
const runtime = machine.command ? machine.runtime : "windows-desktop";
- const driver = new McpDriver({ command, ...(runtime ? { runtime } : {}), ...(machine.driverOverride ? { driverOverride: true as const } : {}), ssh: machine.ssh }, undefined, options.visuals, options.humanStop, options.sessionLabel);
+ const driver = new McpDriver({ command, ...(runtime ? { runtime } : {}), ...(machine.driverOverride ? { driverOverride: true as const } : {}), ...(escapeStop !== undefined ? { escapeStop } : {}), ssh: machine.ssh }, undefined, options.visuals, options.humanStop, options.sessionLabel);
try { await waitForGuestDriver(driver, signal, 30_000); }
catch (error) { await driver.close().catch(() => undefined); throw error; }
const launchWeb = async (launch: Parameters<typeof WebBrowser.launch>[0], webSignal?: AbortSignal) => {
diff --git a/src/mode.ts b/src/mode.ts
index a37e639..cd1fa6b 100644
--- a/src/mode.ts
+++ b/src/mode.ts
@@ -1,4 +1,4 @@
-import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent";
+import type { ExtensionAPI, ExtensionContext, ExtensionCommandContext } from "@earendil-works/pi-coding-agent";
import type { ComputerMode } from "./session.ts";
import { HumanStop } from "./human-stop.ts";
@@ -7,7 +7,7 @@ import { HumanStop } from "./human-stop.ts";
* The human's stop (Escape pressed three times) shows as STOPPED until the human selects a mode again.
*/
export function registerComputerMode(pi: ExtensionAPI, clearFrame: () => Promise<void> = async () => {}, canEnterSafe: () => boolean = () => true,
- humanStop = new HumanStop()): () => ComputerMode {
+ humanStop = new HumanStop()): (() => ComputerMode) & { select: (args: string, ctx: ExtensionCommandContext) => Promise<void>; applyPreference: (value: ComputerMode, ctx: ExtensionContext) => Promise<boolean>; allowsPreference: (value: ComputerMode) => boolean; recordPreference: () => void } {
let mode: ComputerMode = "yolo";
let ui: ExtensionContext["ui"] | undefined;
const show = (ctx?: ExtensionContext) => {
@@ -38,37 +38,49 @@ export function registerComputerMode(pi: ExtensionAPI, clearFrame: () => Promise
};
pi.on("session_start", (_event, ctx) => restore(ctx));
pi.on("session_tree", (_event, ctx) => restore(ctx));
+ const select = async (args: string, ctx: ExtensionCommandContext) => {
+ const selected = args.trim().toLowerCase();
+ if (selected && selected !== "safe" && selected !== "yolo") {
+ ctx.ui.notify("Usage: /computer-mode safe|yolo. Without arguments, show the current mode.", "warning");
+ return;
+ }
+ if (selected === "safe" && selected !== mode && !canEnterSafe()) {
+ ctx.ui.notify("Wait for pending input to finish and close any YOLO agent browser before selecting Safe; its local debugging port stays accessible until it closes. If cleanup was unconfirmed, verify it is closed and restart Pi.", "warning");
+ return;
+ }
+ if ((selected === "safe" || selected === "yolo") && selected !== mode) {
+ // Persist only the user's choice, not window ownership or model-supplied data.
+ pi.appendEntry("computer-use-mode", { mode: selected });
+ mode = selected;
+ await clearFrame();
+ }
+ const resumed = selected !== "" && humanStop.stopped;
+ if (resumed) {
+ humanStop.resume();
+ pi.sendMessage({ customType: "computer-use-stop", display: true, content: "The human turned computer use on their desktop on again." }, { deliverAs: "nextTurn" });
+ }
+ show(ctx);
+ if (humanStop.stopped) {
+ ctx.ui.notify(`Computer mode: STOPPED by Escape (${mode.toUpperCase()} when on). /computer-mode safe or /computer-mode yolo turns it on again.`, "warning");
+ return;
+ }
+ ctx.ui.notify(`Computer mode: ${mode.toUpperCase()}${resumed ? ", on again" : ""}. ${mode === "safe" ? "Only session-opened windows are allowed." : "Desktop-wide UI access is allowed."} Not an OS sandbox.`, "info");
+ };
pi.registerCommand("computer-mode", {
description: "Show or select computer UI access: safe (session-opened windows) or yolo (desktop-wide). Selecting a mode also turns computer use on again after Escape stopped it. Not an OS sandbox.",
getArgumentCompletions: prefix => ["safe", "yolo"].filter(value => value.startsWith(prefix)).map(value => ({ value, label: value })),
- handler: async (args, ctx) => {
- const selected = args.trim().toLowerCase();
- if (selected && selected !== "safe" && selected !== "yolo") {
- ctx.ui.notify("Usage: /computer-mode safe|yolo. Without arguments, show the current mode.", "warning");
- return;
- }
- if (selected === "safe" && selected !== mode && !canEnterSafe()) {
- ctx.ui.notify("Wait for pending input to finish and close any YOLO agent browser before selecting Safe; its local debugging port stays accessible until it closes. If cleanup was unconfirmed, verify it is closed and restart Pi.", "warning");
- return;
- }
- if ((selected === "safe" || selected === "yolo") && selected !== mode) {
- // Persist only the user's choice, not window ownership or model-supplied data.
- pi.appendEntry("computer-use-mode", { mode: selected });
- mode = selected;
- await clearFrame();
- }
- const resumed = selected !== "" && humanStop.stopped;
- if (resumed) {
- humanStop.resume();
- pi.sendMessage({ customType: "computer-use-stop", display: true, content: "The human turned computer use on their desktop on again." }, { deliverAs: "nextTurn" });
- }
- show(ctx);
- if (humanStop.stopped) {
- ctx.ui.notify(`Computer mode: STOPPED by Escape (${mode.toUpperCase()} when on). /computer-mode safe or /computer-mode yolo turns it on again.`, "warning");
- return;
- }
- ctx.ui.notify(`Computer mode: ${mode.toUpperCase()}${resumed ? ", on again" : ""}. ${mode === "safe" ? "Only session-opened windows are allowed." : "Desktop-wide UI access is allowed."} Not an OS sandbox.`, "info");
- },
+ handler: select,
});
- return () => mode;
+ const applyPreference = async (value: ComputerMode, ctx: ExtensionContext): Promise<boolean> => {
+ if (value === "safe" && value !== mode && !canEnterSafe()) {
+ ctx.ui.notify("Close the YOLO agent browser before applying Safe mode.", "warning");
+ return false;
+ }
+ if (value !== mode) { mode = value; await clearFrame(); }
+ show(ctx);
+ return true;
+ };
+ const allowsPreference = (value: ComputerMode) => value !== "safe" || value === mode || canEnterSafe();
+ const recordPreference = () => pi.appendEntry("computer-use-mode", { mode });
+ return Object.assign(() => mode, { select, applyPreference, allowsPreference, recordPreference });
}
diff --git a/src/preferences-controller.ts b/src/preferences-controller.ts
new file mode 100644
index 0000000..a2d0198
--- /dev/null
+++ b/src/preferences-controller.ts
@@ -0,0 +1,80 @@
+import type { ExtensionContext, ExtensionCommandContext } from "@earendil-works/pi-coding-agent";
+import type { CuaImageSettings } from "./cua-image-settings.ts";
+import type { VisualOptions, VisualSettings } from "./native-visuals.ts";
+import type { ComputerMode } from "./session.ts";
+import { readLegacyEscapeStop, writeEscapeStop } from "./computer-settings.ts";
+import { readScopedPreferences, writeScopedPreference, type PreferenceKey, type ScopedValues } from "./scoped-preferences.ts";
+
+type ModeControl = (() => ComputerMode) & {
+ select(value: string, ctx: ExtensionCommandContext): Promise<void>;
+ applyPreference(value: ComputerMode, ctx: ExtensionContext): Promise<boolean>;
+ allowsPreference(value: ComputerMode): boolean;
+ recordPreference(): void;
+};
+type Scope = "global" | "project";
+
+/** Applies user-controlled values without resuming a human stop. */
+export class PreferencesController {
+ private visualBaseline: VisualOptions;
+ private modeBaseline: ComputerMode;
+ private modeFromSession = false;
+ private legacyPreview = false;
+ constructor(private readonly images: CuaImageSettings, private readonly visuals: VisualSettings, private readonly mode: ModeControl) {
+ this.visualBaseline = visuals.current();
+ this.modeBaseline = mode();
+ }
+ async read(ctx: ExtensionContext) {
+ const snapshot = await readScopedPreferences(ctx.cwd, ctx.isProjectTrusted());
+ // The old project file remains authoritative until the user edits this preference.
+ if (ctx.isProjectTrusted() && snapshot.project.escape === undefined) {
+ const legacy = await readLegacyEscapeStop(ctx.cwd);
+ if (legacy !== undefined) {
+ snapshot.project.escape = legacy;
+ snapshot.effective.escape = legacy;
+ snapshot.sources.escape = "project";
+ }
+ }
+ return snapshot;
+ }
+ captureBaseline(ctx: ExtensionContext) {
+ this.visualBaseline = this.visuals.current(); this.modeBaseline = this.mode();
+ const branch = ctx.sessionManager?.getBranch?.() ?? [];
+ this.modeFromSession = branch.some(entry => entry.type === "custom" && entry.customType === "computer-use-mode" && ((entry.data as { mode?: unknown } | undefined)?.mode === "safe" || (entry.data as { mode?: unknown } | undefined)?.mode === "yolo"));
+ const visual = branch.filter(entry => entry.type === "custom" && entry.customType === "computer-use-visuals").at(-1);
+ this.legacyPreview = visual?.type === "custom" && typeof (visual.data as { previews?: unknown } | undefined)?.previews === "boolean";
+ }
+ async apply(ctx: ExtensionContext, applyMode = true, preferSession = false) {
+ const snapshot = await this.read(ctx);
+ const v = snapshot.effective;
+ if (applyMode) await this.mode.applyPreference(preferSession && this.modeFromSession ? this.modeBaseline : v.mode ?? this.modeBaseline, ctx);
+ const image = this.images.settings();
+ this.images.update({ kernel: image.kernel, maxLongEdge: v.cap ?? 1280, scale: v.scale ?? 1 });
+ this.images.setHints(v.hints ?? true);
+ this.images.setGuides({ grid: v.grid ?? false, action: v.action ?? false });
+ const patch: Partial<VisualOptions> = { ...this.visualBaseline };
+ for (const key of ["preset", "observations", "previews", "pointer", "scroll_drag", "keyboard", "typed_text", "reduced_motion", "quality", "opacity"] as const) {
+ if (v[key] !== undefined) Object.assign(patch, { [key]: v[key] });
+ }
+ if (!this.legacyPreview && v.previews === undefined && v.preset === "subtle") patch.previews = false;
+ if (!this.legacyPreview && v.previews === undefined && v.preset === "full") patch.previews = true;
+ this.visuals.restore(patch);
+ return snapshot;
+ }
+ async set(ctx: ExtensionCommandContext, scope: Scope, id: string, value: unknown) {
+ if (scope === "project" && !ctx.isProjectTrusted()) throw new Error("Trust this project before changing its settings.");
+ if (id === "mode") {
+ const snapshot = await this.read(ctx);
+ const project = scope === "project" ? value : snapshot.project.mode;
+ const global = scope === "global" ? value : snapshot.global.mode;
+ const candidate = project ?? global ?? this.modeBaseline;
+ if ((candidate === "safe" || candidate === "yolo") && !this.mode.allowsPreference(candidate))
+ throw new Error("Close the YOLO agent browser before applying Safe mode.");
+ }
+ if (scope === "project" && id === "escape") await readLegacyEscapeStop(ctx.cwd);
+ await writeScopedPreference(ctx.cwd, scope, id as PreferenceKey, value as ScopedValues[PreferenceKey] | undefined);
+ if (scope === "project" && id === "escape") await writeEscapeStop(ctx.cwd, undefined);
+ const result = await this.apply(ctx, id === "mode");
+ if (id === "mode") this.mode.recordPreference();
+ return result;
+ }
+}
diff --git a/src/scoped-preferences.ts b/src/scoped-preferences.ts
new file mode 100644
index 0000000..92f9411
--- /dev/null
+++ b/src/scoped-preferences.ts
@@ -0,0 +1,112 @@
+import { randomUUID } from "node:crypto";
+import { mkdir, readFile, rename, stat, unlink, writeFile } from "node:fs/promises";
+import path from "node:path";
+import { getAgentDir } from "@earendil-works/pi-coding-agent";
+import { lock } from "proper-lockfile";
+
+/** Preferences users can set independently at global and project level. */
+export interface ScopedValues {
+ mode: "safe" | "yolo";
+ escape: boolean;
+ cap: number;
+ scale: number;
+ hints: boolean;
+ grid: boolean;
+ action: boolean;
+ preset: "off" | "subtle" | "full";
+ observations: boolean;
+ previews: boolean;
+ pointer: boolean;
+ scroll_drag: boolean;
+ keyboard: boolean;
+ typed_text: boolean;
+ reduced_motion: boolean;
+ quality: "performance" | "balanced" | "high";
+ opacity: number;
+}
+export type PreferenceKey = keyof ScopedValues;
+export type PreferenceScope = "global" | "project";
+export type Preferences = Partial<ScopedValues>;
+export interface ScopedPreferences {
+ global: Preferences;
+ project: Preferences;
+ effective: Preferences;
+ sources: Partial<Record<PreferenceKey, PreferenceScope>>;
+}
+
+const booleanKeys = new Set(["escape", "hints", "grid", "action", "observations", "previews", "pointer", "scroll_drag", "keyboard", "typed_text", "reduced_motion"]);
+function valid(key: string, value: unknown): boolean {
+ if (booleanKeys.has(key)) return typeof value === "boolean";
+ switch (key) {
+ case "mode": return value === "safe" || value === "yolo";
+ case "preset": return value === "off" || value === "subtle" || value === "full";
+ case "quality": return value === "performance" || value === "balanced" || value === "high";
+ case "cap": return typeof value === "number" && Number.isSafeInteger(value) && value >= 64 && value <= 4096;
+ case "scale": return typeof value === "number" && Number.isFinite(value) && value >= 0.05 && value <= 1;
+ case "opacity": return typeof value === "number" && Number.isFinite(value) && value >= 0 && value <= 1;
+ default: return false;
+ }
+}
+function fileFor(cwd: string, scope: PreferenceScope): string {
+ return path.join(scope === "global" ? getAgentDir() : path.join(cwd, ".pi"), "computer-use-settings.json");
+}
+async function readFileSettings(file: string): Promise<Preferences> {
+ let raw: unknown;
+ try { raw = JSON.parse(await readFile(file, "utf8")); }
+ catch (error) {
+ if ((error as NodeJS.ErrnoException).code === "ENOENT") return {};
+ throw new Error(`Cannot read Computer Use settings at ${file}.`, { cause: error });
+ }
+ if (!raw || typeof raw !== "object" || Array.isArray(raw) || Object.entries(raw).some(([key, value]) => !valid(key, value)))
+ throw new Error(`Invalid Computer Use settings at ${file}.`);
+ return raw as Preferences;
+}
+
+/** Read only trusted project overrides; defaults are applied by each consumer. */
+export async function readScopedPreferences(cwd: string, trusted = true): Promise<ScopedPreferences> {
+ const [global, project] = await Promise.all([
+ readFileSettings(fileFor(cwd, "global")),
+ trusted ? readFileSettings(fileFor(cwd, "project")) : Promise.resolve({} as Preferences),
+ ]);
+ const effective = { ...global, ...project };
+ const sources: ScopedPreferences["sources"] = {};
+ for (const key of Object.keys(global) as PreferenceKey[]) sources[key] = "global";
+ for (const key of Object.keys(project) as PreferenceKey[]) sources[key] = "project";
+ return { global, project, effective, sources };
+}
+
+/** Write one preference without replacing other keys. Undefined clears that override. */
+export async function writeScopedPreference<K extends PreferenceKey>(cwd: string, scope: PreferenceScope, key: K, value: ScopedValues[K] | undefined): Promise<void> {
+ if (scope !== "global" && scope !== "project") throw new Error("Invalid preference scope.");
+ if (value === undefined ? !valid(key, sampleValue(key)) : !valid(key, value)) throw new Error(`Invalid Computer Use setting: ${key}.`);
+ const file = fileFor(cwd, scope);
+ const dir = path.dirname(file);
+ await mkdir(dir, { recursive: true });
+ const release = await lock(file, { realpath: false, retries: { retries: 40, factor: 1.2, minTimeout: 10, maxTimeout: 100 } });
+ try {
+ const settings = await readFileSettings(file);
+ if (value === undefined) delete settings[key];
+ else settings[key] = value;
+ const mode = await stat(file).then(info => info.mode & 0o777).catch((error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return 0o600; throw error; });
+ const temp = path.join(dir, `.computer-use-settings-${randomUUID()}.tmp`);
+ try {
+ await writeFile(temp, `${JSON.stringify(settings, null, 2)}\n`, { mode, flag: "wx" });
+ await rename(temp, file);
+ } finally {
+ await unlink(temp).catch((error: NodeJS.ErrnoException) => { if (error.code !== "ENOENT") throw error; });
+ }
+ } finally { await release(); }
+}
+
+function sampleValue(key: string): unknown {
+ if (booleanKeys.has(key)) return false;
+ switch (key) {
+ case "mode": return "safe";
+ case "preset": return "off";
+ case "quality": return "balanced";
+ case "cap": return 64;
+ case "scale": return 1;
+ case "opacity": return 0;
+ default: return undefined;
+ }
+}
diff --git a/src/visual-preferences.ts b/src/visual-preferences.ts
new file mode 100644
index 0000000..5372c2d
--- /dev/null
+++ b/src/visual-preferences.ts
@@ -0,0 +1,15 @@
+import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent";
+import { VisualSettings, type VisualOptions } from "./native-visuals.ts";
+
+/** Restore human visual preferences from the current session branch. */
+export function registerVisualPreferences(pi: ExtensionAPI, settings: VisualSettings): void {
+ const restore = (ctx: ExtensionContext) => {
+ let saved: Partial<VisualOptions> = { preset: "full" };
+ for (const entry of ctx.sessionManager.getBranch()) {
+ if (entry.type === "custom" && entry.customType === "computer-use-visuals") saved = entry.data as Partial<VisualOptions>;
+ }
+ try { settings.restore(saved); } catch { settings.restore({ preset: "off" }); }
+ };
+ pi.on("session_start", (_event, ctx) => restore(ctx));
+ pi.on("session_tree", (_event, ctx) => restore(ctx));
+}
diff --git a/src/visual-settings-command.ts b/src/visual-settings-command.ts
deleted file mode 100644
index 01dfa2b..0000000
--- a/src/visual-settings-command.ts
+++ /dev/null
@@ -1,32 +0,0 @@
-import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent";
-import { VisualSettings, type VisualOptions } from "./native-visuals.ts";
-
-/** Human-only controls; persistence contains preferences, never images or input text. */
-export function registerVisualSettingsCommand(pi: ExtensionAPI, settings: VisualSettings): void {
- const restore = (ctx: ExtensionContext) => {
- let saved: Partial<VisualOptions> = { preset: "full" };
- for (const entry of ctx.sessionManager.getBranch()) {
- if (entry.type === "custom" && entry.customType === "computer-use-visuals") saved = entry.data as Partial<VisualOptions>;
- }
- try { settings.restore(saved); } catch { settings.restore({ preset: "off" }); }
- };
- pi.on("session_start", (_event, ctx) => restore(ctx));
- pi.on("session_tree", (_event, ctx) => restore(ctx));
- pi.registerCommand("computer-visuals", {
- description: "Show or change native computer visuals: off, subtle, full, or JSON presentation preferences. No input or capture changes.",
- getArgumentCompletions: prefix => ["off", "subtle", "full"].filter(value => value.startsWith(prefix)).map(value => ({ value, label: value })),
- handler: async (args, ctx) => {
- try {
- const text = args.trim();
- if (text) {
- const patch = ["off", "subtle", "full"].includes(text) ? { preset: text as VisualOptions["preset"] } : JSON.parse(text);
- pi.appendEntry("computer-use-visuals", settings.update(patch));
- }
- const value = settings.current();
- ctx.ui.notify(`Computer visuals: ${value.preset}. Previews ${value.previews ? "on" : "off"}; typed text ${value.typed_text ? "on" : "off"}; ${value.quality} quality; opacity ${Math.round(value.opacity * 100)}%; reduced motion ${value.reduced_motion ? "on" : "off"}.`, "info");
- } catch {
- ctx.ui.notify('Use /computer-visuals off|subtle|full, or JSON such as {"previews":false,"reduced_motion":true}. Preferences were not changed.', "warning");
- }
- },
- });
-}
diff --git a/test/computer-commands.integration.test.ts b/test/computer-commands.integration.test.ts
new file mode 100644
index 0000000..72da15a
--- /dev/null
+++ b/test/computer-commands.integration.test.ts
@@ -0,0 +1,20 @@
+import { RpcClient } from "@earendil-works/pi-coding-agent";
+import { mkdtemp, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+import { expect, test } from "vitest";
+
+test("Computer Use exposes only settings, mode and setup commands", async () => {
+ const cwd = await mkdtemp(path.join(tmpdir(), "computer-commands-"));
+ const client = new RpcClient({ cwd,
+ cliPath: path.join(path.dirname(fileURLToPath(import.meta.resolve("@earendil-works/pi-coding-agent"))), "cli.js"),
+ env: { PI_CODING_AGENT_DIR: path.join(cwd, "agent"), PI_OFFLINE: "1" },
+ args: ["--no-session", "--no-extensions", "--no-mcp", "--no-skills", "--no-context-files", "--approve", "--extension", path.resolve("src/index.ts")],
+ });
+ try {
+ await client.start();
+ const names = (await client.getCommands()).filter(command => command.name.startsWith("computer-")).map(command => command.name).sort();
+ expect(names).toEqual(["computer-mode", "computer-settings", "computer-setup"]);
+ } finally { await client.stop(); await rm(cwd, { recursive: true, force: true }); }
+});
diff --git a/test/computer-settings.test.ts b/test/computer-settings.test.ts
new file mode 100644
index 0000000..46daf2b
--- /dev/null
+++ b/test/computer-settings.test.ts
@@ -0,0 +1,114 @@
+import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { afterEach, expect, test, vi } from "vitest";
+import { readEscapeStop, registerComputerSettingsCommand, writeEscapeStop } from "../src/computer-settings.ts";
+import { registerComputerMode } from "../src/mode.ts";
+import { CuaImageSettings } from "../src/cua-image-settings.ts";
+import { VisualSettings } from "../src/native-visuals.ts";
+import { HumanStop } from "../src/human-stop.ts";
+import { PreferencesController } from "../src/preferences-controller.ts";
+import { readScopedPreferences, writeScopedPreference } from "../src/scoped-preferences.ts";
+import type { ExtensionAPI, ExtensionCommandContext } from "@earendil-works/pi-coding-agent";
+import { initTheme } from "@earendil-works/pi-coding-agent";
+import type { Component } from "@earendil-works/pi-tui";
+
+const dirs: string[] = [];
+afterEach(async () => { await Promise.all(dirs.splice(0).map(dir => rm(dir, { recursive: true, force: true }))); });
+async function workspace() { const dir = await mkdtemp(path.join(tmpdir(), "computer-settings-")); dirs.push(dir); return dir; }
+
+test("the shortcut setting defaults to enabled and can be saved without losing existing configuration", async () => {
+ const cwd = await workspace();
+ expect(await readEscapeStop(cwd)).toBe(true);
+ await mkdir(path.join(cwd, ".pi"));
+ const file = path.join(cwd, ".pi", "computer-use.json");
+ await writeFile(file, JSON.stringify({ machines: { office: { ssh: "office" } }, command: "driver" }));
+ await writeEscapeStop(cwd, false);
+ expect(await readEscapeStop(cwd)).toBe(false);
+ expect(JSON.parse(await readFile(file, "utf8"))).toEqual({ machines: { office: { ssh: "office" } }, command: "driver", escapeStop: false });
+ await writeEscapeStop(cwd, true);
+ expect(await readEscapeStop(cwd)).toBe(true);
+});
+
+test("clearing the legacy shortcut override keeps driver and machine configuration", async () => {
+ const cwd = await workspace(); await mkdir(path.join(cwd, ".pi"));
+ const file = path.join(cwd, ".pi", "computer-use.json");
+ await writeFile(file, JSON.stringify({ command: "driver", escapeStop: false, machines: { office: { ssh: "office" } } }));
+ await writeEscapeStop(cwd, undefined);
+ expect(JSON.parse(await readFile(file, "utf8"))).toEqual({ command: "driver", machines: { office: { ssh: "office" } } });
+});
+test("the shortcut setting never overwrites an invalid configuration", async () => {
+ const cwd = await workspace(); await mkdir(path.join(cwd, ".pi"));
+ const file = path.join(cwd, ".pi", "computer-use.json");
+ await writeFile(file, "invalid json");
+ await expect(writeEscapeStop(cwd, false)).rejects.toThrow();
+ expect(await readFile(file, "utf8")).toBe("invalid json");
+});
+
+
+test("the panel saves project and global preferences and clears an override", async () => {
+ initTheme("dark", false);
+ const cwd = await workspace(); const previous = process.env.PI_CODING_AGENT_DIR;
+ process.env.PI_CODING_AGENT_DIR = path.join(cwd, "agent");
+ try {
+ const handlers = new Map<string, (args: string, ctx: ExtensionCommandContext) => Promise<void>>();
+ const pi = { on: vi.fn(), registerCommand: (name: string, command: { handler: (args: string, ctx: ExtensionCommandContext) => Promise<void> }) => handlers.set(name, command.handler), appendEntry: vi.fn(), sendMessage: vi.fn() } as unknown as ExtensionAPI;
+ const stop = new HumanStop(); const mode = registerComputerMode(pi); const images = new CuaImageSettings(); const visuals = new VisualSettings();
+ const preferences = new PreferencesController(images, visuals, mode);
+ registerComputerSettingsCommand(pi, { mode, images, visuals, stop, preferences });
+ let component: Component | undefined;
+ let chosen: string | undefined = "Project";
+ const ctx = { cwd, mode: "tui", hasUI: true, isProjectTrusted: () => true, ui: { select: vi.fn(async () => chosen), notify: vi.fn(), setStatus: vi.fn(), custom: async (factory: (tui: unknown, theme: unknown, keys: unknown, done: () => void) => Component) => {
+ component = factory({ terminal: { rows: 35 }, requestRender: () => {} }, { fg: (_c: string, s: string) => s, bold: (s: string) => s }, {}, () => {});
+ } } } as unknown as ExtensionCommandContext;
+ await writeScopedPreference(cwd, "project", "cap", 777);
+ images.update({ kernel: images.settings().kernel, maxLongEdge: 777, scale: 1 });
+ await handlers.get("computer-settings")!("", ctx);
+ const key = (data: string) => component!.handleInput!(data);
+ expect(component!.render(90).join("\n")).toContain("777");
+ expect(ctx.ui.select).toHaveBeenCalledOnce();
+ key("\r"); await vi.waitFor(async () => expect((await readScopedPreferences(cwd)).project.mode).toBe("safe"));
+ key("\r"); await vi.waitFor(async () => expect((await readScopedPreferences(cwd)).project.mode).toBeUndefined());
+ chosen = "Global"; await handlers.get("computer-settings")!("", ctx);
+ expect(component!.render(90).join("\n")).toContain("1280");
+ expect(component!.render(90).join("\n")).not.toContain("777");
+ key("\r"); await vi.waitFor(async () => expect((await readScopedPreferences(cwd)).global.mode).toBe("safe"));
+ chosen = undefined; component = undefined; await handlers.get("computer-settings")!("", ctx); expect(component).toBeUndefined();
+ } finally { if (previous === undefined) delete process.env.PI_CODING_AGENT_DIR; else process.env.PI_CODING_AGENT_DIR = previous; }
+});
+test("the panel changes the real settings and respects mode safety", async () => {
+ initTheme("dark", false);
+ const cwd = await workspace();
+ const handlers = new Map<string, (args: string, ctx: ExtensionCommandContext) => Promise<void>>();
+ const entries: unknown[] = [];
+ const pi = { on: vi.fn(), registerCommand: (name: string, command: { handler: (args: string, ctx: ExtensionCommandContext) => Promise<void> }) => handlers.set(name, command.handler), appendEntry: (_type: string, data: unknown) => entries.push(data), sendMessage: vi.fn() } as unknown as ExtensionAPI;
+ const stop = new HumanStop();
+ let safeAllowed = false;
+ const mode = registerComputerMode(pi, async () => {}, () => safeAllowed, stop);
+ const images = new CuaImageSettings(); const visuals = new VisualSettings();
+ registerComputerSettingsCommand(pi, { mode, images, visuals, stop });
+ let component: Component | undefined;
+ const ctx = { cwd, mode: "tui", hasUI: true, ui: { notify: vi.fn(), setStatus: vi.fn(), custom: async (factory: (tui: unknown, theme: unknown, keys: unknown, done: () => void) => Component) => {
+ component = factory({ terminal: { rows: 35 }, requestRender: () => {} }, { fg: (_c: string, s: string) => s, bold: (s: string) => s }, {}, () => {});
+ } } } as unknown as ExtensionCommandContext;
+ await handlers.get("computer-settings")!("", ctx);
+ const key = (data: string) => component!.handleInput!(data);
+ const settled = () => new Promise(resolve => setTimeout(resolve, 20));
+ key("\r"); await settled();
+ expect(mode()).toBe("yolo");
+ safeAllowed = true; key("\r"); await settled(); expect(mode()).toBe("safe");
+ stop.stop(); expect(stop.stopped).toBe(true);
+ key("\u001b[B"); key("\r"); await settled(); expect(stop.stopped).toBe(false);
+ key("\u001b[B"); key("\r"); await vi.waitFor(async () => expect(await readEscapeStop(cwd)).toBe(false)); await settled();
+ const original = images.settings();
+ key("\u001b[B"); key("\r"); key("\u0015"); key("800"); key("\r"); await settled(); expect(images.settings().maxLongEdge).toBe(800);
+ key("\r"); key("\u0015"); key("63"); key("\r"); await settled(); expect(images.settings().maxLongEdge).toBe(800);
+ key("\u001b[B"); key("\r"); key("\u0015"); key("0.5"); key("\r"); await settled(); expect(images.settings().scale).toBe(0.5);
+ key("\u001b[B"); key("\r"); await settled(); expect(images.settings().hints).toBe(false);
+ key("\u001b[B"); key("\r"); await settled();
+ key("\u001b[B"); key("\r"); await settled(); expect(images.settings().guides).toEqual({ grid: true, action: true });
+ expect(original).toMatchObject({ maxLongEdge: 1280, scale: 1, version: 0 });
+ key("\u001b[B"); key("\r"); await settled(); expect(visuals.current().preset).toBe("off"); expect(entries).toContainEqual(expect.objectContaining({ preset: "off" }));
+ for (let i = 0; i < 10; i++) key("\u001b[B");
+ key("\r"); await settled(); expect(images.settings()).toEqual(original);
+});
\ No newline at end of file
diff --git a/test/config.test.ts b/test/config.test.ts
index 36d46f2..0a0d6f3 100644
--- a/test/config.test.ts
+++ b/test/config.test.ts
@@ -1,7 +1,7 @@
import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
import path from "node:path";
import { afterEach, expect, test, vi } from "vitest";
-import { loadComputerConfig } from "../src/config.ts";
+import { loadComputerConfig, readMachineSettings } from "../src/config.ts";
// These tests read configuration only; the live WSL interop socket of the machine running them stays out.
vi.mock("../src/wsl-interop.ts", () => ({ liveInterop: vi.fn(async () => undefined) }));
@@ -18,6 +18,19 @@ async function workspace() {
return cwd;
}
+test("the shortcut setting is read for local and remote desktop drivers", async () => {
+ const cwd = await workspace(); await mkdir(path.join(cwd, ".pi"));
+ const file = path.join(cwd, ".pi/computer-use.json");
+ for (const escapeStop of [true, false]) {
+ await writeFile(file, JSON.stringify({ command: "driver", escapeStop }));
+ expect((await loadComputerConfig(cwd)).escapeStop).toBe(escapeStop);
+ expect((await readMachineSettings(cwd)).escapeStop).toBe(escapeStop);
+ }
+ await writeFile(file, JSON.stringify({ command: "driver", escapeStop: "false" }));
+ await expect(loadComputerConfig(cwd)).rejects.toThrow();
+ await expect(readMachineSettings(cwd)).rejects.toThrow();
+});
+
test("opening has no target-file prerequisite and respects a trusted Driver override", async () => {
const cwd = await workspace();
vi.stubEnv("CUA_DRIVER_PATH", "C:\\Trusted\\cua-driver.exe");
diff --git a/test/fixtures/human-stop-driver.mjs b/test/fixtures/human-stop-driver.mjs
index 67646ff..38ffd47 100644
--- a/test/fixtures/human-stop-driver.mjs
+++ b/test/fixtures/human-stop-driver.mjs
@@ -20,7 +20,8 @@ server.setRequestHandler(CallToolRequestSchema, async request => {
const { name, arguments: args } = request.params;
record({ name, arguments: args });
if (stopped) return refusal();
- if (name === "press_key" && args?.key === "escape-x3") {
+ if (name === "press_key" && args?.key === "late-stop") { await new Promise(resolve => setTimeout(resolve, 100)); return refusal(); }
+ if (name === "press_key" && args?.key === "escape-x3" && process.env.CUA_ESCAPE_STOP === "1") {
stopped = true;
// The notification is sent as soon as the hook sees the third press; the running call then stops.
if (mode === "notify") await server.notification({ method: "notifications/cua/human_stop", params: { code: "human_stop" } });
diff --git a/test/fixtures/visual-commands-extension.ts b/test/fixtures/visual-commands-extension.ts
index 5fcc654..9735586 100644
--- a/test/fixtures/visual-commands-extension.ts
+++ b/test/fixtures/visual-commands-extension.ts
@@ -2,14 +2,19 @@ import { writeFile } from "node:fs/promises";
import path from "node:path";
import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
import { CuaImageSettings } from "../../src/cua-image-settings.ts";
-import { registerImageSettingsCommand } from "../../src/image-settings-command.ts";
import { VisualSettings } from "../../src/native-visuals.ts";
-import { registerVisualSettingsCommand } from "../../src/visual-settings-command.ts";
+import { registerVisualPreferences } from "../../src/visual-preferences.ts";
-/** Real Pi owns command dispatch, session entries and tree navigation; this fixture only reads the outcome. */
+/** Real Pi owns session entries and tree navigation; fixture commands create and inspect preferences. */
export default function(pi: ExtensionAPI): void {
const visuals = new VisualSettings(); const images = new CuaImageSettings();
- registerVisualSettingsCommand(pi, visuals); registerImageSettingsCommand(pi, images);
+ registerVisualPreferences(pi, visuals);
+ pi.registerCommand("visual-set", {
+ description: "Save test preferences.",
+ handler: async text => {
+ try { pi.appendEntry("computer-use-visuals", visuals.update(JSON.parse(text))); } catch { /* Invalid fixture input must not save. */ }
+ },
+ });
pi.registerCommand("visual-probe", {
description: "Save the current test preferences.",
handler: async (_args, ctx) => {
diff --git a/test/human-stop.test.ts b/test/human-stop.test.ts
index eaf5daa..05eee5d 100644
--- a/test/human-stop.test.ts
+++ b/test/human-stop.test.ts
@@ -56,6 +56,49 @@ test("the driver stops on the native notification, refuses without reaching the
} finally { await driver.close(); }
});
+test("disabling the shortcut prevents Escape from stopping computer use", async () => {
+ const dir = await mkdtemp(path.join(tmpdir(), "human-stop-")); dirs.push(dir);
+ const log = path.join(dir, "calls.jsonl");
+ const stop = new HumanStop();
+ const driver = new McpDriver({ command: process.execPath, args: [path.resolve("test/fixtures/human-stop-driver.mjs"), log], escapeStop: false }, "0.33.2", undefined, stop);
+ try {
+ const result = await driver.call("press_key", { key: "escape-x3" });
+ expect(result.isError).toBeFalsy();
+ expect(stop.stopped).toBe(false);
+ const records = (await readFile(log, "utf8")).trim().split("\n").map(line => JSON.parse(line) as Entry);
+ expect(records[0].env?.CUA_ESCAPE_STOP).toBe("0");
+ } finally { await driver.close(); }
+});
+
+test("a delayed refusal from before resume cannot stop the session again", async () => {
+ const { driver, stop, entries } = await start();
+ const stops = vi.fn(); stop.onStop(stops);
+ try {
+ await driver.call("list_windows", {});
+ const pending = driver.call("press_key", { key: "late-stop" }).catch(() => undefined);
+ await until(async () => (await entries()).some(entry => entry.arguments?.key === "late-stop"));
+ stop.stop();
+ stop.resume();
+ await pending;
+ expect(stop.stopped).toBe(false);
+ expect(stops).toHaveBeenCalledTimes(1);
+ expect((await driver.call("list_windows", {})).isError).toBeFalsy();
+ } finally { await driver.close(); }
+});
+
+test("a companion observing the stop is replaced when computer use resumes", async () => {
+ const { driver, stop } = await start();
+ const watcher = driver.companion()!;
+ try {
+ await driver.call("list_windows", {});
+ await watcher.call("press_key", { key: "escape-x3" });
+ expect(stop.stopped).toBe(true);
+ stop.resume();
+ expect((await watcher.call("list_windows", {})).isError).toBeFalsy();
+ expect(stop.stopped).toBe(false);
+ } finally { await watcher.close(); await driver.close(); }
+});
+
test("a refusal that names the stop is enough when the notification is missed", async () => {
const { driver, stop } = await start("refuse-only");
try {
diff --git a/test/image-settings-command.test.ts b/test/image-settings-command.test.ts
deleted file mode 100644
index f74c7e5..0000000
--- a/test/image-settings-command.test.ts
+++ /dev/null
@@ -1,43 +0,0 @@
-import type { ExtensionAPI, ExtensionCommandContext } from "@earendil-works/pi-coding-agent";
-import { expect, test, vi } from "vitest";
-import { CuaImageSettings } from "../src/cua-image-settings.ts";
-import { registerImageSettingsCommand } from "../src/image-settings-command.ts";
-
-type Command = { handler(args: string, ctx: ExtensionCommandContext): Promise<void> };
-function fixture() {
- const commands = new Map<string, Command>();
- const images = new CuaImageSettings(() => "bilinear");
- const notify = vi.fn();
- registerImageSettingsCommand({ registerCommand: (name: string, command: Command) => commands.set(name, command) } as unknown as ExtensionAPI, images);
- const ctx = { hasUI: true, ui: { notify } } as unknown as ExtensionCommandContext;
- return { images, notify, run: (input: string) => commands.get("computer-images")!.handler(input, ctx), commands };
-}
-
-test("the human image command changes future native captures without a filter switch", async () => {
- const f = fixture(); const old = f.images.settings();
- await f.run("cap 800"); await f.run("scale 0.5"); await f.run("hints off"); await f.run("guides all");
- expect(f.images.settings()).toMatchObject({ kernel: "bilinear", maxLongEdge: 800, scale: 0.5, hints: false,
- guides: { grid: true, action: true }, version: 4 });
- expect(old).toMatchObject({ maxLongEdge: 1280, scale: 1, version: 0 });
- expect(old.hints).toBeUndefined(); expect(old.guides).toBeUndefined();
- await f.run("reset"); expect(f.images.settings()).toEqual(old);
-});
-
-test("bad image command arguments preserve the full settings tuple", async () => {
- const f = fixture(); await f.run("cap 800"); const saved = f.images.settings();
- for (const input of ["cap 63", "cap 800.5", "cap Infinity", "scale 0", "scale NaN", "hints false", "hints off extra",
- "guides on", "kernel lanczos3", "mode yolo", "reset extra"]) {
- await f.run(input); expect(f.images.settings()).toEqual(saved);
- expect(f.notify.mock.calls.at(-1)?.[1]).toBe("warning");
- }
-});
-
-test("showing image settings does not change captures and instances stay independent", async () => {
- const first = fixture(); const second = fixture();
- await first.run("cap 600"); await first.run("guides grid");
- const saved = first.images.settings(); await first.run(""); await first.run("show");
- expect(first.images.settings()).toEqual(saved);
- expect(second.images.settings()).toMatchObject({ maxLongEdge: 1280, scale: 1, version: 0 });
- expect(first.commands.size).toBe(1);
- expect(first.notify.mock.calls.at(-1)?.[0]).toContain("600");
-});
diff --git a/test/preferences-controller.test.ts b/test/preferences-controller.test.ts
new file mode 100644
index 0000000..ed5d4c0
--- /dev/null
+++ b/test/preferences-controller.test.ts
@@ -0,0 +1,71 @@
+import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { afterEach, expect, test, vi } from "vitest";
+import type { ExtensionAPI, ExtensionCommandContext } from "@earendil-works/pi-coding-agent";
+import { CuaImageSettings } from "../src/cua-image-settings.ts";
+import { VisualSettings } from "../src/native-visuals.ts";
+import { HumanStop } from "../src/human-stop.ts";
+import { registerComputerMode } from "../src/mode.ts";
+import { PreferencesController } from "../src/preferences-controller.ts";
+
+const dirs: string[] = []; const agentDir = process.env.PI_CODING_AGENT_DIR;
+afterEach(async () => { if (agentDir === undefined) delete process.env.PI_CODING_AGENT_DIR; else process.env.PI_CODING_AGENT_DIR = agentDir; await Promise.all(dirs.splice(0).map(dir => rm(dir, { recursive: true, force: true }))); });
+test("a scoped preset preserves a legacy per-setting preview preference", async () => {
+ const root = await mkdtemp(path.join(tmpdir(), "cua-previews-")); dirs.push(root); process.env.PI_CODING_AGENT_DIR = path.join(root, "agent");
+ const pi = { on: vi.fn(), registerCommand: vi.fn(), appendEntry: vi.fn(), sendMessage: vi.fn() } as unknown as ExtensionAPI;
+ const ctx = { cwd: path.join(root, "project"), hasUI: true, isProjectTrusted: () => true, ui: { notify: vi.fn(), setStatus: vi.fn() }, sessionManager: { getBranch: () => [{ type: "custom", customType: "computer-use-visuals", data: { preset: "subtle", previews: false } }] } } as unknown as ExtensionCommandContext;
+ const visuals = new VisualSettings(); visuals.restore({ preset: "subtle", previews: false });
+ const prefs = new PreferencesController(new CuaImageSettings(), visuals, registerComputerMode(pi)); prefs.captureBaseline(ctx);
+ await prefs.set(ctx, "global", "preset", "full");
+ expect(visuals.current()).toMatchObject({ preset: "full", previews: false });
+});
+
+test("a session Safe selection is not widened by the global default on restore", async () => {
+ const root = await mkdtemp(path.join(tmpdir(), "cua-mode-")); dirs.push(root); process.env.PI_CODING_AGENT_DIR = path.join(root, "agent");
+ const branch: unknown[] = []; const events = new Map<string, (event: unknown, ctx: ExtensionCommandContext) => Promise<void>>();
+ const pi = { on: (name: string, handler: (event: unknown, ctx: ExtensionCommandContext) => Promise<void>) => events.set(name, handler), registerCommand: vi.fn(), appendEntry: (customType: string, data: unknown) => branch.push({ type: "custom", customType, data }), sendMessage: vi.fn() } as unknown as ExtensionAPI;
+ const ctx = { cwd: path.join(root, "project"), hasUI: true, isProjectTrusted: () => true, ui: { notify: vi.fn(), setStatus: vi.fn() }, sessionManager: { getBranch: () => branch } } as unknown as ExtensionCommandContext;
+ const mode = registerComputerMode(pi); const prefs = new PreferencesController(new CuaImageSettings(), new VisualSettings(), mode);
+ await prefs.set(ctx, "global", "mode", "yolo");
+ await mode.select("safe", ctx); expect(mode()).toBe("safe");
+ await events.get("session_tree")!({}, ctx); prefs.captureBaseline(ctx); await prefs.apply(ctx, true, true);
+ expect(mode()).toBe("safe");
+});
+test("a blocked Safe change is not persisted", async () => {
+ const root = await mkdtemp(path.join(tmpdir(), "cua-safe-")); dirs.push(root);
+ process.env.PI_CODING_AGENT_DIR = path.join(root, "agent");
+ const pi = { on: vi.fn(), registerCommand: vi.fn(), appendEntry: vi.fn(), sendMessage: vi.fn() } as unknown as ExtensionAPI;
+ const ctx = { cwd: path.join(root, "project"), hasUI: true, isProjectTrusted: () => true, ui: { notify: vi.fn(), setStatus: vi.fn() } } as unknown as ExtensionCommandContext;
+ const mode = registerComputerMode(pi, undefined, () => false);
+ const prefs = new PreferencesController(new CuaImageSettings(), new VisualSettings(), mode);
+ await expect(prefs.set(ctx, "project", "mode", "safe")).rejects.toThrow();
+ expect((await prefs.read(ctx)).effective.mode).toBeUndefined();
+ expect(mode()).toBe("yolo");
+});
+test("restores preferences across sessions, preserves legacy settings and ignores untrusted project overrides", async () => {
+ const root = await mkdtemp(path.join(tmpdir(), "cua-controller-")); dirs.push(root);
+ process.env.PI_CODING_AGENT_DIR = path.join(root, "agent"); const cwd = path.join(root, "project");
+ await mkdir(path.join(cwd, ".pi"), { recursive: true });
+ const config = path.join(cwd, ".pi", "computer-use.json"); await writeFile(config, JSON.stringify({ command: "driver", escapeStop: false }));
+ const ctx = { cwd, hasUI: true, isProjectTrusted: () => true, ui: { notify: vi.fn(), setStatus: vi.fn() } } as unknown as ExtensionCommandContext;
+ const make = () => {
+ const pi = { on: vi.fn(), registerCommand: vi.fn(), appendEntry: vi.fn(), sendMessage: vi.fn() } as unknown as ExtensionAPI;
+ const images = new CuaImageSettings(); const visuals = new VisualSettings(); const stop = new HumanStop();
+ const mode = registerComputerMode(pi, undefined, undefined, stop);
+ return { images, visuals, stop, mode, prefs: new PreferencesController(images, visuals, mode) };
+ };
+ const first = make();
+ await first.prefs.set(ctx, "global", "cap", 900);
+ await first.prefs.set(ctx, "project", "cap", 700);
+ await first.prefs.set(ctx, "global", "escape", true);
+ expect((await first.prefs.read(ctx)).effective.escape).toBe(false);
+ const next = make(); next.stop.stop(); await next.prefs.apply(ctx);
+ expect(next.images.settings().maxLongEdge).toBe(700); expect(next.stop.stopped).toBe(true);
+ await next.prefs.set(ctx, "project", "escape", undefined);
+ expect((await next.prefs.read(ctx)).effective.escape).toBe(true);
+ expect(JSON.parse(await readFile(config, "utf8"))).toEqual({ command: "driver" });
+ const untrusted = { ...ctx, isProjectTrusted: () => false } as ExtensionCommandContext;
+ await next.prefs.apply(untrusted);
+ expect(next.images.settings().maxLongEdge).toBe(900);
+});
diff --git a/test/scoped-mode.test.ts b/test/scoped-mode.test.ts
new file mode 100644
index 0000000..4b617ba
--- /dev/null
+++ b/test/scoped-mode.test.ts
@@ -0,0 +1,20 @@
+import { expect, test, vi } from "vitest";
+import type { ExtensionAPI, ExtensionCommandContext } from "@earendil-works/pi-coding-agent";
+import { registerComputerMode } from "../src/mode.ts";
+import { HumanStop } from "../src/human-stop.ts";
+
+test("restoring a scoped mode does not resume a stopped computer and respects the Safe guard", async () => {
+ const stop = new HumanStop();
+ const pi = { on: vi.fn(), registerCommand: vi.fn(), appendEntry: vi.fn(), sendMessage: vi.fn() } as unknown as ExtensionAPI;
+ const ctx = { hasUI: true, ui: { setStatus: vi.fn(), notify: vi.fn() } } as unknown as ExtensionCommandContext;
+ let safeAllowed = false;
+ const mode = registerComputerMode(pi, async () => {}, () => safeAllowed, stop);
+ stop.stop();
+ expect(await mode.applyPreference("safe", ctx)).toBe(false);
+ expect(mode()).toBe("yolo");
+ expect(stop.stopped).toBe(true);
+ safeAllowed = true;
+ expect(await mode.applyPreference("safe", ctx)).toBe(true);
+ expect(mode()).toBe("safe");
+ expect(stop.stopped).toBe(true);
+});
diff --git a/test/scoped-preferences.test.ts b/test/scoped-preferences.test.ts
new file mode 100644
index 0000000..f14249e
--- /dev/null
+++ b/test/scoped-preferences.test.ts
@@ -0,0 +1,55 @@
+import { mkdtemp, readFile, rm, writeFile, mkdir } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { execFile } from "node:child_process";
+import { promisify } from "node:util";
+import { afterEach, expect, test } from "vitest";
+import { readScopedPreferences, writeScopedPreference } from "../src/scoped-preferences.ts";
+
+const dirs: string[] = [];
+const oldAgentDir = process.env.PI_CODING_AGENT_DIR;
+afterEach(async () => { if (oldAgentDir === undefined) delete process.env.PI_CODING_AGENT_DIR; else process.env.PI_CODING_AGENT_DIR = oldAgentDir; await Promise.all(dirs.splice(0).map(dir => rm(dir, { recursive: true, force: true }))); });
+async function fixture() {
+ const root = await mkdtemp(path.join(tmpdir(), "cua-scopes-")); dirs.push(root);
+ process.env.PI_CODING_AGENT_DIR = path.join(root, "agent");
+ return { a: path.join(root, "a"), b: path.join(root, "b"), root };
+}
+test("project preferences override global per setting and clearing restores inheritance", async () => {
+ const { a, b } = await fixture();
+ await writeScopedPreference(a, "global", "cap", 900);
+ await writeScopedPreference(a, "global", "hints", false);
+ await writeScopedPreference(a, "project", "cap", 700);
+ expect((await readScopedPreferences(a)).effective).toEqual({ cap: 700, hints: false });
+ expect((await readScopedPreferences(a)).sources).toEqual({ cap: "project", hints: "global" });
+ expect((await readScopedPreferences(b)).effective).toEqual({ cap: 900, hints: false });
+ expect((await readScopedPreferences(a, false)).effective).toEqual({ cap: 900, hints: false });
+ await writeScopedPreference(a, "project", "cap", undefined);
+ expect((await readScopedPreferences(a)).effective.cap).toBe(900);
+});
+test("simultaneous writes keep unrelated settings", async () => {
+ const { a } = await fixture();
+ await Promise.all([
+ writeScopedPreference(a, "global", "mode", "safe"),
+ writeScopedPreference(a, "global", "escape", false),
+ writeScopedPreference(a, "global", "cap", 800),
+ ]);
+ expect((await readScopedPreferences(a)).effective).toEqual({ mode: "safe", escape: false, cap: 800 });
+});
+test("concurrent processes do not discard each other's settings", async () => {
+ const { a } = await fixture();
+ const run = promisify(execFile);
+ const moduleUrl = new URL("../src/scoped-preferences.ts", import.meta.url).href;
+ const entries = [["mode", "safe"], ["escape", false], ["cap", 800]] as const;
+ await Promise.all(entries.map(([key, value]) => run(process.execPath, ["--experimental-strip-types", "--input-type=module", "-e",
+ `import { writeScopedPreference } from ${JSON.stringify(moduleUrl)}; await writeScopedPreference(${JSON.stringify(a)}, "global", ${JSON.stringify(key)}, ${JSON.stringify(value)});`], { env: process.env })));
+ expect((await readScopedPreferences(a)).effective).toEqual({ mode: "safe", escape: false, cap: 800 });
+});
+test("invalid settings and malformed files are never overwritten", async () => {
+ const { a } = await fixture();
+ await expect(writeScopedPreference(a, "global", "cap", 2)).rejects.toThrow();
+ await mkdir(path.join(a, ".pi"), { recursive: true });
+ const file = path.join(a, ".pi", "computer-use-settings.json");
+ await writeFile(file, "broken");
+ await expect(writeScopedPreference(a, "project", "mode", "safe")).rejects.toThrow();
+ expect(await readFile(file, "utf8")).toBe("broken");
+});
diff --git a/test/visual-commands.integration.test.ts b/test/visual-commands.integration.test.ts
index 447c471..485fb7d 100644
--- a/test/visual-commands.integration.test.ts
+++ b/test/visual-commands.integration.test.ts
@@ -5,7 +5,7 @@ import path from "node:path";
import { fileURLToPath } from "node:url";
import { expect, test } from "vitest";
-test("human visual commands persist preferences and restore the selected Pi branch without model work", async () => {
+test("visual preferences restore the selected Pi branch without model work", async () => {
const cwd = await mkdtemp(path.join(tmpdir(), "computer-visual-commands-"));
const client = new RpcClient({ cwd,
cliPath: path.join(path.dirname(fileURLToPath(import.meta.resolve("@earendil-works/pi-coding-agent"))), "cli.js"),
@@ -22,21 +22,15 @@ test("human visual commands persist preferences and restore the selected Pi bran
};
try {
await client.start();
- const commands = await client.getCommands();
- expect(commands.filter(command => command.name === "computer-visuals")).toHaveLength(1);
- expect(commands.filter(command => command.name === "computer-images")).toHaveLength(1);
- expect(await client.prompt("/computer-visuals subtle")).toBe("handled");
+ expect(await client.prompt('/visual-set {"preset":"subtle"}')).toBe("handled");
const saved = (await client.getEntries()).entries.filter(entry => entry.type === "custom" && entry.customType === "computer-use-visuals");
expect(saved).toHaveLength(1);
expect((await probe()).visuals).toMatchObject({ preset: "subtle", previews: false });
- expect(await client.prompt('/computer-visuals {"preset":"full","keyboard":false,"quality":"high"}')).toBe("handled");
+ expect(await client.prompt('/visual-set {"preset":"full","keyboard":false,"quality":"high"}')).toBe("handled");
expect((await probe()).visuals).toMatchObject({ preset: "full", keyboard: false, quality: "high" });
expect(await client.prompt(`/visual-navigate ${saved[0].id}`)).toBe("handled");
expect((await probe()).visuals).toMatchObject({ preset: "subtle", previews: false, keyboard: true, quality: "balanced" });
- expect(await client.prompt("/computer-images cap 960")).toBe("handled");
- expect(await client.prompt("/computer-images hints off")).toBe("handled");
- expect((await probe()).images).toMatchObject({ maxLongEdge: 960, hints: false });
- expect(await client.prompt('/computer-visuals {"constructor":false}')).toBe("handled");
+ expect(await client.prompt('/visual-set {"constructor":false}')).toBe("handled");
expect((await probe()).visuals).toMatchObject({ preset: "subtle", previews: false });
expect((await client.getEntries()).entries.filter(entry => entry.type === "custom" && entry.customType === "computer-use-visuals")).toHaveLength(2);
expect((await client.getMessages()).filter(message => message.role === "assistant")).toHaveLength(0);
diff --git a/tsconfig.json b/tsconfig.json
index 8cc2270..843ce26 100644
--- a/tsconfig.json
+++ b/tsconfig.json
@@ -9,5 +9,5 @@
"skipLibCheck": true,
"types": ["node"]
},
- "include": ["src/**/*.ts", "scripts/**/*.ts", "test/**/*.ts", "vitest*.ts"]
+ "include": ["src/**/*.ts", "scripts/**/*.ts", "demo/**/*.ts", "test/**/*.ts", "vitest*.ts"]
}