Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.
TypeScript accessibilityagent-toolsai-agentbrowser-automationcomputer-usecuadesktop-automationelectronpipi-coding-agentpi-extensionpi-packageplaywrighttypescriptui-automationvisionwebview2windowswindows-automationwsl
Route every computer address to the machine that issued it (LPT-929, LPT-924)
alexshpunt-clanker[bot]7d3fa3b9f6d2
diff --git a/docs/computer-resources.md b/docs/computer-resources.mdindex 3571493..132557c 100644--- a/docs/computer-resources.md+++ b/docs/computer-resources.md@@ -13,11 +13,14 @@ The current desktop scope is Windows, with Pi running on it directly or in its WEvery machine is addressed the same way: `computer:machine/local` is this one, and `computer:machine/<name>` is another Windows machine configured under `machines` (see [Work on another machine](#work-on-another-machine)). A machine starts on its first use; nothing needs connecting. Read `computer:session` for the machines, sandboxes and sessions started so far.+No address falls back to this machine. Window, window-state, observation and sequence addresses belong to the machine (or sandbox) that issued them, and every Read, Search and Insert on them runs there. Display addresses name their machine. An address that no machine of this session issued, or that was released, is refused with the reason before anything is sent. Every receipt names its machine in its first line, for example `Computer observed (yolo) on lab`.+| Address | Use || --- | --- || `computer:machine/<name>/windows` | Search its windows || `computer:machine/<name>/apps` | Search its installed apps || `computer:machine/<name>/displays` | Read its displays |+| `computer:machine/<name>/display/primary` | Read its main display; other displays come from `displays` as `computer:machine/<name>/display/<ref>` || `computer:machine/<name>` | Insert commands: `open`, `open_browser`, `open_sandbox`, `open_session`, `webview2_debugging`, file, clipboard, recording and `close` |Search with a name/title substring, or `regex:<pattern>` for a case-insensitive regular expression; sandbox addresses take the same queries. Results include the available native process metadata and addressable app/window resources; an app address opens that app on the machine that listed it. Discovery launches nothing. Display discovery takes no text predicate.@@ -28,7 +31,7 @@ Insert into a machine address with `text` containing `{"action":"open","app":"ExApps with their own scripting (Blender, for example) can build content while computer use checks the result: `put` the script into the session folder, open the app with the script's returned path in `args` (`--factory-startup --python <path>` for Blender), then Read the window with `ocr` and `state`.-Read a returned `computer:window/...` for its native accessibility structure and small overview PNG. Read `computer:display/primary`, or a returned display address, for readonly display pixels; add `capture:{"maxLongEdge":320}` for another size. Successful overviews issue `computer:observation/...` addresses. References are session-local and disappear on close/reload.+Read a returned `computer:window/...` for its native accessibility structure and small overview PNG. Read `computer:machine/<name>/display/primary`, or a display address returned by that machine's `displays`, for readonly display pixels; add `capture:{"maxLongEdge":320}` for another size. Successful overviews issue `computer:observation/...` addresses. References are session-local and disappear on close/reload.Read views:@@ -206,7 +209,7 @@ Use `computer:machine/<name>` with the configured name, for example `computer:ma| --- | --- || Search `computer:machine/lab/windows` | That machine's windows, as `computer:window/...` addresses || Search `computer:machine/lab/apps` | Its installed apps, as `computer:app/...` addresses that open there |-| Read `computer:machine/lab/displays` | Its displays, as `computer:display/...` addresses; Insert `resize` into one to change its resolution |+| Read `computer:machine/lab/displays` | Its displays, as `computer:machine/lab/display/...` addresses; Insert `resize` into one to change its resolution || Insert `{"action":"open","app":"notepad"}` into `computer:machine/lab` | Start an app there and return its window || Insert `{"action":"open_browser"}` | Launch an owned browser there (optional `browser`, `url`) || Insert `{"action":"open_sandbox"}` | Start a Windows Sandbox on that machine (with `import`/`export` folders of that machine) |diff --git a/docs/selected-displays.md b/docs/selected-displays.mdindex 8f167f5..1514a01 100644--- a/docs/selected-displays.md+++ b/docs/selected-displays.md@@ -22,7 +22,7 @@ A native topology token covers monitor identity, physical bounds, work area andSelected source metadata reports `selected_display` or `selected_display_region`, display-local image/crop coordinates and host-global native bounds separately. Only exact native display reads report monitor membership. Window membership remains unknown; proximity to a monitor is not evidence.-`computer:display/primary` keeps the existing primary-read route. It does not select an enumerated monitor by a permanent identity. Temporal sequences accept only windows or the primary display; selected-monitor sequences are outside this change.+`computer:machine/<name>/display/primary` keeps the primary-read route of that machine. It does not select an enumerated monitor by a permanent identity. A listed display's address is `computer:machine/<name>/display/<ref>`; its reference is unknown on any other machine. Temporal sequences accept windows, a machine's primary display and its listed displays.Safe denies discovery, overview and detail. Revoked or cancelled reads withhold metadata/pixels. Paint freshness remains unverified. No selected-display input, window relocation, virtual-desktop management or concurrent-agent arbitration is added.diff --git a/docs/temporal-capture.md b/docs/temporal-capture.mdindex f0a20b9..e596f35 100644--- a/docs/temporal-capture.md+++ b/docs/temporal-capture.md@@ -2,13 +2,13 @@See [the resource guide](computer-resources.md) for the current addresses, views and commands.-A `sequence` Read captures one authorized Windows window or the primary display. Browser windows use the same native path. It does not encode video, raise a window, walk its accessibility tree or grant input authority.+A `sequence` Read captures one authorized Windows window, or a machine's primary or listed display, on the machine that issued its address. Browser windows use the same native path. It does not encode video, raise a window, walk its accessibility tree or grant input authority.## Capture, then selectCapture returns metadata only and sends no image. Capture and selection are separate decisions: read the timestamps, count and stop reason before choosing frames from the returned sequence.-Capture the primary display instead of a window for a read-only display sequence in YOLO. Safe denies display and unowned window captures. Stored frame selection checks current access and the window's lifetime again. Revocation withholds metadata and pixels. A closed or recreated window cannot return its retained frames.+Capture a display (`computer:machine/<name>/display/primary` or one listed by its `displays`) instead of a window for a read-only display sequence in YOLO. Frames are selected through the returned `computer:sequence/...` address, which runs on the machine that captured them. Safe denies display and unowned window captures. Stored frame selection checks current access and the window's lifetime again. Revocation withholds metadata and pixels. A closed or recreated window cannot return its retained frames.## Bounds and settingsdiff --git a/src/computer-address.ts b/src/computer-address.tsindex 044c8fe..ec53e8a 100644--- a/src/computer-address.ts+++ b/src/computer-address.ts@@ -5,10 +5,12 @@ export type ComputerAddress =| { kind: "session"; source: string }/** A machine by its name: `local` for this one, otherwise a name under `machines` in .pi/computer-use.json. */| { kind: "machine"; source: string; ref: string; part?: MachinePart }- | { kind: "app" | "window" | "display" | "sequence" | "window-state" | "sandbox"; source: string; ref: string }+ | { kind: "app" | "window" | "sequence" | "window-state" | "sandbox"; source: string; ref: string }+ /** A display of one machine: `primary`, or a reference listed by that machine's displays. */+ | { kind: "display"; source: string; machine: string; ref: string }| { kind: "observation"; source: string; ref: string; hint?: string; delivery?: "background" | "foreground"; delayMs?: number };-const references = new Set(["app", "window", "display", "observation", "sequence", "window-state", "sandbox", "machine"]);+const references = new Set(["app", "window", "observation", "sequence", "window-state", "sandbox", "machine"]);const parts = new Set(["windows", "apps", "displays"]);/** Resolve this namespace or reject malformed owned addresses before any native work. */@@ -24,15 +26,24 @@ export function parseComputerAddress(source: string): ComputerAddress | undefineif (segments.length !== 1 || query !== undefined) return invalid();return { kind, source };}+ if (kind === "display")+ throw new Error("Display addresses name their machine: computer:machine/<name>/display/primary, or a display listed by Read computer:machine/<name>/displays.");+ const reference = (segment: string) => {+ let ref: string;+ try { ref = decodeURIComponent(segment); } catch { return invalid(); }+ return /^[a-zA-Z0-9_-]{1,160}$/.test(ref) ? ref : invalid();+ };+ if (kind === "machine" && segments.length === 4 && segments[2] === "display") {+ if (query !== undefined) return invalid();+ return { kind: "display", source, machine: reference(segments[1]), ref: reference(segments[3]) };+ }const machinePart = kind === "machine" && segments.length === 3 ? segments[2] : undefined;if (!references.has(kind) || (segments.length !== 2 && !(machinePart && parts.has(machinePart)))) return invalid();- let ref: string;- try { ref = decodeURIComponent(segments[1]); } catch { return invalid(); }- if (!/^[a-zA-Z0-9_-]{1,160}$/.test(ref)) return invalid();+ const ref = reference(segments[1]);if (kind !== "observation") {if (query !== undefined) return invalid();if (kind === "machine") return { kind, ref, source, ...(machinePart ? { part: machinePart as MachinePart } : {}) };- return { kind: kind as "app" | "window" | "display" | "sequence" | "window-state" | "sandbox", ref, source };+ return { kind: kind as "app" | "window" | "sequence" | "window-state" | "sandbox", ref, source };}const result: Extract<ComputerAddress, { kind: "observation" }> = { kind, ref, source };const params = new URLSearchParams(query);@@ -49,8 +60,15 @@ export function parseComputerAddress(source: string): ComputerAddress | undefine}/** Build a computer address from an opaque session-issued reference, or a machine's address from its name. */-export function computerSource(kind: "app" | "window" | "display" | "observation" | "sequence" | "window-state" | "sandbox" | "machine", ref: string, part?: MachinePart): string {+export function computerSource(kind: "app" | "window" | "observation" | "sequence" | "window-state" | "sandbox" | "machine", ref: string, part?: MachinePart): string {const source = `computer:${kind}/${encodeURIComponent(ref)}${part ? `/${part}` : ""}`;parseComputerAddress(source);return source;}++/** The address of one machine's display: `primary` or a reference its displays listing issued. */+export function displaySource(machine: string, ref: string): string {+ const source = `computer:machine/${encodeURIComponent(machine)}/display/${encodeURIComponent(ref)}`;+ parseComputerAddress(source);+ return source;+}diff --git a/src/computer-resources.ts b/src/computer-resources.tsindex 0877330..5ecb8ff 100644--- a/src/computer-resources.ts+++ b/src/computer-resources.ts@@ -1,5 +1,5 @@import { randomUUID } from "node:crypto";-import { parseComputerAddress, computerSource, type ComputerAddress } from "./computer-address.ts";+import { parseComputerAddress, computerSource, displaySource, type ComputerAddress } from "./computer-address.ts";import { parseComputerViews, type ComputerViews } from "./computer-views.ts";import { checkedName, type ProcessHealth } from "./machine-host.ts";import { mkdir, readFile, rm, writeFile } from "node:fs/promises";@@ -24,7 +24,7 @@ import { SETUP_PARTS, SetupError, runSetup, type SetupPart } from "./setup.ts";export const LOCAL_MACHINE = "local";/** How to use a machine address: the same on every machine. */export function machineGuide(source: string): string[] {- return [`Search ${source}/windows for its windows and ${source}/apps for its installed apps; Read ${source}/displays for its displays (Insert resize into a display address to change its resolution).`,+ return [`Search ${source}/windows for its windows and ${source}/apps for its installed apps; Read ${source}/displays for its displays and ${source}/display/primary for its main display (Insert resize into a display address to change its resolution).`,`Insert open, open_browser, open_sandbox, open_session, webview2_debugging, put, get, files, clipboard_read, clipboard_write, record_start, record_stop, audio_record or close into ${source}.`];}/** A rejected request whose validation completed before native dispatch. */@@ -66,7 +66,13 @@ const SETTLE_MS = 300;* follow each other without a pause: each takes about 35 ms, so an animation still differs between them.*/const STEADY_MS = 1_000;-type ObservationTarget = { window_ref: string; display?: never } | { display: string; window_ref?: never };+/** What an observation shows: a window, or one machine's display (display references are per machine; each has a primary). */+type ObservationTarget = { window_ref: string; display?: never; machine?: never } | { display: string; machine: string; window_ref?: never };+const sameTarget = (a: ObservationTarget, b: ObservationTarget) => a.window_ref === b.window_ref && a.display === b.display && a.machine === b.machine;+/** Where a request runs: the session that issued its reference, with that session's machine and sandbox. */+interface Owner { session: ComputerSession; machine: string; sandbox?: string }+/** The machine and sandbox a receipt names. */+type Place = { machine: string; sandbox?: string };interface Snapshot { target: ObservationTarget; result: ComputerResult }/** A running sandbox with its own guest session. */export interface OpenedSandbox {@@ -273,17 +279,69 @@ export class ComputerResources {private readonly closeSession: () => Promise<ComputerResult>, private readonly openSandbox?: SandboxFactory,private readonly startMachine?: MachineFactory, private readonly getMode: () => ComputerMode = () => "yolo") {}- /** The session that issued a window or display reference: a sandbox's or another machine's, otherwise this host's. */- private async sessionFor(cwd: string, target?: ObservationTarget | { window_ref?: string; display?: string }): Promise<ComputerSession> {- for (const environment of [...this.sandboxes.values(), ...this.machines.values()])- if (target?.window_ref ? environment.session.hasWindow(target.window_ref) : target?.display && environment.session.hasDisplay(target.display)) return environment.session;- return this.getSession(cwd);+ /**+ * The session that issued a reference, with its machine. Nothing falls back to this machine: a reference that no+ * session of this computer use holds is refused before anything is sent.+ */+ private async owner(cwd: string, holds: (session: ComputerSession) => boolean, unknown: string): Promise<Owner> {+ for (const sandbox of this.sandboxes.values())+ if (holds(sandbox.session)) return { session: sandbox.session, machine: sandbox.machine ?? LOCAL_MACHINE, sandbox: sandbox.ref };+ for (const [machine, opened] of this.machines) if (holds(opened.session)) return { session: opened.session, machine };+ const local = await this.getSession(cwd);+ if (holds(local)) return { session: local, machine: LOCAL_MACHINE };+ throw new ComputerRequestError(`${unknown} No machine or sandbox of this computer use holds it, so nothing was sent.`);+ }+ /** The owner of a window, or of one machine's display. */+ private async route(cwd: string, target: ObservationTarget, signal?: AbortSignal): Promise<Owner> {+ if (target.window_ref !== undefined)+ return this.owner(cwd, session => session.hasWindow(target.window_ref), "Unknown or closed window reference. Search computer:machine/<name>/windows again.");+ // That machine's session refuses a display reference it did not list, after its own mode check, without native work.+ return { session: await this.machineSession(cwd, target.machine, signal), machine: target.machine };+ }+ /** The owner of a window, display, window state, observation or sequence address. */+ private async routeAddress(cwd: string, address: ComputerAddress, signal?: AbortSignal): Promise<Owner> {+ if (address.kind === "window") return this.route(cwd, { window_ref: address.ref }, signal);+ if (address.kind === "display") return this.route(cwd, { display: address.ref, machine: address.machine }, signal);+ if (address.kind === "window-state") {+ const state = this.windowStates.get(address.ref);+ if (!state) throw new ComputerRequestError("Window state reference expired. Read the window with state view again.");+ return this.route(cwd, { window_ref: state.window_ref }, signal);+ }+ if (address.kind === "observation") return this.route(cwd, this.snapshot(address.ref).target, signal);+ if (address.kind === "sequence")+ return this.owner(cwd, session => session.hasSequence(address.ref), "Sequence is unavailable or evicted. Capture again; sequences do not survive reload or close.");+ throw new ComputerRequestError("This address names no window, display, observation or sequence.");}- private windowOf(address: ComputerAddress): { window_ref?: string; display?: string } | undefined {- return address.kind === "window" ? { window_ref: address.ref }- : address.kind === "display" ? { display: address.ref }- : address.kind === "window-state" ? { window_ref: this.windowStates.get(address.ref)?.window_ref }- : address.kind === "observation" ? this.snapshots.get(address.ref)?.target : undefined;+ /** The machine and sandbox an address runs on, found without starting anything; none for the whole session. */+ private place(address: ComputerAddress): Place | undefined {+ const holder = (holds: (session: ComputerSession) => boolean): Place => {+ for (const sandbox of this.sandboxes.values()) if (holds(sandbox.session)) return { machine: sandbox.machine ?? LOCAL_MACHINE, sandbox: sandbox.ref };+ for (const [machine, opened] of this.machines) if (holds(opened.session)) return { machine };+ return { machine: LOCAL_MACHINE };+ };+ switch (address.kind) {+ case "session": return undefined;+ case "machine": return { machine: address.ref };+ case "display": return { machine: address.machine };+ case "app": { const app = this.apps.get(address.ref); return app && { machine: app.machine }; }+ case "sandbox": { const sandbox = this.sandboxes.get(address.ref); return sandbox && { machine: sandbox.machine ?? LOCAL_MACHINE, sandbox: sandbox.ref }; }+ case "sequence": return holder(session => session.hasSequence(address.ref));+ case "window": return holder(session => session.hasWindow(address.ref));+ case "window-state": { const state = this.windowStates.get(address.ref); return state && holder(session => session.hasWindow(state.window_ref)); }+ case "observation": {+ const target = this.snapshots.get(address.ref)?.target;+ return !target ? undefined : target.machine !== undefined ? { machine: target.machine } : holder(session => session.hasWindow(target.window_ref!));+ }+ }+ }+ /** Name the machine (and sandbox) a request ran on in its receipt. */+ private stamp(place: Place | undefined, result: ComputerResult): ComputerResult {+ return place && result.details.machine === undefined+ ? { ...result, details: { ...result.details, machine: place.machine, ...(place.sandbox ? { sandbox: place.sandbox } : {}) } } : result;+ }+ /** The session of a window, or of one machine's display. */+ private async sessionFor(cwd: string, target: ObservationTarget, signal?: AbortSignal): Promise<ComputerSession> {+ return (await this.route(cwd, target, signal)).session;}/** A configured remote machine, started on its first use. */private async machine(cwd: string, name: string, signal?: AbortSignal): Promise<OpenedMachine> {@@ -302,10 +360,10 @@ export class ComputerResources {private async machineSession(cwd: string, name: string, signal?: AbortSignal): Promise<ComputerSession> {return name === LOCAL_MACHINE ? this.getSession(cwd) : (await this.machine(cwd, name, signal)).session;}- /** Forget a released session's observations and window states. */- private forget(session: ComputerSession): void {+ /** Forget a released session's observations and window states, and the display observations of its machine. */+ private forget(session: ComputerSession, machine?: string): void {for (const [id, entry] of this.snapshots)- if (entry.target.window_ref ? session.hasWindow(entry.target.window_ref) : session.hasDisplay(entry.target.display!)) this.snapshots.delete(id);+ if (entry.target.window_ref !== undefined ? session.hasWindow(entry.target.window_ref) : entry.target.machine === machine) this.snapshots.delete(id);for (const [id, entry] of this.windowStates) if (session.hasWindow(entry.window_ref)) this.windowStates.delete(id);}/**@@ -318,23 +376,23 @@ export class ComputerResources {error => [computerSource("sandbox", sandbox.ref), failure(error, this.getMode())] as const)));let own: ComputerResult;if (name === LOCAL_MACHINE) {- for (const [id, entry] of this.snapshots) if (!this.remoteOwner(entry.target)) this.snapshots.delete(id);- for (const [id, entry] of this.windowStates) if (!this.remoteOwner({ window_ref: entry.window_ref })) this.windowStates.delete(id);+ for (const [id, entry] of this.snapshots)+ if (entry.target.window_ref !== undefined ? !this.remoteWindow(entry.target.window_ref) : entry.target.machine === LOCAL_MACHINE) this.snapshots.delete(id);+ for (const [id, entry] of this.windowStates) if (!this.remoteWindow(entry.window_ref)) this.windowStates.delete(id);own = await this.closeSession().catch(error => failure(error, this.getMode()));} else {const machine = this.machines.get(name);if (!machine) return report(this.getMode(), [], `${computerSource("machine", name)} was not started.`);this.machines.delete(name);- this.forget(machine.session);+ this.forget(machine.session, name);own = await machine.close().catch(error => failure(error, this.getMode()));}for (const [ref, app] of this.apps) if (app.machine === name) this.apps.delete(ref);return report(this.getMode(), [...parts, [computerSource("machine", name), own]]);}- /** Whether a remote machine or a sandbox issued this window or display reference. */- private remoteOwner(target: { window_ref?: string; display?: string }): boolean {- return [...this.sandboxes.values(), ...this.machines.values()].some(environment =>- target.window_ref ? environment.session.hasWindow(target.window_ref) : target.display !== undefined && environment.session.hasDisplay(target.display));+ /** Whether a remote machine or a sandbox issued this window reference. */+ private remoteWindow(ref: string): boolean {+ return [...this.sandboxes.values(), ...this.machines.values()].some(environment => environment.session.hasWindow(ref));}/** Run one launch so session close and Pi shutdown can wait for it; a launch cancelled after success is undone. */private async launch<T extends { close(): Promise<unknown> }>(start: Promise<T>, signal?: AbortSignal): Promise<T> {@@ -393,8 +451,7 @@ export class ComputerResources {// A delivered DOM action carries the page's next snapshot, which becomes the current observation.const observed = result.details.status === "observed" || result.details.status === "delivered" && result.details.dom !== undefined;if (!observed || typeof id !== "string") return result;- for (const [old, entry] of this.snapshots)- if (entry.target.window_ref === target.window_ref && entry.target.display === target.display) this.snapshots.delete(old);+ for (const [old, entry] of this.snapshots) if (sameTarget(entry.target, target)) this.snapshots.delete(old);this.snapshots.set(id, { target, result });if (target.window_ref !== undefined && result.details.ui) {this.lastUi.delete(target.window_ref);@@ -422,7 +479,7 @@ export class ComputerResources {if (address.kind === "observation") {const snapshot = this.snapshot(address.ref);const source = snapshot.target.window_ref !== undefined ? computerSource("window", snapshot.target.window_ref)- : computerSource("display", snapshot.target.display!);+ : displaySource(snapshot.target.machine!, snapshot.target.display!);return `# Computer observation\nSource: ${source}\nObservation: ${computerSource("observation", address.ref)}\n` +(snapshot.result.details.dom ? "Playwright DOM snapshot. Insert one JSON action with a target (aria ref or locator); Read element:crop|highlight,<ref> or image for page pixels. Search filters this snapshot. Write and native input are unavailable here.": snapshot.target.window_ref !== undefined ? "Input needs this still-current observation. Insert sends JSON commands; Write sends literal text."@@ -437,25 +494,30 @@ export class ComputerResources {if (address.kind === "session") {const machines = [LOCAL_MACHINE, ...this.machines.keys()].map(name => `- ${computerSource("machine", name)}`);const sandboxes = [...this.sandboxes.values()].map(sandbox => `- ${computerSource("sandbox", sandbox.ref)} on ${computerSource("machine", sandbox.machine ?? LOCAL_MACHINE)}`);- return ["# Computer use", "Machines (computer:machine/local is this one; a machine configured under machines in .pi/computer-use.json starts on its first use):",+ return ["# Computer use", "Machines (computer:machine/local is this one; a machine configured under machines in .pi/computer-use.json starts on its first use). Every window, display, observation and sequence address belongs to the machine that issued it, and every receipt names that machine:",...machines, ...(sandboxes.length ? ["Sandboxes and isolated sessions:", ...sandboxes] : []),"Insert {\"action\":\"close\"} here to release every machine, browser, sandbox and session this computer use started; the result lists each part."].join("\n");}return `# Computer resource\nSource: ${address.source}\nUse Read for native observation, Search for discovery, and Insert for explicit commands. No native capture occurred during descriptor resolution.`;}- /** Acquire a requested native projection, or inspect one explicitly issued observation. */+ /** Acquire a requested native projection, or inspect one explicitly issued observation; the receipt names its machine. */async read(address: ComputerAddress, cwd: string, views?: readonly string[], signal?: AbortSignal): Promise<ComputerResult | undefined> {+ const place = this.place(address);+ const result = await this.readAddress(address, cwd, views, signal);+ return result && this.stamp(place, result);+ }+ private async readAddress(address: ComputerAddress, cwd: string, views?: readonly string[], signal?: AbortSignal): Promise<ComputerResult | undefined> {const view = parseComputerViews(views);if (address.kind !== "window" && address.kind !== "observation") {if (view.ocr) throw new ComputerRequestError("ocr reads a window, one of its observations or a region of one.");return this.readNative(address, cwd, views, view, signal);}- const session = await this.sessionFor(cwd, this.windowOf(address));const snapshot = address.kind === "observation" ? this.snapshots.get(address.ref) : undefined;const windowRef = address.kind === "window" ? address.ref : snapshot?.target.window_ref;if (view.ocr && windowRef === undefined) throw new ComputerRequestError("ocr reads window observations; display observations have no kept overview pixels.");if (view.changes && address.kind !== "window") throw new ComputerRequestError("changes reads a window address: it takes a fresh capture and compares it with the window's previous observation.");+ const session = (await this.routeAddress(cwd, address, signal)).session;const baseline = view.changes && windowRef !== undefined ? this.lastUi.get(windowRef) : undefined;// The pixel baseline is the window's latest capture with pixels; the fresh capture keeps its overview size.const frame = view.changes && windowRef !== undefined && view.kind === "overview" && view.capture.granularity !== "ui" ? session.windowFrame(windowRef) : undefined;@@ -528,7 +590,8 @@ export class ComputerResources {if (address.kind === "sandbox") this.sandbox(address.ref);return undefined;}- const session = await this.sessionFor(cwd, this.windowOf(address));+ const owner = await this.routeAddress(cwd, address, signal);+ const session = owner.session;if (address.kind === "window-state") {if (views?.length) throw new ComputerRequestError("Cached window state has no capture or UI views. Read the window with state view for a fresh state.");const state = this.windowStates.get(address.ref);@@ -542,12 +605,12 @@ export class ComputerResources {}if (view.kind === "sequence") {if (address.kind !== "window" && address.kind !== "display") throw new ComputerRequestError("Capture a sequence from a window/display address, not an observation.");- if ([...this.sandboxes.values()].some(sandbox => sandbox.session === session)) throw new ComputerRequestError("Temporal capture is not available for sandbox windows.");- const target = address.kind === "window" ? { window_ref: address.ref } : { display: address.ref };+ if (owner.sandbox) throw new ComputerRequestError("Temporal capture is not available for sandbox windows.");+ const target: ObservationTarget = address.kind === "display" ? { display: address.ref, machine: address.machine } : { window_ref: address.ref };if (target.window_ref) for (const [id, entry] of this.windowStates)if (entry.window_ref === target.window_ref) this.windowStates.delete(id);- for (const [id, entry] of this.snapshots) if (entry.target.window_ref === target.window_ref && entry.target.display === target.display) this.snapshots.delete(id);- return session.captureSequence(target, view.options, signal);+ for (const [id, entry] of this.snapshots) if (sameTarget(entry.target, target)) this.snapshots.delete(id);+ return session.captureSequence(address.kind === "window" ? { window_ref: address.ref } : { display: address.ref }, view.options, signal);}if (address.kind === "sequence") {if (view.kind !== "frames") throw new ComputerRequestError("Select sequence images explicitly with frames:first|last|endpoints|changed|all or frames:{\"indexes\":[0]}.");@@ -573,7 +636,7 @@ export class ComputerResources {if (view.kind !== "overview" || Object.keys(view.capture).some(key => key !== "maxLongEdge" && key !== "scale"))throw new ComputerRequestError("A display overview accepts only capture:{\"maxLongEdge\":...,\"scale\":...}; use its observation for region detail.");const { maxLongEdge, scale } = view.capture;- return this.remember({ display: address.ref }, await session.observeDisplay(address.ref, signal, undefined,+ return this.remember({ display: address.ref, machine: address.machine }, await session.observeDisplay(address.ref, signal, undefined,{ ...(maxLongEdge !== undefined ? { maxLongEdge } : {}), ...(scale !== undefined ? { scale } : {}) }));}if (address.kind !== "observation") throw new ComputerRequestError("Use an existing supported computer resource.");@@ -626,8 +689,12 @@ export class ComputerResources {return snapshot.result;}- /** Search permitted discovery metadata or the current collected UI without another native walk. */+ /** Search permitted discovery metadata or the current collected UI without another native walk; the receipt names its machine. */async search(address: ComputerAddress, cwd: string, query: string, limit?: number, signal?: AbortSignal): Promise<ComputerResult> {+ const place = this.place(address);+ return this.stamp(place, await this.searchAddress(address, cwd, query, limit, signal));+ }+ private async searchAddress(address: ComputerAddress, cwd: string, query: string, limit?: number, signal?: AbortSignal): Promise<ComputerResult> {if (address.kind === "sandbox") return this.sandbox(address.ref).session.windows(signal, query);if (address.kind === "machine") {if (address.part !== "windows" && address.part !== "apps")@@ -640,8 +707,8 @@ export class ComputerResources {return Array.isArray(apps) ? { ...listed, details: { ...listed.details, state: { ...listed.details.state,apps: apps.map(app => ({ ...app, app_ref: this.appRef(address.ref, String((app as { app_id: unknown }).app_id)) })) } } } : listed;}- const session = await this.sessionFor(cwd, this.windowOf(address));if (address.kind !== "observation") throw new ComputerRequestError("Search computer:machine/<name>/windows, computer:machine/<name>/apps, a sandbox address or a returned observation address.");+ const session = (await this.routeAddress(cwd, address, signal)).session;if (address.hint || address.delivery || address.delayMs !== undefined) throw new ComputerRequestError("Input options do not select a Search scope.");const snapshot = this.snapshot(address.ref);if (!snapshot.target.window_ref) throw new ComputerRequestError("Display images have no native UI search scope.");@@ -650,8 +717,12 @@ export class ComputerResources {{ operation: "search", observation_id: address.ref, query, ...(limit !== undefined ? { limit } : {}) });}- /** Dispatch exactly one literal Write or JSON Insert against its address-bound target. */+ /** Dispatch exactly one literal Write or JSON Insert against its address-bound target; the receipt names its machine. */async perform(address: ComputerAddress, cwd: string, value: string, tool: "write" | "insert", signal?: AbortSignal): Promise<ComputerResult> {+ const place = this.place(address);+ return this.stamp(place, await this.performAt(address, cwd, value, tool, signal));+ }+ private async performAt(address: ComputerAddress, cwd: string, value: string, tool: "write" | "insert", signal?: AbortSignal): Promise<ComputerResult> {if (address.kind === "window-state") {if (tool !== "insert") throw new ComputerRequestError("Window state grants native control through Insert, never literal Write.");const command = commandObject(value);@@ -729,12 +800,12 @@ export class ComputerResources {let mode;try { mode = displayMode(command); }catch (error) { throw new ComputerRequestError(error instanceof Error ? error.message : String(error)); }- const session = await this.sessionFor(cwd, { display: address.ref });+ const session = await this.sessionFor(cwd, { display: address.ref, machine: address.machine }, signal);// Window and display coordinates from before the change no longer hold. Collect them now:- // a successful change also retires the display references used to recognize them.- const stale = [...this.snapshots].filter(([, entry]) => entry.target.window_ref ? session.hasWindow(entry.target.window_ref) : session.hasDisplay(entry.target.display!)).map(([id]) => id);+ // a successful change also retires the window references used to recognize them.+ const stale = [...this.snapshots].filter(([, entry]) => entry.target.window_ref !== undefined ? session.hasWindow(entry.target.window_ref) : entry.target.machine === address.machine).map(([id]) => id);const result = await session.resizeDisplay(address.ref, mode, signal);- if (result.details.status !== "refused") { for (const id of stale) this.snapshots.delete(id); this.forget(session); }+ if (result.details.status !== "refused") { for (const id of stale) this.snapshots.delete(id); this.forget(session, address.machine); }return result;}if (address.kind === "window" && tool === "insert") {diff --git a/src/provider.ts b/src/provider.tsindex 3826438..5b21fb4 100644--- a/src/provider.ts+++ b/src/provider.ts@@ -5,8 +5,8 @@ import { connectTextEditorPlugin, TEXT_EDITOR_PROTOCOL, TEXT_EDITOR_API_VERSIONimport type { ResourceResolver } from "pi-agent-ide/api/resource";import { connectAgentDocumentation } from "pi-agent-ide/api/documentation";import { readFile } from "node:fs/promises";-import { ComputerResources, requireComputerAddress, computerResultText, ComputerRequestError, type SandboxFactory, type MachineFactory } from "./computer-resources.ts";-import { computerSource } from "./computer-address.ts";+import { ComputerResources, requireComputerAddress, computerResultText, ComputerRequestError, LOCAL_MACHINE, type SandboxFactory, type MachineFactory } from "./computer-resources.ts";+import { computerSource, displaySource } from "./computer-address.ts";import type { WebLogEntry } from "./web-browser.ts";import { diffIsEmpty, type PixelReport, type StateDiff } from "./successor.ts";import type { PixelBox } from "./pixel-diff.ts";@@ -19,7 +19,7 @@ const record = (value: unknown): Record<string, unknown> => value !== null && tyconst label = (value: unknown) => typeof value === "string" ? value.replace(/[\r\n\t]/g, " ").slice(0, 1024) : "";/** Agent guidance shared by the IDE plugin and the standalone tools. */-export const COMPUTER_PATH_HELP = "Read docs:computer for native resources. computer:session, computer:machine/<name> (local, or a configured machine) with its /windows, /apps and /displays, and returned computer:window/ref, display/ref, observation/id, window-state/id or sequence/id. Read window overviews before input; display/detail/frames are readonly.";+export const COMPUTER_PATH_HELP = "Read docs:computer for native resources. computer:session, computer:machine/<name> (local, or a configured machine) with its /windows, /apps and /displays, its /display/primary or listed /display/ref, and returned computer:window/ref, observation/id, window-state/id or sequence/id, each running on the machine that issued it. Read window overviews before input; display/detail/frames are readonly.";export const COMPUTER_VIEWS_HELP = "Computer: image/ui/full; dom alone or dom:{target,boxes,depth,log} on a browser window opened by open_browser or attached with attach_browser returns a fresh Playwright aria snapshot with [ref=...] values of the active tab; log:true adds the retained console/page-error/network history; target is {ref} or locator fields role,name,label,text,placeholder,test_id,css,exact,frames. On a DOM observation: image is a page screenshot, element:crop|highlight,<ref> an element image; Search filters its lines. state alone on a window returns fresh restore or management metadata without pixels/UI, plus whether the window is hung and its process memory/CPU. ocr (alone or with capture options on a window, alone on an observation, or with region:x,y,width,height on an observation) adds Windows OCR lines with boxes in overview pixels; use maxLongEdge 3840 for small text. aria (alone or with ui capture options, on a window) returns its UI as a Playwright aria snapshot with hints as [ref=...]. changes (alone, or with ui, image or full and capture options, on a window) takes a fresh capture and lists only rows changed, added or removed since that window's previous observation, with the new observation's hints, plus changed pixel areas with before/after crops instead of the full image. regions (alone or with image/full options on a window, alone on its observation) lists icons and text areas found by Cua perception with boxes in overview pixels; click a box center with pixel input. capture:{maxLongEdge,scale,max_elements,max_depth,hints,guides}. region:x,y,width,height or element:inspect|scope|highlight|crop|text,hint on an observation; element:text returns one element's full text. sequence:{duration_ms<=60000,fps<=30,max_frames<=300,timeout_ms,maxLongEdge,scale,region} on window/display reports actual_fps and last_frame_age_ms; frames:first|last|all|endpoints|changed or frames:{indexes:[0]} / {from:0,to:2} on sequence. Detail/frames are readonly; text offset/limit select UI lines, never image cells.";export const COMPUTER_SEARCH_HELP = "Search computer:machine/<name>/apps or computer:machine/<name>/windows (local is this machine) for installed-app/permitted-window metadata, or an observation for its collected UI/readonly DOM. Read computer:machine/<name>/displays for monitor metadata. Discovery launches nothing; observation Search performs no new native walk or DOM query. Treat missing matches in a partial projection as unknown.";export const COMPUTER_INPUT_HELP = "Computer Write sends content literally through type_text to a fresh UI/image observation address. On a DOM observation of a browser opened with Insert {\"action\":\"open_browser\"} (YOLO: \"profile\":\"agent\" keeps the agent's own logins between sessions), or of a running Chromium-based browser attached in YOLO with Insert {\"action\":\"attach_browser\"} into its window address (Chrome or Edge through a debugging port, the chrome://inspect/#remote-debugging switch or the Playwright Extension; an Electron/CEF app started with --remote-debugging-port=0 in open's args; a WebView2 app started with open \"debugging\":true or after {\"action\":\"webview2_debugging\",\"app\":\"x.exe\",\"enabled\":true} on its machine; VS Code's integrated browser through the Pi Browser Bridge; closing the session detaches), Insert one Playwright action such as {\"action\":\"fill\",\"target\":{\"label\":\"Name\"},\"text\":\"value\"}; actions target the active tab, and tab/navigation/dialog actions are listed in docs:computer. Insert {\"action\":\"open_sandbox\"} into a machine address to work in a separate disposable Windows Sandbox that does not touch the human's desktop, or {\"action\":\"open_session\"} for an isolated agent session on the same machine (several can run at once, each with its own desktop and foreground input); use the returned computer:sandbox/... address for Search, open and close (docs:computer). Machines are computer:machine/local (this one) and computer:machine/<configured name> (another Windows machine over SSH, started on first use); each takes the same commands. Insert {\"action\":\"put\",\"from\":\"<workspace file>\"}, {\"action\":\"get\",\"name\":\"<file>\",\"to\":\"<workspace path>\"} or {\"action\":\"files\"} into a machine address to move files through the session's own folder on that machine (removed on close); point app Open/Save dialogs there. The same addresses take {\"action\":\"clipboard_write\",\"text\"|\"image\":...}, {\"action\":\"clipboard_read\",\"image\":true}, {\"action\":\"record_start\",\"seconds\":30} and {\"action\":\"record_stop\",\"to\":\"<workspace path>\"} (main-display video) and {\"action\":\"audio_record\",\"seconds\":10,\"transcribe\":true} (playback audio with a transcript) in YOLO. Insert {\"action\":\"timeline\",\"events\":[{at_ms,type,...}]} with delivery=foreground into an observation for timed key/button/move/move_relative/wheel/pen events; everything held is released at the end (docs:computer). Insert {\"action\":\"resize\",\"width\":1920,\"height\":1080} into a display address to change that display's resolution in YOLO; closing the session restores it. Only after the user explicitly allows it, Insert {\"action\":\"setup\",\"part\":\"browser|firefox|perception|mxc|vscode-bridge\"} into a machine address to install a missing optional part there (docs:computer). Insert {\"action\":\"menu\",\"path\":[\"File\",\"Save As...\"]} into an observation to invoke a menu item by its labels, and {\"action\":\"kill\"} into the window address of an app this session opened to force-close it. Insert an action with a Playwright locator target, such as {\"action\":\"click\",\"target\":\"getByRole('button', { name: 'Save' })\"} or fill/check/select, into a native window address to act without a prior Read; it waits until exactly one element is ready. Insert {\"action\":\"wait_for\"|\"verify\",\"conditions\":[...]} into a window address to check UI state with the same locators, count or an aria snapshot template; every action reports what it changed in the window: UI rows, and changed pixel areas with before/after crops after an action on an image or full observation (\"successor\":false skips it), and \"expect\" adds a postcondition; or Insert {\"action\":\"batch\",\"steps\":[...]} into an observation for a fail-stop chain (docs:computer). Use no Write or native input on a DOM observation. Use optional ?hint=...&delivery=foreground&delay_ms=0 only on input observations. No implicit select-all, submit or program execution. Computer Insert sends {\"action\":\"restore\"} to a fresh window-state address for explicit restoration and activation without keys/clicks, or one JSON action to an observation, or {\"action\":\"open\",\"app\":\"Exact installed name\"} (YOLO also {\"action\":\"open\",\"path\":\"C:\\\\...\\\\app.exe\",\"args\":[...]}) to a machine address, and {\"action\":\"close\"} to a machine address or to computer:session for every machine (the result lists each released part). Close releases runtime, not documents. Omit window_ref/observation_id in JSON. Input consumes observations even on refusal; the action's successor observation is current for the next input. With \"successor\":false, delivered pixel-only input keeps its observation until the window is resized; otherwise refresh with Read. One sequence takes up to 2000 key, click and stroke steps. Never replay unknown prefixes. Foreground input requires an observed background refusal and pauses for 10 s after the human's own keyboard or mouse input; restore is a separate explicit activation, not input delivery. Safe/YOLO is user-controlled.";@@ -27,7 +27,8 @@ export const COMPUTER_INPUT_HELP = "Computer Write sends content literally throu/** Present native resources without dumping provenance; full receipts remain in structured data. */export function presentComputerResult(result: ComputerResult, full = false): string {const d = result.details, state = d.state ?? {};- const lines = [`Computer ${d.status} (${d.mode})`];+ // Every routed receipt names the machine (and sandbox) it ran on.+ const lines = [`Computer ${d.status} (${d.mode})${d.machine ? ` on ${d.machine}${d.sandbox ? `, sandbox ${d.sandbox}` : ""}` : ""}`];if (d.window_ref) lines.push(`Window: ${computerSource("window", d.window_ref)}`);if (d.window_state_id && d.status === "observed") {const sample = d.window_state_kind === "management" ? record(state.state) : state;@@ -123,11 +124,12 @@ export function presentComputerResult(result: ComputerResult, full = false): strfor (const raw of rows) {const row = record(raw);const ref = kind === "windows" ? row.window_ref : kind === "displays" ? row.display_ref : undefined;- const source = typeof ref === "string" ? computerSource(kind === "windows" ? "window" : "display", ref)+ const source = typeof ref !== "string" ? undefined : kind === "windows" ? computerSource("window", ref) : displaySource(d.machine ?? LOCAL_MACHINE, ref);+ const shown = source !== undefined ? source: kind === "apps" && typeof row.app_ref === "string" ? computerSource("app", row.app_ref) : "(no controllable reference)";const metadata = Object.fromEntries(["pid", "window_id", "bounds", "minimized", "owned", "width", "height", "primary"].filter(key => row[key] !== undefined).map(key => [key, row[key]]));- lines.push(`${source} | ${label(row.name ?? row.title ?? row.app)} | ${JSON.stringify(metadata)}`);+ lines.push(`${shown} | ${label(row.name ?? row.title ?? row.app)} | ${JSON.stringify(metadata)}`);}}} else {diff --git a/src/session.ts b/src/session.tsindex ab95758..9f7042f 100644--- a/src/session.ts+++ b/src/session.ts@@ -79,6 +79,10 @@ export interface ComputerResult {content: Content[];details: {status: "listed" | "observed" | "delivered" | "refused" | "unknown" | "closed";+ /** The machine the request ran on, by its address name; set by the computer resources for every routed request. */+ machine?: string;+ /** The sandbox or isolated session on that machine the request ran in. */+ sandbox?: string;/** The sandbox an open_sandbox command started. */sandbox_ref?: string;mode: ComputerMode;@@ -567,16 +571,20 @@ export class ComputerSession {return this.queue(async () => {try {validateSequence(options);- if ((target.window_ref !== undefined) === (target.display !== undefined) || (target.display !== undefined && target.display !== "primary"))- return this.result("refused", "Choose exactly one window_ref or display: primary.");+ if ((target.window_ref !== undefined) === (target.display !== undefined)) return this.result("refused", "Choose exactly one window or display.");const entry = target.window_ref ? this.entries.get(target.window_ref) : undefined;- const permitted = () => entry ? this.allowed(entry) : target.display === "primary" && this.getMode() === "yolo";+ // primary is the machine's main display; any other display is one its displays listing issued.+ const display = target.display !== undefined && target.display !== "primary" ? this.displayEntries.get(target.display) : undefined;+ if (target.display !== undefined && target.display !== "primary" && (!this.driver.selectedDisplays || !display))+ return this.result("refused", "Unknown or unavailable display reference. Read the machine's displays again; no primary fallback.");+ const permitted = () => entry ? this.allowed(entry) : target.display !== undefined && this.getMode() === "yolo";const denied = () => { this.sequences.clear(); return this.result("refused", "Sequence source access denied. Safe permits session-owned windows only; retained pixels withheld."); };if (!permitted()) return denied();const settings = { ...this.overviewSettings(), ...(options.maxLongEdge !== undefined ? { maxLongEdge: options.maxLongEdge } : {}),...(options.scale !== undefined ? { scale: options.scale } : {}) };const cap = nativeImageCap(settings);if (entry) { entry.dom = undefined; entry.windowState = undefined; entry.observation = undefined; entry.detail = undefined; }+ else if (display) display.detail = undefined;else this.displayOverview = undefined;let first = true;const sequence = await collectSequence(options, async active => {@@ -584,22 +592,21 @@ export class ComputerSession {if (entry && !await this.current(entry.ref, active)) throw new SequenceStop("target_lost");if (!permitted()) throw new SequenceStop("access_revoked");const nativeStarted = performance.now();- const native = await this.driver.call(entry ? "get_window_state" : "get_desktop_state", entry- ? { ...entry.target, include_accessibility_tree: false, max_image_dimension: cap }- : { max_image_dimension: cap }, active);+ const native = entry ? await this.driver.call("get_window_state", { ...entry.target, include_accessibility_tree: false, max_image_dimension: cap }, active)+ : await this.readDisplay(display, cap, active);const native_completed_at = performance.now();if (!permitted()) throw new SequenceStop("access_revoked");active.throwIfAborted();const state = native.structuredContent;if (native.isError || !state || state.status === "refused" || state.effect === "refused") throw new SequenceStop("capture_failed");- if (!entry && (state.display !== "primary" || state.platform !== "windows")) throw new SequenceStop("capture_failed");+ if (!entry && !this.displayStateMatches(state, display)) throw new SequenceStop("capture_failed");if (entry && ((state.pid !== undefined && state.pid !== entry.target.pid) ||(state.window_id !== undefined && state.window_id !== entry.target.window_id))) throw new SequenceStop("target_lost");const images = native.content.filter((block): block is SequenceFrame["image"] => block.type === "image");const context = this.overview(state, "sequence-frame", !entry, images);if (!context || Math.max(context.width, context.height) > cap || images[0].data.length > 16 * 1024 * 1024)throw new SequenceStop("capture_failed");- let source = this.source(entry ? "window" : "display", state, entry+ let source = display ? selectedDisplaySource(state, this.sessionId, display.ref) : this.source(entry ? "window" : "display", state, entry? { session_id: this.sessionId, window_ref: entry.ref, ...entry.target }: { session_id: this.sessionId, display_id: "primary" });source.semantic_scope = "none"; source.capabilities = { read: true, input: "none", detail: "unsupported" };@@ -838,7 +845,9 @@ export class ComputerSession {const state = native.structuredContent;if (native.isError || state?.status === "refused" || state?.effect === "refused") {if (entry) this.displayEntries.delete(entry.ref);- return this.result("refused", "CUA refused exact-display capture. Discover again; no fallback.");+ // The native reason, such as a locked PC, tells the agent whether discovering again can help.+ const reason = native.content.filter(c => c.type === "text").map(c => c.text).join(" ").slice(0, 600);+ return this.result("refused", `CUA refused exact-display capture${reason ? `: ${reason}` : "."} No fallback.`);}const images = native.content.filter(block => block.type === "image" && block.mimeType === "image/png");if (!this.displayStateMatches(state, entry) || images.length !== 1 ||@@ -1715,6 +1724,8 @@ export class ComputerSession {hasWindow(ref: string): boolean { return this.entries.has(ref); }/** Whether this session issued a still-registered display reference. */hasDisplay(ref: string): boolean { return this.displayEntries.has(ref); }+ /** Whether this session captured the sequence and still retains it. */+ hasSequence(id: string): boolean { return this.sequences.get(id) !== undefined; }/** Close session-opened browsers (deleting their profiles), detach from attached ones and release the native runtime. Other windows/files are not closed or saved. */close(): Promise<ComputerResult> {diff --git a/test/computer-resources.test.ts b/test/computer-resources.test.tsindex 29ea722..7c82651 100644--- a/test/computer-resources.test.ts+++ b/test/computer-resources.test.ts@@ -162,7 +162,7 @@ test.each(["safe", "closed", "reused"] as const)("%s after reading refuses direc});test("display resources and their observations never grant Write or Insert authority", async () => {- const f = await fixture(), display = address("computer:display/primary");+ const f = await fixture(), display = address("computer:machine/local/display/primary");const result = await f.resources.read(display, cwd);expect(result?.details.status).toBe("observed");const observed = address(computerSource("observation", result!.details.agent_image!.image_id as string));@@ -240,7 +240,7 @@ test("an address delivery preference does not block semantic actions", async ()});test("a display overview accepts a size for that Read only", async () => {- const f = await fixture(), display = address("computer:display/primary");+ const f = await fixture(), display = address("computer:machine/local/display/primary");await f.resources.read(display, cwd, ['capture:{"maxLongEdge":320}']);expect(f.captures().at(-1)).toEqual(["get_desktop_state", { max_image_dimension: 320 }, undefined]);await f.resources.read(display, cwd);diff --git a/test/display-observation.integration.test.ts b/test/display-observation.integration.test.tsindex 0a8a17e..746c7b4 100644--- a/test/display-observation.integration.test.ts+++ b/test/display-observation.integration.test.ts@@ -7,7 +7,7 @@ import { DEFAULT_MAX_LONG_EDGE } from "../src/cua-image-settings.ts";import { nestedCalls, SCRIPT_HELPERS } from "./fixtures/standalone-computer.ts";const yoloRead = `${SCRIPT_HELPERS}-const view = await tools.computer_read({path:"computer:display/primary"});+const view = await tools.computer_read({path:"computer:machine/local/display/primary"});text(body(view)); image(view);const input = await insert(observation(view),{action:"click",x:0,y:0});text("observation input: " + input.outcome + " / " + input.effect);`;@@ -28,10 +28,10 @@ for (const mode of ["yolo", "safe"] as const) {tools: ["codemode", "computer_read", "computer_insert"],conversation: [assistantMessage([toolCall({ id: "read", name: mode === "safe" ? "computer_read" : "codemode", arguments: mode === "safe"- ? { path: "computer:display/primary" } : { code: yoloRead } })], { stopReason: "toolUse" }),- assistantMessage([toolCall({ id: "mixed", name: "computer_read", arguments: { path: "computer:display/primary", views: ["ui"] } })], { stopReason: "toolUse" }),- assistantMessage([toolCall({ id: "promote", name: "computer_read", arguments: { path: "computer:display/primary", mode: "yolo" } })], { stopReason: "toolUse" }),- assistantMessage([toolCall({ id: "display-input", name: "computer_insert", arguments: { path: "computer:display/primary", text: '{"action":"click","x":0,"y":0}' } })], { stopReason: "toolUse" }),+ ? { path: "computer:machine/local/display/primary" } : { code: yoloRead } })], { stopReason: "toolUse" }),+ assistantMessage([toolCall({ id: "mixed", name: "computer_read", arguments: { path: "computer:machine/local/display/primary", views: ["ui"] } })], { stopReason: "toolUse" }),+ assistantMessage([toolCall({ id: "promote", name: "computer_read", arguments: { path: "computer:machine/local/display/primary", mode: "yolo" } })], { stopReason: "toolUse" }),+ assistantMessage([toolCall({ id: "display-input", name: "computer_insert", arguments: { path: "computer:machine/local/display/primary", text: '{"action":"click","x":0,"y":0}' } })], { stopReason: "toolUse" }),assistantMessage([toolCall({ id: "close", name: "computer_insert", arguments: { path: "computer:session", text: '{"action":"close"}' } })], { stopReason: "toolUse" }),assistantMessage([text("Read-only display source checked.")]),],diff --git a/test/fixtures/ide-provider-extension.ts b/test/fixtures/ide-provider-extension.tsindex 4ef57dd..24967d7 100644--- a/test/fixtures/ide-provider-extension.ts+++ b/test/fixtures/ide-provider-extension.ts@@ -159,7 +159,7 @@ export default async function fixture(pi: ExtensionAPI): Promise<void> {await denied("insert", { path: observation, text: "not JSON" });await denied("insert", { path: observation, text: JSON.stringify({ action: "click", x: 100, y: 80, window_ref: second }) });await denied("insert", { path: first, text: JSON.stringify({ action: "click", x: 100, y: 80 }) });- await denied("write", { path: "computer:display/primary", content: "not input authority" });+ await denied("write", { path: "computer:machine/local/display/primary", content: "not input authority" });assert((await inputs()).length === 0, "Invalid/Safe/display requests sent native input");view = await ok("read", { path: first }); observation = reference(view, "observation");await control("partial");diff --git a/test/live/machine-sequences.live.ts b/test/live/machine-sequences.live.tsnew file mode 100644index 0000000..8ed2cb0--- /dev/null+++ b/test/live/machine-sequences.live.ts@@ -0,0 +1,51 @@+import { afterAll, expect, test } from "vitest";+import { displaySource, computerSource } from "../../src/computer-address.ts";+import { presentComputerResult } from "../../src/provider.ts";+import { addresses, connectLive, nativeForm, openWindow, text, WINDOWS_POWERSHELL_PATH } from "./remote.ts";++// LPT-924, LPT-929: sequences of a configured machine's windows and displays are captured there, their frames are+// read there, and every receipt names that machine.+const live = await connectLive();+const name = process.env.LIVE_MACHINE ?? "lab";+afterAll(() => live.close());++const images = (result: { content: { type: string }[] }) => result.content.filter(block => block.type === "image").length;++test("a window sequence on the machine returns its frames", async () => {+ const title = `pi-sequence-${Date.now()}`;+ const window = await openWindow(live, live.machine, WINDOWS_POWERSHELL_PATH, nativeForm(title), title);+ try {+ const captured = await live.read(window, ['sequence:{"duration_ms":1000,"fps":5,"maxLongEdge":320}']);+ expect(captured.details.status, text(captured)).toBe("observed");+ expect(captured.details.machine).toBe(name);+ const frames = await live.read(computerSource("sequence", captured.details.sequence!.sequence_id), ["frames:endpoints"]);+ expect(frames.details.status, text(frames)).toBe("observed");+ expect(frames.details.machine).toBe(name);+ expect(images(frames)).toBe(2);+ expect(presentComputerResult(frames)).toMatch(new RegExp(`^Computer observed \\(yolo\\) on ${name}\\n`));+ } finally {+ const state = await live.read(window, ["state"]).catch(() => undefined);+ const target = state?.details.target as { window_id?: number } | undefined;+ if (target?.window_id) await live.closeWindow(target.window_id);+ }+});++test("the machine's primary and listed displays are captured there, with readable frames", async () => {+ const primary = displaySource(name, "primary");+ const overview = await live.read(primary, ['capture:{"maxLongEdge":320}']);+ expect(overview.details.status, text(overview)).toBe("observed");+ expect(overview.details.machine).toBe(name);+ const listed = await live.read(live.displays);+ const [display] = addresses(listed, "display");+ expect(display).toMatch(new RegExp(`^computer:machine/${name}/display/d-`));+ for (const source of [primary, display]) {+ const captured = await live.read(source, ['sequence:{"duration_ms":600,"fps":5,"maxLongEdge":320}']);+ expect(captured.details.status, text(captured)).toBe("observed");+ expect(captured.details.machine).toBe(name);+ const frames = await live.read(computerSource("sequence", captured.details.sequence!.sequence_id), ["frames:last"]);+ expect(frames.details.status, text(frames)).toBe("observed");+ expect(images(frames)).toBe(1);+ }+ // A display reference of this machine is unknown on another one.+ if (name !== "local") await expect(live.read(display.replace(`/machine/${name}/`, "/machine/local/"))).rejects.toThrow();+});diff --git a/test/live/machines.live.ts b/test/live/machines.live.tsindex cf12ba2..219bbda 100644--- a/test/live/machines.live.ts+++ b/test/live/machines.live.ts@@ -22,7 +22,7 @@ test("a machine's apps open there, and closing the machine reports each releasedexpect(closed.details.status, text(closed)).toBe("closed");expect(text(closed)).toMatch(new RegExp(`^${live.machine}: `, "m"));// Its references are gone, and the next use starts the machine again.- // No started machine claims the old window; it falls to the local session, which this harness forbids.+ // No started machine claims the old window; checking whether the local session holds it is forbidden by this harness.await expect(live.read(window, ["state"])).rejects.toThrow(/Live checks use only the remote machine/);expect(addresses(await live.read(live.displays), "display").length).toBeGreaterThan(0);});diff --git a/test/live/remote.ts b/test/live/remote.tsindex 215b52e..573d951 100644--- a/test/live/remote.ts+++ b/test/live/remote.ts@@ -1,4 +1,4 @@-import { computerSource } from "../../src/computer-address.ts";+import { computerSource, displaySource } from "../../src/computer-address.ts";import type { ComputerAddress } from "../../src/computer-address.ts";import { ComputerResources, requireComputerAddress } from "../../src/computer-resources.ts";import { CuaImageSettings } from "../../src/cua-image-settings.ts";@@ -136,5 +136,5 @@ export const text = (result: ComputerResult) => result.content.filter(block => bexport function addresses(result: ComputerResult, kind: "window" | "display"): string[] {const rows = (result.details.state?.[kind === "window" ? "windows" : "displays"] ?? []) as Record<string, unknown>[];return rows.map(row => row[kind === "window" ? "window_ref" : "display_ref"]).filter((ref): ref is string => typeof ref === "string")- .map(ref => computerSource(kind, ref));+ .map(ref => kind === "window" ? computerSource(kind, ref) : displaySource(result.details.machine ?? "local", ref));}diff --git a/test/native-detail.integration.test.ts b/test/native-detail.integration.test.tsindex ac347f9..5dffaa2 100644--- a/test/native-detail.integration.test.ts+++ b/test/native-detail.integration.test.ts@@ -21,7 +21,7 @@ const detail = await tools.computer_read({path:opened.observation,views:[region]text(body(detail)); image(detail);const stale = await insert(opened.observation,{action:"click",element_token:"close"});if (stale.outcome !== "refused" || stale.effect !== "not-applied") throw Error("Old overview still armed");-const overview = await tools.computer_read({path:"computer:display/primary"});+const overview = await tools.computer_read({path:"computer:machine/local/display/primary"});const crop = await tools.computer_read({path:observation(overview),views:[region]});text(body(crop)); image(crop);let nested = "admitted";diff --git a/test/remote-machine.test.ts b/test/remote-machine.test.tsindex 1f4ed04..f06ff24 100644--- a/test/remote-machine.test.ts+++ b/test/remote-machine.test.ts@@ -1,9 +1,10 @@import { execFileSync } from "node:child_process";+import sharp from "sharp";import { mkdir, mkdtemp, writeFile } from "node:fs/promises";import { tmpdir } from "node:os";import path from "node:path";import { expect, test, vi } from "vitest";-import { computerSource } from "../src/computer-address.ts";+import { computerSource, displaySource } from "../src/computer-address.ts";import { ComputerRequestError, ComputerResources, requireComputerAddress, type OpenedMachine, type OpenedSandbox } from "../src/computer-resources.ts";import { loadComputerConfig } from "../src/config.ts";import { checkRemoteWorker } from "../src/machine.ts";@@ -35,15 +36,26 @@ test("machines come only from validated configuration", async () => {}});+const png = (await sharp({ create: { width: 40, height: 30, channels: 3, background: "red" } }).png().toBuffer()).toString("base64");+const image = { type: "image" as const, mimeType: "image/png", data: png };+/** One display's pixels as the native runtime reports them: 1920x1080 scaled to 40x30 here. */+const pixels = { platform: "windows", screenshot_width: 40, screenshot_height: 30, screenshot_original_width: 1920, screenshot_original_height: 1080 };+/** A fake native backend with one window and one display. */function backend(pid: number) {+ const display = { display_id: `D${pid}`, primary: true, monitor_handle: `0x${pid}`, monitor_device_paths: [`\\\\?\\DISPLAY#${pid}`],+ bounds: { x: 0, y: 0, width: 1920, height: 1080 }, work_area: { x: 0, y: 0, width: 1920, height: 1040 }, scale_factor: 1, dpi: 96 };const driver: Driver = {identity: vi.fn(async () => `life-${pid}`), close: vi.fn(async () => {}), selectedDisplays: true,call: vi.fn<Driver["call"]>(async (name, args) => {if (name === "list_windows") return { content: [], structuredContent: { windows: [{ pid, window_id: 7, title: `window ${pid}` }] } };- if (name === "list_displays") return { content: [], structuredContent: { platform: "windows", topology_id: `t${pid}`, displays: [{ display_id: `D${pid}`, primary: true,- monitor_handle: `0x${pid}`, monitor_device_paths: [`\\\\?\\DISPLAY#${pid}`], bounds: { x: 0, y: 0, width: 1920, height: 1080 },- work_area: { x: 0, y: 0, width: 1920, height: 1040 }, scale_factor: 1, dpi: 96 }] } };+ if (name === "list_displays") return { content: [], structuredContent: { platform: "windows", topology_id: `t${pid}`, displays: [display] } };+ if (name === "get_desktop_state") return { content: [image], structuredContent: { ...pixels, display: "primary" } };+ if (name === "get_display_state") return { content: [image], structuredContent: { ...pixels, display, topology_id: `t${pid}` } };+ // A sequence frame: pixels without the UI tree.+ if (name === "get_window_state" && (args as { include_accessibility_tree?: boolean }).include_accessibility_tree === false)+ return { content: [image], structuredContent: { pid, window_id: 7, window_bounds: { x: 0, y: 0, width: 40, height: 30 },+ screenshot_width: 40, screenshot_height: 30, screenshot_original_width: 40, screenshot_original_height: 30 } };// No pixels here: a capture that asks for them reports why, and the successor falls back to the UI.if (name === "get_window_state") return { content: [], structuredContent: { pid, window_id: 7, snapshot_id: "s", elements_complete: true, elements: [],...((args as { include_screenshot?: boolean }).include_screenshot !== false ? { screenshot_error: "No surface." } : {}),@@ -111,9 +123,8 @@ test("an unknown machine refuses, and closing a machine releases it, its referenexpect(() => f.resources.descriptor(address(computerSource("sandbox", "sb-1")))).toThrow(/expired/);await expect(f.resources.perform(address(computerSource("observation", view!.details.observation_id!)), cwd, "late", "write")).rejects.toThrow(/expired/);// A stale remote window address never reaches this host's desktop.- const late = await f.resources.read(address(computerSource("window", ref)), cwd, ["ui"]);- expect(late?.details.status).toBe("refused");- expect(f.names(f.hostDriver).filter(name => name !== "list_windows")).toEqual([]);+ await expect(f.resources.read(address(computerSource("window", ref)), cwd, ["ui"])).rejects.toThrow(/Unknown or closed window reference[\s\S]*nothing was sent/);+ expect(f.names(f.hostDriver)).toEqual([]);});test("closing the computer session releases every machine, the local one included, and reports each", async () => {@@ -133,7 +144,60 @@ test("closing the computer session releases every machine, the local one include});test("removed address forms refuse", () => {- for (const source of ["computer:windows", "computer:apps", "computer:displays"]) expect(() => address(source)).toThrow();+ for (const source of ["computer:windows", "computer:apps", "computer:displays", "computer:machine/lab/display", "computer:machine/lab/display/a/b"]) expect(() => address(source)).toThrow();+ // A display without its machine names the form to use instead.+ expect(() => address("computer:display/primary")).toThrow(/computer:machine\/<name>\/display\/primary/);+});++// LPT-929, LPT-924: every address reaches the machine that issued it, and every receipt names that machine.+test("window sequences and their frames run on the machine that captured them", async () => {+ const f = await fixture();+ const listed = await f.resources.search(address(`${f.machineSource}/windows`), cwd, "window");+ expect(listed.details.machine).toBe("lab");+ const ref = (listed.details.state!.windows as { window_ref: string }[])[0].window_ref;+ const captured = await f.resources.read(address(computerSource("window", ref)), cwd, ['sequence:{"duration_ms":1}']);+ expect(captured?.details).toMatchObject({ status: "observed", machine: "lab" });+ const sequence = address(computerSource("sequence", captured!.details.sequence!.sequence_id));+ const frames = await f.resources.read(sequence, cwd, ["frames:endpoints"]);+ expect(frames?.details).toMatchObject({ status: "observed", machine: "lab" });+ expect(frames!.content.filter(block => block.type === "image")).toHaveLength(1);+ expect(presentComputerResult(frames!)).toMatch(/^Computer observed \(yolo\) on lab\n/);+ expect(f.names(f.hostDriver)).toEqual([]);+});++test("a machine's primary and listed displays are read and captured on that machine only", async () => {+ const f = await fixture();+ const primary = address(displaySource("lab", "primary"));+ const overview = await f.resources.read(primary, cwd);+ expect(overview?.details).toMatchObject({ status: "observed", machine: "lab" });+ // The display observation keeps its machine.+ expect(f.resources.descriptor(address(computerSource("observation", overview!.details.agent_image!.image_id as string)))).toContain(`Source: ${displaySource("lab", "primary")}`);+ const sequence = await f.resources.read(primary, cwd, ['sequence:{"duration_ms":1}']);+ expect(sequence?.details).toMatchObject({ status: "observed", machine: "lab" });+ const listed = await f.resources.read(address(`${f.machineSource}/displays`), cwd);+ const ref = (listed!.details.state!.displays as { display_ref: string }[])[0].display_ref;+ // The listing shows each display with its machine's address.+ expect(presentComputerResult(listed!)).toContain(displaySource("lab", ref));+ const selected = await f.resources.read(address(displaySource("lab", ref)), cwd, ['sequence:{"duration_ms":1}']);+ expect(selected?.details).toMatchObject({ status: "observed", machine: "lab" });+ const frames = await f.resources.read(address(computerSource("sequence", selected!.details.sequence!.sequence_id)), cwd, ["frames:last"]);+ expect(frames?.details.status).toBe("observed");+ // A display reference of one machine is unknown on another.+ const elsewhere = await f.resources.read(address(displaySource("local", ref)), cwd);+ expect(elsewhere?.details).toMatchObject({ status: "refused", machine: "local" });+ expect(presentComputerResult(elsewhere!)).toMatch(/Unknown or unavailable display reference/);+ expect(f.names(f.remoteDriver)).toEqual(["get_desktop_state", "get_desktop_state", "list_displays", "get_display_state"]);+ expect(f.names(f.hostDriver)).toEqual([]);+});++test("an address no machine issued is refused before anything is sent", async () => {+ const f = await fixture();+ await f.resources.search(address(`${f.machineSource}/windows`), cwd, "");+ await expect(f.resources.read(address(computerSource("window", "w-unknown")), cwd, ["ui"])).rejects.toThrow(/Unknown or closed window reference/);+ await expect(f.resources.read(address(computerSource("sequence", "seq-unknown")), cwd, ["frames:last"])).rejects.toThrow(/Sequence is unavailable/);+ await expect(f.resources.read(address(displaySource("other", "primary")), cwd)).rejects.toThrow(/No machine named other/);+ expect(f.names(f.hostDriver)).toEqual([]);+ expect(f.names(f.remoteDriver)).toEqual(["list_windows"]);});test("a remote browser refuses before launch when that machine's worker copy is stale or missing", async () => {diff --git a/test/selected-displays.integration.test.ts b/test/selected-displays.integration.test.tsindex 0b3d8e5..c4f2a03 100644--- a/test/selected-displays.integration.test.ts+++ b/test/selected-displays.integration.test.ts@@ -13,7 +13,7 @@ for (const mode of ["yolo", "safe"] as const) {await writeFile(path.join(cwd, ".pi/computer-use.json"), JSON.stringify({ command: process.execPath,args: [path.resolve("test/fixtures/selected-display-driver.mjs"), log] }));const discover = `const r=await tools.computer_read({path:"computer:machine/local/displays"});-const rows=r.split("\\n").filter(line=>line.startsWith("computer:display/"));+const rows=r.split("\\n").filter(line=>line.startsWith("computer:machine/local/display/"));if(rows.length!==2)throw Error("Native discovery failed");const row=rows.find(line=>JSON.parse(line.slice(line.lastIndexOf(" | ")+3)).bounds.x<0);store("monitor",row.split(" | ")[0]);text(row);`;@@ -33,10 +33,10 @@ image(r);text("cropped");`;extensions: [path.resolve(`test/fixtures/selected-display-${mode === "safe" ? "safe-" : ""}extension.ts`)], isolateUserResources: true,tools: ["codemode", "computer_read", "computer_insert"], conversation: [assistantMessage([toolCall({ id: "discovery", name: mode === "safe" ? "computer_read" : "codemode", arguments: mode === "safe" ? { path: "computer:machine/local/displays" } : { code: discover } })], { stopReason: "toolUse" }),- assistantMessage([toolCall({ id: "overview", name: mode === "safe" ? "computer_read" : "codemode", arguments: mode === "safe" ? { path: "computer:display/d-unowned" } : { code: observe } })], { stopReason: "toolUse" }),+ assistantMessage([toolCall({ id: "overview", name: mode === "safe" ? "computer_read" : "codemode", arguments: mode === "safe" ? { path: "computer:machine/local/display/d-unowned" } : { code: observe } })], { stopReason: "toolUse" }),...(mode === "yolo" ? [assistantMessage([toolCall({ id: "crop", name: "codemode", arguments: { code: crop } })], { stopReason: "toolUse" })] : []),assistantMessage([toolCall({ id: "input", name: mode === "safe" ? "computer_insert" : "codemode", arguments: mode === "safe"- ? { path: "computer:display/d-unowned", text: '{"action":"click","x":1,"y":1}' }+ ? { path: "computer:machine/local/display/d-unowned", text: '{"action":"click","x":1,"y":1}' }: { code: 'const r=await tools.computer_insert({path:load("monitor"),text:JSON.stringify({action:"click",x:1,y:1})});text(r.outcome+": "+r.reason);' } })], { stopReason: "toolUse" }),assistantMessage([toolCall({ id: "close", name: "computer_insert", arguments: { path: "computer:session", text: '{"action":"close"}' } })], { stopReason: "toolUse" }),assistantMessage([text("Read-only monitor selection checked.")]),diff --git a/test/sequence.integration.test.ts b/test/sequence.integration.test.tsindex c1e0313..9211190 100644--- a/test/sequence.integration.test.ts+++ b/test/sequence.integration.test.ts@@ -12,7 +12,7 @@ test("real Pi stores a sequence without images and forwards only explicitly seleawait writeFile(path.join(cwd, ".pi/computer-use.json"), JSON.stringify({ command: process.execPath,args: [path.resolve("test/fixtures/detail-driver.mjs"), log] }));const capture = `${SCRIPT_HELPERS}-const result=await tools.computer_read({path:"computer:display/primary",views:['sequence:{"duration_ms":3000,"fps":2,"max_frames":3}']});+const result=await tools.computer_read({path:"computer:machine/local/display/primary",views:['sequence:{"duration_ms":3000,"fps":2,"max_frames":3}']});if(typeof result!=="string")throw Error("Capture unexpectedly forwarded frames");store("temporal",address(result,"sequence","Sequence"));text(result.match(/^Frames: .*$/m)[0]);`;const select = `const result=await tools.computer_read({path:load("temporal"),views:['frames:{"indexes":[0,2]}']});