Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.
TypeScript accessibilityagent-toolsai-agentbrowser-automationcomputer-usecuadesktop-automationelectronpipi-coding-agentpi-extensionpi-packageplaywrighttypescriptui-automationvisionwebview2windowswindows-automationwsl
Read a sequence as one contact sheet or per-pixel composite (LPT-925)
alexshpunt-clanker[bot]dc79ae3a24fb
diff --git a/docs/computer-resources.md b/docs/computer-resources.mdindex 132557c..b7446b5 100644--- a/docs/computer-resources.md+++ b/docs/computer-resources.md@@ -47,8 +47,8 @@ Read views:| `changes` | Fresh window Read that lists only the rows changed (`~`), added (`+`) or removed (`-`) since that window's previous observation in this session, up to 30 of each with counts, and the changed pixel areas since its latest capture with pixels, with before/after crops instead of the full overview (see [Follow an action](#follow-an-action)). Changed and added rows carry the new observation's hints, which ground input as usual, and Search still finds the whole UI. Add `ui` to compare the UI alone; a window whose pixels cannot be captured falls back to that. The first `changes` Read of a window returns the full observation. Use it when polling or re-reading after a wait. || `regions` | Icons and text areas that [Cua perception](https://github.com/trycua/cua/releases/tag/cua-perception-v0.2.1) finds in the overview pixels (OmniParser v2 icon detector, PP-OCRv5 text; local, about 1–2 s). Alone or with image/full capture options on a window, or alone on its current observation (no new capture). Boxes are in the observation's overview pixels; click a box center with pixel input on that observation. Use it for controls without UI names, such as canvas toolbars or games. Needs the signed extension installed on that machine (see below); otherwise the view refuses with the reason. Detection can miss, split or merge items. || `element:inspect,1:a` | Inspect an actual collected UI hint. Also scope/highlight/crop; crop consumes the overview. `element:text,<hint>` returns one native element's full text. On a DOM observation, `element:crop,<ref>` and `element:highlight,<ref>` return Playwright element images. |-| `sequence:{"duration_ms":2000,"fps":4}` | Bounded temporal capture from a window or primary display: up to 60 s, 30 FPS and 300 frames (`max_frames`, `timeout_ms`, `maxLongEdge`, `scale`, `region`). Returns metadata, the achieved FPS, `last_frame_age_ms` and a sequence address, not frames. Frames are sampled one native capture at a time, so the achieved rate depends on window size and machine. |-| `frames:endpoints` | Explicit readonly selection from a sequence, at most 40 frames per Read. Also all/first/last/changed, `frames:{"indexes":[0,2]}` or inclusive `frames:{"from":0,"to":2}`. |+| `sequence:{"duration_ms":2000,"fps":4}` | Bounded temporal capture from a window or a machine's display: up to 60 s, 30 FPS and 300 frames (`max_frames`, `timeout_ms`, `maxLongEdge`, `scale`, `region`). Returns metadata, the achieved FPS, `last_frame_age_ms` and a sequence address, not frames. Frames are sampled one native capture at a time, so the achieved rate depends on window size and machine. |+| `frames:endpoints` | Explicit readonly selection from a sequence, at most 40 frames per Read. Also all/first/last/changed, `frames:{"indexes":[0,2]}` or inclusive `frames:{"from":0,"to":2}`. Add `"as":"sheet"` for one tiled image with index/time labels, or `"max"`/`"min"` for one per-pixel composite; `region` crops every frame first ([temporal capture](temporal-capture.md#one-image-of-many-frames)). |Examples contain placeholder rectangles/hints. Choose actual targets from the current observation. `offset` and `limit` select text lines, never pixels or image-grid cells. Read a cached observation with `ui` to slice its collected text without another native walk. Acquisition options require a fresh window Read.diff --git a/docs/temporal-capture.md b/docs/temporal-capture.mdindex e596f35..b13d3d3 100644--- a/docs/temporal-capture.md+++ b/docs/temporal-capture.md@@ -46,5 +46,17 @@ Every frame reports source geometry and whether it changed from the previous fraChoose `all`, `first`, `last`, `endpoints`, `changed`, `{indexes:[0,2]}` or inclusive `{from:1,to:3}`. Indexes refer to captured frames, not sampling slots. Selection keeps temporal order. `changed` compares exact image bytes and geometry; it does not claim semantic relevance. Codemode receives an array of frame metadata and image blocks, suitable for explicit forwarding or further processing. Do not store whole image arrays in Codemode storage.+## One image of many frames++Add `as` to a JSON selection to get one image instead of one per frame:++| View | Returns |+| --- | --- |+| `frames:{"as":"sheet"}` | A contact sheet of all frames (at most 40): tiles in time order, left to right, each under a `#index · time` label. The text lists each tile's row, column, frame and time. |+| `frames:{"select":"changed","as":"sheet","columns":4}` | The same for another selection (`select`, `indexes` or `from`/`to`) with a fixed column count (1..10). |+| `frames:{"as":"max"}` or `"min"` | The per-pixel maximum or minimum of the selected frames: bright (or dark) detail from any frame appears at once, for example text revealed by a moving light. Frames must have one size. |++`region:{x,y,width,height}` crops every frame first, in the sequence's frame pixels (the overview size it was captured at); it is clipped to the frames and refused when it misses them. A sheet's long edge is 1600 px by default; tiles shrink together to fit, keeping their aspect ratio. `maxLongEdge` (64..4096) changes that limit, for example 16 tiles of a 640 px region in 4 columns at full size need 2560. A composite keeps the frames' size up to `maxLongEdge`. The image is read-only and grants no input.+The existing debug viewer keeps its three ordinary boundary cards. A temporal dropdown also lists up to 40 sequence image blocks in the latest application-boundary request and lets the human inspect their exact bytes. Selection alone is not forwarding. Byte matching can be ambiguous when screenshots are identical; the viewer lists every retained possible source instead of inventing a timestamp. Asset eviction is visible. Final wire delivery remains unconfirmed.diff --git a/src/computer-resources.ts b/src/computer-resources.tsindex 5ecb8ff..7ee3477 100644--- a/src/computer-resources.ts+++ b/src/computer-resources.ts@@ -613,8 +613,8 @@ export class ComputerResources {return session.captureSequence(address.kind === "window" ? { window_ref: address.ref } : { display: address.ref }, view.options, signal);}if (address.kind === "sequence") {- if (view.kind !== "frames") throw new ComputerRequestError("Select sequence images explicitly with frames:first|last|endpoints|changed|all or frames:{\"indexes\":[0]}.");- return session.sequenceFrames(address.ref, view.selection, signal);+ if (view.kind !== "frames") throw new ComputerRequestError("Select sequence images explicitly with frames:first|last|endpoints|changed|all or frames:{\"indexes\":[0]}; add \"as\":\"sheet\" for one tiled image.");+ return session.sequenceFrames(address.ref, view.selection, signal, view.rendering);}if (view.kind === "frames") throw new ComputerRequestError("Frame selection needs a sequence address.");if (address.kind === "window") {diff --git a/src/computer-views.ts b/src/computer-views.tsindex 457583c..a1cb728 100644--- a/src/computer-views.ts+++ b/src/computer-views.ts@@ -1,6 +1,7 @@import { validateUiCapture, type UiCaptureOptions } from "./ui-resources.ts";import { SEQUENCE_LIMITS, validateSequence, type SequenceOptions, type FrameSelection } from "./temporal-capture.ts";import { validateWebSnapshot, type WebSnapshotOptions } from "./web-browser.ts";+import { validateRendering, type FrameRendering } from "./contact-sheet.ts";/** One native Read mode. Details and sampled frames never grant input authority. */export type ComputerViews =@@ -9,7 +10,8 @@ export type ComputerViews =| { kind: "overview"; capture: UiCaptureOptions }| { kind: "region"; region: { x: number; y: number; width: number; height: number } }| { kind: "sequence"; options: SequenceOptions }- | { kind: "frames"; selection: FrameSelection }+ /** Selected frames, one image each, or one sheet or composite image of them. */+ | { kind: "frames"; selection: FrameSelection; rendering?: FrameRendering }| { kind: "element"; operation: "inspect" | "scope" | "highlight" | "crop" | "text"; hint: string };function json(value: string): Record<string, unknown> {@@ -85,6 +87,15 @@ function parseView(views: readonly string[]): ComputerViews {}if (["all", "first", "last", "endpoints", "changed"].includes(value)) return { kind: "frames", selection: value as FrameSelection };const raw = json(value);+ // Rendering options (as, region, columns, maxLongEdge) go with any selection: select, indexes or from/to.+ const rendering = validateRendering(raw);+ for (const key of ["as", "region", "columns", "maxLongEdge"]) delete raw[key];+ if (rendering && !Object.keys(raw).length) raw.select = "all";+ if (raw.select !== undefined) {+ keys(raw, ["select"]);+ if (!["all", "first", "last", "endpoints", "changed"].includes(raw.select as string)) throw Error("select is all, first, last, endpoints or changed.");+ return { kind: "frames", selection: raw.select as FrameSelection, ...(rendering ? { rendering } : {}) };+ }const indexes = raw.indexes;if (indexes !== undefined) {keys(raw, ["indexes"]);@@ -96,7 +107,7 @@ function parseView(views: readonly string[]): ComputerViews {(raw.to as number) < (raw.from as number) || (raw.to as number) >= SEQUENCE_LIMITS.max_frames || (raw.to as number) - (raw.from as number) >= SEQUENCE_LIMITS.selected)throw Error(`Use an inclusive frame range within 0..${SEQUENCE_LIMITS.max_frames - 1} of at most ${SEQUENCE_LIMITS.selected} frames.`);}- return { kind: "frames", selection: raw as FrameSelection };+ return { kind: "frames", selection: raw as FrameSelection, ...(rendering ? { rendering } : {}) };}const capture: UiCaptureOptions = {};for (const view of views) {diff --git a/src/contact-sheet.ts b/src/contact-sheet.tsnew file mode 100644index 0000000..40e3985--- /dev/null+++ b/src/contact-sheet.ts@@ -0,0 +1,127 @@+import sharp from "sharp";+import type { SequenceFrame } from "./temporal-capture.ts";++/** A rectangle in a sequence's frame pixels. */+export interface FrameRegion { x: number; y: number; width: number; height: number }+/**+ * How selected frames are returned: one image each (the default), one tiled contact sheet, or one per-pixel+ * maximum or minimum composite. `region` crops every frame first; `maxLongEdge` bounds the returned image.+ */+export interface FrameRendering { as: "sheet" | "max" | "min"; region?: FrameRegion; columns?: number; maxLongEdge?: number }+/** Bounds of a rendered sheet or composite. */+export const RENDER_LIMITS = { columns: 10, maxLongEdge: 4096, defaultLongEdge: 1600 } as const;++/** One rendered image of several frames, with where each frame went. */+export interface RenderedFrames {+ png: Buffer; width: number; height: number;+ /** For a sheet, each tile's frame and place; for a composite, the frames combined. */+ tiles: { index: number; timestamp_ms: number; x: number; y: number; width: number; height: number }[];+ columns?: number; rows?: number;+ /** The crop actually applied to every frame, clipped to the frames. */+ region?: FrameRegion;+}++const LABEL_HEIGHT = 18;++/** Clip a region to a frame; refuses one that misses the frame. */+function clip(region: FrameRegion, width: number, height: number): FrameRegion {+ const x = Math.max(0, region.x), y = Math.max(0, region.y);+ const right = Math.min(width, region.x + region.width), bottom = Math.min(height, region.y + region.height);+ if (right <= x || bottom <= y) throw Error(`The region ${region.x},${region.y} ${region.width}x${region.height} lies outside the ${width}x${height} frames.`);+ return { x, y, width: right - x, height: bottom - y };+}++/** Decode a frame, cropped to the region when one is given. */+async function decode(frame: SequenceFrame, region: FrameRegion | undefined): Promise<{ data: Buffer; width: number; height: number; region?: FrameRegion }> {+ const image = sharp(Buffer.from(frame.image.data, "base64")).removeAlpha();+ const meta = await image.metadata();+ const crop = region ? clip(region, meta.width!, meta.height!) : undefined;+ const { data, info } = await (crop ? image.extract({ left: crop.x, top: crop.y, width: crop.width, height: crop.height }) : image)+ .raw().toBuffer({ resolveWithObject: true });+ return { data, width: info.width, height: info.height, region: crop };+}++const escape = (value: string) => value.replace(/[<>&]/g, char => ({ "<": "<", ">": ">", "&": "&" })[char]!);+/** A dark label strip with the frame's index and time, drawn as SVG text. */+function label(text: string, width: number): Buffer {+ return Buffer.from(`<svg xmlns="http://www.w3.org/2000/svg" width="${width}" height="${LABEL_HEIGHT}"><rect width="${width}" height="${LABEL_HEIGHT}" fill="#000"/>` ++ `<text x="4" y="13" font-family="sans-serif" font-size="12" fill="#fff">${escape(text)}</text></svg>`);+}++/**+ * Tile the frames in time order, left to right and top to bottom, each above its index/time label. Tiles keep the+ * frames' aspect ratio and shrink together so the sheet's long edge stays within maxLongEdge.+ */+async function sheet(frames: readonly SequenceFrame[], rendering: FrameRendering): Promise<RenderedFrames> {+ const decoded = await Promise.all(frames.map(frame => decode(frame, rendering.region)));+ const cell = { width: Math.max(...decoded.map(d => d.width)), height: Math.max(...decoded.map(d => d.height)) };+ const columns = Math.min(frames.length, rendering.columns ?? Math.ceil(Math.sqrt(frames.length)));+ const rows = Math.ceil(frames.length / columns);+ const limit = rendering.maxLongEdge ?? RENDER_LIMITS.defaultLongEdge;+ const scale = Math.min(1, limit / Math.max(columns * cell.width, rows * (cell.height + LABEL_HEIGHT)));+ const tile = { width: Math.max(1, Math.floor(cell.width * scale)), height: Math.max(1, Math.floor(cell.height * scale)) };+ const width = columns * tile.width, height = rows * (tile.height + LABEL_HEIGHT);+ const tiles = frames.map((frame, i) => ({ index: frame.index, timestamp_ms: frame.timestamp_ms, x: (i % columns) * tile.width,+ y: Math.floor(i / columns) * (tile.height + LABEL_HEIGHT) + LABEL_HEIGHT, width: tile.width, height: tile.height }));+ const layers = await Promise.all(decoded.map(async (d, i) => {+ const png = await sharp(d.data, { raw: { width: d.width, height: d.height, channels: 3 } })+ .resize(tile.width, tile.height, { fit: "contain", background: "#202020", kernel: "lanczos3" }).png().toBuffer();+ const { x, y } = tiles[i];+ return [{ input: label(`#${frames[i].index} · ${Math.round(frames[i].timestamp_ms)} ms`, tile.width), left: x, top: y - LABEL_HEIGHT },+ { input: png, left: x, top: y }];+ }));+ const png = await sharp({ create: { width, height, channels: 3, background: "#202020" } }).composite(layers.flat()).png().toBuffer();+ return { png, width, height, tiles, columns, rows, ...(decoded[0].region ? { region: decoded[0].region } : {}) };+}++/** The per-pixel maximum or minimum of the frames; every frame must have the same size. */+async function composite(frames: readonly SequenceFrame[], rendering: FrameRendering): Promise<RenderedFrames> {+ const decoded = await Promise.all(frames.map(frame => decode(frame, rendering.region)));+ const { width, height } = decoded[0];+ if (decoded.some(d => d.width !== width || d.height !== height))+ throw Error("A max or min composite needs frames of one size; the window changed size during the sequence. Select frames of one size or use a sheet.");+ const result = Buffer.from(decoded[0].data);+ const pick = rendering.as === "max" ? Math.max : Math.min;+ for (const d of decoded.slice(1)) for (let i = 0; i < result.length; i++) result[i] = pick(result[i], d.data[i]);+ let image = sharp(result, { raw: { width, height, channels: 3 } });+ const limit = rendering.maxLongEdge ?? RENDER_LIMITS.maxLongEdge;+ if (Math.max(width, height) > limit) image = image.resize({ width: width >= height ? limit : undefined, height: height > width ? limit : undefined, kernel: "lanczos3" });+ const { data: png, info } = await image.png().toBuffer({ resolveWithObject: true });+ return { png, width: info.width, height: info.height, tiles: frames.map(frame => ({ index: frame.index, timestamp_ms: frame.timestamp_ms, x: 0, y: 0, width: info.width, height: info.height })),+ ...(decoded[0].region ? { region: decoded[0].region } : {}) };+}++/** Render selected frames as one sheet or composite image. */+export function renderFrames(frames: readonly SequenceFrame[], rendering: FrameRendering): Promise<RenderedFrames> {+ if (!frames.length) throw Error("No frames were selected.");+ return rendering.as === "sheet" ? sheet(frames, rendering) : composite(frames, rendering);+}++/** Validate rendering options of a frames view before any frame is read. */+export function validateRendering(raw: Record<string, unknown>): FrameRendering | undefined {+ if (raw.as === undefined || raw.as === "frames") {+ if (["region", "columns", "maxLongEdge"].some(key => raw[key] !== undefined)) throw Error("region, columns and maxLongEdge apply to as:\"sheet\", \"max\" or \"min\".");+ return undefined;+ }+ if (raw.as !== "sheet" && raw.as !== "max" && raw.as !== "min") throw Error("as is frames (one image each), sheet (one tiled image), max or min (one per-pixel composite).");+ const rendering: FrameRendering = { as: raw.as };+ if (raw.region !== undefined) {+ const region = raw.region as Record<string, unknown>;+ if (!region || typeof region !== "object" || Array.isArray(region) || Object.keys(region).some(key => !["x", "y", "width", "height"].includes(key)) ||+ !["x", "y"].every(key => Number.isSafeInteger(region[key]) && (region[key] as number) >= 0) ||+ !["width", "height"].every(key => Number.isSafeInteger(region[key]) && (region[key] as number) >= 1))+ throw Error("region is {x,y,width,height} in the sequence's frame pixels: x, y >= 0, width, height >= 1.");+ rendering.region = region as unknown as FrameRegion;+ }+ if (raw.columns !== undefined) {+ if (raw.as !== "sheet" || !Number.isSafeInteger(raw.columns) || (raw.columns as number) < 1 || (raw.columns as number) > RENDER_LIMITS.columns)+ throw Error(`columns (1..${RENDER_LIMITS.columns}) applies to a sheet.`);+ rendering.columns = raw.columns as number;+ }+ if (raw.maxLongEdge !== undefined) {+ if (!Number.isSafeInteger(raw.maxLongEdge) || (raw.maxLongEdge as number) < 64 || (raw.maxLongEdge as number) > RENDER_LIMITS.maxLongEdge)+ throw Error(`maxLongEdge is 64..${RENDER_LIMITS.maxLongEdge}.`);+ rendering.maxLongEdge = raw.maxLongEdge as number;+ }+ return rendering;+}diff --git a/src/provider.ts b/src/provider.tsindex 5b21fb4..f26df09 100644--- a/src/provider.ts+++ b/src/provider.ts@@ -20,7 +20,7 @@ const label = (value: unknown) => typeof value === "string" ? value.replace(/[\r/** Agent guidance shared by the IDE plugin and the standalone tools. */export const COMPUTER_PATH_HELP = "Read docs:computer for native resources. computer:session, computer:machine/<name> (local, or a configured machine) with its /windows, /apps and /displays, its /display/primary or listed /display/ref, and returned computer:window/ref, observation/id, window-state/id or sequence/id, each running on the machine that issued it. Read window overviews before input; display/detail/frames are readonly.";-export const COMPUTER_VIEWS_HELP = "Computer: image/ui/full; dom alone or dom:{target,boxes,depth,log} on a browser window opened by open_browser or attached with attach_browser returns a fresh Playwright aria snapshot with [ref=...] values of the active tab; log:true adds the retained console/page-error/network history; target is {ref} or locator fields role,name,label,text,placeholder,test_id,css,exact,frames. On a DOM observation: image is a page screenshot, element:crop|highlight,<ref> an element image; Search filters its lines. state alone on a window returns fresh restore or management metadata without pixels/UI, plus whether the window is hung and its process memory/CPU. ocr (alone or with capture options on a window, alone on an observation, or with region:x,y,width,height on an observation) adds Windows OCR lines with boxes in overview pixels; use maxLongEdge 3840 for small text. aria (alone or with ui capture options, on a window) returns its UI as a Playwright aria snapshot with hints as [ref=...]. changes (alone, or with ui, image or full and capture options, on a window) takes a fresh capture and lists only rows changed, added or removed since that window's previous observation, with the new observation's hints, plus changed pixel areas with before/after crops instead of the full image. regions (alone or with image/full options on a window, alone on its observation) lists icons and text areas found by Cua perception with boxes in overview pixels; click a box center with pixel input. capture:{maxLongEdge,scale,max_elements,max_depth,hints,guides}. region:x,y,width,height or element:inspect|scope|highlight|crop|text,hint on an observation; element:text returns one element's full text. sequence:{duration_ms<=60000,fps<=30,max_frames<=300,timeout_ms,maxLongEdge,scale,region} on window/display reports actual_fps and last_frame_age_ms; frames:first|last|all|endpoints|changed or frames:{indexes:[0]} / {from:0,to:2} on sequence. Detail/frames are readonly; text offset/limit select UI lines, never image cells.";+export const COMPUTER_VIEWS_HELP = "Computer: image/ui/full; dom alone or dom:{target,boxes,depth,log} on a browser window opened by open_browser or attached with attach_browser returns a fresh Playwright aria snapshot with [ref=...] values of the active tab; log:true adds the retained console/page-error/network history; target is {ref} or locator fields role,name,label,text,placeholder,test_id,css,exact,frames. On a DOM observation: image is a page screenshot, element:crop|highlight,<ref> an element image; Search filters its lines. state alone on a window returns fresh restore or management metadata without pixels/UI, plus whether the window is hung and its process memory/CPU. ocr (alone or with capture options on a window, alone on an observation, or with region:x,y,width,height on an observation) adds Windows OCR lines with boxes in overview pixels; use maxLongEdge 3840 for small text. aria (alone or with ui capture options, on a window) returns its UI as a Playwright aria snapshot with hints as [ref=...]. changes (alone, or with ui, image or full and capture options, on a window) takes a fresh capture and lists only rows changed, added or removed since that window's previous observation, with the new observation's hints, plus changed pixel areas with before/after crops instead of the full image. regions (alone or with image/full options on a window, alone on its observation) lists icons and text areas found by Cua perception with boxes in overview pixels; click a box center with pixel input. capture:{maxLongEdge,scale,max_elements,max_depth,hints,guides}. region:x,y,width,height or element:inspect|scope|highlight|crop|text,hint on an observation; element:text returns one element's full text. sequence:{duration_ms<=60000,fps<=30,max_frames<=300,timeout_ms,maxLongEdge,scale,region} on window/display reports actual_fps and last_frame_age_ms; frames:first|last|all|endpoints|changed or frames:{indexes:[0]} / {from:0,to:2} / {select:'changed'} on sequence; add as:'sheet' (one tiled image with index/time labels, optional columns) or as:'max'|'min' (one per-pixel composite), with optional region:{x,y,width,height} in frame pixels and maxLongEdge. Detail/frames are readonly; text offset/limit select UI lines, never image cells.";export const COMPUTER_SEARCH_HELP = "Search computer:machine/<name>/apps or computer:machine/<name>/windows (local is this machine) for installed-app/permitted-window metadata, or an observation for its collected UI/readonly DOM. Read computer:machine/<name>/displays for monitor metadata. Discovery launches nothing; observation Search performs no new native walk or DOM query. Treat missing matches in a partial projection as unknown.";export const COMPUTER_INPUT_HELP = "Computer Write sends content literally through type_text to a fresh UI/image observation address. On a DOM observation of a browser opened with Insert {\"action\":\"open_browser\"} (YOLO: \"profile\":\"agent\" keeps the agent's own logins between sessions), or of a running Chromium-based browser attached in YOLO with Insert {\"action\":\"attach_browser\"} into its window address (Chrome or Edge through a debugging port, the chrome://inspect/#remote-debugging switch or the Playwright Extension; an Electron/CEF app started with --remote-debugging-port=0 in open's args; a WebView2 app started with open \"debugging\":true or after {\"action\":\"webview2_debugging\",\"app\":\"x.exe\",\"enabled\":true} on its machine; VS Code's integrated browser through the Pi Browser Bridge; closing the session detaches), Insert one Playwright action such as {\"action\":\"fill\",\"target\":{\"label\":\"Name\"},\"text\":\"value\"}; actions target the active tab, and tab/navigation/dialog actions are listed in docs:computer. Insert {\"action\":\"open_sandbox\"} into a machine address to work in a separate disposable Windows Sandbox that does not touch the human's desktop, or {\"action\":\"open_session\"} for an isolated agent session on the same machine (several can run at once, each with its own desktop and foreground input); use the returned computer:sandbox/... address for Search, open and close (docs:computer). Machines are computer:machine/local (this one) and computer:machine/<configured name> (another Windows machine over SSH, started on first use); each takes the same commands. Insert {\"action\":\"put\",\"from\":\"<workspace file>\"}, {\"action\":\"get\",\"name\":\"<file>\",\"to\":\"<workspace path>\"} or {\"action\":\"files\"} into a machine address to move files through the session's own folder on that machine (removed on close); point app Open/Save dialogs there. The same addresses take {\"action\":\"clipboard_write\",\"text\"|\"image\":...}, {\"action\":\"clipboard_read\",\"image\":true}, {\"action\":\"record_start\",\"seconds\":30} and {\"action\":\"record_stop\",\"to\":\"<workspace path>\"} (main-display video) and {\"action\":\"audio_record\",\"seconds\":10,\"transcribe\":true} (playback audio with a transcript) in YOLO. Insert {\"action\":\"timeline\",\"events\":[{at_ms,type,...}]} with delivery=foreground into an observation for timed key/button/move/move_relative/wheel/pen events; everything held is released at the end (docs:computer). Insert {\"action\":\"resize\",\"width\":1920,\"height\":1080} into a display address to change that display's resolution in YOLO; closing the session restores it. Only after the user explicitly allows it, Insert {\"action\":\"setup\",\"part\":\"browser|firefox|perception|mxc|vscode-bridge\"} into a machine address to install a missing optional part there (docs:computer). Insert {\"action\":\"menu\",\"path\":[\"File\",\"Save As...\"]} into an observation to invoke a menu item by its labels, and {\"action\":\"kill\"} into the window address of an app this session opened to force-close it. Insert an action with a Playwright locator target, such as {\"action\":\"click\",\"target\":\"getByRole('button', { name: 'Save' })\"} or fill/check/select, into a native window address to act without a prior Read; it waits until exactly one element is ready. Insert {\"action\":\"wait_for\"|\"verify\",\"conditions\":[...]} into a window address to check UI state with the same locators, count or an aria snapshot template; every action reports what it changed in the window: UI rows, and changed pixel areas with before/after crops after an action on an image or full observation (\"successor\":false skips it), and \"expect\" adds a postcondition; or Insert {\"action\":\"batch\",\"steps\":[...]} into an observation for a fail-stop chain (docs:computer). Use no Write or native input on a DOM observation. Use optional ?hint=...&delivery=foreground&delay_ms=0 only on input observations. No implicit select-all, submit or program execution. Computer Insert sends {\"action\":\"restore\"} to a fresh window-state address for explicit restoration and activation without keys/clicks, or one JSON action to an observation, or {\"action\":\"open\",\"app\":\"Exact installed name\"} (YOLO also {\"action\":\"open\",\"path\":\"C:\\\\...\\\\app.exe\",\"args\":[...]}) to a machine address, and {\"action\":\"close\"} to a machine address or to computer:session for every machine (the result lists each released part). Close releases runtime, not documents. Omit window_ref/observation_id in JSON. Input consumes observations even on refusal; the action's successor observation is current for the next input. With \"successor\":false, delivered pixel-only input keeps its observation until the window is resized; otherwise refresh with Read. One sequence takes up to 2000 key, click and stroke steps. Never replay unknown prefixes. Foreground input requires an observed background refusal and pauses for 10 s after the human's own keyboard or mouse input; restore is a separate explicit activation, not input delivery. Safe/YOLO is user-controlled.";diff --git a/src/session.ts b/src/session.tsindex 9f7042f..ca415a6 100644--- a/src/session.ts+++ b/src/session.ts@@ -1,6 +1,7 @@import { randomUUID, createHash } from "node:crypto";import { nativeDisplay, selectedDisplaySource, type NativeDisplay } from "./selected-displays.ts";import { collectSequence, cropSequenceFrame, frameSummary, selectFrames, sequenceMetadata, SequenceStop, SequenceStore, validateSequence, type SequenceOptions, type FrameSelection, type SequenceFrame } from "./temporal-capture.ts";+import { renderFrames, type FrameRendering, type RenderedFrames } from "./contact-sheet.ts";import type { GuideMark } from "./image-guides.ts";import { createUiResource, projectUi, resolveUi, annotateUi, validateUiCapture, type UiResource, type UiRequest, type UiCaptureOptions } from "./ui-resources.ts";import { isSemanticAction, semanticReceipt } from "./semantic-actions.ts";@@ -108,6 +109,8 @@ export interface ComputerResult {captured?: { image: boolean; ui: boolean };sequence?: ReturnType<typeof sequenceMetadata>;selected_frames?: Omit<SequenceFrame, "image">[];+ /** How a sheet or composite Read placed the selected frames in its one image. */+ rendered_frames?: Omit<RenderedFrames, "png">;/** Outcome of a bounded wait_for/verify; the conditions as requested, in order. */verification?: WaitResult & { spec: WaitSpec };/** Declared postcondition of an action: verified now, already true before, not reached, or undecidable. */@@ -631,7 +634,7 @@ export class ComputerSession {}/** Select retained pixels only after rechecking source identity and current access; no native recapture. */- sequenceFrames(id: string, selection: FrameSelection, signal?: AbortSignal): Promise<ComputerResult> {+ sequenceFrames(id: string, selection: FrameSelection, signal?: AbortSignal, rendering?: FrameRendering): Promise<ComputerResult> {return this.queue(async () => {const sequence = this.sequences.get(id);if (!sequence) return this.result("refused", "Sequence is unavailable or evicted. Capture again; resources do not survive reload/close.");@@ -647,6 +650,22 @@ export class ComputerSession {const frames = selectFrames(sequence, selection);const metadata = frames.map(({ image: _image, ...frame }) => frame);if (signal?.aborted) return this.result("refused", "Cancelled during frame selection; pixels withheld.");+ if (rendering) {+ let rendered;+ try { rendered = await renderFrames(frames, rendering); }+ catch (error) { return this.result("refused", `Frames not rendered: ${error instanceof Error ? error.message : error}`); }+ if (signal?.aborted) return this.result("refused", "Cancelled during frame rendering; pixels withheld.");+ if (!permitted()) return denied();+ const { png, ...layout } = rendered;+ const crop = layout.region ? ` of region ${layout.region.x},${layout.region.y} ${layout.region.width}x${layout.region.height}` : "";+ const summary = rendering.as === "sheet"+ ? `Contact sheet of ${frames.length} frames${crop}: ${layout.columns}x${layout.rows} tiles of ${layout.tiles[0].width}x${layout.tiles[0].height} in time order, left to right, each under its "#index · time" label; image ${layout.width}x${layout.height}.\n` ++ layout.tiles.map((tile, i) => `Tile ${i + 1} (row ${Math.floor(i / layout.columns!) + 1}, column ${i % layout.columns! + 1}): frame ${tile.index} at ${tile.timestamp_ms.toFixed(1)} ms`).join("\n")+ : `Per-pixel ${rendering.as === "max" ? "maximum" : "minimum"} of ${frames.length} frames${crop} (frames ${frames.map(frame => frame.index).join(", ")}); image ${layout.width}x${layout.height}. ` ++ `${rendering.as === "max" ? "Bright" : "Dark"} detail from any frame shows at once, for example text revealed by a moving light.`;+ return this.result("observed", `${summary}\nRead only; observe again before input.`, undefined, { selected_frames: metadata, rendered_frames: layout },+ [{ type: "image", mimeType: "image/png", data: png.toString("base64") }]);+ }// Interleave frame labels with their images so both direct calls and Codemode preserve correspondence.const result = this.result("observed", `Selected ${frames.length} read-only sequence frames. Their indexes and measured timestamps label the image order.`, undefined,{ selected_frames: metadata });diff --git a/test/contact-sheet.test.ts b/test/contact-sheet.test.tsnew file mode 100644index 0000000..2426a2d--- /dev/null+++ b/test/contact-sheet.test.ts@@ -0,0 +1,98 @@+import sharp from "sharp";+import { expect, test, vi } from "vitest";+import { renderFrames, validateRendering } from "../src/contact-sheet.ts";+import { parseComputerViews } from "../src/computer-views.ts";+import { presentComputerResult } from "../src/provider.ts";+import { ComputerSession, type Driver } from "../src/session.ts";+import type { SequenceFrame } from "../src/temporal-capture.ts";++/** A frame of solid colour with one bright pixel at (spot, spot). */+async function frame(index: number, width = 64, height = 48, spot?: number, background = { r: 40, g: 40, b: 40 }): Promise<SequenceFrame> {+ const raw = Buffer.alloc(width * height * 3);+ for (let i = 0; i < raw.length; i += 3) { raw[i] = background.r; raw[i + 1] = background.g; raw[i + 2] = background.b; }+ if (spot !== undefined) raw.fill(255, (spot * width + spot) * 3, (spot * width + spot) * 3 + 3);+ const png = await sharp(raw, { raw: { width, height, channels: 3 } }).png().toBuffer();+ return { index, slot: index, timestamp_ms: index * 100, capture_started_ms: 0, capture_latency_ms: 0, sample_latency_ms: 0, hash: String(index), bytes: png.length,+ geometry_changed: false, source: {} as SequenceFrame["source"], image: { type: "image", mimeType: "image/png", data: png.toString("base64") } };+}+const pixel = async (png: Buffer, x: number, y: number) => {+ const { data, info } = await sharp(png).raw().toBuffer({ resolveWithObject: true });+ const at = (y * info.width + x) * info.channels;+ return [data[at], data[at + 1], data[at + 2]];+};++// LPT-925: many frames come back as one image the agent reads at once.+test("a sheet tiles frames in time order under their index and time labels, within the size limit", async () => {+ const frames = await Promise.all(Array.from({ length: 16 }, (_, i) => frame(i, 640, 400)));+ const sheet = await renderFrames(frames, { as: "sheet" });+ expect({ columns: sheet.columns, rows: sheet.rows }).toEqual({ columns: 4, rows: 4 });+ expect(Math.max(sheet.width, sheet.height)).toBeLessThanOrEqual(1600);+ expect((await sharp(sheet.png).metadata()).width).toBe(sheet.width);+ expect(sheet.tiles.map(tile => tile.index)).toEqual(frames.map(f => f.index));+ // Tile 6 is row 2, column 2, below its label strip.+ expect(sheet.tiles[5]).toMatchObject({ x: sheet.tiles[0].width, y: 2 * 18 + sheet.tiles[0].height, timestamp_ms: 500 });+ // The label strip carries white text on black.+ const { data, info } = await sharp(sheet.png).extract({ left: 0, top: 0, width: sheet.tiles[0].width, height: 18 }).raw().toBuffer({ resolveWithObject: true });+ let bright = 0;+ for (let i = 0; i < data.length; i += info.channels) if (data[i] > 200) bright++;+ expect(bright).toBeGreaterThan(10);+ // Columns and a larger limit are honoured: 16 frames of a 640 px region keep full size in a 2-column 1280 px sheet.+ const wide = await renderFrames(frames.slice(0, 4), { as: "sheet", columns: 2, maxLongEdge: 1600 });+ expect({ columns: wide.columns, rows: wide.rows, tile: wide.tiles[0].width }).toEqual({ columns: 2, rows: 2, tile: 640 });+});++test("a region crops every frame first and is clipped to the frames", async () => {+ const frames = await Promise.all([0, 1].map(i => frame(i, 64, 48, 10)));+ const sheet = await renderFrames(frames, { as: "sheet", region: { x: 8, y: 8, width: 100, height: 100 } });+ expect(sheet.region).toEqual({ x: 8, y: 8, width: 56, height: 40 });+ expect(sheet.tiles[0]).toMatchObject({ width: 56, height: 40 });+ // The bright pixel at (10,10) is at (2,2) of the cropped tile.+ expect(await pixel(sheet.png, sheet.tiles[0].x + 2, sheet.tiles[0].y + 2)).toEqual([255, 255, 255]);+ await expect(renderFrames(frames, { as: "sheet", region: { x: 100, y: 0, width: 10, height: 10 } })).rejects.toThrow(/outside the 64x48 frames/);+});++test("max and min composites combine every frame per pixel", async () => {+ const frames = await Promise.all([frame(0, 32, 32, 3), frame(1, 32, 32, 20)]);+ const max = await renderFrames(frames, { as: "max" });+ expect(await pixel(max.png, 3, 3)).toEqual([255, 255, 255]);+ expect(await pixel(max.png, 20, 20)).toEqual([255, 255, 255]);+ expect(await pixel(max.png, 10, 10)).toEqual([40, 40, 40]);+ const min = await renderFrames([await frame(0, 32, 32), await frame(1, 32, 32, undefined, { r: 10, g: 90, b: 40 })], { as: "min" });+ expect(await pixel(min.png, 5, 5)).toEqual([10, 40, 40]);+ await expect(renderFrames([await frame(0, 32, 32), await frame(1, 40, 32)], { as: "max" })).rejects.toThrow(/frames of one size/);+});++test("frames views take a selection with rendering options and refuse invalid ones before reading", () => {+ expect(parseComputerViews(['frames:{"as":"sheet"}'])).toEqual({ kind: "frames", selection: "all", rendering: { as: "sheet" } });+ expect(parseComputerViews(['frames:{"select":"changed","as":"max","region":{"x":0,"y":0,"width":10,"height":10}}']))+ .toEqual({ kind: "frames", selection: "changed", rendering: { as: "max", region: { x: 0, y: 0, width: 10, height: 10 } } });+ expect(parseComputerViews(['frames:{"from":0,"to":15,"as":"sheet","columns":4,"maxLongEdge":2048}']))+ .toEqual({ kind: "frames", selection: { from: 0, to: 15 }, rendering: { as: "sheet", columns: 4, maxLongEdge: 2048 } });+ expect(parseComputerViews(['frames:{"indexes":[1,3]}'])).toEqual({ kind: "frames", selection: { indexes: [1, 3] } });+ for (const bad of [{ as: "video" }, { as: "max", columns: 2 }, { region: { x: 0, y: 0, width: 5, height: 5 } }, { as: "sheet", maxLongEdge: 10 },+ { as: "sheet", region: { x: -1, y: 0, width: 5, height: 5 } }, { as: "sheet", columns: 11 }])+ expect(() => validateRendering(bad)).toThrow();+ expect(() => parseComputerViews(['frames:{"select":"some","as":"sheet"}'])).toThrow(/select is/);+});++test("a sheet Read of a retained sequence returns one image and lists each tile", async () => {+ const png = await sharp({ create: { width: 40, height: 30, channels: 3, background: "red" } }).png().toBuffer();+ const driver: Driver = {+ identity: vi.fn(async () => "life"), close: vi.fn(async () => {}),+ call: vi.fn(async (name: string) => name === "list_windows" ? { content: [], structuredContent: { windows: [{ pid: 42, window_id: 91 }] } } :+ { content: [{ type: "image" as const, mimeType: "image/png", data: png.toString("base64") }], structuredContent: { pid: 42, window_id: 91,+ window_bounds: { x: 10, y: 20, width: 40, height: 30 }, screenshot_width: 40, screenshot_height: 30, screenshot_original_width: 40, screenshot_original_height: 30, elements: [] } }),+ };+ const session = new ComputerSession(driver, () => "yolo");+ const ref = ((await session.windows()).details.state!.windows as { window_ref: string }[])[0].window_ref;+ const captured = await session.captureSequence({ window_ref: ref }, { duration_ms: 400, fps: 10, max_frames: 3 });+ const id = captured.details.sequence!.sequence_id;+ const sheet = await session.sequenceFrames(id, "all", undefined, { as: "sheet" });+ expect(sheet.details.status).toBe("observed");+ expect(sheet.content.filter(block => block.type === "image")).toHaveLength(1);+ expect(sheet.details.rendered_frames!.tiles).toHaveLength(3);+ expect(presentComputerResult(sheet)).toMatch(/Contact sheet of 3 frames: 2x2 tiles[\s\S]*Tile 3 \(row 2, column 1\): frame 2 at/);+ const refused = await session.sequenceFrames(id, "all", undefined, { as: "max", region: { x: 50, y: 0, width: 5, height: 5 } });+ expect(refused.details.status).toBe("refused");+ expect(presentComputerResult(refused)).toMatch(/Frames not rendered: The region .* outside/);+});