Skip to content

Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.

feat!: a window has no region view; detail of part of a window comes from a viewport; docs (LPT-931)

alexshpunt-clanker[bot]c977df4bca21
diff --git a/.pi/skills/computer-use/SKILL.md b/.pi/skills/computer-use/SKILL.md
index b493f0f..bbd09eb 100644
--- a/.pi/skills/computer-use/SKILL.md
+++ b/.pi/skills/computer-use/SKILL.md
@@ -19,7 +19,7 @@ When several agents must work at once, or the work needs foreground input withou
On another configured machine, use its own addresses exactly as for this one, and Delete the machine address when the work there is done.
-For native detail, choose rectangles from the overview already seen and Read `region:x,y,width,height` on its observation, or crop one element. Detail is readonly.
+For work in one part of a window, such as a canvas, a map or a game board, make a viewport with `computer_view` from a rectangle of the overview already seen, then pass its address (`computer:game/pano`) to the input tools: points are pixels of the viewport's image, which has the window's own resolution, and the change list covers only the viewport. Read the viewport to look again, with `ocr` for small text; it follows the window. To see one element closer, crop it with `element:crop`; a crop is readonly.
Read displays only within authorized YOLO scope. If a high-resolution display makes overviews too dense, `computer_display` changes its resolution; the session restores it on close, and earlier display and window observations stop working. Display, detail and frame pixels never become input points.
diff --git a/README.md b/README.md
index 58f19a2..7d91b94 100644
--- a/README.md
+++ b/README.md
@@ -64,7 +64,8 @@ From there the agent can:
- see what each action changed: the UI rows, and a small picture of the window or page with the changed areas outlined and numbered, the one at the input first, plus enlarged before/after crops of them;
- read only what changed since the last read with `changes`;
- run `ocr` on canvases and games, or `regions` for icons and text areas found by Cua perception;
-- zoom into part of the latest overview, or crop one element at full resolution;
+- name a viewport, a rectangle of a window such as a game canvas, with `computer_view`: it shows that part at the window's own resolution, follows the window, takes clicks at points of its image and reports what changed only there;
+- crop one element at full resolution;
- record a short sequence of frames with `computer_record` to watch an animation or a progress bar, then pick the frames it needs;
- read a window's `state` to see whether the app is hung and how much memory it uses.
diff --git a/docs/computer-resources.md b/docs/computer-resources.md
index 0979338..e92a894 100644
--- a/docs/computer-resources.md
+++ b/docs/computer-resources.md
@@ -19,9 +19,10 @@ The desktop scope is Windows, with Pi running on it directly or in its WSL, on t
| --- | --- |
| Read (`computer_read`) | Looks at a window, display, observation, sequence or machine: pixels, UI tree, page snapshot, OCR, window state |
| Search (`computer_search`) | Finds a machine's windows or installed apps, or filters the lines of an observation |
-| Delete (`computer_delete`) | Closes a window like its close button, or releases a sandbox, isolated session, machine or `computer:session` |
+| Delete (`computer_delete`) | Closes a window like its close button, removes a viewport, or releases a sandbox, isolated session, machine or `computer:session` |
| `computer_open` | Starts an app, a Playwright browser, a Windows Sandbox or an isolated session; attaches Playwright to a running browser window; sets WebView2 debugging for an app |
| `computer_window` | Moves, resizes, maximizes, minimizes, focuses or restores a window, or gives it a short name |
+| `computer_view` | Makes, moves or resizes a [viewport](#viewports), a named rectangle of a window, and shows it at the window's own resolution |
| `computer_display` | Changes a display's resolution |
| `computer_click`, `computer_hover`, `computer_drag`, `computer_scroll` | Pointer input on an element or a point |
| `computer_type`, `computer_press`, `computer_select`, `computer_timeline` | Text, keys and chords, list options, checkboxes and menus, timed raw events |
@@ -42,13 +43,14 @@ Several steps in a row go into a Codemode script that calls these tools; each ca
| `computer:machine/<name>/windows`, `/apps` | Its windows and installed apps, for Search |
| `computer:machine/<name>/displays`, `/display/primary` | Its displays, for Read; listed displays are `/display/<ref>` |
| `computer:window/...` | A window, returned by Search or `computer_open` |
+| `computer:window/.../<name>` | A [viewport](#viewports) of that window, made by `computer_view` |
| `computer:observation/...` | One Read of a window or display, or a page snapshot |
| `computer:sequence/...` | Frames captured by `computer_record` `sequence` |
| `computer:app/...` | An installed app from Search; `computer_open` opens it |
| `computer:sandbox/...` | A Windows Sandbox or an isolated session |
| `computer:session` | Everything this computer use started |
-A machine starts on its first use; nothing needs connecting. No address falls back to this machine: window, observation and sequence addresses belong to the machine or sandbox that issued them, and every call on them runs there. An address that no machine of this session issued, or that was released, refuses with the reason before anything is sent. Every receipt names its machine in its first line, for example `Computer observed (yolo) on lab`. Read `computer:session` for the machines, sandboxes, sessions, window names and windows so far.
+A machine starts on its first use; nothing needs connecting. No address falls back to this machine: window, observation and sequence addresses belong to the machine or sandbox that issued them, and every call on them runs there. An address that no machine of this session issued, or that was released, refuses with the reason before anything is sent. Every receipt names its machine in its first line, for example `Computer observed (yolo) on lab`. Read `computer:session` for the machines, sandboxes, sessions, window names, viewports and windows so far.
Results show addresses in a short form, `computer:<machine>/<letter><number><tag>`: `computer:lab/w3qk` for a window, `o` for an observation, `s` for a sequence, `a` for an app, `b` for a sandbox and `d` for a listed display. The two-letter tag is drawn once per Pi run, so an address kept from an earlier run refuses instead of reaching whatever window got the same number. The long forms keep working and stay in structured details.
@@ -67,10 +69,10 @@ Read a window for its UI tree and a small overview image. Overviews have a 1280
| `dom`, `dom:{"target":...,"depth":4,"boxes":true,"log":true}` | A fresh aria snapshot of the active page of a browser opened or attached with `computer_open`, with refs such as `e12` or `f1e3` in frames; `log` adds the retained console and network log. Use alone |
| `state` | The window's minimized, maximized, visible and foreground state and bounds, and whether its process is hung with its memory and CPU time; no pixels or UI. Use alone. `computer_window` changes the state |
| `changes` | A fresh Read that lists only rows changed (`~`), added (`+`) or removed (`-`) since the window's previous observation, and changed pixel areas with before/after crops. Use it when re-reading after a wait |
-| `ocr` | Windows OCR of the overview pixels, lines with boxes in overview pixels. Alone or with capture options on a window, alone on an observation, or with `region:x,y,w,h` on one. Small text needs `capture:{"granularity":"image","maxLongEdge":3840}`. Recognised text can be wrong |
+| `ocr` | Windows OCR of the image, lines with boxes in its pixels. Alone or with capture options on a window, alone on an observation or a [viewport](#viewports). Small text needs a viewport or `capture:{"granularity":"image","maxLongEdge":3840}`. Recognised text can be wrong |
| `regions` | Icons and text areas found by [Cua perception](https://github.com/trycua/cua/releases/tag/cua-perception-v0.2.1) in the overview pixels, for controls without UI names such as canvas toolbars. Needs `/computer-setup perception` on that machine |
| `capture:{"max_elements":500,"max_depth":20,"hints":true,"maxLongEdge":...,"scale":...}` | Limits of one fresh Read; optional `guides` |
-| `region:x,y,w,h` | Readonly detail of a rectangle in the observation's overview pixels |
+| `region:x,y,w,h` | Readonly detail of a rectangle in a display observation's pixels. For part of a window, make a [viewport](#viewports) |
| `element:inspect,1:a` | One element of an observation; also `scope`, `highlight`, `crop` and `text` (its full text). On a page snapshot, `element:crop,<ref>` and `element:highlight,<ref>` |
| `frames:endpoints` | Frames of a sequence: `first`, `last`, `all`, `changed`, `{"indexes":[0,2]}` or `{"from":0,"to":2}`; add `"as":"sheet"` for one tiled image, or `"max"`/`"min"` for a per-pixel composite ([temporal capture](temporal-capture.md#one-image-of-many-frames)) |
@@ -78,11 +80,11 @@ Read an observation with `offset` and `limit` to page through its text; pages co
## Act on windows
-The input tools take a window address, never an observation. The window decides how input goes: a browser opened or attached by computer use runs it through Playwright on its active page, any other window through the native driver and UI Automation. A target is one of:
+The input tools take a window or viewport address, never an observation. The window decides how input goes: a browser opened or attached by computer use runs it through Playwright on its active page, any other window through the native driver and UI Automation. A target is one of:
- **A ref** from the window's latest Read: a native ref such as `1:a` (from `full`, `ui` or `aria`), or a page ref such as `e12` from a `dom` Read. Input spends native refs; its change list brings the new ones.
- **A Playwright locator** such as `getByRole('button', { name: 'Save' })`, resolved with auto-waiting (`timeout_ms`, default 5000).
-- **A point** (`x`, `y`). On a native window it is in pixels of the latest image of that window, from a Read with an image or from an action's change list; without one the tool captures the window first, about 300 ms once. On a browser page a point is in CSS pixels; `space: "window"` uses pixels of a window Read, for example for the browser's own toolbar.
+- **A point** (`x`, `y`). On a native window it is in pixels of the latest image of that window, from a Read with an image or from an action's change list; without one the tool captures the window first, about 300 ms once. On a [viewport](#viewports) it is in pixels of the viewport's image. On a browser page a point is in CSS pixels; `space: "window"` uses pixels of a window Read, for example for the browser's own toolbar.
On native windows the target decides how input goes. What the agent sees structurally is pressed structurally, and what it sees only as pixels gets the real pointer:
@@ -98,6 +100,14 @@ Keys on a browser page go to the page, so browser shortcuts such as `alt+ArrowLe
A refused action sent nothing. An `unknown` outcome may have reached the target: Read again and never replay it blindly. Delivery is not proof of the application's response.
+### Viewports
+
+A viewport is a named rectangle of a window, for work that stays in one part of it, such as a game canvas, a map or a grid. `computer_view` makes one from `x`, `y`, `width` and `height` in pixels of the window's latest image, the same pixels as points of the input tools, and shows it. Its address is the window's name or address with the viewport's name after it: `computer:game/pano`, or `computer:lab/w3qk/pano` for a window without a name. A window can have several viewports. The rectangle is kept relative to the window's top-left corner, so it follows the window when the window moves. `computer_view` on an existing viewport moves or resizes it, and fields left out keep their values. Delete removes a viewport; it also goes away with its window.
+
+The viewport's image has the window's own resolution, up to 1568 px on its long edge; a larger viewport is scaled down to fit. A window larger than its overview shows more detail in a viewport than in the overview. Read the viewport address to look again, and add `ocr` for the text in it. A viewport Read only looks: the window's latest Read and its refs stay current.
+
+Give the viewport address to an input tool as `window`, and points are pixels of the viewport's image; a point outside it refuses. Refs and locators act as on the window. The change list after the action compares only the viewport's pixels, in pixels of its image, and the receipt names the viewport and its window. Input at a point of a viewport first captures the window at full resolution when its current capture is smaller.
+
### Native locators
On native windows, Playwright's own parser reads the locator and this package evaluates it over the window's UI Automation tree with Playwright's semantics:
@@ -115,7 +125,7 @@ CSS and XPath refuse on native windows. Each action takes fresh UI walks until e
Every action reports what it changed, the way Playwright MCP's incremental snapshots do: `~` changed, `+` added and `-` removed elements, plus URL, title or tab changes, such as `~ 3:a Button "Count 0" → "Count 1"`. Changed and added rows carry the new refs. A locator action compares with the walk that resolved its target; a page action compares Playwright's snapshots before and after.
-Every action also compares pixels: on a native window the window's overview pixels, on a browser page Playwright screenshots of the page in CSS pixels. Image 1 is a small copy of the window or page with each changed area outlined and numbered as listed. Before and after crops of the first three areas follow it, with small areas enlarged up to 4x:
+Every action also compares pixels: on a native window the window's overview pixels, in a viewport only the viewport's pixels, on a browser page Playwright screenshots of the page in CSS pixels. Image 1 is a small copy of the window or page with each changed area outlined and numbered as listed. Before and after crops of the first three areas follow it, with small areas enlarged up to 4x:
```text
Pixels changed in 2 areas (overview pixels; 0.07% of the window); image 1 outlines and numbers them on the window:
@@ -232,7 +242,7 @@ else if (r.outcome === "refused") { /* nothing was sent */ }
| `outcome` | `delivered`, `refused`, `unknown`, `observed` or `closed`. `refused` sent nothing; `unknown` may have reached the target |
| `effect` | `applied`, `not-applied` or `unknown`; a refusal after an already delivered step stays `unknown` |
| `stage` | `validation` when the request was rejected before reaching the target, otherwise `target` |
-| `window`, `observation`, `sandbox` | The window, the new current observation and a sandbox `computer_open` started |
+| `window`, `viewport`, `observation`, `sandbox` | The window, the viewport a Read showed or an input went to, the new current observation and a sandbox `computer_open` started |
| `application_effect` | Always `unverified`: delivery is not proof of the application's response |
| `reason`, `code` | Why a refused or unknown action ended |
| `page`, `evaluation`, `verification`, `changes`, `steps` | Present when the action produced them |
diff --git a/docs/observation-granularity.md b/docs/observation-granularity.md
index 7c69e39..a178f1e 100644
--- a/docs/observation-granularity.md
+++ b/docs/observation-granularity.md
@@ -20,7 +20,7 @@ Start with full when learning a layout or identifying controls. Use image-only f
Do not carry a full read's hints into an image-only read. Determine selection or completion from current pixels if the loop has no current UI resource. Stop on layout changes, uncertain state or an expired decision.
-Use UI-only when current reported labels, values and supported native control actions are enough. Local `search` and `inspect` work without another native read. `scope`, `highlight`, `crop` and region detail need image data and refuse after UI-only reads. Current UI nodes have native-screen bounds where reported, not window-image bounds. Do not convert those bounds into pixel input.
+Use UI-only when current reported labels, values and supported native control actions are enough. Local `search` and `inspect` work without another native read. `scope`, `highlight` and `crop` need image data and refuse after UI-only reads. Current UI nodes have native-screen bounds where reported, not window-image bounds. Do not convert those bounds into pixel input.
Image-only accepts image guides but rejects UI query, walk limits and enabled hint labels. UI-only accepts UI query/walk limits but rejects hints and guides. Full keeps the existing controls. Invalid combinations fail explicitly rather than silently changing mode.
diff --git a/src/computer-resources.ts b/src/computer-resources.ts
index c12ae21..c320a31 100644
--- a/src/computer-resources.ts
+++ b/src/computer-resources.ts
@@ -51,6 +51,8 @@ async function cropImages(before: Buffer, after: Buffer, diff: StateDiff): Promi
pixels.crops = cropped.map(item => ({ ...item.region, scale: item.scale }));
return [...(overview ? [image(overview)] : []), ...cropped.flatMap(item => [image(item.before), image(item.after)])];
}
+/** Detail of a rectangle of a window comes from a viewport; a display observation keeps its region view. */
+const WINDOW_REGION = "A window has no region view: make a viewport of that rectangle with computer_view, which shows it at the window's own resolution. Read the viewport, add ocr for its text, and act at points of its image.";
/** Web content up to this far above its topmost element still counts as page, for the page's own margin. */
const PAGE_MARGIN = 12;
/**
@@ -641,11 +643,12 @@ export class ComputerResources {
if (address.kind === "viewport") return this.readViewport(address, cwd, views, signal);
const view = parseComputerViews(views);
if (address.kind !== "window" && address.kind !== "observation") {
- if (view.ocr) throw new ComputerRequestError("ocr reads a window, one of its observations or a region of one.");
+ if (view.ocr) throw new ComputerRequestError("ocr reads a window, one of its observations or a viewport.");
return this.readNative(address, cwd, views, view, signal);
}
const snapshot = address.kind === "observation" ? this.snapshots.get(address.ref) : undefined;
const windowRef = address.kind === "window" ? address.ref : snapshot?.target.window_ref;
+ if (view.kind === "region" && windowRef !== undefined) throw new ComputerRequestError(WINDOW_REGION);
if (view.ocr && windowRef === undefined) throw new ComputerRequestError("ocr reads window observations; display observations have no kept overview pixels.");
if (view.changes && address.kind !== "window") throw new ComputerRequestError("changes reads a window address: it takes a fresh capture and compares it with the window's previous observation.");
const session = (await this.routeAddress(cwd, address, signal)).session;
@@ -655,11 +658,8 @@ export class ComputerResources {
if (frame && view.kind === "overview" && view.capture.maxLongEdge === undefined && view.capture.scale === undefined) Object.assign(view.capture, frame.size);
const target = windowRef === undefined ? undefined : session.windowTarget(windowRef);
if (view.regions && windowRef === undefined) throw new ComputerRequestError("regions reads a window or one of its observations.");
- // A region of a stored observation is read from its own pixels: exact boxes, no native call, input authority kept.
- let result = view.ocr && view.kind === "region" && snapshot && windowRef !== undefined
- ? await this.recognize(session, snapshot.result, windowRef, view.region, signal)
- // An observation keeps its own capture: regions parses it without another native read.
- : view.regions && snapshot ? snapshot.result
+ // An observation keeps its own capture: regions parses it without another native read.
+ let result = view.regions && snapshot ? snapshot.result
: await this.readNative(address, cwd, views, view, signal);
// A window whose pixels cannot be captured still lists its UI changes, unless pixels were asked for explicitly.
if (view.changes && view.kind === "overview" && result && result.details.status !== "observed" && !views?.some(item => item === "full" || item === "image"))
@@ -694,20 +694,20 @@ export class ComputerResources {
const health = await session.processHealth(target);
return health && !health.responding ? { ...result, content: [...result.content, { type: "text", text: unresponsive(health) }], details: { ...result.details, process: health } } : result;
}
- return view.ocr && view.kind !== "region" && windowRef !== undefined ? this.recognize(session, result, windowRef, undefined, signal) : result;
+ return view.ocr && windowRef !== undefined ? this.recognize(session, result, windowRef, signal) : result;
}
- /** Add Windows OCR lines of an observation's unannotated overview pixels, or of one region of them. */
- private async recognize(session: ComputerSession, result: ComputerResult, windowRef: string, region: { x: number; y: number; width: number; height: number } | undefined, signal?: AbortSignal): Promise<ComputerResult> {
+ /** Add Windows OCR lines of an observation's unannotated overview pixels. */
+ private async recognize(session: ComputerSession, result: ComputerResult, windowRef: string, signal?: AbortSignal): Promise<ComputerResult> {
const id = result.details.observation_id;
const image = id === undefined ? undefined : session.observationImage(windowRef, id);
if (!image) throw new ComputerRequestError("This observation's unannotated pixels are no longer kept. Read the window with ocr again.");
let lines;
- try { lines = await session.recognizeText(image, region, signal); }
+ try { lines = await session.recognizeText(image, undefined, signal); }
catch (error) { throw new ComputerRequestError(`Text recognition failed: ${error instanceof Error ? error.message : error}`); }
const listing = lines.length ? lines.map(line => `- "${line.text}" at ${line.x},${line.y} ${line.width}x${line.height}`).join("\n") : "(no text found)";
- const header = `Windows OCR${region ? ` of region ${region.x},${region.y} ${region.width}x${region.height}` : ""}: ${lines.length} lines; boxes are in this observation's overview pixels. Recognised text can be wrong; confirm with pixels or UI before acting on it.`;
- return { ...result, content: [...(region ? [] : result.content), { type: "text", text: `${header}\n${listing}` }], details: { ...result.details, text_lines: lines } };
+ const header = `Windows OCR: ${lines.length} lines; boxes are in this observation's overview pixels. Recognised text can be wrong; confirm with pixels or UI before acting on it.`;
+ return { ...result, content: [...result.content, { type: "text", text: `${header}\n${listing}` }], details: { ...result.details, text_lines: lines } };
}
private async readNative(address: ComputerAddress, cwd: string, views: readonly string[] | undefined, view: ComputerViews, signal?: AbortSignal): Promise<ComputerResult | undefined> {
@@ -742,7 +742,7 @@ export class ComputerResources {
if (view.kind === "state") return session.observeWindowState(address.ref, signal);
this.retireWindow(address.ref, view.kind === "dom" ? "dom" : "native");
if (view.kind === "dom") return this.remember({ window_ref: address.ref }, await session.observeDom(address.ref, view.options, signal));
- if (view.kind !== "overview") throw new ComputerRequestError("Read an overview, then apply region to its returned observation address.");
+ if (view.kind !== "overview") throw new ComputerRequestError("Read an overview of the window, then element views on its returned observation address.");
return this.remember({ window_ref: address.ref }, await session.observe(address.ref, signal, undefined, undefined, undefined, view.capture));
}
if (address.kind === "display") {
@@ -782,13 +782,12 @@ export class ComputerResources {
{ operation: view.operation, observation_id: address.ref, hint: view.hint });
}
if (view.kind === "region") {
+ if (snapshot.target.window_ref !== undefined) throw new ComputerRequestError(WINDOW_REGION);
// Retire the provider handle even when detail refuses; never turn detail into an input target.
this.snapshots.delete(address.ref);
const imageId = snapshot.result.details.agent_image?.image_id;
if (typeof imageId !== "string") throw new ComputerRequestError("This observation has no image for detail.");
- const zoom = { image_id: imageId, ...view.region };
- return snapshot.target.window_ref !== undefined ? session.observe(snapshot.target.window_ref, signal, undefined, zoom)
- : session.observeDisplay(snapshot.target.display!, signal, zoom);
+ return session.observeDisplay(snapshot.target.display!, signal, { image_id: imageId, ...view.region });
}
const checked = await session.checkObservation(snapshot.target, address.ref, signal);
if (checked.details.status !== "observed") { this.snapshots.delete(address.ref); return checked; }
diff --git a/src/computer-views.ts b/src/computer-views.ts
index a1cb728..63d8149 100644
--- a/src/computer-views.ts
+++ b/src/computer-views.ts
@@ -25,7 +25,7 @@ function keys(value: Record<string, unknown>, allowed: readonly string[]): void
}
/** Validate owned views before capturing; text offsets are never image coordinates.
- * `ocr` adds Windows text recognition to a window overview, an observation or a region detail.
+ * `ocr` adds Windows text recognition to a window overview, an observation or a viewport.
* `changes` reads a window's UI and pixels and lists only what changed since its previous observation.
* `regions` adds icons and text areas found by Cua perception to a window overview or its current observation.
* `aria` reads a window's UI as a Playwright aria snapshot whose refs are the observation's hints. */
@@ -50,8 +50,8 @@ export function parseComputerViews(views: readonly string[] = []): ComputerViews
}
if (!views.includes("ocr")) return parseView(views);
const view = parseView(views.filter(view => view !== "ocr"));
- if (view.kind !== "overview" && view.kind !== "region") throw Error("Use ocr alone or with region:x,y,width,height or capture options; it reads text from pixels.");
- if (view.kind === "overview" && view.capture.granularity === "ui") throw Error("ocr needs pixels; use image or full granularity.");
+ if (view.kind !== "overview") throw Error("Use ocr alone or with capture options; it reads text from pixels. For the text of one part of a window, Read a viewport of it with ocr.");
+ if (view.capture.granularity === "ui") throw Error("ocr needs pixels; use image or full granularity.");
return { ...view, ocr: true };
}
diff --git a/src/provider.ts b/src/provider.ts
index a2cc331..cb2bcfe 100644
--- a/src/provider.ts
+++ b/src/provider.ts
@@ -20,7 +20,7 @@ const label = (value: unknown) => typeof value === "string" ? value.replace(/[\r
/** Agent guidance shared by the IDE plugin and the standalone tools. */
export const COMPUTER_PATH_HELP = "Read docs:computer for native resources. computer:session, computer:machine/<name> (local, or a configured machine) with its /windows, /apps and /displays, its /display/primary or listed /display/ref, and returned computer:window/ref, observation/id or sequence/id, each running on the machine that issued it; results show short forms such as computer:<machine>/w3qk (o observation, s sequence), and computer:<name> after computer_window name. A viewport that computer_view made is its window's address with its name after it, such as computer:game/pano or computer:<machine>/w3qk/pano; Read shows its image at full resolution, ocr adds its text. Read only looks; the computer_* tools act.";
-export const COMPUTER_VIEWS_HELP = "Computer: image/ui/full; dom alone or dom:{target,boxes,depth,log} on a browser window opened or attached with computer_open returns a fresh Playwright aria snapshot with [ref=...] values of the active tab; log:true adds the retained console/page-error/network history; target is {ref} or locator fields role,name,label,text,placeholder,test_id,css,exact,frames. On a DOM observation: image is a page screenshot, element:crop|highlight,<ref> an element image; Search filters its lines. state alone on a window returns its minimized, maximized, visible and foreground state and bounds without pixels/UI, plus whether the window is hung and its process memory/CPU; computer_window changes them. ocr (alone or with capture options on a window, alone on an observation, or with region:x,y,width,height on an observation) adds Windows OCR lines with boxes in overview pixels; use maxLongEdge 3840 for small text. aria (alone or with ui capture options, on a window) returns its UI as a Playwright aria snapshot with hints as [ref=...]. changes (alone, or with ui, image or full and capture options, on a window) takes a fresh capture and lists only rows changed, added or removed since that window's previous observation, with the new observation's hints, plus changed pixel areas with before/after crops instead of the full image. regions (alone or with image/full options on a window, alone on its observation) lists icons and text areas found by Cua perception with boxes in overview pixels; click a box centre with computer_click at its x,y. capture:{maxLongEdge,scale,max_elements,max_depth,hints,guides}. region:x,y,width,height or element:inspect|scope|highlight|crop|text,hint on an observation; element:text returns one element's full text. frames:first|last|all|endpoints|changed or frames:{indexes:[0]} / {from:0,to:2} / {select:'changed'} on a sequence address from computer_record; add as:'sheet' (one tiled image with index/time labels, optional columns) or as:'max'|'min' (one per-pixel composite), with optional region:{x,y,width,height} in frame pixels and maxLongEdge. Detail/frames are readonly; text offset/limit select UI lines, never image cells.";
+export const COMPUTER_VIEWS_HELP = "Computer: image/ui/full; dom alone or dom:{target,boxes,depth,log} on a browser window opened or attached with computer_open returns a fresh Playwright aria snapshot with [ref=...] values of the active tab; log:true adds the retained console/page-error/network history; target is {ref} or locator fields role,name,label,text,placeholder,test_id,css,exact,frames. On a DOM observation: image is a page screenshot, element:crop|highlight,<ref> an element image; Search filters its lines. state alone on a window returns its minimized, maximized, visible and foreground state and bounds without pixels/UI, plus whether the window is hung and its process memory/CPU; computer_window changes them. ocr (alone or with capture options on a window, alone on an observation or a viewport) adds Windows OCR lines with boxes in pixels of that image; use maxLongEdge 3840 or a viewport for small text. aria (alone or with ui capture options, on a window) returns its UI as a Playwright aria snapshot with hints as [ref=...]. changes (alone, or with ui, image or full and capture options, on a window) takes a fresh capture and lists only rows changed, added or removed since that window's previous observation, with the new observation's hints, plus changed pixel areas with before/after crops instead of the full image. regions (alone or with image/full options on a window, alone on its observation) lists icons and text areas found by Cua perception with boxes in overview pixels; click a box centre with computer_click at its x,y. capture:{maxLongEdge,scale,max_elements,max_depth,hints,guides}. element:inspect|scope|highlight|crop|text,hint on a window observation; element:text returns one element's full text. For detail of part of a window, computer_view makes a viewport; region:x,y,width,height applies to display observations only. frames:first|last|all|endpoints|changed or frames:{indexes:[0]} / {from:0,to:2} / {select:'changed'} on a sequence address from computer_record; add as:'sheet' (one tiled image with index/time labels, optional columns) or as:'max'|'min' (one per-pixel composite), with optional region:{x,y,width,height} in frame pixels and maxLongEdge. Detail/frames are readonly; text offset/limit select UI lines, never image cells.";
export const COMPUTER_SEARCH_HELP = "Search computer:machine/<name>/apps or computer:machine/<name>/windows (local is this machine) for installed-app/permitted-window metadata, or an observation for its collected UI/readonly DOM. Read computer:machine/<name>/displays for monitor metadata. Discovery launches nothing; observation Search performs no new native walk or DOM query. Treat missing matches in a partial projection as unknown.";
export const COMPUTER_DELETE_HELP = "A computer window address closes that window like its close button (add ?force=true to kill the hung app of a window this session opened); a sandbox, isolated session or machine address releases it; computer:session releases everything computer use started.";
/** Each Insert command the typed tools replaced, by the tool that does it now. */
diff --git a/test/computer-resources.test.ts b/test/computer-resources.test.ts
index fa2fc02..bb0e046 100644
--- a/test/computer-resources.test.ts
+++ b/test/computer-resources.test.ts
@@ -180,9 +180,20 @@ test("display resources and their observations never take input", async () => {
expect(f.captures()).toHaveLength(1);
});
-test("a successful rectangle Read is readonly detail and consumes its overview without a new capture", async () => {
+// LPT-931: a rectangle of a window is a viewport; the region view of a window observation refuses and keeps it current.
+test("a region Read of a window observation refuses before anything is sent and points to computer_view", async () => {
const f = await fixture(), view = await f.read();
- const detail = await f.resources.read(view.observation, cwd, ["region:80,60,40,20"]);
+ await expect(f.resources.read(view.observation, cwd, ["region:80,60,40,20"])).rejects.toThrow(/computer_view/);
+ await expect(f.resources.read(f.window, cwd, ["region:80,60,40,20"])).rejects.toThrow(/computer_view/);
+ expect(f.native().filter(([name]) => name === "zoom")).toHaveLength(0);
+ expect(f.captures()).toHaveLength(1);
+ // The observation is still current for input.
+ expect((await f.resources.perform(view.observation, cwd, JSON.stringify({ action: "click", x: 100, y: 70 }))).details.status).toBe("delivered");
+});
+
+test("an element crop is readonly detail and consumes its overview without a new capture", async () => {
+ const f = await fixture(), view = await f.read();
+ const detail = await f.resources.read(view.observation, cwd, [`element:crop,${view.hint}`]);
expect(detail?.details.status).toBe("observed");
expect(detail?.content.some(block => block.type === "image" && block.mimeType === "image/jpeg")).toBe(true);
expect(detail?.details).not.toHaveProperty("observation_id");
@@ -192,18 +203,17 @@ test("a successful rectangle Read is readonly detail and consumes its overview w
expect(f.inputs()).toHaveLength(0);
});
-// LPT-941: a detail Read spends the window's current observation, also when the region refuses; point input then
-// captures the window again.
-test.each(["region:80,60,40,20", "region:480,300,100,100"])("a point click after a detail Read captures the window again, while a ref of the spent Read still refuses (%s)", async region => {
- const f = await fixture(), view = await f.read(["image"]);
- await f.resources.read(view.observation, cwd, [region]).catch(() => undefined);
+// LPT-941: an element crop spends the window's current observation; point input then captures the window again.
+test("a point click after an element crop captures the window again, while a ref of the spent Read still refuses", async () => {
+ const f = await fixture(), view = await f.read();
+ await f.resources.read(view.observation, cwd, [`element:crop,${view.hint}`]);
const calls = f.native().length;
const clicked = await f.resources.input(f.window, cwd, clickPlan({ window: f.window.source, x: 100, y: 70 }));
expect(clicked.details.status, JSON.stringify(clicked.content)).toBe("delivered");
// The click captured the window first; it did not act on the spent observation.
expect(f.native().slice(calls).map(([name]) => name).filter(name => name !== "list_windows").slice(0, 2)).toEqual(["get_window_state", "click"]);
const full = await f.read();
- await f.resources.read(full.observation, cwd, [region]).catch(() => undefined);
+ await f.resources.read(full.observation, cwd, [`element:crop,${full.hint}`]);
const inputs = f.inputs().length;
await expect(f.resources.input(f.window, cwd, clickPlan({ window: f.window.source, target: full.hint! }))).rejects.toThrow(/is not a ref of this window's latest Read/);
expect(f.inputs()).toHaveLength(inputs);
diff --git a/test/fixtures/detail-driver.mjs b/test/fixtures/detail-driver.mjs
index 2eccd71..7511a81 100644
--- a/test/fixtures/detail-driver.mjs
+++ b/test/fixtures/detail-driver.mjs
@@ -23,7 +23,7 @@ server.setRequestHandler(CallToolRequestSchema, async ({ params: { name, argumen
else if (name === "list_windows") state = { windows };
else if (name === "get_window_state") {
state = { ...args, window_bounds: windows[0].bounds, screenshot_width: 500, screenshot_height: 400,
- elements: [{ element_token: "close", role: "button", label: "Close", actions: ["click"] }] }; bytes = overview;
+ elements: [{ element_token: "close", role: "button", label: "Close", actions: ["click"], frame: { x: 210, y: 180, w: 200, h: 160 }, screenshot_frame: { x: 100, y: 80, w: 100, h: 80 } }] }; bytes = overview;
} else if (name === "zoom") { state = { width: 200, height: 160, x_offset: 200, y_offset: 160 }; bytes = jpeg; mimeType = "image/jpeg"; }
else if (name === "get_desktop_state") {
const native = args.max_image_dimension === 0;
diff --git a/test/live/app-scripting.live.ts b/test/live/app-scripting.live.ts
index d26487e..216790b 100644
--- a/test/live/app-scripting.live.ts
+++ b/test/live/app-scripting.live.ts
@@ -1,5 +1,6 @@
import { afterAll, expect, test } from "vitest";
import { mkdir, writeFile } from "node:fs/promises";
+import { requireComputerAddress } from "../../src/computer-resources.ts";
import { connectLive, text } from "./remote.ts";
// LPT-821: an app builds a scene through its own scripting while computer use observes and verifies it.
@@ -30,10 +31,12 @@ test("a Blender scene built by a script is checked through a window Read", async
let names: string[] = [];
for (let attempt = 0; attempt < 10 && !["PiCube", "PiSphere", "PiGround"].every(name => names.includes(name)); attempt++) {
await new Promise(resolve => setTimeout(resolve, 1500));
- // The outliner sits in the top-right corner; region OCR enlarges it, which small Remote Desktop text needs.
- const read = await live.read(window, ['capture:{"granularity":"image","maxLongEdge":3840}']);
+ // The outliner sits in the top-right corner; a viewport shows it at the window's own resolution and its OCR
+ // enlarges it, which small Remote Desktop text needs.
+ const read = await live.read(window, ["image"]);
const { width, height } = (read.details.agent_image as { output: { width: number; height: number } }).output;
- const corner = await live.read(`computer:observation/${read.details.observation_id}`, [`region:${Math.round(width * 0.7)},0,${Math.round(width * 0.3)},${Math.round(height * 0.4)}`, "ocr"]);
+ await live.resources.view(requireComputerAddress(`${window}/outliner`), live.cwd, { x: Math.round(width * 0.7), y: 0, width: Math.round(width * 0.3), height: Math.round(height * 0.4) });
+ const corner = await live.read(`${window}/outliner`, ["ocr"]);
names = (corner.details.text_lines ?? []).flatMap(line => line.words.map(word => word.text));
}
console.log("outliner words", JSON.stringify(names));
diff --git a/test/live/quick-fixes.live.ts b/test/live/quick-fixes.live.ts
index e3962f8..dd8b462 100644
--- a/test/live/quick-fixes.live.ts
+++ b/test/live/quick-fixes.live.ts
@@ -38,13 +38,14 @@ test("computer_type replace reaches the slider through accessibility", async ()
}
});
-// LPT-941: a refused region Read spends the window's observation; a point click then captures the window again.
-test("a point click after a refused region Read captures the window again", async () => {
+// LPT-941: an element crop spends the window's observation; a point click then captures the window again.
+test("a point click after an element crop captures the window again", async () => {
const title = `pi-spent-${Date.now()}`;
const window = await openWindow(live, live.machine, WINDOWS_POWERSHELL_PATH, nativeForm(title), title);
try {
- const view = await live.read(window, ["image"]);
- await live.read(`computer:observation/${view.details.observation_id}`, ["region:5000,5000,10,10"]).catch(() => undefined);
+ const view = await live.read(window, ["full"]);
+ const hint = view.details.ui!.elements.find(element => /^Count/.test(element.name ?? ""))!.hint;
+ await live.read(`computer:observation/${view.details.observation_id}`, [`element:crop,${hint}`]);
const clicked = await live.resources.input(requireComputerAddress(window), live.cwd, clickPlan({ window, x: 200, y: 250 }));
// The click is real input; right after an app starts, Windows can still hand the foreground back to another window,
// and the driver then answers unknown. What this checks is that the spent observation no longer refuses the click.
diff --git a/test/machine-host.test.ts b/test/machine-host.test.ts
index 80c3bcf..07bc20e 100644
--- a/test/machine-host.test.ts
+++ b/test/machine-host.test.ts
@@ -18,7 +18,8 @@ test("a host whose folder was never used removes nothing on close", async () =>
test("ocr combines only with pixel views", () => {
expect(parseComputerViews(["ocr"])).toMatchObject({ kind: "overview", ocr: true });
- expect(parseComputerViews(["region:1,2,30,40", "ocr"])).toMatchObject({ kind: "region", region: { x: 1, y: 2, width: 30, height: 40 }, ocr: true });
+ // The text of one part of a window comes from a viewport Read with ocr.
+ expect(() => parseComputerViews(["region:1,2,30,40", "ocr"])).toThrow(/viewport/);
expect(parseComputerViews(['capture:{"granularity":"image","maxLongEdge":3840}', "ocr"])).toMatchObject({ kind: "overview", ocr: true });
expect(() => parseComputerViews(["state", "ocr"])).toThrow();
expect(() => parseComputerViews(['capture:{"granularity":"ui"}', "ocr"])).toThrow(/pixels/);
diff --git a/test/native-detail.integration.test.ts b/test/native-detail.integration.test.ts
index a0cd71c..1175dc0 100644
--- a/test/native-detail.integration.test.ts
+++ b/test/native-detail.integration.test.ts
@@ -6,7 +6,7 @@ import { expect, test } from "vitest";
import { DEFAULT_MAX_LONG_EDGE } from "../src/cua-image-settings.ts";
import { SCRIPT_HELPERS } from "./fixtures/standalone-computer.ts";
-test("real Pi forwards only final display crop and window JPEG, and disarms overview input", async () => {
+test("real Pi forwards only final display crop and window element JPEG, and disarms overview input", async () => {
await mkdir(".tmp/native-zoom", { recursive: true });
const cwd = await mkdtemp(path.resolve(".tmp/native-zoom/pi-contract-")); const log = path.join(cwd, "calls.jsonl");
await mkdir(path.join(cwd, ".pi"));
@@ -17,10 +17,10 @@ test("real Pi forwards only final display crop and window JPEG, and disarms over
const opened = await openApp("Notepad");
if (opened.outcome !== "observed" || !opened.observation) throw Error("Open returned no first observation");
const region = "region:100,80,100,80";
-const detail = await tools.computer_read({path:opened.observation,views:[region]});
-text(body(detail)); image(detail);
const ref = hints(opened)[0];
if (!ref) throw Error("Open listed no refs: " + body(opened));
+const detail = await tools.computer_read({path:opened.observation,views:["element:crop," + ref]});
+text(body(detail)); image(detail);
const stale = await tools.computer_click({window:opened.window,target:ref});
if (stale.outcome !== "refused" || stale.effect !== "not-applied") throw Error("Old overview still armed");
const overview = await tools.computer_read({path:"computer:machine/local/display/primary"});
diff --git a/test/viewports.integration.test.ts b/test/viewports.integration.test.ts
new file mode 100644
index 0000000..6c8e638
--- /dev/null
+++ b/test/viewports.integration.test.ts
@@ -0,0 +1,49 @@
+import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
+import path from "node:path";
+import { assistantMessage, getToolExecution, getToolResultText, PiIntegrationTest, testArtifactsDir, text, toolCall } from "pi-coding-agent-test";
+import { expect, test } from "vitest";
+import { SCRIPT_HELPERS } from "./fixtures/standalone-computer.ts";
+
+// LPT-931: a viewport through real Pi: computer_view makes it, Read shows it, an input tool takes points of its image.
+test("real Pi makes a viewport, reads it and clicks at a point of its image", async () => {
+ await mkdir(".tmp/viewports", { recursive: true });
+ const cwd = await mkdtemp(path.resolve(".tmp/viewports/pi-contract-"));
+ const log = path.join(cwd, "calls.jsonl");
+ await mkdir(path.join(cwd, ".pi"));
+ await writeFile(path.join(cwd, ".pi/computer-use.json"), JSON.stringify({
+ command: process.execPath, args: [path.resolve("test/fixtures/observation-granularity-driver.mjs"), log],
+ }));
+ const code = `${SCRIPT_HELPERS}
+const win = address(await windows(), "window");
+await tools.computer_read({path:win,views:["image"]});
+const pano = win + "/pano";
+const made = await tools.computer_view({viewport:pano,x:100,y:50,width:200,height:120});
+if (made.outcome !== "observed" || !made.viewport || !made.viewport.endsWith("/pano")) throw Error("View failed: " + JSON.stringify(made));
+const read = await tools.computer_read({path:made.viewport});
+if (typeof read === "string" || !/Its image is 200x120/.test(read.note ?? JSON.stringify(read))) throw Error("Read of the viewport changed: " + JSON.stringify(read).slice(0, 400));
+const clicked = await tools.computer_click({window:made.viewport,x:10,y:20});
+if (clicked.outcome !== "delivered" || clicked.viewport !== made.viewport) throw Error("Click failed: " + JSON.stringify(clicked));
+const outside = await tools.computer_click({window:made.viewport,x:250,y:20});
+if (outside.outcome !== "refused" || outside.effect !== "not-applied") throw Error("A point outside the viewport was sent");
+const removed = await tools.computer_delete({path:made.viewport});
+if (removed.outcome !== "closed") throw Error("Delete failed: " + JSON.stringify(removed));
+text("Viewport contract passed: " + made.viewport);
+`;
+ try {
+ const result = await new PiIntegrationTest({
+ testName: "viewports", artifactsDir: testArtifactsDir(import.meta.filename), cwd,
+ extensions: [path.resolve("test/fixtures/display-extension.ts")], isolateUserResources: true,
+ tools: ["codemode", "computer_delete"],
+ conversation: [
+ assistantMessage([toolCall({ id: "view", name: "codemode", arguments: { code } })], { stopReason: "toolUse" }),
+ assistantMessage([toolCall({ id: "close", name: "computer_delete", arguments: { path: "computer:session" } })], { stopReason: "toolUse" }),
+ assistantMessage([text("Done")]),
+ ],
+ }).run("Work in a viewport of the window.");
+ expect(getToolExecution(result, "view").isError, getToolResultText(result, "view")).toBe(false);
+ expect(getToolResultText(result, "view")).toMatch(/Viewport contract passed: computer:local\/w1[a-z]{2}\/pano/);
+ const calls = (await readFile(log, "utf8")).trim().split("\n").map(line => JSON.parse(line));
+ // The click went to the viewport's point in the window's full-resolution pixels; the refused one sent nothing.
+ expect(calls.filter(c => c.name === "click").map(c => [c.arguments.x, c.arguments.y])).toEqual([[110, 70]]);
+ } finally { await rm(cwd, { recursive: true, force: true }); }
+});