Skip to content

Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.

9.5 KB
Raw Blame

Temporal capture

See the resource guide for the current addresses, views and commands.

A sequence Read captures one authorized Windows window, or a machine's primary or listed display, on the machine that issued its address. Browser windows use the same native path. It does not encode video, raise a window, walk its accessibility tree or grant input authority.

Capture, then select

Capture returns metadata only and sends no image. Capture and selection are separate decisions: read the timestamps, count and stop reason before choosing frames from the returned sequence.

Capture a display (computer:machine/<name>/display/primary or one listed by its displays) instead of a window for a read-only display sequence in YOLO. Frames are selected through the returned computer:sequence/... address, which runs on the machine that captured them. Safe denies display and unowned window captures. Stored frame selection checks current access and the window's lifetime again. Revocation withholds metadata and pixels. A closed or recreated window cannot return its retained frames.

Bounds and settings

Field Bounds / default
duration_ms Required, 1..60000
fps 1..30, default 4; a goal, not guaranteed throughput
max_frames 1..300, default 40; one selection returns at most 40
timeout_ms At least duration, at most 90000; default 15000
maxLongEdge 64..4096; defaults to current native image setting
scale 0.05..1; defaults to current native image setting
region Optional integer {x,y,width,height} in the first uncropped returned-image pixels

Image settings are frozen before sampling. CUA owns native resizing. Crop extracts only the current CUA frame with the existing Sharp dependency. The initial region must fit; a smaller later raster clips it, and growth does not expand it. Each cropped frame reports requested/applied bounds and native origin when known. No prior overview or detail image is used as a hidden fallback.

The store holds the newest three sequences within 48 MiB of decoded image bytes. One frame is limited to 12 MiB. Count, byte or time limits are explicit. Close/reload drops resources. Base64 and metadata add memory beyond the decoded-byte limit.

Timing and stop reasons

Frames are ordered and indexed from zero. slot is the scheduled sampling slot; timestamp_ms is the measured native-call completion relative to the start. It is not a compositor or paint timestamp. capture_latency_ms measures the native call; sample_latency_ms also includes access/identity checks and crop processing.

actual_fps is (frame count - 1) / sample_span_ms, in seconds. A single frame has no measured FPS. interval_ms reports min/max/mean intervals, making jitter visible. elapsed_ms is the whole capture call, not a fabricated requested duration. The last admitted read may finish after the requested duration; timeout/cancellation bounds further work.

Slow captures and delayed wakes skip expired slots instead of catching up with bursts. skipped_slots lists missed scheduled slots. pending_slots counts unsampled slots left by an early stop; a watch stopped with watch_stop has none, because its duration is only a safety limit. No backend dropped-frame count is invented: CUA provides snapshots, not a continuous stream.

  • completed: the scheduled duration was sampled, including any missed slots.
  • frame_limit: the requested frame cap was reached. The resource is available, but the remaining slots were not sampled.
  • byte_limit, cancelled, timeout, target_lost, access_revoked, capture_failed: partial or failed capture, never full success.

An interrupted resource may contain a known completed prefix. Cancellation withholds the current unfinished frame, stops the sampling loop and never retries a native call. The selected MCP driver owns in-flight cancellation and its 15-second per-call timeout. A sequence creates no detached capture loop; a watch is the one capture that runs in the background.

Every frame reports source geometry and whether it changed from the previous frame. Movement/resize do not change the window lifetime; closed/reused handles stop capture. Native geometry and paint freshness can remain unknown. Capture retires that source's previous overview authority. Observe again before input or detail, even after selecting the last frame.

Selection and further processing

Choose all, first, last, endpoints, changed, {indexes:[0,2]} or inclusive {from:1,to:3}. Indexes refer to captured frames, not sampling slots. Selection keeps temporal order. changed compares exact image bytes and geometry; it does not claim semantic relevance. Codemode receives an array of frame metadata and image blocks, suitable for explicit forwarding or further processing. Do not store whole image arrays in Codemode storage.

One image of many frames

Add as to a JSON selection to get one image instead of one per frame:

View Returns
frames:{"as":"sheet"} A contact sheet of all frames (at most 40): tiles in time order, left to right, each under a #index · time label. The text lists each tile's row, column, frame and time.
frames:{"select":"changed","as":"sheet","columns":4} The same for another selection (select, indexes or from/to) with a fixed column count (1..10).
frames:{"as":"max"} or "min" The per-pixel maximum or minimum of the selected frames: bright (or dark) detail from any frame appears at once, for example text revealed by a moving light. Frames must have one size.

region:{x,y,width,height} crops every frame first, in the sequence's frame pixels (the overview size it was captured at); it is clipped to the frames and refused when it misses them. A sheet's long edge is 1600 px by default; tiles shrink together to fit, keeping their aspect ratio. maxLongEdge (64..4096) changes that limit, for example 16 tiles of a 640 px region in 4 columns at full size need 2560. A composite keeps the frames' size up to maxLongEdge. The image is read-only and grants no input.

Watch while acting

computer_record watch starts capturing a window, or a viewport of it, and returns a sequence address once the first frame is in. Its sequence metadata says running: true with the frames so far, the elapsed time and the rate reached; a stop reason and slot counts come with watch_stop. Capture runs outside the session's queue through a second driver process on the same machine: a driver answers one call at a time, so a capture sent during a 2 s drag would otherwise wait for the drag to end. Starting that process takes a few seconds on the first watch; it stays until the session closes. A sandbox or isolated session has one driver connection, so its watch captures between the session's calls only.

Field Bounds / default
fps 1..30, default 10; a goal, not guaranteed throughput
timeout_ms 1..300000, default 60000; the watch stops by itself then
maxLongEdge 64..4096, default 800; a viewport is captured so its own rectangle gets at most this many pixels on its long edge, never more than its full resolution

A viewport watch asks the driver for the viewport's rectangle only: the driver copies and encodes just that part instead of the whole window. When a frame takes longer than the interval, a second capture starts on time while the first finishes, at most two at once; frames keep the order their captures started in. The rate reached is in the sequence metadata, while the watch runs and in the watch_stop result.

A watch only looks: it does not take the window's observation, so refs and points of the latest Read stay valid for input. A frame with the same bytes as the one before shares its image and costs no memory; 3000 frames or 48 MiB of distinct frames stop the watch by itself. It checks process and window id on every frame, and stops when the window closes or access ends.

Each input or window action on the watched window notes its span, from the call to its result, in every running watch of that window; the receipt lists it with the frames captured during it. Read the sequence address with a frames: view while the watch runs, for example frames:last.

watch_stop ends the watch and returns:

  • a contact sheet of the frames that differ from the one before and the last frame; above 40, the first, the last and those around each action stay and the rest are spread evenly. A frame captured during an action has its action's name on an orange label;
  • the actions with their spans and frames, and the frames that changed most against the frame before, as a share of pixels and a box in frame pixels;
  • the files: one PNG per frame that differs from the one before, and watch.json with every frame's time, file (a repeat names the file of the frame it repeats), size, change and actions. They go to to in the workspace, or to a temporary folder removed when the session closes. video: true adds watch.webp, an animated WebP of up to 200 distinct frames at 640 px, each shown until the next one.

The sequence keeps its frames for frames: Reads after the stop, like any sequence.

The existing debug viewer keeps its three ordinary boundary cards. A temporal dropdown also lists up to 40 sequence image blocks in the latest application-boundary request and lets the human inspect their exact bytes. Selection alone is not forwarding. Byte matching can be ambiguous when screenshots are identical; the viewer lists every retained possible source instead of inventing a timestamp. Asset eviction is visible. Final wire delivery remains unconfirmed.