Skip to content

Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.

Promote develop for 0.1.1 (#160)

alexshpunt-clanker[bot]76bc595b75dc
diff --git a/README.md b/README.md
index 21442fd..83aa638 100644
--- a/README.md
+++ b/README.md
@@ -80,7 +80,11 @@ Input goes to a window by its address and targets an element by a ref from the l
- run a timeline of timed key, button, mouse and pen events, with raw relative mouse motion for pointer-locked apps and pen pressure and tilt;
- invoke, select or set values through UI Automation, and open menu items by their labels.
-What the agent sees as an element it presses through UI Automation in the background, so your pointer and focus stay put. What it sees only as pixels it clicks with the real mouse, and keys are real key presses: that input takes the front window, and it pauses whenever you touch the keyboard or mouse. Nothing is posted to windows as messages that an app could silently drop. To keep foreground UI work off the human desktop, use a sandbox or an isolated session.
+What the agent sees as an element it presses through UI Automation in the background, so your pointer and focus stay put. What it sees only as pixels it clicks with the real mouse, and keys are real key presses: that input brings its window to the front and runs at once, even while you use the computer. Nothing is posted to windows as messages that an app could silently drop. To keep foreground UI work off the human desktop, use a sandbox or an isolated session.
+
+Press Escape three times within a second to stop every agent on your desktop. Their input stops at once, and they refuse to act until you run `/computer-mode safe` or `/computer-mode yolo` in their sessions. An agent's own Escape doesn't count. Sandboxes and isolated sessions keep running.
+
+Several Pi sessions can use the same desktop. Their actions take turns: a string one agent types or a drag it makes finishes before another agent's next action starts.
### Results and checks
@@ -112,6 +116,8 @@ Each session has its own folder on the controlled machine for files an app opens
### Computer visuals
+The agent's cursor carries the Pi session's name, or the project folder's name until the session has one, so you can tell agents apart. A card beside each action names it and shows the text being typed, except into password fields.
+
`/computer-visuals` chooses `off`, `subtle` or `full`, with optional image previews when the native driver supports them. A preview shows the final operation image; it does not prove delivery to a model. `/computer-images` changes screenshot size, hints and image guides for future captures.
## Platforms
@@ -169,7 +175,7 @@ The Windows binaries aren't code-signed. Signing certificates are hard to get fo
| Command | Effect |
| -------------------------------- | -------------------------------------------------------- |
-| `/computer-mode safe\|yolo` | Switch between Safe and YOLO for this session |
+| `/computer-mode safe\|yolo` | Switch between Safe and YOLO; also turns computer use on again after Escape stopped it |
| `/computer-visuals off\|subtle\|full` | Choose native activity cues and image previews |
| `/computer-images` | Show or change capture size, hints and image guides |
| `/computer-setup PART [MACHINE]` | Install an optional part; without a part it lists them |
diff --git a/docs/bundled-runtime.md b/docs/bundled-runtime.md
index c1f0294..ab3d586 100644
--- a/docs/bundled-runtime.md
+++ b/docs/bundled-runtime.md
@@ -1,6 +1,6 @@
# Bundled CUA runtime
-The npm package carries one native runtime: Cua Driver 0.38.0 for Windows x64, built from source `4b547b54` of the [alexshpunt/cua](https://github.com/alexshpunt/cua) fork. [FORK.md](https://github.com/alexshpunt/cua/blob/pi-computer-use/FORK.md) lists the fork's changes. The `windows-desktop` profile needs no separate driver configuration:
+The npm package carries one native runtime: Cua Driver 0.38.0 for Windows x64, built from source `3e69266e` of the [alexshpunt/cua](https://github.com/alexshpunt/cua) fork. [FORK.md](https://github.com/alexshpunt/cua/blob/pi-computer-use/FORK.md) lists the fork's changes. The `windows-desktop` profile needs no separate driver configuration:
```sh
pi install npm:pi-agent-computer-use
@@ -10,7 +10,7 @@ An extension update selects its matching runtime. Computer use prepares that run
## Where the files come from
-The files are published as the fork release [`pi-runtime-0.38.0-4b547b54`](https://github.com/alexshpunt/cua/releases/tag/pi-runtime-0.38.0-4b547b54): `cua-driver.exe`, `cua-driver-uia.exe`, `cua-cursor-theme.exe`, `LICENSE.md` and `THIRD_PARTY_NOTICES.md`. They were cross-built with Rust 1.97.1, cargo-xwin 0.23.1 and LLVM 18, with a static CRT and canonical Windows manifest resources. They are not copies of a GitHub Actions artifact.
+The files are published as the fork release [`pi-runtime-0.38.0-3e69266e`](https://github.com/alexshpunt/cua/releases/tag/pi-runtime-0.38.0-3e69266e): `cua-driver.exe`, `cua-driver-uia.exe`, `cua-cursor-theme.exe`, `LICENSE.md` and `THIRD_PARTY_NOTICES.md`. They were cross-built with Rust 1.97.1, cargo-xwin 0.23.1 and LLVM 18, with a static CRT and canonical Windows manifest resources. They are not copies of a GitHub Actions artifact.
`src/runtime-pins.json` pins the source, the version, the release URL and the SHA-256 of every file. Before packing, `npm pack` runs `scripts/fetch-runtime.mjs`, which downloads the files into `native-runtime/win32-x64/runtime/` and checks each hash. The directory is not committed to Git.
diff --git a/docs/computer-resources.md b/docs/computer-resources.md
index b6fe954..a584aab 100644
--- a/docs/computer-resources.md
+++ b/docs/computer-resources.md
@@ -24,7 +24,8 @@
- Input spends the window's native refs; its change list brings the new ones.
- On native windows, a ref or locator goes through UI Automation in the background, without the human's pointer or focus. When the app does not react to it, use a point instead.
- Drags, `computer_hover` and `computer_timeline` are real input: the window comes to the front. `computer_select` and menus run through UI Automation.
-- While the human uses the keyboard or mouse, real input refuses for the time the refusal gives. Wait; never race the human.
+- Real input runs at once, even while the human uses the computer. Another agent's action on the same desktop finishes first.
+- `human_stop`: the human pressed Escape three times. Send no more input, tell the user what you were doing, and wait until they turn computer use on again.
Native locators run over the UI Automation tree:
diff --git a/docs/native-observability.md b/docs/native-observability.md
index df450d9..b8218f2 100644
--- a/docs/native-observability.md
+++ b/docs/native-observability.md
@@ -15,15 +15,21 @@ Run `/computer-visuals` to show the current settings, or choose a preset:
A JSON patch controls individual categories and rendering preferences:
```text
-/computer-visuals {"observations":true,"previews":false,"pointer":true,"scroll_drag":true,"keyboard":true,"opacity":0.8,"quality":"balanced","reduced_motion":false}
+/computer-visuals {"observations":true,"previews":false,"pointer":true,"scroll_drag":true,"keyboard":true,"typed_text":true,"opacity":0.8,"quality":"balanced","reduced_motion":false}
```
+`typed_text` shows the text an action types or sets on its card, while `keyboard` is on. Text going into a field that UI Automation reports as a password is never sent to the renderer. Control characters show as spaces, and only the last 200 characters are sent.
+
`opacity` is 0..1. `quality` is `performance`, `balanced` or `high`. Preferences are saved on the Pi session branch; navigating to another branch restores its preferences. These commands belong to the human. They do not change Safe/YOLO, capture sizing, input delivery or permission checks.
`/computer-images` controls future captures in this session: `cap 64..4096`, `scale 0.05..1`, `hints on|off`, `guides off|grid|action|all`, or `reset`. Calling it without arguments shows the settings. An individual Read can still override its capture options. Existing images keep their bytes.
## What a cue means
+The agent cursor carries a label: the Pi session's name, or the project folder's name while the session has none. The adapter passes it to the driver process as `CUA_SESSION_LABEL` when the process starts; renaming the session takes effect with the next driver process.
+
+A card stays on screen until the next card replaces it, so consecutive actions do not blink. A card that nothing replaces stays about two seconds and fades out over 500 ms. A clear, a settings change or the end of the session hides it at once.
+
An overview identifies a captured window or display. A detail cue marks a requested physical region only when the adapter has a proven transform. Inspection and browser cues associate the operation with its native window; a Playwright CSS box does not establish a desktop rectangle. Element inspection uses native element bounds when available.
Action cues can report `delivered`, `refused` or `unknown`. `delivered` means input was sent, not that the application changed. Only a completed condition check can report `verified`. Quiet verification samples and background watch frames do not each create another cue.
diff --git a/scripts/sandbox/guest-start.ps1 b/scripts/sandbox/guest-start.ps1
index 7c18da3..192a0e3 100644
--- a/scripts/sandbox/guest-start.ps1
+++ b/scripts/sandbox/guest-start.ps1
@@ -5,8 +5,6 @@ $env:CUA_DRIVER_RS_MCP_HTTP_TOKEN = (Get-Content 'C:\pi\config\token.txt' -Raw).
$env:CUA_DRIVER_RS_TELEMETRY_ENABLED = 'false'
$env:CUA_DRIVER_RS_UPDATE_CHECK = 'false'
$env:DO_NOT_TRACK = '1'
-# Foreground input pauses until 10 s pass without the human's keyboard or mouse input (LPT-835).
-$env:CUA_HUMAN_IDLE_MS = '10000'
$env:CUA_DRIVER_ALLOW_DEBUGGING_LAUNCH = '1'
Start-Process -FilePath 'C:\pi\runtime\cua-driver.exe' -ArgumentList 'serve' -WindowStyle Hidden `
-RedirectStandardOutput 'C:\pi\out\driver.out' -RedirectStandardError 'C:\pi\out\driver.err'
diff --git a/src/computer-tools.ts b/src/computer-tools.ts
index 7c2e536..1f3643b 100644
--- a/src/computer-tools.ts
+++ b/src/computer-tools.ts
@@ -74,8 +74,8 @@ export function openCommand(address: ComputerAddress, params: Static<typeof open
const windowParameters = Type.Object({
window: Type.String({ description: "Window address from {search}, {read} or a name, such as computer:<machine>/w3qk or computer:game." }),
- action: StringEnum(["move", "resize", "maximize", "minimize", "focus", "restore", "name"] as const,
- { description: "focus activates a visible window; restore brings a minimized window back and activates it; name gives the window a short address computer:<as>." }),
+ action: StringEnum(["move", "resize", "maximize", "minimize", "restore", "name"] as const,
+ { description: "restore brings a minimized window back; name gives the window a short address computer:<as>." }),
as: Type.Optional(Type.String({ description: "name: 1-32 lowercase letters, digits, - or _, starting with a letter, such as game. It resolves until the window closes." })),
x: Type.Optional(Type.Integer({ description: "move: new left edge in Windows desktop pixels." })),
y: Type.Optional(Type.Integer({ description: "move: new top edge in Windows desktop pixels." })),
@@ -531,7 +531,7 @@ export function registerComputerTools(pi: ExtensionAPI, resources: ComputerResou
});
pi.registerTool({
...common, name: "computer_window", label: "Computer window",
- description: "Use computer_window to move, resize, maximize, minimize, focus or restore an app window by its address without keys or clicks, or to give it a short name.",
+ description: "Use computer_window to move, resize, maximize, minimize or restore an app window by its address without keys or clicks, or to give it a short name.",
parameters: fillToolNames(windowParameters, names),
annotations: { readOnlyHint: false, destructiveHint: false, openWorldHint: false },
...rendered("computer_window"),
diff --git a/src/driver.ts b/src/driver.ts
index 2e4bb36..78c36d0 100644
--- a/src/driver.ts
+++ b/src/driver.ts
@@ -14,10 +14,9 @@ import { SingleConnectionHttpTransport } from "./http-transport.ts";
import { SSH_OPTIONS, localRunner, remoteCommand, sshRunner } from "./runner.ts";
import { verifyBundledRuntime } from "./native-runtime.ts";
import type { NativePlatform } from "./observation-source.ts";
-import { NativeVisuals, VisualSettings, type WindowObserver } from "./native-visuals.ts";
+import { NativeVisuals, VisualSettings, nativeVisualOptions, type WindowObserver } from "./native-visuals.ts";
+import { reportsHumanStop, type HumanStop } from "./human-stop.ts";
-/** Foreground input waits until the human has not used the keyboard or mouse for this long (owner choice, LPT-835). */
-const HUMAN_IDLE_MS = 10_000;
/** The agent cursor jumps to each action point instead of gliding there. */
const CURSOR_GLIDE_MS = 1;
/** Exact Windows CI artifact qualified for held paths; selection is trusted config, never a tool argument. */
@@ -171,16 +170,32 @@ export class McpDriver implements Driver {
get windowCrop(): boolean { return this.config.runtime === "windows-desktop"; }
/** Overview metadata follows the native implementation, not an agent-chosen filter. */
get overviewKernel(): "bilinear" | "lanczos3" { return ["windows-fast-bilinear", "windows-direct-process", "windows-persistent-wgc", "windows-native-pointer", "windows-coordinate-scroll", "windows-window-restore", "windows-window-management", "windows-desktop"].includes(this.config.runtime ?? "") ? "bilinear" : "lanczos3"; }
+ /**
+ * The human desktop's stop, shared by the session's drivers of human desktops. A sandbox or an isolated session,
+ * reached through an endpoint, is not a human desktop and has none.
+ */
+ private readonly humanStop?: HumanStop;
+ /** This process reported the stop and refuses all input until it exits; the next call after a resume starts a new one. */
+ private stoppedNatively = false;
+ private stopSubscriptions: (() => void)[] = [];
+
constructor(private readonly config: ComputerConfig, private readonly version: "0.33.2" = "0.33.2",
- private readonly visualSettings = new VisualSettings()) {
+ private readonly visualSettings = new VisualSettings(), humanStop?: HumanStop, private readonly sessionLabel?: () => string | undefined) {
+ if (!config.endpoint) this.humanStop = humanStop;
this.presentation = new NativeVisuals(async (name, args) => {
// Presentation never starts or reconnects a backend, including after close or while startup is pending.
- if (!this.client || this.connecting) return undefined;
+ if (!this.client || this.connecting || this.stoppedNatively) return undefined;
if (!this.nativeVisualsAvailable) return false;
const result = CallToolResultSchema.parse(await this.client.callTool({ name, arguments: args }, undefined, { timeout: 2000 }));
return !result.isError;
}, visualSettings);
this.visuals = this.presentation.flash;
+ if (this.humanStop) this.stopSubscriptions.push(this.humanStop.onResume(() => {
+ if (!this.stoppedNatively) return;
+ this.stoppedNatively = false;
+ // Window identities stay: only the stopped driver process is replaced.
+ void this.closeClient().catch(() => undefined);
+ }));
if (this.windowLifetime) this.windowIdentity = new WindowIdentity(config.endpoint ? remoteIdentityPeer(config.endpoint.identityPort, "127.0.0.1", config.endpoint.identityAuth ?? config.endpoint.relay)
: hostIdentityPeer(config.ssh ? sshRunner(config.ssh) : localRunner(config.interop ?? process.env.WSL_INTEROP)));
}
@@ -220,6 +235,10 @@ export class McpDriver implements Driver {
: await this.stdioTransport();
this.transport = transport;
const client = new Client({ name: "pi-agent-computer-use-poc", version: "0.0.0" });
+ // The driver says at once when the human stops input, without waiting for a call.
+ client.fallbackNotificationHandler = async notification => {
+ if (notification.method === "notifications/cua/human_stop") this.reportStop();
+ };
this.client = client;
try {
await client.connect(transport, { signal, timeout: 15_000 });
@@ -241,7 +260,7 @@ export class McpDriver implements Driver {
}
if (this.nativeVisualsAvailable) {
try {
- const configured = CallToolResultSchema.parse(await client.callTool({ name: "set_agent_visuals", arguments: { ...this.visualSettings.current() } }, undefined, { signal, timeout: 2_000 }));
+ const configured = CallToolResultSchema.parse(await client.callTool({ name: "set_agent_visuals", arguments: nativeVisualOptions(this.visualSettings.current()) }, undefined, { signal, timeout: 2_000 }));
this.nativeVisualsAvailable = !configured.isError;
} catch {
// Presentation is optional. A renderer/configuration failure must not prevent native input.
@@ -280,16 +299,26 @@ export class McpDriver implements Driver {
}
/**
- * Variables for the native child, passed to Windows through WSLENV: no telemetry or update checks, a 10 s
- * takeover pause (foreground input refuses while the human used the keyboard or mouse that recently), and launches
- * with remote-debugging switches, which this adapter allows only through YOLO's open by path and arguments.
+ * Variables for the native child, passed to Windows through WSLENV: no telemetry or update checks; three physical
+ * Escape presses stop the agent; agents on one desktop take turns per action; the agent cursor carries the Pi
+ * session's label; and launches with remote-debugging switches, which this adapter allows only through YOLO's open
+ * by path and arguments.
*/
private driverVariables(): Record<string, string> {
+ const label = this.sessionLabel?.()?.replace(/[^\p{L}\p{N} ._-]/gu, "").trim().slice(0, 28);
return { CUA_DRIVER_RS_TELEMETRY_ENABLED: "false", CUA_DRIVER_RS_UPDATE_CHECK: "false",
- ...(this.envelopes ? { CUA_DRIVER_MCP_ENVELOPES: "1" } : {}), DO_NOT_TRACK: "1", CUA_HUMAN_IDLE_MS: String(HUMAN_IDLE_MS),
+ ...(this.envelopes ? { CUA_DRIVER_MCP_ENVELOPES: "1" } : {}), DO_NOT_TRACK: "1",
+ CUA_ESCAPE_STOP: "1", CUA_SHARED_INPUT_LOCK: "1", ...(label ? { CUA_SESSION_LABEL: label } : {}),
CUA_DRIVER_ALLOW_DEBUGGING_LAUNCH: "1" };
}
+ /** The driver reported the human's stop; it refuses everything from now on, so stop this session's desktop work. */
+ private reportStop(): void {
+ if (!this.humanStop) return;
+ this.stoppedNatively = true;
+ this.humanStop.stop();
+ }
+
/** The owned native child; Windows-host control runs through WSL interop, on another machine over SSH. */
private async stdioTransport(): Promise<StdioClientTransport> {
if (this.config.ssh) {
@@ -363,6 +392,8 @@ export class McpDriver implements Driver {
return { content: [], structuredContent: { effect: "refused", code: "held_path_unavailable" }, isError: true };
if (name === "hold_keys" && !this.heldKeys)
return { content: [], structuredContent: { effect: "refused", code: "key_hold_unavailable" }, isError: true };
+ if (this.humanStop?.stopped)
+ return { content: [{ type: "text", text: this.humanStop.message() }], structuredContent: { status: "refused", effect: "refused", code: "human_stop", attempted: false }, isError: true };
const client = await this.connect(signal);
if (this.nativeTools && !this.nativeTools.has(name))
return { content: [], structuredContent: { status: "refused", code: "unsupported_native_capability", platform: this.platform, tool: name, attempted: false, fallback: false }, isError: true };
@@ -372,6 +403,7 @@ export class McpDriver implements Driver {
// every other native call answers well within 15 s.
const timeout = name === "input_timeline" ? 15_000 + timelineLength(args) : name === "launch_app" ? 90_000 : 15_000;
const result = CallToolResultSchema.parse(await client.callTool({ name, arguments: args }, undefined, { signal, timeout }));
+ if (reportsHumanStop(result as Parameters<typeof reportsHumanStop>[0])) this.reportStop();
const content: Content[] = [];
for (const block of result.content) {
if (block.type === "text") content.push({ type: "text", text: block.text });
@@ -383,11 +415,18 @@ export class McpDriver implements Driver {
/** EOF owns shutdown; the SDK escalates only for this child if it does not exit. */
async close(): Promise<void> {
this.presentation.close();
+ for (const unsubscribe of this.stopSubscriptions.splice(0)) unsubscribe();
await this.disconnect();
}
/** Startup cleanup is retryable; only final close disposes presentation and its subscriptions. */
private async disconnect(): Promise<void> {
+ try { await this.closeClient(); }
+ finally { await this.windowIdentity?.close(); }
+ }
+
+ /** End the driver process; the next call starts a new one. */
+ private async closeClient(): Promise<void> {
this.nativeVisualsAvailable = false;
const client = this.client;
const transport = this.transport;
@@ -396,9 +435,6 @@ export class McpDriver implements Driver {
this.nativeTools = undefined;
this.connectedVersion = undefined;
try { await client?.close(); }
- finally {
- try { await transport?.close(); }
- finally { await this.windowIdentity?.close(); }
- }
+ finally { await transport?.close(); }
}
}
diff --git a/src/human-stop.ts b/src/human-stop.ts
new file mode 100644
index 0000000..8bdd0b3
--- /dev/null
+++ b/src/human-stop.ts
@@ -0,0 +1,41 @@
+/**
+ * The human's stop of agent input on their desktop: three Escape presses on the physical keyboard, seen by the
+ * native driver. Until the human resumes it with /computer-mode, every driver of a human desktop in this Pi
+ * session refuses calls. Sandboxes and isolated sessions are not human desktops and keep running.
+ */
+export class HumanStop {
+ private reason?: string;
+ private readonly stopListeners = new Set<(reason: string) => void>();
+ private readonly resumeListeners = new Set<() => void>();
+
+ get stopped(): boolean { return this.reason !== undefined; }
+
+ /** The refusal every call gets while stopped. */
+ message(): string {
+ return "human_stop: the human pressed Escape three times and stopped computer use on their desktop. Nothing was sent. " +
+ "Do not retry or work around it; tell the human and wait until they turn it on again with /computer-mode.";
+ }
+
+ /** The first report wins; later ones while stopped change nothing. */
+ stop(reason = "Escape pressed three times"): void {
+ if (this.reason !== undefined) return;
+ this.reason = reason;
+ for (const listener of this.stopListeners) listener(reason);
+ }
+
+ resume(): void {
+ if (this.reason === undefined) return;
+ this.reason = undefined;
+ for (const listener of this.resumeListeners) listener();
+ }
+
+ onStop(listener: (reason: string) => void): () => void { this.stopListeners.add(listener); return () => { this.stopListeners.delete(listener); }; }
+ onResume(listener: () => void): () => void { this.resumeListeners.add(listener); return () => { this.resumeListeners.delete(listener); }; }
+}
+
+/** A native result that reports the stop: the driver refused, or stopped running input part way. */
+export function reportsHumanStop(result: { structuredContent?: Record<string, unknown>; content: { type: string; text?: string }[] }): boolean {
+ const state = result.structuredContent;
+ return state?.code === "human_stop" || (state?.refusal as { code?: unknown } | undefined)?.code === "human_stop" ||
+ result.content.some(block => block.type === "text" && typeof block.text === "string" && /\bhuman_stop:/.test(block.text));
+}
diff --git a/src/index.ts b/src/index.ts
index cdaaa7e..e7221c6 100644
--- a/src/index.ts
+++ b/src/index.ts
@@ -1,3 +1,4 @@
+import { basename } from "node:path";
import { registerImageSettingsCommand } from "./image-settings-command.ts";
import { registerVisualSettingsCommand } from "./visual-settings-command.ts";
import { VisualSettings } from "./native-visuals.ts";
@@ -17,6 +18,7 @@ import { WebBrowser } from "./web-browser.ts";
import { localRunner } from "./runner.ts";
import { MachineHost } from "./machine-host.ts";
import { machineFactory, sandboxFactory } from "./machine.ts";
+import { HumanStop } from "./human-stop.ts";
/**
* Register native computer resources with ordinary IDE tools when Pi Agent IDE accepts them, otherwise as
@@ -29,14 +31,19 @@ export default async function computerUse(pi: ExtensionAPI): Promise<void> {
registerImageSettingsCommand(pi, images);
registerVisualSettingsCommand(pi, visuals);
let canEnterSafe = () => true;
- const getMode = registerComputerMode(pi, async () => { visuals.clear(); }, () => canEnterSafe());
+ const humanStop = new HumanStop();
+ const getMode = registerComputerMode(pi, async () => { visuals.clear(); }, () => canEnterSafe(), humanStop);
+ let cwdLabel: string | undefined;
+ // The agent cursor names this session, or its project folder until the session has a name.
+ const sessionLabel = () => pi.getSessionName()?.trim() || cwdLabel;
registerSetupCommand(pi);
let session: ComputerSession | undefined;
let loading: Promise<ComputerSession> | undefined;
async function getSession(cwd: string): Promise<ComputerSession> {
if (session) return session;
if (!loading) loading = loadComputerConfig(cwd).then(async config => {
- const driver = new McpDriver(config, undefined, visuals);
+ cwdLabel = basename(cwd);
+ const driver = new McpDriver(config, undefined, visuals, humanStop, sessionLabel);
overviewKernel = driver.overviewKernel ?? "lanczos3";
session = new ComputerSession(driver, getMode, () => images.settings(), undefined,
driver.platform === "windows" ? (options, signal) => WebBrowser.launch({ ...options, interop: config.interop }, signal) : undefined,
@@ -47,7 +54,7 @@ export default async function computerUse(pi: ExtensionAPI): Promise<void> {
finally { loading = undefined; }
}
const openSandbox = sandboxFactory({ settings: () => images.settings(), visuals });
- const connectMachine = machineFactory({ getMode, settings: () => images.settings(), visuals });
+ const connectMachine = machineFactory({ getMode, settings: () => images.settings(), visuals, humanStop, sessionLabel });
const provider = await registerComputerProvider(pi, getSession, async () => {
const result: ComputerResult = session ? await session.close() : {
content: [{ type: "text", text: "Not started." }],
diff --git a/src/machine.ts b/src/machine.ts
index 2448921..8af5d43 100644
--- a/src/machine.ts
+++ b/src/machine.ts
@@ -16,6 +16,7 @@ import { IsolatedSession } from "./isolated-session.ts";
import { ComputerSession, type ComputerMode } from "./session.ts";
import { WebBrowser, WebError } from "./web-browser.ts";
import type { VisualSettings } from "./native-visuals.ts";
+import type { HumanStop } from "./human-stop.ts";
/** The configured machine for a connect command; unknown names refuse before any SSH work. */
export async function machineConfig(cwd: string, name: string): Promise<NonNullable<ComputerConfig["machines"]>[string]> {
@@ -83,13 +84,13 @@ export function sandboxFactory(options: { settings: () => CuaVisionSettings; vis
* Another machine's desktop gets its own driver and session over SSH. Safe/YOLO is the user's choice there too;
* the host overlay is not shown, and browsers start on that machine.
*/
-export function machineFactory(options: { getMode: () => ComputerMode; settings: () => CuaVisionSettings; visuals?: VisualSettings }): MachineFactory {
+export function machineFactory(options: { getMode: () => ComputerMode; settings: () => CuaVisionSettings; visuals?: VisualSettings; humanStop?: HumanStop; sessionLabel?: () => string | undefined }): MachineFactory {
return async (cwd, name, signal) => {
const machine = await machineConfig(cwd, name);
const runner = sshRunner(machine.ssh);
const command = machine.command ?? await prepareRemoteRuntime(runner, signal);
const runtime = machine.command ? machine.runtime : "windows-desktop";
- const driver = new McpDriver({ command, ...(runtime ? { runtime } : {}), ...(machine.driverOverride ? { driverOverride: true as const } : {}), ssh: machine.ssh }, undefined, options.visuals);
+ const driver = new McpDriver({ command, ...(runtime ? { runtime } : {}), ...(machine.driverOverride ? { driverOverride: true as const } : {}), ssh: machine.ssh }, undefined, options.visuals, options.humanStop, options.sessionLabel);
try { await waitForGuestDriver(driver, signal, 30_000); }
catch (error) { await driver.close().catch(() => undefined); throw error; }
const launchWeb = async (launch: Parameters<typeof WebBrowser.launch>[0], webSignal?: AbortSignal) => {
diff --git a/src/mode.ts b/src/mode.ts
index eb829d8..a37e639 100644
--- a/src/mode.ts
+++ b/src/mode.ts
@@ -1,12 +1,27 @@
import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent";
import type { ComputerMode } from "./session.ts";
+import { HumanStop } from "./human-stop.ts";
-/** Register a user command, not a model tool. Restore user policy on reload; never start a native runtime here. */
-export function registerComputerMode(pi: ExtensionAPI, clearFrame: () => Promise<void> = async () => {}, canEnterSafe: () => boolean = () => true): () => ComputerMode {
+/**
+ * Register a user command, not a model tool. Restore user policy on reload; never start a native runtime here.
+ * The human's stop (Escape pressed three times) shows as STOPPED until the human selects a mode again.
+ */
+export function registerComputerMode(pi: ExtensionAPI, clearFrame: () => Promise<void> = async () => {}, canEnterSafe: () => boolean = () => true,
+ humanStop = new HumanStop()): () => ComputerMode {
let mode: ComputerMode = "yolo";
- const show = (ctx: ExtensionContext) => {
- if (ctx.hasUI) ctx.ui.setStatus("computer-use", `Computer: ${mode.toUpperCase()}`);
+ let ui: ExtensionContext["ui"] | undefined;
+ const show = (ctx?: ExtensionContext) => {
+ if (ctx) ui = ctx.hasUI ? ctx.ui : undefined;
+ ui?.setStatus("computer-use", humanStop.stopped ? "Computer: STOPPED" : `Computer: ${mode.toUpperCase()}`);
};
+ humanStop.onStop(() => {
+ show();
+ ui?.notify("Computer use stopped: Escape was pressed three times. Agent input on your desktop is off; /computer-mode safe or /computer-mode yolo turns it on again.", "warning");
+ // Steer a running turn, or wait for the next one: the agent must not keep planning desktop input.
+ pi.sendMessage({ customType: "computer-use-stop", display: true,
+ content: "The human pressed Escape three times and stopped computer use on their desktop. Every computer action on it is refused until they turn it on again with /computer-mode. Do not retry or work around it; tell the human what you were doing and wait." },
+ { deliverAs: "steer" });
+ });
const restore = async (ctx: ExtensionContext) => {
mode = "yolo";
for (const entry of ctx.sessionManager.getBranch()) {
@@ -24,7 +39,7 @@ export function registerComputerMode(pi: ExtensionAPI, clearFrame: () => Promise
pi.on("session_start", (_event, ctx) => restore(ctx));
pi.on("session_tree", (_event, ctx) => restore(ctx));
pi.registerCommand("computer-mode", {
- description: "Show or select computer UI access: safe (session-opened windows) or yolo (desktop-wide). Not an OS sandbox.",
+ description: "Show or select computer UI access: safe (session-opened windows) or yolo (desktop-wide). Selecting a mode also turns computer use on again after Escape stopped it. Not an OS sandbox.",
getArgumentCompletions: prefix => ["safe", "yolo"].filter(value => value.startsWith(prefix)).map(value => ({ value, label: value })),
handler: async (args, ctx) => {
const selected = args.trim().toLowerCase();
@@ -42,8 +57,17 @@ export function registerComputerMode(pi: ExtensionAPI, clearFrame: () => Promise
mode = selected;
await clearFrame();
}
+ const resumed = selected !== "" && humanStop.stopped;
+ if (resumed) {
+ humanStop.resume();
+ pi.sendMessage({ customType: "computer-use-stop", display: true, content: "The human turned computer use on their desktop on again." }, { deliverAs: "nextTurn" });
+ }
show(ctx);
- ctx.ui.notify(`Computer mode: ${mode.toUpperCase()}. ${mode === "safe" ? "Only session-opened windows are allowed." : "Desktop-wide UI access is allowed."} Not an OS sandbox.`, "info");
+ if (humanStop.stopped) {
+ ctx.ui.notify(`Computer mode: STOPPED by Escape (${mode.toUpperCase()} when on). /computer-mode safe or /computer-mode yolo turns it on again.`, "warning");
+ return;
+ }
+ ctx.ui.notify(`Computer mode: ${mode.toUpperCase()}${resumed ? ", on again" : ""}. ${mode === "safe" ? "Only session-opened windows are allowed." : "Desktop-wide UI access is allowed."} Not an OS sandbox.`, "info");
},
});
return () => mode;
diff --git a/src/native-visuals.ts b/src/native-visuals.ts
index 48d4b56..140ad9f 100644
--- a/src/native-visuals.ts
+++ b/src/native-visuals.ts
@@ -4,10 +4,17 @@ import type { ScreenRect } from "./highlight-geometry.ts";
export interface VisualOptions {
preset: "off" | "subtle" | "full";
observations: boolean; previews: boolean; pointer: boolean; scroll_drag: boolean; keyboard: boolean;
+ /** Show the text the agent types beside the action; the native renderer never stores it. */
+ typed_text: boolean;
opacity: number; quality: "performance" | "balanced" | "high"; reduced_motion: boolean;
}
const defaults = (): VisualOptions => ({ preset: "full", observations: true, previews: true, pointer: true,
- scroll_drag: true, keyboard: true, opacity: 0.8, quality: "balanced", reduced_motion: false });
+ scroll_drag: true, keyboard: true, typed_text: true, opacity: 0.8, quality: "balanced", reduced_motion: false });
+/** The settings the native renderer takes; whether typed text is sent at all is decided here. */
+export function nativeVisualOptions(options: VisualOptions): Record<string, unknown> {
+ const { typed_text: _typed, ...native } = options;
+ return native;
+}
/** Human-owned, session-local presentation preferences. No model tool can write them. */
export class VisualSettings {
private value = defaults();
@@ -21,7 +28,7 @@ export class VisualSettings {
const next = { ...base, ...patch };
if (!["off", "subtle", "full"].includes(next.preset) || !["performance", "balanced", "high"].includes(next.quality) ||
!Number.isFinite(next.opacity) || next.opacity < 0 || next.opacity > 1 ||
- [next.observations, next.previews, next.pointer, next.scroll_drag, next.keyboard, next.reduced_motion].some(v => typeof v !== "boolean"))
+ [next.observations, next.previews, next.pointer, next.scroll_drag, next.keyboard, next.typed_text, next.reduced_motion].some(v => typeof v !== "boolean"))
throw Error("Invalid visual settings.");
if (patch.preset === "subtle" && patch.previews === undefined) next.previews = false;
if (patch.preset === "full" && patch.previews === undefined) next.previews = true;
@@ -40,6 +47,8 @@ interface ActivityPayload {
image?: Extract<Content, { type: "image" }>;
action?: string;
outcome?: "delivered" | "refused" | "unknown" | "verified";
+ /** Text the action types, already free of control characters and at most 200 characters. */
+ typedText?: string;
}
export interface WindowFrame extends ActivityPayload { target: WindowTarget; identity: string; bounds: ScreenRect }
export interface DisplayFrame extends ActivityPayload { display: string; bounds: ScreenRect }
@@ -80,9 +89,16 @@ export async function notifyHighlight(observer: WindowObserver | undefined, fram
catch { return { status: "unavailable", reason: "renderer_unavailable" }; }
}
export function notifyWindowObserver(observer: WindowObserver | undefined, target: WindowTarget | undefined, identity: string | undefined,
- bounds: unknown, signal?: AbortSignal, mark?: HighlightMark, image?: Content, action?: string, outcome?: ActivityPayload["outcome"]): Promise<HighlightResult | undefined> {
+ bounds: unknown, signal?: AbortSignal, mark?: HighlightMark, image?: Content, action?: string, outcome?: ActivityPayload["outcome"], typedText?: string): Promise<HighlightResult | undefined> {
return notifyHighlight(observer, target && identity && bounds ? { target: { ...target }, identity, bounds: bounds as ScreenRect,
- ...(mark ? { mark } : {}), ...(image?.type === "image" ? { image } : {}), ...(action ? { action } : {}), ...(outcome ? { outcome } : {}) } : undefined, signal);
+ ...(mark ? { mark } : {}), ...(image?.type === "image" ? { image } : {}), ...(action ? { action } : {}), ...(outcome ? { outcome } : {}),
+ ...(typedText ? { typedText } : {}) } : undefined, signal);
+}
+
+/** What the card shows of a typed string: control characters become spaces and only the last 200 characters stay. */
+export function shownTypedText(text: string | undefined): string | undefined {
+ const shown = text?.replace(/\p{Cc}+/gu, " ").trim();
+ return shown ? [...shown].slice(-200).join("") : undefined;
}
/** One native admission in flight and one replaceable pending image. Native owns animation timing. */
@@ -116,7 +132,8 @@ export class NativeVisuals {
if (mark?.kind !== "action" && !options.observations) return { status: "skipped" };
const args: Record<string, unknown> = { generation, kind: mark?.kind ?? "overview", source,
...(mark?.region ? { region: { ...mark.region } } : {}), ...(mark?.point ? { point: { ...mark.point } } : {}),
- ...(frame.action ? { action: frame.action } : {}), ...(frame.outcome ? { outcome: frame.outcome } : {}) };
+ ...(frame.action ? { action: frame.action } : {}), ...(frame.outcome ? { outcome: frame.outcome } : {}),
+ ...(mark?.kind === "action" && frame.typedText && options.keyboard && options.typed_text ? { typed_text: frame.typedText } : {}) };
// Bound transient transport memory. Oversized images still get a source frame, without a preview.
if (options.previews && image && ["image/png", "image/jpeg"].includes(image.mimeType) && image.data.length <= 4 * 1024 * 1024)
args.image = { data: image.data, mime_type: image.mimeType };
@@ -139,7 +156,7 @@ export class NativeVisuals {
}
private drain(): void {
if (this.busy || this.closed || this.unavailable) return;
- const next = this.config ? { name: "set_agent_visuals" as const, args: { ...this.config }, time: performance.now() } : this.pending;
+ const next = this.config ? { name: "set_agent_visuals" as const, args: nativeVisualOptions(this.config), time: performance.now() } : this.pending;
if (!next) return;
if (this.config) this.config = undefined; else this.pending = undefined;
if (next.name === "present_agent_activity" && next.args.kind !== "clear" && performance.now() - next.time > 250) { this.drain(); return; }
diff --git a/src/provider.ts b/src/provider.ts
index 5336ee0..974cb53 100644
--- a/src/provider.ts
+++ b/src/provider.ts
@@ -49,7 +49,7 @@ const REPLACED: [RegExp, string][] = [
[/^(key|hotkey|press)$/, "computer_press"], [/^hold$/, "computer_press with hold_ms"], [/^(scroll|wheel)$/, "computer_scroll"], [/^drag$/, "computer_drag"], [/^(hover|move_pointer)$/, "computer_hover"],
[/^(select|select_option|menu)$/, "computer_select"], [/^(timeline|sequence)$/, "computer_timeline"],
[/^(open|open_browser|open_sandbox|open_session|attach_browser|webview2_debugging)$/, "computer_open"],
- [/^(move|resize|minimize|maximize|focus|restore|name)$/, "computer_window (a pointer move: computer_hover; resize of a display: computer_display)"], [/^(wait_for|verify)$/, "computer_wait"],
+ [/^(move|resize|minimize|maximize|restore|name)$/, "computer_window (a pointer move: computer_hover; resize of a display: computer_display)"], [/^(wait_for|verify)$/, "computer_wait"],
[/^(navigate|back|forward|reload|wait_load|new_tab|switch_tab|close_tab|dialog|upload|wait_download|cancel_download|evaluate)$/, "computer_browser"],
[/^(put|get|files)$/, "computer_files"], [/^clipboard_/, "computer_clipboard"], [/^(record_start|record_stop|audio_record)$/, "computer_record"],
[/^(close|kill)$/, "Delete (kill: Delete with ?force=true)"], [/^setup$/, "/computer-setup, which only the user runs (ask them)"], [/^batch$/, "Codemode: a script that calls the computer_* tools in order"],
diff --git a/src/runtime-pins.json b/src/runtime-pins.json
index 684fc18..eafc584 100644
--- a/src/runtime-pins.json
+++ b/src/runtime-pins.json
@@ -1,12 +1,12 @@
{
"format": 2,
- "source": "4b547b547edf415908557829fa723f143f781649",
+ "source": "3e69266e1dd832dd57f0ddda25091a01537d84e6",
"version": "0.38.0",
- "url": "https://github.com/alexshpunt/cua/releases/download/pi-runtime-0.38.0-4b547b54/",
+ "url": "https://github.com/alexshpunt/cua/releases/download/pi-runtime-0.38.0-3e69266e/",
"files": {
- "cua-driver.exe": "18d3e3378a4111cf41e3fafd85035525dab1870af77b1ac0725f5b410397f623",
- "cua-driver-uia.exe": "8a724cd67d92a91a6df47df7180d1ecf3be2de7c12b90e777fd9d14ec4ba65be",
- "cua-cursor-theme.exe": "d91276d2301c62dfe9adf5bc8a30846b40ec3057defea1ff670f10c2345deea4",
+ "cua-driver.exe": "d210ca0ce302c8acd6c024b57966e4369291d15ae5d83a3ab2fcdd7f9f1c36cc",
+ "cua-driver-uia.exe": "cd119e33f6726d718e900841127bf47cf2f5c6558e14cfbc48f7347f2b254f2e",
+ "cua-cursor-theme.exe": "19ebe9b201c70c0c921ee727f86bac612fbcbac04a4c1ac6928b93fdb872e0ec",
"LICENSE.md": "c0779290c1d4783169aa3dbfb55feb505e563ef8a004bbf55298ceffcfbda8d9",
"THIRD_PARTY_NOTICES.md": "1fb1119fed1ad8e3397558aeda43b3e323fd80a1cbbe58934546dc6cee9e8bf7"
}
diff --git a/src/session.ts b/src/session.ts
index badf449..0b8d14a 100644
--- a/src/session.ts
+++ b/src/session.ts
@@ -15,7 +15,7 @@ import { isCoordinateScroll, scrollReceipt } from "./scroll-actions.ts";
import { validWindowStatus, windowRestoreStatus } from "./window-restore.ts";
import { validWindowControlState, windowManagementStatus, type WindowManagementCommand } from "./window-management.ts";
import { nativeInput, requestedInputTime, validateInput, type InputCommand, type InputStep } from "./complete-input.ts";
-import { notifyWindowObserver, notifyHighlight, type WindowObserver, type HighlightResult, type HighlightMark } from "./native-visuals.ts";
+import { notifyWindowObserver, notifyHighlight, shownTypedText, type WindowObserver, type HighlightResult, type HighlightMark } from "./native-visuals.ts";
import { screenRect, elementRegion, windowImageMapping, windowRegion, nativeClickPoint, type ScreenRect } from "./highlight-geometry.ts";
import { observationSource, sourceIdentity, type ObservationSource, type SourceIdentity, type NativePlatform } from "./observation-source.ts";
import { actionCapability, type ActionCapability } from "./action-capability.ts";
@@ -1324,7 +1324,8 @@ export class ComputerSession {
}, signal);
if (!this.allowed(entry)) return this.denied(entry.ref);
if (signal?.aborted) return this.result("refused", "Cancelled during observation; captured data withheld.");
- if (native.isError) return this.result("refused", "Driver refused observation.", native.structuredContent, this.addressed(entry, native.structuredContent?.virtual_desktop));
+ if (native.isError) return this.result("refused", native.structuredContent?.code === "human_stop"
+ ? native.content.filter(c => c.type === "text").map(c => c.text).join("\n") : "Driver refused observation.", native.structuredContent, this.addressed(entry, native.structuredContent?.virtual_desktop));
if (!native.structuredContent) throw new Error("Driver returned no structured window state.");
const state = { ...native.structuredContent };
if (hasImage && typeof state.screenshot_error === "string" && state.screenshot_error.trim()) {
@@ -2140,7 +2141,11 @@ export class ComputerSession {
// AX bounds are native. Only native click receipts establish an exact pixel action mark.
const mark = { kind: "action" as const, region: step.element_token ? observed.frames.get(step.element_token) : undefined };
const activityAction = presentationAction(step);
- let highlight = await notifyWindowObserver(this.observer, entry.target, entry.identity, observed.bounds, signal, mark, undefined, activityAction);
+ // A password field's text is never shown.
+ const typedElement = ["type_text", "set_value"].includes(step.action)
+ ? observed.ui?.elements.find(element => step.element_token ? element.element_token === step.element_token : element.focused) : undefined;
+ const typedText = ["type_text", "set_value"].includes(step.action) && !typedElement?.password ? shownTypedText(step.text) : undefined;
+ let highlight = await notifyWindowObserver(this.observer, entry.target, entry.identity, observed.bounds, signal, mark, undefined, activityAction, undefined, typedText);
if (signal?.aborted) return finish(this.result("refused", "Cancelled during action mark; input was not dispatched."), "Cancelled during action mark.");
if (!this.inputAllowed(entry)) return finish(this.denied(entry.ref), "Permission revoked during action mark.");
if (batch && performance.now() - started >= SEQUENCE_ELAPSED_MS) return finish(this.result("refused", "Sequence time budget reached."), `${SEQUENCE_ELAPSED_MS / 60000} min elapsed budget reached during admission; no more steps dispatched.`);
@@ -2177,12 +2182,13 @@ export class ComputerSession {
"In-flight outcome withheld after revocation/cancellation.");
const state = native.structuredContent ?? {};
const nativeMessage = native.content.filter(c => c.type === "text").map(c => c.text).join("\n");
- // Foreground input checks for recent physical input before sending anything (CUA_HUMAN_IDLE_MS).
- const takeover = /human_takeover:/.test(nativeMessage) && (native.isError || state.effect === "refused");
+ // Three physical Escape presses stop input: nothing more is sent, and a running string or timeline stops part way.
+ const stopped = /human_stop:/.test(nativeMessage) || state.code === "human_stop";
+ const stopRefused = stopped && (state.status === "refused" || state.effect === "refused");
// A real press at a point another window covers is refused by the driver before anything is pressed there.
const covering = coveringWindow(nativeMessage);
- const refused = takeover || state.status === "refused" || state.effect === "refused" || ["invalid_action_target", "invalid_arguments", "stale_element_token",
- "conflicting_element_target", "background_unavailable", "background_occluded", "background_uipi_blocked", "key_hold_unavailable", "held_path_unavailable"].includes(String(state.code));
+ const refused = stopRefused || state.status === "refused" || state.effect === "refused" || ["invalid_action_target", "invalid_arguments", "stale_element_token",
+ "conflicting_element_target", "background_unavailable", "background_occluded", "background_uipi_blocked", "key_hold_unavailable", "held_path_unavailable", "input_busy"].includes(String(state.code));
const uncertain = native.isError || ["unknown", "partial"].includes(String(state.status)) || ["unknown", "partial"].includes(String(state.effect));
const inconsistent = !refused && !uncertain && (isSemanticAction(step.action) ? !semanticReceipt(step.action, state)
: isPointerMove(step.action) ? delivery_mode !== "foreground" || !pointerReceipt(step, state)
@@ -2207,7 +2213,7 @@ export class ComputerSession {
: { reason: "Native action did not report usable target geometry." };
}
highlight = await notifyWindowObserver(this.observer, entry.target, entry.identity, observed.bounds, signal,
- { kind: "action", ...(resolvedPoint ? { point: resolvedPoint } : step.element_token && mark.region ? { region: mark.region } : {}) }, undefined, activityAction, status);
+ { kind: "action", ...(resolvedPoint ? { point: resolvedPoint } : step.element_token && mark.region ? { region: mark.region } : {}) }, undefined, activityAction, status, typedText);
if (!this.inputAllowed(entry) || (!batch && signal?.aborted)) {
entry.lastAction = { reason: "In-flight pointer outcome was withheld after revocation or cancellation." };
return finish(this.result("unknown", "Input was dispatched before revocation or cancellation. Do not replay blindly."), "In-flight outcome withheld after input.");
@@ -2216,7 +2222,7 @@ export class ComputerSession {
// A click that closed its window, such as OK in a dialog, names the window of the app the driver found in front.
const after = status === "delivered" ? foregroundAfter(state.foreground_after) : undefined;
const front = after && `The click closed this window; ${await this.windowName(after, signal)} of the same app is now in front.\n`;
- const result = this.result(status, takeover ? `Foreground input paused because the human is using the computer: ${nativeMessage.slice(nativeMessage.indexOf("human_takeover:") + 16)}`
+ const result = this.result(status, stopped ? `The human stopped computer use with Escape. ${status === "refused" ? "Nothing was sent." : "Input stopped part way; observe before anything else once computer use is on again."} Do not retry or work around it; tell the human and wait until they turn it on again with /computer-mode.`
: cover && status === "refused" ? `Nothing was pressed: ${cover}` : `${cover ? `A press stopped: ${cover}\n` : ""}${front ?? ""}Driver returned ${status} for ${name} (requested: ${delivery_mode ?? "background"}). ${isSemanticAction(step.action) ? "A UIA receipt is not app or focus verification. " : ""}Observe again to verify application state.${step.action === "hold" && typeof state.summary === "string" ? `\n${state.summary}` : ""}`,
{ ...state, ...(step.element_token || pointerAction ? { coordinate_grounding: { input_space: step.element_token ? "native_element_token" : "window_image_pixels", transform: "cua_native_once",
overview: { width: observed.width, height: observed.height }, ...(step.x !== undefined ? { requested: { x: step.x, y: step.y } } : {}),
diff --git a/src/visual-settings-command.ts b/src/visual-settings-command.ts
index 96a691d..01dfa2b 100644
--- a/src/visual-settings-command.ts
+++ b/src/visual-settings-command.ts
@@ -23,7 +23,7 @@ export function registerVisualSettingsCommand(pi: ExtensionAPI, settings: Visual
pi.appendEntry("computer-use-visuals", settings.update(patch));
}
const value = settings.current();
- ctx.ui.notify(`Computer visuals: ${value.preset}. Previews ${value.previews ? "on" : "off"}; ${value.quality} quality; opacity ${Math.round(value.opacity * 100)}%; reduced motion ${value.reduced_motion ? "on" : "off"}.`, "info");
+ ctx.ui.notify(`Computer visuals: ${value.preset}. Previews ${value.previews ? "on" : "off"}; typed text ${value.typed_text ? "on" : "off"}; ${value.quality} quality; opacity ${Math.round(value.opacity * 100)}%; reduced motion ${value.reduced_motion ? "on" : "off"}.`, "info");
} catch {
ctx.ui.notify('Use /computer-visuals off|subtle|full, or JSON such as {"previews":false,"reduced_motion":true}. Preferences were not changed.', "warning");
}
diff --git a/test/computer-tools.test.ts b/test/computer-tools.test.ts
index 5ec3e3c..fb20ac6 100644
--- a/test/computer-tools.test.ts
+++ b/test/computer-tools.test.ts
@@ -41,7 +41,7 @@ test("computer_window takes exactly the fields of its action", () => {
expect(windowCommand({ window: "w", action: "resize", width: 800, height: 600 })).toEqual({ action: "resize", width: 800, height: 600, unit: "windows_desktop_pixels" });
expect(windowCommand({ window: "w", action: "restore" })).toEqual({ action: "restore" });
expect(() => windowCommand({ window: "w", action: "move", x: 1 })).toThrow("move needs x and y and nothing else.");
- expect(() => windowCommand({ window: "w", action: "focus", width: 1 })).toThrow("focus takes no coordinates, size or name.");
+ expect(() => windowCommand({ window: "w", action: "minimize", width: 1 })).toThrow("minimize takes no coordinates, size or name.");
expect(windowCommand({ window: "w", action: "name", as: "game" })).toEqual({ action: "name", as: "game" });
expect(() => windowCommand({ window: "w", action: "name" })).toThrow("name needs as and nothing else.");
expect(() => windowCommand({ window: "w", action: "move", x: 1, y: 2, as: "game" })).toThrow("move needs x and y and nothing else.");
diff --git a/test/fixtures/human-stop-driver.mjs b/test/fixtures/human-stop-driver.mjs
new file mode 100644
index 0000000..67646ff
--- /dev/null
+++ b/test/fixtures/human-stop-driver.mjs
@@ -0,0 +1,32 @@
+import { appendFileSync } from "node:fs";
+import { Server } from "@modelcontextprotocol/sdk/server/index.js";
+import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { agentCursor } from "./agent-cursor.mjs";
+import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";
+
+// Deterministic MCP peer for the human stop: `press_key` with key "escape-x3" plays three physical Escape presses
+// during the call, as the native hook would see them. Never native Windows qualification.
+const log = process.argv[2];
+const mode = process.argv[3] ?? "notify";
+const record = entry => appendFileSync(log, JSON.stringify(entry) + "\n");
+record({ event: "start", pid: process.pid, env: Object.fromEntries(["CUA_ESCAPE_STOP", "CUA_SHARED_INPUT_LOCK", "CUA_SESSION_LABEL", "CUA_HUMAN_IDLE_MS"].map(name => [name, process.env[name] ?? null])) });
+let stopped = false;
+const refusal = () => ({ content: [{ type: "text", text: "human_stop: the human pressed Escape three times and stopped agent input." }], isError: true,
+ structuredContent: { status: "refused", effect: "refused", code: "human_stop" } });
+const server = new Server({ name: "cua-driver", version: "0.33.2" }, { capabilities: { tools: {} } });
+server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: [] }));
+server.setRequestHandler(CallToolRequestSchema, async request => {
+ const cursor = agentCursor(request.params.name, request.params.arguments); if (cursor) return cursor;
+ const { name, arguments: args } = request.params;
+ record({ name, arguments: args });
+ if (stopped) return refusal();
+ if (name === "press_key" && args?.key === "escape-x3") {
+ stopped = true;
+ // The notification is sent as soon as the hook sees the third press; the running call then stops.
+ if (mode === "notify") await server.notification({ method: "notifications/cua/human_stop", params: { code: "human_stop" } });
+ return refusal();
+ }
+ return { content: [{ type: "text", text: "{}" }], structuredContent: { effect: "unverifiable", windows: [] } };
+});
+process.stdin.on("end", () => record({ event: "stdin-eof", pid: process.pid }));
+await server.connect(new StdioServerTransport());
diff --git a/test/fixtures/window-management-extension.ts b/test/fixtures/window-management-extension.ts
index 910e7e6..3aa134d 100644
--- a/test/fixtures/window-management-extension.ts
+++ b/test/fixtures/window-management-extension.ts
@@ -118,7 +118,7 @@ export default async function fixture(pi: ExtensionAPI): Promise<void> {
assert(!result.isError && semantic(result).effect === "applied", `Valid desktop move failed: ${body(result)}`);
assert(find(result, "mutation_attempted") === true && find(result, "postcondition_confirmed") === true,
"Delivered management discarded native completion prefix");
- for (const command of [{ action: "resize", width: 180, height: 120 }, { action: "focus" }, { action: "maximize" }, { action: "minimize" }]) {
+ for (const command of [{ action: "resize", width: 180, height: 120 }, { action: "maximize" }, { action: "minimize" }]) {
await control("reset");
const result = await manage(first, command);
assert(!result.isError && semantic(result).effect === "applied", `Valid ${command.action} failed: ${body(result)}`);
@@ -130,7 +130,7 @@ export default async function fixture(pi: ExtensionAPI): Promise<void> {
const ui = await ok("read", { path: first, views: ["ui"] });
await stateRead(first);
await ok("read", { path: reference(ui, "observation") });
- assert(await count("manage_window") === 6, "Read-only state sent a mutation");
+ assert(await count("manage_window") === 5, "Read-only state sent a mutation");
} else if (scenario === "admission") {
const external = reference(await ok("search", { path: "computer:machine/local/windows", query: "Unowned management fixture" }), "window");
await stateRead(external); mode = "safe";
@@ -158,7 +158,7 @@ export default async function fixture(pi: ExtensionAPI): Promise<void> {
const result = await window(bad);
assert(result.isError, `computer_window accepted ${JSON.stringify(bad)}: ${body(result)}`);
}
- await denied("computer_window", { window: "computer:machine/local", action: "focus" });
+ await denied("computer_window", { window: "computer:machine/local", action: "maximize" });
assert(await count("manage_window") === 0, "Invalid computer_window calls reached native management");
const moved = await window({ action: "move", x: -20, y: 120 });
assert(!moved.isError && semantic(moved).effect === "applied" && body(moved).includes("on local"), `computer_window move failed: ${body(moved)}`);
@@ -194,7 +194,7 @@ export default async function fixture(pi: ExtensionAPI): Promise<void> {
await denied("computer_select", { window: first, target: "1:a", checked: true, values: ["a"] });
const external = reference(await ok("search", { path: "computer:machine/local/windows", query: "Unowned management fixture" }), "window");
mode = "safe";
- const refused = await window({ action: "focus" }, external);
+ const refused = await window({ action: "maximize" }, external);
assert(refused.isError && semantic(refused).effect === "not-applied", `Safe mode let computer_window manage a foreign window: ${body(refused)}`);
mode = "yolo";
} else if (scenario === "receipts") {
@@ -213,17 +213,17 @@ export default async function fixture(pi: ExtensionAPI): Promise<void> {
`Cancellation after native completion discarded the delivered prefix: ${body(cancelled)}`);
} else {
await control("minimized");
- const focus = await denied("computer_window", { window: first, action: "focus" });
- assert(semantic(focus).effect === "not-applied" && find(focus, "mutation_attempted") === false && await count("restore_window") === 0,
- "Focus implicitly restored or activated a minimized target");
+ await denied("computer_window", { window: first, action: "focus" });
+ assert(await count("manage_window") === 0 && await count("restore_window") === 0,
+ "Removed focus action reached native management or restored a minimized window");
const restored = await manage(first, { action: "restore" });
assert(!restored.isError && semantic(restored).effect === "applied" && find(restored, "restore_requested") === true,
`Control admission lost existing restore: ${body(restored)}`);
for (const situation of ["other_desktop", "unknown_desktop"]) {
await control("reset"); await control(situation);
const result = await denied("computer_window", { window: first, action: "focus" });
- assert(semantic(result).effect === "not-applied" && find(result, "mutation_attempted") === false,
- `${situation} activation implicitly switched desktops`);
+ assert(result.isError && await count("manage_window") === 0,
+ `${situation} accepted the removed focus action or dispatched native management`);
}
management = false; await control("reset"); await control("minimized");
const managementBefore = await count("manage_window");
diff --git a/test/human-stop.test.ts b/test/human-stop.test.ts
new file mode 100644
index 0000000..eaf5daa
--- /dev/null
+++ b/test/human-stop.test.ts
@@ -0,0 +1,101 @@
+import { mkdtemp, readFile, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import type { ExtensionAPI, ExtensionCommandContext } from "@earendil-works/pi-coding-agent";
+import { afterEach, expect, test, vi } from "vitest";
+import { McpDriver } from "../src/driver.ts";
+import { HumanStop, reportsHumanStop } from "../src/human-stop.ts";
+import { registerComputerMode } from "../src/mode.ts";
+
+const dirs: string[] = [];
+afterEach(async () => { await Promise.all(dirs.splice(0).map(dir => rm(dir, { recursive: true, force: true }))); });
+
+type Entry = { event?: string; pid?: number; env?: Record<string, string | null>; name?: string; arguments?: Record<string, unknown> };
+async function start(mode: "notify" | "refuse-only" = "notify") {
+ const dir = await mkdtemp(path.join(tmpdir(), "human-stop-"));
+ dirs.push(dir);
+ const log = path.join(dir, "calls.jsonl");
+ const stop = new HumanStop();
+ const driver = new McpDriver({ command: process.execPath, args: [path.resolve("test/fixtures/human-stop-driver.mjs"), log, mode] }, "0.33.2", undefined,
+ stop, () => "Fix the build: <LPT-1088>");
+ const entries = async () => (await readFile(log, "utf8")).trim().split("\n").map(line => JSON.parse(line) as Entry);
+ return { driver, stop, entries };
+}
+const until = async (check: () => Promise<boolean>) => {
+ for (let i = 0; i < 100; i++) { if (await check()) return; await new Promise(resolve => setTimeout(resolve, 10)); }
+ throw new Error("condition not reached");
+};
+
+test("the driver stops on the native notification, refuses without reaching the process, and a resume starts a new one", async () => {
+ const { driver, stop, entries } = await start();
+ const stops = vi.fn();
+ stop.onStop(stops);
+ try {
+ expect((await driver.call("list_windows", {})).isError).toBeFalsy();
+ const [first] = await entries();
+ // Three physical Escape presses stop input; agents take turns per action; the cursor names the session; nothing waits for the human.
+ expect(first.env).toEqual({ CUA_ESCAPE_STOP: "1", CUA_SHARED_INPUT_LOCK: "1", CUA_SESSION_LABEL: "Fix the build LPT-1088", CUA_HUMAN_IDLE_MS: null });
+
+ const stopped = await driver.call("press_key", { key: "escape-x3" });
+ expect(stopped.structuredContent).toMatchObject({ code: "human_stop", status: "refused" });
+ expect(stop.stopped).toBe(true);
+ expect(stops).toHaveBeenCalledTimes(1);
+
+ const calls = (await entries()).length;
+ const refused = await driver.call("list_windows", {});
+ expect(refused).toMatchObject({ isError: true, structuredContent: { code: "human_stop", attempted: false } });
+ expect(refused.content[0]).toMatchObject({ type: "text", text: expect.stringContaining("/computer-mode") });
+ expect((await entries()).length).toBe(calls);
+
+ stop.resume();
+ await until(async () => (await entries()).some(entry => entry.event === "stdin-eof" && entry.pid === first.pid));
+ expect((await driver.call("list_windows", {})).isError).toBeFalsy();
+ const starts = (await entries()).filter(entry => entry.event === "start");
+ expect(starts).toHaveLength(2);
+ expect(starts[1].pid).not.toBe(first.pid);
+ } finally { await driver.close(); }
+});
+
+test("a refusal that names the stop is enough when the notification is missed", async () => {
+ const { driver, stop } = await start("refuse-only");
+ try {
+ await driver.call("press_key", { key: "escape-x3" });
+ expect(stop.stopped).toBe(true);
+ } finally { await driver.close(); }
+});
+
+test("native results report the stop by code or by message", () => {
+ expect(reportsHumanStop({ content: [], structuredContent: { code: "human_stop" } })).toBe(true);
+ expect(reportsHumanStop({ content: [], structuredContent: { refusal: { code: "human_stop" } } })).toBe(true);
+ expect(reportsHumanStop({ content: [{ type: "text", text: "Error: human_stop: the human pressed Escape" }] })).toBe(true);
+ expect(reportsHumanStop({ content: [{ type: "text", text: "foreground_lost" }], structuredContent: { code: "foreground_unavailable" } })).toBe(false);
+});
+
+test("the stop shows as STOPPED, tells the agent, and selecting a mode turns computer use on again", async () => {
+ const commands = new Map<string, { handler(args: string, ctx: ExtensionCommandContext): Promise<void> }>();
+ const events = new Map<string, (event: unknown, ctx: ExtensionCommandContext) => unknown>();
+ const sendMessage = vi.fn();
+ const pi = { registerCommand: (name: string, command: never) => commands.set(name, command), on: (name: string, handler: never) => events.set(name, handler),
+ appendEntry: vi.fn(), sendMessage } as unknown as ExtensionAPI;
+ const ctx = { hasUI: true, ui: { setStatus: vi.fn(), notify: vi.fn() }, sessionManager: { getBranch: () => [] } } as unknown as ExtensionCommandContext;
+ const stop = new HumanStop();
+ const getMode = registerComputerMode(pi, async () => {}, () => true, stop);
+ await events.get("session_start")!({}, ctx);
+
+ stop.stop();
+ expect(ctx.ui.setStatus).toHaveBeenLastCalledWith("computer-use", "Computer: STOPPED");
+ expect(ctx.ui.notify).toHaveBeenLastCalledWith(expect.stringContaining("Escape was pressed three times"), "warning");
+ expect(sendMessage).toHaveBeenLastCalledWith(expect.objectContaining({ customType: "computer-use-stop", content: expect.stringContaining("Do not retry") }), { deliverAs: "steer" });
+
+ // Showing the mode does not turn it on.
+ await commands.get("computer-mode")!.handler("", ctx);
+ expect(stop.stopped).toBe(true);
+ expect(ctx.ui.notify).toHaveBeenLastCalledWith(expect.stringContaining("STOPPED"), "warning");
+
+ // Selecting the current mode turns it on again.
+ await commands.get("computer-mode")!.handler("yolo", ctx);
+ expect(stop.stopped).toBe(false);
+ expect(getMode()).toBe("yolo");
+ expect(ctx.ui.setStatus).toHaveBeenLastCalledWith("computer-use", "Computer: YOLO");
+ expect(sendMessage).toHaveBeenLastCalledWith(expect.objectContaining({ content: expect.stringContaining("on again") }), { deliverAs: "nextTurn" });
+});
diff --git a/test/live/takeover.live.ts b/test/live/takeover.live.ts
deleted file mode 100644
index 1b46f97..0000000
--- a/test/live/takeover.live.ts
+++ /dev/null
@@ -1,58 +0,0 @@
-import { afterAll, expect, test } from "vitest";
-import { addresses, connectLive, text } from "./remote.ts";
-import { computerSource } from "../../src/computer-address.ts";
-
-// LPT-835: physical human input pauses foreground agent input for 10 s. Input from the host into a session-opened
-// Windows Sandbox arrives in the guest as physical input, so the host plays the human and the guest runs the agent.
-const live = await connectLive();
-let sandbox: string | undefined;
-afterAll(async () => { if (sandbox) await live.insert(sandbox, { action: "close" }).catch(() => undefined); await live.close(); });
-const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms));
-
-test("human input pauses the guest's foreground input, stops a running timeline and spares background input", async () => {
- const opened = await live.insert(live.machine, { action: "open_sandbox" });
- expect(opened.details.status, text(opened)).toBe("delivered");
- sandbox = computerSource("sandbox", opened.details.sandbox_ref!);
- const cmd = await live.insert(sandbox, { action: "open", app: "Command Prompt" });
- expect(cmd.details.status, text(cmd)).toBe("observed");
- const guest = `computer:window/${cmd.details.window_ref}`;
- const observation = async () => (await live.read(guest, ["image"])).details.observation_id!;
- const key = async (delivery: "foreground" | "background") =>
- live.insert(`computer:observation/${await observation()}?delivery=${delivery}`, { action: "key", key: "a" });
-
- // The human: one click on the restored sandbox window on the host.
- const host = addresses(await live.search(live.windows, "Windows Sandbox"), "window")[0];
- const state = await live.read(host, ["state"]);
- expect((await live.insert(`computer:window-state/${state.details.window_state_id}`, { action: "restore" })).details.status).toBe("delivered");
- const human = async () => {
- const view = await live.read(host, ["image"]);
- const { width, height } = view.details.agent_image!.output as { width: number; height: number };
- const click = await live.insert(`computer:observation/${view.details.observation_id}?delivery=foreground`,
- { action: "click", x: Math.round(width / 2), y: Math.round(height * 0.8) });
- expect(click.details.status, text(click)).toBe("delivered");
- };
-
- await sleep(11_000);
- expect((await key("foreground")).details.status).toBe("delivered");
-
- await human();
- const paused = await key("foreground");
- expect(paused.details.status).toBe("refused");
- expect(text(paused)).toMatch(/human is using the computer: .*retry in \d+ ms/);
- const background = await key("background");
- expect(text(background)).not.toMatch(/human_takeover|human is using/);
-
- await sleep(11_000);
- expect((await key("foreground")).details.status).toBe("delivered");
-
- // A running foreground timeline stops at its next event once the human acts, and releases what it holds.
- const events = Array.from({ length: 16 }, (_, i) => ({ at_ms: i * 500, type: i % 2 ? "key_up" : "key_down", key: "b" }));
- const timeline = live.insert(`computer:observation/${await observation()}?delivery=foreground`, { action: "timeline", events });
- await sleep(2_500);
- await human();
- const stopped = await timeline;
- console.log(text(stopped).split("\n").slice(0, 3).join("\n"));
- expect(text(stopped)).toMatch(/human_takeover/);
- expect(text(stopped)).toMatch(/everything held released: true/);
- expect(text(stopped)).not.toMatch(/16\/16 events sent/);
-});
diff --git a/test/live/typed-tools.live.ts b/test/live/typed-tools.live.ts
index 860b19e..c788024 100644
--- a/test/live/typed-tools.live.ts
+++ b/test/live/typed-tools.live.ts
@@ -8,7 +8,7 @@ import { connectLive, nativeForm, openWindow, WINDOWS_POWERSHELL_PATH } from "./
const live = await connectLive();
afterAll(() => live.close());
const manage = (window: string, fields: Record<string, unknown>) =>
- live.resources.manageWindow(live.resources.address(window), live.cwd, windowCommand({ window, action: "focus", ...fields } as Parameters<typeof windowCommand>[0]));
+ live.resources.manageWindow(live.resources.address(window), live.cwd, windowCommand({ window, action: "restore", ...fields } as Parameters<typeof windowCommand>[0]));
const remove = (source: string) => deleteComputer(live.resources, source, { path: source }, live.cwd);
const gone = async (title: string) => {
for (let i = 0; i < 20; i++) {
diff --git a/test/live/viewports.live.ts b/test/live/viewports.live.ts
index 6cd37f6..e7dd547 100644
--- a/test/live/viewports.live.ts
+++ b/test/live/viewports.live.ts
@@ -43,7 +43,7 @@ test("a viewport shows its part of the window at full resolution, takes clicks i
// The hidden console of the form's PowerShell opens through Windows Terminal, which can take the foreground a moment
// after the form appears; the first real click needs the form in front, so focus it until it stays there.
for (let attempt = 0; attempt < 10; attempt++) {
- await live.resources.manageWindow(requireComputerAddress(window), live.cwd, windowCommand({ window, action: "focus" }));
+ await live.resources.manageWindow(requireComputerAddress(window), live.cwd, { action: "focus" });
await new Promise(resolve => setTimeout(resolve, 500));
const state = (await live.read(window, ["state"])).details.state as { foreground_at_target?: boolean } | undefined;
if (state?.foreground_at_target) break;
diff --git a/test/live/watch.live.ts b/test/live/watch.live.ts
index 54ec125..4005afa 100644
--- a/test/live/watch.live.ts
+++ b/test/live/watch.live.ts
@@ -1,7 +1,7 @@
import { mkdir, readFile, rm, writeFile } from "node:fs/promises";
import { afterAll, expect, test } from "vitest";
import { requireComputerAddress } from "../../src/computer-resources.ts";
-import { dragPlan, windowCommand } from "../../src/computer-tools.ts";
+import { dragPlan } from "../../src/computer-tools.ts";
import { presentComputerResult } from "../../src/provider.ts";
import type { ComputerResult } from "../../src/session.ts";
import { connectLive, openWindow, WINDOWS_POWERSHELL_PATH } from "./remote.ts";
@@ -40,7 +40,7 @@ test("a watch of a viewport sees the panorama move during a drag and stops with
try {
// Windows Terminal on this machine can take the foreground after the form appears; a real drag needs the form in front.
for (let attempt = 0; attempt < 10; attempt++) {
- await live.resources.manageWindow(requireComputerAddress(window), live.cwd, windowCommand({ window, action: "focus" }));
+ await live.resources.manageWindow(requireComputerAddress(window), live.cwd, { action: "focus" });
await new Promise(resolve => setTimeout(resolve, 500));
const state = (await live.read(window, ["state"])).details.state as { foreground_at_target?: boolean } | undefined;
if (state?.foreground_at_target) break;
@@ -102,7 +102,7 @@ test("a watch of a viewport in a maximized window keeps its frame rate during a
const window = await openWindow(live, live.machine, WINDOWS_POWERSHELL_PATH, panoramaForm(title, true), `regex:^${title}$`);
try {
await new Promise(resolve => setTimeout(resolve, 1500));
- await live.resources.manageWindow(requireComputerAddress(window), live.cwd, windowCommand({ window, action: "focus" }));
+ await live.resources.manageWindow(requireComputerAddress(window), live.cwd, { action: "focus" });
const overview = await live.read(window, ["full"]);
const box = overview.details.ui!.elements.find(e => e.name === "Pano")!.bounds.window_image!;
const state = overview.details.state as { screenshot_width: number; screenshot_original_width: number };
diff --git a/test/live/window-focus.live.ts b/test/live/window-focus.live.ts
index 61b9d5d..445dd04 100644
--- a/test/live/window-focus.live.ts
+++ b/test/live/window-focus.live.ts
@@ -1,5 +1,4 @@
import { afterAll, expect, test } from "vitest";
-import { windowCommand } from "../../src/computer-tools.ts";
import { presentComputerResult } from "../../src/provider.ts";
import { connectLive, nativeForm, openWindow, WINDOWS_POWERSHELL_PATH } from "./remote.ts";
@@ -16,7 +15,7 @@ test("focus moves the foreground between two windows of other apps", async () =>
try {
await new Promise(resolve => setTimeout(resolve, 1500));
for (const [window, title] of [[a, `pi-focus-a-${stamp}`], [b, `pi-focus-b-${stamp}`], [a, `pi-focus-a-${stamp}`]] as const) {
- const result = await live.resources.manageWindow(live.resources.address(window), live.cwd, windowCommand({ window, action: "focus" }));
+ const result = await live.resources.manageWindow(live.resources.address(window), live.cwd, { action: "focus" });
expect(result.details.status, presentComputerResult(result)).toBe("delivered");
expect(await foreground()).toBe(title);
}
diff --git a/test/native-visuals.test.ts b/test/native-visuals.test.ts
index a595c8a..3ace281 100644
--- a/test/native-visuals.test.ts
+++ b/test/native-visuals.test.ts
@@ -1,5 +1,5 @@
import { expect, test, vi } from "vitest";
-import { NativeVisuals, VisualSettings, notifyHighlight } from "../src/native-visuals.ts";
+import { NativeVisuals, VisualSettings, nativeVisualOptions, notifyHighlight } from "../src/native-visuals.ts";
import sharp from "sharp";
import { ComputerSession, type Driver, type DriverResult } from "../src/session.ts";
import type { InputCommand } from "../src/complete-input.ts";
@@ -84,12 +84,12 @@ test("settings validate before mutation and unavailable native presentation does
});
-async function actionFixture(outcome: "delivered" | "refused" | "unknown" = "delivered") {
+async function actionFixture(outcome: "delivered" | "refused" | "unknown" = "delivered", options: { password?: boolean; settings?: VisualSettings } = {}) {
const png = { type: "image" as const, mimeType: "image/png", data: (await sharp({ create: {
width: frame.bounds.width, height: frame.bounds.height, channels: 3, background: "white",
} }).png().toBuffer()).toString("base64") };
const send = vi.fn(async (_name: string, _args: Record<string, unknown>) => true);
- const visuals = new NativeVisuals(send);
+ const visuals = new NativeVisuals(send, options.settings);
const driver: Driver = {
visuals: visuals.flash, heldKeys: true, heldPaths: true, captureBoundDrag: true, semanticActions: true,
identity: vi.fn(async () => frame.identity), close: vi.fn(async () => {}),
@@ -100,7 +100,7 @@ async function actionFixture(outcome: "delivered" | "refused" | "unknown" = "del
if (name === "get_window_state") return { content: [png], structuredContent: {
...frame.target, window_bounds: frame.bounds, capture_id: "11111111-1111-4111-8111-111111111111",
screenshot_width: frame.bounds.width, screenshot_height: frame.bounds.height,
- elements: [{ element_token: "field", role: "TextBox", frame: { x: 0, y: 60, w: 80, h: 40 } }],
+ elements: [{ element_token: "field", role: "TextBox", frame: { x: 0, y: 60, w: 80, h: 40 }, ...(options.password ? { password: true } : {}) }],
} };
if (outcome !== "delivered") return { content: [], structuredContent: { effect: outcome } };
return { content: [], structuredContent: name === "semantic_action"
@@ -151,12 +151,38 @@ test.each(nativeFamilies)("native $label retains its presentation family before
]);
for (const args of activities) {
expect(args.source).toEqual({ ...frame.target, identity: frame.identity, bounds: frame.bounds });
- expect(Object.keys(args).every(key => ["generation", "kind", "source", "region", "action", "outcome"].includes(key))).toBe(true);
+ expect(Object.keys(args).every(key => ["generation", "kind", "source", "region", "action", "outcome", "typed_text"].includes(key))).toBe(true);
if (!command.element_token) expect(args).not.toHaveProperty("region");
+ // The human sees what the agent types, before and after the dispatch.
+ expect(args.typed_text).toBe("text" in command ? command.text : undefined);
}
} finally { await f.close(); }
});
+test.each([
+ { label: "a password field", options: { password: true } },
+ { label: "typed text turned off", options: { settings: (() => { const settings = new VisualSettings(); settings.update({ typed_text: false }); return settings; })() } },
+])("typed text is not shown for $label", async ({ options }) => {
+ const f = await actionFixture("delivered", options);
+ try {
+ const listed = await f.session.windows();
+ const ref = (listed.details.state!.windows as { window_ref: string }[])[0].window_ref;
+ const read = await f.session.observe(ref); await tick(); f.send.mockClear();
+ const result = await f.session.act({ window_ref: ref, observation_id: read.details.observation_id!, action: "set_value", element_token: "field", text: "secret" });
+ await tick();
+ expect(result.details.status).toBe("delivered");
+ const activities = f.send.mock.calls.filter(([name]) => name === "present_agent_activity").map(([, args]) => args);
+ expect(activities).toHaveLength(2);
+ for (const args of activities) expect(args).not.toHaveProperty("typed_text");
+ } finally { await f.close(); }
+});
+
+test("the native renderer gets its own settings without the typed text toggle", () => {
+ const options = new VisualSettings().current();
+ expect(options.typed_text).toBe(true);
+ expect(nativeVisualOptions(options)).toEqual(Object.fromEntries(Object.entries(options).filter(([key]) => key !== "typed_text")));
+});
+
test.each(["refused", "unknown"] as const)("native keyboard %s keeps its family without replaying input", async outcome => {
const f = await actionFixture(outcome);
try {
diff --git a/test/window-management.integration.test.ts b/test/window-management.integration.test.ts
index 988deb9..1e04d9e 100644
--- a/test/window-management.integration.test.ts
+++ b/test/window-management.integration.test.ts
@@ -33,7 +33,7 @@ for (const scenario of ["workflow", "admission", "receipts", "restore", "typed"]
expect(getToolResultText(result, "management-proof")).toContain(`IDE window management ${scenario} contract passed`);
const rows = (await readFile(path.join(cwd, "calls.jsonl"), "utf8")).trim().split("\n").filter(Boolean).map(line => JSON.parse(line));
const mutations = rows.filter(row => row.name === "manage_window");
- expect(mutations).toHaveLength({ workflow: 6, admission: 1, receipts: 16, restore: 3, typed: 3 }[scenario]);
+ expect(mutations).toHaveLength({ workflow: 5, admission: 1, receipts: 16, restore: 0, typed: 3 }[scenario]);
for (const { arguments: args } of mutations) {
const fields = args.operation === "move" ? ["x", "y", "unit"] : args.operation === "resize" ? ["width", "height", "unit"] : [];
expect(Object.keys(args).sort()).toEqual(["pid", "window_id", "state_id", "operation", ...fields].sort());
@@ -41,7 +41,7 @@ for (const scenario of ["workflow", "admission", "receipts", "restore", "typed"]
if (fields.length) expect(args.unit).toBe("windows_desktop_pixels");
}
if (scenario === "workflow") {
- expect(mutations.map(row => row.arguments.operation)).toEqual(["move", "resize", "focus", "maximize", "minimize", "move"]);
+ expect(mutations.map(row => row.arguments.operation)).toEqual(["move", "resize", "maximize", "minimize", "move"]);
expect(mutations[0].arguments).toMatchObject({ x: -20, y: 120 });
expect(mutations[1].arguments).toMatchObject({ width: 180, height: 120 });
}