Skip to content

Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.

The agent cursor jumps to each action point instead of gliding (LPT-916)

alexshpunt-clanker[bot]857dd97a39cb
diff --git a/.pi/skills/computer-use/SKILL.md b/.pi/skills/computer-use/SKILL.md
index aec4414..d255465 100644
--- a/.pi/skills/computer-use/SKILL.md
+++ b/.pi/skills/computer-use/SKILL.md
@@ -45,7 +45,7 @@ Read fresh pixels and UI after input and compare them with the expected effect.
Prefer a deterministic check over comparing screenshots: add `"expect":{"conditions":[...]}` to an action, or Insert `wait_for`/`verify` into the window address, and branch on the verdict. Conditions take the same locators, plus `count` (like toHaveCount) and `aria` (a Playwright aria snapshot template, like toMatchAriaSnapshot). Treat `unknown` as undecided, not as false. Read each action's change list before the next step: it is the cheapest proof that something happened. After an action on an `image` or `full` observation it also lists the changed pixel areas with before/after crops, which is how a stroke on a canvas or a moved sprite shows up. When it says nothing changed, check the target instead of repeating the action. Add `"successor":false` only to save the capture in a long run of actions whose effect you don't need, such as the strokes of a drawing; such pixel input then keeps one observation for the next call. In Codemode, branch on the receipt object's `outcome` without try/catch. For a short known chain, use one `batch`; give later native steps a `target` instead of a hint so each is resolved in a fresh UI walk. Page long text with `offset`/`limit` until the footer says it ended, and use `element:text` for one long native element.
-To react to something moving (a game, an animation), loop in Codemode: Read a small `image` overview (`maxLongEdge` 640), decide with pixel checks, or with a classifier model through `models.classify` in Codemode when pixel checks cannot decide (each call is a paid model call), and Insert one pixel action with `"animation":"reactive"` and `"successor":false` so the cursor does not glide first and no change list waits for the window to settle; each turn costs one capture and one input. Before the loop, set a deadline, frame, classification and action limits and a no-progress exit. Keep frames and classifier results inside the program and return only a small summary. Mark a clicked target pending and skip it until the app confirms it; a target can stay visible after it is done. To record a run for review, Insert `record_start`/`record_stop` into the session or machine address. For motion/progress, use a bounded sequence Read, inspect actual timestamps, skipped slots and stop reason, then explicitly Read selected frames. Sampling FPS is a goal. Partial/cancelled capture is not success. Frames are readonly; fresh window Read is required before input or detail.
+To react to something moving (a game, an animation), loop in Codemode: Read a small `image` overview (`maxLongEdge` 640), decide with pixel checks, or with a classifier model through `models.classify` in Codemode when pixel checks cannot decide (each call is a paid model call), and Insert one pixel action with `"successor":false` so no change list waits for the window to settle; each turn costs one capture and one input. Before the loop, set a deadline, frame, classification and action limits and a no-progress exit. Keep frames and classifier results inside the program and return only a small summary. Mark a clicked target pending and skip it until the app confirms it; a target can stay visible after it is done. To record a run for review, Insert `record_start`/`record_stop` into the session or machine address. For motion/progress, use a bounded sequence Read, inspect actual timestamps, skipped slots and stop reason, then explicitly Read selected frames. Sampling FPS is a goal. Partial/cancelled capture is not success. Frames are readonly; fresh window Read is required before input or detail.
Codemode may compose ordinary tools but is not required. Keep compact issued addresses/hints under short store aliases, not screenshots or whole results. Explicitly forward needed image blocks when composing; storage does not extend reference lifetime.
diff --git a/docs/complete-input.md b/docs/complete-input.md
index 8badc3a..1116089 100644
--- a/docs/complete-input.md
+++ b/docs/complete-input.md
@@ -4,7 +4,7 @@ See [the resource guide](computer-resources.md) for the current addresses, views
Send input as one Insert action into a current window overview. CUA remains the only input backend. A display capture or detail cannot ground input.
-For an explicit per-action short cursor glide, use `animation:"reactive"`. Ordinary animation is restored afterward; native click pauses do not change. See [reactive animation and local configuration](reactive-animation.md).
+CUA draws an agent cursor over the desktop and, by default, waits for it to glide to each new point before a pointer action: about 600 ms per 300 px. When the driver starts, the adapter sets that glide to 1 ms, so the agent cursor jumps to each point and still shows the click. A background click on a WinForms button then takes about 22 ms instead of 640 ms. The driver's own pauses between press and release stay: they are what makes double and right clicks take about 200 and 90 ms. If the driver refuses the setting or does not apply it, startup fails before any input.
## Single actions
diff --git a/docs/computer-resources.md b/docs/computer-resources.md
index 0ce3e6a..ba6e186 100644
--- a/docs/computer-resources.md
+++ b/docs/computer-resources.md
@@ -273,7 +273,7 @@ Choose frame values from the measured desktop geometry and the intended layout.
Move and resize require a normal, non-minimized/non-maximized window and request no activation or z-order change. Minimize and maximize may change foreground through Windows itself. Focus activates without restoring a minimized window; use a fresh state and explicit restore for that case. Maximize and focus require confirmed membership on the current virtual desktop. Unknown membership refuses; no operation switches desktops implicitly.
-Omit identity, hint, delivery, animation and sequence fields. A valid dispatch consumes state even on refusal. Read fresh state for another operation. The receipt retains attempted/accepted mutation, observed postcondition, requested/observed bounds and separate foreground/focus measurements. A constraint, cancellation, lost lifetime or incomplete readback stays unknown after possible submission. Never replay that prefix or simulate window controls with keys, clicks or pointer actions. Read a new overview to inspect paint after management.
+Omit identity, hint, delivery and sequence fields. A valid dispatch consumes state even on refusal. Read fresh state for another operation. The receipt retains attempted/accepted mutation, observed postcondition, requested/observed bounds and separate foreground/focus measurements. A constraint, cancellation, lost lifetime or incomplete readback stays unknown after possible submission. Never replay that prefix or simulate window controls with keys, clicks or pointer actions. Read a new overview to inspect paint after management.
## Input
diff --git a/docs/reactive-animation.md b/docs/reactive-animation.md
deleted file mode 100644
index 2634455..0000000
--- a/docs/reactive-animation.md
+++ /dev/null
@@ -1,29 +0,0 @@
-# Reactive cursor animation
-
-See [the resource guide](computer-resources.md) for the current addresses, views and commands.
-
-Ordinary actions keep CUA's current cursor animation. A bounded reactive program can opt into a short glide for one Insert action by adding `"animation":"reactive"` to its JSON.
-
-The adapter reads the current CUA glide, applies the configured reactive glide, checks native state, runs the original input command, then restores and checks the previous glide. Calls are serialized so an ordinary action cannot inherit another action's temporary profile. A sequence opts in once at the top level; each native step restores the ordinary profile before the next step.
-
-## Local configuration
-
-An optional trusted `.pi/reactive-animation.json` contains only:
-
-```json
-{"glide_duration_ms": 1}
-```
-
-The default is 1ms. The value must be an integer in 1..5000. Zero is rejected because CUA uses it for automatic timing, not instant motion. Extra settings, including click-delay settings, are rejected. Reload after changing this local file. No machine-wide setting or runtime pin changes.
-
-This controls decorative cursor glide only. Native click press/release and post-click waits stay unchanged (35ms and 50ms on the inspected Windows Chromium path). A 1ms glide does **not** mean a 1ms click. Identity checks, native transport, profile switching, delivery and scheduling still take time.
-
-The same option works on a configured machine and in a sandbox; their drivers use the configuration of the workspace that connects to them. It matters: by default CUA waits for the visible cursor to arrive at each new point before clicking. On a 4K test machine that made a click on a new spot take 650–930 ms, against about 160 ms with the 1 ms glide.
-
-## Receipts and failures
-
-Successful native receipts include `animation.profile`, `glide_duration_ms`, `previous_glide_duration_ms`, `restored:true` and `timing_ms` for apply, native call and restore. These are adapter wall-clock measurements, not physical-input or compositor timestamps. Include profile-switch overhead when judging reaction latency.
-
-A setter receipt alone does not prove the render thread applied the state. The adapter checks it with at most six native state reads before input and after restoration. A mismatch yields for 5ms before the next read (at most five yields), so the native render thread can apply its queued command. This bounded state-check cost is not a change to native click pauses. Cancellation cleanup does not reuse the cancelled action signal. If restoration cannot be confirmed, the adapter blocks subsequent calls until close and fresh discovery. Input may already have happened: do not replay it.
-
-Fresh observation, permissions, target identity and the normal background/foreground rules are unchanged. This option grants no input authority and adds no new backend. See [complete input](complete-input.md) and [the computer-use skill](../.pi/skills/computer-use/SKILL.md).
diff --git a/docs/semantic-actions.md b/docs/semantic-actions.md
index 740df82..e90d654 100644
--- a/docs/semantic-actions.md
+++ b/docs/semantic-actions.md
@@ -8,7 +8,7 @@ An Insert action accepts explicit `invoke`, `select` and `set_value` intents wit
- `select` calls UIA SelectionItem.Select, not Ctrl+A or a pointer click.
- `set_value` takes `text` (up to 2000 characters) and calls Value.SetValue, or RangeValue.SetValue when Value is unavailable before the call. An empty string is allowed for a text value.
-These actions accept no pixels, modifiers, foreground delivery or cursor animation. They are single actions, not sequence steps. UI-only observations are enough; pixel visibility is not a semantic prerequisite. A disabled control still refuses. Safe access, current observation, exact window lifetime and cancellation checks remain in force.
+These actions accept no pixels, modifiers or foreground delivery. They are single actions, not sequence steps. UI-only observations are enough; pixel visibility is not a semantic prerequisite. A disabled control still refuses. Safe access, current observation, exact window lifetime and cancellation checks remain in force.
The native tool admits one exact snapshot target and one UIA pattern. It has no mouse/key dispatch or foreground retry. Missing patterns, disabled/read-only controls and invalid range values refuse before the application call. Provider failure after a call starts is unknown; never repeat it blindly. Dispatch consumes the overview even on failure.
diff --git a/src/driver.ts b/src/driver.ts
index 9d6bf1e..c4adc6f 100644
--- a/src/driver.ts
+++ b/src/driver.ts
@@ -1,5 +1,6 @@
import { readFile, stat } from "node:fs/promises";
import { createHash } from "node:crypto";
+import { setTimeout as delay } from "node:timers/promises";
import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";
import type { Transport } from "@modelcontextprotocol/sdk/shared/transport.js";
@@ -15,6 +16,8 @@ import type { NativePlatform } from "./observation-source.ts";
/** Foreground input waits until the human has not used the keyboard or mouse for this long (owner choice, LPT-835). */
const HUMAN_IDLE_MS = 10_000;
+/** The agent cursor jumps to each action point instead of gliding there. */
+const CURSOR_GLIDE_MS = 1;
/** Exact Windows CI artifact qualified for held paths; selection is trusted config, never a tool argument. */
export const HELD_PATH_RUNTIME = {
source: "bba07a9c8ccba117c1057a3a0003e95d3693b2b3", version: "0.34.0",
@@ -217,6 +220,7 @@ export class McpDriver implements Driver {
if (!tools.some(tool => tool.name === name && tool.inputSchema.type === "object"))
throw new Error(`native_runtime_schema_missing: ${name}`);
}
+ await this.jumpCursor(client, signal);
return client;
} catch (error) {
await this.close();
@@ -224,6 +228,24 @@ export class McpDriver implements Driver {
}
}
+ /**
+ * Before each pointer action at a new point the driver waits for its agent cursor to glide there: about 600 ms
+ * per 300 px at its default speed. A 1 ms glide makes the cursor jump; 0 would select that speed-based glide.
+ * The setter only queues a render command, so its state is read back, yielding 5 ms between reads.
+ */
+ private async jumpCursor(client: Client, signal?: AbortSignal): Promise<void> {
+ const call = async (name: string, args: Record<string, unknown>) =>
+ CallToolResultSchema.parse(await client.callTool({ name, arguments: args }, undefined, { signal, timeout: 15_000 }));
+ if ((await call("set_agent_cursor_motion", { glide_duration_ms: CURSOR_GLIDE_MS })).isError)
+ throw new Error("The driver refused the agent cursor glide setting.");
+ for (let read = 0; read < 6; read++) {
+ const state = await call("get_agent_cursor_state", {});
+ if ((state.structuredContent?.motion as { glide_duration_ms?: unknown } | undefined)?.glide_duration_ms === CURSOR_GLIDE_MS) return;
+ await delay(5, undefined, { signal });
+ }
+ throw new Error("The driver did not apply the agent cursor glide setting.");
+ }
+
/**
* Variables for the native child, passed to Windows through WSLENV: no telemetry or update checks, a 10 s
* takeover pause (foreground input refuses while the human used the keyboard or mouse that recently), and launches
diff --git a/src/index.ts b/src/index.ts
index 300b7ea..c597ee0 100644
--- a/src/index.ts
+++ b/src/index.ts
@@ -2,7 +2,6 @@ import { registerVisionDebug } from "./vision-debug.ts";
import { CuaImageSettings } from "./cua-image-settings.ts";
import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
import { loadComputerConfig } from "./config.ts";
-import { ReactiveAnimationDriver, loadReactiveAnimationConfig } from "./reactive-animation.ts";
import { McpDriver } from "./driver.ts";
import { registerComputerMode } from "./mode.ts";
import { registerSetupCommand } from "./setup-command.ts";
@@ -31,7 +30,7 @@ export default async function computerUse(pi: ExtensionAPI): Promise<void> {
async function getSession(cwd: string): Promise<ComputerSession> {
if (session) return session;
if (!loading) loading = loadComputerConfig(cwd).then(async config => {
- const driver = new ReactiveAnimationDriver(new McpDriver(config), await loadReactiveAnimationConfig(cwd));
+ const driver = new McpDriver(config);
overviewKernel = driver.overviewKernel ?? "lanczos3";
session = new ComputerSession(driver, getMode, () => images.settings(), driver.platform === "windows" ? highlights.flash : undefined, image => debug.trace.retainOriginal(image),
driver.platform === "windows" ? (options, signal) => WebBrowser.launch({ ...options, interop: config.interop }, signal) : undefined,
diff --git a/src/machine.ts b/src/machine.ts
index 8f94cc9..a82ce3c 100644
--- a/src/machine.ts
+++ b/src/machine.ts
@@ -6,7 +6,6 @@ import { loadComputerConfig, type ComputerConfig } from "./config.ts";
import { ComputerRequestError, machineGuide, type MachineFactory, type SandboxFactory } from "./computer-resources.ts";
import type { CuaVisionSettings } from "./cua-image-settings.ts";
import { McpDriver } from "./driver.ts";
-import { ReactiveAnimationDriver, loadReactiveAnimationConfig } from "./reactive-animation.ts";
import { MachineHost } from "./machine-host.ts";
import { localRunner, sshRunner, type CommandRunner } from "./runner.ts";
import { WindowsSandbox, runnerHost, sandboxOptions, waitForGuestDriver } from "./sandbox.ts";
@@ -60,9 +59,8 @@ export function sandboxFactory(options: { settings: () => CuaVisionSettings; ret
const sandbox = parsed ? await WindowsSandbox.open(runnerHost(runner), machine?.command ?? config.command, parsed, signal)
: await IsolatedSession.open(runnerHost(runner), machine?.command ?? config.command, signal);
// The endpoint selects the guest transport; ssh only makes the runtime hash check read the machine's own bytes.
- // Reactive actions skip the visible cursor glide here too, as on this machine's own desktop.
- const driver = new ReactiveAnimationDriver(new McpDriver({ command: machine?.command ?? config.command, ...(machine?.runtime ?? config.runtime ? { runtime: machine?.runtime ?? config.runtime } : {}),
- endpoint: sandbox.endpoint, ...(machine ? { ssh: machine.ssh } : {}) }), await loadReactiveAnimationConfig(cwd));
+ const driver = new McpDriver({ command: machine?.command ?? config.command, ...(machine?.runtime ?? config.runtime ? { runtime: machine?.runtime ?? config.runtime } : {}),
+ endpoint: sandbox.endpoint, ...(machine ? { ssh: machine.ssh } : {}) });
try { await waitForGuestDriver(driver, signal); }
catch (error) { await driver.close().catch(() => undefined); await sandbox.close(); throw error; }
const guest = new ComputerSession(driver, () => "yolo", options.settings, undefined, options.retainRaw);
@@ -80,8 +78,7 @@ export function machineFactory(options: { getMode: () => ComputerMode; settings:
const machine = await machineConfig(cwd, name);
const refresh = `setup part machine (user: /computer-setup machine ${machine.ssh} ${name}; agent, after asking: Insert {"action":"setup","part":"machine","ssh":"${machine.ssh}","name":"${name}"} into computer:machine/local)`;
const runner = sshRunner(machine.ssh);
- const driver = new ReactiveAnimationDriver(new McpDriver({ command: machine.command, ...(machine.runtime ? { runtime: machine.runtime } : {}), ssh: machine.ssh }),
- await loadReactiveAnimationConfig(cwd));
+ const driver = new McpDriver({ command: machine.command, ...(machine.runtime ? { runtime: machine.runtime } : {}), ssh: machine.ssh });
try { await waitForGuestDriver(driver, signal, 30_000); }
catch (error) { await driver.close().catch(() => undefined); throw error; }
const launchWeb = machine.node && machine.package
diff --git a/src/reactive-animation.ts b/src/reactive-animation.ts
deleted file mode 100644
index fb57727..0000000
--- a/src/reactive-animation.ts
+++ /dev/null
@@ -1,124 +0,0 @@
-import { readFile } from "node:fs/promises";
-import path from "node:path";
-import { setTimeout as delay } from "node:timers/promises";
-import type { Driver, DriverResult, WindowTarget } from "./session.ts";
-
-/** Trusted local cursor-animation timing. Native mouse-event delays are not configurable here. */
-export interface ReactiveAnimationConfig { glide_duration_ms: number }
-
-/** Zero means automatic timing in CUA, so the minimal configured glide is a positive millisecond. */
-export async function loadReactiveAnimationConfig(cwd: string): Promise<ReactiveAnimationConfig> {
- let value: unknown;
- try { value = JSON.parse(await readFile(path.join(cwd, ".pi/reactive-animation.json"), "utf8")); }
- catch (error) {
- if ((error as NodeJS.ErrnoException).code === "ENOENT") return { glide_duration_ms: 1 };
- throw Error("Invalid .pi/reactive-animation.json; use one integer glide_duration_ms in 1..5000.");
- }
- const config = value as Partial<ReactiveAnimationConfig>;
- if (!config || typeof config !== "object" || Array.isArray(config) || Object.keys(config).length !== 1 ||
- !Number.isSafeInteger(config.glide_duration_ms) || config.glide_duration_ms! < 1 || config.glide_duration_ms! > 5000)
- throw Error("Reactive animation config accepts only integer glide_duration_ms in 1..5000.");
- return { glide_duration_ms: config.glide_duration_ms! };
-}
-
-/** Scope existing CUA animation controls to one explicit action, without changing native input delays. */
-export class ReactiveAnimationDriver implements Driver {
- readonly reactiveAnimation = true;
- private tail: Promise<unknown> = Promise.resolve();
- private closed = false;
- private uncertain = false;
- private generation = 0;
- constructor(private readonly base: Driver, private readonly config: ReactiveAnimationConfig) {}
- get platform() { return this.base.platform; }
- get sourceDomain() { return this.base.sourceDomain; }
- get windowLifetime() { return this.base.windowLifetime; }
- get heldPaths() { return this.base.heldPaths; }
- get heldKeys() { return this.base.heldKeys; }
- get selectedDisplays() { return this.base.selectedDisplays; }
- get captureBoundDrag() { return this.base.captureBoundDrag; }
- get semanticActions() { return this.base.semanticActions; }
- get nativePointer() { return this.base.nativePointer; }
- get coordinateScroll() { return this.base.coordinateScroll; }
- get windowRestore() { return this.base.windowRestore; }
- get windowManagement() { return this.base.windowManagement; }
- get overviewKernel() { return this.base.overviewKernel; }
-
- /** Identity and permissions remain with the existing native peer. */
- identity(target: WindowTarget, create: boolean, signal?: AbortSignal) {
- return this.base.identity(target, create, signal);
- }
-
- /** Serialize profile scopes, so normal calls cannot inherit a concurrent reactive animation. */
- async call(name: string, args: Record<string, unknown>, signal?: AbortSignal,
- options?: { animation?: "reactive" }): Promise<DriverResult> {
- const generation = this.generation;
- const run = this.tail.catch(() => undefined).then(async () => {
- if (generation !== this.generation) throw Error("Animation request was closed before execution.");
- if (this.closed || this.uncertain) throw Error("Animation state is unavailable; close and rediscover before further input.");
- signal?.throwIfAborted();
- if (!options?.animation) return this.base.call(name, args, signal);
- const started = performance.now();
- const before = await this.motion(signal);
- let result: DriverResult, nativeMs = 0, restoreMs = 0, applyMs = 0;
- try {
- await this.set(this.config.glide_duration_ms, signal);
- applyMs = performance.now() - started;
- signal?.throwIfAborted();
- this.assertOpen();
- const nativeStart = performance.now();
- result = await this.base.call(name, args, signal);
- nativeMs = performance.now() - nativeStart;
- } finally {
- const restoreStart = performance.now();
- try {
- if (this.closed) throw Error("Animation runtime was closed during input.");
- // Cleanup must not inherit the already-cancelled action signal.
- await this.set(before);
- restoreMs = performance.now() - restoreStart;
- } catch {
- this.uncertain = true;
- throw Error("Cursor animation restoration is unconfirmed. Input may have happened; do not replay. Close and rediscover.");
- }
- }
- return { ...result, structuredContent: { ...result.structuredContent,
- animation: { profile: "reactive", glide_duration_ms: this.config.glide_duration_ms, previous_glide_duration_ms: before, restored: true,
- timing_ms: { apply: applyMs, native: nativeMs, restore: restoreMs } } } };
- });
- this.tail = run;
- return run;
- }
-
- private assertOpen(): void {
- if (this.closed) throw Error("Animation runtime was closed; no further native calls are allowed.");
- }
-
- private async motion(signal?: AbortSignal): Promise<number> {
- this.assertOpen();
- const result = await this.base.call("get_agent_cursor_state", {}, signal);
- const motion = result.structuredContent?.motion as { glide_duration_ms?: unknown } | undefined;
- const value = motion?.glide_duration_ms;
- if (result.isError || typeof value !== "number" || !Number.isFinite(value) || value < 0 || value > 5000)
- throw Error("Native cursor animation state is unavailable; no profile input was sent.");
- return value;
- }
-
- private async set(glide: number, signal?: AbortSignal): Promise<void> {
- this.assertOpen();
- const result = await this.base.call("set_agent_cursor_motion", { glide_duration_ms: glide }, signal);
- if (result.isError) throw Error("Native cursor animation update was refused.");
- // The native setter queues a render command; its receipt is not an applied-state guarantee.
- for (let i = 0; i < 6; i++) {
- if (await this.motion(signal) === glide) return;
- if (i < 5) await delay(5, undefined, { signal });
- }
- throw Error("Native cursor animation did not apply within six bounded state reads.");
- }
-
- /** Release the owned runtime only. A closed profile cannot reconnect while cleanup is running. */
- async close(): Promise<void> {
- this.closed = true; this.generation++;
- await this.base.close();
- await this.tail.catch(() => undefined);
- this.uncertain = false; this.closed = false;
- }
-}
diff --git a/src/session.ts b/src/session.ts
index 3c49df6..ab95758 100644
--- a/src/session.ts
+++ b/src/session.ts
@@ -50,8 +50,6 @@ export interface Driver {
selectedDisplays?: boolean;
/** True only for a qualified native runtime that admits both straight-drag endpoints against one capture. */
captureBoundDrag?: boolean;
- /** True when existing native cursor controls can be scoped and restored for one action. */
- reactiveAnimation?: boolean;
/** True only for an exact qualified Windows runtime with UIA-only semantic dispatch. */
semanticActions?: boolean;
/** True only for the explicitly selected, byte-checked Windows native pointer runtime. */
@@ -64,7 +62,7 @@ export interface Driver {
windowManagement?: boolean;
/** Fixed overview filter of the selected native runtime; never a tool-controlled resize switch. */
overviewKernel?: CuaVisionSettings["kernel"];
- call(name: string, args: Record<string, unknown>, signal?: AbortSignal, options?: { animation?: "reactive" }): Promise<DriverResult>;
+ call(name: string, args: Record<string, unknown>, signal?: AbortSignal): Promise<DriverResult>;
identity(target: WindowTarget, create: boolean, signal?: AbortSignal): Promise<string | undefined>;
close(): Promise<void>;
}
@@ -73,8 +71,6 @@ export type Action = {
window_ref: string;
observation_id: string;
delivery_mode?: "background" | "foreground";
- /** Explicit opt-in to the trusted local reactive cursor-animation timing. */
- animation?: "reactive";
} & (InputCommand | { action: "sequence"; sequence: InputStep[] });
/** A stored Playwright snapshot of the active tab, with the tab list, any held dialog and events since the previous snapshot. */
export interface WebDom extends WebPage { truncated: boolean; matches?: number }
@@ -1539,13 +1535,11 @@ export class ComputerSession {
const change = await this.observationChange(entry, observed, signal).catch((error: unknown) => `the window state could not be read (${error instanceof Error ? error.message : error})`);
if (change) { entry.observation = undefined; return this.result("refused", `Observation expired: ${change}. Read the window again; no input was dispatched.`); }
}
- const { window_ref: _ref, observation_id: _id, delivery_mode, animation, ...command } = action;
+ const { window_ref: _ref, observation_id: _id, delivery_mode, ...command } = action;
const batch = command.action === "sequence";
let commands: InputStep[];
try {
if (delivery_mode !== undefined && !["background", "foreground"].includes(delivery_mode)) throw new Error("Unsupported delivery mode.");
- if (animation !== undefined && (animation !== "reactive" || !this.driver.reactiveAnimation))
- throw Error("Reactive animation requires the scoped native cursor adapter; no input was dispatched.");
if (batch) {
if (Object.keys(command).some(key => !["action", "sequence"].includes(key)) || !Array.isArray(command.sequence) || command.sequence.length < 1 || command.sequence.length > SEQUENCE_STEPS)
throw new Error(`A sequence needs 1..${SEQUENCE_STEPS} steps and no conflicting top-level action fields.`);
@@ -1570,18 +1564,16 @@ export class ComputerSession {
validateInput(nativeStep, observed, batch);
if (isCoordinateScroll(nativeStep)) {
if (!this.driver.coordinateScroll) throw Error("Coordinate scroll requires the explicitly selected native scroll runtime. No input was dispatched.");
- if (animation !== undefined) throw Error("Native coordinate scroll has no overlay animation.");
if (!observed.captureId) throw Error("Coordinate scroll needs this overview's native capture. Observe again.");
}
if (isPointerMove(nativeStep.action)) {
if (!this.driver.nativePointer) throw Error("Pointer movement requires the explicitly selected native pointer runtime. No input was dispatched.");
- if (animation !== undefined) throw Error("Native pointer movement has no overlay animation.");
if (nativeStep.element_token ? !observed.frames.has(nativeStep.element_token) : !observed.captureId)
throw Error("Pointer movement needs a current token with proven geometry or this overview's native capture. Observe again.");
}
if (isSemanticAction(nativeStep.action)) {
if (!this.driver.semanticActions) throw Error("Semantic actions require the explicitly selected qualified Windows semantic runtime. No input was dispatched.");
- if (delivery_mode === "foreground" || animation !== undefined) throw Error("Semantic actions use accessibility only: no foreground or cursor animation.");
+ if (delivery_mode === "foreground") throw Error("Semantic actions use accessibility only: no foreground delivery.");
}
return nativeStep;
});
@@ -1642,9 +1634,7 @@ export class ComputerSession {
// A sequence stops between steps: its current step finishes and releases its own keys and buttons,
// and the driver connection survives the cancellation. A single action stays cancellable in flight.
const nativeSignal = batch ? undefined : signal;
- const native = animation
- ? await this.driver.call(name, args, nativeSignal, { animation })
- : await this.driver.call(name, args, nativeSignal);
+ const native = await this.driver.call(name, args, nativeSignal);
row.completion_ms = performance.now() - started;
if (!this.allowed(entry) || (!batch && signal?.aborted)) return finish(this.result("unknown",
"Input was dispatched before permission was revoked or cancellation. Outcome withheld; native input may still have completed. Do not replay blindly."),
diff --git a/test/agent-cursor.test.ts b/test/agent-cursor.test.ts
new file mode 100644
index 0000000..9bd0b79
--- /dev/null
+++ b/test/agent-cursor.test.ts
@@ -0,0 +1,41 @@
+import { mkdtemp, readFile, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { afterEach, expect, test } from "vitest";
+import { McpDriver } from "../src/driver.ts";
+
+const dirs: string[] = [];
+afterEach(async () => { await Promise.all(dirs.splice(0).map(dir => rm(dir, { recursive: true, force: true }))); });
+
+async function start(mode: "applies" | "refuses" | "ignores") {
+ const dir = await mkdtemp(path.join(tmpdir(), "agent-cursor-"));
+ dirs.push(dir);
+ const log = path.join(dir, "calls.jsonl");
+ const driver = new McpDriver({ command: process.execPath, args: [path.resolve("test/fixtures/agent-cursor-driver.mjs"), log, mode] });
+ const calls = async () => (await readFile(log, "utf8")).trim().split("\n").map(line => JSON.parse(line) as { name: string; arguments: Record<string, unknown> });
+ return { driver, calls };
+}
+
+test("the driver makes the agent cursor jump and checks the setting before any other call", async () => {
+ const { driver, calls } = await start("applies");
+ try {
+ expect((await driver.call("list_windows", {})).isError).toBeFalsy();
+ expect(await calls()).toEqual([
+ { name: "set_agent_cursor_motion", arguments: { glide_duration_ms: 1 } },
+ { name: "get_agent_cursor_state", arguments: {} },
+ { name: "list_windows", arguments: {} },
+ ]);
+ } finally { await driver.close(); }
+});
+
+test("a refused cursor setting stops the driver before any input", async () => {
+ const { driver, calls } = await start("refuses");
+ await expect(driver.call("list_windows", {})).rejects.toThrow("refused the agent cursor glide setting");
+ expect((await calls()).map(call => call.name)).toEqual(["set_agent_cursor_motion"]);
+});
+
+test("a cursor setting that never applies stops the driver after six reads", async () => {
+ const { driver, calls } = await start("ignores");
+ await expect(driver.call("list_windows", {})).rejects.toThrow("did not apply the agent cursor glide setting");
+ expect((await calls()).map(call => call.name)).toEqual(["set_agent_cursor_motion", ...Array(6).fill("get_agent_cursor_state")]);
+});
diff --git a/test/bundled-driver.test.ts b/test/bundled-driver.test.ts
index 9948a0f..9974ee3 100644
--- a/test/bundled-driver.test.ts
+++ b/test/bundled-driver.test.ts
@@ -55,8 +55,8 @@ test("bundled startup checks envelopes/schema, forwards interop and display envi
structuredContent: { code: "unsupported_native_capability", platform: "windows", attempted: false, fallback: false } });
await backend.close();
const calls = (await readFile(log, "utf8")).trim().split("\n").map(line => JSON.parse(line));
- expect(calls.filter(call => call.tool).map(call => call.tool)).toEqual(["list_apps"]);
- expect(calls.filter(call => call.method).map(call => call.method)).toEqual(["initialize", "notifications/initialized", "tools/list", "tools/call"]);
+ expect(calls.filter(call => call.tool).map(call => call.tool)).toEqual(["set_agent_cursor_motion", "get_agent_cursor_state", "list_apps"]);
+ expect(calls.filter(call => call.method).map(call => call.method)).toEqual(["initialize", "notifications/initialized", "tools/list", "tools/call", "tools/call", "tools/call"]);
expect(calls[0].args).toEqual(["mcp", "--direct", "--embedded"]);
expect(calls.at(-1)).toEqual({ eof: true });
} finally {
diff --git a/test/coordinate-scroll.test.ts b/test/coordinate-scroll.test.ts
index af7b5a0..c22f87d 100644
--- a/test/coordinate-scroll.test.ts
+++ b/test/coordinate-scroll.test.ts
@@ -1,7 +1,6 @@
import sharp from "sharp";
import { expect, test, vi } from "vitest";
import { ComputerSession, type Action, type Driver } from "../src/session.ts";
-import { ReactiveAnimationDriver } from "../src/reactive-animation.ts";
import { McpDriver } from "../src/driver.ts";
test("older runtimes refuse coordinate wheel input before starting a native peer", async () => {
@@ -40,7 +39,7 @@ async function fixture() {
}),
};
Object.assign(base, { coordinateScroll: true });
- const driver = new ReactiveAnimationDriver(base, { glide_duration_ms: 1 });
+ const driver = base;
const session = new ComputerSession(driver, () => mode);
const windows = (await session.windows()).details.state!.windows as { window_ref: string }[];
const observe = (ui = false) => session.observe(windows[0].window_ref, undefined, undefined, undefined, undefined, ui ? { granularity: "ui" } : undefined);
diff --git a/test/driver-version.test.ts b/test/driver-version.test.ts
index 5961ce8..04c7578 100644
--- a/test/driver-version.test.ts
+++ b/test/driver-version.test.ts
@@ -3,18 +3,15 @@ import path from "node:path";
import { expect, test } from "vitest";
import { McpDriver } from "../src/driver.ts";
import type { BundledRuntime } from "../src/native-runtime.ts";
-import { ReactiveAnimationDriver } from "../src/reactive-animation.ts";
test("the desktop profile inherits every earlier window capability and checks its bytes before starting", async () => {
const backend = new McpDriver({ command: process.execPath, runtime: "windows-desktop" });
- const wrapped = new ReactiveAnimationDriver(backend, { glide_duration_ms: 1 });
try {
expect([backend.windowManagement, backend.windowRestore, backend.coordinateScroll, backend.nativePointer,
- backend.heldPaths, backend.heldKeys, backend.selectedDisplays, backend.captureBoundDrag, backend.semanticActions,
- wrapped.windowManagement]).toEqual(Array(10).fill(true));
+ backend.heldPaths, backend.heldKeys, backend.selectedDisplays, backend.captureBoundDrag, backend.semanticActions]).toEqual(Array(9).fill(true));
expect(backend.overviewKernel).toBe("bilinear");
await expect(backend.call("list_windows", {})).rejects.toThrow("artifact hash mismatch");
- } finally { await wrapped.close(); }
+ } finally { await backend.close(); }
});
test("management preserves inherited routes but rejects changed bytes before state acquisition", async () => {
const backend = new McpDriver({ command: process.execPath, runtime: "windows-window-management" });
diff --git a/test/fixtures/agent-cursor-driver.mjs b/test/fixtures/agent-cursor-driver.mjs
new file mode 100644
index 0000000..af1ed37
--- /dev/null
+++ b/test/fixtures/agent-cursor-driver.mjs
@@ -0,0 +1,20 @@
+import { appendFileSync } from "node:fs";
+import { Server } from "@modelcontextprotocol/sdk/server/index.js";
+import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";
+
+// Deterministic MCP peer for the startup cursor setting. Mode: applies, refuses, or ignores the glide.
+const [log, mode] = process.argv.slice(2);
+let glide = 0;
+const server = new Server({ name: "cua-driver", version: "0.33.2" }, { capabilities: { tools: {} } });
+server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: [] }));
+server.setRequestHandler(CallToolRequestSchema, async ({ params: { name, arguments: args = {} } }) => {
+ appendFileSync(log, JSON.stringify({ name, arguments: args }) + "\n");
+ if (name === "set_agent_cursor_motion") {
+ if (mode === "refuses") return { content: [{ type: "text", text: "refused" }], isError: true };
+ if (mode === "applies") glide = args.glide_duration_ms;
+ }
+ if (name === "get_agent_cursor_state") return { content: [], structuredContent: { motion: { glide_duration_ms: glide } } };
+ return { content: [], structuredContent: { windows: [] } };
+});
+await server.connect(new StdioServerTransport());
diff --git a/test/fixtures/agent-cursor.mjs b/test/fixtures/agent-cursor.mjs
new file mode 100644
index 0000000..6ba78aa
--- /dev/null
+++ b/test/fixtures/agent-cursor.mjs
@@ -0,0 +1,9 @@
+// The adapter sets the agent cursor glide when the driver starts. Fake peers keep that state without journaling it.
+let glide = 0;
+
+/** Answers the agent cursor tools, or returns undefined for any other tool. */
+export function agentCursor(name, args = {}) {
+ if (name === "set_agent_cursor_motion") glide = args.glide_duration_ms;
+ else if (name !== "get_agent_cursor_state") return undefined;
+ return { content: [], structuredContent: { motion: { glide_duration_ms: glide } } };
+}
diff --git a/test/fixtures/bundled-driver.mjs b/test/fixtures/bundled-driver.mjs
index 91ad0c6..432a018 100644
--- a/test/fixtures/bundled-driver.mjs
+++ b/test/fixtures/bundled-driver.mjs
@@ -5,7 +5,8 @@ import { createInterface } from "node:readline";
// Synthetic MCP peer for package startup/admission, not native GUI evidence.
const log = new URL("./calls.jsonl", import.meta.url);
const record = data => appendFileSync(log, JSON.stringify(data) + "\n");
-const names = ["list_apps", "list_windows", "get_window_state", "get_desktop_state"];
+const names = ["list_apps", "list_windows", "get_window_state", "get_desktop_state", "set_agent_cursor_motion", "get_agent_cursor_state"];
+let glide = 0;
createInterface({ input: process.stdin }).on("line", line => {
const request = JSON.parse(line);
record({ method: request.method, args: process.argv.slice(2) });
@@ -19,7 +20,10 @@ createInterface({ input: process.stdin }).on("line", line => {
else if (request.method === "tools/list") result = { tools: names.map(name => ({ name, inputSchema: { type: "object" } })) };
else if (request.method === "tools/call") {
record({ tool: request.params.name });
- result = { content: [], structuredContent: { apps: [], environment: {
+ if (request.params.name === "set_agent_cursor_motion") glide = request.params.arguments.glide_duration_ms;
+ result = request.params.name.includes("_agent_cursor_")
+ ? { content: [], structuredContent: { motion: { glide_duration_ms: glide } } }
+ : { content: [], structuredContent: { apps: [], environment: {
envelopes: process.env.CUA_DRIVER_MCP_ENVELOPES,
telemetry: process.env.CUA_DRIVER_RS_TELEMETRY_ENABLED,
display: process.env.DISPLAY,
diff --git a/test/fixtures/complete-input-driver.mjs b/test/fixtures/complete-input-driver.mjs
index 0f45c92..cc186cd 100644
--- a/test/fixtures/complete-input-driver.mjs
+++ b/test/fixtures/complete-input-driver.mjs
@@ -2,6 +2,7 @@ import { appendFileSync } from "node:fs";
import sharp from "sharp";
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { agentCursor } from "./agent-cursor.mjs";
import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";
// Deterministic MCP input receipts, not evidence of native application behavior.
@@ -14,6 +15,7 @@ const windows = [];
const server = new Server({ name: "cua-driver", version: "0.33.2" }, { capabilities: { tools: {} } });
server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: [] }));
server.setRequestHandler(CallToolRequestSchema, async ({ params: { name, arguments: args = {} } }) => {
+ const cursor = agentCursor(name, args); if (cursor) return cursor;
appendFileSync(log, JSON.stringify({ name, arguments: args }) + "\n");
let state; let pixels;
if (name === "fixture_identity") state = { identity: "owned-lifetime" };
diff --git a/test/fixtures/coordinate-scroll-driver.mjs b/test/fixtures/coordinate-scroll-driver.mjs
index 996504f..f3679be 100644
--- a/test/fixtures/coordinate-scroll-driver.mjs
+++ b/test/fixtures/coordinate-scroll-driver.mjs
@@ -2,6 +2,7 @@ import { appendFileSync } from "node:fs";
import sharp from "sharp";
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { agentCursor } from "./agent-cursor.mjs";
import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";
// Deterministic authority/receipt witnesses, not Windows application or native scroll evidence.
@@ -15,6 +16,7 @@ let launched = 0, snapshot = 0, response = "success";
const server = new Server({ name: "cua-driver", version: "0.33.2" }, { capabilities: { tools: {} } });
server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: [] }));
server.setRequestHandler(CallToolRequestSchema, async ({ params: { name, arguments: args = {} } }) => {
+ const cursor = agentCursor(name, args); if (cursor) return cursor;
appendFileSync(log, JSON.stringify({ name, arguments: args }) + "\n");
let state, pixels, mimeType = "image/png";
if (name === "fixture_identity") state = { identity: identities.get(args.window_id) };
diff --git a/test/fixtures/coordinate-scroll-extension.ts b/test/fixtures/coordinate-scroll-extension.ts
index 9c34e28..fcaf95d 100644
--- a/test/fixtures/coordinate-scroll-extension.ts
+++ b/test/fixtures/coordinate-scroll-extension.ts
@@ -3,7 +3,6 @@ import { Type } from "@earendil-works/pi-ai";
import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
import { loadComputerConfig } from "../../src/config.ts";
import { McpDriver } from "../../src/driver.ts";
-import { ReactiveAnimationDriver } from "../../src/reactive-animation.ts";
import { registerComputerProvider } from "../../src/provider.ts";
import { ComputerSession, type ComputerMode, type ComputerResult, type WindowTarget } from "../../src/session.ts";
@@ -45,7 +44,7 @@ export default async function fixture(pi: ExtensionAPI): Promise<void> {
log = config.args![1];
driver = new FixtureDriver(config);
}
- return session ??= new ComputerSession(new ReactiveAnimationDriver(driver, { glide_duration_ms: 1 }), () => mode);
+ return session ??= new ComputerSession(driver, () => mode);
};
const close = async (): Promise<ComputerResult> => session ? session.close() : {
content: [{ type: "text", text: "No synthetic runtime is running." }], details: { status: "closed", mode },
diff --git a/test/fixtures/detail-driver.mjs b/test/fixtures/detail-driver.mjs
index c1ca216..2eccd71 100644
--- a/test/fixtures/detail-driver.mjs
+++ b/test/fixtures/detail-driver.mjs
@@ -2,6 +2,7 @@ import { appendFileSync } from "node:fs";
import sharp from "sharp";
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { agentCursor } from "./agent-cursor.mjs";
import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";
// Real MCP boundary with synthetic pixels; this is not native Windows evidence.
@@ -13,6 +14,7 @@ const windows = [];
const server = new Server({ name: "cua-driver", version: process.argv[3] ?? "0.33.2" }, { capabilities: { tools: {} } });
server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: [] }));
server.setRequestHandler(CallToolRequestSchema, async ({ params: { name, arguments: args = {} } }) => {
+ const cursor = agentCursor(name, args); if (cursor) return cursor;
appendFileSync(log, JSON.stringify({ name, arguments: args }) + "\n");
let state; let bytes; let mimeType = "image/png";
if (name === "fixture_identity") state = { identity: "owned-lifetime" };
diff --git a/test/fixtures/fake-driver.mjs b/test/fixtures/fake-driver.mjs
index c986045..964792d 100644
--- a/test/fixtures/fake-driver.mjs
+++ b/test/fixtures/fake-driver.mjs
@@ -1,6 +1,7 @@
import { appendFileSync } from "node:fs";
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { agentCursor } from "./agent-cursor.mjs";
import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";
// Deterministic MCP peer for wrapper contracts, never native Windows qualification.
@@ -12,6 +13,7 @@ const windows = [];
const server = new Server({ name: "cua-driver", version: "0.33.2" }, { capabilities: { tools: {} } });
server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: [] }));
server.setRequestHandler(CallToolRequestSchema, async (request) => {
+ const cursor = agentCursor(request.params.name, request.params.arguments); if (cursor) return cursor;
appendFileSync(log, JSON.stringify(request.params) + "\n");
const { name, arguments: args } = request.params;
let state;
diff --git a/test/fixtures/ide-provider-driver.mjs b/test/fixtures/ide-provider-driver.mjs
index af2c6a6..f9fbf89 100644
--- a/test/fixtures/ide-provider-driver.mjs
+++ b/test/fixtures/ide-provider-driver.mjs
@@ -2,6 +2,7 @@ import { appendFileSync } from "node:fs";
import sharp from "sharp";
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { agentCursor } from "./agent-cursor.mjs";
import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";
// Synthetic native receipts only. This peer never opens or inputs to the real desktop.
@@ -16,6 +17,7 @@ let opened = 0, serial = 0, response = "success", field = "Initial fixture value
const server = new Server({ name: "cua-driver", version: "0.33.2" }, { capabilities: { tools: {} } });
server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: [] }));
server.setRequestHandler(CallToolRequestSchema, async ({ params: { name, arguments: args = {} } }) => {
+ const cursor = agentCursor(name, args); if (cursor) return cursor;
appendFileSync(log, JSON.stringify({ name, arguments: args }) + "\n");
let state, image, mimeType = "image/png";
if (name === "fixture_identity") state = { identity: identities.get(args.window_id) };
diff --git a/test/fixtures/observation-granularity-driver.mjs b/test/fixtures/observation-granularity-driver.mjs
index eef4436..244d8c4 100644
--- a/test/fixtures/observation-granularity-driver.mjs
+++ b/test/fixtures/observation-granularity-driver.mjs
@@ -2,6 +2,7 @@ import { appendFileSync } from "node:fs";
import sharp from "sharp";
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { agentCursor } from "./agent-cursor.mjs";
import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";
// This peer deliberately retains omitted native parts. The adapter must not inherit their authority.
@@ -11,6 +12,7 @@ const bounds = { x: 80, y: 90, width: 1000, height: 640 };
const server = new Server({ name: "cua-driver", version: "0.33.2" }, { capabilities: { tools: {} } });
server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: [] }));
server.setRequestHandler(CallToolRequestSchema, async ({ params: { name, arguments: args = {} } }) => {
+ const cursor = agentCursor(name, args); if (cursor) return cursor;
appendFileSync(log, JSON.stringify({ name, arguments: args }) + "\n");
let state; let image;
if (name === "fixture_identity") state = { identity: `owned:${args.window_id}` };
diff --git a/test/fixtures/pointer-hover-driver.mjs b/test/fixtures/pointer-hover-driver.mjs
index c1c2bf0..4fab171 100644
--- a/test/fixtures/pointer-hover-driver.mjs
+++ b/test/fixtures/pointer-hover-driver.mjs
@@ -2,6 +2,7 @@ import { appendFileSync } from "node:fs";
import sharp from "sharp";
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { agentCursor } from "./agent-cursor.mjs";
import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";
// Deterministic MCP receipts and authority checks, not Windows application evidence.
@@ -15,6 +16,7 @@ let launched = 0, snapshot = 0, response = "success";
const server = new Server({ name: "cua-driver", version: "0.33.2" }, { capabilities: { tools: {} } });
server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: [] }));
server.setRequestHandler(CallToolRequestSchema, async ({ params: { name, arguments: args = {} } }) => {
+ const cursor = agentCursor(name, args); if (cursor) return cursor;
appendFileSync(log, JSON.stringify({ name, arguments: args }) + "\n");
let state, pixels;
if (name === "fixture_identity") state = { identity: identities.get(args.window_id) };
diff --git a/test/fixtures/pointer-hover-extension.ts b/test/fixtures/pointer-hover-extension.ts
index 88c9f45..823da56 100644
--- a/test/fixtures/pointer-hover-extension.ts
+++ b/test/fixtures/pointer-hover-extension.ts
@@ -2,7 +2,6 @@ import { createCodemodeExtension, type ExtensionAPI } from "@earendil-works/pi-c
import { Type } from "@earendil-works/pi-ai";
import { loadComputerConfig } from "../../src/config.ts";
import { McpDriver } from "../../src/driver.ts";
-import { ReactiveAnimationDriver } from "../../src/reactive-animation.ts";
import { registerFixtureComputerTools } from "./standalone-computer.ts";
import { ComputerSession, type ComputerMode, type WindowTarget } from "../../src/session.ts";
@@ -21,7 +20,7 @@ export default function fixture(pi: ExtensionAPI): void {
let mode: ComputerMode = "yolo", session: ComputerSession | undefined, driver: FixtureDriver | undefined;
const getSession = async (cwd: string) => {
driver ??= new FixtureDriver(await loadComputerConfig(cwd));
- return session ??= new ComputerSession(new ReactiveAnimationDriver(driver, { glide_duration_ms: 1 }), () => mode);
+ return session ??= new ComputerSession(driver, () => mode);
};
registerFixtureComputerTools(pi, getSession, async () => session!.close());
pi.registerTool({
diff --git a/test/fixtures/portable-extension.ts b/test/fixtures/portable-extension.ts
index 9541c8e..cb83347 100644
--- a/test/fixtures/portable-extension.ts
+++ b/test/fixtures/portable-extension.ts
@@ -2,7 +2,6 @@ import { createCodemodeExtension, type ExtensionAPI } from "@earendil-works/pi-c
import { loadComputerConfig } from "../../src/config.ts";
import { McpDriver } from "../../src/driver.ts";
import { registerFixtureComputerTools } from "./standalone-computer.ts";
-import { ReactiveAnimationDriver } from "../../src/reactive-animation.ts";
import { ComputerSession } from "../../src/session.ts";
/** Exercise the common tools and real MCP child with deferred native lifetime, not a GUI platform simulation. */
@@ -10,7 +9,7 @@ export default function portableFixture(pi: ExtensionAPI): void {
createCodemodeExtension({ mode: "on" })(pi);
let session: ComputerSession | undefined;
registerFixtureComputerTools(pi, async cwd => session ??= new ComputerSession(
- new ReactiveAnimationDriver(new McpDriver(await loadComputerConfig(cwd)), { glide_duration_ms: 1 })),
+ new McpDriver(await loadComputerConfig(cwd))),
async () => session!.close());
pi.on("session_shutdown", async () => { await session?.close(); });
}
diff --git a/test/fixtures/reactive-animation-extension.ts b/test/fixtures/reactive-animation-extension.ts
deleted file mode 100644
index 5ac3d48..0000000
--- a/test/fixtures/reactive-animation-extension.ts
+++ /dev/null
@@ -1,40 +0,0 @@
-import { appendFile } from "node:fs/promises";
-import path from "node:path";
-import sharp from "sharp";
-import { createCodemodeExtension, type ExtensionAPI } from "@earendil-works/pi-coding-agent";
-import { registerFixtureComputerTools } from "./standalone-computer.ts";
-import { ComputerSession, type Driver } from "../../src/session.ts";
-import { ReactiveAnimationDriver, loadReactiveAnimationConfig } from "../../src/reactive-animation.ts";
-
-/** Real Pi/schema/session fixture. Native animation and physical input need a separate Windows check. */
-export default function reactiveAnimationFixture(pi: ExtensionAPI): void {
- createCodemodeExtension({ mode: "on" })(pi);
- let session: ComputerSession | undefined;
- registerFixtureComputerTools(pi, async cwd => {
- if (!session) {
- let glide = 240, count = 0;
- const png = (await sharp({ create: { width: 500, height: 350, channels: 3, background: "white" } }).png().toBuffer()).toString("base64");
- const base: Driver = {
- identity: async () => "PiComputerUse.0123456789abcdef0123456789abcdef:42:1", close: async () => {},
- async call(name, args) {
- await appendFile(path.join(cwd, "calls.jsonl"), JSON.stringify({ name, args, glide }) + "\n");
- if (name === "list_windows") return { content: [], structuredContent: { windows: [{ pid: 42, window_id: 100 }] } };
- if (name === "get_window_state") return { content: [{ type: "image", mimeType: "image/png", data: png }], structuredContent: {
- elements: [{ element_token: "counter", role: "Button", label: "Counter", frame: { x: 10, y: 10, width: 20, height: 20 } }],
- screenshot_width: 500, screenshot_height: 350, window_bounds: { x: 0, y: 0, width: 500, height: 350 },
- } };
- if (name === "get_agent_cursor_state") return { content: [], structuredContent: { motion: { glide_duration_ms: glide } } };
- if (name === "set_agent_cursor_motion") {
- glide = Number(args.glide_duration_ms);
- return { content: [], structuredContent: { motion: { glide_duration_ms: glide } } };
- }
- if (name === "click") return { content: [], structuredContent: { effect: "unverifiable", count: ++count } };
- throw Error("Unexpected fixture call");
- },
- };
- session = new ComputerSession(new ReactiveAnimationDriver(base, await loadReactiveAnimationConfig(cwd)));
- }
- return session;
- }, async () => session!.close());
- pi.on("session_shutdown", async () => { await session?.close(); });
-}
diff --git a/test/fixtures/selected-display-driver.mjs b/test/fixtures/selected-display-driver.mjs
index 88974bd..2850fe5 100644
--- a/test/fixtures/selected-display-driver.mjs
+++ b/test/fixtures/selected-display-driver.mjs
@@ -2,6 +2,7 @@ import { appendFileSync } from "node:fs";
import sharp from "sharp";
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { agentCursor } from "./agent-cursor.mjs";
import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";
// Synthetic native contract peer, never Windows evidence.
@@ -16,6 +17,7 @@ const pixels = await Promise.all(["blue", "green"].map(async background => {
const server = new Server({ name: "cua-driver", version: "0.33.2" }, { capabilities: { tools: {} } });
server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: [] }));
server.setRequestHandler(CallToolRequestSchema, async ({ params: { name, arguments: args = {} } }) => {
+ const cursor = agentCursor(name, args); if (cursor) return cursor;
appendFileSync(log, JSON.stringify({ name, arguments: args }) + "\n");
let state; let bytes;
if (name === "list_displays") state = { platform: "windows", topology_id: "topology-fixture", displays };
diff --git a/test/fixtures/shell-desktop-driver.mjs b/test/fixtures/shell-desktop-driver.mjs
index b4a10c8..c39baaf 100644
--- a/test/fixtures/shell-desktop-driver.mjs
+++ b/test/fixtures/shell-desktop-driver.mjs
@@ -2,6 +2,7 @@ import { appendFileSync } from "node:fs";
import sharp from "sharp";
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { agentCursor } from "./agent-cursor.mjs";
import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";
// Deterministic wire-contract peer, not evidence of Windows COM or live desktop behavior.
@@ -14,6 +15,7 @@ const pixels = (await sharp({ create: { width: 20, height: 20, channels: 3, back
const server = new Server({ name: "cua-driver", version: "0.33.2" }, { capabilities: { tools: {} } });
server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: [] }));
server.setRequestHandler(CallToolRequestSchema, async ({ params: { name, arguments: args = {} } }) => {
+ const cursor = agentCursor(name, args); if (cursor) return cursor;
appendFileSync(log, JSON.stringify({ name, arguments: args }) + "\n");
let state; let image = false;
if (name === "fixture_identity") state = { identity: "lifetime-" + args.window_id };
diff --git a/test/fixtures/ui-resources-driver.mjs b/test/fixtures/ui-resources-driver.mjs
index ac710ff..657380e 100644
--- a/test/fixtures/ui-resources-driver.mjs
+++ b/test/fixtures/ui-resources-driver.mjs
@@ -2,6 +2,7 @@ import { appendFileSync } from "node:fs";
import sharp from "sharp";
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { agentCursor } from "./agent-cursor.mjs";
import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";
// Deterministic MCP receipts; this is not native UI evidence.
@@ -12,6 +13,7 @@ const windows = [91, 92].map(window_id => ({ pid: 42, window_id, title: `Fixture
const server = new Server({ name: "cua-driver", version: "0.33.2" }, { capabilities: { tools: {} } });
server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: [] }));
server.setRequestHandler(CallToolRequestSchema, async ({ params: { name, arguments: args = {} } }) => {
+ const cursor = agentCursor(name, args); if (cursor) return cursor;
appendFileSync(log, JSON.stringify({ name, arguments: args }) + "\n");
let state; let pixels;
if (name === "fixture_identity") state = { identity: `owned:${args.window_id}` };
diff --git a/test/fixtures/window-capture-failure-driver.mjs b/test/fixtures/window-capture-failure-driver.mjs
index 1e84dbf..d3d159a 100644
--- a/test/fixtures/window-capture-failure-driver.mjs
+++ b/test/fixtures/window-capture-failure-driver.mjs
@@ -2,6 +2,7 @@ import { appendFileSync } from "node:fs";
import sharp from "sharp";
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { agentCursor } from "./agent-cursor.mjs";
import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";
// A real native-shaped partial success: UI was read, but the minimized window has no screenshot.
@@ -10,6 +11,7 @@ const png = await sharp({ create: { width: 100, height: 80, channels: 3, backgro
const server = new Server({ name: "cua-driver", version: "0.33.2" }, { capabilities: { tools: {} } });
server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: [] }));
server.setRequestHandler(CallToolRequestSchema, async ({ params: { name, arguments: args = {} } }) => {
+ const cursor = agentCursor(name, args); if (cursor) return cursor;
appendFileSync(log, JSON.stringify({ name, arguments: args }) + "\n");
let state; let image;
if (name === "fixture_identity") state = { identity: "owned window lifetime" };
diff --git a/test/fixtures/window-management-driver.mjs b/test/fixtures/window-management-driver.mjs
index 7d2e62b..09dd394 100644
--- a/test/fixtures/window-management-driver.mjs
+++ b/test/fixtures/window-management-driver.mjs
@@ -2,6 +2,7 @@ import { appendFileSync } from "node:fs";
import sharp from "sharp";
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { agentCursor } from "./agent-cursor.mjs";
import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";
// Synthetic MCP only: no Windows, pointer, keyboard or activation backend.
@@ -20,6 +21,7 @@ let opened = 0, serial = 0, outcome = "success";
const server = new Server({ name: "cua-driver", version: "0.33.2" }, { capabilities: { tools: {} } });
server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: [] }));
server.setRequestHandler(CallToolRequestSchema, async ({ params: { name, arguments: args = {} } }) => {
+ const cursor = agentCursor(name, args); if (cursor) return cursor;
appendFileSync(log, JSON.stringify({ name, arguments: args }) + "\n");
let state, isError = false, image;
const current = () => { if (!identities.has(args.window_id)) throw Error("Native query reached a closed source"); return samples.get(args.window_id); };
diff --git a/test/fixtures/window-management-extension.ts b/test/fixtures/window-management-extension.ts
index 7e4290b..d330a8c 100644
--- a/test/fixtures/window-management-extension.ts
+++ b/test/fixtures/window-management-extension.ts
@@ -28,8 +28,8 @@ function semantic(result: Result): Record<string, unknown> {
export default async function fixture(pi: ExtensionAPI): Promise<void> {
const source = process.env.PI_WINDOW_MANAGEMENT_PROVIDER_SOURCE ?? path.resolve(import.meta.dirname, "../..");
const load = (name: string) => import(pathToFileURL(path.join(source, "src", name)).href);
- const [{ registerComputerProvider }, { loadComputerConfig }, { McpDriver }, { ComputerSession }, { ReactiveAnimationDriver }] =
- await Promise.all([load("provider.ts"), load("config.ts"), load("driver.ts"), load("session.ts"), load("reactive-animation.ts")]);
+ const [{ registerComputerProvider }, { loadComputerConfig }, { McpDriver }, { ComputerSession }] =
+ await Promise.all([load("provider.ts"), load("config.ts"), load("driver.ts"), load("session.ts")]);
const Base = McpDriver as typeof import("../../src/driver.ts").McpDriver;
let management = true, restoration = true, mode: ComputerMode = "yolo";
let session: ComputerSession | undefined, driver: FixtureDriver | undefined, log: string | undefined;
@@ -52,7 +52,7 @@ export default async function fixture(pi: ExtensionAPI): Promise<void> {
const getSession = async (cwd: string): Promise<ComputerSession> => {
if (!driver) { const config = await loadComputerConfig(cwd); log = config.args[1]; driver = new FixtureDriver(config); }
// Match production wrapping; management/restore capabilities and native calls must survive it unchanged.
- session ??= new ComputerSession(new ReactiveAnimationDriver(driver, { glide_duration_ms: 1 }), () => mode); return session!;
+ session ??= new ComputerSession(driver, () => mode); return session!;
};
const close = async (): Promise<ComputerResult> => session ? session.close() : {
content: [{ type: "text", text: "No synthetic runtime is running." }], details: { status: "closed", mode },
diff --git a/test/fixtures/window-restore-driver.mjs b/test/fixtures/window-restore-driver.mjs
index 76bdbc3..5892335 100644
--- a/test/fixtures/window-restore-driver.mjs
+++ b/test/fixtures/window-restore-driver.mjs
@@ -2,6 +2,7 @@ import { appendFileSync } from "node:fs";
import sharp from "sharp";
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { agentCursor } from "./agent-cursor.mjs";
import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";
// Synthetic native peer only; no real window, activation, pointer or keyboard effects.
@@ -14,6 +15,7 @@ let opened = 0, serial = 0, outcome = "success";
const server = new Server({ name: "cua-driver", version: "0.33.2" }, { capabilities: { tools: {} } });
server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: [] }));
server.setRequestHandler(CallToolRequestSchema, async ({ params: { name, arguments: args = {} } }) => {
+ const cursor = agentCursor(name, args); if (cursor) return cursor;
appendFileSync(log, JSON.stringify({ name, arguments: args }) + "\n");
let state, isError = false, image;
if (name === "fixture_identity") state = { identity: identities.get(args.window_id) };
diff --git a/test/pointer-hover.test.ts b/test/pointer-hover.test.ts
index a7416c2..381e1db 100644
--- a/test/pointer-hover.test.ts
+++ b/test/pointer-hover.test.ts
@@ -70,7 +70,7 @@ test.each([
{ action: "move", x: 10, y: 10, animation: "reactive" },
{ action: "sequence", sequence: [{ action: "key", key: "a" }, { action: "move", x: 10, y: 10 }] },
])("unsupported pointer command cannot send any batch input: %j", async command => {
- const f = await fixture(); Object.assign(f.driver, { reactiveAnimation: true });
+ const f = await fixture();
expect((await f.send(command)).details.status).toBe("refused"); expect(f.inputs()).toHaveLength(0);
});
diff --git a/test/portable-session.test.ts b/test/portable-session.test.ts
index bb2d577..7224c13 100644
--- a/test/portable-session.test.ts
+++ b/test/portable-session.test.ts
@@ -1,7 +1,6 @@
import { expect, test, vi } from "vitest";
import { ComputerSession, type Driver } from "../src/session.ts";
import { observationSource } from "../src/observation-source.ts";
-import { ReactiveAnimationDriver } from "../src/reactive-animation.ts";
function backend(platform: "linux" | "macos") {
const call = vi.fn(async (name: string) => ({ content: [], structuredContent: name === "list_apps"
@@ -9,8 +8,7 @@ function backend(platform: "linux" | "macos") {
: { windows: [{ pid: 42, window_id: 99, title: "Owned test marker" }] } }));
const identity = vi.fn(async () => undefined);
const base: Driver = { platform, windowLifetime: false, call, identity, close: vi.fn(async () => {}) };
- const driver = new ReactiveAnimationDriver(base, { glide_duration_ms: 1 });
- return { driver, call, identity };
+ return { driver: base, call, identity };
}
test.each(["linux", "macos"] as const)("%s exposes metadata but never substitutes reusable IDs for native lifetime", async platform => {
diff --git a/test/reactive-animation.integration.test.ts b/test/reactive-animation.integration.test.ts
deleted file mode 100644
index d8f90ab..0000000
--- a/test/reactive-animation.integration.test.ts
+++ /dev/null
@@ -1,55 +0,0 @@
-import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
-import path from "node:path";
-import { assistantMessage, getToolExecution, getToolExecutions, getToolResultText, PiIntegrationTest,
- testArtifactsDir, text, toolCall } from "pi-coding-agent-test";
-import { expect, test } from "vitest";
-import { nestedCalls, SCRIPT_HELPERS } from "./fixtures/standalone-computer.ts";
-
-test("real Pi scopes configured animation to explicit input and restores normal input across close", async () => {
- await mkdir(".tmp/reactive-animation", { recursive: true });
- const cwd = await mkdtemp(path.resolve(".tmp/reactive-animation/pi-"));
- await mkdir(path.join(cwd, ".pi"));
- await writeFile(path.join(cwd, ".pi/reactive-animation.json"), JSON.stringify({ glide_duration_ms: 2 }));
- const code = `${SCRIPT_HELPERS}
-let win, view, counter;
-const discover = async () => { win = address(await windows(), "window"); };
-const observe = async () => {
- const read = await tools.computer_read({path:win});
- view = observation(read);
- counter = body(read).split("\\n").find(line => line.includes("| Counter |")).split(" | ")[0];
-};
-const send = extra => insert(view, {action:"click",hint:counter,...extra});
-await discover(); await observe();
-const reactive = await send({animation:"reactive"});
-if(reactive.outcome !== "delivered") throw Error("Reactive scope lost");
-if((await send({animation:"reactive"})).outcome !== "refused") throw Error("Consumed observation reused");
-await observe();
-const ordinary = await send({});
-if(ordinary.outcome !== "delivered") throw Error("Ordinary scope changed");
-await insert("computer:session",{action:"close"}); await discover(); await observe();
-const fresh = await send({});
-if(fresh.outcome !== "delivered") throw Error("Close blocked fresh ordinary action");
-text("reactive, ordinary, consumed_refused, fresh_after_close");`;
- try {
- const result = await new PiIntegrationTest({
- testName: "reactive-animation", artifactsDir: testArtifactsDir(import.meta.filename), cwd,
- extensions: [path.resolve("test/fixtures/reactive-animation-extension.ts")], isolateUserResources: true,
- tools: ["codemode", "computer_insert"],
- conversation: [
- assistantMessage([toolCall({ id: "animation", name: "codemode", arguments: { code } })], { stopReason: "toolUse" }),
- assistantMessage([toolCall({ id: "close", name: "computer_insert", arguments: { path: "computer:session", text: '{"action":"close"}' } })], { stopReason: "toolUse" }),
- assistantMessage([text("Done")]),
- ],
- }).run("Use explicit animation only for reactive input. Do not replay a consumed observation.");
- expect(getToolExecution(result, "animation").isError, getToolResultText(result, "animation")).toBe(false);
- expect(getToolResultText(result, "animation")).toContain("fresh_after_close");
- const clicks = nestedCalls(getToolExecutions(result), "animation").filter(call => call.computer.status === "delivered" && call.computer.state?.count !== undefined);
- expect(clicks).toHaveLength(3);
- expect(clicks[0].computer.state.animation).toMatchObject({ glide_duration_ms: 2, restored: true });
- expect(clicks.slice(1).map(call => call.computer.state.animation)).toEqual([undefined, undefined]);
- const calls = (await readFile(path.join(cwd, "calls.jsonl"), "utf8")).trim().split("\n").map(line => JSON.parse(line));
- expect(calls.filter(call => call.name === "click").map(call => call.glide)).toEqual([2, 240, 240]);
- expect(calls.filter(call => call.name === "set_agent_cursor_motion").map(call => call.args.glide_duration_ms)).toEqual([2, 240]);
- expect(calls.filter(call => call.name === "click").every(call => call.args.animation === undefined)).toBe(true);
- } finally { await rm(cwd, { recursive: true, force: true }); }
-});
diff --git a/test/reactive-animation.test.ts b/test/reactive-animation.test.ts
deleted file mode 100644
index 0258290..0000000
--- a/test/reactive-animation.test.ts
+++ /dev/null
@@ -1,157 +0,0 @@
-import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
-import path from "node:path";
-import { expect, test } from "vitest";
-import { ReactiveAnimationDriver, loadReactiveAnimationConfig } from "../src/reactive-animation.ts";
-import type { Driver, DriverResult } from "../src/session.ts";
-
-function fixture() {
- let glide = 240, applied = false;
- const calls: { name: string; args: Record<string, unknown>; signal?: AbortSignal; glide: number }[] = [];
- const control = { failRestore: false, refuse: false, abort: undefined as AbortController | undefined, stale: false };
- const base: Driver = {
- heldPaths: true, heldKeys: true, selectedDisplays: true, captureBoundDrag: true,
- identity: async () => "owned", close: async () => {},
- async call(name, args, signal): Promise<DriverResult> {
- calls.push({ name, args, signal, glide });
- if (name === "get_agent_cursor_state") return { content: [], structuredContent: { session: "owned", motion: { glide_duration_ms: glide } } };
- if (name === "set_agent_cursor_motion") {
- if (applied && args.glide_duration_ms === 240 && control.failRestore) throw Error("Restore failed");
- if (!control.stale) glide = Number(args.glide_duration_ms);
- applied = true;
- return { content: [], structuredContent: { motion: { glide_duration_ms: glide } } };
- }
- control.abort?.abort();
- if (signal?.aborted) throw Error("Native request aborted");
- return { content: [], structuredContent: { effect: control.refuse ? "refused" : "unverifiable" }, isError: control.refuse };
- },
- };
- return { driver: new ReactiveAnimationDriver(base, { glide_duration_ms: 1 }), calls, control, glide: () => glide };
-}
-
-// Remote machines and sandboxes wrap their drivers too (LPT-702); their sources must keep naming their domain.
-test("the wrapper keeps the wrapped driver's source domain", () => {
- for (const domain of ["windows-remote", "windows-sandbox", undefined] as const) {
- const base = { sourceDomain: domain, identity: async () => "owned", close: async () => {}, call: async () => ({ content: [] }) } as Driver;
- expect(new ReactiveAnimationDriver(base, { glide_duration_ms: 1 }).sourceDomain).toBe(domain);
- }
-});
-test("ordinary calls do not change animation or add cursor requests", async () => {
- const f = fixture();
- await f.driver.call("click", { target: "fixture" });
- expect(f.calls.map(c => c.name)).toEqual(["click"]);
- expect(f.glide()).toBe(240);
- expect([f.driver.heldPaths, f.driver.heldKeys, f.driver.selectedDisplays, f.driver.captureBoundDrag]).toEqual([true, true, true, true]);
-});
-
-test("reactive calls use configured glide then restore the exact ordinary value", async () => {
- const f = fixture();
- const result = await f.driver.call("click", { target: "fixture" }, undefined, { animation: "reactive" });
- expect(f.calls.filter(c => c.name === "click")).toMatchObject([{ glide: 1, args: { target: "fixture" } }]);
- expect(f.calls.filter(c => c.name === "set_agent_cursor_motion").map(c => c.args)).toEqual([
- { glide_duration_ms: 1 }, { glide_duration_ms: 240 },
- ]);
- expect(f.glide()).toBe(240);
- expect(result.structuredContent).toMatchObject({ animation: { profile: "reactive", glide_duration_ms: 1, restored: true } });
- await f.driver.call("click", {});
- expect(f.calls.at(-1)?.glide).toBe(240);
-});
-
-test("native refusal and cancellation restore animation without replaying input", async () => {
- for (const cancel of [false, true]) {
- const f = fixture(), controller = new AbortController();
- if (cancel) f.control.abort = controller; else f.control.refuse = true;
- const call = f.driver.call("click", {}, controller.signal, { animation: "reactive" });
- if (cancel) await expect(call).rejects.toThrow(/abort/i); else expect((await call).isError).toBe(true);
- expect(f.glide()).toBe(240);
- expect(f.calls.filter(c => c.name === "click")).toHaveLength(1);
- expect(f.calls.filter(c => c.name === "set_agent_cursor_motion").at(-1)?.signal).toBeUndefined();
- }
-});
-
-test("unapplied motion stops before input and failed restoration blocks later calls", async () => {
- const stale = fixture(); stale.control.stale = true;
- await expect(stale.driver.call("click", {}, undefined, { animation: "reactive" })).rejects.toThrow(/animation/i);
- expect(stale.calls.filter(c => c.name === "click")).toHaveLength(0);
- const failed = fixture(); failed.control.failRestore = true;
- await expect(failed.driver.call("click", {}, undefined, { animation: "reactive" })).rejects.toThrow(/restor/i);
- const before = failed.calls.length;
- await expect(failed.driver.call("click", {})).rejects.toThrow(/animation/i);
- expect(failed.calls).toHaveLength(before);
-});
-
-test("concurrent normal input cannot run inside another call's reactive profile", async () => {
- const f = fixture();
- await Promise.all([
- f.driver.call("click", { id: 1 }, undefined, { animation: "reactive" }),
- f.driver.call("click", { id: 2 }),
- ]);
- expect(f.calls.filter(c => c.name === "click").map(c => c.glide)).toEqual([1, 240]);
-});
-
-test("queued native motion is read back after a render turn before input and after restore", async () => {
- let glide = 240;
- const inputGlides: number[] = [];
- const base: Driver = {
- identity: async () => "owned", close: async () => {},
- async call(name, args) {
- if (name === "set_agent_cursor_motion") {
- setTimeout(() => { glide = Number(args.glide_duration_ms); }, 16);
- return { content: [], structuredContent: { motion: { glide_duration_ms: args.glide_duration_ms } } };
- }
- if (name === "get_agent_cursor_state") return { content: [], structuredContent: { motion: { glide_duration_ms: glide } } };
- inputGlides.push(glide); return { content: [], structuredContent: { effect: "unverifiable" } };
- },
- };
- const driver = new ReactiveAnimationDriver(base, { glide_duration_ms: 1 });
- try {
- await driver.call("click", {}, undefined, { animation: "reactive" });
- expect(inputGlides).toEqual([1]); expect(glide).toBe(240);
- } finally { await driver.close(); }
-});
-
-test("close during profile switching cannot reconnect native controls", async () => {
- let closed = false, notifyRead!: () => void;
- const firstMismatch = new Promise<void>(resolve => { notifyRead = resolve; });
- const afterClose: string[] = [];
- const base: Driver = {
- identity: async () => "owned", close: async () => { closed = true; },
- async call(name) {
- if (closed) afterClose.push(name);
- if (name === "get_agent_cursor_state") {
- notifyRead(); return { content: [], structuredContent: { motion: { glide_duration_ms: 240 } } };
- }
- return { content: [], structuredContent: {} };
- },
- };
- const driver = new ReactiveAnimationDriver(base, { glide_duration_ms: 1 });
- const outcome = Promise.allSettled([driver.call("click", {}, undefined, { animation: "reactive" })]);
- await firstMismatch; await driver.close();
- expect((await outcome)[0].status).toBe("rejected"); expect(afterClose).toEqual([]);
-});
-
-test("close rejects queued profile work but permits a later fresh runtime", async () => {
- const f = fixture();
- const queued = f.driver.call("click", {}, undefined, { animation: "reactive" });
- const outcome = Promise.allSettled([queued]);
- await f.driver.close();
- expect((await outcome)[0].status).toBe("rejected");
- expect(f.calls).toHaveLength(0);
- await f.driver.call("click", {});
- expect(f.calls.map(c => c.name)).toEqual(["click"]);
-});
-
-test("animation setup defaults to1ms and invalid settings fail instead of selecting automatic timing", async () => {
- await mkdir(".tmp/reactive-animation", { recursive: true });
- const dir = await mkdtemp(path.resolve(".tmp/reactive-animation/config-"));
- try {
- expect(await loadReactiveAnimationConfig(dir)).toEqual({ glide_duration_ms: 1 });
- await mkdir(path.join(dir, ".pi"));
- const file = path.join(dir, ".pi/reactive-animation.json");
- for (const value of [{ glide_duration_ms: 0 }, { glide_duration_ms: -1 }, { glide_duration_ms: .5 }, { glide_duration_ms: 5001 }, { glide_duration_ms: 1, click_delay_ms: 0 }]) {
- await writeFile(file, JSON.stringify(value));
- await expect(loadReactiveAnimationConfig(dir)).rejects.toThrow(/animation/i);
- }
- await writeFile(file, JSON.stringify({ glide_duration_ms: 2 }));
- expect(await loadReactiveAnimationConfig(dir)).toEqual({ glide_duration_ms: 2 });
- } finally { await rm(dir, { recursive: true, force: true }); }
-});
diff --git a/test/window-restore.test.ts b/test/window-restore.test.ts
index c24a0a0..e77e8f0 100644
--- a/test/window-restore.test.ts
+++ b/test/window-restore.test.ts
@@ -2,7 +2,6 @@ import { expect, test, vi } from "vitest";
import { ComputerSession, type ComputerMode, type Driver, type DriverResult } from "../src/session.ts";
import { ComputerResources, requireComputerAddress } from "../src/computer-resources.ts";
import { computerSource } from "../src/computer-address.ts";
-import { ReactiveAnimationDriver } from "../src/reactive-animation.ts";
async function fixture() {
let mode: ComputerMode = "yolo", life: string | undefined = "life", serial = 0;
@@ -55,21 +54,6 @@ async function fixture() {
outcome(value: Record<string, unknown>, isError = false) { receipt = value; error = isError; } };
}
-test("production animation wrapper preserves restore capability and dispatches no animation controls", async () => {
- const f = await fixture();
- vi.mocked(f.driver.call).mockClear();
- const wrapper = new ReactiveAnimationDriver(f.driver, { glide_duration_ms: 1 });
- const session = new ComputerSession(wrapper, () => "yolo");
- const window = (await session.windows()).details.state!.windows as { window_ref: string }[];
- const view = await session.observeWindowState(window[0].window_ref);
- expect(view.details.status).toBe("observed");
- const result = await session.restoreWindow(window[0].window_ref, view.details.window_state_id!);
- expect(result.details.status).toBe("delivered");
- expect(f.calls().map(([name]) => name)).toEqual(["list_windows", "get_window_status", "restore_window"]);
- f.setCapability(false);
- expect(wrapper.windowRestore).toBe(false);
- await session.close();
-});
test("fresh metadata-only Read authorizes one exact restore without acquiring pixels or UI", async () => {
const f = await fixture(), view = await f.read();
const before = f.calls().length;