Skip to content

Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.

Docs: a press on a covered point refuses and names the covering window; runtime 2f1a8b1d (LPT-955)

alexshpunt-clanker[bot]449b418f9987
diff --git a/.pi/skills/computer-use/SKILL.md b/.pi/skills/computer-use/SKILL.md
index fa00471..484335a 100644
--- a/.pi/skills/computer-use/SKILL.md
+++ b/.pi/skills/computer-use/SKILL.md
@@ -31,7 +31,7 @@ Know the target, expected effect and stopping point before input. The input tool
- a ref from the window's latest Read;
- a point in pixels of the latest image of the window, for visible controls without UI, canvases and games.
-Reported disabled controls cannot act. A ref or locator is pressed through UI Automation in the background; when the element has no pattern for the action the call uses the real pointer and says so. Points, drags, keys and typing are real input: the window comes to the front. If a UI Automation press is delivered but the app does not react, click the point instead. If real input refuses because the human is using the computer, wait the stated time; never race the human. Never replace a refusal with global input, another backend or a browser script.
+Reported disabled controls cannot act. A ref or locator is pressed through UI Automation in the background; when the element has no pattern for the action the call uses the real pointer and says so. Points, drags, keys and typing are real input: the window comes to the front. If a UI Automation press is delivered but the app does not react, click the point instead. If real input refuses because the human is using the computer, wait the stated time; never race the human. If it refuses because another window covers the point, deal with the window it names first. Never replace a refusal with global input, another backend or a browser script.
For timed input, a button or key held across moves, wheel during a drag, raw relative motion for pointer-locked apps or a pen with pressure and tilt, use one `computer_timeline`. Stop at navigation, a new dialog, changed layout or an effect requiring a new decision. Several known steps go into one Codemode script that checks each receipt. Stop on an `unknown` outcome or cancellation; never replay completed or uncertain input.
diff --git a/docs/bundled-runtime.md b/docs/bundled-runtime.md
index 5741c1f..ad42415 100644
--- a/docs/bundled-runtime.md
+++ b/docs/bundled-runtime.md
@@ -1,6 +1,6 @@
# Bundled CUA runtime
-The npm package carries one native runtime: Cua Driver 0.37.0 for Windows x64, built from the `pi-computer-use` branch of the [alexshpunt/cua](https://github.com/alexshpunt/cua) fork at source `c920b66b`. [FORK.md](https://github.com/alexshpunt/cua/blob/pi-computer-use/FORK.md) lists what the fork adds to upstream Cua Driver. It is the qualified `windows-desktop` profile, so a fresh installation gets every native capability without any configuration:
+The npm package carries one native runtime: Cua Driver 0.37.0 for Windows x64, built from the `pi-computer-use` branch of the [alexshpunt/cua](https://github.com/alexshpunt/cua) fork at source `2f1a8b1d`. [FORK.md](https://github.com/alexshpunt/cua/blob/pi-computer-use/FORK.md) lists what the fork adds to upstream Cua Driver. It is the qualified `windows-desktop` profile, so a fresh installation gets every native capability without any configuration:
```sh
pi install npm:pi-agent-computer-use
@@ -10,7 +10,7 @@ An update installs a new extension version with its exact runtime. Computer use
## Where the files come from
-The files are published as the fork release [`pi-runtime-0.37.0-c920b66b`](https://github.com/alexshpunt/cua/releases/tag/pi-runtime-0.37.0-c920b66b): `cua-driver.exe`, `cua-driver-uia.exe`, `cua-cursor-theme.exe`, `LICENSE.md` and `THIRD_PARTY_NOTICES.md`. They are byte-identical to the fork's CI build of that commit.
+The files are published as the fork release [`pi-runtime-0.37.0-2f1a8b1d`](https://github.com/alexshpunt/cua/releases/tag/pi-runtime-0.37.0-2f1a8b1d): `cua-driver.exe`, `cua-driver-uia.exe`, `cua-cursor-theme.exe`, `LICENSE.md` and `THIRD_PARTY_NOTICES.md`. They are byte-identical to the fork's CI build of that commit.
`src/runtime-pins.json` pins the source, the version, the release URL and the SHA-256 of every file. Before packing, `npm pack` runs `scripts/fetch-runtime.mjs`, which downloads the files into `native-runtime/win32-x64/runtime/` and checks each hash. The directory is not committed to Git.
diff --git a/docs/computer-resources.md b/docs/computer-resources.md
index d7ada62..ab0eb4e 100644
--- a/docs/computer-resources.md
+++ b/docs/computer-resources.md
@@ -96,6 +96,8 @@ No input is posted to a window as messages, so a `delivered` receipt never stand
While the human uses the keyboard or mouse, real input pauses: it refuses with the time since their last input and when to retry, until 10 s pass without it. Never try to outrun the human. When Windows refuses to activate the target (`foreground_unavailable`), the result names the process that holds the foreground; if it is a system service, for example after a UAC prompt, Windows blocks injected input until a person clicks or presses a key on that machine, so ask the user.
+A real press goes to whichever window is on top at its point. After the target comes to the front, the driver checks the point of each click, drag start and timeline press. When another window still covers it, the action refuses before anything is pressed there and names that window with its address: a dialog the target owns (it stays in front of its owner even while the owner is active), another window of the same app, or a window of another app. Close or move that window, or act on it instead. The target's own menus and drop-downs take the press as usual.
+
Keys on a browser page go to the page, so browser shortcuts such as `alt+ArrowLeft` do not run there; use `computer_browser` to go back, or `space: "window"` to press them natively. Pen eraser and barrel, the `x1`/`x2` buttons and `move_relative` exist only natively. Pen pressure and tilt reach apps as real pen input; on a page they arrive as a `pen` pointer in Chrome and Edge. Paint 11 ignores pen pressure; GIMP uses it.
A refused action sent nothing. An `unknown` outcome may have reached the target: Read again and never replay it blindly. Delivery is not proof of the application's response.
@@ -210,7 +212,7 @@ To set one up:
"lab": {
"ssh": "lab-wsl",
"runtime": "windows-desktop",
- "command": "/mnt/c/Users/user/AppData/Local/Programs/PiComputerUse/cua-driver-0.37.0-c920b66b/runtime/cua-driver.exe",
+ "command": "/mnt/c/Users/user/AppData/Local/Programs/PiComputerUse/cua-driver-0.37.0-2f1a8b1d/runtime/cua-driver.exe",
"node": "/mnt/c/Users/user/AppData/Local/Programs/PiComputerUse/node-v22.22.2-win-x64/node.exe",
"package": "/home/user/.local/share/pi-computer-use/package"
}