Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.
TypeScript accessibilityagent-toolsai-agentbrowser-automationcomputer-usecuadesktop-automationelectronpipi-coding-agentpi-extensionpi-packageplaywrighttypescriptui-automationvisionwebview2windowswindows-automationwsl
Pin runtime f754b104: overviews of DPI-unaware windows fill the image and match their UI (LPT-907)
alexshpunt-clanker[bot]005cf46a3a6a
diff --git a/docs/bundled-runtime.md b/docs/bundled-runtime.mdindex 35f03a6..40cc929 100644--- a/docs/bundled-runtime.md+++ b/docs/bundled-runtime.md@@ -1,6 +1,6 @@# Bundled CUA runtime-The npm package carries one native runtime: Cua Driver 0.37.0 for Windows x64, built from the `pi-computer-use` branch of the [alexshpunt/cua](https://github.com/alexshpunt/cua) fork at source `0f170a4b`. [FORK.md](https://github.com/alexshpunt/cua/blob/pi-computer-use/FORK.md) lists what the fork adds to upstream Cua Driver. It is the qualified `windows-desktop` profile, so a fresh installation gets every native capability without any configuration:+The npm package carries one native runtime: Cua Driver 0.37.0 for Windows x64, built from the `pi-computer-use` branch of the [alexshpunt/cua](https://github.com/alexshpunt/cua) fork at source `f754b104`. [FORK.md](https://github.com/alexshpunt/cua/blob/pi-computer-use/FORK.md) lists what the fork adds to upstream Cua Driver. It is the qualified `windows-desktop` profile, so a fresh installation gets every native capability without any configuration:```shpi install npm:pi-agent-computer-use@@ -10,7 +10,7 @@ An update installs a new extension version with its exact runtime. Computer use## Where the files come from-The files are published as the fork release [`pi-runtime-0.37.0-0f170a4b`](https://github.com/alexshpunt/cua/releases/tag/pi-runtime-0.37.0-0f170a4b): `cua-driver.exe`, `cua-driver-uia.exe`, `cua-cursor-theme.exe`, `LICENSE.md` and `THIRD_PARTY_NOTICES.md`. They are byte-identical to the fork's CI build of that commit.+The files are published as the fork release [`pi-runtime-0.37.0-f754b104`](https://github.com/alexshpunt/cua/releases/tag/pi-runtime-0.37.0-f754b104): `cua-driver.exe`, `cua-driver-uia.exe`, `cua-cursor-theme.exe`, `LICENSE.md` and `THIRD_PARTY_NOTICES.md`. They are byte-identical to the fork's CI build of that commit.`src/runtime-pins.json` pins the source, the version, the release URL and the SHA-256 of every file. Before packing, `npm pack` runs `scripts/fetch-runtime.mjs`, which downloads the files into `native-runtime/win32-x64/runtime/` and checks each hash. The directory is not committed to Git.diff --git a/docs/computer-resources.md b/docs/computer-resources.mdindex ba6e186..3571493 100644--- a/docs/computer-resources.md+++ b/docs/computer-resources.md@@ -231,7 +231,7 @@ The `regions` view needs Cua's signed perception extension on the machine whose"lab": {"ssh": "lab-wsl","runtime": "windows-desktop",- "command": "/mnt/c/Users/user/AppData/Local/Programs/PiComputerUse/cua-driver-0.37.0-0f170a4b/runtime/cua-driver.exe",+ "command": "/mnt/c/Users/user/AppData/Local/Programs/PiComputerUse/cua-driver-0.37.0-f754b104/runtime/cua-driver.exe","node": "/mnt/c/Users/user/AppData/Local/Programs/PiComputerUse/node-v22.22.2-win-x64/node.exe","package": "/home/user/.local/share/pi-computer-use/package"}diff --git a/src/driver.ts b/src/driver.tsindex b6a6c88..d25e0e9 100644--- a/src/driver.ts+++ b/src/driver.ts@@ -86,8 +86,8 @@ export const WINDOW_RESTORE_RUNTIME = {} as const;/** Window management plus every earlier native capability; `windows-desktop` selects these bytes. Browser pages use Playwright, not this runtime. */export const WINDOW_MANAGEMENT_RUNTIME = {- source: "0f170a4b2d5f04c153a9e7ddad478fe5b730abf0", version: "0.37.0",- sha256: "fef81c7ac0a4412e9fa1d72ce802562933183f658ca2b0bc951084eb763b627e",+ source: "f754b1046ac1d6807f63365a8ebad10cb10e0de4", version: "0.37.0",+ sha256: "41143483eed9f16d8ea7a1c4f1b5ae13ad38918b490b4dea4c2bd89fa94e00e4",} as const;const forkRuntimes = {"windows-desktop": WINDOW_MANAGEMENT_RUNTIME,diff --git a/src/runtime-pins.json b/src/runtime-pins.jsonindex 86a45a6..3d4b9b5 100644--- a/src/runtime-pins.json+++ b/src/runtime-pins.json@@ -1,12 +1,12 @@{"format": 2,- "source": "0f170a4b2d5f04c153a9e7ddad478fe5b730abf0",+ "source": "f754b1046ac1d6807f63365a8ebad10cb10e0de4","version": "0.37.0",- "url": "https://github.com/alexshpunt/cua/releases/download/pi-runtime-0.37.0-0f170a4b/",+ "url": "https://github.com/alexshpunt/cua/releases/download/pi-runtime-0.37.0-f754b104/","files": {- "cua-driver.exe": "fef81c7ac0a4412e9fa1d72ce802562933183f658ca2b0bc951084eb763b627e",- "cua-driver-uia.exe": "6387b1b0e6da7c61e9d2fbc5c3dd512d9a25f535129d1d4764578c1f58b0162d",- "cua-cursor-theme.exe": "ebaec44ccd472e108fce2771b271288e5e22f1b6f3ce341cd9e42e5c124468d5",+ "cua-driver.exe": "41143483eed9f16d8ea7a1c4f1b5ae13ad38918b490b4dea4c2bd89fa94e00e4",+ "cua-driver-uia.exe": "85e8b6982fcd103306f48c21ef5ac0dae93fbf4fb13cbb91314b9e88ca98ac79",+ "cua-cursor-theme.exe": "eb8afb30ddae9a4d08855952c1fddb708ab7aba50f4734c67f463725c783d108","LICENSE.md": "d0a32419a44fa38d5023d1431dfcbedd1457eb5e7b2e6c7c87ca064facb41b41","THIRD_PARTY_NOTICES.md": "88a24f59293f03b3e03655e8144bcfadc9861bef1182cfeea91639e11f573fd1"}diff --git a/test/live/dpi-scaling.live.ts b/test/live/dpi-scaling.live.tsnew file mode 100644index 0000000..d96af35--- /dev/null+++ b/test/live/dpi-scaling.live.ts@@ -0,0 +1,53 @@+import sharp from "sharp";+import { afterAll, expect, test } from "vitest";+import { connectLive, nativeForm, openWindow, text, WINDOWS_POWERSHELL_PATH as POWERSHELL } from "./remote.ts";++// LPT-907: on a display scaled above 100%, Windows paints a DPI-unaware window at 96 DPI and stretches it on screen.+// Its overview must show that stretched window, so UI bounds, OCR boxes and pixel input agree with the image.+// Run it on a machine scaled above 100% (ZinPC: 150%).+const live = await connectLive();+const pids: number[] = [];+afterAll(async () => { for (const pid of pids) await live.kill(pid); await live.close(); });++type Box = { x: number; y: number; width: number; height: number };+type Element = { role?: string | null; name?: string | null; bounds?: { window_image?: Box } };+const box = (elements: Element[], match: (e: Element) => boolean): Box => {+ const found = elements.find(match)?.bounds?.window_image;+ if (!found) throw new Error(`No overview bounds in ${JSON.stringify(elements.map(e => [e.role, e.name]))}`);+ return found;+};+const centre = (b: Box) => ({ x: Math.round(b.x + b.width / 2), y: Math.round(b.y + b.height / 2) });++for (const dpiAware of [false, true]) test(`a DPI-${dpiAware ? "aware" : "unaware"} window's overview, UI bounds, OCR and pixel clicks agree`, async () => {+ const title = `DPI ${dpiAware ? "aware" : "unaware"} ${Date.now()}`;+ const window = await openWindow(live, live.machine, POWERSHELL, nativeForm(title, { dpiAware }), title);+ pids.push((JSON.parse(JSON.stringify((await live.search(live.windows, title)).details.state)).windows as { pid: number }[])[0].pid);+ const view = await live.read(window, ['capture:{"maxLongEdge":3840}', "full"]);+ const elements = view.details.ui!.elements as Element[];+ const button = box(elements, e => e.role === "Button" && e.name === "Count 0");+ const field = box(elements, e => e.role === "Edit" && e.name === "Field");++ // The text box is drawn where its UI bounds say: white across the middle of its box. A DPI-unaware window used+ // to be painted at 2/3 size in the image's top-left corner, so there the box showed the form's grey or black.+ const image = view.content.find(block => block.type === "image")!;+ const { data, info } = await sharp(Buffer.from(image.data, "base64")).greyscale().raw().toBuffer({ resolveWithObject: true });+ const row = Math.round(field.y + field.height / 2);+ const inside = [0.1, 0.3, 0.5, 0.7, 0.9].map(f => data[row * info.width + Math.round(field.x + field.width * f)]);+ console.log(title, { image: [info.width, info.height], button, field, inside });+ expect(Math.min(...inside), `text box row ${row}: ${inside}`).toBeGreaterThan(245);++ // OCR finds the button's caption inside the button's UI bounds. Its line can pick up the button's edge as "|",+ // and the stretched "0" can read as "O", so the word "Count" is compared.+ const ocr = await live.read(`computer:observation/${view.details.observation_id}`, ["ocr"]);+ const caption = ocr.details.text_lines!.flatMap(line => line.words).find(word => word.text === "Count");+ expect(caption, JSON.stringify(ocr.details.text_lines)).toBeTruthy();+ const at = centre(caption!);+ expect(at.x).toBeGreaterThan(button.x); expect(at.x).toBeLessThan(button.x + button.width);+ expect(at.y).toBeGreaterThan(button.y); expect(at.y).toBeLessThan(button.y + button.height);++ // A pixel click at the button's centre in the overview presses the button.+ const click = await live.insert(`computer:observation/${view.details.observation_id}`, { action: "click", ...centre(button), successor: false });+ expect(click.details.status, text(click)).toBe("delivered");+ const after = await live.read(window, ["ui"]);+ expect((after.details.ui!.elements as Element[]).some(e => e.role === "Button" && e.name === "Count 1"), text(after)).toBe(true);+});diff --git a/test/live/remote.ts b/test/live/remote.tsindex 3c41a1f..215b52e 100644--- a/test/live/remote.ts+++ b/test/live/remote.ts@@ -56,8 +56,13 @@ export async function connectLive(cwd = process.cwd()) {* PowerShell arguments for a disposable native Win32 target: a WinForms window with a counter button, a list box* and a text box behind the standard UIA proxies. Use it where Notepad is missing or would restore the human's tabs.*/-export function nativeForm(title: string): string[] {- const script = `Add-Type -AssemblyName System.Windows.Forms+export function nativeForm(title: string, { dpiAware = false } = {}): string[] {+ // Windows PowerShell is DPI-unaware: above 100% Windows paints its windows at 96 DPI and stretches them.+ // dpiAware switches the process to Per-Monitor V2 first, so the same form paints in physical pixels.+ const aware = dpiAware ? `Add-Type -Name D -Namespace W -MemberDefinition '[DllImport("user32.dll")] public static extern bool SetProcessDpiAwarenessContext(IntPtr v);'+if (-not [W.D]::SetProcessDpiAwarenessContext([IntPtr]-4)) { throw 'DPI awareness was not set' }+` : "";+ const script = `${aware}Add-Type -AssemblyName System.Windows.Forms$f = New-Object Windows.Forms.Form; $f.Text = '${title}'; $f.Width = 420; $f.Height = 300$b = New-Object Windows.Forms.Button; $b.Text = 'Count 0'; $b.Left = 10; $b.Top = 10; $b.Width = 120; $script:n = 0$b.Add_Click({ $script:n++; $b.Text = "Count $script:n" })