Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.
TypeScript accessibilityagent-toolsai-agentbrowser-automationcomputer-usecuadesktop-automationelectronpipi-coding-agentpi-extensionpi-packageplaywrighttypescriptui-automationvisionwebview2windowswindows-automationwsl
Bundle the full UIA tree runtime; text and labels follow Playwright (LPT-911)
alexshpunt-clanker[bot]14792282bd81
diff --git a/docs/bundled-runtime.md b/docs/bundled-runtime.mdindex 2212925..35f03a6 100644--- a/docs/bundled-runtime.md+++ b/docs/bundled-runtime.md@@ -1,6 +1,6 @@# Bundled CUA runtime-The npm package carries one native runtime: Cua Driver 0.37.0 for Windows x64, built from the `pi-computer-use` branch of the [alexshpunt/cua](https://github.com/alexshpunt/cua) fork at source `13dbacca`. [FORK.md](https://github.com/alexshpunt/cua/blob/pi-computer-use/FORK.md) lists what the fork adds to upstream Cua Driver. It is the qualified `windows-desktop` profile, so a fresh installation gets every native capability without any configuration:+The npm package carries one native runtime: Cua Driver 0.37.0 for Windows x64, built from the `pi-computer-use` branch of the [alexshpunt/cua](https://github.com/alexshpunt/cua) fork at source `0f170a4b`. [FORK.md](https://github.com/alexshpunt/cua/blob/pi-computer-use/FORK.md) lists what the fork adds to upstream Cua Driver. It is the qualified `windows-desktop` profile, so a fresh installation gets every native capability without any configuration:```shpi install npm:pi-agent-computer-use@@ -10,7 +10,7 @@ An update installs a new extension version with its exact runtime. Computer use## Where the files come from-The files are published as the fork release [`pi-runtime-0.37.0-13dbacca`](https://github.com/alexshpunt/cua/releases/tag/pi-runtime-0.37.0-13dbacca): `cua-driver.exe`, `cua-driver-uia.exe`, `cua-cursor-theme.exe`, `LICENSE.md` and `THIRD_PARTY_NOTICES.md`. They are byte-identical to the fork's CI build of that commit.+The files are published as the fork release [`pi-runtime-0.37.0-0f170a4b`](https://github.com/alexshpunt/cua/releases/tag/pi-runtime-0.37.0-0f170a4b): `cua-driver.exe`, `cua-driver-uia.exe`, `cua-cursor-theme.exe`, `LICENSE.md` and `THIRD_PARTY_NOTICES.md`. They are byte-identical to the fork's CI build of that commit.`src/runtime-pins.json` pins the source, the version, the release URL and the SHA-256 of every file. Before packing, `npm pack` runs `scripts/fetch-runtime.mjs`, which downloads the files into `native-runtime/win32-x64/runtime/` and checks each hash. The directory is not committed to Git.diff --git a/docs/computer-resources.md b/docs/computer-resources.mdindex 631593f..bbcb9b1 100644--- a/docs/computer-resources.md+++ b/docs/computer-resources.md@@ -231,7 +231,7 @@ The `regions` view needs Cua's signed perception extension on the machine whose"lab": {"ssh": "lab-wsl","runtime": "windows-desktop",- "command": "/mnt/c/Users/user/AppData/Local/Programs/PiComputerUse/cua-driver-0.37.0-desktop-13dbacca/runtime/cua-driver.exe",+ "command": "/mnt/c/Users/user/AppData/Local/Programs/PiComputerUse/cua-driver-0.37.0-0f170a4b/runtime/cua-driver.exe","node": "/mnt/c/Users/user/AppData/Local/Programs/PiComputerUse/node-v22.22.2-win-x64/node.exe","package": "/home/user/.local/share/pi-computer-use/package"}@@ -290,15 +290,15 @@ Insert an action with a `target` into a native **window** address to act without| Locator | Native meaning || --- | --- || `getByRole(role, { name, exact, checked, disabled, selected, expanded, pressed })` | ARIA role mapped from the UIA control type (`Button` → `button`, `Edit` → `textbox`, `ListItem` → `listitem` and `option`, `Text` → `text`, …); the UIA role name also matches. Names match a case-insensitive substring unless `exact` or a regex |-| `getByText`, `getByLabel`, `getByTitle` | Name or value text; the name of labelled controls (fields, lists, checkboxes); the UIA help text |+| `getByText`, `getByLabel`, `getByTitle` | Text content: the smallest element whose visible text matches (static text, buttons, items, group captions), not the label of a container or field; the name of labelled controls (fields, lists, checkboxes); the UIA help text || `getByTestId(id)` | UIA AutomationId || `.filter({ hasText, hasNotText, has, hasNot, visible })`, `.nth(i)`, `.first()`, `.last()`, `.and()`, `.or()`, chaining | As in Playwright; chaining searches inside the previous matches |CSS and XPath selectors refuse on native windows. Native actions with a target are `click`, `double_click` (`dblclick`), `right_click`, `hover`, `type_text`, `scroll`, `invoke`, `select`, `set_value` (`fill`), `check` and `uncheck`; add `timeout_ms` (0..60000, default 5000).-Each action takes fresh UI walks until exactly one element matches and is ready: enabled for input, for pointer input at the same place in two walks, and on screen for hover. Several matches refuse at once (strict mode) and list each match with a locator that names it; nothing is sent. A target that never becomes ready refuses at the timeout with the reason. `check` and `uncheck` send nothing when the element is already in that state, and read the state again afterwards. Elements that UIA reports as off screen still match, as in Playwright; `visible`/`hidden` checks treat them as not visible.+Each action takes fresh UI walks until exactly one element matches and is ready: enabled for input, and for pointer input also on screen and at the same place in two walks. Semantic actions such as `invoke` and `select` also reach elements that are scrolled out or in a closed drop-down. Several matches refuse at once (strict mode) and list each match with a locator that names it; nothing is sent. A target that never becomes ready refuses at the timeout with the reason. `check` and `uncheck` send nothing when the element is already in that state, and read the state again afterwards. Elements that UIA reports as off screen still match, as in Playwright; `visible`/`hidden` checks treat them as not visible.-A walk shows what UIA exposes: a runtime that walks only actionable elements leaves static text and some containers out, and its walks are never complete, so counts and absence stay `unknown` there.+The bundled runtime lists every element of the UIA control view, including static text, disabled controls and named containers. A walk that visited all of it is complete, and then counts, absence and aria mismatches are proven. A walk cut by `max_elements`, `max_depth` or time, or an app reached only through MSAA, is incomplete, and they stay `unknown`.For coordinate scroll, Insert `{"action":"scroll","x":<observed-x>,"y":<observed-y>,"delta_x":1,"delta_y":-2,"unit":"wheel_detents"}` into the current observation. Replace the point placeholders with local overview pixels inside the intended scroll surface. Deltas are signed integer wheel detents in -50..50; omit a zero axis, but provide at least one nonzero axis. Positive X means right and positive Y means down. One detent is 120 native wheel units, not a promised distance in content pixels. The `windows-coordinate-scroll` and inherited `windows-window-restore` / `windows-window-management` profiles are explicitly selected and byte-checked; other runtimes refuse this form. Background refuses without motion. After that refusal, Read a fresh window overview before adding `"delivery_mode":"foreground"`. Foreground moves the real pointer and leaves the window active. The native receipt preserves pointer and Y-then-X axis prefixes; inspect an interrupted prefix without replay. Do not combine coordinate scroll with hints, accessibility scroll fields, modifiers, animation or sequences. Existing hint-based accessibility scroll is unchanged.diff --git a/docs/ui-resources.md b/docs/ui-resources.mdindex 62a1f89..e22aecb 100644--- a/docs/ui-resources.md+++ b/docs/ui-resources.md@@ -6,7 +6,7 @@ Browsers and other Windows apps use the same window Reads. A window overview ret## Observe once, refine locally-Each element has a short `hint`, the exact native `element_token`, reported role/name/value, enabled/selected state, actions, depth and `parent_hint`. `name` comes from the native label, which may fall back to value or help text. Missing properties stay null or absent. An empty `actions` array is not proof that pixel input is impossible.+Each element has a short `hint`, the exact native `element_token`, reported role/name/value, enabled/selected state, actions, depth and `parent_hint`. The walk lists static text, disabled controls and named containers too; `actionable` marks the elements with action patterns, and `checked`, `expanded`, `offscreen`, `focused`, `automation_id` and `class_name` appear when UIA reports them. `name` comes from the native label, which may fall back to value or help text. Missing properties stay null or absent. An empty `actions` array is not proof that pixel input is impossible.`bounds.native_screen` and `bounds.window_image` are different coordinate spaces. Only reported screenshot-local bounds address pixels in the returned image. No adapter conversion invents missing bounds. Tree and pixels are not atomic, and neither proves paint freshness.diff --git a/src/computer-resources.ts b/src/computer-resources.tsindex 0d33298..e042426 100644--- a/src/computer-resources.ts+++ b/src/computer-resources.ts@@ -855,7 +855,7 @@ export class ComputerResources {/*** Resolve a Playwright locator the way Playwright's actions do: take fresh UI walks until exactly one element- * matches and is ready (enabled for input; for pointer input at the same place in two walks; on screen for hover), or+ * matches and is ready (enabled for input; for pointer input also on screen and at the same place in two walks), or* until the timeout. Several matches refuse at once with each match's locator, like Playwright's strict mode.*/private async resolveNative(window: string, session: ComputerSession, selector: ParsedSelector, options: { timeout_ms: number; enabled: boolean; stable: boolean; visible?: boolean; describe: string }, signal?: AbortSignal):@@ -877,7 +877,7 @@ export class ComputerResources {}const element = found[0];if (!element) reason = ui.complete === true && !ui.truncated ? "no element matches it" : "no element matches it in the walked part of the window (the UI walk was incomplete)";- else if (options.visible && !isVisible(element)) reason = "the element is not visible (offscreen)";+ else if (options.visible && !isVisible(element)) reason = "the element is not visible (offscreen); invoke, select and other semantic actions do not need it on screen";else if (options.enabled && element.enabled === false) reason = "the element is disabled";else if (options.stable && previous && JSON.stringify(previous.bounds.native_screen) !== JSON.stringify(element.bounds.native_screen)) reason = "the element is still moving";else if (!options.stable || previous || !element.bounds.native_screen) return { element, observation: fresh };@@ -905,7 +905,7 @@ export class ComputerResources {if (!NATIVE_LOCATOR_ACTIONS.includes(action)) throw new ComputerRequestError(`A target takes one of ${[...NATIVE_LOCATOR_ACTIONS, ...Object.keys(LOCATOR_VERBS)].join(", ")}.`);const describe = typeof target === "string" ? target : JSON.stringify(target);const pointer = ["click", "double_click", "right_click", "hover", "check", "uncheck"].includes(action);- const resolved = await this.resolveNative(window, session, selector, { timeout_ms: (timeout as number | undefined) ?? 5_000, enabled: action !== "hover", stable: pointer, visible: action === "hover", describe }, signal);+ const resolved = await this.resolveNative(window, session, selector, { timeout_ms: (timeout as number | undefined) ?? 5_000, enabled: action !== "hover", stable: pointer, visible: pointer, describe }, signal);const mode = this.getMode();if ("refused" in resolved) return { content: [{ type: "text", text: resolved.refused }], details: { status: "refused", mode, window_ref: window }, isError: true };const { element, observation } = resolved;@@ -991,7 +991,7 @@ export class ComputerResources {// Never reuse a stale observation: a target resolves in fresh UI walks; other steps take one fresh walk.if (step.target) {const action = String(step.command.action), pointer = ["click", "double_click", "right_click"].includes(action);- const resolved = await this.resolveNative(window, session, step.target, { timeout_ms: step.timeout!, enabled: true, stable: pointer, describe: step.describe! }, signal);+ const resolved = await this.resolveNative(window, session, step.target, { timeout_ms: step.timeout!, enabled: true, stable: pointer, visible: pointer, describe: step.describe! }, signal);if ("refused" in resolved) {if (resolved.observation) last = resolved.observation;rows.push({ index, action, outcome: "refused", reason: resolved.refused.slice(0, 600) });diff --git a/src/driver.ts b/src/driver.tsindex e082ced..9d6bf1e 100644--- a/src/driver.ts+++ b/src/driver.ts@@ -83,8 +83,8 @@ export const WINDOW_RESTORE_RUNTIME = {} as const;/** Window management plus every earlier native capability; `windows-desktop` selects these bytes. Browser pages use Playwright, not this runtime. */export const WINDOW_MANAGEMENT_RUNTIME = {- source: "13dbacca94e33fbdce56fd4ed6bd9bebc6aeb5a5", version: "0.37.0",- sha256: "689487a1566b4cb04adee8b30f6732d4ae096290e53dd1d7570064e8a3265b67",+ source: "0f170a4b2d5f04c153a9e7ddad478fe5b730abf0", version: "0.37.0",+ sha256: "fef81c7ac0a4412e9fa1d72ce802562933183f658ca2b0bc951084eb763b627e",} as const;const forkRuntimes = {"windows-desktop": WINDOW_MANAGEMENT_RUNTIME,diff --git a/src/native-locator.ts b/src/native-locator.tsindex 8fbcc57..b756839 100644--- a/src/native-locator.ts+++ b/src/native-locator.ts@@ -48,6 +48,12 @@ const ROLES: Record<string, string[]> = {};/** Roles Playwright's getByLabel looks at: labelled form controls. */const LABELLED = new Set(["textbox", "combobox", "checkbox", "radio", "slider", "spinbutton", "listbox", "switch"]);+/**+ * Roles whose UIA name is their own visible text, as an element's text content is on a page. Containers such as+ * windows, lists and menu bars are left out: their name is a label, and some apps borrow it from a nearby control.+ */+const CONTENT = new Set(["text", "button", "link", "menuitem", "listitem", "option", "tab", "treeitem", "checkbox", "radio",+ "cell", "gridcell", "columnheader", "rowheader", "heading", "tooltip", "group"]);/** Roles whose `checked` state is a checkbox state; for buttons the same UIA toggle is `pressed`. */const CHECKABLE = new Set(["checkbox", "radio", "menuitem", "switch", "treeitem", "listitem"]);@@ -99,8 +105,25 @@ const parseAttributes = (body: unknown): { name: string; attributes: Attribute[]class Tree {readonly order = new Map<string, number>();readonly byHint = new Map<string, UiElement>();+ private readonly children = new Map<string, UiElement[]>();+ private readonly texts = new Map<string, string>();constructor(readonly elements: readonly UiElement[]) {elements.forEach((element, index) => { this.order.set(element.hint, index); this.byHint.set(element.hint, element); });+ for (const element of elements) if (element.parent_hint && this.byHint.has(element.parent_hint))+ (this.children.get(element.parent_hint) ?? this.children.set(element.parent_hint, []).get(element.parent_hint)!).push(element);+ }+ /** Direct children in tree order. */+ childrenOf(element: UiElement): UiElement[] { return this.children.get(element.hint) ?? []; }+ /** Text content, as Playwright reads it: the visible text of this element and its descendants, without container labels. */+ textContent(element: UiElement): string {+ let text = this.texts.get(element.hint);+ if (text === undefined) {+ const role = ariaRole(element);+ const own = CONTENT.has(role) ? [accessibleName(element), role === "text" ? element.value ?? "" : ""] : [];+ text = normalize([...own, ...this.childrenOf(element).map(child => this.textContent(child))].filter(Boolean).join(" "));+ this.texts.set(element.hint, text);+ }+ return text;}/** Whether `element` lies strictly inside `root`. */inside(element: UiElement, root: UiElement): boolean {@@ -110,10 +133,6 @@ class Tree {descendants(roots: readonly UiElement[] | null): UiElement[] {return roots === null ? [...this.elements] : this.elements.filter(element => roots.some(root => this.inside(element, root)));}- /** Own text plus the text of every descendant, as Playwright's has-text sees an element's text content. */- subtreeText(element: UiElement): string {- return [element, ...this.descendants([element])].flatMap(item => [accessibleName(item), item.value ?? ""]).filter(Boolean).join(" ");- }sort(elements: Iterable<UiElement>): UiElement[] {return [...new Set(elements)].sort((a, b) => this.order.get(a.hint)! - this.order.get(b.hint)!);}@@ -141,7 +160,6 @@ function engine(part: ParsedSelector["parts"][number]): (element: UiElement) =>}return element => checks.every(check => check(element));}- case "internal:text": { const match = textMatcher(part.body); return element => match(accessibleName(element)) || (ariaRole(element) === "text" && match(element.value ?? "")); }case "internal:label": { const match = textMatcher(part.body); return element => LABELLED.has(ariaRole(element)) && match(accessibleName(element)); }case "internal:testid": {const [attribute] = parseAttributes(part.body).attributes;@@ -181,7 +199,7 @@ function evaluate(tree: Tree, selector: ParsedSelector, roots: readonly UiElemencase "visible": current = all().filter(element => isVisible(element) === (String(part.body) === "true")); break;case "internal:has-text": case "internal:has-not-text": {const match = textMatcher(part.body), keep = part.name === "internal:has-text";- current = all().filter(element => match(tree.subtreeText(element)) === keep);+ current = all().filter(element => match(tree.textContent(element)) === keep);break;}case "internal:has": case "internal:has-not": {@@ -192,6 +210,13 @@ function evaluate(tree: Tree, selector: ParsedSelector, roots: readonly UiElemencase "internal:and": { const other = new Set(evaluate(tree, nested(part.body), scope)); current = all().filter(element => other.has(element)); break; }case "internal:or": current = tree.sort([...all(), ...evaluate(tree, nested(part.body), scope)]); break;case "internal:describe": break;+ // Like Playwright's text engine: the smallest elements whose text content matches, not their ancestors.+ case "internal:text": {+ const match = textMatcher(part.body);+ scope = current ?? roots;+ current = tree.descendants(scope).filter(element => match(tree.textContent(element)) && !tree.childrenOf(element).some(child => match(tree.textContent(child))));+ break;+ }default: {const match = engine(part);scope = current ?? roots;diff --git a/src/runtime-pins.json b/src/runtime-pins.jsonindex d65cd42..86a45a6 100644--- a/src/runtime-pins.json+++ b/src/runtime-pins.json@@ -1,12 +1,12 @@{"format": 2,- "source": "13dbacca94e33fbdce56fd4ed6bd9bebc6aeb5a5",+ "source": "0f170a4b2d5f04c153a9e7ddad478fe5b730abf0","version": "0.37.0",- "url": "https://github.com/alexshpunt/cua/releases/download/pi-runtime-0.37.0-13dbacca/",+ "url": "https://github.com/alexshpunt/cua/releases/download/pi-runtime-0.37.0-0f170a4b/","files": {- "cua-driver.exe": "689487a1566b4cb04adee8b30f6732d4ae096290e53dd1d7570064e8a3265b67",- "cua-driver-uia.exe": "ab0ece2a0aae0f357e3cf7db82b56f4e2d9d3e15d5c770d6c8beeea410885dba",- "cua-cursor-theme.exe": "a6b3dc598ce3f95372d95ea2da72eb9f960c49fea24adbcb1664839d1528eb1a",+ "cua-driver.exe": "fef81c7ac0a4412e9fa1d72ce802562933183f658ca2b0bc951084eb763b627e",+ "cua-driver-uia.exe": "6387b1b0e6da7c61e9d2fbc5c3dd512d9a25f535129d1d4764578c1f58b0162d",+ "cua-cursor-theme.exe": "ebaec44ccd472e108fce2771b271288e5e22f1b6f3ce341cd9e42e5c124468d5","LICENSE.md": "d0a32419a44fa38d5023d1431dfcbedd1457eb5e7b2e6c7c87ca064facb41b41","THIRD_PARTY_NOTICES.md": "88a24f59293f03b3e03655e8144bcfadc9861bef1182cfeea91639e11f573fd1"}diff --git a/src/ui-resources.ts b/src/ui-resources.tsindex 81c37d2..72b9694 100644--- a/src/ui-resources.ts+++ b/src/ui-resources.ts@@ -16,7 +16,8 @@ export interface UiElement {checked?: "on" | "off" | "mixed"; expanded?: boolean; offscreen?: boolean; focused?: boolean;password?: boolean; read_only?: boolean; required?: boolean; actionable?: boolean;range?: { value: number; min: number; max: number };- hint_display: "pending" | "shown" | "disabled" | "no_bounds" | "outside_image" | "collision" | "off";+ /** Whether the hint label is drawn on the overview; `read_only` marks static text and containers, which get no label. */+ hint_display: "pending" | "shown" | "disabled" | "read_only" | "no_bounds" | "outside_image" | "collision" | "off";}/** Bounded native UIA projection. Completeness is not inferred from an empty search. */export interface UiResource {@@ -166,7 +167,10 @@ export async function annotateUi(image: Content, resource: UiResource, width: nureturn { image, annotation };}const canvas = { x: 0, y: 0, width, height };+ // Labels keep off controls and static text, not off containers (groups, menu bars, title bars) they sit inside.+ const containers = new Set(resource.elements.flatMap(e => e.parent_hint ? [e.parent_hint] : []));const protectedRegions = obstacles.flatMap(e => {+ if (e.actionable === false && containers.has(e.hint)) return [];const r = e.bounds.window_image;return r && r.width * r.height < width * height / 4 ? [r] : [];});@@ -174,6 +178,8 @@ export async function annotateUi(image: Content, resource: UiResource, width: nufor (const e of resource.elements) {if (!enabled) { e.hint_display = e.enabled === false ? "disabled" : e.bounds.window_image ? "off" : "no_bounds"; continue; }if (e.enabled === false) { e.hint_display = "disabled"; continue; }+ // Labels on every text and container would cover the controls; those stay addressable through UI and Search.+ if (e.actionable === false) { e.hint_display = "read_only"; continue; }const target = clipRect(e.bounds.window_image, canvas);if (!target) { e.hint_display = e.bounds.window_image ? "outside_image" : "no_bounds"; continue; }if (selected === e.hint) svg.push(`<rect x="${target.x}" y="${target.y}" width="${target.width}" height="${target.height}" fill="none" stroke="#e52678" stroke-width="2"/>`);diff --git a/test/live/locators.live.ts b/test/live/locators.live.tsindex eea738f..d19e404 100644--- a/test/live/locators.live.ts+++ b/test/live/locators.live.ts@@ -21,7 +21,12 @@ $c = New-Object Windows.Forms.CheckBox; $c.Text = 'Agree'; $c.Left = 10; $c.Top$l = New-Object Windows.Forms.ListBox; $l.Left = 10; $l.Top = 120; $l.Height = 70; [void]$l.Items.AddRange(@('alpha','beta','gamma')); $l.AccessibleName = 'Pick'$tm = New-Object Windows.Forms.Timer; $tm.Interval = 1500; $tm.Add_Tick({ $d.Enabled = $true; $tm.Stop() })$f.Add_Shown({ $tm.Start() })-$f.Controls.AddRange(@($b, $d, $t, $c, $l)); [void]$f.ShowDialog()`;+$s = New-Object Windows.Forms.Label; $s.Left = 10; $s.Top = 200; $s.Width = 300; $s.Text = 'idle'+$b.Add_Click({ $s.Text = "clicked $script:n" })+$m = New-Object Windows.Forms.MenuStrip; $file = New-Object Windows.Forms.ToolStripMenuItem('File'); $reset = New-Object Windows.Forms.ToolStripMenuItem('Reset')+$reset.Add_Click({ $script:n = 0; $b.Text = 'Count 0'; $s.Text = 'reset' }); [void]$file.DropDownItems.Add($reset); [void]$m.Items.Add($file); $f.MainMenuStrip = $m+foreach ($x in @($b, $d, $t, $c, $l, $s)) { $x.Top += 30 }+$f.Controls.AddRange(@($b, $d, $t, $c, $l, $s, $m)); [void]$f.ShowDialog()`;test("a WinForms window is driven by Playwright locators, one call per step", async () => {const window = await openWindow(live, live.machine, WINDOWS_POWERSHELL_PATH,@@ -50,13 +55,21 @@ test("a WinForms window is driven by Playwright locators, one call per step", asawait expectStatus({ action: "verify", conditions: [{ target: `getByRole("textbox", { name: "Name" })`, value: "Ada Lovelace" },{ target: `getByRole("checkbox", { name: "Agree" })`, is: "checked" },- { aria: `- textbox "Name": Ada Lovelace\n- checkbox "Agree" [checked]\n- listitem "beta" [selected]` },+ { aria: `- textbox "Name": Ada Lovelace\n- checkbox "Agree" [checked]\n- list:\n - listitem "beta" [selected]` },] }, "observed", /Verification satisfied/);- // A count is proven only by a complete walk; a runtime that walks just actionable elements leaves it unknown.- const counted = await step({ action: "verify", conditions: [{ target: `getByRole("listitem")`, count: 3 }] });- const [count] = counted.details.verification!.conditions;- expect(count.observed, presentComputerResult(counted)).toBe("3 match(es)");- expect(counted.details.ui!.complete ? ["satisfied"] : ["unknown"]).toContain(count.verdict);+ // The bundled runtime walks the whole structural tree: static text is there, and a complete walk proves counts and absence.+ await expectStatus({ action: "verify", conditions: [+ { target: `getByText("clicked 1")`, is: "visible" },+ { target: `getByRole("listitem")`, count: 3 },+ { target: `getByRole("button", { name: "Nope" })`, is: "absent" },+ { aria: `- list "Pick":\n - listitem "alpha"\n - listitem "beta" [selected]\n - listitem "gamma"\n- text: clicked 1` },+ ] }, "observed", /Verification satisfied/);+ // A mismatch in a complete walk is proven, and the receipt shows the received tree.+ await expectStatus({ action: "verify", conditions: [{ aria: `- button "Nope"` }] }, "observed", /Verification unsatisfied[\s\S]*Received aria snapshot:[\s\S]*- text: clicked 1/);+ // A menu: open File, then pick Reset from its drop-down, both by locator.+ await expectStatus({ action: "click", target: `getByRole("menuitem", { name: "File" })` }, "delivered");+ await expectStatus({ action: "click", target: `getByRole("menuitem", { name: "Reset" })` }, "delivered");+ await expectStatus({ action: "wait_for", conditions: [{ target: `getByText("reset", { exact: true })`, is: "visible" }, { target: `getByRole("button", { name: "Count 0" })`, is: "exists" }], timeout_ms: 3000 }, "observed", /Verification satisfied/);// Strict mode refuses an ambiguous target and lists each match's locator; nothing is sent.await expectStatus({ action: "click", target: `getByRole("button")` }, "refused", /Strict mode: getByRole\("button"\) matches \d+ elements/);await expectStatus({ action: "click", target: `getByRole("button", { name: "Nope" })`, timeout_ms: 800 }, "refused", /Timed out after 800 ms/);@@ -65,7 +78,9 @@ test("a WinForms window is driven by Playwright locators, one call per step", asconst aria = await live.read(window, ["aria"]);const shown = presentComputerResult(aria);console.log(shown, "\n", timings.join("\n"));- expect(shown).toMatch(/- button "Count 1" \[ref=[^\]]+\]/);+ expect(shown).toMatch(/- button "Count 0" \[ref=[^\]]+\]/);+ expect(shown).toMatch(/- menuitem "File" \[ref=[^\]]+\]/);+ expect(shown).toMatch(/- text: reset/);expect(shown).toMatch(/- checkbox "Agree" \[checked\] \[ref=/);expect(shown).toMatch(/- textbox "Name" \[ref=[^\]]+\]: Ada Lovelace/);});diff --git a/test/native-locator.test.ts b/test/native-locator.test.tsindex 565437f..5dc3b8f 100644--- a/test/native-locator.test.ts+++ b/test/native-locator.test.ts@@ -23,6 +23,9 @@ const tree = { elements: [element("status", "Text", "clicked 3"),element("save", "Button", "Save", { help_text: "Save the file" }),element("saveAs", "Button", "Save as…"),+ // WinForms names a MenuStrip after the label before it; that name is a label, not text content.+ element("bar", "MenuBar", "clicked 3"),+ element("file", "MenuItem", "File", { parent_hint: "bar", depth: 2 }),] };// playwright-core loads on first use; under a parallel test run that can take several seconds.beforeAll(() => { parseLocator("getByRole('button')"); }, 60_000);@@ -50,6 +53,12 @@ test("role states filter like Playwright's options; scrolled-out elements stilltest("text, label, test id and title locators read the matching UIA properties", () => {expect(hints(`getByText("clicked")`)).toEqual(["status"]);+ // Like Playwright: the smallest element whose text content matches, so a group matches by its caption and items by their own text.+ expect(hints(`getByText("Options")`)).toEqual(["opts"]);+ expect(hints(`getByText("OK")`)).toEqual(["ok"]);+ expect(hints(`getByText("File")`)).toEqual(["file"]);+ expect(hints(`getByRole("menubar").filter({ hasText: "File" })`)).toEqual(["bar"]);+ expect(hints(`getByRole("menubar").filter({ hasText: "clicked" })`)).toEqual([]);expect(hints(`getByText("Clicked 3", { exact: true })`)).toEqual([]);expect(hints(`getByText("clicked 3", { exact: true })`)).toEqual(["status"]);expect(hints(`getByLabel("Name")`)).toEqual(["name"]);@@ -104,6 +113,8 @@ test("a native window renders as Playwright's own aria snapshot, with hints as r` - text: clicked 3`,` - button "Save" [ref=save]`,` - button "Save as…" [ref=saveAs]`,+ ` - menubar "clicked 3" [ref=bar]:`,+ ` - menuitem "File" [ref=file]`,].join("\n"));});diff --git a/test/ui-conditions.test.ts b/test/ui-conditions.test.tsindex fe91508..b94a653 100644--- a/test/ui-conditions.test.ts+++ b/test/ui-conditions.test.ts@@ -4,7 +4,8 @@ import { combine, evaluateNative, validateWait } from "../src/ui-conditions.ts";import type { UiElement, UiResource } from "../src/ui-resources.ts";beforeAll(() => { parseLocator("getByRole('button')"); }, 60_000);-const element = (fields: Partial<UiElement>): UiElement => ({ hint: "1:a", element_token: "t", index: 0, parent_hint: null, role: null, name: null, value: null,+let serial = 0;+const element = (fields: Partial<UiElement>): UiElement => ({ hint: `1:${serial++}`, element_token: "t", index: 0, parent_hint: null, role: null, name: null, value: null,text_truncated: false, enabled: true, selected: null, actions: [], depth: 0, bounds: {}, hint_display: "off", ...fields });const ui = (elements: UiElement[], complete: boolean | null): UiResource => ({ window_ref: "w", observation_id: "o", snapshot_id: null, captured_at: "",complete, truncated: false, truncation_reason: null, native_total: elements.length, collected: elements.length, matches: elements.length, projection_truncated: false, elements });@@ -18,7 +19,7 @@ test("native absence is decided only by a complete walk; a missing element in atest("native checks compare reported properties and keep ambiguity and missing properties unknown", () => {const tree = ui([element({ role: "Button", name: "Save", enabled: false }), element({ role: "Text", name: "Status", value: "saved Ada" }),- element({ role: "Button", name: "A" }), element({ role: "Button", name: "B" }), element({ role: "Edit", name: "Long", value: "x", text_truncated: true })], true);+ element({ role: "Button", name: "A" }), element({ role: "Button", name: "B" }), element({ role: "Text", name: "Long", value: "x", text_truncated: true })], true);const check = (condition: object) => evaluateNative(tree, validateWait({ action: "verify", conditions: [condition] }, false).conditions)[0];expect(check({ target: { role: "Button", name: "Save" }, is: "disabled" }).verdict).toBe("satisfied");expect(check({ target: { role: "Button", name: "Save" }, is: "enabled" }).verdict).toBe("unsatisfied");diff --git a/test/ui-resources.test.ts b/test/ui-resources.test.tsindex 06237fa..0cddc92 100644--- a/test/ui-resources.test.ts+++ b/test/ui-resources.test.ts@@ -111,6 +111,28 @@ test("render status describes this image, not a previous annotation", async () =expect(resource.elements[2].hint_display).toBe("disabled"); expect(resource.elements[3].hint_display).toBe("no_bounds");});+test("static text and containers from a full UIA tree get no hint label on the overview", async () => {+ const { annotateUi } = await import("../src/ui-resources.ts");+ const f = await fixture(); const view = await f.session.observe(f.ref); const resource = structuredClone(ui(view));+ resource.elements[1].actionable = false;+ const result = await annotateUi({ type: "image", data: f.raw.toString("base64"), mimeType: "image/png" }, resource, 500, 320, true);+ expect(resource.elements[1].hint_display).toBe("read_only");+ expect(result.annotation.labels.map(label => label.hint)).not.toContain(resource.elements[1].hint);+});++test("a read-only container does not push the labels of the controls inside it off the overview", async () => {+ const { annotateUi } = await import("../src/ui-resources.ts");+ const f = await fixture(); const view = await f.session.observe(f.ref); const resource = structuredClone(ui(view));+ const control = resource.elements[1];+ const box = control.bounds.window_image!;+ // A group whose frame surrounds the control with no room for a label outside it.+ const group = { ...structuredClone(control), hint: "g:1", actionable: false, parent_hint: null,+ bounds: { window_image: { x: Math.max(0, box.x - 40), y: Math.max(0, box.y - 40), width: box.width + 80, height: box.height + 80 } } };+ resource.elements = [group, { ...control, parent_hint: "g:1", actionable: true }];+ const result = await annotateUi({ type: "image", data: f.raw.toString("base64"), mimeType: "image/png" }, resource, 500, 320, true);+ expect(result.annotation.labels.map(label => label.hint)).toEqual([control.hint]);+});+test("scoped label placement respects controls excluded from the projection", async () => {const { annotateUi, projectUi } = await import("../src/ui-resources.ts");const f = await fixture(); const view = await f.session.observe(f.ref); const resource = structuredClone(ui(view));