Skip to content

Computer use for the Pi coding agent — Windows apps, browsers and desktops through native UI trees, screenshots and real input.

Bundle the full UIA tree runtime; text and labels follow Playwright (LPT-911)

alexshpunt-clanker[bot]14792282bd81
diff --git a/docs/bundled-runtime.md b/docs/bundled-runtime.md
index 2212925..35f03a6 100644
--- a/docs/bundled-runtime.md
+++ b/docs/bundled-runtime.md
@@ -1,6 +1,6 @@
# Bundled CUA runtime
-The npm package carries one native runtime: Cua Driver 0.37.0 for Windows x64, built from the `pi-computer-use` branch of the [alexshpunt/cua](https://github.com/alexshpunt/cua) fork at source `13dbacca`. [FORK.md](https://github.com/alexshpunt/cua/blob/pi-computer-use/FORK.md) lists what the fork adds to upstream Cua Driver. It is the qualified `windows-desktop` profile, so a fresh installation gets every native capability without any configuration:
+The npm package carries one native runtime: Cua Driver 0.37.0 for Windows x64, built from the `pi-computer-use` branch of the [alexshpunt/cua](https://github.com/alexshpunt/cua) fork at source `0f170a4b`. [FORK.md](https://github.com/alexshpunt/cua/blob/pi-computer-use/FORK.md) lists what the fork adds to upstream Cua Driver. It is the qualified `windows-desktop` profile, so a fresh installation gets every native capability without any configuration:
```sh
pi install npm:pi-agent-computer-use
@@ -10,7 +10,7 @@ An update installs a new extension version with its exact runtime. Computer use
## Where the files come from
-The files are published as the fork release [`pi-runtime-0.37.0-13dbacca`](https://github.com/alexshpunt/cua/releases/tag/pi-runtime-0.37.0-13dbacca): `cua-driver.exe`, `cua-driver-uia.exe`, `cua-cursor-theme.exe`, `LICENSE.md` and `THIRD_PARTY_NOTICES.md`. They are byte-identical to the fork's CI build of that commit.
+The files are published as the fork release [`pi-runtime-0.37.0-0f170a4b`](https://github.com/alexshpunt/cua/releases/tag/pi-runtime-0.37.0-0f170a4b): `cua-driver.exe`, `cua-driver-uia.exe`, `cua-cursor-theme.exe`, `LICENSE.md` and `THIRD_PARTY_NOTICES.md`. They are byte-identical to the fork's CI build of that commit.
`src/runtime-pins.json` pins the source, the version, the release URL and the SHA-256 of every file. Before packing, `npm pack` runs `scripts/fetch-runtime.mjs`, which downloads the files into `native-runtime/win32-x64/runtime/` and checks each hash. The directory is not committed to Git.
diff --git a/docs/computer-resources.md b/docs/computer-resources.md
index 631593f..bbcb9b1 100644
--- a/docs/computer-resources.md
+++ b/docs/computer-resources.md
@@ -231,7 +231,7 @@ The `regions` view needs Cua's signed perception extension on the machine whose
"lab": {
"ssh": "lab-wsl",
"runtime": "windows-desktop",
- "command": "/mnt/c/Users/user/AppData/Local/Programs/PiComputerUse/cua-driver-0.37.0-desktop-13dbacca/runtime/cua-driver.exe",
+ "command": "/mnt/c/Users/user/AppData/Local/Programs/PiComputerUse/cua-driver-0.37.0-0f170a4b/runtime/cua-driver.exe",
"node": "/mnt/c/Users/user/AppData/Local/Programs/PiComputerUse/node-v22.22.2-win-x64/node.exe",
"package": "/home/user/.local/share/pi-computer-use/package"
}
@@ -290,15 +290,15 @@ Insert an action with a `target` into a native **window** address to act without
| Locator | Native meaning |
| --- | --- |
| `getByRole(role, { name, exact, checked, disabled, selected, expanded, pressed })` | ARIA role mapped from the UIA control type (`Button` → `button`, `Edit` → `textbox`, `ListItem` → `listitem` and `option`, `Text` → `text`, …); the UIA role name also matches. Names match a case-insensitive substring unless `exact` or a regex |
-| `getByText`, `getByLabel`, `getByTitle` | Name or value text; the name of labelled controls (fields, lists, checkboxes); the UIA help text |
+| `getByText`, `getByLabel`, `getByTitle` | Text content: the smallest element whose visible text matches (static text, buttons, items, group captions), not the label of a container or field; the name of labelled controls (fields, lists, checkboxes); the UIA help text |
| `getByTestId(id)` | UIA AutomationId |
| `.filter({ hasText, hasNotText, has, hasNot, visible })`, `.nth(i)`, `.first()`, `.last()`, `.and()`, `.or()`, chaining | As in Playwright; chaining searches inside the previous matches |
CSS and XPath selectors refuse on native windows. Native actions with a target are `click`, `double_click` (`dblclick`), `right_click`, `hover`, `type_text`, `scroll`, `invoke`, `select`, `set_value` (`fill`), `check` and `uncheck`; add `timeout_ms` (0..60000, default 5000).
-Each action takes fresh UI walks until exactly one element matches and is ready: enabled for input, for pointer input at the same place in two walks, and on screen for hover. Several matches refuse at once (strict mode) and list each match with a locator that names it; nothing is sent. A target that never becomes ready refuses at the timeout with the reason. `check` and `uncheck` send nothing when the element is already in that state, and read the state again afterwards. Elements that UIA reports as off screen still match, as in Playwright; `visible`/`hidden` checks treat them as not visible.
+Each action takes fresh UI walks until exactly one element matches and is ready: enabled for input, and for pointer input also on screen and at the same place in two walks. Semantic actions such as `invoke` and `select` also reach elements that are scrolled out or in a closed drop-down. Several matches refuse at once (strict mode) and list each match with a locator that names it; nothing is sent. A target that never becomes ready refuses at the timeout with the reason. `check` and `uncheck` send nothing when the element is already in that state, and read the state again afterwards. Elements that UIA reports as off screen still match, as in Playwright; `visible`/`hidden` checks treat them as not visible.
-A walk shows what UIA exposes: a runtime that walks only actionable elements leaves static text and some containers out, and its walks are never complete, so counts and absence stay `unknown` there.
+The bundled runtime lists every element of the UIA control view, including static text, disabled controls and named containers. A walk that visited all of it is complete, and then counts, absence and aria mismatches are proven. A walk cut by `max_elements`, `max_depth` or time, or an app reached only through MSAA, is incomplete, and they stay `unknown`.
For coordinate scroll, Insert `{"action":"scroll","x":<observed-x>,"y":<observed-y>,"delta_x":1,"delta_y":-2,"unit":"wheel_detents"}` into the current observation. Replace the point placeholders with local overview pixels inside the intended scroll surface. Deltas are signed integer wheel detents in -50..50; omit a zero axis, but provide at least one nonzero axis. Positive X means right and positive Y means down. One detent is 120 native wheel units, not a promised distance in content pixels. The `windows-coordinate-scroll` and inherited `windows-window-restore` / `windows-window-management` profiles are explicitly selected and byte-checked; other runtimes refuse this form. Background refuses without motion. After that refusal, Read a fresh window overview before adding `"delivery_mode":"foreground"`. Foreground moves the real pointer and leaves the window active. The native receipt preserves pointer and Y-then-X axis prefixes; inspect an interrupted prefix without replay. Do not combine coordinate scroll with hints, accessibility scroll fields, modifiers, animation or sequences. Existing hint-based accessibility scroll is unchanged.
diff --git a/docs/ui-resources.md b/docs/ui-resources.md
index 62a1f89..e22aecb 100644
--- a/docs/ui-resources.md
+++ b/docs/ui-resources.md
@@ -6,7 +6,7 @@ Browsers and other Windows apps use the same window Reads. A window overview ret
## Observe once, refine locally
-Each element has a short `hint`, the exact native `element_token`, reported role/name/value, enabled/selected state, actions, depth and `parent_hint`. `name` comes from the native label, which may fall back to value or help text. Missing properties stay null or absent. An empty `actions` array is not proof that pixel input is impossible.
+Each element has a short `hint`, the exact native `element_token`, reported role/name/value, enabled/selected state, actions, depth and `parent_hint`. The walk lists static text, disabled controls and named containers too; `actionable` marks the elements with action patterns, and `checked`, `expanded`, `offscreen`, `focused`, `automation_id` and `class_name` appear when UIA reports them. `name` comes from the native label, which may fall back to value or help text. Missing properties stay null or absent. An empty `actions` array is not proof that pixel input is impossible.
`bounds.native_screen` and `bounds.window_image` are different coordinate spaces. Only reported screenshot-local bounds address pixels in the returned image. No adapter conversion invents missing bounds. Tree and pixels are not atomic, and neither proves paint freshness.
diff --git a/src/computer-resources.ts b/src/computer-resources.ts
index 0d33298..e042426 100644
--- a/src/computer-resources.ts
+++ b/src/computer-resources.ts
@@ -855,7 +855,7 @@ export class ComputerResources {
/**
* Resolve a Playwright locator the way Playwright's actions do: take fresh UI walks until exactly one element
- * matches and is ready (enabled for input; for pointer input at the same place in two walks; on screen for hover), or
+ * matches and is ready (enabled for input; for pointer input also on screen and at the same place in two walks), or
* until the timeout. Several matches refuse at once with each match's locator, like Playwright's strict mode.
*/
private async resolveNative(window: string, session: ComputerSession, selector: ParsedSelector, options: { timeout_ms: number; enabled: boolean; stable: boolean; visible?: boolean; describe: string }, signal?: AbortSignal):
@@ -877,7 +877,7 @@ export class ComputerResources {
}
const element = found[0];
if (!element) reason = ui.complete === true && !ui.truncated ? "no element matches it" : "no element matches it in the walked part of the window (the UI walk was incomplete)";
- else if (options.visible && !isVisible(element)) reason = "the element is not visible (offscreen)";
+ else if (options.visible && !isVisible(element)) reason = "the element is not visible (offscreen); invoke, select and other semantic actions do not need it on screen";
else if (options.enabled && element.enabled === false) reason = "the element is disabled";
else if (options.stable && previous && JSON.stringify(previous.bounds.native_screen) !== JSON.stringify(element.bounds.native_screen)) reason = "the element is still moving";
else if (!options.stable || previous || !element.bounds.native_screen) return { element, observation: fresh };
@@ -905,7 +905,7 @@ export class ComputerResources {
if (!NATIVE_LOCATOR_ACTIONS.includes(action)) throw new ComputerRequestError(`A target takes one of ${[...NATIVE_LOCATOR_ACTIONS, ...Object.keys(LOCATOR_VERBS)].join(", ")}.`);
const describe = typeof target === "string" ? target : JSON.stringify(target);
const pointer = ["click", "double_click", "right_click", "hover", "check", "uncheck"].includes(action);
- const resolved = await this.resolveNative(window, session, selector, { timeout_ms: (timeout as number | undefined) ?? 5_000, enabled: action !== "hover", stable: pointer, visible: action === "hover", describe }, signal);
+ const resolved = await this.resolveNative(window, session, selector, { timeout_ms: (timeout as number | undefined) ?? 5_000, enabled: action !== "hover", stable: pointer, visible: pointer, describe }, signal);
const mode = this.getMode();
if ("refused" in resolved) return { content: [{ type: "text", text: resolved.refused }], details: { status: "refused", mode, window_ref: window }, isError: true };
const { element, observation } = resolved;
@@ -991,7 +991,7 @@ export class ComputerResources {
// Never reuse a stale observation: a target resolves in fresh UI walks; other steps take one fresh walk.
if (step.target) {
const action = String(step.command.action), pointer = ["click", "double_click", "right_click"].includes(action);
- const resolved = await this.resolveNative(window, session, step.target, { timeout_ms: step.timeout!, enabled: true, stable: pointer, describe: step.describe! }, signal);
+ const resolved = await this.resolveNative(window, session, step.target, { timeout_ms: step.timeout!, enabled: true, stable: pointer, visible: pointer, describe: step.describe! }, signal);
if ("refused" in resolved) {
if (resolved.observation) last = resolved.observation;
rows.push({ index, action, outcome: "refused", reason: resolved.refused.slice(0, 600) });
diff --git a/src/driver.ts b/src/driver.ts
index e082ced..9d6bf1e 100644
--- a/src/driver.ts
+++ b/src/driver.ts
@@ -83,8 +83,8 @@ export const WINDOW_RESTORE_RUNTIME = {
} as const;
/** Window management plus every earlier native capability; `windows-desktop` selects these bytes. Browser pages use Playwright, not this runtime. */
export const WINDOW_MANAGEMENT_RUNTIME = {
- source: "13dbacca94e33fbdce56fd4ed6bd9bebc6aeb5a5", version: "0.37.0",
- sha256: "689487a1566b4cb04adee8b30f6732d4ae096290e53dd1d7570064e8a3265b67",
+ source: "0f170a4b2d5f04c153a9e7ddad478fe5b730abf0", version: "0.37.0",
+ sha256: "fef81c7ac0a4412e9fa1d72ce802562933183f658ca2b0bc951084eb763b627e",
} as const;
const forkRuntimes = {
"windows-desktop": WINDOW_MANAGEMENT_RUNTIME,
diff --git a/src/native-locator.ts b/src/native-locator.ts
index 8fbcc57..b756839 100644
--- a/src/native-locator.ts
+++ b/src/native-locator.ts
@@ -48,6 +48,12 @@ const ROLES: Record<string, string[]> = {
};
/** Roles Playwright's getByLabel looks at: labelled form controls. */
const LABELLED = new Set(["textbox", "combobox", "checkbox", "radio", "slider", "spinbutton", "listbox", "switch"]);
+/**
+ * Roles whose UIA name is their own visible text, as an element's text content is on a page. Containers such as
+ * windows, lists and menu bars are left out: their name is a label, and some apps borrow it from a nearby control.
+ */
+const CONTENT = new Set(["text", "button", "link", "menuitem", "listitem", "option", "tab", "treeitem", "checkbox", "radio",
+ "cell", "gridcell", "columnheader", "rowheader", "heading", "tooltip", "group"]);
/** Roles whose `checked` state is a checkbox state; for buttons the same UIA toggle is `pressed`. */
const CHECKABLE = new Set(["checkbox", "radio", "menuitem", "switch", "treeitem", "listitem"]);
@@ -99,8 +105,25 @@ const parseAttributes = (body: unknown): { name: string; attributes: Attribute[]
class Tree {
readonly order = new Map<string, number>();
readonly byHint = new Map<string, UiElement>();
+ private readonly children = new Map<string, UiElement[]>();
+ private readonly texts = new Map<string, string>();
constructor(readonly elements: readonly UiElement[]) {
elements.forEach((element, index) => { this.order.set(element.hint, index); this.byHint.set(element.hint, element); });
+ for (const element of elements) if (element.parent_hint && this.byHint.has(element.parent_hint))
+ (this.children.get(element.parent_hint) ?? this.children.set(element.parent_hint, []).get(element.parent_hint)!).push(element);
+ }
+ /** Direct children in tree order. */
+ childrenOf(element: UiElement): UiElement[] { return this.children.get(element.hint) ?? []; }
+ /** Text content, as Playwright reads it: the visible text of this element and its descendants, without container labels. */
+ textContent(element: UiElement): string {
+ let text = this.texts.get(element.hint);
+ if (text === undefined) {
+ const role = ariaRole(element);
+ const own = CONTENT.has(role) ? [accessibleName(element), role === "text" ? element.value ?? "" : ""] : [];
+ text = normalize([...own, ...this.childrenOf(element).map(child => this.textContent(child))].filter(Boolean).join(" "));
+ this.texts.set(element.hint, text);
+ }
+ return text;
}
/** Whether `element` lies strictly inside `root`. */
inside(element: UiElement, root: UiElement): boolean {
@@ -110,10 +133,6 @@ class Tree {
descendants(roots: readonly UiElement[] | null): UiElement[] {
return roots === null ? [...this.elements] : this.elements.filter(element => roots.some(root => this.inside(element, root)));
}
- /** Own text plus the text of every descendant, as Playwright's has-text sees an element's text content. */
- subtreeText(element: UiElement): string {
- return [element, ...this.descendants([element])].flatMap(item => [accessibleName(item), item.value ?? ""]).filter(Boolean).join(" ");
- }
sort(elements: Iterable<UiElement>): UiElement[] {
return [...new Set(elements)].sort((a, b) => this.order.get(a.hint)! - this.order.get(b.hint)!);
}
@@ -141,7 +160,6 @@ function engine(part: ParsedSelector["parts"][number]): (element: UiElement) =>
}
return element => checks.every(check => check(element));
}
- case "internal:text": { const match = textMatcher(part.body); return element => match(accessibleName(element)) || (ariaRole(element) === "text" && match(element.value ?? "")); }
case "internal:label": { const match = textMatcher(part.body); return element => LABELLED.has(ariaRole(element)) && match(accessibleName(element)); }
case "internal:testid": {
const [attribute] = parseAttributes(part.body).attributes;
@@ -181,7 +199,7 @@ function evaluate(tree: Tree, selector: ParsedSelector, roots: readonly UiElemen
case "visible": current = all().filter(element => isVisible(element) === (String(part.body) === "true")); break;
case "internal:has-text": case "internal:has-not-text": {
const match = textMatcher(part.body), keep = part.name === "internal:has-text";
- current = all().filter(element => match(tree.subtreeText(element)) === keep);
+ current = all().filter(element => match(tree.textContent(element)) === keep);
break;
}
case "internal:has": case "internal:has-not": {
@@ -192,6 +210,13 @@ function evaluate(tree: Tree, selector: ParsedSelector, roots: readonly UiElemen
case "internal:and": { const other = new Set(evaluate(tree, nested(part.body), scope)); current = all().filter(element => other.has(element)); break; }
case "internal:or": current = tree.sort([...all(), ...evaluate(tree, nested(part.body), scope)]); break;
case "internal:describe": break;
+ // Like Playwright's text engine: the smallest elements whose text content matches, not their ancestors.
+ case "internal:text": {
+ const match = textMatcher(part.body);
+ scope = current ?? roots;
+ current = tree.descendants(scope).filter(element => match(tree.textContent(element)) && !tree.childrenOf(element).some(child => match(tree.textContent(child))));
+ break;
+ }
default: {
const match = engine(part);
scope = current ?? roots;
diff --git a/src/runtime-pins.json b/src/runtime-pins.json
index d65cd42..86a45a6 100644
--- a/src/runtime-pins.json
+++ b/src/runtime-pins.json
@@ -1,12 +1,12 @@
{
"format": 2,
- "source": "13dbacca94e33fbdce56fd4ed6bd9bebc6aeb5a5",
+ "source": "0f170a4b2d5f04c153a9e7ddad478fe5b730abf0",
"version": "0.37.0",
- "url": "https://github.com/alexshpunt/cua/releases/download/pi-runtime-0.37.0-13dbacca/",
+ "url": "https://github.com/alexshpunt/cua/releases/download/pi-runtime-0.37.0-0f170a4b/",
"files": {
- "cua-driver.exe": "689487a1566b4cb04adee8b30f6732d4ae096290e53dd1d7570064e8a3265b67",
- "cua-driver-uia.exe": "ab0ece2a0aae0f357e3cf7db82b56f4e2d9d3e15d5c770d6c8beeea410885dba",
- "cua-cursor-theme.exe": "a6b3dc598ce3f95372d95ea2da72eb9f960c49fea24adbcb1664839d1528eb1a",
+ "cua-driver.exe": "fef81c7ac0a4412e9fa1d72ce802562933183f658ca2b0bc951084eb763b627e",
+ "cua-driver-uia.exe": "6387b1b0e6da7c61e9d2fbc5c3dd512d9a25f535129d1d4764578c1f58b0162d",
+ "cua-cursor-theme.exe": "ebaec44ccd472e108fce2771b271288e5e22f1b6f3ce341cd9e42e5c124468d5",
"LICENSE.md": "d0a32419a44fa38d5023d1431dfcbedd1457eb5e7b2e6c7c87ca064facb41b41",
"THIRD_PARTY_NOTICES.md": "88a24f59293f03b3e03655e8144bcfadc9861bef1182cfeea91639e11f573fd1"
}
diff --git a/src/ui-resources.ts b/src/ui-resources.ts
index 81c37d2..72b9694 100644
--- a/src/ui-resources.ts
+++ b/src/ui-resources.ts
@@ -16,7 +16,8 @@ export interface UiElement {
checked?: "on" | "off" | "mixed"; expanded?: boolean; offscreen?: boolean; focused?: boolean;
password?: boolean; read_only?: boolean; required?: boolean; actionable?: boolean;
range?: { value: number; min: number; max: number };
- hint_display: "pending" | "shown" | "disabled" | "no_bounds" | "outside_image" | "collision" | "off";
+ /** Whether the hint label is drawn on the overview; `read_only` marks static text and containers, which get no label. */
+ hint_display: "pending" | "shown" | "disabled" | "read_only" | "no_bounds" | "outside_image" | "collision" | "off";
}
/** Bounded native UIA projection. Completeness is not inferred from an empty search. */
export interface UiResource {
@@ -166,7 +167,10 @@ export async function annotateUi(image: Content, resource: UiResource, width: nu
return { image, annotation };
}
const canvas = { x: 0, y: 0, width, height };
+ // Labels keep off controls and static text, not off containers (groups, menu bars, title bars) they sit inside.
+ const containers = new Set(resource.elements.flatMap(e => e.parent_hint ? [e.parent_hint] : []));
const protectedRegions = obstacles.flatMap(e => {
+ if (e.actionable === false && containers.has(e.hint)) return [];
const r = e.bounds.window_image;
return r && r.width * r.height < width * height / 4 ? [r] : [];
});
@@ -174,6 +178,8 @@ export async function annotateUi(image: Content, resource: UiResource, width: nu
for (const e of resource.elements) {
if (!enabled) { e.hint_display = e.enabled === false ? "disabled" : e.bounds.window_image ? "off" : "no_bounds"; continue; }
if (e.enabled === false) { e.hint_display = "disabled"; continue; }
+ // Labels on every text and container would cover the controls; those stay addressable through UI and Search.
+ if (e.actionable === false) { e.hint_display = "read_only"; continue; }
const target = clipRect(e.bounds.window_image, canvas);
if (!target) { e.hint_display = e.bounds.window_image ? "outside_image" : "no_bounds"; continue; }
if (selected === e.hint) svg.push(`<rect x="${target.x}" y="${target.y}" width="${target.width}" height="${target.height}" fill="none" stroke="#e52678" stroke-width="2"/>`);
diff --git a/test/live/locators.live.ts b/test/live/locators.live.ts
index eea738f..d19e404 100644
--- a/test/live/locators.live.ts
+++ b/test/live/locators.live.ts
@@ -21,7 +21,12 @@ $c = New-Object Windows.Forms.CheckBox; $c.Text = 'Agree'; $c.Left = 10; $c.Top
$l = New-Object Windows.Forms.ListBox; $l.Left = 10; $l.Top = 120; $l.Height = 70; [void]$l.Items.AddRange(@('alpha','beta','gamma')); $l.AccessibleName = 'Pick'
$tm = New-Object Windows.Forms.Timer; $tm.Interval = 1500; $tm.Add_Tick({ $d.Enabled = $true; $tm.Stop() })
$f.Add_Shown({ $tm.Start() })
-$f.Controls.AddRange(@($b, $d, $t, $c, $l)); [void]$f.ShowDialog()`;
+$s = New-Object Windows.Forms.Label; $s.Left = 10; $s.Top = 200; $s.Width = 300; $s.Text = 'idle'
+$b.Add_Click({ $s.Text = "clicked $script:n" })
+$m = New-Object Windows.Forms.MenuStrip; $file = New-Object Windows.Forms.ToolStripMenuItem('File'); $reset = New-Object Windows.Forms.ToolStripMenuItem('Reset')
+$reset.Add_Click({ $script:n = 0; $b.Text = 'Count 0'; $s.Text = 'reset' }); [void]$file.DropDownItems.Add($reset); [void]$m.Items.Add($file); $f.MainMenuStrip = $m
+foreach ($x in @($b, $d, $t, $c, $l, $s)) { $x.Top += 30 }
+$f.Controls.AddRange(@($b, $d, $t, $c, $l, $s, $m)); [void]$f.ShowDialog()`;
test("a WinForms window is driven by Playwright locators, one call per step", async () => {
const window = await openWindow(live, live.machine, WINDOWS_POWERSHELL_PATH,
@@ -50,13 +55,21 @@ test("a WinForms window is driven by Playwright locators, one call per step", as
await expectStatus({ action: "verify", conditions: [
{ target: `getByRole("textbox", { name: "Name" })`, value: "Ada Lovelace" },
{ target: `getByRole("checkbox", { name: "Agree" })`, is: "checked" },
- { aria: `- textbox "Name": Ada Lovelace\n- checkbox "Agree" [checked]\n- listitem "beta" [selected]` },
+ { aria: `- textbox "Name": Ada Lovelace\n- checkbox "Agree" [checked]\n- list:\n - listitem "beta" [selected]` },
] }, "observed", /Verification satisfied/);
- // A count is proven only by a complete walk; a runtime that walks just actionable elements leaves it unknown.
- const counted = await step({ action: "verify", conditions: [{ target: `getByRole("listitem")`, count: 3 }] });
- const [count] = counted.details.verification!.conditions;
- expect(count.observed, presentComputerResult(counted)).toBe("3 match(es)");
- expect(counted.details.ui!.complete ? ["satisfied"] : ["unknown"]).toContain(count.verdict);
+ // The bundled runtime walks the whole structural tree: static text is there, and a complete walk proves counts and absence.
+ await expectStatus({ action: "verify", conditions: [
+ { target: `getByText("clicked 1")`, is: "visible" },
+ { target: `getByRole("listitem")`, count: 3 },
+ { target: `getByRole("button", { name: "Nope" })`, is: "absent" },
+ { aria: `- list "Pick":\n - listitem "alpha"\n - listitem "beta" [selected]\n - listitem "gamma"\n- text: clicked 1` },
+ ] }, "observed", /Verification satisfied/);
+ // A mismatch in a complete walk is proven, and the receipt shows the received tree.
+ await expectStatus({ action: "verify", conditions: [{ aria: `- button "Nope"` }] }, "observed", /Verification unsatisfied[\s\S]*Received aria snapshot:[\s\S]*- text: clicked 1/);
+ // A menu: open File, then pick Reset from its drop-down, both by locator.
+ await expectStatus({ action: "click", target: `getByRole("menuitem", { name: "File" })` }, "delivered");
+ await expectStatus({ action: "click", target: `getByRole("menuitem", { name: "Reset" })` }, "delivered");
+ await expectStatus({ action: "wait_for", conditions: [{ target: `getByText("reset", { exact: true })`, is: "visible" }, { target: `getByRole("button", { name: "Count 0" })`, is: "exists" }], timeout_ms: 3000 }, "observed", /Verification satisfied/);
// Strict mode refuses an ambiguous target and lists each match's locator; nothing is sent.
await expectStatus({ action: "click", target: `getByRole("button")` }, "refused", /Strict mode: getByRole\("button"\) matches \d+ elements/);
await expectStatus({ action: "click", target: `getByRole("button", { name: "Nope" })`, timeout_ms: 800 }, "refused", /Timed out after 800 ms/);
@@ -65,7 +78,9 @@ test("a WinForms window is driven by Playwright locators, one call per step", as
const aria = await live.read(window, ["aria"]);
const shown = presentComputerResult(aria);
console.log(shown, "\n", timings.join("\n"));
- expect(shown).toMatch(/- button "Count 1" \[ref=[^\]]+\]/);
+ expect(shown).toMatch(/- button "Count 0" \[ref=[^\]]+\]/);
+ expect(shown).toMatch(/- menuitem "File" \[ref=[^\]]+\]/);
+ expect(shown).toMatch(/- text: reset/);
expect(shown).toMatch(/- checkbox "Agree" \[checked\] \[ref=/);
expect(shown).toMatch(/- textbox "Name" \[ref=[^\]]+\]: Ada Lovelace/);
});
diff --git a/test/native-locator.test.ts b/test/native-locator.test.ts
index 565437f..5dc3b8f 100644
--- a/test/native-locator.test.ts
+++ b/test/native-locator.test.ts
@@ -23,6 +23,9 @@ const tree = { elements: [
element("status", "Text", "clicked 3"),
element("save", "Button", "Save", { help_text: "Save the file" }),
element("saveAs", "Button", "Save as…"),
+ // WinForms names a MenuStrip after the label before it; that name is a label, not text content.
+ element("bar", "MenuBar", "clicked 3"),
+ element("file", "MenuItem", "File", { parent_hint: "bar", depth: 2 }),
] };
// playwright-core loads on first use; under a parallel test run that can take several seconds.
beforeAll(() => { parseLocator("getByRole('button')"); }, 60_000);
@@ -50,6 +53,12 @@ test("role states filter like Playwright's options; scrolled-out elements still
test("text, label, test id and title locators read the matching UIA properties", () => {
expect(hints(`getByText("clicked")`)).toEqual(["status"]);
+ // Like Playwright: the smallest element whose text content matches, so a group matches by its caption and items by their own text.
+ expect(hints(`getByText("Options")`)).toEqual(["opts"]);
+ expect(hints(`getByText("OK")`)).toEqual(["ok"]);
+ expect(hints(`getByText("File")`)).toEqual(["file"]);
+ expect(hints(`getByRole("menubar").filter({ hasText: "File" })`)).toEqual(["bar"]);
+ expect(hints(`getByRole("menubar").filter({ hasText: "clicked" })`)).toEqual([]);
expect(hints(`getByText("Clicked 3", { exact: true })`)).toEqual([]);
expect(hints(`getByText("clicked 3", { exact: true })`)).toEqual(["status"]);
expect(hints(`getByLabel("Name")`)).toEqual(["name"]);
@@ -104,6 +113,8 @@ test("a native window renders as Playwright's own aria snapshot, with hints as r
` - text: clicked 3`,
` - button "Save" [ref=save]`,
` - button "Save as…" [ref=saveAs]`,
+ ` - menubar "clicked 3" [ref=bar]:`,
+ ` - menuitem "File" [ref=file]`,
].join("\n"));
});
diff --git a/test/ui-conditions.test.ts b/test/ui-conditions.test.ts
index fe91508..b94a653 100644
--- a/test/ui-conditions.test.ts
+++ b/test/ui-conditions.test.ts
@@ -4,7 +4,8 @@ import { combine, evaluateNative, validateWait } from "../src/ui-conditions.ts";
import type { UiElement, UiResource } from "../src/ui-resources.ts";
beforeAll(() => { parseLocator("getByRole('button')"); }, 60_000);
-const element = (fields: Partial<UiElement>): UiElement => ({ hint: "1:a", element_token: "t", index: 0, parent_hint: null, role: null, name: null, value: null,
+let serial = 0;
+const element = (fields: Partial<UiElement>): UiElement => ({ hint: `1:${serial++}`, element_token: "t", index: 0, parent_hint: null, role: null, name: null, value: null,
text_truncated: false, enabled: true, selected: null, actions: [], depth: 0, bounds: {}, hint_display: "off", ...fields });
const ui = (elements: UiElement[], complete: boolean | null): UiResource => ({ window_ref: "w", observation_id: "o", snapshot_id: null, captured_at: "",
complete, truncated: false, truncation_reason: null, native_total: elements.length, collected: elements.length, matches: elements.length, projection_truncated: false, elements });
@@ -18,7 +19,7 @@ test("native absence is decided only by a complete walk; a missing element in a
test("native checks compare reported properties and keep ambiguity and missing properties unknown", () => {
const tree = ui([element({ role: "Button", name: "Save", enabled: false }), element({ role: "Text", name: "Status", value: "saved Ada" }),
- element({ role: "Button", name: "A" }), element({ role: "Button", name: "B" }), element({ role: "Edit", name: "Long", value: "x", text_truncated: true })], true);
+ element({ role: "Button", name: "A" }), element({ role: "Button", name: "B" }), element({ role: "Text", name: "Long", value: "x", text_truncated: true })], true);
const check = (condition: object) => evaluateNative(tree, validateWait({ action: "verify", conditions: [condition] }, false).conditions)[0];
expect(check({ target: { role: "Button", name: "Save" }, is: "disabled" }).verdict).toBe("satisfied");
expect(check({ target: { role: "Button", name: "Save" }, is: "enabled" }).verdict).toBe("unsatisfied");
diff --git a/test/ui-resources.test.ts b/test/ui-resources.test.ts
index 06237fa..0cddc92 100644
--- a/test/ui-resources.test.ts
+++ b/test/ui-resources.test.ts
@@ -111,6 +111,28 @@ test("render status describes this image, not a previous annotation", async () =
expect(resource.elements[2].hint_display).toBe("disabled"); expect(resource.elements[3].hint_display).toBe("no_bounds");
});
+test("static text and containers from a full UIA tree get no hint label on the overview", async () => {
+ const { annotateUi } = await import("../src/ui-resources.ts");
+ const f = await fixture(); const view = await f.session.observe(f.ref); const resource = structuredClone(ui(view));
+ resource.elements[1].actionable = false;
+ const result = await annotateUi({ type: "image", data: f.raw.toString("base64"), mimeType: "image/png" }, resource, 500, 320, true);
+ expect(resource.elements[1].hint_display).toBe("read_only");
+ expect(result.annotation.labels.map(label => label.hint)).not.toContain(resource.elements[1].hint);
+});
+
+test("a read-only container does not push the labels of the controls inside it off the overview", async () => {
+ const { annotateUi } = await import("../src/ui-resources.ts");
+ const f = await fixture(); const view = await f.session.observe(f.ref); const resource = structuredClone(ui(view));
+ const control = resource.elements[1];
+ const box = control.bounds.window_image!;
+ // A group whose frame surrounds the control with no room for a label outside it.
+ const group = { ...structuredClone(control), hint: "g:1", actionable: false, parent_hint: null,
+ bounds: { window_image: { x: Math.max(0, box.x - 40), y: Math.max(0, box.y - 40), width: box.width + 80, height: box.height + 80 } } };
+ resource.elements = [group, { ...control, parent_hint: "g:1", actionable: true }];
+ const result = await annotateUi({ type: "image", data: f.raw.toString("base64"), mimeType: "image/png" }, resource, 500, 320, true);
+ expect(result.annotation.labels.map(label => label.hint)).toEqual([control.hint]);
+});
+
test("scoped label placement respects controls excluded from the projection", async () => {
const { annotateUi, projectUi } = await import("../src/ui-resources.ts");
const f = await fixture(); const view = await f.session.observe(f.ref); const resource = structuredClone(ui(view));