Skip to content

An agentic browser harness: an AI side panel & orb in your own logged-in Chrome, Edge, Arc, Brave or even Firefox run by the agent/subscription you already use (Claude Code, Codex, Antigravity). Free, open source, no API key. Also an MCP server.

perf(getPageInfo): say the layout once, as the diagram, and leave bounds and implied roles out of the inventory unless asked

Shahriar74b08cfabc33
diff --git a/docs/guide/features/page-actions.md b/docs/guide/features/page-actions.md
index 348b6d1..409f26c 100644
--- a/docs/guide/features/page-actions.md
+++ b/docs/guide/features/page-actions.md
@@ -21,7 +21,7 @@ by **visible text** survives redesigns that break CSS paths.
| | |
| --- | --- |
-| `page_getPageInfo` | The workhorse. Document metadata, viewport and scroll state, a semantic layout tree with a text diagram, the heading outline, and an inventory of every interactive element — each with its selector, its ARIA role, its live state (disabled, checked, expanded, filled, `aria-current`) and the landmark region it sits in |
+| `page_getPageInfo` | The workhorse. Document metadata, viewport and scroll state, a text diagram of the landmark regions with a selector for each, the heading outline, and an inventory of every interactive element — each with its selector, its ARIA role, its live state (disabled, checked, expanded, filled, `aria-current`) and the landmark region it sits in |
| `page_extractText` | Rendered text of an element or the whole page, a sentence-aligned group at a time — long pages come back through a cursor rather than truncated. Raw HTML is [denied by default](../approvals.md) |
| `page_waitForElement` | Wait until an element is attached, visible, hidden or detached |
| `page_findProgress` | Scan for progress signals worth [monitoring](monitoring.md) |
diff --git a/docs/reference/tools.md b/docs/reference/tools.md
index 404a203..2af0ce0 100644
--- a/docs/reference/tools.md
+++ b/docs/reference/tools.md
@@ -74,8 +74,8 @@ a page tool fails. No parameters.
### page_getPageInfo
-Snapshot the current page: document metadata, viewport and scroll state, a semantic layout tree
-with a text diagram, the heading outline, and an inventory of interactive elements — each with a
+Snapshot the current page: document metadata, viewport and scroll state, a text diagram of the
+landmark regions with a selector for each, the heading outline, and an inventory of interactive elements — each with a
stable selector already computed. The workhorse; start here.
Every element in the inventory carries three things beyond its selector, so you can decide what to
@@ -83,18 +83,20 @@ touch without a second call:
| Field | What it tells you |
| --- | --- |
-| `role` | The computed ARIA role — `link`, `button`, `textbox`, `combobox`, `checkbox`, `tab`, whatever the page declares. The same vocabulary the `role` field of a [target](#element-targets) accepts. |
+| `role` | The computed ARIA role — `link`, `button`, `textbox`, `combobox`, `checkbox`, `tab`, whatever the page declares. The same vocabulary the `role` field of a [target](#element-targets) accepts. Left out, along with `tag`, where the list already says it: an `<a>` under `links` is a link, a `<button>` under `buttons` is a button. |
| `state` | Only the keys that apply: `disabled`, `checked`, `expanded`, `selected`, `required`, `invalid`, `current` (from `aria-current`, which marks the page you are on), `filled` for text inputs, and `value` for the selected option of a `<select>`. Field contents are never reported — `filled` says whether something is typed, not what. |
| `region` | The landmark it lives in, e.g. `navigation “Primary”` or `form “Checkout”`. Use it to tell the main content's “Delete” from the sidebar's. |
Elements hidden from assistive technology — anything inside `aria-hidden="true"` or `inert` — are
-left out of the tree, the outline and the inventory. Each region in the diagram is annotated with
-how many links, buttons and fields its subtree holds, and `interactive.counts` reports the true
+left out of the diagram, the outline and the inventory. Each region in the diagram is annotated with
+how many links, buttons and fields its subtree holds and ends with its selector, which scopes a
+`page_extractText` to that region, and `interactive.counts` reports the true
totals before `maxPerKind` truncates the lists.
| Parameter | Type | Default | Purpose |
| --- | --- | --- | --- |
| `maxPerKind` | integer | `30` | Cap on links, buttons, fields, and forms listed per kind |
+| `geometry` | boolean | `false` | Add each element's `bounds` in document pixels. Ask only when you need coordinates, such as a `point` for `page_trustedClick` |
### page_extractText
@@ -1006,7 +1008,7 @@ tab at the moment it is fetched.
| Resource | Type | What it returns |
| --- | --- | --- |
| `browsentic://page/diagram` | `text/plain` | Text diagram of the page's landmark regions — the cheapest useful view of a page |
-| `browsentic://page/current` | `application/json` | The full `page_getPageInfo` snapshot: metadata, layout tree, headings, interactive inventory |
+| `browsentic://page/current` | `application/json` | The full `page_getPageInfo` snapshot: metadata, layout diagram, headings, interactive inventory |
| `browsentic://page/text` | `text/plain` | The rendered text of the page — the first `page_extractText` group, with no way to page past it |
---
diff --git a/src/daemon/server.ts b/src/daemon/server.ts
index a97c00c..75b321d 100644
--- a/src/daemon/server.ts
+++ b/src/daemon/server.ts
@@ -29,7 +29,7 @@ const RESOURCES = [
{
uri: 'browsentic://page/current',
name: 'Active page snapshot',
- description: 'Full page.getPageInfo snapshot of the active tab: metadata, layout tree, headings, interactive inventory.',
+ description: 'Full page.getPageInfo snapshot of the active tab: metadata, layout diagram, headings, interactive inventory.',
mimeType: 'application/json',
},
{
diff --git a/src/lib/actions/page/get-page-info.test.ts b/src/lib/actions/page/get-page-info.test.ts
new file mode 100644
index 0000000..71f9044
--- /dev/null
+++ b/src/lib/actions/page/get-page-info.test.ts
@@ -0,0 +1,63 @@
+import { beforeEach, describe, expect, test, vi } from 'vitest';
+import { getPageInfo } from './get-page-info';
+
+const PAGE = `
+ <header><nav aria-label="Primary"><a href="/pricing">Pricing</a></nav></header>
+ <main>
+ <h1>Sign in</h1>
+ <form id="login">
+ <input id="email" type="email" aria-label="Email" />
+ <button type="submit">Continue</button>
+ <div role="button">More options</div>
+ </form>
+ </main>`;
+
+const snapshot = (input: { geometry?: boolean } = {}) =>
+ getPageInfo.execute(getPageInfo.input.parse(input)) as {
+ layout: Record<string, unknown> & { diagram: string };
+ interactive: Record<'links' | 'buttons' | 'fields', Record<string, unknown>[]>;
+ };
+
+beforeEach(() => {
+ document.body.innerHTML = PAGE;
+ // happy-dom lays nothing out, and an element with no box counts as hidden.
+ vi.spyOn(Element.prototype, 'getBoundingClientRect').mockReturnValue(new DOMRect(10, 20, 300, 40));
+});
+
+describe('the layout', () => {
+ test('is the diagram alone, with each region ending in its selector', () => {
+ const { layout } = snapshot();
+ expect(Object.keys(layout)).toEqual(['diagram']);
+ expect(layout.diagram.split('\n').slice(1)).toEqual([
+ '├ banner · 300×40 @ (10,20) · 1 links · selector: body > header',
+ '│ └ navigation “Primary” · 300×40 @ (10,20) · 1 links · selector: body > header > nav',
+ '└ main “Sign in” · 300×40 @ (10,20) · 2 buttons · 1 fields · selector: body > main',
+ ' └ form · 300×40 @ (10,20) · 2 buttons · 1 fields · selector: #login',
+ ]);
+ });
+});
+
+describe('the inventory', () => {
+ test('leaves out the tag and role its list already implies, and keeps the ones it does not', () => {
+ const { links, buttons, fields } = snapshot().interactive;
+ expect([links, buttons, fields]).toEqual([
+ [{ selector: 'body > header > nav > a', text: 'Pricing', href: expect.stringContaining('/pricing'), region: 'navigation “Primary”' }],
+ [
+ { selector: '#login > button', text: 'Continue', region: 'form' },
+ { tag: 'div', role: 'button', selector: '#login > div', text: 'More options', region: 'form' },
+ ],
+ [expect.objectContaining({ tag: 'input', role: 'textbox', selector: '#email', kind: 'email' })],
+ ]);
+ });
+
+ test('carries bounds only when geometry is asked for', () => {
+ const bounds = (input: { geometry?: boolean }) => snapshot(input).interactive.buttons.map((button) => button.bounds);
+ expect([bounds({}), bounds({ geometry: true })]).toEqual([
+ [undefined, undefined],
+ [
+ { x: 10, y: 20, width: 300, height: 40 },
+ { x: 10, y: 20, width: 300, height: 40 },
+ ],
+ ]);
+ });
+});
diff --git a/src/lib/actions/page/get-page-info.ts b/src/lib/actions/page/get-page-info.ts
index b9917f8..a97a92a 100644
--- a/src/lib/actions/page/get-page-info.ts
+++ b/src/lib/actions/page/get-page-info.ts
@@ -44,7 +44,7 @@ const LANDMARK_ROLES = new Set([
export const getPageInfo = defineAction({
name: 'page.getPageInfo',
description:
- 'Snapshot the current page: document metadata, viewport and scroll state, a semantic layout tree with a text diagram, the heading outline, and an inventory of interactive elements — each carrying its ARIA role, its live state (disabled, checked, expanded, filled, aria-current) and the landmark region it sits in. When the site registers WebMCP tools, the result also carries a siteTools list — prefer page.callSiteTool over clicking wherever a listed tool covers the step. Visible iframes come back under "frames"; nothing inside one is in this snapshot until page.switchFrame enters it, and "frame" then says which one is in focus.',
+ 'Snapshot the current page: document metadata, viewport and scroll state, a text diagram of the landmark regions with a selector for each, the heading outline, and an inventory of interactive elements — each carrying its ARIA role, its live state (disabled, checked, expanded, filled, aria-current) and the landmark region it sits in. When the site registers WebMCP tools, the result also carries a siteTools list — prefer page.callSiteTool over clicking wherever a listed tool covers the step. Visible iframes come back under "frames"; nothing inside one is in this snapshot until page.switchFrame enters it, and "frame" then says which one is in focus.',
input: z.object({
maxPerKind: z
.number()
@@ -52,8 +52,14 @@ export const getPageInfo = defineAction({
.positive()
.default(30)
.describe('Cap on links, buttons, fields, and forms listed per kind'),
+ geometry: z
+ .boolean()
+ .default(false)
+ .describe(
+ 'Add each element’s "bounds" in document pixels. Off by default: a selector or visible text is what the other tools target by, so ask only when you need coordinates — a point for page.trustedClick or page.dragElement.',
+ ),
}),
- execute({ maxPerKind }) {
+ execute({ maxPerKind, geometry }) {
const { regions, owners } = layoutTree();
const found = collect();
tally(found, owners);
@@ -74,7 +80,7 @@ export const getPageInfo = defineAction({
pageHeight: document.documentElement.scrollHeight,
},
selection: getSelection()?.toString().slice(0, 500) || undefined,
- layout: { regions, diagram: renderDiagram(regions) },
+ layout: { diagram: renderDiagram(regions) },
outline: [...document.querySelectorAll<HTMLElement>('h1,h2,h3,h4,h5,h6')]
.filter(isExposed)
.slice(0, 60)
@@ -82,15 +88,15 @@ export const getPageInfo = defineAction({
level: Number(heading.tagName[1]),
text: accessibleText(heading).slice(0, 120),
})),
- interactive: inventory(found, owners, maxPerKind),
- frames: embeddedFrames(),
+ interactive: inventory(found, owners, maxPerKind, geometry),
+ frames: embeddedFrames(geometry),
};
},
});
const MAX_FRAMES = 20;
-function embeddedFrames() {
+function embeddedFrames(geometry: boolean) {
const frames = [...document.querySelectorAll('iframe,frame')]
.filter(isFrameElement)
.filter(isExposed)
@@ -101,7 +107,7 @@ function embeddedFrames() {
name: frame.name || undefined,
title: frame.title || undefined,
sandbox: sandboxOf(frame),
- bounds: documentBounds(frame),
+ bounds: geometry ? documentBounds(frame) : undefined,
}));
return frames.length ? frames : undefined;
}
@@ -174,7 +180,7 @@ function renderDiagram(regions: Region[]): string {
.map((kind) => ` · ${contains[kind]} ${kind}`)
.join('');
lines.push(
- `${indent}${last ? '└' : '├'} ${regionName(region)} · ${bounds.width}×${bounds.height} @ (${bounds.x},${bounds.y})${counts}`,
+ `${indent}${last ? '└' : '├'} ${regionName(region)} · ${bounds.width}×${bounds.height} @ (${bounds.x},${bounds.y})${counts} · selector: ${region.selector}`,
);
walk(region.children, indent + (last ? ' ' : '│ '));
});
@@ -209,23 +215,29 @@ function inventory(
found: ReturnType<typeof collect>,
owners: Map<HTMLElement, Region>,
cap: number,
+ geometry: boolean,
) {
const regionFor = (el: HTMLElement) => {
const owner = ownerOf(el, owners);
return owner && regionName(owner);
};
+ const entry = (el: HTMLElement, implied?: { tag: string; role: string }) => {
+ const { tag, role, bounds, ...rest } = describeElement(el);
+ const said = implied?.tag === tag && implied.role === role;
+ return { ...(said ? {} : { tag, role }), ...rest, ...(geometry ? { bounds } : {}) };
+ };
return {
links: found.links.slice(0, cap).map((link) => ({
- ...describeElement(link),
+ ...entry(link, { tag: 'a', role: 'link' }),
href: link.href,
region: regionFor(link),
})),
buttons: found.buttons.slice(0, cap).map((button) => ({
- ...describeElement(button),
+ ...entry(button, { tag: 'button', role: 'button' }),
region: regionFor(button),
})),
fields: found.fields.slice(0, cap).map((field) => ({
- ...describeElement(field),
+ ...entry(field),
kind: field instanceof HTMLInputElement ? field.type : field.tagName.toLowerCase(),
region: regionFor(field),
})),