An agentic browser harness: an AI side panel & orb in your own logged-in Chrome, Edge, Arc, Brave or even Firefox run by the agent/subscription you already use (Claude Code, Codex, Antigravity). Free, open source, no API key. Also an MCP server.
TypeScript agentic-browseragentic-browsingagy-cliaiai-agentai-browser-extensionai-toolsbrave-extensionbrowser-agentbrowser-automationbrowser-extensionbrowser-usebrowserskillchrome-extensionclaude-codecodexfirefox-addonmcp-servernanobrowserweb-automation
perf(getPageInfo): say the layout once, as the diagram, and leave bounds and implied roles out of the inventory unless asked
Shahriar74b08cfabc33
diff --git a/docs/guide/features/page-actions.md b/docs/guide/features/page-actions.mdindex 348b6d1..409f26c 100644--- a/docs/guide/features/page-actions.md+++ b/docs/guide/features/page-actions.md@@ -21,7 +21,7 @@ by **visible text** survives redesigns that break CSS paths.| | || --- | --- |-| `page_getPageInfo` | The workhorse. Document metadata, viewport and scroll state, a semantic layout tree with a text diagram, the heading outline, and an inventory of every interactive element — each with its selector, its ARIA role, its live state (disabled, checked, expanded, filled, `aria-current`) and the landmark region it sits in |+| `page_getPageInfo` | The workhorse. Document metadata, viewport and scroll state, a text diagram of the landmark regions with a selector for each, the heading outline, and an inventory of every interactive element — each with its selector, its ARIA role, its live state (disabled, checked, expanded, filled, `aria-current`) and the landmark region it sits in || `page_extractText` | Rendered text of an element or the whole page, a sentence-aligned group at a time — long pages come back through a cursor rather than truncated. Raw HTML is [denied by default](../approvals.md) || `page_waitForElement` | Wait until an element is attached, visible, hidden or detached || `page_findProgress` | Scan for progress signals worth [monitoring](monitoring.md) |diff --git a/docs/reference/tools.md b/docs/reference/tools.mdindex 404a203..2af0ce0 100644--- a/docs/reference/tools.md+++ b/docs/reference/tools.md@@ -74,8 +74,8 @@ a page tool fails. No parameters.### page_getPageInfo-Snapshot the current page: document metadata, viewport and scroll state, a semantic layout tree-with a text diagram, the heading outline, and an inventory of interactive elements — each with a+Snapshot the current page: document metadata, viewport and scroll state, a text diagram of the+landmark regions with a selector for each, the heading outline, and an inventory of interactive elements — each with astable selector already computed. The workhorse; start here.Every element in the inventory carries three things beyond its selector, so you can decide what to@@ -83,18 +83,20 @@ touch without a second call:| Field | What it tells you || --- | --- |-| `role` | The computed ARIA role — `link`, `button`, `textbox`, `combobox`, `checkbox`, `tab`, whatever the page declares. The same vocabulary the `role` field of a [target](#element-targets) accepts. |+| `role` | The computed ARIA role — `link`, `button`, `textbox`, `combobox`, `checkbox`, `tab`, whatever the page declares. The same vocabulary the `role` field of a [target](#element-targets) accepts. Left out, along with `tag`, where the list already says it: an `<a>` under `links` is a link, a `<button>` under `buttons` is a button. || `state` | Only the keys that apply: `disabled`, `checked`, `expanded`, `selected`, `required`, `invalid`, `current` (from `aria-current`, which marks the page you are on), `filled` for text inputs, and `value` for the selected option of a `<select>`. Field contents are never reported — `filled` says whether something is typed, not what. || `region` | The landmark it lives in, e.g. `navigation “Primary”` or `form “Checkout”`. Use it to tell the main content's “Delete” from the sidebar's. |Elements hidden from assistive technology — anything inside `aria-hidden="true"` or `inert` — are-left out of the tree, the outline and the inventory. Each region in the diagram is annotated with-how many links, buttons and fields its subtree holds, and `interactive.counts` reports the true+left out of the diagram, the outline and the inventory. Each region in the diagram is annotated with+how many links, buttons and fields its subtree holds and ends with its selector, which scopes a+`page_extractText` to that region, and `interactive.counts` reports the truetotals before `maxPerKind` truncates the lists.| Parameter | Type | Default | Purpose || --- | --- | --- | --- || `maxPerKind` | integer | `30` | Cap on links, buttons, fields, and forms listed per kind |+| `geometry` | boolean | `false` | Add each element's `bounds` in document pixels. Ask only when you need coordinates, such as a `point` for `page_trustedClick` |### page_extractText@@ -1006,7 +1008,7 @@ tab at the moment it is fetched.| Resource | Type | What it returns || --- | --- | --- || `browsentic://page/diagram` | `text/plain` | Text diagram of the page's landmark regions — the cheapest useful view of a page |-| `browsentic://page/current` | `application/json` | The full `page_getPageInfo` snapshot: metadata, layout tree, headings, interactive inventory |+| `browsentic://page/current` | `application/json` | The full `page_getPageInfo` snapshot: metadata, layout diagram, headings, interactive inventory || `browsentic://page/text` | `text/plain` | The rendered text of the page — the first `page_extractText` group, with no way to page past it |---diff --git a/src/daemon/server.ts b/src/daemon/server.tsindex a97c00c..75b321d 100644--- a/src/daemon/server.ts+++ b/src/daemon/server.ts@@ -29,7 +29,7 @@ const RESOURCES = [{uri: 'browsentic://page/current',name: 'Active page snapshot',- description: 'Full page.getPageInfo snapshot of the active tab: metadata, layout tree, headings, interactive inventory.',+ description: 'Full page.getPageInfo snapshot of the active tab: metadata, layout diagram, headings, interactive inventory.',mimeType: 'application/json',},{diff --git a/src/lib/actions/page/get-page-info.test.ts b/src/lib/actions/page/get-page-info.test.tsnew file mode 100644index 0000000..71f9044--- /dev/null+++ b/src/lib/actions/page/get-page-info.test.ts@@ -0,0 +1,63 @@+import { beforeEach, describe, expect, test, vi } from 'vitest';+import { getPageInfo } from './get-page-info';++const PAGE = `+ <header><nav aria-label="Primary"><a href="/pricing">Pricing</a></nav></header>+ <main>+ <h1>Sign in</h1>+ <form id="login">+ <input id="email" type="email" aria-label="Email" />+ <button type="submit">Continue</button>+ <div role="button">More options</div>+ </form>+ </main>`;++const snapshot = (input: { geometry?: boolean } = {}) =>+ getPageInfo.execute(getPageInfo.input.parse(input)) as {+ layout: Record<string, unknown> & { diagram: string };+ interactive: Record<'links' | 'buttons' | 'fields', Record<string, unknown>[]>;+ };++beforeEach(() => {+ document.body.innerHTML = PAGE;+ // happy-dom lays nothing out, and an element with no box counts as hidden.+ vi.spyOn(Element.prototype, 'getBoundingClientRect').mockReturnValue(new DOMRect(10, 20, 300, 40));+});++describe('the layout', () => {+ test('is the diagram alone, with each region ending in its selector', () => {+ const { layout } = snapshot();+ expect(Object.keys(layout)).toEqual(['diagram']);+ expect(layout.diagram.split('\n').slice(1)).toEqual([+ '├ banner · 300×40 @ (10,20) · 1 links · selector: body > header',+ '│ └ navigation “Primary” · 300×40 @ (10,20) · 1 links · selector: body > header > nav',+ '└ main “Sign in” · 300×40 @ (10,20) · 2 buttons · 1 fields · selector: body > main',+ ' └ form · 300×40 @ (10,20) · 2 buttons · 1 fields · selector: #login',+ ]);+ });+});++describe('the inventory', () => {+ test('leaves out the tag and role its list already implies, and keeps the ones it does not', () => {+ const { links, buttons, fields } = snapshot().interactive;+ expect([links, buttons, fields]).toEqual([+ [{ selector: 'body > header > nav > a', text: 'Pricing', href: expect.stringContaining('/pricing'), region: 'navigation “Primary”' }],+ [+ { selector: '#login > button', text: 'Continue', region: 'form' },+ { tag: 'div', role: 'button', selector: '#login > div', text: 'More options', region: 'form' },+ ],+ [expect.objectContaining({ tag: 'input', role: 'textbox', selector: '#email', kind: 'email' })],+ ]);+ });++ test('carries bounds only when geometry is asked for', () => {+ const bounds = (input: { geometry?: boolean }) => snapshot(input).interactive.buttons.map((button) => button.bounds);+ expect([bounds({}), bounds({ geometry: true })]).toEqual([+ [undefined, undefined],+ [+ { x: 10, y: 20, width: 300, height: 40 },+ { x: 10, y: 20, width: 300, height: 40 },+ ],+ ]);+ });+});diff --git a/src/lib/actions/page/get-page-info.ts b/src/lib/actions/page/get-page-info.tsindex b9917f8..a97a92a 100644--- a/src/lib/actions/page/get-page-info.ts+++ b/src/lib/actions/page/get-page-info.ts@@ -44,7 +44,7 @@ const LANDMARK_ROLES = new Set([export const getPageInfo = defineAction({name: 'page.getPageInfo',description:- 'Snapshot the current page: document metadata, viewport and scroll state, a semantic layout tree with a text diagram, the heading outline, and an inventory of interactive elements — each carrying its ARIA role, its live state (disabled, checked, expanded, filled, aria-current) and the landmark region it sits in. When the site registers WebMCP tools, the result also carries a siteTools list — prefer page.callSiteTool over clicking wherever a listed tool covers the step. Visible iframes come back under "frames"; nothing inside one is in this snapshot until page.switchFrame enters it, and "frame" then says which one is in focus.',+ 'Snapshot the current page: document metadata, viewport and scroll state, a text diagram of the landmark regions with a selector for each, the heading outline, and an inventory of interactive elements — each carrying its ARIA role, its live state (disabled, checked, expanded, filled, aria-current) and the landmark region it sits in. When the site registers WebMCP tools, the result also carries a siteTools list — prefer page.callSiteTool over clicking wherever a listed tool covers the step. Visible iframes come back under "frames"; nothing inside one is in this snapshot until page.switchFrame enters it, and "frame" then says which one is in focus.',input: z.object({maxPerKind: z.number()@@ -52,8 +52,14 @@ export const getPageInfo = defineAction({.positive().default(30).describe('Cap on links, buttons, fields, and forms listed per kind'),+ geometry: z+ .boolean()+ .default(false)+ .describe(+ 'Add each element’s "bounds" in document pixels. Off by default: a selector or visible text is what the other tools target by, so ask only when you need coordinates — a point for page.trustedClick or page.dragElement.',+ ),}),- execute({ maxPerKind }) {+ execute({ maxPerKind, geometry }) {const { regions, owners } = layoutTree();const found = collect();tally(found, owners);@@ -74,7 +80,7 @@ export const getPageInfo = defineAction({pageHeight: document.documentElement.scrollHeight,},selection: getSelection()?.toString().slice(0, 500) || undefined,- layout: { regions, diagram: renderDiagram(regions) },+ layout: { diagram: renderDiagram(regions) },outline: [...document.querySelectorAll<HTMLElement>('h1,h2,h3,h4,h5,h6')].filter(isExposed).slice(0, 60)@@ -82,15 +88,15 @@ export const getPageInfo = defineAction({level: Number(heading.tagName[1]),text: accessibleText(heading).slice(0, 120),})),- interactive: inventory(found, owners, maxPerKind),- frames: embeddedFrames(),+ interactive: inventory(found, owners, maxPerKind, geometry),+ frames: embeddedFrames(geometry),};},});const MAX_FRAMES = 20;-function embeddedFrames() {+function embeddedFrames(geometry: boolean) {const frames = [...document.querySelectorAll('iframe,frame')].filter(isFrameElement).filter(isExposed)@@ -101,7 +107,7 @@ function embeddedFrames() {name: frame.name || undefined,title: frame.title || undefined,sandbox: sandboxOf(frame),- bounds: documentBounds(frame),+ bounds: geometry ? documentBounds(frame) : undefined,}));return frames.length ? frames : undefined;}@@ -174,7 +180,7 @@ function renderDiagram(regions: Region[]): string {.map((kind) => ` · ${contains[kind]} ${kind}`).join('');lines.push(- `${indent}${last ? '└' : '├'} ${regionName(region)} · ${bounds.width}×${bounds.height} @ (${bounds.x},${bounds.y})${counts}`,+ `${indent}${last ? '└' : '├'} ${regionName(region)} · ${bounds.width}×${bounds.height} @ (${bounds.x},${bounds.y})${counts} · selector: ${region.selector}`,);walk(region.children, indent + (last ? ' ' : '│ '));});@@ -209,23 +215,29 @@ function inventory(found: ReturnType<typeof collect>,owners: Map<HTMLElement, Region>,cap: number,+ geometry: boolean,) {const regionFor = (el: HTMLElement) => {const owner = ownerOf(el, owners);return owner && regionName(owner);};+ const entry = (el: HTMLElement, implied?: { tag: string; role: string }) => {+ const { tag, role, bounds, ...rest } = describeElement(el);+ const said = implied?.tag === tag && implied.role === role;+ return { ...(said ? {} : { tag, role }), ...rest, ...(geometry ? { bounds } : {}) };+ };return {links: found.links.slice(0, cap).map((link) => ({- ...describeElement(link),+ ...entry(link, { tag: 'a', role: 'link' }),href: link.href,region: regionFor(link),})),buttons: found.buttons.slice(0, cap).map((button) => ({- ...describeElement(button),+ ...entry(button, { tag: 'button', role: 'button' }),region: regionFor(button),})),fields: found.fields.slice(0, cap).map((field) => ({- ...describeElement(field),+ ...entry(field),kind: field instanceof HTMLInputElement ? field.type : field.tagName.toLowerCase(),region: regionFor(field),})),