An agentic browser harness: an AI side panel & orb in your own logged-in Chrome, Edge, Arc, Brave or even Firefox run by the agent/subscription you already use (Claude Code, Codex, Antigravity). Free, open source, no API key. Also an MCP server.
TypeScript agentic-browseragentic-browsingagy-cliaiai-agentai-browser-extensionai-toolsbrave-extensionbrowser-agentbrowser-automationbrowser-extensionbrowser-usebrowserskillchrome-extensionclaude-codecodexfirefox-addonmcp-servernanobrowserweb-automation
feat(a-eye): picks carry a photograph of the element
Shahriar8f4548de2843
diff --git a/docs/guide/features/a-eye.md b/docs/guide/features/a-eye.mdindex c2446ed..ee5ce55 100644--- a/docs/guide/features/a-eye.md+++ b/docs/guide/features/a-eye.md@@ -10,9 +10,10 @@ Click the **A-Eye** button — the lens in the side panel's composer row. Your cand whatever you hover on the page is outlined as you move. Click to pick it. `↑` widens the pick tothe parent element when you land inside something smaller than you meant; `Esc` cancels.-The pick appears as a chip above the composer, and it goes out with your **next message** — the-element's tag, its role, its selector and the text it held at that moment. The chip clears when the-message sends, so one pick scopes one message. The `×` on the chip drops it without sending.+The pick appears as a chip above the composer — with a small thumbnail of what you picked — and it+goes out with your **next message**: the element's tag, its role, its selector, the text it held at+that moment, and a photograph of it exactly as you saw it. The chip clears when the message sends,+so one pick scopes one message. The `×` on the chip drops it without sending.The site never sees the pick. Every pointer event in the sequence is stopped before the page gets itand the click itself is cancelled, so picking a "Delete" button picks it rather than pressing it.@@ -22,7 +23,8 @@ and the click itself is cancelled, so picking a "Delete" button picks it ratherThe element arrives in the run's system prompt as the **subject of the instruction**. Ask "what doesthis say?" and the answer is about that element; say "translate this" and only that gets translated.It re-reads the element live before acting on it, because a page can change between your pick and-its first tool call.+its first tool call — and when looks matter, it can view the photograph taken at the instant you+picked, even if the page has since moved on.Words win over the pick when they clearly point elsewhere. "Now go to checkout" has left the elementbehind; "compare this with the one below" is about two. The pick resolves ambiguity — it does not@@ -45,9 +47,11 @@ agent is told to ask in words instead, not to ask you to point again.- **One tab.** The lens opens on the tab that was in front when you pressed the button. Switchingtabs mid-pick leaves it behind on the old one; it times out after a minute.+- **One lens at a time.** Starting a new pick — from the button, or the agent asking — dismisses a+ lens already waiting, and whoever opened that one is told it was cancelled.- **Top frame only.** An element inside an iframe picks the iframe, not what is in it.-- **Text, not pixels.** A pick carries the element's rendered text, not an image of it. For layout- or rendering, ask for a [screenshot](screenshots.md) as well.+- **The element, not the page.** The photograph covers what you picked, with a sliver of margin.+ For the page around it, ask for a [screenshot](screenshots.md) as well.- **Long elements are cut.** Picking a whole article sends the first couple of thousand characters;the agent reads the rest through `page_extractText` scoped to the selector.diff --git a/docs/reference/errors.md b/docs/reference/errors.mdindex 605bc97..438d6c0 100644--- a/docs/reference/errors.md+++ b/docs/reference/errors.md@@ -34,7 +34,8 @@ repeatedly: failures carry the *fix* in the message, and a failed tool call neve| `UNSUPPORTED` | Action | e.g. a non-http(s) URL || `TIMEOUT` | Link or action | A wait expired, or the extension did not answer in the window for that action || `ACTION_FAILED` | Action | `execute()` threw — e.g. `back` with no history |-| `PICK_CANCELLED` | Content script | The user dismissed A-Eye without pointing at anything. **Terminal** — ask in words rather than asking them to point again |+| `PICK_CANCELLED` | Content script | The user dismissed A-Eye without pointing at anything — or a newer pick took over the lens. **Terminal** — ask in words rather than asking them to point again |+| `NO_FOCUS_SHOT` | Daemon | `browsentic_focusShot` was called on a run whose instruction carried no A-Eye pick — there is nothing photographed to show || `UNKNOWN_ACTION` | Registry / daemon | Tool-registry skew, or a reserved action reached from outside |## Diagnosticsdiff --git a/docs/reference/tools.md b/docs/reference/tools.mdindex ae33578..2acd828 100644--- a/docs/reference/tools.md+++ b/docs/reference/tools.md@@ -183,7 +183,8 @@ record where a site's search lives. `page_searchSite` is not — it navigates.Hand the page to the user and let them point at one element — **A-Eye**. Their cursor becomes alens, whatever they hover is outlined, and the element they click comes back as a-[described element](#element-targets) plus its rendered text.+[described element](#element-targets) plus its rendered text and a screenshot of it exactly as+they saw it. A new call dismisses a pick already waiting — only one lens can hold the page.It stops everything and waits for a person, so it costs more than any other tool here: reach for itonly when a target is genuinely ambiguous and pointing is quicker than describing. `hint` is thediff --git a/src/daemon/agent/service.ts b/src/daemon/agent/service.tsindex 17d9463..5dc9214 100644--- a/src/daemon/agent/service.ts+++ b/src/daemon/agent/service.ts@@ -11,7 +11,8 @@ import {type RunEvent,} from '@/lib/actions/protocol';import { openTab } from '@/lib/actions/page/open-tab';-import { READ_SITEMAP_ACTION } from '@/lib/actions/reserved';+import { FOCUS_SHOT_ACTION, READ_SITEMAP_ACTION } from '@/lib/actions/reserved';+import { toolNameFor } from '@/lib/actions/tool-names';import { activeRunner, AGENTS, type AgentKind } from '@/lib/agents/catalog';import { SAVE_SITE_MAP_ACTION, SITE_MAPPER_SKILL, validateSiteMapReport } from '@/lib/skills/site-map';import {@@ -71,8 +72,12 @@ interface ActiveRun {/** Host of the tab this run started on — what "this site" means on an approval card. */site?: string;map?: MapRun;+ /** The A-Eye pick photographed when the user attached it to this instruction. */+ focusShot?: string;}+const FOCUS_SHOT_TOOL_NAME = toolNameFor(FOCUS_SHOT_ACTION);+interface Decision {allow: boolean;remember?: boolean;@@ -126,6 +131,19 @@ export class AgentSession {return result;}+ if (action === FOCUS_SHOT_ACTION) {+ const result = run.focusShot+ ? success({ dataUrl: run.focusShot })+ : failure('NO_FOCUS_SHOT', 'No A-Eye pick came with this instruction, so nothing was photographed.');+ emit({+ kind: 'toolResult',+ toolId,+ ok: result.ok,+ summary: result.ok ? 'the picked element, as photographed' : 'no pick attached',+ });+ return result;+ }+if (run.map) {const gate = gateMappingInvoke(run.map, action, input);if (!gate.allow) {@@ -249,6 +267,7 @@ export class AgentSession {abort: new AbortController(),pending: new Map(),ownedTabIds: [],+ focusShot: context?.focus?.shot,};this.runs.set(runId, run);@@ -484,6 +503,11 @@ function focusBlock(focus: FocusedElement | undefined): string | undefined {];if (focus.label) lines.push(`- Label: ${flatten(focus.label)}`);lines.push(`- On: ${flatten(focus.title)} — ${flatten(focus.url)}`);+ if (focus.shot) {+ lines.push(+ `- A screenshot of it, taken at the instant they picked it, is attached — call \`${FOCUS_SHOT_TOOL_NAME}\` once to see it.`,+ );+ }const content = focus.content.slice(0, MAX_FOCUS_CONTENT);const cut = focus.truncated || content.length < focus.content.length;diff --git a/src/daemon/extension-link.ts b/src/daemon/extension-link.tsindex 87dc186..83d3cd7 100644--- a/src/daemon/extension-link.ts+++ b/src/daemon/extension-link.ts@@ -10,6 +10,7 @@ import {} from '@/lib/actions/protocol';import type { ToolDescriptor } from '@/lib/actions/manifest';import { awaitMonitor } from '@/lib/actions/page/await-monitor';+import { PICK_DEFAULT_TIMEOUT_MS, pickElement } from '@/lib/actions/page/pick-element';import { startMonitor } from '@/lib/actions/page/start-monitor';import { typeText, typingDurationMs } from '@/lib/actions/page/type-text';import { AWAIT_DEFAULT_TIMEOUT_MS } from '@/lib/monitor/events';@@ -130,6 +131,7 @@ function timeoutFor(action: string, input: unknown): number {if (action === typeText.name) return typingDurationMs(input) + DEFAULT_TIMEOUT_MS;if (action === startMonitor.name) return DEFAULT_TIMEOUT_MS;if (action === awaitMonitor.name) return (declaredTimeout(input) ?? AWAIT_DEFAULT_TIMEOUT_MS) + 5_000;+ if (action === pickElement.name) return (declaredTimeout(input) ?? PICK_DEFAULT_TIMEOUT_MS) + 5_000;const declared = declaredTimeout(input);return declared != null ? declared + 5_000 : DEFAULT_TIMEOUT_MS;}diff --git a/src/daemon/remote-bridge.ts b/src/daemon/remote-bridge.tsindex 24d3e95..959704e 100644--- a/src/daemon/remote-bridge.ts+++ b/src/daemon/remote-bridge.ts@@ -1,6 +1,7 @@import { randomUUID } from 'node:crypto';import { WebSocket } from 'ws';import { awaitMonitor } from '@/lib/actions/page/await-monitor';+import { PICK_DEFAULT_TIMEOUT_MS, pickElement } from '@/lib/actions/page/pick-element';import { startMonitor } from '@/lib/actions/page/start-monitor';import { failure, type ActionResult } from '@/lib/actions/protocol';import type { ToolDescriptor } from '@/lib/actions/manifest';@@ -119,5 +120,6 @@ function invokeTimeoutFor(action: string, input?: unknown): number {const declared = (input as { timeoutMs?: unknown } | undefined)?.timeoutMs;if (typeof declared === 'number' && declared > 0) return declared + 10_000;if (action === awaitMonitor.name) return AWAIT_DEFAULT_TIMEOUT_MS + 10_000;+ if (action === pickElement.name) return PICK_DEFAULT_TIMEOUT_MS + 10_000;return REQUEST_TIMEOUT_MS;}diff --git a/src/daemon/server.ts b/src/daemon/server.tsindex da59db9..8b6c4cd 100644--- a/src/daemon/server.ts+++ b/src/daemon/server.ts@@ -6,7 +6,7 @@ import {ReadResourceRequestSchema,} from '@modelcontextprotocol/sdk/types.js';import type { ActionResult } from '@/lib/actions/protocol';-import { RESERVED_ACTIONS, RESERVED_PREFIX, SAVE_SITE_MAP_ACTION } from '@/lib/actions/reserved';+import { FOCUS_SHOT_ACTION, RESERVED_ACTIONS, RESERVED_PREFIX, SAVE_SITE_MAP_ACTION } from '@/lib/actions/reserved';import { actionNameFor, assertToolNamesRoundTrip, toolNameFor } from '@/lib/actions/tool-names';import { readAgentConfig } from './agent/config';import type { Bridge } from './control';@@ -15,6 +15,15 @@ import { log } from './log';const STATUS_TOOL = toolNameFor(`${RESERVED_PREFIX}status`);const SCREENSHOT_TOOL = 'page_screenshot';+const PICK_TOOL = 'page_pickElement';++const FOCUS_SHOT_TOOL = {+ name: toolNameFor(FOCUS_SHOT_ACTION),+ description:+ 'Show the screenshot of the element the user pointed at with A-Eye, taken at the instant they picked it. ' ++ 'Only answers when the current instruction arrived with a pick attached.',+ inputSchema: { type: 'object' as const, properties: {}, additionalProperties: false },+};const RESOURCES = [{@@ -148,7 +157,7 @@ export function createMcpServer(bridge: Bridge, version: string, opts: { agentRu'Report whether the Browsentic browser extension is connected, its version, and the active tab. Use this first if a page tool fails.',inputSchema: { type: 'object' as const, properties: {}, additionalProperties: false },},- ...(opts.agentRun ? [SAVE_SITE_MAP_TOOL] : []),+ ...(opts.agentRun ? [SAVE_SITE_MAP_TOOL, FOCUS_SHOT_TOOL] : []),],};});@@ -158,6 +167,8 @@ export function createMcpServer(bridge: Bridge, version: string, opts: { agentRuconst action = actionNameFor(params.name);const result = await bridge.invoke(action, params.arguments ?? {});if (params.name === SCREENSHOT_TOOL) return renderScreenshot(result);+ if (params.name === FOCUS_SHOT_TOOL.name) return renderFocusShot(result);+ if (params.name === PICK_TOOL) return renderPick(result, shouldFence(action, policy) ? tag : undefined);return render(result, shouldFence(action, policy) ? tag : undefined);});@@ -277,6 +288,45 @@ function renderScreenshot(result: ActionResult) {};}+function renderPick(result: ActionResult, fenceWith?: string) {+ if (!result.ok) return render(result, fenceWith);+ const { shot, ...rest } = result.data as { shot?: { dataUrl?: string; width?: number; height?: number } } & Record<+ string,+ unknown+ >;+ if (typeof shot?.dataUrl !== 'string') return render(result, fenceWith);++ const [mimeType, base64] = splitDataUrl(shot.dataUrl);+ const rendered = render({ ok: true, data: rest }, fenceWith);+ return {+ content: [+ ...rendered.content,+ { type: 'image' as const, data: base64, mimeType },+ {+ type: 'text' as const,+ text: `${IMAGE_NOTE} This is the picked element photographed at the instant the user clicked it.`,+ },+ ],+ };+}++function renderFocusShot(result: ActionResult) {+ if (!result.ok) return render(result);+ const dataUrl = (result.data as { dataUrl?: unknown } | null)?.dataUrl;+ if (typeof dataUrl !== 'string') return render(result);++ const [mimeType, base64] = splitDataUrl(dataUrl);+ return {+ content: [+ { type: 'image' as const, data: base64, mimeType },+ {+ type: 'text' as const,+ text: `${IMAGE_NOTE} This is the element the user pointed at with A-Eye, as it stood when they picked it.`,+ },+ ],+ };+}+function splitDataUrl(dataUrl: string): [mimeType: string, base64: string] {const match = /^data:([^;,]+);base64,(.*)$/s.exec(dataUrl);return match ? [match[1], match[2]] : ['image/png', ''];diff --git a/src/daemon/skills/a-eye.md b/src/daemon/skills/a-eye.mdindex b0e2452..60d31eb 100644--- a/src/daemon/skills/a-eye.md+++ b/src/daemon/skills/a-eye.md@@ -8,7 +8,7 @@ A-Eye is pointing, in both directions. The user can point at an element before t## When they pointed first-If they used A-Eye before sending, the element is already in your system prompt under **Focused element (A-Eye)** — its selector, its tag and role, and the text it held at that moment. There is nothing to call: it is the subject of the instruction.+If they used A-Eye before sending, the element is already in your system prompt under **Focused element (A-Eye)** — its selector, its tag and role, and the text it held at that moment. There is nothing to call: it is the subject of the instruction. When that block says a screenshot is attached, `browsentic_focusShot {}` shows you the element photographed at the instant they picked it — call it once when seeing the element matters, and never re-derive it with a fresh capture.Three habits make that worth something:@@ -20,7 +20,7 @@ If the selector no longer resolves, the page has changed under the pick. Say so## When you need them to point-`page_pickElement {}` hands the page to the user: their cursor becomes a lens, whatever they hover is outlined, and the element they click comes back with its selector, its role and its rendered text.+`page_pickElement {}` hands the page to the user: their cursor becomes a lens, whatever they hover is outlined, and the element they click comes back with its selector, its role, its rendered text and a screenshot of it exactly as they saw it.```page_pickElement { hint: "Point at the price you mean" }@@ -28,7 +28,7 @@ page_pickElement { hint: "Point at the price you mean" }Reach for it when a target is genuinely ambiguous and describing it would take longer than pointing — several rows share a label, "the second one" could mean two things, the user said "this one" about something you cannot see. Set `hint` to the question you would have asked in words.-It stops everything and waits for a person, so it costs more than any other tool here. Do not call it to explore a page, do not call it when `page_getPageInfo` would have told you the same thing, and never call it twice in a row — if the first pick did not tell you what you needed, ask in words.+It stops everything and waits for a person, so it costs more than any other tool here. Do not call it to explore a page, do not call it when `page_getPageInfo` would have told you the same thing, and never call it twice in a row — a new call dismisses a pick still waiting, and if the first pick did not tell you what you needed, ask in words.Two refusals, both terminal: `PICK_CANCELLED` means they dismissed it without choosing, and `TIMEOUT` means they never got to it. Both mean the same thing — stop asking them to point, and carry on in words or ask a plain question.diff --git a/src/extension/components/composer.tsx b/src/extension/components/composer.tsxindex fbbd953..bbdd4b1 100644--- a/src/extension/components/composer.tsx+++ b/src/extension/components/composer.tsx@@ -283,6 +283,13 @@ function FocusChip({</span>) : (<>+ {focus.shot && (+ <img+ src={focus.shot}+ alt="The element you picked"+ className="max-h-7 max-w-14 shrink-0 rounded border border-ember/40 object-cover"+ />+ )}<span className="min-w-0 flex-1 truncate text-ink">A-Eye: <span className="font-medium">{focusName(focus)}</span></span>diff --git a/src/extension/components/run-timeline.tsx b/src/extension/components/run-timeline.tsxindex c36a286..b39dcb3 100644--- a/src/extension/components/run-timeline.tsx+++ b/src/extension/components/run-timeline.tsx@@ -240,6 +240,7 @@ const ICONS: Record<string, LucideIcon> = {getPageInfo: Eye,extractText: Eye,pickElement: ScanEye,+ 'browsentic.focusShot': ScanEye,findProgress: Eye,clickElement: MousePointerClick,trustedClick: MousePointerClick,diff --git a/src/lib/actions/page/lens.ts b/src/lib/actions/page/lens.tsindex 998a2f9..a775325 100644--- a/src/lib/actions/page/lens.ts+++ b/src/lib/actions/page/lens.ts@@ -18,11 +18,7 @@ const CURSOR = `url("data:image/svg+xml,${encodeURIComponent(cursorSvg())}") 14const DEFAULT_HINT = 'Click the element you mean';-let picking = false;--export function lensIsUp(): boolean {- return picking;-}+let dismissCurrent: (() => void) | null = null;/*** The page is handed to the user for one click, and the page must not be able to tell the@@ -30,9 +26,12 @@ export function lensIsUp(): boolean {* propagation stopped, default left alone — because preventing `pointerdown` would stop* the browser generating the `click` this waits on. `click` itself is the one that is* fully cancelled, so picking a link never also follows it.+ *+ * Only one lens can hold the page: a new pick dismisses one already waiting, whose caller+ * gets a cancellation. A lens whose caller stopped listening must never wedge the next pick.*/export function pickWithLens({ hint, timeoutMs }: { hint?: string; timeoutMs: number }): Promise<LensOutcome> {- picking = true;+ dismissCurrent?.();const host = document.createElement('div');host.id = HOST_ID;@@ -54,6 +53,8 @@ export function pickWithLens({ hint, timeoutMs }: { hint?: string; timeoutMs: nulet hovered: Element | null = null;return new Promise<LensOutcome>((resolve) => {+ const dismiss = () => settle({ cancelled: true });+ dismissCurrent = dismiss;const timer = setTimeout(() => settle({ timedOut: true }), timeoutMs);const mute = (event: Event) => {@@ -128,7 +129,7 @@ export function pickWithLens({ hint, timeoutMs }: { hint?: string; timeoutMs: nu}host.remove();cursor.remove();- picking = false;+ if (dismissCurrent === dismiss) dismissCurrent = null;resolve(outcome);}});diff --git a/src/lib/actions/page/pick-element.ts b/src/lib/actions/page/pick-element.tsindex ccd1c84..79d3326 100644--- a/src/lib/actions/page/pick-element.ts+++ b/src/lib/actions/page/pick-element.ts@@ -1,18 +1,21 @@import { z } from 'zod';import { ActionError, defineAction } from '../core';import { accessibleText, describeElement } from './dom';-import { lensIsUp, pickWithLens } from './lens';+import { pickWithLens } from './lens';const MAX_CONTENT = 20_000;+export const PICK_DEFAULT_TIMEOUT_MS = 60_000;+export const pickElement = defineAction({name: 'page.pickElement',description:'Ask the user to point at an element — A-Eye. Their cursor becomes a lens, whatever they hover is ' +- 'outlined, and the element they click comes back with its selector, its role and its rendered text. ' +- 'Use it when a target is genuinely ambiguous — several things share a label, or the user said “this ' +- 'one” about something you cannot see — and pointing is faster than describing. It takes over the ' +- 'page and waits for a person, so never call it to explore, and never call it twice in a row.',+ 'outlined, and the element they click comes back with its selector, its role, its rendered text and a ' ++ 'screenshot of it exactly as they saw it. Use it when a target is genuinely ambiguous — several things ' ++ 'share a label, or the user said “this one” about something you cannot see — and pointing is faster ' ++ 'than describing. It takes over the page and waits for a person, so never call it to explore; a new ' ++ 'call dismisses a pick already waiting.',input: z.object({hint: z.string()@@ -31,13 +34,10 @@ export const pickElement = defineAction({.int().min(5_000).max(300_000)- .default(60_000)+ .default(PICK_DEFAULT_TIMEOUT_MS).describe('How long to wait for the user to click before giving up'),}),async execute({ hint, maxContentLength, timeoutMs }) {- if (lensIsUp()) {- throw new ActionError('A-Eye is already waiting for the user to point at something', 'ACTION_FAILED');- }const outcome = await pickWithLens({ hint, timeoutMs });if ('timedOut' in outcome) {throw new ActionError(@@ -52,12 +52,18 @@ export const pickElement = defineAction({const element = outcome.picked;const rendered = element instanceof HTMLElement ? element.innerText : (element.textContent ?? '');const content = rendered.replace(/\n{3,}/g, '\n\n').trim() || accessibleText(element);+ const rect = element.getBoundingClientRect();return {element: describeElement(element),content: content.slice(0, maxContentLength),truncated: content.length > maxContentLength,url: location.href,title: document.title,+ capture: {+ region: { x: rect.x, y: rect.y, w: rect.width, h: rect.height },+ viewport: { w: window.innerWidth, h: window.innerHeight },+ dpr: window.devicePixelRatio || 1,+ },};},});diff --git a/src/lib/actions/protocol.ts b/src/lib/actions/protocol.tsindex 99ddfa4..ea276a5 100644--- a/src/lib/actions/protocol.ts+++ b/src/lib/actions/protocol.ts@@ -91,6 +91,8 @@ export interface FocusedElement {truncated: boolean;url: string;title: string;+ /** Data URL of the element photographed at the instant it was picked. */+ shot?: string;}export interface SavedRecording {diff --git a/src/lib/actions/reserved.ts b/src/lib/actions/reserved.tsindex ef1ec49..9403902 100644--- a/src/lib/actions/reserved.ts+++ b/src/lib/actions/reserved.ts@@ -4,10 +4,12 @@ export const SAVE_SITE_MAP_ACTION = `${RESERVED_PREFIX}saveSiteMap` as const;export const START_RECORDING_ACTION = `${RESERVED_PREFIX}startRecording` as const;export const STOP_RECORDING_ACTION = `${RESERVED_PREFIX}stopRecording` as const;export const READ_SITEMAP_ACTION = `${RESERVED_PREFIX}readSitemap` as const;+export const FOCUS_SHOT_ACTION = `${RESERVED_PREFIX}focusShot` as const;export const RESERVED_ACTIONS = [SAVE_SITE_MAP_ACTION,START_RECORDING_ACTION,STOP_RECORDING_ACTION,READ_SITEMAP_ACTION,+ FOCUS_SHOT_ACTION,] as const;diff --git a/src/lib/bridge/aeye.ts b/src/lib/bridge/aeye.tsindex 7e294aa..83f8137 100644--- a/src/lib/bridge/aeye.ts+++ b/src/lib/bridge/aeye.ts@@ -1,16 +1,28 @@-import { invokeInActiveTab } from '@/lib/actions/client';-import { pickElement } from '@/lib/actions/page/pick-element';+import { browser } from 'wxt/browser';import type { FocusedElement } from '@/lib/actions/protocol';+import { pickInTab, type PickShot } from './pick';export type PickOutcome = { focus: FocusedElement } | { cancelled: true } | { error: string };+interface PickedElement {+ element: { tag: string; role?: string; selector: string; text?: string };+ content: string;+ truncated: boolean;+ url: string;+ title: string;+ shot?: PickShot;+}+/** The panel drives A-Eye through the same action the agent calls, so both see one picker. */export async function pickFocus(): Promise<PickOutcome> {- const result = await invokeInActiveTab(pickElement);+ const [tab] = await browser.tabs.query({ active: true, currentWindow: true });+ if (tab?.id == null) return { error: 'No active tab to point at' };++ const result = await pickInTab({ id: tab.id, windowId: tab.windowId });if (!result.ok) {return result.error.code === 'PICK_CANCELLED' ? { cancelled: true } : { error: result.error.message };}- const { element, content, truncated, url, title } = result.data;+ const { element, content, truncated, url, title, shot } = result.data as PickedElement;return {focus: {tag: element.tag,@@ -21,6 +33,7 @@ export async function pickFocus(): Promise<PickOutcome> {truncated,url,title,+ shot: shot?.dataUrl,},};}diff --git a/src/lib/bridge/invoke.ts b/src/lib/bridge/invoke.tsindex 20c482c..dc9b9e8 100644--- a/src/lib/bridge/invoke.ts+++ b/src/lib/bridge/invoke.ts@@ -13,6 +13,7 @@ import { listRecordings } from '@/lib/actions/page/list-recordings';import { monitorStatus } from '@/lib/actions/page/monitor-status';import { navigate, resolveNavigation, type NavigateInput } from '@/lib/actions/page/navigate';import { openTab } from '@/lib/actions/page/open-tab';+import { pickElement } from '@/lib/actions/page/pick-element';import { readConsole } from '@/lib/actions/page/read-console';import { readNetwork } from '@/lib/actions/page/read-network';import { readRecording } from '@/lib/actions/page/read-recording';@@ -43,6 +44,7 @@ import {import { awaitMonitorDone, monitorStatusFor, startTabMonitor, stopTabMonitor } from '@/lib/bridge/monitor';import { startJobTimer, stopJobTimer, timerStatusFor } from '@/lib/bridge/timer';import { listRecordings as listStoredMeta, readRecordingBody } from '@/lib/bridge/recording-store';+import { pickInTab } from '@/lib/bridge/pick';import { screenshotTab } from '@/lib/bridge/screenshot';import { dragInTab, trustedClickInTab } from '@/lib/bridge/trusted-input';import { closeOpenTab, openNewTab, switchToTab, watchForLoad, type TabRef } from '@/lib/bridge/tabs';@@ -123,6 +125,7 @@ async function dispatch(if (action === switchTab.name) return switchSessionTab({ id: tab.id, windowId: tab.windowId }, input, owner);if (action === closeTab.name) return closeOpenTab({ id: tab.id, windowId: tab.windowId }, input);if (action === screenshot.name) return screenshotTab({ id: tab.id, windowId: tab.windowId }, input, runId);+ if (action === pickElement.name) return pickInTab({ id: tab.id, windowId: tab.windowId }, input);if (action === attachFile.name) return attachStoredFile(tab.id, input);if (action === captureDownload.name) return captureFromPage(tab.id, input);if (action === trustedClick.name) return trustedClickInTab(tab.id, input);diff --git a/src/lib/bridge/pick.ts b/src/lib/bridge/pick.tsnew file mode 100644index 0000000..7ed3260--- /dev/null+++ b/src/lib/bridge/pick.ts@@ -0,0 +1,67 @@+import { invokeInTab } from '@/lib/actions/client';+import { pickElement } from '@/lib/actions/page/pick-element';+import { success, type ActionResult } from '@/lib/actions/protocol';+import { blobToDataUrl, captureViewport } from './screenshot';++export interface PickShot {+ dataUrl: string;+ width: number;+ height: number;+}++interface PickCapture {+ region: { x: number; y: number; w: number; h: number };+ viewport: { w: number; h: number };+ dpr: number;+}++const REGION_PADDING_CSS_PX = 16;+const MAX_SHOT_LONG_SIDE = 1200;+const SHOT_JPEG_QUALITY = 0.85;+const PAINT_SETTLE_MS = 120;++/**+ * Runs the A-Eye pick in the tab, then photographs the picked element before anything+ * can move it — the pre-screenshot that travels with the pick instead of a later re-read.+ * The capture is best-effort: a pick that cannot be photographed is still a pick.+ */+export async function pickInTab(+ tab: { id: number; windowId?: number },+ input?: unknown,+): Promise<ActionResult> {+ const picked = await invokeInTab(tab.id, pickElement.name, input);+ if (!picked.ok) return picked;++ const { capture, ...data } = picked.data as { capture?: PickCapture } & Record<string, unknown>;+ const shot = capture ? await shootRegion(tab.windowId, capture).catch(() => null) : null;+ return success(shot ? { ...data, shot } : data);+}++async function shootRegion(windowId: number | undefined, capture: PickCapture): Promise<PickShot | null> {+ const { viewport, dpr } = capture;+ const x = Math.max(0, capture.region.x - REGION_PADDING_CSS_PX);+ const y = Math.max(0, capture.region.y - REGION_PADDING_CSS_PX);+ const w = Math.min(viewport.w, capture.region.x + capture.region.w + REGION_PADDING_CSS_PX) - x;+ const h = Math.min(viewport.h, capture.region.y + capture.region.h + REGION_PADDING_CSS_PX) - y;+ if (w < 1 || h < 1) return null;++ await new Promise((resolve) => setTimeout(resolve, PAINT_SETTLE_MS));+ const { bitmap } = await captureViewport(windowId, 'jpeg', Math.round(SHOT_JPEG_QUALITY * 100));++ const scale = bitmap.width / viewport.w || dpr;+ const outScale = Math.min(1, MAX_SHOT_LONG_SIDE / Math.max(w * scale, h * scale, 1));+ const out = new OffscreenCanvas(+ Math.max(1, Math.round(w * scale * outScale)),+ Math.max(1, Math.round(h * scale * outScale)),+ );+ const ctx = out.getContext('2d');+ if (!ctx) {+ bitmap.close();+ return null;+ }+ ctx.drawImage(bitmap, x * scale, y * scale, w * scale, h * scale, 0, 0, out.width, out.height);+ bitmap.close();++ const blob = await out.convertToBlob({ type: 'image/jpeg', quality: SHOT_JPEG_QUALITY });+ return { dataUrl: await blobToDataUrl(blob, 'image/jpeg'), width: out.width, height: out.height };+}diff --git a/src/lib/bridge/screenshot.ts b/src/lib/bridge/screenshot.tsindex 95764a0..0c0fd87 100644--- a/src/lib/bridge/screenshot.ts+++ b/src/lib/bridge/screenshot.ts@@ -182,7 +182,7 @@ function fitScale(deviceW: number, deviceH: number, maxLongSide: number): numberreturn Math.min(1, byLong, byArea);}-async function captureViewport(+export async function captureViewport(windowId: number | undefined,format: 'png' | 'jpeg',quality: number | undefined,@@ -217,7 +217,7 @@ function tileStarts(start: number, size: number, step: number): number[] {const delay = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));-function blobToDataUrl(blob: Blob, mime: string): Promise<string> {+export function blobToDataUrl(blob: Blob, mime: string): Promise<string> {return blob.arrayBuffer().then((buffer) => {const bytes = new Uint8Array(buffer);let binary = '';