An agentic browser harness: an AI side panel & orb in your own logged-in Chrome, Edge, Arc, Brave or even Firefox run by the agent/subscription you already use (Claude Code, Codex, Antigravity). Free, open source, no API key. Also an MCP server.
TypeScript agentic-browseragentic-browsingagy-cliaiai-agentai-browser-extensionai-toolsbrave-extensionbrowser-agentbrowser-automationbrowser-extensionbrowser-usebrowserskillchrome-extensionclaude-codecodexfirefox-addonmcp-servernanobrowserweb-automation
fix(android): A-Eye on the phone tab is steered from the picture: hovering outlines the element on the phone, a click picks it with no input reaching the page (disabled buttons, links, fields and touch-handled elements included), a drag still scrolls, ↑ widens and Esc cancels; a tap on the phone itself can no longer be cancelled by the page's touch handlers
Shahriarce287d7cc046
diff --git a/docs/guide/features/android.md b/docs/guide/features/android.mdindex 1d08bc5..7c1e614 100644--- a/docs/guide/features/android.md+++ b/docs/guide/features/android.md@@ -82,10 +82,13 @@ phone's tabs, and timers. The differences:[Blocked sites](blocked-sites.md) are checked against the phone's own address, and[approvals](../approvals.md) work as they do on a desktop tab.-**[A-Eye](a-eye.md)** works on the phone tab too. Press its button in the side panel, and when the-lens shows on the phone, click the element on the phone tab's picture: that tap picks it rather-than pressing it, and the pick carries a photograph of the element from the phone. When the agent-asks you to point at something, it works the same way.+**[A-Eye](a-eye.md)** works on the phone tab too. Press its button in the side panel and the phone+tab's picture becomes the lens: the element under your pointer is outlined on the phone, and a click+picks it without pressing it, so a link is not followed and a field does not open the phone's+keyboard. Disabled buttons can be picked as well. Dragging still scrolls the page, `↑` widens the+outline to the element around it, and `Esc` cancels. A tap on the phone itself picks too. The pick+carries a photograph of the element from the phone. When the agent asks you to point at something,+it works the same way.**Live tools** work there as well. With the Live tool switch on, the agent can write code for thephone's page; you approve it before it runs, as on a desktop tab. You can keep the result as adiff --git a/docs/internals/android.md b/docs/internals/android.mdindex 64973e8..8e776d2 100644--- a/docs/internals/android.md+++ b/docs/internals/android.md@@ -192,9 +192,16 @@ only.## A-Eye on the phone-`page.pickElement` runs its own lens in the bundle's world on the phone. A click on the phone tab's-picture arrives as a tap, and the lens takes the click that follows it and swallows it, so the page-never sees it. The wait is capped at 170 s, under the relay's 180 s per command, so the page's own+`page.pickElement` runs its own lens in the bundle's world on the phone. While it waits, `pickOn`+hands the mirror the lens (`steerLensFromMirror`), and the mirror tab's pointer steers it instead of+touching the page: a hover sends `aim`, a press that stays within 6 px sends `pick`, and `↑` and `Esc`+send `wider` and `cancel`. A press that travels further becomes a touch drag, so the page still+scrolls. The background turns each point into the lens frame's client coordinates (`framePoint`,+`screenPoint` backwards) and calls `__browsenticPhone.lens` in the lens's world, and collapses aims+while one is on its way. The page gets no input at all, so disabled controls, links, fields and+elements whose touch handlers cancel a tap can all be picked. A tap on the phone itself still picks+through the lens's `click` listener; the lens mutes touch events along with pointer and mouse ones,+so the page's own handlers cannot cancel that click. The wait is capped at 170 s, under the relay's 180 s per command, so the page's owntimeout answers first. `pickOn` then photographs the element: `pickClip` pads its box, keeps it onscreen, moves it into document coordinates (plus any frame's offset), and asks `Page.captureScreenshot`for it at device resolution, up to 1,200 px on the long side. The panel's A-Eye button reaches thediff --git a/src/extension/entrypoints/phone/App.tsx b/src/extension/entrypoints/phone/App.tsxindex c07bbcb..73745bc 100644--- a/src/extension/entrypoints/phone/App.tsx+++ b/src/extension/entrypoints/phone/App.tsx@@ -160,7 +160,7 @@ function PhoneFrame({ref={mirror.canvas}tabIndex={0}aria-label="The phone's screen"- className={cn('block h-full w-full touch-none outline-none', (!mirror.image || overlay) && 'opacity-30')}+ className={cn('block h-full w-full touch-none outline-none', mirror.picking && 'cursor-crosshair', (!mirror.image || overlay) && 'opacity-30')}/>{overlay && <div className="absolute inset-0 flex items-center justify-center p-6">{overlay}</div>}{tabsOpen && session && <TabList session={session} mirror={mirror} onClose={() => setTabsOpen(false)} />}diff --git a/src/extension/entrypoints/phone/use-mirror.ts b/src/extension/entrypoints/phone/use-mirror.tsindex 294c0bd..20d797c 100644--- a/src/extension/entrypoints/phone/use-mirror.ts+++ b/src/extension/entrypoints/phone/use-mirror.ts@@ -1,29 +1,36 @@import { useCallback, useEffect, useRef, useState } from 'react';import { browser, type Browser } from 'wxt/browser';-import { PHONE_PORT, isMirrorKey, touchPoint, type FrameMetadata, type MirrorCommand, type MirrorMessage, type PageState, type TouchType } from '@/lib/phone/mirror';+import type { LensCommand } from '@/lib/actions/page/lens';+import { PHONE_PORT, isMirrorKey, touchPoint, type FrameMetadata, type MirrorCommand, type MirrorMessage, type PageState } from '@/lib/phone/mirror';const RECONNECT_MS = 500;const ERROR_SHOWN_MS = 5_000;+/** How far a press travels during A-Eye before it is a drag that scrolls, not a pick. */+const PICK_SLOP_PX = 6;export interface Mirror {canvas: React.RefObject<HTMLCanvasElement | null>;page: PageState | null;image: { width: number; height: number } | null;error: string | undefined;+ /** A-Eye is waiting on the phone: hovering the picture aims its lens and a click picks. */+ picking: boolean;send(command: MirrorCommand): void;}const decode = (data: string) => new Blob([Uint8Array.from(atob(data), (char) => char.charCodeAt(0))], { type: 'image/jpeg' });-/** The page's half of the mirror: frames in and drawn, then acked; touches, wheel and keys out as the phone takes them. */+/** The page's half of the mirror: frames in and drawn, then acked; touches, wheel and keys out as the phone takes them, or to A-Eye's lens while it waits. */export function useMirror(watching: boolean): Mirror {const canvas = useRef<HTMLCanvasElement | null>(null);const port = useRef<Browser.runtime.Port | null>(null);const metadata = useRef<FrameMetadata | null>(null);const watchingNow = useRef(watching);+ const pickingNow = useRef(false);const [page, setPage] = useState<PageState | null>(null);const [image, setImage] = useState<{ width: number; height: number } | null>(null);const [error, setError] = useState<string>();+ const [picking, setPicking] = useState(false);const send = useCallback((command: MirrorCommand) => port.current?.postMessage(command), []);@@ -45,6 +52,10 @@ export function useMirror(watching: boolean): Mirror {metadata.current = message.metadata;from.postMessage({ op: 'ack', frame: message.frame } satisfies MirrorCommand);};+ const lensUp = (active: boolean) => {+ pickingNow.current = active;+ setPicking(active);+ };const connect = () => {const opened = browser.runtime.connect({ name: PHONE_PORT });port.current = opened;@@ -52,10 +63,12 @@ export function useMirror(watching: boolean): Mirror {const message = raw as MirrorMessage;if (message.kind === 'frame') void draw(message, opened);else if (message.kind === 'page') setPage(message);+ else if (message.kind === 'lens') lensUp(message.active);else setError(message.message);});opened.onDisconnect.addListener(() => {if (port.current === opened) port.current = null;+ lensUp(false);if (live) retry = setTimeout(connect, RECONNECT_MS);});if (watchingNow.current) opened.postMessage({ op: 'watch' } satisfies MirrorCommand);@@ -82,21 +95,49 @@ export function useMirror(watching: boolean): Mirror {useEffect(() => {const target = canvas.current;if (!target) return;+ let lensPress: { clientX: number; clientY: number; point: { x: number; y: number }; dragging: boolean } | null = null;const at = (event: { clientX: number; clientY: number }) => {const frame = metadata.current;if (!frame || !target.width) return null;const box = target.getBoundingClientRect();return touchPoint({ x: (event.clientX - box.left) / box.width, y: (event.clientY - box.top) / box.height }, frame, target);};- const touch = (type: TouchType) => (event: PointerEvent) => {- if (type !== 'touchStart' && !target.hasPointerCapture(event.pointerId)) return;- if (type === 'touchStart') {- target.setPointerCapture(event.pointerId);- target.focus();+ const lens = (command: LensCommand) => send({ op: 'lens', lens: command });+ const press = (event: PointerEvent) => {+ target.setPointerCapture(event.pointerId);+ target.focus();+ event.preventDefault();+ const point = at(event);+ if (!point) return;+ if (pickingNow.current) lensPress = { clientX: event.clientX, clientY: event.clientY, point, dragging: false };+ else send({ op: 'touch', type: 'touchStart', ...point });+ };+ const move = (event: PointerEvent) => {+ if (!target.hasPointerCapture(event.pointerId)) {+ const point = pickingNow.current ? at(event) : null;+ if (point) lens({ op: 'aim', ...point });+ return;}+ event.preventDefault();const point = at(event);- if (point) send({ op: 'touch', type, ...point });+ if (!point) return;+ if (lensPress && !lensPress.dragging) {+ if (Math.hypot(event.clientX - lensPress.clientX, event.clientY - lensPress.clientY) < PICK_SLOP_PX) return;+ lensPress.dragging = true;+ send({ op: 'touch', type: 'touchStart', ...lensPress.point });+ }+ send({ op: 'touch', type: 'touchMove', ...point });+ };+ const lift = (type: 'touchEnd' | 'touchCancel') => (event: PointerEvent) => {+ if (!target.hasPointerCapture(event.pointerId)) return;event.preventDefault();+ const point = at(event);+ const pressed = lensPress;+ lensPress = null;+ if (!point) return;+ if (pressed && !pressed.dragging) {+ if (type === 'touchEnd') lens({ op: 'pick', ...point });+ } else send({ op: 'touch', type, ...point });};const wheel = (event: WheelEvent) => {const point = at(event);@@ -105,7 +146,9 @@ export function useMirror(watching: boolean): Mirror {};const key = (event: KeyboardEvent) => {if (event.metaKey || event.ctrlKey || event.altKey) return;- if (isMirrorKey(event.key)) send({ op: 'key', key: event.key });+ if (pickingNow.current && event.key === 'Escape') lens({ op: 'cancel' });+ else if (pickingNow.current && event.key === 'ArrowUp') lens({ op: 'wider' });+ else if (isMirrorKey(event.key)) send({ op: 'key', key: event.key });else if (event.key.length === 1) send({ op: 'text', text: event.key });else return;event.preventDefault();@@ -116,10 +159,10 @@ export function useMirror(watching: boolean): Mirror {event.preventDefault();};const listeners = [- ['pointerdown', touch('touchStart')],- ['pointermove', touch('touchMove')],- ['pointerup', touch('touchEnd')],- ['pointercancel', touch('touchCancel')],+ ['pointerdown', press],+ ['pointermove', move],+ ['pointerup', lift('touchEnd')],+ ['pointercancel', lift('touchCancel')],['wheel', wheel],['keydown', key],['paste', paste],@@ -130,5 +173,5 @@ export function useMirror(watching: boolean): Mirror {};}, [send]);- return { canvas, page, image, error, send };+ return { canvas, page, image, error, picking, send };}diff --git a/src/lib/actions/page/lens.test.ts b/src/lib/actions/page/lens.test.tsindex 6e12977..adcc147 100644--- a/src/lib/actions/page/lens.test.ts+++ b/src/lib/actions/page/lens.test.ts@@ -1,6 +1,6 @@import { afterEach, describe, expect, it, vi } from 'vitest';import { OVERLAY_ATTRIBUTE } from '@/lib/overlay';-import { pickWithLens, type LensOutcome } from './lens';+import { pickWithLens, steerLens, type LensOutcome } from './lens';function stage() {const button = document.createElement('button');@@ -57,4 +57,51 @@ describe('pickWithLens', () => {window.dispatchEvent(new KeyboardEvent('keydown', { key: 'Escape' }));await expect(pick).resolves.toEqual({ cancelled: true });});++ it('is steered without a single page event, so a control the page never lets click is picked all the same', async () => {+ const { button } = stage();+ button.disabled = true;+ const heard = vi.fn();+ button.addEventListener('click', heard);+ const pick = pickWithLens({ timeoutMs: 5_000 });+ vi.spyOn(document, 'elementFromPoint').mockReturnValue(button);+ expect(steerLens({ op: 'aim', x: 10, y: 10 })).toBe(true);+ expect(steerLens({ op: 'pick', x: 10, y: 10 })).toBe(true);+ await expect(pick).resolves.toEqual({ picked: button });+ expect(heard).not.toHaveBeenCalled();+ expect(steerLens({ op: 'cancel' })).toBe(false);+ });++ it('a steered pick after ↑ takes the wider element, and one aimed elsewhere takes what it landed on', async () => {+ const { button } = stage();+ const other = document.createElement('a');+ document.body.append(other);+ const wider = pickWithLens({ timeoutMs: 5_000 });+ vi.spyOn(document, 'elementFromPoint').mockReturnValue(button);+ steerLens({ op: 'aim', x: 10, y: 10 });+ steerLens({ op: 'wider' });+ steerLens({ op: 'pick', x: 10, y: 10 });+ await expect(wider).resolves.toEqual({ picked: document.body });++ const elsewhere = pickWithLens({ timeoutMs: 5_000 });+ steerLens({ op: 'aim', x: 10, y: 10 });+ vi.spyOn(document, 'elementFromPoint').mockReturnValue(other);+ steerLens({ op: 'pick', x: 90, y: 90 });+ await expect(elsewhere).resolves.toEqual({ picked: other });+ });++ it('keeps a finger’s touches from the page, whose handlers could otherwise cancel the tap', async () => {+ const { button } = stage();+ const heard = vi.fn();+ document.addEventListener('touchstart', heard, { capture: true });+ document.addEventListener('touchend', heard, { capture: true });+ const pick = pickWithLens({ timeoutMs: 5_000 });+ button.dispatchEvent(new Event('touchstart', { bubbles: true }));+ button.dispatchEvent(new Event('touchend', { bubbles: true }));+ expect(heard).not.toHaveBeenCalled();+ steerLens({ op: 'cancel' });+ await expect(pick).resolves.toEqual({ cancelled: true });+ document.removeEventListener('touchstart', heard, { capture: true });+ document.removeEventListener('touchend', heard, { capture: true });+ });});diff --git a/src/lib/actions/page/lens.ts b/src/lib/actions/page/lens.tsindex 80b5c00..27a304b 100644--- a/src/lib/actions/page/lens.ts+++ b/src/lib/actions/page/lens.ts@@ -2,6 +2,9 @@ import { isOverlay } from '@/lib/overlay';export type LensOutcome = { picked: Element } | { cancelled: true } | { timedOut: true };+/** A lens driven from outside the page, points in this document's client coordinates. */+export type LensCommand = { op: 'aim'; x: number; y: number } | { op: 'pick'; x: number; y: number } | { op: 'wider' } | { op: 'cancel' };+const HOST_ID = 'browsentic-a-eye';const CURSOR_ID = 'browsentic-a-eye-cursor';@@ -21,13 +24,14 @@ const CURSOR = `url("data:image/svg+xml,${encodeURIComponent(cursorSvg())}") 14const DEFAULT_HINT = 'Click the element you mean';let dismissCurrent: (() => void) | null = null;+let steerCurrent: ((command: LensCommand) => void) | null = null;/*** The page is handed to the user for one click, and the page must not be able to tell the- * difference between that click and nothing at all. The press sequence is only muted —- * propagation stopped, default left alone — because preventing `pointerdown` would stop- * the browser generating the `click` this waits on. `click` itself is the one that is- * fully cancelled, so picking a link never also follows it.+ * difference between that click and nothing at all. The press sequence, touch included, is+ * only muted — propagation stopped, default left alone — because preventing `pointerdown`+ * or `touchstart` would stop the browser generating the `click` this waits on. `click`+ * itself is the one that is fully cancelled, so picking a link never also follows it.** Only one lens can hold the page: a new pick dismisses one already waiting, whose caller* gets a cancellation. A lens whose caller stopped listening must never wedge the next pick.@@ -73,10 +77,7 @@ export function pickWithLens({ hint, timeoutMs }: { hint?: string; timeoutMs: nuconst onClick = (event: MouseEvent) => {swallow(event);- const under = document.elementFromPoint(event.clientX, event.clientY);- if (under && isOverlay(under)) return;- const target = hovered ?? under;- if (target && target !== document.documentElement) settle({ picked: target });+ choose(document.elementFromPoint(event.clientX, event.clientY));};const onKey = (event: KeyboardEvent) => {@@ -91,12 +92,31 @@ export function pickWithLens({ hint, timeoutMs }: { hint?: string; timeoutMs: nu}};+ const steer = (command: LensCommand) => {+ switch (command.op) {+ case 'cancel':+ return settle({ cancelled: true });+ case 'wider':+ if (hovered?.parentElement) aim(hovered.parentElement);+ return;+ case 'aim':+ return aim(document.elementFromPoint(command.x, command.y));+ case 'pick':+ return choose(document.elementFromPoint(command.x, command.y));+ }+ };+ steerCurrent = steer;+const onReflow = () => hovered && draw(hovered);const listeners: [string, EventListener, AddEventListenerOptions][] = [['pointermove', onMove as EventListener, { capture: true, passive: true }],['pointerdown', mute, { capture: true }],['pointerup', mute, { capture: true }],+ ['touchstart', mute, { capture: true, passive: true }],+ ['touchmove', mute, { capture: true, passive: true }],+ ['touchend', mute, { capture: true, passive: true }],+ ['touchcancel', mute, { capture: true, passive: true }],['mousedown', mute, { capture: true }],['mouseup', mute, { capture: true }],['dblclick', swallow, { capture: true }],@@ -109,6 +129,13 @@ export function pickWithLens({ hint, timeoutMs }: { hint?: string; timeoutMs: nu];for (const [type, listener, options] of listeners) window.addEventListener(type, listener, options);+ /** The aimed element wins only when it holds what was pressed, as after ↑; a tap never aimed there picks what it landed on. */+ function choose(under: Element | null): void {+ if (under && isOverlay(under)) return;+ const target = hovered?.contains(under) ? hovered : under;+ if (target && target !== document.documentElement) settle({ picked: target });+ }+function aim(target: Element | null): void {if (target && isOverlay(target)) {hovered = null;@@ -139,11 +166,19 @@ export function pickWithLens({ hint, timeoutMs }: { hint?: string; timeoutMs: nuhost.remove();cursor.remove();if (dismissCurrent === dismiss) dismissCurrent = null;+ if (steerCurrent === steer) steerCurrent = null;resolve(outcome);}});}+/** Steers the lens waiting in this document, if one is; the phone's mirror aims and picks this way, sending the page no input at all. */+export function steerLens(command: LensCommand): boolean {+ if (!steerCurrent) return false;+ steerCurrent(command);+ return true;+}+function label(target: Element, rect: DOMRect): string {const tag = target.tagName.toLowerCase();const id = target.id ? `#${target.id}` : '';diff --git a/src/lib/bridge/phone-backend.test.ts b/src/lib/bridge/phone-backend.test.tsindex 98b315c..166ff2c 100644--- a/src/lib/bridge/phone-backend.test.ts+++ b/src/lib/bridge/phone-backend.test.ts@@ -26,7 +26,7 @@ vi.mock('./socket', () => ({phone.goneOnce = false;return failure('CDP_ERROR', 'Cannot find context with specified id');}- return success({ result: { value: phone.answer(String(params?.expression)) } });+ return success({ result: { value: await phone.answer(String(params?.expression)) } });}if (method === 'Runtime.evaluate' && String(params?.expression).includes('scrollY')) return success({ result: { value: { scrollX: 0, scrollY: 574 } } });return success({});@@ -36,7 +36,15 @@ vi.mock('./socket', () => ({onDaemonClosed: () => undefined,}));-vi.mock('./phone-mirror', () => ({ attachedSession: async () => 'session-1' }));+const mirror = vi.hoisted(() => ({ steer: null as ((command: unknown) => Promise<void>) | null }));++vi.mock('./phone-mirror', () => ({+ attachedSession: async () => 'session-1',+ steerLensFromMirror: (_targetId: string, steer: (command: unknown) => Promise<void>) => {+ mirror.steer = steer;+ return () => (mirror.steer = null);+ },+}));const { pageSideOnPhone, pickClip, shotClip } = await import('./phone-backend');@@ -152,4 +160,19 @@ describe('A-Eye on the phone', () => {expect(String(phone.sent.find(({ params }) => String(params?.expression).includes('page.pickElement'))?.params?.expression)).toContain('"timeoutMs":170000');expect(phone.sent.find(({ method }) => method === 'Page.captureScreenshot')?.params?.clip).toEqual({ x: 4, y: 384, width: 232, height: 72, scale: 1 });});++ test('while the pick waits, the mirror steers the lens at the page point under a touch, and lets go once it is answered', async () => {+ let answer: (value: unknown) => void = () => undefined;+ phone.answer = () => new Promise((resolve) => (answer = resolve));+ phone.viewport = { offsetX: 30, offsetY: 50, clientWidth: 205, clientHeight: 337 };+ const picking = pageSideOnPhone('page.pickElement', {}, SESSION);+ await vi.waitFor(() => expect(mirror.steer).not.toBeNull());+ await mirror.steer!({ op: 'aim', x: 100, y: 200 });+ await mirror.steer!({ op: 'wider' });+ const steered = phone.sent.filter(({ params }) => String(params?.expression).startsWith('__browsenticPhone.lens')).map(({ params }) => params?.expression);+ expect(steered).toEqual(['__browsenticPhone.lens({"op":"aim","x":130,"y":250})', '__browsenticPhone.lens({"op":"wider"})']);+ answer(failure('PICK_CANCELLED', 'dismissed'));+ await picking;+ expect(mirror.steer).toBeNull();+ });});diff --git a/src/lib/bridge/phone-backend.ts b/src/lib/bridge/phone-backend.tsindex 2e328f8..fecdff2 100644--- a/src/lib/bridge/phone-backend.ts+++ b/src/lib/bridge/phone-backend.ts@@ -1,5 +1,6 @@import { browser } from 'wxt/browser';import { z } from 'zod';+import type { LensCommand } from '@/lib/actions/page/lens';import { PICK_DEFAULT_TIMEOUT_MS } from '@/lib/actions/page/pick-element';import { pressKey } from '@/lib/actions/page/press-key';import { scrollTo } from '@/lib/actions/page/scroll-to';@@ -10,7 +11,7 @@ import { PHONE_API } from '@/lib/phone/page-api';import { drag, keyPress, onScreen, tap, typing, wheel, type InputStep, type Point, type Viewport } from '@/lib/phone/touch';import type { EvaluateReply, ToolkitPlace } from './code-toolkit';import { currentTarget, type PhoneSession } from './phone';-import { attachedSession } from './phone-mirror';+import { attachedSession, steerLensFromMirror } from './phone-mirror';import { refusalFor } from './site-guard';import { sendCdp } from './socket';@@ -176,6 +177,14 @@ async function screenPoint(here: Here, point: Point): Promise<Point | null> {return viewport && onScreen(await inTopDocument(here, point), viewport);}+/** `screenPoint` backwards: where a touch would land, in the client coordinates of the frame actions run in. */+async function framePoint(here: Here, point: Point): Promise<Point | null> {+ const viewport = await visualViewport(here);+ if (!viewport) return null;+ const frame = await inTopDocument(here, { x: 0, y: 0 });+ return { x: point.x + viewport.offsetX - frame.x, y: point.y + viewport.offsetY - frame.y };+}+async function play(here: Here, steps: InputStep[]): Promise<CdpResult | null> {for (const step of steps) {const sent = await here.cdp(step.method, step.params);@@ -322,16 +331,26 @@ export function pickClip(corner: Point, capture: PickCapture, layout: LayoutView};}-/** A-Eye on the phone: the pick's own lens in the page, where a tap picks what is under it, then the element photographed before anything moves it. */+/**+ * A-Eye on the phone: the pick's own lens in the page, steered from the mirror tab while it waits+ * (a tap on the phone itself still picks), then the element photographed before anything moves it.+ */async function pickOn(here: Here, input: Record<string, unknown>): Promise<ActionResult> {const timeoutMs = Math.min(Number(input.timeoutMs) || PICK_DEFAULT_TIMEOUT_MS, MAX_PICK_WAIT_MS);- const picked = await inPage(here, 'page.pickElement', { ...input, timeoutMs });+ const { frameId } = await focusedFrame(here);+ const release = steerLensFromMirror(here.targetId, (command) => steerLensOn(here, frameId, command));+ const picked = await inPage(here, 'page.pickElement', { ...input, timeoutMs }, frameId).finally(release);if (!picked.ok) return picked;const { capture, ...data } = picked.data as { capture?: PickCapture } & Record<string, unknown>;const shot = capture ? await pickShot(here, capture).catch(() => null) : null;return success(shot ? { ...data, shot } : data);}+async function steerLensOn(here: Here, frameId: string, command: LensCommand): Promise<void> {+ const placed = 'x' in command ? await framePoint(here, command) : {};+ if (placed) await evaluateIn(here, frameId, `${PHONE_API}.lens(${JSON.stringify({ ...command, ...placed })})`);+}+async function pickShot(here: Here, capture: PickCapture): Promise<{ dataUrl: string; width: number; height: number } | null> {const metrics = await here.cdp('Page.getLayoutMetrics');if (!metrics.ok) return null;diff --git a/src/lib/bridge/phone-mirror.test.ts b/src/lib/bridge/phone-mirror.test.tsindex 2810f74..a8c2321 100644--- a/src/lib/bridge/phone-mirror.test.ts+++ b/src/lib/bridge/phone-mirror.test.ts@@ -157,4 +157,45 @@ describe('the mirror in the background', () => {await settle();expect(socket.sent.at(-1)).toEqual({ method: 'Page.navigateToHistoryEntry', params: { entryId: 1 }, sessionId: `session-${FRONT}` });});++ test('while A-Eye waits, the page is told, its pointer steers the lens with aims collapsed, and the phone is never touched', async () => {+ const { steerLensFromMirror } = await import('./phone-mirror');+ const page = connectPage();+ page.say({ op: 'watch' });+ await settle();+ socket.sent.length = 0;+ const steered: unknown[] = [];+ const release = steerLensFromMirror(FRONT, async (command) => void steered.push(command));+ expect(page.received.at(-1)).toEqual({ kind: 'lens', active: true });+ expect(connectPage().received).toContainEqual({ kind: 'lens', active: true });++ for (let step = 1; step <= 4; step++) page.say({ op: 'lens', lens: { op: 'aim', x: step, y: step } });+ page.say({ op: 'lens', lens: { op: 'pick', x: 4, y: 4 } });+ await settle();+ expect(steered).toEqual([+ { op: 'aim', x: 1, y: 1 },+ { op: 'aim', x: 4, y: 4 },+ { op: 'pick', x: 4, y: 4 },+ ]);+ expect(methods().filter((method) => method.startsWith('Input.'))).toEqual([]);++ release();+ expect(page.received.at(-1)).toEqual({ kind: 'lens', active: false });+ page.say({ op: 'lens', lens: { op: 'pick', x: 4, y: 4 } });+ await settle();+ expect(steered).toHaveLength(3);+ });++ test('a lens left on a tab the mirror moved away from is only ever cancelled', async () => {+ const { steerLensFromMirror } = await import('./phone-mirror');+ const page = connectPage();+ page.say({ op: 'watch' });+ await settle();+ const steered: unknown[] = [];+ steerLensFromMirror(BACK, async (command) => void steered.push(command));+ page.say({ op: 'lens', lens: { op: 'pick', x: 4, y: 4 } });+ page.say({ op: 'lens', lens: { op: 'cancel' } });+ await settle();+ expect(steered).toEqual([{ op: 'cancel' }]);+ });});diff --git a/src/lib/bridge/phone-mirror.ts b/src/lib/bridge/phone-mirror.tsindex 44b573e..9ab7f5f 100644--- a/src/lib/bridge/phone-mirror.ts+++ b/src/lib/bridge/phone-mirror.ts@@ -1,4 +1,5 @@import { browser, type Browser } from 'wxt/browser';+import type { LensCommand } from '@/lib/actions/page/lens';import { PHONE_PORT, keyEvents, type MirrorCommand, type MirrorMessage, type PageState } from '@/lib/phone/mirror';import type { PhoneTarget } from '@/lib/phone/types';import { numbered, onPhoneReset, readPhone, updatePhone, type PhoneSession } from './phone';@@ -7,7 +8,7 @@ import { onCdpEvent, onDaemonClosed, onPhoneClosed, sendCdp, type CdpEvent } froimport { hostOf, patchSession, sessionForTab } from './tab-sessions';type Port = Browser.runtime.Port;-type InputCommand = Extract<MirrorCommand, { op: 'touch' | 'wheel' | 'text' | 'key' }>;+type InputCommand = Extract<MirrorCommand, { op: 'touch' | 'wheel' | 'text' | 'key' | 'lens' }>;/** Discovery also reports a phantom `"1"` beside a restored tab, and every command sent to it hangs (spike Q3). */const REAL_TARGET = /^[0-9A-F]{32}$/i;@@ -30,6 +31,7 @@ let finding: Promise<void> | null = null;const inputs: InputCommand[] = [];let pumping = false;const attachments = new Map<string, Promise<string | null>>();+let lens: { targetId: string; steer(command: LensCommand): Promise<void> } | null = null;const broadcast = (message: MirrorMessage) => {for (const port of ports) port.postMessage(message);@@ -219,9 +221,24 @@ function onEvent(event: CdpEvent): void {if (event.method === 'Page.navigatedWithinDocument') void publishPage();}+/** While an A-Eye pick waits on the phone, the mirror's pointer aims and picks with its lens instead of touching the page. */+export function steerLensFromMirror(targetId: string, steer: (command: LensCommand) => Promise<void>): () => void {+ const held = { targetId, steer };+ lens = held;+ broadcast({ kind: 'lens', active: true });+ return () => {+ if (lens !== held) return;+ lens = null;+ broadcast({ kind: 'lens', active: false });+ };+}++const isAim = (command: InputCommand | undefined) => command?.op === 'lens' && command.lens.op === 'aim';+function queueInput(command: InputCommand): void {const last = inputs.at(-1);if (last?.op === 'touch' && command.op === 'touch' && last.type === 'touchMove' && command.type === 'touchMove') inputs[inputs.length - 1] = command;+ else if (isAim(last) && isAim(command)) inputs[inputs.length - 1] = command;else if (last?.op === 'wheel' && command.op === 'wheel') {inputs[inputs.length - 1] = { ...command, deltaX: last.deltaX + command.deltaX, deltaY: last.deltaY + command.deltaY };} else inputs.push(command);@@ -239,7 +256,10 @@ async function pump(): Promise<void> {}async function dispatchInput(command: InputCommand): Promise<void> {- if (command.op === 'touch') {+ if (command.op === 'lens') {+ const held = lens;+ if (held && (command.lens.op === 'cancel' || held.targetId === followed?.targetId)) await held.steer(command.lens);+ } else if (command.op === 'touch') {const lifted = command.type === 'touchEnd' || command.type === 'touchCancel';await call('Input.dispatchTouchEvent', { type: command.type, touchPoints: lifted ? [] : [{ x: command.x, y: command.y }] });} else if (command.op === 'wheel') {@@ -291,6 +311,7 @@ async function handle(port: Port, command: MirrorCommand): Promise<void> {case 'wheel':case 'text':case 'key':+ case 'lens':return queueInput(command);case 'go': {const refused = await refusalFor(command.url);@@ -340,6 +361,7 @@ export function servePhoneMirror(): void {if (port.name !== PHONE_PORT) return;ports.add(port);if (page) port.postMessage({ kind: 'page', ...page });+ if (lens) port.postMessage({ kind: 'lens', active: true });port.onMessage.addListener((message) => void handle(port, message as MirrorCommand));port.onDisconnect.addListener(() => {ports.delete(port);diff --git a/src/lib/phone/mirror.ts b/src/lib/phone/mirror.tsindex 875fc8c..9ca0f14 100644--- a/src/lib/phone/mirror.ts+++ b/src/lib/phone/mirror.ts@@ -1,3 +1,5 @@+import type { LensCommand } from '@/lib/actions/page/lens';+export const PHONE_PORT = 'browsentic/phone';/** `Page.screencastFrame`'s metadata: the phone's visual viewport when the frame was taken. */@@ -26,6 +28,8 @@ export type MirrorCommand =| { op: 'wheel'; x: number; y: number; deltaX: number; deltaY: number }| { op: 'text'; text: string }| { op: 'key'; key: MirrorKey }+ /** While A-Eye waits on the phone; points are where a touch would land. */+ | { op: 'lens'; lens: LensCommand }| { op: 'go'; url: string }| { op: 'back' }| { op: 'forward' }@@ -45,6 +49,7 @@ export interface PageState {export type MirrorMessage =| { kind: 'frame'; data: string; metadata: FrameMetadata; frame: number }| ({ kind: 'page' } & PageState)+ | { kind: 'lens'; active: boolean }| { kind: 'error'; message: string };/**diff --git a/src/lib/phone/page-api.ts b/src/lib/phone/page-api.tsindex 40ea0c6..0352aa8 100644--- a/src/lib/phone/page-api.ts+++ b/src/lib/phone/page-api.ts@@ -1,16 +1,19 @@import { dispatch } from '@/lib/actions/dispatch';import { resolveTarget, targetSchema } from '@/lib/actions/page/dom';+import { steerLens } from '@/lib/actions/page/lens';import { isFrameElement } from '@/lib/actions/page/switch-frame';export const PHONE_API = '__browsenticPhone';/*** What the phone's page bundle puts in its isolated world: the actions' own `dispatch`, the- * visual viewport a touch is measured from, and the frame element a frame switch enters.+ * visual viewport a touch is measured from, the frame element a frame switch enters, and the+ * A-Eye lens the mirror steers.*/export function installPhoneApi(): void {(globalThis as Record<string, unknown>)[PHONE_API] = {dispatch,+ lens: steerLens,viewport: () => ({left: visualViewport?.offsetLeft ?? 0,top: visualViewport?.offsetTop ?? 0,