Skip to content

An agentic browser harness: an AI side panel & orb in your own logged-in Chrome, Edge, Arc, Brave or even Firefox run by the agent/subscription you already use (Claude Code, Codex, Antigravity). Free, open source, no API key. Also an MCP server.

fix(android): A-Eye on the phone tab is steered from the picture: hovering outlines the element on the phone, a click picks it with no input reaching the page (disabled buttons, links, fields and touch-handled elements included), a drag still scrolls, ↑ widens and Esc cancels; a tap on the phone itself can no longer be cancelled by the page's touch handlers

Shahriarce287d7cc046
diff --git a/docs/guide/features/android.md b/docs/guide/features/android.md
index 1d08bc5..7c1e614 100644
--- a/docs/guide/features/android.md
+++ b/docs/guide/features/android.md
@@ -82,10 +82,13 @@ phone's tabs, and timers. The differences:
[Blocked sites](blocked-sites.md) are checked against the phone's own address, and
[approvals](../approvals.md) work as they do on a desktop tab.
-**[A-Eye](a-eye.md)** works on the phone tab too. Press its button in the side panel, and when the
-lens shows on the phone, click the element on the phone tab's picture: that tap picks it rather
-than pressing it, and the pick carries a photograph of the element from the phone. When the agent
-asks you to point at something, it works the same way.
+**[A-Eye](a-eye.md)** works on the phone tab too. Press its button in the side panel and the phone
+tab's picture becomes the lens: the element under your pointer is outlined on the phone, and a click
+picks it without pressing it, so a link is not followed and a field does not open the phone's
+keyboard. Disabled buttons can be picked as well. Dragging still scrolls the page, `↑` widens the
+outline to the element around it, and `Esc` cancels. A tap on the phone itself picks too. The pick
+carries a photograph of the element from the phone. When the agent asks you to point at something,
+it works the same way.
**Live tools** work there as well. With the Live tool switch on, the agent can write code for the
phone's page; you approve it before it runs, as on a desktop tab. You can keep the result as a
diff --git a/docs/internals/android.md b/docs/internals/android.md
index 64973e8..8e776d2 100644
--- a/docs/internals/android.md
+++ b/docs/internals/android.md
@@ -192,9 +192,16 @@ only.
## A-Eye on the phone
-`page.pickElement` runs its own lens in the bundle's world on the phone. A click on the phone tab's
-picture arrives as a tap, and the lens takes the click that follows it and swallows it, so the page
-never sees it. The wait is capped at 170 s, under the relay's 180 s per command, so the page's own
+`page.pickElement` runs its own lens in the bundle's world on the phone. While it waits, `pickOn`
+hands the mirror the lens (`steerLensFromMirror`), and the mirror tab's pointer steers it instead of
+touching the page: a hover sends `aim`, a press that stays within 6 px sends `pick`, and `↑` and `Esc`
+send `wider` and `cancel`. A press that travels further becomes a touch drag, so the page still
+scrolls. The background turns each point into the lens frame's client coordinates (`framePoint`,
+`screenPoint` backwards) and calls `__browsenticPhone.lens` in the lens's world, and collapses aims
+while one is on its way. The page gets no input at all, so disabled controls, links, fields and
+elements whose touch handlers cancel a tap can all be picked. A tap on the phone itself still picks
+through the lens's `click` listener; the lens mutes touch events along with pointer and mouse ones,
+so the page's own handlers cannot cancel that click. The wait is capped at 170 s, under the relay's 180 s per command, so the page's own
timeout answers first. `pickOn` then photographs the element: `pickClip` pads its box, keeps it on
screen, moves it into document coordinates (plus any frame's offset), and asks `Page.captureScreenshot`
for it at device resolution, up to 1,200 px on the long side. The panel's A-Eye button reaches the
diff --git a/src/extension/entrypoints/phone/App.tsx b/src/extension/entrypoints/phone/App.tsx
index c07bbcb..73745bc 100644
--- a/src/extension/entrypoints/phone/App.tsx
+++ b/src/extension/entrypoints/phone/App.tsx
@@ -160,7 +160,7 @@ function PhoneFrame({
ref={mirror.canvas}
tabIndex={0}
aria-label="The phone's screen"
- className={cn('block h-full w-full touch-none outline-none', (!mirror.image || overlay) && 'opacity-30')}
+ className={cn('block h-full w-full touch-none outline-none', mirror.picking && 'cursor-crosshair', (!mirror.image || overlay) && 'opacity-30')}
/>
{overlay && <div className="absolute inset-0 flex items-center justify-center p-6">{overlay}</div>}
{tabsOpen && session && <TabList session={session} mirror={mirror} onClose={() => setTabsOpen(false)} />}
diff --git a/src/extension/entrypoints/phone/use-mirror.ts b/src/extension/entrypoints/phone/use-mirror.ts
index 294c0bd..20d797c 100644
--- a/src/extension/entrypoints/phone/use-mirror.ts
+++ b/src/extension/entrypoints/phone/use-mirror.ts
@@ -1,29 +1,36 @@
import { useCallback, useEffect, useRef, useState } from 'react';
import { browser, type Browser } from 'wxt/browser';
-import { PHONE_PORT, isMirrorKey, touchPoint, type FrameMetadata, type MirrorCommand, type MirrorMessage, type PageState, type TouchType } from '@/lib/phone/mirror';
+import type { LensCommand } from '@/lib/actions/page/lens';
+import { PHONE_PORT, isMirrorKey, touchPoint, type FrameMetadata, type MirrorCommand, type MirrorMessage, type PageState } from '@/lib/phone/mirror';
const RECONNECT_MS = 500;
const ERROR_SHOWN_MS = 5_000;
+/** How far a press travels during A-Eye before it is a drag that scrolls, not a pick. */
+const PICK_SLOP_PX = 6;
export interface Mirror {
canvas: React.RefObject<HTMLCanvasElement | null>;
page: PageState | null;
image: { width: number; height: number } | null;
error: string | undefined;
+ /** A-Eye is waiting on the phone: hovering the picture aims its lens and a click picks. */
+ picking: boolean;
send(command: MirrorCommand): void;
}
const decode = (data: string) => new Blob([Uint8Array.from(atob(data), (char) => char.charCodeAt(0))], { type: 'image/jpeg' });
-/** The page's half of the mirror: frames in and drawn, then acked; touches, wheel and keys out as the phone takes them. */
+/** The page's half of the mirror: frames in and drawn, then acked; touches, wheel and keys out as the phone takes them, or to A-Eye's lens while it waits. */
export function useMirror(watching: boolean): Mirror {
const canvas = useRef<HTMLCanvasElement | null>(null);
const port = useRef<Browser.runtime.Port | null>(null);
const metadata = useRef<FrameMetadata | null>(null);
const watchingNow = useRef(watching);
+ const pickingNow = useRef(false);
const [page, setPage] = useState<PageState | null>(null);
const [image, setImage] = useState<{ width: number; height: number } | null>(null);
const [error, setError] = useState<string>();
+ const [picking, setPicking] = useState(false);
const send = useCallback((command: MirrorCommand) => port.current?.postMessage(command), []);
@@ -45,6 +52,10 @@ export function useMirror(watching: boolean): Mirror {
metadata.current = message.metadata;
from.postMessage({ op: 'ack', frame: message.frame } satisfies MirrorCommand);
};
+ const lensUp = (active: boolean) => {
+ pickingNow.current = active;
+ setPicking(active);
+ };
const connect = () => {
const opened = browser.runtime.connect({ name: PHONE_PORT });
port.current = opened;
@@ -52,10 +63,12 @@ export function useMirror(watching: boolean): Mirror {
const message = raw as MirrorMessage;
if (message.kind === 'frame') void draw(message, opened);
else if (message.kind === 'page') setPage(message);
+ else if (message.kind === 'lens') lensUp(message.active);
else setError(message.message);
});
opened.onDisconnect.addListener(() => {
if (port.current === opened) port.current = null;
+ lensUp(false);
if (live) retry = setTimeout(connect, RECONNECT_MS);
});
if (watchingNow.current) opened.postMessage({ op: 'watch' } satisfies MirrorCommand);
@@ -82,21 +95,49 @@ export function useMirror(watching: boolean): Mirror {
useEffect(() => {
const target = canvas.current;
if (!target) return;
+ let lensPress: { clientX: number; clientY: number; point: { x: number; y: number }; dragging: boolean } | null = null;
const at = (event: { clientX: number; clientY: number }) => {
const frame = metadata.current;
if (!frame || !target.width) return null;
const box = target.getBoundingClientRect();
return touchPoint({ x: (event.clientX - box.left) / box.width, y: (event.clientY - box.top) / box.height }, frame, target);
};
- const touch = (type: TouchType) => (event: PointerEvent) => {
- if (type !== 'touchStart' && !target.hasPointerCapture(event.pointerId)) return;
- if (type === 'touchStart') {
- target.setPointerCapture(event.pointerId);
- target.focus();
+ const lens = (command: LensCommand) => send({ op: 'lens', lens: command });
+ const press = (event: PointerEvent) => {
+ target.setPointerCapture(event.pointerId);
+ target.focus();
+ event.preventDefault();
+ const point = at(event);
+ if (!point) return;
+ if (pickingNow.current) lensPress = { clientX: event.clientX, clientY: event.clientY, point, dragging: false };
+ else send({ op: 'touch', type: 'touchStart', ...point });
+ };
+ const move = (event: PointerEvent) => {
+ if (!target.hasPointerCapture(event.pointerId)) {
+ const point = pickingNow.current ? at(event) : null;
+ if (point) lens({ op: 'aim', ...point });
+ return;
}
+ event.preventDefault();
const point = at(event);
- if (point) send({ op: 'touch', type, ...point });
+ if (!point) return;
+ if (lensPress && !lensPress.dragging) {
+ if (Math.hypot(event.clientX - lensPress.clientX, event.clientY - lensPress.clientY) < PICK_SLOP_PX) return;
+ lensPress.dragging = true;
+ send({ op: 'touch', type: 'touchStart', ...lensPress.point });
+ }
+ send({ op: 'touch', type: 'touchMove', ...point });
+ };
+ const lift = (type: 'touchEnd' | 'touchCancel') => (event: PointerEvent) => {
+ if (!target.hasPointerCapture(event.pointerId)) return;
event.preventDefault();
+ const point = at(event);
+ const pressed = lensPress;
+ lensPress = null;
+ if (!point) return;
+ if (pressed && !pressed.dragging) {
+ if (type === 'touchEnd') lens({ op: 'pick', ...point });
+ } else send({ op: 'touch', type, ...point });
};
const wheel = (event: WheelEvent) => {
const point = at(event);
@@ -105,7 +146,9 @@ export function useMirror(watching: boolean): Mirror {
};
const key = (event: KeyboardEvent) => {
if (event.metaKey || event.ctrlKey || event.altKey) return;
- if (isMirrorKey(event.key)) send({ op: 'key', key: event.key });
+ if (pickingNow.current && event.key === 'Escape') lens({ op: 'cancel' });
+ else if (pickingNow.current && event.key === 'ArrowUp') lens({ op: 'wider' });
+ else if (isMirrorKey(event.key)) send({ op: 'key', key: event.key });
else if (event.key.length === 1) send({ op: 'text', text: event.key });
else return;
event.preventDefault();
@@ -116,10 +159,10 @@ export function useMirror(watching: boolean): Mirror {
event.preventDefault();
};
const listeners = [
- ['pointerdown', touch('touchStart')],
- ['pointermove', touch('touchMove')],
- ['pointerup', touch('touchEnd')],
- ['pointercancel', touch('touchCancel')],
+ ['pointerdown', press],
+ ['pointermove', move],
+ ['pointerup', lift('touchEnd')],
+ ['pointercancel', lift('touchCancel')],
['wheel', wheel],
['keydown', key],
['paste', paste],
@@ -130,5 +173,5 @@ export function useMirror(watching: boolean): Mirror {
};
}, [send]);
- return { canvas, page, image, error, send };
+ return { canvas, page, image, error, picking, send };
}
diff --git a/src/lib/actions/page/lens.test.ts b/src/lib/actions/page/lens.test.ts
index 6e12977..adcc147 100644
--- a/src/lib/actions/page/lens.test.ts
+++ b/src/lib/actions/page/lens.test.ts
@@ -1,6 +1,6 @@
import { afterEach, describe, expect, it, vi } from 'vitest';
import { OVERLAY_ATTRIBUTE } from '@/lib/overlay';
-import { pickWithLens, type LensOutcome } from './lens';
+import { pickWithLens, steerLens, type LensOutcome } from './lens';
function stage() {
const button = document.createElement('button');
@@ -57,4 +57,51 @@ describe('pickWithLens', () => {
window.dispatchEvent(new KeyboardEvent('keydown', { key: 'Escape' }));
await expect(pick).resolves.toEqual({ cancelled: true });
});
+
+ it('is steered without a single page event, so a control the page never lets click is picked all the same', async () => {
+ const { button } = stage();
+ button.disabled = true;
+ const heard = vi.fn();
+ button.addEventListener('click', heard);
+ const pick = pickWithLens({ timeoutMs: 5_000 });
+ vi.spyOn(document, 'elementFromPoint').mockReturnValue(button);
+ expect(steerLens({ op: 'aim', x: 10, y: 10 })).toBe(true);
+ expect(steerLens({ op: 'pick', x: 10, y: 10 })).toBe(true);
+ await expect(pick).resolves.toEqual({ picked: button });
+ expect(heard).not.toHaveBeenCalled();
+ expect(steerLens({ op: 'cancel' })).toBe(false);
+ });
+
+ it('a steered pick after ↑ takes the wider element, and one aimed elsewhere takes what it landed on', async () => {
+ const { button } = stage();
+ const other = document.createElement('a');
+ document.body.append(other);
+ const wider = pickWithLens({ timeoutMs: 5_000 });
+ vi.spyOn(document, 'elementFromPoint').mockReturnValue(button);
+ steerLens({ op: 'aim', x: 10, y: 10 });
+ steerLens({ op: 'wider' });
+ steerLens({ op: 'pick', x: 10, y: 10 });
+ await expect(wider).resolves.toEqual({ picked: document.body });
+
+ const elsewhere = pickWithLens({ timeoutMs: 5_000 });
+ steerLens({ op: 'aim', x: 10, y: 10 });
+ vi.spyOn(document, 'elementFromPoint').mockReturnValue(other);
+ steerLens({ op: 'pick', x: 90, y: 90 });
+ await expect(elsewhere).resolves.toEqual({ picked: other });
+ });
+
+ it('keeps a finger’s touches from the page, whose handlers could otherwise cancel the tap', async () => {
+ const { button } = stage();
+ const heard = vi.fn();
+ document.addEventListener('touchstart', heard, { capture: true });
+ document.addEventListener('touchend', heard, { capture: true });
+ const pick = pickWithLens({ timeoutMs: 5_000 });
+ button.dispatchEvent(new Event('touchstart', { bubbles: true }));
+ button.dispatchEvent(new Event('touchend', { bubbles: true }));
+ expect(heard).not.toHaveBeenCalled();
+ steerLens({ op: 'cancel' });
+ await expect(pick).resolves.toEqual({ cancelled: true });
+ document.removeEventListener('touchstart', heard, { capture: true });
+ document.removeEventListener('touchend', heard, { capture: true });
+ });
});
diff --git a/src/lib/actions/page/lens.ts b/src/lib/actions/page/lens.ts
index 80b5c00..27a304b 100644
--- a/src/lib/actions/page/lens.ts
+++ b/src/lib/actions/page/lens.ts
@@ -2,6 +2,9 @@ import { isOverlay } from '@/lib/overlay';
export type LensOutcome = { picked: Element } | { cancelled: true } | { timedOut: true };
+/** A lens driven from outside the page, points in this document's client coordinates. */
+export type LensCommand = { op: 'aim'; x: number; y: number } | { op: 'pick'; x: number; y: number } | { op: 'wider' } | { op: 'cancel' };
+
const HOST_ID = 'browsentic-a-eye';
const CURSOR_ID = 'browsentic-a-eye-cursor';
@@ -21,13 +24,14 @@ const CURSOR = `url("data:image/svg+xml,${encodeURIComponent(cursorSvg())}") 14
const DEFAULT_HINT = 'Click the element you mean';
let dismissCurrent: (() => void) | null = null;
+let steerCurrent: ((command: LensCommand) => void) | null = null;
/**
* The page is handed to the user for one click, and the page must not be able to tell the
- * difference between that click and nothing at all. The press sequence is only muted —
- * propagation stopped, default left alone — because preventing `pointerdown` would stop
- * the browser generating the `click` this waits on. `click` itself is the one that is
- * fully cancelled, so picking a link never also follows it.
+ * difference between that click and nothing at all. The press sequence, touch included, is
+ * only muted — propagation stopped, default left alone — because preventing `pointerdown`
+ * or `touchstart` would stop the browser generating the `click` this waits on. `click`
+ * itself is the one that is fully cancelled, so picking a link never also follows it.
*
* Only one lens can hold the page: a new pick dismisses one already waiting, whose caller
* gets a cancellation. A lens whose caller stopped listening must never wedge the next pick.
@@ -73,10 +77,7 @@ export function pickWithLens({ hint, timeoutMs }: { hint?: string; timeoutMs: nu
const onClick = (event: MouseEvent) => {
swallow(event);
- const under = document.elementFromPoint(event.clientX, event.clientY);
- if (under && isOverlay(under)) return;
- const target = hovered ?? under;
- if (target && target !== document.documentElement) settle({ picked: target });
+ choose(document.elementFromPoint(event.clientX, event.clientY));
};
const onKey = (event: KeyboardEvent) => {
@@ -91,12 +92,31 @@ export function pickWithLens({ hint, timeoutMs }: { hint?: string; timeoutMs: nu
}
};
+ const steer = (command: LensCommand) => {
+ switch (command.op) {
+ case 'cancel':
+ return settle({ cancelled: true });
+ case 'wider':
+ if (hovered?.parentElement) aim(hovered.parentElement);
+ return;
+ case 'aim':
+ return aim(document.elementFromPoint(command.x, command.y));
+ case 'pick':
+ return choose(document.elementFromPoint(command.x, command.y));
+ }
+ };
+ steerCurrent = steer;
+
const onReflow = () => hovered && draw(hovered);
const listeners: [string, EventListener, AddEventListenerOptions][] = [
['pointermove', onMove as EventListener, { capture: true, passive: true }],
['pointerdown', mute, { capture: true }],
['pointerup', mute, { capture: true }],
+ ['touchstart', mute, { capture: true, passive: true }],
+ ['touchmove', mute, { capture: true, passive: true }],
+ ['touchend', mute, { capture: true, passive: true }],
+ ['touchcancel', mute, { capture: true, passive: true }],
['mousedown', mute, { capture: true }],
['mouseup', mute, { capture: true }],
['dblclick', swallow, { capture: true }],
@@ -109,6 +129,13 @@ export function pickWithLens({ hint, timeoutMs }: { hint?: string; timeoutMs: nu
];
for (const [type, listener, options] of listeners) window.addEventListener(type, listener, options);
+ /** The aimed element wins only when it holds what was pressed, as after ↑; a tap never aimed there picks what it landed on. */
+ function choose(under: Element | null): void {
+ if (under && isOverlay(under)) return;
+ const target = hovered?.contains(under) ? hovered : under;
+ if (target && target !== document.documentElement) settle({ picked: target });
+ }
+
function aim(target: Element | null): void {
if (target && isOverlay(target)) {
hovered = null;
@@ -139,11 +166,19 @@ export function pickWithLens({ hint, timeoutMs }: { hint?: string; timeoutMs: nu
host.remove();
cursor.remove();
if (dismissCurrent === dismiss) dismissCurrent = null;
+ if (steerCurrent === steer) steerCurrent = null;
resolve(outcome);
}
});
}
+/** Steers the lens waiting in this document, if one is; the phone's mirror aims and picks this way, sending the page no input at all. */
+export function steerLens(command: LensCommand): boolean {
+ if (!steerCurrent) return false;
+ steerCurrent(command);
+ return true;
+}
+
function label(target: Element, rect: DOMRect): string {
const tag = target.tagName.toLowerCase();
const id = target.id ? `#${target.id}` : '';
diff --git a/src/lib/bridge/phone-backend.test.ts b/src/lib/bridge/phone-backend.test.ts
index 98b315c..166ff2c 100644
--- a/src/lib/bridge/phone-backend.test.ts
+++ b/src/lib/bridge/phone-backend.test.ts
@@ -26,7 +26,7 @@ vi.mock('./socket', () => ({
phone.goneOnce = false;
return failure('CDP_ERROR', 'Cannot find context with specified id');
}
- return success({ result: { value: phone.answer(String(params?.expression)) } });
+ return success({ result: { value: await phone.answer(String(params?.expression)) } });
}
if (method === 'Runtime.evaluate' && String(params?.expression).includes('scrollY')) return success({ result: { value: { scrollX: 0, scrollY: 574 } } });
return success({});
@@ -36,7 +36,15 @@ vi.mock('./socket', () => ({
onDaemonClosed: () => undefined,
}));
-vi.mock('./phone-mirror', () => ({ attachedSession: async () => 'session-1' }));
+const mirror = vi.hoisted(() => ({ steer: null as ((command: unknown) => Promise<void>) | null }));
+
+vi.mock('./phone-mirror', () => ({
+ attachedSession: async () => 'session-1',
+ steerLensFromMirror: (_targetId: string, steer: (command: unknown) => Promise<void>) => {
+ mirror.steer = steer;
+ return () => (mirror.steer = null);
+ },
+}));
const { pageSideOnPhone, pickClip, shotClip } = await import('./phone-backend');
@@ -152,4 +160,19 @@ describe('A-Eye on the phone', () => {
expect(String(phone.sent.find(({ params }) => String(params?.expression).includes('page.pickElement'))?.params?.expression)).toContain('"timeoutMs":170000');
expect(phone.sent.find(({ method }) => method === 'Page.captureScreenshot')?.params?.clip).toEqual({ x: 4, y: 384, width: 232, height: 72, scale: 1 });
});
+
+ test('while the pick waits, the mirror steers the lens at the page point under a touch, and lets go once it is answered', async () => {
+ let answer: (value: unknown) => void = () => undefined;
+ phone.answer = () => new Promise((resolve) => (answer = resolve));
+ phone.viewport = { offsetX: 30, offsetY: 50, clientWidth: 205, clientHeight: 337 };
+ const picking = pageSideOnPhone('page.pickElement', {}, SESSION);
+ await vi.waitFor(() => expect(mirror.steer).not.toBeNull());
+ await mirror.steer!({ op: 'aim', x: 100, y: 200 });
+ await mirror.steer!({ op: 'wider' });
+ const steered = phone.sent.filter(({ params }) => String(params?.expression).startsWith('__browsenticPhone.lens')).map(({ params }) => params?.expression);
+ expect(steered).toEqual(['__browsenticPhone.lens({"op":"aim","x":130,"y":250})', '__browsenticPhone.lens({"op":"wider"})']);
+ answer(failure('PICK_CANCELLED', 'dismissed'));
+ await picking;
+ expect(mirror.steer).toBeNull();
+ });
});
diff --git a/src/lib/bridge/phone-backend.ts b/src/lib/bridge/phone-backend.ts
index 2e328f8..fecdff2 100644
--- a/src/lib/bridge/phone-backend.ts
+++ b/src/lib/bridge/phone-backend.ts
@@ -1,5 +1,6 @@
import { browser } from 'wxt/browser';
import { z } from 'zod';
+import type { LensCommand } from '@/lib/actions/page/lens';
import { PICK_DEFAULT_TIMEOUT_MS } from '@/lib/actions/page/pick-element';
import { pressKey } from '@/lib/actions/page/press-key';
import { scrollTo } from '@/lib/actions/page/scroll-to';
@@ -10,7 +11,7 @@ import { PHONE_API } from '@/lib/phone/page-api';
import { drag, keyPress, onScreen, tap, typing, wheel, type InputStep, type Point, type Viewport } from '@/lib/phone/touch';
import type { EvaluateReply, ToolkitPlace } from './code-toolkit';
import { currentTarget, type PhoneSession } from './phone';
-import { attachedSession } from './phone-mirror';
+import { attachedSession, steerLensFromMirror } from './phone-mirror';
import { refusalFor } from './site-guard';
import { sendCdp } from './socket';
@@ -176,6 +177,14 @@ async function screenPoint(here: Here, point: Point): Promise<Point | null> {
return viewport && onScreen(await inTopDocument(here, point), viewport);
}
+/** `screenPoint` backwards: where a touch would land, in the client coordinates of the frame actions run in. */
+async function framePoint(here: Here, point: Point): Promise<Point | null> {
+ const viewport = await visualViewport(here);
+ if (!viewport) return null;
+ const frame = await inTopDocument(here, { x: 0, y: 0 });
+ return { x: point.x + viewport.offsetX - frame.x, y: point.y + viewport.offsetY - frame.y };
+}
+
async function play(here: Here, steps: InputStep[]): Promise<CdpResult | null> {
for (const step of steps) {
const sent = await here.cdp(step.method, step.params);
@@ -322,16 +331,26 @@ export function pickClip(corner: Point, capture: PickCapture, layout: LayoutView
};
}
-/** A-Eye on the phone: the pick's own lens in the page, where a tap picks what is under it, then the element photographed before anything moves it. */
+/**
+ * A-Eye on the phone: the pick's own lens in the page, steered from the mirror tab while it waits
+ * (a tap on the phone itself still picks), then the element photographed before anything moves it.
+ */
async function pickOn(here: Here, input: Record<string, unknown>): Promise<ActionResult> {
const timeoutMs = Math.min(Number(input.timeoutMs) || PICK_DEFAULT_TIMEOUT_MS, MAX_PICK_WAIT_MS);
- const picked = await inPage(here, 'page.pickElement', { ...input, timeoutMs });
+ const { frameId } = await focusedFrame(here);
+ const release = steerLensFromMirror(here.targetId, (command) => steerLensOn(here, frameId, command));
+ const picked = await inPage(here, 'page.pickElement', { ...input, timeoutMs }, frameId).finally(release);
if (!picked.ok) return picked;
const { capture, ...data } = picked.data as { capture?: PickCapture } & Record<string, unknown>;
const shot = capture ? await pickShot(here, capture).catch(() => null) : null;
return success(shot ? { ...data, shot } : data);
}
+async function steerLensOn(here: Here, frameId: string, command: LensCommand): Promise<void> {
+ const placed = 'x' in command ? await framePoint(here, command) : {};
+ if (placed) await evaluateIn(here, frameId, `${PHONE_API}.lens(${JSON.stringify({ ...command, ...placed })})`);
+}
+
async function pickShot(here: Here, capture: PickCapture): Promise<{ dataUrl: string; width: number; height: number } | null> {
const metrics = await here.cdp('Page.getLayoutMetrics');
if (!metrics.ok) return null;
diff --git a/src/lib/bridge/phone-mirror.test.ts b/src/lib/bridge/phone-mirror.test.ts
index 2810f74..a8c2321 100644
--- a/src/lib/bridge/phone-mirror.test.ts
+++ b/src/lib/bridge/phone-mirror.test.ts
@@ -157,4 +157,45 @@ describe('the mirror in the background', () => {
await settle();
expect(socket.sent.at(-1)).toEqual({ method: 'Page.navigateToHistoryEntry', params: { entryId: 1 }, sessionId: `session-${FRONT}` });
});
+
+ test('while A-Eye waits, the page is told, its pointer steers the lens with aims collapsed, and the phone is never touched', async () => {
+ const { steerLensFromMirror } = await import('./phone-mirror');
+ const page = connectPage();
+ page.say({ op: 'watch' });
+ await settle();
+ socket.sent.length = 0;
+ const steered: unknown[] = [];
+ const release = steerLensFromMirror(FRONT, async (command) => void steered.push(command));
+ expect(page.received.at(-1)).toEqual({ kind: 'lens', active: true });
+ expect(connectPage().received).toContainEqual({ kind: 'lens', active: true });
+
+ for (let step = 1; step <= 4; step++) page.say({ op: 'lens', lens: { op: 'aim', x: step, y: step } });
+ page.say({ op: 'lens', lens: { op: 'pick', x: 4, y: 4 } });
+ await settle();
+ expect(steered).toEqual([
+ { op: 'aim', x: 1, y: 1 },
+ { op: 'aim', x: 4, y: 4 },
+ { op: 'pick', x: 4, y: 4 },
+ ]);
+ expect(methods().filter((method) => method.startsWith('Input.'))).toEqual([]);
+
+ release();
+ expect(page.received.at(-1)).toEqual({ kind: 'lens', active: false });
+ page.say({ op: 'lens', lens: { op: 'pick', x: 4, y: 4 } });
+ await settle();
+ expect(steered).toHaveLength(3);
+ });
+
+ test('a lens left on a tab the mirror moved away from is only ever cancelled', async () => {
+ const { steerLensFromMirror } = await import('./phone-mirror');
+ const page = connectPage();
+ page.say({ op: 'watch' });
+ await settle();
+ const steered: unknown[] = [];
+ steerLensFromMirror(BACK, async (command) => void steered.push(command));
+ page.say({ op: 'lens', lens: { op: 'pick', x: 4, y: 4 } });
+ page.say({ op: 'lens', lens: { op: 'cancel' } });
+ await settle();
+ expect(steered).toEqual([{ op: 'cancel' }]);
+ });
});
diff --git a/src/lib/bridge/phone-mirror.ts b/src/lib/bridge/phone-mirror.ts
index 44b573e..9ab7f5f 100644
--- a/src/lib/bridge/phone-mirror.ts
+++ b/src/lib/bridge/phone-mirror.ts
@@ -1,4 +1,5 @@
import { browser, type Browser } from 'wxt/browser';
+import type { LensCommand } from '@/lib/actions/page/lens';
import { PHONE_PORT, keyEvents, type MirrorCommand, type MirrorMessage, type PageState } from '@/lib/phone/mirror';
import type { PhoneTarget } from '@/lib/phone/types';
import { numbered, onPhoneReset, readPhone, updatePhone, type PhoneSession } from './phone';
@@ -7,7 +8,7 @@ import { onCdpEvent, onDaemonClosed, onPhoneClosed, sendCdp, type CdpEvent } fro
import { hostOf, patchSession, sessionForTab } from './tab-sessions';
type Port = Browser.runtime.Port;
-type InputCommand = Extract<MirrorCommand, { op: 'touch' | 'wheel' | 'text' | 'key' }>;
+type InputCommand = Extract<MirrorCommand, { op: 'touch' | 'wheel' | 'text' | 'key' | 'lens' }>;
/** Discovery also reports a phantom `"1"` beside a restored tab, and every command sent to it hangs (spike Q3). */
const REAL_TARGET = /^[0-9A-F]{32}$/i;
@@ -30,6 +31,7 @@ let finding: Promise<void> | null = null;
const inputs: InputCommand[] = [];
let pumping = false;
const attachments = new Map<string, Promise<string | null>>();
+let lens: { targetId: string; steer(command: LensCommand): Promise<void> } | null = null;
const broadcast = (message: MirrorMessage) => {
for (const port of ports) port.postMessage(message);
@@ -219,9 +221,24 @@ function onEvent(event: CdpEvent): void {
if (event.method === 'Page.navigatedWithinDocument') void publishPage();
}
+/** While an A-Eye pick waits on the phone, the mirror's pointer aims and picks with its lens instead of touching the page. */
+export function steerLensFromMirror(targetId: string, steer: (command: LensCommand) => Promise<void>): () => void {
+ const held = { targetId, steer };
+ lens = held;
+ broadcast({ kind: 'lens', active: true });
+ return () => {
+ if (lens !== held) return;
+ lens = null;
+ broadcast({ kind: 'lens', active: false });
+ };
+}
+
+const isAim = (command: InputCommand | undefined) => command?.op === 'lens' && command.lens.op === 'aim';
+
function queueInput(command: InputCommand): void {
const last = inputs.at(-1);
if (last?.op === 'touch' && command.op === 'touch' && last.type === 'touchMove' && command.type === 'touchMove') inputs[inputs.length - 1] = command;
+ else if (isAim(last) && isAim(command)) inputs[inputs.length - 1] = command;
else if (last?.op === 'wheel' && command.op === 'wheel') {
inputs[inputs.length - 1] = { ...command, deltaX: last.deltaX + command.deltaX, deltaY: last.deltaY + command.deltaY };
} else inputs.push(command);
@@ -239,7 +256,10 @@ async function pump(): Promise<void> {
}
async function dispatchInput(command: InputCommand): Promise<void> {
- if (command.op === 'touch') {
+ if (command.op === 'lens') {
+ const held = lens;
+ if (held && (command.lens.op === 'cancel' || held.targetId === followed?.targetId)) await held.steer(command.lens);
+ } else if (command.op === 'touch') {
const lifted = command.type === 'touchEnd' || command.type === 'touchCancel';
await call('Input.dispatchTouchEvent', { type: command.type, touchPoints: lifted ? [] : [{ x: command.x, y: command.y }] });
} else if (command.op === 'wheel') {
@@ -291,6 +311,7 @@ async function handle(port: Port, command: MirrorCommand): Promise<void> {
case 'wheel':
case 'text':
case 'key':
+ case 'lens':
return queueInput(command);
case 'go': {
const refused = await refusalFor(command.url);
@@ -340,6 +361,7 @@ export function servePhoneMirror(): void {
if (port.name !== PHONE_PORT) return;
ports.add(port);
if (page) port.postMessage({ kind: 'page', ...page });
+ if (lens) port.postMessage({ kind: 'lens', active: true });
port.onMessage.addListener((message) => void handle(port, message as MirrorCommand));
port.onDisconnect.addListener(() => {
ports.delete(port);
diff --git a/src/lib/phone/mirror.ts b/src/lib/phone/mirror.ts
index 875fc8c..9ca0f14 100644
--- a/src/lib/phone/mirror.ts
+++ b/src/lib/phone/mirror.ts
@@ -1,3 +1,5 @@
+import type { LensCommand } from '@/lib/actions/page/lens';
+
export const PHONE_PORT = 'browsentic/phone';
/** `Page.screencastFrame`'s metadata: the phone's visual viewport when the frame was taken. */
@@ -26,6 +28,8 @@ export type MirrorCommand =
| { op: 'wheel'; x: number; y: number; deltaX: number; deltaY: number }
| { op: 'text'; text: string }
| { op: 'key'; key: MirrorKey }
+ /** While A-Eye waits on the phone; points are where a touch would land. */
+ | { op: 'lens'; lens: LensCommand }
| { op: 'go'; url: string }
| { op: 'back' }
| { op: 'forward' }
@@ -45,6 +49,7 @@ export interface PageState {
export type MirrorMessage =
| { kind: 'frame'; data: string; metadata: FrameMetadata; frame: number }
| ({ kind: 'page' } & PageState)
+ | { kind: 'lens'; active: boolean }
| { kind: 'error'; message: string };
/**
diff --git a/src/lib/phone/page-api.ts b/src/lib/phone/page-api.ts
index 40ea0c6..0352aa8 100644
--- a/src/lib/phone/page-api.ts
+++ b/src/lib/phone/page-api.ts
@@ -1,16 +1,19 @@
import { dispatch } from '@/lib/actions/dispatch';
import { resolveTarget, targetSchema } from '@/lib/actions/page/dom';
+import { steerLens } from '@/lib/actions/page/lens';
import { isFrameElement } from '@/lib/actions/page/switch-frame';
export const PHONE_API = '__browsenticPhone';
/**
* What the phone's page bundle puts in its isolated world: the actions' own `dispatch`, the
- * visual viewport a touch is measured from, and the frame element a frame switch enters.
+ * visual viewport a touch is measured from, the frame element a frame switch enters, and the
+ * A-Eye lens the mirror steers.
*/
export function installPhoneApi(): void {
(globalThis as Record<string, unknown>)[PHONE_API] = {
dispatch,
+ lens: steerLens,
viewport: () => ({
left: visualViewport?.offsetLeft ?? 0,
top: visualViewport?.offsetTop ?? 0,