An agentic browser harness: an AI side panel & orb in your own logged-in Chrome, Edge, Arc, Brave or even Firefox run by the agent/subscription you already use (Claude Code, Codex, Antigravity). Free, open source, no API key. Also an MCP server.
TypeScript agentic-browseragentic-browsingagy-cliaiai-agentai-browser-extensionai-toolsbrave-extensionbrowser-agentbrowser-automationbrowser-extensionbrowser-usebrowserskillchrome-extensionclaude-codecodexfirefox-addonmcp-servernanobrowserweb-automation
feat(extension): action cues — a glowing ring on every element an agent acts on, captioned with the action and the element’s name, and the page’s edge for an action with no element; off until switched on from the composer, the mic’s menu or Settings, drawn in a closed layer that never touches the page, and cleared before any screenshot
Shahriar4be846f6b9bc
diff --git a/docs/guide/configuration.md b/docs/guide/configuration.mdindex a54bf74..11ce99f 100644--- a/docs/guide/configuration.md+++ b/docs/guide/configuration.md@@ -138,6 +138,7 @@ Below the theme, the same section holds what belongs to one browser alone. None| Block | What it sets || --- | --- |+| **On the page** | **Show what the agent does** — the [action cues](features/action-cues.md) ring on each element an agent acts on. Off until you switch it on; the composer's button and the mic's menu switch the same setting || **Right-click menu** | Whether a right-click on a page offers **Open Browsentic** (the side panel) and **Open Browsentic (Hands Free)**. Both are on until you switch one off. With both on, the browser gathers them under a **Browsentic** entry || **Keyboard shortcuts** | Shows the keys for **Open the popup**, **Open or close the side panel** and **Start or stop hands-free**. The browser owns these: it gives the side panel **Alt+Shift+B** and hands-free **Alt+Shift+H** (**⌥⇧B**, **⌥⇧H** on a Mac) at install unless something else already has them, and **Change shortcuts** opens its own shortcuts page. Firefox has the popup and the sidebar only || **Hands-free** | **Hold to talk** — the same switch as the mic's own menu |diff --git a/docs/guide/features/README.md b/docs/guide/features/README.mdindex c9cb171..aed0043 100644--- a/docs/guide/features/README.md+++ b/docs/guide/features/README.md@@ -13,6 +13,7 @@ first.| [Hands-free](hands-free.md) | Put the panel away and talk to the page through a mic you can drag anywhere || [Instant commands](instant-commands.md) | "go back", "scroll to the top" — the things that run in milliseconds without waking an agent || [A-Eye](a-eye.md) | Point at an element instead of describing it, and let it point back when it needs to |+| [Action cues](action-cues.md) | A glowing ring on whatever the agent is about to click, type into or read || [Profile](profile.md) | Your details and standing rules, saved once, so it fills a form with the real thing instead of guessing || [Blocked sites](blocked-sites.md) | Sites it never reads or acts on, whoever is driving — enforced in the browser, out of the agent's reach || [Skills](skills.md) | How an instruction gets routed to the right playbook, and how to write your own |diff --git a/docs/guide/features/action-cues.md b/docs/guide/features/action-cues.mdnew file mode 100644index 0000000..43b5f23--- /dev/null+++ b/docs/guide/features/action-cues.md@@ -0,0 +1,53 @@+# Action cues++Watching the agent work: a glowing ring on whatever it is about to touch.++---++## What you see++Every time an agent acts in a tab, the element it acts on lights up with a ring in your theme's+colour, and a small caption names the action and the element — **Click · Sign in**, **Fill ·+Email**, **Type · Message**. The ring appears a moment before the action lands, follows the element+if the page scrolls it into view, and fades shortly after the action is done. A failed action+flashes the warning colour instead.++Actions that have no element — reading the page, going to a new address, running a script — glow+around the edge of the page instead, with the caption at the top: **Read page**, **Go to+example.com**.++It works for every agent, whoever is driving: a run from the side panel or the mic, and an MCP+client such as Claude Code or Cursor working in your browser.++## Switching it on++It is off until you switch it on, in any of three places — they are one setting:++- the **Show what the agent does** button in the side panel's composer row, next to **Schedule**+- the same item in the [hands-free](hands-free.md) mic's menu+- **Settings → Extension → On the page**++## What it never does++- **It never touches the page you are on.** The ring is drawn in a closed layer of its own, the+ page's scripts cannot see into it, and nothing in it takes a click — a click that lands on the+ ring reaches the page underneath. It does not move focus, scroll, or listen to anything the page+ does.+- **It never shows what is typed.** The caption names a field by its label — **Fill · Password** —+ never by its contents, and the agent's input never reaches the page's layer at all: only where+ an action lands does.+- **It is never in a screenshot.** Before the agent takes a screenshot, or the captcha solver+ photographs a challenge, every ring on the tab is taken off first, so the agent never sees its+ own cue.+- **It is never drawn on a [blocked site](blocked-sites.md)** — the action is refused before+ anything reaches the page.++## Where it stops++- An element inside a frame is ringed inside that frame, so a ring near the frame's edge is cut off+ by it.+- `chrome://` pages and the Web Store never show one, since the browser keeps extensions out of+ them. A tab that loaded before the extension did shows none for its first action, until the+ extension has been let in.+- The ring sits above a page's own dialogs and popovers. In a browser without popover support it+ sits under a modal dialog instead.diff --git a/docs/guide/features/hands-free.md b/docs/guide/features/hands-free.mdindex 2343327..43a0751 100644--- a/docs/guide/features/hands-free.md+++ b/docs/guide/features/hands-free.md@@ -53,13 +53,14 @@ side panel in every window, and opening the panel anywhere ends hands-free.## The menu-Rest the pointer on the mic for a second and five buttons fan out around it:+Rest the pointer on the mic for a second and six buttons fan out around it:| | || --- | --- || **Attach a file** | Hands a file to this tab's conversation, as dropping it on the panel does — see [Files](files.md) || **Live code** | Lets the agent write a script for this page, as the composer's code toggle does; you approve the code before it runs || **Focus point** | Opens [A-Eye](a-eye.md): point at an element and it rides along with your next instruction |+| **Show what the agent does** | Switches [action cues](action-cues.md) — the ring on each element the agent acts on — on or off, for every tab || **Hold to talk** | Switches between listening all the time and listening only while you hold a key — see below || **Open the side panel** | Brings the panel back and ends hands-free |diff --git a/docs/internals/extension.md b/docs/internals/extension.mdindex 90108e4..01aa22d 100644--- a/docs/internals/extension.md+++ b/docs/internals/extension.md@@ -376,6 +376,44 @@ State: `browsentic/handsFree` in `storage.session` (on while present, with `mute`browsentic/pushToTalk` and `browsentic/orbPosition` — the orb's centre as viewport fractions, so one drag places it on every tab. Hands-free is session stateon purpose: a restarted browser comes back with the panel, not a live microphone.+## Action cues: a ring in whichever frame the agent acts in++Every agent action is ringed on the page — see [Action cues](../guide/features/action-cues.md). The+background drives it, because `invokeForHarness` is the one place that sees each action together+with its tab: the content script's dispatch knows no run, and hears internal sub-calls (a+screenshot is a plan plus a scroll per tile) that are not the agent's actions.++**One table decides what an action looks like.** `cueFor()` in `src/lib/cues/plan.ts` maps each+registry action to an element cue (a target, a point, or the focused element), a page cue (the+viewport's edge) or none, and a test fails if a registry action has no entry — a new action has to+choose. It is built from the **sealed** input, before `releaseForAction`, and copies only the+target fields, points, a named key and a navigation's host — never a value, a text or a query.++**`cued()` wraps the tab-bound half of `dispatch`** (`src/lib/bridge/action-cues.ts`). It sends+`show` to the focused frame for an element cue or the top frame for a page cue, waits at most+`CUE_LEAD_MS` so the ring is painted before the action fires, runs the action, then sends `settle`+with its outcome. A frame that does not answer — frozen, alerting, or without a content script —+costs the action that wait and nothing else, and a cue never fails an action. Blocked sites are+refused by `guardTarget` before `dispatch` runs, so they are never ringed. The switch is+`browsentic/actionCues` in `storage.local`, off unless it is `true`.++**The page gets one element and nothing more.** `exposeCues()` runs in every frame (before the+top-frame return in `content.ts`), and the first cue in a document mounts `#browsentic-cues` on+`documentElement` with its attributes already set — one childList mutation for the page's+observers, and none after it, because every ring, caption and fade is inside a closed shadow root.+Nothing in it takes a pointer, and it never focuses, scrolls or listens to the page. While a cue is+on screen a requestAnimationFrame loop follows the element's box; a target that has not appeared+yet (`waitForElement`) is looked for again every 250 ms.++**It rides in the top layer.** The host is a `popover="manual"`, shown while a cue is up and hidden+when none is, and raised again when a modal dialog, another popover or a fullscreen element has+appeared — so a ring on a cookie banner's button sits above the banner.++**A capture quenches first.** `screenshot`, `findCaptcha`, `solveCaptcha` and `pickElement` carry+`quench` in the table, so `cued()` clears every frame still lit on that tab — the background keeps+which ones, for as long as a cue lingers — and the host answers after two frames, once the page has+repainted without it. A tab with nothing lit is not messaged at all.+## Tab scopingA panel conversation is **bound to the tab it started in**.diff --git a/docs/internals/state.md b/docs/internals/state.mdindex eefdbd2..e9a3718 100644--- a/docs/internals/state.md+++ b/docs/internals/state.md@@ -76,7 +76,8 @@ state under `browsentic/dictation`, and the approvals it has already announced uOnly where the orb was dragged to (`browsentic/orbPosition`), whether hold-to-talk is on(`browsentic/pushToTalk`) and whether this browser's speech service has ever worked(`browsentic/speechService`) are kept in `storage.local`, along with which right-click items this-browser shows (`browsentic/contextMenu`, both on when absent).+browser shows (`browsentic/contextMenu`, both on when absent) and whether agent actions are ringed on+the page (`browsentic/actionCues`, off when absent).**Blocked sites** are `browsentic/blockedSites` in `storage.local` — a list of patterns, up to 500, thatthe daemon never receives. Removing the extension removes it; each browser keeps its own.diff --git a/src/extension/components/composer.tsx b/src/extension/components/composer.tsxindex ffc97bf..c9fe4aa 100644--- a/src/extension/components/composer.tsx+++ b/src/extension/components/composer.tsx@@ -1,5 +1,21 @@import { useEffect, useMemo, useRef, useState, type MouseEvent } from 'react';-import { CalendarClock, Code2, FileText, FileUp, Loader2, Mic, MicOff, Paperclip, RotateCw, ScanEye, Send, Sparkles, Square, X } from 'lucide-react';+import {+ CalendarClock,+ Code2,+ FileText,+ FileUp,+ Loader2,+ Mic,+ MicOff,+ Paperclip,+ RotateCw,+ ScanEye,+ Send,+ Sparkles,+ Square,+ SquareDashedMousePointer,+ X,+} from 'lucide-react';import { SkillMenu, skillMenuItems, type SkillMenuItem } from '@/extension/components/skill-menu';import type { SavedToolMeta } from '@/lib/bridge/saved-tools';@@ -34,6 +50,8 @@ export function Composer({onToggleLiveTools,scheduling,onToggleScheduling,+ actionCues,+ onToggleActionCues,onAttachSkill,onCommand,tools,@@ -65,6 +83,8 @@ export function Composer({onToggleLiveTools: () => void;scheduling: boolean;onToggleScheduling: () => void;+ actionCues: boolean;+ onToggleActionCues: () => void;onAttachSkill: (skill: AttachedSkill | null) => void;onCommand: (command: string) => void;tools: SavedToolMeta[];@@ -293,6 +313,21 @@ export function Composer({><CalendarClock className="size-3.5" /></Button>+ <Button+ variant={actionCues ? 'subtle' : 'ghost'}+ size="icon-sm"+ role="switch"+ aria-checked={actionCues}+ aria-label={actionCues ? 'Hide what the agent does' : 'Show what the agent does'}+ title={+ actionCues+ ? 'Showing what the agent does — a glowing ring marks each element it acts on. Turn it off to hide it.'+ : 'Show what the agent does — mark each element it acts on with a glowing ring.'+ }+ onClick={onToggleActionCues}+ >+ <SquareDashedMousePointer className="size-3.5" />+ </Button><Buttonvariant={voiceEnabled && !voice.error ? 'subtle' : 'ghost'}size="icon-sm"diff --git a/src/extension/components/extension-settings.tsx b/src/extension/components/extension-settings.tsxindex 0ee5600..fe83514 100644--- a/src/extension/components/extension-settings.tsx+++ b/src/extension/components/extension-settings.tsx@@ -4,7 +4,13 @@ import { ExternalLink } from 'lucide-react';import { Button } from '@/extension/components/ui/button';import { Switch } from '@/extension/components/ui/switch';import { HANDS_FREE_SHORTCUT } from '@/lib/settings/shortcuts';-import { openShortcutSettings, useContextMenuChoice, usePushToTalk, useShortcuts } from '@/lib/bridge/use-extension-settings';+import {+ openShortcutSettings,+ useActionCues,+ useContextMenuChoice,+ usePushToTalk,+ useShortcuts,+} from '@/lib/bridge/use-extension-settings';import { useHandsFreeSupported } from '@/lib/bridge/use-speech';import { cn } from '@/lib/utils';@@ -12,6 +18,7 @@ export function ExtensionSettings() {const handsFree = useHandsFreeSupported();return (<>+ <ActionCues /><ContextMenuItems handsFree={handsFree} /><KeyboardShortcuts handsFree={handsFree} />{handsFree && <HandsFree />}@@ -19,6 +26,23 @@ export function ExtensionSettings() {);}+function ActionCues() {+ const [on, setOn] = useActionCues();+ return (+ <SettingsGroup+ title="On the page"+ note="What the page shows while an agent works in it. The composer and the mic’s menu switch the same setting."+ >+ <Row+ title="Show what the agent does"+ note="A glowing ring marks each element the agent clicks, types into or reads, with a short caption. It never takes a click, and screenshots leave it out."+ >+ <Switch checked={on} label="Show what the agent does" onChange={setOn} />+ </Row>+ </SettingsGroup>+ );+}+function ContextMenuItems({ handsFree }: { handsFree: boolean }) {const [choice, setChoice] = useContextMenuChoice();return (diff --git a/src/extension/entrypoints/content.ts b/src/extension/entrypoints/content.tsindex be5ad08..d4f1a1a 100644--- a/src/extension/entrypoints/content.ts+++ b/src/extension/entrypoints/content.ts@@ -1,4 +1,5 @@import { exposeActions } from '@/lib/actions/host';+import { exposeCues } from '@/lib/cues/host';import { exposeFrameProbe } from '@/lib/frames/host';import { exposeHandsFree } from '@/lib/handsfree/host';import { exposeIndicator } from '@/lib/indicator/host';@@ -15,6 +16,7 @@ export default defineContentScript({world.__browsenticActions = true;exposeActions();exposeFrameProbe();+ exposeCues();if (window !== window.top) return;exposeRecorder();exposeMonitor();diff --git a/src/extension/entrypoints/sidepanel/App.tsx b/src/extension/entrypoints/sidepanel/App.tsxindex b42cb52..80f8ff4 100644--- a/src/extension/entrypoints/sidepanel/App.tsx+++ b/src/extension/entrypoints/sidepanel/App.tsx@@ -34,7 +34,7 @@ import { putBytes } from '@/lib/bridge/file-store';import { removeRecording, type StoredRecordingMeta } from '@/lib/bridge/recording-store';import { removeSession } from '@/lib/bridge/session-store';import { useActiveTabUrl } from '@/lib/bridge/use-active-tab-url';-import { useBlockedPattern } from '@/lib/bridge/use-extension-settings';+import { useActionCues, useBlockedPattern } from '@/lib/bridge/use-extension-settings';import { useDaemonState } from '@/lib/bridge/use-daemon-state';import { useFileDrop } from '@/lib/bridge/use-file-drop';import { closeSidePanel } from '@/lib/bridge/side-panel';@@ -61,6 +61,7 @@ export default function App() {const daemon = useDaemonState();const run = useRun();const [voiceEnabled, setVoiceEnabled] = useVoiceEnabled();+ const [actionCues, setActionCues] = useActionCues();const handsFree = useHandsFreeSupported();const storedFiles = useStoredFiles();const sessions = useStoredSessions();@@ -489,6 +490,8 @@ export default function App() {onToggleLiveTools={() => setLiveTools((on) => !on)}scheduling={scheduling}onToggleScheduling={() => setScheduling((on) => !on)}+ actionCues={actionCues}+ onToggleActionCues={() => setActionCues(!actionCues)}onAttachSkill={setAttachedSkill}handsFree={handsFree}onCommand={(command) => {diff --git a/src/lib/bridge/action-cues.test.ts b/src/lib/bridge/action-cues.test.tsnew file mode 100644index 0000000..1ed32d8--- /dev/null+++ b/src/lib/bridge/action-cues.test.ts@@ -0,0 +1,122 @@+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';+import { fakeBrowser } from 'wxt/testing';+import { ACTION_CHANNEL, success } from '@/lib/actions/protocol';+import { CUE_CHANNEL } from '@/lib/cues/events';+import { cueFor } from '@/lib/cues/plan';+import { BLOCKED_SITES_KEY } from '@/lib/settings/blocked-sites';+import { ACTION_CUES_KEY, CUE_LEAD_MS, cued, quenchCues } from './action-cues';+import { forgetFrameFocus, setFramePath } from './frame-focus';+import { invokeForHarness } from './invoke';++type Sent = { tabId: number; frameId?: number; message: { channel: string; op?: string; action?: string } & Record<string, unknown> };++let sent: Sent[];+let answer: (message: Sent['message']) => unknown;++const cues = () => sent.filter((entry) => entry.message.channel === CUE_CHANNEL);+const ops = () => sent.map((entry) => (entry.message.channel === CUE_CHANNEL ? `cue:${entry.message.op}` : `action:${entry.message.action}`));+const tabOn = async (url: string) => (await fakeBrowser.tabs.create({ url, active: true })).id!;++beforeEach(async () => {+ fakeBrowser.reset();+ sent = [];+ answer = (message) => (message.channel === ACTION_CHANNEL ? success({ done: true }) : { ok: true });+ vi.spyOn(fakeBrowser.tabs, 'sendMessage').mockImplementation((async (tabId: number, message: Sent['message'], options?: { frameId?: number }) => {+ sent.push({ tabId, frameId: options?.frameId, message });+ return answer(message);+ }) as never);+ vi.spyOn(fakeBrowser.runtime, 'getManifest').mockReturnValue({ content_scripts: [] } as never);+ await fakeBrowser.storage.local.set({ [ACTION_CUES_KEY]: true });+});++afterEach(() => {+ vi.useRealTimers();+ vi.restoreAllMocks();+});++describe('cued', () => {+ it('draws nothing until the user switches cues on', async () => {+ await fakeBrowser.storage.local.remove(ACTION_CUES_KEY);+ const tabId = await tabOn('https://example.com/');+ await invokeForHarness('page.clickElement', { target: { selector: '#go' } }, tabId);+ expect(ops()).toEqual(['action:page.clickElement']);+ });++ it('rings the element before the action runs and settles it after', async () => {+ const tabId = await tabOn('https://example.com/');+ const result = await invokeForHarness('page.clickElement', { target: { selector: '#go' } }, tabId);++ expect(result.ok).toBe(true);+ expect(ops()).toEqual(['cue:show', 'action:page.clickElement', 'cue:settle']);+ const [show, settle] = cues();+ expect(show.message).toMatchObject({ plan: { kind: 'element', verb: 'Click' }, theme: 'ember' });+ expect(settle.message).toMatchObject({ id: show.message.id, ok: true });+ });++ it('settles a failed action as failed', async () => {+ const tabId = await tabOn('https://example.com/');+ answer = (message) =>+ message.channel === ACTION_CHANNEL ? { ok: false, error: { code: 'TARGET_NOT_FOUND', message: 'none' } } : { ok: true };+ await invokeForHarness('page.clickElement', { target: { selector: '#go' } }, tabId);+ expect(cues().at(-1)?.message).toMatchObject({ op: 'settle', ok: false });+ });++ it('never carries what the agent types', async () => {+ const tabId = await tabOn('https://example.com/');+ await invokeForHarness('page.fillInput', { target: { selector: '#pw' }, value: 'hunter2' }, tabId);+ expect(cues()).toHaveLength(2);+ expect(JSON.stringify(cues())).not.toContain('hunter2');+ });++ it('draws nothing on a blocked site', async () => {+ await fakeBrowser.storage.local.set({ [BLOCKED_SITES_KEY]: ['mybank.com'] });+ const tabId = await tabOn('https://mybank.com/');+ const result = await invokeForHarness('page.clickElement', { target: { selector: '#go' } }, tabId);+ expect(result.ok).toBe(false);+ expect(sent).toEqual([]);+ });++ it('draws nothing once the user has switched cues off', async () => {+ await fakeBrowser.storage.local.set({ [ACTION_CUES_KEY]: false });+ const tabId = await tabOn('https://example.com/');+ await invokeForHarness('page.clickElement', { target: { selector: '#go' } }, tabId);+ expect(ops()).toEqual(['action:page.clickElement']);+ });++ it('sends a page cue to the top frame and an element cue to the frame the agent is in', async () => {+ await setFramePath(500, [{ frameId: 3, url: 'https://ads.example/', selector: 'iframe' }]);+ await cued(500, cueFor('page.getPageInfo', {}), async () => success(null));+ await cued(500, cueFor('page.clickElement', { target: { selector: '#go' } }), async () => success(null));+ await forgetFrameFocus(500);+ expect(cues().map((entry) => [entry.message.op, entry.frameId])).toEqual([+ ['show', 0],+ ['settle', 0],+ ['show', 3],+ ['settle', 3],+ ]);+ });++ it('never holds an action up longer than the lead for a tab that does not answer', async () => {+ vi.useFakeTimers();+ answer = () => new Promise(() => undefined);+ const perform = vi.fn(async () => success(null));+ const running = cued(9, cueFor('page.clickElement', { target: { selector: '#go' } }), perform);+ await vi.advanceTimersByTimeAsync(CUE_LEAD_MS - 1);+ expect(perform).not.toHaveBeenCalled();+ await vi.advanceTimersByTimeAsync(1);+ expect(perform).toHaveBeenCalledOnce();+ await running;+ });++ it('clears a lingering ring before a capture, and asks nothing of a tab with none', async () => {+ await cued(4, cueFor('page.clickElement', { target: { selector: '#go' } }), async () => success(null));+ sent = [];+ await cued(4, cueFor('page.screenshot', {}), async () => success(null));+ expect(ops()).toEqual(['cue:quench']);++ sent = [];+ await quenchCues(4);+ await quenchCues(12);+ expect(sent).toEqual([]);+ });+});diff --git a/src/lib/bridge/action-cues.ts b/src/lib/bridge/action-cues.tsnew file mode 100644index 0000000..7710eef--- /dev/null+++ b/src/lib/bridge/action-cues.ts@@ -0,0 +1,91 @@+import { browser } from 'wxt/browser';+import type { ActionResult } from '@/lib/actions/protocol';+import { CUE_CHANNEL, CUE_FADE_MS, CUE_LINGER_MS, type CueCommand, type CuePlan } from '@/lib/cues/events';+import { focusedFrame, TOP_FRAME } from './frame-focus';+import { readTheme } from './theme';++export const ACTION_CUES_KEY = 'browsentic/actionCues';++export const CUE_LEAD_MS = 120;++const QUENCH_WAIT_MS = 300;++const LIT_FOR_MS = CUE_LINGER_MS + CUE_FADE_MS + 200;++type Unsent<T> = T extends unknown ? Omit<T, 'channel'> : never;++type Timer = ReturnType<typeof setTimeout>;++const lit = new Map<number, Map<number, Timer | undefined>>();++export const actionCuesOn = (value: unknown): boolean => value === true;++export async function readActionCues(): Promise<boolean> {+ const stored = await browser.storage.local.get(ACTION_CUES_KEY);+ return actionCuesOn(stored[ACTION_CUES_KEY]);+}++/**+ * Shows where an action lands while it runs. The ring is asked for first and given a few+ * frames to paint, never longer: a tab that cannot answer — frozen, or held by an alert —+ * costs the action at most CUE_LEAD_MS, and a failed cue never fails the action.+ */+export async function cued(tabId: number, plan: CuePlan, perform: () => Promise<ActionResult>): Promise<ActionResult> {+ if (plan.quench) await quenchCues(tabId);+ if (plan.kind === 'none') return perform();+ const [enabled, theme, frameId] = await Promise.all([+ readActionCues(),+ readTheme(),+ plan.kind === 'page' ? TOP_FRAME : focusedFrame(tabId),+ ]);+ if (!enabled) return perform();++ const id = crypto.randomUUID();+ await within(CUE_LEAD_MS, post(tabId, frameId, { op: 'show', id, plan, theme }));+ light(tabId, frameId, false);++ let ok = false;+ try {+ const result = await perform();+ ok = result.ok;+ return result;+ } finally {+ void post(tabId, frameId, { op: 'settle', id, ok });+ light(tabId, frameId, true);+ }+}++/** Takes every ring off the tab before its pixels are read, so a capture never shows the agent its own cue. */+export async function quenchCues(tabId: number): Promise<void> {+ const frames = lit.get(tabId);+ if (!frames) return;+ lit.delete(tabId);+ for (const timer of frames.values()) clearTimeout(timer);+ await within(+ QUENCH_WAIT_MS,+ Promise.all([...frames.keys()].map((frameId) => post(tabId, frameId, { op: 'quench' }))),+ );+}++function light(tabId: number, frameId: number, settling: boolean): void {+ const frames = lit.get(tabId) ?? new Map<number, Timer | undefined>();+ clearTimeout(frames.get(frameId));+ const out = () => {+ frames.delete(frameId);+ if (!frames.size && lit.get(tabId) === frames) lit.delete(tabId);+ };+ frames.set(frameId, settling ? setTimeout(out, LIT_FOR_MS) : undefined);+ lit.set(tabId, frames);+}++function post(tabId: number, frameId: number, command: Unsent<CueCommand>): Promise<unknown> {+ return browser.tabs.sendMessage(tabId, { channel: CUE_CHANNEL, ...command }, { frameId }).catch(() => undefined);+}++function within(ms: number, work: Promise<unknown>): Promise<void> {+ let timer: ReturnType<typeof setTimeout> | undefined;+ const late = new Promise<void>((resolve) => {+ timer = setTimeout(resolve, ms);+ });+ return Promise.race([work.then(() => undefined), late]).finally(() => clearTimeout(timer));+}diff --git a/src/lib/bridge/hands-free.test.ts b/src/lib/bridge/hands-free.test.tsindex 43cfeaa..7ade2ef 100644--- a/src/lib/bridge/hands-free.test.ts+++ b/src/lib/bridge/hands-free.test.ts@@ -9,6 +9,7 @@ const world: OrbWorld = {state: { muted: false, since: 1 },phase: 'listening',pushToTalk: false,+ cues: true,listeningTab: 7,};@@ -27,6 +28,11 @@ const session = (patch: Partial<TabSession>): TabSession => ({});describe('describeOrb', () => {+ it('carries the action-cues switch to every tab', () => {+ expect(describeOrb(world, 7, null).cues).toBe(true);+ expect(describeOrb({ ...world, cues: false }, 8, null).cues).toBe(false);+ });+it('listens only in the tab the microphone is aimed at', () => {expect(describeOrb(world, 7, null).voice).toBe('listening');expect(describeOrb(world, 8, null).voice).toBe('paused');diff --git a/src/lib/bridge/hands-free.ts b/src/lib/bridge/hands-free.tsindex 95d0d42..13b2573 100644--- a/src/lib/bridge/hands-free.ts+++ b/src/lib/bridge/hands-free.ts@@ -20,6 +20,7 @@ import {import { pickFocusIn } from './aeye';import { putBytes } from './file-store';import { openMicPermissionPage } from './mic-permission';+import { ACTION_CUES_KEY, readActionCues } from './action-cues';import { HANDS_FREE_KEY, PUSH_TO_TALK_KEY, endHandsFree, readHandsFree, writeHandsFree, type HandsFreeState } from './panel-view';import { attachFile, closePanels, onPanelPresence, onTurnSettled, runCommand } from './run-port';import { openSidePanel } from './side-panel';@@ -72,6 +73,7 @@ export interface OrbWorld {state: HandsFreeState;phase: DictationPhase | null;pushToTalk: boolean;+ cues: boolean;listeningTab: number | null;}@@ -85,6 +87,7 @@ export function describeOrb(world: OrbWorld, tabId: number, session: TabSessionrun: pending ? 'approval' : session?.runId ? 'working' : 'idle',voice: world.state.muted ? 'muted' : tabId === world.listeningTab ? (world.phase ?? 'starting') : 'paused',pushToTalk: world.pushToTalk,+ cues: world.cues,...(pending && { approval: approvalOf(pending) }),};}@@ -151,13 +154,14 @@ async function paint(): Promise<void> {}showing = true;- const [daemon, sessions, theme, position, phase, pushToTalk, front, tabs] = await Promise.all([+ const [daemon, sessions, theme, position, phase, pushToTalk, cues, front, tabs] = await Promise.all([readDaemon(),readTabSessions(),readTheme(),readPosition(),readPhase(),readPushToTalk(),+ readActionCues(),frontTab(),browser.tabs.query({}),]);@@ -171,7 +175,7 @@ async function paint(): Promise<void> {/* The tab in front is told afresh until it shows an orb, and healed if the extension was reloaded under it. */if (frontId != null && !orbTabs.has(frontId)) sent.delete(frontId);- const world: OrbWorld = { theme, position, link, state, phase, pushToTalk, listeningTab: target };+ const world: OrbWorld = { theme, position, link, state, phase, pushToTalk, cues, listeningTab: target };await Promise.all(tabs.map(async (tab) => {if (tab.id == null || tab.discarded) return;@@ -441,6 +445,10 @@ function answer(request: OrbRequest, tab: Browser.tabs.Tab & { id: number }, resvoid browser.storage.local.set({ [PUSH_TO_TALK_KEY]: request.on === true });respond({ ok: true });return false;+ case 'cues':+ void browser.storage.local.set({ [ACTION_CUES_KEY]: request.on === true });+ respond({ ok: true });+ return false;case 'talk':/* Letting go always reaches the page: the key can come up in a tab the mic has already left. */if (!request.on || tab.id === listeningTab) {@@ -530,7 +538,8 @@ export function serveHandsFree(): void {}});browser.storage.local.onChanged.addListener((changes) => {- if (THEME_KEY in changes || ORB_POSITION_KEY in changes || SPEECH_SERVICE_KEY in changes || PUSH_TO_TALK_KEY in changes) {+ const repaints = [THEME_KEY, ORB_POSITION_KEY, SPEECH_SERVICE_KEY, PUSH_TO_TALK_KEY, ACTION_CUES_KEY];+ if (repaints.some((key) => key in changes)) {void syncHandsFree();}});diff --git a/src/lib/bridge/invoke.ts b/src/lib/bridge/invoke.tsindex c7661b0..b469758 100644--- a/src/lib/bridge/invoke.ts+++ b/src/lib/bridge/invoke.ts@@ -56,6 +56,9 @@ import { awaitMonitorDone, monitorStatusFor, startTabMonitor, stopTabMonitor } fimport { startJobTimer, stopJobTimer, timerStatusFor } from '@/lib/bridge/timer';import { listRecordings as listStoredMeta, readRecordingBody, recordedOnBlockedSite } from '@/lib/bridge/recording-store';import { pickInTab } from '@/lib/bridge/pick';+import { cued } from '@/lib/bridge/action-cues';+import { cueFor } from '@/lib/cues/plan';+import type { CuePlan } from '@/lib/cues/events';import { screenshotTab } from '@/lib/bridge/screenshot';import { dragInTab, trustedClickInTab } from '@/lib/bridge/trusted-input';import { closeOpenTab, openNewTab, switchToTab, watchForLoad, type TabRef } from '@/lib/bridge/tabs';@@ -82,7 +85,7 @@ export async function invokeForHarness(if (release.refused.length) return failure('SECRET_NOT_RELEASABLE', REFUSED_MESSAGE);if (release.unresolved.length) return failure('SECRET_EXPIRED', EXPIRED_MESSAGE);- const result = await dispatch(action, release.input, tabId, runId);+ const result = await dispatch(action, release.input, tabId, runId, cueFor(action, input));const landed = await guardLanding(action, result, tabId, runId);if (landed) return landed;const { value } = await sealForPage(result, await targetOrigin(tabId, runId));@@ -203,8 +206,9 @@ function hostOf(url: string): string | undefined {async function dispatch(action: string,input: unknown,- tabId?: number,- runId?: string,+ tabId: number | undefined,+ runId: string | undefined,+ cue: CuePlan,): Promise<ActionResult> {if (action === listFiles.name) return listStoredFiles(input);if (action === listRecordings.name) return listStoredRecordings(input);@@ -233,6 +237,17 @@ async function dispatch(? failure('NO_ACTIVE_TAB', 'No active tab to control'): failure('MAPPING_TAB_CHANGED', 'The tab this run started in has been closed.');}+ const { id, windowId, url } = tab;+ return cued(id, cue, () => routeInTab({ id, windowId, url }, action, input, runId, owner));+}++async function routeInTab(+ tab: { id: number; windowId?: number; url?: string },+ action: string,+ input: unknown,+ runId: string | undefined,+ owner: TabSession | null,+): Promise<ActionResult> {if (action === navigate.name) return navigateTab(tab.id, input);if (action === searchSite.name) return searchTab(tab.id, input);if (action === switchTab.name) return switchSessionTab({ id: tab.id, windowId: tab.windowId }, input, owner);diff --git a/src/lib/bridge/use-extension-settings.ts b/src/lib/bridge/use-extension-settings.tsindex 170f035..c7e06da 100644--- a/src/lib/bridge/use-extension-settings.ts+++ b/src/lib/bridge/use-extension-settings.ts@@ -2,6 +2,7 @@ import { useCallback, useEffect, useMemo, useState } from 'react';import { browser } from 'wxt/browser';import { BLOCKED_SITES_KEY, blockedBy, compileBlockedSites } from '@/lib/settings/blocked-sites';import { shortcutsOf, shortcutsPageFor, type Shortcut } from '@/lib/settings/shortcuts';+import { ACTION_CUES_KEY, actionCuesOn } from './action-cues';import { CONTEXT_MENU_KEY, asContextMenuChoice, type ContextMenuChoice } from './context-menu';import { brandFrom } from './identity';import { PUSH_TO_TALK_KEY } from './panel-view';@@ -42,6 +43,8 @@ export const useContextMenuChoice = (): [ContextMenuChoice, (next: ContextMenuChexport const usePushToTalk = (): [boolean, (on: boolean) => void] => useLocalSetting(PUSH_TO_TALK_KEY, isOn);+export const useActionCues = (): [boolean, (on: boolean) => void] => useLocalSetting(ACTION_CUES_KEY, actionCuesOn);+const asPatterns = (value: unknown): string[] =>Array.isArray(value) ? value.filter((entry): entry is string => typeof entry === 'string') : [];diff --git a/src/lib/cues/events.ts b/src/lib/cues/events.tsnew file mode 100644index 0000000..b2213ff--- /dev/null+++ b/src/lib/cues/events.ts@@ -0,0 +1,45 @@+import type { ThemeId } from '@/lib/settings/theme';++export const CUE_CHANNEL = 'browsentic/cue';++export const CUE_LINGER_MS = 700;++export const CUE_FADE_MS = 220;++export interface CueTarget {+ selector?: string;+ text?: string;+ role?: string;+ nth: number;+}++export interface CuePoint {+ x: number;+ y: number;+}++export type CueAnchor =+ | { target: CueTarget; includeHidden?: boolean }+ | { point: CuePoint }+ | { focused: true };++export interface CuePlan {+ kind: 'none' | 'element' | 'page';+ verb: string;+ detail?: string;+ anchors: CueAnchor[];+ quench?: boolean;+}++export type CueCommand =+ | { channel: typeof CUE_CHANNEL; op: 'show'; id: string; plan: CuePlan; theme: ThemeId }+ | { channel: typeof CUE_CHANNEL; op: 'settle'; id: string; ok: boolean }+ | { channel: typeof CUE_CHANNEL; op: 'quench' };++const OPS: readonly unknown[] = ['show', 'settle', 'quench'];++export function isCueCommand(message: unknown): message is CueCommand {+ if (typeof message !== 'object' || message === null) return false;+ const frame = message as { channel?: unknown; op?: unknown };+ return frame.channel === CUE_CHANNEL && OPS.includes(frame.op);+}diff --git a/src/lib/cues/host.test.ts b/src/lib/cues/host.test.tsnew file mode 100644index 0000000..1c1ed32--- /dev/null+++ b/src/lib/cues/host.test.ts@@ -0,0 +1,197 @@+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';+import { fakeBrowser } from 'wxt/testing';+import { OVERLAY_ATTRIBUTE } from '@/lib/overlay';+import { CUE_CHANNEL, CUE_FADE_MS, CUE_LINGER_MS, type CueCommand, type CuePlan } from './events';+import { CUE_HOST_ID, exposeCues } from './host';++type Unsent<T> = T extends unknown ? Omit<T, 'channel'> : never;+type Trigger = (message: unknown, sender: unknown, respond: (value: unknown) => void) => unknown;++let root: ShadowRoot | null;+let watcher: MutationObserver;+let mutations: MutationRecord[];++/* The host answers through sendResponse once it has painted, which the fake's typing leaves out. */+function deliver(command: Unsent<CueCommand>): Promise<unknown> {+ return new Promise((resolve) => {+ void (fakeBrowser.runtime.onMessage.trigger as unknown as Trigger)({ channel: CUE_CHANNEL, ...command }, {}, resolve);+ });+}++async function show(id: string, plan: CuePlan): Promise<unknown> {+ const reply = deliver({ op: 'show', id, plan, theme: 'ember' });+ await vi.advanceTimersByTimeAsync(40);+ return reply;+}++const click = (selector: string): CuePlan => ({ kind: 'element', verb: 'Click', anchors: [{ target: { selector, nth: 0 } }] });++function laidOut<T extends Element>(element: T, box = { left: 100, top: 200, width: 80, height: 30 }): T {+ vi.spyOn(element as Element, 'getBoundingClientRect').mockReturnValue({+ ...box,+ x: box.left,+ y: box.top,+ right: box.left + box.width,+ bottom: box.top + box.height,+ toJSON: () => box,+ } as DOMRect);+ return element;+}++function onPage<T extends HTMLElement>(html: string): T {+ const wrap = document.createElement('div');+ wrap.innerHTML = html;+ const element = wrap.firstElementChild as T;+ document.body.append(element);+ return element;+}++const $ = <T extends Element = HTMLElement>(selector: string) => root!.querySelector(selector) as T;+const $$ = (selector: string) => [...root!.querySelectorAll<HTMLElement>(selector)];+const chip = () => $('.cue:not(.out) .chip').textContent;++beforeEach(() => {+ vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout', 'requestAnimationFrame', 'cancelAnimationFrame', 'performance'] });+ fakeBrowser.reset();+ root = null;+ document.body.innerHTML = '';+ const attach = Element.prototype.attachShadow;+ vi.spyOn(Element.prototype, 'attachShadow').mockImplementation(function (this: Element, init: ShadowRootInit) {+ root = attach.call(this, { ...init, mode: 'open' });+ return root;+ });+ mutations = [];+ watcher = new MutationObserver((records) => mutations.push(...records));+ watcher.observe(document.documentElement, { childList: true, subtree: true, attributes: true, characterData: true });+ exposeCues();+});++afterEach(() => {+ watcher.disconnect();+ vi.useRealTimers();+ vi.restoreAllMocks();+ document.getElementById(CUE_HOST_ID)?.remove();+});++describe('action cues on the page', () => {+ it('rings the element the action will hit, captioned with what a person would call it', async () => {+ laidOut(onPage('<button id="go">Sign in</button>'));+ expect(await show('a', click('#go'))).toEqual({ ok: true });++ const ring = $('.ring');+ expect(ring.classList.contains('hidden')).toBe(false);+ expect(ring.style.transform).toBe('translate(96px, 196px)');+ expect(ring.style.width).toBe('88px');+ expect(ring.style.height).toBe('38px');+ expect(chip()).toBe('Click · Sign in');+ expect($('.edge').classList.contains('hidden')).toBe(true);+ });++ it('adds one marked element to the page on its first cue and never touches the page again', async () => {+ laidOut(onPage('<button id="go">Sign in</button>'));+ await Promise.resolve();+ mutations = [];++ await show('a', click('#go'));+ await deliver({ op: 'settle', id: 'a', ok: true });+ await show('b', { kind: 'page', verb: 'Read page', anchors: [] });+ await vi.advanceTimersByTimeAsync(CUE_LINGER_MS + CUE_FADE_MS + 100);+ const quenched = deliver({ op: 'quench' });+ await vi.advanceTimersByTimeAsync(40);+ await quenched;++ const host = document.getElementById(CUE_HOST_ID)!;+ expect(host.hasAttribute(OVERLAY_ATTRIBUTE)).toBe(true);+ expect(host.getAttribute('aria-hidden')).toBe('true');+ expect(host.style.pointerEvents).toBe('none');+ const records = [...mutations, ...watcher.takeRecords()];+ expect(records).toHaveLength(1);+ expect([...records[0].addedNodes]).toEqual([host]);+ });++ it('leaves focus and scroll where the page had them', async () => {+ const field = onPage<HTMLInputElement>('<input id="q" aria-label="Search">');+ field.focus();+ laidOut(onPage('<button id="go">Sign in</button>'));+ const focus = vi.spyOn(HTMLElement.prototype, 'focus');+ const scrollIntoView = vi.fn();+ Element.prototype.scrollIntoView = scrollIntoView;++ await show('a', click('#go'));+ await vi.advanceTimersByTimeAsync(500);++ expect(document.activeElement).toBe(field);+ expect(focus).not.toHaveBeenCalled();+ expect(scrollIntoView).not.toHaveBeenCalled();+ expect(window.scrollY).toBe(0);+ });++ it('never names a field by what has been typed into it', async () => {+ laidOut(onPage('<label>Password <input id="pw" type="password" value="hunter2"></label>').querySelector('input')!);+ await show('a', { kind: 'element', verb: 'Fill', anchors: [{ target: { selector: '#pw', nth: 0 } }] });+ expect(chip()).toBe('Fill · Password');++ laidOut(onPage('<div id="note" contenteditable="true" aria-label="Message">my secret plan</div>'));+ await show('b', { kind: 'element', verb: 'Type', anchors: [{ target: { selector: '#note', nth: 0 } }] });+ expect(chip()).toBe('Type · Message');+ expect(root!.textContent).not.toContain('secret');+ expect(root!.textContent).not.toContain('hunter2');+ });++ it('glows around the page when there is no element, or the element is not there', async () => {+ await show('a', { kind: 'page', verb: 'Go to', detail: 'example.com', anchors: [] });+ expect($('.edge').classList.contains('hidden')).toBe(false);+ expect(chip()).toBe('Go to example.com');++ await show('b', click('#missing'));+ expect($('.cue:not(.out) .edge').classList.contains('hidden')).toBe(false);+ expect($('.cue:not(.out) .ring').classList.contains('hidden')).toBe(true);+ });++ it('finds an element it is waiting for once the page adds it', async () => {+ await show('a', { kind: 'element', verb: 'Wait for', anchors: [{ target: { selector: '#late', nth: 0 } }] });+ expect($('.ring').classList.contains('hidden')).toBe(true);++ laidOut(onPage('<button id="late">Continue</button>'));+ await vi.advanceTimersByTimeAsync(400);+ expect($('.ring').classList.contains('hidden')).toBe(false);+ expect(chip()).toBe('Wait for · Continue');+ });++ it('lingers once the action is done, then fades and clears', async () => {+ laidOut(onPage('<button id="go">Sign in</button>'));+ await show('a', click('#go'));+ await deliver({ op: 'settle', id: 'a', ok: false });+ expect($('.cue').classList.contains('failed')).toBe(true);++ await vi.advanceTimersByTimeAsync(CUE_LINGER_MS);+ expect($('.cue').classList.contains('out')).toBe(true);+ await vi.advanceTimersByTimeAsync(CUE_FADE_MS);+ expect($$('.cue')).toHaveLength(0);+ });++ it('hands over to the next cue at once', async () => {+ laidOut(onPage('<button id="go">Sign in</button>'));+ await show('a', click('#go'));+ await show('b', { kind: 'page', verb: 'Read page', anchors: [] });+ expect($$('.cue')).toHaveLength(2);+ expect($$('.cue')[0].classList.contains('out')).toBe(true);+ expect($$('.cue')[1].classList.contains('out')).toBe(false);+ });++ it('clears every cue at once when asked to stay out of a capture', async () => {+ laidOut(onPage('<button id="go">Sign in</button>'));+ await show('a', click('#go'));+ const reply = deliver({ op: 'quench' });+ expect($$('.cue')).toHaveLength(0);+ await vi.advanceTimersByTimeAsync(40);+ expect(await reply).toEqual({ ok: true });+ });++ it('rings a point when the action names coordinates instead of an element', async () => {+ await show('a', { kind: 'element', verb: 'Click', anchors: [{ point: { x: 50, y: 60 } }] });+ const ring = $('.ring.point');+ expect(ring.classList.contains('hidden')).toBe(false);+ expect(ring.style.transform).toBe('translate(36px, 46px)');+ });+});diff --git a/src/lib/cues/host.ts b/src/lib/cues/host.tsnew file mode 100644index 0000000..9ae9f86--- /dev/null+++ b/src/lib/cues/host.ts@@ -0,0 +1,309 @@+import { browser } from 'wxt/browser';+import { resolveTarget } from '@/lib/actions/page/dom';+import { OVERLAY_ATTRIBUTE } from '@/lib/overlay';+import { RAIL_PALETTES, RAIL_TONES } from '@/lib/rail/events';+import type { ThemeId } from '@/lib/settings/theme';+import { CUE_FADE_MS, CUE_LINGER_MS, isCueCommand, type CueAnchor, type CuePlan } from './events';+import { cueLabel } from './label';+import { CUE_GLOW, HOST_STYLE, STYLES } from './styles';++export { CUE_CHANNEL } from './events';++export const CUE_HOST_ID = 'browsentic-cues';++const RING_PAD = 4;+const POINT_SIZE = 28;+const CHIP_GAP = 6;+const CHIP_HEIGHT = 22;+const EDGE_CHIP_TOP = 12;+const VIEW_MARGIN = 8;+const RESOLVE_EVERY_MS = 250;+const LIVE_LIMIT_MS = 60_000;+const PAINT_WAIT_MS = 100;++interface Box {+ x: number;+ y: number;+ w: number;+ h: number;+}++interface Mark {+ anchor: CueAnchor;+ element: Element | null;+ ring: HTMLElement;+}++interface Cue {+ id: string;+ plan: CuePlan;+ node: HTMLElement;+ edge: HTMLElement;+ chip: HTMLElement;+ marks: Mark[];+ live: boolean;+ resolvedAt: number;+ timers: ReturnType<typeof setTimeout>[];+}++interface Layer {+ alive: () => boolean;+ show: (id: string, plan: CuePlan, theme: ThemeId) => void;+ settle: (id: string, ok: boolean) => void;+ quench: () => void;+}++/**+ * Where the agent is acting, drawn into the frame it acts in. The page gets one element on+ * its first cue and never another change: every ring, caption and fade lives in a closed+ * shadow root, nothing in it takes a pointer, and nothing here focuses, scrolls or listens+ * to the page.+ */+export function exposeCues(): void {+ let layer: Layer | null = null;++ browser.runtime.onMessage.addListener((message: unknown, _sender, sendResponse) => {+ if (!isCueCommand(message)) return;+ if (message.op === 'settle') {+ layer?.settle(message.id, message.ok);+ sendResponse({ ok: true });+ return;+ }+ if (message.op === 'show') {+ if (!layer?.alive()) layer = mountLayer();+ layer?.show(message.id, message.plan, message.theme);+ } else layer?.quench();+ void painted().then(() => sendResponse({ ok: true }));+ return true;+ });++ document.getElementById(CUE_HOST_ID)?.remove();+ window.addEventListener('pageshow', (event) => {+ if (event.persisted) layer?.quench();+ });+}++function painted(): Promise<void> {+ return new Promise((resolve) => {+ const timer = setTimeout(resolve, PAINT_WAIT_MS);+ requestAnimationFrame(() =>+ requestAnimationFrame(() => {+ clearTimeout(timer);+ resolve();+ }),+ );+ });+}++function mountLayer(): Layer | null {+ if (!document.documentElement) return null;+ const host = document.createElement('div');+ host.id = CUE_HOST_ID;+ host.setAttribute(OVERLAY_ATTRIBUTE, '');+ host.setAttribute('aria-hidden', 'true');+ const raisable = typeof host.showPopover === 'function';+ if (raisable) host.setAttribute('popover', 'manual');+ host.style.cssText = HOST_STYLE;+ const root = host.attachShadow({ mode: 'closed' });+ root.innerHTML = `<style>${STYLES}</style><div class="layer"></div>`;+ const surface = root.querySelector<HTMLElement>('.layer')!;+ document.documentElement.append(host);++ const cues = new Map<string, Cue>();+ let frame = 0;++ function show(id: string, plan: CuePlan, theme: ThemeId): void {+ paintTheme(theme);+ for (const other of cues.values()) fade(other);++ const node = part('cue');+ const edge = part('edge hidden');+ const chip = part('chip hidden');+ const marks = plan.anchors.map((anchor) => ({+ anchor,+ element: null,+ ring: part('point' in anchor ? 'ring point hidden' : 'ring hidden'),+ }));+ node.append(edge, ...marks.map((mark) => mark.ring), chip);+ surface.append(node);++ const cue: Cue = { id, plan, node, edge, chip, marks, live: true, resolvedAt: 0, timers: [] };+ cues.set(id, cue);+ cue.timers.push(setTimeout(() => fade(cue), LIVE_LIMIT_MS));+ resolve(cue);+ raise();+ draw(cue);+ track();+ }++ function settle(id: string, ok: boolean): void {+ const cue = cues.get(id);+ if (!cue?.live) return;+ cue.live = false;+ cue.node.classList.add(ok ? 'done' : 'failed');+ cue.timers.push(setTimeout(() => fade(cue), CUE_LINGER_MS));+ }++ function fade(cue: Cue): void {+ if (cue.node.classList.contains('out')) return;+ cue.live = false;+ cue.node.classList.add('out');+ cue.timers.push(setTimeout(() => drop(cue), CUE_FADE_MS));+ }++ function drop(cue: Cue): void {+ for (const timer of cue.timers) clearTimeout(timer);+ cue.node.remove();+ cues.delete(cue.id);+ if (!cues.size) lower();+ }++ function quench(): void {+ for (const cue of [...cues.values()]) drop(cue);+ if (frame) cancelAnimationFrame(frame);+ frame = 0;+ }++ function paintTheme(theme: ThemeId): void {+ const palette = RAIL_PALETTES[theme];+ surface.style.setProperty('--brand', palette.brand);+ surface.style.setProperty('--ground2', palette.ground2);+ surface.style.setProperty('--ink', palette.ink);+ surface.style.setProperty('--warn', RAIL_TONES[theme].warn);+ surface.style.setProperty('--glow', CUE_GLOW[theme]);+ }++ function resolve(cue: Cue): void {+ cue.resolvedAt = performance.now();+ for (const mark of cue.marks) {+ if (!mark.element?.isConnected) mark.element = locate(mark.anchor);+ }+ caption(cue);+ }++ function caption(cue: Cue): void {+ const named = cue.marks.find((mark) => mark.element)?.element;+ const verb = document.createElement('i');+ verb.textContent = cue.plan.verb;+ const rest = [cue.plan.detail ? ` ${cue.plan.detail}` : '', named ? ` · ${cueLabel(named)}` : ''].join('');+ cue.chip.replaceChildren(verb, rest);+ }++ function draw(cue: Cue): void {+ let anchored: Box | null = null;+ for (const mark of cue.marks) {+ const box = boxOf(mark);+ mark.ring.classList.toggle('hidden', !box);+ if (!box) continue;+ place(mark.ring, box);+ anchored ??= box;+ }+ cue.edge.classList.toggle('hidden', anchored !== null);+ placeChip(cue.chip, anchored);+ }++ function track(): void {+ if (frame) return;+ const tick = () => {+ frame = 0;+ if (!cues.size) return;+ const now = performance.now();+ for (const cue of cues.values()) {+ if (cue.live && now - cue.resolvedAt > RESOLVE_EVERY_MS && cue.marks.some(unplaced)) resolve(cue);+ draw(cue);+ }+ frame = requestAnimationFrame(tick);+ };+ frame = requestAnimationFrame(tick);+ }++ function raise(): void {+ if (!raisable) return;+ try {+ if (host.matches(':popover-open')) {+ if (!underTopLayer(host)) return;+ host.hidePopover();+ }+ host.showPopover();+ } catch {}+ }++ function lower(): void {+ if (!raisable) return;+ try {+ if (host.matches(':popover-open')) host.hidePopover();+ } catch {}+ }++ return { alive: () => host.isConnected, show, settle, quench };+}++function part(className: string): HTMLElement {+ const node = document.createElement('div');+ node.className = className;+ return node;+}++function underTopLayer(host: Element): boolean {+ if (document.fullscreenElement) return true;+ try {+ if (document.querySelector(':modal')) return true;+ return [...document.querySelectorAll(':popover-open')].some((open) => open !== host);+ } catch {+ return false;+ }+}++const unplaced = (mark: Mark): boolean => !('point' in mark.anchor) && !mark.element?.isConnected;++function locate(anchor: CueAnchor): Element | null {+ if ('point' in anchor) return null;+ if ('focused' in anchor) {+ const active = document.activeElement;+ return active && active !== document.body && active !== document.documentElement ? active : null;+ }+ try {+ return resolveTarget(anchor.target, { includeHidden: anchor.includeHidden });+ } catch {+ return null;+ }+}++function boxOf(mark: Mark): Box | null {+ if ('point' in mark.anchor) {+ const { x, y } = mark.anchor.point;+ return { x: x - POINT_SIZE / 2, y: y - POINT_SIZE / 2, w: POINT_SIZE, h: POINT_SIZE };+ }+ if (!mark.element?.isConnected) return null;+ const rect = mark.element.getBoundingClientRect();+ if (rect.width === 0 && rect.height === 0) return null;+ return {+ x: rect.left - RING_PAD,+ y: rect.top - RING_PAD,+ w: rect.width + RING_PAD * 2,+ h: rect.height + RING_PAD * 2,+ };+}++function place(node: HTMLElement, box: Box): void {+ node.style.transform = `translate(${Math.round(box.x)}px, ${Math.round(box.y)}px)`;+ node.style.width = `${Math.round(box.w)}px`;+ node.style.height = `${Math.round(box.h)}px`;+}++function placeChip(chip: HTMLElement, box: Box | null): void {+ chip.classList.remove('hidden');+ const view = { w: document.documentElement.clientWidth || innerWidth, h: document.documentElement.clientHeight || innerHeight };+ const width = chip.offsetWidth;+ const clampX = (x: number) => Math.min(Math.max(x, VIEW_MARGIN), Math.max(VIEW_MARGIN, view.w - width - VIEW_MARGIN));+ const clampY = (y: number) => Math.min(Math.max(y, VIEW_MARGIN), Math.max(VIEW_MARGIN, view.h - CHIP_HEIGHT - VIEW_MARGIN));++ if (!box) {+ chip.style.transform = `translate(${Math.round(clampX((view.w - width) / 2))}px, ${EDGE_CHIP_TOP}px)`;+ return;+ }+ const above = box.y - CHIP_HEIGHT - CHIP_GAP;+ const below = box.y + box.h + CHIP_GAP;+ const y = above >= VIEW_MARGIN ? above : below <= view.h - CHIP_HEIGHT - VIEW_MARGIN ? below : box.y + CHIP_GAP;+ chip.style.transform = `translate(${Math.round(clampX(box.x))}px, ${Math.round(clampY(y))}px)`;+}diff --git a/src/lib/cues/label.ts b/src/lib/cues/label.tsnew file mode 100644index 0000000..3ec6d8e--- /dev/null+++ b/src/lib/cues/label.ts@@ -0,0 +1,34 @@+import { accessibleText, computedRole } from '@/lib/actions/page/dom';++export const LABEL_MAX = 40;++/** What a sighted user would call the element. Something the agent can type into is named, never read aloud. */+export function cueLabel(el: Element): string {+ const role = computedRole(el);+ const typedInto = !isFormField(el) && (isEditable(el) || role === 'textbox' || role === 'searchbox');+ const name = typedInto ? quietName(el) : accessibleText(el);+ return clip(name || role || el.tagName.toLowerCase());+}++function isFormField(el: Element): boolean {+ return el instanceof HTMLInputElement || el instanceof HTMLTextAreaElement || el instanceof HTMLSelectElement;+}++function isEditable(el: Element): boolean {+ return el instanceof HTMLElement && (el.isContentEditable || el.getAttribute('contenteditable') === 'true');+}++function quietName(el: Element): string {+ return (+ el.getAttribute('aria-label')?.trim() ||+ el.getAttribute('aria-placeholder')?.trim() ||+ el.getAttribute('placeholder')?.trim() ||+ el.getAttribute('title')?.trim() ||+ ''+ );+}++export function clip(text: string): string {+ const flat = text.replace(/\s+/g, ' ').trim();+ return flat.length > LABEL_MAX ? `${flat.slice(0, LABEL_MAX - 1).trimEnd()}…` : flat;+}diff --git a/src/lib/cues/plan.test.ts b/src/lib/cues/plan.test.tsnew file mode 100644index 0000000..aa53c99--- /dev/null+++ b/src/lib/cues/plan.test.ts@@ -0,0 +1,67 @@+import { describe, expect, it } from 'vitest';+import { actions } from '@/lib/actions/registry';+import { cueFor, hasCueRule } from './plan';++describe('cueFor', () => {+ it('has a rule for every action, so a new one has to choose its cue', () => {+ const missing = [...actions.keys()].filter((name) => !hasCueRule(name));+ expect(missing).toEqual([]);+ });++ it('copies where an action lands and nothing it types', () => {+ const secret = 'hunter2';+ const plans = [+ cueFor('page.fillInput', { target: { selector: '#pw' }, value: secret }),+ cueFor('page.typeText', { target: { selector: '#pw' }, text: secret }),+ cueFor('page.typeText', { text: secret }),+ cueFor('page.searchSite', { query: secret }),+ cueFor('page.pressKey', { key: 'h' }),+ cueFor('page.selectOption', { target: { selector: 'select' }, value: secret, label: secret }),+ cueFor('page.attachFile', { target: { selector: 'input' }, content: secret, name: secret }),+ ];+ for (const plan of plans) expect(JSON.stringify(plan)).not.toContain(secret);+ expect(JSON.stringify(cueFor('page.pressKey', { key: 'h' }))).not.toContain('"h"');+ });++ it('rings the element an action names, with nothing beyond the target fields', () => {+ expect(cueFor('page.clickElement', { target: { selector: '#go', text: 'Go', extra: 'x' } })).toEqual({+ kind: 'element',+ verb: 'Click',+ anchors: [{ target: { selector: '#go', text: 'Go', role: undefined, nth: 0 } }],+ });+ });++ it('falls back to the focused element for typing, and to the page for the rest', () => {+ expect(cueFor('page.typeText', { text: 'hi' })).toMatchObject({ kind: 'element', anchors: [{ focused: true }] });+ expect(cueFor('page.pressKey', { key: 'Enter' })).toMatchObject({ kind: 'element', detail: 'Enter' });+ expect(cueFor('page.scrollTo', { direction: 'down' })).toMatchObject({ kind: 'page', verb: 'Scroll' });+ expect(cueFor('page.extractText', {})).toMatchObject({ kind: 'page', verb: 'Read' });+ });++ it('marks both ends of a drag, by element or by point', () => {+ expect(cueFor('page.dragElement', { from: { selector: '#card' }, toPoint: { x: 10, y: 20 } }).anchors).toEqual([+ { target: { selector: '#card', text: undefined, role: undefined, nth: 0 } },+ { point: { x: 10, y: 20 } },+ ]);+ });++ it('names only the host a navigation goes to', () => {+ expect(cueFor('page.navigate', { url: 'https://example.com/account?token=abc' })).toEqual({+ kind: 'page',+ verb: 'Go to',+ detail: 'example.com',+ anchors: [],+ });+ expect(cueFor('page.navigate', { action: 'back' })).toMatchObject({ verb: 'Go back' });+ });++ it('clears the page before anything that reads its pixels', () => {+ for (const name of ['page.screenshot', 'page.solveCaptcha', 'page.findCaptcha', 'page.pickElement']) {+ expect(cueFor(name, {})).toMatchObject({ kind: 'none', quench: true });+ }+ });++ it('draws nothing for an action it does not know', () => {+ expect(cueFor('page.somethingNew', { target: { selector: 'a' } })).toMatchObject({ kind: 'none' });+ });+});diff --git a/src/lib/cues/plan.ts b/src/lib/cues/plan.tsnew file mode 100644index 0000000..a474f3e--- /dev/null+++ b/src/lib/cues/plan.ts@@ -0,0 +1,213 @@+import { applyTheme } from '@/lib/actions/page/apply-theme';+import { attachFile } from '@/lib/actions/page/attach-file';+import { auditContrast } from '@/lib/actions/page/audit-contrast';+import { awaitMonitor } from '@/lib/actions/page/await-monitor';+import { callSiteTool } from '@/lib/actions/page/call-site-tool';+import { captureDownload } from '@/lib/actions/page/capture-download';+import { clickElement } from '@/lib/actions/page/click-element';+import { closeTab } from '@/lib/actions/page/close-tab';+import { targetSchema } from '@/lib/actions/page/dom';+import { dragElement } from '@/lib/actions/page/drag-element';+import { extractText } from '@/lib/actions/page/extract-text';+import { fillInput } from '@/lib/actions/page/fill-input';+import { findCaptcha } from '@/lib/actions/page/find-captcha';+import { findProgress } from '@/lib/actions/page/find-progress';+import { findSearch } from '@/lib/actions/page/find-search';+import { focusInput } from '@/lib/actions/page/focus-input';+import { getPageInfo } from '@/lib/actions/page/get-page-info';+import { highlightElement } from '@/lib/actions/page/highlight-element';+import { hoverElement } from '@/lib/actions/page/hover-element';+import { injectCode } from '@/lib/actions/page/inject-code';+import { listDownloads } from '@/lib/actions/page/list-downloads';+import { listFiles } from '@/lib/actions/page/list-files';+import { listRecordings } from '@/lib/actions/page/list-recordings';+import { listSiteTools } from '@/lib/actions/page/list-site-tools';+import { monitorStatus } from '@/lib/actions/page/monitor-status';+import { navigate } from '@/lib/actions/page/navigate';+import { openTab } from '@/lib/actions/page/open-tab';+import { pickElement } from '@/lib/actions/page/pick-element';+import { pointSchema } from '@/lib/actions/page/pointer';+import { pressKey } from '@/lib/actions/page/press-key';+import { readConsole } from '@/lib/actions/page/read-console';+import { readNetwork } from '@/lib/actions/page/read-network';+import { readRecording } from '@/lib/actions/page/read-recording';+import { readTheme } from '@/lib/actions/page/read-theme';+import { runCode } from '@/lib/actions/page/run-code';+import { screenshot } from '@/lib/actions/page/screenshot';+import { scrollTo } from '@/lib/actions/page/scroll-to';+import { searchSite } from '@/lib/actions/page/search-site';+import { selectOption } from '@/lib/actions/page/select-option';+import { selectText } from '@/lib/actions/page/select-text';+import { solveCaptcha } from '@/lib/actions/page/solve-captcha';+import { startDiagnostics } from '@/lib/actions/page/start-diagnostics';+import { startMonitor } from '@/lib/actions/page/start-monitor';+import { startTimer } from '@/lib/actions/page/start-timer';+import { stopDiagnostics } from '@/lib/actions/page/stop-diagnostics';+import { stopMonitor } from '@/lib/actions/page/stop-monitor';+import { stopTimer } from '@/lib/actions/page/stop-timer';+import { submitForm } from '@/lib/actions/page/submit-form';+import { switchFrame } from '@/lib/actions/page/switch-frame';+import { switchTab } from '@/lib/actions/page/switch-tab';+import { timerStatus } from '@/lib/actions/page/timer-status';+import { trustedClick } from '@/lib/actions/page/trusted-click';+import { typeText } from '@/lib/actions/page/type-text';+import { waitForElement } from '@/lib/actions/page/wait-for-element';+import type { CueAnchor, CuePlan, CuePoint, CueTarget } from './events';++type Input = Record<string, unknown>;+type Rule = (input: Input) => CuePlan;++interface Slot {+ target?: string;+ point?: string;+}++interface ElementOptions {+ slots?: Slot[];+ focusedByDefault?: boolean;+ includeHidden?: boolean;+ detail?: (input: Input) => string | undefined;+}++export const NO_CUE: CuePlan = { kind: 'none', verb: '', anchors: [] };++const QUENCH: CuePlan = { ...NO_CUE, quench: true };++const none: Rule = () => NO_CUE;++const quench: Rule = () => QUENCH;++const page =+ (verb: string, detail?: (input: Input) => string | undefined): Rule =>+ (input) =>+ withDetail({ kind: 'page', verb, anchors: [] }, detail?.(input));++const element =+ (verb: string, options: ElementOptions = {}): Rule =>+ (input) => {+ const slots = options.slots ?? [{ target: 'target' }];+ const anchors = slots.flatMap((slot) => anchorFor(input, slot, options.includeHidden));+ const detail = options.detail?.(input);+ if (anchors.length) return withDetail({ kind: 'element', verb, anchors }, detail);+ if (options.focusedByDefault) return withDetail({ kind: 'element', verb, anchors: [{ focused: true }] }, detail);+ return withDetail({ kind: 'page', verb, anchors: [] }, detail);+ };++function withDetail(plan: CuePlan, detail: string | undefined): CuePlan {+ return detail ? { ...plan, detail } : plan;+}++function anchorFor(input: Input, slot: Slot, includeHidden?: boolean): CueAnchor[] {+ const target = slot.target ? targetOf(input[slot.target]) : null;+ if (target) return [includeHidden ? { target, includeHidden } : { target }];+ const point = slot.point ? pointOf(input[slot.point]) : null;+ return point ? [{ point }] : [];+}++function targetOf(value: unknown): CueTarget | null {+ const parsed = targetSchema.safeParse(value);+ if (!parsed.success) return null;+ const { selector, text, role, nth } = parsed.data;+ if (!selector && !text) return null;+ return { selector, text, role, nth };+}++function pointOf(value: unknown): CuePoint | null {+ const parsed = pointSchema.safeParse(value);+ return parsed.success ? { x: parsed.data.x, y: parsed.data.y } : null;+}++const namedKey = (input: Input) =>+ typeof input.key === 'string' && input.key.length > 1 ? input.key : undefined;++const HISTORY_VERBS: Record<string, string> = { back: 'Go back', forward: 'Go forward', reload: 'Reload' };++const navigation: Rule = (input) => {+ const history = typeof input.action === 'string' ? HISTORY_VERBS[input.action] : undefined;+ if (history) return { kind: 'page', verb: history, anchors: [] };+ return withDetail({ kind: 'page', verb: 'Go to', anchors: [] }, hostOf(input.url));+};++function hostOf(url: unknown): string | undefined {+ if (typeof url !== 'string') return undefined;+ try {+ return new URL(url).hostname || undefined;+ } catch {+ return undefined;+ }+}++const RULES = new Map<string, Rule>([+ [clickElement.name, element('Click')],+ [trustedClick.name, element('Click', { slots: [{ target: 'target', point: 'point' }] })],+ [hoverElement.name, element('Hover')],+ [+ dragElement.name,+ element('Drag', {+ slots: [+ { target: 'from', point: 'fromPoint' },+ { target: 'to', point: 'toPoint' },+ ],+ }),+ ],+ [focusInput.name, element('Focus')],+ [fillInput.name, element('Fill')],+ [typeText.name, element('Type', { focusedByDefault: true })],+ [pressKey.name, element('Press', { focusedByDefault: true, detail: namedKey })],+ [selectOption.name, element('Choose')],+ [selectText.name, element('Select')],+ [submitForm.name, element('Submit')],+ [waitForElement.name, element('Wait for')],+ [scrollTo.name, element('Scroll')],+ [extractText.name, element('Read')],+ [attachFile.name, element('Attach', { includeHidden: true })],+ [captureDownload.name, element('Download')],+ [searchSite.name, element('Search')],++ [getPageInfo.name, page('Read page')],+ [navigate.name, navigation],+ [runCode.name, page('Run script')],+ [injectCode.name, page('Add script')],+ [listSiteTools.name, page('Read site tools')],+ [callSiteTool.name, page('Use site tool')],+ [findProgress.name, page('Read page')],+ [findSearch.name, page('Read page')],+ [readTheme.name, page('Inspect styles')],+ [auditContrast.name, page('Inspect styles')],+ [applyTheme.name, page('Restyle page')],++ [screenshot.name, quench],+ [findCaptcha.name, quench],+ [solveCaptcha.name, quench],+ [pickElement.name, quench],++ [highlightElement.name, none],+ [openTab.name, none],+ [switchTab.name, none],+ [closeTab.name, none],+ [switchFrame.name, none],+ [startDiagnostics.name, none],+ [readConsole.name, none],+ [readNetwork.name, none],+ [stopDiagnostics.name, none],+ [startMonitor.name, none],+ [monitorStatus.name, none],+ [awaitMonitor.name, none],+ [stopMonitor.name, none],+ [startTimer.name, none],+ [timerStatus.name, none],+ [stopTimer.name, none],+ [listFiles.name, none],+ [listRecordings.name, none],+ [readRecording.name, none],+ [listDownloads.name, none],+]);++export const hasCueRule = (action: string): boolean => RULES.has(action);++/** Built from the sealed input: only where the action lands is copied, never what it types. */+export function cueFor(action: string, input: unknown): CuePlan {+ const rule = RULES.get(action);+ if (!rule) return NO_CUE;+ return rule(typeof input === 'object' && input !== null ? (input as Input) : {});+}diff --git a/src/lib/cues/styles.ts b/src/lib/cues/styles.tsnew file mode 100644index 0000000..cfe5963--- /dev/null+++ b/src/lib/cues/styles.ts@@ -0,0 +1,101 @@+import type { ThemeId } from '@/lib/settings/theme';+import { CUE_FADE_MS } from './events';++/** --glow in globals.css, theme by theme; the page has no stylesheet to read it from. */+export const CUE_GLOW: Record<ThemeId, string> = {+ ember: '72%',+ midnight: '72%',+ phosphor: '85%',+ daylight: '26%',+};++export const HOST_STYLE = [+ 'all: initial',+ 'position: fixed',+ 'inset: 0',+ 'width: 100vw',+ 'height: 100vh',+ 'max-width: none',+ 'max-height: none',+ 'margin: 0',+ 'padding: 0',+ 'border: 0',+ 'background: transparent',+ 'overflow: visible',+ 'pointer-events: none',+ 'display: block',+ 'z-index: 2147483647',+].join('; ');++export const STYLES = `+ :host { all: initial; }+ * { box-sizing: border-box; margin: 0; padding: 0; pointer-events: none; }+ .layer {+ position: fixed;+ inset: 0;+ overflow: hidden;+ font-family: ui-sans-serif, system-ui, -apple-system, "Segoe UI", sans-serif;+ }+ .cue { --tone: var(--brand); position: absolute; inset: 0; transition: opacity ${CUE_FADE_MS}ms ease-out; }+ .cue.failed { --tone: var(--warn); }+ .cue.out { opacity: 0; }+ .ring {+ position: absolute;+ left: 0;+ top: 0;+ border: 2px solid var(--tone);+ border-radius: 8px;+ box-shadow:+ 0 0 0 1px color-mix(in oklch, var(--tone) 22%, transparent),+ 0 0 18px 1px color-mix(in oklch, var(--tone) var(--glow), transparent),+ inset 0 0 12px -4px color-mix(in oklch, var(--tone) var(--glow), transparent);+ will-change: transform;+ animation: browsentic-cue-in 160ms ease-out, browsentic-cue-pulse 1.4s ease-in-out 160ms infinite;+ }+ .ring.point { border-radius: 999px; }+ .ring.hidden, .chip.hidden, .edge.hidden { display: none; }+ .cue.done .ring, .cue.failed .ring { animation: none; }+ .edge {+ position: absolute;+ inset: 0;+ box-shadow:+ inset 0 0 0 2px color-mix(in oklch, var(--tone) 70%, transparent),+ inset 0 0 38px -6px color-mix(in oklch, var(--tone) var(--glow), transparent);+ animation: browsentic-cue-in 160ms ease-out, browsentic-cue-breathe 1.6s ease-in-out 160ms infinite;+ }+ .cue.done .edge, .cue.failed .edge { animation: none; }+ .chip {+ position: absolute;+ left: 0;+ top: 0;+ max-width: calc(100vw - 16px);+ padding: 3px 9px 4px;+ border: 1px solid color-mix(in oklch, var(--tone) 45%, transparent);+ border-radius: 999px;+ background: var(--ground2);+ color: var(--ink);+ font: 600 11px/1.3 inherit;+ letter-spacing: 0.01em;+ white-space: nowrap;+ overflow: hidden;+ text-overflow: ellipsis;+ box-shadow: 0 6px 26px -10px color-mix(in oklch, var(--tone) var(--glow), transparent);+ will-change: transform;+ }+ .chip i { font-style: normal; color: var(--tone); }+ @keyframes browsentic-cue-in { from { opacity: 0; } }+ @keyframes browsentic-cue-pulse {+ 50% {+ box-shadow:+ 0 0 0 3px color-mix(in oklch, var(--tone) 18%, transparent),+ 0 0 26px 3px color-mix(in oklch, var(--tone) var(--glow), transparent),+ inset 0 0 12px -4px color-mix(in oklch, var(--tone) var(--glow), transparent);+ }+ }+ @keyframes browsentic-cue-breathe { 50% { opacity: 0.65; } }+ @media (prefers-reduced-motion: reduce) {+ .cue { transition: none; }+ .ring, .edge { animation: none; }+ }+ @media print { .layer { display: none; } }+`;diff --git a/src/lib/handsfree/events.ts b/src/lib/handsfree/events.tsindex 9c806f8..7904139 100644--- a/src/lib/handsfree/events.ts+++ b/src/lib/handsfree/events.ts@@ -66,6 +66,8 @@ export interface OrbView {voice: OrbVoice;/** The mic stays off until the hold key is held, and letting go sends. */pushToTalk: boolean;+ /** A ring marks each element the agent acts on. */+ cues: boolean;approval?: OrbApproval;}@@ -98,6 +100,7 @@ export type OrbRequest =| { channel: typeof HANDS_FREE_CHANNEL; op: 'detach'; fileId: string }| { channel: typeof HANDS_FREE_CHANNEL; op: 'move'; position: OrbPosition }| { channel: typeof HANDS_FREE_CHANNEL; op: 'pushToTalk'; on: boolean }+ | { channel: typeof HANDS_FREE_CHANNEL; op: 'cues'; on: boolean }| { channel: typeof HANDS_FREE_CHANNEL; op: 'talk'; on: boolean };export type DictationReport =@@ -110,7 +113,7 @@ export type DictationCommand = { channel: typeof DICTATION_CHANNEL; op: 'talk';const ORB_COMMANDS = new Set(['show', 'hide', 'heard', 'say']);const ORB_REQUESTS = new Set(['sync', 'submit', 'cancel', 'decide', 'openPanel', 'listen', 'grantMic', 'pick', 'attach', 'detach', 'move',- 'pushToTalk', 'talk',+ 'pushToTalk', 'cues', 'talk',]);const DICTATION_REPORTS = new Set(['phase', 'heard']);const DICTATION_COMMANDS = new Set(['talk']);@@ -157,6 +160,18 @@ export const ORB_ICONS = {'m15 15-3-3-3 3',],code: ['m18 16 4-4-4-4', 'm6 8-4 4 4 4', 'm14.5 4-5 16'],+ cues: [+ 'M12.034 12.681a.498.498 0 0 1 .647-.647l9 3.5a.5.5 0 0 1-.033.943l-3.444 1.068a1 1 0 0 0-.66.66l-1.067 3.443a.5.5 0 0 1-.943.033z',+ 'M5 3a2 2 0 0 0-2 2',+ 'M19 3a2 2 0 0 1 2 2',+ 'M5 21a2 2 0 0 1-2-2',+ 'M9 3h1',+ 'M9 21h2',+ 'M14 3h1',+ 'M3 9v1',+ 'M21 9v2',+ 'M3 14v1',+ ],focus: ['M3 7V5a2 2 0 0 1 2-2h2','M17 3h2a2 2 0 0 1 2 2v2',diff --git a/src/lib/handsfree/geometry.test.ts b/src/lib/handsfree/geometry.test.tsindex 037d091..7d84213 100644--- a/src/lib/handsfree/geometry.test.ts+++ b/src/lib/handsfree/geometry.test.ts@@ -12,7 +12,7 @@ import {} from './geometry';const viewport = { width: 1440, height: 900 };-const ITEMS = 5;+const ITEMS = 6;const half = ORB_SIZE / 2 + ORB_INSET;describe('placeOrb', () => {diff --git a/src/lib/handsfree/geometry.ts b/src/lib/handsfree/geometry.tsindex feb3b95..71e862a 100644--- a/src/lib/handsfree/geometry.ts+++ b/src/lib/handsfree/geometry.ts@@ -15,10 +15,10 @@ export type Side = 'top' | 'bottom' | 'left' | 'right';export const ORB_SIZE = 56;export const ORB_INSET = 24;export const MENU_ITEM = 38;-export const MENU_RADIUS = 80;+export const MENU_RADIUS = 88;/* A corner leaves a quarter turn, plus the slack of the orb's inset on either side: 110° at- 96 px keeps five items apart and every one of them on screen. */-const CORNER_RADIUS = 96;+ 118 px keeps six items apart and every one of them on screen. */+const CORNER_RADIUS = 118;const EDGE_SPAN = 150;const CORNER_SPAN = 110;const VIEW_MARGIN = 12;diff --git a/src/lib/handsfree/host.test.ts b/src/lib/handsfree/host.test.tsindex b5c88ca..6f63d7c 100644--- a/src/lib/handsfree/host.test.ts+++ b/src/lib/handsfree/host.test.ts@@ -16,6 +16,7 @@ const view = (patch: Partial<OrbView> = {}): OrbView => ({run: 'idle',voice: 'listening',pushToTalk: false,+ cues: true,...patch,});@@ -222,6 +223,17 @@ describe('the hands-free orb', () => {expect(slots).toEqual([...slots].sort((a, b) => a - b));});+ it('switches action cues from the menu and shows which way they are set', async () => {+ await show();+ const item = $('[data-item="cues"]');+ expect(item.classList.contains('on')).toBe(true);+ item.dispatchEvent(new MouseEvent('click', { bubbles: true }));+ expect(requests('cues')).toEqual([expect.objectContaining({ on: false })]);+ await show({ cues: false });+ expect(item.classList.contains('on')).toBe(false);+ expect(item.querySelector('.tip')?.textContent).toBe('Show what the agent does');+ });+it('rides live code and an A-Eye pick along with the next instruction, then lets them go', async () => {await show();$('[data-item="code"]').dispatchEvent(new MouseEvent('click', { bubbles: true }));diff --git a/src/lib/handsfree/host.ts b/src/lib/handsfree/host.tsindex 746ff82..b477c89 100644--- a/src/lib/handsfree/host.ts+++ b/src/lib/handsfree/host.ts@@ -48,7 +48,7 @@ const SETTLE_MS = 2500;const SPEECH_STAGGER_MS = 45;const REPLY_STAGGER_MS = 32;-type MenuId = 'file' | 'code' | 'focus' | 'talk' | 'panel';+type MenuId = 'file' | 'code' | 'focus' | 'cues' | 'talk' | 'panel';type CaptionKind = 'speech' | 'sent' | 'reply' | 'error' | 'ask' | 'hint';type Unsent<T> = T extends unknown ? Omit<T, 'channel'> : never;@@ -64,6 +64,7 @@ const MENU: { id: MenuId; icon: OrbIcon; label: string }[] = [{ id: 'file', icon: 'file', label: 'Attach a file' },{ id: 'code', icon: 'code', label: 'Live code' },{ id: 'focus', icon: 'focus', label: 'Focus point (A-Eye)' },+ { id: 'cues', icon: 'cues', label: 'Show what the agent does' },{ id: 'talk', icon: 'keyboard', label: `Hold ${HOLD_KEY.name} to talk` },{ id: 'panel', icon: 'panel', label: 'Open the side panel' },];@@ -392,6 +393,10 @@ function mountOrb(first: OrbView): Orb | null {talkItem?.classList.toggle('on', view.pushToTalk);const talkTip = talkItem?.querySelector('.tip');if (talkTip) talkTip.textContent = view.pushToTalk ? `Hold to talk is on — ${HOLD_KEY.name}` : `Hold ${HOLD_KEY.name} to talk`;+ const cuesItem = items.get('cues');+ cuesItem?.classList.toggle('on', view.cues);+ const cuesTip = cuesItem?.querySelector('.tip');+ if (cuesTip) cuesTip.textContent = view.cues ? 'Showing what the agent does' : 'Show what the agent does';button.setAttribute('aria-label', pressLabel());if (!before) return;@@ -734,6 +739,10 @@ function mountOrb(first: OrbView): Orb | null {} else if (id === 'focus') {closeMenu();void pickFocus();+ } else if (id === 'cues') {+ const on = !view.cues;+ void request({ op: 'cues', on });+ hint(on ? 'A ring now marks each element the agent acts on.' : 'The agent’s actions are no longer marked on the page.');} else if (id === 'talk') {const on = !view.pushToTalk;void request({ op: 'pushToTalk', on });diff --git a/vitest.config.ts b/vitest.config.tsindex 7508bbf..0b9f36a 100644--- a/vitest.config.ts+++ b/vitest.config.ts@@ -23,7 +23,12 @@ export default defineConfig({test: {name: 'lib',include: ['src/lib/**/*.test.ts'],- exclude: [...configDefaults.exclude, 'src/lib/actions/page/**', 'src/lib/handsfree/host.test.ts'],+ exclude: [+ ...configDefaults.exclude,+ 'src/lib/actions/page/**',+ 'src/lib/handsfree/host.test.ts',+ 'src/lib/cues/host.test.ts',+ ],},},{@@ -31,7 +36,7 @@ export default defineConfig({test: {name: 'dom',environment: 'happy-dom',- include: ['src/lib/actions/page/**/*.test.ts', 'src/lib/handsfree/host.test.ts'],+ include: ['src/lib/actions/page/**/*.test.ts', 'src/lib/handsfree/host.test.ts', 'src/lib/cues/host.test.ts'],},},{