Skip to content

An agentic browser harness: an AI side panel & orb in your own logged-in Chrome, Edge, Arc, Brave or even Firefox run by the agent/subscription you already use (Claude Code, Codex, Antigravity). Free, open source, no API key. Also an MCP server.

feat(extension): action cues — a glowing ring on every element an agent acts on, captioned with the action and the element’s name, and the page’s edge for an action with no element; off until switched on from the composer, the mic’s menu or Settings, drawn in a closed layer that never touches the page, and cleared before any screenshot

Shahriar4be846f6b9bc
diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md
index a54bf74..11ce99f 100644
--- a/docs/guide/configuration.md
+++ b/docs/guide/configuration.md
@@ -138,6 +138,7 @@ Below the theme, the same section holds what belongs to one browser alone. None
| Block | What it sets |
| --- | --- |
+| **On the page** | **Show what the agent does** — the [action cues](features/action-cues.md) ring on each element an agent acts on. Off until you switch it on; the composer's button and the mic's menu switch the same setting |
| **Right-click menu** | Whether a right-click on a page offers **Open Browsentic** (the side panel) and **Open Browsentic (Hands Free)**. Both are on until you switch one off. With both on, the browser gathers them under a **Browsentic** entry |
| **Keyboard shortcuts** | Shows the keys for **Open the popup**, **Open or close the side panel** and **Start or stop hands-free**. The browser owns these: it gives the side panel **Alt+Shift+B** and hands-free **Alt+Shift+H** (**⌥⇧B**, **⌥⇧H** on a Mac) at install unless something else already has them, and **Change shortcuts** opens its own shortcuts page. Firefox has the popup and the sidebar only |
| **Hands-free** | **Hold to talk** — the same switch as the mic's own menu |
diff --git a/docs/guide/features/README.md b/docs/guide/features/README.md
index c9cb171..aed0043 100644
--- a/docs/guide/features/README.md
+++ b/docs/guide/features/README.md
@@ -13,6 +13,7 @@ first.
| [Hands-free](hands-free.md) | Put the panel away and talk to the page through a mic you can drag anywhere |
| [Instant commands](instant-commands.md) | "go back", "scroll to the top" — the things that run in milliseconds without waking an agent |
| [A-Eye](a-eye.md) | Point at an element instead of describing it, and let it point back when it needs to |
+| [Action cues](action-cues.md) | A glowing ring on whatever the agent is about to click, type into or read |
| [Profile](profile.md) | Your details and standing rules, saved once, so it fills a form with the real thing instead of guessing |
| [Blocked sites](blocked-sites.md) | Sites it never reads or acts on, whoever is driving — enforced in the browser, out of the agent's reach |
| [Skills](skills.md) | How an instruction gets routed to the right playbook, and how to write your own |
diff --git a/docs/guide/features/action-cues.md b/docs/guide/features/action-cues.md
new file mode 100644
index 0000000..43b5f23
--- /dev/null
+++ b/docs/guide/features/action-cues.md
@@ -0,0 +1,53 @@
+# Action cues
+
+Watching the agent work: a glowing ring on whatever it is about to touch.
+
+---
+
+## What you see
+
+Every time an agent acts in a tab, the element it acts on lights up with a ring in your theme's
+colour, and a small caption names the action and the element — **Click · Sign in**, **Fill ·
+Email**, **Type · Message**. The ring appears a moment before the action lands, follows the element
+if the page scrolls it into view, and fades shortly after the action is done. A failed action
+flashes the warning colour instead.
+
+Actions that have no element — reading the page, going to a new address, running a script — glow
+around the edge of the page instead, with the caption at the top: **Read page**, **Go to
+example.com**.
+
+It works for every agent, whoever is driving: a run from the side panel or the mic, and an MCP
+client such as Claude Code or Cursor working in your browser.
+
+## Switching it on
+
+It is off until you switch it on, in any of three places — they are one setting:
+
+- the **Show what the agent does** button in the side panel's composer row, next to **Schedule**
+- the same item in the [hands-free](hands-free.md) mic's menu
+- **Settings → Extension → On the page**
+
+## What it never does
+
+- **It never touches the page you are on.** The ring is drawn in a closed layer of its own, the
+ page's scripts cannot see into it, and nothing in it takes a click — a click that lands on the
+ ring reaches the page underneath. It does not move focus, scroll, or listen to anything the page
+ does.
+- **It never shows what is typed.** The caption names a field by its label — **Fill · Password** —
+ never by its contents, and the agent's input never reaches the page's layer at all: only where
+ an action lands does.
+- **It is never in a screenshot.** Before the agent takes a screenshot, or the captcha solver
+ photographs a challenge, every ring on the tab is taken off first, so the agent never sees its
+ own cue.
+- **It is never drawn on a [blocked site](blocked-sites.md)** — the action is refused before
+ anything reaches the page.
+
+## Where it stops
+
+- An element inside a frame is ringed inside that frame, so a ring near the frame's edge is cut off
+ by it.
+- `chrome://` pages and the Web Store never show one, since the browser keeps extensions out of
+ them. A tab that loaded before the extension did shows none for its first action, until the
+ extension has been let in.
+- The ring sits above a page's own dialogs and popovers. In a browser without popover support it
+ sits under a modal dialog instead.
diff --git a/docs/guide/features/hands-free.md b/docs/guide/features/hands-free.md
index 2343327..43a0751 100644
--- a/docs/guide/features/hands-free.md
+++ b/docs/guide/features/hands-free.md
@@ -53,13 +53,14 @@ side panel in every window, and opening the panel anywhere ends hands-free.
## The menu
-Rest the pointer on the mic for a second and five buttons fan out around it:
+Rest the pointer on the mic for a second and six buttons fan out around it:
| | |
| --- | --- |
| **Attach a file** | Hands a file to this tab's conversation, as dropping it on the panel does — see [Files](files.md) |
| **Live code** | Lets the agent write a script for this page, as the composer's code toggle does; you approve the code before it runs |
| **Focus point** | Opens [A-Eye](a-eye.md): point at an element and it rides along with your next instruction |
+| **Show what the agent does** | Switches [action cues](action-cues.md) — the ring on each element the agent acts on — on or off, for every tab |
| **Hold to talk** | Switches between listening all the time and listening only while you hold a key — see below |
| **Open the side panel** | Brings the panel back and ends hands-free |
diff --git a/docs/internals/extension.md b/docs/internals/extension.md
index 90108e4..01aa22d 100644
--- a/docs/internals/extension.md
+++ b/docs/internals/extension.md
@@ -376,6 +376,44 @@ State: `browsentic/handsFree` in `storage.session` (on while present, with `mute
`browsentic/pushToTalk` and `browsentic/orbPosition` — the orb's centre as viewport fractions, so one drag places it on every tab. Hands-free is session state
on purpose: a restarted browser comes back with the panel, not a live microphone.
+## Action cues: a ring in whichever frame the agent acts in
+
+Every agent action is ringed on the page — see [Action cues](../guide/features/action-cues.md). The
+background drives it, because `invokeForHarness` is the one place that sees each action together
+with its tab: the content script's dispatch knows no run, and hears internal sub-calls (a
+screenshot is a plan plus a scroll per tile) that are not the agent's actions.
+
+**One table decides what an action looks like.** `cueFor()` in `src/lib/cues/plan.ts` maps each
+registry action to an element cue (a target, a point, or the focused element), a page cue (the
+viewport's edge) or none, and a test fails if a registry action has no entry — a new action has to
+choose. It is built from the **sealed** input, before `releaseForAction`, and copies only the
+target fields, points, a named key and a navigation's host — never a value, a text or a query.
+
+**`cued()` wraps the tab-bound half of `dispatch`** (`src/lib/bridge/action-cues.ts`). It sends
+`show` to the focused frame for an element cue or the top frame for a page cue, waits at most
+`CUE_LEAD_MS` so the ring is painted before the action fires, runs the action, then sends `settle`
+with its outcome. A frame that does not answer — frozen, alerting, or without a content script —
+costs the action that wait and nothing else, and a cue never fails an action. Blocked sites are
+refused by `guardTarget` before `dispatch` runs, so they are never ringed. The switch is
+`browsentic/actionCues` in `storage.local`, off unless it is `true`.
+
+**The page gets one element and nothing more.** `exposeCues()` runs in every frame (before the
+top-frame return in `content.ts`), and the first cue in a document mounts `#browsentic-cues` on
+`documentElement` with its attributes already set — one childList mutation for the page's
+observers, and none after it, because every ring, caption and fade is inside a closed shadow root.
+Nothing in it takes a pointer, and it never focuses, scrolls or listens to the page. While a cue is
+on screen a requestAnimationFrame loop follows the element's box; a target that has not appeared
+yet (`waitForElement`) is looked for again every 250 ms.
+
+**It rides in the top layer.** The host is a `popover="manual"`, shown while a cue is up and hidden
+when none is, and raised again when a modal dialog, another popover or a fullscreen element has
+appeared — so a ring on a cookie banner's button sits above the banner.
+
+**A capture quenches first.** `screenshot`, `findCaptcha`, `solveCaptcha` and `pickElement` carry
+`quench` in the table, so `cued()` clears every frame still lit on that tab — the background keeps
+which ones, for as long as a cue lingers — and the host answers after two frames, once the page has
+repainted without it. A tab with nothing lit is not messaged at all.
+
## Tab scoping
A panel conversation is **bound to the tab it started in**.
diff --git a/docs/internals/state.md b/docs/internals/state.md
index eefdbd2..e9a3718 100644
--- a/docs/internals/state.md
+++ b/docs/internals/state.md
@@ -76,7 +76,8 @@ state under `browsentic/dictation`, and the approvals it has already announced u
Only where the orb was dragged to (`browsentic/orbPosition`), whether hold-to-talk is on
(`browsentic/pushToTalk`) and whether this browser's speech service has ever worked
(`browsentic/speechService`) are kept in `storage.local`, along with which right-click items this
-browser shows (`browsentic/contextMenu`, both on when absent).
+browser shows (`browsentic/contextMenu`, both on when absent) and whether agent actions are ringed on
+the page (`browsentic/actionCues`, off when absent).
**Blocked sites** are `browsentic/blockedSites` in `storage.local` — a list of patterns, up to 500, that
the daemon never receives. Removing the extension removes it; each browser keeps its own.
diff --git a/src/extension/components/composer.tsx b/src/extension/components/composer.tsx
index ffc97bf..c9fe4aa 100644
--- a/src/extension/components/composer.tsx
+++ b/src/extension/components/composer.tsx
@@ -1,5 +1,21 @@
import { useEffect, useMemo, useRef, useState, type MouseEvent } from 'react';
-import { CalendarClock, Code2, FileText, FileUp, Loader2, Mic, MicOff, Paperclip, RotateCw, ScanEye, Send, Sparkles, Square, X } from 'lucide-react';
+import {
+ CalendarClock,
+ Code2,
+ FileText,
+ FileUp,
+ Loader2,
+ Mic,
+ MicOff,
+ Paperclip,
+ RotateCw,
+ ScanEye,
+ Send,
+ Sparkles,
+ Square,
+ SquareDashedMousePointer,
+ X,
+} from 'lucide-react';
import { SkillMenu, skillMenuItems, type SkillMenuItem } from '@/extension/components/skill-menu';
import type { SavedToolMeta } from '@/lib/bridge/saved-tools';
@@ -34,6 +50,8 @@ export function Composer({
onToggleLiveTools,
scheduling,
onToggleScheduling,
+ actionCues,
+ onToggleActionCues,
onAttachSkill,
onCommand,
tools,
@@ -65,6 +83,8 @@ export function Composer({
onToggleLiveTools: () => void;
scheduling: boolean;
onToggleScheduling: () => void;
+ actionCues: boolean;
+ onToggleActionCues: () => void;
onAttachSkill: (skill: AttachedSkill | null) => void;
onCommand: (command: string) => void;
tools: SavedToolMeta[];
@@ -293,6 +313,21 @@ export function Composer({
>
<CalendarClock className="size-3.5" />
</Button>
+ <Button
+ variant={actionCues ? 'subtle' : 'ghost'}
+ size="icon-sm"
+ role="switch"
+ aria-checked={actionCues}
+ aria-label={actionCues ? 'Hide what the agent does' : 'Show what the agent does'}
+ title={
+ actionCues
+ ? 'Showing what the agent does — a glowing ring marks each element it acts on. Turn it off to hide it.'
+ : 'Show what the agent does — mark each element it acts on with a glowing ring.'
+ }
+ onClick={onToggleActionCues}
+ >
+ <SquareDashedMousePointer className="size-3.5" />
+ </Button>
<Button
variant={voiceEnabled && !voice.error ? 'subtle' : 'ghost'}
size="icon-sm"
diff --git a/src/extension/components/extension-settings.tsx b/src/extension/components/extension-settings.tsx
index 0ee5600..fe83514 100644
--- a/src/extension/components/extension-settings.tsx
+++ b/src/extension/components/extension-settings.tsx
@@ -4,7 +4,13 @@ import { ExternalLink } from 'lucide-react';
import { Button } from '@/extension/components/ui/button';
import { Switch } from '@/extension/components/ui/switch';
import { HANDS_FREE_SHORTCUT } from '@/lib/settings/shortcuts';
-import { openShortcutSettings, useContextMenuChoice, usePushToTalk, useShortcuts } from '@/lib/bridge/use-extension-settings';
+import {
+ openShortcutSettings,
+ useActionCues,
+ useContextMenuChoice,
+ usePushToTalk,
+ useShortcuts,
+} from '@/lib/bridge/use-extension-settings';
import { useHandsFreeSupported } from '@/lib/bridge/use-speech';
import { cn } from '@/lib/utils';
@@ -12,6 +18,7 @@ export function ExtensionSettings() {
const handsFree = useHandsFreeSupported();
return (
<>
+ <ActionCues />
<ContextMenuItems handsFree={handsFree} />
<KeyboardShortcuts handsFree={handsFree} />
{handsFree && <HandsFree />}
@@ -19,6 +26,23 @@ export function ExtensionSettings() {
);
}
+function ActionCues() {
+ const [on, setOn] = useActionCues();
+ return (
+ <SettingsGroup
+ title="On the page"
+ note="What the page shows while an agent works in it. The composer and the mic’s menu switch the same setting."
+ >
+ <Row
+ title="Show what the agent does"
+ note="A glowing ring marks each element the agent clicks, types into or reads, with a short caption. It never takes a click, and screenshots leave it out."
+ >
+ <Switch checked={on} label="Show what the agent does" onChange={setOn} />
+ </Row>
+ </SettingsGroup>
+ );
+}
+
function ContextMenuItems({ handsFree }: { handsFree: boolean }) {
const [choice, setChoice] = useContextMenuChoice();
return (
diff --git a/src/extension/entrypoints/content.ts b/src/extension/entrypoints/content.ts
index be5ad08..d4f1a1a 100644
--- a/src/extension/entrypoints/content.ts
+++ b/src/extension/entrypoints/content.ts
@@ -1,4 +1,5 @@
import { exposeActions } from '@/lib/actions/host';
+import { exposeCues } from '@/lib/cues/host';
import { exposeFrameProbe } from '@/lib/frames/host';
import { exposeHandsFree } from '@/lib/handsfree/host';
import { exposeIndicator } from '@/lib/indicator/host';
@@ -15,6 +16,7 @@ export default defineContentScript({
world.__browsenticActions = true;
exposeActions();
exposeFrameProbe();
+ exposeCues();
if (window !== window.top) return;
exposeRecorder();
exposeMonitor();
diff --git a/src/extension/entrypoints/sidepanel/App.tsx b/src/extension/entrypoints/sidepanel/App.tsx
index b42cb52..80f8ff4 100644
--- a/src/extension/entrypoints/sidepanel/App.tsx
+++ b/src/extension/entrypoints/sidepanel/App.tsx
@@ -34,7 +34,7 @@ import { putBytes } from '@/lib/bridge/file-store';
import { removeRecording, type StoredRecordingMeta } from '@/lib/bridge/recording-store';
import { removeSession } from '@/lib/bridge/session-store';
import { useActiveTabUrl } from '@/lib/bridge/use-active-tab-url';
-import { useBlockedPattern } from '@/lib/bridge/use-extension-settings';
+import { useActionCues, useBlockedPattern } from '@/lib/bridge/use-extension-settings';
import { useDaemonState } from '@/lib/bridge/use-daemon-state';
import { useFileDrop } from '@/lib/bridge/use-file-drop';
import { closeSidePanel } from '@/lib/bridge/side-panel';
@@ -61,6 +61,7 @@ export default function App() {
const daemon = useDaemonState();
const run = useRun();
const [voiceEnabled, setVoiceEnabled] = useVoiceEnabled();
+ const [actionCues, setActionCues] = useActionCues();
const handsFree = useHandsFreeSupported();
const storedFiles = useStoredFiles();
const sessions = useStoredSessions();
@@ -489,6 +490,8 @@ export default function App() {
onToggleLiveTools={() => setLiveTools((on) => !on)}
scheduling={scheduling}
onToggleScheduling={() => setScheduling((on) => !on)}
+ actionCues={actionCues}
+ onToggleActionCues={() => setActionCues(!actionCues)}
onAttachSkill={setAttachedSkill}
handsFree={handsFree}
onCommand={(command) => {
diff --git a/src/lib/bridge/action-cues.test.ts b/src/lib/bridge/action-cues.test.ts
new file mode 100644
index 0000000..1ed32d8
--- /dev/null
+++ b/src/lib/bridge/action-cues.test.ts
@@ -0,0 +1,122 @@
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
+import { fakeBrowser } from 'wxt/testing';
+import { ACTION_CHANNEL, success } from '@/lib/actions/protocol';
+import { CUE_CHANNEL } from '@/lib/cues/events';
+import { cueFor } from '@/lib/cues/plan';
+import { BLOCKED_SITES_KEY } from '@/lib/settings/blocked-sites';
+import { ACTION_CUES_KEY, CUE_LEAD_MS, cued, quenchCues } from './action-cues';
+import { forgetFrameFocus, setFramePath } from './frame-focus';
+import { invokeForHarness } from './invoke';
+
+type Sent = { tabId: number; frameId?: number; message: { channel: string; op?: string; action?: string } & Record<string, unknown> };
+
+let sent: Sent[];
+let answer: (message: Sent['message']) => unknown;
+
+const cues = () => sent.filter((entry) => entry.message.channel === CUE_CHANNEL);
+const ops = () => sent.map((entry) => (entry.message.channel === CUE_CHANNEL ? `cue:${entry.message.op}` : `action:${entry.message.action}`));
+const tabOn = async (url: string) => (await fakeBrowser.tabs.create({ url, active: true })).id!;
+
+beforeEach(async () => {
+ fakeBrowser.reset();
+ sent = [];
+ answer = (message) => (message.channel === ACTION_CHANNEL ? success({ done: true }) : { ok: true });
+ vi.spyOn(fakeBrowser.tabs, 'sendMessage').mockImplementation((async (tabId: number, message: Sent['message'], options?: { frameId?: number }) => {
+ sent.push({ tabId, frameId: options?.frameId, message });
+ return answer(message);
+ }) as never);
+ vi.spyOn(fakeBrowser.runtime, 'getManifest').mockReturnValue({ content_scripts: [] } as never);
+ await fakeBrowser.storage.local.set({ [ACTION_CUES_KEY]: true });
+});
+
+afterEach(() => {
+ vi.useRealTimers();
+ vi.restoreAllMocks();
+});
+
+describe('cued', () => {
+ it('draws nothing until the user switches cues on', async () => {
+ await fakeBrowser.storage.local.remove(ACTION_CUES_KEY);
+ const tabId = await tabOn('https://example.com/');
+ await invokeForHarness('page.clickElement', { target: { selector: '#go' } }, tabId);
+ expect(ops()).toEqual(['action:page.clickElement']);
+ });
+
+ it('rings the element before the action runs and settles it after', async () => {
+ const tabId = await tabOn('https://example.com/');
+ const result = await invokeForHarness('page.clickElement', { target: { selector: '#go' } }, tabId);
+
+ expect(result.ok).toBe(true);
+ expect(ops()).toEqual(['cue:show', 'action:page.clickElement', 'cue:settle']);
+ const [show, settle] = cues();
+ expect(show.message).toMatchObject({ plan: { kind: 'element', verb: 'Click' }, theme: 'ember' });
+ expect(settle.message).toMatchObject({ id: show.message.id, ok: true });
+ });
+
+ it('settles a failed action as failed', async () => {
+ const tabId = await tabOn('https://example.com/');
+ answer = (message) =>
+ message.channel === ACTION_CHANNEL ? { ok: false, error: { code: 'TARGET_NOT_FOUND', message: 'none' } } : { ok: true };
+ await invokeForHarness('page.clickElement', { target: { selector: '#go' } }, tabId);
+ expect(cues().at(-1)?.message).toMatchObject({ op: 'settle', ok: false });
+ });
+
+ it('never carries what the agent types', async () => {
+ const tabId = await tabOn('https://example.com/');
+ await invokeForHarness('page.fillInput', { target: { selector: '#pw' }, value: 'hunter2' }, tabId);
+ expect(cues()).toHaveLength(2);
+ expect(JSON.stringify(cues())).not.toContain('hunter2');
+ });
+
+ it('draws nothing on a blocked site', async () => {
+ await fakeBrowser.storage.local.set({ [BLOCKED_SITES_KEY]: ['mybank.com'] });
+ const tabId = await tabOn('https://mybank.com/');
+ const result = await invokeForHarness('page.clickElement', { target: { selector: '#go' } }, tabId);
+ expect(result.ok).toBe(false);
+ expect(sent).toEqual([]);
+ });
+
+ it('draws nothing once the user has switched cues off', async () => {
+ await fakeBrowser.storage.local.set({ [ACTION_CUES_KEY]: false });
+ const tabId = await tabOn('https://example.com/');
+ await invokeForHarness('page.clickElement', { target: { selector: '#go' } }, tabId);
+ expect(ops()).toEqual(['action:page.clickElement']);
+ });
+
+ it('sends a page cue to the top frame and an element cue to the frame the agent is in', async () => {
+ await setFramePath(500, [{ frameId: 3, url: 'https://ads.example/', selector: 'iframe' }]);
+ await cued(500, cueFor('page.getPageInfo', {}), async () => success(null));
+ await cued(500, cueFor('page.clickElement', { target: { selector: '#go' } }), async () => success(null));
+ await forgetFrameFocus(500);
+ expect(cues().map((entry) => [entry.message.op, entry.frameId])).toEqual([
+ ['show', 0],
+ ['settle', 0],
+ ['show', 3],
+ ['settle', 3],
+ ]);
+ });
+
+ it('never holds an action up longer than the lead for a tab that does not answer', async () => {
+ vi.useFakeTimers();
+ answer = () => new Promise(() => undefined);
+ const perform = vi.fn(async () => success(null));
+ const running = cued(9, cueFor('page.clickElement', { target: { selector: '#go' } }), perform);
+ await vi.advanceTimersByTimeAsync(CUE_LEAD_MS - 1);
+ expect(perform).not.toHaveBeenCalled();
+ await vi.advanceTimersByTimeAsync(1);
+ expect(perform).toHaveBeenCalledOnce();
+ await running;
+ });
+
+ it('clears a lingering ring before a capture, and asks nothing of a tab with none', async () => {
+ await cued(4, cueFor('page.clickElement', { target: { selector: '#go' } }), async () => success(null));
+ sent = [];
+ await cued(4, cueFor('page.screenshot', {}), async () => success(null));
+ expect(ops()).toEqual(['cue:quench']);
+
+ sent = [];
+ await quenchCues(4);
+ await quenchCues(12);
+ expect(sent).toEqual([]);
+ });
+});
diff --git a/src/lib/bridge/action-cues.ts b/src/lib/bridge/action-cues.ts
new file mode 100644
index 0000000..7710eef
--- /dev/null
+++ b/src/lib/bridge/action-cues.ts
@@ -0,0 +1,91 @@
+import { browser } from 'wxt/browser';
+import type { ActionResult } from '@/lib/actions/protocol';
+import { CUE_CHANNEL, CUE_FADE_MS, CUE_LINGER_MS, type CueCommand, type CuePlan } from '@/lib/cues/events';
+import { focusedFrame, TOP_FRAME } from './frame-focus';
+import { readTheme } from './theme';
+
+export const ACTION_CUES_KEY = 'browsentic/actionCues';
+
+export const CUE_LEAD_MS = 120;
+
+const QUENCH_WAIT_MS = 300;
+
+const LIT_FOR_MS = CUE_LINGER_MS + CUE_FADE_MS + 200;
+
+type Unsent<T> = T extends unknown ? Omit<T, 'channel'> : never;
+
+type Timer = ReturnType<typeof setTimeout>;
+
+const lit = new Map<number, Map<number, Timer | undefined>>();
+
+export const actionCuesOn = (value: unknown): boolean => value === true;
+
+export async function readActionCues(): Promise<boolean> {
+ const stored = await browser.storage.local.get(ACTION_CUES_KEY);
+ return actionCuesOn(stored[ACTION_CUES_KEY]);
+}
+
+/**
+ * Shows where an action lands while it runs. The ring is asked for first and given a few
+ * frames to paint, never longer: a tab that cannot answer — frozen, or held by an alert —
+ * costs the action at most CUE_LEAD_MS, and a failed cue never fails the action.
+ */
+export async function cued(tabId: number, plan: CuePlan, perform: () => Promise<ActionResult>): Promise<ActionResult> {
+ if (plan.quench) await quenchCues(tabId);
+ if (plan.kind === 'none') return perform();
+ const [enabled, theme, frameId] = await Promise.all([
+ readActionCues(),
+ readTheme(),
+ plan.kind === 'page' ? TOP_FRAME : focusedFrame(tabId),
+ ]);
+ if (!enabled) return perform();
+
+ const id = crypto.randomUUID();
+ await within(CUE_LEAD_MS, post(tabId, frameId, { op: 'show', id, plan, theme }));
+ light(tabId, frameId, false);
+
+ let ok = false;
+ try {
+ const result = await perform();
+ ok = result.ok;
+ return result;
+ } finally {
+ void post(tabId, frameId, { op: 'settle', id, ok });
+ light(tabId, frameId, true);
+ }
+}
+
+/** Takes every ring off the tab before its pixels are read, so a capture never shows the agent its own cue. */
+export async function quenchCues(tabId: number): Promise<void> {
+ const frames = lit.get(tabId);
+ if (!frames) return;
+ lit.delete(tabId);
+ for (const timer of frames.values()) clearTimeout(timer);
+ await within(
+ QUENCH_WAIT_MS,
+ Promise.all([...frames.keys()].map((frameId) => post(tabId, frameId, { op: 'quench' }))),
+ );
+}
+
+function light(tabId: number, frameId: number, settling: boolean): void {
+ const frames = lit.get(tabId) ?? new Map<number, Timer | undefined>();
+ clearTimeout(frames.get(frameId));
+ const out = () => {
+ frames.delete(frameId);
+ if (!frames.size && lit.get(tabId) === frames) lit.delete(tabId);
+ };
+ frames.set(frameId, settling ? setTimeout(out, LIT_FOR_MS) : undefined);
+ lit.set(tabId, frames);
+}
+
+function post(tabId: number, frameId: number, command: Unsent<CueCommand>): Promise<unknown> {
+ return browser.tabs.sendMessage(tabId, { channel: CUE_CHANNEL, ...command }, { frameId }).catch(() => undefined);
+}
+
+function within(ms: number, work: Promise<unknown>): Promise<void> {
+ let timer: ReturnType<typeof setTimeout> | undefined;
+ const late = new Promise<void>((resolve) => {
+ timer = setTimeout(resolve, ms);
+ });
+ return Promise.race([work.then(() => undefined), late]).finally(() => clearTimeout(timer));
+}
diff --git a/src/lib/bridge/hands-free.test.ts b/src/lib/bridge/hands-free.test.ts
index 43cfeaa..7ade2ef 100644
--- a/src/lib/bridge/hands-free.test.ts
+++ b/src/lib/bridge/hands-free.test.ts
@@ -9,6 +9,7 @@ const world: OrbWorld = {
state: { muted: false, since: 1 },
phase: 'listening',
pushToTalk: false,
+ cues: true,
listeningTab: 7,
};
@@ -27,6 +28,11 @@ const session = (patch: Partial<TabSession>): TabSession => ({
});
describe('describeOrb', () => {
+ it('carries the action-cues switch to every tab', () => {
+ expect(describeOrb(world, 7, null).cues).toBe(true);
+ expect(describeOrb({ ...world, cues: false }, 8, null).cues).toBe(false);
+ });
+
it('listens only in the tab the microphone is aimed at', () => {
expect(describeOrb(world, 7, null).voice).toBe('listening');
expect(describeOrb(world, 8, null).voice).toBe('paused');
diff --git a/src/lib/bridge/hands-free.ts b/src/lib/bridge/hands-free.ts
index 95d0d42..13b2573 100644
--- a/src/lib/bridge/hands-free.ts
+++ b/src/lib/bridge/hands-free.ts
@@ -20,6 +20,7 @@ import {
import { pickFocusIn } from './aeye';
import { putBytes } from './file-store';
import { openMicPermissionPage } from './mic-permission';
+import { ACTION_CUES_KEY, readActionCues } from './action-cues';
import { HANDS_FREE_KEY, PUSH_TO_TALK_KEY, endHandsFree, readHandsFree, writeHandsFree, type HandsFreeState } from './panel-view';
import { attachFile, closePanels, onPanelPresence, onTurnSettled, runCommand } from './run-port';
import { openSidePanel } from './side-panel';
@@ -72,6 +73,7 @@ export interface OrbWorld {
state: HandsFreeState;
phase: DictationPhase | null;
pushToTalk: boolean;
+ cues: boolean;
listeningTab: number | null;
}
@@ -85,6 +87,7 @@ export function describeOrb(world: OrbWorld, tabId: number, session: TabSession
run: pending ? 'approval' : session?.runId ? 'working' : 'idle',
voice: world.state.muted ? 'muted' : tabId === world.listeningTab ? (world.phase ?? 'starting') : 'paused',
pushToTalk: world.pushToTalk,
+ cues: world.cues,
...(pending && { approval: approvalOf(pending) }),
};
}
@@ -151,13 +154,14 @@ async function paint(): Promise<void> {
}
showing = true;
- const [daemon, sessions, theme, position, phase, pushToTalk, front, tabs] = await Promise.all([
+ const [daemon, sessions, theme, position, phase, pushToTalk, cues, front, tabs] = await Promise.all([
readDaemon(),
readTabSessions(),
readTheme(),
readPosition(),
readPhase(),
readPushToTalk(),
+ readActionCues(),
frontTab(),
browser.tabs.query({}),
]);
@@ -171,7 +175,7 @@ async function paint(): Promise<void> {
/* The tab in front is told afresh until it shows an orb, and healed if the extension was reloaded under it. */
if (frontId != null && !orbTabs.has(frontId)) sent.delete(frontId);
- const world: OrbWorld = { theme, position, link, state, phase, pushToTalk, listeningTab: target };
+ const world: OrbWorld = { theme, position, link, state, phase, pushToTalk, cues, listeningTab: target };
await Promise.all(
tabs.map(async (tab) => {
if (tab.id == null || tab.discarded) return;
@@ -441,6 +445,10 @@ function answer(request: OrbRequest, tab: Browser.tabs.Tab & { id: number }, res
void browser.storage.local.set({ [PUSH_TO_TALK_KEY]: request.on === true });
respond({ ok: true });
return false;
+ case 'cues':
+ void browser.storage.local.set({ [ACTION_CUES_KEY]: request.on === true });
+ respond({ ok: true });
+ return false;
case 'talk':
/* Letting go always reaches the page: the key can come up in a tab the mic has already left. */
if (!request.on || tab.id === listeningTab) {
@@ -530,7 +538,8 @@ export function serveHandsFree(): void {
}
});
browser.storage.local.onChanged.addListener((changes) => {
- if (THEME_KEY in changes || ORB_POSITION_KEY in changes || SPEECH_SERVICE_KEY in changes || PUSH_TO_TALK_KEY in changes) {
+ const repaints = [THEME_KEY, ORB_POSITION_KEY, SPEECH_SERVICE_KEY, PUSH_TO_TALK_KEY, ACTION_CUES_KEY];
+ if (repaints.some((key) => key in changes)) {
void syncHandsFree();
}
});
diff --git a/src/lib/bridge/invoke.ts b/src/lib/bridge/invoke.ts
index c7661b0..b469758 100644
--- a/src/lib/bridge/invoke.ts
+++ b/src/lib/bridge/invoke.ts
@@ -56,6 +56,9 @@ import { awaitMonitorDone, monitorStatusFor, startTabMonitor, stopTabMonitor } f
import { startJobTimer, stopJobTimer, timerStatusFor } from '@/lib/bridge/timer';
import { listRecordings as listStoredMeta, readRecordingBody, recordedOnBlockedSite } from '@/lib/bridge/recording-store';
import { pickInTab } from '@/lib/bridge/pick';
+import { cued } from '@/lib/bridge/action-cues';
+import { cueFor } from '@/lib/cues/plan';
+import type { CuePlan } from '@/lib/cues/events';
import { screenshotTab } from '@/lib/bridge/screenshot';
import { dragInTab, trustedClickInTab } from '@/lib/bridge/trusted-input';
import { closeOpenTab, openNewTab, switchToTab, watchForLoad, type TabRef } from '@/lib/bridge/tabs';
@@ -82,7 +85,7 @@ export async function invokeForHarness(
if (release.refused.length) return failure('SECRET_NOT_RELEASABLE', REFUSED_MESSAGE);
if (release.unresolved.length) return failure('SECRET_EXPIRED', EXPIRED_MESSAGE);
- const result = await dispatch(action, release.input, tabId, runId);
+ const result = await dispatch(action, release.input, tabId, runId, cueFor(action, input));
const landed = await guardLanding(action, result, tabId, runId);
if (landed) return landed;
const { value } = await sealForPage(result, await targetOrigin(tabId, runId));
@@ -203,8 +206,9 @@ function hostOf(url: string): string | undefined {
async function dispatch(
action: string,
input: unknown,
- tabId?: number,
- runId?: string,
+ tabId: number | undefined,
+ runId: string | undefined,
+ cue: CuePlan,
): Promise<ActionResult> {
if (action === listFiles.name) return listStoredFiles(input);
if (action === listRecordings.name) return listStoredRecordings(input);
@@ -233,6 +237,17 @@ async function dispatch(
? failure('NO_ACTIVE_TAB', 'No active tab to control')
: failure('MAPPING_TAB_CHANGED', 'The tab this run started in has been closed.');
}
+ const { id, windowId, url } = tab;
+ return cued(id, cue, () => routeInTab({ id, windowId, url }, action, input, runId, owner));
+}
+
+async function routeInTab(
+ tab: { id: number; windowId?: number; url?: string },
+ action: string,
+ input: unknown,
+ runId: string | undefined,
+ owner: TabSession | null,
+): Promise<ActionResult> {
if (action === navigate.name) return navigateTab(tab.id, input);
if (action === searchSite.name) return searchTab(tab.id, input);
if (action === switchTab.name) return switchSessionTab({ id: tab.id, windowId: tab.windowId }, input, owner);
diff --git a/src/lib/bridge/use-extension-settings.ts b/src/lib/bridge/use-extension-settings.ts
index 170f035..c7e06da 100644
--- a/src/lib/bridge/use-extension-settings.ts
+++ b/src/lib/bridge/use-extension-settings.ts
@@ -2,6 +2,7 @@ import { useCallback, useEffect, useMemo, useState } from 'react';
import { browser } from 'wxt/browser';
import { BLOCKED_SITES_KEY, blockedBy, compileBlockedSites } from '@/lib/settings/blocked-sites';
import { shortcutsOf, shortcutsPageFor, type Shortcut } from '@/lib/settings/shortcuts';
+import { ACTION_CUES_KEY, actionCuesOn } from './action-cues';
import { CONTEXT_MENU_KEY, asContextMenuChoice, type ContextMenuChoice } from './context-menu';
import { brandFrom } from './identity';
import { PUSH_TO_TALK_KEY } from './panel-view';
@@ -42,6 +43,8 @@ export const useContextMenuChoice = (): [ContextMenuChoice, (next: ContextMenuCh
export const usePushToTalk = (): [boolean, (on: boolean) => void] => useLocalSetting(PUSH_TO_TALK_KEY, isOn);
+export const useActionCues = (): [boolean, (on: boolean) => void] => useLocalSetting(ACTION_CUES_KEY, actionCuesOn);
+
const asPatterns = (value: unknown): string[] =>
Array.isArray(value) ? value.filter((entry): entry is string => typeof entry === 'string') : [];
diff --git a/src/lib/cues/events.ts b/src/lib/cues/events.ts
new file mode 100644
index 0000000..b2213ff
--- /dev/null
+++ b/src/lib/cues/events.ts
@@ -0,0 +1,45 @@
+import type { ThemeId } from '@/lib/settings/theme';
+
+export const CUE_CHANNEL = 'browsentic/cue';
+
+export const CUE_LINGER_MS = 700;
+
+export const CUE_FADE_MS = 220;
+
+export interface CueTarget {
+ selector?: string;
+ text?: string;
+ role?: string;
+ nth: number;
+}
+
+export interface CuePoint {
+ x: number;
+ y: number;
+}
+
+export type CueAnchor =
+ | { target: CueTarget; includeHidden?: boolean }
+ | { point: CuePoint }
+ | { focused: true };
+
+export interface CuePlan {
+ kind: 'none' | 'element' | 'page';
+ verb: string;
+ detail?: string;
+ anchors: CueAnchor[];
+ quench?: boolean;
+}
+
+export type CueCommand =
+ | { channel: typeof CUE_CHANNEL; op: 'show'; id: string; plan: CuePlan; theme: ThemeId }
+ | { channel: typeof CUE_CHANNEL; op: 'settle'; id: string; ok: boolean }
+ | { channel: typeof CUE_CHANNEL; op: 'quench' };
+
+const OPS: readonly unknown[] = ['show', 'settle', 'quench'];
+
+export function isCueCommand(message: unknown): message is CueCommand {
+ if (typeof message !== 'object' || message === null) return false;
+ const frame = message as { channel?: unknown; op?: unknown };
+ return frame.channel === CUE_CHANNEL && OPS.includes(frame.op);
+}
diff --git a/src/lib/cues/host.test.ts b/src/lib/cues/host.test.ts
new file mode 100644
index 0000000..1c1ed32
--- /dev/null
+++ b/src/lib/cues/host.test.ts
@@ -0,0 +1,197 @@
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
+import { fakeBrowser } from 'wxt/testing';
+import { OVERLAY_ATTRIBUTE } from '@/lib/overlay';
+import { CUE_CHANNEL, CUE_FADE_MS, CUE_LINGER_MS, type CueCommand, type CuePlan } from './events';
+import { CUE_HOST_ID, exposeCues } from './host';
+
+type Unsent<T> = T extends unknown ? Omit<T, 'channel'> : never;
+type Trigger = (message: unknown, sender: unknown, respond: (value: unknown) => void) => unknown;
+
+let root: ShadowRoot | null;
+let watcher: MutationObserver;
+let mutations: MutationRecord[];
+
+/* The host answers through sendResponse once it has painted, which the fake's typing leaves out. */
+function deliver(command: Unsent<CueCommand>): Promise<unknown> {
+ return new Promise((resolve) => {
+ void (fakeBrowser.runtime.onMessage.trigger as unknown as Trigger)({ channel: CUE_CHANNEL, ...command }, {}, resolve);
+ });
+}
+
+async function show(id: string, plan: CuePlan): Promise<unknown> {
+ const reply = deliver({ op: 'show', id, plan, theme: 'ember' });
+ await vi.advanceTimersByTimeAsync(40);
+ return reply;
+}
+
+const click = (selector: string): CuePlan => ({ kind: 'element', verb: 'Click', anchors: [{ target: { selector, nth: 0 } }] });
+
+function laidOut<T extends Element>(element: T, box = { left: 100, top: 200, width: 80, height: 30 }): T {
+ vi.spyOn(element as Element, 'getBoundingClientRect').mockReturnValue({
+ ...box,
+ x: box.left,
+ y: box.top,
+ right: box.left + box.width,
+ bottom: box.top + box.height,
+ toJSON: () => box,
+ } as DOMRect);
+ return element;
+}
+
+function onPage<T extends HTMLElement>(html: string): T {
+ const wrap = document.createElement('div');
+ wrap.innerHTML = html;
+ const element = wrap.firstElementChild as T;
+ document.body.append(element);
+ return element;
+}
+
+const $ = <T extends Element = HTMLElement>(selector: string) => root!.querySelector(selector) as T;
+const $$ = (selector: string) => [...root!.querySelectorAll<HTMLElement>(selector)];
+const chip = () => $('.cue:not(.out) .chip').textContent;
+
+beforeEach(() => {
+ vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout', 'requestAnimationFrame', 'cancelAnimationFrame', 'performance'] });
+ fakeBrowser.reset();
+ root = null;
+ document.body.innerHTML = '';
+ const attach = Element.prototype.attachShadow;
+ vi.spyOn(Element.prototype, 'attachShadow').mockImplementation(function (this: Element, init: ShadowRootInit) {
+ root = attach.call(this, { ...init, mode: 'open' });
+ return root;
+ });
+ mutations = [];
+ watcher = new MutationObserver((records) => mutations.push(...records));
+ watcher.observe(document.documentElement, { childList: true, subtree: true, attributes: true, characterData: true });
+ exposeCues();
+});
+
+afterEach(() => {
+ watcher.disconnect();
+ vi.useRealTimers();
+ vi.restoreAllMocks();
+ document.getElementById(CUE_HOST_ID)?.remove();
+});
+
+describe('action cues on the page', () => {
+ it('rings the element the action will hit, captioned with what a person would call it', async () => {
+ laidOut(onPage('<button id="go">Sign in</button>'));
+ expect(await show('a', click('#go'))).toEqual({ ok: true });
+
+ const ring = $('.ring');
+ expect(ring.classList.contains('hidden')).toBe(false);
+ expect(ring.style.transform).toBe('translate(96px, 196px)');
+ expect(ring.style.width).toBe('88px');
+ expect(ring.style.height).toBe('38px');
+ expect(chip()).toBe('Click · Sign in');
+ expect($('.edge').classList.contains('hidden')).toBe(true);
+ });
+
+ it('adds one marked element to the page on its first cue and never touches the page again', async () => {
+ laidOut(onPage('<button id="go">Sign in</button>'));
+ await Promise.resolve();
+ mutations = [];
+
+ await show('a', click('#go'));
+ await deliver({ op: 'settle', id: 'a', ok: true });
+ await show('b', { kind: 'page', verb: 'Read page', anchors: [] });
+ await vi.advanceTimersByTimeAsync(CUE_LINGER_MS + CUE_FADE_MS + 100);
+ const quenched = deliver({ op: 'quench' });
+ await vi.advanceTimersByTimeAsync(40);
+ await quenched;
+
+ const host = document.getElementById(CUE_HOST_ID)!;
+ expect(host.hasAttribute(OVERLAY_ATTRIBUTE)).toBe(true);
+ expect(host.getAttribute('aria-hidden')).toBe('true');
+ expect(host.style.pointerEvents).toBe('none');
+ const records = [...mutations, ...watcher.takeRecords()];
+ expect(records).toHaveLength(1);
+ expect([...records[0].addedNodes]).toEqual([host]);
+ });
+
+ it('leaves focus and scroll where the page had them', async () => {
+ const field = onPage<HTMLInputElement>('<input id="q" aria-label="Search">');
+ field.focus();
+ laidOut(onPage('<button id="go">Sign in</button>'));
+ const focus = vi.spyOn(HTMLElement.prototype, 'focus');
+ const scrollIntoView = vi.fn();
+ Element.prototype.scrollIntoView = scrollIntoView;
+
+ await show('a', click('#go'));
+ await vi.advanceTimersByTimeAsync(500);
+
+ expect(document.activeElement).toBe(field);
+ expect(focus).not.toHaveBeenCalled();
+ expect(scrollIntoView).not.toHaveBeenCalled();
+ expect(window.scrollY).toBe(0);
+ });
+
+ it('never names a field by what has been typed into it', async () => {
+ laidOut(onPage('<label>Password <input id="pw" type="password" value="hunter2"></label>').querySelector('input')!);
+ await show('a', { kind: 'element', verb: 'Fill', anchors: [{ target: { selector: '#pw', nth: 0 } }] });
+ expect(chip()).toBe('Fill · Password');
+
+ laidOut(onPage('<div id="note" contenteditable="true" aria-label="Message">my secret plan</div>'));
+ await show('b', { kind: 'element', verb: 'Type', anchors: [{ target: { selector: '#note', nth: 0 } }] });
+ expect(chip()).toBe('Type · Message');
+ expect(root!.textContent).not.toContain('secret');
+ expect(root!.textContent).not.toContain('hunter2');
+ });
+
+ it('glows around the page when there is no element, or the element is not there', async () => {
+ await show('a', { kind: 'page', verb: 'Go to', detail: 'example.com', anchors: [] });
+ expect($('.edge').classList.contains('hidden')).toBe(false);
+ expect(chip()).toBe('Go to example.com');
+
+ await show('b', click('#missing'));
+ expect($('.cue:not(.out) .edge').classList.contains('hidden')).toBe(false);
+ expect($('.cue:not(.out) .ring').classList.contains('hidden')).toBe(true);
+ });
+
+ it('finds an element it is waiting for once the page adds it', async () => {
+ await show('a', { kind: 'element', verb: 'Wait for', anchors: [{ target: { selector: '#late', nth: 0 } }] });
+ expect($('.ring').classList.contains('hidden')).toBe(true);
+
+ laidOut(onPage('<button id="late">Continue</button>'));
+ await vi.advanceTimersByTimeAsync(400);
+ expect($('.ring').classList.contains('hidden')).toBe(false);
+ expect(chip()).toBe('Wait for · Continue');
+ });
+
+ it('lingers once the action is done, then fades and clears', async () => {
+ laidOut(onPage('<button id="go">Sign in</button>'));
+ await show('a', click('#go'));
+ await deliver({ op: 'settle', id: 'a', ok: false });
+ expect($('.cue').classList.contains('failed')).toBe(true);
+
+ await vi.advanceTimersByTimeAsync(CUE_LINGER_MS);
+ expect($('.cue').classList.contains('out')).toBe(true);
+ await vi.advanceTimersByTimeAsync(CUE_FADE_MS);
+ expect($$('.cue')).toHaveLength(0);
+ });
+
+ it('hands over to the next cue at once', async () => {
+ laidOut(onPage('<button id="go">Sign in</button>'));
+ await show('a', click('#go'));
+ await show('b', { kind: 'page', verb: 'Read page', anchors: [] });
+ expect($$('.cue')).toHaveLength(2);
+ expect($$('.cue')[0].classList.contains('out')).toBe(true);
+ expect($$('.cue')[1].classList.contains('out')).toBe(false);
+ });
+
+ it('clears every cue at once when asked to stay out of a capture', async () => {
+ laidOut(onPage('<button id="go">Sign in</button>'));
+ await show('a', click('#go'));
+ const reply = deliver({ op: 'quench' });
+ expect($$('.cue')).toHaveLength(0);
+ await vi.advanceTimersByTimeAsync(40);
+ expect(await reply).toEqual({ ok: true });
+ });
+
+ it('rings a point when the action names coordinates instead of an element', async () => {
+ await show('a', { kind: 'element', verb: 'Click', anchors: [{ point: { x: 50, y: 60 } }] });
+ const ring = $('.ring.point');
+ expect(ring.classList.contains('hidden')).toBe(false);
+ expect(ring.style.transform).toBe('translate(36px, 46px)');
+ });
+});
diff --git a/src/lib/cues/host.ts b/src/lib/cues/host.ts
new file mode 100644
index 0000000..9ae9f86
--- /dev/null
+++ b/src/lib/cues/host.ts
@@ -0,0 +1,309 @@
+import { browser } from 'wxt/browser';
+import { resolveTarget } from '@/lib/actions/page/dom';
+import { OVERLAY_ATTRIBUTE } from '@/lib/overlay';
+import { RAIL_PALETTES, RAIL_TONES } from '@/lib/rail/events';
+import type { ThemeId } from '@/lib/settings/theme';
+import { CUE_FADE_MS, CUE_LINGER_MS, isCueCommand, type CueAnchor, type CuePlan } from './events';
+import { cueLabel } from './label';
+import { CUE_GLOW, HOST_STYLE, STYLES } from './styles';
+
+export { CUE_CHANNEL } from './events';
+
+export const CUE_HOST_ID = 'browsentic-cues';
+
+const RING_PAD = 4;
+const POINT_SIZE = 28;
+const CHIP_GAP = 6;
+const CHIP_HEIGHT = 22;
+const EDGE_CHIP_TOP = 12;
+const VIEW_MARGIN = 8;
+const RESOLVE_EVERY_MS = 250;
+const LIVE_LIMIT_MS = 60_000;
+const PAINT_WAIT_MS = 100;
+
+interface Box {
+ x: number;
+ y: number;
+ w: number;
+ h: number;
+}
+
+interface Mark {
+ anchor: CueAnchor;
+ element: Element | null;
+ ring: HTMLElement;
+}
+
+interface Cue {
+ id: string;
+ plan: CuePlan;
+ node: HTMLElement;
+ edge: HTMLElement;
+ chip: HTMLElement;
+ marks: Mark[];
+ live: boolean;
+ resolvedAt: number;
+ timers: ReturnType<typeof setTimeout>[];
+}
+
+interface Layer {
+ alive: () => boolean;
+ show: (id: string, plan: CuePlan, theme: ThemeId) => void;
+ settle: (id: string, ok: boolean) => void;
+ quench: () => void;
+}
+
+/**
+ * Where the agent is acting, drawn into the frame it acts in. The page gets one element on
+ * its first cue and never another change: every ring, caption and fade lives in a closed
+ * shadow root, nothing in it takes a pointer, and nothing here focuses, scrolls or listens
+ * to the page.
+ */
+export function exposeCues(): void {
+ let layer: Layer | null = null;
+
+ browser.runtime.onMessage.addListener((message: unknown, _sender, sendResponse) => {
+ if (!isCueCommand(message)) return;
+ if (message.op === 'settle') {
+ layer?.settle(message.id, message.ok);
+ sendResponse({ ok: true });
+ return;
+ }
+ if (message.op === 'show') {
+ if (!layer?.alive()) layer = mountLayer();
+ layer?.show(message.id, message.plan, message.theme);
+ } else layer?.quench();
+ void painted().then(() => sendResponse({ ok: true }));
+ return true;
+ });
+
+ document.getElementById(CUE_HOST_ID)?.remove();
+ window.addEventListener('pageshow', (event) => {
+ if (event.persisted) layer?.quench();
+ });
+}
+
+function painted(): Promise<void> {
+ return new Promise((resolve) => {
+ const timer = setTimeout(resolve, PAINT_WAIT_MS);
+ requestAnimationFrame(() =>
+ requestAnimationFrame(() => {
+ clearTimeout(timer);
+ resolve();
+ }),
+ );
+ });
+}
+
+function mountLayer(): Layer | null {
+ if (!document.documentElement) return null;
+ const host = document.createElement('div');
+ host.id = CUE_HOST_ID;
+ host.setAttribute(OVERLAY_ATTRIBUTE, '');
+ host.setAttribute('aria-hidden', 'true');
+ const raisable = typeof host.showPopover === 'function';
+ if (raisable) host.setAttribute('popover', 'manual');
+ host.style.cssText = HOST_STYLE;
+ const root = host.attachShadow({ mode: 'closed' });
+ root.innerHTML = `<style>${STYLES}</style><div class="layer"></div>`;
+ const surface = root.querySelector<HTMLElement>('.layer')!;
+ document.documentElement.append(host);
+
+ const cues = new Map<string, Cue>();
+ let frame = 0;
+
+ function show(id: string, plan: CuePlan, theme: ThemeId): void {
+ paintTheme(theme);
+ for (const other of cues.values()) fade(other);
+
+ const node = part('cue');
+ const edge = part('edge hidden');
+ const chip = part('chip hidden');
+ const marks = plan.anchors.map((anchor) => ({
+ anchor,
+ element: null,
+ ring: part('point' in anchor ? 'ring point hidden' : 'ring hidden'),
+ }));
+ node.append(edge, ...marks.map((mark) => mark.ring), chip);
+ surface.append(node);
+
+ const cue: Cue = { id, plan, node, edge, chip, marks, live: true, resolvedAt: 0, timers: [] };
+ cues.set(id, cue);
+ cue.timers.push(setTimeout(() => fade(cue), LIVE_LIMIT_MS));
+ resolve(cue);
+ raise();
+ draw(cue);
+ track();
+ }
+
+ function settle(id: string, ok: boolean): void {
+ const cue = cues.get(id);
+ if (!cue?.live) return;
+ cue.live = false;
+ cue.node.classList.add(ok ? 'done' : 'failed');
+ cue.timers.push(setTimeout(() => fade(cue), CUE_LINGER_MS));
+ }
+
+ function fade(cue: Cue): void {
+ if (cue.node.classList.contains('out')) return;
+ cue.live = false;
+ cue.node.classList.add('out');
+ cue.timers.push(setTimeout(() => drop(cue), CUE_FADE_MS));
+ }
+
+ function drop(cue: Cue): void {
+ for (const timer of cue.timers) clearTimeout(timer);
+ cue.node.remove();
+ cues.delete(cue.id);
+ if (!cues.size) lower();
+ }
+
+ function quench(): void {
+ for (const cue of [...cues.values()]) drop(cue);
+ if (frame) cancelAnimationFrame(frame);
+ frame = 0;
+ }
+
+ function paintTheme(theme: ThemeId): void {
+ const palette = RAIL_PALETTES[theme];
+ surface.style.setProperty('--brand', palette.brand);
+ surface.style.setProperty('--ground2', palette.ground2);
+ surface.style.setProperty('--ink', palette.ink);
+ surface.style.setProperty('--warn', RAIL_TONES[theme].warn);
+ surface.style.setProperty('--glow', CUE_GLOW[theme]);
+ }
+
+ function resolve(cue: Cue): void {
+ cue.resolvedAt = performance.now();
+ for (const mark of cue.marks) {
+ if (!mark.element?.isConnected) mark.element = locate(mark.anchor);
+ }
+ caption(cue);
+ }
+
+ function caption(cue: Cue): void {
+ const named = cue.marks.find((mark) => mark.element)?.element;
+ const verb = document.createElement('i');
+ verb.textContent = cue.plan.verb;
+ const rest = [cue.plan.detail ? ` ${cue.plan.detail}` : '', named ? ` · ${cueLabel(named)}` : ''].join('');
+ cue.chip.replaceChildren(verb, rest);
+ }
+
+ function draw(cue: Cue): void {
+ let anchored: Box | null = null;
+ for (const mark of cue.marks) {
+ const box = boxOf(mark);
+ mark.ring.classList.toggle('hidden', !box);
+ if (!box) continue;
+ place(mark.ring, box);
+ anchored ??= box;
+ }
+ cue.edge.classList.toggle('hidden', anchored !== null);
+ placeChip(cue.chip, anchored);
+ }
+
+ function track(): void {
+ if (frame) return;
+ const tick = () => {
+ frame = 0;
+ if (!cues.size) return;
+ const now = performance.now();
+ for (const cue of cues.values()) {
+ if (cue.live && now - cue.resolvedAt > RESOLVE_EVERY_MS && cue.marks.some(unplaced)) resolve(cue);
+ draw(cue);
+ }
+ frame = requestAnimationFrame(tick);
+ };
+ frame = requestAnimationFrame(tick);
+ }
+
+ function raise(): void {
+ if (!raisable) return;
+ try {
+ if (host.matches(':popover-open')) {
+ if (!underTopLayer(host)) return;
+ host.hidePopover();
+ }
+ host.showPopover();
+ } catch {}
+ }
+
+ function lower(): void {
+ if (!raisable) return;
+ try {
+ if (host.matches(':popover-open')) host.hidePopover();
+ } catch {}
+ }
+
+ return { alive: () => host.isConnected, show, settle, quench };
+}
+
+function part(className: string): HTMLElement {
+ const node = document.createElement('div');
+ node.className = className;
+ return node;
+}
+
+function underTopLayer(host: Element): boolean {
+ if (document.fullscreenElement) return true;
+ try {
+ if (document.querySelector(':modal')) return true;
+ return [...document.querySelectorAll(':popover-open')].some((open) => open !== host);
+ } catch {
+ return false;
+ }
+}
+
+const unplaced = (mark: Mark): boolean => !('point' in mark.anchor) && !mark.element?.isConnected;
+
+function locate(anchor: CueAnchor): Element | null {
+ if ('point' in anchor) return null;
+ if ('focused' in anchor) {
+ const active = document.activeElement;
+ return active && active !== document.body && active !== document.documentElement ? active : null;
+ }
+ try {
+ return resolveTarget(anchor.target, { includeHidden: anchor.includeHidden });
+ } catch {
+ return null;
+ }
+}
+
+function boxOf(mark: Mark): Box | null {
+ if ('point' in mark.anchor) {
+ const { x, y } = mark.anchor.point;
+ return { x: x - POINT_SIZE / 2, y: y - POINT_SIZE / 2, w: POINT_SIZE, h: POINT_SIZE };
+ }
+ if (!mark.element?.isConnected) return null;
+ const rect = mark.element.getBoundingClientRect();
+ if (rect.width === 0 && rect.height === 0) return null;
+ return {
+ x: rect.left - RING_PAD,
+ y: rect.top - RING_PAD,
+ w: rect.width + RING_PAD * 2,
+ h: rect.height + RING_PAD * 2,
+ };
+}
+
+function place(node: HTMLElement, box: Box): void {
+ node.style.transform = `translate(${Math.round(box.x)}px, ${Math.round(box.y)}px)`;
+ node.style.width = `${Math.round(box.w)}px`;
+ node.style.height = `${Math.round(box.h)}px`;
+}
+
+function placeChip(chip: HTMLElement, box: Box | null): void {
+ chip.classList.remove('hidden');
+ const view = { w: document.documentElement.clientWidth || innerWidth, h: document.documentElement.clientHeight || innerHeight };
+ const width = chip.offsetWidth;
+ const clampX = (x: number) => Math.min(Math.max(x, VIEW_MARGIN), Math.max(VIEW_MARGIN, view.w - width - VIEW_MARGIN));
+ const clampY = (y: number) => Math.min(Math.max(y, VIEW_MARGIN), Math.max(VIEW_MARGIN, view.h - CHIP_HEIGHT - VIEW_MARGIN));
+
+ if (!box) {
+ chip.style.transform = `translate(${Math.round(clampX((view.w - width) / 2))}px, ${EDGE_CHIP_TOP}px)`;
+ return;
+ }
+ const above = box.y - CHIP_HEIGHT - CHIP_GAP;
+ const below = box.y + box.h + CHIP_GAP;
+ const y = above >= VIEW_MARGIN ? above : below <= view.h - CHIP_HEIGHT - VIEW_MARGIN ? below : box.y + CHIP_GAP;
+ chip.style.transform = `translate(${Math.round(clampX(box.x))}px, ${Math.round(clampY(y))}px)`;
+}
diff --git a/src/lib/cues/label.ts b/src/lib/cues/label.ts
new file mode 100644
index 0000000..3ec6d8e
--- /dev/null
+++ b/src/lib/cues/label.ts
@@ -0,0 +1,34 @@
+import { accessibleText, computedRole } from '@/lib/actions/page/dom';
+
+export const LABEL_MAX = 40;
+
+/** What a sighted user would call the element. Something the agent can type into is named, never read aloud. */
+export function cueLabel(el: Element): string {
+ const role = computedRole(el);
+ const typedInto = !isFormField(el) && (isEditable(el) || role === 'textbox' || role === 'searchbox');
+ const name = typedInto ? quietName(el) : accessibleText(el);
+ return clip(name || role || el.tagName.toLowerCase());
+}
+
+function isFormField(el: Element): boolean {
+ return el instanceof HTMLInputElement || el instanceof HTMLTextAreaElement || el instanceof HTMLSelectElement;
+}
+
+function isEditable(el: Element): boolean {
+ return el instanceof HTMLElement && (el.isContentEditable || el.getAttribute('contenteditable') === 'true');
+}
+
+function quietName(el: Element): string {
+ return (
+ el.getAttribute('aria-label')?.trim() ||
+ el.getAttribute('aria-placeholder')?.trim() ||
+ el.getAttribute('placeholder')?.trim() ||
+ el.getAttribute('title')?.trim() ||
+ ''
+ );
+}
+
+export function clip(text: string): string {
+ const flat = text.replace(/\s+/g, ' ').trim();
+ return flat.length > LABEL_MAX ? `${flat.slice(0, LABEL_MAX - 1).trimEnd()}…` : flat;
+}
diff --git a/src/lib/cues/plan.test.ts b/src/lib/cues/plan.test.ts
new file mode 100644
index 0000000..aa53c99
--- /dev/null
+++ b/src/lib/cues/plan.test.ts
@@ -0,0 +1,67 @@
+import { describe, expect, it } from 'vitest';
+import { actions } from '@/lib/actions/registry';
+import { cueFor, hasCueRule } from './plan';
+
+describe('cueFor', () => {
+ it('has a rule for every action, so a new one has to choose its cue', () => {
+ const missing = [...actions.keys()].filter((name) => !hasCueRule(name));
+ expect(missing).toEqual([]);
+ });
+
+ it('copies where an action lands and nothing it types', () => {
+ const secret = 'hunter2';
+ const plans = [
+ cueFor('page.fillInput', { target: { selector: '#pw' }, value: secret }),
+ cueFor('page.typeText', { target: { selector: '#pw' }, text: secret }),
+ cueFor('page.typeText', { text: secret }),
+ cueFor('page.searchSite', { query: secret }),
+ cueFor('page.pressKey', { key: 'h' }),
+ cueFor('page.selectOption', { target: { selector: 'select' }, value: secret, label: secret }),
+ cueFor('page.attachFile', { target: { selector: 'input' }, content: secret, name: secret }),
+ ];
+ for (const plan of plans) expect(JSON.stringify(plan)).not.toContain(secret);
+ expect(JSON.stringify(cueFor('page.pressKey', { key: 'h' }))).not.toContain('"h"');
+ });
+
+ it('rings the element an action names, with nothing beyond the target fields', () => {
+ expect(cueFor('page.clickElement', { target: { selector: '#go', text: 'Go', extra: 'x' } })).toEqual({
+ kind: 'element',
+ verb: 'Click',
+ anchors: [{ target: { selector: '#go', text: 'Go', role: undefined, nth: 0 } }],
+ });
+ });
+
+ it('falls back to the focused element for typing, and to the page for the rest', () => {
+ expect(cueFor('page.typeText', { text: 'hi' })).toMatchObject({ kind: 'element', anchors: [{ focused: true }] });
+ expect(cueFor('page.pressKey', { key: 'Enter' })).toMatchObject({ kind: 'element', detail: 'Enter' });
+ expect(cueFor('page.scrollTo', { direction: 'down' })).toMatchObject({ kind: 'page', verb: 'Scroll' });
+ expect(cueFor('page.extractText', {})).toMatchObject({ kind: 'page', verb: 'Read' });
+ });
+
+ it('marks both ends of a drag, by element or by point', () => {
+ expect(cueFor('page.dragElement', { from: { selector: '#card' }, toPoint: { x: 10, y: 20 } }).anchors).toEqual([
+ { target: { selector: '#card', text: undefined, role: undefined, nth: 0 } },
+ { point: { x: 10, y: 20 } },
+ ]);
+ });
+
+ it('names only the host a navigation goes to', () => {
+ expect(cueFor('page.navigate', { url: 'https://example.com/account?token=abc' })).toEqual({
+ kind: 'page',
+ verb: 'Go to',
+ detail: 'example.com',
+ anchors: [],
+ });
+ expect(cueFor('page.navigate', { action: 'back' })).toMatchObject({ verb: 'Go back' });
+ });
+
+ it('clears the page before anything that reads its pixels', () => {
+ for (const name of ['page.screenshot', 'page.solveCaptcha', 'page.findCaptcha', 'page.pickElement']) {
+ expect(cueFor(name, {})).toMatchObject({ kind: 'none', quench: true });
+ }
+ });
+
+ it('draws nothing for an action it does not know', () => {
+ expect(cueFor('page.somethingNew', { target: { selector: 'a' } })).toMatchObject({ kind: 'none' });
+ });
+});
diff --git a/src/lib/cues/plan.ts b/src/lib/cues/plan.ts
new file mode 100644
index 0000000..a474f3e
--- /dev/null
+++ b/src/lib/cues/plan.ts
@@ -0,0 +1,213 @@
+import { applyTheme } from '@/lib/actions/page/apply-theme';
+import { attachFile } from '@/lib/actions/page/attach-file';
+import { auditContrast } from '@/lib/actions/page/audit-contrast';
+import { awaitMonitor } from '@/lib/actions/page/await-monitor';
+import { callSiteTool } from '@/lib/actions/page/call-site-tool';
+import { captureDownload } from '@/lib/actions/page/capture-download';
+import { clickElement } from '@/lib/actions/page/click-element';
+import { closeTab } from '@/lib/actions/page/close-tab';
+import { targetSchema } from '@/lib/actions/page/dom';
+import { dragElement } from '@/lib/actions/page/drag-element';
+import { extractText } from '@/lib/actions/page/extract-text';
+import { fillInput } from '@/lib/actions/page/fill-input';
+import { findCaptcha } from '@/lib/actions/page/find-captcha';
+import { findProgress } from '@/lib/actions/page/find-progress';
+import { findSearch } from '@/lib/actions/page/find-search';
+import { focusInput } from '@/lib/actions/page/focus-input';
+import { getPageInfo } from '@/lib/actions/page/get-page-info';
+import { highlightElement } from '@/lib/actions/page/highlight-element';
+import { hoverElement } from '@/lib/actions/page/hover-element';
+import { injectCode } from '@/lib/actions/page/inject-code';
+import { listDownloads } from '@/lib/actions/page/list-downloads';
+import { listFiles } from '@/lib/actions/page/list-files';
+import { listRecordings } from '@/lib/actions/page/list-recordings';
+import { listSiteTools } from '@/lib/actions/page/list-site-tools';
+import { monitorStatus } from '@/lib/actions/page/monitor-status';
+import { navigate } from '@/lib/actions/page/navigate';
+import { openTab } from '@/lib/actions/page/open-tab';
+import { pickElement } from '@/lib/actions/page/pick-element';
+import { pointSchema } from '@/lib/actions/page/pointer';
+import { pressKey } from '@/lib/actions/page/press-key';
+import { readConsole } from '@/lib/actions/page/read-console';
+import { readNetwork } from '@/lib/actions/page/read-network';
+import { readRecording } from '@/lib/actions/page/read-recording';
+import { readTheme } from '@/lib/actions/page/read-theme';
+import { runCode } from '@/lib/actions/page/run-code';
+import { screenshot } from '@/lib/actions/page/screenshot';
+import { scrollTo } from '@/lib/actions/page/scroll-to';
+import { searchSite } from '@/lib/actions/page/search-site';
+import { selectOption } from '@/lib/actions/page/select-option';
+import { selectText } from '@/lib/actions/page/select-text';
+import { solveCaptcha } from '@/lib/actions/page/solve-captcha';
+import { startDiagnostics } from '@/lib/actions/page/start-diagnostics';
+import { startMonitor } from '@/lib/actions/page/start-monitor';
+import { startTimer } from '@/lib/actions/page/start-timer';
+import { stopDiagnostics } from '@/lib/actions/page/stop-diagnostics';
+import { stopMonitor } from '@/lib/actions/page/stop-monitor';
+import { stopTimer } from '@/lib/actions/page/stop-timer';
+import { submitForm } from '@/lib/actions/page/submit-form';
+import { switchFrame } from '@/lib/actions/page/switch-frame';
+import { switchTab } from '@/lib/actions/page/switch-tab';
+import { timerStatus } from '@/lib/actions/page/timer-status';
+import { trustedClick } from '@/lib/actions/page/trusted-click';
+import { typeText } from '@/lib/actions/page/type-text';
+import { waitForElement } from '@/lib/actions/page/wait-for-element';
+import type { CueAnchor, CuePlan, CuePoint, CueTarget } from './events';
+
+type Input = Record<string, unknown>;
+type Rule = (input: Input) => CuePlan;
+
+interface Slot {
+ target?: string;
+ point?: string;
+}
+
+interface ElementOptions {
+ slots?: Slot[];
+ focusedByDefault?: boolean;
+ includeHidden?: boolean;
+ detail?: (input: Input) => string | undefined;
+}
+
+export const NO_CUE: CuePlan = { kind: 'none', verb: '', anchors: [] };
+
+const QUENCH: CuePlan = { ...NO_CUE, quench: true };
+
+const none: Rule = () => NO_CUE;
+
+const quench: Rule = () => QUENCH;
+
+const page =
+ (verb: string, detail?: (input: Input) => string | undefined): Rule =>
+ (input) =>
+ withDetail({ kind: 'page', verb, anchors: [] }, detail?.(input));
+
+const element =
+ (verb: string, options: ElementOptions = {}): Rule =>
+ (input) => {
+ const slots = options.slots ?? [{ target: 'target' }];
+ const anchors = slots.flatMap((slot) => anchorFor(input, slot, options.includeHidden));
+ const detail = options.detail?.(input);
+ if (anchors.length) return withDetail({ kind: 'element', verb, anchors }, detail);
+ if (options.focusedByDefault) return withDetail({ kind: 'element', verb, anchors: [{ focused: true }] }, detail);
+ return withDetail({ kind: 'page', verb, anchors: [] }, detail);
+ };
+
+function withDetail(plan: CuePlan, detail: string | undefined): CuePlan {
+ return detail ? { ...plan, detail } : plan;
+}
+
+function anchorFor(input: Input, slot: Slot, includeHidden?: boolean): CueAnchor[] {
+ const target = slot.target ? targetOf(input[slot.target]) : null;
+ if (target) return [includeHidden ? { target, includeHidden } : { target }];
+ const point = slot.point ? pointOf(input[slot.point]) : null;
+ return point ? [{ point }] : [];
+}
+
+function targetOf(value: unknown): CueTarget | null {
+ const parsed = targetSchema.safeParse(value);
+ if (!parsed.success) return null;
+ const { selector, text, role, nth } = parsed.data;
+ if (!selector && !text) return null;
+ return { selector, text, role, nth };
+}
+
+function pointOf(value: unknown): CuePoint | null {
+ const parsed = pointSchema.safeParse(value);
+ return parsed.success ? { x: parsed.data.x, y: parsed.data.y } : null;
+}
+
+const namedKey = (input: Input) =>
+ typeof input.key === 'string' && input.key.length > 1 ? input.key : undefined;
+
+const HISTORY_VERBS: Record<string, string> = { back: 'Go back', forward: 'Go forward', reload: 'Reload' };
+
+const navigation: Rule = (input) => {
+ const history = typeof input.action === 'string' ? HISTORY_VERBS[input.action] : undefined;
+ if (history) return { kind: 'page', verb: history, anchors: [] };
+ return withDetail({ kind: 'page', verb: 'Go to', anchors: [] }, hostOf(input.url));
+};
+
+function hostOf(url: unknown): string | undefined {
+ if (typeof url !== 'string') return undefined;
+ try {
+ return new URL(url).hostname || undefined;
+ } catch {
+ return undefined;
+ }
+}
+
+const RULES = new Map<string, Rule>([
+ [clickElement.name, element('Click')],
+ [trustedClick.name, element('Click', { slots: [{ target: 'target', point: 'point' }] })],
+ [hoverElement.name, element('Hover')],
+ [
+ dragElement.name,
+ element('Drag', {
+ slots: [
+ { target: 'from', point: 'fromPoint' },
+ { target: 'to', point: 'toPoint' },
+ ],
+ }),
+ ],
+ [focusInput.name, element('Focus')],
+ [fillInput.name, element('Fill')],
+ [typeText.name, element('Type', { focusedByDefault: true })],
+ [pressKey.name, element('Press', { focusedByDefault: true, detail: namedKey })],
+ [selectOption.name, element('Choose')],
+ [selectText.name, element('Select')],
+ [submitForm.name, element('Submit')],
+ [waitForElement.name, element('Wait for')],
+ [scrollTo.name, element('Scroll')],
+ [extractText.name, element('Read')],
+ [attachFile.name, element('Attach', { includeHidden: true })],
+ [captureDownload.name, element('Download')],
+ [searchSite.name, element('Search')],
+
+ [getPageInfo.name, page('Read page')],
+ [navigate.name, navigation],
+ [runCode.name, page('Run script')],
+ [injectCode.name, page('Add script')],
+ [listSiteTools.name, page('Read site tools')],
+ [callSiteTool.name, page('Use site tool')],
+ [findProgress.name, page('Read page')],
+ [findSearch.name, page('Read page')],
+ [readTheme.name, page('Inspect styles')],
+ [auditContrast.name, page('Inspect styles')],
+ [applyTheme.name, page('Restyle page')],
+
+ [screenshot.name, quench],
+ [findCaptcha.name, quench],
+ [solveCaptcha.name, quench],
+ [pickElement.name, quench],
+
+ [highlightElement.name, none],
+ [openTab.name, none],
+ [switchTab.name, none],
+ [closeTab.name, none],
+ [switchFrame.name, none],
+ [startDiagnostics.name, none],
+ [readConsole.name, none],
+ [readNetwork.name, none],
+ [stopDiagnostics.name, none],
+ [startMonitor.name, none],
+ [monitorStatus.name, none],
+ [awaitMonitor.name, none],
+ [stopMonitor.name, none],
+ [startTimer.name, none],
+ [timerStatus.name, none],
+ [stopTimer.name, none],
+ [listFiles.name, none],
+ [listRecordings.name, none],
+ [readRecording.name, none],
+ [listDownloads.name, none],
+]);
+
+export const hasCueRule = (action: string): boolean => RULES.has(action);
+
+/** Built from the sealed input: only where the action lands is copied, never what it types. */
+export function cueFor(action: string, input: unknown): CuePlan {
+ const rule = RULES.get(action);
+ if (!rule) return NO_CUE;
+ return rule(typeof input === 'object' && input !== null ? (input as Input) : {});
+}
diff --git a/src/lib/cues/styles.ts b/src/lib/cues/styles.ts
new file mode 100644
index 0000000..cfe5963
--- /dev/null
+++ b/src/lib/cues/styles.ts
@@ -0,0 +1,101 @@
+import type { ThemeId } from '@/lib/settings/theme';
+import { CUE_FADE_MS } from './events';
+
+/** --glow in globals.css, theme by theme; the page has no stylesheet to read it from. */
+export const CUE_GLOW: Record<ThemeId, string> = {
+ ember: '72%',
+ midnight: '72%',
+ phosphor: '85%',
+ daylight: '26%',
+};
+
+export const HOST_STYLE = [
+ 'all: initial',
+ 'position: fixed',
+ 'inset: 0',
+ 'width: 100vw',
+ 'height: 100vh',
+ 'max-width: none',
+ 'max-height: none',
+ 'margin: 0',
+ 'padding: 0',
+ 'border: 0',
+ 'background: transparent',
+ 'overflow: visible',
+ 'pointer-events: none',
+ 'display: block',
+ 'z-index: 2147483647',
+].join('; ');
+
+export const STYLES = `
+ :host { all: initial; }
+ * { box-sizing: border-box; margin: 0; padding: 0; pointer-events: none; }
+ .layer {
+ position: fixed;
+ inset: 0;
+ overflow: hidden;
+ font-family: ui-sans-serif, system-ui, -apple-system, "Segoe UI", sans-serif;
+ }
+ .cue { --tone: var(--brand); position: absolute; inset: 0; transition: opacity ${CUE_FADE_MS}ms ease-out; }
+ .cue.failed { --tone: var(--warn); }
+ .cue.out { opacity: 0; }
+ .ring {
+ position: absolute;
+ left: 0;
+ top: 0;
+ border: 2px solid var(--tone);
+ border-radius: 8px;
+ box-shadow:
+ 0 0 0 1px color-mix(in oklch, var(--tone) 22%, transparent),
+ 0 0 18px 1px color-mix(in oklch, var(--tone) var(--glow), transparent),
+ inset 0 0 12px -4px color-mix(in oklch, var(--tone) var(--glow), transparent);
+ will-change: transform;
+ animation: browsentic-cue-in 160ms ease-out, browsentic-cue-pulse 1.4s ease-in-out 160ms infinite;
+ }
+ .ring.point { border-radius: 999px; }
+ .ring.hidden, .chip.hidden, .edge.hidden { display: none; }
+ .cue.done .ring, .cue.failed .ring { animation: none; }
+ .edge {
+ position: absolute;
+ inset: 0;
+ box-shadow:
+ inset 0 0 0 2px color-mix(in oklch, var(--tone) 70%, transparent),
+ inset 0 0 38px -6px color-mix(in oklch, var(--tone) var(--glow), transparent);
+ animation: browsentic-cue-in 160ms ease-out, browsentic-cue-breathe 1.6s ease-in-out 160ms infinite;
+ }
+ .cue.done .edge, .cue.failed .edge { animation: none; }
+ .chip {
+ position: absolute;
+ left: 0;
+ top: 0;
+ max-width: calc(100vw - 16px);
+ padding: 3px 9px 4px;
+ border: 1px solid color-mix(in oklch, var(--tone) 45%, transparent);
+ border-radius: 999px;
+ background: var(--ground2);
+ color: var(--ink);
+ font: 600 11px/1.3 inherit;
+ letter-spacing: 0.01em;
+ white-space: nowrap;
+ overflow: hidden;
+ text-overflow: ellipsis;
+ box-shadow: 0 6px 26px -10px color-mix(in oklch, var(--tone) var(--glow), transparent);
+ will-change: transform;
+ }
+ .chip i { font-style: normal; color: var(--tone); }
+ @keyframes browsentic-cue-in { from { opacity: 0; } }
+ @keyframes browsentic-cue-pulse {
+ 50% {
+ box-shadow:
+ 0 0 0 3px color-mix(in oklch, var(--tone) 18%, transparent),
+ 0 0 26px 3px color-mix(in oklch, var(--tone) var(--glow), transparent),
+ inset 0 0 12px -4px color-mix(in oklch, var(--tone) var(--glow), transparent);
+ }
+ }
+ @keyframes browsentic-cue-breathe { 50% { opacity: 0.65; } }
+ @media (prefers-reduced-motion: reduce) {
+ .cue { transition: none; }
+ .ring, .edge { animation: none; }
+ }
+ @media print { .layer { display: none; } }
+`;
diff --git a/src/lib/handsfree/events.ts b/src/lib/handsfree/events.ts
index 9c806f8..7904139 100644
--- a/src/lib/handsfree/events.ts
+++ b/src/lib/handsfree/events.ts
@@ -66,6 +66,8 @@ export interface OrbView {
voice: OrbVoice;
/** The mic stays off until the hold key is held, and letting go sends. */
pushToTalk: boolean;
+ /** A ring marks each element the agent acts on. */
+ cues: boolean;
approval?: OrbApproval;
}
@@ -98,6 +100,7 @@ export type OrbRequest =
| { channel: typeof HANDS_FREE_CHANNEL; op: 'detach'; fileId: string }
| { channel: typeof HANDS_FREE_CHANNEL; op: 'move'; position: OrbPosition }
| { channel: typeof HANDS_FREE_CHANNEL; op: 'pushToTalk'; on: boolean }
+ | { channel: typeof HANDS_FREE_CHANNEL; op: 'cues'; on: boolean }
| { channel: typeof HANDS_FREE_CHANNEL; op: 'talk'; on: boolean };
export type DictationReport =
@@ -110,7 +113,7 @@ export type DictationCommand = { channel: typeof DICTATION_CHANNEL; op: 'talk';
const ORB_COMMANDS = new Set(['show', 'hide', 'heard', 'say']);
const ORB_REQUESTS = new Set([
'sync', 'submit', 'cancel', 'decide', 'openPanel', 'listen', 'grantMic', 'pick', 'attach', 'detach', 'move',
- 'pushToTalk', 'talk',
+ 'pushToTalk', 'cues', 'talk',
]);
const DICTATION_REPORTS = new Set(['phase', 'heard']);
const DICTATION_COMMANDS = new Set(['talk']);
@@ -157,6 +160,18 @@ export const ORB_ICONS = {
'm15 15-3-3-3 3',
],
code: ['m18 16 4-4-4-4', 'm6 8-4 4 4 4', 'm14.5 4-5 16'],
+ cues: [
+ 'M12.034 12.681a.498.498 0 0 1 .647-.647l9 3.5a.5.5 0 0 1-.033.943l-3.444 1.068a1 1 0 0 0-.66.66l-1.067 3.443a.5.5 0 0 1-.943.033z',
+ 'M5 3a2 2 0 0 0-2 2',
+ 'M19 3a2 2 0 0 1 2 2',
+ 'M5 21a2 2 0 0 1-2-2',
+ 'M9 3h1',
+ 'M9 21h2',
+ 'M14 3h1',
+ 'M3 9v1',
+ 'M21 9v2',
+ 'M3 14v1',
+ ],
focus: [
'M3 7V5a2 2 0 0 1 2-2h2',
'M17 3h2a2 2 0 0 1 2 2v2',
diff --git a/src/lib/handsfree/geometry.test.ts b/src/lib/handsfree/geometry.test.ts
index 037d091..7d84213 100644
--- a/src/lib/handsfree/geometry.test.ts
+++ b/src/lib/handsfree/geometry.test.ts
@@ -12,7 +12,7 @@ import {
} from './geometry';
const viewport = { width: 1440, height: 900 };
-const ITEMS = 5;
+const ITEMS = 6;
const half = ORB_SIZE / 2 + ORB_INSET;
describe('placeOrb', () => {
diff --git a/src/lib/handsfree/geometry.ts b/src/lib/handsfree/geometry.ts
index feb3b95..71e862a 100644
--- a/src/lib/handsfree/geometry.ts
+++ b/src/lib/handsfree/geometry.ts
@@ -15,10 +15,10 @@ export type Side = 'top' | 'bottom' | 'left' | 'right';
export const ORB_SIZE = 56;
export const ORB_INSET = 24;
export const MENU_ITEM = 38;
-export const MENU_RADIUS = 80;
+export const MENU_RADIUS = 88;
/* A corner leaves a quarter turn, plus the slack of the orb's inset on either side: 110° at
- 96 px keeps five items apart and every one of them on screen. */
-const CORNER_RADIUS = 96;
+ 118 px keeps six items apart and every one of them on screen. */
+const CORNER_RADIUS = 118;
const EDGE_SPAN = 150;
const CORNER_SPAN = 110;
const VIEW_MARGIN = 12;
diff --git a/src/lib/handsfree/host.test.ts b/src/lib/handsfree/host.test.ts
index b5c88ca..6f63d7c 100644
--- a/src/lib/handsfree/host.test.ts
+++ b/src/lib/handsfree/host.test.ts
@@ -16,6 +16,7 @@ const view = (patch: Partial<OrbView> = {}): OrbView => ({
run: 'idle',
voice: 'listening',
pushToTalk: false,
+ cues: true,
...patch,
});
@@ -222,6 +223,17 @@ describe('the hands-free orb', () => {
expect(slots).toEqual([...slots].sort((a, b) => a - b));
});
+ it('switches action cues from the menu and shows which way they are set', async () => {
+ await show();
+ const item = $('[data-item="cues"]');
+ expect(item.classList.contains('on')).toBe(true);
+ item.dispatchEvent(new MouseEvent('click', { bubbles: true }));
+ expect(requests('cues')).toEqual([expect.objectContaining({ on: false })]);
+ await show({ cues: false });
+ expect(item.classList.contains('on')).toBe(false);
+ expect(item.querySelector('.tip')?.textContent).toBe('Show what the agent does');
+ });
+
it('rides live code and an A-Eye pick along with the next instruction, then lets them go', async () => {
await show();
$('[data-item="code"]').dispatchEvent(new MouseEvent('click', { bubbles: true }));
diff --git a/src/lib/handsfree/host.ts b/src/lib/handsfree/host.ts
index 746ff82..b477c89 100644
--- a/src/lib/handsfree/host.ts
+++ b/src/lib/handsfree/host.ts
@@ -48,7 +48,7 @@ const SETTLE_MS = 2500;
const SPEECH_STAGGER_MS = 45;
const REPLY_STAGGER_MS = 32;
-type MenuId = 'file' | 'code' | 'focus' | 'talk' | 'panel';
+type MenuId = 'file' | 'code' | 'focus' | 'cues' | 'talk' | 'panel';
type CaptionKind = 'speech' | 'sent' | 'reply' | 'error' | 'ask' | 'hint';
type Unsent<T> = T extends unknown ? Omit<T, 'channel'> : never;
@@ -64,6 +64,7 @@ const MENU: { id: MenuId; icon: OrbIcon; label: string }[] = [
{ id: 'file', icon: 'file', label: 'Attach a file' },
{ id: 'code', icon: 'code', label: 'Live code' },
{ id: 'focus', icon: 'focus', label: 'Focus point (A-Eye)' },
+ { id: 'cues', icon: 'cues', label: 'Show what the agent does' },
{ id: 'talk', icon: 'keyboard', label: `Hold ${HOLD_KEY.name} to talk` },
{ id: 'panel', icon: 'panel', label: 'Open the side panel' },
];
@@ -392,6 +393,10 @@ function mountOrb(first: OrbView): Orb | null {
talkItem?.classList.toggle('on', view.pushToTalk);
const talkTip = talkItem?.querySelector('.tip');
if (talkTip) talkTip.textContent = view.pushToTalk ? `Hold to talk is on — ${HOLD_KEY.name}` : `Hold ${HOLD_KEY.name} to talk`;
+ const cuesItem = items.get('cues');
+ cuesItem?.classList.toggle('on', view.cues);
+ const cuesTip = cuesItem?.querySelector('.tip');
+ if (cuesTip) cuesTip.textContent = view.cues ? 'Showing what the agent does' : 'Show what the agent does';
button.setAttribute('aria-label', pressLabel());
if (!before) return;
@@ -734,6 +739,10 @@ function mountOrb(first: OrbView): Orb | null {
} else if (id === 'focus') {
closeMenu();
void pickFocus();
+ } else if (id === 'cues') {
+ const on = !view.cues;
+ void request({ op: 'cues', on });
+ hint(on ? 'A ring now marks each element the agent acts on.' : 'The agent’s actions are no longer marked on the page.');
} else if (id === 'talk') {
const on = !view.pushToTalk;
void request({ op: 'pushToTalk', on });
diff --git a/vitest.config.ts b/vitest.config.ts
index 7508bbf..0b9f36a 100644
--- a/vitest.config.ts
+++ b/vitest.config.ts
@@ -23,7 +23,12 @@ export default defineConfig({
test: {
name: 'lib',
include: ['src/lib/**/*.test.ts'],
- exclude: [...configDefaults.exclude, 'src/lib/actions/page/**', 'src/lib/handsfree/host.test.ts'],
+ exclude: [
+ ...configDefaults.exclude,
+ 'src/lib/actions/page/**',
+ 'src/lib/handsfree/host.test.ts',
+ 'src/lib/cues/host.test.ts',
+ ],
},
},
{
@@ -31,7 +36,7 @@ export default defineConfig({
test: {
name: 'dom',
environment: 'happy-dom',
- include: ['src/lib/actions/page/**/*.test.ts', 'src/lib/handsfree/host.test.ts'],
+ include: ['src/lib/actions/page/**/*.test.ts', 'src/lib/handsfree/host.test.ts', 'src/lib/cues/host.test.ts'],
},
},
{