An agentic browser harness: an AI side panel & orb in your own logged-in Chrome, Edge, Arc, Brave or even Firefox run by the agent/subscription you already use (Claude Code, Codex, Antigravity). Free, open source, no API key. Also an MCP server.
TypeScript agentic-browseragentic-browsingagy-cliaiai-agentai-browser-extensionai-toolsbrave-extensionbrowser-agentbrowser-automationbrowser-extensionbrowser-usebrowserskillchrome-extensionclaude-codecodexfirefox-addonmcp-servernanobrowserweb-automation
docs(hands-free): the feature page, which browsers get it, why left Control, how it works inside the extension, and what to do when it will not listen
Shahriarac4093ae4e3d
diff --git a/docs/guide/features/README.md b/docs/guide/features/README.mdindex c0ecb38..e6aa03d 100644--- a/docs/guide/features/README.md+++ b/docs/guide/features/README.md@@ -10,6 +10,7 @@ first.| | || --- | --- || [Conversations](conversations.md) | Voice and text, one conversation per tab, the sessions strip, history |+| [Hands-free](hands-free.md) | Put the panel away and talk to the page through a mic you can drag anywhere || [Instant commands](instant-commands.md) | "go back", "scroll to the top" — the things that run in milliseconds without waking an agent || [A-Eye](a-eye.md) | Point at an element instead of describing it, and let it point back when it needs to || [Skills](skills.md) | How an instruction gets routed to the right playbook, and how to write your own |diff --git a/docs/guide/features/a-eye.md b/docs/guide/features/a-eye.mdindex ee5ce55..b7dd12f 100644--- a/docs/guide/features/a-eye.md+++ b/docs/guide/features/a-eye.md@@ -50,6 +50,9 @@ agent is told to ask in words instead, not to ask you to point again.- **One lens at a time.** Starting a new pick — from the button, or the agent asking — dismisses alens already waiting, and whoever opened that one is told it was cancelled.- **Top frame only.** An element inside an iframe picks the iframe, not what is in it.+- **Never Browsentic's own.** The [hands-free](hands-free.md) mic, the minimized rail and+ Browsentic's cards are skipped: the lens draws nothing over them, and a click on one picks+ nothing. A pick started from the hands-free mic's menu has the mic step aside until it is done.- **The element, not the page.** The photograph covers what you picked, with a sliver of margin.For the page around it, ask for a [screenshot](screenshots.md) as well.- **Long elements are cut.** Picking a whole article sends the first couple of thousand characters;diff --git a/docs/guide/features/conversations.md b/docs/guide/features/conversations.mdindex c8524d6..fdaa9d2 100644--- a/docs/guide/features/conversations.md+++ b/docs/guide/features/conversations.md@@ -6,13 +6,14 @@ How you talk to Browsentic, and how it keeps track of several things at once.## Voice and text-Three ways in:+Four ways in:| | || --- | --- || **Type** | The composer at the bottom of the side panel || **Dictate** | Hands-free in the side panel — press the mic and talk || **Press to talk** | In the popup, for when the panel is not open |+| **Hands-free** | The panel put away, and a mic on the page — see [Hands-free](hands-free.md) |Speech uses the browser's built-in recognition. Nothing is bundled or downloaded — which in Chromemeans audio is streamed to Google for transcription. Type instead if that matters to you; see@@ -66,6 +67,9 @@ and closes it instead.The rail belongs to the minimized state alone. Closing the panel with the browser's own closebutton removes any rail from every page — only the header's collapse button leaves one behind.+The header's detach button, beside collapse, puts the panel away differently: instead of the rail,+a microphone on the page you talk to directly. See [Hands-free](hands-free.md).+### Knowing something is running when you are elsewhereWhile a conversation is working, its tab is marked in two places — a dot on the Browsentic toolbardiff --git a/docs/guide/features/hands-free.md b/docs/guide/features/hands-free.mdnew file mode 100644index 0000000..faed787--- /dev/null+++ b/docs/guide/features/hands-free.md@@ -0,0 +1,129 @@+# Hands-free++Put the side panel away and talk to the page. A microphone floats at the bottom of the page instead,+and everything you would reach for in the panel is a hover away.++Hands-free exists only in browsers that can turn speech into text — Chrome, Edge, and the other+Chromium browsers with a speech service behind them. Anywhere else there is no detach button, no+`/hands-free` and no mic on the page (see [Which browsers](#which-browsers)).++---++## Turning it on++Either of:++- the **detach** button in the side panel's header, just left of minimize+- typing **`/hands-free`** in the composer — the `/` menu offers it as soon as you type `/hand`, and+ `/hand-free` works too++The panel folds down into a mic and closes, and the same mic rises at the bottom middle of the page.+It follows you from tab to tab. Opening the side panel again — from the mic's own menu, the toolbar+icon or the right-click **Open Browsentic** item — turns hands-free off; the panel and the mic never+listen at the same time.++Hands-free lasts until you open the panel or the browser closes. A browser that restarts comes back+with the panel, never with a microphone already on.++## Talking++Just speak. What you say streams in above the mic word by word, and the mic's edge pulses while it+hears you. Pause, and a ring runs once around the mic — when it closes, the instruction is sent.++| Press the mic while… | It… |+| --- | --- |+| the ring is running, or you are mid-sentence | throws away what you said, so you can say it again |+| the agent is working (a turning cyan ring) | stops the run |+| the agent is asking permission (an orange shield) | opens the request, pointing at the mic |+| it is listening | mutes it; press again to listen |+| it needs the microphone | opens the tab where the browser can ask for it |++When a run finishes, its answer streams in above the mic the same way, and fades after a moment.++The mic listens for **the tab in front of you**, and only while the browser window is focused. Switch+tabs and it follows; switch to another app and it stops until you come back. It also stops while the+agent is working, and while a side panel is open anywhere.++## The menu++Rest the pointer on the mic for a second and five buttons fan out around it:++| | |+| --- | --- |+| **Attach a file** | Hands a file to this tab's conversation, as dropping it on the panel does — see [Files](files.md) |+| **Live code** | Lets the agent write a script for this page, as the composer's code toggle does; you approve the code before it runs |+| **Focus point** | Opens [A-Eye](a-eye.md): point at an element and it rides along with your next instruction |+| **Hold to talk** | Switches between listening all the time and listening only while you hold a key — see below |+| **Open the side panel** | Brings the panel back and ends hands-free |++What rides along — the element, live code, attached files — shows as chips beside the mic until you+next speak. Each has a **×**.++## Hold to talk++Switch it on from the menu and the mic stops listening on its own. Hold **left Control** — **control**+on a Mac — and speak; let go, and the mic mutes and sends what you said at once, with no countdown.+The key's name shows on the mic while this is on, and the mic grows and turns pink while you hold it.+The setting is remembered.++Why that key: it sits at the far left on Mac and PC keyboards alike, it is the same physical key on+every layout — QWERTY, AZERTY, Dvorak — and on its own it does nothing in any browser or system.++Control is also the key for shortcuts, so a hold only counts if nothing else joins it. Tap it, or+press another key with it (Ctrl + C), click or scroll while it is down, and the mic never opens; if+it had already opened, what it heard is thrown away rather than sent. A key press a web page makes+up is ignored — only you can open the mic.++## Moving it++Press and hold the mic until it grows a little, then drag it anywhere. It stays where you drop it on+every tab. Against a side or a corner, the menu turns to open away from the edge and deals its+buttons out in that turned order; captions and permission requests move to whichever side has room.++## When the agent needs your OK++The mic turns orange and says what the agent is asking to do. Press it for the request — the same+**Allow**, **Deny** and **Always on ‹host›** the panel offers, or the code itself when the agent+wrote some. If the request comes from a tab you are not looking at, a card appears on the page you+are on, or a system notification when the browser is in the background; clicking it takes you there.++## A-Eye never lands on the mic++The mic, like the rail and Browsentic's cards, is marked as Browsentic's own. A-Eye — yours or the+agent's — never outlines it and never picks it; a click on the mic while A-Eye is up does nothing.+When you start A-Eye from the mic's menu, the mic steps aside until you have picked.++## Which browsers++Browsentic offers hands-free only where it can actually hear you:++| | |+| --- | --- |+| **Chrome, Edge** | Offered |+| **Firefox** | Never offered — it has no speech recognition, and no hidden page to listen from |+| **Brave** | Never offered — it ships the speech API with no service behind it, so every attempt fails |+| **Other Chromium browsers** | Offered until their speech service first fails without ever having transcribed a word. From then on the detach button and `/hands-free` are gone, and a mic already on the page leaves with a note saying why |++A browser whose speech service has transcribed even once keeps hands-free for good: a later failure+there is a dropped connection, not a missing service. And one written off can win it back — dictate+in the panel, and the first word it transcribes brings the detach button back.++## Limits++- **Speech goes to Google** while hands-free is listening, as it does for the panel's dictation —+ see [Limits](../limits.md#speech-goes-to-google).+- **One page at a time can listen.** Chrome runs one speech recognizer at once. If another page+ starts listening, the mic lets it have the microphone and says so; press it to take it back.+- **Background noise can become an instruction.** Anything that sounds like speech is transcribed.+ The ring before sending is your chance to press and throw it away; mute the mic when you are not+ using it.+- **Hold to talk needs the page itself to have the keyboard.** While you are typing inside a frame+ embedded in the page — some rich editors, Google Docs among them — the key goes to that frame and+ the mic does not hear it. Click the page around it first.+- **No mic where Browsentic is not allowed** — `chrome://` pages, the Chrome Web Store, the new tab+ page. The mic stops listening there, and the toolbar icon is the way back to the panel.++## See also++- [Conversations](conversations.md) — typing and dictating in the panel, and minimizing it to the rail+- [Approvals](../approvals.md) — what asks first, and whydiff --git a/docs/guide/limits.md b/docs/guide/limits.mdindex 38ae6dc..7d0d47c 100644--- a/docs/guide/limits.md+++ b/docs/guide/limits.md@@ -108,6 +108,10 @@ Voice input uses Chrome's built-in Web Speech API, which streams audio to GoogleNo model is bundled and nothing is downloaded. If that is not acceptable, type instead — it is onefile to replace the speech engine.+[Hands-free](features/hands-free.md) listens the same way, and for as long as it is on and the+browser is in front — not only while you are talking to it. Switch its **Hold to talk** on and it+listens only while you hold the key.+## Recording and mapping limits**Recordings** run for at most 15 minutes, follow one tab, live in extension storage rather than ondiff --git a/docs/guide/troubleshooting.md b/docs/guide/troubleshooting.mdindex 38268d2..da6fb94 100644--- a/docs/guide/troubleshooting.md+++ b/docs/guide/troubleshooting.md@@ -27,6 +27,10 @@ Those three answer most questions. The daemon log also lives at `~/.browsentic/d| "Browsentic has not been given the microphone yet" | A browser cannot show its microphone prompt inside a side panel or a popup, so a fresh install has never been asked | Press **Allow microphone** and choose **Allow** in the tab that opens || "Microphone access is blocked" | The microphone was refused for the extension | Press **Allow microphone**, then click the icon at the left of that tab's address bar and set **Microphone** to **Allow** || "This browser has no speech service" | Brave and some Chromium builds ship speech recognition without a transcription service behind it | Type instead, or use Chrome or Edge for dictation |+| No detach button in the panel's header, and no `/hands-free` | Hands-free is offered only where speech can be transcribed — never in Firefox or Brave, and not in a browser whose speech service has failed before it ever worked | Use Chrome or Edge. If the service was only out of reach, dictate in the panel once: the first word it transcribes brings the button back |+| "Hands-free is not available here", and the mic left the page | The speech service failed before it had ever transcribed a word in this browser | As above. Open Browsentic from the toolbar to carry on by typing |+| The hands-free mic turns amber and says another page took the microphone | Chrome runs one speech recognizer at a time, and another page or the panel started one | Press the mic to take it back |+| Holding Control does nothing | Hold to talk is off; the key went to a frame embedded in the page (some editors); another key or a click joined it; or the mic is muted | Turn **Hold to talk** on from the mic's menu, click the page outside the embedded editor, and hold left Control on its own || `EXTENSION_OFFLINE` | Browser closed, or not paired | Open the browser; `browsentic sessions` to check pairing || "Load unpacked" cannot see `~/browsentic` | A Flatpak or Snap browser, sandboxed away from your home directory. Snap Chromium is the Ubuntu default | Grant it: `flatpak override --user --filesystem=~/browsentic com.google.Chrome`. Or install somewhere the sandbox can read: `browsentic setup --dir ~/snap/chromium/common/browsentic-extension` || The folder picker does not show `~/browsentic` | It is there; some pickers open elsewhere by default | macOS: press ⇧⌘G and paste the path. Linux: Ctrl+L |diff --git a/docs/internals/extension.md b/docs/internals/extension.mdindex f207bdb..7a1bb55 100644--- a/docs/internals/extension.md+++ b/docs/internals/extension.md@@ -35,8 +35,8 @@ microphone to an extension. `use-speech.ts` therefore reads `navigator.permissio`prompt` it never calls `getUserMedia` — repeated silent refusals would earn the origin a temporaryblock — and offers **Allow microphone** instead. That opens the unlisted `mic-permission.html`entrypoint in a tab, where the prompt can appear. The grant belongs to the extension's origin, so the-panel and the popup both see the `PermissionStatus` change and start listening without a reload.-Firefox prompts from its sidebar on its own and skips the check.+panel, the popup and hands-free mode's offscreen page all see the `PermissionStatus` change and start+listening without a reload. Firefox prompts from its sidebar on its own and skips the check.## The Firefox build is signed, and says who it is@@ -54,8 +54,8 @@ a `browser_specific_settings.gecko` block the Chrome build has no use for:behind whichever agent CLI the user runs.- **`strict_min_version: 140.0`** — the first Firefox that understands the data-collection key.-Two permissions are filtered out of that build, `sidePanel` and `debugger`: Firefox has neither, and-AMO's validator flags each name it does not know. The Firefox sidebar is `sidebar_action`, which WXT+Three permissions are filtered out of that build, `sidePanel`, `debugger` and `offscreen`: Firefox has+none of them, and AMO's validator flags each name it does not know. The Firefox sidebar is `sidebar_action`, which WXTderives from the same entrypoint, and the nine tools that need the debugger are left off the list aFirefox build offers (see [Background vs content script](#background-vs-content-script)).@@ -187,6 +187,94 @@ Pages that refuse content scripts — `chrome://`, the Web Store, the new-tab paThat is the same `TAB_UNREACHABLE` set as everywhere else, and it is why the toolbar icon and the**Open/Close Browsentic** context-menu item stay the guaranteed way back in.+## Hands-free: the orb lives in the page++Detaching is the rail's sibling: the panel closes, and a microphone orb is drawn into the page+instead, which the user talks to. The panel cannot be shrunk, so the fold the user sees — the panel+collapsing into a mic that drops away — is `DetachVeil` playing for 560 ms before `closeSidePanel()`,+and the page's orb rising once the page has widened. Firefox closes its sidebar only inside the+click's gesture, so it skips the fold.++| | |+| --- | --- |+| `src/lib/handsfree/events.ts` | The channel, `OrbView`, the orb's requests, the dictation reports, and the Lucide paths |+| `src/lib/handsfree/host.ts` + `styles.ts` | `exposeHandsFree()` — the orb, its arc menu, caption, countdown ring and approval card, in a closed shadow root |+| `src/lib/handsfree/geometry.ts` | Pure layout: which way the arc opens from the edges the orb is against, and which side a caption or card fits on |+| `src/lib/bridge/hands-free.ts` | `serveHandsFree()` — when the microphone listens and for which tab, what every orb shows, and the orb's requests |+| `src/extension/entrypoints/dictation/` | The offscreen page speech recognition runs in, Chromium only |++**Speech needs a document, and the panel is gone.** The background has no DOM, and recognition in a+content script would run on the page's origin — a prompt per site, and none at all where the page's+`Permissions-Policy` forbids the microphone. So recognition runs in an **offscreen document**+(`dictation.html`, reason `USER_MEDIA`) on the extension's own origin, which the `mic-permission`+tab already granted. The document exists exactly while the orb should listen: creating it starts the+mic, closing it stops it, so it needs no command channel of its own. It reports a `DictationPhase`+and each interim and final transcript; the background keeps the phase in `browsentic/dictation` and+relays transcripts to the one tab listening.++**Chrome runs one recognizer at a time**, and a second one aborts the first. That is why the panel+and hands-free never overlap — the panel's `panelOpened` ends hands-free, and the orb does not listen+while any panel's run port is connected — and why an `aborted` the document did not ask for is read+as another page taking the mic. The document then stops as `yielded` rather than restarting into a+tug-of-war; a press on the orb takes the mic back.++**Hands-free only exists where speech does.** `speech-support.ts` decides it in two layers.+`speechCapable()` is what is knowable without listening: a Firefox build (no offscreen documents, no+recognizer), no `offscreen` API, no recognizer constructor, or a brand known to ship the recognizer+without a service behind it (`Brave`, which fails every attempt with `network`). What that lets+through is settled by the service itself: the panel's dictation and the offscreen page record+`browsentic/speechService` in `storage.local` — `works` on the first transcript, `missing` on a+`network` failure, and `works` always wins, since a service that answered once is only out of reach.+The panel hides the detach button and `/hands-free` behind `useHandsFreeSupported()`, which starts+hidden so nothing flashes; `paint()` ends hands-free on the spot if it finds itself on anywhere+unsupported, and a `missing` found mid-session raises a toast saying why the mic left. Note that+Vitest serves `import.meta.env.FIREFOX` as the string `"false"`, which is truthy — the decision+itself is the pure `capableOf()` so it can be tested at all.++**The microphone listens for one tab**: the one in front of a focused window, whose conversation has+no run and no approval waiting, while the link is up and the user has not muted it. Every other orb+is painted `paused`. `paint()` computes each tab's `OrbView`, sends only the ones that changed since+that tab was last told, and records which tabs answered — a tab with no orb is never aimed at. Like+the rail it is a cache the worker can lose: a revived worker repaints everything.++**The orb owns only what dies with the page**: what has been said since the last send, the countdown+ring (`AUTO_SEND_MS`, the composer's 1.6 s), the A-Eye focus, the live-code toggle and the files+attached since the last send. An instruction goes through the same `instruct` the panel's run port+takes, via `runCommand()`, so it lands in the tab's own conversation and takes the fast path as a+typed one would. A finished turn's last reply or error comes back as a `say` caption, through+`onTurnSettled()`.++**Hold to talk swaps the dictation page's mode, not the orb's.** `browsentic/pushToTalk` in+`storage.local` is a preference, so it is remembered. With it on, the background opens+`dictation.html?hold`, which holds no microphone until a `talk` command arrives — relayed from the+orb only for the tab being listened for — and on `talk: false` calls `stop()` rather than `abort()`,+so Chrome finishes the words already spoken; anything still interim is reported as final, then the+page says `held`. The orb sends on that `held`, or after 2.5 s, whichever comes first. A page left+in the other mode is closed and opened afresh, which is how the toggle takes effect.++The key is `event.code === 'ControlLeft'` (`HOLD_KEY_CODE`) — a physical position, so layout-proof,+present on every platform, and inert on its own everywhere. Because Control is a modifier, a hold+engages only after 250 ms with nothing else pressed, and any other key, a pointer press, a wheel, a+window blur or the tab hiding cancels it and discards what was heard. Only trusted events count: a+page that synthesizes a keydown would otherwise be opening the user's microphone. The listeners are+on the top page's window, since the content script runs in no other frame; an orb whose host the page+removed tears its listeners down on the next event instead of answering beside its replacement.++**Approvals are the orb's to show.** The view carries the waiting approval, the orb turns ember, and+a press opens the same Allow / Deny / Always choice as the timeline, placed on whichever side of the+orb it fits and pointing at it. An approval in a tab nobody is looking at raises a toast on the page+that is, or an OS notification when the browser is in the background.++**A-Eye skips every overlay.** The rail, the toast and the orb carry `data-browsentic-overlay`+(`src/lib/overlay.ts`). A closed shadow root retargets a hit to its host, so the lens sees the host+element under the pointer; it draws no box there and a click there picks nothing, whether the pick+is the user's or the agent's. An orb that starts a pick also hides until the pick is over.++State: `browsentic/handsFree` in `storage.session` (on while present, with `muted` and `since`),+`browsentic/dictation` in `storage.session`, and in `storage.local` `browsentic/speechService`,+`browsentic/pushToTalk` and `browsentic/orbPosition` — the orb's centre as viewport fractions, so one drag places it on every tab. Hands-free is session state+on purpose: a restarted browser comes back with the panel, not a live microphone.+## Tab scopingA panel conversation is **bound to the tab it started in**.diff --git a/docs/internals/state.md b/docs/internals/state.mdindex 7291bb7..e1fe158 100644--- a/docs/internals/state.md+++ b/docs/internals/state.md@@ -58,7 +58,11 @@ leaves history. The daemon holds a copy only while the analyst reads it, in theworkspace at `0600`, and deletes it when the report is in.**Tab sessions** live in `browser.storage.session` under `browsentic/tabSessions`, so they are gone-when the browser closes. So do **diagnostics buffers** (`browsentic/diagnostics`), monitors and+when the browser closes. So does **hands-free mode** (`browsentic/handsFree`, and the listening+state under `browsentic/dictation`) — a restarted browser never comes back with a microphone on.+Only where the orb was dragged to (`browsentic/orbPosition`), whether hold-to-talk is on+(`browsentic/pushToTalk`) and whether this browser's speech service has ever worked+(`browsentic/speechService`) are kept in `storage.local`. So do **diagnostics buffers** (`browsentic/diagnostics`), monitors andtimers — none of what a page reported about itself outlives the browser that reported it.## Relocating