Skip to content

An agentic browser harness: an AI side panel & orb in your own logged-in Chrome, Edge, Arc, Brave or even Firefox run by the agent/subscription you already use (Claude Code, Codex, Antigravity). Free, open source, no API key. Also an MCP server.

docs(hands-free): the feature page, which browsers get it, why left Control, how it works inside the extension, and what to do when it will not listen

Shahriarac4093ae4e3d
diff --git a/docs/guide/features/README.md b/docs/guide/features/README.md
index c0ecb38..e6aa03d 100644
--- a/docs/guide/features/README.md
+++ b/docs/guide/features/README.md
@@ -10,6 +10,7 @@ first.
| | |
| --- | --- |
| [Conversations](conversations.md) | Voice and text, one conversation per tab, the sessions strip, history |
+| [Hands-free](hands-free.md) | Put the panel away and talk to the page through a mic you can drag anywhere |
| [Instant commands](instant-commands.md) | "go back", "scroll to the top" — the things that run in milliseconds without waking an agent |
| [A-Eye](a-eye.md) | Point at an element instead of describing it, and let it point back when it needs to |
| [Skills](skills.md) | How an instruction gets routed to the right playbook, and how to write your own |
diff --git a/docs/guide/features/a-eye.md b/docs/guide/features/a-eye.md
index ee5ce55..b7dd12f 100644
--- a/docs/guide/features/a-eye.md
+++ b/docs/guide/features/a-eye.md
@@ -50,6 +50,9 @@ agent is told to ask in words instead, not to ask you to point again.
- **One lens at a time.** Starting a new pick — from the button, or the agent asking — dismisses a
lens already waiting, and whoever opened that one is told it was cancelled.
- **Top frame only.** An element inside an iframe picks the iframe, not what is in it.
+- **Never Browsentic's own.** The [hands-free](hands-free.md) mic, the minimized rail and
+ Browsentic's cards are skipped: the lens draws nothing over them, and a click on one picks
+ nothing. A pick started from the hands-free mic's menu has the mic step aside until it is done.
- **The element, not the page.** The photograph covers what you picked, with a sliver of margin.
For the page around it, ask for a [screenshot](screenshots.md) as well.
- **Long elements are cut.** Picking a whole article sends the first couple of thousand characters;
diff --git a/docs/guide/features/conversations.md b/docs/guide/features/conversations.md
index c8524d6..fdaa9d2 100644
--- a/docs/guide/features/conversations.md
+++ b/docs/guide/features/conversations.md
@@ -6,13 +6,14 @@ How you talk to Browsentic, and how it keeps track of several things at once.
## Voice and text
-Three ways in:
+Four ways in:
| | |
| --- | --- |
| **Type** | The composer at the bottom of the side panel |
| **Dictate** | Hands-free in the side panel — press the mic and talk |
| **Press to talk** | In the popup, for when the panel is not open |
+| **Hands-free** | The panel put away, and a mic on the page — see [Hands-free](hands-free.md) |
Speech uses the browser's built-in recognition. Nothing is bundled or downloaded — which in Chrome
means audio is streamed to Google for transcription. Type instead if that matters to you; see
@@ -66,6 +67,9 @@ and closes it instead.
The rail belongs to the minimized state alone. Closing the panel with the browser's own close
button removes any rail from every page — only the header's collapse button leaves one behind.
+The header's detach button, beside collapse, puts the panel away differently: instead of the rail,
+a microphone on the page you talk to directly. See [Hands-free](hands-free.md).
+
### Knowing something is running when you are elsewhere
While a conversation is working, its tab is marked in two places — a dot on the Browsentic toolbar
diff --git a/docs/guide/features/hands-free.md b/docs/guide/features/hands-free.md
new file mode 100644
index 0000000..faed787
--- /dev/null
+++ b/docs/guide/features/hands-free.md
@@ -0,0 +1,129 @@
+# Hands-free
+
+Put the side panel away and talk to the page. A microphone floats at the bottom of the page instead,
+and everything you would reach for in the panel is a hover away.
+
+Hands-free exists only in browsers that can turn speech into text — Chrome, Edge, and the other
+Chromium browsers with a speech service behind them. Anywhere else there is no detach button, no
+`/hands-free` and no mic on the page (see [Which browsers](#which-browsers)).
+
+---
+
+## Turning it on
+
+Either of:
+
+- the **detach** button in the side panel's header, just left of minimize
+- typing **`/hands-free`** in the composer — the `/` menu offers it as soon as you type `/hand`, and
+ `/hand-free` works too
+
+The panel folds down into a mic and closes, and the same mic rises at the bottom middle of the page.
+It follows you from tab to tab. Opening the side panel again — from the mic's own menu, the toolbar
+icon or the right-click **Open Browsentic** item — turns hands-free off; the panel and the mic never
+listen at the same time.
+
+Hands-free lasts until you open the panel or the browser closes. A browser that restarts comes back
+with the panel, never with a microphone already on.
+
+## Talking
+
+Just speak. What you say streams in above the mic word by word, and the mic's edge pulses while it
+hears you. Pause, and a ring runs once around the mic — when it closes, the instruction is sent.
+
+| Press the mic while… | It… |
+| --- | --- |
+| the ring is running, or you are mid-sentence | throws away what you said, so you can say it again |
+| the agent is working (a turning cyan ring) | stops the run |
+| the agent is asking permission (an orange shield) | opens the request, pointing at the mic |
+| it is listening | mutes it; press again to listen |
+| it needs the microphone | opens the tab where the browser can ask for it |
+
+When a run finishes, its answer streams in above the mic the same way, and fades after a moment.
+
+The mic listens for **the tab in front of you**, and only while the browser window is focused. Switch
+tabs and it follows; switch to another app and it stops until you come back. It also stops while the
+agent is working, and while a side panel is open anywhere.
+
+## The menu
+
+Rest the pointer on the mic for a second and five buttons fan out around it:
+
+| | |
+| --- | --- |
+| **Attach a file** | Hands a file to this tab's conversation, as dropping it on the panel does — see [Files](files.md) |
+| **Live code** | Lets the agent write a script for this page, as the composer's code toggle does; you approve the code before it runs |
+| **Focus point** | Opens [A-Eye](a-eye.md): point at an element and it rides along with your next instruction |
+| **Hold to talk** | Switches between listening all the time and listening only while you hold a key — see below |
+| **Open the side panel** | Brings the panel back and ends hands-free |
+
+What rides along — the element, live code, attached files — shows as chips beside the mic until you
+next speak. Each has a **×**.
+
+## Hold to talk
+
+Switch it on from the menu and the mic stops listening on its own. Hold **left Control** — **control**
+on a Mac — and speak; let go, and the mic mutes and sends what you said at once, with no countdown.
+The key's name shows on the mic while this is on, and the mic grows and turns pink while you hold it.
+The setting is remembered.
+
+Why that key: it sits at the far left on Mac and PC keyboards alike, it is the same physical key on
+every layout — QWERTY, AZERTY, Dvorak — and on its own it does nothing in any browser or system.
+
+Control is also the key for shortcuts, so a hold only counts if nothing else joins it. Tap it, or
+press another key with it (Ctrl + C), click or scroll while it is down, and the mic never opens; if
+it had already opened, what it heard is thrown away rather than sent. A key press a web page makes
+up is ignored — only you can open the mic.
+
+## Moving it
+
+Press and hold the mic until it grows a little, then drag it anywhere. It stays where you drop it on
+every tab. Against a side or a corner, the menu turns to open away from the edge and deals its
+buttons out in that turned order; captions and permission requests move to whichever side has room.
+
+## When the agent needs your OK
+
+The mic turns orange and says what the agent is asking to do. Press it for the request — the same
+**Allow**, **Deny** and **Always on ‹host›** the panel offers, or the code itself when the agent
+wrote some. If the request comes from a tab you are not looking at, a card appears on the page you
+are on, or a system notification when the browser is in the background; clicking it takes you there.
+
+## A-Eye never lands on the mic
+
+The mic, like the rail and Browsentic's cards, is marked as Browsentic's own. A-Eye — yours or the
+agent's — never outlines it and never picks it; a click on the mic while A-Eye is up does nothing.
+When you start A-Eye from the mic's menu, the mic steps aside until you have picked.
+
+## Which browsers
+
+Browsentic offers hands-free only where it can actually hear you:
+
+| | |
+| --- | --- |
+| **Chrome, Edge** | Offered |
+| **Firefox** | Never offered — it has no speech recognition, and no hidden page to listen from |
+| **Brave** | Never offered — it ships the speech API with no service behind it, so every attempt fails |
+| **Other Chromium browsers** | Offered until their speech service first fails without ever having transcribed a word. From then on the detach button and `/hands-free` are gone, and a mic already on the page leaves with a note saying why |
+
+A browser whose speech service has transcribed even once keeps hands-free for good: a later failure
+there is a dropped connection, not a missing service. And one written off can win it back — dictate
+in the panel, and the first word it transcribes brings the detach button back.
+
+## Limits
+
+- **Speech goes to Google** while hands-free is listening, as it does for the panel's dictation —
+ see [Limits](../limits.md#speech-goes-to-google).
+- **One page at a time can listen.** Chrome runs one speech recognizer at once. If another page
+ starts listening, the mic lets it have the microphone and says so; press it to take it back.
+- **Background noise can become an instruction.** Anything that sounds like speech is transcribed.
+ The ring before sending is your chance to press and throw it away; mute the mic when you are not
+ using it.
+- **Hold to talk needs the page itself to have the keyboard.** While you are typing inside a frame
+ embedded in the page — some rich editors, Google Docs among them — the key goes to that frame and
+ the mic does not hear it. Click the page around it first.
+- **No mic where Browsentic is not allowed** — `chrome://` pages, the Chrome Web Store, the new tab
+ page. The mic stops listening there, and the toolbar icon is the way back to the panel.
+
+## See also
+
+- [Conversations](conversations.md) — typing and dictating in the panel, and minimizing it to the rail
+- [Approvals](../approvals.md) — what asks first, and why
diff --git a/docs/guide/limits.md b/docs/guide/limits.md
index 38ae6dc..7d0d47c 100644
--- a/docs/guide/limits.md
+++ b/docs/guide/limits.md
@@ -108,6 +108,10 @@ Voice input uses Chrome's built-in Web Speech API, which streams audio to Google
No model is bundled and nothing is downloaded. If that is not acceptable, type instead — it is one
file to replace the speech engine.
+[Hands-free](features/hands-free.md) listens the same way, and for as long as it is on and the
+browser is in front — not only while you are talking to it. Switch its **Hold to talk** on and it
+listens only while you hold the key.
+
## Recording and mapping limits
**Recordings** run for at most 15 minutes, follow one tab, live in extension storage rather than on
diff --git a/docs/guide/troubleshooting.md b/docs/guide/troubleshooting.md
index 38268d2..da6fb94 100644
--- a/docs/guide/troubleshooting.md
+++ b/docs/guide/troubleshooting.md
@@ -27,6 +27,10 @@ Those three answer most questions. The daemon log also lives at `~/.browsentic/d
| "Browsentic has not been given the microphone yet" | A browser cannot show its microphone prompt inside a side panel or a popup, so a fresh install has never been asked | Press **Allow microphone** and choose **Allow** in the tab that opens |
| "Microphone access is blocked" | The microphone was refused for the extension | Press **Allow microphone**, then click the icon at the left of that tab's address bar and set **Microphone** to **Allow** |
| "This browser has no speech service" | Brave and some Chromium builds ship speech recognition without a transcription service behind it | Type instead, or use Chrome or Edge for dictation |
+| No detach button in the panel's header, and no `/hands-free` | Hands-free is offered only where speech can be transcribed — never in Firefox or Brave, and not in a browser whose speech service has failed before it ever worked | Use Chrome or Edge. If the service was only out of reach, dictate in the panel once: the first word it transcribes brings the button back |
+| "Hands-free is not available here", and the mic left the page | The speech service failed before it had ever transcribed a word in this browser | As above. Open Browsentic from the toolbar to carry on by typing |
+| The hands-free mic turns amber and says another page took the microphone | Chrome runs one speech recognizer at a time, and another page or the panel started one | Press the mic to take it back |
+| Holding Control does nothing | Hold to talk is off; the key went to a frame embedded in the page (some editors); another key or a click joined it; or the mic is muted | Turn **Hold to talk** on from the mic's menu, click the page outside the embedded editor, and hold left Control on its own |
| `EXTENSION_OFFLINE` | Browser closed, or not paired | Open the browser; `browsentic sessions` to check pairing |
| "Load unpacked" cannot see `~/browsentic` | A Flatpak or Snap browser, sandboxed away from your home directory. Snap Chromium is the Ubuntu default | Grant it: `flatpak override --user --filesystem=~/browsentic com.google.Chrome`. Or install somewhere the sandbox can read: `browsentic setup --dir ~/snap/chromium/common/browsentic-extension` |
| The folder picker does not show `~/browsentic` | It is there; some pickers open elsewhere by default | macOS: press ⇧⌘G and paste the path. Linux: Ctrl+L |
diff --git a/docs/internals/extension.md b/docs/internals/extension.md
index f207bdb..7a1bb55 100644
--- a/docs/internals/extension.md
+++ b/docs/internals/extension.md
@@ -35,8 +35,8 @@ microphone to an extension. `use-speech.ts` therefore reads `navigator.permissio
`prompt` it never calls `getUserMedia` — repeated silent refusals would earn the origin a temporary
block — and offers **Allow microphone** instead. That opens the unlisted `mic-permission.html`
entrypoint in a tab, where the prompt can appear. The grant belongs to the extension's origin, so the
-panel and the popup both see the `PermissionStatus` change and start listening without a reload.
-Firefox prompts from its sidebar on its own and skips the check.
+panel, the popup and hands-free mode's offscreen page all see the `PermissionStatus` change and start
+listening without a reload. Firefox prompts from its sidebar on its own and skips the check.
## The Firefox build is signed, and says who it is
@@ -54,8 +54,8 @@ a `browser_specific_settings.gecko` block the Chrome build has no use for:
behind whichever agent CLI the user runs.
- **`strict_min_version: 140.0`** — the first Firefox that understands the data-collection key.
-Two permissions are filtered out of that build, `sidePanel` and `debugger`: Firefox has neither, and
-AMO's validator flags each name it does not know. The Firefox sidebar is `sidebar_action`, which WXT
+Three permissions are filtered out of that build, `sidePanel`, `debugger` and `offscreen`: Firefox has
+none of them, and AMO's validator flags each name it does not know. The Firefox sidebar is `sidebar_action`, which WXT
derives from the same entrypoint, and the nine tools that need the debugger are left off the list a
Firefox build offers (see [Background vs content script](#background-vs-content-script)).
@@ -187,6 +187,94 @@ Pages that refuse content scripts — `chrome://`, the Web Store, the new-tab pa
That is the same `TAB_UNREACHABLE` set as everywhere else, and it is why the toolbar icon and the
**Open/Close Browsentic** context-menu item stay the guaranteed way back in.
+## Hands-free: the orb lives in the page
+
+Detaching is the rail's sibling: the panel closes, and a microphone orb is drawn into the page
+instead, which the user talks to. The panel cannot be shrunk, so the fold the user sees — the panel
+collapsing into a mic that drops away — is `DetachVeil` playing for 560 ms before `closeSidePanel()`,
+and the page's orb rising once the page has widened. Firefox closes its sidebar only inside the
+click's gesture, so it skips the fold.
+
+| | |
+| --- | --- |
+| `src/lib/handsfree/events.ts` | The channel, `OrbView`, the orb's requests, the dictation reports, and the Lucide paths |
+| `src/lib/handsfree/host.ts` + `styles.ts` | `exposeHandsFree()` — the orb, its arc menu, caption, countdown ring and approval card, in a closed shadow root |
+| `src/lib/handsfree/geometry.ts` | Pure layout: which way the arc opens from the edges the orb is against, and which side a caption or card fits on |
+| `src/lib/bridge/hands-free.ts` | `serveHandsFree()` — when the microphone listens and for which tab, what every orb shows, and the orb's requests |
+| `src/extension/entrypoints/dictation/` | The offscreen page speech recognition runs in, Chromium only |
+
+**Speech needs a document, and the panel is gone.** The background has no DOM, and recognition in a
+content script would run on the page's origin — a prompt per site, and none at all where the page's
+`Permissions-Policy` forbids the microphone. So recognition runs in an **offscreen document**
+(`dictation.html`, reason `USER_MEDIA`) on the extension's own origin, which the `mic-permission`
+tab already granted. The document exists exactly while the orb should listen: creating it starts the
+mic, closing it stops it, so it needs no command channel of its own. It reports a `DictationPhase`
+and each interim and final transcript; the background keeps the phase in `browsentic/dictation` and
+relays transcripts to the one tab listening.
+
+**Chrome runs one recognizer at a time**, and a second one aborts the first. That is why the panel
+and hands-free never overlap — the panel's `panelOpened` ends hands-free, and the orb does not listen
+while any panel's run port is connected — and why an `aborted` the document did not ask for is read
+as another page taking the mic. The document then stops as `yielded` rather than restarting into a
+tug-of-war; a press on the orb takes the mic back.
+
+**Hands-free only exists where speech does.** `speech-support.ts` decides it in two layers.
+`speechCapable()` is what is knowable without listening: a Firefox build (no offscreen documents, no
+recognizer), no `offscreen` API, no recognizer constructor, or a brand known to ship the recognizer
+without a service behind it (`Brave`, which fails every attempt with `network`). What that lets
+through is settled by the service itself: the panel's dictation and the offscreen page record
+`browsentic/speechService` in `storage.local` — `works` on the first transcript, `missing` on a
+`network` failure, and `works` always wins, since a service that answered once is only out of reach.
+The panel hides the detach button and `/hands-free` behind `useHandsFreeSupported()`, which starts
+hidden so nothing flashes; `paint()` ends hands-free on the spot if it finds itself on anywhere
+unsupported, and a `missing` found mid-session raises a toast saying why the mic left. Note that
+Vitest serves `import.meta.env.FIREFOX` as the string `"false"`, which is truthy — the decision
+itself is the pure `capableOf()` so it can be tested at all.
+
+**The microphone listens for one tab**: the one in front of a focused window, whose conversation has
+no run and no approval waiting, while the link is up and the user has not muted it. Every other orb
+is painted `paused`. `paint()` computes each tab's `OrbView`, sends only the ones that changed since
+that tab was last told, and records which tabs answered — a tab with no orb is never aimed at. Like
+the rail it is a cache the worker can lose: a revived worker repaints everything.
+
+**The orb owns only what dies with the page**: what has been said since the last send, the countdown
+ring (`AUTO_SEND_MS`, the composer's 1.6 s), the A-Eye focus, the live-code toggle and the files
+attached since the last send. An instruction goes through the same `instruct` the panel's run port
+takes, via `runCommand()`, so it lands in the tab's own conversation and takes the fast path as a
+typed one would. A finished turn's last reply or error comes back as a `say` caption, through
+`onTurnSettled()`.
+
+**Hold to talk swaps the dictation page's mode, not the orb's.** `browsentic/pushToTalk` in
+`storage.local` is a preference, so it is remembered. With it on, the background opens
+`dictation.html?hold`, which holds no microphone until a `talk` command arrives — relayed from the
+orb only for the tab being listened for — and on `talk: false` calls `stop()` rather than `abort()`,
+so Chrome finishes the words already spoken; anything still interim is reported as final, then the
+page says `held`. The orb sends on that `held`, or after 2.5 s, whichever comes first. A page left
+in the other mode is closed and opened afresh, which is how the toggle takes effect.
+
+The key is `event.code === 'ControlLeft'` (`HOLD_KEY_CODE`) — a physical position, so layout-proof,
+present on every platform, and inert on its own everywhere. Because Control is a modifier, a hold
+engages only after 250 ms with nothing else pressed, and any other key, a pointer press, a wheel, a
+window blur or the tab hiding cancels it and discards what was heard. Only trusted events count: a
+page that synthesizes a keydown would otherwise be opening the user's microphone. The listeners are
+on the top page's window, since the content script runs in no other frame; an orb whose host the page
+removed tears its listeners down on the next event instead of answering beside its replacement.
+
+**Approvals are the orb's to show.** The view carries the waiting approval, the orb turns ember, and
+a press opens the same Allow / Deny / Always choice as the timeline, placed on whichever side of the
+orb it fits and pointing at it. An approval in a tab nobody is looking at raises a toast on the page
+that is, or an OS notification when the browser is in the background.
+
+**A-Eye skips every overlay.** The rail, the toast and the orb carry `data-browsentic-overlay`
+(`src/lib/overlay.ts`). A closed shadow root retargets a hit to its host, so the lens sees the host
+element under the pointer; it draws no box there and a click there picks nothing, whether the pick
+is the user's or the agent's. An orb that starts a pick also hides until the pick is over.
+
+State: `browsentic/handsFree` in `storage.session` (on while present, with `muted` and `since`),
+`browsentic/dictation` in `storage.session`, and in `storage.local` `browsentic/speechService`,
+`browsentic/pushToTalk` and `browsentic/orbPosition` — the orb's centre as viewport fractions, so one drag places it on every tab. Hands-free is session state
+on purpose: a restarted browser comes back with the panel, not a live microphone.
+
## Tab scoping
A panel conversation is **bound to the tab it started in**.
diff --git a/docs/internals/state.md b/docs/internals/state.md
index 7291bb7..e1fe158 100644
--- a/docs/internals/state.md
+++ b/docs/internals/state.md
@@ -58,7 +58,11 @@ leaves history. The daemon holds a copy only while the analyst reads it, in the
workspace at `0600`, and deletes it when the report is in.
**Tab sessions** live in `browser.storage.session` under `browsentic/tabSessions`, so they are gone
-when the browser closes. So do **diagnostics buffers** (`browsentic/diagnostics`), monitors and
+when the browser closes. So does **hands-free mode** (`browsentic/handsFree`, and the listening
+state under `browsentic/dictation`) — a restarted browser never comes back with a microphone on.
+Only where the orb was dragged to (`browsentic/orbPosition`), whether hold-to-talk is on
+(`browsentic/pushToTalk`) and whether this browser's speech service has ever worked
+(`browsentic/speechService`) are kept in `storage.local`. So do **diagnostics buffers** (`browsentic/diagnostics`), monitors and
timers — none of what a page reported about itself outlives the browser that reported it.
## Relocating