Skip to content

An agentic browser harness: an AI side panel & orb in your own logged-in Chrome, Edge, Arc, Brave or even Firefox run by the agent/subscription you already use (Claude Code, Codex, Antigravity). Free, open source, no API key. Also an MCP server.

docs(captcha): what gets answered and what comes back to you, the flag on page snapshots, and the one-shot analyst behind each image round

Shahriar14a50974881d
diff --git a/docs/guide/approvals.md b/docs/guide/approvals.md
index b80d788..29834a3 100644
--- a/docs/guide/approvals.md
+++ b/docs/guide/approvals.md
@@ -53,7 +53,7 @@ depend on declaration order.
| `file-upload` | `page_attachFile` — putting one of your files into a page | confirm |
| `file-download` | `page_captureDownload` — letting a page write a file to your disk | confirm |
| `leaves-pinned-tab` | Moving to a tab the run was not pointed at | confirm |
-| `captcha-solve` | `page_solveCaptcha` — ticking a site's "I am a human" box | confirm |
+| `captcha-solve` | `page_solveCaptcha` — ticking a site's "I am a human" box and answering its image challenge; one yes covers the rest of the run | confirm |
| `code-injection` | `page_injectCode` — installing JavaScript the agent wrote into the page | confirm |
| `external-code-injection` | `page_injectCode` called by an MCP client rather than the side panel | **deny** |
| `external-code-execution` | `page_runCode` called by an MCP client rather than the side panel | **deny** |
diff --git a/docs/guide/features/captcha.md b/docs/guide/features/captcha.md
index 17b54ab..e35117f 100644
--- a/docs/guide/features/captcha.md
+++ b/docs/guide/features/captcha.md
@@ -1,59 +1,95 @@
# Captchas
-What Browsentic does at a "verify you are human" block, and — more importantly — what it does not.
+What Browsentic does at a "verify you are human" block.
---
## The short version
-It will **tick the checkbox**, because that is a click you could have made yourself and it needs a
-real browser-level one to land.
+It **ticks the checkbox** with a real browser-level click, and it **sees an image challenge
+through**: each round, a short-lived vision session looks at the grid, answers, and is closed.
+Only the puzzles it cannot read — Arkose, AWS WAF, drag-to-fit — come back to you.
-It will **not attempt a challenge**. An image grid, an Arkose puzzle, an AWS WAF challenge — those
-come back to you with the widget's location, and the run waits.
+It finds the widget wherever the page put it: inside closed shadow roots, inside cross-origin
+iframes, and inside frames nested within frames.
-Ticking the checkbox is [gated for approval](../approvals.md) by default, because it acts on
-another site's security control.
+Answering a captcha is [gated for approval](../approvals.md) by default, because it acts on
+another site's security control. One approval covers every round of that captcha in the same run.
---
-## Why ordinary targeting cannot see a captcha
+## How the agent knows there is one
-A captcha is the one thing on a page that `page_getPageInfo` reports nothing about. Vendors build
-the widget as a closed shadow root holding a cross-origin iframe holding another shadow root, so
-there is no element for `page_clickElement` to aim at, even though you can plainly see the checkbox.
+`page_getPageInfo` carries a `captcha` field whenever the page holds a captcha that needs
+answering — the vendor, whether it is already `solved`, and the next step. It is read without
+Chrome's debugger, from every frame the browser knows of, so it costs nothing on pages without
+one and is simply absent there. Invisible scoring captchas (reCAPTCHA v3 and the like) are left
+out, because there is nothing to answer.
-The signs you have hit one: a page that never finishes ("Checking your browser", "Just a moment…"),
-a form that refuses without saying why, a visible checkbox no snapshot lists, or a
-`TARGET_NOT_FOUND` for something obviously on screen.
+That field is what makes the agent act: it calls `page_solveCaptcha` as soon as it sees it,
+rather than hunting for a checkbox no selector can reach.
---
## The two tools
-**`page_findCaptcha`** identifies what is blocking you, reading through the shadow roots and frames
-the widget hides in. Read-only — it never clicks anything.
+**`page_findCaptcha`** reports what is there without touching it: the vendor, how many frames
+deep it sits, its on-screen bounds, whether it is satisfied and — while a challenge is open —
+its prompt and grid.
-Recognised: Cloudflare Turnstile, reCAPTCHA v2, reCAPTCHA v3, hCaptcha, GeeTest, Arkose FunCaptcha,
-AWS WAF Captcha.
+Recognised: Cloudflare Turnstile (including the full-page interstitial), reCAPTCHA v2 and v3,
+hCaptcha, GeeTest, Arkose FunCaptcha, AWS WAF Captcha.
-**`page_solveCaptcha`** ticks the checkbox with a real browser-level click and waits for the verdict.
-Three outcomes:
+**`page_solveCaptcha`** gets past it:
+
+1. Finds the widget and waits out one that is still loading (Turnstile's "Verifying…").
+2. Scrolls it into view if it is below the fold, and ticks the checkbox.
+3. If the vendor answers with an image challenge, answers it round by round until the widget
+ is satisfied.
| State | Meaning |
| --- | --- |
| `solved` | The widget accepted it. Carry on |
-| `needsHuman` | The vendor escalated to a challenge a person has to answer. The widget's bounds come back so it can be screenshotted for you; nothing is attempted |
-| `invisible` | A scoring captcha with nothing to click — reCAPTCHA v3 and friends |
+| `challenge` | An image challenge is still open. The result carries it as an image for the caller to answer |
+| `pending` | The widget took the click but had not settled inside the wait |
+| `needsHuman` | A puzzle Browsentic does not answer. Its bounds come back so it can be shown to you |
+| `invisible` | A scoring captcha with nothing to click |
Chrome only. It shows the debugger bar while it runs.
---
+## Image challenges
+
+The extension photographs the challenge and hands it to the daemon, which starts a one-shot
+session of your configured agent to look at that single picture. The session answers and is
+stopped at once; nothing carries over to the next round. The extension then clicks the answer
+with real pointer movement, presses the challenge's own button, and photographs whatever comes
+next.
+
+Two kinds are answered:
+
+- **Tile grids** — reCAPTCHA's 3×3 and 4×4, including the "click verify once there are none
+ left" grids that swap each picked tile for a new picture. Each tile's number is painted on the
+ photograph, and the photograph is taken only once every tile has finished fading in.
+- **Pictures to tap** — hCaptcha's canvas challenges ("click the animal icon that is
+ different"). The answer is a set of points on the photograph.
+
+The analyst needs an agent that can look at images; today that is Claude Code. With another
+agent, or when the analyst gives up (ten rounds, or the time budget), the result comes back as
+`state: "challenge"` with the picture attached, and the agent driving the run answers it itself
+by calling `page_solveCaptcha` again with `tiles`, `points` or `reload`.
+
+A vendor can keep setting new challenges to a browser it distrusts however well each one is
+answered — an automated or headless browser especially. That is the vendor's verdict, not a
+wrong answer.
+
+---
+
## If it is not a captcha
-`CAPTCHA_NOT_FOUND` means there is no known widget on the page, so whatever is blocking the run is
-something else — a rate limit, a login wall, a slow request. Worth knowing rather than guessing.
+`CAPTCHA_NOT_FOUND` means there is no known widget on the page, in any frame, so whatever is
+blocking the run is something else — a rate limit, a login wall, a slow request.
---
diff --git a/docs/guide/troubleshooting.md b/docs/guide/troubleshooting.md
index 6bc3c88..e7e203f 100644
--- a/docs/guide/troubleshooting.md
+++ b/docs/guide/troubleshooting.md
@@ -75,7 +75,8 @@ Those three answer most questions. The daemon log also lives at `~/.browsentic/d
| --- | --- | --- |
| `TAB_UNREACHABLE` on a normal site | The extension needs reloading | ↻ at `chrome://extensions`; ordinary sites otherwise self-heal |
| `TAB_UNREACHABLE` on `chrome://`, the Web Store, the new-tab page | Those pages cannot host a content script | `page_navigate` to an http(s) page — it still works there |
-| `TARGET_NOT_FOUND` for something clearly on screen | The page changed since the snapshot, or it is inside a captcha widget's shadow root | Re-snapshot with `page_getPageInfo`; for a captcha use [`page_findCaptcha`](features/captcha.md) |
+| `TARGET_NOT_FOUND` for something clearly on screen | The page changed since the snapshot, or it is inside a captcha widget's shadow root | Re-snapshot with `page_getPageInfo`; for a captcha use [`page_solveCaptcha`](features/captcha.md) |
+| A captcha keeps setting new image challenges | The vendor distrusts the browser, however well each round is answered — common with automated or headless browsers | Solve one round yourself in the page; a person's answer usually clears the distrust. See [Captchas](features/captcha.md#image-challenges) |
| `DEBUGGER_UNAVAILABLE` | DevTools is open on that tab | Close DevTools, or use `page_clickElement` instead of `page_trustedClick` |
| `RUN_IN_PROGRESS` | One instruction at a time per tab | Cancel the running one, or use another tab |
| `TAB_IN_USE` | That tab belongs to another Browsentic conversation | Switch to it from the Sessions strip |
diff --git a/docs/internals/agent-runs.md b/docs/internals/agent-runs.md
index 0da9037..11428e1 100644
--- a/docs/internals/agent-runs.md
+++ b/docs/internals/agent-runs.md
@@ -223,6 +223,46 @@ agent with the next message.
---
+## The captcha analyst
+
+`page.solveCaptcha` is the one action the daemon keeps a hand in after the extension has answered.
+`invokeOn` routes it through `solveCaptchaWithAnalyst` in
+[`captcha-solver.ts`](../../src/daemon/agent/captcha-solver.ts), for side-panel runs and MCP
+clients alike:
+
+```
+caller ──solveCaptcha {}──► daemon ──► extension tick the checkbox; a challenge opens
+ ◄── state: challenge + photo of the grid
+ answerChallenge ──runAgentJson (task mode, Read only, effort low)
+ ──► extension solveCaptcha { tiles | points | reload }
+ ◄── next round … or state: solved
+caller ◄── solved, analystRounds: n
+```
+
+1. **One session per round.** `answerChallenge()` in
+ [`captcha-analyst.ts`](../../src/daemon/agent/captcha-analyst.ts) writes the round's JPEG to the
+ task workspace at `0600`, asks for one JSON object after `=== ANSWER ===`, and stops the CLI the
+ moment `readAnswer()` accepts its output. The file is deleted in `finally`; nothing carries into
+ the next round. A tile outside the grid or a point off the picture makes the whole answer
+ unusable rather than half-clicked.
+2. **Only where it can see.** It runs when the active runner lists `image` in `Runner.opens` —
+ Claude Code today — at effort `low` whatever the conversation uses.
+3. **Handing back.** An answer the caller supplies itself is passed straight through. When the
+ analyst returns nothing, hits ten rounds, or the budget (`timeoutMs`, 120 s by default, shared
+ by every round) runs low, the open challenge goes back to the caller with its photo, which
+ `server.ts` renders as an MCP image block, and a note that it is now theirs to answer.
+
+The extension side is [`captcha.ts`](../../src/lib/bridge/captcha.ts) on top of
+[`frame-graph.ts`](../../src/lib/bridge/frame-graph.ts): every frame in the tab through
+`Page.getFrameTree` on the root and each auto-attached out-of-process session, a frame's viewport
+origin composed through `DOM.getFrameOwner` of each ancestor, and `DOM.getDocument` with `pierce`
+plus `DOM.querySelectorAll` per shadow root to reach inside closed ones. Page code runs in an
+isolated world per frame. A challenge is photographed only once every tile has finished swapping
+and fading in — reCAPTCHA reports a tile ready about a second before its new picture arrives, so
+the wait is for each clicked tile's picture to change, not for the loading class to go.
+
+---
+
## Skill routing
Skills are markdown with YAML-ish front matter, loaded from three directories, later shadowing
diff --git a/docs/internals/guardrails.md b/docs/internals/guardrails.md
index 2c39c2f..43c40b1 100644
--- a/docs/internals/guardrails.md
+++ b/docs/internals/guardrails.md
@@ -397,8 +397,9 @@ one is **visible in the log rather than assumed away**.
### Two spawn modes
-`run` drives the browser. `task` is a one-shot — the file analyst reading an attached file, turning
-a raw recording trace into steps — that must not reach the browser at all, which is asserted by
+`run` drives the browser. `task` is a one-shot — the file analyst reading an attached file, the
+captcha analyst looking at one round of a challenge, turning a raw recording trace into steps — that
+must not reach the browser at all, which is asserted by
requiring `{"mcpServers":{}}` (or `mcp_servers={}`, or Grok's `--deny MCPTool`) in its argv. `Read`
is deliberately left out of `task`'s deny list, because some tasks are handed a file in the scratch
workspace.
diff --git a/docs/reference/tools.md b/docs/reference/tools.md
index 1020130..5d73f47 100644
--- a/docs/reference/tools.md
+++ b/docs/reference/tools.md
@@ -93,6 +93,12 @@ how many links, buttons and fields its subtree holds and ends with its selector,
`page_extractText` to that region, and `interactive.counts` reports the true
totals before `maxPerKind` truncates the lists.
+When the page holds a captcha that needs answering, the result carries a `captcha` field —
+`vendor`, `label`, `solved`, and while unsolved a `next` step pointing at
+[`page_solveCaptcha`](#page_solvecaptcha). It is read from every frame, including ones inside
+closed shadow roots, without Chrome's debugger, and it is left out when there is none or only an
+invisible scoring captcha.
+
| Parameter | Type | Default | Purpose |
| --- | --- | --- | --- |
| `maxPerKind` | integer | `30` | Cap on links, buttons, fields, and forms listed per kind |
@@ -291,8 +297,8 @@ clicked.
Report what captcha is on the page, without touching it. Ordinary targeting cannot see one:
vendors build the widget as a closed shadow root holding a cross-origin iframe holding another
-shadow root, so `page_getPageInfo` shows nothing where the checkbox visibly is. This reads through
-all of it with Chrome's debugger.
+shadow root, often inside further frames. This reads through all of it with Chrome's debugger.
+To get past one, call [`page_solveCaptcha`](#page_solvecaptcha) directly; this is for re-checking.
Recognises Cloudflare Turnstile (including the full-page interstitial), reCAPTCHA v2 and v3,
hCaptcha, GeeTest, Arkose FunCaptcha and AWS WAF. Takes no parameters.
@@ -302,34 +308,50 @@ hCaptcha, GeeTest, Arkose FunCaptcha and AWS WAF. Takes no parameters.
| `found` | boolean | Whether any known captcha is on the page |
| `vendor`, `label` | string | Which one, e.g. `turnstile` / "Cloudflare Turnstile" |
| `kind` | `"checkbox"` \| `"interactive"` \| `"invisible"` | What the widget asks for |
-| `state` | `"idle"` \| `"pending"` \| `"solved"` \| `"needsHuman"` \| `"invisible"` | Where it has got to |
+| `state` | `"idle"` \| `"loading"` \| `"hidden"` \| `"pending"` \| `"solved"` \| `"challenge"` \| `"needsHuman"` \| `"invisible"` | Where it has got to |
| `solved`, `hasToken` | boolean | Whether it is satisfied, and whether the response field is filled |
-| `bounds` | `{ x, y, width, height }` | The widget's box in viewport coordinates, for a screenshot |
-| `point` | `{ x, y }` | Where its checkbox is, composed across the frame boundary. Absent when there is nothing to click |
+| `frameDepth` | integer | How many frames deep the widget sits, when it is inside any |
+| `bounds` | `{ x, y, width, height }` | The widget's box — or the open challenge's — in viewport coordinates |
+| `point` | `{ x, y }` | Where its checkbox is, composed across every frame boundary. Absent when there is nothing to click |
+| `challenge` | object | While an image challenge is open: its `kind`, `prompt`, and for a grid `rows`, `columns`, `tiles`, `selected` and `dynamic` |
| `note` | string | Why, when there is no `point` or the state needs explaining |
Read-only, so it is allowed during a site-mapping run.
+`page_getPageInfo` reports the same thing more cheaply as its `captcha` field — vendor, `solved`
+and a `next` step — read from every frame without the debugger, and absent when there is none.
+
### page_solveCaptcha
-Tick a captcha's "I am a human" checkbox with a real browser-level click and wait for the widget to
-settle. The checkbox only responds to genuine pointer input, which is the point of it, so no
-synthetic click reaches it.
+Get past a captcha. It finds the widget in any frame at any depth, waits out one that is still
+loading, scrolls it into view, ticks the checkbox with a real browser-level click, and sees an
+image challenge through.
-**Confirm-gated.** Ticking another site's human check is the user's decision; an external MCP
-client with no approval channel gets `DECLINED` under the default `unattended` policy.
+**Confirm-gated.** Answering another site's human check is the user's decision. One approval
+covers every call of this tool for the rest of that run, since a challenge takes a call per
+round. An external MCP client with no approval channel gets `DECLINED` under the default
+`unattended` policy.
-When the vendor escalates to a challenge a person has to answer — an image grid, Arkose, AWS WAF —
-it returns `state: "needsHuman"` with the widget `bounds` and **does not attempt the challenge**.
-Screenshot that region, tell the user, and poll `page_findCaptcha` until they have solved it.
+**Image challenges.** When the configured agent can read images (Claude Code), the daemon answers
+each round with a one-shot vision session that is stopped as soon as it has answered, and the call
+returns only once the widget is satisfied or the analyst gives up. Otherwise — or after ten rounds,
+or when the budget runs out — the result has `state: "challenge"` and carries the challenge as an
+image: an MCP image block beside the JSON. Answer it by calling this tool again with `tiles`,
+`points` or `reload`, and repeat until `solved`.
| Parameter | Type | Default | Purpose |
| --- | --- | --- | --- |
-| `waitMs` | integer 0–120000 | `20000` | How long to wait after clicking for the widget to report a verdict |
-| `timeoutMs` | integer 1000–180000 | `60000` | Overall budget for the attempt, including finding the widget |
-
-Returns the same fields as `page_findCaptcha`, plus `clicked`. Fails with `CAPTCHA_NOT_FOUND` when
-there is no widget to act on.
+| `tiles` | array of integers 1–25 | — | Answer to a `kind: "tiles"` challenge: every matching tile, numbered as painted on the image (1 top-left, along each row). The whole selection; `[]` for none |
+| `points` | array of `{ x, y }`, 1–16 | — | Answer to a `kind: "points"` challenge: where to tap, in pixels on the image as delivered |
+| `reload` | boolean | — | Swap the open challenge for another instead of answering it |
+| `waitMs` | integer 0–120000 | `20000` | How long to wait after each click for the widget to report a verdict |
+| `timeoutMs` | integer 1000–300000 | `120000` | Overall budget for the attempt, image rounds included |
+
+Returns the same fields as `page_findCaptcha`, plus `clicked` and, when the analyst ran,
+`analystRounds`. While a challenge is open, `challenge` adds `image`, `imageWidth`,
+`imageHeight`, `errors` (the vendor refused the last answer) and, for a grid that swaps picked
+tiles, `fresh` (the tiles that just changed). Fails with `CAPTCHA_NOT_FOUND` when there is no
+widget to act on, and with `INVALID_INPUT` for an answer that does not fit the open challenge.
### page_hoverElement