An agentic browser harness: an AI side panel & orb in your own logged-in Chrome, Edge, Arc, Brave or even Firefox run by the agent/subscription you already use (Claude Code, Codex, Antigravity). Free, open source, no API key. Also an MCP server.
TypeScript agentic-browseragentic-browsingagy-cliaiai-agentai-browser-extensionai-toolsbrave-extensionbrowser-agentbrowser-automationbrowser-extensionbrowser-usebrowserskillchrome-extensionclaude-codecodexfirefox-addonmcp-servernanobrowserweb-automation
docs(captcha): what gets answered and what comes back to you, the flag on page snapshots, and the one-shot analyst behind each image round
Shahriar14a50974881d
diff --git a/docs/guide/approvals.md b/docs/guide/approvals.mdindex b80d788..29834a3 100644--- a/docs/guide/approvals.md+++ b/docs/guide/approvals.md@@ -53,7 +53,7 @@ depend on declaration order.| `file-upload` | `page_attachFile` — putting one of your files into a page | confirm || `file-download` | `page_captureDownload` — letting a page write a file to your disk | confirm || `leaves-pinned-tab` | Moving to a tab the run was not pointed at | confirm |-| `captcha-solve` | `page_solveCaptcha` — ticking a site's "I am a human" box | confirm |+| `captcha-solve` | `page_solveCaptcha` — ticking a site's "I am a human" box and answering its image challenge; one yes covers the rest of the run | confirm || `code-injection` | `page_injectCode` — installing JavaScript the agent wrote into the page | confirm || `external-code-injection` | `page_injectCode` called by an MCP client rather than the side panel | **deny** || `external-code-execution` | `page_runCode` called by an MCP client rather than the side panel | **deny** |diff --git a/docs/guide/features/captcha.md b/docs/guide/features/captcha.mdindex 17b54ab..e35117f 100644--- a/docs/guide/features/captcha.md+++ b/docs/guide/features/captcha.md@@ -1,59 +1,95 @@# Captchas-What Browsentic does at a "verify you are human" block, and — more importantly — what it does not.+What Browsentic does at a "verify you are human" block.---## The short version-It will **tick the checkbox**, because that is a click you could have made yourself and it needs a-real browser-level one to land.+It **ticks the checkbox** with a real browser-level click, and it **sees an image challenge+through**: each round, a short-lived vision session looks at the grid, answers, and is closed.+Only the puzzles it cannot read — Arkose, AWS WAF, drag-to-fit — come back to you.-It will **not attempt a challenge**. An image grid, an Arkose puzzle, an AWS WAF challenge — those-come back to you with the widget's location, and the run waits.+It finds the widget wherever the page put it: inside closed shadow roots, inside cross-origin+iframes, and inside frames nested within frames.-Ticking the checkbox is [gated for approval](../approvals.md) by default, because it acts on-another site's security control.+Answering a captcha is [gated for approval](../approvals.md) by default, because it acts on+another site's security control. One approval covers every round of that captcha in the same run.----## Why ordinary targeting cannot see a captcha+## How the agent knows there is one-A captcha is the one thing on a page that `page_getPageInfo` reports nothing about. Vendors build-the widget as a closed shadow root holding a cross-origin iframe holding another shadow root, so-there is no element for `page_clickElement` to aim at, even though you can plainly see the checkbox.+`page_getPageInfo` carries a `captcha` field whenever the page holds a captcha that needs+answering — the vendor, whether it is already `solved`, and the next step. It is read without+Chrome's debugger, from every frame the browser knows of, so it costs nothing on pages without+one and is simply absent there. Invisible scoring captchas (reCAPTCHA v3 and the like) are left+out, because there is nothing to answer.-The signs you have hit one: a page that never finishes ("Checking your browser", "Just a moment…"),-a form that refuses without saying why, a visible checkbox no snapshot lists, or a-`TARGET_NOT_FOUND` for something obviously on screen.+That field is what makes the agent act: it calls `page_solveCaptcha` as soon as it sees it,+rather than hunting for a checkbox no selector can reach.---## The two tools-**`page_findCaptcha`** identifies what is blocking you, reading through the shadow roots and frames-the widget hides in. Read-only — it never clicks anything.+**`page_findCaptcha`** reports what is there without touching it: the vendor, how many frames+deep it sits, its on-screen bounds, whether it is satisfied and — while a challenge is open —+its prompt and grid.-Recognised: Cloudflare Turnstile, reCAPTCHA v2, reCAPTCHA v3, hCaptcha, GeeTest, Arkose FunCaptcha,-AWS WAF Captcha.+Recognised: Cloudflare Turnstile (including the full-page interstitial), reCAPTCHA v2 and v3,+hCaptcha, GeeTest, Arkose FunCaptcha, AWS WAF Captcha.-**`page_solveCaptcha`** ticks the checkbox with a real browser-level click and waits for the verdict.-Three outcomes:+**`page_solveCaptcha`** gets past it:++1. Finds the widget and waits out one that is still loading (Turnstile's "Verifying…").+2. Scrolls it into view if it is below the fold, and ticks the checkbox.+3. If the vendor answers with an image challenge, answers it round by round until the widget+ is satisfied.| State | Meaning || --- | --- || `solved` | The widget accepted it. Carry on |-| `needsHuman` | The vendor escalated to a challenge a person has to answer. The widget's bounds come back so it can be screenshotted for you; nothing is attempted |-| `invisible` | A scoring captcha with nothing to click — reCAPTCHA v3 and friends |+| `challenge` | An image challenge is still open. The result carries it as an image for the caller to answer |+| `pending` | The widget took the click but had not settled inside the wait |+| `needsHuman` | A puzzle Browsentic does not answer. Its bounds come back so it can be shown to you |+| `invisible` | A scoring captcha with nothing to click |Chrome only. It shows the debugger bar while it runs.---+## Image challenges++The extension photographs the challenge and hands it to the daemon, which starts a one-shot+session of your configured agent to look at that single picture. The session answers and is+stopped at once; nothing carries over to the next round. The extension then clicks the answer+with real pointer movement, presses the challenge's own button, and photographs whatever comes+next.++Two kinds are answered:++- **Tile grids** — reCAPTCHA's 3×3 and 4×4, including the "click verify once there are none+ left" grids that swap each picked tile for a new picture. Each tile's number is painted on the+ photograph, and the photograph is taken only once every tile has finished fading in.+- **Pictures to tap** — hCaptcha's canvas challenges ("click the animal icon that is+ different"). The answer is a set of points on the photograph.++The analyst needs an agent that can look at images; today that is Claude Code. With another+agent, or when the analyst gives up (ten rounds, or the time budget), the result comes back as+`state: "challenge"` with the picture attached, and the agent driving the run answers it itself+by calling `page_solveCaptcha` again with `tiles`, `points` or `reload`.++A vendor can keep setting new challenges to a browser it distrusts however well each one is+answered — an automated or headless browser especially. That is the vendor's verdict, not a+wrong answer.++---+## If it is not a captcha-`CAPTCHA_NOT_FOUND` means there is no known widget on the page, so whatever is blocking the run is-something else — a rate limit, a login wall, a slow request. Worth knowing rather than guessing.+`CAPTCHA_NOT_FOUND` means there is no known widget on the page, in any frame, so whatever is+blocking the run is something else — a rate limit, a login wall, a slow request.---diff --git a/docs/guide/troubleshooting.md b/docs/guide/troubleshooting.mdindex 6bc3c88..e7e203f 100644--- a/docs/guide/troubleshooting.md+++ b/docs/guide/troubleshooting.md@@ -75,7 +75,8 @@ Those three answer most questions. The daemon log also lives at `~/.browsentic/d| --- | --- | --- || `TAB_UNREACHABLE` on a normal site | The extension needs reloading | ↻ at `chrome://extensions`; ordinary sites otherwise self-heal || `TAB_UNREACHABLE` on `chrome://`, the Web Store, the new-tab page | Those pages cannot host a content script | `page_navigate` to an http(s) page — it still works there |-| `TARGET_NOT_FOUND` for something clearly on screen | The page changed since the snapshot, or it is inside a captcha widget's shadow root | Re-snapshot with `page_getPageInfo`; for a captcha use [`page_findCaptcha`](features/captcha.md) |+| `TARGET_NOT_FOUND` for something clearly on screen | The page changed since the snapshot, or it is inside a captcha widget's shadow root | Re-snapshot with `page_getPageInfo`; for a captcha use [`page_solveCaptcha`](features/captcha.md) |+| A captcha keeps setting new image challenges | The vendor distrusts the browser, however well each round is answered — common with automated or headless browsers | Solve one round yourself in the page; a person's answer usually clears the distrust. See [Captchas](features/captcha.md#image-challenges) || `DEBUGGER_UNAVAILABLE` | DevTools is open on that tab | Close DevTools, or use `page_clickElement` instead of `page_trustedClick` || `RUN_IN_PROGRESS` | One instruction at a time per tab | Cancel the running one, or use another tab || `TAB_IN_USE` | That tab belongs to another Browsentic conversation | Switch to it from the Sessions strip |diff --git a/docs/internals/agent-runs.md b/docs/internals/agent-runs.mdindex 0da9037..11428e1 100644--- a/docs/internals/agent-runs.md+++ b/docs/internals/agent-runs.md@@ -223,6 +223,46 @@ agent with the next message.---+## The captcha analyst++`page.solveCaptcha` is the one action the daemon keeps a hand in after the extension has answered.+`invokeOn` routes it through `solveCaptchaWithAnalyst` in+[`captcha-solver.ts`](../../src/daemon/agent/captcha-solver.ts), for side-panel runs and MCP+clients alike:++```+caller ──solveCaptcha {}──► daemon ──► extension tick the checkbox; a challenge opens+ ◄── state: challenge + photo of the grid+ answerChallenge ──runAgentJson (task mode, Read only, effort low)+ ──► extension solveCaptcha { tiles | points | reload }+ ◄── next round … or state: solved+caller ◄── solved, analystRounds: n+```++1. **One session per round.** `answerChallenge()` in+ [`captcha-analyst.ts`](../../src/daemon/agent/captcha-analyst.ts) writes the round's JPEG to the+ task workspace at `0600`, asks for one JSON object after `=== ANSWER ===`, and stops the CLI the+ moment `readAnswer()` accepts its output. The file is deleted in `finally`; nothing carries into+ the next round. A tile outside the grid or a point off the picture makes the whole answer+ unusable rather than half-clicked.+2. **Only where it can see.** It runs when the active runner lists `image` in `Runner.opens` —+ Claude Code today — at effort `low` whatever the conversation uses.+3. **Handing back.** An answer the caller supplies itself is passed straight through. When the+ analyst returns nothing, hits ten rounds, or the budget (`timeoutMs`, 120 s by default, shared+ by every round) runs low, the open challenge goes back to the caller with its photo, which+ `server.ts` renders as an MCP image block, and a note that it is now theirs to answer.++The extension side is [`captcha.ts`](../../src/lib/bridge/captcha.ts) on top of+[`frame-graph.ts`](../../src/lib/bridge/frame-graph.ts): every frame in the tab through+`Page.getFrameTree` on the root and each auto-attached out-of-process session, a frame's viewport+origin composed through `DOM.getFrameOwner` of each ancestor, and `DOM.getDocument` with `pierce`+plus `DOM.querySelectorAll` per shadow root to reach inside closed ones. Page code runs in an+isolated world per frame. A challenge is photographed only once every tile has finished swapping+and fading in — reCAPTCHA reports a tile ready about a second before its new picture arrives, so+the wait is for each clicked tile's picture to change, not for the loading class to go.++---+## Skill routingSkills are markdown with YAML-ish front matter, loaded from three directories, later shadowingdiff --git a/docs/internals/guardrails.md b/docs/internals/guardrails.mdindex 2c39c2f..43c40b1 100644--- a/docs/internals/guardrails.md+++ b/docs/internals/guardrails.md@@ -397,8 +397,9 @@ one is **visible in the log rather than assumed away**.### Two spawn modes-`run` drives the browser. `task` is a one-shot — the file analyst reading an attached file, turning-a raw recording trace into steps — that must not reach the browser at all, which is asserted by+`run` drives the browser. `task` is a one-shot — the file analyst reading an attached file, the+captcha analyst looking at one round of a challenge, turning a raw recording trace into steps — that+must not reach the browser at all, which is asserted byrequiring `{"mcpServers":{}}` (or `mcp_servers={}`, or Grok's `--deny MCPTool`) in its argv. `Read`is deliberately left out of `task`'s deny list, because some tasks are handed a file in the scratchworkspace.diff --git a/docs/reference/tools.md b/docs/reference/tools.mdindex 1020130..5d73f47 100644--- a/docs/reference/tools.md+++ b/docs/reference/tools.md@@ -93,6 +93,12 @@ how many links, buttons and fields its subtree holds and ends with its selector,`page_extractText` to that region, and `interactive.counts` reports the truetotals before `maxPerKind` truncates the lists.+When the page holds a captcha that needs answering, the result carries a `captcha` field —+`vendor`, `label`, `solved`, and while unsolved a `next` step pointing at+[`page_solveCaptcha`](#page_solvecaptcha). It is read from every frame, including ones inside+closed shadow roots, without Chrome's debugger, and it is left out when there is none or only an+invisible scoring captcha.+| Parameter | Type | Default | Purpose || --- | --- | --- | --- || `maxPerKind` | integer | `30` | Cap on links, buttons, fields, and forms listed per kind |@@ -291,8 +297,8 @@ clicked.Report what captcha is on the page, without touching it. Ordinary targeting cannot see one:vendors build the widget as a closed shadow root holding a cross-origin iframe holding another-shadow root, so `page_getPageInfo` shows nothing where the checkbox visibly is. This reads through-all of it with Chrome's debugger.+shadow root, often inside further frames. This reads through all of it with Chrome's debugger.+To get past one, call [`page_solveCaptcha`](#page_solvecaptcha) directly; this is for re-checking.Recognises Cloudflare Turnstile (including the full-page interstitial), reCAPTCHA v2 and v3,hCaptcha, GeeTest, Arkose FunCaptcha and AWS WAF. Takes no parameters.@@ -302,34 +308,50 @@ hCaptcha, GeeTest, Arkose FunCaptcha and AWS WAF. Takes no parameters.| `found` | boolean | Whether any known captcha is on the page || `vendor`, `label` | string | Which one, e.g. `turnstile` / "Cloudflare Turnstile" || `kind` | `"checkbox"` \| `"interactive"` \| `"invisible"` | What the widget asks for |-| `state` | `"idle"` \| `"pending"` \| `"solved"` \| `"needsHuman"` \| `"invisible"` | Where it has got to |+| `state` | `"idle"` \| `"loading"` \| `"hidden"` \| `"pending"` \| `"solved"` \| `"challenge"` \| `"needsHuman"` \| `"invisible"` | Where it has got to || `solved`, `hasToken` | boolean | Whether it is satisfied, and whether the response field is filled |-| `bounds` | `{ x, y, width, height }` | The widget's box in viewport coordinates, for a screenshot |-| `point` | `{ x, y }` | Where its checkbox is, composed across the frame boundary. Absent when there is nothing to click |+| `frameDepth` | integer | How many frames deep the widget sits, when it is inside any |+| `bounds` | `{ x, y, width, height }` | The widget's box — or the open challenge's — in viewport coordinates |+| `point` | `{ x, y }` | Where its checkbox is, composed across every frame boundary. Absent when there is nothing to click |+| `challenge` | object | While an image challenge is open: its `kind`, `prompt`, and for a grid `rows`, `columns`, `tiles`, `selected` and `dynamic` || `note` | string | Why, when there is no `point` or the state needs explaining |Read-only, so it is allowed during a site-mapping run.+`page_getPageInfo` reports the same thing more cheaply as its `captcha` field — vendor, `solved`+and a `next` step — read from every frame without the debugger, and absent when there is none.+### page_solveCaptcha-Tick a captcha's "I am a human" checkbox with a real browser-level click and wait for the widget to-settle. The checkbox only responds to genuine pointer input, which is the point of it, so no-synthetic click reaches it.+Get past a captcha. It finds the widget in any frame at any depth, waits out one that is still+loading, scrolls it into view, ticks the checkbox with a real browser-level click, and sees an+image challenge through.-**Confirm-gated.** Ticking another site's human check is the user's decision; an external MCP-client with no approval channel gets `DECLINED` under the default `unattended` policy.+**Confirm-gated.** Answering another site's human check is the user's decision. One approval+covers every call of this tool for the rest of that run, since a challenge takes a call per+round. An external MCP client with no approval channel gets `DECLINED` under the default+`unattended` policy.-When the vendor escalates to a challenge a person has to answer — an image grid, Arkose, AWS WAF —-it returns `state: "needsHuman"` with the widget `bounds` and **does not attempt the challenge**.-Screenshot that region, tell the user, and poll `page_findCaptcha` until they have solved it.+**Image challenges.** When the configured agent can read images (Claude Code), the daemon answers+each round with a one-shot vision session that is stopped as soon as it has answered, and the call+returns only once the widget is satisfied or the analyst gives up. Otherwise — or after ten rounds,+or when the budget runs out — the result has `state: "challenge"` and carries the challenge as an+image: an MCP image block beside the JSON. Answer it by calling this tool again with `tiles`,+`points` or `reload`, and repeat until `solved`.| Parameter | Type | Default | Purpose || --- | --- | --- | --- |-| `waitMs` | integer 0–120000 | `20000` | How long to wait after clicking for the widget to report a verdict |-| `timeoutMs` | integer 1000–180000 | `60000` | Overall budget for the attempt, including finding the widget |--Returns the same fields as `page_findCaptcha`, plus `clicked`. Fails with `CAPTCHA_NOT_FOUND` when-there is no widget to act on.+| `tiles` | array of integers 1–25 | — | Answer to a `kind: "tiles"` challenge: every matching tile, numbered as painted on the image (1 top-left, along each row). The whole selection; `[]` for none |+| `points` | array of `{ x, y }`, 1–16 | — | Answer to a `kind: "points"` challenge: where to tap, in pixels on the image as delivered |+| `reload` | boolean | — | Swap the open challenge for another instead of answering it |+| `waitMs` | integer 0–120000 | `20000` | How long to wait after each click for the widget to report a verdict |+| `timeoutMs` | integer 1000–300000 | `120000` | Overall budget for the attempt, image rounds included |++Returns the same fields as `page_findCaptcha`, plus `clicked` and, when the analyst ran,+`analystRounds`. While a challenge is open, `challenge` adds `image`, `imageWidth`,+`imageHeight`, `errors` (the vendor refused the last answer) and, for a grid that swaps picked+tiles, `fresh` (the tiles that just changed). Fails with `CAPTCHA_NOT_FOUND` when there is no+widget to act on, and with `INVALID_INPUT` for an answer that does not fit the open challenge.### page_hoverElement