Open-source AI browser agent for Chrome and Firefox (monorepo) π§
Vision mode (aka "Tesla" mode)
In vision mode (aka "Tesla" mode, configured under Settings β Advanced β Vision mode), like a Tesla that drives on cameras alone, the run works only from what it sees. With the setting Off, WebBrain is closer to Waymo, which adds other sensors: an Act run reads the page's accessibility tree, element refs and selectors, with screenshots as one more input. In code, storage and traces the feature keeps the name vision mode (visionMode, vision_mode).
In vision mode an Act run sees the tab only as screenshots and acts with the mouse and keyboard at coordinates. There is no accessibility tree, no ref_id and no CSS selector in the model's view. The run starts from a full frame of the viewport; after every action WebBrain sends only what changed on screen (a patch), a new full frame when most of the screen changed, or a one-line "no visible change".
The accessibility-tree path stays available (Settings β Advanced β Vision mode β Off) and is used automatically when the model cannot take images.
The setting
Settings β Advanced:
- Vision mode (storage key
visionMode): Auto (default) or Off. Auto applies to Act runs whose model receives screenshots directly (provider.supportsVision). A separate describe-only vision model cannot aim clicks, so a text-only model keeps the accessibility tree. Ask and Dev are unchanged. The option was called On before; a storedon(ortrue), and an exported configuration that carries it, reads as Auto. - Vision screenshot size (
visionQuality): Economy, Balanced (default) or Sharp. It caps the model-facing image (Economy: long edge 1024 px / 0.6 MP; Balanced: 1440 px / 1.05 MP; Sharp: the provider's own limit). The model can alwayszoomfor detail.
A run decides once, when it starts (_beginVisionRun snapshots both settings), like lean mode. The resolved state is recorded in the trace's runtime config as vision_mode_setting and vision_mode. The Agent class default is off, so embedders and unit tests opt in explicitly; background.js applies the stored setting, whose default is on.
Three things end vision mode inside a run for good, all proofs that only the accessibility tree can give. A site workflow whose completion is proved against the app-owned element inventory (requiresLedger in adapter-workflow.js, form and publish jobs of some adapters) cannot be certified from screenshots: once the planner or a live adoption selects one, resolveVisionMode answers workflow_needs_element_inventory. A request for a whole conversation (a thread to summarize in full, Gmail's read-complete-thread job) is answered only after trusted coverage through its last page with every message expanded (read-completeness.js): once that requirement is armed, it answers complete_thread_read. WhatsApp Web and Telegram's web app confirm who an open chat is with, and bind the chat they read and send in, through the element inventory (a contact check made with click_ax, chat reads and sends bound to refs): on their pages it answers native_chat_needs_element_inventory (_visionNativeChatPage, from the tab's last address). Either way the next step gets the tree tools and the tree prompt (the loop rebuilds the system prompt when the vision state flips).
A file upload leaves vision mode for a few steps only. A native file chooser opens outside the page, where screenshots cannot show or operate it, so a click_at that lands on a file input is stopped by the file-chooser guard (Chrome's CDP guard, Firefox's content-script guard, on every dispatch path). The batch pipeline then arms an upload handoff (_visionNoteUploadHandoff in vision-runtime.js, kept in the run's snapshot) and rewrites the result: the next steps use the element tools, upload_file with the blocked input's selector (the compact tier's discovery call instead), and screenshots come back after the upload. Changes take effect only at step boundaries (_visionUploadHandoffTick, at the top of each step, before adapter notes, cookie banners and the step's tools), so the rest of the click's batch and its screen update stay in vision mode, and the upload's batch stays on the tree path. While the handoff is active resolveVisionMode answers file_upload_needs_element_inventory. It belongs to the page the tab shows when it starts (the click that armed it may itself push a route), and ends after a successful upload_file (the file is in the input in any build's success shape, not a discovery listing or an unconfirmed outcome), when the tab's address changes (the fragment aside), or after five decisions without an upload. A run hands off at most twice; a third chooser click is reported as before (uploads are not available in vision mode). Nested runs keep their own handoff with their snapshot.
No handoff is armed when the upload could not succeed on the tree path either, and the click result says why instead: when only Chrome's protocol guard saw the chooser (protocolOnly: an input in a frame or a closed shadow root, which upload_file cannot reach), and on Chrome's compact tier when the run's message has no attachment (_visionUploadSourceAvailable; compact upload_file attaches only those).
Whenever the run returns to vision after tree-path steps (an upload handoff, a ledger workflow let go, a WhatsApp page left), the step starts on a fresh frame (_visionResumeKeyframe, at the flip in the loop): the old frame is retired first, then a settled full frame is captured with the reason vision resumed. A batch planned on the tree path sends no screen update of its own even when vision is back on by its end (_visionNoteStepTools marks the step), so the return costs one frame. Points on images from before are stale_image, and the older screenshots leave the request. If the capture fails, a retire message tells the model so, and coordinates are refused (no_frame) until take_screenshot succeeds.
Tools
A vision run gets its tier's Act tools with the DOM-targeting ones replaced (VISION_REMOVED_TOOLS in vision-tools.js):
| Removed | Added |
|---|---|
get_accessibility_tree, click_ax, type_ax, set_field, set_checked, hover, drag_drop, inspect_viewport, get_interactive_elements, click, scroll, browser_batch, wait_for_element, find_text, verify_form, iframe_*, get_frames, get_shadow_dom, shadow_dom_query, promote_iframe, carousel_navigate, upload_file, download_resource_from_page, get_captcha_capabilities, apply_captcha_solution |
click_at, scroll_at, hover_at, drag_at, zoom, take_screenshot, wait |
type_text stays with a vision schema (text, clear; it types into the focused field). Everything that does not name a DOM target is kept: press_keys, navigate, go_back/go_forward, read_page, read_pdf, extract_data, fetch_url, downloads, scheduling, scratchpad_write, progress_*, clarify, done, skills and solve_captcha. A CAPTCHA's native provider methods are applied to frame and field bindings only the tree names, so a vision run gets solve_captcha's automatic route only: its schema has no providerTasks (visionSolveCaptchaTool), the native-method tools are removed, and the [CAPTCHA SOLVER] note tells it to call solve_captcha once and let WebBrain check clearance (visionCaptchaSolverNote), not to read the tree. read_page is the cheap way to get exact long text; screenshots are the primary view.
| Tool | What it does |
|---|---|
click_at({x, y, image?, button?, clicks?}) |
A real mouse click at the point. clicks:2 double-clicks, clicks:3 selects a line, button:"right" opens a context menu. |
scroll_at({direction, amount?, x?, y?}) |
Mouse-wheel scrolling of whatever is under the point (page, pane, list, map), in steps of ~100 CSS px. |
hover_at({x, y}) |
Moves the pointer without clicking (hover menus, tooltips). |
drag_at({start_x, start_y, end_x, end_y}) |
Press, move and release (sliders, reordering, selecting). |
zoom({x, y, width, height}) |
Returns the rectangle as a sharper image z<n> (up to 2 image px per CSS px). |
take_screenshot() |
A fresh full frame. |
wait({seconds}) |
Waits 0.2β10 s; the screen update follows. |
The vision prompt (buildVisionActPrompt, used through getVisionActPrompt in tools.js) replaces the tier's tree-centred Act prompt and reuses the shared sensitive-data and plan-to-execution sections verbatim. The universal cookie-banner / paywall / PDF guidance is given in vision terms (visionUniversalPreamble: click_at "Reject all", look at the updated screen, read_pdf instead of paging a PDF with screenshots), and site adapter notes, written for the element tools, arrive with one line saying that click_ax, set_field, refs, selectors and the tree are not available and their facts about the site apply to click_at, type_text, press_keys and scroll_at (VISION_SITE_NOTES_PREFACE).
Images and coordinates
Every image is labeled with an id: f<n> for a full frame, p<n> for a patch, z<n> for a zoom. The model gives a point in the pixels of the image it read it from and names that image with image when it is a patch or zoom; with no image the point is read from the current screen (latest frame plus its patches). VisionSession (vision-session.js) keeps each image's map to the frame and the frame's map to CSS pixels, so the model never rescales anything.
Gemini, Qwen3-VL, GLM-4.xV and InternVL point in relative 0..1000 coordinates; for them the tools and prompt say so and the session converts (coordinates: 'normalized'). Every other model uses image pixels. An image value that is not an exact id ("z2.png", "zoom z2") is refused (unknown_image) instead of being read as the current screen.
A point is refused before any dispatch when:
- its image belongs to an earlier frame (
stale_image), or the area around it changed in a later patch (stale_area); a zoom taken after an earlier action of its batch already shows what that action changed, so it counts from the update the batch produced and only a change after the zoom's capture makes its points stale (a frame-scale capture beside the zoom tells,_visionRefreshZooms); - it lies outside the image;
- it is the second or later coordinate action of one model turn, or any coordinate action while the last screen update failed (
session.observationFailed), and the live screen changed near it since the screen the model last saw, or the screen could not be re-captured to check (_visionStabilityBlock, checked right before dispatch, after every gate; adrag_atis checked at both ends against one capture); - the run has no frame of its own yet (
no_frame); - the tab shows another page than its frame did (
page_changed,_visionStaleLayoutRefusal, a navigation page feedback has not reported yet); a full frame follows. A page that only scrolled since the model looked is not refused, but every point of the batch is checked against the live screen first (session.screenMoved), so content that scrolled under a point stops it; - the browser window was resized or zoomed (CSS viewport size or device pixel ratio) after its frame was taken (
viewport_changed,_visionStaleLayoutRefusal): checked once before each batch that uses points, so a resize page feedback has not reported yet is caught too; a full frame follows. A resize page feedback did report while the model decided sets the decision aside for a fresh full frame before that (visionScreenFeedback).
A refused point also stops the rest of the model's batch: the calls after it (a type_text, say) were planned on that point.
Typing goes to whatever has focus, which only the screen shows. While the run has no frame of this page, or the last screen update failed (a capture or encoding error, or a re-check capture that failed; any later frame, patch or "no visible change" clears it), type_text and press_keys are refused before dispatch (visionScreenUnverified) unless a click_at earlier in the same response passed its check, in which case they type into the field it focused. The refusal stops the rest of the batch, and the model is told to take a screenshot or click the field first.
Every viewport measurement and screenshot (frames, patches, zooms, re-check captures, the layout check before a batch) is raced against the run's Stop and a 15 s deadline (_visionBounded): a hung renderer or debugger ends as an ordinary capture failure (vision_capture_timeout, the usual "could not be captured" note), which is not retried, and Stop ends the wait at once. An observation (the first frame, a screen update, the frame on returning to vision) whose capture failed for a passing reason, such as a navigation destroying the page context or a page not ready to paint (classifyVisionCaptureError), is tried again up to four times within 3 s (_visionCaptureRetrying). Input, timeouts, privacy failures and anything else are never retried. A browser page screenshots cannot capture (chrome:// and similar) is not retried either, and its note tells the model to open a web page with navigate or ask the user instead of taking another screenshot.
Every run starts on a frame it captured itself: _beginVisionRun retires the previous run's screen, so if the first capture fails the model is told so, earlier screenshots leave the request, and coordinates are refused until take_screenshot succeeds. Image ids and epochs continue across sessions of one conversation (another model, a restarted worker), and the current screen is the last full frame in the history by position. A run that does not start on vision (the setting off, a text-only model, a page that needs the element inventory, Ask) after a vision run in the same conversation pushes a retire marker first (_visionRetireEarlierFrames), so the earlier run's last frame and patches are not sent with its requests. Frames within one run are kept across a tree-path interlude, and replaced by the fresh frame on returning to vision.
Capture, retina and zoom
vision-budget.js picks the image size per provider family (detectVisionProfile), as the largest size the provider accepts unchanged: every vision API shrinks larger images before the model sees them, and the model then answers in the shrunk image's pixels.
| Profile | Limits | Format |
|---|---|---|
| Anthropic | long edge β€ 1568, β€ 1.15 MP | WebP |
| OpenAI | β€ 2048, short edge β€ 768 | WebP |
| Gemini | β€ 1536, β€ 1.2 MP; 0..1000 points | WebP |
| Qwen2/2.5-VL, UI-TARS | β€ 1.0 MP, multiples of 28 (so smart_resize is the identity) |
JPEG |
| Qwen3-VL / Qwen3.5+ | β€ 1.05 MP, multiples of 32; 0..1000 points | JPEG |
| GLM-4.xV, InternVL | β€ 1280, β€ 1.0 MP, multiples of 28; 0..1000 points | JPEG |
| Local (llama.cpp, Ollama, LM Studio, WebGPU) | β€ 1280, β€ 1.0 MP, multiples of 28 | JPEG (llama.cpp cannot decode WebP) |
An image never has more pixels than the CSS viewport: a retina capture is downscaled locally, and zoom is the path to fine detail. WebP is ~30% smaller than JPEG for UI screenshots, so uploads are faster; providers bill images by pixels, so the token cost is the same.
Chrome captures with CDP Page.captureScreenshot without a clip: the tab's surface as it is on screen, devicePixelRatio image pixels per CSS pixel from the visual viewport's top left (Chromium keeps the page scrollbar on the right, in right-to-left pages too). The visual viewport, or a zoom's region, is cut out and scaled to the plan locally (cropImage in vision-encoder.js), which also leaves classic scrollbars out. A clip would have the GPU do the scaling, but Chromium serves one by resizing the tab's view to clip Γ scale and rendering the page into it until the capture is done: a visible tab shows a shrunken page in its top-left corner, white around it, on every capture of every screen update. test/vision-pipeline-browser.mjs watches the screencast for that resize.
A clip is still used where only a re-render can serve: a zoom that asks for more detail than the screen has (more image pixels per CSS pixel than the DPR, as most zooms at DPR 1 do), and a pinch-zoomed page, whose surface is not the visual viewport. Two properties of a clip drive that path (verified in the same test):
- Clip coordinates are document coordinates. A clip at
0,0on a scrolled page captures the unrendered top of the document, a blank frame. The clip starts at the visual viewport's page offset (Page.getLayoutMetrics). - The clip is in device-independent pixels, CSS pixels times the browser zoom (
Page.getLayoutMetrics().cssVisualViewport.zoom), and the output is clip Γ scale Γ device scale factor. With the DPR (device scale factor Γ zoom) divided out of the scale, the image comes back at the planned size; whatever size actually comes back is resampled to the plan (rasterizeFrame), so emulated DPRs and compositor quirks cannot change the coordinate map. The legacy auto-screenshot clip (_viewportCaptureClip) applies the same zoom factor.
Firefox captures with tabs.captureTab, whose scale is image pixels per CSS pixel (no DPR factor) and whose rect is page-relative. The rect is always the visual viewport: without one Firefox captures the whole content area, classic scrollbars included.
Screenshot redaction maps element boxes into the captured visual-viewport box (page space), so a field next to a scrollbar is still covered.
Diffs
After a batch WebBrain waits until two consecutive captures match (bounded, ~1 s; a micro change between them, a few 16 px cells such as a blinking caret, a clock or a spinner, counts as a match, isMicroScreenChange, since it never stops). Its own page UI (the activity effect, cursor and Stop button) is hidden once for all of those captures and shown again after the last, rather than around each one, so it does not blink on every step; each capture still asks, which re-hides the UI of a page that navigated in between (_withIndicatorsHidden). It then compares the new frame with what the model has seen (vision-diff.js): 16 px cells, a per-pixel threshold that ignores encoder noise and caret blinks, nearby changes merged, at most 4 padded regions. planFrameUpdate sends:
- nothing but a one-line note when nothing changed: "no visible change" is also the clearest signal that a click missed;
- patches for local changes (typing, a menu, a toast). For a provider with an image grid (28 px for Qwen2/2.5-VL, UI-TARS, GLM-4.xV, InternVL and local servers; 32 px for Qwen3-VL) each patch is grown to multiples of the grid, like frames and zooms, so the server never resizes it and its pixels map 1:1 to the frame;
- a new frame when more than 45% changed, the patch chain is 6 long, patches would cost β₯ 70% of a frame, or the URL, viewport size or DPR changed.
The "seen" buffer copies only patched regions, so changes below the threshold accumulate until they are sent instead of drifting forever.
Only the current frame, its patches and its newest zooms stay in the model request (pruneVisionImages); older images become a short placeholder. At most four zooms of the current frame stay, and stay usable (VISION_MAX_LIVE_ZOOMS; a point on an older one is refused as stale_image with a note to zoom again), and while the kept images exceed about 8 MB of data URL the oldest zooms leave first. Patches are never dropped, since they are the current screen; a long patch chain becomes a new frame instead. Screen updates are app-owned user turns (webbrainAppOwned), never mistaken for a new request.
A capture, encoding or input failure never ends the run: a screen update that cannot be prepared becomes a one-line note and the next update is a full frame; a vision tool that fails returns a tool error (an unknown outcome once input may have gone out).
Page feedback in a vision run
The page monitor (content/page-monitor.js) and page feedback (agent/page-feedback.js) watch the page between decisions as in a tree run. A vision run changes what they show the model and when they set a decision aside:
- No tree text reaches the model. The pre-decision capture still binds keyboard input to the focused field, but its tree text is not added to the conversation; feedback messages carry the events and the URL without a page dump, and applying feedback skips the accessibility-tree read altogether.
- Page churn does not supersede screenshot points.
click_at,hover_atanddrag_atarevision_pointcalls andscroll_atascrollcall (page-feedback-policy.js), so automatic page activity while the model was thinking (a ticker, a live timeline, a spinner) keeps the decision. The pixels around the point are compared with the model's frame right before dispatch (_visionStabilityBlock, mandatory once anything changed), and the monitor rebases the prepared operation over passive or unattributed churn only while the element under the point is the prepared one and has not moved (rebaseVisionPreparation: a click's point, ahover_at's, adrag_at's start, whose end the pixel check holds; a wheel binds no element). A person's input, a navigation or another document still voids it. - Typing rides on its click.
type_textandpress_keysright after aclick_atof the same response are checked by focus: the element focused at the clicked point must be a text field (_visionFocusFollowsClick, the probe'sfocusreading). The page reacting to that focus (a hint, a composer expanding) no longer blocks the typing. - The wheel stays the agent's. Every CDP wheel notch passes the fence, and the monitor counts the wheel events those commands own, so a notch the browser delivers after the next one was fenced is not taken for a person's scroll.
- A set-aside decision is remade on the current screen (
_visionObserveNow: a patch, a new frame or "no visible change"), never on tree text. - An unchanged screen is not sent again before a decision. When the only news is automatic churn, or a viewport change of no known origin (the debugger infobar appearing when a run starts), one capture is compared with what the model last saw; within a micro change no message and no image are added (
_visionScreenUnchanged), and such a resize does not hold the decision for the idle wait. A person's input is always reported, with a fresh look.
Latency
With an instant model the MV3 fixture's four steps (click and type, send, scroll, done) took about 15 s before these changes and about 7.5 s after:
- the settle wait ends on micro changes instead of running its full ~1 s on every step with a caret or a clock on screen;
- the forced re-observation after page churn and the tree read in feedback are gone (one screen update per step);
- in Chrome the file-chooser guard's 500 ms settle doubles as the quiet window for a new tab (
watchOpenedTabssees tabs as they are created) instead of a further 900 ms poll; a tab the click opens later is still rerouted into the run's tab (_visionPlatformLateTab): the watch stays armed for 4 s, a tab that opens while the update's screen settles is rerouted and the screen read again, and one that opens while the model decides is rerouted before the next update; - the first frame is taken after a cookie-banner decline in progress has finished, so it does not show a banner about to disappear.
Input dispatch and safety
click_at is rewritten into an ordinary coordinate click (translateVisionClick) before the batch pipeline runs, so the permission gate, the submit probe and every hard guard (recipient, deletion, money, credentials, publishing) inspect it like any other click. A guard that binds a consequential click to the control and payload it checked (a deletion, a commit after a correction) refuses coordinate clicks on the tree path, where a ref or selector click exists; in a vision run, where the point is all there is, it binds the click to the control the submit probe read at that point (_resourceMutationActionKey), with the same record, stage, sibling and confirmation checks and the same card, and _visionDispatchClick reads the point again right before input goes out (_visionMutationBindingBlock): when the authorized control is no longer there, nothing is sent (resource_changed_before_dispatch) and the model chooses again on the updated screen. A guard's advice names what the run offers (click_at), never a ref or selector. Only the final dispatch differs:
- Chrome: trusted CDP pointer events at exactly the point (
_visionDispatchClick), with the coordinate path's dispatch-time protections kept: a native<select>is focused fortype_textinstead of opening an OS popup no screenshot can show, the file-chooser guard, the recipient dispatch binding,target=_blankredirection into the run tab (also for a tab opened after the click returned, see Latency). The legacy coordinate path's semantic re-targeting, nearby-input redirect and DOM-snapshot progress check are skipped: a vision click lands where the model pointed, and whether it changed anything is for the screen update to show (a DOM text snapshot cannot see a canvas, a map or a value it does not read). Iframes, including cross-origin ones, receive the events natively. - Firefox: no CDP, so
content.jsvisionInputdispatches a synthetic pointer sequence at the exact point with real client coordinates, reproducing a press's focus (and its caret: a click in a text field or editable text puts the caret under the pointer) and a click's activation, moving off the element the pointer was last over the way a mouse does (out and leave events, so a hover menu that closes on mouseleave closes; synthetic events give nothing:hover, so while the pointer rests on an element it and its ancestors carry an attribute, and every:hoverin the selectors of the style rules the content script can read also accepts it, in place (:hoverreads:is(:hover, [data-webbrain-hover]), keeping the rule's place in the cascade, its specificity and the other selectors of its list), which a CSS-only menu or tooltip opens for, until the pointer moves on or a person moves their own pointer, when the page's selectors are put back; a wheel moves the pointer to its point first and goes where the browser would send it: the nearest scroller that can still move that way, an exhausted one passing it on unless itsoverscroll-behaviorkeeps it, never a page locked withoverflow: hidden, and from a frame that cannot scroll to the page around it), selecting the word under the pointer for a double click and its line (a textarea's line, a whole input, the text block) for a triple click, and entering same-origin iframes: the point is mapped into the frame's own coordinates past its border and padding, undoing the scale a CSS transform or zoom draws the frame at (on itself or an ancestor), and a point on a frame drawn rotated, skewed, flipped or in 3D, which no scale maps, is refused (frame_unmappable). The click runs inside the same native file-chooser guard as every agent click (clickWithoutNativeFilePicker), and the agent settles that guard afterwards, so a chooser opened a moment later is caught too. Dispatched events reach a disabled control's listeners, unlike a person's click (orel.click()), so a click or drag that would press a disabled control, disabled by its fieldset included, is refused before anything is sent (disabled_control). A press whosepointerdownthe page cancels gets what a browser gives it: no compatibilitymousedown,mousemoveormouseupuntil its release (the boundary events still go out) and none of the press's defaults below (focus, the caret, a range thumb, a selection, drag and drop, a double click's word), while itsclickstill fires. A drag gives every move and the release in the coordinates of the document under the point, so a slider in a same-origin frame reads one consistent path; its press focuses what it lands on and starts what a person's press would, since synthetic events alone move no native slider, select no text and start no drag and drop: on an<input type=range>the value under the pointer is set as the browser sets it (snapped to the step, aninputevent per change,changeon release; a click's primary press on the track moves the thumb to its point the same way), on selectable text or in a text field the text from the press to the pointer is selected (after a cancelableselectstart; not on links, images, draggable items, other controls oruser-select: none), and on adraggable="true"item HTML5 drag and drop runs as the browser runs it (dragstartwith aDataTransferthe page fills,drag/dragenter/dragleave/dragoverper step,dropwhere the lastdragoverwas accepted,dragend; no pointer or mouse events meanwhile). A press the page captures (setPointerCaptureon its pointerdown, as sliders, canvases and press-and-hold controls do) keeps that capture for the rest of the click or drag: while the sequence runs, the capture methods of the press's window answer for the vision pointer's id (Firefox would throw for it, and no engine routes a synthetic pointer's events), each later pointer event and its compatibility mouse event go to the capturing element in its document's coordinates,gotpointercapturecomes before them andlostpointercaptureright after the release, and the page's own methods are back once the sequence ends. A click that must pass the recipient binding keeps the content click path. A click or drag on a frame the content script cannot enter (another site's frame, a plugin, or a frame still loading) is refused (opaque_frame): its events would stop at the frame element and reach nothing inside it. - Firefox with trusted automation: when Firefox trusted automation is on (Settings β Advanced, the local companion in
firefox-companion/) and its companion reports point input (point_input), a left click and a hover go out as trusted input at the point (bidi/client.jsvisionInput). The content script runs every check it runs for the synthetic sequence (the viewport, a frame it refuses, a disabled control), then marks the element the top document shows at the point, the one the page monitor prepared there (_visionBidiPrepare); the companion checks that the element is still at the point and moves the real pointer there (hover_at), then presses and releases its left button (click_at), so:hover, focus and pointer capture are the browser's own. A click still runs inside the file-chooser guard, settled as the synthetic click's is. The page monitor attributes it as it does the companion's other clicks (a one-time marker on that element). A double, triple, right or middle click, a drag and a wheel stay synthetic (a BiDi wheel scrolls only the window's selected tab, and a run's tab can be in the background), and so does a point over an embedded frame or inside an open shadow root (the monitor binds the shadow host there, but sees the events come from inside it, where it cannot tell them from a person's; the synthetic sequence reaches into both). A companion without point input leaves all vision input synthetic; one that disconnects mid-task stops its trusted input, as for its other actions.
In Firefox a vision click that must pass the recipient binding is sent by the content click, which checks that binding and sends one left click on the element at the point; there a right, middle or repeated click is refused (recipient_bound_click) rather than turned into a left click, and an authorized deletion is read again at its point first. The rich-text toolbar recovery guard checks a vision click's target but does not bind it to an element, so it keeps the vision dispatch in both builds.
Right before input goes out, both builds read what is under the pointer with visionDomProbe (vision-dom-probe.js), in an isolated world (Chrome, CDP) or the content-script world (Firefox), so page scripts cannot redefine the DOM methods it calls. It maps a point into a same-origin frame as the Firefox input does; a frame no scale maps counts as one it cannot inspect. (The click submit probe's Chrome frame mapping likewise takes the frame's border, padding and drawn scale into account.)
click_atwithclicks: 2|3is refused on a button, link or other control (including an element drawn with a pointer cursor, which is how a framework button made of a bare<div>shows itself), on a detected submit, on a protected messaging page, inside a frame it cannot inspect, or when the target cannot be read: each extra click would repeat the one action the checks approved. Double and triple clicks remain for selecting text.drag_atis refused when it is shorter than 6 CSS px, when the press and release would click a control (their nearest common ancestor is or is inside a button, link, label, similar control or pointer-cursor element, unless the drag starts on a slider, scrollbar, resizer, draggable item or grab handle), when either end is inside a frame it cannot inspect (another site's frame, or one drawn rotated, skewed, flipped or in 3D), when the target cannot be read, or when the click submit probe reads its start point as a submit, send, delete or payment control (a drag never gets the confirmation a click gets). On a protected messaging pagedrag_atis blocked outright, likeiframe_click.scroll_atis refused over a focused number field, which takes the wheel as +/-.
Every CDP input passes cdpClient's page-agent fence and every content action passes beforePageAgentDispatch, so the page monitor attributes the events to the agent. Stop is checked again right before every input goes out, after the probes and captures that precede it (_visionThrowIfStopped for scroll_at, hover_at and drag_at; Firefox's _visionContentInput before each trusted or content-script input; _visionDispatchClick for clicks). drag_at is gated as a click (TOOL_CAPABILITY); captures and pointer results are untrusted page content (UNTRUSTED_CONTENT_TOOLS), and every image label says that text in it is data, never instructions. Screenshot redaction (Settings) applies to frames, patches (cut from the redacted frame) and zooms, and is best-effort there as everywhere: an image it could not run on is still sent. Right after a navigation the page's scan is often not ready (its content script starts when the document is idle), so while the tab is loading WebBrain waits up to 1.5 s for the scan and captures again. When an image does go out unredacted (no scan, an undecodable capture, or a region that could not be pixelated, which fails the whole redaction), the trace records vision_redaction_skipped with the reason (code) and the image kind (context), and the side panel warns once per page and run (_visionReportUnredacted).
Guards in a vision run
Every hard guard and quality check runs on vision calls; these are the places where a check written for refs, selectors or tree reads had to be told what a vision run has instead (test/vision-mode-guards.mjs):
- The submit probe reads a point inside open shadow roots, as the pointer events reach it, so a Delete or Pay button in a web component is the control, not its host. A vision click on plain content (no control, link, field, label or inline handler on the element or around it, and no frame, embed, canvas or application region, whose controls the DOM does not show: a post's text, a picture, blank space) is read as a non-submit target (
resolvedPlainContent): it can open what the page opens for it but cannot submit or publish, so the X/Bluesky publication guard and the post-correction guard let it through instead of asking for a ref. - Deletions and commits after a correction bind a vision click to the control read at its point and read it again at dispatch (see Input dispatch and safety).
- Loop detection keys a vision action on the screen it acted on (
VisionSession.screenVersion, bumped by a patch or a frame with other pixels, not by "no visible change" or atake_screenshotof the same screen;_loopScreenIdentity). Accepting ten invitations under the same point, pressing Enter after each typed item or scrolling to the next stretch of a list is progress; the same point, key or wheel on a screen that does not change still nudges and stops (the coordinate stop after eight clicks, with the model's own point in the message). Whether a click or an arrow key changed anything is for the screen update to say: the DOM text snapshot behindnoProgressis not taken in a vision run, since it cannot see a canvas, a map or a value it does not read. Choosing from a native dropdown (nativeSelect) is never a failed click. Loop, stop and recovery messages nameclick_at,zoomand the screen instead of selectors, refs and the tree. - Refusals without dispatch: a call a pre-dispatch guard refuses (publication audit, commit binding, resource mutation, a prior submission after steering) never reaches the loop detector, in any run. It has its own count per run (
_noteRefusalWithoutDispatch): the third identical refusal (arguments compared without image ids) or the sixth of any kind stops the run as the loop detector would, and any call that goes through resets the count. - The delivery checkpoint counts
take_screenshot,scroll_atandwaitlike the tree's reads, scrolls and waits, and a screen update that shows a change ends the streak, so only looking at a screen that stays the same adds up. Zooms are left out: reading a dense screen takes several. - Page feedback keeps a response of
take_screenshot,zoomorwaitwhen a page script moves focus: none of them acts on focus. - Completion: every batch, including one that ends early (a refused
done, a completion hand-back), is followed by one screen update (_executeToolBatch), so the model never acts again on the frame its last actions were chosen on;done's own verification screenshot, an unlabeled device-scale image, is not taken. The opening frame counts as the run's first look at the page (plan evidence); after an action a zoom counts as an observation, as does a screen update once it was captured (auto_screenshot), but nottake_screenshotitself, which only asks for the frame the update after its batch captures; adrag_atis a consequential action like thedrag_dropit replaces; and the status texts on screen (toasts, alerts, live regions) are recorded with each update while a submission waits for proof. Refusals point to the screen update,take_screenshot,read_pageandwaitinstead ofget_accessibility_tree,verify_formorwait_for_element.gmail_count_results, which takes no target, stays in the vision tool list for Gmail's count job. - CAPTCHA: a solved or manually completed challenge is seen gone only through an authoritative root tree read; while a gate is pending, WebBrain makes that read itself before the next page action (
_visionCaptchaRootRead, never shown to the model).
Files
| File | Role (both builds; the first eight are byte-identical) |
|---|---|
vision-mode.js |
setting normalization and per-run resolution |
vision-budget.js |
provider profiles, sizes, token estimates, formats |
vision-diff.js |
frame diff, regions, update planning |
vision-session.js |
image ids, coordinate maps, staleness |
vision-tools.js |
tool schemas, tool-set transform, prompt, labels, click translation |
vision-encoder.js |
canvas decode / resample / encode |
vision-runtime.js |
agent methods: keyframes, observation after batches, native tools, pruning, dispatch preflights |
vision-dom-probe.js |
the in-page probe behind the dispatch checks, and the checks themselves |
agent.js |
platform layer (_visionPlatform*, _visionDispatchClick) and hooks |
content.js |
visionInput (Firefox input path, and the marking for trusted input; also loaded in Chrome) |
bidi/client.js (Firefox) |
visionInput: a left click and a hover as trusted input through the companion |
Tests
npm run test:vision-mode
test/vision-mode.mjs: sizing, profiles, token formulas, diffs, micro changes, sessions, translation, tool surfaces of every tier in both builds, permission classification, prompt, the page-feedback policy for vision calls and the monitor's vision rebase.test/vision-mode-agent.mjs: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation (and aclick_atin a turn that does not offer it, such as a completion recovery turn, refused before it can run as a click), patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools (the page monitor prepared withhover_at's anddrag_at's CSS points, a wheel with none), fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals,drag_aton a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; a drag after an earlier action of its batch re-checked at both ends; a window resized while the model decides (a fresh full frame first) and one page feedback has not reported yet (the batch's points refused); atake_screenshotanddonein one batch after an update that could not be captured, and adrag_atfinished in its own batch, both refused as unverified; a zoom after an earlier action of its batch usable beside that batch's patch except where the screen changed after it; in Chrome, a click that opened no tab costing no poll, tabs opened at once or later being rerouted, and a late-tab watch staying armed past an update and rerouting a tab opened while the screen settled.test/vision-mode-guards.mjs: the guards in a vision run, both builds: a vision click deleting the item the request names is bound to the control read at its point (the same point outside a vision run stays unbound), the deletion card, a changed item, a forbidden item and Enter keep their rules, and an authorized deletion is read again at dispatch, sending nothing when the control moved away; loop detection by screen (repeats after a change are progress, on an unchanged screen they stop), vision loop messages, native dropdowns, arrow keys and clicks without DOM-snapshot verdicts, page-focus churn, the delivery checkpoint, one screen update per batch however it ended; whole-thread reads and WhatsApp/Telegram on the tree path, the Gmail counter, observations and refusal wording after an action (a requested frame verifying nothing until captured, a drag opening verification debt), on-screen status texts, the opening frame as evidence, the CAPTCHA root read and the toolbar check keeping the vision dispatch.test/quality/vision-mode-delete-mv3.mjs(--vague): the installed MV3 extension deleting a note withclick_aton its Delete button and on the site's own confirmation; a request that names the note asks nothing, "delete it" asks once on WebBrain's card, and exactly that note is deleted once.test/quality/vision-mode-repeat-mv3.mjs: the installed MV3 extension pressing a "+" button ten times at the same point (its counter is drawn on a canvas, so the DOM text never changes): all ten presses go out, and a button that does nothing is then stopped by the loop guard, in vision terms.test/monitor-vision-wheel.mjs: the real page monitor in Chromium: two wheel notches fenced before their events arrive stay the agent's (a third, unfenced one is a person's), page churn does not void a prepared wheel, a person's click does.test/vision-pipeline-browser.mjs: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (caret placement, word and line selection, pointer transitions, wheel routing, HTML5 drag and drop, drags in a same-origin frame, native range inputs (dragged, and clicked on the track), text selection, disabled controls, the file-chooser guard, CSS:hoverrevealed under the pointer (a CSS-only menu, a selector listed beside a:hoverkeeping its own cascade, a later rule of equal specificity still winning, a rule inside@media, a shadow root's:host(:hover), a rule added later) and ended by a person's own pointer with the page's selectors put back, pointer capture taken on the press: a thumb with its handlers on itself, a track that captures its thumb, a control that drops the implicit capture, run as from the page and as from a Firefox content script; clicks and drags in a scaled frame and in a frame inside a scaled ancestor, a rotated frame and another origin's frame refused); the DOM probe and the Chrome click submit probe in a scaled, bordered frame; the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames; the click submit probe at a vision point inside open shadow roots and on plain content, in both builds.test/quality/vision-mode-mv3.mjs(--dpr=2,--zoom=1.25,--scroll=100, combinable): the installed MV3 extension with a loopback model that reads the frame labels and fills and submits a form withclick_atandtype_text(sent together in one response), scrolls withscroll_atand follows atarget=_blanklink; two colored markers (viewport-fixed and in the document) must appear in the first frame exactly where the label's scale puts them. The page reacts to the click's focus and a ticker changes it every 250 ms: typing must land on the first try, the wheel must scroll the page, the link's tab must become the run's tab with no extra tab left, no request may carry tree text, and until the link navigates every decision gets exactly one screen update and no browser-state message.
test/firefox-vision-e2e.mjs (npm run test:firefox-vision:e2e; set FIREFOX_BINARY if Firefox is not on PATH or in the macOS default location) runs the Firefox input path in a real headless Firefox. It installs the Firefox build over WebDriver BiDi into an isolated profile and sends vision_input to the real content script, with its Xray world, wrappedJSObject and exportFunction. It checks a CSS :hover menu and the restore when a person moves the mouse, a thumb that captures the pointer, a control that drops the implicit capture, a click on a range track, word and drag text selection, a wheel, a click in a scale(0.5) frame and the refusal of another origin's frame. It is not part of npm run test:vision-mode.
The older MV3 quality fixtures script an accessibility-tree model (they answer with ref_ids read from get_accessibility_tree) under model names that WebBrain recognizes as vision-capable, so they seed visionMode: 'off' with their other settings. A new fixture that scripts tree tools needs the same line.
Limits
upload_fileis not among the vision tools: a click that would open a native file chooser is blocked in both builds and hands the next steps to the element tools for the upload (see above). The guard andupload_filereach the top document and its open shadow roots only, so a file input inside an iframe or a closed shadow root still cannot be uploaded to (Chrome still stops its chooser, through the protocol guard). Page-resource downloads by selector are not offered in vision mode; switch it off for those tasks.- Closed shadow roots and canvas apps are reachable by pointer, but a native
<select>popup is never visible in a screenshot: such a field is focused andtype_textpicks the option. - HTML5 drag and drop: in Firefox a drag that starts on a
draggable="true"item runs the drag-and-drop event sequence itself with a constructedDataTransfer; in Chrome the CDP pointer drag is left to the browser, as fordrag_drop, so a page that relies only ondataTransfermay not react there. - A drag is not shown to the submit, money or publishing checks; it is refused when it would click a control, but a custom "slide to confirm" widget is only as safe as the page makes it.
- The 45-second window that holds back a second click on the same committing caption covers text clicks (
click({text})) only. Like a ref click on the tree path, a repeated vision click relies on the submit confirmation and the deletion and publication guards instead. - Firefox input is synthetic (
isTrusted: false) unless the companion sends it (a left click or a hover, above); a page that requires trusted events for anything else needs Chrome. The emulated:hovercovers the style sheets the content script can read: a:hoverrule in another origin's style sheet does not apply, page scripts that test:hoverthemselves see no hover, and while it lasts a script reading a rule'sselectorTextsees the added:is().