Open-source AI browser agent for Chrome and Firefox (monorepo) ðŸ§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
changelog
Emre Sokullu52bc5ffc07cd
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 96d9c3083..20b6cfa96 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -6,9 +6,27 @@ This changelog was generated from the repository Git history and release tags. V## [Unreleased]+## [44.0.0] - 2026-10-27++### Added+- Vision mode: Act runs whose model takes images now see the page as screenshots and act with the mouse and keyboard at coordinates, in Chrome and Firefox. It is the default for image-capable models; Settings → Advanced → **Vision mode** offers **Auto** and **Off**, and a saved On or true reads as Auto. A run starts from a full frame; after each action WebBrain sends only what changed on screen, a new frame when most of the screen changed, or the note "no visible change". The new tools are `click_at`, `scroll_at`, `hover_at`, `drag_at`, `zoom`, `take_screenshot` and `wait`, and `type_text` types into the focused field. Ask, Dev, text-only models, whole-conversation reads, the WhatsApp Web and Telegram contact checks, and site workflows that need the element inventory keep the accessibility tree. See `docs/vision-mode.md`.+- Vision screenshots are sized for the provider that receives them. The **Vision screenshot size** setting offers Economy, Balanced and Sharp; each provider gets the largest image it accepts without shrinking it, and the per-provider grid keeps patches exact. Retina displays and browser zoom are divided out of the capture so a point the model reads maps back to the same CSS pixel. Models that point in 0..1000 coordinates, such as Gemini, Qwen3-VL, GLM-4.xV and InternVL, are handled as such, and the image format is WebP except where the provider or local server needs JPEG.+- Vision mode in Firefox reproduces what a person's pointer does: a click places the caret or selects a word, a double or triple click selects as it does in the browser, a wheel goes to the nearest scroller that can move, a drag moves native sliders and runs HTML5 drag and drop, a CSS-only hover menu opens, pointer capture is kept, and same-origin frames receive the point in their own coordinates. With **Firefox trusted automation** on and the companion reporting point input, a left click and a hover go out as trusted input instead. Chrome sends trusted CDP pointer events at the point. A click that would open a native file chooser is refused in both builds.++### Changed+- A vision run keeps only the current screen, its patches and its zooms in the model's request; older images become short placeholders. A local change that is too small to send accumulates until it is sent rather than being lost. Navigation, a viewport resize or a change of device pixel ratio starts a new full frame, and points read from an earlier screen or from an area that has since changed are refused before any input goes out.+- Vision steps are quicker. The settle wait ends on micro changes such as a caret or clock instead of waiting its full time, page feedback no longer reads the accessibility tree, an unchanged screen is not sent again before a decision, and a zoom taken after an action of its batch stays usable. The installed Chrome fixture's four steps went from about 15 seconds to about 7.5 seconds.+- A vision decision survives automatic page churn while the pixels around its point still match what the model saw, and a person's input, a navigation or another document still cancels it. Typing and key presses right after a click are checked by whether the clicked point took focus, so a page reacting to that focus no longer blocks them.+### Fixed+- Every hard guard works on coordinate clicks. A click or drag can no longer delete, commit a corrected change or press a disabled control without the same binding, confirmation and re-check at dispatch that a ref or selector click gets. The submit probe reads points inside open shadow roots and treats plain content as a non-submit target; a repeated click on a control, a drag that would click a control or cross a frame it cannot inspect, and a wheel over a focused number field are refused. Deletions, whole-thread reads and protected messaging pages keep the tree path where their checks need it.+- Loop detection now judges a vision action by whether the screen changed, so repeating a click after a visible change is progress and repeating it on an unchanged screen still stops the run. A drag opens completion verification, `take_screenshot` verifies nothing until the screen after its batch is actually captured, and a pending CAPTCHA gate gets WebBrain's own root tree read before the next action. A refused point stops the rest of its batch, and vision tool, capture and encoding failures become tool results or notes instead of ending the run.+- Chrome screenshots under browser zoom and on retina displays now cover the whole viewport and keep points in the right place. The legacy auto-screenshot clip now starts at the scroll offset, so scrolled pages are no longer captured blank, and it divides out the device pixel ratio instead of capturing at twice the intended scale.- Firefox: **download_social_media** saves media instead of stopping with "An unexpected error occurred", and after WebBrain is updated or reloaded, actions in tabs that were already open no longer fail because WebBrain could not load its page scripts into them again. Firefox looks up a script the background page injects from the background page's own folder, not from the extension's root, so every such injection named a file that does not exist. They now name the file from the root.+### Tests+- Vision mode has unit and run-loop coverage in both builds for sizing, diffs, sessions, tool surfaces, prompts, stale points, page churn, viewport resizes, guard behaviour and tree fallback. Chromium tests cover the real capture and dispatch path at device pixel ratios 1, 2 and 3 and at browser zoom, including a scrolled page and a `target=_blank` link. Installed-extension fixtures cover a form submitted at a vision point, a named and a vague deletion, and a canvas counter that needs ten clicks. Firefox input is tested in a real headless Firefox, including CSS hover, native range inputs, pointer capture, scaled frames and trusted input through the companion.+## [42.4.0] - 2026-10-24### Added