Open-source AI browser agent for Chrome and Firefox (monorepo) ðŸ§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Run a translated click_at only while click_at is offered
Emre Sokulluf156bd1df536
diff --git a/docs/vision-mode.md b/docs/vision-mode.mdindex 3352fa3e0..9cac82d2e 100644--- a/docs/vision-mode.md+++ b/docs/vision-mode.md@@ -165,7 +165,7 @@ npm run test:vision-mode```- `test/vision-mode.mjs`: sizing, profiles, token formulas, diffs, micro changes, sessions, translation, tool surfaces of every tier in both builds, permission classification, prompt, the page-feedback policy for vision calls and the monitor's vision rebase.-- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation, patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools (the page monitor prepared with `hover_at`'s and `drag_at`'s CSS points, a wheel with none), fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; a drag after an earlier action of its batch re-checked at both ends; a window resized while the model decides (a fresh full frame first) and one page feedback has not reported yet (the batch's points refused); a `take_screenshot` and `done` in one batch after an update that could not be captured, and a `drag_at` finished in its own batch, both refused as unverified; a zoom after an earlier action of its batch usable beside that batch's patch except where the screen changed after it; in Chrome, a click that opened no tab costing no poll, tabs opened at once or later being rerouted, and a late-tab watch staying armed past an update and rerouting a tab opened while the screen settled.+- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation (and a `click_at` in a turn that does not offer it, such as a completion recovery turn, refused before it can run as a click), patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools (the page monitor prepared with `hover_at`'s and `drag_at`'s CSS points, a wheel with none), fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; a drag after an earlier action of its batch re-checked at both ends; a window resized while the model decides (a fresh full frame first) and one page feedback has not reported yet (the batch's points refused); a `take_screenshot` and `done` in one batch after an update that could not be captured, and a `drag_at` finished in its own batch, both refused as unverified; a zoom after an earlier action of its batch usable beside that batch's patch except where the screen changed after it; in Chrome, a click that opened no tab costing no poll, tabs opened at once or later being rerouted, and a late-tab watch staying armed past an update and rerouting a tab opened while the screen settled.- `test/vision-mode-guards.mjs`: the guards in a vision run, both builds: a vision click deleting the item the request names is bound to the control read at its point (the same point outside a vision run stays unbound), the deletion card, a changed item, a forbidden item and Enter keep their rules, and an authorized deletion is read again at dispatch, sending nothing when the control moved away; loop detection by screen (repeats after a change are progress, on an unchanged screen they stop), vision loop messages, native dropdowns, arrow keys and clicks without DOM-snapshot verdicts, page-focus churn, the delivery checkpoint, one screen update per batch however it ended; whole-thread reads and WhatsApp/Telegram on the tree path, the Gmail counter, observations and refusal wording after an action (a requested frame verifying nothing until captured, a drag opening verification debt), on-screen status texts, the opening frame as evidence, the CAPTCHA root read and the toolbar check keeping the vision dispatch.- `test/quality/vision-mode-delete-mv3.mjs` (`--vague`): the installed MV3 extension deleting a note with `click_at` on its Delete button and on the site's own confirmation; a request that names the note asks nothing, "delete it" asks once on WebBrain's card, and exactly that note is deleted once.- `test/quality/vision-mode-repeat-mv3.mjs`: the installed MV3 extension pressing a "+" button ten times at the same point (its counter is drawn on a canvas, so the DOM text never changes): all ten presses go out, and a button that does nothing is then stopped by the loop guard, in vision terms.diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.jsindex 074828bdc..fbb9ca348 100644--- a/src/chrome/src/agent/agent.js+++ b/src/chrome/src/agent/agent.js@@ -16238,18 +16238,23 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dconst jevPending = this._jevPendingCalls?.get(tc.id);const fnName = tc.function?.name || '';- // A translated click_at runs as `click`, which vision runs do not advertise.- if (!allowedToolNames.has(fnName) && tc._vision?.internalName !== fnName) {+ // A translated click_at runs as `click`, which vision runs do not+ // advertise: it may run only while click_at itself is offered (a+ // completion recovery turn that offers observation and done does not).+ const translatedTool = tc._vision?.internalName && tc._vision.internalName === fnName ? tc._vision.tool || '' : '';+ if (!allowedToolNames.has(fnName) && !(translatedTool && allowedToolNames.has(translatedTool))) {if (fnName) {// A named but unadvertised tool is a rejection before dispatch, so// it goes through loop accounting; otherwise a model can repeat a// retired screenshot call for the whole run. The detector counts it// by name alone (see _checkUnavailableToolLoop), so varying the- // arguments does not evade it, and never as progress.- const result = this._unavailableToolResult(fnName, tabId);- const recovery = await recordPreparationFailure(toolIndex, fnName, {}, result, result.error);+ // arguments does not evade it, and never as progress. A translated+ // call is named as the model called it.+ const calledName = translatedTool || fnName;+ const result = this._unavailableToolResult(calledName, tabId);+ const recovery = await recordPreparationFailure(toolIndex, calledName, {}, result, result.error);if (recovery) return recovery;- if (interruptFailedBrowserAction(toolIndex, fnName)) { navNotices.length = 0; break; }+ if (interruptFailedBrowserAction(toolIndex, calledName)) { navNotices.length = 0; break; }continue;}const error = 'Tool call is missing a function name.';diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.jsindex 3f8c0b7e3..71c2cb3d4 100644--- a/src/firefox/src/agent/agent.js+++ b/src/firefox/src/agent/agent.js@@ -14459,18 +14459,23 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dconst jevPending = this._jevPendingCalls?.get(tc.id);const fnName = tc.function?.name || '';- // A translated click_at runs as `click`, which vision runs do not advertise.- if (!allowedToolNames.has(fnName) && tc._vision?.internalName !== fnName) {+ // A translated click_at runs as `click`, which vision runs do not+ // advertise: it may run only while click_at itself is offered (a+ // completion recovery turn that offers observation and done does not).+ const translatedTool = tc._vision?.internalName && tc._vision.internalName === fnName ? tc._vision.tool || '' : '';+ if (!allowedToolNames.has(fnName) && !(translatedTool && allowedToolNames.has(translatedTool))) {if (fnName) {// A named but unadvertised tool is a rejection before dispatch, so// it goes through loop accounting; otherwise a model can repeat a// retired screenshot call for the whole run. The detector counts it// by name alone (see _checkUnavailableToolLoop), so varying the- // arguments does not evade it, and never as progress.- const result = this._unavailableToolResult(fnName, tabId);- const recovery = await recordPreparationFailure(toolIndex, fnName, {}, result, result.error);+ // arguments does not evade it, and never as progress. A translated+ // call is named as the model called it.+ const calledName = translatedTool || fnName;+ const result = this._unavailableToolResult(calledName, tabId);+ const recovery = await recordPreparationFailure(toolIndex, calledName, {}, result, result.error);if (recovery) return recovery;- if (interruptFailedBrowserAction(toolIndex, fnName)) { navNotices.length = 0; break; }+ if (interruptFailedBrowserAction(toolIndex, calledName)) { navNotices.length = 0; break; }continue;}const error = 'Tool call is missing a function name.';diff --git a/test/vision-mode-agent.mjs b/test/vision-mode-agent.mjsindex bebef1c67..47c34c3ab 100644--- a/test/vision-mode-agent.mjs+++ b/test/vision-mode-agent.mjs@@ -185,6 +185,26 @@ for (const build of (process.env.VISION_BUILDS || 'chrome,firefox').split(','))assert.equal(assistant.tool_calls[0].function.name, 'click_at');});+ test(`${label}: a turn that does not offer click_at does not run one as a click`, async () => {+ const { agent, requests, dispatched } = runAgent(step => {+ if (step === 1) return { toolCalls: [call('click_at', { x: 648, y: 405 })] };+ if (step === 2) return { toolCalls: [call('done', { summary: 'Checked.', outcome: 'partial' })] };+ return { content: 'done' };+ });+ // A completion recovery turn that offers only observation and done.+ let policyCalls = 0;+ agent._completionRecoveryPolicy = (_tab, tools) => (++policyCalls === 1+ ? { kind: 'restricted', tools: tools.filter(tool => ['take_screenshot', 'done'].includes(tool.function.name)), toolChoice: null }+ : null);+ const tab = nextTab++;+ await runLoop(agent, tab, streaming);+ assert.deepEqual(requests[0].tools.sort(), ['done', 'take_screenshot']);+ assert.equal(dispatched.some(entry => entry.name === 'click'), false, JSON.stringify(dispatched));+ const assistant = requests[1].messages.find(message => message.role === 'assistant' && message.tool_calls?.[0]?.function?.name === 'click_at');+ const refusal = requests[1].messages.find(message => message.role === 'tool' && message.tool_call_id === assistant?.tool_calls[0].id);+ assert.match(String(refusal?.content), /Tool click_at is not available/, String(refusal?.content));+ });+test(`${label}: no visible change is reported in one line, without an image`, async () => {const { agent, requests } = runAgent(step => {if (step === 1) return { toolCalls: [call('click_at', { x: 10, y: 10 })] };