Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Run a translated click_at only while click_at is offered

Emre Sokulluf156bd1df536
diff --git a/docs/vision-mode.md b/docs/vision-mode.md
index 3352fa3e0..9cac82d2e 100644
--- a/docs/vision-mode.md
+++ b/docs/vision-mode.md
@@ -165,7 +165,7 @@ npm run test:vision-mode
```
- `test/vision-mode.mjs`: sizing, profiles, token formulas, diffs, micro changes, sessions, translation, tool surfaces of every tier in both builds, permission classification, prompt, the page-feedback policy for vision calls and the monitor's vision rebase.
-- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation, patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools (the page monitor prepared with `hover_at`'s and `drag_at`'s CSS points, a wheel with none), fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; a drag after an earlier action of its batch re-checked at both ends; a window resized while the model decides (a fresh full frame first) and one page feedback has not reported yet (the batch's points refused); a `take_screenshot` and `done` in one batch after an update that could not be captured, and a `drag_at` finished in its own batch, both refused as unverified; a zoom after an earlier action of its batch usable beside that batch's patch except where the screen changed after it; in Chrome, a click that opened no tab costing no poll, tabs opened at once or later being rerouted, and a late-tab watch staying armed past an update and rerouting a tab opened while the screen settled.
+- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation (and a `click_at` in a turn that does not offer it, such as a completion recovery turn, refused before it can run as a click), patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools (the page monitor prepared with `hover_at`'s and `drag_at`'s CSS points, a wheel with none), fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; a drag after an earlier action of its batch re-checked at both ends; a window resized while the model decides (a fresh full frame first) and one page feedback has not reported yet (the batch's points refused); a `take_screenshot` and `done` in one batch after an update that could not be captured, and a `drag_at` finished in its own batch, both refused as unverified; a zoom after an earlier action of its batch usable beside that batch's patch except where the screen changed after it; in Chrome, a click that opened no tab costing no poll, tabs opened at once or later being rerouted, and a late-tab watch staying armed past an update and rerouting a tab opened while the screen settled.
- `test/vision-mode-guards.mjs`: the guards in a vision run, both builds: a vision click deleting the item the request names is bound to the control read at its point (the same point outside a vision run stays unbound), the deletion card, a changed item, a forbidden item and Enter keep their rules, and an authorized deletion is read again at dispatch, sending nothing when the control moved away; loop detection by screen (repeats after a change are progress, on an unchanged screen they stop), vision loop messages, native dropdowns, arrow keys and clicks without DOM-snapshot verdicts, page-focus churn, the delivery checkpoint, one screen update per batch however it ended; whole-thread reads and WhatsApp/Telegram on the tree path, the Gmail counter, observations and refusal wording after an action (a requested frame verifying nothing until captured, a drag opening verification debt), on-screen status texts, the opening frame as evidence, the CAPTCHA root read and the toolbar check keeping the vision dispatch.
- `test/quality/vision-mode-delete-mv3.mjs` (`--vague`): the installed MV3 extension deleting a note with `click_at` on its Delete button and on the site's own confirmation; a request that names the note asks nothing, "delete it" asks once on WebBrain's card, and exactly that note is deleted once.
- `test/quality/vision-mode-repeat-mv3.mjs`: the installed MV3 extension pressing a "+" button ten times at the same point (its counter is drawn on a canvas, so the DOM text never changes): all ten presses go out, and a button that does nothing is then stopped by the loop guard, in vision terms.
diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.js
index 074828bdc..fbb9ca348 100644
--- a/src/chrome/src/agent/agent.js
+++ b/src/chrome/src/agent/agent.js
@@ -16238,18 +16238,23 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
const jevPending = this._jevPendingCalls?.get(tc.id);
const fnName = tc.function?.name || '';
- // A translated click_at runs as `click`, which vision runs do not advertise.
- if (!allowedToolNames.has(fnName) && tc._vision?.internalName !== fnName) {
+ // A translated click_at runs as `click`, which vision runs do not
+ // advertise: it may run only while click_at itself is offered (a
+ // completion recovery turn that offers observation and done does not).
+ const translatedTool = tc._vision?.internalName && tc._vision.internalName === fnName ? tc._vision.tool || '' : '';
+ if (!allowedToolNames.has(fnName) && !(translatedTool && allowedToolNames.has(translatedTool))) {
if (fnName) {
// A named but unadvertised tool is a rejection before dispatch, so
// it goes through loop accounting; otherwise a model can repeat a
// retired screenshot call for the whole run. The detector counts it
// by name alone (see _checkUnavailableToolLoop), so varying the
- // arguments does not evade it, and never as progress.
- const result = this._unavailableToolResult(fnName, tabId);
- const recovery = await recordPreparationFailure(toolIndex, fnName, {}, result, result.error);
+ // arguments does not evade it, and never as progress. A translated
+ // call is named as the model called it.
+ const calledName = translatedTool || fnName;
+ const result = this._unavailableToolResult(calledName, tabId);
+ const recovery = await recordPreparationFailure(toolIndex, calledName, {}, result, result.error);
if (recovery) return recovery;
- if (interruptFailedBrowserAction(toolIndex, fnName)) { navNotices.length = 0; break; }
+ if (interruptFailedBrowserAction(toolIndex, calledName)) { navNotices.length = 0; break; }
continue;
}
const error = 'Tool call is missing a function name.';
diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.js
index 3f8c0b7e3..71c2cb3d4 100644
--- a/src/firefox/src/agent/agent.js
+++ b/src/firefox/src/agent/agent.js
@@ -14459,18 +14459,23 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
const jevPending = this._jevPendingCalls?.get(tc.id);
const fnName = tc.function?.name || '';
- // A translated click_at runs as `click`, which vision runs do not advertise.
- if (!allowedToolNames.has(fnName) && tc._vision?.internalName !== fnName) {
+ // A translated click_at runs as `click`, which vision runs do not
+ // advertise: it may run only while click_at itself is offered (a
+ // completion recovery turn that offers observation and done does not).
+ const translatedTool = tc._vision?.internalName && tc._vision.internalName === fnName ? tc._vision.tool || '' : '';
+ if (!allowedToolNames.has(fnName) && !(translatedTool && allowedToolNames.has(translatedTool))) {
if (fnName) {
// A named but unadvertised tool is a rejection before dispatch, so
// it goes through loop accounting; otherwise a model can repeat a
// retired screenshot call for the whole run. The detector counts it
// by name alone (see _checkUnavailableToolLoop), so varying the
- // arguments does not evade it, and never as progress.
- const result = this._unavailableToolResult(fnName, tabId);
- const recovery = await recordPreparationFailure(toolIndex, fnName, {}, result, result.error);
+ // arguments does not evade it, and never as progress. A translated
+ // call is named as the model called it.
+ const calledName = translatedTool || fnName;
+ const result = this._unavailableToolResult(calledName, tabId);
+ const recovery = await recordPreparationFailure(toolIndex, calledName, {}, result, result.error);
if (recovery) return recovery;
- if (interruptFailedBrowserAction(toolIndex, fnName)) { navNotices.length = 0; break; }
+ if (interruptFailedBrowserAction(toolIndex, calledName)) { navNotices.length = 0; break; }
continue;
}
const error = 'Tool call is missing a function name.';
diff --git a/test/vision-mode-agent.mjs b/test/vision-mode-agent.mjs
index bebef1c67..47c34c3ab 100644
--- a/test/vision-mode-agent.mjs
+++ b/test/vision-mode-agent.mjs
@@ -185,6 +185,26 @@ for (const build of (process.env.VISION_BUILDS || 'chrome,firefox').split(','))
assert.equal(assistant.tool_calls[0].function.name, 'click_at');
});
+ test(`${label}: a turn that does not offer click_at does not run one as a click`, async () => {
+ const { agent, requests, dispatched } = runAgent(step => {
+ if (step === 1) return { toolCalls: [call('click_at', { x: 648, y: 405 })] };
+ if (step === 2) return { toolCalls: [call('done', { summary: 'Checked.', outcome: 'partial' })] };
+ return { content: 'done' };
+ });
+ // A completion recovery turn that offers only observation and done.
+ let policyCalls = 0;
+ agent._completionRecoveryPolicy = (_tab, tools) => (++policyCalls === 1
+ ? { kind: 'restricted', tools: tools.filter(tool => ['take_screenshot', 'done'].includes(tool.function.name)), toolChoice: null }
+ : null);
+ const tab = nextTab++;
+ await runLoop(agent, tab, streaming);
+ assert.deepEqual(requests[0].tools.sort(), ['done', 'take_screenshot']);
+ assert.equal(dispatched.some(entry => entry.name === 'click'), false, JSON.stringify(dispatched));
+ const assistant = requests[1].messages.find(message => message.role === 'assistant' && message.tool_calls?.[0]?.function?.name === 'click_at');
+ const refusal = requests[1].messages.find(message => message.role === 'tool' && message.tool_call_id === assistant?.tool_calls[0].id);
+ assert.match(String(refusal?.content), /Tool click_at is not available/, String(refusal?.content));
+ });
+
test(`${label}: no visible change is reported in one line, without an image`, async () => {
const { agent, requests } = runAgent(step => {
if (step === 1) return { toolCalls: [call('click_at', { x: 10, y: 10 })] };