Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Stop a run that keeps repeating a refused pre-submit action

Emre Sokulluda0fc6cc98f0
diff --git a/docs/vision-mode.md b/docs/vision-mode.md
index 9b8f5e2cd..4244e7068 100644
--- a/docs/vision-mode.md
+++ b/docs/vision-mode.md
@@ -149,6 +149,7 @@ Every hard guard and quality check runs on vision calls; these are the places wh
- **The submit probe** reads a point inside open shadow roots, as the pointer events reach it, so a Delete or Pay button in a web component is the control, not its host. A vision click on plain content (no control, link, field, label or inline handler on the element or around it, and no frame, embed, canvas or application region, whose controls the DOM does not show: a post's text, a picture, blank space) is read as a non-submit target (`resolvedPlainContent`): it can open what the page opens for it but cannot submit or publish, so the X/Bluesky publication guard and the post-correction guard let it through instead of asking for a ref.
- **Deletions and commits after a correction** bind a vision click to the control read at its point and read it again at dispatch (see Input dispatch and safety).
- **Loop detection** keys a vision action on the screen it acted on (`VisionSession.screenVersion`, bumped by a patch or a frame with other pixels, not by "no visible change" or a `take_screenshot` of the same screen; `_loopScreenIdentity`). Accepting ten invitations under the same point, pressing Enter after each typed item or scrolling to the next stretch of a list is progress; the same point, key or wheel on a screen that does not change still nudges and stops (the coordinate stop after eight clicks, with the model's own point in the message). Whether a click or an arrow key changed anything is for the screen update to say: the DOM text snapshot behind `noProgress` is not taken in a vision run, since it cannot see a canvas, a map or a value it does not read. Choosing from a native dropdown (`nativeSelect`) is never a failed click. Loop, stop and recovery messages name `click_at`, `zoom` and the screen instead of selectors, refs and the tree.
+- **Refusals without dispatch**: a call a pre-dispatch guard refuses (publication audit, commit binding, resource mutation, a prior submission after steering) never reaches the loop detector, in any run. It has its own count per run (`_noteRefusalWithoutDispatch`): the third identical refusal (arguments compared without image ids) or the sixth of any kind stops the run as the loop detector would, and any call that goes through resets the count.
- **The delivery checkpoint** counts `take_screenshot`, `scroll_at` and `wait` like the tree's reads, scrolls and waits, and a screen update that shows a change ends the streak, so only looking at a screen that stays the same adds up. Zooms are left out: reading a dense screen takes several.
- **Page feedback** keeps a response of `take_screenshot`, `zoom` or `wait` when a page script moves focus: none of them acts on focus.
- **Completion**: every batch, including one that ends early (a refused `done`, a completion hand-back), is followed by one screen update (`_executeToolBatch`), so the model never acts again on the frame its last actions were chosen on; `done`'s own verification screenshot, an unlabeled device-scale image, is not taken. The opening frame counts as the run's first look at the page (plan evidence); after an action a zoom counts as an observation, as does a screen update once it was captured (`auto_screenshot`), but not `take_screenshot` itself, which only asks for the frame the update after its batch captures; a `drag_at` is a consequential action like the `drag_drop` it replaces; and the status texts on screen (toasts, alerts, live regions) are recorded with each update while a submission waits for proof. Refusals point to the screen update, `take_screenshot`, `read_page` and `wait` instead of `get_accessibility_tree`, `verify_form` or `wait_for_element`. `gmail_count_results`, which takes no target, stays in the vision tool list for Gmail's count job.
diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.js
index dd5d4d646..2886de037 100644
--- a/src/chrome/src/agent/agent.js
+++ b/src/chrome/src/agent/agent.js
@@ -16854,6 +16854,17 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
this._persist(tabId);
return { action: 'return', value: offer?.message || workflowPreSubmitBlock.error, status: 'publication_audit_failed' };
}
+ // The loop detector never sees a call refused here: repeating it is
+ // bounded by its own count instead (_noteRefusalWithoutDispatch).
+ const refusalStop = this._noteRefusalWithoutDispatch(tabId, fnName, fnArgs, workflowPreSubmitBlock);
+ if (refusalStop) {
+ this._appendSyntheticToolResults(tabId, toolCalls, toolIndex + 1, messages, onUpdate, step,
+ () => ({ success: false, skipped: true, error: 'skipped: run stopped after repeated refusals' }));
+ const refusalRunId = this.currentRunId.get(tabId);
+ if (refusalRunId) trace.recordError(refusalRunId, step, 'loop', refusalStop);
+ this._persist(tabId);
+ return { action: 'recover', value: refusalStop, status: 'loop_stopped' };
+ }
onUpdate('warning', { message: workflowPreSubmitBlock.error });
if (workflowPreSubmitBlock.workflowRearmed || workflowPreSubmitBlock.conditionalMutationBlocked) {
this._appendSyntheticToolResults(tabId, toolCalls, toolIndex + 1, messages, onUpdate, step,
@@ -17792,7 +17803,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
}));
return this._pageFeedbackRecoveryResult(tabId, messages, onUpdate) || { action: 'continue' };
}
- if (toolResult?.success !== false && toolResult?.noDispatch !== true && toolResult?.dispatched !== false) this._resetPageFeedbackProgress(tabId);
+ if (toolResult?.success !== false && toolResult?.noDispatch !== true && toolResult?.dispatched !== false) {
+ this._resetPageFeedbackProgress(tabId);
+ this._guardRunState(tabId).refusals = null;
+ }
// A vision click on a file input hands the next steps to upload_file,
// and a successful upload hands them back (vision-runtime.js).
toolResult = this._visionNoteUploadHandoff(tabId, fnName, toolResult, { vision: tc._vision, promptTier });
@@ -22903,6 +22917,27 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
return true;
}
+ /**
+ * A call a pre-dispatch guard refused (publication audit, commit binding,
+ * resource mutation, a prior submission after steering) never reaches the
+ * loop detector. Counted per run until a call goes through: the third
+ * identical refusal (arguments compared without image ids) or the sixth
+ * of any kind returns the message that stops the run.
+ */
+ _noteRefusalWithoutDispatch(tabId, name, args, result) {
+ const state = this._guardRunState(tabId);
+ const refusals = state.refusals ||= { total: 0, bySignature: new Map() };
+ const proposal = JSON.stringify(args || {}, (key, value) => (['image', 'keyframeId', 'imageId'].includes(key) ? undefined : value));
+ const reason = String(result?.errorCode || result?.reasonCode || result?.code || '').slice(0, 80);
+ const signature = JSON.stringify([name, reason, String(result?.error || '').slice(0, 160), proposal]);
+ const repeats = (refusals.bySignature.get(signature) || 0) + 1;
+ refusals.bySignature.set(signature, repeats);
+ refusals.total += 1;
+ if (repeats < 3 && refusals.total < 6) return null;
+ const detail = String(result?.error || 'the action was refused').replace(/\s+/g, ' ').trim().slice(0, 240);
+ return `Stopped: WebBrain refused ${repeats >= 3 ? 'the same action three times' : 'six actions'} in a row without any action going through (${detail}). The task is not complete; nothing more was sent, and any draft is left as it is.`;
+ }
+
_guardRunState(tabId) {
// Untraced runs (the default) have no currentRunId; the task token
// _startTraceRun mints for every run bounds them instead.
diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.js
index 36ea724e9..cedc35361 100644
--- a/src/firefox/src/agent/agent.js
+++ b/src/firefox/src/agent/agent.js
@@ -15007,6 +15007,17 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
this._persist(tabId);
return { action: 'return', value: offer?.message || workflowPreSubmitBlock.error, status: 'publication_audit_failed' };
}
+ // The loop detector never sees a call refused here: repeating it is
+ // bounded by its own count instead (_noteRefusalWithoutDispatch).
+ const refusalStop = this._noteRefusalWithoutDispatch(tabId, fnName, fnArgs, workflowPreSubmitBlock);
+ if (refusalStop) {
+ this._appendSyntheticToolResults(tabId, toolCalls, toolIndex + 1, messages, onUpdate, step,
+ () => ({ success: false, skipped: true, error: 'skipped: run stopped after repeated refusals' }));
+ const refusalRunId = this.currentRunId.get(tabId);
+ if (refusalRunId) trace.recordError(refusalRunId, step, 'loop', refusalStop);
+ this._persist(tabId);
+ return { action: 'recover', value: refusalStop, status: 'loop_stopped' };
+ }
onUpdate('warning', { message: workflowPreSubmitBlock.error });
if (workflowPreSubmitBlock.workflowRearmed || workflowPreSubmitBlock.conditionalMutationBlocked) {
this._appendSyntheticToolResults(tabId, toolCalls, toolIndex + 1, messages, onUpdate, step,
@@ -15916,7 +15927,10 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
}));
return this._pageFeedbackRecoveryResult(tabId, messages, onUpdate) || { action: 'continue' };
}
- if (toolResult?.success !== false && toolResult?.noDispatch !== true && toolResult?.dispatched !== false) this._resetPageFeedbackProgress(tabId);
+ if (toolResult?.success !== false && toolResult?.noDispatch !== true && toolResult?.dispatched !== false) {
+ this._resetPageFeedbackProgress(tabId);
+ this._guardRunState(tabId).refusals = null;
+ }
// A vision click on a file input hands the next steps to upload_file,
// and a successful upload hands them back (vision-runtime.js).
toolResult = this._visionNoteUploadHandoff(tabId, fnName, toolResult, { vision: tc._vision, promptTier });
@@ -19706,6 +19720,27 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
return true;
}
+ /**
+ * A call a pre-dispatch guard refused (publication audit, commit binding,
+ * resource mutation, a prior submission after steering) never reaches the
+ * loop detector. Counted per run until a call goes through: the third
+ * identical refusal (arguments compared without image ids) or the sixth
+ * of any kind returns the message that stops the run.
+ */
+ _noteRefusalWithoutDispatch(tabId, name, args, result) {
+ const state = this._guardRunState(tabId);
+ const refusals = state.refusals ||= { total: 0, bySignature: new Map() };
+ const proposal = JSON.stringify(args || {}, (key, value) => (['image', 'keyframeId', 'imageId'].includes(key) ? undefined : value));
+ const reason = String(result?.errorCode || result?.reasonCode || result?.code || '').slice(0, 80);
+ const signature = JSON.stringify([name, reason, String(result?.error || '').slice(0, 160), proposal]);
+ const repeats = (refusals.bySignature.get(signature) || 0) + 1;
+ refusals.bySignature.set(signature, repeats);
+ refusals.total += 1;
+ if (repeats < 3 && refusals.total < 6) return null;
+ const detail = String(result?.error || 'the action was refused').replace(/\s+/g, ' ').trim().slice(0, 240);
+ return `Stopped: WebBrain refused ${repeats >= 3 ? 'the same action three times' : 'six actions'} in a row without any action going through (${detail}). The task is not complete; nothing more was sent, and any draft is left as it is.`;
+ }
+
_guardRunState(tabId) {
// Untraced runs (the default) have no currentRunId; the task token
// _startTraceRun mints for every run bounds them instead.
diff --git a/test/vision-mode-agent.mjs b/test/vision-mode-agent.mjs
index b6983041d..86dde0797 100644
--- a/test/vision-mode-agent.mjs
+++ b/test/vision-mode-agent.mjs
@@ -689,6 +689,40 @@ for (const build of (process.env.VISION_BUILDS || 'chrome,firefox').split(','))
assert.ok(['none', 'patch', 'key'].includes(messages[update].webbrainVisionFrame.kind), textOf(messages[update]));
});
+ test(`${build}: a click a pre-submit guard keeps refusing stops the run on the third identical refusal`, async () => {
+ const ctx = runAgent((step, _messages, options) => (names(options.tools).includes('click_at')
+ ? { toolCalls: [call('click_at', { x: 300, y: 200 })] }
+ : { toolCalls: [call('done', { summary: 'Not sent.', outcome: 'partial' })] }), { maxSteps: 10 });
+ const refused = [];
+ ctx.agent._workflowPreSubmitDispatchBlock = async (_tab, name, args) => {
+ if (name !== 'click') return null;
+ refused.push(args);
+ return { success: false, dispatched: false, noDispatch: true, errorCode: 'publication_audit_blocked', error: 'The publication audit refused the draft.' };
+ };
+ const result = await runLoop(ctx.agent, nextTab++, false);
+ assert.equal(refused.length, 3, 'the third identical refusal stopped it');
+ assert.equal(ctx.dispatched.filter(entry => entry.name === 'click').length, 0);
+ assert.match(JSON.stringify(result), /refused the same action three times/);
+ });
+
+ test(`${build}: refusals stop counting once an action goes through`, () => {
+ const agent = new Agent({});
+ const tab = 77;
+ const refusal = { success: false, noDispatch: true, errorCode: 'x', error: 'Refused.' };
+ assert.equal(agent._noteRefusalWithoutDispatch(tab, 'click', { x: 1, y: 2, image: 'f1' }, refusal), null);
+ assert.equal(agent._noteRefusalWithoutDispatch(tab, 'click', { x: 1, y: 2, image: 'f2' }, refusal), null);
+ // A dispatched action in between resets the count (the batch clears it).
+ agent._guardRunState(tab).refusals = null;
+ assert.equal(agent._noteRefusalWithoutDispatch(tab, 'click', { x: 1, y: 2, image: 'f3' }, refusal), null);
+ assert.equal(agent._noteRefusalWithoutDispatch(tab, 'click', { x: 1, y: 2, image: 'f4' }, refusal), null);
+ assert.match(agent._noteRefusalWithoutDispatch(tab, 'click', { x: 1, y: 2, image: 'f5' }, refusal), /the same action three times/);
+ // Six different refusals in a row also stop.
+ const other = new Agent({});
+ const stops = [1, 2, 3, 4, 5, 6].map(i => other._noteRefusalWithoutDispatch(tab, 'click', { x: i, y: i }, refusal));
+ assert.deepEqual(stops.map(Boolean), [false, false, false, false, false, true]);
+ assert.match(stops[5], /six actions/);
+ });
+
test(`${build}: a new run never shows or uses the previous run's frame when its first capture fails`, async () => {
const ctx = runAgent(step => {
if (step === 1) return { toolCalls: [call('done', { summary: 'Looked.', outcome: 'success' })] };