Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Read search form buttons at completion instead of matching "search|filter|go"

Claude7874bdc9384a
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 66a0f4095..9c86147f5 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,7 @@ This changelog was generated from the repository Git history and release tags. V
## [Unreleased]
### Fixed
+- When a task ends, a search box with an "Ara", "Suchen" or "Buscar" button no longer counts as a form left unsent. Whether such a button only searches is read by the model in any language, instead of looking for the English words "search", "filter" or "go", which also matched words like "Google".
- A click on a Mastodon post, or on any timeline that marks each post as an item of a feed, is no longer thrown away and asked for again because another post's time or content changed while WebBrain was deciding. Each post now owns its own buttons, and a timestamp ticking over ("2 dk" to "3 dk") is not a change. A change to the clicked post itself, such as its author or its own buttons, still stops the click.
- What a clicked button does is now read by a model in any language, instead of being matched against an English word list. A form's "Kaydet", "Devam" or "送信" is now checked for a validation error like "Save" or "Continue" was, and a quick second click on "Gönder" or "Pagar" is held as a possible duplicate like "Send" or "Pay" was, while "Ekle" or "Filtrele" stays repeatable. Each caption is read once and only after a click that page structure does not already explain, so ordinary clicks get no extra wait. The duplicate-click guard, which the security notes listed as missing in Firefox, works the same in both builds.
- Finishing a task no longer waits long on a slow model. The decision model decides when it is sure; when it answers but is not sure, the active model's second look now has eight seconds in all instead of up to ten seconds for each of two looks, and a look that could not finish in the time left is not started. When the decision model is off or unavailable, the active model keeps its full checks. The active model's reading of the page's status messages at that point is bounded to six seconds. Traces now show where a finished task spent its time and what each verification attempt decided, without any page or model text.
diff --git a/docs/architecture.md b/docs/architecture.md
index 67a9aab8c..88a445f29 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -882,6 +882,10 @@ for a rejection after a caption read as a commit or a next step; a workflow
click submits its form by structure or a caption read as a commit; and a rapid
second text click is a duplicate only for a caption read as a commit, asked
then if the first click left it unread. Words in a CSS selector are not read.
+At a successful done, a visible form made only of search fields, sent with
+GET and holding no other task evidence, counts as a task form still on screen
+unless every one of its submit captions is read as `other` (showing, filtering
+or searching); an empty or unread caption keeps it counted.
Requested field values (a message or post body, release notes, a title) come
from the intent model too. A short value is returned as is; a long one is
diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.js
index 4a11cbfc3..6f424b6b2 100644
--- a/src/chrome/src/agent/agent.js
+++ b/src/chrome/src/agent/agent.js
@@ -1652,6 +1652,29 @@ export class Agent extends LoopDetector {
|| (guard?.requiresSubmission == null && guard?.requiresStateChange === true) || !!guard?.siteWorkflow?.job;
}
+ // A visible form made only of search fields, sent with GET and holding no
+ // task evidence, is a search form unless its submit controls commit
+ // something. What their captions mean is read in any language: when every
+ // one is read as showing, filtering or searching, the form stops counting
+ // as a task form still on screen. A caption that is empty or not read keeps
+ // it counted.
+ async _readCompletionFormSubmits(tabId, pageState = null) {
+ const forms = Array.isArray(pageState?.formDescriptors) ? pageState.formDescriptors : [];
+ const candidates = forms.filter(form => form?.searchCandidate === true && form.relevant === true
+ && Array.isArray(form.submitLabels) && form.submitLabels.length === Number(form.submitCount)
+ && form.submitLabels.every(label => typeof label === 'string' && label.trim()));
+ if (!candidates.length) return;
+ await this._readPageTexts(tabId, 'control_effect', candidates.flatMap(form => form.submitLabels), {},
+ { maxChars: 200, mainTimeoutMs: COMPLETION_READING_TIMEOUT_MS });
+ for (const form of candidates) {
+ if (!form.submitLabels.every(label => this._controlEffectReading(String(label).replace(/\s+/g, ' ').trim()) === 'other')) continue;
+ form.relevant = false;
+ form.utility = true;
+ form.utilityReason = form.searchUtilityReason;
+ pageState.relevantFormCount = Math.max(0, Number(pageState.relevantFormCount || 0) - 1);
+ }
+ }
+
async _readCompletionStatusTexts(tabId, pageState = null) {
const submit = this._currentCompletionSubmitState(tabId);
const live = Array.isArray(pageState?.liveRegionMessages)
@@ -40294,6 +40317,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
// submit say, read in any language for the checks below. Success
// messages are only ever the agent's own readings, never page data.
const readingStartedAt = Date.now();
+ if (outcome === 'success' && pageState) await this._readCompletionFormSubmits(tabId, pageState);
if (outcome === 'success' && pageState) await this._readCompletionStatusTexts(tabId, pageState);
else if (pageState && typeof pageState === 'object') pageState.successMessages = [];
const verificationStartedAt = Date.now();
diff --git a/src/chrome/src/agent/completion-invariant.js b/src/chrome/src/agent/completion-invariant.js
index 9b3ae7926..98bbc89ba 100644
--- a/src/chrome/src/agent/completion-invariant.js
+++ b/src/chrome/src/agent/completion-invariant.js
@@ -175,6 +175,7 @@ export function classifyCompletionForm({
hiddenNamedCount = 0,
editable = [],
submits = [],
+ submitReadings = {},
} = {}) {
const fields = Array.isArray(editable) ? editable : [];
const submitControls = Array.isArray(submits) ? submits : [];
@@ -201,22 +202,31 @@ export function classifyCompletionForm({
// normal search-results state and is not task evidence by itself.
const searchFieldsOnly = normalizedFields.length > 0
&& normalizedFields.every(field => semanticSearchField(field) || conventionalSearchField(field));
- const searchSubmitsOnly = normalizedSubmits.every(control => /search|filter|go/i.test(control.label));
- const searchHasTaskEvidence = !!(
+ // What a submit control's caption means is read by a model in any language
+ // (the agent's control_effect readings, `submitReadings` by label): only
+ // controls read as showing, filtering or searching ("other") leave a
+ // search form a search form. An unread caption is no evidence, so it keeps
+ // the form relevant here; `searchCandidate` tells the agent to read it.
+ const readings = submitReadings && typeof submitReadings === 'object' ? submitReadings : {};
+ const readingOf = control => (Object.prototype.hasOwnProperty.call(readings, control.label) ? readings[control.label] : null);
+ const searchSubmitsOnly = normalizedSubmits.every(control => readingOf(control) === 'other');
+ const searchStructureHasTaskEvidence = !!(
insideDialog
|| normalizedMethod !== 'get'
|| Number(hiddenNamedCount || 0) !== 0
- || !searchSubmitsOnly
|| normalizedFields.some(field => (
field.required
|| field.focused
|| (!!field.name && !conventionalSearchField(field))
))
);
+ const searchHasTaskEvidence = searchStructureHasTaskEvidence || !searchSubmitsOnly;
// Primary content alone is not task evidence: result pages commonly place
// genuine search/filter forms there.
const safeSearchOnly = searchFieldsOnly && !searchHasTaskEvidence;
const hasSemanticSearchField = normalizedFields.some(semanticSearchField);
+ const searchCandidate = searchFieldsOnly && !searchStructureHasTaskEvidence && !searchSubmitsOnly
+ && normalizedSubmits.every(control => readingOf(control) === null || readingOf(control) === 'other');
const onlyField = normalizedFields.length === 1 ? normalizedFields[0] : null;
const passiveUtilityShell = !!(
@@ -251,6 +261,12 @@ export function classifyCompletionForm({
utilityReason,
editableCount: normalizedFields.length,
submitCount: normalizedSubmits.length,
+ ...(searchCandidate && !utility ? {
+ searchCandidate: true,
+ searchUtilityReason: hasSemanticSearchField ? 'semantic_search' : 'conventional_search',
+ // Every submit's caption, so the agent can tell when one has none to read.
+ submitLabels: normalizedSubmits.map(control => control.label.slice(0, 120)).slice(0, 4),
+ } : {}),
};
}
diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.js
index e65600cf4..a41e75dd3 100644
--- a/src/firefox/src/agent/agent.js
+++ b/src/firefox/src/agent/agent.js
@@ -1341,6 +1341,29 @@ export class Agent extends LoopDetector {
|| (guard?.requiresSubmission == null && guard?.requiresStateChange === true) || !!guard?.siteWorkflow?.job;
}
+ // A visible form made only of search fields, sent with GET and holding no
+ // task evidence, is a search form unless its submit controls commit
+ // something. What their captions mean is read in any language: when every
+ // one is read as showing, filtering or searching, the form stops counting
+ // as a task form still on screen. A caption that is empty or not read keeps
+ // it counted.
+ async _readCompletionFormSubmits(tabId, pageState = null) {
+ const forms = Array.isArray(pageState?.formDescriptors) ? pageState.formDescriptors : [];
+ const candidates = forms.filter(form => form?.searchCandidate === true && form.relevant === true
+ && Array.isArray(form.submitLabels) && form.submitLabels.length === Number(form.submitCount)
+ && form.submitLabels.every(label => typeof label === 'string' && label.trim()));
+ if (!candidates.length) return;
+ await this._readPageTexts(tabId, 'control_effect', candidates.flatMap(form => form.submitLabels), {},
+ { maxChars: 200, mainTimeoutMs: COMPLETION_READING_TIMEOUT_MS });
+ for (const form of candidates) {
+ if (!form.submitLabels.every(label => this._controlEffectReading(String(label).replace(/\s+/g, ' ').trim()) === 'other')) continue;
+ form.relevant = false;
+ form.utility = true;
+ form.utilityReason = form.searchUtilityReason;
+ pageState.relevantFormCount = Math.max(0, Number(pageState.relevantFormCount || 0) - 1);
+ }
+ }
+
async _readCompletionStatusTexts(tabId, pageState = null) {
const submit = this._currentCompletionSubmitState(tabId);
const live = Array.isArray(pageState?.liveRegionMessages)
@@ -36389,6 +36412,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
// submit say, read in any language for the checks below. Success
// messages are only ever the agent's own readings, never page data.
const readingStartedAt = Date.now();
+ if (outcome === 'success' && pageState) await this._readCompletionFormSubmits(tabId, pageState);
if (outcome === 'success' && pageState) await this._readCompletionStatusTexts(tabId, pageState);
else if (pageState && typeof pageState === 'object') pageState.successMessages = [];
const verificationStartedAt = Date.now();
diff --git a/src/firefox/src/agent/completion-invariant.js b/src/firefox/src/agent/completion-invariant.js
index 9b3ae7926..98bbc89ba 100644
--- a/src/firefox/src/agent/completion-invariant.js
+++ b/src/firefox/src/agent/completion-invariant.js
@@ -175,6 +175,7 @@ export function classifyCompletionForm({
hiddenNamedCount = 0,
editable = [],
submits = [],
+ submitReadings = {},
} = {}) {
const fields = Array.isArray(editable) ? editable : [];
const submitControls = Array.isArray(submits) ? submits : [];
@@ -201,22 +202,31 @@ export function classifyCompletionForm({
// normal search-results state and is not task evidence by itself.
const searchFieldsOnly = normalizedFields.length > 0
&& normalizedFields.every(field => semanticSearchField(field) || conventionalSearchField(field));
- const searchSubmitsOnly = normalizedSubmits.every(control => /search|filter|go/i.test(control.label));
- const searchHasTaskEvidence = !!(
+ // What a submit control's caption means is read by a model in any language
+ // (the agent's control_effect readings, `submitReadings` by label): only
+ // controls read as showing, filtering or searching ("other") leave a
+ // search form a search form. An unread caption is no evidence, so it keeps
+ // the form relevant here; `searchCandidate` tells the agent to read it.
+ const readings = submitReadings && typeof submitReadings === 'object' ? submitReadings : {};
+ const readingOf = control => (Object.prototype.hasOwnProperty.call(readings, control.label) ? readings[control.label] : null);
+ const searchSubmitsOnly = normalizedSubmits.every(control => readingOf(control) === 'other');
+ const searchStructureHasTaskEvidence = !!(
insideDialog
|| normalizedMethod !== 'get'
|| Number(hiddenNamedCount || 0) !== 0
- || !searchSubmitsOnly
|| normalizedFields.some(field => (
field.required
|| field.focused
|| (!!field.name && !conventionalSearchField(field))
))
);
+ const searchHasTaskEvidence = searchStructureHasTaskEvidence || !searchSubmitsOnly;
// Primary content alone is not task evidence: result pages commonly place
// genuine search/filter forms there.
const safeSearchOnly = searchFieldsOnly && !searchHasTaskEvidence;
const hasSemanticSearchField = normalizedFields.some(semanticSearchField);
+ const searchCandidate = searchFieldsOnly && !searchStructureHasTaskEvidence && !searchSubmitsOnly
+ && normalizedSubmits.every(control => readingOf(control) === null || readingOf(control) === 'other');
const onlyField = normalizedFields.length === 1 ? normalizedFields[0] : null;
const passiveUtilityShell = !!(
@@ -251,6 +261,12 @@ export function classifyCompletionForm({
utilityReason,
editableCount: normalizedFields.length,
submitCount: normalizedSubmits.length,
+ ...(searchCandidate && !utility ? {
+ searchCandidate: true,
+ searchUtilityReason: hasSemanticSearchField ? 'semantic_search' : 'conventional_search',
+ // Every submit's caption, so the agent can tell when one has none to read.
+ submitLabels: normalizedSubmits.map(control => control.label.slice(0, 120)).slice(0, 4),
+ } : {}),
};
}
diff --git a/test/control-effect-readings.mjs b/test/control-effect-readings.mjs
index d8db7dad7..116898663 100644
--- a/test/control-effect-readings.mjs
+++ b/test/control-effect-readings.mjs
@@ -87,6 +87,24 @@ for (const build of ['chrome', 'firefox']) {
assert.equal(agent._isWorkflowFormSubmitAction('click', { text: 'Submit' }, { success: true }), false);
});
+ test(`${build}: a search form whose buttons a model read as searching stops counting at completion`, async () => {
+ const form = (labels, extra = {}) => ({ label: '', relevant: true, utility: false, utilityReason: null, editableCount: 1,
+ submitCount: labels.length, searchCandidate: true, searchUtilityReason: 'semantic_search', submitLabels: labels, ...extra });
+ const agent = new Agent({});
+ const asked = stubPageTextReadings(agent);
+ const state = { relevantFormCount: 5, formDescriptors: [form(['Ara']), form(['Suchen', 'Kaydet']), form(['']),
+ form(['Ara'], { submitCount: 2 }), { label: 'Profil', relevant: true, utility: false, submitCount: 1 }] };
+ // The fixtures' oracle reads "Ara" and "Suchen" as other, "Kaydet" as a commit.
+ agent._pageTextReading = (kind, text) => (kind === 'control_effect' && ['Ara', 'Suchen'].includes(text) ? 'other'
+ : kind === 'control_effect' && text === 'Kaydet' ? 'commit' : null);
+ await agent._readCompletionFormSubmits(1, state);
+ assert.deepEqual(state.formDescriptors.map(item => item.relevant), [false, true, true, true, true]);
+ assert.equal(state.formDescriptors[0].utilityReason, 'semantic_search');
+ assert.equal(state.relevantFormCount, 4);
+ assert.deepEqual([...new Set(asked.map(item => item.text))].sort(), ['Ara', 'Kaydet', 'Suchen'],
+ 'Only complete, non-empty captions of search candidates are read');
+ });
+
test(`${build}: a caption is read once, never when structure decides, and a failed read is no answer`, async () => {
const agent = new Agent({});
const asked = stubPageTextReadings(agent);
diff --git a/test/run.js b/test/run.js
index c3157007e..84eb8beae 100644
--- a/test/run.js
+++ b/test/run.js
@@ -26996,6 +26996,8 @@ test('completion form classification ignores passive localized utility shells wi
{ tag: 'select', type: '', role: '', name: 'filter', value: '' },
],
submits: [{ label: 'Filter' }],
+ // What a model read the caption as.
+ submitReadings: { Filter: 'other' },
}),
{
label: '',
@@ -27015,6 +27017,7 @@ test('completion form classification ignores passive localized utility shells wi
{ tag: 'select', type: '', role: '', name: 'filter', value: 'recent' },
],
submits: [{ label: 'Search' }],
+ submitReadings: { Search: 'other' },
}),
{
label: '',
@@ -27026,6 +27029,24 @@ test('completion form classification ignores passive localized utility shells wi
},
`${label}: retained search query and filter values were treated as task drafts`,
);
+ // A submit caption is read in any language; an unread one keeps the form
+ // relevant and asks the agent to read it, and one read as anything but
+ // showing or searching is task evidence.
+ const searchForm = { outsidePrimaryContent: false,
+ editable: [{ tag: 'input', type: 'search', role: '', name: 'q', value: '' }] };
+ assert.equal(classify({ ...searchForm, submits: [{ label: 'Ara' }], submitReadings: { Ara: 'other' } }).utility, true,
+ `${label}: a search form with a Turkish search button stayed a task form`);
+ assert.deepEqual(classify({ ...searchForm, submits: [{ label: 'Ara' }] }), {
+ label: '', relevant: true, utility: false, utilityReason: null, editableCount: 1, submitCount: 1,
+ searchCandidate: true, searchUtilityReason: 'semantic_search', submitLabels: ['Ara'],
+ }, `${label}: an unread search caption was decided without a reading`);
+ for (const reading of ['commit', 'next_step', 'form_edit']) {
+ const result = classify({ ...searchForm, submits: [{ label: 'Ekle' }], submitReadings: { Ekle: reading } });
+ assert.equal(result.relevant, true, `${label}: a ${reading} caption hid a task form`);
+ assert.equal(result.searchCandidate, undefined, `${label}: a ${reading} caption asked for another reading`);
+ }
+ assert.equal(classify({ ...searchForm, method: 'post', submits: [{ label: 'Ara' }] }).searchCandidate, undefined,
+ `${label}: structure that is task evidence asked for a caption reading`);
for (const [caseName, override] of [
['primary-content form', { outsidePrimaryContent: false }],