Open-source AI browser agent for Chrome and Firefox (monorepo) ðŸ§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Know a search form by its structure, not by English label words
Claude0ac8aa15ac01
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 2d31952f9..99ab67576 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,7 @@ This changelog was generated from the repository Git history and release tags. V## [Unreleased]### Fixed+- Search boxes in any language now share the single approval a search gets, instead of asking again for every search. A German site's box under "Suche" or a Turkish "Ara" button counts as a search when the page marks the form as a search, the field's keyboard says search, or a model reads the button as only searching; the English word "Search" in a label no longer decides it either way. Forms that send data, ask for a password or card, or post elsewhere keep their own approval.- Forms: which field of a form you named in your request, and the value you gave it, is now read by a model in any language instead of by matching words. A request in Turkish such as "ön yazımı ekle" now matches an English form's "Cover letter (optional)", so its value counts as entered and the optional field can no longer be skipped. Before, words in different languages never matched, so the form could not be completed, and a field you asked for could be left empty.- Gmail: the recipient check no longer fails outright when the compose window shows a field such as the subject or a typed address. A coding error, present since version 37, made the check fail there, and the send was refused.- Chat and mail sites: which row of a message a field is, whether a message was sent by you, and whether a sent message was delivered are now read by a model in any language, as button names already are. A Turkish webmail's "Alıcı" and "Bilgi" fields count as To and Cc, Gmail address chips under a Turkish "Gizli" row count as Bcc, a chat message labelled "Siz: …" counts as yours, and a WhatsApp message whose status reads "Okundu" counts as delivered; before, only English labels did, and a Cc field was not checked at all. Discord names a channel or contact from its selected sidebar link in any language, and WhatsApp's contact panel closes through its one header button whatever it is called, so the WhatsApp contact check no longer stops at a Turkish "Kapat". Texts a check reaches only after reading the first ones, such as the recipient rows behind a send button, are read within the same check. A text no model could read yet is taken the cautious way: a field holds no recipient, a message is nobody's in particular, and a status is not delivered.diff --git a/docs/security-model.md b/docs/security-model.mdindex b3833ac73..934fb63ae 100644--- a/docs/security-model.md+++ b/docs/security-model.md@@ -177,6 +177,7 @@ The primary threat: a malicious page crafts content that, when read by the agent| **API mutation override** | A per-conversation `/allow-api` flag, or the default-on persistent setting under General → Advanced, *waives* the permission prompt for write-method network egress (`fetch_url`/`research_url` with POST/PUT/PATCH/DELETE). Neither option waives GET egress or any other capability. Conversation reset clears only the slash-command override. || **Decision models** | OpenRouter, direct TypeSafe and local/custom System One endpoints share independent per-feature switches. Keys are optional for local servers. Compass completion uses its managed decision route. Bounded/redacted task and fresh page evidence are untrusted data. Judges have no browser executor; ordinary tool calls still pass policy, permission and completion layers. Strict Secret Mode, cancellation, quota and cost limits apply. Unavailable or malformed judgments fall through to the active LLM and existing checks. || **`done()` verification** | Decision and tool-free LLM checks prefer vision, then AX. Confident pending/failed outcomes require read-only recovery before another submission. Accepted task-bound proof can satisfy generic evidence despite an unrelated comment form, while explicit recipient, payment, download, authorization and workflow contracts remain required. Existing dialog/form checks are the final fallback. |+| **Grouped search approval** | A same-site GET form with no credential, payment or hidden token field, and no side-effect action address, shares one task-scoped approval for its navigation, typing and submit when the page's structure marks it as a search: a search landmark, a search-typed field or one whose keyboard says search, code names such as `q` or `query`, a search address, or a submit control a model read, in any language, as only searching or filtering. Words in a field's visible label decide nothing. Every other form keeps its fresh confirmation. || **Duplicate-submit guard** | A second text click on the same caption within 45 seconds on the same tab and URL is blocked when a model read the caption, in any language, as committing something (sending, saving, paying, publishing, creating, confirming and so on). A caption read as anything else, or that could not be read, stays repeatable; app-observed popups, links, fields and draft tools are always repeatable. Both builds. || **CLICK occlusion test** | Before clicking, the resolver calls `elementFromPoint()`. If another element is visually on top, the click is refused. || **Modal-scoped click** | When a dialog is open, text clicks are scoped to that subtree so the agent doesn't click a dimmed background element. |diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.jsindex 97c3c9f83..ca7c09ae0 100644--- a/src/chrome/src/agent/agent.js+++ b/src/chrome/src/agent/agent.js@@ -27318,6 +27318,8 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dmethod: String(detected.method || 'GET').slice(0, 20).toUpperCase(),action: String(detected.action || '').slice(0, 300),actionUrl: String(detected.actionUrl || detected.action || '').slice(0, 300),+ searchLandmark: detected.searchLandmark === true,+ searchSubmitCaption: detected.searchSubmitCaption === true,githubCommitDialogLauncher: detected.githubCommitDialogLauncher === true,// Ordinary in-form controls keep their submit confirmation; this only// tells post-correction guards they are not consequential commits.@@ -27362,6 +27364,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dmethod: String(resolvedForm.method || 'GET').slice(0, 20).toUpperCase(),action: String(resolvedForm.action || '').slice(0, 300),actionUrl: String(resolvedForm.actionUrl || resolvedForm.action || '').slice(0, 300),+ searchLandmark: resolvedForm.searchLandmark === true,};}const nonSubmit = rawResults.find(item => item?.resolvedNonSubmitTarget === true);@@ -27723,10 +27726,16 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dariaLabel: compact(el.getAttribute?.('aria-label') || '', 120),placeholder: compact(el.getAttribute?.('placeholder') || '', 120),type: compact(el.type || el.tagName || '', 40),+ enterKeyHint: compact(el.getAttribute?.('enterkeyhint') || '', 20),+ inputMode: compact(el.getAttribute?.('inputmode') || '', 20),value,changed,};});+ // A search landmark around the form says it searches, in any language.+ const searchLandmark = (() => {+ try { return !!form.closest?.('[role="search"],search'); } catch { return false; }+ })();const changedFields = fields.filter(field => field.changed || field.value).slice(0, 8);const changedText = changedFields.length? `Changed/filled fields: ${changedFields.map(field => `${field.label}: ${field.value || '(blank)'}`).join('; ')}.`@@ -27746,6 +27755,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dmethod,action,actionUrl,+ searchLandmark,};};const transactionOrderScan = (root) => {@@ -28134,6 +28144,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dgithubCommitDialogLauncher: githubEditPage&& !controlInModal&& (/^commit changes(?:\.{3}|…)?$/i.test(controlLabel) || controlOpensDialog),+ // The activated control's captions read as only searching,+ // filtering or showing something, in any language.+ searchSubmitCaption: captionsReading(controlCaptions(control)) === 'view',...formSummary,summary: [formSummary.summary,diff --git a/src/chrome/src/agent/permission-gate.js b/src/chrome/src/agent/permission-gate.jsindex e2159bbe0..d43d1fcb5 100644--- a/src/chrome/src/agent/permission-gate.js+++ b/src/chrome/src/agent/permission-gate.js@@ -205,6 +205,13 @@ function submitFieldDescriptor(field = {}) {].map(value => String(value || '').trim()).filter(Boolean).join(' ');}+// A field's code-level names (its type, name, id and autocomplete token),+// never its visible labels: those are in the page's language.+function submitFieldCodeNames(field = {}) {+ return [field.type, field.name, field.id, field.autocomplete]+ .map(value => String(value || '').trim()).filter(Boolean).join(' ');+}+function shortHash(value) {let hash = 2166136261;const text = String(value || '');@@ -311,8 +318,19 @@ export function classifySubmitRisk(submitInfo = {}, currentUrlOrHost = '') {if (SIDE_EFFECT_ACTION_RE.test(`${pathname} ${actionParameters}`)) {return freshSubmitRisk('form action may have side effects');}- const fieldMetadata = fields.map(submitFieldDescriptor).join(' ');- const searchLike = SEARCH_PATH_RE.test(pathname) || SEARCH_FIELD_RE.test(fieldMetadata) || SEARCH_FIELD_RE.test(actionParameters);+ // A search is known by the page's structure, in any language: a search+ // landmark around the form, a search-typed field or one whose keyboard says+ // search, code names such as q or query, a search address, or a submit+ // control a model read as only searching, filtering or showing results.+ const fieldCodeNames = fields.map(submitFieldCodeNames).join(' ');+ const searchKeyboard = fields.some(field => /^search$/i.test(String(field.enterKeyHint || '').trim())+ || /^search$/i.test(String(field.inputMode || '').trim()));+ const searchLike = submitInfo.searchLandmark === true+ || searchKeyboard+ || submitInfo.searchSubmitCaption === true+ || SEARCH_PATH_RE.test(pathname)+ || SEARCH_FIELD_RE.test(fieldCodeNames)+ || SEARCH_FIELD_RE.test(actionParameters);if (!searchLike) return freshSubmitRisk('form is not a resolved search');return {diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.jsindex 959fcf58a..4b05adf42 100644--- a/src/firefox/src/agent/agent.js+++ b/src/firefox/src/agent/agent.js@@ -24747,6 +24747,8 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dmethod: String(detected.method || 'GET').slice(0, 20).toUpperCase(),action: String(detected.action || '').slice(0, 300),actionUrl: String(detected.actionUrl || detected.action || '').slice(0, 300),+ searchLandmark: detected.searchLandmark === true,+ searchSubmitCaption: detected.searchSubmitCaption === true,githubCommitDialogLauncher: detected.githubCommitDialogLauncher === true,// Ordinary in-form controls keep their submit confirmation; this only// tells post-correction guards they are not consequential commits.@@ -24791,6 +24793,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dmethod: String(resolvedForm.method || 'GET').slice(0, 20).toUpperCase(),action: String(resolvedForm.action || '').slice(0, 300),actionUrl: String(resolvedForm.actionUrl || resolvedForm.action || '').slice(0, 300),+ searchLandmark: resolvedForm.searchLandmark === true,};}const nonSubmit = rawResults.find(item => item?.resolvedNonSubmitTarget === true);@@ -25064,10 +25067,16 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dariaLabel: compact(el.getAttribute?.('aria-label') || '', 120),placeholder: compact(el.getAttribute?.('placeholder') || '', 120),type: compact(el.type || el.tagName || '', 40),+ enterKeyHint: compact(el.getAttribute?.('enterkeyhint') || '', 20),+ inputMode: compact(el.getAttribute?.('inputmode') || '', 20),value,changed,};});+ // A search landmark around the form says it searches, in any language.+ const searchLandmark = (() => {+ try { return !!form.closest?.('[role="search"],search'); } catch { return false; }+ })();const changedFields = fields.filter(field => field.changed || field.value).slice(0, 8);const changedText = changedFields.length? `Changed/filled fields: ${changedFields.map(field => `${field.label}: ${field.value || '(blank)'}`).join('; ')}.`@@ -25087,6 +25096,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dmethod,action,actionUrl,+ searchLandmark,};};const transactionOrderScan = (root) => {@@ -25475,6 +25485,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dgithubCommitDialogLauncher: githubEditPage&& !controlInModal&& (/^commit changes(?:\.{3}|…)?$/i.test(controlLabel) || controlOpensDialog),+ // The activated control's captions read as only searching,+ // filtering or showing something, in any language.+ searchSubmitCaption: captionsReading(controlCaptions(control)) === 'view',...formSummary,summary: [formSummary.summary,diff --git a/src/firefox/src/agent/permission-gate.js b/src/firefox/src/agent/permission-gate.jsindex d839063d2..a922c7f10 100644--- a/src/firefox/src/agent/permission-gate.js+++ b/src/firefox/src/agent/permission-gate.js@@ -196,6 +196,13 @@ function submitFieldDescriptor(field = {}) {].map(value => String(value || '').trim()).filter(Boolean).join(' ');}+// A field's code-level names (its type, name, id and autocomplete token),+// never its visible labels: those are in the page's language.+function submitFieldCodeNames(field = {}) {+ return [field.type, field.name, field.id, field.autocomplete]+ .map(value => String(value || '').trim()).filter(Boolean).join(' ');+}+function shortHash(value) {let hash = 2166136261;const text = String(value || '');@@ -302,8 +309,19 @@ export function classifySubmitRisk(submitInfo = {}, currentUrlOrHost = '') {if (SIDE_EFFECT_ACTION_RE.test(`${pathname} ${actionParameters}`)) {return freshSubmitRisk('form action may have side effects');}- const fieldMetadata = fields.map(submitFieldDescriptor).join(' ');- const searchLike = SEARCH_PATH_RE.test(pathname) || SEARCH_FIELD_RE.test(fieldMetadata) || SEARCH_FIELD_RE.test(actionParameters);+ // A search is known by the page's structure, in any language: a search+ // landmark around the form, a search-typed field or one whose keyboard says+ // search, code names such as q or query, a search address, or a submit+ // control a model read as only searching, filtering or showing results.+ const fieldCodeNames = fields.map(submitFieldCodeNames).join(' ');+ const searchKeyboard = fields.some(field => /^search$/i.test(String(field.enterKeyHint || '').trim())+ || /^search$/i.test(String(field.inputMode || '').trim()));+ const searchLike = submitInfo.searchLandmark === true+ || searchKeyboard+ || submitInfo.searchSubmitCaption === true+ || SEARCH_PATH_RE.test(pathname)+ || SEARCH_FIELD_RE.test(fieldCodeNames)+ || SEARCH_FIELD_RE.test(actionParameters);if (!searchLike) return freshSubmitRisk('form is not a resolved search');return {diff --git a/test/run.js b/test/run.jsindex c472b3571..563e5a282 100644--- a/test/run.js+++ b/test/run.js@@ -107553,6 +107553,31 @@ test('submit risk classifier groups only conservative same-site GET searches', (assert.equal(result.risk, SubmitRisk.FRESH_CONFIRMATION, `${label}: ${reason} must keep fresh confirmation`);assert.equal(result.lowRisk, false, `${label}: ${reason} must not be grouped`);}++ // A search is known by the page's structure in any language, never by+ // the words of its labels: a German site's search box under /suche.+ const german = {+ isSubmit: true,+ url: 'https://www.beispiel.de/suche',+ method: 'GET',+ actionUrl: 'https://www.beispiel.de/suche',+ fields: [{ type: 'text', name: 'begriff', label: 'Suchbegriff', placeholder: 'Suchbegriff eingeben', value: 'Fahrrad', changed: true }],+ };+ for (const [info, signal] of [[{ ...german, searchLandmark: true }, 'a search landmark'],+ [{ ...german, fields: [{ ...german.fields[0], enterKeyHint: 'search' }] }, 'a search keyboard'],+ [{ ...german, fields: [{ ...german.fields[0], inputMode: 'search' }] }, 'a search input mode'],+ [{ ...german, searchSubmitCaption: true }, 'a submit control read as searching']]) {+ assert.equal(classify(info, german.url).risk, SubmitRisk.LOW_RISK_SEARCH, `${label}: ${signal} did not mark a search`);+ assert.equal(classify({ ...info, method: 'POST' }, german.url).risk, SubmitRisk.FRESH_CONFIRMATION,+ `${label}: ${signal} grouped a POST form`);+ assert.equal(classify({ ...info, fields: [...info.fields, { type: 'password', name: 'pin', label: 'PIN' }] }, german.url).risk,+ SubmitRisk.FRESH_CONFIRMATION, `${label}: ${signal} grouped a form with a credential field`);+ }+ // Words in a label alone say nothing, in English as in German.+ assert.equal(classify(german, german.url).risk, SubmitRisk.FRESH_CONFIRMATION, `${label}: German label words`);+ assert.equal(classify({ ...german, url: 'https://shop.example/products', actionUrl: 'https://shop.example/products',+ fields: [{ type: 'text', name: 'kw', placeholder: 'Search products', value: 'bike', changed: true }] },+ 'https://shop.example/products').risk, SubmitRisk.FRESH_CONFIRMATION, `${label}: English label words`);}});diff --git a/test/submit-probe-readings.mjs b/test/submit-probe-readings.mjsindex 1a6d64b46..616e1ab49 100644--- a/test/submit-probe-readings.mjs+++ b/test/submit-probe-readings.mjs@@ -118,6 +118,22 @@ for (const build of ['chrome', 'firefox']) {__wbResourceRunToken: 'owned-run', __wbResourceTaskKey: 'owned-task' });assert.equal(held.mutationKind, 'delete');});++ await t.test('a search form reports its landmark, its keyboard and a submit read as searching, in any language', async () => {+ await open('<div role="search"><form id="f" method="get" action="/suche"><input name="begriff" aria-label="Suchbegriff" '+ + 'enterkeyhint="search" value="Fahrrad"><button id="t" type="submit">Suchen</button></form></div>');+ const searched = await probe('#t', caption => (caption === 'Suchen' ? 'view' : read(caption)));+ assert.equal(searched.isSubmit, true);+ assert.equal(searched.searchLandmark, true);+ assert.equal(searched.searchSubmitCaption, true);+ assert.equal(searched.fields?.[0]?.enterKeyHint, 'search');+ // Outside a landmark, with an unread caption, nothing says search.+ await open('<form id="f" method="get" action="/urun"><input name="kw" value="bisiklet"><button id="t" type="submit">Ara</button></form>');+ const plain = await probe('#t', () => null);+ assert.equal(plain.isSubmit, true);+ assert.equal(plain.searchLandmark, false);+ assert.equal(plain.searchSubmitCaption, false);+ });} finally {await browser.close();}