Open-source Python library for AI browser agents, plus Browser Use Cloud: stealth browsers and a hosted agent API.
Python ai-agentsai-toolsbrowser-agentbrowser-automationbrowser-usecdpcloud-browsercomputer-useheadless-browserllmmcpplaywrightpythonremote-browserstealth-browserweb-automationweb-scraping
handle autocomplete fields
Saurav Panda6878bd1ca4fb
diff --git a/browser_use/agent/system_prompts/system_prompt.md b/browser_use/agent/system_prompts/system_prompt.mdindex 86248e26a..cca04084f 100644--- a/browser_use/agent/system_prompts/system_prompt.md+++ b/browser_use/agent/system_prompts/system_prompt.md@@ -75,6 +75,7 @@ Strictly follow these rules while using the browser and navigating the web:- If the <user_request> includes specific page information such as product type, rating, price, location, etc., try to apply filters to be more efficient.- The <user_request> is the ultimate goal. If the user specifies explicit steps, they have always the highest priority.- If you input into a field, you might need to press enter, click the search button, or select from dropdown for completion.+- For autocomplete/combobox fields (e.g. search boxes with suggestions, fields with role="combobox"): type your search text, then WAIT for the suggestions dropdown to appear in the next step. Review the new elements (marked with *[) and click the correct suggestion. Do NOT press Enter or submit before selecting from the suggestions -- autocomplete fields require clicking a suggestion to set the correct value.- Don't login into a page if you don't have to. Don't login if you don't have the credentials.- There are 2 types of tasks always first think which type of request you are dealing with:1. Very specific step by step instructions:diff --git a/browser_use/agent/system_prompts/system_prompt_anthropic.md b/browser_use/agent/system_prompts/system_prompt_anthropic.mdindex 252eff9c1..75c998677 100644--- a/browser_use/agent/system_prompts/system_prompt_anthropic.md+++ b/browser_use/agent/system_prompts/system_prompt_anthropic.md@@ -75,6 +75,7 @@ Strictly follow these rules while using the browser and navigating the web:- If the <user_request> includes specific page information such as product type, rating, price, location, etc., ALWAYS look for filter/sort options FIRST before browsing results. Apply all relevant filters before scrolling through results.- The <user_request> is the ultimate goal. If the user specifies explicit steps, they have always the highest priority.- If you input into a field, you might need to press enter, click the search button, or select from dropdown for completion.+- For autocomplete/combobox fields (e.g. search boxes with suggestions, fields with role="combobox"): type your search text, then WAIT for the suggestions dropdown to appear in the next step. Review the new elements (marked with *[) and click the correct suggestion. Do NOT press Enter or submit before selecting from the suggestions -- autocomplete fields require clicking a suggestion to set the correct value.- Don't login into a page if you don't have to. Don't login if you don't have the credentials.- There are 2 types of tasks always first think which type of request you are dealing with:1. Very specific step by step instructions:diff --git a/browser_use/agent/system_prompts/system_prompt_anthropic_flash.md b/browser_use/agent/system_prompts/system_prompt_anthropic_flash.mdindex bf0d8d643..60baf4c9f 100644--- a/browser_use/agent/system_prompts/system_prompt_anthropic_flash.md+++ b/browser_use/agent/system_prompts/system_prompt_anthropic_flash.md@@ -41,6 +41,7 @@ Strictly follow these rules while using the browser and navigating the web:- If the <user_request> includes specific page information such as product type, rating, price, location, etc., ALWAYS look for filter/sort options FIRST before browsing results. Apply all relevant filters before scrolling through results. This is critical for efficiency.- The <user_request> is the ultimate goal. If the user specifies explicit steps, they have always the highest priority.- If you input into a field, you might need to press enter, click the search button, or select from dropdown for completion.+- For autocomplete/combobox fields (e.g. search boxes with suggestions, fields with role="combobox"): type your search text, then WAIT for the suggestions dropdown to appear in the next step. Review the new elements (marked with *[) and click the correct suggestion. Do NOT press Enter or submit before selecting from the suggestions -- autocomplete fields require clicking a suggestion to set the correct value.- Don't login into a page if you don't have to. Don't login if you don't have the credentials.- There are 2 types of tasks:1. Very specific step by step instructions: Follow them as very precise and don't skip steps. Try to complete everything as requested.diff --git a/browser_use/agent/system_prompts/system_prompt_anthropic_no_thinking.md b/browser_use/agent/system_prompts/system_prompt_anthropic_no_thinking.mdindex 6f279349d..52799c202 100644--- a/browser_use/agent/system_prompts/system_prompt_anthropic_no_thinking.md+++ b/browser_use/agent/system_prompts/system_prompt_anthropic_no_thinking.md@@ -75,6 +75,7 @@ Strictly follow these rules while using the browser and navigating the web:- If the <user_request> includes specific page information such as product type, rating, price, location, etc., ALWAYS look for filter/sort options FIRST before browsing results. Apply all relevant filters before scrolling through results.- The <user_request> is the ultimate goal. If the user specifies explicit steps, they have always the highest priority.- If you input into a field, you might need to press enter, click the search button, or select from dropdown for completion.+- For autocomplete/combobox fields (e.g. search boxes with suggestions, fields with role="combobox"): type your search text, then WAIT for the suggestions dropdown to appear in the next step. Review the new elements (marked with *[) and click the correct suggestion. Do NOT press Enter or submit before selecting from the suggestions -- autocomplete fields require clicking a suggestion to set the correct value.- Don't login into a page if you don't have to. Don't login if you don't have the credentials.- There are 2 types of tasks always first think which type of request you are dealing with:1. Very specific step by step instructions:diff --git a/browser_use/agent/system_prompts/system_prompt_no_thinking.md b/browser_use/agent/system_prompts/system_prompt_no_thinking.mdindex 086cf882e..02fe42673 100644--- a/browser_use/agent/system_prompts/system_prompt_no_thinking.md+++ b/browser_use/agent/system_prompts/system_prompt_no_thinking.md@@ -75,6 +75,7 @@ Strictly follow these rules while using the browser and navigating the web:- If the <user_request> includes specific page information such as product type, rating, price, location, etc., try to apply filters to be more efficient.- The <user_request> is the ultimate goal. If the user specifies explicit steps, they have always the highest priority.- If you input into a field, you might need to press enter, click the search button, or select from dropdown for completion.+- For autocomplete/combobox fields (e.g. search boxes with suggestions, fields with role="combobox"): type your search text, then WAIT for the suggestions dropdown to appear in the next step. Review the new elements (marked with *[) and click the correct suggestion. Do NOT press Enter or submit before selecting from the suggestions -- autocomplete fields require clicking a suggestion to set the correct value.- Don't login into a page if you don't have to. Don't login if you don't have the credentials.- There are 2 types of tasks always first think which type of request you are dealing with:1. Very specific step by step instructions:diff --git a/browser_use/browser/watchdogs/default_action_watchdog.py b/browser_use/browser/watchdogs/default_action_watchdog.pyindex 10c676f6c..ade35c32f 100644--- a/browser_use/browser/watchdogs/default_action_watchdog.py+++ b/browser_use/browser/watchdogs/default_action_watchdog.py@@ -1656,6 +1656,26 @@ class DefaultActionWatchdog(BaseWatchdog):# to update their internal state and trigger re-rendersawait self._trigger_framework_events(object_id=object_id, cdp_session=cdp_session)+ # Step 5: Read back actual value for verification (skip for sensitive data)+ if not is_sensitive:+ try:+ await asyncio.sleep(0.05) # let autocomplete/formatter JS settle+ readback_result = await cdp_session.cdp_client.send.Runtime.callFunctionOn(+ params={+ 'objectId': object_id,+ 'functionDeclaration': 'function() { return this.value !== undefined ? this.value : this.textContent; }',+ 'returnByValue': True,+ },+ session_id=cdp_session.session_id,+ )+ actual_value = readback_result.get('result', {}).get('value')+ if actual_value is not None:+ if input_coordinates is None:+ input_coordinates = {}+ input_coordinates['actual_value'] = actual_value+ except Exception as e:+ self.logger.debug(f'Value readback failed (non-critical): {e}')+# Return coordinates metadata if availablereturn input_coordinatesdiff --git a/browser_use/dom/views.py b/browser_use/dom/views.pyindex 14a618747..42aafb68b 100644--- a/browser_use/dom/views.py+++ b/browser_use/dom/views.py@@ -47,6 +47,8 @@ DEFAULT_INCLUDE_ATTRIBUTES = ['multiple', # Whether multiple files/selections are allowed'inputmode', # Virtual keyboard hint (numeric, tel, email, url, etc.)'autocomplete', # Autocomplete behavior hint+ 'aria-autocomplete', # ARIA autocomplete type (list, inline, both)+ 'list', # Associated datalist element ID'data-mask', # Input mask format (e.g., phone numbers, credit cards)'data-inputmask', # Alternative input mask attribute'data-datepicker', # jQuery datepicker indicator@@ -114,8 +116,10 @@ STATIC_ATTRIBUTES = {'aria-disabled','aria-hidden','aria-pressed',+ 'aria-autocomplete','aria-checked','aria-selected',+ 'list','tabindex','alt','src',diff --git a/browser_use/tools/service.py b/browser_use/tools/service.pyindex bcd2d9fbc..3a8ccbadc 100644--- a/browser_use/tools/service.py+++ b/browser_use/tools/service.py@@ -102,6 +102,21 @@ def handle_browser_error(e: BrowserError) -> ActionResult:raise e+def _is_autocomplete_field(node: EnhancedDOMTreeNode) -> bool:+ """Detect if a node is an autocomplete/combobox field from its attributes."""+ attrs = node.attributes or {}+ if attrs.get('role') == 'combobox':+ return True+ aria_ac = attrs.get('aria-autocomplete', '')+ if aria_ac and aria_ac != 'none':+ return True+ if attrs.get('list'):+ return True+ if attrs.get('aria-haspopup') and (attrs.get('aria-controls') or attrs.get('aria-owns')):+ return True+ return False++class Tools(Generic[Context]):def __init__(self,@@ -417,6 +432,18 @@ class Tools(Generic[Context]):logger.debug(log_msg)+ # Check for value mismatch (non-sensitive only)+ actual_value = None+ if isinstance(input_metadata, dict):+ actual_value = input_metadata.pop('actual_value', None)++ if not has_sensitive_data and actual_value is not None and actual_value != params.text:+ msg += f"\n⚠️ Note: the field's actual value '{actual_value}' differs from typed text '{params.text}'. The page may have reformatted or autocompleted your input."++ # Check for autocomplete/combobox field+ if _is_autocomplete_field(node):+ msg += '\n💡 This is an autocomplete field. Wait for suggestions to appear, then click the correct suggestion instead of pressing Enter.'+# Include input coordinates in metadata if availablereturn ActionResult(extracted_content=msg,diff --git a/tests/ci/interactions/test_autocomplete_interaction.py b/tests/ci/interactions/test_autocomplete_interaction.pynew file mode 100644index 000000000..17b59bfdc--- /dev/null+++ b/tests/ci/interactions/test_autocomplete_interaction.py@@ -0,0 +1,228 @@+"""Test autocomplete/combobox field detection and value readback.++Tests cover:+- Value mismatch detection when JS rewrites input value+- Combobox field detection (role=combobox + aria-autocomplete)+- Datalist field detection (input with list attribute)+- No false positives on plain inputs+- Sensitive data skips value verification+"""++import asyncio++import pytest+from pytest_httpserver import HTTPServer++from browser_use.agent.views import ActionResult+from browser_use.browser import BrowserSession+from browser_use.browser.profile import BrowserProfile+from browser_use.tools.service import Tools++[email protected](scope='session')+def http_server():+ """Create and provide a test HTTP server with autocomplete test pages."""+ server = HTTPServer()+ server.start()++ # Page 1: Input with JS that rewrites value on change (simulates autocomplete replacing text)+ server.expect_request('/autocomplete-rewrite').respond_with_data(+ """+ <!DOCTYPE html>+ <html>+ <head><title>Autocomplete Rewrite Test</title></head>+ <body>+ <input id="search" type="text" />+ <script>+ const input = document.getElementById('search');+ input.addEventListener('change', function() {+ // Simulate autocomplete rewriting the value+ this.value = 'REWRITTEN_' + this.value;+ });+ </script>+ </body>+ </html>+ """,+ content_type='text/html',+ )++ # Page 2: Input with role=combobox + aria-autocomplete=list + aria-controls + listbox+ server.expect_request('/combobox-field').respond_with_data(+ """+ <!DOCTYPE html>+ <html>+ <head><title>Combobox Field Test</title></head>+ <body>+ <div>+ <input id="combo" type="text" role="combobox"+ aria-autocomplete="list" aria-controls="suggestions-list"+ aria-expanded="false" />+ <ul id="suggestions-list" role="listbox" style="display:none;">+ <li role="option">Option A</li>+ <li role="option">Option B</li>+ </ul>+ </div>+ </body>+ </html>+ """,+ content_type='text/html',+ )++ # Page 3: Input with list attribute pointing to a datalist+ server.expect_request('/datalist-field').respond_with_data(+ """+ <!DOCTYPE html>+ <html>+ <head><title>Datalist Field Test</title></head>+ <body>+ <input id="city" type="text" list="suggestions" />+ <datalist id="suggestions">+ <option value="New York">+ <option value="Los Angeles">+ <option value="Chicago">+ </datalist>+ </body>+ </html>+ """,+ content_type='text/html',+ )++ # Page 4: Plain input with no autocomplete attributes+ server.expect_request('/normal-input').respond_with_data(+ """+ <!DOCTYPE html>+ <html>+ <head><title>Normal Input Test</title></head>+ <body>+ <input id="plain" type="text" placeholder="Just a normal input" />+ </body>+ </html>+ """,+ content_type='text/html',+ )++ yield server+ server.stop()++[email protected](scope='session')+def base_url(http_server):+ """Return the base URL for the test HTTP server."""+ return f'http://{http_server.host}:{http_server.port}'++[email protected](scope='module')+async def browser_session():+ """Create and provide a Browser instance for testing."""+ browser_session = BrowserSession(+ browser_profile=BrowserProfile(+ headless=True,+ user_data_dir=None,+ keep_alive=True,+ chromium_sandbox=False,+ )+ )+ await browser_session.start()+ yield browser_session+ await browser_session.kill()++[email protected](scope='function')+def tools():+ """Create and provide a Tools instance."""+ return Tools()+++class TestAutocompleteInteraction:+ """Test autocomplete/combobox detection and value readback."""++ async def test_value_mismatch_detected(self, tools: Tools, browser_session: BrowserSession, base_url: str):+ """Type into a field whose JS rewrites the value on change. Assert the ActionResult notes the mismatch."""+ await tools.navigate(url=f'{base_url}/autocomplete-rewrite', new_tab=False, browser_session=browser_session)+ await asyncio.sleep(0.3)+ await browser_session.get_browser_state_summary()++ input_index = await browser_session.get_index_by_id('search')+ assert input_index is not None, 'Could not find search input'++ result = await tools.input(index=input_index, text='hello', browser_session=browser_session)++ assert isinstance(result, ActionResult)+ assert result.extracted_content is not None+ assert 'differs from typed text' in result.extracted_content, (+ f'Expected mismatch note in extracted_content, got: {result.extracted_content}'+ )++ async def test_combobox_field_detected(self, tools: Tools, browser_session: BrowserSession, base_url: str):+ """Type into a combobox field. Assert the ActionResult includes autocomplete guidance."""+ await tools.navigate(url=f'{base_url}/combobox-field', new_tab=False, browser_session=browser_session)+ await asyncio.sleep(0.3)+ await browser_session.get_browser_state_summary()++ combo_index = await browser_session.get_index_by_id('combo')+ assert combo_index is not None, 'Could not find combobox input'++ result = await tools.input(index=combo_index, text='test', browser_session=browser_session)++ assert isinstance(result, ActionResult)+ assert result.extracted_content is not None+ assert 'autocomplete field' in result.extracted_content, (+ f'Expected autocomplete guidance in extracted_content, got: {result.extracted_content}'+ )++ async def test_datalist_field_detected(self, tools: Tools, browser_session: BrowserSession, base_url: str):+ """Type into a datalist-backed field. Assert the ActionResult includes autocomplete guidance."""+ await tools.navigate(url=f'{base_url}/datalist-field', new_tab=False, browser_session=browser_session)+ await asyncio.sleep(0.3)+ await browser_session.get_browser_state_summary()++ city_index = await browser_session.get_index_by_id('city')+ assert city_index is not None, 'Could not find datalist input'++ result = await tools.input(index=city_index, text='New', browser_session=browser_session)++ assert isinstance(result, ActionResult)+ assert result.extracted_content is not None+ assert 'autocomplete field' in result.extracted_content, (+ f'Expected autocomplete guidance in extracted_content, got: {result.extracted_content}'+ )++ async def test_normal_input_no_false_positive(self, tools: Tools, browser_session: BrowserSession, base_url: str):+ """Type into a plain input. Assert the ActionResult does NOT contain autocomplete guidance."""+ await tools.navigate(url=f'{base_url}/normal-input', new_tab=False, browser_session=browser_session)+ await asyncio.sleep(0.3)+ await browser_session.get_browser_state_summary()++ plain_index = await browser_session.get_index_by_id('plain')+ assert plain_index is not None, 'Could not find plain input'++ result = await tools.input(index=plain_index, text='hello', browser_session=browser_session)++ assert isinstance(result, ActionResult)+ assert result.extracted_content is not None+ assert 'autocomplete field' not in result.extracted_content, (+ f'Got false positive autocomplete guidance on plain input: {result.extracted_content}'+ )++ async def test_sensitive_data_skips_value_verification(self, tools: Tools, browser_session: BrowserSession, base_url: str):+ """Type sensitive data into the rewrite field. Assert no 'differs from typed text' note appears."""+ await tools.navigate(url=f'{base_url}/autocomplete-rewrite', new_tab=False, browser_session=browser_session)+ await asyncio.sleep(0.3)+ await browser_session.get_browser_state_summary()++ input_index = await browser_session.get_index_by_id('search')+ assert input_index is not None, 'Could not find search input'++ # Use tools.act() with sensitive_data to trigger the sensitive code path+ result = await tools.input(+ index=input_index,+ text='secret123',+ browser_session=browser_session,+ sensitive_data={'password': 'secret123'},+ )++ assert isinstance(result, ActionResult)+ assert result.extracted_content is not None+ assert 'differs from typed text' not in result.extracted_content, (+ f'Sensitive data should not show value mismatch: {result.extracted_content}'+ )