Skip to content

Open-source Python library for AI browser agents, plus Browser Use Cloud: stealth browsers and a hosted agent API.

Update system prompts for clarity and adjust bounding box rendering in Python highlights

Magnus Müller8ecc46daf673
diff --git a/.gitignore b/.gitignore
index 19c9d9c16..3072cda88 100644
--- a/.gitignore
+++ b/.gitignore
@@ -57,4 +57,4 @@ token.json
temp-profile-*
-*.png
\ No newline at end of file
+screenshot.png
\ No newline at end of file
diff --git a/browser_use/agent/system_prompt.md b/browser_use/agent/system_prompt.md
index 83640b832..55f90005a 100644
--- a/browser_use/agent/system_prompt.md
+++ b/browser_use/agent/system_prompt.md
@@ -66,8 +66,8 @@ Note that:
</browser_state>
<browser_vision>
-You will be optionally provided with a screenshot of the browser with bounding boxes. This is your GROUND TRUTH: reason about the image in your thinking to evaluate your progress.
-Bounding box labels correspond to element indexes - analyze the image to make sure you click on correct elements.
+You will be provided with a screenshot of the current page with bounding boxes around interactive elements. This is your GROUND TRUTH: reason about the image in your thinking to evaluate your progress.
+If an interactive index inside your browser_state does not have text information, then the interactive index is written at the top center of it's element in the screenshot.
</browser_vision>
<browser_rules>
diff --git a/browser_use/agent/system_prompt_flash.md b/browser_use/agent/system_prompt_flash.md
index a2f9257fc..97d454868 100644
--- a/browser_use/agent/system_prompt_flash.md
+++ b/browser_use/agent/system_prompt_flash.md
@@ -64,8 +64,8 @@ Note that:
</browser_state>
<browser_vision>
-You will be optionally provided with a screenshot of the browser with bounding boxes. This is your GROUND TRUTH: analyze the image to evaluate your progress.
-Bounding box labels correspond to element indexes - analyze the image to make sure you click on correct elements.
+You will be provided with a screenshot of the current page with bounding boxes around interactive elements. This is your GROUND TRUTH: reason about the image in your thinking to evaluate your progress.
+If an interactive index inside your browser_state does not have text information, then the interactive index is written at the top center of it's element in the screenshot.
</browser_vision>
<browser_rules>
diff --git a/browser_use/agent/system_prompt_no_thinking.md b/browser_use/agent/system_prompt_no_thinking.md
index a79cb569b..c155da1f3 100644
--- a/browser_use/agent/system_prompt_no_thinking.md
+++ b/browser_use/agent/system_prompt_no_thinking.md
@@ -66,8 +66,8 @@ Note that:
</browser_state>
<browser_vision>
-You will be optionally provided with a screenshot of the browser with bounding boxes. This is your GROUND TRUTH: analyze the image to evaluate your progress.
-Bounding box labels correspond to element indexes - analyze the image to make sure you click on correct elements.
+You will be provided with a screenshot of the current page with bounding boxes around interactive elements. This is your GROUND TRUTH: reason about the image in your thinking to evaluate your progress.
+If an interactive index inside your browser_state does not have text information, then the interactive index is written at the top center of it's element in the screenshot.
</browser_vision>
<browser_rules>
diff --git a/browser_use/browser/python_highlights.py b/browser_use/browser/python_highlights.py
index a8e7f4d1d..dfd23976a 100644
--- a/browser_use/browser/python_highlights.py
+++ b/browser_use/browser/python_highlights.py
@@ -131,15 +131,17 @@ def draw_enhanced_bounding_box_with_text(
container_width = text_width + padding * 2
container_height = text_height + padding * 2
- # Position in top-left corner (inside if fits, outside if too small)
- if element_width >= container_width and element_height >= container_height:
- # Place inside top-left corner
- bg_x1 = x1 + 2 # Small offset from edge
- bg_y1 = y1 + 2
+ # Position in top center - for small elements, place further up to avoid blocking content
+ # Center horizontally within the element
+ bg_x1 = x1 + (element_width - container_width) // 2
+
+ # Simple rule: if element is small, place index further up to avoid blocking icons
+ if element_width < 60 or element_height < 30:
+ # Small element: place well above to avoid blocking content
+ bg_y1 = max(0, y1 - container_height - 5)
else:
- # Place outside top-left corner
- bg_x1 = x1
- bg_y1 = max(0, y1 - container_height)
+ # Regular element: place inside with small offset
+ bg_y1 = y1 + 2
bg_x2 = bg_x1 + container_width
bg_y2 = bg_y1 + container_height
@@ -482,7 +484,7 @@ async def create_highlighted_screenshot_async(
async def _write_screenshot():
with open(filename, 'wb') as f:
f.write(base64.b64decode(final_screenshot))
- logger.debug('Saved screenshot to screenshot.png')
+ logger.debug('Saved screenshot to ' + str(filename))
await asyncio.to_thread(_write_screenshot)
return final_screenshot