Skip to content

Open-source Python library for AI browser agents, plus Browser Use Cloud: stealth browsers and a hosted agent API.

Document sitemap URL discovery helper

Magnus Müller8621e27e6ccc
diff --git a/RUST_AGENT_INTEGRATION_FEATURES.md b/RUST_AGENT_INTEGRATION_FEATURES.md
index 37d914c59..e80e9625b 100644
--- a/RUST_AGENT_INTEGRATION_FEATURES.md
+++ b/RUST_AGENT_INTEGRATION_FEATURES.md
@@ -261,6 +261,11 @@ Terminal core branch: `magnus/browser-use-rust-integration` at latest pulled mai
- The helper surfaces visible commercial product/plan/package cards with normalized price amounts, currency and billing period, speed/data/network signals, contract terms, offer labels, provider candidates, links, images, and stable selectors.
- Proof: `browser_script_pricing_cards_snapshot_surfaces_commercial_signals`.
+51. Browser-script sitemap URL discovery helper
+ - Terminal browser scripts now expose `sitemap_urls_snapshot(url_or_domain=None, keywords=None, limit=80, max_sitemaps=8, timeout=10.0)`.
+ - The helper reads robots.txt and XML sitemap indexes, follows one nested sitemap level, ranks public routes by task keywords and listing/product/document/search route patterns, and demotes static assets.
+ - Proof: `browser_script_sitemap_urls_snapshot_discovers_public_routes`.
+
## Current Verification
- `python3 -m py_compile browser_use/rust/service.py browser_use/rust/__init__.py browser_use/__init__.py tests/ci/test_rust_agent.py examples/rust_agent/basic.py examples/rust_agent/real_v8_smoke.py`
@@ -289,6 +294,7 @@ Terminal core branch: `magnus/browser-use-rust-integration` at latest pulled mai
- `CARGO_INCREMENTAL=0 cargo test -q -p browser-use-browser browser_script_browser_fetch_many_preserves_order_errors_and_binary -- --nocapture`
- `CARGO_INCREMENTAL=0 cargo test -q -p browser-use-browser browser_script_repeated_item_helpers_surface_actionable_records -- --nocapture`
- `CARGO_INCREMENTAL=0 cargo test -q -p browser-use-browser browser_script_pricing_cards_snapshot_surfaces_commercial_signals -- --nocapture`
+- `CARGO_INCREMENTAL=0 cargo test -q -p browser-use-browser browser_script_sitemap_urls_snapshot_discovers_public_routes -- --nocapture`
- Managed-headless end-to-end:
- `BROWSER_USE_TERMINAL_BINARY=/home/exedev/Developer/terminal/target/debug/browser-use-terminal BROWSER_USE_RUST_BROWSER_MODE=managed-headless BU_TASK='Open https://example.com and report the page title only.' BU_MAX_STEPS=12 timeout 300 uv run python examples/rust_agent/basic.py`
- Output: `Example Domain`