🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
feat(mcp): Add tool to take Aria Snapshot of the page for AI
Karim shoair31ba8e7986e9
diff --git a/docs/ai/mcp-server.md b/docs/ai/mcp-server.mdindex a0abcae..cb32b12 100644--- a/docs/ai/mcp-server.md+++ b/docs/ai/mcp-server.md@@ -6,7 +6,7 @@ The **Scrapling MCP Server** is a new feature that brings Scrapling's powerful W## Features-The Scrapling MCP Server provides thirteen powerful tools for web scraping, split into two modes: one-shot tools that each launch and close their own browser/client, and session tools that open a browser or an HTTP session once and then work through it.+The Scrapling MCP Server provides fourteen powerful tools for web scraping, split into two modes: one-shot tools that each launch and close their own browser/client, and session tools that open a browser or an HTTP session once and then work through it.### One-shot tools@@ -31,12 +31,16 @@ The Scrapling MCP Server provides thirteen powerful tools for web scraping, spli- **`list_sessions`**: List all active sessions with their details and `settings`.#### 🎯 Fetching Through a Session-- **`session_fetch`**: Fetch a single URL through an open browser session (dynamic or stealthy), carrying the per-request options for that call. This is the session counterpart of `fetch`/`stealthy_fetch`.+- **`session_fetch`**: Fetch a single URL through an open browser session (dynamic or stealthy), carrying the per-request options for that call. This is the session counterpart of `fetch`/`stealthy_fetch`. Set `extraction_type="snapshot"` to return an AI ARIA snapshot in the usual response's `content` list.- **`session_make_request`**: Make an HTTP request with any method through a session opened with `open_request_session`, reusing its cookies, connections, and browser fingerprint. This is the session counterpart of `make_request`.#### 📸 Screenshots- **`screenshot`**: Capture a PNG or JPEG screenshot of a page using an open browser session, returned as an image content block the model can actually see (not a base64 string blob). Supports full-page captures, JPEG quality, and the usual readiness controls (`wait`, `wait_selector`, `network_idle`).+#### AI Snapshots++- **`browser_snapshot`**: Read the current whole page in an open browser session as an AI ARIA snapshot. Returns plain text with element roles, names, and references without loading the page again. Supports `depth` and `boxes`.+### Shadow DOMSet `pierce_shadow=true` on `fetch`, `bulk_fetch`, `stealthy_fetch`, `bulk_stealthy_fetch`, or `session_fetch` to include open Shadow DOM content. It defaults to `false`. For persistent sessions, pass it on each `session_fetch` call. See [Shadow DOM](../fetching/dynamic.md#shadow-dom) for selector examples and limits.@@ -204,7 +208,7 @@ You can also set the `SCRAPLING_EXECUTABLE_PATH` environment variable before sta```Open a stealthy browser session on wss://cdp.provider.example/session/abc123, then use it to scrape the product details from https://shop.example.com. Close the session when you're done.```-Both browser session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with `session_fetch` and `screenshot` as usual.+Both browser session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with `session_fetch`, `browser_snapshot`, and `screenshot` as usual.The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser you started yourself with the remote debugging port enabled:```commandlinediff --git a/docs/api-reference/mcp-server.md b/docs/api-reference/mcp-server.mdindex fc50af8..837faf6 100644--- a/docs/api-reference/mcp-server.md+++ b/docs/api-reference/mcp-server.md@@ -5,7 +5,7 @@ search:# MCP Server API Reference-The **Scrapling MCP Server** provides eleven powerful tools for web scraping through the Model Context Protocol (MCP). This server integrates Scrapling's capabilities directly into AI chatbots and agents, allowing conversational web scraping with advanced anti-bot bypass features.+The **Scrapling MCP Server** provides fourteen powerful tools for web scraping through the Model Context Protocol (MCP). This server integrates Scrapling's capabilities directly into AI chatbots and agents, allowing conversational web scraping with advanced anti-bot bypass features.You can start the MCP server by running:@@ -30,7 +30,7 @@ server = ScraplingMCPServer(executable_path="/path/to/chromium")## Response Model-The standardized response structure that's returned by all MCP server tools:+The standardized response structure returned by the fetch and HTTP request tools. Bulk tools return a list of these responses. Session management tools use the models below, `screenshot` returns image and text content blocks, and `browser_snapshot` returns plain text.## ::: scrapling.core.ai.ResponseModelhandler: python@@ -56,6 +56,10 @@ Model classes for session management:The main MCP server class that provides all web scraping tools:+`browser_snapshot(session_id, depth=None, boxes=False)` returns a plain-text AI ARIA snapshot of the current whole page in a dynamic or stealthy session. Use it after `session_fetch` finishes. It reads the existing page without navigation and includes element references. `depth` limits the tree depth, and `boxes=True` adds element positions and sizes in viewport CSS pixels. An unknown session, an HTTP request session, or a missing, closed, or busy page returns an error.++`session_fetch` also accepts `extraction_type="snapshot"` to take an AI ARIA snapshot after loading the requested URL. It returns the usual `status`, `url`, and `content` fields, with one unchanged snapshot string in `content`. The existing `css_selector` can scope the snapshot to exactly one element; zero or multiple matches return an error. Without a selector, the snapshot covers the whole page. `main_content_only` and `pierce_shadow` do not filter snapshots. This extraction type is only available on the MCP `session_fetch` tool; browser fetchers still return the same `Response` object.+## ::: scrapling.core.ai.ScraplingMCPServerhandler: python:docstring:diff --git a/scrapling/core/ai.py b/scrapling/core/ai.pyindex cd12284..b9f7454 100644--- a/scrapling/core/ai.py+++ b/scrapling/core/ai.py@@ -45,6 +45,7 @@ from scrapling.core._types import (SessionType = Literal["dynamic", "stealthy", "static"]BrowserSessionType = Literal["dynamic", "stealthy"]+SessionExtractionType = Literal[extraction_types, "snapshot"]ScreenshotType = Literal["png", "jpeg"]MCP_EXECUTABLE_PATH_ENV = "SCRAPLING_EXECUTABLE_PATH"MCP_AUTH_TOKEN_ENV = "SCRAPLING_MCP_AUTH_TOKEN" # nosec B105 - the name of the variable, not a token@@ -97,7 +98,7 @@ class ResponseModel(BaseModel):"""Request's response information structure."""status: int = Field(description="The status code returned by the website.")- content: list[str] = Field(description="The content as Markdown/HTML or the text content of the page.")+ content: list[str] = Field(description="The page content as Markdown, HTML, text, or an AI ARIA snapshot.")url: str = Field(description="The URL given by the user that resulted in this response.")@@ -355,6 +356,53 @@ class ScraplingMCPServer:for sid, entry in self._sessions.items()]+ async def browser_snapshot(+ self,+ session_id: str,+ depth: Optional[int] = None,+ boxes: bool = False,+ ) -> str:+ """Return the AI ARIA snapshot of the current page in an open browser session as plain text.+ Fetch a page with `session_fetch` first. The snapshot includes element references and preserves the current page.++ :param session_id: ID of an open browser session created with `open_session`.+ :param depth: Limit the snapshot tree depth. Defaults to no limit.+ :param boxes: Include element bounding boxes in viewport CSS pixels.+ """+ return await self._browser_snapshot(session_id, depth=depth, boxes=boxes)++ async def _browser_snapshot(+ self,+ session_id: str,+ css_selector: Optional[str] = None,+ depth: Optional[int] = None,+ boxes: bool = False,+ ) -> str:+ """Reserve the current page and return its AI ARIA snapshot."""+ entry = self._get_session(session_id, expected_type=None)+ if entry.session_type == "static":+ raise ValueError(+ f"Session '{session_id}' is a 'static' session, so it can't take snapshots. "+ f"Open a 'dynamic' or 'stealthy' session for that."+ )++ pool = entry.session.page_pool+ if not pool.pages_count:+ raise ValueError(f"Session '{session_id}' has no page to snapshot. Use session_fetch first.")+ page_info = pool.get_ready_page()+ if page_info is None:+ raise RuntimeError(f"Session '{session_id}' has a busy page. Wait for the current request to finish.")++ try:+ if page_info.page.is_closed():+ raise RuntimeError(f"Session '{session_id}' has a closed page. Use session_fetch to open a new one.")+ return await entry.session._snapshot(page_info.page, depth=depth, boxes=boxes, css_selector=css_selector)+ finally:+ if page_info.page.is_closed():+ pool.remove_page(page_info)+ else:+ page_info.mark_ready()+async def screenshot(self,url: str,@@ -930,7 +978,7 @@ class ScraplingMCPServer:self,url: str,session_id: str,- extraction_type: extraction_types = "markdown",+ extraction_type: SessionExtractionType = "markdown",css_selector: Optional[str] = None,main_content_only: bool = True,wait: int | float = 0,@@ -951,10 +999,10 @@ class ScraplingMCPServer::param url: The URL to request.:param session_id: ID of an open browser session created with `open_session`.- :param extraction_type: The type of content to extract from the page: "markdown", "html", or "text".+ :param extraction_type: The type of content to extract from the page: "markdown", "html", "text", or "snapshot".:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page.:param main_content_only: Whether to extract only the main content of the page. The main content here is the data inside the `<body>` tag.- :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object.+ :param wait: The time (milliseconds) to wait after the page is ready, before returning the `Response` object.:param timeout: The timeout in milliseconds that is used in all operations and waits through the page.:param google_search: Enabled by default, Scrapling will set a Google referer header.:param pierce_shadow: Include open Shadow DOM content in the response. Defaults to False.@@ -997,6 +1045,10 @@ class ScraplingMCPServer:page = await entry.session.fetch(url, **{name: value for name, value in fetch_params.items() if name in fetch_keys})+ if extraction_type == "snapshot":+ return ResponseModel(+ status=page.status, content=[await self._browser_snapshot(session_id, css_selector)], url=page.url+ )return _translate_response(page, extraction_type, css_selector, main_content_only)async def session_make_request(@@ -1098,9 +1150,9 @@ class ScraplingMCPServer:1. When the `open_session` or `open_request_session` tools are used, make sure to close the session with `close_session` after you finish, and use `list_sessions` if you lose track of the open sessions or their effective settings.2. If the user didn't specify which tool to use, start with the `make_request` tool (a plain HTTP request, defaulting to GET; set `method` for POST/PUT/DELETE), then escalate. The `make_request` tool and `bulk_get` (its GET-only bulk version) are suitable only for low-to-mid protection levels.For high-protection levels or websites that require JS loading, use the other tools directly.-3. For all tools, if the `css_selector` resolves to more than one element, all the elements will be returned.+3. For HTML, Markdown, and text extraction, if the `css_selector` resolves to more than one element, all the elements will be returned. Snapshot extraction requires a selector matching exactly one element, or no selector for the whole page.4. For all fetch tools, the `extraction_type` parameter controls the format of the returned content: "markdown" (default) converts the page content to Markdown, "html" returns the raw HTML, and "text" returns the text content of the page.-5. For all fetch tools, `main_content_only` is enabled by default and returns only the content inside the page's `<body>` tag. Pass `main_content_only=False` when you need the full page instead.+5. For HTML, Markdown, and text extraction, `main_content_only` is enabled by default and returns only the content inside the page's `<body>` tag. Pass `main_content_only=False` when you need the full page instead.6. If the task consists of multiple sequential requests to the same website, open a session once, then fetch through it to be more efficient:`open_session` + `session_fetch` per page for browsers, or `open_request_session` + `session_make_request` per request for plain HTTP.7. Sessions hold the session-level configuration set when opened, while `session_fetch`/`session_make_request` carry the per-request options and apply them on each call with the defaults shown in their schemas.@@ -1108,6 +1160,7 @@ class ScraplingMCPServer:8. If you are making multiple parallel one-shot requests, use the bulk version of the tool to be more efficient.9. If you are crawling/browsing a website, be more efficient by using the `css_selector` parameter to only access the parts you are interested in and save money/time. Example: use the `a` selector to extract the urls right away.10. The user can pass a CDP URL to connect to a remote browser session through the `open_session` tool, then use it with the session tools.+11. Set `extraction_type="snapshot"` on `session_fetch` to get an AI ARIA snapshot with element references in its content field. Use `css_selector` to snapshot one element, or omit it for the whole page. `main_content_only` and `pierce_shadow` do not filter snapshots. Use `browser_snapshot` to read the current page again without navigating; it returns plain text. Its `depth` and `boxes` options limit the tree or include element bounding boxes.""",}if self._auth_token:@@ -1192,6 +1245,13 @@ class ScraplingMCPServer:structured_output=True,annotations=_FETCH_TOOL_ANNOTATIONS,)+ server.add_tool(+ self.browser_snapshot,+ title="browser_snapshot",+ description=self.browser_snapshot.__doc__,+ structured_output=False,+ annotations=_FETCH_TOOL_ANNOTATIONS,+ )# Screenshot tool (returns image + url content blocks, not structured JSON)server.add_tool(self.screenshot,diff --git a/scrapling/engines/_browsers/_base.py b/scrapling/engines/_browsers/_base.pyindex 5c12f7d..a7fe0a5 100644--- a/scrapling/engines/_browsers/_base.py+++ b/scrapling/engines/_browsers/_base.py@@ -138,6 +138,19 @@ class SyncSession:"max_pages": self.max_pages,}+ @staticmethod+ def _snapshot(+ page: Page, depth: Optional[int] = None, boxes: bool = False, css_selector: Optional[str] = None+ ) -> str:+ """Return an AI ARIA snapshot of an existing browser page.++ :param page: The browser page to snapshot.+ :param depth: Limit the snapshot tree depth. Defaults to no limit.+ :param boxes: Include element bounding boxes in viewport CSS pixels.+ :param css_selector: Snapshot one matching element instead of the whole page.+ """+ return (page.locator(css_selector) if css_selector else page).aria_snapshot(mode="ai", depth=depth, boxes=boxes)+@staticmethoddef _wait_for_networkidle(page: Page | Frame, timeout: Optional[int] = None):"""Wait for the page to become idle (no network activity) even if there are never-ending requests."""@@ -338,6 +351,21 @@ class AsyncSession:"max_pages": self.max_pages,}+ @staticmethod+ async def _snapshot(+ page: AsyncPage, depth: Optional[int] = None, boxes: bool = False, css_selector: Optional[str] = None+ ) -> str:+ """Return an AI ARIA snapshot of an existing browser page.++ :param page: The browser page to snapshot.+ :param depth: Limit the snapshot tree depth. Defaults to no limit.+ :param boxes: Include element bounding boxes in viewport CSS pixels.+ :param css_selector: Snapshot one matching element instead of the whole page.+ """+ return await (page.locator(css_selector) if css_selector else page).aria_snapshot(+ mode="ai", depth=depth, boxes=boxes+ )+@staticmethodasync def _wait_for_networkidle(page: AsyncPage | AsyncFrame, timeout: Optional[int] = None):"""Wait for the page to become idle (no network activity) even if there are never-ending requests."""diff --git a/tests/ai/test_ai_mcp.py b/tests/ai/test_ai_mcp.pyindex 709ae31..a1f3e79 100644--- a/tests/ai/test_ai_mcp.py+++ b/tests/ai/test_ai_mcp.py@@ -856,7 +856,7 @@ class TestMCPServerAuthentication:monkeypatch.delenv(MCP_AUTH_TOKEN_ENV, raising=False)built = ScraplingMCPServer(auth_token=SHARED_KEY)._build_server("0.0.0.0", 8000)- assert len(built._tool_manager.list_tools()) == 13+ assert len(built._tool_manager.list_tools()) == 14def test_http_without_a_token_refuses_to_serve(self, monkeypatch):"""The streamable-http transport requires authentication unless the caller explicitly opts out"""@@ -913,15 +913,21 @@ class TestServerToolRegistration:@pytest.mark.asyncioasync def test_tools_are_listed_with_expected_schemas(self):- """All 13 tools are advertised, and only the screenshot tool skips the structured output schema"""+ """All 14 tools are advertised, with plain content for screenshots and snapshots"""server = ScraplingMCPServer()._build_server("127.0.0.1", 8000)async with Client(server) as client:assert client.instructionstools = {tool.name: tool for tool in (await client.list_tools()).tools}- assert len(tools) == 13+ assert len(tools) == 14assert tools["screenshot"].output_schema is None- assert all(tool.output_schema is not None for name, tool in tools.items() if name != "screenshot")+ assert tools["browser_snapshot"].output_schema is None+ assert "session_snapshot" not in tools+ assert all(+ tool.output_schema is not None+ for name, tool in tools.items()+ if name not in ("screenshot", "browser_snapshot")+ )@pytest.mark.asyncioasync def test_fetch_tools_expose_real_defaults_and_no_session_id(self):@@ -937,6 +943,7 @@ class TestServerToolRegistration:assert props["google_search"]["default"] is Trueassert props["pierce_shadow"]["type"] == "boolean"assert props["pierce_shadow"]["default"] is False+ assert "snapshot" not in props["extraction_type"]["enum"]request_props = tools["make_request"].input_schema["properties"]assert request_props["method"]["default"] == "GET"@@ -954,6 +961,9 @@ class TestServerToolRegistration:assert session_props["pierce_shadow"]["type"] == "boolean"assert session_props["pierce_shadow"]["default"] is Falseassert "solve_cloudflare" in session_props+ assert session_props["extraction_type"]["default"] == "markdown"+ assert set(session_props["extraction_type"]["enum"]) == {"markdown", "html", "text", "snapshot"}+ assert {"depth", "boxes"}.isdisjoint(session_props)assert set(tools["session_fetch"].input_schema["required"]) >= {"url", "session_id"}open_props = set(tools["open_session"].input_schema["properties"])@@ -976,7 +986,7 @@ class TestServerToolRegistration:assert result.ttl_ms == 3_600_000 and result.cache_scope == "public"annotations = {tool.name: tool.annotations for tool in result.tools if tool.annotations is not None}- assert len(annotations) == 13+ assert len(annotations) == 14for name in ("make_request","bulk_get",@@ -986,6 +996,7 @@ class TestServerToolRegistration:"bulk_stealthy_fetch","session_fetch","session_make_request",+ "browser_snapshot","screenshot",):assert annotations[name].read_only_hint is Truediff --git a/tests/ai/test_browser_snapshot.py b/tests/ai/test_browser_snapshot.pynew file mode 100644index 0000000..9136882--- /dev/null+++ b/tests/ai/test_browser_snapshot.py@@ -0,0 +1,297 @@+import asyncio+from os import getenv+from re import search+from unittest.mock import AsyncMock, Mock++import pytest+from mcp.client import Client+from mcp.types import TextContent++from scrapling.core.ai import ScraplingMCPServer, SessionType, _SessionEntry+from scrapling.core._types import Any+from scrapling.engines._browsers._base import AsyncSession+from scrapling.engines.toolbelt.custom import Response+++SNAPSHOT = '- button "Save" [ref=e2]'+HTML = """<!DOCTYPE html><html><body><main><label>Name<input value="initial"></label>+<button onclick="document.querySelector('output').textContent = document.querySelector('input').value">Save</button>+<output></output></main><button>Outside scope</button><script>+setTimeout(() => { document.querySelector('output').textContent = 'Ready'; document.querySelector('output').id = 'ready'; }, 50);+setTimeout(() => { document.querySelector('input').value = 'after wait'; }, 150);+</script></body></html>"""+++def _server(session_type: SessionType = "dynamic") -> tuple[ScraplingMCPServer, AsyncSession, Mock]:+ server = ScraplingMCPServer()+ session = AsyncSession()+ session._is_alive = True+ page = Mock()+ page.is_closed.return_value = False+ page.aria_snapshot = AsyncMock(return_value=SNAPSHOT)+ session.page_pool.add_page(page).mark_ready()+ server._sessions["browser"] = _SessionEntry(session, session_type)+ return server, session, page++[email protected]("session_type", ["dynamic", "stealthy"])+async def test_browser_snapshot_returns_plain_mcp_text(session_type: SessionType) -> None:+ server, session, page = _server(session_type)+ async with Client(server._build_server("127.0.0.1", 8000)) as client:+ tools = {tool.name: tool for tool in (await client.list_tools()).tools}+ assert "session_snapshot" not in tools+ tool = tools["browser_snapshot"]+ assert tool.output_schema is None+ assert set(tool.input_schema["properties"]) == {"session_id", "depth", "boxes"}+ assert tool.input_schema["required"] == ["session_id"]+ fetch_props = tools["session_fetch"].input_schema["properties"]+ assert set(fetch_props["extraction_type"]["enum"]) == {"markdown", "html", "text", "snapshot"}+ assert "depth" not in fetch_props and "boxes" not in fetch_props+ assert "snapshot" not in tools["fetch"].input_schema["properties"]["extraction_type"]["enum"]+ result = await client.call_tool("browser_snapshot", {"session_id": "browser", "depth": 3, "boxes": True})+ assert not result.is_error+ assert result.structured_content is None+ assert result.content == [TextContent(type="text", text=SNAPSHOT)]+ page.aria_snapshot.assert_awaited_once_with(mode="ai", depth=3, boxes=True)+ assert session.page_pool.pages_count == 1+ assert session.page_pool.pages[0].state == "ready"+ page.goto.assert_not_called()+ page.set_extra_http_headers.assert_not_called()+ page.unroute_all.assert_not_called()+ page.set_default_timeout.assert_not_called()+ page.set_default_navigation_timeout.assert_not_called()+++ "state, message",+ [+ ("missing", "not found"),+ ("dead", "no longer alive"),+ ("static", "'static'"),+ ("empty", "session_fetch"),+ ("busy", "busy"),+ ("closed", "closed"),+ ],+)+async def test_browser_snapshot_errors_reach_the_mcp_client(state: str, message: str) -> None:+ server, session, page = _server("static" if state == "static" else "dynamic")+ if state == "missing":+ server._sessions.clear()+ elif state == "dead":+ session._is_alive = False+ elif state == "empty":+ session.page_pool.clear()+ elif state == "busy":+ session.page_pool.pages[0].mark_busy()+ elif state == "closed":+ page.is_closed.return_value = True+ async with Client(server._build_server("127.0.0.1", 8000)) as client:+ result = await client.call_tool("browser_snapshot", {"session_id": "browser"})+ assert result.is_error+ assert result.content and isinstance(result.content[0], TextContent)+ assert message in result.content[0].text+ page.aria_snapshot.assert_not_awaited()+ if state == "closed":+ assert session.page_pool.pages_count == 0+ elif state == "busy":+ assert session.page_pool.pages[0].state == "busy"+++async def test_browser_snapshot_releases_the_page_after_failure() -> None:+ server, session, page = _server()+ page.aria_snapshot.side_effect = TimeoutError("snapshot timeout")+ with pytest.raises(TimeoutError, match="snapshot timeout"):+ await server.browser_snapshot("browser")+ assert session.page_pool.pages[0].state == "ready"+ page.close.assert_not_called()+ page.aria_snapshot.side_effect = None+ assert await server.browser_snapshot("browser") == SNAPSHOT++[email protected]("session_type", ["dynamic", "stealthy"])[email protected]("css_selector", [None, "main"])+async def test_browser_snapshot_live_mcp_round_trip(session_type: SessionType, css_selector: str | None) -> None:+ server = ScraplingMCPServer(executable_path=getenv("SCRAPLING_EXECUTABLE_PATH"))+ requests: list[str] = []++ async def serve(route: Any) -> None:+ if route.request.is_navigation_request():+ requests.append(route.request.url)+ await route.fulfill(+ status=201,+ content_type="text/html",+ body=HTML,+ )++ async with Client(server._build_server("127.0.0.1", 8000)) as client:+ opened = await client.call_tool("open_session", {"session_type": session_type, "session_id": "browser"})+ assert not opened.is_error+ try:+ session = server._sessions["browser"].session+ await session.context.route("**/*", serve)+ fetched = await client.call_tool(+ "session_fetch",+ {+ "session_id": "browser",+ "url": "https://snapshot.test/",+ "google_search": False,+ "extraction_type": "snapshot",+ "css_selector": css_selector,+ "wait_selector": "#ready",+ "wait": 250,+ },+ )+ assert not fetched.is_error+ assert fetched.structured_content is not None+ assert fetched.structured_content["status"] == 201+ assert fetched.structured_content["url"] == "https://snapshot.test/"+ fetched_snapshot = fetched.structured_content["content"][0]+ assert "after wait" in fetched_snapshot and "initial" not in fetched_snapshot+ assert ("Outside scope" in fetched_snapshot) is (css_selector is None)+ assert requests == ["https://snapshot.test/"]+ page_info = session.page_pool.pages[0]+ page = page_info.page+ match = search(r'button "Save".*?\[ref=([^\]]+)\]', fetched_snapshot)+ assert match is not None+ await page.locator(f"aria-ref={match[1]}").click()+ assert await page.locator("output").inner_text() == "after wait"+ await page.get_by_label("Name").fill("live value")+ navigations: list[str] = []+ page.on("framenavigated", lambda frame: navigations.append(frame.url))+ previous_requests = requests.copy()+ result = await client.call_tool("browser_snapshot", {"session_id": "browser", "boxes": True})+ assert not result.is_error+ assert result.structured_content is None+ assert len(result.content) == 1 and isinstance(result.content[0], TextContent)+ snapshot = result.content[0].text+ assert "live value" in snapshot and "initial" not in snapshot+ assert "Outside scope" in snapshot+ assert "[box=" in snapshot+ assert requests == previous_requests and not navigations+ assert session.page_pool.pages == [page_info] and page_info.state == "ready"+ match = search(r'button "Save".*?\[ref=([^\]]+)\]', snapshot)+ assert match is not None+ await page.locator(f"aria-ref={match[1]}").click()+ assert await page.locator("output").inner_text() == "live value"+ for selector in ("#missing", "button", "["):+ previous_count = len(requests)+ failed = await client.call_tool(+ "session_fetch",+ {+ "session_id": "browser",+ "url": "https://snapshot.test/",+ "google_search": False,+ "extraction_type": "snapshot",+ "css_selector": selector,+ },+ )+ assert failed.is_error+ assert len(requests) == previous_count + 1+ assert session.page_pool.pages == [page_info] and page_info.state == "ready"+ finally:+ closed = await client.call_tool("close_session", {"session_id": "browser"})+ assert not closed.is_error+++async def test_browser_snapshot_reserves_and_releases_the_page_on_cancel() -> None:+ server, session, page = _server()+ entered = asyncio.Event()+ release = asyncio.Event()++ async def capture(**kwargs: object) -> str:+ entered.set()+ await release.wait()+ return SNAPSHOT++ page.aria_snapshot.side_effect = capture+ task = asyncio.create_task(server.browser_snapshot("browser"))+ try:+ await asyncio.wait_for(entered.wait(), 5)+ assert session.page_pool.pages[0].state == "busy"+ assert session.page_pool.get_ready_page() is None+ with pytest.raises(RuntimeError):+ await server.browser_snapshot("browser")+ finally:+ task.cancel()+ with pytest.raises(asyncio.CancelledError):+ await task+ assert session.page_pool.pages[0].state == "ready"+ page.close.assert_not_called()+++ "extraction_type, expected", [("markdown", ["Hello", ""]), ("html", ["<p>Hello</p>", ""]), ("text", ["Hello", ""])]+)+async def test_session_fetch_keeps_existing_formats(extraction_type: Any, expected: list[str]) -> None:+ server, session, page = _server()+ response = Response(+ url="https://snapshot.test/final",+ content="<main><p>Hello</p></main>",+ status=201,+ reason="Created",+ cookies={},+ headers={},+ request_headers={},+ )+ fetch = AsyncMock(return_value=response)+ setattr(session, "fetch", fetch)+ result = await server.session_fetch("https://snapshot.test/", "browser", extraction_type, "p")+ assert result.content == expected+ assert result.status == 201+ assert result.url == "https://snapshot.test/final"+ fetch.assert_awaited_once()+ page.aria_snapshot.assert_not_awaited()+++async def test_session_fetch_snapshot_keeps_raw_content_and_response_metadata() -> None:+ server, session, page = _server()+ snapshot = '\n- button "Save" [ref=e2]\n'+ page.aria_snapshot.return_value = snapshot+ response = Response(+ url="https://snapshot.test/redirected",+ content="<p>Hello</p>",+ status=202,+ reason="Accepted",+ cookies={},+ headers={},+ request_headers={},+ )+ fetch = AsyncMock(return_value=response)+ setattr(session, "fetch", fetch)+ result = await server.session_fetch("https://snapshot.test/start", "browser", extraction_type="snapshot")+ assert result.content == [snapshot]+ assert result.status == 202+ assert result.url == "https://snapshot.test/redirected"+ fetch.assert_awaited_once()+ assert {"extraction_type", "css_selector", "depth", "boxes"}.isdisjoint(fetch.call_args.kwargs)+++async def test_session_fetch_snapshot_error_does_not_refetch() -> None:+ server, session, page = _server()+ response = Response(+ url="https://snapshot.test/final",+ content="<p>Hello</p>",+ status=201,+ reason="Created",+ cookies={},+ headers={},+ request_headers={},+ )+ fetch = AsyncMock(return_value=response)+ setattr(session, "fetch", fetch)+ page.locator.return_value.aria_snapshot = AsyncMock(side_effect=TimeoutError("snapshot timeout"))+ with pytest.raises(TimeoutError, match="snapshot timeout"):+ await server.session_fetch("https://snapshot.test/", "browser", extraction_type="snapshot", css_selector="main")+ fetch.assert_awaited_once()+ assert {"extraction_type", "css_selector", "depth", "boxes"}.isdisjoint(fetch.call_args.kwargs)+ page.aria_snapshot.assert_not_awaited()+ assert session.page_pool.pages[0].state == "ready"+ page.close.assert_not_called()diff --git a/tests/fetchers/test_snapshot.py b/tests/fetchers/test_snapshot.pynew file mode 100644index 0000000..0e520a3--- /dev/null+++ b/tests/fetchers/test_snapshot.py@@ -0,0 +1,106 @@+from os import getenv+from re import search+from unittest.mock import AsyncMock, Mock++import pytest++from scrapling.core._types import Any+from scrapling.fetchers import AsyncDynamicSession, AsyncStealthySession, DynamicSession, StealthySession+++HTML = """<!DOCTYPE html><html><body>+<main><label>Name<input value="initial"></label>+<button onclick="document.querySelector('output').textContent = document.querySelector('input').value">Save</button>+<output></output></main><p>Outside scope</p><script>document.body.dataset.snapshotState = 'kept';</script>+</body></html>"""++[email protected]("session_type", [DynamicSession, StealthySession])[email protected]("css_selector", [None, "main"])+def test_sync_snapshot_forwards_options_and_errors(session_type: Any, css_selector: str | None) -> None:+ page = Mock()+ target = page.locator.return_value if css_selector else page+ target.aria_snapshot.return_value = "snapshot"+ assert session_type._snapshot(page, depth=3, boxes=True, css_selector=css_selector) == "snapshot"+ target.aria_snapshot.assert_called_once_with(mode="ai", depth=3, boxes=True)+ if css_selector:+ page.locator.assert_called_once_with(css_selector)+ page.aria_snapshot.assert_not_called()+ else:+ page.locator.assert_not_called()+ target.aria_snapshot.side_effect = TimeoutError("snapshot timeout")+ with pytest.raises(TimeoutError, match="snapshot timeout"):+ session_type._snapshot(page, css_selector=css_selector)++[email protected]("session_type", [AsyncDynamicSession, AsyncStealthySession])[email protected]("css_selector", [None, "main"])+async def test_async_snapshot_forwards_options_and_errors(session_type: Any, css_selector: str | None) -> None:+ page = Mock()+ target = page.locator.return_value if css_selector else page+ target.aria_snapshot = AsyncMock(return_value="snapshot")+ assert await session_type._snapshot(page, depth=3, boxes=True, css_selector=css_selector) == "snapshot"+ target.aria_snapshot.assert_awaited_once_with(mode="ai", depth=3, boxes=True)+ if css_selector:+ page.locator.assert_called_once_with(css_selector)+ page.aria_snapshot.assert_not_called()+ else:+ page.locator.assert_not_called()+ target.aria_snapshot.side_effect = TimeoutError("snapshot timeout")+ with pytest.raises(TimeoutError, match="snapshot timeout"):+ await session_type._snapshot(page, css_selector=css_selector)+++def _button_ref(snapshot: str) -> str:+ assert "live value" in snapshot+ assert "initial" not in snapshot+ assert "[box=" in snapshot+ match = search(r'button "Save".*?\[ref=([^\]]+)\]', snapshot)+ assert match is not None+ return match[1]++[email protected]("session_type", [DynamicSession, StealthySession])+def test_sync_snapshot_live_state_and_refs(session_type: Any) -> None:+ def setup(page: Any) -> None:+ page.route("**/*", lambda route: route.fulfill(status=200, content_type="text/html", body=HTML))++ with session_type(executable_path=getenv("SCRAPLING_EXECUTABLE_PATH"), google_search=False) as session:+ response = session.fetch("https://snapshot.test/", page_setup=setup)+ assert response.status == 200+ page = session.page_pool.pages[0].page+ page.get_by_label("Name").fill("live value")+ assert "Outside scope" in session_type._snapshot(page)+ snapshot = session_type._snapshot(page, depth=5, boxes=True)+ page.locator(f"aria-ref={_button_ref(snapshot)}").click()+ snapshot = session_type._snapshot(page, css_selector="main", boxes=True)+ assert "Outside scope" not in snapshot+ page.locator(f"aria-ref={_button_ref(snapshot)}").click()+ assert page.locator("output").inner_text() == "live value"+ assert page.locator("body").get_attribute("data-snapshot-state") == "kept"+ assert page.url == "https://snapshot.test/"+ assert session.page_pool.pages_count == 1++[email protected]("session_type", [AsyncDynamicSession, AsyncStealthySession])+async def test_async_snapshot_live_state_and_refs(session_type: Any) -> None:+ async def setup(page: Any) -> None:+ await page.route("**/*", lambda route: route.fulfill(status=200, content_type="text/html", body=HTML))++ async with session_type(executable_path=getenv("SCRAPLING_EXECUTABLE_PATH"), google_search=False) as session:+ response = await session.fetch("https://snapshot.test/", page_setup=setup)+ assert response.status == 200+ page = session.page_pool.pages[0].page+ await page.get_by_label("Name").fill("live value")+ assert "Outside scope" in await session_type._snapshot(page)+ snapshot = await session_type._snapshot(page, depth=5, boxes=True)+ await page.locator(f"aria-ref={_button_ref(snapshot)}").click()+ snapshot = await session_type._snapshot(page, css_selector="main", boxes=True)+ assert "Outside scope" not in snapshot+ await page.locator(f"aria-ref={_button_ref(snapshot)}").click()+ assert await page.locator("output").inner_text() == "live value"+ assert await page.locator("body").get_attribute("data-snapshot-state") == "kept"+ assert page.url == "https://snapshot.test/"+ assert session.page_pool.pages_count == 1