🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
feat(mcp): Add new tools to set and load browser session state
Karim shoair99a228192355
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zipindex 8f9b8ee..d627e1a 100644Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differdiff --git a/agent-skill/Scrapling-Skill/references/mcp-server.md b/agent-skill/Scrapling-Skill/references/mcp-server.mdindex 722cad2..5f5d898 100644--- a/agent-skill/Scrapling-Skill/references/mcp-server.md+++ b/agent-skill/Scrapling-Skill/references/mcp-server.md@@ -1,6 +1,6 @@# Scrapling MCP Server-The Scrapling MCP server exposes fourteen tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), plain HTTP requests and stealth browser rendering with anti-bot bypass, persistent browser session management, mouse actions, batch field filling, keyboard shortcuts, page waits, custom JavaScript, network history, and page screenshots returned as real image content blocks. Every fetch takes one URL. One-shot tools (`make_request`, `browser_fetch_once`) close their own client or browser after the call. Use sessions opened with `browser_open` or `open_request_session` for related requests, browser actions, or network history. Browser session calls reuse one tab, so fetch pages and run actions in order. Close each persistent session when done.+The Scrapling MCP server exposes sixteen tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), plain HTTP requests and stealth browser rendering with anti-bot bypass, persistent browser session management, mouse actions, batch field filling, keyboard shortcuts, page waits, custom JavaScript, network history, and page screenshots returned as real image content blocks. Every fetch takes one URL. One-shot tools (`make_request`, `browser_fetch_once`) close their own client or browser after the call. Use sessions opened with `browser_open` or `open_request_session` for related requests, browser actions, or network history. Browser session calls reuse one tab, so fetch pages and run actions in order. Close each persistent session when done.Fetch and HTTP request tools return a `ResponseModel` with fields: `status` (int), `content` (list of strings), `url` (str). The `browser_screenshot` tool returns a list of MCP content blocks: an `ImageContent` (the screenshot bytes) followed by a `TextContent` (the current page URL). `browser_extract` and `browser_actions` return plain text. Both network tools return structured JSON; the request list supports pagination. `browser_evaluate` returns compact JSON in one plain text block.@@ -111,6 +111,23 @@ Opens an HTTP session (no browser) that stays alive across multiple `session_mak| `impersonate` | str | `"chrome"` | Browser fingerprint to impersonate on every request || `proxy` | str or null | null | Proxy URL used for every request, e.g. `"http://user:pass@host:port"` |+### `browser_save_state` and `browser_load_state` -- Save or restore browser storage++Save cookies, local storage and IndexedDB to a Playwright-compatible JSON file, or replace an open browser session's stored state from that file. Both tools use the session's `save_state` and `load_state` methods. They work immediately after `browser_open`; no `browser_fetch` is required. HTTP sessions are not supported.++| Parameter | Type | Default | Description |+|-----------|------|---------|-------------|+| `session_id` | str | required | ID from `browser_open` |+| `path` | nonempty str | required | JSON file path on the MCP server host; relative paths use its working directory |++Saving overwrites the file, and its parent directory must exist. Loading replaces cookies, local storage and IndexedDB. It leaves `sessionStorage` and open tabs unchanged; fetch or reload pages to use the restored login. The file does not save `sessionStorage`, tabs or browser settings. Finish other calls in the session before saving or loading.++```json+{"session_id": "browser", "path": "state.json"}+```++Save returns `State saved.` and load returns `State loaded.` as plain text, with no storage values in the response. Unknown, closed or HTTP sessions, invalid paths, unreadable files and malformed JSON return errors. These tools do not open or close sessions.+### `browser_fetch` -- Fetch through an open browser session (single URL)Fetches one URL through a stealthy Chromium session opened with `browser_open`, with JavaScript rendering, fingerprint spoofing, and Cloudflare Turnstile/Interstitial bypass. The session holds the browser-level configuration; every parameter here applies to this request only. Raises on a requests session; use `session_make_request` there instead.@@ -356,6 +373,7 @@ The page stays reserved during capture and is released after success, failure, o| Single JavaScript-rendered / SPA page | `browser_fetch_once` || Single page with Cloudflare protection | `browser_fetch_once` with `solve_cloudflare=true` || Related HTTP requests | `open_request_session` + `session_make_request` per URL |+| Reuse browser login state across sessions | `browser_save_state` / `browser_load_state` with `session_id` and `path` || Browser automation or network history | `browser_open` + `browser_fetch` per URL || Capture the current page or an element | `browser_screenshot` after `browser_open` + `browser_fetch` || Read the current page or one element as a snapshot, HTML, Markdown, or text | `browser_extract` with `session_id` |@@ -453,7 +471,7 @@ The MCP server name when registering with a client is `ScraplingServer`. The com## Connecting to remote browsers-`browser_open` doesn't have to launch a browser locally. Pass a `cdp_url` and it connects to an already-running browser through the Chrome DevTools Protocol, whether that browser is on the same machine, another host, or a managed browser provider. The `session_id` you get back is used with `browser_fetch`, `browser_extract`, `browser_actions`, `browser_evaluate`, `browser_screenshot`, `browser_network_requests`, and `browser_network_request` as usual.+`browser_open` doesn't have to launch a browser locally. Pass a `cdp_url` and it connects to an already-running browser through the Chrome DevTools Protocol, whether that browser is on the same machine, another host, or a managed browser provider. The `session_id` you get back is used with `browser_fetch`, `browser_extract`, `browser_actions`, `browser_evaluate`, `browser_screenshot`, `browser_network_requests`, `browser_network_request`, `browser_save_state`, and `browser_load_state` as usual.The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser started with `--remote-debugging-port=9222`, reached as `cdp_url="http://localhost:9222"`.diff --git a/docs/ai/mcp-server.md b/docs/ai/mcp-server.mdindex 5410ed5..f2828c7 100644--- a/docs/ai/mcp-server.md+++ b/docs/ai/mcp-server.md@@ -6,7 +6,7 @@ The **Scrapling MCP Server** is a new feature that brings Scrapling's powerful W## Features-The Scrapling MCP Server provides fourteen tools for web scraping and browser interaction. One-shot tools fetch a single URL and close their own browser or client. Session tools keep cookies and settings across calls, so the AI can continue browser actions on the same tab and inspect network history.+The Scrapling MCP Server provides sixteen tools for web scraping and browser interaction. One-shot tools fetch a single URL and close their own browser or client. Session tools keep cookies and settings across calls, so the AI can continue browser actions on the same tab and inspect network history.### One-shot tools@@ -24,6 +24,13 @@ The Scrapling MCP Server provides fourteen tools for web scraping and browser in- **`close_session`**: Close a persistent session (browser or requests) and free its resources.- **`list_sessions`**: List all active sessions with their details and `settings`.+#### Browser State++- **`browser_save_state`**: Save cookies, local storage and IndexedDB to a Playwright-compatible JSON file, so login state can be reused after the browser session ends. Saving replaces an existing file.+- **`browser_load_state`**: Restore a saved file into an open browser session, including before its first page load. This replaces the current stored state without closing or navigating existing tabs. Fetch or reload the page to use the restored login.++State files are stored on the machine running the MCP server. They do not include `sessionStorage`, open tabs or browser settings.+#### 🎯 Fetching Through a Session- **`browser_fetch`**: Render JavaScript and scrape a single URL through an open stealthy browser session, with fingerprint spoofing and Cloudflare Turnstile/Interstitial bypass. It can also capture a structured page snapshot with element references, positions, and sizes, helping the AI identify controls and plan mouse actions.- **`session_make_request`**: Make fast HTTP requests with any method (GET, POST, PUT, DELETE), browser fingerprint impersonation, matching browser headers, and HTTP/3 support. Reuse cookies, connections, and the browser fingerprint through a session opened with `open_request_session`.@@ -217,7 +224,7 @@ You can also set the `SCRAPLING_EXECUTABLE_PATH` environment variable before sta```Open a stealthy browser session on wss://cdp.provider.example/session/abc123, then use it to scrape the product details from https://shop.example.com. Close the session when you're done.```-The `session_id` you get back is used with `browser_fetch`, `browser_extract`, `browser_actions`, `browser_evaluate`, `browser_screenshot`, `browser_network_requests`, and `browser_network_request` as usual.+The `session_id` you get back is used with `browser_fetch`, `browser_extract`, `browser_actions`, `browser_evaluate`, `browser_screenshot`, `browser_network_requests`, `browser_network_request`, `browser_save_state`, and `browser_load_state` as usual.The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser you started yourself with the remote debugging port enabled:```commandlinediff --git a/scrapling/core/ai/server.py b/scrapling/core/ai/server.pyindex 2f68d99..792e43d 100644--- a/scrapling/core/ai/server.py+++ b/scrapling/core/ai/server.py@@ -210,7 +210,7 @@ def _mcp_tool(tool: Callable[..., Awaitable[Any]]) -> Callable[..., Awaitable[Anasync def wrapped(*args: Any, **kwargs: Any) -> Any:try:return await tool(*args, **kwargs)- except (ValueError, RuntimeError, TypeError, PatchrightError, CurlError) as error:+ except (ValueError, RuntimeError, TypeError, OSError, PatchrightError, CurlError) as error:if isinstance(error, TypeError) and not isinstance(error.__cause__, MsgspecValidationError):raiseraise ToolError(str(error)) from error@@ -327,6 +327,26 @@ class ScraplingMCPServer:await session.start()return self._register_session(session_id, session, "stealthy")+ async def browser_save_state(self, session_id: str, path: NonEmptyString) -> str:+ """Save browser cookies, local storage and IndexedDB as Playwright-compatible JSON. Excludes sessionStorage.++ :param session_id: ID from `browser_open`; no page fetch required.+ :param path: File on the MCP server host; relative to its working directory. Overwrites the file; parent must exist.+ """+ await self._get_session(session_id, ["stealthy"]).session.save_state(path)+ return "State saved."++ async def browser_load_state(self, session_id: str, path: NonEmptyString) -> str:+ """Replace browser cookies, local storage and IndexedDB from a Playwright-compatible JSON file.++ Leaves sessionStorage and open tabs unchanged. Fetch or reload pages to use the restored login.++ :param session_id: ID from `browser_open`; no page fetch required.+ :param path: File on the MCP server host; relative to its working directory.+ """+ await self._get_session(session_id, ["stealthy"]).session.load_state(path)+ return "State loaded."+async def open_request_session(self,session_id: Optional[str] = None,@@ -948,6 +968,18 @@ class ScraplingMCPServer:structured_output=True,annotations=_SESSION_TOOL_ANNOTATIONS,)+ server.add_tool(+ _mcp_tool(self.browser_save_state),+ title="Save browser state",+ structured_output=False,+ annotations=_INPUT_TOOL_ANNOTATIONS,+ )+ server.add_tool(+ _mcp_tool(self.browser_load_state),+ title="Load browser state",+ structured_output=False,+ annotations=_INPUT_TOOL_ANNOTATIONS,+ )server.add_tool(_mcp_tool(self.open_request_session),title="Open HTTP session",diff --git a/tests/ai/test_ai_mcp.py b/tests/ai/test_ai_mcp.pyindex 1f3926b..c3161da 100644--- a/tests/ai/test_ai_mcp.py+++ b/tests/ai/test_ai_mcp.py@@ -43,6 +43,8 @@ MCP_TOOLS = {"open_request_session","session_make_request","browser_open",+ "browser_save_state",+ "browser_load_state","browser_fetch","browser_actions","browser_evaluate",@@ -1300,6 +1302,8 @@ class TestServerToolRegistration:assert tools["browser_extract"].output_schema is Noneassert tools["browser_actions"].output_schema is Noneassert tools["browser_evaluate"].output_schema is None+ assert tools["browser_save_state"].output_schema is None+ assert tools["browser_load_state"].output_schema is Noneassert {"browser_type","browser_fill_form",@@ -1331,6 +1335,8 @@ class TestServerToolRegistration:"browser_extract","browser_actions","browser_evaluate",+ "browser_save_state",+ "browser_load_state",))@@ -1474,7 +1480,7 @@ class TestServerToolRegistration:assert annotations[name].open_world_hint is Trueassert annotations["list_sessions"].read_only_hint is Trueassert annotations["list_sessions"].open_world_hint is False- for name in ("browser_actions", "browser_evaluate"):+ for name in ("browser_actions", "browser_evaluate", "browser_save_state", "browser_load_state"):assert annotations[name].read_only_hint is Falseassert annotations[name].destructive_hint is Trueassert annotations[name].idempotent_hint is Falsediff --git a/tests/ai/test_browser_state.py b/tests/ai/test_browser_state.pynew file mode 100644index 0000000..7d85d1e--- /dev/null+++ b/tests/ai/test_browser_state.py@@ -0,0 +1,141 @@+from contextlib import asynccontextmanager+from errno import EISDIR, ENOENT+from json import loads+from os import getenv, strerror+from pathlib import Path+from types import SimpleNamespace+from unittest.mock import AsyncMock++import pytest+from mcp.client import Client+from mcp.types import CallToolResult, TextContent++from scrapling.core._types import Any, AsyncGenerator+from scrapling.core.ai import ScraplingMCPServer+from scrapling.core.ai.server import _SessionEntry+from tests.fetchers.test_session_state import HTML, READ_STATE, SAVED_STATE, URL, WRITE_STATE, _check_json+++def _status(result: CallToolResult, text: str) -> None:+ assert not result.is_error and result.structured_content is None+ assert result.content == [TextContent(type="text", text=text)]+++@asynccontextmanager+async def _browser() -> AsyncGenerator[tuple[Client, Any], None]:+ server = ScraplingMCPServer(executable_path=getenv("SCRAPLING_EXECUTABLE_PATH"))+ async with Client(server._build_server("127.0.0.1", 8000)) as client:+ opened = await client.call_tool("browser_open", {"session_id": "browser"})+ assert not opened.is_error+ try:+ session = server._sessions["browser"].session+ await session.context.route("**/*", lambda route: route.fulfill(body=HTML, content_type="text/html"))+ yield client, session+ finally:+ assert not (await client.call_tool("close_session", {"session_id": "browser"})).is_error++[email protected]("operation", ["save", "load"])+async def test_browser_state_schema_and_empty_path(operation: str) -> None:+ server = ScraplingMCPServer()+ session = SimpleNamespace(_is_alive=True, save_state=AsyncMock(), load_state=AsyncMock())+ server._sessions["browser"] = _SessionEntry(session, "stealthy")+ name = f"browser_{operation}_state"+ async with Client(server._build_server("127.0.0.1", 8000)) as client:+ tool = next(tool for tool in (await client.list_tools()).tools if tool.name == name)+ assert tool.title == f"{operation.title()} browser state"+ assert tool.input_schema["required"] == ["session_id", "path"]+ properties = tool.input_schema["properties"]+ assert set(properties) == {"session_id", "path"}+ assert properties["path"]["type"] == "string" and properties["path"]["minLength"] == 1+ assert tool.output_schema is None+ assert tool.annotations is not None+ assert tool.annotations.read_only_hint is False+ assert tool.annotations.destructive_hint is True+ assert tool.annotations.idempotent_hint is False+ assert tool.annotations.open_world_hint is True+ result = await client.call_tool(name, {"session_id": "browser", "path": ""})+ assert result.is_error and result.structured_content is None+ assert isinstance(result.content[0], TextContent) and "path" in result.content[0].text+ session.save_state.assert_not_awaited()+ session.load_state.assert_not_awaited()++[email protected]("operation", ["save", "load"])+ "kind, message",+ [("unknown", "not found"), ("dead", "no longer alive"), ("static", "requires a 'stealthy' session")],+)+async def test_browser_state_requires_live_browser(operation: str, kind: str, message: str, tmp_path: Path) -> None:+ server = ScraplingMCPServer()+ session = SimpleNamespace(_is_alive=kind != "dead", save_state=AsyncMock(), load_state=AsyncMock())+ if kind != "unknown":+ server._sessions["browser"] = _SessionEntry(session, "static" if kind == "static" else "stealthy")+ path = tmp_path / "state.json"+ async with Client(server._build_server("127.0.0.1", 8000)) as client:+ result = await client.call_tool(f"browser_{operation}_state", {"session_id": "browser", "path": str(path)})+ assert result.is_error and result.structured_content is None+ assert isinstance(result.content[0], TextContent) and message in result.content[0].text+ assert not path.exists()+ session.save_state.assert_not_awaited()+ session.load_state.assert_not_awaited()+++async def test_browser_state_live_round_trip(tmp_path: Path) -> None:+ path, native_path = tmp_path / "state.json", tmp_path / "native.json"+ arguments = {"session_id": "browser", "path": str(path)}+ fetch = {"session_id": "browser", "url": URL, "google_search": False}+ async with _browser() as (client, source):+ assert source.page_pool.pages_count == 0+ _status(await client.call_tool("browser_save_state", arguments), "State saved.")+ assert loads(path.read_text())["cookies"] == loads(path.read_text())["origins"] == []+ assert source.page_pool.pages_count == 0+ assert not (await client.call_tool("browser_fetch", fetch)).is_error+ await source.page_pool.pages[0].page.evaluate(WRITE_STATE, "saved")+ _status(await client.call_tool("browser_save_state", arguments), "State saved.")+ _check_json(path, await source.context.storage_state(path=native_path, indexed_db=True))+ async with _browser() as (client, restored):+ assert restored.page_pool.pages_count == 0+ _status(await client.call_tool("browser_load_state", arguments), "State loaded.")+ assert restored.page_pool.pages_count == 0+ assert not (await client.call_tool("browser_fetch", fetch)).is_error+ page_info = restored.page_pool.pages[0]+ page = page_info.page+ assert await page.evaluate(READ_STATE) == SAVED_STATE+ assert await page.evaluate("sessionStorage.length") == 0+ await page.evaluate(WRITE_STATE, "changed")+ changed = await page.evaluate(READ_STATE)+ assert changed["indexedDB"]["value"] == "changed" and changed["databases"] == ["auth", "stale"]+ _status(await client.call_tool("browser_load_state", {**arguments, "path": str(native_path)}), "State loaded.")+ assert restored.page_pool.pages == [page_info] and page_info.state == "ready"+ assert page.url == URL and not page.is_closed()+ assert await page.evaluate("sessionStorage.getItem('temporary')") == "changed"+ assert not (await client.call_tool("browser_fetch", fetch)).is_error+ assert restored.page_pool.pages[0].page is page+ assert await page.evaluate(READ_STATE) == SAVED_STATE+++async def test_browser_state_live_file_errors(tmp_path: Path) -> None:+ invalid = tmp_path / "invalid.json"+ invalid.write_text("{")+ cases = [+ ("load", tmp_path / "missing.json", strerror(ENOENT)),+ ("load", invalid, "Expecting property name enclosed in double quotes"),+ ("save", tmp_path / "missing" / "state.json", strerror(ENOENT)),+ ("save", tmp_path, strerror(EISDIR)),+ ]+ async with _browser() as (client, session):+ await session.context.add_cookies([{"name": "kept", "value": "private-token", "url": URL}])+ for operation, path, message in cases:+ result = await client.call_tool(f"browser_{operation}_state", {"session_id": "browser", "path": str(path)})+ assert result.is_error and result.structured_content is None+ assert len(result.content) == 1 and isinstance(result.content[0], TextContent)+ assert message in result.content[0].text and "private-token" not in result.content[0].text+ assert session.page_pool.pages_count == 0+ assert [(cookie["name"], cookie["value"]) for cookie in await session.context.cookies()] == [+ ("kept", "private-token")+ ]diff --git a/tests/ai/test_tool_errors.py b/tests/ai/test_tool_errors.pyindex 68ae65d..7c99401 100644--- a/tests/ai/test_tool_errors.py+++ b/tests/ai/test_tool_errors.py@@ -22,6 +22,9 @@ from scrapling.core._types import AnyRuntimeError("page busy"),PatchrightError("page closed"),CurlError("connection failed"),+ FileNotFoundError("state file missing"),+ PermissionError("state file denied"),+ IsADirectoryError("state path is a directory"),],)async def test_expected_tool_errors_keep_their_message(error: Exception) -> None: