🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
feat(mcp): add two tools to open requests sessions and use them
Karim shoair6e6373c99761
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zipindex 9af98ac..e9a2857 100644Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differdiff --git a/agent-skill/Scrapling-Skill/references/mcp-server.md b/agent-skill/Scrapling-Skill/references/mcp-server.mdindex 33fe2f1..fced8cb 100644--- a/agent-skill/Scrapling-Skill/references/mcp-server.md+++ b/agent-skill/Scrapling-Skill/references/mcp-server.md@@ -1,6 +1,6 @@# Scrapling MCP Server-The Scrapling MCP server exposes eleven tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), three levels of scraping capability (plain HTTP, browser-rendered, and stealth/anti-bot bypass), persistent browser session management, and page screenshots returned as real image content blocks. Fetch tools come in two modes: one-shot tools (`fetch`, `bulk_fetch`, `stealthy_fetch`, `bulk_stealthy_fetch`) each launch and close their own browser, while `session_fetch` fetches through a session opened with `open_session`.+The Scrapling MCP server exposes thirteen tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), three levels of scraping capability (plain HTTP, browser-rendered, and stealth/anti-bot bypass), persistent browser session management, and page screenshots returned as real image content blocks. Fetch tools come in two modes: one-shot tools (`fetch`, `bulk_fetch`, `stealthy_fetch`, `bulk_stealthy_fetch`) each launch and close their own browser, while `session_fetch` and `session_make_request` work through sessions opened with `open_session`/`open_request_session`.All scraping tools return a `ResponseModel` with fields: `status` (int), `content` (list of strings), `url` (str). The `screenshot` tool returns a list of MCP content blocks: an `ImageContent` (the screenshot bytes) followed by a `TextContent` (the post-redirect URL).@@ -100,7 +100,7 @@ Concurrent stealth version. Same parameters as `stealthy_fetch` except `url` is### `open_session` -- Create a persistent browser session-Opens a browser session that stays alive across multiple `session_fetch` calls, avoiding the overhead of launching a new browser each time. It holds the browser-level configuration only; per-request options are passed to `session_fetch`. Returns a `SessionCreatedModel` with `session_id`, `session_type`, `created_at`, `is_alive`, `settings` (the session's effective configuration for the AI agent; empty for CDP sessions), and `message`.+Opens a browser session that stays alive across multiple `session_fetch` calls, avoiding the overhead of launching a new browser each time. It holds the browser-level configuration only; per-request options are passed to `session_fetch`. For plain HTTP requests without a browser, use `open_request_session` instead. Returns a `SessionCreatedModel` with `session_id`, `session_type`, `created_at`, `is_alive`, `settings` (the session's effective configuration for the AI agent; empty for CDP sessions), and `message`.**Key parameters:**@@ -115,11 +115,21 @@ Opens a browser session that stays alive across multiple `session_fetch` calls,Plus the other browser-level session parameters (`proxy`, `real_chrome`, `cdp_url`, `locale`, `timezone_id`, `useragent`, `cookies`, `executable_path`, `additional_args`). Per-request options (`timeout`, `wait`, `google_search`, `network_idle`, `disable_resources`, `wait_selector`, `wait_selector_state`, `extra_headers`, `solve_cloudflare`) are not set here; pass them to `session_fetch`.-One `session_fetch` works with either session type; `solve_cloudflare` only applies to a stealthy session.+One `session_fetch` works with either browser session type; `solve_cloudflare` only applies to a stealthy session.-### `session_fetch` -- Fetch through an open session (single URL)+### `open_request_session` -- Create a persistent HTTP requests session-Fetches one URL through a session opened with `open_session` (dynamic or stealthy). The session holds the browser-level configuration; every parameter here applies to this request only.+Opens an HTTP session (no browser) that stays alive across multiple `session_make_request` calls, keeping cookies, connections, and the browser fingerprint between requests. Returns the same `SessionCreatedModel` receipt and shows in `list_sessions` as a `static` session.++| Parameter | Type | Default | Description |+|---------------|-------------|------------|-------------------------------------------------------------------------------------------------------|+| `session_id` | str or null | null | Custom ID for the session. If omitted, a random 12-char hex ID is generated. Raises if already in use |+| `impersonate` | str | `"chrome"` | Browser fingerprint to impersonate on every request |+| `proxy` | str or null | null | Proxy URL used for every request, e.g. `"http://user:pass@host:port"` |++### `session_fetch` -- Fetch through an open browser session (single URL)++Fetches one URL through a browser session opened with `open_session` (dynamic or stealthy). The session holds the browser-level configuration; every parameter here applies to this request only. Raises on a requests session; use `session_make_request` there instead.| Parameter | Type | Default | Description ||-----------------------|---------------------|--------------|---------------------------------------------------------------------------------|@@ -140,9 +150,13 @@ Fetches one URL through a session opened with `open_session` (dynamic or stealth| `blocked_domains` | list or null | null | Domain names to block for this request (subdomains matched too) || `solve_cloudflare` | bool | false | (Stealthy sessions only) Auto-solve Cloudflare challenges; errors on a dynamic session |-### `close_session` -- Close a persistent browser session+### `session_make_request` -- HTTP request through an open requests session++Makes an HTTP request (any method) through a session opened with `open_request_session`, reusing its cookies, connections, and browser fingerprint across calls. Same parameters as `make_request` plus a required `session_id`, minus the session-level `impersonate`, `proxy`, and `proxy_auth`. Raises on a browser session.++### `close_session` -- Close a persistent session-Closes a session and frees its browser resources. Always close sessions when done.+Closes a session (browser or requests) and frees its resources. Always close sessions when done.| Parameter | Type | Default | Description ||--------------|------|----------|----------------------------------|@@ -186,6 +200,7 @@ Requires an open browser session. Call `open_session` first, then pass the `sess| Cloudflare or strong anti-bot protection | `stealthy_fetch` (with `solve_cloudflare=true` for Turnstile) || Multiple protected pages | `bulk_stealthy_fetch` || Multiple pages from the same site | `open_session` + `session_fetch` per page |+| Multiple plain HTTP requests to one site | `open_request_session` + `session_make_request` per request || Need a screenshot of a page | `open_session` + `screenshot` with `session_id` |Start with `make_request` (fastest, lowest resource cost). Escalate to `fetch` if content requires JS rendering. Escalate to `stealthy_fetch` only if blocked. For multiple pages from the same site, use a persistent session to avoid browser launch overhead.diff --git a/docs/ai/mcp-server.md b/docs/ai/mcp-server.mdindex 610d79b..aa83feb 100644--- a/docs/ai/mcp-server.md+++ b/docs/ai/mcp-server.md@@ -6,7 +6,7 @@ The **Scrapling MCP Server** is a new feature that brings Scrapling's powerful W## Features-The Scrapling MCP Server provides eleven powerful tools for web scraping, split into two modes: one-shot tools that each launch and close their own browser, and session tools that open a browser once and then work through it.+The Scrapling MCP Server provides thirteen powerful tools for web scraping, split into two modes: one-shot tools that each launch and close their own browser/client, and session tools that open a browser or an HTTP session once and then work through it.### One-shot tools@@ -26,11 +26,13 @@ The Scrapling MCP Server provides eleven powerful tools for web scraping, split#### 🔌 Session Management- **`open_session`**: Create a persistent browser session (dynamic or stealthy) that stays open across multiple `session_fetch` calls, avoiding the overhead of launching a new browser each time. It holds the browser-level configuration and returns the session's effective `settings` for the AI agent (empty for CDP sessions).-- **`close_session`**: Close a persistent browser session and free its resources.-- **`list_sessions`**: List all active browser sessions with their details and `settings`.+- **`open_request_session`**: Create a persistent HTTP requests session (no browser) used with `session_make_request`, keeping cookies, connections, and the browser fingerprint (`impersonate`) between requests. It returns the same `settings` receipt and shows in `list_sessions` as a `static` session.+- **`close_session`**: Close a persistent session (browser or requests) and free its resources.+- **`list_sessions`**: List all active sessions with their details and `settings`.#### 🎯 Fetching Through a Session-- **`session_fetch`**: Fetch a single URL through an open session (dynamic or stealthy), carrying the per-request options for that call. This is the session counterpart of `fetch`/`stealthy_fetch`.+- **`session_fetch`**: Fetch a single URL through an open browser session (dynamic or stealthy), carrying the per-request options for that call. This is the session counterpart of `fetch`/`stealthy_fetch`.+- **`session_make_request`**: Make an HTTP request with any method through a session opened with `open_request_session`, reusing its cookies, connections, and browser fingerprint. This is the session counterpart of `make_request`.#### 📸 Screenshots- **`screenshot`**: Capture a PNG or JPEG screenshot of a page using an open browser session, returned as an image content block the model can actually see (not a base64 string blob). Supports full-page captures, JPEG quality, and the usual readiness controls (`wait`, `wait_selector`, `network_idle`).@@ -198,7 +200,7 @@ You can also set the `SCRAPLING_EXECUTABLE_PATH` environment variable before sta```Open a stealthy browser session on wss://cdp.provider.example/session/abc123, then use it to scrape the product details from https://shop.example.com. Close the session when you're done.```-Both session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with `session_fetch` and `screenshot` as usual.+Both browser session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with `session_fetch` and `screenshot` as usual.The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser you started yourself with the remote debugging port enabled:```commandline@@ -436,12 +438,13 @@ This protection runs automatically on all MCP tool responses. Keep `main_content### 6. Use Sessions for Multiple Requests- Use `open_session` to create a persistent browser session when scraping multiple pages, then call `session_fetch` for each page through that session-- `open_session` holds the browser-level configuration (headless, locale, cookies, stealth toggles, etc.); the per-request options (timeout, wait_selector, network_idle, solve_cloudflare, etc.) are passed to `session_fetch` on each call, with their defaults shown in the tool schema-- One `session_fetch` works with both session types; `solve_cloudflare` only applies to a stealthy session and raises a clear error on a dynamic one-- The one-shot tools (`fetch`, `bulk_fetch`, `stealthy_fetch`, `bulk_stealthy_fetch`) never take a session; they always launch and close their own browser+- For multiple plain HTTP requests, use `open_request_session` instead and call `session_make_request` per request; it keeps cookies, connections, and the browser fingerprint (`impersonate`) across calls without a browser+- Sessions hold the session-level configuration set when opened (headless, locale, cookies, stealth toggles, etc. for browsers; `impersonate` and `proxy` for requests sessions); the per-request options (timeout, wait_selector, network_idle, solve_cloudflare, etc.) are passed to `session_fetch`/`session_make_request` on each call, with their defaults shown in the tool schemas+- One `session_fetch` works with both browser session types; `solve_cloudflare` only applies to a stealthy session and raises a clear error on a dynamic one+- The one-shot tools (`make_request`, `bulk_get`, `fetch`, `bulk_fetch`, `stealthy_fetch`, `bulk_stealthy_fetch`) never take a session- Always close sessions with `close_session` when done to free resources- Use `list_sessions` to check which sessions are still active and see the `settings` each was created with (returned for the AI agent; empty for CDP sessions)-- Pass a custom `session_id` to `open_session` to give sessions meaningful names (e.g. `"search"`, `"checkout"`) instead of the random hex default. `open_session` raises if the chosen ID is already in use, so you can detect collisions up front+- Pass a custom `session_id` to the open tools to give sessions meaningful names (e.g. `"search"`, `"checkout"`) instead of the random hex default. They raise if the chosen ID is already in use, so you can detect collisions up front### 7. Capturing Screenshots- `screenshot` only works through an existing browser session, so call `open_session` first (either `dynamic` or `stealthy` works)diff --git a/scrapling/core/ai.py b/scrapling/core/ai.pyindex ca36c8a..f986108 100644--- a/scrapling/core/ai.py+++ b/scrapling/core/ai.py@@ -43,7 +43,8 @@ from scrapling.core._types import (SUPPORTED_HTTP_METHODS,)-SessionType = Literal["dynamic", "stealthy"]+SessionType = Literal["dynamic", "stealthy", "static"]+BrowserSessionType = Literal["dynamic", "stealthy"]ScreenshotType = Literal["png", "jpeg"]MCP_EXECUTABLE_PATH_ENV = "SCRAPLING_EXECUTABLE_PATH"MCP_AUTH_TOKEN_ENV = "SCRAPLING_MCP_AUTH_TOKEN" # nosec B105 - the name of the variable, not a token@@ -66,8 +67,18 @@ _PLAYWRIGHT_FETCH_KEYS = _typed_dict_keys(PlaywrightFetchParams) - _EXCLUDED_FET_STEALTH_FETCH_KEYS = _typed_dict_keys(StealthFetchParams) - _EXCLUDED_FETCH_KEYS-def _session_settings(config: Any) -> Dict[str, Any]:- """Extract the JSON-safe effective settings of a validated session config struct, for the AI agent."""+def _session_settings(session: Any) -> Dict[str, Any]:+ """Extract the JSON-safe effective settings of a session, for the AI agent."""+ if isinstance(session, FetcherSession):+ fields = {"stealthy_headers": session._stealth} | {+ f.removeprefix("_default_"): getattr(session, f)+ for f in FetcherSession.__slots__+ if f.startswith("_default")+ }+ return {+ name: value for name, value in fields.items() if isinstance(value, (str, int, float, bool)) or value is None+ }+ config = session._configif config.cdp_url:return {}return {@@ -94,7 +105,7 @@ class SessionInfo(BaseModel):"""Information about an open browser session."""session_id: str = Field(description="The unique identifier of the session.")- session_type: SessionType = Field(description="The type of the session: 'dynamic' or 'stealthy'.")+ session_type: SessionType = Field(description="The type of the session: 'dynamic', 'stealthy', or 'static'.")created_at: str = Field(description="ISO timestamp of when the session was created.")is_alive: bool = Field(description="Whether the session is still alive and usable.")settings: Dict[str, Any] = Field(@@ -118,7 +129,7 @@ class SessionClosedModel(BaseModel):@dataclassclass _SessionEntry:- session: Any # AsyncDynamicSession | AsyncStealthySession+ session: Any # AsyncDynamicSession | AsyncStealthySession | FetcherSessionsession_type: SessionTypecreated_at: str = field(default_factory=lambda: datetime.now(timezone.utc).isoformat())@@ -200,9 +211,31 @@ class ScraplingMCPServer:)return entry+ def _new_session_id(self, session_id: Optional[str]) -> str:+ """Generate a session ID when none is given, and reject duplicates."""+ session_id = session_id or uuid4().hex[:12]+ if session_id in self._sessions:+ raise ValueError(+ f"Session '{session_id}' already exists. Use a different ID or close the existing session first."+ )+ return session_id++ def _register_session(self, session_id: str, session: Any, session_type: SessionType) -> SessionCreatedModel:+ """Store a started session and build its creation receipt."""+ entry = _SessionEntry(session=session, session_type=session_type)+ self._sessions[session_id] = entry+ return SessionCreatedModel(+ session_id=session_id,+ session_type=session_type,+ created_at=entry.created_at,+ is_alive=True,+ settings=_session_settings(session),+ message=f"Session '{session_id}' ({session_type}) created successfully.",+ )+async def open_session(self,- session_type: SessionType,+ session_type: BrowserSessionType,session_id: Optional[str] = None,headless: bool = True,real_chrome: bool = False,@@ -239,12 +272,7 @@ class ScraplingMCPServer::param allow_webgl: (Stealthy only) Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.:param additional_args: (Stealthy only) Additional arguments to be passed to Playwright's context as additional settings."""- session_id = session_id or uuid4().hex[:12]- if session_id in self._sessions:- raise ValueError(- f"Session '{session_id}' already exists. Use a different ID or close the existing session first."- )-+ session_id = self._new_session_id(session_id)common_kwargs: Dict[str, Any] = dict(proxy=proxy,locale=locale,@@ -271,24 +299,33 @@ class ScraplingMCPServer:session = AsyncDynamicSession(**common_kwargs)await session.start()+ return self._register_session(session_id, session, session_type)- entry = _SessionEntry(session=session, session_type=session_type)- self._sessions[session_id] = entry+ async def open_request_session(+ self,+ session_id: Optional[str] = None,+ impersonate: ImpersonateType = "chrome",+ proxy: Optional[str] = None,+ ) -> SessionCreatedModel:+ """Open a persistent HTTP requests session (no browser) that can be reused across multiple+ `session_make_request` calls, keeping cookies, connections, and the browser fingerprint between requests.+ The session holds this configuration only; per-request options are passed to `session_make_request` on each call.+ It shares `close_session`/`list_sessions` with the browser sessions and shows there as a 'static' session.- return SessionCreatedModel(- session_id=session_id,- session_type=session_type,- created_at=entry.created_at,- is_alive=True,- settings=_session_settings(session._config),- message=f"Session '{session_id}' ({session_type}) created successfully.",- )+ :param session_id: Optional custom session ID. If not provided, a random 12-character hex ID will be generated. Useful for naming sessions for easier management.+ :param impersonate: Browser version to impersonate its fingerprint on every request. It's using the latest chrome version by default.+ :param proxy: The proxy URL used for every request in this session. Format: "http://username:password@localhost:8030".+ """+ session_id = self._new_session_id(session_id)+ session = FetcherSession(impersonate=impersonate, proxy=proxy)+ await session.__aenter__()+ return self._register_session(session_id, session, "static")async def close_session(self,session_id: str,) -> SessionClosedModel:- """Close a persistent browser session and free its resources.+ """Close a persistent session and free its resources.:param session_id: The unique identifier of the session to close. Use list_sessions to see active sessions."""@@ -296,21 +333,24 @@ class ScraplingMCPServer:if entry is None:raise ValueError(f"Session '{session_id}' not found. Use list_sessions to see active sessions.")- await entry.session.close()+ if entry.session_type == "static":+ await entry.session.__aexit__(None, None, None)+ else:+ await entry.session.close()return SessionClosedModel(session_id=session_id,message=f"Session '{session_id}' closed successfully.",)async def list_sessions(self) -> List[SessionInfo]:- """List all active browser sessions with their details, including the effective settings each one was created with."""+ """List all active sessions with their details, including the effective settings each one was created with."""return [SessionInfo(session_id=sid,session_type=entry.session_type,created_at=entry.created_at,is_alive=entry.session._is_alive,- settings=_session_settings(entry.session._config),+ settings=_session_settings(entry.session),)for sid, entry in self._sessions.items()]@@ -346,6 +386,11 @@ class ScraplingMCPServer:raise ValueError("'quality' is only valid when 'image_type' is 'jpeg'.")entry = self._get_session(session_id, expected_type=None)+ if entry.session_type == "static":+ raise ValueError(+ f"Session '{session_id}' is a 'static' session, so it can't take screenshots. "+ f"Open a 'dynamic' or 'stealthy' session for that."+ )screenshot_kwargs: Dict[str, Any] = {"type": image_type, "full_page": full_page}if quality is not None:@@ -910,6 +955,10 @@ class ScraplingMCPServer::param solve_cloudflare: (Stealthy sessions only) Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response."""entry = self._get_session(session_id, expected_type=None)+ if entry.session_type == "static":+ raise ValueError(+ f"Session '{session_id}' is a 'static' session. Use `session_make_request` with it instead."+ )if solve_cloudflare and entry.session_type != "stealthy":raise ValueError(f"Session '{session_id}' is a '{entry.session_type}' session, so it can't solve Cloudflare "@@ -935,6 +984,77 @@ class ScraplingMCPServer:)return _translate_response(page, extraction_type, css_selector, main_content_only)+ async def session_make_request(+ self,+ url: str,+ session_id: str,+ method: SUPPORTED_HTTP_METHODS = "GET",+ extraction_type: extraction_types = "markdown",+ css_selector: Optional[str] = None,+ main_content_only: bool = True,+ params: Optional[Dict] = None,+ data: Optional[Dict[str, str] | str] = None,+ json: Optional[Dict | List] = None,+ headers: Optional[Mapping[str, Optional[str]]] = None,+ cookies: Optional[Dict[str, str]] = None,+ timeout: Optional[int | float] = 30,+ follow_redirects: FollowRedirects = "safe",+ max_redirects: int = 30,+ retries: Optional[int] = 3,+ retry_delay: Optional[int] = 1,+ auth: Optional[Dict[str, str]] = None,+ verify: Optional[bool] = True,+ http3: Optional[bool] = False,+ stealthy_headers: Optional[bool] = True,+ ) -> ResponseModel:+ """Make an HTTP request with any method (GET, POST, PUT, DELETE) through a requests session previously opened+ with `open_request_session`, keeping its cookies, connections, and browser fingerprint across calls. The session+ holds the impersonation and proxy configuration; every option here applies to this request only, with the defaults shown.++ :param url: The URL to request.+ :param session_id: ID of an open requests session created with `open_request_session`.+ :param method: The HTTP method to use: "GET" (default), "POST", "PUT", or "DELETE".+ :param extraction_type: The type of content to extract from the page: "markdown", "html", or "text".+ :param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page.+ :param main_content_only: Whether to extract only the main content of the page. The main content here is the data inside the `<body>` tag.+ :param params: Query string parameters for the request.+ :param data: Form data for the request body. Used with "POST", "PUT", and "DELETE" only.+ :param json: A JSON-serializable object for the request body. Used with "POST", "PUT", and "DELETE" only.+ :param headers: Headers to include in the request.+ :param cookies: Cookies to use in the request.+ :param timeout: Number of seconds to wait before timing out.+ :param follow_redirects: Whether to follow redirects. Defaults to "safe", which follows redirects but rejects those targeting internal/private IPs (SSRF protection).+ Pass True to follow all redirects without restriction.+ :param max_redirects: Maximum number of redirects. Use -1 for unlimited.+ :param retries: Number of retry attempts.+ :param retry_delay: Number of seconds to wait between retry attempts.+ :param auth: HTTP basic auth in dictionary format with `username` and `password` keys.+ :param verify: Whether to verify HTTPS certificates.+ :param http3: Whether to use HTTP3. It might be problematic if used it with `impersonate`.+ :param stealthy_headers: If enabled (default), it creates and adds real browser headers. It also sets a Google referer header.+ """+ entry = self._get_session(session_id, expected_type="static")++ request_kwargs: Dict[str, Any] = dict(+ auth=_normalize_credentials(auth),+ http3=http3,+ verify=verify,+ params=params,+ headers=headers,+ cookies=cookies,+ timeout=timeout,+ retries=retries,+ retry_delay=retry_delay,+ max_redirects=max_redirects,+ follow_redirects=follow_redirects,+ stealthy_headers=stealthy_headers,+ )+ if method != "GET":+ request_kwargs.update(data=data, json=json)++ page = await getattr(entry.session._client, method.lower())(url, **request_kwargs)+ return _translate_response(page, extraction_type, css_selector, main_content_only)+@staticmethoddef _transport_security(allowed_hosts: Sequence[str]) -> Optional[TransportSecuritySettings]:"""Build the DNS-rebinding protection settings for the streamable-http transport."""@@ -960,15 +1080,16 @@ class ScraplingMCPServer:],"cache_hints": {"tools/list": CacheHint(ttl_ms=3_600_000, scope="public")},"instructions": """Follow these instructions precisely:-1. When the `open_session` tool is used, make sure to close the session with `close_session` after you finish, and use `list_sessions` if you lose track of the open sessions or their effective settings.+1. When the `open_session` or `open_request_session` tools are used, make sure to close the session with `close_session` after you finish, and use `list_sessions` if you lose track of the open sessions or their effective settings.2. If the user didn't specify which tool to use, start with the `make_request` tool (a plain HTTP request, defaulting to GET; set `method` for POST/PUT/DELETE), then escalate. The `make_request` tool and `bulk_get` (its GET-only bulk version) are suitable only for low-to-mid protection levels.For high-protection levels or websites that require JS loading, use the other tools directly.3. For all tools, if the `css_selector` resolves to more than one element, all the elements will be returned.4. For all fetch tools, the `extraction_type` parameter controls the format of the returned content: "markdown" (default) converts the page content to Markdown, "html" returns the raw HTML, and "text" returns the text content of the page.5. For all fetch tools, `main_content_only` is enabled by default and returns only the content inside the page's `<body>` tag. Pass `main_content_only=False` when you need the full page instead.-6. If the task consists of multiple sequential requests to the same website, open a session with `open_session` once, then call `session_fetch` per page to be more efficient.-7. Sessions hold the browser-level configuration set at `open_session`, while `session_fetch` carries the per-request options and applies them on each call with the defaults shown in its schema.- The one-shot tools (`fetch`, `bulk_fetch`, `stealthy_fetch`, `bulk_stealthy_fetch`) never touch sessions and always launch their own browser.+6. If the task consists of multiple sequential requests to the same website, open a session once, then fetch through it to be more efficient:+ `open_session` + `session_fetch` per page for browsers, or `open_request_session` + `session_make_request` per request for plain HTTP.+7. Sessions hold the session-level configuration set when opened, while `session_fetch`/`session_make_request` carry the per-request options and apply them on each call with the defaults shown in their schemas.+ The one-shot tools (`make_request`, `bulk_get`, `fetch`, `bulk_fetch`, `stealthy_fetch`, `bulk_stealthy_fetch`) never touch sessions.8. If you are making multiple parallel one-shot requests, use the bulk version of the tool to be more efficient.9. If you are crawling/browsing a website, be more efficient by using the `css_selector` parameter to only access the parts you are interested in and save money/time. Example: use the `a` selector to extract the urls right away.10. The user can pass a CDP URL to connect to a remote browser session through the `open_session` tool, then use it with the session tools.@@ -984,6 +1105,12 @@ class ScraplingMCPServer:server.add_tool(self.open_session, title="open_session", structured_output=True, annotations=_SESSION_TOOL_ANNOTATIONS)+ server.add_tool(+ self.open_request_session,+ title="open_request_session",+ structured_output=True,+ annotations=_SESSION_TOOL_ANNOTATIONS,+ )server.add_tool(self.close_session, title="close_session", structured_output=True, annotations=_SESSION_TOOL_ANNOTATIONS)@@ -1035,7 +1162,7 @@ class ScraplingMCPServer:structured_output=True,annotations=_FETCH_TOOL_ANNOTATIONS,)- # Session-scoped fetch tool+ # Session-scoped fetch toolsserver.add_tool(self.session_fetch,title="session_fetch",@@ -1043,6 +1170,13 @@ class ScraplingMCPServer:structured_output=True,annotations=_FETCH_TOOL_ANNOTATIONS,)+ server.add_tool(+ self.session_make_request,+ title="session_make_request",+ description=self.session_make_request.__doc__,+ structured_output=True,+ annotations=_FETCH_TOOL_ANNOTATIONS,+ )# Screenshot tool (returns image + url content blocks, not structured JSON)server.add_tool(self.screenshot,diff --git a/tests/ai/test_ai_mcp.py b/tests/ai/test_ai_mcp.pyindex 4542968..adc3c99 100644--- a/tests/ai/test_ai_mcp.py+++ b/tests/ai/test_ai_mcp.py@@ -30,7 +30,7 @@ from scrapling.core.ai import (_translate_response,)from scrapling.engines._browsers._validators import PlaywrightConfig, StealthConfig, models_default_values, validate-from scrapling.fetchers import AsyncDynamicSession, AsyncStealthySession+from scrapling.fetchers import AsyncDynamicSession, AsyncStealthySession, FetcherSessiondef test_translate_response_strips_control_characters():@@ -296,6 +296,85 @@ class TestSessionManagement:await server.close_session("dupe")+class TestStaticSessionManagement:+ """Test persistent requests (HTTP) session management"""++ @pytest.fixture+ def server(self):+ return ScraplingMCPServer()++ @pytest.mark.asyncio+ async def test_static_session_lifecycle(self, server, httpbin):+ """Open a requests session, make GET and POST requests through it, then close it"""+ created = await server.open_request_session(session_id="st")+ assert isinstance(created, SessionCreatedModel)+ assert created.session_type == "static"+ assert created.is_alive is True+ assert created.settings["impersonate"] == "chrome"++ response = await server.session_make_request(url=f"{httpbin.url}/html", session_id="st")+ assert isinstance(response, ResponseModel)+ assert response.status == 200++ posted = await server.session_make_request(+ url=f"{httpbin.url}/post", session_id="st", method="POST", json={"key": "value"}, extraction_type="text"+ )+ assert posted.status == 200++ listed = await server.list_sessions()+ assert listed[0].session_type == "static"+ assert listed[0].settings == created.settings++ session = server._sessions["st"].session+ closed = await server.close_session("st")+ assert closed.session_id == "st"+ assert session._is_alive is False++ @pytest.mark.asyncio+ async def test_static_session_keeps_cookies(self, server, httpbin):+ """Cookies set by one request are sent with the next request of the same session"""+ await server.open_request_session(session_id="jar")+ await server.session_make_request(+ url=f"{httpbin.url}/cookies/set/test/value",+ session_id="jar",+ follow_redirects=True,+ extraction_type="text",+ )+ response = await server.session_make_request(+ url=f"{httpbin.url}/cookies", session_id="jar", extraction_type="text"+ )+ assert "test" in "".join(response.content)+ await server.close_session("jar")++ @pytest.mark.asyncio+ async def test_session_ids_are_shared_across_both_open_tools(self, server, monkeypatch):+ """A requests session and a browser session can't share the same ID"""+ monkeypatch.setattr("scrapling.core.ai.AsyncDynamicSession", _FakeDynamicSession)+ await server.open_request_session(session_id="shared")+ with pytest.raises(ValueError, match="already exists"):+ await server.open_session(session_type="dynamic", session_id="shared")+ await server.close_session("shared")++ @pytest.mark.asyncio+ async def test_session_make_request_requires_a_static_session(self, server, monkeypatch):+ """session_make_request rejects browser sessions"""+ monkeypatch.setattr("scrapling.core.ai.AsyncDynamicSession", _FakeDynamicSession)+ await server.open_session(session_type="dynamic", session_id="browser")+ with pytest.raises(ValueError, match="requires a 'static' session"):+ await server.session_make_request(url="https://example.com", session_id="browser")+ await server.close_session("browser")++ @pytest.mark.asyncio+ async def test_session_fetch_and_screenshot_reject_static_sessions(self, server):+ """The browser session tools refuse a static session with a clear error"""+ await server.open_request_session(session_id="st2")+ with pytest.raises(ValueError, match="session_make_request"):+ await server.session_fetch(url="https://example.com", session_id="st2")+ with pytest.raises(ValueError, match="can't take screenshots"):+ await server.screenshot(url="https://example.com", session_id="st2")+ await server.close_session("st2")++class TestExecutablePath:"""Test custom browser executable path plumbing in the MCP browser tools"""@@ -547,8 +626,7 @@ class TestSessionSettingsReceipt:def test_session_settings_extracts_json_safe_fields(self):"""The helper keeps JSON primitives and drops the rest (callables, structs, sequences)"""- config = AsyncStealthySession(headless=True)._config- settings = _session_settings(config)+ settings = _session_settings(AsyncStealthySession(headless=True))assert settings["headless"] is Trueassert settings["timeout"] == 30000assert "cookies" not in settings, "non-primitive fields (list) must be dropped"@@ -556,8 +634,17 @@ class TestSessionSettingsReceipt:def test_cdp_session_reports_empty_settings(self):"""A CDP session drives a remote browser, so the local config is not reported as its settings"""- config = AsyncDynamicSession(cdp_url="ws://127.0.0.1:9222/devtools/browser/x")._config- assert _session_settings(config) == {}+ assert _session_settings(AsyncDynamicSession(cdp_url="ws://127.0.0.1:9222/devtools/browser/x")) == {}++ def test_static_session_settings_extracts_json_safe_fields(self):+ """A static session reports its HTTP defaults (impersonate, proxy, timeout, ...)"""+ settings = _session_settings(FetcherSession(impersonate="chrome", proxy=None))+ assert settings["impersonate"] == "chrome"+ assert settings["proxy"] is None+ assert settings["timeout"] == 30+ assert settings["stealthy_headers"] is True+ assert "headers" not in settings, "non-primitive fields (dict) must be dropped"+ assert all(isinstance(v, (str, int, float, bool)) or v is None for v in settings.values()), settings@pytest.mark.asyncioasync def test_open_session_and_list_report_the_receipt(self, monkeypatch):@@ -768,7 +855,7 @@ class TestMCPServerAuthentication:monkeypatch.delenv(MCP_AUTH_TOKEN_ENV, raising=False)built = ScraplingMCPServer(auth_token=SHARED_KEY)._build_server("0.0.0.0", 8000)- assert len(built._tool_manager.list_tools()) == 11+ assert len(built._tool_manager.list_tools()) == 13def test_http_without_a_token_refuses_to_serve(self, monkeypatch):"""The streamable-http transport requires authentication unless the caller explicitly opts out"""@@ -825,13 +912,13 @@ class TestServerToolRegistration:@pytest.mark.asyncioasync def test_tools_are_listed_with_expected_schemas(self):- """All 11 tools are advertised, and only the screenshot tool skips the structured output schema"""+ """All 13 tools are advertised, and only the screenshot tool skips the structured output schema"""server = ScraplingMCPServer()._build_server("127.0.0.1", 8000)async with Client(server) as client:assert client.instructionstools = {tool.name: tool for tool in (await client.list_tools()).tools}- assert len(tools) == 11+ assert len(tools) == 13assert tools["screenshot"].output_schema is Noneassert all(tool.output_schema is not None for name, tool in tools.items() if name != "screenshot")@@ -853,6 +940,12 @@ class TestServerToolRegistration:assert "data" in request_props and "json" in request_propsassert "method" not in tools["bulk_get"].input_schema["properties"]+ static_props = tools["session_make_request"].input_schema["properties"]+ assert static_props["method"]["default"] == "GET"+ assert "proxy" not in static_props and "impersonate" not in static_props, "session-level params leaked"+ assert set(tools["session_make_request"].input_schema["required"]) >= {"url", "session_id"}+ assert set(tools["open_request_session"].input_schema["properties"]) == {"session_id", "impersonate", "proxy"}+session_props = tools["session_fetch"].input_schema["properties"]assert session_props["timeout"]["default"] == 30000assert "solve_cloudflare" in session_props@@ -878,7 +971,7 @@ class TestServerToolRegistration:assert result.ttl_ms == 3_600_000 and result.cache_scope == "public"annotations = {tool.name: tool.annotations for tool in result.tools if tool.annotations is not None}- assert len(annotations) == 11+ assert len(annotations) == 13for name in ("make_request","bulk_get",@@ -887,11 +980,12 @@ class TestServerToolRegistration:"stealthy_fetch","bulk_stealthy_fetch","session_fetch",+ "session_make_request","screenshot",):assert annotations[name].read_only_hint is Trueassert annotations[name].open_world_hint is True- for name in ("open_session", "close_session"):+ for name in ("open_session", "open_request_session", "close_session"):assert annotations[name].read_only_hint is Falseassert annotations[name].destructive_hint is Falseassert annotations[name].open_world_hint is True