π·οΈ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
refactor(mcp)!: More reliable structure + new tools
Karim shoair73ad18be58af
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zipindex 6d86cc7..7fbc2c1 100644Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differdiff --git a/agent-skill/Scrapling-Skill/references/mcp-server.md b/agent-skill/Scrapling-Skill/references/mcp-server.mdindex 5794c89..37835dd 100644--- a/agent-skill/Scrapling-Skill/references/mcp-server.md+++ b/agent-skill/Scrapling-Skill/references/mcp-server.md@@ -1,10 +1,10 @@# Scrapling MCP Server-The Scrapling MCP server exposes ten tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), three levels of scraping capability (plain HTTP, browser-rendered, and stealth/anti-bot bypass), persistent browser session management, and page screenshots returned as real image content blocks.+The Scrapling MCP server exposes eleven tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), three levels of scraping capability (plain HTTP, browser-rendered, and stealth/anti-bot bypass), persistent browser session management, and page screenshots returned as real image content blocks. Fetch tools come in two modes: one-shot tools (`fetch`, `bulk_fetch`, `stealthy_fetch`, `bulk_stealthy_fetch`) each launch and close their own browser, while `session_fetch` fetches through a session opened with `open_session`.All scraping tools return a `ResponseModel` with fields: `status` (int), `content` (list of strings), `url` (str). The `screenshot` tool returns a list of MCP content blocks: an `ImageContent` (the screenshot bytes) followed by a `TextContent` (the post-redirect URL).-## Tools+## One-shot tools### `get` -- HTTP request (single URL)@@ -66,11 +66,12 @@ Opens a Chromium browser via Playwright to render JavaScript. Suitable for dynam| `cookies` | list or null | null | Playwright-format cookies || `timezone_id` | str or null | null | Browser timezone, e.g. `"America/New_York"` || `locale` | str or null | null | Browser locale, e.g. `"en-GB"` |-| `session_id` | str or null | null | Reuse a persistent session from `open_session` instead of creating a new browser |++This is a one-shot tool: it always launches its own browser. To fetch through a persistent session, use `session_fetch`.### `bulk_fetch` -- Browser fetch (multiple URLs)-Concurrent browser version of `fetch`. Same parameters (including `session_id`) except `url` is replaced by `urls` (list of strings). Each URL opens in a separate browser tab. Returns a list of `ResponseModel`.+Concurrent browser version of `fetch`. Same parameters except `url` is replaced by `urls` (list of strings). Each URL opens in a separate browser tab. Returns a list of `ResponseModel`.### `stealthy_fetch` -- Stealth browser fetch (single URL)@@ -85,17 +86,18 @@ Anti-bot bypass fetcher with fingerprint spoofing. Use this for sites with Cloud| `block_webrtc` | bool | false | Force WebRTC to respect proxy settings (prevents IP leak) || `allow_webgl` | bool | true | Keep WebGL enabled (disabling is detectable by WAFs) || `additional_args` | dict or null | null | Extra Playwright context args (overrides Scrapling defaults) |-| `session_id` | str or null | null | Reuse a persistent stealthy session from `open_session` |-All parameters from `fetch` are also accepted.+All parameters from `fetch` are also accepted. Like `fetch`, this is a one-shot tool that launches its own browser; use `session_fetch` for a stealthy session.### `bulk_stealthy_fetch` -- Stealth browser fetch (multiple URLs)-Concurrent stealth version. Same parameters (including `session_id`) as `stealthy_fetch` except `url` is replaced by `urls` (list of strings). Returns a list of `ResponseModel`.+Concurrent stealth version. Same parameters as `stealthy_fetch` except `url` is replaced by `urls` (list of strings). Returns a list of `ResponseModel`.++## Session tools### `open_session` -- Create a persistent browser session-Opens a browser session that stays alive across multiple fetch calls, avoiding the overhead of launching a new browser each time. Returns a `SessionCreatedModel` with `session_id`, `session_type`, `created_at`, `is_alive`, and `message`.+Opens a browser session that stays alive across multiple `session_fetch` calls, avoiding the overhead of launching a new browser each time. It holds the browser-level configuration only; per-request options are passed to `session_fetch`. Returns a `SessionCreatedModel` with `session_id`, `session_type`, `created_at`, `is_alive`, `settings` (the session's effective configuration for the AI agent; empty for CDP sessions), and `message`.**Key parameters:**@@ -104,17 +106,37 @@ Opens a browser session that stays alive across multiple fetch calls, avoiding t| `session_type` | `"dynamic"` / `"stealthy"` | required | Type of browser session to create || `session_id` | str or null | null | Custom ID for the session. If omitted, a random 12-char hex ID is generated. Raises if already in use || `headless` | bool | true | Run browser hidden or visible |-| `max_pages` | int | 5 | Max concurrent browser tabs (1-50) |-| `proxy` | str or dict or null | null | Proxy for all requests in this session |-| `timeout` | number | 30000 | Default timeout in ms |-| `solve_cloudflare` | bool | false | (Stealthy only) Auto-solve Cloudflare challenges || `hide_canvas` | bool | false | (Stealthy only) Canvas fingerprint noise || `block_webrtc` | bool | false | (Stealthy only) Block WebRTC IP leak || `allow_webgl` | bool | true | (Stealthy only) Keep WebGL enabled |-Plus all other browser session parameters (`google_search`, `real_chrome`, `cdp_url`, `locale`, `timezone_id`, `useragent`, `extra_headers`, `cookies`, `disable_resources`, `network_idle`, `wait_selector`, `wait_selector_state`).+Plus the other browser-level session parameters (`real_chrome`, `cdp_url`, `locale`, `timezone_id`, `useragent`, `cookies`, `executable_path`, `additional_args`). Per-request options (`timeout`, `wait`, `google_search`, `network_idle`, `disable_resources`, `wait_selector`, `wait_selector_state`, `extra_headers`, `proxy`, `solve_cloudflare`) are not set here; pass them to `session_fetch`.++One `session_fetch` works with either session type; `solve_cloudflare` only applies to a stealthy session.++### `session_fetch` -- Fetch through an open session (single URL)++Fetches one URL through a session opened with `open_session` (dynamic or stealthy). The session holds the browser-level configuration; every parameter here applies to this request only.-A dynamic session can only be used with `fetch`/`bulk_fetch`. A stealthy session can only be used with `stealthy_fetch`/`bulk_stealthy_fetch`.+| Parameter | Type | Default | Description |+|-----------------------|---------------------|--------------|---------------------------------------------------------------------------------|+| `url` | str | required | URL to fetch |+| `session_id` | str | required | ID of an open session created with `open_session` |+| `extraction_type` | str | `"markdown"` | `"markdown"` / `"html"` / `"text"` |+| `css_selector` | str or null | null | Narrow content before extraction |+| `main_content_only` | bool | true | Restrict to `<body>` |+| `wait` | number | 0 | Extra wait (ms) after page load before extraction |+| `timeout` | number | 30000 | Timeout in **milliseconds** |+| `google_search` | bool | true | Set a Google referer header |+| `network_idle` | bool | false | Wait until no network activity for 500ms |+| `load_dom` | bool | true | Wait for the page's JavaScript to fully load and execute |+| `disable_resources` | bool | false | Block fonts, images, media, stylesheets, etc. for speed |+| `wait_selector` | str or null | null | CSS selector to wait for before extraction |+| `wait_selector_state` | str | `"attached"` | State for wait_selector: `"attached"` / `"visible"` / `"hidden"` / `"detached"` |+| `extra_headers` | dict or null | null | Additional request headers |+| `blocked_domains` | list or null | null | Domain names to block for this request (subdomains matched too) |+| `proxy` | str or dict or null | null | Proxy for this request |+| `solve_cloudflare` | bool | false | (Stealthy sessions only) Auto-solve Cloudflare challenges; errors on a dynamic session |### `close_session` -- Close a persistent browser session@@ -128,7 +150,7 @@ Returns a `SessionClosedModel` with `session_id` and `message`.### `list_sessions` -- List active sessions-Returns a list of `SessionInfo` objects, each with `session_id`, `session_type`, `created_at`, and `is_alive`.+Returns a list of `SessionInfo` objects, each with `session_id`, `session_type`, `created_at`, `is_alive`, and `settings` (same as `open_session` returns).No parameters.@@ -161,7 +183,7 @@ Requires an open browser session. Call `open_session` first, then pass the `sess| Multiple JS-rendered pages | `bulk_fetch` || Cloudflare or strong anti-bot protection | `stealthy_fetch` (with `solve_cloudflare=true` for Turnstile) || Multiple protected pages | `bulk_stealthy_fetch` |-| Multiple pages from the same site | `open_session` + `fetch`/`stealthy_fetch` with `session_id` |+| Multiple pages from the same site | `open_session` + `session_fetch` per page || Need a screenshot of a page | `open_session` + `screenshot` with `session_id` |Start with `get` (fastest, lowest resource cost). Escalate to `fetch` if content requires JS rendering. Escalate to `stealthy_fetch` only if blocked. For multiple pages from the same site, use a persistent session to avoid browser launch overhead.@@ -253,7 +275,7 @@ The MCP server name when registering with a client is `ScraplingServer`. The com## Connecting to remote browsers-`open_session` doesn't have to launch a browser locally. Pass a `cdp_url` and it connects to an already-running browser through the Chrome DevTools Protocol, whether that browser is on the same machine, another host, or a managed browser provider. Both session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with the fetch and screenshot tools as usual.+`open_session` doesn't have to launch a browser locally. Pass a `cdp_url` and it connects to an already-running browser through the Chrome DevTools Protocol, whether that browser is on the same machine, another host, or a managed browser provider. Both session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with `session_fetch` and `screenshot` as usual.The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser started with `--remote-debugging-port=9222`, reached as `cdp_url="http://localhost:9222"`.diff --git a/docs/ai/mcp-server.md b/docs/ai/mcp-server.mdindex e85ed4b..5d474bc 100644--- a/docs/ai/mcp-server.md+++ b/docs/ai/mcp-server.md@@ -6,27 +6,34 @@ The **Scrapling MCP Server** is a new feature that brings Scrapling's powerful W## Features-The Scrapling MCP Server provides ten powerful tools for web scraping:+The Scrapling MCP Server provides eleven powerful tools for web scraping, split into two modes: one-shot tools that each launch and close their own browser, and session tools that open a browser once and then work through it.-### π Basic HTTP Scraping+### One-shot tools++#### π Basic HTTP Scraping- **`get`**: Fast HTTP requests with browser fingerprint impersonation, generating real browser headers matching the TLS version, HTTP/3, and more!- **`bulk_get`**: An async version of the above tool that allows scraping of multiple URLs at the same time!-### π Dynamic Content Scraping+#### π Dynamic Content Scraping- **`fetch`**: Rapidly fetch dynamic content with Chromium/Chrome browser with complete control over the request/browser, and more!- **`bulk_fetch`**: An async version of the above tool that allows scraping of multiple URLs in different browser tabs at the same time!-### π Stealth Scraping+#### π Stealth Scraping- **`stealthy_fetch`**: Uses our Stealthy browser to bypass Cloudflare Turnstile/Interstitial and other anti-bot systems with complete control over the request/browser!- **`bulk_stealthy_fetch`**: An async version of the above tool that allows stealth scraping of multiple URLs in different browser tabs at the same time!-### πΈ Screenshots-- **`screenshot`**: Capture a PNG or JPEG screenshot of a page using an open browser session, returned as an image content block the model can actually see (not a base64 string blob). Supports full-page captures, JPEG quality, and the usual readiness controls (`wait`, `wait_selector`, `network_idle`).+### Session tools-### π Session Management-- **`open_session`**: Create a persistent browser session (dynamic or stealthy) that stays open across multiple fetch calls, avoiding the overhead of launching a new browser each time.+#### π Session Management+- **`open_session`**: Create a persistent browser session (dynamic or stealthy) that stays open across multiple `session_fetch` calls, avoiding the overhead of launching a new browser each time. It holds the browser-level configuration and returns the session's effective `settings` for the AI agent (empty for CDP sessions).- **`close_session`**: Close a persistent browser session and free its resources.-- **`list_sessions`**: List all active browser sessions with their details.+- **`list_sessions`**: List all active browser sessions with their details and `settings`.++#### π― Fetching Through a Session+- **`session_fetch`**: Fetch a single URL through an open session (dynamic or stealthy), carrying the per-request options for that call. This is the session counterpart of `fetch`/`stealthy_fetch`.++#### πΈ Screenshots+- **`screenshot`**: Capture a PNG or JPEG screenshot of a page using an open browser session, returned as an image content block the model can actually see (not a base64 string blob). Supports full-page captures, JPEG quality, and the usual readiness controls (`wait`, `wait_selector`, `network_idle`).### Key Capabilities- **Smart Content Extraction**: Convert web pages/elements to Markdown, HTML, or extract a clean version of the text content@@ -47,6 +54,14 @@ The way other servers work is that they extract the content, then pass it all toIf you don't know how to write/use CSS selectors, don't worry. You can tell the AI in the prompt to write selectors to match possible fields for you and watch it try different combinations until it finds the right one, as we will show in the examples section.+## Breaking changes++Since version 0.4.15, the MCP server is reworked. If you are upgrading, note:++1. **The Streamable HTTP transport now requires authentication and binds to localhost.** `scrapling-mcp --http` on its own refuses to start; pass `--auth-token` (or the `SCRAPLING_MCP_AUTH_TOKEN` environment variable), or `--no-auth` to serve it unauthenticated on purpose. The default host is now `127.0.0.1` instead of `0.0.0.0`; pass `--host 0.0.0.0` to accept connections from the network.+2. **The one-shot fetch tools no longer accept `session_id`.** `fetch`, `bulk_fetch`, `stealthy_fetch`, and `bulk_stealthy_fetch` always launch their own browser. To fetch through a session, use the new **`session_fetch`** tool (one URL per call, works with dynamic and stealthy sessions).+3. **`open_session` takes browser-level parameters only.** The per-request options (`wait`, `timeout`, `google_search`, `network_idle`, `disable_resources`, `wait_selector`, `wait_selector_state`, `extra_headers`, `proxy`, `solve_cloudflare`) moved to `session_fetch` and are supplied on each call. `max_pages` was removed too, since a session manages a single page per call now.+## InstallationInstall Scrapling with MCP Support, then double-check that the browser dependencies are installed.@@ -182,7 +197,7 @@ You can also set the `SCRAPLING_EXECUTABLE_PATH` environment variable before sta```Open a stealthy browser session on wss://cdp.provider.example/session/abc123, then use it to scrape the product details from https://shop.example.com. Close the session when you're done.```-Both session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with the fetch and screenshot tools as usual.+Both session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with `session_fetch` and `screenshot` as usual.The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser you started yourself with the remote debugging port enabled:```commandline@@ -357,9 +372,9 @@ We will gradually go from simple prompts to more complex ones. We will use ClaudWhen scraping multiple pages from the same site, use a persistent browser session to avoid the overhead of launching a new browser for each request:```- Open a stealthy browser session with 5 pages maximum pool, then use it to scrape the main details in bulk from the first 5 product pages on https://shop.example.com. Close the session when you're done.+ Open a stealthy browser session, then use it to scrape the main details from the first 5 product pages on https://shop.example.com. Close the session when you're done.```- Claude will use `open_session` to create a persistent browser, pass the `session_id` to `bulk_stealthy_fetch` call while opening all pages at the same time, and then call `close_session` at the end. This is significantly faster than launching a new browser for each page.+ Claude will use `open_session` to create a persistent browser, call `session_fetch` for each product page through that session, and then call `close_session` at the end. This is significantly faster than launching a new browser for each page.!!! danger@@ -419,11 +434,12 @@ The MCP server automatically sanitizes scraped content when `main_content_only`This protection runs automatically on all MCP tool responses. Keep `main_content_only=true` (the default) for maximum protection.### 6. Use Sessions for Multiple Requests-- Use `open_session` to create a persistent browser session when scraping multiple pages-- Pass the `session_id` to `fetch` or `stealthy_fetch` calls to reuse the same browser+- Use `open_session` to create a persistent browser session when scraping multiple pages, then call `session_fetch` for each page through that session+- `open_session` holds the browser-level configuration (headless, locale, cookies, stealth toggles, etc.); the per-request options (timeout, wait_selector, network_idle, solve_cloudflare, etc.) are passed to `session_fetch` on each call, with their defaults shown in the tool schema+- One `session_fetch` works with both session types; `solve_cloudflare` only applies to a stealthy session and raises a clear error on a dynamic one+- The one-shot tools (`fetch`, `bulk_fetch`, `stealthy_fetch`, `bulk_stealthy_fetch`) never take a session; they always launch and close their own browser- Always close sessions with `close_session` when done to free resources-- Use `list_sessions` to check which sessions are still active-- A `session_id` from a dynamic session can only be used with `fetch`/`bulk_fetch`, and a stealthy session can only be used with `stealthy_fetch`/`bulk_stealthy_fetch`+- Use `list_sessions` to check which sessions are still active and see the `settings` each was created with (returned for the AI agent; empty for CDP sessions)- Pass a custom `session_id` to `open_session` to give sessions meaningful names (e.g. `"search"`, `"checkout"`) instead of the random hex default. `open_session` raises if the chosen ID is already in use, so you can detect collisions up front### 7. Capturing Screenshotsdiff --git a/docs/api-reference/mcp-server.md b/docs/api-reference/mcp-server.mdindex 4f28ad0..fc50af8 100644--- a/docs/api-reference/mcp-server.md+++ b/docs/api-reference/mcp-server.md@@ -5,7 +5,7 @@ search:# MCP Server API Reference-The **Scrapling MCP Server** provides nine powerful tools for web scraping through the Model Context Protocol (MCP). This server integrates Scrapling's capabilities directly into AI chatbots and agents, allowing conversational web scraping with advanced anti-bot bypass features.+The **Scrapling MCP Server** provides eleven powerful tools for web scraping through the Model Context Protocol (MCP). This server integrates Scrapling's capabilities directly into AI chatbots and agents, allowing conversational web scraping with advanced anti-bot bypass features.You can start the MCP server by running:diff --git a/scrapling/core/ai.py b/scrapling/core/ai.pyindex 9354f64..677ed70 100644--- a/scrapling/core/ai.py+++ b/scrapling/core/ai.py@@ -24,6 +24,7 @@ from scrapling.fetchers import (AsyncDynamicSession,AsyncStealthySession,)+from scrapling.engines._browsers._types import PlaywrightFetchParams, StealthFetchParamsfrom scrapling.core._types import (Optional,Literal,@@ -33,6 +34,7 @@ from scrapling.core._types import (Dict,List,Any,+ Set,Sequence,SetCookieParam,extraction_types,@@ -53,9 +55,25 @@ def _page_pool_size(urls: Sequence[str]) -> int:return min(max(len(urls), 1), _MAX_POOL_PAGES)-def _supplied_params(**params: Any) -> Dict[str, Any]:- """Keep only the parameters the caller set, so the unset ones fall back to the session/fetcher defaults."""- return {name: value for name, value in params.items() if value is not None}+def _typed_dict_keys(typed_dict: Any) -> frozenset:+ """Collect all the keys a TypedDict holds, including the inherited ones."""+ return frozenset(typed_dict.__required_keys__ | typed_dict.__optional_keys__)+++_EXCLUDED_FETCH_KEYS = frozenset({"page_action", "page_setup", "selector_config"})+_PLAYWRIGHT_FETCH_KEYS = _typed_dict_keys(PlaywrightFetchParams) - _EXCLUDED_FETCH_KEYS+_STEALTH_FETCH_KEYS = _typed_dict_keys(StealthFetchParams) - _EXCLUDED_FETCH_KEYS+++def _session_settings(config: Any) -> Dict[str, Any]:+ """Extract the JSON-safe effective settings of a validated session config struct, for the AI agent."""+ if config.cdp_url:+ return {}+ return {+ f: value+ for f in config.__struct_fields__+ if isinstance(value := getattr(config, f), (str, int, float, bool)) or value is None+ }_FETCH_TOOL_ANNOTATIONS = ToolAnnotations(read_only_hint=True, open_world_hint=True)@@ -78,6 +96,10 @@ class SessionInfo(BaseModel):session_type: SessionType = Field(description="The type of the session: 'dynamic' or 'stealthy'.")created_at: str = Field(description="ISO timestamp of when the session was created.")is_alive: bool = Field(description="Whether the session is still alive and usable.")+ settings: Dict[str, Any] = Field(+ default_factory=dict,+ description="The effective settings this session was created with.",+ )class SessionCreatedModel(SessionInfo):@@ -182,57 +204,36 @@ class ScraplingMCPServer:session_type: SessionType,session_id: Optional[str] = None,headless: bool = True,- google_search: bool = True,real_chrome: bool = False,- wait: int | float = 0,- proxy: Optional[str | Dict[str, str]] = None,timezone_id: str | None = None,locale: str | None = None,- extra_headers: Optional[Dict[str, str]] = None,useragent: Optional[str] = None,cdp_url: Optional[str] = None,executable_path: Optional[str] = None,- timeout: int | float = 30000,- disable_resources: bool = False,- wait_selector: Optional[str] = None,cookies: Sequence[SetCookieParam] | None = None,- network_idle: bool = False,- wait_selector_state: SelectorWaitStates = "attached",- max_pages: int = 5,# Stealthy-only params (ignored for dynamic sessions)hide_canvas: bool = False,block_webrtc: bool = False,allow_webgl: bool = True,- solve_cloudflare: bool = False,additional_args: Optional[Dict] = None,) -> SessionCreatedModel:- """Open a persistent browser session that can be reused across multiple fetch calls.- This avoids the overhead of launching a new browser for each request.+ """Open a persistent browser session that can be reused across multiple `session_fetch` calls.+ This avoids the overhead of launching a new browser for each request. Sessions hold the browser-level+ configuration only; per-request options are passed to `session_fetch` on each call.:param session_type: The type of session to open. Use "dynamic" for standard Playwright browser, or "stealthy" for anti-bot bypass with fingerprint spoofing.:param session_id: Optional custom session ID. If not provided, a random 12-character hex ID will be generated. Useful for naming sessions for easier management.:param headless: Run the browser in headless/hidden (default), or headful/visible mode.- :param google_search: Enabled by default, Scrapling will set a Google referer header.:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.- :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.- :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.:param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.:param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc.- :param extra_headers: A dictionary of extra headers to add to the request.:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.:param executable_path: Absolute path to a custom Chromium-compatible browser executable. Overrides the server-wide default for this session.- :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000.- :param disable_resources: Drop requests for unnecessary resources for a speed boost.- :param wait_selector: Wait for a specific CSS selector to be in a specific state.:param cookies: Set cookies for the session. It should be in a dictionary format that Playwright accepts.- :param network_idle: Wait for the page until there are no network connections for at least 500 ms.- :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.- :param max_pages: Maximum number of concurrent pages/tabs in the browser. Defaults to 5. Higher values allow more parallel fetches.:param hide_canvas: (Stealthy only) Add random noise to canvas operations to prevent fingerprinting.:param block_webrtc: (Stealthy only) Forces WebRTC to respect proxy settings to prevent local IP address leak.:param allow_webgl: (Stealthy only) Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.- :param solve_cloudflare: (Stealthy only) Solves all types of the Cloudflare's Turnstile/Interstitial challenges.:param additional_args: (Stealthy only) Additional arguments to be passed to Playwright's context as additional settings."""session_id = session_id or uuid4().hex[:12]@@ -242,25 +243,15 @@ class ScraplingMCPServer:)common_kwargs: Dict[str, Any] = dict(- wait=wait,- proxy=proxy,locale=locale,- timeout=timeout,cookies=cookies,cdp_url=cdp_url,headless=headless,block_ads=True,- max_pages=max_pages,useragent=useragent,timezone_id=timezone_id,real_chrome=real_chrome,- network_idle=network_idle,- wait_selector=wait_selector,- google_search=google_search,- extra_headers=extra_headers,executable_path=self._resolve_executable_path(executable_path),- disable_resources=disable_resources,- wait_selector_state=wait_selector_state,)session: Union[AsyncDynamicSession, AsyncStealthySession]@@ -270,7 +261,6 @@ class ScraplingMCPServer:hide_canvas=hide_canvas,block_webrtc=block_webrtc,allow_webgl=allow_webgl,- solve_cloudflare=solve_cloudflare,additional_args=additional_args,)else:@@ -286,6 +276,7 @@ class ScraplingMCPServer:session_type=session_type,created_at=entry.created_at,is_alive=True,+ settings=_session_settings(session._config),message=f"Session '{session_id}' ({session_type}) created successfully.",)@@ -308,13 +299,14 @@ class ScraplingMCPServer:)async def list_sessions(self) -> List[SessionInfo]:- """List all active browser sessions with their details."""+ """List all active browser sessions with their details, including the effective settings each one was created with."""return [SessionInfo(session_id=sid,session_type=entry.session_type,created_at=entry.created_at,is_alive=entry.session._is_alive,+ settings=_session_settings(entry.session._config),)for sid, entry in self._sessions.items()]@@ -326,11 +318,11 @@ class ScraplingMCPServer:image_type: ScreenshotType = "png",full_page: bool = False,quality: Optional[int] = None,- wait: int | float | None = None,+ wait: int | float = 0,wait_selector: Optional[str] = None,- wait_selector_state: Optional[SelectorWaitStates] = None,- network_idle: bool | None = None,- timeout: int | float | None = None,+ wait_selector_state: SelectorWaitStates = "attached",+ network_idle: bool = False,+ timeout: int | float = 30000,) -> List[ImageContent | TextContent]:"""Capture a screenshot of a web page using an existing browser session and return it as an image.A browser session must be opened first with `open_session` (either `dynamic` or `stealthy`); the session ID is then passed here.@@ -338,13 +330,13 @@ class ScraplingMCPServer::param url: The URL to navigate to and capture.:param session_id: ID of an open browser session created with `open_session`.:param image_type: Image format. Defaults to "png". Use "jpeg" for smaller file sizes.- :param full_page: When True, captures the full scrollable page instead of just the viewport. Defaults to False.+ :param full_page: When True, captures the full scrollable page instead of just the viewport.:param quality: Image quality (0-100) for JPEG only. Raises if passed with `image_type="png"`.- :param wait: Time in milliseconds to wait after page load before capturing. Uses the session's setting when omitted.- :param wait_selector: Optional CSS selector to wait for before capturing. Uses the session's setting when omitted.- :param wait_selector_state: State to wait for the selector. Uses the session's setting when omitted.- :param network_idle: Wait for the page until there are no network connections for at least 500 ms. Uses the session's setting when omitted.- :param timeout: Timeout in milliseconds for page operations. Uses the session's setting when omitted.+ :param wait: Time in milliseconds to wait after page load before capturing.+ :param wait_selector: Optional CSS selector to wait for before capturing.+ :param wait_selector_state: State to wait for the selector.+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.+ :param timeout: Timeout in milliseconds for page operations."""if quality is not None and image_type != "jpeg":raise ValueError("'quality' is only valid when 'image_type' is 'jpeg'.")@@ -367,13 +359,11 @@ class ScraplingMCPServer:await entry.session.fetch(url,page_action=_capture,- **_supplied_params(- wait=wait,- timeout=timeout,- network_idle=network_idle,- wait_selector=wait_selector,- wait_selector_state=wait_selector_state,- ),+ wait=wait,+ timeout=timeout,+ network_idle=network_idle,+ wait_selector=wait_selector,+ wait_selector_state=wait_selector_state,)if "error" in captured:@@ -411,24 +401,24 @@ class ScraplingMCPServer::param url: The URL to request.:param impersonate: Browser version to impersonate its fingerprint. It's using the latest chrome version by default.- :param extraction_type: The type of content to extract from the page: "markdown" (default), "html", or "text".- :param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.- :param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `<body>` tag.+ :param extraction_type: The type of content to extract from the page: "markdown", "html", or "text".+ :param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page.+ :param main_content_only: Whether to extract only the main content of the page. The main content here is the data inside the `<body>` tag.:param params: Query string parameters for the request.:param headers: Headers to include in the request.:param cookies: Cookies to use in the request.:param timeout: Number of seconds to wait before timing out.:param follow_redirects: Whether to follow redirects. Defaults to "safe", which follows redirects but rejects those targeting internal/private IPs (SSRF protection).Pass True to follow all redirects without restriction.- :param max_redirects: Maximum number of redirects. Default 30, use -1 for unlimited.- :param retries: Number of retry attempts. Defaults to 3.- :param retry_delay: Number of seconds to wait between retry attempts. Defaults to 1 second.+ :param max_redirects: Maximum number of redirects. Use -1 for unlimited.+ :param retries: Number of retry attempts.+ :param retry_delay: Number of seconds to wait between retry attempts.:param proxy: Proxy URL to use. Format: "http://username:password@localhost:8030".Cannot be used together with the `proxies` parameter.:param proxy_auth: HTTP basic auth for proxy in dictionary format with `username` and `password` keys.:param auth: HTTP basic auth in dictionary format with `username` and `password` keys.:param verify: Whether to verify HTTPS certificates.- :param http3: Whether to use HTTP3. Defaults to False. It might be problematic if used it with `impersonate`.+ :param http3: Whether to use HTTP3. It might be problematic if used it with `impersonate`.:param stealthy_headers: If enabled (default), it creates and adds real browser headers. It also sets a Google referer header."""results = await ScraplingMCPServer.bulk_get(@@ -481,24 +471,24 @@ class ScraplingMCPServer::param urls: A list of the URLs to request.:param impersonate: Browser version to impersonate its fingerprint. It's using the latest chrome version by default.- :param extraction_type: The type of content to extract from the page: "markdown" (default), "html", or "text".+ :param extraction_type: The type of content to extract from the page: "markdown", "html", or "text".:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.- :param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `<body>` tag.+ :param main_content_only: Whether to extract only the main content of the page. The main content here is the data inside the `<body>` tag.:param params: Query string parameters for the request.:param headers: Headers to include in the request.:param cookies: Cookies to use in the request.:param timeout: Number of seconds to wait before timing out.:param follow_redirects: Whether to follow redirects. Defaults to "safe", which follows redirects but rejects those targeting internal/private IPs (SSRF protection).Pass True to follow all redirects without restriction.- :param max_redirects: Maximum number of redirects. Default 30, use -1 for unlimited.- :param retries: Number of retry attempts. Defaults to 3.- :param retry_delay: Number of seconds to wait between retry attempts. Defaults to 1 second.+ :param max_redirects: Maximum number of redirects. Use -1 for unlimited.+ :param retries: Number of retry attempts.+ :param retry_delay: Number of seconds to wait between retry attempts.:param proxy: Proxy URL to use. Format: "http://username:password@localhost:8030".Cannot be used together with the `proxies` parameter.:param proxy_auth: HTTP basic auth for proxy in dictionary format with `username` and `password` keys.:param auth: HTTP basic auth in dictionary format with `username` and `password` keys.:param verify: Whether to verify HTTPS certificates.- :param http3: Whether to use HTTP3. Defaults to False. It might be problematic if used it with `impersonate`.+ :param http3: Whether to use HTTP3. It might be problematic if used it with `impersonate`.:param stealthy_headers: If enabled (default), it creates and adds real browser headers. It also sets a Google referer header."""normalized_proxy_auth = _normalize_credentials(proxy_auth)@@ -536,9 +526,9 @@ class ScraplingMCPServer:css_selector: Optional[str] = None,main_content_only: bool = True,headless: bool = True, # noqa: F821- google_search: bool | None = None,+ google_search: bool = True,real_chrome: bool = False,- wait: int | float | None = None,+ wait: int | float = 0,proxy: Optional[str | Dict[str, str]] = None,timezone_id: str | None = None,locale: str | None = None,@@ -546,41 +536,39 @@ class ScraplingMCPServer:useragent: Optional[str] = None,cdp_url: Optional[str] = None,executable_path: Optional[str] = None,- timeout: int | float | None = None,- disable_resources: bool | None = None,+ timeout: int | float = 30000,+ disable_resources: bool = False,wait_selector: Optional[str] = None,cookies: Sequence[SetCookieParam] | None = None,- network_idle: bool | None = None,- wait_selector_state: Optional[SelectorWaitStates] = None,- session_id: Optional[str] = None,+ network_idle: bool = False,+ wait_selector_state: SelectorWaitStates = "attached",) -> ResponseModel:"""Use playwright to open a browser to fetch a URL and return a structured output of the result.Only suitable for low-mid protection levels.:param url: The URL to request.- :param extraction_type: The type of content to extract from the page: "markdown" (default), "html", or "text".- :param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.- :param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `<body>` tag.+ :param extraction_type: The type of content to extract from the page: "markdown", "html", or "text".+ :param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page.+ :param main_content_only: Whether to extract only the main content of the page. The main content here is the data inside the `<body>` tag.:param headless: Run the browser in headless/hidden (default), or headful/visible mode.:param disable_resources: Drop requests for unnecessary resources for a speed boost.Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.:param cookies: Set cookies for the next request. It should be in a dictionary format that Playwright accepts.:param network_idle: Wait for the page until there are no network connections for at least 500 ms.- :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page.:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.:param wait_selector: Wait for a specific CSS selector to be in a specific state.:param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.:param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting- rules. Defaults to the system default locale.- :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.+ rules. Defaults to the system locale.+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`.:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.:param executable_path: Absolute path to a custom Chromium-compatible browser executable. Overrides the server-wide default for this request.:param google_search: Enabled by default, Scrapling will set a Google referer header.:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.- :param session_id: Optional session ID from open_session. If provided, reuses the existing browser session instead of creating a new one, and every option you leave out here keeps the value the session was opened with."""results = await self.bulk_fetch(urls=[url],@@ -604,7 +592,6 @@ class ScraplingMCPServer:cookies=cookies,network_idle=network_idle,wait_selector_state=wait_selector_state,- session_id=session_id,)return results[0]@@ -615,9 +602,9 @@ class ScraplingMCPServer:css_selector: Optional[str] = None,main_content_only: bool = True,headless: bool = True, # noqa: F821- google_search: bool | None = None,+ google_search: bool = True,real_chrome: bool = False,- wait: int | float | None = None,+ wait: int | float = 0,proxy: Optional[str | Dict[str, str]] = None,timezone_id: str | None = None,locale: str | None = None,@@ -625,73 +612,63 @@ class ScraplingMCPServer:useragent: Optional[str] = None,cdp_url: Optional[str] = None,executable_path: Optional[str] = None,- timeout: int | float | None = None,- disable_resources: bool | None = None,+ timeout: int | float = 30000,+ disable_resources: bool = False,wait_selector: Optional[str] = None,cookies: Sequence[SetCookieParam] | None = None,- network_idle: bool | None = None,- wait_selector_state: Optional[SelectorWaitStates] = None,- session_id: Optional[str] = None,+ network_idle: bool = False,+ wait_selector_state: SelectorWaitStates = "attached",) -> List[ResponseModel]:"""Use playwright to open a browser, then fetch a group of URLs at the same time, and for each page return a structured output of the result.Only suitable for low-mid protection levels.:param urls: A list of the URLs to request. Batches bigger than 50 URLs are fetched through a pool of 50 concurrent pages.- :param extraction_type: The type of content to extract from the page: "markdown" (default), "html", or "text".- :param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.- :param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `<body>` tag.+ :param extraction_type: The type of content to extract from the page: "markdown", "html", or "text".+ :param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page.+ :param main_content_only: Whether to extract only the main content of the page. The main content here is the data inside the `<body>` tag.:param headless: Run the browser in headless/hidden (default), or headful/visible mode.:param disable_resources: Drop requests for unnecessary resources for a speed boost.Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.:param cookies: Set cookies for the next request. It should be in a dictionary format that Playwright accepts.:param network_idle: Wait for the page until there are no network connections for at least 500 ms.- :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page.:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.:param wait_selector: Wait for a specific CSS selector to be in a specific state.:param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.:param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting- rules. Defaults to the system default locale.- :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.+ rules. Defaults to the system locale.+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`.:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.:param executable_path: Absolute path to a custom Chromium-compatible browser executable. Overrides the server-wide default for this request.:param google_search: Enabled by default, Scrapling will set a Google referer header.:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.- :param session_id: Optional session ID from open_session. If provided, reuses the existing browser session instead of creating a new one, and every option you leave out here keeps the value the session was opened with."""- fetch_params = _supplied_params(+ async with AsyncDynamicSession(wait=wait,+ proxy=proxy,+ locale=locale,timeout=timeout,+ cookies=cookies,+ cdp_url=cdp_url,+ headless=headless,+ block_ads=True,+ max_pages=_page_pool_size(urls),+ useragent=useragent,+ timezone_id=timezone_id,+ real_chrome=real_chrome,+ network_idle=network_idle,+ wait_selector=wait_selector,google_search=google_search,extra_headers=extra_headers,+ executable_path=self._resolve_executable_path(executable_path),disable_resources=disable_resources,- wait_selector=wait_selector,wait_selector_state=wait_selector_state,- network_idle=network_idle,- )- if session_id:- entry = self._get_session(session_id, "dynamic")- tasks = [entry.session.fetch(url, proxy=proxy, **fetch_params) for url in urls]+ ) as session:+ tasks = [session.fetch(url) for url in urls]responses = await gather(*tasks)- else:- async with AsyncDynamicSession(- proxy=proxy,- locale=locale,- cookies=cookies,- cdp_url=cdp_url,- headless=headless,- block_ads=True,- max_pages=_page_pool_size(urls),- useragent=useragent,- timezone_id=timezone_id,- real_chrome=real_chrome,- executable_path=self._resolve_executable_path(executable_path),- **fetch_params,- ) as session:- tasks = [session.fetch(url) for url in urls]- responses = await gather(*tasks)return [_translate_response(page, extraction_type, css_selector, main_content_only) for page in responses]@@ -702,9 +679,9 @@ class ScraplingMCPServer:css_selector: Optional[str] = None,main_content_only: bool = True,headless: bool = True, # noqa: F821- google_search: bool | None = None,+ google_search: bool = True,real_chrome: bool = False,- wait: int | float | None = None,+ wait: int | float = 0,proxy: Optional[str | Dict[str, str]] = None,timezone_id: str | None = None,locale: str | None = None,@@ -713,25 +690,24 @@ class ScraplingMCPServer:hide_canvas: bool = False,cdp_url: Optional[str] = None,executable_path: Optional[str] = None,- timeout: int | float | None = None,- disable_resources: bool | None = None,+ timeout: int | float = 30000,+ disable_resources: bool = False,wait_selector: Optional[str] = None,cookies: Sequence[SetCookieParam] | None = None,- network_idle: bool | None = None,- wait_selector_state: Optional[SelectorWaitStates] = None,+ network_idle: bool = False,+ wait_selector_state: SelectorWaitStates = "attached",block_webrtc: bool = False,allow_webgl: bool = True,- solve_cloudflare: bool | None = None,+ solve_cloudflare: bool = False,additional_args: Optional[Dict] = None,- session_id: Optional[str] = None,) -> ResponseModel:"""Use the stealthy fetcher to fetch a URL and return a structured output of the result.The only fetcher suitable for high-protection websites.:param url: The URL to request.- :param extraction_type: The type of content to extract from the page: "markdown" (default), "html", or "text".- :param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.- :param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `<body>` tag.+ :param extraction_type: The type of content to extract from the page: "markdown", "html", or "text".+ :param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page.+ :param main_content_only: Whether to extract only the main content of the page. The main content here is the data inside the `<body>` tag.:param headless: Run the browser in headless/hidden (default), or headful/visible mode.:param disable_resources: Drop requests for unnecessary resources for a speed boost.Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.@@ -741,12 +717,12 @@ class ScraplingMCPServer::param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.:param network_idle: Wait for the page until there are no network connections for at least 500 ms.:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.- :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page.:param wait_selector: Wait for a specific CSS selector to be in a specific state.:param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.:param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting- rules. Defaults to the system default locale.- :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.+ rules. Defaults to the system locale.+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`.:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.:param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.@@ -756,7 +732,6 @@ class ScraplingMCPServer::param extra_headers: A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.:param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.- :param session_id: Optional session ID from open_session. If provided, reuses the existing browser session instead of creating a new one, and every option you leave out here keeps the value the session was opened with."""results = await self.bulk_stealthy_fetch(urls=[url],@@ -785,7 +760,6 @@ class ScraplingMCPServer:allow_webgl=allow_webgl,solve_cloudflare=solve_cloudflare,additional_args=additional_args,- session_id=session_id,)return results[0]@@ -796,9 +770,9 @@ class ScraplingMCPServer:css_selector: Optional[str] = None,main_content_only: bool = True,headless: bool = True, # noqa: F821- google_search: bool | None = None,+ google_search: bool = True,real_chrome: bool = False,- wait: int | float | None = None,+ wait: int | float = 0,proxy: Optional[str | Dict[str, str]] = None,timezone_id: str | None = None,locale: str | None = None,@@ -807,25 +781,24 @@ class ScraplingMCPServer:hide_canvas: bool = False,cdp_url: Optional[str] = None,executable_path: Optional[str] = None,- timeout: int | float | None = None,- disable_resources: bool | None = None,+ timeout: int | float = 30000,+ disable_resources: bool = False,wait_selector: Optional[str] = None,cookies: Sequence[SetCookieParam] | None = None,- network_idle: bool | None = None,- wait_selector_state: Optional[SelectorWaitStates] = None,+ network_idle: bool = False,+ wait_selector_state: SelectorWaitStates = "attached",block_webrtc: bool = False,allow_webgl: bool = True,- solve_cloudflare: bool | None = None,+ solve_cloudflare: bool = False,additional_args: Optional[Dict] = None,- session_id: Optional[str] = None,) -> List[ResponseModel]:"""Use the stealthy fetcher to fetch a group of URLs at the same time, and for each page return a structured output of the result.The only fetcher suitable for high-protection websites.:param urls: A list of the URLs to request. Batches bigger than 50 URLs are fetched through a pool of 50 concurrent pages.- :param extraction_type: The type of content to extract from the page: "markdown" (default), "html", or "text".- :param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.- :param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `<body>` tag.+ :param extraction_type: The type of content to extract from the page: "markdown", "html", or "text".+ :param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page.+ :param main_content_only: Whether to extract only the main content of the page. The main content here is the data inside the `<body>` tag.:param headless: Run the browser in headless/hidden (default), or headful/visible mode.:param disable_resources: Drop requests for unnecessary resources for a speed boost.Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.@@ -835,12 +808,12 @@ class ScraplingMCPServer::param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.:param network_idle: Wait for the page until there are no network connections for at least 500 ms.:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.- :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page.:param wait_selector: Wait for a specific CSS selector to be in a specific state.:param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.:param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting- rules. Defaults to the system default locale.- :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.+ rules. Defaults to the system locale.+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`.:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.:param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.@@ -850,47 +823,107 @@ class ScraplingMCPServer::param extra_headers: A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.:param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.- :param session_id: Optional session ID from open_session. If provided, reuses the existing browser session instead of creating a new one, and every option you leave out here keeps the value the session was opened with."""- fetch_params = _supplied_params(+ async with AsyncStealthySession(wait=wait,+ proxy=proxy,+ locale=locale,+ cdp_url=cdp_url,timeout=timeout,+ cookies=cookies,+ headless=headless,+ block_ads=True,+ max_pages=_page_pool_size(urls),+ useragent=useragent,+ timezone_id=timezone_id,+ real_chrome=real_chrome,+ hide_canvas=hide_canvas,+ allow_webgl=allow_webgl,+ network_idle=network_idle,+ block_webrtc=block_webrtc,+ wait_selector=wait_selector,google_search=google_search,extra_headers=extra_headers,+ executable_path=self._resolve_executable_path(executable_path),+ additional_args=additional_args,+ solve_cloudflare=solve_cloudflare,disable_resources=disable_resources,- wait_selector=wait_selector,wait_selector_state=wait_selector_state,- network_idle=network_idle,- solve_cloudflare=solve_cloudflare,- )- if session_id:- entry = self._get_session(session_id, "stealthy")- tasks = [entry.session.fetch(url, proxy=proxy, **fetch_params) for url in urls]+ ) as session:+ tasks = [session.fetch(url) for url in urls]responses = await gather(*tasks)- else:- async with AsyncStealthySession(- proxy=proxy,- locale=locale,- cdp_url=cdp_url,- cookies=cookies,- headless=headless,- block_ads=True,- max_pages=_page_pool_size(urls),- useragent=useragent,- timezone_id=timezone_id,- real_chrome=real_chrome,- hide_canvas=hide_canvas,- allow_webgl=allow_webgl,- block_webrtc=block_webrtc,- executable_path=self._resolve_executable_path(executable_path),- additional_args=additional_args,- **fetch_params,- ) as session:- tasks = [session.fetch(url) for url in urls]- responses = await gather(*tasks)return [_translate_response(page, extraction_type, css_selector, main_content_only) for page in responses]+ async def session_fetch(+ self,+ url: str,+ session_id: str,+ extraction_type: extraction_types = "markdown",+ css_selector: Optional[str] = None,+ main_content_only: bool = True,+ wait: int | float = 0,+ timeout: int | float = 30000,+ google_search: bool = True,+ network_idle: bool = False,+ load_dom: bool = True,+ disable_resources: bool = False,+ wait_selector: Optional[str] = None,+ wait_selector_state: SelectorWaitStates = "attached",+ extra_headers: Optional[Dict[str, str]] = None,+ blocked_domains: Optional[Set[str]] = None,+ proxy: Optional[str | Dict[str, str]] = None,+ solve_cloudflare: bool = False,+ ) -> ResponseModel:+ """Fetch a URL through a browser session previously opened with `open_session` and return a structured output of the result.+ The session (dynamic or stealthy) holds the browser-level configuration; every option here applies to this request only, with the defaults shown.++ :param url: The URL to request.+ :param session_id: ID of an open browser session created with `open_session`.+ :param extraction_type: The type of content to extract from the page: "markdown", "html", or "text".+ :param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page.+ :param main_content_only: Whether to extract only the main content of the page. The main content here is the data inside the `<body>` tag.+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object.+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page.+ :param google_search: Enabled by default, Scrapling will set a Google referer header.+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.+ :param load_dom: Enabled by default, wait for all JavaScript on the page to fully load and execute.+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`.+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._+ :param blocked_domains: A list of domain names to block requests to for this request. Subdomains are also matched.+ :param proxy: The proxy to be used with this request, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.+ :param solve_cloudflare: (Stealthy sessions only) Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response.+ """+ entry = self._get_session(session_id, expected_type=None)+ if solve_cloudflare and entry.session_type != "stealthy":+ raise ValueError(+ f"Session '{session_id}' is a '{entry.session_type}' session, so it can't solve Cloudflare "+ f"challenges. Open a 'stealthy' session for that."+ )++ fetch_keys = _STEALTH_FETCH_KEYS if entry.session_type == "stealthy" else _PLAYWRIGHT_FETCH_KEYS+ fetch_params = dict(+ wait=wait,+ timeout=timeout,+ google_search=google_search,+ network_idle=network_idle,+ load_dom=load_dom,+ disable_resources=disable_resources,+ wait_selector=wait_selector,+ wait_selector_state=wait_selector_state,+ extra_headers=extra_headers,+ blocked_domains=blocked_domains,+ proxy=proxy,+ solve_cloudflare=solve_cloudflare,+ )+ page = await entry.session.fetch(+ url, **{name: value for name, value in fetch_params.items() if name in fetch_keys}+ )+ return _translate_response(page, extraction_type, css_selector, main_content_only)+@staticmethoddef _transport_security(allowed_hosts: Sequence[str]) -> Optional[TransportSecuritySettings]:"""Build the DNS-rebinding protection settings for the streamable-http transport."""@@ -916,18 +949,18 @@ class ScraplingMCPServer:],"cache_hints": {"tools/list": CacheHint(ttl_ms=3_600_000, scope="public")},"instructions": """Follow these instructions precisely:-1. When the `open_session` tool is used, make sure to close the session with `close_session` after you finish, and use `list_sessions` if you lose track of the open sessions.-2. If the user didn't specify which tool to use, start with the `get` tool, then escalate. The `get` tool and the bulk version are only suitable for low-mid protection levels.+1. When the `open_session` tool is used, make sure to close the session with `close_session` after you finish, and use `list_sessions` if you lose track of the open sessions or their effective settings.+2. If the user didn't specify which tool to use, start with the `get` tool, then escalate. The `get` tool and the bulk version are suitable only for low-to-mid protection levels.For high-protection levels or websites that require JS loading, use the other tools directly.3. For all tools, if the `css_selector` resolves to more than one element, all the elements will be returned.4. For all fetch tools, the `extraction_type` parameter controls the format of the returned content: "markdown" (default) converts the page content to Markdown, "html" returns the raw HTML, and "text" returns the text content of the page.5. For all fetch tools, `main_content_only` is enabled by default and returns only the content inside the page's `<body>` tag. Pass `main_content_only=False` when you need the full page instead.-6. If the task consists of multiple requests to the same website, open a session to be more efficient.-7. For browser-based tools, if a `session_id` is provided (from open_session), the browser session will be reused instead of creating a new one.- When using a session, browser-level params (headless, locale, useragent, etc.) are ignored since they were set at session creation time, and any per-request param you leave out keeps the value the session was opened with.-8. If you are making multiple requests, use the bulk version of the tool to be more efficient.+6. If the task consists of multiple sequential requests to the same website, open a session with `open_session` once, then call `session_fetch` per page to be more efficient.+7. Sessions hold the browser-level configuration set at `open_session`, while `session_fetch` carries the per-request options and applies them on each call with the defaults shown in its schema.+ The one-shot tools (`fetch`, `bulk_fetch`, `stealthy_fetch`, `bulk_stealthy_fetch`) never touch sessions and always launch their own browser.+8. If you are making multiple parallel one-shot requests, use the bulk version of the tool to be more efficient.9. If you are crawling/browsing a website, be more efficient by using the `css_selector` parameter to only access the parts you are interested in and save money/time. Example: use the `a` selector to extract the urls right away.-10. The user can pass a CDP URL to connect to a remote browser session through the `open_session` tool, then use it in the rest of the tools.+10. The user can pass a CDP URL to connect to a remote browser session through the `open_session` tool, then use it with the session tools.""",}if self._auth_token:@@ -991,6 +1024,14 @@ class ScraplingMCPServer:structured_output=True,annotations=_FETCH_TOOL_ANNOTATIONS,)+ # Session-scoped fetch tool+ server.add_tool(+ self.session_fetch,+ title="session_fetch",+ description=self.session_fetch.__doc__,+ structured_output=True,+ annotations=_FETCH_TOOL_ANNOTATIONS,+ )# Screenshot tool (returns image + url content blocks, not structured JSON)server.add_tool(self.screenshot,diff --git a/tests/ai/test_ai_mcp.py b/tests/ai/test_ai_mcp.pyindex 3e07491..65d5d36 100644--- a/tests/ai/test_ai_mcp.py+++ b/tests/ai/test_ai_mcp.py@@ -1,4 +1,5 @@import base64+import inspectimport structfrom typing import Anyfrom contextlib import contextmanager@@ -23,11 +24,12 @@ from scrapling.core.ai import (SessionClosedModel,_normalize_credentials,_page_pool_size,+ _session_settings,_StaticTokenVerifier,- _supplied_params,+ _STEALTH_FETCH_KEYS,_translate_response,)-from scrapling.engines._browsers._validators import PlaywrightConfig, StealthConfig, validate_fetch+from scrapling.engines._browsers._validators import PlaywrightConfig, StealthConfig, models_default_values, validatefrom scrapling.fetchers import AsyncDynamicSession, AsyncStealthySession@@ -62,11 +64,16 @@ class _FakePage:class _FakeAsyncBrowserSession:instances: list["_FakeAsyncBrowserSession"] = []+ _config_model: Any = PlaywrightConfigdef __init__(self, **kwargs: Any) -> None:self.kwargs = kwargsself.fetch_calls: list[dict[str, Any]] = []self._is_alive = False+ # `executable_path` is validated against the filesystem; drop it so the fake accepts test paths.+ self._config = validate(+ {name: value for name, value in kwargs.items() if name != "executable_path"}, self._config_model+ )type(self).instances.append(self)async def __aenter__(self):@@ -103,6 +110,7 @@ class _FakeDynamicSession(_FakeAsyncBrowserSession):class _FakeStealthySession(_FakeAsyncBrowserSession):instances = []+ _config_model = StealthConfig@pytest_httpbin.use_class_based_httpbin@@ -209,43 +217,31 @@ class TestSessionManagement:await server.close_session(session_id)@pytest.mark.asyncio- async def test_fetch_with_session(self, server, test_url):- """Test fetching with a persistent dynamic session"""+ async def test_session_fetch_reuses_the_session(self, server, test_url):+ """Test fetching a page twice through a persistent dynamic session"""result = await server.open_session(session_type="dynamic", headless=True)session_id = result.session_id- # Fetch using the session- response = await server.fetch(url=test_url, session_id=session_id)+ response = await server.session_fetch(url=test_url, session_id=session_id)assert isinstance(response, ResponseModel)assert response.status == 200# Fetch again with the same session (reuse)- response2 = await server.fetch(url=test_url, session_id=session_id)+ response2 = await server.session_fetch(url=test_url, session_id=session_id)assert isinstance(response2, ResponseModel)assert response2.status == 200await server.close_session(session_id)@pytest.mark.asyncio- async def test_bulk_fetch_with_session(self, server, test_url):- """Test bulk fetching with a persistent dynamic session"""- result = await server.open_session(session_type="dynamic", headless=True, max_pages=5)- session_id = result.session_id-- responses = await server.bulk_fetch(urls=[test_url, test_url], session_id=session_id)- assert len(responses) == 2- assert all(isinstance(r, ResponseModel) for r in responses)-- await server.close_session(session_id)-- @pytest.mark.asyncio- async def test_session_type_mismatch(self, server, test_url):- """Test that using a dynamic session with stealthy_fetch raises an error"""+ async def test_session_fetch_accepts_per_request_overrides(self, server, test_url):+ """A per-request override is honored on a session fetch"""result = await server.open_session(session_type="dynamic", headless=True)session_id = result.session_id- with pytest.raises(ValueError, match="'dynamic' session"):- await server.stealthy_fetch(url=test_url, session_id=session_id)+ response = await server.session_fetch(url=test_url, session_id=session_id, network_idle=True, timeout=45000)+ assert isinstance(response, ResponseModel)+ assert response.status == 200await server.close_session(session_id)@@ -256,20 +252,20 @@ class TestSessionManagement:await server.close_session("nonexistent")@pytest.mark.asyncio- async def test_fetch_with_nonexistent_session(self, server, test_url):+ async def test_session_fetch_with_nonexistent_session(self, server, test_url):"""Test fetching with a session ID that doesn't exist"""with pytest.raises(ValueError, match="not found"):- await server.fetch(url=test_url, session_id="nonexistent")+ await server.session_fetch(url=test_url, session_id="nonexistent")@pytest.mark.asyncio- async def test_fetch_with_closed_session(self, server, test_url):+ async def test_session_fetch_with_closed_session(self, server, test_url):"""Test fetching with a session that has been closed"""result = await server.open_session(session_type="dynamic", headless=True)session_id = result.session_idawait server.close_session(session_id)with pytest.raises(ValueError, match="not found"):- await server.fetch(url=test_url, session_id=session_id)+ await server.session_fetch(url=test_url, session_id=session_id)@pytest.mark.asyncioasync def test_open_session_with_custom_id(self, server):@@ -393,131 +389,8 @@ class TestBulkPagePool:)-async def _noop_page_action(page: Any) -> None:- """Stand-in for the `page_action` the screenshot tool always sends."""---class TestSessionSettingsSurviveFetchKwargs:- """The kwargs the tools send on the session path must not overwrite the settings of the session"""-- def test_dynamic_session_settings_survive_bulk_fetch_kwargs(self):- """Options left out of a `bulk_fetch` call keep the values `open_session` was given"""- session = AsyncDynamicSession(- wait=1500,- timeout=45000,- network_idle=True,- disable_resources=True,- google_search=False,- wait_selector="#main",- wait_selector_state="visible",- extra_headers={"x-test": "1"},- )-- params = validate_fetch(- _supplied_params(- wait=None,- timeout=None,- google_search=None,- extra_headers=None,- disable_resources=None,- wait_selector=None,- wait_selector_state=None,- network_idle=None,- ),- session,- PlaywrightConfig,- )-- assert params.wait == 1500, f"Expected the session's wait 1500, got {params.wait}"- assert params.timeout == 45000, f"Expected the session's timeout 45000, got {params.timeout}"- assert params.network_idle is True, f"Expected the session's network_idle True, got {params.network_idle}"- assert params.disable_resources is True, (- f"Expected the session's disable_resources True, got {params.disable_resources}"- )- assert params.google_search is False, f"Expected the session's google_search False, got {params.google_search}"- assert params.wait_selector == "#main", (- f"Expected the session's wait_selector '#main', got {params.wait_selector}"- )- assert params.wait_selector_state == "visible", (- f"Expected the session's wait_selector_state 'visible', got {params.wait_selector_state}"- )- assert params.extra_headers == {"x-test": "1"}, (- f"Expected the session's extra_headers, got {params.extra_headers}"- )-- def test_stealthy_session_keeps_solving_cloudflare(self):- """A session opened with `solve_cloudflare` keeps solving, and keeps the timeout that comes with it"""- session = AsyncStealthySession(solve_cloudflare=True)-- assert session._config.timeout == 60_000, (- f"Expected solve_cloudflare to raise the session timeout to 60,000, got {session._config.timeout}"- )-- params = validate_fetch(- _supplied_params(- wait=None,- timeout=None,- google_search=None,- extra_headers=None,- disable_resources=None,- wait_selector=None,- wait_selector_state=None,- network_idle=None,- solve_cloudflare=None,- ),- session,- StealthConfig,- )-- assert params.solve_cloudflare is True, (- f"Expected the session's solve_cloudflare True, got {params.solve_cloudflare}"- )- assert params.timeout == 60_000, f"Expected the session's timeout 60,000, got {params.timeout}"-- def test_screenshot_kwargs_keep_session_settings(self):- """The screenshot tool only has a session path, so its unset options must defer to the session too"""- session = AsyncDynamicSession(wait=1500, timeout=45000, network_idle=True, wait_selector="#main")-- params = validate_fetch(- {- "page_action": _noop_page_action,- **_supplied_params(- wait=None,- timeout=None,- network_idle=None,- wait_selector=None,- wait_selector_state=None,- ),- },- session,- PlaywrightConfig,- )-- assert params.page_action is _noop_page_action, "Expected the capture callback to stay on the fetch params"- assert params.wait == 1500, f"Expected the session's wait 1500, got {params.wait}"- assert params.timeout == 45000, f"Expected the session's timeout 45000, got {params.timeout}"- assert params.network_idle is True, f"Expected the session's network_idle True, got {params.network_idle}"- assert params.wait_selector == "#main", (- f"Expected the session's wait_selector '#main', got {params.wait_selector}"- )-- def test_supplied_values_still_override_the_session(self):- """Deferring to the session must not cost the caller the ability to override it per request"""- session = AsyncDynamicSession(wait=1500, timeout=45000, network_idle=True)-- params = validate_fetch(- _supplied_params(wait=None, timeout=15000, network_idle=False),- session,- PlaywrightConfig,- )-- assert params.timeout == 15000, f"Expected the supplied timeout 15000, got {params.timeout}"- assert params.network_idle is False, f"Expected the supplied network_idle False, got {params.network_idle}"- assert params.wait == 1500, f"Expected the untouched wait to stay 1500, got {params.wait}"---class TestSessionParameterForwarding:- """What the tools actually hand to a session's `fetch`, and what they hand to a fresh session"""+class TestSessionFetchForwarding:+ """`session_fetch` forwards its per-request params by name to the session's fetch()."""@pytest.fixture(autouse=True)def reset_fakes(self):@@ -525,123 +398,152 @@ class TestSessionParameterForwarding:_FakeStealthySession.instances = []@staticmethod- def _session_fetch_calls(fake: type[_FakeAsyncBrowserSession]) -> list[dict[str, Any]]:- return fake.instances[0].fetch_calls+ def _fetch_call(fake: type[_FakeAsyncBrowserSession]) -> dict[str, Any]:+ return fake.instances[0].fetch_calls[0]@pytest.mark.asyncio- async def test_bulk_fetch_forwards_nothing_it_was_not_given(self, monkeypatch):- """An option the caller didn't pass must not reach the session as the tool's own default"""+ async def test_dynamic_session_fetch_forwards_the_per_request_params(self, monkeypatch):+ """A dynamic session receives every dynamic per-request param, including explicit None values"""monkeypatch.setattr("scrapling.core.ai.AsyncDynamicSession", _FakeDynamicSession)server = ScraplingMCPServer()opened = await server.open_session(session_type="dynamic")- await server.bulk_fetch(urls=["https://example.com/1"], session_id=opened.session_id)+ await server.session_fetch(url="https://example.com/1", session_id=opened.session_id)++ forwarded = self._fetch_call(_FakeDynamicSession)+ assert forwarded == {+ "wait": 0,+ "timeout": 30000,+ "google_search": True,+ "network_idle": False,+ "load_dom": True,+ "disable_resources": False,+ "wait_selector": None,+ "wait_selector_state": "attached",+ "extra_headers": None,+ "blocked_domains": None,+ "proxy": None,+ }, forwarded+ assert "solve_cloudflare" not in forwarded, "solve_cloudflare must not reach a dynamic session"++ @pytest.mark.asyncio+ async def test_stealthy_session_fetch_forwards_solve_cloudflare(self, monkeypatch):+ """A stealthy session additionally receives solve_cloudflare"""+ monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)+ server = ScraplingMCPServer()+ opened = await server.open_session(session_type="stealthy")- forwarded = self._session_fetch_calls(_FakeDynamicSession)[0]- assert forwarded == {"proxy": None}, f"Expected only the proxy override to be forwarded, got {forwarded}"+ await server.session_fetch(url="https://example.com/1", session_id=opened.session_id, solve_cloudflare=True)++ forwarded = self._fetch_call(_FakeStealthySession)+ assert forwarded.get("solve_cloudflare") is True, forwarded@pytest.mark.asyncio- async def test_bulk_fetch_forwards_what_it_was_given(self, monkeypatch):- """Options the caller did pass still reach the session"""+ async def test_supplied_values_are_forwarded(self, monkeypatch):+ """Per-request overrides reach the session as given"""monkeypatch.setattr("scrapling.core.ai.AsyncDynamicSession", _FakeDynamicSession)server = ScraplingMCPServer()opened = await server.open_session(session_type="dynamic")- await server.bulk_fetch(- urls=["https://example.com/1"], session_id=opened.session_id, network_idle=True, timeout=45000+ await server.session_fetch(+ url="https://example.com/1", session_id=opened.session_id, timeout=45000, wait_selector="#main")- forwarded = self._session_fetch_calls(_FakeDynamicSession)[0]- assert forwarded == {"proxy": None, "network_idle": True, "timeout": 45000}, (- f"Expected only the supplied options to be forwarded, got {forwarded}"- )+ forwarded = self._fetch_call(_FakeDynamicSession)+ assert forwarded["timeout"] == 45000+ assert forwarded["wait_selector"] == "#main"@pytest.mark.asyncio- async def test_fetch_forwards_nothing_it_was_not_given(self, monkeypatch):- """The single-URL wrapper passes the unset options through to `bulk_fetch` untouched"""+ async def test_solve_cloudflare_on_dynamic_session_raises(self, monkeypatch):+ """Asking a dynamic session to solve Cloudflare is a clear error, not a silent no-op"""monkeypatch.setattr("scrapling.core.ai.AsyncDynamicSession", _FakeDynamicSession)server = ScraplingMCPServer()opened = await server.open_session(session_type="dynamic")- await server.fetch(url="https://example.com/1", session_id=opened.session_id)-- forwarded = self._session_fetch_calls(_FakeDynamicSession)[0]- assert forwarded == {"proxy": None}, f"Expected only the proxy override to be forwarded, got {forwarded}"+ with pytest.raises(ValueError, match="can't solve Cloudflare"):+ await server.session_fetch(url="https://example.com/1", session_id=opened.session_id, solve_cloudflare=True)@pytest.mark.asyncio- async def test_bulk_stealthy_fetch_leaves_solve_cloudflare_to_the_session(self, monkeypatch):- """`solve_cloudflare` is kept on the fetch params even when it is False, so it must not be sent unasked"""- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)+ async def test_unknown_session_raises(self):server = ScraplingMCPServer()- opened = await server.open_session(session_type="stealthy", solve_cloudflare=True)-- await server.bulk_stealthy_fetch(urls=["https://example.com/1"], session_id=opened.session_id)+ with pytest.raises(ValueError, match="not found"):+ await server.session_fetch(url="https://example.com/1", session_id="nope")- forwarded = self._session_fetch_calls(_FakeStealthySession)[0]- assert "solve_cloudflare" not in forwarded, (- f"Expected solve_cloudflare to be left to the session, got {forwarded}"- )- @pytest.mark.asyncio- async def test_bulk_stealthy_fetch_forwards_an_explicit_solve_cloudflare(self, monkeypatch):- """A caller can still turn Cloudflare solving on for a single request"""- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)- server = ScraplingMCPServer()- opened = await server.open_session(session_type="stealthy")+class TestModeSplitContract:+ """The one-shot vs session split is derived from the library TypedDicts and must stay in sync."""- await server.bulk_stealthy_fetch(- urls=["https://example.com/1"], session_id=opened.session_id, solve_cloudflare=True+ def test_session_fetch_signature_matches_the_derived_fetch_keys(self):+ """session_fetch exposes exactly the stealth per-request keys (plus url/session_id/extraction trio)"""+ params = set(inspect.signature(ScraplingMCPServer.session_fetch).parameters) - {+ "self",+ "url",+ "session_id",+ "extraction_type",+ "css_selector",+ "main_content_only",+ }+ assert params == set(_STEALTH_FETCH_KEYS), (+ f"session_fetch params drifted from _STEALTH_FETCH_KEYS: {params ^ set(_STEALTH_FETCH_KEYS)}")- forwarded = self._session_fetch_calls(_FakeStealthySession)[0]- assert forwarded.get("solve_cloudflare") is True, (- f"Expected the supplied solve_cloudflare to be forwarded, got {forwarded}"+ def test_session_fetch_defaults_match_the_library(self):+ """Each per-request default equals the library config default so the AI sees the real value"""+ defaults = {+ name: p.default+ for name, p in inspect.signature(ScraplingMCPServer.session_fetch).parameters.items()+ if name in _STEALTH_FETCH_KEYS+ }+ library = models_default_values["StealthConfig"]+ for name, value in defaults.items():+ assert value == library[name], f"session_fetch {name} default {value!r} != library {library[name]!r}"++ def test_one_shot_fetch_tools_have_no_session_id(self):+ """The one-shot tools no longer accept session_id"""+ for tool in (+ ScraplingMCPServer.fetch,+ ScraplingMCPServer.bulk_fetch,+ ScraplingMCPServer.stealthy_fetch,+ ScraplingMCPServer.bulk_stealthy_fetch,+ ):+ assert "session_id" not in inspect.signature(tool).parameters, f"{tool.__name__} still takes session_id"++ def test_open_session_holds_no_per_request_params(self):+ """open_session keeps browser-level params only, none of the per-request fetch keys"""+ params = set(inspect.signature(ScraplingMCPServer.open_session).parameters)+ assert params.isdisjoint(_STEALTH_FETCH_KEYS), (+ f"open_session still carries per-request params: {params & set(_STEALTH_FETCH_KEYS)}")- @pytest.mark.asyncio- async def test_screenshot_forwards_only_the_capture_callback(self, monkeypatch):- """The screenshot tool needs the capture callback, and nothing else the caller didn't ask for"""- monkeypatch.setattr("scrapling.core.ai.AsyncDynamicSession", _FakeDynamicSession)- server = ScraplingMCPServer()- opened = await server.open_session(session_type="dynamic")- result = await server.screenshot(url="https://example.com/1", session_id=opened.session_id)+class TestSessionSettingsReceipt:+ """open_session and list_sessions return the session's effective settings."""++ def test_session_settings_extracts_json_safe_fields(self):+ """The helper keeps JSON primitives and drops the rest (callables, structs, sequences)"""+ config = AsyncStealthySession(headless=True)._config+ settings = _session_settings(config)+ assert settings["headless"] is True+ assert settings["timeout"] == 30000+ assert "cookies" not in settings, "non-primitive fields (list) must be dropped"+ assert all(isinstance(v, (str, int, float, bool)) or v is None for v in settings.values()), settings- forwarded = self._session_fetch_calls(_FakeDynamicSession)[0]- assert set(forwarded) == {"page_action"}, f"Expected only the capture callback to be forwarded, got {forwarded}"- assert isinstance(result[0], ImageContent), f"Expected an image content block, got {type(result[0]).__name__}"+ def test_cdp_session_reports_empty_settings(self):+ """A CDP session drives a remote browser, so the local config is not reported as its settings"""+ config = AsyncDynamicSession(cdp_url="ws://127.0.0.1:9222/devtools/browser/x")._config+ assert _session_settings(config) == {}@pytest.mark.asyncio- @pytest.mark.parametrize(- "tool,fake,fake_name",- [- ("bulk_fetch", _FakeDynamicSession, "AsyncDynamicSession"),- ("bulk_stealthy_fetch", _FakeStealthySession, "AsyncStealthySession"),- ],- )- async def test_a_fresh_session_is_built_with_the_same_settings_as_before(self, monkeypatch, tool, fake, fake_name):- """Without a session ID the tools still build a session with the fetcher defaults they always used"""- monkeypatch.setattr(f"scrapling.core.ai.{fake_name}", fake)+ async def test_open_session_and_list_report_the_receipt(self, monkeypatch):+ """open_session returns the receipt and list_sessions reports the same one"""+ monkeypatch.setattr("scrapling.core.ai.AsyncDynamicSession", _FakeDynamicSession)server = ScraplingMCPServer()- await getattr(server, tool)(urls=["https://example.com/1"])+ created = await server.open_session(session_type="dynamic")- real_session_type = AsyncDynamicSession if fake is _FakeDynamicSession else AsyncStealthySession- config = real_session_type(**fake.instances[0].kwargs)._config- expected = {- "wait": 0,- "timeout": 30000,- "google_search": True,- "extra_headers": None,- "disable_resources": False,- "wait_selector": None,- "wait_selector_state": "attached",- "network_idle": False,- }- for name, value in expected.items():- assert getattr(config, name) == value, (- f"Expected {tool} to keep building a session with {name}={value}, got {getattr(config, name)}"- )+ assert created.settings["headless"] is True+ listed = await server.list_sessions()+ assert listed[0].settings == created.settingsdef _png_height(data: bytes) -> int:@@ -840,7 +742,7 @@ class TestMCPServerAuthentication:monkeypatch.delenv(MCP_AUTH_TOKEN_ENV, raising=False)built = ScraplingMCPServer(auth_token=SHARED_KEY)._build_server("0.0.0.0", 8000)- assert len(built._tool_manager.list_tools()) == 10+ assert len(built._tool_manager.list_tools()) == 11def test_http_without_a_token_refuses_to_serve(self, monkeypatch):"""The streamable-http transport requires authentication unless the caller explicitly opts out"""@@ -897,16 +799,39 @@ class TestServerToolRegistration:@pytest.mark.asyncioasync def test_tools_are_listed_with_expected_schemas(self):- """All 10 tools are advertised, and only the screenshot tool skips the structured output schema"""+ """All 11 tools are advertised, and only the screenshot tool skips the structured output schema"""server = ScraplingMCPServer()._build_server("127.0.0.1", 8000)async with Client(server) as client:assert client.instructionstools = {tool.name: tool for tool in (await client.list_tools()).tools}- assert len(tools) == 10+ assert len(tools) == 11assert tools["screenshot"].output_schema is Noneassert all(tool.output_schema is not None for name, tool in tools.items() if name != "screenshot")+ @pytest.mark.asyncio+ async def test_fetch_tools_expose_real_defaults_and_no_session_id(self):+ """The one-shot tools show real defaults in their schema and no longer take session_id"""+ server = ScraplingMCPServer()._build_server("127.0.0.1", 8000)+ async with Client(server) as client:+ tools = {tool.name: tool for tool in (await client.list_tools()).tools}++ for name in ("fetch", "bulk_fetch", "stealthy_fetch", "bulk_stealthy_fetch"):+ props = tools[name].input_schema["properties"]+ assert "session_id" not in props, f"{name} still exposes session_id"+ assert props["timeout"]["default"] == 30000, f"{name} hides the real timeout default"+ assert props["google_search"]["default"] is True++ session_props = tools["session_fetch"].input_schema["properties"]+ assert session_props["timeout"]["default"] == 30000+ assert "solve_cloudflare" in session_props+ assert set(tools["session_fetch"].input_schema["required"]) >= {"url", "session_id"}++ open_props = set(tools["open_session"].input_schema["properties"])+ assert open_props.isdisjoint(_STEALTH_FETCH_KEYS), (+ f"open_session still exposes per-request params: {open_props & set(_STEALTH_FETCH_KEYS)}"+ )+@pytest.mark.asyncioasync def test_server_metadata_and_tool_annotations(self):"""Server card metadata, cache hints, and tool annotations are advertised to clients"""@@ -922,8 +847,17 @@ class TestServerToolRegistration:assert result.ttl_ms == 3_600_000 and result.cache_scope == "public"annotations = {tool.name: tool.annotations for tool in result.tools if tool.annotations is not None}- assert len(annotations) == 10- for name in ("get", "bulk_get", "fetch", "bulk_fetch", "stealthy_fetch", "bulk_stealthy_fetch", "screenshot"):+ assert len(annotations) == 11+ for name in (+ "get",+ "bulk_get",+ "fetch",+ "bulk_fetch",+ "stealthy_fetch",+ "bulk_stealthy_fetch",+ "session_fetch",+ "screenshot",+ ):assert annotations[name].read_only_hint is Trueassert annotations[name].open_world_hint is Truefor name in ("open_session", "close_session"):