Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

feat(mcp): add two tools to open requests sessions and use them

Karim shoair6e6373c99761
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zip
index 9af98ac..e9a2857 100644
Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differ
diff --git a/agent-skill/Scrapling-Skill/references/mcp-server.md b/agent-skill/Scrapling-Skill/references/mcp-server.md
index 33fe2f1..fced8cb 100644
--- a/agent-skill/Scrapling-Skill/references/mcp-server.md
+++ b/agent-skill/Scrapling-Skill/references/mcp-server.md
@@ -1,6 +1,6 @@
# Scrapling MCP Server
-The Scrapling MCP server exposes eleven tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), three levels of scraping capability (plain HTTP, browser-rendered, and stealth/anti-bot bypass), persistent browser session management, and page screenshots returned as real image content blocks. Fetch tools come in two modes: one-shot tools (`fetch`, `bulk_fetch`, `stealthy_fetch`, `bulk_stealthy_fetch`) each launch and close their own browser, while `session_fetch` fetches through a session opened with `open_session`.
+The Scrapling MCP server exposes thirteen tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), three levels of scraping capability (plain HTTP, browser-rendered, and stealth/anti-bot bypass), persistent browser session management, and page screenshots returned as real image content blocks. Fetch tools come in two modes: one-shot tools (`fetch`, `bulk_fetch`, `stealthy_fetch`, `bulk_stealthy_fetch`) each launch and close their own browser, while `session_fetch` and `session_make_request` work through sessions opened with `open_session`/`open_request_session`.
All scraping tools return a `ResponseModel` with fields: `status` (int), `content` (list of strings), `url` (str). The `screenshot` tool returns a list of MCP content blocks: an `ImageContent` (the screenshot bytes) followed by a `TextContent` (the post-redirect URL).
@@ -100,7 +100,7 @@ Concurrent stealth version. Same parameters as `stealthy_fetch` except `url` is
### `open_session` -- Create a persistent browser session
-Opens a browser session that stays alive across multiple `session_fetch` calls, avoiding the overhead of launching a new browser each time. It holds the browser-level configuration only; per-request options are passed to `session_fetch`. Returns a `SessionCreatedModel` with `session_id`, `session_type`, `created_at`, `is_alive`, `settings` (the session's effective configuration for the AI agent; empty for CDP sessions), and `message`.
+Opens a browser session that stays alive across multiple `session_fetch` calls, avoiding the overhead of launching a new browser each time. It holds the browser-level configuration only; per-request options are passed to `session_fetch`. For plain HTTP requests without a browser, use `open_request_session` instead. Returns a `SessionCreatedModel` with `session_id`, `session_type`, `created_at`, `is_alive`, `settings` (the session's effective configuration for the AI agent; empty for CDP sessions), and `message`.
**Key parameters:**
@@ -115,11 +115,21 @@ Opens a browser session that stays alive across multiple `session_fetch` calls,
Plus the other browser-level session parameters (`proxy`, `real_chrome`, `cdp_url`, `locale`, `timezone_id`, `useragent`, `cookies`, `executable_path`, `additional_args`). Per-request options (`timeout`, `wait`, `google_search`, `network_idle`, `disable_resources`, `wait_selector`, `wait_selector_state`, `extra_headers`, `solve_cloudflare`) are not set here; pass them to `session_fetch`.
-One `session_fetch` works with either session type; `solve_cloudflare` only applies to a stealthy session.
+One `session_fetch` works with either browser session type; `solve_cloudflare` only applies to a stealthy session.
-### `session_fetch` -- Fetch through an open session (single URL)
+### `open_request_session` -- Create a persistent HTTP requests session
-Fetches one URL through a session opened with `open_session` (dynamic or stealthy). The session holds the browser-level configuration; every parameter here applies to this request only.
+Opens an HTTP session (no browser) that stays alive across multiple `session_make_request` calls, keeping cookies, connections, and the browser fingerprint between requests. Returns the same `SessionCreatedModel` receipt and shows in `list_sessions` as a `static` session.
+
+| Parameter | Type | Default | Description |
+|---------------|-------------|------------|-------------------------------------------------------------------------------------------------------|
+| `session_id` | str or null | null | Custom ID for the session. If omitted, a random 12-char hex ID is generated. Raises if already in use |
+| `impersonate` | str | `"chrome"` | Browser fingerprint to impersonate on every request |
+| `proxy` | str or null | null | Proxy URL used for every request, e.g. `"http://user:pass@host:port"` |
+
+### `session_fetch` -- Fetch through an open browser session (single URL)
+
+Fetches one URL through a browser session opened with `open_session` (dynamic or stealthy). The session holds the browser-level configuration; every parameter here applies to this request only. Raises on a requests session; use `session_make_request` there instead.
| Parameter | Type | Default | Description |
|-----------------------|---------------------|--------------|---------------------------------------------------------------------------------|
@@ -140,9 +150,13 @@ Fetches one URL through a session opened with `open_session` (dynamic or stealth
| `blocked_domains` | list or null | null | Domain names to block for this request (subdomains matched too) |
| `solve_cloudflare` | bool | false | (Stealthy sessions only) Auto-solve Cloudflare challenges; errors on a dynamic session |
-### `close_session` -- Close a persistent browser session
+### `session_make_request` -- HTTP request through an open requests session
+
+Makes an HTTP request (any method) through a session opened with `open_request_session`, reusing its cookies, connections, and browser fingerprint across calls. Same parameters as `make_request` plus a required `session_id`, minus the session-level `impersonate`, `proxy`, and `proxy_auth`. Raises on a browser session.
+
+### `close_session` -- Close a persistent session
-Closes a session and frees its browser resources. Always close sessions when done.
+Closes a session (browser or requests) and frees its resources. Always close sessions when done.
| Parameter | Type | Default | Description |
|--------------|------|----------|----------------------------------|
@@ -186,6 +200,7 @@ Requires an open browser session. Call `open_session` first, then pass the `sess
| Cloudflare or strong anti-bot protection | `stealthy_fetch` (with `solve_cloudflare=true` for Turnstile) |
| Multiple protected pages | `bulk_stealthy_fetch` |
| Multiple pages from the same site | `open_session` + `session_fetch` per page |
+| Multiple plain HTTP requests to one site | `open_request_session` + `session_make_request` per request |
| Need a screenshot of a page | `open_session` + `screenshot` with `session_id` |
Start with `make_request` (fastest, lowest resource cost). Escalate to `fetch` if content requires JS rendering. Escalate to `stealthy_fetch` only if blocked. For multiple pages from the same site, use a persistent session to avoid browser launch overhead.
diff --git a/docs/ai/mcp-server.md b/docs/ai/mcp-server.md
index 610d79b..aa83feb 100644
--- a/docs/ai/mcp-server.md
+++ b/docs/ai/mcp-server.md
@@ -6,7 +6,7 @@ The **Scrapling MCP Server** is a new feature that brings Scrapling's powerful W
## Features
-The Scrapling MCP Server provides eleven powerful tools for web scraping, split into two modes: one-shot tools that each launch and close their own browser, and session tools that open a browser once and then work through it.
+The Scrapling MCP Server provides thirteen powerful tools for web scraping, split into two modes: one-shot tools that each launch and close their own browser/client, and session tools that open a browser or an HTTP session once and then work through it.
### One-shot tools
@@ -26,11 +26,13 @@ The Scrapling MCP Server provides eleven powerful tools for web scraping, split
#### 🔌 Session Management
- **`open_session`**: Create a persistent browser session (dynamic or stealthy) that stays open across multiple `session_fetch` calls, avoiding the overhead of launching a new browser each time. It holds the browser-level configuration and returns the session's effective `settings` for the AI agent (empty for CDP sessions).
-- **`close_session`**: Close a persistent browser session and free its resources.
-- **`list_sessions`**: List all active browser sessions with their details and `settings`.
+- **`open_request_session`**: Create a persistent HTTP requests session (no browser) used with `session_make_request`, keeping cookies, connections, and the browser fingerprint (`impersonate`) between requests. It returns the same `settings` receipt and shows in `list_sessions` as a `static` session.
+- **`close_session`**: Close a persistent session (browser or requests) and free its resources.
+- **`list_sessions`**: List all active sessions with their details and `settings`.
#### 🎯 Fetching Through a Session
-- **`session_fetch`**: Fetch a single URL through an open session (dynamic or stealthy), carrying the per-request options for that call. This is the session counterpart of `fetch`/`stealthy_fetch`.
+- **`session_fetch`**: Fetch a single URL through an open browser session (dynamic or stealthy), carrying the per-request options for that call. This is the session counterpart of `fetch`/`stealthy_fetch`.
+- **`session_make_request`**: Make an HTTP request with any method through a session opened with `open_request_session`, reusing its cookies, connections, and browser fingerprint. This is the session counterpart of `make_request`.
#### 📸 Screenshots
- **`screenshot`**: Capture a PNG or JPEG screenshot of a page using an open browser session, returned as an image content block the model can actually see (not a base64 string blob). Supports full-page captures, JPEG quality, and the usual readiness controls (`wait`, `wait_selector`, `network_idle`).
@@ -198,7 +200,7 @@ You can also set the `SCRAPLING_EXECUTABLE_PATH` environment variable before sta
```
Open a stealthy browser session on wss://cdp.provider.example/session/abc123, then use it to scrape the product details from https://shop.example.com. Close the session when you're done.
```
-Both session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with `session_fetch` and `screenshot` as usual.
+Both browser session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with `session_fetch` and `screenshot` as usual.
The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser you started yourself with the remote debugging port enabled:
```commandline
@@ -436,12 +438,13 @@ This protection runs automatically on all MCP tool responses. Keep `main_content
### 6. Use Sessions for Multiple Requests
- Use `open_session` to create a persistent browser session when scraping multiple pages, then call `session_fetch` for each page through that session
-- `open_session` holds the browser-level configuration (headless, locale, cookies, stealth toggles, etc.); the per-request options (timeout, wait_selector, network_idle, solve_cloudflare, etc.) are passed to `session_fetch` on each call, with their defaults shown in the tool schema
-- One `session_fetch` works with both session types; `solve_cloudflare` only applies to a stealthy session and raises a clear error on a dynamic one
-- The one-shot tools (`fetch`, `bulk_fetch`, `stealthy_fetch`, `bulk_stealthy_fetch`) never take a session; they always launch and close their own browser
+- For multiple plain HTTP requests, use `open_request_session` instead and call `session_make_request` per request; it keeps cookies, connections, and the browser fingerprint (`impersonate`) across calls without a browser
+- Sessions hold the session-level configuration set when opened (headless, locale, cookies, stealth toggles, etc. for browsers; `impersonate` and `proxy` for requests sessions); the per-request options (timeout, wait_selector, network_idle, solve_cloudflare, etc.) are passed to `session_fetch`/`session_make_request` on each call, with their defaults shown in the tool schemas
+- One `session_fetch` works with both browser session types; `solve_cloudflare` only applies to a stealthy session and raises a clear error on a dynamic one
+- The one-shot tools (`make_request`, `bulk_get`, `fetch`, `bulk_fetch`, `stealthy_fetch`, `bulk_stealthy_fetch`) never take a session
- Always close sessions with `close_session` when done to free resources
- Use `list_sessions` to check which sessions are still active and see the `settings` each was created with (returned for the AI agent; empty for CDP sessions)
-- Pass a custom `session_id` to `open_session` to give sessions meaningful names (e.g. `"search"`, `"checkout"`) instead of the random hex default. `open_session` raises if the chosen ID is already in use, so you can detect collisions up front
+- Pass a custom `session_id` to the open tools to give sessions meaningful names (e.g. `"search"`, `"checkout"`) instead of the random hex default. They raise if the chosen ID is already in use, so you can detect collisions up front
### 7. Capturing Screenshots
- `screenshot` only works through an existing browser session, so call `open_session` first (either `dynamic` or `stealthy` works)
diff --git a/scrapling/core/ai.py b/scrapling/core/ai.py
index ca36c8a..f986108 100644
--- a/scrapling/core/ai.py
+++ b/scrapling/core/ai.py
@@ -43,7 +43,8 @@ from scrapling.core._types import (
SUPPORTED_HTTP_METHODS,
)
-SessionType = Literal["dynamic", "stealthy"]
+SessionType = Literal["dynamic", "stealthy", "static"]
+BrowserSessionType = Literal["dynamic", "stealthy"]
ScreenshotType = Literal["png", "jpeg"]
MCP_EXECUTABLE_PATH_ENV = "SCRAPLING_EXECUTABLE_PATH"
MCP_AUTH_TOKEN_ENV = "SCRAPLING_MCP_AUTH_TOKEN" # nosec B105 - the name of the variable, not a token
@@ -66,8 +67,18 @@ _PLAYWRIGHT_FETCH_KEYS = _typed_dict_keys(PlaywrightFetchParams) - _EXCLUDED_FET
_STEALTH_FETCH_KEYS = _typed_dict_keys(StealthFetchParams) - _EXCLUDED_FETCH_KEYS
-def _session_settings(config: Any) -> Dict[str, Any]:
- """Extract the JSON-safe effective settings of a validated session config struct, for the AI agent."""
+def _session_settings(session: Any) -> Dict[str, Any]:
+ """Extract the JSON-safe effective settings of a session, for the AI agent."""
+ if isinstance(session, FetcherSession):
+ fields = {"stealthy_headers": session._stealth} | {
+ f.removeprefix("_default_"): getattr(session, f)
+ for f in FetcherSession.__slots__
+ if f.startswith("_default")
+ }
+ return {
+ name: value for name, value in fields.items() if isinstance(value, (str, int, float, bool)) or value is None
+ }
+ config = session._config
if config.cdp_url:
return {}
return {
@@ -94,7 +105,7 @@ class SessionInfo(BaseModel):
"""Information about an open browser session."""
session_id: str = Field(description="The unique identifier of the session.")
- session_type: SessionType = Field(description="The type of the session: 'dynamic' or 'stealthy'.")
+ session_type: SessionType = Field(description="The type of the session: 'dynamic', 'stealthy', or 'static'.")
created_at: str = Field(description="ISO timestamp of when the session was created.")
is_alive: bool = Field(description="Whether the session is still alive and usable.")
settings: Dict[str, Any] = Field(
@@ -118,7 +129,7 @@ class SessionClosedModel(BaseModel):
@dataclass
class _SessionEntry:
- session: Any # AsyncDynamicSession | AsyncStealthySession
+ session: Any # AsyncDynamicSession | AsyncStealthySession | FetcherSession
session_type: SessionType
created_at: str = field(default_factory=lambda: datetime.now(timezone.utc).isoformat())
@@ -200,9 +211,31 @@ class ScraplingMCPServer:
)
return entry
+ def _new_session_id(self, session_id: Optional[str]) -> str:
+ """Generate a session ID when none is given, and reject duplicates."""
+ session_id = session_id or uuid4().hex[:12]
+ if session_id in self._sessions:
+ raise ValueError(
+ f"Session '{session_id}' already exists. Use a different ID or close the existing session first."
+ )
+ return session_id
+
+ def _register_session(self, session_id: str, session: Any, session_type: SessionType) -> SessionCreatedModel:
+ """Store a started session and build its creation receipt."""
+ entry = _SessionEntry(session=session, session_type=session_type)
+ self._sessions[session_id] = entry
+ return SessionCreatedModel(
+ session_id=session_id,
+ session_type=session_type,
+ created_at=entry.created_at,
+ is_alive=True,
+ settings=_session_settings(session),
+ message=f"Session '{session_id}' ({session_type}) created successfully.",
+ )
+
async def open_session(
self,
- session_type: SessionType,
+ session_type: BrowserSessionType,
session_id: Optional[str] = None,
headless: bool = True,
real_chrome: bool = False,
@@ -239,12 +272,7 @@ class ScraplingMCPServer:
:param allow_webgl: (Stealthy only) Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param additional_args: (Stealthy only) Additional arguments to be passed to Playwright's context as additional settings.
"""
- session_id = session_id or uuid4().hex[:12]
- if session_id in self._sessions:
- raise ValueError(
- f"Session '{session_id}' already exists. Use a different ID or close the existing session first."
- )
-
+ session_id = self._new_session_id(session_id)
common_kwargs: Dict[str, Any] = dict(
proxy=proxy,
locale=locale,
@@ -271,24 +299,33 @@ class ScraplingMCPServer:
session = AsyncDynamicSession(**common_kwargs)
await session.start()
+ return self._register_session(session_id, session, session_type)
- entry = _SessionEntry(session=session, session_type=session_type)
- self._sessions[session_id] = entry
+ async def open_request_session(
+ self,
+ session_id: Optional[str] = None,
+ impersonate: ImpersonateType = "chrome",
+ proxy: Optional[str] = None,
+ ) -> SessionCreatedModel:
+ """Open a persistent HTTP requests session (no browser) that can be reused across multiple
+ `session_make_request` calls, keeping cookies, connections, and the browser fingerprint between requests.
+ The session holds this configuration only; per-request options are passed to `session_make_request` on each call.
+ It shares `close_session`/`list_sessions` with the browser sessions and shows there as a 'static' session.
- return SessionCreatedModel(
- session_id=session_id,
- session_type=session_type,
- created_at=entry.created_at,
- is_alive=True,
- settings=_session_settings(session._config),
- message=f"Session '{session_id}' ({session_type}) created successfully.",
- )
+ :param session_id: Optional custom session ID. If not provided, a random 12-character hex ID will be generated. Useful for naming sessions for easier management.
+ :param impersonate: Browser version to impersonate its fingerprint on every request. It's using the latest chrome version by default.
+ :param proxy: The proxy URL used for every request in this session. Format: "http://username:password@localhost:8030".
+ """
+ session_id = self._new_session_id(session_id)
+ session = FetcherSession(impersonate=impersonate, proxy=proxy)
+ await session.__aenter__()
+ return self._register_session(session_id, session, "static")
async def close_session(
self,
session_id: str,
) -> SessionClosedModel:
- """Close a persistent browser session and free its resources.
+ """Close a persistent session and free its resources.
:param session_id: The unique identifier of the session to close. Use list_sessions to see active sessions.
"""
@@ -296,21 +333,24 @@ class ScraplingMCPServer:
if entry is None:
raise ValueError(f"Session '{session_id}' not found. Use list_sessions to see active sessions.")
- await entry.session.close()
+ if entry.session_type == "static":
+ await entry.session.__aexit__(None, None, None)
+ else:
+ await entry.session.close()
return SessionClosedModel(
session_id=session_id,
message=f"Session '{session_id}' closed successfully.",
)
async def list_sessions(self) -> List[SessionInfo]:
- """List all active browser sessions with their details, including the effective settings each one was created with."""
+ """List all active sessions with their details, including the effective settings each one was created with."""
return [
SessionInfo(
session_id=sid,
session_type=entry.session_type,
created_at=entry.created_at,
is_alive=entry.session._is_alive,
- settings=_session_settings(entry.session._config),
+ settings=_session_settings(entry.session),
)
for sid, entry in self._sessions.items()
]
@@ -346,6 +386,11 @@ class ScraplingMCPServer:
raise ValueError("'quality' is only valid when 'image_type' is 'jpeg'.")
entry = self._get_session(session_id, expected_type=None)
+ if entry.session_type == "static":
+ raise ValueError(
+ f"Session '{session_id}' is a 'static' session, so it can't take screenshots. "
+ f"Open a 'dynamic' or 'stealthy' session for that."
+ )
screenshot_kwargs: Dict[str, Any] = {"type": image_type, "full_page": full_page}
if quality is not None:
@@ -910,6 +955,10 @@ class ScraplingMCPServer:
:param solve_cloudflare: (Stealthy sessions only) Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response.
"""
entry = self._get_session(session_id, expected_type=None)
+ if entry.session_type == "static":
+ raise ValueError(
+ f"Session '{session_id}' is a 'static' session. Use `session_make_request` with it instead."
+ )
if solve_cloudflare and entry.session_type != "stealthy":
raise ValueError(
f"Session '{session_id}' is a '{entry.session_type}' session, so it can't solve Cloudflare "
@@ -935,6 +984,77 @@ class ScraplingMCPServer:
)
return _translate_response(page, extraction_type, css_selector, main_content_only)
+ async def session_make_request(
+ self,
+ url: str,
+ session_id: str,
+ method: SUPPORTED_HTTP_METHODS = "GET",
+ extraction_type: extraction_types = "markdown",
+ css_selector: Optional[str] = None,
+ main_content_only: bool = True,
+ params: Optional[Dict] = None,
+ data: Optional[Dict[str, str] | str] = None,
+ json: Optional[Dict | List] = None,
+ headers: Optional[Mapping[str, Optional[str]]] = None,
+ cookies: Optional[Dict[str, str]] = None,
+ timeout: Optional[int | float] = 30,
+ follow_redirects: FollowRedirects = "safe",
+ max_redirects: int = 30,
+ retries: Optional[int] = 3,
+ retry_delay: Optional[int] = 1,
+ auth: Optional[Dict[str, str]] = None,
+ verify: Optional[bool] = True,
+ http3: Optional[bool] = False,
+ stealthy_headers: Optional[bool] = True,
+ ) -> ResponseModel:
+ """Make an HTTP request with any method (GET, POST, PUT, DELETE) through a requests session previously opened
+ with `open_request_session`, keeping its cookies, connections, and browser fingerprint across calls. The session
+ holds the impersonation and proxy configuration; every option here applies to this request only, with the defaults shown.
+
+ :param url: The URL to request.
+ :param session_id: ID of an open requests session created with `open_request_session`.
+ :param method: The HTTP method to use: "GET" (default), "POST", "PUT", or "DELETE".
+ :param extraction_type: The type of content to extract from the page: "markdown", "html", or "text".
+ :param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page.
+ :param main_content_only: Whether to extract only the main content of the page. The main content here is the data inside the `<body>` tag.
+ :param params: Query string parameters for the request.
+ :param data: Form data for the request body. Used with "POST", "PUT", and "DELETE" only.
+ :param json: A JSON-serializable object for the request body. Used with "POST", "PUT", and "DELETE" only.
+ :param headers: Headers to include in the request.
+ :param cookies: Cookies to use in the request.
+ :param timeout: Number of seconds to wait before timing out.
+ :param follow_redirects: Whether to follow redirects. Defaults to "safe", which follows redirects but rejects those targeting internal/private IPs (SSRF protection).
+ Pass True to follow all redirects without restriction.
+ :param max_redirects: Maximum number of redirects. Use -1 for unlimited.
+ :param retries: Number of retry attempts.
+ :param retry_delay: Number of seconds to wait between retry attempts.
+ :param auth: HTTP basic auth in dictionary format with `username` and `password` keys.
+ :param verify: Whether to verify HTTPS certificates.
+ :param http3: Whether to use HTTP3. It might be problematic if used it with `impersonate`.
+ :param stealthy_headers: If enabled (default), it creates and adds real browser headers. It also sets a Google referer header.
+ """
+ entry = self._get_session(session_id, expected_type="static")
+
+ request_kwargs: Dict[str, Any] = dict(
+ auth=_normalize_credentials(auth),
+ http3=http3,
+ verify=verify,
+ params=params,
+ headers=headers,
+ cookies=cookies,
+ timeout=timeout,
+ retries=retries,
+ retry_delay=retry_delay,
+ max_redirects=max_redirects,
+ follow_redirects=follow_redirects,
+ stealthy_headers=stealthy_headers,
+ )
+ if method != "GET":
+ request_kwargs.update(data=data, json=json)
+
+ page = await getattr(entry.session._client, method.lower())(url, **request_kwargs)
+ return _translate_response(page, extraction_type, css_selector, main_content_only)
+
@staticmethod
def _transport_security(allowed_hosts: Sequence[str]) -> Optional[TransportSecuritySettings]:
"""Build the DNS-rebinding protection settings for the streamable-http transport."""
@@ -960,15 +1080,16 @@ class ScraplingMCPServer:
],
"cache_hints": {"tools/list": CacheHint(ttl_ms=3_600_000, scope="public")},
"instructions": """Follow these instructions precisely:
-1. When the `open_session` tool is used, make sure to close the session with `close_session` after you finish, and use `list_sessions` if you lose track of the open sessions or their effective settings.
+1. When the `open_session` or `open_request_session` tools are used, make sure to close the session with `close_session` after you finish, and use `list_sessions` if you lose track of the open sessions or their effective settings.
2. If the user didn't specify which tool to use, start with the `make_request` tool (a plain HTTP request, defaulting to GET; set `method` for POST/PUT/DELETE), then escalate. The `make_request` tool and `bulk_get` (its GET-only bulk version) are suitable only for low-to-mid protection levels.
For high-protection levels or websites that require JS loading, use the other tools directly.
3. For all tools, if the `css_selector` resolves to more than one element, all the elements will be returned.
4. For all fetch tools, the `extraction_type` parameter controls the format of the returned content: "markdown" (default) converts the page content to Markdown, "html" returns the raw HTML, and "text" returns the text content of the page.
5. For all fetch tools, `main_content_only` is enabled by default and returns only the content inside the page's `<body>` tag. Pass `main_content_only=False` when you need the full page instead.
-6. If the task consists of multiple sequential requests to the same website, open a session with `open_session` once, then call `session_fetch` per page to be more efficient.
-7. Sessions hold the browser-level configuration set at `open_session`, while `session_fetch` carries the per-request options and applies them on each call with the defaults shown in its schema.
- The one-shot tools (`fetch`, `bulk_fetch`, `stealthy_fetch`, `bulk_stealthy_fetch`) never touch sessions and always launch their own browser.
+6. If the task consists of multiple sequential requests to the same website, open a session once, then fetch through it to be more efficient:
+ `open_session` + `session_fetch` per page for browsers, or `open_request_session` + `session_make_request` per request for plain HTTP.
+7. Sessions hold the session-level configuration set when opened, while `session_fetch`/`session_make_request` carry the per-request options and apply them on each call with the defaults shown in their schemas.
+ The one-shot tools (`make_request`, `bulk_get`, `fetch`, `bulk_fetch`, `stealthy_fetch`, `bulk_stealthy_fetch`) never touch sessions.
8. If you are making multiple parallel one-shot requests, use the bulk version of the tool to be more efficient.
9. If you are crawling/browsing a website, be more efficient by using the `css_selector` parameter to only access the parts you are interested in and save money/time. Example: use the `a` selector to extract the urls right away.
10. The user can pass a CDP URL to connect to a remote browser session through the `open_session` tool, then use it with the session tools.
@@ -984,6 +1105,12 @@ class ScraplingMCPServer:
server.add_tool(
self.open_session, title="open_session", structured_output=True, annotations=_SESSION_TOOL_ANNOTATIONS
)
+ server.add_tool(
+ self.open_request_session,
+ title="open_request_session",
+ structured_output=True,
+ annotations=_SESSION_TOOL_ANNOTATIONS,
+ )
server.add_tool(
self.close_session, title="close_session", structured_output=True, annotations=_SESSION_TOOL_ANNOTATIONS
)
@@ -1035,7 +1162,7 @@ class ScraplingMCPServer:
structured_output=True,
annotations=_FETCH_TOOL_ANNOTATIONS,
)
- # Session-scoped fetch tool
+ # Session-scoped fetch tools
server.add_tool(
self.session_fetch,
title="session_fetch",
@@ -1043,6 +1170,13 @@ class ScraplingMCPServer:
structured_output=True,
annotations=_FETCH_TOOL_ANNOTATIONS,
)
+ server.add_tool(
+ self.session_make_request,
+ title="session_make_request",
+ description=self.session_make_request.__doc__,
+ structured_output=True,
+ annotations=_FETCH_TOOL_ANNOTATIONS,
+ )
# Screenshot tool (returns image + url content blocks, not structured JSON)
server.add_tool(
self.screenshot,
diff --git a/tests/ai/test_ai_mcp.py b/tests/ai/test_ai_mcp.py
index 4542968..adc3c99 100644
--- a/tests/ai/test_ai_mcp.py
+++ b/tests/ai/test_ai_mcp.py
@@ -30,7 +30,7 @@ from scrapling.core.ai import (
_translate_response,
)
from scrapling.engines._browsers._validators import PlaywrightConfig, StealthConfig, models_default_values, validate
-from scrapling.fetchers import AsyncDynamicSession, AsyncStealthySession
+from scrapling.fetchers import AsyncDynamicSession, AsyncStealthySession, FetcherSession
def test_translate_response_strips_control_characters():
@@ -296,6 +296,85 @@ class TestSessionManagement:
await server.close_session("dupe")
+class TestStaticSessionManagement:
+ """Test persistent requests (HTTP) session management"""
+
+ @pytest.fixture
+ def server(self):
+ return ScraplingMCPServer()
+
+ @pytest.mark.asyncio
+ async def test_static_session_lifecycle(self, server, httpbin):
+ """Open a requests session, make GET and POST requests through it, then close it"""
+ created = await server.open_request_session(session_id="st")
+ assert isinstance(created, SessionCreatedModel)
+ assert created.session_type == "static"
+ assert created.is_alive is True
+ assert created.settings["impersonate"] == "chrome"
+
+ response = await server.session_make_request(url=f"{httpbin.url}/html", session_id="st")
+ assert isinstance(response, ResponseModel)
+ assert response.status == 200
+
+ posted = await server.session_make_request(
+ url=f"{httpbin.url}/post", session_id="st", method="POST", json={"key": "value"}, extraction_type="text"
+ )
+ assert posted.status == 200
+
+ listed = await server.list_sessions()
+ assert listed[0].session_type == "static"
+ assert listed[0].settings == created.settings
+
+ session = server._sessions["st"].session
+ closed = await server.close_session("st")
+ assert closed.session_id == "st"
+ assert session._is_alive is False
+
+ @pytest.mark.asyncio
+ async def test_static_session_keeps_cookies(self, server, httpbin):
+ """Cookies set by one request are sent with the next request of the same session"""
+ await server.open_request_session(session_id="jar")
+ await server.session_make_request(
+ url=f"{httpbin.url}/cookies/set/test/value",
+ session_id="jar",
+ follow_redirects=True,
+ extraction_type="text",
+ )
+ response = await server.session_make_request(
+ url=f"{httpbin.url}/cookies", session_id="jar", extraction_type="text"
+ )
+ assert "test" in "".join(response.content)
+ await server.close_session("jar")
+
+ @pytest.mark.asyncio
+ async def test_session_ids_are_shared_across_both_open_tools(self, server, monkeypatch):
+ """A requests session and a browser session can't share the same ID"""
+ monkeypatch.setattr("scrapling.core.ai.AsyncDynamicSession", _FakeDynamicSession)
+ await server.open_request_session(session_id="shared")
+ with pytest.raises(ValueError, match="already exists"):
+ await server.open_session(session_type="dynamic", session_id="shared")
+ await server.close_session("shared")
+
+ @pytest.mark.asyncio
+ async def test_session_make_request_requires_a_static_session(self, server, monkeypatch):
+ """session_make_request rejects browser sessions"""
+ monkeypatch.setattr("scrapling.core.ai.AsyncDynamicSession", _FakeDynamicSession)
+ await server.open_session(session_type="dynamic", session_id="browser")
+ with pytest.raises(ValueError, match="requires a 'static' session"):
+ await server.session_make_request(url="https://example.com", session_id="browser")
+ await server.close_session("browser")
+
+ @pytest.mark.asyncio
+ async def test_session_fetch_and_screenshot_reject_static_sessions(self, server):
+ """The browser session tools refuse a static session with a clear error"""
+ await server.open_request_session(session_id="st2")
+ with pytest.raises(ValueError, match="session_make_request"):
+ await server.session_fetch(url="https://example.com", session_id="st2")
+ with pytest.raises(ValueError, match="can't take screenshots"):
+ await server.screenshot(url="https://example.com", session_id="st2")
+ await server.close_session("st2")
+
+
class TestExecutablePath:
"""Test custom browser executable path plumbing in the MCP browser tools"""
@@ -547,8 +626,7 @@ class TestSessionSettingsReceipt:
def test_session_settings_extracts_json_safe_fields(self):
"""The helper keeps JSON primitives and drops the rest (callables, structs, sequences)"""
- config = AsyncStealthySession(headless=True)._config
- settings = _session_settings(config)
+ settings = _session_settings(AsyncStealthySession(headless=True))
assert settings["headless"] is True
assert settings["timeout"] == 30000
assert "cookies" not in settings, "non-primitive fields (list) must be dropped"
@@ -556,8 +634,17 @@ class TestSessionSettingsReceipt:
def test_cdp_session_reports_empty_settings(self):
"""A CDP session drives a remote browser, so the local config is not reported as its settings"""
- config = AsyncDynamicSession(cdp_url="ws://127.0.0.1:9222/devtools/browser/x")._config
- assert _session_settings(config) == {}
+ assert _session_settings(AsyncDynamicSession(cdp_url="ws://127.0.0.1:9222/devtools/browser/x")) == {}
+
+ def test_static_session_settings_extracts_json_safe_fields(self):
+ """A static session reports its HTTP defaults (impersonate, proxy, timeout, ...)"""
+ settings = _session_settings(FetcherSession(impersonate="chrome", proxy=None))
+ assert settings["impersonate"] == "chrome"
+ assert settings["proxy"] is None
+ assert settings["timeout"] == 30
+ assert settings["stealthy_headers"] is True
+ assert "headers" not in settings, "non-primitive fields (dict) must be dropped"
+ assert all(isinstance(v, (str, int, float, bool)) or v is None for v in settings.values()), settings
@pytest.mark.asyncio
async def test_open_session_and_list_report_the_receipt(self, monkeypatch):
@@ -768,7 +855,7 @@ class TestMCPServerAuthentication:
monkeypatch.delenv(MCP_AUTH_TOKEN_ENV, raising=False)
built = ScraplingMCPServer(auth_token=SHARED_KEY)._build_server("0.0.0.0", 8000)
- assert len(built._tool_manager.list_tools()) == 11
+ assert len(built._tool_manager.list_tools()) == 13
def test_http_without_a_token_refuses_to_serve(self, monkeypatch):
"""The streamable-http transport requires authentication unless the caller explicitly opts out"""
@@ -825,13 +912,13 @@ class TestServerToolRegistration:
@pytest.mark.asyncio
async def test_tools_are_listed_with_expected_schemas(self):
- """All 11 tools are advertised, and only the screenshot tool skips the structured output schema"""
+ """All 13 tools are advertised, and only the screenshot tool skips the structured output schema"""
server = ScraplingMCPServer()._build_server("127.0.0.1", 8000)
async with Client(server) as client:
assert client.instructions
tools = {tool.name: tool for tool in (await client.list_tools()).tools}
- assert len(tools) == 11
+ assert len(tools) == 13
assert tools["screenshot"].output_schema is None
assert all(tool.output_schema is not None for name, tool in tools.items() if name != "screenshot")
@@ -853,6 +940,12 @@ class TestServerToolRegistration:
assert "data" in request_props and "json" in request_props
assert "method" not in tools["bulk_get"].input_schema["properties"]
+ static_props = tools["session_make_request"].input_schema["properties"]
+ assert static_props["method"]["default"] == "GET"
+ assert "proxy" not in static_props and "impersonate" not in static_props, "session-level params leaked"
+ assert set(tools["session_make_request"].input_schema["required"]) >= {"url", "session_id"}
+ assert set(tools["open_request_session"].input_schema["properties"]) == {"session_id", "impersonate", "proxy"}
+
session_props = tools["session_fetch"].input_schema["properties"]
assert session_props["timeout"]["default"] == 30000
assert "solve_cloudflare" in session_props
@@ -878,7 +971,7 @@ class TestServerToolRegistration:
assert result.ttl_ms == 3_600_000 and result.cache_scope == "public"
annotations = {tool.name: tool.annotations for tool in result.tools if tool.annotations is not None}
- assert len(annotations) == 11
+ assert len(annotations) == 13
for name in (
"make_request",
"bulk_get",
@@ -887,11 +980,12 @@ class TestServerToolRegistration:
"stealthy_fetch",
"bulk_stealthy_fetch",
"session_fetch",
+ "session_make_request",
"screenshot",
):
assert annotations[name].read_only_hint is True
assert annotations[name].open_world_hint is True
- for name in ("open_session", "close_session"):
+ for name in ("open_session", "open_request_session", "close_session"):
assert annotations[name].read_only_hint is False
assert annotations[name].destructive_hint is False
assert annotations[name].open_world_hint is True