Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

refactor(mcp)!: Remove all bulk tools to lower tools definition tokens and more optimizations

Karim shoair84ddced8b2b3
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zip
index f644b47..7d7b955 100644
Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differ
diff --git a/agent-skill/Scrapling-Skill/SKILL.md b/agent-skill/Scrapling-Skill/SKILL.md
index 9469edd..2df07c8 100644
--- a/agent-skill/Scrapling-Skill/SKILL.md
+++ b/agent-skill/Scrapling-Skill/SKILL.md
@@ -255,7 +255,7 @@ data = page.css('.quote .text::text').getall()
### Shadow DOM
-Use `pierce_shadow=True` with dynamic or stealth Python fetchers and sessions to include open shadow roots. It defaults to `False` and can be overridden per request. For MCP, pass `pierce_shadow=true` to `browser_fetch_once`, `browser_fetch_many_once`, or each `browser_fetch` call.
+Use `pierce_shadow=True` with dynamic or stealth Python fetchers and sessions to include open shadow roots. It defaults to `False` and can be overridden per request. For MCP, pass `pierce_shadow=true` to `browser_fetch_once` or each `browser_fetch` call.
Both browser CLI commands accept `--pierce-shadow` and `--no-pierce-shadow`:
diff --git a/agent-skill/Scrapling-Skill/references/mcp-server.md b/agent-skill/Scrapling-Skill/references/mcp-server.md
index 697ae78..c0e1569 100644
--- a/agent-skill/Scrapling-Skill/references/mcp-server.md
+++ b/agent-skill/Scrapling-Skill/references/mcp-server.md
@@ -1,19 +1,19 @@
# Scrapling MCP Server
-The Scrapling MCP server exposes sixteen tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), plain HTTP requests and stealth browser rendering with anti-bot bypass, persistent browser session management, mouse actions, batch field filling, keyboard shortcuts, page waits, custom JavaScript, network history, and page screenshots returned as real image content blocks. Fetch tools come in two modes: one-shot tools (`browser_fetch_once`, `browser_fetch_many_once`) each launch and close their own browser, while `browser_fetch` and `session_make_request` work through sessions opened with `browser_open`/`open_request_session`.
+The Scrapling MCP server exposes fourteen tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), plain HTTP requests and stealth browser rendering with anti-bot bypass, persistent browser session management, mouse actions, batch field filling, keyboard shortcuts, page waits, custom JavaScript, network history, and page screenshots returned as real image content blocks. Every fetch takes one URL. One-shot tools (`make_request`, `browser_fetch_once`) close their own client or browser after the call. Use sessions opened with `browser_open` or `open_request_session` for related requests, browser actions, or network history. Browser session calls reuse one tab, so fetch pages and run actions in order. Close each persistent session when done.
-Fetch and HTTP request tools return a `ResponseModel` with fields: `status` (int), `content` (list of strings), `url` (str). Bulk tools return a list of these responses. The `browser_screenshot` tool returns a list of MCP content blocks: an `ImageContent` (the screenshot bytes) followed by a `TextContent` (the current page URL). `browser_snapshot` and `browser_actions` return plain text. Both network tools return structured JSON; the request list supports pagination. `browser_evaluate` returns compact JSON in one plain text block.
+Fetch and HTTP request tools return a `ResponseModel` with fields: `status` (int), `content` (list of strings), `url` (str). The `browser_screenshot` tool returns a list of MCP content blocks: an `ImageContent` (the screenshot bytes) followed by a `TextContent` (the current page URL). `browser_snapshot` and `browser_actions` return plain text. Both network tools return structured JSON; the request list supports pagination. `browser_evaluate` returns compact JSON in one plain text block.
## Shadow DOM
-Set `pierce_shadow=true` on `browser_fetch_once`, `browser_fetch_many_once`, or `browser_fetch` to include open Shadow DOM content. It defaults to `false`. For persistent sessions, pass it on each `browser_fetch` call. See [Shadow DOM](fetching/dynamic.md#shadow-dom) for selector examples and limits.
+Set `pierce_shadow=true` on `browser_fetch_once` or each `browser_fetch` call to include open Shadow DOM content. It defaults to `false`. See [Shadow DOM](fetching/dynamic.md#shadow-dom) for selector examples and limits.
## One-shot tools
### `make_request` -- HTTP request, any method (single URL)
-Fast HTTP request with browser fingerprint impersonation (TLS, headers). Supports GET (default), POST, PUT, and DELETE via the `method` parameter. Suitable for static pages with no/low bot protection.
+Fast HTTP request with browser fingerprint impersonation (TLS, headers). Supports GET (default), POST, PUT, and DELETE via the `method` parameter. Suitable for static pages with no/low bot protection. Closes its own HTTP client after the call; use `session_make_request` for a persistent session.
**Key parameters:**
@@ -31,7 +31,7 @@ Fast HTTP request with browser fingerprint impersonation (TLS, headers). Support
| `proxy_auth` | dict or null | null | `{"username": "...", "password": "..."}` |
| `auth` | dict or null | null | HTTP basic auth, same format as proxy_auth |
| `timeout` | number | 30 | Seconds before timeout |
-| `retries` | int | 3 | Retry attempts on failure |
+| `retries` | int | 3 | Maximum attempts, including the first |
| `retry_delay` | int | 1 | Seconds between retries |
| `stealthy_headers` | bool | true | Generate realistic browser headers and Google referer |
| `http3` | bool | false | Use HTTP/3 (may conflict with `impersonate`) |
@@ -42,10 +42,6 @@ Fast HTTP request with browser fingerprint impersonation (TLS, headers). Support
| `params` | dict or null | null | Query string parameters |
| `verify` | bool | true | Verify HTTPS certificates |
-### `bulk_get` -- HTTP GET request (multiple URLs)
-
-Async concurrent GET-only version of `make_request`. Same parameters except `url` is replaced by `urls` (list of strings) and there are no `method`/`data`/`json` parameters. All URLs are fetched in parallel. Returns a list of `ResponseModel`.
-
### `browser_fetch_once` -- Stealth browser fetch (single URL)
Uses a stealthy Chromium browser through Patchright for JavaScript rendering, fingerprint spoofing, and anti-bot bypass. Supports dynamic/SPA pages and sites with Cloudflare Turnstile/Interstitial or other strong protections.
@@ -81,11 +77,7 @@ Uses a stealthy Chromium browser through Patchright for JavaScript rendering, fi
| `allow_webgl` | bool | true | Keep WebGL enabled; disabling it can trigger bot detection |
| `additional_args` | dict or null | null | Extra Playwright context args (overrides Scrapling defaults) |
-This one-shot tool creates and closes its own stealthy browser session. To fetch through a persistent session, use `browser_fetch`.
-
-### `browser_fetch_many_once` -- Stealth browser fetch (multiple URLs)
-
-Concurrent stealth browser version of `browser_fetch_once`. Same parameters except `url` is replaced by `urls` (list of strings). Each URL opens in a separate browser tab. Returns a list of `ResponseModel`.
+This one-shot tool creates and closes its own stealthy browser session. For related fetches, later browser actions, or network history, use `browser_open` and `browser_fetch`.
## Session tools
@@ -121,7 +113,7 @@ Opens an HTTP session (no browser) that stays alive across multiple `session_mak
### `browser_fetch` -- Fetch through an open browser session (single URL)
-Fetches one URL through a stealthy browser session opened with `browser_open`. The session holds the browser-level configuration; every parameter here applies to this request only. Raises on a requests session; use `session_make_request` there instead.
+Fetches one URL through a stealthy Chromium session opened with `browser_open`, with JavaScript rendering, fingerprint spoofing, and Cloudflare Turnstile/Interstitial bypass. The session holds the browser-level configuration; every parameter here applies to this request only. Raises on a requests session; use `session_make_request` there instead.
| Parameter | Type | Default | Description |
|-----------------------|--------------|--------------|----------------------------------------------------------------------------------------|
@@ -335,14 +327,11 @@ The page stays reserved during capture and is released after success, failure, o
| Scenario | Tool |
|------------------------------------------|---------------------------------------------------------------|
-| Static page, no bot protection | `make_request` |
-| Multiple static pages | `bulk_get` |
-| JavaScript-rendered / SPA page | `browser_fetch_once` |
-| Multiple JS-rendered pages | `browser_fetch_many_once` |
-| Cloudflare or strong anti-bot protection | `browser_fetch_once` (with `solve_cloudflare=true` for Turnstile) |
-| Multiple protected pages | `browser_fetch_many_once` |
-| Multiple pages from the same site | `browser_open` + `browser_fetch` per page |
-| Multiple plain HTTP requests to one site | `open_request_session` + `session_make_request` per request |
+| Single static page or API request | `make_request` |
+| Single JavaScript-rendered / SPA page | `browser_fetch_once` |
+| Single page with Cloudflare protection | `browser_fetch_once` with `solve_cloudflare=true` |
+| Related HTTP requests | `open_request_session` + `session_make_request` per URL |
+| Browser automation or network history | `browser_open` + `browser_fetch` per URL |
| Capture the current page or an element | `browser_screenshot` after `browser_open` + `browser_fetch` |
| Read the current page's AI ARIA snapshot | `browser_snapshot` with `session_id` |
| Chain mouse, field, keyboard, or wait actions | `browser_actions` with `session_id` |
@@ -350,7 +339,7 @@ The page stays reserved during capture and is released after success, failure, o
| Find API calls, HTTP errors, or redirects | `browser_network_requests` with `session_id` |
| Read recorded request headers or body data | `browser_network_request` with `session_id` and `request_id` |
-Start with `make_request` (fastest, lowest resource cost). Use `browser_fetch_once` when JavaScript rendering is needed or HTTP requests are blocked. Enable `solve_cloudflare` for Cloudflare challenges. For multiple pages from the same site, use a persistent session to avoid browser launch overhead.
+Start with `make_request` for a standalone HTTP request, then use `browser_fetch_once` if JavaScript rendering is needed or HTTP requests are blocked. Enable `solve_cloudflare` for Cloudflare challenges. Use persistent sessions for related requests, browser actions, screenshots, or network history. Reuse the session and fetch URLs in order. Always call `close_session` when done with a persistent session.
## Content extraction tips
@@ -373,7 +362,7 @@ Keep `main_content_only=true` for maximum protection.
## Ad blocking
-All browser-based tools (`browser_fetch_once`, `browser_fetch_many_once`) and persistent sessions (`browser_open`) automatically block requests to ~3,500 known ad and tracker domains. This is always enabled in the MCP server to save tokens and speed up page loads. No configuration needed.
+The `browser_fetch_once` tool and browser sessions opened with `browser_open` automatically block requests to ~3,500 known ad and tracker domains. This is always enabled in the MCP server to save tokens and speed up page loads. No configuration needed.
## Setup
@@ -409,7 +398,7 @@ docker run -p 8000:8000 -e SCRAPLING_MCP_AUTH_TOKEN="<your-token>" pyd4vinci/scr
## Custom browser executable
-Browser-based tools (`browser_fetch_once`, `browser_fetch_many_once`, and `browser_open`) can use a custom Chromium-compatible browser executable instead of the bundled Chromium. This is useful for custom browser builds or lightweight browser engines.
+The `browser_fetch_once` and `browser_open` tools can use a custom Chromium-compatible browser executable instead of the bundled Chromium. This is useful for custom browser builds or lightweight browser engines.
To configure it once for the whole MCP server, pass the executable path when starting the server:
@@ -433,7 +422,7 @@ In a Claude Desktop configuration, add the option to the server arguments:
}
```
-You can also set the `SCRAPLING_EXECUTABLE_PATH` environment variable before starting the server. Tool calls can still pass `executable_path` directly when a single request or session needs a different browser executable. The `scrapling extract fetch` and `scrapling extract stealthy-fetch` CLI commands support the same `--executable-path` option and environment variable fallback.
+You can also set the `SCRAPLING_EXECUTABLE_PATH` environment variable before starting the server. Pass `executable_path` to `browser_fetch_once` or `browser_open` when a single fetch or session needs a different browser executable. The `scrapling extract fetch` and `scrapling extract stealthy-fetch` CLI commands support the same `--executable-path` option and environment variable fallback.
The MCP server name when registering with a client is `ScraplingServer`. The command is the path to the `scrapling-mcp` binary with no arguments (or the `scrapling` binary with `mcp` as the argument on versions before 0.4.13).
diff --git a/docs/ai/mcp-server.md b/docs/ai/mcp-server.md
index 01b87c9..560d60c 100644
--- a/docs/ai/mcp-server.md
+++ b/docs/ai/mcp-server.md
@@ -6,18 +6,15 @@ The **Scrapling MCP Server** is a new feature that brings Scrapling's powerful W
## Features
-The Scrapling MCP Server provides sixteen tools for web scraping and browser interaction, split into two modes: one-shot tools that each launch and close their own browser/client, and session tools that open a browser or an HTTP session once and then work through it.
+The Scrapling MCP Server provides fourteen tools for web scraping and browser interaction. One-shot tools fetch a single URL and close their own browser or client. Session tools keep cookies and settings across calls, so the AI can continue browser actions on the same tab and inspect network history.
### One-shot tools
#### 🚀 Basic HTTP Scraping
- **`make_request`**: Fast HTTP requests with any method (GET, POST, PUT, DELETE) and browser fingerprint impersonation, generating real browser headers matching the TLS version, HTTP/3, and more!
-- **`bulk_get`**: An async GET-only version of the above tool that allows scraping of multiple URLs at the same time!
#### đź”’ Stealth Browser Scraping
-
- **`browser_fetch_once`**: Render JavaScript and scrape dynamic pages with our Stealthy browser. Supports fingerprint spoofing, Cloudflare Turnstile/Interstitial bypass, and control over the browser and request!
-- **`browser_fetch_many_once`**: Scrape multiple URLs at the same time in separate browser tabs, with the same stealth features and protection controls!
### Session tools
@@ -28,8 +25,8 @@ The Scrapling MCP Server provides sixteen tools for web scraping and browser int
- **`list_sessions`**: List all active sessions with their details and `settings`.
#### 🎯 Fetching Through a Session
-- **`browser_fetch`**: Fetch a single URL through an open stealthy browser session, carrying the per-request options for that call. This is the session counterpart of `browser_fetch_once`. It can also capture a structured page snapshot with element references, positions, and sizes, helping the AI identify controls and plan mouse actions.
-- **`session_make_request`**: Make an HTTP request with any method through a session opened with `open_request_session`, reusing its cookies, connections, and browser fingerprint. This is the session counterpart of `make_request`.
+- **`browser_fetch`**: Render JavaScript and scrape a single URL through an open stealthy browser session, with fingerprint spoofing and Cloudflare Turnstile/Interstitial bypass. It can also capture a structured page snapshot with element references, positions, and sizes, helping the AI identify controls and plan mouse actions.
+- **`session_make_request`**: Make fast HTTP requests with any method (GET, POST, PUT, DELETE), browser fingerprint impersonation, matching browser headers, and HTTP/3 support. Reuse cookies, connections, and the browser fingerprint through a session opened with `open_request_session`.
#### 📸 Screenshots
@@ -64,7 +61,7 @@ Only completed requests are saved, including HTTP 4xx and 5xx responses. Failed
### Shadow DOM
-Set `pierce_shadow=true` on `browser_fetch_once`, `browser_fetch_many_once`, or `browser_fetch` to include open Shadow DOM content. It defaults to `false`. For persistent sessions, pass it on each `browser_fetch` call. See [Shadow DOM](../fetching/dynamic.md#shadow-dom) for selector examples and limits.
+Set `pierce_shadow=true` on `browser_fetch_once` or each `browser_fetch` call to include open Shadow DOM content. It defaults to `false`. See [Shadow DOM](../fetching/dynamic.md#shadow-dom) for selector examples and limits.
### Key Capabilities
- **Smart Content Extraction**: Convert web pages/elements to Markdown, HTML, or extract a clean version of the text content
@@ -72,7 +69,6 @@ Set `pierce_shadow=true` on `browser_fetch_once`, `browser_fetch_many_once`, or
- **Anti-Bot Bypass**: Handle Cloudflare Turnstile, Interstitial, and other protections
- **Proxy Support**: Use proxies for anonymity and geo-targeting
- **Browser Impersonation**: Mimic real browsers with TLS fingerprinting, real browser headers matching that version, and more
-- **Parallel Processing**: Scrape multiple URLs concurrently for efficiency
- **Session Persistence**: Reuse browser sessions across multiple requests for better performance
- **Ad Blocking**: All browser-based tools automatically block requests to ~3,500 known ad and tracker domains, saving tokens and speeding up page loads
- **Prompt Injection Protection**: Automatic sanitization of hidden content (CSS-hidden elements, aria-hidden, zero-width characters, HTML comments, template tags) that could be used for prompt injection attacks
@@ -189,7 +185,7 @@ Then, after you've added the server, you need to completely quit and restart the
### Custom Browser Executable
-Browser-based tools (`browser_fetch_once`, `browser_fetch_many_once`, and `browser_open`) can use a custom Chromium-compatible browser executable instead of the bundled Chromium. This is useful for custom browser builds or lightweight browser engines.
+The `browser_fetch_once` and `browser_open` tools can use a custom Chromium-compatible browser executable instead of the bundled Chromium. This is useful for custom browser builds or lightweight browser engines.
To configure it once for the whole MCP server, pass the executable path when starting the server:
@@ -213,7 +209,7 @@ In a Claude Desktop configuration, add the option to the server arguments:
}
```
-You can also set the `SCRAPLING_EXECUTABLE_PATH` environment variable before starting the server. Tool calls can still pass `executable_path` directly when a single request or session needs a different browser executable.
+You can also set the `SCRAPLING_EXECUTABLE_PATH` environment variable before starting the server. The AI can still choose a different browser executable for a single fetch with `browser_fetch_once` or when opening a session with `browser_open`.
### Connecting to Remote Browsers
@@ -313,7 +309,7 @@ We will gradually go from simple prompts to more complex ones. We will use Claud
Scrape the main content from https://example.com and convert it to markdown format.
```
- Claude will use the `make_request` tool to fetch the page and return clean, readable content. If it fails, it will continue retrying every second for 3 attempts, unless you instruct it otherwise. If it fails to retrieve content for any reason, such as protection or if it's a dynamic website, it will automatically try the other tools. If Claude didn't do that automatically for some reason, you can add that to the prompt.
+ Claude can use `make_request` to fetch the page and return clean, readable content. HTTP requests default to three attempts with a one-second delay between attempts. If the page needs JavaScript or blocks HTTP requests, the AI can use `browser_fetch_once` instead. Both tools close their own resources after the call.
A more optimized version of the same prompt would be:
```
@@ -335,33 +331,32 @@ We will gradually go from simple prompts to more complex ones. We will use Claud
Another example of a bit more complex prompt:
```
- Extract product information from these e-commerce URLs using bulk browser fetches:
+ Open a stealthy browser session and visit these e-commerce URLs one at a time:
- https://shop1.com/product-a
- https://shop2.com/product-b
- https://shop3.com/product-c
- Get the product names, prices, and descriptions from each page.
+ Get the product names, prices, and descriptions from each page. Close the session when done.
```
- Claude will use `browser_fetch_many_once` to concurrently scrape all URLs, then analyze the extracted data.
+ Claude can reuse one browser session, call `browser_fetch` for each URL in order, then analyze the extracted data.
4. **More advanced workflow**
Let's say I want to get all the action games available on PlayStation's store first page right now. I can use the following prompt to do that:
```
- Extract the URLs of all games in this page, then do a bulk request to them and return a list of all action games: https://store.playstation.com/en-us/pages/browse
+ Extract the URLs of all games on this page, then visit each game in order and return a list of all action games: https://store.playstation.com/en-us/pages/browse
```
- Note that I instructed it to use a bulk request for all the URLs collected. If I hadn't mentioned it, sometimes it works as intended, and other times it makes a separate request to each URL, which takes significantly longer. This prompt takes approximately one minute to complete.
-
- However, because I wasn't specific enough, it actually used the `browser_fetch_once` here and the `browser_fetch_many_once` in the second step, which unnecessarily consumed a large number of tokens. A better prompt would be:
+ Tell the AI to reuse an HTTP session to avoid launching a browser when the data is available through normal requests:
```
- Use normal requests to extract the URLs of all games in this page, then do a bulk request to them and return a list of all action games: https://store.playstation.com/en-us/pages/browse
+ Open an HTTP session, extract the URLs of all games on this page, then fetch each game in order and return a list of all action games. Close the session when done: https://store.playstation.com/en-us/pages/browse
```
- And if you know how to write CSS selectors, you can instruct Claude to apply the selectors to the elements you want, and it will nearly complete the task immediately.
+ And if you know how to write CSS selectors, you can narrow the content before the AI reads it:
```
- Use normal requests to extract the URLs of all games on the page below, then perform a bulk request to them and return a list of all action games.
- The selector for games in the first page is `[href*="/concept/"]` and the selector for the genre in the second request is `[data-qa="gameInfo#releaseInformation#genre-value"]`.
-
+ Use an HTTP session to extract the URLs of all games on the page below, then fetch each game in order and return a list of all action games.
+ The selector for games in the first page is `[href*="/concept/"]` and the selector for the genre in each game page is `[data-qa="gameInfo#releaseInformation#genre-value"]`.
+ Close the session when done.
+
URL: https://store.playstation.com/en-us/pages/browse
```
@@ -384,7 +379,7 @@ We will gradually go from simple prompts to more complex ones. We will use Claud
```
But a better prompt would be:
```
- Go to the following category URL and extract all product URLs using the CSS selector "a". Then, fetch the first 3 product pages in parallel and extract each product’s price and details.
+ Go to the following category URL and extract all product URLs using the CSS selector "a". Then, reuse the session to fetch the first 3 product pages in order and extract each product’s price and details. Close the session when done.
Keep the output in markdown format to reduce irrelevant content.
@@ -427,11 +422,12 @@ And so on, you get the idea. Your creativity is the key here.
Here is some technical advice for you.
### 1. Choose the Right Tool
-- **`make_request`**: Fast, simple websites
-- **`browser_fetch_once`**: JavaScript/dynamic content and protected sites, including Cloudflare
+- **`make_request`**: Fast HTTP requests for a single page or API call
+- **`browser_fetch_once`**: A single JavaScript/dynamic page or protected site, including Cloudflare
+- **Session tools**: Related requests, browser actions, screenshots, and network history
### 2. Optimize Performance
-- Use bulk tools for multiple URLs
+- Reuse a session and fetch each URL in order
- Disable unnecessary resources
- Set appropriate timeouts
- Use CSS selectors for targeted extraction
@@ -456,12 +452,12 @@ The MCP server automatically sanitizes scraped content when `main_content_only`
This protection runs automatically for HTML, Markdown, and text extraction. Keep `main_content_only=true` (the default) for maximum protection.
-### 6. Use Sessions for Multiple Requests
+### 6. Manage Sessions
+- Use `make_request` or `browser_fetch_once` for a standalone fetch; they close their own resources and do not leave a session for later calls
- Use `browser_open` to create a persistent browser session when scraping multiple pages, then call `browser_fetch` for each page through that session
- For multiple plain HTTP requests, use `open_request_session` instead and call `session_make_request` per request; it keeps cookies, connections, and the browser fingerprint (`impersonate`) across calls without a browser
- Sessions hold the session-level configuration set when opened (headless, locale, cookies, stealth toggles, etc. for browsers; `impersonate` and `proxy` for requests sessions); the per-request options (timeout, wait_selector, network_idle, solve_cloudflare, etc.) are passed to `browser_fetch`/`session_make_request` on each call, with their defaults shown in the tool schemas
- `browser_fetch` can solve Cloudflare challenges through the open stealthy browser session
-- The one-shot tools (`make_request`, `bulk_get`, `browser_fetch_once`, `browser_fetch_many_once`) never take a session
- Always close sessions with `close_session` when done to free resources
- Use `list_sessions` to check which sessions are still active and see the `settings` each was created with (returned for the AI agent; empty for CDP sessions)
- Pass a custom `session_id` to the open tools to give sessions meaningful names (e.g. `"search"`, `"checkout"`) instead of the random hex default. They raise if the chosen ID is already in use, so you can detect collisions up front
diff --git a/docs/api-reference/mcp-server.md b/docs/api-reference/mcp-server.md
index 6eb229c..b25f91d 100644
--- a/docs/api-reference/mcp-server.md
+++ b/docs/api-reference/mcp-server.md
@@ -5,7 +5,7 @@ search:
# MCP Server API Reference
-The **Scrapling MCP Server** provides sixteen tools for web scraping and browser interaction through the Model Context Protocol (MCP). This server integrates Scrapling's capabilities directly into AI chatbots and agents, allowing conversational web scraping with advanced anti-bot bypass features.
+The **Scrapling MCP Server** provides fourteen tools for web scraping and browser interaction through the Model Context Protocol (MCP). This server integrates Scrapling's capabilities directly into AI chatbots and agents, allowing conversational web scraping with advanced anti-bot bypass features.
You can start the MCP server by running:
@@ -30,7 +30,7 @@ server = ScraplingMCPServer(executable_path="/path/to/chromium")
## Response Model
-The standardized response structure returned by the fetch and HTTP request tools. Bulk tools return a list of these responses. Session management tools use the models below, `browser_screenshot` returns an image of the current page or one element, plus the current page URL, as image and text content blocks, and `browser_snapshot` and `browser_actions` return plain text. Both network tools return structured JSON using the models below. `browser_evaluate` returns its result as JSON in a plain text block.
+The standardized response structure returned by the fetch and HTTP request tools, with one response per call. Session management tools use the models below, `browser_screenshot` returns an image of the current page or one element, plus the current page URL, as image and text content blocks, and `browser_snapshot` and `browser_actions` return plain text. Both network tools return structured JSON using the models below. `browser_evaluate` returns its result as JSON in a plain text block.
## ::: scrapling.core.ai.ResponseModel
handler: python
@@ -74,6 +74,9 @@ Model classes for session management:
The main MCP server class that provides all web scraping tools:
+- **`make_request`**: Fetch one URL through HTTP with browser impersonation and automatic client cleanup. Supports GET, POST, PUT, and DELETE.
+- **`browser_fetch_once`**: Fetch one URL through a stealthy browser with JavaScript rendering and Cloudflare handling, then close the browser. Use a persistent session for later actions or network history.
+
- **`browser_snapshot`**: Inspect the current page without reloading it. Structured text includes element roles, names, references, positions, and sizes to help the AI target page controls.
- **`browser_actions`**: Chain mouse moves, hovers, clicks, scrolling, field filling, keyboard shortcuts, and waits in one call. Fill a search box, submit it, wait for results, and open a result without a separate call for each step.
diff --git a/scrapling/core/ai.py b/scrapling/core/ai.py
index 9bcea23..d8f2aed 100644
--- a/scrapling/core/ai.py
+++ b/scrapling/core/ai.py
@@ -3,7 +3,6 @@ from os import environ
from json import dumps
from time import monotonic
from hmac import compare_digest
-from asyncio import gather
from functools import wraps
from contextlib import contextmanager
from datetime import datetime, timezone
@@ -65,13 +64,6 @@ ScreenshotType = Literal["png", "jpeg"]
MCP_EXECUTABLE_PATH_ENV = "SCRAPLING_EXECUTABLE_PATH"
MCP_AUTH_TOKEN_ENV = "SCRAPLING_MCP_AUTH_TOKEN" # nosec B105 - the name of the variable, not a token
-_MAX_POOL_PAGES = 50 # Upper bound of `PagesCount` in scrapling/engines/_browsers/_validators.py
-
-
-def _page_pool_size(urls: Sequence[str]) -> int:
- """Return a page pool size that covers the batch without leaving the validator's bounds."""
- return min(max(len(urls), 1), _MAX_POOL_PAGES)
-
def _typed_dict_keys(typed_dict: Any) -> frozenset:
"""Collect all the keys a TypedDict holds, including the inherited ones."""
@@ -644,86 +636,13 @@ class ScraplingMCPServer:
page = await getattr(session, method.lower())(url, **request_kwargs)
return _translate_response(page, extraction_type, css_selector, main_content_only)
- @staticmethod
- async def bulk_get(
- urls: List[str],
- impersonate: ImpersonateType = "chrome",
- extraction_type: extraction_types = "markdown",
- css_selector: Optional[str] = None,
- main_content_only: bool = True,
- params: Optional[Dict] = None,
- headers: Optional[Mapping[str, Optional[str]]] = None,
- cookies: Optional[Dict[str, str]] = None,
- timeout: Optional[int | float] = 30,
- follow_redirects: FollowRedirects = "safe",
- max_redirects: int = 30,
- retries: Optional[int] = 3,
- retry_delay: Optional[int] = 1,
- proxy: Optional[str] = None,
- proxy_auth: Optional[Dict[str, str]] = None,
- auth: Optional[Dict[str, str]] = None,
- verify: Optional[bool] = True,
- http3: Optional[bool] = False,
- stealthy_headers: Optional[bool] = True,
- ) -> List[ResponseModel]:
- """GET multiple URLs concurrently without JavaScript. Suitable for low-to-mid protection.
-
- :param urls: URLs to fetch.
- :param impersonate: Browser/version to impersonate; "chrome" uses the latest supported Chrome profile.
- Lists pick randomly per attempt; None disables impersonation.
- :param extraction_type: Content output format.
- :param css_selector: Select matching elements after `main_content_only` filtering.
- :param main_content_only: Sanitize <body> content before selection; False uses the full document.
- :param params: Query parameters.
- :param headers: Request headers.
- :param cookies: Request cookies.
- :param timeout: Timeout in seconds.
- :param follow_redirects: "safe" blocks redirects to private/internal IPs; True allows all; False disables redirects.
- :param max_redirects: Redirect limit; -1 means unlimited.
- :param retries: Maximum attempts, including the first; None or values below 1 send once.
- :param retry_delay: Seconds between retries.
- :param proxy: Proxy URL, optionally including credentials.
- :param proxy_auth: Proxy basic auth: {"username": ..., "password": ...}.
- :param auth: Basic auth: {"username": ..., "password": ...}.
- :param verify: Verify TLS certificates.
- :param http3: Use HTTP/3; may conflict with browser impersonation.
- :param stealthy_headers: Fill missing browser headers and use a Google referer if none was supplied.
- """
- normalized_proxy_auth = _normalize_credentials(proxy_auth)
- normalized_auth = _normalize_credentials(auth)
-
- async with FetcherSession() as session:
- tasks: List[Any] = [
- session.get(
- url,
- auth=normalized_auth,
- proxy=proxy,
- http3=http3,
- verify=verify,
- params=params,
- headers=headers,
- cookies=cookies,
- timeout=timeout,
- retries=retries,
- proxy_auth=normalized_proxy_auth,
- retry_delay=retry_delay,
- impersonate=impersonate,
- max_redirects=max_redirects,
- follow_redirects=follow_redirects,
- stealthy_headers=stealthy_headers,
- )
- for url in urls
- ]
- responses = await gather(*tasks)
- return [_translate_response(page, extraction_type, css_selector, main_content_only) for page in responses]
-
async def browser_fetch_once(
self,
url: str,
extraction_type: extraction_types = "markdown",
css_selector: Optional[str] = None,
main_content_only: bool = True,
- headless: bool = True, # noqa: F821
+ headless: bool = True,
google_search: bool = True,
real_chrome: bool = False,
wait: int | float = 0,
@@ -777,97 +696,6 @@ class ScraplingMCPServer:
:param proxy: Proxy URL, or dictionary with server and optional username/password.
:param additional_args: Browser context options that override Scrapling settings.
"""
- results = await self.browser_fetch_many_once(
- urls=[url],
- extraction_type=extraction_type,
- css_selector=css_selector,
- main_content_only=main_content_only,
- headless=headless,
- google_search=google_search,
- real_chrome=real_chrome,
- wait=wait,
- proxy=proxy,
- timezone_id=timezone_id,
- locale=locale,
- extra_headers=extra_headers,
- useragent=useragent,
- hide_canvas=hide_canvas,
- cdp_url=cdp_url,
- executable_path=executable_path,
- timeout=timeout,
- disable_resources=disable_resources,
- wait_selector=wait_selector,
- cookies=cookies,
- network_idle=network_idle,
- pierce_shadow=pierce_shadow,
- wait_selector_state=wait_selector_state,
- block_webrtc=block_webrtc,
- allow_webgl=allow_webgl,
- solve_cloudflare=solve_cloudflare,
- additional_args=additional_args,
- )
- return results[0]
-
- async def browser_fetch_many_once(
- self,
- urls: List[str],
- extraction_type: extraction_types = "markdown",
- css_selector: Optional[str] = None,
- main_content_only: bool = True,
- headless: bool = True, # noqa: F821
- google_search: bool = True,
- real_chrome: bool = False,
- wait: int | float = 0,
- proxy: Optional[str | Dict[str, str]] = None,
- timezone_id: str | None = None,
- locale: str | None = None,
- extra_headers: Optional[Dict[str, str]] = None,
- useragent: Optional[str] = None,
- hide_canvas: bool = False,
- cdp_url: Optional[str] = None,
- executable_path: Optional[str] = None,
- timeout: int | float = 30000,
- disable_resources: bool = False,
- wait_selector: Optional[str] = None,
- cookies: Sequence[SetCookieParam] | None = None,
- network_idle: bool = False,
- wait_selector_state: SelectorWaitStates = "attached",
- block_webrtc: bool = False,
- allow_webgl: bool = True,
- solve_cloudflare: bool = False,
- additional_args: Optional[Dict] = None,
- pierce_shadow: bool = False,
- ) -> List[ResponseModel]:
- """Fetch pages concurrently with a stealth browser and JavaScript rendering.
-
- :param urls: URLs to fetch, with at most 50 concurrent pages.
- :param extraction_type: Content output format.
- :param css_selector: Select matching elements after `main_content_only` filtering.
- :param main_content_only: Sanitize <body> content before selection; False uses the full document.
- :param headless: Hide the browser window.
- :param disable_resources: Block font, image, media, beacon, object, imageset, texttrack, websocket, csp_report, and stylesheet requests.
- :param useragent: User-Agent override; otherwise generated in headless mode, native in headful mode.
- :param cookies: Initial cookies as Playwright cookie dictionaries.
- :param solve_cloudflare: Attempt to solve Cloudflare Turnstile/interstitial challenges before returning.
- :param allow_webgl: Enable WebGL; disabling it can trigger bot detection.
- :param pierce_shadow: Include open Shadow DOM content.
- :param network_idle: Try to wait for 500 ms without network activity; continue if the wait fails.
- :param wait: Extra milliseconds after the page is ready, before returning.
- :param timeout: Navigation and page-operation timeout in milliseconds.
- :param wait_selector: Wait for the first CSS match; continue if the wait fails.
- :param timezone_id: Browser timezone; uses the system timezone if omitted.
- :param locale: Browser language, Accept-Language, and formatting locale; system default if omitted.
- :param wait_selector_state: Target state of `wait_selector`.
- :param real_chrome: Use locally installed Chrome.
- :param hide_canvas: Add noise to canvas operations.
- :param block_webrtc: Disable non-proxied WebRTC UDP to reduce IP leaks.
- :param cdp_url: Connect to an existing Chromium browser over CDP in a new context; launch settings do not apply.
- :param executable_path: Absolute Chromium-compatible executable path; overrides the server default.
- :param google_search: Set a Google referer, overriding any supplied referer.
- :param extra_headers: Additional request headers.
- :param proxy: Proxy URL, or dictionary with server and optional username/password.
- :param additional_args: Browser context options that override Scrapling settings.
- """
async with AsyncStealthySession(
wait=wait,
proxy=proxy,
@@ -877,7 +705,7 @@ class ScraplingMCPServer:
cookies=cookies,
headless=headless,
block_ads=True,
- max_pages=_page_pool_size(urls),
+ max_pages=1,
useragent=useragent,
timezone_id=timezone_id,
real_chrome=real_chrome,
@@ -895,10 +723,9 @@ class ScraplingMCPServer:
disable_resources=disable_resources,
wait_selector_state=wait_selector_state,
) as session:
- tasks = [session.fetch(url) for url in urls]
- responses = await gather(*tasks)
+ page = await session.fetch(url)
- return [_translate_response(page, extraction_type, css_selector, main_content_only) for page in responses]
+ return _translate_response(page, extraction_type, css_selector, main_content_only)
async def browser_fetch(
self,
@@ -1056,25 +883,15 @@ class ScraplingMCPServer:
)
],
"cache_hints": {"tools/list": CacheHint(ttl_ms=3_600_000, scope="public")},
- "instructions": """Follow these instructions precisely:
-1. When the `browser_open` or `open_request_session` tools are used, make sure to close the session with `close_session` after you finish, and use `list_sessions` if you lose track of the open sessions or their effective settings.
-2. If the user didn't specify which tool to use, start with the `make_request` tool (a plain HTTP request, defaulting to GET; set `method` for POST/PUT/DELETE), then escalate. The `make_request` tool and `bulk_get` (its GET-only bulk version) are suitable only for low-to-mid protection levels.
- For high-protection levels or websites that require JS loading, use the stealthy browser tools directly.
-3. For HTML, Markdown, and text extraction, if the `css_selector` resolves to more than one element, all the elements will be returned. Snapshot extraction requires a selector matching exactly one element, or no selector for the whole page.
-4. For all fetch tools, the `extraction_type` parameter controls the format of the returned content: "markdown" (default) converts the page content to Markdown, "html" returns the raw HTML, and "text" returns the text content of the page.
-5. For HTML, Markdown, and text extraction, `main_content_only` is enabled by default and returns only the content inside the page's `<body>` tag. Pass `main_content_only=False` when you need the full page instead.
-6. If the task consists of multiple sequential requests to the same website, open a session once, then fetch through it to be more efficient:
- `browser_open` + `browser_fetch` per page for stealthy browsers, or `open_request_session` + `session_make_request` per request for plain HTTP.
-7. Sessions hold the session-level configuration set when opened, while `browser_fetch`/`session_make_request` carry the per-request options and apply them on each call with the defaults shown in their schemas.
- The one-shot tools (`make_request`, `bulk_get`, `browser_fetch_once`, `browser_fetch_many_once`) never touch sessions.
-8. If you are making multiple parallel one-shot requests, use the bulk version of the tool to be more efficient.
-9. If you are crawling/browsing a website, be more efficient by using the `css_selector` parameter to only access the parts you are interested in and save money/time. Example: use the `a` selector to extract the urls right away.
-10. The user can pass a CDP URL to connect to a remote browser session through the `browser_open` tool, then use it with the session tools.
-11. Set `extraction_type="snapshot"` on `browser_fetch` to get an AI ARIA snapshot with element references and bounding boxes in its content field. Use `css_selector` to snapshot one element, or omit it for the whole page. `main_content_only` and `pierce_shadow` do not filter snapshots. Use `browser_snapshot` to read the current page without navigating; it returns plain text with boxes by default. Set `depth` to limit the tree or `boxes=False` to omit boxes.
-12. Use `browser_actions` for ordered mouse, field, key, and wait actions on the current page. Move/click targets use exactly one selector, current snapshot ref, or viewport (x, y) pair in CSS pixels; fields use one selector/ref. Wheel acts at the current pointer and does not wait for scrolling; coordinate clicks do not wait for navigation. Load waits observe the current document, so prefer a result-specific element for delayed navigation.
-13. Use `browser_snapshot` to inspect results before repeating failed actions or choosing targets that depend on page changes. Actions may partly complete before an error; completed effects are not undone or retried. To submit with Enter, focus the intended control and use a press_key action with key="Enter".
-14. Browser sessions save completed requests and responses across page changes. Failed and unfinished requests are omitted; closing does not wait for captures still running. Use `browser_network_requests` to find IDs, then `browser_network_request` to read one part. Successful non-API traffic is hidden unless include_static=True. Keep up to 1000 requests and 20 MiB of saved response bodies; oldest records are removed when either limit is reached. Only response bodies with a text Content-Type are read and saved; binary or unknown types keep headers and a skip note. Bodies over 1 MiB are skipped; unavailable bodies are marked. `browser_network_request` returns summary and headers as objects, and the full saved body as text. Closing an MCP session removes access to its history. Use next_cursor as after_id to continue the request list.
-""",
+ "instructions": """1. Close sessions when done. Use `list_sessions` to check sessions and settings. Read needed network data before closing; closing removes access to its history.
+2. Unless the user chooses a tool, start with HTTP for low to mid protection; use browsers for strong protection or JavaScript.
+3. Use `make_request` or `browser_fetch_once` for standalone fetches; they close automatically. Use sessions for related requests or follow-up browser actions and network history.
+4. Use `css_selector` to reduce output. HTML/Markdown/text return all matches; snapshots require one match or no selector for the whole page.
+5. Session options persist from opening; fetch options apply per call, with schema defaults.
+6. Each browser session uses one page; finish each call before starting the next in that session.
+7. Snapshots include refs and boxes by default; `main_content_only` and `pierce_shadow` do not filter them.
+8. `browser_actions` chains mouse, field, key and wait actions on the current page. Move/click needs one selector, current snapshot ref, or viewport (x, y) in CSS pixels; fields need one selector/ref. Wheel uses the current pointer without waiting for scrolling; coordinate clicks do not wait for navigation. Load waits observe the current document; wait for a result element for delayed navigation.
+9. Network history saves completed requests and responses across page changes. Failed/unfinished requests are omitted; closing does not wait for running captures. Use `include_static=True` to list successful non-API traffic. Limits: 1000 requests and 20 MiB of saved response bodies; oldest records are removed first. Only response bodies with a text Content-Type are read and saved. Response bodies over 1 MiB are skipped; unavailable bodies are marked.""",
}
if self._auth_token:
base_url = AnyHttpUrl(f"http://{host}:{port}")
@@ -1107,7 +924,6 @@ class ScraplingMCPServer:
structured_output=True,
annotations=_LIST_TOOL_ANNOTATIONS,
)
- # HTTP tools
server.add_tool(
_mcp_tool(self.make_request),
title="Send HTTP request",
@@ -1115,13 +931,6 @@ class ScraplingMCPServer:
structured_output=True,
annotations=_FETCH_TOOL_ANNOTATIONS,
)
- server.add_tool(
- _mcp_tool(self.bulk_get),
- title="Get pages via HTTP requests",
- description=self.bulk_get.__doc__,
- structured_output=True,
- annotations=_FETCH_TOOL_ANNOTATIONS,
- )
server.add_tool(
_mcp_tool(self.browser_fetch_once),
title="Fetch page in browser",
@@ -1129,13 +938,6 @@ class ScraplingMCPServer:
structured_output=True,
annotations=_FETCH_TOOL_ANNOTATIONS,
)
- server.add_tool(
- _mcp_tool(self.browser_fetch_many_once),
- title="Fetch pages in browser",
- description=self.browser_fetch_many_once.__doc__,
- structured_output=True,
- annotations=_FETCH_TOOL_ANNOTATIONS,
- )
# Session-scoped fetch tools
server.add_tool(
_mcp_tool(self.browser_fetch),
diff --git a/tests/ai/test_ai_mcp.py b/tests/ai/test_ai_mcp.py
index 4cc5e3b..8d42a4c 100644
--- a/tests/ai/test_ai_mcp.py
+++ b/tests/ai/test_ai_mcp.py
@@ -1,11 +1,12 @@
import base64
import inspect
+from json import loads
import struct
from typing import Any
from contextlib import contextmanager
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
-from unittest.mock import Mock, patch
+from unittest.mock import AsyncMock, Mock, patch
import pytest
import pytest_httpbin
@@ -25,7 +26,6 @@ from scrapling.core.ai import (
SessionType,
_SessionEntry,
_normalize_credentials,
- _page_pool_size,
_session_settings,
_StaticTokenVerifier,
_STEALTH_FETCH_KEYS,
@@ -35,6 +35,24 @@ from scrapling.engines._browsers._validators import StealthConfig, models_defaul
from scrapling.fetchers import AsyncStealthySession, FetcherSession
+MCP_TOOLS = {
+ "make_request",
+ "browser_fetch_once",
+ "open_request_session",
+ "session_make_request",
+ "browser_open",
+ "browser_fetch",
+ "browser_actions",
+ "browser_evaluate",
+ "browser_snapshot",
+ "browser_screenshot",
+ "browser_network_requests",
+ "browser_network_request",
+ "close_session",
+ "list_sessions",
+}
+
+
def test_translate_response_strips_control_characters():
"""Pages with control chars like U+0008 must not crash the request/fetch path (issue #366)"""
html = "<html><body><p>Hello\x08World</p>\t\n<div>Foo\x0cbar</div></body></html>"
@@ -105,6 +123,159 @@ class _FakeStealthySession:
)
+class TestOneShotTools:
+ @pytest.mark.asyncio
+ async def test_make_request_get_and_post_through_mcp(self, monkeypatch, httpbin):
+ sessions: list[FetcherSession] = []
+
+ def create_session():
+ session = FetcherSession()
+ sessions.append(session)
+ return session
+
+ monkeypatch.setattr("scrapling.core.ai.FetcherSession", create_session)
+ server = ScraplingMCPServer()
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ fetched = await client.call_tool("make_request", {"url": f"{httpbin.url}/html", "css_selector": "h1"})
+ assert not fetched.is_error and fetched.structured_content is not None
+ assert fetched.structured_content["status"] == 200
+ assert fetched.structured_content["url"] == f"{httpbin.url}/html"
+ assert "Herman Melville - Moby-Dick" in "".join(fetched.structured_content["content"])
+ posted = await client.call_tool(
+ "make_request",
+ {
+ "url": f"{httpbin.url}/post",
+ "method": "POST",
+ "json": {"key": "value"},
+ "extraction_type": "text",
+ },
+ )
+ assert not posted.is_error and posted.structured_content is not None
+ assert posted.structured_content["status"] == 200
+ assert loads("".join(posted.structured_content["content"]))["json"] == {"key": "value"}
+ cookies = await client.call_tool(
+ "make_request",
+ {"url": f"{httpbin.url}/cookies/set/test/value", "follow_redirects": True, "extraction_type": "text"},
+ )
+ assert not cookies.is_error and cookies.structured_content is not None
+ assert loads("".join(cookies.structured_content["content"]))["cookies"] == {"test": "value"}
+ fresh = await client.call_tool("make_request", {"url": f"{httpbin.url}/cookies", "extraction_type": "text"})
+ assert not fresh.is_error and fresh.structured_content is not None
+ assert loads("".join(fresh.structured_content["content"]))["cookies"] == {}
+ assert len(sessions) == 4 and all(not session._is_alive for session in sessions)
+ assert not server._sessions
+
+ @pytest.mark.asyncio
+ async def test_browser_fetch_once_renders_and_closes_through_mcp(self, monkeypatch):
+ sessions: list[AsyncStealthySession] = []
+
+ def create_session(**kwargs: Any):
+ session = AsyncStealthySession(**kwargs)
+ sessions.append(session)
+ return session
+
+ monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", create_session)
+ server = ScraplingMCPServer()
+ html = b"""<html><body><p id="answer">Initial</p><p>Outside</p><script>
+setTimeout(() => { const p = document.querySelector('#answer'); p.textContent = 'Rendered'; p.id = 'ready'; }, 50);
+</script></body></html>"""
+ with _serve_html(html) as url:
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool(
+ "browser_fetch_once",
+ {"url": url, "google_search": False, "wait_selector": "#ready", "css_selector": "#ready"},
+ )
+ assert not result.is_error and result.structured_content is not None
+ assert result.structured_content["status"] == 200
+ assert result.structured_content["url"] == url
+ content = "".join(result.structured_content["content"])
+ assert "Rendered" in content and "Initial" not in content and "Outside" not in content
+ assert len(sessions) == 1 and not sessions[0]._is_alive
+ assert sessions[0].page_pool.pages_count == 0
+ assert not server._sessions
+
+ @pytest.mark.asyncio
+ @pytest.mark.parametrize("method", ["GET", "POST", "PUT", "DELETE"])
+ async def test_make_request_forwards_options_and_closes(self, monkeypatch, method):
+ response = Response(
+ url="https://example.com/final",
+ content="<main>ok</main>",
+ status=201,
+ reason="Created",
+ cookies={},
+ headers={},
+ request_headers={},
+ )
+ request = AsyncMock(return_value=response)
+ session = Mock(**{method.lower(): request})
+ context = AsyncMock()
+ context.__aenter__.return_value = session
+ factory = Mock(return_value=context)
+ monkeypatch.setattr("scrapling.core.ai.FetcherSession", factory)
+ server = ScraplingMCPServer()
+ existing = _SessionEntry(Mock(_is_alive=True), "static")
+ server._sessions["existing"] = existing
+ options = {
+ "impersonate": "firefox",
+ "params": {"q": "query"},
+ "data": "body",
+ "json": {"key": "value"},
+ "headers": {"X-Test": "value"},
+ "cookies": {"theme": "dark"},
+ "timeout": 12.5,
+ "follow_redirects": False,
+ "max_redirects": 5,
+ "retries": 2,
+ "retry_delay": 0,
+ "proxy": "http://127.0.0.1:8080",
+ "proxy_auth": {"username": "proxy-user", "password": "proxy-pass"},
+ "auth": {"username": "user", "password": "pass"},
+ "verify": False,
+ "http3": True,
+ "stealthy_headers": False,
+ }
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool("make_request", {"url": "https://example.com", "method": method, **options})
+ assert not result.is_error and result.structured_content is not None
+ assert result.structured_content["status"] == 201
+ assert result.structured_content["url"] == response.url
+ expected = {**options, "auth": ("user", "pass"), "proxy_auth": ("proxy-user", "proxy-pass")}
+ if method == "GET":
+ expected.pop("data")
+ expected.pop("json")
+ request.assert_awaited_once_with("https://example.com", **expected)
+ factory.assert_called_once_with()
+ context.__aexit__.assert_awaited_once_with(None, None, None)
+ assert server._sessions == {"existing": existing}
+ assert existing.session.mock_calls == []
+
+ @pytest.mark.asyncio
+ @pytest.mark.parametrize("tool", ["make_request", "browser_fetch_once"])
+ async def test_failed_one_shot_closes_without_touching_registered_sessions(self, monkeypatch, tool):
+ context = AsyncMock()
+ context.__aenter__.return_value = context
+ request = context.get if tool == "make_request" else context.fetch
+ request.side_effect = RuntimeError("fetch failed")
+ factory = Mock(return_value=context)
+ monkeypatch.setattr(
+ "scrapling.core.ai.FetcherSession" if tool == "make_request" else "scrapling.core.ai.AsyncStealthySession",
+ factory,
+ )
+ server = ScraplingMCPServer()
+ existing = _SessionEntry(Mock(_is_alive=True), "stealthy")
+ server._sessions["existing"] = existing
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool(tool, {"url": "https://example.com"})
+ assert result.is_error
+ assert result.content and isinstance(result.content[0], TextContent)
+ assert "fetch failed" in result.content[0].text
+ request.assert_awaited_once()
+ context.__aexit__.assert_awaited_once()
+ assert context.__aexit__.call_args.args[0] is RuntimeError
+ assert server._sessions == {"existing": existing}
+ assert existing.session.mock_calls == []
+
+
class TestSessionTypeChecks:
@pytest.mark.parametrize(
"session_type, allowed, error",
@@ -157,57 +328,6 @@ class TestSessionTypeChecks:
assert server._sessions["test"].session is session
-@pytest_httpbin.use_class_based_httpbin
-class TestMCPServer:
- """Test MCP server functionality"""
-
- @pytest.fixture(scope="class")
- def test_url(self, httpbin):
- return f"{httpbin.url}/html"
-
- @pytest.fixture
- def server(self):
- return ScraplingMCPServer()
-
- @pytest.mark.asyncio
- async def test_make_request_tool(self, server, test_url):
- """Test the make_request tool method with a default GET"""
- result = await server.make_request(url=test_url, extraction_type="markdown")
- assert isinstance(result, ResponseModel)
- assert result.status == 200
- assert result.url == test_url
-
- @pytest.mark.asyncio
- async def test_make_request_post_tool(self, server, httpbin):
- """Test the make_request tool method with a POST body"""
- result = await server.make_request(
- url=f"{httpbin.url}/post", method="POST", json={"key": "value"}, extraction_type="text"
- )
- assert isinstance(result, ResponseModel)
- assert result.status == 200
-
- @pytest.mark.asyncio
- async def test_bulk_get_tool(self, server, test_url):
- """Test the bulk_get tool method"""
- results = await server.bulk_get(urls=(test_url, test_url), extraction_type="html")
-
- assert len(results) == 2
- assert all(isinstance(r, ResponseModel) for r in results)
-
- @pytest.mark.asyncio
- async def test_browser_fetch_once_tool(self, server, test_url):
- """Test the browser_fetch_once tool method"""
- result = await server.browser_fetch_once(url=test_url, headless=True)
- assert isinstance(result, ResponseModel)
- assert result.status == 200
-
- @pytest.mark.asyncio
- async def test_browser_fetch_many_once_tool(self, server, test_url):
- """Test the browser_fetch_many_once tool method"""
- result = await server.browser_fetch_many_once(urls=(test_url, test_url), headless=True)
- assert all(isinstance(r, ResponseModel) for r in result)
-
-
@pytest_httpbin.use_class_based_httpbin
class TestSessionManagement:
"""Test persistent browser session management"""
@@ -334,6 +454,42 @@ class TestStaticSessionManagement:
def server(self):
return ScraplingMCPServer()
+ @pytest.mark.asyncio
+ async def test_static_session_workflow_through_mcp(self, server, httpbin):
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ opened = await client.call_tool("open_request_session", {"session_id": "http"})
+ assert not opened.is_error
+ session = server._sessions["http"].session
+ try:
+ response = await client.call_tool(
+ "session_make_request",
+ {"url": f"{httpbin.url}/html", "session_id": "http", "css_selector": "h1"},
+ )
+ assert not response.is_error
+ assert response.structured_content is not None
+ assert response.structured_content["status"] == 200
+ assert response.structured_content["url"] == f"{httpbin.url}/html"
+ assert "Herman Melville - Moby-Dick" in "".join(response.structured_content["content"])
+ posted = await client.call_tool(
+ "session_make_request",
+ {
+ "url": f"{httpbin.url}/post",
+ "session_id": "http",
+ "method": "POST",
+ "json": {"key": "value"},
+ "extraction_type": "text",
+ },
+ )
+ assert not posted.is_error
+ assert posted.structured_content is not None
+ assert posted.structured_content["status"] == 200
+ assert loads("".join(posted.structured_content["content"]))["json"] == {"key": "value"}
+ finally:
+ closed = await client.call_tool("close_session", {"session_id": "http"})
+ assert not closed.is_error
+ assert not session._is_alive
+ assert not server._sessions
+
@pytest.mark.asyncio
async def test_static_session_lifecycle(self, server, httpbin):
"""Open a requests session, make GET and POST requests through it, then close it"""
@@ -436,6 +592,28 @@ class TestExecutablePath:
assert _FakeStealthySession.instances[0].kwargs["executable_path"] == "/opt/custom-chromium"
await server.close_session(created.session_id)
+ @pytest.mark.asyncio
+ async def test_browser_open_overrides_global_executable_path(self, monkeypatch):
+ """browser_open prefers the supplied executable path to the server default."""
+ monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
+ server = ScraplingMCPServer(executable_path="/opt/default-chromium")
+
+ created = await server.browser_open(executable_path="/opt/request-chromium")
+
+ assert _FakeStealthySession.instances[0].kwargs["executable_path"] == "/opt/request-chromium"
+ await server.close_session(created.session_id)
+
+ @pytest.mark.asyncio
+ async def test_browser_open_uses_global_executable_path(self, monkeypatch):
+ """browser_open forwards the server executable path default."""
+ monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
+ server = ScraplingMCPServer(executable_path="/opt/default-chromium")
+
+ created = await server.browser_open()
+
+ assert _FakeStealthySession.instances[0].kwargs["executable_path"] == "/opt/default-chromium"
+ await server.close_session(created.session_id)
+
@pytest.mark.asyncio
async def test_browser_fetch_once_overrides_global_executable_path(self, monkeypatch):
"""browser_fetch_once forwards a per-call executable_path instead of the server default"""
@@ -458,45 +636,17 @@ class TestExecutablePath:
assert isinstance(result, ResponseModel)
assert _FakeStealthySession.instances[0].kwargs["executable_path"] == "/opt/default-chromium"
-
-class TestBulkPagePool:
- """Test the page pool sizing of the bulk browser tools"""
-
- @pytest.fixture(autouse=True)
- def reset_fakes(self):
- _FakeStealthySession.instances = []
-
@pytest.mark.asyncio
- @pytest.mark.parametrize("url_count,expected_pages", [(3, 3), (60, 50), (0, 1)])
- async def test_browser_fetch_many_once_sizes_pool_within_validator_bounds(
- self, monkeypatch, url_count, expected_pages
- ):
- """browser_fetch_many_once opens a pool that covers the batch but stays inside the 1..50 `PagesCount` range"""
+ async def test_browser_fetch_once_uses_environment_default(self, monkeypatch):
+ monkeypatch.setenv("SCRAPLING_EXECUTABLE_PATH", "/opt/environment-chromium")
monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
- server = ScraplingMCPServer()
- urls = [f"https://example.com/{index}" for index in range(url_count)]
-
- results = await server.browser_fetch_many_once(urls=urls)
-
- max_pages = _FakeStealthySession.instances[0].kwargs["max_pages"]
- assert max_pages == expected_pages, f"Expected max_pages {expected_pages} for {url_count} URLs, got {max_pages}"
- assert len(results) == url_count, f"Expected {url_count} responses, got {len(results)}"
-
- @pytest.mark.parametrize("url_count", [0, 1, 50, 60, 500])
- def test_page_pool_size_is_accepted_by_session_validation(self, url_count):
- """The computed pool size always passes the real session validation without launching a browser"""
- urls = [f"https://example.com/{index}" for index in range(url_count)]
-
- session = AsyncStealthySession(max_pages=_page_pool_size(urls))
-
- assert session.max_pages == _page_pool_size(urls), (
- f"Expected the session to keep max_pages {_page_pool_size(urls)}, got {session.max_pages}"
- )
+ await ScraplingMCPServer().browser_fetch_once(url="https://example.com")
+ assert _FakeStealthySession.instances[0].kwargs["executable_path"] == "/opt/environment-chromium"
+ assert not _FakeStealthySession.instances[0]._is_alive
class TestOneShotBrowserForwarding:
@pytest.mark.asyncio
- @pytest.mark.parametrize("tool", ["browser_fetch_once", "browser_fetch_many_once"])
@pytest.mark.parametrize(
"options",
[
@@ -529,17 +679,13 @@ class TestOneShotBrowserForwarding:
],
ids=["defaults", "overrides"],
)
- async def test_each_call_uses_a_closed_stealth_session_with_all_options(self, monkeypatch, tool, options):
+ async def test_each_call_uses_a_closed_stealth_session_with_all_options(self, monkeypatch, options):
monkeypatch.delenv("SCRAPLING_EXECUTABLE_PATH", raising=False)
monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
_FakeStealthySession.instances = []
server = ScraplingMCPServer()
- page_count = 2 if "_many_" in tool else 1
- args: dict[str, Any] = (
- {"urls": ["https://example.com/1", "https://example.com/2"]}
- if "_many_" in tool
- else {"url": "https://example.com/1"}
- )
+ existing = _SessionEntry(Mock(_is_alive=True), "stealthy")
+ server._sessions["existing"] = existing
expected = {
"headless": True,
"google_search": True,
@@ -566,17 +712,18 @@ class TestOneShotBrowserForwarding:
"pierce_shadow": False,
**options,
"block_ads": True,
- "max_pages": page_count,
+ "max_pages": 1,
}
async with Client(server._build_server("127.0.0.1", 8000)) as client:
for _ in range(2):
- result = await client.call_tool(tool, {**args, **options})
+ result = await client.call_tool("browser_fetch_once", {"url": "https://example.com/1", **options})
assert not result.is_error
- assert not server._sessions
+ assert server._sessions == {"existing": existing}
+ assert existing.session.mock_calls == []
assert len(_FakeStealthySession.instances) == 2
for session in _FakeStealthySession.instances:
assert session.kwargs == expected
- assert session.fetch_calls == [{}] * page_count
+ assert session.fetch_calls == [{}]
assert not session._is_alive
@@ -634,15 +781,34 @@ class TestBrowserFetchForwarding:
"""Per-request overrides reach the session as given"""
monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
server = ScraplingMCPServer()
- opened = await server.browser_open()
-
- await server.browser_fetch(
- url="https://example.com/1", session_id=opened.session_id, timeout=45000, wait_selector="#main"
- )
-
- forwarded = self._fetch_call(_FakeStealthySession)
- assert forwarded["timeout"] == 45000
- assert forwarded["wait_selector"] == "#main"
+ options = {
+ "wait": 250,
+ "timeout": 45000,
+ "google_search": False,
+ "network_idle": True,
+ "load_dom": False,
+ "pierce_shadow": True,
+ "disable_resources": True,
+ "wait_selector": "#main",
+ "wait_selector_state": "visible",
+ "extra_headers": {"X-Test": "value"},
+ "blocked_domains": ["ads.example.com"],
+ "solve_cloudflare": True,
+ }
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ opened = await client.call_tool("browser_open", {"session_id": "browser"})
+ assert not opened.is_error
+ try:
+ fetched = await client.call_tool(
+ "browser_fetch", {"url": "https://example.com/1", "session_id": "browser", **options}
+ )
+ assert not fetched.is_error
+ assert self._fetch_call(_FakeStealthySession) == {**options, "blocked_domains": {"ads.example.com"}}
+ assert _FakeStealthySession.instances[0]._config.max_pages == 1
+ finally:
+ closed = await client.call_tool("close_session", {"session_id": "browser"})
+ assert not closed.is_error
+ assert not _FakeStealthySession.instances[0]._is_alive
@pytest.mark.asyncio
async def test_unknown_session_raises(self):
@@ -652,7 +818,7 @@ class TestBrowserFetchForwarding:
class TestModeSplitContract:
- """The one-shot vs session split is derived from the library TypedDicts and must stay in sync."""
+ """Session setup and per-request options must stay in sync with the library."""
def test_browser_fetch_signature_matches_the_derived_fetch_keys(self):
"""browser_fetch exposes exactly the stealth per-request keys (plus url/session_id/extraction trio)"""
@@ -679,14 +845,6 @@ class TestModeSplitContract:
for name, value in defaults.items():
assert value == library[name], f"browser_fetch {name} default {value!r} != library {library[name]!r}"
- def test_one_shot_fetch_tools_have_no_session_id(self):
- """The one-shot tools no longer accept session_id"""
- for tool in (
- ScraplingMCPServer.browser_fetch_once,
- ScraplingMCPServer.browser_fetch_many_once,
- ):
- assert "session_id" not in inspect.signature(tool).parameters, f"{tool.__name__} still takes session_id"
-
def test_browser_open_holds_no_per_request_params(self):
"""browser_open keeps browser-level params only, none of the per-request fetch keys"""
params = set(inspect.signature(ScraplingMCPServer.browser_open).parameters)
@@ -980,7 +1138,7 @@ class TestMCPServerAuthentication:
monkeypatch.delenv(MCP_AUTH_TOKEN_ENV, raising=False)
built = ScraplingMCPServer(auth_token=SHARED_KEY)._build_server("0.0.0.0", 8000)
- assert len(built._tool_manager.list_tools()) == 16
+ assert {tool.name for tool in built._tool_manager.list_tools()} == MCP_TOOLS
def test_http_without_a_token_refuses_to_serve(self, monkeypatch):
"""The streamable-http transport requires authentication unless the caller explicitly opts out"""
@@ -1037,13 +1195,15 @@ class TestServerToolRegistration:
@pytest.mark.asyncio
async def test_tools_are_listed_with_expected_schemas(self):
- """All 16 tools are advertised, with plain content for screenshots, snapshots, actions, and JavaScript"""
+ """Single-page and session tools retain their output schemas."""
server = ScraplingMCPServer()._build_server("127.0.0.1", 8000)
async with Client(server) as client:
assert client.instructions
assert "dynamic" not in client.instructions.lower()
tools = {tool.name: tool for tool in (await client.list_tools()).tools}
for name in (
+ "bulk_get",
+ "browser_fetch_many_once",
"browser_type",
"browser_fill_form",
"browser_fill_fields",
@@ -1067,7 +1227,8 @@ class TestServerToolRegistration:
result = await client.call_tool(name, {})
assert result.is_error
- assert len(tools) == 16
+ assert set(tools) == MCP_TOOLS
+ assert {"bulk_get", "browser_fetch_many_once"}.isdisjoint(tools)
assert tools["browser_screenshot"].output_schema is None
assert tools["browser_snapshot"].output_schema is None
assert tools["browser_actions"].output_schema is None
@@ -1094,10 +1255,6 @@ class TestServerToolRegistration:
"browser_stealth_fetch_once",
"browser_stealth_fetch_many_once",
}.isdisjoint(tools)
- assert {
- "browser_fetch_once",
- "browser_fetch_many_once",
- } <= tools.keys()
assert all(
tool.output_schema is not None
for name, tool in tools.items()
@@ -1111,55 +1268,113 @@ class TestServerToolRegistration:
)
@pytest.mark.asyncio
- async def test_fetch_tools_expose_real_defaults_and_no_session_id(self):
- """The one-shot tools show real defaults in their schema and no longer take session_id"""
+ async def test_fetch_tools_keep_single_url_session_schemas(self):
+ """Session tools keep one required URL and their current defaults."""
server = ScraplingMCPServer()._build_server("127.0.0.1", 8000)
async with Client(server) as client:
tools = {tool.name: tool for tool in (await client.list_tools()).tools}
- for name in (
- "browser_fetch_once",
- "browser_fetch_many_once",
- ):
+ for name in ("session_make_request", "browser_fetch"):
props = tools[name].input_schema["properties"]
- assert "session_id" not in props, f"{name} still exposes session_id"
- assert props["timeout"]["default"] == 30000, f"{name} hides the real timeout default"
- assert props["google_search"]["default"] is True
- assert props["pierce_shadow"]["type"] == "boolean"
- assert props["pierce_shadow"]["default"] is False
- for option in ("hide_canvas", "block_webrtc", "solve_cloudflare"):
- assert props[option]["type"] == "boolean"
- assert props[option]["default"] is False
- assert props["allow_webgl"]["default"] is True
- assert props["additional_args"]["default"] is None
- assert "snapshot" not in props["extraction_type"]["enum"]
-
- request_props = tools["make_request"].input_schema["properties"]
- assert request_props["method"]["default"] == "GET"
- assert "data" in request_props and "json" in request_props
- assert "method" not in tools["bulk_get"].input_schema["properties"]
+ assert props["url"]["type"] == "string"
+ assert "urls" not in props
+ assert set(tools[name].input_schema["required"]) >= {"url", "session_id"}
static_props = tools["session_make_request"].input_schema["properties"]
assert static_props["method"]["default"] == "GET"
+ assert "data" in static_props and "json" in static_props
assert "proxy" not in static_props and "impersonate" not in static_props, "session-level params leaked"
- assert set(tools["session_make_request"].input_schema["required"]) >= {"url", "session_id"}
assert set(tools["open_request_session"].input_schema["properties"]) == {"session_id", "impersonate", "proxy"}
session_props = tools["browser_fetch"].input_schema["properties"]
assert session_props["timeout"]["default"] == 30000
+ assert session_props["google_search"]["default"] is True
assert session_props["pierce_shadow"]["type"] == "boolean"
assert session_props["pierce_shadow"]["default"] is False
- assert "solve_cloudflare" in session_props
+ assert session_props["solve_cloudflare"]["type"] == "boolean"
+ assert session_props["solve_cloudflare"]["default"] is False
assert session_props["extraction_type"]["default"] == "markdown"
assert set(session_props["extraction_type"]["enum"]) == {"markdown", "html", "text", "snapshot"}
assert {"depth", "boxes"}.isdisjoint(session_props)
- assert set(tools["browser_fetch"].input_schema["required"]) >= {"url", "session_id"}
- open_props = set(tools["browser_open"].input_schema["properties"])
+ open_props = tools["browser_open"].input_schema["properties"]
+ for option in ("hide_canvas", "block_webrtc"):
+ assert open_props[option]["type"] == "boolean"
+ assert open_props[option]["default"] is False
+ assert open_props["allow_webgl"]["default"] is True
+ assert open_props["additional_args"]["default"] is None
+ open_props = set(open_props)
assert open_props.isdisjoint(_STEALTH_FETCH_KEYS), (
f"browser_open still exposes per-request params: {open_props & set(_STEALTH_FETCH_KEYS)}"
)
+ @pytest.mark.asyncio
+ async def test_one_shot_schemas_keep_original_options_and_defaults(self):
+ async with Client(ScraplingMCPServer()._build_server("127.0.0.1", 8000)) as client:
+ tools = {tool.name: tool for tool in (await client.list_tools()).tools}
+ for name in ("make_request", "browser_fetch_once"):
+ props = tools[name].input_schema["properties"]
+ assert props["url"]["type"] == "string"
+ assert {"urls", "session_id"}.isdisjoint(props)
+ assert tools[name].input_schema["required"] == ["url"]
+ assert props["extraction_type"]["default"] == "markdown"
+ assert set(props["extraction_type"]["enum"]) == {"markdown", "html", "text"}
+ assert tools[name].output_schema == tools["session_make_request"].output_schema
+ request_props = tools["make_request"].input_schema["properties"]
+ assert request_props["method"]["default"] == "GET"
+ assert {"data", "json", "proxy", "impersonate"} <= request_props.keys()
+ assert request_props["timeout"]["default"] == 30
+ assert request_props["follow_redirects"]["default"] == "safe"
+ props = tools["browser_fetch_once"].input_schema["properties"]
+ assert props["timeout"]["default"] == 30000
+ assert props["google_search"]["default"] is True
+ for option in ("pierce_shadow", "hide_canvas", "block_webrtc", "solve_cloudflare"):
+ assert props[option]["type"] == "boolean"
+ assert props[option]["default"] is False
+ assert props["allow_webgl"]["default"] is True
+ assert props["additional_args"]["default"] is None
+ assert not hasattr(ScraplingMCPServer, "bulk_get")
+ assert not hasattr(ScraplingMCPServer, "browser_fetch_many_once")
+
+ @pytest.mark.asyncio
+ @pytest.mark.parametrize("tool", ["make_request", "browser_fetch_once"])
+ @pytest.mark.parametrize("args", [{"url": ["https://example.com"]}, {"urls": ["https://example.com"]}])
+ async def test_one_shot_tools_reject_batches_before_opening_sessions(self, monkeypatch, tool, args):
+ static = Mock()
+ browser = Mock()
+ monkeypatch.setattr("scrapling.core.ai.FetcherSession", static)
+ monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", browser)
+ async with Client(ScraplingMCPServer()._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool(tool, args)
+ assert result.is_error
+ assert result.content and isinstance(result.content[0], TextContent)
+ assert "url" in result.content[0].text
+ static.assert_not_called()
+ browser.assert_not_called()
+
+ @pytest.mark.asyncio
+ @pytest.mark.parametrize("tool, session_type", [("browser_fetch", "stealthy"), ("session_make_request", "static")])
+ @pytest.mark.parametrize(
+ "args, field",
+ [
+ ({"url": ["https://example.com/1", "https://example.com/2"], "session_id": "test"}, "url"),
+ ({"urls": ["https://example.com/1", "https://example.com/2"], "session_id": "test"}, "url"),
+ ({"url": "https://example.com/1"}, "session_id"),
+ ],
+ )
+ async def test_session_fetch_validation_prevents_batches_and_missing_sessions(
+ self, tool, session_type, args, field
+ ):
+ server = ScraplingMCPServer()
+ session = Mock(_is_alive=True)
+ server._sessions["test"] = _SessionEntry(session, session_type)
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool(tool, args)
+ assert result.is_error
+ assert result.content and isinstance(result.content[0], TextContent)
+ assert field in result.content[0].text
+ assert session.mock_calls == []
+
@pytest.mark.asyncio
async def test_server_metadata_and_tool_annotations(self):
"""Server card metadata, cache hints, and tool annotations are advertised to clients"""
@@ -1175,12 +1390,10 @@ class TestServerToolRegistration:
assert result.ttl_ms == 3_600_000 and result.cache_scope == "public"
annotations = {tool.name: tool.annotations for tool in result.tools if tool.annotations is not None}
- assert len(annotations) == 16
+ assert set(annotations) == MCP_TOOLS
for name in (
"make_request",
- "bulk_get",
"browser_fetch_once",
- "browser_fetch_many_once",
"browser_fetch",
"session_make_request",
"browser_snapshot",
@@ -1201,23 +1414,6 @@ class TestServerToolRegistration:
assert annotations[name].open_world_hint is True
[email protected]("enabled", [False, True])
- "tool",
- ["browser_fetch_once", "browser_fetch_many_once"],
-)
-async def test_shadow_option_reaches_browser_session(monkeypatch, tool, enabled):
- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
- server = ScraplingMCPServer()._build_server("127.0.0.1", 8000)
- args: dict[str, Any] = {"urls": ["https://example.com"]} if "_many_" in tool else {"url": "https://example.com"}
- async with Client(server) as client:
- result = await client.call_tool(tool, {**args, "pierce_shadow": enabled})
- assert not result.is_error
- assert _FakeStealthySession.instances[-1].kwargs["pierce_shadow"] is enabled
- assert not _FakeStealthySession.instances[-1]._is_alive
-
-
@pytest.mark.asyncio
async def test_shadow_option_changes_per_session_request(monkeypatch):
monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
@@ -1235,3 +1431,15 @@ async def test_shadow_option_changes_per_session_request(monkeypatch):
finally:
await client.call_tool("close_session", {"session_id": "shadow-test"})
assert not _FakeStealthySession.instances[-1]._is_alive
+
+
[email protected]("enabled", [False, True])
+async def test_shadow_option_reaches_browser_session(monkeypatch, enabled):
+ monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
+ server = ScraplingMCPServer()._build_server("127.0.0.1", 8000)
+ async with Client(server) as client:
+ result = await client.call_tool("browser_fetch_once", {"url": "https://example.com", "pierce_shadow": enabled})
+ assert not result.is_error
+ assert _FakeStealthySession.instances[-1].kwargs["pierce_shadow"] is enabled
+ assert not _FakeStealthySession.instances[-1]._is_alive