Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

feat(mcp)!: rename `get` to `make_request` and add more methods.

Karim shoair1181fc6a1f1c
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zip
index 7fbc2c1..9af98ac 100644
Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differ
diff --git a/agent-skill/Scrapling-Skill/references/mcp-server.md b/agent-skill/Scrapling-Skill/references/mcp-server.md
index 37835dd..33fe2f1 100644
--- a/agent-skill/Scrapling-Skill/references/mcp-server.md
+++ b/agent-skill/Scrapling-Skill/references/mcp-server.md
@@ -6,15 +6,18 @@ All scraping tools return a `ResponseModel` with fields: `status` (int), `conten
## One-shot tools
-### `get` -- HTTP request (single URL)
+### `make_request` -- HTTP request, any method (single URL)
-Fast HTTP GET with browser fingerprint impersonation (TLS, headers). Suitable for static pages with no/low bot protection.
+Fast HTTP request with browser fingerprint impersonation (TLS, headers). Supports GET (default), POST, PUT, and DELETE via the `method` parameter. Suitable for static pages with no/low bot protection.
**Key parameters:**
| Parameter | Type | Default | Description |
|---------------------|------------------------------------|--------------|--------------------------------------------------------------------|
| `url` | str | required | URL to fetch |
+| `method` | `"GET"` / `"POST"` / `"PUT"` / `"DELETE"` | `"GET"` | HTTP method |
+| `data` | dict or str or null | null | Request body (form data). POST/PUT/DELETE only |
+| `json` | dict or list or null | null | Request body (JSON). POST/PUT/DELETE only |
| `extraction_type` | `"markdown"` / `"html"` / `"text"` | `"markdown"` | Output format |
| `css_selector` | str or null | null | CSS selector to narrow content (applied after `main_content_only`) |
| `main_content_only` | bool | true | Restrict to `<body>` content |
@@ -34,9 +37,9 @@ Fast HTTP GET with browser fingerprint impersonation (TLS, headers). Suitable fo
| `params` | dict or null | null | Query string parameters |
| `verify` | bool | true | Verify HTTPS certificates |
-### `bulk_get` -- HTTP request (multiple URLs)
+### `bulk_get` -- HTTP GET request (multiple URLs)
-Async concurrent version of `get`. Same parameters except `url` is replaced by `urls` (list of strings). All URLs are fetched in parallel. Returns a list of `ResponseModel`.
+Async concurrent GET-only version of `make_request`. Same parameters except `url` is replaced by `urls` (list of strings) and there are no `method`/`data`/`json` parameters. All URLs are fetched in parallel. Returns a list of `ResponseModel`.
### `fetch` -- Browser fetch (single URL)
@@ -110,7 +113,7 @@ Opens a browser session that stays alive across multiple `session_fetch` calls,
| `block_webrtc` | bool | false | (Stealthy only) Block WebRTC IP leak |
| `allow_webgl` | bool | true | (Stealthy only) Keep WebGL enabled |
-Plus the other browser-level session parameters (`real_chrome`, `cdp_url`, `locale`, `timezone_id`, `useragent`, `cookies`, `executable_path`, `additional_args`). Per-request options (`timeout`, `wait`, `google_search`, `network_idle`, `disable_resources`, `wait_selector`, `wait_selector_state`, `extra_headers`, `proxy`, `solve_cloudflare`) are not set here; pass them to `session_fetch`.
+Plus the other browser-level session parameters (`proxy`, `real_chrome`, `cdp_url`, `locale`, `timezone_id`, `useragent`, `cookies`, `executable_path`, `additional_args`). Per-request options (`timeout`, `wait`, `google_search`, `network_idle`, `disable_resources`, `wait_selector`, `wait_selector_state`, `extra_headers`, `solve_cloudflare`) are not set here; pass them to `session_fetch`.
One `session_fetch` works with either session type; `solve_cloudflare` only applies to a stealthy session.
@@ -135,7 +138,6 @@ Fetches one URL through a session opened with `open_session` (dynamic or stealth
| `wait_selector_state` | str | `"attached"` | State for wait_selector: `"attached"` / `"visible"` / `"hidden"` / `"detached"` |
| `extra_headers` | dict or null | null | Additional request headers |
| `blocked_domains` | list or null | null | Domain names to block for this request (subdomains matched too) |
-| `proxy` | str or dict or null | null | Proxy for this request |
| `solve_cloudflare` | bool | false | (Stealthy sessions only) Auto-solve Cloudflare challenges; errors on a dynamic session |
### `close_session` -- Close a persistent browser session
@@ -177,8 +179,8 @@ Requires an open browser session. Call `open_session` first, then pass the `sess
| Scenario | Tool |
|------------------------------------------|---------------------------------------------------------------|
-| Static page, no bot protection | `get` |
-| Multiple static pages | `bulk_get` |
+| Static page, no bot protection | `make_request` |
+| Multiple static pages | `bulk_get` |
| JavaScript-rendered / SPA page | `fetch` |
| Multiple JS-rendered pages | `bulk_fetch` |
| Cloudflare or strong anti-bot protection | `stealthy_fetch` (with `solve_cloudflare=true` for Turnstile) |
@@ -186,7 +188,7 @@ Requires an open browser session. Call `open_session` first, then pass the `sess
| Multiple pages from the same site | `open_session` + `session_fetch` per page |
| Need a screenshot of a page | `open_session` + `screenshot` with `session_id` |
-Start with `get` (fastest, lowest resource cost). Escalate to `fetch` if content requires JS rendering. Escalate to `stealthy_fetch` only if blocked. For multiple pages from the same site, use a persistent session to avoid browser launch overhead.
+Start with `make_request` (fastest, lowest resource cost). Escalate to `fetch` if content requires JS rendering. Escalate to `stealthy_fetch` only if blocked. For multiple pages from the same site, use a persistent session to avoid browser launch overhead.
## Content extraction tips
diff --git a/docs/ai/mcp-server.md b/docs/ai/mcp-server.md
index fd3c9ee..610d79b 100644
--- a/docs/ai/mcp-server.md
+++ b/docs/ai/mcp-server.md
@@ -11,8 +11,8 @@ The Scrapling MCP Server provides eleven powerful tools for web scraping, split
### One-shot tools
#### 🚀 Basic HTTP Scraping
-- **`get`**: Fast HTTP requests with browser fingerprint impersonation, generating real browser headers matching the TLS version, HTTP/3, and more!
-- **`bulk_get`**: An async version of the above tool that allows scraping of multiple URLs at the same time!
+- **`make_request`**: Fast HTTP requests with any method (GET, POST, PUT, DELETE) and browser fingerprint impersonation, generating real browser headers matching the TLS version, HTTP/3, and more!
+- **`bulk_get`**: An async GET-only version of the above tool that allows scraping of multiple URLs at the same time!
#### 🌐 Dynamic Content Scraping
- **`fetch`**: Rapidly fetch dynamic content with Chromium/Chrome browser with complete control over the request/browser, and more!
@@ -290,7 +290,7 @@ We will gradually go from simple prompts to more complex ones. We will use Claud
Scrape the main content from https://example.com and convert it to markdown format.
```
- Claude will use the `get` tool to fetch the page and return clean, readable content. If it fails, it will continue retrying every second for 3 attempts, unless you instruct it otherwise. If it fails to retrieve content for any reason, such as protection or if it's a dynamic website, it will automatically try the other tools. If Claude didn't do that automatically for some reason, you can add that to the prompt.
+ Claude will use the `make_request` tool to fetch the page and return clean, readable content. If it fails, it will continue retrying every second for 3 attempts, unless you instruct it otherwise. If it fails to retrieve content for any reason, such as protection or if it's a dynamic website, it will automatically try the other tools. If Claude didn't do that automatically for some reason, you can add that to the prompt.
A more optimized version of the same prompt would be:
```
@@ -404,7 +404,7 @@ And so on, you get the idea. Your creativity is the key here.
Here is some technical advice for you.
### 1. Choose the Right Tool
-- **`get`**: Fast, simple websites
+- **`make_request`**: Fast, simple websites
- **`fetch`**: Sites with JavaScript/dynamic content
- **`stealthy_fetch`**: Protected sites, Cloudflare, anti-bot systems
diff --git a/scrapling/core/ai.py b/scrapling/core/ai.py
index 44dea81..ca36c8a 100644
--- a/scrapling/core/ai.py
+++ b/scrapling/core/ai.py
@@ -40,6 +40,7 @@ from scrapling.core._types import (
extraction_types,
SelectorWaitStates,
FollowRedirects,
+ SUPPORTED_HTTP_METHODS,
)
SessionType = Literal["dynamic", "stealthy"]
@@ -378,13 +379,16 @@ class ScraplingMCPServer:
return [image, TextContent(type="text", text=captured["url"])]
@staticmethod
- async def get(
+ async def make_request(
url: str,
+ method: SUPPORTED_HTTP_METHODS = "GET",
impersonate: ImpersonateType = "chrome",
extraction_type: extraction_types = "markdown",
css_selector: Optional[str] = None,
main_content_only: bool = True,
params: Optional[Dict] = None,
+ data: Optional[Dict[str, str] | str] = None,
+ json: Optional[Dict | List] = None,
headers: Optional[Mapping[str, Optional[str]]] = None,
cookies: Optional[Dict[str, str]] = None,
timeout: Optional[int | float] = 30,
@@ -399,15 +403,18 @@ class ScraplingMCPServer:
http3: Optional[bool] = False,
stealthy_headers: Optional[bool] = True,
) -> ResponseModel:
- """Make GET HTTP request to a URL and return a structured output of the result.
+ """Make an HTTP request to a URL with any method (GET, POST, PUT, DELETE) and return a structured output of the result.
Only suitable for low-mid protection levels.
:param url: The URL to request.
+ :param method: The HTTP method to use: "GET" (default), "POST", "PUT", or "DELETE".
:param impersonate: Browser version to impersonate its fingerprint. It's using the latest chrome version by default.
:param extraction_type: The type of content to extract from the page: "markdown", "html", or "text".
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page.
:param main_content_only: Whether to extract only the main content of the page. The main content here is the data inside the `<body>` tag.
:param params: Query string parameters for the request.
+ :param data: Form data for the request body. Used with "POST", "PUT", and "DELETE" only.
+ :param json: A JSON-serializable object for the request body. Used with "POST", "PUT", and "DELETE" only.
:param headers: Headers to include in the request.
:param cookies: Cookies to use in the request.
:param timeout: Number of seconds to wait before timing out.
@@ -424,28 +431,32 @@ class ScraplingMCPServer:
:param http3: Whether to use HTTP3. It might be problematic if used it with `impersonate`.
:param stealthy_headers: If enabled (default), it creates and adds real browser headers. It also sets a Google referer header.
"""
- results = await ScraplingMCPServer.bulk_get(
- urls=[url],
- impersonate=impersonate,
- extraction_type=extraction_type,
- css_selector=css_selector,
- main_content_only=main_content_only,
+ normalized_proxy_auth = _normalize_credentials(proxy_auth)
+ normalized_auth = _normalize_credentials(auth)
+
+ request_kwargs: Dict[str, Any] = dict(
+ auth=normalized_auth,
+ proxy=proxy,
+ http3=http3,
+ verify=verify,
params=params,
headers=headers,
cookies=cookies,
timeout=timeout,
- follow_redirects=follow_redirects,
- max_redirects=max_redirects,
retries=retries,
+ proxy_auth=normalized_proxy_auth,
retry_delay=retry_delay,
- proxy=proxy,
- proxy_auth=proxy_auth,
- auth=auth,
- verify=verify,
- http3=http3,
+ impersonate=impersonate,
+ max_redirects=max_redirects,
+ follow_redirects=follow_redirects,
stealthy_headers=stealthy_headers,
)
- return results[0]
+ if method != "GET":
+ request_kwargs.update(data=data, json=json)
+
+ async with FetcherSession() as session:
+ page = await getattr(session, method.lower())(url, **request_kwargs)
+ return _translate_response(page, extraction_type, css_selector, main_content_only)
@staticmethod
async def bulk_get(
@@ -950,7 +961,7 @@ class ScraplingMCPServer:
"cache_hints": {"tools/list": CacheHint(ttl_ms=3_600_000, scope="public")},
"instructions": """Follow these instructions precisely:
1. When the `open_session` tool is used, make sure to close the session with `close_session` after you finish, and use `list_sessions` if you lose track of the open sessions or their effective settings.
-2. If the user didn't specify which tool to use, start with the `get` tool, then escalate. The `get` tool and the bulk version are suitable only for low-to-mid protection levels.
+2. If the user didn't specify which tool to use, start with the `make_request` tool (a plain HTTP request, defaulting to GET; set `method` for POST/PUT/DELETE), then escalate. The `make_request` tool and `bulk_get` (its GET-only bulk version) are suitable only for low-to-mid protection levels.
For high-protection levels or websites that require JS loading, use the other tools directly.
3. For all tools, if the `css_selector` resolves to more than one element, all the elements will be returned.
4. For all fetch tools, the `extraction_type` parameter controls the format of the returned content: "markdown" (default) converts the page content to Markdown, "html" returns the raw HTML, and "text" returns the text content of the page.
@@ -981,9 +992,9 @@ class ScraplingMCPServer:
)
# HTTP tools
server.add_tool(
- self.get,
- title="get",
- description=self.get.__doc__,
+ self.make_request,
+ title="make_request",
+ description=self.make_request.__doc__,
structured_output=True,
annotations=_FETCH_TOOL_ANNOTATIONS,
)
diff --git a/tests/ai/test_ai_mcp.py b/tests/ai/test_ai_mcp.py
index 65d5d36..4542968 100644
--- a/tests/ai/test_ai_mcp.py
+++ b/tests/ai/test_ai_mcp.py
@@ -34,7 +34,7 @@ from scrapling.fetchers import AsyncDynamicSession, AsyncStealthySession
def test_translate_response_strips_control_characters():
- """Pages with control chars like U+0008 must not crash the get/fetch path (issue #366)"""
+ """Pages with control chars like U+0008 must not crash the request/fetch path (issue #366)"""
html = "<html><body><p>Hello\x08World</p>\t\n<div>Foo\x0cbar</div></body></html>"
page = Response(
url="https://jfinal.com/doc/1-5",
@@ -126,13 +126,22 @@ class TestMCPServer:
return ScraplingMCPServer()
@pytest.mark.asyncio
- async def test_get_tool(self, server, test_url):
- """Test the get tool method"""
- result = await server.get(url=test_url, extraction_type="markdown")
+ async def test_make_request_tool(self, server, test_url):
+ """Test the make_request tool method with a default GET"""
+ result = await server.make_request(url=test_url, extraction_type="markdown")
assert isinstance(result, ResponseModel)
assert result.status == 200
assert result.url == test_url
+ @pytest.mark.asyncio
+ async def test_make_request_post_tool(self, server, httpbin):
+ """Test the make_request tool method with a POST body"""
+ result = await server.make_request(
+ url=f"{httpbin.url}/post", method="POST", json={"key": "value"}, extraction_type="text"
+ )
+ assert isinstance(result, ResponseModel)
+ assert result.status == 200
+
@pytest.mark.asyncio
async def test_bulk_get_tool(self, server, test_url):
"""Test the bulk_get tool method"""
@@ -422,9 +431,9 @@ class TestSessionFetchForwarding:
"wait_selector_state": "attached",
"extra_headers": None,
"blocked_domains": None,
- "proxy": None,
}, forwarded
assert "solve_cloudflare" not in forwarded, "solve_cloudflare must not reach a dynamic session"
+ assert "proxy" not in forwarded, "proxy is session-level (open_session), never forwarded per request"
@pytest.mark.asyncio
async def test_stealthy_session_fetch_forwards_solve_cloudflare(self, monkeypatch):
@@ -515,6 +524,23 @@ class TestModeSplitContract:
f"open_session still carries per-request params: {params & set(_STEALTH_FETCH_KEYS)}"
)
+ def test_proxy_is_session_level_not_per_request(self):
+ """A session runs one tab, so proxy is set once on open_session, never per request"""
+ assert "proxy" in inspect.signature(ScraplingMCPServer.open_session).parameters
+ assert "proxy" not in inspect.signature(ScraplingMCPServer.session_fetch).parameters
+ assert "proxy" not in _STEALTH_FETCH_KEYS
+
+ @pytest.mark.asyncio
+ async def test_open_session_forwards_proxy_to_the_session(self, monkeypatch):
+ """The session-level proxy reaches the underlying session so it applies to every fetch"""
+ monkeypatch.setattr("scrapling.core.ai.AsyncDynamicSession", _FakeDynamicSession)
+ _FakeDynamicSession.instances = []
+ server = ScraplingMCPServer()
+
+ await server.open_session(session_type="dynamic", proxy="http://user:pass@host:8080")
+
+ assert _FakeDynamicSession.instances[0].kwargs["proxy"] == "http://user:pass@host:8080"
+
class TestSessionSettingsReceipt:
"""open_session and list_sessions return the session's effective settings."""
@@ -822,6 +848,11 @@ class TestServerToolRegistration:
assert props["timeout"]["default"] == 30000, f"{name} hides the real timeout default"
assert props["google_search"]["default"] is True
+ request_props = tools["make_request"].input_schema["properties"]
+ assert request_props["method"]["default"] == "GET"
+ assert "data" in request_props and "json" in request_props
+ assert "method" not in tools["bulk_get"].input_schema["properties"]
+
session_props = tools["session_fetch"].input_schema["properties"]
assert session_props["timeout"]["default"] == 30000
assert "solve_cloudflare" in session_props
@@ -849,7 +880,7 @@ class TestServerToolRegistration:
annotations = {tool.name: tool.annotations for tool in result.tools if tool.annotations is not None}
assert len(annotations) == 11
for name in (
- "get",
+ "make_request",
"bulk_get",
"fetch",
"bulk_fetch",