🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
feat(mcp)!: rename `get` to `make_request` and add more methods.
Karim shoair1181fc6a1f1c
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zipindex 7fbc2c1..9af98ac 100644Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differdiff --git a/agent-skill/Scrapling-Skill/references/mcp-server.md b/agent-skill/Scrapling-Skill/references/mcp-server.mdindex 37835dd..33fe2f1 100644--- a/agent-skill/Scrapling-Skill/references/mcp-server.md+++ b/agent-skill/Scrapling-Skill/references/mcp-server.md@@ -6,15 +6,18 @@ All scraping tools return a `ResponseModel` with fields: `status` (int), `conten## One-shot tools-### `get` -- HTTP request (single URL)+### `make_request` -- HTTP request, any method (single URL)-Fast HTTP GET with browser fingerprint impersonation (TLS, headers). Suitable for static pages with no/low bot protection.+Fast HTTP request with browser fingerprint impersonation (TLS, headers). Supports GET (default), POST, PUT, and DELETE via the `method` parameter. Suitable for static pages with no/low bot protection.**Key parameters:**| Parameter | Type | Default | Description ||---------------------|------------------------------------|--------------|--------------------------------------------------------------------|| `url` | str | required | URL to fetch |+| `method` | `"GET"` / `"POST"` / `"PUT"` / `"DELETE"` | `"GET"` | HTTP method |+| `data` | dict or str or null | null | Request body (form data). POST/PUT/DELETE only |+| `json` | dict or list or null | null | Request body (JSON). POST/PUT/DELETE only || `extraction_type` | `"markdown"` / `"html"` / `"text"` | `"markdown"` | Output format || `css_selector` | str or null | null | CSS selector to narrow content (applied after `main_content_only`) || `main_content_only` | bool | true | Restrict to `<body>` content |@@ -34,9 +37,9 @@ Fast HTTP GET with browser fingerprint impersonation (TLS, headers). Suitable fo| `params` | dict or null | null | Query string parameters || `verify` | bool | true | Verify HTTPS certificates |-### `bulk_get` -- HTTP request (multiple URLs)+### `bulk_get` -- HTTP GET request (multiple URLs)-Async concurrent version of `get`. Same parameters except `url` is replaced by `urls` (list of strings). All URLs are fetched in parallel. Returns a list of `ResponseModel`.+Async concurrent GET-only version of `make_request`. Same parameters except `url` is replaced by `urls` (list of strings) and there are no `method`/`data`/`json` parameters. All URLs are fetched in parallel. Returns a list of `ResponseModel`.### `fetch` -- Browser fetch (single URL)@@ -110,7 +113,7 @@ Opens a browser session that stays alive across multiple `session_fetch` calls,| `block_webrtc` | bool | false | (Stealthy only) Block WebRTC IP leak || `allow_webgl` | bool | true | (Stealthy only) Keep WebGL enabled |-Plus the other browser-level session parameters (`real_chrome`, `cdp_url`, `locale`, `timezone_id`, `useragent`, `cookies`, `executable_path`, `additional_args`). Per-request options (`timeout`, `wait`, `google_search`, `network_idle`, `disable_resources`, `wait_selector`, `wait_selector_state`, `extra_headers`, `proxy`, `solve_cloudflare`) are not set here; pass them to `session_fetch`.+Plus the other browser-level session parameters (`proxy`, `real_chrome`, `cdp_url`, `locale`, `timezone_id`, `useragent`, `cookies`, `executable_path`, `additional_args`). Per-request options (`timeout`, `wait`, `google_search`, `network_idle`, `disable_resources`, `wait_selector`, `wait_selector_state`, `extra_headers`, `solve_cloudflare`) are not set here; pass them to `session_fetch`.One `session_fetch` works with either session type; `solve_cloudflare` only applies to a stealthy session.@@ -135,7 +138,6 @@ Fetches one URL through a session opened with `open_session` (dynamic or stealth| `wait_selector_state` | str | `"attached"` | State for wait_selector: `"attached"` / `"visible"` / `"hidden"` / `"detached"` || `extra_headers` | dict or null | null | Additional request headers || `blocked_domains` | list or null | null | Domain names to block for this request (subdomains matched too) |-| `proxy` | str or dict or null | null | Proxy for this request || `solve_cloudflare` | bool | false | (Stealthy sessions only) Auto-solve Cloudflare challenges; errors on a dynamic session |### `close_session` -- Close a persistent browser session@@ -177,8 +179,8 @@ Requires an open browser session. Call `open_session` first, then pass the `sess| Scenario | Tool ||------------------------------------------|---------------------------------------------------------------|-| Static page, no bot protection | `get` |-| Multiple static pages | `bulk_get` |+| Static page, no bot protection | `make_request` |+| Multiple static pages | `bulk_get` || JavaScript-rendered / SPA page | `fetch` || Multiple JS-rendered pages | `bulk_fetch` || Cloudflare or strong anti-bot protection | `stealthy_fetch` (with `solve_cloudflare=true` for Turnstile) |@@ -186,7 +188,7 @@ Requires an open browser session. Call `open_session` first, then pass the `sess| Multiple pages from the same site | `open_session` + `session_fetch` per page || Need a screenshot of a page | `open_session` + `screenshot` with `session_id` |-Start with `get` (fastest, lowest resource cost). Escalate to `fetch` if content requires JS rendering. Escalate to `stealthy_fetch` only if blocked. For multiple pages from the same site, use a persistent session to avoid browser launch overhead.+Start with `make_request` (fastest, lowest resource cost). Escalate to `fetch` if content requires JS rendering. Escalate to `stealthy_fetch` only if blocked. For multiple pages from the same site, use a persistent session to avoid browser launch overhead.## Content extraction tipsdiff --git a/docs/ai/mcp-server.md b/docs/ai/mcp-server.mdindex fd3c9ee..610d79b 100644--- a/docs/ai/mcp-server.md+++ b/docs/ai/mcp-server.md@@ -11,8 +11,8 @@ The Scrapling MCP Server provides eleven powerful tools for web scraping, split### One-shot tools#### 🚀 Basic HTTP Scraping-- **`get`**: Fast HTTP requests with browser fingerprint impersonation, generating real browser headers matching the TLS version, HTTP/3, and more!-- **`bulk_get`**: An async version of the above tool that allows scraping of multiple URLs at the same time!+- **`make_request`**: Fast HTTP requests with any method (GET, POST, PUT, DELETE) and browser fingerprint impersonation, generating real browser headers matching the TLS version, HTTP/3, and more!+- **`bulk_get`**: An async GET-only version of the above tool that allows scraping of multiple URLs at the same time!#### 🌐 Dynamic Content Scraping- **`fetch`**: Rapidly fetch dynamic content with Chromium/Chrome browser with complete control over the request/browser, and more!@@ -290,7 +290,7 @@ We will gradually go from simple prompts to more complex ones. We will use ClaudScrape the main content from https://example.com and convert it to markdown format.```- Claude will use the `get` tool to fetch the page and return clean, readable content. If it fails, it will continue retrying every second for 3 attempts, unless you instruct it otherwise. If it fails to retrieve content for any reason, such as protection or if it's a dynamic website, it will automatically try the other tools. If Claude didn't do that automatically for some reason, you can add that to the prompt.+ Claude will use the `make_request` tool to fetch the page and return clean, readable content. If it fails, it will continue retrying every second for 3 attempts, unless you instruct it otherwise. If it fails to retrieve content for any reason, such as protection or if it's a dynamic website, it will automatically try the other tools. If Claude didn't do that automatically for some reason, you can add that to the prompt.A more optimized version of the same prompt would be:```@@ -404,7 +404,7 @@ And so on, you get the idea. Your creativity is the key here.Here is some technical advice for you.### 1. Choose the Right Tool-- **`get`**: Fast, simple websites+- **`make_request`**: Fast, simple websites- **`fetch`**: Sites with JavaScript/dynamic content- **`stealthy_fetch`**: Protected sites, Cloudflare, anti-bot systemsdiff --git a/scrapling/core/ai.py b/scrapling/core/ai.pyindex 44dea81..ca36c8a 100644--- a/scrapling/core/ai.py+++ b/scrapling/core/ai.py@@ -40,6 +40,7 @@ from scrapling.core._types import (extraction_types,SelectorWaitStates,FollowRedirects,+ SUPPORTED_HTTP_METHODS,)SessionType = Literal["dynamic", "stealthy"]@@ -378,13 +379,16 @@ class ScraplingMCPServer:return [image, TextContent(type="text", text=captured["url"])]@staticmethod- async def get(+ async def make_request(url: str,+ method: SUPPORTED_HTTP_METHODS = "GET",impersonate: ImpersonateType = "chrome",extraction_type: extraction_types = "markdown",css_selector: Optional[str] = None,main_content_only: bool = True,params: Optional[Dict] = None,+ data: Optional[Dict[str, str] | str] = None,+ json: Optional[Dict | List] = None,headers: Optional[Mapping[str, Optional[str]]] = None,cookies: Optional[Dict[str, str]] = None,timeout: Optional[int | float] = 30,@@ -399,15 +403,18 @@ class ScraplingMCPServer:http3: Optional[bool] = False,stealthy_headers: Optional[bool] = True,) -> ResponseModel:- """Make GET HTTP request to a URL and return a structured output of the result.+ """Make an HTTP request to a URL with any method (GET, POST, PUT, DELETE) and return a structured output of the result.Only suitable for low-mid protection levels.:param url: The URL to request.+ :param method: The HTTP method to use: "GET" (default), "POST", "PUT", or "DELETE".:param impersonate: Browser version to impersonate its fingerprint. It's using the latest chrome version by default.:param extraction_type: The type of content to extract from the page: "markdown", "html", or "text".:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page.:param main_content_only: Whether to extract only the main content of the page. The main content here is the data inside the `<body>` tag.:param params: Query string parameters for the request.+ :param data: Form data for the request body. Used with "POST", "PUT", and "DELETE" only.+ :param json: A JSON-serializable object for the request body. Used with "POST", "PUT", and "DELETE" only.:param headers: Headers to include in the request.:param cookies: Cookies to use in the request.:param timeout: Number of seconds to wait before timing out.@@ -424,28 +431,32 @@ class ScraplingMCPServer::param http3: Whether to use HTTP3. It might be problematic if used it with `impersonate`.:param stealthy_headers: If enabled (default), it creates and adds real browser headers. It also sets a Google referer header."""- results = await ScraplingMCPServer.bulk_get(- urls=[url],- impersonate=impersonate,- extraction_type=extraction_type,- css_selector=css_selector,- main_content_only=main_content_only,+ normalized_proxy_auth = _normalize_credentials(proxy_auth)+ normalized_auth = _normalize_credentials(auth)++ request_kwargs: Dict[str, Any] = dict(+ auth=normalized_auth,+ proxy=proxy,+ http3=http3,+ verify=verify,params=params,headers=headers,cookies=cookies,timeout=timeout,- follow_redirects=follow_redirects,- max_redirects=max_redirects,retries=retries,+ proxy_auth=normalized_proxy_auth,retry_delay=retry_delay,- proxy=proxy,- proxy_auth=proxy_auth,- auth=auth,- verify=verify,- http3=http3,+ impersonate=impersonate,+ max_redirects=max_redirects,+ follow_redirects=follow_redirects,stealthy_headers=stealthy_headers,)- return results[0]+ if method != "GET":+ request_kwargs.update(data=data, json=json)++ async with FetcherSession() as session:+ page = await getattr(session, method.lower())(url, **request_kwargs)+ return _translate_response(page, extraction_type, css_selector, main_content_only)@staticmethodasync def bulk_get(@@ -950,7 +961,7 @@ class ScraplingMCPServer:"cache_hints": {"tools/list": CacheHint(ttl_ms=3_600_000, scope="public")},"instructions": """Follow these instructions precisely:1. When the `open_session` tool is used, make sure to close the session with `close_session` after you finish, and use `list_sessions` if you lose track of the open sessions or their effective settings.-2. If the user didn't specify which tool to use, start with the `get` tool, then escalate. The `get` tool and the bulk version are suitable only for low-to-mid protection levels.+2. If the user didn't specify which tool to use, start with the `make_request` tool (a plain HTTP request, defaulting to GET; set `method` for POST/PUT/DELETE), then escalate. The `make_request` tool and `bulk_get` (its GET-only bulk version) are suitable only for low-to-mid protection levels.For high-protection levels or websites that require JS loading, use the other tools directly.3. For all tools, if the `css_selector` resolves to more than one element, all the elements will be returned.4. For all fetch tools, the `extraction_type` parameter controls the format of the returned content: "markdown" (default) converts the page content to Markdown, "html" returns the raw HTML, and "text" returns the text content of the page.@@ -981,9 +992,9 @@ class ScraplingMCPServer:)# HTTP toolsserver.add_tool(- self.get,- title="get",- description=self.get.__doc__,+ self.make_request,+ title="make_request",+ description=self.make_request.__doc__,structured_output=True,annotations=_FETCH_TOOL_ANNOTATIONS,)diff --git a/tests/ai/test_ai_mcp.py b/tests/ai/test_ai_mcp.pyindex 65d5d36..4542968 100644--- a/tests/ai/test_ai_mcp.py+++ b/tests/ai/test_ai_mcp.py@@ -34,7 +34,7 @@ from scrapling.fetchers import AsyncDynamicSession, AsyncStealthySessiondef test_translate_response_strips_control_characters():- """Pages with control chars like U+0008 must not crash the get/fetch path (issue #366)"""+ """Pages with control chars like U+0008 must not crash the request/fetch path (issue #366)"""html = "<html><body><p>Hello\x08World</p>\t\n<div>Foo\x0cbar</div></body></html>"page = Response(url="https://jfinal.com/doc/1-5",@@ -126,13 +126,22 @@ class TestMCPServer:return ScraplingMCPServer()@pytest.mark.asyncio- async def test_get_tool(self, server, test_url):- """Test the get tool method"""- result = await server.get(url=test_url, extraction_type="markdown")+ async def test_make_request_tool(self, server, test_url):+ """Test the make_request tool method with a default GET"""+ result = await server.make_request(url=test_url, extraction_type="markdown")assert isinstance(result, ResponseModel)assert result.status == 200assert result.url == test_url+ @pytest.mark.asyncio+ async def test_make_request_post_tool(self, server, httpbin):+ """Test the make_request tool method with a POST body"""+ result = await server.make_request(+ url=f"{httpbin.url}/post", method="POST", json={"key": "value"}, extraction_type="text"+ )+ assert isinstance(result, ResponseModel)+ assert result.status == 200+@pytest.mark.asyncioasync def test_bulk_get_tool(self, server, test_url):"""Test the bulk_get tool method"""@@ -422,9 +431,9 @@ class TestSessionFetchForwarding:"wait_selector_state": "attached","extra_headers": None,"blocked_domains": None,- "proxy": None,}, forwardedassert "solve_cloudflare" not in forwarded, "solve_cloudflare must not reach a dynamic session"+ assert "proxy" not in forwarded, "proxy is session-level (open_session), never forwarded per request"@pytest.mark.asyncioasync def test_stealthy_session_fetch_forwards_solve_cloudflare(self, monkeypatch):@@ -515,6 +524,23 @@ class TestModeSplitContract:f"open_session still carries per-request params: {params & set(_STEALTH_FETCH_KEYS)}")+ def test_proxy_is_session_level_not_per_request(self):+ """A session runs one tab, so proxy is set once on open_session, never per request"""+ assert "proxy" in inspect.signature(ScraplingMCPServer.open_session).parameters+ assert "proxy" not in inspect.signature(ScraplingMCPServer.session_fetch).parameters+ assert "proxy" not in _STEALTH_FETCH_KEYS++ @pytest.mark.asyncio+ async def test_open_session_forwards_proxy_to_the_session(self, monkeypatch):+ """The session-level proxy reaches the underlying session so it applies to every fetch"""+ monkeypatch.setattr("scrapling.core.ai.AsyncDynamicSession", _FakeDynamicSession)+ _FakeDynamicSession.instances = []+ server = ScraplingMCPServer()++ await server.open_session(session_type="dynamic", proxy="http://user:pass@host:8080")++ assert _FakeDynamicSession.instances[0].kwargs["proxy"] == "http://user:pass@host:8080"+class TestSessionSettingsReceipt:"""open_session and list_sessions return the session's effective settings."""@@ -822,6 +848,11 @@ class TestServerToolRegistration:assert props["timeout"]["default"] == 30000, f"{name} hides the real timeout default"assert props["google_search"]["default"] is True+ request_props = tools["make_request"].input_schema["properties"]+ assert request_props["method"]["default"] == "GET"+ assert "data" in request_props and "json" in request_props+ assert "method" not in tools["bulk_get"].input_schema["properties"]+session_props = tools["session_fetch"].input_schema["properties"]assert session_props["timeout"]["default"] == 30000assert "solve_cloudflare" in session_props@@ -849,7 +880,7 @@ class TestServerToolRegistration:annotations = {tool.name: tool.annotations for tool in result.tools if tool.annotations is not None}assert len(annotations) == 11for name in (- "get",+ "make_request","bulk_get","fetch","bulk_fetch",